"Salvaging psychotherapy research: a manifesto" by James C. Coyne (renegade psychologist)

[Dolphin]

Salvaging psychotherapy research: a manifesto
June 10

Sub-heading: Fueling Change in Psychotherapy Research with Greater Scrutiny and Public Accountability

http://blogs.plos.org/mindthebrain/2014/06/10/salvaging-psychotherapy-research-manifesto/

James C. Coyne is an influential psychologist who isn't afraid to highlight problems in psychological research.

It starts:

John Ioannidis’s declarations that most positive findings are false and that most breakthrough discoveries are exaggerated or fail to replicate apply have as much to with psychotherapy as they do with biomedicine.

We should take a few tips from Ben Goldacre’s Bad Pharma and clean up the psychotherapy literature, paralleling what is being accomplished with pharmaceutical trials. Sure, much remains to be done to ensure the quality and transparency of drug studies and to get all of the data into public view. But the psychotherapy literature lags far behind and is far less reliable than the pharmaceutical literature.

As it now stands, the psychotherapy literature does not provide a dependable guide to policy makers, clinicians, and consumers attempting to assess the relative costs and benefits of choosing a particular therapy over others. If such stakeholders uncritically depend upon the psychotherapy literature to evaluate the evidence-supported status of treatments, they will be confused or misled.

He makes lots of criticisms of the field, but of course, knowing about such problems is useful for understanding and critiquing any type of research.

 

[Dolphin]

A few more quotes:

There are low thresholds for professional groups such as the American Psychological Association Division 12 or governmental organizations such as the US Substance Abuse and Mental Health Services Administration (SAMHSA) declaring treatments to be “evidence-supported.” Seldom are any treatments deemed ineffective or harmful by these groups.

What would be disallowed as “off label applications” for pharmaceuticals is routinely accepted in psychotherapy workshops.

Psychotherapy research is conducted and evaluated within a club, a mutual admiration society in which members are careful not to disparage others’ results or enforce standards that they themselves might want relaxed when it comes to publishing their own research. There are rivalries between tribes like psychodynamic therapy and cognitive behavior therapy, but suppression of criticism within the tribes and in strenuous efforts to create the appearance that members of the tribes only do what works.

 

[Sean]

Psychotherapy research is conducted and evaluated within a club, a mutual admiration society in which members are careful not to disparage others’ results or enforce standards that they themselves might want relaxed when it comes to publishing their own research.

A mutual protection racket.

It all comes back to the lack of genuinely objective standards. Until they and their ideas are required to meet such standards then they can, and will, continue to muddy the waters and waste human lives by the truckload.

 

[alex3619]

These are things we have been saying for years, most notably about the PACE trial.

 

[alex3619]

Bad science in the evaluation of psychotherapy must be recognized as the current norm, not an anomaly.

Developers of treatments and persons with significant financial gain from a treatment being declared “evidence-supported” should be discouraged from conducting meta-analyses of their own treatments.

PDW's Cochrane review?

We need to shift the culture of doing and reporting psychotherapy research. We need to shift from praising exaggerated claims about treatment and faux evidence generated to promote opportunities for therapists and their professional organizations. Instead, it is much more praiseworthy to provide robust, sustainable, even if more modest claims and to call out hype and hokum in ways that preserve the credibility of psychotherapy.

The alternative is to continue protecting psychotherapy research from stringent criticism and enforcement of standards for conducting and reporting research. We can simply allow the branding of psychotherapies as “evidence supported” to fall into appropriate disrepute.

[My bolding]

I agree with these last points. Indeed I am working toward arguments that major culture shifts are required in areas way beyond even psychotherapy and psychogenic medicine. Yet it is in psychogenic medicine that the issues are most egregious.

 

[Sean]

Developers of treatments and persons with significant financial gain from a treatment being declared “evidence-supported” should be discouraged from conducting meta-analyses of their own treatments.

PDW's Cochrane review?

An outstanding example of how reviews/meta-analyses should not be done.

 

[chipmunk1]

The problem is psychologists make a living selling therapy. Do we expect them to publish research showing that their product is defective?

 

[Cheshire]

There are low thresholds for professional groups such as the American Psychological Association Division 12 or governmental organizations such as the US Substance Abuse and Mental Health Services Administration (SAMHSA) declaring treatments to be “evidence-supported.”:

Do anyone know what is the threshold (if such a threshold exists) above what a medication is considered as efficient or not? Would the 22% PACE trial “recovery” be sufficient in a pharmacology trial?

 

[alex3619]

Do anyone know what is the threshold (if such a threshold exists) above what a medication is considered as efficient or not? Would the 22% PACE trial “recovery” be sufficient in a pharmacology trial?

I think that effect size also matters.

 

[chipmunk1]

Do anyone know what is the threshold (if such a threshold exists) above what a medication is considered as efficient or not? Would the 22% PACE trial “recovery” be sufficient in a pharmacology trial?

A pharmacology trial would usually be double-blinded. The psychobabblers do usually avoid even single-blinded trials.

I am not sure if a medication could be considered efficient without proper controls and blinding.

 

[user9876]

Do anyone know what is the threshold (if such a threshold exists) above what a medication is considered as efficient or not? Would the 22% PACE trial “recovery” be sufficient in a pharmacology trial?

We know that they talk of 22% of (CBT|GET)+SMC but the SMC on its own has a recovery rate under their definition of 15% thus they are over claiming.

We know that they lowered the thresholds post hoc to a level where they are below the trial entry criteria. We don't know if the trial steering committee approved this change and if they did what evidence was presented to them. We also know the published reason for changing the threshold for the sf36 physical function score was wrong and we know that the journal publishing the PACE recovery paper have been told it is wrong and have chosen not to make a correction.

 

[Cheshire]

We know that they talk of 22% of (CBT|GET)+SMC but the SMC on its own has a recovery rate under their definition of 15% thus they are over claiming.

We know that they lowered the thresholds post hoc to a level where they are below the trial entry criteria. We don't know if the trial steering committee approved this change and if they did what evidence was presented to them. We also know the published reason for changing the threshold for the sf36 physical function score was wrong and we know that the journal publishing the PACE recovery paper have been told it is wrong and have chosen not to make a correction.

I perfectly know that, that's why I used " ". Maybe I wasn't sufficiently explicit. I meant: "Even with their overexagerated recovery results do they meet basic pharmacological trial standards?"

 

[chipmunk1]

. I meant: "Even with their overexagerated recovery results do they meet basic pharmacological trial standards?"

Of course they don't. it's very hard to do.

I have been looking for CBT trials that were as close as possible to pharmacological trail standards and so far i haven't found any but one that found CBT was ineffective for most conditions they are attempting to treat.

if someone knows of reliable studies please let me know.

 

[alex3619]

A pharmacology trial would usually be double-blinded. The psychobabblers do usually avoid even single-blinded trials.

I am not sure if a medication could be considered efficient without proper controls and blinding.

How do you blind a patient to psychotherapy? Its chalk and cheese. Inability to blind psychotherapy is one of the reasons that so much of this research is unreliable, and why these studies, even if using RCT format in other ways, are not the highest grade of evidence.

 

[alex3619]

On PACE, isn't it also the case if what I hear about FOI information is correct, that over half of the recovered patients started that way, at least in terms of SF-36? Maybe someone who knows more about that can comment? If you halve that figure, say six percent, and add it to fifteen percent, you get .... twenty one percent. We need all the data from the trials published. It looks very much like a total non-result, a proof of the null hypothesis, is being spun as a massive success.

 

[Sean]

On PACE, isn't it also the case if what I hear about FOI information is correct, that over half of the recovered patients started that way, at least in terms of SF-36?

If true, that would be a serious blow against the already weak PACE results.

 

[alex3619]

If true, that would be a serious blow against the already weak PACE results.

That is why I want it confirmed if I can.

 

[alex3619]

There is a thread on this here that I had forgotten: http://forums.phoenixrising.me/inde...ranges-for-cfq-and-sf36-pf-at-baseline.22366/

I hope to comment more later.

 

[Dolphin]

An open-access published paper based on this has now been published. See: http://forums.phoenixrising.me/inde...rch-a-manifesto-published-article-coyn.33705/