• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

"Salvaging psychotherapy research: a manifesto" (published article) (Coyn

Dolphin

Senior Member
Messages
17,567
The full text of this is free here: http://jebp.psychotherapy.ro/wp-content/uploads/2014/09/James-C.-Coyne-Robin-N.-Kok.pdf

Coyne is a renegade psychologist who is not afraid to challenge other psychological researchers.

SALVAGING PSYCHOTHERAPY RESEARCH: A MANIFESTO
About, Vol XIV, No. 2, 2014 Comments (0)

James C. COYNE* 1, 2 & Robin N. KOK 3,4

1 Department of Health Psychology, University of Groningen, University Medical Center, Groningen, the Netherlands

2 Institute for Health, Health Care Policy and Aging Research, Rutgers, The State University of New Jersey, USA

3 Department of Clinical Psychology and the EMGO institute for Health and Care Research, Faculty of Psychology and Education, VU University Amsterdam, The Netherlands

4 Centre for Mental Health Research, The Australian National University, Canberra, Australia

James C. Coyne & Robin N. Kok

Abstract

Recognition of the unreliability of findings in the biomedical literature, and especially pharmaceutical trials has led to a number of reforms.

These include reporting preregistration of protocols for clinical trials and meta-analysis, reporting standards and making data available to others for all clinical trials, as well as recognition of the influence of conflicts of interest.

These improvements are only partially and inconsistently reflected in the literature evaluating psychotherapies.

The psychotherapy literature is currently of too poor quality to provide a reliable guide to clinicians, consumers, and policymakers.

The literature is dominated by underpowered trials with high risk of bias producing positive effects at a statistically improbable rate.

Meta-analyses that are poorly conducted with undisclosed conflicts of interest compound these problems.

A number of reforms are proposed.

These include accelerating adoption of those already occurring in the pharmaceutical literature.

Additionally, psychotherapy research should parallel the orderly sequence of treatment development seen in the pharmaceutical literature.

Phase III trials providing the effect sizes of treatments should not be conducted until the acceptability of treatment and the feasibility of accruing sufficient numbers of patients are established.

The role of investigator allegiance as a potential and potent source of conflict of interest needs to be recognized.

Yet, enforcement of existing standards could counter many of the deficiencies of the current literature, but such enforcement may only come with pressures emanating from outside the field of psychotherapy.

Keywords: CONSORT, reporting standards, confirmatory bias, investigator allegiance, meta-analysis, conflict of interest.
 

Dolphin

Senior Member
Messages
17,567
The dire state of psychotherapy research

In the broader field of science, many RCTs are underpowered, yet consistently obtain positive results by redefining the primary outcomes after results are known (Chan, Hróbjartsson, Haahr, Gøtzsche, & Altman, 2004; Pinto et al., 2013). A psychotherapy trial with 30 or fewer patients in the smallest cell has less than a 50% probability of detecting a moderate size, significant effect, even if it is present (Coyne, Thombs, & Hagedoorn, 2010). Typical psychotherapy RCTs are small, methodologically flawed studies conducted by investigators with strong allegiances to one of the treatments being evaluated (Munder, Brütsch, Leonhart, Gerger, & Barth, 2013). Moreover, a lack of enforcement of trial preregistration allows for consistently positive results by redefining the primary outcomes after results are known, effectively cherrypicking the most positiveoutcomes from a battery of outcome measures.
 

Dolphin

Senior Member
Messages
17,567
Examination of the studies mustered for treatments being evidence supported by APA Division 12 (http://www.div12.org/empirically-supportedtreatments/) indicates that many studies were of low methodological quality and too underpowered to be reliably counted as evidence of efficacy, yet were included without comment about these problems.

Somebody heavily involved in maintaining this list replies agreeing

THE FUTURE OF EMPIRICALLY SUPPORTED TREATMENTS: A COMMENT ON COYNE AND KOK
About, Vol XIV, No. 2, 2014 Comments (0)

E. David KLONSKY *

Department of Psychology, University of British Columbia, Canada

http://jebp.psychotherapy.ro/about/...cally-supported-treatments-comment-coyne-kok/

[..]

It is time to thank the original Division 12 criteria for all they have done for the field (and they have done a lot), and move on to more methodologically sound, scientifically valid approaches to generating and quantifying psychotherapy outcome research. The field would do well to heed the recommendations of Coyne and Kok.
 

Dolphin

Senior Member
Messages
17,567
Taking an overview, it is striking to see the extent to which the literature continues to depend on small, methodologically flawed RCTs conducted by investigators with strong allegiances to one of the treatments being evaluated. Yet, which treatment is preferred by investigators is a better predictor of the outcome of the trial than the specific treatment being evaluated (Luborsky et al., 2006).
 

SilverbladeTE

Senior Member
Messages
3,043
Location
Somewhere near Glasgow, Scotland
Somewhat off topic, and maybe just fault of a translation?

The psychotherapy literature is currently of too poor quality to provide a reliable guide to clinicians, consumers, and policymakers.

But as said before:
WE ARE NOT "CONSUMERS", WE ARE PATIENTS!

Ugh, vast difference. :/
It's very important. Our systems are just as cuckoo as the Soviets' were, gone that far to an extreme, it's lost sight of important realities!
Seen it so much in last oh 10 or so years, shift away from "patient" to "consumer".

Also comes into why psych' areas are so mucked up, as too as it's easy tool to pull con games of denial of problems etc, because it doesn't rely on hard evidence, so you can make up any rubbish, and the spread of that has reinforced the existing "stupidity by hubris" of the profession (in general).



Good post though! :)
just that issue I think, is a very serious one regardless of their current health, and should be on the look out for and protest against.
He who controls your health, has you by the balls.
 

Dolphin

Senior Member
Messages
17,567
More generally, it is commonplace for psychotherapies with modest support from evidence to be overenthusiastically advocated for conditions even though there is no evidence for their efficacy for that particular condition. What would be disallowed as “off label applications” for pharmaceuticals is routinely accepted in psychotherapy workshops and when under scrutiny, ‘evidence-based’ often turns out to mean ‘evidence-assumed’. Yet the therapists sign off on the attendance list for continuing education credit, go home and the accrediting body gets its pay.
 

Dolphin

Senior Member
Messages
17,567
Psychotherapy research is conducted and evaluated within a mutual admiration society in which members are careful not to disparage others’ results or to embrace standards that they themselves might want relaxed when it comes to evaluation of their own research. There are rivalries between tribes like psychodynamic therapy and cognitive behavior therapy (Leichsenring & Rabung, 2011) that preclude acceptance of common methodological and statistical standards once debate becomes heated (Coyne, Bhar, Pignotti, Tovote, & Beck, 2011). However, within the tribes, there is muted criticism and strenuous efforts to create the appearance that tribe members only ‘do what works’ (Ioannidis, 2012).
 

Dolphin

Senior Member
Messages
17,567
Without preregistration, there is no indication of why the investigators stopped when they did, but such small sample sizes would hardly be justified by any formal power analysis. We believe that this phenomenon reflects investigators simply charging ahead and continuing accrual until resources are exhausted, or worse, a monitoring of incoming data reveals a positive effect and so recruitment is stopped, informed by the results that have been achieved. Formal stopping rules and rules for interim analyses (SPIRIT item 21b, Chan et al., 2013, p. 4) are rarely described in psychotherapy research protocols. Preregistration may take time to be enforced as a requirement for publication, but editors and reviewers can certainly be stopped from accepting small underpowered trials and thereby stop entering spurious effect sizes into the literature. The negative correlation between effect size and sample size and its commensurate overabundance of p-values only just crossing the border of statistical significance cannot be ignored (Kühberger, Fritz, & Scherndl, 2014).
 

jimells

Senior Member
Messages
2,009
Location
northern Maine
Yet, which treatment is preferred by investigators is a better predictor of the outcome of the trial than the specific treatment being evaluated

Wow, that is really F'd up big time. This would be shocking, except that I am well past being shocked by anything anymore.


WE ARE NOT "CONSUMERS", WE ARE PATIENTS!

You're right - we are *not* the consumer in the medical industry. That would be the insurance carriers, since they're the ones with the dough. Patients are a necessary evil, one which the medical industry would rather do without...
 

Dolphin

Senior Member
Messages
17,567
We need to raise stakeholders’ levels of skepticism, disseminate critical appraisal skills widely and provide for their application in evaluating exaggerated claims and methodological flaws in articles published in prestigious, high impact journals. Bad science in the evaluation of psychotherapy must be recognized as the current norm, not an anomaly. Unfortunately, much psychotherapy research is not just bad science – as this is easily spotted by the trained eye – but cargo-cult science: bad science posturing as legitimate science while hiding its shortcomings and inflating its positive results.
 

Dolphin

Senior Member
Messages
17,567
Disclosure of conflicts of interest should be enforced and nondisclosure identified in correction statements and further penalized. Apart from reporting of conflicts of interest, reviewers and meta-analysts should consider investigator allegiance when assessing risk of bias (Coyne, 2013a) and should themselves proactively and voluntarily declare any conflicts of interest, financial or otherwise (Viswanathan et al., 2014 propose some useful questions for this issue). Developers of treatments and persons with significant financial gain from a treatment being declared “evidence-supported” should be discouraged from conducting meta-analyses of their own treatments (e.g., Sanders et al., 2014).
 

Dolphin

Senior Member
Messages
17,567
Meta-analyses of psychotherapy should incorporate techniques to detect significance chasing in primary studies (Kühberger et al., 2014), such as phacking detection techniques (Simonsohn, Nelson, & Simmons, 2014) and tests of excess significance (Ioannidis & Trikalinos, 2007; Ioannidis, 2013) to evaluate the likelihood that patterns of significant findings exceeds likely probability. Publication bias (Duval & Tweedie, 2000; Egger et al., 1997; Sterne & Egger, 2001), significant heterogeneity (Ioannidis, Patsopoulos, & Evangelou, 2007; Thompson & Sharp, 1999) and small-study effects in meta-analyses (Harbord, Egger, & Sterne, 2006) should also be explored and, when present, adequately addressed or controlled for.
 

Dolphin

Senior Member
Messages
17,567
Adverse events and harms should routinely be reported (Vaughan, Goldstein, Alikakos, Cohen, & Serby, 2014), including estimates of lost opportunity costs such as failure to obtain more effective treatment.
 

Dolphin

Senior Member
Messages
17,567
We need to shift the culture of doing and reporting psychotherapy research. We need to shift from praising exaggerated claims about treatment and faux evidence generated to promote opportunities for therapists and their professional organizations. Instead, it is much more praiseworthy to provide robust, sustainable, reproducible, generalizable (even if more modest) claims and to call out hype and hokum in ways that preserve the credibility of psychotherapy.

The alternative is to continue protecting psychotherapy research from stringent criticism and enforcement of standards for conducting and reporting research. We can simply allow the branding of psychotherapies as “evidence supported” to fall into appropriate disrepute. We have the tools and the knowledge, now we need the consensus, cooperation and persistence.
 

jimells

Senior Member
Messages
2,009
Location
northern Maine
Maybe instead of salvaging psychotherapy it should be thrown in the trash. Probably most folks receiving psychotherapy would do better if the "therapists" simply acted as a close friend, which implies the therapist would be equal to the client, instead of setting themselves above the client as some kind of expert. This also implies the therapist would share something about themselves to help equalize the relationship.

Over the years I have seen more therapists than I can shake a stick at. The one person who really helped me wasn't a therapist at all. Her official role was to act as case manager to help me deal with the idiotic medical industry. She also drove me to appointments and sometimes to the grocery, which gave us the opportunity for real conversations, including about her own family.

Of course, all that ended when I finally received Social Security Disability. Along with receiving disability I was kicked off Medicaid, which was paying for the case manager. Now I have no way to get to the doctor's office, and getting to the grocery only happens two or three times a month.
 

Snowdrop

Rebel without a biscuit
Messages
2,933
Thank you for posting this.
It's very good to have something positive to read.
If only it caught the eye of the NHS and NIH.
 
Messages
13,774
Thanks for the excerpts. I wonder how much things are improving. I can't really tell. (Not at all with CFS stuff).
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
I found this interesting:

The American Psychological Association (APA) suppresses discussion of the problem by prohibiting groups of members publicly protesting the quality of what is being offered. As a result, APA continues to allow education credits for bogus and unproven treatments like thought field therapy and somatic experiencing.

Not talking about problems seems to be endemic in the medical profession. Questioning of the DSM (from the other APA) is discouraged. Doctors in general will not question the treatments of their colleagues except for questioning "rogue" practitioners.

So far as I am concerned the entire medical profession has no justifiable right to self-regulate. They have failed, and are continuing to fail, and its becoming more and more obvious. One might argue that oversight bodies that recommend or endorse treatments, such as NICE, could correct this. However what tends to happen, from my current understanding, is that such oversight leads to branding particular bias far too often, rather than establishing truly effective treatments.

We cannot blame the medical profession for this, and neither can we blame individual doctors. However it can be asserted they are the people most responsible for fixing this.