• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Comments submitted to the 2016 Cochrane Review of Exercise Therapy for CFS

Messages
32
Submissions by Robert Courtney to the 2016 Cochrane Review of Exercise Therapy for Chronic Fatigue Syndrome.
https://sites.google.com/site/mecfs...exercise-therapy-for-chronic-fatigue-syndrome

Feel free to copy and post any text elsewhere, but please include a weblink to the original text.

The submissions are very long, and I hope to provide a summary at some point in the near future.

In response to:
Cochrane Database of Systematic Reviews.
Larun L, Brurberg KG, Odgaard-Jensen J, Price JR.
Exercise therapy for chronic fatigue syndrome.
Cochrane Database Syst Rev. 2016; CD003200.

See a list of all versions of the Cochrane review, here:
http://onlinelibrary.wiley.com/wol1/doi/10.1002/14651858.CD003200.pub5/otherversions

Update: The latest version of the Cochrane review (version 5) includes the first two letters (listed below), and the reviewers' responses to them, and can be seen here:
View in Browser: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/full
PDF: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/pdf

The submissions, listed below, are in response to a previous version of the Cochrane review (version 4) which can be seen here. (The latest version differs only in that it includes copies of the first two letters, listed below, but there are no other amendments.)


Submissions:

1. Query re use of post-hoc unpublished outcome data: Scoring system for the Chalder fatigue scale, Wearden 2010.
Robert Courtney
16th April 2016
https://sites.google.com/site/mecfs...-fatigue-syndrome/fine-trial-unpublished-data

2. Assessment of Selective Reporting Bias in White 2011.
Robert Courtney
1st May 2016
https://sites.google.com/site/mecfs...-syndrome/pace-trial-selective-reporting-bias

3. A query regarding the way outcomes for physical function and overall health have been described in the abstract, conclusion and discussions of the review.
Robert Courtney
12th May 2016
https://sites.google.com/site/mecfs...isreporting-of-outcomes-for-physical-function

4. Concerns regarding the use of unplanned primary outcomes in the Cochrane review.
Robert Courtney
3rd June 2016
https://sites.google.com/site/mecfs...ic-fatigue-syndrome/primary-outcome-switching


.
 
Last edited:
Messages
32
These submissions focus on:

• The review authors switching their primary outcomes in the review, and using unplanned analyses, which has had the effect of substantially changing the interpretation and reporting of the primary outcomes of the review;
• A failure to clearly report that there were no significant treatment effects for any pooled outcomes in any measures of health (except for sleep) at follow-up;
• The use of some unpublished and post-hoc data from external studies for the review, despite the reviewers claiming that they've used only published data and pre-defined outcomes. This may have had a significant effect on the outcomes;
• Incorrect and misleading reporting of the review's outcomes in the main discussions of the review;
• An inaccurate assessment of bias relating to the reviewed studies.

These issues (e.g. the failure to clearly report that there were no significant pooled treatment effects for health at follow-up - except for sleep - and the misrepresentation of the review's outcomes in the main discussions) have already caused misreporting of the review elsewhere. (e.g. here.)

These are all serious issues, that we should not be seeing in a Cochrane review.

It seems to me that Cochrane have "pulled a PACE" with this review (e.g. switching primary outcomes and burying/misrepresenting unfavourable outcomes) and it shouldn't be acceptable for Cochrane.
 
Last edited:
Messages
32
For reference...

Main forum thread for the Cochrane review of exercise therapy...

Exercise therapy beneficial for some ME/CFS patients new Cochrane report says
http://forums.phoenixrising.me/inde...-cfs-patients-new-cochrane-report-says.35474/


Other relevant threads...

An abridged version of the Cochrane review of exercise therapy for chronic fatigue syndrome
http://forums.phoenixrising.me/inde...e-therapy-for-chronic-fatigue-syndrome.39940/

Why the Cochrane Collaboration needs to clean up conflicts of interest (on GET for CFS review)
http://forums.phoenixrising.me/inde...icts-of-interest-on-get-for-cfs-review.43690/

New Coyne blog: "An open letter to the Cochrane Collaboration"
http://forums.phoenixrising.me/inde...n-letter-to-the-cochrane-collaboration.43489/

My two detailed comments on the Cochrane Exercise Therapy for CFS review (2015)
http://forums.phoenixrising.me/inde...-for-cfs-review-2015.39801/page-2#post-694387

Ann Intern Med: Review: Exercise therapy reduces fatigue in chronic fatigue syndrome
http://forums.phoenixrising.me/inde...es-fatigue-in-chronic-fatigue-syndrome.44706/

David Tovey (Editor in Chief of Cochrane Library) comments on Cochrane reviews of CFS
http://forums.phoenixrising.me/inde...ry-comments-on-cochrane-reviews-of-cfs.44140/

CMAJ (Canada) recommends exercise for CFS
http://forums.phoenixrising.me/inde...nds-exercise-for-cfs.44396/page-4#post-731307
 
Last edited:

Living Dead

Senior Member
Messages
199
Stupid questions, but:
1. Is this cochrane review finished or in the making?
2. In what way are they open to these letters? (If the review is finished, what is the point?)
3. Did you write these letters (you don't have to answer) and did the author submit them, or just post them online?
 

ScottTriGuy

Stop the harm. Start the research and treatment.
Messages
1,402
Location
Toronto, Canada
Will the Cochrane Review publish your submissions? In what section (eg letters to editors)?

What do you hope will happen (eg CR retraction)?
 
Messages
32
Stupid questions, but:
1. Is this cochrane review finished or in the making?
2. In what way are they open to these letters? (If the review is finished, what is the point?)
3. Did you write these letters (you don't have to answer) and did the author submit them, or just post them online?
These are good questions!
1. This is the Cochrane review for exercise therapy that's already been published.
2. They seem to publish and respond to most letters, so I would expect these to be published with an appropriate response. They can amend their review and easily republish it, if they wish to. The reviews are considered to be an ongoing process, rather than fixed in stone. The current version is already version 4.
3. Yes, I wrote these letters and they have been formally submitted to Cochrane. See the dates listed above, under each title, for the dates that they were submitted.
 
Last edited:
Messages
32
Will the Cochrane Review publish your submissions? In what section (eg letters to editors)?

What do you hope will happen (eg CR retraction)?
They tend to publish the letters that they receive. The way they seem to publish them is to republish the entire review, as a new version (the current version is version 4) which may include any amendments or corrections that they wish to include, with the letters included in the full PDF document. The letters are published towards the end of the PDF document. I'll highlight them here if/when they are published.

I'd hope for some corrections and amendments to the review, but that's a hope on my part rather than a realistic expectation. If I feel that my concerns have not been addressed satisfactorily by the review authors (any responses will be by the review authors, at this stage), then there is the option of embarking on a formal complaints procedure to Cochrane.
 
Last edited:

ScottTriGuy

Stop the harm. Start the research and treatment.
Messages
1,402
Location
Toronto, Canada
... If I feel that my concerns have not been addressed satisfactorily by the review authors (any responses will be by the review authors, at this stage), then there is the option of embarking on a formal complaints procedure to Cochrane.

Good. I understand many doctors in Canada also refer to the Cochrane Review so any corrections reverberate (hopefully).

Keep up the pressure.
 
Messages
32
A preliminary summary (of sorts) of the main issues ... I might try to refine this at a later date to make it more accessible...
A failure to clearly report that there were no significant treatment effects for any pooled outcomes in any measures of health (except for sleep) at follow-up...
This is perhaps the most substantial issue that I have addressed, and it's related to the primary outcome switching in the review. It's an issue that our community needs to be aware of, and to flag up when necessary.

When people refer to the Cochrane review, and if they claim that the review demonstrates universally beneficial health effects, then we need to point out that there were no pooled treatment effects at follow-up for any health outcomes except for sleep. So any beneficial effects, for fatigue, physical function and overall-health, etc., are short-lived. These results are a very important issue for the patient community, but they have been misrepresented in the review itself. This issue is discussed mainly in my submission no.4, and also a little in submission no.3. Some of the follow-up outcomes (i.e. for physical function and overall health) have been misreported in the main discussions in the review (see my submission no.3 for full details) which has reinforced an erroneous impression that there were significant pooled treatment effects at follow-up.

So, I'll briefly try to explain the details. The pre-specified primary outcome (fatigue) was for a pooled treatment effect for all eligible studies immediately after treatment and at follow-up. However this primary outcome for pooled treatment effects was dropped for questionable reasons and replaced with treatment effects for studies grouped together by the specific measure used to assess fatigue (i.e. the various fatigue questionnaires). For the grouped outcomes, two out of the three outcomes had significant treatment effects for fatigue at follow-up, and the other outcome was a non-significant effect. The (two out of three positive, and one non-significant) outcomes for fatigue at follow-up were reported as a primary outcome instead of the pre-specified pooled treatment effect for all the eligible studies. However, the pooled treatment effect, for the same measures, was not a significant outcome for fatigue at follow-up. This null (pre-specified) pooled outcome for fatigue, at follow-up, has been buried and it's almost impossible to find in the review. Uniquely, for the main outcomes, there is no table outlining the details of the analysis.

Furthermore, all the other health outcomes were non-significant at follow-up (except sleep). Unfortunately, physical function and overall-health have been mis-reported in the main text as being positive outcomes at follow-up, when in fact they were non-significant results.

So, all but one health indicator (i.e. fatigue, physical function, overall health, pain, quality of life, depression, and anxiety, but not sleep) demonstrated a non-significant outcome for pooled treatment effects at follow-up for exercise therapy versus passive control.

In a nutshell, fatigue was reported as demonstrating broadly positive outcomes at follow-up, but the (pre-specified) pooled treatment effects were not significant for fatigue at follow-up; and physical function and overall-health were (mis)reported in the main discussion as having positive treatment effects at follow-up but all health outcomes (except sleep) actually demonstrated non-significant treatment effects at follow-up. The actual treatment effects at follow-up are the opposite to the impression gained from a superficial read of the review, or even a careful read of the review. It's only by an in-depth analysis of the entire review that these issues would be noticed.

The way the review is written, any ordinary reader would be under the impression that the main outcomes at follow-up were positive, when in fact the pooled outcomes were not significant, except for sleep only.

Cochrane advises its reviewers to avoid switching primary outcomes, so it will be interesting to see how they attempt to justify their actions with regard to the switching.

I have asked the reviewers to: "Clearly and unambiguously explain that all but one health indicator (i.e. fatigue, physical function, overall health, pain, quality of life, depression, and anxiety, but not sleep) demonstrated a non-significant outcome for pooled treatment effects at follow-up for exercise therapy versus passive control". I think it's a reasonable request.


Switching the primary outcomes, burying the details of the null pooled primary outcomes at follow-up, and misreporting some of the follow-up outcomes (as being positive when they were not significant), has led to a misunderstanding and mis-reporting of the review. This is demonstrated in the following published letter...

CMAJ (Canada) recommends exercise for CFS
http://forums.phoenixrising.me/inde...nds-exercise-for-cfs.44396/page-4#post-731307

The letter claims: "We based the recommendations on the Cochrane systematic review which looked at 8 randomised trials of exercise for chronic fatigue, and together showed a consistent modest benefit of exercise across the different patient groups included. The clear and consistent benefit suggests indication rather than contraindication of exercise."

However, the alleged "clear and consistent benefit" was short-lived when considering the pooled treatment effects, and there were no significant treatment treatment effects for any pooled health outcomes at follow-up, except for sleep.
 
Last edited:

Living Dead

Senior Member
Messages
199
The review was written by:
Not sure who this refers to (i.e is it the patients that have a mental disorder, or the editorial group), but this may be a hurdle to getting a "fair trial". I don't doubt that these people are well-meaning, but I have a feeling that the ability to crunch numbers and interpret biologial research is somewhat lacking in psych* professions.
 
Messages
13,774
That's a lot of comments. Thank you so much for all your work on this.

My thoughts:

1: Also, there's the strange fact that the bimodal result used in Cochrane don't seem to match up with those provided in the FINE RR (which Larun's response to Kindlon highlighted). Hopefully their response to your comment will help clarify what happened there.

Comment 2 is a joy. So good that you went into the details on Larun's vague and evasive response. Her response was such a frustrating brush-off, and this shows why it's so important for patients to not be put off, but keep picking away at the details.

3 is pretty fun too - I especially liked: "It is generally understood that a "positive" treatment effect equates to a significant effect, and I believe that the Cochrane text should reflect this, or at least clarify that the term "positive effect" is being used to indicate a lack of significance."

4 - what?! The review was way more of a mess than I'd realised if all of this is right. It's a bit embarrassing that we didn't notice this before. I'm going to take another look at this tomorrow. To me, this looks like it could be adding up to retraction time, and I don't say that lightly.
 

A.B.

Senior Member
Messages
3,780
Number 4 is ... interesting. My attempt at a summary (hope I get it right, not in an ideal state for this sort of work at the moment)

The reviewers deviated from their pre-specified plan of statistical analysis. According to the original plan, none of the outcomes except for sleep are positive. The review fails to discuss this and instead describes exercise therapy as having a broadly positive effect on outcomes. The short term improvement seen in outcomes could be entirely due to bias inherent in open label studies (lack of long term differences is what one would expect if this were the case).

I don't think this review has any credibility left. Looks like pure manipulation.
 

A.B.

Senior Member
Messages
3,780
Is this really as bad as it looks? They cheated with the outcome switching and failed to acknowledge just how dubious the short term results are due lack of blinding. The conclusion should have been something along the lines of "ineffective long term, of dubious value in short term". Instead they concluded:

Patients with CFS may generally benefit and feel less fatigued following exercise therapy, and no evidence suggests that exercise therapy may worsen outcomes. A positive effect with respect to sleep, physical function and self-perceived general health has been observed, but no conclusions for the outcomes of pain, quality of life, anxiety, depression, drop-out rate and health service resources were possible.
 

A.B.

Senior Member
Messages
3,780
Also a question regarding the statement that "no evidence suggests that exercise therapy may worsen outcomes".

Does the ME Association survey on GET not count as evidence? It may not be strong evidence, but if a survey shows the majority of patients reporting worsening with GET, should this really be ignored completely?

What are the Cochrane rules on inclusion of material?
 
Last edited: