The 12th Invest in ME Research Conference June, 2017, Part 2
MEMum presents the second article in a series of three about the recent 12th Invest In ME International Conference (IIMEC12) in London.
Discuss the article on the Forums.

Comments submitted to the 2016 Cochrane Review of Exercise Therapy for CFS

Discussion in 'Latest ME/CFS Research' started by seaturtle, Jun 21, 2016.

  1. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    Submissions by Robert Courtney to the 2016 Cochrane Review of Exercise Therapy for Chronic Fatigue Syndrome.
    https://sites.google.com/site/mecfs...exercise-therapy-for-chronic-fatigue-syndrome

    Feel free to copy and post any text elsewhere, but please include a weblink to the original text.

    The submissions are very long, and I hope to provide a summary at some point in the near future.

    In response to:
    Cochrane Database of Systematic Reviews.
    Larun L, Brurberg KG, Odgaard-Jensen J, Price JR.
    Exercise therapy for chronic fatigue syndrome.
    Cochrane Database Syst Rev. 2016; CD003200.

    See a list of all versions of the Cochrane review, here:
    http://onlinelibrary.wiley.com/wol1/doi/10.1002/14651858.CD003200.pub5/otherversions

    Update: The latest version of the Cochrane review (version 5) includes the first two letters (listed below), and the reviewers' responses to them, and can be seen here:
    View in Browser: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/full
    PDF: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/pdf

    The submissions, listed below, are in response to a previous version of the Cochrane review (version 4) which can be seen here. (The latest version differs only in that it includes copies of the first two letters, listed below, but there are no other amendments.)


    Submissions:

    1. Query re use of post-hoc unpublished outcome data: Scoring system for the Chalder fatigue scale, Wearden 2010.
    Robert Courtney
    16th April 2016
    https://sites.google.com/site/mecfs...-fatigue-syndrome/fine-trial-unpublished-data

    2. Assessment of Selective Reporting Bias in White 2011.
    Robert Courtney
    1st May 2016
    https://sites.google.com/site/mecfs...-syndrome/pace-trial-selective-reporting-bias

    3. A query regarding the way outcomes for physical function and overall health have been described in the abstract, conclusion and discussions of the review.
    Robert Courtney
    12th May 2016
    https://sites.google.com/site/mecfs...isreporting-of-outcomes-for-physical-function

    4. Concerns regarding the use of unplanned primary outcomes in the Cochrane review.
    Robert Courtney
    3rd June 2016
    https://sites.google.com/site/mecfs...ic-fatigue-syndrome/primary-outcome-switching


    .
     
    Last edited: Jun 25, 2016
    GreyOwl, Sam Carter, duncan and 19 others like this.
  2. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    These submissions focus on:

    • The review authors switching their primary outcomes in the review, and using unplanned analyses, which has had the effect of substantially changing the interpretation and reporting of the primary outcomes of the review;
    • A failure to clearly report that there were no significant treatment effects for any pooled outcomes in any measures of health (except for sleep) at follow-up;
    • The use of some unpublished and post-hoc data from external studies for the review, despite the reviewers claiming that they've used only published data and pre-defined outcomes. This may have had a significant effect on the outcomes;
    • Incorrect and misleading reporting of the review's outcomes in the main discussions of the review;
    • An inaccurate assessment of bias relating to the reviewed studies.

    These issues (e.g. the failure to clearly report that there were no significant pooled treatment effects for health at follow-up - except for sleep - and the misrepresentation of the review's outcomes in the main discussions) have already caused misreporting of the review elsewhere. (e.g. here.)

    These are all serious issues, that we should not be seeing in a Cochrane review.

    It seems to me that Cochrane have "pulled a PACE" with this review (e.g. switching primary outcomes and burying/misrepresenting unfavourable outcomes) and it shouldn't be acceptable for Cochrane.
     
    Last edited: Jun 25, 2016
    TrixieStix, GreyOwl, Kati and 16 others like this.
  3. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    For reference...

    Main forum thread for the Cochrane review of exercise therapy...

    Exercise therapy beneficial for some ME/CFS patients new Cochrane report says
    http://forums.phoenixrising.me/inde...-cfs-patients-new-cochrane-report-says.35474/


    Other relevant threads...

    An abridged version of the Cochrane review of exercise therapy for chronic fatigue syndrome
    http://forums.phoenixrising.me/inde...e-therapy-for-chronic-fatigue-syndrome.39940/

    Why the Cochrane Collaboration needs to clean up conflicts of interest (on GET for CFS review)
    http://forums.phoenixrising.me/inde...icts-of-interest-on-get-for-cfs-review.43690/

    New Coyne blog: "An open letter to the Cochrane Collaboration"
    http://forums.phoenixrising.me/inde...n-letter-to-the-cochrane-collaboration.43489/

    My two detailed comments on the Cochrane Exercise Therapy for CFS review (2015)
    http://forums.phoenixrising.me/inde...-for-cfs-review-2015.39801/page-2#post-694387

    Ann Intern Med: Review: Exercise therapy reduces fatigue in chronic fatigue syndrome
    http://forums.phoenixrising.me/inde...es-fatigue-in-chronic-fatigue-syndrome.44706/

    David Tovey (Editor in Chief of Cochrane Library) comments on Cochrane reviews of CFS
    http://forums.phoenixrising.me/inde...ry-comments-on-cochrane-reviews-of-cfs.44140/

    CMAJ (Canada) recommends exercise for CFS
    http://forums.phoenixrising.me/inde...nds-exercise-for-cfs.44396/page-4#post-731307
     
    Last edited: Jun 21, 2016
    duncan, Sea, actup and 8 others like this.
  4. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
  5. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    I don't expect anyone to read through these submissions, and I'll try to post some summaries over the next few days or weeks, health permitting. Anyone should feel free to copy any excerpts onto this thread, or to attempt to summarize etc.
     
    Daisymay, Simon, Richard7 and 6 others like this.
  6. Living Dead

    Living Dead Senior Member

    Messages:
    165
    Likes:
    266
    Stupid questions, but:
    1. Is this cochrane review finished or in the making?
    2. In what way are they open to these letters? (If the review is finished, what is the point?)
    3. Did you write these letters (you don't have to answer) and did the author submit them, or just post them online?
     
    panckage and Comet like this.
  7. ScottTriGuy

    ScottTriGuy Stop the harm. Start the research and treatment.

    Messages:
    1,089
    Likes:
    4,398
    Toronto, Canada
    Will the Cochrane Review publish your submissions? In what section (eg letters to editors)?

    What do you hope will happen (eg CR retraction)?
     
    Comet likes this.
  8. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    These are good questions!
    1. This is the Cochrane review for exercise therapy that's already been published.
    2. They seem to publish and respond to most letters, so I would expect these to be published with an appropriate response. They can amend their review and easily republish it, if they wish to. The reviews are considered to be an ongoing process, rather than fixed in stone. The current version is already version 4.
    3. Yes, I wrote these letters and they have been formally submitted to Cochrane. See the dates listed above, under each title, for the dates that they were submitted.
     
    Last edited: Jun 21, 2016
    Sea, panckage, duncan and 8 others like this.
  9. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    They tend to publish the letters that they receive. The way they seem to publish them is to republish the entire review, as a new version (the current version is version 4) which may include any amendments or corrections that they wish to include, with the letters included in the full PDF document. The letters are published towards the end of the PDF document. I'll highlight them here if/when they are published.

    I'd hope for some corrections and amendments to the review, but that's a hope on my part rather than a realistic expectation. If I feel that my concerns have not been addressed satisfactorily by the review authors (any responses will be by the review authors, at this stage), then there is the option of embarking on a formal complaints procedure to Cochrane.
     
    Last edited: Jun 21, 2016
    Daisymay, snowathlete, Simon and 10 others like this.
  10. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    BTW, I posted these today because I'm expecting a formal response to the first two submissions quite soon. I've no idea what the response will be. I'll keep you all posted.
     
  11. Living Dead

    Living Dead Senior Member

    Messages:
    165
    Likes:
    266
  12. ScottTriGuy

    ScottTriGuy Stop the harm. Start the research and treatment.

    Messages:
    1,089
    Likes:
    4,398
    Toronto, Canada
    Good. I understand many doctors in Canada also refer to the Cochrane Review so any corrections reverberate (hopefully).

    Keep up the pressure.
     
    Comet, Invisible Woman and seaturtle like this.
  13. Invisible Woman

    Invisible Woman Senior Member

    Messages:
    1,212
    Likes:
    7,419
    A lot of hard work here @seaturtle. Well done & thank you!
     
    SamanthaJ, Daisymay, Dolphin and 6 others like this.
  14. seaturtle

    seaturtle

    Messages:
    32
    Likes:
    280
    A preliminary summary (of sorts) of the main issues ... I might try to refine this at a later date to make it more accessible...
    This is perhaps the most substantial issue that I have addressed, and it's related to the primary outcome switching in the review. It's an issue that our community needs to be aware of, and to flag up when necessary.

    When people refer to the Cochrane review, and if they claim that the review demonstrates universally beneficial health effects, then we need to point out that there were no pooled treatment effects at follow-up for any health outcomes except for sleep. So any beneficial effects, for fatigue, physical function and overall-health, etc., are short-lived. These results are a very important issue for the patient community, but they have been misrepresented in the review itself. This issue is discussed mainly in my submission no.4, and also a little in submission no.3. Some of the follow-up outcomes (i.e. for physical function and overall health) have been misreported in the main discussions in the review (see my submission no.3 for full details) which has reinforced an erroneous impression that there were significant pooled treatment effects at follow-up.

    So, I'll briefly try to explain the details. The pre-specified primary outcome (fatigue) was for a pooled treatment effect for all eligible studies immediately after treatment and at follow-up. However this primary outcome for pooled treatment effects was dropped for questionable reasons and replaced with treatment effects for studies grouped together by the specific measure used to assess fatigue (i.e. the various fatigue questionnaires). For the grouped outcomes, two out of the three outcomes had significant treatment effects for fatigue at follow-up, and the other outcome was a non-significant effect. The (two out of three positive, and one non-significant) outcomes for fatigue at follow-up were reported as a primary outcome instead of the pre-specified pooled treatment effect for all the eligible studies. However, the pooled treatment effect, for the same measures, was not a significant outcome for fatigue at follow-up. This null (pre-specified) pooled outcome for fatigue, at follow-up, has been buried and it's almost impossible to find in the review. Uniquely, for the main outcomes, there is no table outlining the details of the analysis.

    Furthermore, all the other health outcomes were non-significant at follow-up (except sleep). Unfortunately, physical function and overall-health have been mis-reported in the main text as being positive outcomes at follow-up, when in fact they were non-significant results.

    So, all but one health indicator (i.e. fatigue, physical function, overall health, pain, quality of life, depression, and anxiety, but not sleep) demonstrated a non-significant outcome for pooled treatment effects at follow-up for exercise therapy versus passive control.

    In a nutshell, fatigue was reported as demonstrating broadly positive outcomes at follow-up, but the (pre-specified) pooled treatment effects were not significant for fatigue at follow-up; and physical function and overall-health were (mis)reported in the main discussion as having positive treatment effects at follow-up but all health outcomes (except sleep) actually demonstrated non-significant treatment effects at follow-up. The actual treatment effects at follow-up are the opposite to the impression gained from a superficial read of the review, or even a careful read of the review. It's only by an in-depth analysis of the entire review that these issues would be noticed.

    The way the review is written, any ordinary reader would be under the impression that the main outcomes at follow-up were positive, when in fact the pooled outcomes were not significant, except for sleep only.

    Cochrane advises its reviewers to avoid switching primary outcomes, so it will be interesting to see how they attempt to justify their actions with regard to the switching.

    I have asked the reviewers to: "Clearly and unambiguously explain that all but one health indicator (i.e. fatigue, physical function, overall health, pain, quality of life, depression, and anxiety, but not sleep) demonstrated a non-significant outcome for pooled treatment effects at follow-up for exercise therapy versus passive control". I think it's a reasonable request.


    Switching the primary outcomes, burying the details of the null pooled primary outcomes at follow-up, and misreporting some of the follow-up outcomes (as being positive when they were not significant), has led to a misunderstanding and mis-reporting of the review. This is demonstrated in the following published letter...

    CMAJ (Canada) recommends exercise for CFS
    http://forums.phoenixrising.me/inde...nds-exercise-for-cfs.44396/page-4#post-731307

    The letter claims: "We based the recommendations on the Cochrane systematic review which looked at 8 randomised trials of exercise for chronic fatigue, and together showed a consistent modest benefit of exercise across the different patient groups included. The clear and consistent benefit suggests indication rather than contraindication of exercise."

    However, the alleged "clear and consistent benefit" was short-lived when considering the pooled treatment effects, and there were no significant treatment treatment effects for any pooled health outcomes at follow-up, except for sleep.
     
    Last edited: Jun 25, 2016
  15. BurnA

    BurnA Senior Member

    Messages:
    2,086
    Likes:
    9,863
    Well done, great work !
     
  16. Living Dead

    Living Dead Senior Member

    Messages:
    165
    Likes:
    266
    The review was written by:
    Not sure who this refers to (i.e is it the patients that have a mental disorder, or the editorial group), but this may be a hurdle to getting a "fair trial". I don't doubt that these people are well-meaning, but I have a feeling that the ability to crunch numbers and interpret biologial research is somewhat lacking in psych* professions.
     
  17. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,523
    That's a lot of comments. Thank you so much for all your work on this.

    My thoughts:

    1: Also, there's the strange fact that the bimodal result used in Cochrane don't seem to match up with those provided in the FINE RR (which Larun's response to Kindlon highlighted). Hopefully their response to your comment will help clarify what happened there.

    Comment 2 is a joy. So good that you went into the details on Larun's vague and evasive response. Her response was such a frustrating brush-off, and this shows why it's so important for patients to not be put off, but keep picking away at the details.

    3 is pretty fun too - I especially liked: "It is generally understood that a "positive" treatment effect equates to a significant effect, and I believe that the Cochrane text should reflect this, or at least clarify that the term "positive effect" is being used to indicate a lack of significance."

    4 - what?! The review was way more of a mess than I'd realised if all of this is right. It's a bit embarrassing that we didn't notice this before. I'm going to take another look at this tomorrow. To me, this looks like it could be adding up to retraction time, and I don't say that lightly.
     
  18. A.B.

    A.B. Senior Member

    Messages:
    3,751
    Likes:
    23,191
    Number 4 is ... interesting. My attempt at a summary (hope I get it right, not in an ideal state for this sort of work at the moment)

    The reviewers deviated from their pre-specified plan of statistical analysis. According to the original plan, none of the outcomes except for sleep are positive. The review fails to discuss this and instead describes exercise therapy as having a broadly positive effect on outcomes. The short term improvement seen in outcomes could be entirely due to bias inherent in open label studies (lack of long term differences is what one would expect if this were the case).

    I don't think this review has any credibility left. Looks like pure manipulation.
     
  19. A.B.

    A.B. Senior Member

    Messages:
    3,751
    Likes:
    23,191
    Is this really as bad as it looks? They cheated with the outcome switching and failed to acknowledge just how dubious the short term results are due lack of blinding. The conclusion should have been something along the lines of "ineffective long term, of dubious value in short term". Instead they concluded:

     
  20. A.B.

    A.B. Senior Member

    Messages:
    3,751
    Likes:
    23,191
    Also a question regarding the statement that "no evidence suggests that exercise therapy may worsen outcomes".

    Does the ME Association survey on GET not count as evidence? It may not be strong evidence, but if a survey shows the majority of patients reporting worsening with GET, should this really be ignored completely?

    What are the Cochrane rules on inclusion of material?
     
    Last edited: Jun 22, 2016
    SamanthaJ, Daisymay, Dolphin and 4 others like this.

See more popular forum discussions.

Share This Page