Cochrane have now published the authors' responses to my first two letters.
Cochrane's new publication (version 5)...
View in Browser:
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/full
PDF:
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub5/pdf
See pages 120 and 125 in the new PDF for the responses, or do an in-page word search for 'Courtney' in the browser version. (I've copied both responses below as well.)
My two letters discussed the following issues:
1. My first letter discusses the use of post-hoc data from the FINE trial. The Cochrane Review claims that all the data they have analysed was previously published data. (i.e. formally published in a peer-reviewed journal, as per the Cochrane review's protocol.) However, the fatigue data the review has used from the FINE trial is based on the Likert scoring system, whereas the FINE trial only published data based on the biomodal scoring system. The FINE trial's Likert data is actually post-hoc data and was initially published by the FINE authors only in a
BMJ rapid response post, as an after-thought. I queried this issue in my letter.
2. My second letter discusses the PACE trial data and explains the reasons why I believe that the Cochrane review should have categorised the PACE trial data as 'unplanned', and assessed the risk of bias for 'selective reporting' accordingly. The Cochrane review currently categorises the risk of 'selective reporting' bias for the PACE trial as "low', whereas it is my interpretation that the Cochrane reviewers' guidelines indicate (unambiguously) that the risk of bias for the PACE data should be high. I think my argument is fairly robust and water-tight.
All the issues raised in my letters have been entirely dismissed. All of them! Which I find quite bizarre, especially considering that some of the points that I made were factual (i.e. not particularly open to interpretation) and difficult to dispute. Indeed, the authors' response from Cochrane even accepts the main point that I made, in relation to the FINE data, but then the author strangely says that we must "agree to disagree", on all issues, even though she agreed with my main substantive point.
This is the response to my first letter...
Larun said:
Dear Robert Courtney
Thank you for your detailed comments on the Cochrane review 'Exercise Therapy for Chronic Fatigue Syndrome'. We have the greatest respect for your right to comment on and disagree with our work. We take our work as researchers extremely seriously and publish reports that have been subject to rigorous internal and external peer review. In the spirit of openness, transparency and mutual respect we must politely agree to disagree.
The Chalder Fatigue Scale was used to measure fatigue. The results from the Wearden 2010 trial show a statistically significant difference in favour of pragmatic rehabilitation at 20 weeks, regardless whether the results were scored bi-modally or on a scale from 0-3. The effect estimate for the 70 week comparison with the scale scored bi-modally was -1.00 (CI-2.10 to +0.11; p =.076) and -2.55 (-4.99 to -0.11; p=.040) for 0123 scoring. The FINE data measured on the 33-point scale was published in an online rapid response after a reader requested it. We therefore knew that the data existed, and requested clarifying details from the authors to be able to use the estimates in our meta-analysis. In our unadjusted analysis the results were similar for the scale scored bi-modally and the scale scored from 0 to 3, i.e. a statistically significant difference in favour of rehabilitation at 20 weeks and a trend that does not reach statistical significance in favour of pragmatic rehabilitation at 70 weeks. The decision to use the 0123 scoring did does not affect the conclusion of the review.
Regards,
Lillebeth Larun
The author has provided the effect estimates (mean differences) for the (pre-specified) bimodal and (post-hoc) Likert data for fatigue for the FINE trial at 70 weeks (follow-up). These outcomes, using the different scoring methods, are different, so switching the outcomes
may possibly have had an impact on the review's primary outcome (fatigue) at follow-up. Larun hasn't given us the effect estimates for
end-of-treatment, but these would also demonstrate variance between bimodal and Likert scoring, so switching the outcomes
might also have had a significant impact on the primary outcome of the Cochrane review at
end-of-treatment. (Note that the effect estimates given here are mean differences, rather than standardised mean differences, so the differences between the effect estimates may look greater than they actually are simply because they use different scoring scales.)
Larun said: "
The decision to use the 0123 [i.e. Likert] scoring did does not affect the conclusion of the review." But she has provided no evidence to demonstrate this! There is no sensitivity analysis. Are we supposed to accept the word of the author, rather than review the evidence (of a Cochrane review - renowned for their rigour and impartiality!), that switching the review's primary outcome data, from pre-specified to unplanned data, has made no difference to the review's outcomes? Is that supposed to be a rigorous and transparent methodology? I quote from Larun's response: "
In the spirit of openness, transparency..." But where is the transparency here?
Note that Larun has admitted that I am correct with respect to the FINE data (i.e. that it was previously unpublished data; it was not part of the formally published study, but was simply posted informally in a rapid response): "...
the 33-point scale was published in an online rapid response after a reader requested it. We therefore knew that the data existed, and requested clarifying details from the authors..." But then Larun says we must "
agree to disagree". And she's refused to correct her literature either to amend the analyses so they use pre-specified data, or to amend the text of the review so that it indicates that the data is unpublished and post-hoc.
Notice the difference in the effect estimates at 70 weeks for bimodal scoring [-1.00 (CI-2.10 to +0.11; p =.076)] vs Likert scoring [-2.55 (-4.99 to -0.11; p=.040)]. Larun says that both outcomes (i.e. bimodal & Likert) are non-significant at 70 weeks (which isn't true of the data that she has provided above, but her quoted data is slightly different to the published data - see below for further details). However, the significance or non-significance of the FINE data in isolation has limited relevance for a meta-analysis; changing outcomes in this way may have an impact on the review's findings. The PACE trial data was added to the FINE data, only, for the review's published primary analysis, and the combined FINE and PACE data was reported to show a positive and statistically significant effect from exercise therapy.
A friend of mine has commented that the Cochrane reviewers saw, from the BMJ rapid response, that a post-hoc Likert analysis of results allowed for better results to be reported for the FINE trial, so they requested the additional data from the trial's researchers (although they said in the review that they had not) in order to include their own post-hoc analysis in their review.
Note that the review still incorrectly says that all the data is previously published data - even though Larun admits in the letter that it isn't. (i.e. Not published in the formal peer-reviewed sense; we assume that the review wasn't referring to data that might be published in blogs or magazines etc, because the review pretends to analyse formally published data only.)
The authors have ignored my letters and not changed anything in the review, despite admitting in the response that they've used post-hoc data.
The figures for the effect size that Larun has included in the response, quoted above, are slightly different from the data in the Cochrane review. Larun states that the effect size for fatigue at 70 weeks using Likert data is -2.55 (-4.99 to -0.11; p=.040), whereas the Cochrane Review states that it is -2.12 [ -4.49, 0.25 ]. It seems that Larun has quoted the
BMJ rapid response by Wearden et al. rather than her own review's calculations...
BMJ rapid reponse (Wearden et al.) said:
Supportive listening (SL) is still ineffective when compared with GPTAU
(Table 1 and Figure 1). Effect estimates [95% confidence intervals] for 20
week comparisons are: PR versus GPTAU -3.84 [-6.17, -1.52], SE 1.18,
P=0.001; SL versus GPTAU +0.30 [-1.73, +2.33], SE 1.03, P=0.772. Effect
estimates [95% confidence intervals] for 70 week comparisons are: PR
versus GPTAU -2.55 [-4.99,-0.11], SE 1.24, P=0.040; SL versus GPTAU +0.36
[-1.90, 2.63], SE 1.15, P=0.752.
This is the response to my second letter...
Larun said:
Dear Robert Courtney
Thank you for your detailed comments on the Cochrane review 'Exercise Therapy for Chronic Fatigue Syndrome'. We have the greatest respect for your right to comment on and disagree with our work. We take our work as researchers extremely seriously and publish reports that have been subject to rigorous internal and external peer review. In the spirit of openness, transparency and mutual respect we must politely agree to disagree.
Cochrane reviews aim to report the review process in a transparent way, for example, are reasons for the risk of bias stated. We do not agree that Risk of Bias for the Pace trial (White 2011) should be changed, but have presented it in a way so it is possible to see our reasoning. We find that we have been quite careful in stating the effect estimates and the certainty of the documentation. We note that you read this differently.
Regards,
Lillebeth
I'm at a loss to understand what is meant by: "
We do not agree that Risk of Bias for the Pace trial (White 2011) should be changed, but have presented it in a way so it is possible to see our reasoning." I don't think that the review discusses the issue of the PACE data being unplanned, so I'm not sure what is meant by the suggestion the issue has been discussed. This is simply a point-blank refusal to engage with the substantive and serious issues that I raised.