"Selective reporting (outcome bias)" and White et al. (2011)
I don't believe that White et al. (2011) (the PACE Trial) (3) should be classed as having a low risk of bias under "Selective reporting (outcome bias)" (Figure 2, page 15). According to the Cochrane Collaboration's tool for assessing risk of bias (21), the category of low risk of bias is for: "The study protocol is available and all of the study’s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way". This is not the case in the PACE Trial. The three primary efficacy outcomes can be seen in the published protocol (22). None have been reported in the pre-specified way. The Cochrane Collaboration's tool for assessing risk of bias states that a “high risk” of bias applies if any one of several criteria are met, including that “not all of the study’s pre-specified primary outcomes have been reported” or “one or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g. subscales) that were not pre-specified”. In the PACE Trial, the third primary outcome measure (the number of "overall improvers") was never published. Also, the other two primary outcome measures were reported using analysis methods that were not pre-specified (including switching from the bimodal to the Likert scoring method for The Chalder Fatigue Scale, one of the primary outcomes in your review). These facts mean that the “high risk of bias” category should apply.
Selective reporting (outcome bias)
The Cochrane Risk of Bias tool enables the review authors to be transparent about their judgments, but due to the subjective nature of the process it does not guarantee an indisputable consensus. You particularly mention the risk of bias in the PACE trial regarding not providing pre-specified outcomes however the trial did pre-specify the analysis of outcomes. The primary outcomes were the same as in the original protocol, although the scoring method of one was changed and the analysis of assessing efficacy also changed from the original protocol. These changes were made as part of the detailed statistical analysis plan (itself published in full), which had been promised in the original protocol. These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias. The relevant paper also alerted readers to all these changes and gave the reasons for them. Overall, we don’t think that the issues you raise with regard to the risk of selective outcome bias are such as to suspect high risk of bias, but recognize that you may reach different conclusions than us.
Phoenix Rising thread on the blog is here:Why the Cochrane Collaboration needs to clean up conflicts of interest
he is bascally saying they didn't do anything wrong. But he does sayMost of David Tovey's comments look rather weak and non-committal to me...
I'm not getting excited but at least they seem to be somewhat progressive. At least they replied which is more than The Lancet have done.We recognise the need to update the Cochrane policy in respect of academic conflicts, and this process is in the pipeline –
Most of David Tovey's comments look rather weak and non-committal to me...
This response has some positives on data sharing. He also acknowledges the concerns raised in the quotes above.
I would add that the first link that Tovey uses links to the wrong coyne blog post.
"I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochraneto explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study."
He links to this
Which is not the post that Coyne makes reference to chalder, white, Sharpe and their insurance COIs for the second Cochrane review. This is actually the open letter to Cochrane.
This is important as Tovey states:
"James Coyne states that Lillebeth Larun is employed by an insurance company, but I am unclear on what basis this is determined. "
I don't think anyone accused Larun of having insurance COIs and this is plain wrong.
It is unclear whether these are simple errors in the article or intended to obfuscate. The two Cochrane reviews are confusing and I became confused initially.
The impact is that linking to the wrong article downplays the issues and it appears on reading there is no cause for concern for someone unaware of the issues. The major COIs are for the second Cochrane review that Sharpe Chalder and White are directly involved in.
This blog has been corrected on 04/05/16 to remove misleading content caused by a misunderstanding on the author’s part
Continue reading here.David Tovey is the Editor in Chief of the Cochrane Library, and has been working with the Cochrane Editorial Unit (CEU) and the wider Cochrane collaboration in this role since 2009. In this post, he discusses Cochrane's conflict of interest policy and recent calls for re-assessment of its application in the conduct of Cochrane Reviews.
This blog has been corrected on 04/05/16 to remove misleading content caused by a misunderstanding on the author’s part.
I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochrane to explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study.
Cochrane is still fairly unusual within the journal world in that it specifies that in some cases declaration of interests is necessary but insufficient, and that there are individuals or groups of researchers who are not permitted to proceed with a given systematic review. This has been true since 2004, when Cochrane’s Steering Group ratified a commercial sponsorship policy that described circumstances where authorship as proposed within a review could not go ahead. At the time, Cochrane also introduced the post of Funding Arbiter, reporting directly to its Steering Group, to ensure that the policy was followed, and to rule on ambiguous or disputed cases. As Professor Lisa Bero says “The Cochrane policy is strict because, first, there are no journals that prohibit publication of systematic reviews funded by a company with a financial interest in the outcome of the review. Second, to my knowledge, there are no journals that require the majority of authors to be without personal conflicts of interest, prohibit the first author from having a conflict of interest, or prohibit company employees with a conflict of interest from being an author. For example, the BMJ conflict of interest policy states, ‘We are not aiming to eradicate such interests; they are almost inevitable’ and authors with conflicts of interest are not prohibited from being authors of BMJ original research, systematic reviews or meta-analysis articles. The BMJ does prohibit authors with COI from being authors of: Editorials and education articles (clinical reviews, practice articles, state of the art reviews, Minerva pictures, and Endgames), but these are different from systematic reviews.”
David Tovey (email@example.com, @DavidTovey)
18 April 2016
The Cochrane Official Blog is curated and maintained by the Communications & External Affairs Department. To submit items for publication to the blog or to add comments to a blog, please email firstname.lastname@example.org.
The Cochrane Blog presents commentary and personal opinion on topics of interest from a range of contributors to the work of Cochrane. Opinions posted on the Cochrane Blog are those of the individual contributors and do not necessarily reflect the views or policies of Cochrane.