David Tovey (Editor in Chief of Cochrane Library) comments on Cochrane reviews of CFS

T

Tom Kindlon

Senior Member
Messages
1,734
Part of this relates to part of a comment I submitted on the Cochrane Exercise therapy for CFS review (not the one on individual data that hasn't been published yet)

"Selective reporting (outcome bias)" and White et al. (2011)
I don't believe that White et al. (2011) (the PACE Trial) (3) should be classed as having a low risk of bias under "Selective reporting (outcome bias)" (Figure 2, page 15). According to the Cochrane Collaboration's tool for assessing risk of bias (21), the category of low risk of bias is for: "The study protocol is available and all of the study’s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way". This is not the case in the PACE Trial. The three primary efficacy outcomes can be seen in the published protocol (22). None have been reported in the pre-specified way. The Cochrane Collaboration's tool for assessing risk of bias states that a “high risk” of bias applies if any one of several criteria are met, including that “not all of the study’s pre-specified primary outcomes have been reported” or “one or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g. subscales) that were not pre-specified”. In the PACE Trial, the third primary outcome measure (the number of "overall improvers") was never published. Also, the other two primary outcome measures were reported using analysis methods that were not pre-specified (including switching from the bimodal to the Likert scoring method for The Chalder Fatigue Scale, one of the primary outcomes in your review). These facts mean that the “high risk of bias” category should apply.
Click to expand...

Larun's response:

Selective reporting (outcome bias)

The Cochrane Risk of Bias tool enables the review authors to be transparent about their judgments, but due to the subjective nature of the process it does not guarantee an indisputable consensus. You particularly mention the risk of bias in the PACE trial regarding not providing pre-specified outcomes however the trial did pre-specify the analysis of outcomes. The primary outcomes were the same as in the original protocol, although the scoring method of one was changed and the analysis of assessing efficacy also changed from the original protocol. These changes were made as part of the detailed statistical analysis plan (itself published in full), which had been promised in the original protocol. These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias. The relevant paper also alerted readers to all these changes and gave the reasons for them. Overall, we don’t think that the issues you raise with regard to the risk of selective outcome bias are such as to suspect high risk of bias, but recognize that you may reach different conclusions than us.
Click to expand...

My full submission can be read here:
http://forums.phoenixrising.me/inde...ne-exercise-therapy-for-cfs-review-2015.39801

James Coyne has written a blog challenging part of the response by Lillebeth Larun to my comments:
Why the Cochrane Collaboration needs to clean up conflicts of interest
https://jcoynester.wordpress.com/20...tion-needs-to-clean-up-conflicts-of-interest/
Click to expand...
Phoenix Rising thread on the blog is here:
http://forums.phoenixrising.me/inde...icts-of-interest-on-get-for-cfs-review.43690/
 
BurnA

BurnA

Senior Member
Messages
2,087
Snow Leopard said:
Most of David Tovey's comments look rather weak and non-committal to me...
Click to expand...
he is bascally saying they didn't do anything wrong. But he does say
We recognise the need to update the Cochrane policy in respect of academic conflicts, and this process is in the pipeline –
Click to expand...
I'm not getting excited but at least they seem to be somewhat progressive. At least they replied which is more than The Lancet have done.
 
Y

Yogi

Senior Member
Messages
1,132
This response has some positives on data sharing. He also acknowledges the concerns raised in the quotes above.

I would add that the first link that Tovey uses links to the wrong coyne blog post.


"I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochraneto explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study."

He links to this
https://jcoynester.wordpress.com/20...tion-needs-to-clean-up-conflicts-of-interest/


Which is not the post that Coyne makes reference to chalder, white, Sharpe and their insurance COIs for the second Cochrane review. This is actually the open letter to Cochrane.

https://jcoynester.wordpress.com/20...bill-silverman-lies-a-moldering-in-his-grave/

This is important as Tovey states:

"James Coyne states that Lillebeth Larun is employed by an insurance company, but I am unclear on what basis this is determined. "

I don't think anyone accused Larun of having insurance COIs and this is plain wrong.

It is unclear whether these are simple errors in the article or intended to obfuscate. The two Cochrane reviews are confusing and I became confused initially.

The impact is that linking to the wrong article downplays the issues and it appears on reading there is no cause for concern for someone unaware of the issues. The major COIs are for the second Cochrane review that Sharpe Chalder and White are directly involved in.
 
Esther12

Esther12

Senior Member
Messages
13,774
Snow Leopard said:
Most of David Tovey's comments look rather weak and non-committal to me...
Click to expand...

Yeah. It's positive in that it reflects an openness to dialogue, but in some ways I felt that we were past that being a positive to feel particularly pleased about. I remember thinking Coyne's first blog about this was less good than some of his others too. Maybe I'm missing something to all this though?
 
U

user9876

Senior Member
Messages
4,556
To me issues have been raised with the integrity of the Cochrane review both in terms of the protocol for the individual patient data review where the protocol development included the PACE PIs and also the GET review where they defended the outcome switching that PACE did and judged it as if the original outcomes had been published.

Tovey's response is very weak he basically defends the status quo. He does not say he will look further at the issues. He should be looking at how the individual patient review ended up with his with the Pace PIs helping design and fund the protocol; how the Cochrane brand is being used to claim this is independent and he needs to look at how the PACE trial is being mis-classified and outcome switching ignored.

But more than that he needs to look at the entire Cochrane Common Mental disorders group. How is it that these issues were not picked up on and ignored. Are their other issues in other areas where patients are less vocal. How does he fix the group so that the integrity of reviews is brought up to standard and maintained. Otherwise the Cochrane brand is worthless and his inaction is helping to make it so.
 
A.B.

A.B.

Senior Member
Messages
3,780
Cynical me thinks the Cochrane Common Mental Disorders group exists because standards are lower in mental health. Just like with PACE elsewhere, the people in this area don't even seem to realize there is a problem, being so used to fraudulent junk science.

Jonathan Edwards mentioned speaking with a psychiatrist close to PACE who did not understand why lack of blinding plus reliance on subjective outcomes was a problem. Willful ignorance or shocking incompetence?
 
Y

Yogi

Senior Member
Messages
1,132
Yogi said:
This response has some positives on data sharing. He also acknowledges the concerns raised in the quotes above.

I would add that the first link that Tovey uses links to the wrong coyne blog post.


"I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochraneto explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study."

He links to this
https://jcoynester.wordpress.com/20...tion-needs-to-clean-up-conflicts-of-interest/


Which is not the post that Coyne makes reference to chalder, white, Sharpe and their insurance COIs for the second Cochrane review. This is actually the open letter to Cochrane.

https://jcoynester.wordpress.com/20...bill-silverman-lies-a-moldering-in-his-grave/

This is important as Tovey states:

"James Coyne states that Lillebeth Larun is employed by an insurance company, but I am unclear on what basis this is determined. "

I don't think anyone accused Larun of having insurance COIs and this is plain wrong.

It is unclear whether these are simple errors in the article or intended to obfuscate. The two Cochrane reviews are confusing and I became confused initially.

The impact is that linking to the wrong article downplays the issues and it appears on reading there is no cause for concern for someone unaware of the issues. The major COIs are for the second Cochrane review that Sharpe Chalder and White are directly involved in.
Click to expand...


David Tovey has updated his blog.

https://twitter.com/i/web/status/727869799448088577

It would be good to compare to the original version.

At first glance it appears the misleading part about Coyne and Lillibeth Larun has been removed as stated above.

This blog has been corrected on 04/05/16 to remove misleading content caused by a misunderstanding on the author’s part
Click to expand...

How can we compare against the first version if there are any other changes?

David Tovey is the Editor in Chief of the Cochrane Library, and has been working with the Cochrane Editorial Unit (CEU) and the wider Cochrane collaboration in this role since 2009. In this post, he discusses Cochrane's conflict of interest policy and recent calls for re-assessment of its application in the conduct of Cochrane Reviews.

This blog has been corrected on 04/05/16 to remove misleading content caused by a misunderstanding on the author’s part.

DTovey%200416.jpg
I didn’t know Bill Silverman, so I can’t judge whether he would be “a-mouldering in his grave”. However, I recognise that James Coyne has set down a challenge to Cochrane to explain its approach to commercial and academic conflicts of interest and also to respond to criticisms made in relation to the appraisal of the much debated PACE study.

Cochrane is still fairly unusual within the journal world in that it specifies that in some cases declaration of interests is necessary but insufficient, and that there are individuals or groups of researchers who are not permitted to proceed with a given systematic review. This has been true since 2004, when Cochrane’s Steering Group ratified a commercial sponsorship policy that described circumstances where authorship as proposed within a review could not go ahead. At the time, Cochrane also introduced the post of Funding Arbiter, reporting directly to its Steering Group, to ensure that the policy was followed, and to rule on ambiguous or disputed cases. As Professor Lisa Bero says “The Cochrane policy is strict because, first, there are no journals that prohibit publication of systematic reviews funded by a company with a financial interest in the outcome of the review. Second, to my knowledge, there are no journals that require the majority of authors to be without personal conflicts of interest, prohibit the first author from having a conflict of interest, or prohibit company employees with a conflict of interest from being an author. For example, the BMJ conflict of interest policy states, ‘We are not aiming to eradicate such interests; they are almost inevitable’ and authors with conflicts of interest are not prohibited from being authors of BMJ original research, systematic reviews or meta-analysis articles. The BMJ does prohibit authors with COI from being authors of: Editorials and education articles (clinical reviews, practice articles, state of the art reviews, Minerva pictures, and Endgames), but these are different from systematic reviews.”

David Tovey (dtovey@cochrane.org, @DavidTovey)
18 April 2016
The Cochrane Official Blog is curated and maintained by the Communications & External Affairs Department. To submit items for publication to the blog or to add comments to a blog, please email news@cochrane.org.

The Cochrane Blog presents commentary and personal opinion on topics of interest from a range of contributors to the work of Cochrane. Opinions posted on the Cochrane Blog are those of the individual contributors and do not necessarily reflect the views or policies of Cochrane.
Click to expand...
Continue reading here.

This blog still only links to one of Coyne blogs (20th March) rather than 6th March which more clearly demonstrates the insurance COIs and themselves responsible for authoring the Cochrane review.

Thank you to David Tovey for correcting the errors in the blog. However it begs the question have Cochrane appropriately and sufficiently addressed the issue of Conflicts of Interests with White, Sharpe and Chalder given this fundamental confusion over who actually has the serious COIs?

Is it not standard practice that it should it be made clear when edits or removing inaccurate statements in blogs that these should actually be stated clearly?

Thank you also to Richard Smith (former editor of BMJ) for retweeting as well to his followers.
 
Last edited by a moderator:
You must log in or register to reply here.
Back