PACE (again) peer review

[JohntheJack]

My understanding is that criticisms have been made of the peer review process by 'The Lancet'. I've seen J Coyne's blog on the conflict of interest. Could someone, please, point me in the direction of any more criticisms.

From what I can see there was an original fast-track (which in itself was dubious) and then a second review.

Is there anything dodgy about either?

Thanks.

 

[Stewart]

My understanding is that criticisms have been made of the peer review process by 'The Lancet'. I've seen J Coyne's blog on the conflict of interest. Could someone, please, point me in the direction of any more criticisms.

Did you mean this blog post by Coyne, explaining how the sheer number of people involved in PACE mean that the reviewers will almost certainly have had social or institutional connections to members of the PACE team (and therefore not be truly independent)? Or this one where he looks at Peter White's declared (and undeclared) conflicts of interest?

From what I can see there was an original fast-track (which in itself was dubious) and then a second review.

Is there anything dodgy about either?

Thanks.

I don't think there was a second review - I have no recollection of ever reading about a review having taken place - although if you have evidence that there was one then I'm obviously mistaken.

The biggest complaint I've read about the peer review process is the way that the Lancet has tried to pass it off as being way more robust than it actually is. Richard Horton claimed the PACE paper underwent 'endless' rounds of peer review - yet the paper was actually fast-tracked, meaning that it was published 28 days (or less) after it was received. This Lancet article from 1998 explains how, right from the introduction of the fast-track process, peer-reviewers were expected to respond within 48 hours - which doesn't exactly encourage detailed forensic analysis, and perhaps goes some way to explaining how the reviewers might have missed some of the issues with PACE (although not the most glaring ones).

By early 2015 the Lancet was proudly proclaiming that it had got the fast-track process for RCTs down to 20 days - just 10 days to conduct the peer-review and make a decision (!) and then another 10 days from acceptance to publication. It definitely appears that speed is being prioritised over thoroughness. How long did they spend on the peer-review process for PACE? A leisurely 28 days or an almost indecently hasty 20? And how many rounds of robust peer review can you realistically cram into that window?

 

[Esther12]

I don't think there was a second review - I have no recollection of ever reading about a review having taken place - although if you have evidence that there was one then I'm obviously mistaken.

There have been vague references to internal reviews, the Horton quote you mentioned about 'endless' rounds of peer review. The Lancet responded to the PCC investigation into the 30% recovery claim with some evasive BS.

 

[JohntheJack]

Did you mean this blog post by Coyne, explaining how the sheer number of people involved in PACE mean that the reviewers will almost certainly have had social or institutional connections to members of the PACE team (and therefore not be truly independent)? Or this one where he looks at Peter White's declared (and undeclared) conflicts of interest?



I don't think there was a second review - I have no recollection of ever reading about a review having taken place - although if you have evidence that there was one then I'm obviously mistaken.

The biggest complaint I've read about the peer review process is the way that the Lancet has tried to pass it off as being way more robust than it actually is. Richard Horton claimed the PACE paper underwent 'endless' rounds of peer review - yet the paper was actually fast-tracked, meaning that it was published 28 days (or less) after it was received. This Lancet article from 1998 explains how, right from the introduction of the fast-track process, peer-reviewers were expected to respond within 48 hours - which doesn't exactly encourage detailed forensic analysis, and perhaps goes some way to explaining how the reviewers might have missed some of the issues with PACE (although not the most glaring ones).

By early 2015 the Lancet was proudly proclaiming that it had got the fast-track process for RCTs down to 20 days - just 10 days to conduct the peer-review and make a decision (!) and then another 10 days from acceptance to publication. It definitely appears that speed is being prioritised over thoroughness. How long did they spend on the peer-review process for PACE? A leisurely 28 days or an almost indecently hasty 20? And how many rounds of robust peer review can you realistically cram into that window?

Thanks, Stewart. Yes, that was the Coyne blog I had in mind.

The whole fast-tracking does seem in itself questionable.

Yes, there was a second review:
'The Lancet, in response to extensive public commentary, in an unusual procedure, subjected the study to a further peer review process.'

 

[Jonathan Edwards]

Thanks, Stewart. Yes, that was the Coyne blog I had in mind.

The whole fast-tracking does seem in itself questionable.

Yes, there was a second review:
'The Lancet, in response to extensive public commentary, in an unusual procedure, subjected the study to a further peer review process.'

I am not sure that there is anything wrong in fast-tracking. A rigorous peer review, sufficient to identify the sorts of flaws there are in PACE, takes about an hour. My practice was to review manuscripts the day I received them if I could, but certainly within two or three days. There is absolutely no need for prolonged scrutiny here. The problem is that all the people likely to be doing the review were in a branch of medicine that seems to have no understanding of basic principles of reliable evidence. Simon Wessely made it clear to me he had no understanding, as have the PACE authors themselves, repeatedly.

The Lancet used to pride itself on being so knowledgeable in its editorial office not to need formal peer review. It moved to formal peer review but kept th practice of asking chums in the in crowd. How the Wakefield and the tracheal transplant papers got through goodness knows but I think they show how unreliable the whole business is.

 

[user9876]

I am not sure that there is anything wrong in fast-tracking. A rigorous peer review, sufficient to identify the sorts of flaws there are in PACE, takes about an hour. My practice was to review manuscripts the day I received them if I could, but certainly within two or three days. There is absolutely no need for prolonged scrutiny here. The problem is that all the people likely to be doing the review were in a branch of medicine that seems to have no understanding of basic principles of reliable evidence. Simon Wessely made it clear to me he had no understanding, as have the PACE authors themselves, repeatedly.

The Lancet used to pride itself on being so knowledgeable in its editorial office not to need formal peer review. It moved to formal peer review but kept th practice of asking chums in the in crowd. How the Wakefield and the tracheal transplant papers got through goodness knows but I think they show how unreliable the whole business is.

Should there be a different question around who reviewed the protocol prior to the trial. Its a bit late after spending X million running a trial that is not capable of getting meaningful results to apply peer review. This makes me wonder about the process for monitoring the development of the trial and whether external review is really applied or whether the PIs choose friends.

 

[JohntheJack]

I am not sure that there is anything wrong in fast-tracking. A rigorous peer review, sufficient to identify the sorts of flaws there are in PACE, takes about an hour. My practice was to review manuscripts the day I received them if I could, but certainly within two or three days. There is absolutely no need for prolonged scrutiny here. The problem is that all the people likely to be doing the review were in a branch of medicine that seems to have no understanding of basic principles of reliable evidence. Simon Wessely made it clear to me he had no understanding, as have the PACE authors themselves, repeatedly.

The Lancet used to pride itself on being so knowledgeable in its editorial office not to need formal peer review. It moved to formal peer review but kept th practice of asking chums in the in crowd. How the Wakefield and the tracheal transplant papers got through goodness knows but I think they show how unreliable the whole business is.

Thanks, Jonathan. That's interesting.

Yes, I'd thought of the Wakefield and the Macchiarini stuff.

 

[Jonathan Edwards]

Should there be a different question around who reviewed the protocol prior to the trial. Its a bit late after spending X million running a trial that is not capable of getting meaningful results to apply peer review. This makes me wonder about the process for monitoring the development of the trial and whether external review is really applied or whether the PIs choose friends.

For sure, the trial should never have passed ethics committee approval or planning committees. The problem is that if a large proportion of people in medical science are not intelligent enough to understand basic principles of reliable evidence we are stuck. Peer review will be by other people who do not understand. If there is a political agenda in the offing, as there clearly was here if DWP were involved, there is little chance of anything intelligent coming out of it all.

 

[Stewart]

Yes, there was a second review:
'The Lancet, in response to extensive public commentary, in an unusual procedure, subjected the study to a further peer review process.'

Hmm. I take it that quote comes from this ICO decision in 2013?

Given that it seems to be QMUL who made this claim to the Information Commissioner and that (as far as I'm aware) The Lancet has never made a public statement to this effect, there's no way of knowing whether it's true. The most likely time for The Lancet to carry out a second peer review of the PACE study would have been back in 2011, when the initial publication met such an angry backlash - but if they'd done that, you would have expected Horton to use this fact when belittling patient concerns in his subsequent editorial (which he didn't). I would have also thought that if a second peer review had taken place by the start of 2013, he would have used this as a justification for dismissing the Virology blog's open letter in 2015/6 which urged The Lancet to "to seek an independent re-analysis of the individual-level PACE trial data" (on the grounds that an independent re-analysis had already been carried out) - but again he didn't do that and instead gave a much weaker response (which made him look evasive).

I suspect that QMUL may have - either intentionally or unintentionally - misled the Commissioner when responding to this FoI appeal. I obviously can't be sure, as I haven't seen QMUL's submission to the Information Commissioner, but I wonder if they gave the impression that The Lancet's anomalous decison to publish a wide selection of critical letters - along with the authors' response - in May 2011 was in some way the equivalent of a second round of peer review (which of course it wasn't)?

The problem is that all the people likely to be doing the review were in a branch of medicine that seems to have no understanding of basic principles of reliable evidence. Simon Wessely made it clear to me he had no understanding, as have the PACE authors themselves, repeatedly.

The Lancet used to pride itself on being so knowledgeable in its editorial office not to need formal peer review. It moved to formal peer review but kept th practice of asking chums in the in crowd. How the Wakefield and the tracheal transplant papers got through goodness knows but I think they show how unreliable the whole business is.

I think there's a question as to what degree the self-imposed deadlines of The Lancet's fast-track process exacerbate this problem. It *must* discourage them from taking time to identify and then make contact with someone who would be a knowledgable reviewer for a particular paper, and instead make them more dependent on their regular pool of reviewers that they know will get back to them quickly.

And although it has nothing to do with PACE, this blog post provides an interesting summary of a case where The Lancet got their selection of peer reviewers seriously wrong (and then failed miserably to clean up the mess that was created as a consequence).

 

[JohntheJack]

Hmm. I take it that quote comes from this ICO decision in 2013?

Given that it seems to be QMUL who made this claim to the Information Commissioner and that (as far as I'm aware) The Lancet has never made a public statement to this effect, there's no way of knowing whether it's true. The most likely time for The Lancet to carry out a second peer review of the PACE study would have been back in 2011, when the initial publication met such an angry backlash - but if they'd done that, you would have expected Horton to use this fact when belittling patient concerns in his subsequent editorial (which he didn't). I would have also thought that if a second peer review had taken place by the start of 2013, he would have used this as a justification for dismissing the Virology blog's open letter in 2015/6 which urged The Lancet to "to seek an independent re-analysis of the individual-level PACE trial data" (on the grounds that an independent re-analysis had already been carried out) - but again he didn't do that and instead gave a much weaker response (which made him look evasive).

I suspect that QMUL may have - either intentionally or unintentionally - misled the Commissioner when responding to this FoI appeal. I obviously can't be sure, as I haven't seen QMUL's submission to the Information Commissioner, but I wonder if they gave the impression that The Lancet's anomalous decison to publish a wide selection of critical letters - along with the authors' response - in May 2011 was in some way the equivalent of a second round of peer review (which of course it wasn't)?

Yes, it is from that decision.

Maybe. That is an interesting possibility.

 

[Esther12]

There other ICO judgements that contain clearly inaccurate statements that favour PACE/QMUL, so it is possible that they were misled.

I think that the Lancet should make their internal deliberations public so that we can better understand what went wrong... not expecting that to happen though!

 

[RogerBlack]

I think there's a question as to what degree the self-imposed deadlines of The Lancet's fast-track process exacerbate this problem. It *must* discourage them from taking time to identify and then make contact with someone who would be a knowledgable reviewer for a particular paper, and instead make them more dependent on their regular pool of reviewers that they know will get back to them quickly.

I'm not sure I agree with this.
It presumes, for example, that less delay means less scrutiny.
It could also mean you've put a couple more members of staff on it, or streamlined your procedure so you throw away 20% of the papers initially.

"less delay" does not mean "less work done", in a queue of items where you've got to get to all items.

One reason that might worsen quality with faster reviews might be if reviewers take a while to agree, but that's not due to 'carefully selecting'.
I would be frankly astonished if editors took the time to do systematic reviews of the literature before selecting referees.

Roll on actual open peer review, or perhaps better, collaborative peer review.
http://www.nature.com/nature/peerreview/debate/nature04988.html

This approach is pursued by the open-access journal Atmospheric Chemistry and Physics and a growing number of sister journals published by the European Geosciences Union. These journals have a two-stage publication process with public peer review and interactive discussion5.

In the first stage, manuscripts that pass a rapid pre-screening (access review) are immediately published as 'discussion papers' on the journal's website. They are then subject to interactive public discussion for a period of 8 weeks, during which the comments of designated reviewers, additional comments by other interested members of the scientific community, and the authors' replies are published alongside the discussion paper. Reviewers can choose to sign their comments or remain anonymous, but comments by other scientists must be signed.

In the second stage, manuscript revision and peer review are completed in the same way as in traditional journals (with further rounds of non-public revisions and review where required). If accepted, final papers are published in the main journal. To provide a lasting record of review and to secure the authors' publication precedence, every discussion paper and interactive comment remains permanently archived and individually citable.

 

[Wolfiness]

Anyone posted this yet?

https://www.coyneoftherealm.com/blo...tors-article-that-the-authors-refused-to-heed