The 12th Invest in ME Conference, Part 1
OverTheHills presents the first article in a series of three about the recent 12th Invest In ME international Conference (IIMEC12) in London.
Discuss the article on the Forums.

PACE (again) peer review

Discussion in 'General ME/CFS News' started by JohntheJack, Jun 10, 2017.

  1. JohntheJack

    JohntheJack Senior Member

    Messages:
    178
    Likes:
    946
    Swansea, UK
    My understanding is that criticisms have been made of the peer review process by 'The Lancet'. I've seen J Coyne's blog on the conflict of interest. Could someone, please, point me in the direction of any more criticisms.

    From what I can see there was an original fast-track (which in itself was dubious) and then a second review.

    Is there anything dodgy about either?

    Thanks.
     
  2. Stewart

    Stewart Senior Member

    Messages:
    291
    Likes:
    3,432
    Did you mean this blog post by Coyne, explaining how the sheer number of people involved in PACE mean that the reviewers will almost certainly have had social or institutional connections to members of the PACE team (and therefore not be truly independent)? Or this one where he looks at Peter White's declared (and undeclared) conflicts of interest?

    I don't think there was a second review - I have no recollection of ever reading about a review having taken place - although if you have evidence that there was one then I'm obviously mistaken.

    The biggest complaint I've read about the peer review process is the way that the Lancet has tried to pass it off as being way more robust than it actually is. Richard Horton claimed the PACE paper underwent 'endless' rounds of peer review - yet the paper was actually fast-tracked, meaning that it was published 28 days (or less) after it was received. This Lancet article from 1998 explains how, right from the introduction of the fast-track process, peer-reviewers were expected to respond within 48 hours - which doesn't exactly encourage detailed forensic analysis, and perhaps goes some way to explaining how the reviewers might have missed some of the issues with PACE (although not the most glaring ones).

    By early 2015 the Lancet was proudly proclaiming that it had got the fast-track process for RCTs down to 20 days - just 10 days to conduct the peer-review and make a decision (!) and then another 10 days from acceptance to publication. It definitely appears that speed is being prioritised over thoroughness. How long did they spend on the peer-review process for PACE? A leisurely 28 days or an almost indecently hasty 20? And how many rounds of robust peer review can you realistically cram into that window?
     
    Last edited: Jun 10, 2017
    sb4, atleje, JohntheJack and 2 others like this.
  3. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,522
    There have been vague references to internal reviews, the Horton quote you mentioned about 'endless' rounds of peer review. The Lancet responded to the PCC investigation into the 30% recovery claim with some evasive BS.
     
  4. JohntheJack

    JohntheJack Senior Member

    Messages:
    178
    Likes:
    946
    Swansea, UK
    Thanks, Stewart. Yes, that was the Coyne blog I had in mind.

    The whole fast-tracking does seem in itself questionable.

    Yes, there was a second review:
    'The Lancet, in response to extensive public commentary, in an unusual procedure, subjected the study to a further peer review process.'
     
  5. Jonathan Edwards

    Jonathan Edwards "Gibberish"

    Messages:
    5,227
    Likes:
    31,871
    I am not sure that there is anything wrong in fast-tracking. A rigorous peer review, sufficient to identify the sorts of flaws there are in PACE, takes about an hour. My practice was to review manuscripts the day I received them if I could, but certainly within two or three days. There is absolutely no need for prolonged scrutiny here. The problem is that all the people likely to be doing the review were in a branch of medicine that seems to have no understanding of basic principles of reliable evidence. Simon Wessely made it clear to me he had no understanding, as have the PACE authors themselves, repeatedly.

    The Lancet used to pride itself on being so knowledgeable in its editorial office not to need formal peer review. It moved to formal peer review but kept th practice of asking chums in the in crowd. How the Wakefield and the tracheal transplant papers got through goodness knows but I think they show how unreliable the whole business is.
     
  6. user9876

    user9876 Senior Member

    Messages:
    2,583
    Likes:
    18,182
    Should there be a different question around who reviewed the protocol prior to the trial. Its a bit late after spending X million running a trial that is not capable of getting meaningful results to apply peer review. This makes me wonder about the process for monitoring the development of the trial and whether external review is really applied or whether the PIs choose friends.
     
  7. JohntheJack

    JohntheJack Senior Member

    Messages:
    178
    Likes:
    946
    Swansea, UK
    Thanks, Jonathan. That's interesting.

    Yes, I'd thought of the Wakefield and the Macchiarini stuff.
     
  8. Jonathan Edwards

    Jonathan Edwards "Gibberish"

    Messages:
    5,227
    Likes:
    31,871
    For sure, the trial should never have passed ethics committee approval or planning committees. The problem is that if a large proportion of people in medical science are not intelligent enough to understand basic principles of reliable evidence we are stuck. Peer review will be by other people who do not understand. If there is a political agenda in the offing, as there clearly was here if DWP were involved, there is little chance of anything intelligent coming out of it all.
     
    GreyOwl, Wolfiness, atleje and 7 others like this.
  9. Stewart

    Stewart Senior Member

    Messages:
    291
    Likes:
    3,432
    Hmm. I take it that quote comes from this ICO decision in 2013?

    Given that it seems to be QMUL who made this claim to the Information Commissioner and that (as far as I'm aware) The Lancet has never made a public statement to this effect, there's no way of knowing whether it's true. The most likely time for The Lancet to carry out a second peer review of the PACE study would have been back in 2011, when the initial publication met such an angry backlash - but if they'd done that, you would have expected Horton to use this fact when belittling patient concerns in his subsequent editorial (which he didn't). I would have also thought that if a second peer review had taken place by the start of 2013, he would have used this as a justification for dismissing the Virology blog's open letter in 2015/6 which urged The Lancet to "to seek an independent re-analysis of the individual-level PACE trial data" (on the grounds that an independent re-analysis had already been carried out) - but again he didn't do that and instead gave a much weaker response (which made him look evasive).

    I suspect that QMUL may have - either intentionally or unintentionally - misled the Commissioner when responding to this FoI appeal. I obviously can't be sure, as I haven't seen QMUL's submission to the Information Commissioner, but I wonder if they gave the impression that The Lancet's anomalous decison to publish a wide selection of critical letters - along with the authors' response - in May 2011 was in some way the equivalent of a second round of peer review (which of course it wasn't)?

    I think there's a question as to what degree the self-imposed deadlines of The Lancet's fast-track process exacerbate this problem. It *must* discourage them from taking time to identify and then make contact with someone who would be a knowledgable reviewer for a particular paper, and instead make them more dependent on their regular pool of reviewers that they know will get back to them quickly.

    And although it has nothing to do with PACE, this blog post provides an interesting summary of a case where The Lancet got their selection of peer reviewers seriously wrong (and then failed miserably to clean up the mess that was created as a consequence).
     
  10. JohntheJack

    JohntheJack Senior Member

    Messages:
    178
    Likes:
    946
    Swansea, UK
    Yes, it is from that decision.

    Maybe. That is an interesting possibility.
     
    Valentijn and Esther12 like this.
  11. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,522
    There other ICO judgements that contain clearly inaccurate statements that favour PACE/QMUL, so it is possible that they were misled.

    I think that the Lancet should make their internal deliberations public so that we can better understand what went wrong... not expecting that to happen though!
     
    Valentijn and SamanthaJ like this.
  12. RogerBlack

    RogerBlack Senior Member

    Messages:
    883
    Likes:
    2,864
    I'm not sure I agree with this.
    It presumes, for example, that less delay means less scrutiny.
    It could also mean you've put a couple more members of staff on it, or streamlined your procedure so you throw away 20% of the papers initially.

    "less delay" does not mean "less work done", in a queue of items where you've got to get to all items.

    One reason that might worsen quality with faster reviews might be if reviewers take a while to agree, but that's not due to 'carefully selecting'.
    I would be frankly astonished if editors took the time to do systematic reviews of the literature before selecting referees.

    Roll on actual open peer review, or perhaps better, collaborative peer review.
    http://www.nature.com/nature/peerreview/debate/nature04988.html
     
    Last edited: Jun 11, 2017
  13. Wolfiness

    Wolfiness Activity Level 0

    Messages:
    455
    Likes:
    1,945
    UK

See more popular forum discussions.

Share This Page