BMJ blog on problems with peer review [Jul 14] with links to similar articles

[Esther12]

Sometimes when discussing problems around CFS research, one can run in to people who talk as if peer review is rather more effective than it seems to be. Amongst researchers there's far more of an awareness of the problems here, and this might provide useful references to those who want to make that point:

http://blogs.bmj.com/bmj/2014/07/31/jane-feinmann-a-way-forward-for-quality-peer-review/

It's also a nice easy to read summary.

 

[alex3619]

But peer reviewers need more than CONSORT, Professor Altman told The BMJ. “This is a guideline for reporting what was done, but [it] does not give guidance on how to judge whether what was done was sensible. And we can see very clear reporting of very bad, biased methods,” he explains. Which leads us back to Patel’s essential point: reviewers need to learn “to spot fundamental flaws,” and be tested on this skill on a regular basis.

My underlining. Like any studies we know? Where was this training for the PACE trial? We need reviewers who know what they are doing, and sadly its fairly rare in some areas of medicine.

 

[Esther12]

My underlining. Like any studies we know? Where was this training for the PACE trial? We need reviewers who know what they are doing, and sadly its fairly rare in some areas of medicine.

When the peer reviewers have themselves built their careers upon positive claims about the treatments being assessed, they may not be that helpful even with the right training!

 

[alex3619]

When the peer reviewers have themselves built their careers upon positive claims about the treatments being assessed, they may not be that helpful even with the right training!

Yes, this is an obvious caveat. However this arises in part because reviewers are chosen from within a school of thought. Other independent psychiatrists and psychologists could be chosen, who have the right training.

I am concerned though that psychiatry is so pervasively filled with bad research methodology that it might require a non-psychiatrist reviewer. Even medicine is not such a great place. A medical scientist might be the best choice as reviewer, not a doctor. However this is sufficiently removed from psychiatry that I expect pushback from psychiatrists who claim such a reviewer is not qualified.

I think the answer to this is to have methodology reviewers and content reviewers. A methodology reviewer would be a scientist with the appropriate training, including statistics. A content reviewer would be an independent psychiatrist.

 

[Sean]

Probably the single most effective change to the peer review process that would help overcome these problems is to abandon anonymous reviewing. The lack of transparency and accountability arising from anonymity is the main reason poor quality reviewing persists.

 

[alex3619]

Probably the single most effective change to the peer review process that would help overcome these problems is to abandon anonymous reviewing. The lack of transparency and accountability arising from anonymity is the main reason poor quality reviewing persists.

That is part of the mix I would recommend.