Ed: Note Richard Smith is a former editor of the BMJ
http://blogs.bmj.com/bmj/2010/03/22/richard-smith-scrap-peer-review-and-beware-of-“top-journals”/
Richard Smith: Scrap peer review and beware of "top journals"
22 Mar, 10 | by julietwalker
Richard Smith
The neurologist and epidemiologist Cathie Sudlow has written a highly
readable and important piece in the BMJ exposing Science magazine's poor
reporting of a paper on chronic fatigue syndrome, (1) but she reaches the
wrong conclusions on how scientific publishing should change.
For those of you who have missed the story, Science published a case
control study in September that showed a strong link between chronic
fatigue syndrome and xenotropic murine leukaemia virus-related virus
(XMRV). (2)
The study got wide publicity and was very encouraging to the many people
who believe passionately that chronic fatigue syndrome has an infectious
cause.
Unfortunately, as Sudlow describes, the study lacked basic information on
the selection of cases and controls, and, worse, Science has failed to
publish E-letters from Sudlow and others asking for more information.
In the meantime, three other studies have not found an association between
chronic fatigue syndrome and XMRV. (3-5)
To avoid such poor reporting in the future Sudlow urges strengthening the
status quo-more and better prepublication peer review.
Not only is she trying to close the stable door after the horse has bolted
she has also failed to recognise the possibilities of the new Web 2.0
world.
The time has come to move from a world of "filter then publish" to one of
"publish then filter"-and it's happening.
Prepublication peer review is faith based not evidence based, and Sudlow's
story shows how it failed badly at Science.
Her anecdote joins a mountain of evidence of the failures of peer review:
it is slow, expensive, largely a lottery, poor at detecting errors and
fraud, anti-innovatory, biased, and prone to abuse. (6 7)
As two Cochrane reviews have shown, the upside is hard to demonstrate. (8
9) Yet people like Sudlow who are devotees of evidence persist in belief in
peer review. Why?
The world also seems unaware that it is scientifically dangerous to read
only the "top journals".
As Neal Young and others have argued, the "top journals" publish the sexy
stuff. (10)
The unglamorous is published elsewhere or not at all, and yet the evidence
comprises both the glamorous and the unglamorous.
The nave concept that the "top journals" publish the important stuff and
the lesser journals the unimportant is simply false.
People who do systematic reviews know this well.
Anybody reading only the "top journals" receives a distorted view of the
world-as this Science story illustrates.
Unfortunately many people, including most journalists, do pay most
attention to the "top journals."
So rather than bolster traditional peer review at "top journals," we should
abandon prepublication review and paying excessive attention to "top
journals."
Instead, let people publish and let the world decide.
This is ultimately what happens anyway in that what is published is
digested with some of it absorbed into "what we know" and much of it never
being cited and simply disappearing.
Such a process would have worked better with the story that Sudlow tells.
The initial study would have appeared-perhaps to a fanfare of publicity (as
happened) or perhaps not.
Critics would have immediately asked the questions that Sudlow asks.
Instead of hiding behind Science's skirts as has happened, the authors
would have been obliged to provide answers.
If they couldn't, then the wise would disregard their work.
Then follow up studies could be published rapidly.
Unfortunately, unlike physicists, astronomers, and mathematicians, all of
whom have long published in this way, biomedical researchers seem reluctant
to publish without traditional prepublication peer review.
In reality this is probably because of innate conservatism and the grip of
the "top journals" who insist on prepublication review, but biomedical
researchers often say "But our stuff is different from that of physicists
in that it may scare ordinary people. A false story, for example, "Porridge
causes cancer" can create havoc."
My answer to this objection is that this happens now.
Much of what is published in journals is scientifically poor-as the Science
article shows.
Then, many studies are presented at scientific meetings without peer
review, and scientists and their employers are increasingly likely to
report their results through the mass media.
In a world of "publish then filter" we would at least have the full paper
to dissect, whereas reports in the media even if derived from scientific
meetings, include insufficient information for critical appraisal.
So I urge Sudlow, a thinking woman, to reflect further and begin to argue
for something radical and new rather than more of the same.
1. Sudlow C. Science, chronic fatigue syndrome, and me. BMJ 2010;340:c1260
2. Lombardi VC, Ruscetti FW, Das Gupta J, Pfost MA, Hagen KS, Peterson DL,
et al. Detection of an infectious retrovirus, XMRV, in blood cells of
patients with chronic fatigue syndrome. Science 2009;326:585-9.
3. Van Kuppeveld FJM, de Jong AS, Lanke KH, Verhaegh GW, Melchers WJG,
Swanink CMA, et al. Prevalence of xenotropic murine leukaemia virus-related
virus in patients with chronic fatigue syndrome in the Netherlands:
retrospective analysis of samples from an established cohort. BMJ
2010;340:c1018.
4. Erlwein O, Kaye S, McClure MO, Weber J, Willis G, Collier D, et al.
Failure to detect the novel retrovirus XMRV in chronic fatigue syndrome.
PLoS One 2010;5:e8519.
5. Groom HC, Boucherit VC, Makinson K, Randal E, Baptista S, Hagan S, et
al. Absence of xenotropic murine leukaemia virus-related virus in UK
patients with chronic fatigue syndrome. Retrovirology 2010;7:10.
6. Godlee F, Jefferson T. Peer Review in Health Sciences. 2nd ed. London:
BMJ Books; 2003.
7. Smith R. Peer review: A flawed process at the heart of science and
journals. J R Soc Med 2006;99:178-182.
8. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review
for improving the quality of reports of biomedical studies. Cochrane
Database of Systematic Reviews 2007, Issue 1. Art. No.: MR000016. DOI:
10.1002/14651858.MR000016.pub3
9. Demicheli V, Di Pietrantonj C. Peer review for improving the quality of
grant applications. Cochrane Database of Systematic Reviews 2007, Issue 1.
Art. No.: MR000003. DOI: 10.1002/14651858.MR000003.pub2
10. Young NS, Ioannidis JPA, Al-Ubaydli O, 2008 Why Current Publication
Practices May Distort Science. PLoS Med 5(10): e201.
doi:10.1371/journal.pmed.0050201
Competing interest: RS is on the board of the Public Library of Science and
an enthusiast for open access publishing, but he isn't paid and doesn't
benefit financially from open access publishing.
)
