The big idea: should we get rid of the scientific paper? (Guardian editorial, 2022)

[Pyrrhus]

The big idea: should we get rid of the scientific paper?
https://www.theguardian.com/books/2022/apr/11/the-big-idea-should-we-get-rid-of-the-scientific-paper

Excerpt:

As a format it’s slow, encourages hype, and is difficult to correct. A radical overhaul of publishing could make science better.
[...]
The system comes with big problems. Chief among them is the issue of publication bias: reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. So scientists go to great lengths to hype up their studies, lean on their analyses so they produce “better” results, and sometimes even commit fraud in order to impress those all-important gatekeepers. This drastically distorts our view of what really went on.

There are some possible fixes that change the way journals work. Maybe the decision to publish could be made based only on the methodology of a study, rather than on its results (this is already happening to a modest extent in a few journals). Maybe scientists could just publish all their research by default, and journals would curate, rather than decide, which results get out into the world. But maybe we could go a step further, and get rid of scientific papers altogether.

On a related note, PubMed and other literature search engines now list non-peer-reviewed pre-prints from pre-print servers like https://www.biorxiv.org/ . Some prominent researchers have argued that the use of pre-print servers during the coronavirus epidemic has sped research, collaboration, and progress along much faster than could have possibly happened with traditional peer-review journal publishing.

Since pre-print servers like https://www.biorxiv.org/ allow anyone to submit their findings, allow anyone else to leave a critical review of those findings, and allow the authors to easily correct errors in their findings, some have even asked "do we even need traditional peer-review anymore?"

 

[Rufous McKinney]

I'm recalling having to make considerable amounts of comments when I was asked to do some peer review.

In the types of research papers I was involved in, there is considerable background which isn't not understood by the paper authors and I'd have to provide extensive corrections. Not so much in the data collected as in the Background and Set up for What does any of this mean.

To not have any peer review does not sound good to me.

 

[BrightCandle]

To some extent the gold standard in regards to scientific papers has not been peer review its been replication. The more groups that try to replicate a result and determine it is the case and do studies in and around the same area and find the extent of it the stronger the result gets. An actual peer review can potentially spot errors but its a lot less capable than replication.

Saying that the breaking down of Dunning Kruger and why its not real wasn't found by replication but by review, but certainly not by the peer review for the original paper. Sometimes the error is far better hidden than most realise despite how obvious in hindsight the error was in Dunning Kruger and the effect they showed and peer review wont catch this sort of thing. Publishing the raw data would have shown the statistical error they made however and I can't help but think hiding the data especially in Pyschology is part of the way they make stunning results out of data that show nothing. The raw data would have shown the results to be quite random and the processing was faulty, making me think Dunning and Kruger knew what they were doing when they published.

So I am all in favor of preprints and replication mechanism in place of peer review and if we are going to put money somewhere it should be on independent replication which represents a far better review and test of the original paper. I think the news ought to keep to papers that have been replicated rather than publishing on the basis of preprints however as should most of medicine. But preprints and sharing of data are a lot better for research.

 

[Pyrrhus]

To some extent the gold standard in regards to scientific papers has not been peer review its been replication. The more groups that try to replicate a result and determine it is the case and do studies in and around the same area and find the extent of it the stronger the result gets.

Good point. Some have also argued that publication bias of the traditional model disincentivizes replication, since a replication is much less publishable than original research, no matter what the replication effort eventually concludes.

 

[Rufous McKinney]

Good point. Some have also argued that publication bias of the traditional model disincentivizes replication, since a replication is much less publishable than original research, no matter what the replication effort eventually concludes.

this sounds like a very troubling problem, if thats the case. (I believe you). But its also sort of shocking.

How do you get $$ to replicate a study? Suddenly that doent' sound nearly as exciting so why fund that.

Why is it much less publishable, if the replication is necessary to actually "confirm" outcomes?

 

[Murph]

I think speed is the missing ingredient in scientific progress. If you're in a lab on the far side of the world and about to embark on an experiment, it's far better to see a pre-print from another lab doing a related thing, as soon as their experiemnt is finished. The alternative is to wait 4 years for the paper. It's not as helpful in guiding your work, even if the paper contains a small handful of extra goodies like better background section and additional experiments that Reviewer 2 insisted were important.

I also think that rather than judging a paper on what Masthead it appears in (cell, lancet, etc) it should be rated by a large number of peers for clarity, insight, probability of being true, and on how much detail it shows. If a paper comes with 14TB of 24h/day videos of the rats over 3 months from every angle so a person can confirm the intrventions actually happened as stated, and 100MB of raw data, and the code they used to do their calculations, that might be considered a better paper than a 3 page one with a p<.001 on the first page but no way of checking how they found it.

Ultimately the word "paper" gives us a clue. The scientific paper is just a medium for transferring information about findings. The media we have available have changed dramatically. The capacity to capture, store and transmit info have each changed enormously yet the scientific paper has not changed much at all. It should.

 

[Pyrrhus]

Why is it much less publishable, if the replication is necessary to actually "confirm" outcomes?

Because journals want to entice readers with new, sexy findings. Positive replications are seen as "old news". Negative replications are seen as "controversial".

Unless your replication clearly upends decades of established practice, it's probably not going to be considered sexy enough to print.

 

[Shanti1]

I love the rapid and improved access that preprints provide, but I have to admit when I read preprints, my trust level isn't as high as with a peer-review article. If I cite one in scientific communication, I tend to preface with, "it's not peer-reviewed," to warn others that the error rate could be higher.

This sentiment is expressed on medrxiv: https://www.medrxiv.org/content/what-unrefereed-preprint "Readers should therefore be aware that articles on medRxiv have not been finalized by authors, might contain errors, and report information that has not yet been accepted or endorsed in any way by the scientific or medical community. "

For a non-peer-reviewed system to supplant the traditional peer review, I think we need a peer-review-type process implemented in preprint platforms. It sounds like there is something of that nature in place on the biorxiv.org site. Do you think it is enough? It would be great to get the best of both worlds!