shock: Top journal publishes failure to replicate key finding

[Simon]

this is a bit of an opinion piece by me, but it is short

Replication of studies really is the gold standard of scientific research, yet rarely happens leaving a scientific literature that is awash with interesting but unconfirmed findings (see Once Is Not Enough). It's a huge problem in CFS research. One big reason that people don't even attempt repliction - across life sciences - is that journals want novelty, and replication is the exact opposite of that. But here is a case where it did happen:

Nature Biotechnology has made a lot of noise about publishing a replication study with no novelty value, though as they point out it has considerable importance. The new study represents a failure to replicate a very surprising finding, which implied the gene regulation mechanisms inside some plants, if eaten, could affect human gene expression too. Think they were also having a pop at a rival journal (Cell Research) that published the original paper, and turned down the failed replication on the grounds that “it is a bit hard to publish a paper of which the results are largely negative.” Sadly, this attitude is all too common.

Replication is a difficult and thankless task. Until now, journals, funders and academics have shown little interest in it. Nature Biotechnology will remain open to publishing replication studies and rigorous efforts that fail to reproduce findings from other publications of high interest to our readers. It is our view, however, that the best practice is to publish such replication failures in the journal where the original findings were published. That way, the power of the scientific process to consolidate and modify our understanding of initial findings in a report is clearly visible to all.

Perhaps this high profile publication will make it easier for researchers to publish replication studies, and that will be doing science a big favour.

 

[Sparrowhawk]

Thanks @Simon, that's refreshing and encouraging to see such an editorial stance, and furthermore the decision to publish, thus admonishing others to do the same.

 

[Bob]

Speaking of good practice, the ME Association Tweeted a link to this today:
http://blogs.biomedcentral.com/bmcb...r-review-work-biology-direct-suggests-it-can/

It's a blog by another 'bio' journal: BioMed Central.
It says that, for transparency, they publish all peer reviewer comments/reports and the names of peer reviewers, alongside the final published paper.

 

[Esther12]

Speaking of good practice, the ME Association Tweeted a link to this today:
http://blogs.biomedcentral.com/bmcb...r-review-work-biology-direct-suggests-it-can/

It's a blog by another 'bio' journal: BioMed Central.
It says that, for transparency,they publish all peer reviewer comments/reports and the names of peer reviewers, alongside the final published paper.

There really needs to be more openness around peer review. Some journal's processes are an absolute joke.

 

[barbc56]

I though replication is a key component of science.

Yes, absolute transparency!

Very interesting and an important question.

 

[alex3619]

Perhaps this high profile publication will make it easier for researchers to publish replication studies, and that will be doing science a big favour.

I substantially agree with your arguments. There is a move to have online journals for replication studies that have only technical requirements for submission, basically that a study is scientifically sound. So it doesn't matter if its popular, or publishable in a paper journal, the study is still available and can be found by searching or indexing engines. This is an essential step to create continued progress in science. Otherwise science risks becoming just another failed religion.

 

[barbc56]

There's also Pub. Med but it's not as user friendly.

 

[Simon]

There is a move to have online journals for replication studies that have only technical requirements for submission, basically that a study is scientifically sound. So it doesn't matter if its popular, or publishable in a paper journal, the study is still available and can be found by searching or indexing engines. This is an essential step to create continued progress in science.

I'd heard that the worthy goal of a 'replication' journal was struggling to attract any interest from publishers or editors, despite it's obvious usefulness. PLoS One will take anything deemed 'methodologically sound', regardless of importance, and Jonathan Kerr publishd his failed replication of earlier CFS 'identifier' gene expression findings there. But authors have to pay to publish, making it an even less attractive option.

The Nature Biotech editorial points to some other interesting developments though:

• one idea being floated by certain funders is to set aside a portion of a research grant specifically for independent verification of the main study's results before publication; in this scheme, submission to a journal would proceed only after the results were corroborated.
• This summer, the journal Cortex started offering yet another means of improving reproducibility and reducing bias. The mechanism, termed a “Registered Report,” involves peer review of an investigator's experimental design before data are collected. If the scientific question and methods are deemed sound, then authors are offered “in principle acceptance” of their article, irrespective of the study's outcome. [I think pre-review of methodology is in any case a great way to improve research]
  
• The Reproducibility Initiative, A collaboration between the Science Exchange and PLOS ONE, the initiative offers to broker independent validation of a researcher's work in return for a fee, with subsequent publication in the journal. In October, the Laura and John Arnold Foundation provided $1.3 million to the initiative to authenticate 50 high-profile cancer papers from the past two years (only ~$20,000 per study).

 

[Gijs]

Falsification is the key factor for science.

 

[Firestormm]

Is this not a problem only for those seeking professional advancement through publication? What I mean to say is, and to take let's say Rituximab as an example; it is increasingly apparent to me that private funding institutions are now taking over a greater share of the funding of science.

In our little world, any 'interesting' papers, should now see their replication attempts funded without too much effort I would think. Unless of course those holding the money are also uninterested in funding replication studies?

I can see that in the scientific world of competing academics there might be little appeal in replication when it comes to attracting a publisher - though this seems completely daft to me as well - but in our world: there should be less of a barrier to willingness.

Or maybe this does explain why - even when funding is available - few seem willing to approach with a proposal. I mean there is an ocean of 'interesting' studies on ME that should surely be replicated. I just can't see that this publication bias is the only barrier to not replicating.

@Jonathan Edwards Do you have any thoughts about this? Thanks

 

[Hip]

In terms of lack of replication, the fact that Dr John Chia's findings of enterovirus infections in the stomach of ME/CFS patients have not been replicated is a major problem.

Chia's methods were solid, and he demonstrated a strong connection between enterovirus infection and ME/CFS.

Yet in spite of the fact that this enterovirus link is the only real lead lead we have on viral ME/CFS, nobody has bothered to replicate Chia's work. This is crazy. During the XMRV debacle, there were loads of replication studies (far more than were actually needed); yet for some inexplicable reason, nobody has thought to replicate Chia's excellent enterovirus results. Why didn't just one of these researchers try to replicate the enterovirus studies, instead of wasting time and money on an excess amount of XMRV study replications?

Nobody in the ME/CFS community even talks much about these enterovirus results. I cannot understand why there is this lack of interest. Do we want a cure or not?

 

[Sparrowhawk]

Good bloody question, @Hip. Definitely agree it's worth pursuing.

 

[alex3619]

Falsification is the key factor for science.

Falsification is what distinguishes science from nonscience. Freudian psychiatry was a classic case. It could not be properly tested, so Karl Popper called it nonscience, and I think Kuhn agreed - which is itself remarkable because Kuhn and Popper operated in competing paradigms.

I agree we need the enterovirus findings replicated. However to some extent Chia's findings were the actual replication. This research goes back to the 1980s. There was evidence of coxsackie virus in muscle, I think it was viral nucleotides. Indeed after I tested positive an article on this was handed to my by my doctor, who said that CFS (she was up to date on Holmes at least in 1989) might be caused by an enterovirus. This is not a new hypothesis.

 

[Helen]

@Hip and @alex3619 Maybe I´m quite wrong, but doesn´t Ian Lipkin´s research include enteroviruses when he and his group is examinating blood, saliva, faeces and spinal fluid (and I think other fluids too)? Two researchers in Sweden who where in the XMRV circus (as someone here called it) are still researching ME/CFS. I would guess they are eager for revenge...At least I know they are examinating spinal fluid with all analyses that are possible according to the assistent who invited me to take part in the study.

 

[alex3619]

@Helen, there is a difference between acute virus infection and chronic tissue infection. Blood tests only work on acute infections, and with antibody testing can show recent or past acute infections. Slow tissue infections are invisible without biopsy, but I think there are secondary markers for many of them.

 

[Bob]

I think Chia studied gut tissue biopsies, didn't he?

 

[Hip]

@Helen
I understand that Ian Lipkin did include enteroviruses in his recent study, but as Alex points out, chronic enterovirus infections are unfortunately very hard to detect, unless specific types of testing are used. Lipkin's study has not yet been published I believe, so I am not sure exactly what type of enterovirus testing he performed.

If Lipkin just used blood tests and cerebrospinal fluid tests for enterovirus antibodies, this does not count. Most laboratory blood tests for enterovirus antibodies come out negative in ME/CFS patients, and I would think that cerebrospinal fluid tests for enterovirus antibodies will also come out negative. This is because these tests are not sensitive enough for detecting chronic enterovirus infections.

However, Dr John Chia developed and pioneered a new testing method that reliably detects enterovirus infections in ME/CFS patients. His new method uses tiny stomach tissue samples.

What John Chia did was to take a tiny tissue sample (biopsy) from the stomach of ME/CFS patients (using an endoscope), and then he tested that tissue sample for the presence of enteroviruses. Using this method, Chia found that 82% ME/CFS patients have significant enterovirus infections in their stomach tissues, whereas only 20% of the controls had such an infection. Ref: 1

So this is a major finding, as it demonstrates a strong association between enterovirus infection and ME/CFS.

In order to replicate John Chia's results, you would need to conduct a similar study that also tested stomach tissue samples for enteroviruses. This is not a particularly difficult thing to do, and in fact Chia routinely tests his ME/CFS patients for enterovirus infections using the stomach biopsy method.

So it would be very straightforward for another researcher to replicate Chia's results, but nobody seems to have bothered to do this.

To me this is crazy, because Chia found a very positive, strong link between enterovirus infection and ME/CFS. You would have thought that the whole of the medical scientific community would sit up and take note. You would have thought that they would spring into action and investigate further. But nothing further has been done by the wider scientific community.

And even ME/CFS patients, who are usually the most vociferous advocates for further research, have largely ignored Chia's groundbreaking findings. It makes no sense to me. As I say, do we want a cure or not?

 

[Helen]

@Hip , @Bob and @alex3619 Thank you very much for your posts. You opened up my eyes to Dr. Chias indeed interesting stomach biopsy research.