• Phoenix Rising needs funds to operate: please consider donating to support PR

Once Is Not Enough.....by Simon McGrath

View the Post on the Blog

(Guest blogger Simon McGrath focuses on an important topic and stumbling block for ME/CFS - replication studies (or the lack thereof). The XMRV story presents a somewhat unusual theme; a subject receiving enough study that a consensus (at least to date) has been reached and in relative rapid fashion but a recent blog found a significant number of research efforts are never or are haphazardly followed up on. - Cort)



Replication


Replication of important findings is essential for progress in CFS research (and other research too)

A recent review by Harvard professor Anthony Komaroff cited over 150 findings of either biological abnormalities or evidence of chronic infection in CFS patients, yet almost every one was different, with very few replications. Which is disappointing given that replication of findings is the Gold Standard of scientific research. A good number of the cited reports were from the 1990s, suggesting there had been no follow-up or replication for over a decade. The apparent lack of follow-up of promising findings is probably a significant factor in why so little progress has been made in understanding the pathophysiology of CFS.

The Komaroff review quite rightly pointed out that the evidence was consistent with chronic infection being a cause of some cases of CFS, but also acknowledged the limitations of the data.

The titles and abstracts of cited papers in that review (as well as those in other papers) often contain words like 'possible', 'preliminary', 'exploratory', 'suggests' and 'need confirmation' - but rarely contain words like 'validated', 'confirmed' or 'replicated', Suzanne Vernon has pithily described this as a ‘one and done’ approach to research. In fields such as biochemistry there are, of course, many preliminary and provisional findings in recent papers - it's the nature of science - but there are also a respectable number of confirmations (or refutations) of original findings.

It's only by confirming interesting findings that there can be progress: each validated finding is a building block that other researchers can use to make further progress. Of course, CFS is phenomenally challenging to study, but without rigorously following up the leads that do exist to either prove or discount them, it's going to be tough to get anywhere.

CFS studies also tend to be quite small, making findings less reliable and less heeded. To a large extent this is because CFS research is woefully underfunded and well-defined CFS cohorts are expensive to assemble. Nevertheless, many, if not most, studies conclude that their findings need replicating on larger samples – and that replication rarely comes.

Other Roadblocks


A look at the last three months of PubMed citations suggested another problem the ME/CFS research field faces...

Small Study Size


Small study size is another almost common problem. Researchers often dismiss studies as unreliable if they have small sample sizes and they’re generally tagged with a ‘preliminary’ label - meaning the data is nothing more than suggestive. Poorly funded ME/CFS researchers, however, often have no choice than to do the best they can with the funds they have.

A recent autoimmune study had 15 patients and an amino acid study contained 11. Some decently sized biological studies did appear; a Japanese infrared spectrum study (n=57), two PHANU studies on natural killer cells (n=28, , a prognosis study (n=47), a blood pressure study (n=68) and an inflammation study (n=144)) all had decent or good numbers of patients

But contrast these with size and quantity of the behavioral studies over the last couple of months; three CBT trials had 63, 105, 120, 123 and 135 patients, a stress management study had 117, a pscyhosocial study had 84, a ‘psychological aspects’ study 103, a ‘disease-related worries’ study had 140, and a co-morbid psychiatric diagnoses study had 640 patients. All of these studies, not surprisingly, appearing in easily accessible electronic format. They had abundant funding and they will get read.

Its easy, given this pattern, to see why ME/CFS fights an uphill battle against a psychological interpretation. Physiologically based studies are not only dwarfed by not only by the number of behaviorally oriented studies but by their sheer size.

Back to Replication


Replication is key, and not just in CFS research - the CFS scientific literature holds a vast array of unconfirmed findings that anyone can pick from to support just about any hypothesis about the illness - leading to confusion and another reason for researchers in more well-defined disorders to discount findings in ME/CFS or to apply the ‘wastebasket’ term to them.

What's needed to get beyond the ‘wastebasket trap’ is robust replication: repeating the findings on an independent sample of patients and controls, with a substantial sample size, and, ideally, by an independent research group.

More CFS research funding would clearly help, but there’s no guarantee it would lead to more replications, at least not judging by the experience in other, better funded fields.

The lack of replication is a hot topic across life-science research, particularly since John Ioannidis’s landmark paper ‘Why Most Published Research Findings are False’. A recent Nature article noted that a pharmaceutical company’s attempt to replicate 53 published findings of promising drugs managed to fully replicate just 6 (11%) of the findings. Similar work in Germany, at Bayer, found only 25% of preclinical studies could be validated to the point they were worth pursuing. There is even more concern around studies where there are many, many possible links that could arise by chance, as with genetic association studies that look at thousands of genes. One study found that of 166 reported associations between gene variants and common diseases studied 3 or more times, only 6 were replicated consistently.

Gene Expression Studies Illustrate the Need for Replication


The same problem of false positives applies to gene expression studies, which have thrown up many putative associations between expression of specific genes and CFS, but, as yet, no proof. Jonathan Kerr authored several of these studies and he also bravely published research that discounted them too. His PLoS One replication paper last year concluded that, “many of the previously identified reporter genes are study-specific and thus cannot be used as a broad CFS diagnostic”. The problem is that when you look at the expression thousands of different genes it’s highly likely that some of them will appear to correlate with CFS just by chance. If the correlation is by chance, rather than real, it is unlikely to reappear in a replication on an independent sample, which is just what Kerr’s PLoS One study found.

Biomarker Mirage?


A similar problem applies to biomarkers, which have frequently been ‘found’ over the years. One reason so many biological abnormalities appear to identify CFS is that authors look at the results first, then choose a threshold level for the biomarker that best discriminate between controls and patients in their particular sample. This post-hoc tweaking doesn’t mean a whole lot until the finding has been replicated on an independent sample of patients and controls, using the same thresholds as the original study. Again, replication helps to separate the real effects from the statistical artefacts.

A more robust approach is to divide patients into separate exploratory and validation samples, effectively building a replication into the study itself. This approach is often used in gene association studies and was also used in a study from Julia Newton: the diagnostic criterion for autonomic dysfunction in CFS patients established in the phase one (n=40) was shown in an independent phase 2 sample (n=30) to have a positive predictive value of 0.96. This is a much better tactic, though with relatively small patient numbers it still needs confirmation in a larger sample.

A Real, Replicated Bio-abnormality: Natural Killer Cell Functioning


The one shining example of a robustly replicated bio-abnormality is lower natural killer cell cytotoxicity (NKCC) in CFS patients. Natural killer cells protect the body by killing invading organisms and it’s this ability to kill other cells that is reduced in CFS patients. Lower NKCC has been detected at least since 1987, and was most convincingly replicated by Fletcher and Klimas (n=176, p<0.0005 vs controls). There was also a later confirming paper by Brenu, who has now shown these differences are sustained vs controls over time. However, there is still substantial overlap on NKCC levels between individual CFS patients and healthy controls, and as the Komaroff review pointed out, it’s not clear if these findings are specific to CFS - so this bioabnormality may not prove to be such a good biomarker.

Replication Problems with Psychological CFS Research


We’ve focused on biological research but the same lack of replication applies to much psychosocial research too. For instance, some studies have reported spectacular results for CBT for CFS e.g. a study by Deale found physical functioning improved dramatically and another study found 23-44% of CFS patients made a full recovery. Yet a meta-analysis, and the large PACE trial, found only modest gains, and even these self-reported improvements may not be real. But this situation is typical of clinical trials in many fields: small early studies often find impressive results that are not replicated in larger more robust studies, or meta-analyses that look at the data from many studies together.

Reasons Not to Replicate (Why the Low Rate of Replication?)


Most scientists naturally want to pursue their own ideas, so spending time and limited funds on replication studies will never be popular, particularly as journals are often reluctant to publish replications on the grounds they say are not ‘original’ contributions to science. And there may, in fact, be many failed replication attempts that are simply never published, in what is known as the ‘File Drawer Effect’: researchers don’t bother writing up and publishing negative results. Psych File Drawer has been set up to encourage and publish replication attempts in Experimental Psychology. Maybe CFS needs its own File Drawer project. Perhaps there have been unsuccessful – and unpublished - replication attempts of some of the many findings reported in the Komaroff review. If so, other researchers need to know.

Also, replication studies are not a good way to make friends, as XMRV has shown. ‘You’re Wrong’ never goes down well, whether aimed at the original study or the replication attempts. The Reproducibility Project is a bold move to systematically measure reproducibility by attempting to replicate all experiments published during 2008 in three different psychology journals. Brian Nosek, a professor of psychology who helps run the project, was warned not to go ahead “because psychology is under threat and this could look bad”, but pressed ahead anyway - so there may be some very nervous authors out there.

Promising Future?


Over the past two decades I’ve been illthere have been many seemingly-exciting new discoveries in CFS, but little real progress. What’s encouraging now is that there appears to be a substantial and important shift towards more replication studies in CFS research.

The finding that CFS patients have lower NKCC has already been robustly replicated. The three other most important findings in recent years are probably XMRV, Rituximab and the Lights' gene expression on moderate exercise studies. There have already been numerous attempts to replicate the XMRV findings, with the Lipkin’s ‘Mother of All Replication Studies’ due to report soon. Fluge & Mella, the authors of the amazing but small Rituximab study, are planning a large multi-centre trial to attempt replication of their initial findings, while Dr's Enlander and Bell are planning a replication study in the US too. And the Lights are being funded by the NIH to validate the results of their original study with new patients. Hopefully attempts to replicate important findings will become the norm in CFS research.

A new focus on replication of promising findings won’t guarantee success, but is likely to help greatly as research will be increasingly focused on meaningful rather than simply intriguing findings.

(Simon McGrath took a biochemistry degree in the eighties planning to go into research, but then decided to work for a charity instead. He got ill in 1994. Between then and now he's been hospitalised, bedbound for several years and is currently just largely housebound. He posts fairly regularly on the forums under a psuedonym.)

Support Phoenix Rising






(Why not support Ph0enix Rising with a $5 , $10, $15, $20 or more monthly 'subscription'. Click on the Subscribe button and look on the right hand side of the page for options :))​
View the Post on the Blog
 
Last edited by a moderator:

Comments

Simon,

Thank you for this fascinating post and your thoughtful replies to comments. I hope to see you again on this forum,

JAH
 
thanks, Simon, for the great article
 
All that positive feedback is much appreciated, I may have to write another now... Thanks
 
$2m NIH grant to attempt replication of oxidative stress finding in larger cohort
Here's another good example of the shift to replication of interesting findings in CFS research.

2 pilot studies using neuroimaging and markers of oxidative stress found differences between CFS patients and healthy controls, though the differences with MDD patients didn't reach statistical significance. But now these promising findings are being followed up to a) replicate and b) show specificity for CFS vs Major Depression. This is how good science is supposed to work. Promising pilot -> larger replication (or refutation), hopefully, as in this case, with a more comprehensive design - the authors are checking in plasma, urine and CerebroSpinal Fluid on 40 CFS paitients, controls and MDD patients.

NIH project reporter
The applicants recently obtained strong in vivo neuroimaging evidence that suggests increased oxidative stress as a pathophysiological model of chronic fatigue syndrome (CFS). However, the validity and specificity of this hypothesis for CFS remain uncertain, as the primary supporting findings have yet to be replicated, and similar abnormalities were found in major depressive disorder (MDD), a neuropsychiatric disorder that has extensive symptom overlap with CFS.
User-friendly summary of this study over at Research1st

Thanks to @TomKindlon for tweeting about this
 
This is probably the most caustic comment I've come across about reproducibility
Reproducibility separates science from mere anecdote.
Ouch. It's taken from a recent opinion piece in Nature:
In recent years, it has become clear that biomedical science is plagued by findings that cannot be reproduced. This wastes grant money and hinders development of new treatments and cures. The irreproducibility epidemic is exacerbated by a funding structure that rewards publications above all else, whether they are reproducible or not. Science as a system should place more importance on reproducibility... Reproducibility separates science from mere anecdote.
And Nature announced today that it and other Nature Group journals will bring in a raft of editorial measures to increase reproducibilty by improving the consistency and quality of reporting in life-sciences articles. Looks like reproducibility is here to stay across the whole of life sciences.
 
I would like to make several comments on the original article by Simon, which I think makes some very good points.

The Deale article which found cbt/get improved physical functional only used questionairres. http://ajp.psychiatryonline.org/data/Journals/AJP/3674/408.pdf

I suspect the extreme death of funding in CFS and ME research is partly a cause of lack of reproduction of results. Reproducing someone elses results, and especially your own, might be less favoured in an environment of almost no funding. We usually have to compete with many other proposed studies for limited money. My guess is that what is needed is ring-fenced funding for replication studies, to try to promote them happening, as well as funded methods to publish negative or uninteresting results.

The interesting thing about NK cell dysfunction is not so much the low killing power, but the disturbance in NK cell subsets. Similar subset distrubances have been found in T cells and B cells, with the further issue is that there appears to be a decline in mature B cells.

An integrated immune surveillance study looking at all of these issues in large numbers of patients and several types of control groups could prove very interesting.

Psychology, and indeed psychiatry, are indeed in trouble. However, hiding issues and denying everything will only make the problem worse - it needs debate, full disclosure, and very intense analysis of alternatives to fix these issues.

Of all the disciplines in psychiatry, I think psychogenic attribution of disease is in the most trouble. In every case where a cause has been found for a "psychogenic" illness, it has been found to be physical. Every case. I think our generation will find the same for fibro, CFS, ME, and IBS. Its already happening. How is psychogenic medicine, in its current incarnation, not considered a blight on medical research and the medical profession itself?

There is a lot online about The Reproducability Project, for example this:
http://langcog.stanford.edu/papers/OSC-pops2012.pdf
 
Thanks, everyone. This information has been fascinating, partly because I'm going to check reproducibility myself at the end of May. I actually think of it as testing for a null hypothesis. Not sure of the right wording. A recent paper linking IBS or SIBO with rosacea, suggested that Rifaxin would cure the first as well as the latter. I have rosacea and some sort of IBS, so after an exploratory procedure later in May, I'm going to take a course of Rifaxin for two weeks and see if it makes any difference for me. Very rough and ready kind of replication, I suppose, but I'm using the resources available to me.

Other readers here have most likely already tested this link, and I'd love to hear their results.

Cheers, Lynne