• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Once Is Not Enough.....by Simon McGrath

View the Post on the Blog

(Guest blogger Simon McGrath focuses on an important topic and stumbling block for ME/CFS - replication studies (or the lack thereof). The XMRV story presents a somewhat unusual theme; a subject receiving enough study that a consensus (at least to date) has been reached and in relative rapid fashion but a recent blog found a significant number of research efforts are never or are haphazardly followed up on. - Cort)



Replication


Replication of important findings is essential for progress in CFS research (and other research too)

A recent review by Harvard professor Anthony Komaroff cited over 150 findings of either biological abnormalities or evidence of chronic infection in CFS patients, yet almost every one was different, with very few replications. Which is disappointing given that replication of findings is the Gold Standard of scientific research. A good number of the cited reports were from the 1990s, suggesting there had been no follow-up or replication for over a decade. The apparent lack of follow-up of promising findings is probably a significant factor in why so little progress has been made in understanding the pathophysiology of CFS.

The Komaroff review quite rightly pointed out that the evidence was consistent with chronic infection being a cause of some cases of CFS, but also acknowledged the limitations of the data.

The titles and abstracts of cited papers in that review (as well as those in other papers) often contain words like 'possible', 'preliminary', 'exploratory', 'suggests' and 'need confirmation' - but rarely contain words like 'validated', 'confirmed' or 'replicated', Suzanne Vernon has pithily described this as a ‘one and done’ approach to research. In fields such as biochemistry there are, of course, many preliminary and provisional findings in recent papers - it's the nature of science - but there are also a respectable number of confirmations (or refutations) of original findings.

It's only by confirming interesting findings that there can be progress: each validated finding is a building block that other researchers can use to make further progress. Of course, CFS is phenomenally challenging to study, but without rigorously following up the leads that do exist to either prove or discount them, it's going to be tough to get anywhere.

CFS studies also tend to be quite small, making findings less reliable and less heeded. To a large extent this is because CFS research is woefully underfunded and well-defined CFS cohorts are expensive to assemble. Nevertheless, many, if not most, studies conclude that their findings need replicating on larger samples – and that replication rarely comes.

Other Roadblocks


A look at the last three months of PubMed citations suggested another problem the ME/CFS research field faces...

Small Study Size


Small study size is another almost common problem. Researchers often dismiss studies as unreliable if they have small sample sizes and they’re generally tagged with a ‘preliminary’ label - meaning the data is nothing more than suggestive. Poorly funded ME/CFS researchers, however, often have no choice than to do the best they can with the funds they have.

A recent autoimmune study had 15 patients and an amino acid study contained 11. Some decently sized biological studies did appear; a Japanese infrared spectrum study (n=57), two PHANU studies on natural killer cells (n=28, , a prognosis study (n=47), a blood pressure study (n=68) and an inflammation study (n=144)) all had decent or good numbers of patients

But contrast these with size and quantity of the behavioral studies over the last couple of months; three CBT trials had 63, 105, 120, 123 and 135 patients, a stress management study had 117, a pscyhosocial study had 84, a ‘psychological aspects’ study 103, a ‘disease-related worries’ study had 140, and a co-morbid psychiatric diagnoses study had 640 patients. All of these studies, not surprisingly, appearing in easily accessible electronic format. They had abundant funding and they will get read.

Its easy, given this pattern, to see why ME/CFS fights an uphill battle against a psychological interpretation. Physiologically based studies are not only dwarfed by not only by the number of behaviorally oriented studies but by their sheer size.

Back to Replication


Replication is key, and not just in CFS research - the CFS scientific literature holds a vast array of unconfirmed findings that anyone can pick from to support just about any hypothesis about the illness - leading to confusion and another reason for researchers in more well-defined disorders to discount findings in ME/CFS or to apply the ‘wastebasket’ term to them.

What's needed to get beyond the ‘wastebasket trap’ is robust replication: repeating the findings on an independent sample of patients and controls, with a substantial sample size, and, ideally, by an independent research group.

More CFS research funding would clearly help, but there’s no guarantee it would lead to more replications, at least not judging by the experience in other, better funded fields.

The lack of replication is a hot topic across life-science research, particularly since John Ioannidis’s landmark paper ‘Why Most Published Research Findings are False’. A recent Nature article noted that a pharmaceutical company’s attempt to replicate 53 published findings of promising drugs managed to fully replicate just 6 (11%) of the findings. Similar work in Germany, at Bayer, found only 25% of preclinical studies could be validated to the point they were worth pursuing. There is even more concern around studies where there are many, many possible links that could arise by chance, as with genetic association studies that look at thousands of genes. One study found that of 166 reported associations between gene variants and common diseases studied 3 or more times, only 6 were replicated consistently.

Gene Expression Studies Illustrate the Need for Replication


The same problem of false positives applies to gene expression studies, which have thrown up many putative associations between expression of specific genes and CFS, but, as yet, no proof. Jonathan Kerr authored several of these studies and he also bravely published research that discounted them too. His PLoS One replication paper last year concluded that, “many of the previously identified reporter genes are study-specific and thus cannot be used as a broad CFS diagnostic”. The problem is that when you look at the expression thousands of different genes it’s highly likely that some of them will appear to correlate with CFS just by chance. If the correlation is by chance, rather than real, it is unlikely to reappear in a replication on an independent sample, which is just what Kerr’s PLoS One study found.

Biomarker Mirage?


A similar problem applies to biomarkers, which have frequently been ‘found’ over the years. One reason so many biological abnormalities appear to identify CFS is that authors look at the results first, then choose a threshold level for the biomarker that best discriminate between controls and patients in their particular sample. This post-hoc tweaking doesn’t mean a whole lot until the finding has been replicated on an independent sample of patients and controls, using the same thresholds as the original study. Again, replication helps to separate the real effects from the statistical artefacts.

A more robust approach is to divide patients into separate exploratory and validation samples, effectively building a replication into the study itself. This approach is often used in gene association studies and was also used in a study from Julia Newton: the diagnostic criterion for autonomic dysfunction in CFS patients established in the phase one (n=40) was shown in an independent phase 2 sample (n=30) to have a positive predictive value of 0.96. This is a much better tactic, though with relatively small patient numbers it still needs confirmation in a larger sample.

A Real, Replicated Bio-abnormality: Natural Killer Cell Functioning


The one shining example of a robustly replicated bio-abnormality is lower natural killer cell cytotoxicity (NKCC) in CFS patients. Natural killer cells protect the body by killing invading organisms and it’s this ability to kill other cells that is reduced in CFS patients. Lower NKCC has been detected at least since 1987, and was most convincingly replicated by Fletcher and Klimas (n=176, p<0.0005 vs controls). There was also a later confirming paper by Brenu, who has now shown these differences are sustained vs controls over time. However, there is still substantial overlap on NKCC levels between individual CFS patients and healthy controls, and as the Komaroff review pointed out, it’s not clear if these findings are specific to CFS - so this bioabnormality may not prove to be such a good biomarker.

Replication Problems with Psychological CFS Research


We’ve focused on biological research but the same lack of replication applies to much psychosocial research too. For instance, some studies have reported spectacular results for CBT for CFS e.g. a study by Deale found physical functioning improved dramatically and another study found 23-44% of CFS patients made a full recovery. Yet a meta-analysis, and the large PACE trial, found only modest gains, and even these self-reported improvements may not be real. But this situation is typical of clinical trials in many fields: small early studies often find impressive results that are not replicated in larger more robust studies, or meta-analyses that look at the data from many studies together.

Reasons Not to Replicate (Why the Low Rate of Replication?)


Most scientists naturally want to pursue their own ideas, so spending time and limited funds on replication studies will never be popular, particularly as journals are often reluctant to publish replications on the grounds they say are not ‘original’ contributions to science. And there may, in fact, be many failed replication attempts that are simply never published, in what is known as the ‘File Drawer Effect’: researchers don’t bother writing up and publishing negative results. Psych File Drawer has been set up to encourage and publish replication attempts in Experimental Psychology. Maybe CFS needs its own File Drawer project. Perhaps there have been unsuccessful – and unpublished - replication attempts of some of the many findings reported in the Komaroff review. If so, other researchers need to know.

Also, replication studies are not a good way to make friends, as XMRV has shown. ‘You’re Wrong’ never goes down well, whether aimed at the original study or the replication attempts. The Reproducibility Project is a bold move to systematically measure reproducibility by attempting to replicate all experiments published during 2008 in three different psychology journals. Brian Nosek, a professor of psychology who helps run the project, was warned not to go ahead “because psychology is under threat and this could look bad”, but pressed ahead anyway - so there may be some very nervous authors out there.

Promising Future?


Over the past two decades I’ve been illthere have been many seemingly-exciting new discoveries in CFS, but little real progress. What’s encouraging now is that there appears to be a substantial and important shift towards more replication studies in CFS research.

The finding that CFS patients have lower NKCC has already been robustly replicated. The three other most important findings in recent years are probably XMRV, Rituximab and the Lights' gene expression on moderate exercise studies. There have already been numerous attempts to replicate the XMRV findings, with the Lipkin’s ‘Mother of All Replication Studies’ due to report soon. Fluge & Mella, the authors of the amazing but small Rituximab study, are planning a large multi-centre trial to attempt replication of their initial findings, while Dr's Enlander and Bell are planning a replication study in the US too. And the Lights are being funded by the NIH to validate the results of their original study with new patients. Hopefully attempts to replicate important findings will become the norm in CFS research.

A new focus on replication of promising findings won’t guarantee success, but is likely to help greatly as research will be increasingly focused on meaningful rather than simply intriguing findings.

(Simon McGrath took a biochemistry degree in the eighties planning to go into research, but then decided to work for a charity instead. He got ill in 1994. Between then and now he's been hospitalised, bedbound for several years and is currently just largely housebound. He posts fairly regularly on the forums under a psuedonym.)

Support Phoenix Rising






(Why not support Ph0enix Rising with a $5 , $10, $15, $20 or more monthly 'subscription'. Click on the Subscribe button and look on the right hand side of the page for options :))​

View the Post on the Blog
 
Last edited by a moderator:
I couldn't agree more; replication studies are key. It drives me crazy that every time ME/CFS is mentioned, they say "there are no biomarkers," and yet over the years many potential biomarkers have been identified. What has been lacking is replication studies. It's difficult to get funding for replication studies from universities, because they are not seen as enhancing the reputation of the institution or of the researcher. That's one reason why government funded research is so important. Theoretically, it's supposed to be independent of that sort of competitive atmosphere so that it concentrate on the science.

ME/CFS research is caught in a Catch-22: without biomarkers it's difficult to ensure that research is done on a well-defined patient cohort, and because research is done on poorly-define patient cohorts, it's hard to find biomarkers. I think it will be difficult to make real progress with research until we break through this deadlock.

Every time I hear someone from one of our government health research agencies say that there are no biomarkers for ME/CFS, I want to scream "Then why don't you follow up on the dozens of studies that have identified potential biomarkers?!!" The CDC hasn't even followed up on THEIR OWN studies which they said found possible biomarkers.
 
I couldn't agree more; replication studies are key. It drives me crazy that every time ME/CFS is mentioned, they say "there are no biomarkers," and yet over the years many potential biomarkers have been identified. What has been lacking is replication studies. It's difficult to get funding for replication studies from universities, because they are not seen as enhancing the reputation of the institution or of the researcher. That's one reason why government funded research is so important. Theoretically, it's supposed to be independent of that sort of competitive atmosphere so that it concentrate on the science.

ME/CFS research is caught in a Catch-22: without biomarkers it's difficult to ensure that research is done on a well-defined patient cohort, and because research is done on poorly-define patient cohorts, it's hard to find biomarkers. I think it will be difficult to make real progress with research until we break through this deadlock.

Every time I hear someone from one of our government health research agencies say that there are no biomarkers for ME/CFS, I want to scream "Then why don't you follow up on the dozens of studies that have identified potential biomarkers?!!" The CDC hasn't even followed up on THEIR OWN studies which they said found possible biomarkers.
I think if we took a rigorous look at how many positive studies have not been followed up on we'd be in for a bit of shock. One problem may be that with such poor funding some studies are so small that they're not convincing enough for other researchers to take them on...Look at the Non-Hodgkins Lymphoma studies - most have suggested something is going and its been 20 years and we still don't have a really good study on that...A study was presented at the 2009 Reno conference that seemed to nail it- but that study was never published....its's frustrating!
 
Good article, Simon. Thanks.

“because psychology is under threat and this could look bad”

I wonder who is to blame for that situation? Surely not shoddy sub-standard work in the first place by the psychs themselves?

Surely not?
 
Simon, I think we are within 5 years of a biomarker.

One you didn't mention was Komaroff differentiating depression from CFS patients with EEG. I remember his saying he was going to do a bigger one as a follow up.

And whatever happened to the Sakudo results that was able to diagnose 100% of healthy and 93% of ME/CFS patients: http://128.121.104.17/cfs-inform/CFS.case.def/sakudo.etal06.pdf He did another one in 2009: http://www.ncbi.nlm.nih.gov/pubmed/19248775 Last I heard he was trying to get a patent. But that was four years ago. What's the news on that now? I have half a mind to call him.

And then there is some urine tests we are hearing about.

Also, there's Stacy Stevens study showing oxygen absorption and heart rate differences after exercise.

Yeah, we'll have a biomarker soon.
 
Ha! Some good news! Glad you brought that up because I was going to check his earlier results. Sakudo just published again a couple of weeks ago.

Clin Chim Acta. 2012 May 11. [Epub ahead of print]
Visible and near-infrared spectra collected from the thumbs of patients with chronic fatigue syndrome for diagnosis.
Sakudo A, Kuratsune H, Kato YH, Ikuta K.
Source
Department of Virology, Center for Infectious Disease Control, Research Institute for Microbial Diseases, Osaka University, Yamadaoka, Suita, Osaka 565-0871, Japan; Fatigue Clinical Center, 21st Century COE Program, Osaka City University Graduate School of Medicine, Abeno-ku, Osaka 545-8585, Japan.
Abstract
BACKGROUND:
Currently, diagnosis of chronic fatigue syndrome (CFS) is based on clinical symptoms and therefore relies on the experience and skill of the doctors. Here, we have examined the possible diagnosis of CFS based on spectral information and chemometrics analysis, such as principal component analysis (PCA) and soft modeling of class analogy (SIMCA).
METHODS:
Visible and near-infrared (Vis-NIR) spectroscopy was used to examine possible changes in the region of 600-1100nm in thumbs and assessed.
RESULTS:
The Vis-NIR spectra of thumbs from 57 CFS patients and 74 healthy volunteers were subjected to PCA and SIMCA to develop multivariate models to discriminate between CFS patients and healthy individuals. The model was further assessed by the prediction of 120 determinations (60 in the healthy group and 60 in the CFS patient group). The PCA model predicted a discrimination of the masked samples; specifically the SIMCA model correctly predicted 51 of 60 (83.3%) healthy volunteers and 42 of 60 (70%) CFS patients.
CONCLUSIONS:
Despite the relatively small number of subjects involved in this trial, who were exclusively Japanese, our results imply that Vis-NIR spectroscopy of the thumb combined with chemometrics analysis may provide a valuable tool for diagnosing CFS.
Copyright © 2012 Elsevier B.V. All rights reserved.
 
Surprised the author left out the 37 kda Rnase-L research. Replicated by four independent labs, though sample sizes were small, then simply dropped by the research community without real explanation. The test was available in the US up through 2009. This was a better biomarker for identifying Ampligen responders than NK cell function. Science has moved backward.
 
Thanks to all for those kind words.

ixchelkali:
I couldn't agree more; replication studies are key. It drives me crazy that every time ME/CFS is mentioned, they say "there are no biomarkers," and yet over the years many potential biomarkers have been identified. What has been lacking is replication studies. It's difficult to get funding for replication studies from universities, because they are not seen as enhancing the reputation of the institution or of the researcher. That's one reason why government funded research is so important. Theoretically, it's supposed to be independent of that sort of competitive atmosphere so that it concentrate on the science...

Every time I hear someone from one of our government health research agencies say that there are no biomarkers for ME/CFS, I want to scream "Then why don't you follow up on the dozens of studies that have identified potential biomarkers?!!" The CDC hasn't even followed up on THEIR OWN studies which they said found possible biomarkers.
You're right it's shocking the CDC hasn't followed up its own biomarker studies, given it comfortably has the cash to do so. However, I am surprised that more researchers don't follow their own findings, even where funding isn't easy to get.

Consider this situation: you've just published a paper showing you've identified a potential biomarker/bioabnormality for CFS, and have waxed lyrical in the discussion section about the importance of this discovery. There are now 2 options:
  1. Take your promising results and apply for a new grant, pointing out that a replicated finding in CFS would be close to revolutionary and so well worth funding. It doesn't have to be a huge study, but replication of a finding even in a small study would help make a robust case for a definitive study, or may well interest other researchers. Or,
  2. Move on to a new potential biomarker and repeat the process, leaving the potential finding just published to join hundreds of others in the CFS destined for obscurity.
As I said in the article, Jonathan Kerr took route 1 and was unable to validate his earlier gene expression findings, but that result in itself is immensley valuable and helps to move the field on. Other researchers may continue to pursue this line of inquiry - but their work will need to take account of Kerr's findings, pushing research standards higher and improving the chance of real progress in understanding CFS.

This kind of thing happens in other fields all the time - and XMRV is a good example of it unfolding now in CFS research. It's certainly a lot easier for such replication to take place in well-funded fields such as virology (XMRV researchers must have spent $millions, as well as huge numbers of research hours in barely 18 months). The battle for more CFS research funding must continue, but focusing meagre resources on nailing or disproving potential findings is probably going to give more bangs for the bucks that do exist.

ME/CFS research is caught in a Catch-22: without biomarkers it's difficult to ensure that research is done on a well-defined patient cohort, and because research is done on poorly-define patient cohorts, it's hard to find biomarkers. I think it will be difficult to make real progress with research until we break through this deadlock.
Completely agree. Personally, I think the way forward is to take a large cohort defined according to several different criteria, measure a hatful of biological and clinical markers and see if robust clusers emerge. Potentially both the Lipkin/Chronic Fatigue Foundation work and the new CDC clinical study underway could end up doing just this.
 
Simon, I think we are within 5 years of a biomarker.

And whatever happened to the Sakudo results that was able to diagnose 100% of healthy and 93% of ME/CFS patients: http://128.121.104.17/cfs-inform/CFS.case.def/sakudo.etal06.pdf He did another one in 2009: http://www.ncbi.nlm.nih.gov/pubmed/19248775 Last I heard he was trying to get a patent. But that was four years ago. What's the news on that now?
I hope you're right! I'm not as optimistic, mainly because biomarkers are incredibly hard to find generally in medicine. Even with a disease like Rheumatoid Arthritis, where it's known to be an autoimmune disease and cytokine TNF-alpha is known to play an important role in at least a significant number of patients, they haven't yet managed to pin down a biomarker.

As for the Sakudo study, the new paper Cort quoted showed the figures are much less impressive:
The PCA model predicted a discrimination of the masked samples; specifically the SIMCA model correctly predicted 51 of 60 (83.3%) healthy volunteers and 42 of 60 (70%) CFS patients.
According to Lenny Jason, you need at least 85% in both of those measures, and preferably over 90%, to be clinically useful :( .
 
yes, I think immune system variability makes finding a biomarker in immune system diseases harder. I still think the PEM as a result is showing great promise in more than just immune system problems.

Remember the spinal fluid protein study with ME/CFS and Lyme folks? It holds promise also.

I think, actually, we already have many. The question is which is more accessible. Right now, likely the Staci Stevens test is easiest and more accessible.

Tina
 
yes, I think immune system variability makes finding a biomarker in immune system diseases harder. I still think the PEM as a result is showing great promise in more than just immune system problems.

Remember the spinal fluid protein study with ME/CFS and Lyme folks? It holds promise also.

I think, actually, we already have many. The question is which is more accessible. Right now, likely the Staci Stevens test is easiest and more accessible.

Tina
PEM should be good; in fact Lenny Jason has got better specificity and sensitivity with questions about the nature of fatigue than I've seen for any biomarker.

Would like to see a replication of the interesting Stevens work: there was talk this was in the pipeline over a year ago, but no replication seems to have been published yet, AFAIK.
 
Surprised the author left out the 37 kda Rnase-L research. Replicated by four independent labs, though sample sizes were small, then simply dropped by the research community without real explanation. The test was available in the US up through 2009. This was a better biomarker for identifying Ampligen responders than NK cell function. Science has moved backward.
Yes, small studies and I kept expecting to see a big replication giving it would be revolutionary if proven. IIRC, Judy Mikovits said the RNAase-L work was what led her to look for retroviruses in the first place, but the RNAase-L link didn't hold up in their own work.
 
I think the CFI with $10 million will bring the larger cohort. Also, Peterson working with Australians might increase cohort numbers. The CDC CASA project has seven clinics. Ian Lipkin is doing something with multiple clinics.

Finally, our researchers are working together in joint studies, using patients from multiple places.
Also, the organizations are finally starting to work together.

We are more powerful together.
 
As for the Sakudo study, the new paper Cort quoted showed the figures are much less impressive: According to Lenny Jason, you need at least 85% in both of those measures, and preferably over 90%, to be clinically useful :( .

I doubt that a single biomarker will become a replacement to the current process of diagnosing ME/CFS. Perhaps multiple mediocre biomarkers of limited sensitivity or specificity could be combined to strengthen the reason for the diagnosis?
 
Perhaps multiple mediocre biomarkers of limited sensitivity or specificity could be combined to strengthen the reason for the diagnosis?
Maybe! One of the dangers is that such mediocre 'markers' are merely markers of ill-health, rather than specific to CFS, a point Komaroff raises in his paper. Actually, I agree that multiple markers may help, but the more markers you use, the easier it is to find differences by chance between healthy people and CFS patients, so robust replication would be essential.

Usually, potential biomarkers are compared between CFS patients and healthy controls, which isn't that much use in diagnosis - a child can usually manage to spot the difference between such groups. In other areas, biomarkers are useful to detect differences that aren't obvious. E.g in diagnosis in identifying a cancer in someone with mild but generic symptoms, or in prognosis eg discriminating between the likely progression of disease (and best treatment options) of two patients with the same cancer and level of symptoms. For CFS, the equivalent would be in identifying 'true' ME/CFS cases from other cases of chronic fatigue, including those caused by other as-yet-undiagnosed physical illnesses.