Biomarker problems

[Osaca]

Recently there's been more news again of biomarkers claiming certain accuracies amongst patient groups of Long-Covid and ME/CFS patients and from my personal point of views there are usually several problems amongst these studies and the claims they make.

The good studies that make these claims have large sample sizes and validate their findings on subsets which didn't belong to the test data, these studies are extremely rare. Below I will explain why I still believe many of these studies are often sub-optimal and why they should often also apply other classification tools with different goals as well. Furthermore I will detail why specific attention to different onsets should be accounted for in the selection criteria as well as the data. Otherwise reproducibility might always remain a problem.

In short one can say: It's very hard to accurately distinguish between something you don't understand at all, i.e. ME/CFS, and something you hardly understand at all, i.e. a supposedly healthy human body, even if you account for different test and sample sets.

If we look at the Intramural study, which already had the strictest selection criteria of any ME/CFS study ever conducted, with extremely intricate examinations, even in that case it eventually turned out that 3 people didn't have ME/CFS. Statistically this would very roughly mean that you should always expect that at least 15% of people in ME/CFS studies don't actually have ME/CFS (one can argue that the sample size was to small to make this argument, however given the extensive pre-examinations and inclusion criteria it should be more than correct). Furthermore without knowing how ME/CFS works, we should definitely expect that not everything that is nowadays considered to be ME/CFS will be considered to be ME/CFS in the future and instead some diseases not even existing today with different markers might appear. Finally there might exist different subcategories of ME/CFS, for example between different viral onsets EBV vs HHV-6 vs MERS vs SARS-COV-2 and biomarkers could possibly be different for different onsets. Not once have I seen a study that accounts in its data set the different disease onsets.

The current work also seems to more and more rely blindly on machine learning classifiers that researchers often utilize in a way that instead of trying to find maximal differences between specific markers they want to maximise the accuracy of distinguishability between cohorts, i.e. ME/CFS vs HC or LC vs HC. In essence this means the are often looking for biomarker-tests where the difference in specific markers might be very small, but if we group some sets of markers together we get something that applies to everyone that was part of the study. Retrospectively after all classifiers have been tested one then chooses the one classifier that had the highest accuracy amongst all classifiers.

For me this shouldn't always be the main objective. I'd rather have a biomarker that only distinguishes something as low as 70% of cases of ME/CFS, but when it does so, the specific markers differ immensly compared to HC and some other fatiguing diseases. I'm yet to read papers that try to solve such maximisation problems.

Furthermore machine learning classifiers often lack any sort of interpretability. That means if you try to find out why the algorithm thinks that one marker should be prioritised above another one you simply don't know why. In your follow-up study it may now be a marginally different marker making your results irreproducible. As such classical statistical classification methods should be applied as well. Finally blindly applying different machine learning classifiers and then retrospectively choosing the one with the highest accuracy has the notion that what you're doing is solving a maximisation problem amongst classifiers rather than a maximisation problem amongst specific markers. These two problems don't necessarily have to be equivalent in small sample sizes and where the diagnosis of the disease won't be 100% accurate. This can cause problems if you try to reproduce your results in follow-up studies as it might turn out that a different classifier is optimal in different studies. Of course this could also be the case if you would have tried to maximise differences in certain marker values rather than maximise the accuracy of your test, however in that case you might have at least learned something about the disease, which might not be possible if your results lack interpretability. I'm by no means bashing machine learning models, we should just understand how to use them to solve the problem that we actually want to solve. The data should always be made available open source for re-analysis, so that the above problems are minimised.

To maximise chances of finding a biomarker for everyone, not only should the diagnostic criteria and disease severity be as strict as possible in trials, meaning possibly even stricter criteria than CCC+extensive exclusion of everything else, they should also consider very specific viral onsets as well or at least account for these by adding them to the data, especially in your classification models.

In this sense Covid-ME/CFS brings the ideal group of patients for studying which can be longitudinally studied from illness onset and compared to other ME/CFS groups with variying illness durations, such as patients that have ME/CFS from MERS or some other condition (MERS could be particularly good in this case, being from the same viral family). The RECOVER Long-Covid symptom study received a lot of hate (some of it being justified), but if you have a patient that meets the CCC+exclusion criteria and which has a very specific Covid onset which is further proven by a PCR-test and also has a loss of smell, then you can be pretty sure that he has ME/CFS (or at least Long-Covid-ME/CFS which after all could still be different from other forms of ME/CFS and even LC-ME/CFS might not always be the same for example EBV vs HHV-6 reactivation etc). In that sense the RECOVER study was great for study selection criteria and it seems many didn't understand that, probably because study was too open for misinterpretation by the media.

At the end of the day very severe ME/CFS patients are extremely sick. Possibly, sicker than patients from any other disease that exists in this world. This means that somewhere in their body things are going awfully wrong and some day in the future a marker will be found for this. Till then have to try to maximise our chances to find markers that are conclusive enough that we can work our way upstream to the "real problems". Even if these might initially only apply to a subset of patients. At the end of the day you should try to do thing as strict as possible, with the harshest possible set of patients, to hopefully get more conclusive results, which in turn allow you to understand the disease better, helping everybody. I believe Scheibenbogen and at the team at UCSF are both sort of working in this direction.

As such statements along the lines of "100% accurate biomarker for ME/CFS" don't seem very credible at all to me and if not compared to other diseases, probably just measure the inactivity of people in some way, possibly even indirectly.

 

[Wishful]

This means that somewhere in their body things are going awfully wrong

That's not necessarily true. ME could result from some subtle alterations in just a few cells, but those cells are critical and have significant--but hard to differentiate from normal--downstream effects. Maybe IL6 is a bit elevated, and malate is a bit low, and something else is slightly off, but altogether, they cripple collagen production or some such thing. In a modern vehicle, it doesn't take something "awfully wrong" to noticeably reduce driving performance; a bit of dust in the air filter could screw up performance if the computers aren't set up to adjust for that, especially if a few other factors are just slightly off.

I agree that researchers will probably come up with some markers that really just indicate that the patient is in less then perfect health due to a wide variety of factors, but if they adjust the research (and maybe the data), they can convince some politician or health insurance company that this test will reduce costs by giving patients a "Don't bother doing any more tests or treatments" designation.

 

[Osaca]

That's not necessarily true. ME could result from some subtle alterations in just a few cells, but those cells are critical and have significant--but hard to differentiate from normal--downstream effects.

Yes, I was thinking about that as well when I was writing my post. It's definitely a possibility that instead of having something go properly wrong, one instead has hundreds of little things going slightly wrong without at the same time causing anything to go properly wrong from a measurement perspective.

I think I then still decided to go for what I wrote in my post that something must be awfully wrong, because it seems to be the case in all other severe illnesses (which I could be wrong about) and it seems that one is relatively decent at measuring downstream effects, at least all advanced measuring technique studies in ME/CFS always do show some relatively extreme results ( I suppose in the metaphor of the vehicles's performance hindered by a dust of air, you could either measure that somehow more fuel is needed for the same performance or that some parts are running hot, in terms of ME/CFS we haven't even been able to do that reproducibly). Perhaps I also just jumped to the completely wrong conclusion due to my brain fog. Hard to understand what I was attempting to think about when I wrote this.

But as you mention it of course needn't be the case. The question would then be whether we can ever discover things that might only be varying combinations of possibly time dependent slightly off things.

 

[Wishful]

at least all advanced measuring technique studies in ME/CFS always do show some relatively extreme results

Do they? I haven't paid attention to the magnitudes of the abnormalities, because I lack the knowledge to interpret them as minor, major, etc. My impression was that many of the abnormalities were small enough that other experiments didn't find them (or were abnormal the opposite direction), and that many others were compared to "healthy controls", which might just mean that there's a difference between fully healthy people (with normal activity levels) and people who feel unwell and aren't active. I don't recall any reports of "PWME have <whatever marker> at 50x higher than people with comparable other diseases." Some immune markers might be radically high, but not compared to the levels normally found in common infections.

From my readings, we have various factors deviating enough to say that "we are unwell", but nothing "awfully wrong".

The possibility of ME being combinations of factors (different in individuals) is high enough that I think it's more likely that a treatment will be found accidentally. I view ME as a black box, where we are unlikely to understand the mechanisms inside, but by applying different inputs, we might find out ways to make the outputs less awful, or maybe even get it working properly again.

 

[Osaca]

@Wishful The sort of techniques I was referring too were rather some that measure the supposed downstream effects like endothelial dysfunction, so my statement wasn't written particularly well. I feel that whenever they use some advanced method like OCT-A, RVA or functional MRI's for endothelial dysfunction they do get some good results (possibly more results for Long-Covid). If these would be combined and EndoPAT and whether else still exists were used as well, I have the impression that this would give results. Of course these could still be purely downstream results and present in other diseases. But I do feel that endothelial dysfunction is one of the things that actually gives some results. Perhaps I'll be proven wrong in the upcoming years.

As far as bloodmarkers go, I have to agree with you. I don't think there's ever been a significant reproduced finding in ME/CFS. This is the main problem in ME/CFS research. You essentially have no measurments for the basis of theories.

I'm not really sure about the black box approach. I just find the human body too complex and too little is understood about it, independent of ME/CFS. It's sort of like a black box inside of a black box, which makes it harder to get lucky.

I suppose trialing treatments and doing research at the same time as Scheibenbogen proposes and plans to do is the right approach to raise your chances of "getting lucky" whilst learning.

 

[Wishful]

'm not really sure about the black box approach. I just find the human body too complex and too little is understood about it, independent of ME/CFS.

That's the situation where the black box approach does work (and the alternatives don't). You don't have to understand what's in the black box; you just change the inputs and see what happens. I didn't take cumin due to a theory based on knowledge of what it does in the body; I just observed that it had an effect on my symptoms, and it was a reliable effect. Black box solutions!

My view of eastern medicine is that they started with a completely nonsensical basis (four humours, magic energies), but by observing the responses to inputs (herbs, animal parts, etc) and changing their theories (anise is windy, except it treats <whichever ailment> which is supposedly due to too much wind, therefore we'll recategorize anise as earthy), this black box approach to medicine resulted in some treatments that work.


Hmmm, how do eastern experts deal with modern pharmaceuticals? Does each drug get a label of "hot or cold, or windy or earthy, or phlegmatic or melancholic"? Do they have diagrams for how each drug redirects Qi? Do they just avoid thinking about the subject?

 

[Osaca]

That's the situation where the black box approach does work (and the alternatives don't). You don't have to understand what's in the black box; you just change the inputs and see what happens. I didn't take cumin due to a theory based on knowledge of what it does in the body; I just observed that it had an effect on my symptoms, and it was a reliable effect. Black box solutions!

I definitely see your point. The problem as I view it is that you essentially have a blackbox, an ME/CFS patient, inside of your usual blackbox, a human body. Drugs by itself are often blackbox approaches, for the majority of them we don't even know understand their intricacies but rather have rough ideas of their pathways. A drug that would work for ME/CFS would more likely be one that's rather rare and possibly less well understood. Take for example Ampligen. That possibly, sort of works and yet we know nothing about ME/CFS.

I'm not saying this won't work. I think research and trials at the same time is an immensly valuable tool, which has to be used. I just think the combination of complex human body and complex disease makes it a bit more difficult. Even for diseases where one has biomarkers and a rough idea of how they work one still doesn't get very far in terms of therapeutics. In the scenario where you neither understand how the disease works and don't have a biomarker, trial outcomes are even harder to asses as you don't have too many meaningful measurements. Of course a blackbox inside a blackbox is just a blackbox, but it could be a blackbox of too great complexity for which guessing a solution is too complex.

In my naive opinion something comparably easy to do is to try to reproduce previous results. It’s not the sexiest work one can do, but a professor can basically tell his PhD student: These are the 4 most interesting findings in ME/CFS (I'm thinking things like red blood cell deformability, mitochondrial fragmentation and all other results that were never reproduced). Your job is to reproduce them and if they are reproduced, we begin building a theory of ME/CFS on top of them. It’s not the coolest of jobs and you won't get big grants for it, but it could be more significant than pumping out results that turn out to be of little value. Of course it can turn out that you actually can't reproduce the findings. In that case you at least know that you shouldn't be building further hypothesis or even mathematical models, as is currently being done, based on those results.

 

[Mary]

@Osaca - what about the 2-day CPET? ME/CFS patients had worse results than persons in any other group tested. Drawbacks of course are (1) it's expensive and (2) it's too difficult for many people with ME/CFS to even attempt, and for those who can, it has the potential to make them much worse.

https://me-pedia.org/wiki/Two-day_cardiopulmonary_exercise_test

From what I read, it's a conclusive biomarker for the existence of PEM, which is claimed to be the hallmark of ME/CFS. A simple blood test of course would be much easier for most patients to endure and I'm sure would be much cheaper, but we don't have one.

 

[Osaca]

Yeah, I agree, the 2 day or any multiday CPET definitely seems to be a tool to determine, who has moderate? or less severe ME/CFS.

At least it's the only test I know of that seems to have these capabilities, perhaps someone like Systrom can also further optimise it and learn even more from it then. However, I don't see too much room for it in treatment trials as input and output measure as it automatically complicates things and excludes anybody being worse off, which are to a larger extent those people that I argue should be part of trials. I'm not sure how it holds up against the BPS crew. After all it objectively shows forms of extreme problems against exertion, which according to them is psychological. They'll probably find a way to try to argue against anything that isn't a blood test, direct measurement or imaging technique. At least it seems to have enough downsides to not have halted the biomarker search.

Perhaps it would be great to establish a group of patients that are used for multiple trials and to increase reliability of those trials one let's them a priori do a 2 day CPET.

 

[Wishful]

Assume for the moment that ME is a combination of slightly abnormal levels of something in a fairly small number of brain cells: something really hard to identify with present medical technology.

Approach A is to measure all sorts of body fluids/tissues and do CT scans and fMRIs, etc, to find the subtle abnormalities, then try to find a drug that will reach those cells and correct the abnormalities ... without nasty side effects.

Approach B is to get some PWME, and give them all the different herbs, spices, fungi, drugs, etc, that are available for human use. Some of them might provide some changes in ME symptoms. Even the negative ones are useful, since "poking the black box" with them changes the outputs, which lets you consider other possible chemicals to try. There will be lots of individual responses, such as peroxynitrite scavengers making a few people worse, a few better, and most having no response. That probably indicates that it's affecting a downstream mechanism. However, if you test hundreds of thousands of chemicals, you might find some that have a similar effect on the majority of PWME. That might lead to an understanding of the mechanism of ME (abnormal GFAP for example), or the mechanism might remain a mystery, but still lead to a practical treatment.

I'd be more likely to donate to approach B than A.

 

[Osaca]

I don't think anybody is arguing against doing drug trials as well. My post didn't either, it was more about what in my view would be an easy optimisation of the current biomarker search. This optimisiation requires no extra means, costs or grants.

The problem about approach B is simply that we can't get it started. Someone like Scheibenbogen is trying it, but if you don't have the funding you won't get anywhere. There should have been more Ampligen trials years ago, Younger has been talking about Dex-Naltrexone for 10 years, but not even a LDN trial happened. Sometimes the drug companies will be the hurdle, and if you want to trial multiple drugs at the same time you can be sure that there'll be definite resistance from pharmaceutical companies, apart from ethics approval problems.

There should be more drug trials for Long-Covid as well. There's whole lists of sensible drugs: antivirals, JAK-STAT inhibitors, Allogenic CAR T cells, Peginterferon Lambda, immunomodulators, Anticoagulants/antiplatelets, monoclonal antibodies, things for the mitochondria, IVIG, things for the microbiome, Ampligen, Immunadsorption, Beta blockers, drugs that adress microglial cells, Efgartigimod, Peptides, anti inflammatories and a bunch of other things I didn't think of right now (IMO herbs and spices are a waste of resources). There's hundreds of things that could be trialed.

Unfortunately, it's just not happening. Will it ever happen I don't know. It's much harder to start drug trials, especially if thsi disease is not deemed relevant in some way, whilst research is still quite easily possible from your university position. It's probably a lot easier for the majority of researchers and research organisations to advocate for specific research rather than "random" trials.

Should the first few drugs trials for Long-Covid fail, I don't think we'll see a dramatic increase in drug trials for ME/CFS. If there'll be success, it will dribble over to ME/CFS.

Personally I'd donate to sensible research and sensible drug trials.

 

[Consul]

I think Scheibenbogen said there were like 40 drug trials underway in long covid. She was interviewed recently by gez medinger (or whatever his name is). Someone might want to double check that as i may have misheard the number.

 

[Osaca]

I think Scheibenbogen said there were like 40 drug trials underway in long covid. She was interviewed recently by gez medinger (or whatever his name is). Someone might want to double check that as i may have misheard the number.

There are, at least compared to ME/CFS, quite a few for Long-Covid. That's true, clinicaltrials.gov is quite full.

However, that includes trials like vitamin B, vitamin C and other things everbody has tried and which really don't warrant a trial at all. I think the list of useful trials is probably 10 items long (including things that people with ME/CFS have been longing for like Ampligen, Efgartigimod and LDN). I think this list has all the useful ones: www.twitter.com/realfrankbecker/status/1621215657235808256/photo/1. The situation is still extremely dire and doesn't inspire much hope.

I think Hannah Davis is doing some advocacy on the urgency of drug trials and which ones should be trialled. It seems that the team at UCSF will also kick of some more much needed smaller projects and if Polybio finds something in the first round of projects I'm sure there could be some possibilities there as well.

Hopefully RECOVER can be turned around, but so far their plan was to do 5 drug trials I believe.

 

[Osaca]

This is btw Scheibenbogen's plan for ME/CFS treatment trials which she presented at the Berlin conference, in case some people haven't seen it. It also shows how she doesn't have a cent of government funding after 2023 and has to rely on private donations. The most recent attempt to change that failed today (the christian party for the second time proposed a motion for a research strategy for ME/CFS, Long-Covid and Post-Vac in the German parliament, but it was declined. Not that the christian party had done anything in the past, it just seems like political games).

 

[Mary]

Yeah, I agree, the 2 day or any multiday CPET definitely seems to be a tool to determine, who has moderate? or less severe ME/CFS.

I think it does more than this. From what I've read about 2-day CPET testing done to date, ME/CFS patients are the ONLY ones who worsen on day 2. People with MS and congestive heart failure do better on day 2. And again, I think this relates to PEM, the hallmark of ME/CFS. So I think it could be considered a biomarker of sorts. But of course it does have the drawbacks noted above - many are too sick to even attempt it, or it can make some worse. At least it could exclude those who are well enough to do the test and DON'T get worse on day 2 from the ME/CFS category.

 

[Murph]

On the topic of human trials and "black boxes", what we really need is an animal model. If we could give mice me/cfs reliably, we could test drugs way more efficiently. The practical and ethical limitations on human trials mean human trials are always going to be few , and going straight from theoretical benchtop science to human trial has a high miss rate.

Using mice is cheap and fast and also you're allowed to euthanise mice at any time and see what's going on in their muscles and brains, so you can crack the black box open.

Of course the problem is we can't make a decent animal model until we understand what is going on!

But they can give mice ebv now, I wonder if infecting 1000 mice with ebv might generate a cohort of 20 mice with mecfs, who could then be studied further?!

 

[Osaca]

I think it does more than this. From what I've read about 2-day CPET testing done to date, ME/CFS patients are the ONLY ones who worsen on day 2. People with MS and congestive heart failure do better on day 2. And again, I think this relates to PEM, the hallmark of ME/CFS. So I think it could be considered a biomarker of sorts. But of course it does have the drawbacks noted above - many are too sick to even attempt it, or it can make some worse. At least it could exclude those who are well enough to do the test and DON'T get worse on day 2 from the ME/CFS category.

Yes, there were a lot of significantly positive results, differentiating ME/CFS not only from HC but even from MS and idiopathic chronic fatigue. It also seems though that there were unpublished, negative results by the dutch group of researchers that were amongst the leaders in employing the 2-day CPETs. It seems that they might have moved on a bit from using it and are more focused on POTS & chronotopic intolerance and possibly using a lot more reliable detection methods such as tilt table test + neck echo there. They are also struggling with funds and might not be able to do 2-day CPETs anymore, it also wasn't part of the new dutch ME/CFS projects.

As you mention it could also be very interesting that when one does studies to exclude those patients that don't get worse on day 2 or name them ME/CFS* in the data or something similar and acknowledge this in the input data to see if differentations are possible.

On the topic of human trials and "black boxes", what we really need is an animal model. If we could give mice me/cfs reliably, we could test drugs way more efficiently. The practical and ethical limitations on human trials mean human trials are always going to be few , and going straight from theoretical benchtop science to human trial has a high miss rate.

Oh yes definitely! An animal model would be an absolute game changer and it seems like many researchers seem to think that way as well. In fact any model could completely change the situation.

However, as you mention a model either requires some decent measurements or some understanding of some mechanisms either at a macroscopic scale, a microscopic scale or anywhere in-between. Unfortunately, none of this exists for ME/CFS and that worries me a bit when people come up with models if it isn't deemed to be on the more experimental side of things and they claim to have a model of ME/CFS.

Take for example the IFNa-itaconate-shunt-hypothesis. The great part: The model can easily make predictions and as such can very easily be verified or falsified. The "worrying" part: It's based on measurements by Chris Armstrong. From what I know these measurements were never reproduced and Hanson even got slightly conradictory results (I could be wrong on this). To me it makes sense to first verify the measurments and then to start to come up with theories (of course Phair & Davis know this, so there might be things I don't know).

There were actually some humorous comments on mice models and the possibilty to inject them with EBV here: https://forums.phoenixrising.me/threads/targeting-mitochondria-with-asha-091.90207/#post-2435955.

There were recently also some news from Amsterdam that they have a mouse model for Long-Covid https://projecten.zonmw.nl/nl/project/autonome-autoimmuniteit-als-een-oorzaak-van-mecvs-symptomen. However, I'm extremely sceptical of such announcements.

The currently biggest, or at least easiest use for animals might be for something like studying possible mechanisms of viral persistence of SARS-COV-2 (similar to what was done here https://www.biorxiv.org/content/10.1101/2023.04.04.535604v1.full.pdf) where one uses the knowledge of similar studies in Post Ebola or collaborates with Post-Ebola researchers to understand possible differences in humans and mice when it comes to viral persistence.

Scheibenbogen published a paper today on how drug trials for ME/CFS and LC should work https://www.frontiersin.org/articles/10.3389/fmed.2023.1194754/full. To me this is the most sensible pathway, rather than a pure blackbox approach.

 

[Wishful]

(IMO herbs and spices are a waste of resources)

Didn't many of the drugs originate in plant sources? I agree that common herbs&spices are not a top priority for extensive testing (ie. 6 month trial or whatever). I'm talking more of a shotgun approach: try as many different chemicals as possible on some PWME, and see whether the subjects notice any significant responses (positive or negative). For me, at least 6 chemicals had significant positive effects within 24 hrs. Plenty more had negative effects in that time frame. Only one (prednisone) took more than 24 hrs (5 days) to have a noticeable positive effect. Extending the testing time would increase the chances of effects, but that's a cost/benefit decision. However, if you buy a package of spice or even a minimum prescription of some drug, there's no major reason not to take more than one dose.

I think this shotgun approach would be comparatively cheap to do. You don't have to test all chemicals on every patient in the group; you could have millions of PWME each testing a few. Anything with a positive effect on one person would get more people testing it. This doesn't involve expensive blood tests or CPET tests or other tests to try to see miniscule changes in some factor; it's just an attempt to find chemicals that make a reliable dramatic difference.

I'm not impressed by lengthy trials that might involve manipulating the data to achieve the magic statistical result (not actual patient improvement) needed for a claim that it is a treatment, in order to get more funding, citations, etc.