• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

A Guide to XMRV Discussions

asleep

Senior Member
Messages
184
I want to point out two general ideas that keep popping up in discussions about XMRV research. I believe both of these ideas are factually inaccurate and threaten to undermine critical patient support for honest XMRV research.


Idea #1: Finding XMRV is easy

The first idea is that finding XMRV is simple, especially using PCR. There are two aspects to this idea.

The first aspect is that every researcher involved in this field is honest and competent and really wants nothing more than to validate the XMRV finding:

This is science and not a courtroom where motives matter. A scientist's motives are almost irrelevant, what counts is their data. Besides that I think your analysis of motives is incorrect anyway from what I know of these people.

The points I am personally persuaded of after many interactions with people involved in this issue is that there is no conspiracy, and the level of competence in all these studies is generally high.

Despite frequent assurances that all these researchers are honest and smart, we are left with a burning question: why do these scientists continue to produce such deeply flawed research that, from an outside view, does not resemble an honest, competent attempt to find XMRV? Why have none of them even tried to replicated successful methods? Do their actions match the characterizations commonly put forth? I say no, not even remotely.

The second aspect is that finding XMRV should be easy and that all of their attempts should have found it if it was really there:

...but these PCR tests used in ALL the XMRV studies have been capable of finding the level of XMRV infection claimed to be found by WPI.

This approach would find evidence of ANY entrenched XMRV infection.

These types of statements are assertions of opinion, dressed up to look like fact. Generally, the only evidence offered to support these assertions is the claim that these techniques work fine for other well-understood and well-characterized viruses. But there is no reason, logically or historically, that this should be a priori true for a novel virus like XMRV.

Furthermore, none of these negative studies have shown that their PCR or serology methods are capable of finding XMRV in a known positive clinical sample. As such, none of these negative studies have properly ruled out false negatives.

This flaw is usually painted over with apparent incredulity over the fact that the WPI was able to detect XMRV via PCR with purported ease:

The Science 'methods' included using PCR without culturing and finding XMRV sequences in 67% of banked CFS samples. This study was equal to that method and goes well beyond.

I have come to the conclusion that it really shouldn't be that hard to find. Why? Because it wasn't hard for the WPI to find.

They didn't have to go to extraordinary lengths to look for it. ..they didn't have to culture, they didn't have to use the 'wild-type' virus to validate it, they were able to validate it using VP62, they didn't have a particularly sensitive PCR...so why is the virus now, all of a sudden, on the limits of detection of PCR?

At a high level, these arguments assert that the future should pay heed to the past, that XMRV should behave according to our current state of knowledge. This is a fallacious argument.

On a more concrete level, these arguments fail to mention the macaque study, which shows that XMRV has a highly transient presence in the blood. They fail to mention that PCR is a very complex process open to many confounding factors (i.e. not all PCR methods are created equal). They fail to mention the Danielson et al study that showed PCR to be a very unreliable method of detecting XMRV in the blood. They fail to mention that the CFS cohort in the Science paper was highly viremic, which may have unintentionally helped the WPI find XMRV by an otherwise unreliable method.

Taken together, these aspects form an idea that artificially injects more credibility into the negative studies than is warranted. This idea glosses over the deep flaws in these studies and instead props them up on unsupported claims of goodwill and asserted simplicity.


Idea #2: True scientific replication is not important

This idea takes two forms. The first form is a mangling of what scientific replication truly is. Consider Wikipedia’s description of replication:

The results of an experiment performed by a particular researcher or group of researchers are generally evaluated by other independent researchers who repeat the same experiment themselves, based on the original experimental description. (emphasis mine)

In comparison, consider the following description of replication found on this forum:

Using different PCR tests is not a failure to replicate, but is actually what is required to replicate… (emphasis mine)

Note the stark difference. It is simply not true that different techniques constitute replication. Just because two studies use PCR (or serology), doesn’t mean they are the same methods. The devil is in the details.

The second and far more dangerous form of this idea that the XMRV hypothesis can be scientifically dis-proven without anyone actually replicating the WPI’s methods:

Exact replication of WPI's PCR test is less important than people realize...

This is like a cascading effect - the more the negative studies pile up and the more things people try that don't work - it gets harder and harder to see how XMRV will work out - even if they never replicate the study. (emphasis mine)

The idea that a finding can be scientifically dis-proven in this manner is fundamentally flawed. It is a perversion of the scientific method. It is a political argument, not a scientific argument. Often these claims will be accompanied by appeals to an unnamed researcher who confirms that this type of “science” is acceptable. It is not, and anyone who suggests otherwise does not believe in one of the most fundamental building blocks of the scientific method: reproducibility.


Conclusion

In conclusion, I believe these two broad ideas threaten to railroad much-needed public demand for legitimate XMRV research.

First, they threaten to convince people that finding XMRV is easy and therefore all negative studies should be considered equally as valid as the positive studies (even though they don’t replicate their methods). Secondly, they threaten to convince people that there exists some quantity of these flawed negative studies can override the lack of any true replication attempt.

This is, I believe, a dangerous recipe for short-circuiting XMRV research.
 

SOC

Senior Member
Messages
7,849
Wonderful, asleep! I've been thinking much the same thing without being able to sow the eloquence you gave us here. Thanks!
 

JohnnyD

Senior Member
Messages
206
This is, I believe, a dangerous recipe for short-circuiting XMRV research.

Thanks for posting Asleep, it is unfathomable why this is happening and I couldn't agree with you more.
 

cigana

Senior Member
Messages
1,095
Location
UK
Thanks for reposting Asleep. This quote just keeps haunting me:

Using different PCR tests is not a failure to replicate...

What I just can't understand is why some scientists can read the Science article and the Lo/Alter paper, go away and do their own test that is not the same, then come back and claim that their own test was correct and the others must be wrong. If I were a retrovirologist, I would be asking myself am I absolutely sure? I would be asking myself how the hell is it that the Science paper managed to only contaminate, consistently, the CFS samples? How the hell is that the PNAS paper did the same, and got the same rate in controls? Do we really think we know enough about this new retrovirus to be sure?

Why the hell aren't any of these people coming out and saying: "yeah we did the test and got all negatives, so it's really strange that the two original studies got consistent positives - guess this means this XMRV subject is trickier than we all thought and we need to do some more science to find out the truth. I for one am going to work very closely with the positive-study authors and try to get to the bottom of this. Perhaps there are many things we don't yet know about the conditions of the samples that are important. Perhaps we really need to do these studies on highly viremic people, since they might be more likely to harbour XMRV in the blood." ???

Instead they say things like: "We did our own study and because it disagrees with other studies, those other studies must be definitely wrong."

That just doesn't come across as an adventurous-minded, puzzle-solving scientist. It comes across as the type of statement from someone who is any of these:

(a) Very sure both of his/her findings and with a good understanding of why the positive studies are wrong in their consistency.
(b) Arrogant.
(c) Not that clever.
(d) Speaking with vested interests in mind.

Bring on the puzzle-solvers.
 

kurt

Senior Member
Messages
1,186
Location
USA
asleep, I disagree with your analysis and conclusions. This is my understanding:

1. Based on the ability of multiple labs to calibrate successfully to VP62, creating tests significantly more sensitive than WPI's, XMRV can be found by PCR, it is a standard type of testing now, any competent PCR test designer should be able to find it. Contrary to what has been posted on this forum, if you read Cooperatives' study, for example, you will see that they could find live XMRV in a prostate cancer sample. The tests work, and saying they don't just does not hold water anymore.

2. Precise replication is not the same as validation. Most of the follow-on XMRV studies are attempts to validate the original WPI finding. In the PCR world this means creating many different test designs, approaching the problem from many angles, so to speak. This is standard procedure in retroviral research with PCR testing where hundreds of different test designs can find the same viral DNA/RNA. These are independent labs that make their own decisions, and generally want to try out their own tests, nobody can or should control that process, free country, etc. I wish some lab HAD done a precise replication rather than a validation, but also understand that they don't see the value in that because they have to respond to investors or funding agencies and must do something new that contributes to science.

If you feel so strongly that the WPI experiment should be precisely repeated then I suggest all the activists here band together, raise some money and do just that! Again, this is a free country (sort of, in the US anyway). Be sure to use the older PCR methods and design WPI used originally in their Science experiment, with the same patient sample criteria. That may or may not produce the exact results WPI found, would be interesting to see.


You have had your say, this is mine:

Point #3. A few people in an online forum questioning a study does not threaten to derail anything. WPI and those supporting XMRV research will continue regardless of what we say here. And they should, I think XMRV is one of many legitimate research programs for CFS. What bothers me is all the overzealous one-sided reasoning about the current state of affairs in XMRV research. To me this looks like conspiracy theory 101, people are constructing arguments using circuitous logic that can not be defeated, then getting all bent out of shape if anyone disagrees. The hallmark of a conspiracy theory is that people who disagree are branded as plants for the alleged conspirators. That is what you and others are doing here, and I believe the XMRV discussions have become conspiracy theories, and not very scientific. Are you at all open to hearing the other side of the argument? Or have you decided that XMRV is what will validate CFS, so therefore any attack on XMRV is an attack on CFS?

Point #4. There is room in the CFS research community for more than XMRV. It is my opinion that the XMRV hypothesis deserved a fair chance, and it has received that. But XMRV is not the only candidate explanation for CFS that works, the Hydrogen Sulfide explanation, that was published just before WPI took the spotlight, has some very strong supporting evidence, for example. So do enteroviral damage explanations and the new proteonomic study. XMRV is not our last and only hope as some people seem to think. In fact, I think to over-focus on XMRV could short-circuit other important research directions in the CFS world right now, we need a balanced and broad-scale attack to solve this, and major funding. I realize there are people who think that XMRV is the only way we will get funding, and I think they have a point, a dangerous retrovirus would get us a lot of attention, for life. We would become pariah in society. Fortunately, I don't think that is likely and I believe the consensus is emerging in the studies that XMRV is not correlated with CFS. That is not to say there is no XMRV, it may have a presence and role in some human disease, for some patients. We have a lot yet to learn, and thinking could always shift when new evidence is uncovered.

Maybe funding will be harder without an attention-getter like XMRV, but not impossible. We still are here, are sick, and are being short-changed by medical research funding policy makers, and that has to be addressed, with or without XMRV...
 

kurt

Senior Member
Messages
1,186
Location
USA
What I just can't understand is why some scientists can read the Science article and the Lo/Alter paper, go away and do their own test that is not the same, then come back and claim that their own test was correct and the others must be wrong. If I were a retrovirologist, I would be asking myself am I absolutely sure? I would be asking myself how the hell is it that the Science paper managed to only contaminate, consistently, the CFS samples? How the hell is that the PNAS paper did the same, and got the same rate in controls? Do we really think we know enough about this new retrovirus to be sure?

Why the hell aren't any of these people coming out and saying: "yeah we did the test and got all negatives, so it's really strange that the two original studies got consistent positives - guess this means this XMRV subject is trickier than we all thought and we need to do some more science to find out the truth. I for one am going to work very closely with the positive-study authors and try to get to the bottom of this. Perhaps there are many things we don't yet know about the conditions of the samples that are important. Perhaps we really need to do these studies on highly viremic people, since they might be more likely to harbour XMRV in the blood." ???

Instead they say things like: "We did our own study and because it disagrees with other studies, those other studies must be definitely wrong."

What these other scientists know that we don't, they know the track record of retroviral hunts, they know the science, they know the history, they carefully read what WPI wrote and say 'I can find a virus with that sequence in that dilution'. Then when they don't find it they go back to prior history and work all the angles. That is what is happening. There is a history of false early findings in retroviral hunting, including different results for controls. My point is that there are possible explanations for what happened, there can be interactions between reagents, different testing, etc. WPI did at one point admit that they tested the CFS samples more times than the controls, and the controls were a convenient sample (Which is common in this type of research apparently), and not all collected together using the same exact procedures.
 

asleep

Senior Member
Messages
184
1. Based on the ability of multiple labs to calibrate successfully to VP62, creating tests significantly more sensitive than WPI's, XMRV can be found by PCR, it is a standard type of testing now, any competent PCR test designer should be able to find it. Contrary to what has been posted on this forum, if you read Cooperatives' study, for example, you will see that they could find live XMRV in a prostate cancer sample. The tests work, and saying they don't just does not hold water anymore.

I've asked Cort before (here) to provide evidence for the claim that the WPI calibrated their PCR to VP62, but he failed to deliver. Perhaps you have evidence to offer.

But really, this fact doesn't matter. As I noted in my original post, what truly matters is that none of the negative studies have ruled out false negative results by using a positive clinical control sample. I touched upon some of the many reasons why false negatives are a very real possibility. There is some great discussion here by jace, omerbasket, and others about the complexities of PCR, esp when dealing with a novel virus.

Furthermore, finding XMRV in prostate tissue is not the same as finding it in blood. Nor does it prove that one's PCR assay is capable of finding XMRV in clinical blood samples.

In effect, you are promoting here a variation of the "PCR is easy" idea I addressed above. Namely, you are saying that claims of superior sensitivity under different conditions will suffice in lieu of actually proving your assay's ability to find XMRV under the correct clinical conditions. This is not logical.

2. Precise replication is not the same as validation.

Then why is Cooperative Diagnostics trying to peddle the idea that they are the same?

2. Precise replication is not the same as validation. Most of the follow-on XMRV studies are attempts to validate the original WPI finding. In the PCR world this means creating many different test designs, approaching the problem from many angles, so to speak. This is standard procedure in retroviral research with PCR testing where hundreds of different test designs can find the same viral DNA/RNA. These are independent labs that make their own decisions, and generally want to try out their own tests, nobody can or should control that process, free country, etc. I wish some lab HAD done a precise replication rather than a validation, but also understand that they don't see the value in that because they have to respond to investors or funding agencies and must do something new that contributes to science.

Validation studies are important, and no one is trying to control what research labs conduct. What I and others are demanding is proper scientific rigor in interpreting the research that has been done.

While validation has its place, it also has its limits. For example, if a validation study that uses different methods fails to corroborate an original finding, it cannot generally be said to invalidate the finding. Again, saying so is not logical. This is confusing absence of evidence with evidence of absence.

When genuinely inquisitive scientists fail to corroborate a study by validation, they usually fall back on true replication. They do not put out press releases stating that the case is closed, everyone go home.

Point #3. A few people in an online forum questioning a study does not threaten to derail anything.

Here are two Admins from this forum expressing their personal views that XMRV is over (the second is a bit more subtle, but the implication is there):

I just think - and I'm just a laymen obviously - that it doesn't look like its going to work out. Just a personal opinion....

It is my opinion that the XMRV hypothesis deserved a fair chance, and it has received that. (emphasis mine)

I don't begrudge anyone their opinion. But let's not disingenuously pretend that these statements don't influence the opinions of others, especially when they come from individuals who put themselves forward as knowledgeable, connected, and "in the know." I believe these opinions are based on selective evidence and flawed reasoning, and I and others have consistently pointed this out. That such ill-supported statements have a chilling effect on support for XMRV should be quite obvious.

What bothers me is all the overzealous one-sided reasoning about the current state of affairs in XMRV research.

Whoa there cowboy! I would humbly suggest a quick glance in the mirror, but I'm beginning to suspect that yours is malfunctioning...

To me this looks like conspiracy theory 101, people are constructing arguments using circuitous logic that can not be defeated, then getting all bent out of shape if anyone disagrees. The hallmark of a conspiracy theory is that people who disagree are branded as plants for the alleged conspirators. That is what you and others are doing here, and I believe the XMRV discussions have become conspiracy theories, and not very scientific. Are you at all open to hearing the other side of the argument? Or have you decided that XMRV is what will validate CFS, so therefore any attack on XMRV is an attack on CFS?

Really? This is the best you can do? Rather than address my points you instead rely on highly emotionally charged labels to try to indirectly undermine them.

Point #4. There is room in the CFS research community for more than XMRV. It is my opinion that the XMRV hypothesis deserved a fair chance, and it has received that. But XMRV is not the only candidate explanation for CFS that works, the Hydrogen Sulfide explanation, that was published just before WPI took the spotlight, has some very strong supporting evidence, for example. So do enteroviral damage explanations and the new proteonomic study. XMRV is not our last and only hope as some people seem to think. In fact, I think to over-focus on XMRV could short-circuit other important research directions in the CFS world right now, we need a balanced and broad-scale attack to solve this, and major funding. I realize there are people who think that XMRV is the only way we will get funding, and I think they have a point, a dangerous retrovirus would get us a lot of attention, for life. We would become pariah in society. Fortunately, I don't think that is likely and I believe the consensus is emerging in the studies that XMRV is not correlated with CFS. That is not to say there is no XMRV, it may have a presence and role in some human disease, for some patients. We have a lot yet to learn, and thinking could always shift when new evidence is uncovered.

I've never suggested that other research doesn't matter. You've manufactured a false dichotomy here in order to deflect criticism. In other words, you've created a straw man. Pointing out the grave flaws in your reasoning about XMRV (which I have done) is not even close to the same thing as bashing other lines of research (which I have never done, with the exception of psychobabble). You are attempting to shield yourself behind something not even being discussed!

The reason that people are particularly vigilant about XMRV, as opposed to other legitimate lines of research, is that it is by far the foremost hypothesis on the table with real causality explanation potential. People are not willing to settle for half-baked science in hopes that "Gee, maybe something else will come along." If XMRV is the cause, then there is no "something else," and unchallenged, biased conclusions will only take us further from the truth.
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Hi, our understanding of XMRV is so new we are having to relearn everything as we go along. In my opinion it is clear that finding XMRV is very difficult, at least without organ harvesting, and in order to validate or replicate findings properly we need a good understanding of the mechanisms involved in the new virus - which is about more than just validation or replication. If we don't understand the principles, it is hard to judge what is a good replication study. Personally I suspect that only a study done with assistance by the WPI could be a full replication study. The rest are all attempts at validation studies.

There is an abstract in the CROI thread that indicates it is very likely that XMRV is due to recombination of two mouse MLV strains. This does not prove contamination, but it does open the door, and there will be no mouse DNA to be detected. However, it could also be that XMRV has done this many times naturally, leading to multiple epidemic epicenters, possibly including the 1932-1934 Californian outbreak of what looks like it might be ME/CFS.

We need to know the truth about XMRV, one way or the other. This requires good research and analysis, and vigorous debate. I am fairly sure that at least a few scientists would look at discussions here on these forums. It will not influence them unless we say something very pertinent, but they may get many ideas that help them decide where to go next. We are a large global sounding board for research ideas - we are not just explaining things to fellow patients in my opinion.

Bye,
Alex
 

kurt

Senior Member
Messages
1,186
Location
USA
I've asked Cort before (here) to provide evidence for the claim that the WPI calibrated their PCR to VP62, but he failed to deliver. Perhaps you have evidence to offer.

That is in the WPI Science article supplement, here is a direct quote, from p.8:

Figure S1. Gag sequences of XMRV in CFS patients. Partial sequences (nt 649-
1017) from CFS XMRV strains WPI-1130, WPI-1138 and WPI-1169 in
comparison to XMRV strains VP35, VP42 and VP62 derived from prostate
cancer patients. The yellow highlighting denotes the differences from the
reference strain (VP62).

But really, this fact doesn't matter. As I noted in my original post, what truly matters is that none of the negative studies have ruled out false negative results by using a positive clinical control sample. I touched upon some of the many reasons why false negatives are a very real possibility. There is some great discussion here by jace, omerbasket, and others about the complexities of PCR, esp when dealing with a novel virus.

Yes that fact does matter, all the facts here matter. WPI used VP62 as their 'reference strain' and so did the other studies. They all had tests capable of finding sequences in VP62. What was different? Most of the validation attempts (the 'negative studies') used conserved regions of the genome, more stable regions that are present in virtually ALL possible XMRV strains (and I believe in some cases all MuLV strains). So they actually covered more possibilities than WPI did, and still found nothing. As to positive clinical control samples, whether any exist is in dispute, WPI is not a reference point any more than the other labs in this type of validation/consensus process.

Furthermore, finding XMRV in prostate tissue is not the same as finding it in blood. Nor does it prove that one's PCR assay is capable of finding XMRV in clinical blood samples.

I believe the Cooperative study used blood in all their positive controls, not water like some of the other studies. I don't know if the prostate sample was spiked in blood or tissue.

In effect, you are promoting here a variation of the "PCR is easy" idea I addressed above. Namely, you are saying that claims of superior sensitivity under different conditions will suffice in lieu of actually proving your assay's ability to find XMRV under the correct clinical conditions. This is not logical.

What do you mean by 'actually proving your assay's ability to find XMRV'? These labs did the SAME THING that WPI did to calibrate their tests, if WPI could get hits on positive controls, and so did the other tests, how is WPI's test superior? It is not, what is in question is whether there is a real clinical sample. And at least one lab, the CDC did test some of WPIs positive samples and found them negative, using a test calibrated the same way WPI calibrated theirs. What basis do you or anyone have for saying who is right in that case? Science relies on a concurrence of multiple studies in that case, and that is what is now emerging, every week we seem to be getting multiple negative reports. I am simply calling attention to the fact that these are all valid studies, despite the speculation to the contrary. When will people accept that, do we need 1000 negative studies?

Validation studies are important, and no one is trying to control what research labs conduct. What I and others are demanding is proper scientific rigor in interpreting the research that has been done.
While validation has its place, it also has its limits. For example, if a validation study that uses different methods fails to corroborate an original finding, it cannot generally be said to invalidate the finding. Again, saying so is not logical. This is confusing absence of evidence with evidence of absence.
When genuinely inquisitive scientists fail to corroborate a study by validation, they usually fall back on true replication. They do not put out press releases stating that the case is closed, everyone go home.

Unlike you and I and the other CFS patients studying this issue and commenting online, these researchers have a deep-level understanding of PCR and culture testing techniques. They follow standards of scientific rigor, and they are the ones who have to debate whether their tests are valid. I have heard and read the debates and see no validity in the patient-generated arguments that every study but WPI's and the FDA's are hopelessly flawed. That seems to me like wishful thinking by people who believe the XMRV hypothesis will help them. There are definitely CFS patients online who are desperate for help and want to recover who would benefit tremendously if they had a finding like this to take to their respective government health services. Don't you think that desire could be influencing their judgement a little here?

Here are two Admins from this forum expressing their personal views that XMRV is over (the second is a bit more subtle, but the implication is there)

What implication? That we think the odds are stacking up against XMRV? What on earth is wrong with having a viewpoint about that? If all the studies were finding XMRV and we were sitting there saying, nope, it is a falsehood, then you might have reason to question whether there were ulterior motives. But in fact the opposite is happening, but people don't want to believe what is being published.

I don't begrudge anyone their opinion. But let's not disingenuously pretend that these statements don't influence the opinions of others, especially when they come from individuals who put themselves forward as knowledgeable, connected, and "in the know." I believe these opinions are based on selective evidence and flawed reasoning, and I and others have consistently pointed this out. That such ill-supported statements have a chilling effect on support for XMRV should be quite obvious.

No pretense, I hope my statements influence others to think critically about what is going on here. You have not yet pointed out a flaw that I believe agrees with the data. These researcher are not committing huge errors like they are being accused of. They are simply not finding XMRV in CFS samples, that is a scientific result. I believe the argument that their tests are all flawed is based on a misunderstanding of PCR testing, being promoted by people who have their own agendas, and are trying to promote their own theories of what is happening. They have that right, and I have the right to disagree.

The reason that people are particularly vigilant about XMRV, as opposed to other legitimate lines of research, is that it is by far the foremost hypothesis on the table with real causality explanation potential.

There are other viruses (and parasites, bacteria, spirochetes, etc) that can cause immune and neurological problems like we see in CFS, and even ordinary bugs like Herpes can deplete glutathione and damage mitochondria. Which is the foremost hypothesis? I don't think we have enough funded research going on to know for certain at this point.
 

Wonko

Senior Member
Messages
1,467
Location
The other side.
Kurt and others are entitled to their veiwpoint, it's probably based on their knowledge and experience.

I and others are equally entitled to disagree at this point in time, based on our understanding, and change our opinions if, as and when further info comes along.

IMO (probably based on my vastly inferior understanding of the situation and as, stated above by Kurt, my desire for this 'promising' candidate as to WTF's wrong with me to work out) XMRV still looks a likely candiate.

Having said that I'm not 'commited' to XMRV, TBH I dont care, it's not written in stone in the core of my being that I have ME and it's caused by a retroviurs called XMRV - I dont care WTF causes ME - I just want my life back.

So please, people may not agree with him but Kurt and others are entitled to their views, and I find his posts generally to be informative and enlightning - it is possible to learn even from those you dont agree with - in fact it can be the best way to learn.

Give him some credit - his posts are at least better than mine lol
 

SOC

Senior Member
Messages
7,849
Original Research 101:
Discovery Study: Research resulting in a newly discovered result. Result is not considered proved, quality of research is considered a broad measure of the likelihood of a genuine discovery.

Confirmation Study: First replication or validation study that verifies the result of the discovery study. Quality is a again considered a factor in the confidence scientists have in the result.

Validation Study: A validation study is not an "invalidation study". It does not disprove the discovery study because it does not use exactly the same methods as the discovery study. Any change in the methodology could be the source of the failure to validate. Because the discovery is a previously undiscovered result, it is expected that many other methodologies will not find the same result -- that's why it was previously undiscovered. (duh) A validation can be considered a confirmation of the discovery because it reached the same result by a different method, thereby increasing the likelihood the discovery is true.
Summary: A validation study can validate, but not invalidate a discovery.

Replication Study: Research that exactly duplicates the methodology of the discovery study. If the replication study confirms the discovery, the discovery is generally considered proved. If the replication study disproves the discovery and is of roughly the same quality (rigorously reviewed and passed for publication), the discovery is generally considered unproved. When multiple replication studies do not achieve the same result as the discovery paper, the discovery is tentatively considered disproved.
Summary: Multiple replication studies are needed to disprove a discovery.

Besides being basic science, the above is also basic logic.

The best guess I can come up with as to why some of the scientists involved are claiming a negative validation study disproves a discovery study is that those scientists don't generally work in genuine discovery research. Very little science involves making new discoveries. Most involves applying long discovered and confirmed science. A large amount of research involves studying variations and applications of established scientific knowledge. In those areas, the issues of new discovery don't apply. It's more a question of is my method better than your method for achieving some purpose within the bounds of existing knowledge. Perhaps those scientists have lost sight of the details of how validation of new discovery works and rigor necessary to prove or disprove new discoveries.

At this early stage of XMRV research, nothing is absolute. However, the existence of XMRV has so far met the scientific requirements to be generally considered proved -- high quality discovery papers in two different areas -- PC and ME/CFS, and a high-quality validation paper -- Alter/Lo. So far, no one has done a replication paper that would suggest any of the three study results is incorrect. It may yet happen. It's early days, yet. But as of today, it hasn't happened.

Everyone is entitled to their own opinion about what the current state of research tells us, personally. That is a different matter from what is the current state of the research. I choose to believe, at present, that XMRV exists at some level in PC and ME/CFS patients, but understand that this issue is still open to be disproved by multiple high-quality replication studies.

For a clear picture of how the discovery/validation/invalidation process works, see the history of Cold Fusion -- a discovery that was ultimately scientifically disproved through multiple replication studies.
 

kurt

Senior Member
Messages
1,186
Location
USA
Original Research 101:
Discovery Study: Research resulting in a newly discovered result. Result is not considered proved, quality of research is considered a broad measure of the likelihood of a genuine discovery.

Confirmation Study: First replication or validation study that verifies the result of the discovery study. Quality is a again considered a factor in the confidence scientists have in the result.

Validation Study: A validation study is not an "invalidation study". It does not disprove the discovery study because it does not use exactly the same methods as the discovery study. Any change in the methodology could be the source of the failure to validate. ...

Replication Study: Research that exactly duplicates the methodology of the discovery study. If the replication study confirms the discovery, the discovery is generally considered proved. ...

Summary: Multiple replication studies are needed to disprove a discovery.

The best guess I can come up with as to why some of the scientists involved are claiming a negative validation study disproves a discovery study is that those scientists don't generally work in genuine discovery research. Very little science involves making new discoveries. Most involves applying long discovered and confirmed science. A large amount of research involves studying variations and applications of established scientific knowledge. In those areas, the issues of new discovery don't apply. It's more a question of is my method better than your method for achieving some purpose within the bounds of existing knowledge. Perhaps those scientists have lost sight of the details of how validation of new discovery works and rigor necessary to prove or disprove new discoveries.

At this early stage of XMRV research, nothing is absolute. ...

For a clear picture of how the discovery/validation/invalidation process works, see the history of Cold Fusion -- a discovery that was ultimately scientifically disproved through multiple replication studies.

I agree with much of this. But I don't think the problem is that the scientists are out of their element as you and others seem to suggest. Here is my view on that issue. With PCR testing it is possible to have a test that lies somewhat between the formal definitions of replication and validation you included above. That might be part of the problem in the use of that terminology here. Virtually all of the negative PCR studies fit this, they have replicated the calibration process, using the same exact control antigen (VP62), which proves that their own PCR design can find the same viral sequences. But then they run a validation design, they design a test using a differing master mix or process they believe will help confirm (or deny) the original finding. In many cases the labs attempted to achieve a higher resolution, so they actually improved on sensitivity. This is apparently the norm in viral hunts using PCR testing today, so for them this is a perfectly reasonable scientific rigor.

So you are correct I think, nobody has conducted a precise replication by the strictest definition. My prediction is that if someone else conducts a precise replication they will have 0/0 results, whereas if WPI conducts a precise replication they will get somewhat similar results to what they got the first time. I don't think the difference here is due to lack of precise replication. Standard PCR testing should be working here. Something else is going on, perhaps a complex contamination, not in the labs but in reagents, interactions with something in PWC blood. Here is an example. CFS patients have high H2S vs controls, wouldn't that be wild if WPI had stumbled on something that interacts with H2S in their particular PCR test! If they worked on calibration long enough who knows what they may have tweaked out of that test. And their other tests depend on PCR to be valid, so it really comes down to PCR to settle this debate. And as I have tried to explain, that is not as straightforward of an issue as in some other areas in science.

One other difference may be testing different regions of the genome. WPI tested a narrower region than most of the other studies, which mostly looked at conserved regions. The other labs did that to pick up a broader array of possible hits on MuLV type viruses. Maybe the testing of the narrower region has some different interaction than expected, like I mentioned above. But I would not count on a change of the tide if someone replicates WPI precisely, even if they are positive. They would still have to explain why testing the conserved region (validation test) with a replicated calibration produces 0/0 finding.

You remember Pons & Fleishmann! There were quite a few articles on that theory published in electrochemistry journals, I believe at one point an article by those scientists appeared in Nature. There is no way editors can second-guess problems that are not yet apparent.
 

SOC

Senior Member
Messages
7,849
I agree with much of this. But I don't think the problem is that the scientists are out of their element as you and others seem to suggest.

I don't suggest anything. I very deliberately stated that I was making my best guess at a reason scientists would, in good faith, not apply the usual standards of scientific validation, and that perhaps they weren't familiar with the usual practices of discovery research. I do not know enough about the backgrounds of the scientists involved to do more than guess at an honest motive. My guess is as good or as bad as anyone else's guess.

But then they run a validation design, they design a test using a differing master mix or process they believe will help confirm (or deny) the original finding.
[my bolding]

This is exactly my point. They choose to use different methods from the discovery study and by making those changes introduce new factors not in the original study. Those changes could be the source of the failure to confirm. That is exactly why validation studies are not replication studies.

Their "belief" that their process is equal or superior for achieving the result does not make their process equivalent to the process in the discovery paper.

This is all about logic and scientific rigor. Logically, you can't show a method does not achieve it's end by using another method and achieving a different end. All you've proved is that your method does not achieve that end. Not the same thing. This is why replication is critical for disproving a discovery.

Replication studies are not needed to prove a discovery. Validation studies are better in that regard because they show that the finding can be made using multiple methods, which enhances the likelihood that the discovery is true.

Replication studies are a tool for establishing that the discovery could be false. That's why it's so odd that the XMRV deniers have not yet done a replication study.

In many cases the labs attempted to achieve a higher resolution, so they actually improved on sensitivity. This is apparently the norm in viral hunts using PCR testing today, so for them this is a perfectly reasonable scientific rigor.

This is fine for a validation study. Changing the method is NOT a part of a replication study, which by it's very nature should exactly replicate the methods of the discovery study.

If virology has departed from the basic logic of discovery science, that's news to me. Maybe it has happened, but I've yet to hear a logical reason why it would. I'll need more confirmation than your word for it.

So you are correct I think, nobody has conducted a precise replication by the strictest definition. My prediction is that if someone else conducts a precise replication they will have 0/0 results, whereas if WPI conducts a precise replication they will get somewhat similar results to what they got the first time. I don't think the difference here is due to lack of precise replication.

Your prediction is fine as a layman's prediction. Any of us can make predictions. They're not scientific facts. You don't "think the difference here is due to lack of precise replication." Fine. It's a perfectly good thought -- an opinion, in fact. You're entitled to that opinion and are encouraged to share your opinion freely at PR.

I'm not talking about our opinions, but the requirements of scientific proof and disproof.

Standard PCR testing should be working here.

Again with the "should". What should happen based on existing knowledge is not necessarily what happens when new discoveries occur. "Should" is not scientific evidence.

And as I have tried to explain, that is not as straightforward of an issue as in some other areas in science.

That may very well be. That doesn't negate the basic logic involved in scientific proof (and disproof). You can't prove the result is not possible with a given method by using a different method. You can only show that you can't achieve the same result with the different method.

One other difference may be testing different regions of the genome.

Again, it's all in the differences.

But I would not count on a change of the tide if someone replicates WPI precisely, even if they are positive.

Okay, don't. Fine with me. I choose to go by the scientific process -- multiple positive validation and replication studies reinforce the discovered conclusion. Multiple negative replication studies disprove the discovered conclusion. You base your opinions on anything you like.

They would still have to explain why testing the conserved region (validation test) with a replicated calibration produces 0/0 finding.

No, they don't. The 0/0 findings need to be explained by the people who made them.

You remember Pons & Fleishmann! There were quite a few articles on that theory published in electrochemistry journals, I believe at one point an article by those scientists appeared in Nature. There is no way editors can second-guess problems that are not yet apparent.

Of course they can't. That's why the logical, scientific process exists to disprove faulty conclusions.

So far no one has undertaken the established, logical process for disproving that XMRV exists. When they do multiple, quality (rigorously reviewed in a competitive journal) replications that disprove the existence of XMRV, then I'll believe that the preponderance of evidence is that XMRV doesn't exist.
 

asleep

Senior Member
Messages
184
Thank you, Sickofcfs, for your very cogent analysis!

Kurt, I'm not going to address your entire reply to me because you are mostly arguing around my points and because Sickofcfs has roundly defrocked most of your arguments already.

I do want to (re)address one lingering misconception that you've expressed in a few different ways:

What do you mean by 'actually proving your assay's ability to find XMRV'?

You have not yet pointed out a flaw that I believe agrees with the data. These researcher are not committing huge errors like they are being accused of.

All of these negative studies contain the same, basic flaw (which is related to your question): none of them have sufficiently ruled out false negatives. Therefore, their results have very little scientific value.

The ruling out of false negatives is the crux of the difference between validation and replication. Different methods open up endless possibilities for introducing false negatives.

Now, to address your question about 'actually proving your assay's ability to find XMRV':

Very simply, this means that you must demonstrate that your methods can find what you claim they can find. In this case, these studies claim their methods can find XMRV in an XMRV+ clinical sample (i.e. blood drawn from a human being who is infected with XMRV). Yet none of these studies demonstrated this precise ability.

Showing that you can find XMRV in spiked water samples is insufficient. Showing that you can find XMRV in spiked blood samples is insufficient. Showing that you can find XMRV in different tissues (e.g. prostate) is insufficient. Calibrating your assay to a certain sequence strain without showing it's ability to detect XMRV in a genuine positive clinical sample is insufficient.

Hence, why the following statements are misleading:

I believe the Cooperative study used blood in all their positive controls, not water like some of the other studies. I don't know if the prostate sample was spiked in blood or tissue.

These labs did the SAME THING that WPI did to calibrate their tests...

I also want to note that using a genuine XMRV+ clinical control sample is a minimum requirement. It does not 100% rule out false negatives for myriad potential reasons.

(Consider the parallel to positive studies: to be taken seriously, these studies must take sufficient measures to rule out false positives (e.g. contamination), but these measures cannot 100% rule out false positives.)

As to positive clinical control samples, whether any exist is in dispute, WPI is not a reference point any more than the other labs in this type of validation/consensus process.

I grant that there is a bit of an epistemological catch-22 here. However, I would say that "dispute" is probably too strong of a word. Some starting point must be identified in order to bootstrap the science. And here, I think that some credence must be given to the multiple methods (culturing, serology, etc) employed by the WPI (and others). Their ability to find XMRV in a variety of ways, I think, gives their positive samples a temporary benefit of the doubt and places the burden of disproving the genuine positivity of these samples with those who would contest this.
 

SOC

Senior Member
Messages
7,849
A little science history about Pons, Fleishmann, and Cold Fusion

I realized after my last response that Kurt and I mentioned some science history that others may not be aware of, so I thought that for completeness I should add some background.

Two scientists (Pons and Fleishmann) announced some amazing research showing that they could produce safe nuclear energy in small amounts with a remarkably a simple process (Cold Fusion). If verified, this research could be the end of energy crises, nuclear fission (the dangerous kind) energy production, and many other energy production problems.

While Pons and Fleishmann did submit their paper to Nature, a very highly regarded scientific journal, it was rejected because they did not sufficiently describe their methodology, a necessary requirement for reproducible science. Eventually their work was published in a specialist journal, The Journal of Electroanalytical Chemistry .

Many replications of their research were performed, but the overwhelming number did not get the same results. The research is now considered disproved. When scientists duplicated the Pons/Fleishmann results by introducing likely possible scientific errors, the Pons/Fleishmann study was consigned to the murky depths of sloppy, flawed, and generally poor research.

Points of interest:
(1)The Cold Fusion paper was turned down by a highly reputable journal because of its lack of scientific rigor.
(2) The Cold Fusion paper was eventually published by a specialist journal with apparently less rigorous review and standards.
(3) The results were disproved by multiple replication studies which failed to achieve the same results

I said:
For a clear picture of how the discovery/validation/invalidation process works, see the history of Cold Fusion -- a discovery that was ultimately scientifically disproved through multiple replication studies.

Kurt said:
You remember Pons & Fleishmann! There were quite a few articles on that theory published in electrochemistry journals, I believe at one point an article by those scientists appeared in Nature. There is no way editors can second-guess problems that are not yet apparent.

How does this relate to XMRV research?

Firstly, the very best research is published is the the most highly regarded journals. In order to get accepted by those journals, they are reviewed by top experts and held to a very strict standard of scientific proof. The Lombardi and Alter/Lo papers were published in very highly regarded journals with strict scientific standards.

Secondly, the scientific process eventually sorts out erroneous results through the logical, established replication and validation processes.

Thirdly, poor research does get published, as does legitimate research later determined erroneous by developments in the field. It very rarely occurs in the most reputable journals, however.

Finally, editors can make errors, but publication is also dependent on the reviewers who, in the best journals, are experts in their field and can usually find even extremely subtle errors.

That does not completely eliminate the possibility of research published in highly reputable journals being disproved. That is always a possibility through (this is getting tedious to repeat) multiple replication studies.

The process is almost always self-correcting. Eventually.
 

ukxmrv

Senior Member
Messages
4,413
Location
London
No worries SickofCFS,

I think most people have heard of the Cold Fusion saga. It seemed to be very well covered by the media at the time.