• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Dr James Coyne tackles the PACE follow-up paper

Sidereal

Senior Member
Messages
4,856
Uninterpretable: Fatal flaws in PACE Chronic Fatigue Syndrome follow-up study

It's a long read. Here's the summary:

  • The PACE investigators sacrificed any possibility of meaningful long-term follow-up by breaking protocol and issuing patient testimonials about CBT before accrual was even completed.
  • This already fatal flaw was compounded with a loose recommendation for treatment after the intervention phase of the trial ended. The investigators provide poor documentation of which treatment was taken up by which patients and whether there was crossover in the treatment being received during follow up.
  • Investigators’ attempts to correct methodological issues with statistical strategies lapses into voodoo statistics.
  • The primary outcome self-report variables are susceptible to manipulation, investigator preferences for particular treatments, peer pressure, and confounding with mental health variables.
  • The Pace investigators exploited ambiguities in the design and execution of their trial with self-congratulatory, confirmatory bias.
 
Last edited:

Cheshire

Senior Member
Messages
1,129
1When-Cherished-Beliefs-Clash-with-Evidence-218x300.png
 

Sidereal

Senior Member
Messages
4,856
Which statement is to be given precedence? To the extent that features of a randomized trial have been preserved in the follow-up (which we will see, is not actually the case), a lack of between group differences at follow-up should be given precedence over any persistence of change within groups from baseline. That is a not controversial point for interpreting clinical trials.

This is the key point. It is highly unusual and runs contrary to the practice of conduct and reporting of RCTs to focus on the within-group changes when you have between-group comparisons. This is Clinical Trials 101. All groups will tend to report some improvements over time due to various factors like regression toward the mean, placebo effect etc. The key outcome in any RCT is not changes over time but differences between trial arms. And of course we know that this was a null finding in the follow-up paper which is why the press releases have grotesquely spun the story to focus on the within-group changes over time.
 

Sidereal

Senior Member
Messages
4,856
If we’re going to be cautious and qualified in our statements, there are lots of other explanations for similar outcomes in the intervention and control groups that are more plausible. Simply put and without unsubstantiated assumptions, any group differences observed earlier have dissipated. Any advantages of CBT and GET are not sustained.

Exactly. The two other groups have caught up with the CBT and GET groups over time. There is no effect of CBT or GET at 2.5 years' follow-up. None. There is only a significant effect of time but not treatment arm.
 

Sidereal

Senior Member
Messages
4,856
Any investigator group that would deliberately violate protocol in this manner deserves further scrutiny for other violations and threats to the validity of their results. I challenge defenders of the PACE study to cite other precedents for this kind of manipulation of clinical trials participants. What would they have thought if a drug company had done this for the evaluation of their medication?

:thumbsup:
 

Sasha

Fine, thank you
Messages
17,863
Location
UK
BTW, we must go to his blog and thank him and give htm the traffic. He says, "[To be continued later if there is interest in my doing so. If there is, I will discuss the disappearance of objective measures of functioning from the PACE study and you will find out why you should find some 3-D glasses if you are going to search for reports of these outcomes.]

http://blogs.plos.org/mindthebrain/...ace-chronic-fatigue-syndrome-follow-up-study/
 

Sidereal

Senior Member
Messages
4,856
If the investigators are taking their RCT design seriously, they should give precedence to the null findings for group differences at follow-up. They should not be emphasizing the sustaining of benefits within the GET and CBT groups.

The investigators increase their positive spin on the trial in the opening sentence of the Discussion


The main finding of this long-term follow-up study of the PACE trial participants is that the beneficial effects of the rehabilitative CBT and GET therapies on fatigue and physical functioning observed at the final 1 year outcome of the trial were maintained at long-term follow-up 2·5 years from randomisation.

This is incorrect. The main, a priori finding is that any reported advantages of CBT and GET at the end of the trial were lost by long-term follow up. Because an RCT is designed to focus on between group differences, the statement about sustaining of benefits is post-hoc.
 

Sidereal

Senior Member
Messages
4,856
Although it was a long time ago, I recall well my first meeting with Professor Simon Wessely. It was at a closed retreat sponsored by NIH to develop a consensus about the assessment of fatigue by self-report questionnaire. I listened to a lot of nonsense that was not well thought out. Then, I presented slides demonstrating a history of failed attempts to distinguish somatic complaints from mental health symptoms by self-report. Much later, this would become my “Stalking bears, finding bear scat in the woods” slide show.

But then Professor Wessely arrived at the meeting late, claiming to be grumbly because of jet lag and flight delays. Without slides and with devastating humor, he upstaged me in completing the demolition of any illusions that we could create more refined self-report measures of fatigue.

I wonder what he would say now.

But alas, people who suffer from CFS have to contend with a lot more than fatigue. Just ask them.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Perfect additional point of one-eyed interpretation by the authors. I thought peer reviewers were supposed to pick up stuff like this. It is fair enough - good practice even - to point out where analyses are post hoc, but needs to be done consistently
James Coye said:
The investigators further congratulate themselves with

There was some evidence from an exploratory analysis that improvement after the 1 year trial final outcome was not associated with receipt of additional treatment with CBT or GET, given according to need. However this finding must be interpreted with caution because it was a post-hoc subgroup analysis that does not allow the separation of patient and treatment factors that random allocation provides.


However, why is this analysis singled out has exploratory and to be interpreted with caution because it is a post-hoc subgroup analysis when similarly post-hoc subgroup analyses are recommended without such caution?
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
This is the key point. It is highly unusual and runs contrary to the practice of conduct and reporting of RCTs to focus on the within-group changes when you have between-group comparisons. This is Clinical Trials 101.
I think this demonstrates deliberately deceptive conduct. I wonder what a panel reviewing the paper would consider it to be? I guess it depends on who is on the panel.
 

Sidereal

Senior Member
Messages
4,856
I think this demonstrates deliberately deceptive conduct. I wonder what a panel reviewing the paper would consider it to be? I guess it depends on who is on the panel.

Let me see if I can explain this better than I attempted to do above. Simply looking at changes over time WITHIN a trial arm defeats the entire purpose of doing a randomised controlled trial. If you want to look at changes over time you can just naturalistically follow up a group of people getting CBT/GET and not bother with pesky randomisation into CBT/GET vs "control" groups. The whole point of doing an RCT is to show that changes over time in the CBT/GET arm are OVER AND ABOVE the changes over time seen in SMC and APT arms because, as everyone knows, changes in symptom scores that happen over time have many potential causes, not just the effect of a treatment so the difference in change BETWEEN the groups demonstrate the effect of treatment while the remainder is attributed to placebo, SMC treatment, passage of time, regression to the mean etc.
 

ScottTriGuy

Stop the harm. Start the research and treatment.
Messages
1,402
Location
Toronto, Canada
I posted on his blog:

Thank you for writing this response to the waste of research money and harm to patients by these unscrupulous people (can’t bring myself to call them researchers).