• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

A critical commentary and preliminary re-analysis of the PACE trial

Messages
15,786
Can patients with chronic fatigue syndrome really recover after graded exercise or cognitive behavioural therapy? A critical commentary and preliminary re-analysis of the PACE trial
Carolyn Wilshire, Tom Kindlon, Alem Matthees & Simon McGrath

ABSTRACT
BACKGROUND: Publications from the PACE trial reported that 22% of chronic fatigue syndrome patients recovered following graded exercise therapy (GET), and 22% following a specialised form of CBT. Only 7% recovered in a control, no-therapy group. These figures were based on a definition of recovery that differed markedly from that specified in the trial protocol.

PURPOSE: To evaluate whether these recovery claims are justified by the evidence.

METHODS: Drawing on relevant normative data and other research, we critically examine the researchers’ definition of recovery, and whether the late changes they made to this definition were justified. Finally, we calculate recovery rates based on the original protocol-specified definition.

RESULTS: None of the changes made to PACE recovery criteria were adequately justified. Further, the final definition was so lax that on some criteria, it was possible to score below the level required for trial entry, yet still be counted as ‘recovered’. When recovery was defined according to the original protocol, recovery rates in the GET and CBT groups were low and not significantly higher than in the control group (4%, 7% and 3%, respectively).

CONCLUSIONS: The claim that patients can recover as a result of CBT and GET is not justified by the data, and is highly misleading to clinicians and patients considering these treatments.

http://www.tandfonline.com/doi/abs/10.1080/21641846.2017.1259724?journalCode=rftg20&
 
Messages
15,786
Full text at http://www.tandfonline.com.sci-hub.cc/doi/abs/10.1080/21641846.2017.1259724

I haven't read it closely yet, but it seems to be laying out the major flaws with the PACE "Recovery" paper (2013). So there's a lot of material which is familiar to us. I don't think it talks about the initial 2011 paper much, which is the one that covers improvement instead of recovery.

But it's very good to see a discussion of the PACE flaws published, since doctors, therapists, and politicians who can't understand or evaluate research papers themselves will need to hear it from a journal. Hopefully this will be useful to show to doctors, and be considered by evidence review panels, such as NICE. And it makes a nice rebuttal to BPS quacks raving about how great PACE is :rolleyes:

There's a couple graphs that illustrate the questionnaire threshold problems very well:

SF36.jpg


CFQ.jpg
 
Last edited:

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Credit where credit is due

Dr Carolyn Wilshire was the lead author. Co-author Alem Matthees had an exhausting FOI battle to get the recovery data released. @Tom Kindlon and I were co-authors too. And here's our acknowledgement at the end of the paper:
This paper represents a collaborative effort between psychology researchers and patients [Tom & Alem are patient-researchers]. In addition to those named as authors, several others also contributed important ideas, suggestions and information. This paper could not have been written without their help. We also wish to thank Keith Laws for his helpful comments, and David Tuller and Sam Carter for drawing our attention to some of the issues noted in this paper.
Team effort.
 
Last edited:

Cheshire

Senior Member
Messages
1,129
Clear debunking of the lowering of threshold:

In the original study protocol, recovery on this measure was defined as a score of 85 or above [12], a similar figure to that used in previous studies of behavioural interventions for CFS (e.g. [15,16]). This was a reasonable, albeit arguably low threshold: according to reference data from a large British community sample, the vast majority (90%) of people aged 18–59 without a long-term illness or disability actually score 90 or higher ([17]: for summary report, see Bowling et al. [18]). However, in the Recovery paper, the minimum score for a recovery outcome was lowered to 60. This late post-protocol change increased the proportion of participants who met this criterion from 14% to 45%, a more than three-fold increase. The justification given for this change was that the original threshold of 85 was so high that ‘approximately half the general working age population would not meet it’ [3, p. 2229]. This claim is clearly incorrect: it seems to have been based on summary statistics from a large British reference sample reported in Bowling et al. [18], in which almost a third of participants were aged 60 or over, and one-fifth reported a long-term illness or disability that limited their daily activities or the work they could do. In addition, the authors seem to have derived their ‘approximately half’
figure from calculations based on the sample mean and standard deviation, a method which was inappropriate, given the highly skewed distribution of scores (see Figure 1 for illustration).

Indeed, as Figure 1 shows, if we look just at those participants from the British reference
sample who were aged 18–59 and did not have a long-term illness or disability, the median (and modal) score for this highly skewed normative sample was 100 and 93% scored at or above the original recovery threshold of 85 [17]. Their arguments do not therefore justify the lowering of the SF-36 physical function threshold score from the originally specified minimum level.

Thank you so much @Simon and @Tom Kindlon
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Some bonus graphs, not included in the original paper (uses new analysis from FOI data by Alem, but data isnt' in the paper either). They provide a bit more depth, though:

Note how adding in the CFS caseness and health change (CGI) criteria conveniently made only made only a modest difference to the final recovery rate in the published version (2013 PACE paper), but more than halved the recovery rate according to protocol (=planned).

Don't forget that 'planned' is as laid out in the 2007 protocol - "published" is as published in the 2013 PACE recovery paper at Psychological Medicine, where the analysis took place after the trial had completed. The authors have since acknowledged that their analysis was "exploratory" and not predefined.

Cumulative effect of applying the four PACE recovery criteria, for GET

gGUievD9EP4EBP94CiqvHP8i7_uD3L1RFoH1sZNs-rj_ulwqqOrG6k87Ki4cFlJ2k0oBoqUYjW5ug0OS_q7zhOTixfQmepRmZNBR8M2e9B7XViZU2oPxqIXxq7hkKBb9EwufgEhVE7w


Note that PACE applied the primary outcomes of self-rated physical function and fatigue as a joint category, I split them for more detail. A perhaps cleaner, but less pretty version of the graph is this, below - but same info shown

8hHKR-eFRgzZDvOVi0jd6QFuNM_Uk3G1X7QgwuMmoigQrn-50p6fkmkcHDnAkGbEcJmanSv3ePiKikXYVOwPUc9Apj79ATyn64Wz73ONXiptdKTHHWyf6S6xrdAdU9yAbijQwGPlZjM
 
Last edited:

A.B.

Senior Member
Messages
3,780
It's important to keep in mind that this re-analysis only corrects one layer of bias. Underneath this there is another layer of bias which is related to the lack of blinding, an inadequate control group and reliance on subjective measures. One cannot correct this flaw but only point out that even the meager results we're seeing are exaggerated.

This is an important aspect because these things by themselves are enough to produce highly misleading results.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
It's important to keep in mind that this re-analysis only corrects one layer of bias. Underneath this there is another layer of bias which is related to the lack of blinding, an inadequate control group and reliance on subjective measures. One cannot correct this flaw but only point out that even the meager results we're seeing are exaggerated.
Yes, and our paper discusses the problem of relying on self-reports in an unblinded trial. We point out that in these cricumstances the authors should have paid more attention to objctive measures of function, such as walking distance, physical fitness and sickness benefit (were there were no gains, apart from a small one in GET walking).