PACE Trial and PACE Trial Protocol

[oceanblue]

Clinically useful improvement.

... However the decision on whether these changes are clinically significant or clinically useful usually depends on the overall impact on health and this is often measured by patients' subjective responses on how they feel their overall perception of health and function has changed. Subjective experiences therefore help validate objective physiological measures.

In the case of PACE, validation is difficult if not impossible.

The primary outcome measures, SF 36 pf and CFQ, are both based on subjective responses of how people 'feel' while three of the interventions tested are specifically aimed at changing patients' subjective feelings about their symptoms and levels of physical function/fatigue; either positively in the case of CBT and GET (symptoms are benign and the condition reversible) or negatively in the case of APT (symptoms should be heeded and the condition is not reversible).

These results alone cannot be considered as clinically useful without external validation, regardless of the degree of improvement.

External validation by an objective measures could be taken as support for a clinically useful improvement however the only objective measure reported was the 6MWT where marginal improvements were found.

Even if these improvements, while small, were statistically significant, they still cannot be validated by the two primary outcome measures for the reasons stated above and therefore cannot be considered as clinically useful.

Ergo, none of the results reported can be proven to be clinically useful.

I understand your point about the dubious validity of the 'clinically useful difference', but I always think it helps to point out how a study has failed on its own terms, particularly where those terms have already been watered down from the protocol.

the interventions tested are specifically aimed at changing patients' subjective feelings about their symptoms and levels of physical function/fatigue

This suggests that GET has an equal effect on perceptions as CBT and I wonder if this is true.

CBT is based on the premise of treating a phobia. It's a pyschological therapy delivered by psychologists and psychotherapists aimed at changing patient cognitions.

GET is based on the premise of treating deconditioning. It uses an exercise programme devised by physiotherapists and occupational therapists.

Certainly GET tells participants things like 'hurt does not mean harm' and that could well have an effect on pain and general symptoms but it's harder to make the case that it would effect perceptions of fatigue and physical functioning - at least not to the same extent as CBT.

Interestingly, while the reported increases in fatigue/physical functioning for CBT are not matched by improvements in 6MWT, the only objective measure we have, there is at least some correlation between fatigue/sf36 improvements and 6MWT improvement.

I'm not saying that the GET programme has no effect on patients perceptions, just questioning whether it is likely to be as strong as for CBT.

 

[Marco]

I understand your point about the dubious validity of the 'clinically useful difference', but I always think it helps to point out how a study has failed on its own terms, particularly where those terms have already been watered down from the protocol.



This suggests that GET has an equal effect on perceptions as CBT and I wonder if this is true.

CBT is based on the premise of treating a phobia. It's a pyschological therapy delivered by psychologists and psychotherapists aimed at changing patient cognitions.

GET is based on the premise of treating deconditioning. It uses an exercise programme devised by physiotherapists and occupational therapists.

Certainly GET tells participants things like 'hurt does not mean harm' and that could well have an effect on pain and general symptoms but it's harder to make the case that it would effect perceptions of fatigue and physical functioning - at least not to the same extent as CBT.

Interestingly, while the reported increases in fatigue/physical functioning for CBT are not matched by improvements in 6MWT, the only objective measure we have, there is at least some correlation between fatigue/sf36 improvements and 6MWT improvement.

I'm not saying that the GET programme has no effect on patients perceptions, just questioning whether it is likely to be as strong as for CBT.

I'm pretty sure others will be pointing out how PACE failed on its own terms - at least I hope so.

As for whether CBT and GET equally change perceptions, who's to know? Both of them push the notion that the condition is reversable and that symptoms are not to be considered threatening. What we can also say is that CBT did not perform to the authors' expectations and not as well as GET on some measures. I suspect that this came as a major shock to them given the underlying 'theory' but I doubt that they will advocate handing over control to physiotherapists on the basis that GET outperformed CBT.

I can't recall where I saw this mentioned but someone suggested that any improvements seen in CBT and GET are due to boosting the feeling of 'self-efficacy' rather than anything specific to either therapy. On reflection it must have been a psychologist.

As for a clinically useful difference generally, the PACE setup shows that the whole idea of external validation of 'psychological' theories is rather circular and is why they really should have ensured that there were a number of objective measures to provide that validation. Which makes it all the more perplexing why they dropped the actimeter use and have not (as yet?) reported on hours worked/employment outcomes etc.

Or perhaps its not perplexing at all.

 

[anciendaze]

Earlier we turned up some comparison data for the 6 minute walk test showing clinically significant improvements in COPD and heart failure, where you might expect any modest improvement to be considered significant. PACE results did not do well by those standards. Has anyone looked for standards of clinically significant improvements in MS, another remitting-relapsing illness? Because this is acknowledged as an organic disease it might allow distinction between improvements in condition and purely subjective measures.

 

[Marco]

Earlier we turned up some comparison data for the 6 minute walk test showing clinically significant improvements in COPD and heart failure, where you might expect any modest improvement to be considered significant. PACE results did not do well by those standards. Has anyone looked for standards of clinically significant improvements in MS, another remitting-relapsing illness? Because this is acknowledged as an organic disease it might allow distinction between improvements in condition and purely subjective measures.

I don't recall coming across any references to clinically significant differences for MS but I do have a reference for the 6MWT for MS that showed that they started from a lower baseline but the percentage improvement was greater than for the PACE trial.

I also haven't searched extensively, but I'd be surprised if there is much discussion of clinically significant differences in the CBT literature!

 

[oceanblue]

PACE 6MWT for GET not 'clinically useful difference'

For fun (clearly I'm losing all sense of perspective on this thread) I applied the 'clinically useful difference' threshold PACE used on the primary outcomes to the 6MWT, as it's the sole objective measure.

Baseline SD for GET group=87
Clinically useful difference is 0.5 baseline SD = 43.5

Difference between GET and SMC means= +35.3; difference is not 'clinically useful'

On reflection, I think it is extremely reasonable to apply the 'clincially useful difference' test specifically for the 6MWT for GET - surely this is a measure that GET participants should excel at?

 

[urbantravels]

I've been trying to follow this colossal thread all along, but I'm sure I've missed some stuff, so apologies if this has already been discussed --

We've talked about the placebo effect, what about the nocebo effect in the "[evil version of] pacing" arm? If I recall correctly, there was discussion about how the "adaptive pacing" arm participants, as part of the protocol, were told their condition would not improve/could not improve, and that all they could do was to stay within the "envelope" (which was then defined in a very rigid way that most of us would find difficult to live with.) I do not know what, specifically, they were told, or if it's even true that they were told this.

I've seen the manuals for the CBT arm, but not the manuals for the other two arms. Are those posted somewhere for download?

 

[urbantravels]

Another question - about the gap between efficacy and effectiveness. I think there is a name for this gap (i.e. the "something" effect) but I can't call it to mind. Is there a standard or average value for this? Or even a guesstimate at what it might be for a given treatment - any given treatment? Would that gap alone be enough to wipe out the "moderate" benefits of CBT/GET that PACE allegedly showed, when the treatments are actually applied in non-clinical-trial contexts?

 

[Dolphin]

We've talked about the placebo effect, what about the nocebo effect in the "[evil version of] pacing" arm? If I recall correctly, there was discussion about how the "adaptive pacing" arm participants, as part of the protocol, were told their condition would not improve/could not improve, and that all they could do was to stay within the "envelope" (which was then defined in a very rigid way that most of us would find difficult to live with.) I do not know what, specifically, they were told, or if it's even true that they were told this.

Interesting idea to portray it as having a "nocebo" effect.

I've seen the manuals for the CBT arm, but not the manuals for the other two arms. Are those posted somewhere for download?

They're all here: http://www.pacetrial.org/trialinfo/ . Great you're still interested.

Spot quiz on this thread is tomorrow 9am.

Here is one description of APT - but there is plenty of other information as well:

APT
Theoretical Model
The assumptions of simple pacing are that it is beneficial to stabilise activity and to
balance rest with activity. An analogy of balancing the energy “account” to avoid
overdrawing it is often used. Symptoms are regarded as warning signs to be “listened to”.
It is assumed that the symptoms reflect a pathological disturbance, which is not reversed
by undertaking increases in activity. Rather the assumption is that ignoring fatigue and
other symptoms risks activity induced exacerbations of the illness and consequently
impeding natural recovery, whereas good pacing will maximize the chance of natural
recovery and hence improve function in the long term. Activity is therefore planned so as
to balance activity and rest. The aim is to do what can be done on the one hand, within the
limits set by the illness, but to limit activity related exacerbations of symptoms on the
other.

 

[Dolphin]

I'm pretty sure others will be pointing out how PACE failed on its own terms - at least I hope so.

As for whether CBT and GET equally change perceptions, who's to know? Both of them push the notion that the condition is reversable and that symptoms are not to be considered threatening. What we can also say is that CBT did not perform to the authors' expectations and not as well as GET on some measures. I suspect that this came as a major shock to them given the underlying 'theory' but I doubt that they will advocate handing over control to physiotherapists on the basis that GET outperformed CBT.

I can't recall where I saw this mentioned but someone suggested that any improvements seen in CBT and GET are due to boosting the feeling of 'self-efficacy' rather than anything specific to either therapy. On reflection it must have been a psychologist.

Here's information from one of the GET trials previously quoted as showing GET was evidence-based:

(from abstract)
At the end of treatment the exercise group rated themselves as significantly more improved and less fatigued than the control group. A decrease in symptom
focusing rather than an increase in fitness mediated the treatment effect. Graded
exercise appears to be an effective treatment for CFS and it operates in part by reducing the degree to which patients focus on their symptoms.
from
Moss-Morris R, Sharon C, Tobin R, Baldi JC. A randomized controlled graded exercise trial for chronic fatigue syndrome: outcomes and mechanisms of change. J Health Psychol. 2005 Mar;10(2):245-59.

Repeat of a CBT study one:

Full free text at: http://www.iacfsme.org/LinkClick.aspx?link=436&tabid=437

HOW VALID IS THE MODEL BEHIND COGNITIVE BEHAVIOR THERAPY FOR CHRONIC FATIGUE SYNDROME? AN EVALUATION OF THE ADDITIONAL DATA FROM THE TRIAL BY PRINS ET AL.

B. Stouten 1*, PhD
Ellen M. Goudsmit 2, PhD FBPsS

1. Einsteindreef 67A, Utrecht, The Netherlands

2. University of East London, UK



ABSTRACT

The cognitive behavior therapy (CBT) program studied by Prins et al. is based on a model of chronic fatigue syndrome that posits that fatigue and functional impairment are perpetuated by physical inactivity, somatic attributions, focusing on bodily symptoms and a low sense of control. A recent analysis of the data from three trials based on a model devised by Vercoulen et al. concluded that the effect of CBT on fatigue could not be attributed to a persistent increase in physical activity (They are referring to Wiborg et al. (2010)). We therefore examined the effect of treatment on the remaining three variables in the model using data from one of the trials, available in the public domain. The results from the groups given CBT, Guided Support and treatment as usual revealed that CBT had no significant impact on somatic attributions and focusing on bodily symptoms, and that in line with established guidelines, these two variables were not mediating factors. The only variable in the model showing an effect of CBT was sense of control. We submit that there is now sufficient evidence to warrant a review of CFS guidelines which advocate interventions aimed particularly at increasing physical activity and challenging somatic attributions, and that more flexible programs which address loss of control deserve further consideration and evaluation.

--------------------------------------------------------------------------------

Bulletin of the IACFS/ME. 2010;18(2):82-89. 2010 IACFS/ME

As for a clinically useful difference generally, the PACE setup shows that the whole idea of external validation of 'psychological' theories is rather circular and is why they really should have ensured that there were a number of objective measures to provide that validation. Which makes it all the more perplexing why they dropped the actimeter use and have not (as yet?) reported on hours worked/employment outcomes etc.

Or perhaps its not perplexing at all.

That's correct, they haven't reported employment outcomes. I hate when researchers have this excuse that further studies will follow as they can try to claim that missing data might be in that. It of course might never get published if it doesn't suit them - I imagine we won't see at least some of the secondary outcome measures.

 

[anciendaze]

Repeat of a CBT study one:

...The only variable in the model showing an effect of CBT was sense of control.

I begin to wonder if they were measuring patients or therapists.

 

[Dolphin]

PACE 6MWT for GET not 'clinically useful difference'

For fun (clearly I'm losing all sense of perspective on this thread) I applied the 'clinically useful difference' threshold PACE used on the primary outcomes to the 6MWT, as it's the sole objective measure.

Baseline SD for GET group=87
Clinically useful difference is 0.5 baseline SD = 43.5

Difference between GET and SMC means= +35.3; difference is not 'clinically useful'

On reflection, I think it is extremely reasonable to apply the 'clincially useful difference' test specifically for the 6MWT for GET - surely this is a measure that GET participants should excel at?

That's very interesting - good catch. I had missed that although the raw difference might be said to be 45 (= 67 [increase for GET] - 22 [increase for SMC]), that the adjusted figure is smaller than that.

 

[Dolphin]

Another question - about the gap between efficacy and effectiveness. I think there is a name for this gap (i.e. the "something" effect) but I can't call it to mind. Is there a standard or average value for this? Or even a guesstimate at what it might be for a given treatment - any given treatment? Would that gap alone be enough to wipe out the "moderate" benefits of CBT/GET that PACE allegedly showed, when the treatments are actually applied in non-clinical-trial contexts?

Don't know too much about it but I put
efficacy effectiveness gap
into Google and it came out with the "efficacy-effectiveness gap"! If anyone knows another name for it, feel free to post it.

 

[oceanblue]

That's very interesting - good catch. I had missed that although the raw difference might be said to be 45 (= 67 [increase for GET] - 22 [increase for SMC]), that the adjusted figure is smaller than that.

Ah, I hadn't spotted that but presumably adjusted figures are what they use for the other comparisons?
In any event, applying confidence intervals to the raw data means they still wouldn't achieve a CUD.

 

[Dolphin]

Ah, I hadn't spotted that but presumably adjusted figures are what they use for the other comparisons?
In any event, applying confidence intervals to the raw data means they still wouldn't achieve a CUD.

Yes, apologies for not being clear, I would think the adjusted difference is the one to use, as you did.

 

[Bob]

Here's a blog article on The Psychologist website (The British Psychological Society)...
It's, surprisingly, quite a balanced and informed article, especially compared to the recent newspaper articles...
There's a couple of errors or contradictions that I've noticed, esp re pacing/APT and re depression/anxiety...
There's a helpful facility for comments.

Fatigue evidence gathers PACE
http://www.thepsychologist.org.uk/blog/11/blogpost.cfm?threadid=1947&catid=48

 

[anciendaze]

Since I haven't been able to tempt anyone into making a fool of themselves, I guess I will have to risk exposing my cognitively-impaired dyscalculia to ridicule.

My arguments about outliers were meant to suggest a major problem with the mathematical models in use. One problem is that those beyond about 3 SD below the mean of the assumed distribution would have negative scores for physical activity. Tabloid headlines like "Study Proves UK has 80,600 Zombies!" could result. (I got this by a quick look at a table for the complementary cumulative distribution function, plus a guess at the current UK population. My remembered values for both gave results within an order of magnitude. Detailed calculations on meaningless data are a waste of time.)

On the more rational side of responses, we see that the model must break down before reaching 3 SD below the mean. Since the study groups were between 1 SD and a little over 2 SD, the question of exactly when, where and how the model breaks down is germane to questions about interpretation of published results.

 

[oceanblue]

re

Repeat of a CBT study one:
...The only variable in the model showing an effect of CBT was sense of control.

I begin to wonder if they were measuring patients or therapists.

brilliant!

 

[Marco]

Since I haven't been able to tempt anyone into making a fool of themselves, I guess I will have to risk exposing my cognitively-impaired dyscalculia to ridicule.

My arguments about outliers were meant to suggest a major problem with the mathematical models in use. One problem is that those beyond about 3 SD below the mean of the assumed distribution would have negative scores for physical activity. Tabloid headlines like "Study Proves UK has 80,600 Zombies!" could result. (I got this by a quick look at a table for the complementary cumulative distribution function, plus a guess at the current UK population. My remembered values for both gave results within an order of magnitude. Detailed calculations on meaningless data are a waste of time.)

On the more rational side of responses, we see that the model must break down before reaching 3 SD below the mean. Since the study groups were between 1 SD and a little over 2 SD, the question of exactly when, where and how the model breaks down is germane to questions about interpretation of published results.

While I agree that the reported statistics are nonsense - because neither the population nor patient samples on these measures are any way approximating a normal distribution, it is what was published and it is possible to use their reported figures and statistical analysis to highlight just how ridiculous, for example, equating an SF36 pf score of 60 to 'normal' really is.

I'm happy to abandon statistical purity if used for this purpose.

After all, the authors can't come back and complain that 'you can't say that' because 1 SD or whatever is meaningless in this case!

 

[oceanblue]

Another way of looking at the PACE results

I went back to that graph of the distribution of SF36 scores for the general uk adult population (30% of whom are 65 or over) - and where the PACE results fit on it - and tried to calculate/estimate some numbers.

For example, the baseline Sf-36 scores of 38 corresponds to about the bottom 10% of the UK population

The control SMC group at 52 weeks scored around 50, corresponding to the bottom 13% of the population.

The CBT/GET groups at 52 weeks scored about 58, corresponding to the bottome 15% of the population.

So the net effect of CBT or GET, after 1 year, was to move particpants from around the bottom 13% of SF-36 scores to around the bottom 15%.

Let's party.

22% of the UK population used in this study reported a long-term illness, though the authors of the relevant study (Bowling) say the face-to-face interview method used probably leads to under-reporting of ill-health.

This proably isn't an entirely fair way of presenting things, but it's at least as fair as the 'within the normal range' stunt pulled by the PACE authors.

This underlying graph is taken from page 9 of the open access article (notations added by me): Bowling SF-36 normative data. Although the article and picture are freely available there may be copyright issues so please don't reproduce this pic

 

[biophile]

Thanks for the compliments! Still been working on some more graphs.

Esther12's idea of a wiki collaboration and oceanblue's dream of a comprehensive rebuttal piece (and a user-friendly version) are noteworthy. There is a wealth of information contained within this thread, and this forum in general.

oceanblue wrote: When Bleijenberg & Knoop said PACE had used a 'strict definition of recovery' it was because they incorrectly thought that PACE has used a healthy population to define 'within the norm'. Which is pretty unimpressive in an editorial.

Indeed. Bad peer review? The editorial even used the word "recovery", not "normal" as PACE does. A misunderstanding or typo on their behalf? Or a lack of fact checking after a naive assumption that the PACE authors would stick to the protocol? Wishful thinking? Public relations or spin from comrades?

Dolphin wrote: Just to correct this typo/similar: they used "working age" not people actually in employment. ETA: I see you make this point later on.

Re-reading what I originally wrote, I made a slight modification, thanks. I needed to correct an error in the first graph's label as well.

Dolphin wrote: Note on the point of using 80 as the threshold: if one have a recovered group that are like normal, it means it is possible the could be derived from the same distribution. If most of the scores were 80/85 and a few 90+, such a group (which might have a mean in the low 80s or even a bit higher) could still be different from a healthy population with mean of 92/93. So shouldn't be described as a normal group.

Good point. The "threshold" of normal is not the same as average normal.

ancientdaze wrote:

If we can't directly demonstrate the absurdity of some assumptions going into this study, maybe we can produce absurdities coming out. For the moment, forget the lower bounds. If you take the numbers in the study as meaningful, what do they tell you about the healthy population, including allegedly fully-recovered ME/CFS patients? My first thought is that those experiencing recovery to full health become outliers rare enough to ignore.

[...]

Another inference from that assumed normal distribution is that a few ME/CFS patients should be running marathons, while those at the other extreme exhibit negative physical activity. Considering the estimated total of sufferers, there shouldn't be any problem finding a few of each in the UK.

I agree. 41% of the GET group reported feeling "much better or very much better" (vs 25% for SMC). If even half of these were "recovered" (PACE's original definition in the protocol) and/or a healthy normal distance in the 6MWT ie 600m, the authors would be proudly displaying these figures.

Snow Leopard wrote: Yes, I agree and that is why I believe a 6 minute walking test is hardly objective if it does not consider the overall impact on symptoms and activity levels over the next few days.

Exactly. The GET group may merely be pushing themselves more and paying a heftier one off price for a one off test that was absorbed into the statistics on "non-serious adverse events", "serious adverse events", "serious adverse reactions" etc.