PACE Trial and PACE Trial Protocol

[oceanblue]

PACE used wrong data to define 'normal' fatigue

We've already shown how the PACE definition of 'normal' for SF36 function is wrong, but it's been harder to pin down the problems with the Fatigue defintion. I'm pretty sure there I've now found data showing PACE used the wrong fatigue data to set its norms, because their 'normative' sample was very sick. Since I failed to excite anyone with my previous post,

Chalder Fatigue Scale normative data is suspect

I'm having one last go at explaining this:

• Only 14% of the original sample exceeded the PACE threshold of 18 for normal fatigue
• but 38% of that same sample met the original Chalder defintion for fatigued (that's the same definition used in the protocol)
• so 24% (38%-14%) of the sample are defined as "normal" by PACE yet meet it's protocol definition of fatigued. That's even worse than the situation for SF36. (It also means the sample was very fatigued, emphasised by another paper quoting just 10% as they typical level of fatigue in GP attenders - this was not a suitable normative sample).

One point of explanation. The Cella 2010 paper that was used for PACE's definition of 18 as 'normal' used a subset of a sample of 15,000 patients published in 1994. However, we know that the mean for Cella subsample was 14.2, higher than the mean of the original sample of 13.8 (a statistically significant difference) so we can conclude that the Cella sample is at least as fatigued as the original sample, hence the figures above should apply to the Cella data too. My original post has the details.

OK, I've done my best, if no one's interested I promise I won't mention it again.

 

[Graham]

I'm interested, oceanblue, honest! We knew about the inconsistencies in "normal", which is why we focused on the fatigue scale rather than the sf36 scale in our animated graphics. The fatigue scale is so utterly wrong in so many ways. But I didn't know about the Cella paper, so I hooked back on your original posting, and thanks for that: now I understand much more about it.

To be honest, the whole fatigue thing was rubbish. But it is more effective and easy to use their results to prove how feeble the improvement was (or non-existent rather), than to try to convince them that they had done such a bad job. It would have been much harder if they had actually come up with more impressive figures.

 

[Esther12]

I'm interested too... I'm just struggling to keep up with all the good points you keep making. Thanks for highlighting what you think is particularly important... I'll try to make the time to really go over it when I'm feeling better.

 

[Sean]

I am following this as closely as I can for now, OB. Don't ever think these kind of posts are ignored or unappreciated. At the very least they provide useful background info and ideas.

 

[anciendaze]

You may well be right, oceanblue. My take on the whole business is that, after careful study: 1) we can't figure out what they mean by CFS; 2) we believe they used a very strange definition of normal; 3) we don't believe they were serious about getting objective data; 4) we don't believe the reported improvements in objective scores were clinically significant; 5) we can't tell if reported improvements were statistically significant because of the huge difference between the number intended to treat and the number actually used in reported measures; 6) we're not even sure the measures of significance were appropriate because the distributions appear to violate prerequisites for determination of necessary parameters. For a scientific paper these lapses are damning.

 

[oceanblue]

Thank you all for your kind replies . I realise my info would have been more useful in the 2 week Lancet-letter window but I only recently managed to get my head around the nicities of the fatigue studies.

 

[oceanblue]

Response bias? The graph, with bells & Whistles

New version of the graph showing SF36 scores for PACE at 52 weeks if they had stayed in line with the original WSAS data (see 'WSAS' projected line).



Second version adds error bars nb these are standard errors, about half this size of error bars for 95% confidence limits. I'm not entirely sure how to intepret these as we are looking at the trend across a series of data, not comparing two different points. As its a scatter plot there are 2 sets of error bars per data point: sf36 error and 6mwt error.



don't know why these images are so badly mangled when I upload them; the much clearer originals are availabe on request

And I won't be the least bit offended if no one replies to this post...

 

[Bob]

Unrecorded drop outs from the PACE Trial...

Here's some text from Action for ME's recent edition of Interaction Magazine...

I can't quite get my head around the wording of the text taken from the PACE Trial paper, so I can't confirm if the interpretation is correct.

Had any of us picked up on this previously? I'd completely missed this bit of the paper.

It's written by Prof Derek Pheby.


--------------------------------------------------------------------------------


Interaction magazine - Page 5 - top right hand corner...

The analysis was ostensibly on the basis of treatment intention. This is the correct way to analyse the results of a randomised controlled trial, because if the analysis only involves those subjects who complete the trial, it disregards any participants who drop out because of adverse effects, and may therefore give an unjustifiably favourable impression of the treatment under examination.

The authors state, though: "We excluded participants from the intention-to-treat population for whom we had no primary outcome data in the final analysis."

In other words, this was not an intention-to-treat analysis at all, since only data regarding participants who stayed the course long enough for outcomes to be assessed were included in it.


Here's the relevant quote from the PACE Trial paper:

We excluded participants from the intention-to-treat
population for whom we had no primary outcome data
in the final analysis, which used restricted maximum
likelihood. The per-protocol analysis excluded
participants who were ineligible after randomisation,
treated at a second centre, or did not received adequate
treatment, adjusting for actual stratification factors.
Statistical analyses were done with Stata version 10,
SAS version 9.1, and SPSS version 18.

 

[oceanblue]

Unrecorded drop outs from the PACE Trial...

Had any of us picked up on this previously? I'd completely missed this bit of the paper.

Here's the relevant quote from the PACE Trial paper:

We excluded participants from the intention-to-treat
population for whom we had no primary outcome data
in the final analysis, which used restricted maximum
likelihood. The per-protocol analysis excluded
participants who were ineligible after randomisation,
treated at a second centre, or did not received adequate
treatment, adjusting for actual stratification factors.
Statistical analyses were done with Stata version 10,
SAS version 9.1, and SPSS version 18.

Thought this had been mentioned briefly, but Pheby is right. If they were going to do an intention to treat analysis they should at least have used a 'last observation carried forward' approach, though even that would be too conservative if people had dropped out due to relapse. That said, using last-observation-carried-forward would probably only affect the results (including the difference with SMC) by 5% and I'm not sure that's so important.

I'm not sure why they couldn't get primary data on 5%: it's only 2 short questionnaires to complete by post or phone, and a carer could presumably do this if someone was in a really bad way.

Disappointing that Derek Pheby didn't mention the PACE self-report gains were not matched by gains in objective measures.

 

[Bob]

Thought this had been mentioned briefly, but Pheby is right. If they were going to do an intention to treat analysis they should at least have used a 'last observation carried forward' approach, though even that would be too conservative if people had dropped out due to relapse. That said, using last-observation-carried-forward would probably only affect the results (including the difference with SMC) by 5% and I'm not sure that's so important.

I'm not sure why they couldn't get primary data on 5%: it's only 2 short questionnaires to complete by post or phone, and a carer could presumably do this if someone was in a really bad way.

Disappointing that Derek Pheby didn't mention the PACE self-report gains were not matched by gains in objective measures.

Ah, right, I was getting confused again! Thanks for that ocean.

I thought that Pheby was referring to participants who were not recorded in any of the data in the paper.

Yes, the paper did record the number of drop outs didn't it...
When I looked at those figures, I seem to remember agreeing with you, that they wouldn't have made much difference to the outcomes.

 

[Dolphin]

Some data on 6 min walk vs SF-36 PF that could be plotted

It would be interesting to get/watch out for baseline data from other studies and see does the same equation hold. Although I don't think the 6 minute walking test has been used in many ME/CFS trials. Perhaps it was in the Jason et al. (2007) study.

I just checked, Jason et al. (2007) does have 6 minute walk test data along with SF-36 PF data. It would be interesting to plot the points on a graph (and ideally make a regression line/make an approximate regression line visually) and see how it compares to the baseline in the PACE trial.

6 min walk Mean (SD) (presumably in feet)

Baseline 12 Month

CBT
1346.35 (296.76) 1542.60 (634.11)

COG (incl. encouragement to pace)
1389.50 (385.51) 1513.50 (270.95)

ACT (exercise program)
1335.27 (280.99) 1378.40 (208.92)

RELAX (relaxation and stretching, yoga, etc.)
1317.78 (296.55) 1429.33 (286.19)

b Higher scores indicate better outcome

Physical functioningb Mean (SD)

Baseline 12 Month

CBT
46.36 (27.44) 58.64 (30.44)

COG (incl. encouragement to pace)
45.65 (23.71) 61.09 (23.74)

ACT (exercise program)
39.17 (15.65) 39.72 (27.63)

RELAX (relaxation and stretching, yoga, etc.)
53.77 (26.66) 61.20 (27.70)

b Higher scores indicate better outcome

from:

Jason, LA., Torres-Harding, S., Friedberg, F., Corradi, K., Njoku, MG., Donalek, J., Reynolds, N., Brown, M. Weitner, BB., Rademaker, A and Papernik, M. Non-pharmacologic interventions for CFS: a randomized trial. Journal of Clinical Psychology in Medical Settings, 2007, 14, 4, 275-296.

 

[Sean]

Nice consistency between the objective and subjective results. Exercise clearly didn't work. CBT did no better overall than pacing. And the SD for CBT outcome on the 6mwt, is more than thrice the therapeutic gain.

Not much joy there for PACE lovers.

 

[Dolphin]

"Health in mind and body: bridging the gap"
http://www.foundation.org.uk/events/audios/audiopdf.htm?e=440&s=1200

20 minute audio and slides, PACE bit starts about 10 mins in.

Thanks oceanblue.

Regarding the slide from this paper:

BMJ. 1994 March 19; 308(6931): 763766. PMCID: PMC2539651

Copyright notice
Population based study of fatigue and psychological distress.
T. Pawlikowska, T. Chalder, S. R. Hirsch, P. Wallace, D. J. Wright, and S. C. Wessely

Free full text: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2539651/?tool=pubmed

some of us dissected this graph in this thread (started a few posts in): http://forums.phoenixrising.me/show...a-should-not-be-exclusion-criteria-for-ME-CFS

Basically, the GHQ-12 measures various symptoms incl. some physical symptoms so describing it as a measure of psychological distress is somewhat misleading. The same problem of confounding is probably related to a lot of the points made.

ETA: Oceanblue has corrected me that I was thinking of a similar but different study. His point seems stronger: http://forums.phoenixrising.me/show...s-from-the-90s-that-keep-cropping-up&p=179419

The Diabetes type 1 CBT/motivational interventional is not really analagous to ME/CFS: with diabetes, it is very clear what are the sort of behaviours to encourage; it is much less clear cut in ME/CFS.

 

[Dolphin]

3. Only participants who visited their GP were included
To complicate things, Cella didn't use all the data from the original mailing. Instead, data was only used from respondents who subsequently visited their GP about a viral or other complaint and were selected as part of another study. So anyone who was very healthy and never visited their GP would not be included. Those who visited their GP more often would consequently have more chances to make this cohort than those who rarely visited their GP. All of this is likely to bias the sample against healthy individuals.

I hadn't thought of that. Good point.

4.Data from the original study indicate this is an unhealthy cohort.
According to Pawlikowska, 38% of patients had a score about the original Chalder bimodal cut off of 3 (as used in the PACE protocol) and 18.3% of patients were substantially fatigues for 6 months or longer. Whoa, that looks unhealthy, esp as the paper quotes a 1990 paper that found only 10% of GP practice patients had fatigue for one month or more. I think there are some US studies indicating fatigue of over 6 months in the population is much less than 18%.

Well spotted.

 

[Dolphin]

One point of explanation. The Cella 2010 paper that was used for PACE's definition of 18 as 'normal' used a subset of a sample of 15,000 patients published in 1994. However, we know that the mean for Cella subsample was 14.2, higher than the mean of the original sample of 13.8 (a statistically significant difference) so we can conclude that the Cella sample is at least as fatigued as the original sample, hence the figures above should apply to the Cella data too. My original post has the details.

Where did you get the figure of 13.8 from?[/QUOTE]Oceanblue explained this (wasn't in the paper - he did some calculations) and I'm convinced the figure for Cella is higher than the Pawlikowska figure.

 

[Snow Leopard]

"Health in mind and body: bridging the gap"
http://www.foundation.org.uk/events/audios/audiopdf.htm?e=440&s=1200

20 minute audio and slides, PACE bit starts about 10 mins in.

I didn't bother with the audio, but I was concerned with the following:

50% of all new hospital outpatients have physical symptoms
unaccounted for by physical disease.

This hardly provides confidence in the medical model. It is also a non sequitur to then blame it on mental health. The biomolecular pathways between cognition, stress, behaviour etc and physical symptoms have not been explained at all. Basically all we have is two competing opinions (mental vs physical). The fact is that it is not scientific to decide between them until the underlying science is known.

 

[Dolphin]

ETA: should mention that the Cella CFQ data is not normally distributed and therefore, like the SF36 data, is not suitable for use with parametic stats, such as the 'within 1 SD of the mean' formula used by PACE:

Similarly to the CFS group, the community group scores were not distributed normally but were positively skewed. Values of skewness for the nonclinical group ranged between 0.40 (Item 5) and 1.06 (Item 9) with a mean skewness of 0.77.

That's only within each item. The total score is the relevant statistic in the PACE Trial.

In Pawlikowska et al. (1994) it says:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2539651/pdf/bmj00432-0041.pdf

In this community survey symptoms of fatigue fitted a normal distribution

 

[oceanblue]

@oceanblue: ETA: should mention that the Cella CFQ data is not normally distributed

Similarly to the CFS group, the community group scores were not distributed normally but were positively skewed. Values of skewness for the nonclinical group ranged between 0.40 (Item 5) and 1.06 (Item 9) with a mean skewness of 0.77.

That's only within each item. The total score is the relevant statistic in the PACE Trial.

In Pawlikowska et al. (1994) it says: "In this community survey symptoms of fatigue fitted a normal distribution "
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2539651/pdf/bmj00432-0041.pdf

1. They quoted figures for item 5 and 9 but I'm pretty sure they are referring to the whole data as not normally distributed (eg mean skewness figure).
2. It's the Cella data that's most relevant to the PACE trial, and that doesn't seem to be normally distributed. And did you notice that the Pawlikowska claim of normal distribution were not backed up by any data eg skewness or Shapiro-Wilk test? The graph of the data certainly looks non-normal, with a positive skew and median and mode higher than the mean. I'd put money on that not being a normal distribution.

 

[oceanblue]

Thanks oceanblue.

Regarding the slide from this paper:
some of us dissected this graph in this thread (started a few posts in): http://forums.phoenixrising.me/show...a-should-not-be-exclusion-criteria-for-ME-CFS

Basically, the GHQ-12 measures various symptoms incl. some physical symptoms so describing it as a measure of psychological distress is somewhat misleading. The same problem of confounding is probably related to a lot of the points made.

The Diabetes type 1 CBT/motivational interventional is not really analagous to ME/CFS: with diabetes, it is very clear what are the sort of behaviours to encourage; it is much less clear cut in ME/CFS.

I've started a new thread on the Wessely papers from the 90s, and have replied there re the graph:

This relates to the Pawlikowska paper and had been discussed on the PACE thread...

Could you explain a bit more about diabetes and behaviour?

 

[Dolphin]

Originally Posted by Dolphin
@oceanblue: ETA: should mention that the Cella CFQ data is not normally distributed
Similarly to the CFS group, the community group scores were not distributed normally but were positively skewed. Values of skewness for the nonclinical group ranged between 0.40 (Item 5) and 1.06 (Item 9) with a mean skewness of 0.77.
That's only within each item. The total score is the relevant statistic in the PACE Trial.

In Pawlikowska et al. (1994) it says: "In this community survey symptoms of fatigue fitted a normal distribution "
http://www.ncbi.nlm.nih.gov/pmc/arti...00432-0041.pdf

1. They quoted figures for item 5 and 9 but I'm pretty sure they are referring to the whole data as not normally distributed (eg mean skewness figure).
2. It's the Cella data that's most relevant to the PACE trial, and that doesn't seem to be normally distributed. And did you notice that the Pawlikowska claim of normal distribution were not backed up by any data eg skewness or Shapiro-Wilk test? The graph of the data certainly looks non-normal, with a positive skew and median and mode higher than the mean. I'd put money on that not being a normal distribution.

This isn't important I think but anyway:
1. I'm pretty sure they are referring to the individual items:

Item distribution was inspected with distribution plots and skewness values; normality assumption was tested with ShapiroWilk test.

2. As I think you know, they are mainly assuming normality here to work out percentiles (as far as I can see), so one is more interested in what percentiles are chosen by the figures.

I think a problem looking at the graph is 11 & 12 are combined. 11 is for "fatigue the same as usual" (11 * 1). I would think it would have a lot higher frequency. Indeed if one looks at Table 3, in Cella & Chalder, one can see that item 11 represents 22.9% of all the scores (.866 -.637) for people who don't have CFS, while only 7.8% scored 12.

We can also that 16.7% of the non-CFS cases scored 19 or more (that may have been pointed out before).

Note: one thing the central limit theorem tells one is that if one adds together sufficient numbers of skewed distributions (that are the same), one gets a normal distribution. I think this also extends under certain conditions to skewed distributions that are not the same distribution (which is what causes so many things in nature to be normally distributed as far as I know). What sort of graph one would get adding together 11 graphs of equal weight with mean skewness of 0.77, I'm not sure.
ETA: Normally the central limit theorem is expressed in terms of the distribution of the mean of the items. However, the distribution of the mean will have exactly the same distribution as the total score - it's just a scalar multiple of it.

I don't know too much about skewness. Given the range of scores is only 0,1,2,3, perhaps one can get a good idea of the relative frequencies of the four scores using the mean skewness figure of 0.77?