PACE Trial and PACE Trial Protocol

[Bob]

This is something I worked on some time ago, but never got around to posting it.
I might have made a mistake or two, as I'm going from memory, and I'm a novice statistician.
I'm hoping that Graham, or any other statisticians, will pick up any errors.


Usually, a 'normal range' (more appropriately known as a 'reference range') is calculated by using +/- 2 SD from the mean of a well defined population, as per this helpful Wikipedia page:
http://en.wikipedia.org/wiki/Reference_range

A 'normal range' analysis is intended to get rid of the top and bottom outliers of a well defined population. It cuts off the top and bottom 2.5% of values (5% in total), leaving the middle 95% as an indication of 'normal' values for a well defined population. For example, it could be used to describe the 'normal' walking speeds for healthy women of the same height.

It isn't appropriate to use this calculation (i.e. to use 'standard deviations') with SF-36 PF normative data because the data does not have a 'normal distribution' (i.e. it has a skewed distribution.) Using standard deviations for data that isn't normally distributed just doesn't give predictable or useful results, leading to a meaningless analysis. This is well known to all statisticians. (Except, apparently those working at the Lancet.)

The PACE Trial employed what seems to be a very unusual methodology, and decided to calculate +/- 1 SD of the general population (instead of +/- 2 SD of a well defined group), to give the 'normal range'.
So, in theory, this would cut off the top and bottom 16% (32% in total) for the general population, so it would include 68% of the population, if the data values were normally distributed.

I'm not sure what this methodology is intended to demonstrate, or what evidence it is based on, as I can't find this methodology described in any scientific literature.

It isn't appropriate to use this calculation for SF-36 PF scores, because the data isn't normally distributed. So, before we even do the calculation, we know it's going to give us meaningless results, as it proved to do.


If we assume that the 'normal range' in the PACE Trial was intended to cut off the bottom 16% of values of the general population (as +/- 1 SD is always assumed to do), then we need to know the 16th percentile, to know the statistically appropriate threshold for their inappropriate normal range analysis.

But in any case, their analysis is meaningless for all the other reasons we know about. I'm just exploring the issue.

A common methodology might cut off the bottom 2.5% of values of the healthy population, to define the 'normal range'. But there are even problems here, because the 'healthy' population is not a very well defined group.

 

[Graham]

I know I'm one of the resident mathematicians here, but I really don't think this is an area for statistics. When you take a driving test, you either meet the standard, or you don't. There is no calculation to work out where you lie relative to the general population. The same happens (or should happen) with exams. Is "normal level" a suitable target, or is it "healthy"? I go for the latter, especially when there appears to be an obesity epidemic in the developed world, severely affecting the performance of "healthy" adults. Would I accept that I was cured of ME if my activity levels became those of someone who was very obese?

One thing is beginning to bother me though. We are spending a lot of energy trying to be precise about our evaluations, and it is all very complex. But the gross violation of setting 60 as normality doesn't need anything sophisticated. The ball is in their court, not ours. They have to prove their case: we do not have to prove them wrong, we only have to find fault with their proof. So the simpler we make that message the better.

Sorry, am I being a wet blanket here? You know me well enough to understand that I'm not being critical.

 

[Bob]

I know I'm one of the resident mathematicians here, but I really don't think this is an area for statistics. When you take a driving test, you either meet the standard, or you don't. There is no calculation to work out where you lie relative to the general population. The same happens (or should happen) with exams. Is "normal level" a suitable target, or is it "healthy"? I go for the latter, especially when there appears to be an obesity epidemic in the developed world, severely affecting the performance of "healthy" adults. Would I accept that I was cured of ME if my activity levels became those of someone who was very obese?

One thing is beginning to bother me though. We are spending a lot of energy trying to be precise about our evaluations, and it is all very complex. But the gross violation of setting 60 as normality doesn't need anything sophisticated. The ball is in their court, not ours. They have to prove their case: we do not have to prove them wrong, we only have to find fault with their proof. So the simpler we make that message the better.

Sorry, am I being a wet blanket here? You know me well enough to understand that I'm not being critical.

Yes, I agree with much of that. My post was just intended for us PACE geeks who use this particular thread.

But I don't agree that the ball is currently in their court, and that they have to prove their case.
We aimed the ball at their court, but they blocked it, so it's still in ours.

In other words, they don't feel that they have to prove anything, because they've been published in the Lancet, and their propaganda was widely reported in the media. Whereas, we still haven't managed to get people to sit up and take notice of, or to acknowledge, any of the short-comings. I don't think our patient charities even understand the results of the PACE Trial. And most people, including patients, still quote the "30%" figure. (Although I think I have noticed a change recently, as I've seen a few patients discussing the results more accurately over the past few days, in newspaper discussions etc.)

So I think we still have a lot to prove.

 

[Bob]

This is something that I've only just noticed...

This passage is taken from the full protocol:

11.1 Assumptions
At one year we assume that 60% will improve with CBT, 50% with GET, 25% with APT
and 10% with SSMC. The existing evidence suggests that at one year follow up, 50 to
63% of participants with CFS/ME had a positive outcome, by intention to treat, in the
three RCTs of rehabilitative CBT, [18, 25, 26] with 69% improved after an educational
rehabilitation that closely resembled CBT.E 433 This compares to 18 to 63% improved in the
two RCTs of GET, [23' 241 and 47% improvement in a clinical audit of GET.E 54] Having usual
medical care allowed 6% to 17% to improve by one year in two RCTs.E 18' 253 There are no
previous RCTs of APT to guide us,[11' 12] but we estimate that APT will be at least as
effective as the control treatments of relaxation and flexibility used in previous RCTs, with 26% to 27% improved on primary outcomes.E23' 26] We propose that a clinically important
difference would be between 2 and 3 times the improvement rate of SSMC.

I haven't read the literature reviews re CBT/GET etc., in detail, so I'm not very clued up on any past studies.

But I've just noticed, in this passage, that it gives the results for 'relaxation and flexibility controls' used in previous RCTs, as 26% to 27% of patients improving for primary outcomes. This seems to be patients who achieve SF-36 PF scores of 75 and above.

This is very interesting because it means that it could be said that CBT and GET (in the PACE Trial) were found to be less effective than relaxation in other studies, at least in terms of individual outcomes.

Has anyone else looked into this more deeply at all?


Edit:

After reading the relaxation paper in more detail, it seems that the relaxation group was the control group in that paper, so I don't think we can legitimately compare the improvements attributable to CBT/GET in the PACE Trial, with improvements seen in the relaxation group from the relaxation paper.

Randomised controlled trial of graded exercise in patients with the chronic fatigue syndrome
Kathy Y Fulcher, Peter D White
BMJ 1997, 314:1647-1652.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2126868/pdf/9180065.pdf

 

[Graham]

Yes, you are right of course Bob about where the ball is! And I know it is just geekiness here - no wonder I feel at home.

I'm impressed with your spotting of the relaxation bit. I have read that paragraph several times, and just blanked over it, but of course, it is so useful!

 

[Bob]

Yes, you are right of course Bob about where the ball is! And I know it is just geekiness here - no wonder I feel at home.

At home with the geeks!

I'm impressed with your spotting of the relaxation bit. I have read that paragraph several times, and just blanked over it, but of course, it is so useful!

I only noticed it today, and it was your new video that promoted me to have another look at that paragraph.
I wanted to see how they came up with the 50% (CBT) and 60% (GET) figures.

If we can say that relaxation is more effective than CBT and GET were found to be in the PACE Trial, then that's quite useful, isn't it.

 

[Graham]

I've just been watching a recording of the Channel 4 programme on Ecstasy. Do you think they would fund a follow up study for the value of Ecstasy on people with ME?

 

[Dolphin]

This is something that I've only just noticed...

This passage is taken from the full protocol:



I haven't read the literature reviews re CBT/GET etc., in detail, so I'm not very clued up on any past studies.

But I've just noticed, in this passage, that it gives the results for 'relaxation and flexibility controls' used in previous RCTs, as 26% to 27% of patients improving for primary outcomes. This seems to be patients who achieve SF-36 PF scores of 75 and above.

This is very interesting because it means that it could be said that CBT and GET (in the PACE Trial) were found to be less effective than relaxation in other studies, at least in terms of individual outcomes.

Has anyone else looked into this more deeply at all?


(They say that most studies use an SF-36 PF threshold of 75 points to indicate an 'improvement' or a 'positive outcome', and this was their original intent in the PACE Trial, so I wonder if we should make a FOI request to get these results.)

I just checked reference 23. It refers to Fulcher and White (1997) - free at:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2126868/pdf/9180065.pdf
27% was the percentage who were graded as "Very much better" and "Much better". This was the primary outcome for this trial, rather than the PACE Trial:

Outcome measures: The main outcome measure was the self rated clinical
global impression change score, which is a validated
measure of overall change compared with study onset,
with seven possible scores from “very much worse”
(score 7) to “very much better” (score 1).28 Secondary
outcome measures included assessments of strength
and fitness, symptoms, and functional capacity, as
described above.

I read somewhere that person gave the answer to Peter White in an appointment, which makes wonder was there any chance this could add bias to that result.

 

[Bob]

I just checked reference 23. It refers to Fulcher and White (1997) - free at:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2126868/pdf/9180065.pdf
27% was the percentage who were graded as "Very much better" and "Much better".

Thanks for looking that up, Dolphin.

I read somewhere that person gave the answer to Peter White in an appointment, which makes wonder was there any chance this could add bias to that result.

I can't work out what you are referring to there? What do you mean by 'the answer'?

 

[Dolphin]

I can't work out what you are referring to there? What do you mean by 'the answer'?

This was the primary outcome measure:

Outcome measures: The main outcome measure was the self rated clinical
global impression change score, which is a validated
measure of overall change compared with study onset,
with seven possible scores from “very much worse”
(score 7) to “very much better” (score 1).28

So presumably you go in and have a little chat with him to help you answer the question. So if PDW asked: are you able to go for any walks these days: the GET person might say, "yes, I go for 3 x 5-10 minute walks a week on average". PDW might follow that up with: "were you able to do this before you started?" Person with CFS might say "no". PDW might say: "that looks like a lot of progress". A patient who wasn't doing the GET might be doing just as much activity in total, just not going for walks. He potentially might be able to persuade them they didn't make much progress.

 

[Bob]

I've just been checking out that paper that's referenced in the PACE Trial, regarding relaxation (as discussed above):

Randomised controlled trial of graded exercise in patients with the chronic fatigue syndrome
Kathy Y Fulcher, Peter D White
BMJ 1997, 314:1647-1652.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2126868/pdf/9180065.pdf


It's supposed to be a controlled trial, but I can't work out how to interpret the data with respect to the controls.

So I'm not sure if the "27%" figure for a positive outcome for the 'relaxation & flexibility' group takes into account the control, or not. ("Flexibility" just refers to stretching exercises.)

The mean SF-36 physical function scores are also given. For the relaxation & flexibility group, the mean scores for SF-36 physical function increased from 47 to 55. This is an increase of 8 points, which is about the same as both CBT (7.1) and GET (9.4) in the PACE Trial.

However, again, I can't work out how it this takes account of the control.

I'll read it again, but if anyone else is interested, and wants to have a look, then I'd be grateful for any interpretations.

 

[Dolphin]

I've just been checking out that paper that's referenced in the PACE Trial, regarding relaxation (as discussed above):

Randomised controlled trial of graded exercise in patients with the chronic fatigue syndrome
Kathy Y Fulcher, Peter D White
BMJ 1997, 314:1647-1652.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2126868/pdf/9180065.pdf


It's supposed to be a controlled trial, but I can't work out how to interpret the data with respect to the controls.

So I'm not sure if the "27%" figure for a positive outcome for the 'relaxation & flexibility' group takes into account the control, or not. ("Flexibility" just refers to stretching exercises.)

The mean SF-36 physical function scores are also given. For the relaxation & flexibility group, the mean scores for SF-36 physical function increased from 47 to 55. This is an increase of 8 points, which is about the same as both CBT (7.1) and GET (9.4) in the PACE Trial.

However, again, I can't work out how it this takes account of the control.

I'll read it again, but if anyone else is interested, and wants to have a look, then I'd be grateful for any interpretations.

27% was the percentage who said they were "Very much better" (7%) or "Much better" (20%). This was the primary outcome for the trial. There was no "no treatment"/"waiting list" control group.

In the PACE Trial, the GET group went from 36.7 to 57.7 at 12 months, while the CBT group went from 39.0 to 58.2 so I'm afraid I wouldn't say the results are similar to the relaxation & flexibility group. If one restricts it to 12 weeks, GET went from 36.7 to 48.1 while the CBT group went from 39.0 to 51.0.

 

[Bob]

27% was the percentage who said they were "Very much better" (7%) or "Much better" (20%). This was the primary outcome for the trial. There was no "no treatment"/"waiting list" control group.

Thanks Dolphin. It says it was a controlled trial, but there was no control group. So, do you know how this works in terms of making adjustments for placebo effect etc.?

 

[Dolphin]

Thanks Dolphin. It says it was a controlled trial, but there was no control group. So, do you know how this works in terms of making adjustments for placebo effect etc.?

There was a control group, matched for therapist time. Arguably a better control group. Although having both is more interesting.
I'm not an expert on the placebo effect and am slightly rusty on the trial to remember what might be different between the two groups.

The correspondence (i.e. published letters) are at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2127599/pdf/9361552.pdf

A few e-letters are here: http://www.bmj.com/content/314/7095/1647?tab=responses (I don't think the BMJ accepted e-letters at the time, and not many patients were on the internet, so there aren't many).

 

[Dolphin]

From:
http://apaththroughthevalley.wordpr...cise-testing-in-cfsme-research-and-treatment/

Summary of a talk by Dr. Chris Snell. Video can be seen at:

Extract mentioning PACE Trial:

There are problems with the PACE trial:
- they very selectively reported results
- they only took high-functioning CFS patients
- they used the 6-minute walking test (see below)
- patients at the end of the trial, were still walking at a severely disabled speed, even when they had improved the distance they could walk. If a patient who needed a heart transplant could only walk this speed due to a lung problem, they would not be allowed onto the transplant list because they would not be deemed well enough to actually survive.
- there is no mention of any improved functioning in any other area for any of the trial participants



There are problems with the 6-minute walking test (as used in the PACE trial as a measure of functioning) and other similar tests
- they assume that the patient does not exercise to exhaustion, or anywhere near exhaustion. All the understanding of the results are based on the assumption that it was just a casual exercise experience that they could easily repeat.
- it does not work for specific groups of unhealthy people, it is designed only for a healthy population, so results from an unhealthy group can’t be interpreted validly. This is because they rely on the heart rate as a measure of energy production, but the way the heart rate and energy production are linked in a healthy person is not necessarily the same in a sick person. Many studies in fact show that the link between them in certain diseases is very different – meaning you cannot rely on these tests in sick individuals, without first carrying out studies to determine what the connection is.
- the american heart association says not to use tests with heart-rate measures, as many people use heart-rate controlling medication (eg: for POTS, migraines)

The best way to assess physical function is to use “cardio-pulmonary exercise testing”, which is to measure:
etc

If people want to discuss Dr. Chris Snell video in general, you'll probably get a bigger audience by starting a new thread on it.

 

[urbantravels]

Please do start a new thread. I've often thought Dr. Snell and the PFL have some of the most compelling arguments against weak subjective measures like the 6-minute walk test.

 

[Bob]

There was a control group, matched for therapist time. Arguably a better control group. Although having both is more interesting.
I'm not an expert on the placebo effect and am slightly rusty on the trial to remember what might be different between the two groups.

The correspondence (i.e. published letters) are at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2127599/pdf/9361552.pdf

A few e-letters are here: http://www.bmj.com/content/314/7095/1647?tab=responses (I don't think the BMJ accepted e-letters at the time, and not many patients were on the internet, so there aren't many).

It seems that the relaxation group was the control group in that paper, so I don't think we can legitimately compare the improvements attributable to CBT/GET in the PACE Trial, with improvements seen in the relaxation group from the other paper, after all.

 

[Bob]

I have a feeling that this has been posted repeatedly, but it's a helpful reminder re the 'normal range'...

Chronic Fatigue Syndrome - A Clinically Empirical Approach to Its Definition and Study
W. C. Reeves, D. Wagner, R. Nisenbaum, J. F. Jones, B. Gurbaxani, L. Solomon, et al.
BMC Medicine, Vol. 3, 2005.
doi:10.1186/1741-7015-3-19
http://www.biomedcentral.com/1741-7015/3/19

"We defined substantial reduction in occupational, educational, social, or recreational activities as
scores lower than the 25th percentile of published US population [11] on the physical function (≤ 70) [...] subscales of the SF-36"

"Functional impairment, fatigue and an accompanying symptom complex characterize CFS. We defined functional impairment as scores ≤ 70 on the physical function [...] subscales of the SF-36."


So Reeves et al. 2005, defined "functional impairment", and a "substantial reduction in occupational, educational, social, or recreational activities", as an SF-36 PF score of 70 or less.

Note that the PACE paper cited the Reeves 2003 paper, not the 2005 paper.

 

[Graham]

I have just been reading through the protocols, getting information for the third video, and I noticed that they only intended to buy 12 actometers. You know I can't resist it! I did the sums. If we spread the 12 actometers over the 640 patients, then over the first six months of the trial, which included three assessments on each of the fatigue and sf-36 scales, each patient could have the actometer for 3.4 days, and that includes collection and return. So if everything worked without a hitch: I don't think they were ever serious about using them to back up their data.

 

[Dolphin]

I have just been reading through the protocols, getting information for the third video, and I noticed that they only intended to buy 12 actometers. You know I can't resist it! I did the sums. If we spread the 12 actometers over the 640 patients, then over the first six months of the trial, which included three assessments on each of the fatigue and sf-36 scales, each patient could have the actometer for 3.4 days, and that includes collection and return. So if everything worked without a hitch: I don't think they were ever serious about using them to back up their data.

Are you sure that's right? Remember this trial was spread over several years: it wasn't all done in one year.

And they still said they were going to use at the start still - they have not said this hasn't happened, to the best of my knowledge.