PACE Trial and PACE Trial Protocol

[anciendaze]

Reminds me of that physics joke about eliminating inconvenient variables: 'Assume a cow is a sphere'...

Spherical cows have a long and honorable history in physics, where solving a different problem may yield useful insight.

This disease brings to mind the daunting complexity of compressible aerodynamics, pioneered by Theodore von Karman, who once said, "an aerodynamicist is willing to assume anything, except responsibility."

A similar disconnect shows up in this research.

 

[Sean]

Spherical cows have a long and honorable history in physics,...

Certainly do, when used appropriately to eliminate irrelevant variables/factors, as opposed to inconvenient but relevant ones, like PEM in ME/CFS for example.

The whole trick, of course, is knowing which ones to ignore and which to pay attention to.

"an aerodynamicist is willing to assume anything, except responsibility."

He he.

 

[oceanblue]

My point is that I have not been able to find any part of a healthy, working-age population in that range...

What does that tell you about using statistics derived from the general population for comparison?

I think we are saying the same thing.

And an SF36 score of 60 can be attained by someone who can only walk a few hundred yards and climb a single flight of steps: that's hardly a measure of 'normal' for someone of working age.

 

[biophile]

PACE didn't use normative data from a working age population as claimed

oceanblue wrote: Bigger problems in this case are a) they didn't use the figures for a working age population and b) the 'mean -1SD' formula itself as a basis for establishing a 'normal' threshold, regardless of whether or not that population is Gaussian.

[oceanblue expresses suspicion regarding the PACE definition of threshold for "normal", and also quotes the published 2007 protocol]:

"The SF-36 physical function sub-scale [29] measures physical function, and has often been used as a primary outcome measure in trials of CBT and GET. We will count a score of 75 [(out of a maximum of 100)or more, or a 50% increase from baseline in SF-36 sub-scale score] as a positive outcome. A score of 70 is about one standard deviation below the mean score about 85, depending on the study for the UK adult population [51,52]."

I agree, something seems questionable about what dataset they used to calculate 60 as the lower threshold of "normal". In an earlier post, Dolphin gave us a mean(SD) of 84.15(23.28) from Velanovich 2007 for a US general population including the elderly, which he found was suspiciously close to the normative values used in the PACE trial for the UK working age population to calculate the threshold of "normal" (84-24=60). I looked into references 51-52 in the published PACE protocol (White et al 2007) and came to a similar suspicion, as outlined below.

Reference [51] is Jenkinson et al 1993, "Short form 36 (SF 36) health survey questionnaire: normative data for adults of working age". I didn't find a total mean+/-SD (more on that later) but Tables 4 and 5 break down physical functioning in age brackets and gender. Using the PACE method for calculating the lower threshold for "normal", only a minority of age brackets came close to 60, the males 55-64yrs @ 57.9 points, females 45-54yrs @ 66.5 points, and females 55-64yrs @ 51.3 points; the rest are 70-80 and suggest 60 is inappropriate for a "working age population". C Jenkinson was a co-author of Bowling et al 1999 and the latter refer to Jenkinson et al 1993 as the [Oxford (Central England) Healthy Life Survey 1991-1992 (ages 18-64)], describe it as "the most commonly used normative dataset", and for that sample give a total mean(SD)=88.4(17.9), which when used in the PACE definition for "normal", gives us a threshold of about 70 points for the working age population and is being dragged down by the older age groups (PACE cohorts were 38+/-11yrs).

Reference [52] is Bowling et al 1999 which we are already familiar with, the "1992 Omnibus sample", a general population including the elderly. I can't find the total sample mean(SD) so I attempted to calculate it for the working age population and the total general population based on the data given in Table 3. Working age population (16-64yrs), n=1532, mean(SD)=89.77(18.03), therefore PACE "normal" threshold ? 72 when rounded to the nearest point (70 was mentioned in the protocol and 75 was the original safe threshold for "improvement"). Total population (combining the male and female groups), n=2042 (lacking gender data for n=5, close enough for the point to be made), mean(SD)=83.84(24.30) therefore PACE "normal" threshold ? 60 when rounded to the nearest point (60 of course being the revised threshold for "normal" in the PACE results paper).

So the calculated estimates for the general population are consistent with the PACE revised definition of "normal" threshold while the calculated estimates for the "working age population" are consistent with the original PACE definition of "improvement" threshold. We now have evidence that White et al didn't use PF/SF-36 normative data from a working age population as they claimed in the Lancet paper, apparently a general population was used instead which included the elderly and not to mention the diseased as well. This invalidates their figures for the proportion of participants who moved into the "normal range" for both primary outcomes at 52 weeks.

We have previously established that using the SD to help define "normal" in a heavily skewed distribution is highly misleading, and that PACE should have used normative data from an appropriate age bracket ie 35-44yrs since PACE cohorts were 38+/-11yrs which from both Bowling et al 1999 and Jenkinson et al 1993 gives us a threshold of about 80. Including people with illness in the normative data is also inappropriate. Now we know that the wrong dataset was used as well, and considering their familiarity with PF/SF-36 in the original protocol and the unexpected change in the final paper which gives us a lower score for no good reason, the question needs to be asked; was this just another simple mistake or was it more spin and perhaps even worse? At the very least the authors need to issue a correction in the Lancet and a press release, which seems unlikely. This also doesn't look good for the "prestigious" Lancet peer-review process.

As a side note, recently on another thread, urbantravels posted [VanNess 2010 - PEM in Women.pdf]. Although admittedly a small study with only 23 CFS patients and 25 healthy controls, Table 1 on the SF-36 subscales [mean(SD)] is interesting: Physical function, CFS=26.84(17.41), controls=96.84(4.48). These patients were more impaired than the PACE trial participants, and to reach White et al's dubious definition of "normal" (using data from the controls in this study) they would have had to score 90!

 

[biophile]

oceanblue wrote: Well, if that's the case then GET basically made no difference to physical condition. They got nothin'!

oceanblue wrote: And an SF36 score of 60 can be attained by someone who can only walk a few hundred yards and climb a single flight of steps: that's hardly a measure of 'normal' for someone of working age.

Yet we still had to put up with "psychotherapy and exercise cures CFS" rubbish in the news despite the Oxford criteria CFS patients still on average being as impaired on the 6WMD and PF/SF-36 as patients of other serious medical diseases.

ancientdaze wrote: I wish to point out that a normal (Gaussian) distribution should have mean, median and mode approximately equal. Here they are using a mean of 85 when the population mode is 95 or 100.

As the majority of the population are scoring 95-100/100 points, wouldn't be surprised if the median is 95 or 100 as well.

ancientdaze wrote (about PF/SF-36 scores): If you look at the population data carefully, you will see that the healthy population has a much stronger central tendency than a normal distribution. I've tried to find a comparable group equally far from the mode for comparison. This isn't an exhaustive search, but what I've found so far includes: people recovering from surgery, people with serious conditions like cancer, COPD or heart failure, people well over 65. It is virtually impossible to find healthy people way out in that tail for comparison. In every other case, so far, the comparison group has serious organic problems resulting from either a known disease or aging.

Good idea/point.

ancientdaze wrote: This disease brings to mind the daunting complexity of compressible aerodynamics, pioneered by Theodore von Karman, who once said, "an aerodynamicist is willing to assume anything, except responsibility." A similar disconnect shows up in this research.

Hehe.

Sean wrote: Certainly do, when used appropriately to eliminate irrelevant variables/factors, as opposed to inconvenient but relevant ones, like PEM in ME/CFS for example. The whole trick, of course, is knowing which ones to ignore and which to pay attention to.

Precisely.

 

[oceanblue]

PACE didn't use normative data from a working age population as claimed

Reference [51] is Jenkinson et al 1993, "Short form 36 (SF 36) health survey questionnaire: normative data for adults of working age". ... C Jenkinson was a co-author of Bowling et al 1999 and the latter refer to Jenkinson et al 1993 as the [Oxford (Central England) Healthy Life Survey 1991-1992 (ages 18-64)], describe it as "the most commonly used normative dataset", and for that sample give a total mean(SD)=88.4(17.9), which when used in the PACE definition for "normal", gives us a threshold of about 70 points for the working age population and is being dragged down by the older age groups (PACE cohorts were 38+/-11yrs).

Reference [52] is Bowling et al 1999 which we are already familiar with, the "1992 Omnibus sample", a general population including the elderly. ...Working age population (16-64yrs), n=1532, mean(SD)=89.77(18.03), therefore PACE "normal" threshold ? 72 when rounded to the nearest point (70 was mentioned in the protocol and 75 was the original safe threshold for "improvement"). Total population (combining the male and female groups), n=2042 (lacking gender data for n=5, close enough for the point to be made), mean(SD)=83.84(24.30) therefore PACE "normal" threshold ? 60 when rounded to the nearest point (60 of course being the revised threshold for "normal" in the PACE results paper).

So the calculated estimates for the general population are consistent with the PACE revised definition of "normal" threshold while the calculated estimates for the "working age population" are consistent with the original PACE definition of "improvement" threshold. We now have evidence that White et al didn't use PF/SF-36 normative data from a working age population as they claimed in the Lancet paper, apparently a general population was used instead which included the elderly and not to mention the diseased as well. This invalidates their figures for the proportion of participants who moved into the "normal range" for both primary outcomes at 52 weeks.

Including people with illness in the normative data is also inappropriate. Now we know that the wrong dataset was used as well, and considering their familiarity with PF/SF-36 in the original protocol and the unexpected change in the final paper which gives us a lower score for no good reason, the question needs to be asked; was this just another simple mistake or was it more spin and perhaps even worse? At the very least the authors need to issue a correction in the Lancet and a press release, which seems unlikely. This also doesn't look good for the "prestigious" Lancet peer-review process.

Great work - you must have put a huge amount of effort has gone into producing these figures, which clearly show that PACE used the wrong data to calculate it's 'within the normal range' criterion. We now need to figure out the best way to use these figures. And yes, it is hard to see how PACE made a mistake like this, given that they started off with the right figures in the protocol....

 

[Sean]

Nice one, biophile.

"and perhaps even worse?"

Difficult to see this manipulation of the protocol, especially after the data was collected, as anything but a very deliberate attempt to produce a convenient but unscientific (and unethical) result.

Such behaviour is dangerously close to blatant fraud.

 

[Snow Leopard]

Reference [51] is Jenkinson et al 1993, "Short form 36 (SF 36) health survey questionnaire: normative data for adults of working age".

Look at table 7. It gives the data for patients Respondents not reporting long standing illness.

The mean PF score was 92.5 (SD=13.4). I pointed out this in my letter to the Lancet.

Since the trial subjects had all other illnesses ruled out, normal therefore cannot be defined as anything <80 and certainly higher if you rule out the elderly.

 

[Enid]

I'm feeling much more comfortable now about ruling out the obvious - recall a time at a critical stage when emerging fro my flat to hear water flowing somewhere in the propery - having neither the time nor energy to contact those about I left. As it happened and ignored by all others around there was a burst pipe flooding a flat next door and all those below.! QED. Bit brighter than the rest. You really have to laugh at these people.

 

[oceanblue]

Look at table 7. It gives the data for patients Respondents not reporting long standing illness.

The mean PF score was 92.5 (SD=13.4). I pointed out this in my letter to the Lancet.

Since the trial subjects had all other illnesses ruled out, normal therefore cannot be defined as anything <80 and certainly higher if you rule out the elderly.

Glad you mentioned that in your letter. Yes, for working age the threshold would probably come out at 85 - the threshold the protocol specified for 'recovery'.

 

[Mark]

Has there been any progress in summarising the main points of critique of the PACE trial, in a wiki section or elsewhere?

This latest point regarding the 'normal' functioning data, for example, ultimately needs to be summarised into a few sentences - a headline which "the man in the street" could understand, and a referenced summary of the relevant technical points. The same sort of thing is needed for all the points of critique.

For example: would it be correct to say in bold something similar to "the study authors changed [1] their definition of "normal" functioning, partway through the study, downgrading it from the original x [2] to y, where y was determined as the lowest-functioning 15% of patients referred to doctors [3]."

(That last sentence probably isn't quite accurate but that's the kind of stark series of well-referenced sentences we need - clear statements of referenced fact that make it clear to anyone what a swindle this is).

I know there are several anti-PACE articles and critiques, but nothing I've seen yet to hammer the key points home to people who don't have the technical background. And of course the public and doctors here have all seen a uniformly pro-PACE agenda across all papers - the truth of this matter is completely invisible to everybody that matters in a practical sense in our lives.

I'm sorry to say it, but as a community we never seem to respond well enough in circumstances like this - we always seem to fail to put our case as strongly as it merits. We desperately need some better summarising of the key points and a good quality, referenced, easy to read document setting out the flaws in the research. Something I can point my family at. At the moment, we still seem to have nothing like that at all - and that's just ridiculous. I think it's partly because we lack a good quality wiki (for the time being...) but we do have a wiki and plenty of people who are willing and able to work together on this task...so what's happening?...

 

[Enid]

Ok Mark - anyone up to it - it is an enormous job apart from all the people/charities/pressure groups/gov/current science/history etc to pull together. I for one do not have it. And it is impossible to ride roughshod over all these working in their way for ME. It's a grass roots thing until we can finally clobber the real bogies (oh and with science revealing all which it is). One day this will be written up for the pathetics/education of Medics.

 

[oceanblue]

Has there been any progress in summarising the main points of critique of the PACE trial, in a wiki section or elsewhere?

I'm sorry to say it, but as a community we never seem to respond well enough in circumstances like this - we always seem to fail to put our case as strongly as it merits. We desperately need some better summarising of the key points and a good quality, referenced, easy to read document setting out the flaws in the research. Something I can point my family at. At the moment, we still seem to have nothing like that at all - and that's just ridiculous. I think it's partly because we lack a good quality wiki (for the time being...) but we do have a wiki and plenty of people who are willing and able to work together on this task...so what's happening?...

Ah, Mark, I feel we have let you down

But you are right and this is important work - 150,000-ish words of this thread is problably a little indigestible. I'm thinking of having a go myself, but it won't be for a good month as I'm still looking at other stuff. One option would be to have a new 'PACE Summaries - no geeky posts here' thread where people could post summaries of either the whole thing or parts that they felt were most important. It might be hard to get a single consensus document but we could probably agree on most of it and then people could add summaries of parts they feld were important.

Using the main forum, rather than the wiki, might give the project more attention though perhaps moderators and others would feel this was inappropriate. Comments welcome.

 

[Sean]

This latest point regarding the 'normal' functioning data, for example, ultimately needs to be summarised into a few sentences...

Don't need many words at all to get this point across. Use the graphic developed on the PACE thread that compares PACE's definitions of 'normal' or 'recovered' to the real normal for the general (healthy) population. Indeed, I would suggest keeping any word based explanation down to a minimum for this point, given we have such a clear graphic. Probably one or two other criticisms that could be presented graphically.

Also agree completely that we have to do a good summary of the thread (and related stuff), and that it is a pretty big task, that is not going to happen overnight. But a very, very important one. Apart from any other reason, most of the serious flaws in PACE are the same flaws as in the basic psycho-social CBT/GET model, so any hits we can score on PACE will have broader applicability.

I have serious family stuff to deal with at the moment, for the next month or so, including some travel, which is always an effort. But after that I am going to look at it. Also needs at least 2-3 people separately going over the thread, to reduce any chance of bias or missed/misinterpreted points (for example, many of us, including me, are not so sharp on the details of stats stuff).

We made a good start, we just have to keep the momentum going, as best we can. This is primarily a boring long distance race, not an exciting sprint.

 

[Graham]

Hi all

I'm a new member, but I have been working with Bob for some time on an analysis of the PACE trial. We have put together a blog with a couple of animations that just focuses on a small but representative part of the study and shows how ineffective the therapies were. We are hoping that it could be a "dummies' guide", but have avoided calling it that so that it doesn't put off professionals. This Phoenix thread, quite rightly, has torn the study to pieces: there is just so much that is wrong about it. That is why it took us so long to decide on what to include in the blog. In the end we decided not to get embroiled in all the technical stuff, but just to focus on how even their own results actually decry the use of GET and CBT, and to start off with a softly-softly approach so that we don't put off people who have, so far, fallen for the psychological propaganda. It was just so easy to keep adding bits, but eventually we went for a ruthless prune.

The blog is at meanalysis.blogspot.com, but the animations are through Vimeo which obscures the image at the end. There is an alternative version at meetup.org.uk/pace2.html but you will need Quicktime for that. Both have a link to a pdf download.

We would be interested in your opinions (don't worry, I taught maths for 40 years - I can take punishment!), but I'm afraid that the quality of the commentary is about as good as my present equipment allows. I'm told that it is much more effective than counting sheep if you are having trouble getting to sleep.

I can manage long-distance boring!

 

[Dolphin]

4th letter published this week.

http://bit.ly/f0noE0
i.e.
http://www.imt.ie/opinion/2011/04/caution-is-still-required-on-get-and-me.html

I wonder will we top the 5 they published in reply to a piece last year.

5th letter published this week.

http://bit.ly/ev0scy i.e.
http://www.imt.ie/opinion/2011/04/cfs-has-developed-multiple-research-definitions.html

CFS has developed multiple research definitions

 

[Jenny]

This article is in print form now in the April issue of The Psychologist. I'm going to try to write a letter in response. They publish quite long letters so it's worth a go.

Also, it would be worth thinking about submitting an academic article to this journal. Articles can be shorter than those in most journals. The Psychologist has a very large readership compared to most academic publications - over 40,000.

Guidelines for submitting are here:

http://www.thepsychologist.org.uk/contribute/how.cfm

I think you have to be a psychologist to submit a paper.

If anyone would like to collaborate with me to submit something, let me know. Before I had to give up work I was an academic psychologist and journal editor.

Jenny

Oceanblue and I have sent a the following letter to The Psychologist commenting on their piece on the PACE trial:

Your summary of the findings of the PACE trial (White et al., 2011), which evaluated the effectiveness of CBT and graded exercise therapy as treatments for ME/CFS (Fatigue evidence gathers PACE, April 2011), gave a somewhat misleading impression of the outcomes of this study. You said that self-reports on measures of fatigue and physical function showed that 30% of CBT patients and 28% of exercise patients had returned to normal function. This suggests that nearly one-third had recovered with these treatments. Unfortunately this is far from the case.

First, the thresholds for normal were set so low they could include those with considerable disability. The authors defined within the normal range as a Short Form-36 Physical Function (SF-36) score of 60 or more (0-100 scale), yet the problems with physical functioning that characterise CFS were defined by a SF-36 score of up to 65 - which overlaps with normality. The situation with fatigue scores is similar, so that a participant may have met the trial fatigue criteria for CFS yet simultaneously have met the criteria for normal. Consequently the figures you quoted tell us little meaningful about the PACE trials effectiveness.

Secondly, it is of some concern that the authors inexplicably changed the criteria for positive primary outcomes originally proposed by them in the protocol for the study
(White et al., 2007). On the Chalder Fatigue Scale, for example, they stated that a positive outcome would be a 50% reduction in self-reported fatigue, or a score of 3 or less. And on the SF-36 scale of physical function a score of 75 or more, or a 50% increase from baseline would be required. So the figures you reported are misleading.

Thirdly, you omitted to mention the disappointing outcomes on more objective measures of functioning. For example, after a year of treatment, patients receiving graded exercise therapy had on average increased the distance they were able to walk in 6 minutes from 312 to 379 metres. Even patients suffering from serious chronic cardiopulmonary diseases can manage more than this (in a sample of over 1,000 such patients the mean distance walked was 393 metres [Ross, Murthy, Wollak, & Jackson, 2010]), and at normal walking speed people typically cover around 500 metres. CBT treatment had no significant effect on walking distance.

Perhaps these results are unsurprising, given that the treatments focused on reducing patients assumed fear of engaging in activity, and completely failed to acknowledge the complexity of this illness. We are much concerned that exaggerated claims for these treatments will create a false impression of the effectiveness of PACE amongst psychologists, and will continue to divert scarce resources away from effective medical treatments for this devastating condition.

References
Ross, R.M., Murthy, J.N., Wollak, I.D., & Jackson, A.J. (2010). The six-minute walk test accurately estimates mean peak oxygen uptake. BMC Pulmonary Medicine, 10.31. http://www.biomedcentral.com/1471-2466/10/31

White P.D., Sharpe M.C., Chalder T., DeCesare J.C., Walwyn R., and the PACE Trial Group (2007). Protocol for the PACE trial: A randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BioMed Central Neurology, 2007;7:6. http://www.biomedcentralcom/1471-2377/71/6

White, P.D., Goldsmith, K.A., Johnson, A.L, Potts, L., Walwyn, R., DeCesare, J.C, Baber, H.L., Burgess, M., Clark, L.V., Cox, D.L., Bavinton, J., Angus, B.J., Murphy, G., Murphy, M., ODowd, H., Wilks, D., McCrone, P., Chalder, T., Sharpe, M.C., on behalf of the PACE trial management group (2011). Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE). The Lancet, published online February 2011. DOI:10.1016/So140-6736(11)60006-2.

 

[Sean]

Well done, Jenny and Oceanblue.

 

[Sean]

Hi all

I'm a new member, but I have been working with Bob for some time on an analysis of the PACE trial. We have put together a blog with a couple of animations that just focuses on a small but representative part of the study and shows how ineffective the therapies were.

Ooh, lovely. Those animations are exactly the sort of thing we need to clearly highlight the critical issues.

One small suggestion, the 'Qualifying Level' marker in the second video is pretty faint, could be a little more solid.

Thanks for that, guys.

 

[Dolphin]

Oceanblue and I have sent a the following letter to The Psychologist commenting on their piece on the PACE trial:

Well done and thanks. Fingers crossed they publish it.