'Recovery' from chronic fatigue syndrome after treatments given in the PACE trial

[Sean]

"Do you want to go back to the life that gave you ME?"

[snip]

He said,'Do you think it's down to you?' On my report he has written: "[name] is making good progress. [Name]'s only remaining barriers are [name]'s illness beliefs"

Standard double-bind blame-the-victim thuggery.

The coward's creed.

All I have come to expect from this bunch of a-holes.

 

[Graham]

So, looking at the sf-36 and the scores that indicate recovery, being able to play a game of football, do the vacuuming, walk up a flight of stairs or walk more that a mile are the kind of activity that makes a 38 year old susceptible to ME. No wonder I went down with it. It's time for a national campaign to do away with staircases, and ban sports.

 

[Simon]

From the paper:

Creating criteria for recovery from domains that are measured on a continuum requires the setting of operational thresholds based on population studies or trial eligibility criteria (Powell et al. 2004 ; Knoop et al. 2007 ; Malouff et al. 2008).

In this context it is important to note that recovery does not mean being free of all symptoms; population studies show that the average person in the UK reports a mean of four symptoms in any 2-week period (McAteer et al. 2011).

The three most common symptoms reported were fatigue, headache and joint pain; symptoms consistent with CFS (McAteer et al. 2011).

Anyone care to explain? Should I feel enraged or is this reasonable? I haven't read the papers cited. Thanks.

Just been reading the full text: Ascertaining the size of the symptom iceberg in a UK-wide community-based survey

Guess what? It's a poor study, misleadingly quoted.

1. Unrepresenative sample
The response rate was only 33% ie most people didn't respond. 60% female and older than the UK working age population. And as it was a healthy questionnaire, it's likely that those with health problems were more likely to respond. Witnessed by:

2. 44% of the sample had a chronic illness
Should be nearer 15%, I think, for working age population which was the group they targeted here.
So while the average number of symptoms was 3.6, it was 4.8 for those whith chronic illness and 2.7 for non-chronic. The non-chronically ill may still have worse than typical in the whole population as they chose to answer a health questionnaire that most people ignored.

They say:

A comparison of general health (as measured by the Short Form–36) among our responders with UK norms for a working-age population24,25 showed very similar scores for all dimensions except bodily pain, in which this sample had poorer scores. This similarity between the general health of this study's sample and the working-age UK general population further supports the generalisability of these results.

However, no data is presented and is seems rather suspect given that their sample has far more chronically ill people than the working age UK populaiton.

3. The symptoms resported are not generally severe
Typically around 15% or less of those reporting a symptom say it is severe.

So maybe we are looking at an average of 2 mild or moderate symptoms for a healthy person.
Which is rather different from the picture the authors have tried to paint.

 

[Firestormm]

Simon are you bored?

 

[Esther12]

Thanks Simon.

I don't think we've put as much work into examining the references in the recovery paper (I know I haven't) as there were already obvious problems with it. It's good to see some of it being looked into, and depressingly unsurprising to see that there 'evidence based' are dodgy.

 

[Tom Kindlon]

My new 1000-word BMJ rapid response (i.e. e-letter) on the £5m PACE Trial, "PACE Trial: Simply giving a reason why an outcome measure was changed is not necessarily sufficient".

It lists all sorts of reasons why I'm unhappy with the changes made to the PACE Trial outcome measures.

I'm afraid I knew it would be too long to get published, so didn't spend as much time on the wording as other letters where I'm hoping they might be published in the print edition.

http://www.bmj.com/content/347/bmj.f5963?tab=responses

 

[alex3619]

Don't forget to click "like" on Toms articles at the BMJ.

 

[peggy-sue]

My "like" was the first one.
(smug little so-and-so that I am)

 

[Tom Kindlon]

My new 1000-word BMJ rapid response (i.e. e-letter) on the £5m PACE Trial, "PACE Trial: Simply giving a reason why an outcome measure was changed is not necessarily sufficient".

It lists all sorts of reasons why I'm unhappy with the changes made to the PACE Trial outcome measures.

I'm afraid I knew it would be too long to get published, so didn't spend as much time on the wording as other letters where I'm hoping they might be published in the print edition.

http://www.bmj.com/content/347/bmj.f5963?tab=responses

Sean Lynch, a psychiatrist, has now responded, approx. 24 hours after my comment went up. Authors get automatic alerts for responses - I wonder whether there might be a connection?

His defence of the changes to the PACE Trial protocol largely seem to depend people trusting him as some sort of expert, perhaps an unbiased expert. There is little substance in them. It's like an appeal to authority.

Sean Lynch was one of the co-authors of the Royal Colleges of Report on CFS (1996) (Word file of this is available here:
http://bit.ly/HFQeOM ). As well as recommending CBT and GET, and little else, it gave very few suggestions for biological research.

He calls for more trials like the PACE Trial:

I feel that there is a need for a new debate now. PACE has in my view made an important contribution to the research evidence and has been a well-designed and conducted trial, but there is a need for further research to build on its contribution in my opinion.

[..]

Pragmatic trials might have advantages of larger sample size and seeing how well the best- evidenced treatments of that time are applied and fare in real world situations and their cost-effectiveness (6).

so that'd be CBT and GET trials and the like.

 

[Tom Kindlon]

Psychological Medicine said they received 15 letters to the trial but only 6 were published. If there are one or more people here whose letter wasn't published, perhaps you could consider posting it under the White et al. response on this page: http://www.bmj.com/content/347/bmj.f5963?tab=responses .

I'd like to respond to Sean Lynch but ideally I'd like at least one other person to post after he posted

Multiple, frequent changes in a protocol or method of analysis might raise valid concerns about a trial and this might beg the question about the need for independent secondary analysis of data. As far as I can see, from the correspondence and what has been written in the public domain so far, this does not appear to be the case here and the alterations mentioned do appear well justified in my view. Ultimately, if there are serious concerns, secondary analyses can be performed, however.

etc.

The White et al. letter can be read here: http://www.bmj.com/content/347/bmj.f5731?tab=responses

 

[biophile]

The domains chosen and the criteria for recovery on each were defined before we undertook the analysis. [...] We changed some of the thresholds for measuring recovery from those of the original protocols (White et al. 2007); we made the changes before analysis and to more accurately reflect recovery.

There is a separate thread for the PACE Trial statistical analysis plan:

http://forums.phoenixrising.me/index.php?threads/pace-trial-statistical-analysis-plan.26520

Just noting here that upon first glance/search of the 45 page document, there appears to be no mention whatsoever of the new definition of recovery, nor the definition of 'normal' fatigue and physical function which underpins it.

 

[Esther12]

There is a separate thread for the PACE Trial statistical analysis plan:

http://forums.phoenixrising.me/index.php?threads/pace-trial-statistical-analysis-plan.26520

Just noting here that upon first glance/search of the 45 page document, there appears to be no mention whatsoever of the new definition of recovery, nor the definition of 'normal' fatigue and physical function which underpins it.

They came up with them after seeing the results. I'm not sure that they were ever even looked at by the steering group.

 

[biophile]

@Esther12. At first I was going to lightly disagree with you because of this ...

The domains chosen and the criteria for recovery on each were defined before we undertook the analysis. [...] We changed some of the thresholds for measuring recovery from those of the original protocols (White et al. 2007); we made the changes before analysis and to more accurately reflect recovery.

However, I found some notes I took on this issue from previous discussions.

We have this from the 2011 Lancet paper:

The statistical analysis plan was finalised, including changes to the original protocol, and was approved by the trial steering committee and the data monitoring and ethics committee before outcome data were examined.

I assumed that the 'post-hoc' analyses mentioned in the 2011 Lancet paper meant after the trial began but before seeing the actual data. However, the statistical analysis plan mentions nothing whatsoever about the most notorious 'post-hoc' analysis, i.e. normal range in fatigue and physical function. Furthermore, the revised definition of recovery, which is fundamentally based on this so-called normal range, is not mentioned either in the statistical analysis plan.

The recovery paper, which was published 2 years after the 2011 Lancet paper, stated that, "we made the changes before analysis". This of course does not guarantee that they made the changes before working on previous papers involving the same PACE data which was being re-used in the recovery paper, it just means that they technically made the changes before conducting the statistical analysis for the recovery paper.

As an open-label trial the researchers could have gained obvious impressions that their favoured therapies were failing to meet expectations anyway. But you may indeed be right that they made some of the changes after seeing the data.

In the statistical analysis plan it states that: "The aim of this paper is to make public and to report in detail the planned analyses that were approved by the Trial Steering Committee in May 2010..."

So, when did the Trial Steering Committee approve the 'normal range' in fatigue and physical function, then? Unlike the first PACE paper, the recovery paper does not mention the trial steering committee. So it is also possible that the steering committee was not involved with these later changes, which would then effectively be unapproved and unplanned.

 

[Dolphin]

Thanks biophile for drawing all that together.

 

[biophile]

Another clue from the PACE trial FAQ#2 (late 2011) ...

http://www.pacetrial.org/faq/faq2.html

Steering committee approval is only mentioned in these two themes but not for normal range nor for recovery:

14. Why did you change entry criteria while the trial was recruiting participants?

27. Why did you change the analysis plan of the primary outcomes?

Aside, it states this about recovery:

"Being within the normal population range for these two outcomes does not necessarily mean the patient had recovered from CFS, so we are analysing separately the numbers of patients who recovered after treatment."

 

[biophile]

Guess what? It's a poor study, misleadingly quoted.

I am shocked, SHOCKED I tell you (not), that PACE would inaccurately cite such a study in order to support their spurious hint that trial participants are 'recovered' despite still being disabled with multiple lingering symptoms. ;-)

PACE also claimed that, "The NNT of 7 for recovery after both CBT and GET is within the range of the effects found for drug treatments in both general medical and psychiatric conditions (Leucht et al. 2012)."

I examined the cited paper and noticed that the drug treatments reviewed were found to be superior to a placebo control, which CBT and GET in the PACE Trial were not tested against, being an open-label trial with uncontrolled and various levels of optimism and/or encouragement between the different groups.

The 2008 Cochrane systematic review on CBT for CFS included a sub-analysis which suggests that merely being on a waiting list for treatment provides a 'positive influence' on questionnaire-answering behaviour as effective as actual CBT!

 

[Esther12]

@biophile: Also, the PACE researchers have said somewhere (I can't remember where now) that their 'normal range' analysis was prompted by a Lancet peer reviewer (would be nice to have all the peer review notes out in the open).

They've always been very careful in their phrasing about this.

 

[biophile]

@biophile: Also, the PACE researchers have said somewhere (I can't remember where now) that their 'normal range' analysis was prompted by a Lancet peer reviewer (would be nice to have all the peer review notes out in the open). They've always been very careful in their phrasing about this.

Thanks. Good point. I vaguely remembered something like that.
I think I managed to track it down too. It is from White et al's unpublished reply to Hooper:
http://www.meactionuk.org.uk/whitereply.htm

Normal ranges - The primary analysis compared the mean differences in the primary outcome scores across treatment arms, which are in the paper. The normal range analysis was plainly stated as post hoc, given in response to a reviewer’s request. We give the results of the proportions with both primary outcomes within normal ranges, described a priori, using population derived anchors.

SF-36 scores (page 31) - The definition of a “normal range” for the SF36 in the paper is different from that given in the protocol for “recovery”. Firstly, being within a “normal range” is not necessarily the same as being “recovered”. Secondly, the normal range we gave in the paper was directly taken from a study of the most representative sample of the adult population of England (mean - 1 SD = 84 – 24 = 60). The threshold SF36 score given in the protocol for recovery (85) was an estimated mean (without a standard deviation) derived from several population studies. We are planning to publish a paper comparing proportions meeting various criteria for recovery or remission, so more results pertinent to this concern will be available in the future. We did however make a descriptive error in referring to the sample we referred to in the paper as a “UK working age population”, whereas it should have read “English adult population”, and have made this clear in our response to correspondence.

I am still not 100% sure whether the Lancet reviewer in question had requested the normal range analysis itself or just requested that PACE describe the analysis as post-hoc. Remember that PACE previously used mean minus SD to come up with the threshold for a 'positive outcome' in physical function (>=75/100 points).

However, even if the 'normal range' used later in the Lancet paper was just tacked on at the last moment because a peer reviewer wanted it, this still would not explain why PACE then abandoned the original definition of recovery with a pathetically erroneous justification for the changes, apparently without any approval or oversight, and then used this makeshift and flawed 'normal range' to base the new definition of recovery on.

Either way it looks even more certain now that PACE made up the recovery criteria *after* first seeing the trial data.
I am not an expert on conducting clinical trials, but I am under the impression that this is frowned upon.
Yes, they have always been very careful in their phrasing about this in public.
But the allusion that all changes were approved before seeing the trial data appears to be false.

 

[Esther12]

However, even if the 'normal range' used later in the Lancet paper was just tacked on at the last moment because a peer reviewer wanted it, this still would not explain why PACE then abandoned the original definition of recovery with a pathetically erroneous justification for the changes, apparently without any approval or oversight, and then used this makeshift and flawed 'normal range' to base the new definition of recovery on.

Either way it looks even more certain now that PACE made up the recovery criteria *after* first seeing the trial data.
I am not an expert on conducting clinical trials, but I am under the impression that this is frowned upon.
Yes, they have always been very careful in their phrasing about this in public.
But the allusion that all changes were approved before seeing the trial data appears to be false.

I think secretive peer review is an utter joke. It wouldn't surprise me at all if someone like Wessely was peer reviewing, and coming up with helpful way to make their results look more exciting and positive for patients.

I'm not sure if they've ever really implied that ALL changes were approved before seeing the trial data. Some people do seem to have assumed that this is what they've said, but they never really come close to it. I think a lot of people's reading of PACE is affected by CFS bigotry.

 

[Bob]

I am still not 100% sure whether the Lancet reviewer in question had requested the normal range analysis itself or just requested that PACE describe the analysis as post-hoc.

My vague memory tells me that the reviewer demanded that a threshold for improvement should be included. I don't think they stipulated what the threshold should be, but just stipulated that there should be one. (Like you say, a threshold was originally included in the protocol, but it was dropped before the peer review stage.) But I might be wrong about this, and I cannot find the relevant information. I'm sure that I've read it somewhere, but don't rely on my memory. It's hopeless.