PACE Trial and PACE Trial Protocol

[Dolphin]

But I disagree with your presentation in one key area. If we are to get people who are not on our side to actually take notice, we must make the facts hammer the points home, and not indulge in opinion (such as "one of the most flagrant..."). I know you are right - it was appalling, but I am sure that we are more likely to get people to read it if we let the facts pile up and prove the case for us. For that reason, I would avoid the topic of conflict of interests - it is both relevant and important, but you have to earn credit in the reader's mind before you can start to dish the dirt. It is more the sort of thing that you keep up your sleeve for questions/follow-ups afterwards.

Unfortunately, people critical of these trials have already been painted as angry reactionaries (or some such thing), so I think that if we come out with all guns blazing, we will be ignored. Perhaps we ought to adopt the hunting style of the Komodo dragon - it just aims at getting a quick bite in, then leaves the wound to fester away, until finally the target dies.

I tend to agree with you but just to point out that in the 29 letters in the thread of submitted letters: http://forums.phoenixrising.me/show...-letters-that-were-not-accepted-by-the-Lancet , "insurance" isn't mentioned at all and "conflict(s) of interest" is only mentioned in one letter.

 

[Dolphin]

OK Mark, here comes the pushy newbie.

I generally find it is easier to work with something and knock it into shape, so here are my ideas. I'm not the sensitive sort, so I really won't mind how much of it you feel is rubbish: all that concerns me is that we end up with something useful and good.

My thinking went along the lines of how I used to mark statistics coursework. First determining the measurement systems, and setting up the targets for assessing success or failure. In a decent scientific study, these should be trialled and then the decisions more or less set in stone. (From what I can see, items 2. 3 and 4 were changed during the study.) Then the sampling, splitting into groups, and setting up a control group. Then the acquisition and processing of the data, followed by the actual results (and whether the results were successful or not according to the preset criteria). In addition, I have added another item - things that come to light after the trial.

1. They abandoned objective measures.
2. They used a fatigue scale where the average score was 28/33, so the only real way was up (Bill's random walk).
3. They redefined Normal
4. They set a very low target of an improvement in the average score of 2/33 in fatigue and 8/100 in SF36.
5. The use of the Oxford Criteria mean that there were many subgroups. A therapy that helped one subgroup, but did nothing for the others, even harming a few, would nevertheless show an improvement.
6. Many were excluded from the sample (your no.1)
7. They did not set up a control group, although they almost treated the SMC group as though it was one. Members of that group didn't receive the same attention of course, and if results are weak, a proper control group is essential to evaluate the placebo effect.
8. They trivialise adverse events.
9. The actual results were very poor: if you subtract the improvement made by the SMC group from the GET+SMC or CBT+SMC results, SMC in general outperforms GET or CBT. That could mean a large placebo effect: it could mean that having specialist medical attention prescribe drugs to help with pain/sleep is significant.
10. Subsequently we are told that it takes 7 or 8 patients to follow the therapy for a year for one to show a clinically significant improvement (not a return to near normality). That means that 6 or 7 have wasted their time. Sorry, that's a bit emotive ;-)

I think the order should be different though: it should be in an order that gives the most impact. I think perhaps 8, 9 and 10 create the initial impact, then the others home in on the reality of why it was such a bad study.

Over to you!

Well done on that, each point is summarised in a short way.

However, it would be nice to get a more complete list before then cutting them down.
For example, something about response bias would be good - then it might come under objective measures. I'd love if one or more people could go through the list of letters that the Lancet rejected http://forums.phoenixrising.me/show...-letters-that-were-not-accepted-by-the-Lancet and come up a longer list which might become a shorter list. I would feel more confident that a higher percentage of the points were covered if that was done.

 

[Angela Kennedy]

Well done on that, each point is summarised in a short way.

However, it would be nice to get a more complete list before then cutting them down.
For example, something about response bias would be good - then it might come under objective measures. I'd love if one or more people could go through the list of letters that the Lancet rejected http://forums.phoenixrising.me/show...-letters-that-were-not-accepted-by-the-Lancet and come up a longer list which might become a shorter list. I would feel more confident that a higher percentage of the points were covered if that was done.

And one needs to remember the wrong sort of pacing being trialled. Pacing is generally understood as an autonomous flexible self-management strategy, not the therapist imposed limitations and demands for diary keeping etc. that is 'APT'.

Also- I would say avoid the term 'sub-groups'. This is not accurate as far as the PACE trial goes, unless you're talking about sub-groups of Oxford, which is NOT ME or Canadian defined ME/CFS.

 

[oceanblue]

I generally find it is easier to work with something and knock it into shape, so here are my ideas.

1. They abandoned objective measures.
2. They used a fatigue scale where the average score was 28/33, so the only real way was up (Bill's random walk).
3. They redefined Normal
4. They set a very low target of an improvement in the average score of 2/33 in fatigue and 8/100 in SF36.
5. The use of the Oxford Criteria mean that there were many subgroups. A therapy that helped one subgroup, but did nothing for the others, even harming a few, would nevertheless show an improvement.
6. Many were excluded from the sample (your no.1)
7. They did not set up a control group, although they almost treated the SMC group as though it was one. Members of that group didn't receive the same attention of course, and if results are weak, a proper control group is essential to evaluate the placebo effect.
8. They trivialise adverse events.
9. The actual results were very poor: if you subtract the improvement made by the SMC group from the GET+SMC or CBT+SMC results, SMC in general outperforms GET or CBT. That could mean a large placebo effect: it could mean that having specialist medical attention prescribe drugs to help with pain/sleep is significant.
10. Subsequently we are told that it takes 7 or 8 patients to follow the therapy for a year for one to show a clinically significant improvement (not a return to near normality). That means that 6 or 7 have wasted their time. Sorry, that's a bit emotive ;-)

Hi Graham - I think this is great work and it would be very helpful to have a comprehensive list of the main flaws with PACE, which could then be fleshed out. Some comments from me:

• I'd have the weak results as the first point as this is the very best they can claim, then move on to the flaws that undermine even these weak results.
• 6MWT deserves a point of its own as the sole objective measure, with no gain here from CBT and only a small GET gain (and only 69% completed the test at 52 weeks).
• 'Normal' wasn't redefined but was a new post-hoc measure that overlapped with the trial definition of 'severe disabling fatigue'. Recovery, which was defined in the protocol, wasn't reported.
• as Angela says, APT was a pretty unusual version of pacing with it's 70% rule (though I don't think there's anything out of place about using a diary to monitor activity levels), which makes the pacing findings questionable.

Might be worth going through the Lancet letters to check for more points as Dolphin suggests, but I'm not volunteering!

 

[Angela Kennedy]

APT was a pretty unusual version of pacing with it's 70% rule (though I don't think there's anything out of place about using a diary to monitor activity levels), which makes the pacing findings questionable.

The issue with using a diary is only one when the practice is imposed by a therapist. Pacing in other illnesses tends to be just that: pacing oneself according to one's limitations. It is in this illness that people feel the need to tell sufferers how to pace! Very little (actually if any) research has been done on how people 'pace' in ME, although a lot of authoritative-sounding advice has sprung up (!) as the references show in the PACE article. Where has this '70%' rule actually come from, for example, or the idea you need to do less than optimum? How accurate IS that a description of how people manage to cope with Canadian defined ME/CFS?

Obviously, someone who is too ill to hold a pen and notebook/diary for long is going to find that activity overrides any hopes of 'pacing' oneself. I'm looking further into the Pesek/Jason/Taylor article on 'empirical investigation', but the idea that pacing is not to be recommended is a serious potential problem- bearing in mind seriously affected, Canadian criteria-type illness sufferers.

So there's various problems with the way PACE defined and applied their own notion of 'pacing' which could have serious harmful ramifications for people with Canadian defined ME/CFS, or Ramsay defined ME, or similar illness that might get them refused for research but as clinical patients have CBT/GET imposed on them.

 

[Dolphin]

[*]'Normal' wasn't redefined but was a new post-hoc measure that overlapped with the trial definition of 'severe disabling fatigue'. Recovery, which was defined in the protocol, wasn't reported.

They were pretty close to redefining normal. Certainly mean-1SD (the basis for normal fatigue/physical functioning in the final paper) was different:
http://www.biomedcentral.com/1471-2377/7/6

We will count a score of 75 (out of a maximum of 100) or more, or a 50% increase from baseline in SF-36 sub-scale score as a positive outcome. A score of 70 is about one standard deviation below the mean score (about 85, depending on the study) for the UK adult population [51,52].

As you know, in the final paper, this was 60.

For fatigue: http://www.biomedcentral.com/1471-2377/7/6

A positive outcome will be a 50% reduction in fatigue score, or a score of 3 or less, this threshold having been previously shown to indicate normal fatigue

In the final paper, it was 18 or less. One can't score 18 and have a bimodal score of 3 or less.

 

[Dolphin]

Don't forget wdb's wiki

Wiki Home Baseace Trial Criticisms
http://forums.phoenixrising.me/showwiki.php?title=Pace+Trial+Criticisms

This should be useful.

 

[oceanblue]

They were pretty close to redefining normal. Certainly mean-1SD (the basis for normal fatigue/physical functioning in the final paper) was different:
http://www.biomedcentral.com/1471-2377/7/6
As you know, in the final paper, this was 60.

What I meant was that 'within the normal range' was a new post-hoc measure applying to both fatigue and function being within these paramaters, and there isn't a direct equivalent of this in the protocol - so this measure wasn't redefined. They certainly used 'normal' with a reasonable threshold in the protocol. However, in both cases the formula, "mean +/- 1SD", was the same. the threshold difference came about because the final paper wrongly used the mean/SD from the whole adult population, not the working age one. I suppose this illustrates the problem of trying to summarise PACE flaws in a single line.

I also think the bigger issue was the use of a phrase like 'within the normal range' while omitting the predefined Recovery data, as this inevitably resulted in the wacko 'normal' figures being quoted as recovery figures in the media (as well as in the BMJ and the accompanying Lancet editorial). The media-savvy authors would have known this would happen.

 

[Graham]

Wiki Home Baseace Trial Criticisms
http://forums.phoenixrising.me/showwiki.php?title=Pace+Trial+Criticisms

This should be useful.

I have just looked at wdb's wiki, and sit here in awe of a superior being. Let's face it, mine was just a childish starting scribble. I'm going to have to up my game to play with you big boys and girls.

I like all your comments, but now two thought processes are milling around my head. The wiki summary is spot on - exactly the sort of thing that a wiki entry should be, and carries the sort of critical information that we want to get across, so how can we help wdp to make the most of it? How does Mark's idea sit in that context? The idea of 10 points sits well with a populist approach - shades of a Dummies' Guide (although I'd be wary of calling it that if we want doctors to read it): and we do need a populist approach to get our message across - one that attracts sympathy and understanding. But would it be useful enough to be worth the extra effort? How would we envisage using it? I am happy to put in the effort if it stood a chance of being effective (I have had ME since 1999, my son since 1989 - plenty of motivation there!)

I have looked at all the rejected PACE letters, but to be honest, I have picked more ideas than that from the comments on this thread of the forum. I find the level of your analysis, the precision of your comments, and the width of your knowledge quite overwhelming! Will flattery get me everywhere?

 

[Dolphin]

I have just looked at wdb's wiki, and sit here in awe of a superior being. Let's face it, mine was just a childish starting scribble. I'm going to have to up my game to play with you big boys and girls.

I like all your comments, but now two thought processes are milling around my head. The wiki summary is spot on - exactly the sort of thing that a wiki entry should be, and carries the sort of critical information that we want to get across, so how can we help wdp to make the most of it? How does Mark's idea sit in that context? The idea of 10 points sits well with a populist approach - shades of a Dummies' Guide (although I'd be wary of calling it that if we want doctors to read it): and we do need a populist approach to get our message across - one that attracts sympathy and understanding. But would it be useful enough to be worth the extra effort? How would we envisage using it? I am happy to put in the effort if it stood a chance of being effective (I have had ME since 1999, my son since 1989 - plenty of motivation there!)

I have looked at all the rejected PACE letters, but to be honest, I have picked more ideas than that from the comments on this thread of the forum. I find the level of your analysis, the precision of your comments, and the width of your knowledge quite overwhelming! Will flattery get me everywhere?

As this was replying to my post, I'll reply even though I know it was really for lots of people.

I have let my "homework" slide because of the PACE Trial - there is other (ME) stuff that I really should have been doing and need to get down to. Also, I have to admit that I haven't read the wdb wiki but hopefully I will - I think I've got stuck in the mode of reading and responding to messages in threads (I have some E-mails from a couple of people that I haven't read either on PACE that people know about). Anyway, basically, I'm not saying much in this message but that it's up to you and others how much time and effort you want to put into it if any. Only do what you want to. But probably few people in the future are going to read this whole thread so summaries are certainly welcome.

And well done for reading the letters to the editor that didn't make it.

Oh, and glad to have you as part of the "team".

 

[Graham]

Just a quick extra thought that came to me in the shower. WDB's wiki is an excellent summary of what is wrong with the PACE trial, so why do we want another? Wouldn't it be better to title the "10 point summary" along the lines of "Reasons why CBT and GET should not be used as a panacea for ME, and what should replace it"? We would use the PACE trial as the main basis for arguing the case, but be able to draw in other studies as evidence. Isn't that really what our aim should be - to dump the whole idea of just shunting off people with ME to therapy? Again, that approach could lend it to being assertive, analytical and dispassionate, whereas a destruction of the PACE trial must carry a negative tone. The "replace it" part could open a new can of worms, with all the unproven stuff around, but I was thinking more along the lines of simply ensuring that the illness really wasn't something else, and providing help with pain relief and sleeping problems.

I know most of you are more or less at your limits, so I would be happy to get this all going if you would be willing to give me advice and criticism along the way, and contribute pieces according to your energies and knowledge.

And thanks Dolphin for the welcome!

 

[Dolphin]

Prof Malcolm Hooper's complaint to the editor of The Lancet re The PACE Trial


It looks like Prof Malcolm Hooper has now published his critical analysis of the PACE Trial...


"REPORT: COMPLAINT TO THE RELEVANT EXECUTIVE EDITOR OF THE LANCET ABOUT THE PACE TRIAL ARTICLES PUBLISHED BY THE LANCET"
http://www.meactionuk.org.uk/COMPLAINT-to-Lancet-re-PACE.htm

Download it as a Word doc format here:
http://www.meactionuk.org.uk/COMPLAINT-to-Lancet-re-PACE.doc

I finally finished reading it (have been reading it on and off for 10 days or so).
I'm afraid I didn't make detailed summary notes. At this stage, I've heard some of the points before so I may biased in saying that what I found of most interest (IIRC) was the detailed points on the trial report in the Lancet [maybe from (Word document) page 35 (end of page) on]. Like this thread, I think a lot of the points could be summarised in a lot fewer words.

 

[Bob]

I finally finished reading it (have been reading it on and off for 10 days or so).
I'm afraid I didn't make detailed summary notes. At this stage, I've heard some of the points before so I may biased in saying that what I found of most interest (IIRC) was the detailed points on the trial report in the Lancet [maybe from (Word document) page 35 (end of page) on]. Like this thread, I think a lot of the points could be summarised in a lot fewer words.

Well, that's a coincidence... I finished reading it last night as well...

For me, I thought the strongest arguments were in explaining why the study does not apply to ME patients because of the inappropriate diagnostic criteria used for selecting patients.

I also noticed that it says (section 9, under 'clinically useful difference'), that in the physical function primary outcome measurements, neither of the differences measured for both CBT and GET were 'clinically useful'. Did we all reach that conclusion? I had thought it was just one of the physical function measurements which didn't meet 'statistical significance'. I'm afraid that I don't retain this sort of info very well, and I haven't checked my notes yet.

 

[oceanblue]

Well, that's a coincidence... I finished reading it last night as well...

For me, I thought the strongest arguments were in explaining why the study does not apply to ME patients because of the inappropriate diagnostic criteria used for selecting patients.

I also noticed that it says (section 9, under 'clinically useful difference'), that in the physical function primary outcome measurements, neither of the differences measured for both CBT and GET were 'clinically useful'. Did we all reach that conclusion? I had thought it was just one of the physical function measurements which didn't meet 'statistical significance'. I'm afraid that I don't retain this sort of info very well, and I haven't checked my notes yet.

I don't have that critique in front of me, but the mean difference between GET and SMC was 9.4, thereby exceeding the 'clinically useful difference' threshold of 8 points, wheras the mean difference between CBT and SMC was 7.1 ie below the clinically useful difference. However, neither CBT nor GET was above the 8 point threshold at 95% confidence levels (see figure 2), though the means themselves were both significantly different from SMC. Hope this helps.

 

[Bob]

I don't have that critique in front of me, but the mean difference between GET and SMC was 9.4, thereby exceeding the 'clinically useful difference' threshold of 8 points, wheras the mean difference between CBT and SMC was 7.1 ie below the clinically useful difference.

Yes, that's what I thought...

Here's the quote:

The relevant figures are presented in Table 3. A basic appraisal suggests that the differences between the average for CBT, and for GET, as compared with SMC on one of the two primary outcomes - physical function - were not clinically useful, at 7.4 points and 6.9 points, respectively.

 

[oceanblue]

Yes, that's what I thought...

A basic appraisal suggests that the differences between the average for CBT, and for GET, as compared with SMC on one of the two primary outcomes - physical function - were not clinically useful, at 7.4 points and 6.9 points, respectively

Odd, don't know where he gets those figures from - doesn't seem to be from the 'mean difference' line in Table 3.

 

[Esther12]

Odd, don't know where he gets those figures from - doesn't seem to be from the 'mean difference' line in Table 3.

From the way the Lancet's editor talked about it, I was worried there were errors in there. I can't say if this is the case myself though, as I've not looked through it.

 

[Bob]

From the way the Lancet's editor talked about it, I was worried there were errors in there. I can't say if this is the case myself though, as I've not looked through it.

But the biggest mistake was in publishing the paper in the first place. That was the Lancet editor's major mistake.
Not to mention the myriad of 'mistakes' in the methodology of the paper itself!
The editor of the Lancet seems to be pretty ignorant and uninformed about ME and science, so what he says is pretty meaningless.
He's backed himself into a corner, and seems to be saying all sorts to defend his indefensible position, including lashing out at the entire ME community.
Hooper's paper seems like a pretty heavyweight and substantial piece of work to me.
I think we can allow him one or two errors, if there are any, and I'm not aware of any yet (please see my next post.)

(Esther, if this post comes across as a bit ranty, it isn't aimed at you, but just at the Lancet and the authors!)

 

[Bob]

Odd, don't know where he gets those figures from - doesn't seem to be from the 'mean difference' line in Table 3.

Ah, they are a simple subtraction of the averages, rather than the adjusted mean difference.
So the paragraph is accurate.

 

[oceanblue]

Ah, they are a simple subtraction of the averages, rather than the adjusted mean difference.

I see. I'm pretty sure the right figures to use - in terms of statistical validity - are the ones comparing the difference in gains between baseline and 52 weeks, not the difference in values 52 weeks.