• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

The PACE Trial – The Results

currer

Senior Member
Messages
1,409
It is shocking that once the statistics are clearly interpreted, PACE, from being an endorsement of CBT and GET as "treatments" for ME, becomes a proof of the opposite.

PACE CLEARLY SHOWS THAT CBT AND GET DO NOT WORK.

It is only the distortion of their own statistics that allowed the PACE authors to claim the opposite.

We must ensure that this messsage gets out to all the ME communities and advocates.
 

currer

Senior Member
Messages
1,409
Most people do not have time to do a sophisticated check of all the maths ina research paper. We rely on the authors and the peer review process to do this.

Now PACE did not get a proper peer review before it was published. If it had the inadequacy of its statistical interpretation would have come to light.

Bob has done a great job in "peer reviewing" this paper. it looks as if no-one else has been able to see what has been done to the statistices to enable the false conclusions of this paper to be drawn.

I think Professor White must have some awareness of this because of his statement last year that the PACE trial was not investigating ME as "operationally defined"

Can anyone else find this quote?
 

currer

Senior Member
Messages
1,409
Could the inadequacy of the PACE results, and the dubiety of its conclusions (if these were recognised by the authors) be the reason PACE did not go through a proper PEER review?

And yet its conclusions have been trumpeted all over the press and used to undermine our status as deserving proper medical recognition and welfare benefits.
 

jace

Off the fence
Messages
856
Location
England
Here's the link to the correspondence between White and Hooper, with the The PACE trial paper refers to chronic fatigue syndrome (CFS) which is operationally defined; it does not purport to be studying CFS/ME” and CFS defined simply as a principal complaint of fatigue that is disabling, having lasted six months, with no alternative medical explanation (Oxford criteria)” - http://www.meactionuk.org.uk/Hoopers-initial-response-to-PDW-letter.htm

Yet despite the cohort in PACE were unlikely to have a neurological disease as defined in G93.3, they still did not show any statistically significant improvements....

ETA How many harms were caused by the protocols being tested in PACE is key. If we could say "13% are said to have improved (yet the improvement was so small that the 'improvers' were possibly still sick enough that they still qualified to enter the study) but (say) 46% became more ill... I do hope the FOI gets answered.
 

currer

Senior Member
Messages
1,409
If the real statistical findings of the PACE trial were understood, PACE could not be used to justify the current NICE guidelines.

In fact NICE (with regard to ME) would have to be changed.
 

Shell

Senior Member
Messages
477
Location
England
The PACE trial paper refers to chronic fatigue syndrome (CFS) which is operationally defined; it does not purport to be studying CFS/ME” and CFS defined simply as a principal complaint of fatigue that is disabling, having lasted six months, with no alternative medical explanation (Oxford criteria)

Unlerlined and highlighted by me. I don't undertstand that part of the sentence. Did all or most of the subjects have a dx of depression? I was under the impression they did. For what we used to call endogenous depression, which was a severe organic based depression without psychoses, fatigue was/is a major symptom. Those patients with EnD did not respond well to any talking therepies because their depression was caused by the way their brains were not taking up seretonim and something to do with dopermine (can't remember, it's been a long time since I was a psychi nurse).

Even with reactive depression CBT was not a one-size-fits-all approach. We used different councelling methods depending on patient need (old fashioned view isn't it?)

It was understood that exercise and relaxation therepies could help people with lots of different kinds of depression as it helped boost seretonim. Even so, the results were mixed.
All this was known and documented back before I was training (started 1986).
So Wessley n White should have KNOWN the outcomes wouldn't be that good.

Who were these fatigued people without depression? And if they had depression in what way was their fatigue different? And how did they differentiate the symptoms if ME wasn't in the mix?
 

currer

Senior Member
Messages
1,409
Unlerlined and highlighted by me. I don't undertstand that part of the sentence. Did all or most of the subjects have a dx of depression? I was under the impression they did. For what we used to call endogenous depression, which was a severe organic based depression without psychoses, fatigue was/is a major symptom. Those patients with EnD did not respond well to any talking therepies because their depression was caused by the way their brains were not taking up seretonim and something to do with dopermine (can't remember, it's been a long time since I was a psychi nurse).

Even with reactive depression CBT was not a one-size-fits-all approach. We used different councelling methods depending on patient need (old fashioned view isn't it?)

It was understood that exercise and relaxation therepies could help people with lots of different kinds of depression as it helped boost seretonim. Even so, the results were mixed.
All this was known and documented back before I was training (started 1986).
So Wessley n White should have KNOWN the outcomes wouldn't be that good.

Who were these fatigued people without depression? And if they had depression in what way was their fatigue different? And how did they differentiate the symptoms if ME wasn't in the mix?

This is a very good point, shell.

As I understand it, CBT does not help organic depression. And yes, as psychiatrists they would know this.
One needs to question again the validity of the papers that make large claims for CBT and depression.

The Oxford criteria used for the PACE trial do not distinguish adequately between other idiopathic fatigue states and ME.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Bob has done a great job in "peer reviewing" this paper. it looks as if no-one else has been able to see what has been done to the statistics to enable the false conclusions of this paper to be drawn.

Thanks currer. I'd just like to highlight, again, that the information contained in my posts really has been a huge ongoing community effort. The reason that I haven't named anyone, is because not everyone involved is a member of this forum, but also, if I were to try to create a list of names, I would be sure to unfairly miss some people out, and also I expect the list would be very long. Anyone interested in seeing who was involved, from this forum, should have a quick look through the enormously long PACE Trial thread:
http://forums.phoenixrising.me/index.php?threads/pace-trial-and-pace-trial-protocol.3928/

And then, in case anyone missed it, and wants to read a deeper analysis, there's Phoenix Rising's excellent analysis:
ME Analysis - Evaluating the results of the PACE study
http://evaluatingpace.phoenixrising.me/homepageanim.html
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
It is shocking that once the statistics are clearly interpreted, PACE, from being an endorsement of CBT and GET as "treatments" for ME, becomes a proof of the opposite.

Unfortunately, there are many questionable aspects of the methodology, but it's very difficult to officially challenge methodologies that are weak, but often scientifically acceptable, especially when they have been published in the Lancet. So I've just stuck to the basics of their results in these posts. These are quite revealing on their own, aren't they!

In the protocol, a measure for a 'positive outcome' was proposed, but dropped before publication. It would be good to get that sort of analysis released under a FOI request, as well as the deterioration rates, because there's not much doubt that the proposed primary results would be even worse than a response rate of 13%.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
This post is intended to explain the actual primary results of the PACE Trial (as opposed to the spun results.)
All questions, comments, feedback and corrections are very welcome.
Hi Bob

Great work on all of this. Afraid I've only just got round to printing it out and readling it all, but it's a very good summary of the PACE trial and the way it's been misrepresented. Thank you.

I'd like to make some constructive suggestions too; inevitably these take up more space than saying 'a brilliant job', which it is, but they might help make what you've done even better. Some of the comments are about making things clearer, others in making things persuasive to those who have already accepted the 'official' PACE line.

Primary Results/Improvement rates:
I think the 'Number Needed to Treat' data is exactly the right way to present the key results of PACE, as it shows how many people have to be treated for just one patient to improve by a modest amount. Curiously, Michael Sharpe used these statistics when asked to explain the results of the PACE trial, giving a very different picture to that of the Official launch.

To simplify: I would just use NNT for those improving in both fatigue and function as this is probably the most important measure, and not bother with separate NNTs for fatigue, function. I think it would also help if you first explain what a 'Clinically Useful Difference' is, either using the 0.5 SD definition or just saying the 'minimum useful improvement' in both fatigue and function. Then present the data.

Clinical Effectiveness /Therapeutic Effect Size:
It might help just to explain that while the 'primary' results above are categorical - i.e. what proportion of patients reach a threshold, these relate findings relate to continuous measures i.e. how much the average patient score improved.[/quote]

Presenting the data in a neutral way
Maybe it's just me, but I find evidence far more persuasive when it's presented in a straightforward way, without pushing the data to appear truly dramatic. The PACE results are very poor when viewed this way, which is why I'm not comfortable hyping the findings a little:
This means that approximately 87% of patients did not benefit from CBT or GET, the only treatments recommended for ME by NICE.

By contrast, 58% (physical function) and 65% (fatigue) of the SMC group achieved a clinically useful outcome.
Technically, this is more or less accurate, but it does give a slightly skewed picture, which I will try to illustrate with an example. Let's say 60% of the SMC group improved and instead of GET/CBT, the trial was of NewWonderDrug. And let's say 95% of patients on NewWonderDrug improved, a pretty decent result. The approach in quotes would present that as "65% of patients did not benefit from NewWonderDrug/Only 35% improved. By contrast 60% of the SMC group showed a clinically useful outcome". I'm not sure that's a sound way of presenting the data.

I still think the NNT figures are the fairest way of presenting the data. Sure, the PACE authors tried to present the results in a way that wasn't fair or accurate but I don't think we should follow their lead on this. The naked results are powerful enough on their own, in my view.

For similar reasons, I think describing fatigue as a 'self-reported subjective' symptom is over-egging things a bit. In contrast to physical function, there is no objective measure of fatigue, whoever is doing the experiment. I think it's appropriate to point out once that it is a self-reported measure and so liable to reporting bias, but to continually label a subjective phenomenom as subjective might irritate some people you want to influence. Again, I think the argument is strong enough without embelishment.

That's it. I think you've done a fantastic job here, and it must have been a huge amount of work. I just think if you rein it in a little in a couple of places it will be even more persuasive. 'Less is more'.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Just to illustrate that the authors are fully aware of the actual results, for those who haven't looked through my references, this is one of the references that I included in my first post:

Michael Sharpe on ABC National Radio, The Health Report.
Comparison of treatments for chronic fatigue syndrome - the PACE trial.
http://www.abc.net.au/rn/healthreport/stories/2011/3192571.htm#transcript
“We have a number needed to treat; I think it's about seven to get a clinically important treatment benefit with CBT and GET. What this trial isn't able to answer is how much better are these treatments than really not having very much treatment at all.”

So, Michael Sharpe, in perhaps an unguarded moment, states that the 'number needed to treat' (NNT) is 1 in 7.
That means that 7 patients need to be treated for a clinical benefit to be seen in 1 patient.

Obviously, this is very different to the "30%" figure promoted in the media, which would give a NNT of approx 1 in 3.

I think Sharpe is refering to the secondary post-hoc analysis: Participants who acheived a CUD in both primary outcome measures.
This secondary post-hoc analysis puts the results in a slightly more favourable light than the primary results, which give an average NNT of 1 in 8 patients.

So, to be clear, the primary results give a NNT of 1 in 8. (13% response rate)
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Bob, what are the chances of getting your original posts on this thread independently checked? Not that I don't trust them of course, but it could prove quite useful to have them semi-officially verified before they are used - for instance - at the occasion of the NICE Review in (I think) August next year (2013). Would take dosh though I suppose unless one or all of the ME Charities are interested in funding something. Only I see it has now been splashed around the internet and I am a 'tad' concerned that without independent verification it could be spurned (if you see what I mean).
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Bob, what are the chances of getting your original posts on this thread independently checked? Not that I don't trust them of course, but it could prove quite useful to have them semi-officially verified before they are used - for instance - at the occasion of the NICE Review in (I think) August next year (2013). Would take dosh though I suppose unless one or all of the ME Charities are interested in funding something. Only I see it has now been splashed around the internet and I am a 'tad' concerned that without independent verification it could be spurned (if you see what I mean).

It's a good question Firestorrm.
However, there's not really very much to check or verify, as I've only presented the most basic/fundamental primary results, which are presented in the paper itself. (Although, they are a little hidden in the data tables, and not clarified in the text or the discussion.)

I haven't really challenged their published data (as opposed to the spin), or questioned much of the methodology, or questioned their statistics.
I've questioned the 'normal range' analysis, but that was only a post-hoc analysis anyway. It's not a primary outcome analysis, and wasn't intended to demonstrate the effectiveness of the treatments. So for that analysis to be found wanting, doesn't really change much.

It would be very useful to get a critique published in a scientific journal, and maybe some of us could work towards that, but I'm out of my depth when it comes to getting anything published. I wouldn't know where to start.

I think that political leverage is also very important, so it would be helpful if the patient organisations are aware of these results.
Posting these results was just intended to bring people's attention towards the most basic of the results.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Hi Bob

Great work on all of this. Afraid I've only just got round to printing it out and readling it all, but it's a very good summary of the PACE trial and the way it's been misrepresented. Thank you.

I'd like to make some constructive suggestions too; inevitably these take up more space than saying 'a brilliant job', which it is, but they might help make what you've done even better. Some of the comments are about making things clearer, others in making things persuasive to those who have already accepted the 'official' PACE line.

Thanks very much for the feedback Simon. Much appreciated.
I'll consider it in detail, and respond to it, a bit later.
 

biophile

Places I'd rather be.
Messages
8,977
Calculating the NNT is simple to confirm. The basic formula is:

NNT = 100 / ( [ % responding in therapy group ] minus [ % responding in control group ] ).

The result is often then rounded up to the nearest whole number.

However, if you want a published reference for NNT in PACE, this can be used instead:

Primary outcomes were self-reported assessments of fatigue (Chalder fatigue questionnaire) and physical function (short form-36 physical function subscale). Clinically meaningful improvement in both fatigue and function scores were achieved by 61% of participants in the graded exercise group and 59% in the CBT group, compared to 42% of subjects in the adaptive pacing group and 45% in the specialized care only group. Compared to specialized care only, the NNT was 7 for graded exercise (P = .004) and 8 for CBT (P = .015).

Glickman-Simon R, Tenkku LE. Carpal tunnel syndrome, low back pain, chronic fatigue syndrome, preventing preeclampsia, preventing complex regional pain syndrome. Explore (NY). 2012 Jan;8(1):65-7. PMID:22828246 http://www.thblack.com/links/RSD/EXPLORE2012_8_65_preventingCRPS.pdf

This can be represented in the following illustration (not from above source):

NNT-CUD.png


Although technically correct, I'm not 100% comfortable with it, because someone can still point out it obscures the likelihood that the participants who achieved CUD threshold for both fatigue and physical function in the CBT (59%) and GET (61%) groups improved more than those who achieved CUD threshold in the SMC group (45%), which would explain the group average differences whereas NNT x CUD does not. That said, the additional improvements over SMC would be minimal and White et al showed no apparent discomfort during their promotion of the dubious "normal range".

Keep in mind that the authors' own estimates in the original protocol had implied that the NNT would be 2 for CBT and 2.5 (3) for GET, the "response" goalposts equate to 9-14/33 points on CFQ and 20-30/100 points on physical function SF-36. The difference between expectations and outcomes are obvious, and the best phrase to use when this happens in research is that the "truth is wearing off" for CBT and GET (when larger better conducted studies yield much less impressive results compared to earlier smaller studies).

PS to Bob: All reported primary outcomes and response rates in fatigue and physical function were post-hoc too, not just "normal range", which itself had a NNT of 7 for CBT and 8 for GET.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
The PACE trial paper refers to chronic fatigue syndrome (CFS) which is operationally defined; it does not purport to be studying CFS/ME” and CFS defined simply as a principal complaint of fatigue that is disabling, having lasted six months, with no alternative medical explanation (Oxford criteria)



Unlerlined and highlighted by me. I don't undertstand that part of the sentence. Did all or most of the subjects have a dx of depression? I was under the impression they did. For what we used to call endogenous depression, which was a severe organic based depression without psychoses, fatigue was/is a major symptom. Those patients with EnD did not respond well to any talking therepies because their depression was caused by the way their brains were not taking up seretonim and something to do with dopermine (can't remember, it's been a long time since I was a psychi nurse).

Even with reactive depression CBT was not a one-size-fits-all approach. We used different councelling methods depending on patient need (old fashioned view isn't it?)

It was understood that exercise and relaxation therepies could help people with lots of different kinds of depression as it helped boost seretonim. Even so, the results were mixed.
All this was known and documented back before I was training (started 1986).
So Wessley n White should have KNOWN the outcomes wouldn't be that good.

Who were these fatigued people without depression? And if they had depression in what way was their fatigue different? And how did they differentiate the symptoms if ME wasn't in the mix?



33% of participants had "any depressive disorder", but bi-polar depressive patients were excluded from the Trial.

47% of participants were said to have "any psychiatric disorder", which included: depressive disorder and any anxiety disorder, including phobias, obsessive-compulsive disorder, and post-traumatic stress disorder. The following psychiatric disorders were excluded: psychosis, bipolar disorder, substance misuse, an organic brain disorder, or an eating disorder.

67% of participants were said to fit the Reeves et al 2003 criteria for CFS.

The entry criteria required 'fatigue' as the 'principle' symptom, and so any ME patients, who experience primary complaints other than 'fatigue', would be excluded from the trial.

The secondary outcome measures assessed the participants for some issues other than fatigue and physical function, such as: sleep; post-exertional malaise; and poor concentration and memory.



The PACE trial paper refers to chronic fatigue syndrome (CFS) which is operationally defined; it does not purport to be studying CFS/ME” and CFS defined simply as a principal complaint of fatigue that is disabling, having lasted six months, with no alternative medical explanation (Oxford criteria)

Yes, where it says 'with no alternative medical explanation' that does seem a bit slippery, because patients with depression, anxiety, post-traumatic stress disorder, phobias, and OCD were included in the Trial. However, they did all have to have a principle complaint of fatigue.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Calculating the NNT is simple to confirm. The basic formula is:

NNT = 100 / ( [ % benefiting from therapy ] minus [ % benefiting from control ] ).

The result is often then rounded up to the nearest whole number.

However, if you want a published reference for NNT in PACE, this can be used instead:

Primary outcomes were self-reported assessments of fatigue (Chalder fatigue questionnaire) and physical function (short form-36 physical function subscale). Clinically meaningful improvement in both fatigue and function scores were achieved by 61% of participants in the graded exercise group and 59% in the CBT group, compared to 42% of subjects in the adaptive pacing group and 45% in the specialized care only group. Compared to specialized care only, the NNT was 7 for graded exercise (P = .004) and 8 for CBT (P = .015).

Glickman-Simon R, Tenkku LE. Carpal tunnel syndrome, low back pain, chronic fatigue syndrome, preventing preeclampsia, preventing complex regional pain syndrome. Explore (NY). 2012 Jan;8(1):65-7. PMID:22828246
http://www.thblack.com/links/RSD/EXPLORE2012_8_65_preventingCRPS.pdf

I'd like to explain to everyone why the secondary results outlined in the above quote are slightly difference to the primary results that I have outlined.

I have presented the results intended to be the primary measure of clinical effectiveness. (Except that the originally intended measure of a 'positive outcome', and a 'clinically important difference' were dropped in favour of a post-hoc definition of a CUD.)

There was also a "secondary", "post-hoc" analysis included in the published paper, which analysed the results for participants who achieved a CUD for both primary outcomes. This placed the therapies in a slightly more favourable light, and it is one of the analyses that the authors seem to prefer promoting. This secondary analysis is outlined in the above quote.

I have stuck to the primary outcome results, for the sake of simplicity, clarity and rigour. Also, the primary results helpfully differentiate between fatigue and disability, which is informative.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
It's a good question Firestorrm.
However, there's not really very much to check or verify, as I've only presented the most basic/fundamental primary results, which are presented in the paper itself. (Although maybe they are a little hidden in the data tables, and not clarified in the text or the discussion.)

I haven't really challenged their published data, or questioned much of the methodology, or questioned their statistics.
I've questioned the 'normal range' analysis, but that was only a post-hoc analysis anyway. It's not a primary outcome analysis, and wasn't intended to demonstrate the effectiveness of the treatments. So for that analysis to be found wanting, doesn't really change much.

It would be very useful to get a critique published in a scientific journal, and maybe some of us could work towards that, but I'm out of my depth when it comes to getting anything published. I wouldn't know where to start.

I think that political leverage is also very important, so it would be helpful if the patient organisations are aware of these results.
Posting these results was just intended to bring people's attention towards the most basic of the results.

Thanks.

As I said to you just now, I wouldn't think it was hard to get this on all/some of the ME Charities' agendas and maybe go for an endorsement. Might prove a useful document to challenge the perception of these therapies and the way in which they will (I suspect) be presented at the NICE Review next year. I'm all for patients being afforded all the relevant information in order to make an informed choice. It was ever the way in which PACE results were marketed that sent shivers down my spine. Its NICE to see that misinformation being smacked in the mouth especially with their own (hidden) figures. I do like that approach. Yin and Yang old chap :)
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Thanks.

As I said to you just now, I wouldn't think it was hard to get this on all/some of the ME Charities' agendas and maybe go for an endorsement. Might prove a useful document to challenge the perception of these therapies and the way in which they will (I suspect) be presented at the NICE Review next year. I'm all for patients being afforded all the relevant information in order to make an informed choice. It was ever the way in which PACE results were marketed that sent shivers down my spine. Its NICE to see that misinformation being smacked in the mouth especially with their own (hidden) figures. I do like that approach. Yin and Yang old chap :)

Thanks Firestormm.

Yes, there does seem to be some justice in using the results of the PACE Trial to discredit their hypothetical model of illness, doesn't there!

I agree that the patient organisations need to be on board with this. I don't know how much they currently understand about the PACE Trial results but I've never seen them explaining the results to their patients.

I think I will do a bit more work on what I've written (just tidying it up a bit), and then maybe send it around.