Former BMJ editor Richard Smith's blog post on PACE

[Sean]

The PACE authors will have known that CBT is not magic, from prior experience. They knew they were looking for a fairly modest effect.

The PACE authors expected (i.e. predicted) much larger effect sizes for CBT and GET than the data delivered. This size of this discrepancy is one of the most revealing facts in the whole saga.

(Outcome defined as % patients reporting improvement on the two primary measures.
UMC = Usual non-specialist Medical Care.
SSMC = Standard Specialist Medical Care – which they later contracted to SMC.
APT assumed equivalent in effect to relaxation and flexibility therapies.)

Expected Outcomes (as advantage over UMC = 6-17%)

CBT: 33-57% (possibly 33-63%)
GET: 1-57% (probably around 30-41%)
APT: 9-21%​

Measured Outcomes (1 year)

Fatigue (as advantage over SSMC = 65%)
CBT: 11%
GET: 15%
APT: 0%

Physical Function (as advantage over SSMC = 58%)
CBT: 13%
GET: 12%
APT: -9%​

Measured Outcomes (2.5 years)

Null result. (No statistically significant difference between trial arms. Plus no advantage from post-intervention addition of CBT and/or GET to APT or SSMC.)​

With the exception of a statistically but not clinically significant 6MWD result for the GET arm, they also failed to deliver anything on secondary objective measures at 1 year: 6MWD on the other 3 arms, Self-paced Step Test, and Client Service Receipt Inventory (employment participation, reliance on welfare or on disability/income insurance, total service usage and costs).

Clearly they seriously overestimated the therapeutic (and hence explanatory and predictive) power of the cognitive-behavioural model that PACE was testing, and the reliability of the data it was built on.

Why they did so is for them to answer.

http://bmcneurol.biomedcentral.com/articles/10.1186/1471-2377-7-6

Assumptions
The existing evidence does not allow precise estimates of improvement with the trial treatments. However the available data suggests that at one year follow up, 50 to 63% of participants with CFS/ME had a positive outcome, by intention to treat, in the three RCTs of rehabilitative CBT [18, 25, 26], with 69% improved after an educational rehabilitation that closely resembled CBT [43]. This compares to 18 and 63% improved in the two RCTs of GET [23, 24], and 47% improvement in a clinical audit of GET [56]. Having usual rather than specialist medical care allowed 6% to 17% to improve by one year in two RCTs [18, 25]. There are no previous RCTs of APT to guide us [11, 12], but we estimate that APT will be at least as effective as the control treatments of relaxation and flexibility used in previous RCTs, with 26% to 27% improved on primary outcomes [23, 26]. We propose that a clinically important difference would be between 2 and 3 times the improvement rate of SSMC.

 

[Jonathan Edwards]

I don't see why you need a Bonferoni adjustment if you way all criteria have to agree. The problem is when people have a number of outcomes and try to pick some to support their theory. If they all agree then that should reinforce a decision - if they don't agree then a careful examination should take place into whether outcomes are really independent or if there is just bias in certain variables.

You would not need an adjustment if you require consistency. But you would if you said 'we'll say it works if measure a or measure b or measure c shows a significant improvement, we don't mind which because they would all be good.'

I thought one of the claims for CBT over other talking therapies was that is was easier to reproduce reliably. I'm surprised that no one has done experiments to look at the reproducability claims and different skill levels and types of people giving the therapy. But then such experiments would need easy to measure effects so maybe very hard.

Maybe a certain eminent colleague of the PACE team should be reminded of that claim. I think some work has been done on how dependent results are on individual therapists. But I suspect that there is nothing conclusive. It might also be hard to know what it is about a particular therapist's delivery that 'worked' and even harder to simulate it.

 

[Jenny]

You would not need an adjustment if you require consistency. But you would if you said 'we'll say it works if measure a or measure b or measure c shows a significant improvement, we don't mind which because they would all be good.'



Maybe a certain eminent colleague of the PACE team should be reminded of that claim. I think some work has been done on how dependent results are on individual therapists. But I suspect that there is nothing conclusive. It might also be hard to know what it is about a particular therapist's delivery that 'worked' and even harder to simulate it.

I'm a bit out of date on all this, but when I was doing research in this field there was a lot of evidence to suggest that the benefits of therapy largely come down to the quality of the therapeutic relationship. So differences in technique and theoretical perspective have minimal effects on outcomes.

 

[Jonathan Edwards]

I'm a bit out of date on all this, but when I was doing research in this field there was a lot of evidence to suggest that the benefits of therapy largely come down to the quality of the therapeutic relationship. So differences in technique and theoretical perspective have minimal effects on outcomes.

An interesting comment, Jenny. It makes me wonder how one measures the quality of a relationship? And since it seems doubtful whether or not there are any benefits from CBT I am not sure what there would be to correlate with.

You may be right that the quality of the personal relationship is all that really matters and not the technique or theory. But then why call it CBT and make special claims for it? And what distinguishes the quality of a personal relationship from the quality of a therapeutic relationship? Couldn't a dedicated friend do as well?

I don't know the answer to any of these questions but I would be interested any what evidence there is for this sort of thing.

 

[Jenny]

An interesting comment, Jenny. It makes me wonder how one measures the quality of a relationship? And since it seems doubtful whether or not there are any benefits from CBT I am not sure what there would be to correlate with.

You may be right that the quality of the personal relationship is all that really matters and not the technique or theory. But then why call it CBT and make special claims for it? And what distinguishes the quality of a personal relationship from the quality of a therapeutic relationship? Couldn't a dedicated friend do as well?

I don't know the answer to any of these questions but I would be interested any what evidence there is for this sort of thing.

Here is a rather old abstract which reviews research into the relationship between outcomes and the working relationship (sometimes referred to as the working alliance).

Research summary on the therapeutic relationship and psychotherapy outcome.
By Lambert, Michael J.; Barley, Dean E.
Psychotherapy: Theory, Research, Practice, Training, Vol 38(4), 2001, 357-361.
Abstract
Factors that influence client outcome can be divided into four areas: extratherapeutic factors, expectancy effects, specific therapy techniques, and common factors. Common factors such as empathy, warmth, and the therapeutic relationship have been shown to correlate more highly with client outcome than specialized treatment interventions. The common factors most frequently studied have been the person-centered facilitative conditions (empathy, warmth, congruence) and the therapeutic alliance. Decades of research indicate that the provision of therapy is an interpersonal process in which a main curative component is the nature of the therapeutic relationship. Clinicians must remember that this is the foundation of our efforts to help others. The improvement of psychotherapy may best be accomplished by learning to improve one's ability to relate to clients and tailoring that relationship to individual clients. (PsycINFO Database Record (c) 2012 APA, all rights

...............................................

The quality of the working alliance is usually measured using a self-report inventory, completed by therapist and/or client.

As the abstract says the key factors are often seen to be empathy, warmth and congruence (or genuineness) (all these have quite precise definitions), but more recent work has suggested others.

A dedicated friend would have to be quite special to be consistently helpful.

What special claims are you referring to that CBT has over and above other forms of therapy? One of the main advantages claimed by government in its Improving Access to Psychological Therapies programme is that it can be brief and therapists don't need a lot of training (so it's cheap).

 

[Jonathan Edwards]

To be honest, that does not look like reliable evidence, @Jenny.

I suspect a session is scored highly on these things if the patient is improving, and less often if they are still unwell - i.e. the correlation is self-fulfilling. What if the therapist thinks they were warm and empathetic and the patient disagrees?

Why would a dedicated friend have to be more special than a therapist if warmth and empathy are what matter? Surely a dedicated friend would be likely to score higher than a therapist on those? When my wife was psychotic I was the only person who was exempt from her paranoia. Although her illness made her believe that everyone, and especially all doctors and therapists, were conspiring against her it seemed to stop at making her believe I was against her. But that did not mean that even I could make her better. When she recovered with ECT she began to think her psychiatrist was warm and empathetic. In fact the first person she believed was not against her was the ECT specialist.

So from a scientific point of view I am sceptical about this sort of 'evidence'.

The special claim made for CBT in ME/CFS is that it is effective therapy, whereas there is no such claim for any other form of therapy or support from dedicated friends. The suggestion that it does not need much training does not seem to be the view of the person most associated with use of CBT in ME/CFS.

 

[Jenny]

To be honest, that does not look like reliable evidence, @Jenny.

I suspect a session is scored highly on these things if the patient is improving, and less often if they are still unwell - i.e. the correlation is self-fulfilling. What if the therapist thinks they were warm and empathetic and the patient disagrees?

Why would a dedicated friend have to be more special than a therapist if warmth and empathy are what matter? Surely a dedicated friend would be likely to score higher than a therapist on those? When my wife was psychotic I was the only person who was exempt from her paranoia. Although her illness made her believe that everyone, and especially all doctors and therapists, were conspiring against her it seemed to stop at making her believe I was against her. But that did not mean that even I could make her better. When she recovered with ECT she began to think her psychiatrist was warm and empathetic. In fact the first person she believed was not against her was the ECT specialist.

So from a scientific point of view I am sceptical about this sort of 'evidence'.

The special claim made for CBT in ME/CFS is that it is effective therapy, whereas there is no such claim for any other form of therapy or support from dedicated friends. The suggestion that it does not need much training does not seem to be the view of the person most associated with use of CBT in ME/CFS.

Yes I agree. There's been a lot of criticism of this sort of research, and I'm not particularly defending it. In longitudinal studies though, the quality if the relationship early on, even when the patient isn't improving, predicts outcome. But the main point is that when you compare outcomes across different modes of therapy, a lot of the research suggests that type of therapy doesn't make much difference. (That's over-simplifying a bit though as it depends on the condition being treated - there is some evidence that mild depression is helped more by CBT.)

As Eysenck showed in the 1950s, two thirds of patients with psychiatric conditions improve spontaneously, with no therapy, so this has to be taken account of too. It gets complicated. Just from personal experience though I do think the quality of the relationship has some small part to play.

I think dedicated friends are paradoxically often too close to the patient to help much (for example I'm not good at helping my husband with his bouts of depression as It's difficult to hide my frustration when his illness has such an effect on my life), but not close enough in that it can be quite difficult to feel and express empathy rather than sympathy. And it's sometimes hard not to slip into advice mode.

 

[jimells]

As Eysenck showed in the 1950s, two thirds of patients with psychiatric conditions improve spontaneously, with no therapy, so this has to be taken account of too.

By "therapy" do you mean any form of treatment?

 

[Jenny]

By "therapy" do you mean any form of treatment?

Can't remember what forms of treatment he was looking at, but it probably was a range. CBT wasn't around then though.

 

[Jonathan Edwards]

Yes I agree. There's been a lot of criticism of this sort of research, and I'm not particularly defending it. In longitudinal studies though, the quality if the relationship early on, even when the patient isn't improving, predicts outcome. But the main point is that when you compare outcomes across different modes of therapy, a lot of the research suggests that type of therapy doesn't make much difference. (That's over-simplifying a bit though as it depends on the condition being treated - there is some evidence that mild depression is helped more by CBT.)

As Eysenck showed in the 1950s, two thirds of patients with psychiatric conditions improve spontaneously, with no therapy, so this has to be taken account of too. It gets complicated. Just from personal experience though I do think the quality of the relationship has some small part to play.

I think dedicated friends are paradoxically often too close to the patient to help much (for example I'm not good at helping my husband with his bouts of depression as It's difficult to hide my frustration when his illness has such an effect on my life), but not close enough in that it can be quite difficult to feel and express empathy rather than sympathy. And it's sometimes hard not to slip into advice mode.

I am very ready to believe all of that. Which seems to indicate that clinical psychological science may not have caught up with common sense yet!

 

[Large Donner]

I am very ready to believe all of that. Which seems to indicate that clinical psychological science may not have caught up with common sense yet!

There's a team of BPS experts this very minute who have just run off to design a study to measure the effectiveness of common sense.

They have however created a new term called "applied common sense therapy", the results or which are expected in five years once they receive the check for five million pounds and have found a lock up box secure enough to keep the data away from common sense activists.

They are expected to confirm that common sense should be applied upon people by external experts and duly funded by the state.

 

[charles shepherd]

As it is now over a month since I asked for an apology and retraction regarding an inaccurate accusation about the MEA I have posted this reply today

It is awaiting moderation

Dear Dr Smith
It is now over a month since I sent a reply to this blog item - see below

I pointed out that your opening sentence:

'Several times when I was the editor of The BMJ the journal was declared the worst medical journal in the world by the ME (Myalgic Encephalomyelitis) Association.'

was inaccurate and therefore requested an apology and retraction for what is a derogatory statement about the MEA.

I assume that you read your own blog and the comments it producesI am therefore surprised and disappointed that you have not responded

A prompt response would now be appreciated

If not, I will pursue an apology and retraction through other routes

Dr Charles Shepherd

Hon Medical Adviser, MEA


Five quick points:

1 Richard Smith is mistaken when he states that The ME Association (MEA) has 'several times' described the BMJ as the 'worst medical journal in the world'

Yes, The MEA has been very critical for a considerable period of time on the way in which the BMJ has covered the subject of ME/CFS, and the choice of people it has commissioned to provide material and editorial comment on the illness. But we have never used this description.

In fact, I was very happy to ask Fiona Godlee, currently editor in chief at the BMJ, to chair the meeting at the Royal Society of Medicine on controversies surrounding ME/CFS that I helped to organise earlier this year:

http://www.meassociation.org.u...

I think an apology would therefore be appropriate

2 Having worked in hospital psychiatry, I know that mental illness is just as real and horrible as physical illness.

Our criticism is with the way in which a group of mainly psychiatrists have put forward a largely psychosocial model of causation and management (ie abnormal illness beliefs and behaviours associated with deconditioning treated with cognitive behaviour therapy/CBT and graded exercise therapy/GET) that is flawed; is not helping a large proportion of patients (in the case of CBT); makes a significant proportion worse (in the case of GET); and has for a long period of time dating back to the early 1980s deflected scientific interest and funding into research aimed at understanding the biomedical aspects of what is undoubtedly a wide spectrum of conditions involving both clinical presentations and disease pathways

To quote from my recent article in the Daily Telegraph:

'The argument here is not with mental illness, which is just as real and horrible as physical illness. As with any long-term illness, some people will develop mental health problems where talking therapies can clearly be of help.

The argument is with a simplistic and seriously flawed model of causation that patients know is wrong and which has seriously delayed progress in understanding the underlying cause of ME and developing effective forms of treatment.

Opening the 2015 research collaborative section of neuropathology, Jose Montoya, professor of medicine at the University of Stanford, said: “I have a wish and a dream that medical and scientific societies will apologise to their ME patients."

I agree – the time has come for doctors and scientists to apologise for the very neglectful way in which ME has been researched and treated over the past 60 years. Doctors need to start listening to their patients and there must now be increased investment in biomedical research to gain a better understanding of the disease process and to develop treatments that these patients desperately need.

Full article: http://www.telegraph.co.uk/new...

3 Abuse and harassment (which I have personal experience of as well) has no role to play in this debate and I have been forthright in my condemnation of people who use this tactic

4 The paper just published in Lancet Psychiatry, which examines data on long term outlook from the PACE trial, found very little difference between CBT, GET and Adaptive Pacing

If you want to look at patient evidence (1428 respondents) on these three approaches to management please read the summary of our latest MEA report:

http://www.meassociation.org.u...

5 I welcome the fact that Richard Smith is making it very clear that the research groups involved are making a serious mistake in refusing to release the PACE trial data that is being requested and that they are going against basic scientific principles

Given the growing strength of criticism on this issue from patients, clinicians and academics it looks as though the the Canutes at King's College would be wise to stop digging and give way to what is a perfectly reasonable request

Dr Charles Shepherd

Hon Medical Adviser, MEA

 

[charles shepherd]

As it is now over a month since I asked for an apology and retraction regarding an inaccurate accusation about the MEA I have posted this reply today

It is awaiting moderation

Dear Dr Smith
It is now over a month since I sent a reply to this blog item - see below

I pointed out that your opening sentence:

'Several times when I was the editor of The BMJ the journal was declared the worst medical journal in the world by the ME (Myalgic Encephalomyelitis) Association.'

was inaccurate and therefore requested an apology and retraction for what is a derogatory statement about the MEA.

I assume that you read your own blog and the comments it producesI am therefore surprised and disappointed that you have not responded

A prompt response would now be appreciated

If not, I will pursue an apology and retraction through other routes

Dr Charles Shepherd

Hon Medical Adviser, MEA

Dear Charles,
Many thanks for your comment to Richard Smith's blog. I have alerted him to it. I expect that he will reply shortly.
Best wishes,
Juliet


--
Juliet Dobson
Web editor and blogs editor

 

[Esther12]

Dear Charles,
Many thanks for your comment to Richard Smith's blog. I have alerted him to it. I expect that he will reply shortly.
Best wishes,
Juliet


--
Juliet Dobson
Web editor and blogs editor

Did you ever get a response from Richard Smith to this?

 

[Bob]

Did you ever get a response from Richard Smith to this?

Richard Smith issued an apology and corrected the article. See the bottom of the article:
http://blogs.bmj.com/bmj/2015/12/16...lege-should-release-data-from-the-pace-trial/