A cost effectiveness of the PACE trial

[Dolphin]

Here's the question on informal care. As one can see, it was filled in by the participant, not a family member or friend.

In the last 6 months, have you received help from friends or relatives on any of the following tasks, as a consequence of your fatigue?

Type of help/Please circle one/Average number of hours help per week/Who provides this care?/Do they live in your house?

Child Care No Yes (circle No if you have no children)

Personal care (E.g. washing, dressing etc.) No Yes

Help in & around the house (E.g. cooking, cleaning etc.) No Yes

Help outside the home (E.g. shopping, transport etc.) No Yes

Other No Yes

Total hours

This was asked before the trial started, at 24 weeks and 52 weeks.
When one sees a figure of x number of pounds, it seems very exact. However, I would imagine people don't necessarily answer these questions that accurately - they can involve picking figures out of the air a bit. One could imagine that little biases could easily influence somebody from say saying they got 8 hours vs 10 hours help a week on average (or whatever).

I don't know whether referring specifically to "fatigue" as the cause for the need for help could affect results at all? One might think the other option is CFS (or ME) but there is also simply not specifying the illness and see how much help people got.

Again, as I think it was pointed out before, it comes back to actometers - it would have been more interesting to see their results. As well as possible inaccurate reporting, we also don't know whether people have lowered their standards/been more reluctant to look for help or, as Sam Carter suggested, some people could have been doing house work as their exercise (I know one case where this did occur with somebody in the GET group).

 

[Esther12]

Thanks a lot for that Dolphin. I can't seem to stop myself from continually assuming that their figures are based upon better evidence than they are. Seeing the actual questionnaires really helps.

I have no idea if this is true, but it seems perfectly plausible that:

At the start of the trial, patients want to emphasise their problems to those assessing them, in order to ensure that they are taken seriously and helped (particularly with a diagnosis like CFS).

Following treatments in which medical staff emphasised the importance of doing an activity like household tasks (GET) or some CBT equivalent, patients could well want to down-play any assistance they get out of a sense of pride.

Without knowing how these sorts of factors affect questionnaire results, the results themselves are of little value.

Anyone know if these questionnaires have been tested in other, similar circumstances? I googled the questions, and nothing came up.

 

[Dolphin]

Thanks a lot for that Dolphin. I can't seem to stop myself from continually assuming that their figures are based upon better evidence than they are. Seeing the actual questionnaires really helps.

No problem. In case anyone missed it (nobody downloaded the file), the full protocol booklet with all the questionnaires can be downloaded from: t: http://www.mediafire.com/?92x9s920bsftyx0 .

I have found it very useful to have this book as questionnaires can show up in all sorts of studies.

This particular questionnaire is on page 168.

 

[Esther12]

I did download it yesterday (or the day before). Not sure why it didn't register.

It might be a long time before I get the chance to give it a proper look though.

 

[Dolphin]

I did download it yesterday (or the day before). Not sure why it didn't register.

It might be a long time before I get the chance to give it a proper look though.

No problem. It is a big document. But the questionnaires are all at the back in alphabetical order so fairly handy.

EQ5d is on page 181.

 

[Dolphin]

Of course, as has been said, there are also "costs", mainly in terms of their time and energy, for the participant in doing such courses. Loads of healthy people don't do regular exercise because it involves sacrifices; for people with M.E. or CFS who are already struggling, doing such a course, with all the homework, involves bigger sacrifices/is a serious commitment, especially for people with other responsibilities.

I didn't find this piece that exciting: http://www.nhs.uk/news/2012/08augus...onic-fatigue-syndrome-not-cost-effective.aspx

However, was interested in the second paragraph here:

Based on the statistical models used by the researchers, CBT and graded exercise therapy were found to be most cost-effective, while specialist medical care and pacing were the least cost-effective.

The researchers did not consider patient preference, which could have an impact.

I am interested in this concept of patient preference. I don't know too much about it. It appears to me it could be a concept that could be useful for us.

 

[WillowJ]

I don't know whether referring specifically to "fatigue" as the cause for the need for help could affect results at all? One might think the other option is CFS (or ME) but there is also simply not specifying the illness and see how much help people got.

This is a good point. Some people might reason that they get help because of pain, or because of PER, and so forth, and not specifically because of 'fatigue'.

 

[WillowJ]

here is a great comment on the PLOS One article by TKindlon:

Title: Data for predefined primary outcome measure "overall improvers" has thus far not been published


The protocol for the PACE Trial was published in 2007 (1). It gave the following primary outcome measures:

“Primary outcome measures – Primary efficacy measures

Since we are interested in changes in both symptoms and disability we have chosen to designate both the symptoms of fatigue and physical function as primary outcomes. This is because it is possible that a specific treatment may relieve symptoms without reducing disability, or vice versa. Both these measures will be self-rated.

The 11 item Chalder Fatigue Questionnaire measures the severity of symptomatic fatigue [27], and has been the most frequently used measure of fatigue in most previous trials of these interventions. We will use the 0,0,1,1 item scores to allow a possible score of between 0 and 11. A positive outcome will be a 50% reduction in fatigue score, or a score of 3 or less, this threshold having been previously shown to indicate normal fatigue [27].

The SF-36 physical function sub-scale [29] measures physical function, and has often been used as a primary outcome measure in trials of CBT and GET. We will count a score of 75 (out of a maximum of 100) or more, or a 50% increase from baseline in SF-36 sub-scale score as a positive outcome. A score of 70 is about one standard deviation below the mean score (about 85, depending on the study) for the UK adult population [51,52].

Those participants who improve in both primary outcome measures will be regarded as overall improvers.”

The details for none of these have been published so far.

In the Lancet paper (2), the authors made reference to changing the primary outcome measures for the Chalder fatigue questionnaire and the SF-36 to continuous measures: "We used continuous scores for primary outcomes to allow a more straightforward interpretation of the individual outcomes, instead of the originally planned composite measures (50% change or meeting a threshold score)."

It should be pointed out that the protocol had already predefined secondary outcome measure using continuous values for the Chalder fatigue questionnaire ["1. The Chalder Fatigue Questionnaire Likert scoring (0,1,2,3) will be used to compare responses to treatment."] so that data could have been published anyway without making changes to the protocol, just not as the primary outcome measure.

No mention was made of the third primary outcome measure, "overall improvers". In the Lancet paper, post-hoc definitions for the percentages of participants improved for the Chalder fatigue questionnaire, the SF-36 Physical Functioning questionnaire, as well as both of them, were introduced. It is unclear why such criteria were necessary as these are dichotomous like the original primary outcome measure criteria so the argument for using continuous criteria doesn't apply. Also, the primary outcome measures used a validated definition for fatigue caseness.

The threshold to qualify as improved was much lower with these new post-hoc criteria. This can be seen in when one looks at the percentage that improved with specialist medical care (SMC) under the new definitions of improvement: 65% for fatigue, 58% for physical function and 45% for both (2).

The calculations the authors had made in designing the paper were for a much lower improvement rate (1):

"The existing evidence does not allow precise estimates of improvement with the trial treatments. However the available data suggests that at one year follow up, 50 to 63% of participants with CFS/ME had a positive outcome, by intention to treat, in the three RCTs of rehabilitative CBT [18,25,26], with 69% improved after an educational rehabilitation that closely resembled CBT [43]. This compares to 18 and 63% improved in the two RCTs of GET [23,24], and 47% improvement in a clinical audit of GET [56]. Having usual rather than specialist medical care allowed 6% to 17% to improve by one year in two RCTs [18,25]. There are no previous RCTs of APT to guide us [11,12], but we estimate that APT will be at least as effective as the control treatments of relaxation and flexibility used in previous RCTs, with 26% to 27% improved on primary outcomes [23,26]. We propose that a clinically important difference would be between 2 and 3 times the improvement rate of SSMC."

For a therapy to be three times the improvement rate of SSMC would mean 90% to 135% of patients doing that therapy would be improved (numbers over 100% impossible).

A letter has previously been published explaining the problem of using 0.5SD of the baseline values as cut-offs for a clinically useful difference (3). Basically because the baseline values were artificially restricted, the standard deviation is artificially restricted. In this circumstance other values should be used for a clinical useful difference.

The current paper (4) again uses these post-hoc criteria for improvement rather than the prespecified primary outcome measure.

It would be good if the data for the prespecified primary outcome measure for overall improvers could be released.


References:

1. White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; on behalf of the PACE trial group. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BioMed Cent Neurol 2007; 7: 6.
http://www.biomedcentral.com/1471-2377/7/6

2. White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, et al. (2011) Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet 377: 823–836.

3. Giakoumakis J. The PACE trial in chronic fatigue syndrome. Lancet. 2011 May 28;377(9780):1831; author reply 1834-5. Epub 2011 May 16.
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60689-2/fulltext

4. McCrone P, Sharpe M, Chalder T, Knapp M, Johnson AL, et al. (2012) Adaptive Pacing, Cognitive Behaviour Therapy, Graded Exercise, and Specialist Medical Care for Chronic Fatigue Syndrome: A Cost-Effectiveness Analysis.PLoS ONE 7(8):e40808.
doi:10.1371/journal.pone.0040808

 

[alex3619]

Professor Michael Hyland, a Fellow of the British Psychological Society, comments:
"The PACE trial is a well-conducted study which has provided evidence, contrary to the assertion of patient groups, that adaptive pacing does not lead to benefit whereas cognitive behaviour therapy (CBT) and graded exercise therapy do.

How can he be so wrong? First, so far as I am aware we, as patients, all agree that adaptive pacing is of no use and probably dangerous. It seems he cannot tell the difference between pacing and adaptive pacing, which is the logical negation of pacing. This is a win for pro-PACE spin when a professor can't spot such simple errors.

People are not reading the studies in detail. They are not investigating methodologies. They are not taking into account the masses of contrary evidence. They are not recognizing how so many terms have been redefined and then used as though the definition were the same as the original. This appears to include the media, doctors, and even professors. Its sad.

Benefit from CBT and GET? The hard evidence says otherwise, in study after study, and this all agrees with patient claims. When are so called experts going to check the evidence before they make claims?

Bye, Alex

 

[biophile]

[edit: made a few modifications to original post]

I crunched a few numbers. [Table 3], CBT does little to reduce total societal costs vs SMC:
SMC: £22,088 / (£12,001 x 2) = -7.97% reduction or £1,914 less than pre-randomization.
CBT: £20,288 / (£11,333 x 2) = -10.49% reduction or £2,378 less than pre-randomization.
According to the authors, there were no significant difference between any groups except for CBT vs APT.

Meanwhile total health costs went up for CBT vs SMC:
SMC: £1424 / (£770 x 2) = -7.53% reduction or £116 less than pre-randomization.
CBT: £2322 / (£755 x 2) = +53.78% increase or £812 more than pre-randomization.
The authors indicate that the difference is significant, although health costs are part of total societal costs. When looking at the above values, the extra cost of CBT is not clearly recovered by the decrease in total societal costs vs SMC and has no obvious advantage for overall service costs.

Anyway, from the Discussion section:

"The major contributors to societal costs were lost employment and informal care costs. The cost-effectiveness of CBT was more apparent from this perspective than from the narrower healthcare perspective; CBT dominated APT, GET and SMC, while GET dominated APT and SMC. Informal care costs were substantial for each group and significantly lower after receiving CBT and GET when compared to APT and SMC. CFS affects patients in many ways and can have a major effect on family members. This study suggests that CBT and GET could ameliorate this effect. However, with the exception of a difference between CBT and APT, there were no significant differences in either lost work time or benefits between the treatments during follow up. In fact, benefits increased across all four treatments."

So back to [Table 3] for lost employment:
SMC: £14,157 / (7,499 x 2) = -5.61% reduction or £841 less than pre-randomization.
CBT: £13,958 / (7,978 x 2) = -12.52% reduction or £1,998 less than pre-randomization.

Again to [Table 3] for informal care:
SMC: £6,507 / (£3,732 x 2) = -12.82% reduction or £957 less than pre-randomization cost.
CBT: £4,008 / (£2,601 x 2) = -22.95% reduction or £1,194 less than pre-randomization cost.

While CBT seems to be better at reducing lost employment and informal care (the bulk of societal costs), these were selected from the overall service costs which show no obvious advantage for CBT when factoring in costs of therapy. "However, with the exception of a difference between CBT and APT, there were no significant differences in either lost work time or benefits between the treatments during follow up. In fact, benefits increased across all four treatments."

From the section on Welfare benefits and other financial payments:

"Receipt of benefits due to illness or disability increased slightly from baseline to follow-up (Table 4). Patients in the SMC group had the lowest level of receipt at baseline but the figures at followup were similar between groups. Relatively few patients were in receipt of income-related benefits or payments from income protection schemes and differences between groups were not substantial."

Increased slightly? The combined proportion of participants receiving all types of financial payments (assuming no overlap) was SMC 37% to 60%, APT 50% to 68%, CBT 48% to 63%, GET 53% to 72%. If these figures were reversed and favoured CBT/GET I doubt they would be calling such differences "slight". And from the section on Methods:

"We excluded welfare benefits or payments from other sources such as private pensions and income protection schemes from the economic costs. However, we do report the receipt of these given that they are important financial outlays."

Wait, what? Although lost employment does not really change much, For some unknown reason their cost analyses did not take into account the increasing use of welfare benefits or other financial payments. This could have significant impact on cost-effectiveness, as benefits/payments cost a lot of money per person. Notice how all groups in the trial reported improvements in symptoms but also had increased usage of welfare benefits and other financial payments? I think I already highlighted the potential irony here for the underlying rationale of CBT/GET if these are associated.

The cost-effectiveness of CBT and GET are minimal and fragile at best when it comes to overall service costs. I still haven't looked at the other cost-effectiveness analyses which AFAIK are not based around actual monetary exchange but based entirely around hypothetical value of QALY gained (EQ-5D), self-reported fatigue and physical function. As others have pointed out, there are potential flaws and biases in these measures as used in the PACE Trial.

 

[biophile]

The news has been met with interest from healthcare experts, with Professor Michael Sharpe from Oxford University saying, “This new evidence should encourage health service commissioners to provide these treatments to all those patients who need them”.

http://www.egovmonitor.com/node/53027

LOL, of course the hyped news is going to be met with "interest" from Sharpe, he was a co-author of the paper and a principal investigator of the trial in question, and has invested part of his career and reputation on CBT/GET. I wonder if Sharpe met the data on poor employment/welfare/insurance outcomes with interest too? You know, the data that was downplayed in the paper and ignored during cost-effectiveness analyses (except for employment losses), while suspiciously remaining out of the news headlines and articles? Classic spin 101.

Wasn't one of the recent failed FOIs (which included the request for data on deterioration rates based on the reverse criteria for improvement) denied because of a pending publication of another PACE paper? I see no such deterioration data in this new paper. Unless there is another PACE paper being peer-reviewed, we have been scammed once again.

 

[Firestormm]

Child Care No Yes (circle No if you have no children)

Just choosing one for a moment. Does that mean that all the childless people in the study, circling 'No', will improve the results? i.e. Every 'No' counts as a positive i.e. my fatigue has reduced so much I don't need anyone to help me with those kids I don't have


The 'good old' Huffington Post carried this tale of predictable euphoria: http://www.huffingtonpost.co.uk/201...ptoms-chronic-fatigue-syndrome_n_1732004.html posted here originally: http://www.mecfsforums.com/index.php/topic,13031.msg139651.html#new

Chronic Fatigue Syndrome Treatments 'Good Value' (Can You Spot Signs? PICTURES)

Straight off the bat we have the first set of signs:

If you sleep for eight hours a night, yet never feel rested and are constantly tired, irritable and unmotivated, these could be signs of chronic fatigue syndrome (CFS), say experts.

My comment is that the above means you are normal and stressed. I mean who the hell are these people? Eight hours sleep A NIGHT? If only. So that's it then folks. Way to reduce a condition to normality. No wonder we get ridiculed.

 

[Snow Leopard]

Just to clarify my previous comment, twenty or so thousand pounds was referring to the (substantial) total economic costs after receiving CBT/GET.

Nevertheless, delivering CBT itself to all CFS patients in the UK would cost hundreds of millions of pounds (to apparently save tens of millions), up to a cost of billions if the 2.6% prevalence is to be believed.

Hell, I hope that they do spend that money - then they will find out how serious CFS really is.

 

[Firestormm]

I gotta crunch these figures when I get some time. Thanks to all for posting what you have.

 

[Dolphin]

Good analysis, biophile

Increased slightly? The combined proportion of participants receiving all types of financial payments was SMC 37% to 60%, APT 50% to 68%, CBT 48% to 63%, GET 53% to 72%. If these figures were reversed and favoured CBT/GET I doubt they would be calling such differences "slight".

Just a small point on this: I'm not sure we can be 100% sure there is no overlap i.e. one person doesn't get more than one payment. However, adding them together gives the best indicator of total costs i.e. if some person, X1, gets Benefit A & B, while two people, X2 & X3, get benefits A& B respectively (i.e. one each), costs are likely similar (again we can't be 100% sure they're the same as there can be means testing, but it's likely the best we can do with the figures).

 

[Esther12]

I've not digested some of the above posts (looks like biophile did what I failed to, in going back to some raw data and really trying to get to grips with it, and that looks interesting).

I want to quickly make a... Pro-PACE post:

I'm perfectly happy to use the fact that more patients were on benefits after CBT/GET than before to attack the value of these treatments, and especially the misleading way in which their efficacy has been exaggerated... but it's probably actually a good thing.

Lots of people with CFS are unable to get access to benefits which they deserve, and which will help them manage their condition. While some CFS centres refuse to help with these problems, it's probably a good sign for the way in which patients were treated that more of them ended up on benefits. (Although, this positive only counts once you've already accepted that the treatments aren't effective at improving capacity.)

Okay, now that's done we can go back to slating them.

 

[Dolphin]

Just choosing one for a moment. Does that mean that all the childless people in the study, circling 'No', will improve the results? i.e. Every 'No' counts as a positive i.e. my fatigue has reduced so much I don't need anyone to help me with those kids I don't have

It shouldn't because they also asked them before the therapies so such people presumably got no help then for their non-existent kids*

* presuming people don't help patients with any imaginary kids.

 

[Simon]

I crunched a few numbers. [Table 3],

Thanks for the analysis.

Data Crunched for costs
I now have the data in an excel spreadsheet with some bells and whistle:
- 6 month pre-baseline data pro-rated to 12 months for comparison with post-baseline
- subtotals eg for PACE vs other healthcare costs
- separate table for pre vs post ,and all vs SMC

But it won't let me upload (extension not allowed). People can PM with an email address if they want it edit: or get from google docs

Couple of important points when using the table data to compare pre/post:

1. Looks like you can't make exact direct comparisons between pre & post figures because they use a more sophisticated approach in their modelling:

   Healthcare, informal care and societal costs
   (including lost production and informal care) during the 12 months
   after randomisation in each treatment group were compared using
   regression models controlling for baseline costs and clustering for
   centre.

2. For the 12 month post-randomisation data, the healthcare totals slightly overstate the costs based on the indiviudal items (though 6-month data adds up perfectly). No idea why, but the GET & CBT figures are overstated by more than the SMC ones.

more later

 

[Dolphin]

I've not digested some of the above posts (looks like biophile did what I failed to, in going back to some raw data and really trying to get to grips with it, and that looks interesting).

I want to quickly make a... Pro-PACE post:

I'm perfectly happy to use the fact that more patients were on benefits after CBT/GET than before to attack the value of these treatments, and especially the misleading way in which their efficacy has been exaggerated... but it's probably actually a good thing.

Lots of people with CFS are unable to get access to benefits which they deserve, and which will help them manage their condition. While some CFS centres refuse to help with these problems, it's probably a good sign for the way in which patients were treated that more of them ended up on benefits. (Although, this positive only counts once you've already accepted that the treatments aren't effective at improving capacity.)

Okay, now that's done we can go back to slating them.

Ok, don't think Barts should get any credit for this. There is a woman on MEA FB complaining how her service in Essex won't write any letters for benefit applications. They claim it's some sort of "conflict of interest".

Some posts from her from the thread:

• CFS clinics and 'conflicts of interest' in supporting benefits calims.

  At the Essex service i ahve been told that I cannot use them for medical evidence for benefts calims because they consider it a 'conflict of interest'.
  
  I would be interested in hearing if anyone else has had this said to them and if the MEA know anything about it.
  
  Thank you!

• Thanks - I will see if I can get a written statement about it but i know it is happening in a lot of CFS clinics. Sussex and Sheffield being two of them.
  
  I recieved a letter from someone attending the Sheffield clinic as an example of what their therapist had to do to try and support his claim it states:
  
  "The Chronic Fatigue Syndrome/Myalgic Encephalomyelitis (CFS/ME) Service is not currently able to provide medical reports related to applications for Disability Living allowance). The Therapy service providing a range of of strategies to enable individuals to self manage their symptoms and does not provide medical interventions."
  
  In this case the OT writing the letter gos on to say the person attended the clinic and what ME is and that hiscondition is likely to persist.

• OK - I spoke to the admin at Southend and she told me that the policy comes from St. Barts - she said it is from Prof Peter White.
  
  The clinic have told me I need to do less but then won't provide evidence for benefits claims because they 'aim for recovery' - it just goes round and round doesn't it.
  
  She also said that GP's sign a form when they refer someone to the clinic saying that they are responsible for benefits claims.
  
  I reapeated that i understood that but that i wanted to know what the 'conflict of interests were' and she again said 'we aim for recovery'.
  
  I asked what happens if people don't get better and she said 'they would be dischraged'.
  
  From someone else who attended the clinic but decided it was'thelping I am aware that they then send out a letter saying that you are not ready for recovery but are welcome to return when you feel you want to recover - it is basically a 'blame the patient' letter.

  She mentions somewhere she may become homeless because of the policy.

 

[Bob]

This paper is confusing the hell out of me, and I haven't even begun to read it yet!!!
I'm going to have a look at it today.

I'm getting really irritated by the media reports that CBT and GET are the 'best' or 'most cost effective' therapies. They only compared them with SMC and APT. APT was a completely untested experimental novel therapy invented solely for use in the PACE Trial, so it's not surprising it was a complete failure. And SMC was the control group, so I'm not certain that it is appropriate to talk about the cost benefits of this group.

Also, SMC just consisted of normal medical care, that you would expect to get from a GP, with a tiny bit of extra advice, and so these are costs which are going to be paid by the NHS whatever extra therapies are given. So CBT and GET should only be considered as supplementary treatments, on top of SMC, not as alternative costs. I don't know if they've taken this into account, but I don't expect that they have.