• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

A cost effectiveness of the PACE trial

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
GET QALY improvement not significantly better than SMC?

CBT produced the largest QALY gain, significantly more than SMC. After controlling for baseline utility, the difference between CBT and SMC was 0.05 (95% CI 0.01 to 0.09). No other differences between treatment groups were statistically significant.
This implies the GET QALY gains were not significantly different than for SMC, though table 6 does show a QALY gain of 0.034 for GET compard to SMC, which is a tad confusing.

Fatigue & Function cost-effectiveness may be overstated

ICER indicates the [cost per QALY gained or] cost per unit
reduction in fatigue or disability for one treatment against another...
ICERs constructed with the CFQ and SF-36 PF data used the
differences in proportions achieving clinically important changes.
This looks to me like they used the proportion who improved, while ignoring the proportion who worsened by the same amount (point © Dolphin), which could overstate the benefit (depends on proportions who worsened on CBT/GET vs SMC).


Re: Proportion on Benefit may go up due to time lag
A good point, particularly given that these patients hadn't been ill that long:

SMC: 25 months (25% less than 15 months)
CBT: 36 months (25% less than 16 months)
GET: 35 months (25% less than 18 months)

The authors could run analyses that would throw light on this: were people with a shorter illness duration more likely to be awarded benefits during the trial?
Although we don't know the outcome of this analysis, the SMC group has the shortest average illness duration so is a crude proxy of this. At baseline, SMC has the lowest proportion on illness/disability benefits but has caught up by 12 months (proportion increasing from 34% to 58%, Table 4). So this is consistent with the idea that the increases in proportion on benefits is reflecting a time lag in awarding benefits rather than a deteoriation in health - though clearly it isn't conclusive.

Just as important, none of the interventions reduced benefit levels, which is hardly a ringing endorsement of said treatments.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
This is my initial understanding of the conclusions of the paper...
(As usual, I'm not sure if I've got this correct, because I'm struggling to grasp the complexity of this paper, so any feedback would be welcome.)


CBT and GET were more expensive than SMC to administer.

Compared to SMC, CBT and GET improved no overall or total costs in this paper, by a significant amount (there were differences in the various itemised costs.) (Even overall societal costs were not significantly improved by CBT or GET, compared to SMC.)

So there were no overall significant savings for CBT and GET, over SMC, in: welfare payments and private financial payments costs; health care costs; lost employment costs; or lost production costs, or societal costs. There were significant differences in informal care costs, which is an item included in societal costs.


There were a few sets of cost data used for calculating cost effectiveness in this paper:

Health costs per QALYs gained per individual, with a QALY valued at £30,000.
Societal savings per QALY gained per individual.
Societal savings per individual improved for Chalder fatigue.
Societal savings per individual improved for SF-36 physical function.


They do not say that CBT and GET are hugely cost effective.
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Pacing is one of the prefered and highly successful treatments, as decided by patients. Adaptive Pacing Therapy is its logical opposite. So APT may well be close to the worse possible therapy.

If CBT/GET is not massively better than a very bad treatment, what kind of message is that?

Bye, Alex
 

Purple

Bundle of purpliness
Messages
489
Would the means-testing of ESA in the WRAG have an effect? (i.e. less people would be receiving illness disability benefits because they have partners to support them/their own means rather than because they are less sick but people who don't have means would still receive income-related ESA and therefore there is no change in income welfare benefits). Or is this not applicable? Does the paper break down or list which benefits exactly they refer to?
 

Dolphin

Senior Member
Messages
17,567
Would the means-testing of ESA in the WRAG have an effect? (i.e. less people would be receiving illness disability benefits because they have partners to support them/their own means rather than because they are less sick but people who don't have means would still receive income-related ESA and therefore there is no change in income welfare benefits). Or is this not applicable? Does the paper break down or list which benefits exactly they refer to?
Data is in this table: http://www.plosone.org/article/info...RI=info:doi/10.1371/journal.pone.0040808.t004

Trial was end of 2005 (or so) (?) to end of 2009 (or so) (?). I think ESA changes started in 2008 (?) so wouldn't have applied to a lot of the people anyway.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Societal benefits for CBT & GET probably overstated (and important data not shown)

The paper justifies the cost-effectiveness of CBT & GET on Healthcare costs alone. But it also plays up the enhanced cost-effectiveness using Societal costs, which includes costs of informal care as well as healthcare.
The findings we report suggest that such investment [in CBT/GET] would be justified in terms of improved quality of life of patients and would actually be cost saving if all costs including societal costs are considered.

... The cost-effectiveness of CBT was more apparent from this [Societal] perspective than from the narrower healthcare perspective; CBT dominated APT, GET and SMC, while GET dominated APT and SMC.
Yes, not only is CBT & GET better than SMC alone, it saves society money too. Or maybe it doesn't, depending on the assumptions used.

The major contributor to CBT/CFS Societal savings were from Informal Care, since lost employment was not significantly different between any of the groups. The authors chose to value this informal care using national mean earnings of £14.60 an hour. That's not real money, it's what the informal carers would earn if they worked instead of providing care, and were payed the average national wage. A more conservative way to value this is by using the minimum legal wage in the UK of £5.93 an hour. The authors looked at this option in their 'sensitivity' analysis and reported that "it did not have a large impact on cost-effectiveness".

'Not a large impact' isn't a very precise term, particularly when CBT and GET didn't have a large impact on QALY outcomes either. No data is presented, though under 'Limitations' the authors say:
we made assumptions regarding the value of unpaid care from family and friends and lost employment. However, sensitivity analyses revealed that the results were robust for alternative assumptions.
This may not be entirely correct. Table 6 shows the Societal cost savings for CBT & GET relative to SMC, using national mean wages to value savings in informal care. However, as the figures below show, those savings vanish when the informal care is valued at the miminum wage instead:

societal.jpg


This looks rather different. GET now has a cost of £225 (as opposed to a saving of £472) while CBT is neutral instead of saving nearly £700. So the authors claim that:
there would actually be cost saving if all costs including societal costs are considered
doesn't apply to the minimum wage scenario. Similarly, the claim that CBT & GET 'dominate' (i.e. is more effective and saves money) SMC doesn't apply using the minimum wage scenario. And these figures are hardly consistent with the claim that "results were robust for alternative assumptions".

The authors could easily have mentioned the impact of using the mimimum wage in the paper rather than saying the sensitivity analysis didn't reveal 'large' differences. Or they could have put the data in an online appendix. It doens't look good when unflattering data is hidden, and the value of sensitivity analysis is diminished if the results of it aren't fairly or adequately reported.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Would the means-testing of ESA in the WRAG have an effect? (i.e. less people would be receiving illness disability benefits because they have partners to support them/their own means rather than because they are less sick but people who don't have means would still receive income-related ESA and therefore there is no change in income welfare benefits). Or is this not applicable? Does the paper break down or list which benefits exactly they refer to?

Good question.
I'll go with Dolphin's answer that the change to ESA would have had limited effect. But it could have had some effect.

I wonder if part of the CBT and GET therapy was to encourage people not to claim disability benefits. Has anyone come across that in the PACE Trial literature?

I imagine that illness benefits are: IB, ESA and DLA.
Whereas income benefits would be: income support and job seekers.
 

Dolphin

Senior Member
Messages
17,567
GET QALY improvement not significantly better than SMC?
CBT produced the largest QALY gain, significantly more than SMC. After controlling for baseline utility, the difference between CBT and SMC was 0.05 (95% CI 0.01 to 0.09).​
No other differences between treatment groups were statistically significant​
.​

This implies the GET QALY gains were not significantly different than for SMC, though table 6 does show a QALY gain of 0.034 for GET compard to SMC, which is a tad confusing.
Yes, it is an important point you highlight. Generally, if there is no statistically significant difference, it can end there. APT also shows up as a QALY gain of 0.0149 for APT compard to SMC. But because of the type of calculations they did, combining two outcomes (QALYs and cost) as well I think as the involvement of CBT it seems it doesn't seem to matter. I believe the adding of CBT into figures 1 and 2, pushes the probabilities for SMC down i.e. if it was GET vs SMC (without brining in CBT), SMC would be higher.
 

Dolphin

Senior Member
Messages
17,567
Fatigue & Function cost-effectiveness may be overstated
ICER indicates the [cost per QALY gained or] cost per unit
reduction in fatigue or disability for one treatment against another...
ICERs constructed with the CFQ and SF-36 PF data used the
differences in proportions achieving clinically important changes.

This looks to me like they used the proportion who improved, while ignoring the proportion who worsened by the same amount (point © Dolphin), which could overstate the benefit (depends on proportions who worsened on CBT/GET vs SMC).
That's a very astute point, Simon. I can't say I had spotted the problem of using it, in the way you have highlighted it (although was generally uncomfortable with it).

This would be a problem with any categorical measure: I wonder how it is dealt with in other situations?

I wonder also how often minimum important differences are used in such calculations or whether much larger improvements are usually used in such calculations?

BTW, thanks for the acknowledgement :) but my comments are copyright free, on this forum anyway (if somebody copied a whole chunk, I might have a problem: but probably not if it was done in a comment I would agree with).
 

Dolphin

Senior Member
Messages
17,567
This was the question they were asked:

What benefits (if any) do you currently receive?

Name of benefit Please circle

Income Support I
Incapacity Benefit 2
Disability Living Allowance
- care component 3
- mobility component 4
Disabled Person's Tax Credit 5
Severe Disablement Allowance 6
Council Tax Benefit 7
Housing Benefit 8
Jobseeker's Allowance 9
Working Tax Credit 10
Statutory Sick Pay 11
State retirement pension 12
Other (please specify) 13
 

Dolphin

Senior Member
Messages
17,567
Thanks Simon, I wonder if it also assumes that everybody who is caring for somebody, would otherwise be working extra hours.
I think it is relevant to measure such things. Indeed, if we are ever to get expensive drugs paid for by national health services, this is one of the ways that will help us argue for them: the overall burden on society is reduced if our health improves.
 

Dolphin

Senior Member
Messages
17,567
Societal benefits for CBT & GET probably overstated (and important data not shown)

<snip>

The authors could easily have mentioned the impact of using the mimimum wage in the paper rather than saying the sensitivity analysis didn't reveal 'large' differences. Or they could have put the data in an online appendix. It doens't look good when unflattering data is hidden, and the value of sensitivity analysis is diminished if the results of it aren't fairly or adequately reported.
Excellent post, Simon. Really interesting stuff you've unearthed there.

I find it odd that they haven't included the data in online appendices - I have seen papers with lots of (5-10+) online appendices with such information.

And of course, just to remind people, these therapies are only seen as value-for-money at a societal level as they have excluded "welfare benefits or payments from other sources such as private pensions and income protection schemes from the economic costs". It looks very likely they wouldn't have been value-for-money if this data had been included. They could have calculated two sets of figures, one including them, one not (and done sensitivity analyses on the welfare benefits or payments from other sources if they had doubts about figures to include - they did manage to enter mean national figures for all sorts of other things in the paper).
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Just for the record...

Competing interests:

PDW has done voluntary and paid consultancy work for the United Kingdom Departments of Health and Work and Pensions and Swiss Re (a reinsurance company)...

MS has done voluntary and paid consultancy work for the United Kingdom government, has done consultancy work for the insurance company Aegon...

http://www.plosone.org/article/info:doi/10.1371/journal.pone.0040808
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
"welfare benefits or payments from other sources such as private pensions and income protection schemes from the economic costs".

These are a redistribution, rather than a specific cost, thought the overhead itself probably should be counted (eg cost of managing the welfare programme/fund).
 

user9876

Senior Member
Messages
4,556
The authors chose to value this informal care using national mean earnings of £14.60 an hour. That's not real money, it's what the informal carers would earn if they worked instead of providing care, and were payed the average national wage.

Mean average earnings have traditionally been pushed up by bonuses in the city. Most people use median (typical) earnings. In april 2011 the median hourly rate excluding overtime and including men, women, full and part time workers was £11.15.

http://www.ons.gov.uk/ons/dcp171778_241497.pdf

I've looked at the source they quote since I was having trouble finding a 14.60 mean wage. The median figure in their 2010 reference seems to be 11.14
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
I'm very grateful to you chaps for getting stuck into this one.

I really do not understand why they even tried to drill down as much as they did. I can only think that like the PACE Trial itself they expected more than was revealed. Unfortunately, the reality is that only the headlines will ever be considered by those with the authority to do something about it.

It is wrong and I think there is a publication bias happening that doesn't allow sufficient peer review or critique; but the authors will use this paper as a further justification for continuing to promote these therapies in the absence of anything else that might be considered - and it has an effect not only on our condition of course it will I am sure be used to support the case for the adoption of such therapies across the board especially for long term conditions.

Still, one 'good' thing to come from this is that they have opened the door so to speak in allowing patient support groups and patients themselves (as well as those who commission such services who retain some open-mindedness) to judge the effectiveness of their treatments in terms of economic reality.

What I mean is: if a patient completes a course of therapy and their circumstances have not improved, then they now have a valid basis to perhaps hold the therapist/service/commissioners/promoters to account - if only to ask 'Why am I not able to work now?' or 'Why do I still need to depend on benefits and on the unpaid care of others?' in other words 'Your therapy has not been as effective as it said on the tin!'

I wonder if trading standards would ever be interested? :lol:

I would also suggest the thread is 'stickied'. I will certainly need to keep this one in mind and refer back to it frequently I think - once my head allows me to understand the things you boffins are talking about :)
 

Dolphin

Senior Member
Messages
17,567
Dolphin said:
"welfare benefits or payments from other sources such as private pensions and income protection schemes from the economic costs".
These are a redistribution, rather than a specific cost, thought the overhead itself probably should be counted (eg cost of managing the welfare programme/fund).
Is there no type of analysis that could use such information? I have seen them excluded before from productivity cost calculations but thought there might be other ways of looking at the problem.