• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

A cost effectiveness of the PACE trial

Dolphin

Senior Member
Messages
17,567
I was just looking at Simon's file of comparisons.
Using the raw figures for informal care, CBT and GET don't involve savings that are that big compared to SMC: £237 and £408 respectively. However, when they adjust for baseline, these jump to £1165 (95% CI: £239-2894) and £1173 (95% CI: £740 to £1569). I wonder did they have options in terms of what baseline factors they could adjust for and maybe chose factors that helped increase this gap?
Can anybody think about this. They say they
Control[ling] for baseline
. This presumably means that the higher the initial costs, the bigger the savings expected, and this is controlled for. However, would the trend be so big that it would likely make such a difference given the following: if one looks at APT, it had the second highest informal care costs initially and it reduced the least?
 

biophile

Places I'd rather be.
Messages
8,977
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1199645

Biopsychosocial Medicine: An Integrated Approach to Understanding Illness[6] is the product of a two-day conference held under the auspices of One Health—an organization that seeks to promote a system of healthcare based on this approach—and the Novartis Foundation. In his preface the convenor, Peter White tells us that his rationale for organizing the conference was a concern that medicine is travelling up a blind alley in its attempt to help patients improve their health and reduce their disability. ‘This blind alley is the biomedical approach...’

6. White P, ed. Biopsychosocial Medicine: an Integrated Approach to Understanding Illness. Oxford: Oxford University Press, 2005. [242 pp; ISBN 0-19-53034-X; £70]

Oh the irony, considering how White has lead us up a blind alley of his own with the cognitive behavioural model of CFS (functional somatic syndrome predominantly perpetuated by unhelpful illness beliefs and deconditioning), and after the PACE results even the stubborn MRC are now shifting the funding towards biomedical research.
[...] Marmot's chapter, however, is closely followed by a warning from George Davey Smith against the too ready assumption that an association signifies cause and effect. To illustrate the pitfalls of confounding and bias he offers a striking set of ‘cautionary tales’. An example is the peptic ulcer story: such was the consensus that this condition was stress-induced that otherwise careful researchers overlooked important epidemiological and other evidence pointing to an infective causation.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Dolphin said:
I was just looking at Simon's file of comparisons.
Using the raw figures for informal care, CBT and GET don't involve savings that are that big compared to SMC: £237 and £408 respectively. However, when they adjust for baseline, these jump to £1165 (95% CI: £239-2894) and £1173 (95% CI: £740 to £1569). I wonder did they have options in terms of what baseline factors they could adjust for and maybe chose factors that helped increase this gap?

Can anybody think about this. They say they

Control[ling] for baseline
. This presumably means that the higher the initial costs, the bigger the savings expected, and this is controlled for. However, would the trend be so big that it would likely make such a difference given the following: if one looks at APT, it had the second highest informal care costs initially and it reduced the least?

Well spotted Dolphin. I can't understand how they could adjust the difference from £237 to £1165 or from £408 to £1173. It seems far-fetched.

There do seem to be quite a few apparent inconsistencies in this paper, based on the adjusted sums.
I'm also surprised that, for Societal costs ("i.e. healthcare, informal care and lost production costs"), they say there are no statistical differences apart from CBT vs APT.
Because Simon's comparisons with SMC seem fairly substantial, for CBT vs SMC at least:
Total societal costs APT 1,592 CBT -532 GET -244 SMC 0

And my figures, which are the differences between adjusted totals, in Table 3, still seem quite substantial for CBT vs SMC at least:
Total societal costs APT 1617 CBT -464 GET -197 SMC 0


Edit:
But, there again, individual societal savings, per person improved, and per QALY gained, might be statistically different, which adds more layers of confusion!
 

user9876

Senior Member
Messages
4,556
Does anyone understand how they calculated the relationship between therapies in Figure 1?
I've been wondering about fig 1 and fig 2. The x axis is for the amount that the governemt is prepared to spend for a QALYs. So if they are not prepared to spend anything then clearly SMC has a probability of 1 of getting sucess. As we move from the edge case then you look at the probability that you get a better treatment given the spending constraint. My guess is costs are fixed and so that they are doing is looking at the probability distribution for QALYs and asking which is most likely to deliver the QALYs gain necessary to get to the threshold cost. I've not managed to work out how they do this as I'm not sure if the graph shows individual values or if they all the points for a given threshold add up to 1.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
I've been wondering about fig 1 and fig 2. The x axis is for the amount that the governemt is prepared to spend for a QALYs. So if they are not prepared to spend anything then clearly SMC has a probability of 1 of getting sucess. As we move from the edge case then you look at the probability that you get a better treatment given the spending constraint. My guess is costs are fixed and so that they are doing is looking at the probability distribution for QALYs and asking which is most likely to deliver the QALYs gain necessary to get to the threshold cost. I've not managed to work out how they do this as I'm not sure if the graph shows individual values or if they all the points for a given threshold add up to 1.

Yes, my thoughts about it are similar. And I can't work out what calculations they've used. I'm interested to find out how/why SMC goes from a probability of 1, to almost zero.

The only explanation I can find is (if it is the relevant explanation):

"Interpretation of the cost-effectiveness results was made using
cost-effectiveness acceptability curves [18]. Net benefit values were
computed for each study participant, defined as the value of a
QALY multiplied by the number of QALYs gained minus the cost
(from both healthcare and societal perspectives)."

(There's a bit more text after that quote in the paper, about how they fit the values into the graphs.)
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
RE: measuring fatigue/disability improvements:
That's a very astute point, Simon. I can't say I had spotted the problem of using it, in the way you have highlighted it (although was generally uncomfortable with it).

This would be a problem with any categorical measure: I wonder how it is dealt with in other situations?

I wonder also how often minimum important differences are used in such calculations or whether much larger improvements are usually used in such calculations?
No idea how such categorical issues are dealt with in general, but like you I do wonder if it's normal to show cost-effectiveness calculations for "minimum useful differences".
This was the question they were asked:
What benefits (if any) do you currently receive?

Name of benefit Please circle

Income Support I
...
Statutory Sick Pay [SSP] 11
State retirement pension 12
Other (please specify) 13
Ah, as always, good to see the underlying questions. If SSP was included as a benefit (despite being payed by employers) then there wouldn't be such a time delay for people with CFS getting on benefits - SSP is straightforward to claim. On the other hand, many employers give sick pay at a much higher rate (mine did initially) in which case people may not be aware part of their salary is "SSP" if they continue to be paid at their normal rate. So they might not say they were on SSP.
Can anybody think about this. They say they
Control[ling] for baseline. This presumably means that the higher the initial costs, the bigger the savings expected, and this is controlled for. However, would the trend be so big that it would likely make such a difference given the following: if one looks at APT, it had the second highest informal care costs initially and it reduced the least?
If you are comparing using Table 6, note that individual costs for informal care are not given but instead they give 'Societal' costs which also includes employment. I have a feeling that employment losses were diferent for APT and that might complicate the situation.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
GET QALY improvement not significantly better than SMC?

CBT produced the largest QALY gain, significantly more than SMC. After controlling for baseline utility, the difference between CBT and SMC was 0.05 (95% CI 0.01 to 0.09). No other differences between treatment groups were statistically significant.

This implies the GET QALY gains were not significantly different than for SMC, though table 6 does show a QALY gain of 0.034 for GET compard to SMC, which is a tad confusing.

Also, the differences in societal costs are not significant between CBT/GET and SMC, so that's two statistically insignificant figures that they've used in that section, in order to work out that GET is dominant over SMC.
 

Dolphin

Senior Member
Messages
17,567
If you are comparing using Table 6, note that individual costs for informal care are not given but instead they give 'Societal' costs which also includes employment. I have a feeling that employment losses were diferent for APT and that might complicate the situation.
No, I was replying to this (raw figures are from Table 3):
Using the raw figures for informal care, CBT and GET don't involve savings that are that big compared to SMC: £237 and £408 respectively. However, when they adjust for baseline, these jump to £1165 (95% CI: £239-2894) and £1173 (95% CI: £740 to £1569). I wonder did they have options in terms of what baseline factors they could adjust for and maybe chose factors that helped increase this gap?
 
Messages
5,238
Location
Sofa, UK
Mean average earnings have traditionally been pushed up by bonuses in the city. Most people use median (typical) earnings. In april 2011 the median hourly rate excluding overtime and including men, women, full and part time workers was £11.15.

http://www.ons.gov.uk/ons/dcp171778_241497.pdf

I've looked at the source they quote since I was having trouble finding a 14.60 mean wage. The median figure in their 2010 reference seems to be 11.14
Indeed, and this is simply abuse of statistics, spin and manipulation of results; science with proper peer review shouldn't stand for it.

The issue with the PACE authors' abuse of averages is explained very well here:

http://evaluatingpace.phoenixrising.me/aps2details.html
http://evaluatingpace.phoenixrising.me/aps2more.html
 

Dolphin

Senior Member
Messages
17,567
But I couldn't see any figures for baseline-adjusted informal care costs for APT vs SMC which were the basis for the comments I made re CBT/GET. Did I miss them?
You've lost me now.
In this comment: http://forums.phoenixrising.me/inde...s-of-the-pace-trial.18722/page-10#post-285886 I was commenting on this comment by myself, where there is no mention of APT:
Using the raw figures for informal care, CBT and GET don't involve savings that are that big compared to SMC: £237 and £408 respectively. However, when they adjust for baseline, these jump to £1165 (95% CI: £239-2894) and £1173 (95% CI: £740 to £1569). I wonder did they have options in terms of what baseline factors they could adjust for and maybe chose factors that helped increase this gap?
 
Messages
13,774
Sorry, not following this thread, but I just noticed that SMC press briefing. Ignore if it's already been mentioned:

01 August 2012
Do the best treatments for CFS cost more?

Speakers:
Professor Michael Sharpe, Professor of Psychological Medicine, University of Oxford
Dr Paul McCrone, Professor of Health Economics, Kings College London

Chronic Fatigue Syndrome (CFS, or ME) is a condition characterised by deliberating, long lasting fatigue that is not attributable to another underlying condition. The PACE group in 2011 demonstrated that Cognitive Behavioural Therapy (CBT) and Graded Exercise Therapy (GET) were the most effective therapies for CFS. Authors of a follow up paper came to the SMC to discuss a new paper which explores whether therapies that are most effective at treating CFS also the most cost-effective.

http://www.sciencemediacentre.org/pages/press_briefings/index.php?&showArticle=676
 

WillowJ

คภภเє ɠรค๓թєl
Messages
4,940
Location
WA, USA
Ok, don't think Barts should get any credit for this. There is a woman on MEA FB complaining how her service in Essex won't write any letters for benefit applications. They claim it's some sort of "conflict of interest".

Some posts from her from the thread:


  • She mentions somewhere she may become homeless because of the policy.
wow

thx for info, D
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Has anyone had any joy with Figure 1 yet?

I still can't work it out.

It seems to me that, for Figure 1, values for SMC and GET should be roughly identical when a QALY is valued at around £20,000, because the two lines cross over on the graph at £20,000.

If the relative value of SMC is 'zero' (and I'm not certain about this), then I think that the relative net benefit for GET should be valued at about 'zero' when a QALY is valued at £20,000.
 

Dolphin

Senior Member
Messages
17,567
Figures 1 & 2 come from a statistical technique called bootstrapping. An educational article on it is at: http://www.stat.rutgers.edu/home/mxie/RCPapers/bootstrap.pdf . Basically, it means one doesn't have to know what type of distributions the data takes (e.g. Normal distribution, Poisson Distribution, etc.), one just uses a computer to sample randomly from the data one has, get the computer to do this numerous times, and collate the results. This means one doesn't have to do any fancy mathematical calculations to work out what a model suggests the results should be.

I have come across the technique a few times (i.e. am no expert), however, my impression is the problems advocates are likely to have with the paper are not here. [Although, as I pointed out before, if one removed CBT from the figures, SMC wouldn't have such low percentages (for a lot of the graph)].
 

WillowJ

คภภเє ɠรค๓թєl
Messages
4,940
Location
WA, USA
Societal benefits for CBT & GET probably overstated (and important data not shown)

The paper justifies the cost-effectiveness of CBT & GET on Healthcare costs alone. But it also plays up the enhanced cost-effectiveness using Societal costs, which includes costs of informal care as well as healthcare.

Yes, not only is CBT & GET better than SMC alone, it saves society money too. Or maybe it doesn't, depending on the assumptions used.

The major contributor to CBT/CFS Societal savings were from Informal Care, since lost employment was not significantly different between any of the groups. The authors chose to value this informal care using national mean earnings of £14.60 an hour. That's not real money, it's what the informal carers would earn if they worked instead of providing care, and were payed the average national wage. A more conservative way to value this is by using the minimum legal wage in the UK of £5.93 an hour. The authors looked at this option in their 'sensitivity' analysis and reported that "it did not have a large impact on cost-effectiveness".

'Not a large impact' isn't a very precise term, particularly when CBT and GET didn't have a large impact on QALY outcomes either. No data is presented, though under 'Limitations' the authors say:

This may not be entirely correct. Table 6 shows the Societal cost savings for CBT & GET relative to SMC, using national mean wages to value savings in informal care. However, as the figures below show, those savings vanish when the informal care is valued at the miminum wage instead:

View attachment 3659

This looks rather different. GET now has a cost of £225 (as opposed to a saving of £472) while CBT is neutral instead of saving nearly £700. So the authors claim that:

doesn't apply to the minimum wage scenario. Similarly, the claim that CBT & GET 'dominate' (i.e. is more effective and saves money) SMC doesn't apply using the minimum wage scenario. And these figures are hardly consistent with the claim that "results were robust for alternative assumptions".

The authors could easily have mentioned the impact of using the mimimum wage in the paper rather than saying the sensitivity analysis didn't reveal 'large' differences. Or they could have put the data in an online appendix. It doens't look good when unflattering data is hidden, and the value of sensitivity analysis is diminished if the results of it aren't fairly or adequately reported.

how likely is it that carers would make the mean national wage? shouldn't they use the mean wage of women? (more carers would tend to be women) (which would account for those who might chose not to work anyway and those who would get paid less - or do you not have a men/women wage difference in UK?) and rather than using mean, shouldn't they use a median or something like that?

ETA: I see this has been partially mentioned already -
Mean average earnings have traditionally been pushed up by bonuses in the city. Most people use median (typical) earnings. In april 2011 the median hourly rate excluding overtime and including men, women, full and part time workers was £11.15.

http://www.ons.gov.uk/ons/dcp171778_241497.pdf

I've looked at the source they quote since I was having trouble finding a 14.60 mean wage. The median figure in their 2010 reference seems to be 11.14
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia

user9876

Senior Member
Messages
4,556
how likely is it that carers would make the mean national wage? shouldn't they use the mean wage of women? (more carers would tend to be women) (which would account for those who might chose not to work anyway and those who would get paid less - or do you not have a men/women wage difference in UK?) and rather than using mean, shouldn't they use a median or something like that?

ETA: I see this has been partially mentioned already -

Using the mean seems strange.

If it is true that ME is more prevelant in men then women I'm not sure it makes sense to use the mean or median wage for women.
An alternative option would be to use the wage paid to someone working as a carer. I suspect this is just over the minimum wages.

In terms of societal costs spending money on carers isn't that big a cost, money recirculates around the economy. They really big cost is the opportunity cost - that is what would people with ME and their families be able to do if they weren't ill. These are very hard to estimate and I suspect won't change with small changes in levels of disability,