• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Davis, Racaniello et al: An open letter to PLoS One

mango

Senior Member
Messages
905
Dear PLoS One Editors:

In 2012, PLoS One published “Adaptive Pacing, Cognitive Behaviour Therapy, Graded Exercise, and Specialist Medical Care for Chronic Fatigue Syndrome: A Cost-Effectiveness Analysis.” This was one in a series of papers highlighting results from the PACE study—the largest trial of treatments for the illness, also known as ME/CFS. Psychologist James Coyne has been seeking data from the study based on PLoS’ open-access policies, an effort we support. [...]

http://www.virology.ws/2016/05/23/an-open-letter-to-plos-one/
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Just to clarify, while this is the paper where the authors are refusing to make the dataset available (depsite agreeing to as a condition of publishing), the issue raised by Prof Racaniello and colleagues is diffferent, and points to a significant error in the paper.

Basically, the paper claims that CBT/GET are cost-effective regardless of assumptions made about the economic cost of care provided by friends and family. Yet it appears this isn't the case at all.

Here's the bit of David Tuller's analysis the letter cites:

The PLoS One paper argued that the graded exercise and cognitive behavior therapies were the most cost-effective treatments from a societal perspective. In reaching this conclusion, the investigators valued so-called “informal” care—unpaid care provided by family and friends–at the replacement cost of a homecare worker. The PACE statistical analysis plan (approved in 2010 but not published until 2013) had included two additional, lower-cost assumptions. The first valued informal care at minimum wage, the second at zero compensation.
The PLoS One paper itself did not provide these additional findings, noting only that “sensitivity analyses revealed that the results were robust for alternative assumptions.”

Commenters on the PLoS One website, including [patient] Tom Kindlon, challenged the claim that the findings would be “robust” under the alternative assumptions for informal care. In fact, they pointed out, the lower-cost conditions would reduce or fully eliminate the reported societal cost-benefit advantages of the cognitive behavior and graded exercise therapies.

In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings. However, McCrone did not explain or even mention the apparently erroneous sensitivity analyses he had cited in the paper, which had found the societal cost-benefit advantages for graded exercise therapy and cognitive behavior therapy to be “robust” under all assumptions. Instead, he argued that the two lower-cost approaches were unfair to caregivers because families deserved more economic consideration for their labor.

“In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial,” McCrone wrote. “Likewise, to assume it only has the value of the minimum wage is also very restrictive.”

In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were “robust” for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.

“Remember it was the investigators themselves that chose the alternative assumptions,” wrote Kindlon. “If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.

They are asking for the lead author, Dr Paul McCrone, to either publish the evidence for the claim of a 'robust' finding, or for PLoS One to issue its own correction:

PLoS One has an obligation to question Dr. McCrone about the contradiction between the text of the paper and his subsequent comments, so he can either provide a reasonable explanation, produce the actual sensitivity analyses demonstrating “robustness” under all three assumptions outlined in the statistical analysis plan, or correct the paper’s core finding that CBT and GET are “cost-effective” no matter how informal care is valued. Should he fail to do so, PLoS One has an obligation itself to correct the paper, independent of the disposition of the issue of access to trial data.
 
Last edited:

Tom Kindlon

Senior Member
Messages
1,734
I believe it was @Simon who first noticed how much they had valued informal care.

Here's a quick comment I wrote elsewhere which perhaps summarises it very briefly.
---

This open letter is by five US academics

It's about the £5 million PACE Trial and the investigators' questionable claim that CBT and graded exercise therapy were found to be cost-effective.

It was really outrageous that they wrote:

"Fourth, we made assumptions regarding the value of unpaid care from family and friends and lost employment. However, sensitivity analyses revealed that the results were robust for alternative assumptions."

They (well, Paul McCrone) have subsequently admitted that this is not the case and the results are very different with alternative assumptions.
 

A.B.

Senior Member
Messages
3,780
Can anyone simplify this? I don't understand the argument being made :confused:

The paper reported that CBT and GET were cost effective under all assumptions. Three different assumptions were made for "informal care", which would be care provided by family members:

1. That informal care was valued at the cost of a homecare worker.
2. That informal care was valued at the minimum wage.
3. That informal care was valued at zero.

Readers, including Tom Kindlon, pointed out that under assumption 2 and 3, CBT and GET would not be cost effective.

The lead author conceded that this was correct, but no steps were taken to correct the paper, which still reports that CBT and GET are cost effective no matter how informal care is valued. Instead the lead author began arguing that valuing informal care at minimum wage or zero would be controversial.

James Coyne had previously requested the relevant data in order to perform a cost effectiveness analysis. The authors have refused, going against the journals data sharing policy.

The letter points out that, since the lead author has conceded that the conclusion is wrong, there is no need to actually perform an independent cost effectiveness analysis. The lead author must now either provide an explanation or correct the paper. If he refuses, the journal will have to correct the paper.

At least that's how I understand all this.

PS: if we want to simplify it even further, we could say that readers have apparently found an error with the statistical analysis, but the authors have taken no action to correct it after admitting its existence (whilst refusing to release the data for an independent analysis).
 

Dolphin

Senior Member
Messages
17,567
The paper reported that CBT and GET were cost effective under all assumptions. Three different assumptions were made for "informal care", which would be care provided by family members:

1. That informal care was valued at the cost of a homecare worker.
They said:

Alternative methods exist for valuing informal care, with the opportunity cost and replacement cost approaches being the most recognised. We adopted the former and valued informal care at £14.60 per hour based on national mean earnings

Office for National Statistics (2010) Annual survey of hours and earnings (ASHE) - 2010 results. Available: http://www.statistics.gov.uk/statbase/Product.asp?vlnk=1951. Accessed 2011 Oct 3

I think £14.60 is national mean earnings for everyone, not homecare workers.

Mean earnings are higher than median earnings as they include millionaires, etc.

By way of comparison, the minimum wage was £5.93. Homecare workers don't tend to be that well paid.
 
Last edited:

A.B.

Senior Member
Messages
3,780
They said:

I think £14.60 is national mean earnings for everyone, not homecare workers.

Mean earnings are higher than median earnings as they include millionaires, etc.

Thanks for pointing this out. I'm going by the letter which says

In reaching this conclusion, the investigators valued so-called “informal” care—unpaid care provided by family and friends–at the replacement cost of a homecare worker.
 

Large Donner

Senior Member
Messages
866
The PACE authors have repeatedly cited this inaccurate claim of cost-effectiveness to justify their continued promotion of these interventions.

Yet the claim is not supported by the evidence, and it is not necessary to obtain the study data to draw this conclusion.

Why are they making the claim in bold (my bold)?

Also for clarity, the PLOS one paper data issue raised by James Coyne being processed by FOI is still unsettled and the QMUL FOI data release is to do with the original Lancet paper. Is that correct because Its getting very confusing now?
 

Large Donner

Senior Member
Messages
866
I think £14.60 is national mean earnings for everyone, not homecare workers.

Mean earnings are higher than median earnings as they include millionaires, etc.

By way of comparison, the minimum wage was £5.93. Homecare workers don't tend to be that well paid.

The workers are not necessarily paid much more than minimum wage but they are usually employed by an agency who charge a considerable amount more than they pay the worker. They also often get paid mileage and transport costs.
 

A.B.

Senior Member
Messages
3,780
This statistical analysis plan was not published until over a year after the cost-effectiveness paper was published. This is why we didn't know how they had promised to analyse the data when the cost-effectiveness paper came out initially.

They do have a habit of selectively reporting information. The fitness graph (which totally contradicts any claims of improvement) took years to be published. The data is still withheld.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Also for clarity, the PLOS one paper data issue raised by James Coyne being processed by FOI is still unsettled and the QMUL FOI data release is to do with the original Lancet paper. Is that correct because Its getting very confusing now?
All FOI requests for PACE data are processed by QMUL because they hold all the trial data relating to all publications. So QMUL would deal with requests for data from the original Lancet paper or the PLOS paper. James Coyne has asked PLOS to obtain the data from the authors, as part of their contractual agreement with PLOS. I think James also made a request to QMUL for data but it wasn't a formal FOI request; QMUL converted his request to an FOI request then rejected it. Clear as mud?
 
Last edited:

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Yet the claim is not supported by the evidence, and it is not necessary to obtain the study data to draw this conclusion.
Why are they making the claim in bold (my bold)?
That's answered in the blog...
Virology blog said:
In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings.
Virology blog said:
Given that Dr. McCrone, the lead author, directly contradicted in his comments what the paper itself claimed about sensitivity analyses having confirmed the “robustness” of the findings under other assumptions, it is clearly not necessary to scrutinize the study data to confirm that this central finding cannot be supported.
 

Large Donner

Senior Member
Messages
866
All FOI requests for PACE data are processed by QMUL because they hold all the trial data relating to all publications. So QMUL would deal with requests for data from the original Lancet paper or the PLOS paper. James Coyne has asked PLOS one to obtain the data from the authors, as part of their contractual agreement with PLOS. I think James also made a request to QMUL for data but it wasn't a formal FOI request; QMUL converted his request to an FOI request then rejected it. Clear as mud?

Thanks Bob. The request that is going through appeal now was filed by whom and against which study in which publication?
 

Yogi

Senior Member
Messages
1,132
All FOI requests for PACE data are processed by QMUL because they hold all the trial data relating to all publications. So QMUL would deal with requests for data from the original Lancet paper or the PLOS paper. James Coyne has asked PLOS one to obtain the data from the authors, as part of their contractual agreement with PLOS. I think James also made a request to QMUL for data but it wasn't a formal FOI request; QMUL converted his request to an FOI request then rejected it. Clear as mud?


JC made the request to Paul McCrone (this Health economist) at Kings College who automatically regarded it as a FOI.
https://jcoynester.wordpress.com/20...ating-refusal-to-release-the-pace-trial-data/
 

Yogi

Senior Member
Messages
1,132
I have not read this PLOS One paper in detail so if anyone knows of a good summary which explains the issues and regarding the QALY it would be good?


“In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial,” McCrone wrote. “Likewise, to assume it only has the value of the minimum wage is also very restrictive.”

In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were “robust” for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.

“Remember it was the investigators themselves that chose the alternative assumptions,” wrote Kindlon. “If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.”

I have a question.

Carers allowance in UK is per week £62.30 for full time carers. This is the ridiculously low amount that full time carers (minimum 35 hours) are valued at in the UK. When the PACE trial was done it would have been about £45-55 per week due to inflation. This is the official rate.


Who specifically sets the rate of £62.30 you may ask?

Answer: The DWP who were the funders of the PACE trial and who Peter White works for!

The hourly rate at 2016 levels is £1.78.

Why was that not used?

https://www.gov.uk/carers-allowance/overview
 
Last edited:
Messages
13,774
Why are they making the claim in bold (my bold)?

I thought it was a mistake to emphasise that access to the data was not needed in the way they did.

I do understand wanting to make clear that what is already in the public sphere allows us to make certain criticisms confidently, but getting access to PACE data is going to be important for drawing other people's attention to these problems imo.

A small quibble with otherwise great work though. Thanks to all the signatories.

edit: Also I wrote this before half the posts on this page had been made... slow typer!
 
Last edited:

panckage

Senior Member
Messages
777
Location
Vancouver, BC
1. That informal care was valued at the cost of a homecare worker.
2. That informal care was valued at the minimum wage.
3. That informal care was valued at zero.

Readers, including Tom Kindlon, pointed out that under assumption 2 and 3, CBT and GET would not be cost effective.
I'm sorry i must be really stupid but there still seems to be something missing from this explanation. If the more expensive care (1.) is considered cost effective how could lowering the cost not be considered cost effective?