Good piece by Jerome Burne (independent journalist) on PACE Trial (September 26)

[AndyPR]

The link is wrong somehow, this should work http://healthinsightuk.org/2016/10/...cting-medical-advice-and-what-to-do-about-it/

 

[anciendaze]

Peter White continues to come across as a true Oaf of Hypocrisy, even before people realize the trial was structured in such a way nobody could be said to have been made worse unless the authors wanted to admit this. They were not about to admit such, or to admit that they were cherry-picking 640 patients from an intake of 3158 diagnosed by NHS doctors to get even the questionable results left by reanalysis. Eliminate the negative, accentuate whatever positive you find, and you have anecdotal evidence promoted as the result of an objective, randomized, blinded study. There are problems with all three adjectives.

 

[AndyPR]

Please read and leave comments on the Hippocratic Post and this latest article as he said it is read by doctors and he is trying to get published in mainstream UK media.

I have a comment waiting on moderation on the Health Insight article, I chose to weight it towards MEGA and how the patient reaction is very much informed by the levels of distrust that have built up. The Hippocratic Post requires you to be a medical or healthcare professional to register and comment.

ETA: Health Insight comment is now readable.

 

[Esther12]

I've not looked into the details of all that blog but:

I’m grateful to the indefatigable Dr Zoe Harcombe, a hugely well-informed nutritionist with a clear grip on bio-medical statistics, for this explanation. She cheerfully admitted she didn’t understand the workings of this particular McGuffin, but that didn’t matter. She knew what its underlying assumptions were and what it was designed to do.

Zoe Harcombe gets slagged off by the World Cancer Research Fund here:

http://wcrf-uk.org/uk/blog/articles/2010/11/zoe-harcombe’s-advice-based-solid-scientific-evidence

I'm not well informed enough to say anything about the quality of Harcombe's work/analysis, but maybe worth being cautious unless you do have time to look in to the details?

I thought that the analysis of the BMJ's/media's response to the release of PACE data was good though.

 

[Barry53]

It's a really good article. When faced with such pig-headed obstinacy (refusal to accept blame let alone make reparations), a really good counter strategy is what is currently happening: applying intensive unrelenting peer pressure (but strictly 100% ethical), which although currently on quite a slow ramp up, will snowball I've no doubt. It's just hard to see at the moment because we haven't got to the more dramatic bit yet, but I'm sure it will - provided the pressure stays on.

 

[slysaint]

Not sure but I think this is the same article
http://healthinsightuk.org/2016/09/...fraction-as-effective-as-experts-claim-it-is/
well worth a read

 

[deleder2k]

I'm discussing with a MD about PACE and the reanalysis after the data was made public.

He asks: "Given that the reanalysis is a more correct measure for the true effect of CBT, how do we know that the original protocol way of measuring is more accurate?"

What did they really change, and what does it mean practically that a patient is considered an improver, vs an improver in the original protocol?

I haven't read all about PACE in detail, but I think I remember that an "improver" could really be very sick, and that the difference between improvers and non-improvers are very small. Change of SF36 score? Also some very considered improved even though they got worse, isn't that true? Does anyone have some facts to back this up?

 

[Esther12]

I'm discussing with a MD about PACE and the reanalysis after the data was made public.

He asks: "Given that the reanalysis is a more correct measure for the true effect of CBT, how do we know that the original protocol way of measuring is more accurate?"

What did they really change, and what does it mean practically that a patient is considered an improver, vs an improver in the original protocol?

I haven't read all about PACE in detail, but I think I remember that an "improver" could really be very sick, and that the difference between improvers and non-improvers are very small. Change of SF36 score? Also some very considered improved even though they got worse, isn't that true? Does anyone have some facts to back this up?

I think that going into the details of the recovery criteria makes it clear that the one the PACE team preferred is just absurd.

For 'improvement', it's less simple imo. The fact that it's a nonblinded trial and CBT & GET involved telling patients that these treatments were effective during the trial mean that the relatively minor improvements in self-report outcomes (CFQ and SF36-PF) used to class people are improvers could easily just be a result of response bias (the poor results for the trial's more objective outcomes also indicate that this was a problem). More demanding criteria for 'improvements', like those used in the protocol, reduce the danger of making big claims about efficacy based on minor changes in scores stemming from methodological problems with the trial.

(I'd have to go back the the Lancet paper to check the exact criteria for 'improvement', but there was no requirement that patients reported, for example, an SF36-PF score of at least 50. It was instead that they had to report an improvement of 10 points. 0->10 would count.)

Also, that they refused to release results for outcomes from their protocol is worrying in itself, as it made it hard to compare their results with what they predicted. They had predicted 10% of the SMC only group would count as overall improvers - their results were 10%. They had predicted 60% of the CBT+SMC group would count as overall improvers - their results were 20%. Yet this was sold as being a big success.

 

[Barry53]

What did they really change, and what does it mean practically that a patient is considered an improver, vs an improver in the original protocol?

The following is an extract from Prof. Vincent Racaniello's article "No ‘Recovery’ in PACE Trial, New Analysis Finds":-

For example, the changes created an overlap between trial eligibility criteria for severe disabling fatigue, and the new “normal range”. Trial participants could consequently be classified as recovered without clinically significant improvements to self-reported physical function or fatigue, and in some cases without any improvement whatsoever on these outcome measures. Approximately 13% of participants at baseline simultaneously met the trial eligibility criteria for ‘significant disability’ and the revised recovery criteria for normal self-reported physical function.

There is more.

http://www.virology.ws/2016/09/21/no-recovery-in-pace-trial-new-analysis-finds/

 

[Valentijn]

What did they really change, and what does it mean practically that a patient is considered an improver, vs an improver in the original protocol?

According to the protocol, "improvement" (not "recovery") required a bimodal-scored Chalder Fatigue Scale (CFQ) of 3 or less (out of 11, recruitment required 6 or higher) or a 50% reduction. For the SF36 Physical Functioning subscale, scores had to be at least 75 (out of 100, 65 or lower required for recruitment) or a 50% increase.

In the PACE paper published in 2011, this changed to a 2 point decrease on the likert-scored CFQ (out of 33), and a 8 point increase on the SF36-PF.

So that goes from requiring a 50% reduction on the CFQ to as little as a 6% reduction (2 out of 33 points). SF36-PF went from requiring 15% or greater increase to 12% or greater increase.

SF36 thresholds were always pretty lax for people starting at the low and high end of the scale. But the changes made it very easy for people in the middle of the range to be considered "improved". The bigger effect comes from the CFQ scoring, since the points were tripled and the change needed for improvement was dramatically decreased.

Scores on the SF36-PF would be expected to have clumps for no disability, moderate disability, and severe disability, since there's a lot of overlap in the questions. Hence if someone was improved, you'd expect a bigger increase, since you'd generally do a bit better on all activities. Answering 2 questions as being a bit better, on the other hand, could simple reflect having been borderline for those activities both before and after treatment.

 

[Barry53]

I think it's important to remember that this propoganda is not really aimed at us. They don't expect to be able to win patients over. It is aimed at the general public, at colleagues, at NICE and at funders (these must be our target audience too). The word from the BPS lot will be that "we offered an olive branch even though we were not at fault, we've tried to engage and unite the community but unfortunately a small vocal minority are just very self destructive

I think this take is absolutely right.

 

[deleder2k]

Thanks for the answers.

Do we know whether or not the analysis plan was completed before the data was opened? http://www.wolfson.qmul.ac.uk/image...otocol_based_analysis_final_8th_Sept_2016.pdf


I guy here wrote in March said this "It is questionable whether all changes to the published trial protocol were approved:"

It is questionable whether all changes to the published trial protocol were approved:

While the investigators stated that the recovery criteria was changed before the analysis was conducted [1] and therefore should be regarded as “pre-specified” [2], there is evidence suggesting that it was changed well after the investigators were already familiar with the distribution of scores for each component of the recovery criteria, including access to summary statistics and histograms:

a) Before appearing as part of the revised recovery criteria, the controversial post-hoc ‘normal range’ for fatigue and physical function appears to be first introduced during the peer review stage [3] of the 2011 Lancet publication on the main trial results (well after the authors were unblinded to trial data). [4]

b) The statistical analysis plan outlines the final planned or pre-specified analyses that were approved prior to database lock and unblinding of trial data, but does not mention changes to the recovery criteria. [5] It describes the summary data they had access to early on when preparing the first paper on the main results. It discusses the possibility of future post-hoc changes and exploratory analyses, including those based on trial data.

c) Changes to the primary outcomes were approved by an oversight body (e.g. Trial Steering Committee) [5] but there was no confirmation that the revised recovery criteria was similarly approved, not in the paper itself [1] or the investigators’ response to David Tuller when the issue of changes to the recovery criteria was raised. [4] Note that statements in the PACE trial literature about approval of changes typically refers to the main changes to the primary outcomes, not specifically to the revised recovery criteria. When referring to the overlap between the normal range for fatigue and physical function, with the trial eligibility criteria for severe disabling fatigue, Bruce Levin (Professor of Biostatistics, Columbia University) has stated: “I find it nearly inconceivable that a trial’s data monitoring committee would have approved such a protocol problem if they were aware of it.”. [6]

d) The original primary outcome i.e. positive outcome was the proportion of individuals meeting a threshold, but the new primary outcome was group-level averages; in other words the new primary outcome was not a direct replacement of the original. The closest (but much less stringent) equivalent to the original primary outcome is the secondary outcome for individual-level “clinically useful difference” (CUD). But this was also described in the Lancet publication as a post-hoc analysis [4] and not mentioned in the statistical analysis plan either [5] (group-level CUD is mentioned and approved) so it remains unclear if it was explicitly approved by the Trial Steering Committee.

The PACE investigators have not reported all pre-specified outcomes and have not provided adequate details about the timing of the changes; this is contrary to modern expectations of researchers (e.g. see AllTrials.net and compare-trials.org). A FOIA request to clarify this issue and make sure a possible post-hoc or exploratory analysis was not being misleadingly described as a pre-specified, was dismissed as part of a “vexatious” campaign. [7] The PACE investigators stated that the trial would be conducted in accordance to the Declaration of Helsinki, however, the refusal to clarify the timing of changes to the protocol is equivalent to withholding information about methods, which is against the declaration. [8]

The changes to the protocol were (generally) poorly justified, and the change to the threshold for normal physical function was demonstrably erroneous. The stated justification for this drastic change, incorrectly asserted that about half the general working age population score under 85, but it is actually 17.6%, so the change was unjustified. Note that 92.3% of the working age English population who do not report chronic disabilities score 85 to 100, and 61.4% score the maximum of 100, with scores under 80 being extreme outliers. [9,10] Again, PACE investigators claim that attempts to raise these issues and have them corrected are simply part of a “vexatious” campaign. [11,12]

1. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3776285
2. http://www.virology.ws/2015/10/30/pace-trial-investigators-respond-to-david-tuller
3. http://www.meactionuk.org.uk/whitereply.htm
4. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065633
5. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4226009
6. http://www.virology.ws/2015/10/21/trial-by-error-i/
7. https://www.whatdotheyknow.com/request/timing_of_changes_to_pace_trial
8. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4396123
9. http://link.springer.com/article/10.1007/s11136-014-0819-0
10. http://www.bmj.com/content/350/bmj.h227/rr-16
11. https://ico.org.uk/media/action-weve-taken/decision-notices/2015/1043579/fs_50558352.pdf
12. http://www.thelancet.com/journals/lanpsy/article/PIIS2215-0366(15)00114-5/fulltext

Comment from "Almat": http://www.stats.org/pace-research-sparked-patient-rebellion-challenged-medicine/