• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

David Tuller / Trial by Error: The Science Media Centre’s Desperate Efforts to Defend PACE

Cheshire

Senior Member
Messages
1,129
This week, the Journal of Health Psychology published a special issue containing a raft of commentaries on the PACE trial. Most of them slammed the study for its many, many unacceptable flaws. Not surprisingly, Sir Simon Wessely’s lackeys at the Science Media Centre immediately posted three comments from “experts” lauding the trial and criticizing the JHP commentaries. I thought it might be helpful to deconstruct these rather pathetic efforts at defending the indefensible.

http://www.virology.ws/2017/08/02/t...dia-centres-desperate-efforts-to-defend-pace/
 
Messages
15,786
Good to see the SMC getting slammed. My comment:
The Science Media Centre is a perversion, taking money from the public and also funded by the industries and institutions which they promote and defend. SMC's defense of the indefensible PACE Trial almost certainly results from the money the SMC received from the university of one of the principal PACE investigators. They are a marketing organization, not a independent purveyor of science.
 
Messages
2,087
It means they saw the data (and even could have analyzed it) before they made the changes, rather than declaring the thresholds beforehand. So they could have selected the thresholds which they knew would give them the results they wanted.
But how is that established?

If a recovery criteria requires post hoc analysis, why does changing it, imply they saw the data.

Sorry I am just confused with the terms.

I am trying to figure out the 'evidence' that they saw the data or at least were aware of it.
 
Messages
15,786
I am trying to figure out the 'evidence' that they saw the data or at least were aware of it.
If it's not declared beforehand (such as in the protocol), it's always going to be under suspicion of being done post-hoc. And in the case of the recovery data, the PACE team won't even claim that it was pre-planned or pre-approved, plus their protocol had different standards which they abandoned. So it would be naively generous to assume that the analysis wasn't post-hoc, and it's essentially up to them to prove it if it wasn't.
 
Messages
15,786
One more question, if the recovery threshold was weakened, or indeed any outcome measure, would that change be expected to apply equally to all arms of the trial?
Yes, the same lowered thresholds applied to everyone. But the glory of post-hoc analysis is that researchers can play with the numbers until they find the best ones. So maybe lowering the SF36-PF to 75 doesn't help, and maybe 70 makes pacing look most effective, but 60 comes out looking great for CBT or GET.

It might be interesting to look at the data released from the FOIA to see how other thresholds would have stacked up against the original 85 and the eventual 60, come to think of it :D
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Good to see the SMC getting slammed. My comment:
Here was my comment on David's blog -
The SMC claims to be an independent provider of news and views on science. On numerous occasions its been shown to be engaged in scientific politics, promoting views of research from specific industries and scientific groups, and clearly operates as a public relations organization a lot of the time. Its not a news distributor, its about persuading on matters of science. They even wrote about manipulating views about CFS/ME advocates. This is medical and scientific politics. At best the SMC should be viewed as a partisan think tank.

On arguments from authority, or citing authoritative processes, its designed to persuade without providing evidence. Its also a double edged sword. When the PACE fiasco settles anyone who is supporting PACE will have to face whatever views the scientific and medical communities then take. It would be evidence not only of failure at places like the Lancet, SMC, MRC and NICE, especially in the review process, but also of failure to even recognize they have a problem.

This is such a big scope that I think a fully independent Royal Commission in the UK is required to investigate, and include investigation of NICE and the MRC at the least.
 

Dolphin

Senior Member
Messages
17,567
But how is that established?

If a recovery criteria requires post hoc analysis, why does changing it, imply they saw the data.

Sorry I am just confused with the terms.

I am trying to figure out the 'evidence' that they saw the data or at least were aware of it.
The normal range analyses i.e. SF-36 physical functioning score of 60 or more and Chalder Fatigue questionnaire Likert score of 18 or less were called post-hoc in the Lancet 2011 paper. That means they were calculated after seeing the data. These two normal range analyses were then used in the 2013 recovery paper as alterations to the recovery criteria in the protocol.
 

SilverbladeTE

Senior Member
Messages
3,043
Location
Somewhere near Glasgow, Scotland
What is needed and whether or not it happens are two different things. One thing I will say is that if nobody pushes for change then nothing changes.

Only revolution will change thing in UK, as has been proven, time after time, "Inquiries" are a load of white wash and bollocks, veering things up.
Whole system is involved in grotesque cronyism, nepotism, corruption, and every kind of corruption, perversion and exploitation including treasonous under funding/lunatic cutbacks of personnel of the military and essential civil infrastructure.
If we want genuine democracy, we'll have to fight for it
 
Messages
2,087
Yes, the same lowered thresholds applied to everyone. But the glory of post-hoc analysis is that researchers can play with the numbers until they find the best ones. So maybe lowering the SF36-PF to 75 doesn't help, and maybe 70 makes pacing look most effective, but 60 comes out looking great for CBT or GET.
These two normal range analyses were then used in the 2013 recovery paper as alterations to the recovery criteria in the protocol.

So, are you saying that when the changed the recovery definition it lead to greater results for the non SMC arms only?


Trying to work out how the reanalysis of the recovery paper as per the original protocol, resulted in a non statistical difference between arms, when there was much more of a difference under changed protocol in the original paper.

Another way of saying it, does the argument that lowering the recovery threshold results in more patients beings recovered, hold true - if someone could say it also means the SMC arm should have more patients recovered.

Or are you saying the recovery threshold could have been choosen exactly to give more patients recovered in the CBT and GET arms and not the SMC arm ?

Apologies if this doesnt make sense.
 

Londinium

Senior Member
Messages
178
Another way of saying it, does the argument that lowering the recovery threshold results in more patients beings recovered, hold true - if someone could say it also means the SMC arm should have more patients recovered.

In a blinded trial that would be correct: lowering the threshold for success wouldn't be sufficient in and of itself to skew the results to the treatment arm. In an unblinded trial, where some level of placebo effect is expected, lowering the success threshold might be just enough to capture the increase due to placebo effect in the treatment arm. The control arm (SMC) would not get the same placebo effect* and so lowering the success threshold may well then turn a null result into a successful trial. Reanalysis of the 'recovery' paper shows that had the trial outcome for success not been altered the difference recovery rates for all four arms would not have been statistically significant.

*separate issue, but lots of reasons why SMC as designed would not have given same level of placebo effect.
 
Messages
13,774
I most liked Tuller's response to the MRC comment, which was maybe the one which was least infuriating. He did make me feel more infuriated by it than I was though. With Macleod's comment being so completely rubbish and misguided it's hard to really go through calmly explaining what's wrong with it. It would need everything wrong with the PACE trial to be re-explained.

I wonder if an FOI request could be lodged with Oxford to find out who signed off on the statement, and what evidence they have to support the claims made within it?

Or are you saying the recovery threshold could have been choosen exactly to give more patients recovered in the CBT and GET arms and not the SMC arm ?

They could have been, but we wouldn't be able to know that. The changes made it easier for those in all groups to be classed as 'recovered', but particularly favoured those groups most likely to have positively biased the self-report outcomes that were used to define recovery.