Andrew Gelman The PACE trial and the problems with discrete, yes/no thinking

[Cheshire]

As he promised, new blog post from Andrew Gelman.

But today I want to talk about something slightly different, which is the flawed way that research results get interpreted and translated into policy.

There are big problems here, which I attribute in part to one of our favorite villains, deterministic thinking.

http://andrewgelman.com/2016/01/06/...l-inference-resolve-reverse-causal-questions/

 

[sarah darwins]

Not sure I'm thrilled with that post. It seems to be steering the argument into rather abstract, murky waters, the kind where the BPS crowd thrive.

In the end, there were some much more fundamental problems with PACE, not least that it recruited using bogus diagnostic criteria crafted to fit the same theory the PACE researchers were trying to confirm.

I think we should be kicking the legs out from under the whole pseudo-scientific edifice behind PACE rather than getting too far into questions of which conclusions might be drawn from such a study (when the answer is none, ever).

 

[Esther12]

I was hoping for more on the mediation paper's statistics. It seemed like he jumped to those more abstract points before having really understood a lot of the controversies around PACE.

where the BPS crowd thrive.

LOL - yeah.

 

[A.B.]

Same feeling here. Missing the elephant in the room while focusing on some minor (if relevant) issue. Getting the feeling he can't quite believe it's really as bad as claimed.

The question of what conclusions should be drawn is important but the question of whether we can draw any conclusions at all even more so.

How bizarrely inappropriate the entire BPS model is should also raise eyebrows.

 

[Jonathan Edwards]

I am coming to think that Andrew Gelman has missed the whole point. It really does seem that, like the psychiatrists, he does not see why PACE is NOT a gold standard trial. Randomisation and statistical significance are only some of the things you need for a useful study. A Rolls Royce with no brake pads is not much use to you.

 

[A.B.]

Maybe they're just not used to such blatant incompetence and assume the criticism, while being justified, is probably also exaggerated. Oh well.

Also frustrating that this crowd, while being very concerned about the quality of research, seems entirely unconcerned with whether claims of psychogenic illness are even valid logically and scientifically. There's one reason why so much research in psychology and psychiatry is piss poor. They should be concerned about this as well.

 

[Sasha]

I am coming to think that Andrew Gelman has missed the whole point. It really does seem that, like the psychiatrists, he does not see why PACE is NOT a gold standard trial. Randomisation and statistical significance are only some of the things you need for a useful study. A Rolls Royce with no brake pads is not much use to you.

I think that point really needs making on his blog - I hope you'll post there.

One or two other people have said it but I think it needs reinforcing. He's potentially a useful ally but he doesn't seem to appreciate quite the extent to which PACE is useless.

 

[Jonathan Edwards]

I think that point really needs making on his blog - I hope you'll post there.

One or two other people have said it but I think it needs reinforcing. He's potentially a useful ally but he doesn't seem to appreciate quite the extent to which PACE is useless.

I think I have already said it, Sasha. I am not sure how useful an advocate is if they don't have a grasp of the problem.

 

[Sasha]

I think I have already said it, Sasha. I am not sure how useful an advocate is if they don't have a grasp of the problem.

Ah well. I've left a comment.

 

[A.B.]

Ah well. I've left a comment.

An excellent comment.

 

[Kyla]

yeah, I have a few issues with this one as well.

Is anyone else having an issue with commenting? I've tried three times and not gotten the comment posted (and the previous ones were on older posts showing newer comments that would have come long after)

I don't think I posted anything offensive or even questionable, so not sure if this is a moderation issue or something else.

 

[sarah darwins]

Also frustrating that this crowd, while being very concerned about the quality of research, seems entirely unconcerned with whether claims of psychogenic illness are even valid logically and scientifically. There's one reason why so much research in psychology and psychiatry is piss poor. They should be concerned about this as well.

Right on. That's what I was trying to say.

If I postulate that shape-shifting lizard people are sneaking into houses at night and that their toxic emissions are making some people ill, it doesn’t matter how well-designed my study to test the efficacy of lizard repellent is. Doesn’t matter if I can find some statistical support for those people gaining some improvements in the presence of lizard repellent. My theory is still nonsense.

 

[alex3619]

I was not particularly happy with the blog either. However I don't think it demonstrates a lack of understanding.

Let me boil it down. Even without considering methodological problems, if you presume the study data is right, then you still cannot be sure enough of the findings to use them in public policy.

So best case scenario is you still cannot use the findings in formulating policy. Once you consider other things it just gets worse from there.

 

[Sasha]

An excellent comment.

Thanks! Got a nice comment from Andrew and a correction (so I've asked for another one!).

 

[Bob]

yeah, I have a few issues with this one as well.

Is anyone else having an issue with commenting? I've tried three times and not gotten the comment posted (and the previous ones were on older posts showing newer comments that would have come long after)

I don't think I posted anything offensive or even questionable, so not sure if this is a moderation issue or something else.

No problem for me. He usually accepts posts quickly, and he doesn't seem easily offended by criticisms. Sometimes comments to a blog can go into a spam box for no reason, only to be discovered by the blog owner weeks later. Otherwise perhaps clear your browser cache just in case it's a minor bug on your system.

 

[Sasha]

Good comment by someone called "Zach" (who also made some good comments before, IIRC):

http://andrewgelman.com/2016/01/06/...olve-reverse-causal-questions/#comment-257824

 

[Kyla]

No problem for me. He usually accepts posts quickly, and he doesn't seem easily offended by criticisms. Sometimes comments to a blog can go into a spam box for no reason, only to be discovered by the blog owner weeks later. Otherwise perhaps clear your browser cache just in case it's a minor bug on your system.

hmmm. have tried in two different browsers on two devices.

Now I'm afraid I've been labelled a "vexatious" "militant"

Maybe I will email him and ask if I have been spam-filtered.

 

[Bob]

Now I'm afraid I've been labelled a "vexatious" "militant"

Hehe... I know you're not being entirely serious, but isn't it ridiculous for such a thought to cross our minds, however fleeting, after posting very reasonable comments.

But what I've noticed about the comments after Andrew's blog is that he engages with everyone respectfully and with an open & curious mind, and there is a meaningful dialogue going on. That's instead of the usual situation whereby we are constantly banging our heads against a brick wall trying to correct ignorance and prejudice. It's so refreshing to read the comments section.

I know that Andrew didn't quite hit all the right notes on this blog, but he seems genuinely interested in what patients have to say, and receptive to information, and not at all defensive. Very refreshing.

I hope he continues to maintain an interest in the subject. As he learns more, he might become bolder in his criticisms. I'm not sure how many read his blog but he's got an academic following, and he might be influential.

 

[Kyla]

well, he doesn't seem to have an email address posted on the site, so I can't ask why my comments aren't going through. oh well

FWIW Here was the comment I was trying to post on today's blogpost
My main concerns with this post were that he seems to have the common misperception that GET and CBT for ME is harmless even if it isn't effective. and the related point that often goes with this, that it the only option available.

If anyone wants to comment with any part of this, or the whole thing, feel free:

I have been really enjoying your continued PACE analysis, but I have to say I have a few issues with some of the finer details of this one. I of course understand you are not an expert in the ME&CFS field and the politics, history and medical findings are quite a bit to wade through, but hope you will indulge me to point out a few things...


In terms of PACE based therapies, the objections patients and ME clinicians have (beyond the poor science) is not just that the specific GET/CBT proposed is ineffective, but that it is in fact harmful to a great number of patients. The fact that exercise and a bit of talking therapy sound on their face to be something that couldn't possibly cause harm is a large part of the problem.

There is a large body of literature showing abnormal response to exertion in ME&CFS - so much so that the Institute of Medicine has proposed renaming the disease to 'Systemic Exertion Intolerance Disease'.

For an overview of some of the exercise/muscle/exertion findings try:
http://www.jacobspublishers.com/images/Physiology/J_J_Physiology_1_2_007.pdf


Every survey of ME patients ever done has shown the majority of patients reporting harms from GET, the recent MEA report shows 74% saying GET worsened their condition, and if you review the qualitative section at the end there are a large number of patients (including a few from the PACE trial) describing how GET rendered them permanently bedbound or wheelchair dependent. see: http://www.meassociation.org.uk/wp-...No-decisions-about-me-without-me-30.05.15.pdf

and in fact in the few cases where Objective measures have been used in GET/CBT studies the results have been negative for GET, even in studies that are heavily skewed to a 'biopsychosocial' approach: https://www.researchgate.net/public...atigue_syndrome_The_role_of_physical_activity

The Belgium government has dismantled its GET&CBT program based on results they collected showing these therapies had negative effects on employment rates and other disability measures.

In the PACE trial specifically, the “Fitness” data for GET – one of the original outcome measures – also showed GET having the worst outcomes of all trial arms. It was only ever published in a tiny graph in the supplemental materials of one of the papers, and it is impossible to tell in this format if it rose to the level of statistical significance, but it is noteworthy that this has never been addressed by the authors. In particular as an exercise therapy like GET should have improved “fitness” (in this case measured by a step-test), not worsened it.


Tom Kindlon also has some excellent analysis of harms relating to PACE specifically :
https://scholar.google.com/citations?user=CWv7ODUAAAAJ&hl=en


its worth noting as well, that the PACE investigators used their own non-standard criteria to select patients (The Oxford criteria) which doesn’t require Post-exertional symptoms, neurological symptoms, or autonomic irregularities, and requires ‘fatigue’ to be the primary complaint. They also cherry picked the 640 trial subjects from a pool of 3158 patients. So even if their results were valid, it is not a given that they can be expanded to ME or CFS patients who fit different criteria and experience Post-exertional malaise. Both the IOM report and the NIH P2P report have suggested these criteria (Oxford) should not be used for research as this overly broad definition is likely to “impair progress and cause harm”.

In terms of large RCTs, I'm not sure if PACE was indeed the largest, but would have to verify

I believe Ampligen trials were comparable size. And in fact an FDA review of Ampligen has found it to be safe, though the FDA was not convinced of efficacy despite 40% of patients reporting substantial improvement. And while there have been a lot of mis-steps in regard to Ampligen, one could argue that in this case pharmacological trials are being held to a much higher evidence standard than non-pharmacological treatments. http://www.cortjohnson.org/blog/2015/02/05/congressional-hearing-ampligen-roadblocks-fda-called/

The ongoing phase-3 Rituximab trials are also a large RCT (for M.E.) , and judging by the extremely promising results from three previous trials, will hopefully put some of this debate to bed. Ie – if results are comparable to previous trials, the weak argument of “its all we can offer” would be off the table. Here is a summary of the Rituximab results so far:

http://simmaronresearch.com/2015/07...al-boosts-hopes-for-chronic-fatigue-syndrome/

 

[Mark]

Perhaps that post is over the character limit for comments Kyla. Or if Andrew personally reviews/moderates all comments himself before publishing, perhaps he's saving that one up for bedtime reading.