• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

"Tug of War: Epic battle over data in controversial paper on chronic fatigue syndrome"

Kyla

ᴀɴɴɪᴇ ɢꜱᴀᴍᴩᴇʟ
Messages
721
Location
Canada
http://andrewgelman.com/2015/12/18/28362/

good short blog post on PACE-gate.
Mostly a summary of correspondence with James Coyne from someone outside the issue.

I'm unfamiliar with the author or with this blog but from a quick look Andrew Gelman is a statistician, and the blog seems to mostly cover statistical methods in medicine & social science (feel free to correct me if someone is more familiar)

Nice to see people outside of ME&CFS research taking an interest.

As the article is quite short I won't post an excerpt. click through here:
http://andrewgelman.com/2015/12/18/28362/
 

Dolphin

Senior Member
Messages
17,567
http://andrewgelman.com/2015/12/18/28362/

good short blog post on PACE-gate.
Mostly a summary of correspondence with James Coyne from someone outside the issue.

I'm unfamiliar with the author or with this blog but from a quick look Andrew Gelman is a statistician, and the blog seems to mostly cover statistical methods in medicine & social science (feel free to correct me if someone is more familiar)

Nice to see people outside of ME&CFS research taking an interest.

As the article is quite short I won't post an excerpt. click through here:
http://andrewgelman.com/2015/12/18/28362/
What Coyne may be referring to is the comments by the Barts Chronic Fatigue Service (headed by Peter Denton White) on the draft NICE guidelines.

Parking badge one is the first comment here:
http://forums.phoenixrising.me/inde...ice-guidelines-insight-into-their-views.1239/
(If you ever want to find it, it's in my signature!)
 

Cheshire

Senior Member
Messages
1,129
This Andrew Gelman blog post, and the comments that follow are well worth reading.

Yes, in this comment in particular Robert seems to point to very interesting things.

Thanks for an interesting blog. Interesting to hear your views on mediation analyses. It seems to me that there is plenty of room for guess-work, false assumptions, error and statistical noise in mediation analyses, perhaps especially in the case of analyses that use self-report questionnaires (that may be vulnerable to response bias, therapist allegiance and other biases).

A brief note about the particular Secondary Mediation Analysis in question…

Chalder et al. [1] used the “single mediation model” for their methodology, which is explained in detail in a book by MacKinnon [2]. Explaining the methodology MacKinnon says a temporal separation between variables must be observed (i.e. changes in mediating variable must occur before changes in the mediated variable) for a mediation effect to be empirically and robustly established.

Chalder et al. were working to this model and acknowledged that they failed to establish a temporal or causal relationship between variables, and therefore did not empirically establish a mediation effect: “Given the pattern of change in the mediators was similar to the pattern of change in the outcomes it is possible that the variables were affecting each other reciprocally”.

However, despite the lack of empirical evidence to support a mediation effect, the investigators concluded that they had established mediation effects, e.g: “Our main finding was that fear avoidance beliefs were the strongest mediator for both CBT and GET.”

The study’s conclusion relied upon an assumption that the investigators’ favoured hypothetical model of illness for ME/CFS has a robust empirical evidence base and is applicable to this study. The hypothesis is based upon the idea that symptoms and disability in ME/CFS are perpetuated by false illness beliefs, fear, and an avoidance of activity.

However, the prestigious National Academy of Medicine (formerly known as the Institute of Medicine) recently released a comprehensive report [3] into ME/CFS that rejected such a hypothetical model of illness, and unambiguously concluded that ME/CFS does not have a psychological or cognitive-behavioural basis, but is an organic illness that requires biomedical research.

Chalder et al. discussed the possibility that more frequent measurements may have potentially demonstrated a temporal separation between the variables, and therefore a mediation effect. However, this raises the possibility of whether changes in the primary outcome variables (self-report physical function and fatigue) may, in fact, have occurred before changes in the presumed mediator variables. Such an outcome would entirely contradict the investigators’ premature conclusions. According to MacKinnon [2] and Wiedermann et al. [4], unexpected outcomes should not be ruled out.

Chalder et al. concluded that symptoms and physical impairment, in ME/CFS patients, are mediated by activity avoidance and other factors. However, from a common sense point of view, this seems like rather a convoluted conclusion, and it seems more likely that symptoms would be the direct cause of activity avoidance in any illness. And physical impairment is identical to activity avoidance. To conclude that activity avoidance causes fatigue (rather than fatigue being a direct cause of activity avoidance), is similar to concluding that a person has flu because they’ve taken a day off work, rather than the obvious conclusion that they’ve taken a day off work because they have flu.

In the case of fatigue, it seems reasonable to consider the possibility that, as the symptom of fatigue fluctuates, patients may intuitively or rationally adapt their activity levels according to what is comfortable and safe. i.e. patients reduce activity levels because they are fatigued. The investigators have concluded that patients are fatigued because they have reduced activity levels.

Perhaps patients’ perspectives and insights would help clarify the issues but, unfortunately, patients were not consulted for this study.

References:

1. Chalder T, Goldsmith KA, White PD, Sharpe M, Pickles AR. Rehabilitative therapies for chronic fatigue syndrome: a secondary mediation analysis of the PACE trial. Lancet Psychiatry 2015; 2: 141–52.

2. MacKinnon DP. Introduction to Statistical Mediation Analysis. Taylor and Francis: New York 2008.

3. http://iom.nationalacademies.org/Reports/2015/ME-CFS.aspx

4. Wiedermann W, von Eye A. Direction of Effects in Mediation Analysis. Psychol Methods 2015; 20: 221-44.

Intuitively, I think there is something of importance here, but my brain is struggling to make any sense of it. Maybe more knowledgeable members could help! @Woolie @user9876 and many more!
 
Messages
3,263
I found this an interesting comment, @Cheshire. I don't know much about mediation analysis I'm afraid. However, technically, there was in fact a temporal separation in the measures used. The mediator ("fear avoidance" etc.) was measured at 12 weeks after trial commencement, and the outcome (scores on self reported fatigue and physical function) was measured at at 52 weeks.

But having said that, the figures suggest that both mediators and outcomes actually changed at around the same time - around 12 weeks - so measuring the outcome at 52 weeks doesn't add very much. It would seem to me to demonstrate mediation, you need to do more than just use measures taken at different time points. You'd need to show that there was an early time point where the mediator changed - but not the outcome variable - and another, later time point when the outcome then changed. And it doesn't seem to me that the mediation analysis was set up to show this. And the figures suggest this actually wasn't the case. But maybe others may know more?

The other problem I see with the paper was that there was a pretty big fishing expedition going on. There are around 10 possible mediator variables (anxiety, catastrophisation, all-or-nothing behaviour, etc., etc.). There were also two preferred treatments - so there were 10x2= 20 possible effects that could be significant. In other words 20 chances to get some sort of "hit". Normally, you would need to correct for this by requiring a more stringent level of significance for your analysis (it would have to be 20 times lower). I'm not sure if this also applies to mediation analysis, perhaps someone else may be able to comment?

And of course, the elephant in the room: if the study design was dodgy in the first place (which I think it was), then doing a mediation analysis is kind of meaningless...
 

RustyJ

Contaminated Cell Line 'RustyJ'
Messages
1,200
Location
Mackay, Aust

Dolphin

Senior Member
Messages
17,567
I tried to follow the links to the NICE site, but they send me to the home page, not the relevant documents. Is there a better link?
If you Google a phrase, they should come up (I found one with an active NICE page in the last week or two). The comments are spread across several files as I recall.
 

RustyJ

Contaminated Cell Line 'RustyJ'
Messages
1,200
Location
Mackay, Aust
If you Google a phrase, they should come up (I found one with an active NICE page in the last week or two). The comments are spread across several files as I recall.

Huh, I've done it before plenty of time, but just didn't think of it this time.
 
Messages
3,263
PACE mediatin comparison.jpg
@Woolie, have you considered posting your comments on the blog? I think they'd nicely add to the discussion. If you have any questions, the blogger seems well informed and seems to be willing to answer questions.
@Bob, I don't feel sure enough of this stuff to post on the blog. But anyone, feel free to borrow anything and use it yourself if its useful.

I'm looking again at the figures. Maybe things aren't quite as negative as I suggested earlier. Here they are.

The ticks on the base axis are time points (0 weeks, 12 weeks, 24 weeks and 52 weeks).

You can see that the fear avoidance scores started separating at 12 weeks, but in fact looking now at it again, those for the outcomes (self-rated fatigue and physical function) don't really start separating till 24 weeks.

What do people think?
 

biophile

Places I'd rather be.
Messages
8,977
What do people think?

I think the evidence for no improvement to the step test etc means the correlation of fear avoidance with fatigue and physical functions scores could be largely meaningless. They talk about how patients need to fear activity less and do more activity, yet the evidence shows patients aren't doing more activity as a result even if they fear activity less (or say so on a questionnaire). No surprise that that 6MWD scores are (inversely) correlated with fear avoidance either.
 

user9876

Senior Member
Messages
4,556
View attachment 14036
@Bob, I don't feel sure enough of this stuff to post on the blog. But anyone, feel free to borrow anything and use it yourself if its useful.

I'm looking again at the figures. Maybe things aren't quite as negative as I suggested earlier. Here they are.

The ticks on the base axis are time points (0 weeks, 12 weeks, 24 weeks and 52 weeks).

You can see that the fear avoidance scores started separating at 12 weeks, but in fact looking now at it again, those for the outcomes (self-rated fatigue and physical function) don't really start separating till 24 weeks.

What do people think?

The problem is they are answers to questions which may be similar in content some might have a leading/lagging effect within the therapy. I think it may start off with a message that patients shouldn't fear activity. I think I would prefer the graphs to be drawn in terms of question answers changed rather than on more abstract score lines - also to see the distributions. As Biophile says the figures don't seem to be anchored in objective measures (which I think they treated as mediators rather than outcomes).

My overall impression of mediation analysis and in particular this one (I've not read others) is that it is throwing complex stats at a problem and just looking for a result when they should be doing more work to form a detailed hypothesis and verify it experimentally using simpler techniques. I would be particularly worried that the base assumptions in regression and mediation analysis are not met by the data sets. In particular around linearity of scales and independence of variables (Do individual questions on the different questionnaires correlate).
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
@Bob, I don't feel sure enough of this stuff to post on the blog. But anyone, feel free to borrow anything and use it yourself if its useful.
Without the data being available, I doubt if any us feel confident about interpreting their analysis, even if we were knowledgeable about mediation analyses. I think it's probably impossible to interpret the data just by using the graphs.

The ticks on the base axis are time points (0 weeks, 12 weeks, 24 weeks and 52 weeks).

You can see that the fear avoidance scores started separating at 12 weeks, but in fact looking now at it again, those for the outcomes (self-rated fatigue and physical function) don't really start separating till 24 weeks.

What do people think?
There's so much that could be said about this study, but to stick to some basic data...

Fear-avoidance was said to have the largest purported mediation effect, but the size of the mediation effect for fear-avoidance ranges from 17% to 61%. 61% was the absolute maximum effect for any variable in the study.

Are we looking at the purported mediation effects on the primary outcome measures after 12 weeks? (i.e. the mediation variables can't have a mediating effect until after the interventions have affected the mediator variables, which are measured at 12 weeks.) If so, I wonder if the improvements in the primary outcome measures between 12 & 52 weeks are clinically useful, taken in isolation from the improvements before 12 weeks? If the improvement after 12 weeks were not clinically useful, then we're looking at the effects the mediator variables would have on a non-useful clinical benefit seen in the primary outcome between 12-52 weeks.

The effects of fear-avoidance range from 17% to 61%, so we might be looking at as little as 17% of a non-useful change in outcomes after 12 weeks.

Even if we're looking at the whole improvements between 0-52 weeks, the mediation analysis only applies to 15% of PACE trial participants, and it only applies to a very modest mean improvement. If isolated, the individual mediation effects, would probably not have caused a useful clinical outcome.

Then, as others have said, the objectively measured outcomes were null, and the long term benefits were null, so the whole exercise is a waste of time.

I might have just written a load of twaddle, but I think that might be because the study is twaddle.
 
Last edited: