Lipkin's Monster ME/CFS Study: Microbes, Immunity & Big Data
The Microbe Discovery Project outlines an ambitious new study by top researchers that has collected patient samples, but needs desperately funds to complete the work.
Discuss the article on the Forums.

"Tug of War: Epic battle over data in controversial paper on chronic fatigue syndrome"

Discussion in 'General ME/CFS News' started by Kyla, Dec 18, 2015.

  1. Kyla

    Kyla ᴀɴɴɪᴇ ɢꜱᴀᴍᴩᴇʟ

    Messages:
    721
    Likes:
    4,221
    Canada
    http://andrewgelman.com/2015/12/18/28362/

    good short blog post on PACE-gate.
    Mostly a summary of correspondence with James Coyne from someone outside the issue.

    I'm unfamiliar with the author or with this blog but from a quick look Andrew Gelman is a statistician, and the blog seems to mostly cover statistical methods in medicine & social science (feel free to correct me if someone is more familiar)

    Nice to see people outside of ME&CFS research taking an interest.

    As the article is quite short I won't post an excerpt. click through here:
    http://andrewgelman.com/2015/12/18/28362/
     
  2. Dolphin

    Dolphin Senior Member

    Messages:
    10,557
    Likes:
    27,562
    What Coyne may be referring to is the comments by the Barts Chronic Fatigue Service (headed by Peter Denton White) on the draft NICE guidelines.

    Parking badge one is the first comment here:
    http://forums.phoenixrising.me/inde...ice-guidelines-insight-into-their-views.1239/
    (If you ever want to find it, it's in my signature!)
     
  3. Dolphin

    Dolphin Senior Member

    Messages:
    10,557
    Likes:
    27,562
    Blog reveals this:
     
  4. Never Give Up

    Never Give Up Collecting improvements, until there's a cure.

    Messages:
    938
    Likes:
    3,879
    This Andrew Gelman blog post, and the comments that follow are well worth reading.
     
    Lyratic, snowathlete, Bob and 3 others like this.
  5. Cheshire

    Cheshire Senior Member

    Messages:
    1,056
    Likes:
    7,990
    Yes, in this comment in particular Robert seems to point to very interesting things.

    Intuitively, I think there is something of importance here, but my brain is struggling to make any sense of it. Maybe more knowledgeable members could help! @Woolie @user9876 and many more!
     
    Valentijn, ScottTriGuy, Bob and 2 others like this.
  6. Woolie

    Woolie Senior Member

    Messages:
    1,770
    Likes:
    13,151
    I found this an interesting comment, @Cheshire. I don't know much about mediation analysis I'm afraid. However, technically, there was in fact a temporal separation in the measures used. The mediator ("fear avoidance" etc.) was measured at 12 weeks after trial commencement, and the outcome (scores on self reported fatigue and physical function) was measured at at 52 weeks.

    But having said that, the figures suggest that both mediators and outcomes actually changed at around the same time - around 12 weeks - so measuring the outcome at 52 weeks doesn't add very much. It would seem to me to demonstrate mediation, you need to do more than just use measures taken at different time points. You'd need to show that there was an early time point where the mediator changed - but not the outcome variable - and another, later time point when the outcome then changed. And it doesn't seem to me that the mediation analysis was set up to show this. And the figures suggest this actually wasn't the case. But maybe others may know more?

    The other problem I see with the paper was that there was a pretty big fishing expedition going on. There are around 10 possible mediator variables (anxiety, catastrophisation, all-or-nothing behaviour, etc., etc.). There were also two preferred treatments - so there were 10x2= 20 possible effects that could be significant. In other words 20 chances to get some sort of "hit". Normally, you would need to correct for this by requiring a more stringent level of significance for your analysis (it would have to be 20 times lower). I'm not sure if this also applies to mediation analysis, perhaps someone else may be able to comment?

    And of course, the elephant in the room: if the study design was dodgy in the first place (which I think it was), then doing a mediation analysis is kind of meaningless...
     
    Sidereal, Roy S, Valentijn and 7 others like this.
  7. Bob

    Bob

    Messages:
    9,846
    Likes:
    33,923
    England (south coast)
    @Woolie, have you considered posting your comments on the blog? I think they'd nicely add to the discussion. If you have any questions, the blogger seems well informed and seems to be willing to answer questions.
     
    Roseblossom, Valentijn and Sasha like this.
  8. RustyJ

    RustyJ Contaminated Cell Line 'RustyJ'

    Messages:
    1,188
    Likes:
    885
    Noosaville, Aust

    I tried to follow the links to the NICE site, but they send me to the home page, not the relevant documents. Is there a better link?
     
  9. Dolphin

    Dolphin Senior Member

    Messages:
    10,557
    Likes:
    27,562
    If you Google a phrase, they should come up (I found one with an active NICE page in the last week or two). The comments are spread across several files as I recall.
     
    RustyJ likes this.
  10. RustyJ

    RustyJ Contaminated Cell Line 'RustyJ'

    Messages:
    1,188
    Likes:
    885
    Noosaville, Aust
    Huh, I've done it before plenty of time, but just didn't think of it this time.
     
    Dolphin likes this.
  11. Woolie

    Woolie Senior Member

    Messages:
    1,770
    Likes:
    13,151
    PACE mediatin comparison.jpg
    @Bob, I don't feel sure enough of this stuff to post on the blog. But anyone, feel free to borrow anything and use it yourself if its useful.

    I'm looking again at the figures. Maybe things aren't quite as negative as I suggested earlier. Here they are.

    The ticks on the base axis are time points (0 weeks, 12 weeks, 24 weeks and 52 weeks).

    You can see that the fear avoidance scores started separating at 12 weeks, but in fact looking now at it again, those for the outcomes (self-rated fatigue and physical function) don't really start separating till 24 weeks.

    What do people think?
     
    Valentijn likes this.
  12. biophile

    biophile Places I'd rather be.

    Messages:
    1,520
    Likes:
    16,574
    I think the evidence for no improvement to the step test etc means the correlation of fear avoidance with fatigue and physical functions scores could be largely meaningless. They talk about how patients need to fear activity less and do more activity, yet the evidence shows patients aren't doing more activity as a result even if they fear activity less (or say so on a questionnaire). No surprise that that 6MWD scores are (inversely) correlated with fear avoidance either.
     
    snowathlete, Bob, Dolphin and 5 others like this.
  13. user9876

    user9876 Senior Member

    Messages:
    2,421
    Likes:
    16,429
    The problem is they are answers to questions which may be similar in content some might have a leading/lagging effect within the therapy. I think it may start off with a message that patients shouldn't fear activity. I think I would prefer the graphs to be drawn in terms of question answers changed rather than on more abstract score lines - also to see the distributions. As Biophile says the figures don't seem to be anchored in objective measures (which I think they treated as mediators rather than outcomes).

    My overall impression of mediation analysis and in particular this one (I've not read others) is that it is throwing complex stats at a problem and just looking for a result when they should be doing more work to form a detailed hypothesis and verify it experimentally using simpler techniques. I would be particularly worried that the base assumptions in regression and mediation analysis are not met by the data sets. In particular around linearity of scales and independence of variables (Do individual questions on the different questionnaires correlate).
     
  14. Bob

    Bob

    Messages:
    9,846
    Likes:
    33,923
    England (south coast)
    Without the data being available, I doubt if any us feel confident about interpreting their analysis, even if we were knowledgeable about mediation analyses. I think it's probably impossible to interpret the data just by using the graphs.

    There's so much that could be said about this study, but to stick to some basic data...

    Fear-avoidance was said to have the largest purported mediation effect, but the size of the mediation effect for fear-avoidance ranges from 17% to 61%. 61% was the absolute maximum effect for any variable in the study.

    Are we looking at the purported mediation effects on the primary outcome measures after 12 weeks? (i.e. the mediation variables can't have a mediating effect until after the interventions have affected the mediator variables, which are measured at 12 weeks.) If so, I wonder if the improvements in the primary outcome measures between 12 & 52 weeks are clinically useful, taken in isolation from the improvements before 12 weeks? If the improvement after 12 weeks were not clinically useful, then we're looking at the effects the mediator variables would have on a non-useful clinical benefit seen in the primary outcome between 12-52 weeks.

    The effects of fear-avoidance range from 17% to 61%, so we might be looking at as little as 17% of a non-useful change in outcomes after 12 weeks.

    Even if we're looking at the whole improvements between 0-52 weeks, the mediation analysis only applies to 15% of PACE trial participants, and it only applies to a very modest mean improvement. If isolated, the individual mediation effects, would probably not have caused a useful clinical outcome.

    Then, as others have said, the objectively measured outcomes were null, and the long term benefits were null, so the whole exercise is a waste of time.

    I might have just written a load of twaddle, but I think that might be because the study is twaddle.
     
    Last edited: Dec 21, 2015
    Sean, Woolie, Daisymay and 3 others like this.
  15. Sean

    Sean Senior Member

    Messages:
    3,188
    Likes:
    17,361
    Hard to write sense about nonsense, other than to point out it is nonsense.
     
    snowathlete, alex3619 and Bob like this.
  16. worldbackwards

    worldbackwards A unique snowflake

    Messages:
    2,091
    Likes:
    10,354
    Earth
    Valentijn, Roseblossom, Sean and 6 others like this.
  17. Bob

    Bob

    Messages:
    9,846
    Likes:
    33,923
    England (south coast)
    Andrew Gelman's Tweet:
     
    Valentijn, Yogi and BurnA like this.
  18. adreno

    adreno PR activist

    Messages:
    4,736
    Likes:
    9,896
    More good stuff. The PACE authors are slowly becoming the laughing stock of the scientific community. Soon an attempt to spin null findings into positive results will be known as 'Pulling a PACE'.
     
    Valentijn, MeSci, Roseblossom and 6 others like this.
  19. BurnA

    BurnA Senior Member

    Messages:
    1,998
    Likes:
    8,929
    The power of Twitter needs to be emphasised even more. I know you've tried already Bob but we need to get more people on Twitter following, liking and retweeting.
    We can talk all we like here amongst ourselves but one influential tweet will reach the numbers of people we can only dream of.
     
    Valentijn, Sean and Bob like this.
  20. Mark

    Mark Acting CEO

    Messages:
    5,182
    Likes:
    5,410
    Sofa, UK
    "The PACE Scandal"

    That has a nice ring to it. :D And to think, it wasn't even anyone from the world of ME/CFS advocacy who came up with that one. What a year it's been! :D
     
    Valentijn, MeSci, Daisymay and 7 others like this.

See more popular forum discussions.

Share This Page