1. Patients launch $1.27 million crowdfunding campaign for ME/CFS gut microbiome study.
    Check out the website, Facebook and Twitter. Join in donate and spread the word!
Knitting Equals Pleasure, Despite ME/CFS
Jody Smith loves knitting. Again. She thought her days of knitting and purling were long over but ... she's back ...
Discuss the article on the Forums.

PACE Trial and PACE Trial Protocol

Discussion in 'Latest ME/CFS Research' started by Dolphin, May 12, 2010.

  1. Dolphin

    Dolphin Senior Member

    Messages:
    6,870
    Likes:
    6,155
    There is loss of information with dichotomous variables, or composite variable, both of which could be said to be problems with the original primary outcome measures.

    However, it was they who came up with the such variables.

    Also, their proposal presumably used such variables when the reviewers reviewed whether the MRC would fund it.

    And similarly, when presumably used such variables when ethics reviewers reviewed it.

    And it wasn't as if they gave up on dichotomous variables: the two post hoc variables they introduced, "normal functioning" and "improvement" were dichotomous variables.

    Also, they already had a secondary outcome measure that used continuous scoring for the Chalder fatigue scale.

    Given this and, for example, the drastic reduction in the recovery thresholds, it's difficult to believe it was anything other than running away from demanding thresholds that would give them not good results.

    Also, and this has been asked (and maybe answered?) before, do we know whether they had seen all the data before making this change? There is a lot of talk somewhere about having data but not broken down by intervention, but I can't remember whether we know what they knew before making this change?
     
    Valentijn and biophile like this.
  2. Esther12

    Esther12 Senior Member

    Messages:
    5,383
    Likes:
    5,882
    They said:

    Lancet PACE:

    re most of what @Dolphin said: I'd have no problem with them choosing to release data as they did in addition to the outcome measures from the protocol... they could have then argued that these new outcome measures were of greater value with all of the data available for people to assess for themselves. But their refusal to release the outcome measures they originally developed does look just like "running away from demanding thresholds that would give them not good results."
     
    Valentijn and biophile like this.
  3. alex3619

    alex3619 Senior Member

    Messages:
    7,688
    Likes:
    12,531
    Logan, Queensland, Australia
    @Dolphin, I question whether they would need to see all the data before realizing their thresholds would make the study a total failure instead of the disaster it is. They would likely have seen enough to realize they had a problem, long before all the data was in.
     
  4. biophile

    biophile Places I'd rather be.

    Messages:
    1,404
    Likes:
    4,892
    @Dolphin . The definitions for a 'clinically useful difference' (and 'normal range' in fatigue or physical function) on an individual level were not mentioned in the Statistical Analysis Plan and are described as post-hoc in the Lancet paper.

    The normal range was introduced during peer-review, so they submitted the paper without it. It is unclear whether the reviewer specified the exact same normal range that White et al used or just suggested it in general. It is unknown when the post-hoc 'clinically useful difference' (individual level) was added, but it is described as 'post-hoc'. They claimed to the Lancet (in response to Hooper) that "we defined clinically useful differences before the analysis of outcomes", but they seemed to discuss group differences not individual differences. Group CUD was introduced in the SAP and not apparently described as post-hoc in the Lancet paper, whereas individual CUD was not mentioned in the SAP and is described as post-hoc in the Lancet paper, so it is plausible that individual CUD was introduced after being unblinded to the data in general.

    Then there is the revised criteria for 'recovery' which was based on the same post-hoc 'normal range' first introduced when the February 2011 Lancet paper was undergoing fast tracked peer-review. They claim in the 2013 recovery paper that the criteria for recovery was changed before doing the analyses for that paper, but failed to mention that they had already known about the normal range outcome and already knew the mean(SD) of physical function scores in general. It would not take a genius to figure out from just looking at those figures and other results known in 2010/2011 that the unofficially expected recovery rate of 25% for CBT/GET was looking in serious doubt if using the original criteria for recovery.

    All their main outcomes for individual clinical improvement (and recovery) for fatigue and physical function appear to have been introduced *after* being unblinded to data in general. If this was a drug trial designed and conducted by people with careers partly built on two of the drugs being tested, there would be an uproar, but instead we are told that PACE was held up to the highest standards and is definitive evidence and that the principal investigators were 'utterly impartial'. Given the numerous published blunders associated with the normal range in physical function, there appears to have been a severe lack of scrutiny. Not to mention that changes were apparently not approved by the relevant trial oversight committees.

    When you say talk of having the data but not broken down by intervention, I think you meant for the purposes of monitoring safety? There are additional cues, conscious or subconscious, which could have affected decisions to change the thresholds, besides of course the disappointing data itself. The FINE Trial tested similar therapies and was essentially a null result. And as I think you have mentioned before, there was little stopping those involved from getting general impressions from other staff about how the trial was going, i.e. much worse than anticipated.

    There is no doubt the original goalposts would have given very sobering results. I agree with Esther12 about releasing both sets of outcomes for comparisons, instead of the resistance and stonewalling we have witnessed instead. Their reasoning for changing the physical function threshold for recovery has also been debunked.
     
    Last edited: Mar 28, 2014
  5. Sean

    Sean Senior Member

    Messages:
    1,307
    Likes:
    2,337
    Me too. Got no problem with different analyses being done on the same data. But the original analysis plan must be delivered, in addition to any alternative post-hoc analysis. (Either that or the data must be released so others can do the original analysis.)
     
    Dolphin and Valentijn like this.
  6. Dolphin

    Dolphin Senior Member

    Messages:
    6,870
    Likes:
    6,155
  7. Tom Kindlon

    Tom Kindlon Senior Member

    Messages:
    257
    Likes:
    797
  8. Tom Kindlon

    Tom Kindlon Senior Member

    Messages:
    257
    Likes:
    797
  9. Dolphin

    Dolphin Senior Member

    Messages:
    6,870
    Likes:
    6,155
    (no major surprise)
    Healthy Controls in an ME/CFS study had average SF-36 PF scores of 95

    from:

    A piece by Cort Johnson on the IACFS/ME conference:

     
    Valentijn, Esther12 and biophile like this.
  10. Dolphin

    Dolphin Senior Member

    Messages:
    6,870
    Likes:
    6,155
  11. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
    OK, I'm sure I'm being incredibly stupid, and have possibly completely misinterpreted this, but I think we've missed this haven't we? (And, if so, how the heck did we manage to miss it?)

    PACE Trial

    Cohen's d effect size.

    CBT vs APT 0.33 (CI 95% 0.10, 0.56)

    GET vs APT 0.28 (CI 95% 0.06, 0.51)



    It doesn't specificy if this is fatigue or physical function or combined fatigue and physical function. Or some other measures.

    Can anyone guess where I found it? ;)



    Edit: @biophile gets the prize.

    It's from the Castell et al. meta-analysis:

    Castell, B. D., Kazantzis, N. and Moss-Morris, R. E. (2011), Cognitive Behavioral Therapy and Graded Exercise for Chronic Fatigue Syndrome: A Meta-Analysis. Clinical Psychology: Science and Practice, 18: 311–324.
    http://onlinelibrary.wiley.com/doi/10.1111/j.1468-2850.2011.01262.x/abstract
     
    Last edited: Apr 16, 2014
    Dolphin, biophile and Sean like this.
  12. biophile

    biophile Places I'd rather be.

    Messages:
    1,404
    Likes:
    4,892
    Sean and Bob like this.
  13. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
  14. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
    BTW, I interpret these as 'small' effect sizes, not 'moderate'.


    And, looking at the results in the PACE trial paper, I can't imagine that the effect size would change much with CBT vs SMC and GET vs SMC.
     
  15. biophile

    biophile Places I'd rather be.

    Messages:
    1,404
    Likes:
    4,892
    Yay! At first I did a Google search but that was not successful. Then I vaguely remembered that paper having similar outcomes for CBT/GET in general so I had a closer look. I'm not sure if we have talked about it yet, I think I have seen those results before but probably ignored them because it was compared to APT rather than SMC.

    Interpreting Cohen's d is arbitrary. Also, the PACE Trial recruitment process decreased the standard deviation in the scores and this could have increased the apparent size of Cohen's d without the actual effect being larger.
     
    Bob likes this.
  16. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
    I don't think it's entirely arbitrary, is it?

    This is the standard interpretation:

    d=0.2 (small effect)
    d=0.5 (moderate effect)
    d=0.8 (large effect)
     
  17. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
    :) I think you deserve a phoenix rising prize for that bit of detective work! :)

    I can't remember us discussing it before, but I have sometimes got a very bad memory.
    I've looked at the Castell paper many times, and I'm sure i must have looked at Table 3 in the past!
     
  18. Dolphin

    Dolphin Senior Member

    Messages:
    6,870
    Likes:
    6,155
    Paper says
     
    biophile and Bob like this.
  19. Bob

    Bob

    Messages:
    8,851
    Likes:
    12,406
    South of England
    @Dolphin, thanks for that.

    I guess that makes the published effect size quite uninteresting for our purposes. Not quite as exciting as I thought!
     
  20. biophile

    biophile Places I'd rather be.

    Messages:
    1,404
    Likes:
    4,892
    If I recall, the 'standard interpretation' was just a very rough guideline from Cohen himself (who also suggested that the interpretation depends on the outcomes being measured), but it has just been lazily parroted without much consideration. The advantage of Cohen's d and similar versions is that various outcome measures can be combined. However, the result is not particularly intuitive. In a previous Phoenix Rising article, Simon gave a helpful visual of what d=0.27 looks like:

    [​IMG]

    As I said before though, I am weary about how it depends on the standard deviation.
     
    Simon, Bob and Sean like this.

See more popular forum discussions.

Share This Page