1. Patients launch $1.27 million crowdfunding campaign for ME/CFS gut microbiome study.
    Check out the website, Facebook and Twitter. Join in donate and spread the word!
ME/CFS: A disease at war with itself
We can all agree that ME/CFS is a nasty disease, particularly in its severe form, but there are abundant nasty diseases in the world. What is unique and particularly confounding about our disease is that so much controversy surrounds it, and not only surrounds it, but invades it too.
Discuss the article on the Forums.

PACE Trial and PACE Trial Protocol

Discussion in 'Latest ME/CFS Research' started by Dolphin, May 12, 2010.

  1. anciendaze

    anciendaze Senior Member

    Messages:
    871
    Likes:
    936
    While I've said this before, I will say it again. The parametric measures of significance being used depend on the assumption of a Gaussian (normal) underlying distribution. That is not merely falsifiable, but demonstrably false.

    In checking for possible selection effects, you must consider those who are not in the study. The combined numbers of those who declined to participate and those who dropped out are roughly equivalent to those who completed trials. You also have 31% of the most 'successful' group not completing both 6MWT, the only objective measure presented. One might expect those who did not take a test involving a short walk to be in worse shape than those who did. Plenty of room for non-random selection. The surprise is that they couldn't make this look better.

    You should also check on evidence of selecting forces, even beyond investigator bias. With 93% of the most 'successful' group reporting one or more adverse events, even after a redefinition of adverse events to make them less likely, these forces are in evidence.
  2. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    Hi Jenny, both sound great.

    I'm overcommitted myself at the moment (e.g. have reviewers' comments to deal with on a paper I've submitted) so can't help but have highlighted your message to a few people.

    Best of luck.
  3. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    Good way of putting it. It would have been interesting if some biological measures (that might be part of a "pathological model") had been used e.g. measures of oxidative stress, viral titres, etc. what the results would have been.

    I'm hoping letters and the like can put pressure on them to be honest. If they never know when somebody may point out that they've done some misleading, they might be a bit more careful with claims eventually. Throughout the 2000s, generally they were able to get away with all sorts of rubbish in journals.

    I'm not sure how you're getting percentiles from the Bowling paper?
    If one looks at the top left histogram in figure 1, the percentages look a lot smaller.
  4. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    The response bias and "social desirability effects" point is still relevant.

    It is thing one if "patients (were) believing and behaving as if they were healthy" and there was objective proof of this e.g. actometers or whatever.

    However, questionnaires that may not actually tell one how patients are behaving (because participants give the answers the researchers want to hear, etc.) so they don't give proof that the "patients are behaving as if they are healthy".
  5. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    And don't forget that they don't have the 6MWD for 31% of the participants for GET.
    If one did a "last value carried forward" analysis, the Cohen's d values would likely be smaller again.

    Good point about using the 6WMD for normal functioning (or part of the recovery definition).
  6. Sean

    Sean Senior Member

    Messages:
    1,214
    Likes:
    1,961
    Small but significant practical point arising from this post (and some subsequent posts):

    Better to quote the post numbers because they are invariant, unlike the number of posts per page which is a customisable setting in each member's control panel (Settings > General Settings > Thread Display Options). My settings are for 30 posts per page, which means I only have 25 pages for this thread. Or just use a direct link to each post you are discussing.
  7. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    You learn something new everyday (or I do anyway!). Have joined you with that setting.
  8. Sean

    Sean Senior Member

    Messages:
    1,214
    Likes:
    1,961
    Just the one?

    I usually end up learning several new things a day here, even if I don't want to.

    ;)
  9. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    343
    UK
    According to my quick calculation, using the baseline mean for the missing 31% and adding them back in to the sample would give a mean 20m lower. Assuming this led to a difference with SMC also 20m lower that would give a Cohen's d of 0.15, which is classed as 'trivial' (and probably not significant too). Are you sure that the missing particpants are excluded and not included under 'last value carried forward'? I couldn't see anything in the paper that made this clear.
  10. Dolphin

    Dolphin Senior Member

    Messages:
    6,713
    Likes:
    5,541
    Maybe not 100% sure but the low percentages taking the test suggest it. Haven't seen anything in the paper or protocol paper to suggest they used "last value carried forward" for secondary outcome measures. Somebody else who is used to reading papers came to a similar conclusion.
  11. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    343
    UK
    Well, if that's the case then GET basically made no difference to physical condition. They got nothin'!

    Checking over the paper I found this:
    does this mean that drop outs are excluded if they didn't get SF36/CFQ data for them? Which seems to contradict the principle of ITT, and could make a significant difference if these drop-outs had in fact deteriorated.
  12. biophile

    biophile Places I'd rather be.

    Messages:
    1,386
    Likes:
    4,563
    Reply to thread post range 701-720

    Hehe, so true!

    Equivalent claims are common, but it looks like the PACE results dispute the "most" part, when using these outcomes on a modified clinical global impression: 41% for CBT/GET vs 31% for APT and 25% for SMC.

    In the original protocol: "We propose that a clinically important difference would be between 2 and 3 times the improvement rate of SSMC." We are not given odds ratios for primary measures, and I don't know how they arrived at these values, but we are given OR for the clinical global impression:

    APT vs SMC: 13 (0821); p=031 | CBT vs SMC: 22 (1239); p=0011 | GET vs SMC: 20 (1235); p=0013.

    CBT vs APT: 17 (1027); p=0034 | GET vs APT: 15 (1023); p=0028.

    I agree. Pacing is not about avoiding all exacerbations, and I also think "perceived energy" is not the only measurement or necessarily the best description. CFS is not just the reduction or absence of something "good" but also the increase or presence or something "bad", both have to be taken into account.

    Good points.

    The inconsistency you mention is probably because they don't understand pacing nor believe in it. Then of course there are suspicions that they don't want it to succeed because it is a threat to them, and the rationale behind it competes/conflicts with their own approach. This is just as much of an ideological struggle as a scientific one. The PACE trial was probably never designed to give pacing the best chance, but more about demonstrating their own CBT/GET approach is superior.

    Some beliefs and cognitions can be rather difficult to change, but I agree there is double-speak and a disconnect between hypothesis and reality. I think they have toned down some of the rhetoric or at least worded it better for different audiences, and it is difficult to say how much goalpost-shifting has occurred over 20 years. The so-called "Wessely School" see themselves as the moderates between the purely organic position the purely psychological position. Their current theme in a soundbite is: CFS symptoms are "physical" but "functional" and misinterpreted as a disease process, primarily perpetuated by cognitive and behavioural factors. Wessely seems to think of CFS as a delayed recovery from an event that everyone else recovers from naturally, that's why he allows for an infectious "trigger". Their comparison to anorexia nervosa is a good example of how they view the "physiology" of CFS. There is an article on the KCL website about the "Physiological Aspects of CFS" which is a better example, limited to the effects of deconditioning, anxiety, hyperventilation, stress, depression and circadian rhythm disturbance, and we are told (in bold!) "It is important to point out that these changes are reversible with physical rehabilitation and/or exercise."

    In my opinion, psychologists like Leonard Jason and Fred Friedberg are the real "moderates". They acknowledge that psychological factors can play a role in CFS but apparently have not been seduced by psychobabble, hyperbole, and the convenient dismissal of biomedical research.

    I agree, and "inflated self-efficacy" is an interesting confounder for a "mind over body" attitude.

    I just redid it and you're right, pooled meanSD of SMC at 0 and 52 weeks vs GET advantage over SMC at 52 weeks was Cohen's d=0.348. I probably did something stupid under a haze of brain fog so I deleted it.

    Good points.

    I'm afraid that you are correct. I would be scared to have Chalder as a therapist.
  13. biophile

    biophile Places I'd rather be.

    Messages:
    1,386
    Likes:
    4,563
    Some of the news articles on PACE talked about "hope" or "relief" for patients as a result of the trial. But after reading other peoples comments, and attempts at examining the PACE trial for myself (some of which caused a crash), I do not feel such hope or relief but instead I feel dismay.

    What about the majority of patients who don't gain much from CBT/GET or experience adverse effects from these in the real world? Not looking forward to, and feel sorry for all the patients who will now have to endure a renewed blanket push for CBT/GET as a result of a large flawed study. CBT/GET studies often exclude people who have already undergone therapy, but no doubt that patients who have already tried these without success will be goaded into further attempts despite the failings.

    The continuous onslaught of flawed psychological research and the uncritical swallowing of it are concerning, and combined with illness limitations these concerns make me feel powerless against the psychobabble juggernaut. It is reasonable to view White et al as "sincere ideologues" but after the PACE trial "trust" isn't the first thing that comes to mind. I don't think they fabricated data but they definitely engaged in spin.

    I agree. As I recall, the PACE authors stated they didn't use biological measures because none were validated or reliable or whatever.

    Has anything changed in the 2010's so far?

    I agree, I just find it hard to believe that White et al are not already aware of this problem to some degree. They just don't seem to care all that much about it, or as a more cynical possibility, let it slide because it works in their favour. They also showed resistance to actigraphy and gave a rather strange excuse. If they believe "deconditioning" was a major factor, the 6WMT was probably expected to show a much larger improvement than it did.

    Thanks for the advice Sean, I changed the titles.

    Another good point/idea.
  14. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    343
    UK
    Protocol uses 'within 1SD of the mean' in a very different way

    The biggest con of the published trial was the creation of a post-hoc measure of 'normal', defined as within 1 SD of the mean, which they wrongly calculated as a SF36 score of 60.

    Someone pointed out to me that the protocol also used this 'within 1 SD of the mean' formula, but in a far saner way:
    So, for SF36, 'within 1 SD of the mean' is interpreted as:
    Protocol: 5 points higher = 'Improved', but not recovered.
    Published: normal

    Which is a big difference, and I think goes to show both how far PACE moved the goalposts and also how well the PACE authors originally thought the trial was going to work.

    Note also that the Protocol uses the correct figure for working age SF36 norms, while the published Lancet paper uses the wrong figures, giving a mean-SD threshold 10 points too low..
  15. anciendaze

    anciendaze Senior Member

    Messages:
    871
    Likes:
    936
    At the risk of being tiresome, I wish to point out that a normal (Gaussian) distribution should have mean, median and mode approximately equal. Here they are using a mean of 85 when the population mode is 95 or 100. You can go through an arithmetical calculation of squaring, adding and taking root values to get a "standard deviation" for numbers following any distribution. This "standard deviation" does not have the same meaning as 1 SD for a normal distribution. We still await clarification of just what the number means.

    The discrepancies here are on the same order of magnitude as the results trumpeted as proof of effectiveness, and the bound used in the trial is set by subtracting "1 SD" from the offset mean, roughly doubling the effect of discrepancies between mean and mode. Someone has paid 5M pounds for gibberish.
  16. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    343
    UK
    Though it turns out (by chance?) that the 'mean - 1SD' formula sets at threshold at the 17th percentile, which is very close to where it would be if the distribution were normal. Bigger problems in this case are a) they didn't use the figures for a working age population and b) the 'mean -1SD' forumla itself as a basis for establishing a 'normal' threshold, regardless of whether or not that populaton is gaussian.
  17. anciendaze

    anciendaze Senior Member

    Messages:
    871
    Likes:
    936
    If you look at the population data carefully, you will see that the healthy population has a much stronger central tendency than a normal distribution. I've tried to find a comparable group equally far from the mode for comparison. This isn't an exhaustive search, but what I've found so far includes: people recovering from surgery, people with serious conditions like cancer, COPD or heart failure, people well over 65. It is virtually impossible to find healthy people way out in that tail for comparison. In every other case, so far, the comparison group has serious organic problems resulting from either a known disease or aging.

    The argument about percentiles if the distribution were the assumed Gaussian distribution is a little like saying "if all horses are zebras...".
  18. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    343
    UK
    What I was trying to say is if you want used that formula on a normal population the threshold would be set as 'better than the bottom 15%', and the threshold happens to be effectively the same using non-normal SF36 scores and the same formula. It may be a coincidence, but the effect is the same. Had the result been the 10th percentile or the 25th percentile we could argue that the use of the formula on a non-normal distribution was producing a very different type of threshold - but it isn't.

    And yes, I agree, down at 60 you do not have healthy people, that's the problem of using mean-1SD (or rather the 15th percentile) to define 'normal' when 22%+ of the same population report long-term health issues.
  19. anciendaze

    anciendaze Senior Member

    Messages:
    871
    Likes:
    936
    My point is that I have not been able to find any part of a healthy, working-age population in that range. I can find plenty of people in the categories listed above, plus those with such obvious problems as rheumatoid arthritis or serious obesity. When it comes to finding otherwise healthy, deconditioned people to compare, I have a problem. Those diagnosed with CFS according to these criteria appear to be sui generis.

    What does that tell you about using statistics derived from the general population for comparison?
  20. Sean

    Sean Senior Member

    Messages:
    1,214
    Likes:
    1,961
    Reminds me of that physics joke about eliminating inconvenient variables: 'Assume a cow is a sphere'...

See more popular forum discussions.

Share This Page