1. Patients launch a $1.27 million crowdfunding campaign for ME/CFS gut microbiome study.
    Check out the website, Facebook and Twitter. Join in donate and spread the word!
ME/CFS: In Free Fall Through the Looking Glass
Jody Smith continues to try to put into words the horror of the altered state that hobbles the brains of those with ME/CFS...
Discuss the article on the Forums.

PACE Trial and PACE Trial Protocol

Discussion in 'Latest ME/CFS Research' started by Dolphin, May 12, 2010.

  1. anciendaze

    anciendaze Senior Member

    Messages:
    834
    Likes:
    782
    I would agree, if I thought the numbers had any significance.

    Unfortunately, measures of statistical significance used here depend on accurate values for parameters describing a particular distribution. That distribution has known characteristics: 1) it is symmetrical; 2) mean, median and mode are very close together; 3) a predictably small number of outliers exceed a given number of standard deviations from the mean. None of these characteristics is true for the population used to establish "normal" as defined by this study. Some gross departures are readily apparent.

    One measure of the validity of an assumed standard deviation is stability in a control group. This also fails here. Standard deviations increased over time. Considering the central role of standard deviation in establishing criteria for "recovery", as well as statistical significance, weakness in this pillar of the statistics slips by with surprisingly little concern voiced by investigators.

    The mean value from which deviations should be measured also seems uncertain. If the two parameters carrying all possible information about an assumed distribution are as indefinite as these appear, no confidence in calculated values is warranted.

    What the numbers released show is a heterogeneous cohort created by entry requirements dispersing over time. Random walk models would show similar changes.

    Selection effects are apparent at multiple stages. Roughly half those they attempted to enroll declined. Classic values for bias created by dropouts are hidden by treating those who did not complete the only objective measure presented at the end of the trial as though they fully completed. Data from Actometers was simply dropped. With some 30% of those 'completing' having no objective data whatsoever for improvement, plus dispersal of a meaningless cohort, there should be no problem getting numbers in the range claimed as "moderate improvement".

    Lack of selection shows up in one area, adverse events. There were more than 3,000 adverse events distributed over 600 participants. Any drug trial with this rate of adverse events would be halted. An attempt to redefine adverse events during a trial, to make them less likely, as was done here, will typically get you investigated. We are told few were "serious adverse events" based on review by three independent assessors who turn out to be less than independent.

    Acceptance of this study by a medical establishment does more to undermine confidence in mainstream medicine than to validate the results. If medical professionals can't tell when they are getting bamboozled, great savings are possible in funding medical research. Legislators take note.
  2. Bob

    Bob

    Messages:
    7,561
    Likes:
    8,836
    England, UK
    That's not the way I see it...
    If the additional average improvement in the GET or CBT groups, over and above the SMC-alone group, is insignificant (or it's a negative change), then the GET and CBT can be considered to be useless.
    So, for example, if the SMC+GET group had a worse overall improvement after 52 weeks than the SMC-alone group, then it would be reasonable to assume that the GET was harmful.
  3. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    342
    UK
    Maybe you could spell this out. As far as I can see, the SMC+GET group improved more over 52 weeks -relative to its baseline - than the SMC alone group and that's the mean difference. If what you have said is correct then you conclusions are correct too - but I don't think that's the case.
  4. Bob

    Bob

    Messages:
    7,561
    Likes:
    8,836
    England, UK
    Sorry, my last post just added confusion to the discussion... I was just giving an example of why I thought the mean difference was an important measurement, after you said that the right figures to use, in terms of statistical validity, are the ones comparing the difference in gains between baseline and 52 weeks. I wasn't saying that the actual mean differences were negative. It was just an example of why they are important.
    As you said, the only statistically insignificant improvement, according to the mean differences stated in the paper, was the CBT for physical function.

    I think I might have missed the meaning of your earlier post anyway... I'm not sure what you meant by saying "in terms of statistical validity"?
  5. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    Maybe my expectations are low because a large percentage of the hundreds of full papers I have read are on ME or CFS, but I'm not sure this paper is bad enough not to be published somewhere and I'm not convinced it's bad enough to be retracted (not that I'm an expert on what gets retracted). ETA: Maybe I'm being over-generous and just taking the paper on face value rather than the fact that they have changed outcome measures.

    However if being published in a high impact journal is supposed to mean that a paper is of very high quality, then I don't think the Lancet should have published it.
  6. Snow Leopard

    Snow Leopard Senior Member

    Messages:
    2,206
    Likes:
    1,543
    Australia
  7. Esther12

    Esther12 Senior Member

    Messages:
    5,107
    Likes:
    4,908
  8. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    342
    UK
    Another one from Cella in the pipeline that could well be using PACE data.
  9. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    It looks like they might be running away (in all papers so far*) from this measure that was in the protocol paper:
    e.g. one would think one would look for correlations between that and the WSAS.

    Also, I wonder is there any significance to the fact that they only give us the data on change for CBT rather than across all the groups or looking to see if there were different patterns for different groups. I'm afraid I don't trust PACE trial authors with much.

    * I think it will get mentioned somewhere eventually.
  10. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    Comment from a PACE Trial participant

    FWIW, a comment from a PACE Trial participant:

  11. Esther12

    Esther12 Senior Member

    Messages:
    5,107
    Likes:
    4,908
    I started reading this new paper, and had trouble getting much useful info out of it. Should I print it out and give it a proper go when I'm feeling sharper, or is it just lacking in info that would let us judge the efficacy of CBT/GET?

    It looks like table 2 might be worthwhile for understanding the correlation between different outcome measures... but it looks like all treatment groups are just lumped in together, so a lot of info is lost.
  12. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
  13. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    What would the CBT PACE Trial results look like if we stripped out response biases

    This is of course speculative. But it is transparent so people can decide if any of the jumps are too big.

    I have been reading:
    This contains the baseline data for all of the PACE Trial divided into quartiles based on WSAS scores.
    The second quartile has a mean (SD) 6MWD of 334.9 (82.1).
    This is very like the mean for the CBT group at baseline 333 (86) (I think this is for only those who did the 6 MWT at 52 weeks.
    Of course, some of the individuals are the same but it is probably only around 20% (the baseline figure of 333 is for 123 it appears (there is the possibility it is for 161).

    Anyway, the untreated CBT group is quite similar in other ways to this 2nd quartile (WSAS) group:
    CBT vs Everyone (who make up 2nd quartile)

    CDC total: 4.6 (1.8) vs 4.5 (1.7)

    Jenkins Sleep scale: 12.5 (4.9) vs 11.9 (4.3)

    Fatigue: 27.7 (3.7) vs 27.6 (3.6)

    SF-36 PF: 39.0 (15.3) vs 41.2 (13.9)

    HADS-D: 83 (37) vs 7.6 (3.4)

    HADS-A: 8.1 (4.3) vs 7.8 (3.8)
    -------------

    Many of us think that we can't trust the subjective data in the PACE Trial paper because of all sorts of response biases - people wanting to please their therapist, believe that all the effort was worthwhile, are more optimistic, etc.

    The nearest we have to an objective outcome measure is the 6MWT.

    In the CBT group, for the group we had data for, this went up from 333 to 354 which was actually a tiny bit less than the specialist medical care group.

    One could assume that there was no chance whatsoever and this was completely to do with response bias e.g. being willing to push oneself more as well as having more practice at the test.
    In which case, the "real scores" would be the baseline scores.

    However, it is possible that the patients did actually improve a small bit. This could be due to the passage of time, for example - for example, quite a large percentage of people whose illness started with EBV get better over time.

    The Cella, Sharpe & Chalder (2011) paper doesn't give us data on what people are like without treatment if their baseline score is 354m.
    What it does give us is what people's scores are like if their baseline score is 349.8m. Maybe we could say that would be a more accurate baseline score as after CBT, people might possibly be willing to push themselves a tiny bit harder, or their might be a training effect from having done the test before. So all we're saying is alloting 4.2m for this.

    We then have information about what the final score for the CBT participants would represent, compared to what the CBT participants reported:
    I think the first two measures are probably the most interesting. I'd be "willing to let them keep" the slightly better HADS-A and HADS-D scores as SMC and/or CBT may help those perhaps a little more than a similar improvement in fitness due to the passage of time but no CBT. Perhaps I'd "give them" the improvement in sleep as SMC and/or CBT may help there also.
  14. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    6MWT - looks like those who did it at 52 weeks were the same as average at baseline

    This paper:
    contains the baseline data for all but one of the PACE participants (the WSAS score was missing for this one person).

    From Table 2, one can calculate that the average was 6 Minute Walking Distances (6MWD) of 320.7683881m for the participants at baseline.

    Table 6 in the Lancet PACE Trial papers gives data on the 6MWD at baseline and at 52 weeks.
    If one calculates the average for the figures given, it is 321.2796875. This may not be exactly accurate as the numbers were rounded to the nearest natural number so it could be slightly higher or lower.

    I presume the baseline data in Table 6 is just for those who completed the test at 52 weeks and not for everyone.

    Of course, baseline data still doesn't tell us whether people who were not doing as well at 52 weeks didn't do the repeat test which seems perfectly plausible.
  15. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    342
    UK
    Re possible evidence of response bias in PACE

    This is very interesting. As I understand it what you're saying is this:
    • By chance, the WSAS 2nd quartile has very similar figures to the overall CBT group at baseline, including the one objective measure, 6MWT, of 335m
    • Similarly, by chance, the WSAS 1st quartile has a 6MWT of 350m, very similar to the 6MWT of the CBT group at 52 weeks.
    • So you're proposing we use Q2 as a proxy for the CBT baseline group and Q1 as a proxy for the CBT group at 52 weeks...
    • ... then compare the subjective scores, esp SF36 and CFQ with the objective 6MWT scores
    • and we see that the SF36/CFQ scores associated with the Q1 baseline group (proxy for CBT at 52 weeks) are much lower than those seen in the real CBT @ 52 weeks group, despite having the same 6MWT.
    • providing some evidence that the CBT figures for SF36 & CFQ at 52 weeks have been artifically inflated by response bias.
    Is that a fair summary?
    It's fascinating stuff and possibly quite brilliant :D. I have a hunch there may be some flaw, but I can't see what it is and on the face of it this is evidence of response bias. Bit complicated to explain though, as you and I have shown.
  16. Esther12

    Esther12 Senior Member

    Messages:
    5,107
    Likes:
    4,908
    That is interesting... and all the more reason to want to have access to this data divided up in to the different treatment groups.
  17. oceanblue

    oceanblue Senior Member

    Messages:
    1,174
    Likes:
    342
    UK
    CBT results for cohort 2 similar to PACE on fatigue

    This study gives us results for 384 patients from a secondary care specialist CFS clinic, which must be one of the largest groups reported on, though I'm not sure how many of these had CFQ fatigue data for before and after. The follow-up looks very similar to PACE at 1 year, and had a similar number of CBT session: 10-15. From Fig 1 there was a roughly 6 point fall in mean fatigue scores from 24 to 18, compared with a 7.4 point fall for PACE from 27.7 to 20.3. The PACE group were more fatigued to start with and may have been ill for less time, but they look broadly similar results. Nb these are pre/post results, with no control. The point is that the results are hardly spectacular after a year, and leave the average particpant far from healthy (with no objective measure of progress reported). Yet more data indicating that CBT does not seem to be the answer.
  18. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    Thanks. Yes, you've explained it well.

    I think it works better for some measures more than others: six minute walking distance is probably measuring something quite a lot different from depression and anxiety so unfortunately, just because the 6MWD in quartile one (pre-treatment) is similar to the 6MWD at the end of CBT, doesn't mean that one can tell say that the anxiety and depression scores should be the same.

    However, I find it quite unsatisfactory that the CBT group can have a 6MWD of 333m before treatment, that it only goes up to 354m but the SF-36 Physical Functioning scores jump from 39.0 to 58.2. While another group of CFS patients whose 6MWD is 349.8 give as their level of SF-36 PF physical functioning 45.2.

    SF-36 Physical Functioning results are often called in the CFS world (in the literature, in talks, etc.) as patients' level of physical functioning. Often from the way it is said, people might think it is the result of some objective test like an exercise test, actometers, energy usage using double labelled water, or whatever.

    It is similar to the result we keep pointing to in the Wiborg et al. (2010) review which found that the CBT and controls groups both only increased by a small amount on the actometer readings but the CBT had much larger decreases in fatigue (and when one looks at the same studies, in two of them (as I recall) the CBT groups had much better physical functioning scores).
  19. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    The SMC group are on 348m at 52 weeks so I thought I'd add in their results too at 52 weeks also (I've added in the CBT group at 52 weeks to highlight the difference):
  20. Dolphin

    Dolphin Senior Member

    Messages:
    6,472
    Likes:
    4,756
    Technically, there is a slight flaw in this use of data from the Cella et al. (2011) paper: it's based on not all the data. Ideally, what one would like is for regression to be performed on all the baseline data and see what it represents. Although as I suggested, the distance walked at 52 weeks might be slightly inflated due to issues like people perhaps being willing to push themselves more (perhaps only applies to those who did GET and CBT (maybe to a lesser extent)) and perhaps a bit of a learning effect from doing the test at 0, 24 and 52 weeks. Also, people may be more used to going for continuous walks so might know what pace they can cope with more than when they do the test originally.

    Anyway, if anything, the value of 45.2 might be a slighly inflated value for a 6MWD of 349.8 at baseline because these people are the people who scored the top quartile of WSAS so may have a slight tendency to portray themselves as more able than some others with similar 6MWD scores.

    With a group in the second lowest quartile of scoring on the WSAS (who might have a tendency to mark themselves a bit lower on questionnaires), they had a 6MWD of 309.1 and a SF-36 PF score of 35.5. The baseline scores of APT and GET were respectively 314m & 312m and 37.2 & 36.7.

See more popular forum discussions.

Share This Page