PACE raw data available

Valentijn

Senior Member
Messages
15,786
Likes
45,660
Some nice visualisations of the trial data by Athanasia Mowinckel
https://twitter.com/DrMowinckels/media


One thing I'm seeing is that the improvers on questionnaire scores look like outliers. Nearly everyone is still in the same clumps they were in at the start of the trial.

The other thing is that there's no similar outliers in the objective measurement, the 6 Minute Walk Test (6MWT). Even the very highest scores are on the poor end of average, so it would suggest these weren't healthy people at the end of the trial.

Put those two together, and I think we're seeing the effects of a bit of brainwashing upon a small minority: self-reports claim improvement, but they can't keep up on the objective performance measures.
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,720
Those graphs really show up how slight the between group differences are at 52 weeks.

One amendment I'd like to see would be a bulge of appropriate width at the bottom of the 6 minute walk tests at 52 weeks to show visually all the people who didn't do the 6 minute walk. As far as I'm concerned this means they walked 0 metres. I think there were more of these in the GET group, completely wiping out any overall improvement in the average for that group.

And it's a pity we don't have the step test data.
 

wdb

Senior Member
Messages
1,392
Likes
2,976
Location
London
As far as I'm concerned this means they walked 0 metres.
I think you would have trouble justifying that, even if on the day a participant was unable to make the journey to the testing centre, do the walk and make the journey back home again that does not equate to being unable to walk a single meter.
 

user9876

Senior Member
Messages
4,556
Likes
18,067
I think you would have trouble justifying that, even if on the day a participant was unable to make the journey to the testing centre, do the walk and make the journey back home again that does not equate to being unable to walk a single meter.
But in most cases an intention to treat principle should be followed so that those with missing data don't get discounted but are included in any significance testing.
 

wdb

Senior Member
Messages
1,392
Likes
2,976
Location
London
But in most cases an intention to treat principle should be followed so that those with missing data don't get discounted but are included in any significance testing.
So how would that apply to including missing scores in a histogram or mean calculation of a single outcome measure ?
 

user9876

Senior Member
Messages
4,556
Likes
18,067
Don't know. I think histogram data is used to get a feel and check the distributions its not really used for making inferences.
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,720
I think you would have trouble justifying that, even if on the day a participant was unable to make the journey to the testing centre, do the walk and make the journey back home again that does not equate to being unable to walk a single meter.
I'm sure you're right, it couldn't be justified actually including them as scoring zero in a calculation of a mean.

However, any mean calculated for comparison between groups where one group had a higher drop out rate than the other should be cited with the proviso that this may invalidate any statistically significant 'difference'.

To take a silly example, if 1 out of 10 in group A walked 500 metres and the rest didn't walk, and if 10 out of 10 in group B walked an average of 300 metres, the averages would be A = 500, B = 300, but I know which group I'd say had done better!

This shows how important it is to use something like a continuously worn actometer throughout a study rather than a single walk test.
 

wdb

Senior Member
Messages
1,392
Likes
2,976
Location
London
However, any mean calculated for comparison between groups where one group had a higher drop out rate than the other should be cited with the proviso that this may invalidate any statistically significant 'difference'.
There is no shortage of places they could have been more upfront about the limitations of study.

These are the numbers I get for missing 52w walk scores
Code:
               APT CBT GET SMC
 Missing       48  38  50  42
 Out of        159 161 160 160
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,720
There is no shortage of places they could have been more upfront about the limitations of study.

These are the numbers I get for missing 52w walk scores
Code:
               APT CBT GET SMC
Missing       48  38  50  42
Out of        159 161 160 160
I get about the same. The particular comparison I was concerned about is the GET versus SMC, since if I remember rightly, GET was the group that had a slightly better average distance walked at 52 weeks. This was used to justify the claim that GET is effective. If there were some way of taking into account that more people failed to do the walk in GET than in SMC, this would, I suggest, wipe out that small difference.

Here are a few approximate counts I made comparing these groups for numbers who walked 50 or more metres less than at the start, and numbers who didn't do the second walk, and combining these to show how many did significantly worse:
APT CBT GET SMC
50m less: 18 18 10 17
No walk: 47 38 49 39
Total : 65 56 59 56

I suspect these between group differences on this measure are insignificant statistically. The APT group may be deemed slightly worse, but given that they've spent a year being told to do 70% of their base activity level, that's hardly surprising.

What they do show (to me) is that about 35% to 40% of participants in all groups got significantly worse between the particular days they did the walking test at the start of the trial and the day they did, or were unable to do, the walking test after a year. If that doesn't show the biopsychosocial model is a dead duck, I don't know what does!

What this particular measure doesn't show, of course, is whether, as a result of pushing themselves to do this walk, many of the participants suffered PEM afterwards. But then that opens up the whole hornets nest of the invalidity of all measures used in the trial. And this is not the place for that discussion.
 

wdb

Senior Member
Messages
1,392
Likes
2,976
Location
London
I agree in principle but I worry how defensible the assumption is that everyone who missed the walking follow up got significantly worse. I imagine it would be counter argued that some may have missed the test for reasons unrelated to their health or due to logistical constraints. Also looking the SF36 follow up data whilst this does show lower scores than for those did complete the 52w walk the majority did subjectively report improvement.


Rplot03.png
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,720
Hi @wdb, I'm beginning to wish I hadn't started this diversion. I completely agree that the point I'm making about the non-walkers does not provide publication worthy information.

I was simply trying to point to a weakness of using comparison of mean distance walked for the different groups because so many did not do the second walk.

This is probably a diversion too far. Any published refutation of the PACE trial will need to stick mainly to the points already made about the protocol based outcomes. That is sufficient to demolish the trial.

Any post hoc mini-analysis like mine is not really useful for this unless the PACE investigators themselves were to start promoting the walking results as evidence of benefit of GET. Then I think it would be worth pointing to the fact that many didn't do the second walk, thus greatly weakening the data.
 
Messages
2,051
Likes
10,097
Any post hoc mini-analysis like mine is not really useful for this unless the PACE investigators themselves were to start promoting the walking results as evidence of benefit of GET. Then I think it would be worth pointing to the fact that many didn't do the second walk, thus greatly weakening the data.
I like to think of it as a neat corollary to the reams of mediation analyses that were used to string PACE out for several years and keep providing headlines for the papers ('Researchers reveal that "patient fear" responsible for inactivity', etc).

Even if there's no millage in this, I imagine that a cottage industry of "More PACE bullshit - part 33" type articles could at some point prove lucrative, or at least entertaining for the troops. :balloons:
 

Sidereal

Senior Member
Messages
4,856
Likes
17,274
If data are not missing at random on the 6MWT at follow-up, as they clearly are not judging by that tweet, that renders their analysis (where they showed that in completers GET group walked a significantly longer distance) uninterpretable.
 

A.B.

Senior Member
Messages
3,780
Likes
23,078
Is there some way to determine if the improvement in the GET arm is more due to sicker patients dropping out than anything else?

The average distance walked of the sample will increase if some of the poorly performing patients are removed.
 

Sidereal

Senior Member
Messages
4,856
Likes
17,274
Is there some way to determine if the improvement in the GET arm is more due to sicker patients dropping out than anything else?
You could use a regression model to impute the missing walking test scores with predicted scores based on SF-36 scores for instance or you could do a more conservative analysis where you carry over their baseline scores in a last observation carried forward type of analysis. There are also more complex methods available such as multiple imputation which is a more valid approach because it adds back in the variability that you lose if you use simple imputation methods like the aforementioned ones. But all imputation methods are inherently problematic since you are essentially filling in (making up) the missing data. Still, this is far preferable to producing an extremely biased and invalid analysis based on available cases only, which is what they did, even though they clearly saw that the sicker patients were missing the walking assessment and, worse yet, the attrition problem was bigger in the GET group. Every single decision that they made in this trial was biased in favour of CBT/GET.