PACE Trial and PACE Trial Protocol

[Simon]

Why would that cause a delay in releasing the Analysis Strategy?

well, if they had diluted the original protocol definition of recovery, they might not want such a change to be public until they had come up with some alternatives with more impressive-looking results. Otherwise, the change to protocol might be 'misinterpreted'; this way they have more control. Though that's all wild speculation from me.

 

[Dolphin]

Why would that cause a delay in releasing the Analysis Strategy?

well, if they had diluted the original protocol definition of recovery, they might not want such a change to be public until they had come up with some alternatives with more impressive-looking results. Otherwise, the change to protocol might be 'misinterpreted'; this way they have more control. Though that's all wild speculation from me.

Ok, thanks.
Also thinking about it, a full paper gives plenty of scope to justify definitions i.e. with words (maybe you were thinking that, but you didn't say it explicitly so thought I would).

 

[Dolphin]

This was probably pointed out once or more, but probably no harm pointing it out:

In the protocol paper, they say:

Secondary outcome measure:


The self-rated Clinical Global Impression (CGI) change score (range 1 – 7) provides a self-rated global measure of change, and has been used in previous trials [45]. As in previous trials, we will consider scores of 1 or 2 as a positive outcome ("very much better" and "much better") and the rest as non-improvement [23].

If 3-7 are non-improvement, what they are effectively saying is that 1-2 are improvement. Or certainly there can't be two non-improvement groups.

But they decided to come up with a new measure of improvement, post hoc (see below). This means they knew the other data on improvement at the time.

These post hoc criteria give higher numbers for improvement and are the figures one sees floating around for improvement i.e. when they're talking about it.

e.g.

How many patients improved and how many were back to normal?
Around six out of ten patients made an improvement in both fatigue and physical ability after CBT or GET, compared to four out of ten patients who improved with APT or SMC. The number of patients returning to normal levels of fatigue and physical function was about three out of ten after CBT or GET; about twice as many as those who received APT or SMC.
http://www.pacetrial.org/docs/participantsnewsletter4.pdf

Details about the post hoc measures:

A clinically useful difference between the means of the primary outcomes was defined as 0·5 of the SD of these measures at baseline, equating to 2 points for Chalder fatigue questionnaire and 8 points for short form-36. A secondary post-hoc analysis compared the proportions of participants who had improved between baseline and 52 weeks by 2 or more points of the Chalder fatigue questionnaire, 8 or more points of the short form-36, and improved on both. In another post-hoc analysis, we compared the proportions of participants who had scores of both primary outcomes within the normal range at 52 weeks. This range was defined as less than the mean plus 1 SD scores of adult attendees to UK general practice of 14·2 (+4·6) for fatigue (score of 18 or less) and equal to or above the mean minus 1 SD scores of the UK working age population of 84 (–24) for physical function (score of 60 or more).

64 (42%) of 153 participants in the APT group improved
by at least 2 points for fatigue and at least 8 points for
physical function at 52 weeks, compared with 87 (59%) of
148 participants for CBT, 94 (61%) of 154 participants for
GET, and 68 (45%) of 152 participants for SMC. More
participants improved after CBT compared with APT
(p=0·0033) or SMC (p=0·0149), and more improved with
GET compared with APT (p=0·0008) or SMC (p=0·0043);
APT did not differ from SMC (p=0·61; webappendix p 2).

----

 

[Bob]

House of Lords Debate.

Watch a recording of the House of Lords PACE Trial debate (starts at 17:25:55):
http://www.parliamentlive.tv/Main/Player.aspx?meetingId=12482&wfs=true

There's an interesting titbit, at 18:19:10

The minister says that the Cochrane group have carried out an independent review of the PACE Trial's primary outcomes:
"The main outcome data have already been independently analysed by the Cochrane collaborative group, and a paper is being prepared for publication."

Interesting, but the results depend on what data they had access to.
If they only had access to selective data, then I don't hold out much hope for added clarity.

 

[Dolphin]

House of Lords Debate.

Watch a recording of the House of Lords PACE Trial debate (starts at 17:25:55):
http://www.parliamentlive.tv/Main/Player.aspx?meetingId=12482&wfs=true

There's an interesting titbit, at 18:19:10

The minister says that the Cochrane group have carried out an independent review of the PACE Trial's primary outcomes:
"The main outcome data have already been independently analysed by the Cochrane collaborative group, and a paper is being prepared for publication."

Interesting, but the results depend on what data they had access to.
If they only had access to selective data, then I don't hold out much hope for added clarity.

Cochrane reviewers often ask authors for more details. It will be interesting to see how rigorous they are with the PACE Trial.

 

[Bob]

The most ironic moment in the debate was when one of the speakers (sitting next to Countess Mar) agreed that it is important to be wary of media 'spin', with regards to any medical trial results, and then proceeded to assert that there was a "60%" improvement rate, without mentioning that most of this was seen in the SMC group, and that only a fraction of this was attributable to CBT or GET.

 

[Dolphin]

The most ironic moment in the debate was when one of the speakers (sitting next to Countess Mar) agreed that it is important to be wary of media 'spin', with regards to any medical trial results, and then proceeded to assert that there was a "60%" improvement rate, without mentioning that most of this was seen in the SMC group, and that only a fraction of this was attributable to CBT or GET.

And as I pointed out yesterday http://forums.phoenixrising.me/inde...pace-trial-protocol.3928/page-106#post-331616, this measure was post hoc. They had a measure they they were using for non-improvement.

 

[user9876]

Cochrane reviewers often ask authors for more details. It will be interesting to see how rigorous they are with the PACE Trial.

To do a proper review of the PACE trial they need to look at the choice of primary outcomes and not just the outcomes.

They then need to examine the data in detail i.e. not just an sf36-pf and CFQ score but they need to look at the different dimentions and how these vary. I.e how each patients scores changed at different points.

It would be interesting to know who commissioned the review and under what terms the review is happening. Has all the critisism actually had an effect on the way the PACE trial is seen in the MRC and government. Or is this queen mary's giving a limited remit to try and get some additional support.

 

[Dolphin]

To do a proper review of the PACE trial they need to look at the choice of primary outcomes and not just the outcomes.

They then need to examine the data in detail i.e. not just an sf36-pf and CFQ score but they need to look at the different dimentions and how these vary. I.e how each patients scores changed at different points.

It would be interesting to know who commissioned the review and under what terms the review is happening. Has all the critisism actually had an effect on the way the PACE trial is seen in the MRC and government. Or is this queen mary's giving a limited remit to try and get some additional support.

Based on this Queen Mary response to the FOI request for the positive outcomes (primary outcome measures in the protocol) and recovery rates (as specified in the outcome):

The information you have requested is not held. The requested data relating to the recovery rates and positive outcomes do not exist. That is to say that such analyses have not been done and there is no intention to do so
http://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po

I'm not too hopeful. This was only a few months ago.

 

[user9876]

Based on this Queen Mary response to the FOI request for the positive outcomes (primary outcome measures in the protocol) and recovery rates (as specified in the outcome):
I'm not too hopeful. This was only a few months ago.

But given the raw data these the recovery rates would be trivial to generate things like the recovery rates.

If I was to review someones results I would feel very uncomfortable basing any analysis on someone elses summary statistics as I would want to test lots of the basic assumptions.

 

[Dolphin]

But given the raw data these the recovery rates would be trivial to generate things like the recovery rates.

Yes, agree with you. I'm pretty sure it's all in a database to do lots of other calculations.

If I was to review someones results I would feel very uncomfortable basing any analysis on someone elses summary statistics as I would want to test lots of the basic assumptions.

Yes, but reviewers usually have lots of things to be doing so may only dig so far.
It depends what the remit was e.g. whether this is the update of CBT or GET reviews in which case they have to look at various studies. Not sure I've seen Cochrane reviews on individual studies, but I'm far from an expert on the topic (in terms of what types of things they generally do, etc.).

 

[user9876]

I was assuming that the review is more that a peer review of a paper and that someone was paying for some researcher time to look at the data and check the interpretation of it.

As a peer reviewer for a journal you are basically just doing a very basic sanity check.

 

[Dolphin]

I was assuming that the review is more that a peer review of a paper and that someone was paying for some researcher time to look at the data and check the interpretation of it.

As a peer reviewer for a journal you are basically just doing a very basic sanity check.

Even ordinary Cochrane reviews of fields are a bit different from being a peer reviewer for a journal, who as you say, often don't dig deep. Anyway, not sure if I'm adding much to this particular discussion.

 

[Bob]

The House of Lords debate is now in Hansard:
http://www.publications.parliament.uk/pa/ld201213/ldhansrd/text/130206-gc0001.htm#130206114000195

Edit: the link has been changed and updated.

 

[Bob]

I haven't got much hope for the Cochrane analysis. My guess is that they have access only to selective data, and that they are able to access the (selective) data only because the authors want them to. I've had a quick look at the Cochrane review of CBT for CFS, and my understanding, on first reading, is that it took all the results at face value. As Dolphin suggests, an analysis of an individual paper might be different, but would they really be motivated to demolish a paper written by their peers?

 

[Graham]

I just watched the whole debate. Depressed? Me? It was as bad as a school management meeting! How on earth does the Countess manage to restrain herself? I'll need some counselling soon.

 

[user9876]

I haven't got much hope for the Cochrane analysis. My guess is that they have access only to selective data, and that they are able to access the (selective) data only because the authors want them to. I've had a quick look at the Cochrane review of CBT for CFS, and my understanding, on first reading, is that it took all the results at face value. As Dolphin suggests, an analysis of an individual paper might be different, but would they really be motivated to demolish a paper written by their peers?

They should be because the pace trial is so very disfunctional I find it hard to see how they can retain credibility whilst skirting around the problems. However they might not care about retaining crediblity since no one in the scientific establishment seems prepared to critise PACE.

 

[Bob]

The original Hansard link is now redundant.
Here's the permanent link:
http://www.publications.parliament.uk/pa/ld201213/ldhansrd/text/130206-gc0001.htm#130206114000195

 

[Dolphin]

Regarding the issue of the investigators changing the outcome measures:
These were done in one direction: to lower the thresholds required. There is no particular reason to do this: they could have also added in new thresholds as well as the original ones.

These therapies involve discussing how one is with therapists. In the case of GET, this would include the level of exercise one is doing. In CBT, how one is doing with goals. I'm not sure it would be practical to have such therapists sworn to secrecy and I never saw any suggestion that they were. I can imagine practical problems were discussed in team meetings.

Also, the issue of adverse events involved centre leaders as I recall it. So centre leaders can get a sense through that.

This was a very long running trial. Who knows what could happen. For example, perhaps some of the patients went back to the same centres for therapy and would mention how they were, what therapy they had previously, etc.

The measures used for the primary outcome measures and in the recovery paper are not hidden away the way they could be with some trials.

 

[Dolphin]

Lords debate mentioned there will be a two and a half year follow up.
I don't think I've ever seen any details on how this might be done. This would seem to give PDW and co a fairly clean slate as to what they report on, what thresholds they use, etc. So I'm not hopeful on that one.