• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

PACE Trial statistical analysis plan

Dolphin

Senior Member
Messages
17,567
Probably not important:

Therapy and other treatment received Summaries will be given of treatment received under the intervention policies; these will include:

[..]

ii) Median (lower and upper quartile, minimum and maximum) of proportion of telephone sessions per participant

They didn't do this:
†86% of sessions were received face-to-face and 14% by telephone. ‡94% of sessions were received face-to-face and 6% by telephone.
(i.e. "Therapy sessions attended†" and "Specialist medical care sessions attended").

It is slightly odd they didn't report this. If one looks at:
part i (just above it):
i) SSMC and APT/CBT/GET received
a. Median (lower and upper quartile, minimum and maximum) number of SSMC sessions attended
b. Median (lower and upper quartile, minimum and maximum) number of APT/CBT/GET sessions
attended

They did report median and lower and upper quartile (i.e. IQR), so they could easily have done so in the same table which makes me wonder whether they wanted to hide it e.g. perhaps the figures for CBT or GET were different in some way.

They didn't report minimum and maximum for either (i) or (ii) either.
 

Dolphin

Senior Member
Messages
17,567
Baseline comparability of randomised groups
The following participant-level baseline variables will be summarised both overall and between randomised interventions:
i) Oxford criteria met (yes; no)
? They were all supposed to before they could take part in the trial.

ii) Centre (Barts, Bristol, Edinburgh, Kings, Oxford, Royal Free)
Don't think so

iii) Diagnostic criteria (neither met; CDC met only;London met only; both met)
Didn't do for "neither met" or "both met" (if they'd give one, one could have worked out the other).

iv) Current depressive disorder (present or absent)
Yes.

v) GAD (yes, no)
vi) Agoraphobia (yes, no)
vii) Panic disorder (yes, no)
Not individually but they did report "Any psychiatric disorder"
*Psychiatric disorders included any depressive disorder and any anxiety disorder, including phobias, obsessive-compulsive disorder, and post-traumatic stress disorder.


viii) Fibromyalgia (met, unmet)

Yes (given in Bourke et al., 2013)

ix) Duration of CFS/ME since start of illness

Yes

x) Taking hypnotics, analgesics or antidepressants

Not for analgesics (unless it's in Bourke et al, which I can't easily check). Yes for hypnotics and antidepressants.

Analgesic data was used in terms of costs, which isn't the same thing.
Specific types of medication (analgesics, antidepressants, anxiolytics, and hypnotics) were recorded and average costs assumed for each type [15].

xi) Number of other medications/treatments taken

No

xii) CFQ Score (continuous)

Yes

xiii) SF-36PF score

Yes

xiv) Age at randomisation (years) (continuous)

Yes

xv) Age at randomisation (years) (18 to 29; 30 to 39;40 to 49; 50 to 59; 60+)

No

xvi) Sex (male; female)

Yes

xvii) Ethnicity (white; other, unless ‘other’ can be split further)

They gave %White, so can calculate % other from that.

xviii) Marital status (married and co-habiting, single, divorced/separated/widowed)

Not in Lancet paper anyway

xix) Group membership (none; self-help only; national only; both)

They gave: "Any ME group membership"

xx) Employment status

I don't think they do specifically which seems a bit odd. This would normally be something like %homemaker, %retired, %employed, %unemployed, etc. I don't see anything like that in Lancet paper or McCrone cost paper.

xxi) Health care costs

Yes

xxii) Social care costs

Yes

xxiii) Cost of lost employment

Yes
 
Last edited:

Dolphin

Senior Member
Messages
17,567
Not sure if it's that important but here's the CONSORT flow diagram from the statistical plan and then what they actually published.

I haven't checked the bits before randomised, but they're quite a bit different below that. The box for "Allocation Care Providers" is not in the Lancet one.

CONSORTinstatisticalplan_zps6574c767.jpg


CONSORTinLancet_zps7454fccf.jpg
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
It's not much good recording expectations if you don't publish them.

They also said:
"we will publish these data after the end of the trial."

Perhaps it might be worthwhile somebody requesting this data.


Would be hard for them to argue their way out of that lol

I bet there are practitioner effects also, I strongly suspect CBT/GET was delivered in a more positive way than APT.
 

user9876

Senior Member
Messages
4,556
I wonder what they meant/had in mind with regard to consumer opinions. They didn't show much interest or empathy with opinions expressed on the papers.

The consistency of effects points is interesting. If one looks at objective measures, the results are fairly consistent for CBT with no benefit over APT or SMC. GET did improve with regard to the 6 minute walking test, but not dramatically.

I assumed with the consistency of effects they would be talking about the profile of how patients were affected - the consistency of effect over the patient population. i.e. the looking at the distributions that they have not published. When I first read the lancet paper (before reading anything else about the trial) I thought the increased SDs suggested they had some people with better scores but others with no change or deterioration. I met a hypnotist once who said he works on the basis that some people are very suggestible and will comply with his requests. Others he had no effect on.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Thanks for all your work and analysis on this
I think this one might be reasonably important:

If one looks at the Lancet paper, it appears they have only done it for fatigue and disability and not participant-rated CGI.

They repeat in point 4 that CGI is supposed to be adjusted for:

One can see them doing it for fatigue and disability in Table 3. All apart from one means the numbers are multiplied by 5 (the odd one out is a p value of .38 which then becomes .99. p-values can only be between 0 and 1 so this makes sense.

However, Table 5 has: Participant-rated clinical global impression of change in overall health
Odds ratio (positive change vs negative or minimum changes)
Compared with specialist medical care

APT: 1·3 (0·8–2·1); p=0·31

CBT: 2·2 (1·2–3·9); p=0·011

GET: 2·0 (1·2–3·5); p=0·013

Compared with adaptive pacing therapy

CBT: 1·7 (1·0–2·7); p=0·034

GET: 1·5 (1·0–2·3); p=0·028

I think both of the CBT and both of the GET results would no longer be statistically significant with bonferroni adjustment (basically multiplying by 5 at that level)
Wow, I think that s very important, not least when it comes to looking for consistency across outcomes. This means that neithe CGI rating nor 6MWT improved significantly for CBT (and remember that SF36 Physical Function didn't improve by a clinically Useful Amount relative to control either. For GET, CGI didn't improve significantly while 6MWT only improved by a small amount.

I was just thinking a bit more about the lack of the histograms:



By not publishing histograms, it enabled them to make claims about "return to normal" that others couldn't easily see were dubious.

Similarly with regard to recovery, histograms would likely show that recovery (at 85/90/95/100 in the SF-36 PF, for example) was not that common.

These might be good things,,, to do a Freedom of Information Act request on. These should already have been created and thus there should be no work involved in anyone preparing them.
i agree it would be good to do an FOI for these. However, I'm not sure I would have expected then to publish the histograms; I think presented could mean simply produced as a standard step in any stats analysis, though still worth asking for.
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
Thanks for all your work and analysis on this
Wow, I think that s very important, not least when it comes to looking for consistency across outcomes. This means that neithe CGI rating nor 6MWT improved significantly for CBT (and remember that SF36 Physical Function didn't improve by a clinically Useful Amount relative to control either. For GET, CGI didn't improve significantly while 6MWT only improved by a small amount..

I personally think that the magic "less than 0.05=significant" is nonsense. It is clear there is a difference and Bonferroni adjustment is not needed in this sense. Of course it is still not a large difference.

But I do feel frustrated that certain people put so much stock into the minor changes on self-report questionnaires in this trial, yet if it was a non-blinded pharmacological trial they would repeatedly tell us that the results are meaningless. It is a double standard.

Addition:

If someone ever asks me about the PACE trial and CBT/GET in general, I give them a simple answer. I say that if it was a drug, it would not be approved as the evidence base is poor. The reason why the evidence base is poor is because there have been no blinded trials and there is no evidence of objective improvements, which are necessary to demonstrate efficacy in non-blinded trials. Bogging down in the details often just confuses people, so that is the message I give.

Remember the Ampligen results? The CBT/GET results are no better when taken on the same level (not blinded etc).
 
Last edited:

Dolphin

Senior Member
Messages
17,567
Simon said:
Thanks for all your work and analysis on this
Wow, I think that s very important, not least when it comes to looking for consistency across outcomes. This means that neithe CGI rating nor 6MWT improved significantly for CBT (and remember that SF36 Physical Function didn't improve by a clinically Useful Amount relative to control either. For GET, CGI didn't improve significantly while 6MWT only improved by a small amount.

I personally think that the magic "less than 0.05=significant" is nonsense. It is clear there is a difference and Bonferroni adjustment is not needed in this sense. Of course it is still not a large difference.
I have mixed feelings about Bonferroni adjustments. They seem too severe. Similarly making p<0.05 a strict cut-off is far from ideal.

However, it is they who promised making the Bonferroni adjustment for the CGI and then didn't do it in an case that didn't suit them (but did in the other two cases (SF36PF and CFQ)). Also, they are making claims based on p<0.05 in other areas and don't report most effect sizes (or odds ratios and the like).

ETA: I've just noticed they did publish odds ratios for the CGIs and one can see how marginal the comparisons are with APT by the fact that both CIs include 1.0.

So I accept your basic point about objective measures, but still think it is interesting to highlight what they did and didn't do.
 
Last edited:

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
I personally think that the magic "less than 0.05=significant" is nonsense. It is clear there is a difference and Bonferroni adjustment is not needed in this sense. Of course it is still not a large difference.

But I do feel frustrated that certain people put so much stock into the minor changes on self-report questionnaires in this trial, yet if it was a non-blinded pharmacological trial they would repeatedly tell us that the results are meaningless. It is a double standard.
I agree that it is too simplistic to say that p=0.051 means 'nothing doing' and p=0.049 means "woo-hoo, result!" as if these represent two entirely different worlds. I also agree that bonferroni is too severe, and there are other ways of correcting for mulitple comparisons that are less strict eg False discovery rate.

But don't forget this is a very large study and any decent effect should be significant (confidence intervals shrink as sample size increases); the fact that CGI and Physical Function for CBT failed to reach significance is further evidence that whatever is going on here isn't very impressive.
 

Dolphin

Senior Member
Messages
17,567
Minor, I imagine:

Doctor variables will be summarised by:
i) Discipline (for example, psychiatrist/physician/GP)
ii) Grade (for example, Consultant/SpR/SHO)

They didn't report part (ii) in full

Lancet paper appendix said:
All the physicians and GPs had completed training. 4 psychiatrists had completed training; the rest were trainees.

The line before in the statistical plan they said:
iv) Employment grade (for health economic analysis)

I have no idea whether it's of any importance that did not adjust based on grade/type in the cost effectiveness analysis:
The cost of SMC was based on the cost per hour of consultant physician time in face-to-face contact with patients, which was £169 [12].

The breakdown was:
300 (47%) participants were treated by physicians, 184 (29%) by psychiatrists, and 149 (23%) by GPs. All the physicians and GPs had completed training. 4 psychiatrists had completed training; the rest were trainees.

------
ETA:
Actually the cost effectiveness paper has:
(iv) reduced the cost of standardised medical care by 50% to reflect the possibility of it being provided by a less senior doctor.
and
reported:
"No other sensitivity analyses (i.e. changing the value of informal care, lost employment and standardised medical care) had a large impact on costeffectiveness."
 
Last edited:

Dolphin

Senior Member
Messages
17,567
Comments in italics:
Unblinding of randomised intervention

While this trial is not blinded, due to impracticability, a number of steps were taken to minimise bias arising from this. The apparent success of these steps will be assessed where possible:

1. Extent of any unblinding of the Trial Steering and Data Monitoring Committees or the blinded statisticians will be reported.

There is no mention of blinding in Lancet paper so doesn't seem to have been reported there

2. Extent of primary outcomes data collected over the phone will be reported by randomised intervention.

Done

3. The degree of self-declared expectations of the trial outcome among the trial team by professional role (that is, SSMC doctor, APT/CBT/GET therapist, therapy leader, centre leader, research staff ) and centre by randomised intervention was collected.

Not done

4. Participant preferences will be reported by randomised intervention.

Here's the question:
A6.24 Preferred treatment group
As you know, neither you nor the research nurse can choose which treatment you will receive. That having been said, we would be interested to know which of the four treatments you would prefer if you had a choice. Please place a tick against the treatment you would prefer to receive. If there are two or more treatments that you would prefer, please choose only one.

1 Adaptive Pacing Therapy
2 Cognitive Behavioural Therapy
3 Graded Exercise Therapy
4 Standardised Specialist Medical Care alone
5 Don't know
I'm pretty sure this wasn't published.


5. Participant expectations of outcome will be reported by randomised intervention.

How logical does this type of treatment seem to you?

Extremely Moderately Somewhat Slightly Not at all

How confident are you that this treatment will help your illness

Extremely Moderately Somewhat Slightly Not at all

They published some data from this.
After participants had been told their treatment allocation, but before treatment began, they rated how logical their proposed treatment seemed and how confi dent they were that it would help them (5-point Likert scale with moderately and extremely condensed into a positive response to help with interpretation).

6. Proportion and type of discrepancies between preferred intervention and randomised intervention will be reported by randomised intervention.

Pretty sure wasn't reported.
 

Dolphin

Senior Member
Messages
17,567
(Minor)
Additional analyses
The CBT versus GET contrast will be reported, recognising its exploratory status.
This is after they listed the primary analyses.
I'm not sure they did it?
I think the primary ones wouldn't have been significant anyway.
 
Last edited:

Dolphin

Senior Member
Messages
17,567
(Minor)
Adverse events
Adverse events will be tabulated separately by type (non-serious adverse events, serious adverse events, serious adverse reactions and suspected unexpected serious adverse reactions), by time (weeks 0 to 12, weeks 12 to 26, weeks 26 to 52, and overall weeks 0 to 52), and by randomised intervention.
As has been highlighted in other threads/information, we only got serious adverse reactions by intervention.

The only time period reported on was 0 to 52 weeks.
 

Dolphin

Senior Member
Messages
17,567
(Possibly minor)

All adverse events leading to withdrawal (which constitute significant adverse events) will be summarised by randomised intervention, and whether the participant withdrew from the whole trial or intervention only.
They did report the numbers with "Withdrawn due to worsening" which they define further as: "withdrawal from treatment due to explicit worsening, or a serious adverse reaction".
I presume that covers all withdrawals due to adverse events but can we be sure? They use "serious adverse reaction" which means an assessor has to have decided the adverse event was due to the treatment. But a patient might have felt an adverse event was due to a treatment.

They don't report info on the second part:
whether the participant withdrew from the whole trial or intervention only
----
General point: I haven't looked at CONSORT flow charts closely but I find the one they did unsatisfactory and not as detailed as some I saw where they said what withdrawals were due to.

If people withdraw from a trial because a therapy has made them worse, that can bias results. On the other hand, the numbers of withdrawals were small in the trial.

I can't remember how they might have dealt with such situations (some trials use a form of intention-to-treat where baseline values are carried forward; others (if I recall correctly) use sensitivity analyses where they look at assuming bad results for missing data.
----
ETA:
Discontinuation and withdrawals from intervention

Discontinuation and withdrawals from intervention will be listed by intervention, participant identification number, centre, who made decision for withdrawal, whether the participant withdrew from intervention or trial, the reason for withdrawal, and interval post-randomisation (in days). Reasons for discontinuation and withdrawal from intervention will be tabulated by time (week 0 to week 12, week 12 to week 26, week 26 to week 52 and week 0 to week 52), randomised intervention and reason for withdrawal.
As pointed out above, they haven't reported the reasons for discontinuation.

More detailed descriptions of adverse events will be published separately.
This sounds like it might be a paper?
 
Last edited:

Dolphin

Senior Member
Messages
17,567
Primary analysis (including method of analysis)

All serious adverse events (SAEs, SARs and SUSARs combined) will be tabulated in relation to the intervention. Any doubling in harms observed between interventions will be highlighted. The percentages of participants with SAEs, SARs and SUSARS, and the three combined, as well as number of non-serious AE and percentage of participants with one or more non-serious AEs, will be reported by intervention group, including differences between groups with 95% CIs.

"Web Appendix Table D: Description of Serious Adverse Events" was not tabulated in relation to the intervention.
 

Dolphin

Senior Member
Messages
17,567
(the first paragraph here can probably be skipped)

Predictors of cost

Participant characteristics will be used in a regression model to explain differences in baseline costs. We will test the hypothesised associations with both healthcare and societal costs, as well as using multivariable modelling of other possible predictors identified from univariate analyses. Subsequent regression models will be used to explain variations in follow-up costs, and these will also include clinical characteristics from preceding periods. Two types of regression model will be used. First, we will construct ordinary least squares models, with bootstrapping used to produce reliable 95% CIs around the regression coefficients. Second, we will construct generalised linear models with a log link and gamma distribution to account for the skewness that is likely in the costs data.

Independent variables will include demographic characteristics (such as age, gender and marital status), year of randomisation, clinical variables (such as fatigue score, disability, depression, anxiety) and benefits status (whether receiving benefits and whether benefits are in dispute).

These (i.e. in the last paragraph) weren't discussed in the cost effectiveness paper

Predictors of cost-effectiveness/cost-utility
The net-benefit approach allows multivariable analyses of economic data. This will enable us to identify predictors of cost-effectiveness and cost-utility. This will be done using regression models as described above. In particular we hypothesise that age and gender will predict cost-effectiveness and cost-utility.
This wasn't reported on.
 

Dolphin

Senior Member
Messages
17,567
The main analyses will use an informal care unit cost based on the replacement method (where the cost of a homecare worker is used as a proxy for informal care). We will alternatively use a zero cost and a cost based on the national minimum wage for informal care.

The zero cost analysis doesn't get a mention in the cost-effectiveness paper:

Sensitivity analyses were conducted around key parameters in the analyses about which assumptions had been made. Specifically we (i) estimated the cost of therapy required to reverse the findings from the initial analysis, (ii) used the minimum wage rate (£5.93 per hour) and the unit cost of a homecare worker to value informal care, (iii) used the minimum wage rate to value lost production, (iv) reduced the cost of standardised medical care by 50% to reflect the possibility of it being provided by a less senior doctor.

No other sensitivity analyses (i.e. changing the value of informal care, lost employment and standardised medical care) had a large impact on costeffectiveness.

Fourth, we made assumptions regarding the value of unpaid care from family and friends and lost employment. However, sensitivity analyses revealed that the results were robust for alternative assumptions.

SMcGrath highlighted the problems with such claims if unpaid care was calculated at the minimum wage in a post here:
Can the authors show the Sensitivity Analysis results for Societal Benefits for CBT & GET?
http://www.plosone.org/annotation/listThread.action?root=53389

The lead author essentially agreed with SMcGrath, giving some data.
 

Dolphin

Senior Member
Messages
17,567
[TC=Trudie Chalder]
Competing interests
[..]
TC has received royalties from Sheldon Press and Constable and Robinson.

This contrasts with the Lancet paper
TC has done consultancy work for insurance companies and has received royalties from Sheldon Press and Constable and Robinson.
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
Though I don't agree with the practise, (since space is no longer limited in many journals) It was typical not to report all of the stuff that is in an analysis plans and protocols.

The practise does indeed invite cherry picking, which is clearly evident in the reporting of the PACE trial. I am suprised about the level of "will be reported" data that haven't been reported almost 3 years later. Given the time that has passed, I wonder what their excuse is!?!