PACE Trial and PACE Trial Protocol

[anciendaze]

I've written elsewhere that distributions resulting from problems involving multiple interacting organ systems are very likely to be Lévy distributions, since efficiency of each system combines multiplicatively. A simple, obvious example is the combination of COPD and heart failure, and individuals with both problems function way below the mean, if they are still living.

We don't really know if even the PACE authors believe the full distribution of SF-36 pf scores has a symmetrical right tail of people performing far above the mean. In the absence of such a tail the mean, mode and median values will not be the same, and common inferences about the meaning of "one-standard deviation below the mean" will be invalid. I suspect anyone familiar with psychology would know how readers who don't have the time to check all references and assumptions would interpret that criterion. Stated differently, I believe this continuing unresolved problem reflects a deliberate attempt to mislead.

From an academic standpoint, the distribution from which samples are drawn need not be Gaussian provided the Central Limit Theorem (CLT) applies. This places a great deal of weight on the sampling process itself, plus some unstated assumptions. What we know in this case is that patients entering the trial were chosen according to subjective values applied by a small group of people. It is hard to find objective criteria for the illness definition in use. A small sprinkling of randomness will not correct inherent systematic bias.

I think the common prerequisites for the CLT, independent identically-distributed random processes, are clearly not present. More sophisticated proofs of the CLT depend on a large number of different distributions, each possessing a mean and bounded variance. Analytic Lévy distributions do not have bounded variance. It seems quite likely they have mixed samples with organic disease and psychological problems. I do not consider two a large number. If the organic disease(s) involved combine multiplicatively to reduce performance, the main requirement for getting a Gaussian distribution out disappears entirely.

We don't know how many times they applied sampling criteria before they got something out that looked like a set of normal distributions for each group. We do know they had so much trouble getting the numbers they wanted that they went back and revised the entry criteria. If changing these by 5 points did not invalidate results, then, a fortiori, changing a score by 5 points during the trial will not invalidate the null hypothesis that there was no benefit. This implies that modified recovery criteria should be more stringent, not less.

Do we have any way to evaluate the number of patients considered recovered by no more than 5 points?

In the absence of answers to the objections above, I believe we should consider the entire 5-million pound study an expression of unsubstantiated opinion.

 

[user9876]

I've written elsewhere that distributions resulting from problems involving multiple interacting organ systems are very likely to be Lévy distributions, since efficiency of each system combines multiplicatively. A simple, obvious example is the combination of COPD and heart failure, and individuals with both problems function way below the mean, if they are still living.

Where there are multiple interacting systems I suspect observed distributions are actually the sum of a number of different (not necessarily independent) random processes and hence likely to be multi-modal and complex. Hence if you want to model them something like a Gaussian mixture model or a phase type distribution would seem appropriate. These are basically composite distributions either a sum of Gaussians or Exponential distributions that are then fitted to the data. In the case of phase type distributions they can be created using some form of Markov chain which can represent something of the structure of the problem.

More generally there are issues around noise and error distributions on measurements. I've generally been a fan of robust statistics which tries to remove assumptions of normality in noise distributions. ftp://ess.r-project.org/Research-Reports/94.pdf

 

[anciendaze]

@user9876, the crucial difference in my hypothesis is that I believe measures like overall performance arise through multiplication of performance of individual organ systems. This is likely to occur in many examples of pathological situations. Too often medical statistics based on distributions of healthy people are applied to those who are clearly ill. While instrumental errors are more likely to be normally distributed, distributions connected to pathology are not.

Remember that the classic normal biometric distribution of adult heights came from draft physicals of healthy adult males, and that independent variation in length of individual bones does combine additively to produce total height. (You don't have to look far to find statistical problems if you combine distributions for male and female heights, or apply these numbers to patients with osteoporosis.) I don't have to engage in special pleading to suggest that emphasis on problems limited to single organ systems will distort measures of health in such a way that some patients will necessarily "fall through the cracks" of the resulting system.

In the case of ME/CFS, the problems I have currently identified are: dysautonomia, immune dysfunction, mitochondrial dysfunction, hypovolemia and reduced cardiac output. By themselves, each of these does not look terribly bad in most cases. The cumulative effects are something else. You can die while bouncing between specialists, and this is not peculiar to ME/CFS.

Another aspect gets into subjective experience, though this is also the result of objective organic conditions which are hard to study prior to autopsy. Damage to dorsal root ganglia will produce sensory inputs which are apparent to patients, but completely invisible in terms of clinical signs. This overlaps with perception of pain and fatigue by MS patients, who have clear damage to efferent nerves that produces clinical signs visible to doctors. No ethical doctor will tell MS patients to simply push themselves past limits set by perceived effort.

The problem for ME patients is that they have no way to distinguish boundaries between safe effort and effort which will result in a crash, before they wind up in the ER. Their nerves are always telling them something is wrong. Cognitive impairment is another indication which is generally disregarded by doctors. ME patients pay attention to these episodes as precursors of syncope, which might even involve them in falls or automobile accidents. I believe honest statistics about these potential problems will show that patients are not simply "catastrophizing". Too many have experienced such consequences.

The assumption above of damage within the CNS is perhaps the most controversial. In addition to recent evidence of neuroinflammation, we have older results on the proteome of CFS patients cerebrospinal fluid. There are also high-resolution MRI studies using voxel-based morphometry which show progressive reductions in volume of structures within the brain as a function of time since onset.

This evidence is considerably more impressive than repeated subjective studies by the same people based on an idée fixe about functional somatization disorders. Dodgy use of statistics dealing with primarily subjective measures does nothing to reduce suspicion of deliberate bias and obfuscation.

 

[anciendaze]

In the above post, perhaps I've leaned too hard on the subjective nature of PACE. To be even-handed I should give the authors their due. The most impressive objective measure of success for PACE is that they were given five million pounds to conduct the study.

 

[Graham]

Imagine what they could have got for doing a proper job!

 

[alex3619]

@anciendaze, the way I put what you describe as a multiplication effect years ago is like this. Most diseases require a loss of substantive organ function before you see pathology. In the case of the heart it can be as much as 80% loss of function. Yet what if you have seven interdependent organs or systems with a loss of 20% function? The resulting function across the systems is then only 21%. I found it easier to explain using numbers.

My point about non-Gaussian was primarily aimed at the general population, not patient response to treatment. Its about making comparisons and the use of SD to define SF-36 cutoff points in the general population. Patient SF-36 data distribution is probably much more like a Gaussian distribution, though primarily as an artifact of the study. Not too sick, not too well, just right for the study. The selection process creates artificial boundaries.

 

[anciendaze]

@alex3619 :

My point about distributions within individual groups is that these could look like Gaussian distributions, if you did not have many data points, and none you had were extreme. The researchers certainly had the opportunity to exclude any patients they didn't want. We also don't know how many times researchers rolled the dice to get these sample distributions, but we do know they had trouble meeting original criteria.

One aspect of working with distributions that lack bounded variance is that sample numbers and bounds completely determine sample variance. We know that these authors manipulated criteria to change bounds and increase numbers when they ran into trouble. A less obvious manipulation turns up in the criteria for adverse events. These were changed as well, making it harder to register an adverse response, and virtually impossible to register a serious adverse response if patients did not die. Taken together, these changes, and dropping objective measures promised in the proposal that got funding, strongly suggest the result was preordained, not the result of experiment.

The discrepancy between the press release and actual numbers tells an interesting story all by itself. Professional psychologists and psychiatrists can hardly claim to be naive about the likely press interpretation of what they said.

Incidentally, I'm not arguing against the exponential distributions people are talking about above. The tail of a Lévy distribution is essentially exponential. I'm going beyond raw data to suggest a mechanism based on combining distributions multiplicatively. I think the clue shows up when performance of an individual organ system goes to zero. No matter how well other organs perform total patient performance will be zero, the patient being dead.

(I also need to caution people about how they may be combining deviations from the mean additively. I made the same mistake many years ago as a student, considering only extreme cases. The expected value of the variance of Gaussian distributions combines in a less dramatic way. )

Your point about subjective criteria is well taken. I think PACE could become a case study in the number of ways you can present subjective choices as objective data.

 

[anciendaze]

@Graham, I strongly suspect they did exactly what they were expected to do, show that the patients were to blame for their illness. This reduces demands for disability payments, and it has already been shown that most patients are highly averse to proposed therapies, which reduces visible costs in a second way. Arguing that you need government assistance because you are not competent to make your own decisions is a losing proposition when dealing with people who already feel they are competent to make decisions for everyone else.

 

[anciendaze]

Just a note for people who wonder how a travesty like PACE can be accepted as "evidence-based medicine". The BBC has an article on medical doctors understanding of risks reported in statistics. Since doctors are very much herd animals, who are strongly influenced by peer pressure, the fact that the majority misunderstand statistics in simple situations should be a serious caution.

Don't expect mere exposure to mathematics to change this. Consider what happened to major banks in 2008. This article mentions what Gigerizer found when he talked to people at Goldman-Sachs about risk management after the crash.

 

[alex3619]

@anciendaze , nice article links on Gigerenzer. I think we should all read them.

 

[Tom Kindlon]

Someone sent me this after reading my letters and thought I might find it interesting but wanted to stay anonymous if I was going to re-use it.
-----

PACE Trial participants classified as 'recovered' should reflect a healthy working age population. Alternative chronic disabling health problems are usually excluded during the diagnosis of CFS, and 97% of trial participants were 18-59 years of age.

Here is a histogram and brief summary of the scores for the working age population aged 16-64 years who answered no to the following question in the ONS Omnibus Survey 1992 cited by PACE via Bowling et al.: "Do you have any long-term illness, health problem or handicap which limits your daily activities or the work you can do? (Include problems which are due to old age.)"

mean±S.D. is 95.0±10.2 and median(IQR) is 100(95-100). 1.7% score 0-55, 6.0% score 60-80, 92.3% score 85 or more.

Depending on the question asked on the SF-36, close to or well over 90% responded that they have no physical limitations, except for the question about vigorous activities which was 67% (with 26% limited a little and 7% limited a lot).

This group should be strict enough, but does not factor in temporary illness, outliers or non-representative scores, errors due to data entry, or questionable answers (such as reporting more limitations climbing one flight of stairs than several).

Further excluding those scoring under 60 or 85 increases the mean somewhat and has a greater effect deceasing the S.D, but does not change the median(IQR) since it is far less susceptible to the effects of outliers.

It is extremely unlikely the PACE Trial subgroup classified as 'recovered' came anywhere near these distributions. It would most probably appear the opposite to the above histogram, clumping around 60 and thinning out as the score increases.

 

[Dolphin]

New thread:

Peter Denton White/Queen Mary, University of London again refuse to release data from £5m PACE Trial
http://forums.phoenixrising.me/inde...se-to-release-data-from-£5m-pace-trial.31436/

 

[alex3619]

Someone sent me this after reading my letters and thought I might find it interesting but wanted to stay anonymous if I was going to re-use it.
-----

If we can get a graph of the likely PACE scores, based on what we do know (inferred from the averages and SD) then we can do a composite indicative graph (not accurate of course) with different ranges superimposed.

 

[Dolphin]

There's a new (open access) PACE Trial paper - journal accepts e-letters:

The planning, implementation and publication of a complex intervention trial for chronic fatigue syndrome: the PACE trial

I've started a thread on it here: http://forums.phoenixrising.me/inde...-intervention-trial-cfs-the-pace-trial.31575/

 

[Graham]

I'm sure that all of you in the UK have been overjoyed to see that our economy has now recovered. Which just goes to show that it isn't only psychiatrists who can define their own version of recovery.

Mind you, the economy had been depressed.

 

[Dolphin]

In a reply to another paper, Frank Twisk mentioned the PACE Trial
http://www.rehab.research.va.gov/jour/2013/509/pdf/letterstotheeditor509.pdf

This would imply that many subjects with ME/CFS are physically incapable of improving their activity levels gradually. This is illustrated by the observation that even moderate exercise induces postexertional malaise, an increase in pain, “fatigue,” etc., and various (durable) abnormalities, e.g., an increase in metabolite detecting pain receptors [6], the fact that ME/CFS patients are unable to sustain a graded walking program starting at low levels and exhibit exercise intolerance, as implicated by reduced total activity after 4 to 10 days [6], and the observation that the increase in distance walked in 6 min after cognitive behavioral therapy and Graded Exercise Therapy is very minimal and insufficient to qualify as clinical improvement [7].

7. White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Davinton J, Angus BJ, Murphy G, Murphy M, O’Dowd H, Wilks D, McCrone P, Chalder T, Sharp M; PACE Trial Management Group. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011;377(9768): 823–36. [PMID:21334061] http://dx.doi.org/10.1016/S0140-6736(11) 60096-2

 

[Tom Kindlon]

This is a letter Adrian Baldwin and I just had published in Evidence Based Mental Health.

http://ebmh.bmj.com/content/early/2014/09/19/eb-2014-101961.extract

Unfortunately it is not open access

It is about the recovery definition used in the PACE Trial.

Friedberg & Adamowicz say that the recovery criteria in the PACE Trial were similar to those used by Deale et al (2001) in an earlier CBT trial, echoing PACE Trial authors claimed. We disagree pointing out how they are different.

We then also say the use of a threshold of SF-36 PF of 60 or more to define recovery is inappropriate, including by quoting population data.

 

[Dolphin]

Saw this posted on an ME mailing list. Haven't looked in to it:

NHS National Research Ethics Advisors’ Panel: PACE, SMILE, & CoIs minutes


National Research Ethics Advisors’ Panel PACE minutes [extracts]

https://dl.dropboxusercontent.com/u/89858260/NREAP on PACE.pdf

National Research Ethics Advisors’ Panel SMILE minutes [extracts]

https://dl.dropboxusercontent.com/u/89858260/NREAP on SMILE.pdf

National Research Ethics Advisors’ Panel Conflict of Interests minutes [extracts]

https://dl.dropboxusercontent.com/u/89858260/NREAP on Conflicts of Interest.pdf

 

[Graham]

Just a couple of quotes from your links Dolphin:

Principles that "might be applied by" Research Ethics C (council? committee?)

The following steps might be taken in order to mitigate the competing interest (N.B. these measures should be applied in a proportionate manner in accordance with the seriousness of the competing interest):

The investigator's financial interests/other competing interests should be publically declared and described in the participant information sheet
…..
Encourage researchers to make their research datasets publically available to allow independent validation of results
…..

and "The legal position within the UK":

Lord Falconer of Thoroton (2000). Freedom of information bill (Hansard). Column 261 - 5.

The 1998 Data Protection Act allows medical data to be used for any medical research purpose without the need for the consent of individuals. It is not necessary to define the term ‘medical research,’ nor to make specific provision for it to include the monitoring of public health, which for these purposes is regarded as medical research It is clear that many practitioners are confused between the requirements of the Data Protection Act and those of the various regulatory and representative bodies within the sector.

Information Commissioner. (2002). Use and disclosure of health data: guidance on the application of the Data Protection Act 1998.

...It is a common misconception, for instance, that the act always requires consent of data subjects to the processing of their data.

Boyd, P. (2003). Health research and the Data Protection Act 1998. Journal of Health Services Research and Policy. 8(sup 1): S1 - S7.

The two most widely held misconceptions are that the act creates an overarching requirement to obtain explicit consent for the processing of all health data and that the requirements of the act are additional to good professional standards, medical ethics and confidentiality. In fact, in most cases the act will almost never require consent for the processing of data for research purposes, unless consent is also a more general legal requirement.

Also, towards the end of the third link, page 4 of 10 in the final document, in a table asking questions like "will control of data rest in inappropriate hands?" (e.g. those with financial interests), two of the solutions are "Study should be registered on a publically accessible database" and "Encourage researcher to make research dataset publically available " (their spelling mistake, not mine!).

 

[worldbackwards]

Ho ho