'Recovery' from chronic fatigue syndrome after treatments given in the PACE trial

[Dolphin]

2. Changing protocol after research is scientifcly not done

If anyone is not going to write a letter but has references that make this point, feel free to share them on the thread or to me personally. Thanks.

 

[Esther12]

I think that points 1 and 2 could be turned in to something, but without expansion would be better left out.

re point 3: There are problems with the language there. I had a quick go at a re-write for you (assuming English is not your first language, it was a valiant effort! - if English is your first language, then I've just been rather rude)

White et al. sets a standard for recovered CFS patients in the following way: "The mean (SD) scores for a demographically representative English adult population were 86.3 (22.5) for males and 81.8 (25.7) for females (Bowling et al 1999) . We derived a mean (SD) score of 84 (24) for the whole sample, giving a normal range of 60 or above for physical functioning”.

White et al. use a score according to the general population (60), without taking into account age. According to table 3 increases in age lead to significant decreses in physical functioning (Bowling et al 1999) http://jpubhealth.oxfordjournals.org/content/21/3/255.full.pdf html

The SF36 PF scores according to table 3 (Bowling et al 1999) show the following means: (16-24): 83.4 (25-34): 81 (35-44): 79.9 (45-54): 66.3 (55-64) : 51.7

‘’The average age of the participants in the study of White et al was 38, 77% of them were female and 93% were of white ethnicity’’.

In Bowling et al. 1999 those aged (35-44 years) tabel3, the mean - 1sd for physical functioning is a score of 79.9.

This is much higher than the score of 60 used by White et al.

Using a general population, of whom 28% are aged over 65 has led to White et al claiming that low levels of physical functioning are normal for the working age patients that were being assessed. According to White et al 2013, 22% of CFS patients who received cognitive behavioural therapy had recovered, however it is only once patients fall in to the ages of 55-64 or above that a score of 60 can be said to fall within the mean -1sd of an age matched population.

Firestormm:

I don't think much of substance was said in the HofL debate.

Point 1 - as you say is relevant given how results were sold and presented in the media (although I agree that these sorts of problems are not really surprising in trials).

Point 2 - as you say, it's pretty pathetic, especially if these dramatic changes are combined with a refusal to release data in the manner laid out in their protocol as supplementary information. Just because those speaking in the House of Lords do not think it's fair for patients to want access to the results from a publicly funded trial in the manner laid out in the trial's protocol, and should be happy with however it is the researchers choose to present their results to us, doesn't mean that there's any reason to think that they are right.

 

[Marco]

Comparing patients with various conditions (including after treatment) against SF36 age matched norms is not unusual (particularly when such norms are available).

These dystonia patients would be considered 'recovered' on SF36 PF according to PACE but still far off the age matched norm!

http://www.springerimages.com/Images/MedicineAndPublicHealth/1-10.1007_s00415-009-5060-3-0

 

[Gijs]

------------------------------------------------------------------------------------------------------------------------------------
Tabel 3
age 16-24 25-34 35-44 45-54 55-64 65-74 75-84 85+
N for each group n = 204 n =415 n =319 n =297 n = 297 n=281 n=296 n=36
Mean (SD), 95.5 (12.1) 94.5 (13.5) 93.3 (13.4) 87.2 (20.9) 78.0 (26.3) 72.7 (26.7) 57.9 (28.6) 39.3 (31.5)
physical function
---------------------------------------------------------------------------------------------------------------------------------
Male 86.3 (22.5) n=925
Female 81.8 (25.7) n=1117
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Bowling et. al., 1999

Standard for recovery: ’The average age of the participants in the study of White et al, mean (SD) 38 (12), 77% of them were female. The mean (SD) scores for a demographically representative English adult population were 86.3 (22.5) for males and 81.8 (25.7) for females (Bowling et al 1999) . We derived a mean (SD) score of 84 (24) for the whole sample, giving a normal range of 60 or above for physical functioning''.

The average age (38) in the studied group of White et.al. falls between 26 and 50 years. The standard to determine recovery should consider the average score in this group of age not the average general standard stated by White et.al., because the average age of this population falls between 16 and 85+ years table 3. The standard for normal physical function in this age group will be substantially lower because an older group of age steal a proper standard. The standard of White et. al. therefore is not representative for participants with an age between 26 to 50 years.

------------------------------------------------
Example on data table 3 *
-------------------------------------------------
age normal range or above
------------------------------------------------
(25-34) 81
(35-44) 79.9
(45-54) 66.3
------------------------------------------------
Average score in the age from 25 and 54 years = 75.7

*This standard is statistically not accurate because you need individual data but it is an indication to show that the standard for recovery stated by White et.al., is not correct without taking age by groups in to account.

The standard giving a normal range need to be adjusted.

 

[Graham]

Sorry to harp on, but you are still using the mean and variance. Are you sure that the data is Normally distributed? I suspect it is strongly skewed. If we calculate the mean and variance in income of adults in the UK in 2007/8 we get £26800 and £29500, which gives us a "normal range" of income of between £56300 and minus £2700. In contrast, if we use the median (the middle value) as an average, it is a more sensible £18500, and the quartiles (i.e. the middle half of the earners) get between £11800 and £29500. I did the calculations for the Chalder Fatigue scale, and found that the data there had the same coefficient of skew as the incomes. Have you tried the median and percentiles to estimate normal range? It may be quite different.

Recently both Simon and I followed a statistics course with a slant on psychology, and before any calculations were made, it was emphasised that the data had to be checked to make sure that it was "well-behaved". It still is common to use means and variances, but that doesn't make it right: but I have seen several studies using the median and percentiles.

 

[Gijs]

No problem Graham. If you want do good statistics you need all data of individuals. But it is clear that the standard white et. al. used is not representative for his population by age and sexe. His population is relatively young and that substantially influence the outcomes. It is not comparable.

 

[user9876]

No problem Graham. If you want do good statistics you need all data of individuals. But it is clear that the standard white et. al. used is not representative for his population by age and sexe. His population is relatively young and that substantially influence the outcomes. It is not comparable.

If you look at what data is available you will notice that the worse the scores the larger the standard deviation. I could explain this in terms of heathy people being described as a random variable hence forming some kind of distribution but the many health problems both with long term health issues and aging as being fairly randomly (and independantly distribued).

Looking at the distibutions from Bowling's paper the basic graph looks a bit like an exponential distribution in shape (although not necessarily in spread) along with some noise created by the addition of people who are 65+, those with long term health problems and those with short term health problems (ill in last 2 weeks, visited hospital in previous year).

Including people non healthy people thus has two effects firstly it reduces the mean and secondly it leads to considerable increases in the standard deviation.

Looking at the Bowling paper
22% had long term health problems (mean 52 std 29)
16% had accute health problems
11% were 75+
13% were between 65 and 74

Note that the standard deviations increase considerably with age as the means decline. My explanation would be that people are having more health problems and hence the distribution is affected by increasing numbers of people with health problems.

The important to note that the inclusion of people with health problems in the data decreases the mean and increases the standard deviation hence giving a very wide range for normal.

Graham is of course right in that it is wrong to characterise the data using the standard deviation and the mean is not a robust measure if we consider the distribution to be multimodal. Hence we should use the median, percentiles (or the median absolute deviation)

http://www.archive.official-documents.co.uk/document/doh/survey96/tab5-18.htm

Give some median and percentile figures but they note the numbers with health problems rises with aging.
http://www.archive.official-documents.co.uk/document/doh/survey96/tab5-3.htm
Hence the sf36-pf scores decrease.

Even including these issues they have a 25 percentile of 75. That is only 25% of the population fall below the score of 75. When looking at the 35-44 age group (given average age in PACE of 38) the 25 percentile is 90.

To demonstrate issues with the mean for the 35-44 age group there is a median of 95 and mean of 89 (25th percentile 90). But as the health of people declines the median moves more towards the mean (as would be expected). So the mean for 75+ is 52 with a median of 55. For the overall population the mean is 81 compared to the median of 95.

 

[user9876]

If anyone is not going to write a letter but has references that make this point, feel free to share them on the thread or to me personally. Thanks.

From offical EU Guidance on changing trial protocols

http://www.archive.official-documents.co.uk/document/doh/survey96/tab5-3.htm

Talks about the legal needs to notify the ethics commitee and national bodies for drug trials wanting to make substantial protocol changes (Page 12 onwards) with examples of substantial (page 14,15) including:

change of primary or secondary endpoint which is likely to have a significant impact on the safety or scientific value of the clinical trial;

a change in the definition of the end of the trial, even if the trial has in practice already ended;

(n) a change of study design which is likely to have a significant impact on primary or major secondary statistical analysis or the risk/benefit assessment.

Also page 23 section 5.1 of the european medical agencies statistical principles for clinical trials

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500002928.pdf

talks about prespecification of the analysis

5.1 Prespecification of the Analysis

When designing a clinical trial the principal features of the eventual statistical analysis of the
data should be described in the statistical section of the protocol. This section should include
all the principal features of the proposed confirmatory analysis of the primary variable(s) and
the way in which anticipated analysis problems will be handled. In case of exploratory trials
this section could describe more general principles and directions.

The statistical analysis plan (see Glossary) may be written as a separate document to be
completed after finalising the protocol. In this document, a more technical and detailed
elaboration of the principal features stated in the protocol may be included (see section 7.1).
The plan may include detailed procedures for executing the statistical analysis of the primary
and secondary variables and other data. The plan should be reviewed and possibly updated as
a result of the blind review of the data (see 7.1 for definition) and should be finalised before
breaking the blind. Formal records should be kept of when the statistical analysis plan was
finalised as well as when the blind was subsequently broken.

If the blind review suggests changes to the principal features stated in the protocol, these
should be documented in a protocol amendment. Otherwise, it will suffice to update the
statistical analysis plan with the considerations suggested from the blind review. Only results
from analyses envisaged in the protocol (including amendments) can be regarded as
confirmatory.

In the statistical section of the clinical study report the statistical methodology should be
clearly described including when in the clinical trial process methodology decisions were
made (see ICH E3).

There is

 

[Graham]

You are entirely right Gijs. White et al were utterly wrong to do what they did.

Actually I would argue that to include healthy and ill people's scores in one distribution is adding together two very different types. It would be like adding adult males' heights to those of children of various ages. You generally get a Normal (or Gaussian) distribution when there are a myriad of small effects all combining (as in intelligence test scores), but I would argue that there isn't much of a distribution at all for healthy people, because the tests are not designed for that. Each illness would have its own distribution which may not at all be Normal.

I don't think any of this "normative" sets of data have a single distribution. The only way to set targets or levels is to actually look at the scores of healthy people in the right age range and make an educated and informed decision. Statistics is a tool, not a compulsion.

Speech over (sorry, but it's the maths teacher in me, Gijs!)

 

[biophile]

Tossing in some more physical function data from a general population:

http://www.ncbi.nlm.nih.gov/pubmed/9817123

http://share.eldoc.ub.rug.nl/FILES/root2/1998/Tranvaann/Aaronson_1998_Journal_of_Clinical_Epidem.pdf

Mean (SD) [range]
Age: 43.1 (18.1) [16–97].
General population PF: 85.2 (23.1)
[0.7% floor ie 0 and 45.0% ceiling ie 100]

Chronic health conditions: 56%.

No chronic health condition (all ages): 94.1 (13.2).

1 chronic health condition: 88.5 (18.8).

>1 chronic health condition: 70.0 (28.3).

Migraine attack past 2 weeks PF: 84.0 (21.3).

Metastatic stage of cancer PF: 59.2 (26.3).

 

[biophile]

The only way to set targets or levels is to actually look at the scores of healthy people in the right age range and make an educated and informed decision.

The several CFS papers I have come across which report PF scores for healthy age-matched controls report mean(SD) scores of around 96(4). Admittedly the sample sizes were much smaller too.

 

[Dolphin]

------------------------------------------------------------------------------------------------------------------------------------
Tabel 3
age 16-24 25-34 35-44 45-54 55-64 65-74 75-84 85+
N for each group n = 204 n =415 n =319 n =297 n = 297 n=281 n=296 n=36
Mean (SD), 95.5 (12.1) 94.5 (13.5) 93.3 (13.4) 87.2 (20.9) 78.0 (26.3) 72.7 (26.7) 57.9 (28.6) 39.3 (31.5)
physical function
---------------------------------------------------------------------------------------------------------------------------------
Male 86.3 (22.5) n=925
Female 81.8 (25.7) n=1117
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Bowling et. al., 1999

Standard for recovery: ’The average age of the participants in the study of White et al, mean (SD) 38 (12), 77% of them were female. The mean (SD) scores for a demographically representative English adult population were 86.3 (22.5) for males and 81.8 (25.7) for females (Bowling et al 1999) . We derived a mean (SD) score of 84 (24) for the whole sample, giving a normal range of 60 or above for physical functioning''.

The average age (38) in the studied group of White et.al. falls between 26 and 50 years. The standard to determine recovery should consider the average score in this group of age not the average general standard stated by White et.al., because the average age of this population falls between 16 and 85+ years table 3. The standard for normal physical function in this age group will be substantially lower because an older group of age steal a proper standard.The standard of White et. al. therefore is not representative for participants with an age between 26 to 50 years.

------------------------------------------------
Example on data table 3 *
-------------------------------------------------
age normal range or above
------------------------------------------------
(25-34) 81
(35-44) 79.9
(45-54) 66.3
------------------------------------------------
Average score in the age from 25 and 54 years = 75.7

*This standard is statistically not accurate because you need individual data but it is an indication to show that the standard for recovery stated by White et.al., is not correct without taking age by groups in to account.

The standard giving a normal range need to be adjusted.

While I can see why Graham criticised it, it can still be useful to show that that using the PACE Trial investigators own methods (mean - 1 SD), the threshold could be different if the the range of ages was restricted, as Gijs did.

 

[kaffiend]

Pasted below is nice article (with references) on the validity of fatigue scales used in CFS. The article recommends using the Fatigue Severity Scale - at a glance it would appear to be very sensitive to CFS type fatigue.

http://www.cfids.org/archives/2002rr/2002-rr4-article02.asp

Is there any way to start a wiki-type project to pool the main criticisms of this paper? A lot of important points are being brought up in this thread.

 

[Dolphin]

Is there any way to start a wiki-type project to pool the main criticisms of this paper? A lot of important points are being brought up in this thread.

At least a few of us hope to/plan to submit letters. We would then intend to collate letters that are published as well as those that are not published, the way we did with the Lancet PACE Trial paper:

PACE Trial - letters that were published and authors' response (and editorial)
http://forums.phoenixrising.me/show...ublished-and-authors-response-(and-editorial)

PACE Trial - letters that were not accepted by the Lancet:
http://forums.phoenixrising.me/show...-letters-that-were-not-accepted-by-the-Lancet

A wiki would be good, but my preference is, for the moment, that people would focus on submitting letters. Having a complaint/criticism published in a journal gives it more status.

 

[kaffiend]

Has anyone identified a clear case in which the change to the Chalder Fatigue Questionnaire (CFQ) from binary scoring to Likert scale had an effect on meeting case definition criteria? Is it possible to score worse at 52 weeks and be classed as recovered? The study has a heavy reliance on this tool.

 

[Dolphin]

Has anyone identified a clear case in which the change to the Chalder Fatigue Questionnaire (CFQ) from binary scoring to Likert scale had an effect on meeting case definition criteria? Is it possible to score worse at 52 weeks and be classed as recovered? The study has a heavy reliance on this tool.

Regarding the latter, I gave an example here:
http://forums.phoenixrising.me/inde...n-in-the-pace-trial.21628/page-11#post-332889

First number is baseline, second is at 52 weeks:
CFQ (Likert): 17 --> 18 (i.e. a deterioration) e.g. 22222211111-->33222111111 but now within normal range
CFQ (bimodal): 6 --> 5 e.g. scores in last line translates to 11111100000 --> 11111000000

 

[Graham]

While I can see why Graham criticised it, it can still be use to show that that using the PACE Trial investigators own methods (mean - 1 SD), the threshold could be different if the the range of ages was restricted, as Gijs did.

Too true Dolphin. I never meant it as a criticism: it was more an attempt to add to its faults. I'm not too good at differentiating between what I mean and what I actually say!

Kaffiend, you may find our table on the Chalder Fatigue Scale useful.

We also carried out a survey on Chalder Fatigue Scores.

http://evaluatingpace.phoenixrising.me/aps6furtherdetai.html

and

http://evaluatingpace.phoenixrising.me/aps6survey.html

 

[Dolphin]

ME Association response to PACE trial recovery paper | 15 February 2013
by Dr. Charles Shepherd:
http://www.meassociation.org.uk/?p=14460

Useful letter, has a chance of being published I would have thought.

Note: I wouldn't recommend making your letter public until the letters are published in the journal (or you are told it is not going in) but perhaps the ME Association can get away with it.

 

[Firestormm]

ME Association response to PACE trial recovery paper | 15 February 2013
by Dr. Charles Shepherd:
http://www.meassociation.org.uk/?p=14460

Useful letter, has a chance of being published I would have thought.

Note: I wouldn't recommend making your letter public until the letters are published in the journal (or you are told it is not going in) but perhaps the ME Association can get away with it.

Dolphin, Does the journal that published the paper routinely publish letters do you know? I know that some are more amenable that others to do so - and on-line. I presume there is a process and not all submitted letters 'make the grade'. Hopefully this one will and will prompt a response. Am guessing that authors are obliged to respond?

 

[Dolphin]

Dolphin, Does the journal that published the paper routinely publish letters do you know? I know that some are more amenable that others to do so - and on-line. I presume there is a process and not all submitted letters 'make the grade'. Hopefully this one will and will prompt a response. Am guessing that authors are obliged to respond?

One can get free access to Cambridge journals published in 2012 incl. this journal to see what they published. They published plenty of letters - more than some journals, but less than others.

I don't think any journal publishes all letters. Some journals allow e-letters/rapid responses/online comments which can be a much lower standard and still go up on the site.

Charles' letter hasn't covered all the points that could be made so I hope more letters will be submitted. I would hope they would publish more than one, given it's a £5m trial, but who knows for definite. We can collate the ones that aren't published which would be a good way of summarising points. I would hope to highlight a list of submitted letters so they got some attention.