Note: this is actually a follow-up on my previous post The Sound of Silence. This was provoked by "expert opinions" on the Hornig/Lipkin study made public by the UK's Science Media Center, and in particular the comment that this was only "a small study". Readers looking for more topical material will be disappointed.
When I commented on the omissions in the PACE trial one might have thought I was being unduly negative. Actually, I was using an estimate made by P.D. White that 40% of patients in that trial failed to respond. I would be willing to adjust that number if any data were available to allow others to independently judge what percentage responded.
The size of the cohorts studied w.r.t. the favored PACE therapies was actually smaller than those in the cytokine study above which had 298 cases and 348 controls. In the CBT arm there were 148. In the GET arm there were 154. Of these only 60% were said to be responders, based on criteria outsiders cannot check. Please note that PACE carefully avoided comparing performance with any healthy controls, using instead data on the general population which included many people over the age of 65 and some with serious medical problems which would be exclusionary for participants. The "control arm" actually received specialist medical care (SMC), and performance on the 6-minute walk test advanced by some 22 meters. This raises questions about the extent to which this group could be considered a valid control.
There was actually a hidden positive bias in my estimate that selectivity in the trial was such that we could only say that 384 out of 3158 in the initial intake might be said to benefit. In all arms of the trial roughly 1/3 of the participants failed to complete the 6-minute walk test both at the beginning and end of the trial. This means there was no objective data at all from them on any possible therapeutic benefit. If we are only to judge results based on what was actually measured we should exclude those, reducing the objectively-measured responders to 256 out of 3158, or a little over 8% of the original intake. Please note that only about 1/4 of these were subject to any particular intervention. We are getting down to the point where behavior of individual patients can significantly affect group means. This raises real concerns about rates of misdiagnosis.
(There was a proposal to use actigraphy to measure total patient activity in the original protocol, but this was dropped at about the time it might have shown that patients were displacing activity from a fixed energy budget to participate in therapy, as has happened in other studies. This left any claim to objective measures hanging by a slender thread.)
The claim that the trial was randomized only applies to the assignment of subjects to different arms of the trial. It says nothing about any systematic biases introduced during the selection process which reduced the original 3158 to 640 who completed the trial. There is every reason to assume the authors did their best to select suitable subjects to demonstrate the effects of their preferred therapies. Since these selections were all subject to review -- by the same people -- this amounts to an enormous systematic influence on the outcome. The authors certainly had their innings.
The "Oxford definition" of the illness, which seems to have been the primary basis for this study, dates back to 1991. If there were ever any doubt that UK physicians working for the NHS had been exposed to this it should be gone by now. Despite this the authors had to reject the vast majority of patients in the initial intake referred by such doctors as unsuitable. We still don't know what to do with the bulk of patients ordinary NHS doctors classify as suffering from "CFS". This study did not shed much light on that aspect of the problem.
(If this were any more than an excuse to deny or defer disability benefits, and/or effective treatment options, I should think policymakers would be troubled by such an outcome.)
A "step test" used to measure physical performance failed to show any such change following a year of therapy. This comes on top of a measure of physical performance on the six-minute walk test which left the best group, including responders, short of the threshold for performance used to classify patients with stage III heart failure, which none were said to have.
So, to sum up, we see no improvement in physical performance, no improvement in employment, no reduction in demand for benefits and/or services. Millions of pounds and a year of therapy went in, and what came out? I see no reason to reject the null hypothesis.
When I commented on the omissions in the PACE trial one might have thought I was being unduly negative. Actually, I was using an estimate made by P.D. White that 40% of patients in that trial failed to respond. I would be willing to adjust that number if any data were available to allow others to independently judge what percentage responded.
The size of the cohorts studied w.r.t. the favored PACE therapies was actually smaller than those in the cytokine study above which had 298 cases and 348 controls. In the CBT arm there were 148. In the GET arm there were 154. Of these only 60% were said to be responders, based on criteria outsiders cannot check. Please note that PACE carefully avoided comparing performance with any healthy controls, using instead data on the general population which included many people over the age of 65 and some with serious medical problems which would be exclusionary for participants. The "control arm" actually received specialist medical care (SMC), and performance on the 6-minute walk test advanced by some 22 meters. This raises questions about the extent to which this group could be considered a valid control.
There was actually a hidden positive bias in my estimate that selectivity in the trial was such that we could only say that 384 out of 3158 in the initial intake might be said to benefit. In all arms of the trial roughly 1/3 of the participants failed to complete the 6-minute walk test both at the beginning and end of the trial. This means there was no objective data at all from them on any possible therapeutic benefit. If we are only to judge results based on what was actually measured we should exclude those, reducing the objectively-measured responders to 256 out of 3158, or a little over 8% of the original intake. Please note that only about 1/4 of these were subject to any particular intervention. We are getting down to the point where behavior of individual patients can significantly affect group means. This raises real concerns about rates of misdiagnosis.
(There was a proposal to use actigraphy to measure total patient activity in the original protocol, but this was dropped at about the time it might have shown that patients were displacing activity from a fixed energy budget to participate in therapy, as has happened in other studies. This left any claim to objective measures hanging by a slender thread.)
The claim that the trial was randomized only applies to the assignment of subjects to different arms of the trial. It says nothing about any systematic biases introduced during the selection process which reduced the original 3158 to 640 who completed the trial. There is every reason to assume the authors did their best to select suitable subjects to demonstrate the effects of their preferred therapies. Since these selections were all subject to review -- by the same people -- this amounts to an enormous systematic influence on the outcome. The authors certainly had their innings.
The "Oxford definition" of the illness, which seems to have been the primary basis for this study, dates back to 1991. If there were ever any doubt that UK physicians working for the NHS had been exposed to this it should be gone by now. Despite this the authors had to reject the vast majority of patients in the initial intake referred by such doctors as unsuitable. We still don't know what to do with the bulk of patients ordinary NHS doctors classify as suffering from "CFS". This study did not shed much light on that aspect of the problem.
(If this were any more than an excuse to deny or defer disability benefits, and/or effective treatment options, I should think policymakers would be troubled by such an outcome.)
A "step test" used to measure physical performance failed to show any such change following a year of therapy. This comes on top of a measure of physical performance on the six-minute walk test which left the best group, including responders, short of the threshold for performance used to classify patients with stage III heart failure, which none were said to have.
So, to sum up, we see no improvement in physical performance, no improvement in employment, no reduction in demand for benefits and/or services. Millions of pounds and a year of therapy went in, and what came out? I see no reason to reject the null hypothesis.
