JonathanWest
3h ago
Contributor
I've had a look at the paper. There are a number of issues which are extremely obvious if you've ever read Ben Goldacre's "Bad Science".
The first is that the trial is very small. It is so small that it would take a huge difference in outcomes to be fairly sure that the difference wasn't down to mere chance.
The second is that although the groups were randomised, there was no attempt to control for the placebo effect. The comparison was between existing treatment + lightning programme and existing treatment alone. There's no way that which programme a patient is allocated to can be hidden from the patient, and so no way of assessing how much improvement might be down to placebo effect rather than lightning programme.
Third, the improvements do look somewhat marginal, and there are quite a few things measured in the SF-36 questionnaire, and there doesn't seem to have been any attempt to assess the statistical significance of the data in the light of this. If you're testing for multiple outcomes, there is a much increased chance that one of your outcomes shows what initially appears to be a statistically significant difference. In fact there seems to be no mention of confidence intervals and therefore no attempt at all to assess whether the results are statistically significant.
We normally think of a statistically significant outcome as being one where there is no more than a 5% probability of seeing that outcome by chance. But if you're measuring 36 different things, then there's an 85% probability that one or more of them will fall into the 5% bracket, just by chance.
As far as I can see, the combination of multiple outcomes, small sample size (100 patients is a
verysmall trial), no mention of confidence intervals and lack of control for the placebo effect means that the numbers obtained by the trial are essentially meaningless.
I welcome research into ME. I would like to see more of it. I'm aware of the suffering it can cause. (A close relative of mine suffered from it.) But I regret to say that it appears that this paper has not made any significant addition to our knowledge of it. It merely serves to give false hope to the suffering patients who are looking for something that will help and who do not have the statistical knowledge to realise how meaningless this paper appears to be.