ME Analysis - Patient created Web document relating to the PACE study

[user9876]

In your analysis you talk about using better measures than the mean and standard deviation. I would go further and argue that the mean and standard deviation, and median and percentiles are not valid for the way they have processes the data from the chandler fatigue scale. Basically they say lets treat it as a likert scale and add up all the values and use this as a measure. That is simply wrong!

The chandler fatigue scale is a multi factor scale (including influances of physical fatigue and mental fatigue - look at the principle component analysis quoted in their original paper). A likert scale requires all the questions to relate to one underlying thing or attitude. You should show there is a high degree of correlation between the answers of all the questions. What they have created is a scale that is not necessarily monotonic - that is an increase the things you are measuring will not necessarily lead to an increase in the measure they are using. They could talk about the mode but that wouldn't be very useful.


Even if they had a single thing they were measuring they need to worry about whether it is an ordinal or interval scale. That is is the scale linear - does the gap between 1 and 2 represent the same gap as between 4 and 5. This is complicated by two factors, firstly the scale on which questions are answered. People answering the scale need to understand it as even and the scale needs to represent the full extremes of the possible answers. If we see bunching at the end of a scale then the end points probably represent a much bigger interval than the middle points.

The second factor is when you add up all the results of different questions, since there are no units it is hard to say that the resulting sum evenly covers the overall scale. I beleive psyciatrists tend to say but our intervals are of the same order of magnatide hence its ok.

If the scale is really an interval scale then you can use the mean and std and do things like regression on the results however if it is not you cann't.

As long as the results are ordered even if not evenly then you have an ordinal scale which you can quote the median and percentiles for. But the mean and std make no sense.

I would argue that if their questionaires only measure one (unseperable) concept then they have an ordinal scale. They may be able to argue that they have an interval scale but it is up to them to demonstrate that they have the correct properties. Where they have things like the chandler fatigue scale then there analysis is just wrong as its measuring some unknown function of two or more different things.

 

[Graham]

Agreed. Just on a very simple level, an improvement of 4 points on the fatigue scale can mean so many different things. But I suppose that this is also true of exam marks. An improvement from grade E to grade D gives absolutely no indication of what a person can and can't do.

Your argument reflects the way I think about the whole approach, but it is very mathematical in nature and we wanted to make this report accessible, as opposed to the more technical and finely accurate analyses produced elsewhere. It took me a long time to find a way to present the Bimodal/Likert conflict in graphical form, and yet now it looks so obvious: I did think about the "ordinal scale", but I just couldn't come up with a way of tying it all together in an accessible way. If you come up with anything, I'd be very interested to see it.

The main point is, of course, simply that even on their terms, there is no evidence of CBT or GET being worthwhile, and it is possible that their assessments made it harder to record harm. If we had been given information about medians and quartiles, we would have had a better picture of what happened to the majority of scores, and it would have underlined the fact that most patients hardly improved. As I see it, if we are not careful we can overwhelm that simple message by being too precise.

If they had stuck to the original Bimodal scoring, then the fatigue scale would simply have been a list of how many items gave difficulties. That would have been much clearer, but my guess is that no-one would have recorded much of a change in scores over the year.

 

[Enid]

Can I just add that it seems to me impossible to come up with anything of value from PACE. Yes from mostly bedbound - gradual increase in activity - my Doc and Osteopath encouraging, muscles moving and walking a little more, feeling better - then crash. Back to bed, feeling worse (after about 6 months of improvement). My Osteopath declined to treat and my Doc likewise. Additional problems set in - passings out, severe GI problems etc - all as we all know. Measure of fatigue in isolation from the rest of the illness is nonesense to me. Or to be blunt pathetic. Long term study like Graham's much more revealing.

 

[Graham]

For those of you who have been involved in the survey, and have a link to the campaign page, I have added a copy of the email that I sent to Richard Horton, editor of The Lancet, on 17th May, in response to his comment in an interview where he issued a challenge to the patient community to engage in a proper scientific discussion of the issues raised by the PACE trial. I haven't had any reply.

I then wrote to the deputy editor, Dr James, asking for an acknowledgement that the email had been received. That was a week ago - no reply.

It is an interesting interpretation of the term "discussion".

Now to consider the next moves.

 

[biophile]

Graham, you will probably never receive a reply from Horton. AFAIK, no one here who has emailed him has ever received one. I received an auto-reply when he was on holiday, which indicates that I got the address correct. Also, his "challenge" was issued over a year ago, it is unfortunate that due to illness it takes us so long to respond in detail, but he probably expected a response within weeks, so for him it would be old news by now. However, if the Lancet publish another PACE paper or related article/editorial then it might be a good time to bring the issue up again.

 

[Graham]

True enough. To be honest, I didn't really expect one. But it was the proper thing for me to do as a first step. If he chooses to ignore it, then it leaves me more free to take the next steps.

On the issue of time though, as it has taken a well-funded, well-supported and healthy PACE team over a year now to fail to come up with their next level of analysis of the data, I don't think we have done too badly

 

[biophile]

I agree. They have several papers planned, and this FOI request suggests another paper is getting closer to submission for publication (if there is no recovery or employment data in the next paper we will know something is wrong) :

http://www.whatdotheyknow.com/request/request_for_deterioration_rates

 

[user9876]

Your argument reflects the way I think about the whole approach, but it is very mathematical in nature and we wanted to make this report accessible, as opposed to the more technical and finely accurate analyses produced elsewhere. It took me a long time to find a way to present the Bimodal/Likert conflict in graphical form, and yet now it looks so obvious: I did think about the "ordinal scale", but I just couldn't come up with a way of tying it all together in an accessible way. If you come up with anything, I'd be very interested to see it.
.

Yes my argument is very mathematical, I spend most of my time talking with engineers and mathematicions, and I'm not good at making things accessable. In other areas we are trying to use simulation, visualisation and games to get across complex ideas. I'm trying to think about how such an approach could be used with things like the PACE trial.

I don't think we will ever change their minds but it would be good if we could get the wider medical community to understand how bad their methodologies are.

 

[user9876]

For those of you who have been involved in the survey, and have a link to the campaign page, I have added a copy of the email that I sent to Richard Horton, editor of The Lancet, on 17th May, in response to his comment in an interview where he issued a challenge to the patient community to engage in a proper scientific discussion of the issues raised by the PACE trial. I haven't had any reply.

I then wrote to the deputy editor, Dr James, asking for an acknowledgement that the email had been received. That was a week ago - no reply.

It is an interesting interpretation of the term "discussion".

Now to consider the next moves.

How about trying:

Erik Engstrom
Reed Elsevier
1-3 Strand
London WC2N 5JR
United Kingdom


He is the CEO of Reed Elsevier who own the Lancet. I would have thought that they would have an interest that one of there major brands could become devalued because their review and editorial process is so weak that they publish papers with bad stats and where the entry conditions of someone being sick overlaps with exit conditions of someone within a "normal" range of activity.

Another thought is to push links to your we site on shareholder's message boards such as
http://uk.messages.finance.yahoo.com/UK_Stocks/forumview?bn=UKF-Stocks-REL.L

Seems to me that publishers will only really take critisism seriously if it hits their reputation and share price.

 

[Graham]

Nice suggestions! I have added them to my "to do" list (it seems to keep growing no matter how much I get done!).

I cannot get my head around why the medical world find this so difficult to understand. You and I clearly think alike on this. If I translate the situation to my world as a maths teacher, it looks utterly stupid.

If I had a class of 30 students who struggled with maths who were in their last year coming up to major exams, and I knew that 4 of them suffered badly from exam nerves and would benefit from six months of concentrating on exam techniques, relaxation tricks, improved pacing and timing, rereading questions after answering to make sure that they have done what was asked etc., the current ME strategy would then be to devote all the lessons for the next six months on exam techniques. Obviously it would boost those 4 students' marks, and may help some of the others pick up a couple of extra marks. But it would be utterly unprofessional and stupid for three main reasons.

Firstly, I could not justify spending so much time with the whole class on this when only a few need it. The appropriate thing to do would be to identify those 4 students and deal with them separately.

The second is that it would prevent me from actively helping the majority, and spending time and resources on their needs.

The third and biggest fault is that exam nerves are not a maths problem: they can affect any subject. The appropriate action would be for the school to set up special classes for those who are identified as having exam nerves, freeing me to deal with the specific problems of helping students to improve their mathematical understanding. There would be a risk there of interpreting exam nerves as being a maths problem, rather than an exam problem. Why can't they see that?

Can you just imagine the uproar from the parents if I spent 6 months not doing any maths but just concentrating on exam techniques? And if I produced evidence at the end that suggested there was a weak improvement over those 6 months (mostly caused by the great improvement of those 4 averaged out over the whole group), who would fall for it?

By the way, 4 in 30 is the correct proportion. CBT and GET, according to PACE, have a "number needed to treat" of 7 and 8 (or it might be 8 and 7).

Rant over!

 

[Enid]

What really galls is that these numbskulls have and do turn a blind eye to all the scientific discoveries/pathologies found over years, let alone medical treatments now being used for some symptoms. Who are the "trained" Docs, GP's they speak of - the NHS doesn't carry out the range of testing needed to identify the problems found everywhere else but the UK - so what are these apparently trained people trained in.

 

[Dolphin]

This is an impressive project. Even too impressive.

Now, the question is who is going to be your audience. Because if you know who your target audience is, you will know how to prepare it in a way that- A. will not raise antagonism. B. will get them to understand. C. will (hopefully) get them on your side.

Think of your worst opponents and think how you are going to get to them.

Love your enemies because they bring out the best in you. Friedrich Nietzsche

Because, being the devil's advocate, I can see how people (who do not understand this illness) are going to ask- well, if they can produce something like that, why can't they do it on a regular basis...
There are very good ways to deal with those kind of questions, but I think you have to be prepared for them.
(The first time my accomplishments were used against me like that, I was quite devastated, but with time I learned. Although, it is still one of the hardest things to deal with, for me).


The best to learn from are drug companies. They are just amazing in marketing.

I also think that for the above reasons you need to make a very short summary that can be viewed in seconds and convey the message. I would personally take off the "phoenix rising" logo, just like drug companies don't put their logo on the studies they sponsor. Drug companies are experts in creating such pamphlets which they distribute to every physician on earth. Physicians are busy people and don't have time to read lengthy documents which are not in their direct field of expertise. The rocking chart in the corner (which is really cute) creates a feeling of pleasant leisure. This is definitely not the impression you want to create. Also, the animations should preferably be seen by a common program and not something you have to download.

Also, physicians want to help patients, but they also don't want to cause any harm, so if they have something which is possibly effective, but can't cause harm (and they don't have anything better to offer) they will gladly go with it. Many of them have seen the paper in Lancet (most probably only read the abstract) and it clearly conveyed the message that it may or may not help your patient, but will definitely cause no harm. I doubt many (including the reviewers?) went to look at the supplementary material or asked themselves what is the difference between adverse response and adverse event and what is the meaning of such a large number of adverse events in a supposedly healthy population of young people and why were the serious adverse events of the treatment group clustered with those of the non-treatment group.

The main point of the PACE trial was- this causes no harm and may even be effective in some patients.
Most people (including physicians) don't take CBT very seriously. They tend to think-what harm can " talk therapy" do. Because they don't take it seriously, they are ready to use it without putting much thought into it.
Physical exercise is seen as the ultimate remedy for everything, so it can only be beneficial.
Very few physicians, for instance, are aware of the fact that acupuncture can be fatal.

The point you have to stress, is that it can cause harm.
GET is not harmless, and CBT is not harmless either. Use clear animations to show this. (just like you did with the 6 min. walk test). Look at the adverse responses, but also at the adverse events in the supplementary material. Compare it to adverse events of medications. Stress the fact that CBT and GET are serious medical interventions.

Your goal is not to convince everyone, but to create a discussion. Turn this into a controversial issue and get as many people as possible involved. (See for instance the debate regarding antidepressants).
If you can get a good medical journal to publish a " for" and " against" this will be a good step in the right direction.

Please don't see this as criticism of your work. On the contrary. I think it is excellent and it would be a shame if it doesn't create the proper impact.
Those are just some minor suggestions which I hope can help.
I would even consider trying to get a PR company involved.

FWIW, the issue of the reporting of harms in the PACE Trial is dealt with in quite a lot of detail in section 6 of this paper which was published in a peer-reviewed journal:

Kindlon T. Reporting of Harms Associated with Graded Exercise Therapy and Cognitive Behavioural Therapy in Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Bulletin of the IACFS/ME. 2011;19(2):59-111. Free at: http://iacfsme.org/BULLETINFALL2011/Fall2011KindlonHarmsPaperABSTRACT/tabid/501/Default.aspx

If people think important points have been missed, they could write a letter to the editor for this journal.

 

[Dolphin]

One idea to get published would be the Chalder Fatigue Scale data. That is a scale that is used in a lot of studies. Then in the introduction and discussion sections, would could potentially discuss how this might relate to the PACE Trial.
Here is an example of such a study:

Fatigue in Myalgic Encephalomyelitis
http://www.iacfs.net/BULLETINFALL2008/Fall08GoudsmitFatigueinMyalgicEnceph/tabid/292/Default.aspx

Even if you didn't manage to make all your points in any one paper, it could be quoted in future studies/papers/letters.

 

[Graham]

Thanks for the suggestion, Dolphin. I have two difficulties with this though. The first is that there is masses of good-quality, scientific-style debunking of the psychological interpretation of ME, but it doesn't seem to be getting through to mainstream understanding. So would it be worth my time and effort (especially considering my second difficulty)? Or is it something to postpone? Is it better for me to focus on accessible stuff for the time being?

The second is that I have no experience of how to go about it, and my linguistic style is far from what is required. Putting the ideas into that format would be tough, very tough, for me. Stand me in front of a group of a hundred people who need to be convinced, give me a pen and a board, and I'd do it! I've had a life of talking, not writing.

 

[Dolphin]

Or is it something to postpone? Is it better for me to focus on accessible stuff for the time being?

No particular rush on it I suppose.

But basically, the world of the Lancet and the like I don't think deals with data in the format you have. And indeed for some of us who try to enter the world a bit through published pieces of one sort or another, it would be difficult to quote any data or analysis you have until it is published somewhere.

 

[WillowJ]

Keep at it, Dolphin. I've been telling Graham from the beginning that if we really want to reach GP's/PCP's, we've got to produce a version which can be submitted to a peer-reviewed journal. (preferably one which is already indexed on PubMed, no offense to IACFS/ME)

 

[Bob]

One idea to get published would be the Chalder Fatigue Scale data. That is a scale that is used in a lot of studies. Then in the introduction and discussion sections, would could potentially discuss how this might relate to the PACE Trial.
Here is an example of such a study:

Fatigue in Myalgic Encephalomyelitis
http://www.iacfs.net/BULLETINFALL2008/Fall08GoudsmitFatigueinMyalgicEnceph/tabid/292/Default.aspx

Even if you didn't manage to make all your points in any one paper, it could be quoted in future studies/papers/letters.

It's a great idea to aim to get it published, but would Graham have to think about ethical considerations, and designing consent forms?

 

[Dolphin]

The second is that I have no experience of how to go about it, and my linguistic style is far from what is required. Putting the ideas into that format would be tough, very tough, for me. Stand me in front of a group of a hundred people who need to be convinced, give me a pen and a board, and I'd do it! I've had a life of talking, not writing.

Ok. I don't want to put any unnecessary pressure on you. But I have read quite a lot of papers and sometimes, it is obvious that English isn't the writer's/writers' first language.

Of course, probably for many high-impact journals e.g. the Lancet or BMJ, the English can be quite good. But they aren't the only journals out there. Also, there can be people out there who can help with writing once drafts are done: different people have different skills and knowledge.

Not giving any guarantees I'll volunteer myself I'm afraid, but if a draft came my way, I might help in some way (maybe more with references, although I can sometimes coming up with the odd synonym and the like to make something look a bit fancier. And I might know others who might help too. And WillowJ is good with words.

 

[Dolphin]

It's a great idea to aim to get it published, but would Graham have to think about ethical considerations, and designing consent forms?

Afraid I'm not an expert on that end of things. One could contact other people who have published questionnaire data and see what they did.

 

[Graham]

Hmm, consent forms. I can't even get in touch with all the people who took part in the study. Two of them have vanished off the radar (i.e. left PR or changed their email addresses). But would the initial announcement be sufficient? (The statement that I would use their responses totally anonymously in our study.)

As for the offer of help, Dolphin, and the suggestion of Willow J wielding a whip over me while I sort out my grammar – I would have had you both earmarked for that anyway!

I have just finished the first of a possible short series of YouTube videos enticing people to our study, and at the moment am phrasing a letter to MPs in the light of the fact that the consultation period "Liberating the NHS: Greater Choice and Control" runs out on 20th July, and I want to draw their attention to the fact that patient power at grassroots level is no good if there is no coherent structure that enables us to ensure that all the GP consortia are following proper guidelines and actually providing support (cue the AfME report). What with that and starting up a Facebook page, I'm really pushing the boundaries of what a tired old ex-maths teacher is able to get his brain around.

The YouTube video combined a standard video, a couple of graphics and the race animation, taken at different sizes and frame rates. The software I used, Blender, was free, very powerful, and has hideously poor documentation. My brain hurts!