BMJ comments on new PACE trial data analysis

[worldbackwards]

Dr Chalder has written:

'Whilst some correspondents have complained that the primary outcomes of the trial were self-rated, we argue that these are the most appropriate measures to judge improvement in an illness that is currently defined by symptoms.'

Ultimately, what this means is: we think the symptoms are subjective, so what does it matter if the assessment of the treatment is. Which rather underlines everything that is wrong with PACE.

 

[Snow Leopard]

Dr Chalder has written:

'Whilst some correspondents have complained that the primary outcomes of the trial were self-rated, we argue that these are the most appropriate measures to judge improvement in an illness that is currently defined by symptoms.'

Sadly, in response to criticism, they have been pushing this line for years.

Objective impacts on functioning are easy to measure, but they are not interested. When those objective impacts have been measured, they have shown no improvement. Hence it challenges their worldview to use such measures.

Yet the Norwegian Rituximab trial investigators have chosen to use objective measures in addition to subjective measures, making the claim by Chalder look even more silly.

 

[Sean]

Beware their slippery language. For a long time, they haven't said that illness-beliefs cause ME/CFS. They say that illness-beliefs perpetuate the illness. (They say the cause doesn't matter. It's the perpetuating illness-beliefs that matter.)

If they were being consistent with the disease model they are advocating, their answer would be something like:

We do not claim that false illness beliefs are a predisposing or precipitating factor, only that they are the primary perpetuating factor.

Otherwise, they are just playing slippery games with words.

 

[WillowJ]

A good question. In rheumatology whenever something is reported everyone puts in a grant to do the same thing so they can get a paper too (they find some slightly different angle to make it publishable). I am not sure what has been happening in the ME field.

I think to a large degree, this has not happened.

I think possibly part of the the underlying reason is that there is no medical specialty. So there is no one to, say, lend respectability to conferences of specialists (or board certify them in, idk, physiopathy --global systems failure of the human body? ). And in the case of ME, for instance, we have in the US no Institute or broad category of spending to appeal to, if the individual line item is too low (in the US that's how it works: there is a large budget for, say, autoimmune disease ($838 M est 20145, and for arthritis (237 M), and for neurodegenerative ($1662 M; MS and epilepsy can also apply), and also individual budgets for specific common diseases like Lupus ($95 M) and MS ($115 M)... then for ME/CFS, $5 M. This is NIH spending.

But in the US, ME belongs to the Trans-NIH ME/CFS Research Working Group in the Office of Research of Women's Health, which seems a pretty unique assignment, and this office has no research budget at all. It would be standard for a disease to be housed in an Institute. They could apply to the various large disease budgets, but as none of them know what ME is, and the general perception is not favorable, I wouldn't guess the success rate would be helpful.

Word is that people who apply for grants to study ME/CFS, and even sometimes people who actually get grants and then try to get published, are blocked most often on the grounds that "CFS is psychological so medical research is not needed" and sometimes "this journal doesn't take any CFS papers". Dr. Lipkin mentioned this in a recent year, and it was brought up at the San Francisco conferences, but I couldn't tell via Twitter how many people discussed it. BMJ and the Lancet have taken no medical papers since a pro-medical editorial in the Lancet in 1996 (if I have my dates straight). JAMA pediatrics took one, but I understand typically JAMA won't take any ME/CFS papers as a matter of editorial policy (I guess they see it as similar as not taking any on little green men!).

With no medical papers in top journals, no professional association, and viturally no funding available (particularly for medical studies), some researchers report pressure from their universities not to become involved in ME/CFS research, to water down the criteria, not attend ME events, etc.

Dr. Mary Ann Fletcher and Dr. Nancy Klimas recently moved universities and fortunately in the post-XMRV era they were able to find a supportive one (they noted what a welcome contrast this was to their previous university). Dr. Sonia Marshall-Grandsnik's team also moved (although I do not recall if they cited a cause). Dr. Chris Snell's team had to take their ME/CFS research away from his university but they weren't able to find a new university home--they are currently a nonprofit org. Someone else was drummed out of his university and eventually turned up in a different country altogether. And that list is confined only to people currently involved in research.

This problem with interest and funding* began to change only with the entry of XMRV in 2009 or 2010 (even though it was later withdrawn). Some people began to get the idea that it might not be a junk diagnosis after all, and some key people figured out the disease was serious. The Stanford conference was a major indication that things are changing, with important people from the university there (not only the normal ME people). The involvement of Stephen Holgate and newer big names is also impressive. But of course this is all very, very new.

*(private funding has changed, and the ratio of US funds spent on psychological vs physiological research has changed, looking at 2013 studies)

In this Wonderland context, it seems most people (from earlier days) left the field and I know of one who kept treating patients but firewalled himself and quit publishing. I'm sure there are some negative unpublished studies as well, but I expect that's a much less common explanation than it would be in a normal context like rheumatoid arthritis.

 

[Sasha]

That's like either (1) randomly giving the placebo to someone in the treatment group of a drug trial and telling them that you sometimes give them the placebo instead of the treatment, or alternatively (2) saying that it 's ok not to take the pills if they don't want to but they don't have to withdraw from the trial. With the added twist that in the context of PACE, the placebo / control was arguably the better 'treatment' anyway (depending on how much credence you're prepared to give to the results of an unblinded trial with subjective measures).

Does the fact that I'm shocked mean that I am hopelessly naïve? Whatever the shortcomings of PACE, I actually do like to believe that the investigators and therapists involved acted in good faith.

I think it depends what you mean by 'good faith'. Practitioners sometimes get roped into trials by their superiors but don't themselves believe in the therapy. In such a case, the practitioner is faced with the choice of putting the patient in front of them first, or the patients downstream who will be treated off the back of the trial results (and mistreated, if the results are misleading because of protocol breaches).

So maybe the therapist in this case was trying to do right by the patient in front of them and protect them from GET harming them.

Or maybe they just didn't understand the therapy...

 

[Jonathan Edwards]

Ultimately, what this means is: we think the symptoms are subjective, so what does it matter if the assessment of the treatment is. Which rather underlines everything that is wrong with PACE.

Sadly, in response to criticism, they have been pushing this line for years.

Objective impacts on functioning are easy to measure, but they are not interested. When those objective impacts have been measured, they have shown no improvement. Hence it challenges their worldview to use such measures.

Yet the Norwegian Rituximab trial investigators have chosen to use objective measures in addition to subjective measures, making the claim by Chalder look even more silly.

I actually disagree, respectfully, and only to try to sharpen the case that needs to be made, with both these analyses and think Dr Chalder would be correct to challenge them.

All symptoms are subjective. Even if you have a broken leg. Fluge and Mella have used secondary physical measures but these will have no statistical validity. They have chosen a symptom score as the primary trial endpoint and I think this is right. What matters for the ME patient is how awful they feel. Walking time might reflect that, but it might not. Going to work might reflect that and it might not. Moreover, all these so-called objective measures like actometry and walking time and employment are, in the terms of the placebo effect, subjective measures. They can all very credibly be affected by beliefs. So actually they are no better and they are less to the point. If there were objective measures like a CRP that would be different. But even measures of muscle physiology are open to interpretation as subjective because they might be due to de-conditioning. It is no good falling into the same trap as the PACE authors of discounting the power of placebo in all its indirect and subtle guises.

So for me the important point is that a subjective endpoint is the right endpoint if you can design a blinded study. If you cannot do a blinded study you really need something better than even actometry to even think it is worth doing a trial. I am not sure we have that. So it is not that PACE should have used other endpoints. It was never going to be valid anyway. We need different methodologies for answering questions about unblindable therapies. Not just endpoints.

 

[A.B.]

We need different methodologies for answering questions about unblindable therapies.

Is it even possible implement blinding in psychotherapy research? A placebo treatment must be indistinguishable from the real thing for both patient and therapist. In the context of psychotherapy this sounds like an impossible goal. I think it's also apparent that lowering the standards of research isn't the answer. The logical conclusion is that psychotherapy research of this type is a waste of time and resources.

Are there any alternatives to blinded studies?

 

[worldbackwards]

What matters for the ME patient is how awful they feel. Walking time might reflect that, but it might not. Going to work might reflect that and it might not. Moreover, all these so-called objective measures like actometry and walking time and employment are, in the terms of the placebo effect, subjective measures. They can all very credibly be affected by beliefs. So actually they are no better and they are less to the point.

Certainly a valid point, but I think that actions, independently assessed, at least represent as good a chance, if not better, of getting to the bottom of improvement than symptoms vaguely remembered (amusing myself here by arguing with a good researcher!). A fitness test is closer to the mark than a subjective assessment in my opinion. Whether someone attends work is, I agree, not very relevant if the idea has been pushed on them that this is what they should be doing. But it's interesting as to whether they can maintain that activity or not. The fact that more were on benefits at the end of the trial shouldn't just be dismissed as irrelevant.

Probably, in lieu of your abstract 'perfect trial' (which I agree probably can't be done), it would be best to take as many measures as possible to paint a picture of changes in quality of life, for good or ill. What is particularly notable about the measures used for the headlines in PACE is that the things that made the treatments look good are frequently hyped and the ones that didn't are usually dismissed, which I think is both sour grapes and a sign of bad faith on the assessors parts.

 

[Jonathan Edwards]

Is it even possible to do implement blinding in psychotherapy research? A placebo treatment must be indistinguishable from the real thing for both patient and therapist. In the context of psychotherapy this sounds like an impossible goal. I think it's also apparent that lowering the standards of research isn't the answer. The logical conclusion is that psychotherapy research of this type is a waste of time and resources.

Are there any alternatives to blinded studies?

I am not at all sure that there are bulletproof alternatives but I think there are lots of ways to mitigate the placebo effect and the 'please-the-therapist' effect. We mentioned these on another thread. One thing is to remove all therapists from the conducting of the trial - administration of treatment and assessment. But I think there other things one can do with using analysis of response kinetics (it there is a response). Another thing I am keen on which I used in a trial in the 1980s was to personalised the endpoint for each case, and perhaps also personalise the treatment according to some set formula relating to the type and severity of problems for that individual. As long as one has a clearly defined formula that can then be applied in guidelines if a trial is successful, even if it does not prescribe exactly the same thing for every case.

Perhaps the key thing is to be found in somebody's signature here - I forget who. Science needs to set out to discover what is the right answer, not to produce the answer you want. So studies should not be designed to corroborate theories in this sort of situation so much as just to find out what works best.

 

[Esther12]

What matters for the ME patient is how awful they feel.

While I agree that this is complicated and there are no ideal outcome measures, for me personally, I think that we should not emphasise how people feel over what they can do. I've been able to improve how I feel, but not the amount I can do, and these ongoing limitations are a key part of my health problems. I'd rather be able to do the amount of stuff I was prior to falling ill, without having an improvement in how I feel, than feel as well as I did before falling ill without being able to do any more than I do now. When how I feel and activity levels are so tied up this is a bit difficult to talk about.

I'd be more enthusiastic about results from a double-blind trial that found a treatment leading to a substantial improvement in the amount of activity patient's performed than a substantial improvement in their fatigue scores. Also, while there is a danger that a nonblinded trial could lead to patients improving their activity levels just to please a therapist/fit in with expectations, I suspect that this is less of a danger than having them just fill in a questionnaire more positively, especially if their activity levels are measured for any sustained period of time.

 

[Jonathan Edwards]

While I agree that this is complicated and there are no ideal outcome measures, for me personally, I think that we should not emphasise how people feel over what they can do. I've been able to improve how I feel, but not the amount I can do, and these ongoing limitations are a key part of my health problems. I'd rather be able to do the amount of stuff I was prior to falling ill, without having an improvement in how I feel, than feel as well as I did before falling ill without being able to do any more than I do now. When how I feel and activity levels are so tied up this is a bit difficult to talk about.

I'd be more enthusiastic about results from a double-blind trial that found a treatment leading to a substantial improvement in the amount of activity patient's performed than a substantial improvement in their fatigue scores. Also, while there is a danger that a nonblinded trial could lead to patients improving their activity levels just to please a therapist/fit in with expectations, I suspect that this is less of a danger than having them just fill in a questionnaire more positively, especially if their activity levels are measured for any sustained period of time.

That is fairly persuasive, I think.

My reservation about what PWME can do would be that it would be a very individual issue. One person might be able to work at their old job even when as bad as they get whereas another might not be able to get back to work without more or less full recovery. So maybe this is where individualised targets come in. The great thing about individualised targets is that you can vary them as much as you like as long as you stipulate exactly what they are before you randomise a patient to a treatment. We did this for lupus and it worked well.
So you can say that grade 1 for Jack is getting back to being a postman. Grade 2 is getting back to a desk job in the sorting office, grade 3 is no change, grade 4 is ending up bedbound - or whatever - whereas for Jill, grade 1 is continuing to work from home as a travel agent, plus picking up the kids from school, grade 2 is just managing to continue the job, grade 3 is no change ... - or whatever. The beauty of the technique is that all the personalised details drop out of the final analysis which indicates whether or not there is optimisation, improvement, stasis or deterioration in whatever way the person most wishes to be better.

 

[user9876]

Dr Chalder has written:

'Whilst some correspondents have complained that the primary outcomes of the trial were self-rated, we argue that these are the most appropriate measures to judge improvement in an illness that is currently defined by symptoms.'

I am not sure which correspondents she is referring to but if this is a response to my post it is non-sequitur. I suspect that most of us would agree that a subjective self-rated measure of symptoms should be the primary endpoint in a therapeutic trial in ME. It is what Fluge and Mella used. The problem comes if the treatment is unblindable, because a subjective end point is then valueless. It is likely that there is currently no recognised methodology for getting around the problem of assessing unblindable therapies like CBT and GET in ME. However this does not validate doing an unblinded trial and saying 'this is the best we can do'. If it is no good it is no good.

Its worse than just having a non-blinded trial. One of the things that both CBT and GET aim to do is to get patients to reinterpret how they see their symptoms. So having an end point based on asking about symptom levels makes no sense.

There was some work on the placebo effect on asthma where they found subjective reports being good for a drug treatment vs a sham acupuncture. The subjective results were quite similar for both but the objective breathing tests showed improvements for the drug treatment. Hence I would be concerned about just relying on subjective end points although it is hard to see how to create more objective end points. I guess the trial wasn't blinded.

I would have thought it also depends on the size of the effect. I have a vague memory of the Ampligen trial that had unimpressive differences with subjective tests but better walking test results for the drug arm.

 

[Sasha]

Is Trudie Chalder's response the final say or are further responses allowed (that is, can people still write in to counter her arguments about subjective outcome measures)?

 

[user9876]

Sadly, in response to criticism, they have been pushing this line for years.

Objective impacts on functioning are easy to measure, but they are not interested. When those objective impacts have been measured, they have shown no improvement. Hence it challenges their worldview to use such measures.

Yet the Norwegian Rituximab trial investigators have chosen to use objective measures in addition to subjective measures, making the claim by Chalder look even more silly.

I think that secondary objective measures are good but it would be good to know how they correlate to see if they make sense. PACE view the correlations between the subjective and objective measures for those they claim are recovered as a national secret and view even asking about it as harassment.

 

[Aurator]

Can anyone give me some examples of the kind of specialist medical care ME/CFS patients in the UK are supposed to be routinely offered? When Chalder et al. say that "adding cognitive behaviour therapy (CBT) and graded exercise therapy (GET ) to specialist medical care (SMC) both had greater success in reducing fatigue and physical disability than adding adaptive pacing therapy (APT) to specialist medical care (SMC) or SMC alone", what SMC do they have in mind?
I've never been offered any medical care, specialist or otherwise, for my ME/CFS apart from GET/CBT.

 

[Esther12]

The great thing about individualised targets is that you can vary them as much as you like as long as you stipulate exactly what they are before you randomise a patient to a treatment.

That's interesting. I'd never thought of something like that. Good old randomisation.

 

[user9876]

I think it depends what you mean by 'good faith'. Practitioners sometimes get roped into trials by their superiors but don't themselves believe in the therapy. In such a case, the practitioner is faced with the choice of putting the patient in front of them first, or the patients downstream who will be treated off the back of the trial results (and mistreated, if the results are misleading because of protocol breaches).

So maybe the therapist in this case was trying to do right by the patient in front of them and protect them from GET harming them.

Or maybe they just didn't understand the therapy...

They did have monitoring of the therapists and claimed a good measure of conformity to their manuals. However, they did or have not reported on how well patients complied with treatment programs

 

[Jonathan Edwards]

Can anyone give me some examples of the kind of specialist medical care ME/CFS patients in the UK are supposed to be routinely offered? When Chalder et al. say that "adding cognitive behaviour therapy (CBT) and graded exercise therapy (GET ) to specialist medical care (SMC) both had greater success in reducing fatigue and physical disability than adding adaptive pacing therapy (APT) to specialist medical care (SMC) or SMC alone", what SMC do they have in mind?
I've never been offered any medical care, specialist or otherwise, for my ME/CFS apart from GET/CBT.

As far as I know the usual specialist medical care being offered, before and after PACE, has been CBT and referral to a physio for some good exercises, or maybe not. So in the PACE trial maybe SMC was seeing a doctor who says "Well you have chosen the standard care - which is actually no care at all (thinks: LOL!) other than the treatments you are not supposed to have because they are in the other arms, so are you feeling better? Hopefully, I can put you down as 'not much'.

 

[A.B.]

I too would like to know what "specialist medical care" in the PACE trial meant. Probably little more than nothing.

 

[Sean]

The issue for me is not so much that the more objective measures are better in some sense, but that (until recently at least) they are hardly being used at all, with subjective measures being almost the only source of primary outcome data in this field.

The real value of using objective measures is when you compare them with subjective measures. If both types of measure agree, then you probably have something. When they conflict, there is a problem, and they conflict big time in this field.

Which, if I put on my cynical hat, is why the psychosocial school of ME/CFS are so reluctant to use objective measures and give them their due place alongside subjective. They know that objective measures do not confirm the results they are touting from subjective measures.

I want both types of measure to be used, and used properly. Subjective are needed for the full picture, but objective are needed to keep the subjective honest. If patients are being persuaded to self-report improved physical function, but without any objective support for it, there is a serious problem, which is the current situation in the ME/CFS circus.

And that is before we factor in how modest the subjective improvements are on their own.