NICE guideline for ME/CFS is unethical – Dr Diane O’Leary, Kennedy Institute of Ethics | 23 August 2

[A.B.]

To clarify: I don't actually know if PACE factored these dropouts into the calculations correctly or not, but I'd be interested to here the opinions of people better qualified than I am. And does the reanalysis fully account for it?

There is no way to know what the data would look like without the drop outs. To fill in the missing data, the most recent available results for that person can be carried forward, or the average of the available results of that group can be used. There are some other strategies. In the case of PACE the dropouts are probably related to some degree to worsening of health and therefore the average walking distance would be overestimated. I don't remember what PACE did but think it was one of these two strategies, and I'm not a statistician.

If it comes up in a debate, one can point out that the results are probably an overestimate and consistent with poor health in any case.

 

[Valentijn]

Investigators may have been "less than encouraging" to people they thought might produce poor results. Not an option I would once have even thought of, but now for PACE seems a worryingly non-trivial possibility.

There doesn't seem to be much difference between the walking test dropouts in different trial arms. APT and GET dropouts had slightly better average doctor and patient CGI scores than CBT dropouts, but SMC dropouts even worse. CBT dropouts had bigger average improvements in fatigue than the other three groups. GET dropouts reported bigger increases in physical functioning than APT, while CBT and SMC dropouts reported an average decrease in physical function.

So there doesn't seem to be any obvious pattern of self-rated or doctor-rated poorer-performing CBT or GET patients dropping out, or for better-performing SMC or APT patients dropping out of the 6MWT. The only potentially feasible selection would have been in favor of CBT at the expense of GET, if the SF36-PF were assumed to be the best predictor of performance. Though that would require an intent to downgrade GET to not outshine CBT, which seems unlikely when PACE's GET incorporated CBT to a considerable extent, and since it would have likely made CBT look slightly worse when compared to SMC.

 

[Snow Leopard]

To clarify: I don't actually know if PACE factored these dropouts into the calculations correctly or not, but I'd be interested to here the opinions of people better qualified than I am. And does the reanalysis fully account for it?

Who cares? The PACE results show that the effect was very poor and patients were still severely ill. A person of average fitness in their mid 30s easily can walk around 700+m in 6 minutes. I am in my mid 30s, I have been ill my whole adult life and I can walk (pushing as hard as possible) 800+m in 6 minutes (albeit with severe symptoms for a week or so afterwards).

 

[Jonathan Edwards]

There is another issue here I wonder about. Across all arms of the PACE trial there were a lot of dropouts from the 6mwt, and we have no available data clarifying why. Various possibilities come to mind:-

• People may have felt unable to do the final 6mwt.
• Investigators may have been "less than encouraging" to people they thought might produce poor results. Not an option I would once have even thought of, but now for PACE seems a worryingly non-trivial possibility.
• People may have felt too demotivated to do it.
• Other things I haven't thought of?

All of these will almost certainly have applied. Murphy's law applies for bias - if it can happen a particular way it will.

The key point is that there is a significant likelihood that the people who got worse would be more likely to drop out in the GET group because it would be a 'failure'. For SMC it would just be a slow walk.

 

[Jonathan Edwards]

Even (especially!) if they don't know they're incompetent, they still get punished with removal of their license to practice, or other restrictions and requirements.

I think you are missing the point Valentijn. In relevant comparable situations there would be no penalty. If a medical student causes harm while doing a procedure because they are unaware of dangers they are not punished. They are more likely to be counselled and given closer attention. The same for junior doctors who actually have licenses. For the people deciding on guidelines at NICE there are no licenses to take away. These people have to be at one remove from those who actually produce the treatments so they are not expected to have all the relevant expertise. They have to rely on 'experts' and the definition of an expert is someone who is said to be an expert. If they turn out to be incompetent then nobody necessarily is to know that.

I am not wanting to defend anyone but I think it is important that NICE is not deluged with complaints based on spurious arguments. I have a strong suspicion that the main reason why the media have not been more helpful is because they perceive that ME/CFS advocates are all singing from different hymn sheets so that they cannot hear a clear tune.

 

[Invisible Woman]

They have to rely on 'experts' and the definition of an expert is someone who is said to be an expert. If they turn out to be incompetent then nobody necessarily is to know that.

I find this .... bizarre. In my (much shorter than planned) career I was used to people relying on experts - I was one of those experts.

I worked for an employer who made sure I had the qualifications & expertise to do my job. There were processes in place to constantly monitor how things were going and if there I were problems we'd review the situation and resolve it. There would be regular oversight

If a client's representative chose an inappropriate expert / solution that would have fairly serious career limitations for them. If I thought a client was going down the wrong path I'd advise them & my boss and we would arrange a meeting with the appropriate expert to offer best advice.

I'm not really sure what the point of NICE is if they don't take full responsibility for the decisions they make.

 

[Barry53]

I find this .... bizarre. In my (much shorter than planned) career I was used to people relying on experts - I was one of those experts.

I worked for an employer who made sure I had the qualifications & expertise to do my job. There were processes in place to constantly monitor how things were going and if there I were problems we'd review the situation and resolve it. There would be regular oversight

If a client's representative chose an inappropriate expert / solution that would have fairly serious career limitations for them. If I thought a client was going down the wrong path I'd advise them & my boss and we would arrange a meeting with the appropriate expert to offer best advice.

I'm not really sure what the point of NICE is if they don't take full responsibility for the decisions they make.

It's a bit weird isn't it. Normally an expert is someone who has "earned their stripes", and can be relied upon for their expertise. I think that in the case of the BPS crowd it comes back the fundamental problem that the lunatics have taken over the asylum and convinced most people they are legitimate and sane.

I agree with @Jonathan Edwards that NICE cannot be expected to know everything and have to rely on experts. But I do also think there is a due diligence responsibility on NICE, to try their best to ensure the experts they are relying on are in fact reliable. Give the current lay of the land I think their due diligence should be flagging up some major red flags.

I'm no legal expert, but I think in UK law the notion of due diligence features strongly in many areas.

 

[Jonathan Edwards]

If a client's representative chose an inappropriate expert / solution that would have fairly serious career limitations for them.

The problem is that in science and medicine it is not necessarily clear who an 'inappropriate expert' is. In the case of PACE we have the man who wrote the book on how to do psychiatry trials saying the trial is a thing of beauty. He is supposed to be the expert. And it seems nobody in the field is interested in challenging that. As someone outside the field I have said that I think the trial is no use for a very basic reason - that it is unblinded and has subjective endpoints. However, almost everybody, including members on PR and reviewers of my manuscripts about this have said that this is not an important criticism - at least until some of them have come around to agreeing. So am I the expert who picks out the inappropriate expert or have I got it wrong too. There is nobody who is endowed with some special power to adjudicate. Dr O'Leary quotes the US Institute of Medicine but the same institute was of the opposite opinion until recently and who knows if they are really experts?

It is easy enough to have appropriate experts if one is dealing with facts but whether or not PACE provides evidence for the value of CBT or GET is not a fact. It is something on which one has an opinion. I see the situation as pretty black and white but, as I say, remarkably few people have agreed with my main reason for thinking that.

 

[Jonathan Edwards]

I But I do also think there is a due diligence responsibility on NICE, to try their best to ensure the experts they are relying on are in fact reliable. Give the current lay of the land I think their due diligence should be flagging up some major red flags.

But if they do what some of the media people have done and canvassed opinion from statisticians and academics close to the field they will have been told that PACE is not perfect but not too bad. I think in English law a court is obliged to go with the consensus of such experts. Unless of course on cross examination those experts can be shown to not know what they are talking about. But that requires someone to know enough to cross examine in a way that would reveal that.

The most obvious port of call for NICE is Cochrane. Up until now not a pink streak of a flag has been raised. The reviewers have swallowed the story. But red flags have been waved at Cochrane so we will have to see what happens at the next round. (Note that a new Cochrane review on exercise therapy is currently on the stocks, written by the PACE authors together with staunch supporters.)

 

[Cheshire]

As someone outside the field I have said that I think the trial is no use for a very basic reason - that it is unblinded and has subjective endpoints. However, almost everybody, including members on PR and reviewers of my manuscripts about this have said that this is not an important criticism - at least until some of them have come around to agreeing.

I strongly disagree with you on this. We may not have understood the importance of your point immediatly, but I have seen this argument used more and more these last months (in comments and on PR) to the point of being central in some comments. I now think this is the major point that makes PACE invalid, though that was not the case at first.

 

[A.B.]

I now think this is the major point that makes PACE invalid

It makes most of the "psychotherapy for physical symptoms" clinical trials literature invalid and claims of existence of psychosomatic symptoms seems to rely on results of these trials as well. This branch of science is on the brink of ruin and doesn't know it yet.

 

[Daisymay]

As someone outside the field I have said that I think the trial is no use for a very basic reason - that it is unblinded and has subjective endpoints.

I agree, and is part of the problem not that a lot of psych research is similarly flawed, but they dare n't acknowledge this is the case because to do so would mean accepting that a great deal of psych research is invalid.

 

[Large Donner]

I am not wanting to defend anyone but I think it is important that NICE is not deluged with complaints based on spurious arguments. I have a strong suspicion that the main reason why the media have not been more helpful is because they perceive that ME/CFS advocates are all singing from different hymn sheets so that they cannot hear a clear tune.

I think the main reason the media is not more helpful is that they are singing hymns written by the SMC otherwise the features on ME would be more balanced in terms of total articles written and sides of the argument reported. They would have also seen PACE for what it is by now if it wasn't for the SMC writing the articles for them.

From that perspective its got nothing to do with ME advocates really. Its just bloody lazy spoon fed, sit on yer arse copy and paste "journalism" and no doubts, favours for favours.

The reason the media is not more useful in EVERYTHING is because they dont do journalism. And one of the main reasons for that is that editors and media tycoons mix with politicians and people who can influence and they both use each other.

 

[Invisible Woman]

The problem is that in science and medicine it is not necessarily clear who an 'inappropriate expert' is.

I find this whole thing crazy - this is directed at the situation rather than your good self @ Jonathon Edwards.

Maybe I am oversimplifying matters but in my field, if it was unclear who the expert should be, we'd get all parties in a room and we'd discuss it. We would deliberately include people with opposing a views ..and yes, the discussion could become "robust".

The thing is our experts would not be nameless. All those involved would be accountable and there would be a deliberate paper trail.

NICE just seem to want it everyway. No names, no accountability. The experts can't be held accountable or have their credentials checked as they can remain nameless and the decision maker can't be held accountable as they aren't an expert.

Clear as mud.

 

[Valentijn]

There is another issue here I wonder about. Across all arms of the PACE trial there were a lot of dropouts from the 6mwt, and we have no available data clarifying why.

Something which is also interesting is that of the 178 patients who didn't take the walking test, 33 also didn't provide any other endpoint data. Of those, only 10 were excluded from analysis, presumably because they also lacked tho doctor-rated CGI scores. So that's 23 patients included in their analysis who gave no data at the end of the study, but were somehow included in assessment anyhow despite missing fatigue and SF36-PF scores.

I looked at changes in SF36-PF scores compared between walkers and non-walkers (of those who had scores at the end) in each arm, and there are some interesting things there. There was little difference in improvement between APT walkers and non-walkers, while CBT walkers scored 10 points higher on the SF36-PF than non-walkers.

If failing to walk was pretty random, we'd expect to see similar scores for walkers and non-walkers in each arm. If failing to walk was due to worse health, we'd expect to see lower scores across the board for non-walkers. So that does suggest that there was another influence, which might not necessarily be as heavy-handed as therapists having a private word with those doing worse in the CBT group and suggesting they sit it out, but might simply be the result of a bias inherent in some therapies but not others.

Similarly, those in the GET arm who performed worse in the baseline 6MWT were more likely to dropout from the 52 week 6MWT. It's a bit interesting that the objective walking test seems to correlate with dropping out for GET patients, while the more subjective SF36-PF correlated with dropping out for CBT patients.

Note that these aren't proper statistical comparisons, and might just be due to completely random variations, math errors, or other processing errors.

 

[Esther12]

But if they do what some of the media people have done and canvassed opinion from statisticians and academics close to the field they will have been told that PACE is not perfect but not too bad. I think in English law a court is obliged to go with the consensus of such experts. Unless of course on cross examination those experts can be shown to not know what they are talking about. But that requires someone to know enough to cross examine in a way that would reveal that.

The most obvious port of call for NICE is Cochrane. Up until now not a pink streak of a flag has been raised. The reviewers have swallowed the story. But red flags have been waved at Cochrane so we will have to see what happens at the next round. (Note that a new Cochrane review on exercise therapy is currently on the stocks, written by the PACE authors together with staunch supporters.)

We need some cross examination.

Part of the trouble is that Wessely and colleagues have helped set the standards for this sort of research. And they've managed to set them very low. Lots of people in medical research seem to want to just fit in with the consensus rather than take responsibility for making their own moral judgements about what claims would be misleading to patients.

The Larun responses to Courtney and Kindlon on the last exercise review don't inspire much confidence in Cochrane.

 

[Jonathan Edwards]

I strongly disagree with you on this. We may not have understood the importance of your point immediatly, but I have seen this argument used more and more these last months (in comments and on PR) to the point of being central in some comments. I now think this is the major point that makes PACE invalid, though that was not the case at first.

No, you are not disagreeing. We agree that many people have come around to seeing it as an important point. However, it took me at least a year to get it across. And in terms of reviewers for journals I had to fight hard to resist the suggestion that I should remove the argument because it would make most psychologists' research appear a waste of time. David Tuller disagrees with me on it - amicably. I think Tom has disagreed with me on it. I am pretty sure James Coyne has disagreed with me (he may have been one of the reviewers!). Important advocates behind the scenes have disagreed with me in personal conversations.

My point is that if it takes that long for an opinion to be accepted it is clearly an opinion, not a fact. So those who seek to do due diligence and establish the consensus view may well not get to see that it might be crucial.

 

[Jonathan Edwards]

I think the main reason the media is not more helpful is that they are singing hymns written by the SMC otherwise the features on ME would be more balanced in terms of total articles written and sides of the argument reported.

I am referring to very specific individuals within the media who are better than that. People who have very nearly run stories but who on doing due diligence find they do not have back up to go ahead.

 

[Jonathan Edwards]

I find this whole thing crazy - this is directed at the situation rather than your good self @ Jonathon Edwards.

Maybe I am oversimplifying matters but in my field, if it was unclear who the expert should be, we'd get all parties in a room and we'd discuss it. We would deliberately include people with opposing a views ..and yes, the discussion could become "robust".

The thing is our experts would not be nameless. All those involved would be accountable and there would be a deliberate paper trail.

NICE just seem to want it everyway. No names, no accountability. The experts can't be held accountable or have their credentials checked as they can remain nameless and the decision maker can't be held accountable as they aren't an expert.

Clear as mud.

No names is the way science has always done it, (with the exception of the Physiological Society, where referees are members of the audience when a paper is presented and there is a vote). No names is important if you want papers reviewed by good junior scientists who will lose their job if they put a foot wrong. But it has proved a rotten system.

But NICE generally works on an open basis. The problem here is that the people who dispute the quality of evidence have no standing in the field. I have no standing in psychiatry or even in ME/CFS. There is no reason for NICE staff, who are at one remove from those in the field, to think that people like me have anything relevant to say. Life is like that in medicine because it reflects the real uncertainties that our lives often depend on, I think.

 

[Large Donner]

I am referring to very specific individuals within the media who are better than that. People who have very nearly run stories but who on doing due diligence find they do not have back up to go ahead.

Surely there are multiple reasons to highlight whats wrong with PACE not just your main critique....

As someone outside the field I have said that I think the trial is no use for a very basic reason - that it is unblinded and has subjective endpoints.

Its self evident enough that quote, that especially science journalists, should be all over it. Aside from that how about changing protocols, dropping objective measures, refusing data release, connections to the DWP and insurance industry, denial of the existence of the value of tests like the 2 day CPET test.

I really don't think that picking one reason over another why PACE provably stinks, when there's smelly fish in the BPS school, is a good enough explanation why journalists could take the stance...

the media have not been more helpful is because they perceive that ME/CFS advocates are all singing from different hymn sheets so that they cannot hear a clear tune.

.....its more likely as you also say.......

People who have very nearly run stories but who on doing due diligence find they do not have back up to go ahead.

....if you mean that those who want to run the good stories don't want to go up against the establishment.