PACE Trial and PACE Trial Protocol

[biophile]

I'm pretty sure this just means that the reviewer insisted that the 'normal range' was labelled as post-hoc, to make it clear the researchers were taking libertys. Which means that PACE had originally not owned up to this.

Interesting. Are you just offering an explanation for the peer reviewer's role but still agreeing that the authors defined "normal" after seeing the data?

I just want to be crystal clear that the way they have used "post-hoc analysis" in the PACE trial definitely means "after seeing the data" rather than "after the experiment design was settled" or whatever. And if they claim all changes were approved before seeing the data, how can they then get away with effectively replacing recovery with normal without such approval?

I must admit that after all the spin and discrepancies and obfuscation from these people I have trouble trusting them or giving them the benefit of the doubt as I used to, but I would delete my previous post if it was based on a serious misinterpretation on my behalf.

 

[oceanblue]

Interesting. Are you just offering an explanation for the peer reviewer's role but still agreeing that the authors defined "normal" after seeing the data?

Yes.

And if they claim all changes were approved before seeing the data, how can they then get away with effectively replacing recovery with normal without such approval?

As ever, they have been slippery. It was the change to the primary outcomes they had approved, and I think that's what they claimed. However, it was post-hoc secondary outcomes (% normal, % improvers) they relied on heavily in declaring the results moderately effective.

It's also worth noting that they claimed they didn't include the protocol 'recovery' definition (which I don't think was affected one way or another by the approved changes) because of lack of space - yet they found space for their dodgy post-hoc normal definition. The authors have said they will publish the protocol recovery data later (alongside new, no doubt even more creative, defintions of 'recovery').

Effectively they got away with replacing normal with recovery because they dropped recovery from the main paper (which got the main publicity) but will eventually publish it. I suspect nothing in the protocol binds them to publishing secondary outcomes like recovery in the main paper. It's sneaky and deceptive but probably doesn't break any rules.

I just want to be crystal clear that the way they have used "post-hoc analysis" in the PACE trial definitely means "after seeing the data" rather than "after the experiment design was settled" or whatever.

Your interpretation is exactly right. And in the world of proper science, post-hoc analysis is viewed with a huge degree of scepticism.

 

[Esther12]

Just a note for me to go back and re-read the posts from 1113 on. Lots of stuff I want to take the time to really understand, but I'm feeling a bit wiped right now. Thanks for the discussion people.

 

[Snow Leopard]

It's sneaky and deceptive but probably doesn't break any rules.

It's also unscientific.

 

[Dolphin]

Patient satisfaction

Earlier in the year somebody (who I don't think is on the forum) sent me and some others the full text of the following in connection with the PACE Trial:

Patient satisfaction with anaesthesia care: what is patient satisfaction, how should it be measured, and what is the evidence for assuring high patient satisfaction?

Best Pract Res Clin Anaesthesiol. 2006 Jun;20(2):331-46.

Heidegger T, Saal D, Nuebling M.

Source: Department of Anaesthesiology, Cantonal Hospital St Gallen, Rorschacherstrasse 95, 9007 St Gallen, Switzerland. thomas.heidegger@kssg.ch

Abstract

Patient satisfaction is a part of outcome quality.

Many theories of satisfaction include patients' expectation.

One definition of satisfaction is therefore the degree of congruence between expectation and accomplishment.

The involvement of patients as well as experts is therefore an important step in the development of an instrument to measure patient satisfaction.

Results of single-item ratings or overall satisfaction surveys are over-optimistic and do not represent the true indication of care.

The construction of highly standardized (psychometric) questionnaires should include elements of content validity, criterion and construct validity, reliability and practicability.

Based on the few available studies in anaesthesia, patient satisfaction is primarily determined by information and communication.

There is great potential for improvement in this area.

However, we do not know the best way to continuously improve patient satisfaction with anaesthesia care, or to what extent decisions should be shared between the anaesthetist and the patient.

PMID: 16850781
[PubMed - indexed for MEDLINE]

I put it in my "to be read" bundle but haven't been in any rush to read it. Anyway, I'm just making my way through it now; I have found it useful to have explained again the construction of questionnaires but I imagine reading the full text would be a (very) minority interest.

Anyway, this quote might be of most interest:

In the case of patient satisfaction, most studies [14,40] have shown that questions simply relating to overall satisfaction are not adequate: first, because they lead to highly skewed distributions with over-estimation of satisfaction; patients are notoriously satisfied (strong ceiling effects) when asked in a general way only, but we know that they will report deficits beyond this when we ask in a more concrete manner. And second, such overall satisfaction questions do not help in formulating improvement strategies; one only knows that 97% of the patients are satisfied and 3% are dissatisfied, but not what should be improved.

14. Fung D & Cohen MM. Measuring patient satisfaction with anaesthesia care: a review of current methodology. Anesthesia and Analgesia 1998; 87: 10891098.
40. Jenkinson C, Coulter A, Bruster S et al. Patients experiences and satisfaction with health care: results of a questionnaire study of specific aspects of care. Quality and Safety in Health Care 2002; 11: 335339.

The authors of the PACE Trial refer on a few occasions (between the paper and response to the letters) to the satisfaction ratings.

 

[Esther12]

Thanks D. I often find that reading non-CFS papers does more to improve my understanding of the issues around CFS than CFS papers do.

This is not a good sign.

So many of our concerns about the use of questionnaires, etc in CFS research are widely discussed in papers looking at other illnesses, but CFS papers tend to be written by/for morons. Also - didn't BACME promote some satisfaction figures for CFS clinics recently? Now that they're having to admit just how ineffective their treatments usually are, patient satisfaction at having a nice chat looks like their best pitch for continued funding.

 

[Graham]

I did a satisfaction survey at school many years ago, testing how useful, important and effective the maths department were, and got amazingly good results. Well, OK, I took my sample outside the maths office as the maths staff were going in for a cup of coffee, so it may not quite have been pukka. Strangely enough, when I showed the results to the class, the kids saw through it. I wonder why those psychologists don't realise that we can see through it as well?

 

[Dolphin]

Thanks D. I often find that reading non-CFS papers does more to improve my understanding of the issues around CFS than CFS papers do.

This is not a good sign.

So many of our concerns about the use of questionnaires, etc in CFS research are widely discussed in papers looking at other illnesses, but CFS papers tend to be written by/for morons. Also - didn't BACME promote some satisfaction figures for CFS clinics recently? Now that they're having to admit just how ineffective their treatments usually are, patient satisfaction at having a nice chat looks like their best pitch for continued funding.

I remember hearing a very high satisfaction rating for the service in Belfast a few years ago: the service was one (IIRC) OT offering group CBT and a psychiatrist who seemed not only to be from the Wessely School of Thought but also may have lacked something in "bedside manner"/could be rude. It had an average satisfaction rating of around 9 out of 10.

 

[Bob]

I don't know if this has been posted before, but it's a pack of lies, so I thought I'd flag it up.

Centre for Psychiatry Newsletter. Issue 2. Spring 2011.

FINDING THE RIGHT PACE for TREATING CHRONIC FATIGUE SYNDROME, by Peter White

http://issuu.com/lisakass/docs/centre_for_psychiatry_newsletter_-_issue_2

See page 4. I've quoted the offending extract, below.

What The Trial Showed.

...

CBT and GET were more effective in improving both fatigue and physical function
than either SMC alone or APT.

This was the case no matter how we defined CFS or ME or whether patients also
had a comorbid depressive illness, suffered by a third of our patients.

By a year, some six out of ten patients made a clinically useful improvement in
both fatigue and function after both CBT and GET compared to about four out of
ten for APT and 45% for SMC alone.

About three out of ten were within normal population ranges for both fatigue and
function a year after both CBT and GET; about twice the numbers than for APT and
SMC alone.

Serious adverse reactions to trial treatments were uncommon (2% or less for all
four interventions), with no differences between arms.

In fact, SMC was more effective than the improvement that is attributable to CBT or GET.
Notice how he also purposely confuses how many patients improved due to "CBT and GET", rather CBT+SMC and GET+SMC.
And then there's the 'normal population range' nonsense.


There's this info as well, which we already know about, but just thought I'd repost in case anyone missed it:

FUTURE PAPERS

We are now writing papers that include the health economic
outcomes (to test cost-effectiveness), the moderators and mediators of the
interventions (to find whom they best work for and why), and long-term follow up
to see if effects last.

 

[Esther12]

Thanks for that Bob.

In fact, SMC was more effective than the improvement that is attributable to CBT or GET.

He does say 'SMC alone'. It's not totally clear, but not a lie either. The 'within normal range' stuff has started seeming funny to me now.

 

[Bob]

He does say 'SMC alone'. It's not totally clear, but not a lie either. The 'within normal range' stuff has started seeming funny to me now.

But he says that CBT and GET were more effective than SMC alone. They weren't.
CBT and GET were less effective than SMC alone, or in other words SMC was more effective than CBT and GET.
White knows this very well. This is definitely an absolute lie, unless all three statements were mistakes. He is misrepresenting the facts to his own advantage.

CBT+SMC and GET+SMC were more effective than SMC alone.

 

[Esther12]

But he says that CBT and GET were more effective than SMC alone. They weren't. CBT and GET were less effective than SMC alone, or in other words SMC was more effective than CBT and GET.

CBT+SMC and GET+SMC were more effective than SMC alone.

I know what you mean. Elsewhere in the piece though, it is explained that everyone gets SMC. Really, I don't think that article is any more damning than any of the other things they've done, but just more of the same.

 

[Graham]

I am just putting together an argument as to how it is possible that it was quite difficult for some patients to record worsening score. It isn't quite ready yet, but basically, if you have ME and, on the Chalder scale, rate 6 things as really bad (score 3 each), two as bad (score 2 each) and the remaining 3 items (like muscle weakness) not affected by the ME (score 1 each), you would score 25. But unless the treatment affected the things that hadn't been part of the problem, you can only drop your score to 27, even if it nearly kills you. Trying to put together the data on the 26 patients in the recent Goudsmit study suggests that it didn't quiet work like that for those 26 patients - there was a cluster of 5 scoring 11 (bimodal) which transferred to 32 or 33 on the Likert scale, so they couldn't get much worse: 5 of the remaining 8 on a score of 11 transposed to 30, so they could deteriorate by 3 points. The other three on 11 converted to 27, 28 and 29. Those on lower bimodal scores of 8,9 or 10 seem to have put half in the score 3 category and half in the score 2, so they could drop by 4 or 5 points if things got bad.

As you can see, I'm still working on it, but clearly overall it is easier for scores to improve than worsen. I just need to work out how to be coherent.

 

[Esther12]

I am just putting together an argument as to how it is possible that it was quite difficult for some patients to record worsening score. It isn't quite ready yet, but basically, if you have ME and, on the Chalder scale, rate 6 things as really bad (score 3 each), two as bad (score 2 each) and the remaining 3 items (like muscle weakness) not affected by the ME (score 1 each), you would score 25. But unless the treatment affected the things that hadn't been part of the problem, you can only drop your score to 27, even if it nearly kills you. Trying to put together the data on the 26 patients in the recent Goudsmit study suggests that it didn't quiet work like that for those 26 patients - there was a cluster of 5 scoring 11 (bimodal) which transferred to 32 or 33 on the Likert scale, so they couldn't get much worse: 5 of the remaining 8 on a score of 11 transposed to 30, so they could deteriorate by 3 points. The other three on 11 converted to 27, 28 and 29. Those on lower bimodal scores of 8,9 or 10 seem to have put half in the score 3 category and half in the score 2, so they could drop by 4 or 5 points if things got bad.

As you can see, I'm still working on it, but clearly overall it is easier for scores to improve than worsen. I just need to work out how to be coherent.

Dolphin (and others?) wrote some posts on this. In this thread. Somewhere.

It will take you much longer to look for them than work it all out again though.

 

[Graham]

Yes, sorry, I'm not claiming that I came up with the idea - virtually everything I am working on belongs to you guys. It's just that there is so much stuff, I lose track of where it is. The Goudsmit stuff is new though, I think, and it lists the bimodal scores and the Likert scores, which is useful. I'm trying to get a diagram or graphic to display it, to see if it can be easier to understand. What I still can't get my head around is how anyone can score 0 points on any question - how can you end up feeling better than when you were well? Unless of course the therapist has been putting 120% effort into your rehabilitation.

 

[Dolphin]

I am just putting together an argument as to how it is possible that it was quite difficult for some patients to record worsening score. It isn't quite ready yet, but basically, if you have ME and, on the Chalder scale, rate 6 things as really bad (score 3 each), two as bad (score 2 each) and the remaining 3 items (like muscle weakness) not affected by the ME (score 1 each), you would score 25. But unless the treatment affected the things that hadn't been part of the problem, you can only drop your score to 27, even if it nearly kills you. Trying to put together the data on the 26 patients in the recent Goudsmit study suggests that it didn't quiet work like that for those 26 patients - there was a cluster of 5 scoring 11 (bimodal) which transferred to 32 or 33 on the Likert scale, so they couldn't get much worse: 5 of the remaining 8 on a score of 11 transposed to 30, so they could deteriorate by 3 points. The other three on 11 converted to 27, 28 and 29. Those on lower bimodal scores of 8,9 or 10 seem to have put half in the score 3 category and half in the score 2, so they could drop by 4 or 5 points if things got bad.

As you can see, I'm still working on it, but clearly overall it is easier for scores to improve than worsen. I just need to work out how to be coherent.

Great you're doing this, Graham.

Basically what you are talking about is ceiling effects of individual items I think (rather than total scores). Stouten discusses this in: http://www.biomedcentral.com/1472-6963/5/37 . As I have mentioned somewhere before, one doesn't need to read through his maths to get the points he makes. One point I have reminded him of is that the figures he has are minimums (minima?) for the percentage of items with maximum scoring and it would be good to get the actual percentages whenever possible.

Not sure that "muscle weakness" is the best example to make the point you are make (threw me a bit as some would say it's a core/near core symptom of ME, although is perhaps not always obvious) except if you're saying that whole new symptoms have to appear for them to show up rather than worsening of existing symptoms.

The more I think about it, the more I think the Chalder fatigue scale (Likert scoring) is weird. As I mentioned before, population studies generally score around 11 (mainly 10-12) but as somebody (Snow Leopard?) pointed out, in MS trials mean scores were around 8! Perhaps one can say that in trials, esp. of psychological therapies which might influence thinking, it has weird properties/is not suitable. A lot of trials have used bimodal scoring which at least merges 0 and 1 (on the 0-3 scale) but it has it's own problems.

The thing about people scoring 0's of course is that it brings down the whole score - a jump from 3 to 0 is 3 points (a lot) and even 2 to 0 on one item would be enough to be classed as an improver using their criterion of an improvement of 2 points. I'd feel much happier if scores of 0 were "banned".

 

[Graham]

Thanks for that reference (I think, but my brain is flagging, so I'll check the maths tomorrow!): it looks pretty useful, but I am going to need to make it a lot easier to understand.

Your comment on muscle weakness puzzled me. When it comes to opening bottle tops, lifting a heavy bag, wringing out a cloth, I have the same strength as I had before ME - I just don't have the stamina. When it comes to leg muscles, then obviously as I don't cycle any more, my legs have lost some of their strength, but this isn't a factor that I can attribute directly to ME. One of my first obvious symptoms with ME was weird sensations in my legs, so when I first went down with ME, my GP did that test where they run a stick along the sole of your foot. I nearly kicked him across the room - no loss of strength there! So I interpret the question to mean that my actual strength hasn't diminished but my energy has. The trouble is that "do you feel weak?" to me is the same question. My answer would be no - only when I am tired or have run out of energy.

As I will need to give an example, perhaps I should change this one - but am I unusual in the ME community for this? Would most people with ME "tick" all eleven boxes? Am I misunderstanding the questions?

 

[Dolphin]

Thanks for that reference (I think, but my brain is flagging, so I'll check the maths tomorrow!): it looks pretty useful, but I am going to need to make it a lot easier to understand.

Your comment on muscle weakness puzzled me. When it comes to opening bottle tops, lifting a heavy bag, wringing out a cloth, I have the same strength as I had before ME - I just don't have the stamina. When it comes to leg muscles, then obviously as I don't cycle any more, my legs have lost some of their strength, but this isn't a factor that I can attribute directly to ME. One of my first obvious symptoms with ME was weird sensations in my legs, so when I first went down with ME, my GP did that test where they run a stick along the sole of your foot. I nearly kicked him across the room - no loss of strength there! So I interpret the question to mean that my actual strength hasn't diminished but my energy has. The trouble is that "do you feel weak?" to me is the same question. My answer would be no - only when I am tired or have run out of energy.

As I will need to give an example, perhaps I should change this one - but am I unusual in the ME community for this? Would most people with ME "tick" all eleven boxes? Am I misunderstanding the questions?

I think you might be better to give other examples. I think the Goudsmit paper talks about the two items that were tending to not show up.

Here is one piece of data on symptoms where muscle weakness is pretty high. But it is probably a lot to do with severity how much one has and/or notices certain symptoms.

Also here is (UK) ME Association survey 2010 symptoms:

 

[alex3619]

Your comment on muscle weakness puzzled me. When it comes to opening bottle tops, lifting a heavy bag, wringing out a cloth, I have the same strength as I had before ME - I just don't have the stamina.

Hi Graham, I had lots of hand strength for most of the time I was ill - I was the guy who could open any bottle, jar etc in half a second. Now I cant even grip the jar top - muscle weakness appeared some years ago for me, but I don't really recall when that was. However, most of my problem is stamina, or rather recovery after effort. Once I get exhausted, I stay exhausted for a long time. So people who do not have muscle weakness yet might develop it later, and indeed I suspect this is the case for many symptoms. Bye, Alex

 

[Dolphin]

I don't know if this has been posted before, but it's a pack of lies, so I thought I'd flag it up.

Centre for Psychiatry Newsletter. Issue 2. Spring 2011.

FINDING THE RIGHT PACE for TREATING CHRONIC FATIGUE SYNDROME, by Peter White

http://issuu.com/lisakass/docs/centre_for_psychiatry_newsletter_-_issue_2

Thanks for this, Bob. Even though I haven't read it in full I'm interested in hearing about coverage like this. Indeed, even a slight change in how something is said could be a bit damning/useful.

BTW, does anyone know whether there is any way to download such files? I like to store things in my PACE Trial folder but didn't see any easy way of storing this.