Discussion in 'General ME/CFS News' started by Bob, Aug 17, 2012.
Happy to help with that old bean
I found myself a bit too tired to read this properly now, but just want to say thanks for the referencing which included quotes. This is clearly the best way to do things for writing like this, and I need to get in the habit more.
I'm not really comfortable with removing the gains which occurred from SMC from the SMC+CBT/GET figures without explicitly stating that this is what's been done either.
It's absurd for promoters of CBT/GET to fail to mention the gains from SMC, and that SMC was included with CBT and GET in PACE (and it's funny to see how often you've quoted them doing so, and I think I can remember other's too)... but they're quacks. In any full response, particularly because of the prejudices which surround CFS, I think that we should try to be tediously clear about exactly how we're using and presenting data. This can make for pieces of unreadable length, but there could always be a foot-note for more technical discussion.
Also, I'm far from clear what the most meaningful and honest way of presenting these results is. It would have been useful to have an arm which was intended to be a placebo control. Instead we had SMC, which gave patients much less 'therapy' time, and was also given to participants who were also receiving CBT and GET. I'm not sure how this should best be accounted for in our assessments as to what affect CBT and GET had upon patient's questionnaire scores.
I think an opening paragraph explaining these problems could help with clarity.
(sorry... gone really tired. Off to do something mindless).
I have fully explained about the SMC control group, and exactly how the results for SMC need to be taken into account.
The PACE Trial was clearly designed to be interpreted in the way I have gone about it, and not in the way it has been promoted in the media. I have purely used the published results from Table 3: "Primary outcomes of fatigue and physical function."
The primary results are very simple, and are clearly laid out in Table 3. I've not used any other data at all to present the main results.
The primary outcome results from Table 3 (Table 3: "Primary outcomes of fatigue and physical function") are as follows:
1. The "mean difference from SMC", the results of which are exactly as I have presented them, and
2. The "number improved from baseline", exactly as I have presented them, except that I have shown the 'difference from SMC'.
The 'difference from SMC' is exactly the way that the authors have intended the results to be interpreted.
This is what led to Sharpe talking about a NNT of 1 in 7, except that he has used the secondary post-hoc analysis to reach this figure, possibly because that gives a slightly more favourable result for CBT and GET than the primary results of a NNT of 1 in 8.
In order to arrive at a NNT of 1 in 8, one has to know the 'difference from SMC', which shows the proportion of patients whose improvements are attributable to CBT and GET. (i.e. an average of 13%.)
I have explained that my results do not include improvements seen in the SMC group, and that the results presented are attributable to CBT and GET.
The SMC group does not necessarily indicate any therapeutic effect at all. So it is not helpful to add the improvements of the SMC group onto the improvements for CBT and GET. In fact, this is counter-productive, and the study was designed with the SMC control group precisely to be able to see the effects of CBT and GET.
I'm in favour of maximum clarity, and that was my intention.
And I also think that accurate simplicity is important for a general forum audience.
So I presented the data as clearly as possible, and avoided unnecessary complexity, and getting bogged down in loads of impenetrable technical details.
But I haven't missed out any important details, in relation to the data that I have presented, as far as I am aware.
I will try to tidy it up a bit so that it is easier to read through.
Thanks again for your comments Simon.
Something to keep in mind is that this was all intended purely for a forum audience, so I wasn't worried about antagonising any psychiatrists.
If I was aiming to get it published anywhere, then I'd be more careful with my phrasing.
I've now posted Michael Sharpe's comment (in a recent post.)
I think I will leave these results as presented. I've clearly laid out all of the primary results so people can use them if helpful. I disagree that the analysis for both fatigue and physical function is the most important measure. It is a "secondary" and "post-hoc" analysis, and to my mind it adds confusion to the results. (The secondary post-hoc analysis places CBT and GET in a slightly more favourable light, so I can't imagine why they decided to include it!) And the primary results have the advantage of differentiating fatigue and physical function, which I think is helpful.
Thanks for that. I'm going to re-read it after a brain rest, in a few days, and then I'll tidy it up a bit. I'll try to clarify the info about a CUD. Thanks for pointing out that it could be more informative.
Thanks. I'll try to add clarity to that as well.
You are absolutely not alone there Simon. I think I would usually totally agree with you on that.
But my posts were intended for a forum audience, amongst friends, so I indulged myself in a tiny bit of sensationalism!
Having said that, I did think it important to highlight the issues that you have quoted, below, but maybe I could have gone about it slightly differently. I'll have a look at it all when I come back to it.
I could include a short new section on the NICE guidelines.
Thanks Simon. I'll think about what you've said, but I think it's OK to present the figures that I've presented. They are there in the paper itself, and I think that patients and others are able to interpret them.
And a 13% improvement rate does equate to a NNT of 1 in 8. It's exactly the same information. So I've just provided both sets of info.
I'll think further about what you've said, and possibly tone down my phrasing.
But I am basing my presentation exactly on the results that were published in Table 3, as explained in my previous post, and on the way the study was designed.
I would say that it has clearly been designed so that the 'difference from SMC' would show us the results attributable to the other therapies. This is confirmed by Sharpe's reference to the NNT.
The only primary results that they have provided, apart from the average effects, is how many acheived a CUD.
So I think that the data that they have presented can only lead us to conclude that an extra 13% benefited from CBT and GET, and that 87% were not shown to benefit from CBT and GET.
To my mind, this is exactly the same as giving a NNT. Don't you think so?
The way I've phrased the bit about "87% not responding to treatment" maybe is a little too absolute, so I'll have a look at that as well, but it does make the point very clearly, based on the results we have been given. If we are being absolute sticklers for technicalities, then maybe it might be more accurate to say that "87% were not shown to benefit."
There are shortcomings in the data, and if the authors wanted to present more information, on how each patient responded to treatment, then we could make a deeper analysis, but we can't.
So, your example is exactly how I would present the results if that's the only data that was presented, and if that's how the study was designed.
95% of patients did not respond to the NewWonderDrug, as far as we know, and it would be misleading to say so. Only 35% improved as a result of the drug, as far as the data tells us, and 60% in the control group improved.
Well, that's my thinking behind it all anyway. It seems to me that the Trial was designed to be interpreted in this way. But, it seems that all we are discussing here is the difference between a NNT and a percentage response rate.
The reason that I used the phrase "by contrast" was because I was highlighting the fact that the SMC group performed better than the therapy groups, because you wouldn't know this from reading some of promotional material. But I'll have another look at it. I agree that it is best not to be sensationalist.
I thought I only used the word 'subjective' twice, but I'll look at that as well.
I do think that the difference between a subjective 'symptom' and 'physical disability' is very important to highlight though. I might try to explain that without appearing sensationalist.
I forgot to mention that the results were subject to response bias, so I might add that as well.
I hope I haven't been too rambling... Just trying to explain all of my thinking, and it took more time than I expected... Always open to further discussion... Time for bed now.
Edit: Thanks again for the feedback Simon. It's much appreciated. I'll keep it all in mind when/if I get around to writing a revised version.
Esther12, oh, sorry, it's late and my earlier post might have come across as a bit ranty. I wasn't intending to rant at you Esther, as you know. I just wanted to explain and clarify.
edit: Bob: I possibly deserved a bit of a rant. It's quite possible I had just failed to take in much of what you wrote properly, as I'm semi-conscious here. I shouldn't really have replied on such a brief reading, but it was in my head, so had to choose between posting it or forgetting it.
Sorry Bob, I could well have missed bits. My eyes started to glaze over once it got more mathematical, so I only properly read the start, and then saw someone else saying what I had thought to myself. (I'm afraid I've not fully taken in your reply either... I should really step away from the PC and in to my bed).
You seemed to begin with a strong emphasis on the 13% figure, but didn't explain at the start that this is very different from the figures being promoted by the PACE researchers because you had attempted to account for the impact of SMC.
Actually,I just had another quick look, and I'm not sure that this bit is right:
It seems like there's some ambiguity over this, but I don't think we can really say the above. My memory could be off, but I also think that early on in the literature related to PACE, SMC was presented as more of a control group than it came to be in the final paper. It could be worth quoting from their description of SMC?
Okay, I'm going to have to crash. My eyes won't stay open. Night all.
No you didn't, Esther! Not at all.
It's always good to have feedback.
I'm just over-tired.
Thanks. If it's not clear enough to you, then it could certainly be clearer. I'll work on it.
It's not my attempt to account for SMC. It was the way the study was designed. The SMC group was proposed as a control group, and used as one. See Table 3, and it is set out as a control group. The relevant analysis is labelled as the "mean difference from SMC". The results have been muddied, but the actual details are very clear.
Yes, I understand how you have come to this conclusion. The authors have consistently referred to SMC as if it was just another therapy group, and often (incorrectly) saying that CBT and GET were more successful than SMC. But when you look at the design of the study, and the way the results are laid out, it's clear that the SMC group should be regarded as a control group. I think it has just been convenient for some to pretend otherwise, perhaps so that it can be (incorrectly) reported that CBT and GET are more effective than standard medical care..
Heads up, you 'orrible little dedicated overworking grunts!
Step away from the computer.
Leave the room. Do Not Look Back.
Make a nice cup of your favourite beverage.
Go outside, and be with the sky/sea/trees/birds etc for an hour. Or longer.
Squad – wait for it, wait for it... – diiiiiiiismissed!
The value of the work that has been done on PACE by many people on this forum who are mathmatically inclined, is because the patient charities seem to have been unable themselves to get the presentation of the statistics and outcomes checked.
Most people do not enjoy mathmatics and glaze over when figures are presented to them, and it seems this has enabled the PACE authors to hide the reality of their findings in a big pile of statistics.
It also seems that PACE evaded adequate peer review, so these inadequacies did not come to light.
We need precisely what Bob and others have provided here, which is a simple, clear exposition of exactly what PACE found.
And surprise, surprise, it agrees with what we intuitively know - that CBT and GET have little value, and cannot be promoted as treatments.
This information need to get out to the wider ME community and also to the patient charities, because no-one has hitherto been able to challenge the accuracy of the PACE findings before.
It is widely stated, even among patients that 30% improved on CBT/GET.
This is absolutely untrue and this fact needs to be understood.
Every time we see this figure wrongly quoted, we now can correct it.
Me too. I think a couple of nights of interrupted sleep (it's too hot!) had more of an impact than I realised.
Ah, yes, I was so tired last night that I couldn't remember that's why I felt it is so important to give such prominence to the "13%" figure.
And now the MRC has started saying that 60% improved as a result of CBT and GET.
Worryingly, I think I actually dreamt about control groups and statistics last night!
Strangely, it wasn't a even nightmare, which I think makes it even more worrying!
Definitely time for some sky/sea/trees/birds etc.
It's more than "just" the charities not putting together the data to fight pace. Does anyone know which charity Tony Johnson who oversaw the trial statistics is connected to ?
(from a MRC report which I think has now been deleted)
"I am not a PI because of familial involvement with one of the charities, a perspective that
has enabled me to play a vital role in ensuring that all involved in PACE maintain absolute neutrality to all trial treatments in presentation, documentation, and assessment."
Arghhhh, computer ate my full post, so you'll now get a very short version Lucky escape.
Ah, that makes sense, though given how helpful your piece is I suspect it will reach a far wider audience.
I'm pretty sure all CUD thresholds are secondary, even those in Table 3:
The 0.5 SD was used to assess if there was a clinically useful difference between the means of the primary outcomes.
Agreed that 95% would be misleading, but I think saying 65% do not respond could be misleading too. With 60% of the control group improving, the maximum improvement that is mathematically possible, even for "PerfectDrug", is 40% - less than for the control group. Giving both figures - in this case 95% vs 60% gives the full picture. For PACE it's 45% for the SMC/control group vs 60% for CBT, which I think gives a fair view of the results, and they are unimpressive.
Sorry if I got overxcited about a few subjective/self-report adjectives for Fatigue!
I see Bob gave you figures on people considered to have a depressive disorder in PACE.
I do not think it safe to assume that people with CF/Oxford-'CFS' have mainly primary depression (which, yes, is organic), regardless of how many are assessed to have 'depression' (or 'anxiety' or whatever) by psychiatrists (the DSM is in crisis because it cannot be shown that the diagnostic categories are reproducible or separate from one another, nor does a given DSM diagnosis suggest a treatment plan which is likely to work--this could be because research is inadequate as researchers have not typically explored many biological causes and have evidently been classing various diseases together during research).
People may end up in a CF group because they have been inadequately assessed. We in fact find that if we try a little bit harder to diagnose them, we find additional diseases like Lupus and stuff. Keep in mind that the latest Newcastle report, which found a 40% misdiagnosis rate, was a huge improvement over prior years in which the misdiagnosis rate was much higher. Also it's possible that some other diagnoses are still being missed (especially for diseases which are harder to diagnose because they lack a test, take some years to develop, or are rare).
It's not uncommon for psychiatrists (and other docs/allied health professionals) to decide that people with various chronic diseases have 'depression'. This is partly because chronic disease obviously causes grief and loss, because the pathology of various diseases can theoretically have similarities to the organic mechanisms of diseases of depression, and because the depression probes interpret loss of vitality, difficulty sleeping, and the like to contribute to the diagnosis of 'depression' even if they are actually related to something entirely different.
That being said, some investigators do seem to actively look for patients who seem depressed to them in some 'CFS' studies, and as we know Jason found that MDD was very easy to mistake for CFS when using, for instance, the CDC's 'Empirical' surveys (which use more symptoms than Oxford, but also allow depression or sadness to substitute for some of the Fukuda requirements). Of course, infection and all sorts of other things can look like depression especially to someone who typically expects to find that.
In my wee meeting yesterday, one of my colleagues, actually said at one point (quite out of the blue and it made me rather angry) that 'only 65% of those in PACE improved' thinking her comment was a positive point in her argument.
By the time I had thought about it she was on to something else. I tell you Bob the message hasn't even gotten through to even other patients.
She also came up with a statistic (out of the blue) that '13% of diagnoses were reported as wrong' citing some UK data storage bank. The name of which escapes me. BANE? Is that it? I'd never heard of it of course and it rather flies in the face of the two studies last year that I had built in to our written submission (the 40 and 50% alternate diagnosis ones).
I tell you. There are too many people singing from too many hymn sheets. Gods only know what anyone else thinks of the way in which we present ourselves. Damn worrying. And I'm still in a mood.
Bob, thank you for explaining the numbers to me. As Firestorm says the mixed figures from different studies can be so confusing. And as a tutor wrote on the bottom of my essay on statistics (pre-dissertation thing) "You really don't understand this do you?" LOL - the truth hurts.
willow - yes, you're right. It's a personal bug-bare of mine that depression gets treated so badly and with so little understanding. The DSM seems to get worse with each new publication. I've seen it used to prop up a broken table once- excellent use, should be the only one allowed.
The standard of psychiatrists is appalling imho. I worked with two excellent ones in 16 years of psychi nursing, the rest were on a scale of bad to dangerous.
I believe that the "it's all in yer 'ead" and "yuppie flu" reporting on ME has damaged the care of people with all kinds of depression as well as our care.
Sorry, I'm getting off topic.
I quite agree! there's no excuse for the way people with depression and other psychiatric-classified diseases are stigmatized and neglected.
The things is that 'all in the head' is standard response to any patient or disease which seems complicated or odd. This must not be allowed to continue. They should have learned better by now as so many 'imaginary' diseases have become demonstrated. Somehow we have to institute strict mores against this.
Back from my short break.
So that everyone can understand why the MRC is saying that 60% of participants improved (incorrectly attributed to CBT and GET), and why Michael Sharpe used a 'number needed to treat' (NNT) figure of "1 in 7", I'm including a breakdown of the secondary post-hoc analysis, from the PACE Trial paper, below. (I might include this in my first post, when I revise it.)
So, this secondary post-hoc analysis looks at the number of participants who achieved a clinically useful outcome in both of primary outcome measures (SF-36 physical function and Chalder fatigue), as opposed to achieving a clinically useful outcome in each of the primary outcome measures.
From the PACE Trial paper:
"A secondary post-hoc analysis compared the proportions of participants who had improved between baseline and 52 weeks by 2 or more points of the Chalder fatigue questionnaire, 8 or more points of the short form-36, and improved on both."
Proportion of participants who improved in both primary outcomes:
SMC alone 45%
Average for SMC+CBT and SMC+GET = 60% (This is the result that the MRC is mis-using, saying it is attributable to CBT and GET.)
However, when the improvements for the SMC group are taken into account, then the following results are apparent:
Improvements attributable to CBT 14% (NNT = 1 in 8) (rounded up)
Improvements attributable to GET 16% (NNT = 1 in 7) (rounded up)
Average for CBT and GET 15% (NNT = 1 in 7) (This is the 'number needed to treat' figure, that Michael Sharpe used.)
Remember that these are not the primary results, but are determined using a secondary post-hoc analysis.
I've included these figures just so that everyone can understand where the MRC's "60%" figure comes from, and where Michael Sharpe's "NNT = 1 in 7" figure comes from.
This secondary analysis places CBT and GET in a slightly more favourable light that the figures I presented, in my opening post, which are taken from Table 3 of the PACE Trial which presents the primary outcome results.
You can also try a Google Site Search
Separate names with a comma.