Misrepresentation of Randomized Controlled Trials in Press Releases & News Coverage: A Cohort Study
- Thread starter Dolphin
- Start date
alex3619
Senior Member
- Messages
- 13,810
- Location
- Logan, Queensland, Australia
"It’s not news that press releases are skewed. Previous research found that most press releases left out important caveats on safety or applicability of the research, and many flat out exaggerated the importance of results. “Our study adds to these results showing that ‘‘spin’’ in press releases and the news is related to the presence of ‘‘spin’’ in the published article,” say the authors. In other words – the root of the problem lies in how we write up research results in the first place."
Another quote from the article in Scientific American. Nice timing, together with another thread, as I am writing two connected blogs related to this.
Its clear that misleading press releases are causing major problems all over medical science. However, in the UK if a science journalist wants to follow up and do some in depth investigating, where do they go? Which "experts" do they interview? Do they use the Science Media Centre as a source? The impact of normal misreporting in science is likely to be increased by the particular circumstances around ME and CFS psychobabble.
Bye, Alex
Another quote from the article in Scientific American. Nice timing, together with another thread, as I am writing two connected blogs related to this.
Its clear that misleading press releases are causing major problems all over medical science. However, in the UK if a science journalist wants to follow up and do some in depth investigating, where do they go? Which "experts" do they interview? Do they use the Science Media Centre as a source? The impact of normal misreporting in science is likely to be increased by the particular circumstances around ME and CFS psychobabble.
Bye, Alex
biophile
Places I'd rather be.
- Messages
- 8,977
[edit: this post is in reply to this thread and any other recent threads covering similar issues, such as "Psychologists are facing up to problems with replication" and "Are most positive findings in health psychology false.... or at least somewhat exaggerated?".]
The number of papers and articles on research misconduct and misrepresentation have increased significantly in recent years. This is very good news as sooner or later the spotlight may be shone on the CFS backwater. I doubt "they" will believe yet it applies to CFS, unless there are specific examples published in journals. However, if psychology and psychiatry are the backwater of the scientific and medical communities, and CFS is a backwater subject in psychology and psychiatry, then I can imagine the problem must be rife in CFS research. Guilt by association is not enough though, we need specific examples. I guess some outsiders would consider the WPI/Mikovits/XMRV saga an example, but it would be unusual if a backwater subject in backwater disciples (biopsychosocial approach to CFS) was somehow free of this problem. I bet some patients/advocates are waiting with glee for scandalous examples to be exposed, which may be inevitable and have been suspected for a long time now. Personally I doubt there is much outright fraud, but plenty of more grey area examples such as selective reporting and spin doctoring, etc.
With the PACE Trial, the "spin" about normal and recovery was in the accompanying editorial and from the authors themselves at the Lancet press conference.
The number of papers and articles on research misconduct and misrepresentation have increased significantly in recent years. This is very good news as sooner or later the spotlight may be shone on the CFS backwater. I doubt "they" will believe yet it applies to CFS, unless there are specific examples published in journals. However, if psychology and psychiatry are the backwater of the scientific and medical communities, and CFS is a backwater subject in psychology and psychiatry, then I can imagine the problem must be rife in CFS research. Guilt by association is not enough though, we need specific examples. I guess some outsiders would consider the WPI/Mikovits/XMRV saga an example, but it would be unusual if a backwater subject in backwater disciples (
Dolphin said:
With the PACE Trial, the "spin" about normal and recovery was in the accompanying editorial and from the authors themselves at the Lancet press conference.
alex3619
Senior Member
- Messages
- 13,810
- Location
- Logan, Queensland, Australia
It was the spin from the PACE trial that inspired me to start research for a book. There is a lot more than just involving PACE. Unsupported claims are made all the time in this research, typically by implying more recovery than shown, or by implying causation when all they have is association. Bye, Alex
biophile
Places I'd rather be.
- Messages
- 8,977
The Lancet are complicit in the spin doctoring. 19 months later they still have not even published a correction for the editorial in question (there were reliable rumours about a planned correction last year, after previous resistance, but which now seems to have amounted to lip service from the Lancet). I tried to have another error corrected in a different editorial, but there was no interest either even though the error was admitted. I wonder about their commitment to accuracy.
Bob
Senior Member
- Messages
- 16,455
- Location
- England (south coast)
How are these for examples of 'spin' within the published PACE Trial paper?:
"Interpretation
CBT and GET can safely be added to SMC to moderately improve outcomes for chronic fatigue
syndrome, but APT is not an effective addition."
Here, they forget to qualify their remarks.
Firstly, severely affected patients were not investigated, and so the results do not apply to severely affected patients.
Secondly, CBT 'moderately improved' outcomes only in one of the two primary outcomes. There was no clinical improvement for 'physical function'. (Not to mention the results of the six minute walking distance test.)
Thirdly, it says that CBT and GET can be 'safely' added to SMC, but they have not published the 'deterioration rates' using the same measure as the 'improvement rates', so they are basing their assertion only on their own arbitrary measures of safety.
So the whole paragraph is full of cleverly-crafted half-truths.
“Panel 2: Research in context
Interpretation
In the pacing, graded activity, and cognitive behaviour therapy:
a randomised evaluation (PACE) trial, we affirm that cognitive
behaviour therapy and graded exercise therapy are moderately
effective outpatient treatments for chronic fatigue syndrome
when added to specialist medical care, as compared with
adaptive pacing therapy or specialist medical care alone.”
At no place in the text of the paper, or the discussion of the results, do they mention that CBT failed to clinically improve physical function or physical disability. (This information is only to be found with an analysis of the data tables.)
They repeatedly refer to CBT as being 'moderately effective', without ever qualifying their assertion. But CBT moderately improved just one of the two primary outcomes, so CBT could equally be called 'clinically ineffective'.
In the published paper, the only place where they make a reference to CBT failing to clinically improve physical function (one of the two primary outcomes) was in a fleeting oblique reference to it, as follows:
“Mean differences between groups on primary
outcomes almost always exceeded predefined clinically
useful differences for CBT and GET when compared
with APT and SMC.”
An example of ambiguous language:
“Discussion
When added to SMC, CBT and GET had greater success
in reducing fatigue and improving physical function than
did APT or SMC alone”
From reading this, the reader might think that CBT and GET were more successful therapies than SMC, and that CBT and GET were all-round successful therapies.
Not true. The SMC group saw greater improvements than the incremental additional improvements for CBT or GET.
And CBT did not demonstrate clinical success at improving physical function.
In such a high profile, multi-million pound medical research trial, would it have been so difficult to use wording that was unambiguous, such as:
"When used as a supplement to SMC, the incremental/additional effect size of GET was 'moderate'. For CBT, the incremental effect size for fatigue was 'moderate', but there was not a clinically useful improvement for physical function." ?
"Interpretation
CBT and GET can safely be added to SMC to moderately improve outcomes for chronic fatigue
syndrome, but APT is not an effective addition."
Here, they forget to qualify their remarks.
Firstly, severely affected patients were not investigated, and so the results do not apply to severely affected patients.
Secondly, CBT 'moderately improved' outcomes only in one of the two primary outcomes. There was no clinical improvement for 'physical function'. (Not to mention the results of the six minute walking distance test.)
Thirdly, it says that CBT and GET can be 'safely' added to SMC, but they have not published the 'deterioration rates' using the same measure as the 'improvement rates', so they are basing their assertion only on their own arbitrary measures of safety.
So the whole paragraph is full of cleverly-crafted half-truths.
“Panel 2: Research in context
Interpretation
In the pacing, graded activity, and cognitive behaviour therapy:
a randomised evaluation (PACE) trial, we affirm that cognitive
behaviour therapy and graded exercise therapy are moderately
effective outpatient treatments for chronic fatigue syndrome
when added to specialist medical care, as compared with
adaptive pacing therapy or specialist medical care alone.”
At no place in the text of the paper, or the discussion of the results, do they mention that CBT failed to clinically improve physical function or physical disability. (This information is only to be found with an analysis of the data tables.)
They repeatedly refer to CBT as being 'moderately effective', without ever qualifying their assertion. But CBT moderately improved just one of the two primary outcomes, so CBT could equally be called 'clinically ineffective'.
In the published paper, the only place where they make a reference to CBT failing to clinically improve physical function (one of the two primary outcomes) was in a fleeting oblique reference to it, as follows:
“Mean differences between groups on primary
outcomes almost always exceeded predefined clinically
useful differences for CBT and GET when compared
with APT and SMC.”
An example of ambiguous language:
“Discussion
When added to SMC, CBT and GET had greater success
in reducing fatigue and improving physical function than
did APT or SMC alone”
From reading this, the reader might think that CBT and GET were more successful therapies than SMC, and that CBT and GET were all-round successful therapies.
Not true. The SMC group saw greater improvements than the incremental additional improvements for CBT or GET.
And CBT did not demonstrate clinical success at improving physical function.
In such a high profile, multi-million pound medical research trial, would it have been so difficult to use wording that was unambiguous, such as:
"When used as a supplement to SMC, the incremental/additional effect size of GET was 'moderate'. For CBT, the incremental effect size for fatigue was 'moderate', but there was not a clinically useful improvement for physical function." ?
I think a spot light on research misconduct will start to change peoples behaviour. Currently I think many people think they can get away with spinning their results, mis quoting other articles. Reviewers think they can get away with a quick read and a few comments rather than a detailed review. If people feel the spot light may fall on them then they may behave better.
However, I think certain researchers have developed such a degree of arrogance that it will not change their behaviour. I wonder if Wessely and White have got away with such poor work for so many years that they don't recognise the problems. Reading Wesselys papers in particular I feel he lacks an analytic logical thought process but his writing style allows him to gloss over incoherant arguments.
Snow Leopard
Hibernating
- Messages
- 5,902
- Location
- South Australia
The problem with misrepresentation goes much deeper though, because it is a sociological problem relating to prestige and funding of science and scientists trying to justify their work.
It can't merely be solved by somehow forcing scientists themselves to be more honest, but we need to increase both the general societal knowledge of how science works, its importance and so forth, such that scientists are not completed to fall in this marketing trap in the first place.
It can't merely be solved by somehow forcing scientists themselves to be more honest, but we need to increase both the general societal knowledge of how science works, its importance and so forth, such that scientists are not completed to fall in this marketing trap in the first place.
alex3619
Senior Member
- Messages
- 13,810
- Location
- Logan, Queensland, Australia
In the last several decades the funding for science has become more and more dependent on private funding, including corporations doing applied research. In addition government funded research has been more about justification and spinning results - can't afford to waste a penny, or if that fails can't afford to be seen wasting a penny. To this end its important that research results are spun as effective. Future funding depends on it. This is a fundamental distortion of the funding base, leading to Zombie Science. There is a desire by some to hide failure wherever possible. The old notion of science as objective enquiry is disappearing. The other side of the distortion is that corporations want particular results. There is a lot of pressure to give it to them. Bye, Alex
This is one of the things that motivates me to write letters, e-letters, etc.
I have a different perspective but from research into IT. I think one of the big problems is the preasure within universities to get resarch grants and to publish lots of papers to meet the statistics. This means that academics tend not to challange the popular research directions. Imagine doing research into autoimmunity and ME and trying to publish - many of the best journals will get psyciatrists to do the review as they claim to have the expertise. When as a researcher you are judged every year against the publications you have in tier 1 journals its not an encouragement to work in that area.
Researchers in industrial research often have more freedom as there is not the preasure to publish and get research grants - however I can't think of many industrial research labs. R&D functions have a very different role which is to develop product.
Definately a good thing to do looking back at some papers you have picked them up on some very important points (as have others).
Unfortunately White et al are getting away with dismissing critisism from patients as personal attacks - again more unjustified spin. Perhaps thats a sign of their weakness since they can't argue the points.
biophile
Places I'd rather be.
- Messages
- 8,977
I fear that much of the research into CFS is some sort of "post-modern science".
The posts from user9876 and alex3619 also reminded me of the neuroscientist Ramachandran: "too much of the Victorian sense of adventure [in science] has been lost" [...] "But where I'd really like to go is back in time. I'd go to the Victorian age, before science had professionalized and become just another 9–5 job, with power-brokering and grants nightmares. Back then scientists just had fun. People like Darwin and Huxley; the whole world was their playground." (http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran).
CBT/GET proponents have done a good job at dismissing criticism without adequately addressing it, convincing outsiders that critics have nothing important to say, and downplaying the fact that their own research cannot demonstrate much if any objective benefit (despite presumed benefits and supposed recoveries, which influence policy and opinion).
The posts from user9876 and alex3619 also reminded me of the neuroscientist Ramachandran: "too much of the Victorian sense of adventure [in science] has been lost" [...] "But where I'd really like to go is back in time. I'd go to the Victorian age, before science had professionalized and become just another 9–5 job, with power-brokering and grants nightmares. Back then scientists just had fun. People like Darwin and Huxley; the whole world was their playground." (http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran).
CBT/GET proponents have done a good job at dismissing criticism without adequately addressing it, convincing outsiders that critics have nothing important to say, and downplaying the fact that their own research cannot demonstrate much if any objective benefit (despite presumed benefits and supposed recoveries, which influence policy and opinion).
As people are taught as children, "sticks and stones may break my bones, but words will never hurt me": if one makes a point that is fair, valid, and focused on the facts, it's not the end of the world if it is dismissed. If you've been on internet forums a while, you likely have taken bigger hits. So I wouldn't let that put people off.
These days CBT/GET proponents aren't necessarily annoyed by internet criticism as they've already had plenty. What they won't like is criticism in front of their peers with letters and e-letters. That's one of the reasons I think it's worth doing.
These days CBT/GET proponents aren't necessarily annoyed by internet criticism as they've already had plenty. What they won't like is criticism in front of their peers with letters and e-letters. That's one of the reasons I think it's worth doing.
alex3619
Senior Member
- Messages
- 13,810
- Location
- Logan, Queensland, Australia
Hi user9876, I agree with this too. The problems are multifactorial, and layered. I see it as a mutually reinforcing system. Other factors I didn't discuss include media limitations. The rise of the internet has seen budgets for journalism decline - investigation is becoming more superficial and often overlooked in favour of regurgitating press releases: churnalism in the articles that have been posted lately.
A decline in public understanding of science might also be involved, but I am less sure about this: anti nuclear protestors back in the 1950s (I think) walked around with placards like "I don't want no damned atoms around here!". I suspect science education was better for a while in the late twentieth century, but its in decline again in many countries (though not in Asia).
Industrial research also has to show results though. It may be less competitive (as in grants) but its still competitive in the long run. In the USA total research funding has gone up, but funding into pure research is in massive decline: a very great percentage is focussed on product development.
Economic rationalism is something I keep coming back to. Things began to change in the 1980s, and one of the factors was the drive to cut waste. An overemphasis on cutting costs is bad if universally applied. Its important not to waste money, but an excessive drive to cut non-productive research can lead to restriction to either short term goals or an emphasis on just how good the research is: spin. In other words if it looks good its ok, so spin has a managerial and financial benefit.
Then there is product testing, especially pharmaceuticals. The scientists have non-disclosure agreements. A company can fund a whole lot of studies, then pick the most favourable for publication. This practice is one which led to the term Zombie Science being coined. Combine this with editorial and reviewer bias in journals and what is published tends to be even more distorted. I also suspect that with modern demands on time from medical professionals, too many are reading abstracts instead of papers. Who has time for more if they have a busy medical practice? Abstracts are frequently distorted and sometimes misleading, reading the paper is critical - the abstract just lets you know the paper might be interesting. Anyone who does research should know that, but too many who talk about science seem to be unaware of it.
I am fairly sure there are other factors that are slipping my mind at the moment.
Bye, Alex
alex3619
Senior Member
- Messages
- 13,810
- Location
- Logan, Queensland, Australia
Hi, if you look at the history of pseudoscience, this is a hallmark feature: I see it again and again. They use pursuasive rhetoric to combat criticism, not reason and evidence. Bye, Alex
PS This is similar to what biophile said in post 16.
There is a drive for RCTs to be registered, with some journals saying they won't publish RCTs that haven't been registered. This should make it harder for some selective reporting.
A more recent phenomenon is a drive to register reviews and meta-analyses so that again there can't be selective reporting.
A huge chunk of the problem we face is that the psych crowd are telling the political and economic elites (and the 'moral' thugs) what they want to hear, so the psychs get lots of funding and political protection.