Daisymay
Senior Member
- Messages
- 754
The MRC PACE Trial will the results be reliable?
Margaret Williams 16th October 2010
In Magical Medicine: how to make a disease disappear reference was made to the meta-research of Professor John Ioannidis and to his disturbing findings of the high level of bias in medical research (Known biases in Random Controlled Trials may not have been avoided in the PACE Trial: http://www.meactionuk.org.uk/magical-medicine.htm , page 272 ff).
It may be of interest and is certainly of relevance -- to the ME community that a new article considers the work of Professor Ioannidis in more depth (Lies, Damned Lies, and Medical Science; David H Freedman; The Atlantic, November 2010: http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269 ). The introduction immediately captures attention:
Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong, so why are doctors to a striking extent still drawing upon misinformation in their everyday practice?
The following are notable extracts from that article (note that the extracts are not always in sequential order as they appear in Freedmans article) :
(Ioannidis) has become one of the worlds foremost experts on the credibility of medical research.He charges that as much as 90 percent of the published information doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the fields top journals, where it is heavily cited. Ioannidis may be one of the most influential scientists aliveTo say that Ioannidis work has been embraced would be an understatement. His PLoS Medicine paper (the one quoted in Magical Medicine) is the most downloaded in the journals history.Other researchers are eager to work with him: he has published papers with 1,328 different co-authors at 538 institutions in 43 countries.Last year he received, by his estimate, invitations to speak at 1,000 conferences and institutions around the world.Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change.
Discussing randomised controlled trials (RCTs), of which the PACE Trial is one, Ioannidis says: I realised even our gold-standard research had a lot of problems. Baffled, he started looking for the specific ways in which studies were going wrong. And before long he discovered that the range of errors being committed was astonishing: from what questions the researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analysed the data, to how they presented their results, to how particular studies came to be published in medical journals.
This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. The studies were biased, he says. Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there. Researchers headed into their studies wanting certain results and, lo and behold, they were getting them.
At every step in the process, there is room to distort results. A way to make a stronger claim or to select what is going to be concluded says Ioannidis. There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded .
In the late 1990s Ioannidisstarted chipping away at the problem in a series of papers that pointed out specific ways certain papers were getting misleading results. Other meta-researchers were also starting to spotlight disturbingly high rates of error in the medical literature. Ioannidis wanted to get the big picture across, and to do so with solid data, clear reasoning, and good statistical analysis.A pervasive theme of ancient Greek literature is that you need to pursue the truth, no matter what the truth might be, he says.
in PLoS MedicineIoannidis laid out a detailed mathematical proof thatresearchers will come up with wrong findings most of the time. Simply put, if youre attracted to ideas that have a good chance of being wrong, and if youre motivated to prove them rightyoull probably succeed in proving wrong theories right.The article spelled out his belief that researchers werechasing career-advancing findings rather than good science, and even using the peer-review processto suppress opposing views.
Ioannidis zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years as judged by the science communitys two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments.Of the 49 articles, 41 percent had been convincingly shown to be wrong or significantly exaggerated.
Ioannidis points out that obviously questionable findings cram the pages of top medical journals (and) suggests a simple approach: ignore them all.
Even when the evidence shows that a particular research idea is wrong, if you have thousands of scientists who have invested their careers in it, theyll continue to publish papers on it, he says.
Its like an epidemic, in the sense that theyre infected with these wrong ideas, and theyre spreading it to other researchers through journals .
Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correctin one case for at least 12 years after the results were discredited.
Doctors needs to rely on instinct and judgment to make choices he says. But these choices should be as informed as possible by the evidence. And if the evidence isnt good, doctors should know that, too. And so should patients.If we dont tell the public about these problems, were no better than nonscientists who claim they can heal .
as long as careers remain contingent on producing a stream of research thats dressed up to seem more right than it is, scientists will keep on delivering exactly that .
In the light of Professor Ioannidis findings and mindful of what is already known (and has been shown) about the PACE Trial in Magical Medicine, it would seem essential that the international ME community applies forensic scrutiny to the raw data upon which the PACE Trials results depend (and as a publicly-funded trial, the raw data must be disclosed, otherwise suspicion might be legitimately aroused), especially as Dutch researchers have now accepted that the validity and reliability of measuring physical activity is questionable or unknown (Evering RM et al; Clin Rehabil 2010; October 13; Epub ahead of print: http://www.ncbi.nlm.nih.gov/pubmed/20943713).
Given that the Department of Health is one of the co-funders of the PACE Trial and given that it is a matter of record that The Department funds research to support policy (Hansard: 11th May 2000:461W-462W), the tenacity with which the Investigators involved with the PACE Trial strive to produce policy-based evidence that validates cognitive restructuring as the intervention of choice for patients with ME/CFS seems illustrative of the serious criticisms advanced by Ioannidis, because policy-based evidence is not science-based evidence, nor does it pursue truth.
May be reposted
Margaret Williams 16th October 2010
In Magical Medicine: how to make a disease disappear reference was made to the meta-research of Professor John Ioannidis and to his disturbing findings of the high level of bias in medical research (Known biases in Random Controlled Trials may not have been avoided in the PACE Trial: http://www.meactionuk.org.uk/magical-medicine.htm , page 272 ff).
It may be of interest and is certainly of relevance -- to the ME community that a new article considers the work of Professor Ioannidis in more depth (Lies, Damned Lies, and Medical Science; David H Freedman; The Atlantic, November 2010: http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269 ). The introduction immediately captures attention:
Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong, so why are doctors to a striking extent still drawing upon misinformation in their everyday practice?
The following are notable extracts from that article (note that the extracts are not always in sequential order as they appear in Freedmans article) :
(Ioannidis) has become one of the worlds foremost experts on the credibility of medical research.He charges that as much as 90 percent of the published information doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the fields top journals, where it is heavily cited. Ioannidis may be one of the most influential scientists aliveTo say that Ioannidis work has been embraced would be an understatement. His PLoS Medicine paper (the one quoted in Magical Medicine) is the most downloaded in the journals history.Other researchers are eager to work with him: he has published papers with 1,328 different co-authors at 538 institutions in 43 countries.Last year he received, by his estimate, invitations to speak at 1,000 conferences and institutions around the world.Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change.
Discussing randomised controlled trials (RCTs), of which the PACE Trial is one, Ioannidis says: I realised even our gold-standard research had a lot of problems. Baffled, he started looking for the specific ways in which studies were going wrong. And before long he discovered that the range of errors being committed was astonishing: from what questions the researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analysed the data, to how they presented their results, to how particular studies came to be published in medical journals.
This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. The studies were biased, he says. Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there. Researchers headed into their studies wanting certain results and, lo and behold, they were getting them.
At every step in the process, there is room to distort results. A way to make a stronger claim or to select what is going to be concluded says Ioannidis. There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded .
In the late 1990s Ioannidisstarted chipping away at the problem in a series of papers that pointed out specific ways certain papers were getting misleading results. Other meta-researchers were also starting to spotlight disturbingly high rates of error in the medical literature. Ioannidis wanted to get the big picture across, and to do so with solid data, clear reasoning, and good statistical analysis.A pervasive theme of ancient Greek literature is that you need to pursue the truth, no matter what the truth might be, he says.
in PLoS MedicineIoannidis laid out a detailed mathematical proof thatresearchers will come up with wrong findings most of the time. Simply put, if youre attracted to ideas that have a good chance of being wrong, and if youre motivated to prove them rightyoull probably succeed in proving wrong theories right.The article spelled out his belief that researchers werechasing career-advancing findings rather than good science, and even using the peer-review processto suppress opposing views.
Ioannidis zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years as judged by the science communitys two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments.Of the 49 articles, 41 percent had been convincingly shown to be wrong or significantly exaggerated.
Ioannidis points out that obviously questionable findings cram the pages of top medical journals (and) suggests a simple approach: ignore them all.
Even when the evidence shows that a particular research idea is wrong, if you have thousands of scientists who have invested their careers in it, theyll continue to publish papers on it, he says.
Its like an epidemic, in the sense that theyre infected with these wrong ideas, and theyre spreading it to other researchers through journals .
Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correctin one case for at least 12 years after the results were discredited.
Doctors needs to rely on instinct and judgment to make choices he says. But these choices should be as informed as possible by the evidence. And if the evidence isnt good, doctors should know that, too. And so should patients.If we dont tell the public about these problems, were no better than nonscientists who claim they can heal .
as long as careers remain contingent on producing a stream of research thats dressed up to seem more right than it is, scientists will keep on delivering exactly that .
In the light of Professor Ioannidis findings and mindful of what is already known (and has been shown) about the PACE Trial in Magical Medicine, it would seem essential that the international ME community applies forensic scrutiny to the raw data upon which the PACE Trials results depend (and as a publicly-funded trial, the raw data must be disclosed, otherwise suspicion might be legitimately aroused), especially as Dutch researchers have now accepted that the validity and reliability of measuring physical activity is questionable or unknown (Evering RM et al; Clin Rehabil 2010; October 13; Epub ahead of print: http://www.ncbi.nlm.nih.gov/pubmed/20943713).
Given that the Department of Health is one of the co-funders of the PACE Trial and given that it is a matter of record that The Department funds research to support policy (Hansard: 11th May 2000:461W-462W), the tenacity with which the Investigators involved with the PACE Trial strive to produce policy-based evidence that validates cognitive restructuring as the intervention of choice for patients with ME/CFS seems illustrative of the serious criticisms advanced by Ioannidis, because policy-based evidence is not science-based evidence, nor does it pursue truth.
May be reposted