Esther12
Senior Member
- Messages
- 13,774
I know a few of us have been following the AllTrials campaign (and generally feeling rather sceptical about it, as the BMJ publishes letters from White where he gives the impression of supporting it other than for concerns about patient confidentiality, despite refusing to release the data on PACE's protocol defined outcome measures). This piece seemed to make a number of points relevant to arguments White and co. have put forth while fighting freedom of information requests for data from PACE.
http://www.bmj.com/content/347/bmj.f1880?ijkey=KJaN2gQyuYf46yS&keytype=ref
Head to Head
Head to Head
Are clinical trial data shared sufficiently today? No
BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f1880 (Published 9 July 2013)
Cite this as: BMJ 2013;347:f1880
The AllTrials campaign asks for all trials to be registered and their results published. Ben Goldacre says we need the evidence to make informed decisions about medicines. John Castellani (doi:10.1136/bmj.f1881) says mandatory disclosure could affect patient privacy, stifle discovery, and allow competitors or unscrupulous actors to use the information [this is not a reference to CFS patients! - E12]
When discussing transparency it is important to be clear on what is being requested, as obfuscation is sometimes used to avoid discussing simple fixes. At stake are four levels of information about trials: (1) knowledge that a trial has been conducted, from a clinical trials register; (2) a brief summary of a trial’s results, in an academic journal article or regulatory summary; (3) longer details about the trial’s methods and results, from a clinical study report where available; (4) individual patient data. The AllTrials campaign calls only for the first three to be published.
The status quo is plainly unsatisfactory. The most current review—with no cherry picking permitted—estimates that around half of all trials for the treatments being used today have gone unpublished; and that trials with positive results are twice as likely to be disseminated.1 This is a problem for both industry and academic trials.
Although some in industry claim that these problems are in the past, in reality all supposed fixes have failed. In 2005, journal editors passed regulations stating that they would publish only registered trials: the evidence now shows these regulations have been widely ignored.2 In 2007, US legislation was passed requiring all trials since 2008 to post results on clinicaltrials.gov within a year of completion: the best published evidence shows this law has been ignored by 60-90% of trials.3 If industry representatives believe these problems have been fixed, they should present published evidence to support their case, with methods and results that are available for public scrutiny.
Even if the latest rules on transparency were to be implemented perfectly—starting from now—they would still do nothing to improve the evidence base for the treatments we use today, because they all cover only trials from the past few years. More than 80% of the medicines prescribed this year were generic, and came on the market more than a decade ago. We need the results of trials on these treatments, which are still available, albeit on paper. It is both practical and reasonable to request that these documents should be simply scanned, and shared.
The arguments against this level of transparency are conflicted and misguided. John Castellani, of the Pharmaceutical Research and Manufacturers of America (PhRMA), has claimed previously that it’s enough for regulators alone to see all the information on trials, and to see it behind closed doors. But this goes against the fundamental principles of science: we rely on transparency about methods and results, so that every experiment can be double checked and critically appraised. Although he might not realise it, Castellani’s position also exposes patients to real and unnecessary risks. Many of the most notable recent problems with medicines—problems with rofecoxib (Vioxx) and rosiglitazone (Avandia), for example, and problems with the evidence base for oseltamivir (Tamiflu)—were spotted by independent academics and doctors, and not by regulators. This isn’t because regulators are incompetent; on the contrary, they are highly trained, intelligent, and well motivated. But risks and benefits can be difficult to detect, and like everything in science, these problems benefit from many eyes.
For similar reasons, it is peculiar to see industry argue that information should not be shared simply because there might be disputes about interpretation: disputed interpretations are widespread throughout science and medicine, they are normal, and this open debate is how we get closer to the truth. And likewise, we do not silence medical scaremongers in the media by hiding information about trials; if anything, routinely withholding trial results is more likely to undermine public trust.
Overall, the lack of progress on transparency has been startling. Some worry that these problems should not be discussed in public, while we fix them quietly behind closed doors. But the problem of withheld trial results has been documented since at least 1986,4 and industry has successfully delayed remedial efforts for three decades. The latest strategy has been to raise the spectre of patient privacy.
In February, for example, PhRMA released a colourful statement that misleadingly suggested that I and the BMJ have somehow called for the reckless public release of full individual patient data sets. They made this claim, despite the head of press relations at PhRMA already knowing that neither I nor the AllTrials campaign call for individual patient data to be published.5 6
The BMJ has recently called for individual patient data to be made more widely available, in an editorial.7
Was this reckless and unreasonable? I don’t believe so. Where industry has shared data with researchers, it has been only piecemeal, and after enormous battles. But in many fields, there is already a long history of sensible and cautious sharing of detailed datasets—for example, to conduct individual patient data meta-analyses. These produce better estimates of treatment benefits, and improve care for patients, with appropriate concern for confidentiality. The Early Breast Cancer Trialists Collaborative Group’s meta-analyses, already published, represent just one notable example.8 The YODA project at Yale is looking at best practice for data sharing, as are many other groups.9 What’s more, the European Medicines Agency (EMA) has fully committed to sharing individual patient data after 2014, and are consulting only on the best mechanism to do so.10 These are reasonable and responsible things to discuss, as evidence based medicine moves forwards and becomes more effective.
Is patient confidentiality also an issue when clinical study reports are shared, as AllTrials and I have suggested they should be? Clinical study reports are long documents—often thousands of pages—but they are important, because analyses have shown that the information published in academic journal reports on clinical trials can be misleading or inaccurate, when compared with these longer, definitive sources of information.11 12 These reports certainly do contain some information about individuals—for example, in narrative descriptions of adverse events—but such information can easily be removed, or shared only with named researchers, if this is deemed necessary. Some industry figures have claimed that removing this material is either impossible or prohibitively expensive. But in 2010 the European ombudsman made a ruling of maladministration against the EMA, for claiming exactly that. The ombudsman examined the clinical study reports requested from the agency in detail, and concluded that the administrative burden of removing patient information, where necessary, was small. The European ombudsman has also stated clearly that there is no important commercially confidential information in these reports—the fact that a drug is not as good as claimed is not, in itself, something any company can hope to ethically withhold from doctors and patients.13 Since then, the EMA has released 1.6 million pages of clinical study reports14 under its new policy.13 Because these documents are so informative—and because the EMA holds only a small proportion of all the clinical study reports in existence—alltrials.net is asking for all existing clinical study reports to be made available, on all medicines currently in use.
This campaign has rapidly snowballed to become the mainstream position in the United Kingdom. AllTrials is now supported by more than 50 000 individuals, and 250 organizations, including more than 100 patient groups, the National Institute for Health and Care Excellence, academic funders such as the Medical Research Council and the Wellcome Trust, royal colleges, the Royal Pharmaceutical Society, the British Pharmacological Society, and the Faculty of Pharmaceutical Medicine, to name but a few. Ironically, within 24 hours of PhRMA denouncing our calls for greater transparency, GlaxoSmithKline—the world’s fourth largest drug company—signed up as supporters of alltrials.net. They have committed to do the very thing that PhRMA says is impossible, and share all clinical study reports going back to the foundation of the company.
If the transparency we ask for is practical, and reasonable, then what lies behind the colourful denunciations of PhRMA? Speaking to policy staff in some signatory organizations, one worrying theme recurs. We knew that withholding trial data was common, people have said, and we knew that it harms patients, but we felt embarrassed to talk about it, because even raising the issue seemed somehow subversive. This is a worrying state of affairs, and a testament to the power of aggressive lobbying by industry. But it is also perhaps a testament to the capture of key opinion leaders, and the dangers of longstanding inaction at senior levels in the medical establishment. In the UK, we have seen the same phenomena during prominent inquiries into failing hospitals: many senior staff, in numerous organizations, all saw a problem, but most were too busy—or too anxious about workplace conflict—to put patients first.
The problem of missing trials is one of the greatest ethical and practical problems facing medicine today. It also represents a bizarre paradox: we can spend millions of dollars on a trial, hoping it is free from bias, trying to detect a modest difference between two treatment groups; and then at the final moment we let all those biases and errors back in, by permitting half the results to disappear. Future generations may well look back at our tolerating this in amazement, in the same way that we look back on mediaeval bloodletting. The AllTrials movement is driving the solution forwards: patients need industry to engage constructively with this widespread consensus, on the practical details—urgently—so that we can all move on.
Notes
Cite this as: BMJ 2013;347:f1880
Footnotes
http://www.bmj.com/content/347/bmj.f1880?ijkey=KJaN2gQyuYf46yS&keytype=ref
Head to Head
Head to Head
Are clinical trial data shared sufficiently today? No
BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f1880 (Published 9 July 2013)
Cite this as: BMJ 2013;347:f1880
- Ben Goldacre, Wellcome research fellow in epidemiology
The AllTrials campaign asks for all trials to be registered and their results published. Ben Goldacre says we need the evidence to make informed decisions about medicines. John Castellani (doi:10.1136/bmj.f1881) says mandatory disclosure could affect patient privacy, stifle discovery, and allow competitors or unscrupulous actors to use the information [this is not a reference to CFS patients! - E12]
When discussing transparency it is important to be clear on what is being requested, as obfuscation is sometimes used to avoid discussing simple fixes. At stake are four levels of information about trials: (1) knowledge that a trial has been conducted, from a clinical trials register; (2) a brief summary of a trial’s results, in an academic journal article or regulatory summary; (3) longer details about the trial’s methods and results, from a clinical study report where available; (4) individual patient data. The AllTrials campaign calls only for the first three to be published.
The status quo is plainly unsatisfactory. The most current review—with no cherry picking permitted—estimates that around half of all trials for the treatments being used today have gone unpublished; and that trials with positive results are twice as likely to be disseminated.1 This is a problem for both industry and academic trials.
Although some in industry claim that these problems are in the past, in reality all supposed fixes have failed. In 2005, journal editors passed regulations stating that they would publish only registered trials: the evidence now shows these regulations have been widely ignored.2 In 2007, US legislation was passed requiring all trials since 2008 to post results on clinicaltrials.gov within a year of completion: the best published evidence shows this law has been ignored by 60-90% of trials.3 If industry representatives believe these problems have been fixed, they should present published evidence to support their case, with methods and results that are available for public scrutiny.
Even if the latest rules on transparency were to be implemented perfectly—starting from now—they would still do nothing to improve the evidence base for the treatments we use today, because they all cover only trials from the past few years. More than 80% of the medicines prescribed this year were generic, and came on the market more than a decade ago. We need the results of trials on these treatments, which are still available, albeit on paper. It is both practical and reasonable to request that these documents should be simply scanned, and shared.
The arguments against this level of transparency are conflicted and misguided. John Castellani, of the Pharmaceutical Research and Manufacturers of America (PhRMA), has claimed previously that it’s enough for regulators alone to see all the information on trials, and to see it behind closed doors. But this goes against the fundamental principles of science: we rely on transparency about methods and results, so that every experiment can be double checked and critically appraised. Although he might not realise it, Castellani’s position also exposes patients to real and unnecessary risks. Many of the most notable recent problems with medicines—problems with rofecoxib (Vioxx) and rosiglitazone (Avandia), for example, and problems with the evidence base for oseltamivir (Tamiflu)—were spotted by independent academics and doctors, and not by regulators. This isn’t because regulators are incompetent; on the contrary, they are highly trained, intelligent, and well motivated. But risks and benefits can be difficult to detect, and like everything in science, these problems benefit from many eyes.
For similar reasons, it is peculiar to see industry argue that information should not be shared simply because there might be disputes about interpretation: disputed interpretations are widespread throughout science and medicine, they are normal, and this open debate is how we get closer to the truth. And likewise, we do not silence medical scaremongers in the media by hiding information about trials; if anything, routinely withholding trial results is more likely to undermine public trust.
Overall, the lack of progress on transparency has been startling. Some worry that these problems should not be discussed in public, while we fix them quietly behind closed doors. But the problem of withheld trial results has been documented since at least 1986,4 and industry has successfully delayed remedial efforts for three decades. The latest strategy has been to raise the spectre of patient privacy.
In February, for example, PhRMA released a colourful statement that misleadingly suggested that I and the BMJ have somehow called for the reckless public release of full individual patient data sets. They made this claim, despite the head of press relations at PhRMA already knowing that neither I nor the AllTrials campaign call for individual patient data to be published.5 6
The BMJ has recently called for individual patient data to be made more widely available, in an editorial.7
Was this reckless and unreasonable? I don’t believe so. Where industry has shared data with researchers, it has been only piecemeal, and after enormous battles. But in many fields, there is already a long history of sensible and cautious sharing of detailed datasets—for example, to conduct individual patient data meta-analyses. These produce better estimates of treatment benefits, and improve care for patients, with appropriate concern for confidentiality. The Early Breast Cancer Trialists Collaborative Group’s meta-analyses, already published, represent just one notable example.8 The YODA project at Yale is looking at best practice for data sharing, as are many other groups.9 What’s more, the European Medicines Agency (EMA) has fully committed to sharing individual patient data after 2014, and are consulting only on the best mechanism to do so.10 These are reasonable and responsible things to discuss, as evidence based medicine moves forwards and becomes more effective.
Is patient confidentiality also an issue when clinical study reports are shared, as AllTrials and I have suggested they should be? Clinical study reports are long documents—often thousands of pages—but they are important, because analyses have shown that the information published in academic journal reports on clinical trials can be misleading or inaccurate, when compared with these longer, definitive sources of information.11 12 These reports certainly do contain some information about individuals—for example, in narrative descriptions of adverse events—but such information can easily be removed, or shared only with named researchers, if this is deemed necessary. Some industry figures have claimed that removing this material is either impossible or prohibitively expensive. But in 2010 the European ombudsman made a ruling of maladministration against the EMA, for claiming exactly that. The ombudsman examined the clinical study reports requested from the agency in detail, and concluded that the administrative burden of removing patient information, where necessary, was small. The European ombudsman has also stated clearly that there is no important commercially confidential information in these reports—the fact that a drug is not as good as claimed is not, in itself, something any company can hope to ethically withhold from doctors and patients.13 Since then, the EMA has released 1.6 million pages of clinical study reports14 under its new policy.13 Because these documents are so informative—and because the EMA holds only a small proportion of all the clinical study reports in existence—alltrials.net is asking for all existing clinical study reports to be made available, on all medicines currently in use.
This campaign has rapidly snowballed to become the mainstream position in the United Kingdom. AllTrials is now supported by more than 50 000 individuals, and 250 organizations, including more than 100 patient groups, the National Institute for Health and Care Excellence, academic funders such as the Medical Research Council and the Wellcome Trust, royal colleges, the Royal Pharmaceutical Society, the British Pharmacological Society, and the Faculty of Pharmaceutical Medicine, to name but a few. Ironically, within 24 hours of PhRMA denouncing our calls for greater transparency, GlaxoSmithKline—the world’s fourth largest drug company—signed up as supporters of alltrials.net. They have committed to do the very thing that PhRMA says is impossible, and share all clinical study reports going back to the foundation of the company.
If the transparency we ask for is practical, and reasonable, then what lies behind the colourful denunciations of PhRMA? Speaking to policy staff in some signatory organizations, one worrying theme recurs. We knew that withholding trial data was common, people have said, and we knew that it harms patients, but we felt embarrassed to talk about it, because even raising the issue seemed somehow subversive. This is a worrying state of affairs, and a testament to the power of aggressive lobbying by industry. But it is also perhaps a testament to the capture of key opinion leaders, and the dangers of longstanding inaction at senior levels in the medical establishment. In the UK, we have seen the same phenomena during prominent inquiries into failing hospitals: many senior staff, in numerous organizations, all saw a problem, but most were too busy—or too anxious about workplace conflict—to put patients first.
The problem of missing trials is one of the greatest ethical and practical problems facing medicine today. It also represents a bizarre paradox: we can spend millions of dollars on a trial, hoping it is free from bias, trying to detect a modest difference between two treatment groups; and then at the final moment we let all those biases and errors back in, by permitting half the results to disappear. Future generations may well look back at our tolerating this in amazement, in the same way that we look back on mediaeval bloodletting. The AllTrials movement is driving the solution forwards: patients need industry to engage constructively with this widespread consensus, on the practical details—urgently—so that we can all move on.
Notes
Cite this as: BMJ 2013;347:f1880
Footnotes
- Competing interests: I have read and understood the BMJ Group policy on declaration of interests and declare the following interests: I am a doctor, academic, and writer. I make income from talking and writing about problems in science, including publication bias. My research fellowship in epidemiology is funded by the Wellcome Trust, who are signatories to AllTrials, and support other aspects of open science.
- Provenance and peer review: commissioned; not peer reviewed.
- ↵
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess2010;14:1-193. www.hta.ac.uk/fullmono/mon1408.pdf. - ↵
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA2009;302:977-84.
CrossRefMedline - ↵
Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ2012;344:d7373.
CrossRefMedline - ↵
Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol1986;4:1529-41.
Abstract/FREE Full Text - ↵
PhRMA Catalyst blog post, comments, 2 Feb 2013. http://catalyst.phrma.org/clinical-trials-discovering-and-sharing-knowledge/. - ↵
PhRMA press release, 4 Feb 2013. http://phrma.org/media/releases/phrma-statement-clinical-trials-bad-pharma - ↵
Godlee F, Groves T. The new BMJ policy on sharing data from drug and device trials. BMJ2012;345:e7888.
FREE Full Text - ↵
Darby S, McGale P, Correa C, Taylor C, Arriagada R, Clarke M, et al. Early Breast Cancer Trialists’ Collaborative Group. Effect of radiotherapy after breast-conserving surgery on 10-year recurrence and 15-year breast cancer death: meta-analysis of individual patient data for 10 801 women in 17 randomised trials. Lancet2011;378:1707-16.
CrossRefMedlineWeb of Science - ↵
Center for Outcomes Research and Evaluation. Yale University Open Data Access (YODA) Project. http://medicine.yale.edu/core/projects/yodap/index.aspx - ↵
European Medicines Agency. Release of data from clinical trials, 2012. www.ema.europa.eu/ema/index.jsp?curl=pages/special_topics/general/general_content_000555.jsp&mid=WC0b01ac0580607bfa. - ↵
Vedula SS, Li T, Dickersin K. Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med 2013;10:e1001378.
CrossRefMedline - ↵
Doshi P, Jones M, Jefferson T. Rethinking credible evidence synthesis. BMJ2012;344:d7898.
FREE Full Text - ↵
Draft recommendation of the European ombudsman 2560/2007/BEH, 2010. www.ombudsman.europa.eu/cases/draftrecommendation.faces/en/4883/html.bookmark. - ↵
Doshi P, Jefferson T. The first 2 years of the European Medicines Agency’s policy on access to documents: secret no longer. JAMA Intern Med2013;173:380-2.
CrossRefMedlineWeb of Science
