• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

BMJ Editorial - Missing clinical trial data

Daisymay

Senior Member
Messages
754
http://www.bmj.com/content/344/bmj.d8158

BMJ Editorial - Missing clinical trial data

BMJ2012;344doi: 10.1136/bmj.d8158(Published 3 January 2012)
Cite this as:BMJ2012;344:d8158

1.Richard Lehman, senior research fellow1,
2.Elizabeth Loder, clinical epidemiology editor2
Author Affiliations
1.eloder@bmj.com

A threat to the integrity of evidence based medicine

Clinical medicine involves making decisions under uncertainty. Clinical research aims to reduce this uncertainty, usually by performing experiments on groups of people who consent to run the risks of such trials in the belief that the resulting knowledge will benefit others. Most clinicians assume that the complex regulatory systems that govern human research ensure that this knowledge is relevant, reliable, and properly disseminated. It generally comes as a shock to clinicians, and certainly to the public, to learn that this is far from the case.

The linked cluster of papers on unpublished evidence should reinforce this sense of shock. These articles confirm the fact that a large proportion of evidence from human trials is unreported, and much of what is reported is done so inadequately. We are not dealing here with trial design, hidden bias, or problems of data analysiswe are talking simply about the absence of the data. And this is no academic matter, because missing data about harm in trials can harm patients, and incomplete data about benefit can lead to futile costs to health systems. Moreover, researchers or others who deliberately conceal trial results have breached their ethical duty to trial participants.

The linked articles look closely at the extent, causes, and consequences of unpublished evidence from clinical trials. Hart and colleagues incorporated unpublished evidence into existing meta-analyses of nine drugs approved by the US Food and Drug Administration in 2001 and 2002.1 These reanalyses produced identical estimates of drug efficacy in just three of 41 cases (7%); in the remaining cases, estimates of drug efficacy were evenly split between more (19/41) and less (19/41). It is sometimes assumed that incorporation of missing data will reduce estimates of drug benefits, but this study shows that publication bias can cut both ways. Each increment of data can change the overall picture, but in most cases with no certainty that the picture is complete.

A fundamental step towards tackling this problem was taken in 2005, when, as Chan describes in the Research Methods and Reporting section, prior registration of all trials became a condition for later publication.2 Chan details the ways in which authors of systematic reviews can search for unpublished evidence, and he strikes an optimistic note when he states that Key stakeholdersincluding medical journal editors, legislators, and funding agenciesprovide enforcement mechanisms that have greatly improved adherence to registration practices.

However, two studies we publish give little cause for optimism that this adherence extends to timely sharing of trial results. A survey of publicly funded research in the United States between 2005 and 2008 by Ross and colleagues shows that registration is not followed by reporting of summary results within 30 months of completion in more than half of trials.3 Even at three years, one third remain unpublished. The US Food and Drug Administration Amendments Act of 2007 made publication of a results summary on ClinicalTrials.gov within 12 months mandatory for all eligible trials in the US initiated or ongoing as of September 2007Prayle and colleagues examine the extent to which this has happened.4 The tally stands at 22%. When the word mandatory turns out to mandate so little, the need for stronger mechanisms of enforcement becomes very clear.

Most clinical interventions in current use, however, are based on trials carried out before the era of mandatory registration, and here the task of data retrieval by systematic reviewers and national advisory bodies becomes impossible. Wieseler and colleagues show that the different documents available to researchers and regulatorsinternally produced study reports, study findings published in peer reviewed journals, and results posted in results registriessupplement each other, but that reporting quality is highest in study reports. However, the effort required to find and collate these sources can be prodigious and seldom guarantees completeness.5 In their just published Cochrane review update on antiviral treatments for influenza, Jefferson and colleagues describe a painstaking search for information from undisclosed trials stretching over several years.6

There is an Alice in Wonderland feel to these investigators effortsacting on the publics behalf, searching over hill and dale and among the paperwork of regulatory bodies and drug companies to put together pieces of data that should have been freely available in the first place. Even when data on individual participants are made available, they only form part of the jigsaw, and Ahmed and colleagues describe the problems of fitting in such data when the whole picture is not known.7

Finally, to find the randomised clinical trials that have been published in the medical literature, nearly every student, clinician, or researcher turns first to Medline among the biomedical databases. But Wieland and colleagues find that many reports of randomised controlled trials entered into Medline between 2006 and 2011 have not been indexed as such; thus, simply entering the search term randomised controlled trial into this database will miss many of these trials, despite the best efforts of the Cochrane Collaboration and the US National Library of Medicine.8

What is clear from the linked studies is that past failures to ensure proper regulation and registration of clinical trials, and a current culture of haphazard publication and incomplete data disclosure, make the proper analysis of the harms and benefits of common interventions almost impossible for systematic reviewers. Our patients will have to live with the consequences of these failures for many years to come. Retrospective disclosure of full individual participant data would be an important first step towards better understanding of the benefits and harms of many kinds of treatment. A model for this is provided by Medtronics recent agreement to release full individual participant data relating to its controversial bone productrecombinant human bone morphogenetic protein-2to independent analysis teams; so there is no longer any convincing reason for other companies to refuse similar disclosure of de-identified participant data from all past trials.9

The main challenge is to ensure better systems for the future. Because the optimal systematic review would have complete information about every trialthe full protocol, final study report, raw dataset, and any journal publications and regulatory submissions,2 10 a prospective system of research governance should insist on nothing less. This may require the global organisation of a suitable shared database for all raw data from human trialsan obvious next step for the World Health Organization after its excellent work on the International Clinical Trials Registry Platform Search Portal. Concealment of data should be regarded as the serious ethical breach that it is, and clinical researchers who fail to disclose data should be subject to disciplinary action by professional organisations. This may achieve quicker results than legislation in individual countries, although this is also desirable.
These changes have been long called for,11 and delay has already caused harm. The evidence we publish shows that the current situation is a disservice to research participants, patients, health systems, and the whole endeavour of clinical medicine.

Notes
Cite this as: BMJ 2012;344:d8158
Footnotes
Research, doi:10.1136/bmj.d7202
Research Methods and Reporting, doi:10.1136/bmj.d8013
Research, doi:10.1136/bmj.d7292
Research, doi:10.1136/bmj.d7373
Research, doi:10.1136/bmj.d8141
Research, doi:10.1136/bmj.d7501
Research, doi:10.1136/bmj.d7762
Competing interests: Both authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.
References
1. ?
Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ2012;344:d7202.
2. ?
Chan A-W. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ2012;344:d8013.
3. ?
Ross JS, Tse T, Zarin DA, Hui X, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ2012;344:d7292.
4. ?
Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ2012;344:d7373.
5. ?
Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on the quality of reporting of clinical drug trials: a comparison of registry reports, clinical study reports and journal publications. BMJ2012;344:d8141.
6. ?
Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev2011;12:CD008965.
7. ?
Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ2012;344:d7762.
8. ?
Wieland SL, Robinson KA, Dickersin K. Understanding why evidence from randomised clinical trials may not be retrieved from Medline: comparison of indexed and non-indexed records. BMJ2012;344:d7501.
9. ?
Krumholz HM, Ross JS. A model for dissemination and independent analysis of industry data. JAMA2011;306:1593-4.
FREE Full Text
10. ?
Jefferson T, Doshi P, Thompson M, Heneghan C. Ensuring safe and effective drugs: who can do what it takes? BMJ2011;342:c7258.
FREE Full Text
11. ?
Chalmers I. Underreporting research is scientific misconduct. JAMA1990;263:1405-8.
FREE Full Text
PDF
Easy Read
Respond to this article
 

Daisymay

Senior Member
Messages
754
http://www.naturalnews.com/034577_quack_science_clinical_trials_BMJ.html

Busted! Scientists leave out data to produce bogus findings
by S. L. Baker, features writer

(NaturalNews) Clinical trials of drugs and other medical therapies are carefully carried out and are the very gold standard of scientific proof, right? According to an in-depth review of this question just published in the British Medical Journal (BMJ,) the answer is no. In fact, the BMJ is sounding the alarm that data reported by scientists is too often not the truth -- because the researchers leave out inconvenient evidence. The result of facts-gone-missing could well be harming patients, spiking up healthcare costs by the selling of medical treatments based on bogus findings, and threatening the very integrity of medicine.

These warnings come from multiple papers released by the BMJ. The whistle-blowing authors of these articles examined the extent, causes, consequences of hidden facts, figures, and other data scientists discover as they do human trials. It turns out this is no "once in a while" kind of problem, either. The BMJ claims a "large proportion of evidence from human trials is unreported, and much of what is reported is done so inadequately."

In an editorial, Dr. Richard Lehman from the University of Oxford and BMJ Clinical Epidemiology Editor, Dr. Elizabeth Loder, nail the current state of medical research as a "culture of haphazard publication and incomplete data disclosure." They call for full access to raw trial data to allow better understanding of the benefits and harms of many treatments.

Bottom line: when data is left out, the missing facts distort the scientific record and published results of a study. This then leads doctors to make potentially dangerous clinical decisions about what drugs or procedures patients need because the docs are relying on skewed and even bogus "evidence."


Conveniently missing facts left out of drug trials and more
Papers in the current issue of BMJ include a study by Dr. Beth Hart and colleagues, which document how unpublished data is "conveniently missing" from many published meta-analyses of drug trials. That's right. Big Pharma's pills and potions are often pushed based on studies that simply ignore and leave out major data about what was really discovered about a medication lacking of benefits, potential dangers, side effects and more. Dr. Hart's team argues that access to full trial data is necessary to allow drugs to be independently assessed.

Two additional studies show the requirements for mandatory trial registration and timely sharing of results are poorly followed, if at all. For example, it turns out that less than half of US National Institutes of Health funded trials are published in a peer reviewed journal within 30 months of completion and only 22 percent of trials that are supposed to be subject to mandatory reporting had results available within one year of completion.

"When the word mandatory turns out to mandate so little, the need for stronger mechanisms of enforcement becomes very clear," the researchers report.

And what happens when ethical, dedicated scientists try to assess true harms vs benefits of Big Pharma drugs and other interventions? It's not a pretty picture for their careers, apparently. Additional studies published in the special BMJ issue highlight the many difficulties these researchers face when they try to buck the system.

Dr. Lehman and Dr. Loder, however, are bravely speaking out and directly saying that a concealment of data in clinical trials is anything but unusual. They label this "a serious ethical breach" and demand that clinical researchers who fail to disclose data "should be subject to disciplinary action by professional organizations. These changes have long been called for, and delay has already caused harm. The evidence we publish shows that the current situation is a disservice to research participants, patients, health systems, and the whole endeavor of clinical medicine."

For more information:

http://www.bmj.com/

http://www.naturalnews.com/medical_errors.html
 
Messages
5,238
Location
Sofa, UK
What is clear from the linked studies is that past failures to ensure proper regulation and registration of clinical trials, and a current culture of haphazard publication and incomplete data disclosure, make the proper analysis of the harms and benefits of common interventions almost impossible for systematic reviewers.

Wow! So much for the 'evidence' in 'evidence-based medicine'...

Long past time that medical science got to grips with this serious issue - and they are still only looking at the tip of the iceberg here. I didn't spot any particular emphasis on the (likely) widespread practice of commercial interests to selectively publish only those results that are convenient to them. A friend of mine has told me that she conducted cancer research that was industry-funded, along with about 20 other similar studies. Her results weren't published "because she got the wrong results". Other studies with opposite conclusions were published. I have no reason to doubt her and every reason to trust her - and if this is true, then the corruption of the research data must be widespread, and devastating to the reliability of the available evidence base.

How should this work, in order to provide reliable, credible scientific data?

It should not be permissible to conduct scientific research and not publish the results. It should be an intrinsic requirement of the scientific method that the experimental protocols are published before the experiment is conducted, and the results must then be reported, whatever those results may be. Any science that doesn't register and conform to those requirements should be considered inadmissable. Failing to publish full results - even in cases of 'failed' experiments - should be considered unacceptable. All the information should be published in full - and it should all be publicly available for scrutiny.

Some of these requirements will be absolute anathema to vested interests and commercial interests - what is needed would be a revolution in scientific practice, but these are minimum requirements for the maintenance of public credibility in science. It is simply not possible for a rational member of the public to trust that scientific data is genuinely 'scientific' unless science is conducted openly - there is too much evidence now of the distortion of research by commercial interests. The crisis in science can only be solved by a radical change in scientific customs and practices, and scientists can't continue blaming the press and the public for the crisis - science needs to look more thoughtfully at itself and consider the changes it needs to make in order to restore public trust.
 

Daisymay

Senior Member
Messages
754
Absolutely agree Mark, but then again would it not be possible for researchers to provide their raw data but for that to have been fabricated? How would we know if it was accurate? And what about the abuse of statistical analysis that would have to be looked at too.
 
Messages
13,774
I feel a bit irritated with the BMJ for publsihing this after the piss poor way they dealt with these issues and CFS.

Good that their finally drawing some attention to these obvious and serious problems though.

Maybe in 50 years time we'll come up with something semi-professional.
 

markmc20001

Guest
Messages
877
Wow! So much for the 'evidence' in 'evidence-based medicine'...

Long past time that medical science got to grips with this serious issue - and they are still only looking at the tip of the iceberg here. I didn't spot any particular emphasis on the (likely) widespread practice of commercial interests to selectively publish only those results that are convenient to them. A friend of mine has told me that she conducted cancer research that was industry-funded, along with about 20 other similar studies. Her results weren't published "because she got the wrong results". Other studies with opposite conclusions were published. I have no reason to doubt her and every reason to trust her - and if this is true, then the corruption of the research data must be widespread, and devastating to the reliability of the available evidence base.

How should this work, in order to provide reliable, credible scientific data?

It should not be permissible to conduct scientific research and not publish the results. It should be an intrinsic requirement of the scientific method that the experimental protocols are published before the experiment is conducted, and the results must then be reported, whatever those results may be. Any science that doesn't register and conform to those requirements should be considered inadmissable. Failing to publish full results - even in cases of 'failed' experiments - should be considered unacceptable. All the information should be published in full - and it should all be publicly available for scrutiny.

Some of these requirements will be absolute anathema to vested interests and commercial interests - what is needed would be a revolution in scientific practice, but these are minimum requirements for the maintenance of public credibility in science. It is simply not possible for a rational member of the public to trust that scientific data is genuinely 'scientific' unless science is conducted openly - there is too much evidence now of the distortion of research by commercial interests. The crisis in science can only be solved by a radical change in scientific customs and practices, and scientists can't continue blaming the press and the public for the crisis - science needs to look more thoughtfully at itself and consider the changes it needs to make in order to restore public trust.

Always right on the money sir.

It is the small elite group of scientists in charge, manufacturing the false consensus that concern me the most.
 
Messages
5,238
Location
Sofa, UK
Absolutely agree Mark, but then again would it not be possible for researchers to provide their raw data but for that to have been fabricated? How would we know if it was accurate? And what about the abuse of statistical analysis that would have to be looked at too.

Fair points, there is still more than what I've mentioned that would be needed for a full open and rigorous process. Open publication and discussion of papers and of raw data would go a long way with the problems of statistical analysis I think. If there was an open public discussion of PACE, and full disclosure of the raw data, it would be much harder to twist the results as has clearly happened.

As regards the fabrication of data, the conventional response to that question is to say that peer review can never deal with that problem, and this is why all research needs to be confirmed by other groups before it is accepted; professional reputation is supposed to provide a check on fabrication of data but it doesn't always work. The requirement for results to be independently repeated does provide some checks and balances, but clearly it doesn't completely solve the problem of trust. I think that, ultimately, trust requires openness, transparency, and engagement, and that's often hard to achieve and for many scientists it doesn't come naturally, it's not the way they are used to operating, but it's the only way. Completely solving all these problems might in the end prove impossible, but there's certainly an awful lot more that could be done. I'm encouraged by the trend towards open publication though, things do seem to be moving in the right direction there. I suppose financial interests and the funding of research are the biggest problems though, we've got ourselves into a hell of a mess there over the last couple of decades...
 

Enid

Senior Member
Messages
3,309
Location
UK
Wow too - thanks for posting dasimay. Big implications rightly here.
 

Tony Mach

Show me the evidence.
Messages
146
Location
Upper Palatinate, Bavaria
It should not be permissible to conduct scientific research and not publish the results. It should be an intrinsic requirement of the scientific method that the experimental protocols are published before the experiment is conducted, and the results must then be reported, whatever those results may be. Any science that doesn't register and conform to those requirements should be considered inadmissable. Failing to publish full results - even in cases of 'failed' experiments - should be considered unacceptable. All the information should be published in full - and it should all be publicly available for scrutiny.

Always right on the money sir.

It is the small elite group of scientists in charge, manufacturing the false consensus that concern me the most.

And why stop at the small elite group of scientists, when there is a small fringe group of scientists to whom you can address the same criticism? Or is the case of the 5AZA slide something different?
 

floydguy

Senior Member
Messages
650
OK, let's make a connection to ME/CFS research: The 5AZA slide comes to my mind.

If the same level of scrutiny was made of all research than we wouldn't be where are currently. Mikovits is clearly a maverick so there should be intense scrutiny but I guess the question is how much is due to the fact that many just don't like her? For me, one of the shocking revelations about the research world is how much it is driven by personality not by "science" . I've never been naive enough to think it wasn't driven by money but I did naively think they were above the behavior of pre-teen girls.
 

markmc20001

Guest
Messages
877
And why stop at the small elite group of scientists, when there is a small fringe group of scientists to whom you can address the same criticism? Or is the case of the 5AZA slide something different?

Heya Tony,

I ignore the media and concensus, and try to find the truth by looking at the evidence I can interpret.

I also know the difficulties associated with repeating experiments or getting consistent results from my days designing automation for use in class 100 clean rooms. Ignoring or adjusting variables can and will change the results in experiments.

Here is my non-expert opinion:

1) No studies to date have replicated exactly the Lombardi Assays, so there is no way of knowing if Dr Mikovitz found any virus.

2) The virus in the previous studies were calibrated to a VP62 clone, and never calibrated to the virus that Mikovitz found.
(See Burzynski Movie for how FDA scientist interviewed claims protocols have been designed to fail at the NCI, especially when evaluating alternative treatments)

3) Dr Lo stands by his MLV research findings, but somehow retracts his findings based on consensus? (I'm not looking for concensus. I'm looking for the truth)

Given the three bits of information above, it is impossible to determine if Mikovitz did find a virus in my opinion.

Now for my other opinions:

1) I don't believe governments act in good faith all the time. See Tuskegee experiments.

2) I think the worldwide media is capable of selectively reporting news to suit special interests. See most recent USA presidential elections where Ron Paul is a front runner and barely mentioned by media. When he is mentioned, he is framed in a negative light or asked questions on the most controversial parts of his policy.

3) And yes, I think it is possible their is a small group of scientists that can influence a consensus based on what suits coorporate masters agenda. Let's not be naive here, meaningful medical research simply doesn't get financed without corporate/government funding.

I think corporate america has so much power in today's world, it can essentially co-opt whoever they want in certain influential positions. Just ask former head of Centers for Disease Control Julie Gerberding, who is now president of Merck vaccines.

http://adventuresinautism.blogspot.com/2009/12/julie-gerberding-named-head-of-merck.html

Or better yet, ask Jack Abramoff how government officials can be made to do anything.

http://www.cbsnews.com/video/watch/?id=7387331n