Ioannidis study: Empirical Evaluation of Very Large Treatment Effects of Medical Interventions

[Dolphin]

This is not to do with ME/CFS or any other condition specifically.
21-minute radio interview on the study is at this link:

http://www.scpr.org/programs/airtalk/2012/10/24/28991/many-medical-study-results-really-are-too-good-or-/

Many medical study results really are too good (or too bad) to be true

AirTalk | October 24th, 2012, 10:56am

If you happen to come across medical study results that claim a treatment has a “very large effect,” those results are likely either exaggerated or flat out wrong, according to researchers at Stanford University’s School of Medicine.

A statistical analysis of nearly 230,000 trials led by Dr. John Ioannidis, published today in the Journal of the American Medical Association, found that 90 percent of studies showing "very large effects” in initial reports on medical treatments are less effective or nonexistent when additional trials are conducted. Dr. Ioannidis attributes this misleading trend to a variety of causes including the fact that many studies’ sample sizes are too small and that the results are often based on intermediate effects only.

How surprising is it that the allegedly dramatic effects achieved by many medical treatments are exaggerated or false? What can medical professionals do differently to avoid making false claims about treatments?

Guest:


Dr. John Ioannidis, MD, Professor of Medicine and Health Research & Policy, Stanford University's School of Medicine; Co-author, "Empirical Evaluation of Very Large Treatment Effects of Medical Interventions," published in The Journal of the American Medical Association this week
Dr. Ivan Oransky, MD, Executive Editor, Reuters Health; teaches medical journalism at New York University’s Science, Health, and Environmental Reporting Program; co-creator of the blog Retraction Watch focused on retractions of studies in science journals

 

[Dolphin]

Abstract:

Empirical Evaluation of Very Large Treatment Effects of Medical Interventions

Tiago V. Pereira, PhD; Ralph I. Horwitz, MD; John P. A. Ioannidis, MD, DSc

JAMA. 2012;308(16):1676-1684. doi:10.1001/jama.2012.13444.

Original Contribution | October 24/31, 2012

ABSTRACT*

Context Most medical interventions have modest effects, but occasionally some clinical trials may find very large effects for benefits or harms.

Objective

To evaluate the frequency and features of very large effects in medicine.

Data Sources

Cochrane Database of Systematic Reviews (CDSR, 2010, issue 7).

Study Selection

We separated all binary-outcome CDSR forest plots with comparisons of interventions according to whether the first published trial, a subsequent trial (not the first), or no trial had a nominally statistically significant (P < .05) very large effect (odds ratio [OR], ≥5).

We also sampled randomly 250 topics from each group for further in-depth evaluation.

Data Extraction

We assessed the types of treatments and outcomes in trials with very large effects, examined how often large-effect trials were followed up by other trials on the same topic, and how these effects compared against the effects of the respective meta-analyses.

Results

Among 85 002 forest plots (from 3082 reviews), 8239 (9.7%) had a significant very large effect in the first published trial, 5158 (6.1%) only after the first published trial, and 71 605 (84.2%) had no trials with significant very large effects.

Nominally significant very large effects typically appeared in small trials with median number of events: 18 in first trials and 15 in subsequent trials.

Topics with very large effects were less likely than other topics to address mortality (3.6% in first trials, 3.2% in subsequent trials, and 11.6% in no trials with significant very large effects) and were more likely to address laboratory-defined efficacy (10% in first trials,10.8% in subsequent, and 3.2% in no trials with significant very large effects).

First trials with very large effects were as likely as trials with no very large effects to have subsequent published trials.

Ninety percent and 98% of the very large effects observed in first and subsequently published trials, respectively, became smaller in meta-analyses that included other trials; the median odds ratio decreased from 11.88 to 4.20 for first trials, and from 10.02 to 2.60 for subsequent trials.

For 46 of the 500 selected topics (9.2%; first and subsequent trials) with a very large-effect trial, the meta-analysis maintained very large effects with P < .001 when additional trials were included, but none pertained to mortality-related outcomes.

Across the whole CDSR, there was only 1 intervention with large beneficial effects on mortality, P < .001, and no major concerns about the quality of the evidence (for a trial on extracorporeal oxygenation for severe respiratory failure in newborns).

Conclusions Most large treatment effects emerge from small studies, and when additional trials are performed, the effect sizes become typically much smaller. Well-validated large effects are uncommon and pertain to nonfatal outcomes.

*I gave each sentence its own paragraph