Evidence of bias in treatment effects in controlled trials with different interventions and outcomes

[Dolphin]

(From 2008)
Full title: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes:meta-epidemiological study

Free full text: http://www.bmj.com/content/336/7644/601
---

BMJ. 2008 Mar 15;336(7644):601-5. doi: 10.1136/bmj.39465.451748.AD. Epub 2008 Mar 3.
Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study.
Wood L1, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA.
Author information

Abstract

OBJECTIVE:

To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome.

DESIGN:
Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses.

DATA SOURCES:
146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes.

MAIN OUTCOME MEASURES:

Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome.

A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates.

RESULTS:

In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)).

In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding.

There was little evidence for a difference between trials of drug and non-drug interventions.

Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses.

CONCLUSIONS:

The average bias associated with defects in the conduct of randomised trials varies with the type of outcome.

Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.

PMID:
18316340
PMCID:
PMC2267990
DOI:
10.1136/bmj.39465.451748.AD
[PubMed - indexed for MEDLINE]

 

[Woolie]

Great article, @Dolphin!

The key finding from this metanalysis was that lack of blinding in an efficacy trial increased the reported treatment effects when the outcome measure was subjective - that is, when it involved patients rating their own health, or doctors rating it for them on a subjective scale.

Here's fig 2, the results showing degree of bias for various types of nonblinded trials. I've put a red box around the most important finding. Note the difference is significant.

They said - and I have to laugh - that this finding was "unexpected"! That is, they didn't expect that bias due to lack of blinding would be bigger for trials using subjective than objective measures. Have they been living under a rock or something???

It is especially interesting that bias was inherent in all studies relying on subjective measures, including both patient ratings and doctor ratings. So bias in these situations is unlikely to be the result of a magical mind-over-body placebo effect - not unless you really believe the doctor's mind can influence the patient's body.

Its just plain old boring confirmation bias.