Observer bias in RCTs with measurement scale outcomes: review of trials w/ both blinded & nonblinded

[Dolphin]

A minority interest, I imagine.

This could be a useful reference for people who wanted something on the issue:

Free full text: http://www.cmaj.ca/content/185/4/E201.long


CMAJ. 2013 Mar 5;185(4):E201-11. doi: 10.1503/cmaj.120744. Epub 2013 Jan 28.

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors.

Hróbjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S.

Source

Nordic Cochrane Centre, Rigshospitalet Department 7811, Copenhagen, Denmark. ah@cochrane.dk

Abstract

BACKGROUND:

Clinical trials are commonly done without blinded outcome assessors despite the risk of bias.

We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales.

METHODS:

We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome.

We searched PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press and Google Scholar for relevant studies.

Two investigators agreed on the inclusion of trials and the outcome scale.

For each trial, we calculated the difference in effect size (i.e., standardized mean difference between nonblinded and blinded assessments).

A difference in effect size of less than 0 suggested that nonblinded assessors generated more optimistic estimates of effect.

We pooled the differences in effect size using inverse variance random-effects meta-analysis and used metaregression to identify potential reasons for variation.

RESULTS:

We included 24 trials in our review.

The main meta-analysis included 16 trials (involving 2854 patients) with subjective outcomes.

The estimated treatment effect was more beneficial when based on nonblinded assessors (pooled difference in effect size -0.23 [95% confidence interval (CI) -0.40 to -0.06]).

In relative terms, nonblinded assessors exaggerated the pooled effect size by 68% (95% CI 14% to 230%).

Heterogeneity was moderate (I(2) = 46%, p = 0.02) and unexplained by metaregression.

INTERPRETATION:

We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes.

A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.

 

[biophile]

Similarly (and free full text too) ...

Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors.

Hróbjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S.

BMJ. 2012 Feb 27;344:e1119. doi: 10.1136/bmj.e1119.

Nordic Cochrane Centre, Rigshospitalet Department 3343, Blegdamsvej 9, 2100 Copenhagen Ø, Denmark. ah@cochrane.dk

OBJECTIVE: To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes.

DESIGN: Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios--the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias.

DATA SOURCES: PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome.

RESULTS: We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective--for example, qualitative assessment of the patient's function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor's overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%).

CONCLUSIONS: On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.

PMID: 22371859

http://www.bmj.com/content/344/bmj.e1119