Citation Analysis Reveals Disproportionate Emphasis on Positive Claims in Study Abstracts
University of Bristol
Evidence-based medicine relies on evidence from high-quality randomized trials and observational studies (1). However, meta-analyses, the cornerstone of evidence-based medicine, may be at odds with the prevailing views of the relevant research community. One factor that may contribute to the distortion of an evidence base is selective citation of studies that support (or do not support) a particular treatment. Greenberg (2) showed that citation distortions (such as a bias against citing studies that refute or weaken a particular belief) create an unfounded impression of authority and consensus.
We used a recent meta-analysis of cognitive behavioural therapy in schizophrenia (3) to explore patterns of citations within this literature. Studies included in the meta-analysis (k = 50) were coded according to whether or not they provided support for the intervention (defined as P < 0.05 in the direction of therapeutic benefit). We also coded whether or not the study abstract reported that the study provided support for the intervention, given evidence that abstracts may over-state the findings of a study (4). Finally, we coded which other studies within the literature (i.e., those included in the meta-analysis) were cited by each individual study. Independent coding was by two authors (DFH and NEJR), and discrepancies resolved by mutual consent.
Our results indicated that abstracts are more likely to report a favourable outcome (42 studies, 84%) compared with the data reported within the meta-analysis (20 studies, 40%). We also observed a clear pattern of citation distortion: studies that claimed support in the abstract received 138 citations (95%), while those that showed support for the intervention on the basis of the data reported in the meta-analysis received 58 citations (40%). A binomial test indicated that the number of citations to studies with abstracts reporting a favourable outcome was greater than expected (95% of citations from 84% of abstracts, P < 0.001).
These findings could explain discrepancies between the evidence base as indicated by a meta-analysis, and the beliefs held by practitioners and clinicians within the field. While we used data from one specific literature, we have no particular reason to believe that our findings are unique to this literature. Claims from highly cited observational studies have been shown to persist even after strong contradictory evidence from randomized trials (5). Since citation distortions may be driven, in part, by inflated claims in study abstracts, readers should not rely solely on the abstract of an article.
Acknowledgements
MRM is a member of the United Kingdom Centre for Tobacco and Alcohol Studies, a UKCRC Public Health Research: Centre of Excellence. Funding from British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged.
References
1. Greenhalgh T, Howick J, Maskrey N, Evidence Based Medicine Renaissance G. Evidence based medicine: a movement in crisis? BMJ. 2014; 348: g3725.
2. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. BMJ. 2009; 339: b2680.
3. Jauhar S, McKenna PJ, Radua J, Fung E, Salvador R, Laws KR. Cognitive-behavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. British Journal of Psychiatry. 2014; 204(1): 20-9.
4. Park IU, Peacey MW, Munafo MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. 2014; 506(7486): 93-6.
5. Tatsioni A, Bonitsis NG, Ioannidis JP. Persistence of contradicted claims in the literature. Journal of the American Medical Association. 2007; 298(21): 2517-26.
Conflict of Interest:
None declared
Submit response
Published August 19, 2014
