BACKGROUND
Systematic reviews are an increasingly important part of the clinical evidence base. Well-conducted reviews provide solid evidence for clinical decision-making, though there are a number of important biases that can impact on the validity of study conclusions. One of the quality markers for systematic reviews is the thoroughness of the search. Our study’s purpose was to assess the frequency that systematic reviews are limited to studies published in English.
METHODS
We searched in PUBMED using the search term “systematic review [pt]” through December 15, 2018. We reviewed the first 250 systematic reviews for the presence of any language restriction and the number of non-English articles included or excluded. This review was done in duplicate with disagreement reached by consensus.
RESULTS
Among 250 systematic reviews, we found that 84 (34%) explicitly excluded non-English articles, 80 (32%) had no statement about excluding non-English trials, though none included non-English articles, and 86 (34%) indicated that they did their search without language restriction. Of the 86 reviews without language restrictions, only 19 (22%) included non-English trials, representing 2% of the total articles included in these studies.
DISCUSSION
A study that examined systematic reviews through 1993 found that most had language restrictions with 93% excluding at least one non-English randomized controlled trial.1 This exclusion of non-English studies is labeled “English-language bias” or “Tower of Babel bias.” We found that in the subsequent two decades, limiting searches to English-only is still common, with only 34% reporting having no language restrictions and few reviews including non-English trials. While 86% of journals are published in English, and the likelihood of citation is higher for English-language articles,2 this does not imply that non-English trials are lower quality. One 1995 study found no difference in quality design elements between English and other language trials,3 though this study was limited to European language articles. Studies examining the impact of including non-English trials on review estimates of effect have been mixed.4,5 Articles published in German are less likely to be significant than those published in English.6 If this is true for other languages, there is a potential for bias in outcomes among reviews that are limited to English-only. Including studies published in non-English languages may have increased resource challenges with respect to costs, time, and expertise in non-English languages; however, inclusion will help ensure generalizability and reduce the risk of bias.
CONCLUSIONS
Limiting systematic reviews to English-only is common in systematic reviews and could result in biased estimates of effect and reduce generalizability. Important research questions for those conducting systematic reviews are whether online translation programs, such as Google Translate, are sufficiently accurate to allow the translation of non-English trials and whether such trials are of comparable quality to those published in English.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they have no conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Contributor Information
Jeffrey L. Jackson, Email: jjackson@mcw.edu.
Akira Kuriyama, Email: akira.kuriyama.jpn@gmail.com.
References
- 1.Gregoire G, Derderian F, Le LJ. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol. 1995;48(1):159–163. doi: 10.1016/0895-4356(94)00098-B. [DOI] [PubMed] [Google Scholar]
- 2.Winkmann G, Schlutius S, Schweim HG. Citation rates of medical German-language journals in English-language papers - do they correlate with the impact factor, and who cites? Dtsch Med Wochenschr. 2002;127(4):138–143. doi: 10.1055/s-2002-19716. [DOI] [PubMed] [Google Scholar]
- 3.Moher D, Fortin P, Jadad AR, et al. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet. 1996;347(8998):363–366. doi: 10.1016/S0140-6736(96)90538-3. [DOI] [PubMed] [Google Scholar]
- 4.Juni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002;31(1):115–123. doi: 10.1093/ije/31.1.115. [DOI] [PubMed] [Google Scholar]
- 5.Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64. doi: 10.1186/s12874-017-0347-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326–329. doi: 10.1016/S0140-6736(97)02419-7. [DOI] [PubMed] [Google Scholar]