Abstract
Study objective
Publication bias compromises the validity of systematic reviews. This problem can be addressed in part through searching clinical trials registries to identify unpublished studies. This study aims to determine how often systematic reviews published in emergency medicine journals include clinical trials registry searches.
Methods
We identified all systematic reviews published in the 6 highest-impact emergency medicine journals between January 1 and December 31, 2013. Systematic reviews that assessed the effects of an intervention were further examined to determine whether the authors described searching a clinical trials registry and whether this search identified relevant unpublished studies.
Results
Of 191 articles identified through PubMed search, 80 were confirmed to be systematic reviews. Our sample consisted of 41 systematic reviews that assessed a specific intervention. Eight of these 41 (20%) searched a clinical trials registry. For 4 of these 8 reviews, the registry search identified at least 1 relevant unpublished study.
Conclusion
Systematic reviews published in emergency medicine journals do not routinely include searches of clinical trials registries. By helping authors identify unpublished trial data, the addition of registry searches may improve the validity of systematic reviews.
INTRODUCTION
Systematic reviews and meta-analyses have the potential to provide clinicians with a methodologically rigorous synthesis of all the available evidence on a particular topic. These methods are of particular importance to emergency physicians, who routinely encounter a breadth of clinical questions with limited time to find answers. These factors make it difficult to either memorize best practices or review all published studies during the course of clinical care. Because of the ongoing rapid expansion in emergency care research and because there are neither clinical guidelines nor a well-defined standard of care for many of the conditions that emergency physicians must manage, systematic reviews and meta-analyses often serve as the most appropriate source of information to guide decision making.
Although systematic reviews are often informative, they are susceptible to bias. In particular, publication bias occurs because of the nonpublication of negative, nonsignificant, or otherwise unfavorable results and is a major threat to the validity of systematic reviews. Several methods of assessing for publication bias are commonly used: contacting leading investigators in the field; reviewing conference proceedings, published abstracts, and bibliographies of included articles; constructing funnel plots; and analyzing with statistical techniques such as Egger’s test. Each of these methods, however, suffers from substantial limitations. For example, nonresponse limits the utility of contacting investigators, conference proceedings cannot identify data that have not yet been presented in poster or abstract format, and the risk of false-positive results and a lack of power for detecting bias in reviews with few included studies limits funnel plot utility.1–3 Clinical trials registries, on the other hand, were developed in part to address the problem of publication bias and may provide a more consistent method of identifying unpublished trials. However, the extent to which registry searches are used by emergency medicine researchers is unknown. Our objective was to determine how frequently systematic reviews published in emergency medicine journals assess for publication bias through the use of clinical trials registry searches.
MATERIALS AND METHODS
Study Design
We identified systematic reviews published between January 1, 2013, and December 31, 2013, in the 6 highest-impact-factor emergency medicine journals: Annals of Emergency Medicine, Academic Emergency Medicine, American Journal of Emergency Medicine, Injury, Journal of Trauma and Acute Care Surgery, and Resuscitation. Systematic reviews were identified by searching MEDLINE through PubMed, using the following search terms: (Meta-analysis[Publication Type] OR meta-analysis[Title/ Abstract] OR meta-analysis[MeSH Terms] OR review [Publication Type] OR search*[Title/Abstract]). This search strategy has been shown to be sensitive for the identification of systematic reviews published in general medical journals.4 We used the advanced search feature of PubMed to restrict the search to articles with print or online publication dates between January 1, 2013, and December 31, 2013, in the 6 aforementioned journals. One author (C.W.J.) subsequently reviewed the tables of contents from all issues published in 2013 for these 6 journals to identify systematic reviews missed by the PubMed search.
Two study authors (L.G.K. and C.W.J.) independently reviewed the full text of each article retrieved by the search strategy to determine whether it described results from a systematic review. We classified articles as systematic reviews according to previously developed criteria: the article included a clear statement identifying the topic of the review, authors provided a detailed description of the methods and data-sources used to identify evidence included in the review, the article included explicit inclusion and exclusion criteria, and the results included at least 1 study that met these inclusion criteria.4 We limited this analysis to systematic reviews that assessed interventions in humans to focus on reviews for which clinical trials registry searches should be most useful.
Outcome Measures
The main outcome of interest was whether each review included a search of at least 1 clinical trials registry. This outcome was independently assessed by 2 authors (L.G.K. and C.W.J.) for each included study. Acceptable registries included clinicaltrials.gov and other clinical trials registries meeting the World Health Organization Registry Criteria, version 2.1.5 When registry searches had been performed, we reviewed the text to determine whether the searches identified any relevant ongoing or completed but unpublished trials. If the results of the registry search were unavailable or ambiguous, we attempted to contact the corresponding author by e-mail for clarification of search results. We also determined whether review authors provided detailed search terms to allow replication of the clinical trials registry search. Other data points included funding source(s), stated compliance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement,6 and the number of studies included in each review. Discrepancies between authors about review classifications or the inclusion of clinical trials registry searches were resolved by consensus.
Primary Data Analysis
Interrater agreement for the primary outcome was calculated with Cohen’s κ. Statistical analyses were performed with PASW (version 18.0; IBM, Armonk, NY).
RESULTS
Our PubMed search retrieved 191 results, of which 111 were deemed not systematic reviews. Of the 80 remaining systematic reviews, 40 were determined to be interventional. The table of contents search identified 1 additional interventional review, yielding a final sample of 41 reviews (Appendix E1, available online at http://www.annemergmed.com). The majority of included reviews were published in Annals of Emergency Medicine, Resuscitation, and Injury (Table). A clinical trials registry search was performed for 8 (20%) of the interventional reviews. Four of these 8 (50%) identified 1 or more relevant unpublished trials as a result of the registry search. Of these 4, 2 reviews identified completed, unpublished studies through the trials registry search, and 3 reviews identified relevant ongoing studies. For 2 reviews, the specific terms used to perform the clinical trials registry search were provided. Of the 16 reviews that reported compliance with PRISMA, 2 included a registry search. Six of 25 reviews not reporting PRISMA compliance performed registry searches.
Table.
Characteristics of systematic reviews published in 6 emergency medicine journals from January 2013 through December 2013.
| Review Characteristics | Interventional Reviews (n=41) |
|---|---|
| Journal, n | |
| Academic Emergency Medicine | 3 |
| American Journal of Emergency Medicine | 3 |
| Annals of Emergency Medicine | 8 |
| Injury | 11 |
| Journal of Trauma and Acute Care Surgery | 3 |
| Resuscitation | 13 |
| Funding source, n* | |
| Industry | 1 |
| NIH/government | 6 |
| Other | 3 |
| None | 32 |
| Number of studies included in review, median (range)† | 16 (3–182) |
NIH, National Institutes of Health.
One review was performed with both NIH and industry support.
n=39. Two reviews did not report the total number of included studies.
Cohen’s κ coefficient for interrater reliability with respect to whether systematic reviews included clinical trials registry searches was 0.93, indicating excellent agreement.
LIMITATIONS
Our study has several limitations. It is possible that authors conducted a clinical trials registry search but did not describe this in the article if it was perceived to be unhelpful. We examined systematic reviews published in emergency medicine journals. Little is known about the use of clinical trials registries for systematic reviews published in other medical specialties or in general medical journals. Additionally, we limited this analysis to reviews published in 2013; recently conducted systematic reviews are more likely to benefit from including registry searches because the consistent registration of clinical trials only began in 2005. Registry use by authors of systematic reviews has almost certainly increased during the past decade and is likely to continue to increase.
DISCUSSION
Among systematic reviews of interventions published in emergency medicine journals in 2013, publication bias was not routinely addressed with clinical trials registry searches. For authors who did search clinical trial registries, approximately half identified relevant unpublished studies, indicating the potential benefit of this approach.
Clinical trials registry searches provide systematic review authors with the ability to directly identify relevant ongoing or completed but unpublished studies. The use of this technique offers several significant advantages compared with traditional methods of assessing for unpublished data. First, registry searches allow the identification of specific relevant studies compared with funnel plots and statistical methods of assessing for publication bias, which can provide insight only about the statistical likelihood that unpublished data may exist. Funnel plots may be unhelpful when reviews have few included studies; however, this limitation does not apply to clinical trials registry searches.7 Additionally, funnel plot asymmetry can occur for reasons other than publication bias, such as heterogeneity across studies of varying sample size (eg, smaller studies may study only sicker patients).8 Furthermore, even experts often disagree when interpreting funnel plots.3 Registry searches also have an advantage over reviewing reference lists from included studies or conference abstracts because these sources are likely subject to the same reporting biases as published articles.
Another important feature of registry use is that registry searches are particularly well suited to identifying large unpublished trials. Unfortunately, approximately 30% of large randomized trials are not published.9 Funnel plots are inherently unable to identify the presence of unpublished trials that are large relative to published studies. This is particularly concerning because the presence of large unpublished studies can significantly alter the interpretation of the effect estimate from available studies.10 Conversely, large trials, which are almost always intended for publication, are the most likely to be registered, thereby allowing clinical trials registries to address this important limitation of funnel plots.
Finally, clinical trials registry searches have the potential not only to help identify the existence of unpublished trials but also to facilitate access to results from these trials. Results are available in clinicaltrials.gov for an estimated 22% of large unpublished trials.9 Even for trials for which results are not available, registry entries typically include information about study size. Knowing the sample size of unpublished trials allows the authors of systematic reviews to more accurately estimate the potential influence of unpublished trials on the effect estimate from the published trials.
Incomplete or missing registry entries still constrain the utility of registries to serve as a comprehensive resource for identifying unpublished data. However, registration of clinical trials has increased rapidly during the past decade, and the amount and quality of information available in trials registries will likely continue to improve, rendering this concern less important over time.
Among interventional systematic reviews published recently in emergency medicine journals, clinical trials registry searches were not routinely used to assess for publication bias. However, when registry searches were performed they often provided additional information about the presence of unpublished data. The routine use of trials registry searches by systematic review authors may help to limit or define the presence of publication bias, thereby improving the validity of systematic reviews.
Supplementary Material
Editor’s Capsule Summary.
What is already known on this topic
Publication bias remains a major threat to the validity of systematic reviews and meta-analyses.
What question this study addressed
Whether authors of systematic reviews in emergency medicine reported that they checked trial registries for relevant unpublished trials.
What this study adds to our knowledge
Trial registries were checked in only 20% of the 41 reviews published in 2013.
How this is relevant to clinical practice
This finding should sensitize journal editors and reviewers to the importance of ensuring that systematic reviews have checked the appropriate registries for unpublished articles.
Acknowledgments
Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). The authors have stated that no such relationships exist. Dr. Jones is an investigator on an unrelated study sponsored by Roche Diagnostics, which provides funding to his department. Dr. Platts-Mills is supported by the National Institute on Aging through grant K23 AG038548.
Footnotes
Author contributions: All authors contributed to the study concept and design and to data analysis and interpretation. LGK and CWJ were responsible for data acquisition and drafting of the initial article; all authors contributed to subsequent revisions. CWJ takes responsibility for the paper as a whole.
Presented at the Society for Academic Emergency Medicine annual meeting, May 2014, Dallas, TX.
The National Institute on Aging had no role in the study design; collection, analysis, and interpretation of the data; writing of the article; or decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
References
- 1.Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53:1119–1129. doi: 10.1016/s0895-4356(00)00242-0. [DOI] [PubMed] [Google Scholar]
- 2.Lau J, Ioannidis JP, Terrin N, et al. The case of the misleading funnel plot. BMJ. 2006;333:597–600. doi: 10.1136/bmj.333.7568.597. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58:894–901.1. doi: 10.1016/j.jclinepi.2005.01.006. [DOI] [PubMed] [Google Scholar]
- 4.Montori VM, Wilczynski NL, Morgan D, et al. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330:68. doi: 10.1136/bmj.38336.804167.47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.International Clinical Trials Registry Platform. [Accessed October 15, 2014];WHO Registry Network. c2014 [cited June 11, 2014]. Updated April 2009. Available at: http://www.who.int/ictrp/network/en/
- 6.Moher D, Liberati A, Tetzlaff J, et al. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi: 10.1136/bmj.b2535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med. 2001;20:641–654. doi: 10.1002/sim.698. [DOI] [PubMed] [Google Scholar]
- 8.Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. doi: 10.1136/bmj.d4002. [DOI] [PubMed] [Google Scholar]
- 9.Jones CW, Handler L, Crowell KE, et al. Non-publication of large randomized clinical trials: cross sectional analysis. BMJ. 2013;347:f6104. doi: 10.1136/bmj.f6104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Jefferson TL, Jones MA, Doshi P, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2014;(4):CD008965. doi: 10.1002/14651858.CD008965.pub4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.