Abstract
Background
There is broad recognition of the importance of evidence in informing clinical decisions. When information from all studies included in a systematic review (“review”) does not contribute to a meta-analysis, decision-makers can be frustrated. Our objectives were to use the field of eyes and vision as a case study and examine the extent to which authors of Cochrane reviews conducted meta-analyses for their review’s pre-specified main outcome domain and the reasons that some otherwise eligible studies were not incorporated into meta-analyses.
Methods
We examined all completed systematic reviews published by Cochrane Eyes and Vision, as of August 11, 2017. We extracted information about each review’s outcomes and, using an algorithm, categorized one outcome as its “main” outcome. We calculated the percentage of included studies incorporated into meta-analyses for any outcome and for the main outcome. We examined reasons for non-inclusion of studies into the meta-analysis for the main outcome.
Results
We identified 175 completed reviews, of which 125 reviews included two or more studies. Across these 125 reviews, the median proportions of studies incorporated into at least one meta-analysis for any outcome and for the main outcome were 74% (interquartile range [IQR] 0–100%) and 28% (IQR 0–71%), respectively. Fifty-one reviews (41%) could not conduct a meta-analysis for the main outcome, mostly because fewer than two included studies measured the outcome (21/51 reviews) or the specific measurements for the outcome were inconsistent (16/51 reviews).
Conclusions
Outcome choice during systematic reviews can lead to few eligible studies included in meta-analyses. Core outcome sets and improved reporting of outcomes can help solve some of these problems.
Keywords: Systematic review, Outcomes, Meta-analysis, Core outcome sets, Loss of information, Clinical trials
Background
There is broad recognition of the importance of evidence in determining clinical decision-making [1]. For evidence-based healthcare, decision-makers (e.g., patients, clinicians, guideline developers) increasingly rely on systematic reviews (“reviews”) [1]. Reviews identify primary studies, such as clinical trials and observational studies, that have addressed the research question of interest. This research question typically defines the population, interventions, and comparators; these defined aspects in turn help delineate the primary studies eligible for the review.
Reviews may or may not include quantitative syntheses of data across studies (“meta-analyses”). When appropriately conducted, meta-analyses provide decision-makers with summary estimates (e.g., relative risks) and accompanying estimates of uncertainty (e.g., 95% confidence intervals) that convey information about treatment effectiveness or safety succinctly [2]. Often, however, meta-analyses cannot be conducted because the studies address somewhat different clinical questions, assess different outcomes than the systematic reviewer (“reviewer”) had pre-specified, are methodologically heterogeneous, or are poorly-reported (e.g., inadequate information about results). In these circumstances, a study may be eligible for the review, but may not contribute to a meta-analysis [3]. When a review includes multiple studies, but these studies cannot be included in the meta-analysis, both doers (i.e., reviewers) and users of reviews (i.e., decision-makers) can be frustrated. Decision-makers want to know how treatments compare quantitatively; they may not be able to get reliable information about this when only some included studies contribute data to the meta-analysis or when no meta-analysis is possible [4].
Outcomes are measures or events used to assess the effectiveness and/or safety of clinical interventions [5]. A frequent reason for non-conduct of meta-analyses is that the studies assess different outcomes or assess the same outcomes, but do so differently. These scenarios can occur even among high-quality studies.
Although outcomes are fundamental to reviews of interventions, outcomes are typically not considered when determining the eligibility of a primary study in such reviews [6]. This is because outcomes inform meta-analyses, not whether the primary study is eligible for the review. Consistent with guidance in the Cochrane Handbook for Systematic Reviews of Interventions [6], we believe that studies that address the population, interventions, and comparators of interest should be included and cataloged in systematic reviews even if they do not report outcomes of interest. Outcome choice in a review is crucial because: (1) outcomes serve as yardsticks for basing conclusions about treatments; and (2) which outcomes are chosen and how they are defined can impact how many meta-analyses can be done and how many studies can be included in them [7–11].
Outcomes may be assessed differently in different studies because an “outcome” (a seemingly monolithic entity) actually comprises five elements: domain, e.g., visual acuity; specific measurement, e.g., Snellen chart; specific metric, e.g., ≥3 lines of vision lost; method of aggregation, e.g., proportion; and time-points, e.g., 6 months [9, 12]. Another example of the application of this five-element framework to clearly specify a particular data point of interest related to the outcome of “anxiety” is mean (method of aggregation) change (specific metric) in anxiety (domain) measured through the Hamilton Anxiety Rating Scale (specific measurement) from baseline to 1 year (time-point) [9, 12].
We previously demonstrated, through case studies in the fields of eyes and vision [11] and HIV/AIDS [10], that reviewers and clinical trialists addressing the same research question often examine different outcomes. In addition, inconsistency in outcome reporting across eligible studies prevents incorporation of all eligible studies into meta-analyses. For instance, a 2017 Cochrane systematic review comparing non-steroidal anti-inflammatory drugs (NSAIDs) with corticosteroids for inflammation after cataract surgery [13] included 48 trials, none of which reported data for the review’s pre-specified primary outcome, “proportion of patients with intraocular inflammation at 1 week after surgery.”
To document the extent and determinants of this problem, we embarked on the current case study in the field of eyes and vision. Our objectives were to examine the extent to which Cochrane reviews in eyes and vision conducted meta-analyses for the main outcome domain and the reasons why some otherwise eligible studies were not incorporated into meta-analyses.
Methods
Reviews examined
We examined all completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews as of August 11, 2017. We excluded reviews that were still in the protocol stage.
Data extraction
We developed a data extraction form in the Systematic Review Data Repository (SRDR), an open-source platform for extracting and archiving data [14, 15]. Using a pilot-tested form, two individuals (from among SM, HK, BTS, and IJS) independently extracted data, resolving discrepancies through discussion. We extracted the following data: year published, population (i.e., eye function/region affected), and types of interventions and comparators. We extracted the numbers of primary, secondary, and other, i.e., non-primary and non-secondary, outcome domains. We also extracted the number of studies included in the review and in ≥1 meta-analysis for any, any primary, any secondary, and any other domain.
“Main” outcome domains
We categorized one domain from each review as its “main” outcome domain (Table 1). For reviews that named only one primary outcome domain, we categorized it as the main outcome domain; for reviews that named more than one primary outcome domain (or named more than one secondary outcome domain), we categorized the primary outcome domain (or secondary outcome domain) with the highest number of included studies as the main outcome domain. For reviews that did not name any primary or secondary outcome domains, we categorized the “other”, i.e., nonprimary and non-secondary, outcome domain with the highest number of included studies as the main outcome domain.
Table 1.
Algorithm for categorizing the “main” outcome domain for each systematic review
| Scenario | If | Then | Number of systematic reviews (N = 175) |
|---|---|---|---|
| n (%) | |||
| 1 | The review named only 1 primary outcome domain | we categorized that outcome domain as the main outcome domain. | 131 (75) |
| 2 | The review named >1 primary outcome domain | we categorized the primary outcome domain with the highest number of included studies as the main outcome domain. | 41 (23) |
| 3 | The review did not name any primary outcome domain, but named ≥1 secondary outcome domain | we categorized the secondary outcome domain with the highest number of included studies as the main outcome domain. | 0 (0) |
| 4 | If the review did not name any primary or secondary outcome domains | we categorized the “other” (i.e., non-primary and non-secondary) outcome domain with the highest number of included studies as the main outcome domain. | 3 (2) |
Note
In scenarios 2, 3, and 4, if there were two or more possible outcome domains that had the same number of included studies (“Then” column), we categorized the first outcome listed in the Methods section as the main outcome domain
For each main outcome domain, we extracted the other four elements specified: specific measurement, specific metric, method of aggregation, and time-points. For the main outcome domain, we also extracted the numbers of studies that reported measuring it, reported any data, reported any meta-analyzable data, and were incorporated into ≥ 1 meta-analysis. We considered data for a given outcome from a given study to be “meta-analyzable” if the study reported adequate information so that it could be incorporated into a meta-analysis. For categorical outcomes, meta-analyzable meant that either of these conditions were met: (1) total number of participants and number of participants with the outcome were reported for each study arm; and (2) the between-group treatment effect (e.g., relative risk) and an uncertainty estimate (e.g., 95% confidence interval) were reported. For continuous and time-to-event outcomes, meta-analyzable meant that either of these conditions were met: (1) mean and uncertainty estimates were reported for each study arm; and (2) the between-group treatment effect (e.g., mean difference) and an uncertainty estimate were reported.
Results
Reviews examined
We identified 175 completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews (Table 2). The reviews were published between January 1, 2005 and August 11, 2017 (median = 2014). The most common populations were patients with retinal/choroidal disease (35 reviews; 20%) and visual impairment/low vision (33 reviews; 19%). The most common types of interventions/comparators were drugs (74 reviews; 42%) and surgeries (67 reviews; 38%).
Table 2.
Characteristics of systematic reviews examined
| Characteristic | Number of systematic reviews (N = 175) |
|---|---|
| n (%) | |
| Year published | |
| 2003–2005 | 3 (2) |
| 2006–2008 | 12 (7) |
| 2009–2011 | 15 (9) |
| 2012–2014 | 68 (39) |
| 2015–2017 | 77 (44) |
| Population (function/region of eye) addressed | |
| Retinal/choroidal disease | 35 (20) |
| Visual impairment/low vision | 33 (19) |
| Optic nerve, including glaucoma | 32 (18) |
| Ocular surface | 31 (18) |
| Lens | 18 (10) |
| Ocular vasculature | 5 (3) |
| Other | 21 (12) |
| Interventions and comparators examineda | |
| Drug | 74 (42) |
| Surgery | 67 (38) |
| Other procedure | 31 (18) |
| Device | 15 (9) |
| Supplements | 6 (3) |
| Screening/testing | 5 (3) |
| Other intervention | 26 (15) |
| Number of outcome domains examined | |
| Median | 6 |
| Interquartile range | 5 to 8 |
| Range | 1 to 19 |
| Number of primary outcome domains examined | |
| Median | 1 |
| Interquartile range | 1 to 1 |
| Range | 0 to 5 |
| Number of secondary outcome domain examined | |
| Median | 4 |
| Interquartile range | 3 to 6 |
| Range | 0 to 12 |
| Number of other outcome domains examined | |
| Median | 1 |
| Interquartile range | 0 to 2 |
| Range | 0 to 6 |
| Number of studies included | |
| Median | 3 |
| Interquartile range | 1 to 9 |
| Range | 0 to 137 |
aMore than one category could apply
Incorporation of studies into meta-analyses for any outcome domain
The 175 included reviews examined a median of 6 total outcome domains, including a median of 1 primary outcome domain, 4 secondary outcome domains, and 1 other outcome domain.
The 175 reviews included a median of 3 studies (IQR 1–9); 125 reviews (71%) included ≥2 studies. For these 125 reviews, Fig. 2 plots the percentage of studies incorporated into a meta-analysis for any outcome domain (blue line) and for the main outcome domain (red bars). Among these reviews, 44/125 reviews (35%) incorporated every included study into ≥1 meta-analysis (for any outcome domain). Conversely, 33/125 reviews (26%) did not incorporate any study into any meta-analysis for any outcome, i.e., they did not conduct any meta-analysis. The remaining 48/125 reviews (38%) incorporated only a subset of their studies into ≥1 meta-analysis. These 48 reviews included a median of 12.5 studies (IQR 6–22), and the meta-analyses in these reviews incorporated a median of 6.5 studies (IQR 4–13).
Fig. 2.
Percentage of studies included in the review that were incorporated into a meta-analysis for any outcome (blue line) and for the review’s main outcome (red bars)
Notes: This Figure excludes the 50 systematic reviews in whom a meta-analysis was not possible: 35 systematic reviews that each included 0 studies (i.e., “empty reviews”) and 15 systematic reviews included that each included only 1 study. When the blue line is non-0 but the red bars are 0, it implies that the systematic review did not conduct a meta-analysis for the main outcome, but did so for ≥1 of the remaining outcomes
Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 74% (IQR 0–100%). Among the 92 reviews that conducted a meta-analysis, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 93% (IQR 64–100%).
Characteristics of main outcome domains
Almost all reviews (172/175 reviews; 98%) named ≥1 primary outcome domain (Table 1). Three in four reviews (131/175 reviews; 75%) each named exactly one primary outcome domain, which we categorized as their main outcome domain. The most frequent main outcome domains across the 175 reviews were visual acuity (31%) and intraocular pressure (6%) (Table 3). Thirty-eight outcome domains were main outcome domains in just one review each. The main outcome was categorical in 70% and continuous in 29% of reviews. Most main outcome domains (98%) were efficacy outcomes, i.e., not safety outcomes.
Table 3.
Characteristics of main outcome domains in all 175 systematic reviews examined
| Characteristic | Number of systematic reviews (N = 175) n (%) |
|---|---|
| Main outcome domain | |
| Visual acuity | 55 (31) |
| Intraocular pressure | 11 (6) |
| Visual field | 7 (4) |
| Visual impairment/vision loss | 5 (3) |
| Success of surgery/procedure | 5 (3) |
| Failure of trabeculectomy | 4 (2) |
| Progression of age-related macular degeneration | 3 (2) |
| Reading speed | 3 (2) |
| Ocular symptoms (unspecified) | 3 (2) |
| Symptoms of dry eye | 3 (2) |
| Vision-related quality of life | 3 (2) |
| Resolution of infection | 3 (2) |
| Active trachoma | 3 (2) |
| Healing of keratitis | 3 (2) |
| Other | 64 (37) |
| Type of main outcome domain | |
| Categorical | 122 (70) |
| Continuous | 50 (29) |
| Other (i.e., time-to-event) | 2 (1) |
| Not reported | 1 (0) |
| Goal of main outcome domain | |
| Efficacy | 172 (98) |
| Safety | 3 (2) |
Incorporation of studies into meta-analyses for the main outcome domain
Among the 125 reviews including ≥2 studies, only 18 reviews (14%) incorporated all their studies into a meta-analysis for the main outcome domain. Conversely, 51/125 reviews (41%) did not incorporate any study into the meta-analysis for the main outcome domain, i.e., they did not conduct any meta-analysis for the main outcome domain. The remaining 56/125 reviews (45%) incorporated only a subset of their studies into the meta-analysis for the main outcome domain. These 56 reviews included a median of 12 studies each, and the meta-analyses for the main outcome domain in these reviews incorporated a median of 4 studies each.
Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for the main outcome domain was 28% (IQR 0–71%). Among the 74 reviews that conducted meta-analyses for the main outcome domain, the median proportion of studies incorporated was 67% (IQR 39–91%).
Meta-analysis conduct for the main outcome domain
Figure 1 illustrates a cascading effect of loss of information as regards the main outcome domain in the 175 reviews. Thirty-five reviews (20%) included no studies, i.e., were empty reviews, and 15 (9%) included one study each (Fig. 1). Of the 125 reviews including ≥2 studies, i.e., those in which a meta-analysis could theoretically be done for the main outcome if ≥2 studies reported meta-analyzable data, only 74 reviews (59%) conducted a meta-analysis for the main outcome.
Fig. 1.
Conduct of meta-analyses for the main outcome domain
Reasons for non-conduct of meta-analyses for the main outcome domain
Among the 125 reviews including ≥2 studies, 51 reviews (41%) did not conduct a meta-analysis for the main outcome domain. For 21/51 reviews (41%), fewer than two studies measured the review’s main outcome (Table 4). When ≥2 studies reported meta-analyzable data, there were numerous reasons why reviewers did not conduct a meta-analysis, most frequently due to inconsistency in outcome elements among the included studies. Specifically, data could not be meta-analyzed because the specific measurements used (16/51 reviews; 31%) and time-points examined (9/51 reviews; 18%) were inconsistent among studies.
Table 4.
Reasons for non-conduct of a meta-analysis for the systematic review’s main outcome even when ≥2 studies were included in the systematic review (N = 51 of 125 reviews that included ≥2 studies)
| Reason | Number of systematic reviews (N = 51) | |
|---|---|---|
| n (%) | ||
| When meta-analyzable data1 for the review’s main outcome domain were NOT REPORTED by ≥2 studies (n = 23 reviews) | ||
| < 2 studies measured the review’s main outcome | 21 | (41) |
| < 2 studies reported any data for the review’s main outcome | 1 | (2) |
| < 2 studies reported any meta-analyzable data1for the review’s main outcome | 1 | (2) |
| When meta-analyzable data1 for the review’s main outcome domain were REPORTED by ≥2 studies (n = 28 reviews)2 | ||
| Reasons related to inconsistencies in outcome elements | ||
| Studies used inconsistent specific measurements | 16 | (31) |
| Studies used inconsistent specific metrics | 0 | (0) |
| Studies used inconsistent methods of aggregation | 0 | (0) |
| Studies reported data at inconsistent time-points | 9 | (18) |
| Reasons related to heterogeneity | ||
| Studies were clinically heterogeneous | 7 | (14) |
| Studies were methodologically heterogeneous | 2 | (4) |
| Studies were statistically heterogeneous | 0 | (0) |
1For categorical outcomes, we considered data to be meta-analyzable if either of the following scenarios were met [1]: total number of participants and number of participants with the outcome of interest were reported for each study arm; and [2] the between-group treatment effect (e.g., relative risk, odds ratio) and an estimate of uncertainty (e.g., 95% confidence interval) were reported. For continuous and time-to-event outcomes, we considered data to be meta-analyzable if either of the following scenarios were met [1]: mean and estimate of uncertainty (e.g., standard deviation) were reported for each study arm; and [2] the between-group treatment effect (e.g., mean difference) and an estimate of uncertainty (e.g., 95% confidence interval) were reported
2More than one reason could apply
Figure 2 demonstrates that the loss of information for the main outcome domain (red bars) was similar in pattern to the loss of information when considering any outcome domain (blue line).
Discussion
Through a case study of all Cochrane reviews in the field of eyes and vision, the current work demonstrates three major areas that need improvement.
First, primary studies addressing similar research questions should align their outcomes better. Studies often could not be incorporated into meta-analyses because the outcomes were not aligned, either because the domains or ≥1 of the other four outcome elements did not overlap. Among the reviews including ≥2 studies, only 59 and 74% could conduct a meta-analysis for the main outcome and for any outcome, respectively. In other words, even when reviews included ≥2 studies, 41 and 26% of reviews missed opportunities to conduct a meta-analysis to succinctly convey information regarding the main outcome and any outcome, respectively.
Second, reviews and primary studies should align their outcomes better. When looking at reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median percentages of included studies incorporated into the meta-analysis for the main outcome and for any outcome were 28 and 74%, respectively. This suggests that, approximately 7 in 10 studies that reviewers include are not incorporated into the meta-analysis for the main outcome, and 1 in 4 studies are not incorporated into the meta-analysis for any outcome. In previous work, we demonstrated poor overlap between outcomes in clinical trials and reviews, and possible differences in the types of outcomes they examine [10, 11]. For HIV/AIDS, we demonstrated that reviewers examined more long-term clinical outcomes and patient-centered outcomes than did clinical trialists. Such differences may arise because: (1) reviews may more directly inform clinical practice guidelines, and (2) reviewers may be less affected by common constraints faced by clinical trialists, e.g., costs and sample size [10].
Our findings beg the question of who should prioritize outcomes for measurement and reporting in research. It has aptly been stated that achieving consensus in outcome use across research “cannot be left to serendipity.” [16] One deliberate and fundamental aspect of the solution to the problem of outcome inconsistency is the development of “core outcome sets.” A core outcome set is a minimum set of outcomes that should be measured and reported in all clinical trials addressing a given condition [17]. Core outcome sets are increasingly common in various health fields; a 2018 systematic review identified 307 core outcomes sets [18]. However, outcome inconsistency remains widespread; 40% of recent (2019) published Cochrane reviews explicitly noted this problem [19].
We [10, 11] and others [20] have argued that, as stakeholders in a given field, systematic reviewers should both participate in the development of and adopt core outcome sets for that field. By broadening the participation in outcome prioritization efforts, this could potentially help ensure that the outcomes that are measured and reported in research are widely relevant and important. Two aspects of core outcome sets are worthy of clarification. First, core outcome sets do not stifle innovation; they are simply meant to represent a minimum set of outcomes that should be reported. Once a core outcome set exists for a given topic, clinical trialists working in that topic area should explicitly specify the intention to measure and report the outcomes in the set. Second, core outcome sets are not static; they can and should be updated as the field advances and new knowledge emerges.
The third major area in need of improvement that our study demonstrates is the reporting of outcomes in primary studies. Results data from primary studies were often not meta-analyzable even when outcomes might have been aligned. In addition, outcome domains were frequently not reported in primary studies or ≥1 of the outcome elements were frequently missing or inadequately reported (e.g., “worsening of disease” without clarification of how “worsening” was defined). It is possible that the studies measured these outcomes, but did not report measuring them or reported them inadequately. If such selective reporting, either non-reporting or inadequate reporting, of outcomes in the included studies occurred as a function of the direction of the outcome’s results, it would be suggestive of outcome reporting bias [21]. In this case study, we relied on the reviewers’ reporting of the extent to which the primary studies reported the outcomes. Because we did not examine the reports of the primary studies (or their protocols), we are unable to comment definitively on whether non-reporting of the outcomes indicates outcome reporting bias. However, outcome reporting bias in primary studies has been documented to be a widespread problem across reviews [22–26], and, as such, is a likely explanation for some outcomes not being reported.
Implications
For the evidence-based medicine paradigm to work, decision-makers must be able to rely on systematic reviews, which in turn rely on the results of primary studies. For results of primary studies to be actionable, there (1) needs to be alignment in outcomes considered important to both primary study researchers and reviewers, and (2) those outcomes need to be reported completely. Important discussions need to be had regarding who should choose outcomes for the field and how such choices should be made. We, in conjunction with others, suggest that these discussions should include, at the least, clinicians, patients, clinical trialists, systematic reviewers, regulators, and other decision-makers [27].
We have demonstrated that the choice of outcomes for systematic reviews may have led to loss of information through non-incorporation of results from included studies into meta-analyses. The most substantial drops in the percentage of reviews conducting meta-analyses for the main outcome domain appeared to be due to inadequate numbers of studies reporting the outcome and, when there were adequate numbers of studies for a meta-analysis (i.e., ≥2 studies), differences in the specific measurements and time-points used.
Our findings also demonstrate that even when focusing on reviews that conducted meta-analyses for their main outcome domain, only about 2 in 3 studies were incorporated into those meta-analyses. As such, non-incorporation of included studies into meta-analyses represents two main problems. First, it represents missed opportunities for using research to inform decision-making through evidence synthesis. This contributes considerably towards research waste [28–30]. Second, non-incorporation of included studies into meta-analyses represents a failed obligation on the part of the researchers (both trialists and reviewers) [31]. As a community of researchers, both parties have a solemn obligation to research participants to ensure that their participation will lead to a useful contribution to science; failing to agree upon outcomes that should be collected and adequately reported likely violates this obligation.
Other solutions
Core outcome sets are integral to solving the problems this study illustrates. Other parts of the solution are worth discussing. We agree with existing recommendations against studies being excluded from systematic reviews solely on the basis of the lack of relevant outcome data [3]. Thankfully, such recommendations have been associated with a reduction in the number of reviews excluding studies solely on the basis of outcome data [32]. As the current study demonstrates, the review team’s choice of outcomes may not align with that of the primary studies. This may be particularly true for eyes and vision, a field with few core outcome sets [4, 18]. We also encourage reviewers to report an outcome matrix [23, 24], a transparent and simple way to indicate all fully-reported, partially-reported, or non-reported outcomes in each included study.
Large numbers of empty reviews and reviews including only one study
Twenty-percent of the reviews we examined were empty and 9% included only one study each. While such reviews are useful in driving primary research, the possible reasons for the paucity of studies in them are worth exploring. One possibility is that these represent topics that primary researchers have not yet studied. Another is that only observational studies addressing these topics may exist; Cochrane reviews typically include only randomized trials. It also is possible that these topics reflect the priorities of Cochrane Eyes and Vision and the authors of these reviews, rather than of the field at-large.
Limitations
Our study has certain limitations. First, we focused on Cochrane reviews within one field. Loss of information due to the choice of review outcomes could be a bigger, similar, or smaller problem in non-Cochrane systematic reviews in eyes and vision or systematic reviews in other fields. Second, we analyzed in-depth the extent of incorporation of included studies into meta-analyses only for the main outcome domain. Meta-analyses of other primary, secondary, and other outcome domains may have incorporated higher percentages of included studies. However, Fig. 2 suggests that this is likely not the case. It is possible that our algorithm for categorizing the “main” outcome for each review could have impacted our findings. But, in reviews where more than one outcome domain could have served as the main outcome, we categorized as the main outcome the outcome that the highest number of included studies had reported. Our results thus represent the best-case scenario. Third, most outcome domains (98%) were efficacy outcomes. Selective outcome reporting has also been reported to be a problem for safety outcomes [33]. Fourth, we relied on the reviews to determine whether or not each included study did the following for the main outcome domain: reported measuring it, reported any results for it, and reported meta-analyzable data for it. Related to this, we did not examine the appropriateness or feasibility of the reviewers’ being able to conduct meta-analyses when the included studies reported data in a format different from what the reviewers were interested. As such, our results document what was actually done in the reviews.
Conclusions
This case study of all Cochrane systematic reviews addressing an entire field (eyes and vision) demonstrates that only 59 and 74% of the reviews including ≥2 studies could conduct a meta-analysis for the main outcome and for any outcome, respectively. In evidence-based healthcare, such loss of information represents missed opportunities and a failed obligation by researchers to research participants to ensure that their participation will lead to a useful contribution to science. Core outcome sets and improved outcome reporting can help solve some of these problems.
Acknowledgements
The authors acknowledge Elizabeth Clearfield and Dr. Roy Chuck for their contributions to the concepts presented in this work.
Abbreviations
- IQR
Interquartile range
- NSAID
Non-steroidal anti-inflammatory drug
- SRDR
Systematic Review Data Repository
Authors’ contributions
IJS, KL, and KD conceptualized and designed the study. IJS and KD analyzed and interpreted the data. IJS drafted the manuscript. All authors provided critical feedback to the manuscript. All authors read and approved the final manuscript.
Funding
The National Institutes of Health (Grant Number UG EY020522) funded this work. The funder played no role in the design of the study; collection, analysis, and interpretation of the data, or writing of the manuscript.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
Drs. Saldanha and Dickersin and Ms. Lindsley and Money declare affiliation with Cochrane Eyes and Vision during conduct of the work related to this manuscript. Ms. Kimmel and Mr. Smith were research assistants on this study. All of the reviews examined in this manuscript were produced by Cochrane Eyes and Vision. No other disclosures are reported.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Contributor Information
Ian J. Saldanha, Email: ian_saldanha@brown.edu
Kristina B. Lindsley, Email: K.B.Lindsley@umcutrecht.nl
Sarah Money, Email: sarah.e.money@gmail.com.
Hannah J. Kimmel, Email: hannah_kimmel@brown.edu
Bryant T. Smith, Email: bryant_smith@brown.edu
Kay Dickersin, Email: kdicker3@jhu.edu.
References
- 1.Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011. [PubMed]
- 2.Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111. doi: 10.1002/jrsm.12. [DOI] [PubMed] [Google Scholar]
- 3.Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. Version 1.02 ed. London: Cochrane; 2016. [Google Scholar]
- 4.Saldanha IJ, Le JT, Solomon SD, Repka MX, Akpek EK, Li T. Choosing Core outcomes for use in clinical trials in ophthalmology: perspectives from three ophthalmology outcomes working groups. Ophthalmology. 2019;126(1):6–9. doi: 10.1016/j.ophtha.2018.09.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Meinert CL. Clinical trials dictionary: terminology and usage recommendation. 2nd ed Hoboken, NJ: Wiley; 2012.
- 6.Higgins Julian P.T., Thomas James, Chandler Jacqueline, Cumpston Miranda, Li Tianjing, Page Matthew J., Welch Vivian A., editors. Cochrane Handbook for Systematic Reviews of Interventions. 2019. [Google Scholar]
- 7.Mayo-Wilson Evan, Li Tianjing, Fusco Nicole, Bertizzolo Lorenzo, Canner Joseph K., Cowley Terrie, Doshi Peter, Ehmsen Jeffrey, Gresham Gillian, Guo Nan, Haythornthwaite Jennifer A., Heyward James, Hong Hwanhee, Pham Diana, Payne Jennifer L., Rosman Lori, Stuart Elizabeth A., Suarez-Cuervo Catalina, Tolbert Elizabeth, Twose Claire, Vedula Swaroop, Dickersin Kay. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. Journal of Clinical Epidemiology. 2017;91:95–110. doi: 10.1016/j.jclinepi.2017.07.014. [DOI] [PubMed] [Google Scholar]
- 8.Mayo-Wilson Evan, Li Tianjing, Fusco Nicole, Dickersin Kay. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study) Research Synthesis Methods. 2017;9(1):2–12. doi: 10.1002/jrsm.1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400. doi: 10.1371/journal.pone.0109400. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;84:85–94. doi: 10.1016/j.jclinepi.2017.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Saldanha IJ, Lindsley K, Do DV, et al. Comparison of clinical trial and systematic review outcomes for the 4 Most prevalent eye diseases. JAMA Ophthalmology. 2017;135(9):933–940. doi: 10.1001/jamaophthalmol.2017.2583. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Zarin DA, Tse T, Williams RJ, Califf RM, ide NC. The ClinicalTrials.gov results database--update and key issues. N Engl J Med. 2011;364(9):852–860. doi: 10.1056/NEJMsa1012065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Juthani VV, Clearfield E, Chuck RS. Non-steroidal anti-inflammatory drugs versus corticosteroids for controlling inflammation after uncomplicated cataract surgery. Cochrane Database Syst Rev. 2017;7:Cd010516. doi: 10.1002/14651858.CD010516.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ip S, Hadar N, Keefe S, et al. A web-based archive of systematic review data. Syst Rev. 2012;1:15. doi: 10.1186/2046-4053-1-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Li T, Vedula SS, Hadar N, Parkin C, Lau J, Dickersin K. Innovations in data collection, management, and archiving for systematic reviews. Ann Intern Med. 2015;162(4):287–294. doi: 10.7326/M14-1603. [DOI] [PubMed] [Google Scholar]
- 16.Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8:39. doi: 10.1186/1745-6215-8-39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132. doi: 10.1186/1745-6215-13-132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. PLoS One. 2018;13(12):e0209869. doi: 10.1371/journal.pone.0209869. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Williamson PR, Oliveira RD, Clarke M, et al. The relevance and uptake of core outcome sets in Cochrane systematic reviews Submitted.
- 20.Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5:11. doi: 10.1186/s13643-016-0188-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–740. doi: 10.1503/cmaj.1041086. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Dwan Kerry, Kirkham Jamie J, Williamson Paula R, Gamble Carrol. Selective reporting of outcomes in randomised controlled trials in systematic reviews of cystic fibrosis. BMJ Open. 2013;3(6):e002709. doi: 10.1136/bmjopen-2013-002709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Kirkham JJ, Altman DG, Chan AW, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ. 2018;362:k3802. doi: 10.1136/bmj.k3802. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. doi: 10.1136/bmj.c365. [DOI] [PubMed] [Google Scholar]
- 25.Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179–2195. doi: 10.1002/sim.5356. [DOI] [PubMed] [Google Scholar]
- 26.Page MJ, McKenzie JE, Kirkham J, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;(10):Mr000035. [DOI] [PMC free article] [PubMed]
- 27.Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280. doi: 10.1186/s13063-017-1978-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet (London, England) 2014;383(9912):156–165. doi: 10.1016/S0140-6736(13)62229-1. [DOI] [PubMed] [Google Scholar]
- 29.Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014;9(7):e102670. doi: 10.1371/journal.pone.0102670. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Macleod MR, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet (London, England) 2014;383(9912):101–104. doi: 10.1016/S0140-6736(13)62329-6. [DOI] [PubMed] [Google Scholar]
- 31.Law A, Lindsley K, Rouse B, Wormald R, Dickersin K, Li T. Missed opportunity from randomised controlled trials of medical interventions for open-angle glaucoma. Br J Ophthalmol. 2017;101(10):1315–1317. doi: 10.1136/bjophthalmol-2016-309695. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Dwan KM, Williamson PR, Kirkham JJ. Do systematic reviews still exclude studies with “no relevant outcome data”? BMJ. 2017;358:j3919. doi: 10.1136/bmj.j3919. [DOI] [PubMed] [Google Scholar]
- 33.Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501. doi: 10.1136/bmj.g6501. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.