21st Century Evidence: Randomized Controlled Trials Versus Systematic Reviews and Meta-Analyses

In this issue of Critical Care Medicine, there are two systematic review and meta-analyses (SRMAs) assessing IV vitamin C therapy in patients with sepsis. Although most readers are likely familiar with using randomized clinical trials (RCTs) in clinical decisions, some may be less familiar with using SRMAs. In this foreword, we will review benefits and downsides of SRMAs and address the value of SRMAs in relation to RCTs and their respective roles in clinical decision-making.

An SRMA is an original, reproducible scientific work that answers a focused research question both qualitatively (evidence evaluation and distillation through systematic review) and quantitatively (meta-analysis). SRMAs are informed by a careful literature search in multiple databases such as Medline, Embase, and the Cochrane database, and often include evaluation of unpublished sources, such as trial registries. The investigators will assess studies for clinical heterogeneity (as there are almost always some differences between trials). In the absence of substantial heterogeneity, a meta-analysis is performed to generate summary- effect estimates for the outcomes of interest. At this point, investigators will evaluate for statistical heterogeneity both qualitatively (visually via a Forrest plot) and quantitatively (commonly with Cochran Q test and/or I² statistic). If there is important statistical heterogeneity, subgroup analysis can be performed to help explain the inconsistency in findings. In addition, included studies are evaluated for risk of bias, based on methodology, selective outcome reporting bias, and publication bias. A properly done SRMA will follow standardized reporting guidelines, including The Preferred Reporting Items for Systematic Reviews and Meta-Analyses, which includes a checklist for reporting results. Ideally, an SRMA should be formally registered prior to initiation to ensure important decisions are made a priori (1, 2).

As part of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, clinical practice guidelines and resultant recommendations must be informed by a systematic review (and meta-analysis if possible). Pooled effect estimates from an SRMA are evaluated using GRADE to ascribe the certainty of evidence as high, moderate, low, or very low. Using this summarized data evaluating benefits, harms, and certainty of evidence, guidelines panels decide on recommendations—strong or conditional, for or against an intervention of interest.

Outside of guidelines, most clinicians are comfortable using results from clinical trials to inform their practice. Considered the gold standard for clinical investigation, RCTs offer the ability to directly compare two (or more) groups, whereas randomization ensures balance of both known and unknown confounders between groups. It can be argued that well-done RCTs offer the most trustworthy information for the target population from which the study sample was drawn. However, RCTs may not always be generalizable to patients who are different from the study sample (such as those often excluded from RCTs—severely critically ill, elderly, and pregnant patients). Pragmatic trials with fewer exclusion criteria are more likely to be generalizable.

By combining multiple RCTs into a single pooled effect estimate, meta-analyses offer a larger sample size and, therefore, greater precision in detecting clinically meaningful effects (3). Other advantages of meta-analyses include the ability to evaluate and compare multiple interventions not directly studied in comparison (via a technique called network meta-analyses), improved external validity by including trials with more diverse populations, and an ability to examine between-study subgroups for factors that may influence treatment effects (even more relevant for individual patient data meta-analyses) (3). Some methodologists consider findings from a well-done SRMA of RCTs to be the highest level of evidence.

Despite the advantages of SRMAs, some clinicians may find it difficult to use SRMAs as compared with RCTs to inform clinical practice. There is always a degree of clinical heterogeneity between the studies included in SRMAs, and detractors often describe SRMAs as “combining apples and oranges.” For practicing clinicians, the further the data are from the patient, the less they might trust the data and more likely to consider that their patient is the exception to the rule. This is further complicated when the results of SRMAs are not concordant with the largest or the most recent RCT examining the same clinical question.

If the summary estimate of the SRMA matches with the largest and most trustworthy of the included RCTs, then the clinician does not need to decide which of the two to follow. However, discordance between an SRMA and the largest or most trustworthy RCT is also possible (4). If the SRMA results differ from largest or most reliable RCT, this may be a situation where one should consider following the RCT, particularly if the SRMA includes small trials at high risk for bias. Additionally, meta-analyses on the same topic may not agree, with a recent example being the role of corticosteroids in sepsis and septic shock, where two meta-analyses found disparate results (5, 6). In this case, the methodological quality of the SRMAs should be assessed—using a tool such as A Measurement Tool to Assess Systematic Reviews (7, 8). If both SRMAs are well done, it is important to examine carefully the included studies and their details to understand why differing results were obtained.

Ultimately, SRMAs are only as good as the studies they include. The discerning clinician should review SRMAs with the same careful lens that they use to appraise traditional RCT (9) and be aware that the quality and reproducibility of each study included in the SRMA will influence the results of the SRMA. If the SRMA meets those exacting criteria, we would suggest that a clinician should be comfortable using the results of a well-done SRMA as well as a large RCT to inform clinical care.