Meta-analyses: How to critically appraise them?

In this issue of the Journal, Green et al¹ used meta-analysis to test the hypothesis that normal myocardial perfusion imaging (MPI) study and coronary computed tomography angiography (CCTA) study were able to identify patients at low risk for cardiac events at long-term follow-up. The authors concluded that both modalities have high long-term “warranty” for adverse cardiac events.

Meta-analyses are a great resource for summarizing evidence for clinical decision-making, if conducted in an unbiased fashion with use of rigorous methodology.² Given an exponential increase in the number of publications on meta-analyses and the fact that they are vulnerable to biases, it is important to critically appraise them prior to using the results in clinical practice.² In this editorial, we will focus on ‘how to appraise meta-analyses’ with examples of dos and don’ts with the aim to help our readership become informed consumers of the knowledge produced from such manuscripts.

WHY META-ANALYZE?^2,3

Meta-analysis is usually performed for several reasons: (1) Increase power and precision; for example, Golwala et al⁴ compared the treatment effect of implantable cardioverter defibrillators (ICDs) to optimal medical therapy on all-cause mortality in non-ischemic cardiomyopathy (NICM) patients in a meta-analysis. The individual studies were small and yielded statistically non-significant treatment effects. When the authors meta-analyzed the six studies with a resultant increase in the power, the summary treatment effect was statistically significant in favor of ICD use, yielding a more precise estimate than the individual studies.⁴ (2) Settle controversies arising from apparently conflicting studies, as demonstrated in the same meta-analysis by Golwala et al.⁴ The recently conducted DANISH trial⁵ suggested no improvement in all-cause mortality in NICM patients undergoing ICD placement for primary prevention. However, when Golwala et al⁴ pooled results from five other studies, a statistically significant treatment effect was observed in favor of ICDs.⁴ (3) Generate and test new hypothesis; in a recent meta-analysis by Bajaj et al,⁶ the authors investigated reasons for differences in the rates of mortality in patients with acute pulmonary embolism undergoing catheter-based treatment. A meta-regression analysis conducted by the authors showed that higher intrapulmonary thrombolytic usage in part explained the differences in rates of mortality across studies, prompting the need to assess this hypothesis in future studies.⁶ (4) Help with design and sample size calculations for future studies; while trying to explain heterogeneity observed across studies, authors can usually attribute it to differences in eligibility criteria, definitions of endpoints, and baseline characteristics across studies. A close investigation of these differences may guide future study design. The summary treatment effects from meta-analysis may also help with power and sample size calculations for future studies. (5) Help with comparative effectiveness research; network meta-analysis can help compare different treatment options where direct comparisons are lacking. Bajaj et al⁷ in their study compared all U.S. Food and Drug Administration-approved strategies for stroke prophylaxis in non-valvular atrial fibrillation and demonstrated equivalent efficacy of all available treatments. However, the safety profiles of these treatments were different. Finally, based on these data, a ranking order was established taking into account efficacy and safety for each treatment. These data may aid clinicians in individualizing treatment in patients with atrial fibrillation in the absence of direct comparisons.⁷

HOW TO START?²

After a research question is outlined, the second step involves identification and screening of individual studies, followed by determination of eligibility for inclusion in the meta-analysis. It is very important to be overinclusive during the identification and screening phase to avoid missing any relevant studies on the research topic in question. A medical librarian is a great resource during this stage and may help search several electronic databases in an efficient manner. For example, Bajaj et al⁸ were able to devise an electronic search strategy for MEDLINE, Web of Science, and SCOPUS databases with a comprehensive list of Medical Subject Headings (MeSH) terms suggested by medical librarian; this helped them to be thorough with their screening search. Finally, eligibility and exclusion criteria based on a priori definition of population, study design, exposure/treatment(s), and outcome(s) should be used to identify studies for meta-analysis. A common criticism of meta-analysis, i.e. comparing apples and oranges, can be avoided at this stage. Authors should be upfront about the research question, use specific and valid definitions of exposure/treatment and outcome, and outline eligibility/exclusion criteria at this stage. An expert in the field is an invaluable resource at this step. It is extremely important to follow the definitions while extracting data from individual studies prior to conducting analysis.

In some cases, the eligible studies may not report the results in the format desired. Efforts should be made to contact the corresponding author of the original study to obtain the summary data in the desired format to allow for maximum inclusivity and in turn improve power and precision of the meta-analysis. For example, Gupta et al,⁹ in a meta-analysis, investigated the effect of zero vs non-zero coronary artery calcium on the initiation, continuation, or intensification of pharmacological and lifestyle preventive therapies. However, some of the eligible studies in this meta-analysis did not report the results in a format desired by the authors. Gupta et al⁹ were able to procure data from the authors of these studies in the desired format which allowed inclusion of two additional studies in their meta-analysis.

HOW TO PRESENT AND INTERPRET DATA?²

After the studies have been deemed eligible for inclusion based on quality control measures mentioned in the previous section, a qualitative assessment of studies based on standardized scales is recommended. Lastly, meticulously collected data can be pooled using standard statistical models.² A biostatistician is a great resource at this stage of meta-analysis. Traditionally, results of the statistical analysis are presented via a forest plot. This plot is a visual representation of effect estimates and their variability in each individual study as well as the pooled estimate from all the included studies. In the absence of variability (different direction and numerical measure of the treatment effect across studies), heterogeneity is thought to be minimal and discussion of data should focus on pooled estimate. However, when the heterogeneity is high, the discussion should focus on reasons of heterogeneity such as clinical heterogeneity (mixing apples with oranges) or true variability in treatment effects. The review by Kalra et al² thoroughly summarizes the presentation techniques for meta-analysis.

HOW TO APPRAISE?

There are several guidelines available to authors and reviewers for standardized conduction and reporting of meta-analysis. That is, the Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) guidelines are available for both meta-analysis of randomized controlled trials^10,11 and network meta-analysis.¹² Additionally, the Meta-analysis of Observational Studies in Epidemiology (MOOSE) guidelines are also available to authors conducting a meta-analysis of observational studies.¹³ These are very comprehensive guidelines and should be employed by all authors conducting meta-analyses. Based on these guidelines, we prepared a table for rapid appraisal of meta-analysis (Table 1). This table can aid beginners and those without a statistical background to screen the quality and validity of meta-analyses.

Table 1.

Quick appraisal of meta-analysis

Appraisal points
Clear and specific question
Specific eligibility/exclusion criteria for included studies
Clear and valid definition of exposure/treatment in a priori fashion
Clear and valid definition of outcomes in a priori fashion
Appropriate/comprehensive search strategy with the aim of including all relevant studies
Quality assessment of individual studies with standardized tools
Using appropriate mathematical modeling techniques
Assessment of heterogeneity
Attempt to explain the heterogeneity
Appropriate conclusion

LIMITATIONS OF META-ANALYSIS²

As is true with all study designs, meta-analyses have limitations and are often criticized. Several of these limitations can be overcome if a standardized methodology, as outlined in the guideline documents, is employed. Nevertheless, it is worthwhile highlighting a few important limitations: (1) Inclusion of poor-quality studies can render the summary estimate invalid; this issue should be dealt with during the design phase. Poor-quality studies should be excluded as they may introduce a significant bias and affect validity and generalizability of meta-analysis. (2) Oversimplification of an entire field: there is an inherent desire for authors, readers, and reviewers to oversimplify the results and give the summary estimate undue importance. It should be noted that the summary estimate is meant to reflect a treatment effect in a well-defined clinical situation determined by eligibility and exclusion criteria. Hence, the results should aid thoughtful clinical decision making, not replace it. (3) Disagreement with individual studies; results of a meta-analysis may disagree with those of included studies. This situation potentially occurs when the studies included in a meta-analysis have a wide dispersion in their individual treatment effects. This again highlights the need to employ careful inclusion/exclusion criteria to ensure comparison of similar study populations and endpoints. In most of the cases, this can be explained by reviewing individual studies carefully.

In summary, although meta-analysis is a powerful tool to summarize literature, its validity depends on the methodological rigor and critical appraisal on part of the authors, reviewers, and editors.