Meta‐analysis: A Brief Introduction

Abstract

Meta‐analysis is the process of combining data from multiple sources and analyzing it together to increase power and provide a clearer picture of the effect of intervention or exposure on an outcome. The process is not complicated, but requires a great deal of attention to detail. A specific set of inclusion criteria for studies must be defined. Published or available study results may be affected by publication bias of several different types, so the researcher should be sure to conduct a thorough search of available databases in order to include unpublished findings. Following selection, the group of studies should be examined using funnel plots or statistical tests. Meta‐analysis models themselves must be selected to properly reflect the combined studies. Both fixed‐ and random‐effects modeling are discussed. Two case studies are presented, illustrating a well‐conducted meta‐analysis and a meta‐analysis that was more controversial.

Introduction

Meta‐analysis is the process of combining data from multiple studies with the goal of better estimating the true effect of a specific intervention or exposure on a particular outcome. Individual studies are often insufficiently powered to draw definitive results, and even larger studies are often unable to adequately estimate differences in risk of rare adverse events. The systematic combination of results from multiple studies, even if they are inconclusive or contradictory, can help to more definitively determine the true measure of effect.

Meta‐analyses have become quite common in the biomedical literature and have gained attention in the mainstream press. ¹ , ² Numerous references on the theory and proper procedures for meta‐analysis exist in the medical and statistical literature. ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ In this paper, we review the basics of meta‐analysis, including issues in study identification and selection, issues of publication and other bias, and choices in statistical modeling. We follow this with two illustrative case studies. Our goal is to encourage appropriate use of meta‐analysis methods and increase understanding of published systematic reviews.

Study Selection

It is our opinion that perhaps the most critical part of any meta‐analysis is the evaluation of studies for inclusion in the analysis. Inclusion of a particular study in a meta‐analysis should be based on criteria established well before undertaking study evaluation. The criteria should be rigorous, but not overly restrictive. The goal is for the studies to be similar, but they obviously will not be identical. Study heterogeneity can impact the validity of the analysis; careful attention to selection criteria and appropriate modeling of study heterogeneity are essential. Eligible studies should be similar in at least two ways—the hypothesis addressed and the type of the study.

Hypothesis addressed

A good meta‐analysis first carefully defines the question of interest; studies are selected for inclusion in the meta‐analysis only if they address this hypothesis. Adequate definition of the problem is necessary for any study, but is especially important in meta‐analysis as it directly impacts the selection of studies for combined analysis. ³ In meta‐analysis of treatment studies, this question may be whether an intervention is efficacious or whether an intervention increases the risk of adverse events.

Types of studies

Meta‐analyses commonly combine information from randomized trials because of their typically rigorous study designs and randomization schemes, but well‐conducted observational studies can also be used in the same way. ⁵ Sample characteristics, outcomes analyzed, and type of treatment all may influence whether a study is included or excluded from the analysis. More detailed methods for the selection of studies are described elsewhere. ³

Identifying studies

Researchers conducting a meta‐analysis have numerous data sources available to them. It is essential that the researcher investigate as many as possible to both bolster the strength of the meta‐analysis and mitigate potential publication bias. We have listed many of these sources in Table 1 . The meta‐analyst should conduct their search in a systematic fashion, keeping track of search terms, keywords, and procedures to produce the most comprehensive list of eligible studies.

Table 1.

Database resources.

Data source	Location
Government sponsored
Clinical trials data bank	ClinicalTrials.gov
Australia New Zealand Clinical Trials Registry	http://www.anzctr.org.au
Chinese Clinical Trial Register (ChiCTR)	http://www.chictr.org
Clinical Trials Registry—India (CTRI)	http://www.ctri.in
German Clinical Trials Register (DRKS)	http://www.germanctr.de
Iranian Registry of Clinical Trials (IRCT)	http://www.irct.ir
Japan Primary Registries Network	http://rctportal.niph.go.jp (in Japanese) http://www.umin.ac.jp/ctr (English available) http://www.clinicaltrials.jp (English available) https://dbcentre3.jmacct.med.or.jp (English available)
The Netherlands National Trial Register (NTR)	http://www.trialregister.nl (in Dutch)
Sri Lanka Clinical Trials Registry (SLCTR)	http://www.slctr.lk/
International Standard Randomized Controlled Trial Number	http://www.isrctn.org/, http://www.controlled‐trials.com/isrctn/
Drugs@FDA	http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm
Aggregator databases
WHO ICTRP	http://www.who.int/ictrp/en/
Meta‐register of controlled trials	http://www.controlled‐trials.com/mrct/
Disease/location specific registries
Physician data query (PDQ) database	http://www.cancer.gov/cancerinfo/pdq/cancerdatabase
European Leukemia Trials Registry	http://www.leukemia‐registry.eu
Clinical Trial Registry of the University Medical Center Freiburg	http://www.uniklinik‐freiburg.de/zks/live/uklregister/Oeffentlich/extern_en.html
German Registry for Somatic Gene‐Transfer Trials (DeReG)	http://www.dereg.de
Industry registries
AstraZeneca	http://www.astrazenecaclinicaltrials.com/
Eli Lilly	http://www.lillytrials.com/
Roche	http://www.roche‐trials.com/
GlaxoSmithKline	http://www.gsk‐clinicalstudyregister.com/
Bristol‐Myers Squibb	http://www.bms.com/clinical_trials/Pages/clinical_trial_registry.aspx
PhRMA Clinical Study Results Database	http://www.clinicalstudyresults.org/
FDA
Drugs@FDA	http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm

The most common source of studies is published journal articles. These can be searched through MEDLINE/PubMed, or through subscription databases such as EMBASE. Additional sources outside of journals are referred to as “gray literature,” including book chapters, abstracts, presentations, unpublished results, theses, or letters. ⁹ , ¹⁰ The Cochrane Collaboration's CENTRAL Registry of controlled clinical trials combines these in an effort to compile all available information, both in journals and gray literature. ¹¹

Clinical trials registries afford another avenue for identifying studies. Disease‐ and pharmaceutical‐specific trials registries have been in existence for some time. However, neither is as comprehensive as a mandatory trials registry, such as the Clinical Trials Data Bank. ¹² There are currently multiple national and international trials registries, as well as meta‐registries. The Food and Drug Administration (FDA) can provide similar information; however, their database was not designed to be a public‐access registry. ¹³

Publication and related bias

No matter how thorough the search, the researcher is frequently limited to published results. Bias in study reporting—where publication is influenced by the direction or strength of the results—is the most obvious source of error. Reporting bias can refer to several situations, influencing the “if,”“when,” or “how” of publication.

Due to the nature of the source data for meta‐analyses, reporting bias of all types can contribute substantial error, leading the researcher to draw flawed or incorrect conclusions. ¹⁴ , ¹⁵ , ¹⁶ Publication bias describes the situation where positive results are preferentially published over inconclusive or negative results. This can refer to action on the part of either the publisher or the submitter. At least one major journal provides authors with instructions outlining that already known or negative results will only be accepted under extraordinary circumstances. ¹⁷

Additionally, researchers themselves may refrain from submitting negative or inconclusive results for publication, either in anticipation of rejection or simply deciding not to put forth the effort to publish. ¹⁸ Time‐lag bias has also been observed, where nonpositive results may take a significantly longer period of time to be submitted and published. ¹⁷ , ¹⁸ In a related fashion, positive results can lead to multiple publication bias, where more than one paper is published reporting the same results. ¹⁷ , ¹⁹ Another type of bias is outcome reporting bias, where primary hypothesis results are nonsignificant, but secondary hypotheses may have a positive result. These results are then reported as though they were the primary result. For example, a publication of a study of setraline reported results for several hypotheses, but did not identify the primary hypothesis as it was reported to the FDA, and the discussion focused more on significant secondary hypotheses. ¹³ , ²⁰ Reporting results in this fashion not only contributes to publication bias, but has potential to obscure the original study design, further hampering the meta‐analyst. There is also evidence that legitimately significant effect sizes may be inflated in published literature in comparison to results submitted for FDA approval. ¹³ , ²¹

Analysis Methods

Assessing publication bias

Regardless of the source of reporting bias, the impact remains the same. The meta‐analyst may be left to contend with an incomplete list of questionably reported results. Funnel plots provide a graphical approach to investigating bias. These display the effect size estimates plotted against the sample sizes or standard errors. An unbiased sample of studies should appear as a roughly symmetric funnel centered on the population average effect. Departures from this shape, particularly if one side of the funnel is missing, indicates selection bias is likely present. Figure 1 shows two funnel plots of hypothetical odds ratio (OR) results generated with Stata 10 software (StataCorp, College Station, TX, USA). ²² The vertical lines in the middle of each plot represent the log of the pooled OR estimate and the dashed lines represent approximate point‐wise 95% confidence limits. The symmetry in Figure 1A suggests no publication bias in the selected studies. We removed the studies having smaller effect and sample sizes and plotted the remainder in Figure 1B . The asymmetry in this plot would suggest publication bias toward larger studies and positive results. Statistical significance tests, including Begg's test ²³ and Egger's test, ²⁴ have been developed to indicate significance of the asymmetry. However, departures from symmetry can also result from poor study methodology, spurious reporting, real relationships between effect size and standard error, sampling variability, and random chance. ¹ Thus, meta‐analysts should research the sources of asymmetry in these funnel plots before drawing the conclusion of publication bias.

Figure 1.

(A) A funnel plot with pooled estimate (solid line) and approximate point‐wise 95% confidence limits showing no apparent publication bias. (B) A funnel plot suggesting publication bias in favor of larger studies and positive results.

Combining studies

Once studies have been selected, data must be combined appropriately to estimate the effect or association of interest and answer the research question. Analysis of data from multiple studies may be divided into two types: combined analysis of summary measures and pooled analysis of subject‐level data. Pooled analysis of subject‐level data from multiple studies is increasingly common and generally preferred if the data are available, but may be too restrictive in some areas of research. Methods for this type of analysis can be found in several references. ²⁵ , ²⁶ , ²⁷

As subject‐level data may be difficult to obtain, a more common approach is to combine measures of effect from the selected studies using weighted averaging or a regression model. The idea is to calculate the average measure of this effect where study‐specific estimates are weighted by some function of the sample size and/or standard error of the estimate, reflecting the relative importance of each study. Models for combining effect estimates can be further divided into two categories: fixed effects and random effects. Fixed‐effect models assume the studies being combined are multiple replicates of the same study where the true effect measure being estimated in each study is the same. By contrast, random‐effects models assume that each study is chosen at random from a multitude of possible studies. The true effect measure is assumed to be different for each study due to, e.g., differences in the precise populations sampled or differences in study methodology. The choice of method can have a substantial impact on the results, particularly if sample sizes or event rates are small for most of the studies. In particular, if a study has no events, it will be excluded from a fixed‐effects analysis of ratio estimates as the study‐specific estimate is undefined. Careful thought should be given to the choice of statistical model and the assumptions underlying each model when combining data from multiple studies. Further insight into the dif erences between fixed‐ and random‐effects models can be found elsewhere. ²⁸ , ²⁹

Heterogeneity in either the populations sampled or the manner in which the data were modeled should raise suspicion of the validity or reliability of the results, as it can introduce confounding and other defects into the meta‐analysis. Unfortunately, heterogeneity is typically unavoidable. The extent of its influence can be investigated by diagnostic analysis of the sensitivity of the results to one or more of the studies, similar to the methods used in regression model diagnostics. Additionally, the Q‐statistic ³⁰ or the I²‐statistic ³¹ can be calculated to test for the homogeneity or constancy of the effects or associations reported among studies. Moreover, to correct for sources of observed heterogeneity, fixed‐ and random‐effects meta‐analytic regression models can include study‐level covariate information if it has been consistently reported in the selected studies.

Presentation of results

As the results from a meta‐analysis summarize the results of multiple studies, the presentation of the results should describe both final pooled summary statistics and the data from each study. It is customary to produce a table of descriptive information on each study under consideration, which concisely summarizes each study's design, targeted population, characteristics of the sample (e.g., demographic information), and the statistical methods used to analyze the data. This allows the reader to assess the heterogeneity among the studies which might influence the quality of the results. The more similar the selected studies are with respect to sample selection and effect or association modeling, the better the meta‐analytic sample will represent the population under study. The meta‐analytic sample will thus have qualities similar to a simple random sample and the summary statistics from the meta‐analysis will tend to be unbiased and considered reliable.

We recommend that the original results from each study, in terms of the effect estimate, their standard errors, and statistical significance (i.e., 95% confidence intervals or p values), should be reported in a single table along with the results of the meta‐analysis. The information from this table as well as the respective contributions of each study to the pooled meta‐analytic estimate can also be displayed graphically using a forest plot. ³² , Figure 2 displays a forest plot of hypothetical ORs on the log‐scale. Each study is summarized by a row in the plot from left to right by a study identifier, a plot of the reported OR estimate and 95% confidence interval, a box proportional in size to the sample size relative to the other studies, the tabulated OR and 95% confidence interval, and the relative weight given to the study. The last row represents the overall pooled results from a fixed‐effects model. The plotted values can be easily compared to the null hypothesis of no association represented by the solid vertical line or to the pooled estimate represented by the diamond shape and dashed vertical line. The 95% confidence interval for the pooled OR (0.96, 1.31) includes 1. Thus, at the 0.05 significance level we would not reject a null hypothesis of no association. Furthermore, the I² test statistic suggests that if the population of studies from which these results came were homogeneous, then the probability of observing these data would be low (p= 0.016). The source of the heterogeneity is unclear from the data presented and should spur the meta‐analyst to review the included studies. Under the assumption that each included study meets the preset inclusion criteria and there is no information with which to extend the model, for example by meta‐regression, then the heterogeneity may be attributed to random sampling error. A random‐effects model would then be a more appropriate choice, which produces a pooled OR of 1.11 with 95% confidence interval (0.91, 1.34).

Figure 2.

A forest plot from a weighted meta‐analysis of odds ratios.

Case Study 1

Almost half of cancer survivors are younger than 65, but subsequent employment status has not been well examined. de Boer et al. published what we feel to be a good example of a well‐conducted meta‐analysis on observational studies. At the outset, specific study inclusion and exclusion criteria were set, based on presence of a control group, average age between 18 and 60, and assessment of employment status after cancer treatment. MEDLINE and four additional subscription databases were searched for eligible studies. Individual study characteristics were then reported in a table. Additionally, quality of each study was assessed on twelve aspects using the methodological index for nonrandomized studies (MINOR) and reported as well. Publication bias was examined using the Egger statistic, from which it was determined that none was present. Heterogeneity was quantified using the I²‐statistic. To account for heterogeneity, separate meta‐analyses were run within cancer diagnosis subgroups and using random‐effects models. Furthermore, associations were calculated using the multivariate meta‐regression models that adjusted for other employment‐related variables. A forest plot of results was included. The researchers were able to conclude that cancer survivors were about 1.37 times more likely to be unemployed as healthy controls. This meta‐analysis serves as a good example of the procedure, with straightforward inclusion/exclusion criteria, an intensive search of available literature, assessment of publication bias, multiple methods for addressing study heterogeneity, and the proper use of random‐effects methods. ³³

Case Study 2

Rosiglitazone is a drug used to control blood glucose levels in patients with type 2 diabetes mellitus. While the drug was extensively studied with respect to efficacy, none of the trials were large enough to conclusively establish the risk of cardiovascular (CV) events. Nissen and Wolski reported on results of a meta‐analysis assessing the impact of Rosiglitazone on risk of myocardial infarction (MI) and cardiac death (CD). ³⁴ They identified 116 studies from published and unpublished reports, out of which 48 met their criteria for inclusion. A fixed‐effects analysis concluded that Rosiglitazone increased the risk of myocardial infarction (RR = 1.43 [1.03–1.98], p= 0.03) but could not conclude that there was increased risk of CD (RR = 1.64 [0.98–2.74], p= 0.06). After these findings were published, the FDA issued a public safety alert about potential for increased risk of CV events and sales of the drug plummeted.

While the authors followed proper procedures in study selection, less careful attention was paid to the choice of statistical model. Of the 48 studies, 6 had no MI in either study drug arm and 23 had no CD and were consequently excluded from the fixed‐effects analyses. A subsequent reanalysis of the same data using a random‐effects model showed insufficient evidence for a difference in the risk of MI (RR = 1.51 [0.91–2.48], p= 0.11), but concluded that there was a signif cant increase in the risk of CD (RR = 2.37 [1.38–4.07], p= 0.0017). ²⁸ The authors of this analysis used it as a cautionary tale of over‐reliance on meta‐analysis for setting public policy, and expressed confidence that large ongoing randomized trials for evaluating long‐term cardiac outcomes would provide more conclusive evidence. However, the recent inconclusive results of one of these trials, the RECORD trial, may indicate that this confidence was misplaced and argue for the value of meta‐analysis even when data from large trials are available. ³⁵ , ³⁶

Conclusions

Meta‐analysis is generally a relatively simple analytic procedure. The effort that surrounds meta‐analysis is not concentrated in the data analysis, but rather in the strict attention to detail, both in the background research and the correct selection of analytical models. This requires carefully defining a hypothesis, establishing and strictly adhering to inclusion and exclusion criteria, and thoroughly searching through all available resources for eligible studies, assessing potential bias and heterogeneity, and appropriately selecting a model. Only after these preliminary objectives have been achieved should the meta‐analyst proceed with the relatively straightforward process of conducting the data analysis and reporting findings.

Inattention to these details can lead to a meta‐analysis that is inaccurate and misleading, as shown in our second case study. It is not our intention to discourage researchers from attempting a meta‐analysis, but rather to encourage vigilance in approaching all aspects of this important process.