Systematic Reviews and Meta Analysis of Published Studies: An overview and Best Practices

Abstract

Systematic reviews and meta-analytic approaches are widely used in the clinical arena to integrate outcome data from published studies in a patient population that address a set of related research hypotheses. The credibility of this line of research is dependent on how the studies are chosen, how the data are assembled and how the results are reported. In this brief report, we provide an overview of the minimum set of reporting requirements for systematic reviews and meta-analyses based on the PRISMA (Preferred Reporting Items of Systematic reviews and Meta-Analyses) guidelines. As with any research, following a set of established guidelines is essential for quality and consistency of the findings across studies, and for assessment of clinical utility.

Introduction

The Cochrane collaboration (1) defines systematic review as a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. A meta-analysis then utilizes appropriate statistical methods to assimilate the results of the studies included in the systematic review to address a set of related research hypotheses. Meta-analysis of published studies uses summary data from the published studies, and thus is restricted in its scope (i.e., testing new hypotheses for which summary data is not previously published). A pooled analysis on the other hand utilizes individual subject data from the included studies, and can help answer new research hypotheses (if the relevant data are available and collected in a similar manner). Meta analyses of published studies are typically performed when individual subject data is not readily accessible / available and under a framework when the findings from the multiple studies are conflicting or unclear. In such cases, an integration of the existing outcome data in a relatively quick manner (as opposed to a pooled analysis that requires considerably more effort) becomes important.

As with any published research, the credibility of a systematic review is dependent on how the studies are chosen, how the data are assembled and how the results are reported. Two recent examples of a well conducted meta analysis are as follows: a) a meta-analysis performed to understand the effectiveness of low dose computed tomography for lung cancer screening, while awaiting the results from the prospective definitive randomized trial (2), and b) a meta analysis performed to potentially resolve the conflicting data on the choice of therapy for previously untreated, transplant ineligible, elderly myeloma patients, while awaiting the results from the pooled analysis of individual patient data, and prospective randomized trials (3).

Extensive work in the literature in the mid 1980’s and early 1990’s demonstrated that the reporting quality of systematic reviews and the approaches utilized for meta-analysis were generally inconsistent and incompletely reported (4, 5). Thus, in 1996, the QUOROM Statement (QUality Of Reporting Of Meta-analyses) was developed by a group of international researchers, which focused on the minimum set of reporting requirements for systematic reviews and meta-analyses of randomized controlled trials (6). This guideline was subsequently updated in 2009 to incorporate the conceptual and practical advances in this arena, and was renamed PRISMA (Preferred Reporting Items of Systematic reviews and Meta-Analyses) (7, 8). The PRISMA (and previously the QUOROM) guidelines have been adopted as a minimal standard for publication requirements by several journals. While the main focus of the guidelines are on randomized trials, it can be used as a basis for reporting systematic reviews of other types of research (7, 8).

In this brief report, we provide an overview of the PRISMA guidelines, which includes the PRISMA flow diagram and the PRISMA checklist for systematic reviews and meta-analysis. We use a hypothetical example to illustrate some of the concepts. Together, the checklist and the flow diagram provide the necessary guidance and documentation essential for conducting and reporting systematic reviews and meta-analysis.

The PRISMA Approach

The PRISMA flow diagram is a schematic representation that maps out the number of studies identified, the number excluded, and the reasons for exclusions, and the final number of studies included in the systematic review and meta-analysis. This is analogous to the CONSORT diagram that is followed when reporting results from prospective clinical trials (9). The PRISMA checklist includes a comprehensive list of 27 items that pertain to the content of performing and reporting a systematic review and meta-analysis. Below is a synopsis of the main topics that is covered in the checklist (7, 8):

1. Title description,

2. Abstract structure and content,

3. Introduction section that clearly outlines the rationale and the hypothesis for the purported meta-analysis,

4. Methods section, which includes the search criteria, selection process for studies that are included in the review, assembly of data, statistical approaches used to assimilate the results and assessment of selection and/or publication bias,

5. Results section that includes a flow diagram for the final set of studies included and their quality, information on the individual study characteristics and results, bias assessment, summary of each outcome analysis with estimates and confidence intervals, and subgroup and/or sensitivity analyses results,

6. Discussion section that summarizes the main findings and highlights any limitations, and finally,

7. Role of any funding source in the systematic review to assess potential conflict of interest.

Additional considerations in the interpretation of the findings from a meta-analysis include the weighting scheme used to adjust for the study size when integrating the results (with greater weight assigned to (larger) studies that provide more information), and the between study heterogeneity. In a summary meta analysis, the reasons for such heterogeneity are likely caused by the differences in a) the definitions of the outcome across the individual studies, b) dosage and schedule of treatments, c) inclusion/exclusion criteria for subjects in the individual studies etc. In the Gopal et al. (2) meta-analysis, the heterogeneity in the control arm (using chest x-ray versus no screening) across the trials was addressed by doing two separate analyses: including all studies, and including only those that used chest x-ray as the control arm. In the case of Kapoor et al. (3), sensitivity analyses by leaving out one trial at a time was performed to more fully capture the impact of the trial level heterogeneity on the final conclusions.

Hypothetical Example

Assume that 6 published reports testing the efficacy of a control regimen (C) to an experimental regimen (E) to treat patients with advanced lung cancer were identified using the Cochrane collaboration recommended optimal search strategy. Table 1 is a summary of the hypothetical outcome data from the individual studies. As an aside, in instances where relevant summary statistics are not readily available from the published articles, the information on observed number of events and reported P-value from the log-rank-test statistic can be used to derive the point estimates and its associated standard error (or variance) (10). Given the somewhat differing levels of agreement in the overall survival (OS) outcome for the optimal treatment (C versus E) in this example, a meta-analysis of these studies can be conducted to assimilate the results. Under a random effects model, the summary HR and the 95% CI would be1.134 (0.89, 1.44; p-value: 0.3), thus suggesting that there is no statistically significant difference in OS between the treatments. Suppose that the study II outcomes were 1.5(1.4, 1.6), and 0.001 for the HR (95% CI) and the variance (log HR), and the variance (log HR) for studies 5 and 6 were 0.02, then the summary HR and the 95% CI would be 1.24 (0.99, 1.6; p-value: 0.06) under a random effects model, suggesting a trend towards favorable outcome for the experimental regimen E.

Table 1.

Data from six hypothetical trials with similar sample size in advanced lung cancer comparing C (control regimen) versus E (experimental regimen)

Study Number	Overall Survival
	Hazard Ratio (C versus E) (95% CI)	Variance (log HR)
I	1.7 (1.29, 2.24)	0.02
II	1.04 (0.74, 1.46)	0.03
III	1.5 (1.07, 2.11)	0.03
IV	1.3 (0.99, 1.72)	0.02
V	0.9 (0.78, 1.03)	0.005
VI	0.8 (0.75, 0.85)	0.001

The between trial heterogeneity in both of the above scenarios is high, thus requiring additional (sensitivity / subgroup) analyses to be performed in order to understand and identify those studies that contribute to the heterogeneity. In this example, if studies 5 and 6 are excluded, then the heterogeneity across the remaining 4 trials becomes insignificant, and the overall HR of 1.4 (p=0.002) shows a statistically significant improvement in OS for the experimental regimen over the control regimen. Removal of studies 2, 5, and 6 results in a complete disappearance of heterogeneity and a much stronger evidence in favor of E.

Summary

The reporting quality of systematic reviews and meta-analyses has considerable variation, thus limiting the ability to comprehensively assess the strengths and weaknesses of the reviews and the clinical utility of the results. A well conducted systematic review should be a thorough process of collecting, reviewing and presenting all available evidence, and a well conducted meta-analysis should utilize the appropriate statistical techniques to extract and combine data across studies to produce a summary result. The PRISMA guideline referenced in this brief report is a recommended standard for reporting systematic reviews and meta-analyses.