Systematic Literature Reviews and Meta-Analyses

Abstract

Background

Because of the rising number of scientific publications, it is important to have a means of jointly summarizing and assessing different studies on a single topic. Systematic literature reviews, meta-analyses of published data, and meta-analyses of individual data (pooled reanalyses) are now being published with increasing frequency. We here describe the essential features of these methods and discuss their strengths and weaknesses.

Methods

This article is based on a selective literature search. The different types of review and meta-analysis are described, the methods used in each are outlined so that they can be evaluated, and a checklist is given for the assessment of reviews and meta-analyses of scientific articles.

Results

Systematic literature reviews provide an overview of the state of research on a given topic and enable an assessment of the quality of individual studies. They also allow the results of different studies to be evaluated together when these are inconsistent. Meta-analyses additionally allow calculation of pooled estimates of an effect. The different types of review and meta-analysis are discussed with examples from the literature on one particular topic.

Conclusions

Systematic literature reviews and meta-analyses enable the research findings and treatment effects obtained in different individual studies to be summed up and evaluated.

Every year, there is a great increase in the number of scientific publications. For example, the literature database PubMed registered 361 000 new publications in 1987, with 448 000 in 1997 and 766 000 in 2007 (research in Medline, last updated in January 2009). These figures make it clear how increasingly difficult it is for physicians in private practice, clinicians and scientists to obtain comprehensive current information on any given medical topic. This is why it is necessary to summarize and critically analyze individual studies on the same theme.

Summaries of individual studies are mostly prepared when the results of individual studies are unclear or inconsistent. They are also used to study relationships for which the individual studies do not have adequate statistical power, as the number of cases is too low (1).

The Cochrane Collaboration undertakes systematic processing and summary of the primary literature for many therapeutic topics, particularly randomized clinical studies (www.cochrane.org). They have published a handbook for the performance of systematic reviews and meta-analyses of randomized clinical studies (2). Cook et al. have published methodological guidelines for this process (3). Instructions of this sort help to lay down standards for the summary of individual studies. Guidelines have also been drawn up for the publication of meta-analyses on randomized clinical studies (4) and on observational studies (5).

Publications on individual studies may be summarized in various forms (1, 6– 10):

• Narrative reviews

• Systematic review articles

• Meta-analyses of published data

• Pooled reanalyses (meta-analyses with individual data).

These terms are often not clearly allocated in the literature. The aim of the present article is to describe and distinguish these forms and to allow the reader to perform a critical analysis of the results of individual studies and the quality of the systematic review or meta-analysis.

Methods

The various types of systematic reviews and meta-analyses of scientific articles will be defined and the procedure will be explained. A selective literature search was performed for this purpose.

A "review" is the qualitative summary of the results of individual studies (1). A distinction is made between narrative reviews and systematic reviews (table 1). Narrative reviews (A) mostly provide a broad overview of a specific topic (1, 11). They are therefore a good way of rapidly obtaining current information on research on a given topic. However, the articles to be included are selected subjectively and unsystematically (1, 11). For some time, the Deutsches Ärzteblatt has been using the term "selective literature review" for this type of review. Narrative reviews will not be further discussed in this article.

Table 1. Various types of summary of individual studies.

	A	B	C	D	E
	Narrative review	Systematic review	Meta-analysis of published data	Pooled reanalysis	Prospectively planned meta-analysis
Preparation of a detailed study protocol and analysis plan	–	+	+	+	+
Literature search for suitable studies in accordance with prospectively defined inclusion and exclusion criteria	–	+	+	+	+
Quantitative summary of the results (calculation of pooled estimates, examination of heterogeneity, sensitivity analyses)	–	–	+	+	+
Analysis of individual data	–	–	–	+	+
Common study protocol for the individual studies and prospectively planned analysis	–	–	–	–	+

In contrast, systematic review articles (B) claim that, if possible, they consider all published studies on a specific theme—after the application of previously defined inclusion and exclusion criteria (11). The aim is to extract relevant information systematically from the publications. What is important is to analyze the methodological quality of the included publications and to investigate the reasons for any differences between the results in the different studies. The results of each study are presented and analyzed according to defined criteria, such as study design and mode of recruitment.

The same applies to the meta-analysis of published data (C). In addition, the results are quantitatively summarized using statistical methods and pooled effect estimates (glossary) are calculated (1).

Glossary.

• Aggregated data

  The summary of individual data

• Bias

  Distortion of study results from systematic errors

• Confidence interval:

  The confidence interval is the range within which the true value lies with a specified probability, usually 95%.

• Confounder

  A confounder is a factor which is linked to both the studied disease and the studied exposure. For this reason, it can either enhance or weaken the true association between the disease and the target parameter.

• Effect estimates

  An effect estimate, such as the odd ratio or relative risk, estimates the extent of the change in the frequency of a disease caused by a specific exposure.

• Exposure

  Contact with a specific risk factor

• Forest plot:

  A forest plot is a graphical representation of the individual studies, as well as the pooled estimate. The effect estimate of each individual study is generally represented on the horizontal or vertical axis, with a confidence interval. The larger the area of the effect estimate of the individual study, the greater is the weight of the study, as a result of the study size and other factors. The pooled effect estimate is mostly represented in the form of a diamond.

• Funnel plot:

  In a funnel plot, the study size is plotted against the effect estimates of the individual studies. The variances or the standard error of the effect estimate of the individual studies is given, rather than the study size. Smaller studies give larger variances and standard errors. The effect estimates from large studies are less scattered around the pooled effect estimate than are the effect estimates of small studies. This gives the shape of a funnel. A publication bias can be visualized with the help of funnel plots.

• Heterogeneity:

  Statistical heterogeneity describes the differences between the studies with respect to the effect estimates. These may be caused by methodological differences between the studies, such as differences in study population or study size, or differences in the methods of measurement.

• Individual data:

  In individual data, all data (e.g. age, gender, diagnosis) are at the level of the individual.

• Odds ratio:

  In medicine and epidemiology, the odds is the ratio of the probability of exposure and the probability of not being exposed. The quotient of the odds of the cases and the odds of the controls gives the odds ratio. For rare diseases, the odds ratio is an approximation to the relative risk.

• Original data:

  See individual data

• Publication bias:

  Publication bias means that studies which failed to find any influence of exposure on the target disease ("negative studies") are more rarely published than studies which showed a positive or statistically significant association. Publication bias can be visualized with funnel plots.

• Risk factor:

  A risk factor modifies the probability of the development of a specific disease. This can, for example, be an external environmental effect or an individual predisposition.

• Relative risk:

  To calculate the relative risk, the probability that an exposed individual falls ill is divided by the probability that a non-exposed person falls ill. The relative risk is calculated on the basis of incident diseases.

• Sensitivity analyses:

  Using sensitivity analyses, it is examined whether excluding individual studies from the analysis influences the pooled estimate. This tests the stability of the pooled effect estimate.

• Subgroup analysis:

  In subgroup analysis, separate groups in the study population, such as a homogenous ethnic group, are analyzed separately.

A pooled reanalysis (D) is a quantitative compilation of original data (glossary) from individual studies for combined analysis (1). The authors of each study included in the analysis then provide individual data (glossary). These are then compiled in a combined database and analyzed according to standard criteria fixed in advance. This form of pooled reanalysis is also referred to as "meta-analysis of individual data".

In a prospectively planned meta-analysis (E), the summary of the individual studies and the combined analysis is included in the planning of the individual studies. For this reason, the individual studies are performed in a standard manner. Prospectively planned meta-analyses will not be further discussed in this article.

It is essential for all forms of summary—except the narrative review—that they should include a prospectively prepared study protocol, with descriptions of the questions to be answered, the hypotheses, the inclusion and exclusion criteria, the selection of studies, and, where applicable, the combination of the data and the recoding of the individual data (only for pooled reanalysis).

Types of study summaries

The procedure for the summary of the studies will now be presented (modified from [7, 10, 12, 13]). This is intended to enable the reader to assess whether a given summary fulfils specific criteria (Box).

Box. Checklist for the analysis of a systematic summary.

• Was there an a priori study protocol?

• Was there an a priori hypothesis?

• Was there a detailed description of the literature search used?

• Were prospectively specified inclusion and exclusion criteria clearly described and applied?

• Was the possible heterogeneity between the studies considered?

• Was there a clear description of the statistical methods used?

• Were the limitations of the summary discussed?

1. Was the question to be answered specified in advance?

The question to be answered in the review or meta-analysis and the hypotheses must be clearly defined and laid down in writing prospectively in a study protocol.

2. Were the inclusion and exclusion criteria specified in advance?

On the basis of the inclusion and exclusion criteria, it is decided whether the studies found in the literature search (see point 3) are included in the review/meta-analysis.

3. Were precautions taken to find all studies performed with reference to the specific question to be answered?

An extensive literature search must be performed for studies on the topic. If at all possible, this should be in several literature databases. To avoid bias, all relevant articles should be considered, whatever their language. Moreover, a search should be performed in the literature lists of the articles found and for unpublished studies in congress volumes, as well as with search machines on the Internet.

4. Was the relevant information extracted from the published articles or were the original data combined?

For a systematic review article (B) and for a meta-analysis of published data (C), relevant information should be extracted from the publications.

For a pooled reanalysis (D), authors of all identified studies must be contacted and requested to provide individual data. The individual data must then be coded according to standard specifications, compiled in a combined database and analyzed.

5. Was a descriptive analysis of the data performed?

In all forms of summary, it is usual for the most important characteristics of the individual studies to be presented in overview tables. Table 2 shows an example of such a table, taken from a meta-analysis with published data (C) (14). This helps to make the differences between the studies clear with respect to the data examined.

Table 2. Characteristics of studies included in a systematic review on the association between oral contraceptives and cervical neoplasia (14).

Study (country)	Study design	Type of cervical neoplasia	Measure of HPV	Cases/controls status	Year of diagnosis for cases	Use of hormonal contraceptives in controls
						% ever use of oral contraceptives (% use for > 5 years)	% ever use of injectable contraceptives (% use for > 5 years)
Case-control studies, including those with population (pop) and/or hospital controls (hosp)
Brinton/Jones, 1986 (USA) (e1–e2)	pop	Invasive/in situ	None	772/801	1982–1984	51 (18)	NK
Peters, 1986 (USA) (e3)	pop	Invasive*1	None	200/200	1980–1981	26 (NK)*2	NK
Ebeling, 1987 (Germany) (e4)	hosp	Invasive	None	129/275	1983–1985	66 (46)	NK
Brinton, 1990 (4 countries*4) (e5, e6)	pop/hosp	Invasive	FISH	759/1429	1986–1987	25 (11)	6 (1)
WHO, 1993 (9 countries*5) (e7– e12)	hosp	Invasive/in situ	None	3848/13 644	1979–1988	41 (8)	15 (4)*3
Ursin, 1994 (USA) (e13)	pop	Invasive*6	None	195/386	1977–1991	81 (36)	NK
Cuzick, 1996 (GB) (e14)	pop	Invasive	None	121/241	1985–1991	92 (62)	NK
Madeleine, 2001 (USA) (e15)	pop	In situ*6	PCR/ serology	132/478	1990–1996	84 (29)	NK
Berrington, 2002 (GB) (e16)	pop	Invasive	Serology	221/393	1984–1988	88 (47)	NK
Moreno, 2002 (8 studies*7) (e17)	pop/hosp	Invasive/in situ*1	PCR	2171/2299	1985–1997	36 (11)	NK

NK, not known; FISH, fluorescent in situ hybridation; ^*1 squamous cell carcinoma only; ^*2 ever use → 2 years’ use;

^*3 relative risks for injectable contrceptives adjusted for oral contraceptive use; ^*4 Costa Rica, Colombia, Mexico, Panama;

^*5 Australia, Chile, Colombia, Israel, Kenya, Mexico, Nigeria, Philippines, Thailand; ^*6 adenocarcinoma of the cervix only;

^*7 Brasil, Colombia, Morocco, Paraguay, Peru, Philippines, Spain, Thailand (Shortened from: Smith J, Green J, Berrington de Gonzalez A et al.: Cervical cancer and use of hormonal contraceptives: a systematic review. Lancet 2003; 361: 1159–67. With the kind permission of Elsevier)

6. Were the calculations of the effect estimates of the individual studies and of the pooled effect estimate presented?

How were the effect estimates of the individual studies calculated?—Systematic review articles (B) usually contain tables with the effect estimates of the individual studies. In a meta-analysis of published data (C), the effect estimates of individual studies (for example, odds ratio or relative risk, see Glossary) are either directly extracted from the publications or recalculated in a standard manner from the data in each publication (figure 1). Depending on the nature of the factors and target parameters (binary, categorical or continuous variables), a logistic or a linear regression model is used to calculate the effect estimates of the individual studies in the meta-analyses of published data (C) and pooled reanalyses (D).

Figure 1.

The results of the individual studies and the pooled estimate, presented as forest plots on the association between oral contraceptives and cervical carcinoma, as an example of the meta-analysis of published data (14); N.A. = not available; * never use means <2 years use. CI = confidence interval

(Shortened from: Smith J, Green J, Berrington de Gonzalez A et al.: Cervical cancer and use of hormonal contraceptives: a systematic review. Lancet 2003; 361: 1159–67. With the kind permission of Elsevier).

How was the pooled effect estimate calculated?—The effect estimates of the individual studies are combined by statistical procedures to give a common pooled effect estimate (9) (figure 1). In meta-analyses with published data (C), two methods are mostly used to calculate a pooled effect estimate: either the fixed effect model or the random effect model (15, 16). They differ with respect to assumptions about the heterogeneity of the estimate between individual studies (see point 7). The method used should be given in the publication and justified. The effect estimates of the individual studies and the pooled effect estimates can be graphically presented in the form of so-called forest plots (Glossary; Figure 1; [14]).

In pooled reanalyses (D), the pooled effect estimates are mostly calculated by logistic or linear regression. However, the statistical analysis must adequately allow for the origin of the data sets from different studies. The results of the pooled reanalyses can be presented like the results of a single combined study (table 3).

Table 3. Results of a pooled reanalysis of the association between oral contraceptives and cervical carcinoma—analyzed and presented in a similar manner to an individual study (24).

Duration of use	Time since last use	Cases/controls	Mean duration of use in years (cases)	RR	95% CI*	p-value*
Never		7356/21682	–	1.00	–	–
5+ years	Current user	880/1466	11.1	1.90	1.69–2.13	s.
	2–9 years	747/1510	9.3	1.28	N. A.	s.
	10+ years	412/1654	8.1	0.94	N. A.	n. s.

Trend test: χ² = 66.2; p < 0.0001

RR, relative risk, adjusted for age, study or study center, age at first sexual intercourse,

number of sex partners, number of full-term pregnancies, smoking and screening status;

^* Information taken from the publication; CI, confidence interval; N.A., not available;

s., significance at the level α = 5%; n.s., not significant at the level α = 5%

(Shortened and modified from: International Collaboration of Epidemiological Studies of Cervical Cancer: Cervical cancer and hormonal contraceptives: collaborative reanalysis of individual data for 16,573 women with cervical cancer and 35,509 women without cervical cancer from 24 epidemiological studies. Lancet 2007; 370: 1609–21. With the kind permission of Elsevier)

7. Were problems considered in the interpretation of pooled estimates?

Was the heterogeneity between the estimates considered?—There may be marked differences between the estimates in the individual studies. This statistical heterogeneity (glossary) between the studies may be caused by differences in study design, study populations (age, gender, ethnic group), methods of recruitment, diagnosis, or methods of measurement (17, 18). The methodological heterogeneity between the studies can be visualized in an overview table, in which the most important characteristics of the individual studies are presented (table 2). The heterogeneity can be formally investigated with the help of statistical tests. If there is statistical heterogeneity between the studies, the random effect model, rather than the fixed effect model, should be used for the calculation of the pooled estimate (7, 15, 16). There is, however, no clear definition as to when the statistical heterogeneity between the studies is so large that the pooled effect estimate should not be calculated (1, 19). In addition, the heterogeneity between the studies should be examined by subgroup analysis (glossary). For example, this might involve combined analysis of only studies with the same characteristics in the study population, such as homogenous age groups, the same ethnic groups or the same histological findings. Moreover, studies with the same characteristics—such as study quality or study size—may be considered separately in subgroup analyses. This may indicate whether the effect of the corresponding risk factors (glossary) is different in the different subgroups.

Were sensitivity analyses performed?—Like subgroup analyses, sensitivity analyses (glossary) serve to test the stability of the pooled estimate. It is, for example, possible that the pooled effect estimate is mainly determined by one large study. If this study is excluded from the analysis, the pooled effect estimate may change. This must be borne in mind in the discussion and interpretation of the results.

Was a possible publication bias considered?—A publication bias (glossary) can be visualized with a so-called funnel plot (glossary) (7, 20– 22). Figure 2 shows an example with simulated data. In the upper funnel plot (Figure 2a), there is a roughly funnel shaped distribution of the effect estimates of the individual studies around the pooled effect estimates (middle broken line). There is no publication bias here. In the lower funnel plot (Figure 2b), the small studies are missing, which in this example show no increased risk. For this reason, there is probably a publication bias, because these studies had not been published.

Figure 2.

Visualization of publication bias with funnel plots of simulated data a) No publication bias; b) Publication bias; SE = standard error; OR = odds ratio

8. How were the results interpreted?

In the interpretation of the results, possible limitations should be discussed and considered. For example, the reliability of the results of individual studies can be limited by the inadequate quality of the individual studies—for example, by selection of the study population or from aggregated data (glossary).

Results

The method section describes the individual steps for the extraction of the relevant points which must be considered in the systematic summary of scientific articles (Box). This checklist can also be used to analyze the quality of systematic review articles or meta-analysis.

Publications on the association between the administration of oral contraceptives and the development of cervical carcinoma were used as examples of the performance of a systematic literature review (B), a meta-analysis of published data (C), and a pooled reanalysis (D). This association has been scientifically studied for a long period.

In 1996, La Vecchia et al. published a systematic review article (B) on this topic, including six studies (23). Their overview table contained a variety of information on the individual studies. No pooled effect estimate was calculated.

In 2003, Smith et al. (14) presented a meta-analysis of published data (C) of 28 studies on the same topic. The included studies were first summarized in a descriptive overview, as is common in systematic review articles (table 2). This table shows that the study methods were heterogenous (glossary); for example, HPV was detected in different ways (table 2). The heterogeneity was also formally investigated with statistical tests and various subgroup analyses were performed. In contrast to the systematic review article (B) of LaVecchia et al., pooled effect estimates were calculated with the published data (figure 1). The effect estimates for the individual studies and the pooled effect estimates with their confidence intervals (glossary) were presented as a forest plot (figure 1).

In 2007, a pooled reanalysis (D) was published for 24 studies on the same topic for which the original data were available (24). In contrast to the meta-analysis of published data, the pooled effect estimates were calculated from the original data and only the combined results were presented (table 3). This kind of analysis is only possible in a pooled reanalysis, as the original data with precise information on all parameters for each participant are then available. Nevertheless, here too it is necessary to consider that the individual data (glossary) are derived from different studies.

Discussion

Systematic review articles (B) can provide a comprehensive overview of the current state of research (1). They are also necessary for the development of S2 and S3 guidelines for formal evidence-based research (25). Meta-analyses of published data (C) are performed to calculate additional pooled effect estimates from the individual studies (1). Like systematic review articles, they are feasible whether the authors of the original articles are prepared to cooperate or not.

The calculated pooled effect estimates may be of limited validity for various reasons. Firstly, it has not been clearly defined what is the maximum order of heterogeneity between the studies which is negligible and which then allows a meaningful calculation of a pooled effect estimate (1, 19). If the individual studies are too heterogenous, a pooled effect estimate should not be calculated. Secondly, the pooled effect estimate is mostly calculated from aggregated data. Subgroup analyses and the consideration of potential confounders (glossary) are often impossible, or only possible to a limited extent (1, 19). Thirdly, publication bias is also a problem for the meta-analysis of published data.

In a pooled reanalysis (D), potential confounders and risk factors can be more easily considered (7), as they are usually only published in an aggregated form. With the individual data, the outcome parameters, risk factors, and confounders used in the analysis can be categorized in a standard manner and properly incorporated in the analysis. Individual data can be removed in accordance with the prospective specifications in the study protocol, without it being necessary to exclude the whole study. The disadvantages of pooled reanalysis are that it demands a great deal of time and money and that it is dependent on the willingness of the authors of the individual studies to cooperate. If not all authors send their individual data, this may result in biased results.

The level of evidence of the type of summary increases from the systematic review to the meta-analysis of published data to the pooled reanalysis. It is important that all three forms of summary should be performed with high quality.

Key messages.

• The various forms of summary can be categorized as systematic review articles, meta-analyses of published data, and pooled reanalyses.

• Systematic review articles can provide a rapid overview of the status of research on a specific topic.

• Meta-analyses of published data and pooled reanalyses additionally permit the calculation of pooled effect estimates.

• Pooled reanalyses allow a detailed evaluation on the basis of individual data.

• Like any original study, all these types of summary must have an a priori study protocol, laying down in detail the research questions, the hypothesis, the literature search, the inclusion and exclusion criteria, and the analysis strategies.