Meta-Analysis: Drawing Conclusions When Study Results Vary

Summary

Low-dose aspirin has been suggested to positively impact a number of clinical outcomes associated with oxidative stress; however, results of clinical trials surrounding its effect on a woman’s ability to achieve and sustain pregnancy have been inconclusive. A meta-analysis is an advantageous tool in this situation. Meta-analyses allow researchers to formally and systematically pool together all relevant research in order to clarify findings and form conclusions based on all currently available information. The purpose of this chapter is to describe how to perform a meta-analysis, clarify the impact of model selection, and provide examples of implementation.

1. Introduction

Because of its low cost, wide availability, and apparent safety, low-dose aspirin has become an increasingly popular therapy to consider for a number of conditions requiring a treatment with anti-inflammatory, vasodilatory, and platelet aggregation inhibition properties. Its efficacy has been and is currently being investigated in relation to improving health issues ranging from cardiovascular disease (CVD) (1) to reproduction and sub fertility (2–10). Elevated levels of oxidative stress (OS) are associated with both of these conditions, as well as their respective risk factors. Low-dose aspirin has been suggested to have a cytoprotective effect against OS. As a result, improving understanding of how low-dose aspirin use affects oxidative stress may translate to improved understanding of its impact on corresponding clinical outcomes.

While there has been a lot of research concerning the effects of low-dose aspirin use on both individuals at risk for CVD and individuals demonstrating signs of sub fertility who desire to become pregnant, the results of the research surrounding its effect on a woman’s ability to achieve and sustain pregnancy have been inconsistent. These discrepant findings and/or findings occurring in trials that lack significant power have made it difficult to draw conclusions concerning the hypothesized benefits and, in turn, form inferences concerning the effects on oxidative stress.

Under circumstances like this where results of different studies are conflicting or inconclusive, it can be advantageous to perform a meta-analysis. Meta-analyses allow researchers to formally and systematically pool together all relevant research in order to clarify findings and form conclusions based on all currently available information. Essentially, this procedure allows researchers to address: “Was an effect seen in the trials included in the analysis?” or “Will an effect be seen in a given trial in the future?” (11). The purpose of this chapter is to describe how to perform a meta-analysis, clarify the impact of model selection, and provide examples of implementation.

2. Materials

Software for data extraction can be accessed through The Cochrane Collaboration website: http://www.cochrane.org (12). As noted in the procedure section of this chapter, data extraction should be performed with the use of structured forms, which are made available through this website.

3. Methods

3.1. Procedure

A meta-analysis is comprised of four main steps: identifying pertinent studies/ trials, determining criteria for inclusion and exclusion, data extraction, and data analysis (11).

1. To begin, one must first seek out all information concerning studies or trials relevant to the topic of interest. In order to ensure the scientific integrity of a meta-analysis, however, this must be done in a systematic and explicit manor to allow future researchers who follow the same procedure to achieve the same results. This generally begins with a thorough search of multiple electronic databases, for example, MEDLINE. All articles are then read to determine if they are applicable to the topic at hand. Next, the references of selected studies are searched manually to ensure recovery of any articles related to studies not identified in the computerized search. For completeness, this step is repeated for any new studies identified. Moreover, the list is often verified through submission to an expert in the field, or by having a second independent researcher perform the search.

2. The second step is clearly defining the criteria for inclusion and exclusion of studies from the meta-analysis. Explicitly stating these criteria helps to guarantee reproducibility as stated in step one, as well as avoid unnecessary sources of bias (see Note 1). Such criteria include, but are not limited to, factors surrounding the design of a study or trial, the presentation of the study or trial, and treatments undergone in a study or trial.

3. Data from the identified studies (or trials) are then extracted for both determining eligibility and analyzing. All data extraction should be done with the use of structured forms. Details on data combination are described elsewhere (13).

4. Data from those studies (or trials) included in the meta-analysis are compiled and analyzed statistically. Analysis is performed with the use of either the fixed-effects (Mantel–Haenszel (14)) or the random-effects (DerSimonian and Laird (15)), or in rare cases the Peto (16) model to determine both a test statistic and confidence interval (CI), as well as any other relevant information (17). Table 30.1 provides formulas for these methods (see Notes 2 and 3).

Table 30.1.

Formulas for calculating odds ratios and their corresponding 95% confidence intervals

	Model
	Mantel–Haenszel	Peto	Random-effects
ORM	$\frac{{\displaystyle \sum }({\text{weight}}_{\mathrm{i}}\times {\text{OR}}_{\mathrm{i}})}{{\displaystyle \sum }{\text{weight}}_{\mathrm{i}}}$	eΣ (Oi − Ei) / Σ variancei	${}_{\mathrm{e}}{\displaystyle \sum }(w_{\mathrm{i}}^{*}\times {\text{ln}\text{OR}}_{\mathrm{i}})/{\displaystyle \sum (w_{\mathrm{i}}^{*})}$
	${\text{OR}}_{\mathrm{i}}=\frac{(a_{\mathrm{i}}\times d_{\mathrm{i}})}{(b_{\mathrm{i}}\times c_{\mathrm{i}})}$	${\text{ln}\text{OR}}_{\mathrm{P}}=\frac{{\displaystyle \sum }(O_{\mathrm{i}}-E_{\mathrm{i}})}{{\displaystyle \sum }{\text{variance}}_{\mathrm{i}}}$	${\text{ln}\text{OR}}_{\text{RE}}=\frac{{\displaystyle \sum }(w_{\mathrm{i}}^{*}\times {\text{ln}\text{OR}}_{\mathrm{i}})}{{\displaystyle \sum }(w_{\mathrm{i}}^{*})}$
	${\text{weight}}_{\mathrm{i}}=\frac{1}{{\text{variance}}_{\mathrm{i}}}$	$E_{\mathrm{i}}=\frac{(e_{\mathrm{i}}\times g_{\mathrm{i}})}{n_{\mathrm{i}}}$	$w_{\mathrm{i}}^{*}=\frac{1}{[D+1/w_i]},w_{\mathrm{i}}=\frac{1}{{\text{variance}}_{\mathrm{i}}}$
variancei	$\frac{n_{\mathrm{i}}}{b_{\mathrm{i}}\times c_{\mathrm{i}}}$	$\frac{(E_{\mathrm{i}}\times f_{\mathrm{i}}\times h_{\mathrm{i}})}{(n_{\mathrm{i}}\times (n_{\mathrm{i}}-1))}$	$\frac{n_{\mathrm{i}}}{b_{\mathrm{i}}\times c_{\mathrm{i}}}$
95% Confidence Interval	${\mathrm{e}}^{{\text{lnOR}}_{\text{mh}}\pm 1.96\sqrt{{\text{variance OR}}_{\text{mh}}}}$	${\mathrm{e}}^{{\text{lnOR}}_{\mathrm{P}}\backslash \text{pm}\pm 1.96\sqrt{{\text{variance}}_{\mathrm{i}}}}$	${\mathrm{e}}^{{\text{lnOR}}_{\text{dl}}+1.96\backslash \text{pm}\times \sqrt{{\text{variance}}_{\mathrm{s}}^{*}}},{\text{variance}}_{\mathrm{s}}^{*}={\displaystyle \sum }{\text{weight}}_{\mathrm{i}}^{*}$

The fixed-effects model weights the studies by the inverse of the variance of estimates and produces a CI that takes into account the random variation within each trial. This model aims to understand: “Was an effect seen in the trials included in the analysis?” Fixed-effects model weighting thus constitutes a direct synthesis of the literature (11, 18).

Conversely, the random-effects model includes a between-study component of variance. It is sometimes thought to be a more conservative approach, because the CIs produced are wider than those produced from the fixed-effects model due to the contribution of the inter-study variability to the total variability. ‘Random-effects models use study power to estimate inter-study covariance, and the weighting of each study in the model is determined by both intra-study and inter-study variance. Accordingly, the random-effects model attempts to answer the question: “Will an effect be seen in a given trial in the future?”’ (11, 18).

Thoughtful consideration should be used in determining which model is most appropriate for statistical analysis. While the random-effects model can sometimes be considered more conservative, its use is accompanied by a loss of precision. Moreover, the strong assumptions it requires may not be valid in practice. “Petitti suggests that the random-effects model should be used only when the absence of inter-study heterogeneity can be assumed, because any significant between-study heterogeneity dominates the weights assigned to the studies (11). As a consequence, in the presence of substantial between-study heterogeneity, small and large trials become weighted the same, and the summary statistic is greatly affected by the inclusion of small trials in the analysis. The fixed-effects model, however, takes into account only between-study variances and thus weights studies according to [their] sample size.” (18)

3.2. Applications and Results

1. In response to inconsistent evidence, Gelbaya et al. conducted a meta-analysis to examine the impact of low-dose aspirin use on women undergoing in vitro fertilization (IVF). They began by “[searching] four electronic databases – MEDLINE, EMBASE, Cochrane Controlled Trials Register (CENTRAL), and The UK National Research Register of ongoing and completed research projects undertaken in or for the UK National Health Service – from January 1980 to March 2006 using the key words ‘(aspirin or acetylsalicylic acid) and (IVF or ICSI).’” (19). To follow, they performed manual searches consistent with the above listed recommendations, and finally verified the findings through an independent search by a second investigator.

   They outlined the inclusion and exclusion criteria, such as they would include “studies investigating the effect of low-dose aspirin alone or in conjunction with heparin or glucocorticoids on IVF or ICSI outcome” (19) and there would not be any language restrictions. As a result, they had six trials to be included in the meta-analysis. After extracting the data, they determined that due “to significant heterogeneity among trials, [they would use] the random-effects model (15) to derive the summary estimates of the effect of treatment” (19).

   Statistical analysis was conducted to examine the rate of clinical pregnancy (CP) per embryo transfer (ET) between individuals ingesting daily low-dose aspirin and those who received a placebo, or, no treatment. Their results indicated no statistically significant “[difference] between aspirin and no treatment groups (RR 1.09, 95% CI 0.92–1.29)” (19). Furthermore, they had similar findings when they examined other outcomes of interest.

2. To assess and evaluate the conclusions mentioned above, Ruopp et al. conducted a reanalysis of the work described above (18). We began in a similar fashion, having two reviewers conduct independent searches of MEDLINE, Web of Science, EMBASE, TOXLINE, DART, and the Cochrane Database of Systematic Reviews and following through with a manual search of the references for each selected trial/ study.

   Inclusion criteria included prospective trials using low-dose aspirin (<150 mg) during IVF randomized or matched controlled trials. Unlike Gelbaya et al., abstracts from conferences, subgroups of infertile patients, and frozen ET were included; however, Roupp et al. only included those written in English.

   After determining which studies were to be included, we extracted and analyzed the data using 2 × 2 structured forms and Cochrane Review Manager Software (version 4.1; Update Software, Oxford, England), respectively. We calculated odds ratios, 95% confidence intervals, and heterogeneity. As opposed to Gelbaya et al., Ruopp et al. assumed fixed effects in utilizing tthe Mantel–Haenszel method to calculate heterogeneity. This allowed us to pool data from all of the studies and assign weights on the basis of sample size.

   From analysis of the ten studies that met the selection criteria as well as a subgroup analysis evaluating only the studies included in Gelbaya et al., Ruopp et al. arrived at somewhat different conclusions. The results of this analysis are illustrated in Table 30.2, from Ruopp et al. When comparing clinical pregnancy per embryo transfer rates between the low-dose aspirin and no-treatment groups, Ruopp et al. found a significant risk ratio of 1.15 with data from all ten studies, and a risk ratio of 1.12 in the subgroup of studies selected by Gelbaya et al. (both greater than the value estimated by Gelbaya et al.). This and other similar results led Ruopp et al. to conclude that the effect of low-dose aspirin on women undergoing in vitro fertilization is still uncertain (18).

   Variation between the two meta-analyses described above can be attributed to differences in statistical modeling, study selection, and statistical inference. Table 30.2 highlights these differences demonstrating how model selection can lead to different results in some cases. And, while the variation is not necessarily substantial, in some cases this can determine whether or not findings are considered significant.

Table 30.2.

Summary statistics and CIs using fixed-and random-effects models for full and subgroup analysis

Outcome	Model
	Fixed effects, risk ratio (95% CI)	Random effects, risk ratio (95% CI)
Full analysis: all studies
Pregnancy rate	1.15 (1.03, 1.27)	1.14 (0.95, 1.35)
Implantation rate	1.08 (0.69, 1.71)	1.00 (0.34, 2.93)
Miscarriage rate	1.19 (0.86, 1.65)	1.18 (0.85, 1.64)
Subgroup analysis: studies including only fresh ET
Pregnancy rate	1.16 (1.04, 1.29)	1.15 (0.98, 1.36)
Implantation rate	1.32 (0.81, 2.16)	1.49 (0.50, 4.46)
Miscarriage rate	1.19 (0.86, 1.65)	1.18 (0.85, 1.64)
Subgroup analysis: studies included by Gelbaya et al.
Pregnancy rate	1.12 (1.00, 1.25)	1.09 (0.92, 1.29)
Implantation rate	0.89 (0.48, 1.66)	0.89 (0.48, 1.66)
Miscarriage rate	1.17 (0.84, 1.63)	1.17 (0.84, 1.63)