Statistical Methods for Quantifying Between-study Heterogeneity in Meta-analysis with Focus on Rare Binary Events

Abstract

Meta-analysis, the statistical procedure for combining results from multiple independent studies, has been widely used in medical research to evaluate intervention efficacy and drug safety. In many practical situations, treatment effects vary notably among the collected studies, and the variation, often modeled by the between-study variance parameter τ², can greatly affect the inference of the overall effect size. In the past, comparative studies have been conducted for both point and interval estimation of τ². However, most are incomplete, only including a limited subset of existing methods, and some are outdated. Further, none of the studies covers descriptive measures for assessing the level of heterogeneity, nor are they focused on rare binary events that require special attention. We summarize by far the most comprehensive set including 11 descriptive measures, 23 estimators, and 16 confidence intervals. In addition to providing synthesized information, we further categorize these methods according to their key features. We then evaluate their performance based on simulation studies that examine various realistic scenarios for rare binary events, with an illustration using a data example of a gestational diabetes meta-analysis. We conclude that there is no uniformly “best” method. However, methods with consistently better performance do exist in the context of rare binary events, and we provide practical guidelines based on numerical evidences.

1. Introduction

Meta-analysis, the statistical procedure for synthesizing information from multiple studies, has been widely used in many research areas including social, psychological and especially medical sciences. Meta-analysis is a powerful tool in drug safety evaluation, where the number of cases (adverse events) can be very limited in a single study. The U.S. Food and Drug Administration (FDA) released a draft guidance for industry titled “Meta-Analyses of Randomized Controlled Clinical Trials to Evaluate the Safety of Human Drugs or Biological Products” in November 2018, which demonstrates the importance of meta-analysis in the development of new drugs. Such meta-analysis often involves binary outcomes of rare events, which are the focus of this study.

The primary goal of a meta-analysis is usually to estimate and infer the overall effect size, where the variability in the effect estimates from component studies should be properly accounted for. Besides the within-study sampling errors, the variability may come from diverse characteristics of individual studies such as disparities in trial protocols, subjects’ conditions, and population features, etc. When the study-wise differences exist, we call these studies (statistically) heterogeneous and the heterogeneity is typically measured by a between-study variance parameter τ². Also, descriptive measures have been widely used by clinicians to provide a more intuitive interpretation about the heterogeneity for ease of understanding.

For point estimation of τ², the DerSimonian and Laird (DL) estimator [10], most widely used in the field, has been frequently challenged for its default use in many software packages, largely due to its sizable negative bias when the heterogeneity level is high [39, 47, 2, 35, 34]. Many modifications over the DL estimator have been suggested based on the method of moments. Other approaches such as likelihood-based and other nonparametric methods can also be applied. For interval estimation of τ², different types of confidence intervals (CIs) have been constructed to gauge the estimation uncertainty. However, nearly all these methods were constructed without a special consideration of dichotomous data and their performance remains unclear in the context of rare binary events, in which some may produce large bias or even fail to work.

Comparative studies and review papers exist for both point and interval estimation of τ² but not for descriptive measures. For example, Veroniki et al. [46], Langan et al. [28], Petropoulou and Mavridis [37] reviewed and compared most of the existing estimators of τ², among which only Petropoulou and Mavridis [37] conducted simulation studies to evaluate their performance. Previous comparisons about CIs (e.g., [48, 25, 45]) were largely limited to several similar types of CIs. As detailed in Tables 2 and 5, none of these papers covers descriptive measures for quantifying the level of heterogeneity, nor do they focus on rare binary events. And most of them are far from being complete, some even outdated, which motivates us to conduct this study to provide useful guidance to clinicians and bio-statisticians.

Table 2:

Overview of 23 estimators for the between-study variance τ²

Estimators	Abbreviation	Reference	Iterative?	Sign	Effect Measure
Method of Moments
Hedges and Olkin	HO	[17]	No	> = 0
Two-step Hedges and Olkin	HO2	DerSimonian and Kacker [9]	No	> = 0
DerSimonian and Laird	DL	DerSimonian and Laird [10]	No	> = 0
Positive DerSimonian and Laird	DLp	Kontopantelis et al. [27]	No	> 0
Two-step DerSimonian and Laird	DL2	DerSimonian and Kacker [9]	No	> = 0
Multistep Dersimonian and Laird	DLM	van Aert and Jackson [44]	No	> = 0
Paule and Mandel	PM	Paule and Mandel [36]	Yes	> = 0
Improved Paule and Mandel	IPM	Bhaumik et al. [2]	Yes	> = 0	OR
Hartung and Makambi	HM	Hartung and Makambi [16]	No	> 0
Hunter and Schmidt	HS	Hunter and Schmidt [20]	No	> = 0
Lin, Chu and Hodges	LCH	Lin et al. [31]	No	> = 0
Likelihood-based
Maximum Likelihood	ML	Hardy and Thompson [14]	Yes	> = 0
Restricted maximum likelihood	REML	Viechtbauer [47]	Yes	> = 0
Approximate restricted maximum likelihood	AREML	Morris [33]	Yes	> = 0
Model error variance (Least squares)
Sidik and Jonkman	SJ	Sidik and Jonkman [39]	No	> 0
Sidik and Jonkman (HO prior)	SJHO	Sidik and Jonkman [40]	No	> 0
Bayesian
Rukhin Bayes	RB0	Rukhin [38]	Yes	> = 0
Positive Rukhin Bayes	RBp	Rukhin [38]	Yes	> 0
Empirical Bayes (Equivalent to PM)	EB	Morris [33]	Yes	> = 0
Fully Bayes	FB	Smith et al. [41]	Yes	> 0
Bayes Modal	BM	Chung et al. [6, 5]	Yes	> 0
Other nonparametric
Malzahn, Böhning, and Holling	MBH	Malzahn et al. [32]	No	> = 0	SMD
Non-parametric bootstrap DerSimonian and Laird	DLb	Kontopantelis et al. [27]	No	> = 0

Table 5:

Existing comparative studies on constructing CIs for τ² in random-effects meta-analysis

Review paper	CI methods reviewed/compared	Effect measure	Recommendations
Knapp et al. [25]	QP, MQP, BT, PLML, WML	MD/OR	QP and MQP
Viechtbauer [48]	QP, BT, PL, W, SJ, BS	OR	QP
Veroniki et al. [46]	PL, W, BT, BJ, J, QP, SJ, BS, BC	Generic	—
van Aert et al. [45]	QP, BJ, J	OR	None recommended when pki < 0.1 in combination with either K ≥ 80 or (K ≥ 40 and nki < 30)

The paper is organized as follows. In Section 2, we introduce notation and frequently used terms in meta-analysis. Section 3 reviews existing descriptive measures quantifying the level of heterogeneity. In Section 4, we list estimators for τ² and briefly summarize two recently developed ones that are not included in any of the existing review papers. In Section 5, different types of confidence intervals for τ² are described and categorized. In Section 6, we compare the performance, in terms of bias and mean squared error (MSE) for point estimators and empirical coverage probability and width for CIs, in a large collection of scenarios that are designed to mimic practical situations. In Section 7, we re-analyze the data from a meta-analysis [1] of 20 trials of type 2 diabetes mellitus after gestational diabetes with focus on the heterogeneity among the component studies. The final section provides recommendations in terms of choosing appropriate estimators and CIs in meta-analysis of rare binary events as well as a brief discussion.

2. Notation & frequently used terms

Suppose a meta-analysis includes K independent studies and the kth study contains n_k subjects (k = 1, …, K). In study k, let θ_k be the true but unknown treatment effect and y_k be the observed treatment effect such that E[y_k|θ_k] = θ_k and $\text{Var}\left[y_k\mid {\theta }_k\right]={\sigma }_k^2$, the within-study variance. Typically $s_k^2$, an estimate of ${\sigma }_k^2$, is reported along with y_k in published studies and it is often treated as a known quantity in practice (i.e., indistinguishable from ${\sigma }_k^2$). When the study-specific effects θ_k’s are treated as random variables rather than constants, we assume E[θ_k] = θ and Var[θ_k] = τ², where θ, a parameter of main interest in the meta-analysis, represents the overall treatment effect across different studies, and τ² measures the between-study heterogeneity. There exist two main parametric models, namely Re (random effect) and Fe (fixed effect), to combine results from component studies. The Re model assumes that y_k = θ_k + ϵ_k, where θ_k ~ N(θ, τ²) and ${ϵ}_k~N\left(0,{\sigma }_k^2\right)$. When τ² = 0, it is reduced to the Fe model y_k = θ + ϵ_k, where a common treatment effect θ is assumed for all component studies (i.e., θ_k ≡ θ). These models can be used with any effect measure, as long as the assumed normality is (approximately) valid.

For binary responses, we denote the number of events by x_k0 (x_k1) and the number of subjects by n_k0 (n_k1) in the control (treatment) group. The probability of having an event in the control (treatment) group is denoted by p_k0 (p_k1). Effect measures for binary outcomes include risk difference (RD, p_k1 − p_k0), risk ratio (RR, p_k1/p_k0) and odds ratio (OR, [p_k1/(1 − p_k1)]/[p_k0/(1 − p_k0)]). For rare binary events, RR≈OR. A logarithm transformation of the odds ratio (LOR) is often used in meta-analysis for a much faster convergence to asymptotic normality, and the within-study variance ${\sigma }_k^2$ is then estimated by $s_k^2=\frac{1}{x_{k0}}+\frac{1}{n_{k0}-x_{k0}}+\frac{1}{x_{k1}}+\frac{1}{n_{k1}-x_{k1}}$. Gart [13] added a continuity correction factor of 0.5 to all the cells so that

$$y_k=\text{log}\frac{x_{k1}+0.5}{n_{k1}-x_{k1}+0.5}-\text{log}\frac{x_{k0}+0.5}{n_{k0}-x_{k0}+0.5},$$

and ${\sigma }_k^2$ is estimated by

$$s_k^2=\frac{1}{x_{k0}+0.5}+\frac{1}{n_{k0}-x_{k0}+0.5}+\frac{1}{x_{k1}+0.5}+\frac{1}{n_{k1}-x_{k1}+0.5},$$

which will be used in our numerical evaluation of rare binary events.

Next, we introduce the (generalized) Q statistic [9] and related terms, which will frequently appear in the paper. For any parameter of interest, we use the corresponding letter/symbol with a hat to denote its estimate. For example, we use $\widehat{\theta }$ to denote the estimate of the overall treatment effect θ. The Q statistic is defined as the weighted sum of squared deviations between the estimated overall treatment effect and observed treatment effect in each individual study, namely

$$Q=\sum _{k=1}^K{w}_k{\left(y_k-\widehat{\theta }\right)}^2,$$ (1)

where w_k is a positive weight assigned to study k, and $\widehat{\theta }={\sum }_{k=1}^K{w}_k{y}_k/{\sum }_{k=1}^K{w}_k$, the weighted average of the estimated study-specific effects. A commonly used weighting scheme is to set $w_k={\left[\widehat{\text{Var}}\left(y_k\right)\right]}^{-1}$, i.e., the inverse of the estimated variance of y_k. Under this inverse-variance weighing scheme, the variance of $\widehat{\theta }$ can be given by $1/{\sum }_{k=1}^K{w}_k$ if we treat w_k’s as known constants (i.e., indistinguishable from [Var(y_k)]⁻¹). Further, this scheme yields $w_k=1/s_k^2$ for the Fe model, and $w_k=1/\left(s_k^2+{\widehat{\tau }}^2\right)$ for the Re model, where ${\widehat{\tau }}^2$ can be any estimator discussed in Section 4. Under the Fe (Re) model with the inverse-variance weights, we denote the corresponding Q statistic by Q_Fe (Q_Re) and the corresponding $\widehat{\theta }$ by ${\widehat{\theta }}_{Fe}\left({\widehat{\theta }}_{Re}\right)$ with variance v_Fe (v_Re). In fact, the Cochran’s Q statistic is Q_Fe, also known as the DerSimonian and Laird’s Q test statistic [10].

Throughout this paper, we use ${\chi }_{df}^2$ to denote a chi-squared distribution with df degrees of freedom, and use ${\chi }_{df,\alpha }^2$ to denote its 100α-th percentile.

3. Descriptive measures quantifying between-study heterogeneity

As mentioned in the introduction, (statistical) heterogeneity exists when true effects being evaluated differ among studies in a meta-analysis. Assessing the extent of heterogeneity is essential for model selection between Fe and Re models and decision making. An obvious choice is by estimating the variance parameter τ², as is typically done in a random-effects meta-analysis. As pointed out by Higgins and Thompson [18], this measure does not facilitate comparison of heterogeneity across meta-analyses of different types of outcomes (e.g., the survival time can be either continuous or discrete). Also, its scale is specific to a chosen effect metric and the interpretation can be difficult. For example, odds ratio is a commonly used effect measure for binary data. Still, the variance of log-odds ratio is not easy to understand for many non-statisticians. Alternatively, one may test the existence of the between-study heterogeneity (e.g., through Cochran’s Q-test [7]), and use the corresponding test statistic or p-value to indicate the extent of heterogeneity. However, such measures depend on the scale of effect sizes or the number of component studies K. To overcome these limitations, effort has been devoted to development of various descriptive measures that can provide more intuitive information about the heterogeneity.

Table 1 summarizes 11 descriptive heterogeneity measures in the literature. Note that all these measures are general-purpose and none is specifically designed for binary outcomes. Takkouche et al. [42] proposed two measures, R_I and CV_B, to quantify the level of heterogeneity in five published meta-analyses. The statistic R_I was developed to estimate τ²/(τ²+σ²), the proportion of total variation in the effect estimates that is due to between-study heterogeneity. This quantity is also known as the intra-class correlation in the context of cluster sampling. Here, the within-study variances ${\sigma }_k^2’\text{s}$ are assumed to be constant, i.e., ${\sigma }_k^2\equiv {\sigma }^2$, which is estimated by $1/{\sum }_{k=1}^{K}1/s_k^2$, making $R_I=\frac{{\widehat{\tau }}^2}{{\widehat{\tau }}^2+K/{\sum }_{k=1}^{K}1/s_k^2}$. The other statistic CV_B estimates the between-study coefficient of variation τ/|θ| by $\sqrt{{\widehat{\tau }}^2}/\left|\widehat{\theta }\right|$. Obviously, CV_B is affected by the overall treatment effect θ and is undefined when θ = 0.

Table 1:

Descriptive measures quantifying the between-study heterogeneity

Name	f (θ, τ2, σ2, K)	Formula	Ref.	Interpretation	Assume ${\sigma }_k^2\equiv {\sigma }^2?$
RI	$\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$	$\frac{{\widehat{\tau }}^2}{{\widehat{\tau }}^2+K/{\sum }_{k-1}^{K}1/s_k^2}$	[42]	Proportion of total variation in the estimates of treatment effect due to between-study heterogeneity	Yes
CVb	$\frac{\tau }{\left|\theta \right|}$	$\frac{\sqrt{{\widehat{\tau }}^2}}{\left|\widehat{\theta }\right|}$	[42]	Between-study coefficient of variation	No
H2	$\frac{{\tau }^2+{\sigma }^2}{{\sigma }^2}$	$\frac{Q_{Fe}}{K-1}$	[18]	Relative excess in QFe over its degrees of freedom	Yes, but can be used for different ${\sigma }_k^2$.
R2	$\frac{{\tau }^2+{\sigma }^2}{{\sigma }^2}\approx \frac{v_{Re}}{v_{Fe}}$	$\frac{{\sum }_{k=1}^K\frac{1}{s_k^2}}{{\sum }_{k=1}^K\frac{1}{s_k^2+{\widehat{\tau }}^2}}$	[18]	Inflation in the confidence interval for a single summary estimate under Re model compared with Fe model	Yes, but can be used for different ${\sigma }_k^2$.
$I_{HT}^2$	$\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$	$1-\frac{K-1}{Q_{Fe}}$	[18]	Same as RI	Yes
$I_R^2$	$\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$	$1-\frac{{\sum }_{k=1}^K\frac{1}{s_k^2+{\widehat{\tau }}^2}}{{\sum }_{k=1}^K\frac{1}{s_k^2}}$	[24]	Same as RI	Yes
Rb	$\frac{{\tau }^2}{v_{Re}}\approx \frac{1}{K}{\sum }_{k=1}^K\frac{{\tau }^2}{{\sigma }_k^2+{\tau }^2}$	$\frac{1}{K}{\sum }_{k=1}^K\frac{{\widehat{\tau }}^2}{s_k^2+{\widehat{\tau }}^2}$	[8]	Proportion of the between-study heterogeneity τ2 relative to vRe, the variance of ${\widehat{\theta }}_{Re}$.	No
$H_r^2$	$\frac{{\tau }^2+{\sigma }^2}{{\sigma }^2}$	$\frac{\pi Q_r^2}{2K(K-1)}$, $Q_r={\sum }_{k=1}^K\frac{1}{s_k}\left|y_k-{\widehat{\theta }}_{Fe}\right|$	[31]	Same as H2	Yes
$I_r^2$	$\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$	$1-\frac{2K(K-1)}{\pi Q_r^2}$	[31]	Same as RI	Yes
$H_m^2$	$\frac{{\tau }^2+{\sigma }^2}{{\sigma }^2}$	$\frac{\pi Q_m^2}{2K^2}$, $Q_m={\sum }_{k=1}^K\frac{1}{s_k}\left|y_k-{\widehat{\theta }}_m\right|$, ${\widehat{\theta }}_m$ is weighted median estimate	[31]	Same as H2	Yes
$I_m^2$	$\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$	$\frac{Q_m^2-2K^2/\pi }{Q_m^2}$	[31]	Same as RI	Yes

Under the assumption of a common within-study variance σ², Higgins and Thompson [18] formulated a general heterogeneity measure as a function of the overall treatment effect θ, the between-study variance τ², the within-study variance σ², and the number of component studies, namely, f(θ, τ², σ², K). They proposed three criteria that such a measure should satisfy in general in order to facilitate its comparability and interpretability, including (i) dependence on the extent of heterogeneity, (ii) scale invariance, i.e. f(θ, τ², σ², K) = f(a + bθ, b²τ², b²σ², K) for any a and b, and (iii) size invariance, i.e. f(θ, τ², σ², K₁) = f(θ, τ², σ², K₂) for any positive integers K₁ and K₂. Criterion (i) implies that the function f should increase monotonically with τ². Criterion (ii) implies that f should be a function of the ratio $\rho \equiv \frac{{\tau }^2}{{\sigma }^2}$ and that θ should not be involved. Criterion (iii) implies that f does not depend on K. It can be shown that any monotonically increasing function of ρ satisfies the three criteria. Based on this, three statistics, H², R² and I² were proposed. The first, H², estimates the quantity ρ + 1 by equating the observed value of Q_Fe to its expectation so that $H^2=\frac{Q_{Fe}}{K-1}$ can be interpreted as relative excess in Q_Fe over its expected value, the degrees of freedom K −1. The second, R², attempts to estimate ρ+1 as well; but here, ρ+1 is approximated by v_Re/v_Fe so that $R^2={\widehat{v}}_{Re}/{\widehat{v}}_{Fe}={\sum }_{k=1}^K\frac{1}{s_k^2}/{\sum }_{k=1}^K\frac{1}{s_k^2+{\widehat{\tau }}^2}$, which can be interpreted as the inflation in the confidence interval for ${\widehat{\theta }}_{Re}$ under the Re model compared with ${\widehat{\theta }}_{Fe}$ under the Fe model. Both H² and R² should be at least 1, where 1 means perfect homogeneity; and the larger the value, the more heterogeneous the studies. In practice, the authors suggested to use H and R because clinicians may be more familiar with standard deviations than variances. The third statistic, I², estimates a different function of ρ, i.e. $\frac{\rho }{1+\rho }=\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$, which represents the proportion of total variance that is due to between-study variation. Higgins and Thompson [18] suggested to compute I² by $I_{HT}^2=1-\frac{K-1}{Q_{Fe}}$, which leads to a convenient relationship $I_{HT}^2=1-\frac{1}{H^2}$. Jackson et al. [24] suggested to compute I² by $I_R^2=1-\frac{{\widehat{v}}_{Fe}}{{\widehat{v}}_{Re}}=1-{\sum }_{k=1}^K\frac{1}{s_k^2+{\widehat{\tau }}^2}/{\sum }_{k=1}^K\frac{1}{s_k^2}$, which leads to another convenient relationship $I_R^2=1-\frac{1}{R^2}$. Both $I_{HT}^2$ and $I_R^2$ are usually expressed as percentages between 0% and 100%, where a value of 0% corresponds to no observed heterogeneity, while larger values indicate increasing levels of heterogeneity. They estimate the same quantity as R_I does, but with different within-study variance estimates. Among these measures (i.e. H², R², $I_{HT}^2$ or $I_R^2$), $I_{HT}^2$ is most popular and in the literature, I² typically represents $I_{HT}^2$ as $I_R^2$ is much less known. Higgins and Green [19] empirically provided a rough guide to the interpretation of I² using overlapping intervals: a value in [0,0.4] suggests that heterogeneity may not be that important; [0.3, 0.6] may represent moderate heterogeneity; [0.5,0.9] may represent substantial heterogeneity; and [0.75,1] implies considerable heterogeneity.

The assumption of a constant within-study variance is probably untrue in many real life data. Thus, Crippa et al. [8] lifted this assumption and proposed a new measure R_b, defined as $R_b=\frac{1}{K}{\sum }_{k=1}^K\frac{{\widehat{\tau }}^2}{s_k^2+{\widehat{\tau }}^2}$, to assess the contribution of the between-study variance τ² to v_Re (i.e., the variance of the pooled random-effects estimate ${\widehat{\theta }}_{Re}$). It can be viewed as an average of the study-specific proportions of the study-specific variances due to between-study heterogeneity. They showed that the quantity τ²/v_Re underlying R_b is a strictly increasing function of τ² and is scale-invariant. However, this quantity depends on K and so is not size-invariant. They further showed that R_I ≥ max(R_b, $I_{HT}^2$). When ${\sigma }_k^2\equiv {\sigma }^2$ and σ² is estimated by s², R_b, R_I, $I_{HT}^2$ and $I_R^2$ all yield the same quantity $\frac{{\widehat{\tau }}^2}{s^2+{\widehat{\tau }}^2}$. The authors conducted a simulation study to examine the performance of R_I, $I_{HT}^2$ and R_b. Both R_I and $I_{HT}^2$ tend to be positively biased and this overestimation increases as K increases. Confidence intervals based on R_I and $I_{HT}^2$ give lower coverage probabilities compared to those based on R_b and the difference becomes more obvious when the within-study variances vary more and when the heterogeneity level increases.

To reduce the impact of outlying studies, Lin et al. [31] proposed new robust measures $H_r^2$, $H_m^2$, $I_r^2$ and $I_m^2$, which are analogous to and have the same interpretations as H² and I², respectively. These methods were developed upon the absolute deviation measures Q_r and Q_m rather than the usual squared deviation measure Q, as defined in Table 1 and will be described in more detail in Section 4.

All the measures except for CV_B depend on the precision of the study-specific effects. As the sample sizes of the component studies increase, ${\sigma }_k^2’\text{s}$ would decrease to zero so that R_I, R_B and all I²’s would increase to 1 and all H²’s and R² would become arbitrarily large, even when there is little between-study heterogeneity. The measure CV_B avoids this drawback but has its own limitation: it would approach +∞ as θ goes to 0. Finally, we mention that some of the measures involve the estimated value ${\widehat{\tau }}^2$. In principle, ${\widehat{\tau }}^2$ can be any estimator of τ², but most software uses the DL estimator ${\widehat{\tau }}_{DL}^2$ as the default choice.

4. Estimators

We summarize 23 estimators for τ² in Table 2, among which most can be applied to all kinds of effect measures except for the improved Paule and Mandel estimator (IPM, [2]) and Malzahn, Böhning, and Holling (MBH, [32]). IPM is specifically designed to work with OR for binary outcomes, and MBH can be only used for standardized mean difference (SMD). All estimators can be divided into five groups: method of moments, likelihood-based, model error variance (least squares), Bayes, and other nonparametric estimators. Some have closed form expressions while the others require numerical solutions. Some produce only positive estimates while the others require truncation to zero when a negative value occurs. Some properties of the estimators are summarized in Table 2.

Table 3 shows previous studies that reviewed and compared (large) subsets of these estimators. Recommendations were made either based on their own simulations or conclusions from the literature. Among them, Veroniki et al. [46], Langan et al. [28] and Petropoulou and Mavridis [37] are the most comprehensive. Veroniki et al. [46] reviewed 17 estimators as listed in Table 3, including all the method of moments estimators except for the IPM, multistep DL and LCH estimators, all three likelihood-based estimators, the SJ estimator, all the Bayesian estimators, and DL_b. Langan et al. [28] and Petropoulou and Mavridis [37] added IPM, MBH, and SJ_HO into the comparison. Note that IPM was briefly summarized but not compared with other estimators in Veroniki et al. [46]. Also, EB mentioned in [37] has been shown to be equivalent to PM. Langan et al. [28] also added RB estimators with different priors, RB_u and RB_a.

Table 3:

Existing comparative studies for various estimators of the between-study variance τ².

Review paper	Estimators compared	Effect measure	Recommendations
Viechtbauer [47]	HO, DL, HS, ML, REML	SMD and MD	REML
Sidik and Jonkman [40]	HO, DL, SJ, SJHO, ML, REML, EB	OR	SJHO when τ2 is expected to be small or moderate; SJ when τ2 is expected to be large.
Kontopantelis et al. [27]	HO, HO2, DL, DL2, DLb, DLp, SJ, SJHO, ML, RB, RBp	Generic	DLb
Veroniki et al. [46]	HO, HO2, DL, DL2, DLp, DLb, PM, HM, HS, ML, REML, AREML, SJ, RB, RBp, FB, BM	Generic	PM
Langan et al. [28]	Estimators in Veroniki et al. [46] except for FB plus IPM, SJHO, RBu, RBa, MBH	RR, OR, SMD, MD and Generic	PM
Petropoulou and Mavridis [37]	Estimators in Langan et al. [28] except for RBu, RBa	OR and MD	DLb and DLp
Langan et al. [29]	DL, HO, PM, PMHO, PMDL, HM, SJ, SJHO, REML	OR and Generic	REML, PM and PMDL for continuous outcomes and non-rare binary events

Two newly proposed estimators, the LCH estimators [31] and the multistep DL estimator DL_M [44], are included in our pool. We mark them in bold in Table 2 and provide a brief description for each in below. The IPM estimator [2] is described as well because it is the only method specifically designed for rare binary events. More details about other estimators can be found in [46] and references therein.

Lin, Chu and Hodges (LCH)

Lin et al. [31] proposed two alternative estimators, ${\widehat{\tau }}_r^2$ and ${\widehat{\tau }}_m^2$, designed to be less affected by outliers than conventional estimators based on the Q statistics in (1). For the purpose of robustness, they are based on Q_r and Q_m, defined as the weighted sums of absolute differences between the study-specific treatment effects and the overall treatment effect, namely

$$Q_r=\sum _{k=1}^K\frac{1}{s_k}\left|y_k-{\widehat{\theta }}_{Fe}\right|,Q_m=\sum _{k=1}^K\frac{1}{s_k}\left|y_k-{\widehat{\theta }}_m\right|.$$

Here, ${\widehat{\theta }}_{Fe}={\sum }_{k=1}^K\frac{y_k}{s_k^2}/{\sum }_{k=1}^K\frac{1}{s_k^2}$, the fixed-effect estimate of θ as defined in Section 2, and ${\widehat{\theta }}_m$ is the weighted median estimator that is the solution to the equation ${\sum }_{k=1}^K{w}_k\left[I\left(\theta \ge y_i\right)-0.5\right]=0$, where I(·) is the indicator function. The estimators ${\widehat{\tau }}_r^2$ and ${\widehat{\tau }}_m^2$, based on Q_r and Q_m, respectively, can be derived similarly as ${\widehat{\tau }}_{DL}^2$ by equating observed Q_r and Q_m to their corresponding expected values.

Multistep DL

We first introduce the generalized method of moments (GMM) estimator of τ² based on the Q statistic in (1). DerSimonian and Kacker [9] showed that if the weights w_k’s are treated as known constants, the expected value of Q is

$$E(Q)={\tau }^2\left(\sum _{k=1}^K{w}_k-\frac{{\sum }_{k=1}^K{w}_k^2}{{\sum }_{k=1}^K{w}_k}\right)+\left(\sum _{k=1}^K{w}_k{\sigma }_k^2-\frac{{\sum }_{k=1}^K{w}_k^2{\sigma }_k^2}{{\sum }_{k=1}^K{w}_k}\right).$$ (2)

By equating Q to its expected value, replacing ${\sigma }_k^2$ by $s_k^2$ in (2), solving for τ² and truncating any negative solution to zero:

$${\widehat{\tau }}_{GMM}^2=\text{max}\left\{\frac{Q-\left({\sum }_{k=1}^K{w}_k{s}_k^2-\frac{{\sum }_{k=1}^K{w}_k^2{s}_k^2}{{\sum }_{k=1}^K{w}_k}\right)}{{\sum }_{k=1}^K{w}_k-\frac{{\sum }_{k=1}^K{w}_k^2}{{\sum }_{k=1}^K{w}_k}},0\right\}.$$ (3)

The DL estimator ${\widehat{\tau }}_{DL}^2$ [10] is a special case of ${\widehat{\tau }}_{GMM}^2$, with $w_k=1/s_k^2$ and Q = Q_Fe.

As discussed in Section 2, the inverse-variance weighing scheme yields $w_k=1/\left(s_k^2+{\widehat{\tau }}^2\right)$ when calculating the (generalized) Q statistic (1) under the Re model. Recall that the original DL estimator ${\widehat{\tau }}_{DL}^2$ can be obtained by specifying $w_k=1/s_k^2$ in (3), which is equivalent to setting ${\widehat{\tau }}^2=0$ in the Re weights. The two-step DL method [9] first obtains ${\widehat{\tau }}_{DL}^2$ and then sets ${\widehat{\tau }}^2={\widehat{\tau }}_{DL}^2$ in the Re weights to obtain ${\widehat{\tau }}_{D{L}_2}^2$ from (3).

van Aert and Jackson [44] proposed the multistep DL estimator as a natural extension of the two-step DL estimator. The M-step DL estimator ${\widehat{\tau }}_{D{L}_M}^2$ can be obtained recursively by computing ${\widehat{\tau }}_{DL}^2$, ${\widehat{\tau }}_{D{L}_2}^2$, …, ${\widehat{\tau }}_{D{L}_M}^2$ using (3). It has been shown that the limit of the multistep DL estimator, ${\widehat{\tau }}_{D{L}_{\infty }}^2$, when it exists, is equivalent to the PM estimator. As further suggested by the authors, divergence problems seldom happen in practice and the convergence is usually achieved quickly.

Improved Paule and Mandel (IPM)

For meta-analysis of rare binary events, Bhaumik et al. [2] adopted a standard binomial-normal random-effects model (labeled BN_BA), which can be specified by

$$x_{ki}~\text{Binomial}\left(n_{ki},p_{ki}\right)\text{ for }i=0,1;$$

$$\text{logit}\left(p_{k0}\right)={\mu }_k,\text{logit}\left(p_{k1}\right)={\mu }_k+{\theta }_k;$$

$${\mu }_k~N\left(\mu ,{\sigma }^2\right),{\theta }_k~N\left(\theta ,{\tau }^2\right),{\mu }_k\perp {\theta }_k\text{ for }k=1,\dots ,K.$$

They proposed a simple average estimator, ${\widehat{\theta }}_{sa}$, for the overall treatment effect θ and then developed the IPM estimator for τ² based on ${\widehat{\theta }}_{sa}$ and the iterative PM method. The treatment effect θ_k (measured by log-odds ratio) in study k is estimated with a correction factor a added to each cell count, namely, y_ka = log[(x_k1 + a)/(n_k1 − x_k1 + a)] − log[(x_k0 + a)/(n_k0 − x_k0 + a)]. The simple average estimator for θ is then given by ${\widehat{\theta }}_{sa}={\sum }_{k=1}^K{y}_{ka}/K$. The authors further proved that a should be $\frac{1}{2}$ in order for ${\widehat{\theta }}_{sa}$ to be the least biased for large samples. They noticed that the PM estimator for τ² depends on $s_k^2$ and proposed to improve PM by borrowing strength from all component studies when estimating each within-study variance,

$$s_k^2(*)=\frac{1}{n_{k1}+1}\left[\text{exp}\left(-\widehat{\mu }-{\widehat{\theta }}_{s\frac{1}{2}}+\frac{{\tau }^2}{2}\right)+2+\text{exp}\left(\widehat{\mu }+{\widehat{\theta }}_{s\frac{1}{2}}+\frac{{\tau }^2}{2}\right)\right]+\frac{1}{n_{k0}+1}[\text{exp}(-\widehat{\mu })+2+\text{exp}(\widehat{\mu })].$$

Denote the corresponding weights by $w_k(*)\equiv 1/\left[s_k^2(*)+{\tau }^2\right]$ and ${\widehat{\tau }}_{IPM}^2$ can be obtained by solving Q − (K − 1) = 0 iteratively with weights w_k(*) in the calculation of Q.

5. Confidence intervals

Table 4 reports 16 existing methods for constructing CIs for τ² in terms of key features including whether the algorithm for computing a CI is iterative, whether truncation for non-negativity is needed, which distribution is used for construction, and whether the CI is exact under the Re model. All the methods are general-purpose and so can be applied to meta-analysis of binary events except for the generalized variable approach [43], which is specifically designed for the mean difference (MD) metric based on normally distributed outcomes. Some of the CIs are obtained via a test-inverting process based on different statistics for testing ${\mathcal{H}}_0:{\tau }^2=0$.

Table 4:

CI methods for τ² in random-effects meta-analysis.

Method	Abbreviation	Iterative? (Y/N)	Truncation to 0? (Y/N)	Distribution Used	Exact Method for Re ?(Y/N)	Reference
CIs based on (modified) Q statistics
Q-Profile	QP	Y	Y	${\chi }_{K-1}^2$	Y	[15, 25]
Modified Q-Profile	MQP	Y	Y	${\chi }_{K-1}^2$	N	[15, 25]
Biggerstaff and Tweedie	BT	Y	Y	Ga(r, λ)	N	[3]
Biggerstaff and Jackson	BJ	Y	Y	A positive linear combination of ${\chi }_1^2$	Y	[4]
Jackson	J	Y	Y	A positive linear combination of ${\chi }_1^2$	Y	[21]
Approximate Jackson	AJ	N	Y	Normal	N	[23]
Unequal-tail Q-profile	UTQ	Y	Y	${\chi }_{K-1}^2$	Y	[22]
Profile likelihood CIs
PL based on ML estimation	PLML	Y	Y	${\chi }_1^2$	N	[14]
PL based on REML estimation	PLREML	Y	Y	${\chi }_1^2$	N	[48]
Wald CIs
Wald based on ML estimation	WML	N	Y	N(0,1)	N	[3, 49]
Wald based on REML estimation	WREML	N	Y	N(0, l)	N	[49]
Others
Sidik and Jonkman	SJ	N	N	${\chi }_{K-1}^2$	N	[39]
Sidik and Jonkman with HO priori	SJHO	N	N	${\chi }_{K-1}^2$	N	[40]
Bayesian credible intervals	—	Y	N	—	N	[46]
Bootstrap	BSP/BSNP	Y	Y	—	N	[11, 27]
Generalized variable approach	GV	Y	Y	—	N	[43]

In Table 5, we list existing review papers on constructing confidence intervals for τ². Clearly, none of these reviews is comprehensive.

5.1. Confidence intervals based on (modified) Q statistics

Q-profile and modified Q-profile CIs

Knapp et al. [25] and Viechtbauer [48] considered the Q-profile CIs based on the generalized Q statistic in (1) with weights $w_k=1/\left({\tau }^2+s_k^2\right)$, denoted by Q(τ²), which depends on τ² and treats $s_k^2’\text{s}$ as known constants. It can be shown that Q(τ²) follows a ${\chi }_{K-1}^2$ distribution under the Re model for any τ². It follows that $P\left({\chi }_{K-1,\alpha /2}^2<Q\left({\tau }^2\right)<{\chi }_{K-1,1-\alpha /2}^2\right)=1-\alpha$. Based on the test-inversion principle, a 100(1 − α)% confidence interval for τ² can be obtained as the interval (${\tilde{\tau }}_l^2$, ${\tilde{\tau }}_u^2$) satisfying $Q\left({\tilde{\tau }}_l^2\right)={\chi }_{K-1,1-\alpha /2}^2$ and $Q\left({\tilde{\tau }}_u^2\right)={\chi }_{K-1,\alpha /2}^2$. Since τ² is non-negative, ${\tilde{\tau }}_l^2$ is truncated to 0 if $Q(0)<{\chi }_{K-1,1-\alpha /2}^2$ (meaning that ${\tilde{\tau }}_l^2$ is negative); and the CI is set to [0, 0] (or {0}, the set containing only zero) if $Q(0)<{\chi }_{K-1,\alpha /2}^2$ (meaning that ${\tilde{\tau }}_u^2$ is also negative). This type of CIs is referred to as the Q-profile (QP) CIs as we are profiling Q(τ²) with different τ² values when solving the above equations for ${\tilde{\tau }}_l^2$ and ${\tilde{\tau }}_u^2$ iteratively.

Knapp et al. [25] considered the fact that $s_k^2’\text{s}$ are only estimates and so have error variability, and constructed CIs using the test statistic ${\tilde{Q}}_r$ that replaces the weights in Q(τ²) with regularized variants $w_{rk}=r_k/\left({\tau }^2+s_k^2\right)$ to achieve a closer approximation to ${\chi }_{K-1}^2$, where the regularization factor r_k is derived through a moment matching approach based on approximating the distribution of ${\tau }^2+s_k^2$ by a scaled χ² distribution [15]. The lower bound ${\tilde{\tau }}_l^2$ is obtained by profiling ${\tilde{Q}}_r\left({\tau }^2\right)$ while the upper bound ${\tilde{\tau }}_u^2$ is still obtained by profiling Q(τ²), satisfying ${\tilde{Q}}_r\left({\tilde{\tau }}_l^2\right)={\chi }_{K-1,1-\alpha /2}^2$ and $Q\left({\tilde{\tau }}_u^2\right)={\chi }_{K-1,\alpha /2}^2$. We refer to this type of CIs as the modified Q-profile (MQP) CIs.

Like the Q-profile CIs, the MQP CIs need left truncation to zero if the lower bound ${\tilde{\tau }}_l^2$ turns out to be negative, and they are set to {0} if the upper bound ${\tilde{\tau }}_u^2$ is also negative. The same rule applies to all other types of CIs based on (modified) Q statistics in Section 5.1, as discussed below.

BT and BJ CIs based on Cochran’s Q statistic

Biggerstaff and Tweedie [3] proposed to approximate the distribution of the Cochran’s Q statistic Q_Fe by a gamma distribution with a shape parameter r(τ²) ≡ E²(Q_Fe)/Var(Q_Fe) and a scale parameter λ(τ²) ≡ Var(Q_Fe)/E(Q_Fe). The mean and variance of Q_Fe under the Re model are given by E(Q_Fe) = (K − 1) + (S₁ − S₂/S₁)τ² and $\text{Var}\left(Q_{Fe}\right)=2(K-1)+4\left(S_1-S_2/S_1\right){\tau }^2+2\left(S_2+S_2^2/S_1^2-2S_3/S_1\right){\tau }^4$, where $S_r\equiv {\sum }_{k=1}^K{\left[1/s_k^2\right]}^r$. CIs for τ² can be obtained similarly based on this gamma approximation instead of ${\chi }_{K-1}^2$ using the above profiling approach, which we refer to as the BT intervals.

Biggerstaff and Jackson [4] derived the exact CDF of Q_Fe under the Re model, denoted by F_Q(q; τ²), as a positive linear combination of ${\chi }_1^2$ random variables, whose cumulative distribution function can be obtained using Farebrother’s algorithm [12] via the CompQuad-Form package in R. They then obtained (${\tilde{\tau }}_l^2$, ${\tilde{\tau }}_u^2$) by solving the two equations numerically, $F_Q\left(c{\widehat{\tau }}_{uDL}^2+K-1;{\tilde{\tau }}_l^2\right)=1-\alpha /2$ and $F_Q\left(c{\widehat{\tau }}_{uDL}^2+K-1;{\tilde{\tau }}_u^2\right)=\alpha /2$, where c = S₁ − S₂/S₁ and ${\widehat{\tau }}_{uDL}^2=\left[Q_{Fe}-(K-1)\right]/c$ is the untruncated version of the DL estimator of τ². This type of CIs is referred to as the BJ intervals.

Jackson and approximate Jackson CIs

Following the numerical approach in [4], Jackson [21] proposed CIs by test inversion based on the generalized Q in (1), which is also distributed as a positive linear combination of ${\chi }_1^2$ random variables under the Re model. Jackson et al. [23] further proposed to apply the arcsinh transformation to the untruncated version of ${\widehat{\tau }}_{GMM}^2$ for variance stabilization and then constructed CIs for τ² based on a normal approximation. These types of CIs are referred to as the Jackson (J) and approximate Jackson (AJ) CIs, respectively. Based on simulation, Jackson further commented that weighting component studies by the reciprocal of their within-study standard errors (i.e. s_k), rather than by their variances (i.e. $1/s_k^2$) as the convention dictates, appears to provide a sensible and viable option when there is little a priori knowledge about the extent of heterogeneity.

Unequal-tail Q profile CIs

Jackson and Bowden [22] advocated to use unequal tail probabilities to obtain shorter intervals whenever such methods are justifiable. For example, when constructing a 100(1 − α)% unequal-tail Q-profile (UTQ) confidence interval, the lower and upper bounds, ${\tilde{\tau }}_l^2$ and ${\tilde{\tau }}_u^2$, are obtained by solving $Q\left({\tilde{\tau }}_l^2\right)={\chi }_{K-1,1-{\alpha }_1}^2$ and $Q\left({\tilde{\tau }}_u^2\right)={\chi }_{K-1,{\alpha }_2}^2$, respectively, where α₂ > α₁ and α₁ + α₂ = α. They further suggested to use a pre-specified α-split with α₁ = 0.01 and α₂ = 0.04 for a 95% CI, which was shown to be able to retain the nominal coverage and reduce the width under the Re model. Obviously, the idea of unequal tails can be applied to all kinds of confidence intervals. In our numerical evaluation, we examine the performance of the Q-profile CIs with α₁ = 0.01 and α₂ = 0.04 as a representative.

5.2. Profile likelihood confidence intervals

Under the Re model, Hardy and Thompson [14] proposed the profile likelihood CIs based on maximum likelihood (ML) estimation, referred to as PL_ML. The profile log-likelihood for τ² takes into account the fact that θ is also unknown and must be estimated, given by $l\left({\widehat{\theta }}_{ML}\left({\tau }^2\right),{\tau }^2\right)$, where the log-likelihood function of (θ, τ²) is given by

$$l\left(\theta ,{\tau }^2\right)=-\frac{K}{2}\text{ln }2\pi -\frac{1}{2}\sum _{k=1}^K\text{ln}\left({\tau }^2+{\sigma }_k^2\right)-\frac{1}{2}\sum _{k=1}^K\frac{{\left(y_k-\theta \right)}^2}{{\tau }^2+{\sigma }_k^2},$$

and given the value of τ², the ML estimator of θ can be obtained by

$${\widehat{\theta }}_{ML}\left({\tau }^2\right)=\sum _{k=1}^K\frac{1}{{\tau }^2+{\sigma }_k^2}{y}_k/\sum _{k=1}^K\frac{1}{{\tau }^2+{\sigma }_k^2}.$$

Then a 100(1 − α)% CI for τ² is given by the set of τ² values satisfying $l\left({\widehat{\theta }}_{ML}\left({\tau }^2\right),{\tau }^2\right)>l\left({\widehat{\theta }}_{ML}\left({\widehat{\tau }}_{ML}^2\right),{\widehat{\tau }}_{ML}^2\right)-{\chi }_{1,1-\alpha }^2/2$.

Viechtbauer [48] proposed to construct profile likelihood CIs based on restricted maximum likelihood (REML) estimation, referred to as PL_REML. The 100(1 − α)% CI for τ² is given by the set of τ² values satisfying $l_R\left({\tau }^2\right)>l_R\left({\widehat{\tau }}_{\mathit{\text{REML}}}^2\right)-{\chi }_{1,1-\alpha }^2/2$, where the restricted log-likelihood function of τ² is given by

$$l_R\left({\tau }^2\right)=-\frac{K}{2}\text{ln }2\pi -\frac{1}{2}\sum _{k=1}^K\text{ln}\left({\tau }^2+{\sigma }_k^2\right)-\frac{1}{2}\sum _{k=1}^K\text{ln}\frac{1}{{\tau }^2+{\sigma }_k^2}-\frac{1}{2}\sum _{k=1}^K\frac{{\left(y_k-{\widehat{\theta }}_{ML}\left({\tau }^2\right)\right)}^2}{{\tau }^2+{\sigma }_k^2},$$

and ${\widehat{\tau }}_{\mathit{\text{REML}}}^2$ is the REML estimate of τ² (by maximizing l_R). Viechtbauer [48] found that the REML-based CIs were slightly more accurate than the ML-based CIs in terms of coverage probability, especially for small K.

Because ML and REML estimates of τ² require non-negativity, the lower bounds of profile likelihood (PL) intervals are always non-negative and the upper bounds are strictly positive after applying the same truncation for Q-profile CIs.

5.3. Wald confidence intervals

The Wald test statistics for testing ${\mathcal{H}}_0:{\tau }^2=0$ under the Re model have the form $W={\widehat{\tau }}^2/SE\left({\widehat{\tau }}^2\right)$, where ${\widehat{\tau }}^2$ can be ${\widehat{\tau }}_{ML}^2$ or ${\widehat{\tau }}_{\mathit{\text{REML}}}^2$, and the standard error is estimated by

$$\widehat{SE}\left({\widehat{\tau }}_{ML}^2\right)=\sqrt{2{\left[\sum _{k=1}^K{w}_{ML.k}^2\right]}^{-1}},$$

$$\widehat{SE}\left({\widehat{\tau }}_{\mathit{\text{REML}}}^2\right)=\sqrt{2{\left[\sum _{k=1}^K{w}_{\mathit{\text{REML}}.k}^2-2\frac{{\sum }_{k=1}^K{w}_{\mathit{\text{REML}}.k}^3}{{\sum }_{k=1}^K{w}_{\mathit{\text{REML}}.k}}+{\left(\frac{{\sum }_{k=1}^K{w}_{\mathit{\text{REML}}.k}^2}{{\sum }_{k=1}^K{w}_{\mathit{\text{REML}}.k}}\right)}^2\right]}^{-1}}$$

with $w_{ML.k}=1/\left({\widehat{\tau }}_{ML}^2+s_k^2\right)$ and $w_{\mathit{\text{REML}}.k}=1/\left({\widehat{\tau }}_{\mathit{\text{REML}}}^2+s_k^2\right)$. We label the Wald statistics based on ML and REML estimation by W_ML and W_REML, respectively. The corresponding 100(1 − α)% Wald (W) CI for τ² can be easily obtained by ${\widehat{\tau }}_{ML}^2\pm z_{1-\alpha /2}\widehat{SE}\left({\widehat{\tau }}_{ML}^2\right)$ or ${\widehat{\tau }}_{\mathit{\text{REML}}}^2\pm z_{1-\alpha /2}\widehat{SE}\left({\widehat{\tau }}_{\mathit{\text{REML}}}^2\right)$ [3, 48], where z_α is the 100α-th percentile of the standard normal distribution. Negative lower bounds of the Wald CIs should be truncated to 0 since both ML and REML estimates of τ² are constrained to be non-negative.

5.4. Other confidence intervals

Sidik and Jonkman (SJ) CIs

Sidik and Jonkman [39] proposed confidence intervals based on the SJ estimator of τ², which is derived from the weighted residual sum of squares in the framework of a linear regression model. Let the crude estimate ${\widehat{\tau }}_0={\sum }_{k=1}^K{\left(y_k-\overline{y}\right)}^2/K$ be an a priori value for τ². Then the SJ estimator is given by ${\widehat{\tau }}_{SJ}^2=\frac{{\widehat{\tau }}_0^2}{K-1}{\sum }_{k=1}^K{\widehat{w}}_k{\left(y_k-{\widehat{\theta }}_0\right)}^2$, where ${\widehat{w}}_k=1/\left(s_k^2+{\widehat{\tau }}_0^2\right)$, and ${\widehat{\theta }}_0={\sum }_{k=1}^K{\widehat{w}}_k{y}_k/{\sum }_{k=1}^K{\widehat{w}}_k$. It follows that $(K-1){\widehat{\tau }}_{SJ}^2/{\tau }^2$ has an asymptotic distribution of ${\chi }_{K-1}^2$. Thus an approximate 100(1 − α)% confidence interval can be calculated by

$$\frac{(K-1){\widehat{\tau }}_{SJ}^2}{{\chi }_{K-1,1-\alpha /2}^2}\le {\tau }^2\le \frac{(K-1){\widehat{\tau }}_{SJ}^2}{{\chi }_{K-1,\alpha /2}^2}.$$

Since ${\widehat{\tau }}_{SJ}^2$ is always positive, the SJ confidence intervals have positive lower and upper bounds. Sidik and Jonkman [40] later proposed an improved estimator ${\widehat{\tau }}_{S{J}_{HO}}^2$ by using ${\widehat{\tau }}_{HO}^2$ as the a priori value. Then improved confidence intervals can be constructed correspondingly.

Bayesian credible intervals

Bayesian credible (BC) intervals can be obtained when a Bayesian approach is employed and posterior samples are drawn from the (joint) posterior distribution of all parameters involved using an MCMC algorithm. The lower and upper points of a 100(1 − α)% CI can be the 100(α/2)th and 100(1 − α/2)th percentiles of the posterior sample of τ²’s, or determined by the region that gives the highest posterior density. Such intervals may be heavily affected by the prior selection when the number of studies K is small.

Bootstrap CIs

Bootstrap techniques can be used to obtain confidence intervals for nearly all τ² estimators. For nonparametric bootstrap (denoted by BS_NP), we sample K studies with replacement from the observed set of studies B times to get B bootstrap samples. For parametric bootstrap (denoted by BS_P), we first obtain the parameter estimates and then generate B samples from the assumed distributions with these estimates. For each (parametric or nonparametric) sample, we calculate the corresponding estimate ${\widehat{\tau }}^2$. Then the 100(α/2)th and 100(1 − α/2)th percentiles of the B estimates of τ² are respectively the lower and upper bounds of a 100(1 − α)% bootstrap confidence interval. In our numerical experiment, we only perform the nonparametric bootstrap procedure for the DL estimator for illustration.

The generalized variable (GV) approach

For meta-analysis of normally distributed outcomes, Tian [43] proposed inference procedures based on the generalized pivotal quantity for τ². A pivotal quantity is a function of observations and parameters such that the distribution of the function does not depend on the parameters including nuisance parameters. Let ${\sigma }_{k0}^2\left({\sigma }_{k1}^2\right)$ be the population variance of the control (treatment) group in study k; let $s_{k0}^2\left(s_{k1}^2\right)$ be the corresponding sample variance. For normally distributed outcomes, it is well known that $V_{ki}\equiv \left(n_{ki}-1\right)s_{ki}^2/{\sigma }_{ki}^2~{\chi }_{n_{ki}-1}^2$ for k = 1, …, K and i = 0, 1. Denote Q in (1) with weight $w_k=1/\left({\sigma }_{k0}^2/n_{k0}+{\sigma }_{k1}^2/n_{k1}+{\tau }^2\right)$ by Q(τ²), which follows ${\chi }_{K-1}^2$ and is a monotonic decreasing function of τ². Thus, given a real number η ≥ 0, there exists a unique ${\tau }_{\eta }^2\ge 0$ such that $Q\left({\tau }_{\eta }^2\right)=\eta$. Based on this, Tian [43] defined the generalized pivotal quantity $R_{{\tau }^2}$ for τ² as $R_{{\tau }^2}={\tau }_{\eta }^2$ if η ≤ Q(0) and $R_{{\tau }^2}=0$ otherwise. Given the observed treatment effects y_k’s and sample variances $s_{ki}^2’\text{s}$, the distribution of $R_{{\tau }^2}$ does not depend on any nuisance parameters. A series of $R_{{\tau }^2}$ values can be obtained by first simulating $V_{ki}~{\chi }_{n_{ki}-1}^2$ and $\eta ~{\chi }_{K-1}^2$ and setting ${\sigma }_{ki}^2=\left(n_{ki}-1\right)s_{ki}^2/V_{ki}$ in Q(τ²) for k = 1, …, K and i = 0, 1, and then solving for ${\tau }_{\eta }^2$. A 100(1 − α)% confidence interval is given by ($R_{{\tau }^2,\alpha /2}$, $R_{{\tau }^2,1-\alpha /2}$), where the lower and upper bounds are the 100(α/2)th and 100(1 − α/2)th percentiles of the generated $R_{{\tau }^2}’\text{s}$.

6. Simulation focusing on rare binary events

For meta-analysis of rare binary events, Li and Wang [30] conducted a comprehensive simulation study to compare the performance of various estimators of the overall treatment effect θ measured by log-odds ratio, where a flexible binomial-normal model was used to accommodate treatment groups with unequal variability. This model, labeled BN_LW, specifies the event probabilities by

$$\text{logit}\left(p_{k0}\right)={\mu }_k-\omega {\theta }_k,\text{ logit}\left(p_{k1}\right)={\mu }_k+(1-\omega ){\theta }_k,$$

where μ_k ~ N(μ, σ²), θ_k ~ N(θ, τ²), μ_k ⊥ θ_k, and ω is a constant in [0, 1]. The random-effects model BN_BA in [2] is a special case of BN_LW with ω = 0. Further, when $\omega =\frac{1}{2}$, it is reduced to the model in [41], which assumes the equality of the variances of logit(p_k0) and logit(p_k1).

In this section, we adopt the same model and simulation setup from [30], to examine the performance of various methods. Results are summarized in Sections 6.1 and 6.2 for estimating the between-study variance τ² of log-odds ratios θ_k’s. Here, bias and MSE are reported for point estimation, and the actual coverage probability and width of confidence intervals are reported for interval estimation. To be specific, we set the number of studies K to 10, 20 and 50 to reflect different sizes of meta-analysis. We generate the number of events x_ki from Binomial(n_ki, p_ki) for k = 1, …, K and i = 0, 1. The number of subjects in the control group, n_k0’s are generated from Uniform[2000, 3000] to examine large-sample performance and from Uniform[20, 1000] to examine small-sample performance, and then rounded to the nearest integers. To allow varying allocation ratios across studies, the within-study sample sizes are set to follow the relationship n_k1 = R_kn_k0, where ${\text{log}}_2{R}_k~N\left({\text{log}}_{2}R,{\sigma }_R^2\right)$, R ∈ {1, 2, 4} and ${\sigma }_R^2=0.5$. For small sample sizes, as noted in [30], the range [20, 1000] is chosen so that the empirical means of $\text{min}{\left(n_{k0}{p}_{k0},n_{k1}{p}_{k1}\right)}_{k=1}^K$ in all the settings are below one while it still allows for cases where most component studies have small sample sizes but a few can have sample sizes close to 1000. To generate p_ki’s, we fix σ² at 0.5, and set τ² ∈ {0, 0.25, 0.5, 0.75, 1} for evaluating different estimators and τ² ∈ {0, 0.1, 0.2, ⋯, 0.9, 1} for evaluating different types of CIs. We further set θ ∈ {−1, 0, 1} to reflect different directions of the overall treatment effect, set μ ∈ {−2.5, −5} to represent low and very low incidence rates of the binary event (i.e., 0.076 and 0.0067 in the probability scale), and set ω ∈ {0, 0.5, 1} to represent smaller/equal/larger variability in the control group, compared to the treatment group. For each setting, 1000 datasets are simulated to compute empirical values of the performance measures by taking the average.

6.1. Comparison of different heterogeneity estimators

We compare all the methods listed in Table 2 except for FB and MBH. Since the full Bayesian method can be greatly affected by the prior choice and other factors (such as convergence), we exclude FB from our simulation. The MBH method is designed specifically for standard mean difference thus not suitable for binary events. In addition, the empirical Bayes method EB is equivalent to PM and the multistep DL method has the property that DL_∞ converges to PM. Therefore, we include PM in the comparison and leave EB and DL_M out. We use heat maps to visualize the bias and MSE results where the rows of each map represent different methods and columns represents different τ² values in [0, 1].

Large-sample results

Figure 1 presents the bias and MSE results of different estimators for μ = −2.5 and μ = −5 based on large-sample settings with R = 1, K = 50, θ = 0, and w = 0. As shown in Figure 1(a), as the event of interest becomes rarer, all methods seem to produce more bias when estimating τ². Almost all methods underestimate the between-study heterogeneity when τ² > 0. The RBp estimator, however, consistently overestimates τ² when the event is very rare (μ = −5). As τ² increases, most estimators produce more bias except for BM and RB_p; the bias from BM first increases then decreases, and the bias from RB_p decreases for very rare events (μ = −5). When the events are not that rare (μ = −2.5), most estimators have similarly low bias except for the one-step DL estimators (DL, DLp, DL_b), HM, HS, and BM. However, IPM stands out with the lowest bias when the incidence rate becomes very low, especially when τ² ≥ 0.5. The HS, HM, BM and one-step DL family methods remain the worst and should be avoided in terms of bias. All three likelihood-based methods, ML, REML and AREML, produce similar results with a moderate level of bias. In terms of MSE, most methods have similar performance except for HM and BM, which are the most inefficient according to Figure 1(b). Those with relatively large magnitude of bias tend to have relatively large MSE.

Figure 1:

Large-sample performance of different τ² estimators based on settings with R = 1, K = 50, θ = 0, and w = 0.

We next discuss the potential impacts of R, K, θ, and w on the estimation performance for the large-sample case. Figures S1 and S2 in the Supplementary Material (SM) show the bias and MSE results for different R and K values, respectively, based on settings with μ = −2.5, θ = 0 and w = 0. We can see that when τ² < 0.5, regardless of R and K, all the methods perform somewhat similarly and have both bias and MSE close to zero except for BM which has much larger bias. As K increases, MSE decreases significantly for every estimator when τ² ≥ 0.5 but bias for a few estimators seems not to get closer to zero (e.g., DL for τ² = 1, BM for τ² = 0.5, and 0.75). However, the heat maps show very similar color patterns both vertically and horizontally, indicating that the impact of R and K on the relative performance of these methods is merely marginal. Figures S3 and S4 in the SM show the bias and MSE results for different θ and w values, respectively, based on settings with R = 1, K = 50 and μ = −5. When θ = −1, bias decreases as w increases while this trend reverses when θ = 1. This effect of w is minimal when there is no treatment effect (θ = 0). Similar trends are observed but less obvious for MSE. Also, we find that IPM maintains the best performance in terms of both bias and MSE while DL, DLp, DL_b, HS, HM, and BM are among the worst in nearly all the settings considered.

Small-sample results

Figure 2 presents the bias and MSE results of different estimators for μ = −2.5 and μ = −5 based on small-sample settings with R = 1, K = 50, θ = 0, and w = 0. From Figure 2(a), we can see that when τ² > 0, the underestimation observed in the large-sample results for all the estimators but RB_p is much more severe for small samples, where the magnitude of bias increases substantially for very rare events (μ = −5). Note that RB_p consistently overestimates τ² for both μ = −2.5 and μ = −5, and unlike most other estimators, the bias decreases as τ² increases. When events are not that rare (μ = −2.5), IPM is still the least biased. However, for very rare events (μ = −5), SJ becomes the least biased estimator for τ² ≥ 0.5. The problem of SJ is that it significantly overestimates τ² when there is no or little heterogeneity, due to its positive nature. From Figure 2(b) we can see that MSE does not change much when μ = −2.5 but dramatically increases when μ = −5 compared to results from large samples. For very rare events (μ = −5), SJ is the most efficient method except for τ² = 0 and IPM seems to be the second best in terms of MSE. Note that when τ² = 1, RB_p has smaller MSE than IPM for very rare events, but it does not perform as well as IPM for smaller τ² values.

Figure 2:

Small-sample performance of different τ² estimators based on settings with R = 1, K = 50, θ = 0, and w = 0.

The impacts of R, K, θ, and w on the estimation bias and MSE for the small-sample case are shown in Figure S5–S8 of the SM. Since several methods (e.g., the likelihood-based methods) failed in some small-sample settings for very rare events (μ = −5), we show results for μ = −2.5 in these figures. Although the effect of K on MSE becomes more significant for small samples (i.e., MSE decreases more as K increases), it is still the case that both R and K have little impact on the relative performance of different methods. Also, similar trends for both bias and MSE occur when w and θ change as in the large-sample case. For these μ = −2.5 settings, IPM seems to be the best estimator due to its consistent top-level performance across various settings. This also agrees with the results in the left panels of Figure 2. On the other hand, DL, DL_p, DL_b, HM, HS, and BM should be used with caution due to their generally large bias.

6.2. Comparison of different types of CIs

Among those summarized in Table 4, we compared 14 different types of 95% CIs for the heterogeneity parameter τ² in Figures 3 and 4, excluding Bayesian credible intervals and the GV method as before. As mentioned in Section 5, BS_NP represents the nonparametric bootstrap procedure combined with the DL estimator and UTQ represents the unequal-tail Q-profile CI with α₁ = 0.01 and α₂ = 0.04. Again, from our (unreported) simulation results, we find that the influences of R, θ, and w on the empirical coverage probability are marginal.

Figure 3:

Actual coverage probabilities of different types of 95% CIs for different K values based on large-sample settings with R = 1, μ = −5, θ = 0, and w = 0.

Figure 4:

Actual coverage probabilities of different types of 95% CIs for both large- and small-sample cases and different μ values based on settings with R = 1, K = 20, θ = 0, and w = 0.

Figure 3 shows actual coverage probabilities of different types of CIs for different K values based on large-sample settings with R = 1, μ = −5, θ = 0, and w = 0. When there is no between-study heterogeneity (τ² = 0), all the methods provide 100% coverage except for SJ and SJ_HO, which produce strictly positive intervals and so have zero coverage. When τ² is small, as K increases, the methods based on (modified) Q statistics gain some improvement in coverage except for AJ, which achieves relatively high coverage for all K and τ² values. As τ² gets larger, most methods do not improve their coverage by increasing K.

Figure 4 presents actual coverage probabilities of different types of CIs for both large- and small-sample cases and different μ values based on settings with R = 1, K = 20, θ = 0, and w = 0. When μ = −2.5, most methods have actual coverage close to the nominal level 0.95. Among all, the nonparametric bootstrap CI has the lowest coverage, followed by the two Wald CIs when τ² > 0. The influence of sample sizes is not obvious except for J, SJ and SJ_HO that improve their coverage for large sample sizes when τ² is small. For very rare events (μ = −5), the impact of sample sizes is much more severe and some of the CIs (e.g., SJ_HO, J, UTQ) do not even achieve 50% coverage in most small-sample settings. In the large-sample settings, PL_ML, PL_REML, and AJ maintain the nominal 95% coverage quite well at all positive levels of τ². As the sample sizes become small, all methods fail to do so for very rare events when τ² ≥ 0.3. Still, PL_ML and PL_REML, and AJ are among those with the highest coverage. We also find that when τ² ≥ 0.4, SJ joins the top-performing group with the following order SJ ≈ PL_REML > PL_ML > AJ. This matches with the estimation results reported in Section 6.1 that for very rare events coupled with small samples, the SJ estimator is the least biased and has the smallest MSE when τ² ≥ 0.5. In such situations, the Q statistic-based CIs have generally low coverage and thus should be avoided; meanwhile the Wald and nonparametric bootstrap CIs have moderate coverage instead of being the worst in the other three cases.

Figure 5 shows width curves of different types of CIs under the same settings of Figure 4, where for all CIs, the width shows an increasing pattern as τ² increases. The influence of sample sizes on the CI width is only obvious when μ = −5, where all the CIs become narrower when sample sizes decrease. Though anti-intuitive, a closer examination reveals that when events are very rare and sample sizes are small, many simulation iterations produce confidence intervals of a point {0}, which makes the average width become smaller. In the first three situations (either μ = −2.5 or large samples), BT and BJ produce the widest intervals, and PL and AJ intervals, which offer higher coverage than most other methods, have moderate widths among all. Unsurprisingly, the nonparametric bootstrap procedure produces the narrowest CIs. In the last situation (very rare events coupled with small samples), PL and AJ intervals are among the widest. Here, CIs with shorter widths are not necessarily desirable as they may reflect more {0} intervals due to sparsity. SJ produces intervals with moderate widths though it also provides higher coverage when τ² is large. Overall, we recommend PL and AJ intervals in meta-analysis of rare binary events for their high coverage. For very rare events with small samples, we recommend SJ intervals if we know there exists at least moderate-level heterogeneity. Besides, AJ and SJ intervals are much easier to obtain than PL intervals.

Figure 5:

Width curves of different types of 95% CIs for both large- and small-sample cases and different μ values based on settings with R = 1, K = 20, θ = 0, and w = 0.

7. Example: Type 2 diabetes mellitus after gestational diabetes

Women with gestational diabetes are believed to have a higher chance to develop type 2 diabetes. Bellamy et al. [1] performed a comprehensive systematic review and meta-analysis to assess the strength of this association. They selected 20 cohort studies that included 675,455 women with/without gestational diabetes and 10,859 type 2 diabetic events from 205 reports between Jan 1, 1960, and Jan 31, 2009, from Embase and Medline (see Table S.1 of the SM). We reanalyzed the data focusing on inference about the heterogeneity parameter τ². We note that the overall event rate is ~1.61% and many studies have very small sample sizes with zero event counts. So this data example fits in the scenario of very rare events coupled with small sample sizes. Recall that in this scenario, SJ gives the least bias and most efficient estimator when there exists a moderate or large level of heterogeneity and IPM is the second best which tends to underestimate τ².

Point estimates for the heterogeneity parameter τ² and the corresponding inverse-variance weighted estimates for the overall treatment effect θ (measured by log-odds ratio) are summarized in Table 6. Here, most methods give an estimate between 0.4 and 0.7 for τ², where the estimate from IPM is 0.563 and that from SJ is 0.679. This seems to suggest a moderate to high level of heterogeneity, especially after accounting for the underestimation from IPM. The RB_p method, which has been shown to severely overestimate τ² for very rare events, not surprisingly gives the largest estimate of 1.162. On the other hand, the HS estimate is much smaller than the others. The resulting estimated odds ratios do not vary as much except for the one from RB_p. Table 7 shows the confidence intervals from all the compared methods. BT gives a very large upper bound, which seems to be odd. All CIs except for those from BT, BJ, and Wald methods exclude zero, among which SJ yields the shortest interval with the largest lower bound and the upper bound in line with that from PL and AJ methods. Recall that SJ tends to produce the best interval with higher coverage and relatively shorter width when there exists at least moderate-level heterogeneity, as reported in Section 6.2. In this example, we lean toward reporting the SJ interval, among the top performing methods PL, AJ and SJ. Based on the estimation and inference results above, we believe that these studies are heterogeneous‥

Table 6:

Data example of gestational diabetes meta-analysis: estimates for τ² and θ from different methods

Estimator	HO	HO2	DL	DL2	DLp	DLb	PM	IPM	HM	HS
${\widehat{\tau }}^2$	0.220	0.418	0.466	0.411	0.466	0.265	0.413	0.563	0.419	0.046
$\widehat{\theta }$	2.093	2.136	2.146	2.135	2.146	2.104	2.135	2.162	2.137	2.092
OR	8.112	8.469	8.547	8.457	8.547	8.197	8.461	8.691	8.470	8.099
Estimator	LCHmean	LCHmedian	ML	REML	AREML	SJ	SJHO	RB0	RBp	BM
${\widehat{\tau }}^2$	0.519	0.298	0.396	0.449	0.433	0.679	0.290	0.198	1.162	0.195
$\widehat{\theta }$	2.155	2.111	2.132	2.142	2.139	2.180	2.110	2.088	2.235	2.088
OR	8.626	8.260	8.432	8.520	8.493	8.846	8.245	8.072	9.345	8.067

Table 7:

Data example of gestational diabetes meta-analysis: confidence intervals for τ² from different methods.

Method CI	QP (0.109, 1.603)	MQP (0.106, 1.603)	UTQ (0.083, 1.403)	BT [0, 8.610)	BJ [0, 2.660)	J (0.048, 1.540)	AJ (0.004, 1.396)
Method CI	SJ (0.393,1.449)	SJHO (0.168, 0.620)	BSNP (0.012, 0.670)	PLML (0.113, 1.285)	PLREML (0.129, 1.458)	WML [0, 0.841)	WREML [0, 0.966)

8. Discussion and recommendations

Based on our comprehensive simulation studies for large-sample meta-analysis of rare binary events, we recommend the IPM method for estimating the heterogeneity parameter τ² if reducing estimation bias is of high priority, especially when the events are extremely rare. Most of the methods do not differ much in terms of MSE. We suggest to avoid using HM, HS and BM since they have relatively large bias and MSE compared with other estimators. The most widely used DL estimator and its one-step variants DL_p and DL_b do not perform satisfactorily and hence should be avoided. For small-sample meta-analysis of rare events, IPM is still recommended and SJ also performs much better than the other estimators in terms of both bias and MSE when τ² ≥ 0.5 and the events are extremely rare. In terms of interval estimation, we recommend the profile likelihood methods (PL_ML and PL_REML) and the approximate Jackson method AJ in general situations. Among the three, PL_REML usually produces higher coverage but with wider intervals. The SJ method is a good candidate when events are extremely rare, sample sizes are small, and τ² ≥ 0.4. We did not examine the performance of Bayesian methods because of the computation burden, convergence detection issue, and potential sensitivity to prior choices. However, Bayesian hierarchical modeling can be a good alternative especially when meaningful prior information is available.

We notice that most estimators for τ² are negatively biased in our simulation, an interesting phenomenon observed in other simulation studies with binary outcomes [26, 39, 40, 2] as well. In simulation studies with continuous outcomes [27], most of the estimators show positive bias when τ² is small (< 0.1) and the magnitude of bias of RB_p is much larger than the other estimators; for larger τ² values, the HS and ML estimators are negatively biased and the magnitude increases as τ² increases [47]. Viechtbauer [47] provides some analytical results for the bias of estimators HO, DL, HS, ML, and REML. Most of these results were derived based on the homogeneous within-study variance assumption (${\sigma }_k^2={\sigma }^2$). Under this assumption, the bias due to truncation is always positive for DL, HO and REML with all levels of heterogeneity and is negative for HS and ML when τ² ≥ 0.5. However, we believe that in the rare events context, it is the sparsity (caused by zero counts) and lack of resolution in estimating the within-study variances that cause the large magnitude of underestimation for many methods. This underestimation is much reduced by the IPM estimator where the within-study variance estimates are improved by pooling information from all the studies.

Finally, we should mention that, when synthesizing information from multiple studies to obtain more reliable conclusions, one should not simply rely on one point estimate or one p-value (especially those from the default methods in software packages) without considering the rich selection of statistical tools offered in the literature. Each of the above reviewed models or methods has its own limitations. In practice, all kinds of evidence should be combined and evaluated together with the specific characteristics of component studies included in the meta-analysis.