Comparisons of various estimates of the I² statistic for quantifying between-study heterogeneity in meta-analysis

Abstract

Assessing heterogeneity between studies is a critical step in determining whether studies can be combined and whether the synthesized results are reliable. The $I^2$ statistic has been a popular measure for quantifying heterogeneity, but its usage has been challenged from various perspectives in recent years. In particular, it should not be considered an absolute measure of heterogeneity, and it could be subject to large uncertainties. As such, when using $I^2$ to interpret the extent of heterogeneity, it is essential to account for its interval estimate. Various point and interval estimators exist for $I^2$. This article summarizes these estimators. In addition, we performed a simulation study under different scenarios to investigate preferable point and interval estimates of $I^2$. We found that the Sidik–Jonkman method gave precise point estimates for $I^2$ when the between-study variance was large, while in other cases, the DerSimonian–Laird method was suggested to estimate $I^2$. When the effect measure was the mean difference or the standardized mean difference, the $Q$-profile method, the Biggerstaff–Jackson method, or the Jackson method was suggested to calculate the interval estimate for $I^2$ due to reasonable interval length and more reliable coverage probabilities than various alternatives. For the same reason, the Kulinskaya–Dollinger method was recommended to calculate the interval estimate for $I^2$ when the effect measure was the log odds ratio.

1. Introduction

Meta-analysis is a statistical tool to synthesize evidence from different studies and is widely used in medical research. Assessing heterogeneity between the collected studies is a critical step to examine whether the studies may be properly combined and the synthesized results are reliable.^1,2 In this article, heterogeneity refers to the variation in underlying treatment effects across studies.³

A classical method to detect heterogeneity is the chi-squared $Q$ test; the distribution of the $Q$ statistic is approximately ${\chi }_{k-1}^2$ (k is the number of studies) under the null hypothesis that all studies in a meta-analysis are homogeneous. However, the $Q$ test alone does not suffice to describe the amount of heterogeneity because only $P$-values are produced to indicate a binary decision of either the presence or absence of heterogeneity. The $I^2$ statistic has been a popular alternative to quantify heterogeneity because of its attractive interpretation as the proportion of total variation caused by heterogeneity rather than within-study sampling error.^4,5 Specifically, the $I^2$ statistic can be conceptualized as ${\tau }^2/\left({\tau }^2+{\sigma }^2\right)\times 100\mathrm{\%}$, where ${\tau }^2$ is the between-study variance caused by heterogeneity and ${\sigma }^2$ is a summary of within-study variances. It ranges from $0\mathrm{\%}$ to $100\mathrm{\%}$. The Cochrane Handbook provides a rough yet widely used rule to interpret this measure: $I^2\le 40\mathrm{\%}$ may indicate unimportant heterogeneity, $30\mathrm{\%}\le I^2\le 60\mathrm{\%}$ may represent moderate heterogeneity, $50\mathrm{\%}\le I^2\le 90\mathrm{\%}$ may represent substantial heterogeneity, and $75\mathrm{\%}\le I^2\le 100\mathrm{\%}$ implies considerable heterogeneity.³ These ranges overlap with each other because they are vague, and the true heterogeneity should be evaluated with caution, using both statistical and clinical knowledge.⁶

Over the past few years, the usage of $I^2$ has been challenged from many perspectives, and it should not be used as an absolute measure.^7–9 Several studies have demonstrated shortcomings of $I^2$. The $I^2$ statistic may be particularly unreliable in meta-analyses with a small number of studies (e.g. < 10).^10,11 The sample sizes within individual studies can inflate or deflate $I^2$ under different circumstances.^11,12 Moreover, $I^2$ inherits the following misunderstanding about the distribution of the $Q$ statistic under the null hypothesis: the ${\chi }_{k-1}^2$ approximation holds for large within-study sample sizes, but it is not accurate for small and moderate sample sizes.^13,14

In response to the above shortcomings, the associated $95\mathrm{\%}$ confidence interval (CI) should be reported to accompany the $I^2$ statistic.^10,15,16 Also, under the null hypothesis, distributions of the $Q$ statistic for different effect measures (e.g. standardized mean difference [SMD]) have been proposed to adjust for the inaccuracy of the standard chi-square approximation^17–20; they provide a solution to calculating the CI of $I^2$. CIs may be more desirable than point estimates of $I^2$ because they give an appreciation of the spectrum of possible extents of heterogeneity (e.g. mild to moderate). A spectrum of the $I^2$ statistic can be more robust to nuisance factors compared with the point estimate, enabling appropriate interpretation of the overall estimate of the intervention effect.¹⁶ Methods to calculate $95\mathrm{\%}$ CIs of $I^2$ have been discussed in former research.^4,21 Nevertheless, more intensive studies need to be implemented to compare different methods’ performance (e.g. the coverage probability) in practical situations.

This article uses a simulation study under various scenarios to obtain informative conclusions of preferable point and interval estimates of $I^2$. The rest of the article is organized as follows. We first review the setups of a meta-analysis, including various types of effect measures, in Section 2. Section 3 reviews various point and interval estimators of $I^2$. Section 4 presents the simulation study comparing the multiple estimators. We conclude this article with a brief discussion in Section 5.

2. Setups of meta-analysis

2.1. Common-effect and random-effects models

Consider a meta-analysis that collects $k$ independent studies. Let ${\mu }_i$ be the underlying true effect size in study $i(i=1,\dots ,k)$. Each study reports an estimate of the effect size and its sample variance, denoted by $y_i$ and $s_i^2$. These data are commonly modeled as $y_i\sim N\left({\mu }_i,s_i^2\right)$. Although $s_i^2$ is subject to sampling error, it is usually treated as a fixed, known value. This assumption is generally valid if each study’s sample size is reasonably large.

If study-specific true effect sizes are assumed to follow a normal distribution, that is, ${\mu }_i\stackrel{\text{iid}}{\sim }N\left(\mu ,{\tau }^2\right)$, then this is the random-effect (RE) model that accounts for heterogeneity. Here, $\mu$ is the overall mean effect size, and ${\tau }^2$ is the between-study variance. If ${\tau }^2=0$, then ${\mu }_i=\mu$ for all studies. This implies that the collected studies are homogeneous, and it leads to the common-effect (CE) model. The RE model encompasses within-study ($s_i^2$) and between-study (${\tau }^2$) variation, in contrast to the CE model that includes within-study variation only. We denote $w_{i,\mathrm{C}\mathrm{E}}=1/s_i^2$ as the weight assigned to each study under the CE model. The $Q$ statistic is defined as $Q=\sum w_{i,\mathrm{C}\mathrm{E}}{\left(y_i-{\hat{\mu }}_{\mathrm{C}\mathrm{E}}\right)}^2$, where ${\hat{\mu }}_{\mathrm{C}\mathrm{E}}=\sum w_{i,\mathrm{C}\mathrm{E}}{y}_i/\sum w_{i,\mathrm{C}\mathrm{E}}$ is the pooled CE estimate of the overall effect size $\mu$. It follows a ${\chi }_{k-1}^2$ distribution under the null hypothesis. Under the RE model, using the between-study variance estimate ${\hat{\tau }}^2$, the overall mean effect size is estimated as

$${\hat{\mu }}_{\text{RE}}\left({\hat{\tau }}^2\right)=\frac{\sum w_{i,\text{RE}}\left({\hat{\tau }}^2\right)y_i}{\sum w_{i,\text{RE}}\left({\hat{\tau }}^2\right)}$$ (1)

where $w_{i,\mathrm{R}\mathrm{E}}\left({\hat{\tau }}^2\right)=1/\left(s_i^2+{\hat{\tau }}^2\right)$.

2.2. Meta-analysis with a continuous outcome

Suppose each study in a meta-analysis compares a treatment group with a control group. Denote $n_{i0}$ and $n_{i1}$ as the sample sizes in the control and treatment groups in study $i$. The continuous outcome measures of participants in each group are assumed to follow normal distributions. The subject-level data in each arm have means ${\mu }_{i0}$ and ${\mu }_{i1}$ and variances ${\gamma }_{i0}^2$ and ${\gamma }_{i1}^2$. The sample means are denoted as ${\bar{y}}_{i0}$ and ${\bar{y}}_{i1}$, and the sample variances are denoted as $s_{i0}^2$ and $s_{i1}^2$ for $i=1,\dots ,k$.

If the outcome measures have a meaningful scale and all studies in the meta-analysis are reported on the same scale, the mean difference (MD) between the two groups, ${\mu }_i={\mu }_{i1}-{\mu }_{i0}$, is often used as the effect size. An estimate of the MD can be obtained from each study, denoted as $y_i={\bar{y}}_{i1}-{\bar{y}}_{i0}$. The variances of samples in two arms are frequently assumed to be equal, i.e., ${\gamma }_{i0}^2={\gamma }_{i1}^2={\gamma }_i^2$. The ${\gamma }_i^2$ is estimated as the pooled sample variance $s_{i\mathrm{P}}^2=\left[\left(n_{i0}-1\right)s_{i0}^2+\left(n_{i1}-1\right)s_{i1}^2\right]/\left(n_{i0}+n_{i1}-2\right)$. Therefore, the estimated within-study variance of $y_i$ is $s_i^2=\left(\frac{1}{n_{i0}}+\frac{1}{n_{i1}}\right)s_{i\mathrm{P}}^2$.

Another commonly used effect measure for continuous outcomes is the SMD, because this unit-free measure permits different scales in the collected studies and is deemed more comparable across studies.²² The SMD effect measure is ${\mu }_i=\left({\mu }_{i1}-{\mu }_{i0}\right)/{\gamma }_i$. Known as Cohen’s $d$, it is frequently estimated as follows: $y_i=\left({\bar{y}}_{i1}-{\bar{y}}_{i0}\right)/s_{i\mathrm{P}}$. The exact within-study variance of Cohen’s $d$ can be derived as a complicated form of gamma functions,²³ but researchers often use different simpler forms to approximate it.^24–26 For example, $s_i^2=\frac{1}{n_{i0}}+\frac{1}{n_{i1}}+\frac{y_i^2}{2\left(n_{i0}+n_{i1}-2\right)}$. As $s_i^2$ depends on $y_i$, they are correlated. The correlation may increase as the sample sizes decrease, because the coefficient of $y_i^2$ in the formula increases. Cohen’s $d$ is shown to be biased in small sample sizes.²⁴ Therefore, we do not consider it further. Instead, we study the bias-corrected estimator Hedge’s $g$, which is usually adopted when sample sizes are small. Suggested by Hedges and Olkin,^24(p86) it is computed as $y_i=\left[1-\frac{3}{4\left(n_{i0}+n_{i1}\right)-9}\right]\frac{{\bar{y}}_{i1}-{\bar{y}}_{i0}}{s_{i\mathrm{P}}}$ with an estimated variance $s_i^2=\frac{1}{n_{i0}}+\frac{1}{n_{i1}}+\frac{y_i^2}{2\left(n_{i0}+n_{i1}\right)}$.

Except for this formula to estimate the within-study variance, Lin and Aloe²⁷ summarized many other formulas. Using different formulas can result in different estimates of the overall SMD, but this topic is beyond the scope of this paper. Like Cohen’s $d$, the observed data $y_i$ and $s_i^2$ are also correlated when using Hedge’s $g$ as the effect measure, which may affect the estimation results of meta-analyses.²⁸

2.3. Meta-analysis with a binary outcome

Suppose a $2\times 2$ table is available from each collected study in a meta-analysis with a binary outcome (i.e. individual-level outcomes are reported from $k$ studies). Denote $n_{i00}$ and $n_{i01}$ as the number of participants without and with an event in the control group, respectively; $n_{i10}$ and $n_{i11}$ are the data cells in the treatment group. The sample sizes in the control and treatment groups are $n_{i0}=n_{i00}+n_{i01}$ and $n_{i1}=n_{i10}+n_{i11}$. Also, denote $p_{i0}$ and $p_{i1}$ as the population event rates in the two groups.

The odds ratio (OR) is frequently used as the effect measure for a binary outcome; its true value in the study $i$ is ${\mathrm{O}\mathrm{R}}_i=\left[p_{i1}\left(1-p_{i0}\right)\right]/\left[p_{i0}\left(1-p_{i1}\right)\right]$. Using individual-level data, the OR is estimated by ${\widehat{\mathrm{O}\mathrm{R}}}_i=\left(n_{i00}{n}_{i11}\right)/\left(n_{i01}{n}_{i10}\right)$. The ORs are usually combined on a logarithmic scale in meta-analyses, because the distribution of the estimated log OR, $y_i=\log {\widehat{\mathrm{OR}}}_i$, is better approximated by a normal distribution. The within-study variance of $y_i$ is estimated as $s_i^2=\frac{1}{n_{i00}}+\frac{1}{n_{i01}}+\frac{1}{n_{i10}}+\frac{1}{n_{i11}}$.

Moreover, the risk ratio (RR) and risk difference (RD) are also popular effect metrics, but they are not discussed in this article. Although RRs are more interpretable measures of association for clinicians,^29,30 the debate continues over the merits of the OR versus RR and their interpretations.^31,32 Doi et al.³³ argued that RRs should no longer be used in meta-analyses, because the RR depends on prevalence more so than on the strength of exposure-outcome association that it is supposed to reflect. Specifically, the RR is a ratio of two conditional probabilities that vary with outcome prevalence, whereas the OR is a true effect magnitude measure representing the multiplicative increase in odds of outcome from an unexposed state to an exposed state. The RD can be easily computed from the OR with the fixed baseline risk. When generating simulated meta-analyses for RDs and RRs under the RE model, it is unrealistic to naturally limit $p_{i0}$ and $p_{i1}$ within the range [0, 1] if the true overall effect size is given. This is because the normality assumption ${\mu }_i\stackrel{\mathrm{iid}}{\sim }N\left(\mu ,{\tau }^2\right)$ can generate extreme values of a non-zero $\tau$. For example, a true RD of study $i$ is simulated from $N\left(\mathrm{0.2,\; 0.2}\right)$ as ${\mu }_i=0.8$, then $p_{i1}=p_{i0}+{\mu }_i$ will be beyond 1 if $p_{i0}$ is fixed to larger than 0.2. To overcome this issue, an alternative method is truncating such improper probabilities so they are between 0 and 1, but this constraint can produce bias which cannot be distinguished from the bias caused by sampling error.^28,34 Thus, the undesired effect of bounding the probabilities can be problematic, and inevitable when conducting simulation studies for RRs and RDs. Although some meta-analysts try to explore other models to simulate data, there still does not exist a general method that fixes the biased problem and is well accepted in the literature. Bakbergenuly et al.³⁴ evaluated the performance of a number of data-generating models, such as the binomial generalized linear mixed model with logit link function and the beta-binomial model, when effects are RRs. It appears no gold standard was concluded, and they encouraged future research to explore this topic. Therefore, we focus on analyzing the results of ORs when studies return binary outcomes.

When sample sizes are small, some data cells may be 0, even if the event is not rare. In general, if a $2\times 2$ table contains zero cells, a fixed value of 0.5 is added to each data cell to reduce bias and avoid computational errors.^35–37 Although this continuity correction may not be optimal in some cases and alternative corrections can be used,^38–41 we use the adding 0.5 correction if it is not specially mentioned in the following sections.

3. Estimates of the ${\mathit{I}}^2$ statistic

3.1. Point estimates

Because point estimates of the between-study variance ${\tau }^2$ are used to calculate $I^2$ intervals, we first introduce these point estimators. As $I^2$ depends on ${\tau }^2$, these estimators further lead to point estimates of $I^2$.

3.1.1. Method-of-moments approach

The estimator of ${\tau }^2$ can be derived from the method-of-moments approach, which is based on the generalized $Q$ statistic,⁴² $Q_a=\sum a_i{\left(y_i-{\hat{\mu }}_a\right)}^2$, where $a_i$ represents the weight assigned to the study $i$ and ${\hat{\mu }}_a=\sum a_i{y}_i/\sum a_i$. By equating $Q_a$ to its expected value, the general formula for the heterogeneity variance can be derived as

$${\hat{\tau }}^2=\text{max}\left\{0,\frac{Q_a-\left(\sum a_i{s}_i^2-\frac{\sum a_i^2{s}_i^2}{\sum a_i}\right)}{\sum a_i-\frac{\sum a_i^2}{\sum a_i}}\right\}.$$ (2)

The DerSimonian–Laird (DL) estimator uses the CE model weights $a_i=w_{i,\mathrm{C}\mathrm{E}}$, leading to⁴³:

$${\hat{\tau }}_{\text{DL}}^2=\text{max}\left\{0,\frac{\sum w_{i,\text{CE}}{\left(y_i-{\hat{\mu }}_{\text{CE}}\right)}^2-(k-1)}{\sum w_{i,\text{CE}}-\frac{\sum w_{i,\text{CE}}^2}{\sum w_{i,\text{CE}}}}\right\}$$

Note that the DL estimators can produce negative variance estimates and are truncated to zero in such cases.

3.1.2. Sidik–Jonkman (SJ) method

Sidik and Jonkman⁴⁴ proposed a two-step estimator producing positive ${\tau }^2$ estimates

$${\hat{\tau }}_{\text{SJ}}^2=\frac{1}{k-1}\sum \frac{1}{1+\left(s_i^2/{\hat{\tau }}_0^2\right)}{\left(y_i-{\hat{\mu }}_{\text{SJ}}\right)}^2$$

where ${\hat{\tau }}_0^2=\sum {\left(y_i-\bar{y}\right)}^2/k$ is the initial heterogeneity variance estimate and ${\hat{\mu }}_{\mathrm{S}\mathrm{J}}$ is calculated from equation (1) with weights $w_{i,\mathrm{R}\mathrm{E}}\left(s_i^2/{\hat{\tau }}_0^2\right)=1/\left[1+\left(s_i^2/{\hat{\tau }}_0^2\right)\right]$.

3.1.3. Restricted maximum likelihood (REML) method

Based on the marginal distribution of the RE model, $y_i\sim N\left(\mu ,s_i^2+{\tau }^2\right)$, the maximum likelihood (ML) estimate ${\hat{\tau }}_{\mathrm{M}\mathrm{L}}^2$ is obtained by maximizing the log-likelihood function:

$$l\left(\mu ,{\tau }^2\right)=-\frac{k}{2}\text{log}(2\pi )-\frac{1}{2}\sum \text{log}\left(s_i^2+{\tau }^2\right)-\frac{1}{2}\sum \frac{{\left(y_i-\mu \right)}^2}{s_i^2+{\tau }^2}$$

To derive the REML estimator, the above log-likelihood function is transformed to exclude the parameter $\mu .$⁴⁵ By doing so, REML avoids assuming $\mu$ is known and is therefore thought to be an improvement on the ML estimator.⁴⁶ The modified log-likelihood function is

$$l_{\text{R}}\left({\tau }^2\right)=-\frac{k}{2}\text{log}(2\pi )-\frac{1}{2}\sum \text{log}\left(s_i^2+{\tau }^2\right)-\frac{1}{2}\sum \frac{{\left[y_i-{\hat{\mu }}_{\text{RE}}\left({\hat{\tau }}_{\text{ML}}^2\right)\right]}^2}{s_i^2+{\tau }^2}-\frac{1}{2}\text{log}\left(\sum \frac{1}{s_i^2+{\tau }^2}\right)$$

By maximizing this modified $\log$-likelihood function to ${\tau }^2$, the formula of the between-study variance estimate is

$${\hat{\tau }}_{\text{REML}}^2=\text{max}\left\{0,\frac{\sum a_i^2\left[{\left(y_i-{\hat{\mu }}_{\text{RE}}\left({\hat{\tau }}_{\text{ML}}^2\right)\right)}^2-s_i^2\right]}{\sum a_i^2}+\frac{1}{\sum a_i}\right\}$$

where $a_i=1/\left(s_i^2+{\hat{\tau }}_{\text{REML}}^2\right)$. The REML estimate is calculated by using an iteration scheme. Fisher scoring algorithm is used for the iteration of the REML estimates in this article, as implemented in the R package “metafor.”⁴⁷

3.2. Interval estimates

3.2.1. Interval estimates for ${\mathit{I}}^2\mathbf{b}\mathbf{a}\mathbf{s}\mathbf{e}\mathbf{d}\mathbf{o}\mathbf{n}\mathbf{t}\mathbf{h}\mathbf{e}\mathit{Q}$ statistic

The $I^2$ statistic is originated from the $Q$ statistic by assuming within-study variances are equal (i.e. $s_i^2={\sigma }^2$) and by equating the observed $Q$ with its expectation, so we have

$$I^2=\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}=\frac{Q-(k-1)}{Q}$$ (3)

which is a function of $Q$.⁴⁸ A widely used truncation (i.e. $I^2$ is set to 0 if $Q\le k-1$) is applied because conceptually the $I^2$ statistic should be non-negative. Using equation (3), the $I^2$ interval estimate can be calculated by evaluating quantiles from the cumulative distribution function (CDF) of the $Q$ statistic (i.e. $F_Q$). Biggerstaff and Jackson²¹ (BJ) developed three approaches to approximate the distribution of $Q$ under the RE model.

The two-moment gamma approximation of $F_Q$, with shape parameter $\alpha$ and scale parameter $\beta$, is obtained by matching the first two moments of the gamma and $Q$ distributions. Explicit expressions for the mean and variance of $Q$ are

$$\text{E}(Q)=k-1+\left(S_1-\frac{S_2}{S_1}\right){\tau }^2$$

$$\text{Var}(Q)=2(k-1)+4\left(S_1-\frac{S_2}{S_1}\right){\tau }^2+2\left(S_2+\frac{S_2^2}{S_1^2}-2\frac{S_3}{S_1}\right){\tau }^4$$

where $S_r=\sum w_{i,\mathrm{C}\mathrm{E}}^r$. The proof for the two formulas is included by Biggerstaff and Tweedie.⁴⁹ Using any non-negative estimates for ${\tau }^2$ in the above two formulas, the first two moments of $Q$ can be estimated, denoted as $\stackrel{\mathrm{ˆ}}{\mathrm{E}}\left(Q\right)$ and $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left(Q\right)$. Therefore, solving equations $\mathrm{E}\left(Q\right)=\alpha \beta$ and $\mathrm{V}\mathrm{a}\mathrm{r}\left(Q\right)=\alpha {\beta }^2$ by plugging in estimated values gives $\hat{\alpha }=\left[\stackrel{\mathrm{ˆ}}{\mathrm{E}}\right(Q){]}^2/\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(Q)$ and $\hat{\beta }=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left(Q\right)/\stackrel{\mathrm{ˆ}}{\mathrm{E}}\left(Q\right)$. The $F_Q$ is then approximated by computing the gamma CDF with $\hat{\alpha }$ and $\hat{\beta }$.

The Pearson type III distribution provides an extension of the previous two-moment gamma approximation by adding the third central moment (TCM) of $Q$, which is derived similarly to the variance of $Q$:

$$\text{TCM}(Q)=\text{E}\left[{(Q-\text{E}(Q))}^3\right]=8(k-1)+24\left(S_1-\frac{S_2}{S_1}\right){\tau }^2+24\left(S_2-2\frac{S_3}{S_1}+\frac{S_2^2}{S_1^2}\right){\tau }^4+8\left(S_3-3\frac{S_4}{S_1}+3\frac{S_2{S}_3}{S_1^2}-\frac{S_2^3}{S_1^3}\right){\tau }^6$$

Matching all three moments with parameters of the Pearson type III distribution, emphasizing the dependence on ${\tau }^2$, gives

$$r\left({\tau }^2\right)=\frac{4\text{Var}{(Q)}^3}{\text{TCM}{(Q)}^2},\theta \left({\tau }^2\right)=\frac{2\text{Var}(Q)}{\text{TCM}(Q)},\text{and}\gamma \left({\tau }^2\right)=\text{E}(Q)-\frac{2\text{Var}{(Q)}^2}{\text{TCM}(Q)}$$

Therefore, the approximation of $F_Q$ can easily be calculated from the Pearson type III CDF with location parameter $\gamma \left({\tau }^2\right)$, shape parameter $r\left({\tau }^2\right)$ and rate parameter $\theta \left({\tau }^2\right)$. This approximation can be obtained by plugging in ${\hat{\tau }}^2$ to the three parameters. Note that although the three-moment Pearson type III approximation is intended as an improvement on the two-moment gamma approximation as it matches a further moment, it has support $\left[\gamma \left({\hat{\tau }}^2\right),\mathrm{\infty }\right)$, hence it is not appropriate to approximate $F_Q$ when values of $Q$ are extremely small, especially if those values are less than $\gamma \left({\hat{\tau }}^2\right)$.

A further approximation expected to be more accurate in the tails of the distribution is the saddlepoint approximation, given in the present case by Kuonen⁵⁰ using the Barndorff–Nielsen formulation. This requires the cumulant generating function of $Q$, denoted by $K\left(s\right)$, and its first two derivates, given by

$$K(s)=-\frac{1}{2}\sum _{i=1}^{k-1}\text{log}\left(1-2{\lambda }_{i}s\right),K^{(1)}(s)=\sum _{i=1}^{k-1}\frac{{\lambda }_i}{1-2{\lambda }_{i}s},\text{and}{K}^{(2)}(s)=2\sum _{i=1}^{k-1}{\left(\frac{{\lambda }_i}{1-2{\lambda }_{i}s}\right)}^2$$

where $s<1/2{\lambda }_1$ and ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_{k-1}\ge 0$ with ${\lambda }_k=0$ are the ordered eigenvalues of $\mathrm{S}={\mathbf{\Sigma }}^{1/2}\mathbf{A}{\mathbf{\Sigma }}^{1/2}$. Here, and $\mathbf{\Sigma }$ is the diagonal matrix with entries $s_i^2+{\tau }^2$. Let $\mathbf{A}=\mathbf{W}-\left(1/\sum _i w_{i,\mathrm{C}\mathrm{E}}\right)\mathit{w}{\mathit{w}}^t$, where $\mathbf{W}$ is the diagonal matrix containing the $w_{i,\mathrm{C}\mathrm{E}}=1/s_i^2,\mathit{w}$ is the vector containing the $w_{i,\mathrm{C}\mathrm{E}}$, and the superscript t denotes matrix transpose. Plugging in the ${\tau }^2$ estimate, the saddlepoint approximating $\mathrm{C}\mathrm{D}\mathrm{F}{F}_S\left(x\right)\approx P(Q\le x)$ can be calculated in two steps. First, we solve the equation $K^{\left(1\right)}\left(\hat{s}\right)=x$ for $\hat{s}$, the solution referred to as the saddlepoint. Next, we compute $a=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(\hat{s}\right)\sqrt{2[\hat{s}x-K(\hat{s}\left)\right]}$ and $b=\hat{s}\sqrt{K^{\left(2\right)}\left(\hat{s}\right)}$. The saddlepoint approximation is then given by $F_S\left(x\right)=\varphi \left(a+\frac{1}{a}\log \left(\frac{b}{a}\right)\right)$, where $\varphi$ is the standard normal CDF. Our objective is to obtain the $I^2$ interval through evaluating the quantiles of the $Q$ statistic, so $F_S\left(x\right)$ is used to estimate $x$ for given probabilities (e.g. 0.025 and 0.975).

The test-based method⁵¹ provides another way to compute the CI of $Q$, and hence for the $I^2$ statistic.¹⁶ Appendix A2 by Higgins and Thompson⁴ discussed in detail for conducting the test-based method to calculate the $95\mathrm{\%}$ CI of the heterogeneity measure

$$H=\sqrt{\frac{Q}{k-1}}$$

where $H$ is defined to be 1 whenever $Q\le k-1$. Because $I^2=1-1/H^2$ is a monotone-increasing function with $H^2$, we briefly present results that are used to estimate the $95\mathrm{\%}$ CI of $H$ here, and the corresponding $95\mathrm{\%}$ CI for $I^2$ can be readily calculated via the relationship between $I^2$ and $H^2$. The logarithm of $Q$ is used in this method to remove some of the skew inherent in the distribution of $Q$. A test-based standard error of $\log \left(H\right)$ is

$$\text{SE}[\text{log}(H)]=\{\begin{array}{c}\sqrt{\left\{\frac{1}{2(k-2)}\left[1-\frac{1}{3{(k-2)}^2}\right]\right\}},Q\le k\\ \frac{\text{log}(Q)-\text{log}(k-1)}{2(\sqrt{2Q}-\sqrt{2k-3})},Q>k\end{array}$$

Then a $95\mathrm{\%}$ CI for $H$ follows as $\mathrm{e}\mathrm{x}\mathrm{p}\{\log (H)\pm 1.96\mathrm{S}\mathrm{E}[\log \left(H\right)\left]\right\}$. Therefore, a test-based interval estimate for $I^2$ is constructed using the lower and upper bounds of $H$.

The non-parametric bootstrap CI of $I^2$ can be obtained by sampling $k$ studies with replacement from the observed pairs $\left(y_i,s_i^2\right)$, and $I^2$ is estimated for each bootstrap sample using equation (3). Repeating the process $B$ (e.g. 1000) times, a $95\mathrm{\%}\mathrm{C}\mathrm{I}$ is given by the 2.5th and 97.5th percentiles of the $B{I}^2$ values.

In sum, five methods to estimate $I^2$ intervals are summarized in this subsection. It should be noted that the three methods using the approximated $F_Q$ need first to estimate ${\tau }^2$, whereas the other two methods do not depend on ${\hat{\tau }}^2$, as shown in Table 1.

Table 1.

Summary of the five methods to calculate confidence intervals for I² based on the Q statistic.

Methods to estimate I2 intervals		Use the estimated τ2
Biggerstaff and Jacksons approximated cumulative distribution functions of Q	Two-moment gamma (TMG)	√
	Pearson type III distribution (PIII)	√
	Saddlepoint approximation (S)	√
Test-based approach (T)		—
Non-parametric bootstrap (NPBS)		—

(a) Methods requiring the estimated between-study variance, √; and (b) methods not requiring the estimated between-study variance, —.

3.2.2. Interval estimates for ${\mathit{I}}^2$ based on the between-study variance

Consider the DL estimate of the between-study variance ${\tau }^2$, the $Q$ statistic can be expressed in the form of ${\hat{\tau }}^2$ via equation (2) when $a_i=w_{i,\mathrm{C}\mathrm{E}}$. Note that $Q=k-1$ when ${\hat{\tau }}^2=0$, this setting matches with the widely used truncation that $Q$ is truncated as $k-1$ if $Q\le k-1$. Replacing the $Q$ statistic with the form of ${\hat{\tau }}^2$ in equation (3), $I^2$ can be expressed as a function of the estimated between-study variance

$$I^2=\frac{{\hat{\tau }}^2}{{\hat{\tau }}^2+\frac{(k-1)\sum s_i^{-2}}{{\left(\sum s_i^{-2}\right)}^2-\sum s_i^{-4}}}$$ (4)

In this expression, the summary of the within-study variance (i.e. the moment-based sampling error) is treated as follows:

$${\hat{\sigma }}^2=\frac{(k-1)\sum s_i^{-2}}{{\left(\sum s_i^{-2}\right)}^2-\sum s_i^{-4}}$$

Nevertheless, considering $I^2$ as a function of ${\hat{\tau }}^2$ depends on the accuracy of the summary estimate ${\hat{\sigma }}^2$ because the calculation or interpretation of the $I^2$ statistic can be seriously distorted if ${\hat{\sigma }}^2$ provides a misleading estimate.⁵² Nevertheless, we use the moment-based sampling error throughout this article because it is consistent with the definition of $I^2$. Improving the summary estimate of the within-study variance will be explored in our future research. For a given meta-analysis, interval estimates of $I^2$ can be calculated from interval estimates of the between-study variance via the monotone-increasing function $I^2\left({\hat{\tau }}^2\right)$ in equation (4). Therefore, calculating the CI for $I^2$ is one step further than estimating the CI for ${\tau }^2$. Researchers have conducted comprehensive overviews to compare estimation methods for ${\tau }^2$ and its uncertainty.^53,54 Although equation (4) is derived to use the DL estimate of ${\tau }^2$, other methods can be used to calculate ${\hat{\tau }}^2$ because different estimators aim to estimate the same true between-study variance. Three-point estimators of ${\tau }^2$ in Section 3.1 and the following six interval estimators of ${\tau }^2$ are considered to obtain the CI for ${\tau }^2$, and thus the CI for $I^2$, in this article.

Specifically, we summarize interval estimation methods for the between-study variance below:

• Profile likelihood CI of the REML estimator (PL-REML). The PL method⁵⁵ is based on the log-likelihood function and is an iterative process that provides CIs for the between-study variance, considering the fact that $\mu$ needs to be estimated as well. The $95\mathrm{\%}$ PL CI for ${\tau }^2$ consists of the values that are not rejected by the likelihood ratio test with ${\tau }^2$ under the null hypothesis. For the REML estimator, the ${\tau }^2$ values in the CI are obtained by solving

  $$l_R\left({\tau }^2\right)>l_R\left({\hat{\tau }}_{\text{REML}}^2\right)-\frac{1}{2}{\chi }_{1,0.95}^2$$

  where ${\chi }_{\mathrm{1,0.95}}^2=3.841$ is the 95th quantile of the ${\chi }_1^2$ distribution. The method produces wide CIs with very high coverage probabilities when ${\tau }^2=0$, and the coverage probabilities reduce to the nominal level as ${\tau }^2$ increasing.⁵⁶

• Q-profile $CI\left(QP\right)$. The QP method is based on the generalized $Q$ statistic ($Q_a$ in Section 3.1) when $a_i=1/\left(s_i^2+{\tau }^2\right)$, which follows ${\chi }_{k-1}^2$. Viechtbauer⁵⁶ shows that the $Q$-profile CI is obtained by iteratively solving $Q_a\left({\tilde{\tau }}_{\mathrm{L}}^2\right)={\chi }_{k-\mathrm{1,0.975}}^2$ and $Q_a\left({\tilde{\tau }}_{\mathrm{U}}^2\right)={\chi }_{k-\mathrm{1,0.025}}^2$, where ${\tilde{\tau }}_{\mathrm{L}}^2$ and ${\tilde{\tau }}_{\mathrm{U}}^2$ are the lower and upper confidence limits, respectively. The corresponding CIs have been shown to achieve nominal coverage probabilities even in small samples.⁵⁶ However, the estimated within-study variance $s_i^2$ is not the true within-study variance ${\sigma }_i^2$ for each study. Therefore, in practice, the generalized $Q$ statistic no longer follows the assumed chi-squared distribution. This method is implemented in the R package “metafor” as the default approach to compute the CI for ${\tau }^2$.

• Biggerstaff and Jackson CI (BJ). Using the CDF of $Q,F_Q\left(x;{\tau }^2\right)$, Biggerstaff and Jackson²¹ proposed a method to calculate a $95\mathrm{\%}$ CI for the between-study variance by obtaining the solutions of the equations:

  $$\left(1-F_Q\left(x;{\tau }^2\right)=0.025,F_Q\left(x;{\tau }^2\right)=0.025\right)$$

  When $F_Q\left(x;{\tau }^2=0\right)<0.025$, the interval is set as $\left[\mathrm{0,\; 0}\right]$. If $1-F_Q\left(x;{\tau }^2=0\right)>0.025$, the lower bound of CI is set equal to 0. The CDF $F_Q\left(x,{\tau }^2\right)$ may be calculated using the algorithm by Farebrother⁵⁷ for the positive linear combination of chi-squared random variables.
• Jackson $CI\left(J\right)$. An extension of the BJ CI is suggested by Jackson⁵⁸ using $Q_a$. The generalized statistic $Q_a$ has been shown to be as a linear combination of ${\chi }^2$ random variables so that methods like BJ can be used. The CDF of $Q_a$, $F_{Q_a}\left(x;{\tau }^2\right)$, is a continuous and strictly decreasing function of ${\tau }^2$. The $95\mathrm{\%}\mathrm{C}\mathrm{I}$ of ${\tau }^2$ is obtained as:

  $$\left(1-F_{Q_a}\left(x;{\tau }^2\right)=0.025,F_{Q_a}\left(x;{\tau }^2\right)=0.025\right)$$

  When $F_{Q_a}\left(x;{\tau }^2=0\right)<0.025$, the interval is set as $\left[\mathrm{0,\; 0}\right]$. If $1-F_{Q_a}\left(x;{\tau }^2=0\right)>0.025$, the lower bound of CI is set equal to 0. For moderate ${\tau }^2$, Jackson recommends using the $J$ interval with weights $a_i=1/s_i$, which are used in this article. The BJ and $J$ CIs for ${\tau }^2$ are calculated using the R code provided by Jackson.⁵⁸
• Sidik and Jonkman CI (SJ). Sidik and Jonkman⁴⁴ propose a method based on the SJ estimator with the 2.5th and 97.5th quantiles of the ${\chi }_{k-1}^2$ distribution:

  $$\left(\frac{(k-1){\hat{\tau }}_{\text{SJ}}^2}{{\chi }_{k-1,0.975}^2},\frac{(k-1){\hat{\tau }}_{\text{SJ}}^2}{{\chi }_{k-1,0.025}^2}\right)$$

  As ${\hat{\tau }}_{\mathrm{S}\mathrm{J}}^2$ takes non-negative values, the interval should also be non-negative. Simulation studies indicate that the SJ intervals have very poor coverage probability when ${\tau }^2$ is small, but as $k$ and ${\tau }^2$ increase the coverage probability becomes close to the nominal value.^44,56

• Bootstrap CI. For any consistent and non-negative estimator of ${\tau }^2$, parametric bootstrap CIs can be obtained by generating $k$ values from the distribution $y_i\sim N\left({\hat{\mu }}_{\mathrm{R}\mathrm{E}}\left({\hat{\tau }}^2\right),{\hat{\tau }}^2+s_i^2\right)$, where ${\hat{\tau }}^2$ is the between-study variance estimate and ${\hat{\mu }}_{\mathrm{R}\mathrm{E}}\left({\hat{\tau }}^2\right)$ given by equation (1). Next, estimate the between-study variance based on the bootstrap sample. After repeating this process $B$ (e.g. 1000) times, the CI is constructed by taking the 2.5th and 97.5th percentiles of the distribution of ${\hat{\tau }}^2$ values. Non-parametric bootstrap CIs are obtained via a similar process, where $k$ studies are sampled with replacement from the observed pairs ($y_i,s_i^2$). For each bootstrap sample, ${\tau }^2$ can be estimated using the same specified method (e.g. REML). Repeating the process $B$ times, a $95\mathrm{\%}\mathrm{C}\mathrm{I}$ is given by the 2.5th and 97.5th percentiles of the $B{\hat{\tau }}^2$ values. The normal distribution assumption of observed effects is not required in the non-parametric bootstrap method, but its coverage performance has been doubted because of the substantial deviation from the nominal level in simulation studies.⁵⁶

So far, for generic effect measures, multiple approaches to calculating $95\mathrm{\%}\mathrm{C}\mathrm{I}\mathrm{s}$ of $I^2$ are presented in two directions: based on the $Q$ statistic or based on the between-study variance. Nevertheless, these methods have been criticized by researchers because they can be unreliable in real-world meta-analysis.^8,59 For the $Q$-profile method, the null distribution of the generalized $Q$ statistic follows ${\chi }_{k-1}^2$ may not be an accurate approximation, especially when study-specific sample sizes are small or moderate. Three methods based on three approximations of $F_Q$ proposed by Biggerstaff and Jackson²¹ also require sufficiently large studies (i.e. large sample sizes of studies) and the assumption that effect sizes are normally distributed. For the test-based method, Hoaglin⁸ points out: (1) the CI involving the test-based standard error is valid only under the null hypothesis; (2) the standard normal approximation, $Z=\sqrt{2Q}-\sqrt{2k-3}$, used in the method requires “large” degrees of freedom (e.g. over 100); and (3) subtracting $\log (k-1$) is not exactly the same as subtracting the mean of $\log \left(Q\right)$. Therefore, the test-based CI for $I^2$ can be unreliable to reflect heterogeneity. Other approaches to improve the estimation of CIs for ${\tau }^2$, and thus for $I^2$, have been discussed recently.^21,54,60 For example, Knapp et al.⁶⁰ suggested a modified $Q$ profile method using a different weighting scheme for the generalized $Q$ statistic to determine the lower bound of the interval for ${\tau }^2$, and the upper bound is still the same as that of the original $Q$-profile method. However, the improvement of the modified $Q$-profile method is subtle, and the weighting scheme for the lower bound is lacking when effect measures are SMDs.

To handle the problem that ${\chi }_{k-1}^2$ can be an inaccurate null distribution, Kulinskaya et al.^18,20 proposed a series of methods, which provide appropriate CIs for ${\tau }^2$, by combining the $Q$-profile method with corrected null approximations of the $Q$ statistic. The distribution of $Q$ under the null hypothesis of homogeneity depends on statistics used to estimate the effects and the weights. Two methods to estimate CIs for ${\tau }^2$, thus for $I^2$, are introduced for two effect measures, the SMD and the OR, as follows:

• Kulinskaya–Dollinger–Bjøkestøl CI (KDB). When using Hedge’s $g$ as the estimator of SMD, Kulinskaya et al.¹⁸ derived $O(1/n)$ corrections to moments of $Q$ and suggested using the chi-squared distribution with degrees of freedom equal to the estimate of the corrected first moment, denoted by ${\chi }_{\mathrm{E}\left(Q\right)}^2$, to approximate the distribution of $Q$. The detailed expression of $\mathrm{E}\left(Q\right)$ is provided along with the R code by Kulinskaya et al.,¹⁸ and they are not presented here because the concrete form is complicated. The upper and lower confidence limits for ${\tau }^2$ can be calculated iteratively from the lower and upper quantiles of ${\chi }_{\mathrm{E}\left(Q\right)}^2$:

  $$Q\left({\tau }_{\text{L}}^2\right)={\chi }_{\text{E}(Q),0.975}^2,Q\left({\tau }_U^2\right)={\chi }_{\text{E}(Q),0.025}^2$$

  Then, the corresponding CI for $I^2$ is obtained via equation (4).
• Kulinskaya–Dollinger CI (KD). When effect measures are log ORs, Kulinskaya and Dollinger²⁰ obtain corrected approximations for the mean and variance of the $Q$ statistic under the null hypothesis. They then match those corrected moments to construct a gamma distribution that closely fits the null distribution of $Q$, and their simulations confirm that the gamma approximation outperforms the chi-squared approximation.²⁰ The improved approximation blends theoretical derivation with simulation results. Let ${\mathrm{E}}_{\mathrm{K}\mathrm{D}}\left(Q\right)$ denote the corrected expectation of $Q$ when ${\tau }^2=0$. This corrected first moment can be written as

  $${\text{E}}_{\text{KD}}(Q)=k-1-0.687\left[k-1-{\text{E}}_{\text{th}}(Q)\right]$$

  where ${\mathrm{E}}_{\mathrm{t}\mathrm{h}}\left(Q\right)$ is a theoretical moment obtained from their general expansion of the mean of $Q$ for arbitrary binary effect measures. The detailed expression of ${\mathrm{E}}_{\mathrm{t}\mathrm{h}}\left(Q\right)$ is presented in Appendix B.3 of Kulinskaya and Dollinger.²⁰ For large sample sizes, ${\mathrm{E}}_{\mathrm{t}\mathrm{h}}\left(Q\right)$ converges to $k-1$. The corrected variance of $Q$, denoted by ${\mathrm{V}\mathrm{a}\mathrm{r}}_{\mathrm{K}\mathrm{D}}\left(Q\right)$, is a quadratic function of the corrected mean and it is calculated by

  $${\text{Var}}_{\text{KD}}(Q)=4.74(k-1)-12.17{\text{E}}_{\text{KD}}(Q)+\frac{9.42}{k-1}{\left[{\text{E}}_{\text{KD}}(Q)\right]}^2$$

  Then, the shape parameter $\alpha$ of the gamma distribution approximating $F_Q$ is estimated by $\hat{\alpha }={\left[{\mathrm{E}}_{\mathrm{K}\mathrm{D}}\left(Q\right)\right]}^2/{\mathrm{V}\mathrm{a}\mathrm{r}}_{\mathrm{K}\mathrm{D}}\left(Q\right)$, and the scale parameter $\beta$ is estimated by $\hat{\beta }={\mathrm{V}\mathrm{a}\mathrm{r}}_{\mathrm{K}\mathrm{D}}\left(Q\right)/{\mathrm{E}}_{\mathrm{K}\mathrm{D}}\left(Q\right)$. Therefore, the KD interval estimate of ${\tau }^2$ is obtained by iteratively solving:

  $$Q\left({\tau }_{\text{L}}^2\right)=F_{Q,0.975},Q\left({\tau }_{\text{U}}^2\right)=F_{Q,0.025}$$

  The corresponding CI for $I^2$ is calculated via equation (4). Different from all other methods, the KD interval estimate of ${\tau }^2$ is based on the $2\times 2$ table where 0.5 is added to each cell regardless of the existence of zero cells; this change is adjusted in the programming.

Among the methods presented in this subsection, four (PL-REML, SJ, PBS-${\tau }^2$, and NPBS-${\tau }^2$) need to use the estimated ${\tau }^2$, whereas others can directly calculate interval estimates of ${\tau }^2$ without using the point estimate.

4. Simulation study

4.1. Simulation settings

We conducted simulation studies to investigate the performance of different interval estimators of the $I^2$ statistic. Following the framework by Morris et al.⁶¹ to design simulations:

• Aims. The primary goal is to compare the performance of different methods’ $95\mathrm{\%}$ CIs for $I^2$. The secondary aim is to compare three-point estimators of $I^2$.

• Data-generating mechanisms. The number of studies in a simulated meta-analysis was set to $k=\mathrm{5,20}$, and 50. Denote $n=\left(n_1,\dots ,n_k\right)$, where $n_i$ represents the sample size of the study $i(i=1,\dots ,k)$. When $k=5$, a vector $n$ represents sample sizes of an artificial meta-analysis was fixed as (10, 20, 30, 40, 50), then we gradually increased it to (50, 75, 100, 125, 150), and to (150, 250, 350, 450, 550). Three different settings indicated the considered sample sizes were small, medium, and large. When $k=20$, the sample size vector was specified as four replicates of $n$ when $k=5$. For example, considering $10\le n_i\le 50$ and $i=1,\dots ,20$, the sample size vector was set by combining four vectors (10, 20, 30, 40, 50). Similarly, 10 replicates of $n$ when $k=5$ were used to construct the sample size vector when $k=50$. The control/treatment allocation ratio was set to $1:1$ in all studies, which is commonly used in real-world applications. Specifically, $n_{i0}=n_{i1}=n_i/2$, where $n_{i0}$ participants were assigned to the control group and $n_{i1}$ participants were assigned to the treatment group.

When effect measures were MDs, each participan’s outcome measure was sampled from $N\left({\mu }_{i0},{\gamma }_i^2\right)$ in the control group or $N\left({\mu }_{i0}+{\mu }_i,{\gamma }_i^2\right)$ in the treatment group. Without loss of generality, the baseline effect ${\mu }_{i0}$ of the study $i$ was generated from $N\left(\mathrm{0,\; 1}\right)$. The study-specific standard deviation ${\gamma }_i$ was sampled from $U\left(\mathrm{1,5}\right)$, and it was generated anew for each simulated meta-analysis. The MD ${\mu }_i$ was sampled from $N\left(\mu ,{\tau }^2\right)$. Table 2 shows the specified values for the overall MD $\mu$ and the between-study standard deviation $\tau$.

Table 2.

Vectors of the between-study standard deviation (τ) and specified values of the true overall effect size (μ).

	Overall mean difference	Overall standardized mean difference		Overall log odds ratio
Range of the sample size ni	μ = 0 or 1	μ = 0	μ = 0.8	μ = 0	μ = 1
10 ≤ ni ≤ 50	τ = (0.50, 1.10, 2.20)	τ = (0.18, 0.37, 0.73)	τ = (0.19, 0.38, 0.76)	τ = (0.37, 0.73, 1.46)	τ = (0.39, 0.78, 1.56)
50 ≤ ni ≤ 150	τ = (0.30, 0.60, 1.20)	τ = (0.10, 0.20, 0.40)	τ = (0.10, 0.21, 0.42)	τ = (0.20, 0.40, 0.80)	τ = (0.21, 0.43, 0.85)
150 ≤ ni ≤ 550	τ = (0.20, 0.30, 0.60)	τ = (0.05, 0.11, 0.21)	τ = (0.06, 0.11, 0.22)	τ = (0.11, 0.21, 0.43)	τ = (0.11, 0.23, 0.46)

Given the range of study-specific sample sizes and the true overall effect size, a between-study standard deviation is chosen from one of three values in the corresponding vector, and it is used to generate meta-analyses.

When effect measures were SMDs, each participant’s outcome measure was generated from $N\left({\mu }_{i0},{\gamma }_i^2\right)$ in the control group or $N\left({\mu }_{i1},{\gamma }_i^2\right)$ in the treatment group. The baseline effect ${\mu }_{i0}$ of $i$ th study was generated from $N\left(\mathrm{0,\; 1}\right)$, and the study-specific standard deviation ${\gamma }_i$ was generated anew for each meta-analysis by sampling from $U\left(\mathrm{1,5}\right)$. The SMD ${\mu }_i=\left({\mu }_{i1}-{\mu }_{i0}\right)/{\gamma }_i$ was sampled from the normal distribution $N\left(\mu ,{\tau }^2\right)$, so ${\mu }_{i1}={\mu }_i{\gamma }_i+{\mu }_{i0}$. The overall SMD $\mu$ and the between-study standard deviation $\tau$ were set as in Table 2.

When effect measures were $\log$ ORs, the event numbers $n_{i01}$ and $n_{i11}$ in the control and treatment groups were sampled from $\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{a}\mathrm{l}\left(n_{i0},p_{i0}\right)$ and $\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{a}\mathrm{l}\left(n_{i1},p_{i1}\right)$, respectively. The event rate in the control group $p_{i0}$ was sampled from $U\left(\mathrm{0.3,0.7}\right)$ representing a common event,⁶² and it was generated anew for each meta-analysis. The event rate in the treatment group $p_{i1}$ was calculated using $p_{i0}$ and the study-specific log OR ${\mu }_i$; specifically, $p_{i1}={\left[1+e^{-{\mu }_i}\left(1-p_{i0}\right)/p_{i0}\right]}^{-1}$. The study-specific $\log$ OR ${\mu }_i$ was sampled from $N\left(\mu ,{\tau }^2\right)$. The settings of the overall $\log$ OR $\mu$ and the between-study standard deviation $\tau$ were presented in Table 2.

For each simulation setting above, 10,000 meta-analyses were generated. For a simulated meta-analysis, the study-specific effect size and the within-study variance were estimated as $y_i$ and $s_i^2$ in Section 2. The RE model was applied to each simulated meta-analysis, and the between-study variance was estimated by three methods (DL, SJ, and REML) introduced in Section 3.1. We skipped simulated meta-analyses whose REML estimates of ${\tau }^2$ could not be obtained (e.g. the solution did not converge) until enough simulated meta-analyses were generated.

• Estimands of interest. We estimated $I^2$ and the corresponding $95\mathrm{\%}$ CI for each simulated meta-analysis. The true value of $I^2$ was calculated by equation (4) with the true between-study variance ${\tau }^2$.

• Methods to be evaluated. For MDs, we compared 12 methods to calculate $95\mathrm{\%}$ CIs of $I^2$, five methods (TMG, PIII, S, T, and NPBS-Q) introduced in Section 3.2.1 and seven methods (PL-REML, QP, BJ, J, SJ, NPBS-${\tau }^2$, and PBS-${\tau }^2$) introduced in Section 3.2.2. These 12 methods were also compared when effect measures were SMDs or log ORs, but the KDB CI or the KD CI was added to the comparison. Among the methods needing to use the estimated between-study variance, the SJ method used the SJ estimate of ${\tau }^2$, and other methods used the REML estimate of ${\tau }^2$. Moreover, estimated $I^2$ using three different estimators (DL, SJ, and REML) of ${\tau }^2$ were also compared.

• Performance measures. Coverage probabilities of $95\mathrm{\%}$ CIs, lengths of interval estimates, standard deviations of lengths, biases, and root mean squared errors were examined.

We provide all R code for the simulations at the Open Science Framework (https://osf.io/qu26v/).

4.2. Simulation results

4.2.1. Properties of ${\mathit{I}}^2$ estimates

For the point estimates of $I^2$ using the DL, SJ, and REML estimators of ${\tau }^2$, the SJ method stood out when the between-study variance was large. Table 3 shows estimates of bias for the three estimation methods when the estimand was MD. Generally, the SJ methods had the highest bias compared to DL and REML when $\tau$ was small or moderate, but the lowest bias when $\tau$ was large. Additionally, DL and REML estimates of $I^2$ had extremely similar performance and were often biased downward, particularly when $\tau$ was large.

Table 3.

Biases of estimated I² using the estimated between-study variance of three methods: DerSimonian–Laird (DL), Sidik–Jonkman (SJ), and restricted maximum likelihood (REML) methods. Results are from the simulation study using mean differences as effect measures.

	The true overall mean difference μ = 0 or 1
	ni = 10–50			ni = 50–150			ni = 150–550
Method	τ = 0.5	τ = 1.1	τ = 2.2	τ = 0.3	τ = 0.6	τ = 1.2	τ = 0.2	τ = 0.3	τ = 0.6
k=5
DL	0.020	−0.125	−0.123	−0.021	−0.140	−0.128	−0.069	−0.137	−0.140
SJ	0.204	−0.003	−0.075	0.155	−0.022	−0.082	0.099	−0.007	−0.084
REML	0.015	−0.127	−0.122	−0.025	−0.144	−0.127	−0.074	−0.142	−0.140
k=20
DL	0.031	−0.024	−0.016	−0.023	−0.048	−0.023	−0.046	−0.054	−0.028
SJ	0.306	0.080	0.002	0.241	0.055	−0.007	0.180	0.070	−0.007
REML	0.032	−0.019	−0.009	−0.025	−0.047	−0.018	−0.048	−0.054	−0.022
k=50
DL	0.044	0.008	−0.001	−0.008	−0.014	−0.007	−0.024	−0.021	−0.010
SJ	0.330	0.095	0.011	0.262	0.071	0.003	0.197	0.085	0.004
REML	0.045	0.014	0.004	−0.007	−0.011	−0.004	−0.024	−0.020	−0.007

To illustrate this point, consider the case where $k=20$ and $n_i$ was between 50 and 150. When $\tau =0.3$, estimates of the bias were $-\mathrm{0.023,0.241}$, and −0.025 for DL, SJ, and REML methods, respectively. The SJ method’s magnitude of estimated bias was more than 10 times that of DL or REML. As $\tau$ increased to 0.6, the magnitudes of bias were approximately equal (−0.048 for DL, 0.055 for SJ, and – 0.047 for REML). However, when $\tau$ was 1.2, SJ had the lowest estimated bias at −0.007 compared to −0.023 and −0.018 for DL and REML. This held true across nearly all parameter combinations studied, as well as for SMD and $\log$ OR. The estimated magnitude of the bias for the $\log$ OR was higher than that of MD or SMD (Tables S1 to S4 in the Supplemental Material). This was particularly striking when $\mu =1$ (Table S4 in the Supplemental Material). RMSE followed a similar, but far less extreme, pattern for all parameter combinations and estimands studied (Tables S5 to S9 in the Supplemental Material).

4.2.2. CIs for MDs and SMDs

Table 4 shows the simulation-based coverage probabilities in studies of the MD for each of the CI methods introduced for the $I^2$ statistic. CIs based on the BJ estimate of $F_Q$ generally behaved similarly. When the number of studies was small $(k=5)$, interval coverage for the TMG, PIII, and S methods decreased with increasing ${\tau }^2$, regardless of the size of the individual studies in the meta-analysis. For example, when an individual study sample size $n_i$ was between 10 and 15, we observed over-conservative coverage when $\tau =0.5$ ($100\mathrm{\%}$ TMG, $99.2\mathrm{\%}$ PIII, and $99.9\mathrm{\%}$ S), very close to the nominal coverage when $\tau =1.1$ ($95.5\mathrm{\%}$ TMG, $95.1\mathrm{\%}$ PIII, and $95.5\mathrm{\%}\mathrm{S})$. As $k$ increased to 20 or 50, often a non-linear relationship between CI coverage and the between-study variance was present. This trend depended on $n_i$, highlighting the importance of considering $k,n_i$, and ${\tau }^2$ together when conducting a meta-analysis.

Table 4.

Coverage probabilities (in percentage, %) of estimated I² 95% confidence intervals of 12 methods in simulated studies where effect measures are mean differences.

	The true overall mean difference μ=0 or 1
	ni = 10–50			ni = 50–150			ni = 150–550
Method	τ = 0.5	τ = 1.1	τ = 2.2	τ = 0.3	τ = 0.6	τ = 1.2	τ = 0.2	τ = 0.3	τ = 0.6
k=5
TMG	100.0	95.5	83.4	100.0	95.0	82.8	100.0	96.8	82.3
PIII	99.2	95.1	83.4	99.6	94.7	82.8	99.6	96.6	82.4
S	99.9	95.5	83.1	100.0	95.0	82.3	100.0	96.9	81.8
T	93.8	90.9	70.7	95.5	92.6	72.1	95.4	93.4	75.2
NPBS-Q	68.0	65.8	63.2	66.7	64.1	61.9	64.9	63.8	61.5
PL-REML	96.4	97.0	93.4	97.6	97.8	94.1	98.0	98.0	94.2
QP	93.3	93.7	93.9	94.7	94.9	94.8	94.9	94.8	94.8
BJ	93.2	93.7	93.7	94.6	94.8	94.9	95.2	94.7	94.7
J	93.7	93.9	93.9	94.8	94.8	94.9	94.7	94.9	94.7
SJ	46.6	76.0	86.7	55.7	79.0	88.2	64.3	76.9	87.7
PBS-τ2	99.9	97.2	84.5	100.0	97.1	83.7	100.0	98.1	83.6
NPBS-τ2	73.2	69.5	66.8	71.4	68.0	65.7	69.5	67.8	65.8
k=20
TMG	99.3	92.1	95.1	99.8	90.6	93.4	97.2	90.5	93.4
PIII	97.7	91.1	95.0	99.1	90.4	93.8	96.7	90.1	93.8
S	98.4	91.4	95.2	99.4	90.2	93.8	96.9	90.0	93.7
T	90.2	76.4	59.2	94.1	78.8	62.0	92.6	80.9	63.0
NPBS-Q	88.5	85.6	83.1	85.7	83.5	81.0	85.0	83.4	80.3
PL-REML	93.6	92.3	92.9	96.4	93.8	93.9	95.6	94.3	94.5
QP	91.4	92.5	92.6	94.1	94.2	94.1	95.1	95.3	94.8
BJ	91.5	93.3	94.0	94.0	94.5	94.2	94.9	94.6	94.5
J	93.1	93.1	93.0	94.5	94.2	94.1	95.1	95.0	94.7
SJ	8.1	58.8	83.8	18.0	68.2	87.6	32.6	63.8	87.7
PBS-τ2	98.6	91.9	92.8	99.1	91.0	91.2	95.0	90.9	91.7
NPBS-τ2	88.9	88.2	87.8	86.1	86.1	85.7	85.6	85.5	85.1
k=50
TMG	96.8	95.1	97.8	96.7	94.5	97.1	93.7	93.5	96.4
PIII	95.1	93.7	97.3	96.4	94.3	97.3	93.4	93.6	96.7
S	95.6	94.3	96.4	96.4	94.3	96.6	93.4	93.6	95.8
T	86.1	73.6	55.3	91.7	78.0	58.7	87.4	80.4	60.7
NPBS-Q	92.7	92.2	90.7	91.3	90.0	88.6	90.4	89.3	87.7
PL-REML	89.7	91.1	91.4	94.0	94.2	94.5	94.8	94.9	94.9
QP	89.1	90.7	91.2	94.4	94.4	94.5	95.0	95.1	94.8
BJ	89.3	92.5	93.5	94.3	94.5	94.6	95.1	94.9	95.0
J	91.6	91.7	91.9	94.7	94.4	94.5	95.2	94.9	94.8
SJ	0.3	35.4	79.9	1.8	50.4	87.4	8.1	43.9	86.8
PBS-τ2	92.7	92.9	93.3	92.9	93.6	94.0	92.5	93.3	93.5
NPBS-τ2	92.1	91.8	91.8	90.4	91.1	91.4	90.2	90.7	90.9

TMG: two-moment gamma; PIII: Pearson type III; S: Saddlepoint; T: test-based; NPBS: non-parametric bootstrap; Q: the Q statistic; PL: profile-likelihood; REML: restricted maximum likelihood; QP: Q-profile; BJ: Biggerstaff and Jackson; J: Jackson; SJ: Sidik and Jonkman; PBS: parametric bootstrap; τ²: the between-study variance.

The performance of the T approach mimicked that of TMG, PIII, and S, but with lower CI coverage across different simulation settings, which was clearly illustrated with a large number of studies but a small within-study sample size. Table 4 shows, when $k=50$ and $n_i$ was between 10 and 50, the coverage probabilities were $86.1\mathrm{\%},73.6\mathrm{\%}$, and $55.3\mathrm{\%}$ for $\tau =0.5,\tau =1.1$, and $\tau =2.2$, respectively. This was below the average coverage of other intervals based on the $Q$ statistic. For the T approach, when $k$ was fixed, the coverage probability decreased as $\tau$ increased. This finding indicates that the commonly used test-based method was inappropriate for calculating a $95\mathrm{\%}\mathrm{C}\mathrm{I}$ of $I^2$ if the meta-analysis is highly heterogeneous. The similarities in coverage trajectory over parameters of meta-analyses among the T, TMG, PIII, and S methods likely reflect the fact that all four methods use the approximated distribution of the $Q$ statistic, while the decrease in overall coverage probability for the T method may be because it does not use the estimate of ${\tau }^2$. Interestingly, although the coverage probability was worse than the other three methods, the average interval length of the T method was comparatively shorter (Table 5).

Table 5.

Average lengths (in percentage, %) of estimated I² 95% confidence intervals of 12 methods are shown with standard deviations in parentheses. Results are from the simulation study using mean differences as effect measures.

	The true overall mean difference μ = 0 or 1
	ni = 10–50			ni = 50–150			ni = 150–550
Method	τ = 0.5	τ = 1.1	τ = 2.2	τ = 0.3	τ = 0.6	τ = 1.2	τ = 0.2	τ = 0.3	τ = 0.6
k=5
TMG	72.7 (10.0)	79.9 (11.7)	88.0 (11.5)	72.6 (9.8)	79.5 (11.5)	87.7 (11.5)	74.0 (10.4)	78.4 (11.4)	87.3 (11.2)
PIII	72.3 (10.0)	77.1 (12.3)	69.7 (22.4)	72.4 (9.7)	77.3 (11.8)	70.5 (22.0)	73.6 (10.2)	76.7 (11.5)	72.5 (20.3)
S	72.7 (10.0)	79.7 (11.7)	83.4 (15.5)	72.6 (9.8)	79.4 (11.5)	83.3 (15.5)	74.0 (10.4)	78.3 (11.4)	84.0 (14.3)
T	77.0 (8.6)	69.0 (18.9)	44.3 (28.2)	77.3 (7.8)	69.9 (17.9)	44.6 (27.9)	76.3 (9.9)	71.4 (16.4)	48.5 (27.6)
NPBS-Q	41.2 (30.0)	58.9 (29.2)	77.9 (21.7)	41.0 (29.6)	58.3 (28.7)	77.6 (21.2)	44.7 (29.9)	55.6 (29.3)	76.1 (22.4)
PL-REML	83.3 (9.3)	79.5 (15.9)	58.1 (27.3)	83.4 (8.8)	80.2 (14.9)	58.6 (27.0)	83.3 (9.6)	81.3 (13.8)	62.3 (26.0)
QP	83.7 (18.7)	80.5 (19.9)	57.6 (29.1)	84.0 (17.6)	81.2 (18.8)	57.8 (28.7)	84.2 (17.4)	82.2 (18.5)	61.7 (28.0)
BJ	83.2 (18.1)	80.1 (19.8)	58.0 (28.4)	83.5 (17.4)	80.9 (18.8)	58.3 (28.1)	83.7 (17.1)	81.8 (18.4)	62.0 (27.4)
J	83.7 (18.5)	82.7 (19.7)	60.1 (30.5)	83.9 (17.6)	82.8 (18.6)	59.3 (29.7)	84.5 (17.3)	83.7 (18.2)	63.5 (29.0)
SJ	53.6 (13.7)	50.7 (13.7)	37.0 (16.6)	54.3 (13.3)	51.6 (13.1)	37.5 (16.4)	53.9 (13.1)	52.1 (13.2)	39.6 (16.2)
PBS-τ2	74.7 (9.3)	81.1 (10.7)	85.0 (14.5)	74.5 (9.1)	80.7 (10.4)	84.8 (14.4)	75.8 (9.6)	79.7 (10.4)	85.4 (12.9)
NPBS-τ2	46.5 (30.7)	62.4 (28.2)	80.6 (20.6)	45.8 (30.1)	61.5 (27.9)	80.4 (20.1)	49.4 (30.1)	59.1 (28.7)	78.7 (21.2)
k=20
TMG	58.9 (12.0)	65.1 (11.7)	35.6 (13.9)	57.8 (11.5)	65.1 (11.0)	35.5 (13.7)	61.8 (11.5)	66.3 (10.3)	40.0 (14.5)
PIII	58.0 (11.2)	57.1 (13.0)	26.6 (11.3)	57.3 (10.9)	58.8 (12.1)	27.6 (11.5)	60.5 (10.6)	60.8 (11.1)	31.2 (12.5)
S	58.3 (11.6)	60.1 (12.7)	29.7 (12.2)	57.5 (11.2)	61.1 (11.9)	30.4 (12.3)	60.9 (11.1)	62.9 (11.1)	34.3 (13.2)
T	52.9 (8.0)	38.2 (15.2)	12.2 (8.3)	53.1 (7.7)	40.5 (14.9)	13.0 (8.6)	52.5 (9.1)	43.5 (14.1)	15.2 (9.8)
NPBS-Q	50.1 (18.2)	59.9 (14.7)	33.6 (15.9)	48.6 (17.7)	59.7 (14.4)	33.8 (15.6)	53.8 (16.0)	60.2 (13.7)	37.9 (16.7)
PL-REML	60.8 (11.2)	50.6 (14.5)	21.0 (10.4)	60.3 (10.9)	52.5 (13.9)	21.9 (10.7)	61.2 (10.1)	55.3 (13.0)	25.1 (11.8)
QP	65.7 (14.0)	53.1 (15.9)	21.0 (10.5)	65.2 (13.8)	54.9 (15.2)	21.7 (10.6)	65.9 (12.3)	58.2 (14.2)	25.0 (11.8)
BJ	62.9 (12.4)	52.3 (14.3)	23.8 (10.5)	62.5 (12.5)	53.8 (14.1)	24.3 (10.6)	63.6 (10.9)	56.7 (13.2)	27.3 (11.5)
J	66.8 (14.2)	59.7 (18.7)	21.8 (12.1)	65.7 (13.9)	59.5 (17.2)	22.1 (11.5)	67.8 (12.6)	63.2 (15.9)	25.6 (13.1)
SJ	28.7 (3.3)	24.6 (4.8)	13.9 (4.7)	29.2 (2.8)	25.7 (4.4)	14.6 (4.8)	28.7 (3.1)	26.4 (4.2)	16.0 (4.9)
PBS-τ2	58.4 (12.5)	60.8 (13.6)	27.1 (13.3)	57.5 (12.2)	62.0 (12.5)	28.1 (13.4)	61.3 (11.8)	63.8 (11.5)	32.3 (14.7)
NPBS-τ2	53.5 (20.3)	62.2 (16.5)	30.2 (18.8)	51.3 (19.5)	62.1 (15.7)	31.1 (18.8)	56.5 (17.4)	62.7 (14.9)	35.7 (20.2)
k=50
TMG	48.8 (7.9)	39.0 (7.9)	16.4 (4.3)	48.5 (7.7)	40.0 (7.6)	16.5 (4.2)	50.2 (6.3)	43.1 (7.4)	18.8 (4.7)
PIII	48.0 (7.6)	35.9 (7.8)	14.3 (3.7)	48.0 (7.5)	37.5 (7.7)	14.8 (3.7)	49.1 (6.3)	40.7 (7.6)	16.9 (4.2)
S	48.2 (7.7)	36.7 (7.8)	14.8 (4.1)	48.1 (7.6)	38.1 (7.7)	15.2 (4.0)	49.3 (6.4)	41.2 (7.6)	17.3 (4.5)
T	40.9 (6.6)	22.0 (7.8)	5.9 (2.6)	41.5 (6.3)	23.7 (7.9)	6.3 (2.6)	39.4 (7.3)	26.8 (8.4)	7.5 (3.1)
NPBS-Q	46.3 (10.9)	38.3 (10.6)	16.0 (5.1)	45.2 (10.4)	39.2 (10.6)	16.1 (5.0)	47.4 (9.2)	42.0 (10.6)	18.3 (5.7)
PL-REML	47.2 (7.5)	31.8 (8.2)	11.1 (3.4)	46.9 (7.0)	33.4 (8.0)	11.6 (3.4)	46.2 (6.5)	36.4 (8.1)	13.4 (3.9)
QP	52.7 (9.4)	33.7 (9.0)	11.2 (3.5)	52.4 (8.9)	35.4 (8.6)	11.7 (3.4)	51.3 (7.8)	39.0 (8.9)	13.6 (4.0)
BJ	49.0 (7.3)	33.5 (7.5)	13.9 (4.2)	49.0 (7.1)	34.6 (7.4)	14.1 (4.0)	48.2 (6.5)	37.6 (7.6)	16.0 (4.5)
J	56.2 (11.1)	38.4 (13.5)	11.4 (3.5)	55.1 (10.1)	38.5 (12.0)	11.8 (3.4)	56.1 (9.4)	43.6 (12.7)	13.6 (4.0)
SJ	18.3 (1.5)	15.1 (2.2)	7.9 (1.8)	18.8 (1.1)	16.0 (2.0)	8.4 (1.8)	18.3 (1.3)	16.5 (1.9)	9.3 (1.9)
PBS-τ2	47.2 (8.6)	35.3 (8.6)	12.1 (3.8)	47.3 (8.4)	37.2 (8.4)	12.7 (3.8)	48.8 (6.8)	40.5 (8.2)	14.7 (4.3)
NPBS-τ2	48.0 (12.7)	37.0 (12.2)	12.4 (4.8)	45.9 (11.6)	38.2 (12.0)	12.8 (4.7)	47.8 (10.3)	41.4 (12.1)	14.9 (5.5)

TMG: two-moment gamma; PIII: Pearson type III; S: Saddlepoint; T: Test-based; NPBS: non-parametric bootstrap; Q: the Q statistic; PL: profile-likelihood; REML: restricted maximum likelihood; QP: Q-profile; BJ: Biggerstaff and Jackson; J: Jackson; SJ: Sidik and Jonkman; PBS: parametric bootstrap; τ²: the between-study variance.

The NPBS-Q interval had unacceptable coverage across most of the simulation scenarios. This method hardly achieved nominal coverage probability, with the highest coverage estimated at $92.7\mathrm{\%}$. In general, the coverage of the NPBS-Q interval decreased modestly with larger $\tau$ and $n_i$. However, the primary drive of the low coverage was the number of studies available for resampling in the bootstrap operation, where $k=50$ provided the best coverage. In the case of a small number of studies, the NPBS-Q interval was shorter than other $F_Q$-based intervals.

Compared with $I^2$ intervals based on $F_Q$, most intervals based on ${\tau }^2$ were found to have coverage consistently closer to the nominal $95\mathrm{\%}$. Specifically, the QP, BJ, and J intervals stood out as well-performing in areas where the $F_Q$-based intervals under-performed. PL-REML also performed well, but it was slightly farther from the nominal coverage probability compared to QP, BJ, and J; this was evident when the within-study sample size was between 150 and 550. For example, when $k=20$ and $\tau =0.3$, the coverage probabilities were $95.6\mathrm{\%},95.1\mathrm{\%},94.9\mathrm{\%}$, and $95.1\mathrm{\%}$ for PL-REML, QP, BJ, and J, respectively. Additionally, when $\tau =0.6$, the coverage probabilities of these methods are closer to the nominal coverage, with coverage probabilities of $94.5\mathrm{\%},94.8\mathrm{\%},94.5\mathrm{\%}$, and $94.7\mathrm{\%}$. Interestingly, when $k=50$ and $n_i$ was between 10 and 50, the coverage of the QP, BJ, J, and PL-REML intervals dropped, and the TMG, PIII, and S methods were preferred in this case.

Two ${\tau }^2$-based intervals would not be recommended for practice based on poor coverage probabilities given by the simulation. For different simulation scenarios, the lowest CI coverage was found for the SJ method, which was consistent with prior simulation studies for the interval of ${\tau }^2.$⁵⁴ Table 5 shows that the interval length of the SJ method tended to be the shortest of all studied methods and decreased with increasing $\tau$. Additionally, similar to the NPBS-Q, the NPBS-${\tau }^2$ failed to reach the nominal coverage in any scenario.

The CIs for the SMD when $\mu =0$ or $\mu =0.8$ showed the same trend as described above (Tables S10 to S13 in the Supplemental Material) with two important caveats. First, when $k=50$ and $n_i$ was between 10 and 15, the QP, BJ, and J methods worked well, and they were close to the nominal coverage. This was in direct contrast to the CIs based on $F_Q$, whose coverage probabilities were consistently under $90\mathrm{\%}$ when $\tau$ was moderate or large (Table S7 in the Supplemental Material). Second, the KDB interval also worked well under all studied scenarios, and it had comparable performance to the QP, BJ, and J intervals. These points made any of the QP, BJ, J, or KDB interval estimation methods a reliable choice when the estimand of interest was SMD, regardless of the other meta-analysis parameters.

4.2.3. CIs for $\mathbf{l}\mathbf{o}\mathbf{g}\mathit{O}\mathit{R}$

Similar trends of CI coverage were observed for $\log$ OR as MD, but with a greater magnitude of departure from the nominal coverage level. For both $\mu =0$ and $\mu =1$, the TMG, PIII, and S methods had decreasing coverage with increasing $\tau$. In Table 6, when $k=5$ and $n_i$ was between 10 and 50, a drop from $100\mathrm{\%}$ coverage to $75\mathrm{\%}$ was seen from all three intervals as $\tau$ increased. As $k$ increased to 20 and 50, this decrease in coverage presented for both moderate and large values of $\tau$ and the severity of departure from nominal coverage increased. The $T$ interval also showed a drop in coverage as $\tau$ increased, the magnitude of which was exacerbated as $k$ increased. NPBS-Q, SJ, and NPBS-${\tau }^2$ showed unacceptable coverage in all scenarios studied. PBS-${\tau }^2$ was generally over-conservative when $\tau$ was mild, but it performed poorly as $\tau$ increased.

Table 6.

Coverage probabilities (in percentage, %) of estimated I² 95% confidence intervals of 13 methods when the true overall log odds ratio is 0. Results are from the simulation study using log odds ratios as effect measures.

	The true overall log odds ratio μ = 0
	ni = 10–50			ni = 50–150			ni = 150–550
Method	τ = 0.37	τ = 0.73	τ = 1.46	τ = 0.20	τ = 0.40	τ = 0.80	τ = 0.11	τ = 0.21	τ = 0.43
k=5
TMG	100.0	100.0	74.8	100.0	100.0	82.3	100.0	100.0	83.3
PIII	100.0	100.0	74.9	100.0	100.0	82.3	100.0	100.0	83.3
S	100.0	100.0	74.8	100.0	100.0	82.2	100.0	100.0	83.1
T	98.2	98.6	92.5	96.9	96.3	88.9	96.4	95.5	85.9
NPBS-Q	64.6	61.7	49.7	64.8	63.7	61.8	65.1	65.1	63.5
PL-REML	99.3	99.8	96.2	98.7	98.9	95.4	98.3	98.5	94.8
QP	96.7	97.4	97.2	95.6	95.7	95.9	95.0	95.3	95.2
BJ	96.8	97.3	97.1	95.6	96.1	96.7	95.1	95.5	95.7
J	97.0	97.3	97.1	95.4	95.6	96.3	94.9	95.1	95.4
SJ	51.0	80.0	91.3	54.0	78.5	88.1	54.2	78.6	87.8
PBS-τ2	100.0	100.0	76.5	100.0	100.0	82.6	100.0	100.0	83.8
NPBS-τ2	68.4	66.1	58.5	66.0	64.8	65.0	67.1	66.1	65.3
KD	94.1	95.0	95.5	95.0	95.3	95.6	94.9	95.2	95.1
k=20
TMG	100.0	83.2	67.5	99.9	89.9	87.8	99.9	91.5	91.1
PIII	100.0	83.2	68.0	99.8	89.9	88.0	99.9	91.4	91.2
S	99.9	83.2	67.9	99.7	89.6	88.0	99.6	91.1	91.2
T	99.1	94.3	54.7	97.2	90.0	80.0	96.8	92.2	78.5
NPBS-Q	82.5	75.3	35.4	85.5	84.3	76.2	85.6	85.1	82.8
PL-REML	99.6	93.5	86.8	98.8	95.0	94.7	98.4	94.7	94.4
QP	97.0	97.0	95.3	95.3	95.6	96.0	95.1	95.1	94.9
BJ	97.0	96.2	89.0	95.5	96.0	95.9	95.2	95.2	95.4
J	97.4	97.2	94.0	95.0	95.5	96.3	94.8	95.0	95.1
SJ	10.8	68.6	91.3	14.7	67.7	87.9	16.1	66.9	87.6
PBS-τ2	100.0	82.0	67.9	99.8	89.4	87.8	99.8	91.1	91.1
NPBS-τ2	79.2	75.5	59.5	84.7	84.6	82.8	84.9	85.2	84.7
KD	94.6	95.2	93.7	94.8	95.2	95.4	95.0	95.0	94.8
k=50
TMG	100.0	76.9	44.2	99.6	91.2	89.1	99.4	93.3	93.5
PIII	100.0	77.0	44.9	99.6	91.2	89.2	99.4	93.2	93.5
S	100.0	76.8	44.9	99.6	91.2	89.0	99.4	93.2	93.4
T	99.5	83.2	14.9	97.5	90.1	74.0	97.1	89.5	79.5
NPBS-Q	85.5	72.5	8.4	90.5	88.4	75.6	90.8	90.3	88.2
PL-REML	98.0	87.6	64.6	97.8	94.5	93.9	97.2	95.1	95.0
QP	96.7	95.8	87.5	95.3	95.5	96.0	95.5	95.0	95.1
BJ	96.4	93.2	54.7	95.4	95.6	93.2	95.4	95.4	95.8
J	97.5	96.3	80.2	95.0	95.3	95.9	95.4	94.9	95.4
SJ	0.4	48.6	88.2	1.1	49.8	87.5	1.3	48.1	87.8
PBS-τ2	100.0	75.7	43.5	99.6	90.8	88.6	99.4	93.1	93.0
NPBS-τ2	80.3	71.9	36.2	88.8	88.0	86.2	90.1	90.2	90.2
KD	93.3	93.7	87.9	94.9	95.0	95.2	95.3	94.9	94.9

TMG: two-moment gamma; PIII: Pearson type III; S: Saddlepoint; T: test-based; NPBS: non-parametric bootstrap; Q: the Q statistic; PL: profile-likelihood; REML: restricted maximum likelihood; QP: Q-profile; BJ: Biggerstaff and Jackson; J: Jackson; SJ: Sidik and Jonkman; PBS: parametric bootstrap; τ²: the between-study variance. KD: Kulinskaya-Dollinger.

When $\mu =0$, the QP, BJ, J, and KD intervals maintained the closest to the nominal coverage probability under most parameter combinations. The exception was when $k=50,n_i$ was between 10 and 50, and $\tau =1.46$. A large drop in coverage was observed for all four intervals, particularly the BJ CI, with a coverage of $54\mathrm{\%}$ in this case (Table 6), corresponding to a large decrease in CI length (Table S14 in the Supplemental Material). This was magnified when $\mu =1$, where the dip in coverage and the decrease in CI length were also observed for the case that $n_i$ was between 50 and 150 (Table 7 and Table S15 in the Supplemental Material). The KD interval only showed a major drop in coverage for $\mu =1$ when $k=50$, $n_i$ was between 10 and 50, and $\tau$ was moderate to large. In most parameter combinations, the KD interval outperformed other methods with respect to the coverage probability or the average interval length. Therefore, in the case of the log OR, we suggested using the KD interval of $I^2$.

Table 7.

Coverage probabilities (in percentage, %) of estimated I² 95% confidence intervals of 13 methods when the true overall log odds ratio is 1. Results are from the simulation study using log odds ratios as effect measures.

	The true overall log odds ratio μ = 1
	ni = 10–50			ni = 50–150			ni = 150–550
Method	τ =0.39	τ =0.78	τ =1.56	τ =0.21	τ =0.43	τ =0.85	τ =0.11	τ =0.23	τ =0.46
k=5
TMG	100.0	100.0	73.4	100.0	100.0	80.4	100.0	100.0	82.5
PIII	100.0	100.0	73.5	100.0	100.0	80.5	100.0	100.0	82.5
S	100.0	100.0	73.3	100.0	100.0	80.3	100.0	100.0	82.4
T	98.8	98.9	91.4	97.2	96.5	87.4	96.1	95.0	83.1
NPBS-Q	61.4	57.4	45.0	64.8	63.4	59.1	64.6	64.3	62.2
PL-REML	99.5	99.9	96.1	98.8	99.1	95.5	98.2	98.5	94.9
QP	97.3	97.4	96.9	95.8	95.9	96.3	94.7	95.0	95.4
BJ	97.2	97.3	96.8	95.8	96.1	96.6	94.8	95.2	95.6
J	97.4	97.4	96.9	95.7	95.9	96.5	94.6	95.0	95.4
SJ	54.8	83.1	90.2	52.8	79.0	88.5	52.0	78.1	87.6
PBS-τ2	100.0	100.0	75.7	100.0	100.0	81.0	100.0	100.0	83.1
NPBS-τ2	65.5	62.1	53.0	66.6	65.3	64.2	66.8	66.0	65.2
KD	94.8	95.5	95.3	94.9	95.3	95.8	94.5	94.8	95.3
k=20
TMG	100.0	80.3	63.0	99.9	89.1	86.4	99.8	91.1	91.0
PIII	100.0	80.3	63.6	99.8	89.0	86.5	99.7	91.1	91.0
S	99.8	80.3	63.4	99.7	89.0	86.4	99.6	91.1	91.0
T	99.5	90.8	47.6	97.9	89.2	78.0	96.8	87.6	78.8
NPBS-Q	78.1	68.4	30.4	84.8	82.5	71.8	85.8	84.5	82.1
PL-REML	99.7	91.4	82.9	99.1	95.0	94.2	98.0	94.5	94.9
QP	96.0	95.7	91.6	95.9	96.2	96.7	95.3	95.1	95.2
BJ	95.9	94.6	85.2	95.9	96.3	95.5	95.2	95.1	95.7
J	96.6	95.9	89.9	95.7	96.0	96.8	95.2	95.0	95.5
SJ	14.0	77.9	86.1	14.1	68.2	89.6	12.3	67.7	88.6
PBS-τ2	100.0	79.2	63.4	99.9	88.7	86.2	99.8	90.6	90.7
NPBS-τ2	75.1	70.2	51.6	83.3	82.6	80.1	84.9	84.8	84.8
KD	94.8	94.5	90.3	94.9	95.5	95.9	94.9	94.9	95.0
k=50
TMG	100.0	71.1	34.6	99.8	89.3	85.0	99.4	93.2	93.0
PIII	100.0	71.2	35.3	99.8	89.3	85.2	99.4	93.2	93.0
S	100.0	71.1	35.2	99.8	88.9	84.9	99.4	92.6	92.4
T	99.9	74.0	9.2	97.8	89.3	67.7	97.2	89.6	79.0
NPBS-Q	78.0	61.6	5.6	89.6	86.7	69.7	90.7	90.4	87.1
PL-REML	97.0	82.3	53.3	98.0	93.7	91.6	97.7	95.1	94.9
QP	94.1	91.4	73.9	95.7	95.8	95.7	95.3	95.6	95.5
BJ	93.6	87.8	44.0	95.7	95.3	91.1	95.4	95.6	95.5
J	95.5	92.2	65.5	95.4	95.9	95.2	95.1	95.5	95.6
SJ	0.7	66.6	73.9	0.7	49.8	89.1	0.6	50.0	87.7
PBS-τ2	100.0	69.9	33.8	99.7	89.0	84.5	99.4	93.0	92.4
NPBS-τ2	75.1	70.2	51.6	83.3	82.6	80.1	84.9	84.8	84.8
KD	93.3	90.8	75.9	94.8	94.8	94.3	95.2	95.4	95.2

TMG: two-moment gamma; PIII: Pearson type III; S: Saddlepoint; T: test-based; NPBS: non-parametric bootstrap; Q: the Q statistic; PL: profile-likelihood; REML: restricted maximum likelihood; QP: Q-profile; BJ: Biggerstaff and Jackson; J: Jackson; SJ: Sidik and Jonkman; PBS: parametric bootstrap; τ²: the between-study variance. KD: Kulinskaya-Dollinger.

5. Discussion

In this article, for a meta-analysis, we have compared different methods to calculate the point estimate and the $95\mathrm{\%}$ CI of the $I^2$ statistic. For point estimates of $I^2$, the SJ method is suggested to be used when ${\tau }^2$ is large. Otherwise, the DL method gives a less biased estimate for $I^2$ based on the simulation studies. The interval estimates of $I^2$ are grouped into two categories by their derivation. One group is the methods based on the approximation of the CDF for the $Q$ statistic; another group is the methods viewing $I^2$ as the function of ${\tau }^2$ in equation (4), and they calculate the interval of $I^2$ based on the interval of ${\tau }^2$. Based on the simulation studies, we would suggest the following guidelines:

• When the effect measure is the MD or SMD, use the QP, BJ, or J method to calculate the $95\mathrm{\%}\mathrm{C}\mathrm{I}$ for $I^2$;

• When the effect measure is the $\log$ OR, use the KD method to calculate the $95\mathrm{\%}\mathrm{C}\mathrm{I}$ for $I^2$.

In the case of the $\log \mathrm{O}\mathrm{R}$, the KD method is recommended because it generally outperforms the other methods with respect to the coverage probability of the $95\mathrm{\%}$ CI for $I^2$. Except for the KDB method for the SMD and the KD method for the log OR, all other methods can be used to calculate the CI of the $I^2$ statistic for any type of effect measure.

Although the $I^2$ statistic is widely used to measure the heterogeneity of meta-analyses, it suffers from large uncertainties and should not be used as an absolute measure of heterogeneity. However, the CI of $I^2$ provides an appreciation of the spectrum of possible extents of heterogeneity, which can be more robust to nuisance factors. In practice, meta-analysts should report the $95\mathrm{\%}$ CI for $I^2$ using the recommended methods, which have reasonable interval lengths and provide much more reliable coverage probabilities than the currently used methods (e.g. the test-based method).

Based on the simulation framework in this article, other simulation settings can be considered in future research. For example, when the effect measure is the MD, the chi-squared distribution can be used to generate the study-specific standard deviation. Further studies can provide additional clarity on the guidelines for $I^2$, with our work serving as the baseline.