Estimation of heterogeneity variance based on a generalized Q statistic in meta‐analysis of log‐odds‐ratio

Abstract

For estimation of heterogeneity variance ${\tau }^2$ in meta‐analysis of log‐odds‐ratio, we derive new mean‐ and median‐unbiased point estimators and new interval estimators based on a generalized $Q$ statistic, $Q_F$, in which the weights depend on only the studies' effective sample sizes. We compare them with familiar estimators based on the inverse‐variance‐weights version of $Q$, $Q_{\mathit{IV}}.$ In an extensive simulation, we studied the bias (including median bias) of the point estimators and the coverage (including left and right coverage error) of the confidence intervals. Most estimators add $0.5$ to each cell of the $2\times 2$ table when one cell contains a zero count; we include a version that always adds $0.5$. The results show that: two of the new point estimators and two of the familiar point estimators are almost unbiased when the total sample size $n\ge 250$ and the probability in the Control arm ($p_{\mathit{iC}}$) is 0.1, and when $n\ge 100$ and $p_{\mathit{iC}}$ is 0.2 or 0.5; for $0.1\le {\tau }^2\le 1$, all estimators have negative bias for small to medium sample sizes, but for larger sample sizes some of the new median‐unbiased estimators are almost median‐unbiased; choices of interval estimators depend on values of parameters, but one of the new estimators is reasonable when $p_{\mathit{iC}}=0.1$ and another, when $p_{\mathit{iC}}=0.2$ or $p_{\mathit{iC}}=0.5$; and lack of balance between left and right coverage errors for small $n$ and/or $p_{\mathit{iC}}$ implies that the available approximations for the distributions of $Q_{\mathit{IV}}$ and $Q_F$ are accurate only for larger sample sizes.

Highlights.

What is already known

• Use of inverse‐variance weights based on estimated variances in the conventional $Q$ statistic makes it very difficult to approximate the null distribution of $Q$.

• Related moment‐based estimators of the heterogeneity variance (${\tau }^2$), such as the DerSimonian‐Laird estimator, have considerable bias.

What is new

• We study estimation of ${\tau }^2$ based on $Q_F$, a generalized $Q$ statistic with fixed effective‐sample‐size weights.

• For point estimation of ${\tau }^2$ we consider both mean‐unbiased and novel median‐unbiased estimators. The new estimators are based on the Farebrother approximation to the distribution of $Q_F$.

• In an extensive simulation study, we compared four new point estimators of ${\tau }^2$ and two new interval estimators with traditional estimators.

• We provide practical guidelines for choosing appropriate point and interval estimators for LOR.

• For median bias, which was not studied previously, an important finding is that the majority of the standard estimators have negative bias for larger sample sizes. This means that their median values of ${\widehat{\tau }}^2$ are too low. Two new estimators consistently result in almost median‐unbiased estimation for moderate to large $n$.

Potential impact for RSM readers outside the authors' field

• We provide practical guidelines for choosing appropriate point and interval estimators of ${\tau }^2$ in meta‐analysis of log‐odds‐ratio.

• Some popular estimators of heterogeneity variance ${\tau }^2$ such as the DL and REML point estimators and PL intervals have unacceptable bias or coverage. We recommend instead new point and interval estimators of ${\tau }^2$ that use constant effective‐sample‐size weights. Some of these estimators are already implemented in metafor.

1. INTRODUCTION

As the measure of effect, meta‐analyses of binary outcomes from randomized trials most often use the odds ratio (OR), preferably analyzed as log‐odds‐ratio (LOR). The customary random‐effects analyses assess heterogeneity by using Cochran's $Q$ statistic ¹ and estimate the between‐study variance, ${\tau }^2$, for use in inverse‐variance weights. The resulting weights, based on estimated variances, underlie various shortcomings of that approach. Thus, for assessing heterogeneity and estimating ${\tau }^2$, we have studied $Q_F$, a version of Cochran's $Q$ statistic in which the weights involve only the studies' arm‐level sample sizes. $Q_F$ belongs to a class of generalized $Q$ statistics, introduced by DerSimonian and Kacker, ² in which the $w_i$ are arbitrary positive constants.

Our initial motivation came from random‐effects meta‐analyses of mean difference (MD) and standardized mean difference (SMD) ³ and, later, LOR, ⁴ where a weighted mean whose weights involved only those effective sample sizes performed well in estimating the overall effect.

Further developments produced $Q_F$ and accurate approximations to its distribution function, for testing heterogeneity of MD, ⁵ SMD, ⁵ and three binary effect measures (LOR, log‐relative‐risk, and risk difference). ⁶ Those studies also examined approximations for the customary $Q$ statistic ($Q_{\mathit{IV}}$) and investigated estimates for ${\tau }^2$ in MD and SMD.

In the present paper, we derive from $Q_F$ new point and interval estimators of ${\tau }^2$ for meta‐analysis of LOR. In particular, we study mean‐ and median‐unbiased estimators. Similar new estimators of ${\tau }^2$ are available in the procedure rma in metafor. ⁷ However, the performance of these point and interval estimators of ${\tau }^2$ had not been evaluated by simulations. Therefore, we carried out an extensive simulation study of the bias of the point estimators and the coverage of the confidence intervals. For comparison we included familiar point and interval estimators of ${\tau }^2$.

Section 2 briefly reviews study‐level estimation of LOR. Section 3 reviews the generic random‐effects model and describes the $Q$ statistic. Section 4 describes random‐effects models for LOR. Section 5 discusses approximations to the distributions of $Q_F$ and $Q_{\mathit{IV}}$. Section 6 introduces new point and interval estimators of ${\tau }^2$ for LOR. Section 7 describes the simulation design, and Section 8 summarizes the results. Section 9 examines an example of meta‐analysis using LOR. Section 10 offers a summary and discussion. The Supporting Information provides further details on the interval estimators, the simulation results, and the example.

2. STUDY‐LEVEL ESTIMATION OF LOG‐ODDS‐RATIO

Analyses of log‐odds‐ratio usually adopt binomial distributions to model the numbers of events. In Study $i$ ($i=1,\dots ,K$), $X_{\mathit{iT}}$ and $X_{\mathit{iC}}$ denote the numbers of events in the $n_{\mathit{iT}}$ subjects in the Treatment arm and the $n_{\mathit{iC}}$ subjects in the Control arm. Thus, we treat $X_{\mathit{iT}}$ and $X_{\mathit{iC}}$ as independent binomial variables:

$$X_{\mathit{iT}}\sim \mathrm{Bin}\left(n_{\mathit{iT}},p_{\mathit{iT}}\right)\text{and}{X}_{\mathit{iC}}\sim \mathrm{Bin}\left(n_{\mathit{iC}},p_{\mathit{iC}}\right).$$ (1)

The log‐odds‐ratio for Study $i$ is

$${\theta }_i={\log }_e\left(\frac{p_{\mathit{iT}}\left(1-p_{\mathit{iC}}\right)}{p_{\mathit{iC}}\left(1-p_{\mathit{iT}}\right)}\right)\text{estimated}\mathrm{by}{\widehat{\theta }}_i={\log }_e\left(\frac{{\stackrel{⌣}{p}}_{\mathit{iT}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iC}}\right)}{{\stackrel{⌣}{p}}_{\mathit{iC}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iT}}\right)}\right),$$ (2)

where ${\stackrel{⌣}{p}}_{\mathit{ij}}$ is an estimate of $p_{\mathit{ij}}$.

The customary estimators of $p_{\mathit{iT}}$ and $p_{\mathit{iC}}$ are ${\tilde{p}}_{\mathit{ij}}=X_{\mathit{ij}}/n_{\mathit{ij}}$ (maximum likelihood). A reasonable alternative adds 0.5 to $X_{\mathit{ij}}$ and adds 1 to $n_{\mathit{ij}}$: ${\widehat{p}}_{\mathit{ij}}=\left(X_{\mathit{ij}}+0.5\right)/\left(n_{\mathit{ij}}+1\right)$ eliminates bias of order $1/n_{\mathit{ij}}$ and provides the least biased estimator of log‐odds. ⁸

As inputs, a two‐stage meta‐analysis uses estimates of the ${\theta }_i$ (${\widehat{\theta }}_i$) and estimates of their variances (${\widehat{v}}_i^2$). The (conditional, given $p_{\mathit{ij}}$ and $n_{\mathit{ij}}$) asymptotic variance of ${\widehat{\theta }}_i$, derived by the delta method, is

$$v_i^2=\mathrm{Var}\left({\widehat{\theta }}_i\right)=\frac{1}{n_{\mathit{iT}}{p}_{\mathit{iT}}\left(1-p_{\mathit{iT}}\right)}+\frac{1}{n_{\mathit{iC}}{p}_{\mathit{iC}}\left(1-p_{\mathit{iC}}\right)},$$ (3)

estimated by substituting ${\tilde{p}}_{\mathit{ij}}$ or ${\widehat{p}}_{\mathit{ij}}$ for $p_{\mathit{ij}}$. The estimator of the variance using the ${\widehat{p}}_{\mathit{ij}}$ and $n_{\mathit{ij}}+1$ instead of $n_{\mathit{ij}}$ in (3) is unbiased in large samples, but Gart et al. ⁸ note that it overestimates the variance for small sample sizes. As we explain in Section 4, the unconditional variance depends on the mechanism generating the $p_{\mathit{iC}}$ and the $p_{\mathit{iT}}$.

3. RANDOM‐EFFECTS MODEL AND THE $Q$ STATISTIC.

In the most widely used method for two‐stage meta‐analysis, ⁹ Cochran's $Q$ statistic serves as the basis for testing heterogeneity and estimating ${\tau }^2$, for use in the inverse‐variance weights, ${\widehat{w}}_i=1/\left({\widehat{v}}_i^2+{\widehat{\tau }}^2\right)$. $Q$ is a weighted sum of the squared deviations of the estimated effects ${\widehat{\theta }}_i$ from their weighted mean ${\overline{\theta }}_w=\sum w_i{\widehat{\theta }}_i/\sum w_i$:

$$Q=\sum w_i{\left({\widehat{\theta }}_i-{\overline{\theta }}_w\right)}^2.$$ (4)

In calculating $Q$, an estimate of ${\tau }^2$ is not yet available, so $w_i$ is simply $1/{\widehat{v}}_i^2$, the reciprocal of the estimated variance of ${\widehat{\theta }}_i$ (as in Cochran ¹ ). We denote the result by $Q_{\mathit{IV}}$.

The process underlying the DerSimonian‐Laird ⁹ estimator of ${\tau }^2$ (${\widehat{\tau }}_{\mathit{DL}}^2$) derives the expected value of $Q_{\mathit{IV}}$ and rearranges the resulting expression to obtain ${\tau }^2$ in terms of $\mathrm{E}\left(Q_{\mathit{IV}}\right)$ and the $v_i^2=\mathrm{Var}\left({\widehat{\theta }}_i\right)$. Substituting the observed value of $Q_{\mathit{IV}}$ for $\mathrm{E}\left(Q_{\mathit{IV}}\right)$ and the ${\widehat{v}}_i^2$ for the $v_i^2$ then yields ${\widehat{\tau }}_{\mathit{DL}}^2$. The convenient step of plugging in the ${\widehat{v}}_i^2$, however, lacks justification; it inderlies the documented shortcomings of the DL method.

As an alternative, we base estimates of ${\tau }^2$ on $Q_F$, studied by Kulinskaya et al., ⁵ in which $w_i={\tilde{n}}_i=n_{\mathit{iC}}{n}_{\mathit{iT}}/n_i$, the effective sample size in Study $i$ ($n_i=n_{\mathit{iC}}+n_{\mathit{iT}}$).

In studying properties of estimators under the random‐effects model, we start by taking as given the observed study‐level effects ${\widehat{\theta }}_i$; that is, we condition on those values. The model includes a distribution for the true ${\theta }_i$, conventionally ${\theta }_i\sim N\left(\theta ,{\tau }^2\right)$. Taking expectations with respect to that model yields unconditional estimators, for comparison with the conditional ones.

The random‐effects model assumes that the ${\widehat{\theta }}_i$ are unbiased estimators of the ${\theta }_i$ and that the $v_i^2=\mathrm{Var}\left({\widehat{\theta }}_i|{\theta }_i\right)$ are the corresponding variances (i.e., the true conditional variances).

To develop estimators of ${\tau }^2$ based on $Q_F$, we define $W=\sum w_i$, $q_i=w_i/W$, and ${\mathrm{\Theta }}_i={\widehat{\theta }}_i-\theta$. In this notation, and expanding ${\overline{\theta }}_w$, Equation (4) can be written as

$$Q=W\left[\sum q_i\left(1-q_i\right){\mathrm{\Theta }}_i^2-\sum _{i\cancel{=}j}{q}_i{q}_j{\mathrm{\Theta }}_i{\mathrm{\Theta }}_j\right].$$ (5)

We distinguish between the conditional distribution of $Q$ (given the ${\theta }_i$) and the unconditional distribution, and the respective moments of ${\mathrm{\Theta }}_i$. For instance, the conditional second moment of ${\mathrm{\Theta }}_i$, denoted by $M_{2i}^c$, is $v_i^2$; and the unconditional second moment, denoted by $M_{2i}$, is $\mathrm{E}\left({\mathrm{\Theta }}_i^2\right)=\mathrm{Var}\left({\widehat{\theta }}_i\right)=\mathrm{E}\left(v_i^2\right)+{\tau }^2$.

Under the REM, it is straightforward to obtain the first moment of $Q_F$ as

$$\mathrm{E}\left(Q_F\right)=W\left[\sum q_i\left(1-q_i\right)\mathrm{E}\left({\mathrm{\Theta }}_i^2\right)\right]=W\left[\sum q_i\left(1-q_i\right)\left(\mathrm{E}\left(v_i^2\right)+{\tau }^2\right)\right].$$ (6)

This expression is similar to Equation (4) in DerSimonian and Kacker ² ; they use $v_i^2+{\tau }^2$ instead of the unconditional variance $\mathrm{E}\left(v_i^2\right)+{\tau }^2$.

Our simulations yield an exact calculation of conditional central moments of LOR, following the implementation of Kulinskaya and Dollinger. ¹⁰

4. RANDOM‐EFFECTS META‐ANALYSIS OF LOG‐ODDS‐RATIO

The standard REM for LOR assumes that $\text{logit}\left(p_{\mathit{iT}}\right)=\text{logit}\left(p_{\mathit{iC}}\right)+{\theta }_i$ for ${\theta }_i\sim N\left(\theta ,{\tau }^2\right)$. The intercept ${\alpha }_i=\text{logit}\left(p_{\mathit{iC}}\right)$ may also be random. Further, $p_{\mathit{iC}}$ and $p_{\mathit{iT}}$ may be correlated. Equation (3) gives the conditional variance of ${\widehat{\theta }}_i$. The full (unconditional) variance of ${\widehat{\theta }}_i$ depends on the generation mechanism for the $p_{\mathit{iC}}$ and was derived in Kulinskaya et al. ¹¹

Conventionally, the $p_{\mathit{iC}}$ are assumed to be fixed. Then

$$\mathrm{E}\left(v_i^2\right)=\frac{1}{n_{\mathit{iT}}{\stackrel{⌣}{p}}_{\mathit{iT}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iT}}\right)}+\frac{1}{n_{\mathit{iC}}{p}_{\mathit{iC}}\left(1-p_{\mathit{iC}}\right)}+{\tau }^2\left(1+\frac{1}{2n_{\mathit{iT}}}\left({\left[{\stackrel{⌣}{p}}_{\mathit{iT}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iT}}\right)\right]}^{-1}-2\right)\right),$$ (7)

where ${\stackrel{⌣}{p}}_{\mathit{iT}}=\text{expit}\left({\alpha }_i+\theta \right)$. For use in $Q_F$, this unconditional variance can be estimated by substituting ${\widehat{p}}_{\mathit{iC}}$ for $p_{\mathit{iC}}$ and ${\overline{p}}_{\mathit{iT}}=\text{expit}\left({\widehat{\alpha }}_i+\widehat{\theta }\right)$ for ${\stackrel{⌣}{p}}_{\mathit{iT}}$, where ${\widehat{\alpha }}_i=\text{logit}\left({\widehat{p}}_{\mathit{iC}}\right)$ and $\widehat{\theta }$ is the estimated LOR. We refer to the estimate ${\overline{p}}_{\mathit{iT}}$ of $p_{\mathit{iT}}$ as model‐based. Alternatively, a naïve estimate uses ${\widehat{p}}_{\mathit{iT}}$. Such a naïve estimate has the advantage that it maintains the variance inflation of $\mathrm{E}\left(v_i^2\right)$ in comparison with $v_i^2$ (using $v_i^2$ from Equation (3)). Whenever ${\widehat{p}}_{\mathit{ij}}$ is used in Equation (7), $n_{\mathit{ij}}+1$ replaces $n_{\mathit{ij}}$.

We use these results in Sections 5 and 6.

5. APPROXIMATIONS TO THE DISTRIBUTIONS OF $Q_F$ AND $Q_{\mathit{IV}}$

Our new interval estimators of ${\tau }^2$ (Section 6.2) involve the cumulative distribution function of $Q_F$. For LOR, $Q_F$ is a quadratic form in asymptotically normal variables. The Farebrother algorithm, ¹² applicable for quadratic forms in normal variables, provides a satisfactory approximation to the cdf for larger sample sizes ($n\ge 100$), though it may not behave well for small $n$. ⁶ To apply it, we plug in estimated variances. As in Reference [13], we denote the Farebrother approximation for $Q$ with effective‐sample‐size weights by F SSW. We further distinguish between a “model‐based” version and a “naïve” version of F SSW, according to whether we use ${\overline{p}}_{\mathit{iT}}$ or ${\widehat{p}}_{\mathit{iT}}$ in Equation (7).

The null distribution of $Q_{\mathit{IV}}$ is usually approximated by the chi‐square distribution with $K-1$ degrees of freedom. For LOR, as also for both MD and SMD, this approximation is not accurate for small sample sizes. ¹⁴ For LOR, Kulinskaya and Dollinger ¹⁰ provided an improved approximation to the null distribution of $Q_{\mathit{IV}}$ based on fitting two moments of the gamma distribution; we denote this approximation by KD. For comparison, we study point and interval estimators of ${\tau }^2$ based on the chi‐square and KD approximations.

6. POINT AND INTERVAL ESTIMATORS OF ${\tau }^2$ FOR LOR

6.1. Point estimators

The unconditional variance of $\widehat{\theta }$ in the customary fixed‐intercept model, Equation (7), can be written as a sum of two terms,

$$M_{2i}=\mathrm{E}\left(\mathrm{Var}\left({\widehat{\theta }}_i|p_{\mathit{ij}},n_{\mathit{ij}}\right)\right)+{\tau }^2\left(1+\frac{1}{2n_{\mathit{iT}}}\left({\left[{\stackrel{⌣}{p}}_{\mathit{iT}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iT}}\right)\right]}^{-1}-2\right)\right),$$ (8)

where $\mathrm{E}\left(\mathrm{Var}\left({\widehat{\theta }}_i|p_{\mathit{ij}},n_{\mathit{ij}}\right)\right)={\left[n_{\mathit{iT}}{\stackrel{⌣}{p}}_{\mathit{iT}}\left(1-{\stackrel{⌣}{p}}_{\mathit{iT}}\right)\right]}^{-1}+{\left[n_{\mathit{iC}}{p}_{\mathit{iC}}\left(1-p_{\mathit{iC}}\right)\right]}^{-1}$, $p_{\mathit{iC}}=\text{expit}\left({\alpha }_i\right)$, and ${\stackrel{⌣}{p}}_{\mathit{iT}}=\text{expit}\left({\alpha }_i+\theta \right)$. Rearranging the terms in Equation (6) with $\mathrm{E}\left({\mathrm{\Theta }}_i^2\right)=M_{2i}$ and replacing $p_{\mathit{iC}}$ and ${\stackrel{⌣}{p}}_{\mathit{iT}}$ with ${\widehat{p}}_{\mathit{iC}}$ and ${\widehat{p}}_{\mathit{iT}}$ (or ${\overline{p}}_{\mathit{iT}}$) give the naïve (or model‐based) moment estimator of ${\tau }^2$.

$${\widehat{\tau }}_U^2=\frac{Q/W-\sum q_i\left(1-q_i\right){\widehat{v}}_i^2}{\sum q_i\left(1-q_i\right)C_i},$$ (9)

where ${\widehat{v}}_i^2={\left[n_{\mathit{iT}}{\widehat{p}}_{\mathit{iT}}\left(1-{\widehat{p}}_{\mathit{iT}}\right)\right]}^{-1}+{\left[n_{\mathit{iC}}{\widehat{p}}_{\mathit{iC}}\left(1-{\widehat{p}}_{\mathit{iC}}\right)\right]}^{-1}$ and $C_i=1+\frac{1}{2n_{\mathit{iT}}}\left({\left[{\widehat{p}}_{\mathit{iT}}\left(1-{\widehat{p}}_{\mathit{iT}}\right)\right]}^{-1}-2\right)$. DerSimonian and Kacker ² obtain a similar result; they use the conditional estimate, ${\widehat{v}}_i^2$, instead of the unconditional estimate, $\widehat{\mathrm{E}\left(v_i^2\right)}$, obtaining

$${\widehat{\tau }}_M^2=\frac{Q/W-\sum q_i\left(1-q_i\right){\widehat{v}}_i^2}{\sum q_i\left(1-q_i\right)}.$$ (10)

We study both estimators with effective‐sample‐size weights. With the conditional estimated variances in Equation (10), we denote the estimator by SSC (for “Sample Sizes Conditional”); with the unconditional estimated variances, as in Equation (9), it is SSU (for “Sample Sizes Unconditional”). These estimators differ by a term of order $O\left(1/n_i\right)$ and will be very similar for large sample sizes.

The estimators ${\widehat{\tau }}_U^2$ and ${\widehat{\tau }}_M^2$ arose from setting the observed value of $Q$ equal to its expected value and solving for ${\tau }^2$. Instead of the expected value, one could use the median of the distribution of $Q$ given ${\tau }^2$. ¹⁵ , ¹⁶ , ¹⁷ If the true (or approximate) cumulative distribution function is $F\left(\cdot |{\tau }^2\right)$, a point estimator of ${\tau }^2$ can be found as

$${\widehat{\tau }}_{\mathit{med}}^2=\max \left(0,\left\{{\tau }^2:F\left(Q|{\tau }^2\right)=0.5\right\}\right).$$ (11)

In the Farebrother approximation to the distribution of $Q_F$ (Section 5), one can use either the conditional estimated variances or the unconditional estimated variances. We denote the resulting estimators by SMC and SMU (“Sample sizes Median (Un)Conditional”), respectively.

Choosing between the “model‐based” and the “naïve” estimate of $p_{\mathit{iT}}$ in $M_{2i}$ (8) yields “model‐based” and “naïve” versions of SSU and SMU: SSU model or SSU naïve and SMU model or SMU naïve.

The SSC and SMC estimators can be obtained from the procedure rma in metafor ⁷ by choosing as the method “GENQ” or “GENQM,” respectively, and specifying $\mathit{\text{nbar}}$ weights.

For comparison, our simulations (Section 7) include four estimators that use inverse‐variance weights: DerSimonian‐Laird ⁹ (DL), restricted maximum‐likelihood (REML), Mandel‐Paule ¹⁸ (MP), and an estimator (KD) based on the work of Kulinskaya and Dollinger ¹⁰ and discussed by Bakbergenuly et al. ⁴ KD uses an improved non‐null first moment of $Q$ and has better performance than most other estimators of ${\tau }^2$. In their review of methods for estimating the between‐study variance, Veroniki et al. ¹⁹ explain that DL is (by default) the most widely used, and they conclude that both REML and MP are better.

A perennial question involves whether analysts should add $1/2$ to each of $X_{\mathit{iT}},X_{\mathit{iC}},n_{\mathit{iT}}-X_{\mathit{iT}},n_{\mathit{iC}}-X_{\mathit{iC}}$ only when one of them is zero, or in all studies. This is equivalent to using $\tilde{p}$ (whenever possible) or $\widehat{p}$ (always), respectively, when estimating ${\theta }_i$ and $v_i^2$. To obtain evidence on this issue, we included the corresponding two versions, “only” and “always,” of DL, REML, MP, SSC, and SMC (we follow the prevalent practice of omitting “double‐zero” studies, in which two of those cell counts are zero).

Table 1 gives the full list of point estimators.

TABLE 1.

Point and interval estimators of ${\tau }^2$ in the simulations.

Estimator	Description	Add $1/2$
		“Only”	“Always”
Point estimators
DL	DerSimonian‐Laird, 9 a moment estimator based on ${\chi }^2$ approximation to distribution of $Q_{\mathit{IV}}$	x	x
REML	Restricted Maximum Likelihood	x	x
MP	Mandel‐Paule, 18 a moment estimator based on ${\chi }^2$ approximation to distribution of $Q_{\mathit{IV}}$	x	x
KD	Kulinskaya‐Dollinger, 10 Bakbergenuly et al. 4 a moment estimator based on improved Gamma approximation to $Q_{\mathit{IV}}$		x
SSC	Equation (10), effective‐sample‐size weights, conditional variance (3) of $\widehat{\theta }$	x	x
SSU model	Equation (9), effective‐sample‐size weights, model‐based estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x
SSU naïve	Equation (9), effective‐sample‐size weights, naïve estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x
SMC	Median‐unbiased, Equation (11), effective‐sample‐size weights, conditional variance (3) of $\widehat{\theta }$	x	x
SMU model	Median‐unbiased, Equation (11), effective‐sample‐size weights, model‐based estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x
SMU naïve	Median‐unbiased, Equation (11), effective‐sample‐size weights, naïve estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x
Confidence intervals
QP	Q‐profile, Viechtbauer, 20 Appendix S1.2	x	x
PL	Profile Likelihood, Hardy & Thompson, 21 Appendix S1.1	x	x
KD	Kulinskaya–Dollinger, 10 Bakbergenuly et al., 4 Appendix S1.3, profiled improved Gamma approximation to distribution of $Q_{\mathit{IV}}$		x
FPC	Farebrother Profile, that is, profiled Farebrother approximation to distribution of $Q_F$, effective‐sample‐size weights, conditional variance (3) of $\widehat{\theta }$	x	x
FPU model	Farebrother Profile, effective‐sample‐size weights, model‐based estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x
FPU naïve	Farebrother Profile, effective‐sample‐size weights, naïve estimate of $p_{\mathit{iT}}$ in unconditional variance (7) of $\widehat{\theta }$		x

6.2. Interval estimators

Straightforward use of the cumulative distribution function $F\left(\cdot |{\tau }^2\right)$ also yields a $100\left(1-\alpha \right)\%$ confidence interval for ${\tau }^2$:

$$\left\{{\tau }^2\ge 0:F\left(Q|{\tau }^2\right)\in \left[\alpha /2,1-\alpha /2\right]\right\}.$$

We use both the conditional estimated variances and the unconditional estimated variances in the Farebrother approximation to $Q_F$ (Section 5); we refer to the resulting profile estimators as FPC and FPU (“Farebrother Profile (Un)Conditional”) intervals. Jackson ²² introduced a similar approach using conditional variances. The FPC interval can be obtained from the confint procedure in metafor ⁷ for “GENQ” or “GENQM” objects that used $\mathit{\text{nbar}}$ weights. For the FPU intervals, we further distinguish between a “model‐based” version and a “naïve” version, according to whether we use ${\overline{p}}_{\mathit{iT}}$ or ${\widehat{p}}_{\mathit{iT}}$ in $M_{2i}$ (8). Kulinskaya and Hoaglin ¹³ give the higher unconditional moments of $\widehat{\theta }$ required for the FPU intervals.

Our simulations (Section 7) also include the profile‐likelihood interval, ²¹ the Q‐profile interval, ²⁰ and the KD interval. ¹⁰ Table 1 gives the full list, and Section S1 in the Supporting Information gives further details.

7. SIMULATION DESIGN

Our simulation design followed that described in Bakbergenuly et al. ⁴ Briefly, we varied five parameters: the overall true effect ($\theta$), the between‐studies variance (${\tau }^2$), the number of studies $\left(K\right)$, the studies' total sample size ($n$ or $\overline{n}$, the average sample size), and the probability in the control arm ($p_{\mathit{iC}}$). We kept the proportion of observations in the control arm ($f$) at $1/2$.

The values of $\theta$ (0, 0.1, 0.5, 1, 1.5, and 2) aim to represent the range containing most values encountered in practice. LOR is a symmetric effect measure, so the sign of $\theta$ is not relevant.

The values of ${\tau }^2$ (0(0.1)1) systematically cover a reasonable range.

The numbers of studies ($K=5$, 10, and 30) reflect the sizes of many meta‐analyses and have yielded valuable insights in previous work.

In practice, many studies' total sample sizes fall in the ranges covered by our choices $(n=20$, 40, 100, and 250 when all studies have the same $n$, and $\overline{n}=30$, 60, 100, and 160 when sample sizes vary among studies). The choices of sample sizes corresponding to $\overline{n}$ follow a suggestion of Sánchez‐Meca and Marín‐Martínez, ²³ who constructed the studies' sample sizes to have skewness 1.464, which they regarded as typical in behavioral and health sciences. For $K=5,$ Table 2 lists the sets of five sample sizes. The simulations for $K=10$ and $K=30$ used each set of unequal sample sizes twice and six times, respectively.

TABLE 2.

Values of parameters in the simulations.

Parameter	Equal study sizes	Unequal study sizes
$K$ (number of studies)	5, 10, 30
$n$ or $\overline{n}$ (average (individual) study size—total of the two arms) For $K=10$ and $K=30$, the same set of unequal study sizes is used twice or six times, respectively.	20, 40, 100, 250	30 (12,16,18,20,84), 60 (24,32,36,40,168), 100 (64,72,76,80,208), 160 (124,132,136,140,268)
$f$ (proportion of observations in the control arm)	1/2
$p_{\mathit{iC}}$ (probability in the control arm)	0.1, 0.2, 0.5
$\theta$ (true value of LOR)	0, 0.1, 0.5, 1, 1.5, 2
${\tau }^2$ (variance of random effects)	0 (0.1)1

The values of $p_{\mathit{iC}}$, 0.1, 0.2, and 0.5, provide a typical range of small to medium risks.

The values of $p_{\mathit{iC}}$ and the true effect ${\theta }_i$ defined the probabilities $p_{\mathit{iT}}$, and the counts $X_{\mathit{iC}}$ and $X_{\mathit{iT}}$ were generated from the respective binomial distributions. We used a total of $\mathrm{10,000}$ repetitions for each combination of parameters (which we also call a situation). We discarded “double‐zero” and “double‐n” studies and reduced the observed value of $K$ accordingly. Next, we discarded repetitions with $K<3$ and used the observed number of repetitions for analysis.

The simulations used R statistical software. ²⁴ We used metafor for all methods of interest that it implemented. Table 1 gives the full list of point and interval estimators of ${\tau }^2$. The user‐friendly R programs implementing our methods are available on OSF. ²⁵

8. SIMULATION RESULTS

Our eprint ²⁶ reports the full simulation results. Here we describe the most important findings. We also refer to Figures S1 through S10 in the Supporting Information.

8.1. Bias in point estimation of ${\tau }^2$ for LOR

The relation of each estimator's bias to ${\tau }^2$ is roughly linear, with variation in intercept and, especially, in slope among the situations in the simulation. The slope varies most with $n$ and $K$ and to a lesser extent with $p_{\mathit{iC}}$, $\theta$, and whether sample sizes are equal or unequal. In one of the more extreme examples, when $p_{\mathit{iC}}=.1$, $\theta =0$, and $K=5$, the bias of SMC “only” when $n=20$ is $+0.29$ at ${\tau }^2=0$ and $-0.22$ at ${\tau }^2=1$, whereas when $n=250$, it is $+0.09$ at ${\tau }^2=0$ and $+0.35$ at ${\tau }^2=1$. The estimators' traces combine to form various patterns (Figure 1).

FIGURE 1.

Bias of estimators of between‐study variance of LOR (the “only” versions of DL, REML, MP, and SMC; SSC “always”; KD; and the model versions of SMU and SSU) versus ${\tau }^2$, for equal sample sizes $n=20,40,100$, and $250$, $p_{\mathit{iC}}=0.1$, $\theta =0$, and $f=0.5$. [Colour figure can be viewed at wileyonlinelibrary.com]

For small sample sizes, all estimators of ${\tau }^2$ have considerable bias, positive at ${\tau }^2=0$ and negative for larger ${\tau }^2$. The traces are roughly parallel. The pattern becomes tighter as $K$ increases. When $p_{\mathit{iC}}=0.1$, a few estimators have bias that is almost constant in ${\tau }^2$ when $n\ge 100.$ When $n=250$, the traces form a fan‐shaped pattern, in which the bias ranges from 0.02 to 0.07 when ${\tau }^2=0$ and from $-0.2$ to $+0.35$ when ${\tau }^2=1$. The flattening and fan‐shaped pattern occur at somewhat smaller $n$ as $\theta$ increases. The fan‐shaped pattern appears by $n=100$ when $p_{\mathit{iC}}=0.2$ and by $n=40$ when $p_{\mathit{iC}}=.5$. For larger sample sizes, some estimators have negative trends (or no trend), and other have positive trends.

The best estimators, almost unbiased when $n\ge 250$ and $p_{\mathit{iC}}=0.1$, are MP “only,” KD, SSU model, and SSC “always.” The same estimators are recommended for larger values of $p_{\mathit{iC}}$, where they generally become less biased earlier. When $p_{\mathit{iC}}=0.2$ or $p_{\mathit{iC}}=0.5$, various transitions occur at smaller sample sizes (Figures S1 and S2).

For comparison, our simulations included the popular point estimators DL and REML. We focus on their “only” version. The trace for the bias of DL “only” is in the middle of the traces for the collection of estimators, or slightly lower. When $n=250$ or $\overline{n}=160$, its negative bias, increasing in size as ${\tau }^2$ increases, stands out (usually alone).

The trace for REML “only” lies very close to that for DL, except that when $n=250$ (or, less often, $\overline{n}=160$), it is close to zero instead of trending negative.

To summarize, we recommend MP “only,” KD, SSU model, and SSC “always”; and we emphatically recommend against using DL and REML.

8.2. Median bias of estimators of ${\tau }^2$ for LOR

We define median bias as $P\left({\widehat{\tau }}^2\ge {\tau }^2\right)-P\left({\widehat{\tau }}^2\le {\tau }^2\right)$. For a median, the median bias is zero.

For $0.1\le {\tau }^2\le 1$, all estimators have negative bias for small to medium sample sizes, $n\le 40$ (Figure 2). Interestingly, the median bias of most estimators is almost constant across the range of nonzero ${\tau }^2$ values. That level, however, varies with $n$ and $K$. As $n$ increases, the bias becomes less negative (but not small), but increasing $K$ tends to make it more negative. DL “only,” REML “only,” and MP “only” depart from the almost constant pattern. As $K$ increases, the trace of DL “only” declines steeply (e.g., from $-0.20$ at ${\tau }^2=0.1$ to $-0.68$ at ${\tau }^2=1$ when $p_{\mathit{iC}}=0.2$, $\theta =0$, $n=100$, and $K=30$, Figure S3), and the trace of REML “only” declines less steeply (and only when $p_{\mathit{iC}}=0.5$, Figure S4). The trace of MP “only” tends to have a positive slope when $n=40$ and $n=100$ and $\theta \ge 1$ and $p_{\mathit{iC}}=0.1$ and a negative slope when $n=20$ and $n=40$ and $p_{\mathit{iC}}=0.5$.

FIGURE 2.

Median bias of estimators of between‐study variance of LOR (the “only” versions of DL, REML, MP, and SSC; SMC “always”; KD; and the model versions of SMU and SSU) versus ${\tau }^2,$ for equal sample sizes $n=20,40,100$ and $250$, $p_{\mathit{iC}}=0.1$, $\theta =0$ and $f=0.5$. [Colour figure can be viewed at wileyonlinelibrary.com]

When $n=20$ or $\overline{n}=30$ and $p_{\mathit{iC}}=0.1$, none of the estimators are satisfactory. KD comes closest, with median bias around $-0.25$ at best. For $p_{\mathit{iC}}=0.2$ and $p_{\mathit{iC}}=0.5$, SMC “always” and SMC “only” are better; in a few situations one or both have small positive or negative bias.

The majority of the standard estimators have negative bias for larger values of $n$. This means that their median values of ${\widehat{\tau }}^2$ are too low. When $n\ge 100$ for $p_{\mathit{iC}}=0.1$ and $n\ge 40$ for $p_{\mathit{iC}}\ge 0.2$, the new median‐unbiased estimators perform well. We recommend SMC “always” and SMU model. Both consistently result in almost median‐unbiased estimation across the range of $p_{\mathit{iC}}$ values.

8.3. Coverage of interval estimators of ${\tau }^2$ for LOR

The results of our simulations show a few general patterns, but specific choices of interval estimators depend on values of parameters or combinations of parameters. In particular, we often separate $K=30$ from $K=5$ and $K=10$.

At ${\tau }^2=0$ coverage is always too high (essentially 1.00). For small $n$, it is roughly flat when ${\tau }^2\ge 0.1$ and $K=5$ or $K=10$.

When $p_{\mathit{iC}}=0.1$ and $\theta =0$, coverage of most estimators remains too high when $n\le 100$ (Figure 3). KD and FPU model are close to 0.95 when $n=100$. When $K=30$ and $n=20$ or $n=40$, all estimators except KD break down: most have coverage $<0.90$ at ${\tau }^2=0.1$, and their coverage declines steeply as ${\tau }^2$ increases. Some form of breakdown persists for all $\theta$. As $\theta$ increases, and $n=20$ or $n=40$, coverage at $0.1\le {\tau }^2\le 1$ moves toward 0.95. KD and FPC model (for $n\ge 40$) seem the best choices.

FIGURE 3.

Coverage of 95% confidence intervals for between‐study variance of LOR (the “only” versions of PL, QP and FPC; KD; and the model version of FPU) versus ${\tau }^2$, for equal sample sizes $n=20,40,100$ and $250$, $p_{\mathit{iC}}=0.1$, $\theta =0$ _, and $f=0.5$. [Colour figure can be viewed at wileyonlinelibrary.com]

When $p_{\mathit{iC}}=0.2$ or $p_{\mathit{iC}}=0.5$ (Figures S5 and S6), the picture is simpler. Coverage when $n=20$ or $n=40$ becomes closer to (or equals) 0.95 as $p_{\mathit{iC}}$ increases, except for the “always” versions of PL, QP, and FPC when $p_{\mathit{iC}}=0.2$. For $K=30$, KD seems the best single choice.

Coverage of PL is usually too high, and in some situations it seems trapped at 1.00.

8.4. Left and right coverage error

It is often informative to approach estimation of coverage by separating its complement, the coverage error or “miscoverage,” into two parts, corresponding to whether the value of the parameter is to the left of the lower confidence limit or to the right of the upper confidence limit. Efron and Tibshirani in Reference [27], section 13.5 denote these by “miss left” and “miss right.” A CI that had a small miss‐left percentage would overcover on the left; if it had a large percentage, it would undercover. The confidence intervals that we studied aim to have miscoverage equal to 2.5% on each side (Section 6.2), but an approximation for the distribution of $Q$ may not provide the desired balance, even if the overall coverage is close to 95%. Thus, our simulations included the miss‐left and miss‐right percentages.

In general, the miss‐left percentages are typically lower than 2.5% for small sample sizes and/or control‐arm probabilities, but they improve for $n\ge 100$ and for larger $p_{\mathit{iC}}$ (Figures 4, S7, S8). A low miss‐left percentage means that the confidence interval includes an excess of low ${\tau }^2$ values.

FIGURE 4.

Miss‐left probability of PL, QP, KD, FPC, and FPU 95% confidence intervals for between‐study variance of LOR versus ${\tau }^2$, for equal sample sizes $n=20,40,100$ and $250$, $p_{\mathit{iC}}=0.1$, $\theta =0$, and $f=0.5$. Solid lines: the “only” versions of PL, QP and FPC; KD; and the model version of FPU. Dashed lines: the “always” version of FPC and the naïve version of FPU. [Colour figure can be viewed at wileyonlinelibrary.com]

The miss‐left percentages of KD, FPU naïve, and FPC “always” are closer to 2.5% from $n=100$, and miss‐left percentages of QP are typically lower than nominal when $p_{\mathit{iC}}=0.1$ but improve for larger $p_{\mathit{iC}}$.

The only exceptions to typically lower than nominal miss‐left percentages are the FPC “only” and FPU model intervals, whose percentages for $n\ge 40$ are often higher than nominal for $K=30$ and sometimes when $K=10$. KD occasionally has high percentages for high values of $\theta$ when $K=30$. PL miss‐left percentages are especially low for all sample sizes.

The miss‐right percentages are often higher than nominal, but they improve for larger $n$ and $p_{\mathit{iC}}$ (Figures 5, S9, S10). $K=30$ is especially challenging.

FIGURE 5.

Miss‐right probability of PL, QP, KD, FPC, and FPU 95% confidence intervals for between‐study variance of LOR versus ${\tau }^2$, for equal sample sizes $n=20,40,100$ and $250$, $p_{\mathit{iC}}=0.1$, $\theta =0$, and $f=0.5$. Solid lines: the “only” versions of PL, QP and FPC; KD; and the model version of FPU. Dashed lines: the “always” version of FPC and the naïve version of FPU. [Colour figure can be viewed at wileyonlinelibrary.com]

For $p_{\mathit{iC}}=0.1$, the miss‐right percentages of KD, QP “always,” and FPC “always” are closest to 2.5% when $n=20$ and $K=5$, but increase for larger $n$. When $K=5$ or $10$, the percentages of FPC “only” and QP “only” are reasonably close to 2.5% when $n\ge 40$. The percentages of KD are consistently close to 2.5% for $K=10$. For $K=30$, the range of results is larger. The percentages of KD, QP “only,” and FPC “always” are close to 2.5% when $n\ge 100$. The percentages for all other estimators are typically higher, with the exception of FPU model, which has lower than nominal percentages. PL produces erratic percentages, from 0 to above 10%. For larger values of $p_{\mathit{iC}}$, the miss‐right percentages of all estimators, with the exception of PL, improve earlier, so that those of KD, QP “only,” and FPC “always” are all close to 2.5% when $n=40$ and $p_{\mathit{iC}}=0.5$, even when $K=30$.

Lack of balance in the miss‐left and miss‐right percentages for small sample sizes and/or control‐arm probabilities agrees with our findings ¹³ that the chi‐square approximation for $Q_{\mathit{IV}}$ and the Farebrother approximation for $Q_F$ are accurate only for larger sample sizes.

9. EXAMPLE: SMOKING CESSATION

Stead et al. ²⁸ conducted a systematic review of clinical trials on the use of physician advice for smoking cessation. We use the data from the subgroup of interventions in which the treatment involved only one visit (Comparison 3.1.4, p. 54). The first version of the report was published in 2001. In an update, published in 2004, 17 studies included this comparison. The 2013 update included one more study, by Unrod (2007). For each study, Table S1 gives the number of subjects in the treatment and control arms and the number who were nonsmokers at the longest follow‐up time reported (either 6 months or 12 months). The definition of “nonsmoker” varied among the studies. Some studies required sustained abstinence, and others only asked about smoking status at that time. Stead et al. ²⁸ analyzed relative risk. Kulinskaya and Hoaglin ⁶ showed that both OR and RR are reasonable effect measures for these data. Here we consider estimation of ${\tau }^2$ in two meta‐analyses of LOR.

The studies were mostly balanced, though two studies had substantially more subjects in the treatment arm. Sample sizes varied from 182 to 3128, with an average of 836 patients per study. The mean probabilities of smoking cessation in both arms were rather low, at $0.058$ in the treatment arm and $0.043$ in the control arm.

The standard IV‐based meta‐analysis of LOR for the original 17 studies gives $\widehat{\theta }=0.4774$ with standard error $0.1148$ and $p<0.0001$ for the intervention effect (${\widehat{\tau }}_{\mathit{MP}}^2=0.0754$). The fixed‐weights estimate of $\theta$ is higher, at $0.7127$. In testing for heterogeneity, $Q_{\mathit{IV}}=24.84$, and the chi‐square approximation on 16 df provides a p‐value of $.079$ ($I^2=38.18\%$). The p‐values for the KD and F SSW naïve methods (which Kulinskaya and Hoaglin ⁶ recommend) are very close, at 0.035 and 0.038, respectively.

Table 3 shows striking differences: the SSC and SSU estimates of ${\tau }^2$ are more than twice as large as the standard IV estimates. This agrees with our simulations, which showed large positive biases of SSC and SSU for low values of ${\tau }^2$. All confidence intervals have zero as the lower limit (Table 4). This result contradicts the significant p‐values from the KD and F SSW naïve tests, but it can be explained by the inflated confidence level of all six confidence intervals near zero. Adding $1/2$ to the numbers of events somewhat reduces the estimated values and the upper confidence limits. This would somewhat reduce the positive biases and the inflation in coverage. Because of the large sample sizes, there are only minor differences between conditional and unconditional model‐based or naïve estimators of ${\tau }^2$ and confidence intervals.

TABLE 3.

Estimated values of ${\tau }^2$ in the meta‐analysis of LOR for the data of Stead et al. on the use of physician advice for smoking cessation.

Example	Add 1/2	DL	REML	MP	KD	SSC	SSU model	SSU naïve	SMC	SMU model	SMU naïve
Stead et al. (17 studies)	Always	0.0627	0.0696	0.0701	0.0887	0.1715	0.1725	0.1680	0.2014	0.2043	0.1972
	Only	0.0675	0.0769	0.0754		0.1776			0.2098
Stead et al. (18 studies)	Always	0.0514	0.0537	0.0576	0.0740	0.1634	0.1642	0.1601	0.1913	0.1939	0.1873
	Only	0.0555	0.0597	0.0623		0.1697			0.1997

TABLE 4.

95% confidence intervals for the between‐study variance in the meta‐analysis of LOR for the data of Stead et al. on the use of physician advice for smoking cessation.

Example	Add 1/2	QP	PL	KD	FPC	FPU model	FPU naïve
Stead et al. (17 studies)	Always	[0, 0.3745]	[0, 0.3445]	[0, 0.3910]	[0, 0.7583]	[0, 0.7554]	[0, 0.7389]
	Only	[0, 0.3957]	[0, 0.3713]		[0, 0.8075]
Stead et al. (18 studies)	Always	[0, 0.3267]	[0, 0.2915]	[0, 0.3419]	[0, 0.6992]	[0, 0.6969]	[0, 0.6820]
	Only	[0, 0.3455]	[0, 0.3139]		[0, 0.7458]

Addition of the 18th study somewhat increased the p‐value for the standard $Q$ test, to $0.107$, and the KD p‐value to $0.052$, but hardly affected the p‐values of the SSW‐based methods. The recommended F SSW naïve test rejects homogeneity at the 5% significance level, with $p=0.033$. The estimated ${\tau }^2$ values are somewhat smaller, and all confidence intervals still start at zero.

To better understand the properties of the estimation methods for very small values of $p_{\mathit{iC}}$ in this example, we performed additional simulations with 1000 repetitions in the relevant intervals of values, ${\tau }^2\in \left[0,0.30\right]$ and $\theta \in \left[0.4,0.8\right]$, keeping the sample sizes as in the example and using the probabilities $p_{\mathit{iC}}=X_{\mathit{iC}}/n_{\mathit{iC}}$. Figure 6 shows the results. In agreement with our main series of simulations, KD, MP “only”, SSC “always,” and SSU model are the least biased estimators of ${\tau }^2$, whereas SMC “only” is the least median‐biased. Table S2 gives summary statistics for these five estimators when ${\tau }^2=0.05$ and $0.2$, and $\theta =0.5$ and $0.7$. Additionally, Table S2 provides summary statistics when $\theta =0.5$ and ${\tau }^2=0.2$, but the control arm probabilities are less extreme, at $p_{\mathit{iC}}=\left(X_{\mathit{iC}}/n_{\mathit{iC}}\right)+.1$.

FIGURE 6.

Bias, median bias, and coverage at 95% nominal confidence level of point and interval estimators of ${\tau }^2$ for the example of meta‐analysis by Stead et al. Solid lines: the “only” versions of DL, REML, MP, SSC, and SMC; KD; the model versions of SSU and SMU point estimators; the “only” versions of PL, QP, and FPC; FPU model, and KD intervals. Dashed lines: the “always” versions of SSC and SMC point estimators. [Colour figure can be viewed at wileyonlinelibrary.com]

Both lower and upper quartiles for the first four estimators of ${\tau }^2$ are comparable, though the upper quartile of SMC “only” is somewhat higher. However, the three effective‐sample‐size estimators have a much longer right tail than KD and MP “only.” This shape explains their much higher mean values. Comparing the medians, SMC “only” is almost median‐unbiased in all five scenarios, whereas SSC “always” and SSU model have the lowest medians, followed by MP “only” and then KD. This pattern is especially noticeable when the control arm probabilities are low. The lower the median, the more often these estimators would underestimate true heterogeneity. The differences between distributions decrease quite considerably for larger control arm probabilities, so the KD and MP “only” densities practically coincide, as do the densities of SSC “always” and SSU model (Figure 7). Overall, KD is the best mean‐unbiased estimator, and SMC “only” is the best median‐unbiased estimator. Which one to prefer depends very much on the context of the research. All confidence intervals provide reasonable coverage, though KD is sometimes too liberal, and PL too conservative (Figure 6).

FIGURE 7.

Density plots of the point estimators of ${\tau }^2$ for the example of meta‐analysis by Stead et al.: the “only” versions of MP and SMC; the “always” version of SSC; the model version of SMU; and KD. [Colour figure can be viewed at wileyonlinelibrary.com]

10. DISCUSSION

Estimation of heterogeneity variance ${\tau }^2$ is an important part of any meta‐analysis. Apart from its importance per se, it also affects the value of an estimated pooled effect and its estimated variability. In this paper, we studied estimation of ${\tau }^2$ based on $Q_F$, a generalized $Q$ statistic with fixed effective‐sample‐size weights, and compared four proposed new point estimators of ${\tau }^2$ (SSC, SMC, SSU, and SMU) and two new interval estimators (FPC and FPU) with traditional estimators based on the $Q$ statistic with inverse‐variance weights, $Q_{\mathit{IV}}$.

The new point estimators based on the expected value of $Q_F$ involve estimates (${\widehat{v}}_i$) of the variances of the ${\widehat{\theta }}_i$. We obtained SSC from the conditional variances and SSU from the unconditional variances of the ${\widehat{\theta }}_i$. Similarly, our novel point estimators based on the median of the Farebrother approximation to the distribution of $Q_F$ are SMC and SMU, and our profile interval estimators are FPC and FPU.

For each unconditional estimator, we investigated two approaches to estimation of $p_{\mathit{iT}}$ (which is used in the calculation of the second and fourth central moments of ${\widehat{\theta }}_i$), “naïve” estimation of $p_{\mathit{iT}}$ from $X_{\mathit{iT}}$ and $n_{\mathit{iT}}$ and “model‐based” estimation using the fixed‐effects meta‐analysis estimate of the overall effect to obtain ${\overline{p}}_{\mathit{iT}}$ from ${\widehat{p}}_{\mathit{iC}}$. Additionally, we considered two versions of the traditional and the unconditional estimators of ${\tau }^2$: adding $1/2$ to the four cell counts always, or only when one of those counts is zero. Thus, our simulation study examined a total of 15 point estimators of ${\tau }^2$ and 9 interval estimators (Table 1).

For mean bias, the best estimators are MP “only,” KD, SSU model, and SSC “always.” The DL and REML estimators have the worst bias. For median bias, which was not studied previously, an important finding is that the majority of the standard estimators have negative bias for larger values of $n$. This means that their median values of ${\widehat{\tau }}^2$ are too low. We recommend SMC “always” and SMU model. Both consistently result in almost median‐unbiased estimation across the range of $p_{\mathit{iC}}$ values for moderate to large $n$.

Coverage of the confidence intervals depends on $p_{\mathit{iC}}$, $n$, and $K$. There is also some dependence on $\theta$ and ${\tau }^2$. However, KD, FPC “always,” and FPU model are the best choices overall. PL is the worst choice. We also considered left and right coverage errors separately. Typically, the miss‐left rates are below, and the miss‐right rates are above, the nominal 2.5% level, especially so for small sample sizes and/or control arm probabilities. Arguably, this is not bad, as it would reduce the width of the confidence intervals. ²⁹ Lack of balance in the miss‐left and miss‐right rates agrees with our findings ¹³ that the chi‐square approximation for $Q_{\mathit{IV}}$ and the Farebrother approximation for $Q_F$ are accurate only for larger sample sizes.

Alongside the novel two‐stage estimators using sample‐size‐based weights, our simulation study involved a variety of established inverse‐variance‐based two‐stage estimators of heterogeneity. With a very few exceptions, such as MP and, especially, KD, their performance is not impressive. One‐stage meta‐analysis is often suggested as a better choice. However, we previously considered quality of one‐stage estimation in GLMM‐based binomial‐normal models. ¹¹ , ³⁰ and found it lacking. Also, it is useful to remember that GLMMs are still asymptotic methods that use normal likelihood.

Ideally, for meta‐analyses of log‐odds‐ratio, a single point estimator of ${\tau }^2$ would have acceptably low bias (and, perhaps, median bias) for a substantial region of $p_{\mathit{iC}}$, $\theta$, $n$, $K$, and ${\tau }^2$. Similarly, a single confidence interval would have close to nominal coverage of ${\tau }^2$. Our results show some progress toward those goals, by demonstrating advantages of new estimators in some situations and, importantly, by demonstrating that some popular estimators have unacceptable bias or coverage. The goals, however, require further research. In the interim, effective education can help users avoid methods that perform poorly.

AUTHOR CONTRIBUTIONS

Elena Kulinskaya: Conceptualization; funding acquisition; methodology; software; visualization; writing – original draft; writing – review and editing. David C. Hoaglin: Conceptualization; investigation; methodology; writing – original draft; writing – review and editing.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Supporting information

Data S1: Supporting Information.

Kulinskaya E, Hoaglin DC. Estimation of heterogeneity variance based on a generalized Q statistic in meta‐analysis of log‐odds‐ratio. Res Syn Meth. 2023;14(5):671‐688. doi: 10.1002/jrsm.1647

DATA AVAILABILITY STATEMENT

Our full simulation results are available as an arXiv e‐print (arXiv:2208.00707v1). The user‐friendly R program implementing all studied estimators of heterogeneity variance \$\tau^2\$ in meta‐analysis of log‐odds‐ratio with related confidence intervals is available at https://osf.io/5n3vd