Exact inference for the random-effect model for meta-analyses with rare events

Abstract

Meta-analysis allows for the aggregation of results from multiple studies to improve statistical inference for the parameter of interest. In recent years, random-effect meta-analysis has been employed to synthesize estimates of incidence rates of adverse events across heterogeneous clinical trials to evaluate treatment safety. However, the validity of existing approaches relies on asymptotic approximation as the number of studies becomes large. In practice, a limited number of trials are typically available for analysis. Moreover, adverse events are typically rare; thus, study-specific incidence rate estimates may be unstable or undefined. In this paper, we present a method for construction of an exact confidence interval for the location parameter of the beta-binomial model through inversion of exact tests. The coverage level of the proposed confidence interval is guaranteed to achieve at least the nominal level, regardless of the number of studies or the with-in study sample size, making it particularly applicable to the study of rare-event data.

1 |. INTRODUCTION

Interval estimation for a binomial probability is one of the most well-studied and methodologically important problems in statistics. For instance, premarketing and postmarketing evaluation of the safety of pharmaceutical products often lies in the estimation of the incidence rate of adverse events (AEs). When a single trial is available for assessment, inference for the incidence rate may be made with the simple binomial model or various asymptotic methods.^1–3 However, AEs are typically rare; therefore, single trials are not sufficiently powered to make sound inference. A recent example involves Mylotarg, a targeted treatment for acute myeloid leukemia (AML) that was voluntarily withdrawn from the US market after confirmatory trials identified safety concerns such as early mortality (EM). The incidence rate of EM was estimated to be 1∕14 = 7.1% with an exact binomial 95% confidence interval (CI) of [0.2%, 33.9%] based on data from a multicenter Phase II trial.⁴ As the results of additional studies were made available, a meta-analytic approach enabled the synthesis of information from individual trials to make more reliable inference on the rate of EM. The simplest of such methods is the fixed-effect (FE) meta-analysis that assumes that the underlying study-specific incidence rates are identical; thus, the data may be aggregated and similarly evaluated with the binomial model. For example, additional studies reported 2 out of 8, 0 out of 6, 0 out of 6, and 2 out of 7 patients experienced EM. Combining these results yields an estimated incidence rate of (1 + 2 + 0 + 0 + 2)∕(14 + 8 + 6 + 6 + 7) = 12.1% with a narrower 95% CI of [4.1%, 26.2%].

However, the assumption of a common incidence rate across studies is often too restrictive. Studies typically differ in various aspects such as patient inclusion criteria and treatment administration resulting in heterogeneity of study-specific effects. A more realistic model allows the study-specific rates to follow a random distribution. The objective of statistical inference is therefore to estimate the parameter (oftentimes the location parameter) characterizing such a distribution and is the aim of random-effect meta-analysis.⁵ Within current literature, the most popular random-effect model assumes that

$${\widehat{\mu }}_i\mid {\mu }_i~\text{N}\left({\mu }_i,{\sigma }_i^2\right),{\mu }_i~\text{N}\left({\mu }_0,{\tau }^2\right),i=1,\dots ,K$$ (1)

or equivalently that

$${\widehat{\mu }}_i~\text{N}\left({\mu }_0,{\sigma }_i^2+{\tau }_0^2\right),$$

where ${\widehat{\mu }}_i$ is the point estimator of the study-specific effect μ_i, μ₀ is the overall effect, ${\sigma }_i^2$ is the known within-study variance, ${\tau }_0^2$ is the unknown between-study variance, and K is the number of studies. When the outcome of interest is binary, μ_i and its empirical estimator are commonly taken as logit(π_i), where π_i is the underlying probability of observing the event in the ith study.

In their seminal paper, DerSimonian-Laird (DL) proposed to estimate μ₀ with the optimal linear combination of ${\widehat{\mu }}_i$ where the weights are constructed via a simple-moment estimator for the between-study variance ${\tau }_0^2$.⁶ The associated 100(1 − α)% CI for μ₀ is constructed based on asymptotic approximation requiring that K → ∞. Although the DL method is widely used in practice, the number of studies utilized in a meta-analysis is typically not large and frequently less than 7.⁷ In the Mylotarg example, for instance, the total number of available trials for analysis amounted to 7. As a result, ${\tau }_0^2$ may be imprecisely estimated and the finite sample performance of the DL CI can be poor. Performance issues are further exacerbated for binomial outcomes when study-specific sample sizes are small and/or p_i is close to zero (or one) as the normality assumption of (1) is violated.^8–12 Additionally, the logit proportion and its estimate of variance become undefined for zero-event studies leading to the use of arbitrary continuity corrections.¹³ These corrections may induce bias and do not alleviate the normality violation. Nonlinear random-effect models have thus been proposed such as the normal-binomial and beta-binomial (BB) models, but the performance of these methods still relies on a large number of individual studies.^10,14,15

To address these limitations, we propose a method for constructing an exact CI for the location parameter of a BB model based on the inversion of exact tests. The coverage level of the CI is guaranteed to achieve at least the nominal level up to Monte Carlo error with simulations confirming that the CI is not overly conservative. Moreover, our implementation of the proposed method allows for efficient estimation on a personal computer. In Section 2, we present our procedure for construction of the exact CI. The performance of our method is then evaluated based on extensive simulation studies and several meta-analyses aiming to assess the safety and efficacy of Mylotarg in Section 3. We conclude with a discussion in Section 4.

2 |. METHODS

2.1 |. Data structure

The observed data consist of ${\mathcal{D}}^0=\left\{Y_i\mid i=1,\dots ,K\right\}$ where Y_i is the number of events out of n_i subjects in the ith study for i = 1, …, K. To allow for between-study variation, we assume a BB random-effect model:

$$Y_i\mid {\pi }_i\stackrel{\text{ind}.}{~}\text{Bin}\left(n_i,{\pi }_i\right),{\pi }_i\stackrel{\text{ind}.}{~}\text{Beta}\left({\alpha }_0,{\beta }_0\right),i=1,\dots ,K,$$

or equivalently that

$$Y_i~\text{BB}\left(n_i,{\alpha }_0,{\beta }_0\right).$$

We require α₀, β₀ > 1 to ensure the unimodality of the random-effect distribution and hence the identifiability of the location parameter. This model implies

$$\text{E}\left(\frac{Y_i}{n_i}\right)=\frac{{\alpha }_0}{{\alpha }_0+{\beta }_0}={\mu }_0$$

and

$$\text{Var}\left(\frac{Y_i}{n_i}\right)=\frac{{\mu }_0\left(1-{\mu }_0\right)}{n_i}+\left(1-\frac{1}{n_i}\right){\nu }_0={\mu }_0\left(1-{\mu }_0\right)\left\{\frac{1}{n_i}+\left(1-\frac{1}{n_i}\right){\tau }_0\right\},$$

where ν₀ = μ₀(1 − μ₀)τ₀ is the variance of the random-effect distribution and τ₀ = (α₀+ β₀ + 1)⁻¹ represents the between-study heterogeneity. For clarity of presentation, we reparameterize the BB distribution with respect to μ₀ and ν₀ based on the following equivalences:

$${\alpha }_0={\mu }_0\left\{\frac{{\mu }_0\left(1-{\mu }_0\right)-{\nu }_0}{{\nu }_0}\right\}\text{and}{\beta }_0=\left(1-{\mu }_0\right)\left\{\frac{{\mu }_0\left(1-{\mu }_0\right)-{\nu }_0}{{\nu }_0}\right\}.$$ (2)

Remark 1.

While the standard BB model allows α₀, β₀ > 0, our additional requirement of α₀, β₀ > 1 implies that

$${\nu }_0\le {\mu }_0\left(1-{\mu }_0\right)\text{min}\left(\frac{{\mu }_0}{1+{\mu }_0},\frac{1-{\mu }_0}{2-{\mu }_0}\right)={\nu }_{\text{sup}}\left({\mu }_0\right),$$ (3)

which significantly reduces the parameter space of the standard BB model to

$$\left\{\left({\mu }_0,{\nu }_0\right)\mid {\nu }_0\le {\nu }_{\text{sup}}\left({\mu }_0\right)\right\}.$$

The reduced parameter space in the (μ₀, ν₀) plane is illustrated in Figure 1. The constraint on α₀ and β₀ serves a dual purpose. First, it guarantees that the distribution of π_i is unimodal and thus facilitates the interpretation of μ₀ = E(π_i). Second, it ensures that μ₀ is identifiable when all of the Y_i are close to zero. For instance, consider the toy example:

$$\left\{\left(Y_i,n_i\right)=(0,1000)\mid i=1,\dots ,6\right\}.$$

Intuitively, this presents strong evidence that all of the π_i are close to zero and so is μ₀. However, without our unimodality constraint, one cannot rule out the possibility that π_i ~ Beta(0.001, 0.003), which approximately assigns 3/4 and 1/4 probabilities to π_i = 0 and π_i = 1, respectively. Under this model, there is about 17% probability that all six π_i are very close to zero by chance and generate the observed data. The corresponding μ₀ = 1∕4 which clearly contradicts our intuition that μ₀ should be ≈ 0 as all six observed incidence rates are zero. This paradox is caused by the second mode of the Beta(0.001, 0.003) distribution and can be resolved by the proposed constraint on α₀ and β₀.

FIGURE 1.

The boundary of the parameter space in the (μ, v) plane for the standard beta-binomial model (solid) and our restricted model (dashed)

2.2 |. Proposed exact CI for the location parameter

With the aim of constructing an interval for μ₀ guaranteed to achieve at least the nominal level when K is small, we regard ν₀ as a nuisance parameter. To this end, we consider an unconditional test for testing

$$H_0:{\mu }_0=\mu$$

based on a test statistic $T\left(\mu ;{\mathcal{D}}^0\right)$ that is a function of both μ and the observed data ${\mathcal{D}}^0$. For now, we assume that larger values of $T\left(\mu ;{\mathcal{D}}^0\right)$ provide more evidence against H₀ and defer our discussion about the specific choice of $T\left(\mu ;{\mathcal{D}}^0\right)$ to the subsequent section. To control the type I error, the unconditional test eliminates ν₀ with the profile p value:

$$p\left(\mu ;{\mathcal{D}}^0\right)=\underset{\nu }{\text{sup}}P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,\nu }\right)\ge T\left(\mu ;D^0\right)\right\}=\underset{\nu }{\text{sup}}p\left(\mu ,\nu ;D^0\right),$$

where the probability is with respect to the random data ${\mathcal{D}}^{\mu ,\nu }$ following the mixed-effect BB model with parameters μ and ν. Thus, we would reject the null hypothesis, if $p\left(\mu ;{\mathcal{D}}^0\right)<\alpha$, where α stands for the type I error (rather than the parameter of the Beta distribution). The corresponding 100(1 − α)% CI is the set of μ for which $p\left(\mu ;{\mathcal{D}}^0\right)\ge \alpha$.

Assuming that the true values of μ and ν are covered by the finite intervals [μ_L, μ_U] and [ν_L, ν_U], respectively, an exact CI for μ₀ may then be constructed with the following generic two-step procedure:

1. For each value (μ, ν) in a dense H × J grid G = [μ_L, μ_U] × [ν_L, ν_U], compute $p\left(\mu ,\nu ;{\mathcal{D}}^0\right)$ based on the null distribution of the test statistic $T\left(\mu ;{\mathcal{D}}^{\mu ,\nu }\right)$.

2. Obtain the (1 − α) level CI for μ₀ as

   $$\left\{\underset{\mu \in \left[{\mu }_L,{\mu }_U\right]}{\text{inf}}{\Omega }_{1-\alpha }\left({\mathcal{D}}^0\right),\underset{\mu \in \left[{\mu }_L,{\mu }_U\right]}{\text{sup}}{\Omega }_{1-\alpha }\left({\mathcal{D}}^0\right)\right\},$$

   where ${\Omega }_{1-\alpha }\left({\mathcal{D}}^0\right)=\left\{(\mu ,\nu )\mid p\left(\mu ,\nu ;{\mathcal{D}}^0\right)\ge \alpha \right\}$ is the 95% confidence region for (μ₀, ν₀).

However, the exact distribution of $T\left(\mu ;{\mathcal{D}}^{\mu ,\nu }\right)$ which is needed to calculate $p\left(\mu ,\nu ;{\mathcal{D}}^0\right)$, is typically not tractable. We therefore propose to augment step (A) in the above procedure by approximating the null distribution through Monte Carlo simulation. Specifically, for each (μ, ν) ∈ G, we:

• (A.1)Generate ${\mathcal{D}}_m=\left\{Y_{im}\mid i=1,\dots ,K\right\}$ where Y_im ~ BB(n_i, μ, ν) for m = 1, …, M.
• (A.2)Compute $T_m=T\left(\mu ;{\mathcal{D}}_m\right)$ and approximate the distribution of $T\left(\mu ;{\mathcal{D}}^{\mu ,\nu }\right)$ with the empirical distribution of {T_m|m = 1, …, M}.
• (A.3)Compute the p value as $p\left(\mu ,\nu ;{\mathcal{D}}^0\right)=M^{-1}{\sum }_{m=1}^{M}I\left\{T_m\ge T\left(\mu ;{\mathcal{D}}^0\right)\right\}$.

While the above procedure guarantees that the resulting CI will achieve at least nominal coverage up to Monte Carlo error, the computational complexity is of O(MHJ) which may be restrictive when estimating the CI on a personal computer. In the next section, we describe our choice of test statistic as well as the details of the numerical computation that allows for the CI to be efficiently constructed.

2.3 |. Proposed test statistic and details of computation

The speed of our proposed procedure clearly rests on the choice of $T\left(\mu ;{\mathcal{D}}^0\right)$. Our goal in constructing the test statistic is thus to balance computational efficiency with performance. Here, we propose the use of a simple Wald statistic based on a DL-type estimator. In particular, we suggest to estimate μ₀ with

$$\widehat{\mu }=\frac{{\sum }_{i=1}^K\left(Y_i/n_i\right)w_i^{-1}}{{\sum }_{i=1}^K{w}_i^{-1}},$$

where

$$\widehat{\nu }=\text{max}\left\{0,\frac{{\sum }_{i=1}^K\left\{{\left(\frac{{\tilde{Y}}_i}{{\tilde{n}}_i}\right)}^2-\frac{{\widehat{\mu }}_{\text{int}}}{{\tilde{n}}_i}\right\}}{{\sum }_{i=1}^K\left(1-\frac{1}{{\tilde{n}}_i}\right)}-{\widehat{\mu }}_{\text{int}}^2\right\},{\widehat{\mu }}_{\text{int}}=\frac{{\sum }_{i=1}^K{\tilde{Y}}_i}{{\sum }_{i=1}^K{\tilde{n}}_i},$$

$w_i=\left\{{\widehat{\mu }}_{\text{int}}\left(1-{\widehat{\mu }}_{\text{int}}\right)\right\}/n_i+\left(1-1/n_i\right)\widehat{\nu }$, and

$$\left({\tilde{Y}}_i,{\tilde{n}}_i\right)=\{\begin{array}{l}\left(Y_i+1,n_i+2\right)\text{if}{Y}_i=0\text{or}{Y}_i=n_i\text{for}i=1,\dots ,K\hfill \\ \left(Y_i,n_i\right)\text{otherwise}\hfill \end{array}.$$

The continuity correction is employed to ensure that $\text{Var}\left(\frac{Y_i}{n_i}\mid {\pi }_i\right)>0$ and $\widehat{\nu }$ is well defined when Y_i = 0 or Y_i = n_i in all K studies. The test statistic is taken accordingly as

$$T\left(\mu ;{\mathcal{D}}^0\right)=\left(\sum _{i=1}^K{w}_i^{-1}\right){(\widehat{\mu }-\mu )}^2$$ (4)

and may be computed quickly.

Moreover, a desirable property of $T\left(\mu ;{\mathcal{D}}^0\right)$ is that our empirical results suggest that for a fixed μ and ν₁ < ν₂

$$P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,{\nu }_1}\right)\ge t\right\}<P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,{\nu }_2}\right)\ge t\right\}$$ (5)

for t in the tail region of interest. This inequality implies that

$$p\left(\mu ;{\mathcal{D}}^0\right)=\underset{\nu }{\text{sup}}p\left(\mu ,\nu ;{\mathcal{D}}^0\right)\approx p\left\{\mu ,{\nu }_{\text{sup}}(\mu );{\mathcal{D}}^0\right\},$$

which can greatly simplify the calculation of $p\left(\mu ;{\mathcal{D}}^0\right)$. While the verification of (5) is not straightforward analytically and only holds approximately, the underlying rational is obvious: a larger variance in the distribution of π_i implies higher variability for the test statistics and in turn increases the tail probability². Figure 2 contains an example confidence region based on the proposed test statistic in the (μ, ν) plane supporting the claim in (5). We leverage this property to substantially reduce the computational complexity by O(J) operations by simply considering values along the boundary of the reduced BB parameter space illustrated in Figure 2. In practice, our CI may be constructed based on the following simple procedure:

• (Step 1)

  Obtain the CI based on the asymptotic chi-square approximation to $T\left(\mu ;{\mathcal{D}}^0\right)$, denoted by $\left[{\mu }_{\text{LB}}^{\text{MOM}},{\mu }_{\text{UB}}^{\text{MOM}}\right]$.

• (Step 2)
  Letting $\tilde{p}\left(\mu ;{\mathcal{D}}^0\right)=p\left\{\mu ,{\nu }_{\text{sup}}(\mu );{\mathcal{D}}^0\right\}$, compute

  $$\tilde{p}\left({\mu }_{\text{LB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right)\text{and}\tilde{p}\left({\mu }_{\text{UB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right),$$

  where ${\mu }_{\text{LB}}^{\widetilde{\text{MOM}}}=\text{max}\left\{s,{\tilde{\mu }}_{\text{LB}}^{\text{MOM}}\right\}$, ${\mu }_{\text{UB}}^{\widetilde{\text{MOM}}}=\text{min}\left\{1-s,{\tilde{\mu }}_{\text{UB}}^{\text{MOM}}\right\}$, and s is the chosen grid size.
• (Step 3)
  Identify the starting points for the lower and upper confidence bounds, respectively, as

  $${\tilde{\mu }}_{\text{inf}}={\mu }_{\text{LB}}^{\widetilde{\text{MOM}}}I\left\{\tilde{p}\left({\mu }_{\text{LB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right)\ge \alpha \right\}+\widehat{\mu }I\left\{\tilde{p}\left({\mu }_{\text{LB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right)<\alpha \right\}$$

  and

  $${\tilde{\mu }}_{\text{sup }}={\mu }_{\text{UB}}^{\widetilde{\text{MOM}}}I\left\{\tilde{p}\left({\mu }_{\text{UB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right)\ge \alpha \right\}+\widehat{\mu }I\left\{\tilde{p}\left({\mu }_{\text{UB}}^{\widetilde{\text{MOM}}};{\mathcal{D}}^0\right)<\alpha \right\}.$$
• (Step 4)
  Iterate out to the left and right along the boundary provided in (3) to compute $\tilde{p}\left(\mu ;{\mathcal{D}}^0\right)$ using steps A.1 to A.3 until [μ_LB, μ_UB], where

  $${\mu }_{\text{LB}}=\underset{\mu }{\text{inf}}\left[\mu \mid \tilde{p}\left(\mu ;{\mathcal{D}}^0\right)\ge \alpha \right],$$

  and

  $${\mu }_{\text{UB}}=\underset{\mu }{\text{sup}}\left[\mu \mid \tilde{p}\left(\mu ;{\mathcal{D}}^0\right)\ge \alpha \right].$$

  In practice, we would compute

  $$\left\{\tilde{p}\left({\tilde{\mu }}_{\text{inf}}-ks;{\mathcal{D}}^0\right),k=1,2,\dots \right\}\text{and }\left\{\tilde{p}\left({\tilde{\mu }}_{\text{sup}}+ks;{\mathcal{D}}^0\right),k=1,2,\dots \right\},$$

  and let

  $${\mu }_{\text{LB}}=\text{min}\left\{{\tilde{\mu }}_{\text{inf}}-ks\mid \tilde{p}\left({\tilde{\mu }}_{\text{inf}}-ks;{\mathcal{D}}^0\right)\ge \alpha ,k\ge 0\right\},$$

  and

  $${\mu }_{\text{UB}}=\text{max}\left\{{\tilde{\mu }}_{\text{sup}}+ks\mid \tilde{p}\left({\tilde{\mu }}_{\text{sup}}+ks;{\mathcal{D}}^0\right)\ge \alpha ,k\ge 0\right\}.$$
• (Step 5)
  Compute $p\left(\mu ;{\mathcal{D}}^0\right)={\text{sup}}_{\nu }p\left(\mu ,\nu ;{\mathcal{D}}^0\right)$ for μ ∈ [μ_LB−δ, μ_LB] for some small δ > 0. To this end, we need to calculate $p\left(\mu ,\nu ;{\mathcal{D}}^0\right)$ for many pairs of ν and μ. For example, to compute the lower bound of the CI, we would need to compute $p\left({\mu }^{(j)},{\nu }^{\left(j_i\right)};{\mathcal{D}}^0\right)$ and approximate $p\left({\mu }^{(j)};{\mathcal{D}}^0\right)$ by

  $$\text{max}\left\{p\left({\mu }^{(j)},{\nu }^{\left(j_i\right)};{\mathcal{D}}^0\right)\mid i=1,\dots ,I_j\right\},$$

  where {μ⁽¹⁾, μ⁽²⁾, … , μ^(J)} are J equally spaced points within the interval [μ_LB −δ, μ_LB] and $\left\{{\nu }^{\left(j_1\right)},{\nu }^{\left(j_2\right)},\dots ,{\nu }^{\left(j_{I_j}\right)}\right\}$ are I_j equally spaced points within the interval [0, ν_sup(μ^(j))]. The lower end of the CI is given by

  $$\text{min}\left\{{\mu }^{(j)}\mid p\left({\mu }^{(j)};{\mathcal{D}}^0\right)\ge \alpha ,j=1,\dots ,J\right\}.$$
• (Step 6)

  Likewise, we can calculate the upper end of the CI by computing $p\left(\mu ;{\mathcal{D}}^0\right)$ for μ ∈ [μ_UB, μ_UB + δ].

FIGURE 2.

Example confidence region in the (μ, v) plane illustrating the stochastic dominance property in (5). The dashed lined is the boundary of the parameter space for the restricted beta-binomial model

This algorithm uses the fact that $p\left(\mu ;{\mathcal{D}}^0\right)\approx p\left\{\mu ,{\nu }_{\text{sup }}(\mu );{\mathcal{D}}^0\right\}$ and the cut off values based on the latter as μ_LB and μ_UB are very close to the ends of the CI. In practice, further gains in computational speed may be achieved with standard parallelization techniques.

Remark 2.

While we may consider other scales (such as logit) in deciding the grid points for searching the confidence bounds of μ, we prefer the original scale with the intention to control the precision on the probability scale. In practice, the grid size s needs to be chosen according to the required precision of the incidence rate of interest. Oftentimes, s = 0.001 is a sensible choice. The number of Monte Carlo iterations in calculating $p\left(\mu ,\nu ;{\mathcal{D}}^0\right)$ should also depend on the desired precision. For example, if we want to estimate a p-value of 0.05 with a standard error of 0.005 then M needs to be ≥ 2000. The tuning parameter, δ, in steps 5 and 6 can be set as a multiple of s such as 10s in practice.

Remark 3.

We note that the selection of the test statistic is not unique. A simple alternative is $\tilde{T}\left(\mu ;{\mathcal{D}}^0\right)={\left({\sum }_{i=1}^K{Y}_i-\mu {\sum }_{i=1}^K{n}_i\right)}^2$. In our limited experience, we have found that the corresponding exact CI is similar to that based on $T\left(\mu ;{\mathcal{D}}^0\right)$ when K is small. One potential advantage of this test statistic is that the aforementioned approximation to $p\left\{\mu ,{\nu }_{\text{sup}}(\mu );{\mathcal{D}}^0\right\}$ appears to become an equality, ie, $p\left(\mu ;{\mathcal{D}}^0\right)=p\left\{\mu ,{\nu }_{\text{sup}}(\mu );{\mathcal{D}}^0\right\}$. Thus, we may skip steps 5 and 6 in the proposed algorithm to further accelerate the computation.

3 |. NUMERICAL STUDIES

3.1 |. Simulation study

We evaluated the performance of the proposed method in finite samples through simulation study. Throughout, the observed data were generated as

$$Y_i\mid {\pi }_i~\text{Bin}\left(n_i,{\pi }_i\right),{\pi }_i~\text{Beta}\left({\alpha }_0,{\beta }_0\right),i=1,\dots ,K,$$

for different values of K and (α₀, β₀) to represent varying degrees of heterogeneity and event rates. Specifically, we consider a rare-event rate setting with

$$\text{Setting}\text{1a:}\left({\alpha }_0,{\beta }_0\right)=(1.2,75)\text{and}\text{Setting }1\text{b}:\left({\alpha }_0,{\beta }_0\right)=(3.6,225)$$

and a low event rate setting with

$$\text{Setting}\text{2a: }\left({\alpha }_0,{\beta }_0\right)=(1.2,10)\text{and}\text{Setting }2\text{b}:\left({\alpha }_0,{\beta }_0\right)=(3.6,30)$$

so that (μ₀, τ₀) = (0.016, 0.013) and (0.016, 0.004) in Settings 1a and 1b, respectively, and (μ₀, τ₀) = (0.107, 0.082) and (0.107, 0.029) in Settings 2a and 2b, respectively. For all scenarios, we set K = 5, 10, 15, and 20 and let the study specific sample sizes be n₁ = n₂ = 50, n₃ = n₄ = 100, and n_i = 150 for i ≥ 5/ For comparison, we present the performance of the CIs based on the DL and Sidik-Jonkman (SJ) methods with (1) the logistic transformation and 0.5 continuity correction (DL-LGT and SJ-LGT) and (2) the arcsine transformation (DL-ASIN and SJ-ASIN) as well as the CI based on the asymptotic chi-square approximation to the proposed test statistics (4) denoted as the method-of-moment estimators (MOM). We have also obtained the exact CI under the FE model where the pooled data ${\sum }_{i=1}^K{Y}_i~\text{Bin}\left({\sum }_{i=1}^K{n}_i,{\mu }_0\right)$. In Tables 1 and 2, we summarize the performance of the 95% CIs from the various procedures based on the median interval length and empirical coverage probability from 2000 simulated data sets.

TABLE 1.

The empirical coverage probabilities (CovP) of the 95% confidence intervals and the median lengths (Length) of the various methods in the rare-event rate setting and moderate study-specific sample sizes

(1a) α0 = 1.2,β0=75
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.962 (0.040)	0.954 (0.028)	0.958 (0.023)	0.950 (0.019)
MOM	0.821 (0.025)	0.85 (0.018)	0.876 (0.016)	0.900 (0.014)
DL-LGT	0.882 (0.032)	0.848 (0.021)	0.831 (0.017)	0.841 (0.015)
SJ-LGT	0.958 (0.037)	0.958 (0.023)	0.961 (0.019)	0.970 (0.016)
DL-ASIN	0.792 (0.026)	0.751 (0.018)	0.699 (0.015)	0.648 (0.013)
SJ-ASIN	0.822 (0.028)	0.781 (0.019)	0.734 (0.016)	0.684 (0.014)
FE	0.890 (0.024)	0.790 (0.015)	0.788 (0.011)	0.786 (0.010)
(1b) α0 = 3.6,β0=30
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.978 (0.039)	0.98 (0.025)	0.976 (0.018)	0.971 (0.015)
MOM	0.874 (0.024)	0.903 (0.015)	0.920 (0.012)	0.920 (0.011)
DL-LGT	0.898 (0.031)	0.837 (0.019)	0.769 (0.015)	0.714 (0.012)
SJ-LGT	0.959 (0.034)	0.961 (0.021)	0.964 (0.017)	0.958 (0.014)
DL-ASIN	0.858 (0.025)	0.840 (0.017)	0.812 (0.014)	0.776 (0.012)
SJ-ASIN	0.878 (0.028)	0.877 (0.019)	0.854 (0.015)	0.814 (0.013)
FE	0.948 (0.026)	0.914 (0.015)	0.905 (0.011)	0.906 (0.010)

TABLE 2.

The empirical coverage probabilities (CovP) of the 95% confidence intervals and the median lengths (Length) of the various methods in the low event rate setting and moderate study-specific sample sizes

(2a) α0 = 1.2,β0=10
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.962 (0.222)	0.955 (0.157)	0.961 (0.116)	0.960 (0.094)
MOM	0.802 (0.122)	0.882 (0.101)	0.909 (0.087)	0.920 (0.076)
DL-LGT	0.864 (0.131)	0.878 (0.091)	0.840 (0.073)	0.816 (0.063)
SJ-LGT	0.890 (0.151)	0.905 (0.107)	0.882 (0.087)	0.844 (0.076)
DL-ASIN	0.832 (0.136)	0.850 (0.102)	0.836 (0.085)	0.819 (0.075)
SJ-ASIN	0.838 (0.137)	0.854 (0.104)	0.844 (0.086)	0.820 (0.075)
FE	0.478 (0.058)	0.426 (0.035)	0.428 (0.028)	0.425 (0.024)
(2b) α = 3.6,β=30
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.990 (0.176)	0.968 (0.103)	0.959 (0.074)	0.969 (0.061)
MOM	0.836 (0.084)	0.883 (0.066)	0.914 (0.056)	0.925 (0.049)
DL-LGT	0.888 (0.096)	0.906 (0.067)	0.920 (0.055)	0.922 (0.047)
SJ-LGT	0.924 (0.106)	0.938 (0.075)	0.940 (0.062)	0.945 (0.053)
DL-ASIN	0.866 (0.096)	0.892 (0.070)	0.896 (0.058)	0.900 (0.050)
SJ-ASIN	0.887 (0.099)	0.904 (0.072)	0.907 (0.059)	0.909 (0.052)
FE	0.688 (0.059)	0.626 (0.036)	0.625 (0.028)	0.642 (0.024)

With the exception of the SJ-LGT method, all comparison methods fail to achieve the nominal coverage level of 95%. For instance, the empirical coverage probabilities for the remaining methods based on the random-effect model range from 64.8% to 93.2% with K = 20. Although the SJ-LGT method is generally the best performing method of the DL and SJ methods considered, the coverage probability substantially deteriorates with increasing K in Setting 2a. The MOM method exhibits more stable behavior but falls well below the nominal level for K = 20 in Setting 1a. As expected, the FE method exhibits significant undercoverage with the best performance in Setting 1b where the between-study heterogeneity is the smallest.

The proposed method is conservative for K = 5 and the settings with lower heterogeneity (Settings 1b and 2b), but overall, we find that the coverage levels are only moderately higher than 95% in the scenarios considered. As expected, the length of the proposed CI decreases with increasing K. With K = 20, the length is generally on par with the competing methods, although yields a coverage level closer to the desired 95%.

For completeness, we also considered analogous scenarios with smaller study-specific sample sizes. Specifically, for both settings, we set the study specific sample sizes to: n₁ = n₂ = 10, n₃ = n₄ = 15, and n_i = 20 for i ≥ 5. The results are summarized in Tables 3 and 4. When μ₀ = 0.107, the FE method exhibits undercoverage while the empirical coverage levels of CIs based on the new method are all above the nominal level. The empirical coverage level of CIs based on SJ-LGT are also satisfactory. When μ₀ = 0.016, the empirical coverage levels of CIs based on DL-LGT, SJ-LGT, DL-ASIN, and SJ-ASIN are substantially lower than the nominal level. A potential reason is that the bias caused by the 0.5 correction in zero event studies becomes nonnegligible relative to the true incidence rates which are very low.

TABLE 3.

The empirical coverage probabilities (CovP) of the 95% confidence intervals and the median lengths (Length) of the various methods in the rare-event rate setting and small study-specific sample sizes

(2a) α = 1.2,β=75
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.961 (0.098)	0.952 (0.046)	0.944 (0.035)	0.956 (0.029)
MOM	0.999 (0.115)	1.000 (0.068)	0.998 (0.054)	0.999 (0.045)
DL-LGT	0.696 (0.112)	0.308 (0.064)	0.088 (0.051)	0.026 (0.042)
SJ-LGT	0.696 (0.113)	0.308 (0.065)	0.088 (0.051)	0.026 (0.043)
DL-ASIN	0.944 (0.088)	0.796 (0.052)	0.573 (0.042)	0.285 (0.036)
SJ-ASIN	0.962 (0.088)	0.840 (0.054)	0.586 (0.042)	0.295 (0.036)
FE	0.964 (0.077)	0.968 (0.040)	0.960 (0.033)	0.950 (0.029)
(2b) α = 3.6,β=225
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.962 (0.098)	0.964 (0.046)	0.956 (0.035)	0.964 (0.029)
MOM	1.000 (0.115)	1.000 (0.068)	0.998 (0.053)	0.999 (0.045)
DL-LGT	0.697 (0.112)	0.270 (0.064)	0.086 (0.050)	0.020 (0.042)
SJ-LGT	0.696 (0.113)	0.270 (0.065)	0.086 (0.050)	0.020 (0.043)
DL-ASIN	0.942 (0.088)	0.828 (0.053)	0.584 (0.042)	0.276 (0.036)
SJ-ASIN	0.958 (0.088)	0.853 (0.052)	0.598 (0.042)	0.285 (0.036)
FE	0.966 (0.077)	0.981 (0.040)	0.962 (0.033)	0.962 (0.029)

TABLE 4.

The empirical coverage probabilities (CovP) of the 95% confidence intervals and the median lengths (Length) of the various methods in the low event rate setting and small study-specific sample sizes

(2a) α = 1.2,β=10
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.970 (0.229)	0.962 (0.146)	0.960 (0.119)	0.960 (0.104)
MOM	0.834 (0.160)	0.859 (0.110)	0.894 (0.095)	0.900 (0.086)
DL-LGT	0.920 (0.189)	0.882 (0.133)	0.860 (0.110)	0.869 (0.095)
SJ-LGT	0.960 (0.215)	0.964 (0.148)	0.968 (0.121)	0.974 (0.104)
DL-ASIN	0.914 (0.151)	0.882 (0.107)	0.888 (0.091)	0.874 (0.079)
SJ-ASIN	0.930 (0.169)	0.927 (0.120)	0.934 (0.099)	0.916 (0.086)
FE	0.840 (0.154)	0.803 (0.098)	0.823 (0.077)	0.814 (0.065)
(2b) α = 3.6,β=30
	K = 5	K = 10	K = 15	K = 20
Method	CovP (Length)	CovP (Length)	CovP (Length)	CovP (Length)
Exact	0.990 (0.218)	0.983 (0.134)	0.981 (0.102)	0.975 (0.087)
MOM	0.906 (0.155)	0.912 (0.100)	0.922 (0.079)	0.926 (0.069)
DL-LGT	0.924 (0.177)	0.881 (0.116)	0.831 (0.093)	0.774 (0.080)
SJ-LGT	0.974 (0.198)	0.973 (0.133)	0.968 (0.107)	0.970 (0.092)
DL-ASIN	0.967 (0.145)	0.931 (0.095)	0.920 (0.076)	0.900 (0.066)
SJ-ASIN	0.974 (0.162)	0.957 (0.110)	0.955 (0.089)	0.948 (0.077)
FE	0.934 (0.154)	0.900 (0.098)	0.910 (0.076)	0.914 (0.065)

Furthermore, we have investigated the validity of (5):

$$P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,{\nu }_1}\right)\ge t\right\}<P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,{\nu }_2}\right)\ge t\right\},\text{for}{\nu }_1<{\nu }_2,$$

via numerical studies. Specifically, with μ₀ at the previous simulation setting, we estimate the cumulative distribution function of $T\left({\mu }_0;{\mathcal{D}}^{{\mu }_0,\nu }\right)$ for ν = ν_sup(μ₀)k∕5, k = 1, …, 5, using Monte Carlo simulation. To this end, we simulated 10⁶ test statistics for each combination of (μ₀, ν, n₁, …, n_K) and obtained the empirical survival function. We have plotted the survival functions for μ₀ = 0.016 and μ₀ = 0.107 with moderate study specific sample sizes in Figures 3 and 4, respectively, to examine whether they are stochastically ordered according to the value of ν. Likewise, we have plotted the survival function for μ₀ = 0.107 with small study specific sample sizes in Figure 5. The simulation results confirm the inequality (5) and especially the claim that

$$\underset{\nu \le {\nu }_{\text{sup }}(\mu )}{\text{sup}}P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,\nu }\right)\ge t\right\}=P\left\{T\left(\mu ;{\mathcal{D}}^{\mu ,{\nu }_{\text{sup }p}(\mu )}\right)\ge t\right\}$$

under the given settings.

FIGURE 3.

The empirical survival functions for the test statistic with μ = 0.016 for the moderate study-specific sample size setting. The five survival curves from bottom to top correspond to increasing values of v

FIGURE 4.

The empirical survival functions for the test statistic with μ = 0.107 for the moderate study-specific sample size setting. The five survival curves from bottom to top correspond to increasing values of v

FIGURE 5.

The empirical survival functions for the test statistic with μ = 0.107 for the small study-specific sample size setting. The five survival curves from bottom to top correspond to increasing values of v

3.2 |. Real data analysis

AML is an aggressive cancer that begins in the bone marrow and results in an elevated level of immature white blood cells called myeloid blasts in the bloodstream. CD33 is an antigen expressed on the blast cells in the majority of AML patients and has been identified as a suitable candidate for targeted treatment of the disease. One such drug, Mylotarg, was approved by the FDA in 2000 for the treatment of patients with CD33-positive AML in first relapse who are 60 years of age or older and who are not candidates for other types of cytotoxic chemotherapy. Confirmatory trials revealed safety concerns, including higher rates of EM and veno-occlusive disease (VOD) and failed to confirm a clinical benefit. Here, we illustrate the proposed method in three meta-analyses of 14 trials aiming to evaluate the safety and/or efficacy of an unfractionated regimen of Mylotarg: 2 doses of 6 mg/m² given 14 days apart.⁴ The studies consisted of three pivotal phase II multicenter trials as well as dose-escalation and postmarketing studies in patients with relapsed or refractory AML. The efficacy of the treatment regimen was evaluated based on meta-analyses of complete remission (CR) rate across the studies presented in Table A1 in the Appendix. Safety was assessed with meta-analyses of EM and VOD based on the studies presented in Tables A2 and A3. We also provide some discussion of the comparison of the 6 mg/m² unfractionated regimen to a fractionated dosing regimen of 3 mg/m² given on days 1, 4, and 7 for which very few studies were available for analysis.

The 95% CIs and corresponding lengths based on the proposed exact method as well as the FE, SJ-LGT, and SJ-ASIN methods for the 6 mg/m² regimen are presented in Table 5. The number of available studies for analysis is quite small for each of the outcomes. Additionally, all studies have 16 or fewer patients and the number of zero event studies is high. For example, CR was not observed in four of the six studies considered. As a result, we observe substantial differences in the point estimates for the three methods, ranging from 0.013 to 0.035, as well as the corresponding CIs. The SJ-LGT interval is more than twice the length of SJ-ASIN interval and longer than the CI from the exact method. Similar though less extreme patterns are observed for the EM and VOD analyses. Consistent with our simulation results and intuition, we also find the intervals from the FE method are shorter than those from the proposed method. These differences highlight the utility of our method in practice as its validity does not rely on the choice of transformation or large sample approximation.

TABLE 5.

Random effects meta-analyses of the efficacy (complete remission (CR)) and safety (early mortality (EM) and veno-occlusive disease (VOD)) of the 6 mg/m² regimen of Mylotarg based on the proposed exact method and the SJ-LGT and SJ-ASIN methods. Presented are the 95% confidence intervals and corresponding lengths for each of the methods

Study	CR	EM	VOD
K	6	5	7
Exact	0.007–0.128 (0.121)	0.049–0.340 (0.291)	0.072–0.367 (0.295)
SJ-LGT	0.027–0.185 (0.158)	0.064–0.342 (0.279)	0.112–0.421 (0.310)
SJ-ASIN	0.003–0.076 (0.073)	0.001–0.252 (0.251)	0.012–0.306 (0.294)
FE	0.004 – 0.121 (0.116)	0.041–0.262 (0.221)	0.105–0.350 (0.245)

In addition to evaluating the 6 mg/m² unfractionated treatment regimen, interest also lied in comparison to a fractionated dosing regimen of 3 mg/m². However, only two studies evaluated CR and EM for the fractionated regimen and three studies for VOD. We therefore constructed exact binomial CIs based on the pooled data, effectively assuming a FE model, as it is difficult to support the modeling assumptions for a random-effect model in this setting. The resulting CIs for CR, EM, and VOD were estimated to be (0.153, 0.379), (0.035, 0.170), and (0, 0.042), respectively. While these results suggest some benefit in efficacy for the fractioned treatment versus the unfractionated treatment as well as lower rates of VOD and EM, they must be interpreted with caution due to the small number of studies and the use of the FE approach. The confidence intervals based on the proposed random-effect approach are wider as expected, namely (0.060, 0.724), (0.030, 0.377), and (0, 0.169) respectively, but we again caution that there is insufficient information to support the related model assumptions for the application of this method.

4 |. DISCUSSION

In this paper, we have proposed a random-effect model for combining multiple binomial random variables with underlying probabilities following a parametric distribution. In particular, we introduced an exact CI for the location parameter of the BB model. The method is valid regardless of the number of studies or the sample size within each of the studies and is thus especially suitable to study rare-event data where the event probability is close to zero. While the performance of the method depends on the choice of the test statistic, we choose the modified DL test statistic to ensure that our method yields similar performance as the DL method when K is not small as the DL method is asymptotically optimal. A closely related problem and a direction warranting future research is to extend our method to combine a group of two-by-two tables which are used to characterize the between group difference in binary outcomes from multiple studies.

Finally, we note that when the number of studies is less than 4 we oftentimes lack sufficient information to either support or refute the involved model assumptions. We therefore advise practitioners to avoid over interpreting the results despite the proposed method maintaining validity under correctly specified model assumptions. Additionally, we emphasize that having a small number of studies by itself is not necessarily a good reason for using the simple FE model since its implicit assumption is even stronger than that of the random-effect model. As a general suggestion, the choice between random versus FE approaches should be based on the observed data and one should always clearly state the key assumptions together with the analysis results.

APPENDIX

DATA FOR MYLOTARG EXAMPLE

TABLE A1.

Complete remission rates for the studies of the two regimens of Mylotarg

(a) 6 mg/m2
	Events	Total
Study 101	0	8
Study 102	1	14
Study 103	0	6
Study 100374	1	6
Piccaluga, 2004	0	7
van der Heiden, 2006	0	16
(b) 3 mg/m2
	Events	Total
MyloFrance 1	15	57
Brethon, 2006	1	6

TABLE A2.

Rates of early mortality for the studies of the two regimens of Mylotarg

(a) 6 mg/m2
	Events	Total
Study 101	2	8
Study 102	1	14
Study 103	0	6
Study 100374	0	6
Piccaluga, 2004	2	7
(b) 3 mg/m2
	Events	Total
MyloFrance 1	4	57
Thomas, 2005	3	24

TABLE A3.

Rates of veno-occlusive disease for the studies of the two regimens of Mylotarg

(a) 6 mg/m2
	Events	Total
Study 101	1	8
Study 102	6	14
Study 103	0	6
Study 100374	2	6
Thomas, 2005	1	6
Piccaluga, 2004	0	7
Zwaan, 2003	0	1
(b) 3 mg/m2
	Events	Total
MyloFrance 1	0	57
Thomas, 2005	0	24
Brethon, 2006	0	6