Prior Effective Sample Size in Conditionally Independent Hierarchical Models

Abstract

Prior effective sample size (ESS) of a Bayesian parametric model was defined by Morita, et al. (2008, Biometrics, 64, 595–602). Starting with an ε-information prior defined to have the same means and correlations as the prior but to be vague in a suitable sense, the ESS is the required sample size to obtain a hypothetical posterior very close to the prior. In this paper, we present two alternative definitions for the prior ESS that are suitable for a conditionally independent hierarchical model. The two definitions focus on either the first level prior or second level prior. The proposed methods are applied to important examples to verify that each of the two types of prior ESS matches the intuitively obvious answer where it exists. We illustrate the method with applications to several motivating examples, including a single-arm clinical trial to evaluate treatment response probabilities across different disease subtypes, a dose-finding trial based on toxicity in this setting, and a multicenter randomized trial of treatments for affective disorders.

1 Introduction

Recently, a definition for the effective sample size (ESS) of a given prior π(θ) with respect to a sampling model p(Y | θ) was proposed by Morita, Thall, and Müller (2008) (MTM). The ESS provides an easily interpretable index of the informativeness of a prior with respect to a given likelihood. The approach is to first define an ε-information prior π₀(θ) having the same means and correlations as π(θ) but being vague in a suitable sense, and then define the ESS to be the sample size n of hypothetical outcomes Y_n = (Y₁, ···Y_n) that, starting with π₀(θ), yields a hypothetical posterior π_n(θ | Y_n) very close to π(θ). MTM define the distance between π(θ) and π_n(θ | Y_n) in terms of the trace of the negative second derivative matrices of log{π(θ)} and log{π_n(θ | Y_n)}. The ESS is defined as the interpolated value of n that minimizes this “prior-to-posterior” distance. While this definition is suitable for a wide range of models and applications, it fails for hierarchical models.

In this paper, we propose two extensions of the definition for prior ESS that are applicable to two-stage conditionally independent hierarchical models (CIHMs) (Kass and Steffey, 1989). Our approach is pragmatic in that the ESS can always be evaluated either analytically or by using a simulation-based approach. We validate the definitions by verifying that they match the intuitively obvious answers in important special cases where such answers exist. We focus on the class of CIHMs due to their practical importance in settings where data are collected from exchangeable subgroups, such as study centers, schools, cities, etc. Important areas of application include meta-analysis (cf. Berry and Stangl, 1996; Berlin and Colditz, 1999), and clinical trial design (cf. Thall et al., 2003).

Moreover, we restrict attention to CIHMs as the most commonly used versions of hierarchical models. For more complex hierarchical models, one might report prior ESS values for appropriate sub-models, although it might become less meaningful to report an overall ESS.

A two-level CIHM for K sub groups is defined as follows. Let Y_k = (Y_k,1, …, Y_k,nk) denote the vector of outcomes for sub-group k and let = (Y₁,…, Y_K), with the K vectors assumed to be distributed independently conditional on hyperparameters. We use f(·) generically to indicate the sampling model of observable data, which may be a single variable Y_k,j, a vector Y_k, or the vector of all n₁ +···+n_K observations. The nature of the argument will clarify the specific meaning of f(·). In the first level, Y_k follows distribution f(Y_k | θ_k). In the second level, the subgroup-specific parameters θ=(θ₁, ···, θ_K) are assumed to be i.i.d. with prior π₁(θ_k | θ̃), where the hyperparameter θ̃ has a hyperprior π₂(θ̃ | φ) with known φ. The model is summarized in equation (1).

$$\begin{gathered} \text{Sampling model}f({\mathcal{Y}}_M\mid \mathit{\theta })={\prod }_{k=1}^{K}f({\mathbf{Y}}_k\mid {\mathit{\theta }}_k)(\text{Level 1})\text{Prior}{\pi }_1(\mathit{\theta }\mid \stackrel{\sim }{\mathit{\theta }})={\prod }_{k=1}^K{\pi }_1({\mathit{\theta }}_k\mid \stackrel{\sim }{\mathit{\theta }}) \\ (\text{Level 2})\text{Hyperprior}{\pi }_2(\stackrel{\sim }{\mathit{\theta }}\mid \mathit{\phi }). \end{gathered}$$ (1)

A common example of a CIHM (1) is a conjugate normal/inverse χ²-normal-normal model. Let Inv-χ²(ν, S) denote a scaled inverse χ² distribution with ν degrees of freedom, mean $\frac{\nu S}{\nu -2}$ for ν > 2, and variance $\frac{2{\nu }^2{S}^2}{{(\nu -2)}^2(\nu -4)}$ for ν > 4. This model has a normal sampling distribution Y_k,i | θ_k ~ N(θ_k,σ²) with known σ², where N(μ, σ²) denotes a normal distribution with mean μ, and variance σ². Independent conjugate normal priors θ_k | μ̃, γ̃² ~ N(μ̃, γ̃²) on the location parameters θ₁, ···, θ_K are assumed, with a normal/inverse χ² hyperprior μ̃ | μ_φ, $\stackrel{\sim }{\mu }\mid {\mu }_{\phi },{\tau }_{\phi }^2~\text{N}({\mu }_{\phi },{\tau }_{\phi }^2)$ and γ̃² | ν_φ, S_φ ~ Inv-χ²(ν_φ, S_φ). Once the methodology is established, we will explain how to compute ESS when σ² is not assumed to be known but rather is random with its own prior.

To compute an ESS under a CIHM, we will consider the following two cases, which address different inferential objectives. In case 1, the target is the marginalized prior,

$${\pi }_{12}(\mathit{\theta }\mid \mathit{\phi })=\int {\pi }_1(\mathit{\theta }\mid \stackrel{\sim }{\mathit{\theta }}){\pi }_2(\stackrel{\sim }{\mathit{\theta }}\mid \mathit{\phi })d\stackrel{\sim }{\mathit{\theta }}.$$ (2)

An example is a setting where θ₁,…,θ_K are treatment effects in K different disease subtypes and they are the parameters of primary interest. In case 2, the target prior is

$${\pi }_2(\stackrel{\sim }{\mathit{\theta }}\mid \mathit{\phi }).$$ (3)

For example, this would arise if an overall treatment effect θ̃ obtained by averaging over K clinical centers in a multi-center clinical trial is the parameter of primary interest. We propose two definitions for the ESS under a CIHM, one for each case, allowing the possibility that both types of ESS may be of interest in a given analysis. For later reference we define the marginal likelihood

$$f_1({\mathbf{Y}}_k\mid \stackrel{\sim }{\mathit{\theta }})=\int f({\mathbf{Y}}_k\mid {\mathit{\theta }}_k){\pi }_1({\mathit{\theta }}_k\mid \stackrel{\sim }{\mathit{\theta }})d{\mathit{\theta }}_k$$ (4)

by integrating with respect to the level I prior.

Section 2 presents motivating examples. We briefly summarize the MTM formulation in Section 3. The two definitions of ESS in CIHMs and accompanying computational methods are given in Section 3. In Section 4 we compute the ESS for the three motivating examples. In Section 5 we discuss some standard CIHMs, and we close with a brief discussion in Section 6.

2 Motivating examples

2.1 A Single-Arm Sarcoma Trial

Thall et al. (2003) present a design for a single-arm phase II trial to examine the efficacy of the targeted drug imatinib for sarcoma, a disease with many subtypes. Since sarcomas are uncommon, the goal was to construct a design that allowed the efficacy of imatinib to be evaluated in K = 10 sarcoma subtypes. This was achieved by assuming the following CIHM, where the treatment effects differ across subtypes. The parameters of primary interest were the subtype-specific tumor response probabilities, ξ₁, ···, ξ₁₀. Let Ga(a_φ, b_φ) denote a gamma distribution with mean a_φ/b_φ and variance $a_{\phi }/b_{\phi }^2$. Denoting θ_k = log{ξ_k/(1 − ξ_k)}, it was assumed that θ₁, …, θ₁₀ were i.i.d. N(μ̃, γ̃⁻¹) and that μ̃, and the precision parameter γ̃ followed independent normal and gamma hyperpriors, respectively. Elicitation of prior probabilities characterizing association between pairs of ξ_k’s yielded the hyperpriors μ̃~ N(−1.386,10) and γ̃ ~ Ga(2, 20), so that E(γ̃) = .10 and var(γ̃) = .005. In summary, the trial design assumed the following model:

$$\begin{array}{llll}\text{Sampling}\text{model}\hfill & \hfill {\mathbf{Y}}_{k,m}\mid {\theta }_k& ~\hfill & \text{Bin}(m,{\xi }_k)\text{indep}.\text{for}\text{all}k\hfill \\ \text{Prior}\hfill & \hfill {\theta }_k\mid \stackrel{\sim }{\mu },\stackrel{\sim }{\gamma }& ~\hfill & \text{N}(\stackrel{\sim }{\mu },{\stackrel{\sim }{\gamma }}^{-1})\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \text{Hyperpriors}\hfill & \hfill \stackrel{\sim }{\mu }& ~\hfill & \text{N}(-1.386,10)\hfill \\ \hfill & \hfill \stackrel{\sim }{\gamma }& ~\hfill & \text{Ga}(2,20).\hfill \end{array}$$ (5)

Thall et al. (2003) used the marginal posterior probability Pr(ξ_k > 0.30 | ) to define an early stopping criterion in disease subtype k, which was computed based on the posterior π(θ | ) under (5). Thus, 10 stopping rules were applied, one for each subtype. Note that included the data from all ten subtypes in order to exploit the association among the θ_k’s induced by the hierarchical model. This rule was first applied after observing a minimum of eight patients in each disease subtype, and subsequently at sample sizes of 17, 23, and 30 patients. Thus, an overly informative prior, for example, with a prior ESS ≥ 40 might be considered inappropriate since the prior, rather than patient response data, would dominate early termination decisions. Because both a prior and hyperprior were specified, the methods in MTM are not applicable, and it is not obvious how to determine the ESS of this model. We will show below that the ESS can be determined for this model using an approach that is coherent in the sense that it gives the intuitively obvious answer in cases where the ESS exists.

2.2 A CRM Dose-finding Trial for Multiple Patient Subgroups

As an extension of the HM in (5) we consider a phase I dose-finding trial with multiple patient subgroups using a model-based design. We assume an implementation that generalizes the continual reassessment method (O’Quigley et al, 1990). Suppose that there are K = 4 subgroups (k = 1,…, 4) with population proportions (.40, .30, .20, .10). Each patient in each subgroup receives one of six doses, 100, 200, 300, 400, 500, 600, denoted by d₁,…,d₆, with standardized doses $x_z=log(d_z)-1/6{\sum }_{l=1}^{6}log(d_l)$. The outcome variable is the indicator Y_k,i = 1 if the i^th patient in subgroup k suffers toxicity, 0 if not. The probability of toxicity in subgroup k under dose x_i is denoted by π_k(x_i,α_k,β_k) = Pr(Y_k,i = 1 | x_i,α_k,/β_k) with logit (π_k(x_i, α_k,/β_k)} = α_k + β_kx_i, for k = 1,2,3,4. We have a CRM-type goal of finding the “optimal” dose $x_k^{\ast }$ in each subgroup k. Optimal is defined as the posterior mean of ${\pi }_k(x_k^{\ast })$ being closest to some fixed target π^*. The maximum sample size is 36, with the cohort size of 1, starting at the lowest dose d₁, and not skipping a dose level when escalating, with target toxicity probability π^* = .30. The parameters of primary interest are θ_k = (α_k, β_k), k = 1,2, 3, 4. It is assumed that α₁, …, α₄ and β₁,…, β₄ are i.i.d. $\text{N}({\stackrel{\sim }{\mu }}_{\alpha },{\stackrel{\sim }{\sigma }}_{\alpha }^2)$ and $\text{N}({\stackrel{\sim }{\mu }}_{\beta },{\stackrel{\sim }{\sigma }}_{\beta }^2)$, respectively, and that (μ̃_α and μ̃_β follow independent normal hyperpriors. For the variance hyperparameters ${\stackrel{\sim }{\sigma }}_{\alpha }^2$ and ${\stackrel{\sim }{\sigma }}_{\beta }^2$, following Gelman (2006, Section 4.3) we assume that σ̃_α and σ̃_β are uniform on [0, U_φ]. Denoting the dose assigned to the i^th patient by x_[i], in summary we assume

$$\begin{array}{llll}\text{Sampling}\text{model}\hfill & \hfill Y_{k,i}\mid {\mathit{\theta }}_k,x_{[i]}& ~\hfill & \text{Bernoulli}({\pi }_k(x_{[i]},{\mathit{\theta }}_k))\text{indep}.\text{for}\text{all}k\hfill \\ \text{Prior}\hfill & \hfill {\alpha }_k\mid {\stackrel{\sim }{\mu }}_{\alpha },{\stackrel{\sim }{\sigma }}_{\alpha }^2& ~\hfill & \text{N}({\stackrel{\sim }{\mu }}_{\alpha },{\stackrel{\sim }{\sigma }}_{\alpha }^2)\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \hfill & \hfill {\beta }_k\mid {\stackrel{\sim }{\mu }}_{\beta },{\stackrel{\sim }{\sigma }}_{\beta }^2& ~\hfill & \text{N}({\stackrel{\sim }{\mu }}_{\beta },{\stackrel{\sim }{\sigma }}_{\beta }^2)\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \text{Hyperpriors}\hfill & \hfill {\stackrel{\sim }{\mu }}_{\alpha }\mid {\mu }_{\alpha ,\phi },{\sigma }_{\alpha ,\phi }^2& ~\hfill & \text{N}({\mu }_{\alpha ,\phi },{\sigma }_{\alpha ,\phi }^2)\hfill \\ \hfill & \hfill {\stackrel{\sim }{\mu }}_{\beta }\mid {\mu }_{\beta ,\phi },{\sigma }_{\beta ,\phi }^2& ~\hfill & \text{N}({\mu }_{\beta ,\phi },{\sigma }_{\beta ,\phi }^2)\hfill \\ \hfill & \hfill {\stackrel{\sim }{\sigma }}_{\alpha },{\stackrel{\sim }{\sigma }}_{\beta }\mid U_{\phi }& ~\hfill & \text{U}(0,U_{\phi }).\hfill \end{array}$$ (6)

We will later, in Section 3.4, discuss how the ESS in this example depends not only on the assumed probability model and hyperparameters, but also on design choices like the adaptive dose-finding algorithm, the population proportions, etc. Note that we assume that (μ̃_α, ${\stackrel{\sim }{\sigma }}_{\alpha }^2$) and (μ̃_β, ${\stackrel{\sim }{\sigma }}_{\beta }^2$) are independent, in order to have a reasonably parsimonious model. We can use elicited information to solve for the hyperprior means μ_α,φ and μ_β,φ, as follows. Given the standardized doses, those hyperprior means are calculated based on the elicited values E{π_k (x₂ = −.403)} = .25 at the second lowest dose and E{π_k(x₅ = .513)} = .75 at the second highest dose for all the subgroups. These give μ_α,φ= −0.131 and μ_β,φ= 2.398. We will evaluate the ESS under several combinations of ( ${\sigma }_{\alpha ,\phi }^2,{\sigma }_{\beta ,\phi }^2$, U_φ) in a sensitivity analysis.

2.3 A Multicenter Randomized Trial

When analyzing data from a multicenter trial, it often is important to examine the inter-center variability of treatment effects, i.e., treatment-by-center interaction (Gray, 1994), since substantial variation among treatment effects across centers may cause a regulatory agency to question the generalizability of results obtained from such a trial before giving approval for a new therapy. As a third example, we consider a multicenter randomized clinical trial reported by Stangl (1996). The trial was carried out to examine the inter-center variability of the effect of imipramine hydrochloride for preventing the recurrence of depression. The primary outcome was time to the first recurrence of a depressive episode, denoted by T_jk,1, …,T_jk,njk for n_jk patients receiving treatment j at the k^th center, for j = 1,2 and k = 1,…, K. A total of 150 patients were enrolled in K = 5 centers. For each (j, k), the recurrence times T_jk,1, …, T_jk,njk were assumed to be i.i.d. exponentially distributed with recurrence rate θ_jk. Working with the transformed parameters ζ_k = log(θ_1k/θ_2k) and η_k = log(θ_2k)> the priors were assumed to be ${\zeta }_k~\text{i}.\text{i}.\text{d}.\text{N}({\stackrel{\sim }{\mu }}_{\zeta },{\stackrel{\sim }{\sigma }}_{\zeta }^2)$ and ${\eta }_k~\text{i}.\text{i}.\text{d}.\text{N}({\stackrel{\sim }{\mu }}_{\eta },{\stackrel{\sim }{\sigma }}_{\eta }^2)$. The hyperparameter ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ of the inter-center heterogeneity of the treatment effect log ratios is of primary interest in this example, while ${\stackrel{\sim }{\sigma }}_{\eta }^2$ represents the inter-center heterogeneity in the effect of the control treatment arm. Lognormal hyperpriors were assumed with ${\stackrel{\sim }{\sigma }}_{\zeta }~\text{LN}(m_{\phi },s_{\phi }^2)$, and σ_η ~ LN(−0.22,1), where LN(μ, σ²) denotes the lognormal distribution of e^X for X ~ N(μ, σ²). The model is summarized as follows:

$$\begin{array}{llll}\text{Sampling}\text{model}\hfill & \hfill T_{jk,i}\mid {\theta }_{jk}& ~\hfill & \text{Exp}({\theta }_{jk})\text{indep}.\text{for}j=1,2,\text{and}\text{all}k\hfill \\ \text{Priors}\hfill & \hfill {\zeta }_k\mid {\stackrel{\sim }{\mu }}_{\zeta },{\stackrel{\sim }{\sigma }}_{\zeta }^2& ~\hfill & \text{N}({\stackrel{\sim }{\mu }}_{\zeta },{\stackrel{\sim }{\sigma }}_{\zeta }^2)\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \hfill & \hfill {\eta }_k\mid {\stackrel{\sim }{\mu }}_{\eta },{\stackrel{\sim }{\sigma }}_{\eta }^2& ~\hfill & \text{N}({\stackrel{\sim }{\mu }}_{\eta },{\stackrel{\sim }{\sigma }}_{\eta }^2)\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \text{Hyperpriors}\hfill & \hfill {\stackrel{\sim }{\sigma }}_{\zeta }\mid m_{\phi },s_{\phi }^2& ~\hfill & \text{LN}(m_{\phi },s_{\phi }^2),\hfill \\ \hfill & \hfill {\stackrel{\sim }{\sigma }}_{\eta }& ~\hfill & \text{LN}(-0.22,1^2),\hfill \\ \hfill & \hfill {\stackrel{\sim }{\mu }}_{\zeta },{\stackrel{\sim }{\mu }}_{\eta }& ~\hfill & \text{U}(-20,20).\hfill \end{array}$$ (7)

Stangl assumed two alternative sets of hyperparameters (m_φ, $s_{\phi }^2$), to represent two types of prior belief on ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ in a Bayesian sensitivity analysis. The first choice was $(m_{\phi },s_{\phi }^2)=(-1.61,{0.50}^2)$, which places substantial prior belief on smaller ${\stackrel{\sim }{\sigma }}_{\zeta }^2$, and the second was $(m_{\phi },s_{\phi }^2)=(0,{0.50}^2)$, which places prior weight on larger ${\stackrel{\sim }{\sigma }}_{\zeta }^2$. We will evaluate the ESS of each prior on ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ under case 2 of our proposed methods.

Event time data, like the recurrence time, often includes extensive censoring. In the presence of censoring, the amount of information, and thus the ESS, depends on the number of observed events in addition to the sample size. The ESS computation in this example, therefore, needs to account for censoring cases which can occur depending on study duration. We will discuss details of the ESS computation in the presence of censoring and other relevant design details in Section 3.4.

In any CIHM, the prior choice is subject to two competing desiderata. On one hand, an informative hyperprior that expresses a belief of strong association among the θ_k’s is needed to borrow strength across subpopulations. In some settings, it is appropriate to use an informative prior that reflects accurate and comprehensive prior knowledge. If the hyperprior is elicited from an area expert, then the ESS provides an easily understood numerical value that the expert may use, if desired, to modify his/her original elicited values. On the other hand, in some settings it is necessary to avoid excessively informative priors that may compromise the objectivity of one’s conclusions. In practice, many arbitrary choices are made for technical convenience while formulating a model. A skeptical reviewer may like to quantify the prior information as being equivalent to a certain number of hypothetical observations, i.e., a prior ESS. Such a summary immediately allows a reader to judge the relative contributions of the prior and the data to the final conclusion.

3 Effective sample size in CIHMs

3.1 Prior Effective Sample Size in Non-hierarchical Models

In this subsection, we review and formalize the heuristic definition of ESS for non-hierarchical models given by MTM. We give formal definitions of the ε-information prior and a prior-to-posterior distance. The intuitive motivation for MTM’s method is to mimic the rationale for why the ESS of a beta distribution, Be(a,b), equals a + b. A binomial variable Y with binomial sample size n and success probability θ following a Be(a, b) prior implies a Be(a + Y, b + n − Y) posterior. Thus, saying that a given Be(a, b) prior has ESS m = a + b requires the implicit reasoning that the Be(a, b) may be identified with a Be(c + Y, d + m − Y) posterior arising from a previous Be(c, d) prior having a very small amount of information. A simple way to formalize this is to set c + d, = ε for an arbitrarily small value ε > 0 and solve for m = a + b − (c + d) = a + b − ε.

In a generic, non-hierarchical model, let f(Y | θ) denote the distribution of a random, possibly vector-valued outcome Y and π(θ | θ̃) the prior on the parameter vector θ = (θ₁, …, θ_d), with hyperparameters θ̃. The definition of the ESS of π(θ | θ̃) given f(Y | θ) requires the notions of an ε-information prior and the distance between the prior and a hypothetical posterior corresponding to a sample of a given size used to update an ε-information prior. The following definition formalizes the heuristic definition given by MTM. Denote ${\sigma }_{\pi ,j}^2={var}_{\pi }({\theta }_j)$.

Definition of epsilon-information prior

Let π(θ | θ̃) be a prior with ${\sigma }_{\pi ,j}^2<{\overline{V}}_j(\stackrel{\sim }{\mathit{\theta }})\le \infty$, where V̄_j(θ̃) is a fixed bound, for all j = 1,…, d. Given arbitrarily small ε > 0. the prior π₀(θ | θ̃₀) defined on the same domain and in the same parametric family as π(θ | θ̃) is an ε-information prior if E_π0(θ) = E_π(θ), Corr_π0(θ_j, θ_j′) = Corr_π(θ_j, θ_j′) for all j ≠ j′ and

$$\left\{\frac{{\overline{V}}_j(\stackrel{\sim }{\mathit{\theta }})-{\sigma }_{{\pi }_0,j}^2}{{\overline{V}}_j(\stackrel{\sim }{\mathit{\theta }})-{\sigma }_{\pi ,j}^2}\right\}\frac{{\sigma }_{\pi ,j}^2}{{\sigma }_{{\pi }_0,j}^2}<\epsilon \text{for}j=1,\dots ,d.$$ (8)

In the case V̄_j(θ̃) = ∞, condition (8) reduces to ${\sigma }_{\pi ,j}^2/{\sigma }_{{\pi }_0,j}^2<\epsilon$. Thus, in this case one may simply choose θ̃₀, subject to the constraints on the means and correlations, so that ${\sigma }_{{\pi }_0,j}^2$ is large enough to ensure that this inequality holds. When V̄_j(θ̃) < ∞, note that (8) can be written in the form $\{{\overline{V}}_j(\stackrel{\sim }{\mathit{\theta }})/{\sigma }_{{\pi }_0,j}^2-1\}<\epsilon \{{\overline{V}}_j(\stackrel{\sim }{\mathit{\theta }})/{\sigma }_{\pi ,j}^2-1\}$. Thus, in this case one should choose θ̃₀ so that ${\sigma }_{{\pi }_0,j}^2$ is sufficiently close to V̄_j(θ̃) from below to ensure this inequality For example, if π(θ | θ̃) is a beta distribution, Be(α̃,β̃), then ${\sigma }_{\pi }^2=\mu (1-\mu )/(\stackrel{\sim }{\alpha }+\stackrel{\sim }{\beta }+1)$ and V̄(θ̃) = μ(1 − μ), where μ = α̃/(α̃ + β̃). One may construct π₀(θ | θ̃₀)as a Be(α̃/c, β̃/c) by choosing c > 0 large enough so that 1/c < ε, since this implies that the above inequality holds.

Alternatively, a proper non-informative prior could be considered for π₀(θ | θ̃₀). However, this might not be appropriate for defining ESS. For example, the Jeffreys prior for a binomial success probability is $\text{Be}(\frac{1}{2},\frac{1}{2})$, which gives ESS values that are too large by an additive factor of 1 due to the implied sample size of 1.

To see how the definition works for θ of dimension > 1, consider a bivariate normal distribution with mean μ̃ = (μ̃₁, μ̃₂)′, variances ${\stackrel{\sim }{\sigma }}_1^2,{\stackrel{\sim }{\sigma }}_2^2$ and covariance σ̃₁₂. An ε-information prior is specified by using the same means μ̃ but variances $c_1^2{\stackrel{\sim }{\sigma }}_1^2$ and $c_2^2{\stackrel{\sim }{\sigma }}_2^2$ and covariance c₁c₂<σ̃₁₂ for arbitrarily large c₁ > 0 and c₂ > 0. As a practical guideline, we suggest choosing c large enough so that a further increase would not change the ESS by more than 0.1.

Given the likelihood $f({\mathbf{Y}}_n\mid \mathit{\theta })={\prod }_{i=1}^{n}f(Y_i\mid \mathit{\theta })$ of a i.i.d. sample Y_n = (Y₁, …, Y_n) and ε-information prior π₀(θ | θ̃₀), the posterior is

$${\pi }_n(\mathit{\theta }\mid {\stackrel{\sim }{\mathit{\theta }}}_0,{\mathbf{Y}}_n)\propto {\pi }_0(\mathit{\theta }\mid {\stackrel{\sim }{\mathit{\theta }}}_0)f({\mathbf{Y}}_n\mid \mathit{\theta }).$$

MTM define a distance between the prior π(θ | θ̃) and π_n(θ | θ̃₀, Y_n) obtained from a hypothetical sample Y_n of size n starting with π₀. To do this, MTM use the prior variance ${\sigma }_{\pi ,j}^2$ and the average posterior variance ${\sigma }_{{\pi }_0,j,n}^2$ under the ε-information prior. The average is with respect to the prior predictive distribution f(Y_n | θ̃) = ∫ f(Y_n | θ) π(θ | θ̃)dθ. Let ${\sigma }_{{\pi }_0,j,n}^2({\mathbf{Y}}_n)$ denote the posterior variance of θ_j conditional on Y_n, under the ε-information prior π₀. Then ${\sigma }_{{\pi }_0,j,n}^2=\int {\sigma }_{{\pi }_0,j,n}^2({\mathbf{Y}}_n)df({\mathbf{Y}}_n\mid \stackrel{\sim }{\mathit{\theta }})$. In cases where these variances cannot be computed analytically, we use approximations, denoted by ${\widehat{\sigma }}_{\pi ,j}^{-2}$ and ${\widehat{\sigma }}_{{\pi }_0,j,n}^{-2}$. For example, one could use the negative second derivatives of the log distributions, as in MTM. We define the distance between π and π_0,n to be

$$\mathrm{\Delta }(n,\pi ,{\pi }_0)=\sum _{j=1}^d\mid {\sigma }_{\pi ,j}^{-2}-{\sigma }_{{\pi }_0,j,n}^{-2}\mid .$$ (9)

When interest focuses on only one of the parameters, say θ_j, then we use $\mid {\sigma }_{\pi ,j}^{-2}-{\sigma }_{{\pi }_0,j,n}^{-2}\mid$ to define a distance between the marginal prior on θ_j versus the marginal posterior on θ_j under the ε-information prior and a sample of size n.

Definition of ESS in the non-CIHM case

The effective sample size (ESS) of a parametric prior π(θ | θ̃) for which ${\sigma }_{\pi ,j}^2<\infty$ for all j = 1, …, d, with respect to the likelihood f(Y_n | θ) is the interpolated integer n that minimizes the distance Δ(n, π, π₀).

Computation of Δ(n, π, π₀) is carried out either analytically or using a simulation-based numerical approximation.

3.2 Conditionally independent hierarchical models

We extend the definition of ESS to accommodate two-level CIHMs in a balanced study design with K subgroups each having sample size m, i.e., Y_k = (Y_k,1, ···, Y_k,m) for each k = 1, ···, K and the total sample size of = (Y₁, …, Y_K) is M = m × K. To accommodate the hierarchial structure, we propose the following two alternative definitions of ESS for CIHMs. Recall the discussion following (1). Under a CIHM interest may focus on θ₁, …, θ_K (case 1) or on θ̃ (case 2).

Case 1

When θ = (θ₁, …, θ_K) are the parameters of primary interest, the ESS is a function of the target marginal prior π₁₂(θ | φ) and the sampling model f(·). In this case, we constructively define an ε-information prior, π_12,0(θ | φ), as follows. First, specify an ε-information prior π_1,0(θ | θ̃) of π₁(θ | θ̃), as described in Section 3.1, and use this to define π_12,0(θ | φ) = ∫ π_1,0(θ | θ̃) π₂(θ̃ | φ)dθ̃. A proof that π_12,0 defined in this way is in fact an ε-information prior is given in the appendix.

An alternative way to define an ε-information prior for case 1 could be to first specify π₁₂(θ | φ) and then compute π_12,0. However, this approach is tractable only if π₁₂(θ | φ) can be specified analytically. Consequently, we do not use this alternative approach.

Given the likelihood $f({\mathcal{Y}}_M\mid \mathit{\theta })={\prod }_{k=1}^{K}f({\mathbf{Y}}_k\mid {\mathit{\theta }}_k)$ and π_12,0(θ | φ), we denote the hypothetical posterior by π_12,M(θ | φ, ) ∝ π_12,0(θ | φ)f( | θ). Denote θ̄₁₂ = E_π12(θ | φ), the prior mean of θ under π₁₂(θ | φ) and let = E( | θ̄₁₂) denote the prior mean of under θ̄₁₂. We will later discuss how to proceed in problems where is not meaningfully defined, due to censoring, the use of covariates, and other reasons. Let Σ_π12 denote the marginal variance-covariance matrix of θ under π₁₂, and similarly for Σ_π12,M ( ) under π_12,M(θ | φ, ). Often exact evaluation is not possible and approximations Σ̂_π12 and Σ̂_π12,M must be used. For example, one could use the negative inverse Hessian matrices of log(π₁₂) and log(π_12,M) evaluated at θ̄₁₂. Similarly to Δ(·) in equation (9) for a non-hierarchical model, in case 1 of a CIHM we define the distance between π₁₂(θ | φ) and π_12,M(θ | φ, ) to be

$${\mathrm{\Delta }}_1(M,{\pi }_1,{\pi }_2,{\pi }_{1,0})=\left|\mathit{det}({\mathrm{\sum }}_{{\pi }_{12}}^{-1})-\mathit{det}\{{\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\overline{\mathcal{Y}}}_M)\}\right|.$$ (10)

Definition of ESS for Case 1

The ESS of π₁₂(θ | φ) with respect to f(·), denoted by ESS₁₂(θ | φ), is the interpolated value of the sample size M minimizing Δ₁(M, π₁, π₂, π_1,0).

We use determinants in the definition of Δ₁ to incorporate the off-diagonal elements of the variance/covariance matrices and thereby account for association induced by the hyperprior among the parameters, θ₁, …, θ_K, which is a key aspect of a CIHM. The determinants quantify the total amounts of variability of the prior π₁₂ and the hypothetical posterior π_12,M. The same definition of distance could be used for non-hierarchical models in place of (9). However, the trace used in (9) is easier to evaluate and leads to exactly the same ESS in all cases that we considered. Although our choice seems to make sense intuitively, it still is arbitrary. We validate the definition by investigating important cases of CIHMs, given in Section 5 and the supplementary materials.

Case 2

When the hyperparameters θ̃ are of primary interest, the ESS is a function of the target prior π₂(θ̃ | φ) and the marginal likelihood (4). We define an ε-information prior, π_2,0(θ̃ | φ₀), and use the marginal likelihood f₁( | θ̃) to update π_2,0(θ̃ | φ₀) to obtain the hypothetical posterior π_2,M(θ̃ | φ₀, ). Let ${\overline{\stackrel{\sim }{\mathit{\theta }}}}_2={\text{E}}_{{\pi }_2}(\stackrel{\sim }{\mathit{\theta }}\mid \mathit{\phi })$ denote the prior mean of θ̃ under π₂(θ̃ | φ), and let ${\overline{\mathcal{Y}}}_{M,2}=E({\mathcal{Y}}_M\mid {\overline{\stackrel{\sim }{\mathit{\theta }}}}_2)$. We define a distance between π₂(θ̃ | φ) and π_2,M(θ̃ | φ₀, ) for sample size M as in the definition in (11), using the variance/covariance matrices Σ_π2 under the prior π₂(θ̃ | φ) and Σ_π2,M( ) under π_2,M(θ̃ | φ₀, ). When the variance/covariance matrices are not available in closed form, one can again use a numerical approximation.

Definition of ESS for Case 2

The ESS of π₂(θ̃ | φ) with respect to f_M( | θ̃), denoted by ESS₂(θ̃ | φ), is the interpolated value of M minimizing

$${\mathrm{\Delta }}_2(M,{\pi }_2,{\pi }_{2,0})=\left|\mathit{det}({\mathrm{\sum }}_{{\pi }_2}^{-1})-\mathit{det}\{{\mathrm{\sum }}_{{\pi }_{2,M}}^{-1}({\overline{\mathcal{Y}}}_{M,2})\}\right|.$$ (11)

Definition (11) is equivalent to the non-hierarchical ESS definition, simply because after marginalizing with respect to the group specific parameters θ_k the hierarchical model (1) reduces to a non-hierarchical model. If interest is focused on a subvector θ̃_s of θ̃, the ESS can be determined similarly in terms of the marginal hyperprior π₂(θ̃_s | φ).

An important aspect of the definition is that the ESS could be infinite when it is not possible to achieve a comparably informative posterior with any finite sample size. For example, recall the stylized normal/normal CIHM introduced after (1). Assume we fix γ̃² = 1, μ̃ ~ N(0, 0.01) and K = 10. In particular, we keep the number K of subpopulations fixed and consider an increasing number of samples per group. No matter how many observations, it is impossible to match the prior variance Var(μ̃) = 0.01 as the posterior variance under π_0,n. Formally, the minimization (11) has no solution. This is desireable. The informative prior μ̃ ~ N(0, 0.01) cannot be interpreted as the posterior under any hypothetical sample having finite sample size, because, in this example, the number of groups is fixed at K = 10. However, while no sample size of a hypothetical prior study under the same experimental design, i.e., fixed K, can justify the employed prior, it still is possible to report a meaningful finite ESS under an alternative experimental design. For example, one could report the required sample size of a hypothetical prior study with fixed m = 20, σ² = 1, and increasing K. In this example we find ESS = 2100 with K = 105.

3.3 Algorithm for ESS Computation

Let θ̄₁₂ denote the prior mean vector of θ under π₁₂(θ | φ), and let ${\overline{\stackrel{\sim }{\mathit{\theta }}}}_2$ denote the prior mean vector of θ̃ under π₂(θ̃ | φ). Let M_max = m_max × K be a positive integer multiple of K chosen so that it is reasonable to assume that ESS ≤ M_max.

Algorithm 1, ESS for Case 1

• Step 1Evaluate the target marginal prior π₁₂(θ | φ)
• Step 2Evaluate ε-information prior π_12,0(θ | φ)
• Step 3For each M = 0, K, 2K, …, M_max, compute Δ₁(M, π₁, π₂, π_1,0).
• Step 4ESS₁₂(θ | φ) is the interpolated value of M minimizing Δ₁(M, π₁, π₂, π_1,0).

When ${\mathrm{\sum }}_{{\pi }_{12}}^{-1}$ cannot be evaluated analytically, then one can use the negative Hessian of the log prior as approximation ${\widehat{\mathrm{\sum }}}_{{\pi }_{12}}^{-1}=-H_{{\pi }_{12}}({\overline{\mathit{\theta }}}_{12})$. The Hessian H_π12 is the matrix of second derivatives of the log(π₁₂) with respective to θ_j. The Hessian is evaluated at θ̄₁₂. When π₁₂(θ | φ) itself is not available in closed form, then we use the following simulation-based approximation. First, simulate a Monte Carlo sample θ̃⁽¹⁾, …, θ̃^(T) from π₂(θ̃ | φ) for large T, e.g. T = 100,000. Use the Monte Carlo average $T^{-1}{\sum }_{t=1}^T\frac{\partial }{\partial {\theta }_k}{\pi }_1(\mathit{\theta }\mid {\stackrel{\sim }{\mathit{\theta }}}^{(t)})$ in place of $\int \frac{\partial }{\partial {\theta }_k}{\pi }_1(\mathit{\theta }\mid \stackrel{\sim }{\mathit{\theta }}){\pi }_2(\stackrel{\sim }{\mathit{\theta }}\mid \mathit{\phi })d\stackrel{\sim }{\mathit{\theta }}$, and similarly for the second order partial derivatives. When θ̄₁₂ = E_π12(θ | φ) cannot be computed analytically, first simulate a Monte Carlo sample θ⁽¹⁾, …, θ^(T) from π₁₂(θ | φ) = ∫ π₁(θ | θ̃) π₂(θ̃ | φ)dθ̃ for large T, and compute the mean $T^{-1}{\sum }_{t=1}^T{\mathit{\theta }}^{(t)}$. Similarly, when ${\mathrm{\sum }}_{12,M}^{-1}({\overline{\mathcal{Y}}}_M)$ is not available in closed form one can use the negative Hessian of log π_12,M(θ̄₁₂ | , φ) the as a convenient approximation, ${\widehat{\mathrm{\sum }}}_{12,M}^{-1}({\overline{\mathcal{Y}}}_M)=-H_{{\pi }_{12,M}}({\overline{\mathit{\theta }}}_{12},{\overline{\mathcal{Y}}}_M)$. The posterior π_12,M is evaluated conditional on and the Hessian is evaluated at θ̄₁₂.

Algorithm 2, ESS for Case 2

• Step 1Specify π_2,0(θ̃ | φ₀).
• Step 2For each M = 0, K, 2K, …, M_max, compute Δ₂(M, π₂, π_2,0).
• Step 3ESS₂(θ̃ | φ) is the interpolated value of M minimizing Δ₂(M, π₂, π_2,0).

When ${\mathrm{\sum }}_{{\pi }_2}^{-1}$ or ${\mathrm{\sum }}_{{\pi }_2,M}^{-1}({\overline{\mathcal{Y}}}_{M,2})$ are not available in closed form we proceed similarly as before, using the negative Hessians to approximate ${\mathrm{\sum }}_{{\pi }_2}^{-1}$ and ${\mathrm{\sum }}_{{\pi }_2,M}^{-1}$. The posterior π_2,M is evaluated conditional on . The Hessian is evaluated at ${\overline{\stackrel{\sim }{\mathit{\theta }}}}_2$. In many cases, steps 2 and 3 may be repeated to compute ESS₂(θ̃ | φ) for subvectors of θ̃ that are of interest.

3.4 Numerical Evaluation of ESS

Equations (10) and (11) define a distance between prior and posterior as a difference of determinants of a prior precision matrix versus a posterior precision matrix. The posterior precision matrix is evaluated using the plug-in values in Δ₁ and in Δ₂, respectively. In some studies, however, the definition of or as an expectation is simply not meaningful. This is explained most easily by the following examples. Many phase I studies involve choosing doses for future patients as a function of the outcomes of earlier patients. In such settings, it is not meaningful to represent a sample of size M by a mean or . More generally, outcome adaptive designs for clinical studies make it difficult to meaningfully define . Similar difficulties arise when studies involve censored outcomes or regression on covariates.

In these cases, we use an extended definition of Δ₁ that replaces the plug in of by full prior predictive marginalization. Specifically, we replace $det\{\mid {\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\overline{\mathcal{Y}}}_M)\mid \}$ by an expectation of $det\{\mid {\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\mathcal{Y}}_M)\mid \}$, averaging over possible study realizations with M patients. The sampling scheme might include additional variables, like assigned dose, patient-specific covariates, or censoring times. Let x_i denote these additional variables for the i^th patient. Let g(x_i | Y₁, …, Y_i−1, x₁, …, x_i−1, θ̄₁₂) denote a probability model for x_i. When x_i is a deterministic function of the conditioning variables, we interpret g(·) as a degenerate distribution with a single point mass. For example, in the case of an adaptive dose escalation design, the dose for the i^th patient is a function of the earlier outcomes. Thus, the outcomes are not independent, making the interpretation of tricky. We replace use of the plug in by the expectation of $det\{\mid {\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\mathcal{Y}}_M)\mid \}$ with respect to the marginal model,

$$f_M({\mathcal{Y}}_M\mid {\overline{\mathit{\theta }}}_{12})=\int \prod _{i=1}^{M}f(Y_i\mid Y_1,\dots ,Y_{i-1},x_i,{\overline{\mathit{\theta }}}_{12})g(x_i\mid Y_1,\dots ,Y_{i-1},x_1,\dots ,x_{i-1},{\overline{\mathit{\theta }}}_{12})\text{d}{x}_1\cdots \text{d}{x}_M.$$ (12)

In words, f_M(·) is the marginal distribution of possible study outcomes when we carry out the study under a given design, and the prior mean θ̄₁₂. The predictive model (12) highlights the fact that ESS is defined in the context of an assumed experimental design. This includes details like the expected extent of censoring, distribution of covariates x_i, and more. Recall the stylized normal/normal CIHM stated below (1). One could consider two alternative experiments, either increasing the number of observations within each subpopulation, k = 1, …, K for fixed K, or alternatively one could consider increasing the number of groups K with a fixed number of observations per group. The two experiments naturally report different ESS values for the same prior, simply because the information that is being accrued with hypothetical future patients differs.

In the definition of the ESS, Case 1, for such studies we replace $det\{\mid {\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\overline{\mathcal{Y}}}_M)\mid \}$ in definition of Δ₁ by

$$\int det\{{\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\mathcal{Y}}_M)\}{f}_M({\mathcal{Y}}_M\mid {\overline{\mathit{\theta }}}_{12})d{\mathcal{Y}}_M$$ (13)

where Σ⁻¹(Y_M) = Var(θ | ) is the posterior variance under data and prior π₁₂. Similarly, we replace $det\{\mid {\mathrm{\sum }}_{{\pi }_{2,M}}^{-1}({\overline{\mathcal{Y}}}_{M,2})\mid \}$ in the definition of Δ₂ in Case 2 by

$$\int det\{{\mathrm{\sum }}_{{\pi }_{2,M}}^{-1}({\mathcal{Y}}_M)\}{f}_{M,2}({\mathcal{Y}}_M\mid {\overline{\stackrel{\sim }{\mathit{\theta }}}}_2)d{\mathcal{Y}}_M$$ (14)

where

$$f_{M,2}({\mathcal{Y}}_M\mid {\overline{\stackrel{\sim }{\mathit{\theta }}}}_2)=\int \int \prod _{i=1}^{M}f(Y_i\mid Y_1,\dots ,Y_{i-1},x_i,\mathit{\theta })g(x_i\mid Y_1,\dots ,Y_{i-1},x_i,\dots ,x_{i-1},\mathit{\theta })\text{d}{x}_1\cdots \text{d}{x}_M{\pi }_1(\mathit{\theta }\mid {\overline{\stackrel{\sim }{\mathit{\theta }}}}_2)\text{d}\mathit{\theta }.$$ (15)

For the standard CIHMs discussed later, in §5, the definitions using (13) instead of $det\{\mid {\mathrm{\sum }}_{{\pi }_{12,M}}^{-1}({\overline{\mathcal{Y}}}_M)\mid \}$ and (14) in place of $det\{\mid {\mathrm{\sum }}_{{\pi }_{2,M}}^{-1}({\overline{\mathcal{Y}}}_{M,2})\mid \}$ in the definitions of Δ₁ and Δ₂, respectively, give the same ESS as the original definitions. That is why we recommend using the simplified definitions (10) and (11) when possible. We recommend using the extended definitions only for problems where the definition of or is not meaningful.

Algorithms 1 and 2 remain unchanged. The only additional detail is the evaluation of Δ₁ and Δ₂. For Δ₁, we use numerical evaluation of the integral (13).

Algorithm 3, Numerical Evaluation of Δ₁

Evaluation of (13) as a Monte Carlo integral.

Repeat L iterations of Steps 1–6:

Step 1: Generate Y1 ~ f(Y1 | x1, θ̄12)

Repeat for i = 2, …, M:

Step 2: Evaluate g(xi | Y1, …, Yi−1, xi, …, xi−1, θ̄12). If g(·) is a deterministic function (as in dose allocation), record the value as xi. If g(·) is a non-degenerate distribution generate xi accordingly.

Step 3: Generate Yi ~ f(Yi | Y1, …, Yi−1, xi, θ̄12), using xi from Step 2.

Step 4: The vector = (Y1, …, YM) is a simulated outcome vector.

Step 5: Evaluate the posterior variance covariance matrix Σ(θ | ). This evaluation might require Markov chain Monte Carlo integration when the model is not conjugate.

Step 6: Evaluate det(Σ−1( )).

Step 7: The average over all L iterations of steps 1 through 6.

$$\frac{1}{L}\sum det({\mathrm{\sum }}^{-1}({\mathcal{Y}}_M))\approx \int det\{{\mathrm{\sum }}^{-1}({\mathcal{Y}}_M)\}{f}_M({\mathcal{Y}}_M\mid {\overline{\mathit{\theta }}}_{12}){dY}_M,$$ (16)

approximates the desired integral. The sum is over the L repeat simulations of Steps 1 through 6, plugging in the vector that is generated in each step of the simulation.

With a minor variation, the same algorithm can be used to evaluate (14). Step 1 is replaced by

Step 1′: Generate $\mathit{\theta }~{\pi }_1(\mathit{\theta }\mid {\overline{\stackrel{\sim }{\mathit{\theta }}}}_2)$ and generate Y₁ ~ f(Y₁ | x₁, θ).

The rest of the algorithm for computing ESS remains unchanged, with θ replacing θ̄₁₂. It is important that a new value θ is generated for each of the L repetitions of steps 1 through 6. This approximates the outer integral with respect to θ.

4 Application to the motivating examples

4.1 ESS for the sarcoma trial

The goal of the sarcoma trial was to estimate the efficacy of a molecularly targeted drug within each disease subtype. Recall that ξ_k was the response probability in subtype k, and θ_k = log{ξ_k/(1 − ξ_k)}. The likelihood for M patients with m patients in each subtype having response indicators Y_k,m = (Y_k1, …, Y_km) is

$$f({\mathcal{Y}}_M\mid \mathit{\theta })\propto \prod _{k=1}^K\prod _{i=1}^m{\xi }_k{({\theta }_k)}^{Y_{ki}}{\{1-{\xi }_k({\theta }_k)\}}^{1-Y_{ki}}.$$ (17)

For this example, based on the normal-gamma hierarchical prior described in Section 2, a closed form of the target prior in case 1 can be found analytically, as we will show in Section 5. Based on this prior and the likelihood (17), applying Algorithm 1 of subsection 3.3 gives ESS₁₂(θ | φ_μ̃ = (−1.386, 10), φ_γ̃ = (2, 20)) = 2.6. This ESS value is reasonable, since a separate early stopping rule was applied for each subtype in this study. For comparison, if one assumes instead that φ_μ̃ = (−1.386, 1) and φ_γ̃= (20, 1), then ESS₁₂(θ | φ_μ̃) = 702.2. This illustrates the important practical point that a seemingly reasonable choice of ${\mathit{\phi }}_{\stackrel{\sim }{\mu }}=({\mu }_{\phi },{\tau }_{\phi }^2)$ and φ_γ̃ = (a_φ, b_φ) may give an excessively informative prior. Figure 1 gives plots of the ESS₁₂ values as a function of ${\tau }_{\phi }^2$ for each of the four pairs φ_γ̃ = (2, 20), (5, 20), (10, 20), and (10, 10), when μ_φ = −1.386. Figure 1 shows that, as the variance parameter ${\tau }_{\phi }^2$ in the hyperprior of μ̃ gets larger, the ESS₁₂ values decrease, which is intuitively correct. Similarly, ESS₁₂ increases with the prior mean of the precision parameter γ̃ under π₂(γ̃ | φ_γ̃).

Figure 1.

Plots of ESS₁₂ as a function of variance parameter ${\tau }_{\phi }^2$ in the hyperprior of μ̃ for the four sets of φ_γ̃ = (2, 20), (5, 20), (10, 20), and (10, 10), when μ_φ = −1.386.

If one wishes to focus, instead, on the overall mean treatment effect across all disease subtypes, (case 2), it leads to consideration of the hyperprior π₂(μ̃ | φ_μ̃). The ESS in this case may be computed using Algorithm 2 of subsection 3.3, which gives ESS₂(μ̃ | φ) = 3.7. We conclude that the hierarchical prior used by Thall et al. (2003) is reasonable under both points of view, either when one is concerned with inference on success probabilities for the sarcoma subtypes, or when one is interested in the average response probability for all sarcomas.

4.2 A CRM Dose-finding Trial for Multiple Patient Subgroups

In the second example, the goal of the CRM-type dose-finding study is to find the optimal dose in each subgroup. Recall that the probability of toxicity in each subgroup is modeled by π_k(x_i, θ_k) with logit{π_k(x_i, θ_k)} = α_k + β_kx_i, where θ_k = (α_k, β_k). Let m_1, m₂, m₃, m₄ denote the number of patients in the four subgroups, let x_k,_[i] denote the dose assignment for the i^th patient in subgroup k, and let D_mk = {Y_k,1, x_k,[1], …, Y_k,mk, x_k,[mk]} denote the dose assignments and the responses observed in subgroup k. The likelihood for all $M={\sum }_{k=1}^4{m}_k$ patients is

$$f({\mathcal{Y}}_M\mid {\mathbf{x}}_M,\mathit{\theta })\propto \prod _{k=1}^4\prod _{i=1}^{m_k}{\pi }_k{(x_{k,[i]},{\mathit{\theta }}_k)}^{Y_{k,i}}{\{1-{\pi }_k(x_{k,[i]},{\mathit{\theta }}_k)\}}^{1-Y_{k,i}}.$$ (18)

Following the extended ESS definition given in Section 3.4, we compute ESS values under several combinations of ( ${\sigma }_{\alpha ,\phi }^2,{\sigma }_{\beta ,\phi }^2$, U_φ) for ${\sigma }_{\alpha ,\phi }^2={\sigma }_{\beta ,\phi }^2$ using Algorithm 3. Dose levels are determined on the basis of the observed data. As mentioned before, ESS is always defined in the context of an assumed experimental design. In the case of this study, the assumed relative accrual rates (0.4, 0.3, 0.2, 0.1) are part of the design, and are used in the evaluation of ESS.

Posterior summaries are evaluated using Markov chain Monte Carlo posterior simulation (with three parallel chains of length 5,000 with a burn-in of 500). The posterior means at the six dose levels in each subgroup are required in the dose-escalation/de-escalation algorithm for the respective new cohort. Table 1 summarizes the results of the computations, including the subgroup ESS values based on the assumed relative accrual rates (0.4, 0.3, 0.2, 0.1). Those computations suggest that, considering the maximum sample size of 36, the prior choices that are summarized in the first two rows of the table are defensible. The implied ESS is reasonably small compared to the sample size, since in these cases the per-subgroup sample size is less than 2. The prior in the third row may or may not be considered suitably non-informative for a dose-finding trial, but the prior in row 4 clearly is far too informative. Without computing ESS, it would be very difficult to make such determinations.

Table 1.

ESS values computed for several sets of level 2 prior parameters ( ${\sigma }_{\alpha ,\phi }^2,{\sigma }_{\beta ,\phi }^2$, U_φ). The per-subgroup ESS values are based on assumed relative accrual rates .40, .30, .20, .10 for k = 1, 2, 3, 4.

${\sigma }_{\alpha ,\phi }^2={\sigma }_{\beta ,\phi }^2$	Uφ	ESS	Per-subgroup ESS
			1	2	3	4
100	5	6.6	2.6	2.0	1.3	0.7
25	5	7.1	2.8	2.1	1.4	0.7
4	5	9.1	3.6	2.7	1.8	0.9
25	2	25.8	10.3	7.7	5.2	2.6

Computation of frequentist operating characteristics (OCs) by computer simulation, and calibration of prior hyperparameters as well as design parameters on that basis, has become standard practice in Bayesian clinical trial design. An important question is how computing the prior’s ESS can improve this process. This is especially important logistically because obtaining both prior ESS and the design’s OCs are computationally intensive. However, computing an ESS takes even less time than the time required for OCs under only one scenario. This is the case because in contrast to OC simulation, the evaluation of ESS does not require the simulation of complete trial histories. Also, in practice, after establishing hyperparameters that determine prior mean values, one guesses numerical values of second order hyperparameters, including variances and correlations, simulates the trial using that prior, and examines the resulting OCs. Since the numerical values of prior hyper-variances have no meaning per se, in the absence of ESS, one must rely on the OCs alone to determine whether the prior is acceptable and reasonable. This process often is repeated many times, until satisfactory choices are found, and is quite time-consuming and tedious. When the ESS of each prior is computed, however, it may be used to calibrate the hyper-variances so that they give a reasonable ESS, before running any simulations of complete trial histories. This saves a great deal of time because, as noted above, computing an ESS is far less time-consuming than running simulations.

Aside from obtaining a design’s OCs, because the ESS is readily interpretable, it is a useful tool for deciding whether a prior is reasonable per se. Recall that the ESS is a property of the prior, likelihood, and experimental design. In contrast, obtaining the OCs of a given design also requires the specification of particular scenarios (fixed outcome probabilities) under which the design is evaluated. In the absence of ESS, it is possible to obtain OCs that look reasonable when in fact the prior is undesirably informative. That is, one may unknowingly be misled by simulation results. This is especially worrying when the clinical outcome is a complex function of treatment, as in a dose-response setting. As an example, consider the CRM dose-finding trial with heterogeneous subgroups under the prior with level 2 prior parameters ${\sigma }_{\alpha ,\phi }^2={\sigma }_{\beta ,\phi }^2=25$ and U_φ = 2 in the last row of Table 1. Recall that the ESS = 25.8 for that prior. A simulation of the trial assuming these level 2 hyper-prior parameters is summarized in Table 2, which shows that, despite the overly informative prior in terms of ESS, the subgroup-specific CRM design is most likely to choose the right doses in subgroups 1, 2, and 3. In contrast, the dose selection is clearly wrong in subgroup 4, since the highest dose level would be most desirable in that subgroup. In the assumed scenario, the dose-toxicity curve in subgroup 4 is qualitatively different from the other subgroups, with true π₄(x_i) = .10 or .20 for all doses, well below the target of .30. Since subgroup 4 has the lowest accrual rate of 10% and thus accrues on average .10 × 36 = 3.6 patients, the data from the trial cannot overcome the overly informative prior (ESS = 2.7) in this subgroup. Based on the ESS, however, one would never assume the prior with ${\sigma }_{\phi }^2=25$ and U_φ = 2. This example shows how use of the prior ESS complements the evaluation of OCs. In summary, in the context of Bayesian clinical trial design, prior ESS is both a tool for calibrating hyper-variances and thus speeding up the process of simulation to obtain the design’s OCs, and also a simple summary statistic that helps investigators decide whether a prior is acceptable.

Table 2.

Simulation study of the CRM dose-finding trial with heterogeneous subgroups assuming the prior with level 2 prior parameters ${\sigma }_{\alpha ,\phi }^2={\sigma }_{\beta ,\phi }^2=25$ and U_φ = 2. The percentage of times that the method selected each dose level as the final MTD in each subgroup (%recommendation) and the numbers of patients who were treated at each dose level (No. of patients) out of 36 patients are summarized. Correct selections are marked in bold-face.

Subgroup	Dose level	d1	d2	d3	d4	d5	d6
1	True prob. tox	.05	.10	.15	.30	.50	.65
	%recommendation	.05	.16	.30	.36	.07	.04
	No. of patients	2.1	3.1	4.5	3.6	1.1	.4
2	True prob. tox	.10	.20	.30	.45	.60	.70
	%recommendation	.08	.26	.39	.24	.02	.01
	No. of patients	1.8	2.9	3.3	2.0	.6	.1
3	True prob. tox	.15	.30	.45	.55	.65	.75
	%recommendation	.12	.34	.32	.16	.02	.01
	No. of patients	1.2	2.2	1.9	1.0	.3	.1
4	True prob. tox	.10	.10	.10	.20	.20	.20
	%recommendation	.05	.20	.32	.31	.09	.02
	No. of patients	.5	.9	1.0	.8	.2	.1

4.3 ESS for a Multicenter Randomized Trial

In the third example, recall that θ_1k and θ_2k denote the affective disorder recurrence rates in the two treatments arms for trial center k, and these are reparameterized as ζ_k = log(θ_1k/θ_2k) and η_k = log(θ_2k). Recall that T_jk,i is the time to the event for patient i receiving treatment j at center k. Denote the observed time to an event at T or right-censoring by T^o and δ = I[T^o = T]. Let ${\mathcal{T}}_M^o$ and δ_m denote the vectors of all event observation times and indicators in the sample of M = m × K patients. The likelihood under the exponential model is

$$f({\mathcal{T}}_M^o,{\mathit{\delta }}_M\mid \mathit{\theta })=\prod _{k=1}^K\prod _{j=1}^2\prod _{i=1}^{m_{j,k}}{\{{\theta }_{jk}\mathit{exp}(-{\theta }_{jk}{T}_{jk,i})\}}^{{\delta }_{\mathit{jki}}}{\{\mathit{exp}(-{\theta }_{jk}{T}_{jk,i}^o)\}}^{1-{\delta }_{\mathit{jki}}},$$ (19)

where θ_1k = exp(ζ_k + η_k) and θ_2k = exp(η_k). Recall that the hyperparameter ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ characterizing inter-center variability is of primary interest (Case 2). Based on the hierarchical model described in Section 2 and the likelihood (19), we use Algorithm 2 to compute ${\text{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi },s_{\phi }^2)$, which are ${\text{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi }=-1.61,s_{\phi }^2={0.5}^2)=27.6$ and ${\text{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi }=0,s_{\phi }^2={0.5}^2)=9.1$. The two ESS values indicate that the first prior may be too informative, since the total sample size of the trial is 150. One may obtain a less informative hyperprior for ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ by specifying larger $s_{\phi }^2$. For example, $s_{\phi }^2={0.75}^2$, 1.0², and 1.25² give ${\text{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi }=0,s_{\phi }^2)=5.0$, 1.4, and 0.4. Table 3 summarizes the ${\text{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi },s_{\phi }^2)$ values computed for several sets of (m_φ, $s_{\phi }^2$) pairs.

Table 3.

${\mathit{ESS}}_2({\stackrel{\sim }{\sigma }}_{\zeta }^2\mid m_{\phi },s_{\phi }^2)$ values computed for several sets of (m_φ, $s_{\phi }^2$), where ${\stackrel{\sim }{\sigma }}_{\zeta }^2$ represents the inter-center heterogeneity of the treatment effect, and m_φ and $s_{\phi }^2$ respectively denote the mean and variance parameters of the lognormal hyperprior of σ̃_ζ.

mφ	$s_{\phi }^2$	ESS2
−1.61	0.52	27.6
−1.61	0.752	12.1
−1.61	1.02	8.4
−1.61	1.252	6.6
−0.5	0.52	15.5
−0.5	0.752	8.0
−0.5	1.02	5.7
−0.5	1.252	2.1
0.0	0.52	9.1
0.0	0.752	5.0
0.0	1.02	1.4
0.0	1.252	0.4

As we mentioned in Section 2.3, the ESS computation in this example accounts for censoring in the process of numerical computation of the distance between the prior and posterior. For the above computations, we set-up the study duration at 2 years, as it was done in Stangl’s study design. That is, patients are followed for 2 years, or until they experience the event. If patients do not experience the event by the end of the study, they are treated as censored cases. Given the event rates in the two treatment groups, the proportion of censored cases goes up as the study duration becomes shorter. When the study duration is 2 years, the ESS is computed to be 27.6, as shown in the first row of Table 3. When the study duration is 10 years, the ESS value is computed to be 24.5, while the ESS is 36.3 in case the duration is 0.5 years. That is, as the proportion of censored cases rises up, the ESS goes up, and vice-versa.

From Figure 7 of Stangl (1996), it can readily be seen by inspection that prior 1 is too informative by comparing the posterior and prior of the 3rd stage variances. The ESS formalizes this judgment by quantifying the informativeness of prior 1. Moreover, the ESS validates prior 2 as being reasonable, whereas this in not obvious using only a graphical evaluation.

5 Validation in Some Standard CIHMs

The aim of the proposed ESS definitions is to provide an easily interpretable way to quantify the prior informativeness of commonly used CIHMs. The definition of ESS obtained by matching a given prior with a posterior distribution under an ε-information prior could be argued to be a natural choice. In order to formalize this, however, many detailed technical choices must be made. The definitions that we have proposed here are two of many possible formalizations. We arrived at the definitions after considering many alternatives, and evaluating the performance of each for several widely used CIHMs. The examples (one is in this section and two are in the supplementary materials) report the implied ESS for these models. As standard CIHMs, we use hierarchical extensions of standard models listed in Chapter 5 of Congdon (2005). In this section, we show how to compute the ESSs in cases 1 and 2 analytically in the normal CIHM with known sampling variance, and we argue that the obtained ESSs are sensible. In the supplementary materials, we provide computational details for a hierarchical model for binary responses recorded over different subpopulations as in the sarcoma trial of Section 2.1, and also for an alternative hierarchical model for the multicenter randomized trial analysis given in Section 2.3, assuming an exponential sampling model for the observed recurrence times.

Perhaps the most widely used CIHM is a hierarchical model with normal sampling model and conjugate prior and hyperprior. A typical example of such a CIHM is discussed in Gelman et al. (1995, Section 5.5), for a study of special coaching programs to prepare students for the scholastic aptitude test (SAT). The observed outcomes are SAT scores, where Y_ki denotes score of the i-th student in school k. The scores are assumed to be normally distributed with school specific means θ₁, …, θ_k and known variance σ². The model is completed with conjugate hyperpriors, as follows:

$$\begin{array}{llll}\text{Sampling}\text{model}\hfill & \hfill Y_{k,1},\cdots Y_{k,m}\mid {\theta }_k,{\sigma }^2& ~\hfill & \text{N}({\theta }_k,{\sigma }^2)\text{indep}.\text{for}\text{all}k\hfill \\ \text{Prior}\hfill & \hfill {\theta }_k\mid \stackrel{\sim }{\mu },{\stackrel{\sim }{\gamma }}^2& ~\hfill & \text{N}(\stackrel{\sim }{\mu },{\stackrel{\sim }{\gamma }}^2)\text{i}.\text{i}.\text{d}.\text{for}\text{all}k\hfill \\ \text{Hyperpriors}\hfill & \hfill \stackrel{\sim }{\mu }\mid {\mu }_{\phi },{\tau }_{\phi }^2& ~\hfill & \text{N}({\mu }_{\phi },{\tau }_{\phi }^2)\hfill \\ \hfill & \hfill {\stackrel{\sim }{\gamma }}^2\mid {\nu }_{\phi },S_{\phi }& ~\hfill & \text{Inv}-{\chi }^2({\nu }_{\phi },S_{\phi }).\hfill \end{array}$$

Thus, for K schools, μ̃ represents an overall effect and γ̃² represents inter-school variability. The fixed hyperparameters are ${\mathit{\phi }}_{\stackrel{\sim }{\mu }}=({\mu }_{\phi },{\tau }_{\phi }^2)$ and φ_γ̃² = (ν_φ, S_φ). Let $r={\tau }_{\phi }^2/({\stackrel{\sim }{\gamma }}^2+{\tau }_{\phi }^2)$ denote the intra-class correlation. In this model the ESS in both cases 1 and 2 can be found analytically, because both the marginal prior π₁₂ and the posterior π_12,M are multivariate normal, thus the information matrices of the prior and posterior can be obtained analytically as the inverses of their variance-covariance matrices. Details are given in the supplementary materials.

Case 1

When the main goal is inference on the school-specific SAT scores θ₁, …, θ_K, assuming fixed prior variance γ̃²,

$${\text{ESS}}_{12}(\mathit{\theta }\mid {\mathit{\phi }}_{\stackrel{\sim }{\mu }})=\frac{{\sigma }^2}{{\stackrel{\sim }{\gamma }}^2}{\left(\frac{1-r}{1+(K-1)r}\right)}^{1/K}\times K.$$ (20)

This formula confirms what one would expect intuitively. The precision 1/γ̃² of the conjugate normal prior for each school is equivalent to σ²/γ̃² observations (SAT scores). Thus K independent priors for θ₁, …, θ_K, corresponding to r = 0, would be equivalent to Kσ²/γ̃² observations. The second factor in (20) is obtained analytically from the determinant of the information matrix of π₁₂ for the normal CIHM with conjugate priors (computational details are given in the supplementary materials). This factor accounts for the dependence among θ₁, …, θ_K in the two-level prior. Given K, as the correlation among θ₁, …, θ_K increases, this factor reduces the ESS₁₂. That is, as r → 1, the ESS₁₂ → 0. In particular, for K = 2 schools, ${\text{ESS}}_{12}(\mathit{\theta }\mid {\mathit{\phi }}_{\stackrel{\sim }{\mu }})=\frac{{\sigma }^2}{{\stackrel{\sim }{\gamma }}^2}{\left(\frac{1-r}{1+r}\right)}^{1/2}\times 2$. Figure 2 shows a plot of ESS₁₂(θ | φ_μ̃) as a function of r and γ̃² for K = 5 and σ² = 1.0.

Figure 2.

Plots of ESS₁₂(θ | φ_μ̃) as a function of prior variance γ̃² and r for the normal/inverse χ²-normal-normal model, when K = 5 and σ² = 1.0.

To validate the plausibility of (20), we consider three limiting cases, each of which leads to a non-hierarchical model having an obvious prior ESS. (i) Given γ̃², as r → 0, i.e. as ${\tau }_{\phi }^2\to 0$, which is the limiting case with K independent schools, ESS₁₂ → K × σ²/γ̃² and σ²/γ̃² is the ratio of the data variance to the prior variance, which is the prior ESS of a non-hierarchical model (Spiegelhalter et al. 1994, Section 3.1.2; MTM, Section 5, Example 3). (ii) In contrast, given γ̃², as ${\tau }_{\phi }^2\to +\infty$, and consequently r → 1, and there is one common effect θ = θ₁ = … = θ_k, i.e. this limiting case is a non-hierarchical model where schools are ignored. The marginal prior variance ${\stackrel{\sim }{\gamma }}^2+{\tau }_{\phi }^2$

for the common effect diverges, however, which is reflected by the fact that ESS₁₂ → 0, as desired, (iii) On the other hand, consider the case where ${\tau }_{\phi }^2$ is fixed and γ̃² → 0. Again, the model reduces to one common effect θ = θ₁ = … = θ_K, but now with finite marginal prior variance ${\tau }_{\phi }^2$. In this case, to learn with certainty that all effects are equal would require extremely reliable data, i.e., a very large number of students in each school, which is (correctly) reflected by the fact that ESS₁₂(θ | φ_μ̃) → +∞. If we allow γ̃² to be random, ESS₁₂(θ | φ) takes the same form as expression (20), but with γ̃² replaced by its prior mean μ_γ̃². (see Supplementary Materials, section 1).

Case 2

If the primary focus is the population parameter μ̃, averaging across all schools,

$${\text{ESS}}_2(\stackrel{\sim }{\mu }\mid \mathit{\phi })=\frac{{\sigma }^2}{{\tau }_{\phi }^2}{\left(1-\frac{{\mu }_{{\stackrel{\sim }{\gamma }}^2}}{{\tau }_{\phi }^{2}K}\right)}^{-1},$$ (21)

under the restriction that ${\tau }_{\phi }^2>K^{-1}{\mu }_{{\stackrel{\sim }{\gamma }}^2}$. An explanation of this restriction is given in the supplementary materials, Section 1 (Normal/Inverse χ²-Normal-Normal Model).

6 Discussion

We have proposed two practical definitions for the prior ESS of a conditionally independent hierarchical model. The main limitations are the need for numerical evaluations in many cases and the use of several arbitrary choices in the definitions. The arbitrary choices in the definition include the construction of an ε-information prior, evaluation of the posterior variance/covariance matrices conditional on and , and evaluation of the distances Δ₁ and Δ₂ at the prior means θ̄₁₂ and ${\overline{\stackrel{\sim }{\mathit{\theta }}}}_2$. The definitions of the two distances, based on the determinants of the information matrices, are reasonable but arbitrary. Alternative choices could be investigated. For example, one could use L₂ distance after a transformation of the variance-covariance matrix to some suitable standard form. E.g. the Cayley transformation maps the variance-covariance matrix Σ to the orthogonal matrix (I − Σ)(I + Σ)⁻¹. The main strengths of the method are the constructive nature of the definitions, and validation by matching prior ESS values obtained by the method with intuitively correct prior ESS values in special but important cases.

One of the reasons for the big success of hierarchical models, in particular, in inference for biomedical studies is the following feature. The hierarchical model constructs an informative prior for any given subgroup by borrowing strength from other subgroups and using the information contained in the data. If this feature is important, then the investigator might wish assurance that the hyperprior should not be excessively informative. The proposed ESS facilitates such judgements.

An important area of practical application for the proposed prior ESS summaries is design and inference for early phase clinical trials with small to moderate sample sizes. A concern about inappropriately influential prior information is one of the main impediments to a more widespread use of Bayesian methods in clinical trials. With the ever growing pressure for efficient use of resources and higher ethical standards, clinical trial designs are becoming increasingly more complex. Trial designs now routinely include the use of multiple outcomes, borrowing strength across different disease sub-types or patient subpopulations, the use of biomarkers, adaptive allocation, sequential stopping, etc. Such complexity may lead naturally to the use of a CIHM. The ability to compute prior ESS in such models provides a natural basis for calibrating priors as well as explaining the model to non-statisticians. Prior ESS values provide a similar tool when using a CIHM in a meta-analysis, which is an area of extensive activity. The two case studies in Section 2 are typical examples. It may be argued that, without an understanding of the prior ESS, it is impossible to carry out a fair regulatory review of a clinical trial protocol using a method based on a Bayesian CIHM. The proposed methodology may be used to mitigate this concern.

Overall, we believe that reporting ESS is most useful in judging the relative role of the prior, either informative or non-informative, in a clinical trial design, and as a tool to greatly facilitate the process of design simulation to calibrate the prior and obtain OCs. In this setting, both investigators and regulators may be very concerned that the trial conduct should not be overly biased by (sometimes inappropriately) optimistic priors. The ESS also is helpful to report and interpret the results of a data analysis, since comparing the ESS to the data sample size allows one to judge the prior’s relative informativeness.

Appendix

In Case 1 of a CIHM, the fact that constructing an ε-information prior π_1,0(θ | θ̃) for π₁(θ | θ̃) ensures that π_12,0(θ | φ) = ∫ π_1,0(θ | θ̃)π₂(θ̃ | φ)dθ̃ is an ε-information prior for π₁₂(θ | φ) can be proved as follows:

Assume for simplicity that V̄_j = ∞; the proof in the case V̄_j < ∞ is similar. Accounting for the variability at both levels of the CIHM, the variance of each θ_j can be written in the expanded form ${\sigma }_{{\pi }_{12}}^2({\theta }_j)=E_{{\pi }_2}\{{\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})\}+{\sigma }_{{\pi }_2}^2\{E_{{\pi }_1}({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})\}$. Similarly, using π_1,0 in place of π₁, ${\sigma }_{{\pi }_{12,0}}^2({\theta }_j)=E_{{\pi }_2}\{{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}+{\sigma }_{{\pi }_2}^2\{E_{{\pi }_{1,0}}({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}$. Since the definition of ε-information prior ensures that E_π1,0(θ_j|θ̃₀) = E_π1(θ_j|θ̃), the second terms of the two expansions are identical. Denoting this common term by c_j, and also denoting $a_j=E_{{\pi }_2}\{{\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})\}$ and $b_j=E_{{\pi }_2}\{{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}$, the ratio of variances may be written as

$$\frac{{\sigma }_{{\pi }_{12}}^2({\theta }_j)}{{\sigma }_{{\pi }_{12,0}}^2({\theta }_j)}=\frac{E_{{\pi }_2}\{{\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})\}+c_j}{E_{{\pi }_2}\{{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}+c_j}=\frac{a_j+c_j}{b_j+c_j}.$$ (22)

Given θ̃, under the assumption V̄_j = ∞ one can choose θ̃₀ so that $c_j/{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)<\epsilon /2$ and ${\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})/{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)<\epsilon /2$. Writing these inequalities as $c_j-{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\epsilon /2<0$ and ${\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})-{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\epsilon /2<0$, taking the expectations with respect to π₂ and reversing the algebra in each inequality, it follows that $c_j/E_{{\pi }_2}\{{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}=c_j/b_j<\epsilon /2$ and $E_{{\pi }_2}\{{\sigma }_{{\pi }_1}^2({\theta }_j\mid \stackrel{\sim }{\mathit{\theta }})\}/E_{{\pi }_2}\{{\sigma }_{{\pi }_{1,0}}^2({\theta }_j\mid {\stackrel{\sim }{\mathit{\theta }}}_0)\}=a_j/b_j<\epsilon /2$. Writing the right-hand side of (22) as (a_j/b_j + c_j/b_j)/(1 + c_j/b_j), since c_j/b_j > 0 this is bounded above by a_j/b_j + c_j/b_j, which by the foregoing is bounded above by ε/2 + ε/2 = ε.