Frequentist accuracy of Bayesian estimates

Abstract

In the absence of relevant prior experience, popular Bayesian estimation techniques usually begin with some form of “uninformative” prior distribution intended to have minimal inferential influence. Bayes rule will still produce nice-looking estimates and credible intervals, but these lack the logical force attached to experience-based priors and require further justification. This paper concerns the frequentist assessment of Bayes estimates. A simple formula is shown to give the frequentist standard deviation of a Bayesian point estimate. The same simulations required for the point estimate also produce the standard deviation. Exponential family models make the calculations particularly simple, and bring in a connection to the parametric bootstrap.

1 Introduction

The past two decades have witnessed a greatly increased use of Bayesian techniques in statistical applications. Objective Bayes methods, based on neutral or uniformative priors of the type pioneered by Jeffreys, dominate these applications, carried forward on a wave of popularity for Markov chain Monte Carlo (MCMC) algorithms. Good references include Ghosh (2011), Berger (2006), and Kass and Wasserman (1996).

Suppose then that having observed data x from a known parametric family f_μ(x), I wish to estimate t(μ), a parameter of particular interest. In the absence of relevant prior experience, I assign an uninformative prior π(μ), perhaps from the Jeffreys school. Applying Bayes rule yields θ̂, the posterior expectation of t(μ) given x,

$$\widehat{\theta }=E\{t(\mu )\mid x\}.$$ (1.1)

How accurate is θ̂? The obvious answer, and the one almost always employed, is to infer the accuracy of θ̂ according to the Bayes posterior distribution of t(μ) given x. This would obviously be correct if π(μ) were based on genuine past experience. It is not so obvious for uninformative priors. I might very well like θ̂ as a point estimate, based on considerations of convenience, coherence, smoothness, admissability, or esthetic Bayesian preference, but not trust what is after all a self-selected choice of prior as determining θ̂’s accuracy. Berger (2006) makes this point at the beginning of his Section 4.

As an alternative, this paper proposes computing the frequentist accuracy of θ̂. That is, regardless of its Bayesian provenance, we consider θ̂ simply as a function of the data x, and compute its frequentist variability.

Our main result, presented in Section 2, is a general accuracy formula for the delta-method standard deviation of θ̂: general in the sense that it applies to all prior distributions, uninformative or not. Even in complicated situations the formula is computationally inexpensive: the same MCMC calculations that give the Bayes estimate θ̂ also provide its frequentist standard deviation. A lasso-type example is used for illustration. Note: Many of the examples that follow use Jeffreys priors; this is only for simplified exposition, and is not a limitation of the theory.

In fact several of our examples will demonstrate near equality between Bayesian and frequentist standard deviations. That does not have to be the case; Remark 1 of Section 6 discusses a class of reasonable examples where the frequentist accuracy can be less than half of its Bayesian counterpart. Other examples will calculate frequentist standard deviations for situations where there is no obvious Bayesian counterpart, e.g., for the upper endpoint of a 95% credible interval.

The general accuracy formula takes on a particularly simple form when f_μ(x) represents a p-parameter exponential family, Section 3. Exponential family structure also allows us to substitute parametric bootstrap sampling for MCMC calculations, at least for uninformative priors. This has computational advantages. More importantly, it helps connect Bayesian inference with the seemingly super-frequentist bootstrap world, a central theme of this paper.

The general accuracy formula provides frequentist standard deviations for Bayes estimators, but nothing more. Better inferences, in the form of second order-accurate confidence intervals are developed in Section 4, again in an exponential family bootstrap context. Section 5 uses the accuracy formula to compare hierarchical and empirical Bayes methods. The paper concludes with remarks, details, and extensions in Section 6.

The frequentist properties of Bayes estimates is a venerable topic, nicely reviewed in Chapter 4 of Carlin and Louis (2000). Particular attention focuses on large-sample behavior, where “the data swamps the prior” and θ̂ converges to the maximum likelihood estimator (see Result 8, Section 4.7 of Berger, 1985), in which case the Bayes and frequentist standard deviations are nearly the same. Our accuracy formula provides some information about what happens before the data swamps the prior.

Some other important Bayesian-cum-frequentist topics are posterior and preposterior model checking as in Little (2006) or Chapter 6 of Gelman et al. (1995); Bayesian consistency, Diaconis and Freedman (1986); confidence matching priors, going back to Welch and Peers (1963); and empirical Bayes analysis as in Morris (1983). Johnstone and Silverman (2004) provide, among much else, asymptotic bounds for the frequentist accuracy of empirical Bayes estimates.

Sensitivity analysis — modifying the prior as a check on the stability of posterior inference — is a staple of Bayesian model selection. The methods of this paper amount to modifying the data as a posterior stability check (see Lemma 1 of Section 2). The implied suggestion here is to consider both techniques when the prior is in doubt.

2 The general accuracy formula

We wish to estimate the frequentist accuracy of a Bayes posterior expectation θ̂ = E{t(μ)|x} (1.1), where t(μ) is a parameter of particular interest. Here μ is an unknown parameter vector existing in parameter space Ω with prior density π(μ), while x is a sufficient statistic taking its values in, say, p-dimensional space,

$$x\in {\mathcal{R}}^p,$$ (2.1)

drawn from density f_μ(x) in a known parametric family

$$\mathcal{F}=\{f_{\mu }(x),\mu \in \mathrm{\Omega }\}.$$ (2.2)

We write the expectation and covariance of x given μ as

$$x\sim (m_{\mu },V_{\mu })$$ (2.3)

with V_μ a p × p matrix. Denote the gradient of log f_μ(x) with respect to x by

$${\alpha }_x(\mu )={\nabla }_{x}log{f}_{\mu }(x)={\left(\cdots \frac{\partial }{\partial x_i}log{f}_{\mu }(x)\cdots \right)}^{\top }$$ (2.4)

Lemma 1

The gradient of θ̂ = E{t(μ)|x} with respect to x is the posterior covariance of t(μ) with α_x(μ),

$${\nabla }_x\widehat{\theta }=cov\{t(\mu ),{\alpha }_x(\mu )\mid x\}.$$ (2.5)

Proof

Write θ̂ = A(x)/B(x) where

$$A(x)={\int }_{\mathrm{\Omega }}t(\mu )\pi (\mu )f_{\mu }(x)d\mu \text{and}B(x)={\int }_{\mathrm{\Omega }}\pi (\mu )f_{\mu }(x)d\mu .$$ (2.6)

Denoting the gradient operator ∇_x by primes, so α_x(μ) = (log f_μ(x))′ (2.4) we calculate

$$\begin{array}{l}{A}^{\prime }(x)={\int }_{\mathrm{\Omega }}t(\mu ){\alpha }_x(\mu )\pi (\mu )f_{\mu }(x)d\mu ,\\ B^{\prime }(x)=\int {\alpha }_x(\mu )\pi (\mu )f_{\mu }(x)d\mu .\end{array}$$ (2.7)

Using (A/B)′ = (A/B)[A′/A − B′/B] gives

$$\begin{array}{l}{\widehat{\theta }}^{\prime }=\widehat{\theta }·\left[\frac{{\int }_{\mathrm{\Omega }}t(\mu ){\alpha }_x(\mu )\pi (\mu )f_{\mu }(x)d\mu }{{\int }_{\mathrm{\Omega }}t(\mu )\pi (\mu )f_{\mu }(x)d\mu }-\frac{{\int }_{\mathrm{\Omega }}{\alpha }_x(\mu )\pi (\mu )f_{\mu }(x)d\mu }{{\int }_{\mathrm{\Omega }}\pi (\mu )f_{\mu }(x)d\mu }\right]\\ =\widehat{\theta }·\left[\frac{E\{t(\mu ){\alpha }_x(\mu )\mid x\}}{E\{t(\mu )\mid x\}}-E\{{\alpha }_x(\mu )\mid x\}\right]\\ =E\{t(\mu ){\alpha }_x(\mu )\mid x\}-E\{t(\mu )\mid x\}E\{{\alpha }_x(\mu )\mid x\}\\ =cov\{t(\mu ),{\alpha }_x(\mu )\mid x\}.\end{array}$$

A sufficient condition for the interchange of integration and differentiation in (2.7) is that $t(\mu )\pi (\mu )f_{\mu }^{\prime }(x)$ be bounded in absolute value by a function g(μ, x̃) having ∫_Ωg(μ, x̃) dμ < ∞ for x̃ in an open neighborhood of x, and similarly for $\pi (\mu )f_{\mu }^{\prime }(x)$. See Section 2.4 of Casella and Berger (2002). Remark 2 of Section 6 presents a more computational derivation of Lemma 1, where the crucial condition is only that the gradient $f_{\mu }^{\prime }(x)$ exist continuously in a neighborhood of x.

Lemma 1 leads immediately to the general accuracy formula, general in the sense of applying to all choices of prior, not necessarily uninformative ones.

Theorem 1

The delta-method approximation for the frequentist standard deviation of θ̂ = E{t(μ)|x} is

$$\widehat{\text{sd}}={\left[cov{\{t(\mu ),{\alpha }_x(\mu )\mid x\}}^{\top }{V}_{\widehat{\mu }}cov\{t(\mu ),{\alpha }_x(\mu )\mid x\}\right]}^{1/2},$$ (2.8)

where μ̂ is the value of μ having $m_{\widehat{\mu }}=x$.

Proof

The theorem is an immediate consequence of Lemma 1 and the usual delta-method estimate of a statistic s(x), as described for instance in Section 4.6 of Rice (2007). Suppose for the sake of convenient notation that x is unbiased for μ, so m_μ = μ in (2.3), and μ̂ = x in (2.8). Assuming that the gradient s′(x) = ∇_x(s) exists continuously in a neighborhood of μ, a Taylor series expansion gives

$$s(x)=s(\mu )+s^{\prime }(\mu )(x-\mu )+o(x-\mu ).$$ (2.9)

The delta method (or “propagation of errors” as it is known in the physical science literature) ignores the o(x − μ) term in (2.9), and approximates the standard deviation of s(x) by

$${\text{sd}}_{\mu }(s)\doteq {\left[s^{\prime }{(\mu )}^{\top }{V}_{\mu }{s}^{\prime }(\mu )\right]}^{1/2},$$ (2.10)

at the final step, plugging in an unbiased or maximum likelihood estimate μ̂ for μ. The theorem applies (2.10) to s(x) = θ̂ = E{t(μ)|x}, using s′(x) = cov{t(μ), α_x(μ)|x} (2.5).

The delta method can be used to estimate bias as well as standard deviation, by extending (2.9) to a second-order Taylor series. Instead, the exponential family development of Section 4 provides second-order accurate frequentist confidence intervals for θ̂, correcting for bias as well as other effects.

A useful special case of Theorem 1 appears in Meneses et al. (1990). Fraser (1990, Sect. 2) makes use of ∇_xlog f_μ(x) in likelihood-based procedures for calculating tail probabilities. The goal of his 1990 paper is related to ours in the sense that likelihood methods enjoy a flat-prior Bayesian interpretation. Fraser’s work can be thought of as a continuation of the matching priors theory of Welch and Peers (1963), in which prior distributions are constructed to have favorable frequentist properties (as opposed to finding the frequentist properties of arbitrary priors, our goal here).

Several points about the general accuracy formula (2.6) are worth emphasizing.

Implementation

Suppose

$$\{{\mu }_1,{\mu }_2,{\mu }_3,\dots ,{\mu }_B\}$$ (2.11)

is a sample of size B from the posterior distribution of μ given x. Each μ_i gives corresponding values of t(μ) and α_x(μ) (2.4),

$$t_i=t({\mu }_i)\text{and}{\alpha }_i={\alpha }_x({\mu }_i).$$ (2.12)

Then t̄= Σt_i/B approximates the posterior expectation θ̂, while

$$\widehat{cov}=\sum _{i=1}^B({\alpha }_i-\overline{\alpha })(t_i-\overline{t})/B\left[\overline{\alpha }=\sum {\alpha }_i/B\right]$$ (2.13)

estimates the posterior covariance (2.5), so the same simulations that give θ̂ also provide its frequentist standard deviation. (This assumes that V_μ̂ is easily available, as it will be in our applications.)

Posterior sampling

The posterior sample {μ₁, μ₂, …, μ_B} will typically be obtained via MCMC, after a suitable burn-in period. The nonindependence of the μ’s does not invalidate (2.13), but suggests that large values of B may be required for computational accuracy. The bootstrap-based posterior sampling method of Section 3 produces independent values μ_i. Independence permits simple assessments of the required size B; see (3.12).

Exponential families

Section 3 shows that α_x(μ) (2.4) has a simple form, not depending on x, in exponential families.

Factorization

If

$$f_{\mu }(x)=g_{\mu }(x)h(x)$$ (2.14)

then the gradient

$${\nabla }_{x}log{f}_{\mu }(x)={\nabla }_{x}log{g}_{\mu }(x)+{\nabla }_{x}logh(x).$$ (2.15)

The last term does not depend on μ, so cov{t(μ), ∇_xlog f_μ(x)|x} equals cov{t(μ), ∇_xlog g_μ(x)|x} and we can take

$${\alpha }_x(\mu )={\nabla }_{x}log{g}_{\mu }(x)$$ (2.16)

in the lemma and the theorem.

Sufficiency

If x = (y, z) where x is p-dimensional and y = Y(x) is a q-dimensional sufficient statistic, we can write f_μ(x) = g_μ(y)h(z) and

$${\alpha }_x(\mu )={\nabla }_{x}log{f}_{\mu }(x)={\nabla }_{x}log{g}_{\mu }(y)+{\nabla }_{x}logh(z).$$ (2.17)

As in (2.15), the last term does not depend on μ so we can take α_x(μ) = ∇_xlog g_μ(y). Letting α_y(μ) = ∇_ylog g_μ(y), a q-dimensional vector,

$${\alpha }_x(\mu )={Y^{\prime }}^{\top }{\alpha }_y(\mu ),$$ (2.18)

where Y′ is the q × p matrix (∂y_i/∂x_j). From (2.8) we get

$$\widehat{\text{sd}}={\left[cov{\{t(\mu ),{\alpha }_y(\mu )\mid y\}}^{\top }{Y}^{\prime }{V}_{\widehat{\mu }}{Y^{\prime }}^{\top }cov\{t(\mu ),{\alpha }_y(\mu )\mid y\}\right]}^{1/2}.$$ (2.19)

Notice that Y′V_μ̂Y′^⊤ is the delta-method estimate of the covariance matrix of y when μ equals μ̂. In this approximate sense the theorem automatically accounts for sufficiency. However we can avoid the approximation if in the first place we work with y and its actual covariance matrix. (This will be the case in the exponential family setup of Section 3.) More importantly, working with y makes $\widehat{\text{cov}}$ in (2.13) lower-dimensional, and yields better estimation properties when substituted into (2.8).

Vector parameter of interest

The lemma and theorem apply also to the case where the target parameter t(μ) is vector-valued, say K-dimensional, as is θ̂ = E{t(μ)|x}. Then ∇_xθ̂ and cov{t(μ), α_x(μ)|x} in (2.5) become p × K matrices, yielding K × K approximate frequentist covariance matrix $\widehat{\text{var}}$ for θ̂ = E{t(μ)|x},

$$\widehat{var}=cov{\{t(\mu ),{\alpha }_x(\mu )\mid x\}}^{\top }{V}_{\widehat{\mu }}cov\{t(\mu ),{\alpha }_x(\mu )\mid x\},$$ (2.20)

with α_x(μ) and $V_{\widehat{\mu }}$ the same as before

Discrete statistic x

Suppose in (2.2) is the one-dimensional Poisson family f_μ(x) = exp(−μ)· μ^x/x!, x a nonnegative integer. We can still calculate α_x(μ) = log(μ) (2.4) (ignoring the term due to x!, as in (2.15)). For μ greater than, say, 10, the Poisson distribution ranges widely enough to smooth over its discrete nature, and we can expect formula (2.8) to apply reasonably well. Section 5 discusses a multidimensional discrete application.

Sd bias correction

Replacing cov{t(μ), α_x(μ)|x} in (2.8) with its nearly unbiased estimate $\widehat{cov}$ (2.13) upwardly biases the sd estimate. Remark 4 of Section 6 discusses a simple bias correction. Bias was negligible in the numerical examples that follow.

As an example of Theorem 1 in action, we will consider the Diabetes Data of Efron et al. (2004): n = 442 diabetes patients each have had observed a vector x of p = 10 predictor variables (age, sex, body mass index, blood pressure, and six blood serum measurements),

$${\mathit{x}}_i=(x_{i1},x_{i2},\dots ,x_{i10})\text{for}i=1,2,\dots ,n=442,$$ (2.21)

and also a response variable y_i measuring disease progression at one year after entry. Standardizing the predictors and response variables suggests a normal linear model

$$\mathit{y}=\mathit{X}\alpha +\mathit{e}\text{with}\mathit{e}\sim {\mathcal{N}}_n(0,\mathit{I}).$$ (2.22)

Here X is the n × p matrix having ith row x_i, while y is the vector of n responses.

Park and Casella (2008) consider applying a Bayesian version of the lasso (Tibshirani, 1996) to the Diabetes Data. In terms of our model (2.22) (they do not standardize the response) Park and Casella take the prior distribution for α to be

$$\pi (\alpha )=e^{-\lambda L_1(\alpha )},$$ (2.23)

with L₁(α) the L₁ norm ${\sum }_1^{10}\mid {\alpha }_j\mid$, and λ having value (in our standardized setup)

$$\lambda =0.37$$ (2.24)

The Laplace-type prior (2.23) results in the posterior mode of α given y coinciding with the lasso estimate

$${\widehat{\alpha }}_{\lambda }=arg\underset{\alpha }{max}\{{\Vert \mathit{y}-\mathit{X}\alpha \Vert }^2/2+\lambda L_1(\alpha )\},$$ (2.25)

as pointed out in Tibshirani (1996). The choice λ = 0.37 was obtained from marginal maximum likelihood considerations. In this sense Park and Casella’s analysis is empirical Bayesian, but we will ignore that here and assume prior (2.23)–(2.24) to be pre-selected. (The lasso itself plays no role in their calculations or the ones here except as motivation.)

An MCMC algorithm was used to produce (after burn-in) B = 10, 000 samples α_i from the posterior distribution π(α|y), under assumptions (2.22)–(2.24),

$$\{{\alpha }_i,i=1,2,\dots ,B=10,000\}.$$ (2.26)

From these we can approximate the Bayes posterior expectation θ̂ = E{γ|y} for any parameter of interest γ = t(α),

$$\widehat{\theta }=\sum _{i=1}^{B}t({\alpha }_i)/B.$$ (2.27)

Note: It is helpful here and in what follows to denote the parameter of interest as γ = t(μ) with θ̂ = E{γ|x} indicating its posterior expectation.

We can now apply Theorem 1 to estimate the frequentist standard deviation of θ̂. In terms of the general notation (2.2), μ becomes α, while we can take x to be the sufficient statistic β̂ = X^⊤y in model (2.22). (We could take x = α̂ = (X^⊤X)⁻¹X^⊤y, but our choices make α the natural parameter vector and β̂ the sufficient statistic in the exponential family form (3.1).) Section 3 shows that α_x(μ) (2.4) equals α in an exponential family. With $\widehat{cov}$ computed as in (2.13), the computational form of Theorem 1 yields frequentist standard deviation

$$\widehat{\text{sd}}={\left\{{\widehat{cov}}^{\top }G\widehat{cov}\right\}}^{1/2}\left[G={\mathit{X}}^{\top }\mathit{X}\right],$$ (2.28)

since G is the variance matrix V of β̂.

As a univariate “parameter of special interest,” consider estimating

$${\gamma }_{125}={\mathit{x}}_{125}\alpha ,$$ (2.29)

the diabetes progression for patient 125. (Patient 125 fell near the center of the y response scale.) The 10,000 values γ̂_125,i = x₁₂₅α_i were nearly normally distributed,

$$\mathcal{N}(0.248,{0.072}^2).$$ (2.30)

Formula (2.28) gave frequentist standard deviation 0.071 for the posterior expectation of γ₁₂₅, θ̂₁₂₅ = 0.248 = Σγ̂_125,i/10, 000, almost the same as the posterior standard deviation, but having a quite different interpretation. The near equality here is no fluke, but can turn out differently for other linear combinations γ = xα; see Remark 1 of Section 6.

Suppose we are interested in the posterior cumulative distribution function (cdf) of γ₁₂₅. For a given value c define

$$t_c(\alpha )=\{\begin{array}{ll}1\hfill & \text{if}{\mathit{x}}_{125}\alpha \le c\hfill \\ 0\hfill & \text{if}{\mathit{x}}_{125}\alpha >c\hfill \end{array}$$ (2.31)

so E{t_c(α)|y} = Pr{γ₁₂₅ ≤ c|y}. The MCMC sample (2.26) provides B = 10, 000 posterior values t_ci, from which we obtain the estimated cdf(c) value ${\sum }_1^B{t}_{ci}/B$ and its standard deviation (2.8); for example c = 0.3 gives

$$Pr\{{\gamma }_{125}\le 0.3\mid \mathit{y}\}=0.762\pm 0.304,$$ (2.32)

0.304 being the frequentist standard deviation of the posterior Bayes cdf 0.762.

The heavy curve in Figure 1 traces the posterior cdf of γ₁₂₅. Dashed vertical bars indicate ± one frequentist standard deviation. If we take the prior (2.23) literally then the cdf curve is exact, but if not, the large frequentist standard errors suggest cautious interpretation, in the same way we might react to a disturbing sensitivity analysis on the choice of prior.

Figure 1.

Heavy curve is posterior cdf of γ₁₂₅ (2.29), Diabetes Data; vertical dashed lines indicate ± one frequentist standard error. The estimated curve is quite uncertain from a frequentist viewpoint. The upper 0.90 value ĉ = 0.342 has frequentist standard error 0.069, as indicated by the horizontal bar.

The cdf curve equals 0.90 at ĉ = 0.342, this being the upper endpoint of a one-sided Bayes 90% credible interval. The frequentist standard deviation of ĉ is 0.069 (obtained from $\widehat{\text{sd}}(\text{cdf}(\widehat{c}))$ divided by the posterior density at ĉ, the usual delta-method approximation), giving coefficient of variation 0.069/0.342 = 0.20.

For θ₁₂₅ itself we were able to compare the frequentist standard deviation 0.071 with its Bayes posterior counterpart 0.072 (2.30). No such comparison is possible for the posterior cdf estimates: the cdf curve in Figure 1 is exact under prior (2.23)–(2.24). We might add a hierarchical layer of Bayesian assumptions in front of (2.23)–(2.24) in order to assess the curve’s variability, but it is not obvious how to do so here. (Park and Casella, 2008, Section 3.2, make one suggestion.)

The frequentist error bars of Figure 1 extend below zero and above one, a reminder that standard deviations are a relatively crude inferential tool. Section 4 discusses more sophisticated frequentist methods.

3 A bootstrap version of the general formula

A possible disadvantage of Section 2’s methodology is the requirement of a posterior sample {μ₁, μ₂, …, μ_B} from π(μ|x) (2.11). This section discusses a parametric bootstrap approach to the general accuracy formula that eliminates posterior sampling, at the price of less generality: a reduction of scope to exponential families and to priors π(μ) that are at least roughly uninformative. On the other hand, the bootstrap methodology makes the computational error analysis, i.e., the choice of the number of replications B, straightforward, and, more importantly, helps connect Bayesian and frequentist points of view.

A p-parameter exponential family can be written as

$$\mathcal{F}:\left\{f_{\alpha }\left(\widehat{\beta }\right)=e^{{\alpha }^{\top }\widehat{\beta }-\psi (\alpha )}{f}_0\left(\widehat{\beta }\right),\alpha \in \mathcal{A}\right\}.$$ (3.1)

Here α is the natural or canonical parameter vector, and β̂ is the p-dimensional sufficient statistic. The expectation parameter β = E_α{β̂} is a one-to-one function of α, say β = A(α), with β̂ equaling the maximum likelihood estimate (MLE) of β. The parameter space for α is a subset of , p-dimensional space, as is the corresponding space for β. The function ψ(α) provides the multiplier necessary for f_α(β̂) to integrate to 1.

In terms of the generic notation (2.1)–(2.2), α is μ and β̂ is x. The expectation and covariance of β̂ given α,

$$\widehat{\beta }\sim (\beta ,V_{\alpha }),$$ (3.2)

can be obtained by differentiating ψ(α).

The general accuracy formula (2.8) takes a simplified form in exponential families.

Theorem 2

The delta-method approximation for the frequentist standard deviation of θ̂ = E{t(α)|β̂} in exponential family (3.1) is

$$\widehat{\text{sd}}={\left[cov{\left\{t(\alpha ),\alpha \mid \widehat{\beta }\right\}}^{\top }{V}_{\widehat{\alpha }}cov\left\{t(\alpha ),\alpha \mid \widehat{\beta }\right\}\right]}^{1/2},$$ (3.3)

where α̂, the natural parameter vector corresponding to β̂, is the MLE of α.

Proof

The gradient ∇_x log f_μ(x) in (2.4) is now

$$\begin{array}{l}{\nabla }_{\widehat{\beta }}log{f}_{\alpha }\left(\widehat{\beta }\right)={\nabla }_{\widehat{\beta }}\left\{{\alpha }^{\top }\widehat{\beta }-\psi (\alpha )+log{f}_0\left(\widehat{\beta }\right)\right\}\\ =\alpha +{\nabla }_{\widehat{\beta }}log{f}_0\left(\widehat{\beta }\right).\end{array}$$ (3.4)

The final term does not depend upon α so, as in (2.15), what was called α_x(μ) in (2.4) becomes simply α, reducing (2.8) to (3.3).

Parametric bootstrap resampling can be employed to calculate both θ̂ and $\widehat{\text{sd}}$, as suggested in Efron (2012). We independently resample B times from the member of having parameter vector α equal α̂,

$$f_{\widehat{\alpha }}(·)\to \{{\beta }_1,{\beta }_2,\dots ,{\beta }_i,\dots ,{\beta }_B\}$$ (3.5)

(β_i being short for the conventional bootstrap notation ${\widehat{\beta }}_i^{\ast }$). Each β_i gives a corresponding natural parameter vector α_i = A⁻¹(β_i). Let π_i = π(α_i), and define the “conversion factor”

$$R_i=f_{{\alpha }_i}\left(\widehat{\beta }\right)/f_{\widehat{\alpha }}({\beta }_i),$$ (3.6)

the ratio of the likelihood to the bootstrap density. (See (3.13)–(3.15) for the evaluation of R_i.)

The discrete distribution that puts weight

$$p_i={\pi }_i{R}_i/\sum _{j=1}^B{\pi }_j{R}_j$$ (3.7)

on α_i for i = 1, 2, …, B, approximates the conditional distribution of α given β̂. To see this let t_i = t(α_i) and ${\widehat{\theta }}_B={\sum }_1^B{t}_i{p}_i$, so

$${\widehat{\theta }}_B=\left[\sum _{i=1}^B{t}_i{\pi }_i{f}_{{\alpha }_i}\left(\widehat{\beta }\right)/f_{\widehat{\alpha }}({\beta }_i)\right]/\left[\sum _{i=1}^B{\pi }_i{f}_{{\alpha }_i}\left(\widehat{\beta }\right)/f_{\widehat{\alpha }}({\beta }_i)\right].$$ (3.8)

Since the β_i are drawn from bootstrap density f_α̂(·), (3.8) represents an importance sampling estimate of

$$\left[{\int }_{\mathcal{A}}t(\alpha )\pi (\alpha )f_{\alpha }\left(\widehat{\beta }\right)d\alpha \right]/\left[{\int }_{\mathcal{A}}\pi (\alpha )f_{\alpha }\left(\widehat{\beta }\right)\right],$$ (3.9)

which equals E{t(α)|β̂}.

The same argument applies to any posterior calculation. In particular, cov{t(α), α|β̂} in (3.3) is approximated by

$$\widehat{cov}=\sum _{i=1}^B{p}_i({\alpha }_i-\overline{\alpha })\left(t_i-\widehat{\theta }\right)\left[\widehat{\alpha }=\sum p_i{\alpha }_i,\widehat{\theta }=\sum p_i{t}_i\right].$$ (3.10)

Implementing Theorem 2 now follows three algorithmic steps:

1. Generate a parametric bootstrap sample β₁, β₂, …, β_B (3.5).

2. for each β_i calculate α_i, t_i = t(α_i), and p_i (3.7).

3. Compute $\widehat{cov}$ (3.10).

Then θ̂_B = Σp_it_i approximates θ̂ = E{t(α)|β̂}, and has delta-method frequentist standard deviation

$$\widehat{\text{sd}}={\left[{\widehat{cov}}^{\top }{V}_{\widehat{\alpha }}\widehat{cov}\right]}^{1/2}.$$ (3.11)

(The matrix V_α̂ can be replaced by the empirical covariance matrix of β₁, β₂, …, β_B or, with one further approximation, by the inverse of the covariance matrix of α₁, α₂, …, α_B.) Remark 3 of Section 6 develops an alternative expression for $\widehat{\text{sd}}$. In what follows, θ̂_B is called simply θ̂.

An MCMC implementation sample {μ_i, i = 1, 2, …, B} (2.11) approximates a multidimensional posterior distribution by an equally weighted distribution on B nonindependent points. The bootstrap implementation (3.5)–(3.7) puts unequal weights on B i.i.d. (independent and identically distributed) points.

Independent resampling permits a simple analysis of “internal accuracy,” the error due to stopping at B resamples rather than letting B → ∞. Define P_i = π_iR_i and Q_i = t_iP_i = t_iπ_iR_i. Since the pairs (P_i, Q_i) are independently resampled, standard delta-method calculations show that θ̂ = ΣQ_i/ΣP_i has internal squared coefficient of variation approximately

$${\widehat{\text{cv}}}_{int}^2=\frac{1}{B}\left[\sum _{i=1}^B{\left(\frac{Q_i}{\overline{Q}}-\frac{P_i}{\overline{P}}\right)}^2/B\right],$$ (3.12)

Q̄ = ΣQ_i/B and P̄ = ΣP_i/B. See Remark 3 of Section 6.

There are two sources of approximation in applying the general accuracy formula: Monte Carlo error due to stopping at B replications, and delta-method error in estimating the true standard deviation. For bootstrap sampling, formula (3.12) assesses the Monte Carlo error. The better bootstrap confidence intervals of Section 4 improve upon the inferential approximations of the delta method.

The conversion factor R_i (3.6) can be defined for any family {f_α(β̂)}, but it has a simple expression in exponential families:

$$R_i=\xi ({\alpha }_i)e^{\mathrm{\Delta }({\alpha }_i)},$$ (3.13)

where Δ(α) is the “half deviance difference”

$$\mathrm{\Delta }(\alpha )={(\alpha -\widehat{\alpha })}^{\top }\left(\beta +\widehat{\beta }\right)-2[\psi (\alpha )-\psi (\widehat{\alpha })],$$ (3.14)

and, to a good approximation (Efron, 2012, Lemma 1),

$$\xi (\alpha )=1/{\pi }^{\text{Jeff}}(\alpha ),$$ (3.15)

with π^Jeff(α) = |V_α|^1/2, Jeffreys invariant prior for α. If our prior π(α) is π^Jeff(α) then

$${\pi }_i{R}_i=e^{\mathrm{\Delta }({\alpha }_i)}.$$ (3.16)

The bootstrap distribution f_α̂(·) locates its resamples α_i near the MLE α̂. A working definition of an informative prior π(α) might be one that places substantial probability far from α̂. In that case, R_i is liable to take on enormous values, destabilizing the importance sampling calculations. Park and Casella’s prior (2.23)–(2.24) for the Diabetes Data would be a poor choice for bootstrap implementation (though this difficulty can be mitigated by recentering the parametric bootstrap resampling distribution).

Table 1 displays the cell infusion data, which we will use to illustrate bootstrap implementation of the general accuracy formula. Human muscle cell colonies were infused with mouse nucleii. Five increasing infusion proportions of mouse nucleii were tried, cultured over time periods ranging from one to five days, and observed to see if they thrived or did not. The table shows that 52 of the 61 colonies in the highest proportion/days category thrived, etc.

Table 1. Cell infusion data.

Human cell colonies were infused with mouse nucleii in 5 different proportions, over time periods varying from 1 to 5 days, and observed to see if they did or did not thrive. The table displays number thriving over number of colonies. For example, 5 of the 31 colonies in the lowest infusion/days category thrived.

		Days
		1	2	3	4	5
Proportions	1	5/31	3/28	20/45	24/47	29/35
	2	15/77	36/78	43/71	56/71	66/74
	3	48/126	68/116	145/171	98/119	114/129
	4	29/92	35/52	57/85	38/50	72/77
	5	11/53	20/52	20/48	40/55	52/61

Letting (s_jk, n_jk) be the number of successes and number of colonies in the jkth cell, we assume independent binomial variation,

$$s_{jk}\stackrel{\text{ind}}{\sim }\text{Bi}(n_{jk},{\xi }_{jk})j=1,2,\dots ,5,k=1,2,\dots ,5.$$ (3.17)

An additive logistic regression model fit the data reasonably well,

$$\text{logit}({\xi }_{jk})={\alpha }_0+{\alpha }_1{I}_j+{\alpha }_2{I}_j^2+{\alpha }_3{D}_k+{\alpha }_4{D}_k^2,$$ (3.18)

with I_j the infusion proportions 1, 2, …, 5, and D_k the days 1, 2, …, 5. Model (3.18) is a five-parameter exponential family (3.1).

For our parameter of special interest t(α) we will take

$$\gamma =\sum _{j=1}^5{\xi }_{j5}/\sum _{j=1}^5{\xi }_{j1},$$ (3.19)

the ratio of overall probability of success on Day 5 compared to Day 1, and calculate its posterior distribution assuming Jeffreys prior π^Jeff(α) on α. Warning: Jeffreys prior is convenient for illustrative purposes here, but can be dangerous to use in multidimensional situations. An alternative analysis based on conjugate priors appears below.

B = 2000 parametric bootstrap samples were generated according to

$$s_{jk}^{\ast }\stackrel{\text{ind}}{\sim }\text{Bi}\left(n_{jk},{\widehat{\xi }}_{jk}\right),j=1,2,\dots ,5,k=1,2,\dots ,5,$$ (3.20)

where ξ̂_jk is the MLE of ξ_jk obtained from model (3.18). These gave bootstrap MLEs α₁, α₂, …, α_i, …, α₂₀₀₀ and corresponding bootstrap estimates γ_i = t(α_i) as in (3.19). The weights p_i (3.7) that convert the bootstrap sample into a posterior distribution are

$$p_i=e^{{\mathrm{\Delta }}_i}/\sum _{j=1}^{2000}{e}^{{\mathrm{\Delta }}_j}$$ (3.21)

according to (3.16), with Δ_i the half binomial deviance difference (3.14); see Remark 5, Section 6.

The heavy curve in Figure 2 is the estimated posterior density, that is, a smoothed version of the discrete distribution putting weight p_i on γ_i = t(α_i). Its expectation

Figure 2.

Posterior density of ratio γ (3.19) given the cell infusion data; binomial model (3.17)–(3.18), Jeffries prior π^Jeff(α). From B = 2000 parametric bootstrap replications (3.20), posterior expectation 3.34 has frequentist $\widehat{\text{sd}}=0.273$. Solid line segment shows central 0.90 credible interval [2.92, 3.80]. Frequentist sd of 0.90 content is 0.042. Light dashed line is posterior density of γ using the conjugate prior density (3.23) with c₀ = 0.2 and b₀ equal the MLE β̂. Starred points indicate the raw unweighted bootstrap density.

$${\widehat{\theta }}_B=\sum _1^B{p}_{i}t({\alpha }_i)=3.34$$ (3.22)

is a Monte Carlo estimate of the posterior expectation of γ given the data. (B = 2000 resamples was excessive, formula (3.12) giving internal coefficient of variation only 0.002.)

How accurate is θ̂? Formula (3.11) yields $\widehat{\text{sd}}=0.273$ as its frequentist standard deviation. This is almost the same as the Bayes posterior standard deviation [Σp_i(γ_i − θ̂)²]^1/2 = 0.272.

In this case we can see why the Bayesian and frequentist standard deviations might be so similar: the Bayes posterior density is nearly the same as the raw bootstrap density (weight 1/B on each value γ_i). This happens whenever the parameter of interest has low correlation with the weights p_i (Lemma 3 of Efron, 2013). The bootstrap estimate of standard deviation [Σ(γ_i − γ̄)²]^1/2 equals 0.270, and it is not surprising that both the Bayes posterior sd and the frequentist delta-method sd are close to 0.270.

Integrating the solid curve in Figure 2 gives [2.92, 3.80] as the 0.90 central credible interval for γ. Defining t_i to be 1 or 0 as γ_i does or does not fall into this interval, formula (3.11) yields $\widehat{\text{sd}}=0.042$ for the frequentist standard deviation of the interval’s content. The two endpoints have sd’s 0.22 and 0.31. More interestingly, their frequentist correlation (calculated using (2.20); see Remark 6 of Section 6) is 0.999. This strongly suggests that replications of the muscle data experiment would show the 0.90 credible interval shifting left or right, without much change in length.

As an alternative to π^Jeff(α) we also considered conjugate priors for the exponential family (3.17)–(3.18). In terms of (3.1), conjugate priors have the form

$${\pi }_{c_0,b_0}(\alpha )=e^{c_0\left({\alpha }^{\top }{b}_0-\psi (\alpha )\right)},$$ (3.23)

Diaconis and Ylvisaker (1979). The p × p second-derivative matrix of −log π_c0b0 (α) is c₀ψ̈(α), with ψ̈(α) = (∂²ψ/∂α_i∂α_j), compared to ψ̈(α) for −log f_α(β̂), so small values of c₀ make π_c0,b0(α) more diffuse than the distribution of the MLE α̂. Spiegelhalter and Smith (1982), writing in a model selection context, recommend setting b₀ equal to β̂, the MLE of β = E_αβ̂, with c₀ perhaps 1/n for an iid sample of size n. For the non-iid data of Table 1, rough information calculations suggest c₀ on the order of 0.01.

Posterior expectations and standard deviations (not frequentist standard deviations from the general accuracy formula) are given in Table 2, for six choices of c₀ ranging from 0.005 to 0.2. These do not differ much from each other or from the Jeffreys moments, and all are close to the unweighted bootstrap values.

Table 2.

Posterior expectation and standard deviation of γ (3.19) for Jeffreys prior and six choices of c₀ for conjugate prior (3.23), b₀ = β̂. At right are expectation and standard deviation for the unweighted bootstrap distribution.

	Jeff	Conjugate priors						boot
		.005	.01	.025	.05	.1	.2
E	3.335	3.348	3.348	3.349	3.349	3.349	3.350	3.361
Sd	.272	.274	.273	.271	.268	.263	.252	.270

Note: R function freqacc, available from the author, calculates frequentist standard deviations for Bayes estimates obtained either by MCMC as in Section 2 or by bootstrap reweighting as here; the function assumes exponential family form (3.1).

4 Improved inferences

The general accuracy formula of Theorem 1 and Theorem 2 computes frequentist standard deviations for Bayesian estimates. Standard deviations are a good start but not the last word in assessing the accuracy of a point estimator. A drawback is apparent in Figure 1, where the standard error bars protrude beyond the feasible interval [0, 1].

This section concerns bootstrap methods that provide better frequentist inference for Bayesian estimates. A straightforward bootstrap approach would begin by obtaining a preliminary set of resamples, say

$$f_{\widehat{\alpha }}(·)\to {\widehat{b}}_1^{\ast },{\widehat{b}}_2^{\ast },\dots ,{\widehat{b}}_K^{\ast }$$ (4.1)

in the exponential family setup (3.1); for each ${\widehat{b}}_k^{\ast }$ calculating ${\widehat{\theta }}_k^{\ast }=\widehat{E}\{t(\alpha )\mid {\widehat{b}}_k^{\ast }\}$, the posterior expectation of t(α) given sufficient statistic ${\widehat{b}}_k^{\ast }$; and finally using { ${\widehat{\theta }}_1^{\ast },{\widehat{\theta }}_2^{\ast },\dots ,{\widehat{\theta }}_K^{\ast }$} to form a bootstrap confidence interval corresponding to the point estimate θ̂ = E{t(α)|β̂}, perhaps the BC_a interval (Efron, 1987). By construction, such intervals would not protrude beyond [0, 1] in the equivalent of Figure 1, and would take into account bias and interval asymmetry as well as standard deviation.

The roadblock to the straightforward approach is excessive computation. Bootstrap confidence intervals require K, the number of replicates, to be on the order of 1000. Each of these would require further simulations, {μ₁, μ₂, …, μ_B} as in (2.11) or {β₁, β₂, …, β_B} as in (3.5), B also exceeding 1000, in order to accurately calculate the ${\widehat{\theta }}_k^{\ast }$. Note: The change in notation from (3.5) to (4.1) is intended to emphasize that each ${\widehat{b}}_k^{\ast }$ needs to be followed, at least in the straightforward approach, by its own second-level bootstrap sample (3.5).

A shortcut method for bootstrap confidence calculations that, like Theorems 1 and 2, requires no additional replications, will be developed next. The shortcut requires exponential family structure (3.1), but otherwise applies equally to MCMC or bootstrap implementation (2.11) or (3.5).

Bayes theorem says that the posterior density g(α|β̂) corresponding to exponential family density f_α(β̂) (3.1) is

$$g\left(\alpha \mid \widehat{\beta }\right)=\pi (\alpha )f_{\alpha }\left(\widehat{\beta }\right)/f\left(\widehat{\beta }\right)\left[f\left(\widehat{\beta }\right)={\int }_{\mathcal{A}}\pi (\alpha )f_{\alpha }\left(\widehat{\beta }\right)d\alpha \right]$$ (4.2)

Suppose now we change the observed sufficient statistic vector β̂ to a different value b.

Lemma 2

The posterior distributions corresponding to exponential family form an exponential family ,

$$\mathcal{G}=\left\{g(\alpha \mid b)=e^{{(b-\widehat{\beta })}^{\top }\alpha -\varphi (b)}g\left(\alpha \mid \widehat{\beta }\right)\text{for}b-\widehat{\beta }\in \widehat{\mathcal{B}}\right\},$$ (4.3)

where

$$e^{\varphi (b)}={\int }_{\mathcal{A}}{e}^{{(b-\widehat{\beta })}^{\top }\alpha }g\left(\alpha \mid \widehat{\beta }\right)d\alpha .$$ (4.4)

is a p-parameter exponential family with roles reversed from ; now α is the sufficient statistic and b the natural parameter vector; is the convex set of vectors b − β̂ for which the integral in (4.4) is finite.

( is not the familiar conjugate family, Diaconis and Ylvisaker, 1979, though there are connections.)

Proof

From (3.1) we compute

$$\begin{array}{l}g(\alpha \mid b)=\pi (\alpha )f_{\alpha }(b)/f(b)\\ =\left[\frac{\pi (\alpha )f_{\alpha }\left(\widehat{\beta }\right)}{f\left(\widehat{\beta }\right)}\right]\left[\frac{f_{\alpha }(b)}{f_{\alpha }\left(\widehat{\beta }\right)}\right]\left[\frac{f\left(\widehat{\beta }\right)}{f(b)}\right].\end{array}$$ (4.5)

But

$$f_{\alpha }(b)/f_{\alpha }\left(\widehat{\beta }\right)=e^{{(b-\widehat{\beta })}^{\top }\alpha }\left[f_0(b)/f_0\left(\widehat{\beta }\right)\right],$$ (4.6)

yielding

$$g(\alpha \mid b)=g\left(\alpha \mid \widehat{\beta }\right)e^{{\left(b-\widehat{\beta }\right)}^{\top }\alpha }\left[\frac{f_0(b)f\left(\widehat{\beta }\right)}{f_0\left(\widehat{\beta }\right)f(b)}\right].$$ (4.7)

The final factor does not depend on α and so must equal exp(−ϕ(b)) in (4.3)–(4.4) in order for (4.7) to integrate to 1.

In Sections 2 and 3, g(α|β̂) was approximated by a discrete distribution putting weight p_i on α_i, say

$$\widehat{g}\left({\alpha }_i\mid \widehat{\beta }\right)=p_i\text{for}i=1,2,\dots ,B;$$ (4.8)

$p_i={\pi }_i{R}_i/{\sum }_1^B{\pi }_j{R}_j$ in bootstrap implementation (3.5)–(3.9); and p_i = 1/B in the MCMC implementation (2.11) where the μ_i play the role of the α_i.

Substituting ĝ(α|β̂) for g(α|β̂) in (4.3) produces the empirical posterior family . Define

$$W_i(b)=e^{{\left(b-\widehat{\beta }\right)}^{\top }{\alpha }_i}.$$ (4.9)

Then can be expressed as

$$\widehat{\mathcal{G}}:\left\{\widehat{g}({\alpha }_i\mid b)=W_i(b)p_i/\sum _{j=1}^B{W}_j(b)p_j\text{for}i=1,2,\dots ,B\right\},$$ (4.10)

b ∈ , i.e., the discrete distribution putting weight proportional to W_i(b)p_i on α_i. (Note: differs from the empirical exponential family in Section 6 of Efron, 2012.)

We can now execute the “straightforward bootstrap approach” (4.1) without much additional computation. The kth bootstrap replication ${\widehat{\theta }}_k^{\ast }=\widehat{E}\{t(\alpha )\mid {\widehat{b}}_k^{\ast }\}$ is estimated from $\widehat{g}({\alpha }_i\mid {\widehat{b}}_k^{\ast })$, using the importance sampling formula, as

$${\widehat{\theta }}_k^{\ast }=\sum _{i=1}^B{t}_i{W}_i\left({\widehat{b}}_k^{\ast }\right)p_i/\sum _{i=1}^B{W}_i\left({\widehat{b}}_k^{\ast }\right)p_i[t_i=t({\alpha }_i)].$$ (4.11)

Aside from step (4.1), usually comparatively inexpensive to carry out, we can obtain ${\widehat{\theta }}_1^{\ast },{\widehat{\theta }}_2^{\ast },\dots ,{\widehat{\theta }}_K^{\ast }$ from just the original calculations for θ̂ = Σt_ip_i, and use the ${\widehat{\theta }}_k^{\ast }$ values to construct a bootstrap confidence interval. (In particular, there is no need for new MCMC simulations for each new ${\widehat{b}}_k^{\ast }$.)

Section 6 of Efron (2012) carries out this program under the rubric “bootstrap after bootstrap.” It involves, however, some numerical peril: the weighting factors $W_i({\widehat{b}}_k^{\ast })$ can easily blow up, destabilizing the estimates ${\widehat{\theta }}_k^{\ast }$. The peril can be avoided by local resampling, that is, by considering alternate data values b very close to the actual β̂, rather than full bootstrap resamples as in (4.1).

This suggests the abc system of confidence intervals (“approximate bootstrap confidence,” Di-Ciccio and Efron, 1992, not to be confused with “Approximate Bayesian Computation,” as in Fearnhead and Prangle, 2012). The abc algorithm approximates full bootstrap confidence intervals using only a small number of resamples b in the immediate neighborhood of the observed sufficient statistic β̂.

Figure 3 shows again the posterior cdf from Figure 1 for γ₁₂₅, the progression parameter for patient 125 in the diabetes study. The heavy vertical bars indicate abc 68% central frequentist confidence limits for the Bayes posterior cdf values. Now the confidence limits stay within [0, 1]. (95% limits are much wider, nearly filling the interval [0, 1] for some values of c, indicating that perhaps we are asking too much of the Diabetes data set.) Remark 7 of Section 6 discusses the details of the abc calculations.

Figure 3.

Vertical bars are 68% central abc confidence limits for patient 125’s posterior cdf, Figure 1. They remain within the feasible interval [0, 1], unlike Figure 1’s standard deviation bars, shown here as light dashed vertical lines.

Standard confidence intervals, say $\widehat{\theta }\pm \widehat{\text{sd}}$ for approximate 68% coverage, require only the original point estimate θ̂ and its accuracy estimate $\widehat{\text{sd}}$, which in our case is what the general accuracy formula efficiently provides. The standard intervals are “first order accurate,” with their actual coverage probabilities converging to the nominal value at rate n^−1/2 as sample size n grows large.

The abc algorithm provides second order accuracy, that is, coverage errors approaching zero at rate n⁻¹. This is more than a theoretical nicety. As the examples in DiCiccio and Efron (1992) show, the abc intervals often come close to exact small-sample intervals when the latter exist. Three corrections are made to the standard intervals: for bias, for acceleration (i.e., changes in standard deviation between the interval endpoints), and for nonnormality. The algorithm depends on exponential family structure, provided by the empirical posterior family (4.10), and a smoothly varying point estimate.

In our situation the point estimate is the empirical posterior expectation (4.11) of t(α) given sufficient statistic b, say θ̂ = s(b),

$$\widehat{\theta }=s(b)=\sum _{i=1}^B{t}_i{W}_i(b)p_i/\sum _{i=1}^B{W}_i(b)p_i.$$ (4.12)

For b near β̂, the values explored in the abc algorithm, the smoothness of the kernel W_i(b) (4.9), makes s(b) smoothly differentiable.

What parameter is the intended target of the abc intervals? The answer, from DiCiccio and Efron (1992), is θ = s(β), the value of s(b) if sufficient statistic b equals its expectation β. It is not γ = t(α), the true value of the parameter of special interest.

Abc’s output includes bias, an assessment of the bias of θ̂ = s(β̂) as an estimator of θ, not as an estimate of γ. The more interesting quantity definitional bias,

$$\theta -\gamma =E\left\{t(\widehat{\alpha })\mid \widehat{\beta }=\beta \right\}-t(\alpha )$$ (4.13)

depends on the prior π(α). It seems reasonable to ask that an uninformative prior not produce large definitional biases. The parameter γ₁₂₅ (2.29) has MLE 0.316 ± 0.076, compared with Bayes estimate and frequentist standard deviation 0.248 ± 0.071, giving a relative difference of

$$\widehat{\delta }=\frac{\widehat{\theta }-\widehat{\gamma }}{\text{sd}(\widehat{\gamma })}=\frac{0.248-0.316}{0.076}=-\mathrm{0.90.}$$ (4.14)

In other words, the Park and Casella prior (2.23) shifts the estimate for patient 125 about 0.9 standard deviations downward, a quite substantial effect.

Figure 4 shows the relative difference estimates for all 442 diabetes patients. Most of the δ̂’s are less extreme than that for patient 125. Even though prior (2.23) looks like a strong shrinker, and not at all uninformative, its effects on the patient estimates are mostly moderate.

Figure 4.

Relative differences (4.14) for the 442 diabetes patients, Park and Casella prior (2.23): Bayes estimate minus MLE, divided by MLE standard deviation.

5 Hierarchical and empirical Bayes accuracy

Modern scientific technology excels at the simultaneous execution of thousands, and more, parallel investigations, the iconic example being microarray studies of genetic activity. Both hierarchical and empirical Bayes methods provide natural statistical tools for analyzing large parallel data sets. This section compares the accuracy of the two methods, providing some intuition as to why, often, there is not much difference.

A typical hierarchical model begins with a hyperprior π(α) providing a hyperparameter α, which determines a prior density g_α(δ); N realizations are generated from g_α(·), say

$$\mathit{\delta }=({\delta }_1,{\delta }_2,\dots ,{\delta }_k,\dots ,{\delta }_N);$$ (5.1)

finally, each parameter δ_k provides an observation z_k according to density h_δk (z_k), yielding a vector z of N observations,

$$\mathit{z}=(z_1,z_2,\dots ,z_k,\dots ,z_N).$$ (5.2)

The functional forms π(·), g_α(·), and h_δ(·) are known, but not the values of α and δ. Here we will assume that the pairs (δ_k, z_k) are generated independently for k = 1, 2, …, N. We wish to estimate the parameter δ from the observations z.

If α were known then Bayes theorem would directly provide the conditional distribution of δ_k given z_k,

$$g_{\alpha }({\delta }_k\mid z_k)=g_{\alpha }({\delta }_k)h_{{\delta }_k}(z_k)/f_{\alpha }(z_k),$$ (5.3)

where f_α(z_k) is the marginal density of z_k given α,

$$f_{\alpha }(z_k)=\int g_{\alpha }(\delta )h_{\delta }(z_k)d\delta .$$ (5.4)

The empirical Bayes approach estimates the unknown value of α from the observed vector z, often by marginal maximum likelihood,

$$\widehat{\alpha }=arg\underset{\alpha }{max}\left\{\prod _{i=1}^N{f}_{\alpha }(z_k)\right\},$$ (5.5)

and then infers the individual δ_k’s according to g_α̂(δ_k|z_k). Hierarchical Bayes inference aims instead for the full posterior distribution of δ_k given all the observations z,

$$g({\delta }_k\mid \mathit{z})=\int g_{\alpha }({\delta }_k\mid z_k)\pi (\alpha \mid \mathit{z})d\alpha .$$ (5.6)

We will employ the general accuracy formula to compare the frequentist variability of the two approaches. Note: In the example that follows, all of the calculations can be carried out in terms of the marginal densities f_α(·), rendering it unneccessary to specify the prior densities g_α(·).

As a working example we consider the prostate cancer microarray data (Singh et al., 2002). Each of 102 men, 52 prostate cancer patients and 50 controls, has had the activity of N = 6033 genes measured, as discussed in Section 5 of Efron (2012). A test statistic z_k comparing cancer patients with controls has been calculated for each gene, which we will assume here follows a normal translation model

$$z_k\sim \mathcal{N}({\delta }_k,1),$$ (5.7)

where δ_k is gene_k’s effect size (so h_δ(z) in (5.3)–(5.4) is the normal kernel $\varphi (z-\delta )=exp\{-{(z-\delta )}^2/2\}/\sqrt{2\pi }$. “Null” genes have δ_k = 0 and z_k ~ (0, 1), but of course the investigators were looking for nonnull genes, those having large δ_k values, either positive or negative.

Binning the data simplifies the Bayes and empirical Bayes analyses. For Figure 5 the data has been put into J = 49 bins , each of width 0.2, with centers c_j,

Figure 5.

Prostate data: dots are log counts for 49 bins (5.8)–(5.9). Dashed quadratic curve would fit the dots if all genes were null, δ_k = 0 in (5.7). Eighth-degree polynomial, heavy curve, gives a much better fit, indicating that some genes have large effect sizes.

$$c_j=-4.4,-4.2,\dots ,\mathrm{5.2.}$$ (5.8)

Let y_j be the count in bin ,

$$y_j=\#\{z_k\in {\mathcal{Z}}_j\}.$$ (5.9)

The dots in Figure 5 are the log counts log(y_j). The dashed quadratic curve would give a good fit to the dots if all the genes were null, but it is obviously deficient in the tails, suggesting some large effect sizes.

An eighth-degree polynomial, the solid curve, provided a good fit to the data. It was obtained from a Poisson regression GLM (generalized linear model). The counts y_j (5.9) were assumed to be independent Poisson variates,

$$y_j\stackrel{\text{ind}}{\sim }\text{Poi}({\mu }_j),j=1,2,\dots ,J=49,$$ (5.10)

with

$${\mu }_j=E_{\alpha }\{y_j\}=e^{\mathit{x}(c_j)\alpha }.$$ (5.11)

Here x(c_j) is the nine-dimensional row vector

$$\mathit{x}(c_j)=(1,c_j,c_j^2,\dots ,c_j^8),$$ (5.12)

the c_j being the bin centers (5.8), while α is an unknown parameter vector, α ∈ . There is a small loss of information in going from the full data vector z to the binned counts that we will ignore here.

Model (5.10)–(5.12) is a nine-parameter exponential family f_α(β̂) (3.1) with α the natural parameter vector. Its sufficient statistic is

$$\widehat{\beta }={\mathit{X}}^{\top }\mathit{y},$$ (5.13)

where X is the 49×9 matrix having jth row x(c_j), and y is the 49-vector of counts; β̂ has covariance matrix

$$V_{\alpha }={\mathit{X}}^{\top }\text{diag}({\mathit{\mu }}_{\alpha })\mathit{X},$$ (5.14)

diag(μ_α) the diagonal matrix with diagonal elements (5.11).

We are now ready to apply the accuracy formula in the exponential family form of Theorem 2 (3.3). A notable feature of this example is that the parameter of interest t(α) is itself a posterior expectation: let τ(δ) be an “interesting function” of an individual parameter δ in (5.1), for instance the indicator of whether or not δ = 0,

$$\tau (\delta )=I_0(\delta ).$$ (5.15)

Letting (δ₀, z₀) represent a hypothetical (parameter, observation) pair, we define t(α) to be the conditional expectation of τ(δ₀) given z₀, α, and the sufficient statistic β̂,

$$t(\alpha )=E\left\{\tau ({\delta }_0)\mid z_0,\alpha ,\widehat{\beta }\right\}.$$ (5.16)

In the prostate study, for example, with τ(δ) = I₀(δ) and z₀ = 3, t(α) is the conditional probability of a gene being null given a z-value of 3. However, α is unobserved and t(α) must be inferred. The hierarchical Bayes estimate is

$$\widehat{\theta }=E\left\{t(\alpha )\mid \widehat{\beta }\right\}=E\left\{\tau ({\delta }_0)\mid z_0,\widehat{\beta }\right\},$$ (5.17)

as compared to the empirical Bayes MLE estimate t(α̂). (Notice that we now require three levels of parameter definition: in addition to θ̂ being the posterior expectation of t(α) (5.17), t(α) itself is the posterior expectation of τ(δ₀) (5.16).)

The hyperprior π(α) is usually taken to be uninformative in hierarchical Bayes applications, making them good candidates for the bootstrap implementation of Section 3. Let α̂ be the MLE of hyperparameter α, obtained in the prostate study by Poisson regression from model (5.10)–(5.12), glm(y ~ X, poisson)$coef in language R. From α̂ we obtain parametric bootstrap samples ${\mathit{y}}_i^{\ast }$, i = 1, 2,…, B,

$$y_{ij}^{\ast }\stackrel{\text{ind}}{\sim }\text{Poi}({\widehat{\mu }}_j),j=1,2,\dots ,J,$$ (5.18)

where μ̂_j = exp(x(c_j)α̂). The ${\mathit{y}}_i^{\ast }$ vector yields β_i and α_i, (3.5) and (3.6): β_i = X^⊤ y* and α_i = glm(y* ~ X, poisson)$coef.

If for convenience we take π(α) to be Jeffreys prior, then the weights π_iR_i in (3.7) become

$${\pi }_i{R}_i=e^{\mathrm{\Delta }({\alpha }_i)}$$ (5.19)

(3.16), where, for Poisson regression, the half deviance difference Δ(α_i) is

$${\mathrm{\Delta }}_i={({\alpha }_i-\widehat{\alpha })}^{\top }\left({\beta }_i+\widehat{\beta }\right)-2\sum _{j=1}^J({\mu }_{ij}-{\widehat{\mu }}_j),$$ (5.20)

μ̂_j = exp(x(c_j)α̂) and μ_ij = exp(x(c_j)α_i) (Efron, 2012, Sect. 5). Letting t_i be the conditional expectation (5.16),

$$t({\alpha }_i)=E\left\{\tau ({\delta }_0)\mid z_0,{\alpha }_i,\widehat{\beta }\right\},$$ (5.21)

the hierarchical Bayes estimate θ̂ (5.17) is

$$\widehat{\theta }=\sum _{i=1}^B{p}_i{t}_i\left[p_i=e^{{\mathrm{\Delta }}_i}/\sum _{k=1}^B{c}^{{\mathrm{\Delta }}_k}\right],$$ (5.22)

and has frequentist standard $\widehat{\text{sd}}$ (3.11) from Theorem 2.

Figure 6 applies to the prostate data, taking τ(δ), the function of interest, to be δ itself; that is, the hierarchical Bayes estimate (5.17) is

Figure 6.

Hierarchical Bayes estimate θ̂ = E{δ₀|z₀, β̂} as a function of z₀, prostate study data. Vertical bars indicate ± one frequentist standard deviation $\widehat{\text{sd}}$ (3.11). Calculated from B = 4000 parametric bootstrap samples (5.18).

$$\widehat{\theta }=E\left\{{\delta }_0\mid z_0,\widehat{\beta }\right\},$$ (5.23)

the posterior expected effect size for a gene having z = z₀. The calculations assume Poisson regression model (5.10)–(5.12), beginning with Jeffreys prior π(α). B = 4000 bootstrap samples (5.18) provided the Bayesian estimates, as in (5.19)–(5.22). (Tweedie’s formula, Efron (2011), says that θ̂ in (5.23) equals z₀ + d/dz{log f_α̂(z)}|_z0, again allowing us to avoid explicit characterization of the prior distributions g_α(·).)

The heavy curve in Figure 6 shows θ̂ as a function of z₀. It stays near zero for z₀ in [−2, 2), suggesting nullity for genes having small z values, and then swings away from the horizontal axis, indicating nonnull effect sizes for large |z₀|, but always with strong “regression to the mean” behavior: |θ̂| < |z₀|. The vertical bars span plus or minus one frequentist standard deviation $\widehat{\text{sd}}$ (3.11).

There was very little difference between the hierarchical and empirical Bayes results. The graph of the empirical Bayes estimates

$$t(\widehat{\alpha })=E\left\{{\delta }_0\mid z_0,\alpha =\widehat{\alpha },\widehat{\beta }\right\}$$ (5.24)

follows the curve in Figure 6 to within the line width. Table 3 gives numerical comparisons for z₀ = −4, −3, …, 4. The estimated standard deviations for the empirical Bayes estimates, line 5, are a little bigger than those on line 3 for hierarchical Bayes, but that may just reflect the fact that the former are full bootstrap estimates while the latter are delta-method sd’s.

Table 3.

Comparison of hierarchical and empirical Bayes estimates for expected effect sizes in the prostate study. (1) Bayes estimate θ̂ (5.23); (2) empirical Bayes estimate E{δ₀|z₀, α = α̂, β̂}; (3) Bayes posterior sd [Σp_i(t_i − θ̂)²]^1/2; (4) frequentist sd of θ̂ (3.11); (5) bootstrap sd (5.25).

z0	−4	−3	−2	−1	0	1	2	3	4
1. Bayes est	−2.221	−1.480	−.329	−.093	−.020	.127	.357	1.271	3.042
2. Emp Bayes est	−2.217	−1.478	−.331	−.092	−.020	.126	.360	1.266	3.021
3. Bayes sd	.756	.183	.074	.036	.030	.039	.071	.131	.336
4. Bayes freq sd	.740	.183	.075	.035	.029	.038	.068	.131	.349
5. Emp Bayes sd	.878	.187	.074	.037	.030	.039	.072	.139	.386

Particularly striking is the agreement between the frequentist sd estimates for θ̂ (3.11), line 4, and the posterior Bayes sd estimates, line 3. This is the predicted asymptotic behavior (Berger, 1985, Sect. 4.7.8) if the effect of the prior distribution has indeed been swamped by the data. It cannot be assumed, though, that agreement would hold for estimates other then (5.23).

The empirical Bayes estimate t(α̂) = E{δ₀|z₀, α = α̂, β̂} had its standard deviation $\overline{\text{sd}}$, line 5 of Table 3, calculated directly from its bootstrap replications,

$$\overline{\text{sd}}={\left[\sum _1^B{(t_i-\overline{t})}^2/B\right]}^{1/2}\left[\overline{t}=\sum _1^B{t}_i/B\right],$$ (5.25)

as compared with the Bayes posterior standard deviation, line 3,

$$\widehat{\text{sd}}={\left[\sum _1^B{p}_i{\left(t_i-\widehat{\theta }\right)}^2\right]}^{1/2}\left[\widehat{\theta }=\sum _1^B{p}_i{t}_i\right].$$ (5.26)

(See Remark 8 of Section 6 concerning the calculation of t_i.) The difference comes from weighting the B bootstrap replications t_i according to p_i (3.7), rather than equally. Lemma 3 of Efron (2012) shows that the discrepancy, small in Table 3, depends on the empirical correlation between p_i and t_i.

There is a similar relation between lines 4 and 5 of the table. Remark 9 shows that $\overline{\text{sd}}$ line 5, is approximated by

$$\overline{\text{sd}}\doteq {\left[{\overline{cov}}^{\top }{V}_{\widehat{\alpha }}\overline{cov}\right]}^{1/2},$$ (5.27)

where $\overline{cov}$ is the unweighted bootstrap covariance between α_i and t_i,

$$\overline{cov}=\sum _1^B({\alpha }_i-\overline{\alpha })(t_i-\overline{t})/B\left[\overline{\alpha }=\sum _1^B{\alpha }_i/B\right].$$ (5.28)

This compares with the weighted version (3.10)–(3.11) of line 4. Weighting did not matter much in Table 3, leaving the three standard deviations more alike than different.

The eighth-degree polynomial fit used in Figure 5 might be excessive. For each of the B = 4000 bootstrap samples ${\mathit{y}}_i^{\ast }$, the “best” polynomial degree $m_i^{\ast }$ was selected according to the AIC criterion, as detailed in Section 5 of Efron (2012). Only degrees m = 0 through 8 were considered. The top row of Table 4 shows that 32% of the 4000 bootstrap samples gave $m_i^{\ast }=4$, compared to 51% for $m_i^{\ast }=8$. (None of the samples had $m_i^{\ast }$ less than 4.)

Table 4.

Polynomial model selection for the prostate study data. Row 1: raw bootstrap proportions for best polynomial fit, AIC criterion; row 2: corresponding Bayes posterior probabilities, Jeffreys prior; row 3: frequentist standard deviations for the Bayes estimates.

m	4	5	6	7	8
1. Bootstrap %	32%	10%	5%	1%	51%
2. Bayes exp	36%	12%	5%	2%	45%
3. Freq sd	±32%	±16%	±8%	±3%	±40%

Let $t_i^{(m)}$ be the indicator for model m selection,

$$t_i^{(m)}=\{\begin{array}{ll}1\hfill & \text{if}{m}_i^{\ast }=m\hfill \\ 0\hfill & \text{if}{m}_i^{\ast }\ne m.\hfill \end{array}$$ (5.29)

Then

$${\widehat{\theta }}^{(m)}=\sum _{i=1}^B{p}_i{t}_i^{(m)}$$ (5.30)

is the posterior probability of the region in the space of possible α vectors where degree m is best; for instance, θ̂⁽⁴⁾ equals 36% on row 2.

We can apply Theorem 2 (3.11) to obtain frequentist standard deviations for the θ̂^(m). These are shown in row 3. The results are discouraging, with θ̂⁽⁴⁾ = 36% having $\widehat{\text{sd}}=32\%$ and so on. (These numbers differ from those in Table 2 of Efron, 2012, where the standard deviations were assessed by the potentially perilous “bootstrap after bootstrap” method.) There was a strong negative frequentist correlation of −0.84 between θ̂⁽⁴⁾ and θ̂⁽⁸⁾ (using (2.20)). All of this suggests that the MLE α̂ lies near the boundary between and , but not near the other regions. Bayesian model selection, of the limited type considered above, is frequentistically unstable for the prostate data.

6 Remarks

This section presents remarks, details, and extensions of the previous material.

Remark 1. Relation of Bayes and frequentist standard deviations

In several of our examples the posterior Bayes estimate θ̂ had its posterior standard deviation ${\widehat{\text{sd}}}_{\text{Bayes}}$ quite close to ${\widehat{\text{sd}}}_{\text{freq}}$, the frequentist sd. Why this might happen, or might not, is easy to understand in the diabetes data example (2.29)–(2.30).

Let α̃ be the 10, 000 × 10 matrix with ith row α_i − ᾱ, so

$${\mathrm{\sum }}_{\alpha }={\stackrel{\sim }{\mathit{\alpha }}}^{\top }\stackrel{\sim }{\mathit{\alpha }}/B(B=10,000)$$ (6.1)

is the empirical covariance matrix of the α_i vectors. For any fixed row vector x we define as our parameter of special interest γ_x = xα (x = x₁₂₅ in (2.29)). Each α_i gives t_i = xα_i, with average t̄ = xᾱ. The vector t̃ of centered values t̃_i = t_i − t̄ is given by

$$\stackrel{\sim }{\mathit{t}}=\stackrel{\sim }{\mathit{\alpha }}{\mathit{x}}^{\top }.$$ (6.2)

Then

$${\widehat{\text{sd}}}_{\text{Bayes}}^2=\sum _1^B{\stackrel{\sim }{\mathit{t}}}^2/B=\mathit{x}{\mathrm{\sum }}_{\alpha }{\mathit{x}}^{\top }.$$ (6.3)

Also, from (2.13),

$${\widehat{cov}}^{\top }=\stackrel{\sim }{\mathit{t}}\stackrel{\sim }{\mathit{\alpha }}/B=\mathit{x}{\mathrm{\sum }}_{\alpha },$$ (6.4)

yielding

$${\widehat{\text{sd}}}_{\text{freq}}^2=\mathit{x}{\mathrm{\sum }}_{\alpha }G{\mathrm{\sum }}_{\alpha }{\mathit{x}}^{\top }$$ (6.5)

from (2.28).

The variance ratio rat(x) equals

$$\text{rat}(\mathit{x})={\left(\frac{{\widehat{\text{sd}}}_{\text{freq}}}{{\widehat{\text{sd}}}_{\text{Bayes}}}\right)}^2=\frac{\mathit{x}{\mathrm{\sum }}_{\alpha }G{\mathrm{\sum }}_{\alpha }{\mathit{x}}^{\top }}{\mathit{x}{\mathrm{\sum }}_{\alpha }{\mathit{x}}^{\top }}.$$ (6.6)

Suppose $H={\mathrm{\sum }}_{\alpha }^{1/2}G{\mathrm{\sum }}_{\alpha }^{1/2}$ has spectral decomposition H = ΓdΓ^⊤, d the diagonal matrix of eigenvalues. Then (6.6) reduces to

$$\text{rat}(\mathit{x})=\sum _1^p{d}_i{v}_i^2/\sum v_i^2\left(\mathit{v}=\mathit{x}{\mathrm{\sum }}_{\alpha }^{1/2}\mathrm{\Gamma }\right).$$ (6.7)

Table 5 shows the eigenvalues d_i. We see that rat(x) could vary from 1.014 down to 0.098. For the 442 diabetes patients, rat(x_i) ranged from 0.991 to 0.670, averaging 0.903; rat(x₁₂₅) = 0.962 was near the high end. A spherically uniform choice of v in (6.7) would yield an average rat(x) of 0.800.

Table 5.

Eigenvalue d_i for the variance ratio rat(x) (6.7).

di

1.014

1.009

.986

.976

.961

.944

.822

.710

.482

.098

The fact that the eigenvalues in Table 5 are mostly less than one relates to the Park and Casella prior (2.23). A flat prior for model (2.22) has cov(α) = G⁻¹, giving H = I and eigenvalues d_i = 1 in (6.7). The Park and Casella prior (2.23) is a “shrinker,” making σ_α and H less than I.

A more general but less transparent formula for ${({\widehat{\text{sd}}}_{\text{freq}}/{\widehat{\text{sd}}}_{\text{Bayes}})}^2$ is available for possibly nonlinear parameters t(α). As before, let p_i be the weight on α_i, with p_i equaling 1/B or (3.7) in Sections 2 and 3, respectively, giving t̄ = Σp_it_i and ᾱ = Σp_iα_i. Define $s_i=\sqrt{p_i}(t_i-\overline{t})$ and matrix M,

$$M=\text{diag}(p_i^{1/2})\stackrel{\sim }{\mathit{\alpha }}{V}_{\widehat{\alpha }}{\stackrel{\sim }{\mathit{\alpha }}}^{\top }\text{diag}(p_i^{1/2}),$$ (6.8)

where α̃ has rows α_i − ᾱ and V_α̂ is as in (3.11). The spectral decomposition M = ΓdΓ′ has p = rank(α̃) nonzero eigenvalues d_i, with corresponding eigenvectors Γ_i, giving, after straightforward calculations,

$${\left(\frac{{\widehat{\text{sd}}}_{\text{freq}}}{{\widehat{\text{sd}}}_{\text{Bayes}}}\right)}^2=\frac{{\sum }_1^p{d}_i{s_i}^2}{{\sum }_1^p{s}_i^2}\left(s_i={\mathit{s}}^{\top }{\mathrm{\Gamma }}_i\right)$$ (6.9)

for θ̂ = Σp_it_i; the ratio can range from a high of d₁ to a low of d_p, depending on how t(α) aligns with the eigenvectors of M.

Remark 2. A computational verification of Lemma 1

Working directly with the implementation values μ_i, α_i, and t_i (2.11)–(2.12), we can verify Lemma 1 in the form in which it is actually used computationally. For x̃ any point in the sample space of the sufficient statistic, define

$$W_{\mu }(\stackrel{\sim }{x})=f_{\mu }(\stackrel{\sim }{x})/f_{\mu }(x),$$ (6.10)

x the observed statistic. Letting x̃ = x + dx with dx → 0,

$$W_{\mu }(\stackrel{\sim }{x})=\frac{f_{\mu }(x)+f_{\mu }^{\prime }(x)dx+o(dx)}{f_{\mu }(x)}=1+{\alpha }_x(\mu )dx+r(x)$$ (6.11)

where the remainder r(x) = o(dx)/f_μ(x). Here we are assuming that f_μ(x) has continuous gradient $f_{\mu }^{\prime }(\stackrel{\sim }{x})$ in a neighborhood of x, and that f_μ(x) > 0.

The importance sampling estimate of E{t(μ)|x̃} is

$$\begin{array}{l}\widehat{\theta }(\stackrel{\sim }{x})=\sum _{i=1}^B{t}_i{W}_i(\stackrel{\sim }{x})/\sum _{i=1}^B{W}_i(\stackrel{\sim }{x})\\ =\frac{\sum t_i(1+{\alpha }_{i}dx+r_i)}{\sum (1+{\alpha }_{i}dx+r_i)},\end{array}$$ (6.12)

with W_i = W_μi (x̃), α_i = α_x(μ_i), and r_i = o_i(dx)/f_μi (x). Denoting t̄ = Σt_i/B, $\overline{t\alpha }=\sum t_i{\alpha }_i/B$ etc., (6.12) gives

$$\widehat{\theta }(x+dx)=\frac{\overline{t}[1+(\overline{t\alpha }/\overline{t})dx+\overline{tr}/\overline{t}]}{1+\overline{\alpha }dx+\overline{r}}.$$ (6.13)

Since t̄ = θ̂(x) and $\overline{tr}$ and r̄ are o(dx), letting dx → 0 yields

$$\begin{array}{l}\widehat{\theta }(x+dx)=\widehat{\theta }+(\overline{t\alpha }/\overline{t}\overline{\alpha })dx+o(dx)\\ =\widehat{\theta }+\widehat{\text{cov}}·dx+o(dx),\end{array}$$ (6.14)

$\widehat{\text{cov}}$ as in (2.13). This verifies Lemma 1 as employed in the computational form of Theorem 1: $\widehat{\text{sd}}={({\widehat{\text{cov}}}^{\top }{V}_{\widehat{\mu }}\widehat{\text{cov}})}^{1/2}$ (3.11).

Remark 3. An alternative form of Lemma 1

Lemma 1 assumes the computational form ${\nabla }_{\widehat{\beta }}\widehat{\theta }=\widehat{\text{cov}}(t,\alpha )$ (3.10) in an exponential family (3.1). Defining

$$O_i=Q_i/\overline{Q}-P_i/\overline{P}$$ (6.15)

as in (3.12), an equivalent expression for $\widehat{\text{cov}}$ turns out to be

$$\widehat{\text{cov}}=\widehat{\theta }·{cov}_{\ast }(O,\alpha ),$$ (6.16)

where cov_* is the usual unweighted bootstrap covariance

$${cov}_{\ast }=\sum _{i=1}^B({\alpha }_i-\overline{\alpha })O_i/B\left[\overline{\alpha }=\sum _1^B{\alpha }_i/B\right].$$ (6.17)

(Notice that $\overline{O}=0$.) This leads to a convenient formula for the frequentist coefficient of variation $\widehat{\text{sd}}/\mid \widehat{\theta }\mid$ of θ̂,

$$\widehat{\text{cv}}={\left({cov}_{\ast }^{\top }{V}_{\widehat{\alpha }}{cov}_{\ast }\right)}^{1/2};$$ (6.18)

as compared with the internal $\text{cv}{\text{sd}}_{\ast }(O)/\sqrt{B}$ (3.12).

Remark 4. Bias correction for $\widehat{\text{sd}}$

Monte Carlo calculation of $\widehat{\text{sd}}$, either by MCMC or bootstrap methods, can be improved by a downward internal bias correction. Define Ŏ_i = θ̂O_i (6.15), ${\stackrel{⌣}{\alpha }}_i=V_{\widehat{\alpha }}^{1/2}{\alpha }_i$, and vector

$$C_B={cov}_{\ast }\left(\stackrel{⌣}{O},\stackrel{⌣}{\alpha }\right)=\sum _{i=1}^B{\stackrel{⌣}{\alpha }}_i{\stackrel{⌣}{O}}_i/B.$$ (6.19)

Then formula (6.18) can be reexpressed as

$${\widehat{\text{sd}}}^2={\Vert C_B\Vert }^2.$$ (6.20)

Let C_∞ denote the limit of C_B as the number of parametric bootstrap replications B → ∞. The last expression in (6.19) suggests that C_B has approximate bootstrap expectation and covariance

$$C_B\sim (C_{\infty },D_B),$$ (6.21)

with D_B the component of covariance from stopping at B replications rather than going on to infinity. Combining (6.20) and (6.21) gives

$${\widehat{\text{sd}}}^2={\widehat{\text{sd}}}_{\infty }^2+\text{tr}(D_B)$$ (6.22)

( ${\widehat{\text{sd}}}_{\infty }$ being the ideal sd estimate when B → ∞), indicating an upward bias in $\widehat{\text{sd}}$.

The bias-corrected sd estimate for θ̂ is given by

$${\stackrel{⌣}{\text{sd}}}^2={\widehat{\text{sd}}}^2-\text{tr}(D_B).$$ (6.23)

Jackknife calculations provide a convenient estimate of tr(D_B): the B bootstrap replications are divided into J groups of B/J each (e.g., J = 20); C_Bj is computed as in (6.19) but with the jth group of replications removed, giving the J × p matrix C with rows C_Bj; finally the p × p sample covariance matrix of C gives the estimate

$$\text{tr}(D_B)=\frac{{(J-1)}^2}{J}\text{tr}(\text{cov}\mathit{C}).$$ (6.24)

D_B decreases at rate 1/B, and the large choices of B in our examples made the bias correction (6.23) insignificant.

Remark 5. Binomial deviance difference

The binomial GLM for the cell infusion data analysis (3.17)–(3.18), has half deviance difference

$$\mathrm{\Delta }=\sum _{j,k=1}^5\left\{({\eta }_{jk}-{\widehat{\eta }}_{jk})\left({\xi }_{jk}+{\widehat{\xi }}_{jk}\right)-2log\left[(1+e^{{\eta }_{jk}})/\left(1+e^{{\widehat{\eta }}_{jk}}\right)\right]\right\},$$ (6.25)

where η_jk = log(ξ_jk/(1 − ξ_jk)). Here we have suppressed subscript i.

Remark 6. A vector parameter example

The joint frequentist behavior of the 0.90 credible interval endpoints [0.292, 0.380] in Figure 2 involved the vector parameter form (2.20) of the general accuracy formula, carried out by the bootstrap sampling method of Section 3.

With I_c(γ) the indicator function of γ ≤ c, we define the bivariate parameter replication t_i = (I_2.92(γ_i), I_3.80(γ_i)) for i = 1, 2,…, B = 2000. Then $\widehat{cov}$ (3.10) is a 2 × 2 matrix, as is $\widehat{var}$ (3.11). The weighted bootstrap density f̂(γ) had numerical derivatives (d_lo, d_up) = (0.466, 0.330) at the interval endpoints;

$${\left(\begin{array}{cc}{d}_{\text{lo}}& 0\\ 0& d_{\text{up}}\end{array}\right)}^{-1}\widehat{var}\left(\begin{array}{cc}{d}_{\text{lo}}& 0\\ 0& d_{\text{up}}\end{array}\right)=\left(\begin{array}{cc}.0476& .0678\\ .0678& .0968\end{array}\right)$$ (6.26)

is the usual delta-method covariance matrix estimate for the endpoints, giving them frequentist standard deviations 0.218 and 0.311, and correlation 0.999.

Remark 7. Abc calculations for the diabetes data

The abc algorithm (DiCiccio and Efron, 1992) provides second-order accurate confidence intervals for scalar parameters θ = T(β) in p-parameter exponential families (3.1) It does this by recomputing the MLE θ̂ = T(β̂) for values of b near β̂ (only 4p+4 recomputations are needed), calculating 2p+2 numerical second derivatives, and using these to make second-order adjustments to the standard intervals $\widehat{\theta }\pm c·\widehat{\text{sd}}$. An R version of abc is available from the author.

The solid bars in Figure 3 are abc intervals for the point estimates

$${\widehat{\theta }}_c=\widehat{Pr}\left\{{\gamma }_{125}\le c\mid \widehat{\beta }\right\}$$ (6.27)

(2.29). Here Ĝ (4.10) was the p-parameter exponential family, p = 10, with α_i (2.26) the B = 10, 000 MCMC vectors, weights p_i = 1 in (4.8). Taking Ĝ’s reversed roles of α and β into consideration, the abc call was

$$\text{ab}c(\text{TT},\text{ahat},\mathrm{S},\text{bhat},\text{mu})$$ (6.28)

where mu was the function

$$\text{mu}(b)=\sum _{i=1}^B{W}_i(b){\alpha }_i/\sum _{i=1}^B{W}_i(\beta )\left[W_i(b)=e^{{\left(b-\widehat{\beta }\right)}^{\top }{\alpha }_i}\right],$$ (6.29)

bhat= β̂ = X^⊤y, ahat=mu(bhat), and S the p × p covariance matrix of the α_i; TT was the function

$$\text{TT}(a)=\sum _{i=1}^b{W}_i(b)t_{ci}/\sum _{i=1}^B{W}_i(b),b={\text{mu}}^{-1}(a),$$ (6.30)

where t_ci = t_c(α_i) (2.31), while mu⁻¹(·) was the inverse function of mu(·), calculated to accuracy 10⁻¹¹ using Newton–Raphson iteration. (The inversion is necessary because θ̂ = s(b) (4.12) is a function of the natural parameter b of Ĝ, but abc requires θ̂ stated in terms of the expectation parameter, a in the case of Ĝ.)

Table 6 displays a portion of the abc output going into Figure 3. Besides θ̂ and $\widehat{\text{sd}}$, it shows the three second-order correction coefficients described in DiCiccio and Efron (1992): acceleration a and bias-correction z₀ are mostly ignorable, but the quadratic coefficient c_q is not. It has a major effect on the abcq limits, a version of abc that is purely local in the sense of only recomputing T(b) for b near β̂.

Table 6.

abc calculations for the diabetes data, Figure 3; (a, z₀, c_q) are the three coefficients that adjust the standard limits $\widehat{\theta }\pm \widehat{\text{sd}}$ to second-order accuracy (DiCiccio and Efron, 1992). The abc limits, columns 7 and 8, were not much different than the purely local abcq limits, columns 9 and 10.

c	θ̂	$\widehat{\text{sd}}$	a	z0	cq	abc		abcq
						lo	up	lo	up
.04	.00	.01	.00	.27	1.50	.00	.06	.00	.03
.08	.01	.03	.01	.21	1.17	.00	.13	.01	.09
.12	.04	.08	.00	.12	.88	.00	.25	.02	.22
.16	.11	.19	.00	.05	.60	.02	.44	.03	.45
.2	.25	.32	.00	.02	.33	.05	.63	.04	.68
.24	.46	.40	.00	−.02	.05	.13	.80	.08	.86
.28	.67	.36	.00	−.02	−.23	.28	.92	.23	.95
.32	.84	.24	.00	−.03	−.50	.49	.98	.47	.96
.36	.94	.12	.00	−.03	−.78	.71	.99	.73	.97

The abc limits in Figure 3 involve one nonlocal recomputation. They enjoy tranformation invariance, monotone transformations of the parameter of interest producing the same transformation of the interval endpoints, which might be helpful for parameters like θ_c restricted to interval [0, 1]. However in this case they were not much different than the abcq versions.

Remark 8. Tweedie’s formula for the prostate data

Both Bayes and empirical Bayes hierarchical analyses require evaluation of t_i = E{τ(δ₀)|z₀, α_i, β̂} (5.21) for i = 1, 2,…, B. This is straightforward when τ(δ) = δ as in Figure 6. Tweedie’s formula (Efron, 2011) says that

$$E\{{\delta }_0{\mid z_0,\alpha \}=z_0+\frac{d}{dz}log{f}_{\alpha }(z)\mid }_{z_0},$$ (6.31)

where f_α (z) is the marginal density (5.4). In terms of notation (5.11)–(5.12),

$$t_i=c_{j0}+{\stackrel{.}{\mathit{x}}}_{j_0}{\alpha }_i,$$ (6.32)

where j₀ is the bin index (5.9) for z₀, and

$${\stackrel{.}{\mathit{x}}}_j=(0,1,2c_j,3c_j^2,\dots ,8c_j^7).$$ (6.33)

Theoretically there is a version of Tweedie’s formula applying to any function τ(δ) (called “Bayes rule in terms of f” in Efron, 2013). The case τ (δ) = δ, however, is particularly favorable to GLM modeling of the marginal density f(z) (5.4). Other choices of τ(δ) may require non-GLM models for f, returning hierarchical Bayes analysis to the general, nonexponential family framework of Section 2.

Remark 9. Empirical Bayes sd formula

The empirical Bayes standard deviation formula (5.27) is easy to derive in exponential families. We assume, for convenience, that the sufficient statistic x takes on only a finite number J of possible values, so that the marginal density f_α(·) is represented by a J-vector f_α. Let Q be the gradient of t(α) = E{τ(δ₀)|z₀, α} with respect to f (specific formulas for Q are given in Efron, 2013), and ${\stackrel{.}{\mathit{f}}}_{\alpha }$ the J × p derivative matrix (∂f_αj/∂α_k). Then a first-order Taylor expansion gives

$$t(\widehat{\alpha })-t(\alpha )\doteq {\mathit{Q}}^{\top }{\stackrel{.}{\mathit{f}}}_{\alpha }(\widehat{\alpha }-\alpha ).$$ (6.34)

This yields

$$\text{sd}{(t(\widehat{\alpha }))}^2\doteq {\mathit{Q}}^{\top }{\stackrel{.}{\mathit{f}}}_{\alpha }{\mathrm{\sum }}_{\alpha }{\stackrel{.}{\mathit{f}}}_{\alpha }\mathit{Q}[{\mathrm{\sum }}_{\alpha }={cov}_{\alpha }(\widehat{\alpha })]$$ (6.35)

and

$$cov(t(\widehat{\alpha }),\widehat{\alpha })\doteq {\mathit{Q}}^{\top }{\stackrel{.}{\mathit{f}}}_{\alpha }{\mathrm{\sum }}_{\alpha },$$ (6.36)

so

$$\text{sd}{(t(\widehat{\alpha }))}^2\doteq cov{(t(\widehat{\alpha }),\widehat{\alpha })}^{\top }{\mathrm{\sum }}_{\alpha }^{-1}cov(t(\widehat{\alpha }),\widehat{\alpha }).$$ (6.37)

But ${\mathrm{\sum }}_{\alpha }^{-1}\doteq {cov}_{\alpha }(\widehat{\beta })=V_{\alpha }$ in exponential families, giving

$$\text{sd}{(t(\widehat{\alpha }))}^2\doteq cov{(t(\widehat{\alpha }),\widehat{\alpha })}^{\top }{V}_{\alpha }cov(t(\widehat{\alpha }),\widehat{\alpha }).$$ (6.38)

Formula (5.27) for $\overline{\text{sd}}$ is the bootstrap evaluation of (6.38).