High dimensional Bernstein-von Mises: simple examples

Abstract

In Gaussian sequence models with Gaussian priors, we develop some simple examples to illustrate three perspectives on matching of posterior and frequentist probabilities when the dimension p increases with sample size n: (i) convergence of joint posterior distributions, (ii) behavior of a non-linear functional: squared error loss, and (iii) estimation of linear functionals. The three settings are progressively less demanding in terms of conditions needed for validity of the Bernstein-von Mises theorem.

The Bernstein-von Mises theorem is a formalization of conditions under which Bayesian posterior credible intervals agree approximately with frequentist confidence intervals constructed from likelihood theory. It is traditionally formulated in situations in which the number of parameters p is fixed and the sample size n → ∞. The situation is very different in high dimensional settings in which p is allowed to grow with n. In this primarily expository paper, we use simple Gaussian sequence models to draw some conclusions about when a version of Bernstein-von Mises can hold.

We begin with a somewhat informal statement of the classical theorem. Suppose that Y₁, …, Y_n are i.i.d. observations from a distribution P_θ having density p_θ(y)dμ(y) where $\theta \in \Theta \subset {\mathbb{R}}^p$. The log-likelihood for a single observation

$${\ell }_{\theta }=\log p_{\theta }\left(y\right),$$

and, as usual, the score function vector and Fisher information matrix are given by

$${\stackrel{.}{\ell }}_{\theta }=(\partial ∕\partial \theta )\log p_{\theta }\left(y\right)I_{\theta }=E_{\theta }{\stackrel{.}{\ell }}_{\theta }{\stackrel{.}{\ell }}_{\theta }^T.$$

Writing Yⁿ = (Y₁, …, Y_n) for the full data, the log-likelihood

$$L_n\left(\theta \right)=\sum _{k=1}^n{\ell }_{\theta }\left(Y_k\right),$$

and we write ${\widehat{\theta }}_{\mathit{MLE}}$ for a maximizer of L_n(θ). Classical likelihood theory says that any (nice) estimator satisfies the information bound

$$Va{r}_{\theta }\widehat{\theta }\ge n^{-1}{I}_{\theta }^{-1},$$

in the usual ordering of nonnegative definite matrices, and that the bound is asymptotically attained by the MLE, which is also asymptotically Gaussian:

$${\widehat{\theta }}_{MLE}\mid \theta ~N_p(\theta ,n^{-1}{I}_{\theta }^{-1}).$$

Now suppose that π(θ) is the density of a prior distribution with respect to Legesgue measure. Then the posterior distribution of θ given Yⁿ is given by Bayes' rule; we denote it simply by P_θ|Yⁿ.

The Bernstein-von Mises theorem says, informally, that this posterior distribution is, in large samples, approximately normal with mean approximately the MLE, ${\widehat{\theta }}_{\mathit{MLE}}$ and variance matrix approximately $n^{-1}{I}_{{\theta }_0}^{-1}$ (here θ₀ is the `true' value of θ generating the observations Y₁, …, Y_n). Using the scalar case for simplicity, and writing ${\sigma }_n^2=n^{-1}{I}_{{\theta }_0}^{-1}$ and $z_{\alpha }={\tilde{\Phi }}^{-1}\left(\alpha \right)$, we have that an approximate 100(1 – α)% credible interval for θ would be given by ${\widehat{\theta }}_{\mathit{MLE}}\pm z_{\alpha ∕2}{\widehat{\sigma }}_n$. This is exactly the same as the frequentist confidence interval based on asymptotic normality of the MLE. Thus in large samples the effect of the prior density π disappears: “the data overwhelms the prior”.

A somewhat more formal statement uses the notion of variation distance between probability measures P and Q, and an equivalent expression in terms of the densities p = dP/dμ and q = dQ/dμ relative to a dominating measure μ:

$$\parallel P-Q\parallel =\underset{A}{\max }\mid P\left(A\right)-Q\left(A\right)\mid =\frac{1}{2}\int \mid p-q\mid d\mu .$$

Suppose that π(θ) is continuous and positive at the `true' value θ₀, and that θ → P_θ is differentiable in quadratic mean and satisfies a further mild separation condition, then

$$\parallel P_{\theta \mid Y^n}-N({\widehat{\theta }}_{MLE},n^{-1}{I}_{{\theta }_o}^{-1})\parallel \to 0.$$ (1)

in probability under $P_{{\theta }_0}^n$.

In other words, the variation distance between posterior and the approximating Gaussian distribution is a random variable depending on Yⁿ, and which converges to zero in probability under repeated draws from P_θ0.

A development of the Bernstein-von Mises theorem as formulated above may be found in van der Vaart (1998, §10.2). A proof due to Bickel is given in Lehmann and Casella (1998, §6.8). Extension from independent to dependent sampling settings are possible, see e.g. Borwanker et al. (1971); Heyde and Johnstone (1979). For further references and methods of proof of the classical results, see Ghosh and Ramamoorthi (2003, §1.4 and §1.5).

1. Growing Gaussian location model

In nonparametric and semiparametric settings the situation is very different. Even frequentist consistency of nonparametric Bayesian methods is a difficult issue with a large literature of both positive and negative results (e.g. Ghosh and Ramamoorthi (2003); Ghosal and van der Vaart, (2010)). One cannot therefore expect Bernstein-von Mises phenomena in any great generality for the full posterior.

In this largely expository paper, we do some simple calculations in symmetric Gaussian sequence models. The Gaussian sequence structure makes possible an elementary set of examples that avoid the technical challenges posed by, and sophistication needed for, posterior Gaussian approximation in high dimensional settings (see references in Section 5). Nevertheless, the Gaussian examples can conveniently illustrate some of the issues related to validity of the Bernstein-von Mises theorem in high dimensional models. Depending on the frequentist or Bayesian perspective, we assume that p = p(n) grows with n, and one, or both, of

$$\begin{array}{cc}\hfill & \left(\mathbf{D}\right)\mathrm{Data}:\stackrel{‒}{Y}\mid \theta ~N_p(\theta ,{\sigma }_n^{2}I),\mathrm{and}\hfill \\ \hfill & \left(\mathbf{P}\right)\mathrm{Prior}:\theta ~N_p(0,{\tau }_n^{2}I).\hfill \end{array}$$

The notation $\stackrel{‒}{Y}$ suggests an average (Y₁+⋯+Y_n)/n of observations individually of variance ${\sigma }_0^2$, so that in this case ${\sigma }_n^2={\sigma }_0^2∕n$. [If p were held fixed, not depending on n, then ${\sigma }_n^2$ would match with the definition given in the introductory section.] We also allow the prior variance ${\tau }_n^2$ to depend on the sample size n.

Our goal is to compare the Bayesian posterior distribution $\mathcal{L}(\theta \mid Y)$ with frequentist distributions, in particular those of the MLE $\mathcal{L}({\widehat{\theta }}_{\mathit{MLE}}\mid \theta )$ and of the posterior mean Bayes estimator $\mathcal{L}({\widehat{\theta }}_B\mid \theta )$. A key simplification is that since both prior and likelihood are Gaussian, so also is the posterior distribution, and hence all the behavior will be determined by centering and scaling. Thus from standard results, the posterior is given by

$$\begin{array}{cc}\hfill \theta \mid \stackrel{‒}{Y}& =\stackrel{‒}{y}~N_p(w_{n}y,w_n{\sigma }_n^{2}I),\hfill \\ \hfill & w_n={\tau }_n^2∕({\sigma }_n^2+{\tau }_n^2).\hfill \end{array}$$ (2)

Remarks: 1. The reference to Gaussian sequence models becomes clearer if, as will be helpful later, we write out assumptions (D) and (P) in co-ordinates:

$$\begin{array}{cc}\hfill & \left({\mathbf{D}}_{\mathrm{seq}}\right)\mathrm{Data}:{\stackrel{‒}{y}}_k={\theta }_k+{\sigma }_n{∊}_k,and\hfill \\ \hfill & \left({\mathbf{P}}_{\mathrm{seq}}\right)\mathrm{Prior}:{\theta }_k={\tau }_n{\zeta }_k,\hfill \end{array}$$

with ε_k and ζ_k all i.i.d standard Gaussian, for k = 1, …, p(n).

Strictly speaking, the indexing by n of parameters σ_n, τ_n and p(n) creates a sequence of sequence models. However, one can, as needed for almost sure results, think of the infinite sequences {$({∊}_k,{\zeta }_k),k\in \mathbb{N}$} as being drawn from a single common probability space.

2. We also consider the infinite sequence Gaussian white noise model

$$Y_t={\int }_0^{t}f\left(s\right)ds+{\sigma }_n{W}_{t}0\le t\le 1$$ (3)

or equivalently, when expressed in any orthonormal basis {φ_k(t)} for L₂[0, 1],

$$y_k={\theta }_k+{\sigma }_n{∊}_k{∊}_k\stackrel{\mathrm{ind}}{~}N(0,1),$$ (4)

where it is assumed that ${\sigma }_n={\sigma }_0∕\sqrt{n}$. For some examples, it is helpful to use doubly indexed orthonormal bases {${\phi }_{jk}\left(t\right),k=1,\dots ,2^j,j\in \mathbb{N}$} such as arise with systems of orthonormal wavelets.

The forthcoming book Johnstone (2010) will have more on estimation in such Gaussian sequence models.

We develop three perspectives on the Bernstein-von Mises phenomenon:

1. global convergence of the posterior,

2. behavior of a non-linear functional ${\Vert \theta -\widehat{\theta }\Vert }^2$, and of

3. linear functionals Lf, in the white noise model (3) – (4).

We shall see that these situations are progressively “less demanding” in terms of validity of the Bernstein-von Mises phenomenon. Indeed, case (1) requires that w_n → 1 at a sufficiently fast rate, while setting (2) needs only w_n → 1. In case (3), the formulation itself delivers w_n → 1, and covers at least all bounded linear functionals.

2. Global convergence of posterior

The first calculation considers the p—dimensional posterior distribution (2) and shows that the convergence in (1) occurs, even in the best possible case that θ₀ = 0, only if the shrinkage factor w_n approaches 1 at a sufficiently fast rate.

Proposition 1

Let θ₀ = 0. The variation distance between posterior distribution $P_{\theta \mid Y^n}$ and $N({\widehat{\theta }}_{\mathit{MLE}},n^{-1}{I}_{{\theta }_0}^{-1})$ converges to zero inP_θ0 — probability if and only if $\sqrt{p}{\sigma }_n^2∕{\tau }_n^2\to 0$, or equivalently, if

$$w_n=1-o(1∕\sqrt{p_n}).$$ (5)

PROOF. We introduce notation P_y,n(dθ) for the posterior distribution of θ|Yⁿ = y and Q_y,n(dθ) for the distribution centered at ${\widehat{\theta }}_{\mathit{MLE}}=\stackrel{‒}{y}$. Thus

$$\begin{array}{cc}\hfill & P_{y,n}\left(d\theta \right)\leftrightarrow N_p(w_n\stackrel{‒}{y},{\sigma }_n^2{w}_{n}I)\hfill \\ \hfill & Q_{y,n}\left(d\theta \right)\leftrightarrow N_p(\stackrel{‒}{y},{\sigma }_n^{2}I).\hfill \end{array}$$ (6)

Let $\rho (P,Q)=\int \sqrt{p}\sqrt{q}d\mu$ denote the Hellinger affinity between two probability measures P,Q having densities p, q with respect to a common dominating measure μ. We recall an elementary bound (van der Vaart, 1998, p. 212) for variation distance in terms of Hellinger distance and hence Hellinger affinity:

$$2[1-\rho (P,Q)]\le \parallel P-Q\parallel \le \sqrt{8}{[1-\rho (P,Q)]}^{1∕2}$$ (7)

Thus ∥P_y,n − Q_y,n∥ → 0 if and only if ρ(P_y,n, Q_y,n) → 1. We recall also that affinity commutes with products:

$$\rho (\Pi P_i,\Pi Q_i)=\Pi \rho (P_i,Q_i).$$

An elementary calculation shows that

$${\rho }^2(N({\theta }_1,{\sigma }_1^2),N({\theta }_2,{\sigma }_2^2)=\left(\frac{2{\sigma }_1{\sigma }_2}{{\sigma }_1^2+{\sigma }_2^2}\right)\exp \{-\frac{{({\theta }_1-{\theta }_2)}^2}{2({\sigma }_1^2+{\sigma }_2^2)}\}.$$ (8)

When applied to P_y,n and Q_y,n, we set ${\theta }_{1_i}=w_n{\stackrel{‒}{y}}_i$, ${\theta }_{2_i}={\stackrel{‒}{y}}_i$ and ${\sigma }_1^2={\sigma }_n^2{w}_n$, ${\sigma }_2^2={\sigma }_n^2$ to obtain

$$\rho (P_{y,n},Q_{y,n})=\exp \{-\frac{p}{2}\log \frac{1}{2}(w_n^{1∕2}+w_n^{-1∕2})-\frac{{(1-w_n)}^2{\parallel \stackrel{‒}{y}\parallel }^2}{4(1+w_n){\sigma }_n^2}\}.$$ (9)

Introduce $r_n={\sigma }_n^2∕{\tau }_n^2=w_n^{-1}-1$. Suppose first that $p{r}_n^2\to 0$. Since w_n = (1 + r_n)⁻¹, we have

$$\log \frac{1}{2}(w_n^{1∕2}+w_n^{-1∕2})=-\frac{1}{2}\log (1+r_n)+\log (1+\frac{1}{2}{r}_n)\le 2c_1{r}_n^3$$

for $r_n\le \frac{1}{2}$, say. When p → ∞, we have with probability tending to one that ${\Vert \stackrel{‒}{y}\Vert }^2∕{\sigma }_n^2<2p$, and so for $r_n\le \frac{1}{2}$,

$$\frac{{(1-w_n)}^2}{4(1+w_n)}\frac{{\parallel \stackrel{‒}{y}\parallel }^2}{{\sigma }_n^2}\le \frac{2p{r}_n^2}{(1+r_n)(1+r_n∕2)}\le c_{2}p{r}_n^2.$$

Consequently, when $p{r}_n^2\to 0$,

$$\rho (P_{y,n},Q_{y,n})\ge \exp \{-c_{1}p{r}_n^3-c_{2}p{r}_n^2\}\to 1.$$

Suppose now that $p{r}_n^2$ does not approach 0. Again with probability tending to one, ${\Vert \stackrel{‒}{y}\Vert }^2∕{\sigma }_n^2>p∕2$, and since $\frac{1}{2}(w_n^{1∕2}+w_n^{-1∕2})>1$, we have from (9) that

$$-\log \rho (P_{y,n},Q_{y,n})>\frac{p{(1-w_n)}^2}{8(1+w_n)}>c_3\min \{p{r}_n^2,p\}$$

which cannot converge to zero if $p{r}_n^2$ does not.

Remark. If θ₀ = θ_0n ≠ 0, so that the data mean differs from the prior mean, then the rate condition is replaced by

$$w_n-1=o(1∕q_n\left({\theta }_{0n}\right)),q_n\left({\theta }_{0n}\right)=\sqrt{p_n}+\Vert {\theta }_{0n}\Vert ∕{\sigma }_n.$$

Example. We illustrate the result by considering estimation in the Gaussian white noise model (3). When expressed in a suitable orthonormal basis of wavelets, we obtain $y_{jk}\stackrel{\underset{~}{ind}}{}N({\theta }_{j,k},{\sigma }_n^2)$, for k = 1, …, 2^j, and $j\in \mathbb{N}$. Pinsker's theorem (Pinsker, 1980) describes the minimax linear estimator of f, or equivalently of (θ_jk), under squared error loss when it is assumed that f has α mean square derivatives and shows that such minimax linear estimators are asymptotically minimax among all estimators as σ_n → 0.

Pinsker's estimator is necessarily posterior mean Bayes for a corresponding Gaussian prior. The mean square differentiability condition can be equivalently expressed in terms of the coefficients as

$$\sum _{j,k}{2}^{2j\alpha }{\theta }_{j,k}^2\le C^2,$$

and the corresponding least favorable Gaussian prior puts

$${\theta }_{j,k}\stackrel{\mathrm{ind}}{~}N(0,{\tau }_j^2){\tau }_j^2={\sigma }_n^2{({\mu }_n{2}^{-j\alpha }-1)}_{+},$$ (10)

where μ_n = c_αn(C/σ_n)^2α/(2α+1). The constant c_αn satisfies bounds independent of n, c_1α ≤ c_αn ≤ c_2α, whose precise values are unimportant here–for further details see Johnstone (2010).

We consider the validity of the Bernstein-von Mises phenomenon for the collection of coefficients {θ_jk, k = 1, …, 2^j} at a given level j = j(n)–possibly fixed, or possibly varying with n.

The prior variances ${\tau }_j^2$ decrease with j, and vanish above a “critical level” j_* = j_*(α, C;n). Since j_* ~ (2/(2α+1)) log(C/σ_n) grows with n, so does the number of parameters θ_j*,k at the critical level. From (10), we conclude that

$${\tau }_{j\ast }^2∕{\sigma }_n^2\le 2^{\alpha }-1,$$

and hence that w_n ≤ 1 − 2^−α does not approach 1, so that the condition of Proposition 1 fails.

On the other hand, at a fixed level j₀, we have p = 2^jo fixed and ${\tau }_{j_0}^2∕{\sigma }_n^2={\mu }_n{2}^{-j\ast \alpha }-1\to \infty$, so that $\sqrt{p}{\sigma }_n^2∕{\tau }_{j_0}^2\to 0$ and so Proposition 1 applies. Thus we may say informally that the Bernstein-von Mises phenomenon holds at a fixed level but fails at the critical level.

3. Behavior of the squared loss

In this section, we pay homage to a remarkable paper by Freedman (1999), itself stimulated by Cox (1993), which sets out the failure of the Bernstein-von Mises theorem in a simple sequence model of function estimation in Gaussian white noise. To further simplify the calculations, we use the growing Gaussian location model (D), (P), yielding results parallel to, but not identical with, Freedman's. Hence, define

$$T_n(\theta ,Y)={\parallel \theta -{\widehat{\theta }}_B\parallel }^2=\sum _{k=1}^{p\left(n\right)}{({\theta }_k-{\widehat{\theta }}_k)}^2.$$

The posterior distribution of θ|Y is described by (2); in particular the shrinkage factor $w_n={\tau }_n^2∕({\sigma }_n^2+{\tau }_n^2)$ again plays a critical role.

Theorem 2 (Bayesian) The posterior distribution $\mathcal{L}(T_n\mid Y)$ is given by

$$T_n=C_n+\sqrt{D_n{Z}_{1n}},$$

where

$$C_n=p{\sigma }_n^2{w}_n$$ (11)

$$\sqrt{D_n}=\sqrt{2p{\sigma }_n^2{w}_n}$$ (12)

and the random variable Z_1n has mean 0, variance 1 and converges in distribution to N(0, 1) as n → ∞.

Proof. From (2), the posterior distribution of T_n given Y is ${\sigma }_n^2{w}_n{\chi }_{\left(p\right)}^2$ and in particular it is free of Y. Hence we have the representation

$$T_n=p{\sigma }_n^2{w}_n+\sqrt{2p{\sigma }_n^2}{w}_n{Z}_{1}n,$$

and the theorem follows because $({\chi }_p^2-p)∕\sqrt{2p}\Rightarrow N(0,1)$ as p → ∞.

Turn now to the frequentist perspective, in which θ is a fixed and unknown (sequence of) parameters. We will therefore use the decomposition y_k = θ_k + σ_nε_k with ${∊}_k\stackrel{\underset{~}{iid}}{}N(0,1)$, c.f. (D_seq) above. Since ${\widehat{\theta }}_{B,k}=w_n{y}_k$ we have

$${\theta }_k-{\widehat{\theta }}_{B,k}=(1-w_n){\theta }_k-w_n{\sigma }_n{∊}_k.$$ (13)

Some of the conclusions will be valid only for “most” θ: to formulate this it is useful to give θ a distribution. The natural one to use is (P), despite the possible confusion arising because, for the frequentist, this is not an a priori law!

Theorem 3 (Frequentist) The conditional distribution $\mathcal{L}(T_n\mid \theta )$ is given by

$$T_n=C_n+\sqrt{F_n}{Z}_{2n}\left(\theta \right)+\sqrt{G_n\left(\theta \right)}{Z}_{3n}(\theta ,∊),$$ (14)

where C_n is as in Theorem 2, while Z_3n (θ, ε) has mean 0 and variance 1.

If θ is distributed according to (P), then Z_2n(θ) has mean 0, variance 1 and converges in distribution to N(0, 1) as n → ∞. In addition, if w_n ∞ w = 1 − cosω,

$$\begin{array}{c}\hfill \sqrt{F_n}~\sqrt{D_n}\cos \omega \hfill \\ \hfill \sqrt{G_n\left(\theta \right)}~\sqrt{D_n}\sin \omega ,\hfill \end{array}$$ (15)

and

$$Z_{3n}(\theta ,\cdot )\Rightarrow N(0,1)$$ (16)

Formulas (15) and (16) hold as n → ∞, for almost all θ's generated from (P).

Proof. Using (13), and ${(1-w_n)}^2{\tau }_n^2+w_n^2{\sigma }_n^2={\sigma }_n^2{w}_n$, we may write

$$\begin{array}{cc}\hfill T_n& =\sum _k{[(1-w_n){\theta }_k-w_n{\sigma }_n{∊}_k]}^2\hfill \\ \hfill & =p{\sigma }_n^2{w}_n+\sqrt{2p{\tau }_n^2}{(1-w_n)}^2\cdot \frac{\Sigma {\theta }_k^2-p{\tau }_n^2}{\sqrt{2p{\tau }_n^2}}+R_n(\theta ,∊),\hfill \end{array}$$ (17)

with

$$R_n(\theta ,∊)=-2w_n(1-w_n){\sigma }_n\Sigma {\theta }_k{∊}_k+w_n^2{\sigma }_n^2\Sigma ({∊}_n^2-1)$$

This leads immediately to the representation (14) after observing that ${\tau }_n^2(1-w_n)={\sigma }_n^2{w}_n$ and setting

$$\begin{array}{cc}\hfill \sqrt{F_n}& =\sqrt{2p}{\sigma }_n^2{w}_n(1-w_n),\hfill \\ \hfill Z_{2n}\left(\theta \right)& =\Sigma ({\theta }_k^2-{\tau }_n^2)∕\sqrt{2p}{\tau }_n^2,\hfill \\ \hfill G_n\left(\theta \right)& =\mathrm{Var}{R}_n(\theta ,∊)=4w_n^2{(1-w_n)}^2{\sigma }_n^2\Sigma {\theta }_k^2+w_n^4\cdot 2p{\sigma }_n^4\hfill \\ \hfill & =G_{1n}\left(\theta \right)+G_{2n}\hfill \end{array}$$

Turning to the final assertions, we may rewrite

$$R_n(\theta ,∊)=\sqrt{G_{1n}\left(\theta \right)}{Z}_{4n}\left(\theta \right)+\sqrt{G_{2n}}{Z}_{5n},$$

where

$$Z_{4n}\left(\theta \right)=\Sigma {\theta }_k{∊}_k∕{\left(\Sigma {\theta }_k^2\right)}^{1∕2},Z_{5n}=(\Sigma {∊}_k^2-p)∕\sqrt{2p}.$$

Using again ${\sigma }_n^2{w}_n={\tau }_n^2(1-w_n)$, we have

$$G_{1n}\left(\theta \right)=2p{\sigma }_n^4\cdot w_n^2(2w_n-2w_n^2)\cdot p^{-1}\sum _1^p{({\theta }_k∕{\tau }_n)}^2.$$

For almost all θ's generated from (P), $p^{-1}{\Sigma }_1^p{({\theta }_k∕{\tau }_n)}^2\to 1$, and since G_n (θ) = G_1n(θ) + G_2n, (15) follows.

Clearly Z_4n(θ) ~ N(0, 1), free of θ, while Z_5n ⇒ N (0, 1) and so (16) follows.

Remark. The doctrinaire frequentist would not contemplate the joint distribution of (θ, Y ) in (D, P); but anyone else would observe that in that joint distribution, $T_n~{\sigma }_n^2{w}_n{\chi }_{\left(p\right)}^2$, as follows easily in two ways, either from the proof of Theorem 2, or from (17).

The Bernstein-von Mises theorem fails if lim w_n = w < 1, as may be seen in Figure 1. For the Bayesian, conditional on $Y,\theta -{\widehat{\theta }}_B$ is a noise vector, and Theorem 2 says that the distribution of ${\Vert \theta -{\widehat{\theta }}_B\Vert }^2$ is approximately normal with mean C_n and standard deviation $\sqrt{D_n}$. For the frequentist, $E[{\widehat{\theta }}_B\mid \theta ]=w_n\theta$ is biased (also asymptotically), and some of ${\Vert {\widehat{\theta }}_B-\theta \Vert }^2$ comes from this bias. As a result, Theorem 3 says that, conditional on $\theta ,{\Vert {\widehat{\theta }}_B-\theta \Vert }^2$ is approximately normal with mean $C_n+\sqrt{F_n}{Z}_{2n}\left(\theta \right)$ and standard deviation $\sqrt{G_n\left(\theta \right)}$. Comparing (12) and (15) shows that the frequentist SD is smaller than the Bayesian $\sqrt{G_n\left(\theta \right)}<\sqrt{D_n}$.

Fig 1.

The top panel, for the Bayesian, has $\theta -{\widehat{\theta }}_B$ as a noise vector, and the posterior distribution $\mathcal{L}(T\mid Y)$ approximately N(C_n, D_n). The bottom panel, for the frequentist, shows the effect of the bias of $E[{\widehat{\theta }}_B\mid \theta ]$ for θ, with $\mathcal{L}(T\mid \theta )$ approximately $N(C_n+\sqrt{F_n}{Z}_{2n}\left(\theta \right),G_{2n}\left(\theta \right))$.

Under the assumption (P), ${\theta }_i\stackrel{\underset{~}{iid}}{}N(0,{\tau }_n^2)$, the `wobble' in the frequentist mean be arbitrarily large relative to $\sqrt{D_n}$: from the law of the iterated logarithm, with probability one

$$\lim \inf Z_{2n}\left(\theta \right)∕\sqrt{2\log \log p}=1$$

.

By contrast, if lim w_n = 1, then the wobble disappears: $\sqrt{F_n}=o\left(\sqrt{D_n}\right)$ and the Bayesian SD equals the frequentist SD asymptotically: $\sqrt{G_n\left(\theta \right)}~\sqrt{D_n}$.

4. Linear Functionals

We turn now to the least demanding of our three scenarios for the Bernstein-von Mises theorem: the behavior of linear functionals. We change the setting slightly to the infinite sequence Gaussian white noise model (3). We consider linear functionals Lf such as integrals ${\int }_{B}f$ or derivatives f^(r) (t₀): if f has expansion $f\left(t\right)=\Sigma {\theta }_k{\phi }_k\left(t\right)$, then on setting $a_k=L{\phi }_k$, we have

$$Lf=\Sigma {\theta }_{k}L{\phi }_k=\Sigma {\theta }_k{a}_k.$$

Again, for maximum simplicity, we consider Gaussian priors on the coefficients:

$${\theta }_k\stackrel{\mathrm{ind}}{~}N(0,{\tau }_k^2).$$ (18)

In order that $\Sigma {\theta }_k^2<\infty$ with probability 1, it is necessary and sufficient that $\Sigma {\tau }_n^2<\infty$.

Consequently, the posterior laws are Gaussian

$${\theta }_k\mid y_k\stackrel{\mathrm{ind}}{~}N(w_{kn}{y}_k,w_{kn}{\sigma }_n^2),$$

again with

$$w_{kn}={\tau }_k^2∕({\sigma }_n^2+{\tau }_k^2)$$ (19)

so that the posterior mean estimate

$${\widehat{Lf}}_n=\sum _n{a}_k{w}_{kn}{y}_k.$$

Centering at posterior mean. For the Bayesian, the posterior distribution

$$Lf\mid y~N({\widehat{Lf}}_n,V_{yn}),V_{yn}={\sigma }_n^2\sum _k{a}_k^2{w}_{kn}$$

while for the frequentist, the conditional distribution

$${\widehat{Lf}}_n\mid f~N(E_f{\widehat{Lf}}_n,V_{Fn}),V_{Fn}={\sigma }_n^2\sum _k{a}_k^2{w}_{kn}^2.$$

The Bayesian might use 100(1 − α)% posterior credible intervals of the form ${\widehat{Lf}}_n\pm z_{\alpha ∕2}\sqrt{V_{yn}}$, while the frequentist might employ 100(1 − α)% confidence intervals ${\widehat{Lf}}_n\pm z_{\alpha ∕2}\sqrt{V_{Fn}}$. This leads us to conider the variance ratio

$$\frac{V_{Fn}}{V_{yn}}=\frac{\Sigma a_k^2{w}_{kn}^2}{\Sigma a_k^2{w}_{kn}}<1,$$ (20)

from which we see that the frequentist intervals are narrower–because the frequentist bias $E_{f}L\widehat{f}-Lf$ is being ignored for now, along with the attendant implications for coverage (but see below).

As sample size n → ∞, the noise variance ${\sigma }_n^2\to 0$ and so for a given Gaussian prior (18), the weights (19) converge marginally: w_kn → 1 for each fixed k. This alone does not imply convergence of the variance ratio V_Fn/V_yn → 1, as a later example shows. A sufficient condition is that the linear functional Lf be bounded (as a mapping from L₂[0, 1] to $\mathbb{R}$.) This amounts to saying that Lf has the representation $Lf={\int }_0^{1}a\left(t\right)f\left(t\right)dt$ with ∫ a²(t)dt < ∞, or equivalently, in sequence terms, that $\Sigma a_k^2\le \infty$.

Proposition 4

Let ${\mathbb{P}}_f^n$ denote the measure corresponding to (3). If Lf = ∫ af is a bounded linear functional, then the variation distance between Bayesian and frequentist distributions converges to zero:

$$\Vert N({\widehat{Lf}}_n,V_{yn})-N({\widehat{Lf}}_n,V_{Fn})\Vert \stackrel{{\mathbb{P}}_f^n}{\to }0.$$ (21)

Proof. We again use the Hellinger affinity (7) and apply (8) to the laws $P=N({\widehat{Lf}}_n,V_{yn})$ and $Q=N({\widehat{Lf}}_n,V_{Fn})$ to obtain

$${\rho }^2(P,Q)=\frac{2\sqrt{V_{Fn}∕V_{yn}}}{1+V_{Fn}∕V_{yn}}.$$

In view of (7), the merging in (21) occurs if and only if

$$V_{Fn}∕V_{yn}\to 1.$$

When $\Sigma a_k^2<\infty$, this convergence follows from (20) and the dominated convergence theorem.

Remarks. 1. Examples of bounded functionals include polynomials $a\left(t\right)={\Sigma }_{k=0}^K{c}_k{t}^k$ and “regions of interest” a(t) = I{t ε B}.

2. Examples of unbounded functionals are given by evaluation of a function (or its derivatives) at a point: Lf = f^(r)(t₀). We shall see that the variance ratio does not converge to 1, and so the Bernstein-von Mises theorem fails. Indeed, in the Fourier basis

$$\phi 0\left(t\right)\equiv 1,\{\begin{array}{cc}{\phi }_{2k-1}\left(t\right)=\sqrt{2}\sin 2\pi kt\hfill & \hfill \\ {\phi }_{2k}\left(t\right)=\sqrt{2}\sin 2\pi kt,\hfill & \hfill \end{array}\phantom{\}}k=1,2,\dots$$

we find that, a_k = Lφ_k = d^rφ_k(t₀)/dt^r and an easy calculation shows that $a_{2k-1}^2+a_{2k}^2=2{\left(2\pi k\right)}^{2r}$. We use a Gaussian prior (18) with ${\tau }_{2k-1}^2={\tau }_{2k}^2=k^{-2m}$ and 2m > 2r + 1. It follows from (19) that, after writing V_1n and V_2n for V_yn and V_Fn respectively, we have

$$V_{jn}=2{\left(2\pi \right)}^{2r}{\sigma }_n^2\sum _k{k}^{2r}{(1+{\sigma }_n^2{k}^{2m})}^{-j}.$$

As λ → 0, sums of the form

$$\sum _{k=0}^{\infty }{k}^p{(1+\lambda k^q)}^{-r}~k{\lambda }^{-(p+1)∕q},$$

with $\kappa =\kappa (p,r;q)={\int }_0^{\infty }{v}^p{(1+v^q)}^{-r}dv=\Gamma (r-\mu )\Gamma \left(\mu \right)∕\left(q\Gamma \left(r\right)\right)$ and μ (p+1)/q. In the present case, with p = 2r, q = 2m and r = j, we conclude that

$$\frac{V_{Fn}}{V_{yn}}\to 1-\frac{2r+1}{2m}<1.$$

Centering at the MLE. For a bounded linear functional, the MLE $L{\widehat{f}}_M=\Sigma a_k{y}_k$ is well defined and unbiased, with mean $E\left(L{\widehat{f}}_M\right)=Lf$ and frequentist variance $V_{Mn}=\mathrm{Var}\left(L{\widehat{f}}_M\right)={\sigma }_n^2{\Sigma }_k{a}_k^2$. A frequentist might prefer to use 100(1 − α)% intervals $L{\widehat{f}}_M\pm z_{\alpha ∕2}\sqrt{V_{Mn}}$ which will have the correct coverage property. However, extra conditions are required for the Bernstein-von Mises result to hold in this case.

Proposition 5

Assume that Lf = ∫ af is a bounded linear functional. Suppose also that the coefficients of θ_k = 〈f, φ_k〉 of the `true' f, and the variances ${\tau }_k^2$ of the Gaussian prior together satisfy Σ|a_kθ_k/τ_k|< ∞. Then the distance between Bayesian and frequentist distributions

$$\parallel N({\widehat{Lf}}_n,V_{yn})-N(L{\widehat{f}}_M,V_{Mn})\parallel \stackrel{{\mathbb{P}}_f^n}{\to }0.$$ (22)

Proof. The argument is a slight elaboration of that used in the previous proposition. We use (7) and $P=N({\widehat{Lf}}_n,V_{yn})$ as before, but now $Q=N(L{\widehat{f}}_M,V_{Mn})$ and (8) yields

$${\rho }^2(P,Q)=\frac{2\sqrt{V_{yn}{V}_{Mn}}}{V_{yn}+V_{Mn}}\exp \{-\frac{1}{2}\frac{{(L{\widehat{f}}_M-{\widehat{Lf}}_n)}^2}{V_{yn}+V_{Mn}}\}.$$

As before $V_{yn}∕V_{Mn}=\Sigma a_k^2{w}_{kn}^2∕\Sigma a_k^2\to 1$ by dominated convergence. Using this and the expression $V_{Mn}={\sigma }_n^2\Sigma a_k^2$ and in view of the bounds (7), the conclusion (22) is equivalent to ${\sigma }_n^{-1}\mid L{\widehat{f}}_M-{\widehat{Lf}}_n\mid \stackrel{{\mathbb{P}}_f^n}{\to }0$. We may write

$${\sigma }_n^{-1}(L{\widehat{f}}_M-{\widehat{Lf}}_n)\stackrel{\mathcal{D}}{=}\sum _k{a}_k{\sigma }_n^{-1}(1-w_{kn}){\theta }_k+\Sigma a_k(1-w_{kn}){∊}_k.$$

The stochastic term has mean 0 and variance $\Sigma a_k^2{(1-w_{kn})}^2\to 0$, again by dominated convergence. Thus we may focus on the deterministic term, and note that the merging in (22) occurs if and anly if

$${\sigma }_n\sum _k\frac{a_k{\theta }_k}{{\sigma }_n^2+{\tau }_k^2}\to 0.$$

The bound ${\sigma }_n{\tau }_n∕({\sigma }_n^2+{\tau }_k^2)\le \frac{1}{2}$ along with the dominated convergence theorem then shows that $\Sigma \mid a_k{\theta }_k{\tau }_k^{-1}\mid <\infty$ is a sufficient condition for (22) as claimed.

5. Related work

As remarked earlier, this paper avoids the important Gaussian approximation part of the Bernstein-von Mises phenomenon by focusing on examples with Gaussian likelihoods and priors. A growing literature addresses the approximation challenges; we give a brief listing here, and refer to the books (Ghosh and Ramamoorthi, 2003; Ghosal and van der Vaart, 2010) and the survey discussion in Ghosal (2010, §2.7) for more detailed discussion.

Ghosal (1997, 1999, 2000) developed posterior normality results for the full posterior in cases where the dimension of the parameter space increases sufficiently slowly. In each case, the emphasis is on conditions under which a non-Gaussian likelihood and appropriate prior sequence can yield approximate Guassian posteriors. However Ghosal (2000, Sec. 4) specializes his results to our setting (D) with ${\sigma }_n^2=1∕n$ and notes that one can choose priors-in general not Gaussian-so that the posterior distribution centered by the MLE is approximately Gaussian if p³(log p)/n → 0.

In survival analysis, Bernstein-von Mises theorems for the cumulative hazard function are established by Kim and Lee (2004) and for the cumulative hazard and fixed dimensional covariate regression parameter in a proportional hazards model in Kim (2006).

Boucheron and Gassiat (2009) develop a Bernstein-von Mises theorem for discrete probability distributions of growing dimension, and consider application to functionals such as Shannon and Renyi entropies.

In a semiparametric setting, where a finite dimensional parameter of interest can be separated from an infinite dimensional nuisance parameter, Castillo (2008) obtains conditions leading to a Bernstein-von Mises theorem on the parametric part, clarifying an earlier work of Shen (2002).

Rivoirard and Rousseau (2009) give conditions under which Bernstein-von Mises holds for linear functionals of a nonparametrically specified probability density function.