EXACT MINIMAX ESTIMATION OF THE PREDICTIVE DENSITY IN SPARSE GAUSSIAN MODELS

Abstract

We consider estimating the predictive density under Kullback–Leibler loss in an ℓ₀ sparse Gaussian sequence model. Explicit expressions of the first order minimax risk along with its exact constant, asymptotically least favorable priors and optimal predictive density estimates are derived. Compared to the sparse recovery results involving point estimation of the normal mean, new decision theoretic phenomena are seen. Suboptimal performance of the class of plug-in density estimates reflects the predictive nature of the problem and optimal strategies need diversification of the future risk. We find that minimax optimal strategies lie outside the Gaussian family but can be constructed with threshold predictive density estimates. Novel minimax techniques involving simultaneous calibration of the sparsity adjustment and the risk diversification mechanisms are used to design optimal predictive density estimates.

1. Introduction

Statistical prediction analysis aims to use past data to choose a probability distribution that will be good in predicting the behavior of future samples. This well-established subject [Aitchison and Dunsmore (1975), Geisser (1993)] finds application in game theory, econometrics, information theory, machine learning, mathematical finance, etc.

In this paper we study predictive density estimation in a high-dimensional setting and, in particular, explore the consequences of sparsity assumptions on the unknown parameters.

1.1. Main results

We begin by describing some of our main results: fuller references, background and interpretation follow in Section 1.2.

We work in the simplest Gaussian model for high-dimensional prediction:

$$X\sim N_n(\theta ,v_{x}I),Y\sim N_n(\theta ,v_{y}I),X\perp \perp Y|\theta .$$ (1)

On the basis of the “past” observation vector X, we seek to predict the distribution of a future observation Y. The past and future observations are independent, but are linked by the common mean parameter θ, assumed to be unknown. Note, however, that the variances, assumed here to be known, may differ. We write p(x|θ, v_x) and p(y|θ, v_y) for the probability densities of X and Y, respectively.

We seek estimators $\widehat{p}\left(y|x\right)$ of the future observation density p(y|θ, v_y), and to compare their performance under sparsity assumptions on θ. We recall two natural ways of generating large classes of estimators. Perhaps simplest are the “plug-in” or estimative densities: given a point estimate $\widehat{\theta }\left(X\right)$, simply set $\widehat{p}\left(y|x\right)=p\left(y|\widehat{\theta }\right)$. We often use the abbreviation $p[\widehat{\theta }]$. Second, given any prior measure π(dθ), proper or improper, such that the posterior π(dθ|x) is well defined, the Bayes predictive density is

$${\widehat{p}}_{\pi }\left(y|x\right)={\displaystyle \int p\left(y|\theta ,v_y\right)\pi \left(d\theta |x\right)}.$$ (2)

The important case of a uniform prior measure π(dθ) = dθ leads to predictive density ${\widehat{p}}_U\left(y|x\right)$, easily seen to correspond to N_n(x, (v_x + v_y)I).

We will examine similarities and differences between high-dimensional prediction and high-dimensional estimation. In particular, ${\widehat{p}}_U\left(y|x\right)$ plays in prediction the role of the maximum likelihood estimator ${\widehat{\theta }}_{\mathrm{MLE}}\left(x\right)=x$ in the multinormal mean estimation setting. In contrast to the corresponding plug-in estimate $p[{\widehat{\theta }}_{\mathrm{MLE}}]$, the density ${\widehat{p}}_U$ incorporates the variability of the location estimate which leads to a flattening of the estimator: v_x + v_y>v_y.

To evaluate the performance of a predictive density estimator $\widehat{p}\left(y|x\right)$, we use the familiar Kullback–Leibler “distance” as loss function:

$$L\left(\theta ,\widehat{p}\left(\cdot |x\right)\right)={\displaystyle \int p\left(y|\theta ,v_y\right)\log \frac{p\left(y|\theta ,v_y\right)}{\widehat{p}\left(y|x\right)}dy}.$$

The corresponding K–L risk function follows by averaging over the distribution of the past observation:

$$\rho \left(\theta ,\widehat{p}\right)={\displaystyle \int L\left(\theta ,\widehat{p}\left(\cdot |x\right)p\left(x|\theta ,v_x\right)\right)dx}.$$

Given a prior measure π(dθ), the average or integrated risk is

$$B(\pi ,\widehat{p})={\displaystyle \int \rho (\theta ,\widehat{p})\pi \left(d\theta \right)}.$$ (3)

The Bayes predictive density (2) can be shown to minimize both the posterior expected loss ${\displaystyle \int L\left(\theta ,\widehat{p}\left(\cdot |x\right)\right)\pi \left(d\theta |x\right)}$ and the integrated risk $B(\pi ,\widehat{p})$ in the class of all density estimates. This is a general fact in statistical decision theory [Brown (1974)], the resulting minimum the Bayes K–L risk:

$$B(\pi )=\underset{\widehat{p}}{\text{inf}}B(\pi ,\widehat{p}).$$ (4)

Our main focus is on how to optimize the predictive risk $\rho \left(\theta ,\widehat{p}\right)$ in a high-dimensional setting under an ℓ₀ sparsity condition on the parameter space. Thus, let ‖θ‖₀ = #{i:θ_i ≠0} and

$${\mathrm{\Theta }}_n\left[s\right]=\{\theta \in {\mathbb{ℝ}}^n:\Vert \theta {\Vert }_0\le s\}.$$ (5)

This “exact” sparsity condition has been widely used in estimation; in this paper we initiate study of its implications for predictive density estimation.

The minimax K–L risk for estimation over Θ is given by

$$R_N\left(\mathrm{\Theta }\right)=\underset{\widehat{p}}{\text{inf}}\underset{\theta \in \mathrm{\Theta }}{\sup }\rho \left(\theta ,\widehat{p}\right),$$ (6)

where the infimum is taken over all measurable predictive density estimators $\widehat{p}\left(y|x\right)$. For comparison, we write $R_{\epsilon }(\mathrm{\Theta })={\text{inf}}_{\widehat{\theta }}{\text{sup}}_{\mathrm{\Theta }}\rho \left(\theta ,p\left[\widehat{\theta }\right]\right)$ for the minimax risk restricted to the sub-class ℰ of plug-in or “estimative” densities.

To state our main results, henceforth we will assume v_x = 1 and introduce the key parameters

$$r=v_y/v_x=v_y,v_w={(1+r^{-1})}^{-1}.$$ (7)

Here v_w is the “oracle variance” which would be the variance of the UMVUE for θ, were both X and Y observed.

In our asymptotic model, the dimensionality n → ∞ and the sparsity s = s_n may depend on n, but the variance ratio r remains fixed. The notation a_n ~ b_n denotes a_n/b_n → 1 as n → ∞.

Theorem 1A

Fix r ∈ (0, ∞). If η_n = s_n/n → 0, then

$$R_N\left({\mathrm{\Theta }}_n\left[s_n\right]\right)~\frac{1}{1+r}{s}_n\log \left(n/s_n\right)=\frac{1}{1+r}n{\eta }_n\log {\eta }_n^{-1}.$$ (8)

The minimax risk is proportional to the sparsity s_n, with a logarithmic penalty factor. The case where s_n ≡ s remains constant is included. The expression is quite analogous to that obtained for point estimation with quadratic loss, namely, 2s_nlog(n/s_n) [Donoho and Johnstone (1994), Donoho et al. (1992) and Johnstone (2013), Chapter 8.8, hereafter cited as Johnstone (2013)]. However, we shall see that quite different phenomena emerge in the predictive density setting.

Indeed, the future-to-past variance ratio r is an important parameter of the predictive estimation problem. The minimax risk increases as r decreases: we need to estimate the future observation density based on increasingly noisy past observations (in relative terms, r = v_y/v_x), and so the difficulty of the density estimation problem increases. However, the rate of convergence with n in (8) does not depend on r, and so exact determination of the constants is needed to show the role of r in this prediction problem.

The inefficiency of plug-in estimators is an immediate consequence of Theorem 1A. Let $q(\theta ,\widehat{\theta })=E\Vert \widehat{\theta }\left(X\right)-\theta {\Vert }^2$ denote the risk of point estimator $\widehat{\theta }$ under squared-error loss. It is straightforward to show for a plug-in density estimate $p\left[\widehat{\theta }\right]$ that $\rho \left(\theta ,p\left[\widehat{\theta }\right]\right)=q\left(\theta ,\widehat{\theta }\right)/\left(2r\right)$. Hence, from the point estimation minimax risk just cited,

$$R_{\epsilon }\left({\mathrm{\Theta }}_n\left[s_n\right]\right)\sim \frac{1}{r}{s}_n\log \left(n/s_n\right)\sim \left(1+\frac{1}{r}\right)R_N\left({\mathrm{\Theta }}_N\left[s_n\right]\right).$$

The inefficiency of plug-in estimators thus equals the oracle precision,

$$1/v_w=1+1/r,$$

and becomes arbitrarily large as the variance ratio r → 0.

We turn now to the asymptotically least favorable priors and optimal estimators in Theorem 1A. Let δ_λ denote unit point mass at λ and

$$\pi \left[\eta ,\mathrm{\lambda }\right]=\left(1-\eta \right){\delta }_0+\eta {\delta }_{\mathrm{\lambda }}$$ (9)

be a univariate two-point prior: this is a sparse prior when η is small and λ large. Let

$${\mathrm{\lambda }}_e=\sqrt{2\log {\eta }_n^{-1}\left(1-{\eta }_n\right)},{\mathrm{\lambda }}_f=\sqrt{v_w}{\mathrm{\lambda }}_e.$$ (10)

In point estimation based on X, we recall that λ_e is essentially the threshold of detectability corresponding to sparsity η_n = s_n/n. Although Y is not yet observed, we will see that in the prediction setting the UMVUE scaled threshold λ_f < λ_e plays a partly analogous role.

Build a sparse high-dimensional prior from i.i.d. draws:

$${\pi }_n^{\mathrm{IID}}\left(d\theta \right)={\displaystyle \prod _{i=1}^n\pi \left[{\eta }_n,{\mathrm{\lambda }}_f\right]\left(d{\theta }_i\right)}.$$ (11)

If the sparsity s_n increases without bound with n, then this i.i.d. prior with scale λ_f is asymptotically least favorable:

Theorem 1B

If s_n → ∞ and s_n/n → 0, then

$$B\left({\pi }_n^{\mathrm{IID}}\right)=R_N\left({\mathrm{\Theta }}_n\left[s_n\right]\right)\cdot \left(1+o\left(1\right)\right).$$

The assumption that s_n → ∞ ensures that ${\pi }_n^{\mathrm{IID}}$ concentrates on Θ[s_n], namely, that ${\pi }_n^{\mathrm{IID}}\left(\mathrm{\Theta }[s_n]\right)\to 1$ as n → ∞. This hypothesis is not needed for Theorem 1A; indeed, a sparse prior built from “independent blocks” is asymptotically least favorable assuming only s_n/n → 0. This more elaborate prior is described in Section 5.

Some of the novel aspects of the predictive density estimation problem appear in the description of optimal estimators, that is, ones that asymptotically attain the minimax bound in Theorem 1A. In point estimation, the simplest asymptotically minimax rule for sparsity s_n is given by co-ordinatewise hard thresholding ${\widehat{\theta }}_i\left(x\right)=x_{i}I\left\{|x_i|\ge {\mathrm{\lambda }}_e\right\}$. For prediction, we consider the following class of univariate density estimators as analogs of hard thresholding:

$${\widehat{p}}_T\left(y_1|x_1\right)=\{\begin{array}{cc}{\widehat{p}}_{\pi }\left(y_1|x_1\right),& \text{if}|x_1|\le {\mathrm{\lambda }}_e,\\ {\widehat{p}}_U\left(y_1|x_1\right),& \text{if}|x_1|<{\mathrm{\lambda }}_e.\end{array}$$ (12)

The univariate density estimates are combined to form a multivariate predictive density estimate via a product rule

$${\widehat{p}}_T\left(y|x\right)={\displaystyle \prod _{i=1}^n{\widehat{p}}_T\left(y_i|x_i\right)}.$$ (13)

The threshold λ_e in (12) is that corresponding to estimation based on X at sparsity η_n = s_n/n. Above the threshold, the uniform prior predictive density ${\widehat{p}}_U$ corresponds to the (unbiased) MLE. Below threshold, we shall need the flexibility of the Bayes predictive density (2). Indeed, as explained in Section 4, it does not suffice to use π = δ₀, point mass at 0, which would be the predictive analog of thresholding to zero in point estimation.

Instead, we use a sparse univariate cluster prior π = π_CL[η, r] given by

$$\pi =\left(1-\eta \right){\delta }_0+\frac{\eta }{2K}{\displaystyle \sum _{k=1}^K\left({\delta }_{{\mu }_k}+{\delta }_{-{\mu }_k}\right)}.$$ (14)

The points μ_k = μ_k(r) for k = 1,…, K are geometrically spaced to cover an interval [ν_η, λ_e + a] containing [λ_f, λ_e], as described in more detail below. The key point is that it is necessary to “diversify” the predictive risk by introducing prior support points to cover [–λ_e, –λ_f] ∪ [λ_f, λ_e].

More specifically, for a parameter a = a_η given below, let μ_η be the positive root of the overshoot equation

$${\mu }^2+2a\mu ={\mathrm{\lambda }}_e^2,$$ (15)

that occurs in sparse minimax point estimation [e.g., Johnstone (2013), equation (8.48)], and then set ${\nu }_{\eta }=\sqrt{v_w}{\mu }_{\eta }$: since μ_η < λ_e, we have ν_η <λ_f. The support points

$${\mu }_1={\nu }_{\eta },{\mu }_{k+1}={\left(1+2r\right)}^k{\nu }_{\eta },k\ge 1,$$ (16)

with K = max{k : μ_k ≤ λ_e + a}. We choose $a_{\eta }=\sqrt{{2\log \mathrm{\lambda }}_f}$.

Theorem 1C

Assume η_n = s_n/n → 0. Let ${\widehat{p}}_{T,\mathrm{CL}}\left(y|x\right)$ be the product predictive threshold estimator defined by (12) and (13) using the cluster prior π_CL [η_n, r ]. Then ${\widehat{p}}_{T,\mathrm{CL}}$ is asymptotically minimax:

$$\underset{{\mathrm{\Theta }}_n\left[s_n\right]}{max}\rho (\theta ,{\widehat{p}}_{T,\mathrm{CL}})=R_N\left({\mathrm{\Theta }}_n\left[s_n\right]\right)\left(1+o\left(1\right)\right).$$

Note that the number of positive support points in the cluster prior K = K_η increases as r decreases. For any fixed η, the cluster prior contains in total (2K_η + 1) support points. Also, for any fixed r ∈ (0, ∞) as η → 0, we have

$$K\left(r\right)=\underset{\eta \to 0}{\lim }{K}_n=\left[\frac{\log \left(1+r^{-1}\right)}{2\log \left(1+2r\right)}\right].$$

Thus, K(r) is apiecewise constant, right continuous function with jumps as shown in Table 1.

Table 1.

Number K(r) of positive support points in the cluster prior π_CL[η, r] as r varies

r	0.1073	0.1235	0.1465	0.1826	0.2485	0.4196	>0.4196
K(r)	7	6	5	4	3	2	1

The results presented above assume v_x = 1. These results can be easily extended to the general case by noting that the minimax risk remains invariant and the scale of past observations and parameter is divided by $\sqrt{v_x}$.

1.2. Background and previous work

The relative entropy predictive risk $\rho \left(\theta ,\widehat{p}\right)$ measures the exponential rate of divergence of the joint likelihood ratio over a large number of independent trials [Larimore (1983)]. The minimal predictive risk estimate maximizes the expected growth rate in repeated investment scenarios [Cover and Thomas (1991), Chapters 6, 15]. In data compression, $L\left(\theta ,\widehat{p}\left(\cdot |x\right)\right)$ reflects the excess average code length that we need if we use the conditional density estimate $\widehat{p}$ instead of the true density to construct a uniquely decodable code for the data Y given the past x [McMillan (1956)]. Following Bell and Cover (1980), ℓ₀-constrained minimax optimal predictive density estimates in on our model can be used for construction of optimal predictive schemes for gambling, sports betting, portfolio selection and sparse coding [Mukherjee (2013), Chapter 1.3].

Aitchison (1975), Murray (1977) and Ng (1980) showed that in most parametric models there exist Bayes predictive density estimates which are decision theoretically better than the maximum likelihood plug-in estimate. An important issue in predictive inference has always been to compare the performance of the class ℰ of point estimation (PE) based plug-in density estimates [Barndorff-Nielsen and Cox (1996)] with that of the optimal predictive density estimate. In parameter spaces of fixed dimension, large sample attributes of the predictive risk of efficient plug-in and Bayes density estimates have been studied by Komaki (1996), Hartigan (1998) and Aslan (2006).

The high-dimensional predictive density estimation problem studied in this paper is relevant to a number of contemporary applications, including data compression, sequential investment with side information and sports betting (SM).

Analogy with point estimation

Decision theoretic parallels between predictive density estimation under Kullback–Leibler loss and point estimation under quadratic loss have been explored in our Gaussian model by George, Liang and Xu (2006), Ghosh, Mergel and Datta (2008), Komaki (2004), Xu and Zhou (2011) and George, Liang and Xu (2012). For unconstrained parameter spaces $\mathrm{\Theta }={\mathbb{ℝ}}^n$, fundamental ideas in Gaussian point estimation theory can be extended to yield optimal predictive density estimates [Brown, George and Xu (2008), Fourdrinier et al. (2011), Komaki (2001)]. For ellipsoids, Xu and Liang (2010) established an analog of the theorem of Pinsker (1980) by proving that the class of all linear predictive density estimates [see (17)] is minimax optimal.

For sparse estimation, instead of parallels, we found contrasts. Minimax risks in the predictive density problem depend on r, but this dependence is not emphasized in the admissibility results in unrestricted spaces. As we have seen, under sparsity construction of optimal minimax estimators requires the notion of diversification of the future risk over the interval [λ_f, λ_e] in a way strongly dependent on r. Thus, efficiency of the prediction schemes depend on careful calibration of the sparsity adjustment and the risk diversification mechanisms.

1.3. Further results

Other classes of estimators

The class of linear estimates ℒ are Bayes rules based on conjugate product normal priors. The resulting estimators

$${\widehat{p}}_{L,\alpha }={\displaystyle \prod _{i=1}^{n}N\left({\alpha }_i{X}_i,{\alpha }_i+r\right)},{\alpha }_i\in \left[0,1\right],$$ (17)

are still Gaussian but have larger variance than the future density $p\left(y|\theta ,r\right)=\varphi \left(y|\theta ,r\right)$. We choose the name “linear” because the conjugate prior implies linearity of the posterior mean in X.

The class $\mathcal{G}$ contains all product Gaussian density estimates $p[\widehat{\theta },\widehat{d}]={\displaystyle {\prod }_{i=1}^{n}N\left({\widehat{\theta }}_i,{\widehat{d}}_i\right)}$. Clearly, $\mathcal{G}$ contains both ℒ and ℰ, the latter introduced after (6). The minimax risks R_ℒ(Θ) and $R_{\mathcal{G}}(\mathrm{\Theta })$ are defined by restricting the infimum in (6) to ℒ and $\mathcal{G}$, respectively.

We have seen after Theorem 1a that $R_{\epsilon }\left({\mathrm{\Theta }}_n\left[s_n\right]\right)~\left(1+r^{-1}\right)R_N\left({\mathrm{\Theta }}_n[s_n]\right)$. It turns out that extending ℰ to $\mathcal{G}$ does not help, while, as is typical for sparse estimation, the class of linear estimators ℒ performs very poorly.

Proposition 1

Fix r ∈ (0, ∞). If s_n/n → 0, then

$$\begin{array}{l}(R_{ℒ}({\mathrm{\Theta }}_n\left[s_n\right])=\left(n/2\right)\log (1+r^{-1}),\\ R_{ℒ}({\mathrm{\Theta }}_n\left[s_n\right])/R_N({\mathrm{\Theta }}_n\left[s_n\right])\to \infty \mathit{and}\\ R_{\mathcal{G}}\{{\mathrm{\Theta }}_n\left[s_n\right])~R_{ℰ}({\mathrm{\Theta }}_n\left[s_n\right]).\end{array}$$

Univariate prediction problem

The product structure of our high-dimensional model (1), estimators (13) and priors (11), along with concentration of measure, implies that many aspects of our multivariate results can be understood and proved through an associated univariate prediction problem.

In the univariate setting, assume that the past observation X|θ ~ N(θ, 1) and the future observation Y|θ ~ N(θ, r). Assume that X and Y are independent given θ. In addition, suppose that θ is random with distribution π(dθ), assumed to belong to

$$\mathfrak{m}(\eta )=\left\{\pi \in \mathcal{P}\left(\mathbb{ℝ}\right):\pi \left(\theta \ne 0\right)\le \eta \right\},$$ (18)

where $\mathcal{P}\left(\mathbb{ℝ}\right)$ is the collection of all probability measures in ℝ.

A predictive density estimator $\widehat{p}\left(y|x\right)$ is evaluated through its integrated risk $B\left(\pi ,\widehat{p}\right)$ defined at (3). The minimax risk for this univariate prediction problem is given by

$$\beta \left(\eta ,r\right):=\underset{\widehat{p}}{\text{inf}}\underset{\pi \in \mathfrak{m}\left(\eta \right)}{\sup }B\left(\pi ,\widehat{p}\right),$$ (19)

and we study sparsity through the asymptotic regime η → 0. Recall definition (10) of the scaled threshold λ_f = λ_f,_η.

Theorem 2

1. Fix r ∈ (0, ∞). As η → 0,

   $$\beta \left(\eta ,r\right)=\frac{1}{2r}\eta {\mathrm{\lambda }}_f^2\left(1+o\left(1\right)\right).$$ (20)

2. An asymptotically least favorable prior is given by the two-point distribution π[η, λ_f(η)] of (9).

3. An asymptotically minimax estimator is given by the thresholding construction (12) combined with sparse univariate cluster prior π = π_CL [η, r] defined at (14).

1.4. Organization of the paper

The main results of the paper are multivariate, Theorems 1A, 1B and 1C. However, the main technical issues in the proofs are best handled in the univariate setting of Theorem 2, whose parts a, b and c correspond to Theorems 1a, 1b and 1c, respectively. Section 2 has an overview: it first reviews some connections between the multivariate and univariate settings, then gives heuristic derivations for the lower and upper bounds of univariate Theorem 2. Section 3 and Section 4, respectively, contain the technical proofs for the lower and upper bound on the univariate minimax risk, Theorem 2b and 2c, respectively. Together, they complete the proof of Theorem 2. Proofs of the multivariate results in Theorems 1a, 1b and 1c are completed in Section 5. This section also contains a heuristic proof of Proposition 1 whose rigorous proof is presented in the supplementary material [Mukherjee and Johnstone (2015)].

Glossary

[The notation (6)+2 refers to text 2 lines after equation (6)].

Estimators: Bayes ${\widehat{p}}_{\pi }$ (2), Uniform prior ${\widehat{p}}_U$ (2)+2, Threshold ${\widehat{p}}_T$ (12), Multivariate product $\widehat{p}\left(y|x\right)$ (13); Univariate $\widehat{p}\left(y_1|x_1\right)$.

Classes of estimators and multivariate minimax risks: all nonlinear N, R_N (6), estimative ℰ, R_ℰ (6)+2, “linear” ℒ, R_ℒ (17), Gaussian $\mathcal{G}$, $R_{\mathcal{G}}$ (17)+4.

Univariate minimax risk: β (19).

Parameter spaces: multivariate Θ_n[s] (5); univariate m(η) (18).

Priors: Univariate: two point π[η,λ] (9), cluster π_CL[η,r] (14), Multivariate: ${\pi }_n^{\mathrm{IID}}$ (11).

Parameters: variance ratio r = v_y/v_x, oracle variance v_w (7), sparsity η (9), thresholds λ_e, λ_f (10), cluster prior: overshoot a (15), ν_η (15)+2.

2. Proof overview and interpretation

2.1. Connections between multivariate and univariate settings

Many aspects of the multivariate theorem may be understood, and in part proved, through a discussion of the univariate prediction problem of Theorem 2. An obvious connection between the univariate and multivariate approaches runs as follows: suppose that a multivariate predictive estimator is built as a product of univariate components

$$\widehat{p}\left(y|x\right)={\displaystyle \prod _{i=1}^n{\widehat{p}}_1\left(y_i|x_i\right)}.$$ (21)

Suppose also that to a vector θ = (θ_i) we associate a univariate (discrete) distribution ${\pi }_n^e=n^{-1}{\displaystyle {\sum }_{i=1}^n{\delta }_{{\theta }_i}}$. Since the true multivariate future density p(Y|θ, r) is also a product of univariate components, it is then readily seen that the multivariate and univariate Bayes K–L risks are related by

$$\rho \left(\theta ,\widehat{p}\right)={\displaystyle \sum _{i=1}^n\rho \left({\theta }_i,{\widehat{p}}_1\right)=nB\left({\pi }_n^e,{\widehat{p}}_1\right)}.$$ (22)

The sparsity condition Θ_n[s_n] in the multivariate problem corresponds to requiring that the prior $\pi ={\pi }_n^e$ in the univariate problem satisfies

$$\pi \{{\theta }_1\ne 0\}\le s_n/n={\eta }_n,$$

and thus belongs to the class m(η) defined in (18). Next, we outline the minimax risk calculations for the sparse predictive density estimation problem.

As a first illustration, to which we return later, consider the maximum risk of a product rule over Θ_n[s_n]: using (22) and (3), we have

$$\underset{{\mathrm{\Theta }}_n\left[s_n\right]}{\text{sup}}\rho \left(\theta ,\widehat{p}\right)=n\left[\left(1-{\eta }_n\right)\rho \left(0,{\widehat{p}}_1\right)+{\eta }_n\underset{\theta \in \mathbb{ℝ}}{\text{sup}}\rho \left(\theta ,{\widehat{p}}_1\right)\right].$$ (23)

In the univariate problem, using ${\widehat{p}}_1$, we have the somewhat parallel bound

$$\underset{\mathfrak{m}\left(\eta \right)}{\text{sup}}B\left(\pi ,\widehat{p}\right)=\left(1-\eta \right)\rho \left(0,{\widehat{p}}_1\right)+\eta \underset{\theta \in \mathbb{ℝ}}{\text{sup}}\rho (\theta ,{\widehat{p}}_1).$$ (24)

Consequently, a careful study of the two univariate quantities

$\begin{array}{ll}\text{risk at zero}:& \rho (0,{\widehat{p}}_1),\\ \text{maximum risk}:& \underset{\theta }{\sup }\rho (\theta ,{\widehat{p}}_1)\end{array}$ (25)

is basic for upper bounds for both univariate and multivariate cases.

2.2. Theorem 2B: Univariate lower bound heuristics

To understand the apperance of ${\mathrm{\lambda }}_f^2$ in the minimax risks, we turn to a heuristic discussion of the lower bound, first in the univariate case.

We use the two point priors (9) and the definition (19):

$$\beta \left(\eta ,r\right)\ge B\left(\pi \left[\eta ,\mathrm{\lambda }\right]\right)=\left(1-\eta \right)\rho \left(0,{\widehat{p}}_{\pi }\right)+\eta \rho \left(\mathrm{\lambda },{\widehat{p}}_{\pi }\right)\ge \eta \rho \left(\mathrm{\lambda },{\widehat{p}}_{\pi }\right),$$ (26)

and look for a good bound for $\rho \left(\mathrm{\lambda },{\widehat{p}}_{\pi }\right)$ for a suitable choice of λ.

The key is a mixture representation for predictive risk of a Bayes estimator in terms of quadratic risk, where the weighted mixture is over noise levels v ∈ [v_w,1], with v_w being the oracle variance, (7). Brown, George and Xu (2008), Theorem 1, show that the predictive risk of the Bayes predictive density estimate ${\widehat{p}}_{\pi }$ is

$$\rho \left(\theta ,{\widehat{p}}_{\pi }\right)=\frac{1}{2}-{\displaystyle {\int }_{v_w}^{1}q\left(\theta ,{\widehat{\theta }}_{\pi ,v};v\right)\frac{dv}{v^2},}$$ (27)

where $q\left(\theta ,{\widehat{\theta }}_{\pi ,v};v\right)=E_{\theta }{\left[{\widehat{\theta }}_{\pi ,v}\left(W\right)-\theta \right]}^2$ is the quadratic risk of the Bayes location estimate ${\widehat{\theta }}_{\pi ,v}$ for prior π when W ~ N(θ, v). In point estimation with quadratic loss, it is known [Johnstone (2013), Chapter 8], that as η → 0 an approximately least favorable prior in the class m(η) is given, for noise level v = 1, by the sparse two-point prior π[η, λ_e(η)] defined in (9) and ${\mathrm{\lambda }}_e\left(\eta \right)=\sqrt{2\log {\eta }^{-1}\left(1-\eta \right)}$. This prior has the remarkable property that points θ ≤ λ_e are “invisible” in the sense that even when θ is true, the Bayes estimator ${\widehat{\theta }}_{\pi }={\widehat{\theta }}_{\pi ,1}$ effectively estimates 0 rather than θ and so makes a mean squared error

$$q(\theta ,{\widehat{\theta }}_{\pi };1)~{\theta }^2\text{for}0\le \theta \le {\mathrm{\lambda }}_e.$$ (28)

Two issues arise as the noise level v varies. First, the region of invisibility will scale, becoming $0\le \theta \le \sqrt{v}{\mathrm{\lambda }}_e$at scale v. As v varies in [v_w, 1], the intersection of all regions of invisibility will be $0\le \theta \le \sqrt{v_w}{\mathrm{\lambda }}_e={\mathrm{\lambda }}_f$ as defined at (10). The second issue is that for a given prior π and predictive Bayes rule ${\widehat{p}}_{\pi }$in (27), the Bayes rules ${\widehat{\theta }}_{\pi ,v}$ vary with v. We return to this second point in the next section; for now we can hope that for all v∈[v_w,1],

$$q\left({\mathrm{\lambda }}_f,{\widehat{\theta }}_{\pi ,\mathrm{v}};v\right)\underset{\sim }{>}{\mathrm{\lambda }}_f^2,$$ (29)

and so, from mixture representation (27),

$$\rho \left({\mathrm{\lambda }}_f,{\widehat{p}}_{\pi }\right)\underset{\sim }{>}\frac{{\mathrm{\lambda }}_f^2}{2}{\displaystyle {\int }_{v_w}^1\frac{dv}{v^2}=\frac{{\mathrm{\lambda }}_f^2}{2r}},$$

since the integral evaluates to $v_w^{-1}-1=r^{-1}$. From this we can conjecture that for π = π[η, λ_f] ∈ m(η),

$$B\left(\pi \right)>\eta \rho \left({\mathrm{\lambda }}_f,{\widehat{p}}_{\pi }\right)\underset{\sim }{>}\eta \frac{{\mathrm{\lambda }}_f^2}{2r}.$$ (30)

A full proof, with slightly modified definitions, is given in Section 3.

2.3. Theorem 2C: Univariate upper bound heuristics

We now turn to a heuristic discussion of constructing a density estimate to show that the lower bound (30) is asymptotically correct. Pursuing the analogy with point estimation, we know that in that setting optimal estimators can be found within the family of hard thresholding rules $\widehat{\theta }\left(x\right)=xI\left\{|x|>\mathrm{\lambda }\right\}$. The natural analog for predictive density estimation would have the form

$${\widehat{p}}_{T,{\pi }_0}\left[\mathrm{\lambda }\right]\left(y|x\right)=\{\begin{array}{c}{\widehat{p}}_U\left(y|x\right),|x|>\mathrm{\lambda },\\ {\widehat{p}}_{{\pi }_0}\left(y|x\right),|x|\le \mathrm{\lambda }.\end{array}$$ (31)

To see this, note that ${\widehat{p}}_U$ is the predictive Bayes rule corresponding to the uniform prior π(dθ) = dθ, which leads to the MLE $\widehat{\theta }\left(x\right)=x$ in point estimation, while ${\widehat{p}}_{{\pi }_0}\left(y|x\right)$ denotes the predictive Bayes rule corresponding to a prior concentrated entirely at 0, so that

$${\widehat{p}}_{{\pi }_0}\left(y|x\right)=\varphi \left(y|0,r\right)$$ (32)

is a normal density with mean zero and variance r.

For the upper bound, according to definition (19), we seek an estimator ${\widehat{p}}_1$ for which ${\text{sup}}_{\mathfrak{m}\left(\eta \right)}B\left(\pi ,{\widehat{p}}_1\right)\sim \eta {\mathrm{\lambda }}_f^2/\left(2r\right)$ as η → 0. In bound (24), the first component is the risk at zero, $\rho \left(0,{\widehat{p}}_1\right)$, and it turns out that this determines the possible values of the threshold λ in (31). Thus, in order that

$\rho \left(0,{\widehat{p}}_{T,{\pi }_0}\left[\mathrm{\lambda }\right]\right)=o\left(\eta {\mathrm{\lambda }}_f^2\right),$

it follows [see (51)] that the threshold λ should be chosen as λ = λ_e ~ (2log η⁻¹)^1/2 and not smaller.

Turning to the second part of (25), we seek an estimator ${\widehat{p}}_1$ with

$\underset{\theta }{\sup }\rho \left(\theta ,{\widehat{p}}_1\right)=\frac{{\mathrm{\lambda }}_f^2}{2r}\cdot \left(1+o\left(1\right)\right).$ (33)

We first argue that the hard thresholding analog (31) cannot work. Decompose the predictive risk of a univariate threshold estimator ${\widehat{p}}_{\mathrm{T}}$ with threshold λ_e into contributions due to X above and below the threshold

$$\begin{array}{l}\rho \left(\theta ,{\widehat{p}}_T\right)=E_{\theta }L\left(\theta ,\widehat{p}\left(\cdot |X\right)\right)\\ =E_{\theta }\left[L\left(\theta ,{\widehat{p}}_U\left(\cdot |X\right)\right),|X|>{\mathrm{\lambda }}_e\right]+E_{\theta }\left[L\left(\theta ,{\widehat{p}}_{\pi }\left(\cdot |X\right)\right),|X|\le {\mathrm{\lambda }}_e\right]\\ ={\rho }_A\left(\theta \right)+{\rho }_B\left(\theta \right),\end{array}$$ (34)

say. With the “zero prior,” the K–L loss is just quadratic in θ,

$$L\left(\theta ,{\widehat{p}}_{{\pi }_0}\left(Y|X\right)\right)=E_{\theta }\log \frac{\varphi \left(Y|\theta ,r\right)}{\varphi \left(Y|0,r\right)}=\frac{{\theta }^2}{2r},$$

and so, in particular, for θ ≤ λ_e we see that

$$\rho \left(\theta ,{\widehat{p}}_{T,{\pi }_0}\right)\ge {\rho }_B\left(\theta \right)\underset{\sim }{>}\frac{{\theta }^2}{2r}{P}_{\theta }\left[|X|\le {\mathrm{\lambda }}_e\right]$$ (35)

could be as large as ${\mathrm{\lambda }}_e^2/\left(2r\right)$, and hence larger than our target risk ${\mathrm{\lambda }}_f^2/\left(2r\right)$.

Bearing in mind the role that two-point priors play in the lower bound, it is perhaps natural to ask next if the threshold rule ${\widehat{p}}_{T,\mathrm{LF}}$ with π₀ in (31) replaced by the (symmetrized) two-point prior π[η, λ_f] could cut off the growth of the quadratic θ²/(2r) for |θ| ≥ λ_f. The 3-point prior π₃[η, λ_f] ∈ m(η) places probability η/2 at the two nonzero atoms at ±λ_f. Remarks in Section 3 show that π₃[η, λ_f] is also asymptotically least favorable for the univariate prediction problem as η → 0. Indeed, it can be shown (see Section 4) that for this prior and for λ_f ≤|θ|≤ λ_e,

$$\begin{array}{l}\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{LF}}\right)~\rho B\left(\theta \right)\\ \le \frac{1}{2r}\left\{{\mathrm{\lambda }}_f^2-\left(|\theta |-{\mathrm{\lambda }}_f\right)\left[\left(1+2r\right){\mathrm{\lambda }}_f-|\theta |\right]\right\}+o\left({\mathrm{\lambda }}_f^2\right).\end{array}$$ (36)

Consequently, the risk bound dips below ${\mathrm{\lambda }}_f^2/\left(2r\right)$ for λ_f ≤ |θ| ≤ (1 + 2r)λ_f but increases thereafter. So, ${\widehat{p}}_{T,\mathrm{LF}}$ is minimax optimal if λ_e < (1 + 2r)λ_f, which occurs if r is sufficiently large, r > 0.4196 in Table 1. However, the upper bound exceeds our target risk ${\mathrm{\lambda }}_f^2/\left(2r\right)$ if r ≤ 0.4196. Section S.2 of the supplementary material [Mukherjee and Johnstone (2015)] shows rigorously that ${\widehat{p}}_{T,\mathrm{LF}}$ is indeed minimax suboptimal for low values of r.

As π₃[η,λ_f] fails to produce minimax optimal density estimates, the strategy then is to introduce extra support points |μ_k| ≤ λ_e into the prior chosen to “pull down” the risk ${\rho }_B\left(\theta \right)=E_{\theta }\left[L\left(\theta ,{\widehat{p}}_{\pi }\left(\cdot |X\right)\right),|X|\le {\mathrm{\lambda }}_e\right]$ below ${\mathrm{\lambda }}_f^2/\left(2r\right)$ whenever it would otherwise exceed this level. The schematic diagram in Figure 1 illustrates this bounding of the maximum risk. The extra support points added in [λ_f, λ_e] and [−λ_e, −λ_f] distribute the predictive risk across that range—“risk diversification”—and keep the maximum risk below ${\mathrm{\lambda }}_f^2/\left(2r\right)(1+o(1\left)\right)$.

Fig. 1.

Schematic diagram of the risk of univariate threshold density estimates for θ ≥ 0. The dotted line is the risk of density estimator ${\widehat{p}}_{T,\mathrm{LF}}$ based on the 3-point prior π₃[η, λ_f]. The addition of appropriately spaced prior mass points (shown in red) up to λ_e pulls down the risk function of the cluster prior-based density estimate ${\widehat{p}}_{T,\mathrm{CL}}$below ${\mathrm{\lambda }}_f^2/\left(2r\right)$until the effect of thresholding at λ_e takes over.

To prove that this works, we obtain upper bounds on ρ_B (θ) for ${\widehat{p}}_{T,\mathrm{CL}}$ by focusing, when $\theta \in [{\mu }_k,{\mu }_{k+1}]$, only on the prior support point μ_k. The main inequality is obtained in (50), namely,

${\rho }_B\left(\theta \right)\le \frac{1}{2r}\left[{\mathrm{\lambda }}_f^2+\underset{k}{min}{q}_k\left(\theta \right)\right]+o\left({\mathrm{\lambda }}_f^2\right).$

where q_k(θ) is a quadratic polynomial that is O(λ_f) on [μ_k, μ_k+1]. Putting together this and other bounds, we can then finally establish the uniform bound (33). The details are in Section 4.

3. Theorem 2B: Univariate lower bound proof

This section is devoted to a proof of the lower bound part of Theorem 2. The heuristic discussion of the last section indicated the importance of two-point sparse priors and the invisibility property (28). To formulate a precise statement about the upper limit of invisibility, we start with noise level 1 and bring in the positive solution μ_η of the overshoot equation (15), namely, ${\mu }^2+2a\mu ={\mathrm{\lambda }}_e^2$. Here the “overshoot” parameter a = a_η should satisfy both $a_{\eta }\to \infty$ and a_η = o(μ_η); we make the specific choice $a_{\eta }=\sqrt{2\log {\mathrm{\lambda }}_{f,\eta }}$.

In preparation for the range of variance scales in mixture representation (27), we consider the collection of two-point priors π[η, μ] for 0≤μ≤μ_η. Using a temporary notation for this section, let ${\widehat{\theta }}_{\mu }\left(x\right)=E\left[\theta |x\right]$ be the Bayes rule for squared error loss for the prior π[η, μ]. The next result shows that when the true parameter is actually μ, and this nonzero support point $\mu \le {\mu }_{\eta }$, then the Bayes rule for π[η,μ] “gets it wrong” by effectively estimating 0 and making an error of size μ², uniformly in $\mu \le {\mu }_{\eta }$.

Lemma 3

There exists ε_η ↘ 0 as η → 0 such that for all μ in [0,μ_η],

$$q\left(\mu ,{\widehat{\theta }}_{\mu };1\right)\ge {\mu }^2\left[1-{\epsilon }_{\eta }\right].$$

Proof

Using standard calculations for the two-point prior, the Bayes rule ${\widehat{\theta }}_{\mu }=\mu p(\mu |x)=\mu /\left[1+m\left(x\right)\right]$, with

$$m\left(x\right)=\frac{p\left(0|x\right)}{p\left(\mu |x\right)}=\frac{1-\eta }{\eta }\frac{\varphi \left(x\right)}{\varphi \left(x-\mu \right)}=\exp \left\{\frac{1}{2}{\mathrm{\lambda }}_e^2-x\mu +\frac{1}{2}{\mu }^2\right\}.$$ (37)

Consequently,

$$\begin{array}{l}q\left(\mu ,{\widehat{\theta }}_{\mu };1\right)=E_{\mu }{\left[{\widehat{\theta }}_{\mu }-\mu \right]}^2={\mu }^2{E}_{\mu }{\left[{\left(1+m\left(X\right)\right)}^{-1}-1\right]}^2\\ ={\mu }^2{E}_0{\left[1+m^{-1}\left(\mu +Z\right)\right]}^{-2},\end{array}$$

where Z ~ N(0, 1), and from (37), $m^{-}{}^1(\mu +z)=\text{exp}\left\{\frac{1}{2}\left({\mu }^2+2\mu \mathrm{z}-{\mathrm{\lambda }}_e^2\right)\right\}$.

Now, using definition (15) of μ_η, for 0 ≤ μ ≤ μ_η, we have

$${\mu }^2+2\mu z-{\mathrm{\lambda }}_e^2\le {\mu }_{\eta }^2+2{\mu }_{\eta }{z}_{+}-{\mathrm{\lambda }}_e^2=-2{\mu }_{\eta }\left(a-z_{+}\right),$$

so that for 0 ≤ μ ≤μ_η,

$${\mu }^{-2}q\left(\mu ,{\widehat{\theta }}_{\mu };1\right)\ge E_0\left\{{\left[1+\exp \left(-{\mu }_{\eta }\left(a-Z_{+}\right)\right)\right]}^{-2},Z<a\right\}=1-{\epsilon }_{\eta },$$

say. For each fixed z, we have ${\mu }_{\eta }\left(a-z_{+}\right)\to \infty$ since $a\to \infty$, and so from the dominated convergence theorem we conclude that $\epsilon \left(\eta \right)\to 0$. □

With these preparations, we return to the lower bound in the prediction problem. As η → 0, an asymptotically least favorable distribution is given by a sparse two-point prior with the nonzero support point scaled using the oracle standard deviation ${\nu }_w^{1/2}$. We shall prove the following:

Lemma 4

Let μ_η be the positive solution to overshoot equation (15) with $a_{\eta }=\sqrt{2\log {\mathrm{\lambda }}_{f,\eta }}.\mathit{Set}{\nu }_{\eta }={\nu }_w^{1/2}{\mu }_{\eta }$ and consider the two-point prior π[η, ν_η]. Then as η → 0,

$$\beta \left(\eta ,r\right)\ge B\left(\pi \left[\eta ,{\nu }_{\eta }\right]\right)\ge \frac{\eta {\mathrm{\lambda }}_f^2}{2r}\left(1+o\left(1\right)\right).$$

We note here that since a_η = o(μ_η), the overshoot equation implies that

$${\mu }_{\eta }~{\mathrm{\lambda }}_{e,\eta }\text{and}{\nu }_{\eta }~{\mathrm{\lambda }}_{f,\eta }.$$ (38)

A stronger conclusion, used in the next section, also follows from the overshoot equation, namely,

$${\mathrm{\lambda }}_{f,\eta }^2-{\nu }_{\eta }^2={\nu }_w\left({\mathrm{\lambda }}_{e,\eta }^2-{\mu }_{\eta }^2\right)={\nu }_w\cdot 2a{\mu }_{\eta }\le 2a{\nu }_w{\mathrm{\lambda }}_{e,\eta }=2a\sqrt{{\nu }_w}{\mathrm{\lambda }}_{f,\eta }.$$ (39)

Proof of Lemma 4

Recall (26) and (27) in the heuristic discussion. We now clarify the dependence on scale v of the Bayes rule ${\widehat{\theta }}_{\pi ,\nu }$ in the mixture representation (27). Passing from noise level v to noise level 1 by dividing parameters and estimates by v^1/2, we obtain the invariance relation

$$q\left(\theta ,{\widehat{\theta }}_{\pi \left[\eta ,\mathrm{\lambda }\right],\nu };\nu \right)=\nu q\left({\nu }^{-1/2}\theta ,{\widehat{\theta }}_{\pi [\eta ,{\nu }^{-1/2}\mathrm{\lambda }]};1\right).$$

Now set θ = ν_η and substitute into (27) to obtain, for π = π[η, ν_η],

$$\rho \left({\nu }_{\eta },{\widehat{p}}_{\pi }\right)=\frac{1}{2}{\displaystyle {\int }_{{\nu }_w}^{1}q\left(v^{-1/2}{v}_{\eta },{\widehat{\theta }}_{\pi \left[\eta ,v^{-1/2}{v}_{\eta }\right]};1\right)}\frac{dv}{v}.$$ (40)

Now apply Lemma 3 with μ = v^−1/2ν_η being bounded above by $v_w^{-1/2}{\nu }_{\eta }={\mu }_{\eta }$. For all $v\in \left[v_w,1\right]$ we obtain

$$q\left(v^{-1/2},{\widehat{\theta }}_{v^{-1/2}{v}_{\eta }};1\right)\ge v^{-1}{v}_{\eta }^2\left[1-{\epsilon }_{\eta }\right].$$

Putting this into the mixture representation, we get

$$\rho \left(v_{\eta },{\widehat{p}}_{\pi }\right)\ge \frac{1}{2}{v}_{\eta }^2\left[1-{\epsilon }_{\eta }\right]{\displaystyle {\int }_{v_w}^1\frac{dv}{v^2}=\frac{{\nu }_{\eta }^2}{2r}}\left[1-{\epsilon }_{\eta }\right].$$

Taking into account both (26) and (38), we have established the lemma.

Based on the discussion in Section 2, the above lemma establishes a lower bound on the asymptotic minimax risk β (η, r) in Theorem 2. Similarly, the symmetric 3-point prior

$${\pi }_3[\eta ,{\nu }_{\eta }]=(1-\eta ){\delta }_0+\left(\eta /2\right)\{{\delta }_{{\nu }_{\eta }}+\delta {-}_{{\nu }_{\eta }}\}$$

will also be asymptotically least favorable over m (η) as η → 0.

4. Theorem 2C: Univariate upper bound proof

The upper bound on the predictive minimax risk β (η, r) is derived from the upper bound on the maximum Bayes risk of ${\widehat{p}}_{T,\mathrm{CL}}$ over m (η). In this section we will prove the following lemma which along with Lemma 4 completes the proof of Theorem 2.

Lemma 5

For any r ∈ (0, ∞) we have, as η → 0,

$$\underset{\pi \in \mathfrak{m}\left(\eta \right)}{\sup }B\left(\pi ,{\widehat{p}}_{T,\mathrm{CL}}\right)\le \frac{\eta {\mathrm{\lambda }}_f^2}{2r}\left(1+o\left(1\right)\right).$$

We consider a threshold predictive density estimate ${\widehat{p}}_T$ which uses the Bayes predictive density estimate from prior π below the threshold λ_e and ${\widehat{p}}_U$ above the threshold λ_e. We bound the maximum predictive risk over m(η):

$$\underset{\pi \in \mathfrak{m}\left(\eta \right)}{\sup }B\left(\pi ,{\widehat{p}}_T\right)\le \left(1-\eta \right)\rho \left(0,{\widehat{p}}_T\right)+\eta \underset{\theta }{\sup }\rho \left(\theta ,{\widehat{p}}_T\right).$$ (41)

Next, as in (34), we decompose the predictive risk of ${\widehat{p}}_T$ into contributions due to X above and below the threshold. We calculate explicit expressions for ρ_A and ρ_B. The predictive loss of ${\widehat{p}}_U$ (see Appendix A.2) is given by

$$L\left(\theta ,{\widehat{p}}_U\left(·|x\right)\right)=a_{1r}+a_{2r}{\left(\theta -x\right)}^2$$ (42)

with $a_{1r}=\frac{1}{2}\left[\log \left(1+r^{-1}\right)-{\left(1+r\right)}^{-1}\right]$ and $a_{2r}=\frac{1}{2}{\left(1+r\right)}^{-1}$. Hence, the above threshold term

$${\rho }_A(\theta )=a_{1r}{P}_{\theta }\left(|X|>{\mathrm{\lambda }}_e\right)+a_{2r}{E}_{\theta }\left[{\left(X-\theta \right)}^2,|X|>{\mathrm{\lambda }}_e\right].$$ (43)

As ρ_B (θ) depends on the prior π used below the threshold, we restrict our attention to the specific choice of the cluster prior. The risk functions of the hard threshold density estimate ${\widehat{p}}_T{,}_{{\pi }_0}$ and that of ${\widehat{p}}_{T,\mathrm{LF}}$ can be easily derived from the calculations with the cluster prior.

According to (58) in the Appendix, the Bayes predictive density for a discrete prior $\pi ={\displaystyle {\sum }_{k=-K}^K{\pi }_k{\delta }_{{\mu }_k}}$ is given by

$${\widehat{p}}_{\pi }\left(y|x\right)={\displaystyle \sum _{-K}^K\varphi \left(y|{\mu }_k,r\right)}{\pi }_k\varphi \left(x-{\mu }_k\right)/m\left(x\right),$$ (44)

where $m\left(x\right)={\displaystyle {\sum }_k{\pi }_k}\varphi \left(x-{\mu }_k\right)$ denotes the marginal density of π. The K–L loss of ${\widehat{p}}_{\pi }\left(·|x\right)$is given by

$$L\left(\theta ,{\widehat{p}}_{\pi }\left(\cdot |x\right)\right)={\mathrm{E}}_{\theta }\log \frac{\varphi \left(Y|\theta ,r\right)}{{\widehat{p}}_{\pi }\left(Y|x\right)}.$$

A simple but informative upper bound for the K–L loss is obtained by retaining only the kth term in (44):

$$\begin{array}{l}L(\theta ,{\widehat{p}}_{\pi }(\cdot |x))\le {\mathrm{E}}_{\theta }\log \frac{\varphi (Y|\theta ,r)}{\varphi (Y|{\mu }_k,r)}-\log \frac{{\pi }_k\varphi (x-{\mu }_k)}{{\pi }_0\varphi \left(x\right)}+\log \frac{m\left(x\right)}{{\pi }_0\varphi \left(x\right)}\\ =\frac{1}{2r}{(\theta -{\mu }_k)}^2+\frac{1}{2}\left({\mu }_k^2-2x{\mu }_k\right)-\log \frac{{\pi }_k}{{\pi }_0}+d\left(x\right)\end{array}$$ (45)

where we have set d(x) = log[m(x)/(π₀ϕ(x))].

We are now ready to analyze the bound (41). We follow the steps recalled in the quadratic loss case [see Section S.4 of Mukherjee and Johnstone (2015)] and evaluate the predictive risk at the origin and the maximum risk of the threshold density estimate ${\widehat{p}}_T$. This organization helps to make clear the new features of the predictive loss setting.

Risk at zero

It is easy to show that $\rho \left(0,{\widehat{p}}_T\right)=O\left(\eta {\mathrm{\lambda }}_f\right)$. First, from (43), we have

$${\rho }_A\left(0\right)=2a_{1r}\stackrel{\sim }{\mathrm{\Phi }}\left({\mathrm{\lambda }}_e\right)+{\mathrm{a}}_{2r}{q}_A\left(0\right)=O\left(\eta {\mathrm{\lambda }}_f\right),$$

where q_A(0) is defined in (S.4.2) and the above calculation follows by using $\stackrel{\sim }{\mathrm{\Phi }}\left({\mathrm{\lambda }}_e\right)\le {\mathrm{\lambda }}^{-1}\varphi \left({\mathrm{\lambda }}_e\right)=O\left({\mathrm{\lambda }}_e^{-1}\eta \right)$ and the quadratic risk-at-zero bound (S.4.4).

For the below-threshold term, we set k = 0 in (45), note that μ₀ = 0 and apply Jensen’s inequality to obtain

$${\rho }_B\left(0\right)=E_0\left[L\left(0,{\widehat{p}}_{\pi }\left(·|x\right)\right),|x|\le \mathrm{\lambda }\right]\le E_0\left[d\left(X\right)\right]\le \log E_0\left[m\left(X\right)/\left({\pi }_0\varphi \left(X\right)\right)\right].$$

Since $E_0\left[m\left(X\right)/\varphi \left(X\right)\right]={\displaystyle \int m}\left(x\right)dx=1$ and π₀ = −η, we obtain that

$${\rho }_B\left(0\right)\le -\log \left(1-\eta \right)\le \eta .$$

Consequently, ρ_B(0) = O(η) and so ρ(0, ${\widehat{p}}_T$,_CL) = O(ηλ_f). Note that the above calculations hold for any ${\widehat{p}}_{T,\pi }$ with π being a discrete prior in m(η).

Maximum risk

From decomposition (41), our goal is to show that

$$\underset{\theta }{\sup }\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{CL}}\right)={\left(2r\right)}^{-}{}^1{\mathrm{\lambda }}_f^2\left(1+o\left(1\right)\right).$$ (46)

We first isolate the main term in the contributions from ρ_A(θ) and ρ_B(θ). From (43), clearly ρ_A(θ) ≤ a_1r + a_2r = O(1), which does not contribute. We turn to

$${\rho }_B(\theta )=E_{\theta }\left[L\left(\theta ,{\widehat{p}}_{\pi }\left(·|X\right)\right),|X|\le {\mathrm{\lambda }}_e\right]$$

and returning to (45), we begin by claiming that for $|x|\le {\mathrm{\lambda }}_e$ the final term $d\left(x\right)\le \log 2$. Indeed

$$\frac{m\left(x\right)}{{\pi }_0\varphi \left(x\right)}=1+{\displaystyle \sum _{|k|=1}^K\frac{{\pi }_k}{{\pi }_0}\frac{\varphi \left(x-{\mu }_k\right)}{\varphi \left(x\right)}}=1+{\displaystyle \sum _{|k|=1}^K\frac{{\pi }_k}{{\pi }_0}\exp \left\{x{\mu }_k-\frac{{\mu }_k^2}{2}\right\}}.$$ (47)

For $|x|\le {\mathrm{\lambda }}_e$, we have

$$x{\mu }_k-{\mu }_k^2/2\le {\mathrm{\lambda }}_e|{\mu }_k|-{\mu }_k^2/2\le {\mathrm{\lambda }}_e^2/2=\log {\eta }^{-1}\left(1-\eta \right).$$

Since ${\pi }_0=1-\eta$, we arrive at

$$E_{\theta }\left[d\left(X\right),|X|\le {\mathrm{\lambda }}_e\right]\le \log 2.$$ (48)

The dependence of (45) on θ may then be seen by writing x = θ + z. The first two terms in (45) then take the form

$$\frac{1}{2r}\{{\left[\theta -\left(1+r\right){\mu }_k\right]}^2-\left(r^2+r\right){\mu }_k^2\}-{\mu }_{k}z,$$

while, after recalling that ${\pi }_k=\eta /\left(2K\right)$ and that ${\mathrm{\lambda }}_e^2=2\log \left(1-\eta \right){\eta }^{-1}$, the third term becomes

$$\frac{1}{2}{\mathrm{\lambda }}_e^2+\log \left(2K\right)=\frac{1}{2r}\left(1+r\right){\mathrm{\lambda }}_f^2+\log \left(2K\right).$$

We may therefore rewrite (45) as

$$L\left(\theta ,{\widehat{p}}_{\pi }\left(\cdot |x\right)\right)\le \frac{1}{2r}\left[{\mathrm{\lambda }}_f^2+q_k\left(\theta \right)\right]-{\mu }_k\left(x-\theta \right)+\log \left(2K\right)+d\left(x\right),$$ (49)

where the kth quadratic polynomial

$$q_k\left(\theta \right)={\left[\theta -\left(1+r\right){\mu }_k\right]}^2-r^2{\mu }_k^2+r\left({\mathrm{\lambda }}_f^2-{\mu }_k^2\right).$$

Denote the last three terms of (49) by J_k(x, θ). From (16) and (48) we see that

$$E_{\theta }\left[J_k,|X|\le {\mathrm{\lambda }}_e\right]\le {\mu }_k+\log \left(2K\right)+\log 2\le {\mathrm{\lambda }}_e+a+\log \left(4K\right)=o\left({\mathrm{\lambda }}_f^2\right).$$

Consequently, we obtain the key bound

$${\rho }_B\left(\theta \right)\le \frac{1}{2r}\left[{\mathrm{\lambda }}_f^2+\underset{k}{min}{q}_k\left(\theta \right)\right]+o\left({\mathrm{\lambda }}_f^2\right).$$ (50)

Now we use the geometric structure of the support points μ_k, defined at (16). We bound min_k q_k(θ) above by considering the quadratic polynomial q_k(θ) on I_k = [μ_k,μ_k+1] and observe that these 2K intervals cover the range (–λ_e –a,–λ_f) ∪ (λ_f, λ_e +a) of interest. See Figure 2. Note that q_k(θ) achieves its maximum on I_k at both endpoints and that

Fig. 2.

Schematic diagram demonstrating the behavior of the quadratic polynomials q_k(θ) in the interval [μ₁, μ_K+1]. Here K = 4. The maximum of min_kq_k(θ) for θ ∈ [μ₁, μ_K+1] is bounded by q₁(μ₁).

$$q_k\left({\mu }_{k+1}\right)=q_k\left(\left(1+2r\right){\mu }_k\right)=q_k\left({\mu }_k\right)=r\left({\mathrm{\lambda }}_f^2-{\mu }_k^2\right).$$

These maxima decrease with k and so are bounded by $q_1\left({\nu }_{\eta }\right)=r\left({\mathrm{\lambda }}_f^2-{\nu }_{\eta }^2\right)$. Appealing now to bound (39), we have for λ_f < |θ| < λ_e + a,

$$\underset{k}{min}{q}_k\left(\theta \right)\le r\left({\mathrm{\lambda }}_f^2-v_{\eta }^2\right)\le 2r\sqrt{v_w}a{\mathrm{\lambda }}_f.$$

Returning to (50), we now see that the last two terms are each $o\left({\mathrm{\lambda }}_f^2\right)$ and so the final bound (46) is proven. This completes the proof of Lemma 5.

These calculations apply to threshold density estimates based on Bayes estimates of discrete priors. In particular, for ${\widehat{p}}_{T,\mathrm{LF}}$ which is based on the 3-point prior π₃[η, ν_η], we have K = 1 and the bound (36). Thus, the difference ${\rho }_B\left(\theta \right)-{\mathrm{\lambda }}_f^2/2r$ in this case is negligible when |θ| ≤ μ₂.

Similarly, the asymptotic risk function of the hard threshold plug-in density estimate ${\widehat{p}}_{T,{\pi }_0}$ (for which K = 0 in our calculations above) exceeds the minimax risk β(η, r) for |θ| ∈ [λ_f, λ_e] and so is minimax suboptimal for any fixed r. Figure 3 shows the numerical evaluation of the risk functions for the different univariate threshold density estimates.

Fig. 3.

Numerical evaluation of the asymptotic risk ρ_B(θ) for r = 0.25 of univariate threshold density estimates: hard threshold plug-in estimate ${\widehat{p}}_{T,{\pi }_0}$ (red), ${\widehat{p}}_{T,\mathrm{LF}}$ (green) and the cluster prior-based minimax optimal estimate ${\widehat{p}}_{T,\mathrm{CL}}$ (blue). The brown boxes show the nonzero support point of the cluster prior and the univarate asymptotic minimax risk $\beta \left(\eta ,r\right)={\left(2r\right)}^{-1}{\mathrm{\lambda }}_f^2$ and the threshold λ_e are respectively denoted by dotted horizontal and vertical lines. The plot on left has η = e⁻²⁰ (very high sparsity), λ_f = 2.83, λ_e = 6.32 and the right one has η = 0.05 (moderate sparsity), λ_f = 1.09, λ_e = 2.45.

Also, note that any threshold estimate ${\widehat{p}}_T\left[\mathrm{\lambda }\right]$ with threshold size λ less than λ_e will be minimax suboptimal, as its risk at the origin will not be negligible as compared to β(η, r). By (34) and (43) we have

$\begin{array}{l}\rho \left(0,{\widehat{p}}_T\left[\mathrm{\lambda }\right]\right)\ge 2a_{2r}E\left[Z^{2}I\left\{Z>\mathrm{\lambda }\right\}\right]=2a_{2r}\left\{\mathrm{\lambda }\varphi \left(\mathrm{\lambda }\right)+2\stackrel{\sim }{\mathrm{\Phi }}\left(\mathrm{\lambda }\right)\right\}\\ \ge \mathrm{\lambda }\varphi \left(\mathrm{\lambda }\right)/\left(1+r\right),\end{array}$ (51)

and so for any fixed ε > 0,

$$\underset{1\le {\mathrm{\lambda }<\mathrm{\lambda }}_e\left(\eta \right)-\epsilon }{\lim \inf }\frac{\rho \left(0,{\widehat{p}}_T\left[\mathrm{\lambda }\right]\right)}{\beta \left(\eta ,r\right)}\to \infty .$$

Thus, ${\widehat{p}}_T\left[\mathrm{\lambda }\right]$ is suboptimal unless λ ≥ λ_e.

5. Theorem 1: Multivariate minimax risk

Here we will use the univariate minimax results developed in the previous sections to evaluate the asymptotic multivariate minimax risk R_n = R_N(Θ_n[s_n]) over the sparse parameter space Θ_n[s_n].

5.1. Lower bound proof: Theorem 1B and an extension

We first prove a lower bound for the multivariate minimax risk under only the assumption that s_n/n → 0—without requiring, as in Theorem 1B, that also s_n → ∞. This is done using an “independent blocks” sparse prior, along the lines of Johnstone (2013), Chapter 8.6, that we will show to be asymptotically least favorable. This result establishes the lower bound half of Theorem 1a. At the end of the subsection, we prove Theorem 1B using the simpler i.i.d. prior.

Let π_S(τ; m) denote a single spike prior of scale τ on ℝ^m: choose an index I ∈ {1, …, m} at random and set θ = τe_I, where e_I is a unit length vector in the ith coordinate direction. We will use a scale τ_m = λ_m − logλ_m which is somewhat smaller than ${\mathrm{\lambda }}_m=\sqrt{2\log m}$.

The independent blocks prior π^IB on Θ[s_n] is built by dividing {1, …, n} into s_n contiguous blocks B_j,j = 1, …, m each of length m = m_n = [n/s_n]. Draw components θ_i in each block B_j according to an independent copy of π_S(ν_m; m) where the scale $v_m=\sqrt{v_w}{\tau }_m$ is matched to the prediction setting. Finally, set θ_i = 0 for the remaining n − m_ns_n components. Thus, π^IB is supported on Θ[s_n] since any draw θ from π^IB has exactly s_n nonzero components.

The lower bound half of Theorem 1A follows from the following result, the analog of Theorem 1B for the independent blocks prior.

Theorem 6

Fix r ∈ (0, ∝). If s_n/n → 0, then

$$R_N\left({\mathrm{\Theta }}_n\left[s_n\right]\right)\ge B\left({\pi }_n^{\mathrm{IB}}\right)\ge {\left(1+r\right)}^{-1}{s}_n\log \left(n/s_n\right).$$

Proof

Bounding maximum risk by Bayes risk and using the product structure shows that

$R_n=R_N\left({\mathrm{\Theta }}_n\left[s_n\right]\right)\ge B\left({\pi }_n^{\mathrm{IB}}\right)=s_{n}B\left({\pi }_S\left(v_m;m\right)\right).$ (52)

Next, using $B_Q^v$ to denote the Bayes risk for noise level v, the multivariate form of the connecting equation and scale invariance enable us to write

$$B\left({\pi }_S\left(v_m;m\right)\right)=\frac{1}{2}{\displaystyle {\int }_{v_w}^1{B}_Q^v\left({\pi }_S\left(v_m;m\right)\right)\frac{dv}{v^2}}=\frac{1}{2}{\displaystyle {\int }_{v_w}^1{B}_Q\left({\pi }_S\left(\frac{v_m}{\sqrt{v}};m\right)\right)\frac{dv}{v}}.$$

The next lemma, proved in Section S.5 of Mukherjee and Johnstone (2015), provides a uniform lower bound for the quadratic loss Bayes risk of a single spike prior. It is a multivariate analog of Lemma 3.

Proposition 7

Suppose that y ~ N_n(0, I). Set ${\mathrm{\lambda }}_n=\sqrt{2\log n}$ and τ_n = λ_n − log λ_n. Then there exists ε_n → 0 such that uniformly in τ ∈ [0, τ_n],

$B_q\left({\pi }_S\left(\tau ;n\right)\right)\ge {\tau }^2\left(1-{\epsilon }_n\right).$

Noting that v ∈ [v_w, 1] implies that $v_m/\sqrt{v}\le v_m/\sqrt{v_w}={\tau }_m$, and then applying the proposition,

$B\left({\pi }_S\left(v_m;m\right)\right)\ge \frac{\left(1-{\epsilon }_m\right)}{2}{\displaystyle {\int }_{v_m}^1\frac{v_m^2}{v^2}dv=\left(1-{\epsilon }_m\right)\frac{v_m^2}{2r}}.$

Combining this with (52) and the definition of ν_m, we obtain

$R_n\ge \left(1-{\epsilon }_m\right)s_n{v}_w{\tau }_m^2/\left(2r\right)\sim {\left(1+r\right)}^{-1}{s}_n\log \left(n/s_n\right).$ (53)

Proof of Theorem 1B

Note that because of the product structure of the problem and the prior ${\pi }_n^{\mathrm{IID}}$ we have

$B\left({\pi }_n^{\mathrm{IID}}\right)={\displaystyle \sum _{i=1}^n\beta \left({\eta }_n,r\right)}=n\beta \left({\eta }_n,r\right),$

which is asymptotically equal to R_N(Θ[s_n]), using the univariate Theorem 2 [cf. (20)] and

$${\left(2r\right)}^{-1}{\mathrm{\lambda }}_f^2={\left(2r\right)}^{-1}{v}_w{\mathrm{\lambda }}_f^2\sim {\left(1+r\right)}^{-1}\log {\eta }_n^{-1}\text{as}n\to \infty .$$ (54)

Also, as s_n → ∞, ${\pi }_n^{\mathrm{IID}}\left(\mathrm{\Theta }\left[s_n\right]\right)\to 1$ by application of Chebyshev’s inequality and, hence, ${\pi }_n^{\mathrm{IID}}$ is an asymptotically least favorable prior under the conditions of Theorem 1B. □

5.2. Upper bound proof: Theorem 1C

First, an upper bound on R_N(Θ_n[s_n]) is derived based on the maximum risk of the multivariate product threshold density estimate ${\widehat{p}}_{T,\mathrm{CL}}$ defined in Theorem 1C. Using the product structure of the threshold estimate as well as that of the unknown future density

$${\widehat{p}}_{T,\mathrm{CL}}\left(y|x\right)={\displaystyle \prod _{i=1}^n{\widehat{p}}_{T,\mathrm{CL}}}\left(y_i|x_i\right)\text{and}p\left(y|\theta ,r\right)={\displaystyle \prod _{i=1}^{n}p}\left(y_i|{\theta }_i,r\right),$$

the risk of our multivariate threshold estimate simplifies as an agglomerative coordinate wise risk of the respective univariate density estimates

$$\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{CL}}\right)=E_{\theta }\log \frac{p\left(y|\theta ,r\right)}{{\widehat{p}}_{T,\mathrm{CL}}\left(y|x\right)}={\displaystyle \sum _{i=1}^n\rho \left({\theta }_i,{\widehat{p}}_T\right)}.$$

Now, maximizing over θ ∈ Θ_n[s_n], we have

$$R_n\le \underset{{\mathrm{\Theta }}_n\left[s_n\right]}{\sup }\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{CL}}\right)\le \left(n-s_n\right)\rho \left(0,{\widehat{p}}_{T,\mathrm{CL}}\right)+s_n\underset{\theta }{\sup }\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{CL}}\right).$$

From the univariate study, we know that $\rho \left(0,{\widehat{p}}_{T,\mathrm{CL}}\right)=O\left({\eta }_n{\mathrm{\lambda }}_f\right)$, which makes $\left(n-s_n\right)\rho \left(0,{\widehat{p}}_{T,\mathrm{CL}}\right)=O\left(s_n{\mathrm{\lambda }}_f\right)$ negligible relative to

$$s_n\underset{\theta }{\sup }\rho \left(\theta ,{\widehat{p}}_{T,\mathrm{CL}}\right)={\left(2r\right)}^{-1}{s}_n{\mathrm{\lambda }}_f^2\left(1+o\left(1\right)\right),$$

where we used (46). Thus, taking account also of (54), we have the desired upper bound on the minimax risk

$$R_n\le {\left(2r\right)}^{-1}{s}_n{\mathrm{\lambda }}_f^2\left(1+o\left(1\right)\right)\sim {\left(1+r\right)}^{-1}{s}_n\log \left(n/s_n\right).$$ (55)

Completion of Proof of Theorems 1A, 1B and 1C

As the lower bound (53) and upper bound (55) on R_n match asymptotically, the first order asymptotic minimax risk of Theorem 1a is achieved, and the proof of all parts is done.

5.3. Proof of Proposition 1

Estimates in ℒ and $\mathcal{G}$ are products of the form (21) and so $R_{ℒ,n}=R_{ℒ}\left({\mathrm{\Theta }}_n\left[s_n\right]\right)$ can be studied using the associated univariate problem and decomposition (23). It is shown in Appendix A.2 that

$$\rho \left(\theta ,{\widehat{p}}_{L,\alpha }\right)=\frac{1}{2}\log \left(1+\frac{\alpha }{r}\right)+\frac{{\left(1-\alpha \right)}^2}{2\left(r+\alpha \right)}\left[{\theta }^2-\frac{\alpha }{1-\alpha }\right].$$ (56)

Thus, ${\sup }_{\theta }\rho \left(\theta ,{\widehat{\rho }}_{L,\alpha }\right)$ is infinite unless α = 1, that is, the uniform prior estimate ${\widehat{p}}_U$, in which case $\rho \left(\theta ,{\widehat{p}}_U\right)\equiv \frac{1}{2}\log \left(1+r^{-1}\right)$. Thus,

$R_{ℒ,n}=\frac{n}{2}\log \left(1+r^{-1}\right)\gg \frac{s_n}{1+r}\log \left(\frac{n}{s_n}\right)\sim R_n.$

In particular, R_ℒ,_n/R_n → ∝ when s_n/n → 0.

We turn to the Gaussian class $\mathcal{G}$. Since $\mathcal{E}\subset \mathcal{G}$, clearly $R_{\mathcal{G},n}<R_{\mathcal{E},n}={\left(2r\right)}^{-1}n{\eta }_n{\mathrm{\lambda }}_e^2$. We give here a heuristic argument for the reverse inequality, which gives the idea for the rigorous proof given in Section S.3 of the supplementary material [Mukherjee and Johnstone (2015)]. From the decomposition (23), any near-optimal estimator in $\mathcal{G}$ must have univariate risk at 0 bounded as follows:

$$\rho \left(0,{\widehat{p}}_1\right)\le r^{-1}{\eta }_n{\mathrm{\lambda }}_e^2.$$ (57)

Now from (60) we know that the risk at the origin for the univariate Gaussian density estimate $p\left[\widehat{\theta },\widehat{d}\right]$ is

$$\rho \left(0,p\left[\widehat{\theta },\widehat{d}\right]\right)=2^{-1}{E}_0\left\{\log \left(r^{-1}\widehat{d}\right)+{\widehat{d}}^{-1}\left(r+{\widehat{\theta }}^2\right)-1\right\},$$

which for any fixed choice of $\widehat{\theta }$ achieves its minimum at $d_{\text{opt}}\left[\widehat{\theta }\right]=r+{\widehat{\theta }}^2$. Thus, for such an optimal choice of $\widehat{d}$,

$$\rho \left(0,p\left[\widehat{\theta },d_{\text{opt}}\left(\widehat{\theta }\right)\right]\right)=E_0\log \left(1+r^{-1}{\widehat{\theta }}^2\right),$$

and for this to satisfy (57), we must have $\widehat{\theta }\left(x\right)\approx 0$ for |x| ≤ λ_e(1 + o(1)). Thus, $\widehat{p}$ would approximately need to have the threshold structure (31), (32) for |x| ≤ λ_e and so the bound (35) shows that

$$\rho \left(\theta ,{\widehat{p}}_1\right)\ge \frac{{\theta }^2}{2r}{P}_{\theta }\left(|X|\le {\mathrm{\lambda }}_e\right)\sim \frac{{\mathrm{\lambda }}_e^2}{2r}.$$

Returning to decomposition (23), we can now see that $R_{\mathcal{G},n}\underset{\sim }{>}{\left(2r\right)}^{-1}{s}_n{\mathrm{\lambda }}^2\sim R_{\mathcal{E},n}$, which completes the heuristic argument.

6. Discussion

Avoiding thresholding

The asymptotic minimax rules ${\widehat{p}}_T$ described in Theorems 1c and 2c are based on thresholding. It would be desirable to construct a prior π for which the Bayes predictive density ${\widehat{p}}_{\pi }$ in (2) is itself asymptotically minimax, without any use of the discontinuous thresholding operation.

Consider, then, a symmetric univariate prior π_∞[η, r] whose support consists of the origin and infinite number of equidistant clusters each containing 2K points in the same spatial alignment as for π_CL[η, r]:

$${\pi }_{\infty }\left[\eta ,r\right]=\left(1-\eta \right){\delta }_0+\frac{1-\eta }{2}{\displaystyle \sum _{j=0}^{\infty }{\eta }^{j+1}}{\displaystyle \sum _{k=1}^K{q}_k({\delta }_{{\mu }_{\mathit{jk}}}+\delta -{\mu }_{\mathit{jk}})},$$

where μ_jk = jλ_e + μ_k and for k = 2, …, K and γ = log η⁻¹, we have q_k = γ^−k and $q_1=1-{\displaystyle {\sum }_2^K{q}_k}$.

Based on π_∞[η_n, r], one can construct a multivariate prior ${\pi }_{n,\infty }^{\mathrm{IID}}$ using (11), which heuristic arguments indicate will not only be least favorable but also yield a minimax optimal density estimate. A detailed proof is forthcoming.

Approximate sparsity and other extensions

Starting from Johnstone (2013), Chapters 8 and 13, the ℓ₀ sparsity results presented here can be extended to obtain minimax optimal predictive density estimates over weak and strong ℓ_p sparse parameter spaces. An interesting topic for future work will be whether, as in point estimation [Donoho and Johnstone (1994)], the phenomena seen here can be generalized to a family of loss functions. Simple analogues of the connecting equations [Brown, George and Xu (2008), Theorem 1] between the predictive and quadratic PE regimes do not exist in those cases, though some of the decision theoretic parallels can still be proved particularly for the ℓ₂ loss [Gatsonis (1984)].

APPENDIX

A.1. Bayes density estimate for discrete priors

The posterior distribution for the discrete prior $\pi ={\displaystyle {\sum }_{k=-K}^K{\pi }_k{\delta }_{{\mu }_k}}$ is given by

$\pi \left({\mu }_k|x\right)={\left\{m\left(x\right)\right\}}^{-1}\varphi \left(x|{\mu }_k,1\right){\pi }_k\text{where}m\left(x\right)={\displaystyle \sum _k{\pi }_k\varphi \left(x|{\mu }_k,1\right)}.$

So, for the Bayes predictive density based on the prior π,

$${\widehat{p}}_{\pi }\left(y|x\right)={\displaystyle \sum _{k=-K}^K\varphi \left(y|{\mu }_k,r\right)\pi \left({\mu }_k|x\right)=}{\displaystyle \sum _{k=-K}^K\varphi \left(y|{\mu }_k,r\right)\frac{\varphi \left(x|{\mu }_k,1\right){\pi }_k}{m\left(x\right)}}.$$ (58)

A.2. K–L risk for gaussian and linear density estimates

The predictive risk of the univariate Gaussian density estimate $p\left[\widehat{\theta },\widehat{d}\right]=N\left(\widehat{\theta },\widehat{d}\right)$ is given by

$$\rho \left(\theta ,p\left[\widehat{\theta },\widehat{d}\right]\right)=E_{\theta }\left\{\log \varphi \left(Y|\theta ,r\right)\right\}-E_{\theta }\left\{\log \varphi \left(Y|\widehat{\theta }\left(X\right),\widehat{d}\left(X\right)\right)\right\},$$

where the expectation is over X ~ N(θ, 1) and Y ~ N(θ, r). Noting that $E_{\theta }\left\{\log \varphi \left(Y|\widehat{\theta },\widehat{d}\right)|X=x\right\}=-\frac{1}{2}\log \left(2\pi \widehat{d}\left(x\right)\right)-\left(2\widehat{d}\left(x\right)\right)-1\left\{r+{\left(\widehat{\theta }\left(x\right)-\theta \right)}^2\right\}$ and $E_{\theta }\log \varphi \left(Y|\theta ,r\right)=-\frac{1}{2}\log \left(2\pi r\right)-\frac{1}{2}$, we obtain

$$L\left(\theta ,\widehat{p}\left(\cdot |x\right)\right)=\frac{1}{2}\log \left(r^{-1}\widehat{d}\right)+\frac{r+{\left(\widehat{\theta }\left(x\right)-\theta \right)}^2}{2\widehat{d}}-\frac{1}{2},$$ (59)

and the following expression for the K–L risk of members in $\mathcal{G}$:

$$\rho \left(\theta ,p\left[\widehat{\theta },\widehat{d}\right]\right)=\frac{1}{2}\left[E_{\theta }\log \left(r^{-1}\widehat{d}\right)+E_{\theta }\left\{\frac{r+{\left(\widehat{\theta }-\theta \right)}^2}{\widehat{d}}\right\}-1\right].$$ (60)

Consider now “linear” estimators. Starting with the conjugate prior θ ~ N(0, α/(1 − α)) for 0 ≤ α ≤ 1, standard calculations show that the posterior density π(θ|x) is N(αx, α) and the predictive density ${\widehat{p}}_{L,\alpha }$, being the convolution of Gaussians, compare (2), is seen to be N(αx, r + α). Now, using $\widehat{d}=r+\alpha$ and $\widehat{\theta }=\alpha X$ in (60), we get

$$\rho \left(\theta ,{\widehat{p}}_{L,\alpha }\right)=\frac{1}{2}\left[\log \left(1+r^{-1}\alpha \right)+{\left(r+\alpha \right)}^{-1}\left\{r+E_{\theta }{\left(\alpha X-\theta \right)}^2\right\}-1\right].$$

The linear risk formula (56) now follows from the quadratic risk of αX. Next, we present some details about the risk of the particular linear estimate ${\widehat{p}}_U$.

Proof of (42)

The estimator ${\widehat{p}}_U={\widehat{p}}_{L,1}$ is given by the N(x, 1 + r) distribution, and so from (59)

$$L\left(\theta ,{\widehat{p}}_U\left(\cdot |x\right)\right)=\frac{1}{2}\log \left(1+r^{-1}\right)+\frac{r+{\left(\theta -x\right)}^2}{2\left(1+r\right)}-\frac{1}{2},$$

from which (42) is immediate.