Posterior consistency in conditional distribution estimation

Abstract

A wide variety of priors have been proposed for nonparametric Bayesian estimation of conditional distributions, and there is a clear need for theorems providing conditions on the prior for large support, as well as posterior consistency. Estimation of an uncountable collection of conditional distributions across different regions of the predictor space is a challenging problem, which differs in some important ways from density and mean regression estimation problems. Defining various topologies on the space of conditional distributions, we provide sufficient conditions for posterior consistency focusing on a broad class of priors formulated as predictor-dependent mixtures of Gaussian kernels. This theory is illustrated by showing that the conditions are satisfied for a class of generalized stick-breaking process mixtures in which the stick-breaking lengths are monotone, differentiable functions of a continuous stochastic process. We also provide a set of sufficient conditions for the case where stick-breaking lengths are predictor independent, such as those arising from a fixed Dirichlet process prior.

1. Introduction

One of the most common problems in data analysis is the need to characterize the dependence of a response on predictors in a flexible manner. We want to avoid parametric assumptions on the response density and how features, such as the mean, variance, skewness, shape and even modality, change with predictors. Nonparametric estimates of the conditional distribution [1, 2] are appealing in this context, but in most applications one requires not just a point estimate but also a characterization of uncertainty. For this reason, and because of excellent practical performance in a rich variety of application areas, Bayesian approaches for conditional distribution estimation have become popular in recent years. The most common class of models are infinite mixture models due in part to the rich literature on algorithms for posterior computation using Markov chain Monte Carlo (MCMC) [3–5] and fast approximations [6]. Such MCMC algorithms are straightforward to implement, and the output can be used to estimate exact posterior densities for functionals of interest.

The ever increasing literature on new nonparametric Bayes models and exciting new applications in areas ranging from finance to biostatistics to machine learning has generated considerable enthusiasm. However, there is a clear lack of frequentist asymptotic theory supporting these models. The emphasis of this article is on substantially closing this gap focusing on a new class of generalized stick-breaking process (gSB) priors, which encompasses a number of the most widely applied priors as special cases.

In the absence of predictors, there is a rich theory and methods literature on nonparametric Bayes methods for estimating a density f using mixture models of the form

$$y_i~f,f~\mathrm{\Pi },$$ (1.1)

where Π is a mixture prior of the form ${\sum }_{h=1}^{\infty }{\pi }_{h}k(y;{\theta }_h)$ for suitably chosen kernel k, atoms and weights {(θ_h, π_h), h = 1, …, ∞} with ${\sum }_{h=1}^{\infty }{\pi }_h=1$ almost surely. The most common choice of Π is the Dirichlet process mixture of normals, first introduced by [7]. Original works on Dirichlet process can be found in [8, 9]. Support of Π in (1.1) and asymptotic properties of the posterior are now well-understood [10–15].

Recent literature has focused on generalizing model (1.1) to the density regression setting in which the entire conditional distribution of y given x changes flexibly with predictors. Bayesian density regression views the entire conditional density f(y | x) as a function valued parameter and allows its center, spread, skewness, modality and other such features to vary with x. For data {(y_i, x_i), i = 1, …, n} let

$$y_i\mid {\mathbf{x}}_i~f(·\mid {\mathbf{x}}_i),\{f(·\mid \mathbf{x}),\mathbf{x}\in X\}~{\mathrm{\Pi }}_{\mathcal{X}},$$ (1.2)

where is the predictor space and is a prior for the class of conditional densities {f_x, x ∈ } indexed by the predictors. Refer, for example, to [16–21] and [22] among others.

The primary focus of this recent development has been infinite mixture models of the form

$$f(y\mid \mathbf{x})=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x})\varphi \left\{\frac{y-{\mu }_h(\mathbf{x})}{{\sigma }_h}\right\},$$ (1.3)

where ϕ is the standard normal density, {π_h(x), h = 1, 2, …} are predictor-dependent probability weights that sum to one almost surely for each x ∈ , and (μ_h, σ_h) ~ G₀ independently, with G₀ a base probability measure on ×ℜ⁺, ⊂ , the space of all → ℜ functions. A single finite mixture of Gaussians is inadequate to represent the shape of the density f(y | x) for different levels of the predictor x unless the number of components is huge. By using an infinite mixture we inherently allow for uncertainty in the number of components needed to characterize the data and bypass the difficult issue of selecting the number of components.

(1.1) is similar in spirit to kernel mixtures used in non-parametric smoothing approaches. However, a major advantage of using a Bayesian paradigm is that we do not need to deal with optimizing tuning parameters, which becomes difficult in higher dimensions. The new adaptation results [23, 24] reveal that even a single prior specification can adapt to the unknown correct smoothness level of the true density and optimizes estimation in an asymptotic minimax sense. For conditional densities, smoothing needs to be done over the response space as well as the predictor space, making the choice of optimal smoothing even more difficult, especially when the predictors have varying degrees of influence on the response. A Bayesian approach offers an easier practical solution in this case.

To our knowledge, only [25] have considered formalizing the notions of support for dependent stick-breaking processes. We focus on a novel class of gSB processes, which express the probability weights π_h(x) in stick-breaking form, with the stick lengths constructed through mapping continuous stochastic processes to the unit interval using a monotone differentiable link function. This class includes dependent Dirichlet processes [26] as a special case.

Only a few papers have considered asymptotic properties of the posterior in conditional density estimation. [22] considers posterior consistency in estimating conditional distributions focusing exclusively on logistic Gaussian process priors [27]. Such priors lack the computational simplicity of the countable mixture priors in (1.3). [28] considers posterior consistency in conditional distribution estimation through a limited information approach by approximating the likelihood by the quantiles of the true distribution. [29, 30] provide sufficient conditions for showing posterior consistency in estimating an autoregressive conditional density and a transition density rather than regression with respect to another covariate.

In this article, focusing on model (1.3), we initially provide sufficient conditions on the prior and true data-generating model under which the prior leads to weak and various types of strong posterior consistency. In this context, we first define notions of weak and L₁-integrated neighborhoods. We then show that the sufficient conditions are satisfied for gSB priors. The theory is illustrated through application to a model relying on probit transformations of Gaussian processes, an approach related to the probit stick-breaking process of [21] and [31]. We also considered Gaussian mixtures of fixed-π dependent processes [26, 32].

[33] showed posterior consistency in conditional density estimation using kernel stick breaking process mixtures of Gaussians in a very recent unpublished article. They approximated a conditional density by a smooth mixture of linear regressions as in [34] to demonstrate the KL property. In this paper, we have shown KL support using a more direct approach of approximating the true density by a kernel mixture of a compactly supported conditional measure.

The fundamental contribution of this article is formalizing the notion of support of the gSB process mixture of Gaussians on the space of conditional densities and formulating sufficient conditions to ensure that it leads to a consistent posterior. In doing so, a key technical contribution is the development of a novel method of constructing a sieve for the proposed class of priors. It has been noted by [35] that the usual method of constructing a sieve by controlling prior probabilities is unable to lead to a consistency theorem in the multivariate case. This is because of the explosion of the L₁-metric entropy with increasing dimension. They developed a technique specific to the Dirichlet process in the multivariate case for showing weak and strong posterior consistency. The proposed sieve¹ avoids the pitfall mentioned by [35] in showing consistency using multivariate mixtures. Our sieve construction has been recently used for studying convergence rates in multivariate density estimation [36, 37].

2. Notations

Throughout the paper, Lebesgue measure on ℜ or ℜ^p is denoted by λ and the set of natural numbers by ℕ. The supremum and the L₁-norms are denoted by ||·||_∞ and ||·||₁ respectively. The indicator function of a set B is denoted by 1_B. Let L_p(ν, M) denote the space of real valued measurable functions defined on M with ν-integrable pth absolute power. For two density functions f, g, the Kullback-Leibler divergence is given by K(f, g) = ∫ log(f/g)fdλ. A ball of radius r with centre x₀ relative to the metric d is defined as B(x₀, r; d). The diameter of a bounded metric space M relative to a metric d is defined to be sup{d(x, y) : x, y ∈ M}. The ε-covering number N(ε, M, d) of a semi-metric space M relative to the semi-metric d is the minimal number of balls of radius ε needed to cover M. The logarithm of the covering number is referred to as the entropy. “≾” stands for inequality up to a constant multiple or if the constant multiple is irrelevant to the given situation. δ₀ stands for a distribution degenerate at 0 and supp(ν) for the support of a measure ν.

3. Conditional density estimation

In this section, we will define the space of conditional densities and construct a prior on this space. It is first necessary to generalize the topologies to allow appropriate neighborhoods to be constructed around an uncountable collection of conditional densities indexed by predictors. With such neighborhoods in place, we then state our main theorems providing sufficient conditions under which various modes of posterior consistency hold for a broad class of predictor-dependent mixtures of Gaussian kernels.

Let = ℜ be the response space and be the covariate space which is a compact subset of ℜ^p. Unless otherwise stated, we will assume = [0, 1]^p without loss of generality. Let denote the space of densities on × w.r.t. the Lebesgue measure and denote a subset of the space of conditional densities satisfying,

$${\mathcal{F}}_d=\{g:\mathcal{X}\times \mathcal{Y}\to (0,\infty ),{\int }_{\mathcal{Y}}g(\mathbf{x},y)dy=1\forall \mathbf{x}\in \mathcal{X},\mathbf{x}\mapsto g(\mathbf{x},·)\text{continuous}\text{as}\mathrm{a}\text{function}\text{from}\mathcal{X}\to L_1(\lambda ,\mathcal{Y})\}.$$

Suppose y_i is observed independently given the covariates x_i, i = 1, 2, … which are drawn independently from a probability distribution Q on . Assume that Q admits a density q with respect to the Lebesgue measure.

If we define h(x, y) = q(x)f(y | x) and h₀(x, y) = q(x)f₀(y | x) then h, h₀ ∈ . Throughout the paper, h₀ is assumed to be a fixed density in which we alternatively refer to as the true data generating density and {f₀(· | x), x ∈ } is referred to as the true conditional density. The density q(x) will be needed only for theoretical investigation. In practice, we do not need to know it or learn it from the data.

We propose to induce a prior on the space of conditional densities through a prior for a collection of mixing measures = {G_x, x ∈ } using the following predictor-dependent mixture of kernels

$$f(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi \left(\frac{y-\mu }{\sigma }\right){dG}_{\mathbf{x}}(\psi ),$$ (3.1)

where ψ = (μ, σ), and

$$G_{\mathbf{x}}\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x}){\delta }_{\{{\mu }_h(\mathbf{x}),{\sigma }_h\}},({\mu }_h,{\sigma }_h)~G_0,$$ (3.2)

where π_h(x) ≥ 0 are random functions of x such that ${\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x})=1$ a.s. for each fixed x ∈ . ${\{{\mu }_h(\mathbf{x}),\mathbf{x}\in X\}}_{h=1}^{\infty }$are i.i.d. realizations of a real valued stochastic process, i.e., G₀ is a probability distribution over ×ℜ⁺, where ⊂ , being the space of functions from to ℜ. Hence for each x ∈ , G_x is a random probability measure over the measurable Polish space (ℜ × ℜ⁺, (ℜ × ℜ⁺)). We are interested the following two important special cases.

3.1. Predictor dependent countable mixtures of Gaussian linear regressions

We define the predictor dependent countable mixtures of Gaussian linear regressions (MGLR_x) as

$$f(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\delta }\right){dG}_{\mathbf{x}}(\mathit{\beta },\sigma ),$$

and

$$G_{\mathbf{x}}=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x}){\delta }_{({\mathit{\beta }}_h,{\sigma }_h)},({\mathit{\beta }}_h,{\sigma }_h)~G_0$$ (3.3)

where π_h(x) ≥ 0 are random functions of x such that ${\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x})=1$ a.s. for each fixed x ∈ and G₀ = G_0,β × G_0,σ is a probability distribution on ℜ^p × ℜ⁺ where G_0,β and G_0,σ are probability distributions on ℜ^p and ℜ⁺ respectively. For a particular choice of π_h(x)’s, we obtain the probit stick-breaking mixtures of Gaussians which have been previously applied to real data applications by [21, 31, 38]. The latter two articles considered probit transformations of Gaussian processes in constructing the stick-breaking weights.

3.2. Gaussian mixtures of fixed-π dependent processes

In (3.1), set G_x as in (3.2) with π_h(x) ≡ π_h for all x ∈ where π_h ≥ 0 are random probability weights ${\sum }_{h=1}^{\infty }{\pi }_h=1$ a.s. and ${\{{\mu }_h(\mathbf{x}),\mathbf{x}\in X\}}_{h=1}^{\infty }$ are as in (3.2). Examples include fixed-π dependent Dirichlet process mixtures of Gaussians [26]. Versions of the fixed π-DDP have been applied to ANOVA [32], survival analysis [39, 40], spatial modeling [41], and many more.

A Gaussian process is a common choice for constructing stochastic processes π_h(x)’s and μ_h(x)’s. Recall that a Gaussian process {α(x) : x ∈ } is defined as a stochastic process for which any finite dimensional representation {α(x₁), …, α(x_p)}, p ≥ 1 has a joint Gaussian distribution. We denote by GP(μ, c) a Gaussian process with mean function μ : → ℝ and c : × → ℝ.

4. Notions of posterior consistency for conditional densities

We recall the definition of posterior consistency through yⁿ = (y₁, …, y_n) and xⁿ = (x₁, …, x_n).

Definition 4.1

The posterior (· | yⁿ, xⁿ) is consistent at {f₀(· | x), x ∈ } with respect to a given topology if (U^c | yⁿ, xⁿ) → 0 a.s. for an arbitrary neighborhood U of {f₀(· | x), x ∈ } in that topology.

Here a.s. consistency at {f₀(· | x), x ∈ } means that the posterior distribution concentrates around a neighborhood of {f₀(· | x), x ∈ } for almost every sequence ${\{y_i,{\mathbf{x}}_i\}}_{i=1}^{\infty }$ generated by i.i.d. sampling from the joint density q(x)f₀(y | x).

We define the weak and ν-integrated L₁ neighborhoods of a collection of conditional densities {f₀(· | x), x ∈ } in the following. A sub-base of a weak neighborhood is defined as

$$W_{\epsilon ,g}(f_0)=\left\{f:f\in {\mathcal{F}}_d,\left|{\int }_{\mathcal{X}\times \mathcal{Y}}gh-{\int }_{\mathcal{X}\times \mathcal{Y}}{gh}_0\right|<\epsilon \right\},$$ (4.1)

for a bounded continuous function g : × → ℜ. A weak neighborhood base is formed by finite intersections of neighborhoods of the type (4.1). Define a ν-integrated L₁ neighborhood

$$S_{\epsilon }(f_0:\nu )=\left\{f:f\in {\mathcal{F}}_d,\int {\Vert f(·\mid \mathbf{x})-f_0(·\mid \mathbf{x})\Vert }_1\nu (\mathbf{x})d\mathbf{x}<\epsilon \right\}$$ (4.2)

for any measure ν with supp(ν) ⊂ . Observe that under the topology in (4.2), can be identified to a closed subset of L₁(λ×ν, ×supp(ν)) making it a complete separable metric space. Thus measurability issues won’t arise with these topologies.

In the following, we define the Kullback-Leibler (KL) property of at a given f₀ ∈ . Note that we define a KL-type neighborhood around the collection of conditional densities f₀ through defining a KL neighborhood around the joint density h₀, while keeping Q fixed at its true unknown value.

Definition 4.2

For any f₀ ∈ , such that h₀(x, y) = q(x)f₀(y | x) is the true joint data-generating density, we define an ε-sized KL neighborhood around f₀ as

$$K_{\epsilon }(f_0)=\{f:f\in {\mathcal{F}}_d,KL(h_0,h)<\epsilon ,h(\mathbf{x},y)=q(\mathbf{x})f(y\mid \mathbf{x})\forall y\in \mathcal{Y},\mathbf{x}\in \mathcal{X}\},$$

where KL(h₀, h) = ∫ h₀ log(h₀/h). Then, is said to have KL property at f₀ ∈ , denoted f₀ ∈ KL( ), if {K_ε(f₀)} > 0 for any ε > 0.

Another definition we would require for showing the KL support is the notion of weak neighborhood of a collection of mixing measures = {G_x, x ∈ } where G_x is a probability measure on S × ℜ⁺ for each x ∈ . Here S = ℜ^p or ℜ depending on the cases considered above. We formulate the notion of a sub-base of the weak neighborhood of = {G_x, x ∈ } below.

Definition 4.3

For a bounded continuous function g : S × ℜ⁺ × → ℜ and ε > 0, a sub-base of the weak neighborhood of a conditional probability measure {F_x, x ∈ } is defined as

$$\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\left|{\int }_{S\times \mathfrak{R}+\times \mathcal{X}}g(s,\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(s,\sigma )q(\mathbf{x})d\mathbf{x}-g(s,\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(s,\sigma )q(\mathbf{x})d\mathbf{x}\right|<\epsilon \right\}$$ (4.3)

A conditional probability measure {G_x, x ∈ } lies in the weak support of if assigns positive probability to every basic neighborhood generated by the sub-base of the type (4.3). In the sequel, we will also consider a neighborhood of the form

$$\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\underset{\mathbf{x}\in \mathcal{X}}{sup}\left|{\int }_{S\times {\mathfrak{R}}^{+}}\{g(s,\sigma ){dG}_{\mathbf{x}}(s,\sigma )-g(s,\sigma ){dF}_{\mathbf{x}}(s,\sigma )\}\right|<\epsilon \right\}.$$ (4.4)

for a bounded continuous function g : S × ℜ⁺ → ℜ.

5. Posterior consistency in MGLR_x mixture of Gaussians

5.1. Kullback-Leibler property

We will work with a specific choice of motivated by the probit stick breaking process construction in [21]. Let

$${\pi }_h(\mathbf{x})=\mathrm{\Phi }\{{\alpha }_h(\mathbf{x})\}\prod _{l<h}[1-\mathrm{\Phi }\{{\alpha }_l(\mathbf{x})\}],$$ (5.1)

where α_h ~ GP(0, c_h), for h = 1, 2, …, ∞. Assume the following holds.

• S1c_h is chosen so that α_h ~ GP(0, c_h) has continuous path realizations
• S2
  for any continuous function under the GP(0, c_h) prior for α_h g : ↦ ℜ,

  $${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\alpha }_h(\mathbf{x})-g(\mathbf{x})\mid <\epsilon \right\}>0$$

  h = 1, …, ∞ and for any ε > 0.

• S3G₀ is absolutely continuous with respect to λ(ℜ^p × ℜ⁺).

Consider the subset ${\mathcal{F}}_d^{\ast }\subset {\mathcal{F}}_d$ satisfying the following conditions.

• A1f is nowhere zero and bounded by M < ∞.
• A2| f(y | x) log f(y | x)dyq(x)dx| < ∞.
• A3 $\mid {\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}f(y\mid \mathbf{x})log\frac{f(y\mid \mathbf{x})}{{\psi }_{\mathbf{x}}(y)}\mathit{dyq}(\mathbf{x})d\mathbf{x}\mid <\infty$,

  where ψ_x(y) = inf_{t∈[y−1,y+1]} f(t | x).

• A4∃ η > 0 such that |y|^2(1+η) f(y | x)dyq(x)dx < ∞.
• A5(x, y) ↦ f(y | x) is jointly continuous.

Remark 5.1

A1 is usually satisfied by common densities arising in practice. A4 imposes a minor tail restriction; e.g., a mean regression model with continuous mean function and a heavy-tailed t residual density with 4 degrees of freedom satisfies A4. Conditions A2 and A3 are more subtle, but are also mild. A flexible class of models which satisfies A1–A5 is as follows. Let y_i = μ(x_i) + ε_i, with μ : → ℜ continuous and ε_i ~ f_xi, where $f_x(\epsilon )={\sum }_{h=1}^H{\pi }_h(x)\psi (\epsilon ;{\mu }_h,{\sigma }_h^2)$ for some H ≥ 1, ${\sum }_{h=1}^H{\pi }_h(x)=1$, π_h : → [0, 1] continuous and ψ is Gaussian or t with greater than 2 degrees of freedom.

Remark 5.2

S2 is satisfied if c_h(x, x′) = e^{−A_h||x−x′||2} and the prior for A_h has full support on ℝ⁺.

The following theorem characterizes the subset of for which has the KL property. The proof of Theorem 5.3 is provided in Appendix C.

Theorem 5.3

f₀ ∈ KL( ) for each f₀ in ${\mathcal{F}}_d^{\ast }$ if satisfies S1–S3.

Remark 5.4

The conditions are satisfied for a class of gSB process mixtures in which the stick-breaking lengths are constructed through mapping continuous stochastic processes to the unit interval using a monotone differentiable link function.

To prove Theorem 5.3, we need several auxiliary results related to the support of the prior which might be of independent interest. The key idea for showing that the true f₀ satisfies {K_ε(f₀)} > 0 for any ε > 0 is to impose certain tail conditions on f₀(y | x) and approximate it by $\stackrel{\sim }{f}(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma }\right)d{\stackrel{\sim }{G}}_{\mathbf{x}}(\mathit{\beta },\sigma )$, where {G̃_x, x ∈ } is compactly supported. Observe that,

$$KL(h_0,h)={\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{f_0(y\mid \mathbf{x})}{\stackrel{\sim }{f}(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}+{\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{\stackrel{\sim }{f}(y\mid \mathbf{x})}{f(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}.$$ (5.2)

We construct such an f̃ in Theorem 5.3 which makes the first term in the right hand side of (5.2) sufficiently small. The following lemma (which is similar to Lemma 3.1 in [12] and Theorem 3 in [11]) guarantees that the second term in the right hand side of (5.2) is also sufficiently small if {G_x, x ∈ } lies inside a finite intersection of neighborhoods of {G̃_x, x ∈ } of the type (4.4).

Lemma 5.5

Assume that f₀ ∈ satisfies y²f₀(y | x)dyq(x)dx < ∞. Suppose $\stackrel{\sim }{f}(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma }\right)d{\stackrel{\sim }{G}}_{\mathbf{x}}(\mathit{\beta },\sigma )$, where ∃a > 0 and 0 < σ < σ̄ such that

$${\stackrel{\sim }{G}}_{\mathbf{x}}({[-a,a]}^p\times (\underset{\_}{\sigma },\overline{\sigma }))=1\forall \mathbf{x}\in \mathcal{X},$$ (5.3)

so that G̃_x has compact support for each x ∈ . Then given any ε > 0, ∃ a finite intersection W of neighborhoods of {G̃_x, x ∈ } of the type (4.4) such that for any conditional density $f(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma }\right){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )$, x ∈ , with {G_x, x ∈ } ∈ W,

$${\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{\stackrel{\sim }{f}(y\mid \mathbf{x})}{f(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}<\epsilon .$$ (5.4)

The proof is similar to Theorem 3 in [11] and is omitted here. In order to ensure that the weak support of is sufficiently large to contain all densities f̃ satisfying the assumptions of Lemma 5.5, we define a collection of fixed conditional probability measures on (ℜ^p × ℜ⁺, (ℜ^p × ℜ⁺)) denoted by ${\mathcal{G}}_{\mathcal{X}}^{\ast }$ satisfying

1. x ↦ F_x(B) is a continuous function of x ∈ , ∀ B ∈ (ℜ^p × ℜ⁺).

2. For any sequence of sets A_n ⊂ ℜ^p × ℜ⁺ ↓ ∅, F_x(A_n) ↓ 0.

Next we state the theorem characterizing the weak support of which will be proved in Appendix B.

Theorem 5.6

If satisfies S1–S3, then any $\{F_{\mathbf{x}},\mathbf{X}\in \mathcal{X}\}\in {\mathcal{G}}_{\mathcal{X}}^{\ast }$ lies in the weak support of .

Corollary 5.7

Assume S1–S3 hold and assume $F_{\mathbf{x}}\in {\mathcal{G}}_{\mathcal{X}}^{\ast }$ is compactly supported, i.e., there exists a, σ, σ̄ > 0 such that F_x([−a, a]^p × [σ, σ̄]) = 1. Then for a bounded uniformly continuous function g : ℜ^p × ℜ⁺ → [0, 1] satisfying g(β, σ) → 0 as ||β|| → ∞, σ → ∞,

$${\mathcal{P}}_{\mathcal{X}}\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\underset{\mathbf{x}\in \mathcal{X}}{sup}\left|{\int }_{{\mathfrak{R}}^p\times {\mathfrak{R}}^{+}}\{g(\mathit{\beta },\sigma ){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )-g(\mathit{\beta },\sigma ){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\}\right|<\epsilon \right\}>0.$$ (5.5)

Proof

The proof is similar to Theorem 5.6 with the L₁ convergence in (B.1) replaced by convergence uniformly in x. This is because under the assumptions of Corollary 5.7, the uniformly continuous sequence of functions ${\sum }_{k=1}^{n}g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n})F_{\mathbf{x}}(A_{k,n})$ on monotonically decreases to ∫_C g(β, σ)dF_x(β, σ) as n → ∞ where C is given by [−a, a]^p × [σ, σ̄].

The proof of the following corollary is along the lines of the proof of Theorem 5.6 and is omitted here.

Corollary 5.8

Under the assumptions of Corollary 5.7 for any k₀ ≥ 1,

$${\mathcal{P}}_{\mathcal{X}}\left\{{\cap }_{j=1}^{k_0}{U}_j\right\}>0,$$ (5.6)

where U_j’s are neighborhoods of the type (5.5).

5.2. Strong Consistency with the q-integrated L₁ neighborhood

To obtain strong consistency in the q-integrated L₁ topology, we need a very straight forward extension of Theorem 2 of [11] below.

Theorem 5.9

Suppose f₀ ∈ KL( ) and there exist subsets ⊂ with

1. log N (ε, , ||·||₁) = o(n),

2. ${\mathrm{\Pi }}_{\mathcal{X}}({\mathcal{F}}_n^c)\le c_2{e}^{-n{\beta }_2}$ for some c₂, β₂ > 0,

then the posterior is strongly consistent with respect to the q-integrated L₁ neighborhood.

Before stating the main theorem on strong consistency, we consider a hierarchical extension of MGLR_x where the bandwidths are taken to be random. We define a sequence of random inverse-bandwidths A_h of the Gaussian process α_h, h ≥ 1 each having ℜ⁺ as its support. Since the first few atoms suffice to explain most of the dependence of y on x, we expect that the variability due to the covariate in the stochastic process Φ{α_h} decreases as h increases. This is achieved through a carefully chosen prior for the covariance kernel c_h of the Gaussian process α_h.

Let α₀ denote the base Gaussian process on [0, 1]^p with covariance kernel c₀(x, x′) = τ²e^{−||x−x′||2}. Then ${\alpha }_h(\mathbf{x})={\alpha }_0(A_h^{1/2}\mathbf{x})$ for each x∈ The variability of α_h with respect to the covariate is shrunk or stretched to the rectangle ${[0,A_h^{1/2}]}^p$ as A_h decreases or increases. A_h’s are constructed to be stochastically decreasing to δ₀ in the following manner. We assume that there exist η, η₀ > 0 and a sequence δ_n = O((log n)²/n^5/2) such that P (A_h > δ_n) ≤ exp{−n^−η₀h^(η₀+2)/η log h} for each h ≥ 1. Also assume that there exists a sequence r_n ↑ ∞ such that $r_n^p{n}^{\eta }{(logn)}^{p+1}=o(n)$ and P (A_h > r_n) ≤ e⁻ⁿ. We will discuss how to construct such a sequence of random variables in the Remark 5.12 following Theorem 5.10.

The following theorem provides sufficient conditions for strong posterior consistency in the q-integrated L₁ topology. The proof is provided in Appendix D.

Theorem 5.10

Let π_h’s satisfy (5.1) with α_h ~ GP(0, c_h) where c_h(x, x′) = τ²e^{−A_h||x−x′||2}, h ≥1, τ² > 0 fixed.

• C1There exist sequences a_n, h_n ↑ ∞, l_n ↓ 0 with $\frac{a_n}{l_n}=O(n),\frac{h_n}{l_n}=O(e^n)$, and constants d₁, d₂ > 0 such that G₀{B(0; a_n) × [l_n, h_n]}^c < d₁e^−d₂n for some d₁, d₂ > 0.
• C2A_h’s are constructed as in the second last paragraph before Theorem 5.10.

then f₀ ∈ KL( ) implies that achieves strong posterior consistency in q-integrated L₁ topology at f₀.

Remark 5.11

Verification of condition C1 of Theorem 5.10 is particularly simple. For example, if G₀ is a product of multivariate normals on β and an inverse Gamma prior on σ², the condition C1 is satisfied with $a_n=O(\sqrt{n})$, h_n = eⁿ, $l_n=O(\frac{1}{\sqrt{n}})$. It follows from [42] that f₀ ∈ KL( ) is still satisfied when we have the additional assumptions C1–C2 together with S1–S3 on the prior .

Remark 5.12

Since we need $r_n^p{n}^{\eta }{(logn)}^{p+1}=o(n),r_n^p$ can be chosen to be O(n^η₁) for some 0 < η₁ < 1. Let d be such that dη₁/p ≥ 1 and set η₀ = 3d. Let A_h = c_hB_h, where $B_h^d~\mathit{Exp}(\lambda )$ and c_h = (h^(3d+2)/η log h)^−1/d for any 0 < η < 1. Then P (A_h > n^η₁/p) ≤ P(B_h > n^η₁/p) ≤ e^−ndη₁/p ≤ e⁻ⁿ and P(A_h > (log n)²n^−5/2) ≤ exp{−n^−3dh^(3d+2)/η log h}.

Remark 5.13

The theory of strong posterior consistency can be generalized to an arbitrary monotone differentiable link function L : ℜ ↦ [0, 1] which is Lipschitz, i.e., there exists a constant K > 0 such that |L(x) − L(x′)| ≤ K |x − x′| for all x, x′ ∈ . Also, as long as the π_h(x)’s satisfy the hypothesis of Lemma Appendix A.1 and possess the required tail probability in Lemma 5.15, general predictor dependent mixing weights can be used.

Below we will develop several auxiliary results required to prove Theorem 5.10. They are stated below as some of them might be of independent interest. Let ${\varphi }_{\mathit{\beta },\sigma }(\mathbf{x},y):=\frac{1}{\sigma }\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma }\right)$ for y ∈ and x ∈ . From [12], we obtain for ${\sigma }_2>{\sigma }_1>\frac{{\sigma }_2}{2}$ and for each x ∈ ,

$${\int }_{\mathcal{Y}}\mid {\varphi }_{{\mathit{\beta }}_1,{\sigma }_1}(\mathbf{x},y)-{\varphi }_{{\mathit{\beta }}_2,{\sigma }_2}(\mathbf{x},y)\mid dy\le {\left(\frac{2}{\pi }\right)}^{1/2}\frac{\Vert {\mathit{\beta }}_2-{\mathit{\beta }}_1\Vert \sqrt{p}}{{\sigma }_2}+\frac{3({\sigma }_2-{\sigma }_1)}{{\sigma }_1}$$

Construct a sieve for (β, σ) as

$${\mathrm{\Theta }}_{a,h,l}=\{{\varphi }_{\mathit{\beta },\sigma }:\Vert \mathit{\beta }\Vert \le a,l\le \sigma \le h\}.$$ (5.7)

In the following Lemma, we provide an upper bound to N (Θ_a,h,l, ε, d_SS). The proof is omitted as it follows trivially from Lemma 4.1 in [12].

Lemma 5.14

There exist constants d₁, d₂ > 0 such that $N({\mathrm{\Theta }}_{a,h,l},\epsilon ,d_{SS})\le d_1{\left(\frac{a}{l}\right)}^p+d_{2}log\frac{h}{l}+1$.

In the proof of Theorem 5.10, we will verify the sufficient conditions of Theorem 5.9. We calibrate by a carefully chosen sequence of subsets ⊂ . The fundamental problem with mixture models ∫ N (y; μ, σ²I_p)dP (μ) in estimating a multivariate density lies in attempting to compactify the model space by {∫ N (y, μ, σ²I_p)dP (μ) : P ((−a_n, a_n]^p) > 1 − δ} for each σ leading to an entropy $a_n^p$ growing exponentially with the dimension p. Here we marginalize P in ∫ N(y; μ, σ²I_p)dP (μ) to yield the following construction ${\sum }_{h=1}^{m_n}{\pi }_{h}N(y;{\mu }_h,{\sigma }^2{I}_p):\Vert {\mu }_h\Vert \le a_n$, h = 1, …, m_n, ${\sum }_{h=m_n+1}^{\infty }{\pi }_h<\epsilon$ leading to an entropy m_n log a_n where m_n is related to the tail-decay of $P({\sum }_{h=m_n+1}^{\infty }{\pi }_h>\epsilon )$. With this idea in place, we extend the construction of for conditional densities below.

Before constructing a sieve, we briefly review alternative definitions [43] of a Gaussian process as a Banach space valued element below. A Borel measurable random element W with values in a separable Banach space ( , ||·||) is called Gaussian if the random variable b^*W is normally distributed for any element b^* ∈ , the dual space of . Recall that in general, the reproducing kernel Hilbert space (RKHS) ℍ attached to a zero-mean Gaussian process W is defined as the collection of all EHW for H ranging over the closed linear span of the variables b^*W in L₂(ν, M) with inner product

$${\langle \mathrm{E}W(·)H_1;\mathrm{E}W(·)H_2\rangle }_{ℍ}=\mathrm{E}{H}_1{H}_2,$$ (5.8)

The RKHS can be viewed as a subset of and the RKHS norm ||·||_ℍ stronger than the Banach space norm ||·||.

In particular, if W is a Borel measurable zero-mean Gaussian random element in a complete separable subspace of ℓ^∞ (T), the Banach space of uniformly bounded functions g : T → ℝ equipped with the uniform norm ||g|| = sup{|g(t)| : t ∈ T }, then the RKHS is actually the completion of the linear space of functions t ↦ EW (t)H relative to the inner product (5.8) where H, H₁ and H₂ are finite linear combinations of the form Σ_i a_iW (s_i) with a_i ∈ ℝ and s_i in the index set of W. See Theorem 2.1 of [43] for details.

Next we turn to constructing the sieve. Assume ε > 0 is given. Let ${ℍ}_1^a$ denote a unit ball in the RKHS of the covariance kernel τ²e^{−a||x−x′||2}and is a unit ball in ℂ [0, 1]^p. For numbers M, m, r, δ, construct a sequence of subsets {B_h, h = 1, …, m} of ℂ [0, 1]^p as follows.

$$B_h=\{\begin{array}{l}(M\sqrt{r/\delta }{ℍ}_1^r+\frac{\epsilon }{m^2}{\mathbb{B}}_1)\cup ({\cup }_{a<\delta }M{ℍ}_1^a+\frac{\epsilon }{m^2}{\mathbb{B}}_1),\text{if}h=1,\dots ,m^{\eta }\hfill \\ {\cup }_{a<{\delta }_n}{M}_n{ℍ}_1^a+\frac{\epsilon }{m^2}{\mathbb{B}}_1,\text{if}h=m^{\eta }+1,\dots ,m.\hfill \end{array}$$

The idea is to construct

$$\begin{array}{l}{\mathcal{F}}_n=\{f:f(y\mid \mathbf{x})=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x})\frac{1}{{\sigma }_h}\varphi \left(\frac{y-{\mathbf{x}}^{\prime }{\mathit{\beta }}_h}{{\sigma }_h}\right),{\{{\varphi }_{{\mathit{\beta }}_h,{\sigma }_h}\}}_{h=1}^{m_n}\\ \in {\mathrm{\Theta }}_{a_n,h_n,l_n},{\alpha }_h\in B_{h,n},h=1,\dots ,m_n,\underset{\mathbf{x}\in \mathcal{X}}{sup}\sum _{h\ge m_n+1}{\pi }_h(\mathbf{x})\le \epsilon \}.\end{array}$$ (5.9)

for appropriate sequences a_m, l_n, h_n, M_n, m_n, r_n, δ_n to be chosen in the proof of Theorem 5.10.

The following lemma is also crucial to the proof of Theorem 5.10 which allows us to calculate the rate of decay of P( π_h(x) > ε) with m_n.

Lemma 5.15

Let π_h’s satisfy (5.1) with α_h ~ GP(0, c_h) where c_h(x, x′) = τ²e^{−A_h||x−x′||2}, h ≥ 1, τ² > 0 fixed. Then for some constant C₇ > 0,

$${\mathrm{\Pi }}_{\mathcal{X}}\left({\Vert \sum _{h=m_n+1}^{\infty }{\pi }_h\Vert }_{\infty }>\epsilon \right)\le e^{-C_7{m}_{n}log{m}_n}+\sum _{n=m_n^{\eta }+1}^{m_n}P(A_h>{\delta }_n).$$ (5.10)

Proof

Let $W_h=-log[1-\mathrm{\Phi }\{{\alpha }_h^{\prime }\}]$ where ${\alpha }_h^{\prime }={inf}_{\mathbf{x}\in \mathcal{X}}{\alpha }_h(\mathbf{x})$, Z_h ~ Ga(1, γ₀). We will choose an appropriate value for γ₀ in the sequel. Let t₀ = −log ε > 0. Observe that

$${\mathrm{\Pi }}_{\mathcal{X}}\left({\Vert \sum _{h=m_n+1}^{\infty }{\pi }_h\Vert }_{\infty }>\epsilon \right)={\mathrm{\Pi }}_{\mathcal{X}}\left(-\sum _{h=m_n^{\eta }+1}^{m_n}log\{1-\mathrm{\Phi }({\alpha }_h^{\prime })\}<t_0\right).$$

Observe that ${\mathrm{\Pi }}_{\mathcal{X}}\left(-{\sum }_{h=1}^{m_n}log\{1-\mathrm{\Phi }({\alpha }_h)\}<t_0\right)={\mathrm{\Pi }}_{\mathcal{X}}({\mathrm{\Lambda }}_h<t_0)$ where Λ_h ~ Ga(m_n, 1). Then it is easy to show that (Λ_h < t₀) ≾ e^{−m_n log m_n}. However, the calculation gets complicated when α_h’s are i.i.d realizations of a zero mean Gaussian process. The proof relies on the fact that the supremum of Gaussian processes has sub-Gaussian tails.

Below we calculate the rate of decay of ${\mathrm{\Pi }}_{\mathcal{X}}\left({\Vert {\sum }_{h=m_n+1}^{\infty }{\pi }_h\Vert }_{\infty }>\epsilon \right)$ with m_n. We will show that there exists γ₀, depending on ε and τ but not depending on n, such that

$${\mathrm{\Pi }}_{\mathcal{X}}\left(\sum _{h=m_n^{\eta }+1}^{m_n}{W}_h+t_0\right)\le \xi {({\delta }_n)}^{m_n-m_n^{\eta }}{\mathrm{\Pi }}_{\mathcal{X}}\left(\sum _{h=m_n^{\eta }+1}^{m_n}{Z}_h<t_0\right)+\sum _{h=m_n^{\eta }+1}^{m_n}P(A_h>{\delta }_n).$$ (5.11)

where there exists a constant C₅ > 0 such that ξ(x) = C₅x^p/2 for x > 0. Observe that ${\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=m_n^{\eta }+1}^{m_n}{W}_h<t_0\right)\le {\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=m_n^{\eta }+1}^{m_n}{W}_h<t_0,A_h\le {\delta }_n,h=m_n^{\eta }+1,\dots ,m_n\right)+{\sum }_{h=m_n^{\eta }+1}^{m_n}P(A_h>{\delta }_n)$.

Since ${\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=m_n^{\eta }+1}^{m_n}{W}_h<t_0\right)={\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=m_n^{\eta }+1}^{m_n}({\tau }^{\prime }/\tau )W_h<{\tau }^{\prime }{t}_0/\tau \right)$ for some τ′ < 1, we can re-parameterize t₀ as τ′t₀/τ and τ as τ′. Hence without loss of generality we assume τ < 1.

Define g : [0, t₀] → ℜ, t ↦ −Φ⁻¹(1 − e^−t). It holds that g is a continuous function on (0, t₀]. Assume α₀ ~ GP(0, c₀) where c₀(x, x′) = τ²e^{−||x−x′||2}. For $h=m_n^{\eta }+1,\dots ,m_n$,

$$P(\underset{\mathbf{x}\in \mathcal{X}}{sup}{\alpha }_h(\mathbf{x})\ge \lambda ,A_h\le {\delta }_n)\le P(\underset{\mathbf{x}\in \sqrt{{\delta }_n}\mathcal{X}}{sup}{\alpha }_0(\mathbf{x})\ge \lambda ).$$

Below we estimate $P({sup}_{\mathbf{x}\in \sqrt{{\delta }_n}\mathcal{X}}{\alpha }_0(\mathbf{x})\ge \lambda )$ for large enough λ following Theorem 5.2 of [44]. However extra care is required to identify the role of δ_n. Since $N(\epsilon ,\sqrt{{\delta }_n}\mathcal{X},\Vert ·\Vert )\le C_1{(\sqrt{{\delta }_n}/\epsilon )}^p$,

$${\int }_0^{\epsilon }{\{logN(\epsilon ,\sqrt{{\delta }_n}\mathcal{X},\Vert ·\Vert )\}}^{1/2}d\epsilon \le C_2\epsilon \{1+\sqrt{log(1/\epsilon )}\}.$$

for some constant C₂ > 0. Hence

$$\begin{array}{l}P(\underset{\mathbf{x}\in r_n\mathcal{X}}{sup}{\alpha }_0(\mathbf{x})\ge \lambda )\le C_3{(\sqrt{{\delta }_n}\lambda )}^{p}exp[-1/2{\{\lambda -C_2/\lambda (1+\sqrt{log\lambda })\}}^2/{\tau }^2]\\ \le C_3{\delta }_n^{p/2}{\lambda }^{p+2}\{1-\mathrm{\Phi }(\lambda /{\tau }^2)\}\le C_4{\delta }_n^{p/2}\{1-\mathrm{\Phi }(\lambda )\}.\end{array}$$

for constants C₃, C₄ > 0. The last inequality holds for all large λ because τ < 1. Hence there exists t₁ ∈ (0, t₀) sufficiently small and independent of n such that for all t ∈ (0, t₁), ${\mathrm{\Pi }}_{\mathcal{X}}\{{sup}_{\mathbf{x}\in \sqrt{{\delta }_n}\mathcal{X}}{\alpha }_0(\mathbf{x})\ge g(t)\}\le C_4{\delta }_n^{p/2}\mathrm{\Phi }\{-g(t)\}$. Observe that

$${\mathrm{\Pi }}_{\mathcal{X}}\{\underset{\mathbf{x}\in \sqrt{{\delta }_n}\mathcal{X}}{sup}{\alpha }_0(\mathbf{x})\ge g(t)\}\le C_4{\delta }_n^{p/2}\mathrm{\Phi }\{-g(t)\}<C_5{\delta }_n^{p/2}(1-e^{-{\gamma }_{0}t}),$$

for any γ₀ > 1. Further choose γ₀ large enough such that 2(1 − e^−γ₀t) > 1 ∀ t ∈ [t₁, t₀]. Hence P(W_h ≤ t, A_h ≤ δ_n) ≤ ξ(δ_n)P(Z_h < t) ∀ t ∈ (0, t₀] where $\xi ({\delta }_n)=C_5{\delta }_n^{p/2}$, with C₅ = max{2, C₄}. Applying Lemma Appendix E.1, we conclude (5.11) by induction. Lemma Appendix E.1 is proved in Appendix E. As ${\sum }_{h=1}^{m_n}{Z}_h~\text{Ga}(m_n,{\gamma }_0),{\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=1}^{m_n}{Z}_h<t_0\right)\le e^{-C_6{m}_{n}log{m}_n}$ for some constant C₆ > 0. Since $\xi {({\delta }_n)}^{m_n-m_n^{\eta }}{\mathrm{\Pi }}_{\mathcal{X}}\left({\sum }_{h=1}^{m_n}{Z}_h<t_0\right)\le (e^{-C_7{m}_{n}log{m}_n})$ for some constant C₇ > 0, the result follows immediately.

5.3. Prior specification and posterior computation

To illustrate the applicability of the proposed methods, we mention the prior choices and key steps for posterior computation for the MGLR_x model. Recall that

$$f(y\mid \mathbf{x})-\int \frac{1}{2}\varphi \left(\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma }\right){dG}_{\mathbf{x}}(\mathit{\beta },\sigma ),$$ (5.12)

$$G_{\mathbf{x}}=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x}){\delta }_{({\mathit{\beta }}_h,{\sigma }_h^2)},({\mathit{\beta }}_h,{\sigma }_h^{-2})~N({\mathit{\beta }}_0,{\mathrm{\sum }}_0)\times \text{Ga}({\alpha }_{\sigma },{\beta }_{\sigma }),$$ (5.13)

where π_h(x) = Φ{α_h(x)}Π_l<h{1−Φ{α_l(x)}. We assume α_h ~ GP(0, c_h), where $c_h(\mathbf{x},{\mathbf{x}}^{\prime })=\frac{1}{{\tau }_{\alpha }}{e}^{-A_h{\Vert \mathbf{x}-{\mathbf{x}}^{\prime }\Vert }^2}$, τ_α ~ Ga(ν_α/2, ν_α/2). See Remark 5.12 for constructing prior for A_h’s. If the y_i’s are standardized, we would expect that the total variance ${\sum }_{h=1}^{\infty }{\pi }_h{\sigma }_h^2$ should be around 1. Hence choose a_σ = 1, b_σ = 10 so that the $E({\sigma }_h^{-2})=0.1$. We can resort to an MCMC algorithm, which is a hybrid of data augmentation, the exact block Gibbs sampler of [45] and Metropolis Hastings sampling to sample from the posterior of (5.12). [45] proposed the exact block Gibbs sampler as an efficient approach to posterior computation in infinite-dimensional Dirichlet process mixture models, modifying the block Gibbs sampler of [46] to avoid truncation approximations. The exact block Gibbs sampler combines characteristics of the retrospective sampler [47] and the slice sampler [4, 48]. Introduce γ₁, …, γ_n such that π_h(x_i) = P(γ_i = h), h = 1, 2, …, ∞. Then

$${\gamma }_i~\sum _{h=1}^{\infty }{\pi }_h({\mathbf{x}}_i){\delta }_h=\sum _{h=1}^{\infty }1(u_i<{\pi }_h({\mathbf{x}}_i)){\delta }_h$$

where u_i ~ U(0, 1).

We continue up to $h^{\ast }=max\{h_1^{\ast },\dots ,h_n^{\ast }\}$, where $h_i^{\ast }$ is the minimum integer satisfying ${\sum }_{l=1}^{h_i^{\ast }}{\pi }_l({\mathbf{x}}_i)>1-min\{u_1,\dots ,u_n\}$, i = 1, …, n. The Markov chain adaptively estimates the desired number of components h^* at each iteration of the MCMC, thus making it more efficient than a finite mixture model with a pre-specified large number of components. Here we describe the key steps for the posterior computation.

1. Update u_i’s and stick breaking random variables: Generate

   $$u_i\mid -~U(0,{\pi }_{{\gamma }_i}({\mathbf{x}}_i))$$
   where π_h(x_i) = Φ{α_h(x_i)}Π_l<h[1 − Φ{α_l(x_i)}]. For i = 1, …, n, introduce latent variables Z_h(x_i), h = 1, 2, … such that Z_h(x_i) ~ N(α_h(x_i), 1). Thus π_h(x_i) = P(Z_h(x_i) > 0, Z_l(x_i) < 0 for l < h). Then

   $$Z_h({\mathbf{x}}_i)\mid -~\{\begin{array}{l}\mathrm{N}({\alpha }_h({\mathbf{x}}_i),1)I_{{ℝ}^{+}},h={\gamma }_i\hfill \\ \mathrm{N}({\alpha }_h({\mathbf{x}}_i),1)I_{{ℝ}^{-}},h<{\gamma }_i.\hfill \end{array}$$
   Let Z_h = (Z_h(x₁), …, Z_h(x_n))′ and α_h = (α_h(x₁), …, α_h(x_n))′. Letting (Σ_h)_ij = e^{−A_h||xi₋x_j||}, Z_h ~ N(α_h, I) and ${\mathit{\alpha }}_h~\mathrm{N}(0,\frac{1}{{\tau }_{\alpha }}{\mathbf{\sum }}_h)$,

   $${\mathit{\alpha }}_h\mid -~\mathrm{N}({({\tau }_{\alpha }{\mathbf{\sum }}_h^{-1}+{\mathbf{I}}_n)}^{-1}{\mathbf{Z}}_h,{({\tau }_{\alpha }{\mathbf{\sum }}_h^{-1}+{\mathbf{I}}_n)}^{-1})$$
   Continue up to $h^{\ast }=max\{h_1^{\ast },\dots ,h_n^{\ast }\}$, where $h_i^{\ast }$ is the minimum integer satisfying ${\sum }_{l=1}^{h_i^{\ast }}{\pi }_l({\mathbf{x}}_i)>1-min\{u_1,\dots ,u_n\}$, i = 1, …, n. Now

   $${\tau }_{\alpha }\mid -~\text{Ga}\left(\frac{1}{2}({nh}^{\ast }+{\nu }_{\alpha }),\frac{1}{2}\left(\sum _{l=1}^{h^{\ast }}{\mathit{\alpha }}_k^{\prime }{\mathbf{\sum }}_l^{-1}{\mathit{\alpha }}_k+{\nu }_{\alpha }\right)\right),$$

   while κ_α is updated using a Metropolis Hastings step.

2. Update allocation to atoms: Update (γ₁, …, γ_n)|– as multinomial random variables with probabilities

   $$P({\gamma }_i=h)\propto \mathrm{N}(y_i;{\mathbf{x}}_i^{\prime }{\mathit{\beta }}_h,{\tau }_h^{-1})I(u_i<{\pi }_h({\mathbf{x}}_i)),h=1,\dots ,h^{\ast }.$$
3. Update component-specific locations and precisions: Let n_h = #{i : γ_i = h}, l = 1, 2, …, h^*. Let Y_h = (y_i : γ_i = h) be a n_h dimensional vector and X_h is the corresponding n_h × p dimensional covariate matrix.

   $$\begin{array}{l}{\mathit{\beta }}_h\mid -~\mathrm{N}({({\mathbf{X}}_h^{\prime }{\mathbf{X}}_h+{\mathrm{\sum }}_0^{-1})}^{-1}({\mathbf{X}}_h^{\prime }{Y}_h+{\mathrm{\sum }}_0^{-1}{\mathit{\beta }}_0),{({\mathbf{X}}_h^{\prime }{\mathbf{X}}_h+{\mathrm{\sum }}_0^{-1})}^{-1})\\ {\tau }_h\mid -~\text{Ga}\left(\frac{n_h}{2}+{\alpha }_{\tau },{\beta }_{\tau }+\sum _{i:{\gamma }_i=h}{(y_i-{\mathbf{x}}_i^{\prime }{\mathit{\beta }}_h))}^2\right),h=1,2,\dots ,h^{\ast }\end{array}$$

   Update A_h’s in a Metropolis Hastings step.

At each iteration of the MCMC, we obtain samples from the full conditional distributions of the parameters, which after discarding a burn-in can be used to get summary statistics of posterior distribution of the parameters or functionals of interest.

6. Posterior consistency in Gaussian mixture of fixed-π dependent processes

6.1. Kullback-Leibler property

The following theorem verifies that has KL property at $f_0\in {\mathcal{F}}_d^{\ast }$. The proof of Theorem 6.1 is somewhat similar to that of Theorem 5.3 and can be found in Appendix F.

Theorem 6.1

f₀ ∈ KL( ) for each f₀ in ${\mathcal{F}}_d^{\ast }$ if satisfies

• T1G₀ is specified by μ_h ~ GP(μ, c), σ_h ~ G₀,_σ where c is chosen so that GP(0, c) has continuous path realizations and Π_σ is absolutely continuous w.r.t. Lebesgue measure on ℜ⁺.
• T2For every k ≥2, (π₁, …, π_k) is absolutely continuous w.r.t. to the Lebesgue measure on S_k−1.
• T3
  For any continuous function g : ↦ ℜ,

  $${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\mu }_h(\mathbf{x})-g(\mathbf{x})\mid <\epsilon \right\}>0$$

  h = 1, …, ∞ and for any ε > 0.

6.2. Strong consistency with the q-integrated L₁ neighborhood

Next we summarize the consistency theorem with respect to the q-integrated L₁ topology. The proof of Theorem 6.2 is also similar to that of Theorem 5.10 and is provided in Appendix G.

Theorem 6.2

Let μ_h(x) = x′β_h + η_h(x), β_h ~ G_β and η_h ~ GP(0, c), h = 1, …, ∞ where c(x, x′) = τ²e^{−A||x−x′||2}, A^{p(1+η₂)/η₂} ~ Ga(a, b) for some η₂ > 0.

• F1There exist sequences a_n, h_n ↑ ∞, l_n ↓ 0 with $\frac{a_n}{l_n}=O(n),\frac{h_n}{l_n}=O(e^n)$, and constants d₁, d₂, d₃ and d₄ > 0 such that G_β{B(0; a_n)}^c < d₁e^−d₂n and G_0,σ{[l_n, h_n]}^c ≤ d₃e^−d₄n.
• F2 $P({\sum }_{h=n}^{\infty }{\pi }_h>\epsilon )\precsim O(e^{-n^{1+{\eta }_2}{(logn)}^{(p+1)}})$.

  then f₀ ∈ KL( ) implies that achieves strong posterior consistency at f₀ with respect to the q-integrated L₁ topology.

Remark 6.3

F2 is satisfied if π_h’s are made to decay more rapidly than the usual Beta(1, α) stick-breaking random variables, e.g, if π_h = ν_hΠ_l<h(1−ν_h) and if ν_h ~ Beta(1, α_h) where α_h = h^1+η₂(log h)^p+1α₀ for some α₀ > 0, then F2 is satisfied. Large value of α_h for the higher indexed weights favors smaller number of components.

Remark 6.4

A Gaussian kernel is used here for technical simplification. One can obtain similar results using a variety of kernels e.g. t, Laplace, etc. However, the KL support conditions A1–A5 will be different for different kernels. Refer to [49] for a catalogue of conditions for various kernels in a density estimation framework.

7. Discussion

We have provided sufficient conditions to show posterior consistency in estimating the conditional density via predictor dependent mixtures of Gaussians which include probit stick-breaking mixtures of Gaussians and the fixed-π dependent processes as special cases. The problem is of interest, providing a more flexible and informative alternative to the usual mean regression. For both the models, we need the same set of tail conditions (mentioned in ${\mathcal{F}}_d^{\ast }$) on f₀ for KL support. Although the first prior is flexible in the weights and the second one in the atoms through their corresponding GP terms, S1, S2, T1 and T3 show that verification of KL property only requires that both the GP terms have continuous path realizations and desired approximation property. Moreover, for the second prior, any set of weights summing to one a.s. T2 suffices for showing KL property. Careful investigations of the prior for the GP kernel for the first model and the probability weights for the second one are required for strong consistency. For the first one we need the covariate dependence of the higher indexed GP terms in the weights to fade off. On the other hand, for the second model, the atoms can be i.i.d. realizations of a GP with Gaussian covariance kernel with inverse- Gamma bandwidth while limiting the model complexity through a sequence of probability weights which are allowed to decay rapidly. This suggests that full flexibility in the weights should be down-weighted by an appropriately chosen prior while full flexibility in the atoms should be accompanied by a restriction imposing fewer number of components. It would be interesting to see how the conditions on the bandwidth can be modified when we actually use a sieve Bayes prior, i.e. a prior with number of components k_n diverging to ∞.

Another interesting direction is to consider rates of convergence of the posterior and Bernstein von-Mises (BvM) type results. For infinite dimensional parameters [50], there has been quite a few positive BvM results very recently for linear functionals of a probability density function [51] and for general classes of linear and non-linear functionals in a Gaussian white noise model [52]. We conjecture that such BvM-type results hold for linear functionals of conditional density (e.g. conditional mean, conditional cdf) too under appropriate conditions on the prior and the true data generating conditional density.

Appendix A. A useful lemma

To prove Theorem 5.6, we need an auxiliary lemma which we state below.

Lemma Appendix A.1

If {π_h(x), h = 1, …, ∞} constructed as in (5.1) satisfies S1 and S2 then

$${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\pi }_1(\mathbf{x})-F_{\mathbf{x}}(A_1)\mid <{\epsilon }_1,\dots ,\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\pi }_k(\mathbf{x})-F_{\mathbf{x}}(A_k)\mid <{\epsilon }_k\right\}>0.$$ (A.1)

for a measurable partition {A_i, i = 1, …, k} of ℜ^p × ℜ⁺, ε_i > 0 and a conditional cdf {F_x, x ∈ }.

Proof

Without loss of generality, let 0 < F_x(A_i) < 1, i = 1, …, k ∀ x ∈ . We want to show that for any ε_i > 0, i = 1, …, k, (A.1) holds. Construct continuous functions g_i : ↦ ℜ, 0 < g_i(x) < 1 ∀x ∈ , i = 1, …, k−1 such that

$$g_1(\mathbf{x})=F_{\mathbf{x}}(A_1),g_i(\mathbf{x})\prod _{l<i}\{1-g_l(\mathbf{x})\}=F_{\mathbf{x}}(A_i),2\le i\le k-1,g_k(\mathbf{x})=1\forall \mathbf{x}.$$ (A.2)

As 0 < Fx(A_i) < 1, i = 1, …, k ∀ x ∈ , it is trivial to find g_i, i = 1, …, k satisfying (A.2) since one can solve back for the g_i’s from (A.2). ${\sum }_{i=1}^k{F}_{\mathbf{x}}(A_i)=1$ enforces g_k ≡1. Since Φ is a continuous function, for any ε_i > 0, i = 1, …, k − 1,

$${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid \mathrm{\Phi }\{{\alpha }_i(\mathbf{x})\}-g_i(\mathbf{x})\mid <{\epsilon }_i\right\}>0$$ (A.3)

and for i = k,

$${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid \mathrm{\Phi }\{{\alpha }_k(\mathbf{x})\}-1\mid <{\epsilon }_k\right\}={\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{inf}{\alpha }_k(\mathbf{x})>{\mathrm{\Phi }}^{-1}(1-{\epsilon }_k)\right\}.$$ (A.4)

Choose M > Φ⁻¹(1 − ε_k) + ε_k. We have 0 < M < 1 and

$$\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\alpha }_k(\mathbf{x})-M\mid <{\epsilon }_k\right\}\subset \left\{\underset{\mathbf{x}\in \mathcal{X}}{inf}{\alpha }_k(\mathbf{x})>{\mathrm{\Phi }}^{-1}(1-{\epsilon }_k)\right\}.$$

Hence by assumption, { α_k(x) > Φ⁻¹(1 − ε_k)} > 0. Let S_k−1 denote the k-dimensional simplex. For notational simplicity let p_i(x) = Φ{α_i(x)}, g_i(x) = F_x(A_i), i = 1, …, k−1 and g_k(x) = 1. Let z = (z₁, …, z_p)′, f_i : S_k−1 → ℜ, z ↦ z_iΠ_l<i(1 − z_l), i = 2, …, k and f₁(z) = z₁. Let p(x) = (p₁(x), …, p_k(x)) and g(x) = (g₁(x), …, g_k(x)). Then we need to show that

$${\mathcal{P}}_{\mathcal{X}}\{{\Vert f_1(\mathit{p})-f_1(\mathit{g})\Vert }_{\infty }<{\epsilon }_1,\dots ,{\Vert f_{k-1}(\mathit{p})-f_{k-1}(\mathit{g})\Vert }_{\infty }<{\epsilon }_{k-1},{\Vert f_k(\mathit{p})-1\Vert }_{\infty }<{\epsilon }_k\}>0.$$

Note that for 2 ≤i ≤k,

$${\Vert f_i(\mathit{p})-f_i(\mathit{g})\Vert }_{\infty }\le (i-1){\Vert p_i-g_i\Vert }_{\infty }+\sum _{l<i}{\Vert f_l(\mathit{p})-f_l(\mathit{g})\Vert }_{\infty }.$$

Thus one can get ${\epsilon }_i^{\ast }>0$, i = 1, …, k, such that

$$\begin{array}{l}\{{\Vert p_i-g_i\Vert }_{\infty }<{\epsilon }_i^{\ast },i=1,\dots ,k\}\subset \{{\Vert f_1(\mathit{p})-f_1(\mathit{g})\Vert }_{\infty }<{\epsilon }_1,\dots ,\\ {\Vert f_{k-1}(\mathit{p})-f_{k-1}(\mathit{g})\Vert }_{\infty }<{\epsilon }_{k-1},{\Vert f_k(\mathit{p})-1\Vert }_{\infty }<{\epsilon }_k\}.\end{array}$$

But since ${\mathcal{P}}_{\mathcal{X}}\{{\Vert p_i-g_i\Vert }_{\infty }<{\epsilon }_i^{\ast },i=1,\dots ,k\}={\prod }_{i=1}^k{\mathcal{P}}_{\mathcal{X}}\{{\Vert p_i-g_i\Vert }_{\infty }<{\epsilon }_i^{\ast }\}$, the result follows immediately.

Appendix B. Proof of Theorem 5.6

Fix $\{F_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}\in {\mathcal{G}}_{\mathcal{X}}^{\ast }$. Without loss of generality it is enough to show that for a uniformly continuous function g : ℜ^p×ℜ⁺× → [0, 1] and ε > 0,

$${\mathcal{P}}_{\mathcal{X}}\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\left|{\int }_{{\mathfrak{R}}^p\times {\mathfrak{R}}^{+}\times \mathcal{X}}\{g(\mathit{\beta },\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )q(\mathbf{x})d\mathbf{x}-g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )q(\mathbf{x})d\mathbf{x}\}\right|<\epsilon \right\}>0.$$

Furthermore, it suffices to assume g(β, σ, x) → 0 uniformly in x ∈ as ||β|| → ∞, σ → ∞.

Fix ε > 0, there exist a, σ, σ̄ > 0 not depending on x such that F_x([−a, a]^p×[σ, σ̄]) > 1 − ε for all x ∈ . Let C = [−a, a]^p × [σ, σ̄].

$$\begin{array}{l}{\int }_{{\mathfrak{R}}^p\times {\mathfrak{R}}^{+}\mathcal{X}}\{g(\mathit{\beta },\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )-g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\}q(\mathbf{x})d\mathbf{x}\le \\ {\int }_{\mathcal{X}}\left\{\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x})g({\mathit{\beta }}_h,{\sigma }_h,\mathbf{x})-{\int }_{C}g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\right\}q(\mathbf{x})d\mathbf{x}+\epsilon .\end{array}$$

where π_h’s are specified by 5.1 with c_h satisfying S1 and S2 and (β_h, σ_h) ~ G₀. Now for each x ∈ , construct a Riemann sum approximation of ∫_C g(β, σ, x)dF_x(β, σ).

Let {A_k,n, k = 1, …, n} be a sequence of partitions of C with increasing refinement as n increases. Assume max_1≤k≤n diam(A_k,n) → 0 as n ↑ ∞. Fix (β̃_k,n, σ̃_k,n) ∈ A_k,n, k = 1, …, n. Then by DCT as n → ∞,

$${\int }_{\mathcal{X}}\{\sum _{k=1}^{n}g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n},\mathbf{x})F_{\mathbf{x}}(A_{k,n})\}q(\mathbf{x})d\mathbf{x}\to {\int }_{\mathcal{X}}{\int }_{C}g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )q(\mathbf{x})d\mathbf{x}.$$ (B.1)

Hence there exists n₁ such that for n ≥ n₁

$$\begin{array}{l}\left|{\int }_{{\mathfrak{R}}^p\times {\mathfrak{R}}^{+}\mathcal{X}}\{g(\mathit{\beta },\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )-g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\}q(\mathbf{x})d\mathbf{x}\right|\le \\ \left|{\int }_{\mathcal{X}}\left\{\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x})g({\mathit{\beta }}_h,{\sigma }_h,\mathbf{x})-\sum _{k=1}^{n}g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n},\mathbf{x})F_{\mathbf{x}}(A_{k,n})\right\}q(\mathbf{x})d\mathbf{x}\right|+2\epsilon .\end{array}$$

Consider the set

$${\mathrm{\Omega }}_1=\left\{({\pi }_h,h=1,\dots ,\infty ):\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\pi }_1(\mathbf{x})-F_{\mathbf{x}}(A_{1,n_1})\mid <\frac{\epsilon }{n_1},\dots ,\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\pi }_{n_1}(\mathbf{x})-F_{\mathbf{x}}(A_{n_1,n_1})\mid <\frac{\epsilon }{n_1}\right\}.$$

By Lemma Appendix A.1 which is proved in Appendix A, (Ω₁) > 0. Since ${\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x})=1$ a.s. there ∃ Ω with (Ω) = 1, such that for each ω= {π_h, h = 1, …, ∞} ∈ Ω, $g_n(\mathbf{x})={\sum }_{h=1}^n{\pi }_h(\mathbf{x})\to 1$ as n → ∞ for each x in . Note that this convergence is uniform since, g_n(·), n ≥ 1 are continuous functions defined on a compact set monotonically increasing to a continuous function identically equal to 1. Hence for each ω = {π_h, h = 1, …, ∞} ∈ Ω, g_n(x) → 1 uniformly in x. By Egoroff’s theorem, there exists a measurable subset Ω₂ of Ω₁ with (Ω₂) > 0 such that within this subset g_n(x) → 1 uniformly in x and uniformly in ω in Ω₂. Thus there exists a positive integer n_ε ≥ n₁ not depending on x and ω, such that ${\sum }_{h=n_{\epsilon }+1}^{\infty }{\pi }_h(\mathbf{x})<\epsilon$ on Ω₂. Moreover, one can find a K > 0 independent of x such that g(β, σ, x) < ε if ||β|| > K and σ > K. Let A₁ = {(β, σ) : ||β|| > K, σ > K}. Let Ω₃ = Ω₂ ∩ {(β_n1+1, σ_n1+1) ∈ A₁, …, (β_nε−1, σ_nε−1) ∈ A₁}. For ω ∈ Ω₃,

$$\begin{array}{l}\left|{\int }_{R^p\times R^{+}\mathcal{X}}\{g(\mathit{\beta },\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )-g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\}q(\mathbf{x})d\mathbf{x}\right|\le \\ {\int }_{\mathcal{X}}\left\{\sum _{k=1}^{n_1}\left|{\pi }_k(\mathbf{x})g({\mathit{\beta }}_k,{\sigma }_k,\mathbf{x})-g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n},\mathbf{x})F_{\mathbf{x}}(A_{k,n_1})\right|\right\}q(\mathbf{x})d\mathbf{x}+4\epsilon \end{array}$$

and

$$\begin{array}{l}{\int }_{\mathcal{X}}\left\{\sum _{k=1}^{n_1}\left|{\pi }_k(\mathbf{x})g({\mathit{\beta }}_k,{\sigma }_k,\mathbf{x})-g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n},\mathbf{x})F_{\mathbf{x}}(A_{k,n_1})\right|\right\}q(\mathbf{x})d\mathbf{x}\\ \le \sum _{k=1}^{n_1}{\int }_{\mathcal{X}}{\pi }_k(\mathbf{x})\left|g({\mathit{\beta }}_k,{\sigma }_k,\mathbf{x})-g({\stackrel{\sim }{\mathit{\beta }}}_{k,n},{\stackrel{\sim }{\sigma }}_{k,n},\mathbf{x})\right|q(\mathbf{x})d\mathbf{x}+\epsilon .\end{array}$$

There exists sets B_k, k = 1, …, n₁ depending on n₁ but independent of x such that if (β_k, σ_k) ∈ B_k,|g(β_k, σ_k, x) − g(β̃_k,n1, σ̃_k,n1, x)|< ε. So for ω ∈ Ω₄ = Omega;₃ ∩ {(β₁, σ₁) ∈ B₁, …, (β_n1, σ_n1) ∈ B_n1},

$$\left|{\int }_{{\mathfrak{R}}^p\times {\mathfrak{R}}^{+}\mathcal{X}}\{g(\mathit{\beta },\sigma ,\mathbf{x}){dG}_{\mathbf{x}}(\mathit{\beta },\sigma )-g(\mathit{\beta },\sigma ,\mathbf{x}){dF}_{\mathbf{x}}(\mathit{\beta },\sigma )\}q(\mathbf{x})d\mathbf{x}\right|<5\epsilon .$$

Now since (Ω₂) > 0 and the sets {(β_n1+1, σ_n1+1) ∈ A₁, …, (β_nε−1, σ_nε−1) ∈ A₁} and {(β₁, σ₁) ∈ B₁, …, (β_n1, σ_n1) ∈ B_n1} are independent from Ω₂ and have positive probability, it follows that (Ω₄) > 0.

Appendix C. Proof of Theorem 5.3

Without loss of generality, assume that the covariate space is [ζ, 1]^p for some 0 < ζ < 1. The proof is essentially along the lines of Theorem 3.2 of [12]. The f̃ in (5.2) will be constructed so as to satisfy the assumptions of Lemma 5.5 and such that ${\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{f_0(y\mid \mathbf{x})}{f(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}<\frac{\epsilon }{2}$ for any ε > 0. Define a sequence of conditional densities $f_n(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi (\frac{y-{\mathbf{x}}^{\prime }\mathit{\beta }}{\sigma })d{\stackrel{\sim }{G}}_{n,\mathbf{x}}(\mathit{\beta },\sigma )$, n ≥ 1 where for σ_n = n^−η,

$${dG}_{n,\mathbf{x}}(\mathit{\beta },\sigma )=\frac{I_{{\beta }_1\in [-n,n]}{f}_0({\mathbf{x}}^{\prime }\beta \mid \mathbf{x}){\prod }_{j=2}^p{\delta }_0({\beta }_j){\delta }_{{\sigma }_n}(\sigma )}{{\int }_{-n}^n{f}_0(x_1{\beta }_1\mid \mathbf{x})d{\beta }_1}.$$ (C.1)

Define

$$f_n(y\mid \mathbf{x})=\frac{{\int }_{-{nx}_1}^{{nx}_1}\frac{1}{{\sigma }_n}\varphi \left(\frac{y-t}{{\sigma }_n}\right)f_0(t\mid \mathbf{x})dt}{{\int }_{-{nx}_1}^{{nx}_1}{f}_0(t\mid \mathbf{x})dt}.$$ (C.2)

Proceeding as in Theorem 3.2 of [12], an application of DCT using the conditions A1–A5 yields

$${\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{f_0(y\mid \mathbf{x})}{f_n(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}\to 0\text{as}n\to \infty .$$

Therefore one can simply choose f̃ = f_n0 for sufficiently large n₀. f_n0 satisfies the assumptions of Lemma 5.5 since {G_n0,x, x ∈ } is compactly supported. Also $\{G_{n_0,\mathbf{x}},\mathbf{x}\in \mathcal{X}\}\in {\mathcal{G}}_{\mathcal{X}}^{\ast }$ as x → G_n0,_x(A) is continuous. Hence there exists a finite intersection W of neighborhoods of {G_n0,x, x ∈ } the type (5.5) such that for any {G_x, x ∈ } ∈ W, the second term of (5.2) is arbitrarily small. The conclusion of the theorem follows immediately from Corollary 5.8.

Appendix D. Proof of Theorem 5.10

Consider the sequence of sieves defined by (5.9) for given ε > 0 and for sequences a_n, h_n, l_n, M_n, m_n, r_n to be chosen later with ${\delta }_n=K_1\epsilon /(M_n{m}_n^2)$ for some constant K₁. We will first show that given ξ > 0, there exist c₁, c₂ > 0 and sequences m_n and M_n, such that ${\mathrm{\Pi }}_{\mathcal{X}}({\mathcal{F}}_n^c)\le c_1{e}^{-{nc}_2}$ and log N (δ, , ||·||) < nξ.

For f₁, f₂ ∈ , we have for each x ∈ ,

$${\Vert f_1(·\mid \mathbf{x})-f_2(·\mid \mathbf{x})\Vert }_1\le \sum _{h=1}^{m_n}{\Vert {\pi }_h^{(1)}-{\pi }_h^{(2)}\Vert }_{\infty }+2\epsilon .$$

Let Θ_π,n = {π^m_n = (π₁, π₂, …, π_mn) : α_h ∈ B_h,n, h = 1, …, m_n}. Fix ${\pi }_1^{m_n},{\pi }_2^{m_n}\in {\mathrm{\Theta }}_{\pi ,n}$. Note that since |Φ(x₁) − Φ(x₂)| < K₂ |x₁ − x₂| for a global constant K₂ > 0, we have

$${\Vert \mathrm{\Phi }({\alpha }_{h,1})-\mathrm{\Phi }({\alpha }_{h,2})\Vert }_{\infty }\le K_2{\Vert {\alpha }_{h,1}-{\alpha }_{h,2}\Vert }_{\infty }.$$

The above fact together with the proof of Lemma Appendix A.1 show that if we can make ${\Vert {\alpha }_{h,1}-{\alpha }_{h,2}\Vert }_{\infty }<\frac{\epsilon }{m_n^2}$, h = 1, …, m_n, we would have ${\sum }_{h=1}^{m_n}{\Vert {\pi }_h^{(1)}-{\pi }_h^{(2)}\Vert }_{\infty }<\epsilon$. From the proof of Theorem 3.1 in [42] it follows that for $h=1,\dots ,m_n^{\eta }$ and for sufficiently large M_n, r_n,

$$logN(2\epsilon /m_n^2,B_{h,n},{\Vert ·\Vert }_{\infty })\le K_3{r}_n^{p}log{\left(\frac{M_n{m}_n^2\sqrt{r_n/{\delta }_n}}{\epsilon }\right)}^{p+1}+2log\frac{K_4{M}_n{m}_n^2}{\epsilon }.$$ (D.1)

for global constants K₃, K₄ > 0. For $M_n^2>16K_5{r}_n^p{(log(r_n/\epsilon ))}^{1+p}$, r_n > 1 we have for $h=1,\dots ,m_n^{\eta }$,

$$P({\alpha }_h\notin B_{h,n})\le P(A_h>r_n)+e^{-M_n^2/2}.$$ (D.2)

Hence for sufficiently large M_n, we have for $h=m_n^{\eta }+1,\dots ,m_n$,

$$logN(3\epsilon /m_n^2,B_{h,n},{\Vert ·\Vert }_{\infty })\le 2log\frac{K_4{M}_n{m}_n^2}{\epsilon }.$$ (D.3)

For $h=m_n^{\eta }+1,\dots ,m_n$,

$$\begin{array}{l}P({\alpha }_h\notin B_{h,n})\le P(A_h>{\delta }_n)+{\int }_{a=0}^{{\delta }_n}P({\alpha }_h\notin B_{h,n}\mid A_h=a)g_{A_h}(a)da\\ \le P(A_h>{\delta }_n)+(1-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(e^{-{\varphi }_0^{{\delta }_n}(\epsilon /m_n^2)})+M_n)).\end{array}$$

where ${\varphi }_0^{\kappa }(\epsilon )$ denotes the concentration function of the Gaussian process with covariance kernel c(x, x′) = τ²e^{−κ||x−x′||2}. Now

$${\varphi }_0^{{\delta }_n}(\epsilon /m_n^2)\le -logP(\mid W_0\mid \le \epsilon /m_n^2)=K_6\mid log(\epsilon /m_n^2)\mid$$

for some constant K₆ > 0. Hence if $M_n\ge K_7\mid log(\epsilon /m_n^2)\mid$ for some K₇ > 0, then it follows from the proof of Theorem 3.1 in [42] that

$$P({\alpha }_h\notin B_{h,n})\le P(A_h>{\delta }_n)+e^{-M_n^2/2}.$$ (D.4)

From (D.1) and (D.3),

$$log(N(\epsilon ,B_{1,n}\times \cdots \times B_{m_n,n},{\Vert ·\Vert }_{\infty })\le 2m_{n}log\frac{K_4{M}_n{m}_n^2}{\epsilon }+m_n^{\eta }{r}_n^{p}log{\left(\frac{M_n{m}_n^2\sqrt{r_n/{\delta }_n}}{\epsilon }\right)}^{p+1}.$$ (D.5)

Also from (D.2) and (D.4),

$$\sum _{h=1}^{m_n}P({\alpha }_h\notin B_{h,n})\le m_n{e}^{-M_n^2/2}+\sum _{h=1}^{m_n^{\eta }}P(A_h>r_n)+\sum _{h=m_n^{\eta }+1}^{m_n}P(A_h>{\delta }_n).$$

We will show that with $m_n=O(\frac{n}{logn}),{\mathrm{\Pi }}_{\mathcal{X}}({\mathcal{F}}_n^c)<e^{-n{\xi }_0}$ for some ξ₀ assumption C1, we have

$${\mathrm{\Pi }}_{\mathcal{X}}({\mathrm{\Theta }}_{a_n,h_n,l_n}^c)\precsim m_{n}O(e^{-n})\precsim O(e^{-n}).$$ (D.6)

With m_n = O(n/log n), ${\sum }_{h=1}^{m_n^{\eta }}P(A_h>r_n)\le m_n^{\eta }{e}^{-n}\precsim e^{-n},{\sum }_{h=m_n^{\eta }+1}^{m_n}P(A_h>{\delta }_n)\le (m_n-m_n^{\eta })e^{-n^{-{\eta }_0}{m}_n^{{\eta }_0+2}log{m}_n}\precsim e^{-m_{n}log{m}_n}$.

With $m_n=\frac{n}{logn},m_{n}log{m}_n>\frac{n}{2}$ for large enough n and it follows from Lemma 5.15 that

$${\mathrm{\Pi }}_{\mathcal{X}}\left(\underset{\mathbf{x}\in \mathcal{X}}{sup}\sum _{h=m_n+1}^{\infty }{\pi }_h(\mathbf{x})>\epsilon \right)\precsim O(e^{-n/2}).$$ (D.7)

Thus with M_n = O(n^1/2),

$$\sum _{h=1}^{m_n}P({\alpha }_h\notin B_{h,n})\precsim e^{-n}.$$ (D.8)

(D.6), (D.7) and (D.8) together imply that ${\mathrm{\Pi }}_{\mathcal{X}}({\mathcal{F}}_n^c)\precsim O(e^{-n})$.

Also $m_n^{\eta }{r}_n^{p}log{\left(\frac{M_n\sqrt{r_n/{\delta }_n}}{\epsilon }\right)}^{p+1}=o(n)$ for the choice of the sequence r_n. With m_n = n/(C log n) for some large C > 0, one can make

$$log(N(\epsilon ,B_{1,n}\times \cdots \times B_{m_n,n},{\Vert ·\Vert }_{\infty })<n\xi$$ (D.9)

for any ξ > 0. Also from Lemma 5.14,

$$m_{n}logN({\mathrm{\Theta }}_{a_n,h_n,l_n},\epsilon ,{\Vert ·\Vert }_{\infty })\le m_{n}log\left\{d_1{\left(\frac{a_n}{l_n}\right)}^p+d_{2}log\frac{h_n}{l_n}+1\right\}<n\xi$$ (D.10)

for any ξ > 0. Combining (D.9) and (D.10), log N ( , 4ε, ||·||₁) < nξ for any ξ > 0.

Appendix E. Another useful lemma

We state without proof the following Lemma needed to prove Theorem 6.1.

Lemma Appendix E.1

For non-negative r.v.s A_i, B_i, if P(A_i ≤ u) ≤ C_i P(B_i ≤ u) for u ∈ (0, t₀), t₀ > 0, i = 1, 2, P(A₁ + A₂ ≤ t₀) ≤ C₁C₂P(B₁ + B₂ ≤ t₀).

Appendix F. Proof of Theorem 6.1

Proof

Once again we approximate f₀(y|x) by $\stackrel{\sim }{f}(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi (\frac{y-\mu }{\sigma })d{\stackrel{\sim }{G}}_{\mathbf{x}}(\mu ,\sigma )$, so that the first term of 5.2 is arbitrarily small. We construct such an f̃ analogous to that in Theorem 5.3. Lemma Appendix F.1 is a variant of Lemma 5.5 which ensures that the second term in (5.2) is also sufficiently small. Before that we need a different notion of neighborhood of {F_x, x ∈ } which we formulate below.

$$\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\underset{\mathbf{x}\in \mathcal{X}}{sup}\left|{\int }_{\mathfrak{R}\times {\mathfrak{R}}^{+}}\{g(\mu ,\sigma ){dG}_{\mathbf{x}}(\mu ,\sigma )-g(\mu ,\sigma ){dF}_{\mathbf{x}}(\mu ,\sigma )\}\right|<\epsilon \right\}.$$ (F.1)

Lemma Appendix F.1

Assume that f₀ ∈ satisfies y²f₀(y | x)dyq(x)dx < ∞. Suppose $\stackrel{\sim }{f}(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi (\frac{y-\mu }{\sigma })d{\stackrel{\sim }{G}}_{\mathbf{x}}(\mu ,\sigma )$, where ∃ a > 0 and 0 < σ < σ̄ such that

$${\stackrel{\sim }{G}}_{\mathbf{x}}([-a,a]\times (\underset{\_}{\sigma },\overline{\sigma }))=1\forall \mathbf{x}\in \mathcal{X},$$ (F.2)

so that G̃_x has compact support for each x ∈ . Then given any ε > 0, ∃ a neighborhood W of {G̃_x, x ∈ } which is a finite intersection of neighborhoods of the type (F.1) such that for any conditional density $f(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi (\frac{y-\mu }{\sigma }){dG}_{\mathbf{x}}(\mu ,\sigma )$, x ∈ , with {G_x, x ∈ } ∈ W,

$${\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{\stackrel{\sim }{f}(y\mid \mathbf{x})}{f(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}<\epsilon .$$ (F.3)

The proof of Lemma Appendix F.1 is similar to that of Lemma 5.5 and is omitted here. To characterize the support of , we define a collection of fixed conditional probability measures {F_x, x ∈ } on (ℜ × ℜ⁺, (ℜ × ℜ⁺)) denoted by ${\mathcal{G}}_{\mathcal{X}}^{\ast \ast }$ satisfying x ↦ ∫_ℜ×ℜ⁺ g(μ, σ)dF_x(μ) is a continuous function of x for all bounded uniformly continuous functions g : ℜ × ℜ⁺ → [0, 1].

Theorem Appendix F.2

Assume the following holds.

• T1G₀ is specified by μ_h ~ GP(μ, c), σ_h ~ G_0,σ where c is chosen so that GP(0, c) has continuous path realizations and Π_σ is absolutely continuous w.r.t. Lebesgue measure on ℜ⁺.
• T2For every k ≥ 2, (π₁, …, π_k) is absolutely continuous w.r.t. to the Lebesgue measure on S_k−1.
• T3
  For any continuous function g : ↦ ℜ,

  $${\mathcal{P}}_{\mathcal{X}}\left\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\mu }_h(\mathbf{x})-g(\mathbf{x})\mid <\epsilon \right\}>0$$

  h = 1, …, ∞ and for any ε > 0.

Then for a bounded uniformly continuous function g : ℜ × ℜ⁺ : [0, 1] satisfying g(μ, σ) → 0 as |μ| → ∞, σ → ∞,

$${\mathcal{P}}_{\mathcal{X}}\left\{\{G_{\mathbf{x}},\mathbf{x}\in \mathcal{X}\}:\underset{\mathbf{x}\in \mathcal{X}}{sup}\left|{\int }_{\mathfrak{R}\times {\mathfrak{R}}^{+}}\{g(\mu ,\sigma ){dG}_{\mathbf{x}}(\mu ,\sigma )-g(\mu ,\sigma ){dF}_{\mathbf{x}}(\mu ,\sigma )\}\right|<\epsilon \right\}>0.$$ (F.4)

Proof

It suffices to assume that g is coordinatewise monotonically increasing on ℜ × ℜ⁺. Let ε > 0 be given and ψ(x) = ∫_ℜ×ℜ⁺ g(μ, σ)dF_x(μ, σ). Let n_ε be such that (Ω₁) > 0 where ${\mathrm{\Omega }}_1=\{{\sum }_{h=n_{\epsilon }+1}^{\infty }{\pi }_h<\epsilon \}$. Then in Ω₁,

$$\left|{\int }_{\mathfrak{R}\times {\mathfrak{R}}^{+}}\{g(\mu ,\sigma ){dG}_{\mathbf{x}}(\mu ,\sigma )-\psi (\mathbf{x})\}\right|\le \sum _{k=1}^{n_{\epsilon }}{\pi }_k\mid g({\mu }_k(\mathbf{x}),{\sigma }_k)-\psi (x)\mid +\epsilon .$$

Define Ω₂ = { |g(μ_k(x), σ_k) − ψ(x)|< ε, k = 1, …, n_ε}. For a fixed σ_k, there exists a δ such that |g(μ_k(x), σ_k) − ψ(x)|< ε/2 if ${sup}_{\mathbf{x}\in \mathcal{X}}\mid {\mu }_k(\mathbf{x})-g_{{\sigma }_k}^{-1}\psi (\mathbf{x})\mid <\delta$ where $g_{{\sigma }_k}^{-1}$ denotes the inverse of g(·, σ_k) for fixed σ_k. Hence there exists a neighborhood B_k of σ_k such that for σ_k ∈ B_k and ${sup}_{\mathbf{x}\in \mathcal{X}}\mid {\mu }_k(\mathbf{x})-g_{{\sigma }_k}^{-1}\psi (\mathbf{x})\mid <\delta$, we have |g(μ_k(x), σ_k) − ψ(x)| < ε. Since for each k = 1, …, n_ε, ${\mathcal{P}}_{\mathcal{X}}\{{\sigma }_k\in B_k,{sup}_{\mathbf{x}\in \mathcal{X}}\mid {\mu }_k(\mathbf{x})-g_{{\sigma }_k}^{-1}\psi (\mathbf{x})\mid <\delta \}=$

$${\int }_{{\sigma }_k\in B_k}{\mathcal{P}}_{\mathcal{X}}\{\underset{\mathbf{x}\in \mathcal{X}}{sup}\mid {\mu }_k(\mathbf{x})-g_{{\sigma }_k}^{-1}\psi (\mathbf{x})\mid <\delta \}{dG}_{0,\sigma }({\sigma }_k)>0,$$

(Ω₂) > 0. The conclusion of the theorem follows from the independence of Ω₁ and Ω₂.

f̃ in (5.2) will be constructed so as to satisfy the assumptions of Lemma Appendix F.1 and such that ${\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}{f}_0(y\mid \mathbf{x})log\frac{f_0(y\mid \mathbf{x})}{f(y\mid \mathbf{x})}\mathit{dyq}(\mathbf{x})d\mathbf{x}<\frac{\epsilon }{2}$ for any ε > 0. Define a sequence of conditional densities $f_n(y\mid \mathbf{x})=\int \frac{1}{\sigma }\varphi (\frac{y-\mu }{\sigma })d{\stackrel{\sim }{G}}_{n,\mathbf{x}}(\mu ,\sigma )$, n ≥ 1 where for σ_n = n^−η,

$${dG}_{n,\mathbf{x}}(\mu ,\sigma )=\frac{I_{\mu \in [-n,n]}{f}_0(\mu \mid \mathbf{x}){\delta }_{{\sigma }_n}(\sigma )}{{\int }_{-n}^n{f}_0(\mu \mid \mathbf{x})}.$$ (F.5)

As before define the approximator

$$f_n(y\mid \mathbf{x})=\frac{{\int }_{-n}^n\frac{1}{{\sigma }_n}\varphi (\frac{y-t}{{\sigma }_n})f_0(t\mid \mathbf{x})dt}{{\int }_{-n}^n{f}_0(t\mid \mathbf{x})dt}.$$ (F.6)

f̃ will be chosen to be f_n0 for some large n₀. f_n0 satisfies the assumptions of Lemma Appendix F.1 since {G_n0,_x, x ∈ } is compactly supported. Moreover $\{G_{n_0,\mathbf{x}},\mathbf{x}\in \mathcal{X}\}\in {\mathcal{G}}_{\mathcal{X}}^{\ast \ast }$ as x → ∫_ℜ×ℜ⁺ g(μ, σ)dG_n0,x(μ, σ) is continuous function of x for all bounded uniformly continuous function g. Hence there exists a finite intersection W of neighborhoods of {G_n0x, x ∈ } the type (F.1) such that for any {G_x, x ∈ } ∈ W, the second term of (5.2) is arbitrarily small. The conclusion of the theorem follows immediately from a variant of Corollary 5.8 applied to neighborhoods of the type (F.1).

Appendix G. Proof of Theorem 6.2

Proof

As before we establish q-integrated L₁ consistency of Gaussian mixtures of fixed-π dependent processes by verifying the conditions of Theorem 5.9. Let ${\varphi }_{\mu ,\sigma }(\mathbf{x},y):=\frac{1}{\sigma }\varphi (\frac{y-\mu (\mathbf{x})}{\sigma })$ for y ∈ and x ∈ . Construct B_n as

$$B_n=\left(M_n\sqrt{\frac{r_n}{{\delta }_n}}{ℍ}_1^{r_n}+\frac{\epsilon l_n\sqrt{\pi }}{4\sqrt{2}}{\mathbb{B}}_1\right)\cup \left({\cup }_{a<{\delta }_n}{M}_n{ℍ}_1^a+\frac{\epsilon l_n\sqrt{\pi }}{4\sqrt{2}}{\mathbb{B}}_1\right).$$

with ${\delta }_n=\frac{K_1\epsilon l_n}{M_n}$ for some constant K₁ > 0. Let

$${\mathrm{\Theta }}_n=\{{\varphi }_{\mu ,\sigma }:\Vert \mathit{\beta }\Vert \le a_n,\eta \in B_n,l_n\le \sigma \le h_n\}.$$ (G.1)

It is easy to see that

$$\begin{array}{l}logN({\mathcal{F}}_n,4\epsilon ,\Vert ·\Vert )\le K_2{m}_n{r}_n^p{\left\{log\left(\frac{8\sqrt{2}{M}_n\sqrt{r_n/{\delta }_n}}{\epsilon \sqrt{\pi }{l}_n}\right)\right\}}^{p+1}+m_{n}log\frac{K_4{M}_n^2}{\epsilon }\\ m_{n}log\frac{K_3{M}_n}{\epsilon l_n}+m_{n}log\left\{d_1{\left(\frac{a_n}{l_n}\right)}^p+d_{2}log\frac{h_n}{l_n}+1\right\}.\end{array}$$ (G.2)

Note that ${\mathrm{\Pi }}_{\mathcal{X}}({\mathcal{F}}_n^c)\le m_{n}P({\mathrm{\Theta }}_n^c)+P({\sum }_{h=m_n}^{\infty }{\pi }_h>\epsilon )$ and $P({\mathrm{\Theta }}_n^c)\le \{P(\Vert \mathit{\beta }\Vert \ge a_n)+P(\sigma \in {[l_n,h_n]}^c)+P(\eta \in B_n^c)\}$. It follows from the proof of Theorem 3.1 of [42] that

$$P(\eta \in B_n^c)\le P(A>r_n)+e^{-M_n^2/2}$$

if $M_n^2>r_n^p\{log\left(\frac{8\sqrt{2}{M}_n\sqrt{r_n/{\delta }_n}}{\epsilon \sqrt{\pi }{l}_n}\right)$. Since A^{p(1+η₂)/η₂} ~ Ga(a, b), Lemma 4.9 of [42] indicates that $P(A>r_n)\precsim exp\{-r_n^{p(1+{\eta }_2)/{\eta }_2}\}$. Hence with M_n = O(n^1/2), m_n = O{n/(log n)^p+1}^1/(1+η₂) and $r_n^p=O\{n^{{\eta }_2/(1+{\eta }_2)}\}P({\mathrm{\Theta }}_n^c)\precsim e^{-n}$ and

$$P(\sum _{h=m_n}^{\infty }{\pi }_h>\epsilon )\precsim exp\{-m_n^{1+{\eta }_2}{(log{m}_n)}^{(p+1)}\}\precsim e^{-n}.$$ (G.3)

Also, the first term in the right hand side of (G.2) can be made smaller than nξ since $m_n{r}_n^p=O(n/{(logn)}^{p+1})$. Also by F1, the last two terms of the right hand side of (G.2) can be made to grow at o(n).