Bayesian nonparametric regression with varying residual density

Abstract

We consider the problem of robust Bayesian inference on the mean regression function allowing the residual density to change flexibly with predictors. The proposed class of models is based on a Gaussian process prior for the mean regression function and mixtures of Gaussians for the collection of residual densities indexed by predictors. Initially considering the homoscedastic case, we propose priors for the residual density based on probit stick-breaking (PSB) scale mixtures and symmetrized PSB (sPSB) location-scale mixtures. Both priors restrict the residual density to be symmetric about zero, with the sPSB prior more flexible in allowing multimodal densities. We provide sufficient conditions to ensure strong posterior consistency in estimating the regression function under the sPSB prior, generalizing existing theory focused on parametric residual distributions. The PSB and sPSB priors are generalized to allow residual densities to change nonparametrically with predictors through incorporating Gaussian processes in the stick-breaking components. This leads to a robust Bayesian regression procedure that automatically down-weights outliers and influential observations in a locally-adaptive manner. Posterior computation relies on an efficient data augmentation exact block Gibbs sampler. The methods are illustrated using simulated and real data applications.

1 Introduction

Nonparametric regression offers a more flexible way of modeling the effect of covariates on the response compared to parametric models having restrictive assumptions on the mean function and the residual distribution. Here we consider a fully Bayesian approach. The response y ∈ 𝒴 corresponding to a set of covariates x = (x₁, x₂, …, x_p)′ ∈ 𝒳 can be expressed as

$$y=\eta (\mathbf{x})+\epsilon$$ (1)

where η(x) = 𝖤(y | x) is the mean regression function under the assumption that the residual density has mean zero, i.e., 𝖤(ε | x) = 0 for all x ∈ 𝒳. Our focus is on obtaining a robust estimate of η while allowing heavy tails to down-weight influential observations. We propose a class of models that allows the residual density to change nonparametrically with predictors x, with homoscedasticity arising as a special case.

There is a substantial literature proposing priors for flexible estimation of the mean function, typically using basis function representations such as splines or wavelets (Denison et al 2002). Most of this literature assumes a constant residual density, possibly up to a scale factor allowing heteroscedasticity. Yau and Kohn (2003) allow the mean and variance to change with predictors using thin plate splines. In certain applications, this structure may be overly restrictive due to the specific splines used and the normality assumption. Chan et al (2006) also used splines for heteroscedastic regression, but with locally adaptive estimation of the residual variance and allowance for uncertainty in variable selection. Nott (2006) considered the problem of simultaneous estimation of the mean and variance function by using penalized splines for possibly non Gaussian data. Due to the lack of conjugacy, these methods rely on involved sampling techniques using Metropolis Hastings, requiring proposal distributions to be chosen that may not be efficient in all cases. The residual density is assumed to have a known parametric form and heavy-tailed distributions have not been considered. In addition, since basis function selection for multiple predictors is highly computationally demanding, additive assumptions are typically made that rule out interactions.

Gaussian process (GP) regression (Adler 1990; Ghosal and Roy 2006; van der Vaart and van Zanten 2008, 2009; Neal 1998) is an increasingly popular choice, which avoids the need to explicitly choose the basis functions, while having many appealing computational and theoretical properties. For articles describing some of these properties, refer to Adler (1990), Cramér and Leadbetter (1967), van der Vaart and van Zanten (2008) and van der Vaart and Wellner (1996). A wide variety of functions can arise as the sample paths of the Gaussian process. GP priors can be chosen that have support on the space of all smooth functions while facilitating Bayes computation through conjugacy properties. In particular, the GP realizations at the data points are simply multivariate Gaussian. As shown by Choi and Schervish (2007), GP priors also lead to consistent estimation of the regression function under normality assumptions on the residuals. van der Vaart and van Zanten (2009) demonstrated that a Gaussian process prior with an inverse-gamma bandwidth leads to an optimal rate of posterior convergence in a mean regression problem with Gaussian errors. Recently, Choi (2009) extended the results of Choi and Schervish (2007) to allow for non-Gaussian symmetric residual distributions (for example, the Laplace distribution) which satisfy certain regularity conditions and the induced conditional density belongs to a location-scale family. Although they require mild assumptions on the parametric scale family, the results depend heavily on parametric assumptions. In particular, their theory of posterior consistency is not applicable to an infinite mixture prior on the residual density. We extend their result allowing a rich class of residual distributions through PSB mixtures of Gaussians in Section 3.

There is a rich literature on Bayesian methods for density estimation using mixture models of the form

$$y_i~f({\theta }_i),{\theta }_i~P,P~\Pi ,$$ (2)

where f(·) is a parametric density and P is an unknown mixing distribution assigned a prior Π. The most common choice of Π is the Dirichlet process (Ferguson 1973), 1974). Lo (1984) showed that Dirichlet process mixtures of normals have dense support on the space of densities with respect to Lesbesgue measure, while Escobar and West (1995) developed methods for posterior computation and inference. James et al (2005) considered a broader class of normalized random measures for Π.

In order to combine methods for Bayesian nonparametric regression with methods for Bayesian density estimation, one can potentially use mixture model (2) for the residual density in (1). A number of authors have considered nonparametric priors for the residual distribution in regression. For example, Kottas and Gelfand (2001) proposed mixture models for the error distributions in median regression models. To ensure identifiability of the regression coefficients, the residual distribution is constrained to have median zero. Their approach is very flexible but has the unappealing property of producing a residual density that is discontinuous at zero. In addition, the approach of mixing uniforms leads to blocky looking estimates of the residual density particularly for sparse data. Lavine and Mockus (2005) allow both a regression function for a single predictor and the residual distribution to be unknown subject to a monotonicity constraint. A number of recent papers have focused on generalizing model (2) to the density regression setting in which the entire conditional distribution of y given x changes flexibly with predictors. Refer, for example, to Müller et al (1996); Griffin and Steel (2006, 2008); Dunson et al (2007) and Dunson and Park (2008b) among others. Bush and MacEachern (1996) consider estimating the random block effects non parametrically in an anova-type mean linear-regression model with a t-residual density rather than density regression.

Although these approaches are clearly highly flexible, there are several issues that provide motivation for this article. First, to simplify inferences and prior elicitation, it is appealing to separate the mean function η(x) from the residual distribution in the specification, which is accomplished by only a few density regression methods. The general framework of separately modeling the mean function and residual distribution nonparametrically was introduced by Griffin and Steel (2008). They allow the residual distribution to change flexibly with predictors using the order-based Dirichlet process (Griffin and Steel 2006). On the other hand, we want to able to have a computationally simpler specification with straightforward prior elicitation. Chib and Greenberg (2010) develops a nonparametric model jointly for continuous and categorical responses where they model the mean of the link function and residual density separately. The mean is modeled using flexible additive splines and the residual density is modeled using a DP scale mixture of normals. However they didn’t allow the residual distribution to change flexibly with the predictors. Often we have strong prior information regarding the form of the regression function. Most density regression models do not allow incorporation of prior information on the mean function separately from the residual densities. Second, in many applications, the main interest is in inference on η or in prediction, and the residual distribution can be considered as a nuisance. Third, the use of residual distribution with zero mean has rarely been attempted in the nonparametric Bayes literature. This is one of the important contributions of the paper. Finally, we would like to be able to provide a specification with theoretical support. In particular, it would be appealing to show strong posterior consistency in estimating η without requiring restrictive assumptions on η or the residual distribution. Current density regression models lack such theoretical support. In addition, computation for density regression can be quite involved, particularly in cases involving more than a few predictors, and one encounters the curse of dimensionality. Our goal is to obtain a computationally convenient specification that allows consistent estimation of the regression function, while being flexible in the residual distribution specification to obtain robust estimates.

We propose to place a Gaussian process prior on η and to allow the residual density to be unknown through a probit stick-breaking (PSB) process mixture. The basic PSB process specification was proposed by Chung and Dunson (2009) in developing a density regression approach that allows variable selection. On the other hand we are concerned with robust estimation of the mean regression function allowing the residual distribution to change flexibly with predictors. While we want to model the mean regression function nonparametrically, we also want to be able to incorporate our prior knowledge for the regression function quite easily. Here, we propose four novel variants of PSB mixtures for the residual distribution. The first uses a scale mixture of Gaussians to obtain a prior with large support on unimodal symmetric distributions. The next is based on a symmetrized location-scale PSB mixture, which is more flexible in avoiding the unimodality constraint, while constraining the residual density to be symmetric and have mean zero. In addition, we show that this prior leads to strong posterior consistency in estimating η under weak conditions.

To allow the residual density to change flexibly with predictors, we generalize the above priors through incorporating probit transformations of Gaussian processes in the weights. The last two prior specifications allow changing residual variances and tail heaviness with predictors, leading to a highly robust specification which is shown to have better performance in simulation studies and out of sample prediction. It will be shown in some small sample simulated examples that the heteroscedastic symmetrized location-scale PSB mixture leads to even more robust inference than the heteroscedastic scale PSB mixture without compromising out of sample predictive performance.

Section 2 proposes the class of models under consideration. Section 3 shows consistency properties. Section 4 develops efficient posterior computation through an exact block Gibbs sampler. Section 5 describes measures of influence to study robustness properties of our proposed methods. Section 6 contains simulation study results, Section 7 applies the methods to the Boston housing data and body fat data, and Section 8 discusses the results. Proofs are included in the Appendix.

2 Nonparametric regression modeling

2.1 Data Structure and Model

Consider n observations with the ith observation recorded in response to the covariate x_i = (x_i1, x_i2, … x_ip)′. Let X = (x₁, … x_n)′ be the predictor matrix for all n subjects. The regression model can be expressed as

$$y_i=\eta ({\mathbf{x}}_i)+{\epsilon }_i,{\epsilon }_i~f_{{\mathbf{x}}_i},i=1,\dots ,n.$$

We assume that the response y ∈ 𝒴 is continuous and x ∈ 𝒳 where 𝒳 ⊂ ℝ^p is compact. Also, the residuals ε_i are sampled independently, with f_x denoting the residual density specific to predictor value x_i = x. We focus initially on the case in which the covariate space 𝒳 is continuous, with the covariates arising from a fixed, non-random design or consisting of i.i.d realizations of a random variable. We choose a prior on the regression function η(x) that has support on a large subset of 𝒞^∞(𝒳), the space of smooth real valued 𝒳 → ℝ functions. The priors proposed for {f_x, x ∈ 𝒳} will be chosen to have large support so that heavy-tailed distributions and outliers will automatically be accommodated, with influential observations downweighted in estimating η.

2.2 Prior on the Mean Regression Function

We assume that η ∈ ℱ = {g : 𝒳 → ℝ is a continuous function}, with η assigned a Gaussian process (GP) prior, η ~ GP(μ, c), where μ is the mean function and c is the covariance kernel. A Gaussian process is a stochastic process {η(x) : x ∈ 𝒳} such that any finite dimensional distribution is multivariate normal, i.e., for any n and x₁, …, x_n, η(X) ≔ (η(x₁), …, η(x_n))′ ~ N(μ(X), Σ^η), where μ(X) = (μ(x₁),…, μ(x_n))′ and ${\Sigma }_{ij}^{\eta }=c({\mathbf{x}}_i,{\mathbf{x}}_j)$. Naturally the covariance kernel c(·, ·) must satisfy, for each n and x₁, …, x_n, that the matrix Σ^η is positive definite. The smoothness of the covariance kernel essentially controls the smoothness of the sample paths of {η(x) : x ∈ 𝒳}. For an appropriate choice of c, a Gaussian process has large support in the space of all smooth functions. More precisely, the support of a Gaussian process is the closure of the reproducing kernel Hilbert space generated by the covariance kernel with a shift by the mean function (Ghosal and Roy 2006). For example, when 𝒳 ⊂ ℝ, the eigenfunctions of the univariate covariance kernel, $c(x,x\prime )=\frac{1}{\tau }{e}^{-\kappa {(x-x\prime )}^2}$, span C^∞(𝒳) if κ is allowed to vary freely over ℝ⁺. Thus we can see that the Gaussian process prior has a rich class of functions as its support and hence is appealing as a prior on the mean regression function. Refer to Rasmussen and Williams (2006) as an introductory textbook on Gaussian processes.

We follow common practice in choosing the mean function in the GP prior to correspond to a linear regression, μ(X) = Xβ, with β denoting unknown regression coefficients. As a commonly used covariance kernel, we took the Gaussian kernel $c(\mathbf{x},\mathbf{x}\prime )=\frac{1}{\tau }{e}^{-\kappa {\parallel \mathbf{x}-\mathbf{x}\prime \parallel }^2}$, where τ and κ are unknown hyperparameters, with κ controlling the local smoothness of the sample paths of η(x). Smoother sample paths imply more borrowing of information from neighboring x values.

2.3 Priors for Residual Distribution

Motivated by the problem of robust estimation of the regression function η, we consider five different types of priors for the residual distributions {f_x, x ∈ 𝒳} as enumerated below. The first prior corresponds to the t distribution, which is widely used for robust modeling of residual distributions (West 1984); Lange et al 1989; Fonseca et al 2008), while the remaining priors are flexible nonparametric specifications.

P1. Heavy tailed parametric error distribution

Following many previous authors, we first consider the case in which the residual distributions follow a homoscedastic Student-t distribution with unknown degrees of freedom. As the Student-t with low degrees of freedom is heavy tailed, outliers are allowed. By placing a hyperprior on the degrees of freedom, ν_σ ~ Ga(a_ν, b_ν), with Ga(a, b) denoting the Gamma distribution with mean a/b, one can obtain a data adaptive approach to down-weighting outliers in estimating the mean regression function. However, note that this specification assumes that the same degrees of freedom and tail-heaviness holds for all x ∈ 𝒳. Following West (1987), we express the Student-t distribution as a scale mixture of normals for ease in computation. In addition, we allow an unknown scale parameter, letting ε_i ~ N(0, σ²/ϕ_i), with ϕ_i ~ Ga(ν_σ/2, ν_σ/2), σ⁻² ~ Ga(a, b).

P2. Nonparametric error distribution

Let 𝒴 = ℜ be the response space and 𝒳 be the covariate space which is a compact subset of ℜ^p. Let ℱ denote the space of densities on 𝒳 × 𝒴 w.r.t the Lebesgue measure and ℱ_d denotes the space of all conditional densities subject to mean zero,

$${ℱ}_d=\{g:𝒳\times 𝒴\to (0,\infty ),{\int }_{𝒴}g(\mathbf{x},y)dy=1,{\int }_{𝒴}yg(\mathbf{x},y)dy=0\forall \mathbf{x}\in 𝒳\}.$$

We propose to induce a prior on the space of mean zero conditional densities through a prior for a collection of mixing measures {P_x, x ∈ 𝒳} using the following predictor-dependent mixture of kernels.

$$P_{\mathbf{x}}=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x}){\delta }_{\{{\mu }_h(\mathbf{x}),{\sigma }_h\},}{\mu }_h~P_0,{\sigma }_h~P_{0,\sigma }$$ (3)

where π_h(x) ≥ 0 are random functions of x such that ${\mathrm{\Sigma }}_{h=1}^{\infty }{\pi }_h(\mathbf{x})=1$ a.s. for each fixed x ∈ 𝒳. ${\{{\mu }_h(\mathbf{x}),\mathbf{x}\in X\}}_{h=1}^{\infty }$ are iid realizations of a real valued stochastic process, i.e., P₀ is a probability distribution over a function space ℱ_𝒳. Here P_0,σ is a probability distribution on ℜ⁺. Hence for each x ∈ 𝒳, P_x is a random probability measure over the measurable Polish space (ℜ × ℜ⁺, ℬ(ℜ × ℜ⁺)). Before proposing the prior, we first review the probit stick breaking process specification and its relationship to the Dirichlet process. Rodriguez and Dunson (2011) introduce the probit stick-breaking process in broad settings and discuss some smoothness and clustering properties. A probability measure P ∈ 𝒫 on (𝒴, ℬ(𝒴)) follows a probit stick-breaking process with base measure P₀ if it has a representation of the form

$$P(\cdot )=\sum _{h=1}^{\infty }{\pi }_h{\delta }_{{\theta }_h}(\cdot ),{\theta }_h~P_0,$$ (4)

where the atoms ${\{{\theta }_h\}}_{h=1}^{\infty }$ are independent and identically distributed from P₀ and the random weights are defined as ${\pi }_h=\Phi ({\alpha }_h){\prod }_{l<h}\{1-\Phi ({\alpha }_l)\},{\alpha }_h~\mathrm{N}({\mu }_{\alpha },{\sigma }_{\alpha }^2),h=1,\dots ,\infty$. Here Φ(·) denotes the cumulative distribution function for the standard normal distribution. Note that expression (4) is identical to the stick-breaking representation (Sethuraman 1994) of the Dirichlet process (DP), but the DP is obtained by replacing the stick-breaking weight Φ(α_h) with a beta(1, α) distributed random variable. Hence, the PSB process differs from the DP in using probit transformations of Gaussian random variables instead of betas for the stick lengths, with the two specifications being identical in the special case in which μ_α = 0, σ_α = 1 and the DP precision parameter is α = 1. Rodriguez and Dunson (2011) also mentioned the possibility of constructing a variety of predictor dependent models e.g., latent Markov random fields, spatio-temporal processes, etc by using probit transformation latent Gaussian processes. Such latent Gaussian processes can be updated using data augmentation Gibbs sampling as in continuation-ratio probit models for survival analysis (Albert and Chib 2001). While we follow similar computational strategies as in Rodriguez and Dunson (2011), they didn’t consider robust regression using predictor dependent residual density.

Under the symmetric about zero assumption, we propose two nonparametric priors for the residual density f_x for all x ∈ 𝒳. The first prior is a predictor dependent PSB scale mixture of Gaussians which enforces symmetry about zero and unimodality, and the next is a symmetrized location-scale PSB mixture of Gaussians, which we develop to satisfy the symmetric about zero assumption while allowing multimodality.

P2a. Heteroscedastic scale PSB mixtures

To allow the residual density to change flexibly with predictors, while maintaining the constraint that each of the predictor-dependent residual distributions is unimodal and symmetric about zero, we propose the following specification

$$f(\cdot )=\int \mathrm{N}(\cdot ;0,{\tau }^{-1})P_{\mathbf{x}}(d\tau ),P_{\mathbf{x}}=\sum _{h=1}^{\infty }{\pi }_h(\mathbf{x}){\delta }_{{\tau }_h},{\tau }_h~\mathrm{G}\mathrm{a}({\alpha }_{\tau },{\beta }_{\tau }),$$ (5)

where π_h(x) = Φ{α_h(x)} ∏_l<h[1−Φ{α_l(x)}] is the predictor-dependent probability weight on the hth mixture component, and the α_h’s are drawn independently from zero mean Gaussian processes having covariance kernel $c_{\alpha }(\mathbf{x},\mathbf{x}\prime )=\frac{1}{{\tau }_{\alpha }}{e}^{-{\kappa }_{\alpha }{\parallel \mathbf{x}-\mathbf{x}\prime \parallel }^2}$. This implies $f_{\mathbf{x}}(\cdot )={\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x})\mathrm{N}(\cdot ;0,{\tau }_h^{-1})$ and is a highly-flexible specification that enforces smoothly changing mixture weights across the predictor space, so that the residual densities at x and x′ will tend to be similar if x is located close to x′, as measured by κ_α‖x − x′‖².

Clearly, the specification allows the residual variance to change flexibly with predictors, as we have $\text{var}(\epsilon |\mathbf{x})={\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x}){\tau }_h^{-1})$. However, unlike the previously proposed methods for heteroscedastic nonlinear regression, we do not just allow the variances to vary, but allow any aspect of the density to vary, including the heaviness of the tails. This allows locally adaptive downweighting of outliers in estimating the mean function. Previous methods, which instead assume a single heavy-tailed residual distribution, such as a t-distribution, can lead to a lack of robustness due to global estimation of a single degree of freedom parameter. In addition, due to the form of our specification, posterior computation becomes very straightforward using a data augmentation Gibbs sampler, which involves simple steps for sampling from conjugate full conditional distributions. Even under the assumption of Gaussian residual distributions, posterior computation for heteroscedastic models tends to be complex, with Gibbs sampling typically not possible due to the lack of conditional conjugacy.

P2b. Heteroscedastic symmetric PSB (sPSB) location-scale mixtures

The PSB scale mixture in (5) restricts the residual density to be unimodal. As this is a very restrictive assumption, it is appealing to define a prior with larger support that allows multimodal residual densities, while enforcing the symmetric about zero assumption so that the residual density is constrained to have mean zero. To accomplish this, we propose a novel symmetrized PSB process specification, which is related to the symmetrized Dirichlet process proposed by Tokdar (2006). We define

$$f(\cdot )=\int \mathrm{N}(\cdot ;\mu ,{\tau }^{-1})d{P}_{\mathbf{x}}^s(\mu ,\tau ),d{P}_{\mathbf{x}}^s(\mu ,\tau )=\frac{1}{2}d{P}_{\mathbf{x}}(-\mu ,\tau )+\frac{1}{2}d{P}_{\mathbf{x}}(\mu ,\tau ),$$ (6)

where the atoms (μ_h, τ_h) are drawn independently from P₀ a priori, with P₀ chosen as a product of a $\mathrm{N}({\mu }_0,{\sigma }_0^2)$ and Ga(α_τ, β_τ measure. The difference between the sPSB process prior and the PSB process prior is that instead of just placing probability weight π_h on atom (μ_h, τ_h), we place probability τ_h/2 on (−μ_h, τ_h) and (μ_h, τ_h). The resulting residual density under (6) has the form $f(\cdot )={\sum }_{h=1}^{\infty }\frac{{\pi }_h(\mathbf{x})}{2}\{\mathrm{N}(\cdot .,;-{\mu }_h,{\tau }_h^{-1})+\mathrm{N}(\cdot ;{\mu }_h,{\tau }_h^{-1})\}$. Clearly, each of the realizations corresponds to a mixture of Gaussians that is constrained to be symmetric about zero. The same comments made for the heteroscedastic scale PSB mixture apply here, but (6) is more flexible in allowing multi-modal residual distributions, with modality changing flexibly with predictors. Posterior computation is again straightforward, as will be shown later.

P2c. Homoscedastic scale PSB process mixture of Gaussians

A simpler homoscedastic version of (5) is to consider

$$f(\cdot )=\int \mathrm{N}(\cdot ;0,{\tau }^{-1})P(d\tau ),P=\sum _{h=1}^{\infty }{\pi }_h{\delta }_{{\tau }_h},{\tau }_h~\mathrm{G}\mathrm{a}({\alpha }_{\tau },{\beta }_{\tau }),$$ (7)

where the weights {π_h} are specified as in

$${\pi }_h={\nu }_h\prod _{l<h}(1-{\nu }_h),{\nu }_h=\Phi ({\alpha }_h),{\alpha }_h~\mathrm{N}({\mu }_{\alpha },{\sigma }_{\alpha }^2).$$ (8)

This implies that $f(\cdot )={\sum }_{h=1}^{\infty }{\pi }_h\mathrm{N}(\cdot ;0,{\tau }_h^{-1})$, so that the unknown density of the residuals is expressed as a countable mixture of Gaussians centered at zero but with varying variances. Observations will be automatically allocated to clusters, with outlying clusters corresponding to components having large variance (low τ_h). By choosing a hyperprior on μ_α while letting σ_α = 1, we allow the data to inform more strongly about the posterior distribution on the number, sizes and allocation to clusters.

P2d. Location-scale symmetrized PSB (sPSB) mixture of Gaussians

A homoscedastic version of (6) is the following.

$$f(\cdot )=\int \mathrm{N}(\cdot ;\mu ,{\tau }^{-1})d{P}^s(\mu ,\tau ),d{P}^s(\mu ,\tau )=\frac{1}{2}dP(-\mu ,\tau )+\frac{1}{2}dP(\mu ,\tau ),P=\sum _{h=1}^{\infty }{\pi }_h{\delta }_{({\mu }_h,{\tau }_h)},({\mu }_h,{\tau }_h)~P_0,$$ (9)

where the prior on the weights π_h are given by (8) and the prior for (μ_h, τ_h) are exactly as in 2b.

3 Consistency properties

Let f ~ Π_u and f ~ Π_s denote the priors for the unknown residual density defined in expressions (7) and (9) respectively. It is appealing for Π_u and Π_s to have support on a large subset of 𝒮_u and 𝒮_s respectively, where 𝒮_s denotes the set of densities on ℝ with respect to Lebesgue measure that are symmetric about zero and 𝒮_u ⊂ 𝒮_s is the subset of 𝒮_s corresponding to unimodal densities. We characterize the weak support of Π_u, denoted by wk(Π_u) ⊂ 𝒮_u, in the following lemma.

Lemma 1 wk(Π_u) = 𝒞_m, where 𝒞_m = {f : f ∈ 𝒮_u, $h(x)=f(\sqrt{x}),x>0$ is a completely monotone function}.

A function h(x) on (0,∞) is completely monotone in x if it is infinitely differentiable and ${(-1)}^m\frac{d^m}{d{x}^m}h(x)\ge 0$ for all x and for all m ∈ {1, 2, …, ∞}. Chu (1973) proved that if f is a density on ℝ which is symmetric about zero and unimodal, it can be written as a scale mixture of normals,

$$f(x)=\int {\sigma }^{-1}\varphi ({\sigma }^{-1}x)g(\sigma )d\sigma$$

for some density g on ℝ, if and only if $h(x)=f(\sqrt{x}),x>0$, is a completely monotone function, where ϕ is the standard normal pdf. This restriction places a smoothness constraint on f(x), but still allows a broad variety of densities.

Definition 1 Letting f ~ Π, f₀ is in the Kullback-Leibler(KL) support of Π if

$$\Pi (f:\int f_0(y)\text{log}\frac{f_0(y)}{f(y)}dy<\epsilon )>0,\forall \epsilon >0$$

The set of densities f in the Kullback-Leibler support of Π is denoted by KL(Π).

Let 𝒮̃_s denote the subset of 𝒮_s corresponding to densities satisfying the following regularity conditions.

1. f is nowhere zero and bounded by M < ∞

2. | ∫_ℜ f (y) log f (y)dy| < ∞

3. $|{\int }_{\Re }f(y)\text{log}\frac{f(y)}{{\psi }_1(y)}dy|<\infty$, where ψ₁ (y) = inf_{t∈[y−1, y+1]} f(t)

4. there exists ψ > 0 such that ∫_ℜ |y|^2(1+ψ) f (y)dy < ∞

Lemma 2 KL(Π_s) ⊇ 𝒮̃_s.

Remark 1 The above assumptions on f are standard regularity conditions introduced by Tokdar (2006) and Wu and Ghoshal (2008) to prove that f ∈ KL(Π), where Π is a general stick breaking prior which has all compactly supported probability distributions as its support. (1) is usually satisfied by common densities arising in practice. (4) imposes a minor tail restriction e.g., t-density with (2 + δ) degrees of freedom for some δ > 0 satisfies (4). (1)–(4) are satisfied by a finite mixture of t-densities or even by an infinite mixture of t-densites with (2 + δ) degrees of freedom for some δ > 0 and bounded component specific means and variances.

From Lemma 2, it follows that the sPSB location-scale mixture has KL-support on a large subset of the set of densities symmetric about zero. These conditions are important in verifying that the priors are flexible enough to approximate any density subject to the noted restrictions.

We provide fairly general sufficient conditions to ensure strong and weak posterior consistency in estimating the mean regression function and the residual density, respectively. We focus on the case in which a GP prior is chosen for η and an sPSB location-scale mixture of Gaussians is chosen for the residual density as in (9). Similar results can be obtained for the homoscedastic scale PSB process mixture under stronger restrictions on the true residual density. Although showing consistency results using predictor dependent mixtures of normals as the prior for the residual density in (5) and (6) is a challenging task, one can anticipate such results given the theory in Pati et al (2010) and Norets and Pelenis (2010). Indeed Norets and Pelenis (2011) showed posterior consistency of the regression coefficients in a mean linear regression model with covariate dependent nonparametric residuals using the kernel stick-breaking process Dunson and Park (2008a). However, showing posterior consistency of the mean regression when we have a Gaussian process prior on the regression function and predictor dependent residuals is quite challenging and is a topic of future research.

For this section, we assume x_i’s are non random and arising from a fixed design, though the proofs are easily modified for random x_i’s. When the covariate values are fixed in advance, we consider the neighborhood based on the empirical measure of the design points. Let Q_n be the empirical probability measure of the design points, $Q_n(\mathbf{x})=\frac{1}{n}{\sum }_{i=1}^n{I}_{{\mathbf{x}}_i}(\mathbf{x})$ . Based on Q_n, we define a strong L₁ neighborhood of radius Δ > 0 as in Choi (2005) around the true regression function η₀. Letting ∥η − η₀∥_1,n = ∫_x∈𝒳 |η(x) − η₀(x)|dQ_n(x) set,

$$S_n({\eta }_0;\Delta )=\{\eta :{\Vert \eta -{\eta }_0\Vert }_{1,\eta }<\Delta \}$$ (10)

We introduce the following notation. Let f₀ denote an arbitrary fixed density in 𝒮̃_s, η₀ denote an arbitrary fixed regression function in ℱ, and

$$f_{0i}=f_0(y-{\eta }_0({\mathbf{x}}_i))f_{\eta i}=f(y-\eta ({\mathbf{x}}_i)).$$

For any two densities f and g, let

$$K(f,g)={\int }_{ℝ}f(y)\text{log}\{f(y)/g(y)\}dy,V(f,g)={\int }_{ℝ}f(y){[{\text{log}}_{+}\{f(y)/g(y)\}]}^{2}dy,$$

where log₊ x = max(log x, 0). Set K_i(f, η) = K(f_0i, f_ηi) and V_i(f, η) = V (f_0i, f_ηi) for i = 1, …, n.

For technical simplicity assume 𝒳 = [0, 1]^p, τ = 1 and μ ≡ 0. Denote a mean zero Gaussian process {W_x : x ∈ [0, 1]^p} with covariance kernel c(x, x′) = e^{−∥x−x′∥2} by W. Rescaling the sample paths of an infinitely smooth Gaussian process is a powerful technique to improve the approximation of α- Hölder functions from the RKHS of the scaled process $\{W_{\mathbf{x}}^{\kappa }=W_{\sqrt{\kappa }\mathbf{x}}:\mathbf{x}\in {[0,1]}^d\}$ with κ > 0. Intuitively, for large values of κ, the scaled process traverses the sample path of an unscaled process on the larger interval ${[0,\sqrt{\kappa }]}^p$, thereby incorporating more “roughness”. van der Vaart and van Zanten (2009) studied rescaled Gaussian processes $W^{\kappa }=\{W_{\sqrt{\kappa }\mathbf{x}}:\mathbf{x}\in {[0,1]}^p\}$ for a positive random variable κ stochastically independent of W and showed that with a Gamma prior on κ^p/2, one obtains the minimax-optimal rate of convergence for arbitrary smooth functions.

Assumption1: η ~ W^κ with the density g of √κ on the positive real line satisfying

$$C_1{x}^p\text{exp}(-D_{1}x{\text{log}}^{q}x)\le g(x)\le C_2{x}^p\text{exp}(-D_{2}x{\text{log}}^{q}x),$$

for positive constants C₁, C₂, D₁, D₂ and every sufficiently large x > 0. Next we state the lemma on prior positivity due to van der Vaart and van Zanten (2009).

Lemma 3 If η satisfies Assumption 1 then P(∥η − η₀∥_∞ < ε) > 0 ∀ ε > 0, if η₀ is continuous.

In order to prove posterior consistency for our proposed model, we rely on a theorem of Amewou-Atisso et al (2003), which is a modification of the celebrated Schwartz (1965) theorem to accommodate independent but not identically distributed data.

Theorem 1 Suppose η as in Assumption 1 with q ≥ p + 2 and f ~ Π_s, with Π_s defined in (9). In addition, assume the data are drawn from the true density f₀ (y_i − η₀ (x_i)), with {x_i} fixed and non-random, f₀ ∈ 𝒮̃_s, η₀ ∈ ℱ and f₀ following the additional regularity conditions,

1. ∫ y⁴ f₀ (y) dy < ∞ and ∫ f₀ (y) | log f₀ (y)|² dy < ∞.

2. ${\int }_{ℝ}{f}_0(y){\left|\text{log}\frac{f_0(y)}{{\psi }_1(y)}\right|}^{2}dy<\infty$ , where ψ₁(y) = inf_{t∈[y−1,y+1]} f₀(t).

Let 𝒰 be a weak neighborhood of f₀ and 𝒲_n = 𝒰×S_n (η₀; Δ), with 𝒲_n ⊂ 𝒮̃_s × ℱ. Then the posterior probability

$$({\Pi }_s\times W^{\kappa })({𝒲}_n|y_1,\dots ,y_n,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n)=\frac{{\int }_{{𝒲}_n}{\prod }_{i=1}^n{f}_{\eta i}(y_i)d{\Pi }_s(f)d{W}^{\kappa }(\eta )}{{\int }_{{\widetilde{𝒮}}_s\times ℱ}{\prod }_{i=1}^n{f}_{\eta i}(y_i)d{\Pi }_s(f)d{W}^{\kappa }(\eta )}\to 1a.s.$$

Theorem 1 ensures weak consistency of the posterior of the residual density and strong consistency of the posterior of the regression function η.

4 Posterior computation

We first describe the choice of hyperpriors and hyperparameters for the regression function. We choose the typical conjugate prior for the regression coefficients in the mean of the GP, β ~ N(β₀, Σ₀), where β₀ = 0 and Σ₀ = cI is a common choice corresponding to a ridge regression shrinkage prior. The prior on τ is given by $\tau ~\mathrm{G}\mathrm{a}(\frac{{\nu }_{\tau }}{2},\frac{{\nu }_{\tau }}{2})$. We let κ ~ Ga(α_κ, β_κ) with small β_κ, and large α_κ. Normalizing the predictors prior to analysis, we find that the data are quite informative about κ under these priors, so as long as the priors are not overly informative, inferences are robust. The parameter τ controls the heaviness of the tails of the prior for the regression function. In fact, choosing a Ga(ν_τ/2, ν_τ/2) prior induces a heavy tailed t-process with ν_τ degrees of freedom as a prior for the regression function. We chose ν_τ to be 3. κ controls the correlation of the Gaussian process at two points in the covariate space similar to a spatial decay parameter in a spatial random effects model. Although a discrete uniform prior for κ is computationally efficient in leading to a griddy Gibbs update step, there can be sensitivity to the choice of grid. A gamma prior for κ eliminates such sensitivity at some associated computational price in terms of requiring a Metropolis-Hastings update that tends to mix slowly. We choose the parameters α_κ and β_κ so that the mean correlation is 0.1 for two points separated by a distance √p in the covariate space.

Next we describe the hyperprior and associated hyperparameter choices for P1 and P2a-d.

1. P1: Since the responses are normalized and the covariates are scaled to lie in the interval [0, 1], using a single decay parameter appears to be reasonable. ν_σ controls the tail-heaviness of the prior for the scaling ϕ. To accommodate outliers with the mean being fixed at 1, we assume ϕ_i ~ Ga(ν_σ/2, ν_σ/2) with ν_σ ~ Ga(α_ν, β_ν). We took Σ₀ = 5I, α_ν = 1, β_ν = 1. a and b are fixed at 3/2 to resemble a t-distribution with 3 degrees of freedom without the scaling ϕ_i.

2. P2a & P2b: We assume κ_α ~ Ga(γ_κ, δ_κ) and ${\tau }_{\alpha }~\mathrm{G}\mathrm{a}(\frac{{\nu }_{\alpha }}{2},\frac{{\nu }_{\alpha }}{2})$. Assuming y_i are normalized, we can expect the overall variance to be close to one, so the variance of the residuals, $\mathit{\text{Var}}(\epsilon |\mathbf{x})={\sum }_{h=1}^{\infty }{\pi }_h(\mathbf{x}){\tau }_h^{-1}$, should be less than one. We set α_τ = 1 and choose a hyperprior on β_τ, β_τ ~ Ga(1, k₀) with k₀ > 1 so that the prior mean of τ_h is significantly less than one. Different values of k₀ are tried out to assess robustness of the posteriors. In Sections 5 and 6, we choose γ_κ = 1, δ_κ = 5, ν_α = 1, k₀ = 10, μ₀ = 0, σ₀ = 1.

3. P2c & P2d: Same choices as above with except k₀ = 5, μ_α = 0, σ_α = 1.

For brevity, we provide details for posterior computation only for P1, P2a–b.

4.1 Gaussian process regression with t residuals (P1)

Let Y = (y₁, … y_n)′, η = (η(x₁), η(x₂), …, η(x_n))′ and define a matrix T such that T_ij = e^{−κ∥x_i−x_j∥2}. Hence ${\mathit{\Sigma }}^{\eta }=\frac{1}{\tau }\mathbf{T}.$. Assume Ω = diag(1/ϕ_i : i = 1, …, n) and ϕ = (ϕ₁, … , ϕ_n)′. Then we have

$$\mathbf{Y}|\mathit{\eta }~\mathrm{N}(\mathit{\eta },{\sigma }^2\mathbf{\Omega }),\mathit{\eta }|\mathit{\beta },\tau ,\kappa ~\mathrm{N}(\mathbf{X}\mathit{\beta },{\tau }^{-1}\mathbf{T}),\mathit{\beta }~\mathrm{N}({\mathit{\beta }}_0,{\mathit{\Sigma }}_0){\varphi }_i~\mathrm{G}\mathrm{a}(\frac{{\nu }_{\sigma }}{2},\frac{{\nu }_{\sigma }}{2}),{\nu }_{\sigma }~\mathrm{G}\mathrm{a}({\alpha }_{\nu },{\beta }_{\nu }),{\sigma }^{-2}~\mathrm{G}\mathrm{a}(a,b)\kappa ~\mathrm{G}\mathrm{a}({\alpha }_{\kappa },{\beta }_{\kappa }),\tau ~\mathrm{G}\mathrm{a}(\frac{{\nu }_{\tau }}{2},\frac{{\nu }_{\tau }}{2}).$$

Next we provide the full conditional distributions needed for Gibbs sampling. Due to conjugacy, η, β, σ⁻², ϕ and τ have closed form full conditional distributions, while ν_α and κ are updated by using Metropolis Hastings steps within the Gibbs sampler. Let V_η = (τT⁻¹ + σ⁻² Ω⁻¹)⁻¹ and $V_{\beta }={(\tau \mathbf{X}\prime {\mathbf{T}}^{-1}\mathbf{X}+{\sum }_0^{-1})}^{-1}$.

$$\mathit{\eta }|\mathbf{Y},\mathit{\beta },{\sigma }^{-2},\tau ,\kappa ,{\nu }_{\sigma },\varphi ~\mathrm{N}(V_{\mathit{\eta }}(\tau {\mathbf{T}}^{-1}\mathbf{X}\mathit{\beta }+{\sigma }^{-2}{\mathbf{\Omega }}^{-1}\mathbf{Y}),V_{\mathit{\eta }})\mathit{\beta }|\mathbf{Y},\eta ,{\sigma }^{-2},\tau ,\kappa ,{\nu }_{\sigma },\mathit{\varphi }~\mathrm{N}(V_{\mathit{\beta }}(\tau \mathbf{X}\prime {\mathbf{T}}^{-1}\mathit{\eta }+{\sum }_0^{-1}{\mathit{\beta }}_0),V_{\mathit{\beta }}){\sigma }^{-2}|\mathbf{Y},\mathit{\eta },\mathit{\beta },\tau ,\kappa ,{\nu }_{\sigma },\mathit{\varphi }~\mathrm{G}\mathrm{a}(\frac{n}{2}+a,\frac{1}{2}\sum _{i=1}^n{\varphi }_i{(y_i-{\eta }_i)}^2+b\left)\tau \right|\mathbf{Y},\mathit{\eta },\mathit{\beta },{\sigma }^{-2},\kappa ,{\nu }_{\sigma },\mathit{\varphi }~\mathrm{G}\mathrm{a}(\frac{n+{\nu }_{\tau }}{2},\frac{1}{2}\{(\mathit{\eta }-\mathbf{X}\mathit{\beta })\prime {\mathbf{T}}^{-1}(\mathit{\eta }-\mathbf{X}\mathit{\beta })+{\nu }_{\tau }\}){\varphi }_i|\mathbf{Y},\mathit{\eta },\mathit{\beta },{\sigma }^{-2},\kappa ,{\nu }_{\sigma }~\mathrm{G}\mathrm{a}(\frac{{\nu }_{\sigma }+1}{2},\frac{1}{2}\{{\sigma }^{-2}{(y_i-{\eta }_i)}^2+{\nu }_{\sigma }\}).$$

4.2 Heteroscedastic PSB mixture of normals (P2a)

We propose a Markov chain Monte Carlo algorithm, which is a hybrid of data augmentation, the exact block Gibbs sampler of Papaspiliopoulos (2008) and Metropolis Hastings sampling. Papaspiliopoulos (2008) proposed the exact block Gibbs sampler as an efficient approach to posterior computation in Dirichlet process mixture models, modifying the block Gibbs sampler of Ishwaran and James (2001) to avoid truncation approximations. The exact block Gibbs sampler combines characteristics of the retrospective sampler (Papaspiliopoulos and Roberts 2008) and the slice sampler (Walker 2007); Kalli et al 2010). We included the label switching moves introduced by Papaspiliopoulos and Roberts (2008) for better mixing. Introduce γ₁, …, γ_n such that π_h(x_i) = P(γ_i = h), h = 1, 2, … ,∞. Then

$${\gamma }_i~\sum _{h=1}^{\infty }{\pi }_h({\mathbf{x}}_i){\delta }_h=\sum _{h=1}^{\infty }1(u_i<{\pi }_h({\mathbf{x}}_i)){\delta }_h$$

where u_i ~ U(0, 1). The MCMC steps are given below.

1. Update u_i’s and stick breaking random variables: Generate

$$u_i|-~U(0,{\pi }_{{\gamma }_i}({\mathbf{x}}_i))$$

where π_h(x_i) = Φ{α_h(x_i)} ∏_l<h[1 − Φ {α_l(x_i)}]. For i = 1, …, n, introduce latent variables Z_h(x_i), h = 1, 2, … such that Z_h(x_i) ~ N(α_h(x_i), 1). Thus π_h(x_i) = P(Z_h(x_i) > 0, Z_l(x_i) < 0 for l < h). Then

$$Z_h({\mathbf{x}}_i)|-~\{\begin{array}{c}\mathrm{N}({\alpha }_h({\mathbf{x}}_i),1)I_{ℝ+},h={\gamma }_i\\ \mathrm{N}({\alpha }_h({\mathbf{x}}_i),1)I_{ℝ-},h<{\gamma }_i.\end{array}$$

Let Z_h = (Z_h(x₁), …, Z_h(x_n))′ and α_h = (α_h(x₁), … , α_h(x_n))′. Letting (Σ_α)_ij = e^{−κ_α∥x_i−x_j∥}, Z_h ~ N(α_h, I) and ${\mathbf{\alpha }}_h~\mathrm{N}(0,\frac{1}{{\tau }_{\alpha }}{\mathbf{\Sigma }}_{\alpha })$,

$${\mathbf{\alpha }}_h|-~\mathrm{N}({({\tau }_{\alpha }{\mathbf{\Sigma }}_{\alpha }^{-1}+{\mathbf{I}}_n)}^{-1}{\mathbf{Z}}_h,{({\tau }_{\alpha }{\mathbf{\Sigma }}_{\alpha }^{-1}+{\mathbf{I}}_n)}^{-1})$$

Continue up to $h=1,\dots ,h^{*}=\text{max}\{h_1^{*},\dots ,h_n^{*}\},\text{where}{h}_i^{*}$ is the minimum integer satisfying ${\sum }_{l=1}^{h_i^{*}}{\pi }_l({\mathbf{x}}_i)>1-\text{min}\{u_1,\mathrm{\dots },u_n\},i=1,\dots ,n.$. Now

$${\tau }_{\alpha }|-~\mathrm{G}\mathrm{a}(\frac{1}{2}(n{h}^{*}+{\nu }_{\alpha }),\frac{1}{2}(\sum _{l=1}^{h^{*}}{\mathbf{\alpha }}_{\kappa }^{\prime }{\mathbf{\Sigma }}_{\alpha }^{-1}{\mathbf{\alpha }}_{\kappa }+{\nu }_{\alpha })),$$

while κ_α is updated using a Metropolis Hastings step.

2. Update allocation to atoms: Update (γ₁, … , γ_n)|− as multinomial random variables with probabilities

$$P({\gamma }_i=h)\propto \mathrm{N}(y_i;\eta ({\mathbf{x}}_i),{\tau }_h^{-1})I(u_i<{\pi }_h({\mathbf{x}}_i)),h=1,\dots ,h^{*}.$$

.

3. Update component-specific locations and precisions: Letting n_l = #{i : γ_i = l}, l = 1, 2, … ; h*,

$${\tau }_l|-~\mathrm{G}\mathrm{a}(\frac{n_l}{2}+{\alpha }_{\tau },{\beta }_{\tau }+\sum _{i:{\gamma }_i=l}{(y_i-\eta ({\mathbf{x}}_i))}^2),l=1,2,\dots ,h^{*}{\beta }_{\tau }|-~\mathrm{G}\mathrm{a}(1,\sum _{l=1}^{k^{*}}{\tau }_l+k_0).$$

4. Update the mean regression function: Letting $\mathit{\wedge }=\text{diag}({\tau }_{{\gamma }_1}^{-1},\dots ,{\tau }_{{\gamma }_n}^{-1})$,

$$\mathit{\eta }|-~\mathrm{N}({(\tau {\mathbf{T}}^{-1}+{\mathit{\wedge }}^{-1})}^{-1}(\tau {\mathbf{T}}^{-1}\mathbf{X}\mathit{\beta }+{\mathit{\wedge }}^{-1}\mathbf{Y}),{(\tau {\mathbf{T}}^{-1}+{\mathit{\wedge }}^{-1})}^{-1})\mathit{\beta }|-~\mathrm{N}({(\tau \mathbf{X}\prime {\mathbf{T}}^{-1}\mathbf{X}+\tau {\sum }_0^{-1})}^{-1}(\tau \mathbf{X}\prime {\mathbf{T}}^{-1}\eta +{\sum }_0^{-1}{\beta }_0),{(\tau \mathbf{X}\prime {\mathbf{T}}^{-1}\mathbf{X}+{\sum }_0^{-1})}^{-1})\tau |-~\mathrm{G}\mathrm{a}(\frac{n+{\nu }_{\tau }}{2},\frac{1}{2}\{(\mathit{\eta }-\mathbf{X}\mathit{\beta })\prime {\mathbf{T}}^{-1}(\mathit{\eta }-\mathbf{X}\mathit{\beta })\prime +{\nu }_{\tau }\}).$$

5. Update κ in a Metropolis Hastings step.

4.3 Heteroscedastic sPSB process location-scale mixture (P2b)

We will need the following changes in the updating steps from the previous case.

1. Update allocation to atoms: Update (γ₁, … , γ_n)|− as multinomial random variables with probabilities

$$P({\gamma }_i=h)\propto \frac{1}{2}\{\mathrm{N}(y_i;\eta ({\mathbf{x}}_i)+{\mu }_h,{\tau }_h^{-1})+\mathrm{N}(y_i;\eta ({\mathbf{x}}_i)-{\mu }_h,{\tau }_h^{-1})\}I(u_i<{\pi }_h({\mathbf{x}}_i)),h=1,\dots ,h^{*}.$$

h = 1, … , h*.

3. Component-specific locations and precisions: Let n_l = #{i : γ_i = l}, l = 1,2, … , h* and m_l = ∑_i:γi=l(y_i − η_i). The atoms of the base measure location is updated from a mixture of normals as

$${\mu }_l|-~p_l\mathrm{N}(\frac{{\mu }_0{\sigma }_0^{-2}+{\tau }_l{n}_l}{{\sigma }_0^{-2}+n_l{\tau }_l},\frac{1}{{\sigma }_0^{-2}+n_l{\tau }_l})+(1-p_l)\mathrm{N}(\frac{{\mu }_0{\sigma }_0^{-2}-{\tau }_l{n}_l}{{\sigma }_0^{-2}+n_l{\tau }_l},\frac{1}{{\sigma }_0^{-2}+n_l{\tau }_l}),$$

where $p_l\propto \text{exp}\left\{\frac{1}{2}\right(\frac{{\mu }_0{\sigma }_0^{-2}+{\tau }_l{n}_l}{{\sigma }_0^{-2}+n_l{\tau }_l}\left)\right\}$.

$${\tau }_l|-~p_l\mathrm{G}\mathrm{a}(\frac{n_l}{2}+{\alpha }_{\tau },{\beta }_{\tau }+\sum _{i:{\gamma }_i=l}{\{y_i-\eta ({\mathbf{x}}_i)-{\mu }_l\}}^2)+(1-p_l)\mathrm{G}\mathrm{a}(\frac{n_l}{2}+{\alpha }_{\tau },{\beta }_{\tau }+\sum _{i:{\gamma }_i=l}\{y_i-\eta ({\mathbf{x}}_i)+{\mu }_l\}2),$$

where $p_l\propto {\left\{\frac{1}{{\beta }_{\tau }+\frac{1}{2}{\sum }_{i:{\gamma }_i=l}{\{y_i-\eta ({\mathbf{x}}_i)-{\mu }_l\}}^2)}\right\}}^{\frac{n_l}{2}+\alpha }$ .

4. Update the mean regression function: Let $\mathbf{\wedge }=\text{diag}({\tau }_{{\gamma }_1}^{-1},\dots ,{\tau }_{{\gamma }_n}^{-1})$, μ* = (μ_γ1, μ_γ2, … , μ_γn) and W = (τT⁻¹ + ∧⁻¹)⁻¹. Hence

$$\mathit{\eta }|-p\mathrm{N}(\mathit{\eta };\mathbf{W}\{\tau {\mathbf{T}}^{-1}\mathbf{X}\beta +{\mathit{\wedge }}^{-1}(\mathbf{Y}-{\mathit{\mu }}^{*})\},\mathbf{W})+(1-p)\mathrm{N}(\mathit{\eta };\mathbf{W}\{\tau {\mathbf{T}}^{-1}\mathbf{X}\mathit{\beta }+{\mathit{\wedge }}^{-1}(\mathbf{Y}+{\mathit{\mu }}^{*})\},\mathbf{W})$$

where $p\propto \text{exp}[\frac{1}{2}\{\tau {\mathbf{T}}^{-1}\mathbf{X}\mathit{\beta }+{\mathit{\wedge }}^{-1}(\mathbf{Y}-{\mathit{\mu }}^{*}))\prime \mathbf{W}\mathbf{X}\mathit{\beta }+{\mathit{\wedge }}^{-1}(\mathbf{Y}-{\mathit{\mu }}^{*})-(\mathbf{Y}-{\mathit{\mu }}^{*})\prime {\mathit{\wedge }}^{-1}(\mathbf{Y}-{\mathit{\mu }}^{*})\}]$.

5 Measures of influence

There has been limited work on sensitivity of the posterior distribution to perturbations of the data and outliers. Arellano-Vallea et al (2000) use deletion diagnostics to assess sensitivity, but their methods are computationally expensive in requiring posterior computation with and without data deleted. Weiss (1996) proposed an alternative that perturbs the posterior instead of the likelihood, and only requires samples from the full posterior. Following Weiss (1996), let f(y_i|Θ̃, x_i) denote the likelihood of the data y_i, define

$${\delta }_i^{*}(\tilde{\Theta })=\frac{f(y_i+\delta |\tilde{\Theta },{\mathbf{x}}_i)}{f(y_i|\tilde{\Theta },{\mathbf{x}}_i)},$$

, for some small δ > 0 and let p_i (Θ̃|Y) denote a new perturbed posterior,

$$p_i(\tilde{\Theta }|\mathbf{Y})=\frac{p(\tilde{\Theta }+\mathbf{Y}){\delta }_i^{*}(\tilde{\Theta })}{𝖤({\delta }_i^{*}(\tilde{\Theta })|\mathbf{Y})}.$$

Since the responses are normalized prior to analysis, it is reasonable to assume that the perturbation is less than 0.1. We vary δ in [0.01, 0.1] over a grid of width 0.01 and obtain the average of results. Denote by L_i the influence measure, which is a divergence measure between the unperturbed posterior p(Θ̃|Y) and the perturbed posterior p_i(Θ̃|Y),

$$L_i=\frac{1}{2}\int |p(\tilde{\Theta }|\mathbf{Y})-p_i(\tilde{\Theta }|\mathbf{Y})d\tilde{\Theta }.$$

L_i is bounded and takes values in [0, 1]. When p(Θ̃|Y) = p_i(Θ̃|Y), L_i = 0 indicating that the perturbation ${\delta }_i^{*}$ has no influence. On the other hand, if L_i = 1, the supports of p(Θ̃|Y) and p_i(Θ̃|Y) are disjoint indicating maximum influence. We can define an influence measure as $L=\frac{1}{n}{\sum }_{i=1}^n{L}_i$. Clearly L also takes values in [0, 1] with L = 0 ⇒ L_i = 0 ∀ i = 1, 2, … , n. Also L = 1 ⇒ L_i = 1 ∀ i = 1, 2, … , n. Weiss (1996) provided a sample version of L_i, i = 1, … , n. Letting Θ̃₁, … , Θ̃_M be the posterior samples with B the burn-in,

$${\widehat{L}}_i=\frac{1}{M-B}\sum _{k=B+1}^M\frac{1}{2}|\frac{{\delta }_i^{*}({\tilde{\Theta }}_k)}{\widehat{E}({\delta }_i^{*}(\tilde{\Theta }))}-1|,$$

where $\widehat{𝖤}\{{\delta }_i^{*}(\tilde{\Theta })\}=\frac{1}{M-B}{\sum }_{k=B+1}^M{\delta }_i^{*}({\tilde{\Theta }}_k)$ . Our estimated influence measure is $\widehat{L}=\frac{1}{n}{\sum }_{i=1}^n{\widehat{L}}_i$ . We will calculate the influence measure for our proposed methods and compare their sensitivity.

6 Simulation studies

To assess the performance of our proposed approaches, we consider a number of simulation examples, (i) linear model, homoscedastic error with no outliers, (ii) linear model, homoscedastic error with outliers (iii) linear model, heteroscedastic errors and outliers, (iv) non-linear model with heteroscedastic errors and outliers and (v) non-linear model with heteroscedastic errors and outliers, but with fewer true predictors. We let the heaviness of the tails and error variance change with x in cases (iii), (iv) and (v). We considered the following methods of assessing the performance, namely, mean squared prediction error (MSPE), coverage of 95% prediction intervals, mean integrated squared error (MISE) in estimating the regression function at the points for which we have data, point wise coverage of 95% credible intervals for the regression function and the influence measure (L̂) as described in Section 5. We also consider a variety of sample sizes in the simulation, n=30, 60, 80 and simulate 10 covariates independently from U(0, 1). Let z be 10-dim vector of i.i.d U(0, 1) random variables independent of the covariates.

Generation of errors in heteroscedastic case and outliers: Let f_xi (ε_i) = p_xi N(ε_i; 0, 1) + q_xi N(ε_i; 0, 5) where $p_{{\mathbf{x}}_i}=\Theta ({\mathbf{x}}_i^{\prime }\mathbf{z})$ . The outliers are simulated from the model with error distribution $f_{{\mathbf{x}}_i}^{\mathrm{o}}(\cdot )$, which is a mixture of truncated normal distributions as follows. In the heteroscedastic case, $f_{{\mathbf{x}}_i}^{\mathrm{o}}({\epsilon }_i)=p_{{\mathbf{x}}_i}\mathrm{T}{\mathrm{N}}_{(-\infty ,3)\cup (3,\infty )}({\epsilon }_i;0,1)+q_{{\mathbf{x}}_i}T{\mathrm{N}}_{(-\infty ,-3\sqrt{5})\cup (3\sqrt{5},\infty )}({\epsilon }_i;0,5)$ where TN_ℛ(·;μ, σ²) denotes a truncated normal distribution with mean μ and standard deviation σ over the region ℛ. We consider the following five cases.

1. Case (i): y_i = 2.3 + 5.7x_1i + ε_i, ε_i ~ N(0, 1) with no outliers.

2. Case (ii): y_i = 2.3 + 5.7x_1i + ε_i, ε_i ~ 0.95N(0, 1) + 0.05N(0, 10).

3. Case (iii): y_i = 1.2 + 5.7x_1i + 4.7x_2i + 0.12x_3i −8.9x_4i + 2.4x_5i + 3.1x_6i + 0.01_x7i + ε_i, ε_i ~ f_xi, with 5% outliers generated from $f_{{\mathbf{x}}_i}^o({\epsilon }_i)$ .

4. Case (iv): $y_i=1.2+5.7x_{1i}+3.4x_{1i}^2+4.7x_{i2}+0.89x_{i2}^2+0.12x_{i3}-8.9x_{i4}{x}_{i8}+2.4x_{i5}{x}_{i9}+3.1x_{i6}+x_{i6}^2+0.01x_{i7}+{\epsilon }_i,{\epsilon }_i~f_{x_i}$ with 5% outliers generated from $f_{{\mathbf{x}}_i}^{\mathrm{o}}({\epsilon }_i)$.

5. Case (v): y_i = 1.2 + 5.7 sin x_1i + 3.4 exp(x_2i) + 4.7 log |x_i3| + ε_i, ε_i ~ f_xi with 5% outliers generated from $f_{{\mathbf{x}}_i}^{\mathrm{o}}({\epsilon }_i)$.

For each of the cases and for each sample size n, we took the first $\frac{n}{2}$ samples as the training set and the next $\frac{n}{2}$ samples as the test set. We also compare the MSPE of the proposed methods with robust regression using M-estimation (Huber 1964), Bayesian additive regression trees (Chipman et al 2010), and Treed Gaussian processes (Gramacy and Lee 2008). The MCMC algorithms described in Section 5 are used to obtain samples from the posterior distribution. The results for model P1 given here are based on 20,000 samples obtained after a burn-in period of 3,000. The results for models P2a–d are based on 20,000 samples obtained after a period of 7,000. Rapid convergence was observed based on diagnostic tests of Geweke (1992) and Raftery and Lewis (1992). In addition, the mixing was very good for model P1. For models P2a–d, we use the label switching moves by Papaspiliopoulos and Roberts (2008), which lead to adequate mixing. Tables 1, 2 and 3 summarize the performance of all the methods based on 50 replicated datasets.

Table 1.

Simulation results under homoscedastic residuals (Cases (i) and (ii))

n=40	MSPE	cov(y)a	Case (i) MISE	cov(η)b	L	MSPE	cov(y)	Case (ii) MISE	cov(η)	L
P1	0.2997	1	0.0248	1	0.0017	0.6043	1	0.0232	1	0.0027
P2a	0.2821	0.9980	0.0141	1	0.0015	0.5983	0.9740	0.0173	1	0.0019
P2b	0.2798	1	0.0144	1	0.0015	0.5987	0.9745	0.0169	1	0.0017
P2c	0.2980	1	0.0156	1	0.0016	0.5980	0.9750	0.0189	1	0.0020
P2d	0.2869	1	0.0155	1	0.0016	0.5983	0.9750	0.0190	1	0.0020
M-Estimation	0.2820		0.0140			0.6013		0.0177
BART	0.3510	0.6866	0.0714			0.7051	0.7845	0.0950
Treed GP	0.3042	0.9134	0.0256			0.6968	0.9365	0.0803
n=60	MSPE	cov(y)	MISE	cov(η)	L	MSPE	cov(y)	MISE	cov(η)	L
P1	0.2990	1	0.0246	1	0.0019	0.5776	1	0.0242	1	0.0023
P2a	0.2769	0.9947	0.0103	1	0.0017	0.5471	0.95	0.0143	0.97	0.0016
P2b	0.2752	0.9963	0.0104	1	0.0016	0.5541	0.95	0.0141	0.98	0.0016
P2c	0.2852	0.9945	0.0176	1	0.0019	0.5664	0.95	0.0142	0.99	0.0021
P2d	0.2826	0.9960	0.0173	1	0.0018	0.5561	0.95	0.0141	0.98	0.0021
M-Estimation	0.2759		0.0103			0.5623		0.0139
BART	0.3314	0.6753	0.0539			0.6725	0.7777	0.1098
Treed GP	0.3000	0.9193	0.0218			0.6880	0.9301	0.1198
n=80	MSPE	cov(y)	MISE	cov(η)	L	MSPE	cov(y)	MISE	cov(η)	L
P1	0.2913	1	0.0252	1	0.0021	0.5583	1	0.0172	1	0.0022
P2a	0.2592	0.9940	0.0086	1	0.0021	0.4989	0.97	0.0050	1	0.0014
P2b	0.2574	0.9956	0.0069	1	0.0020	0.4898	0.98	0.0067	1	0.0010
P2c	0.2724	0.9976	0.0187	1	0.0020	0.5104	0.98	0.0103	1	0.0017
P2d	0.2716	0.9976	0.0189	1	0.0020	0.5002	0.98	0.0097	1	0.0018
M-Estimation	0.2720		0.0079			0.5431		0.0068
BART	0.3128	0.6525	0.0437			0.6509	0.7815	0.1098
Treed GP	0.2886	0.9301	0.0175			0.6532	0.9224	0.1031

^acov(y) denotes the coverage of the 95% predictive intervals of the test cases

^bcov(η) denotes the coverage of the 95% credible intervals of the mean regression function

Table 2.

Simulation results under heteroscedastic residuals (Cases (iii) and (iv))

n=40	MSPE	cov(y)	Case (iii) MISE	cov(η)	L	MSPE	cov(y)	Case (iv) MISE	cov(η)	L
P1	0.4833	1	0.3612	1	0.0027	0.4416	1	0.3274	1	0.0029
P2a	0.2570	0.9990	0.1394	1	0.0025	0.2783	0.9923	0.1583	0.98	0.0023
P2b	0.2586	0.9990	0.1298	1	0.0025	0.2712	0.9867	0.1501	0.97	0.0017
P2c	0.3057	1	0.2412	1	0.0026	0.3334	0.9967	0.2213	0.99	0.0024
P2d	0.3024	1	0.2304	1	0.0026	0.3216	0.9967	0.2192	0.99	0.0022
M-Estimation	0.2613		0.1376			0.2889		0.1663
BART	0.4639	0.8444	0.3413			0.4103	0.8833	0.2675
Treed GP	0.3320	0.7834	0.1979			0.3548	0.8268	0.2108
n=60	MSPE	cov(y)	MISE	cov(η)	L	MSPE	cov(y)	MISE	cov(η)	L
P1	0.2254	1	0.1154	1	0.0023	0.2367	1	0.1067	1	0.0021
P2a	0.1744	0.9973	0.0572	1	0.0020	0.2178	1	0.0562	0.97	0.0019
P2b	0.1712	0.9878	0.0567	1	0.0016	0.2099	1	0.0656	0.98	0.0017
P2c	0.1952	0.9998	0.0854	1	0.0021	0.2216	1	0.0879	0.99	0.0020
P2d	0.1934	0.9998	0.0799	1	0.0020	0.2208	1	0.0812	0.99	0.0020
M-Estimation	0.1746		0.0564			0.2125		0.0678
BART	0.3429	0.8546	0.2217			0.3385	0.9122	0.1799
Treed GP	0.2047	0.8349	0.0779			0.2611	0.8867	0.0899
n=80	MSPE	cov(y)	MISE	cov(η)	L	MSPE	cov(y)	MISE	cov(η)	L
P1	0.1636	1	0.0454	1	0.0018	0.1855	1	0.0346	1	0.0019
P2a	0.1509	0.9976	0.0373	0.95	0.0015	0.1653	1	0.0321	0.9952	0.0014
P2b	0.1578	0.9931	0.0324	1	0.0013	0.1614	1	0.0312	0.9932	0.0010
P2c	0.1589	0.9960	0.0404	1	0.0017	0.1774	1	0.0329	0.9980	0.0016
P2d	0.1567	0.9969	0.0401	1	0.0017	0.1770	1	0.0320	0.9969	0.0016
M-Estimation	0.1582		0.0364			0.1832		0.0325
BART	0.2284	0.9265	0.1098			0.2491	0.9490	0.1083
Treed GP	0.1655	0.8876	0.0427			0.2022	0.8923	0.0548

Table 3.

Simulation results under heteroscedastic residuals (Case (v))

n=40	MSPE	cov(y)	MISE	cov(η)	L
P1	0.6666	0.9800	0.5856	1	0.0033
P2a	0.5233	0.9770	0.3980	0.9812	0.0025
P2b	0.5231	0.9854	0.3745	0.9765	0.0019
P2c	0.5875	0.9850	0.4452	1	0.0029
P2d	0.5788	0.9859	0.4223	1	0.0028
M-Estimation	0.5531		0.3671
BART	0.4956	0.8980
Treed GP	0.7224	0.8123	0.6132
n=60	MSPE	cov(y)	MISE	cov(η)	L
P1	0.3828	1	0.2911	0.9985	0.0031
P2a	0.3745	0.9832	0.2617	0.9840	0.0022
P2b	0.3767	0.9812	0.2601	0.9867	0.0020
P2c	0.3810	0.9900	0.2800	0.9998	0.0027
P2d	0.3800	0.9906	0.2798	0.9998	0.0026
M-Estimation	0.3939		0.2824
BART	0.3930	0.9313	0.2668
Treed GP	0.4225	0.9023	0.3217
n=80	MSPE	cov(y)	MISE	cov(η)	L
P1	0.3599	0.9901	0.2759	0.9998	0.0029
P2a	0.3503	0.9762	0.2582	0.9765	0.0022
P2b	0.3519	0.9712	0.2545	0.9715	0.0019
P2c	0.3560	0.9856	0.2656	0.9885	0.0025
P2d	0.3557	0.9800	0.2677	0.9881	0.0024
M-Estimation	0.3905		0.2887
BART	0.3594	0.9442	0.2867
Treed GP	0.4489	0.9125	0.3509

Tables 1, 2 and 3 clearly show that in small samples both of the heteroscedastic methods (P2a and P2b) have substantially reduced MSPE and MISE relative to the heavy tailed parametric error model in most of the cases, interestingly even in the homoscedastic cases. This may be because discrete mixture of Gaussians better approximate a single normal than a t-distribution in small samples. Methods P2a and P2b also did a better job than method P1 in allowing uncertainty in estimating the mean regression and predicting the test sample observations. The homoscedastic versions 4 and 5 perform better than the parametric models but worse than the heteroscedastic models. In some cases, the heavy tailed t-residual distribution results in overly conservative predictive and credible intervals. As seen from the value of the influence statistic, the heteroscedastic PSB process mixtures result in more robust inference compared to the parametric error model, the sPSB process mixture of normals being more robust than the symmetric and unimodal version. As the sample size increases, the difference in the predictive performances between the parametric and the nonparametric models is reduced and in some cases the parametric error model performs as well as the nonparametric approaches, which is as expected given the Central Limit Theorem.

Table 1 shows that, in the simple linear model with normal homoscedastic errors, all the models perform similarly in terms of mean squared prediction error, though the methods P2a and P2b are somewhat better than the rest. Also, in estimating the mean regression function in case (i), methods P2a and P2b performed better than all the other methods. In case (ii)(Table 1), methods P2a and P2b are most robust in terms of estimation and prediction in presence of outliers. However, there is no significant difference between methods P2a and P2b and methods P2c and P2d in cases (i) and (ii). In cases (iii) and (iv), when the residual distribution is heteroscedastic, methods P2a and P2b perform significantly better than the parametric model P1 and the homoscedastic models P2c and P2d in both estimation and prediction, since the heteroscedastic PSB mixture is very flexible in modeling the residual distribution. This is quite evident from the MSPE values under cases (iii) and (iv) in Table 2. Huber’s M-Estimation method performs similarly to methods P2a–d in cases (i) and (ii) but did not do as well in estimation and prediction in cases (iii) and (iv) when the underlying mean function is actually non-linear with heteroscedastic residual distribution. Also BART failed to perform well in estimating the mean function in small samples in these cases. On the other hand, GP based approaches perform quite well in these cases in estimating the regression function with methods P2a and P2b performing better than the rest. Treed GP performed close to method P1 in estimation and prediction as both the methods are based on GP priors on the mean function and have a parametric error distribution. In not allowing heteroscedastic error variance, BART and Treed GP underestimate uncertainty in prediction, leading to overly narrow predictive intervals.

In case (v)(Table 3), where the true model is generated using comparatively less number of true signals, BART performed slightly better in terms of prediction than other methods in small samples. However, as the sample size increased, BART performed poorly while the GP prior on the mean can accommodate the non-linearity resulting in substantially better predictive performances.

7 Applications

7.1 Boston housing data application

To compare our proposed approaches to alternatives, we applied the methods to a commonly used data set from the literature, the Boston housing data. The response is the median value of the owner-occupied homes (measured in 1000$) in 506 census tracts in the Boston area, and there are 13 predictors (12 continuous, 1 binary) that might help to explain the variation in the median value across tracts. We predict the median value of the owner occupied homes of which the first 253 is taken as the training set and the remaining 253 as the test set. Out of sample predictive performance of our three methods is compared to competitors in Table 4. The parametric model P1, and the mixture models P2a–d and the M-Estimation methods perform very closely to each other in terms of prediction and did better than BART and Treed GP. Methods P1 and P2a even perform slightly better than method P2b, P2c and P2d. As in the simulation examples, BART and Treed GP underestimate the uncertainty in prediction. On the other hand, the predictive intervals of the methods P1, P2a–d are more conservative and accommodate uncertainty in predicting regions with outliers quite flexibly. Also the model P2b appears to be more robust compared to models P1, P2a, P2c & P2d in terms of the influence measure.

Table 4.

Boston housing data and body fat data results

Boston housing data					body fat data
Methods	MSPE	cov(y)	L	corr(Ytest, Ypred)a	MSPE	cov(y)	L	corr(Ytest, Ypred)
P1	0.0012	0.99	0.0034	0.9894	0.0055	1	0.0020	0.9972
P2a	0.0013	0.99	0.0027	0.9901	0.0031	1	0.0017	0.9984
P2b	0.0016	0.99	0.0020	0.9863	0.0029	1	0.0017	0.9989
P2c	0.0014	0.99	0.0030	0.9879	0.0034	1	0.0019	0.9969
P2d	0.0013	0.99	0.0029	0.9881	0.0032	1	0.0018	0.9975
M-Estimation	0.0016			0.9858	0.0375			0.9710
BART	0.0024	0.92		0.9836	0.0355	0.95		0.9655
Treed GP	0.0053	0.91		0.9524	0.1526	0.98		0.9250

^acorr(Ytest, Ypred) denotes the sample correlation between the test and predicted y

7.2 Body fat data application

With the increasing trend in obesity and concerns about associated adverse health effects, such as heart disease and diabetes, it has become even more important to obtain accurate estimates of body fat percentage. It is well known that body mass index, which is calculated based only on weight and height, can produce a misleading measure of adiposity as it does not take into account muscle mass or variability in frame size. As a gold standard for measuring percentage of body fat, one can rely on under water weighing techniques, and age and body circumference measurements have also been widely used as additional predictors. We consider a commonly-used data set from Statlib (http://lib.stat.cmu.edu/datasets/bodyfat), which contains the following 15 variables; percentage of body fat(%), body density from underwater weighing (gm/cm³), age (year), weight (lbs.), height (inches), and ten body circumferences (neck, chest, abdomen, hip, thigh, knee, ankle, biceps, forearm, wrist, all in cm). Percentage of body fat is given from Siri’s (1956) equation:

$$\text{Percentage of body Fat}=\frac{495}{\text{Density}}-450$$

We predict the percentage of body fat(%) taking the first 126 as the training set and the remaining 126 as the test set. We summarize the predictive performances in Table 4.

Table 4 suggests that the nonparametric regression procedures with heteroscedastic residual distribution P2a and P2b perform better than the parametric models P1 and models P2c and P2d, BART, M-Estimation and Treed GP in predicting the percentage of body fat.

8 Discussion

We have developed a novel regression model that can accommodate a large range of non-linearity in the mean function and at the same time can flexibly deal with outliers and heteroscedasticity. Based on preliminary simulation results, it appears that our methods P2a and P2b can outperform contemporary nonparametric regression methods, such as Huber’s M-Estimation method, BART and treed Gaussian processes. We also provide theoretical support for the proposed methodology when both the mean and the residuals are modeled nonparametrically.

One possible future direction is to relax the symmetry assumption on the residual distribution and introduce a model for median regression based on conditional PSB mixtures for allowing possibly asymmetric residual densities constrained to have zero median. Conditional DP mixtures are well known in the literature (Doss 1985); Burr and Doss 2005) and it is certainly interesting to extend our approach via a conditional PSB. In that way we can hope to obtain a more robust estimate of the regression function. It is challenging to extend our theoretical results to conditional PSB and develop a fast algorithm for computation. Another possible theoretical direction is to prove posterior consistency using heteroscedastic mixtures. Currently we only have results for the homoscedastic PSBP mixture.

A Proofs of main results

Proof of Lemma 1

It follows from Chu (1973) that

$$f\in {𝒞}_m\iff f(x)=\int {\sigma }^{-1}\varphi ({\sigma }^{-1}x)g(\sigma )d\sigma$$

for some density g on ℝ⁺. Recall from Ongaro and Cattaneo (2004) that a collection of random weights ${\{{\pi }_h\}}_{h=1}^{\infty }$ with ${\sum }_{h=1}^{\infty }{\pi }_h=1$ a.s. is said to have a full support if for any m ≥ 1, (π₁; …, π_m) admits a positive joint density with respect to Lebesgue measure on the simplex $\{(p_1,\dots ,p_m):{\sum }_{i=1}^m{p}_i\le 1\}$. Ongaro and Cattaneo (2004) showed that if π_h’s have a full support, the weak support of

$$P=\sum _{h=1}^{\infty }{\pi }_h{\delta }_{{\theta }_h},{\theta }_h~G_0$$

is the set of all probability measures whose support is contained the support of G₀. Since

$${({\pi }_1,\dots ,{\pi }_m)}^{\underset{=}{d}}(\Theta ({\alpha }_1),\Theta ({\alpha }_2)\{1-\Theta ({\alpha }_1)\},\dots ,\Theta ({\alpha }_m)\prod _{i=1}^{m-1}\{1-\Theta ({\alpha }_i)\}),{\alpha }_i~\mathrm{N}({\mu }_{\alpha },{\sigma }_{\alpha }^2),$$

π_h’s have a full support and hence the weak support of $P={\sum }_{h=1}^{\infty }{\pi }_h{\delta }_{{\tau }_h}$ defined in (7) is all probability measures on ℝ⁺. It follows that the weak support of the induced prior Π_u on 𝒮_u, denoted by wk(Π_u), is precisely 𝒞_m.

Proof of Lemma 2

It follows from Tokdar (2006) that if we can show that the weak support of Π_s contains all probability measures symmetric about zero and having compact support, then f ∈ S̃_s ⟹ f ∈ K L(Π_s). The argument given in Lemma 1 shows that the weak support of the PSB prior in (4) is the set of all probability measures on ℝ × ℝ⁺. Now we will show that an arbitrary P^s is in a weak neighborhood of P^s if P̃ is in a weak neighborhood of P. We state a lemma to prove our claim.

Lemma 4 Let P̃_n be a sequence of probability measures and P̃ be a fixed probability measure. Then $({\tilde{P}}_n\Rightarrow \tilde{P})\Rightarrow ({\tilde{P}}_n^s\Rightarrow {\tilde{P}}^s)$, with ${\tilde{P}}_n^s$ and P̃^s the symmetrised versions of P̃_n and P̃, respectively, where the symmetrizing operation is as defined in (9).

Proof Assume P̃_n ⟹ P̃. We have to show that for any bounded function ϕ on ℝ × ℝ⁺,

$$\int \varphi (t,\tau )d{\tilde{P}}_n^s(t,\tau )\to \int \varphi (t,\tau )d{\tilde{P}}^s(t,\tau )\text{as}n\to \infty .$$

Now,

$$\int \varphi (t,\tau )d{\tilde{P}}_n^s(t,\tau )=\frac{1}{2}\int \varphi (t,\tau )d{\tilde{P}}_n(t,\tau )+\frac{1}{2}\int \varphi (t,\tau )d{\tilde{P}}_n(-t,\tau )=\int \frac{1}{2}\{\varphi (t,\tau )+\varphi (-t,\tau )\}d{\tilde{P}}_n(t,\tau ).$$

Since $\psi (t,\tau )=\frac{1}{2}\{\varphi (t,\tau )+\varphi (-t,\tau )\}$ is also a bounded continuous function and P̃_n ⟹ P̃,

$$\int \frac{1}{2}\{\varphi (t,\tau )+\varphi (-t,\tau )\}d{\tilde{P}}_n(t,\tau )\to \int \frac{1}{2}\{\varphi (t,\tau )+\varphi (-t,\tau )\}d\tilde{P}(t,\tau )=\int \varphi (t,\tau )d{\tilde{P}}^s(t,\tau )$$

as n → ∞. This completes the proof of Lemma 4.

Lemma 4 in fact shows that the weak support of Π_s contains all probability measures symmetric about zero. With an appeal to Tokdar (2006), f ∈ S̃_s ⟹ f ∈ K L(Π_s).

Proof of Theorem 1

In order to prove the theorem we need the following variant of Theorem 2.1 of Amewou-Atisso et al (2003) and Theorem 1 of Choi and Schervish (2007) which we state as Lemma 5. Existence of exponentially consistent tests is a typical tool in showing strong consistency.

Definition 2 Let 𝒲 ⊂ 𝒮̃_s × ℱ. A sequence of test functions ${\Theta }_n({\{y_i,{\mathbf{x}}_i\}}_{i=1}^n)$ is said to be exponentially consistent for testing

$$H_0:(f,\eta )=(f_0,{\eta }_0)\text{against}{H}_1:(f,\eta )\in {𝒲}_n$$

if there exists constants C₁, C₂, C > 0 such that

1. ${𝖤}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}({\Theta }_n)\le C_1{e}^{-nC},$

2. ${\text{inf}}_{(f,\eta )\in {𝒲}_n}{𝖤}_{{\mathrm{\Pi }}_{i=1}^n{f}_{\eta i}}({\Theta }_n)\ge 1-C_2{e}^{-nC}.$

Lemma 5 Let Π̃ = (Π_s × π) be the prior on S̃_s × ℱ. Let U_n be a sequence of subsets of S̃_s × ℱ. Suppose that there exists test functions ${\{{\Theta }_n\}}_{n=1}^{\infty }$, sets Θ_n ⊂ S̃_s × ℱ, n ≥ 1 and constants C₁, C₂, c₁, c₂ > 0 such that

1. ${\sum }_{n=1}^{\infty }{E}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}{\Theta }_n<\infty .$

2. ${\text{sup}}_{(f,\eta )\in U_n^c\cap {\Theta }_n}{E}_{{\mathrm{\Pi }}_{i=1}^n{f}_{\eta i}}(1-{\Theta }_n)\le C_1{e}^{-c_{1}n}.$

3. $\tilde{\Pi }({\Theta }_n^c)\le C_2{e}^{-c_{2}n}.$

4. For all δ > 0 and for almost every data sequence ${\{y_i,{\mathbf{x}}_i\}}_{i=1}^{\infty }$,

   $$\tilde{\Pi }\{(f,\eta ):K_i(f,\eta )<\delta {\forall }_i,\sum _{i=1}^{\infty }\frac{V_i(f,\eta )}{i^2}<\infty \}>0.$$

Then $\tilde{\Pi }\{(f,\eta )\in U_n^c|(Y_1,{\mathbf{x}}_1),\dots ,(Y_n,{\mathbf{x}}_n)\}\to 0a.s.[P_{f_0,{\eta }_0}]$.

In this case U_n = 𝒲_n = 𝒰 × S_n (f₀, Δ) ∀ n ≥ 1. As in van der Vaart and van Zanten (2009), we construct Θ_n = ℱ × Θ_1n where ${\Theta }_{1n}={\cup }_{a<r_n}{M}_n{ℍ}_1^a+\epsilon {𝔹}_1$ where ℍ₁ and 𝔹₁ are unit ball of the RKHS of W^a and unit ball of the Banach space of C[0, 1]^p respectively, r_n, M_n are increasing sequences to be chosen later. The nth test is constructed by combining a collection of tests one for each of the finitely many elements of Θ_n. It follows from the proof of Theorem 3.1 in van der Vaart and van Zanten (2009) that under Assumption 1, there exists constants d₁, d₂, K > 0 such that

1. $\tilde{\Pi }({\Theta }_n^c)\le \text{exp}\{-d_1{r}_n^p{\text{log}}^q(r_n)\}+\text{exp}\{-M_n^2/8\}.$

2. $\text{log}N(\epsilon ,{\Theta }_{1n},{\Vert ·\Vert }_{\infty })\le K{r}_n^p{\left(\text{log}\frac{M_n}{\epsilon }\right)}^{p+1}.$

Choosing M_n = O(n^1/2), $r_n^p=O(n/{(\text{log}n)}^{p+2})$, we observe that

1. $\tilde{\Pi }({\Theta }_n^c)\le \text{exp}\{-d_{2}n\}.$

2. log N (ε, Θ_1n, ‖·‖_∞) = o(n).

for some constant d₂ > 0.

In order to verify 1 and 2 of Lemma 5, we will write 𝒲_n as a disjoint union of two easily tractable regions. The particular form of 𝒲_n that is of interest to us is 𝒲_1n ∪ 𝒲_2n, where for any Δ > 0,

$${𝒲}_{1n}={𝒰}^c\times \{\eta :{\Vert \eta -\eta \Vert }_{1,n}\le \Delta \}{𝒲}_{2n}=\{(f,\eta ):{\Vert \eta -\eta \Vert }_{1,n}>\Delta \}.$$

We will establish the existence of a consistent sequence of tests for each of these regions by considering the following variants of Proposition 3.1 and Proposition 3.3 of Amewou-Atisso et al (2003).

Proposition 1 There exists an exponentially consistent sequence of tests for

$$H_0:(f,\eta )=(f_0,{\eta }_0)\mathit{\text{against}}{H}_1:(f,\eta )\in {𝒲}_{2n}\cap {\Theta }_n.$$

Proof Let 0 < t < Δ/2 and assume N_t = N(t, Θ_1n, ‖·‖_∞). Let η¹, …, η^N_t ∈ Θ_1n be such that for each η ∈ Θ_1n there exists j such that ‖η − η^j‖_∞ < t. If ‖η − η₀‖_1,n > Δ, ‖η^j − η₀‖_1,n > Δ/2. It follows from Lemma 3.2 Amewou-Atisso et al (2003) that there exists a set $A_i^j$ and a constant C > 0 depending on f₀ such that ${\alpha }_i^j≔P_{f_{0_i}}(A_i^j)\le \frac{1}{2}-C|{\eta }^j({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)|$. and ${\gamma }_j^i≔P_{f_{\eta }{j}_i}(A_i)\ge \frac{1}{2}$. If i ≤ n and i ∉ K_n, set A_i = ℝ, so that ${\alpha }_i^j={\gamma }_i^j=1$. Thus

$$\underset{n\to \infty }{\text{lim inf}}\frac{1}{n}\sum _{i=1}^n({\gamma }_i^j-{\alpha }_i^j)\ge C\Delta /2$$

From Lemma 3.1 and Lemma 3.2 of Amewou-Atisso et al (2003), it follows that there exist test functions ${\Theta }_n^j$ based on $\{I_{A_i^j},i=1,\dots ,n\}$ such that ${\mathrm{E}}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}{\Theta }_n^j<e^{-n{C}_1}$ and ${\mathrm{E}}_{{\mathrm{\Pi }}_{i=1}^n{f}_{\eta }{j}_i}(1-{\Theta }_n^j)<e^{-n{C}_2}$ for constants C₁, C₂ > 0 Now define ${\Theta }_n={\text{max}}_{1\le j\le N_t}{\Theta }_n^j$.

Then

$${\mathrm{E}}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}{\Theta }_n\le \sum _{j=1}^{N_t}{\mathrm{E}}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}{\Theta }_n^j\le \sum _{j=1}^{N_t}{e}^{-n{C}_1}\le N_t{e}^{-n{C}_1}\le e^{-n{C}_3}.$$

for some constant C3 > 0. Clearly ${\sum }_{n=1}^{\infty }{\mathrm{E}}_{{\mathrm{\Pi }}_{i=1}^n{f}_{0_i}}{\Theta }_n<\infty$.

Next we consider the type II error probability. The type II error probability of Θ_n is no larger than the type II error probability of any of the $\{{\Theta }_n^j,j=1,\dots ,N_t\}$ and hence exponentially small.

Proposition 2 There exists an exponentially consistent sequence of tests for

$$H_0:(f,\eta )=(f_0,{\eta }_0)\mathit{\text{against}}{H}_1:(f,\eta )\in {𝒲}_{1n}$$

Proof Without loss of generality take

$$𝒰=\{f:\int \Theta (y)f(y)dy-\int \Theta (y)f_0(y)dy<\epsilon \}$$

where 0 ≤ Θ ≤ 1 and Θ is Lipschitz continuous. Hence there exists M > 0 such that |Θ(y₁) − Θ(y₂)| < M|y₁ − y₂|. Set Θ̃_i(y) = Θ{y − η₀(x_i)}. Notice that 𝖤_f0i Θ̃_i = 𝖤_f0Θ. Now

$${𝖤}_{f_{\eta i}}{\tilde{\Theta }}_i=\int {\tilde{\Theta }}_i(y)f_{\eta i}(y)dy=\int \Theta (y)f[y-\{\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)\}]\ge \int \Theta [y-\{\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)\}]f[y-\{\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)\}]dy-\int |\Theta (y)-\Theta [y-\{\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)\}]|f[y-\{\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)\}]dy\ge \int \Theta (y)f(y)dy-M|\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)|\ge {𝖤}_{f_0}\Theta +\epsilon -M|\eta ({\mathbf{x}}_i)-{\eta }_0({\mathbf{x}}_i)|$$

Hence $1/n{\sum }_{i=1}^n{𝖤}_{f_{\eta i}}{\tilde{\Theta }}_i\ge E_{f_0}\Theta +\epsilon -M\Delta$ for any f ∈ U^c. Now choosing Δ < ε/M and applying Lemma 3.1 of Amewou-Atisso et al (2003) we complete the proof.

It remains to verify the second sufficient condition of Theorem 1. Under the assumptions, it follows from Lemma 2 that f₀ ∈ K L(Π_s). We will present an important lemma which is similar to Lemma 5.1 of Tokdar (2006). It guarantees that K(f₀, f_θ) and V(f₀, f_θ) are continuous at θ = 0. First we state and prove some properties of the prior Π_s described in (9) which will be used to prove the lemma.

Lemma 6 If Π_s is the prior described in (9) and $P_0(t,\tau )=N(t;{\mu }_0,{\sigma }_0^2)\times Ga(\tau ;{\alpha }_{\tau },{\beta }_{\tau })$, with α_τ > 0 and β_τ > 0. Then,

$$\int \tau d{P}^s(t,\tau )<\infty a.s.,\int t^{2}d{P}^s(t,\tau )<\infty a.s.,\int \tau t^{2}d{P}^s(t,\tau )<\infty a.s.,-\infty <\int (\text{log}\tau )d{P}^s(t,\tau )<\infty a.s.$$ (11)

Proof

$$\int {\int }_{\tau >0,t\in ℝ}\tau d{P}^s(t,\tau )dP={\int }_{\tau >0,t\in ℝ}\tau \int d{P}^s(t,\tau )dP=\frac{1}{2}{\int }_{\tau >0,t\in ℝ}\tau \mathrm{N}(t;{\mu }_0,{\sigma }_0^2)\mathrm{G}\mathrm{a}(\tau ;{\alpha }_{\tau },{\beta }_{\tau })dtd\tau +\frac{1}{2}{\int }_{\tau >0,t\in ℝ}\tau \mathrm{N}(t;-{\mu }_0,{\sigma }_0^2)\mathrm{G}\mathrm{a}(\tau ;{\alpha }_{\tau },{\beta }_{\tau })dtd\tau ={\int }_{\tau >0}\tau \mathrm{G}\mathrm{a}(\tau ;{\alpha }_{\tau },{\beta }_{\tau })d\tau <\infty .$$

The proofs of ∫ t²dP^s(t,τ) < ∞ a.s. and ∫ τt²dP^s(t,τ) < ∞ a.s. are similar. Since α_τ > 0, choose an integer m large enough such that ${\alpha }_{\tau }>\frac{1}{m}$.

$$\int {\int }_{\tau >0,t\in ℝ}(\text{log}\tau )d{P}^s(t,\tau )dP={\int }_{\tau >0}(\text{log}\tau )\mathrm{G}\mathrm{a}(\tau ;{\alpha }_{\tau },{\beta }_{\tau })d\tau =C{\int }_{\tau >0}(\text{log}\tau ){\tau }^{{\alpha }_{\tau }-1}{e}^{-{\beta }_{\tau }\tau }d\tau =C{\int }_{\tau >0}({\tau }^{1/m}\text{log}\tau ){\tau }^{{\alpha }_{\tau }-\frac{1}{m}-1}{e}^{-{\beta }_{\tau }\tau }d\tau >-\infty$$

since τ^1/m log τ is bounded in [0, 1]. Also ∫_τ>0 (log τ)τ^α_τ−1 e^−β_ττ dτ ≤ ∫_τ>0 ττ^α_τ−1 e^−β_ττ dτ < ∞.

Lemma 7 Under the conditions of the Theorem 1, if f(·) = ∫ N(·; t, τ⁻¹)dP^s(t,τ) and f_θ(y) = f(y − θ), then

1. ${\text{lim}}_{\theta \to 0}\int f_0(y)\text{log}\frac{f_0(y)}{f_{\theta }(y)}dy=\int f_0(y)\text{log}\frac{f_0(y)}{f(y)}dy.$

2. ${\text{lim}}_{\theta \to 0}\int f_0(y){({\text{log}}_{+}\frac{f_0(y)}{f_{\theta }(y)})}^{2}dy=\int f_0(y){({\text{log}}_{+}\frac{f_0(y)}{f(y)})}^{2}dy.$

Proof Clearly τϕ{τ(y−θ−t)} → τϕ{τ(y−t)} as θ → 0. Since $\int \tau \varphi \{\tau (y-\theta -t)\}d{P}^s(t,\tau )\le \frac{1}{\sqrt{2\pi }}\int \tau d{P}^s(t,\tau )<\infty$, so by DCT f_θ(y) → f(y) as θ → 0. Hence

$$\text{log}\frac{f_0(y)}{f_t(y)}\to \text{log}\frac{f_0(y)}{f(y)}\text{as}t\to 0{\left({\text{log}}_{+}\frac{f_0(y)}{f_t(y)}\right)}^2\to {\left({\text{log}}_{+}\frac{f_0(y)}{f(y)}\right)}^2\text{as}t\to 0.$$

To apply DCT again, we have to bound the function |log f_θ(y)|by an integrable function.

$$|\text{log}{f}_{\theta }(y)|\le \text{log}\sqrt{2\pi }+|\text{log}\int \tau e^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )|.$$

Let c = ∫ τdP^s(t,τ) < ∞. Then

$$|\text{log}\int \tau e^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )|\le |\text{log}c|+|\text{log}\int \frac{\tau }{c}{e}^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )|.$$

Now since $\int \tau e^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )\le c,|\text{log}\int \frac{\tau }{c}{e}^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )|=-\text{log}\int \frac{\tau }{c}{e}^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )$. Hence, by Jensen’s inequality applied to −log x, we get, $-\text{log}\int \frac{\tau }{c}{e}^{-\frac{\tau }{2}{(y-t-\theta )}^2}d{P}^s(t,\tau )\le \text{log}c-\int (\text{log}\tau )d{P}^s(t,\tau )+\frac{1}{2}\int \tau {(y-t-\theta )}^{2}d{P}^s(t,\tau )$.

Now since θ → 0, w.l.o.g assume |θ| ≤ 1. Hence

$$\int \tau {(y-t-\theta )}^{2}d{P}^s(t,\tau )\le 4(y^2\int \tau d{P}^s(t,\tau )+\int \tau t^{2}d{P}^s(t,\tau )+1)\Rightarrow |\text{log}{f}_{\theta }(y)|\le \text{log}\sqrt{2\pi }+|\text{log}c|+\text{log}c-\int (\text{log}\tau )d{P}^s(t,\tau )+2(y^2\int \tau d{P}^s(t,\tau )+\int \tau t^{2}d{P}^s(t,\tau )+1)$$

which is clearly f₀-integrable according to the assumptions of the lemma and from the properties of Π_s proved in Lemma 6. Similarly |log f_θ(y)|² can be bounded by an f₀-integrable function. The conclusion of the lemma follows from a simple application of DCT.

Lemma 2 together with the assumption (2) of the Theorem 1 guarantees Π{f : K(f₀, f) < δ, V(f₀, f) < ∞} > 0 for all δ > 0. Since (11) holds, we may assume

$$\Pi (𝒰)>0,\text{where}𝒰=\{f:K(f_0,f)<\delta ,V(f_0,f)<\infty ,(11)\text{holds}\}.$$ (12)

Now for every f(·) = ∫ N(·; t, τ⁻¹)dP^s(t,τ) ∈ 𝒰, using Lemma 7, choose δ_f such that for |θ| < δ_f,

$$K(f_0,f_{\theta })<2K(f_0,f),V(f,f_{\theta })<2V(f_0,f).$$

Now if ‖η − η₀‖ < δ_f, |η(x_i) − η₀(x_i)| < δ_f, for i = 1, …, n. So if f ∈ 𝒰 and ‖η − η₀‖ < δ_f, we have

$$K_i(f,\eta )=\int f_{0_i}\text{log}\frac{f_{0_i}}{f_{\eta i}}=\int f_0\text{log}\frac{f_0}{f_{(\eta -{\eta }_0)i}}<2K(f_0,f),V_i(f,\eta )=\int f_{0_i}{({\text{log}}_{+}\frac{f_{0_i}}{f_{\eta i}})}^2=\int f_0{({\text{log}}_{+}\frac{f_0}{f_{(\eta -{\eta }_0)i}})}^2<2V(f_0,f).$$

From (12) and Lemma 3 we have,

$$\Pi \{(f,\eta ):f\in 𝒰,\Vert \eta -{\eta }_0\Vert <{\delta }_f\}>0.$$

Hence

$$\Pi \{(f,\eta ):K_i(f,\eta )<2\delta {\forall }_i,\sum _{i=1}^{\infty }\frac{V_i(f,\eta )}{i^2}<\infty \}>0.$$

This ensures weak consistency of the posterior of the residual density and strong consistency of the posterior of the regression function η.