Locally Adaptive Bayes Nonparametric Regression via Nested Gaussian Processes

Abstract

We propose a nested Gaussian process (nGP) as a locally adaptive prior for Bayesian nonparametric regression. Specified through a set of stochastic differential equations (SDEs), the nGP imposes a Gaussian process prior for the function’s mth-order derivative. The nesting comes in through including a local instantaneous mean function, which is drawn from another Gaussian process inducing adaptivity to locally-varying smoothness. We discuss the support of the nGP prior in terms of the closure of a reproducing kernel Hilbert space, and consider theoretical properties of the posterior. The posterior mean under the nGP prior is shown to be equivalent to the minimizer of a nested penalized sum-of-squares involving penalties for both the global and local roughness of the function. Using highly-efficient Markov chain Monte Carlo for posterior inference, the proposed method performs well in simulation studies compared to several alternatives, and is scalable to massive data, illustrated through a proteomics application.

1 Introduction

We consider the nonparametric regression problem

$$Y(t)=U(t)+\epsilon (t),t\in \mathcal{T}=[t_0,t_U],$$ (1)

where U: → ℝ is an unknown mean regression function to be estimated at = {t₀, t₁, t₂, …, t_J < t_U}, t₀ = 0, and $\mathit{\epsilon }={\{\epsilon (t_1),\epsilon (t_2),\cdots ,\epsilon (t_J)\}}^{\prime }~{\mathrm{N}}_J(\mathbf{0},{\sigma }_{\epsilon }^2\mathit{I})$ a J-dimensional multivariate normal distribution with mean vector 0 and covariance matrix ${\sigma }_{\epsilon }^2\mathit{I}$. We are particularly interested in allowing the smoothness of U to vary locally as a function of t. For example, consider the protein mass spectrometry data in panel (a) of Figure 1. There are clearly regions of t across which the function is very smooth and other regions in which there are distinct spikes, with these spikes being quite important. An additional challenge is that the data are generated in a high-throughput experiment with J = 11, 186 observations. Hence, we need a statistical model which allows locally-varying smoothness, while also permitting efficient computation even when data are available at a large number of locations along the function.

Figure 1.

(a) Plot of protein mass spectrometry data: observed intensities versus mass to charge ratio m/z; (b) Posterior mean ( ) and 95% credible interval of U (red shades); (c) Posterior mean and 95% credible interval of U for a local region; (d) Posterior mean and 95% credible interval of rate of intensity changes DU.

A commonly used approach for nonparametric regression is to place a Gaussian process (GP) prior (Neal, 1998; Rasmussen and Williams, 2006; Shi and Choi, 2011) on the unknown U, where the GP is usually specified by its mean and covariance function (e.g. squared exponential). The posterior distribution of U( ) can be conveniently obtained as a multivariate Gaussian distribution. When carefully-chosen hyperpriors are placed on the parameters in the covariance kernel, GP priors have been shown to lead to large support, posterior consistency (Ghosal and Roy, 2006; Choi and Schervish, 2007) and even near minimax optimal adaptive rates of posterior contraction (Van der Vaart and Van Zanten, 2008a). However, the focus of this literature has been on isotropic Gaussian processes, which have a single bandwidth parameter controlling global smoothness, with the contraction rate theory assuming the true function has a single smoothness level. There has been applied work allowing the smoothness of a multivariate regression surface to vary in different directions by using predictor-specific bandwidths in a GP with a squared exponential covariance (Savitsky et al., 2011; Zou et al., 2010). Bhattacharya, Pati, and Dunson (2011) recently showed that a carefully-scaled anisotropic GP leads to minimax optimal adaptive rates in anisotropic function classes including when the true function depends on a subset of the predictors. However, the focus was on allowing a single smoothness level for each predictor, while our current interest is allowing smoothness to vary locally in nonparametric regression in a single predictor.

There is a rich literature on locally-varying smoothing. One popular approach relies on free knot splines, for which various strategies have been proposed to select the number of knots and their locations, including stepwise forward and/or backward knots selection (Friedman and Silverman, 1989; Friedman, 1991; Luo and Wahba, 1997), accurate knots selection scheme (Zhou and Shen, 2001) and Bayesian knots selection (Smith and Kohn, 1996; Denison et al., 1998; Dimatteo et al., 2001) via Gibbs sampling (George and McCulloch, 1993) or reversible jump Markov chain Monte Carlo (Green, 1995). Although many of these methods perform well in simulations, such free knot approaches tend to be highly computationally demanding making their implementation in massive data sets problematic.

In addition to free knot methods, adaptive penalization approaches have also been proposed. An estimate of U is obtained as the minimizer of a penalized sum of squares including a roughness penalty with a spatially-varying smoothness parameter (Wahba, 1995; Ruppert and Carroll, 2000; Pintore et al., 2006; Crainiceanu et al., 2007). Other smoothness adaptive methods include wavelet shrinkage (Donoho and Johnstone, 1995), local polynomial fitting with variable bandwidth (Fan and Gijbels, 1995), L-spline (Abramovich and Steinberg, 1996; Heckman and Ramsay, 2000), mixture of splines (Wood et al., 2002) and linear combination of kernels with varying bandwidths (Wolpert et al., 2011). Additionally, adaptive penalization approaches have been applied to non-Gaussian data (Krivobokova et al., 2008; Wood et al., 2008; Scheipl and Kneib, 2009). The common theme of these approaches is to reduce the constraint on the single smoothness level assumption and to implicitly allow the derivatives of U, a common measurement of the smoothness of U, to vary over t.

In this paper, we instead propose a nested Gaussian process (nGP) prior to explicitly model the expectation of the derivative of U as a function of t and to make full Bayesian inference using an efficient Markov chain Monte Carlo (MCMC) algorithm scalable to massive data. More formally, our nGP prior specifies a GP for U’s mth-order derivative D^mU centered on a local instantaneous mean function A: → ℝ which is in turn drawn from another GP. Both GPs are defined by stochastic differential equations (SDEs), related to the method proposed by Zhu et al. (2011). However, Zhu et al. (2011) centered their process on a parametric model, while we instead center on a higher-level GP to allow nonparametric locally-adaptive smoothing. Along with the observation equation (1), SDEs can be reformulated as a state space model (Durbin and Koopman, 2001). This reformulation facilitates the application of simulation smoother (Durbin and Koopman, 2002), an efficient MCMC algorithm with (J) computational complexity which is essential to deal with large scale data. We will show that the nGP prior has large support and its posterior distribution is asymptotically consistent. In addition, the posterior mean or mode of U under the nGP prior can be shown to correspond to the minimizer of a penalized sum of squares with nested penalty functions.

The remainder of the paper is organized as follows. Section 2 defines the nGP prior and discusses some of its properties. Section 3 outlines an efficient Markov chain Monte Carlo (MCMC) algorithm for posterior computation. Section 4 presents simulation studies. The proposed method is applied to a mass spectra dataset in Section 5. Finally, Section 6 contains several concluding remarks and outlines some future directions.

2 Nested Gaussian Process Prior

2.1 Definition and Properties

The nGP defines a GP prior for the mean regression function U and the local instantaneous mean function A through the following SDEs with parameters σ_U ∈ ℝ⁺ and σ_A ∈ ℝ⁺:

$$D^{m}U(t)=A(t)+{\sigma }_U{\stackrel{.}{W}}_U(t),m\in \mathbb{N}\ge 2$$ (2)

$$D^{n}A(t)={\sigma }_A{\stackrel{.}{W}}_A(t),n\in \mathbb{N}\ge 1$$ (3)

where Ẇ_U (t) and Ẇ_A(t) are two independent Gaussian white noise processes with mean function E{Ẇ_U (t)} = E{Ẇ_A(t)} = 0 and covariance function E{Ẇ_U (t)Ẇ_U (t′)} = E{Ẇ_A(t)Ẇ_A(t′)} = δ(t − t′) a delta function; $D^m=\frac{d^m}{{dt}^m}$ an mth-order differential operator. The initial value of U and its derivatives up to order m − 1 at t₀ = 0 are denoted as $\mathit{\mu }={({\mu }_0,{\mu }_1,\cdots ,{\mu }_{m-1})}^{\prime }~{\mathrm{N}}_m(\mathbf{0},{\sigma }_{\mu }^2\mathit{I})$. Similarly, the initial values of A and its derivatives till order n − 1 at t₀ = 0 are denoted as $\mathit{\alpha }={({\alpha }_0,{\alpha }_1,\cdots ,{\alpha }_{n-1})}^{\prime }~{\mathrm{N}}_n(\mathbf{0},{\sigma }_{\alpha }^2\mathit{I})$. In addition, we assume that μ, α, Ẇ_U (·) and Ẇ_A(·) are mutually independent. The definition of nGP naturally induces a prior for U with varying smoothness. Indeed, the SDE (2) suggests that E{D^mU(t) | A(t)} = A(t). Thus, the smoothness of U, measured by D^mU, is expected to be centered on a function A varying over t.

We first recall the definition of the reproducing kernel Hilbert space (RKHS) generated by the zero-mean Gaussian process W = {W(t): t ∈ } and the results on the support of W, which will be useful to explore the theoretical properties of the nGP prior. Let (Ω, , P) be the probability space for W such that for any t₁, t₂,…, t_k ∈ with k ∈ ℕ, {W(t₁), W(t₂),…, W (t_k)}′ follow a zero-mean multivariate normal distribution with covariance matrix induced through the covariance function : × → ℝ, defined by (s, t) = E{W (s)W (t)}. The RKHS generated by W is the completion of the linear space of all functions

$$t\mapsto \sum _{i=1}^k{a}_i{\mathcal{K}}_W(s_i,t),a_1,\dots ,a_k\in ℝ,s_1,\dots ,s_k\in \mathcal{T},k\in \mathbb{N},$$

with the inner product

$${\langle \sum _{i=1}^k{a}_i{\mathcal{K}}_W(s_i,·),\sum _{j=1}^l{b}_j{\mathcal{K}}_W(t_j,·)\rangle }_{{\mathcal{H}}_{{\mathcal{K}}_W}}=\sum _{i=1}^k\sum _{j=1}^l{a}_i{b}_j{\mathcal{K}}_W(s_i,t_j),$$

which satisfies the reproducing property f(t) = 〈f, (t, ·)〉 for any f ∈ : → ℝ.

With the specification of the RKHS , we are able to define the support of W as the closure of (Lemma 5.1, Van der Vaart and Van Zanten, 2008b). We apply this definition to characterize the support of the nGP prior, which is formally stated in Theorem 1.

Theorem 1

The support of nested Gaussian process U is the closure of RKHS = ⊕ ⊕ ⊕ , the direct sum of RKHSs , , and with reproducing kernels (s, t), (s, t), (s, t) and (s, t) respectively.

The proof is in Appendix A. It is of interest to note that the support of the Gaussian process Ũ = Ũ₀ + Ũ₁ as the prior for polynomial smoothing spline is the closure of RKHS = ⊕ with ⊂ (the proof is given in Appendix A). Hence, it is clear that the nGP prior includes GP prior for polynomial smoothing spline as a special case when ${\sigma }_{\alpha }^2\to 0$ and ${\sigma }_A^2\to 0$.

The nGP prior can generate functions U arbitrarily close to any function U₀ in the support of the prior. From Theorem 1 it is clear that the support is large and hence the sample paths from the proposed prior can approximate any function in a broad class. As a stronger property, it is also appealing that the posterior distribution concentrate in arbitrarily small neighborhoods of the true function U₀ which generated the data as the sample size J increases, with this property referred to as posterior consistency. More formally, a prior Π on Θ achieves posterior consistency at the true parameter θ₀ if for any neighborhoods , the posterior distribution Π ( | Y₁, Y₂,…, Y_J) → 1 almost surely under Π_θ0, the true joint distribution of observations ${\{Y_j\}}_{j=1}^J$. For our case, the parameters θ = (U, σ_ε) lie in the product space Θ = × ℝ⁺ and have a prior Π_θ = Π_U × Π_σε, for which Π_U is an nGP prior for U and Π_σε is a prior distribution for σ_ε. The L₁ neighborhood of θ₀ = (U₀, σ_ε,0) is defined as ${\mathcal{U}}_{\epsilon }=\left\{(U,{\sigma }_{\epsilon }):{\Vert U-U_0\Vert }_1={\int }_0^{t_U}\mid U(t)-U_0(t)\mid dt<\epsilon ,\mid {\sigma }_{\epsilon }-{\sigma }_{\epsilon ,0}\mid <\epsilon \right\}$.

Under those specifications and regularity conditions detailed in Appendix A, the results on strong posterior consistency for the Bayes nonparametric regression with nGP prior is given as follows.

Theorem 2

Let ${\{Y_j\}}_{j=1}^J$ be the independent but non-identical observations following normal distributions ${\{N_1(U(t_j),{\sigma }_{\epsilon }^2)\}}_{j=1}^J$ with unknown mean function U and unknown ${\sigma }_{\epsilon }^2$ at design points t₁, t₂,…t_J. Suppose U follows an nGP prior and the Assumptions 1,2 and 3 hold. Then for every θ₀ ∈ Θ and every ε > 0,

$$\mathrm{\Pi }({\mathcal{U}}_{\epsilon }\mid Y_1,Y_2,\dots ,Y_J)\to 1a.s.\mathit{under}{\mathrm{\Pi }}_{{\theta }_0}.$$

The proof is based on the strong consistency theorem by Choi and Schervish (2007) and is detailed in Appendix A.

2.2 The Bayes Estimate as The Nested Smoothing Spline

We show in Theorem 4 that the posterior mean of U under an nGP prior can be related to the minimizer, namely the nested smoothing spline (nSS) Û, of the following penalized sum-of-squares with nested penalties,

$$\text{nPSS}=\frac{1}{J}\sum _{j=1}^J{\{Y(t_j)-U(t_j)\}}^2+{\lambda }_U{\int }_{\mathcal{T}}{\{D^{m}U(t)-A(t)\}}^{2}dt+{\lambda }_A{\int }_{\mathcal{T}}{\{D^{n}A(t)\}}^{2}dt,$$ (4)

where λ_U ∈ ℝ⁺ and λ_A ∈ ℝ⁺ are the smoothing parameters which control the smoothness of unknown functions U(t) and A(t) respectively. The following Theorem 3 provides the explicit forms for nSS, for which the proofs are included in Appendix A.

Theorem 3

The nested smoothing spline Û (t) has the form

$$\begin{array}{l}\widehat{U}(t)=\sum _{i=0}^{m-1}{\mu }_i{\phi }_i(t)+\sum _{j=1}^J{\nu }_j{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_j,t)+\sum _{i=0}^{n-1}{\alpha }_i{\phi }_{m+i}(t)+\sum _{j=1}^J{\beta }_j{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_j,t)\\ ={\mathit{\mu }}^{\prime }{\mathit{\phi }}_{\mu }(t)+{\mathit{\nu }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{U}}(t)+{\mathit{\alpha }}^{\prime }{\mathit{\phi }}_{\alpha }(t)+{\mathit{\beta }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{A}}(t),\end{array}$$

where μ = (μ₀, μ₁, ···, μ_m−1)′, ν = (ν₁, ν₁, ···, ν_J)′, α = (α₀, α₁, ···, α_n−1)′ and β = (β₁, β₂, ···, β_J)′ are the coefficients for the bases

$$\begin{array}{l}{\mathit{\phi }}_{\mu }(t)={\{{\phi }_0(t),{\phi }_1(t),\cdots ,{\phi }_{m-1}(t)\}}^{\prime },{\mathit{R}}_{\stackrel{\sim }{U}}(t)={\{{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_1,t),{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_2,t),\cdots ,{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_J,t)\}}^{\prime },\\ {\mathit{\phi }}_{\alpha }(t)={\{{\phi }_m(t),{\phi }_{m+1}(t),\cdots ,{\phi }_{m+n-1}(t)\}}^{\prime },{\mathit{R}}_{\stackrel{\sim }{A}}(t)={\{{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_1,t),{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_2,t),\cdots ,{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_J,t)\}}^{\prime }.\end{array}$$

In addition, the nested penalized sum-of-squares can be written as

$$\mathit{nPSS}=\frac{1}{J}{(\mathit{Y}-{\mathit{\phi }}_{\mu }\mathit{\mu }-{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }-{\mathit{\phi }}_{\alpha }\mathit{\alpha }-{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta })}^{\prime }(\mathit{Y}-{\mathit{\phi }}_{\mu }\mathit{\mu }-{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }-{\mathit{\phi }}_{\alpha }\mathit{\alpha }-{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta })+{\lambda }_U{\mathit{\nu }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\lambda }_A{\mathit{\beta }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta },$$

where

$$\begin{array}{l}\mathit{Y}={\{Y(t_1),Y(t_1),\cdots ,Y(t_J)\}}^{\prime },\\ {\mathit{\phi }}_{\mu }={\{{\mathit{\phi }}_{\mu }(t_1),{\mathit{\phi }}_{\mu }(t_2),\cdots ,{\mathit{\phi }}_{\mu }(t_J)\}}^{\prime },{\mathit{\phi }}_{\alpha }={\{{\mathit{\phi }}_{\alpha }(t_1),{\mathit{\phi }}_{\alpha }(t_2),\cdots ,{\mathit{\phi }}_{\alpha }(t_J)\}}^{\prime },\\ {\mathit{R}}_{\stackrel{\sim }{U}}=\{{\mathit{R}}_{\stackrel{\sim }{U}}(t_1),{\mathit{R}}_{\stackrel{\sim }{U}}(t_2),\cdots ,{\mathit{R}}_{\stackrel{\sim }{U}}(t_J)\},{\mathit{R}}_{\stackrel{\sim }{A}}=\{{\mathit{R}}_{\stackrel{\sim }{A}}(t_1),{\mathit{R}}_{\stackrel{\sim }{A}}(t_2),\cdots ,{\mathit{R}}_{\stackrel{\sim }{A}}(t_J)\}.\end{array}$$

The coefficients μ, ν, α and β of the nested smoothing spline Û (t) are given as

$$\begin{array}{l}\mathit{\mu }={\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }{\mathit{S}}^{-1}\mathit{Y},\\ \mathit{\nu }={\mathit{S}}^{-1}\left\{\mathit{I}-\left({\mathit{\phi }}_{\mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }+{\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }\right){\mathit{S}}^{-1}\right\}\mathit{Y},\\ \mathit{\alpha }={\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }{\mathit{S}}^{-1}\mathit{Y},\\ \mathit{\beta }=\frac{{\lambda }_U}{{\lambda }_A}\mathit{\nu },\end{array}$$

where ${\mathit{\phi }}_{\mu \mid \alpha }={\mathit{\phi }}_{\mu }^{\prime }-{\mathbf{\sum }}_{\mu \alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathit{\phi }}_{\alpha }^{\prime },{\mathit{\phi }}_{\alpha \mid \mu }={\mathit{\phi }}_{\alpha }^{\prime }-{\mathbf{\sum }}_{\alpha \mu }{\mathbf{\sum }}_{\mu \mu }^{-1}{\mathit{\phi }}_{\mu }^{\prime },{\mathbf{\sum }}_{\mu \mid \alpha }={\mathbf{\sum }}_{\mu \mu }-{\mathbf{\sum }}_{\mu \alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathbf{\sum }}_{\alpha \mu },{\mathbf{\sum }}_{\alpha \mid \mu }={\mathbf{\sum }}_{\alpha \alpha }-{\mathbf{\sum }}_{\alpha \mu }{\mathbf{\sum }}_{\mu \mu }^{-1}{\mathbf{\sum }}_{\mu \alpha },{\mathbf{\sum }}_{\mu \mu }={\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\mu },{\mathbf{\sum }}_{\mu \alpha }={\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}^{-1}{\phi }_{\alpha },{\mathbf{\sum }}_{\alpha \mu }={\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\mu },{\mathbf{\sum }}_{\alpha \alpha }={\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }\mathit{and}\mathit{S}={\mathit{M}}_{\stackrel{\sim }{U}}+\frac{{\lambda }_U}{{\lambda }_A}{\mathit{R}}_{\stackrel{\sim }{A}}={\mathit{R}}_{\stackrel{\sim }{U}}+J{\lambda }_U\mathit{I}+\frac{{\lambda }_U}{{\lambda }_A}{\mathit{R}}_{\stackrel{\sim }{A}}$.

By applying Theorem 3, it’s easy to show that the nested smoothing spline Û (t) is a linear smoother, expressed in the matrix form as, Û = K_λU,λAY, a linear combination of observations with the weight matrix K_λUλA where K_λUλA = φ_μB_μ+R_ŨB_ν+ φ_αB_α+ R_ÃB_β with ${\mathit{B}}_{\mu }={\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }{\mathit{S}}^{-1},{\mathit{B}}_{\nu }={\mathit{S}}^{-1}\left\{\mathit{I}-\left({\mathit{\phi }}_{\mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }+{\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }\right){\mathit{S}}^{-1}\right\},{\mathit{B}}_{\alpha }={\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }{\mathit{S}}^{-1}$ and ${\mathit{B}}_{\beta }=\frac{{\lambda }_U}{{\lambda }_A}{\mathit{B}}_{\nu }$.

Theorem 4 below shows the main result of this subsection, i.e. the posterior mean of U under the nGP prior is equivalent to the nSS Û when ${\sigma }_{\mu }^2\to \infty$ and ${\sigma }_{\alpha }^2\to \infty$. The proof is in Appendix A

Theorem 4

Given observations Y = {Y(t₁), Y(t₂),…, Y(t_J)}′, the posterior mean of U(t) with nested Gaussian process prior is denoted as ${\overline{U}}_{{\sigma }_{\mu }^2,{\sigma }_{\alpha }^2}(t)=E\{U(t)\mid \mathit{Y},{\sigma }_{\mu }^2,{\sigma }_{\alpha }^2,{\sigma }_U^2,{\sigma }_A^2,{\sigma }_{\epsilon }^2\}$. We have

$$\underset{{\sigma }_{\mu }^2\to \infty }{lim}\underset{{\sigma }_{\alpha }^2\to \infty }{lim}{\overline{U}}_{{\sigma }_{\mu }^2,{\sigma }_{\alpha }^2}(t)=\widehat{U}(t),$$

where Û(t) is the nested smoothing spline.

3 Posterior Computation

To complete a Bayesian specification, we choose priors for the initial values, covariance parameters in the nGP and residual variance. In particular, we let $\mathit{\mu }~{\mathrm{N}}_m(\mathbf{0},{\sigma }_{\mu }^2\mathit{I}),\mathit{\alpha }~{\mathrm{N}}_m(\mathbf{0},{\sigma }_{\alpha }^2\mathit{I}),{\sigma }_{\epsilon }^2~\text{invGamma}(a,b),{\sigma }_U^2~\text{invGamma}(a,b)$ and ${\sigma }_A^2~\text{invGamma}(a,b)$, where invGamma(a, b) denotes the inverse gamma distribution with shape parameter a and scale parameter b. In the applications shown below, the data are rescaled so that the absolute value of the maximum observation is less than 100. We choose diffuse but proper priors by letting ${\sigma }_{\mu }^{-1}={\sigma }_{\alpha }^{-1}=a=b=0.01$ as a default to allow the data to inform strongly, and have observed good performance in a variety of settings for this choice. In practice, we have found the posterior distributions for these hyperparameters to be substantially more concentrated than the prior in applications we have considered, suggesting substantial Bayesian learning.

With this prior specification, we propose an MCMC algorithm for posterior computation. This algorithm consists of two iterative steps: (1) Given the ${\sigma }_{\epsilon }^2,{\sigma }_U^2,{\sigma }_A^2$ and Y, draw posterior samples of μ, U = {U(t₁), U (t₂),…, U (t_J)}′, α and A = {A(t₁), A(t₂),…, A(t_J)}′; (2) Given the μ, U, α, A and Y, draw posterior samples of ${\sigma }_{\epsilon }^2,{\sigma }_U^2$ and ${\sigma }_A^2$.

In the first step, it would seem natural to draw U and A from their multivariate normal conditional posterior distributions. However, this is extremely expensive computationally in high dimensions involving O(J³) computations in inverting J × J covariance matrices, which do not have any sparsity structure that can be exploited. To reduce this computational bottleneck in GP models, there is a rich literature relying on low rank matrix approximations (Smola and Bartlett, 2001; Lawrence et al., 2002; Quinonero-Candela and Rasmussen, 2005). Of course, such low rank approximations introduce some associated approximation error, with the magnitude of this error unknown but potentially substantial in our motivating mass spectrometry applications, as it is not clear that typical approximations having sufficiently low rank to be computationally feasible can be accurate.

To bypass the need for such approximations, we propose a different approach that does not require inverting J × J covariance matrices but instead exploits the Markovian property implied by SDEs (2) and (3). The Markovian property is represented by a stochastic difference equation, namely the state equation, as illustrated for the case when m = 2 and n = 1 as follows which is easily extended to cases with higher order of m and n. Given m = 2 and n = 1, nested Gaussian process U(t) along with its first order derivative D¹U(t) and A(t) follow the exact state equation:

$${\mathit{\theta }}_{j+1}={\mathit{G}}_j{\mathit{\theta }}_j+{\mathit{\omega }}_j,$$ (5)

where θ_j+1 = {U(t_j+1), D¹U(t_j+1), A(t_j+1)}′, ω_j ~ N₃ (0, W_j), ${\mathit{G}}_j=\left(\begin{array}{ccc}1& {\delta }_j& \frac{{\delta }_j^2}{2}\\ 0& 1& {\delta }_j\\ 0& 0& 1\end{array}\right)$ and ${\mathit{W}}_j=\left(\begin{array}{ccc}\frac{{\delta }_j^3}{3}{\sigma }_U^2+\frac{{\delta }_j^5}{20}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_U^2+\frac{{\delta }_j^4}{8}{\sigma }_A^2& \frac{{\delta }_j^3}{6}{\sigma }_A^2\\ \frac{{\delta }_j^2}{2}{\sigma }_U^2+\frac{{\delta }_j^4}{8}{\sigma }_A^2& {\delta }_j{\sigma }_U^2+\frac{{\delta }_j^3}{3}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_A^2\\ \frac{{\delta }_j^3}{6}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_A^2& {\delta }_j{\sigma }_A^2\end{array}\right)$ with δ_j = t_j+1 − t_j. The proof is in Appendix A. The exact state equation combined with the observation equation (1) forms a state space model (West and Harrison, 1997; Durbin and Koopman, 2001), for which the latent states θ_j’s can be efficiently sampled by a simulation smoother algorithm (Durbin and Koopman, 2002) with O(J) computation complexity.

Given the μ, U, α and A, posterior samples of ${\sigma }_{\epsilon }^2$ can be obtained by drawing from the inverse-gamma conditional posterior while ${\sigma }_U^2$ and ${\sigma }_A^2$ can be updated in Metropolis-Hastings (MH) steps. We have found that typical MH random walk steps tend to be sticky and it is preferable to use MH independence chain proposals in which one samples candidates for ${\sigma }_U^2$ and ${\sigma }_A^2$ from approximations to their conditional posteriors that are easy to sample from. To accomplish this, we rely on the following approximate state equation.

When δ_j is sufficient small, the exact state equation (5) can be approximated by

$${\mathit{\theta }}_{j+1}={\stackrel{\sim }{\mathit{G}}}_j{\mathit{\theta }}_j+{\stackrel{\sim }{\mathit{H}}}_j{\stackrel{\sim }{\mathit{\omega }}}_j,$$ (6)

where ω̃_j ~ N₂ (0, W̃_j), ${\stackrel{\sim }{\mathit{G}}}_j=\left(\begin{array}{ccc}1& {\delta }_j& 0\\ 0& 1& {\delta }_j\\ 0& 0& 1\end{array}\right),{\stackrel{\sim }{\mathit{H}}}_j=\left(\begin{array}{cc}0& 0\\ 1& 0\\ 0& 1\end{array}\right)$ and ${\stackrel{\sim }{\mathit{W}}}_j=\left(\begin{array}{cc}{\sigma }_U^2{\delta }_j& 0\\ 0& {\sigma }_A^2{\delta }_j\end{array}\right)$. The above approximate state equation is derived by applying the Euler approximation (chapter 9, Kloeden and Platen, 1992), essentially a first-order Taylor approximation, to the SDEs (2) and (3). Given the θ_j’s, the ${\sigma }_U^2$ and ${\sigma }_A^2$ in the above approximate state equation can be easily sampled as a Bayesian linear regression model with the given coefficients.

Finally, we outline the proposed MCMC algorithm as follows:

• (1)For the state space model with the observation equation (1) and the exact state equation (5), update the latent states μ, U, α and A by using the simulation smoother.
• (2)Sample ${\sigma }_{\epsilon }^2$ from the posterior distribution $\text{invGamma}\left(a+\frac{1}{2}J,b+\frac{1}{2}{\sum }_{j=1}^J{\{Y(t_j)-U(t_j)\}}^2\right)$.
• (3a)Given ${\sigma }_{\epsilon }^2,{\sigma }_U^2$ and ${\sigma }_A^2$, we sample the latent states μ*, U*, α* and A* for the approximate state space model with the observation equation (1) and the approximate state equation (6).
• (3b)Given μ*, U*, α* and A*, the proposal ${\sigma }_U^{2\ast }$ and ${\sigma }_A^{2\ast }$ is drawn from the posterior distributions $\text{invGamma}\left(a+\frac{1}{2}J,b+\frac{1}{2}{\sum }_{j=0}^{J-1}\frac{{\{{DU}^{\ast }(t_{j+1})-{DU}^{\ast }(t_j)-A^{\ast }(t_j){\delta }_j\}}^2}{{\delta }_j}\right)$ and $\text{invGamma}\left(a+\frac{1}{2}J,b+\frac{1}{2}{\sum }_{j=0}^{J-1}\frac{{\{A^{\ast }(t_{j+1})-A^{\ast }(t_j)\}}^2}{{\delta }_j}\right)$, respectively.
• (3c)
  The proposal ${\sigma }_U^{2\ast }$ and ${\sigma }_A^{2\ast }$ will be accepted with the probability min

  $$min\left\{\prod _{j=0}^{J-1}\frac{f_{\mathrm{N},3}({\mathit{\theta }}_{j+1}-{\mathit{G}}_j{\mathit{\theta }}_j\mid \mathbf{0},{\mathit{W}}_j^{\ast })f_{\mathrm{N},2}\left({\stackrel{\sim }{\mathit{H}}}_j({\mathit{\theta }}_{j+1}^{\ast }-{\stackrel{\sim }{\mathit{G}}}_j{\mathit{\theta }}_j^{\ast })\mid \mathbf{0},{\stackrel{\sim }{\mathit{W}}}_j\right)}{f_{\mathrm{N},3}({\mathit{\theta }}_{j+1}-{\mathit{G}}_j{\mathit{\theta }}_j\mid \mathbf{0},{\mathit{W}}_j)f_{\mathrm{N},2}\left({\stackrel{\sim }{\mathit{H}}}_j({\mathit{\theta }}_{j+1}^{\ast }-{\stackrel{\sim }{\mathit{G}}}_j{\mathit{\theta }}_j^{\ast })\mid \mathbf{0},{\stackrel{\sim }{\mathit{W}}}_j^{\ast }\right)},1\right\},$$

  where f_N,k(X | 0, Σ) denotes the probability density function of the k-dimensional normal random vector with mean 0 and covariance matrix Σ; θ_j, W_j and W̃_j are specified in equations (5) and (6); Similar notions hold for ${\mathit{\theta }}_j^{\ast },{\mathit{W}}_j^{\ast }$ and ${\stackrel{\sim }{\mathit{W}}}_j^{\ast }$ with μ, U, α A, ${\sigma }_U^2$ and ${\sigma }_A^2$ replaced by μ*, U*, α* A*, ${\sigma }_U^{2\ast }$ and ${\sigma }_A^{2\ast }$ correspondingly.

We also consider an empirical Bayes approach, which modifies the above MCMC algorithm to use conditional method of moment estimators for ${\sigma }_{\epsilon }^2,{\sigma }_U^2$ and ${\sigma }_A^2$. In particular, focus on the m = 2, n = 1 case and assume δ_j = δ sufficiently small and observations are equally spaced. Then, given Y (t_j), U (t_j) and A(t_j), we have ${\stackrel{\sim }{\sigma }}_{\epsilon }^2=\frac{1}{J}{\sum }_{j=1}^J{\{Y(t_j)-U(t_j)\}}^2,{\stackrel{\sim }{\sigma }}_U^2=\frac{1}{J}{\sum }_{j=0}^{J-1}\frac{{\{DU(t_{j+1})-DU(t_j)-A(t_j)\delta \}}^2}{\delta }$ and ${\stackrel{\sim }{\sigma }}_A^2=\frac{1}{J}{\sum }_{j=0}^{J-1}\frac{{\{A(t_{j+1})-A(t_j)\}}^2}{\delta }$ derived from the observation equation (1) and the approximate state equation (6). Given ${\stackrel{\sim }{\sigma }}_{\epsilon }^2,{\stackrel{\sim }{\sigma }}_U^2$ and ${\stackrel{\sim }{\sigma }}_A^2$, the latent states can be easily sampled using the first step of the above MCMC algorithm.

4 Simulations

We conducted a simulation study to assess the performance of the proposed method, Bayesian nonparametric regression via an nGP prior (BNR-nGP), and compared it to several alternative methods: cubic smoothing spline (SS, Wahba, 1990), wavelet method with the soft minimax threshold (Wavelet1, Donoho and Johnstone, 1994), wavelet method with the soft Stein’s unbiased estimate of risk for threshold choice (Wavelet2, Donoho and Johnstone, 1995) and hybrid adaptive splines (HAS, Luo and Wahba, 1997). For BNR-nGP, we take the posterior mean as the estimate, which is based on the draws from the proposed MCMC algorithm with 1,500 iterations, discarding the first 500 as the burn-in stage and saving remaining ones. The initial values of ${\sigma }_{\epsilon }^2,{\sigma }_U^2$ and ${\sigma }_U^2$ are chosen as method of moment estimators. The other methods are implemented in R (R Development Core Team, 2011), along with the corresponding R packages for Wavelet methods (wmtsa, Constantine and Percival, 2010) and hybrid adaptive splines (bsml, Wu et al., 2011).

Our first simulation study focuses on four functions adapted from Donoho and Johnstone (1994) with different types of locally-varying smoothness. The functions are plotted in Figure 2, for which the smoothness levels vary, for example, abruptly in panel (a) or gradually in panel (d). For each function, equally-spaced observations are obtained with Gaussian noise, for which the signal-to-noise ratio is $\frac{SD(U)}{{\sigma }_{\epsilon }}=7$. We use the mean squared error (MSE) $\frac{1}{J}{\sum }_{j=1}^J{\{\widehat{U}(t_j)-U_0(t_j)\}}^2$ to compare the performance of different methods based on 100 replicates. The simulation results are summarized in Table 1. Among all methods, SS performs worst, which is not surprising since it can not adapt to the locally-varying smoothness. Among the remaining methods, BNR-nGP performs well in general for all cases with either the smallest or the second smallest average MSE across 100 replicates. In contrast, Wavelet2 and HAS may perform better for a given function, but their performances are obviously inferior for another function (e.g. Heavisine for Wavelet2 and Doppler for HAS). This suggests the nGP prior is able to adapt to a wide variety of locally-varying smoothness profiles.

Figure 2.

Four locally-varying smoothness functions: true function ( ) and 128 observations ( ).

Table 1.

Average MSE and the interquartile range of MSE (in parentheses) for Bayesian nonparametric regression with nGP prior (BNR-nGP), smoothing spline (SS), wavelet method with the soft minimax threshold (Wavelet1), wavelet method with the soft Stein’s unbiased estimate of risk for threshold choice (Wavelet2) and Hybrid adaptive spline (HAS).

Example	BNR-nGP	SS	Wavelet1	Wavelet2	HAS
Blocks	0.950(0.166)	3.018(0.248)	2.750(0.748)	1.237(0.341)	0.539(0.113)
Bumps	1.014(0.185)	26.185(0.787)	3.433(0.938)	1.195(0.282)	0.904(0.258)
Heavisine	0.320(0.058)	0.337(0.087)	0.702(0.230)	1.620(0.460)	0.818(0.122)
Doppler	0.989(0.183)	3.403(0.361)	1.517(0.402)	0.695(0.179)	3.700(0.534)
MS Region 1(×10−3)	1.498(0.266)	2.293(0.513)	2.367(0.616)	6.048(3.441)	72.565(39.596)
MS Region 2(×10−3)	0.840(0.375)	0.798(0.490)	0.948(0.587)	1.885(0.493)	7.958(5.559)

We further compare the proposed method and the alternative methods for analyzing mass spectrometry data. The 100 datasets are generated by the ‘virtual mass spectrometer’ (Coombes et al., 2005a), which considers the physical principles of the instrument. One set of these simulated data is plotted in Figure 3 with σ_ε = 66. The simulated data have been shown to accurately mimic real data (Morris et al., 2005) and are available at http://bioinformatics.mdanderson.org/Supplements/Datasets/Simulations/index.html. Since the analysis of all observations (J=20,695) of a given dataset is computational infeasible for HAS, we focus on the analysis of the observations within two regions with 5 < km=z < 8 (region 1 with J=2,524) and 20 < km=z < 25 (region 2 with J=2,235) respectively. Those two regions represent the unique feature of mass spectrometry data. More specifically, with smaller km/z values the peaks are much taller and sharper than the peaks in the region with larger km/z values. The results in Table 1 indicate that the BNR-nGP performs better than the other smoothness adaptive methods for both regions in terms of smaller average MSE and narrower interquartile range of MSE. Although the smoothing spline seems to work well with smaller average MSE in region 2, the peaks are clearly over-smoothed, leading to large MSEs at these important locations. In contrast, BNR-nGP had excellent performance relative to the competitors across locations.

Figure 3.

The plot of one simulated mass spectrometry data (J=20,695).

We finally compare the full Bayes inference approach with the empirical Bayes approach detailed in Section 3. The performance of the two methods are measured by MSEs. The observations are simulated based on the exact state equation (5) and the observation equation (1) with ${\sigma }_{\epsilon }^2=1,{\sigma }_U^2=10,{\sigma }_A^2=1$, θ₀ = 0 and J = 256. The observations are equally spaced with δ = 1 and we simulate 100 replicate datasets. The full Bayes approach is implemented as in the previous simulation study. The MCMC algorithm took 86 seconds for 1500 iterations on a PC with 2.33GHz Intel(R)Xeon(R)CPU. The trace plots and autocorrelation plots imply fast convergence and rapid mixing, especially for the samples of U(t) and ${\sigma }_{\epsilon }^2$. For the posterior means of ${\sigma }_{\epsilon }^2,{\sigma }_U^2,{\sigma }_A^2$, their MSEs are 0.003, 4.319 and 0.142 while the average MSE of posterior mean of U(t) is 0.622. We also apply the empirical Bayes approach with method of moment estimators of ${\sigma }_{\epsilon }^2,{\sigma }_U^2,{\sigma }_A^2$. Their MSE are 0.002, 11.459 and 0.183 respectively. The average MSE of U(t) is 0.629 when we take the posterior mean as the estimator. Although the empirical Bayes approach with the method of moment estimators is simple to implement and computationally less intensive, the MSEs of ${\sigma }_U^2$ and ${\sigma }_A^2$ are inflated significantly, possibly due to approximating the differential equations in deriving the method of moment estimators.

5 Applications

We apply the proposed method to protein mass spectrometry (MS) data. Protein MS plays an important role in proteomics for identifying disease-related proteins in the samples (Cottrell and London, 1999; Tibshirani et al., 2004; Domon and Aebersold, 2006; Morris et al., 2008). For example, Panel (a) of Figure 1 plots 11, 186 intensities in a pooled sample of nipple aspirate fluid from healthy breasts and breasts with cancer versus the mass to charge ratio m/z of ions (Coombes et al., 2005b). Analysis of protein MS data involves several steps, including spectra alignment, signal extraction, baseline subtraction, normalization and peak detection. As an illustration of our method, we focus on the second step, i.e., estimate the intensity function adjusted for measurement errors. Peaks in the intensity function may correspond to proteins that differ in the expression levels between cancer and control patients.

We fit the Bayes nonparametric regression with nGP prior and ran the MCMC algorithm for 11,000 iterations with the first 1000 iterations discarded as burn-in and every 10th draw retained for analysis. The trace plots and autocorrelation plots suggested the algorithm converged fast and mixed well. Panel (b) of Figure 1 plots the posterior mean of U and its pointwise 95% credible interval. Note that the posterior mean of U is adapted to the various smoothness at different regions, which is more apparently illustrated by the Panel (c) of Figure 1. Panel (d) of Figure 1 demonstrates the posterior mean and 95% credible interval of rate of intensity change DU, which suggests a peak around 4 km/z.

6 Discussion

We have proposed a novel nested Gaussian process prior, which is designed for flexible nonparametric locally adaptive smoothing while facilitating efficient computation even in large data sets. Most approaches for Bayesian locally adaptive smoothing, such as free knot splines and kernel regression with varying bandwidths, encounter substantial problems with scalability. Even isotropic Gaussian processes, which provide a widely used and studied prior for nonparametric regression, face well known issues in large data sets, with standard approaches for speeding up computation relying on low rank approximations. It is typically not possible to assess the accuracy of such approximations and whether a low rank assumption is warranted for a particular data set. However, when the function of interest is not smooth but can have very many local bumps and features, high resolution data may be intrinsically needed to obtain an accurate estimate of local features of the function, with low rank approximations having poor accuracy. This seems to be the case in mass spectroscopy applications, such as the motivating proteomics example we considered in Section 5. We have simultaneously addressed two fundamental limitations of typical isotropic Gaussian process priors for nonparametric Bayes regression: (i) the lack of spatially-varying smoothness; and (ii) the lack of scalability to large sample sizes. In addition, this was accomplished in a single coherent Bayesian probability model that fully accounts for uncertainty in the function without relying on multistage estimation.

Although we have provided an initial study of some basic theoretical properties, the fundamental motivation in this paper is to obtain a practically useful method. We hope that this initial work stimulates additional research along several interesting lines. The first relates to generalizing the models and computational algorithms to multivariate regression surfaces. Seemingly this will be straightforward to accomplish using additive models and tensor product specifications. The second is to allow for the incorporation of prior knowledge regarding the shapes of the functions; in some applications, there is information available in the form of differential equations or even a rough knowledge of the types of curves one anticipates, which could ideally be incorporated into an nGP prior. Finally, there are several interesting theoretical directions, such as showing rates of posterior contraction for true functions belonging to a spatially-varying smoothness class.

A Appendix: Proofs of Theoretical Results

A.1 Proof of Theorem 1

Its proof requires the results of the following lemma.

Lemma 1

The nested Gaussian process U can be written as U(t) = Ũ₀(t) + Ũ₁(t) + Ã₀(t) + Ã₁(t), the summation of mutually independent Gaussian processes with the corresponding mean functions E{U₀(t)} = E{U₁(t)} = E{A₀(t)} = E {A₁(t)} = 0 and covariance functions

$$\begin{array}{l}{\mathcal{K}}_{{\stackrel{\sim }{U}}_0}(s,t)={\sigma }_{\mu }^2{\mathcal{R}}_{{\stackrel{\sim }{U}}_0}(s,t)={\sigma }_{\mu }^2\sum _{i=0}^{m-1}{\phi }_i(s){\phi }_i(t),\\ {\mathcal{K}}_{{\stackrel{\sim }{U}}_1}(s,t)={\sigma }_U^2{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(s,t)={\sigma }_U^2{\int }_{\mathcal{T}}{G}_m(s,u)G_m(t,u)du,\\ {\mathcal{K}}_{{\stackrel{\sim }{A}}_0}(s,t)={\sigma }_{\alpha }^2{\mathcal{R}}_{{\stackrel{\sim }{A}}_0}(s,t)={\sigma }_{\alpha }^2\sum _{i=0}^{n-1}{\phi }_{m+i}(s){\phi }_{m+i}(t),\\ {\mathcal{K}}_{{\stackrel{\sim }{A}}_1}(s,t)={\sigma }_A^2{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(s,t)={\sigma }_A^2{\int }_{\mathcal{T}}{G}_{m+n}(s,u)G_{m+n}(t,u)du,\end{array}$$

respectively, where ${\phi }_i(t)=\frac{t^i}{i!}$ and $G_m(s,u)=\frac{{(s-u)}_{+}^{m-1}}{(m-1)!}$.

Proof

We specify U(t) = Ũ(t) + Ã(t) and A(t) = D^m Ã(t). By SDEs (2) and (3),

$$D^m\stackrel{\sim }{U}(t)={\sigma }_U{\stackrel{.}{W}}_U(t),$$ (7)

$$D^{m+n}\stackrel{\sim }{A}(t)={\sigma }_A{\stackrel{.}{W}}_A(t).$$ (8)

By applying stochastic integration to SDEs (7) and (8), it can be shown that

$$\begin{array}{l}\stackrel{\sim }{U}(t)={\stackrel{\sim }{U}}_0(t)+{\stackrel{\sim }{U}}_1(t)=\sum _{i=0}^{m-1}{\mu }_i{\phi }_i(t)+{\sigma }_U^2{\int }_{\mathcal{T}}{G}_m(t,u){\stackrel{.}{W}}_U(u)du,\\ \stackrel{\sim }{A}(t)={\stackrel{\sim }{A}}_0(t)+{\stackrel{\sim }{A}}_1(t)=\sum _{i=0}^{n-1}{\alpha }_i{\phi }_{m+i}(t)+{\sigma }_A^2{\int }_{\mathcal{T}}{G}_{m+n}(t,u){\stackrel{.}{W}}_A(u)du\end{array}$$

given the initial values μ and α. Since Ũ₀(t), Ũ₁(t), Ã₀(t) and Ã₁(t) are the linear combination of Gaussian random variables at every t, they are Gaussian processes defined over t, whose mean functions and covariance functions can be easily derived as required. In addition, Ũ₀(t), Ũ₁(t), Ã₀(t) and Ã₁(t) are mutually independent due to the mutually independent assumption of μ, α, Ẇ_U (·) and Ẇ_A(·) in the definition of nGP.

Given the above lemma, the Theorem 1 can be proven as follows. We aim to characterize , the RKHS of U with the reproducing kernel (s, t). The support of U, a mean-zero Gaussian random element, is the closure of (Van der Vaart and Van Zanten, 2008b, Lemma 5.1).

By Loève’s Theorem (Berlinet and Thomas-Agnan, 2004, Theorem 35), the RKHSs generated by the processes Ũ₀(t), Ũ₁(t), Ã₀(t) and Ã₁(t) with covariance functions (s, t), (s, t), (s, t) and (s, t) (given in Lemma 1) are congruent to RKHSs , , and , respectively. Based on Theorem 5 of Berlinet and Thomas-Agnan (2004), we conclude that (s, t) = (s, t) + (s, t) + (s, t) + (s, t) is the reproducing kernel of the RKHS

$$\begin{array}{l}{\mathcal{H}}_{{\mathcal{K}}_U}={\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{U}}_0}}\oplus {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{U}}_1}}\oplus {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{A}}_0}}\oplus {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{A}}_1}}\\ =\{U(t):U(t)={\stackrel{\sim }{U}}_0(t)+{\stackrel{\sim }{U}}_1(t)+{\stackrel{\sim }{A}}_0(t)+{\stackrel{\sim }{A}}_1(t),{\stackrel{\sim }{U}}_0(t)\in {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{U}}_0}},{\stackrel{\sim }{U}}_1(t)\in {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{U}}_1}},{\stackrel{\sim }{A}}_0(t)\in {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{A}}_0}},{\stackrel{\sim }{A}}_1(t)\in {\mathcal{H}}_{{\mathcal{K}}_{{\stackrel{\sim }{A}}_1}}\}\end{array}$$

A.2 Proof of Theorem 2

We first specify a couple of regularity conditions given by:

• Assumption 1 t_j arises according to an infill design: for each S_j = t_j+1 − t_j, there exists a constant 0< C_d ≤ 1 such that ${max}_{1\le j<J}{S}_j<\frac{t_U}{C_{d}J}$.

• Assumption 2 The prior distributions ${\mathrm{\Pi }}_{{\sigma }_{\mu }^2},{\mathrm{\Pi }}_{{\sigma }_U^2},{\mathrm{\Pi }}_{{\sigma }_{\alpha }^2}$ and ${\mathrm{\Pi }}_{{\sigma }_A^2}$ satisfy an exponential tail condition. Specifically, there exist sequences M_J, ${\sigma }_{\mu ,J}^2,{\sigma }_{U,J}^2,{\sigma }_{\alpha ,J}^2$ and ${\sigma }_{A,J}^2$ such that: (i) ${\mathrm{\Pi }}_{{\sigma }_{\mu }^2}({\sigma }_{\mu }^2>{\sigma }_{\mu ,J}^2)=e^{-C_{\mu }J},{\mathrm{\Pi }}_{{\sigma }_U^2}({\sigma }_U^2>{\sigma }_{U,J}^2)=e^{-C_{U}J},{\mathrm{\Pi }}_{{\sigma }_{\alpha }^2}({\sigma }_{\alpha }^2>{\sigma }_{\alpha ,J}^2)=e^{-C_{\alpha }J}$ and ${\mathrm{\Pi }}_{{\sigma }_A^2}({\sigma }_A^2>{\sigma }_{A,J}^2)=e^{-C_{A}J}$, for some positive constants C_μ, C_U, C_α and C_A; (ii) $M_J^2{\sigma }_J^{-2}\ge C_{g}J$, for every C_g > 0 and ${\sigma }_J^{-2}$, the minimal element of { ${\sigma }_{\mu ,J}^{-2},{\sigma }_{U,J}^{-2},{\sigma }_{\alpha ,J}^{-2},{\sigma }_{A,J}^{-2}$}.

• Assumption 3 The prior distribution Π_σε is continuous and the σ_ε,0 lies in the support of Π_σε

Similar to the proof of Lemma 1, we specify U = Ũ₀ + Ũ₁ + Ã₀ + Ã₁, which is a mean zero Gaussian process with continuous and differentiable covariance function (s, t) = (s, t) + (s, t) + (s, t) + (s, t).

We aims to verify the suficient conditions of the strong consistency theorem (Theorem 1, Choi and Schervish, 2007) for nonparametric regression: (I) prior positivity of neighborhoods and (II) existence of uniformly exponentially consistent tests and sieves Θ_J with ${\mathrm{\Pi }}_U({\mathrm{\Theta }}_J^C)\le C_{1}exp(-C_{2}J)$ for some positive constants C1 and C2.

Given U is a Gaussian process with continuous sample path and continuous covariance function, it follows from Theorem 4 of Ghosal and Roy (2006) that Π_U (||U − U₀||_∞ < δ) > 0 for any δ > 0. In addition, for every δ > 0, ${\mathrm{\Pi }}_{{\sigma }_{\epsilon }}\left(\mid \frac{{\sigma }_{\epsilon }}{{\sigma }_{\epsilon ,0}}-1\mid <\delta \right)>0$ under Assumption 3. Hence, we can define a neighborhood $B_{\delta }=\left\{(U,{\sigma }_{\epsilon }):{\Vert U-U_0\Vert }_{\infty }<\delta ,\mid \frac{{\sigma }_{\epsilon }}{{\sigma }_{\epsilon ,0}}-1\mid <\delta \right\}$ such that Π_(U,σε)(B_δ) > 0 satisfying the condition (I).

From Theorem 2 of Choi and Schervish (2007), we can show that for a sequence of M_J, there exist uniformly exponentially consistent tests for the sieves Θ_J = {U: ||U||_∞ < M_J, ||DU ||_∞ < M_J} under the infill design Assumption 1. What remains is to verify the exponentially small probability of ${\mathrm{\Theta }}_{J,0}^C=\{U:{\Vert U\Vert }_{\infty }>M_J\}$ and ${\mathrm{\Theta }}_{J,1}^C=\{U:{\Vert DU\Vert }_{\infty }>M_J\}$. Using Borell’s inequality (Proposition A.2.7, Van der Vaart and Wellner, 1996), we have

$${\mathrm{\Pi }}_U({\Vert U\Vert }_{\infty }>M_J)\le C_{1}exp\left(-\frac{C_3{M}_J^2}{{\sigma }^2}\right)$$

for some positive constants C₁ and C₃, and

$$\begin{array}{l}{\sigma }^2:=\underset{t\in [0,t_U]}{sup}\left\{E{({\stackrel{\sim }{U}}_0+{\stackrel{\sim }{U}}_1+{\stackrel{\sim }{A}}_0+{\stackrel{\sim }{A}}_1)}^2\right\}\\ =\underset{t\in [0,t_U]}{sup}\left\{{E\stackrel{\sim }{U}}_0^2+{E\stackrel{\sim }{U}}_1^2+E{\stackrel{\sim }{A}}_0^2+E{\stackrel{\sim }{A}}_1^2\right\}\\ ={\sigma }_{\mu }^2\sum _{i=0}^{m-1}{\phi }_i^2(t_U)+\frac{{\sigma }_U^2{t}_U^{2m-1}}{(m-1)!(m-1)!(2m-1)}+{\sigma }_{\alpha }^2\sum _{i=0}^{n-1}{\phi }_{m+i}^2(t_U)+\frac{{\sigma }_A^2{t}_U^{2m+2n-1}}{(m+n-1)!(m+n-1)!(2m+2n-1)}.\end{array}$$

By applying the Borel-Cantelli theorem, we have

$${\mathrm{\Pi }}_U({\Vert U\Vert }_{\infty }>M_J)\le C_{1}exp(-C_{2}J),$$

almost surely under the exponential tail Assumption 2. By the similar arguments, we can show that Π_U (||DU||_∞ > M_J) ≤ C₁ exp(−C₂J).

Hence, the conditions (I) and (II) hold, which leads to the strong consistency for Bayesian nonparametric regression with nGP prior.

A.3 Proof of the Support of GP Prior for Polynomial Smoothing Spline

Note that $\stackrel{\sim }{U}(t)={\stackrel{\sim }{U}}_0(t)+{\stackrel{\sim }{U}}_1(t)={\sum }_{i=0}^{m-1}{\mu }_i{\phi }_i(t)+{\sigma }_U^2{\int }_{\mathcal{T}}{G}_m(t,u){\stackrel{.}{W}}_U(u)du$ is the prior for the polynomial smoothing spline (Wahba, 1990, Section 1.5). By the similar arguments in Theorem 1, we can show that the support of Ũ is the closure of RKHS = ⊕ .

Thus, (s, t) − (s, t) = (s, t) = ⊕ a nonnegative kernel, which implies that ⊂ by Corollary 4 of Aronszajn (1950).

A.4 Proof of Theorem 3

Let U(t) = Ũ(t) + Ã(t) and A(t) = D^m Ã(t). The nested penalized sum-of-square (4) can be written as:

$$\text{nPSS}=\frac{1}{J}\sum _{j=1}^J{\left\{Y(t_j)-\stackrel{\sim }{U}(t_j)-\stackrel{\sim }{A}(t_j)\right\}}^2+{\lambda }_U{\int }_{\mathcal{T}}{\left\{D^m\stackrel{\sim }{U}(t)\right\}}^{2}dt+{\lambda }_A{\int }_{\mathcal{T}}{\left\{D^{m+n}\stackrel{\sim }{A}(t)\right\}}^{2}dt,$$ (9)

where Ũ(t) is the m-order polynomial smoothing spline and Ã(t) is the (m+ n)-order polynomial smoothing spline.

By the classical RKHS theory of the polynomial smoothing spline (Wahba, 1990, Section 1.2), there exists a unique decomposition of Ũ(t):

$$\begin{array}{l}\stackrel{\sim }{U}(t)={\stackrel{\sim }{U}}_0(t)+{\stackrel{\sim }{U}}_1(t)\\ =\sum _{i=0}^{m-1}{\mu }_i{\phi }_i(t)+{\int }_{\mathcal{T}}{G}_m(t,u)D^m\stackrel{\sim }{U}(t)du\end{array}$$

with Ũ₀(t) ∈ and Ũ₁(t) ∈ · = {f(t): D^m f(t) = 0, t ∈ } nad = {f(t): Dⁱf(t) absolutely continuous for i = 0, 1, ···, m − 1, D^mf(t) ∈ ( )} are the RKHSs with reproducing kernel (s, t) and (s, t) respectively, where φ_i(t), G_m(t, u), (s, t) and (s, t) are defined in Theorem 1 with ( ) = {f(t): ∫ f²(t)dt < ∞} the space of squared integrable functions defined on index set .

Given = {t_j: j = 1, 2, ···, J}, Ũ₁(t) can be uniquely written as ${\stackrel{\sim }{U}}_1(t)={\sum }_{j=1}^J{\nu }_j{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_j,t)+{\eta }_{{\stackrel{\sim }{U}}_1}(t)$, where η_Ũ1(·) ∈ orthogonal to (t_j, ·) with inner product 〈 (t_j, ·), η_Ũ1 (·)〉 = ∫ D^m (t_j, u)D^m η_Ũ1 (u)du = 0 for j = 1, 2, ···, J.

As a result,

$$\begin{array}{l}{\int }_{\mathcal{T}}{\left\{D^m\stackrel{\sim }{U}(t)\right\}}^{2}dt={\int }_{\mathcal{T}}{\left[D^m\left\{\sum _{i=0}^{m-1}{\mu }_i{\phi }_i(t)+\sum _{j=1}^J{\nu }_j{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_j,t)+{\eta }_{{\stackrel{\sim }{U}}_1}(t)\right\}\right]}^{2}dt\\ =\sum _{j=1}^J\sum _{j^{\prime }=1}^J{\nu }_j{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_j,t_{j^{\prime }}){\nu }_{j^{\prime }}+{\int }_{\mathcal{T}}{\{D^m{\eta }_{{\stackrel{\sim }{U}}_1}(t)\}}^{2}dt\\ ={\mathit{\nu }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\langle {\eta }_{{\stackrel{\sim }{U}}_1}(·),{\eta }_{{\stackrel{\sim }{U}}_1}(·)\rangle }_{{\mathcal{H}}_{{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}}}.\end{array}$$

By similar arguments,

$$\begin{array}{l}\stackrel{\sim }{A}(t)={\stackrel{\sim }{A}}_0(t)+{\stackrel{\sim }{A}}_1(t)\\ =\sum _{i=0}^{n-1}{\alpha }_i{\phi }_{m+i}(t)+\sum _{j=1}^J{\beta }_j{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_j,t)+{\eta }_{{\stackrel{\sim }{A}}_1}(t),\end{array}$$

and

$${\int }_{\mathcal{T}}{\left\{D^m\stackrel{\sim }{A}(t)\right\}}^{2}dt={\mathit{\beta }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta }+{\langle {\eta }_{{\stackrel{\sim }{A}}_1}(·),{\eta }_{{\stackrel{\sim }{A}}_1}(·)\rangle }_{{\mathcal{H}}_{{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}}},$$

where Ã₀(t) ∈ and Ã₁(t) ∈ with = {f(t): D^m+nf(t) = 0, t ∈ } and = {f(t): Dⁱf(t) absolutely continuous for i = 0, 1, ···, m + n − 1, D^m+nf(t) ∈ ( )} RKHSs with reproducing kernel (s, t) and (s, t) respectively; η_Ã1 (·) ∈ is orthogonal to (t_j, ·) with inner product 〈 (t_j, ·), η_Ã1 (·)〉 = ∫ D^m (t_j, u)D^m η_Ã1 (u)du = 0 for j = 1, 2,···, J.

Note that η_Ũ1(t_j) = 〈 (t_j, ·), η_Ũ1(·)〉 = 0 and η_Ã1 (t_j) = 〈 (t_j, ·), η_Ã1 (·)〉 = 0 due to the reproducing property of (t_j, ·) and (t_j, ·). It then follows from expression (9) that

$$\text{nPSS}=\frac{1}{J}{(\mathit{Y}-{\mathit{\phi }}_{\mu }\mathit{\mu }-{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }-{\mathit{\phi }}_{\alpha }\mathit{\alpha }-{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta })}^{\prime }(\mathit{Y}-{\mathit{\phi }}_{\mu }\mathit{\mu }-{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }-{\mathit{\phi }}_{\alpha }\mathit{\alpha }-{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta })+{\lambda }_U{\mathit{\nu }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\lambda }_A{\mathit{\beta }}^{\prime }{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta }+{\lambda }_U{\langle {\eta }_{{\stackrel{\sim }{U}}_1}(·),{\eta }_{{\stackrel{\sim }{U}}_1}(·)\rangle }_{{\mathcal{H}}_{{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}}}+{\lambda }_A{\langle {\eta }_{{\stackrel{\sim }{A}}_1}(·),{\eta }_{{\stackrel{\sim }{A}}_1}(·)\rangle }_{{\mathcal{H}}_{{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}}},$$

which is minimized when 〈 η_Ũ1 (·), η_Ũ1 (·)〉 = 〈 η_Ã1 (·), η_Ã1 (·)〉 = 0. Thus, η_Ũ1 (·) = η_Ã1 (·) = 0 and we obtain the forms of Û(t) and nPSS(t) as required.

To obtain the coefficients μ, ν, α and β, we first take partial derivatives of nested penalized sum-of-squares nPSS with respective to μ, ν, α and β and set them to zeros:

$$\frac{\partial \text{nPSS}}{\partial \mathit{\mu }}={\mathit{\phi }}_{\mu }^{\prime }({\mathit{\phi }}_{\mu }\mathit{\mu }+{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\mathit{\phi }}_{\alpha }\mathit{\alpha }+{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta }-\mathit{Y})=\mathbf{0},$$ (10)

$$\frac{\partial \text{nPSS}}{\partial \mathit{\nu }}={\mathit{R}}_{\stackrel{\sim }{U}}({\mathit{\phi }}_{\mu }\mathit{\mu }+{\mathit{M}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\mathit{\phi }}_{\alpha }\mathit{\alpha }+{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta }-\mathit{Y})=\mathbf{0},$$ (11)

$$\frac{\partial \text{nPSS}}{\partial \mathit{\alpha }}={\mathit{\phi }}_{\alpha }^{\prime }({\mathit{\phi }}_{\mu }\mathit{\mu }+{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\mathit{\phi }}_{\alpha }\mathit{\alpha }+{\mathit{R}}_{\stackrel{\sim }{A}}\mathit{\beta }-\mathit{Y})=\mathbf{0},$$ (12)

$$\frac{\partial \text{nPSS}}{\partial \mathit{\beta }}={\mathit{R}}_{\stackrel{\sim }{A}}({\mathit{\phi }}_{\mu }\mathit{\mu }+{\mathit{R}}_{\stackrel{\sim }{U}}\mathit{\nu }+{\mathit{\phi }}_{\alpha }\mathit{\alpha }+{\mathit{M}}_{\stackrel{\sim }{A}}\mathit{\beta }-\mathit{Y})=\mathbf{0},$$ (13)

where M_Ũ = R_Ũ + Jλ_UI and M_Ã = R_Ã + Jλ_AI. It follows from equations (11) and (13) that

$$\begin{array}{l}\mathit{\nu }={\mathit{S}}^{-1}(\mathit{Y}-{\mathit{\phi }}_{\mu }\mathit{\mu }-{\mathit{\phi }}_{\alpha }\mathit{\alpha }),\\ \mathit{\beta }=\frac{{\lambda }_U}{{\lambda }_A}\mathit{\nu }.\end{array}$$

Substituting them into equations (10) and (12) with some algebra leads to

$$\begin{array}{l}{\mathbf{\sum }}_{\mu \mu }\mathit{\mu }+{\mathbf{\sum }}_{\mu \alpha }\mathit{\alpha }={\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}^{-1}\mathit{Y},\\ {\mathbf{\sum }}_{\alpha \mu }\mathit{\mu }+{\mathbf{\sum }}_{\alpha \alpha }\mathit{\alpha }={\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}\mathit{Y},\end{array}$$

from which we obtain

$$\begin{array}{l}\mathit{\mu }={({\mathbf{\sum }}_{\mu \mu }-{\mathbf{\sum }}_{\mu \alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathbf{\sum }}_{\alpha \mu })}^{-1}({\mathit{\phi }}_{\mu }^{\prime }-{\mathbf{\sum }}_{\mu \alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathit{\phi }}_{\alpha }^{\prime }){\mathit{S}}^{-1}\mathit{Y}={\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }{\mathit{S}}^{-1}\mathit{Y},\\ \mathit{\alpha }={({\mathbf{\sum }}_{\alpha \alpha }-{\mathbf{\sum }}_{\alpha \mu }{\mathbf{\sum }}_{\mu \mu }^{-1}{\mathbf{\sum }}_{\mu \alpha })}^{-1}({\mathit{\phi }}_{\alpha }^{\prime }-{\mathbf{\sum }}_{\alpha \mu }{\mathbf{\sum }}_{\mu \mu }^{-1}{\mathit{\phi }}_{\mu }^{\prime }){\mathit{S}}^{-1}\mathit{Y}={\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }{\mathit{S}}^{-1}\mathit{Y}.\end{array}$$

It is then straightforward to show

$$\begin{array}{l}\mathit{\nu }={\mathit{S}}^{-1}\left\{\mathit{I}-\left({\mathit{\phi }}_{\mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }+{\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }\right){\mathit{S}}^{-1}\right\}\mathit{Y},\\ \mathit{\beta }=\frac{{\lambda }_U}{{\lambda }_A}\mathit{\nu }\end{array}$$

as desired.

A.5 Proof of Theorem 4

The proof is based on the following results of Lemma 2.

Lemma 2

For the observations Y = {Y(t₁), Y(t₂), …, Y(t_J)}′ and the nested Gaussian process U(t), we have

$$\begin{array}{l}E\{U(t)\}=0,\\ E\{\mathit{Y}\}=\mathbf{0},\\ \mathit{Cov}\{U(t),\mathit{Y}\}={\sigma }_{\mu }^2{\mathit{\phi }}_{\mu }^{\prime }(t){\mathit{\phi }}_{\mu }^{\prime }+{\sigma }_U^2{\mathit{R}}_{\stackrel{\sim }{U}}^{\prime }(t)+{\sigma }_{\alpha }^2{\mathit{\phi }}_{\alpha }^{\prime }(t){\mathit{\phi }}_{\alpha }^{\prime }+{\sigma }_A^2{\mathit{R}}_{\stackrel{\sim }{A}}^{\prime }(t),\\ \mathit{Cov}\{\mathit{Y},\mathit{Y}\}={\sigma }_{\mu }^2{\mathit{\phi }}_{\mu }{\mathit{\phi }}_{\mu }^{\prime }+{\sigma }_U^2{\mathit{R}}_{\stackrel{\sim }{U}}+{\sigma }_{\alpha }^2{\mathit{\phi }}_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }+{\sigma }_A^2{\mathit{R}}_{\stackrel{\sim }{A}}+{\sigma }_{\epsilon }^2\mathit{I}.\end{array}$$

Proof

Let U(t) = Ũ(t) + Ã(t) and A(t) = D^m Ã(t). From SDEs (2) and (3),

$$\begin{array}{r}{D}^m\stackrel{\sim }{U}(t)={\sigma }_U{\stackrel{.}{W}}_U(t),\\ D^{m+n}\stackrel{\sim }{A}(t)={\sigma }_A{\stackrel{.}{W}}_A(t).\end{array}$$

Thus, given the initial value μ, it can be shown that $\stackrel{\sim }{U}(t)={\sum }_{i=0}^{m-1}{\mu }_i{\phi }_i(t)+{\sigma }_U^2{\int }_{\mathcal{T}}{G}_m(t,u){\stackrel{.}{W}}_U(u)du$, a (m−1)-fold integrated Wiener process (Shepp, 1966). Similarly, $\stackrel{\sim }{A}(t)={\sum }_{i=0}^{n-1}{\alpha }_i{\phi }_{m+i}(t)+{\sigma }_A^2{\int }_{\mathcal{T}}{G}_{m+n}(t,u){\stackrel{.}{W}}_A(u)du$, a (m + n − 1)-fold integrated Wiener process.

It is obvious that E {U(t)} = 0 and E {Y} = 0. Given the mutually independent assumption of μ, α, Ẇ_U (·) and Ẇ_A(·),

$$\begin{array}{l}\text{Cov}\{U(t),Y(t_j)\}=\text{Cov}\{U(t),U(t_j)\}\\ =\text{Cov}\left\{\stackrel{\sim }{U}(t),\stackrel{\sim }{U}(t_j)\right\}+\text{Cov}\left\{\stackrel{\sim }{A}(t),\stackrel{\sim }{A}(t_j)\right\}\\ =\mathrm{E}\left\{\stackrel{\sim }{U}(t)\stackrel{\sim }{U}(t_j)\right\}+\mathrm{E}\left\{\stackrel{\sim }{A}(t)\stackrel{\sim }{A}(t_j)\right\}\\ ={\sigma }_{\mu }^2\sum _{i=0}^{m-1}{\phi }_i(t){\phi }_i(t_j)+{\sigma }_U^2{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t,t_j)+{\sigma }_{\alpha }^2\sum _{i=0}^{n-1}{\phi }_{m+i}(t){\phi }_{m+i}(t_j)+{\sigma }_A^2{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t,t_j),\end{array}$$

and

$$\begin{array}{l}\text{Cov}\{Y(t_j),Y(t_{j^{\prime }})\}=\text{Cov}\{U(t_j),U(t_{j^{\prime }})\}+{\sigma }_{\epsilon }^2\\ ={\sigma }_{\mu }^2\sum _{i=0}^{m-1}{\phi }_i(t_j){\phi }_i(t_{j^{\prime }})+{\sigma }_U^2{\mathcal{R}}_{{\stackrel{\sim }{U}}_1}(t_j,t_{J^{\prime }})+{\sigma }_{\alpha }^2\sum _{i=0}^{n-1}{\phi }_{m+i}(t_j){\phi }_{m+i}(t_{j^{\prime }})+{\sigma }_A^2{\mathcal{R}}_{{\stackrel{\sim }{A}}_1}(t_j,t_{j^{\prime }})+{\sigma }_{\epsilon }^2\end{array}$$

for j = 1, 2, ···, J and j′ = 1, 2, ···, J. The lemma holds.

The proof of Theorem 4 is given as follows. By Lemma 2 and the results on conditional multivariate normal distribution (Searle, 1982),

$$\begin{array}{l}\mathrm{E}\{U(t)\mid \mathit{Y},{\sigma }_{\mu }^2,{\sigma }_{\alpha }^2,{\sigma }_U^2,{\sigma }_A^2,{\sigma }_{\epsilon }^2\}=\text{Cov}\{U(t),\mathit{Y}\}{\text{Cov}}^{-1}\{\mathit{Y},\mathit{Y}\}\mathit{Y}\\ =[{\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }(t){\mathit{\phi }}_{\mu }^{\prime }+{\mathit{R}}_{\stackrel{\sim }{U}}^{\prime }(t)+{\rho }_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }(t){\mathit{\phi }}_{\alpha }^{\prime }+{\rho }_A{\mathit{R}}_{\stackrel{\sim }{A}}^{\prime }(t)]\times {[{\rho }_{\mu }{\mathit{\phi }}_{\mu }{\mathit{\phi }}_{\mu }^{\prime }+{\rho }_{\alpha }{\mathit{\phi }}_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }+{\rho }_A{\mathit{R}}_{\stackrel{\sim }{A}}+{\mathit{R}}_{\stackrel{\sim }{U}}+J{\lambda }_U\mathit{I}]}^{-1}\mathit{Y}\\ ={\mathit{\phi }}_{\mu }^{\prime }(t)\left({\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\right)\mathit{Y}+{\mathit{R}}_{\stackrel{\sim }{U}}^{\prime }(t){\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\mathit{Y}+{\mathit{\phi }}_{\alpha }^{\prime }(t)\left({\rho }_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\right)\mathit{Y}+{\rho }_A{\mathit{R}}_{\stackrel{\sim }{A}}^{\prime }(t){\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\mathit{Y}\end{array}$$

where ${\rho }_{\mu }={\sigma }_{\mu }^2/{\sigma }_U^2,{\rho }_{\alpha }={\sigma }_{\alpha }^2/{\sigma }_U^2,{\rho }_A={\sigma }_A^2/{\sigma }_U^2,J{\lambda }_U={\sigma }_{\epsilon }^2/{\sigma }_U^2$ and ${\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}={\rho }_{\mu }{\mathit{\phi }}_{\mu }{\mathit{\phi }}_{\mu }^{\prime }+{\mathit{S}}_{{\rho }_{\alpha }}$ with ${\mathit{S}}_{{\rho }_{\alpha }}={\rho }_{\alpha }{\mathit{\phi }}_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }+\mathit{S}={\rho }_{\alpha }{\mathit{\phi }}_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }+{\rho }_A{\mathit{R}}_{\stackrel{\sim }{A}}+{\mathit{R}}_{\stackrel{\sim }{U}}+J{\lambda }_U\mathit{I}$. We are going to evaluate the limits of ${\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1},{\rho }_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}$ and ${\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}$ when ρ_μ → +∞ and ρ_α → + ∞

It can be verified that

$$\begin{array}{l}{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}={\mathit{S}}_{{\rho }_{\alpha }}^{-1}-{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }{\left({\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }\right)}^{-1}{\left\{\mathit{I}+{\rho }_{\mu }^{-1}{\left({\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }\right)}^{-1}\right\}}^{-1}{\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1},\\ {\mathit{S}}_{{\rho }_{\alpha }}^{-1}={\mathit{S}}^{-1}-{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }{\left({\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }\right)}^{-1}{\left\{\mathit{I}+{\rho }_{\alpha }^{-1}{\left({\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }\right)}^{-1}\right\}}^{-1}{\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}.\end{array}$$ (14)

It follows that ${\mathit{S}}_{\infty }^{-1}=\underset{{\rho }_{\alpha }\to +\infty }{lim}{\mathit{S}}_{{\rho }_{\alpha }}^{-1}={\mathit{S}}^{-1}-{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }{({\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha })}^{-1}{\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}={\mathit{S}}^{-1}-{\mathit{S}}^{-1}{\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathit{\phi }}_{\alpha }^{\prime }{\mathit{S}}^{-1}$ and ${\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{\infty }^{-1}{\mathit{\phi }}_{\mu }={\mathbf{\sum }}_{\mu \mid \alpha }$ and ${\mathit{S}}_{\infty }^{-1}{\mathit{\phi }}_{\mu }={\mathit{S}}^{-1}{\mathit{\phi }}_{\mu \mid \alpha }^{\prime }$.

As a result,

$$\begin{array}{l}{\mathbf{\sum }}_{\infty \infty }^{-1}=\underset{{\rho }_{\mu }\to +\infty }{lim}\underset{{\rho }_{\alpha }\to +\infty }{lim}{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\\ ={\mathit{S}}^{-1}-{\mathit{S}}^{-1}\left({\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathit{\phi }}_{\alpha }^{\prime }+{\mathit{\phi }}_{\mu \mid \alpha }^{\prime }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }\right){\mathit{S}}^{-1}\\ ={\mathit{S}}^{-1}-{\mathit{S}}^{-1}\left\{{\mathit{\phi }}_{\mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }+{\mathit{\phi }}_{\alpha }\left({\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathit{\phi }}_{\alpha }^{\prime }-{\mathbf{\sum }}_{\alpha \alpha }^{-1}{\mathbf{\sum }}_{\alpha \mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }\right)\right\}{\mathit{S}}^{-1}\\ ={\mathit{S}}^{-1}\left\{\mathit{I}-\left({\mathit{\phi }}_{\mu }{\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }+{\mathit{\phi }}_{\alpha }{\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }\right){\mathit{S}}^{-1}\right\}.\end{array}$$

By expression (14),

$$\begin{array}{l}{\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}={\rho }_{\mu }\left[\mathit{I}-{\left\{\mathit{I}+{\rho }_{\mu }^{-1}{\left({\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }\right)}^{-1}\right\}}^{-1}\right]{\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}\\ ={\left({\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }\right)}^{-1}{\left\{\mathit{I}+{\rho }_{\mu }^{-1}{\left({\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}{\mathit{\phi }}_{\mu }\right)}^{-1}\right\}}^{-1}{\mathit{\phi }}_{\mu }^{\prime }{\mathit{S}}_{{\rho }_{\alpha }}^{-1}.\end{array}$$

It follows that $\underset{{\rho }_{\mu }\to +\infty }{lim}\underset{{\rho }_{\alpha }\to +\infty }{lim}{\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}={\mathbf{\sum }}_{\mu \mid \alpha }^{-1}{\mathit{\phi }}_{\mu \mid \alpha }{\mathit{S}}^{-1}$. By similar arguments, $\underset{{\rho }_{\mu }\to +\infty }{lim}\underset{{\rho }_{\alpha }\to +\infty }{lim}{\rho }_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}={\mathbf{\sum }}_{\alpha \mid \mu }^{-1}{\mathit{\phi }}_{\alpha \mid \mu }{\mathit{S}}^{-1}$.

Hence, ${\mathbf{\sum }}_{\infty \infty }^{-1}\mathit{Y}=\mathit{\nu },{\rho }_A{\mathbf{\sum }}_{\infty \infty }^{-1}\mathit{Y}=\mathit{\beta },\underset{{\rho }_{\mu }\to +\infty }{lim}\underset{{\rho }_{\alpha }\to +\infty }{lim}{\rho }_{\mu }{\mathit{\phi }}_{\mu }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\mathit{Y}=\mathit{\mu }$ and $\underset{{\rho }_{\mu }\to +\infty }{lim}\underset{{\rho }_{\alpha }\to +\infty }{lim}{\rho }_{\alpha }{\mathit{\phi }}_{\alpha }^{\prime }{\mathbf{\sum }}_{{\rho }_{\mu }{\rho }_{\alpha }}^{-1}\mathit{Y}=\mathit{\alpha }$. The theorem holds.

A.6 Proof of the Exact State Equation

When m = 2 and n = 1, the SDEs (2) and (3) can be written as,

$$D^1\mathit{\theta }(t)=\mathit{C}\mathit{\theta }(t)+\mathit{D}\stackrel{.}{\mathit{W}}(t),$$

where $\mathit{\theta }(t)=\left\{\begin{array}{c}U(t)\\ D^{1}U(t)\\ A(t)\end{array}\right\},\mathit{C}=\left(\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right),\mathit{D}=\left(\begin{array}{cc}0& 0\\ {\sigma }_U& 0\\ 0& {\sigma }_A\end{array}\right)$ and $\stackrel{.}{\mathit{W}}(t)=\left\{\begin{array}{c}{\stackrel{.}{W}}_U(t)\\ {\stackrel{.}{W}}_A(t)\end{array}\right\}$.

As a result,

$$\begin{array}{l}{\mathit{\theta }}_{j+1}=exp(\mathit{C}{\delta }_j){\mathit{\theta }}_j+{\int }_0^{{\delta }_j}exp\{\mathit{C}({\delta }_j-u)\}\mathit{D}\stackrel{.}{\mathit{W}}(t_j+u)du\\ ={\mathit{G}}_j{\mathit{\theta }}_j+{\mathit{\omega }}_j,\end{array}$$

where ${\mathit{G}}_j=exp(\mathit{C}{\delta }_j)=\mathit{I}+{\delta }_j\mathit{C}+{\delta }_j^2\mathit{CC}/2=\left(\begin{array}{ccc}1& {\delta }_j& \frac{{\delta }_j^2}{2}\\ 0& 1& {\delta }_j\\ 0& 0& 1\end{array}\right)$ and ω_j ~ N₃ (0, W_j) with

$$\begin{array}{l}{\mathit{W}}_j={\int }_0^{{\delta }_j}exp\{\mathit{C}({\delta }_j-u)\}{\mathit{DD}}^{\prime }exp\{{\mathit{C}}^{\prime }({\delta }_j-u)\}du\\ =\left(\begin{array}{ccc}\frac{{\delta }_j^3}{3}{\sigma }_U^2+\frac{{\delta }_j^5}{20}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_U^2+\frac{{\delta }_j^4}{8}{\sigma }_A^2& \frac{{\delta }_j^3}{6}{\sigma }_A^2\\ \frac{{\delta }_j^2}{2}{\sigma }_U^2+\frac{{\delta }_j^4}{8}{\sigma }_A^2& {\delta }_j{\sigma }_U^2+\frac{{\delta }_j^3}{3}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_A^2\\ \frac{{\delta }_j^3}{6}{\sigma }_A^2& \frac{{\delta }_j^2}{2}{\sigma }_A^2& {\delta }_j{\sigma }_A^2\end{array}\right)\end{array}$$

as required.