Stochastic dynamic models and Chebyshev splines

Abstract

In this article, we establish a connection between a stochastic dynamic model (SDM) driven by a linear stochastic differential equation (SDE) and a Chebyshev spline, which enables researchers to borrow strength across fields both theoretically and numerically. We construct a differential operator for the penalty function and develop a reproducing kernel Hilbert space (RKHS) induced by the SDM and the Chebyshev spline. The general form of the linear SDE allows us to extend the well-known connection between an integrated Brownian motion and a polynomial spline to a connection between more complex diffusion processes and Chebyshev splines. One interesting special case is connection between an integrated Ornstein–Uhlenbeck process and an exponential spline. We use two real data sets to illustrate the integrated Ornstein–Uhlenbeck process model and exponential spline model and show their estimates are almost identical.

1. INTRODUCTION

There exists a one-to-one correspondence between a reproducing kernel Hilbert space (RKHS) and Gaussian stochastic processes by the Kolmogorov consistency theorem (Cramér & Leadbetter, 1967; Wahba, 1990). This correspondence has led to the development of connections between general smoothing spline models and Gaussian stochastic processes which enables researchers to borrow strength across different fields. For example, stochastic models corresponding to smoothing splines have been used to derive the generalized maximum likelihood estimate of smoothing parameters and to construct Bayesian confidence intervals (Wahba, 1990). A specific connection between an L-spline and an integrated Brownian motion has been explored to develop efficient O(n) algorithms (Wecker & Ansley, 1983).

Specific connections between more complex processes and related smoothing splines have not been studied extensively. The purpose of this article is to establish a connection between a stochastic dynamic model (SDM) driven by a linear stochastic differential equation (SDE) and a Chebyshev spline. We construct a differential operator for the penalty function and develop an RKHS corresponding to the SDM and the Chebyshev spline. The general form of the SDE allows us to establish a connection between more complex diffusion processes and Chebyshev splines. As an interesting special case, an integrated Ornstein–Uhlenbeck process will be connected to an exponential spline.

The extended connection between the SDM driven by a linear SDE and a Chebyshev spline can be explored to motivate new SDMs based on spline models, and vice versa. As an illustration, we will present a partial spline model motivated by an SDM and an SDM motivated by logistic spline. We differ from the smoothing spline literature where one usually builds SDMs that connect to spline models of interest; here we will build spline models that connect to SDMs. We will present the construction of penalties corresponding to SDMs, which may be regarded as priors for regression functions in non-parametric regression models.

Estimation and computational methods for Chebyshev splines may be used to compute the posterior means of the corresponding SDMs, and vice versa. We will use two real data sets to show that the posterior mean of the integrated Ornstein–Uhlenbeck process in an SDM matches the penalized least squares estimate of the corresponding exponential spline.

Gaussian stochastic models have been widely used in many fields including physics, engineering, finance and biology (Ansley & Kohn, 1986; Rue & Martino, 2009; Stuart, 2010; Griebel & Hegland, 2010; Lindgren, Rue, & Lindstrom, 2011; Papaspiliopoulis et al., 2012). Some of the interesting special cases are given in Bishwal (2008). On the other hand, much work has been done in the area of smoothing splines. For instance, Pintore, Speckman, & Holmes (2006) used an RKHS representation to derive the smoothing spline with a particular inhomogeneous differential operator penalty; Furrer & Nychka (2007) built a framework to understand the asymptotic properties of Kriging and splines; in this framework Kriging estimators are interpreted as generalized smoothing splines. The differential operator L = D^q in Pintore, Speckman, & Holmes (2006) is a special case of that considered in this paper.

The remainder of this article is organized as follows. In Section 2, we introduce the SDM. In Section 3, we compute mean and covariance functions for a general class of Gaussian models driven by a linear SDE and construct corresponding Chebyshev splines. We extend the SDM in Section 4 with general differential operators and connect them to general Chebyshev splines. We use two real data sets to confirm the theoretical results in Section 5 and conclude with some remarks in Section 6.

2. STOCHASTIC DYNAMIC MODELS

We present an SDM driven by an SDE in the Section 2.1. Then we develop an equivalent stochastic integration equation and use this equivalent model to compute the mean and covariance functions of the dynamic system in Section 2.2.

2.1. Stochastic Dynamic Models

Consider a temporal SDM

$$Y(t,\omega )=U(t,\omega )+\epsilon (t,\omega ),\omega \in \mathrm{\Omega },t\in [0,T],$$ (1)

where Y(t, ω) is the observation at time t, U(t, ω) is a latent stochastic process of interest observed at time t on the path ω, ε(t, ω) is an error term, Ω is the sample path probability space, and T is a positive real number which may be the right end point of time in consideration. We assume that U(·) and ε(·) are independent and ε(·) is a Gaussian white noise process with variance σ².

In this paper, we will first consider the following stochastic dynamic system for the latent stochastic process U(·)

$$\frac{d^{q}U(t)}{\mathrm{d}{t}^q}=a_0(t)U_q+b(t)V(t),q\ge 0,t\in [0,T],$$ (2)

where U_k = d^kU(0)/dt^k for k = 0, …, q are initial values of U(·) and its derivatives up to the order q, and a₀(t) and b(t) are integrable deterministic functions. We assume that U₀, …, U_q are mutually independent and square integrable random variables. A more general equation for U(·) that corresponds to the general Chebyshev splines will be considered in Section 4.

We assume that the stochastic process V(·) in (2) is independent of U₀, …, U_q and is driven by the following SDE

$$\mathrm{d}V(t,\omega )=\beta (t,V(t,\omega ))\mathrm{d}t+\sigma (t,V(t,\omega ))\mathrm{d}B(t,\omega ),V(0,\omega )=0,t\in [0,T],$$ (3)

where β(t, V(t, ω)) is a drift term, σ²(t, V(t, ω)) is a diffusion coefficient, and B(t, ω) is a standard Brownian motion observed at time t on the path ω. We assume that the SDE (3) has a unique, continuous and adapted strong solution, and V(·) has finite moments of any order p ∈ [1,∞). This assumption holds when the coefficients in (3) satisfy the Lipschitz condition, |σ(t, x) − σ(t, y)|² + |β(t, x) − β(t, y)|² ≤ c₁(T)|x − y|², and the growth condition, |σ(t, x)|² + |β(t, x)|² ≤ c₂(T)(1 + x²) for all x, y ∈ R and t ∈ [0, T], where c₁(T) and c₂(T) are positive constants depending on T only (Ikeda & Watanabe, 1989).

Together, Equations (1), (2) and (3) define the SDM we study in this paper. The SDE (3) is a parametric Itô SDE with initial value equal to zero. The assumption about the initial value may be removed (see Remark 3.1). Zhu, Song, & Taylor (2011) considered a similar SDM with a₀(t) = 0 and b(t) = 1 in the stochastic dynamic system (2) without the initial condition V(0) = 0, and used an Ornstein–Uhlenbeck process V(·) to model the rate of changes of prostate specific antigen profiles. We consider the more general Equation (2) for the connection to Chebyshev splines.

It is noteworthy that a special case of the SDM has been connected to polynomial smoothing splines. Specifically, let a₀(t) = b(t) = 1, β(t, V(t)) = 0 and σ(t, V(t)) = σ_V, then U(·) is the same as the random effects model (1.5.8) in Wahba (1990) for a polynomial spline of order q + 1 (refer to Section S.5 in the Supplementary Materials for details). When q = 1 and V(·) is a standard Brownian motion, then U(·) is an integrated Brownian motion. Alternatively, one may want to use an Ornstein–Uhlenbeck process V(·) to model exponentially decreasing correlation. In this case, U(·) is an integrated Ornstein–Uhlenbeck process.

2.2. An Equivalent Stochastic Integration Equation

In this section, we first introduce a stochastic integration equation that is equivalent to the stochastic dynamic system (2). We then use the stochastic integration equation to compute the mean and covariance functions of the stochastic process U(·).

When q = 0, the stochastic dynamic system (2) has the form

$$U(t)=a_0(t)U_0+b(t)V(t),$$ (4)

and it is straightforward to compute the mean and covariance functions of U(·) in this case.

In the following discussion, we assume that q ≥ 1.We will show that the following stochastic integration equation is equivalent to the stochastic dynamic system (2)

$$U(t)=\sum _{i=0}^{q-1}{\psi }_i(t)U_i+A_q(t)U_q+{\int }_0^t{\psi }_q(t-s)\mathrm{d}[b(s)V(s)],$$ (5)

where ψ_i(t) ≜ tⁱ/i!, $A_q(t)\triangleq {\int }_0^t{\psi }_{q-1}(t-s)a_0(s)\mathrm{d}s$, and V(·) is a stochastic process driven by the SDE defined in (3). We first show that (5) can be rewritten as

$$U(t)=\sum _{i=0}^{q-1}{\psi }_i(t)U_i+A_q(t)U_q+{\int }_0^t{\psi }_{q-1}(t-s)b(s)V(s)\mathrm{d}s.$$ (6)

To see that (5) can be rewritten as (6), applying Itô formula to b(s)V(s)(t − s)^q, s ≤ t, leads to

$${\int }_0^t{(t-s)}^q\mathrm{d}[b(s)V(s)]+{\int }_0^{t}b(s)V(s)\mathrm{d}{(t-s)}^q=b(s)V(s){(t-s)}^q{|}_0^t=0.$$

Hence, we have

$${\int }_0^t{(t-s)}^q\mathrm{d}[b(s)V(s)]=q{\int }_0^{t}b(s)V(s){(t-s)}^{q-1}\mathrm{d}s.$$ (7)

Suppose that U(·) is given by the stochastic integration Equation (5). By Equation (7), we have the expression (6).

When q = 1, it is easy to see that taking the derivative on both sides of (6) with respect to t leads to (2). When q ≥ 2, taking the derivative on both sides of (6) with respect to t leads to

$$\frac{\mathrm{d}U(t)}{\mathrm{d}t}=\sum _{i=1}^{q-1}{\psi }_{i-1}(t)U_i+A_{q-1}(t)U_q+{\int }_0^t{\psi }_{q-2}(t-s)b(s)V(s)\mathrm{d}s+[{\psi }_{q-1}(t-s)b(s)V(s)]{|}_{s=t}=\sum _{i=1}^{q-1}{\psi }_{i-1}(t)U_i+A_{q-1}(t)U_q+{\int }_0^t{\psi }_{q-2}(t-s)b(s)V(s)\mathrm{d}s.$$

Continuing this process to calculate higher order derivatives, we get d^qU(t)/dt^q = a₀(t)U_q + b(t)V(t). Therefore, the stochastic integration Equation (5) or (6) implies the stochastic differential system (2).

Conversely, assuming that U(·) is given by the stochastic differential system (2) and reversing the process above, it is not difficult to show that U(·) satisfies the stochastic integration Equation (5) or (6). The following lemma summarizes the above discussion.

Lemma 2.1

Suppose that the stochastic process V(·) is driven by the SDE (3). Then, the stochastic dynamic system (2) is equivalent to the stochastic integration Equation (5) or (6).

Note that when a₀(t) = b(t) = 1 and V(t) = σ_V B(t), we have A_q(t) = ψ_q(t) and $U(t)={\sum }_{i=0}^q{\psi }_i(t)U_i+{\sigma }_V{\int }_0^t{\psi }_q(t-s)\mathrm{d}B(s)$ which is the stochastic process corresponding to the polynomial spline of order q + 1 (Kimeldorf & Wahba, 1970a,b; Wahba, 1990).

The mean and covariance functions of U(·) listed in the following proposition can be computed directly based on (4) when q = 0 or by applying the Fubini’s theorem to Equation (6) when q ≥ 1.

Proposition 2.1

For the stochastic process U(·) defined in (2) or equivalently in (5) or (6), when q ≥ 1, we have

$$\mathrm{E}U(t)=\sum _{i=0}^{q-1}{\psi }_i(t){\mu }_i+A_q(t){\mu }_q+{\int }_0^t{\psi }_{q-1}(t-s)b(s)\mathrm{E}V(s)\mathrm{d}s,$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^{q-1}{\psi }_i(s){\psi }_i(t){\sigma }_i^2+A_q(s)A_q(t){\sigma }_q^2+{\int }_0^t{\int }_0^s{\psi }_{q-1}(t-u){\psi }_{q-1}(s-\upsilon )b(u)b(\upsilon )\text{Cov}(V(u),V(\upsilon ))\mathrm{d}u\mathrm{d}\upsilon ,$$

where μ_i ≜ E U_i and ${\sigma }_i^2\triangleq \text{Var}(U_i)$ When q = 0, we have E U(t) = a₀(t)μ₀ + b(t) E V(t) and $\text{Cov}(U(s),U(t))=a_0(s)a_0(t){\sigma }_0^2+b(s)b(t)\text{Cov}(V(s),V(t))$.

There exists a unique RKHS that is congruent to the Hilbert space generated by the stochastic process U(·) (Berlinet & Thomas-Agnan, 2004). In the next section, we will make specific connections between Gaussian processes driven by a linear SDE and an RKHS.

3. GAUSSIAN PROCESS DRIVEN BY A LINEAR SDE AND CONNECTION TO CHEBYSHEV SPLINES

The stochastic process V(·) driven by the SDE (3) is a Markovian process. However, it is not necessarily a Gaussian process. In this section, we consider a Gaussian process driven by the following linear SDE

$$\mathrm{d}V(t)=[{\beta }_0(t)+{\beta }_1(t)V(t)]\mathrm{d}t+\sigma (t)\mathrm{d}B(t),V(0)=0,t\in [0,T],$$ (8)

where β₀(t), β₁(t) and σ(t) are deterministic, measurable, and bounded functions of time t (Karatzas & Shreve, 1988). The SDE (8) is a special case of (3) when β(t, V(t, ω)) is linear in V(·) and σ(t, V(t, ω)) is deterministic.

In this section, we establish a connection between the SDM where V(·) is driven by the linear SDE (8) and Chebyshev splines. In Section 3.1, we apply results in Section 2.2 to compute the mean and covariance functions for Gaussian processes driven by the linear SDE. In Section 3.2, we build connections between Gaussian processes and Chebyshev splines. In the subsections, we provide examples of some specific SDMs and their corresponding spline models.

3.1. Gaussian Process Driven by the Linear SDE (8)

Denote $\mathrm{\Psi }(t)\triangleq \text{exp}({\int }_0^t{\beta }_1(s)\mathrm{d}s)$. By the Itô formula, one can show that (Karatzas & Shreve, 1988, pp. 354–355)

$$V(t)=\mathrm{\Psi }(t)[{\int }_0^t\frac{{\beta }_0(s)}{\mathrm{\Psi }(s)}\mathrm{d}s+{\int }_0^t\frac{\sigma (s)}{\mathrm{\Psi }(s)}\mathrm{d}B(s)].$$ (9)

Lemma 3.1

Suppose that the stochastic process V(·) is a Gaussian process driven by the linear SDE (8). Then, we have

$$\mathrm{E}V(t)=\mathrm{\Psi }(t){\int }_0^t\frac{{\beta }_0(s)}{\mathrm{\Psi }(s)}\mathrm{d}s,$$

$$\text{Cov}(V(s),V(t))=\mathrm{\Psi }(s)\mathrm{\Psi }(t){\int }_0^{s\wedge t}{\left(\frac{\sigma (u)}{\mathrm{\Psi }(u)}\right)}^2\mathrm{d}u.$$

The proof of Lemma 3.1 can be found in the Supplementary Materials. For q ≥ 1, denote

$$F(t,u)\triangleq I(u\le t){\int }_u^t{\psi }_{q-1}(t-\tau )b(\tau )\mathrm{\Psi }(\tau )\mathrm{d}\tau ,$$

where I(u ≤ t) = 1 when u ≤ t and 0 otherwise. Combining results in Proposition 2.1 and Lemma 3.1, we have the following theorem.

Theorem3.1

Suppose that the stochastic process V(·) is a Gaussian process driven by the linear SDE (8) and $U_i~\mathrm{N}({\mu }_i,{\sigma }_i^2)$ for i = 0, …, q. Then, the stochastic process U(·) is a Gaussian process with the following mean and covariance functions

$$\mathrm{E}U(t)=\sum _{i=0}^{q-1}{\psi }_i(t){\mu }_i+A_q(t){\mu }_q+{\int }_0^{t}F(t,s)\frac{{\beta }_0(s)}{\mathrm{\Psi }(s)}\mathrm{d}s,$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^{q-1}{\psi }_i(s){\psi }_i(t){\sigma }_i^2+A_q(s)A_q(t){\sigma }_q^2+{\int }_0^{s\wedge t}F(s,u)F(t,u){\left(\frac{\sigma (u)}{\mathrm{\Psi }(u)}\right)}^2\mathrm{d}u,$$

where, for q = 0, we take ${\sum }_{i=0}^{q-1}\triangleq 0$, A₀(t) ≜ a₀(t), and F(t, u) ≜ b(t) Ψ(t)I(u ≤ t).

The proof of Theorem 3.1 is provided in the Supplementary Materials.

Remark 3.1

We have assumed that V(0) = 0 in the SDE (8). This assumption can be removed from the construction as follows. Let

$$\overline{V}(t)=\mathrm{\Psi }(t)[U_q+{\int }_0^t\frac{{\beta }_0(s)}{\mathrm{\Psi }(s)}\mathrm{d}s+{\int }_0^t\frac{\sigma (s)}{\mathrm{\Psi }(s)}\mathrm{d}B(s)].$$

Then, V̄(·) satisfies

$$\mathrm{d}\overline{V}(t)=[{\beta }_0(t)+{\beta }_1(t)\overline{V}(t)]\mathrm{d}t+\sigma (t)\mathrm{d}B(t),\overline{V}(0)=U_q,t\in [0,T].$$ (10)

When a₀(t) = Ψ(t), b(t) = 1, and V(·) is a solution of linear SDE (8), the stochastic differential system (2) is equivalent to

$$\frac{d^{q}U(t)}{\mathrm{d}{t}^q}=\overline{V}(t).$$ (11)

This means that d^qU(t)/dt^q is equal to the solution V̄(t) of the linear SDE (10) which starts from U_q. Both V̄(·) and U(·) defined in (11) are Gaussian processes.

3.2. Construction of a Chebyshev Spline Model

We now construct a Chebyshev spline model such that, up to a known function, the penalized least squares estimate equals the best linear unbiased estimator of the SDM defined by (1), (2) and (8).We note that the following development is in the opposite direction of the common approach employed in the spline literature where one constructs a stochastic process to connect with a smoothing spline model.

Define a differential operator

$$Lf(t)\triangleq \frac{d}{\mathrm{d}t}\left(\frac{f^{(q)}(t)}{\mathrm{\Psi }(t)}\right)=\frac{f^{(q+1)}(t)-{\beta }_1(t)f^{(q)}(t)}{\mathrm{\Psi }(t)}.$$ (12)

We may write L as Lf (u) = D [D^qf (u)/Ψ(u)] where D^q = d^q/dt^q is the qth derivative operator. The differential operator L is a special case of the general differential operator L_q+1 defined in Equation (4.64) of Gu (2013) for the Chebyshev spline with w_i(t) = 1 for i = 1, …, q and w_q+1(t) = Ψ(t). Then, from Equation (4.65) in Gu (2013), the Chebyshev system on [0, T] is ψ₀(t), ψ₁(t), …, ψ_q−1(t), ψ̃_q(t) and they span the null space ℋ₀ = span{ψ₀(t), ψ₁(t), …, ψ_q−1(t), ψ̃_q(t)} of the differential operator L, where

$${\tilde{\psi }}_q(t)={\int }_0^t{\psi }_{q-1}(t-s)\mathrm{\Psi }(s)\mathrm{d}s.$$

Under an inner product ${\sum }_{i=0}^q{f}^{(i)}(0)g^{(i)}(0)$, {ψ₀(t),ψ₁(t), …, ψ_q−1(t),ψ̃_q(t)} forms an orthonormal basis of the null space ℋ₀. Furthermore, it can be shown that the Green’s function associated with the differential operator L is

$$G(t,u)=I(u\le t){\int }_u^t{\psi }_{q-1}(t-\tau )\mathrm{\Psi }(\tau )\mathrm{d}\tau ,$$

which is given by the relation (4.67) in Gu (2013).

Consider the model space

$$W_2^{q+1}[0,T]\triangleq \{f:f,f\prime ,\dots ,f^{(q)}\text{are absolutely continuous and}{\int }_0^T{(Lf(t))}^{2}h(t)\mathrm{d}t<\infty \}$$ (13)

with an inner product

$$(f,g)\triangleq \sum _{i=0}^q{f}^{(i)}(0)g^{(i)}(0)+{\int }_0^{T}Lf(u)Lg(u)h(u)\mathrm{d}u,$$ (14)

where h(u) ≜ (Ψ(u)/σ(u))² is a weight function. Following the same arguments as in Section 4.5.2 of Gu (2013), we have the following result.

Theorem 3.2

Assume that a₀(t) = Ψ(t) and b(t) = 1. Then, $W_2^{q+1}[0,T]$ is an RKHS. Let us denote the kernel of L as ${\mathscr{H}}_0=\{f\in W_2^{q+1}[0,T]:Lf=0\}$. Then, ψ₀(t), ψ₁(t), …, ψ_q−1(t), A_q(t) = ψ̃_q(t) form an orthonormal basis of ℋ₀, and $W_2^{q+1}[0,T]$ can be decomposed into $W_2^{q+1}[0,T]={\mathscr{H}}_0\oplus {\mathscr{H}}_1$, where

$${\mathscr{H}}_0=\mathit{\text{span}}\{{\psi }_0(t),{\psi }_1(t),\dots ,{\psi }_{q-1}(t),{\tilde{\psi }}_q(t)\},$$

$${\mathscr{H}}_1=\{f\in W_2^{q+1}[0,T]:f(0)=f\prime (0)=\cdots =f^{(q)}(0)=0\}.$$

The reproducing kernels of ℋ₀ and ℋ₁ are respectively given by

$$R_0(s,t)=\sum _{i=0}^{q-1}{\psi }_i(s){\psi }_i(t)+{\tilde{\psi }}_q(s){\tilde{\psi }}_q(t),$$

$$R_1(s,t)={\int }_0^{s\wedge t}G(s,u)G(t,u){[h(u)]}^{-1}\mathrm{d}u.$$

The proof of Theorem 3.2 is straightforward based on the fact that A_q(t) = ψ̃_q(t) and G(t, u) = F(t, u) when a₀(t) = Ψ(t) and b(t) = 1.

Remark 3.2

In Theorem 3.2 we assumed that a₀(t) = Ψ(t). In general, assume that a₀(t) is strictly positive and a₀(0) = 1. It is not difficult to check that Dⁱψ_j(0) = 1 if i = j and 0 otherwise for 0 ≤ i, j ≤ q − 1. Furthermore, D^qψ_j(0) = 0 for 0 ≤ j ≤ q − 1, DⁱA_q(0) = 0 for 0 ≤ i ≤ q − 1, and D^qA_q(0) = a₀(0) = 1. Then {ψ₀(t), ψ₁(t), …, ψ_q−1(t), A_q(t)} forms an orthonormal basis of a subspace of $W_2^{q+1}[0,T]$ under the inner product ${\sum }_{i=0}^q{f}^{(i)}(0)g^{(i)}(0)$. In addition, we have Lψ_i(t) = 0 for i = 0, 1, …, q− 1. It is easy to check that D^qA^q(t) = a₀(t). Therefore, LA_q(t) = 0 iff a₀(t) = Ψ(t). Consequently, A_q(t) does not belong to the space ${\mathscr{H}}_0=\{f\in W_2^{q+1}[0,T]:Lf=0\}$ when a₀(t) ≠ Ψ(t). Nevertheless, when a₀(t) ≠ Ψ(t), a partial spline model may be constructed. See Section 3.3.2 for an example.

Consider the following nonparametric regression model

$$Y_i=f(t_i)+{\epsilon }_i,i=1,\dots ,n,t_i\in [0,T].$$ (15)

Assume that $f\in W_2^{q+1}[0,T]$. A Chebyshev spline is the solution to the following penalized least Squares

$$\underset{f\in W_2^{q+1}[0,T]}{\text{min}}\{\frac{1}{n}\sum _{i=1}^n{[Y_i-f(t_i)]}^2+\lambda {\int }_0^T{(Lf(t))}^{2}h(t)\mathrm{d}t\},$$ (16)

where λ is a smoothing parameter and L is given in (12). Let y = (Y₁, …, Y_n)′, $\mathrm{\Sigma }={\{R_1(t_i,t_j)\}}_{i,j=1}^n,S={\{({\psi }_0(t_i),\dots ,{\psi }_{q-1}(t_i),A_q(t_i))\}}_{i=1}^n$, and M = Σ + nλI_n, where I_n is an n × n identity matrix. The solution to (16) can be represented as (Wang, 2011)

$${\hat{f}}_{\lambda }(t)=\sum _{i=0}^{q-1}{d}_i{\psi }_i(t)+d_q{A}_q(t)+\sum _{k=1}^n{c}_i{R}_1(t,t_i),$$ (17)

Where

$$(d_0,\dots ,d_q)\prime ={(S\prime M^{-1}S)}^{-1}S\prime M^{-1}\mathbf{y},$$

$$(c_1,\dots ,c_n)\prime =M^{-1}[I_n-S{(S\prime M^{-1}S)}^{-1}S\prime M^{-1}]\mathbf{y}.$$

Now consider n observations based on the SDM (1)

$$Y(t_i)=U(t_i)+\epsilon (t_i),i=1,\dots ,n,t_i\in [0,T],$$ (18)

where the stochastic processes U(·) and V(·) are defined in (2) and (8), respectively. The best linear unbiased estimator of U(t) is the posterior mean E (U(t)|Y(t_i), i = 1, …, n) (Wahba, 1990). Denote

$$\mu (t)\triangleq {\int }_0^{t}G(t,s){\beta }_0(s)/\mathrm{\Psi }(s)\mathrm{d}s.$$ (19)

In this article, we assume that μ(t) is known. Subtracting μ(t_i) on both sides of (18) and following the same arguments as in Wahba (1990) and Gu (2013), we have the following connection between the Chebyshev spline and the best linear unbiased estimator of U(t).

Proposition 3.1

Assume that a₀(t) = Ψ(t), b(t) = 1, and U₀, …, $U_q\stackrel{iid}{~}\mathrm{N}(0,a)$. Denote

$${Û}_a(t)=\mathrm{E}(U(t)|Y(t_i),i=1,\dots ,n)$$

as the posterior mean where the dependence on the variance a is expressed explicitly. For any fixed t ∈ [0, T], when λ = σ²/n, we have

$$\underset{a\to \infty }{\text{lim}}{Û}_a(t)=\mu (t)+{\hat{f}}_{\lambda }(t),$$ (20)

where f̂_λ (t) in (20) is the penalized least squares solution to (16) with observations y = (Y(t₁) − μ(t₁), …, Y(t_n) − μ(t_n))′.

Remark 3.3

The penalty can be simplified as follows

$${\int }_0^T{(Lf(t))}^{2}h(t)\mathrm{d}t={\int }_0^T{\left[\frac{f^{(q+1)}(t)-{\beta }_1(t)f^{(q)}(t)}{\sigma (t)}\right]}^2\mathrm{d}t.$$ (21)

It is clear that the construction of RKHS including inner product, basis function of the space ℋ₀ and reproducing kernel of the space ℋ₁ is independent of the function β₀(t), while whether there exists a drift term μ(t) defined in (19) depends on if β₀(t) = 0. The condition n λ = σ² does not depend on σ(t) since σ(t) is involved in the penalty. When σ(t) is a constant, say σ_V, it may be absorbed into the smoothing parameter and then we have the standard condition $n\lambda ={\sigma }^2/{\sigma }_V^2$.

Remark 3.4

Assume that β₀(t) = 0. Then μ(t) = 0. Proposition 3.1 states that the best linear unbiased estimator of U(t) coincides with the smoothing spline estimate as a → ∞. This link has been explored to derive the generalized maximum likelihood (restricted maximum likelihood) estimate of the smoothing parameter λ. The variance function of Û_a(t) as a → ∞ has been used to construct Bayesian confidence intervals. See Wang (2011) for details.

Remark 3.5

Proposition 3.1 extends existing results to the case when μ(t) ≠ 0. Denote

$$\mathbf{Û}={\mathit{\text{lim}}}_{a\to \infty }(U_a(t_1),\dots ,U_a(t_n))\prime$$

as the best linear unbiased estimates at design points, Y = (Y(t₁), …, Y(t_n))′, and μ = (μ(t₁), …, μ(t_n))′. Then, it can be seen that

$$\mathbf{Û}=\mathit{\mu }+({\hat{f}}_{\lambda }(t_1),\dots ,{\hat{f}}_{\lambda }(t_n))\prime =\mathit{\mu }+H(\lambda )(\mathbf{Y}-\mathit{\mu })=(I_n-H(\lambda ))\mathit{\mu }+H(\lambda )\mathbf{Y},$$

Where H (λ) = I_n − nλM⁻¹[I_n − S(S′M⁻¹S)⁻¹S′M⁻¹] is the smoothing matrix. Thus, Û has the typical form of a shrinkage estimator.

3.3. Ornstein–Uhlenbeck Process and Exponential Spline

Consider an Ornstein–Uhlenbeck process V(·) that satisfies the SDE

$$\mathrm{d}V(t)=\theta (\mu -V(t))\mathrm{d}t+{\sigma }_V\mathrm{d}B(t),V(0)=0,$$ (22)

where μ is the equilibrium value of the process, θ > 0 is the speed of reversion, and σ_V is the volatility. It is easy to see that the Ornstein–Uhlenbeck process is a special case of (8) with β₀(t) = μθ, β₁(t) = −θ, and σ(t) = σ_V.

The mean and covariance functions of V(·) are

$$\mathrm{E}V(t)=\mu [1-\text{exp}(-\theta t)],$$

$$\text{Cov}(V(s),V(t))=\frac{{\sigma }_V^2}{2\theta }\text{exp}(-\theta |s-t|)[1-\text{exp}(-2\theta s\wedge t)].$$

In this subsection we assume that q ≥ 1. See Remark 3.7 for the exponential spline with q = 0. It is easy to check that Ψ(t) = exp(−θt), $h(t)={\mathrm{e}}^{-2\theta t}{\sigma }_V^{-2}$ and $F(t,u)=I(u\le t){\int }_u^t{\psi }_{q-1}(t-\tau ){\mathrm{e}}^{-\theta \tau }b(\tau )\mathrm{d}\tau$. Assume that $U_i~\mathrm{N}({\mu }_i,{\sigma }_i^2)$. Then the mean and covariance functions of U(·) are

$$\mathrm{E}U(t)=\sum _{i=0}^{q-1}{\psi }_i(t){\mu }_i+A_q(t){\mu }_q+\mu {\int }_0^t{\psi }_{q-1}(t-s)b(s)(1-{\mathrm{e}}^{-\theta s})\mathrm{d}s$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^{q-1}{\psi }_i(s){\psi }_i(t){\sigma }_i^2+A_q(s)A_q(t){\sigma }_q^2+{\sigma }_V^2{\int }_0^{s\wedge t}F(s,u)F(t,u){\mathrm{e}}^{2\theta u}\mathrm{d}u.$$

The differential operator in (12) reduces to Lf(u) = [f^(q+1)(u) + θf^(q)(u)] e^θu. Therefore, the penalty term in (16) is equal to $(\lambda /{\sigma }_V^2){\int }_0^T{[f^{(q+1)}(t)+\theta f^{(q)}(t)]}^2\mathrm{d}t$. Note that f^(q+1) + θf^(q) = D^q−1(D² + θD)f, where D² + θD is differential operator for exponential spline (Wang, 2011).

3.3.1. Exponential spline

Consider a special case of model (2) with a₀(t) = e^−θt and b(t) = 1. When q = 1, the stochastic process U(·) can be represented as a summation of drift terms and an integrated Ornstein–Uhlenbeck process

$$U(t)=U_0+U_1\frac{1-{\mathrm{e}}^{-\theta t}}{\theta }+{\int }_0^t(t-s)\mathrm{d}V(s)=U_0+U_1\frac{1-{\mathrm{e}}^{-\theta t}}{\theta }+{\int }_0^{t}V(s)\mathrm{d}s.$$ (23)

The penalty ${\int }_0^T{[(D^2+\theta D)f]}^2\mathrm{d}t$ is the same as that for an exponential spline. It is easy to check that the basis functions of the space ℋ₀ in Theorem 3.2 are ψ₀(t) = 1 and A₁(t) = (1 − e^−θt)/θ, and the reproducing kernel of the space ℋ₁ is

$$R_1(s,t)=\frac{{\sigma }_V^2}{{\theta }^2}[s\wedge t-\frac{{\mathrm{e}}^{-\theta s}+{\mathrm{e}}^{-\theta t}}{\theta }({\mathrm{e}}^{\theta s\wedge t}-1)+\frac{{\mathrm{e}}^{-\theta (s+t)}}{2\theta }({\mathrm{e}}^{2\theta s\wedge t}-1\left)\right],$$ (24)

which is equivalent to that of an exponential spline (Wang, 2011, pp 41–44). Furthermore,

$$\mu (t)={\int }_0^t\frac{F(t,u){\beta }_0(u)}{\mathrm{\Psi }(u)}\mathrm{d}u={\int }_0^t\mu \theta {\mathrm{e}}^{\theta u}\mathrm{d}u{\int }_u^t{\mathrm{e}}^{-\theta \tau }\mathrm{d}\tau =\mu {\theta }^{-1}(\theta t-1+{\mathrm{e}}^{-\theta t}).$$ (25)

Therefore, when μ = 0, the integrated Ornstein–Uhlenbeck process U(·) defined in (23) is the corresponding stochastic process for the exponential spline. The case when μ ≠ 0 provides an extension of the exponential spline. We note that the integrated Ornstein–Uhlenbeck stochastic process with drift U(t) represents a major deviation from existing spline literature which involves the Brownian motion only.

The mean function of stochastic process U(·) is

$$\mathrm{E}[U(t)]={\mu }_0+{\mu }_1\frac{1-{\mathrm{e}}^{-\theta t}}{\theta }+\mu (t+\frac{{\mathrm{e}}^{-\theta t}-1}{\theta }).$$

When μ = 0, $\mu =0,\overline{V}(t)=\mathrm{\Psi }(t)[U_1+{\int }_0^t{\sigma }_V{\mathrm{e}}^{\theta s}dB(s)]$ in Remark 3.1 satisfies

$$\mathrm{d}\overline{V}(t)=-\theta \overline{V}(t)\mathrm{d}t+{\sigma }_V\mathrm{d}B(t),\overline{V}(0)=U_1,t\in [0,T].$$

Therefore, $U(t)=U_0+{\int }_0^t\overline{V}(s)\mathrm{d}s$ and E[U(t)] = μ₀ + μ₁(1 − e^−θt)/θ.

When q > 1, the stochastic process U(·) can be represented as

$$U(t)=\sum _{i=1}^{q-1}{\psi }_i(t)U_i+A_q(t)U_q+{\int }_0^t{\psi }_{q-1}(t-s)V(s)\mathrm{d}s.$$ (26)

The penalty ${\int }_0^T{[D^{(q-1)}(D^2+\theta D)f]}^2\mathrm{d}t$ adds polynomials up to the order q − 1 to the kernel space ℋ₀. Specifically, the orthonormal basis functions of ℋ₀ are ψ₀(t), ψ₁(t), …, ψ_q−1(t), and A_q(t), where $A_q(t)={\int }_0^t{\psi }_{q-1}(t-s){\mathrm{e}}^{-\theta s}\mathrm{d}s$. The function A_q(t) can be calculated recursively by A_q(t) = ψ_q−1(t)/θ − A_q−1(t)/θ and A₁(t) = (1 − e^−θt)/θ. The Chebyshev spline in this case may be called a polynomial-exponential spline.

3.3.2. Partial exponential spline

Consider another special case of model (2) with a₀(t) = b(t) = 1. For q ≥ 1, the stochastic process U(·) can be represented as

$$U(t)=\sum _{i=0}^q{\psi }_i(t)U_i+{\int }_0^t{\psi }_{q-1}(t-s)V(s)\mathrm{d}s.$$ (27)

Replacing a Brownian motion by the Ornstein–Uhlenbeck process V(·), the stochastic process U(·) in (27) extends the stochastic model for polynomial splines (Wahba, 1990) (see Eq. (S.5) in the Supplementary Materials). The mean and covariance of the stochastic process U(·) are

$$\mathrm{E}U(t)=\sum _{i=0}^q{\psi }_i(t){\mu }_i+{\int }_0^t{\psi }_{q-1}(t-s)\mu (1-{\mathrm{e}}^{-\theta s})\mathrm{d}s,$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^q{\psi }_i(s){\psi }_i(t){\sigma }_i^2+{\sigma }_V^2{\int }_0^{s\wedge t}F(s,u)F(t,u){\mathrm{e}}^{2\theta u}\mathrm{d}u,$$

where $F(t,u)=I(u\le t){\int }_u^t{\psi }_{q-1}(t-\tau ){\mathrm{e}}^{-\theta \tau }\mathrm{d}\tau$.

For simplicity, we now consider the special case when q = 1. The stochastic process U(·) is given by

$$U(t)=U_0+U_{1}t+{\int }_0^t(t-s)\mathrm{d}V(s)=U_0+U_{1}t+{\int }_0^{t}V(s)\mathrm{d}s.$$ (28)

The integrated Ornstein–Uhlenbeck process in (28) extends the integrated Brownian motion [see (S.6) in Section S.5 of the Supplementary Materials]. The mean and covariance functions are

$$\mathrm{E}U(t)={\mu }_0+{\mu }_{1}t+\mu (t+\frac{{\mathrm{e}}^{-\theta t}-1}{\theta }),$$

$$\text{Cov}(U(s),U(t))={\sigma }_0^2+st{\sigma }_1^2+R_1(s,t),$$ (29)

where R₁ is given by (24).

Note that Theorem 3.2 does not apply in this case since a₀(t) ≠ Ψ(t). Compared with the exponential spline model, the basis function (1 − exp(−θt))/θ in ℋ₀ has been replaced by the function ψ₁(t) = t which is not orthogonal to the space ℋ₁ (Remark 3.2). Nevertheless, the stochastic process U(·) defined in (28) can be connected to the following partial spline model

$$Y_i={\alpha }_1+{\alpha }_2{t}_i+f(t_i)+{\epsilon }_i,i=1,\dots ,n,t_i\in [0,T].$$ (30)

Assume that $f\in ℳ\triangleq W_2^2[0,T]\ominus \{1,(1-\text{exp}(-\theta t))/\theta \}$ under the inner product (14) with Lf (u) = [D²f (u) + θDf(u)] e^θu and $h(u)={\mathrm{e}}^{-2\theta u}{\sigma }_V^{-2}$. Let α̂₁, α̂₂ and f̂_λ(t) be the solution to the following penalized least squares

$$\underset{{\alpha }_1\in R,{\alpha }_2\in R,f\in ℳ}{\text{min}}\{\frac{1}{n}\sum _{i=1}^n{[Y_i-({\alpha }_1+{\alpha }_2{t}_i+f(t_i))]}^2+\lambda {\int }_0^T{(Lf(t))}^{2}h(t)\mathrm{d}t\}.$$ (31)

Following similar arguments as in Section 3.2 one can show that lim_a→∞ Û_a(t) = μ(t)+ α̂₁ + α̂₂t + f̂_λ(t), where α̂₁, α̂₂ and f̂_λ(t) are the penalized least squares solutions to (31) with observations Y_i = Y(t_i) − μ(t_i) for i = 1, …, n where μ(t) is given in (25).

3.4. Logistic Spline and Its Corresponding SDM

The logistic spline is a special case of the Chebyshev spline with q = 0 and penalty ${\int }_0^T{[f\prime (t)-{\beta }_1(t)f(t)]}^2\mathrm{d}t$, where

$${\beta }_1(t)=\theta \gamma {\mathrm{e}}^{-\theta t}/(1+\gamma {\mathrm{e}}^{-\theta t}),\theta >0,\gamma >0.$$ (32)

To construct the corresponding SDM, consider a process V(·) driven by the following linear SDE

$$\mathrm{d}V(t)={\beta }_1(t)V(t)\mathrm{d}t+{\sigma }_V\mathrm{d}B(t),V(0)=0.$$ (33)

Equation (33) is a special case of (8) with β₀(t) = 0, β₁(t) is given in (32), and σ(t) = σ_V. It is not difficult to check that μ(t) = 0, Ψ(t) = (1 + γ)/(1 + γe^−θt), and

$$V(t)=\frac{{\sigma }_{V}B(t)}{1+\gamma {\mathrm{e}}^{-\theta t}}+\frac{{\sigma }_V\gamma }{1+\gamma {\mathrm{e}}^{-\theta t}}{\int }_0^t{\mathrm{e}}^{-\theta s}\mathrm{d}B(s).$$

Then, E V(t) = 0 and

$$\text{Cov}(V(s),V(t))=\mathrm{\Psi }(s)\mathrm{\Psi }(t){\int }_0^{s\wedge t}{\left(\frac{{\sigma }_V}{\mathrm{\Psi }(u)}\right)}^2\mathrm{d}u=\frac{{\sigma }_V^2}{(1+\gamma {\mathrm{e}}^{-\theta s})(1+\gamma {\mathrm{e}}^{-\theta t})}[s\wedge t+\frac{2\gamma }{\theta }(1-{\mathrm{e}}^{-\theta (s\wedge t)})+\frac{{\gamma }^2}{2\theta }(1-{\mathrm{e}}^{-2\theta (s\wedge t)}\left)\right].$$ (34)

According to (21), the penalty ${\int }_0^T{(Lf(t))}^{2}h(t)\mathrm{d}t={\sigma }_V^{-2}{\int }_0^T{[f\prime (t)-{\beta }_1(t)f(t)]}^2\mathrm{d}t$ is the same as that for the logistic spline up to a multiplying constant which can be absorbed into the smoothing parameter λ (Wang, 2011). Consider the special case of stochastic dynamic system (2) with a₀(t) = Ψ(t) and b(t) = 1. The basis function of the space ℋ₀ is Ψ(t) and the Green’s function is Ψ(t)I(u ≤ t). Then, the reproducing kernel for the space ℋ₁ is $R_1(s,t)=\mathrm{\Psi }(s)\mathrm{\Psi }(t){\int }_0^{s\wedge t}{[{\sigma }_V/\mathrm{\Psi }(u)]}^2\mathrm{d}u$. R₁ has the form in (34) which is the same as the reproducing kernel for the logistic spline (Eq. (2.61) in Wang (2011)). Thus, the SDM (1), (2) and (33) consist of the stochastic model for the logistic spline.

Remark 3.6

When q = 1, following similar argument it can be shown that the SDM (1), (2) and (33) is the stochastic model for the logistic spline discussed by Gu (2013, p. 161). The cases when q > 1 can be regarded as extensions of the logistic spline where basis functions ψ₀(t), …, ψ_q−1(t) are added to the null space ℋ₀.

Remark 3.7

The connection between an SDM driven by (33) and a Chebyshev spline discussed in this section holds for a general function β₁(t). Specifically, consider the SDE (33) with an unspecified β₁(t). Assume that q = 0, a₀(t) = Ψ(t) and b(t) = 1. Then, $V(t)={\sigma }_V\mathrm{\Psi }(t){\int }_0^t\mathrm{d}B(s)/\mathrm{\Psi }(s)$ and $U(t)=\mathrm{\Psi }(t)U_0+{\sigma }_V\mathrm{\Psi }(t){\int }_0^t\mathrm{d}B(s)/\mathrm{\Psi }(s)$. According to Theorem 3.2, the RKHS $W_2^1[0,T]=\{\mathrm{\Psi }(t)\}\oplus {\mathscr{H}}_1$, where ℋ₁has the reproducing kernel $R_1(s,t)=\mathrm{\Psi }(s)\mathrm{\Psi }(t){\int }_0^{s\wedge t}{[{\sigma }_V/\mathrm{\Psi }(u)]}^2\mathrm{d}u$. The penalty of the corresponding Chebyshev spline is ${\sigma }_V^{-2}{\int }_0^T{[f\prime (t)-{\beta }_1(t)f(t)]}^2\mathrm{d}t$. The logistic spline is a special case with β₁(t) = θγe^−θt/(1 + γe^−θt). It is not difficult to check that the function β₁(t) = −θ corresponds to the exponential spline with β₀(t) = 0. Other functions may be considered for β₁(t). For example, β₁(t) = −θt corresponds to a Chebyshev spline with the kernel space spanned by Ψ(t) = e^−θt2/2, the Gaussian function.

4. EXTENDED STOCHASTIC DYNAMIC MODELS AND THEIR CONNECTIONS TO CHEBYSHEV SPLINES

Motivated by the Chebyshev splines (Karlin & Ziegler, 1966; Kimeldorf & Wahba, 1971; Gu, 2013), we now consider a more general dynamic system by replacing D^q in (2) with the following differential operator

$${\mathrm{\Pi }}_q=D\frac{1}{a_1}D\frac{1}{a_2}\cdots D\frac{1}{a_q},$$ (35)

where a₁, …, a_q are strictly positive and (i + 1)th differentiable functions with a_i(0) = 1. Specifically, we assume the following stochastic dynamic system for U(·)

$${\mathrm{\Pi }}_{q}U(t)=a_0(t){\mathrm{\Pi }}_{q}U(0)+b(t)V(t),q\ge 0,t\in [0,T].$$ (36)

As in previous sections, we will consider two types of models for V(·): the general diffusion process determined by the SDE (3) and the Gaussian model driven by the linear SDE (8).

The following stochastic differential system has been considered by Kimeldorf & Wahba (1971):

$${\mathrm{\Pi }}_{q+1}U(t)=\frac{\mathrm{d}B(t)}{\mathrm{d}t},$$ (37)

where ${\mathrm{\Pi }}_{q+1}=D\frac{1}{a_0}D\frac{1}{a_1}D\frac{1}{a_2}\cdots D\frac{1}{a_q}$ is a (q + 1)-order differential operator of Chebyshev splines (Wahba, 1978, 1990). The stochastic dynamic system (37) is equivalent to Π_qU(t) = a₀(t)[Π_qU(0) + B(t)]. The stochastic dynamic system (36) extends (37) in two aspects: (1) Brownian motion B(t) is replaced by a general solution V(·) of an SDE, and (2) the coefficient b(t) of V(·) does not necessarily equal to a₀(t).

4.1. General Stochastic Dynamic Models

Following Kimeldorf & Wahba (1971) and Wahba (1990), denote

$${\mathrm{\Pi }}_0=I,$$

$${\mathrm{\Pi }}_1=D\frac{1}{a_q},$$

$$\begin{array}{c}\hfill {\mathrm{\Pi }}_2=D\frac{1}{a_{q-1}}D\frac{1}{a_q},\hfill \\ \hfill ⋮\hfill \\ \hfill {\mathrm{\Pi }}_{q-1}=D\frac{1}{a_2}D\frac{1}{a_3}\cdots D\frac{1}{a_q},\hfill \end{array}$$

where I is the identity operator. In addition, define the following functions:

$${\omega }_0(t)=a_q(t),$$

$$\begin{array}{c}\hfill {\omega }_1(t)=a_q(t){\int }_0^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1},\hfill \\ \hfill ⋮\hfill \\ \hfill {\omega }_{q-1}(t)=a_q(t){\int }_0^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}{\int }_0^{t_{q-1}}{a}_{q-2}(t_{q-2})\mathrm{d}{t}_{q-2}\cdots {\int }_0^{t_2}{a}_1(t_1)\mathrm{d}{t}_1,\hfill \end{array}$$

$${\omega }_q(t)=a_q(t){\int }_0^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}{\int }_0^{t_{q-1}}{a}_{q-2}(t_{q-2})\mathrm{d}{t}_{q-2}\cdots {\int }_0^{t_2}{a}_1(t_1)\mathrm{d}{t}_1{\int }_0^{t_1}{a}_0(t_0)\mathrm{d}{t}_0.$$

Note that for i, j = 0, 1, 2, …, q, we have

$$({\mathrm{\Pi }}_i{\omega }_j)(0)=\{\begin{array}{cc}\hfill 1\hfill & \hfill \text{if}i=j\hfill \\ \hfill 0\hfill & \hfill \text{else}.\hfill \end{array}$$ (38)

Define

$$X(t)=a_q(t){\int }_0^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}\cdots {\int }_0^{t_2}{a}_1(t_1)\mathrm{d}{t}_1{\int }_0^{t_1}b(t_0)\mathrm{d}{t}_0{\int }_0^{t_0}\mathrm{d}V(u)=a_q(t){\int }_0^{t}b(t_0)V(t_0)\mathrm{d}{t}_0{\int }_{t_0}^t{a}_1(t_1)\mathrm{d}{t}_1\cdots {\int }_{t_{q-2}}^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}.$$ (39)

We note a typo in Equation (7.3) of Kimeldorf & Wahba (1971) where … ${\int }_0^{t_2}{a}_1(t_1)\mathrm{d}W(t_1)$ should be … ${\int }_0^{t_2}{a}_1(t_1)W(t_1)\mathrm{d}{t}_1$.

Following similar arguments as in the Section 2.2, the stochastic differential system (36) is equivalent to the following stochastic integration equation

$$U(t)=\sum _{i=0}^q{\omega }_i(t)U_i+X(t),$$ (40)

where U_i = Π_iU(0). Let μ_i = EU_i and ${\sigma }_i^2=\text{Var}(U_i)$. Applying the Fubini’s Theorem to the stochastic integration equation (40), we have the following results which extends Proposition 2.1.

Proposition 4.1

Suppose that U(·) is given by the stochastic integration equation (40), where V(·) is driven by the SDE (3). Then,

$$\mathrm{E}U(t)=\sum _{i=0}^q{\omega }_i(t){\mu }_i+a_q(t){\int }_0^{t}b(\tau )\mathrm{E}V(\tau )\mathrm{d}\tau {\int }_{\tau }^t{a}_1(t_1)\mathrm{d}{t}_1\cdots {\int }_{t_{q-2}}^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1},$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^q{\omega }_i(s){\omega }_i(t){\sigma }_i^2+\text{Cov}(X(s),X(t)),$$

$$\text{Cov}(X(s),X(t))=a_q(s)a_q(t){\int }_0^s{\int }_0^{t}b(u)b(\upsilon )\text{Cov}(V(u),V(\upsilon ))\mathrm{d}u\mathrm{d}\upsilon {\int }_u^s{a}_1(s_1)\mathrm{d}{s}_1{\int }_{\upsilon }^t{a}_1(t_1)\mathrm{d}{t}_1\cdots {\int }_{s_{q-2}}^s{a}_{q-1}(s_{q-1})\mathrm{d}{s}_{q-1}{\int }_{t_{q-2}}^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}.$$

4.2. Gaussian Models Driven by a Linear SDE

Denote

$$\tilde{F}(t,u)=a_q(t){\int }_u^{t}b(\tau )\mathrm{\Psi }(\tau )\mathrm{d}\tau {\int }_{\tau }^t{a}_1(t_1)\mathrm{d}{t}_1{\int }_{t_1}^t{a}_2(t_2)\mathrm{d}{t}_2\cdots {\int }_{t_{q-2}}^t{a}_{q-1}(t_{q-1})\mathrm{d}{t}_{q-1}.$$

The following results extend Theorem 3.1 when the SDM is driven by the linear SDE (8).

Theorem 4.1

Suppose that the stochastic process V(·) is a Gaussian process driven by the linear SDE (8) and $U_i~\mathrm{N}({\mu }_i,{\sigma }_i^2)$ for i = 0, …, q. Then, the stochastic process U(·) given by (36) or (40) is a Gaussian process with the following mean and covariance functions

$$\mathrm{E}U(t)=\sum _{i=0}^q{\omega }_i(t){\mu }_i+{\int }_0^t\tilde{F}(t,u)\frac{{\beta }_0(u)}{\mathrm{\Psi }(u)}\mathrm{d}u,$$

$$\text{Cov}(U(s),U(t))=\sum _{i=0}^q{\omega }_i(s){\omega }_i(t){\sigma }_i^2+{\int }_0^{s\wedge t}\tilde{F}(s,u)\tilde{F}(t,u){\left(\frac{\sigma (u)}{\mathrm{\Psi }(u)}\right)}^2\mathrm{d}u.$$

The proof of Theorem 4.1 is provided in the Supplementary Materials.

4.3. Connection to Chebyshev Splines

Again, let $\mathrm{\Psi }(t)\triangleq \text{exp}({\int }_0^t{\beta }_1(s)\mathrm{d}s)$ and h(t) ≜ (Ψ(t)/σ(t))² be a weight function. Consider the following differential operator

$$\tilde{L}f(u)=\frac{d}{\mathrm{d}u}\left(\frac{{\mathrm{\Pi }}_{q}f(u)}{\mathrm{\Psi }(u)}\right)=\frac{D{\mathrm{\Pi }}_{q}f(u)-{\beta }_1(u){\mathrm{\Pi }}_{q}f(u)}{\mathrm{\Psi }(u)}.$$

Consider the model space

$${\tilde{W}}_2^{q+1}[0,T]\triangleq \{f:f,{\mathrm{\Pi }}_{1}f,\dots ,{\mathrm{\Pi }}_{q}f\text{are absolutely continuous and}{\int }_0^T{(\tilde{L}f(t))}^{2}h(t)\mathrm{d}t<\infty \}$$

with inner product

$$(f,g)=\sum _{i=0}^q{\mathrm{\Pi }}_{i}f(0){\mathrm{\Pi }}_{i}g(0)+{\int }_0^T\tilde{L}f(u)\tilde{L}g(u)h(u)\mathrm{d}u.$$ (41)

Theorem 4.2

Assume that a₀(t) = Ψ(t) and b(t) = 1. Then, the space ${\tilde{W}}_2^{q+1}[0,T]$ is an RKHS with inner product (41). Let ${\tilde{\mathscr{H}}}_0=\{f\in {\tilde{W}}_2^{q+1}[0,T]:\tilde{L}f=0\}$ be the kernel of L̃. Then, {ω₀, ω₁,…, ω_q} forms an orthonormal basis of ℋ̃₀ and ${\tilde{W}}_2^{q+1}[0,T]$ can be decomposed into ${\tilde{W}}_2^{q+1}[0,T]={\tilde{\mathscr{H}}}_0\oplus {\tilde{\mathscr{H}}}_1$, where

$${\tilde{\mathscr{H}}}_0=\mathit{\text{span}}\{{\omega }_0,{\omega }_1,\dots ,{\omega }_q\},$$

$${\tilde{\mathscr{H}}}_1=\{f\in {\tilde{W}}_2^{q+1}[0,T]:f(0)={\mathrm{\Pi }}_{1}f(0)=\dots ={\mathrm{\Pi }}_{q}f(0)=0\}.$$

The reproducing kernels of ℋ̃₀ and ℋ̃₁ are respectively given by

$${\tilde{R}}_0(s,t)=\sum _{i=0}^q{\omega }_i(s){\omega }_i(t),$$

$${\tilde{R}}_1(s,t)={\int }_0^{s\wedge t}\tilde{F}(s,u)\tilde{F}(t,u){[h(u)]}^{-1}\mathrm{d}u.$$

The proof of Theorem 4.2 is provided in the Supplementary Materials.

Now consider the nonparametric regression model (15) where $f\in {\tilde{W}}_2^{q+1}[0,T]$. A general Chebyshev spline is the solution to the penalized least squares (16) with $W_2^{q+1}[0,T]$ and L being replaced by ${\tilde{W}}_2^{q+1}[0,T]$ and L̃, respectively. Again, up to a constant function, the smoothing spline estimate equals the posterior mean of the SDM (1), (8) and (36). Specifically, assume that U₀, …, $U_q\stackrel{iid}{~}\mathrm{N}(0,a)$. Denote Û_a = E(U(t)|Y(t_i), i = 1, …, n) and f̃_λ(t) as the penalized least squares solution with observations (Y(t₁) − μ̃(t₁), …, Y(t_n) − μ̃(t_n))′, where

$$\tilde{\mu }(t)\triangleq {\int }_0^t\tilde{F}(t,s){\beta }_0(s)/\mathrm{\Psi }(s)\mathrm{d}s.$$

Then for any fixed t ∈ [0, T], when λ = σ²/n, it can be shown that lim_a→∞ Û_a(t) = μ̃(t) + f̃_λ(t).

4.4. Chebyshev Splines to Stochastic Dynamic Models

In the previous sections we started with SDE driven SDMs and built corresponding Chebyshev splines. We now describe a strategy that builds an SDE driven SDM that corresponds to the general Chebyshev spline defined in Section 4.3.

With the general Chebyshev spline, a₀(t), …, a_q(t) and a general weight function h(t) > 0 are given. Assume that $a_i(t)\in W_2^{i+1}[0,t]$ for i = 0, …, q are strictly positive functions with a_i(0) = 1. From SDMs to Chebyshev splines, we have set $a_0(t)=\mathrm{\Psi }(t)=\text{exp}({\int }_0^t{\beta }_1(s)\mathrm{d}s)$ and h(t) = (a₀(t)/σ(t))² in the previous sections, where β₁(t) and σ(t) are given in the linear SDE (8). Reversely, now we set ${\beta }_1(t)=a_0^{\prime }(t)/a_0(t)$ and $\sigma (t)=a_0(t)/\sqrt{h(t)}$. Consider the following stochastic dynamic system

$${\mathrm{\Pi }}_{q}U(t)=a_0(t)U_q+V(t),$$ (42)

$$\mathrm{d}V(t)={\beta }_1(t)V(t)\mathrm{d}t+\sigma (t)\mathrm{d}B(t),V(0)=0,$$ (43)

with initial conditions U_i = Π_iU(0) for i = 0, …, q. Then, it is not difficult to show that the general Chebyshev spline equals the posterior mean of the SDM (1), (42), and (43).

5. APPLICATIONS

Proposition 3.1 implies that with appropriate choices of parameters in the SDM and the smoothing parameter, the posterior mean coincides with the corresponding Chebyshev spline estimate. In particular, as discussed in Section 3.3.1, with U₀, $U_1\stackrel{iid}{~}\mathrm{N}(0,a)$, a → ∞ and $n\lambda ={\sigma }^2/{\sigma }_V^2$, the posterior mean of the integrated Ornstein–Uhlenbeck process (23) coincides with the exponential spline estimate, in which σ² is the variance of ε(t, ω) the measurement error in Equation (1) and ${\sigma }_V^2$ is the volatility term of integrated Ornstein–Uhlenbeck process (23), reflecting the fluctuation of the process. We now use two real data applications to illustrate this theoretical result.

Glomerular filtration rate (GFR) measures the flow rate of filtered fluid through the kidney. Progression of kidney disease is often assessed by change in GFR. Therefore it is important to estimate the trajectory of GFR based on observations (Li et al., 2014). The left panel of Figure 1 shows estimated GFR (eGFR) from a patient with chronic kidney disease. We first consider an exponential spline model discussed in Section 3.3.1 since the profile is close to exponential decay. To estimate the speed of reversion parameter θ, as in Wang (2011), we first fit a nonlinear regression model which is motivated by Equation (25) as y_i = β₁ + β₂ exp(−β₃t_i) + ε_i, i = 1, …, 62, and then set θ = β̂³ where β̂₃ is the non-linear least square estimate of β₃.We then fit the exponential spline using the ssr function in the R ASSIST package (Wang, 2011) with θ = β̂₃ and smoothing parameter selected by the generalized maximum likelihood method. The exponential spline fit is shown in the left panel of Figure 1 as the solid blue line.

Figure 1.

GFR (a) and PSA (b) examples. Circles are observations, posterior means are dashed red lines, and exponential spline fits are solid blue lines.

The Ornstein–Uhlenbeck process provides a natural model for a process stabilizing around some equilibrium point. Such phenomena is often observed in the biological or biomedical application. For example, we consider modelling the eGFR profile by the SDM with integrated Ornstein–Uhlenbeck process (23), for which the parameters have natural interpretations in terms of the convergence of process. This is a special case of the stochastic velocity model with Ornstein–Uhlenbeck process discussed in Zhu, Song, & Taylor (2011). Thus the Markov chain Monte Carlo (MCMC) algorithm developed in Zhu, Song, & Taylor (2011) can be used to compute the posterior mean of the integrated Ornstein–Uhlenbeck process. Since patients will eventually lose renal function, we set the equilibrium value μ = 0.We estimated the parameter θ as the posterior mean of MCMC samples of θ, and set U₀, $U_1\stackrel{iid}{~}\mathrm{N}(0,{10}^4)$. The MCMC was run for 45,000 iterations, in which the first 35,000 runs are discarded as the burn in and every 10th draw is saved. The posterior mean is shown in the left panel of Figure 1 as the red dashed line. As expected from the theoretical result, the exponential spline estimate is almost identical to the posterior mean of the integrated Ornstein–Uhlenbeck process.

We further present another real data application for the prostate specific antigen (PSA) profile commonly used to monitor recurrence of prostate cancer after treatment. Different from the GFR example, the rate of change of the PSA profile converges or the slope of the PSA profile is stabilized at a non-zero value (Figure 1, Zhu, Song, & Taylor, 2011). It’s reasonable to fit an SDM with integrated Ornstein–Uhlenbeck process and non-zero μ. For such a case, we illustrate that exponential spline can also be applied with a simple transformation. To make the estimates by two methods comparable, we fix θ̂; = 1. 15 and μ̂ = 0. 385 (posterior means in Table 3, Zhu, Song, & Taylor, 2011), while other parameters were estimated either by the MCMC algorithm for SDM or generalized maximum likelihood method for the exponential spline. To fit the exponential spline, the transformed variable z(t) = y(t)− μ̂{t + (e^−θ̂t − 1)/θ̂} is used as observations. The right panel of Figure 1 shows the observations and estimated profiles. Again, the exponential spline estimate is almost identical to the posterior mean of the integrated Ornstein–Uhlenbeck process.

6. DISCUSSION

Under the general framework of isometric mapping between the Hilbert space spanned by a second order stochastic process and the RKHS generated by its covariance kernel, we establish a specific connection between an SDM driven by a linear SDE and the Chebyshev spline. This connection provides a statistical structure and mechanism for estimating sample paths in a stochastic dynamic model as well as a justification for the somewhat ad hoc penalty in a penalized least squares. Our results extend the well-known connection between the integrated Brownian and the polynomial spline to the connection between an SDM and the Chebyshev spline, which is mutually beneficial for these two different areas. For example, fitting spline models with large data can be computationally expensive. Instead, we may fit the corresponding SDMs with efficient algorithms based on the Markov property (Kohn & Ansley, 1987; Zhu, Song, & Taylor, 2011).

In this paper we have assumed σ(t), β₀(t), and β₁(t) in (8) are known, which in practice need to be estimated. Under the assumption of a₀(t) = Ψ(t) and b(t) = 1, the SDM (1), (2) and (8) are determined by σ(t), β₀(t), and β₁(t). Similarly, the Chebyshev spline model may contain unknown parameters _ (Heckman & Ramsay, 2000). For example, the parameter θ in the penalty ${\int }_0^T{[(D^2+\theta D)f]}^2\mathrm{d}t$ of the exponential spline is usually unknown and corresponds to an unknown speed of reversion in the integrated Ornstein–Uhlenbeck process. Estimation methods have been proposed for parameters in Chebyshev spline models (Heckman & Ramsay, 2000; Wang & Ke, 2009 and references therein) and SDMs (Zhu, Song, & Taylor, 2011, and references therein). Connection and comparison between these parameter estimation methods in these two different fields will be studied in the future.

One important feature of the SDM considered in this paper is that the process V(·) can be non-stationary which makes it more flexible. For special cases discussed in this paper, we have assumed that σ(t) is a constant. With a general σ(t), Equation (21) corresponds to an adaptive penalty for spatial inhomogeneous functions (see Pintore, Speckman, & Holmes, 2006; Liu & Guo, 2010, and references therein). As a future research topic, we will explore the corresponding non-stationary processes to fit spline models with varying smoothing parameters.