Bayesian Local Extremum Splines

Summary

We consider shape restricted nonparametric regression on a closed set $\mathcal{X}\subset ℝ$, where it is reasonable to assume the function has no more than H local extrema interior to $\mathcal{X}$. Following a Bayesian approach we develop a nonparametric prior over a novel class of local extremum splines. This approach is shown to be consistent when modeling any continuously differentiable function within the class considered, and is used to develop methods for testing hypotheses on the shape of the curve. Sampling algorithms are developed, and the method is applied in simulation studies and data examples where the shape of the curve is of interest.

1. Introduction

This paper considers Bayesian modeling of an unknown function $f_0:\mathcal{X}\to ℝ$, where it is known that f₀ has at most H local extrema, or change points, interior to $\mathcal{X}$, and one wishes to estimate the function subject to constraints or test the hypothesis the function has a specific shape. For example, one may wish to consider a monotone function versus one having an N shape. We propose a spline construction that allows for nonparametric estimation of shape-constrained functions having at most H change points. The approach places a prior over a knot set dense in $\mathcal{X}$, and, to sample over the models defined by this knot set, a Markov chain Monte Carlo algorithm is developed to sample models. The method allows for nonparametric hypothesis testing of different shapes within the class of functions considered.

The shape-constrained regression literature focuses primarily on functions that are monotone, convex, or have a single minimum; that is, cases with H ≤ 1. Ramgopal et al. (1993), Lavine & Mockus (1995), and Bornkamp & Ickstadt (2009) consider priors over cumulative distribution functions used to model monotone curves. Holmes & Mallick (2003), Neelon & Dunson (2004), Meyer (2008), and Shively et al. (2009) develop spline-based approaches for monotone functions. Hans & Dunson (2005) design a prior for umbrella-shaped functions, while Shively et al. (2011) propose methods for fixed- and free- knot splines that model continuous segments having a single unknown change point.

Extending these approaches to broader shape constraints is not straightforward. For example, to obtain H = 3 change points, one could define a prior over B-spline bases de Boor (2001, page 87)) having four monotone segments that alternately increase and decrease. However, for even a moderate number of pre-specified knots and a known number of change points, allowing for uncertainty in the locations of the change points leads to a daunting computational problem. Bayesian computation via Markov chain Monte Carlo is subject to slow mixing and convergence rates in alternating between updating the spline coefficients conditionally on the change points and vice versa, and it is not clear how to devise algorithms that can efficiently update both simultaneously. These difficulties are compounded by allowing for the possibility that some of the change points should be removed, which is commonly the situation in applications. By defining a new spline basis based on the number of change points, we bypass these issues.

Little work has been done on nonparametric Bayesian testing of curve shapes. Salomond (2014) and Scott et al. (2015) consider Bayesian nonparametric testing for monotonic versus an unspecified nonparametric alternative, but do not consider shapes beyond monotonicity. Our approach allows for testing of all shapes, where shape is defined as the type and sequence of extrema. For example, one can use this approach to test for an umbrella shape verses an N-shaped curve and use the same procedure to test the umbrella shape against monotone alternatives.

We propose a new approach to incorporating shape constraints based on splines that are carefully constructed to induce curves having a particular number of extrema. This is similar in spirit to the I-spline construction of Ramsay (1988) or the C-spline construction for convex splines (Meyer, 2008; Meyer et al., 2011), both of which create a spline construction based upon the derivative of the spline. When paired with positivity constraints on the spline coefficients, our construction enforces shape restrictions on the curve of interest by limiting the number of change points.

Another key aspect of our approach is that we place a prior over a countable dense set of knots, which allows the number of the splines in the model space to grow. This bypasses the sensitivity to choice of the number of knots, while facilitating computation and theory on consistency. In particular, we propose a prior over nested model spaces where the location of the knots is known for each model. This allows for a straightforward reversible jump Markov chain Monte Carlo algorithm (Green, 1995) based upon Godsill (2001). This is different from much of the previous Bayes literature allowing unknown numbers of knots (Biller, 2000; DiMatteo et al., 2001). In these methods, the knot locations are unknown, and the reversible jump Markov chain Monte Carlo proposal must propose a knot to add or delete as well as its location. Such algorithms are notoriously inefficient.

2. Model

2·1. Local Extremum Spline Construction

Let ℱ^H be a set of functions defined on the closed set $\mathcal{X}\subset ℝ$, such that for f₀ ∈ ℱ^H, f₀ is continuously differentiable and has H or fewer local extrema interior to $\mathcal{X}$. Such functions can be modeled using B-spline approximations of the form

$$f\left(x\right)={\displaystyle \sum _{k=1}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}\left(x\right)}.$$ (1)

Here, β_k is a scalar coefficient and B_(j,k)(x) is a B-Spline function of order j defined on the knot set $\mathcal{T}={\left\{{\tau }_k\right\}}_{k=1}^K$, τ₁ ≤…≤ τ_K, which includes end knots. For any knot set, de Boor (2001, page 145) showed that there exist spline approximations such that ‖f − f₀‖_∞ ≤ Δ ‖f₀‖_∞, where Δ is the maximum difference between adjacent knots. Though this construction can be used to model f₀ with arbitrary accuracy, it does not ensure that the approximating function f is itself in ℱ^H.

We force f ∈ ℱ^H to have at most H local extrema by defining a new spline basis

$$B_{\left(j,k\right)}^{\ast }\left(x\right)=M{\displaystyle {\int }_{-\infty }^x\left\{{\displaystyle \prod _{h=1}^H\left(\xi -{\alpha }_h\right)}\right\}{B}_{\left(j,k\right)}\left(\xi \right)d\xi ,}$$ (2)

where B_(j,k)(x) is a B-spline constructed using the knot set $\mathcal{T}$, {α₁, …, α_h} are distinct change points, and M is a fixed integer. Letting $B_{\left(j,0\right)}^{\ast }\left(x\right)=1$, if β_k ≥ 0, for all k ≥ 1, any linear combination of local extremum spline basis functions for any distinct values of α₁,…, α_H in (2) will be in ℱ^H.

Proposition 1

If $f\left(x\right)={\displaystyle {\sum }_{k=0}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}^{\ast }}\left(x\right)$ for any K ≥ 1 with M ∈ {−1, 1}, j ≥ 1, and β_k ≥ 0 for all k ≥ 1, then f ∈ ℱ^H.

This result follows from the constraint on the β_k coefficients. By forcing β_k ≥ 0 for k ≥ 1, the sign of the derivative is controlled by the polynomial $M{\displaystyle {\prod }_{h=1}^H\left(x-{\alpha }_h\right)}$, which allows a maximum of H local extrema located at the change points {α₁, …, α_H}. When β_k = … = β_k+1 = 0 and α_h ∈ [τ_k+j, τ_k+j+1], α_h does not define a unique extremum. In this case, there is a flat region, and multiple configurations of the change point parameters can give the same curve. Otherwise, the extrema are uniquely defined for all ${\alpha }_h\in \mathcal{X}$, and fewer than H extrema can be considered if ${\alpha }_h\notin \mathcal{X}$.

Theorem 1

For any f₀ ∈ ℱ^H and ε > 0 there exist a knot set $\mathcal{T}$ and a local extremum spline f^LX defined on this knot set such that

$${\Vert f_0-f^{LX}\Vert }_{\infty }<\epsilon .$$

The flexibility of local extremum splines is attributable to the B-splines used in their construction. The proof of Theorem 1 assumes that M can be chosen to be positive or negative, which allows all functions in ℱ^H to be approximated. If M is fixed, then any function with H − 1 extrema can be modeled. For exactly H extrema, the approach is limited to modeling functions that are either initially increasing or initially decreasing, and this depends on the sign of M.

Though the polynomial weighting does not affect the ability of the local extremum spline to model arbitrary functions in ℱ^H, it does impact the magnitude of the spline, ${\sup }_{x\in \mathcal{X}}|B_{\left(j,k\right)}^{\ast }\left(x\right)|$, which may cause difficulty in the prior specification. To minimize this effect it is often beneficial to construct the splines on the interval (−0·5, 0·5). Additionally, it is often beneficial to multiply M by a fixed constant to aid in prior specification.

2·2. Infill Process Prior

Bayesian methods for automatic knot selection (Biller, 2000; DiMatteo et al., 2001) commonly define priors over the number and location of knots. Using free knots presents computational challenges, while fixed knots are too inflexible; we address this by defining a prior over a branching process where the children of each generation represent knot locations that are binary infills of the previous generation. This defines a nested set of spline models such that successive generations produce knots that can be arbitrarily close.

To make these ideas explicit, define ${\mathcal{T}}_N=\left\{a/2^{N+1}:a=1,3,\dots ,2^{N+1}-1\right\}$ with N ∈ {0, 1, 2, 3, …}. Assume for the sake of exposition, and consider an infinite complete binary tree. In this tree, each node at a given depth N is uniquely labeled using an element from ${\mathcal{T}}_N$. If the node’s label is a/2^N+1, its children are labeled (2a − 1)/2^N+2 and (2a + 1)/2^N+2. For example, the node labeled 3/8 at N + 2 has children labeled 5/16 and 7/16, and the root node labeled 1/2 has children labeled 1/4 and 3/4.

We induce a prior on the set of local extremum spline basis functions through a branching process over this tree. The process starts at the root node N = 0 where the generation of children occurs via two independent Bernoulli experiments having probability of success ζ. On each success, a child is generated, and its label is added to the knot set. This process repeats until it dies out. If ζ < 0·5, the probability of extinction is 1 (Feller, 1974, page 297). To favor parsimony, we define the probability of success for a node at a given depth N to be 0·5^N+1, which decreases the probability of adding a new node the larger the tree becomes. The tree ℳ generated from this process corresponds to a knot set ${\mathcal{T}}_{\mathcal{M}}$. We complete the knot set by adding end knots {0, 1}.

Letting $K=|{\mathcal{T}}_{\mathcal{M}}|$ be the number of knots for tree ℳ including end knots, there are K + j − 1 basis functions. Letting β_k denote the coefficient on $B_{\left(j,k\right)}^{\ast }\left(x\right)$, we choose the prior:

$$p\left({\beta }_k|\mathcal{M}\right)=\pi 1_{\left({\beta }_k=0\right)}+\left(1-\pi \right)\text{Exp}\left({\beta }_k;\mathrm{\lambda }\right),1\le k\le K+j-1,$$ (3)

where Exp(β_k; λ) is an exponential distribution with rate parameter λ, π is the prior probability of β_k = 0, and the β_k are drawn independently conditionally on ℳ, π, and λ. For the intercept, we let β₀ ~ N(0, c), and we allow for greater adaptivity to the data through hyperpriors, π ~Be(ν, ω) and λ ~ Ga(δ, κ)1(λ > ε), which is a truncated gamma distribution, that is truncated slightly above zero to guarantee posterior consistency. In practice, this value is set to 10⁻⁵, making the prior indistinguishable from the Gamma distribution.

To allow uncertainty in locations of the change points, we choose the prior

$$p\left(\alpha \right)={\displaystyle \prod _{h=1}^H\text{TN}\left\{{\alpha }_h;\left(b-a\right)/2,1,a,b\right\}}$$ (4)

where TN{(b − a)/2, 1, a, b} is a normal distribution with mean (b − a)/2 and variance 1, truncated below by a and above by b with $\mathcal{X}\subset \left[a,b\right]$. If ${\alpha }_h\le \inf \mathcal{X}$ or ${\alpha }_h\ge \sup \mathcal{X}$, then the change point is removed. We assume that M is pre-specified corresponding to prior knowledge of whether the function is initially increasing or decreasing, though generalizations to place a Bernoulli or alternative prior on M are straightforward.

The prior for the change point parameters is defined such that $\mathcal{X}\subset \left[a,b\right]$. A change point placed outside of $\mathcal{X}$ allows the derivative of f to be non-zero at inf $\mathcal{X}$ or sup $\mathcal{X}$. In practice, results are insensitive to the choice of a and b. In what follows, we choose $a=\inf \left(\mathcal{X}\right)-\mathrm{\Delta }$ and $b=\sup \left(\mathcal{X}\right)+\mathrm{\Delta }$, where $\mathrm{\Delta }=\left\{\sup \left(\mathcal{X}\right)-\inf \left(\mathcal{X}\right)\right\}/2$.

2·3. Prior Properties

Define ℱ^H+ as the space of continuously differentiable functions with H or fewer local extrema, such that, for all f₀ ∈ ℱ^H+ having exactly H extrema, the first extremum from the left is a maximum, and, for all functions in f₀ ∈ ℱ^H+ having less than H extrema, the function is also in ℱ^H−1. Conversely, define ℱ^H− as the set of continuously differentiable functions with H or fewer local extrema, such that for all functions having exactly H extrema, the first from the left is a minimum, and for all functions f₀ ∈ ℱ^H− having less than H extrema, they are also in ℱ^H−1. The prior places positivity in ε−neighborhoods of any f₀ in ℱ^H− or ℱ^H+ depending on the sign of M.

Lemma 1

Letting f^LX be a randomly generated local extremum spline from the prior defined in §2·2 for all f₀ ∈ ℱ^H−1,

$$pr({\Vert f^{LX}-f_0\Vert }_{\infty }<\epsilon )>0.$$

This holds for all f₀ ∈ ℱ^H+ if H is odd and M < 0 or H is even and M > 0. Otherwise, if H is even and M > 0 or H is odd and M < 0, this holds for all f₀ ∈ ℱ^H−.

Using this result we can show posterior consistency. Assume that Y = (y₁,…, y_n)^T are observed at locations (x₁,…,x_n) such that $y_i~\mathrm{N}\{f_0(x_i),{\sigma }_0^2\}$. Following Choi & Schervish (2007), assume that the design points are independent and identically distributed from some probability distribution Q on the interval $\mathcal{X}$, or observed using a fixed design such that max(|x_i − x_i+1|) < (K₁n)⁻¹, where 0 < K₁ < 1 and i < n. Define the neighborhoods W_ε,n = {(f, σ) : ∫ |f(x) − f₀(x)|dQ_n(x) < ε, |σ/σ₀ − 1| < ε} and U_ε = {(f, σ) : d_Q(f, f₀) < ε, |σ/σ₀ − 1| < ε}, where d_Q(f₁, f₂) = inf {ε > 0 : Q[{x : |f₁(x) − f₂(x)| > ε}] < ε}. Under the assumption that the prior over σ assigns positive probability to every ε–neighborhood of σ₀, one has:

Theorem 2

Let f^LX be a randomly generated curve from the prior defined in §2·2 with f₀ ∈ ℱ^H−1. If $P_{f_0,{\sigma }_0}$ is the joint distribution of ${\left\{y_i\right\}}_{i=1}^{\infty }$ conditionally on ${\left\{x_i\right\}}_{i=1}^{\infty }$, ${\left\{Z_i\right\}}_{i=1}^{\infty }$ is a sequence of open subsets in ℱ^H−1 that is defined by W_ε,n for fixed designs or by U_ε for random designs, and ∏_n is the posterior distribution of f₀ given ${\left\{y_i\right\}}_{i=1}^n$, then

$${\displaystyle {\prod }_n(f\in {\mathcal{Z}}_n^C|y_1,\dots ,y_n)\to 0\mathit{\text{almost surely}}\left[P_{f_0,{\sigma }_0}\right]}.$$

Further, for all H odd if M < 0, this relation holds for f₀ ∈ ℱ^H+, otherwise it holds for f₀ ∈ ℱ^H−. Similarly, for H even if M > 0, then f₀ ∈ ℱ^H+, otherwise it holds for f₀ ∈ ℱ^H−.

The proof of this consistency result follows from Choi & Schervish (2007) and the prior positivity result above. The condition on the prior over σ² can be satisfied with an inverse-Gamma distribution.

2·4. Bayes Factors for Testing Curve Shapes

Our approach allows one to define the shape of the curve through the α vector and to place prior probability on a class of functions having a given shape, i.e the number and type of extrema in $\mathcal{X}$. When there are flat regions of f₀ the shape of the curve is not uniquely identifiable based upon the configuration of α, and hypothesis tests may be inconclusive. For an example of this, see the consistency arguments for monotone curve testing in Scott et al. (2015). In what follows, we assume that $|f_0^{\prime }(x)|>0$ at all points in $\mathcal{X}$ except within flat regions.

Let ℍ₁ and ℍ₂ denote two distinct and non-nested sets of α values, corresponding to distinct shapes. These sets are defined by the number of ${\alpha }_h\in \mathcal{X}$, the number of ${\alpha }_h\le \inf (\mathcal{X})$, and the number of ${\alpha }_h\ge \sup (\mathcal{X})$. One can compute pr(Y|f₀ ∈ ℍ₁) and pr(Y|f₀ ∈ ℍ₂), with the corresponding Bayes factor between the two shapes being

$$B{F}_{12}=\frac{\text{pr}(Y|f_0\in {ℍ}_1)}{\text{pr}(Y|f_0\in {ℍ}_2)}.$$ (5)

This quantity is not available analytically, but can be estimated through posterior simulation by monitoring the α and β vectors.

Any two shapes falling within ℱ^H can be compared using this approach. Alternatively, one may be interested in the hypothesis that f₀ is in a class of functions with at least K extrema. For example, one may wish to assess whether or not the function is monotone. In this case, one can define ℍ₁ to correspond to functions in ℱ^H with F or more extrema and ${ℍ}_2={ℍ}_1^c$ to functions with less than F extrema. The value of H can be elicited as an upper bound on the number of extrema to avoid highly irregular functions. For such tests, the following result holds.

Proposition 2

Let ℍ₁ be the class of functions in ℱ^H with F or more extrema and ${ℍ}_2={ℍ}_1^c\cap {\mathcal{F}}^H$. If f₀ ∈ ℍ₁, then

$$B{F}_{12}\to \infty$$

as n → ∞

This result, an application of Theorem 1 in Walker et al. (2004), It follows from the fact that local extremum spline representations having fewer than F change points can never be arbitrarily close to the function of interest.

3. Posterior Computation

We rely on Godsill (2001) to develop a reversible jump Markov chain Monte Carlo algorithm to sample between models. Consider moves between models ℳ and ℳ′, where the model ℳ′ has one extra knot that is a child of a node also in ℳ. As described further in the Supplementary Material, most of the local extremum spline basis functions for model ℳ and ℳ′ are identical, with only j + 2 different functions. Let β_−ℳ denote the coefficients on all the splines that are the same as well as σ², π and λ, which are parameters shared between both models. The remaining spline coefficients are β_ℳ and ${\beta }_{{\mathcal{M}}^{\prime }}$ for models ℳ and ℳ′, respectively. As in Godsill, given the shared vector β_−ℳ, we marginalize β_ℳ and ${\beta }_{{\mathcal{M}}^{\prime }}$out of the posterior to compute p(ℳ′ | Y, β_−ℳ) and p(ℳ | Y, β_−ℳ). This marginalization requires numerical integration of multivariate normal distributions, which is performed using Genz (1992) and Genz & Kwong (2000). The probability of a move between two models is determined by the ratio

$$h=\frac{q\left(\mathcal{M};{\mathcal{M}}^{\prime }\right)p\left({\mathcal{M}}^{\prime }|Y,{\beta }_{-\mathcal{M}}\right)}{q\left({\mathcal{M}}^{\prime };\mathcal{M}\right)p\left(\mathcal{M}|Y,{\beta }_{-\mathcal{M}}\right)},$$ (6)

where a knot insertion is made with probability min(1, h), a knot deletion is made with probability min(1, 1/h), and q(ℳ; ℳ′) is the transition probability between ℳ and ℳ′.

All proposals are made between models that are nested and differ by only one knot. When the current model has no children we propose a knot insertion with unit probability. Otherwise, the proposal adds or deletes a knot with probability 1/2, and the inserted or deleted knot is chosen uniformly. For a knot insertion, as we are going from model ℳ to ℳ′, the available knots are represented by all failures in the branching process that generated ℳ. A knot deletion going from model ℳ′ to ℳ represents all of the nodes in the branching process that generated ℳ′ that do not have any children. All other parameters, including the spline coefficients, are sampled in Gibbs steps described in the supplement.

The posterior distribution is often multimodal, with the sampler getting stuck in a single mode, when widely different parameter values have relatively large support by the data, with low posterior density between these isolate modes. To increase the probability of jumps between modes, a parallel tempering algorithm (Geyer, 1991, 2011) is implemented.

4. Simulation

4·1. Simulation Specification

We investigate our approach through simulations for functions having 0, 1, or 2 local extrema interior to $\mathcal{X}$. For all simulations, we place a Ga(1, 1) prior over σ. For the hyper prior on π, we let ν = 2 and ω = 18, which puts low prior probability on flat curves. Additionally, for the hyper prior over λ, we let δ = 0·2 and κ = 2, which favors smaller values of β. All local extremum splines were constructed using B-splines of order 2 with M = 100.

The Markov chain Monte Carlo algorithm was implemented in the R programming language with some subroutines written in C++ and is available from the first author. Depending on the complexity of the function, the algorithm took between 60 and 90 seconds per 50, 000 samples using one core of a 3·3 gigahertz Intel i7-5830k processor. Parallelizing the tempering algorithm on multiple cores may substantially reduce the computation time. Additional information on the convergence of the algorithm, as well as impact of the B-spline order used, is provided in the Supplementary Material.

4·2. Curve Fitting

We compare the local extremum spline approach to other nonparametric methods, including Bayesian P-splines (Lang & Brezger, 2004), a smoothing spline method described in Green & Silverman (1993), and a frequentist Gaussian process approach described in Chapter 5 of Shi & Choi (2011). We consider seven different curves with between 0 and 2 extrema and compare the fits of the other approaches and of a local extremum spline specified to have at most H = 2 change points. The following true curves are investigated:

$$\begin{array}{ll}{f}_1\left(x\right)=10x^2,\hfill & f_2\left(x\right)=2+20\Phi \left\{\left(x-0\cdot 5\right)/0\cdot 071\right\},\hfill \\ f_3\left(x\right)=5\cos \left(\pi x\right),\hfill & f_4\left(x\right)=10{\left(x-0\cdot 5\right)}^2,\hfill \\ f_5\left(x\right)=-2\cdot 5+10\exp \left\{-50{\left(x-0\cdot 35\right)}^2\right\},\hfill & f_6\left(x\right)=1+2\cdot 5\sin \left\{2\pi \left(x+8\right)\right\}+10x,\hfill \\ f_7\left(x\right)=5min\left(2\pi x\right)/{\left(x+0\cdot 75\right)}^3-2\cdot 5\left(x+10\cdot 5\right).\hfill & \hfill \end{array}$$

We set y_i = f_j(x_i) + ε_i with ε_i ~ N(0, σ²). Functions f₁, f₂ and f₃ are monotone, f₄ and f₅ have one change point, and f₆ and f₇ have two change points. For each simulation, a total of 100 equidistant points were sampled in $\mathcal{X}=\left[0,1\right]$. We consider σ² = 1, 4. For each simulation condition, 250 data sets were generated, fitted and compared using the mean squared error, $n^{-1}{\displaystyle {\sum }_{i=1}^n{\left\{\widehat{f}\left(x_i\right)-f\left(x_i\right)\right\}}^2}$, for the local extremum spline, smoothing spline, Bayesian P-spline, and Gaussian process approaches.

For the local extrema approach, we collected 50,000 Markov chain Monte Carlo samples, with the first 10, 000 samples disregarded as burn-in. For the parallel tempering algorithm, we specify 12 parallel chains with {κ₁, …, κ₁₂} = {1/30, 1/24, 1/12, 1/9, 1/5, 1/3·5, 1/2, 1/1·7, 1/1·3, 1/1·2, 1/1·1, 1}, and monitor the target chain with κ₁₂ = 1. The P-spline approach was defined using 30 equally-spaced knots, and the prior over the second-order random walk smoothing parameter was IG(1, 0·0005), distribution, which was one of the recommended choices in Lang & Brezger (2004).·In this approach, 25,000 posterior samples were taken, discarding the first 5,000 as burn in. For the smoothing spline method, the R function ‘smooth.spline’ was used. Finally, the Gaussian process approach used a frequentist implementation given in the R package ‘GPFDA.’

Table 1 gives the integrated mean squared error of the various approaches. All numbers marked with an asterisk are significantly different from local extremum splines. The local extremum approach integrated mean square error is always smaller than the others, and in most cases it is significantly different at the 0·05 level. Generally, when there is a high signal-to-noise ratio, the methods perform similarly, but when the ratio decreases, specifically in flat regions, the local extremum approach was superior as it removed artifactual bumps from the estimate.

Table 1.

Estimated mean squared error for all functions. For each function, the left value represents the simulation condition σ² = 1 and the right value represents the simulation condition σ² = 4. Asterisks signify that the number is significantly different than the local extremum spline at the one-sided 0·05 level.

True Function	Local Extremum Splines	Smoothing Splines	Bayesian P-Splines	Gaussian Process
f1	1·60/0·49	2·11*/0·58	2·28*/0·55	2·15*/0·71*
f2	2·59/0·09	4·19*/0·13*	3·82*/0·11*	5·26*/0·15*
f3	1·57/0·49	2·43*/0·67*	2·26*/0·92*	2·64*/0·79*
f4	1·70/0·49	2·10*/0·56*	2·15*/0·49	1·90*/ 0·59*
f5	2·55/0·61	3·69*/1·12*	3·39*/0·98*	3·90*/1·14*
f6	2·17/0·69	2·57/0·72	5·16*/0·72	2·44/0·79*
f7	2·38/0·66	3·39*/1·05*	3·96*/0·85*	3·30*/0·90*

4·3. Hypothesis Testing

We perform a simulation experiment investigating the method’s ability to correctly identify the shape of the response function for three sets of hypotheses. In the first case, the null hypothesis is the set of all functions with one or more extremum, and the alternative, ℍ₁, is the set of all monotone functions. In the second test, the null consists of all monotone functions, and the alternative, ℍ₂, is all functions with one or more extremum. Finally, for the third test the null hypothesis is the set of functions having at most one extrema, and the alternative, ℍ₃, is the set of functions with two extrema first having a local maximum followed by a local minimum. Functions are defined on $\mathcal{X}=\left[0,1\right]$. The nine functions used in this simulation are:

$$\begin{array}{ll} \hfill & {ℍ}_1\hfill \\ g_1\left(x\right)=2+0\cdot 5x+\Phi \left\{\left(x-0\cdot 5\right)/0\cdot 071\right\},\hfill & g_2\left(x\right)=0\cdot 5\sin \left\{2\pi \left(x+8\right)\right\}+4\cdot 75x,\hfill \\ g_3\left(x\right)=1+2\cdot 25x,\hfill & \hfill \end{array}$$

$$\begin{array}{ll} \hfill & {ℍ}_2\hfill \\ g_4\left(x\right)=-4{\left(x-0\cdot 75\right)}^2,\hfill & g_5\left(x\right)=1+2x-1\cdot 56\exp \left\{-50{\left(x-0\cdot 5\right)}^2\right\},\hfill \\ g_6\left(x\right)=15{\left(x-0\cdot 5\right)}^3{1}_{\left(x<0\cdot 5\right)}+0\cdot 3\left(x-0\cdot 5\right)-\exp \left\{-250\left(x-0\cdot 25\right)\right\},\hfill & \hfill \end{array}$$

$$\begin{array}{ll} \hfill & {ℍ}_3\hfill \\ g_7\left(x\right)=0.85\sin \left\{2\pi \left(x+8\right)\right\}+4\cdot 75x,\hfill & g_8\left(x\right)=g_5\left(x\right),\hfill \\ g_9\left(x\right)=5\sin \left(2\pi x\right)/{\left(x+0\cdot 75\right)}^3-2\cdot 5\left(x+10\cdot 5\right)+2.\hfill & \hfill \end{array}$$

For the simulation, data are generated assuming y_i = g_j(x_i) + ε_i, where ε_i ~ N(0, σ²) and σ² = 1. We consider sample sizes n = 100, 200, 300, and 400, with 50 data sets constructed where points are sampled evenly across $\mathcal{X}$, for each sample condition. The local extremum approach is as above except, but 150,000 posterior samples are taken with the first 10,000 disregarded as burn-in. For tests ℍ₁ and ℍ₂, the local extremum approach is compared with the Bayesian method of Salomond (2014) and the frequentist methods of Baraud et al. (2005) and Wang & Meyer (2011). For the method of Baraud et al. we use the test where ℓ_n = 25, and for the method of Wang and Meyer we use k = 4 splines, which were the most powerful tests presented in the respective articles.

The Bayesian tests produce Bayes factors, while the frequentist tests have corresponding test statistics. We compare the methods based upon area under the receiver operating curve. For the simulation, the false positive rate was computed from the values of the test statistics for the other functions not in the test set. As a frequentist calibration of our Bayesian test, one can choose a threshold on the Bayes factor to control the type I error rate at a specified level based on an approximation to the distribution of the Bayes factor under the null hypothesis. We describe this approximation in the Supplementary Material.

Figure 1 shows the receiver operating curve for hypothesis ℍ₁. This shows that the local extremum approach is superior to the other three approaches across all false positive rates. Further, the estimated area under the receiver operating curve is 0·94, better than the approaches of Salomond at 0·86, Baraud at 0·77, and Wang and Meyer at·0·74. When looking at the impact of sample size on the tests, the power of the local extrema approach increases as the sample size increases, does so at a rate greater than competitors, and is similarly superior for hypothesis ℍ₂, data not shown.

Fig. 1.

The receiver operating curve for the four tests defined for hypothesis ℍ₁ for all 1,400 simulations. The black line represents the local extremum spline, dashed line the approach of Salomond, dashed-dotted line the approach of Baraud, and dotted line the approach of Wang and Meyer.

For hypothesis ℍ₃, there is not an equivalent methodology in the literature, but the performance of our approach is excellent. The area under the receiver operator curve is 0·94. For the Bayes factor cut point of 6, Table 2 gives results across all simulation conditions.·Our test achieves high power for function g₇, even though it differs this function is only slightly different from g₃. Function g₈ is the same as g₅, this simulation gives evidence that the departure from monotonicity may be due to the pronounced U shape in the data and not necessarily because there are two extrema, which requires more data to conclude in favor of ℍ₃.

Table 2.

Percent of samples where the model was correctly chosen as having two extrema, which is hypothesis ℍ₃, using a cut point of 6.

Function	n
	100	200	300	400
g7	78	90	98	96
g8	14	32	22	46
g9	76	88	98	100

4·4. Seasonal Influenza and Pneumonia Death Rate

In temperate climates, the prevalence of influenza peaks in the winter months while dropping in the warmer months. Estimating this seasonal effect as well as departures from this effect, may be of interest when estimating the magnitude of an influenza epidemic. Here, we expect a peak in the winter months followed by a trough in the summer months. Parametric models for this pattern may not be adequate to model the observed phenomena, and smoothing approaches do not guarantee this pattern. We use local extremum splines, setting H = 2, to estimate this trend for Virginia, North Carolina and South Carolina for data collected by the Centers for Disease Control and Prevention National Center for Health Statistics Mortality surveillance branch.

Figure 2 plots the estimated mortality rates, estimated using an additive model defined by a quadratic trend representing a decrease in mortality over time, a seasonal component defined using local extremum spline, and a P-spline that represents departures from the overall trend. This seasonal component is different from the trend published by the Centers for Disease Control (Viboud et al., 2010), mainly due to the asymmetry in the local extrema approach during the winter months, which cannot be captured by a single sinusoidal function.

Fig. 2.

Estimate of the expected rate of seasonal influenza and pnuemonia deaths using the local extremum spline, black line, compared to the observed rate of influenza and pnuemonia deaths estimated using the Center for Disease Control’s standard approach, gray line. Dots represent observed state level influenza and pneumonia percentages.

Appendix 1

Proofs of results

Proof of Proposition 1

It is well known that ${\displaystyle {\sum }_{k=1}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}\left(x\right)}$ is continuous for j ≥ 1 and for j ≥ 1 and for all $x\in \mathcal{X}$. Further, ${\displaystyle {\prod }_{h=1}^H\left(x-{\alpha }_h\right)}$ is a polynomial; therefore, ${\displaystyle {\prod }_{h=1}^H\left(x-{\alpha }_h\right){\displaystyle {\sum }_{k=1}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}\left(x\right)}}$ is continuous with anti-derivative ${\displaystyle {\sum }_{k=0}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}^{\ast }\left(x\right)}$.

If β_k ≥ 0 for all k ≥ 1, then ${\displaystyle {\sum }_{k=0}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}\left(x\right)}\ge 0$ for all $x\in \mathcal{X}$ and $f^{\prime }\left(x\right)={\displaystyle {\prod }_{h=1}^H\left(x-{\alpha }_h\right)}{\displaystyle {\sum }_{k=1}^{K+j-1}{\beta }_k{B}_{\left(j,k\right)}\left(x\right)}$ can only change sign when x = α_h. Thus, there are at most H local extrema interior to $\mathcal{X}$, with $f\in {\mathcal{F}}^H$. □

Proof of Theorem 1

Consider f₀ ∈ ℱ^H, where f₀ has exactly H change-points. Functions with less than H change points can be modeled by removing the required change point parameters from $\mathcal{X}$ and continuing with the proof below.

Let f^BS be a taut B-spline approximation of f₀ of order j + 1 defined on the knot set $\mathcal{T}$

having exactly H extrema such that

$${\Vert f_0-f^{\text{BS}}\Vert }_{\infty }<\mathrm{\Delta }C.$$

Here f^BS is defined on $\mathcal{T}$, where Δ = max_k |τ_k − τ_k+j| < 1. As f₀ and f^BS are continuous and differentiable, we define C such that ‖f₀‖ < C < ∞ and ‖f^BS‖ < C. The measurable set of taut spline functions $L_{f^{\text{BS}}}^{\ast }=\left\{f^{\text{BS}}:{\Vert f_0-f^{\text{BS}}\Vert }_{\infty }<\mathrm{\Delta }C\right\}$ can be shown to exist (de Boor, 2001) and we define a map $\mathcal{G}:L_{f^{\text{BS}}}^{\ast }\to L_{f^{\text{LX}}}^{\ast }$ where $L_{f^{\text{LX}}}^{\ast }$ a subset of all possible local extremum spline functions with H change points. Consider

$${\Vert f^{\text{BS}}-f^{LX}\Vert }_{\infty }=\underset{x\in \mathcal{X}}{\sup }|f^{\text{BS}}\left(x\right)-f^{\text{LX}}\left(x\right)|$$ (A1)

and let β₀ = f^BS(0). For the exactly H extrema ${\alpha }_1^{\text{BS}}<\mathit{ldots}<{\alpha }_H^{\text{BS}}$ in f^BS defined by the taut spline, set ${\alpha }_h={\alpha }_h^{\text{BS}}$. Additionally, if $f^{\text{BS}}\left({\alpha }_1^{\text{BS}}\right)-f^{\text{BS}}\left(0\right)\ge 0$ with H odd, then set M = −1; otherwise set M = 1. In the case where $f^{\text{BS}}\left({\alpha }_1^{\text{BS}}\right)-f^{\text{BS}}\left(0\right)<0$ with H odd, then set M = 1 otherwise set M = −1.

Rewriting the right hand side of (A1) in a form based upon the derivative, we have

$$\underset{x\in \mathcal{X}}{\sup }\left|{\displaystyle {\int }_{-\infty }^x{\displaystyle \sum _{k=1}^{K+j-1}{\kappa }_k{B}_{\left(j,k\right)}\left(\xi \right)-{\beta }_{k}G\left(\xi \right)B_{\left(j,k\right)}\left(\xi \right)d\xi }}\right|,$$

$$\le {\displaystyle \sum _{k=1}^{K+j-1}\underset{x\in \mathcal{X}}{\sup }}\left|{\displaystyle {\int }_{{\tau }_k}^x{\kappa }_k{B}_{\left(j,k\right)}\left(\xi \right)-{\beta }_{k}G\left(\xi \right)B_{\left(j,k\right)}\left(\xi \right)d\xi }\right|,$$

where the derivative of f^BX is based upon the derivative formula for B-Splines (de Boor, 2001) and $G\left(\xi \right)={\displaystyle {\prod }_{h=1}^H\left(\xi -{\alpha }_h\right)}$.

Because of the taut spline construction of f^BS, we know that for all k, h such that α_h ∉ [τ_k, τ_k+j−1] one has sgn(κ_k) = sgn(G(x)), for all x ∈ [τ_k, τ_k+j−1]. Here sgn(·) is the signum function. On each of these intervals let

$${\beta }_k=\frac{{\displaystyle {\int }_{{\tau }_k}^{{\tau }_{k+j-1}}{\kappa }_k{B}_{\left(j,k\right)}\left(\xi \right)d\xi }}{{\displaystyle {\int }_{{\tau }_k}^{{\tau }_{k+j-1}}G\left(\xi \right)B_{\left(j,k\right)}\left(\xi \right)d\xi }}.$$

As B_(j,k)(x) ≥ 0, we have β_k ≥ 0; further, one has

$${\displaystyle {\int }_{{\tau }_k}^{{\tau }_{k+j-1}}{\kappa }_k{B}_{\left(j,k\right)}\left(\xi \right)-{\beta }_{k}G\left(\xi \right)B_{\left(j,k\right)}\left(\xi \right)d\xi =0}$$

for all intervals such that α_h ∈ [τ_k, τ_k+j−1].

For the at most H coefficients defined on splines that are nonzero in the intervals α_h ∈ [τ_k, τ_k+j−1], set these coefficients to zero. As there are a finite number of intervals whose error is non-zero and f^BS is bounded, the maximum error is at most (H + 1)(j + 1)ΔC for any x and

$${\Vert f^{\text{BS}}-f^{\text{LX}}\Vert }_{\infty }\le \left(H+1\right)\left(j+1\right)\mathrm{\Delta }C.$$

Consequently, for any ε, consider taut B-spline constructions on knot sets $\mathcal{T}$ such that Δ ≤ ε[{2(H + 1)(j + 1)}C]⁻¹ that also have ‖f₀ − f^BS‖_∞ < ε/2. Then one has

$${\Vert f_0-f^{LX}\Vert }_{\infty }\le {\Vert f_0-f^{\text{BS}}\Vert }_{\infty }+{\Vert f^{\text{BS}}-f^{\text{LX}}\Vert }_{\infty }=\frac{\epsilon }{2}+\frac{\epsilon }{2},$$

completing the proof. □

Proof of Lemma 1

The function $\mathcal{G}$ in Theorem 1 is measurable. If $L_{f^{\text{BS}}}^{\ast }$ is measurable on some abstract measure space, one has $\text{pr}\left({\Vert f^{\text{LX}}-f_0\Vert }_{\infty }<\epsilon |{\mathcal{T}}_{\mathcal{M}}\right)>0$ for any ε > 0 and some ${\mathcal{T}}_{\mathcal{M}}$. Given the prior puts probability over knot sets having knot spacings that are arbitrarily close, that is Δ ≤ ε[{2(H + 1)(j + 1)}C]⁻¹ as in Theorem 1, we conclude that $\text{pr}\left({\Vert f_0-f^{\text{LX}}\Vert }_{\infty }<\epsilon \right)=\text{pr}\left({\Vert f^{\text{LX}}-f_0\Vert }_{\infty }<\epsilon |{\mathcal{T}}_{\mathcal{M}}\right)\text{pr}\left({\mathcal{T}}_{\mathcal{M}}\right)>0$ for all ε > 0. □

Proof of Theorem 2

We verify the conditions given in A1 and A2 of Theorem 1 of Choi & Schervish (2007). If there is positive prior probability, Lemma 1, within all neighborhoods of (f₀, σ²), one can use Choi & Schervish (2007), section 4, to show that the conditions of A1 of Theorem 1 are met. To verify A2 we have that ℱ^H+ and ℱ^H+ are subsets of all continuous differentiable functions on $\mathcal{X}$ which were considered in Choi & Schervish (2007); consequently, we appeal to Theorem 2 and 3 of Choi & Schervish (2007) to construct suitable tests for both random and fixed designs using W_ε,n and U_ε. We need only verify (iii) in part A2.

As in Choi & Schervish (2007), assume that $M_n=\mathcal{O}\left(n^{\alpha }\right)$ with 1/2 < α < 1. We show that $\text{pr}\left({\Vert f^{\mathit{\text{LX}}}\left(x\right)\Vert }_{\infty }<M_n\right)\le C_0\exp \left(-n{C}_1\right)$ and $\text{pr}\left({\Vert {f^{\prime }}^{\mathit{\text{LX}}}\left(x\right)\Vert }_{\infty }>M_n\right)\le C_2\exp \left(-n{C}_3\right)$ for some C₀, C₁, C₂, C₃ > 0. Define $B_{\left(j,k,\mathcal{M},\alpha \right)}^{\ast }\left(X\right)$ as the design matrix given model ℳ and a particular α figuration. Let $A=\underset{\mathcal{M},k,\alpha ,x}{\sup }|B_{\left(j,k,\mathcal{M},\alpha \right)}^{\ast }\left(X\right)|$ and K_ℳ be the number of spline coefficients in model ℳ then

$$\text{pr}\left\{{\Vert f^{LX}\left(x\right)\Vert }_{\infty }>M_n\right\}={\displaystyle \int pr\left\{{\Vert {\displaystyle \sum _{k=1}^{K_{\mathcal{M}}}{\beta }_k{B}_{\left(j,k,\mathcal{M},\alpha \right)}^{\ast }}\left(X\right)\Vert }_{\infty }>M_n|\mathcal{M}\right\}d\alpha }d\mathcal{M}d\pi d\mathrm{\lambda }\le {\displaystyle \int pr\left\{{\displaystyle \sum _{k=1}^{K_{\mathcal{M}}}{\Vert {\beta }_k{B}_{\left(j,k,\mathcal{M},\alpha \right)}^{\ast }\left(X\right)\Vert }_{\infty }>M_n|\mathcal{M},\beta >0}\right\}d\alpha }d\mathcal{M}d\pi d\mathrm{\lambda }\le {\displaystyle \int pr\left\{{\displaystyle \sum _{k=1}^{K_{\mathcal{M}}}{\beta }_{k}A>M_n|\mathcal{M},\beta >0}\right\}d\alpha }d\mathcal{M}d\pi d\mathrm{\lambda }\le \exp \left(-M_{n}t\right){\displaystyle \int {\displaystyle \sum _{k=1}^{\mathcal{M}}\left\{{\left(\frac{\mathrm{\lambda }-\pi t}{\mathrm{\lambda }-t}\right)}^{K_{\mathcal{M}}}\text{pr}\left(\mathcal{M}\right)\right\}d\alpha d\pi d\mathrm{\lambda }.}}$$ (A2)

Where the last inequality comes from the the Chernoff bounds.

Now let pr*(ℳ) be the probability of a branching process where ζ < 0.5 is constant for all children, then there exists a $\mathcal{K}$ such that {pr*(ℳ)}² ≥ pr (ℳ) for all ℳ such that $K_{\mathcal{M}}\ge \mathcal{K}$. Partition the sum into the finite sum where $K_{\mathcal{M}}<\mathcal{K}$ and the infinite sum $K_{\mathcal{M}}\ge \mathcal{K}$. As the finite sum is finite for all 0 < t < λ, continuing with (A2):

$$\le \exp \left(-M_{n}t\right){\displaystyle \int C_1+\left[\sum ^{K_{\mathcal{M}}\ge \mathcal{K}}{\left(\frac{\mathrm{\lambda }-\pi t}{\mathrm{\lambda }-t}\right)}^{K_{\mathcal{M}}}{\left\{p{r}^{\ast }\left(\mathcal{M}\right)\right\}}^2\right]d\alpha d\pi d\mathrm{\lambda }}\le \exp \left(-M_{n}t\right){\displaystyle \int C_1+C_2\left[\sum ^{K_{\mathcal{M}}\ge \mathcal{K}}{\left(\frac{\mathrm{\lambda }-\pi t}{\mathrm{\lambda }-t}\zeta \right)}^{K_{\mathcal{M}}}p{r}^{\ast }\left(\mathcal{M}\right)\right]d\alpha d\pi d\mathrm{\lambda }}\le \exp \left(-M_{n}t\right){\displaystyle \int \left(C_1+C_2\right)d\alpha d\pi d\mathrm{\lambda }}$$

where the last inequality exists as λ is bounded above zero, which implies one can choose some t < λ such that (λ − πt)/(λ − t)ζ < 1. This implies that

$$\text{pr}\left\{{\Vert f^{LX}\left(x\right)\Vert }_{\infty }>M_n\right\}\le C_0\exp \left(-n{C}_1\right).$$

A derivation similar to the above can be used to show the same holds for pr(‖f′^LX(x)‖_∞ > M_n) ≤ C₂ exp(−nC₃). One can find a $B=\underset{\mathcal{M},k,\alpha ,x}{\sup }|B_{\left(j,k,\mathcal{M},\alpha \right)}^{\prime \ast }\left(X\right)|$ and substitute B for A and $B_{\left(j,k,\mathcal{M},\alpha \right)}^{\prime \ast }\left(X\right)$ for $B_{\left(j,k,\mathcal{M},\alpha \right)}^{\ast }\left(X\right)$ in the above derivation.