Estimation and Inference in Generalized Additive Coefficient Models for Nonlinear Interactions with High-Dimensional Covariates

Abstract

In the low-dimensional case, the generalized additive coefficient model (GACM) proposed by Xue and Yang [Statist. Sinica 16 (2006) 1423–1446] has been demonstrated to be a powerful tool for studying nonlinear interaction effects of variables. In this paper, we propose estimation and inference procedures for the GACM when the dimension of the variables is high. Specifically, we propose a groupwise penalization based procedure to distinguish significant covariates for the “large p small n” setting. The procedure is shown to be consistent for model structure identification. Further, we construct simultaneous confidence bands for the coefficient functions in the selected model based on a refined two-step spline estimator. We also discuss how to choose the tuning parameters. To estimate the standard deviation of the functional estimator, we adopt the smoothed bootstrap method. We conduct simulation experiments to evaluate the numerical performance of the proposed methods and analyze an obesity data set from a genome-wide association study as an illustration.

1. Introduction

Regression analysis is a commonly used statistical tool for modeling the relationship between a scalar dependent variable Y and one or more explanatory variables denoted as T = (T₁, T₂, …, T_p)^T. To study the marginal effects of the predictors on the response, one may fit a generalized linear model (GLM),

$$E(Y|\mathbf{T})=\mu (\mathbf{T})=g^{-1}\{\eta (\mathbf{T})\},\eta (\mathbf{T})=\sum _{\ell =1}^p{\alpha }_{\ell 0}{T}_{\ell },$$ (1)

where g is a known monotone link function, and α_ℓ0, 1 ≤ ℓ ≤ p, are unknown parameters. Sometimes, the effect of one variable may change with other variables; that is, there is an interaction effect. By letting T₁ = 1, to incorporate the interaction effects of T and the other variables, denoted as X = (X₁,…, X_d)^T, model (1) can be modified to E(Y|X, T) = μ(X, T) = g⁻¹{η(X, T)} with

$$\eta (\mathbf{X},\mathbf{T})={\alpha }_{10}+\sum _{\ell =2}^p{\alpha }_{\ell 0}{T}_{\ell }+\sum _{k=1}^d{\alpha }_{1k}{X}_k+\sum _{\ell =2}^p\sum _{k=1}^d{\alpha }_{\ell k}{X}_k{T}_{\ell },$$ (2)

where α_ℓk for 0 ≤ k ≤ d and 1 ≤ ℓ ≤ p are parameters. After a direct reformulation, model (2) can be written as

$$\eta (\mathbf{X},\mathbf{T})=\sum _{\ell =1}^p({\alpha }_{\ell 0}+\sum _{k=1}^d{\alpha }_{\ell k}{X}_k)T_{\ell }.$$ (3)

Here the effect of each T_ℓ changes linearly with X_k. However, in practice, this simple linear relationship may not reflect the true changing patterns of the coefficient with other covariates. We here use an example of gene and environment (G × E) interactions for illustration. It has been noticed in the literature that obesity is linked to genetic factors. Their effects, however, can be altered under different environmental factors such as sleeping hours [Knutson (2012)] and physical activity [Wareham, van Sluijs and Ekelund (2005)]. To have a rough idea of how the effects of the genetic factors change with the environment, we explore data from the Framingham Heart Study [Dawber, Meadors and Moore (1951)]. In Figure 1 we plot the estimated mean body mass index (BMI) against sleeping hours per day and activity hours per day, respectively, for people with three possible genotype categories represented by AA, Aa and aa, and for one single nucleotide polymorphism (SNP). A detailed description and the analysis of this data set are given in Section 5. We define allele A as the minor (less frequent) allele. This figure clearly shows different nonlinear curves for the three groups in each of the two plots. By letting T_ℓ be the indicator for the group ℓ, the linear function in model (3) is clearly misspecified.

Fig. 1. Plots of the estimated BMI against sleeping hours per day (left panel) and activity hours per day (right panel) for the three genotypes AA (solid line), Aa (dashed line) and aa (dotted line) of SNP rs242263 in the Framingham study, where A is the minor allele.

To relax the linearity assumption, we allow each α_ℓk X_k term to be an unknown nonlinear function of X_k, and thus extend model (3) to the generalized additive coefficient model (GACM)

$$\eta (\mathbf{X},\mathbf{T})=\sum _{\ell =1}^p\{{\alpha }_{\ell 0}+\sum _{k=1}^d{\alpha }_{\ell k}(X_k)\}{T}_{\ell }=\sum _{\ell =1}^p{\alpha }_{\ell }(\mathbf{X})T_{\ell }.$$ (4)

For identifiability, the functional components satisfy E{α_ℓk(X_k)} = 0 for 1 ≤ k ≤ d and 1 ≤ ℓ ≤ p. The conditional variance of Y is modeled as a function of the mean, that is, var(Y|X, T) = V{μ(X, T)} = σ²(X, T). In each coefficient function of the GACM, covariates X_k are continuous variables. If some of them are discrete, they will enter linearly. For example, if X_k is binary, we let α_ℓk(X_k) = α_ℓk X_k. In such a case, model (4) turns out to be a partially linear additive coefficient model. The linearity of (4) in T_ℓ is particularly appropriate when those factors are discrete, for example, SNPs in a genome-wide association study (GWAS), as in the data example of Section 5.

For the low-dimensional case that the dimensions of X and T are fixed, estimation of model (4) has been studied; see Liu and Yang (2010), Xue and Liang (2010), Xue and Yang (2006) for a spline estimation procedure and Lee, Mammen and Park (2012) for a backfitting algorithm. In modern data applications, model (4), however, is particularly useful when p is large. For example, in GWAS, the number of SNPs, which is p, can be very large, but the dimension of X such as the environmental factors, which is d, is inevitably relatively small. Moreover, the number of variables in T which have nonzero effects is small. It therefore, poses new challenges to apply model (4) to the high-dimensional case including: (i) how to identify those important variables in T, (ii) how to estimate the coefficient functions for the important covariates and (iii) how to conduct inferences for the nonzero coefficient functions. For example, it is of interest to know whether they are a function of a specific parametric form such as constant, linear or quadratic, etc.

In the high-dimensional data setting, studying nonlinear interaction effects has found much attention in recent years, and a few strategies have been proposed. For example, Jiang and Liu (2014) proposed to detect variables under the general index model, which enables the study of high-order interactions among components of continuous predictors, which are assumed to have a multivariate normal distribution. Moreover, Lian (2012) considered variable selection in varying coefficient models which allows the coefficient functions to depend on one index variable, such as a time-dependent variable.

When we would like to see how the effect of each genetic factor changes under the influence of multiple environmental variables, the proposed high-dimensional GACM (4) becomes a natural approach to consider, since both the index model [Jiang and Liu (2014)] and the varying coefficient model [Lian (2012)] cannot address this question; the former is used to study interactions of components in a set of continuous predictors, and the latter only allows one index variable. For model selection and estimation, we apply a groupwise penalization method. Moreover, most existing high-dimensional nonparametric modeling papers [Lian (2012), Meier, van de Geer and Bühlmann (2009), Ravikumar et al. (2009), Wang et al. (2014), Huang, Horowitz and Wei (2010)] focus on variable selection and estimation. In this paper, after variable selection, we also propose a simultaneous inferential tool to further test the shape of the coefficient function for each selected variable, which has not been studied in the previous works.

To this end, we aim to address questions (i)–(iii). Specifically, for estimation and model selection, we apply a groupwise regularization method based on a penalized quasi-likelihood criterion. The penalty is imposed on the L₂ norm of the spline coefficients of the spline estimators for α_ℓ(·). We establish the asymptotic consistency of model selection and estimation for the proposed group penalized estimators with the quasi-likelihood criterion in the high-dimensional GACM (4). We allow p to grow with n at an almost exponential order. Importantly, establishment of these results is technically more difficult than other work based on least squares, since no closed-form of the estimators exists from the penalized quasi-likelihood method.

After selecting the important variables, the next question of interest is what shapes the nonzero coefficient functions may have. Then we need to provide an inferential tool to further check whether a coefficient function has some specific parametric form. For example, when it is a constant or a linear function, the corresponding covariate has no or linear interaction effects with another covariate, respectively. For global inference, we construct simultaneous confidence bands (SCBs) for the nonparametric additive functions based on a two-step estimation procedure. By using the selected variables, we first propose a refined two-step spline estimator for the function of interest, which is proved to have a pointwise asymptotic normal distribution and oracle efficiency. We then establish the bounds for the SCBs based on the absolute maxima distribution of a Gaussian process and on the strong approximation lemma [Csörgő and Révész (1981)]. Some other related works on SCBs for nonparametric functions include Claeskens and Van Keilegom (2003), Hall and Titterington (1988), Härdle and Marron (1991), among others. We provide an asymptotic formula for the standard deviation of the spline estimator for the coefficient function, which involves unknown population parameters to be estimated. The formula has somewhat complex expressions and contains many parameters. Direct estimation therefore may be not accurate, particularly with the small or moderate sample sizes. As an alternative, the bootstrap method provides us a reliable way to calculate the standard deviation by avoiding estimating those population parameters. We here apply the smoothed bootstrap method suggested by Efron (2014), which advocated that the method can improve coverage probability to calculate the pointwise estimated standard deviations for the estimators of the coefficient functions. This method was originally proposed for calculating the estimated standard deviation of the estimate of a parameter of interest, such as the conditional mean. We extend this method to the case of functional estimation. We demonstrate by simulation studies in Section 4 that compared to the traditional resampling bootstrap method, the smoothed bootstrap method can successfully improve the empirical coverage rate.

The paper is organized as follows. Section 2 introduces the B-spline estimation procedure for the nonparametric functions, describes the adaptive group Lasso estimators and the initial Lasso estimators and presents asymptotic results. Section 3 describes the two-step spline estimators and introduces the simultaneous confidence bands and the bootstrap methods for calculating the estimated standard deviation. Section 4 describes simulation studies, and Section 5 illustrates the method through the analysis of an obesity data set from a genome-wide association study. Proofs are in the Appendix and additional supplementary material [Ma et al. (2015)].

2. Penalization based variable selection

Let (Y_i, ${\mathbf{X}}_i^{\mathrm{T}},{\mathbf{T}}_i^{\mathrm{T}}$), i = 1,…, n, be random vectors that are independently and identically distributed as (Y, X^T, T^T), where X_i = (X_i1, …, X_id)^T and T_i = (T_i1, …, T_ip)^T. Write the negative quasi-likelihood function $Q(\mu ,y)={\int }_{\mu }^y\{(y-\zeta )/V(\zeta )\}d\zeta$. Estimation of the mean function can be achieved by minimizing the negative quasi-likelihood of the observed data

$$\sum _{i=1}^{n}Q\{g^{-1}\{\eta ({\mathbf{X}}_i,{\mathbf{T}}_i)\},Y_i\}.$$ (5)

2.1. Spline approximation

We approximate the smooth functions α_ℓk(·), 1 ≤ k ≤ d and 1 ≤ ℓ ≤ p in (4) by B-splines. As in most work on nonparametric smoothing, estimation of the functions α_ℓk(·) is conducted on compact sets. Without loss of generality, let the compact set be χ = [0, 1]. Let $G_n^0$ be the space of polynomial splines of order q ≥ 2. We introduce a sequence of spline knots

$$t_{-q-1}=\cdots =t_{-1}=t_0=0<t_1<\cdots <t_N<1=t_{N+1}=\cdots =t_{N+q},$$

where N ≡ N_n is the number of interior knots. In the following, let J_n = N_n + q. For 0 ≤ j ≤ N, let H_j = t_{j + 1} − t_j be the distance between neighboring knots and let H = max_0≤s≤N H_j. Following Zhou, Shen and Wolfe (1998), to study asymptotic properties of the spline estimators for α_ℓk(·), we assume that max_{0≤j≤N−1} |H_{j + 1} − H_j| = o(N⁻¹) and H/min_0≤j≤N H_j ≤ M, where M > 0 is a predetermined constant. Such an assumption is necessary for numerical implementation. In practice, we can use the quantiles as the locations of the knots. Let {b_j,k(x_k) : 1 ≤ j ≤ J_n}^T be the qth order B spline basis functions given on page 87 of de Boor (2001). For positive numbers a_n and b_n, a_n ≍ b_n means that lim_n→∞ a_n/b_n = c, where c is some nonzero finite constant. For 1 ≤ j ≤ J_n, we adopt the centered B-spline functions given in Xue and Yang (2006) such that B_j,k(x_k) = √N[b_j,k(x_k) − {E(b_j,k)/E(b_1,k)}b_1,k(x_k)], so that E{B_j,k(X_k)} = 0 and var{B_j,k(X_k)} ≍ 1. Define the space G_n of additive spline functions as the linear space spanned by B(x) = {1, B_j,k(x_k), 1 ≤ j ≤ J_n, 1 ≤ k ≤ d}^T , where x = (x₁,…, x_d)^T. According to the result on page 149 of de Boor (2001), for α_ℓk(·) satisfying condition (C3) in Appendix A.2 such that ${\alpha }_{\ell k}^{(r-1)}(x_k)\in C^{0,1}[0,1]$ for given integer r ≥ 1, where C^{0, 1} [0, 1] is the space of Lipschitz continuous functions on [0, 1] defined in Appendix A.2, there is a function

$${\alpha }_{\ell k}^0(x_k)=\sum _{j=1}^{J_n}{\gamma }_{j,\ell k}{B}_j(x_k)\in G_n^0,$$ (6)

such that ${\text{sup}}_{x_k\in [0,1]}|{\alpha }_{\ell k}^0(x_k)-{\alpha }_{\ell k}(x_k)|=O(J_n^{-r})$. Then for every 1 ≤ ℓ ≤ p, α_ℓ(x) can be approximated well by a linear combination of spline functions in $G_n^0$, so that

$${\alpha }_{\ell }(\mathbf{X})\approx {\alpha }_{\ell }^0(\mathbf{X})={\gamma }_{\ell 0}+\sum _{k=1}^d\sum _{j=1}^{J_n}{\gamma }_{j,\ell k}{B}_{j,k}(x_k)=B{(\mathbf{X})}^{\mathrm{T}}{\mathit{\gamma }}_{\ell },$$ (7)

where ${\mathit{\gamma }}_{\ell }={({\gamma }_{\ell 0},{\mathit{\gamma }}_{\ell 1}^{\mathrm{T}},\mathrm{\dots },{\mathit{\gamma }}_{\ell d}^{\mathrm{T}})}^{\mathrm{T}}$, in which γ_ℓk = (γ_j,ℓk : 1 ≤ j ≤ J_n)^T. Thus the minimization problem in (5) is equivalent to finding ${\stackrel{\sim }{\mathit{\gamma }}}^0={({\stackrel{\sim }{\mathit{\gamma }}}_{\ell }^{0\mathrm{T}},1\le \ell \le p)}^{\mathrm{T}}$ with ${\stackrel{\sim }{\mathit{\gamma }}}_{\ell }^0={({\stackrel{\sim }{\gamma }}_{\ell 0}^0,{\stackrel{\sim }{\mathit{\gamma }}}_{\ell 1}^{0\mathrm{T}},\dots ,{\stackrel{\sim }{\mathit{\gamma }}}_{\ell d}^{0\mathrm{T}})}^{\mathrm{T}}$ and ${\stackrel{\sim }{\mathit{\gamma }}}_{\ell k}^0={({\stackrel{\sim }{\gamma }}_{j,\ell k}^0:1\le j\le J_n)}^{\mathrm{T}}$ to minimize ${\sum }_{i=1}^{n}Q[g^{-1}\{{\sum }_{\ell =1}^{p}B{({\mathbf{X}}_i)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell }{T}_{\ell }\},Y_i]$. The components of the additive coefficients are estimated by ${\stackrel{\sim }{\alpha }}_{\ell k}^0(x_k)={\sum }_{j=1}^{J_n}{\stackrel{\sim }{\gamma }}_{j,\ell k}^0{B}_j(x_k)=B{(\mathbf{X})}^{\mathrm{T}}{\stackrel{\sim }{\mathit{\gamma }}}_{\ell k}^0$ and ${\stackrel{\sim }{\alpha }}_{\ell 0}^0={\stackrel{\sim }{\gamma }}_{\ell 0}^0$.

2.2. Adaptive group Lasso estimator

We now describe the procedure for estimating and selecting the additive coefficient functions by using the adaptive group Lasso. The estimators are obtained by minimizing a penalized negative quasi-likelihood criterion. We establish asymptotic selection consistency as well as the convergence rate of the estimators to the true nonzero functions. For any vector a = (a₁, …, a_s)^T, let its L₂ norm be ${\Vert \mathbf{a}\Vert }_2=\sqrt{a_1^2+\cdots +a_s^2}$. For any measurable L₂-integrable function ϕ on [0, 1]^d, define the L₂ norm as ‖ϕ‖² = E{ϕ²(X)}.

We are interested in identifying the significant components of the vector T = (T₁, …, T_p)^T. Let s, a fixed number, be the total number of nonzero α_ℓ's and I₁ = {ℓ: ‖α_ℓ‖ ≠ 0, 1 ≤ ℓ ≤ p}. Let I₂ be the complementary set of I₁; that is, I₂ = {ℓ: α_ℓ(·) ≡ 0, 1 ≤ ℓ ≤ p}. Recalling the approximation given in (7), γ_ℓ is zero if and only if each element of γ_ℓ is zero; that is, ‖γ_ℓ‖₂ = 0. We apply the adaptive group Lasso approach in Huang, Horowitz and Wei (2010) for variable selection in model (4). In order to identify zero additive coefficients, we penalize the L₂ norm of the coefficients γ_ℓ for 1 ≤ ℓ ≤ p. Let w_n = (w_n1, …, w_np)^T be a given vector of weights, which needs to be chosen appropriately to achieve selection consistency. Their choice will be discussed in Section 2.3. We consider the penalized negative quasi-likelihood

$$L_n(\mathit{\gamma })=\sum _{i=1}^{n}Q\left[g^{-1}\right\{\sum _{\ell =1}^p{B}^{\mathrm{T}}({\mathbf{X}}_i){\mathit{\gamma }}_{\ell }{T}_{\ell }\},Y_i]+n{\lambda }_n\sum _{\ell =1}^p{w}_{n\ell }{\Vert {\mathit{\gamma }}_{\ell }\Vert }_2,$$ (8)

where λ_n is a regularization parameter controlling the amount of shrinkage. The estimator $\widehat{\mathit{\gamma }}={({\widehat{\mathit{\gamma }}}_1^{\mathrm{T}},\mathrm{\dots },{\widehat{\mathit{\gamma }}}_p^{\mathrm{T}})}^{\mathrm{T}}$ is obtained by minimizing (8). Minimization of (8) is solved by local quadratic approximation as adopted by Fan and Li (2001).

For ℓ = 1, …, p, the ℓth additive coefficient function is estimated by

$${\widehat{\alpha }}_{\ell }(\mathbf{X})={\widehat{\gamma }}_{\ell 0}+\sum _{k=1}^d\sum _{j=1}^{J_n}{\widehat{\gamma }}_{j,\ell k}{B}_{j,k}(x_k)=B^{\mathrm{T}}(\mathbf{X}){\widehat{\mathit{\gamma }}}_{\ell }.$$

We will make the following two assumptions on the order requirements of the tuning parameters. Write w_{n, I1} = (w_nℓ : ℓ ∈ I₁).

Assumption 1. $J_n^2{\{nlog(n)\}}^{-1}\to 0$ and λ_n ‖w_{n, I1}‖₂ → 0, as n → ∞.

Assumption 2. $n{\lambda }_n{\Vert w_{n,I_1}\Vert }_2+n^{1/2}{J}_n^{1/2}\sqrt{log(p{J}_n)}+n{J}_n^{-r}=o(n{\lambda }_n{w}_{n\ell })$, for all ℓ ∈ I₂.

The following theorem presents the selection consistency and estimation properties of the adaptive group Lasso estimators.

Theorem 1. Under conditions (C1)–(C5) in the Appendix and Assumptions 1 and 2: (i) as n → ∞, P(‖α̂_ℓ‖ > 0, ℓ ∈ I₁ and ‖α̂_ℓ‖ = 0, ℓ ∈ I₂) → 1, and (ii) $\Vert {\widehat{\alpha }}_{\ell }-{\alpha }_{\ell }\Vert =O_p({\lambda }_n{\Vert w_{n,I_1}\Vert }_2+n^{-1/2}{J}_n^{1/2}+J_n^{-r}),\ell \in I_1$.

2.3. Choice of the weights

We now discuss how to choose the weights used in (8) based on the initial estimates. For low-dimensional data settings with p < n, an unpenalized estimator such as least squares estimator [Zou (2006)] can be used as an initial estimate. For high-dimensional settings with p ≫ n, it has been discussed [Meier and Bühlmann (2007)] that the Lasso estimator is a more appropriate choice. Following Huang, Horowitz and Wei (2010), we obtain an initial estimate with the group Lasso by minimizing

$$L_{n1}(\mathit{\gamma })=\sum _{i=1}^{n}Q\left[g^{-1}\right\{\sum _{\ell =1}^{p}B{({\mathbf{X}}_i)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell }{T}_{\ell }\},Y_i]+n{\lambda }_{n1}\sum _{\ell =1}^p{\Vert {\mathit{\gamma }}_{\ell }\Vert }_2,$$

with respect to $\mathit{\gamma }={({\mathit{\gamma }}_1^{\mathrm{T}},\mathrm{\dots },{\mathit{\gamma }}_p^{\mathrm{T}})}^{\mathrm{T}}$. Denote the resulting estimators by $\stackrel{\sim }{\mathit{\gamma }}={({\stackrel{\sim }{\mathit{\gamma }}}_1^{\mathrm{T}},\mathrm{\dots },{\stackrel{\sim }{\mathit{\gamma }}}_p^{\mathrm{T}})}^{\mathrm{T}}$. Let Ĩ₁ = {ℓ : ‖γ̃_ℓ‖₂ ≠ 0, 1 ≤ ℓ ≤ p}, and let s̃ be the number of elements in Ĩ₁.

Under conditions (C1)–(C5) in the Appendix, and when ${\lambda }_{n1}\ge C{n}^{-1/2}{J}_n^{1/2}\times \sqrt{log(p{J}_n)}$ for a sufficiently large constant C, we have: (i) the number of estimated nonzero functions are bounded; that is, as n → ∞, there exists a constant 1 < C₁ < ∞ such that P(s̃ ≤ C₁s) → 1; (ii) if λ_n1 → 0, then P(‖γ̃_ℓ‖₂ > 0 for all l ∈ I₁) → 1; (iii) ${\Vert \stackrel{\sim }{\mathit{\gamma }}-\mathit{\gamma }\Vert }_2=O_p({\lambda }_{n1}+n^{-1/2}{J}_n^{1/2}+J_n^{-r})$. We refer to Theorems 1 (i) and (ii) of Huang, Horowitz and Wei (2010) for the proofs of (i) and (ii), and Theorem 1 in our paper for the proof of (iii).

The weights we use are $w_{n\ell }={\Vert {\stackrel{\sim }{\mathit{\gamma }}}_{\ell }\Vert }_2^{-1}$, if ‖γ̃_ℓ‖₂ > 0; w_nℓ = ∞, if ‖γ̃_ℓ‖₂ = 0.

Remark 1. Assumptions 1 and 2 give the order requirements of J_n and λ_n. Based on the condition that $J_n^2{\{nlog(n)\}}^{-1}\to 0$ given in Assumption 1, we need J_n ≪ {n log(n)}^1/2, where a_n ≪ b_n denotes that a_n/b_n = o(1) for any positive numbers a_n and b_n, and λ_n needs to satisfy $n^{-1/2}{J}_n^{1/2}\sqrt{log(p{J}_n)}\times {\{{min}_{\ell \in I_2}(w_{n\ell })\}}^{-1}\ll {\lambda }_n\ll 1$. From the above theoretical properties of the group Lasso estimators, we know that, with probability approaching 1, ‖γ̃_ℓ‖₂ > 0 for nonzero components, and then the corresponding weights w_nℓ are bounded away from 0 and infinity for ℓ ∈ I₁. By defining 0 · ∞ = 0, the components not selected by the group Lasso are not included in the adaptive group Lasso procedure. Let J_n ≍ n^{1/(2r + 1)}, so that J_n has the optimal order for spline regression. If p = exp[o{n^{2r/(2r + 1)}}], then $n^{-1/2}{J}_n^{1/2}\sqrt{log(p{J}_n)}\to 0$. This means the dimension p can diverge with the sample size at an almost exponential rate.

2.4. Selection of tuning parameters

Tuning parameter selection always plays an important role in model and variable selection. An underfitted model can lead to severely biased estimation, and an overfitted model can seriously degrade the estimation efficiency. Among different data-driven methods, the Bayesian information criterion (BIC) tuning parameter selector has been shown to be able to identify the true model consistently in the fixed dimensional setting [Wang, Li and Tsai (2007)]. In the high-dimensional setting, an extend BIC (EBIC) and a generalized information criterion have been proposed by Chen and Chen (2008) and Fan and Tang (2013), respectively. In this paper, we adopt the EBIC method [Chen and Chen (2008)] to select the tuning parameter λ_n in (8). Specifically, the EBIC(λ_n) is defined as

$$2\sum _{i=1}^n\left(Q\right[g^{-1}\left\{\sum _{\ell =1}^{p}B{({\mathbf{X}}_i)}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{\ell }{T}_{i\ell }\right\},Y_i\left]\right)+s^{\ast }(1+d{J}_n)log(n)+2\nu log\left(\begin{array}{c}p\\ s^{\ast }\end{array}\right),$$

where ${({\widehat{\mathit{\gamma }}}_{\ell })}_{\ell =1}^p$ is the minimizer of (8) for a given λ_n, s* is the number of nonzero estimated functions ${({\widehat{\alpha }}_{\ell })}_{\ell =1}^p$ and 0 ≤ ν ≤ 1 is a constant. Here we use ν = 0.5. When ν = 0, the EBIC is ordinary BIC.

We use cubic B-splines for the nonparametric function estimation, so that q = 4. In the penalized estimation procedure, we let the number of interior knots N = ⎿cn^{1/(2q + 1)}⏌ satisfy the optimal order, where ⎿a⏌ denotes the largest integer no greater than a and c is a constant. In the simulations, we take c = 2.

3. Inference and the bootstrap smoothing procedure

3.1. Background

After model selection, our next step is to conduct statistical inference for the coefficient functions of those important variables. We will establish a simultaneous confidence band (SCB) based on a two-step estimator for global inference. An asymptotic formula of the SCB will be provided based on the distribution of the maximum value of the normalized deviation of the spline functional estimate. To improve accuracy, we calculate the estimated standard deviation in the SCB by using the nonparametric bootstrap smoothing method as discussed in Efron (2014). For specificity, we focus on the construction of α_ℓ1(x₁), with α_ℓk(x_k)for k ≥ 2 defined similarly, for ℓ ∈ Î₁, where Î₁ = {ℓ : ‖α̂_ℓ‖ ≠ 0, 1 ≤ ℓ ≤ p}.

Although the one-step penalized estimation in Section 2 can quickly identify nonzero coefficient functions, no asymptotic distribution is available for the resulting estimators. Thus we construct the SCB based on a refined two-step spline estimator for α_ℓ1(x₁), which will be shown to have the oracle property that the estimator of α_ℓ1(x₁) has the same asymptotic distribution as the univariate oracle estimator obtained by pretending that α_ℓ0 and α_ℓk (X_k) for ℓ ∈ Î₁, k ≥ 2 and α_ℓ(X) for ℓ ∉ Î₁ are known. See Horowitz, Klemelä and Mammen (2006), Horowitz and Mammen (2004), Liu, Yang and Härdle (2013) for kernel-based two-step estimators in generalized additive models, which also have the oracle property but are not as computationally efficient as the two-step spline method. We next introduce the oracle estimator and the proposed two-step estimator before we present the SCB.

3.2. Oracle estimator

In the following, we describe the oracle estimator of α_ℓ1(x₁). We rewrite model (4) as

$$\mu (\mathbf{X},\mathbf{T})=g^{-1}\{\eta (\mathbf{X},\mathbf{T})\}=\sum _{\ell \in {\widehat{I}}_1}{\alpha }_{\ell 1}(X_1)T_{\ell }+\sum _{\ell \in {\widehat{I}}_1}\{{\alpha }_{\ell 0}+\sum _{k\ge 2}{\alpha }_{\ell k}(X_k)\}{T}_{\ell }+\sum _{\ell \notin {\widehat{I}}_1}{\alpha }_{\ell }(X)T_{\ell }.$$ (9)

By assuming that α_ℓ0 and α_ℓk(X_k) for ℓ ∈ Î₁, k ≥ 2 and α_ℓ(X) for ℓ ∉ Î₁ are known, estimation in (9) involves only the nonparametric functions α_ℓ1(X₁) of a scalar covariate X₁. It will be shown in Theorem 2 that the estimator achieves the univariate optimal convergence rate when the optimal order for the number of knots is applied. We estimate α₁(x₁) = {α_ℓ1(x₁), ℓ ∈ Î₁}^T by minimizing the negative quasi-likelihood function as follows. Denote the oracle estimator by ${\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)=B_1^{\mathcal{S}}{(x_1)}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{\ell 1}^{\text{OR}}$, where ${\widehat{\mathit{\gamma }}}_{\ell 1}^{\text{OR}}$ is defined directly below, $B_1^{\mathcal{S}}(x_1)=\{B_{j,1}^{\mathcal{S}}(x_1),1\le j\le J_n^{\mathcal{S}}\}$ where $B_{j,1}^{\mathcal{S}}(x_1)$ is the centered B-spline function defined in the same way as B_{j, 1}(x₁) in Section 2, but with $N^{\mathcal{S}}=N_n^{\mathcal{S}}$ interior knots and $J_n^{\mathcal{S}}=N_n^{\mathcal{S}}+q$. Rates of increase for $J_n^{\mathcal{S}}$ are described in Assumptions 3 and 4 below. Let α_{ℓ, −1}(X_i) = α_ℓ0 + Σ_{k ≥ 2} α_ℓk(X_ik). Then ${\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}={\{{({\widehat{\mathit{\gamma }}}_{\ell 1}^{\text{OR}})}^{\mathrm{T}},\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$ is obtained by minimizing the negative quasi-likelihood

$$L_n^{\text{OR}}(\mathit{\gamma },1)=\sum _{i=1}^{n}Q\left[g^{-1}\right\{\sum _{\ell \in {\widehat{I}}_1}{B}_1^{\mathcal{S}}{(X_{i1})}^{\mathrm{T}}{\mathit{\gamma }}_{\ell 1}{T}_{i\ell }+\sum _{\ell \in {\widehat{I}}_1}{\alpha }_{\ell ,-1}({\mathbf{X}}_i)T_{i\ell }+\sum _{\ell \notin {\widehat{I}}_1}{\alpha }_{\ell }({\mathbf{X}}_i)T_{i\ell }\},Y_i],$$ (10)

where γ_,1 = {(γ_ℓ1)^T, ℓ ∈ Î₁}^T. Similarly, the oracle estimator of α₀ = {α_ℓ0, ℓ ∈ Î₁}^T, which is denoted as ${\widehat{\mathit{\alpha }}}_0^{\text{OR}}={\{{\widehat{\alpha }}_{\ell 0}^{\text{OR}},\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}={\{{\widehat{\mathit{\gamma }}}_{\ell 0}^{\text{OR}},\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$, is obtained by minimizing $L_n^{\text{OR}}({\mathit{\gamma }}_{,0})={\sum }_{i=1}^{n}Q[g^{-1}\{{\sum }_{\ell \in {\widehat{I}}_1}{\gamma }_{\ell 0}{T}_{i\ell }+{\sum }_{\ell \in {\widehat{I}}_1}{\alpha }_{\ell ,-0}({\mathbf{X}}_i)T_{i\ell }+{\sum }_{\ell \notin {\widehat{I}}_1}{\alpha }_{\ell }({\mathbf{X}}_i)T_{i\ell }\},Y_i]$, where γ_,0 = (γ_ℓ0, ℓ ∈ Î₁) and ${\alpha }_{\ell ,-0}({\mathbf{X}}_i)={\sum }_{k=1}^d{\alpha }_{\ell k}(X_{ik})$.

3.3. Initial estimator

The oracle estimator is infeasible because it assumes knowledge of the other functions. In order to obtain the two-step estimators of α_ℓ1(x₁) for ℓ ∈ Î₁, we first need initial estimators for α_ℓ0 and α_ℓk(x_k) for k ≥ 2 and ℓ ∈ Î₁ , denoted as ${\widehat{\alpha }}_{\ell 0}^{\text{ini}}={\widehat{\gamma }}_{\ell 0}^{\text{ini}}$ and ${\widehat{\alpha }}_{\ell k}^{\text{ini}}(x_k)=B_k^{\text{ini}}{(x_k)}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{\ell k}^{\text{ini}}$, where $B_k^{\text{ini}}(x_k)={\{B_{j,k}^{\text{ini}}(x_k):1\le j\le J_n^{\text{ini}}\}}^{\mathrm{T}}$ and $B_{j,k}^{\text{ini}}(x_k)$ are B-spline functions with the number of interior knots $N_n^{\text{ini}}$ and $J_n^{\text{ini}}=N_n^{\text{ini}}+q$. Rates of increase for $J_n^{\text{ini}}$ are described in Assumptions 3 and 4 below. We need an undersmoothed procedure in the first step, so that the approximation bias can be reduced, and the difference between the two-step and oracle estimators is asymptotically negligible. We obtain ${\widehat{\mathit{\gamma }}}_{{\widehat{I}}_1}^{\text{ini}}={\{{({\widehat{\mathit{\gamma }}}_{\ell }^{\text{ini}})}^{\mathrm{T}}:\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$, where ${\widehat{\mathit{\gamma }}}_{\ell }^{\text{ini}}={\{{\widehat{\mathit{\gamma }}}_{\ell 0}^{\text{ini}},{({\widehat{\mathit{\gamma }}}_{\ell k}^{\text{ini}})}^{\mathrm{T}}\}}^{\mathrm{T}}$, by minimizing the negative quasi-likelihood ${\sum }_{i=1}^{n}Q[g^{-1}\{{\sum }_{\ell \in {\widehat{I}}_1}B{({\mathbf{X}}_i)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell }{T}_{\ell }\},Y_i]$. The adaptive group Lasso penalized estimator γ̂_Î1 = {(γ̂_ℓ)^T : ℓ ∈ Î₁}^T obtained in Section 2 can also be used as the initial estimator. We, however, refit the model with the selected variables and obtain the initial estimator ${\widehat{\mathit{\gamma }}}_{{\widehat{I}}_1}^{\text{ini}}$ in order to improve estimation accuracy in high-dimensional data settings.

3.4. Final estimator

In the second step, we construct the two-step estimator of α_ℓ1 for ℓ ∈ Î₁. We replace α_ℓ0 and α_ℓk(X_k) by the initial estimators ${\widehat{\alpha }}_{\ell 0}^{\text{ini}}$ and ${\widehat{\alpha }}_{\ell k}^{\text{ini}}(X_k)$ for ℓ ∈ Î₁ and k ≥ 2 and replace α_ℓ(X) for ℓ ∉ Î₁ by α̂_ℓ(X) = 0. Let ${\widehat{\alpha }}_{\ell ,-1}^{\text{ini}}({\mathbf{X}}_i)={\widehat{\alpha }}_{\ell 0}^{\text{ini}}+{\sum }_{k\ge 2}{\widehat{\alpha }}_{\ell k}^{\text{ini}}(X_{ik})$. Denote the two-step spline estimator of α_ℓ1(x₁) as ${\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)=B_1^{\mathcal{S}}{(x_1)}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{\ell 1}^{\mathcal{S}}$ with ${\widehat{\mathit{\gamma }}}_{,1}^{\mathcal{S}}={\{{({\widehat{\mathit{\gamma }}}_{\ell 1}^{\mathcal{S}})}^{\mathrm{T}},\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$ minimizing

$$L_n^{\mathcal{S}}({\mathit{\gamma }}_{,1})=\sum _{i=1}^{n}Q\left[g^{-1}\right\{\sum _{\ell \in {\widehat{I}}_1}{B}_1^{\mathcal{S}}{(X_{i1})}^{\mathrm{T}}{\mathit{\gamma }}_{\ell 1}{T}_{i\ell }+\sum _{\ell \in {\widehat{I}}_1}{\widehat{\alpha }}_{\ell ,-1}^{\text{ini}}({\mathbf{X}}_i)T_{i\ell }+\sum _{\ell \notin {\widehat{I}}_1}{\widehat{\alpha }}_{\ell }({\mathbf{X}}_i)T_{i\ell }\},Y_i].$$ (11)

Then the two-step of α_ℓ0, denoted as ${\widehat{\alpha }}_{\ell 0}^{\mathcal{S}}={\widehat{\gamma }}_{\ell 0}^{\mathcal{S}}$, is obtained in the same way as ${\widehat{\alpha }}_{\ell 0}^{\text{OR}}$ by replacing α_ℓ,0(X_i) with ${\widehat{\alpha }}_{\ell ,0}^{\text{ini}}({\mathbf{X}}_i)={\sum }_{k=1}^d{\widehat{\alpha }}_{\ell k}^{\text{ini}}(X_{ik})$ for ℓ ∈ Î₁ and replacing α_ℓ(X_i) with α̂_ℓ(X_i) = 0 for ℓ ∉ Î₁. Let ${\widehat{\mathit{\alpha }}}_0^{\mathcal{S}}={\{{\widehat{\alpha }}_{\ell 0}^{\mathcal{S}},\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$.

3.5. Asymptotic normality and uniform oracle efficiency

We now establish the asymptotic normality and uniform oracle efficiency for the oracle and final estimators. Let $Z_{ij\ell ,1}=B_{j,1}^{\mathcal{S}}(X_{i1})T_{i\ell }$ and $Z_{i,1}={(Z_{ij\ell ,1},1\le j\le J_n^{\mathcal{S}},\ell \in {\widehat{I}}_1)}^{\mathrm{T}}$. Let s* be the number of elements in Î₁. By Theorem 1, P(s* = s) → 1. For simplicity of notation, denote ${\sigma }_i^2={\sigma }^2({\mathbf{X}}_i,{\mathbf{T}}_i)$ and η_i = η(X_i, T_i). Define $s^{\ast }\times s^{\ast }{J}_n^{\mathcal{S}}$ matrix B(x₁) as

$$\left[\begin{array}{ccccccccc}{B}_{1,1}^{\mathcal{S}}(x_1)& \cdots & B_{J_n^{\mathcal{S}},1}^{\mathcal{S}}(x_1)& 0& \cdots & 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 0& \cdots & 0& 0& \cdots & 0& B_{1,1}^{\mathcal{S}}(x_1)& \cdots & B_{J_n^{\mathcal{S}},1}^{\mathcal{S}}(x_1)\end{array}\right].$$

To establish the asymptotic distribution of the two-step estimator, in addition to Assumptions 1 and 2 given in Section 2, we make the following two assumptions on the number of basis functions $J_n^{\mathcal{S}}$ and $J_n^{\text{ini}}$:

Assumption 3. (i) $s^{\ast }{(J_n^{\mathcal{S}})}^2{\{nlog(n)\}}^{-1}=o(1)$ and $s^{\ast }{(J_n^{\mathcal{S}})}^{-r}=o(1)$, and (ii) $n{(logn)}^{-1}{(J_n^{\mathcal{S}}{J}_n^{\text{ini}})}^{-1}\to \infty$, as n → ∞.

Assumption 4. ${(n/J_n^{\mathcal{S}})}^{1/2}{(J_n^{\text{ini}})}^{-r}\to 0$, as n → ∞.

First we describe the asymptotic normality of the oracle estimator ${\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)$ of α_ℓ1(x₁). Let ${\widehat{\alpha }}_1^{\text{OR}}(x_1)={\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1),\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$. Let $b_1(x_1)=E\{{\widehat{\alpha }}_1^{\text{OR}}(x_1)|\mathbb{X},\mathbb{T}\}$ and $b_{\ell 1}(x_1)=E\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)|\mathbb{X},\mathbb{T}\}$, for ℓ ∈ Î₁, where $(\mathbb{X},\mathbb{T})={({\mathbf{X}}_i,{\mathbf{T}}_i)}_{i=1}^n$.

Theorem 2. Under conditions (C1)–(C5) and Assumption 3(i), for any vector a ∈ R^s* with ‖a‖₂ = 1, for any x₁ ∈ [0, 1], ${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\text{OR}}(x_1)-b_1(x_1)\}\to N(0,1)$, where

$${\sigma }_n^2(x_1)=B^{\mathcal{S}}(x_1){[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}{B}^{\mathcal{S}}{(x_1)}^{\mathrm{T}},$$ (12)

where g˙⁻¹ (η_i) is the first-order derivative of g⁻¹(η_i) with respect to η_i, and

$$\sum _{\ell \in {\widehat{I}}_1}{\Vert {\widehat{\alpha }}_{\ell 1}^{\text{OR}}-b_{\ell 1}\Vert }^2=O_p(s^{\ast }{J}_n^{\mathcal{S}}{n}^{-1}),\sum _{\ell \in {\widehat{I}}_1}{\Vert b_{\ell 1}-{\alpha }_{\ell 1}\Vert }^2=O_p\{{(s^{\ast })}^2{(J_n^{\mathcal{S}})}^{-2r}\}.$$

Thus for ℓ ∈ Î₁, ${\sigma }_{n1}^{-1}(x_1)\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-b_{\ell 1}(x_1)\}\to N(0,1)$, where

$${\sigma }_{n1}^2(x_1)={\mathbf{e}}_{\ell }^{\mathrm{T}}{\sigma }_n^2(x_1){\mathbf{e}}_{\ell },$$ (13)

and e_ℓ is the s*-dimensional vector with the ℓth element 1 and other elements 0, and ${\Vert {\widehat{\mathit{\alpha }}}_0^{\text{OR}}-{\mathit{\alpha }}_0\Vert }_2=O_p(\sqrt{s^{\ast }/n})$.

The next result shows the uniform oracle efficiency of the two-step estimator that the difference between the two-step estimator ${\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)$ and oracle estimator ${\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)$ is uniformly asymptotically negligible, and thus the two-step estimator is oracle in the sense that it has the same asymptotic distribution as the oracle estimator. Let ${\widehat{\alpha }}_1^{\mathcal{S}}(x_1)={\{{\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1),\ell \in {\widehat{I}}_1\}}^{\mathrm{T}}$.

Theorem 3. Under conditions (C1)–(C5) in the Appendix and Assumptions 1–3,

$$\underset{x_1\in [0,1]}{sup}{\Vert {\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_1^{\text{OR}}(x_1)\Vert }_{\infty }=O_p\{{(n^{-1}logn)}^{1/2}+{(J_n^{\text{ini}})}^{-r}\},$$

${\Vert {\widehat{\mathit{\alpha }}}_0^{\mathcal{S}}-{\widehat{\mathit{\alpha }}}_0^{\text{OR}}\Vert }_2=o_p(n^{-1/2})$, and furthermore under Assumption 4,

$$\underset{x_1\in [0,1]}{sup}|{\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_1^{\text{OR}}(x_1)\}|=o_p(1),$$

for any vector a ∈ R^s* with ‖a‖₂ = 1 and ${\sigma }_n^2(x_1)$ given in (12). Hence, for any x₁ ∈ [0, 1], ${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-b_1(x_1)\}\to N(0,1)$.

Remark 2. Under Assumptions 1 and 2, by Theorem 1, with probability approaching 1, s* = s, which is a fixed number. In the second step, by letting $J_n^{\mathcal{S}}\asymp n^{1/(2r+1)}$, the nonparametric functions α_ℓ1 for ℓ ∈ Î₁ are approximated by spline functions with the optimal number of knots. By the conditions that $(n/J_n^{\mathcal{S}}){(J_n^{\text{ini}})}^{-1}\to 0$ and $n{(logn)}^{-1}{(J_n^{\mathcal{S}}{J}_n^{\text{ini}})}^{-1}\to \infty$ given in Assumptions 3 and 4, $J_n^{\text{ini}}$ needs to satisfy $n^{1/(2r+1)}\ll J_n^{\text{ini}}\ll n^{2r/(2r+1)}{(logn)}^{-1}$ where r ≥ 1. By using the adaptive group lasso estimator as the initial estimator, Assumption 1 requires that $J_n^{\text{ini}}\ll {\{nlog(n)\}}^{1/2}$. Hence $n^{1/(2r+1)}\ll J_n^{\text{ini}}\ll {\{nlog(n)\}}^{1/2}$. We therefore can let $J_n^{\text{ini}}\asymp n^{(1+\vartheta )/(2r+1)}$, where ϑ is any small positive number close to 0. This increase in the number of basis functions ensures undersmoothing in the first step in order that the uniform difference between the two-step and the oracle estimators become asymptotically negligible. Based on Assumptions 1 and 2, the tuning parameter λ_n needs to satisfy $n^{-1/2}{(J_n^{\text{ini}})}^{1/2}\sqrt{log(p{J}_n^{\text{ini}})}{\{{min}_{\ell \in I_2}(w_{n\ell })\}}^{-1}\ll {\lambda }_n\ll 1$.

Remark 3. The number of interior knots has the same order requirement as the number of basis functions. In the first step, with the undersmoothing requirement as discussed in Remark 2, we let the number of interior knots Nⁱⁿⁱ = ⎿cn^{(1+0.01)/(2q+1)}⏌, where c is a constant, by assuming that r = q. In the simulations, we let c = 2. In the second-step estimation, we use BIC to select the number of knots N^S, so the optimal N^S ranges in [⎿n^1/(2q+1)⏌, ⎿2n^1/(2q+1)⏌] by minimizing BIC: $\text{BIC}(N^S)=2L_n^{\mathcal{S}}({\widehat{\mathit{\gamma }}}_{,1}^{\mathcal{S}})+d(N^S+q)log(n)$.

3.6. Simultaneous confidence bands

In this section, we propose a SCB for α_ℓ1(x₁) by studying the asymptotic behavior of the maximum of the normalized deviation of the spline functional estimate. To construct asymptotic SCBs for α_ℓ1(x₁) over the interval x₁ ∈ [0, 1] with confidence level 100(1 − α)%, α ∈ (0, 1), we need to find two functions l_ℓn(x₁) and u_ℓn(x₁) such that

$$\underset{n\to \infty }{lim}P(l_{\ell n}(x_1)\le {\alpha }_{\ell 1}(x_1)\le u_{\ell n}(x_1)\text{for all}{x}_1\in [0,1])=1-\alpha .$$ (14)

In practice, we consider a variant of (14) and construct SCBs over a subset S_n,1 of [0, 1] with S_n,₁ becoming denser as n → ∞. We, therefore, partition [0, 1] according to L_n equally spaced intervals based on 0 < ξ₀ < ξ₁ < ··· < ξ_Ln < ξ_Ln+1 = 1 where L_n → ∞ as n → ∞. Let S_n,1 = (ξ₀,…,ξ_Ln). Define d_Ln(α) = 1 − {2 log(L_n + 1)}⁻¹[log{−(1/2)log(1 − α)} + (1/2) {log log(L_n + 1) + log(4π)}], and Q_Ln(α) = {2 log(L_n + 1)}^1/2d_Ln(α).

Theorem 4. Under conditions (C1)–(C5) in the Appendix, and $L_n\asymp J_n^{\mathcal{S}}\asymp n^{1/(2r+1)}$ and $n^{1/(2r+1)}\ll J_n^{\text{ini}}\ll n^{2r/(2r+1)}{\{log(n)\}}^{-1}$, we have

$$\underset{n\to \infty }{lim}P\{\underset{x_1\in S_{n,1}}{\text{sup}}|{\sigma }_{n1}^{-1}(x_1)\{{\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)-{\alpha }_{\ell 1}(x_1)\}|\le Q_{L_n}(\alpha )\}=1-\alpha ,$$

and thus an asymptotic 100(1 − α)% confidence band for α_ℓ1(x₁) over x₁ ∈ S_n,1 is

$${\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)\pm {\sigma }_{n1}(x_1)Q_{L_n}(\alpha ).$$ (15)

Remark 4. Compared to the pointwise confidence intervals with width 2Z_{1 − α/2}σ_n(x₁), the width of the confidence bands (15) is inflated by a rate {2 log(L_n +1)}^1/2d_Ln (α) / Z_1−α/2, where Z_1−α/2 is the cut-off point of the 100(1 − α)th percentile of the standard normal.

3.7. Bootstrap smoothing for calculating the standard error

Theorem 4 establishes a thresholding value Q_Ln(α) for the SCB. One critical question is how to estimate the standard deviation σ_n1(x₁) in order to construct the SCB. We can use a sample estimate of σ_n1(x₁) according to the asymptotic formula given in (12),which may have approximation error and thus lead to inaccurate results for inference. The bootstrap estimate of the standard deviation provides an alternative way. We here propose a bootstrap smoothed confidence band by adopting the nonparametric bootstrap smoothing idea from Efron (2014), which can eliminates discontinuities in jumpy estimates. The procedure is described as follows.

Let D = {D₁,…, D_n} be the data we have, where D_i = {Y_i, X_i, (T_i,ℓ, ℓ ∈ Î₁)}. Denote ${\mathbf{D}}^{\ast }=\{{\mathbf{D}}_1^{\ast },\dots ,{\mathbf{D}}_n^{\ast }\}$ as a nonparametric bootstrap sample from {D₁,…, D_n}, and ${\mathbf{D}}_{(j)}^{\ast }=\{{\mathbf{D}}_{(j)1}^{\ast },\dots ,{\mathbf{D}}_{(j)n}^{\ast }\}$ as the jth bootstrap sample in B draws. Let ${\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)$ be the two-step estimator of α_ℓ1(x₁) by using the data ${\mathbf{D}}_{(j)}^{\ast }$. We first present an empirical standard deviation by the traditional resampling method which is given as

$${\widehat{\sigma }}_{\ell 1,B}(x_1)={[\sum _{j=1}^B{\{{\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)-{\widehat{\alpha }}_{\ell 1,}^{\ast \mathcal{S}}.(x_1)\}}^2/(B-1)]}^{1/2},$$ (16)

where ${\widehat{\alpha }}_{\ell 1,}^{\ast \mathcal{S}}.(x_1)={\sum }_{j=1}^B{\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)/B$. Then a 100(1 − α)% unsmoothed bootstrap SCB for α_ℓ1(x₁) over x₁ ∈ S_n,1 is given as

$${\widehat{\alpha }}_{\ell 1}^{\ast \mathcal{S}}(x_1)\pm {\widehat{\sigma }}_{\ell 1,B}(x_1)Q_{L_n}(\alpha ).$$ (17)

Another choice is the smoothed bootstrap SCB which eliminates discontinuities in the estimates [Efron (2014)]. Let

$${\stackrel{\sim }{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)=\sum _{j=1}^B{\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)/B$$

be the smoothed estimate of α_ℓ1(x₁) obtained by averaging over the bootstrap replications. Let $C_{(j)i}^{\ast }=\#\{{\mathbf{D}}_{(j)i\prime }^{\ast }={\mathbf{D}}_i\}$ be the number of elements in ${\mathbf{D}}_{(j)i\prime }^{\ast }$ equaling D_i.

Proposition 1. At each point x₁ ∈ S_n,1, the nonparametric delta-method estimate of the standard deviation for the smoothed bootstrap statistic ${\stackrel{\sim }{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)$ is ${\stackrel{\sim }{\sigma }}_{\ell 1}(x_1)={\{{\sum }_{i=1}^n{cov}_i^2(x_1)\}}^{1/2}$, where ${cov}_i(x_1)={cov}_{\ast }\{C_{(j)i}^{\ast },{\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)\}$ which is the bootstrap covariance between $C_{(j)i}^{\ast }$ and ${\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)$.

The proof of Proposition 1 essentially follows the same arguments as the proof for Theorem 1 in Efron (2014). Based on Proposition 1, to construct the smoothed bootstrap SCB, we use the nonparametric estimate of the standard deviation given as

$${\stackrel{\sim }{\sigma }}_{\ell 1,B}(x_1)={\{\sum _{i=1}^n{\widehat{cov}}_{i,B}^2(x_1)\}}^{1/2},$$ (18)

Where

$${\widehat{cov}}_{\ell i,B}(x_1)=\sum _{j=1}^B(C_{(j)i}^{\ast }-C_{\cdot i}^{\ast })({\widehat{\alpha }}_{\ell 1,(j)}^{\ast \mathcal{S}}(x_1)-{\widehat{\alpha }}_{\ell 1,}^{\ast \mathcal{S}}.(x_1))/B$$

with $C_{\cdot i}^{\ast }={\sum }_{j=1}^B{C}_{(j)i}^{\ast }/B$. The 100(1 − α)% smoothed bootstrap SCB for α_ℓ1(x₁) over x₁ ∈ S_n,1 is given as

$${\stackrel{\sim }{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)\pm {\stackrel{\sim }{\sigma }}_{\ell 1,B}(x_1)Q_{L_n}(\alpha ).$$ (19)

4. A simulation study

In this section, we present a simulation study to evaluate the finite sample performance of our proposed penalized estimation procedure and the simultaneous confidence bands. More numerical studies are located in the supplementary materials [Ma et al. (2015)].

Example 1. In this example, we use 1286 SNPs located on the sixth chromosome from the Framingham Heart Study to simulate the binary response from the logistic model

$$logit\{P(Y_i=1|{\mathbf{X}}_i,{\mathbf{T}}_i)\}=\sum _{\ell =1}^p{\alpha }_{\ell }({\mathbf{X}}_i)T_{i\ell }=\sum _{\ell =1}^p\{{\alpha }_{\ell 0}+\sum _{k=1}^2{\alpha }_{\ell k}(X_{ik})\}{T}_{i\ell },$$ (20)

with the four SNPs ss66063578, ss66236230, ss66194604 and ss66533844 selected from the real data analysis in Section 5 as important covariates and the other SNPs as unimportant covariates, so that s = 4 (the number of important covariates), p = 1286 and the sample size n = 300. The three possible allele combinations are coded as 1, 0 and −1 for each SNP The covariates X_ik, k = 1, 2, are simulated environmental effects, which are generated from independent uniform distributions on [0, 1]. We generate the coefficient functions as α₁₀ = 0.5, α₁₁(x₁) = 4cos(2πx₁), α₁₂(x₂) = 5{(2x₂ − 1)² − 1/3}, α₂₀ = 0.5, α₂₁(x₁) = 6x₁ − 3, α₂₂(x₂) = 4{sin(2πx₂) + cos(2πx₂)}, α₃₀ = 0.5, α₃₁(x₁) = 4sin(2πx₁), α₃₂(x₂) = 6x₂ − 3, α₄₀ = 0.5, α₄₁(x₁) = 4cos(2πx₁), α₄₂(x₂) = 5{(2x₂ − 1)² − 1/3} and α_ℓ(X_i) = 0 for l = 5,…, 1286. We conducted 500 replications for each simulation. We fit the data with the GACM (20) by using the adaptive group lasso (AGL) and group lasso (GL). In the literature, the generalized varying coefficient model [GVCM; Lian (2012)], which considers one index variable in the coefficient function for each predictor T_iℓ, has been widely used to study nonlinear interactions. To apply the GVCM method [Lian (2012)] in this setting, we first perform principal component analysis (PCA) on X_i and then use the first principal component as the index variable in the GVCM. Then we apply the AGL and GL methods to the GVCM: $logit\{P(Y_i=1|{\mathbf{X}}_i,{\mathbf{T}}_i)\}={\sum }_{\ell =1}^p{\alpha }_{\ell }(U_i)T_{i\ell }$, where U_i is the first principal component obtained by PCA on X_i. Moreover, we also fit the data with the parametric logistic regression by assuming linear coefficient functions (3) with the AGL method. We also compare our proposed method with the conventional screening method by parametric logistic regression for Genome-Wide Association Studies [GWAS; Murcray, Lewinger and Gauderman (2009)]. In the screening method, we fit a logistic model for each SNP: $logit\{P(Y_i=1|{\mathbf{X}}_i,T_{i\ell })\}={\alpha }_0+{\mathit{\alpha }}^{\mathrm{T}}{\mathbf{X}}_i+{\beta }_{\ell }{T}_{i\ell }+{\sum }_{k=1}^2{\beta }_{\ell k}{X}_{ik}{T}_{i\ell }$, for ℓ = 1,…, 1286. Then we conduct a likelihood ratio test for the genetic and interaction effects of H₀ : β_ℓ = β_ℓ1 = β_ℓ2 = β_ℓ3 = 0. Let α₀ = 0.05 be the overall type I error for the study and M = 1286 be the number of SNPs in this study. We apply the multiple testing correction procedure for GWAS with H₀ rejected when the p-value < α₀/M_eff, where M_eff is the Cheverud–Nyholt estimate of the effective number of tests [Cheverud (2001), Nyholt (2004)] calculated by $M_{\text{eff}}=1+M^{-1}{\sum }_{j=1}^M{\sum }_{k=1}^M(1-r_{jk}^2)$ and r_jk are the correlation coefficients of the SNPs, and we obtain M_eff = 1275.65.

Table 1 presents the percentages of correct-fitting (C) (exactly the important covariates are selected), over-fitting (O) (both the important covariates and some unimportant covariates are selected) and incorrect-fitting (I) (some of the important covariates are not selected), the average true positives (TP), that is, the average number of selected covariates among the important covariates, the average false positives (FP), that is, the average number of selected covariates among the unimportant covariates, and the average model errors (MR), the latter defined as ${\sum }_{i=1}^n{\{{\widehat{\mu }}_i({\mathbf{X}}_i,{\mathbf{T}}_i)-{\mu }_i({\mathbf{X}}_i,{\mathbf{T}}_i)\}}^2/n$, where μ̂_i(X_i, T_i) and μ_i(X_i, T_i) are the estimated and true conditional means for Y_i, respectively. We see that by fitting the proposed GACM, the GL method has larger percentage of over-fitting as well as larger average false positives than the AGL methods. The AGL improves the correct-fitting percentage by 26%. As a result, the AGL reduces the model fitting error by (0.083 − 0.059)/0.059 = 40.7% compared to the GL method. Moreover, both the logistic model and the GVCM fail to identify those important covariates with incorrect-fitting percentage close to or being 1. Furthermore, by using the screening method with logistic regression, the average true positive is 1.056, which is much less than 4 (the number of those important SNPs). This further illustrates that the traditional screening method is not an effective tool to identify important genetic factors in this context. In addition, we observe that the results for the AGL method in Table 1 are comparable to the results in Table S.1 of Example 2 (in the supplementary materials) at p = 1000 with the simulated SNPs in terms of having similar correct-fitting percentages and MR values.

Table 1.

Variable selection and estimation results by the adaptive group lasso and the group lasso with the GACM and GVCM, respectively, and parametric logistic regression with adaptive group lasso and screening methods based on 500 replications. The columns of C, O and I show the percentage of correct-fitting, over-fitting and incorrect-fitting. The columns TP, FP and MR show true positives, false positives and model errors, respectively

		C	O	I	TP	FP	MR
GACM	AGL	0.410	0.460	0.130	3.860	0.870	0.059
	GL	0.140	0.764	0.096	3.904	2.540	0.083
GVCM	AGL	0.030	0.000	0.970	1.636	5.685	0.142
	GL	0.060	0.000	0.940	2.076	20.670	0.120
Logistic regression	AGL	0.000	0.000	1.000	1.872	1.174	0.159
	Screening	0.000	0.000	1.000	1.056	0.786	0.141

Next, we investigate the empirical coverage rates of the unsmoothed and smoothed SCBs given in (17) and (19). To calculate the unsmoothed and smoothed bootstrap standard deviations (16) and (18), we use B = 500 bootstrap replications. The confidence bands are constructed at L_n = 20 equally spaced points. At 95% confidence level, Table 2 reports the empirical coverage rates (cov) and the sample averages of median and mean standard deviations (sd.median and sd.mean), respectively, for the unsmoothed SCB (17) and smoothed SCB (19) for coefficient functions α_ℓ1(x₁), ℓ = 1, 2, 3, 4. We see that the smoothed bootstrap method leads to better performance, having empirical coverage rates closer to the nominal confidence level 0.95.

Table 2. The empirical coverage rates (cov) and the sample average of median and mean of the standard deviations (sd.median and sd.mean)for the unsmoothed SCB (17) and smoothed SCB (19) for the coefficient functions α_ℓ1(x₁) for ℓ = 1, 2, 3, 4.

	Unsmoothed bootstrap			Smoothed bootstrap
	cov	sd.median	sd.mean	cov	sd.median	sd.mean
α11	0.610	0.689	0.809	0.818	0.735	0.982
α21	0.628	0.563	0.725	0.846	0.666	0.932
α31	0.636	0.736	0.832	0.869	0.837	1.053
α41	0.646	0.768	0.843	0.882	0.891	1.064

5. Data application

We illustrate our method via analysis of the Framingham Heart Study [Dawber, Meadors and Moore (1951)] to investigate the effects of G × E interactions on obesity. People are defined as obese when their body mass index (BMI) is 30 or greater: this is the definition of being obese made by the U.S. Centers for Disease Control and Prevention; see http://www.cdc.gov/obesity/adult/defining.html. We defined the response variable to be Y = 1 for BMI ≥ 30; and Y = 0 for BMI < 30. We use X₁ = sleeping hours per day; X₂ = activity hours per day; and X₃ = diastolic blood pressure as the environmental factors, and use single nucleotide polymorphisms (SNPs) located in the sixth chromosome as the genetic factors. The three possible allele combinations are coded as 1, 0 and −1. As in the simulation, we thus are fitting a multiplicative risk model in the SNPs. For details on genotyping, see http://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?studyid=phs000007.v3.p2. A total of 1286 SNPs remain in our analysis after eliminating SNPs with minor allele frequency <0.05, those with departure from Hardy–Weinberg equilibrium and those having correlation coefficient with the response between −0.1 and 0.1. We have n = 300 individuals left in our study after deleting observations with missing values.

To see possible nonlinear main effects of the environmental factors, we first fit a generalized additive model by using X₁, X₂ and X₃ as predictors such that

$$E(Y_i|{\mathbf{X}}_i,{\mathbf{T}}_i)=g^{-1}\{\eta ({\mathbf{X}}_i)\}\text{with}\eta ({\mathbf{X}}_i)=m_0+\sum _{k=1}^3{m}_k(X_{ik}).$$ (21)

Figure S.1 given in the supplementary material [Ma et al. (2015)] depicts the plots of m̂_k(·) for k = 1, 2, 3 by one-step cubic spline estimation. Clearly the estimate of each nonparametric function has a nonlinear pattern. We refer to Section S.2 for the detailed description of this figure. Based on the plots shown in Figure S.1, we fit the GACM model

$$\eta ({\mathbf{X}}_i,{\mathbf{T}}_i)=\sum _{\ell =1}^{1287}\{{\alpha }_{\ell 0}+\sum _{k=1}^3{\alpha }_{\ell k}(X_{ik})\}{T}_{i\ell },$$ (22)

where T_i = (T_i1,T_i2,…, T_i1287)^T with T_i1 = 1, and T_iℓ are the SNP covariates for ℓ = 2,…, 1287. The nonparametric function α_ℓk(·) is estimated by cubic splines, and the number of interior knots for each step is selected based on the criterion described in Section 2.4. We select variables in model (22) by the proposed adaptive group lasso (AGL) and the group lasso (GL). To compare the proposed model with linear models, we perform the group lasso by assuming linear interaction effects (Linear) such that ${\alpha }_{\ell }({\mathbf{X}}_i)={\alpha }_{\ell 0}+{\sum }_{k=1}^3{\beta }_{\ell k}{X}_{ik}$, and we also perform the lasso by assuming no interaction effects (No interaction) such that α_ℓ(X_i) = α_ℓ0. We also apply the screening method with parametric logistic regression (Screening) as described in Example 2. Table 3 reports the variable selection results in these five scenarios. After model selection, we calculate the estimated leave-one-out cross-validation prediction error (CVPE) for the model with the selected variables as shown in the last row of Table 3. Among the selected SNPs by the AGL method, two SNPs, rs4714924 and rs6543930, have been scientifically confirmed by Randall et al. (2013) to have strong associations with obesity. Moreover, compared to the linear, no interaction and screening methods, our proposed AGL with GACM method enables us to identify more genetic factors, which may be important to the response but missed out by other methods. As a result, it has the smallest CVPE (0.078), so that it significantly improves model prediction compared to other methods. We also see that the logistic model that completely ignores interactions has the largest CVPE (0.152). The screening method has the second largest CVPE (0.149), which is larger than that of the penalization method (0.124) obtained by fitting the same logistic regression model but including interaction considered. This result demonstrates that the screening method is not as effective as the penalization method for analysis of this data set, a result which also agrees with our simulations.

Table 3.

Variable selection results for the group lasso (GL) and the adaptive group lasso (AGL) in model (22), the group lasso by assuming linear interaction effects (linear), the lasso by assuming no interaction effects (no interaction) and the screening method (screening). The symbol ✓ indicates that the SNP was selected into the model. The last row shows the cross validation prediction errors (CVPE)

SNPs	GL	AGL	Linear	No interaction	Screening
rs9296244	✓	✓
rs6910353	✓	✓
rs3130813	✓	✓
rs9353447				✓	✓
rs4714924	✓	✓	✓		✓
rs242263	✓	✓	✓
rs282123	✓
rs282128	✓	✓
rs6929006			✓
rs9353711	✓
rs12199154	✓	✓
rs2277114	✓
rs749517	✓
rs729888	✓
rs203139			✓
rs6914589	✓	✓
rs6543930	✓	✓
CVPE	0.099	0.078	0.124	0.152	0.149

Next we fit the final GACM selected variables from the AGL procedure as

$$\eta ({\mathbf{X}}_i,{\mathbf{T}}_i)=\sum _{\ell =1}^{10}\{{\alpha }_{\ell 0}+\sum _{k=1}^3{\alpha }_{\ell k}(X_{ik})\}{T}_{i\ell }.$$ (23)

To illustrate the main effects of the environmental factors, Figure 2 plots the smoothed two-step estimated functions ${\stackrel{\sim }{\alpha }}_{1k}^{\mathcal{S}}(\cdot )$ of the functions ${\alpha }_{1k}^{\mathcal{S}}(\cdot )$, for k = 1, 2, 3, and the associated 95% smoothed SCBs (upper and lower solid lines). The plots of the functional estimates have the same nonlinear change patterns as the corresponding plots in Figure S.1, although because of the addition of the SCBs, the scale of the plot has changed.

Fig. 2. Plots of the smoothed two-step estimated functions ${\stackrel{\sim }{\alpha }}_{1k}^{\mathcal{S}}(\cdot )$ for k = 1, 2, 3 and the associated 95% SCBs based on model (23).

To illustrate the effects of the genetic factors changing with the environmental factors, in Figure 3 we plot the smoothed two-step estimated functions ${\stackrel{\sim }{\alpha }}_{6k}^{\mathcal{S}}(\cdot )$ and the associated 95% smoothed SCBs of the coefficient functions ${\alpha }_{6k}^{\mathcal{S}}(\cdot )$ for the SNP rs242263. To further demonstrate how the probability of developing obesity changes with the environmental factors for each category of SNP rs242263, Figure 4 plots the estimated conditional probability of obesity against each environmental factor by letting T_iℓ = 0 for ℓ ≠ 6. Letting A be the minor allele, the curves are for aa (solid line), Aa (dashed line) and AA (dotted line). Figure 3 indicates different changing patterns of the interaction effects under different environments. For example, sleeping hours seem to have an overall more significant interaction effect with this particular SNP than the other two variables. The effect of this SNP changes from positive to negative and then to positive again as the sleeping hours increase. The coefficient functions of the SNP have an increasing pattern along with the activity hours and diastolic blood pressure, respectively. From Figure 4, we observe that there are stronger differences among the levels AA, Aa, and aa of SNP rs242263 for both large and small values of the environmental factors. There are other interesting results worth further study. For example, in the 2–6 hours per day sleeping range, the AA group (dotted lines) have much higher rates of obesity than the aa group (solid line), but the opposite occurs in the 6–9 hour range. For those with low amounts of activity per day, again the AA group is more obese than the aa group, while when activity increases, the AA group is less obese than the aa group. A similar noticeable difference occurs between the <60 diastolic blood pressure group, those who are hypotensive, and the >90 group, those who are hypertensive, although there are few subjects in the former group.

Fig. 3. Plots of the smoothed two-step estimated functions ${\stackrel{\sim }{\alpha }}_{5k}^{\mathcal{S}}(\cdot )$ for k = 1, 2, 3 and the associated 95% SCBs based on model (23).

Fig. 4.

Plots of the estimated conditional probability of obesity against each environmental factor by letting T_iℓ = 0 for ℓ ≠ 5. With A being the minor allele, the curves are aa (solid line), Aa (dashed line) and AA (dotted line), based on model (23).

6. Discussions

The generalized additive coefficient model (GACM) proposed by Xue and Yang (2006) and Xue and Liang (2010) has been demonstrated to be a powerful tool for studying nonlinear interaction effects of variables. To promote the use of the GACM in modern data applications such as gene-environment (G × E) interaction effects in GWAS, we have proposed estimation and inference procedures for the GACM when the dimension of the variables is high. Specifically, we have devised a groupwise penalization method in the GACM for simultaneous model selection and estimation. We showed by numerical studies that we can effectively identify important genetic factors by using the proposed nonparametric model while traditional generalized parametric models such as logistic regression model fails to do so when nonlinear interactions exist. Moreover, by comparing with the conventional screening method with logistic regression as commonly used in the GWAS community, our proposed groupwise penalization method with the GACM has been demonstrated to be more effective for variable selection and model estimation. After identifying those important covariates, we have further constructed simultaneous confidence bands for the nonzero coefficient functions based on a refined two-step estimator. We estimate the standard deviation of the functional estimator by a smoothed bootstrap method as proposed in Efron (2014). The method was shown to have good numerical performance by reducing variability as well as improving the empirical coverage rate of the proposed simultaneous confidence bands. Our methods can be extended to longitudinal data settings through marginal models or mixed-effects models. More work, however, is needed to understand the properties of the estimators in such new settings. Moreover, extending this work to the setting with the dimensions for both genetic and environmental factors growing with the sample size can be a future project to be considered. Some associated theoretical properties with respect to model selection and estimation as well as inference need to be carefully investigated.

Appendix

Denote the space of the qth order smooth functions as C^(q)([0, 1]) = {ϕ|ϕ^(q) ∈ C[0, 1]}. For any s × s symmetric matrix A, denote its L_q norm as ‖A‖_q = max_{ς∈R^s,‖ς‖2=1} ‖Aς‖_q. Let ${\Vert \mathbf{A}\Vert }_{\infty }={max}_{1\le i\le s}{\sum }_{j=1}^s|a_{ij}|$. For a vector a, let ‖a‖_∞ = max_1≤i≤s |a_i|.

Let C^{0, 1} (χ_w) be the space of Lipschitz continuous functions on χ_w, that is,

$$C^{0,1}({\chi }_w)=\{\phi :{\Vert \phi \Vert }_{0,1}=\underset{w\ne w^{\text{'}},w,w^{\text{'}}\in {\chi }_w}{sup}\frac{|\phi (w)-\phi (w^{\text{'}})|}{|w-w^{\text{'}}|}<+\infty \},$$

in which ‖φ‖_{0, 1} is the C^{0, 1}-norm of φ. Denote q_j(η, y) = ∂^j Q{g⁻¹(η), y}/∂η^j, so that

$$q_1(\eta ,y)=\frac{\mathit{\partial }}{\mathit{\partial }\eta }Q\{g^{-1}(\eta ),y\}=-\{y-g^{-1}(\eta )\}{\rho }_1(\eta ),$$

$$q_2(\eta ,y)=\frac{{\mathit{\partial }}^2}{\mathit{\partial }{\eta }^2}Q\{g^{-1}(\eta ),y\}={\rho }_2(\eta )-\{y-g^{-1}(\eta )\}{\rho }_1^{\text{'}}(\eta ),$$

where ρ_j(η) = {ġ⁻¹ (η)}^j/V{g⁻¹ (η)}.

A.1. Assumptions

Throughout the paper, we assume the following regularity conditions:

(C1) The joint density of X, denoted by f(x), is absolutely continuous, and there exist constants 0 < c_f ≤ C_f < ∞, such that c_f ≤min_{x∈[0, 1]^d} f(x) ≤ max_{x∈[0, 1]^d} f(x) ≤ C_f.

(C2) The function V is twice continuously differentiable, and the link function g is three times continuously differentiable. The function q₂(η, y) < 0 for η ∈ R and y in the range of the response variable.

(C3) For 1 ≤ ℓ ≤ p, 1 ≤ k ≤ d, ${\alpha }_{\ell k}^{(r-1)}(x_k)\in C^{0,1}[0,1]$, for given integer r ≥ 1. The spline order satisfies q ≥ r.

(C4) Let ε_i = Y_i − μ(X_i, T_i), 1 ≤ i ≤ n. The random variables ε₁,…, ε_n are i.i.d. with E(ε_i) = 0 and var(ε_i|X_i, T_i) = σ²(X_i, T_i). Furthermore, their tail probabilities satisfy P(|ε_i| > x) < K exp(−Cx²), i = 1,…, n, for all x ≥ 0 and for some positive constants C and K.

(C5) The eigenvalues of $E({\mathbf{T}}_{I_1}{\mathbf{T}}_{I_1}^{\mathrm{T}}|\mathbf{X}=\mathbf{x})$, where T_I1 = (T_ℓ, ℓ ∈ I₁)^T, are uniformly bounded away from 0 and ∞ for all x ∈ [0, 1]^d. There exist constants 0 < c₁ < C₁ < ∞, such that $c_1\le E(T_{\ell }^2|\mathbf{X}=\mathbf{x})\le C_1$, for all x ∈ [0, 1]^d, ℓ ∈ I₂.

Conditions (C1)–(C5) are standard conditions for nonparametric estimation. Condition (C1) is the same as condition (C1) in Xue and Yang (2006) and condition (C5) in Xue and Liang (2010). The first condition in (C2) gives the assumptions on V and the link function g, which can be found in condition (E) of Lam and Fan (2008). The second condition in (C2) guarantees that the negative quasi-likelihood function Q{g⁻¹(η), y} is convex in η ∈ R, which is also given in condition (D) of Lam and Fan (2008) and (a) of condition 1 in Carroll et al. (1997). Condition (C3) is typical for polynomial spline smoothing; see the same condition given in Section 5.2 of Huang (2003). Condition (C4) is the same as assumption (A2) given in Huang, Horowitz and Wei (2010). Condition (C5) is given in condition (C5) of Xue and Liang (2010) and condition (A5) in Ma and Yang (2011b).

A.2. Preliminary lemmas

Define ${\alpha }_{\ell }^0(\mathbf{x})={\sum }_{k=1}^d{\alpha }_{\ell k}^0(x_k)=B{(\mathbf{x})}^{\mathrm{T}}{\mathit{\gamma }}_{\ell }$, where ${\alpha }_{\ell k}^0(x_k)$ is defined in (6). Let γ_I1 = (γ_ℓ : ℓ ∈ I₁)^T. To prove Theorem 1, we next define the oracle estimator of γ_I1 by minimizing the penalized negative quasi-likelihood with all irrelevant predictors eliminated as such

$$L_n({\mathit{\gamma }}_{I_1})=\sum _{i=1}^{n}Q\left[g^{-1}\right\{\sum _{\ell \in I_1}B{({\mathbf{X}}_i)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell }{T}_{\ell }\},Y_i]+n{\lambda }_n\sum _{\ell \in I_1}{w}_{n\ell }{\Vert {\mathit{\gamma }}_{\ell }\Vert }_2,$$ (24)

so that ${\widehat{\mathit{\gamma }}}_{I_1}^0={({\widehat{\mathit{\gamma }}}_{\ell }^0:\ell \in I_1)}^{\mathrm{T}}=arg{min}_{{\mathit{\gamma }}_{I_1}}{L}_n({\mathit{\gamma }}_{I_1})$. Define ${\widehat{\mathit{\gamma }}}_{I_2}^0={({\widehat{\mathit{\gamma }}}_{\ell }^0:\ell \in I_2)}^{\mathrm{T}}$ with ${\widehat{\mathit{\gamma }}}_{\ell }^0\equiv {\mathbf{0}}_{d{J}_n+1}$ for ℓ ∈ I₂, where 0_{d Jn+1} is a (d J_n + 1)-dimensional zero vector. We next present several lemmas, whose detailed proofs are given in the online supplementary materials [Ma et al. (2015)]. Lemma A.1 is used for the proof of Theorem 1, while Lemma A.2 is needed in the proof of Theorem 3.

Lemma A.1. nder the conditions of Theorem 1, one has

$${\Vert {\widehat{\mathit{\gamma }}}_{I_1}^0-{\mathit{\gamma }}_{I_1}\Vert }_2=O_p({\lambda }_n\Vert w_{n,I_1}\Vert +n^{-1/2}{J}_n^{1/2}+J_n^{-r}),$$ (25)

and as n → ∞,

$$P\{\widehat{\mathit{\gamma }}={({\widehat{\mathit{\gamma }}}_{I_1}^{0\mathrm{T}},{\widehat{\mathit{\gamma }}}_{I_2}^{0\mathrm{T}})}^{\mathrm{T}}\}\to 1.$$ (26)

Lemma A.2. Under conditions (C1)–(C5) and Assumptions 1–3,

$${\Vert {\widehat{\mathit{\gamma }}}_{,1}^{\mathcal{S}}-{\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}\Vert }_{\infty }=O_p(\sqrt{logn/\left(J_n^{\mathcal{S}}n\right)}+{\left(J_n^{\mathcal{S}}\right)}^{-1/2}{\left(J_n^{\text{ini}}\right)}^{-r}).$$ (27)

A.3. Proof of Theorem 1

By (25) and (26),

$$\begin{array}{c}\sum _{\ell \in I_1}\Vert {\widehat{\alpha }}_{\ell }-{\alpha }_{\ell }\Vert \asymp {\Vert {\widehat{\mathit{\gamma }}}_{I_1}-{\mathit{\gamma }}_{I_1}\Vert }_2=O_p({\lambda }_n\Vert w_{n,I_1}\Vert +n^{-1/2}{J}_n^{1/2}+J_n^{-r}),\\ P(\Vert {\widehat{\alpha }}_{\ell }\Vert >0,\ell \in I_1\text{and}\Vert {\widehat{\alpha }}_{\ell }\Vert =0,\ell \in I_2)\to 1.\end{array}$$

A.4. Proof of Theorem 2

Let γ_,1 = (γ_ℓ1, ℓ ∈ Î₁)^T, where γ_ℓ1 is defined in (7). By Taylor's expansion, from (10), one has

$${\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}-{\mathit{\gamma }}_{,1}={[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i^{\ast })\}}^2/{\sigma }_i^2]}^{-1}\times [\sum _{i=1}^n{Z}_{i,1}\{Y_i-g^{-1}({\eta }_i^0)\}({\dot{g}}^{-1}({\eta }_i^0)/{\sigma }_i^2)],$$

Where ${\eta }_i^0={\sum }_{\ell =1}^p\{{\alpha }_{\ell 0}+{\sum }_{k=2}^d{\alpha }_{\ell k}(X_{ik})\}{T}_{i\ell }+{\sum }_{\ell =1}^p{B}^{\mathcal{S}}{(x_1)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell 1}{T}_{i\ell }$ and

$${\eta }_i^{\ast }=\sum _{\ell =1}^p\{{\alpha }_{\ell 0}+\sum _{k=2}^d{\alpha }_{\ell k}(X_{ik})\}{T}_{i\ell }+\sum _{\ell =1}^p{B}^{\mathcal{S}}{(x_1)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell 1}^{\ast }{T}_{i\ell },$$

where ${\mathit{\gamma }}_{,1}^{\ast }={({\mathit{\gamma }}_{\ell 1}^{\ast },\ell \in {\widehat{I}}_1)}^{\mathrm{T}}\in ({\mathit{\gamma }}_{,1},{\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}})$. Following similar reasoning as the proofs for (25), we have ${\Vert {\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}-{\mathit{\gamma }}_{,1}\Vert }_2=o_p(1)$. Then ${\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}-{\mathit{\gamma }}_{,1}=({\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}+{\widehat{\mathit{\gamma }}}_{,1\mu }^{\text{OR}})+o_p(1)$, where

$${\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}={[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}[\sum _{i=1}^n{Z}_{i,1}{\epsilon }_i\{{\dot{g}}^{-1}({\eta }_i)/{\sigma }_i^2\}],$$

$${\widehat{\mathit{\gamma }}}_{,1\mu }^{\text{OR}}={[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}\times [\sum _{i=1}^n{Z}_{i,1}\{g^{-1}({\eta }_i)-g^{-1}({\eta }_i^0)\}\{{\dot{g}}^{-1}({\eta }_i)/{\sigma }_i^2\}].$$ (28)

Therefore, $var({\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}|\mathbb{X},\mathbb{T})={[{\sum }_{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}$. By Theorem 5.4.2 of DeVore and Lorentz (1993), for sufficiently large n, there exist constants 0 < c_B ≤ C_B < ∞, such that $c_B{\mathbf{I}}_{J_n^{\mathcal{S}}\times J_n^{\mathcal{S}}}\le E(B_1^{\mathcal{S}}(X_{i1})B_1^{\mathcal{S}}{(X_{i1})}^{\mathrm{T}})\le C_B{\mathbf{I}}_{J_n^{\mathcal{S}}\times J_n^{\mathcal{S}}}$. By condition (C5), for n large enough, there are constants 0 < C_T, C′ < ∞, such that

$$E[Z_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]\le C^{\text{'}}E[\{B_1^{\mathcal{S}}(X_{i1})B_1^{\mathcal{S}}{(X_{i1})}^{\mathrm{T}}\}\otimes {\{E(T_{\ell }{T}_{{\ell }^{\text{'}}}|\mathbf{X})\}}_{\ell ,{\ell }^{\text{'}}\in {\widehat{I}}_1}]\le C{C}_T{s}^{\ast }E\{B_1^{\mathcal{S}}(X_{i1})B_1^{\mathcal{S}}{(X_{i1})}^{\mathrm{T}}\}\otimes {\mathbf{I}}_{s^{\ast }\times s^{\ast }}\le C^{\text{'}}{C}_T{C}_B{s}^{\ast }{\mathbf{I}}_{J_n^{\mathcal{S}}\times J_n^{\mathcal{S}}}\otimes {\mathbf{I}}_{s^{\ast }\times s^{\ast }}=C{s}^{\ast }{\mathbf{I}}_{J_n^{\mathcal{S}}{s}^{\ast }\times J_n^{\mathcal{S}}{s}^{\ast }},$$

where C = C′C_TC_B. Similarly, we have $E[Z_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]\ge c{\mathbf{I}}_{J_n^{\mathcal{S}}{s}^{\ast }\times J_n^{\mathcal{S}}{s}^{\ast }}$ for some constant 0 < c < ∞. Thus, following the same reasoning as the proof for (S.5) in the supplementary materials [Ma et al. (2015)], we have with probability 1, for n → ∞,

$$C^{-1}{(s^{\ast })}^{-1}{n}^{-1}{\mathbf{I}}_{J_n^{\mathcal{S}}{s}^{\ast }\times J_n^{\mathcal{S}}{s}^{\ast }}\le {[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}\le c^{-1}{n}^{-1}{\mathbf{I}}_{J_n^{\mathcal{S}}{s}^{\ast }\times J_n^{\mathcal{S}}{s}^{\ast }}.$$ (29)

By the Lindeberg central limit theorem, it can be proved that

$${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{B^{\mathcal{S}}(x_1){\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}\}\to N(0,1),$$ (30)

for any a ∈ R^s* with ‖a‖₂ = 1. Since ${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\text{OR}}(x_1)-b_1(x_1)\}={\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{B^{\mathcal{S}}(x_1){\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}\}+o_p(1)$, by (30) and Slutsky's theorem, we have

$${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\text{OR}}(x_1)-b_1(x_1)\}\to N(0,1).$$ (31)

By (28) and (29), with probability approaching 1,

$$\sum _{\ell \in I_1}{\Vert {\widehat{\alpha }}_{\ell 1}^{\text{OR}}-b_{\ell 1}\Vert }^2\asymp {\Vert {\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}\Vert }_2^2\le c^{-2}{n}^{-2}[\sum _{i=1}^n{\epsilon }_i{Z}_{i,1}^{\mathrm{T}}({\dot{g}}^{-1}({\eta }_i)/{\sigma }_i^2)][\sum _{i=1}^n{Z}_{i,1}{\epsilon }_i({\dot{g}}^{-1}({\eta }_i)/{\sigma }_i^2)]\asymp c^{-2}{n}^{-1}E[Z_{i,1}^{\mathrm{T}}{Z}_{i,1}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]\asymp s^{\ast }{J}_n^{\mathcal{S}}{n}^{-1};$$

$${\Vert {\mathbf{a}}^{\mathrm{T}}({\widehat{\alpha }}_1^{\text{OR}}-b_1)\Vert }^2\le C_a{\Vert {\widehat{\mathit{\gamma }}}_{,1e}^{\text{OR}}\Vert }_2^2\le C_a{c}^{-1}{n}^{-2}\left(\sum _{i=1}^n{\epsilon }_i{Z}_{i,1}^{\mathrm{T}}\right)\left(\sum _{i=1}^n{Z}_{i,1}{\epsilon }_i\right)\asymp C_a{c}^{-1}{n}^{-1}E(Z_{i,1}^{\mathrm{T}}{Z}_{i,1})\asymp s^{\ast }{J}_n^{\mathcal{S}}{n}^{-1}.$$

Since ${sup}_{x_1\in [0,1]}|{\alpha }_{\ell 1}(x_1)-B_1^{\mathcal{S}}{(x_1)}^{\mathrm{T}}{\mathit{\gamma }}_{\ell 1}|=O\{{(J_n^{\mathcal{S}})}^{-r}\}$, it can be proved that $\Vert {\mathbf{a}}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{,1\mu }^{\text{OR}}\Vert \le {\Vert {\widehat{\mathit{\gamma }}}_{,1\mu }^{\text{OR}}\Vert }_2=O_p\{{(s^{\ast })}^{1/2}{(J_n^{\mathcal{S}})}^{-r}\}$, and $\Vert {\mathbf{a}}^{\mathrm{T}}(b_1-{\alpha }_1^0)\Vert \asymp \Vert {\mathbf{a}}^{\mathrm{T}}{\widehat{\mathit{\gamma }}}_{,1\mu 2}^{\text{OR}}\Vert =O_p\{{(s^{\ast })}^{1/2}{(J_n^{\mathcal{S}})}^{-r}\}$. Hence

$$\Vert {\mathbf{a}}^{\mathrm{T}}(b_1-{\alpha }_1)\Vert \le \Vert {\mathbf{a}}^{\mathrm{T}}(b_1-{\alpha }_1^0)\Vert +\Vert {\mathbf{a}}^{\mathrm{T}}({\alpha }_1^0-{\alpha }_1)\Vert =O_p\{s^{\ast }{(J_n^{\mathcal{S}})}^{-r}\}.$$

By (31), ${\{{\mathbf{e}}_{\ell }^{\mathrm{T}}{\sigma }_n^2(x_1){\mathbf{e}}_{\ell }\}}^{-1/2}\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-b_{\ell 1}\}\to N(0,1)$, and ${sup}_{\ell \in {\widehat{I}}_1}|{\widehat{\alpha }}_{\ell 0}^{\text{OR}}-{\alpha }_{\ell 0}|=O_p(n^{-1/2})$ follows from the central limit theorem.

A.5. Proof of Theorem 3

By (27) in Lemma A.2,

$$\underset{x_1\in [0,1]}{sup}{\Vert {\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_1^{\text{OR}}(x_1)\Vert }_{\infty }\le \underset{x_1\in [0,1]}{sup}\sum _{j=1}^{J_n^{\mathcal{S}}}|B_{j,1}^{\mathcal{S}}(x_1)|{\Vert {\widehat{\mathit{\gamma }}}_{,1}^{\mathcal{S}}-{\widehat{\mathit{\gamma }}}_{,1}^{\text{OR}}\Vert }_{\infty }.$$

The right-hand side is bounded by $O_p\{{(n^{-1}logn)}^{1/2}+{(J_n^{\text{ini}})}^{-r}\}\cdot {\Vert {\widehat{\mathbf{\alpha }}}_0^{\mathcal{S}}-{\widehat{\mathbf{\alpha }}}_0^{\text{OR}}\Vert }_2=o_p(n^{-1/2})$ can be proved following the same procedure and thus omitted. By (29), with probability approaching 1, for large enough n, for any x₁ ∈ [0, 1], and a ∈ R^s* with ‖a‖₂ = 1, one has

$${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^2(x_1)\mathbf{a}\le c_Z^{-1}{n}^{-1}{\mathbf{a}}^{\mathrm{T}}{B}^{\mathcal{S}}(x_1)B^{\mathcal{S}}{(x_1)}^{\mathrm{T}}\mathbf{a}\le c^{-1}{J}_n^{\mathcal{S}}{n}^{-1}{\mathbf{a}}^{\mathrm{T}}\mathbf{a},$$

$${\mathbf{a}}^{\mathrm{T}}{\sigma }_n^2(x_1)\mathbf{a}\ge C_Z^{-1}{(s^{\ast })}^{-1}{n}^{-1}{\mathbf{a}}^{\mathrm{T}}{B}^{\mathcal{S}}(x_1)B^{\mathcal{S}}{(x_1)}^{\mathrm{T}}\mathbf{a}\ge C^{-1}{J}_n^{\mathcal{S}}{(s^{\ast })}^{-1}{n}^{-1}{\mathbf{a}}^{\mathrm{T}}\mathbf{a},$$

where ${\sigma }_n^2(x_1)$ is defined in (12). Thus

$$\underset{x_1\in [0,1]}{sup}|{\mathbf{a}}^{\mathrm{T}}{\sigma }_n^{-1}(x_1)\{{\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_1^{\text{OR}}(x_1)\}|\le \underset{x_1\in [0,1]}{sup}{\Vert {\sigma }_n^{-1}(x_1)\Vert }_2{\Vert {\widehat{\alpha }}_1^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_1^{\text{OR}}(x_1)\Vert }_2=O_p[s^{\ast }\{{(logn/J_n^{\mathcal{S}})}^{1/2}+{(n/J_n^{\mathcal{S}})}^{1/2}{(J_n^{\text{ini}})}^{-r}\}]=o_p(1).$$

A.6. Proof of Theorem 4

Using the strong approximation lemma given in Theorem 2.6.7 of Csörgő and Révész (1981), we can prove by the same procedure as Lemma A.7 in Ma, Yang and Carroll (2012) that

$$\underset{x_1\in [0,1]}{sup}|{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-b_{\ell 1}(x_1)-{\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)|=o_{\mathrm{a}.\mathrm{s}.}(n^t)$$ (32)

for some t < –r/(2r + 1) < 0, where ${\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)$ is

$${\mathbf{e}}_{\ell }^{\mathrm{T}}{B}^{\mathcal{S}}(x_1){[\sum _{i=1}^n{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}}{\{{\dot{g}}^{-1}({\eta }_i)\}}^2/{\sigma }_i^2]}^{-1}[\sum _{i=1}^n{Z}_{i,1}{e}_i\{{\dot{g}}^{-1}({\eta }_i)/{\sigma }_i^2\}],$$

and e_i, 1 ≤ i ≤ n, are i.i.d. N(0, 1) independent of Z_{i, 1}. For ${\sigma }_n^2(x_1)$ defined in (12) and ${\sigma }_{n1}(x_1)\asymp {(J_n^{\mathcal{S}}/n)}^{1/2}\{1+o_p(1)\}$ uniformly in x₁ ∈ [0, 1]. By (32), $J_n^{\mathcal{S}}\asymp n^{1/(2r+1)}$ and t < –r/(2r + 1) < 0, we have

$$\underset{x_1\in [0,1]}{sup}|{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-b_{\ell 1}(x_1)-{\widehat{\alpha }}_{\ell 1,\epsilon }^0(xl1)\}|=o_{\mathrm{a}.\mathrm{s}.}({\{log(L_n+1)\}}^{-1/2}{(n/J_n^{\mathcal{S}})}^{1/2}{n}^t)=o_{\mathrm{a}.\mathrm{s}.}({\{log(L_n+1)\}}^{-1/2}{n}^{r/(2r+1)-t})=o_{\mathrm{a}.\mathrm{s}.}(1).$$ (33)

Define $\eta (x_1)={\sigma }_{n1}^{-1}(x_1){\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)$. It is apparent that {η(ξ_J)|Z_{i, 1}, 1 ≤ i ≤ n} = N(0, 1), so {η(ξ_J)} = N(0,1) for 0 ≤ J ≤ L_n. Moreover, the eigenvalues of ${(E{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}})}^{-1}\asymp J_n^{\mathcal{S}}$. Then with probability approaching 1, for J ≠ J′,

$$|E\{\eta ({\xi }_J)\eta ({\xi }_{J^{\text{'}}})\}|\asymp (n/J_n^{\mathcal{S}})n^{-1}|{\mathbf{e}}_{\ell }^{\mathrm{T}}{B}^{\mathcal{S}}({\xi }_J){(E{Z}_{i,1}{Z}_{i,1}^{\mathrm{T}})}^{-1}{B}^{\mathcal{S}}{({\xi }_{J^{\text{'}}})}^{\mathrm{T}}{\mathbf{e}}_{\ell }|\asymp |{\mathbf{e}}_{\ell }^{\mathrm{T}}{B}^{\mathcal{S}}({\xi }_J)B^{\mathcal{S}}{({\xi }_{J^{\text{'}}})}^{\mathrm{T}}{\mathbf{e}}_{\ell }|=\sum _{j=1}^{J_n^{\mathcal{S}}}{B}_{j,1}^{\mathcal{S}}({\xi }_J)B_{j,1}^{\mathcal{S}}({\xi }_{J^{\text{'}}})$$

and ${\sum }_{j=1}^{J_n^{\mathcal{S}}}{B}_{j,1}^{\mathcal{S}}({\xi }_J)B_{j,1}^{\mathcal{S}}({\xi }_{J^{\text{'}}})\asymp C$ for a constant 0 < C < ∞ when |j_J – j_J′| ≤ (q − 1) and ${\sum }_{j=1}^{J_n^{\mathcal{S}}}{B}_{j,1}^{\mathcal{S}}({\xi }_J)B_{j,1}^{\mathcal{S}}({\xi }_{J^{\text{'}}})=0$ when |j_J – j_J′| > (q – 1), in which j_J denotes the index of the knot closest to ξ_J from the left. Therefore, by $L_n\asymp J_n^{\mathcal{S}}$, there exist constants 0 < C₁ < ∞ and 0 < C₂ < ∞ such that with probability approaching 1, for J ≠ J′, $|E\{\eta ({\xi }_J)\eta ({\xi }_{J^{\text{'}}})\}|\le C_1^{-|j_J-j_{J^{\text{'}}}|}\le C_2^{-|J-J^{\text{'}}|}$. By Lemma A1 given in Ma and Yang (2011a), we have

$$\underset{n\to \infty }{lim}P\{\underset{0\le J\le L_n}{sup}|{\{2\text{log}(L_n+1)\}}^{-1/2}\eta ({\xi }_J)|\le d_{N_n}(\alpha )\}=1-\alpha ,$$

and hence

$$\underset{n\to \infty }{lim}P\{\underset{x_1\in S_{n,1}}{sup}|{\{2log(L_n+1)\}}^{-1/2}{\sigma }_n^{-1}(x_1){\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)|\le d_{N_n}(\alpha )\}=1-\alpha .$$ (34)

Furthermore, according to the result on page 149 of de Boor (2001), we have

$$\underset{x_1\in [0,1]}{sup}|{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)\{b_{\ell 1}(x_1)-{\alpha }_{\ell 1}(x_1)\}|=O_p({\{log(L_n+1)\}}^{-1/2}{(n/J_n^{\mathcal{S}})}^{1/2}{(J_n^{\mathcal{S}})}^{-r})=o_p(1).$$ (35)

Moreover, ${\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-{\alpha }_{\ell 1}(x_1)={\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)+\{{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-b_{\ell 1}(x_1)-{\widehat{\alpha }}_{\ell 1,\epsilon }^{\text{OR}}(x_1)\}+\{b_{\ell 1}(x_1)-{\alpha }_{\ell 1}(x_1)\}$. Hence by (33) and (35), we have

$$\underset{n\to \infty }{lim}P\{\underset{x_1\in S_{n,1}}{sup}{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)|{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)-{\alpha }_{\ell 1}|\le d_{N_n}(\alpha )\}=\underset{n\to \infty }{lim}P\{\underset{x_1\in S_{n,1}}{sup}{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)|{\widehat{\alpha }}_{\ell 1,\epsilon }^0(x_1)|\le d_{N_n}(\alpha )\}=1-\alpha ,$$ (36)

where the last step follows from (34). By the oracle property given in Theorem 3, and $J_n^{\mathcal{S}}\asymp n^{1/(2r+1)}$ and $n^{1/(2r+1)}\ll J_n^{\text{ini}}$, we have

$$\underset{x_1\in [0,1]}{sup}{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)|{\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)-{\widehat{\alpha }}_{\ell 1}^{\text{OR}}(x_1)|=O_p[log{(L_n+1)}^{-1/2}{(n/J_n^{\mathcal{S}})}^{1/2}{(n^{-1}logn)}^{1/2}+{(J_n^{\text{ini}})}^{-r}]=o_p(1).$$ (37)

Therefore, by (36) and (37), we have

$$\underset{n\to \infty }{lim}P\{\underset{x_1\in S_{n,1}}{sup}{\{log(L_n+1)\}}^{-1/2}{\sigma }_{n1}^{-1}(x_1)|{\widehat{\alpha }}_{\ell 1}^{\mathcal{S}}(x_1)-{\alpha }_{\ell 1}(x_1)|\le d_{N_n}(\alpha )\}=1-\alpha ,$$

and hence the result in Theorem 4 is proved.