Variable Selection in Nonparametric Varying-Coefficient Models for Analysis of Repeated Measurements

SUMMARY

Nonparametric varying-coefficient models are commonly used for analysis of data measured repeatedly over time, including longitudinal and functional responses data. While many procedures have been developed for estimating the varying-coefficients, the problem of variable selection for such models has not been addressed. In this article, we present a regularized estimation procedure for variable selection that combines basis function approximations and the smoothly clipped absolute deviation (SCAD) penalty. The proposed procedure simultaneously selects significant variables with time-varying effects and estimates the nonzero smooth coefficient functions. Under suitable conditions, we have established the theoretical properties of our procedure, including consistency in variable selection and the oracle property in estimation. Here the oracle property means that the asymptotic distribution of an estimated coefficient function is the same as that when it is known a priori which variables are in the model. The method is illustrated with simulations and two real data examples, one for identifying risk factors in the study of AIDS and one using microarray time-course gene expression data to identify the transcription factors related to the yeast cell cycle process.

1 Introduction

Varying-coefficient (VC) models (Hastie and Tibshirani, 1993) are commonly used for studying the time-dependent effects of covariates on responses measured repeatedly. Such models can be used for longitudinal data where subjects are often measured repeatedly over a given period of time, so that the measurements within each subject are possibly correlated with each other (Diggle et al., 1994; Rice, 2004). Another setting is the functional response models (Rice, 2004), where the ith response is a smooth real function y_i(t), i = 1,…, n, t ∈ = [0, T], although in practice only y_i(t_ij), j = 1,…, J_i are observed. For both settings, the response Y (t) is a random process and the predictor X(t) = (X⁽¹⁾(t),…,X^(p)(t))^T is a p-dimensional random process. In applications, observations for n randomly selected subjects are obtained as (y_i(t_ij); x_i(t_ij)) for the ith subject at discrete time point t_ij, i = 1,…,n and j = 1,…,J_i. The linear varying-coefficient model can be written as

$$y_i(t_{ij})={\mathit{x}}_i{(t_{ij})}^T\mathit{\beta }(t_{ij})+{\epsilon }_i(t_{ij}),$$ (1)

where β(t) = (β₁(t),…,β_p(t))^T is a p-dimensional vector of smooth functions of t, and ε_i(t), i = 1,…,n are independently identically distributed random processes, independent of x_i(t). This model has the simplicity of linear models but also has the flexibility of allowing time-varying covariate effects. Since its introduction to the longitudinal data setting by Hoover, Rice, Wu and Yang (1998), many methods have been developed for estimation and inference of Model (1). See, for example, Wu, Chiang and Hoover (1998) and Fan and Zhang (2000) for the local polynomial kernel method; Huang, Wu and Zhou (2002, 2004) and Qu and Li (2006) for basis expansion and the spline methods, and Chiang, Rice and Wu (2001) for the smoothing spline method. One important problem not well studied in the literature is how to select significant variables in Model (1). The goal of this paper is to estimate β(t) nonparametrically and select relevant predictors x_k(t) with non-zero functional coefficient β_k(t), based on observations {(y_i(t_ij), x_i(t_ij)); i = 1,…,n, j = 1,…,J_i}.

Traditional methods for variable selection include hypothesis testing and using information criteria such as AIC and BIC. Recently, regularized estimation has received much attention as a way of performing variable selection for parametric regression models (see Bickel and Li, 2006 for a review). Existing regularization procedures for variable selection include LASSO (Tibshirani, 1996), smoothly clipped absolute deviation (SCAD) (Fan and Li, 2001) and their extensions (Yuan and Lin, 2006; Zou and Hastie, 2005; Zou, 2006). However, these regularized estimation procedures were developed for regression models where parameters are Euclidean and cannot be applied directly to the nonparametric varying-coefficients models where parameters are nonparametric smooth functions. Moreover, these procedures are studied mainly for independent data; an exception is Fan and Li (2004) where the SCAD penalty was used for variable selection in longitudinal data analysis. On the other hand, some recent results on regularization methods with diverging number of parameters (e.g., Fan & Peng, 2004; Huang, Horowitz & Ma, 2007) are related to nonparametric settings, but the model error due to function approximation has not been of concern in that literature.

Regularized estimation for selecting relevant variables in nonparametric settings has been developed in the general context of Smoothing Spline Analysis of Variance (ANOVA) Models. Lin and Zhang (2006) developed COSSO for component selection and smoothing in Smoothing Spline ANOVA. The COSSO method was extended by Zhang and Lin (2006) to nonparametric regression in exponential families, by Zhang (2006) to Support Vector Machines and by Leng and Zhang (2007) to hazard regression. Lin and Zhang (2006) studied rates of convergence of COSSO estimators; the rest of the papers focused on computational algorithms and did not provide theoretical results. Asymptotic distribution results have not been developed for COSSO and its extensions. Moreover, these approaches have focused on independent data, not on longitudinal data that we study in this paper.

In this paper we use the method of basis expansion for estimating smooth functions in varying coefficient models because of the simplicity of the basis expansion approach as illustrated in Huang et al. (2002). We extend the application of the SCAD penalty to a nonparametric setting. In addition to developing a computational algorithm, we study the asymptotic property of the estimator. We show that our procedure is consistent in variable selection, that is, the probability that it correctly selects the true model tends to one. Furthermore, we show that our procedure has an oracle property, that is, the asymptotic distribution of an estimated coefficient function is the same as that when it is known a priori which variables are in the model. Such results are new in nonparametric settings.

The rest of the paper is organized as follows. We first describe in Section 2 the regularized estimation procedure using basis expansion and the SCAD penalty. A computational algorithm and method of selecting the tuning parameter are given in Section 3. We then present theoretical results in Section 4, including the consistency in variable selection and the oracle property. Some simulation results are shown in Section 5. Section 6 illustrates the proposed method using two real data examples, a CD4 dataset and a microarray time-course gene expression dataset. Technical proofs are gathered in the Appendix.

2 Basis Function Expansion and Regularized Estimation Using the SCAD Penalty

Huang, Wu and Zhou (2002) proposed to estimate the unknown time-varying coefficient functions using basis expansion. Suppose that the coefficient β_k(t) can be approximated by a basis expansion ${\beta }_k(t)\approx {\sum }_{l=1}^{L_k}{\gamma }_{kl}{B}_{kl}(t)$, where L_k is the number of basis functions in approximating the function β_k(t). Model (1) becomes

$$y_i(t_{ij})\approx \sum _{k=1}^p\sum _{l=1}^{L_k}{\gamma }_{kl}{x}_i^{(k)}(t_{ij})B_{kl}(t_{ij})+{\epsilon }_i(t_{ij}).$$ (2)

The parameters γ_kl in the basis expansion can be estimated by minimizing

$$\frac{1}{n}\sum _{i=1}^n{w}_i\sum _{j=1}^{J_i}{\left\{y_i(t_{ij})-\sum _{k=1}^p\sum _{l=1}^{L_k}{\gamma }_{kl}{x}_i^{(k)}(t_{ij})B_{kl}(t_{ij})\right\}}^2,$$ (3)

where w_i are weights, taking the value 1 if we treat all observations equally or 1/n_i if we treat each subject equally. An estimate of β_k(t) is obtained by ${\widehat{\beta }}_k(t)\approx {\sum }_{l=1}^{L_k}{\widehat{\gamma }}_{kl}{B}_{kl}(t)$ where γ̂_kl are minimizers of (3). Various basis systems such as Fourier bases, polynomial bases and B-spline bases can be used in the basis expansion. Huang, Wu and Zhou (2002) studied consistency and rates of convergence of such estimators for general basis choices and Huang, Wu and Zhou (2004) studied asymptotic normality of the estimators when the basis functions B_kl are splines.

Now suppose some variables are not relevant in the regression so that the corresponding coefficient functions are zero functions. We introduce a regularization penalty to (3) so that these zero coefficient functions will be estimated as identically zero. To this end, it is convenient to rewrite (3) using function space notation. Let denote all functions that have the form ${\sum }_{l=1}^{L_k}{\gamma }_{kl}{B}_{kl}(t)$ for a given basis system {B_kl(t)}. Then (3) can be written as (1/n) ${\sum }_i{w}_i{\sum }_j{\{y_i(t_{ij})-{\sum }_k{g}_k(t_{ij})x_i^{(k)}(t_{ij})\}}^2$ where $g_k(t)={\sum }_{l=1}^{L_k}{\gamma }_{kl}{B}_{kl}(t)\in {\mathcal{G}}_k$. Let p_λ(u), u ≥0, be a nonnegative penalty function that depends on a penalty parameter λ. It is assumed that p_λ(0) = 0 and p_λ(u) is nondecreasing as a function of u. Let ||g_k|| denote the L₂-norm of the function g_k. We minimize over g_k ∈ the penalized criterion

$$\frac{1}{n}\sum _{i=1}^n{w}_i\sum _{j=1}^{J_i}{\{y_i(t_{ij})-\sum _{k=1}^p{g}_k(t_{ij})x_i^{(k)}(t_{ij})\}}^2+\sum _{k=1}^p{p}_{\lambda }(\parallel g_k\parallel ).$$ (4)

There are two sets of tuning parameters: the L_k’s control the smoothness of the coefficient functions and the λ governs variable selection or sparsity of the model. Selection of these parameters will be discussed in Section 4. In our implementation of the method, we use B-splines as the basis functions. Thus L_k = n_k + d + 1 where n_k is the number of interior knots for g_k and d is the degree of the spline. The interior knots of the splines can be either equally spaced or placed on the sample quantiles of the data so that there are about the same number of observations between any two adjacent knots. We use equally spaced knots for all numerical examples in this paper.

There are many ways to specify the penalty function p_λ(·). If p_λ(u) = λu, then the penalty term in (4) is similar to that in COSSO. However, we will use the SCAD penalty function by Fan and Li (2001), defined as

$$p_{\lambda }(u)=\{\begin{array}{ll}\lambda u\hfill & \text{if}0\le u\le \lambda ,\hfill \\ -\frac{(u^2-2a\lambda u+{\lambda }^2)}{2(a-1)}\hfill & \text{if}\lambda <u<a\lambda ,\hfill \\ \frac{(a+1){\lambda }^2}{2}\hfill & \text{if}u\ge a\lambda .\hfill \end{array}$$ (5)

The penalty function (5) is a quadratic spline function with two knots at λ and aλ, where a is another tuning parameter. Fan and Li (2001) suggested that a = 3.7 is a reasonable choice, which was also adopted in this paper. Use of the SCAD penalty allows us to obtain nice theoretical properties such as consistent variable selection and the oracle property for the proposed method.

For g_k(t) = Σ_l γ_klB_kl(t) ∈, its squared L₂-norm can be written as a quadratic form in γ_k = (γ_k1,…,γ_kLk)^T. Let R_k = (r_ij) _{Lk× Lk} be a matrix with entries $r_{ij}={\int }_{\mathcal{T}}{B}_{ki}(t)B_{kj}(t)dt$. Then $\parallel g_k\mid {\mid }^2={\mathit{\gamma }}_k^T{\mathit{R}}_k{\mathit{\gamma }}_k\triangleq \parallel {\mathit{\gamma }}_k\mid {\mid }_k^2$. Set $\mathit{\gamma }={({\mathit{\gamma }}_1^T,\dots ,{\mathit{\gamma }}_p^T)}^T$. The penalized weighted least squares criterion (4) can be written as

$$pl(\mathit{\gamma })=\frac{1}{n}\sum _{i=1}^n{w}_i\sum _{j=1}^{J_i}{\left\{(y_i(t_{ij})-\sum _{k=1}^p\sum _{l=1}^{L_k}{\gamma }_{kl}{x}_i^{(k)}(t_{ij})B_{kl}(t_{ij})\right\}}^2+\sum _{k=1}^p{p}_{\lambda }(\parallel {\mathit{\gamma }}_k\mid {\mid }_k).$$ (6)

To express the criterion function (6) using vectors and matrices, we define

$$\mathit{B}(t)=\left(\begin{array}{ccccccccc}{B}_{11}(t)& \dots & B_{1L_1}(t)& 0& \dots & 0& 0& \dots & 0\\ & ⋮& & & ⋮& & & ⋮& 0\\ 0& \dots & 0& 0& \dots & 0& B_{p1}(t)& \dots & B_{p{L}_p}(t)\end{array}\right),$$

U_i(t_ij) = (x_i(t_ij)^TB(t_ij))^T, U_i = (U_i(t_i1),…,U_i(t_iJi))^T, and $\mathit{U}={({\mathit{U}}_1^T,\dots ,{\mathit{U}}_n^T)}^T$. We also define y_i = (y_i(t_i1),…,y_i(t_iJi))^T and $\mathit{y}={({\mathit{y}}_1^T,\dots ,{\mathit{y}}_n^T)}^T$. The criterion function (6) can be rewritten as

$$\begin{array}{l}pl(\mathit{\gamma })=\frac{1}{n}\sum _{i=1}^n{w}_i{({\mathit{y}}_i-{\mathit{U}}_i\mathit{\gamma })}^T({\mathit{y}}_i-{\mathit{U}}_i\mathit{\gamma })+\sum _{k=1}^p{p}_{\lambda }(\parallel {\mathit{\gamma }}_k\mid {\mid }_k)\\ ={(\mathit{y}-\mathit{U}\gamma )}^T\mathit{W}(\mathit{y}-\mathit{U}\gamma )+\sum _{k=1}^p{p}_{\lambda }(\parallel {\mathit{\gamma }}_k\mid {\mid }_k),\end{array}$$ (7)

where W is a diagonal weight matrix with w_i repeated n_i times.

Remark

The penalized weighted least squares criterion here does not take into account the within-subject correlation typically present in the longitudinal data, because the correlation structure is usually unknown a priori. A popular method for dealing with within-subject correlation is to employ a working correlation structure as in the generalized estimation equations (GEE, Zeger and Liang, 1996). The idea of GEE can be easily adapted into our methodolgy. Specifically, the weight matrix W in (7) can be constructed using a working correlation structure, such as AR or compound symmetry. Actually the criteria (4) and (6) correspond to a working independence correlation structure. Using working independence correlation won’t affect the consistency of variable selection (Section 5). On the other hand, using an appropriate working correlation structure may yield more efficient function estimation if the structure is correctly specified.

3 Computational Algorithm

Because of non-differentiability of the penalized loss (7), the commonly-used gradient-based optimization method is not applicable. In this section, we develop an iterative algorithm using local quadratic approximation of the non-convex penalty p_λ(||γ_k||₂). Following Fan and Li (2001), in a neighborhood of a given positive u₀ ∈ ℝ⁺,

$$p_{\lambda }(u)\approx p_{\lambda }(u_0)+\frac{1}{2}\{p_{\lambda }^{\prime }(u_0)/u_0\}(u^2-u_0^2).$$ (8)

In our algorithm, a similar quadratic approximation is used by substituting u with ||γ_k||_k in (8), k = 1,…,p. Given an initial value of ${\mathit{\gamma }}_k^0$ with $\parallel {\mathit{\gamma }}_k^0\mid {\mid }_k>0$, p_λ(||γ_k||_k) can be approximated by a quadratic form

$$p_{\lambda }(\parallel {\mathit{\gamma }}_k^0\mid {\mid }_k)+\frac{1}{2}\{p_{\lambda }^{\prime }(\parallel {\mathit{\gamma }}_k^0\mid {\mid }_k)/\parallel {\mathit{\gamma }}_k^0\mid {\mid }_k\}\{{\mathit{\gamma }}_k^T{\mathit{R}}_k{\mathit{\gamma }}_k-{({\mathit{\gamma }}_k^0)}^T{\mathit{R}}_k{\mathit{\gamma }}_k^0\}.$$

As a consequence, removing an irrelevant constant, the penalized loss (7) becomes

$$pl(\mathit{\gamma })=\frac{1}{n}{(\mathit{y}-\mathit{U}\gamma )}^T\mathit{W}(\mathit{y}-\mathit{U}\gamma )+\frac{1}{2}{\mathit{\gamma }}^T{\mathit{\Omega }}_{\lambda }({\mathit{\gamma }}^0)\mathit{\gamma },$$ (9)

where ${\mathit{\Omega }}_{\lambda }({\mathit{\gamma }}^0)=\mathit{diag}\{(p_{\lambda }^{\prime }(\parallel {\mathit{\gamma }}_1^0\mid {\mid }_2)/\parallel {\mathit{\gamma }}_1^0\mid {\mid }_2){\mathit{R}}_1,\dots ,(p_{\lambda }^{\prime }(\parallel {\mathit{\gamma }}_K^0\mid {\mid }_2)/\parallel {\mathit{\gamma }}_K^0\mid {\mid }_2){\mathit{R}}_p\}$ where R_k are defined above equation (6). This is a quadratic form whose minimizer satisfies

$$\left\{{\mathit{U}}^T\mathit{WU}+\frac{n}{2}{\mathit{\Omega }}_{\lambda }({\mathit{\gamma }}^0)\right\}\mathit{\gamma }={\mathit{U}}^T\mathit{Wy}.$$ (10)

The above discussion leads to the following algorithm:

Step 1: Initialize γ = γ⁽¹⁾.

Step 2: Given γ^(m), update γ to γ^(m+1) by solving equation (10) where γ⁰ is set to be γ^(m)

Step 3: Iterate Step 2 until convergence of γ.

The initial estimation of γ in Step 1 can be obtained using a ridge regression, which substitutes p_λ(||γ_k||₂) in the penalized loss (7) with the quadratic function $\parallel {\mathit{\gamma }}_k\mid {\mid }_2^2$. The ridge regression has a closed form solution. At any iteration of Step 2, if some $\parallel {\mathit{\gamma }}_k^{(m)}\mid {\mid }_2$ is smaller than a cutoff value ε > 0, we set γ̂_k = 0 and treat x^(k)(t) as irrelevant. In our implementation of the algorithm ε is set to 10⁻³.

Remark

Due to the non-convexity of pl(γ), the algorithm sometimes may fail to converge and the search falls into an infinite loop surrounding several local minimizers. In this situation, we choose the local minimizer that gives the smallest value of pl(γ).

4 Selection of Tuning Parameters

To implement the proposed method, we need to choose the tuning parameters: L_k, k = 1,…,p and λ, where L_k controls the smoothness of β̂_k(t), while λ determines the variable selection. In section 5, we will show that our method is consistent in variable selection when these tuning parameters grow or decay at a proper rate with n. In practice, however, we need a data-driven procedure to select the tuning parameters. We propose to use the “leave-one-observation-out” cross-validation (CV) when the errors are independent and use the “leave-one-subject-out” cross-validation for correlated errors.

Below we develop some computational shortcuts using suitable approximations to prevent actual computation of leave-one-out estimates. Although the derivation of model selection criteria generally holds, we focus our presentation on the situation when L_k = L for all β_k(t), k = 1,…, p. To simplify our presentation, we also set W = I; extension to general W is straightforward, simply replacing y with W^1/2y and U with W^1/2U in the derivation below. At the convergence of our algorithm, the nonzero components are estimated as γ̂ = {U^TU + (n/2)Ω_λ(γ̂)}⁻¹U^Ty, which can be considered as the solution of the ridge regression

$$\parallel \mathit{y}-\mathit{U}\gamma \mid {\mid }_2^2+\frac{n}{2}{\mathit{\gamma }}^T{\mathit{\Omega }}_{\lambda }(\widehat{\mathit{\gamma }})\mathit{\gamma }.$$ (11)

The hat matrix of this ridge regression is denoted by H(L, λ) = U{U^TU+(n/2) Ω_λ(γ̂)}⁻¹U^T. The fitted y from this ridge regression satisfies ŷ = H(L, λ)y.

When the errors ε_ij are independent for different t_ij, j = 1,…,J_i, leave-one-observation-out CV is a reasonable choice for selecting tuning parameters. For the ridge regression (11), computation of leave-one-out estimates for CV can be avoided (Hastie and Tibshirani, 1993). The CV criterion has a closed form expression

$$C{V}_1(L,\lambda )=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^{J_i}\frac{\{{(y_i(t_{ij})-{\mathit{U}}_i^T(t_{ij})\widehat{\mathit{\gamma }}\}}^2}{{[1-{\{\mathit{H}(L,J)\}}_{(ij),(ij)}]}^2},$$ (12)

where {H(L; λ)}_(ij);(ij) is the diagonal element of H(L, λ) that corresponds to the observation at time t_ij. The corresponding GCV criterion, which replaces the diagonal elements in the above CV formula by their average, is

$$GC{V}_1(L,\lambda )=\frac{1}{n}\frac{\parallel \mathit{y}-\mathit{H}(L,\lambda )\mathit{y}\mid {\mid }_2^2}{{[1-tr\{\mathit{H}(L,\lambda )\}/n]}^2}.$$ (13)

When the errors ε_ij are not independent and the correlation structure of ε_i(t) is unknown, the leave-one-observation-out CV is unsuitable, we need to use leave-one-subject-out CV (Rice and Silverman, 1991; Hoover et al., 1998; and Huang et al., 2002). Let γ̂⁽⁻ⁱ⁾ be the solution of (6) after deleting the ith subject. The leave-one-subject-out CV can be written as

$$C{V}_2(L,\lambda )=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^{J_i}{\{y_i(t_{ij})-{\mathit{U}}_i^T(t_{ij}){\widehat{\mathit{\gamma }}}^{(-i)}\}}^2.$$ (14)

The CV₂(L, λ) can be viewed as an estimate of the true prediction error. However, computation of CV₂ is intensive, since it requires minimization of (6) n times. To overcome this difficulty, we propose to minimize a delete-one version of (11) instead of (6) in computing CV₂, where γ̂ in Ω_λ(γ̂) of (11) is the estimate based on the whole dataset. The approximated CV error is

$$AC{V}_2(L,\lambda )=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^{J_i}{\{y_i(t_{ij})-{\mathit{U}}_i^T(t_{ij}){\widehat{\mathit{\gamma }}}^{★(-i)}\}}^2=\sum _{i=1}^n\parallel {\mathit{y}}_i-{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}\mid {\mid }_2^2,$$ (15)

where γ̂^★(−i) is obtained by solving (11) instead of (6), deleting the ith subject. It turns out a computation shortcut is available for (15) as we discuss next.

The computation shortcut relies on the following “leave-one-subject-out” formula, the proof of which is given in Appendix A.

Lemma 1

Define ${\stackrel{\sim }{\mathit{y}}}^{(i)}=({\mathit{y}}_1^T,\dots ,{\mathit{y}}_{i-1}^T,{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)},{\mathit{y}}_{i+1}^T,\dots ,{\mathit{y}}_n^T)$, and let γ̃⁽ⁱ⁾ be the minimizer of (11) with y substituted by ỹ⁽ⁱ⁾. Then, U_iγ̂^★(−i) = U_iγ̂⁽ⁱ⁾.

Let A = {U^TU +(n/2)Ω_λ(γ̂)}⁻¹U^T. Partition A = (A₁,…,A_n) into blocks of columns with each block corresponding to one subject. Then

$$\widehat{\mathit{\gamma }}=\mathit{Ay}=\sum _{i=1}^n{\mathit{A}}_i{\mathit{y}}_i$$

and

$${\stackrel{\sim }{\mathit{\gamma }}}^{(i)}=\mathit{A}{\stackrel{\sim }{\mathit{y}}}^{(i)}=\sum _{k\ne i}{\mathit{A}}_k{\mathit{y}}_k+{\mathit{A}}_i{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}=\widehat{\mathit{\gamma }}-{\mathit{A}}_i{\mathit{y}}_i+{\mathit{A}}_i{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}.$$

As a consequence of Lemma 1,

$${\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}={\mathit{U}}_i{\stackrel{\sim }{\mathit{\gamma }}}^{(i)}={\mathit{U}}_i\widehat{\mathit{\gamma }}-{\mathit{U}}_i{\mathit{A}}_i({\mathit{y}}_i-{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}).$$

It follows that

$${\mathit{y}}_i-{\mathit{U}}_i{\widehat{\mathit{\gamma }}}^{★(-i)}={(\mathit{I}-{\mathit{U}}_i{\mathit{A}}_i)}^{-1}({\mathit{y}}_i-{\mathit{U}}_i\widehat{\mathit{\gamma }})={\{\mathit{I}-{\mathit{H}}_{ii}(L,\lambda )\}}^{-1}({\mathit{y}}_i-{\mathit{U}}_i\widehat{\mathit{\gamma }}).$$

Therefore,

$$AC{V}_2(L,\lambda )=\frac{1}{n}\sum _{i=1}^n\parallel {\{\mathit{I}-{\mathit{H}}_{ii}(L,\lambda )\}}^{-1}({\mathit{y}}_i-{\mathit{U}}_i\widehat{\mathit{\gamma }})\mid {\mid }_2^2.$$ (16)

where H_ii(L, λ) is the (i, i)-th diagonal block corresponding to observations for the i-th subject. Since J_i is usually not a big number, inversion of the J_i-dimensional matrices I − H_ii won’t create a computation burden in the evaluation of (16). Using (16) as an approximation of the leave-one-subject-out CV helps us avoid actual computation of many delete-one estimates. The quality of the approximation is illustrated in Figure 1 using a dataset from the simulation study reported in Section 6.

Figure 1.

Illustration of the approximation of CV scores for one simulated dataset for different values of L and λ. The ACV given in(16) approximates the leave-one-subject-out CV in (14) well.

5 Large Sample Properties

For parametric regression models, Fan and Li (2001) established the large sample properties of the SCAD penalized estimates. They showed that the SCAD penalty enables consistent variable selection and has an oracle property for parameter estimation | the estimates of the nonzero regression coefficients behave as if the subset of relevant variables is already known. We show that similar large sample properties hold for our proposed method for the varying coefficients models. Note that some care needs to be taken in developing an asymptotic theory for a nonparametric setting. For example, the rate of convergence of a nonparametric estimate is not root-n. The approximation error due to use of basis expansion needs also to be carefully studied. For simplicity we shall only present our asymptotic results for working independence case. Technical proofs of all asymptotic results are given in the Appendix.

We focus our asymptotic analysis for being a space of spline functions. Extension to general basis expansions can be obtained by combining the technical arguments in this paper with those in Huang et al. (2002). Define L_n = max_1≤k≤p L_k and ρ_n = max_1≤k≤p inf_g∈ ||β_k − g||L_∞. Thus, ρn characterizes the approximation error due to spline approximation. Assume that only s predictors are relevant in the Model (1). Without loss of generality, let β_k(t), k = 1,…,s, be the non-zero coefficient functions, and β_k(t) ≡ 0, k = s + 1,…,p.

We make the following technical assumptions:

(C1) The response and covariate processes (y_i(t); x_i(t)), i = 1,…, n, are i.i.d. as (y(t), X(t)), and the observation time points t_ij are i.i.d. from an unknown density f (t) on [0, T ], where f(t) are uniformly bounded away from infinity and zero.

(C2) The eigenvalues of the matrix E{X(t)X^T(t)} are uniformly bound away from zero and infinity for all t.

(C3) There exists a positive constant M₁ such that |X_k(t)| ≤ M₁ for all t and 1 ≤ k ≤ p.

(C4) There exists a positive constant M₂ such that E{ε²(t)} ≤ M₂ for all t.

(C5) lim sup_n(max_k L_k/min_k L_k) < ∞.

(C6) The process ε(t) can be decomposed as the sum of two independent stochastic processes, that is, ε(t) = ε⁽¹⁾(t) + ε⁽²⁾(t), where ε⁽¹⁾ is an arbitrary mean zero process and ε⁽²⁾ is a process of measurement errors that are independent at different time points and have mean zero and constant variance σ².

The same set of assumptions has been used in Huang et al. (2004). Our results can be extended to cases with deterministic observation times; the assumption on independent observation times in (C1) can also be relaxed (Remarks 3.1 and 3.2 of Huang et al., 2004).

Define $r_n=n^{-1}{[{\sum }_{i=1}^n{w}_i^{-2}\{L_n{J}_i+J_i(J_i-1)\}]}^{1/2}$. When w_i = 1/J_i, or w_i = 1/J_i with J_i uniformly bounded, we can show that r_n ≍ (L_n/n)^1/2

Theorem 1

Suppose Conditions (C1)–(C5) hold, lim_n→∞ ρ_n = 0 and

$$\underset{n\to \infty }{lim}{n}^{-1}{L}_{n}log(L_n)=0.$$ (17)

Then, with a choice of λ_n such that λ_n → 0 and λ_n/max{r_n, ρ_n} → ∞, we have

1. β̂_k = 0, k = s + 1,…,p, with probability approaching 1.

2. ||β̂_k − β_k||_L2 = O_p(max(r_n, ρ_n)), k = 1,…,s.

Part (a) of Theorem 1 says that the proposed penalized least squares method is consistent in variable selection, that is, it can identify the zero coefficient functions with probability tending to 1. Part (b) of the the theorem provides the rate of convergence in estimating the non-zero coefficient functions.

Corollary 1

Suppose that Conditions (C1)–(C4) hold and that β_k(t) have bounded second derivatives, k = 1,…,s, and β_k(t) = 0, k = s + 1,…,p. Let be a space of splines with degree no less than 1 and with L_k equally spaced interior knots, where L_k ≍ n^1/5, k = 1,…,p. If λ_n → 0 and n^2/5 λ_n → ∞, then,

1. β̂_k = 0, k = s + 1,…,p, with probability approaching 1.

2. ||β̂_k − β_k||_L2 = O_p(n^−2/5), k = 1,…, s.

Note that the rate of convergence given in part (b) of Corollary 1 is the optimal rate for nonparametric regression with independent, identically distributed data under the same smoothness assumption on the unknown function (Stone, 1982).

Now we consider the asymptotic variance of proposed estimate. Let β⁽¹⁾ = (β₁,…, β_s)^T denote the vector of non-zero coefficient functions and let β̂⁽¹⁾ = (β̂₁,…,β̂_s)^T its estimate obtained by minimizing (6). Let ${\widehat{\mathit{\gamma }}}^{(1)}={({\mathit{\gamma }}_1^T,\dots ,{\mathit{\gamma }}_s^T)}^T$. Let U⁽¹⁾ denote the selected columns of U corresponding to β⁽¹⁾ and similarly let ${\mathit{\Omega }}_{\lambda }^{(1)}$ denote the selected diagonal blocks of Ω_λ. By part (a) of Theorem 1, with probability tending to one, γ̂_k, k = s + 1,…,p, are vectors of 0’s. Thus the quadratic approximation (9) yields

$${\widehat{\mathit{\gamma }}}^{(1)}={\left\{{({\mathit{U}}^{(1)})}^T\mathit{W}{\mathit{U}}^{(1)}+\frac{n}{2}{\mathit{\Omega }}_{{\lambda }_n}^{(1)}({\widehat{\mathit{\gamma }}}^{(1)})\right\}}^{-1}{({\mathit{U}}^{(1)})}^T\mathit{Wy}.$$ (18)

Let ${\mathit{U}}_i^{(1)}$ be the rows of U⁽¹⁾ corresponding to subject i. Let

$$\begin{array}{l}{\mathit{H}}^{(1)}={({\mathit{U}}^{(1)})}^T\mathit{W}{\mathit{U}}^{(1)}+\frac{n}{2}{\mathit{\Omega }}_{\lambda }^{(1)}({\widehat{\mathit{\gamma }}}^{(1)})\\ =\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_i{\mathit{U}}_i^{(1)}+\frac{n}{2}{\mathit{\Omega }}_{\lambda }^{(1)}({\widehat{\mathit{\gamma }}}^{(1)}).\end{array}$$ (19)

The asymptotic variance of γ̂⁽¹⁾ is

$$\begin{array}{l}\text{Avar}({\widehat{\mathit{\gamma }}}^{(1)})={({\mathit{H}}^{(1)})}^{-1}{({\mathit{U}}^{(1)})}^T\mathit{W}cov(\mathit{y})\mathit{W}{\mathit{U}}^{(1)}{({\mathit{H}}^{(1)})}^{-1}\\ ={({\mathit{H}}^{(1)})}^{-1}\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_{i}cov({\mathit{y}}_i){\mathit{W}}_i{\mathit{U}}_i^{(1)}{({\mathit{H}}^{(1)})}^{-1},\end{array}$$ (20)

whose diagonal blocks give Avar(γ̂_k), k = 1,…,s. Since β̂⁽¹⁾(t) = (B⁽¹⁾)^T(t)γ̂⁽¹⁾ where B⁽¹⁾(t) are the first s rows of B(t), we have that Avar {β̂⁽¹⁾(t)} = (B⁽¹⁾)^T(t)Avar(γ̂⁽¹⁾)B⁽¹⁾(t), the diagonal elements of which are Avar{β̂_k(t)}, k = 1,…,s.

Let β̃_k(t) = E{β̂_k(t)} be the conditional mean given all observed X’s. Let β̃⁽¹⁾(t) = (β̃₁(t),… β̃_s(t)). For positive definite matrices A, let A ^1/2 denote the unique square root of A and let A^−1/2 = (A⁻¹)^1/2. Let Avar^*{β̂⁽¹⁾(t)} denote a modification of Avar{β̂⁽¹⁾(t)} by replacing Ω_λn in (19) with 0.

Theorem 2

Suppose Conditions (C1)–(C6) hold, lim_{n → ∞} ρ_n = 0, lim_{n→ ∞} L_n max_i J_i/n = 0, and (17) holds. Then, as n → ∞, {Avar^* (β̂⁽¹⁾(t))}^−1/2{β̂⁽¹⁾(t) − β̃⁽¹⁾(t)} → N(0, I) in distribution, and in particular, {Avar^* (β̂_k(t))}^−1/2{β̂_k(t) − β̃_k(t)} → N (0, 1), k = 1,…, s.

Note that Avar^*{β̂⁽¹⁾(t)} is exactly the same asymptotic variance of the non-penalized weighted least squares estimate using only those covariates corresponding to nonzero coefficient functions (Theorem 3, Huang et al., 2004). Theorem 2 implies that our penalized least squares estimate has the so-called oracle property — the joint asymptotic distribution of estimates of nonzero coefficient functions are the same as that of the non-penalized least squares estimate that utilizes the information about zero coefficient functions.

Theorem 2 can be used to construct pointwise asymptotic confidence intervals for β̃_k(t), k = 1,…,s. Theorem 4 and Corollary 1 of Huang et al. (2004) discuss how to bound the biases β̃_k(t) − β_k(t) and under what conditions the biases are negligible relative to variance. Those results are applicable to the current context. When constructing the confidence intervals, in order to improve finite sample performance, we still use (19) and the sandwich formula (20) to calculate the variance. In actual calculation, we need replace cov(y_i) in (19) by its estimate ${\mathit{e}}_i{\mathit{e}}_i^T$ where e_i is the residual vector for subject i.

6 Monte Carlo Simulation

We conducted simulation studies to assess the performance of the proposed procedure. In each simulation run, we generated a simple random sample of 200 subjects according to the model used in Huang et al. (2002), which assumes

$$Y(t_{ij})={\beta }_0(t_{ij})+\sum _{k=1}^{23}{\beta }_k(t_{ij})x_k(t_{ij})+\tau \epsilon (t_{ij}),i=1,\dots ,200,j=1,\dots ,J_i.$$

The first three variables x_i(t), i = 1,…,3, are the true relevant covariates, which are simulated the same way as in Huang et al. (2002): x₁(t) is sampled uniformly from [t/10, 2+ t/10] at any given time point t; x₂(t), conditioning on x₁(t), is Gaussian with mean zero and variance (1 + x₁(t))/(2 + x₁(t)); x₃(t), independent of x₁ and x₂, is a Bernoulli random variable with success rate 0.6. In addition to x_k, k = 1, 2, 3, 20 redundant variables x_k(t), k = 4,…,23, are simulated to demonstrate the performance of variable selection, where each x_k(t), independent of each other, is a random realization of a Gaussian process with covariance structure cov(x_k(t); x_k(s)) = 4 exp(− |t − s|). The random error ε(t) is given by Z(t) + E(t), where Z(t) has the same distribution as x_k(t), k = 4,…,23, and E(t) are independent measurement errors from N(0, 4) distribution at each time t. The parameter τ in the model is used to control the signal-to-noise ratio of the model. We consider models with τ = 1.00, 4.00, 5.66 and 8.00. The coefficients β_k(t), k = 0,…,3, corresponding to the constant term and the first three variables, are given by

$$\begin{array}{c}{\beta }_0(t)=15+20sin(\frac{\pi t}{60}),{\beta }_1(t)=2-3cos(\frac{\pi (t-25)}{15}),\\ {\beta }_2(t)=6-0.2t,{\beta }_3(t)=-4+\frac{{(20-t)}^3}{2000},\end{array}$$

(see solid lines of Figure 2) while the remaining coefficients, corresponding to the irrelevant variables, are given by β_k(t) = 0, k = 4,…, 23. The observation time points t_ij are generated following the same scheme as in Huang et al. (2002), where each subject has a set of “scheduled” time points {1,…,30}, and each scheduled time has a probability of 60% of being skipped. Then, the actual observation time t_ij is obtained by adding a random perturbation from Uniform(− 0.5, 0.5) to the nonskipped scheduled time.

Figure 2.

Simulation study for τ = 1.00. True (solid lines) and average of estimated (dashed lines) time-varying coefficients β₀(t) (a), β₁(t) (b), β₂(t) (c) and β₃(t) (+/− 1 point-wise SD) over 500 replications.

For each simulated dataset, the penalized least squares criterion (4) was minimized over a space of cubic splines with equally spaced knots. The number of knots and the tuning parameter λ were selected by minimizing the ACV₂ criterion in (15) as described in Section 4 (see Figure 1 for an example). We repeated the simulations 500 times. Table 1 shows the results of variable selection of our proposed procedure for four different models with different levels of noise variances. Clearly the level of noise had a great effect on how well the proposed method selected the exact correct models. When the noise level was low (τ = 1.00), the method selected the exact correct model in 79% of the replications. In addition, out of these 500 replications, the variables 1–3 were selected in each of the runs. In contrast, the number of times each of the 20 irrelevant variables were selected among the 500 simulation runs ranged from 3 to 12 with an average of 7.3 times (i.e., 1.46%). However, when the noise level was high (τ = 8.00), the method selected the exact true model in only 26% of the replications and the average number of times that the three relevant variables were selected was reduced to 2.24. However, it is interesting to note that the average number of times that the irrelevant variables were selected remained to be small (See Aver.I in Table 1). The simulations indicate that the proposed procedure indeed provides an effective method for selecting variables with time-varying coefficients.

Table 1.

Simulation results for models with different noise levels (τ) based on 500 replications. Perc.T: percentage of replications that the exact true model was selected; Aver.R: average of the numbers of the three relevant variables that were selected in the model; Aver.I: average of the numbers of the irrelevant variables that were selected in the model; Aver.Bias²: average of the squared biases; Aver.Var: average of the empirical variances; 95%Cov.Prob: 95% coverage probability based on the replications when the true models were selected. For Aver.Bias², Aver.Var and 95%Cov.Prob, all values were averaged over a grid of 300 points and 500 replications. Numbers in parentheses are results of the oracle estimators that assume the true models are known.

	Noise level (τ)
	1.00	4.00	5.66	8.00
	Variable selection
Perc.T	0.79	0.68	0.48	0.26
Aver.R	3.00	2.94	2.73	2.24
Aver.I	0.29	0.47	0.55	0.54
	Aver.Bias2
β0(t)	0.021 (0.023)	0.39 (0.37)	1.60 (0.42)	4.55 (0.42)
β1(t)	0.003 (0.004)	0.067 (0.067)	0.29 (0.076)	0.97 (0.073)
β2(t)	0.000089 (0.0002)	0.0022 (0.0022)	0.0047 (0.0043)	0.026 (0.0089)
β3(t)	0.00023 (0.0004)	0.0079 (0.0060)	0.061 (0.010)	0.88 (0.02)
	Aver.Var
β0(t)	0.98 (0.95)	14.11 (12.53)	28.29 (23.98)	49.09 (47.25)
β1(t)	0.12 (0.11)	1.95 (1.52)	4.08 (2.84)	7.15 (5.52)
β2(t)	0.049 (0.047)	0.70 (0.67)	1.37 (1.31)	3.06 (2.63)
β3(t)	0.15 (0.15)	2.13 (2.01)	4.70 (3.96)	9.98 (7.88)
	95%Cov.Prob
β0(t)	0.929	0.935	0.937	0.926
β1(t)	0.924	0.929	0.930	0.929
β2(t)	0.936	0.940	0.946	0.942
β3(t)	0.944	0.949	0.947	0.949

To examine how well the method estimates the coefficient functions, Figure 2 shows the estimates of the time-varying coefficients of β_k(t), k = 0, 1, 2, 3 for the model with τ = 1.00, indicating that the estimates fit the true function very well. Table 1 shows the averages of the squared biases and the empirical variances of the estimates of β_k(t), k = 0, 1, 2, 3 averaged over a grid of 300 points. In general, the biases were small, except for the estimates of β₀(t) when the noise level was high, in which case the proposed method did not select the variables well. Similarly, the variances of the estimates of β₀(t) were also relatively larger due to the fact that the magnitude of true β₀(t) is larger than the other three functions. As a comparison, we also calculated the biases and the empirical variances of the estimates based on the true models (i.e., the oracle estimators, see Table 1). Again, we observed that the both biases and empirical variances were very comparable to the oracle estimators when the noise level was low. When the noise level was high, due to the fact that the method tends to select fewer relevant variables, the biases and the empirical variances became large.

Finally, we examined the coverage probabilities of the theoretical point-wise confidence intervals based on Theorem 2. For each model, the empirical coverage probabilities averaged over a grid of 300 points were presented in Table 1 for β₀(t), β₁(t), β₂(t) and β₃(t), respectively, for nominal 95% point-wise confidence intervals. The slight under-coverage is caused by bias: the GCV criterion tried to balance the bias and variance. When we used the number of knots larger than that suggested by GCV (under-smoothing), the empirical coverage probabilities got closer to the nominal level. For example, for the model with τ = 1, the average empirical coverage probabilities became 0.946, 0.949, 0.942 and 0.947 when the number of interior knots were fixed at 5.

7 Applications to Real Datasets

To demonstrate the effectiveness of the proposed methods in selecting the variables with time-varying effects and in estimating the time-varying effects, we present in this Section results from the analysis of two real datasets, including a longitudinal AIDS dataset (Kaslow et al., 1987) and a repeated measurement microarray time course gene expression dataset (Spellman et al., 1998).

7.1 Application to AIDS data

Kaslow et al. (1987) reported a Multicenter AIDS Cohort Study to obtain the repeated measurements of physical examinations, laboratory results and CD4 cell counts and percentages of the homosexual men who became HIV-positive during 1984 and 1991. All individuals were scheduled to have their measurements made at semi-annual visits, but because many individuals missed some of their scheduled visits and the HIV infections happened randomly during the study, there were unequal numbers of repeated measurements and different measurement times per individual. As a subset of the cohort, our analysis focuses on the 283 homosexual men who become HIV-positive and aims to evaluate the effects of cigarette smoking, pre-HIV infection CD4 percentage, and age at HIV infection on the mean CD4-percentage after infection. In this subset, the number of repeated measurements per subject ranged from 1 to 14, with a median of 6 and a mean of 6.57. The number of distinct measurement time points was 59. Let t_ij be the time in years of the jth measurement of the ith individual after HIV infection, y_ij be the ith individual’s CD4 percentage at time t_ij, and x_i1 be the smoking status of the ith individual, taking a value of 1 or 0 for smoker or non-smoker. In addition, let x_i2 be the centered age at HIV infection for the ith individual and x_i3 be the centered pre-infection CD4 percentage. We assume the following VC model for y_ij,

$$y_{ij}={\beta }_0(t_{ij})+\sum _{k=1}^3{x}_{ik}{\beta }_k(t_{ik})+{\epsilon }_{ij},$$

where β₀(t) is the baseline CD4 percentage, representing the mean CD4 percentage t years after the infection for a non-smoker with average pre-infection CD4 percentage and average age at HIV infection, and β₁(t), β₂(t), β₃(t) measures the time-varying effects for cigarette smoking, age at HIV infection and pre-infection CD4 percentage, respectively, on the post-infection CD4 percentage at time t.

Our procedure identified two non-zero coefficients, β₀(t) and β₃(t), indicating that cigarette smoking and age of HIV infection do not have any effects on the poster-infection CDS4 percentage. Figure 3 shows the fitted coefficient functions (solid curves) and their 95% point-wise confidence intervals, indicating that the baseline CD4 percentage of the population decreases with time and pre-infection CD4 percentage appears to be positively associated with high post-infection CD4 percentage. These results agreed with those obtained by the local smoothing methods of Wu and Chiang (2002) and Fan and Zhang (2002) and by the basis function approximation method of Huang et al. (2002).

Figure 3.

Application to AIDS data. Estimated coefficient functions for intercept (a) and pre-infection CD4 percentages (b) (solid curves) and their point-wise 95% confidence intervals (dashed lines).

To assess our method in more challenging cases, we added 30 redundant variables, which are generated by randomly permuting the values of each predictor variable 10 times, and applied our method to the augmented dataset. We repeated the procedure 50 times. For the three observed covariates, “smoking status” was never selected, “age” was selected twice, and “pre-infection CD4 percentage” was always selected, wheras among the 30 redundant variables, two were selected twice, four were selected once, and the rest were never selected.

7.2 Application to yeast cell cycle gene expression data

The cell cycle is a tightly regulated set of processes by which cells grow, replicate their DNA, segregate their chromosomes, and divide into daughter cells. The cell cycle process is commonly divided into G1-S-G2-M stages, where the G1 stage stands for “GAP 1”, the S stage stands for “Synthesis”, during which DNA replication occurs, the G2 stage stands for “GAP 2” and the M stage stands for “mitosis”, which is when nuclear (chromosomes separate) and cytoplasmic (cytokinesis) division occur. Coordinate expression of genes whose products contribute to stage-specific functions is a key feature of the cell cycle (Simon et al., 2001). Transcription factors (TFs) have been identified that have roles in regulating transcription of a small set of yeast cell-cycle regulated genes; these include Mbp1, Swi4, Swi6, Mcm1, Fkh1, Fkh2, Ndd1, Swi5 and Ace2. Among these, Swi6, Swi4 and Mbp1 have been reported to function during the G1 stage; McM1, Ndd1 and Fkh1/2 have been reported to function during the G2 stage; and Ace2, Swi5 and Mcm1 have been reported to function during the M stage (Simon et al., 2001). However, it is not clear where these TFs regulate all cell cycle genes.

We apply our methods to identify the key transcription factors that play roles in the network of regulations from a set of gene expression measurements captured at different time points during the cell cycle. The dataset we use comes from the α-factor synchronized cultures from Spellman et al. (1998). They monitored genome-wide mRNA levels for 6178 yeast ORFs simultaneously using several different methods of synchronization including an α-factor-mediated G1 arrest, which covers approximately two cell-cycle periods with measurements at 7-min intervals for 119 mins with a total of 18 time points. Using a model-based approach, Luan and Li (2003) identified 297 cell-cycle-regulated genes based on the α-factor synchronization experiments. Let y_it denote the log-expression level for gene i at time point t during the cell cycle process, for i = 1, ···, 297 and t = 1, ···, 18. We then applied the mixture model approach (Chen et al., 2007; Wang et al., 2007) using the ChIP data of Lee et al. (2002) to derive the binding probabilities x_ik for these 297 cell-cycle-regulated genes for a total of 96 transcriptional factors with at least one nonzero binding probability in the 297 genes. We assume the following VC model to link the binding probabilities to the gene expression levels

$$y_{it}={\beta }_0(t)+\sum _{k=1}^{96}{\beta }_k(t)x_{ik}+{\epsilon }_{it},$$

where β_k(t) models the transcription effect of the kth TF on gene expression at time t during the cell cycle process. In this model, we assume that gene expression levels are independent conditioning on effects of the relevant transcription factors. In addition, for a given gene i, ∈_its are also assumed to be independent over different time points due to the cross-sectional nature of the experiments. Our goal is to identify the transcriptional factors that might be related to the expression patterns of these 297 cell-cycle-regulated genes. Since different transcriptional factors may regulate the gene expression at different time points during the cell cycle process, their effects on gene expression are expected to be time-dependent.

We applied our method using the GCV defined in (13) for selecting the tuning parameter in order to identify the TFs that affect the expression changes over time for these 297 cell-cycle-regulated genes in the α-factor synchronization experiment. Our procedure identified a total of 71 TFs that are related to yeast cell-cycle processes, including 19 of the 21 known and experimentally verified cell-cycle related TFs, all showing time-dependent effects of these TFs on gene expression levels. In addition, the effects followed similar trends between the two cell cycle periods. The other two TFs, CBF1 and GCN4, were not selected by our procedure. The minimum p-values over 18 time points from simple linear regressions are 0.06 and 0.14, respectively, also indicating that CBF1 and GCN4 were not related to expression variation over time. The 52 additional TFs that were selected by our procedure almost all showed estimated periodic transcriptional effects. The identified TFs include many pairs of cooperative or synergistic pairs of TFs involved in the yeast cell cycle process reported in the literature (Banerjee and Zhang, 2003; Tsai et al., 2005). Of these 52 TFs, 34 of them belong to the cooperative pairs of the TFs identified by Banerjee and Zhang (2003). Overall, the model can explain 43% of the total variations of the gene expression levels.

Figure 4 shows the estimated time-dependent transcriptional effects of nine of the experimentally verified TFs that are known to regulate cell-cycle-regulated genes at different stages. The top panel shows the transcriptional effects of three TFs, Swi4, Swi6 and Mbp1, which regulate gene expression at the G1 phase (Simon et al., 2001). The estimated transcriptional effects of these three TFs are quite similar with peak effects obtained at approximately the time points corresponding to the G1 phase of the cell cycle process. The middle panel shows the transcriptional effects of three TFs, Mcm1, Fkh2 and Ndd1, which regulate gene expression at the G2 phase (Simon et al., 2001). The estimated transcriptional effects of two of these three TFs, Fkh2 and Ndd1, are similar with peak effects obtained at approximately the time points corresponding to the G2 phase of the cell cycle process. The estimated effect of Mcm1 was somewhat different, however Mcm1 is also known to regulate gene expression at the M phase (Simon et al., 2001). The bottom panel of Figure 4 shows the transcriptional effects of three TFs, Swi5, Ace2 and Mcm1, which regulate gene expression at the M phase (Simon et al., 2001), indicating similar transcriptional effects of these three TFs with peak effects at approximately the time points corresponding to the M phase of the cell cycle. These plots demonstrated that our procedure can indeed identify the known cell-cycle-related TFs and point to the stages when these TFs function to regulate the expressions of cell cycle genes. As a comparison, only two of these nine TFs (SWI4 and FKH2) show certain periodic patterns in the observed gene expressions over the two cell-cycle periods, indicating that by only examining the expression profiles, we may miss many relevant transcription factors. Our method incorporates both time-course gene expression and ChIP-chip binding data and enables us to identify more transcription factors that are related to the cell-cycle process.

Figure 4.

Application to yeast cell cycle gene expression data. Estimated time-varying coefficients for selected transcription factors (TFs)(solid lines) and point-wise 90% confidence intervals (dotted lines). Top panel, TFs that regulate genes expressed at the G1 phase; middle panel, TFs that regulate genes expressed at the G2 phase; bottom panel, TFs that regulate genes expressed at the M phase.

Finally, to assess false identifications of the TFs that are related to a dynamic biological procedure, we randomly assigned the observed gene expression profiles to the 297 genes while keeping the values and the orders of the expression values for a given profile unchanged and applied our procedure again to the new datasets. The goal of such permutations was to scramble the response data and to create new expression profiles that do not depend on the motif binding probabilities in order to see to what extent the proposed procedure identifies the spurious variables. We repeated this procedure 50 times. Among the 50 runs, 5 runs selected 4 TFs, 1 run selected 3 TFs, 16 runs selected 2 TFs and the rest of the 28 runs did not select any of the TFs, indicating that our procedure indeed selects the relevant TFs with few false positives.

8 Discussion

We have proposed a regularized estimation procedure for nonparametric varying-coefficient models that can simultaneously perform variable selection and estimation of smooth coefficient functions. The procedure is applicable to both longitudinal and functional responses. Our method extends the SCAD penalty of Fan and Li (2001) from parametric settings to a nonparametric setting. We have shown that the proposed method is consistent in variable selection and has the oracle property that the asymptotic distribution of the nonparametric estimate of a smooth function with variable selection is the same as that when knowing which variables are in the model. This oracle property has been obtained for parametric settings, but our result seems to be the first of its kind in a nonparametric setting. A computation algorithm is developed based on local quadratic approximation to the criterion function. Simulation studies indicated that the proposed procedure is very effective in selecting relevant variables and in estimating the smooth regression coefficient functions. Results from application to the yeast cell cycle dataset indicates that the procedure can be effective in selecting the transcription factors that potentially play important roles in the regulation of gene expressions during the cell cycle process.

We need regularization for two purposes in our method: controlling the smoothness of the function estimates and variable selection. In our approach, the number of basis functions/the number of knots of splines is used to control the smoothness, while the SCAD parameter is used to set the threshold for variable selection. Thus, the two tasks of regularization are disentangled. To achieve finer control of smoothness, however, it is desirable to use the roughness penalty for introducing smoothness in function estimation, as in penalized splines or smoothing splines. Following the idea of COSSO, we could replace Σ_k p_λ(||g_k||) in (4) by λΣ_k ||g_k||_W where ||·||_W denotes a Sobolev norm, and then minimize the modified criterion over the corresponding Sobolev space. However, such an extension of COSSO is not very attractive compared with our method, because a single tuning parameter λ serves two different purposes of regularization. It is an interesting open question to develop a unified penalization method that does both smoothing and variable selection but also has the two tasks decoupled in some manner. Recent results on regularization methods with a diverging number of parameters (Fan and Peng, 2004; Huang et al., 2008) may provide theoretical insights into such a procedure.

As we explained in Section 2, our methodology applies when a working correlation structure is used to take into account within-subject correlation for longitudinal data. We only presented asymptotic results for the working independence case, however. Extension of these results to parametrically estimated correlation structures is relatively straightforward, using the arguments in Huang, Zhang and Zhou (2007); one needs to assume that the ratio of the smallest and largest eigenvalues of the weighting matrix W is bounded. Extension to nonparametrically estimated correlation structures (Wu and Pourahmadi, 2003; Huang et al., 2006; Huang, Liu and Liu, 2007) would be more challenging. Careful study of the asymptotics of our methodology for general working correlation structures is left for future research.

Appendix

A.1 Proof of Lemma 1

Use proof by contradiction. Suppose U_iγ̂^★(−i) ≠ U_iγ̃⁽ⁱ⁾. Denote (11) as pl_ridge(γ, y). Then,

$$\begin{array}{l}{pl}_{\mathit{ridge}}({\stackrel{\sim }{\mathit{\gamma }}}^{(i)},{\stackrel{\sim }{\mathit{y}}}^{(i)})=\frac{1}{n}\sum _{k=1}^n\parallel {\stackrel{\sim }{\mathit{y}}}_k^{(i)}-{\mathit{U}}_k{\stackrel{\sim }{\mathit{\gamma }}}_k^{(i)}\mid {\mid }_2^2+\frac{1}{2}{({\stackrel{\sim }{\mathit{\gamma }}}^{(i)})}^T{\mathrm{\sum }}_{\lambda }(\widehat{\mathit{\gamma }}){\stackrel{\sim }{\mathit{\gamma }}}^{(i)}\\ >\frac{1}{n}\sum _{k\ne i}\parallel {\mathit{y}}_k-{\mathit{U}}_k{\stackrel{\sim }{\mathit{\gamma }}}_k^{(i)}\mid {\mid }_2^2+\frac{1}{2}{({\stackrel{\sim }{\mathit{\gamma }}}^{(i)})}^T{\mathrm{\sum }}_{\lambda }(\widehat{\mathit{\gamma }}){\stackrel{\sim }{\mathit{\gamma }}}^{(i)}\\ \ge \frac{1}{n}\sum _{k\ne i}\parallel {\mathit{y}}_k-{\mathit{U}}_k{\widehat{\mathit{\gamma }}}_k^{\ast (-i)}\mid {\mid }_2^2+\frac{1}{2}{({\widehat{\mathit{\gamma }}}^{\ast (-i)})}^T{\mathrm{\sum }}_{\lambda }(\widehat{\mathit{\gamma }}){\widehat{\mathit{\gamma }}}^{\ast (-i)}\\ =p{l}_{\mathit{ridge}}({\widehat{\mathit{\gamma }}}^{\ast (-i)},{\stackrel{\sim }{\mathit{y}}}^{(i)}).\end{array}$$

This contradicts the fact that γ̃⁽ⁱ⁾ minimizes pl_ridge(γ̃⁽ⁱ⁾, ỹ⁽ⁱ⁾).

A.2 Useful lemmas

This subsection provides some lemmas that facilitate the proof of Theorems 1 and 2. Define ỹ_i = E(y_i|x_i), $\stackrel{\sim }{\mathit{\gamma }}={({\sum }_{i=1}^n{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{U}}_i)}^{-1}({\sum }_{i=1}^n{\mathit{U}}_i^T{\mathit{W}}_i{\stackrel{\sim }{\mathit{y}}}_i)$, and (β̃(t) = B(t)γ̃. Here β̃(t) can be regarded as the conditional mean of β̂(t). We have a bias-variance decomposition β̂−β = (β̂ − β̃) + (β̃ − β). Lemmas 3 and 5 quantify the rate of the convergence of ||β̃ − β||_L2 and ||β̂−β̃||_L2, respectively. The rate given in Lemma 5 on the variance term will be refined in part (b) of Theorem 1.

Lemma 2

Suppose that (17) holds. There exists an interval [M₃, M₄], 0 < M₃ < M₄ < ∞, such that all the eigenvalues of $n^{-1}{L}_n{\sum }_{i=1}^n{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{U}}_i$ fall in [M₃, M₄] with probability approaching 1 as n → ∞.

Lemma 3

Suppose that (17) holds. Then, ||β̃ − β||_L2 = O_P (ρ_n).

Lemmas 2 and 3 are from Lemmas A.3 and A.7 in Huang et al. (2004), respectively. The proofs are omitted.

Define $r_n=n^{-1}{[{\sum }_{i=1}^n{w}_i^{-2}\{L_n{J}_i+J_i(J_i-1)\}]}^{1/2}$. When w_i = 1/J_i, or w_i = 1/J_i with J_i uniformly bounded, we can show that r_n ≍ (L_n/n)^1/2.

Lemma 4

$n^{-1}{L}_n^{1/2}{sup}_{\mathit{v}}(\mid {\mathit{\epsilon }}^T\mathit{WUv}\mid /\mid \mathit{v}\mid )=O_P(r_n)$, where |v| = ||v|| _l2

Proof of Lemma 4

By the Cauchy-Schwarz inequality,

$$\underset{\mathit{v}}{sup}\left\{{\left(\frac{\mid {\mathit{\epsilon }}^T\mathit{WUv}\mid }{\mid \mathit{v}\mid }\right)}^2\right\}\le {\mathit{\epsilon }}^T\mathit{WU}{\mathit{U}}^T\mathit{W}\mathit{\epsilon }.$$ (21)

The expected value of the right hand side equals

$$\begin{array}{l}E({\mathit{\epsilon }}^T\mathit{WU}{\mathit{U}}^T\mathit{W}\mathit{\epsilon })=E\left\{\left(\sum _{i=1}^n{\mathit{\epsilon }}_i^T{\mathit{W}}_i{\mathit{U}}_i\right)\left(\sum _{i=1}^n{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{\epsilon }}_i\right)\right\}\\ =\sum _{i=1}^{n}E({\mathit{\epsilon }}_i^T{\mathit{W}}_i{\mathit{U}}_i{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{\epsilon }}_i).\end{array}$$ (22)

Note that

$$\begin{array}{l}E({\mathit{\epsilon }}_i^T{\mathit{W}}_i{\mathit{U}}_i{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{\epsilon }}_i)\\ =w_i^{-2}E\left[\sum _{k,l}{\{\sum _j{\epsilon }_i(t_{ij})X_{ik}(t_{ij})B_{kl}(t_{ij})\}}^2\right]\\ =w_i^{-2}\sum _{k,l}[\sum _{j}E\{{\epsilon }_i{(t_{ij})}^2{X}_{ik}{(t_{ij})}^2{B}_{kl}{(t_{ij})}^2\}\\ +\sum _{j\ne j^{\prime }}E\{{\epsilon }_i(t_{ij})X_{ik}(t_{ij})B_{kl}(t_{ij}){\epsilon }_i(t_{i{j}^{\prime }})X_{ik}(t_{i{j}^{\prime }})B_{kl}(t_{i{j}^{\prime }})\}].\end{array}$$

On the other hand,

$$\begin{array}{l}E\{B_{kl}(t_{ij})B_{kl}(t_{i{j}^{\prime }})\}=E\{B_{kl}(t_{ij})\}E\{B_{kl}(t_{i{j}^{\prime }})\}\\ \le {\{\underset{t\in [0,T]}{sup}f(t){\int }_0^T{B}_{kl}(t)dt\}}^2\le C_1^2{L}_k^{-2}\end{array}$$

and $E\{B_{kl}{(t)}^2\}\le E\{B_{kl}(t)\}\le C_1{L}_k^{-1}$. Therefore,

$$E({\mathit{\epsilon }}_i^T{\mathit{W}}_i{\mathit{U}}_i{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{\epsilon }}_i)\le C_2{w}_i^{-2}\{J_i+J_i(J_i-1)L_n^{-1}\}.$$ (23)

Combining (22) and (23) and using the Markov inequality, we obtain

$${\mathit{\epsilon }}^T\mathit{WU}{\mathit{U}}^T\mathit{W}\mathit{\epsilon }=O_P\left[\sum _{i=1}^n{w}_i^{-2}\{J_i+J_i(J_i-1)L_n^{-1}\}\right],$$

which together with (21) yields the desired result.

Lemma 5

Suppose that (17) holds and that λ_n, r_n, ρ_n → 0 and λ_n/ρ_n → ∞ as n → ∞. Then, ||β̂ − β̃||_L2 =O_P {r_n + (λ_nρ_n)^1/2}.

Proof of Lemma 5

Using properties of B-spline basis functions (see Section A.2 of Huang et al., 2004), we have

$$\parallel {\widehat{\mathit{\beta }}}_k-{\stackrel{\sim }{\mathit{\beta }}}_k\mid {\mid }_{L_2}^2=\parallel {\widehat{\mathit{\gamma }}}_k-{\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k^2\asymp L_k^{-1}\parallel {\widehat{\mathit{\gamma }}}_k-{\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_{l_2}^2.$$

Sum over k to obtain

$$\parallel \widehat{\mathit{\beta }}-\stackrel{\sim }{\mathit{\beta }}\mid {\mid }_{L_2}^2=\sum _{k=1}^p\parallel {\widehat{\mathit{\gamma }}}_k-{\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k^2\asymp L_n^{-1}\parallel \widehat{\mathit{\gamma }}-\stackrel{\sim }{\mathit{\gamma }}\mid {\mid }_{l_2}^2.$$

Let $\widehat{\mathit{\gamma }}-\stackrel{\sim }{\mathit{\gamma }}={\delta }_n{L}_n^{1/2}\mathit{v}$ with δ_n a scaler and v a vector satisfying ||v||_l2= 1. It follows that ||β̂−β̃||_L2 ≍ δ_n. We first show that δ_n = O_p(r_n + λ_n) and then refine this rate of convergence in the second step of the proof.

Let ε_i = (ε_i(t_i1), …, ε_i(t_iJi))^T, i = 1, …, n, and $\mathit{\epsilon }={({\mathit{\epsilon }}_1^T,\dots ,{\mathit{\epsilon }}_n^T)}^T$. By the definition of γ̃, U^TW(y − ε − U_γ̃) = 0. Since $\widehat{\mathit{\gamma }}=\stackrel{\sim }{\mathit{\gamma }}+{\delta }_n{L}_n^{1/2}\mathit{v}$,

$$\begin{array}{l}{(\mathit{y}-\mathit{U}\widehat{\mathit{\gamma }})}^T\mathit{W}(\mathit{y}-\mathit{U}\widehat{\mathit{\gamma }})-{(\mathit{y}-\mathit{U}\stackrel{\sim }{\mathit{\gamma }})}^T\mathit{W}(\mathit{y}-\mathit{U}\stackrel{\sim }{\mathit{\gamma }})\\ =-2{\delta }_n{L}_n^{1/2}{\mathit{v}}^T{\mathit{U}}^T\mathit{W}(\mathit{y}-\mathit{U}\stackrel{\sim }{\mathit{\gamma }})+{\delta }_n^2{L}_n{\mathit{v}}^T{\mathit{U}}^T\mathit{WUv}\\ =-2{\delta }_n{L}_n^{1/2}{\mathit{v}}^T{\mathit{U}}^T\mathit{W}\mathit{\epsilon }+{\delta }_n^2{L}_n{\mathit{v}}^T{\mathit{U}}^T\mathit{WUv}.\end{array}$$

Hence,

$$\begin{array}{l}pl(\widehat{\mathit{\gamma }})-pl(\stackrel{\sim }{\mathit{\gamma }})=-\frac{2{\delta }_n{L}_n^{1/2}}{n}{\mathit{v}}^T{\mathit{U}}^T\mathit{W}\mathit{\epsilon }+\frac{{\delta }_n^2{L}_n}{n}{\mathit{v}}^T{\mathit{U}}^T\mathit{WUv}\\ +\sum _{k=1}^p\{p_{{\lambda }_n}(\parallel {\widehat{\mathit{\gamma }}}_k\mid {\mid }_k)-p_{{\lambda }_n}(\parallel {\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k)\},\end{array}$$ (24)

where v_k’s are components of v when the latter is partitioned as γ̃.

By the definition of γ̂, pl(γ̂) − pl(γ̃) ≤ 0. According to Lemma 4,

$$\frac{L_n^{1/2}}{n}{\mathit{v}}^T{\mathit{U}}^T\mathit{W}\mathit{\epsilon }=\frac{L_n^{1/2}}{n}\sum _{i=1}^n{{\mathit{\epsilon }}_i}^T{\mathit{W}}_i{\mathit{U}}_i\mathit{v}=O_P(r_n).$$ (25)

By Lemma 2,

$$\frac{L_n}{n}{\mathit{v}}^T{\mathit{U}}^T\mathit{WUv}=\frac{L_n}{n}\sum _{i=1}^n{\mathit{v}}^T{\mathit{U}}_i^T{\mathit{W}}_i{\mathit{U}}_i\mathit{v}\ge M_3,$$ (26)

with probability approaching 1. Using the inequality |p_λ(a) − p_λ(b)| ≤ λ|a − b| we obtain

$$\sum _{k=1}^p\{p_{{\lambda }_n}(\parallel {\widehat{\mathit{\gamma }}}_k\mid {\mid }_k)-p_{{\lambda }_n}(\parallel {\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k)\}\ge \sum _{k=1}^p-{\lambda }_n\parallel {\widehat{\mathit{\gamma }}}_k-{\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k\asymp -{\lambda }_n{\delta }_n.$$ (27)

Therefore, $0\ge -O_p(r_n){\delta }_n+M_3{\delta }_n^2-{\lambda }_n{\delta }_n$ with probability approaching 1, which implies δ_n = O_p(r_n + λ_n).

Now we proceed to improve the obtained rate and show that δ_n = O_P {r_n + (λ_nρ_n)^1/2}. For k = 1, …, p, we have |||γ̂_k||_k − ||γ̃_k||_k| ≤ ||γ̂_k − γ̃_k||_k = o_p(1) and

$$\mid \parallel {\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k-\parallel {\mathit{\beta }}_k\mid {\mid }_{L_2}\mid \le \mid \parallel {\stackrel{\sim }{\mathit{\beta }}}_k\mid {\mid }_{L_2}-\parallel {\mathit{\beta }}_k\mid {\mid }_{L_2}\mid \le \parallel {\stackrel{\sim }{\mathit{\beta }}}_k-{\mathit{\beta }}_k\mid {\mid }_{L_2}=O_p({\rho }_n)=o_P(1).$$ (28)

It follows that ||γ̂_k||_k → ||β_k||_L2 and ||γ̃_k||_k → ||β_k||_L2with probability. Since ||β_k||_L2 > 0 for k = 1, …, s and λ_n → 0, we have that, with probability approaching 1, ||γ̂_k||_k > aλ_n, ||γ̃_k||_k > aλ_n, k = 1, …, s, On the other hand, ||β_k||_L2 = 0 for k = s + 1, …, p, so (28) implies that ||γ̃_k||_k = O_P (ρ_n). Since λ_n/ρ_n → ∞, we have that with probability approaching 1, ||γ̃_k||_k < λ_n, k = s + 1, …, p. Consequently, by the definition of p_λ(·), P {p_λn(||γ̃_k||H_k) =p_λn(||γ̂||_Hk)} → 1, k = 1, … s, and P{p_λn(||γ̃_k||_Hk) = λ_n(||γ̃_k|| _Hk} → 1, k = s + 1, …, p.

Therefore,

$$\sum _{k=1}^p\{p_{{\lambda }_n}(\parallel {\widehat{\mathit{\gamma }}}_k\mid {\mid }_k)-p_{{\lambda }_n}(\parallel {\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k)\}={\lambda }_n\sum _{k=s+1}^p\parallel {\stackrel{\sim }{\mathit{\gamma }}}_k\mid {\mid }_k\ge -O_P({\lambda }_n{\rho }_n).$$

This result combined with (24), (25) and (26) imply that, with probability approaching 1,

$$pl(\widehat{\mathit{\gamma }})-pl(\stackrel{\sim }{\mathit{\gamma }})\ge -O_p(r_n){\delta }_n+M_3{\delta }_n^2-O_p({\lambda }_n{\rho }_n),$$

which in turn implies δ_n = O_P {r_n + (λ_nρ_n)^1/2}. The proof is complete.

A.3 Proof of Theorem 1

To prove part (a) of Theorem 1, we use proof by contradiction. Suppose that for n large enough, there exists a constant η > 0 such that with probability at least η, there exists a k₀ > s such that β̂ _k0(t) ≠ 0. Then ||γ̂_k0||_k0 = ||β̂ _k0 (·)||_L2 > 0. Let γ̂^* be a vector constructed by replacing γ̂_k0 with 0 in γ̂. Then,

$$pl(\widehat{\mathit{\gamma }})-pl({\widehat{\mathit{\gamma }}}^{\ast })=\frac{1}{n}(\parallel \mathit{y}-\mathit{U}\widehat{\mathit{\gamma }}\mid {\mid }^2-\parallel \mathit{y}-\mathit{U}{\widehat{\mathit{\gamma }}}^{\ast }\mid {\mid }^2)+p_{{\lambda }_n}(\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{k_0}).$$ (29)

By Lemma 5 and the fact that β_k0 (t) = 0, ||γ̂_k0||_k0 = ||β̂_k0|| = O_p{r_n + (λ_nρ_n)^1/2}. Since λ_n/max(r_n, ρ_n) → ∞, we have that λ_n > ||γ̂_k0||_k0 and thus p_λn (||γ̂_k0||_k0) = λ_n||γ̂_k0||_k0 with probability approaching 1. To analyze the first term on the right hand side of (29), note that

$$\begin{array}{l}\parallel \mathit{y}-\mathit{U}\widehat{\mathit{\gamma }}\mid {\mid }^2-\parallel \mathit{y}-\mathit{U}{\widehat{\mathit{\gamma }}}^{\ast }\mid {\mid }^2\\ \ge -{(\mathit{y}-\mathit{U}{\widehat{\mathit{\gamma }}}^{\ast })}^T\mathit{WU}(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })\\ =-2{(\mathit{y}-\mathit{U}\stackrel{\sim }{\mathit{\gamma }})}^T\mathit{WU}(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })-2{(\stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })}^T{\mathit{U}}^T\mathit{WU}(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast }).\end{array}$$ (30)

By the Cauchy-Schwarz inequality and Lemma 2,

$$\begin{array}{l}{(\stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })}^T({\mathit{U}}^T\mathit{WU})(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })\\ \le {\{{(\stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })}^T({\mathit{U}}^T\mathit{WU})(\stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })\}}^{1/2}{\{{(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })}^T({\mathit{U}}^T\mathit{WU})(\stackrel{\sim }{\mathit{\gamma }}-{\mathit{\gamma }}^{\ast })\}}^{1/2}\\ \le M_4\frac{n}{L_n}\parallel \stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast }\mid {\mid }_{l_2}\parallel \widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast }\mid {\mid }_{l_2}.\end{array}$$

It follows from the triangle inequality and Lemmas 3 and 5 that,

$$\parallel \stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast }\mid {\mid }_{l_2}\le \parallel \stackrel{\sim }{\mathit{\gamma }}-\widehat{\mathit{\gamma }}\mid {\mid }_{l_2}+\parallel {\stackrel{\sim }{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}=O_p[L_n^{1/2}\{r_n+{({\lambda }_n{\rho }_n)}^{1/2}+{\rho }_n\}].$$

Hence,

$${(\stackrel{\sim }{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })}^T({\mathit{U}}^T\mathit{WU})(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })=O_p[L_n^{1/2}\{r_n+{({\lambda }_n{\rho }_n)}^{1/2}+{\rho }_n\}]\frac{n}{L_n}\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}.$$ (31)

Lemma 4 implies that

$$\mid {(\mathit{y}-\mathit{U}\stackrel{\sim }{\mathit{\gamma }})}^T\mathit{WU}(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })\mid =\mid {\mathit{\epsilon }}^T\mathit{WU}(\widehat{\mathit{\gamma }}-{\widehat{\mathit{\gamma }}}^{\ast })\mid =O_P\left(\frac{n{r}_n}{L_n^{1/2}}\right)\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}.$$ (32)

Combining (29)–(32), we get

$$\begin{array}{l}pl(\widehat{\mathit{\gamma }})-pl({\widehat{\mathit{\gamma }}}^{\ast })\\ \ge -O_p\left(\frac{r_n}{L_n^{1/2}}\right)\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}-O_p\left(\frac{r_n+{({\lambda }_n{\rho }_n)}^{1/2}+{\rho }_n}{L_n^{1/2}}\right)\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}+\frac{{\lambda }_n}{L_n^{1/2}}\parallel {\widehat{\mathit{\gamma }}}_{k_0}\mid {\mid }_{l_2}.\end{array}$$ (33)

Note that on the right hand side of (33), the third term dominates both the first term and the second term, since λ_n/max(r_n, ρ_n) → ∞. This contradicts the fact that pl(γ̂) − pl(γ^*) ≤ 0. We have thus proved part (a).

To prove part (b), write β = ((β⁽¹⁾)^T, (β⁽²⁾)^T)^T, where β⁽¹⁾ = (β₁, …, β_s)^T and β⁽²⁾ = (β_s+1, …, β_p)^T, and write γ = ((γ⁽¹⁾)^T, (γ⁽²⁾)^T)^T, where ${\mathit{\gamma }}^{(1)}={({\mathit{\gamma }}_1^T,\dots ,{\mathit{\gamma }}_s^T)}^T$ and ${\mathit{\gamma }}^{(2)}={({\mathit{\gamma }}_{s+1}^T,\dots ,{\mathit{\gamma }}_p^T)}^T$. Similarly, write ${\mathit{U}}_i=({\mathit{U}}_i^{(1)},{\mathit{U}}_i^{(2)})$ and U = (U⁽¹⁾; U⁽²⁾). Define the oracle version of γ̃,

$$\begin{array}{l}{\stackrel{\sim }{\mathit{\gamma }}}_{\mathit{oracle}}=arg{min}_{\mathit{\gamma }={({\mathit{\gamma }}^{{(1)}^T},{\mathbf{0}}^T)}^T}\frac{1}{n}\sum _{i=1}^n{w}_i{({\stackrel{\sim }{\mathit{y}}}_i-{\mathit{U}}_i\mathit{\gamma })}^T({\stackrel{\sim }{\mathit{y}}}_i-{\mathit{U}}_i\mathit{\gamma })\\ =\left(\begin{array}{c}{({\sum }_i{\mathit{U}}_i^{{(1)}^T}{\mathit{W}}_i{\mathit{U}}_i^{(1)})}^{-1}({\sum }_i{\mathit{U}}_i^{{(1)}^T}{\mathit{W}}_i{\stackrel{\sim }{\mathit{y}}}_i)\\ \mathbf{0}\end{array}\right),\end{array}$$ (34)

which is obtained as if the information of the nonzero components were given; the corresponding vector of coefficient functions is denoted as β̃_oracle. Note that the true vector of coefficient functions is β = ((β ⁽¹⁾)^T, 0^T)^T. By Lemma 3, ||β̃_oracle − β||_L2 = O_P (ρ_n) = o_P (1). By Lemmas 3 and 5, ||β̂ − β||_L2 = o_P (1). Thus, with probabiltiy approaching one, ||β̃_k,oracle||_L2 → ||β_k||_L2 > 0 and ||β̂_k||_L2 → ||β_k||_L2 > 0, for k = 1, …, s. On the other hand, by the definition of β̃_oracle, ||β̃_k;oracle||_L2 = 0 and by part (a) of the theorem, with probability approaching one, ||β̂_k||_L2 = 0, for k = s + 1, …, p. Consequently,

$$\sum _{k=1}^p{p}_{{\lambda }_n}(\parallel {\stackrel{\sim }{\beta }}_{k,\mathit{oracle}}\mid {\mid }_{L_2})=\sum _{k=1}^p{p}_{{\lambda }_n}(\parallel {\widehat{\beta }}_k\mid {\mid }_{L_2}),$$ (35)

with probability approaching 1. ¿From part (a) of the theorem, with probability approaching 1, γ̂ = ((γ̂⁽¹⁾)^T, 0^T)^T. Let $\widehat{\mathit{\gamma }}-{\stackrel{\sim }{\mathit{\gamma }}}_{\mathit{oracle}}={\delta }_n{L}_n^{1/2}\mathit{v}$ with v = ((v⁽¹⁾)^T, 0^T)^T and ||v⁽¹⁾||_l2 = 1. Then, $\parallel \widehat{\mathit{\beta }}-{\stackrel{\sim }{\mathit{\beta }}}_{\mathit{oracle}}\mid {\mid }_{L_2}\asymp L_n^{-1}\parallel \widehat{\mathit{\gamma }}-{\stackrel{\sim }{\mathit{\gamma }}}_{\mathit{oracle}}\mid {\mid }_{l_2}={\delta }_n$. By (24) and (35), and the argument following (24),

$$\begin{array}{l}0\ge pl(\widehat{\mathit{\gamma }})-pl({\stackrel{\sim }{\mathit{\gamma }}}_{\mathit{oracle}})=-\frac{2{\delta }_n{L}_n^{1/2}}{n}{\mathit{v}}^T{({\mathit{U}}^{(1)})}^T\mathit{W}\mathit{\epsilon }+\frac{{\delta }_n^2{L}_n}{n}{\mathit{v}}^T{({\mathit{U}}^{(1)})}^T\mathit{W}{\mathit{U}}^{(1)}\mathit{v}\\ \ge -O_p(r_n){\delta }_n+M_3{\delta }_n^2.\end{array}$$

Hence, ||β̂ − β̃||_L2 ≍ δ_n = Op(r_n), which together with ||β_oracle − −β||_L2 = o_P (ρ_n) implies that ||β̂ − β||_L² = O_p(ρ_n + r_n). The desired result follows.

A.4 Proof of Corollary 1

By Theorem 6.27 in Schumaker (1981), ${inf}_{g\in {\mathcal{G}}_k}\parallel {\beta }_k-g\mid {\mid }_{L_{\infty }}=O(L_k^{-2})$, and thus ${\rho }_n=O({\sum }_{k=1}^p{L}_k^{-2})=O(L_n^{-2})$. Moreover, r_n ≍ (L_n/n)^1/2. Hence, the convergence rate is $\parallel {\widehat{\mathit{\beta }}}_k-{\mathit{\beta }}_k\mid {\mid }_{L_2}=O_P\{{(L_n/n)}^{1/2}+L_n^{-2}\}$. When L_n ≍ n^1/5, ${(L_n/n)}^{1/2}+L_n^{-2}\asymp n^{-2/5}$. The proof is complete.

A.5 Proof of Theorem 2

According to the proof of Lemma 5, with probability approaching one, ||γ̃_k||_k > aλ_n, ||γ̂ _k||_k > aλ_n, and thus p_λn(||γ̃_k||_k) = p_λn(||γ̂_k||_k), k = 1, …, s. By Theorem 1, with probability approaching 1, γ̂ = ((γ̂⁽¹⁾)^T, 0^T)^T is a local minimizer of pl(γ). Note that pl(γ⁽¹⁾) is quadratic in γ⁽¹⁾ = (γ₁, …, γ_s)^T when ||γ_k||_k > aλ_n, k = 1, …, s. Therefore, ∂pl(γ)/∂γ⁽¹⁾|_{γ⁽¹⁾=γ̂⁽¹⁾,γ⁽²⁾=0} = 0, which implies

$${\widehat{\mathit{\gamma }}}^{(1)}={\left(\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_i{\mathit{U}}_i^{(1)}\right)}^{-1}\left(\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_i{\mathit{Y}}_i\right).$$

Let

$${\stackrel{\sim }{\mathit{\gamma }}}^{(1)}={\left(\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_i{\mathit{U}}_i^{(1)}\right)}^{-1}\left(\sum _{i=1}^n{({\mathit{U}}_i^{(1)})}^T{\mathit{W}}_{i}E({\mathit{Y}}_i\mid {\mathit{x}}_i)\right).$$

Let Avar^*(γ̂⁽¹⁾) be a modification of Avar(γ̂⁽¹⁾) given in (20) by replacing Ω_λn in (19) with 0. Applying Theorem V.8 of Petrov (1975) as in the proof of Theorem 4.1 of Huang (2003) and using the arguments in the proof of Lemma A.8 of Huang et al. (2004), we obtain that, for any vector c_n with dimension ${\sum }_{k=1}^s{L}_s$ and whose components are not all zero,

$${\{{\mathit{c}}_n^T{\text{Avar}}^{\ast }({\widehat{\mathit{\gamma }}}^{(1)}){\mathit{c}}_n\}}^{-1/2}{\mathit{c}}_n^T({\widehat{\mathit{\gamma }}}^{(1)}-{\stackrel{\sim }{\mathit{\gamma }}}^{(1)})\to N(0,1)in\mathit{distribution}.$$

For any s-vector a_n whose components are not all zero, choosing c_n = (B⁽¹⁾(t))^Ta_n yields

$${[{\mathit{a}}_n^T{\text{Avar}}^{\ast }\{{\widehat{\mathit{\beta }}}^{(1)}(t)\}{\mathit{a}}_n]}^{-1/2}{\mathit{a}}_n^T\{{\widehat{\mathit{\beta }}}^{(1)}(t)-{\stackrel{\sim }{\mathit{\beta }}}^{(1)}(t)\}\to N(0,1)in\mathit{distribution},$$

which in turn yields the desired result.