VARIABLE SELECTION IN NONPARAMETRIC ADDITIVE MODELS

Abstract

We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is “small” relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansions with B-spline bases. With this approximation, the problem of component selection becomes that of selecting the groups of coefficients in the expansion. We apply the adaptive group Lasso to select nonzero components, using the group Lasso to obtain an initial estimator and reduce the dimension of the problem. We give conditions under which the group Lasso selects a model whose number of components is comparable with the underlying model, and the adaptive group Lasso selects the nonzero components correctly with probability approaching one as the sample size increases and achieves the optimal rate of convergence. The results of Monte Carlo experiments show that the adaptive group Lasso procedure works well with samples of moderate size. A data example is used to illustrate the application of the proposed method.

1. Introduction

Let (Y_i, X_i), i = 1, …, n, be random vectors that are independently and identically distributed as (Y, X), where Y is a response variable and X = (X₁, …, X_p)′ is a p-dimensional covariate vector. Consider the nonparametric additive model

$$Y_i=\mu +{\displaystyle \sum _{j=1}^p{f}_j(X_{\mathit{\text{ij}}})+{\epsilon }_i},$$ (1)

where µ is an intercept term, X_ij is the jth component of X_i, the f_j’s are unknown functions, and ε_i is an unobserved random variable with mean zero and finite variance σ². Suppose that some of the additive components f_j are zero. The problem addressed in this paper is to distinguish the nonzero components from the zero components and estimate the nonzero components. We allow the possibility that p is larger than the sample size n, which we represent by letting p increase as n increases. We propose a penalized method for variable selection in (1) and show that the proposed method can correctly select the nonzero components with high probability.

There has been much work on penalized methods for variable selection and estimation with high-dimensional data. Methods that have been proposed include the bridge estimator [Frank and Friedman (1993), Huang, Horowitz and Ma (2008)]; least absolute shrinkage and selection operator or Lasso [Tibshirani (1996)], the smoothly clipped absolute deviation (SCAD) penalty [Fan and Li (2001), Fan and Peng (2004)], and the minimum concave penalty [Zhang (2010)]. Much progress has been made in understanding the statistical properties of these methods. In particular, many authors have studied the variable selection, estimation and prediction properties of the Lasso in high-dimensional settings. See, for example, Meinshausen and Bühlmann (2006), Zhao and Yu (2006), Zou (2006), Bunea, Tsybakov and Wegkamp (2007), Meinshausen and Yu (2009), Huang, Ma and Zhang (2008), van de Geer (2008) and Zhang and Huang (2008), among others. All these authors assume a linear or other parametric model. In many applications, however, there is little a priori justification for assuming that the effects of covariates take a linear form or belong to any other known, finite-dimensional parametric family. For example, in studies of economic development, the effects of covariates on the growth of gross domestic product can be nonlinear. Similarly, there is evidence of nonlinearity in the gene expression data used in the empirical example in Section 5.

There is a large body of literature on estimation in nonparametric additive models. For example, Stone (1985, 1986) showed that additive spline estimators achieve the same optimal rate of convergence for a general fixed p as for p = 1. Horowitz and Mammen (2004) and Horowitz, Klemelä and Mammen (2006) showed that if p is fixed and mild regularity conditions hold, then oracle-efficient estimates of the f_j’s can be obtained by a two-step procedure. Here, oracle efficiency means that the estimator of each f_j has the same asymptotic distribution that it would have if all the other f_j’s were known. However, these papers do not discuss variable selection in nonparametric additive models.

Antoniadis and Fan (2001) proposed a group SCAD approach for regularization in wavelets approximation. Zhang et al. (2004) and Lin and Zhang (2006) have investigated the use of penalization methods in smoothing spline ANOVA with a fixed number of covariates. Zhang et al. (2004) used a Lasso-type penalty but did not investigate model-selection consistency. Lin and Zhang (2006) proposed the component selection and smoothing operator (COSSO) method for model selection and estimation in multivariate nonparametric regression models. For fixed p, they showed that the COSSO estimator in the additive model converges at the rate n^−d/(2d+1), where d is the order of smoothness of the components. They also showed that, in the special case of a tensor product design, the COSSO correctly selects the nonzero additive components with high probability. Zhang and Lin (2006) considered the COSSO for nonparametric regression in exponential families.

Meier, van de Geer and Bühlmann (2009) treat variable selection in a nonparametric additive model in which the numbers of zero and nonzero f_j’s may both be larger than n. They propose a penalized least-squares estimator for variable selection and estimation. They give conditions under which, with probability approaching 1, their procedure selects a set of f_j’s containing all the additive components whose distance from zero in a certain metric exceeds a specified threshold. However, they do not establish model-selection consistency of their procedure. Even asymptotically, the selected set may be larger than the set of nonzero f_j’s. Moreover, they impose a compatibility condition that relates the levels and smoothness of the f_j’s. The compatibility condition does not have a straightforward, intuitive interpretation and, as they point out, cannot be checked empirically. Ravikumar et al. (2009) proposed a penalized approach for variable selection in nonparametric additive models. In their approach, the penalty is imposed on the ℓ₂ norm of the nonparametric components, as well as the mean value of the components to ensure identifiability. In their theoretical results, they require that the eigenvalues of a “design matrix” be bounded away from zero and infinity, where the “design matrix” is formed from the basis functions for the nonzero components. It is not clear whether this condition holds in general, especially when the number of nonzero components diverges with n. Another critical condition required in the results of Ravikumar et al. (2009) is similar to the irrepresentable condition of Zhao and Yu (2006). It is not clear for what type of basis functions this condition is satisfied. We do not require such a condition in our results on selection consistency of the adaptive group Lasso.

Several other recent papers have also considered variable selection in nonparametric models. For example, Wang, Chen and Li (2007) and Wang and Xia (2008) considered the use of group Lasso and SCAD methods for model selection and estimation in varying coefficient models with a fixed number of coefficients and covariates. Bach (2007) applies what amounts to the group Lasso to a nonparametric additive model with a fixed number of covariates. He established model selection consistency under conditions that are considerably more complicated than the ones we require for a possibly diverging number of covariates.

In this paper, we propose to use the adaptive group Lasso for variable selection in (1) based on a spline approximation to the nonparametric components. With this approximation, each nonparametric component is represented by a linear combination of spline basis functions. Consequently, the problem of component selection becomes that of selecting the groups of coefficients in the linear combinations. It is natural to apply the group Lasso method, since it is desirable to take into the grouping structure in the approximating model. To achieve model selection consistency, we apply the group Lasso iteratively as follows. First, we use the group Lasso to obtain an initial estimator and reduce the dimension of the problem. Then we use the adaptive group Lasso to select the final set of nonparametric components. The adaptive group Lasso is a simple generalization of the adaptive Lasso [Zou (2006)] to the method of the group Lasso [Yuan and Lin (2006)]. However, here we apply this approach to nonparametric additive modeling.

We assume that the number of nonzero f_j’s is fixed. This enables us to achieve model selection consistency under simple assumptions that are easy to interpret. We do not have to impose compatibility or irrepresentable conditions, nor do we need to assume conditions on the eigenvalues of certain matrices formed from the spline basis functions. We show that the group Lasso selects a model whose number of components is bounded with probability approaching one by a constant that is independent of the sample size. Then using the group Lasso result as the initial estimator, the adaptive group Lasso selects the correct model with probability approaching 1 and achieves the optimal rate of convergence for nonparametric estimation of an additive model.

The remainder of the paper is organized as follows. Section 2 describes the group Lasso and the adaptive group Lasso for variable selection in nonparametric additive models. Section 3 presents the asymptotic properties of these methods in “large p, small n” settings. Section 4 presents the results of simulation studies to evaluate the finite-sample performance of these methods. Section 5 provides an illustrative application, and Section 6 includes concluding remarks. Proofs of the results stated in Section 3 are given in the Appendix.

2. Adaptive group Lasso in nonparametric additive models

We describe a two-step approach that uses the group Lasso for variable selection based on a spline representation of each component in additive models. In the first step, we use the standard group Lasso to achieve an initial reduction of the dimension in the model and obtain an initial estimator of the nonparametric components. In the second step, we use the adaptive group Lasso to achieve consistent selection.

Suppose that each X_j takes values in [a, b] where a < b are finite numbers. To ensure unique identification of the f_j’s, we assume that E f_j(X_j) = 0, 1 ≤ j ≤ p. Let a = ξ₀ < ξ₁ < ⋯ < ξ_K < ξ_K+1 = b be a partition of [a, b] into K subintervals I_Kt = [ξ_t, ξ_t+1), t = 0, …, K − 1, and I_KK = [ξ_K, ξ_K+1], where K ≡ K_n = n^υ with 0 < υ < 0.5 is a positive integer such that max_1≤k≤K+1 |ξ_k − ξ_k−1| = O(n^−υ). Let 𝒮_n be the space of polynomial splines of degree l ≥ 1 consisting of functions s satisfying: (i) the restriction of s to I_Kt is a polynomial of degree l for 1 ≤ t ≤ K; (ii) for l ≥ 2 and 0 ≤ l ′ ≤ l − 2, s is l′ times continuously differentiable on [a, b]. This definition is phrased after Stone (1985), which is a descriptive version of Schumaker (1981), page 108, Definition 4.1.

There exists a normalized B-spline basis {ϕ_k, 1 ≤ k ≤ m_n} for 𝒮_n, where m_n ≡ K_n + l [Schumaker (1981)]. Thus, for any f_nj ∈ 𝒮_n, we can write

$$f_{\mathit{\text{nj}}}(x)={\displaystyle \sum _{k=1}^{m_n}{\beta }_{\mathit{\text{jk}}}{\varphi }_k(x),1\le j\le p.}$$ (2)

Under suitable smoothness assumptions, the f_j’s can be well approximated by functions in 𝒮_n. Accordingly, the variable selection method described in this paper is based on the representation (2).

Let ${\Vert \mathbf{a}\Vert }_2\equiv {({\displaystyle {\sum }_{j=1}^m{|a_j|}^2})}^{1/2}$ denote the ℓ₂ norm of any vector a ∈ ℝ^m. Let β_nj = (β_j1, …, β_jmn)′ and ${\mathit{\beta }}_n=({\mathit{\beta }}_{n1}^{\prime },\dots ,{\mathit{\beta }}_{\mathit{\text{np}}}^{\prime })\prime$ . Let w_n = (w_n1, …, w_np)′ be a given vector of weights, where 0 ≤ w_nj ≤ ∞, 1 ≤ j ≤ p. Consider the penalized least squares criterion

$$L_n(\mu ,{\mathit{\beta }}_n)={{\displaystyle \sum _{i=1}^n\left[Y_i-\mu -{\displaystyle \sum _{j=1}^p{\displaystyle \sum _{k=1}^{m_n}{\beta }_{\mathit{\text{jk}}}{\varphi }_k(X_{\mathit{\text{ij}}}})}\right]}}^2+{\lambda }_n{\displaystyle \sum _{j=1}^p{w}_{\mathit{\text{nj}}}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2,}$$ (3)

where λ_n is a penalty parameter. We study the estimators that minimize L_n(µ, β_n) subject to the constraints

$${\displaystyle \sum _{i=1}^n{\displaystyle \sum _{k=1}^{m_n}{\beta }_{\mathit{\text{jk}}}{\varphi }_k(X_{\mathit{\text{ij}}})=0,1\le j\le p.}}$$ (4)

These centering constraints are sample analogs of the identifying restriction E f_j(X_j) = 0, 1 ≤ j ≤ p. We can convert (3) and (4) to an unconstrained optimization problem by centering the response and the basis functions. Let

$${\overline{\varphi }}_{\mathit{\text{jk}}}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\varphi }_k(X_{\mathit{\text{ij}}}),{\psi }_{\mathit{\text{jk}}}(x)}={\varphi }_k(x)-{\overline{\varphi }}_{\mathit{\text{jk}}}.$$ (5)

For simplicity and without causing confusion, we simply write ψ_k(x) = ψ_jk(x). Define

$$Z_{\mathit{\text{ij}}}=({\psi }_1(X_{\mathit{\text{ij}}}),\dots ,{\psi }_{m_n}(X_{\mathit{\text{ij}}}))\prime .$$

So, Z_ij consists of values of the (centered) basis functions at the ith observation of the jth covariate. Let Z_j = (Z_1j, …, Z_nj)′ be the n × m_n “design” matrix corresponding to the jth covariate. The total “design” matrix is Z = (Z₁, …, Z_p). Let Y = (Y₁ − Y̅, …, Y_n − Y̅)′. With this notation, we can write

$$L_n({\mathit{\beta }}_n;\lambda )={\Vert \mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n\Vert }_2^2+{\lambda }_n{\displaystyle \sum _{j=1}^p{w}_{\mathit{\text{nj}}}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2.}$$ (6)

Here, we have dropped µ in the argument of L_n. With the centering, µ̂ = Y̅. Then minimizing (3) subject to (4) is equivalent to minimizing (6) with respect to β_n, but the centering constraints are not needed for (6).

We now describe the two-step approach to component selection in the nonparametric additive model (1).

Step 1. Compute the group Lasso estimator. Let

$$L_{n1}({\mathit{\beta }}_n,{\lambda }_{n1})={\Vert \mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n\Vert }_2^2+{\lambda }_{n1}{\displaystyle \sum _{j=1}^p{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2.}$$

This objective function is the special case of (6) that is obtained by setting w_nj = 1, 1 ≤ j ≤ p. The group Lasso estimator is β̃_n ≡ β̃_n(λ_n1) = arg min_βn L_n1(β_n; λ_n1).

Step 2. Use the group Lasso estimator β̃_n to obtain the weights by setting

$$w_{\mathit{\text{nj}}}=\{\begin{array}{cc}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2^{-1},\hfill & \text{\hspace{1em}if }{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2>0,\hfill \\ \infty ,\hfill & \text{\hspace{1em}if }{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2=0.\hfill \end{array}$$

The adaptive group Lasso objective function is

$$L_{n2}({\mathit{\beta }}_n;{\lambda }_{n2})={\Vert \mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n\Vert }_2^2+{\lambda }_{n2}{\displaystyle \sum _{j=1}^p{w}_{\mathit{\text{nj}}}}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2.$$

Here, we define 0 · ∞ = 0. Thus, the components not selected by the group Lasso are not included in Step 2. The adaptive group Lasso estimator is β̂_n ≡ β̂_n(λ_n2) = arg min_βn L_n2(β_n; λ_n2). Finally, the adaptive group Lasso estimators of µ and f_j are

$${\widehat{\mu }}_n=\overline{Y}\equiv n^{-1}{\displaystyle \sum _{i=1}^n{Y}_i,{\widehat{f}}_{\mathit{\text{nj}}}(x)}={\displaystyle \sum _{k=1}^{m_n}{\widehat{\beta }}_{\mathit{\text{jk}}}{\psi }_k(x),1\le j\le p.}$$

3. Main results

This section presents our results on the asymptotic properties of the estimators defined in Steps 1 and 2 of Section 2.

Let k be a nonnegative integer, and let α ∈ (0, 1] be such that d = k + α > 0.5. Let ℱ be the class of functions f on [0, 1] whose kth derivative f^(k) exists and satisfies a Lipschitz condition of order α:

$$|f^{(k)}(s)-f^{(k)}(t)|\le C|s-t{|}^{\alpha }\text{\hspace{1em}\hspace{1em}for }s,t\in [a,b].$$

In (1), without loss of generality, suppose that the first q components are nonzero, that is, f_j(x) ≠ 0, 1 ≤ j ≤ q, but f_j(x) ≡ 0, q + 1 ≤ j ≤ p. Let A₁ = {1, …, q} and A₀ = {q + 1, …, p}. Define ${\Vert f\Vert }_2={[{\displaystyle {\int }_a^b{f}^2(x)\mathit{\text{dx}}}]}^{1/2}$ for any function f, whenever the integral exists.

We make the following assumptions.

(A1) The number of nonzero components q is fixed and there is a constant c_f > 0 such that min_1≤j≤q‖f_j‖₂ ≥ c_f.

(A2) The random variables ε₁, …, ε_n are independent and identically distributed with Eε_i = 0 and Var(ε_i) = σ². Furthermore, their tail probabilities satisfy P(|ε_i| > x) ≤ K exp(−Cx²), i = 1, …, n, for all x ≥ 0 and for constants C and K.

(A3) E f_j(X_j) = 0 and f_j ∈ ℱ, j = 1, …, q.

(A4) The covariate vector X has a continuous density and there exist constants C₁ and C₂ such that the density function g_j of X_j satisfies 0 < C₁ ≤ g_j (x) ≤ C₂ < ∞ on [a, b] for every 1 ≤ j ≤ p.

We note that (A1), (A3) and (A4) are standard conditions for nonparametric additive models. They would be needed to estimate the nonzero additive components at the optimal ℓ₂ rate of convergence on [a, b], even if q were fixed and known. Only (A2) strengthens the assumptions needed for nonparametric estimation of a nonparametric additive model. While condition (A1) is reasonable in most applications, it would be interesting to relax this condition and investigate the case when the number of nonzero components can also increase with the sample size. The only technical reason that we assume this condition is related to Lemma 3 given in the Appendix, which is concerned with the properties of the smallest and largest eigenvalues of the “design matrix” formed from the spline basis functions. If this lemma can be extended to the case of a divergent number of components, then (A1) can be relaxed. However, it is clear that there needs to be restriction on the number of nonzero components to ensure model identification.

3.1. Estimation consistency of the group Lasso

In this section, we consider the selection and estimation properties of the group Lasso estimator. Define Ã₁ = {j : ‖β̃_nj‖₂ ≠ 0, 1 ≤ j ≤ p}. Let |A| denote the cardinality of any set A ⊆ {1, …, p}.

THEOREM 1. Suppose that (A1) to (A4) hold and ${\lambda }_{n1}\ge C\sqrt{n\text{ log}({\mathit{\text{pm}}}_n)}$ for a sufficiently large constant C.

1. With probability converging to 1, |Ã₁| ≤ M₁|A₁| = M_1q for a finite constant M₁ > 1.

2. If $m_n^2\text{ log}({\mathit{\text{pm}}}_n)/n\to 0\mathit{\text{ and }}({\lambda }_{n1}^2{m}_n)/n^2\to 0\mathit{\text{ as }}n\to \infty$ , then all the nonzero β_nj, 1 ≤ j ≤ q, are selected with probability converging to one.

3. $${\displaystyle \sum _{j=1}^p{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2=O_p\left(\frac{m_n^2\text{ log}({\mathit{\text{pm}}}_n)}{n}\right)+O_p\left(\frac{m_n}{n}\right)+O\left(\frac{1}{m_n^{2d-1}}\right)+O\left(\frac{4m_n^2{\lambda }_{n1}^2}{n^2}\right)}.$$

Part (i) of Theorem 1 says that, with probability approaching 1, the group Lasso selects a model whose dimension is a constant multiple of the number of nonzero additive components f_j, regardless of the number of additive components that are zero. Part (ii) implies that every nonzero coefficient will be selected with high probability. Part (iii) shows that the difference between the coefficients in the spline representation of the nonparametric functions in (1) and their estimators converges to zero in probability. The rate of convergence is determined by four terms: the stochastic error in estimating the nonparametric components (the first term) and the intercept µ (the second term), the spline approximation error (the third term) and the bias due to penalization (the fourth term).

Let ${\tilde{f}}_{\mathit{\text{nj}}}(x)={\displaystyle {\sum }_{j=1}^{m_n}{\tilde{\beta }}_{\mathit{\text{jk}}}\psi (x)}$ , 1 ≤ j ≤ p. The following theorem is a consequence of Theorem 1.

THEOREM 2. Suppose that (A1) to (A4) hold and that ${\lambda }_{n1}\ge C\sqrt{n\text{ log}({\mathit{\text{pm}}}_n)}$ for a sufficiently large constant C. Then:

1. Let Ã_f = {j : ‖f̃_nj‖₂ > 0, 1 ≤ j ≤ p}. There is a constant M₁ > 1 such that, with probability converging to 1, |Ã_f| ≤ M_1q.

2. If (m_n log(pm_n))/n → 0 and $({\lambda }_{n1}^2{m}_n)/n^2\to 0\mathit{\text{ as }}n\to \infty$ , then all the nonzero additive components f_j, 1 ≤ j ≤ q, are selected with probability converging to one.

3. $${\Vert {\tilde{f}}_{\mathit{\text{nj}}}-f_j\Vert }_2^2=O_p\left(\frac{m_n\text{log}({\mathit{\text{pm}}}_n)}{n}\right)+O_p\left(\frac{1}{n}\right)+O\left(\frac{1}{m_n^{2d}}\right)+O\left(\frac{4m_n{\lambda }_{n1}^2}{n^2}\right),j\in {\tilde{A}}_2,$$

   where Ã₂ = A₁ ∪ Ã₁.

Thus, under the conditions of Theorem 2, the group Lasso selects all the nonzero additive components with high probability. Part (iii) of the theorem gives the rate of convergence of the group Lasso estimator of the nonparametric components.

For any two sequences {a_n, b_n, n = 1, 2,…}, we write a_n ≍ b_n if there are constants 0 < c₁ < c₂ < ∞ such that c₁ ≤ a_n/b_n ≤ c₂ for all n sufficiently large.

We now state a useful corollary of Theorem 2.

COROLLARY 1. Suppose that (A1) to (A4) hold. If ${\lambda }_{n1}\asymp \sqrt{n\text{log}({\mathit{\text{pm}}}_n)}$ and m_n ≍ n^1/(2d+1), then:

1. If n^−2d/(2d+1) log(p) → 0 as n → ∞, then with probability converging to one, all the nonzero components f_j, 1 ≤ j ≤ q, are selected and the number of selected components is no more than M₁q.

2. $${\Vert {\tilde{f}}_{\mathit{\text{nj}}}-f_j\Vert }_2^2=O_p(n^{-2d/(2d+1)}\text{log}({\mathit{\text{pm}}}_n)),j\in {\tilde{A}}_2.$$

For the λ_n1 and m_n given in Corollary 1, the number of zero components can be as large as exp(o(n^2d/(2d+1))). For example, if each f_j has continuous second derivative (d = 2), then it is exp(o(n^4/5)), which can be much larger than n.

3.2. Selection consistency of the adaptive group Lasso

We now consider the properties of the adaptive group Lasso. We first state a general result concerning the selection consistency of the adaptive group Lasso, assuming an initial consistent estimator is available. We then apply to the case when the group Lasso is used as the initial estimator. We make the following assumptions.

(B1) The initial estimators β̃_nj are r_n-consistent at zero:

$$r_n\underset{j\in A_0}{\text{max }}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2=O_P(1),r_n\to \infty ,$$

and there exists a constant c_b > 0 such that

$$\mathrm{P}(\underset{j\in A_1}{\text{min}}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2\ge c_b{b}_{n1})\to 1,$$

where b_n1 = min_j∈A1‖β_nj‖₂.

(B2) Let q be the number of nonzero components and s_n = p − q be the number of zero components. Suppose that:

1. $$\frac{m_n}{n^{1/2}}+\frac{{\lambda }_{n2}{m}_n^{1/4}}{n}=o(1),$$
2. $$\frac{n^{1/2}{\text{log}}^{1/2}(s_n{m}_n)}{{\lambda }_{n2}{r}_n}+\frac{n}{{\lambda }_{n2}{r}_n{m}_n^{(2d+1)/2}}=o(1).$$

We state condition (B1) for a general initial estimator, to highlight the point that the availability of an r_n-consistent estimator at zero is crucial for the adaptive group Lasso to be selection consistent. In other words, any initial estimator satisfying (B1) will ensure that the adaptive group Lasso (based on this initial estimator) is selection consistent, provided that certain regularity conditions are satisfied. We note that it follows immediately from Theorem 1 that the group Lasso estimator satisfies (B1). We will come back to this point below.

For ${\widehat{\mathit{\beta }}}_n\equiv ({\widehat{\mathit{\beta }}}_{n1}^{\prime },\dots ,{\widehat{\mathit{\beta }}}_{\mathit{\text{np}}}^{\prime })\prime \text{ and }{\mathit{\beta }}_n\equiv ({\mathit{\beta }}_{n1}^{\prime },\dots ,{\mathit{\beta }}_{\mathit{\text{np}}}^{\prime })\prime$ , we say β̂_n =₀ β_n if sgn₀(‖β̂_nj‖) = sgn₀(‖β_nj‖), 1 ≤ j ≤ p, where sgn₀(|x|) = 1 if |x| > 0 and = 0 if |x| = 0.

THEOREM 3. Suppose that conditions (B1), (B2) and (A1)–(A4) hold. Then:

1. $$\mathrm{P}({\widehat{\mathit{\beta }}}_n{=}_0{\mathit{\beta }}_n)\to 1.$$
2. $${\displaystyle \sum _{j=1}^q{\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2=O_p\left(\frac{m_n^2}{n}\right)+O_p\left(\frac{m_n}{n}\right)+O\left(\frac{1}{m_n^{2d-1}}\right)+O\left(\frac{4m_n^2{\lambda }_{n2}^2}{n^2}\right)}.$$

This theorem is concerned with the selection and estimation properties of the adaptive group Lasso in terms of β̂_n. The following theorem states the results in terms of the estimators of the nonparametric components.

THEOREM 4. Suppose that conditions (B1), (B2) and (A1)–(A4) hold. Then:

1. $$\mathrm{P}({\Vert {\widehat{f}}_{\mathit{\text{nj}}}\Vert }_2>0,j\in A_1\mathit{\text{ and }}{\Vert {\widehat{f}}_{\mathit{\text{nj}}}\Vert }_2=0,j\in A_0)\to 1.$$
2. $${\displaystyle \sum _{j=1}^q{\Vert {\widehat{f}}_{\mathit{\text{nj}}}-f_j\Vert }_2^2=O_p\left(\frac{m_n}{n}\right)+O_p\left(\frac{1}{n}\right)+O\left(\frac{1}{m_n^{2d}}\right)+O\left(\frac{4m_n{\lambda }_{n2}^2}{n^2}\right)}.$$

Part (i) of this theorem states that the adaptive group Lasso can consistently distinguish nonzero components from zero components. Part (ii) gives an upper bound on the rate of convergence of the estimator.

We now apply the above results to our proposed procedure described in Section 2, in which we first obtain the the group Lasso estimator and then use it as the initial estimator in the adaptive group Lasso.

By Theorem 1, if ${\lambda }_{n1}\asymp \sqrt{n\text{log}({\mathit{\text{pm}}}_n)}$ and m_n ≍ n^1/(2d+1) for d ≥ 1, then the group Lasso estimator satisfies (B1) with $r_n\asymp n^{d/(2d+1)}/\sqrt{\text{log}({\mathit{\text{pm}}}_n)}$ . In this case, (B2) simplifies to

$$\frac{{\lambda }_{n2}}{n^{(8d+3)/(8d+4)}}=o(1)\text{\hspace{1em}and\hspace{1em}}\frac{n^{1/(4d+2)}{\text{log}}^{1/2}({\mathit{\text{pm}}}_n)}{{\lambda }_{n2}}=o(1).$$ (7)

We summarize the above discussion in the following corollary.

COROLLARY 2. Let the group Lasso estimator β̃_n ≡ β̃_n(λ_n1) with ${\lambda }_{n1}\asymp \sqrt{n\text{log}({\mathit{\text{pm}}}_n)}$ and m_n ≍ n^1/(2d+1) be the initial estimator in the adaptive group Lasso. Suppose that the conditions of Theorem 1 hold. If λ_n2 ≤ O(n^1/2) and satisfies (7), then the adaptive group Lasso consistently selects the nonzero components in (1), that is, part (i) of Theorem 4 holds. In addition,

$${\displaystyle \sum _{j=1}^q{\Vert {\widehat{f}}_{\mathit{\text{nj}}}-f_j\Vert }_2^2=O_p(n^{-2d/(2d+1)})}.$$

This corollary follows directly from Theorems 1 and 4. The largest λ_n2 allowed is λ_n2 = O(n^1/2). With this λ_n2, the first equation in (6) is satisfied. Substitute it into the second equation in (6), we obtain p = exp(o(n^2d/(2d+1))), which is the largest p permitted and can be larger than n. Thus, under the conditions of this corollary, our proposed adaptive group Lasso estimator using the group Lasso as the initial estimator is selection consistent and achieves optimal rate of convergence even when p is larger than n. Following model selection, oracle-efficient, asymptotically normal estimators of the nonzero components can be obtained by using existing methods.

4. Simulation studies

We use simulation to evaluate the performance of the adaptive group Lasso with regard to variable selection. The generating model is

$$y_i=f(x_i)+{\epsilon }_i\equiv {\displaystyle \sum _{j=1}^p{f}_j(x_{\mathit{\text{ij}}})+{\epsilon }_i,i=1,\dots ,n.}$$ (8)

Since p can be larger than n, we consider two ways to select the penalty parameter, the BIC [Schwarz (1978)] and the EBIC [Chen and Chen (2008, 2009)]. The BIC is defined as

$$\mathit{\text{BIC}}(\lambda )=\text{log}({\text{RSS}}_{\lambda })+{\mathit{\text{df}}}_{\lambda }·\frac{\text{log }n}{n}.$$

Here, RSS_λ is the residual sum of squares for a given λ, and the degrees of freedom df_λ = q̂_λm_n, where q̂_λ is the number of nonzero estimated components for the given λ. The EBIC is defined as

$$\mathit{\text{EBIC}}(\lambda )=\text{log}({\text{RSS}}_{\lambda })+{\mathit{\text{df}}}_{\lambda }·\frac{\text{log }n}{n}+\nu ·{\mathit{\text{df}}}_{\lambda }·\frac{\text{log }p}{n},$$

where 0 ≤ ν ≤ 1 is a constant. We use ν = 0.5.

We have also considered two other possible ways of defining df: (a) using the trace of a linear smoother based on a quadratic approximation; (b) using the number of estimated nonzero components. We have decided to use the definition given above based on the results from our simulations. We note that the df for the group Lasso of Yuan and Lin (2006) requires an initial (least squares) estimator, which is not available when p > n. Thus, their df is not applicable to our problem.

In our simulation example, we compare the adaptive group Lasso with the group Lasso and ordinary Lasso. Here, the ordinary Lasso estimator is defined as the value that minimizes

$${\Vert \mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n\Vert }_2^2+{\lambda }_n{\displaystyle \sum _{j=1}^p{\displaystyle \sum _{k=1}^{m_n}|{\beta }_{\mathit{\text{jk}}}|.}}$$

This simple application of the Lasso does not take into account the grouping structure in the spline expansions of the components. The group Lasso and the adaptive group Lasso estimates are computed using the algorithm proposed by Yuan and Lin (2006). The ordinary Lasso estimates are computed using the Lars algorithms [Efron et al. (2004)]. The group Lasso is used as the initial estimate for the adaptive group Lasso.

We also compare the results from the nonparametric additive modeling with those from the standard linear regression model with Lasso. We note that this is not a fair comparison because the generating model is highly nonlinear. Our purpose is to illustrate that it is necessary to use nonparametric models when the underlying model deviates substantially from linear models in the context of variable selection with high-dimensional data and that model misspecification can lead to bad selection results.

EXAMPLE 1. We generate data from the model

$$y_i=f(x_i)+{\epsilon }_i\equiv {\displaystyle \sum _{j=1}^p{f}_j(x_{\mathit{\text{ij}}})+{\epsilon }_i,i=1,\dots ,n,}$$

where f₁(t) = 5t, f₂(t) = 3(2t − 1)², f₃(t) = 4 sin(2πt)/(2 − sin(2πt)), f₄(t) = 6(0.1 sin(2πt) + 0.2 cos(2πt) + 0.3 sin(2πt)² + 0.4 cos(2πt)³ + 0.5 sin(2πt)³), and f₅(t) = ⋯ = f_p(t) = 0. Thus, the number of nonzero functions is q = 4. This generating model is the same as Example 1 of Lin and Zhang (2006). However, here we use this model in high-dimensional settings. We consider the cases where p = 1000 and three different sample sizes: n = 50, 100 and 200. We use the cubic B-spline with six evenly distributed knots for all the functions f_k. The number of replications in all the simulations is 400.

The covariates are simulated as follows. First, we generate $w_{i1},\dots ,w_{\mathit{\text{ip}}},u_i,u_i^{\prime },{\upsilon }_i$ independently from N(0, 1) truncated to the interval [0, 1], i = 1, …, n. Then we set x_ik = (w_ik + tu_i)/(1 + t) for k = 1, …, 4 and x_ik = (w_ik + tυ_i)/(1 + t) for k = 5, …, p, where the parameter t controls the amount of correlation among predictors. We have Corr(x_ik, x_ij) = t²/(1 + t²), 1 ≤ j ≤ 4, 1 ≤ k ≤ 4, and Corr(x_ik, x_ij) = t²/(1 + t²), 4 ≤ j ≤ p, 4 ≤ k ≤ p, but the covariates of the nonzero components and zero components are independent. We consider t = 0, 1 in our simulation. The signal to noise ratio is defined to be sd(f)/sd(ε). The error term is chosen to be ε_i ~ N(0, 1.27²) to give a signal-to-noise ratio (SNR) 3.11 : 1. This value is the same as the estimated SNR in the real data example below, which is the square root of the ratio of the sum of estimated components squared divided by the sum of residual squared.

The results of 400 Monte Carlo replications are summarized in Table 1. The columns are the mean number of variables selected (NV), model error (ER), the percentage of replications in which all the correct additive components are included in the selected model (IN), and the percentage of replications in which precisely the correct components are selected (CS). The corresponding standard errors are in parentheses. The model error is computed as the average of $n^{-1}{\displaystyle {\sum }_{i=1}^n{[\widehat{f}(x_i)-f(x_i)]}^2}$ over the 400 Monte Carlo replications, where f is the true conditional mean function.

TABLE 1.

Example 1. Simulation results for the adaptive group Lasso, group Lasso, ordinary Lasso, and linear model with Lasso, n = 50, 100 or 200, p = 1000. NV, average number of the variables being selected; ME, model error; IN, percentage of occasions on which the correct components are included in the selected model; CS, percentage of occasions on which correct components are selected, averaged over 400 replications. Enclosed in parentheses are the corresponding standard errors. Top panel, independent predictors; bottom panel, correlated predictors

		Adaptive group Lasso				Group Lasso				Ordinary Lasso				Linear mode with Lasso
		NV	ME	IN	CS	NV	ME	IN	CS	NV	ME	IN	CS	NV	ME	IN	CS
		Independent predictors
n = 200	BIC	4.15	26.72	90.00	80.00	4.20	27.54	90.00	58.25	9.73	28.44	95.00	18.00	3.35	31.89	0.00	0.00
		(0.43)	(4.13)	(0.30)	(0.41)	(0.43)	(4.45)	(0.30)	(0.54)	(6.72)	(5.55)	(0.22)	(0.40)	(1.75)	(5.65)	(0.00)	(0.00)
	EBIC	4.09	26.64	92.00	81.75	4.18	27.40	92.00	60.00	9.58	28.15	95.00	32.50	3.30	32.08	0.00	0.00
		(0.38)	(4.06)	(0.24)	(0.39)	(0.40)	(4.33)	(0.24)	(0.50)	(6.81)	(5.25)	(0.22)	(0.47)	(1.86)	(5.69)	(0.00)	(0.00)
n = 100	BIC	4.73	28.26	85.00	70.00	5.03	29.07	85.00	35.00	17.25	29.50	82.50	12.00	6.35	31.57	5.00	0.00
		(1.18)	(5.71)	(0.36)	(0.46)	(1.22)	(6.01)	(0.36)	(0.48)	(8.72)	(5.89)	(0.38)	(0.44)	(2.91)	(7.22)	(0.22)	(0.00)
	EBIC	4.62	28.07	84.25	74.00	4.90	28.87	84.25	38.00	15.93	29.35	84.00	27.75	5.90	31.53	5.00	0.00
		(0.89)	(5.02)	(0.36)	(0.42)	(1.20)	(5.72)	(0.36)	(0.50)	(9.06)	(5.25)	(0.36)	(0.45)	(2.97)	(6.40)	(0.22)	(0.00)
n = 50	BIC	4.75	28.86	80.00	65.00	5.12	29.97	80.00	32.00	18.53	30.05	75.00	11.00	12.53	32.52	22.50	0.00
		(1.22)	(5.72)	(0.41)	(0.48)	(1.29)	(6.15)	(0.41)	(0.48)	(12.67)	(6.26)	(0.41)	(0.31)	(3.80)	(8.37)	(0.43)	(0.00)
	EBIC	4.69	28.94	78.00	65.00	5.01	29.82	78.00	36.00	17.27	30.50	77.50	26.00	10.33	31.64	20.00	0.00
		(1.98)	(6.48)	(0.40)	(0.48)	(1.21)	(6.11)	0.40)	(0.49)	(15.32)	(7.89)	(0.39)	(0.44)	(3.19)	(8.17)	(0.41)	(0.00)
		Correlated predictors
n = 200	BIC	3.20	27.76	66.00	60.00	3.85	28.12	66.00	30.00	9.13	28.80	56.00	11.00	1.08	32.18	0.00	0.00
		(1.27)	(4.74)	(0.46)	(0.50)	(1.49)	(4.76)	(0.46)	(0.46)	(7.02)	(5.36)	(0.51)	(0.31)	(0.33)	(8.99)	(0.00)	(0.00)
	EBIC	3.23	27.60	68.00	63.00	3.92	27.85	68.00	31.00	9.24	28.22	58.00	13.75	1.30	32.00	0.00	0.00
		(1.24)	(4.34)	(0.45)	(0.49)	(1.68)	(4.50)	(0.45)	(0.48)	(7.18)	(5.30)	(0.52)	(0.44)	(1.60)	(8.92)	(0.00)	(0.00)
n = 100	BIC	2.88	27.88	60.00	56.00	3.28	28.33	60.00	22.00	8.80	28.97	52.00	8.00	1.00	32.24	0.00	0.00
		(1.91)	(4.88)	(0.50)	(0.56)	(1.96)	(4.92)	(0.50)	(0.42)	(10.22)	(5.45)	(0.44)	(0.26)	(0.00)	(9.20)	(0.00)	(0.00)
	EBIC	3.04	27.78	61.75	58.00	3.44	28.16	61.75	24.00	9.06	28.55	54.00	10.00	1.00	32.09	0.00	0.00
		(1.46)	(4.85)	(0.49)	(0.54)	(1.52)	(4.90)	(0.49)	(0.43)	(11.24)	(5.42)	(0.46)	(0.28)	(0.00)	(8.98)	(0.00)	(0.00)
n = 50	BIC	2.50	28.36	48.50	38.00	3.10	29.37	48.50	20.00	8.01	30.48	30.00	5.00	1.00	33.28	0.00	0.00
		(1.64)	(5.32)	(0.50)	(0.55)	(1.78)	(5.98)	(0.50)	(0.41)	(11.42)	(6.77)	(0.46)	(0.23)	(0.00)	(9.42)	(0.00)	(0.00)
	EBIC	2.48	28.57	48.00	38.00	3.07	30.13	48.00	18.00	8.24	30.89	32.00	6.00	1.00	33.25	0.00	0.00
		(1.62)	(5.51)	(0.51)	(0.55)	(1.76)	(7.60)	(0.51)	(0.40)	(11.46)	(6.40)	(0.48)	(0.24)	(0.00)	(9.38)	(0.00)	(0.00)

Table 1 shows that the adaptive group Lasso selects all the nonzero components (IN) and selects exactly the correct model (CS) more frequently than the other methods do. For example, with the BIC and n = 200, the percentage of correct selections (CS) by the adaptive group Lasso ranges from 65.25% to 81%, which is much higher than the ranges 30–57.75% for the group Lasso and 12–15.75% for the ordinary Lasso. The adaptive group Lasso and group Lasso perform better than the ordinary Lasso in all of the experiments, which illustrates the importance of taking account of the group structure of the coefficients of the spline expansion. Correlation among covariates increases the difficulty of component selection, so it is not surprising that all methods perform better with independent covariates than with correlated ones. The percentage of correct selections increases as the sample size increases. The linear model with Lasso never selects the correct model. This illustrates the poor results that can be produced by a linear model when the true conditional mean function is nonlinear.

Table 1 also shows that the model error (ME) of the group Lasso is only slightly larger than that of the adaptive group Lasso. The models selected by the group Lasso nest and, therefore, have more estimated coefficients than the models selected by the adaptive group Lasso. Therefore, the group Lasso estimators of the conditional mean function have a larger variance and larger ME. The differences between the MEs of the two methods are small, however, because as can be seen from the NV column, the models selected by the group Lasso in our experiments have only slightly more estimated coefficients than the models selected by the adaptive group Lasso.

EXAMPLE 2. We now compare the adaptive group Lasso with the COSSO [Lin and Zhang (2006)]. This comparison is suggested to us by the Associate Editor. Because the COSSO algorithm only works for the case when p is smaller than n, we use the same set-up as in Example 1 of Lin and Zhang (2006). In this example, the generating model is as in (8) with 4 nonzero components. Let X_j = (W_j + tU)/(1 + t), j = 1, …, p, where W₁, …, W_p and U are i.i.d. from N(0, 1), truncated to the interval [0, 1]. Therefore, corr(X_j, X_k) = t²/(1 + t²) for j ≠ k. The random error term ε ~ N(0, 1.32²). The SNR is 3:1. We consider three different sample sizes n = 50, 100 or 200 and three different number of predictors p = 10, 20 or 50. The COSSO estimator is computed using the Matlab software which is publicly available at http://www4.stat.ncsu.edu/~hzhang/cosso.html.

The COSSO procedure uses either generalized cross-validation or 5-fold cross-validation. Based the simulation results of Lin and Zhang (2006) and our own simulations, the COSSO with 5-fold cross-validation has better selection performance. Thus, we compare the adaptive group Lasso with BIC or EBIC with the COSSO with 5-fold cross-validation. The results are given in Table 2. For independent predictors, when n = 200 and p = 10, 20 or 50, the adaptive group Lasso and COSSO have similar performance in terms of selection accuracy and model error. However, for smaller n and larger p, the adaptive group Lasso does significantly better. For example, for n = 100 and p = 50, the percentage of correct selection for the adaptive group Lasso is 81–83%, but it is only 11% for the COSSO. The model error of the adaptive group Lasso is similar to or smaller than that of the COSSO. In several experiments, the model error of the COSSO is 2 to more than 7 times larger than that of the adaptive group Lasso. It is interesting to note that when n = 50 and p = 20 or 50, the adaptive group Lasso still does a descent job in selecting the correct model, but the COSSO does poorly in these two cases. In particular, for n = 50 and p = 50, the COSSO did not select the exact correct model in all the simulation runs. For dependent predictors, the comparison is even mode favorable to the adaptive group Lasso, which performs significantly better than COSSO in terms of both model error and selection accuracy in all the cases.

TABLE 2.

Example 2. Simulation results comparing the adaptive group Lasso and COSSO. n = 50, 100 or 200, p = 10, 20 or 50. NV, average number of the variables being selected; ME, model error; IN, percentage of occasions on which all the correct components are included in the selected model; CS, percentage of occasions on which correct components are selected, averaged over 400 replications. Enclosed in parentheses are the corresponding standard errors

		p = 10				p = 20				p = 50
		NV	ME	IN	CS	NV	ME	IN	CS	NV	ME	IN	CS
		Independent predictors
n = 200	AGLasso(BIC)	4.02	0.27	100.00	98.00	4.01	0.34	96.00	92.00	4.10	0.88	98.00	90.00
		(0.14)	(0.10)	(0.00)	(0.14)	(0.40)	(0.10)	(0.20)	(0.27)	(0.39)	(0.19)	(0.14)	(0.30)
	AGLasso(EBIC)	4.02	0.27	100.00	99.00	4.05	0.32	100.00	94.00	4.08	0.87	98.00	90.00
		(0.14)	(0.09)	(0.00)	(0.10)	(0.22)	(0.09)	(0.00)	(0.24)	(0.30)	(0.16)	(0.14)	(0.30)
	COSSO(5CV)	4.06	0.29	100.00	98.00	4.10	0.37	100.00	92.00	4.49	1.53	94.00	84.00
		(0.24)	(0.07)	(0.00)	(0.14)	(0.39)	(0.11)	(0.00)	(0.27)	(1.10)	(0.86)	(0.24)	(0.37)
n = 100	AGLasso(BIC)	4.06	0.56	99.00	90.00	4.11	0.63	98.00	87.00	4.27	1.04	93.00	81.00
		(0.24)	(0.19)	(0.10)	(0.30)	(0.42)	(0.26)	(0.14)	(0.34)	(0.58)	(0.64)	(0.26)	(0.39)
	AGLasso(EBIC)	4.06	0.54	99.00	91.00	4.10	0.59	98.00	89.00	4.22	1.01	93.00	83.00
		(0.24)	(0.21)	(0.10)	(0.31)	(0.39)	(0.22)	(0.14)	(0.31)	(0.56)	(0.60)	(0.26)	(0.38)
	COSSO(5CV)	4.17	0.53	96.00	89.00	4.18	1.04	83.00	63.00	4.89	6.63	30.00	11.00
		(0.62)	(0.19)	(0.20)	(0.31)	(0.96)	(0.64)	(0.38)	(0.49)	(1.50)	(1.29)	(0.46)	(0.31)
n = 50	AGLasso(BIC)	4.18	0.72	98.00	84.00	4.25	0.99	96.00	79.00	4.30	1.06	90.00	71.00
		(0.66)	(0.56)	(0.14)	(0.36)	(0.72)	(0.60)	(0.20)	(0.41)	(0.89)	(0.68)	(0.30)	(0.46)
	AGLasso(EBIC)	4.16	0.70	98.00	84.00	4.24	1.02	94.00	78.00	4.27	1.04	92.00	73.00
		(0.64)	(0.52)	(0.14)	(0.36)	(0.70)	(0.62)	(0.20)	(0.42)	(0.86)	(0.64)	(0.27)	(0.45)
	COSSO(5CV)	4.41	1.77	61.00	58.00	5.06	5.53	33.00	20.00	5.96	7.60	8.00	0.00
		(1.08)	(1.35)	(0.46)	(0.42)	(1.54)	(1.88)	(0.47)	(0.40)	(2.20)	(2.07)	(0.27)	(0.00)
		Correlated predictors
n = 200	AGLasso(BIC)	3.75	0.49	82.00	70.00	3.71	1.20	75.00	66.00	3.50	1.68	68.00	62.00
		(0.61)	(0.14)	(0.39)	(0.46)	(0.68)	(0.89)	(0.41)	(0.46)	(0.92)	(1.29)	(0.45)	(0.49)
	AGLasso(EBIC)	3.75	0.49	82.00	70.00	3.73	1.18	75.00	68.00	3.58	1.60	70.00	65.00
		(0.61)	(0.14)	(0.39)	(0.46)	(0.65)	(0.88)	(0.41)	(0.45)	(0.84)	(1.27)	(0.46)	(0.46)
	COSSO(5CV)	3.70	0.53	69.00	41.00	3.89	1.24	57.00	36.00	4.11	1.76	41.00	16.00
		(0.58)	(0.17)	(0.46)	(0.49)	(0.60)	(0.90)	(0.50)	(0.48)	(0.86)	(1.33)	(0.49)	(0.37)
n = 100	AGLasso(BIC)	3.72	1.40	78.00	68.00	3.68	1.78	70.00	64.00	3.02	3.07	63.00	59.00
		(0.66)	(0.70)	(0.40)	(0.45)	(0.74)	(1.15)	(0.46)	(0.48)	(1.58)	(2.37)	(0.49)	(0.51)
	AGLasso(EBIC)	3.70	1.46	75.00	66.00	3.71	1.74	72.00	64.00	3.20	2.98	65.00	60.00
		(0.72)	(0.78)	(0.41)	(0.46)	(0.68)	(1.06)	(0.42)	(0.48)	(1.42)	(1.96)	(0.46)	(0.50)
	COSSO(5CV)	3.98	1.42	41.00	26.00	4.14	1.76	30.00	6.00	4.24	6.88	8.00	0.00
		(0.64)	(0.74)	(0.49)	(0.42)	(2.27)	(1.11)	(0.46)	(0.24)	(2.96)	(2.91)	(0.27)	(0.00)
n = 50	AGLasso(BIC)	3.30	2.26	70.00	62.00	3.06	3.02	65.00	60.00	2.87	4.01	52.00	42.00
		(1.16)	(1.09)	(0.46)	(0.49)	(1.52)	(2.14)	(0.46)	(0.50)	(1.56)	(3.69)	(0.44)	(0.52)
	AGLasso(EBIC)	3.32	2.20	70.00	64.00	3.10	3.01	68.00	62.00	2.90	3.88	50.00	42.00
		(1.14)	(1.06)	(0.46)	(0.48)	(1.51)	(2.12)	(0.45)	(0.49)	(1.54)	(3.62)	(0.42)	(0.52)
	COSSO(5CV)	4.14	3.77	25.00	6.00	4.20	6.98	5.00	0.00	4.90	9.93	1.00	0.00
		(2.25)	(2.02)	(0.44)	(0.24)	(2.88)	(2.82)	(0.22)	(0.00)	(3.30)	(4.08)	(0.10)	(0.00)

5. Data example

We use the data set reported in Scheetz et al. (2006) to illustrate the application of the proposed method in high-dimensional settings. For this data set, 120 twelve-week old male rats were selected for tissue harvesting from the eyes and for microarray analysis. The microarrays used to analyze the RNA from the eyes of these animals contain over 31,042 different probe sets (Affymetric GeneChip Rat Genome 230 2.0 Array). The intensity values were normalized using the robust multi-chip averaging method [Irizzary et al. (2003)] method to obtain summary expression values for each probe set. Gene expression levels were analyzed on a logarithmic scale.

We are interested in finding the genes that are related to the gene TRIM32. This gene was recently found to cause Bardet–Biedl syndrome [Chiang et al. (2006)], which is a genetically heterogeneous disease of multiple organ systems including the retina. Although over 30,000 probe sets are represented on the Rat Genome 230 2.0 Array, many of them are not expressed in the eye tissue and initial screening using correlation shows that most probe sets have very low correlation with TRIM32. In addition, we are expecting only a small number of genes to be related to TRIM32. Therefore, we use 500 probe sets that are expressed in the eye and have highest marginal correlation in the analysis. Thus, the sample size is n = 120 (i.e., there are 120 arrays from 120 rats) and p = 500. It is expected that only a few genes are related to TRIM32. Therefore, this is a sparse, high-dimensional regression problem.

We use the nonparametric additive model to model the relation between the expression of TRIM32 and those of the 500 genes. We estimate model (1) using the ordinary Lasso, group Lasso, and adaptive group Lasso for the nonparametric additive model. To compare the results of the nonparametric additive model with that of the linear regression model, we also analyzed the data using the linear regression model with Lasso. We scale the covariates so that their values are between 0 and 1 and use cubic splines with six evenly distributed knots to estimate the additive components. The penalty parameters in all the methods are chosen using the BIC or EBIC as in the simulation study. Table 3 lists the probes selected by the group Lasso and the adaptive group Lasso, indicated by the check signs. Table 4 shows the number of variables, the residual sums of squares obtained with each estimation method. For the ordinary Lasso with the spline expansion, a variable is considered to be selected if any of the estimated coefficients of the spline approximation to its additive component are nonzero. Depending on whether BIC or EBIC is used, the group Lasso selects 16–17 variables, the adaptive group Lasso selects 15 variables and the ordinary Lasso with the spline expansion selects 94–97 variables, the linear model selects 8–14 variables. Table 4 shows that the adaptive group Lasso does better than the other methods in terms of residual sum of squares (RSS). We have also examined the plots (not shown) of the estimated additive components obtained with the group Lasso and the adaptive group Lasso, respectively. Most are highly nonlinear, confirming the need for taking into account nonlinearity.

TABLE 3.

Probe sets selected by the group Lasso and the adaptive group Lasso in the data example using BIC or EBIC for penalty parameter selection. GL, group Lasso; AGL, adaptive group Lasso; Linear, linear model with Lasso

Probes	GL(BIC)	AGL(BIC)	Linear(BIC)	GL(EBIC)	AGL(EBIC)	Linear(EBIC)
1389584_at	✓	✓	✓	✓	✓	✓
1383673_at	✓	✓	✓	✓	✓	✓
1379971_at	✓	✓	✓	✓	✓	✓
1374106_at	✓		✓	✓		✓
1393817_at	✓	✓	✓	✓	✓
1373776_at	✓	✓	✓	✓	✓
1377187_at	✓	✓	✓	✓	✓
1393955_at	✓	✓	✓	✓	✓
1393684_at	✓	✓		✓	✓
1381515_at	✓	✓		✓	✓
1382835_at	✓	✓	✓	✓	✓
1385944_at	✓	✓	✓	✓	✓
1382263_at	✓	✓	✓	✓	✓	✓
1380033_at	✓	✓		✓	✓
1398594_at	✓		✓			✓
1376744_at	✓	✓		✓	✓
1382633_at	✓	✓		✓	✓
1383110_at			✓			✓
1386683_at			✓			✓

TABLE 4.

Analysis results for the data example. No. of probes, the number of probe sets selected; RSS, the residual sum of squares of the fitted model

	BIC		EBIC
	No. of probe sets	RSS	No. of probe sets	RSS
Adaptive group Lasso	15	1.52e–03	15	1.52e–03
Group Lasso	17	3.24e–03	16	3.40e–03
Ordinary Lasso	97	2.96e–07	94	8.10e–08
Linear regression with Lasso	14	2.62e–03	8	3.75e–03

In order to evaluate the performance of the methods, we use cross-validation and compare the prediction mean square errors (PEs). We randomly partition the data into 6 subsets, each set consisting of 20 observations. We then fit the model with 5 subsets as training set and calculate the PE for the remaining set which we consider as test set. We repeat this process 6 times, considering one of the 6 subsets as test set every time. We compute the average of the numbers of probes selected and the prediction errors of these 6 calculations. Then we replicate this process 400 times (this is suggested to us by the Associate Editor). Table 5 gives the average values over 400 replications. The adaptive group Lasso has smaller average prediction error than the group Lasso, the ordinary Lasso and the linear regression with Lasso. The ordinary Lasso selects far more probe sets than the other approaches, but this does not lead to better prediction performance. Therefore, in this example, the adaptive group Lasso provides the investigator a more targeted list of probe sets, which can serve as a starting point for further study.

TABLE 5.

Comparison of adaptive group Lasso, group Lasso, ordinary Lasso, and linear regression model with Lasso for the data example. ANP, the average number of probe sets selected averaged across 400 replications; PE, the average of prediction mean square errors for the test set

	Adaptive group Lasso		Group Lasso		Ordinary Lasso		Linear model with Lasso
	ANP	PE	ANP	PE	ANP	PE	ANP	PE
BIC	15.75	1.86e–02	16.45	2.89e–02	78.48	1.40e–02	9.25	2.26e–02
	(0.85)	(0.47e–02)	(0.88)	(0.49e–02)	(3.62)	(0.90e–02)	(0.88)	(1.41e–2)
EBIC	15.55	1.78e–02	16.75	1.99e–02	80.00	1.23e–02	9.15	2.03e–02
	(0.82)	(0.42e–02)	(0.84)	(0.47e–02)	(3.50)	(0.89e–02)	(0.86)	(1.39e–02)

It is of interest to compare the selection results from the adaptive group Lasso and the linear regression model with Lasso. The adaptive group Lasso and the linear model with Lasso select different sets of genes. When the penalty parameter is chosen with the BIC, the adaptive group Lasso selects 5 genes that are not selected by the linear model with Lasso. In addition, the linear model with Lasso selects 5 genes that are not selected by the adaptive group Lasso. When the penalty parameter is selected with the EBIC, the adaptive group Lasso selects 10 genes that are not selected by the linear model with Lasso. The estimated effects of many of the genes are nonlinear, and the Monte Carlo results of Section 4 show that the performance of the linear model with Lasso can be very poor in the presence of nonlinearity. Therefore, we interpret the differences between the gene selections of the adaptive group Lasso and the linear model with Lasso as evidence that the selections produced by the linear model are misleading.

6. Concluding remarks

In this paper, we propose to use the adaptive group Lasso for variable selection in nonparametric additive models in sparse, high-dimensional settings. A key requirement for the adaptive group Lasso to be selection consistent is that the initial estimator is estimation consistent and selects all the important components with high probability. In low-dimensional settings, finding an initial consistent estimator is relatively easy and can be achieved by many well-established approaches such as the additive spline estimators. However, in high-dimensional settings, finding an initial consistent estimator is difficult. Under the conditions stated in Theorem 1, the group Lasso is shown to be consistent and selects all the important components. Thus the group Lasso can be used as the initial estimator in the adaptive Lasso to achieve selection consistency. Following model selection, oracle-efficient, asymptotically normal estimators of the nonzero components can be obtained by using existing methods. Our simulation results indicate that our procedure works well for variable selection in the models considered. Therefore, the adaptive group Lasso is a useful approach for variable selection and estimation in sparse, high-dimensional nonparametric additive models.

Our theoretical results are concerned with a fixed sequence of penalty parameters, which are not applicable to the case where the penalty parameters are selected based on data driven procedures such as the BIC. This is an important and challenging problem that deserves further investigation, but is beyond the scope of this paper. We have only considered linear nonparametric additive models. The adaptive group Lasso can be applied to generalized nonparametric additive models, such as the generalized logistic nonparametric additive model and other nonparametric models with high-dimensional data. However, more work is needed to understand the properties of this approach in those more complicated models.

APPENDIX: PROOFS

We first prove the following lemmas. Denote the centered versions of 𝒮_n by

$${𝒮}_{\mathit{\text{nj}}}^0=\{f_{\mathit{\text{nj}}}:f_{\mathit{\text{nj}}}(x)={\displaystyle \sum _{k=1}^{m_n}{b}_{\mathit{\text{jk}}}{\psi }_k(x),({\beta }_{j1},\dots ,{\beta }_{{\mathit{\text{jm}}}_n})\in {ℝ}^{m_n}}\},1\le j\le p,$$

where ψ_k’s are the centered spline bases defined in (5).

LEMMA 1. Suppose that f ∈ ℱ and E f(X_j) = 0. Then under (A3) and (A4), there exists an $f_n\in {𝒮}_{\mathit{\text{nj}}}^0$ satisfying

$$\Vert f_n-f{\Vert }_2=O_p(m_n^{-d}+m_n^{1/2}{n}^{-1/2}).$$

In particular, if we choose m_n = O(n^1/(2d+1)), then

$$\Vert f_n-f{\Vert }_2=O_p(m_n^{-d})=O_p(n^{-d/(2d+1)}).$$

PROOF. By (A4), for f ∈ ℱ, there is an $f_n^{*}\in {𝒮}_n$ such that ${\Vert f-f_n^{*}\Vert }_2=O(m_n^{-d})$ . Let $f_n=f_n^{*}-n^{-1}{\displaystyle {\sum }_{i=1}^n{f}_n^{*}}(X_{\mathit{\text{ij}}})$ . Then $f_n\in {𝒮}_{\mathit{\text{nj}}}^0\text{ and }|f_n-f|\le |f_n^{*}-f|+|P_n{f}_n^{*}|$ , where P_n is the empirical measure of i.i.d. random variables X_1j, …, X_nj. Consider

$$P_n{f}_n^{*}=(P_n-P)f_n^{*}+P(f_n^{*}-f).$$

Here, we use the linear functional notation, for example, Pf = ∫ fdP, where P is the probability measure of X_1j. For any ε > 0, the bracketing number $N_{[·]}(\epsilon ,{𝒮}_{\mathit{\text{nj}}}^0,L_2(P))\text{ of }{𝒮}_{\mathit{\text{nj}}}^0$ satisfies $\text{log}{N}_{[·]}(\epsilon ,{𝒮}_{\mathit{\text{nj}}}^0,L_2(P))\le c_1{m}_n\text{log}(1/\epsilon )$ for some constant c₁ > 0 [Shen and Wong (1994), page 597]. Thus, by the maximal inequality; see, for example, van der Vaart (1998, page 288), $(P_n-P)f_n^{*}=O_p(n^{-1/2}{m}_n^{1/2})$ . By (A4), $|P(f_n^{*}-f)|\le C_2{\Vert f_n^{*}-f\Vert }_2=O(m_n^{-d})$ for some constant C₂ > 0. The lemma follows from the triangle inequality.

LEMMA 2. Suppose that conditions (A2) and (A4) hold. Let

$$T_{\mathit{\text{jk}}}=n^{-1/2}{m}_n^{1/2}{\displaystyle \sum _{i=1}^n{\psi }_k(X_{\mathit{\text{ij}}}){\epsilon }_i,}1\le j\le p,1\le k\le m_n,$$

and T_n = max_{1≤j≤p,1≤k≤mn} |T_jk|. Then

$$\mathrm{E}(T_n)\le C_1{n}^{-1/2}{m}_n^{1/2}\sqrt{\text{log}({\mathit{\text{pm}}}_n)}{(\sqrt{2C_2{m}_n^{-1}n\text{log}({\mathit{\text{pm}}}_n)}+4\text{log}(2{\mathit{\text{pm}}}_n)+C_2{\mathit{\text{nm}}}_n^{-1})}^{1/2},$$

where C₁ and C₂ are two positive constants. In particular, when m_n log(pm_n)/n → 0,

$$\mathrm{E}(T_n)=O(1)\sqrt{\text{log}({\mathit{\text{pm}}}_n)}.$$

PROOF. Let $s_{\mathit{\text{njk}}}^2={\displaystyle {\sum }_{i=1}^n{\psi }_k^2(X_{\mathit{\text{ij}}})}$ . Conditional on X_ij’s, T_jk’s are sub-Gaussian. Let $s_n^2={\text{max }}_{1\le j\le p,1\le k\le m_n}{s}_{\mathit{\text{njk}}}^2$ . By (A2) and the maximal inequality for sub-Gaussian random variables [van der Vaart and Wellner (1996), Lemmas 2.2.1 and 2.2.2],

$$\mathrm{E}(\underset{1\le j\le p,1\le k\le m_n}{\text{max }}|T_{\mathit{\text{jk}}}||\{X_{\mathit{\text{ij}}},1\le i\le n,1\le j\le p\})\le C_1{n}^{-1/2}{m}_n^{1/2}{s}_n\sqrt{\text{log}({\mathit{\text{pm}}}_n)}.$$

Therefore,

$$\mathrm{E}(\underset{1\le j\le p,1\le k\le m_n}{\text{max }}|T_{\mathit{\text{jk}}}|)\le C_1{n}^{-1/2}{m}_n^{1/2}\sqrt{\text{log}({\mathit{\text{pm}}}_n)}\mathrm{E}(s_n),$$ (9)

where C₁ > 0 is a constant. By (A4) and the properties of B-splines,

$$|{\psi }_k(X_{\mathit{\text{ij}}})|\le |{\varphi }_k(X_{\mathit{\text{ij}}})|+|{\overline{\varphi }}_{\mathit{\text{jk}}}|\le 2\text{\hspace{1em}and\hspace{1em}}\mathrm{E}{({\psi }_k(X_{\mathit{\text{ij}}}))}^2\le C_2{m}_n^{-1}$$ (10)

for a constant C₂ > 0, for every 1 ≤ j ≤ p and 1 ≤ k ≤ m_n. By (10),

$${\displaystyle \sum _{i=1}^n\mathrm{E}{[{\psi }_k^2(X_{\mathit{\text{ij}}})-\mathrm{E}{\psi }_k^2(X_{\mathit{\text{ij}}})]}^2\le 4C_2{\mathit{\text{nm}}}_n^{-1}}$$ (11)

and

$$\underset{1\le j\le p,1\le k\le m_n}{\text{max }}{\displaystyle \sum _{i=1}^n\mathrm{E}{\psi }_k^2(X_{\mathit{\text{ij}}})}\le C_2{\mathit{\text{nm}}}_n^{-1}.$$ (12)

By Lemma A.1 of van de Geer (2008), (10) and (11) imply

$$\mathrm{E}\left(\underset{1\le j\le p,1\le k\le m_n}{\text{max }}\left|{\displaystyle \sum _{i=1}^n\{{\psi }_k^2(X_{\mathit{\text{ij}}})-\mathrm{E}{\psi }_k^2(X_{\mathit{\text{ij}}})\}}\right|\right)\le \sqrt{2C_2{m}_n^{-1}n\text{log}({\mathit{\text{pm}}}_n)}+4\text{log}(2{\mathit{\text{pm}}}_n).$$

Therefore, by (12) and the triangle inequality,

$$\mathrm{E}{s}_n^2\le \sqrt{2C_2{m}_n^{-1}n\text{log}({\mathit{\text{pm}}}_n)}+4\text{log}(2{\mathit{\text{pm}}}_n)+C_2{\mathit{\text{nm}}}_n^{-1}.$$

Now since ${\mathrm{E}s}_n\le {({\mathrm{E}s}_n^2)}^{1/2}$ , we have

$$\mathrm{E}{s}_n\le {(\sqrt{2C_2{m}_n^{-1}n\text{log}({\mathit{\text{pm}}}_n)}+4\text{log}(2{\mathit{\text{pm}}}_n)+C_2{\mathit{\text{nm}}}_n^{-1})}^{1/2}.$$ (13)

The lemma follows from (9) and (13).

Denote

$${\mathit{\beta }}_A=({\mathit{\beta }}_j^{\prime },j\in A)\prime \text{\hspace{1em}and\hspace{1em}}{\mathbf{Z}}_A=({\mathbf{Z}}_j,j\in A).$$

Here, β_A is an |A|m_n × 1 vector and Z_A is an n × |A|m_n matrix. Let ${\mathbf{C}}_A={\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A/n$ . When A = {1, …, p}, we simply write C = Z′Z/n. Let ρ_min(C_A) and ρ_max(C_A) be the minimum and maximum eigenvalues of C_A, respectively.

LEMMA 3. Let m_n = O(n^γ) where 0 < γ < 0.5. Suppose that |A| is bounded by a fixed constant independent of n and p. Let $h\equiv h_n\asymp m_n^{-1}$ . Then under (A3) and (A4), with probability converging to one,

$$c_1{h}_n\le {\rho }_{\text{min}}({\mathbf{C}}_A)\le {\rho }_{\text{max}}({\mathbf{C}}_A)\le c_2{h}_n,$$

where c₁ and c₂ are two positive constants.

PROOF. Without loss of generality, suppose A = {1, …, k}. Then Z_A = (Z₁, …, Z_q). Let $\mathbf{b}=({\mathbf{b}}_1^{\prime },\dots ,{\mathbf{b}}_q^{\prime })\prime$ , where b_j ∈ R^m_n. By Lemma 3 of Stone (1985),

$${\Vert {\mathbf{Z}}_1{\mathbf{b}}_1+\cdots +{\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2\ge c_3({\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2)$$

for a certain constant c₃ > 0. By the triangle inequality,

$${\Vert {\mathbf{Z}}_1{\mathbf{b}}_1+\cdots +{\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2\le {\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2.$$

Since Z_Ab = Z₁b₁ + ⋯ + Z_qb_q, the above two inequalities imply that

$$c_3({\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2)\le {\Vert {\mathbf{Z}}_A\mathbf{b}\Vert }_2\le {\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2.$$

Therefore,

$$c_3^2({\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2^2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2^2)\le {\Vert {\mathbf{Z}}_A\mathbf{b}\Vert }_2^2\le 2({\Vert {\mathbf{Z}}_1{\mathbf{b}}_1\Vert }_2^2+\cdots +{\Vert {\mathbf{Z}}_q{\mathbf{b}}_q\Vert }_2^2).$$ (14)

Let ${\mathbf{C}}_j=n^{-1}{\mathbf{Z}}_j^{\prime }{\mathbf{Z}}_j$ . By Lemma 6.2 of Zhou, Shen and Wolf (1998),

$$c_{4}h\le {\rho }_{\text{min}}({\mathbf{C}}_j)\le {\rho }_{\text{max}}({\mathbf{C}}_j)\le c_{5}h,j\in A.$$ (15)

Since ${\mathbf{C}}_A=n^{-1}{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A$ , it follows from (14) that

$$c_3^2({\mathbf{b}}_1^{\prime }{\mathbf{C}}_1{\mathbf{b}}_1+\cdots +{\mathbf{b}}_q^{\prime }{\mathbf{C}}_q{\mathbf{b}}_q)\le \mathbf{b}\prime {\mathbf{C}}_A\mathbf{b}\le 2({\mathbf{b}}_1^{\prime }{\mathbf{C}}_1{\mathbf{b}}_1+\cdots +{\mathbf{b}}_q^{\prime }{\mathbf{C}}_q{\mathbf{b}}_q).$$

Therefore, by (15),

$$\begin{array}{cc}\frac{{\mathbf{b}}_1^{\prime }{\mathbf{C}}_1{\mathbf{b}}_1}{{\Vert \mathbf{b}\Vert }_2^2}+\cdots +\frac{{\mathbf{b}}_q^{\prime }{\mathbf{C}}_q{\mathbf{b}}_q}{{\Vert \mathbf{b}\Vert }_2^2}\hfill & =\frac{{\mathbf{b}}_1^{\prime }{\mathbf{C}}_1{\mathbf{b}}_1}{{\Vert {\mathbf{b}}_1\Vert }_2^2}\frac{{\Vert {\mathbf{b}}_1\Vert }_2^2}{{\Vert \mathbf{b}\Vert }_2^2}+\cdots +\frac{{\mathbf{b}}_q^{\prime }{\mathbf{C}}_q{\mathbf{b}}_q}{{\Vert {\mathbf{b}}_q\Vert }_2^2}\frac{{\Vert {\mathbf{b}}_q\Vert }_2^2}{{\Vert \mathbf{b}\Vert }_2^2}\hfill \\ \hfill & \ge {\rho }_{\text{min}}({\mathbf{C}}_1)\frac{{\Vert {\mathbf{b}}_1\Vert }_2^2}{{\Vert \mathbf{b}\Vert }_2^2}+\cdots +{\rho }_{\text{min}}({\mathbf{C}}_q)\frac{{\Vert {\mathbf{b}}_q\Vert }_2^2}{{\Vert \mathbf{b}\Vert }_2^2}\hfill \\ \hfill & \ge c_{4}h.\hfill \end{array}$$

Similarly,

$$\frac{{\mathbf{b}}_1^{\prime }{\mathbf{C}}_1{\mathbf{b}}_1}{{\Vert \mathbf{b}\Vert }_2^2}+\cdots +\frac{{\mathbf{b}}_q^{\prime }{\mathbf{C}}_q{\mathbf{b}}_q}{{\Vert \mathbf{b}\Vert }_2^2}\le c_{5}h.$$

Thus, we have

$$c_3^2{c}_{4}h\le \frac{\mathbf{b}\prime {\mathbf{C}}_A\mathbf{b}}{\mathbf{b}\prime \mathbf{b}}\le 2c_{5}h.$$

The lemma follows.

PROOF OF THEOREM 1. The proof of parts (i) and (ii) essentially follows the proof of Theorem 2.1 of Wei and Huang (2008). The only change that must be made here is that we need to consider the approximation error of the regression functions by splines. Specifically, let ξ_n = ε_n + δ_n, where δ_n = (δ_n1, …, δ_nn)′ with ${\delta }_{\mathit{\text{ni}}}={\displaystyle {\sum }_{j=1}^{q_n}(f_{0j}(X_{\mathit{\text{ij}}})-f_{\mathit{\text{nj}}}(X_{\mathit{\text{ij}}}))}$ . Since ${\Vert f_{0j}-f_{\mathit{\text{nj}}}\Vert }_2=O(m_n^{-d})=O(n^{-d/(2d+1)})$ for m_n = n^1/(2d+1), we have

$${\Vert {\mathit{\delta }}_n\Vert }_2\le C_1\sqrt{{\mathit{\text{nqm}}}_n^{-2d}}=C_{1}q{n}^{1/(4d+2)}$$

for some constant C₁ > 0. For any integer t, let

$${\chi }_t=\underset{|A|=t}{\text{max}}\underset{{\Vert U_{A_k}\Vert }_2=1,1\le k\le t}{\text{max}}\frac{|{\mathit{\xi }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A(\mathbf{s})\Vert }_2}\text{\hspace{1em}and\hspace{1em}}{\chi }_t^{*}=\underset{|A|=t}{\text{max}}\underset{{\Vert U_{A_k}\Vert }_2=1,1\le k\le t}{\text{max}}\frac{|{\mathit{\epsilon }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A(\mathbf{s})\Vert }_2},$$

where $V_A(S_A)={\mathit{\xi }}_n^{\prime }({\mathbf{Z}}_A{({\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A)}^{-1}{\overline{S}}_A-(I-P_A)X\mathit{\beta }$ for N(A) = q₁ = m ≥ 0, $S_A=(S_{A_1}^{\prime },\dots ,S_{A_m}^{\prime })\prime ,S_{A_k}=\lambda \sqrt{d_{A_k}}{U}_{A_k}$ and ‖U_Ak‖₂ = 1.

For a sufficiently large constant C₂ > 0, define

$${\mathrm{\Omega }}_{t_0}=\{(\mathbf{Z},{\mathit{\epsilon }}_n):x_t\le \sigma C_2\sqrt{(t\vee 1)m_n\text{log}({\mathit{\text{pm}}}_n),}\forall t\ge t_0\}$$

and

$${\mathrm{\Omega }}_{t_0}^{*}=\{(\mathbf{Z},{\mathit{\epsilon }}_n):x_t^{*}\le \sigma C_2\sqrt{(t\vee 1)m_n\text{log}({\mathit{\text{pm}}}_n),}\forall t\ge t_0\},$$

where t₀ ≥ 0.

As in the proof of Theorem 2.1 of Wei and Huang (2008),

$$(\mathbf{Z},{\mathit{\epsilon }}_n)\in {\mathrm{\Omega }}_q\Rightarrow |{\tilde{A}}_1|\le M_{1}q$$

for a constant M₁ > 1. By the triangle and Cauchy–Schwarz inequalities,

$$\frac{|{\mathit{\xi }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A(\mathbf{s})\Vert }_2}=\frac{|{\mathit{\epsilon }}_n^{\prime }{V}_A(\mathbf{s})+{\mathit{\delta }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A(\mathbf{s})\Vert }_2}\le \frac{|{\mathit{\epsilon }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A\Vert }_2}+\Vert {\mathit{\delta }}_n\Vert .$$ (16)

In the proof of Theorem 2.1 of Wei and Huang (2008), it is shown that

$$\mathrm{P}({\mathrm{\Omega }}_0^{*})\ge 2-\frac{2}{p^{1+c_0}}-\text{exp}\left(\frac{2p}{p^{1+c_0}}\right)\to 1.$$ (17)

Since

$$\frac{|{\mathit{\delta }}_n^{\prime }{V}_A(\mathbf{s})|}{{\Vert V_A(\mathbf{s})\Vert }_2}\le {\Vert {\mathit{\delta }}_n\Vert }_2\le C_1{\mathit{\text{qn}}}^{1/(2(2d+1))}$$

and m_n = O(n^1/(2d+1)), we have for all t ≥ 0 and n sufficiently large,

$${\Vert {\mathit{\delta }}_n\Vert }_2\le C_1{\mathit{\text{qn}}}^{1/(2(2d+1))}\le \sigma C_2\sqrt{(t\vee 1)m_n\text{log}(p)}.$$ (18)

It follows from (16), (17) and (18) that P(Ω₀) → 1. This completes the proof of part (i) of Theorem 1.

Before proving part (ii), we first prove part (iii) of Theorem 1. By the definition of ${\tilde{\mathit{\beta }}}_n\equiv ({\tilde{\mathit{\beta }}}_{n1}^{\prime },\dots ,{\tilde{\mathit{\beta }}}_{\mathit{\text{np}}}^{\prime })\prime$ ,

$${\Vert \mathbf{Y}-\mathbf{Z}{\tilde{\mathit{\beta }}}_n\Vert }_2^2+{\lambda }_{n1}{\displaystyle \sum _{j=1}^p{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2\le {\Vert \mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n\Vert }_2^2}+{\lambda }_{n1}{\displaystyle \sum _{j=1}^p{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2.}$$ (19)

Let A₂ = {j : ‖β_nj‖₂ ≠ 0 or ‖β̃_nj‖₂ ≠ 0} and d_n2 = |A₂|. By part (i), d_n2 = O_p(q). By (19) and the definition of A₂,

$${\Vert \mathbf{Y}-{\mathbf{Z}}_{A_2}{\tilde{\mathit{\beta }}}_{n{A}_2}\Vert }_2^2+{\lambda }_{n1}{\displaystyle \sum _{j\in A_2}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2\le {\Vert \mathbf{Y}-{\mathbf{Z}}_{A_2}{\mathit{\beta }}_{n{A}_2}\Vert }_2^2}+{\lambda }_{n1}{\displaystyle \sum _{j\in A_2}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2.}$$ (20)

Let η_n = Y − Zβ_n. Write

$$\mathbf{Y}-{\mathbf{Z}}_{A_2}{\tilde{\mathit{\beta }}}_{n{A}_2}=\mathbf{Y}-\mathbf{Z}{\mathit{\beta }}_n-{\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2})={\mathit{\eta }}_n-{\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}).$$

We have

$${\Vert \mathbf{Y}-{\mathbf{Z}}_{A_2}{\tilde{\mathit{\beta }}}_{n{A}_2}\Vert }_2^2={\Vert {\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2})\Vert }_2^2-2{\mathit{\eta }}_n^{\prime }{\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2})+{\mathit{\eta }}_n^{\prime }{\mathit{\eta }}_n.$$

We can rewrite (20) as

$${\Vert {\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2})\Vert }_2^2-2{\mathit{\eta }}_n^{\prime }{\mathbf{Z}}_{A_2}({\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2})\le {\lambda }_{n1}{\displaystyle \sum _{j\in A_1}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2}-{\lambda }_{n1}{\displaystyle \sum _{j\in A_1}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2}.$$ (21)

Now

$$\left|{\displaystyle \sum _{j\in A_1}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2}-{\displaystyle \sum _{j\in A_1}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2}\right|\le \sqrt{|A_1|}·{\Vert {\tilde{\mathit{\beta }}}_{n{A}_1}-{\mathit{\beta }}_{n{A}_1}\Vert }_2\le \sqrt{|A_1|}·{\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2.$$ (22)

Let ν_n = Z_A2(β̃_nA2 − β_nA2). Combining (20), (21) and (22) to get

$${\Vert {\mathit{\nu }}_n\Vert }_2^2-2{\mathit{\eta }}_n^{\prime }{\mathit{\nu }}_n\le {\lambda }_{n1}\sqrt{|A_1|}·{\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2.$$ (23)

Let ${\mathit{\eta }}_n^{*}$ be the projection of η_n to the span Z_A2, that is, ${\mathit{\eta }}_n^{*}={\mathbf{Z}}_{A_2}{({\mathbf{Z}}_{A_2}^{\prime }\times {\mathbf{Z}}_{A_2})}^{-1}{\mathbf{Z}}_{A_2}^{\prime }{\mathit{\eta }}_n$ . By the Cauchy–Schwarz inequality,

$$2|{\mathit{\eta }}_n^{\prime }{\mathit{\nu }}_n|\le 2{\Vert {\mathit{\eta }}_n^{*}\Vert }_2·{\Vert {\mathit{\nu }}_n\Vert }_2\le 2{\Vert {\mathit{\eta }}_n^{*}\Vert }_2^2+\frac{1}{2}{\Vert {\mathit{\nu }}_n\Vert }_2^2.$$ (24)

From (23) and (24), we have

$${\Vert {\mathit{\nu }}_n\Vert }_2^2\le 4{\Vert {\mathit{\eta }}_n^{*}\Vert }_2^2+2{\lambda }_{n1}\sqrt{|A_1|}·{\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2.$$

Let c_n* be the smallest eigenvalue of ${\mathbf{Z}}_{A_2}^{\prime }{\mathbf{Z}}_{A_2}/n$ . By Lemma 3 and part (i), $c_{n*}{\asymp }_p{m}_n^{-1}$ . Since ${\Vert {\mathit{\nu }}_n\Vert }_2^2\ge {\mathit{\text{nc}}}_{n*}{\Vert {\tilde{\mathit{\beta }}}_{{\mathit{\text{nA}}}_2}-{\mathit{\beta }}_{{\mathit{\text{nA}}}_2}\Vert }_2^2$ and 2ab ≤ a² + b²,

$${\mathit{\text{nc}}}_{n*}{\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2^2\le 4{\Vert {\mathit{\eta }}_n^{*}\Vert }_2^2+\frac{{(2{\lambda }_{n1}\sqrt{|A_1|})}^2}{2{\mathit{\text{nc}}}_{n*}}+\frac{1}{2}{\mathit{\text{nc}}}_{n*}{\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2^2.$$

It follows that

$${\Vert {\tilde{\mathit{\beta }}}_{n{A}_2}-{\mathit{\beta }}_{n{A}_2}\Vert }_2^2\le \frac{8{\Vert {\mathit{\eta }}_n^{*}\Vert }_2^2}{{\mathit{\text{nc}}}_{n*}}+\frac{4{\lambda }_{n1}^2|A_1|}{n^2{c}_{n\mathrm{*}}^2}.$$ (25)

Let $f_0({\mathbf{X}}_i)={\displaystyle {\sum }_{j=1}^p{f}_{0j}(X_{\mathit{\text{ij}}})}$ and f_0A(X_i) = ∑_j∈A f_0j(X_ij). Write

$${\eta }_i=Y_i-\mu -f_0({\mathbf{X}}_i)+(\mu -\overline{Y})+f_0({\mathbf{X}}_i)-{\displaystyle \sum _{j\in A_2}{Z}_{\mathit{\text{ij}}}^{\prime }{\mathit{\beta }}_{\mathit{\text{nj}}}}={\epsilon }_i+(\mu -\overline{Y})+f_{A_2}({\mathbf{X}}_i)-f_{n{A}_2}({\mathbf{X}}_i).$$

Since |µ − Y̅|² = O_p(n⁻¹) and ${\Vert f_{0j}-f_{\mathit{\text{nj}}}\Vert }_{\infty }=O(m_n^{-d})$ , we have

$${\Vert {\mathit{\eta }}_n^{*}\Vert }_2^2\le 2{\Vert {\mathit{\epsilon }}_n^{*}\Vert }_2^2+O_p(1)+O({\mathit{\text{nd}}}_{n2}{m}_n^{-2d}),$$ (26)

where ${\mathit{\epsilon }}_n^{*}$ is the projection of ε_n = (ε₁, …, ε_n)′ to the span of Z_A2. We have

$${\Vert {\mathit{\epsilon }}_n^{*}\Vert }_2^2={\Vert {({\mathbf{Z}}_{A_2}^{\prime }{\mathbf{Z}}_{A_2})}^{-1/2}{\mathbf{Z}}_{A_2}^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2\le \frac{1}{{\mathit{\text{nc}}}_{n*}}{\Vert {\mathbf{Z}}_{A_2}^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2.$$

Now

$$\underset{A:|A|\le d_{n2}}{\text{max}}{\Vert {\mathbf{Z}}_A^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2=\underset{A:|A|\le d_{n2}}{\text{max}}{\displaystyle \sum _{j\in A}{\Vert {\mathbf{Z}}_j^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2}\le d_{n2}{m}_n\underset{1\le j\le p,1\le k\le m_n}{\text{max}}{|{𝒵}_{\mathit{\text{jk}}}^{\prime }\mathit{\epsilon }|}^2,$$

where 𝒵_jk = (ψ_k(X_1j), …, ψ_k(X_nj))′. By Lemma 2,

$$\underset{1\le j\le p,1\le k\le m_n}{\text{max}}{|{𝒵}_{\mathit{\text{jk}}}^{\prime }{\mathit{\epsilon }}_n|}^2={\mathit{\text{nm}}}_n^{-1}\underset{1\le j\le p,1\le k\le m_n}{\text{max}}{|{(m_n/n)}^{1/2}{𝒵}_{\mathit{\text{jk}}}^{\prime }{\mathit{\epsilon }}_n|}^2=O_p(1){\mathit{\text{nm}}}_n^{-1}\text{log}({\mathit{\text{pm}}}_n).$$

It follows that,

$${\Vert {\mathit{\epsilon }}_n^{*}\Vert }_2^2=O_p(1)\frac{d_{n2}\text{log}({\mathit{\text{pm}}}_n)}{c_{n*}}.$$ (27)

Combining (25), (26) and (27), we get

$${\Vert {\tilde{\mathit{\beta }}}_{A_2}-{\mathit{\beta }}_{A_2}\Vert }_2^2\le O_p\left(\frac{d_{n2}\text{log}({\mathit{\text{pm}}}_n)}{{\mathit{\text{nc}}}_{n*}^2}\right)+O_p\left(\frac{1}{{\mathit{\text{nc}}}_{n*}}\right)+O\left(\frac{d_{n2}{m}_n^{-2d}}{c_{n*}}\right)+\frac{4{\lambda }_{n1}^2|A_1|}{n^2{c}_{n*}^2}.$$

Since d_n2 = O_p(q), $c_{n*}{\asymp }_p{m}_n^{-1}\text{ and }{c}_n^{*}{\asymp }_p{m}_n^{-1}$ , we have

$${\Vert {\tilde{\mathit{\beta }}}_{A_2}-{\mathit{\beta }}_{A_2}\Vert }_2^2\le O_p\left(\frac{m_n^2\text{log}({\mathit{\text{pm}}}_n)}{n}\right)+O_p\left(\frac{m_n}{n}\right)+O\left(\frac{1}{m_n^{2d-1}}\right)+O\left(\frac{4m_n^2{\lambda }_{n1}^2}{n^2}\right).$$

This completes the proof of part (iii).

We now prove part (ii). Since ${\Vert f_j\Vert }_2\ge c_f>0,1\le j\le q,{\Vert f_j-f_{\mathit{\text{nj}}}\Vert }_2=O(m_n^{-d})$ and ‖f_nj‖₂ ≥ ‖f_j‖₂ − ‖f_j − f_nj‖₂, we have ‖f_nj‖₂ ≥ 0.5c_f for n sufficiently large. By a result of de Boor (2001), see also (12) of Stone (1986), there are positive constants c₆ and c₇ such that

$$c_6{m}_n^{-1}{\Vert {\mathit{\beta }}_n\Vert }_2^2\le {\Vert f_{\mathit{\text{nj}}}\Vert }_2^2\le c_7{m}_n^{-1}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2.$$

It follows that ${\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2\ge c_7^{-1}{m}_n{\Vert f_{\mathit{\text{nj}}}\Vert }_2^2\ge 0.25c_7^{-1}{c}_f^2{m}_n$ . Therefore, if ‖β_nj‖₂ ≠ 0 but ‖β̃_nj‖₂ = 0, then

$${\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2\ge 0.25c_7^{-1}{c}_f^2{m}_n.$$ (28)

However, since (m_n log(pm_n))/n → 0 and $({\lambda }_{n1}^2{m}_n)/n^2\to$ , (28) contradicts part (iii).

PROOF OF THEOREM 2. By the definition of f̃_j, 1 ≤ j ≤ p, parts (i) and (ii) follow from parts (i) and (ii) of Theorem 1 directly.

Now consider part (iii). By the properties of spline [de Boor (2001)],

$$c_6{m}_n^{-1}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2\le {\Vert {\tilde{f}}_{\mathit{\text{nj}}}-f_{\mathit{\text{nj}}}\Vert }_2^2\le c_7{m}_n^{-1}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2^2.$$

Thus,

$${\Vert {\tilde{f}}_{\mathit{\text{nj}}}-f_{\mathit{\text{nj}}}\Vert }_2^2=O_p\left(\frac{m_n\text{log}({\mathit{\text{pm}}}_n)}{n}\right)+O_p\left(\frac{1}{n}\right)+O\left(\frac{1}{m_n^{2d}}\right)+O\left(\frac{4m_n{\lambda }_{n1}^2}{n^2}\right).$$ (29)

By (A3),

$${\Vert f_j-f_{\mathit{\text{nj}}}\Vert }_2^2=O(m_n^{-2d}).$$ (30)

Part (iii) follows from (29) and (30).

In the proofs below, for any matrix H, denote its 2-norm by ‖H‖, which is equal to its largest eigenvalue. This norm satisfies the inequality ‖Hx‖ ≤ ‖H‖‖x‖ for a column vector x whose dimension is the same as the number of the columns of H.

Denote ${\mathit{\beta }}_{{\mathit{\text{nA}}}_1}=({\mathit{\beta }}_{\mathit{\text{nj}}}^{\prime },j\in A_1)\prime ,{\widehat{\mathit{\beta }}}_{{\mathit{\text{nA}}}_1}=({\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}^{\prime },j\in A_1)\prime$ and Z_A1 = Z_j, j ∈ A₁). Define ${\mathbf{C}}_{A_1}=n^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1}$ . Let ρ_n1 and ρ_n2 be the smallest and largest eigenvalues of C_A1, respectively.

PROOF OF THEOREM 3. By the KKT, a necessary and sufficient condition for β̂_n is

$$\{\begin{array}{cc}2{\mathbf{Z}}_j^{\prime }(\mathbf{Y}-\mathbf{Z}{\widehat{\mathit{\beta }}}_n)={\mathrm{\lambda }}_{n2}{w}_{\mathit{\text{nj}}}\frac{{\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}}{\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert },\hfill & {\Vert {\widehat{\mathit{\beta }}}_j\Vert }_2\ne 0,j\ge 1,\hfill \\ 2{\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-\mathbf{Z}{\widehat{\mathit{\beta }}}_n)\Vert }_2\le {\mathrm{\lambda }}_{n2}{w}_{\mathit{\text{nj}}},\hfill & \Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert =0,j\ge 1.\hfill \end{array}$$ (31)

Let ν_n = (w_njβ̂_j/(2‖β̂_nj‖), j ∈ A₁)′. Define

$${\widehat{\mathit{\beta }}}_{n{A}_1}={({\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1})}^{-1}({\mathbf{Z}}_{A_1}^{\prime }\mathbf{Y}-{\mathrm{\lambda }}_{n2}{\mathit{\nu }}_n).$$ (32)

If β̂_nA1 =₀ β_nA1, then the equation in (31) holds for ${\widehat{\mathit{\beta }}}_n\equiv {\widehat{\mathit{\beta }}}_{{\mathit{\text{nA}}}_1}^{\prime },\mathbf{0}\prime )\prime$ . Thus, since Zβ̂_n = Z_A1β̂_nA1 for this β̂_n and {Z_j, j ∈ A₁} are linearly independent,

$${\widehat{\mathit{\beta }}}_n{=}_0{\mathit{\beta }}_n\text{\hspace{1em}\hspace{1em}if}\{\begin{array}{cc}{\widehat{\mathit{\beta }}}_{n{A}_1{=}_0}{\mathit{\beta }}_{n{A}_1},\hfill & \hfill \\ {\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})\Vert }_2\le {\lambda }_{n2}{w}_{\mathit{\text{nj}}}/2,\hfill & & \forall j\notin A_1.\hfill \end{array}$$

This is true if

$${\widehat{\mathit{\beta }}}_n{=}_0{\mathit{\beta }}_n\text{ if}\{\begin{array}{cc}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2-{\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }_2<{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2,\hfill & \forall j\in A_1,\hfill \\ {\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})\Vert }_2\le {\lambda }_{n2}{w}_{\mathit{\text{nj}}}/2,\hfill & \forall j\notin A_1.\hfill \end{array}$$

Therefore,

$$\mathrm{P}({\widehat{\mathit{\beta }}}_n{\ne }_0{\mathit{\beta }}_n)\le \mathrm{P}({\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2\ge {\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2,\exists j\in A_1)+\mathrm{P}{\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})\Vert }_2>{\lambda }_{n2}{w}_{\mathit{\text{nj}}}/2,\exists j\notin A_1).$$

Let f_0j (X_j) = (f_0j(X_1j), …, f_0j(X_nj))′ and δ_n = ∑_j∈A1 f_0j (X_j) − Z_A1β_nA1. By Lemma 1, we have

$$n^{-1}{\Vert {\mathit{\delta }}_n\Vert }^2=O_p({\mathit{\text{qm}}}_n^{-2d}).$$ (33)

Let ${\mathbf{H}}_n={\mathbf{I}}_n-{\mathbf{Z}}_{A_1}{({\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1})}^{-1}{\mathbf{Z}}_{A_1}^{\prime }$ . By (32),

$${\widehat{\mathit{\beta }}}_{n{A}_1}-{\mathit{\beta }}_{n{A}_1}=n^{-1}{\mathbf{C}}_{A_1}^{-1}({\mathbf{Z}}_{A_1}^{\prime }({\mathit{\epsilon }}_n+{\mathit{\delta }}_n)-{\lambda }_{n2}{\mathit{\nu }}_n)$$ (34)

and

$$\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1}={\mathbf{H}}_n{\mathit{\epsilon }}_n+{\mathbf{H}}_n{\mathit{\delta }}_n+{\lambda }_{n2}{\mathbf{Z}}_{A_1}{\mathbf{C}}_{A_1}^{-1}{\mathbf{\nu }}_n/n.$$ (35)

Based on these two equations, Lemma 5 below shows that

$$\mathrm{P}({\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2\ge {\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2,\exists j\in A_1)\to 0,$$

and Lemma 6 below shows that

$$\mathrm{P}({\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})\Vert }_2>{\lambda }_{n2}{w}_{\mathit{\text{nj}}}/2,\exists j\notin A_1)\to 0.$$

These two equations lead to part (i) of the theorem.

We now prove part (ii) of Theorem 3. As in (26), for η_n = Y − Zβ_n and

$${\mathit{\eta }}_{n1}^{*}={\mathbf{Z}}_{A_1}{({\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1})}^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\eta }}_n,$$

we have

$${\Vert {\mathit{\eta }}_{n1}^{*}\Vert }_2^2\le 2{\Vert {\mathit{\epsilon }}_{n1}^{*}\Vert }_2^2+O_p(1)+O({\mathit{\text{qnm}}}_n^{-2d}),$$ (36)

where ${\mathit{\epsilon }}_{n1}^{*}$ is the projection of ε_n = (ε₁, …, ε_n)′ to the span of Z_A1. We have

$${\Vert {\mathit{\epsilon }}_{n1}^{*}\Vert }_2^2={\Vert {({\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1})}^{-1/2}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2\le \frac{1}{n{\rho }_{n1}}{\Vert {\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2^2=O_p(1)\frac{|A_1|}{{\rho }_{n1}}.$$ (37)

Now similarly to the proof of (25), we can show that

$${\Vert {\widehat{\mathit{\beta }}}_{n{A}_1}-{\mathit{\beta }}_{n{A}_1}\Vert }_2^2\le \frac{8{\Vert {\mathit{\eta }}_{n1}^{*}\Vert }_2^2}{n{\rho }_{n1}}+\frac{4{\lambda }_{n2}^2|A_1|}{n^2{\rho }_{n1}^2}.$$ (38)

Combining (36), (37) and (38), we get

$${\Vert {\widehat{\mathit{\beta }}}_{n{A}_1}-{\mathit{\beta }}_{n{A}_1}\Vert }_2^2=O_p\left(\frac{8}{n{\rho }_{n1}^2}\right)+O_p\left(\frac{1}{n{\rho }_{n1}}\right)+O\left(\frac{1}{m_n^{2d-1}}\right)+O\left(\frac{4{\lambda }_{n2}^2}{n^2{\rho }_{n1}^2}\right).$$

Since ${\rho }_{n1}{\asymp }_p{m}_n^{-1}$ , the result follows.

The following lemmas are needed in the proof of Theorem 3.

LEMMA 4. For ν_n = (w_njβ̃_j/(2‖β̃_nj‖), j ∈ A₁)′, under condition (B1),

$${\Vert {\mathit{\nu }}_n\Vert }^2=O_p(h_n^2)=O_p({(b_{n1}^2{c}_b)}^{-2}{r}_n^{-1}+q{b}_{n1}^{-1}).$$

PROOF. Write

$${\Vert {\mathit{\nu }}_n\Vert }^2={\displaystyle \sum _{j\in A_1}{w}_j^2=}{\displaystyle \sum _{j\in A_1}{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }^{-2}={\displaystyle \sum _{j\in A_1}\frac{{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^2-{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }^2}{{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^2·{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }^2}}}+{\displaystyle \sum _{j\in A_1}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^{-1}}.$$

Under (B2),

$${\displaystyle \sum _{j\in A_1}\frac{|{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^2-{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }^2|}{{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^2·{\Vert {\tilde{\mathit{\beta }}}_{\mathit{\text{nj}}}\Vert }^2}\le M{c}_b^{-2}{b}_{n1}^{-4}\Vert {\tilde{\mathit{\beta }}}_n-{\mathit{\beta }}_n\Vert }$$

and ${\displaystyle {\sum }_{j\in A_1}{\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }^{-2}\le {\mathit{\text{qb}}}_{n1}^{-2}}$ . The claim follows.

Let ρ_n3 be the maximum of the largest eigenvalues of $n^{-1}{\mathbf{Z}}_j^{\prime }{\mathbf{Z}}_j,j\in A_0$ , that is, ${\rho }_{n3}={\text{max }}_{j\in A_0}{\Vert n^{-1}{\mathbf{Z}}_j^{\prime }{\mathbf{Z}}_j\Vert }_2$ . By Lemma 3,

$$b_{n1}\asymp O(m_n^{1/2}),{\rho }_{n1}{\asymp }_p{m}_n^{-1},{\rho }_{n2}{\asymp }_p{m}_n^{-1}\text{\hspace{1em}and\hspace{1em}}{\rho }_{n3}{\asymp }_p{m}_n^{-1}.$$ (39)

LEMMA 5. Under conditions (B1), (B2), (A3) and (A4),

$$\mathrm{P}({\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2\ge {\Vert {\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2,\exists j\in A_1)\to 0.$$ (40)

PROOF. Let T_nj be an m_n × qm_n matrix with the form

$${\mathbf{T}}_{\mathit{\text{nj}}}=({\mathbf{0}}_{m_n},\dots ,{\mathbf{0}}_{m_n},{\mathbf{I}}_{m_n},{\mathbf{0}}_{m_n},\dots ,{\mathbf{0}}_{m_n}),$$

where O_mn is an m_n × m_n matrix of zeros and I_mn is an m_n × m_n identity matrix, and I_mn is at the jth block. By (34), ${\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}=n^{-1}{\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}({\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n+{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\delta }}_n-{\lambda }_{n2}{\mathit{\nu }}_n)$ . By the triangle inequality,

$${\Vert {\widehat{\mathit{\beta }}}_{\mathit{\text{nj}}}-{\mathit{\beta }}_{\mathit{\text{nj}}}\Vert }_2\le n^{-1}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2+n^{-1}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\delta }_n\Vert }_2+n^{-1}{\lambda }_{n2}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathit{\nu }}_n\Vert }_2.$$ (41)

Let C be a generic constant independent of n. The first term on the right-hand side

$$\begin{array}{cc}\underset{j\in A_1}{\text{max }}{n}^{-1}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2\hfill & \le n^{-1}{\rho }_{n1}^{-1}{\Vert {\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2\hfill \\ \hfill & =n^{-1/2}{\rho }_{n1}^{-1}{\Vert n^{-1/2}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\epsilon }}_n\Vert }_2\hfill \\ \hfill & =O_p(1)n^{-1/2}{\rho }_{n1}^{-1}{m}_n^{-1/2}{({\mathit{\text{qm}}}_n)}^{1/2}.\hfill \end{array}$$ (42)

By (33), the second term

$$\begin{array}{cc}\underset{j\in A_1}{\text{max }}{n}^{-1}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathit{\delta }}_n\Vert }_2\hfill & \le {\Vert {\mathbf{C}}_{A_1}^{-1}\Vert }_2·{\Vert n^{-1}{\mathbf{Z}}_{A_1}^{\prime }{\mathbf{Z}}_{A_1}\Vert }_2^{1/2}·{\Vert n^{-1}{\mathit{\delta }}_n\Vert }_2\hfill \\ \hfill & =O_p(1){\rho }_{n1}^{-1}{\rho }_{n2}^{1/2}{q}^{1/2}{m}_n^{-d}.\hfill \end{array}$$ (43)

By Lemma 4, the third term

$$\underset{j\in A_1}{\text{max }}{n}^{-1}{\lambda }_{n2}{\Vert {\mathbf{T}}_{\mathit{\text{nj}}}{\mathbf{C}}_{A_1}^{-1}{\mathit{\nu }}_n\Vert }_2\le n{\lambda }_{n2}{\rho }_{n1}^{-1}{\Vert {\mathit{\nu }}_n\Vert }_2=O_p(1){\rho }_{n1}^{-1}{n}^{-1}{\lambda }_{n2}{h}_n.$$ (44)

Thus, (40) follows from (39), (42)–(44) and condition (B2a).

LEMMA 6. Under conditions (B1), (B2), (A3) and (A4),

$$\mathrm{P}({\Vert {\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})\Vert }_2>{\lambda }_{n2}{w}_{\mathit{\text{nj}}}/2,\exists j\notin A_1)\to 0.$$ (45)

PROOF. By (35), we have

$${\mathbf{Z}}_j^{\prime }(\mathbf{Y}-{\mathbf{Z}}_{A_1}{\widehat{\mathit{\beta }}}_{n{A}_1})={\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\epsilon }}_n+{\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\delta }}_n+\lambda n^{-1}{\mathbf{Z}}_j^{\prime }{\mathbf{Z}}_{A_1}{\mathbf{C}}_{A_1}^{-1}{\mathit{\nu }}_n.$$ (46)

Recall s_n = p − q is the number of zero components in the model. By Lemma 2,

$$\mathrm{E}\left(\underset{j\notin A_1}{\text{max }}{\Vert n^{-1/2}{\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\epsilon }}_n\Vert }_2\right)\le O(1){\{\text{log}(s_n{m}_n)\}}^{1/2}.$$ (47)

Since w_nj = ‖β̂_nj‖⁻¹ = O_p (r_n) for j ∉ A₁ and by (47), for the first term on the right-hand side of (46), we have

$$\begin{array}{c}\mathrm{P}({\Vert {\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\epsilon }}_n\Vert }_2>{\lambda }_{n2}{w}_{\mathit{\text{nj}}}/6,\exists j\notin A_1)\hfill \\ \le \mathrm{P}({\Vert {\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\epsilon }}_n\Vert }_2>C{\lambda }_{n2}{r}_n,\exists j\notin A_1)+o(1)\hfill \\ =\mathrm{P}\left(\underset{j\notin A_1}{\text{max }}{\Vert n^{-1/2}{\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\epsilon }}_n\Vert }_2>C{n}^{-1/2}{\lambda }_{n2}{r}_n\right)+o(1)\hfill \\ \le O(1)\frac{n^{1/2}{\{\text{log}(s_n{m}_n)\}}^{1/2}}{C{\lambda }_{n2}{r}_n}+o(1).\hfill \end{array}$$ (48)

By (33), the second term on the right-hand side of (46)

$$\begin{array}{cc}\underset{j\notin A_1}{\text{max }}{\Vert {\mathbf{Z}}_j^{\prime }{\mathbf{H}}_n{\mathit{\delta }}_n\Vert }_2\hfill & \le n^{1/2}\underset{j\notin A_1}{\text{max }}{\Vert n^{-1}{\mathbf{Z}}_j^{\prime }{\mathbf{Z}}_j\Vert }_2^{1/2}·{\Vert {\mathbf{H}}_n\Vert }_2·{\Vert {\mathit{\delta }}_n\Vert }_2\hfill \\ \hfill & =O(1)n{\rho }_{n3}^{1/2}{q}^{1/2}{m}_n^{-d}.\hfill \end{array}$$ (49)

By Lemma 4, the third term on the right-hand side of (46)

$$\begin{array}{c}\underset{j\notin A_1}{\text{ max }}{\lambda }_{n2}{n}^{-1}{\Vert {\mathbf{Z}}_j{\mathbf{Z}}_{A_1}{\mathbf{C}}_{A_1}^{-1}{\mathit{\nu }}_n\Vert }_2\hfill \\ \le {\lambda }_{n2}\underset{j\notin A_1}{\text{ max }}{\Vert n^{-1/2}{\mathbf{Z}}_j\Vert }_2·{\Vert n^{-1/2}{\mathbf{Z}}_{A_1}{\mathbf{C}}_{A_1}^{-1/2}\Vert }_2·{\Vert {\mathbf{C}}_{A_1}^{-1/2}\Vert }_2·{\Vert {\mathit{\nu }}_n\Vert }_2\hfill \\ ={\lambda }_{n2}{\rho }_{n3}^{1/2}{\rho }_{n1}^{-1/2}{O}_p(q{b}_{n1}^{-1}).\hfill \end{array}$$ (50)

Therefore, (45) follows from (39), (48), (49), (50) and condition (B2b).

PROOF OF THEOREM 4. The proof is similar to that of Theorem 2 and is omitted.