ADAPTIVE ROBUST VARIABLE SELECTION

Abstract

Heavy-tailed high-dimensional data are commonly encountered in various scientific fields and pose great challenges to modern statistical analysis. A natural procedure to address this problem is to use penalized quantile regression with weighted L₁-penalty, called weighted robust Lasso (WR-Lasso), in which weights are introduced to ameliorate the bias problem induced by the L₁-penalty. In the ultra-high dimensional setting, where the dimensionality can grow exponentially with the sample size, we investigate the model selection oracle property and establish the asymptotic normality of the WR-Lasso. We show that only mild conditions on the model error distribution are needed. Our theoretical results also reveal that adaptive choice of the weight vector is essential for the WR-Lasso to enjoy these nice asymptotic properties. To make the WR-Lasso practically feasible, we propose a two-step procedure, called adaptive robust Lasso (AR-Lasso), in which the weight vector in the second step is constructed based on the L₁-penalized quantile regression estimate from the first step. This two-step procedure is justified theoretically to possess the oracle property and the asymptotic normality. Numerical studies demonstrate the favorable finite-sample performance of the AR-Lasso.

1. Introduction

The advent of modern technology makes it easier to collect massive, large-scale data sets. A common feature of these data sets is that the number of covariates greatly exceeds the number of observations, a regime opposite to conventional statistical settings. For example, portfolio allocation with hundreds of stocks in finance involves a covariance matrix of about tens of thousands of parameters, but the sample sizes are often only in the order of hundreds (e.g., daily data over a year period (Fan et al., 2008)). Genome-wide association studies in biology involve hundreds of thousands of single-nucleotide polymorphisms (SNPs), but the available sample size is usually in hundreds too. Data-sets with large number of variables but relatively small sample size pose great unprecedented challenges, and opportunities, for statistical analysis.

Regularization methods have been widely used for high-dimensional variable selection (Bickel and Li, 2006; Bickel et al., 2009; Efron et al., 2007; Fan and Li, 2001; Lv and Fan, 2009; Tibshirani, 1996; Zhang, 2010; Zou, 2006). Yet, most existing methods such as penalized least-squares or penalized likelihood (Fan and Lv, 2011) are designed for light-tailed distributions. Zhao and Yu (2006) established the irrepresentability conditions for the model selection consistency of the Lasso estimator. Fan and Li (2001) studied the oracle properties of nonconcave penalized likelihood estimators for fixed dimensionality. Lv and Fan (2009) investigated the penalized least-squares estimator with folded-concave penalty functions in the ultra-high dimensional setting and established a nonasymptotic weak oracle property. Fan and Lv (2008) proposed and investigated the sure independence screening method in the setting of light-tailed distributions. The robustness of the aforementioned methods have not yet been thoroughly studied and well understood.

Robust regularization methods such as the least absolute deviation (LAD) regression and quantile regression have been used for variable selection in the case of fixed dimensionality. See, for example, Li and Zhu (2008); Wang, Li and Jiang (2007); Wu and Liu (2009); Zou and Yuan (2008). The penalized composite likelihood method was proposed in Bradic et al. (2011) for robust estimation in ultra-high dimensions with focus on the efficiency of the method. They still assumed sub-Gaussian tails. Belloni and Chernozhukov (2011) studied the L₁-penalized quantile regression in high-dimensional sparse models where the dimensionality could be larger than the sample size. We refer to their method as robust Lasso (R-Lasso). They showed that the R-Lasso estimate is consistent at the near-oracle rate, and gave conditions under which the selected model includes the true model, and derived bounds on the size of the selected model, uniformly in a compact set of quantile indices. Wang (2012) studied the L₁-penalized LAD regression and showed that the estimate achieves near oracle risk performance with a nearly universal penalty parameter and established also a sure screening property for such an estimator. van de Geer and Müller (2012) obtained bounds on the prediction error of a large class of L₁ penalized estimators, including quantile regression. Wang et al. (2012) considered the nonconvex penalized quantile regression in the ultra-high dimensional setting and showed that the oracle estimate belongs to the set of local minima of the nonconvex penalized quantile regression, under mild assumptions on the error distribution.

In this paper, we introduce the penalized quantile regression with the weighted L₁-penalty (WR-Lasso) for robust regularization, as in Bradic et al. (2011). The weights are introduced to reduce the bias problem induced by the L₁-penalty. The exibility of the choice of the weights provides exibility in shrinkage estimation of the regression coefficient. WR-Lasso shares a similar spirit to the folded-concave penalized quantile-regression (Wang et al., 2012; Zou and Li, 2008), but avoids the nonconvex optimization problem. We establish conditions on the error distribution in order for the WR-Lasso to successfully recover the true underlying sparse model with asymptotic probability one. It turns out that the required condition is much weaker than the sub-Gaussian assumption in Bradic et al. (2011). The only conditions we impose is that the density function of error has Lipschitz property in a neighborhood around 0. This includes a large class of heavy-tailed distributions such as the stable distributions, including the Cauchy distribution. It also covers the double exponential distribution whose density function is nondifferentiable at the origin.

Unfortunately, because of the penalized nature of the estimator, WR-Lasso estimate has a bias. In order to reduce the bias, the weights in WR-Lasso need to be chosen adaptively according to the magnitudes of the unknown true regression coefficients, which makes the bias reduction infeasible for practical applications.

To make the bias reduction feasible, we introduce the adaptive robust Lasso (AR-Lasso). The AR-Lasso first runs R-Lasso to obtain an initial estimate, and then computes the weight vector of the weighted L₁-penalty according to a decreasing function of the magnitude of the initial estimate. After that, AR-Lasso runs WR-Lasso with the computed weights. We formally establish the model selection oracle property of AR-Lasso in the context of Fan and Li (2001) with no assumptions made on the tail distribution of the model error. In particular, the asymptotic normality of the AR-Lasso is formally established.

This paper is organized as follows. First, we introduce our robust estimators in Section 2. Then, to demonstrate the advantages of our estimator, we show in Section 3 with a simple example that Lasso behaves sub-optimally when noise has heavy tails. In Section 4.1, we study the performance of the oracle-assisted regularization estimator. Then in Section 4.2, we show that when the weights are adaptively chosen, WR-Lasso has the model selection oracle property, and performs as well as the oracle-assisted regularization estimate. In Section 4.3, we prove the asymptotic normality of our proposed estimator. The feasible estimator, AR-Lasso, is investigated in Section 5. Section 6 presents the results of the simulation studies. Finally, in Section 7 we present the proofs of the main theorems. Additional proofs, as well as the results of a genome-wide association study, are provided in the supplementary Appendix (Fan et al., 2013).

2. Adaptive Robust Lasso

Consider the linear regression model

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },$$ (2.1)

where y is an n-dimensional response vector, X = (x₁, …, x_n)^T = (x̃₁, ···, x̃_p) is an n × p fixed design matrix, β = (β₁,…, β_p)^T is a p-dimensional regression coefficient vector, and ε = (ε₁, …, ε_n)^T is an n-dimensional error vector whose components are independently distributed and satisfy P(ε_i ≤ 0) = τ for some known constant τ ∈ (0, 1). Under this model, ${\mathbf{x}}_i^T\mathit{\beta }$ is the conditional τth-quantile of y_i given x_i. We impose no conditions on the heaviness of the tail probability or the homoscedasticity of ε_i. We consider a challenging setting in which log p = o(n^b) with some constant b > 0. To ensure the model identifiability and to enhance the model fitting accuracy and interpretability, the true regression coefficient vector β^* is commonly imposed to be sparse with only a small proportion of nonzeros (Fan and Li, 2001; Tibshirani, 1996). Denoting the number of nonzero elements of the true regression coefficients by s_n, we allow s_n to slowly diverge with the sample size n and assume that s_n = o(n). To ease the presentation, we suppress the dependence of s_n on n whenever there is no confusion. Without loss of generality, we write ${\mathit{\beta }}^{\ast }={({\mathit{\beta }}_1^{\ast T},{\mathbf{0}}^T)}^T$, i.e. only the first s entries are non-vanishing. The true model is denoted by

$${\mathcal{M}}_{\ast }=\text{supp}({\mathit{\beta }}^{\ast })=\{1,\cdots ,s\},$$

and its complement, ${\mathcal{M}}_{\ast }^c=\{s+1,\cdots ,p\}$, represents the set of noise variables.

We consider a fixed design matrix in this paper and denote by S = (S₁, ···, S_n)^T = (x̃₁, ···, x̃_s) the submatrix of X corresponding to the covariates whose coefficients are non-vanishing. These variables will be referred to as the signal covariates and the rest will be called noise covariates. The set of columns that correspond to the noise covariates is denoted by Q = (Q₁, ···, Q_n)^T = (x̃_s+1, ···, x̃_p). We standardize each column of X to have L₂-norm $\sqrt{n}$.

To recover the true model and estimate β^*, we consider the following regularization problem

$$\underset{\mathit{\beta }\in {\mathbf{R}}^p}{min}\left\{\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathbf{x}}_i^T\mathit{\beta })+n{\lambda }_n\sum _{j=1}^p{p}_{{\lambda }_n}(\mid {\beta }_j\mid )\right\},$$ (2.2)

where ρ_τ (u) = u(τ − 1{u ≤ 0}) is the quantile loss function, and p_λn(·) is a nonnegative penalty function on [0, ∞) with a regularization parameter λ_n ≥ 0. The use of quantile loss function in (2.2) is to overcome the difficulty of heavy tails of the error distribution. Since P(ε ≤ 0) = τ, (2.2) can be interpreted as the sparse estimation of the conditional τth quantile. Regarding the choice of p_λn(·), it was demonstrated in Lv and Fan (2009) and Fan and Lv (2011) that folded-concave penalties are more advantageous for variable selection in high dimensions than the convex ones such as the L₁-penalty. It is, however, computationally more challenging to minimize the objective function in (2.2) when p_λ(·) is folded-concave. Noting that with a good initial estimate ${\widehat{\mathit{\beta }}}^{\mathit{ini}}={({\widehat{\beta }}_1^{\mathit{ini}},\cdots ,{\widehat{\beta }}_p^{\mathit{ini}})}^T$ of the true coefficient vector, we have

$$p_{{\lambda }_n}(\mid {\beta }_j\mid )\approx p_{{\lambda }_n}(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid )+p_{{\lambda }_n}^{\prime }(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid )\left(\mid {\beta }_j\mid -\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid \right).$$

Thus, instead of (2.2) we consider the following weighted L₁-regularized quantile regression

$$L_n(\mathit{\beta })=\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathbf{x}}_i^T\mathit{\beta })+n{\lambda }_n{\Vert \mathbf{d}\circ \mathit{\beta }\Vert }_1,$$ (2.3)

where d = (d₁, ···, d_p)T is the vector of non-negative weights, and ∘ is the Hadamard product, i.e., the componentwise product of two vectors. This motivates us to define the weighted robust Lasso (WR-Lasso) estimate as the global minimizer of the convex function L_n(β) for a given non-stochastic weight vector:

$$\widehat{\mathit{\beta }}=arg{min}_{\mathit{\beta }}{L}_n(\mathit{\beta }).$$ (2.4)

The uniqueness of the global minimizer is easily guaranteed by adding a negligible L₂-regularization in implementation. In particular, when d_j = 1 for all j, the method will be referred to as robust Lasso (R-Lasso).

The adaptive robust Lasso (AR-Lasso) refers specifically to the two-stage procedure in which the stochastic weights ${\widehat{d}}_j=p_{{\lambda }_n}^{\prime }(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid )$ for j = 1, ···, p are used in the second step for WR-Lasso and are constructed using a concave penalty p_λn(·) and the initial estimates, ${\widehat{\beta }}_j^{\mathit{ini}}$, from the first step. In practice, we recommend using R-Lasso as the initial estimate and then using SCAD to compute the weights in AR-Lasso. The asymptotic result of this specific AR-Lasso is summarized in Corollary 1 in Section 5 for the ultra-high dimensional robust regression problem. This is a main contribution of the paper.

3. Suboptimality of Lasso

In this section, we use a specific example to illustrate that, in the case of heavy-tailed error distribution, Lasso fails at model selection unless the non-zero coefficients, ${\beta }_1^{\ast },\cdots ,{\beta }_s^{\ast }$, have a very large magnitude. We assume that the errors ε₁, ···, ε_n have the identical symmetric stable distribution and the characteristic function of ε₁ is given by

$$E[exp(iu{\epsilon }_1)]=exp(-{\mid u\mid }^{\alpha }),$$

where α ∈ (0, 2). By Nolan (2012), E|ε₁|^p is finite for 0 < p < α, and E|ε₁|^p = ∞ for p ≥ α. Furthermore as z → ∞,

$$P(\mid {\epsilon }_1\mid \ge z)~c_a{z}^{-\alpha },$$

where $c_{\alpha }=sin(\frac{\pi \alpha }{2})\mathrm{\Gamma }(\alpha )/\pi$ is a constant depending only on α, and we use the notation ~ to denote that two terms are equivalent up to some constant. Moreover, for any constant vector a = (a₁, ···, a_n)^T, the linear combination a^Tε has the following tail behavior

$$P(\mid {\mathbf{a}}^T\mathit{\epsilon }\mid >z)~{\Vert \mathbf{a}\Vert }_{\alpha }^{\alpha }{c}_{\alpha }{z}^{-\alpha },$$ (3.1)

with ||·||_α denoting the L_α-norm of a vector.

To demonstrate the suboptimality of Lasso, we consider a simple case in which the design matrix satisfies the conditions that S^TQ = 0, $\frac{1}{n}{\mathbf{S}}^T\mathbf{S}={\mathbf{I}}_s$, the columns of Q satisfy |supp(x̃_j)| = m_n = O(n^1/2) and supp(x̃_k) ∩ supp(x̃_j) = Ø for any k ≠ j and k, j ∈ {s + 1, ···, p}. Here, m_n is a positive integer measuring the sparsity level of the columns of Q. We assume that there are only fixed number of true variables, i.e., s is finite, and that max_ij |x_ij| = O(n^1/4). Thus, it is easy to see that p = O(n^1/2). In addition, we assume further that all nonzero regression coefficients are the same and ${\beta }_1^{\ast }=\cdots ={\beta }_s^{\ast }={\beta }_0>0$.

We first consider R-Lasso, which is the global minimizer of (2.4). We will later see in Theorem 2 that by choosing the tuning parameter

$${\lambda }_n=O({(logn)}^2\sqrt{(logp)/n}),$$

R-Lasso can recover the true support = {1, ···, s} with probability tending to 1. Moreover, the signs of the true regression coefficients can also be recovered with asymptotic probability one as long as the following condition on signal strength is satisfied

$${\lambda }_n^{-1}{\beta }_0\to \infty ,\mathrm{i}.\mathrm{e}.{(logn)}^{-2}\sqrt{n/(logp)}{\beta }_0\to \infty .$$ (3.2)

Now, consider Lasso, which minimizes

$${\stackrel{\sim }{L}}_n(\mathit{\beta })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }_2^2+n{\lambda }_n{\Vert \mathit{\beta }\Vert }_1.$$ (3.3)

We will see that for (3.3) to recover the true model and the correct signs of coefficients, we need a much stronger signal level than that is given in (3.2). By results in optimization theory, the Karush–Kuhn–Tucker (KKT) conditions guaranteeing the necessary and sufficient conditions for β̃ with = supp(β̃) being a minimizer to (3.3) are

$$\begin{array}{c}{\stackrel{\sim }{\mathit{\beta }}}_{\mathcal{M}}+n{\lambda }_n{({\mathbf{X}}_{\mathcal{M}}^T{\mathbf{X}}_{\mathcal{M}})}^{-1}sgn({\stackrel{\sim }{\mathit{\beta }}}_{\mathcal{M}})={({\mathbf{X}}_{\mathcal{M}}^T{\mathbf{X}}_{\mathcal{M}})}^{-1}{\mathbf{X}}_{\mathcal{M}}^T\mathbf{y},\\ {\Vert {\mathbf{X}}_{{\mathcal{M}}^c}^T(\mathbf{y}-{\mathbf{X}}_{\mathcal{M}}{\stackrel{\sim }{\mathit{\beta }}}_{\mathcal{M}})\Vert }_{\infty }\le n{\lambda }_n,\end{array}$$

where is the complement of , is the subvector formed by entries of β with indices in , and and are the submatrices formed by columns of X with indices in and , respectively. It is easy to see from the above two conditions that for Lasso to enjoy the sign consistency, sgn(β̃) = sgn(β^*) with asymptotic probability one, we must have these two conditions satisfied with = with probability tending to 1. Since we have assumed that Q^TS = 0 and n⁻¹S^TS = I, the above sufficient and necessary conditions can also be written as

$${\stackrel{\sim }{\mathit{\beta }}}_{{\mathcal{M}}^{\ast }}+{\lambda }_{n}sgn({\stackrel{\sim }{\mathit{\beta }}}_{{\mathcal{M}}^{\ast }})={\mathit{\beta }}_{{\mathcal{M}}^{\ast }}^{\ast }+n^{-1}{\mathbf{S}}^T\mathit{\epsilon },$$ (3.4)

$${\Vert {\mathbf{Q}}^T\mathit{\epsilon }\Vert }_{\infty }\le n{\lambda }_n,$$ (3.5)

Conditions (3.4) and (3.5) are hard for Lasso to hold simultaneously. The following proposition summarizes the necessary condition, whose proof is given in the supplementary material (Fan et al., 2013).

Proposition 1

In the above model, with probability at least 1 − e^−c̃₀, where c̃₀ is some positive constant, Lasso does not have sign consistency, unless the following signal condition is satisfied

$$n^{\frac{3}{4}-\frac{1}{\alpha }}{\beta }_0\to \infty .$$ (3.6)

Comparing this with (3.2), it is easy to see that even in this simple case, Lasso needs much stronger signal levels than R-Lasso in order to have a sign consistency in the presence of a heavy-tailed distribution.

4. Model Selection Oracle Property

In this section, we establish the model selection oracle property of WR-Lasso. The study enables us to see the bias due to penalization, and that an adaptive weighting scheme is needed in order to eliminate such a bias. We need the following condition on the distribution of noise.

Condition 1

There exist universal constants c₁ > 0 and c₂ > 0 such that for any u satisfying |u| ≤ c₁, f_i(u)’s are uniformly bounded away from 0 and ∞ and

$$\mid F_i(u)-F_i(0)-u{f}_i(0)\mid \le c_2{u}^2,$$

where f_i(u) and F_i(u) are the density function and distribution function of the error ε_i, respectively.

Condition 1 implies basically that each f_i(u) is Lipschitz around the origin. Commonly used distributions such as the double-exponential distribution and stable distributions including the Cauchy distribution all satisfy this condition.

Denote by H = diag{f₁(0), ···, f_n(0)}. The next condition is on the sub-matrix of X that corresponds to signal covariates and the magnitude of the entries of X.

Condition 2

The eigenvalues of $\frac{1}{n}{\mathbf{S}}^T\mathbf{HS}$ are bounded from below and above by some positive constants c₀ and 1/c₀, respectively. Furthermore,

$${\kappa }_n\equiv \underset{ij}{max}\mid x_{ij}\mid =o(\sqrt{n}{s}^{-1}).$$

Although Condition 2 is on the fixed design matrix, we note that the above condition on κ_n is satisfied with asymptotic probability one when the design matrix is generated from some distributions. For instance, if the entries of X are independent copies from a sub-exponential distribution, the bound on κ_n is satisfied with asymptotic probability one as long as $s=o(\sqrt{n}/(logp))$; if the components are generated from sub-Gaussian distribution, then the condition on κ_n is satisfied with probability tending to one when $s=o(\sqrt{n/(logp)})$.

4.1. Oracle Regularized Estimator

To evaluate our newly proposed method, we first study how well one can do with the assistance of the oracle information on the locations of signal covariates. Then, we use this to establish the asymptotic property of our estimator without the oracle assistance. Denote by ${\widehat{\mathit{\beta }}}^o={({({\widehat{\mathit{\beta }}}_1^o)}^T,{\mathbf{0}}^T)}^T$ the oracle regularized estimator (ORE) with ${\widehat{\mathit{\beta }}}_1^o\in {\mathbf{R}}^s$ and 0 being the vector of all zeros, which minimizes L_n(β) over the space { $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in {\mathbf{R}}^p:{\mathit{\beta }}_2=\mathbf{0}\in {\mathbf{R}}^{p-s}$}. The next theorem shows that ORE is consistent, and estimates the correct sign of the true coefficient vector with probability tending to one. We use d₀ to denote the first s elements of d.

Theorem 1

Let ${\gamma }_n=C_1(\sqrt{s(logn)/n}+{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2)$ with C₁ > 0 a constant. If Conditions 1 and 2 hold and ${\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2\sqrt{s}{\kappa }_n\to 0$, then there exists some constant c > 0 such that

$$P\left({\Vert {\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast }\Vert }_2\le {\gamma }_n\right)\ge 1-n^{-cs}.$$ (4.1)

If in addition ${\gamma }_n^{-1}{min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid \to \infty$, then with probability at least 1−n^−cs,

$$sgn({\widehat{\mathit{\beta }}}_1^o)=sgn({\mathit{\beta }}_1^{\ast }),$$

where the above equation should be understood componentwisely.

As shown in Theorem 1, the consistency rate of ${\widehat{\mathit{\beta }}}_1^o$ in terms of the vector L₂-norm is given by γ_n. The first component of γ_n, $C_1\sqrt{s(logn)/n}$, is the oracle rate within a factor of log n, and the second component C₁λ_n||d₀||₂ reflects the bias due to penalization. If no prior information is available, one may choose equal weights d₀ = (1, 1, ···, 1)^T, which corresponds to R-Lasso. Thus for R-Lasso, with probability at least 1 − n^−cs, it holds that

$${\Vert {\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast }\Vert }_2\le C_1(\sqrt{s(logn)/n}+\sqrt{s}{\lambda }_n).$$ (4.2)

4.2. WR-Lasso

In this section, we show that even without the oracle information, WR-Lasso enjoys the same asymptotic property as in Theorem 1 when the weight vector is appropriately chosen. Since the regularized estimator β̂ in (2.4) depends on the full design matrix X, we need to impose the following conditions on the design matrix to control the correlation of columns in Q and S.

Condition 3

With γ_n defined in Theorem 1, it holds that

$${\Vert \frac{1}{n}{\mathbf{Q}}^T\mathbf{HS}\Vert }_{2,\infty }<\frac{{\lambda }_n}{2{\Vert {\mathbf{d}}_1^{-1}\Vert }_{\infty }{\gamma }_n},$$

where ||A||_2,∞ = sup_x≠0 ||Ax||_∞/||x||₂ for a matrix A and vector x, and ${\mathbf{d}}_1^{-1}={(d_{s+1}^{-1},\cdots ,d_p^{-1})}^T$. Furthermore, log(p) = o(n^b) for some constant b ∈ (0, 1).

To understand the implications of Condition 3, we consider the case of f₁(0) = ⋯ = f_n(0) ≡ f(0). In the special case of Q^T S = 0, Condition 3 is satisfied automatically. In the case of equal correlation, that is, n⁻¹X^TX having off-diagonal elements all equal to ρ, the above Condition 3 reduces to

$$\mid \rho \mid <\frac{{\lambda }_n}{4f(0){\Vert {\mathbf{d}}_1^{-1}\Vert }_{\infty }\sqrt{s}{\gamma }_n}.$$

This puts an upper bound on the correlation coefficient ρ for such a dense matrix.

It is well known that for Gaussian errors, the optimal choice of regularization parameter λ_n has the order $\sqrt{(logp)/n}$ (Bickel et al., 2009). The distribution of the model noise with heavy tails demands a larger choice of λ_n to filter the noise for R-lasso. When ${\lambda }_n\ge \sqrt{(logn)/n}$, γ_n given in (4.2) is in the order of $C_1{\lambda }_n\sqrt{s}$. In this case, Condition 3 reduces to

$${\Vert n^{-1}{\mathbf{Q}}^T\mathbf{HS}\Vert }_{2,\infty }<O(s^{-1/2}).$$ (4.3)

For WR-Lasso, if the weights are chosen such that ${\Vert {\mathbf{d}}_0\Vert }_2=O(\sqrt{s(logn)/n}/{\lambda }_n)$ and ||d₁||_∞ = O(1), then γ_n is in the order of $C_1\sqrt{s(logn)/n}$, and correspondingly, Condition 3 becomes

$${\Vert n^{-1}{\mathbf{Q}}^T\mathbf{HS}\Vert }_{2,\infty }<O({\lambda }_n\sqrt{n/(s(logn))}).$$

This is a more relaxed condition than (4.3), since with heavy-tailed errors, the optimal γ_n should be larger than $\sqrt{(logp)/n}$. In other words, WR-Lasso not only reduces the bias of the estimate, but also allows for stronger correlations among the signal and noise covariates. However, the above choice of weights depends on unknown locations of signals. A data-driven choice will be given in Section 5, in which the resulting AR-Lasso estimator will be studied.

The following theorem shows the model selection oracle property of the WR-Lasso estimator.

Theorem 2

Suppose Conditions 1 – 3 hold. In addition, assume that min_j≥s+1 d_j > c₃ with some constant c₃ > 0,

$${\gamma }_n{s}^{3/2}{\kappa }_n^2{({log}_{2}n)}^2=o(n{\lambda }_n^2),{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2{\kappa }_{n}max\{\sqrt{s},{\Vert {\mathbf{d}}_0\Vert }_2\}\to 0,$$ (4.4)

and ${\lambda }_n>2\sqrt{(1+c)(logp)/n}$, where κ_n is defined in Condition 2, γ_n is defined in Theorem 1, and c is some positive constant. Then, with probability at least 1− O(n^−cs), there exists a global minimizer $\widehat{\mathit{\beta }}={({({\widehat{\mathit{\beta }}}_1^o)}^T,{\widehat{\mathit{\beta }}}_2^T)}^T$ of L_n(β) which satisfies

1. β̂₂ = 0;

2. ${\Vert {\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast }\Vert }_2\le {\gamma }_n$.

Theorem 2 shows that the WR-Lasso estimator enjoys the same property as ORE with probability tending to one. However, we impose non-adaptive assumptions on the weight vector $\mathbf{d}={({\mathbf{d}}_0^T,{\mathbf{d}}_1^T)}^T$. For noise covariates, we assume min_j>s d_j > c₃, which implies that each coordinate needs to be penalized. For the signal covariates, we impose (4.4), which requires ||d₀||₂ to be small.

When studying the nonconvex penalized quantile regression, Wang et al. (2012) assumed that κ_n is bounded and the density functions of ε_i’s are uniformly bounded away from 0 and ∞ in a small neighborhood of 0. Their assumption on the error distribution is weaker than our Condition 1. We remark that the difference is because we have weaker conditions on κ_n and the penalty function (See Condition 2 and (4.4)). In fact, our Condition 1 can be weakened to the same condition as that in Wang et al. (2012) at the cost of imposing stronger assumptions on κ_n and the weight vector d.

Belloni and Chernozhukov (2011) and Wang (2012) imposed the restricted eigenvalue assumption of the design matrix and studied the L₁-penalized quantile regression and LAD regression, respectively. We impose different conditions on the design matrix and allow flexible shrinkage by choosing d. In addition, our Theorem 2 provides a stronger result than consistency; we establish model selection oracle property of the estimator.

4.3. Asymptotic Normality

We now present the asymptotic normality of our estimator. Define V_n = (S^THS)^−1/2 and Z_n = (Z_n1, ⋯, Z_nn)^T = SV_n with Z_nj ∈ R^s for j = 1, ⋯, n.

Theorem 3

Assume the conditions of Theorem 2 hold, the first and second order derivatives $f_i^{\prime }(u)$ and $f_i^{″}(u)$ are uniformly bounded in a small neighborhood of 0 for all i = 1, ⋯, n, and that ${\Vert {\mathbf{d}}_0\Vert }_2=O(\sqrt{s/n}/{\lambda }_n)$, max_i ||H^1/2Z_ni||₂ = o(s^−7/2(log s)⁻¹), and $\sqrt{n/s}{min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid \to \infty$. Then, with probability tending to 1 there exists a global minimizer $\widehat{\mathit{\beta }}={({({\widehat{\mathit{\beta }}}_1^o)}^T,{\widehat{\mathit{\beta }}}_2^T)}^T$ of L_n(β) such that β̂₂ = 0. Moreover,

$${\mathbf{c}}^T{({\mathbf{Z}}_n^T{\mathbf{Z}}_n)}^{-1/2}{\mathbf{V}}_n^{-1}[({\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast })+\frac{n{\lambda }_n}{2}{\mathbf{V}}_n^2{\stackrel{\sim }{\mathbf{d}}}_0]\stackrel{\mathcal{D}}{\to }N\left(0,\tau (1-\tau )\right),$$

where c is an arbitrary s-dimensional vector satisfying c^Tc = 1, and d̃₀ is an s-dimensional vector with the jth element $d_{j}sgn({\beta }_j^{\ast })$.

The proof of Theorem 3 is an extension of the proof on the asymptotic normality theorem for the LAD estimator in Pollard (1990), in which the theorem is proved for fixed dimensionality. The idea is to approximate L_n(β₁, 0) in (2.4) by a sequence of quadratic functions, whose minimizers converge to normal distribution. Since L_n(β₁, 0) and the quadratic approximation are close, their minimizers are also close, which results in the asymptotic normality in Theorem 3.

Theorem 3 assumes that max_i ||H^1/2Z_ni||₂ = o(s^−7/2(log s)⁻¹). Since by definition ${\sum }_{i=1}^n{\Vert {\mathbf{H}}^{1/2}{\mathbf{Z}}_{ni}\Vert }_2^2=s$, it is seen that the condition implies s = o(n^1/8). This assumption is made to guarantee that the quadratic approximation is close enough to L_n(β₁, 0). When s is finite, the condition becomes max_i ||Z_ni||₂ = o(1), as in Pollard (1990). Another important assumption is ${\lambda }_n\sqrt{n}{\Vert {\mathbf{d}}_0\Vert }_2=O(\sqrt{s})$, which is imposed to make sure that the bias 2⁻¹nλ_nc^TV_nd₀ caused by the penalty term does not diverge. For instance, using R-Lasso will create a non-diminishing bias and thus cannot be guaranteed to have asymptotic normality.

Note that we do not assume a parametric form of the error distribution. Thus, our oracle estimator is in fact a semiparametric estimator with the error density as the nuisance parameter. Heuristically speaking, Theorem 3 shows that the asymptotic variance of $\sqrt{n}({\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast })$ is $n\tau (1-\tau ){\mathbf{V}}_n{\mathbf{Z}}_n^T{\mathbf{Z}}_n{\mathbf{V}}_n$. Since V_n = (S^T HS)^−1/2 and Z_n = SV_n, if the model errors ε_i are i.i.d. with density function f_ε(·), then this asymptotic variance reduces to $\tau (1-\tau ){(n^{-1}{f}_{\epsilon }^2(0){\mathbf{S}}^T\mathbf{S})}^{-1}$. In the random design case where the true covariate vectors ${\{{\mathbf{S}}_i\}}_{i=1}^n$ are i.i.d. observations, n⁻¹S^TS converges to $E[{\mathbf{S}}_1^T{\mathbf{S}}_1]$ as n → ∞, and the asymptotic variance reduces to $\tau (1-\tau ){(f_{\epsilon }^2(0)E[{\mathbf{S}}_1^T{\mathbf{S}}_1])}^{-1}$. This is the semiparametric efficiency bound derived by Newey and Powell (1990) for random designs. In fact, if we assume that (x_i, y_i) are i.i.d., then the conditions of Theorem 3 can hold with asymptotic probability one. Using similar arguments, it can be formally shown that $\sqrt{n}({\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast })$ is asymptotically normal with covariance matrix equal to the aforementioned semiparametric efficiency bound. Hence, our oracle estimator is semiparametric efficient.

5. Properties of the Adaptive Robust Lasso

In previous sections, we have seen that the choice of the weight vector d plays a pivotal role for the WR-Lasso estimate to enjoy the model selection oracle property and asymptotic normality. In fact, conditions in Theorem 2 require that min_j≥s+1 d_j > c₃ and that ||d₀||₂ does not diverge too fast. Theorem 3 imposes an even more stringent condition, ${\Vert {\mathbf{d}}_0\Vert }_2=O(\sqrt{s/n}/{\lambda }_n)$, on the weight vector d₀. For R-Lasso, ${\Vert {\mathbf{d}}_0\Vert }_2=\sqrt{s}$ and these conditions become very restrictive. For example, the condition in Theorem 3 becomes λ_n = O(n^−1/2), which is too low for a thresholding level even for Gaussian errors. Hence, an adaptive choice of weights is needed to ensure that those conditions are satisfied. To this end, we propose a two-step procedure.

In the first step, we use R-Lasso, which gives the estimate β̂ⁱⁿⁱ. As has been shown in Belloni and Chernozhukov (2011) and Wang (2012), R-Lasso is consistent at a near-oracle rate $\sqrt{s(logp)/n}$ and selects the true model as a submodel (in other words, R-Lasso has the sure screening property using the terminology of Fan and Lv (2008)) with asymptotic probability one, namely,

$$\text{supp}({\widehat{\mathit{\beta }}}^{\mathit{ini}})\supseteq \text{supp}({\mathit{\beta }}^{\ast })\text{and}{\Vert {\widehat{\mathit{\beta }}}_1^{\mathit{ini}}-{\mathit{\beta }}_1^{\ast }\Vert }_2=O(\sqrt{s(logp)/n}).$$

We remark that our Theorem 2 also ensures the consistency of R-Lasso. Compared to Belloni and Chernozhukov (2011), Theorem 2 presents stronger results but also needs more restrictive conditions for R-Lasso. As will be shown in latter theorems, only the consistency of R-Lasso is needed in the study of AR-Lasso, so we quote the results and conditions on R-Lasso in Belloni and Chernozhukov (2011) with the mind of imposing weaker conditions.

In the second step, we set d̂ = (d̂₁, ···, d̂_p)^T with ${\widehat{d}}_j=p_{{\lambda }_n}^{\prime }(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid )$ where p_λn(|·|) is a folded concave penalty function, and then solve the regularization problem (2.4) with a newly computed weight vector. Thus, vector d̂₀ is expected to be close to the vector ${(p_{{\lambda }_n}^{\prime }(\mid {\beta }_1^{\ast }\mid ),\cdots ,p_{{\lambda }_n}^{\prime }(\mid {\beta }_s^{\ast }\mid ))}^T$ under L₂-norm. If a folded concave penalty such as SCAD is used, then $p_{{\lambda }_n}^{\prime }(\mid {\beta }_j^{\ast }\mid )$ will be close, or even equal, to zero for 1 ≤ j ≤ s and thus the magnitude of ||d̂₀||₂ is negligible.

Now, we formally establish the asymptotic properties of AR-Lasso. We first present a more general result and then highlight our recommended procedure, which uses R-Lasso as the initial estimate and then uses SCAD to compute the stochastic weights, in Corollary 1. Denote by ${\mathbf{d}}^{\ast }=(d_1^{\ast },\cdots ,d_p^{\ast })$ with $d_j^{\ast }=p_{{\lambda }_n}^{\prime }(\mid {\beta }_j^{\ast }\mid )$. Using the weight vector d̂, AR-Lasso minimizes the following objective function

$${\widehat{L}}_n(\mathit{\beta })=\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathbf{x}}_i^T\mathit{\beta })+n{\lambda }_n{\Vert \widehat{\mathbf{d}}\circ \mathit{\beta }\Vert }_1.$$ (5.1)

We also need the following conditions to show the model selection oracle property of the two-step procedure.

Condition 4

With asymptotic probability one, the initial estimate satisfies ${\Vert {\widehat{\mathit{\beta }}}^{\mathit{ini}}-{\mathit{\beta }}^{\ast }\Vert }_2\le C_2\sqrt{s(logp)/n}$ with some constant C₂ > 0.

As discussed above, if R-Lasso is used to obtain the initial estimate, it satisfies the above condition. Our second condition is on the penalty function.

Condition 5

$p_{{\lambda }_n}^{\prime }(t)$ is non-increasing in t ∈ (0, ∞) and is Lipschitz with constant c₅, that is,

$$\mid p_{{\lambda }_n}^{\prime }(\mid {\beta }_1\mid )-p_{{\lambda }_n}^{\prime }(\mid {\beta }_2\mid )\mid \le c_5\mid {\beta }_1-{\beta }_2\mid ,$$

for any β₁, β₂ ∈ R. Moreover, $p_{{\lambda }_n}^{\prime }(C_2\sqrt{s(logp)/n})>\frac{1}{2}{p}_{{\lambda }_n}^{\prime }(0+)$ for large enough n, where C₂ is defined in Condition 4.

For the SCAD (Fan and Li, 2001) penalty, $p_{{\lambda }_n}^{\prime }(\beta )$ is given by

$$p_{{\lambda }_n}^{\prime }(\beta )=1\{\beta \le {\lambda }_n\}+\frac{{(a{\lambda }_n-\beta )}_{+}}{(a-1){\lambda }_n}1\{\beta >{\lambda }_n\},$$ (5.2)

for a given constant a > 2, and it can be easily verified that Condition 5 holds if ${\lambda }_n>2{(a+1)}^{-1}{C}_2\sqrt{s(logp)/n}$.

Theorem 4

Assume conditions of Theorem 2 hold with d = d^* and γ_n = a_n, where

$$a_n=C_3\left(\sqrt{s(logn)/n}+{\lambda }_n({\Vert {\mathbf{d}}_0^{\ast }\Vert }_2+C_2{c}_5\sqrt{s(logp)/n})\right),$$

with some constant C₃ > 0, and ${\lambda }_{n}s{\kappa }_n\sqrt{(logp)/n}\to 0$. Then, under Conditions 4 and 5, with probability tending to one, there exists a global minimizer $\widehat{\mathit{\beta }}={({\widehat{\mathit{\beta }}}_1^T,{\widehat{\mathit{\beta }}}_2^T)}^T$ of (5.1) such that β̂₂ = 0 and ${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le a_n$.

The results in Theorem 4 are analogous to those in Theorem 2. The extra term ${\lambda }_n\sqrt{s(logp)/n}$ in the convergence rate a_n, compared to the convergence rate γ_n in Theorem 2, is caused by the bias of the initial estimate β̂ⁱⁿⁱ. Since the regularization parameter λ_n goes to zero, the bias of AR-Lasso is much smaller than that of the initial estimator β̂ⁱⁿⁱ. Moreover, the AR-Lasso β̂ possesses the model selection oracle property.

Now we present the asymptotic normality of the AR-Lasso estimate.

Condition 6

The smallest signal satisfies ${min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid >2C_2\sqrt{(\mathit{s}logp)/n}$. Moreover, it holds that $p_{{\lambda }_n}^{″}(\mid \beta \mid )=o(s^{-1}{\lambda }_n^{-1}{(\mathit{n}logp)}^{-1/2})$ for any $\mid \beta \mid >2^{-1}{min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid$.

The above condition on the penalty function is satisfied when the SCAD penalty is used and ${min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid \ge 2a{\lambda }_n$ where a is the parameter in the SCAD penalty (5.2).

Theorem 5

Assume conditions of Theorem 3 hold with d = d^* and γ_n = a_n, where a_n is defined in Theorem 4. Then, under Conditions 4 – 6, with asymptotic probability one, there exists a global minimizer β̂ of (5.1) having the same asymptotic properties as those in Theorem 3.

With the SCAD penalty, conditions in Theorems 4 and 5 can be simplified and AR-Lasso still enjoys the same asymptotic properties, as presented in the following corollary.

Corollary 1

Assume ${\lambda }_n=O(\sqrt{s(logp)(loglog\mathit{n})/n}),logp=o(\sqrt{n}),{min}_{1\le j\le s}\mid {\beta }_j^{\ast }\mid \ge 2a{\lambda }_n$ with a the parameter in the SCAD penalty, and κ_n = o(n^1/4s^−1/2(log n)^−3/2(log p)^1/2). Further assume that ${\Vert n^{-1}{\mathbf{Q}}^T\mathbf{HS}\Vert }_{2.\infty }<C_4\sqrt{(logp)(loglog\mathit{n})/logn}$ with C₄ some positive constant. Then, under Conditions 1 and 2, with asymptotic probability one, there exists a global minimizer $\widehat{\mathit{\beta }}={({\widehat{\mathit{\beta }}}_1^T,{\widehat{\mathit{\beta }}}_2^T)}^T$ of L̂_n(β) such that

$${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le O(\sqrt{s(logn)/n}),sgn({\widehat{\mathit{\beta }}}_1)=sgn({\mathit{\beta }}_1^{\ast }),\mathit{and}{\widehat{\mathit{\beta }}}_2=\mathbf{0}.$$

If in addition, max_i ||H^1/2Z_ni||₂ = o(s^−7/2(log s)⁻¹), then we also have

$${\mathbf{c}}^T{({\mathbf{Z}}_n^T{\mathbf{Z}}_n)}^{-1/2}{\mathbf{V}}_n^{-1}({\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast })\stackrel{\mathcal{D}}{\to }N(0,\tau (1-\tau )),$$

where c is an arbitrary s-dimensional vector satisfying c^Tc = 1.

Corollary 1 provides sufficient conditions for ensuring the variable selection sign consistency of AR-Lasso. These conditions require that R-Lasso in the initial step has the sure screening property. We remark that in implementation, AR-Lasso is able to select the variables missed by R-Lasso, as demonstrated in our numerical studies in the next section. The theoretical comparison of the variable selection results of R-Lasso and AR-Lasso would be an interesting topic for future study. One set of (p, n, s, κ_n) satisfying conditions in Corollary 1 is log p = O(n^b₁), s = o(n^(1−b₁)/2) and κ_n = o(n^b₁/4(log n)^−3/2) with b₁ ∈ (0, 1/2) some constant. Corollary 1 gives one specific choice of λ_n, not necessarily the smallest λ_n, which makes our procedure work. In fact, the condition on λ_n can be weakened to ${\lambda }_n>2{(a+1)}^{-1}{\Vert {\widehat{\mathit{\beta }}}_1^{\mathit{ini}}-{\mathit{\beta }}_1\Vert }_{\infty }$. Currently, we use the L₂-norm ${\Vert {\widehat{\mathit{\beta }}}_1^{\mathit{ini}}-{\mathit{\beta }}_1\Vert }_2$ to bound this L_∞-norm, which is too crude. If one can establish ${\Vert {\widehat{\mathit{\beta }}}_1^{\mathit{ini}}-{\mathit{\beta }}_1\Vert }_{\infty }=O_p(\sqrt{n^{-1}logp})$ for an initial estimator ${\widehat{\mathit{\beta }}}_1^{\mathit{ini}}$, then the choice of λ_n can be as small as $O(\sqrt{n^{-1}logp})$, the same order as that used in Wang (2012). On the other hand, since we are using AR-Lasso, the choice of λ_n is not as sensitive as R-Lasso.

6. Numerical Studies

In this section we evaluate the finite sample property of our proposed estimator with synthetic data. Please see the supplementary material (Fan et al., 2013) for a real life data set analysis, where we provide results of an eQTL study on the CHRNA6 gene.

To assess the performance of the proposed estimator and compare it with other methods, we simulated data from the high-dimensional linear regression model

$$y_i={\mathbf{x}}_i^T{\mathit{\beta }}_0+{\epsilon }_i,\mathbf{x}~\mathcal{N}(0,{\mathbf{\sum }}_{\mathit{x}}),$$

where the data had n = 100 observations and the number of parameters was chosen as p = 400. We fixed the true regression coefficient vector as

$${\mathit{\beta }}_0=\{2,0,1.5,0,.80,0,0,1,0,1.75,0,0,.75,0,0,0.3,0,\dots ,0\}.$$

For the distribution of the noise, ε, we considered six symmetric distributions: Normal with variance 2 ( (0, 2)), a scale mixture of Normals for which ${\sigma }_i^2=1$ with probability 0.9 and ${\sigma }_i^2=25$ otherwise (MN₁), a different scale mixture model where ${\epsilon }_i~\mathcal{N}(0,{\sigma }_i^2)$ and σ_i ~ Unif(1, 5) (MN₂), Laplace, Student’s t with degrees of freedom 4 with doubled variance ( $\sqrt{2}\times t_4$) and Cauchy. We take τ = 0.5, corresponding to L₁-regression, throughout the simulation. Correlation of the covariates, Σ_x were either chosen to be identity (i.e. Σ_x = I_p) or they were generated from an AR(1) model with correlation 0.5, that is Σ_x(i,j) = 0.5^|i−j|.

We implemented five methods for each setting:

1. L₂-Oracle, which is the least squares estimator based on the signal covariates.

2. Lasso, the penalized least-squares estimator with L₁-penalty as in Tibshirani (1996).

3. SCAD, the penalized least-squares estimator with SCAD penalty as in Fan and Li (2001).

4. R-Lasso, the robust Lasso defined as the minimizer of (2.4) with d = 1.

5. AR-Lasso, which is the adaptive robust lasso whose adaptive weights on the penalty function were computed based on the SCAD penalty using the R-Lasso estimate as an initial value.

The tuning parameter, λ_n, was chosen optimally based on 100 validation data-sets. For each of these data-sets, we ran a grid search to find the best λ_n (with the lowest L₂ error for β) for the particular setting. This optimal λ_n was recorded for each of the 100 validation data-sets. The median of these 100 optimal λ_n were used in the simulation studies. We preferred this procedure over cross-validation because of the instability of the L₂ loss under heavy tails.

The following four performance measures were calculated:

1. L₂ loss, which is defined as ||β^* − β̂||₂.

2. L₁ loss, which is defined as ||β^* − β̂||₁.

3. Number of noise covariates that are included in the model, that is the number of false positives (FP).

4. Number of signal covariates that are not included, i.e. the number of false negatives (FN).

For each setting, we present the average of the performance measure based on 100 simulations. The results are depicted in Tables 1 and 2. A boxplot of the L₂ losses under different noise settings is also given in Figure 1 (the L₂ loss boxplot for the independent covariate setting is similar and omitted). For the results in Tables 1 and 2, one should compare the performance between Lasso and R-Lasso and that between SCAD and AR-Lasso. This comparison reflects the effectiveness of L₁-regression in dealing with heavy-tail distributions. Furthermore, comparing Lasso with SCAD, and R-Lasso with AR-Lasso, shows the effectiveness of using adaptive weights in the penalty function.

Table 1.

Simulation Results with Independent Covariates

		L2 Oracle	Lasso	SCAD	R-Lasso	AR-Lasso
(0, 2)	L2 loss	0.833	4.114	3.412	5.342	2.662
	L1 loss	0.380	1.047	0.819	1.169	0.785
	FP, FN	-	27.00, 0.49	29.60, 0.51	36.81, 0.62	17.27, 0.70
MN1	L2 loss	0.977	5.232	4.736	4.525	2.039
	L1 loss	0.446	1.304	1.113	1.028	0.598
	FP, FN	-	26.80, 0.73	29.29, 0.68	34.26, 0.51	16.76, 0.51
MN2	L2 loss	1.886	7.563	7.583	8.121	5.647
	L1 loss	0.861	2.085	2.007	2.083	1.845
	FP, FN	-	20.39, 2.28	23.25, 2.19	24.64, 2.29	11.97, 2.57
Laplace	L2 loss	0.795	4.056	3.395	4.610	2.025
	L1 loss	0.366	1.016	0.799	1.039	0.573
	FP, FN	-	26.87, 0.62	29.98, 0.49	34.76, 0.48	18.81, 0.40
$\sqrt{2}\times t_4$	L2 loss	1.087	5.303	5.859	6.185	3.266
	L1 loss	0.502	1.378	1.256	1.403	0.951
	FP, FN	-	24.61, 0.85	36.95, 0.76	33.84, 0.84	18.53, 0.82
Cauchy	L2 loss	37.451	211.699	266.088	6.647	3.587
	L1 loss	17.136	30.052	40.041	1.646	1.081
	FP, FN	-	27.39, 5.78	34.32, 5.94	27.33, 1.41	17.28, 1.10

Table 2.

Simulation Results with Correlated Covariates

		L2 Oracle	Lasso	SCAD	R-Lasso	AR-Lasso
(0, 2)	L2 loss	0.836	3.440	3.003	4.185	2.580
	L1 loss	0.375	0.943	0.803	1.079	0.806
	FP, FN	-	20.62, 0.59	23.13, 0.56	22.72, 0.77	14.49, 0.74
MN1	L2 loss	1.081	4.415	3.589	3.652	1.829
	L1 loss	0.495	1.211	1.055	0.901	0.593
	FP, FN	-	18.66, 0.77	15.71, 0.75	26.65, 0.60	13.29, 0.51
MN2	L2 loss	1.858	6.427	6.249	6.882	4.890
	L1 loss	0.844	1.899	1.876	1.916	1.785
	FP, FN	-	15.16, 2.08	14.77, 1.96	18.22, 1.91	7.86, 2.71
Laplace	L2 loss	0.803	3.341	2.909	3.606	1.785
	L1 loss	0.371	0.931	0.781	0.927	0.573
	FP, FN	-	19.32, 0.62	21.60, 0.38	24.44, 0.46	12.90, 0.55
$\sqrt{2}\times t_4$	L2 loss	1.122	4.474	4.259	4.980	2.855
	L1 loss	0.518	1.222	1.201	1.299	0.946
	FP, FN	-	20.00, 0.76	18.49, 0.91	23.56, 0.79	13.40, 1.05
Cauchy	L2 loss	31.095	217.395	243.141	5.388	3.286
	L1 loss	13.978	31.361	36.624	1.461	1.074
	FP, FN	-	25.59, 5.48	32.01, 5.43	20.80, 1.16	12.45, 1.17

Fig. 1.

Boxplots for L₂ Loss with Correlated Covariates

Our simulation results reveal the following facts. The quantile based estimators were more robust in dealing with the outliers. For example, for the first mixture model (MN₁) and Cauchy, R-Lasso outperformed Lasso, and AR-Lasso outperformed SCAD in all of the four metrics, and significantly so when the error distribution is the Cauchy distribution. On the other hand, for the light-tail distributions such as the normal distribution, the efficiency loss was limited. When the tails get heavier, for instance for the Laplace distribution, quantile based methods started to outperform the least-squares based approaches, more so when the tails got heavier.

The effectiveness of weights in AR-Lasso is self-evident. SCAD outperformed Lasso and AR-Lasso outperformed R-Lasso in almost all of the settings. Furthermore, for all of the error settings AR-Lasso had significantly lower L₂ and L₁ loss as well as a smaller model size compared to other estimators.

It is seen that when the noise does not have heavy tails, that is for the normal and the Laplace distribution, all the estimators are comparable in terms of L₁ loss. As expected, estimators that minimize squared loss worked better than R-Lasso and AR-Lasso estimators under Gaussian noise, but their performances deteriorated as the tails got heavier. In addition, in the two heteroscedastic settings, AR-Lasso had the best performance among others.

For Cauchy noise, least squares methods could only recover 1 or 2 of the true variables on average. On the other hand, L₁-estimators (R-Lasso and AR-Lasso) had very few false negatives, and as evident from L₂ loss values, these estimators only missed variables with smaller magnitudes.

In addition, AR-Lasso consistently selected a smaller set of variables than R-Lasso. For instance, for the setting with independent covariates, under the Laplace distribution, R-Lasso and AR-Lasso had on average 34.76 and 18.81 false positives, respectively. Also note that AR-Lasso consistently outperformed R-Lasso: It estimated β^* (lower L₁ and L₂ losses), and the support of β^* (lower averages for the number of false positives) more efficiently.

7. Proofs

In this section, we prove Theorems 1, 2 and 4 and provide the lemmas used in these proofs. The proofs of Theorems 3 and 5 and Proposition 1 are given in the supplementary Appendix (Fan et al., 2013).

We use techniques from empirical process theory to prove the theoretical results. Let $v_n(\mathit{\beta })={\sum }_{i=1}^n{\rho }_{\tau }(y_i-{\mathbf{x}}_i^T\mathit{\beta })$. Then $L_n(\mathit{\beta })=v_n(\mathit{\beta })+n{\lambda }_n{\sum }_{j=1}^p{d}_j\mid {\beta }_j\mid$. For a given deterministic M > 0, define the set

$${\mathcal{B}}_0(M)=\{\mathit{\beta }\in {\mathbf{R}}^p:{\Vert \mathit{\beta }-{\mathit{\beta }}^{\ast }\Vert }_2\le M,\text{supp}(\mathit{\beta })\subseteq \text{supp}({\mathit{\beta }}^{\ast })\}.$$

Then, define the function

$$Z_n(M)=\underset{\mathit{\beta }\in {\mathcal{B}}_0(M)}{sup}\frac{1}{n}\mid (v_n(\mathit{\beta })-v_n({\mathit{\beta }}^{\ast }))-E(v_n(\mathit{\beta })-v_n({\mathit{\beta }}^{\ast }))\mid .$$ (7.1)

Lemma 1 in Section 7.4 gives the rate of convergence for Z_n(M).

7.1. Proof of Theorem 1

We first show that for any $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathbf{0}}^T)}^T\in {\mathcal{B}}_0(M)$ with $M=o({\kappa }_n^{-1}{s}^{-1/2})$,

$$E[v_n(\mathit{\beta })-v_n({\mathit{\beta }}^{\ast })]\ge \frac{1}{2}{c}_{0}cn{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2^2,$$ (7.2)

for sufficiently large n, where c is the lower bound for f_i(·) in the neighborhood of 0. The intuition follows from the fact that β^* is the minimizer of the function Ev_n(β) and hence in Taylor’s expansion of E[v_n(β) − v_n(β^*)] around β^*, the first order derivative is zero at the point β = β^*. The left-hand side of (7.2) will be controlled by Z_n(M). This yields the L₂-rate of convergence in Theorem 1.

To prove (7.2), we set $a_i=\mid {\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })\mid$. Then, for β ∈ (M),

$$\mid a_i\mid \le {\Vert {\mathbf{S}}_i\Vert }_2{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le \sqrt{s}{\kappa }_{n}M\to 0.$$

Thus if ${\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })>0$, by E1{ε_i ≤ 0} = τ, Fubini’s theorem, mean value theorem, and Condition 1 it is easy to derive that

$$\begin{array}{l}E[{\rho }_{\tau }({\epsilon }_i-a_i)-{\rho }_{\tau }({\epsilon }_i)]=E[a_i(1\{{\epsilon }_i\le a_i\}-\tau )-{\epsilon }_{i}1\{0\le {\epsilon }_i\le a_i\}]\\ =E[{\int }_0^{a_i}1\{0\le {\epsilon }_i\le s\}ds]={\int }_0^{a_i}(F_i(s)-F_i(0))ds=\frac{1}{2}{f}_i(0)a_i^2+o(1)a_i^2,\end{array}$$ (7.3)

where the o(1) is uniformly over all i = 1, ···, n. When ${\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })<0$, the same result can be obtained. Furthermore, by Condition 2,

$$\sum _{i=1}^n{f}_i(0)a_i^2={({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })}^T{\mathbf{S}}^T\mathbf{HS}({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })\ge c_{0}n{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2^2.$$

This together with (7.3) and the definition of v_n(β) proves (7.2).

The inequality (7.2) holds for any $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathbf{0}}^T)}^T\in {\mathcal{B}}_0(M)$, yet ${\widehat{\mathit{\beta }}}^o={({({\widehat{\mathit{\beta }}}_1^o)}^T,{\mathbf{0}}^T)}^T$ may not be in the set. Thus, we let $\stackrel{\sim }{\mathit{\beta }}={({\stackrel{\sim }{\mathit{\beta }}}_1^T,{\mathbf{0}}^T)}^T$, where

$${\stackrel{\sim }{\mathit{\beta }}}_1=u{\widehat{\mathit{\beta }}}_1^o+(1-u){\mathit{\beta }}_1^{\ast },\text{with}u=M/(M+{\Vert {\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast }\Vert }_2),$$

which falls in the set (M). Then, by the convexity and the definition of ${\widehat{\mathit{\beta }}}_1^o$,

$$L_n(\stackrel{\sim }{\mathit{\beta }})\le u{L}_n({\widehat{\mathit{\beta }}}_1^o,\mathbf{0})+(1-u)L_n({\mathit{\beta }}_1^{\ast },\mathbf{0})\le L_n({\mathit{\beta }}_1^{\ast },\mathbf{0})=L_n({\mathit{\beta }}^{\ast }).$$

Using this and the triangle inequality we have

$$\begin{array}{l}E[v_n(\stackrel{\sim }{\mathit{\beta }})-v_n({\mathit{\beta }}^{\ast })]=\{v_n({\mathit{\beta }}^{\ast })-E{v}_n({\mathit{\beta }}^{\ast })\}-\{v_n(\stackrel{\sim }{\mathit{\beta }})-E{v}_n(\stackrel{\sim }{\mathit{\beta }})\}+L_n(\stackrel{\sim }{\mathit{\beta }})-L_n({\mathit{\beta }}^{\ast })+n{\lambda }_n{\Vert {\mathbf{d}}_0\circ {\mathit{\beta }}_1^{\ast }\Vert }_1-n{\lambda }_n{\Vert {\mathbf{d}}_0\circ {\mathit{\beta }}_1^{\ast }\Vert }_1\\ \le {nZ}_n(M)+n{\lambda }_n{\Vert {\mathbf{d}}_0\circ ({\mathit{\beta }}_1^{\ast }-{\stackrel{\sim }{\mathit{\beta }}}_1)\Vert }_1.\end{array}$$ (7.4)

By the Cauchy-Schwarz inequality, the very last term is bounded by $n{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2{\Vert {\stackrel{\sim }{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le n{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_{2}M$.

Define the event ${\mathcal{E}}_n=\{Z_n(M)\le 2M{n}^{-1/2}\sqrt{\mathit{s}logn}\}$. Then by Lemma 1,

$$P({\mathcal{E}}_n)\ge 1-exp(-c_{0}s(logn)/8).$$ (7.5)

On the event , by (7.4), we have

$$E[v_n(\stackrel{\sim }{\mathit{\beta }})-v_n({\mathit{\beta }}^{\ast })]\le 2M\sqrt{sn(logn)}+n{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_{2}M.$$

Taking $M=2\sqrt{s/n}+{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2$. By Condition 2 and the assumption ${\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2\sqrt{s}{\kappa }_n\to 0$, it is easy to check that $M=o({\kappa }_n^{-1}{s}^{-1/2})$. Combining these two results with (7.2), we obtain that on the event ,

$$\frac{1}{2}{c}_{0}n{\Vert {\stackrel{\sim }{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2^2\le (2\sqrt{sn(logn)}+n{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2)(2\sqrt{s/n}+{\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2),$$

which entails that

$${\Vert {\mathit{\beta }}_1^{\ast }-{\stackrel{\sim }{\mathit{\beta }}}_1\Vert }_2\le O({\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2+\sqrt{s(logn)/n}).$$

Note that ${\Vert {\mathit{\beta }}_1^{\ast }-\stackrel{\sim }{\mathit{\beta }}\Vert }_2\le M$ implies ${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le 2M$. Thus, on the event ,

$${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le O({\lambda }_n{\Vert {\mathbf{d}}_0\Vert }_2+\sqrt{s(logn)/n}).$$

The second result follows trivially.

7.2. Proof of Theorem 2

Since ${\widehat{\mathit{\beta }}}_1^o$ defined in Theorem 1 is a minimizer of L_n(β₁, 0), it satisfies the KKT conditions. To prove that $\widehat{\mathit{\beta }}={({({\widehat{\mathit{\beta }}}_1^o)}^T,{\mathbf{0}}^T)}^T\in {\mathbf{R}}^p$ is a global minimizer of L_n(β) in the original R^p space, we only need to check the following condition

$${\Vert {\mathbf{d}}_1^{-1}\circ {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\widehat{\mathit{\beta }}}_1^o)\Vert }_{\infty }<n{\lambda }_n,$$ (7.6)

where ${\rho }_{\tau }^{\prime }(\mathbf{u})={({\rho }_{\tau }^{\prime }(u_i),\cdots ,{\rho }_{\tau }^{\prime }(u_n))}^T$ for any n-vector u = (u₁, ···, u_n)^T with ${\rho }_{\tau }^{\prime }(u_i)=\tau -1\{u_i\le 0\}$. Here, ${\mathbf{d}}_1^{-1}$ denotes the vector ${(d_{s+1}^{-1},\cdots ,d_p^{-1})}^T$. Then the KKT conditions and the convexity of L_n(β) together ensure that β̂ is a global minimizer of L(β).

Define events

$$A_1=\{{\Vert {\widehat{\mathit{\beta }}}_1^o-{\mathit{\beta }}_1^{\ast }\Vert }_2\le {\gamma }_n\}{A}_2=\{\underset{\mathit{\beta }\in \mathcal{N}}{sup}{\Vert {\mathbf{d}}_1^{-1}\circ {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)\Vert }_{\infty }<n{\lambda }_n\},$$

where γ_n is defined in Theorem 1 and

$$\mathcal{N}=\{\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in R^p:{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le {\gamma }_n,{\mathit{\beta }}_2=\mathbf{0}\in {\mathbf{R}}^{p-s}\}.$$

Then by Theorem 1 and Lemma 2 in Section 7.4, P(A₁ ∩ A₂) ≥1 − o(n^−cs). Since β̂ ∈ on the event A₁, the inequality (7.6) holds on the event A₁ ∩ A₂. This completes the proof of Theorem 2.

7.3. Proof of Theorem 4

The idea of the proof follows those used in the proof of Theorems 1 and 2. We first consider the minimizer of L̂_n(β) in the subspace { $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in {\mathbf{R}}^p:{\mathit{\beta }}_2=\mathbf{0}$}. Let $\mathit{\beta }={({\mathit{\beta }}_1^T,\mathbf{0})}^T$, where ${\mathit{\beta }}_1={\mathit{\beta }}_1^{\ast }+{\stackrel{\sim }{a}}_n{\mathbf{v}}_1\in {\mathbf{R}}^s$ with ${\stackrel{\sim }{a}}_n=\sqrt{s(logn)/n}+{\lambda }_n({\Vert {\mathbf{d}}_0^{\ast }\Vert }_2+C_2{c}_5\sqrt{s(logp)/n})$, ||v₁||₂ = C, and C > 0 is some large enough constant. By the assumptions in the theorem we have ${\stackrel{\sim }{a}}_n=o({\kappa }_n^{-1}{s}^{-1/2})$. Note that

$${\widehat{L}}_n({\mathit{\beta }}_1^{\ast }+{\stackrel{\sim }{a}}_n{\mathbf{v}}_1,\mathbf{0})-{\widehat{L}}_n({\mathit{\beta }}_1^{\ast },\mathbf{0})=I_1({\mathbf{v}}_1)+I_2({\mathbf{v}}_1),$$ (7.7)

where $I_1({\mathbf{v}}_1)={\Vert {\rho }_{\tau }(\mathbf{y}-\mathbf{S}({\mathit{\beta }}_1^{\ast }+{\stackrel{\sim }{a}}_n{\mathbf{v}}_1))\Vert }_1-{\Vert {\rho }_{\tau }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1^{\ast })\Vert }_1$ and $I_2({\mathbf{v}}_1)=n{\lambda }_n({\Vert {\widehat{\mathbf{d}}}_0\circ ({\mathit{\beta }}_1^{\ast }+{\stackrel{\sim }{a}}_n{\mathbf{v}}_1)\Vert }_1-{\Vert {\widehat{\mathbf{d}}}_0\circ {\mathit{\beta }}_1^{\ast }\Vert }_1)$ with ${\Vert {\rho }_{\tau }(\mathbf{u})\Vert }_1={\sum }_{i=1}^n{\rho }_{\tau }(u_i)$ for any vector u = (u₁, ···, u_n)^T. By the results in the proof of Theorem 1, $E[I_1({\mathbf{v}}_1)]\ge 2^{-1}{c}_{0}n{\Vert {\stackrel{\sim }{a}}_n{\mathbf{v}}_1\Vert }_2^2$, and moreover, with probability at least 1 − n^−cs,

$$\mid I_1({\mathbf{v}}_1)-E[I_1({\mathbf{v}}_1)]\mid \le {nZ}_n(C{a}_n)\le 2{\stackrel{\sim }{a}}_n\sqrt{s(logn)n}{\Vert {\mathbf{v}}_1\Vert }_2.$$

Thus, by the triangle inequality,

$$I_1({\mathbf{v}}_1)\ge 2^{-1}{c}_0{\stackrel{\sim }{a}}_n^{2}n{\Vert {\mathbf{v}}_1\Vert }_2^2-2{\stackrel{\sim }{a}}_n\sqrt{s(logn)n}{\Vert {\mathbf{v}}_1\Vert }_2.$$ (7.8)

The second term on the right side of (7.7) can be bounded as

$$\mid I_2({\mathbf{v}}_1)\mid \le n{\lambda }_n{\Vert \widehat{\mathbf{d}}\circ ({\stackrel{\sim }{a}}_n{\mathbf{v}}_1)\Vert }_1\le {na}_n{\lambda }_n{\Vert {\widehat{\mathbf{d}}}_0\Vert }_2{\Vert {\mathbf{v}}_1\Vert }_2.$$ (7.9)

By triangle inequality and Conditions 4 and 5, it holds that

$${\Vert {\widehat{\mathbf{d}}}_0\Vert }_2\le {\Vert {\widehat{\mathbf{d}}}_0-{\mathbf{d}}_0^{\ast }\Vert }_2+{\Vert {\mathbf{d}}_0^{\ast }\Vert }_2\le c_5{\Vert {\widehat{\mathit{\beta }}}_1^{\mathit{ini}}-{\mathit{\beta }}_1^{\ast }\Vert }_2+{\Vert {\mathbf{d}}_0^{\ast }\Vert }_2\le C_2{c}_5\sqrt{s(logp)/n}+{\Vert {\mathbf{d}}_0^{\ast }\Vert }_2.$$ (7.10)

Thus, combining (7.7)–(7.10) yields

$${\widehat{L}}_n({\mathit{\beta }}^{\ast }+{\stackrel{\sim }{a}}_n{\mathbf{v}}_1)-{\widehat{L}}_n({\mathit{\beta }}^{\ast })\ge 2^{-1}{c}_{0}n{a}_n^2{\Vert {\mathbf{v}}_1\Vert }_2^2-2{\stackrel{\sim }{a}}_n\sqrt{s(logn)n}{\Vert {\mathbf{v}}_1\Vert }_2-{na}_n{\lambda }_n({\Vert {\mathbf{d}}_0^{\ast }\Vert }_2+C_2{c}_5\sqrt{s(logp)/n}){\Vert {\mathbf{v}}_1\Vert }_2.$$

Making ||v₁||₂ = C large enough, we obtain that with probability tending to one, L̂_n(β^*+ ã_nv) − L̂_n(β^*) > 0. Then, it follows immediately that with asymptotic probability one, there exists a minimizer β̂₁ of L̂_n(β₁, 0) such that ${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le C_3{\stackrel{\sim }{a}}_n\equiv a_n$ with some constant C₃ > 0.

It remains to prove that with asymptotic probability one,

$${\Vert {\widehat{\mathbf{d}}}_1^{-1}\circ {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\widehat{\mathit{\beta }}}_1)\Vert }_{\infty }<n{\lambda }_n.$$ (7.11)

Then by KKT conditions, $\widehat{\mathit{\beta }}={({\widehat{\mathit{\beta }}}_1^T,{\mathbf{0}}^T)}^T$ is a global minimizer of L̂_n(β).

Now we proceed to prove (7.11). Since ${\beta }_j^{\ast }=0$ for all j = s + 1, ···, p, we have that $d_j^{\ast }=p_{{\lambda }_n}^{\prime }(0+)$. Furthermore, by Condition 4, it holds that $\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid \le C_2\sqrt{s(logp)/n}$ with asymptotic probability one. Then, it follows that

$$\underset{j>s}{min}{p}_{{\lambda }_n}^{\prime }(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid )\ge p_{{\lambda }_n}^{\prime }(C_2\sqrt{s(logp)/n}).$$

Therefore, by Condition 5 we conclude that

$${\Vert {({\widehat{\mathbf{d}}}_1)}^{-1}\Vert }_{\infty }={(\underset{j>s}{min}{p}_{{\lambda }_n}^{\prime }(\mid {\widehat{\beta }}_j^{\mathit{ini}}\mid ))}^{-1}<2/p_{{\lambda }_n}^{\prime }(0+)=2{\Vert {({\mathbf{d}}_1^{\ast })}^{-1}\Vert }_{\infty }.$$ (7.12)

From the conditions of Theorem 2 with γ_n = a_n, it follows from Lemma 2 (inequality (7.20)) that, with probability at least 1 − o(p^−c),

$$\underset{{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le C_3{a}_n}{sup}{\Vert {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)\Vert }_{\infty }<\frac{n{\lambda }_n}{2{\Vert {({\mathbf{d}}_1^{\ast })}^{-1}\Vert }_{\infty }}(1+o(1)).$$ (7.13)

Combining (7.12)–(7.13) and by the triangle inequality, it holds that with asymptotic probability one,

$$\underset{{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le C_3{a}_n}{sup}{\Vert {({\widehat{\mathbf{d}}}_1)}^{-1}\circ {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)\Vert }_{\infty }<n{\lambda }_n.$$

Since the minimizer β̂₁ satisfies ${\Vert {\widehat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2<C_3{a}_n$ with asymptotic probability one, the above inequality ensures that (7.11) holds with probability tending to one. This completes the proof.

7.4. Lemmas

This subsection contains Lemmas used in proofs of Theorems 1,2 and 4.

Lemma 1

Under Condition 2, for any t > 0, we have

$$P(Z_n(M)\ge 4M\sqrt{s/n}+t)\le exp(-{nc}_0{t}^2/(8M^2)).$$ (7.14)

Proof

Define ρ(s, y) = (y−s)(τ − 1{y− s ≤ 0}). Then, v_n(β) in (7.1) can be rewritten as $v_n(\mathit{\beta })={\sum }_{i=1}^n\rho ({\mathbf{x}}_i^T\mathit{\beta },y_i)$. Note that the following Lipschitz condition holds for ρ(·; y_i)

$$\mid \rho (s_1,y_i)-\rho (s_2,y_i)\mid \le max\{\tau ,1-\tau \}\mid s_1-s_2\mid \le \mid s_1-s_2\mid .$$ (7.15)

Let W₁, ···, W_n be a Rademacher sequence, independent of model errors ε₁, ···, ε_n. The Lipschitz inequality (7.15) combined with the symmetrization theorem and Concentration inequality (see, for example, Theorems 14.3 and 14.4 in Büuhlmann and van de Geer (2011)) yields that

$$\begin{array}{l}E[Z_n(M)]\le 2E\underset{\mathit{\beta }\in {\mathcal{B}}_0(M)}{sup}\left|\frac{1}{n}\sum _{i=1}^n{W}_i(\rho ({\mathbf{x}}_i^T\mathit{\beta },y_i)-\rho ({\mathbf{x}}_i^T{\mathit{\beta }}^{\ast },y_i))\right|\\ \le 4E\underset{\mathit{\beta }\in {\mathcal{B}}_0(M)}{sup}\left|\frac{1}{n}\sum _{i=1}^n{W}_i({\mathbf{x}}_i^T\mathit{\beta }-{\mathbf{x}}_i^T{\mathit{\beta }}^{\ast })\right|.\end{array}$$ (7.16)

On the other hand, by the Cauchy-Schwarz inequality

$$\begin{array}{l}\left|\sum _{i=1}^n{W}_i({\mathbf{x}}_i^T\mathit{\beta }-{\mathbf{x}}_i^T{\mathit{\beta }}^{\ast })\right|=\left|\sum _{j=1}^s(\sum _{i=1}^n{W}_i{x}_{ij})({\beta }_j-{\beta }_j^{\ast })\right|\\ \le {\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2{\{\sum _{j=1}^s{\mid \sum _{i=1}^n{W}_i{x}_{ij}\mid }^2\}}^{1/2}.\end{array}$$

By Jensen’s inequality and concavity of the square root function, E(X^1/2) ≤ (EX)^1/2 for any non-negative random variable X. Thus, these two inequalities ensure that the very right hand side of (7.16) can be further bounded by

$$\underset{\mathit{\beta }\in {\mathcal{B}}_0(M)}{sup}{\Vert \mathit{\beta }-{\mathit{\beta }}^{\ast }\Vert }_{2}E{\left\{{\sum }_{j=1}^s{\mid \frac{1}{n}\sum _{i=1}^n{W}_i{x}_{ij}\mid }^2\right\}}^{1/2}\le M{\left\{{\sum }_{j=1}^{s}E{\mid \frac{1}{n}\sum _{i=1}^n{W}_i{x}_{ij}\mid }^2\right\}}^{1/2}=M\sqrt{s/n}.$$ (7.17)

Therefore, it follows from (7.16) and (7.17) that

$$E[Z_n(M)]\le 4M\sqrt{s/n}.$$ (7.18)

Next since n⁻¹S^TS has bounded eigenvalues, for any $\mathit{\beta }=({\mathit{\beta }}_1^T,{\mathbf{0}}^T)\in {\mathcal{B}}_0(M)$,

$$\frac{1}{n}\sum _{i=1}^n{({\mathbf{x}}_i^T(\mathit{\beta }-{\mathit{\beta }}^{\ast }))}^2=\frac{1}{n}{({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })}^T{\mathbf{S}}^T\mathbf{S}({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })\le c_0^{-1}{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2^2\le c_0^{-1}{M}^2.$$

Combining this with the Lipschitz inequality (7.15), (7.18), and applying Massart’s concentration theorem (see Theorem 14.2 in Bühlmann and van de Geer (2011)) yields that for any t > 0,

$$P(Z_n(M)\ge 4M\sqrt{s/n}+t)\le exp(-{nc}_0{t}^2/(8M^2)).$$

This proves the Lemma.

Lemma 2

Consider a ball in R^s around ${\mathit{\beta }}^{\ast }:\mathcal{N}=\{\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in R^p;{\mathit{\beta }}_2=0,{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2\le {\gamma }_n\}$ with some sequence γ_n → 0. Assume that min_j>s d_j > c₃, $\sqrt{1+{\gamma }_n{s}^{3/2}{\kappa }_n^2}{log}_{2}n=o(\sqrt{n}{\lambda }_n)$, n^1/2λ_n(log p)^−1/2 → ∞, and ${\kappa }_n{\gamma }_n^2=o({\lambda }_n)$. Then under Conditions 1–3, there exists some constant c > 0 such that

$$P\left(\underset{\mathit{\beta }\in \mathcal{N}}{sup}{\Vert {\mathbf{d}}_1^{-1}\circ {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)\Vert }_{\infty }\ge n{\lambda }_n\right)\le o(p^{-c}),$$

where ${\rho }_{\tau }^{\prime }(u)=\tau -1\{u\le 0\}$.

Proof

For a fixed j ∈ {s + 1, ···, p} and $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in \mathcal{N}$, define

$${\gamma }_{\mathit{\beta },j}({\mathbf{x}}_i,y_i)=x_{ij}[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)-E[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)]],$$

where ${\mathbf{x}}_i^T=(x_{i1},\cdots ,x_{ip})$ is the i-th row of the design matrix. The key for the proof is to use the following decomposition

$$\underset{\mathit{\beta }\in \mathcal{N}}{sup}{\Vert \frac{1}{n}{\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)\Vert }_{\infty }\le \underset{\mathit{\beta }\in \mathcal{N}}{sup}{\Vert \frac{1}{n}{\mathbf{Q}}^{T}E[{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)-{\rho }_{\tau }^{\prime }(\mathit{\epsilon })]\Vert }_{\infty }+{\Vert \frac{1}{n}{\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathit{\epsilon })\Vert }_{\infty }+\underset{j>s}{max}\underset{\mathit{\beta }\in \mathcal{N}}{sup}\frac{1}{n}\sum _{i=1}^n\mid {\gamma }_{\mathit{\beta },j}({\mathbf{x}}_i,y_i)\mid .$$ (7.19)

We will prove that with probability at least 1 − o(p^−c),

$$I_1\equiv \underset{\mathit{\beta }\in \mathcal{N}}{sup}{\Vert \frac{1}{n}{\mathbf{Q}}^{T}E[{\rho }_{\tau }^{\prime }(\mathbf{y}-\mathbf{S}{\mathit{\beta }}_1)-{\rho }_{\tau }^{\prime }(\mathit{\epsilon })]\Vert }_{\infty }<\frac{{\lambda }_n}{2{\Vert {\mathbf{d}}_1^{-1}\Vert }_{\infty }}+o({\lambda }_n),$$ (7.20)

$$I_2\equiv n^{-1}{\Vert {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathit{\epsilon })\Vert }_{\infty }=o({\lambda }_n),$$ (7.21)

$$I_3\equiv \underset{j>s}{max}\underset{\mathit{\beta }\in \mathcal{N}}{sup}\mid \frac{1}{n}\sum _{i=1}^n{\gamma }_{\mathit{\beta },j}({\mathbf{x}}_i,y_i)\mid =o_p({\lambda }_n).$$ (7.22)

Combining (7.19)–(7.22) with the assumption min_j>s d_j > c₃ completes the proof of the Lemma.

Now we proceed to prove (7.20). Note that I₁ can be rewritten as

$$I_1=\underset{j>s}{max}\underset{\mathit{\beta }\in \mathcal{N}}{sup}\mid \frac{1}{n}\sum _{i=1}^n{x}_{ij}E[{\rho }_{\tau }^{\prime }({\epsilon }_i)-{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })]\mid .$$ (7.23)

By Condition 1,

$$E[{\rho }_{\tau }^{\prime }({\epsilon }_i)-{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })]=F_i({\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }))-F_i(0)=f_i(0){\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })+{\stackrel{\sim }{I}}_i,$$

where F(t) is the cumulative distribution function of ε_i, and ${\stackrel{\sim }{I}}_i=F_i({\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }))-F_i(0)-f_i(0){\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })$. Thus, for any j > s,

$$\sum _{i=1}^n{x}_{ij}E[{\rho }_{\tau }^{\prime }({\epsilon }_i)-{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })]=\sum _{i=1}^n(f_i(0)x_{ij}{\mathbf{S}}_i^T)({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })+\sum _{i=1}^n{x}_{ij}{\stackrel{\sim }{I}}_i.$$

This together with (7.23) and Cauchy-Schwartz inequality entails that

$$I_1\le {\Vert \frac{1}{n}{\mathbf{Q}}^T\mathbf{HS}({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })\Vert }_{\infty }+\underset{j>s}{max}\mid \frac{1}{n}\sum _{i=1}^n{x}_{ij}{\stackrel{\sim }{I}}_i\mid ,$$ (7.24)

where H = diag{f₁(0, ···, f_n(0))}. We consider the two terms on the right hand side of (7.24) one by one. By Condition 3, the first term can be bounded as

$${\Vert \frac{1}{n}{\mathbf{Q}}^T\mathbf{HS}({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })\Vert }_{\infty }\le {\Vert \frac{1}{n}{\mathbf{Q}}^T\mathbf{HS}\Vert }_{2,\infty }{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2<\frac{{\lambda }_n}{2{\Vert {\mathbf{d}}_1^{-1}\Vert }_{\infty }}.$$ (7.25)

By Condition 1, $\mid {\stackrel{\sim }{\mathbf{I}}}_i\mid \le c{({\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }))}^2$. This together with Condition 2 ensures that the second term of (7.24) can be bounded as

$$\underset{j>s}{max}\left|\frac{1}{n}\sum _{i=1}^n{x}_{ij}{\stackrel{\sim }{I}}_i\right|\le \frac{{\kappa }_n}{n}\sum _{i=1}^n\mid {\stackrel{\sim }{I}}_1\mid \le C\frac{{\kappa }_n}{n}\sum _{i=1}^n{({\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }))}^2\le C{\kappa }_n{\Vert {\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\Vert }_2^2.$$

Since β ∈ , it follows from the assumption ${\lambda }_n^{-1}{\kappa }_n{\gamma }_n^2=o(1)$ that

$$\underset{j>s}{max}\left|\frac{1}{n}\sum _{i=1}^n{x}_{ij}{\stackrel{\sim }{I}}_i\right|\le C{\kappa }_n{\gamma }_n^2=o({\lambda }_n).$$

Plugging the above inequality and (7.25) into (7.24) completes the proof of (7.20).

Next we prove (7.21). By Hoeffding’s inequality, if ${\lambda }_n>2\sqrt{(1+c)(logp)/n}$ with c is some positive constant, then

$$\begin{array}{l}P({\Vert {\mathbf{Q}}^T{\rho }_{\tau }^{\prime }(\mathit{\epsilon })\Vert }_{\infty }\ge n{\lambda }_n)\le \sum _{j=s+1}^{p}2exp\left(-\frac{n^2{\lambda }_n^2}{4{\sum }_{i=1}^n{x}_{ij}^2}\right)\\ =2exp\left(log(p-s)-n{\lambda }_n^2/4\right)\le O(p^{-c}).\end{array}$$

Thus, with probability at least 1 − O(p^−c), (7.21) holds.

We now apply Corollary 14.4 in Bühlmann and van de Geer (2011) to prove (7.22). To this end, we need to check conditions of the Corollary. For each fixed j, define the functional space Γ_j = {γ_β,j : β ∈ }. First note that E[γ_β,j(x_i, y_i)] = 0 for any γ_β,j ∈ Γ_j. Second, since the ${\rho }_{\tau }^{\prime }$ function is bounded, we have

$$\frac{1}{n}\sum _{i=1}^n{\gamma }_{\mathit{\beta },j}^2({\mathbf{x}}_i,y_i)=\frac{1}{n}\sum _{i=1}^n{x}_{ij}^2{({\rho }_{\tau }^{\prime }(y_i={\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)-E({\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }({\epsilon }_i)))}^2\le 4.$$

Thus, ${\Vert {\gamma }_{\mathit{\beta },j}\Vert }_n\equiv {\left(n^{-1}{\sum }_{i=1}^n{\gamma }_{\mathit{\beta },j}^2{({\mathbf{x}}_i,y_i)}^2\right)}^{1/2}\le 2$.

Third, we will calculate the covering number of the functional space Γ_j, N(·, Γ_j, ||·||₂). For any $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T\in \mathcal{N}$ and $\stackrel{\sim }{\mathit{\beta }}={({\stackrel{\sim }{\mathit{\beta }}}_1^T,{\stackrel{\sim }{\mathit{\beta }}}_2^T)}^T\in \mathcal{N}$, by Condition 1 and the mean value theorem,

$$E[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)]-E[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\stackrel{\sim }{\mathit{\beta }})-{\rho }_{\tau }^{\prime }({\epsilon }_i)]=F_i({\mathbf{S}}_i^T(\stackrel{\sim }{\mathit{\beta }}-{\mathit{\beta }}_1^{\ast }))-F_i({\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }))=f_i(a_{1i}){\mathbf{S}}_i^T({\mathit{\beta }}_1-{\stackrel{\sim }{\mathit{\beta }}}_1),$$ (7.26)

where F(t) is the cumulative distribution function of ε_i, and a_1i lies on the segment connecting ${\mathbf{S}}_i^T({\stackrel{\sim }{\mathit{\beta }}}_1-{\mathit{\beta }}_1^{\ast })$ and ${\mathbf{S}}_i^T({\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast })$. Let κ_n = max_ij |x_ij|. Since f_i(u)’s are uniformly bounded, by (7.26),

$$\mid x_{ij}E[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)]-x_{ij}E[{\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\stackrel{\sim }{\mathit{\beta }})-{\rho }_{\tau }^{\prime }({\epsilon }_i)]\mid \le C\mid x_{ij}{\mathbf{S}}_i^T({\mathit{\beta }}_1-{\stackrel{\sim }{\mathit{\beta }}}_1)\mid \le C{\Vert x_{ij}{\mathbf{S}}_i\Vert }_2{\Vert {\mathit{\beta }}_1-{\stackrel{\sim }{\mathit{\beta }}}_1\Vert }_2\le C\sqrt{s}{\kappa }_n^2{\Vert {\mathit{\beta }}_1-{\stackrel{\sim }{\mathit{\beta }}}_1\Vert }_2,$$ (7.27)

where C > 0 is some generic constant. It is known (see, for example, Lemma 14.27 in Bühlmann and van de Geer (2011)) that the ball in R^s can be covered by (1 + 4γ_n/δ)^s balls with radius δ. Since ${\rho }_{\tau }^{\prime }(y_i-{\mathbf{x}}_i^T\mathit{\beta })-{\rho }_{\tau }^{\prime }({\epsilon }_i)$ can only take 3 different values {−1, 0, 1}, it follows from (7.27) that the covering number of Γ_j is $N(2^{2-k},{\mathrm{\Gamma }}_j,{\Vert ·\Vert }_2)=3{(1+C^{-1}{2}^k{\gamma }_n{s}^{1/2}{\kappa }_n^2)}^s$. Thus, by calculus, for any 0 ≤ k ≤ (log₂ n)/2,

$$\begin{array}{l}log(1+N(2^{2-k},\mathrm{\Gamma },{\Vert ·\Vert }_2))\le log(6)+slog(1+C^{-1}{2}^k{\gamma }_n{s}^{1/2}{\kappa }_n^2)\\ \le log(6)+C^{-1}{2}^k{\gamma }_n{s}^{3/2}{\kappa }_n^2\le 4(1+C^{-1}{\gamma }_n{s}^{3/2}{\kappa }_n^2)2^{2k}.\end{array}$$

Hence, conditions of Corollary 14.4 in Bühlmann and van de Geer (2011) are checked and we obtain that for any t > 0,

$$P\left(\underset{\mathit{\beta }\in \mathcal{N}}{sup}\left|\frac{1}{n}\sum _{i=1}^n{\gamma }_{\mathit{\beta },j}({\mathbf{x}}_i,y_i)\right|\ge \frac{8}{\sqrt{n}}(3\sqrt{1+C^{-1}{\gamma }_n{s}^{3/2}{\kappa }_n^2}{log}_{2}n+4+4t)\right)\le 4exp\left(-\frac{n{t}^2}{8}\right).$$

Taking $t=\sqrt{C(logp)/n}$ with C > 0 large enough constant we obtain that

$$\begin{array}{l}P\left(\underset{j>s}{max}\underset{\mathit{\beta }\in \mathcal{N}}{sup}\mid \frac{1}{n}\sum _{i=1}^n{\gamma }_{\mathit{\beta },j}({\mathbf{x}}_i,y_i)\mid \ge \frac{24}{\sqrt{n}}\sqrt{1+C^{-1}{\gamma }_n{s}^{3/2}{\kappa }_n^2}{log}_{2}n\right)\\ \le 4(p-s)exp\left(-\frac{\mathit{C}logp}{8}\right)\to 0.\end{array}$$

Thus if $\sqrt{1+{\gamma }_n{s}^{3/2}{\kappa }_n^2}{log}_{2}n=o(\sqrt{n}{\lambda }_n)$, then with probability at least 1 − o(p^−c), (7.22) holds. This completes the proof of the Lemma.