Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions

Abstract

Data subject to heavy-tailed errors are commonly encountered in various scientific fields. To address this problem, procedures based on quantile regression and Least Absolute Deviation (LAD) regression have been developed in recent years. These methods essentially estimate the conditional median (or quantile) function. They can be very different from the conditional mean functions, especially when distributions are asymmetric and heteroscedastic. How can we efficiently estimate the mean regression functions in ultra-high dimensional setting with existence of only the second moment? To solve this problem, we propose a penalized Huber loss with diverging parameter to reduce biases created by the traditional Huber loss. Such a penalized robust approximate quadratic (RA-quadratic) loss will be called RA-Lasso. In the ultra-high dimensional setting, where the dimensionality can grow exponentially with the sample size, our results reveal that the RA-lasso estimator produces a consistent estimator at the same rate as the optimal rate under the light-tail situation. We further study the computational convergence of RA-Lasso and show that the composite gradient descent algorithm indeed produces a solution that admits the same optimal rate after sufficient iterations. As a byproduct, we also establish the concentration inequality for estimating population mean when there exists only the second moment. We compare RA-Lasso with other regularized robust estimators based on quantile regression and LAD regression. Extensive simulation studies demonstrate the satisfactory finite-sample performance of RA-Lasso.

1 Introduction

Our era has witnessed the massive explosion of data and a dramatic improvement of technology in collecting and processing large data sets. We often encounter huge data sets that the number of features greatly surpasses the number of observations. It makes many traditional statistical analysis tools infeasible and poses great challenge on developing new tools. Regularization methods have been widely used for the analysis of high-dimensional data. These methods penalize the least squares or the likelihood function with the L₁-penalty on the unknown parameters (Lasso, Tibshirani (1996)), or a folded concave penalty function such as the SCAD (Fan and Li, 2001) and the MCP(Zhang, 2010). However, these penalized least-squares methods are sensitive to the tails of the error distributions, particularly for ultrahigh dimensional covariates, as the maximum spurious correlation between the covariates and the realized noises can be large in those cases. As a result, theoretical properties are often obtained under light-tailed error distributions (Bickel, Ritov and Tsybakov, 2009; Fan and Lv, 2011). Besides regularization methods, traditional stagewise selection methods (e.g forward selection) have also been extended to the high-dimensional setting. For instance, Fan and Lv (2008) proposed a Sure Independence Screening method and Wang (2009) studied the stagewise selection methods in high-dimension setting. These methods are usually built on marginal correlations between the response and covariates, hence they also need light-tail assumptions on the errors.

To tackle the problem of heavy-tailed errors, robust regularization methods have been extensively studied. Li and Zhu (2008), Wu and Liu (2009) and Zou and Yuan (2008) developed robust regularized estimators based on quantile regression for the case of fixed dimensionality. Belloni and Chernozhukov (2011) studied the L₁-penalized quantile regression in high dimensional sparse models. Fan, Fan, and Barut (2014) further considered an adaptively weighted L₁ penalty to alleviate the bias problem and established the oracle property and asymptotic normality of the corresponding estimator. Other robust estimators were developed based on Least Absolute Deviation (LAD) regression. Wang (2013) studied the L₁-penalized LAD regression and showed that the estimator achieves near oracle risk performance under the high dimensional setting.

The above methods essentially estimate the conditional median (or quantile) regression, instead of the conditional mean regression function. In the applications where the mean regression is of interest, these methods are not feasible unless a strong assumption is made that the distribution of errors is symmetric around zero. A simple example is the heteroscedastic linear model with asymmetric noise distribution. Another example is to estimate the conditional variance function such as ARCH model (Engle, 1982). In these cases, the conditional mean and conditional median are very different. Another important example is to estimate large covariance matrix without assuming light-tails. We will explain this more in details in Section 5. In addition, LAD-based methods tend to penalize strongly on small errors. If only a small proportion of samples are outliers, they are expected to be less efficient than the least squares based method.

A natural question is then how to conduct ultrahigh dimensional mean regression when the tails of errors are not light? How to estimate the sample mean with very fast concentration when the distribution has only bounded second moment? These simple questions have not been carefully studied. LAD-based methods do not intend to answer these questions as they alter the problems of the study. This leads us to consider Huber loss as another way of robustification. The Huber loss (Huber, 1964) is a hybrid of squared loss for relatively small errors and absolute loss for relatively large errors, where the degree of hybridization is controlled by one tuning parameter. Lambert-Lacroix and Zwald (2011) proposed to use the Huber loss together with the adaptive LASSO penalty for the robust estimation. However, they needed the strong assumption that the distribution of errors is symmetric around zero. Unlike their method, we waive the symmetry requirement by allowing the regularization parameter to diverge (or converge if its reciprocal is used) in order to reduce the bias induced by the Huber loss when the distribution is asymmetric. In this paper, we consider the regularized approximate quadratic (RA-Lasso) estimator with an L₁ penalty and show that it admits the same L₂ error rate as the optimal error rate in the light-tail situation. In particular, if the distribution of errors is indeed symmetric around 0 (where the median and mean agree), this rate is the same as the regularized LAD estimator obtained in Wang (2013). Therefore, the RA-Lasso estimator does not lose efficiency in this special case. In practice, since the distribution of errors is unknown, RA-Lasso is more flexible than the existing methods in terms of estimating the conditional mean regression function.

A by-product of our method is that the RA-Lasso estimator of the population mean has the exponential type of concentration even in presence of the finite second moment. Catoni (2012) studied this type of problem and proposed a class of losses to result in a robust M-estimator of mean with exponential type of concentration. We further extend his idea to the sparse linear regression setting and show that Catoni loss is another choice in order to reach the optimal rate.

As done in many other papers, estimators with nice sampling properties are typically defined through the optimization of a target function such as the penalized least-squares. The properties that are established are not necessarily the same as the ones that are computed. Following the framework of Agarwal, Negahban, and Wainwright (2012), we propose the composite gradient descent algorithm for solving the RA-Lasso estimator and develop the sampling properties by taking computational error into consideration. We show that the algorithm indeed produces a solution that admits the same optimal L₂ error rate as the theoretical estimator after sufficient number of iterations.

This paper is organized as follows. First, in Section 2, we introduce the RA-Lasso estimator and give the non-asymptotic upper bound for its L₂ error. We show that it has the same rate as the optimal rate under light-tails. In Section 3, we study the property of the composite gradient descent algorithm for solving our problem and show that the algorithm produces a solution that performs as well as the theoretical solution. In Section 4, we apply the idea to robust estimation of mean and large covariance matrix. In Section 5, we show similar results for Catoni loss in robust sparse regression. Section 6 gives estimation of residual variance. Numerical studies are given in Section 7 and 8 to compare our method with two competitors. Proofs of Theorems 1 and 2 are given in the appendix, which together imply the main result (Theorem 3). Proof of Theorem 5 regarding the concentration of the robust mean estimator is also given in the appendix. Proofs of supporting lemmas and remaining theorems are given in an on-line supplementary file. The relevant matlab code is available at the site: http://orfe.princeton.edu/~jqfan/papers/15/RA-Lasso.zip.

2 RA-Lasso estimator

We consider the linear regression model

$$y_i={\mathbf{x}}_i^T{\beta }^{\ast }+{ϵ}_i,$$ (2.1)

where ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$ are independent and identically distributed (i.i.d) p-dimensional covariate vectors, ${\left\{{ϵ}_i\right\}}_{i=1}^n$ are i.i.d errors, and β* is a p-dimensional regression coefficient vector. The i.i.d assumption on ε_i indeed allows conditional heteroscedastic models, where ε_i can depend on x_i. For example, it can be ${ϵ}_i=\sigma \left({\mathbf{x}}_i\right){\widetilde{ϵ}}_i$, where σ(x_i) is a function of x_i and ${\widetilde{ϵ}}_i$ is independent of x_i. We consider the high-dimensional setting, where log(p) = O(n^b) for some constant 0 < b < 1. The distributions of x and ε|x are assumed to both have mean 0. Under this assumption, β* is related to the mean effect of y conditioning on x, which is assumed to be of interest. β* differs from the median effect of y conditioning on x, especially under the heteroscedastic models or more general models. Therefore, the LAD-based methods are not applicable.

To adapt for different magnitude of errors and robustify the estimation, we propose to use the Huber loss (Huber, 1964):

$${\ell }_{\alpha }\left(x\right)=\{\begin{array}{cc}2{\alpha }^{-1}\mid x\mid -{\alpha }^{-2}\hfill & \text{if}\mid x\mid >{\alpha }^{-1};\hfill \\ x^2\hfill & \text{if}\mid x\mid \le {\alpha }^{-1}.\hfill \end{array}\phantom{\}}$$ (2.2)

The Huber loss is quadratic for small values of x and linear for large values of x. The parameter α controls the blending of quadratic and linear penalization. The least squares and the LAD can be regarded as two extremes of the Huber loss for α = 0 and α = ∞, respectively. Deviated from the traditional Huber's estimator, the parameter α converges to zero in order to reduce the biases of estimating the mean regression function when the conditional distribution of ε_i is not symmetric. On the other hand, α cannot shrink too fast in order to maintain the robustness. In this paper, we regard α as a tuning parameter, whose optimal value will be discussed later in this section. In practice, α needs to be tuned by some data-driven method. By letting α vary, we call ${\ell }_{\alpha }\left(x\right)$ the robust approximate quadratic (RA-quadratic) loss.

To estimate β*, we propose to solve the following convex optimization problem:

$$\widehat{\beta }=\underset{\beta }{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{\ell }_{\alpha }(y_i-{\mathbf{x}}_i^T\beta )+{\lambda }_n\sum _{j=1}^p\mid {\beta }_j\mid .$$ (2.3)

To assess the performance of $\widehat{\beta }$, we study the property of ${\Vert \widehat{\beta }={\beta }^{\ast }\Vert }_2$, where ∥·∥₂ is the Euclidean norm of a vector. When λ_n converges to zero sufficiently fast, $\widehat{\beta }$ is a natural M-estimator of ${\beta }_{\alpha }^{\ast }={\mathrm{argmin}}_{\beta }\mathrm{E}{\ell }_{\alpha }(y-{\mathbf{x}}^{\prime }\beta )$, which is the population minimizer under the RA-quadratic loss and varies by α. In general, ${\beta }_{\alpha }^{\ast }$ differs from β*. But, since the RA-quadratic loss approximates the quadratic loss as α tends to 0, ${\beta }_{\alpha }^{\ast }$ is expected to converge to β*. This property will be established in Theorem 1. Therefore, we decompose the statistical error $\widehat{\beta }-{\beta }^{\ast }$ into the approximation error ${\beta }_{\alpha }^{\ast }-{\beta }^{\ast }$ and the estimation error $\widehat{\beta }-{\beta }_{\alpha }^{\ast }$. The statistical error ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$ is then bounded by

$${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2\le \underset{\text{approximation error}}{\underbrace{{\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2}}+\underset{\text{estimation error}}{\underbrace{{\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2}}.$$

In the following, we give upper bounds of the approximation and estimation error, respectively. We show that ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$ is upper bounded by the same rate as the optimal rate under light tails, as long as the two tuning parameters α and λ_n are properly chosen. We first give the upper bound of the approximation error under some moment conditions on x and ε|x. We assume that ${\Vert {\beta }^{\ast }\Vert }_2\le {\rho }_2$, where the radius ρ₂ is a sufficiently large constant. This is a mild assumption, which is implied by (C2) and a reasonable assumption that var(y) > ∞, since $\mathrm{var}\left(y\right)\ge {\left({\beta }^{\ast }\right)}^T\mathrm{E}\left(\mathbf{x}{\mathbf{x}}^T\right){\beta }^{\ast }\ge {\kappa }_l{\Vert {\beta }^{\ast }\Vert }_2^2$.

Theorem 1

Under the following conditions:

• (C1)E{E(|ε|^k|x)}² ≤ M_k < ∞, for some k ≥ 2.
• (C2)0 < κ_l ≤ λ_min(E[xx^T]) ≤ λ_max(E[xx^T]) ≤ κ_u < ∞,
• (C3)For any $\nu \in {\mathbb{R}}^p$, x^Tν is sub-Gaussian with parameter at most ${\kappa }_0^2{\Vert \nu \Vert }_2^2$, i.e. $\mathrm{E}\exp \left(t{\mathbf{x}}^T\nu \right)\le \exp (t^2{\kappa }_0^2{\Vert \nu \Vert }_2^2∕2)$, for any $t\in \mathbb{R}$,

there exosts a universal positive constant C₁, such that ${\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2\le C_1\sqrt{{\kappa }_u}{\kappa }_l^{-1}({\kappa }_0^k+\sqrt{M_k}){\alpha }^{k-1}$.

Theorem 1 reveals that the approximation error vanishes faster if higher moments of ε|x exist. We next give the non-asymptotic upper bound of the estimation error ${\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2$. This part differs from the existing work regarding the estimation error of high dimensional regularized M-estimator (Negahban, et al., 2012; Agarwal, Negahban, and Wainwright, 2012) as the population minimizer ${\beta }_{\alpha }^{\ast }$ now varies with α. However, we will show that the upper bound of the estimation error does not depend on α, given a uniform sparsity condition.

In order to be solvable in the high-dimensional setting, β* is usually assumed to be sparse or weakly sparse, i.e. many elements of β* are zero or small. By Theorem 1, ${\beta }_{\alpha }^{\ast }$ converges to β* as α goes to 0. In view of this fact, we assume that ${\beta }_{\alpha }^{\ast }$ is uniformly weakly sparse when α is sufficiently small. In particular, we assume that there exists a small constant r > 0, such that ${\beta }_{\alpha }^{\ast }$ belongs to an L_q-ball with a uniform radius R_q that

$$\sum _{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }\mid }^q\le R_q,$$ (2.4)

for all α ε (0, r] and some q ε (0,1]. When the conditional distribution of ε_i is symmetric, ${\beta }_{\alpha ,j}^{\ast }={\beta }_j^{\ast }$ for all α and j. Therefore the condition reduces to that β* is in the L_q-ball. When the conditional distribution of ε_i is asymmetric, we give a sufficient condition showing that if β* belongs to an L_q-ball with radius R_q/2, (2.4) holds for all $\alpha \le c{\left\{R_q{p}^{-(2-q)∕2}\right\}}^{\frac{1}{q(k-1)}}$, where c is a positive constant. In fact, for any $q\in (0,1],{\mid r_1\mid }^q+{\mid r_2\mid }^q\ge {(\mid r_1\mid +\mid r_2\mid )}^q\ge {\mid r_1+r_2\mid }^q$. Using this,

$$\sum _{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }\mid }^q\le \sum _{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }-{\beta }_j^{\ast }\mid }^q+\sum _{j=1}^p{\mid {\beta }_j^{\ast }\mid }^q\le p^{(2-q)∕2}{\left(\sum _{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }-{\beta }_j^{\ast }\mid }^2\right)}^{q∕2}+\sum _{j=1}^p{{\beta }_j^{\ast }\mid }^q.$$

By Theorem 1, ${\sum }_{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }-{\beta }_j^{\ast }\mid }^2=O\left({\alpha }^{2(k-1)}\right)$. Hence, if ${\sum }_{j=1}^p{\mid {\beta }_j^{\ast }\mid }^q\le R_q∕2$, we have ${\sum }_{j=1}^p{\mid {\beta }_{\alpha ,j}^{\ast }\mid }^q\le R_q$ for all $\alpha \le c{\left\{R_q{p}^{-(2-q)∕2}\right\}}^{\frac{1}{q(k-1)}}$.

Since the RA-quadratic loss in convex, we show that with high probability the estimation error $\widehat{\Delta }=\widehat{\beta }-{\beta }_{\alpha }^{\ast }$ belongs to a star-shaped set, which depends on α and the threshold level η of signals.

Lemma 1

Under Conditions (C1) and (C3), with the choice of ${\lambda }_n={\kappa }_{\lambda }\sqrt{\left(\log p\right)∕n}$ and $\alpha \ge L{\lambda }_n∕\left(4v\right)$, where v and L are positive constants depending on M₂ and κ₀, and κ_λ is a sufficiently large constant such that ${\kappa }_{\lambda }^2>32v$, it holds with probability greater than 1 – 2 exp(–c₀n) that,

$$\widehat{\Delta }=\widehat{\beta }-{\beta }_{\alpha }^{\ast }\in {\mathbb{C}}_{\alpha \eta }≔\{\Delta \in {\mathbb{R}}^p:{\Vert {\Delta }_{S_{\alpha \eta }^c}\Vert }_1\le 3{\Vert {\Delta }_{S_{\alpha \eta }}\Vert }_1+4{\Vert {\beta }_{\alpha ,S_{\alpha \eta }^c}^{\ast }\Vert }_1\},$$

where $c_0={\kappa }_{\lambda }^2∕\left(32v\right)-1$, η is a positive constant, $S_{\alpha \eta }=\{j:\mid {\beta }_{\alpha ,j}^{\ast }\mid >\eta \}$ and ΔS_αη denotes the subvector of Δ with indices in set S_αη.

We further verify a restricted strong convexity (RSC) condition, which has been shown to be critical in the study of high dimensional regularized M-estimator (Negahban, et al., 2012; Agarwal, Negahban, and Wainwright, 2012). Let

$$\delta {\mathcal{L}}_n(\Delta ,\beta )={\mathcal{L}}_n(\beta +\Delta )-{\mathcal{L}}_n\left(\beta \right)-{[\nabla {\mathcal{L}}_n\left(\beta \right)]}^T\Delta ,$$ (2.5)

where ${\mathcal{L}}_n\left(\beta \right)=\frac{1}{n}{\sum }_{i=1}^n{\ell }_{\alpha }(y_i-{\mathbf{x}}_i^T\beta )$, Δ is a p-dimensional vector and $\nabla {\mathcal{L}}_n\left(\beta \right)$ is the gradient of ${\mathcal{L}}_n$ at the point of β.

Definition 1

The loss function ${\mathcal{L}}_n$ satisfies RSC condition on a set S with curvature ${\kappa }_{\mathcal{L}}>0$ and tolerance ${\tau }_{\mathcal{L}}$ if

$$\delta {\mathcal{L}}_n(\Delta ,\beta )\ge {\kappa }_{\mathcal{L}}{\Vert \Delta \Vert }_2^2-{\tau }_{\mathcal{L}}^2,forall\Delta \in S.$$

Next, we show that with high probability the RA-quadratic loss (2.2) satisfies RSC for $\beta ={\beta }_{\alpha }^{\ast }$ and all $\Delta \in {\mathbb{C}}_{\alpha \eta }\cap \{\Delta :{\Vert \Delta \Vert }_2\le 1\}$ with uniform constants ${\kappa }_{\mathcal{L}}$ and ${\tau }_{\mathcal{L}}$ that do not depend on α. To prove the RSC at ${\beta }_{\alpha }^{\ast }$ and a stronger version in Lemma 4, we first give a uniform lower bound of $\delta {\mathcal{L}}_n(\Delta ,\beta )$ for all ${\Vert \beta \Vert }_2\le 4{\rho }_2$, ${\Vert \Delta \Vert }_2\le 8{\rho }_2$ and $\alpha \le c_u{\rho }_2^{-1}$, where c_u is a positive constant, depending on M_k, κ_l, κ_u and κ0.

Lemma 2

Under conditions (C1)-(C3), for all ${\Vert \beta \Vert }_2\le 4{\rho }_2$, ${\Vert \Delta \Vert }_2\le 8{\rho }_2$ and $\alpha \le c_u{\rho }_2^{-1}$, there exist uniform positive constants $c_1^{\prime }$ and $c_2^{\prime }$ such that, with probability at least $1-c_1^{\prime }\exp (-c_2^{\prime }n)$,

$$\delta {\mathcal{L}}_n(\Delta ,\beta )\ge {\kappa }_1{\Vert \Delta \Vert }_2({\Vert \Delta \Vert }_2-{\kappa }_2\sqrt{\left(\log p\right)∕n}{\Vert \Delta \Vert }_1).$$ (2.6)

Lemma 3

Suppose conditions (C1)-(C3) hold and assume that

$$8{\kappa }_2{\kappa }_{\lambda }^{-q∕2}\sqrt{R_q}{\left(\frac{\log p}{n}\right)}^{(1-q)∕2}\le 1,$$ (2.7)

by choosing η = λ_n, with probability at least $1-c_1^{\prime }\exp (-c_2^{\prime }n)$, the RSC condition holds for $\delta {\mathcal{L}}_n(\Delta ,{\beta }_{\alpha }^{\ast })$ for any $\Delta \in {\mathbb{C}}_{\alpha \eta }\cap \{\Delta :{\Vert \Delta \Vert }_2\le 1\}$ with ${\kappa }_{\mathcal{L}}={\kappa }_1∕2$ and ${\tau }_{\mathcal{L}}^2=4R_q{\kappa }_1{\kappa }_2{\kappa }_{\lambda }^{1-q}{\left(n^{-1}\log p\right)}^{1-(q∕2)}$.

Lemma 3 shows that, even though ${\beta }_{\alpha }^{\ast }$ is unknown and the set ${\mathbb{C}}_{\alpha \eta }$ depends on α, RSC holds with uniform constants that do not depend on α. This further gives the following upper bound of the estimation error ${\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2$, which also does not depend on α.

Theorem 2

Under conditions of Lemma 1 and 3, there exist positive constants c₁, c₂, and C₂ such that with probability at least 1 – c₁ exp(–c₂n),

$${\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2\le C_2{k}_l^{-2}{\kappa }_{\lambda }^{2-q}{R}_q{\left(n^{-1}\log p\right)}^{1-(q∕2)}.$$

Finally, Theorems 1 and 2 together lead to the following main result, which gives the non-asymptotic upper bound of the statistical error ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$.

Theorem 3

Under conditions of Lemmas 1 and 3, with probability at least 1 – c₁ exp(–c₂n),

$${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2\le d_1{\alpha }^{k-1}+d_2\sqrt{R_q}{[\left(\log p\right)∕n]}^{1∕2-q∕4},$$ (2.8)

where the constants $d_1=C_1\sqrt{{\kappa }_u}{\kappa }_l^{-1}({\kappa }_0^k+\sqrt{M_k})$ and $d_2=C_2{\kappa }_l^{-2}{\kappa }_{\lambda }^{2-q}$.

Next, we compare our result with the existing results regarding the robust estimation of high dimensional linear regression model.

1. When the conditional distribution of ε is symmetric around 0, then ${\beta }_{\alpha }^{\ast }={\beta }^{\ast }$ for any α, which has no approximation error. If ε has heavy tails in addition to being symmetric, we would like to choose α sufficiently large to robustify the estimation. Theorem 2 implies that ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$ has a convergence rate of $\sqrt{R_q}{[\left(\log p\right)∕n]}^{1∕2-q∕4}$, where $R_q={\sum }_{j=1}^p{\mid {\beta }_j^{\ast }\mid }^q$. The rate is the same as the minimax rate (Raskutti, Wainwright, and Yu, 2011) for weakly sparse model under the light tails. In a special case that q = 0, ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$ converges at a rate of $\sqrt{s\left(\log p\right)∕n}$, where s is the number of nonzero elements in β*. This is the same rate as the regularized LAD estimator in Wang (2013) and the regularized quantile estimator in Belloni and Chernozhukov (2011). It suggests that our method does not lose efficiency for symmetric heavy-tailed errors.

2. If the conditional distribution of ε is asymmetric around 0, the quantile and LAD based methods are inconsistent, since they estimate the median instead of the mean. Theorem 3 shows that our estimator still achieves the optimal rate as long as $\alpha \le {\left\{d_1^{-1}{d}_2{R}_q{[\left(\log p\right)∕n]}^{1-\frac{q}{2}}\right\}}^{\frac{1}{2(k-1)}}$. Recall from conditions in Lemmas 1 and 3 that we also need to choose α, such that $c_l\sqrt{\left(\log p\right)∕n}\le \alpha \le c_u{\rho }_2^{-1}$ for some constants c_l and c_u. Given the sparsity condition (2.7), α can be chosen to meet the above three requirements. In terms of estimating the conditional mean effect, errors with heavy but asymmetric tails give the case where the RA-Lasso has the biggest advantage over the existing estimators.

In practice, the distribution of errors is unknown. Our method is more flexible than the existing methods as it does not require symmetry and light-tail assumptions. The tuning parameter α plays a key role by adapting to errors with different shapes and tails. In reality, the optimal values of tuning parameters α and λ_n can be chosen by a two-dimensional grid search using cross-validation or information-based criterion, for example, Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC). More specifically, the searching grid is formed by partitioning a rectangle in the scale of (log(α), log(λ_n)). The optimal values are then found by the combination that minimizing AIC, BIC or the cross-validated measurement (such as Mean Squared Error).

3 Geometric convergence of computational error

The gradient descent algorithm (Nesterov, 2007; Agarwal, Negahban, and Wainwright, 2012) is usually applied to solve the convex problem (2.3). For example, we can replace the RA-quadratic loss with its local isotropic quadratic approximation (LQA) and iteratively solve the following optimization problem:

$${\widehat{\beta }}^{t+1}=\underset{{\Vert \beta \Vert }_{1\le \rho }}{\mathrm{argmin}}\left\{{\mathcal{L}}_n\left({\widehat{\beta }}^t\right)+{[\nabla {\mathcal{L}}_n\left({\widehat{\beta }}^t\right)]}^T(\beta -{\widehat{\beta }}^t)+\frac{{\gamma }_u}{2}{\Vert \beta -{\widehat{\beta }}^t\Vert }_2^2+{\lambda }_n{\Vert \beta \Vert }_1\right\},$$ (3.1)

where γ_u is a sufficiently large fixed constant whose condition is specified in (3.3) and the side constraint “∥β∥₁ ≤ ρ” is introduced to guarantee good performance in the first few iterations and ρ is allowed to be sufficiently large such that β* is feasible. The isotropic local quadratic approximation allows an expedient computation. To solve (3.1), the update can be computed by a two-step procedure. We first solve (3.1) without the norm constraint, which is the soft-threshold of the vector ${\widehat{\beta }}^t-\frac{1}{\gamma u}\nabla {\mathcal{L}}_n\left({\widehat{\beta }}^t\right)$ at level λ_n, and call the solution $\check{\beta }$. If ${\Vert \check{\beta }\Vert }_1\le \rho$, set ${\widehat{\beta }}^{t+1}=\check{\beta }$. Otherwise, ${\widehat{\beta }}^{t+1}$ is obtained by further projecting $\check{\beta }$ onto the L₁-ball {β : ∥β∥₁ ≤ ρ}. The projection can be done (Duchi, et al., 2008) by soft-thresholding $\check{\beta }$ at level π_n, where π_n is given by the following procedure: (1) sort ${\left\{\mid {\check{\beta }}_j\mid \right\}}_{j=1}^p$ into $b_1\ge b_2\ge \dots \ge b_p$; (2) find $J=\max \{1\le j\le p:b_j-({\sum }_{r=1}^j{b}_r-\rho )∕j>0\}$ and $({\sum }_{r=1}^J{b}_j-\rho )∕J$.

Agarwal, Negahban, and Wainwright (2012) considered the computational error of such firstorder gradient descent method. They showed that, for a convex and differentiable loss functions $\ell \left(x\right)$ and decomposable penalty function p(β), the error ${\Vert {\widehat{\beta }}^t-{\beta }^{\ast }\Vert }_2$ has the same rate as ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$ for all sufficiently large t, where ${\beta }^{\ast }={\mathrm{argmin}}_{\beta }\mathrm{E}\ell (\mathbf{x},y;\beta )$, and $\widehat{\beta }={\mathrm{argmin}}_{\beta }\frac{1}{n}{\sum }_{i=1}^n\ell ({\mathbf{x}}_i,y_i,\beta )+p\left(\beta \right)$. Different from their setup, our population minimizer ${\beta }_{\alpha }^{\ast }$ varies by α. Nevertheless, as ${\beta }_{\alpha }^{\ast }$ converges to the true effect β*, by a careful control of a, we can still show that ${\Vert {\widehat{\beta }}^t-{\beta }^{\ast }\Vert }_2$ has the same rate as ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$, where $\widehat{\beta }$ is the theoretical solution of (2.3) and ${\widehat{\beta }}^t$ is as defined in (3.1).

The key is that the RA-quadratic loss function ${\mathcal{L}}_n$ satisfies the restricted strong convexity (RSC) condition and the restricted smoothness condition (RSM) with some uniform constants, namely $\delta {\mathcal{L}}_n(\Delta ,\beta )$ as defined in (2.5) satisfies the following conditions:

$$\mathrm{RSC}:\delta {\mathcal{L}}_n(\Delta ,\beta )\ge \frac{{\gamma }_l}{2}{\Vert \Delta \Vert }_2^2-{\tau }_l{\Vert \Delta \Vert }_1^2,$$ (3.2)

$$\mathrm{RSM}:\delta {\mathcal{L}}_n(\Delta ,\beta )\le \frac{{\gamma }_u}{2}{\Vert \Delta \Vert }_2^2+{\tau }_u{\Vert \Delta \Vert }_1^2,$$ (3.3)

for all β and Δ in some set of interest, with parameters γ_l, τ_l, γ_u and τ_u that do not depend on α. We show that such conditions hold with high probability.

Lemma 4

Under condition (C1)-(C3), for all ${\Vert \beta \Vert }_2\le 4{\rho }_2$, ${\Vert \Delta \Vert }_2\le 8{\rho }_2$ and $\alpha \le c_u{\rho }_2^{-1}$, with probability greater than 1 – c₁ exp(–c₂n), (3.2) and (3.3) hold with γ_l = κ1, ${\tau }_l={\kappa }_1{\kappa }_2^2\left(\log p\right)∕\left(2n\right)$, γ_u = 3κ_u, τ_u = κ_u(log p)/n.

We further give an upper bound of computational error ${\Vert {\widehat{\beta }}^t-\widehat{\beta }\Vert }_2$ in Theorem 4. It shows that with high probability, ${\Vert {\widehat{\beta }}^t-\widehat{\beta }\Vert }_2$ is dominated by ${\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2$ after sufficient iterations, as long as $R_q{\left(\frac{\log p}{n}\right)}^{1-(q∕2)}=o\left(1\right)$, which is required for consistency of any method over the weak sparse L_q ball by the known minimax results (Raskutti, Wainwright, and Yu, 2011). Denote $r_n^2=R_q{\left(\frac{\log p}{n}\right)}^{1-(q∕2)}$. Theorems 3 and 4 below imply that, with high probability,

$$\begin{array}{cc}\hfill {\Vert {\widehat{\beta }}^t-{\beta }^{\ast }\Vert }_2& \le {\Vert {\widehat{\beta }}^t-\widehat{\beta }\Vert }_2+{\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2+{\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2\hfill \\ \hfill & \le \sqrt{d_3}{r}_n{({\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2^2+r_n^2)}^{1∕2}+d_2{r}_n+d_1{\alpha }^{k-1}\hfill \\ \hfill & \le {\left\{d_3(d_2^2+1)\right\}}^{1∕2}{r}_n^2+d_2{r}_n+d_1{\alpha }^{k-1}\hfill \\ \hfill & \le 2d_2{r}_n+d_1{\alpha }^{k-1},\hfill \end{array}$$

when the sample size is large enough to ensure $r_n\le d_2{\left\{d_3(d_2^2+1)\right\}}^{-1∕2}$. Therefore, ${\Vert {\widehat{\beta }}^t-{\beta }^{\ast }\Vert }_2$ has the same rate as ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$. Hence, from a statistical point of view, there is no need to iterate beyond t steps.

Theorem 4

Under conditions of Theorem 3, suppose we choose λ_n as in Lemma 1 and also satisfying

$${\lambda }_n\ge \frac{32\rho }{1-\kappa }{\left(1-\frac{64{\kappa }_u\mid S_{\alpha \eta }\mid \log p}{n{\stackrel{‒}{\gamma }}_l}\right)}^{-1}\left[1+{\kappa }_1{\kappa }_2^2\left(\frac{{\stackrel{‒}{\gamma }}_l}{12{\kappa }_u}+\frac{128{\kappa }_u\mid S_{\alpha \eta }\mid \log p}{n{\stackrel{‒}{\gamma }}_l}\right)+8{\kappa }_u\right]\frac{\log p}{n},$$

where |S_αη| denotes the cardinality of set S_αη and ${\stackrel{‒}{\gamma }}_l={\gamma }_l-64{\tau }_l\mid S_{\alpha \eta }\mid$, then with probability at least 1 – c₁ exp(–c₂n), there exists a generic positive constant d₃ such that

$${\Vert {\widehat{\beta }}^t-\widehat{\beta }\Vert }_2^2\le d_3{R}_q{\left(\frac{\log p}{n}\right)}^{1-(q∕2)}\left[{\Vert \widehat{\beta }-{\beta }_{\alpha }^{\ast }\Vert }_2^2+R_q{\left(\frac{\log p}{n}\right)}^{1-(q∕2)}\right],$$ (3.4)

for all iterations

$$t\ge \frac{2\log (({\varphi }_n\left({\widehat{\beta }}^0\right)-{\varphi }_n\left(\widehat{\beta }\right))∕{\delta }^2)}{\log (1∕\kappa )}+{\log }_2{\log }_2\left(\frac{\rho {\lambda }_n}{{\delta }^2}\right)\left(1+\frac{\log 2}{\log (1∕\kappa )}\right),$$

where ${\varphi }_n\left(\beta \right)={\mathcal{L}}_n\left(\beta \right)+{\lambda }_n{\Vert \beta \Vert }_1$ and ${\widehat{\beta }}^0$ is the initial value satisfying ${\Vert {\widehat{\beta }}^0-{\beta }^{\ast }\Vert }_2\le {\rho }_2$, $\delta ={\epsilon }^2∕(1-\kappa )$ is the tolerance level, κ, and ε are some constants as will be defined in (19) and (20) in the on-line supplementary file, respectively.

4 Robust estimation of mean and covariance matrix

The estimation of mean can be regarded as a univariate linear regression where the covariate equals to 1. In that special case, we have more explicit concentration result for the RA-mean estimator, which is the estimator that minimizes the RA-quadratic loss. Let ${\left\{y_i\right\}}_{i=1}^n$ be an i.i.d sample from some unknown distribution with E(y_i) = μ and var(y_i) = σ². The RA-mean estimator ${\widehat{\mu }}_{\alpha }$ of μ is the solution of

$$\sum _{i=1}^n\psi \left[\alpha (y_i-\mu )\right]=0,$$

for parameter α → 0, where the influence function ψ(x) = x if |x| ≤ 1, ψ(x) = 1, if x > 1 and ψ(x) = −1 if x < −1. The following theorem gives the exponential type of concentration of ${\widehat{\mu }}_{\alpha }$ around μ.

Theorem 5

Assume $\frac{\log (1∕\delta )}{n}\le 1∕8$ and let $\alpha =\sqrt{\frac{\log (1∕\delta )}{n{v}^2}}$ where v ≥ σ. Then,

$P\left(\mid {\widehat{\mu }}_{\alpha }-\mu \mid \ge 4v\sqrt{\frac{\log (1∕\delta )}{n}}\right)\le 2\delta$.

The above result provides fast concentration of the mean estimation with only two moments assumption. This is very useful for large scale hypothesis testing (Efron, 2010; Fan, Han, and Gu, 2012) and covariance matrix estimation (Bickel and Levina, 2008; Fan, Liao and Mincheva, 2013), where uniform convergence is required. Taking the estimation of large covariance matrix as an example, in order for the elements of the sample covariance matrix to converge uniformly, the aforementioned authors require the underlying multivariate distribution be sub-Gaussian. This restrictive assumptions can be removed if we apply the robust estimation with concentration bound. Regarding σ_ij = E X_iX_j as the expected value of the random variable X_iX_j (it is typically not the same as the median of X_iX_j), it can be estimated with accuracy

$$P\left(\mid {\widehat{\sigma }}_{ij}-{\sigma }_{ij}\mid \ge 4v\sqrt{\frac{\log (1∕\delta )}{n}}\right)\le 2\delta ,$$

where $v\ge {\max }_{i,j\le p}\sqrt{\mathrm{var}\left(X_i{X}_j\right)}$ and ${\widehat{\sigma }}_{ij}$ is RA-mean estimator using data ${\left\{X_{ik}{X}_{jk}\right\}}_{k=1}^n$. Since there are only O(p²) elements, by taking δ = p^–a for a > 2 and the union bound, we have

$$P\left\{\underset{i,j\le p}{\max }\mid {\widehat{\sigma }}_{ij}-{\sigma }_{ij}\mid \ge 4v\sqrt{\frac{a\log p}{n}}\right\}\le 2p^{2-a},$$

when ${\max }_{i\le p}\mathrm{E}{X}_i^4$ is bounded. This robustified covariance estimator requires much weaker condition than the sample covariance and has far wide applicability than the sample covariance. It can be regularized further in the same way as the sample covariance matrix.

5 Connection with Catoni loss

Catoni (2012) considered the estimation of the mean of heavy-tailed distribution with fast concentration. He proposed an M-estimator by solving

$$\sum _{i=1}^n{\psi }_c\left[\alpha (y_i-\theta )\right]=0,$$

where the influence function ψ_c(x) is chosen such that –log(1–x+x²/2) ≤ ψ_c(x) ≤ log(1+x+x²/2). He showed that this M-estimator has the exponential type of concentration by only requiring the existence of the variance. It performed as well as the sample mean under the light-tail case.

Catoni's idea can also be extended to the linear regression setting. Suppose we replace the RA-quadratic loss ${\ell }_{\alpha }\left(x\right)$ in (2.3) with Catoni loss

$${\ell }_{\alpha }^c\left(x\right)=\frac{2}{\alpha }{\int }_0^x{\psi }_c\left(\alpha t\right)dt,$$

where the influence function ψ_c(t) is given by

$${\psi }_c\left(t\right)=\mathrm{sgn}\left(t\right)\{-\log (1-\mid t\mid +t^2∕2)I(\mid t\mid <1)+\log \left(2\right)I(\mid t\mid \ge 1)\}.$$

Let ${\widehat{\beta }}^c$ be the corresponding solution. Then, ${\widehat{\beta }}^c$ has the same non-asymptotic upper bound as the RA-Lasso, which is stated as follows.

Theorem 6

Suppose condition (C1) holds for k = 2 or 3, (C2), (C3) and (2.7) hold. Then there exist generic positive constants c₁, c₂, d₄ and d₅, depending on M_k, κ₀, κ_l, κ_u and κ_λ, such that with probability at least 1 – c₁ exp(–c₂n),

$${\Vert {\widehat{\beta }}^c-{\beta }^{\ast }\Vert }_2\le d_4{\alpha }^{k-1}+d_5\sqrt{R_q}{[\left(\log p\right)∕n]}^{1∕2-q∕4}.$$

Unlike the RA-lasso, the order of bias of ${\widehat{\beta }}^c$ cannot be further improved, even when higher conditional moments of errors exist beyond the third order. The reason is that the Catoni loss is not exactly the quadratic loss over any finite intervals. Similar results regarding the computational error of ${\widehat{\beta }}^c$ could also be established as in Theorem 4, since the RSC/RSM conditions also hold for Catoni loss with uniform constants.

6 Variance Estimation

We estimate the unconditional variance σ² = Eε² based on the RA-Lasso estimator and a cross-validation scheme. To ease the presentation, we assume the data set can be evenly divided into J folds with m observations in each fold. Then, we estimate σ² by

$${\widehat{\sigma }}^2=\frac{1}{J}\sum _{j=1}^J\frac{1}{m}\sum _{i\in \text{fold}j}{(y_i-{\mathbf{x}}_i^T{\widehat{\beta }}^{(-j)})}^2,$$

where ${\widehat{\beta }}^{(-j)}$ is the RA-Lasso estimator obtained by using data points outside the j-th fold. We show that ${\widehat{\sigma }}^2$ is asymptotically efficient. Different from the existing cross-validation based method (Fan, Guo, and Hao, 2012), light-tail assumption is not needed due to the utilization of the RA-Lasso estimator.

Theorem 7

Under conditions of Theorem 3, if $R_q\left(\log p\right){}^{1-q∕2}∕n^{(1-q)∕2}\to 0$ for q ε (0,1), and $\alpha =O\left({\left\{R_q{[\left(\log p\right)∕n]}^{1-\frac{q}{2}}\right\}}^{\frac{1}{2(k-1)}}\right)$, then

$$\sqrt{n}({\widehat{\sigma }}^2-{\sigma }^2)\stackrel{\mathcal{D}}{\to }N(0,\mathrm{E}{ϵ}^4-{\sigma }^4).$$

7 Simulation Studies

In this section, we assess the finite sample performance of the RA-Lasso and compare it with other methods through various models. We simulated data from the following high dimensional model

$$y_i={\mathbf{x}}_i^T{\beta }^{\ast }+{ϵ}_i,{\mathbf{x}}_i\sim N(0,I_p),$$ (7.1)

where we generated n = 100 observations and the number of parameters was chosen to be p = 400. We chose the true regression coefficient vector as

$${\beta }^{\ast }={(3,\dots ,3,0,\dots ,0)}^T,$$

where the first 20 elements are all equal to 3 and the rest are all equal to 0. To involve various shapes of error distributions, we considered the following five scenarios:

1. Normal with mean 0 and variance 4 (N(0,4));

2. Two times the t-distribution with degrees of freedom 3 (2t₃);

3. Mixture of Normal distribution(MixN): 0.5N(−1, 4) + 0.5N(8, 1);

4. Log-normal distribution (LogNormal): ε = e^1+1.2Z, where Z is standard normal.

5. Weibull distribution with shape parameter = 0.3 and scale parameter = 0.5 (Weibull).

In order to meet the model assumption, the errors were standardized to have mean 0. Table 1 categorizes the five scenarios according to the shapes and tails of the error distributions.

Table 1.

Summary of the shapes and tails of five error distributions

	Light Tail	Heavy Tail
Symmetric	N (0, 4)	2t3
Asymmetric	MixN	LogNormal, Weibull

To obtain our estimator, we iteratively applied the gradient descent algorithm. We compared RA-Lasso with two other methods in high-dimensional setting: (a) Lasso: the penalized least-squares estimator with L₁-penalty as in Tibshirani (1996); and (b) R-Lasso: the R-Lasso estimator in Fan, Fan, and Barut (2014), which is the same as the regularized LAD estimator with L₁-penalty as in Wang (2013). Their performance under the five scenarios was evaluated by the following four measurements:

• (1)L₂ error, which is defined as ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_2$.
• (2)L₁ error, which is defined as ${\Vert \widehat{\beta }-{\beta }^{\ast }\Vert }_1$.
• (3)Number of false positives (FP), which is number of noise covariates that are selected.
• (4)Number of false negatives (FN), which is number of signal covariates that are not selected.

We also measured the relative gain of RA-Lasso with respect to R-Lasso and Lasso, in terms of the difference to the oracle estimator. The oracle estimator ${\widehat{\beta }}_{\text{oracle}}$ is defined to be the least square estimator by using the first 20 covariates only. Then, the relative gain of RA-Lasso with respect to Lasso (RG_A,L) in L₂ and L₁ norm are defined as

$$\frac{{\Vert {\widehat{\beta }}_{\text{Lasso}}-{\beta }^{\ast }\Vert }_2-{\Vert {\widehat{\beta }}_{\text{oracle}}-{\beta }^{\ast }\Vert }_2}{{\Vert {\widehat{\beta }}_{\text{RA-Lasso}}-{\beta }^{\ast }\Vert }_2-{\Vert {\widehat{\beta }}_{\text{oracle}}-{\beta }^{\ast }\Vert }_2}\text{and}\frac{{\Vert {\widehat{\beta }}_{\text{Lasso}}-{\beta }^{\ast }\Vert }_1-{\Vert {\widehat{\beta }}_{\text{oracle}}-{\beta }^{\ast }\Vert }_1}{{\Vert {\widehat{\beta }}_{\text{RA-Lasso}}-{\beta }^{\ast }\Vert }_1-{\Vert {\widehat{\beta }}_{\text{oracle}}-{\beta }^{\ast }\Vert }_1}.$$

The relative gain of RA-Lasso with respect to R-Lasso (RG_A,R) is defined similarly.

For RA-Lasso, the tuning parameters λ_n and α were chosen optimally based on 100 independent validation datasets. We ran a 2-dimensional grid search to find the best (λ_n, α) pair that minimizes the mean L₂-loss of the 100 validation datasets. Such an optimal pair was then used in the simulations. Similar method was applied in choosing the tuning parameters in Lasso and R-Lasso.

The above simulation model is based on the additive model (7.1), in which error distribution is independent of covariates. However, this homoscedastic model makes the conditional mean and the conditional median differ only by a constant. To further examine the deviations between the mean regression and median regression, we also simulated the data from the heteroscedastic model

$$y_i={\mathbf{x}}_i^T{\beta }^{\ast }+c^{-1}{\left({\mathbf{x}}_i^T{\beta }^{\ast }\right)}^2{ϵ}_i,{\mathbf{x}}_i\sim N(0,I_p),$$ (7.2)

where the constant $c=\sqrt{3}{\Vert {\beta }^{\ast }\Vert }^2$ makes $\mathrm{E}{\left[c^{-1}{\left({\mathbf{x}}_i^T{\beta }^{\ast }\right)}^2\right]}^2=1$. Note that ${\mathbf{x}}_i^T{\beta }^{\ast }\sim N(0,{\Vert {\beta }^{\ast }\Vert }^2)$ and therefore c is chosen so that the average noise levels is the same as that of ε_i. For both the homoscedastic and the heteroscedastic models, we ran 100 simulations for each scenario. The mean of each performance measurement is reported in Table 2 and 3, respectively.

Table 2.

Simulation results of Lasso, R-Lasso and RA-Lasso under homoscedastic model. (7.1)

		Lasso	R-Lasso	RA-Lasso	RGA,L	RGA,R
N(0, 4)	L2 loss	4.54	4.40	4.53	1.00	0.96
	L1 loss	27.21	29.11	27.21	1.00	1.08
	FP, FN	52.10, 0.09	66.36, 0.17	52.10, 0.09
2t3	L2 loss	6.04	5.10	5.47	1.14	0.91
	L1 loss	35.22	33.07	30.42	1.19	1.10
	FP, FN	47.13, 0.34	65.84, 0.22	41.34, 0.28
MixN	L2 loss	6.14	6.44	6.13	1.00	1.06
	L1 loss	40.46	46.18	38.48	1.06	1.23
	FP, FN	65.99, 0.34	80.31, 0.33	58.05, 0.39
LogNormal	L2 loss	11.08	12.16	10.10	1.14	1.30
	L1 loss	53.17	57.18	51.58	1.04	1.14
	FP, FN	26.5, 15.00	27.20, 6.90	37.20, 3.90
Weibull	L2 loss	7.77	7.11	6.62	1.23	1.10
	L1 loss	55.65	50.49	42.93	1.34	1.20
	FP, FN	78.70, 0.71	77.13, 0.56	62.27,0.52

Table 3.

Simulation results of Lasso, R-Lasso and RA-Lasso under heteroscedastic model (7.2).

		Lasso	R-Lasso	RA-Lasso	RGA,L	RGA,R
N(0, 4)	L2 loss	4.60	4.34	4.60	1.00	0.93
	L1 loss	27.16	27.14	27.15	1.00	1.00
	FP, FN	48.78, 0.10	58.25, 0.27	48.78, 0.10
2t3	L2 loss	8.08	6.71	6.70	1.26	1.01
	L1 loss	41.16	42.76	38.52	1.08	1.12
	FP, FN	55.33, 0.67	71.67, 0.33	45.33, 0.33
MixN	L2 loss	6.26	6.54	6.25	1.00	1.06
	L1 loss	41.26	46.95	39.25	1.06	1.23
	FP, FN	65.98, 0.34	80.30, 0.32	58.80 0.34
LogNormal	L2 loss	10.86	9.19	8.48	1.43	1.13
	L1 loss	57.52	57.18	53.20	1.10	1.09
	FP, FN	29.70, 5.70	54.10, 2.00	54.30, 1.50
Weibull	L2 loss	7.40	8.81	5.53	1.53	1.92
	L1 loss	40.95	47.82	34.65	1.23	1.48
	FP, FN	38.87,0.96	35.31, 2.90	58.15,0.39

Tables 2 and 3 indicate that our method had the biggest advantage when the errors were asymmetric and heavy-tailed (LogNormal and Weibull). In this case, R-Lasso had larger L₁ and L₂ errors due to the bias for estimating the conditional median instead of the mean. Even though Lasso did not have bias in the loss component (quadratic loss), it did not perform well due to its sensitivity to outliers. The advantage of our method is more pronounced in the heteroscedastic model than in the homoscedastic model. Both of them clearly indicate that if the errors come from asymmetric and heavy-tailed distributions, our method is better than both Lasso and R-Lasso. When the errors were symmetric and heavy-tailed (2t₃), our estimator performed closely as R-Lasso, both of which outperformed Lasso. The above two cases evidently showed that RA-Lasso was robust to the outliers and did not lose efficiency when the errors were indeed symmetric. Under the light-tailed scenario, if the errors were asymmetric (MixN), our method performed similarly as Lasso. R-Lasso performed worse, since it had bias. For the regular setting (N(0, 4)), where the errors were light-tailed and symmetric, the three methods were comparable with each other.

In conclusion, RA-Lasso is more flexible than Lasso and R-Lasso. The tuning parameter α automatically adapts to errors with different shapes and tails. It enables RA-Lasso to render consistently satisfactory results under all scenarios.

8 Real Data Example

In this section, we use a microarray data to illustrate the performance of Lasso, R-Lasso and RA-Lasso. Huang, et al. (2011) studied the role of innate immune system on the development of atherosclerosis by examining gene profiles from peripheral blood of 119 patients. The data were collected using Illumina HumanRef8 V2.0 Bead Chip and are available on Gene Expression Omnibus. The original study showed that the toll-like receptors (TLR) signaling pathway plays an important role on triggering the innate immune system in face of atherosclerosis. Under this pathway, the “TLR8” gene was found to be a key atherosclerosis-associated gene. To further study the relationship between this key gene and the other genes, we regressed it on another 464 genes from 12 different pathways (TLR, CCC, CIR, IFNG, MAPK, RAPO, EXAPO, INAPO, DRS, NOD, EPO, CTR) that are related to the TLR pathway. We applied Lasso, R-Lasso and RA-Lasso to this data. The tuning parameters for all methods were chosen by using five-fold cross validation. Figure 1 shows our choice of the penalization parameter based on the cross validation results. For RA-Lasso, the choice of a was insensitive to the results and was fixed at 5. We then applied the three methods with the above choice of tuning parameters to select significant genes. The QQ-plots of the residuals from the three methods are shown in Figure 2. The selected genes by the three methods are reported in Table 4. After the selection, we regressed the expression of TLR8 gene on the selected genes, the t-values from the refittings are also reported in Table 4.

Figure 1.

Five-fold cross validation results: black dot marks the choice of the penalization parameter.

Figure 2.

QQ plots of the residuals from three methods.

Table 4.

Selected genes by Lasso, R-Lasso and RA-Lasso.

Lasso	CRK 0.23
R-Lasso	CSF3	IL10	AKT1	KPNB1	TLR2	GRB2	MAPK1
	−2.46	2.24	1.68	1.49	1.41	−1.06	0.98
	DAPK2	TOLLIP	TLR1	TLR3	SHC1	PSMD1	F12
	0.7	−0.68	0.52	0.33	−0.28	0.27	0.24
	EPOR	TJP1	GAB2
	−0.17	−0.12	−0.01
RA-Lasso	CSF3	CD3E	BTK	CLSPN	RELA	AKT1	IRS2
	−2.95	2.67	2.37	1.93	1.88	1.61	1.55
	IL10	MAP2K4	PMAIP1	BCL2L11	AKT3	DUSP10	IRF4
	1.52	1.17	−1.14	−1.13	−1.01	0.97	−0.95
	IFI6	TLR1	PSMB8	KPNB1	IFNG	FADD	TJP1
	0.86	0.82	0.79	0.77	−0.74	0.65	−0.57
	CR2	IL2	PSMC2	HSPA8	SHC1	SPI1	IFNA6
	0.57	−0.47	0.38	−0.35	−0.33	−0.28	0.28
	FYN	EPOR	MASP1	PRKCZ	TOLLIP	BAK1
	−0.24	0.24	−0.24	0.24	−0.19	0.14

Table 4 shows that Lasso only selected one gene. R-Lasso selected 17 genes. Our proposed RA-Lasso selected 34 genes. Eight genes (CSF3, IL10, AKT1, TOLLIP, TLR1, SHC1, EPOR, and TJP1) found by R-Lasso were also selected by RA-Lasso. Compared with Lasso and R-Lasso, our method selected more genes, which could be useful for a second-stage confirmatory study. It is clearly seen from Figure 2 that the residuals from the fitted regressions had heavy right tail and skewed distribution. We learn from the simulation studies in Section 7 that RA-Lasso tends to perform better than Lasso and R-Lasso in this situation. For further investigation, we randomly chose 24 patients as the test set; applied three methods to the rest patients to obtain the estimated coefficients, which in return were used to predict the responses of 24 patients. We repeated the random splitting 100 times, the boxplots of the Mean Absolute/Squared Error of predictions are shown in Figure 3. RA-Lasso has better predictions than Lasso and R-Lasso.

Figure 3.

Boxplot of the Mean Absolute/Squared Error of predictions.

Appendix: Proofs of Theorem 1, 2 and 5

Proof of Theorem 1

Let $\ell \left(x\right)=x^2$. Since β* minimizes $\mathrm{E}\ell (y-{\mathbf{x}}^T\beta )$, it follows from condition(C2) that

$$\mathrm{E}[\ell (y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-\ell (y-{\mathbf{x}}^T{\beta }^{\ast })]={({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })}^T\mathrm{E}\left(\mathbf{x}{\mathbf{x}}^T\right)({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\ge {\kappa }_l{\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2^2.$$ (A.1)

Let $g_{\alpha }\left(x\right)=\ell \left(x\right)-{\ell }_{\alpha }\left(x\right)={(\mid x\mid -{\alpha }^{-1})}^{2}I(\mid x\mid >{\alpha }^{-1})$. Then, since ${\beta }_{\alpha }^{\ast }$ is the minimizer of $E{\ell }_{\alpha }(y-{\mathbf{x}}^T\beta )$, we have

$$\begin{array}{cc}\hfill & \mathrm{E}[\ell (y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-\ell (y-{\mathbf{x}}^T{\beta }^{\ast })]\hfill \\ \hfill =& \mathrm{E}[\ell (y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-{\ell }_{\alpha }(y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })]+\mathrm{E}[{\ell }_{\alpha }(y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-{\ell }_{\alpha }(y-{\mathbf{x}}^T{\beta }^{\ast })]+\mathrm{E}[{\ell }_{\alpha }(y-{\mathbf{x}}^T{\beta }^{\ast })-\ell (y-{\mathbf{x}}^T{\beta }^{\ast })]\hfill \\ \hfill \le & \mathrm{E}\left[g_{\alpha }(y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })\right]-\mathrm{E}\left[g_{\alpha }(y-{\mathbf{x}}^T{\beta }^{\ast })\right].\hfill \end{array}$$

By Taylor's expansion, we have

$$\mathrm{E}[\ell (y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-{\ell }_{\alpha }(y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })]\le 2\mathrm{E}\left[(z-{\alpha }^{-1})I(z>{\alpha }^{-1})\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right],$$ (A.2)

where $\tilde{\beta }$ is a vector lying between β* and ${\beta }_{\alpha }^{\ast }$ and $z=\mid y-{\mathbf{x}}^T\tilde{\beta }\mid$. With P_ε denoting the distribution of ε conditioning on x and E_ε the corresponding expectation, we have

$$\begin{array}{cc}\hfill {\mathrm{E}}_{ϵ}\left[(z-{\alpha }^{-1})I(z>{\alpha }^{-1})\right]=& {\int }_0^{\infty }{P}_{ϵ}(zI(z>{\alpha }^{-1})>t)dt-{\alpha }^{-1}{P}_{ϵ}(z>{\alpha }^{-1})\hfill \\ \hfill =& {\int }_0^{\infty }{P}_{ϵ}(z>t\text{and}z>{\alpha }^{-1})dt-{\alpha }^{-1}{P}_{ϵ}(z>{\alpha }^{-1})\hfill \\ \hfill =& {\int }_{{\alpha }^{-1}}^{\infty }{P}_{ϵ}(z>t)dt+{\int }_0^{{\alpha }^{-1}}{P}_{ϵ}(z>{\alpha }^{-1})dt-{\alpha }^{-1}{P}_{ϵ}(z>{\alpha }^{-1})\hfill \\ \hfill \le & {\int }_{{\alpha }^{-1}}^{\infty }\frac{{\mathrm{E}}_{ϵ}\left(z^k\right)}{t^k}dt\le {\alpha }^{k-1}{\mathrm{E}}_{ϵ}\left(z^k\right).\hfill \end{array}$$

Therefore, $\mathrm{E}[\ell (y-{\mathbf{x}}^T{\beta }_{\alpha }^{\ast })-\ell (y-{\mathbf{x}}^T{\beta }^{\ast })]$ is further bounded by

$$\begin{array}{cc}\hfill & 2{\alpha }^{k-1}\mathrm{E}\left\{{\mid y-{\mathbf{x}}^T\tilde{\beta }\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}\hfill \\ \hfill =& 2{\alpha }^{k-1}\mathrm{E}\left\{{\mid ϵ+{\mathbf{x}}^T({\beta }^{\ast }-\tilde{\beta })\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}\hfill \\ \hfill =& 2{\left(2\alpha \right)}^{k-1}\left[\mathrm{E}\left\{{\mid ϵ\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}+\mathrm{E}\left\{{\mid {\mathbf{x}}^T({\beta }^{\ast }-\tilde{\beta })\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}\right]\hfill \end{array}$$ (A.3)

Note that,

$$\begin{array}{cc}\hfill \mathrm{E}\left\{{\mid ϵ\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}& =\mathrm{E}\left\{\mathrm{E}({\mid ϵ\mid }^k\mid \mathbf{x})\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\right\}\le {\left[\mathrm{E}{\left\{\mathrm{E}({\mid ϵ\mid }^k\mid \mathbf{x})\right\}}^2\right]}^{\frac{1}{2}}{\left[\mathrm{E}{\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid }^2\right]}^{\frac{1}{2}}\hfill \\ \hfill & \le \sqrt{M_k{\kappa }_u}{\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2.\hfill \end{array}$$

where the last inequality follows from (C1) and (C2). On the other hand, by (C3), ${\mathbf{x}}^T({\beta }^{\ast }-\tilde{\beta })$ is sub-Gaussian, hence its 2k-th moment is bounded by $c^2{\kappa }_0^{2k}$, for a universal positive constant c depending on k only. Then,

$$\begin{array}{cc}\hfill \mathrm{E}\left\{{\mid {\mathbf{x}}^T({\beta }^{\ast }-\tilde{\beta })\mid }^k\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid \right\}& \le {\left\{\mathrm{E}{\mid {\mathbf{x}}^T({\beta }^{\ast }-\tilde{\beta })\mid }^{2k}\right\}}^{\frac{1}{2}}{\left\{\mathrm{E}{\mid {\mathbf{x}}^T({\beta }_{\alpha }^{\ast }-{\beta }^{\ast })\mid }^2\right\}}^{\frac{1}{2}}\hfill \\ \hfill & \le c{\kappa }_0^k\sqrt{{\kappa }_u}{\Vert {\beta }_{\alpha }^{\ast }-{\beta }^{\ast }\Vert }_2.\hfill \end{array}$$

These results together with (A.1) and (A.3) completes the proof.

Proof of Theorem 2

Let A₁ and A₂ denote the events that Lemma 1 and Lemma 3 hold, respectively. By Theorem 1 of Negahban, et al. (2012), within $A_1\cap A_2$, it holds that

$$\begin{array}{cc}\hfill {\Vert \Delta \Vert }_2^2& \le 9\frac{{\lambda }_n^2}{{\kappa }_{\mathcal{L}}^2}\mid S_{\alpha \eta }\mid +\frac{{\lambda }_n}{{\kappa }_{\mathcal{L}}^2}\{2{\tau }_{\mathcal{L}}^2+4{\Vert {\beta }_{S_{\alpha \eta }^c}^{\ast }\Vert }_1\}\hfill \\ \hfill & \le \frac{36{\lambda }_n^2{R}_q}{{\kappa }_1^2{\eta }^q}+\frac{4{\lambda }_n}{{\kappa }_1^2}\left\{8R_q{\kappa }_1{\kappa }_2{{\kappa }_{\lambda }}^{1-q}{\left(\frac{\log p}{n}\right)}^{1-(q∕2)}+4R_q{\eta }^{1-q}\right\}\hfill \\ \hfill & \stackrel{\left(ii\right)}{=}\frac{36}{{\kappa }_1^2}{R}_q{\lambda }_n^{2-q}+\frac{16}{{\kappa }_1^2}{R}_q{\lambda }_n^{2-q}\left\{2{\kappa }_1{\kappa }_2{\left(\frac{\log p}{n}\right)}^{\frac{1}{2}}+1\right\}\hfill \\ \hfill & \stackrel{\left(ii\right)}{\le }{C}_2{k}_l^{-2}{\kappa }_{\lambda }^{2-q}{R}_q{\left(n^{-1}\log p\right)}^{1-(q∕2)},\hfill \end{array}$$

where (i) follows from the choice of η = λ_n, in (ii) we assume that the sample size n is large enough such that 2κ₁κ₂(n⁻¹ log p)^1/2 ≤ 1 and observe that κ₁ = κ_l/4. On the other hand, by Lemma 1 and 3, $P(A_1\cap A_2)\ge 1-c_1\exp (-c_{2}n)$, where $c_1=\max \{2,c_1^{\prime }\}$ and $c_2=\min \{c_0,c_2^{\prime }\}$.

Proof of Theorem 5

The proof follows the same spirit as the proof of Proposition 2.4 in Catoni (2012). The influence function ψ(x) of RA-quadratic loss satisfies

$$-\log (1-x+x^2)\le \psi \left(x\right)\le \log (1+x+x^2).$$

Using this and independence, with $r\left(\theta \right)=\frac{1}{\alpha n}{\sum }_{i=1}^n\psi \left[\alpha (Y_i-\theta )\right]$, we have

$$\begin{array}{cc}\hfill \mathrm{E}\left\{\exp \left[\alpha nr\left(\theta \right)\right]\right\}& \le {\left(\mathrm{E}\{\exp \left\{\psi \left[\alpha (Y_i-\theta )\right]\right\}\right)}^n\hfill \\ \hfill & \le {\{1+\alpha (\mu -\theta )+{\alpha }^2[{\sigma }^2+{(\mu -\theta )}^2]\}}^n\hfill \\ \hfill & \le \exp \{n\alpha (\mu -\theta )+n{\alpha }^2[v^2+{(\mu -\theta )}^2]\}.\hfill \end{array}$$

Similarly, $\mathrm{E}\left\{\exp [-\alpha nr\left(\theta \right)]\right\}\le \exp \{-n\alpha (\mu -\theta )+n{\alpha }^2[v^2+{(\mu -\theta )}^2]\}$. Define

$$\begin{array}{cc}\hfill B_{+}\left(\theta \right)=& \mu -\theta +\alpha [v^2+{(\mu -\theta )}^2]+\frac{\log (1∕\delta )}{n\alpha }\hfill \\ \hfill B_{-}\left(\theta \right)=& \mu -\theta -\alpha [v^2+{(\mu -\theta )}^2]-\frac{\log (1∕\delta )}{n\alpha }\hfill \end{array}$$

By Chebyshev inequality,

$$P(r\left(\theta \right)>B_{+}\left(\theta \right))\le \frac{\mathrm{E}\left\{\exp \left[\alpha nr\left(\theta \right)\right]\right\}}{\exp \{\alpha n(\mu -\theta )+n{\alpha }^2[v^2+{(\mu -\theta )}^2]+\log (1∕\delta )\}}\le \delta$$

Similarly, P(r(θ) < B₋(θ)) ≤ δ.

Let θ₊ be the smallest solution of the quadratic equation B₊(θ₊ = 0 and θ₋ be the largest solution of the equation B₋(θ_) = 0. Under the assumption that $\frac{\log (1∕\delta )}{n}\le 1∕8$ and the choice of $\alpha =\sqrt{\frac{\log (1∕\delta )}{n{v}^2}}$, we have ${\alpha }^2{v}^2+\frac{\log (1∕\delta )}{n}\le 1∕4$. Therefore,

$$\begin{array}{cc}\hfill {\theta }_{+}=& \mu +2\left(\alpha v^2+\frac{\log (1∕\delta )}{\alpha n}\right){\left(1+\sqrt{1-4\left({\alpha }^2{v}^2+\frac{\log (1∕\delta )}{n}\right)}\right)}^{-1}\hfill \\ \hfill & \le \mu +2\left(\alpha v^2+\frac{\log (1∕\delta )}{\alpha n}\right).\hfill \end{array}$$

Similarly,

$$\begin{array}{cc}\hfill {\theta }_{-}=& \mu -2\left(\alpha v^2+\frac{\log (1∕\delta )}{\alpha n}\right){\left(1+\sqrt{1-4\left({\alpha }^2{v}^2+\frac{\log (1∕\delta )}{n}\right)}\right)}^{-1}\hfill \\ \hfill & \ge \mu -2\left(\alpha v^2+\frac{\log (1∕\delta )}{\alpha n}\right).\hfill \end{array}$$

With $\alpha =\sqrt{\frac{\log (1∕\delta )}{n{v}^2}}$, ${\theta }_{+}\le \mu +4v\sqrt{\frac{\log (1∕\delta )}{n}}$, ${\theta }_{-}\ge \mu -4v\sqrt{\frac{\log (1∕\delta )}{n}}$ Since the map $\theta \mapsto r\left(\theta \right)$ is non-increasing, under event $\{B_{-}\left(\theta \right)\le r\left(\theta \right)\le B_{+}\left(\theta \right)\}$

$$\mu -4v\sqrt{\frac{\log (1∕\delta )}{n}}\le {\theta }_{-}\le {\widehat{\mu }}_{\alpha }\le {\theta }_{+}\le \mu +4v\sqrt{\frac{\log (1∕\delta )}{n}},$$

i.e. $\mid {\widehat{\mu }}_{\alpha }-\mu \mid \le 4v\sqrt{\frac{\log (1∕\delta )}{n}}$. Meanwhile, $P(B_{-}\left(\theta \right)\le r\left(\theta \right)\le B_{+}\left(\theta \right))>1-2\delta$.