Regularized Quantile Regression and Robust Feature Screening for Single Index Models

Abstract

We propose both a penalized quantile regression and an independence screening procedure to identify important covariates and to exclude unimportant ones for a general class of ultrahigh dimensional single-index models, in which the conditional distribution of the response depends on the covariates via a single-index structure. We observe that the linear quantile regression yields a consistent estimator of the direction of the index parameter in the single-index model. Such an observation dramatically reduces computational complexity in selecting important covariates in the single-index model. We establish an oracle property for the penalized quantile regression estimator when the covariate dimension increases at an exponential rate of the sample size. From a practical perspective, however, when the covariate dimension is extremely large, the penalized quantile regression may suffer from at least two drawbacks: computational expediency and algorithmic stability. To address these issues, we propose an independence screening procedure which is robust to model misspecification, and has reliable performance when the distribution of the response variable is heavily tailed or response realizations contain extreme values. The new independence screening procedure offers a useful complement to the penalized quantile regression since it helps to reduce the covariate dimension from ultrahigh dimensionality to a moderate scale. Based on the reduced model, the penalized linear quantile regression further refines selection of important covariates at different quantile levels. We examine the finite sample performance of the newly proposed procedure by Monte Carlo simulations and demonstrate the proposed methodology by an empirical analysis of a real data set.

1. Introduction

Single index regression models are widely assumed to avoid the “curse of dimensionality”. Let Y be a response variable and x be the associated covariate vector. Traditional single index regression model is referred to as

$$Y=m\left({\mathbf{x}}^{\mathrm{T}}{\beta }_0\right)+\epsilon ,$$ (1.1)

where m(·) is an unknown regression function, β₀ consists of unknown index parameters, and ε is a random error with E(ε | x) = 0 and var(ε | x) = σ². Model (1.1) has been well studied in the literature. See, for example, Powell, Stock and Stoker (1989) and Härdle, Hall and Ichimura (1993). Zhu, Huang and Li (2012) studied the following heteroscedastic single index regression model

$$Y=m\left({\mathbf{x}}^{\mathrm{T}}{\beta }_0\right)+\sigma \left({\mathbf{x}}^{\mathrm{T}}{\beta }_0\right)\epsilon ,$$ (1.2)

for unknown functions m(·) and σ(·), where ε has mean zero and is assumed to be independent of x. Zhu, Huang and Li (2012) developed an estimation procedure for β₀ and m(·) under a quantile loss function when the dimension of x is finite.

The goal of regression analysis amounts to characterizing how the conditional distribution of the response variable Y varies with the realizations of the covariate vector x = (X₁, …, X_pn)^T. In this paper, we focus on ultrahigh dimensional situation. Thus, we denote by p_n the dimension of x to emphasize the dependence of p_n on the sample size n. Denote by F (y | x) the conditional distribution of Y given x. In this paper, we study a general class of single index models that include models (1.1) and (1.2) as special cases. Specifically, we assume that there exists β₀ ∈ ℝ^p_n such that

$$F(y\mid \mathbf{x})=F(y\mid {\mathbf{x}}^{\mathrm{T}}{\beta }_0),\mathrm{for}\mathrm{all}y\in \mathbb{ℝ}.$$ (1.3)

That is, the conditional distribution of (Y | x) is fully characterized through a single linear combination of predictors x^T β₀. Consequently, the “curse of dimensionality” issue is avoided and simultaneously the model interpretability is maintained via a single index structure. Because the conditional distributional function F (· | ·) is unknown, the index parameter β₀ is not identifiable. The direction of β₀, instead of its true value, is of our primary interest. We refer to model (1.3) as a conditional distribution based single index model (CDSIM for short) in order to distinguish it from models (1.1) and (1.2).

When the covariate dimension is high, it is natural to assume that some covariates are irrelevant. The presence of irrelevant covariates may substantially deteriorate the precision of parameter estimation and the accuracy of response prediction (Altham, 1984). In the context of linear regression or generalized linear regression, many regularization methods, such as the LASSO (Tibshirani, 1996), the SCAD (Fan and Li, 2001; Zou and Li, 2008), the adaptive LASSO (Zou, 2006), the MCP (Zhang, 2010), the hard thresholding penalty (Zheng, Fan and Lv, 2014) and general penalty functions (Fan and Lv, 2013) have been proposed to remove those irrelevant covariates and simultaneously estimate the nonzero coefficients. Naik and Tsai (2001), Kong and Xia (2007), Zhu, Qian and Lin (2011) and Liang et al. (2010) developed some regularization methods for single-index regression. Recently, Wang, Wu and Li (2012) investigated the nonconvex penalized quantile regression for analyzing heterogeneity in the ultrahigh dimensional setting. Fan, Fan and Barut (2014) proposed two-step adaptive robust LASSO based on weighted L₁-penalized quantile regression to deal with heavy-tailed high dimensional data.

In this paper, we consider variable selection and feature screening for model (1.3) when the covariate dimension p_n is ultrahigh. We further assume β₀ is sparse. Denote by A the active index set, β_A the nonzero entries of β₀ and x_A the collection of all active covariates. When β₀ is sparse, model (1.3) reduces to

$$F(y\mid \mathbf{x})=F(y\mid {\mathbf{x}}_A^{\mathrm{T}}{\beta }_A),\mathrm{for}\mathrm{all}y\in \mathbb{ℝ}.$$ (1.4)

Our goal is to identify A and if possible, to estimate β_A. To the best of our knowledge, there is few variable selection method designed for model (1.3) or (1.4) with ultrahigh-dimensional covariates.

In this paper we introduce two approaches to accomplish our goal: a penalized linear quantile regression and an independence screening procedure. When model (1.3) is true, the quantile functions of (Y | x) always vary with the realizations of (x^T β₀). In other words, the quantile function admits a single index structure. This motivates us to implement a penalized quantile regression to exclude irrelevant covariates and simultaneously estimate the direction of β₀. The advantage of using quantile regression is that the quantile function characterizes equivalently the distributional function (1.3) and it is resilient to outliers and extreme values in the response. We show that, although the true quantile functions of (Y | x) are possibly nonlinear, the resulting estimator obtained from penalized linear quantile regression remains consistent up to a proportionality constant. This strategy helps to reduce the computational complexity substantially in estimating (1.3) in that the linear quantile regression procedure avoids estimating nonlinear quantile functions. Due to its computational efficiency, it is appealing for ultrahigh dimensional data analysis. We show that the penalized linear quantile regression estimate has the oracle property under mild regularity conditions even when p_n tends to ∞ in an exponential rate of n.

From a practical perspective, when the covariate dimension is extremely large, even the penalized linear quantile regression may suffer from at least two serious drawbacks: computational inexpediency and algorithmic instability (Fan, Samworth and Wu, 2009). To further reduce the computational complexity in selecting important covariates from ultrahigh dimensional candidates, we further introduce an independence screening procedure which ranks the importance of each covariate through its distance correlation with the marginal distribution function of the response in model (1.3) and the implicit model (1.4). Since the distribution function is bounded and monotone, we can reasonably expect that this new independence screening procedure still works in the presence of outliers or extreme values in the response variable. In addition, it is computationally efficient and hence offers a useful complement, rather an alterative, to the penalized quantile regression approach since the proposed independence screening can precede the penalized quantile regression when the latter fails to produce a reliable solution within a tolerant time. Based on the reduced model, the penalized quantile regression may further refine selection of important covariates at different quantile levels. We show that this new independence screening procedure has the sure screening property even when p_n is ultrahigh.

This paper is organized as follows. In Section 2, we propose the penalized linear quantile regression and study the consistency and the oracle property of the resulting estimator. We propose a robust independence screening procedure and establish its sure screening property in Section 3. We compare the finite sample performance of our proposals with several competitors in Section 4. All technical proofs are given in the Appendix.

2. Penalized Linear Quantile Regression

In this section, we will construct an estimate for the direction of β₀ in model (1.3) via the penalized linear quantile regression.

2.1 The Methodology

Model (1.3) and its sparse structure (1.4) indicate that the quantile functions of (Y | x) at different quantile levels are all functions of (x^T β₀) and (${\mathbf{x}}_A^{\mathrm{T}}{\beta }_A$) if the sparsity principle applies. This motivates us to estimate β₀ through the quantile functions at different levels. Similar to Zhu, Huang and Li (2012), we first show that linear quantile regression can be used to estimate the direction of β₀ in model (1.3). To be specific, we denote by ρ_τ (r) = τ r − rI(r < 0), the check loss function at the τ th quantile, for τ ∈ (0, 1). Define

$${ℒ}_{\tau }(u,\mathbf{b})=E\left\{{\rho }_{\tau }(Y-u-{\mathbf{x}}^{\mathrm{T}}\mathbf{b})\right\}\mathrm{and}(u_{\tau },{\beta }_{\tau })=\underset{u,\mathbf{b}}{\mathrm{argmin}}\left\{{ℒ}_{\tau }(u,\mathbf{b})\right\},$$ (2.1)

where b = (b₁, …, b_pn)^T ∈ ℝ^p_n.

Lemma 1

If x satisfies the linearity condition that

$$E\{\mathbf{x}-E\left(\mathbf{x}\right)\mid {\mathbf{x}}^T{\beta }_0\}=var\left(\mathbf{x}\right){\beta }_0{\left\{{\beta }_0^{T}var\left(\mathbf{x}\right){\beta }_0\right\}}^{-1}{\beta }_0^T\{\mathbf{x}-E\left(\mathbf{x}\right)\},$$

then β_τ is proportional to β₀ in model (1.3).

The linearity condition is satisfied when x follows an elliptically contour distribution (Li, 1991). Hall and Li (1993) demonstrated that, regardless of the covariate distribution, the linearity condition always offers an ideal approximation to the reality as long as p_n is sufficient large. Therefore, the linearity condition is typically regarded as mild in an ultrahigh dimensional setting. Lemma 1 implies that the indices of zero entries in both β₀ and β_τ coincide. To estimate the direction of β₀ in model (1.3), it amounts to estimating β_τ defined in (2.1). This lemma can be proved by using similar arguments used in Zhu, Huang and Li (2012). Thus, we omit its proof to save space.

When the covariate dimension is large, it is desirable to exclude irrelevant covariates and simultaneously estimate β_τ in (2.1). Note that β_τ is identifiable because the linear quantile loss function L_τ (u, b) is convex. Suppose that {(x_i, Y_i), i = 1, 2, …, n} is a random sample from (1.3). We consider the following penalized linear quantile regression to produce a sparse estimator of β_τ:

$$Q(u,\mathbf{b})=n^{-1}\sum _{i=1}^n{\rho }_{\tau }(Y_i-u-{\mathbf{x}}_i^{\mathrm{T}}\mathbf{b})+\sum _{j=1}^{p_n}{p}_{\lambda }(\mid b_j\mid ),$$ (2.2)

where p_λ(·) is a penalty function with a regularization parameter λ. In this paper, we use the SCAD penalty (Fan and Li, 2001) and the MCP penalty (Zhang, 2010). The MCP function is defined as

$$p_{\lambda }\left(b\right)=\lambda (\mid b\mid -\frac{b^2}{2a\lambda })I(0\le \mid b\mid <a\lambda )+\frac{a{\lambda }^2}{2}I(\mid b\mid \ge a\lambda ),$$

where a > 1. The SCAD penalty is defined as follows,

$$p_{\lambda }\left(b\right)=\lambda \mid b\mid I(0\le \mid b\mid <\lambda )+\frac{a\lambda \mid b\mid -(b^2+{\lambda }^2)∕2}{a-1}I(\lambda \le \mid b\mid \le a\lambda )+\frac{(a+1){\lambda }^2}{2}I(\mid b\mid >a\lambda ),$$

where a = 3.7 is suggested by Fan and Li (2001). By minimizing the objective function Q(u, b), we obtain the estimators (${\widehat{u}}_{\tau },{\widehat{\beta }}_{\tau }$) at the τ -th quantile. In symbols, the resulting estimators are given by

$$({\widehat{u}}_{\tau },{\widehat{\beta }}_{\tau })=\underset{u,\mathbf{b}}{\mathrm{argmin}}\left\{Q(u,\mathbf{b})\right\}.$$ (2.3)

2.2 The Oracle Property

In this section we study the oracle property of the estimators obtained from the penalized linear quantile regression. Without loss of generality, we assume the first q_n components of x are active and the rest are inactive, where q_n(« p_n) is a small positive integer representing the sparsity level. In other words, A = {1, 2, …, q_n}. We define the oracle estimator at the population level by

$${ℒ}_{\tau }(u,{\mathbf{b}}_1)=E\left\{{\rho }_{\tau }(Y-u-{\mathbf{x}}_A^{\mathrm{T}}{\mathbf{b}}_1)\right\}\mathrm{and}(u_{\tau }^o,{\beta }_{\tau 1}^o)=\underset{u,{\mathbf{b}}_1}{\mathrm{argmin}}\left\{{ℒ}_{\tau }(u,{\mathbf{b}}_1)\right\},$$ (2.4)

where b₁ = (b₁, …, b_qn)^T ∈ ℝ^q_n. We further write ${\beta }_{\tau }^0={({\beta }_{\tau 1}^{o\mathrm{T}},0^{\mathrm{T}})}^{\mathrm{T}}$, where ${\beta }_{\tau 1}^o$ represents a q_n-dimensional vector of nonzero components associated with the active covariates and 0 denotes a (p_n − q_n)-dimensional vector of zeros. Accordingly, we define the oracle estimator ${\widehat{\beta }}_{\tau }^0={({\widehat{\beta }}_{\tau 1}^{o\mathrm{T}},0^{\mathrm{T}})}^{\mathrm{T}}$ at the sample level by

$$\begin{array}{cc}\hfill {ℒ}_{\tau n}(u,{\mathbf{b}}_1)& =n^{-1}\sum _{i=1}^n\left\{{\rho }_{\tau }(Y_i-u-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\mathbf{b}}_1)\right\}\mathrm{and}\hfill \\ \hfill ({\widehat{u}}_{\tau }^o,{\widehat{\beta }}_{\tau 1}^o)& =\underset{u,{\mathbf{b}}_1}{argmin}\left\{{ℒ}_{\tau n}(u,{\mathbf{b}}_1)\right\}.\hfill \end{array}$$ (2.5)

We assume the following regularity conditions to investigate the consistency of the oracle estimator defined in (2.5) and the oracle property of the resulting estimator obtained from the penalized linear quantile regression (2.3).

(C1) The covariates x satisfy the sub-exponential tail probability uniformly in p_n. That is, there exist positive constants t₀ and C such that

$$\underset{1\le k\le p_n}{\max }E\left\{\exp (t\mid X_k\mid )\right\}\le C<\infty ,\mathrm{for}0<t\le t_0.$$ (2.6)

(C2) There exist positive constants 0 < C₁ ≤ C₂ < ∞, such that

$$C_1\le {\lambda }_{\min }\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^{\mathrm{T}}\right)\right\}\le {\lambda }_{\max }\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^{\mathrm{T}}\right)\right\}\le C_2,$$

where λ_min and λ_max represent the smallest and largest eigenvalues, respectively. Assume further that {(x_i,A, Y_i), i = 1, …, n} are in general positions (Koenker, 2005, Section 2.2).

(C3) The probability density function of Y − x^T β_τ conditional on x, denoted by f (· | x), is uniformly bounded away from 0 and ∞ in the neighborhood around $u_{\tau }^o$.

(C4) The true model size q_n satisfies q_n = O(n^c₁) for 0 ≤ c₁ < 1/2.

(C5) For ${\beta }_{\tau 1}^o={({\beta }_{\tau ,1}^o,{\beta }_{\tau ,2}^o,\dots ,{\beta }_{\tau ,q_n}^o)}^{\mathrm{T}}$, there exist positive constants c₂ and C such that 2c₁ < c₂ ≤ 1 and

$$\underset{1\le j\le q_n}{\min }\mid {\beta }_{\tau ,j}^o\mid \ge C{n}^{-(1-c_2)∕2}.$$

Condition (C1) is concerned with the moments of the covariates, which follows immediately when the covariates are bounded, or when x has a multivariate normal distribution. This condition is widely assumed in high dimensional inference. See, for instance, Bickel and Levina (2008). Condition (C2) requires that the design matrix of the true model at the population level be well behaved. Condition (C3) is a common assumption on the conditional distribution function of (Y − x^T β_τ) conditional on x. Condition (C4) allows the sparsity size q_n can diverge as the sample size n goes to the infinity. Condition (C5) requires that the smallest true signal decay to zero at a slow rate.

Lemma 2 below states the consistency of the oracle estimators ${\widehat{u}}_{\tau }^o\mathrm{and}{\widehat{\beta }}_{\tau 1}^o$.

Lemma 2

Under Conditions (C1)-(C4), the oracle estimators ${\widehat{u}}_{\tau }^o\mathrm{and}{\widehat{\beta }}_{\tau 1}^o$ satisfy

$$\Vert {\widehat{\beta }}_{\tau 1}^o-{\beta }_{\tau 1}^o\Vert =O_p\left(\sqrt{q_n∕n}\right)and\Vert {\widehat{u}}_{\tau }^o-u_{\tau }^o\Vert =O_p(\sqrt{q_n}∕n).$$ (2.7)

Next we study the theoretical property of the oracle estimator ${\widehat{\beta }}_{\tau }^0={({\widehat{\beta }}_{\tau 1}^{o\mathrm{T}},0^{\mathrm{T}})}^{\mathrm{T}}$.

Theorem 1

(The Oracle Property) Suppose Conditions (C1)-(C5) hold, and log p_n = o(n^{min{c2−2θ,θ}}) with 0 < θ < (c₂ − c₁)/2 and λ = o {n^{−(1−c₂)/2}}. Let B_n(λ) be the set of local minima ${\widehat{\beta }}_{\tau }$ of the objective function Q(u, b) defined in (2.2) with the SCAD or the MCP penalty and the tuning parameter λ. The oracle estimator ${\widehat{\beta }}_{\tau }^0={({\widehat{\beta }}_{\tau 1}^{o\mathrm{T}},0^{\mathrm{T}})}^{\mathrm{T}}$ satisfies

$$\mathrm{Pr}\{{\widehat{\beta }}_{\tau }^o\in B_n\left(\lambda \right)\}\to 1,asn\to \infty .$$

Theorem 1 implies that the oracle estimator ${\widehat{\beta }}_{\tau }^o$ is a local minimizer of the objective function (2.2) with the probability approaching one as n → ∞. This result extends Theorem 2.4 of Wang, Wu and Li (2012) from the linear quantile regression model to model (1.3). The results in Lemmas 1, 2 and Theorem 1 imply that ${\widehat{\beta }}_{\tau }$ from the penalized linear quantile regression is a consistent estimator of the direction of β₀ in model (1.3). It can detect the non-zero components of β₀ and simultaneously estimate its direction. From a technical perspective, Wang, Wu and Li (2012) assumed all covariates are bounded uniformly while we relax their assumption to condition (C1) which only requires the distributions of the covariates have sub-exponential tails. In practice, the linear quantile regression estimator obtained with the LASSO penalty can serve as an initial value in our algorithm to minimize the objective function Q(u, b).

3. Robust SIS based on Distance Correlation

Next we propose a new robust feature screening procedure for model (1.3) through using distance correlation.

3.1 The Methodology

Theorem 1 indicates that the oracle property of the penalized quantile regression holds asymptotically true for log p_n = o(n^δ) for some δ > 0. This suffices for many problems from a theoretical perspective. From a practical perspective, however, when the covariate dimension is extremely large, even the penalized linear quantile regression suffers from at least two serious drawbacks: computational expediency and algorithmic stability (Fan, Samworth and Wu, 2009). When the penalized linear quantile regression fails to produce a reliable solution within a tolerant time, an independence screening procedure better precede the penalized linear quantile regression. This strategy helps to reduce the ultrahigh dimensionality down to a relatively moderate scale. To this end, we propose an independence screening procedure which excludes irrelevant covariates and hence reduces the computational complexity of subsequent penalized quantile regressions at all different quantile levels. In other words, the independence screening procedure is expected to have a sure screening property and to be independent of the quantile levels. In addition, it is expected to behave well when extreme values and/or outliers are present in the observed response values because subsequent quantile regression automatically has such a robustness property.

We first briefly review the definition of distance correlation (Szekely, Rizzo and Bakirov, 2007). The distance covariance between two random variables X and Y is defined by

$${\mathrm{dcov}}^2(X,Y)=S_1+S_2-2S_3,$$ (3.1)

where $S_1=E(\mid X-\tilde{X}\mid \mid Y-\tilde{Y}\mid ),S_2=E(\mid X-\tilde{X}\mid )E(\mid Y-\tilde{Y}\mid ),S_3=E\left\{E(\mid X-\tilde{X}\mid \mid X)E(\mid Y-\tilde{Y}\mid \mid Y)\right\},\mathrm{and}(\tilde{X},\tilde{Y})$ is an independent copy of (X, Y). Then, the distance covariance between X and Y is defined by

$$\mathrm{dcorr}(X,Y)=\frac{\mathrm{dcov}(X,Y)}{\sqrt{\mathrm{dcov}(X,Y)\mathrm{dcov}(Y,Y)}}.$$ (3.2)

Szekely, Rizzo and Bakirov (2007) pointed out that dcorr(X, Y) = 0 if and only if X and Y are independent and dcorr(X, Y) is strictly increasing in the absolute value of the Pearson correlation between X and Y. Motivated by these properties, Li, Zhong and Zhu (2012) proposed an sure independence screening to rank all predictors using their distance correlations with the response variable, called as DC-SIS, and proved its sure screening property for ultrahigh dimensional data.

Next, we denote by X_k the kth predictor with k = 1, …, p_n and propose to quantify the importance of X_k through its distance correlation with the marginal distribution function of Y, denoted by F (Y). That is,

$${\omega }_k=\mathrm{dcorr}\{X_k,F\left(Y\right)\},$$ (3.3)

where F (y) = E {1(Y ≤ y)} and 1(·) denotes an indicator function. This is a modification of the marginal utility in Li, Zhong and Zhu (2012) in that here we use F (Y) instead of Y. The marginal utility defined in (3.3) has several distinctive and appealing advantages comparing with the existing measurements.

1. It is obvious that dcorr{X_k, F (Y)} = 0 if and only if X_k and Y are independent. Following similar arguments in Li, Zhong and Zhu (2012), we can see that this new independence screening procedure based on (3.3) is model-free and hence is applicable to model (1.3) and its sparse model structure (1.4).

2. Since F (Y) is a bounded function for all types of Y, we can naturally expect that the independence screening procedure using (3.3) has a reliable performance when the response is the heavy-tailed and when extreme values are present in the response values.

3. If one suspects that the covariates also contain some extreme values, then one can use the utility dcorr{F_k (X_k), F (Y)} to rank the importance of X_k, where F_k (x) = E {1(X_k ≤ x)}.

In the sequel we introduce how to implement the marginal utility (3.3) in the screening procedure. Let {(x_i, Y_i), i = 1, · · · , n} be a random sample from the population (x, Y). We first estimate the distance covariance between X_k and F (Y) through the moment estimation method,

$${\widehat{\mathrm{dcov}}}^2\{X_k,F\left(Y\right)\}={\widehat{S}}_{k,1}+{\widehat{S}}_{k,2}-2{\widehat{S}}_{k,3},$$ (3.4)

where

$$\begin{array}{cc}\hfill {\widehat{S}}_{k,1}& =\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid X_{ik}-X_{jk}\mid \mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid ,\hfill \\ \hfill {\widehat{S}}_{k,2}& =\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid X_{ik}-X_{jk}\mid \frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid ,\mathrm{and}\hfill \\ \hfill {\widehat{S}}_{k,3}& =\frac{1}{n^3}\sum _{i=1}^n\sum _{j=1}^n\sum _{l=1}^n\mid X_{ik}-X_{lk}\mid \mid F_n\left(Y_j\right)-F_n\left(Y_l\right)\mid .\hfill \end{array}$$

are the corresponding estimators of S_k,1, S_k,2, S_k,3, and $F_n\left(y\right)=n^{-1}{\Sigma }_{i=1}^{n}1(Y_i\le y)$. We estimate ω_k with

$${\widehat{\omega }}_k=\widehat{\mathrm{dcorr}}\{X_k,F\left(Y\right)\}=\frac{\widehat{\mathrm{dcov}}(X_k,F\left(Y\right))}{\sqrt{\widehat{\mathrm{dcov}}(X_k,X_k)\widehat{\mathrm{dcov}}(F\left(Y\right),F\left(Y\right))}}.$$ (3.5)

Our proposed independence screening procedure retains the covariates with the ${\widehat{\omega }}_k$ values larger than a user-specified threshold. Denote

$$\widehat{A}=\{k:{\widehat{\omega }}_k\ge c{n}^{-\kappa },\mathrm{for}1\le k\le p_n\}$$

for some pre-specified thresholds c > 0 and 0 ≤ κ < 1/2. The constants c and κ control the signal strength and will be defined in Condition (C6) below. We refer to this approach as the distance correlation based robust independence screening procedure (DC-RoSIS).

3.2 Sure Screening Property

We first state the consistency of ${\widehat{\omega }}_k$ defined in (3.5), which paves the road for proving the sure screening property of the DC-RoSIS procedure.

Theorem 2

Under Condition (C1), for any 0 < γ < 1/2−κ, there exist positive constants c₁ and c₂ such that

$$Pr(\underset{1\le k\le p}{\max }\mid {\widehat{\omega }}_k-{\omega }_k\mid \ge c{n}^{-\kappa })\le O\left(p[\exp \{-c_1{n}^{1-2(\kappa +\gamma )}\}+n\exp (-c_2{n}^{\gamma })]\right)$$ (3.6)

We remark here that to derive the consistency of the estimated marginal utility, we do not assume any moment condition on the response. To prove the sure screening property, we further assume the following condition.

(C6) The marginal utility defined in (3.3) satisfies $\underset{k\in A}{\min }{\omega }_k\ge 2c{n}^{-\kappa }$, for some constants c > 0 and 0 ≤ κ < 1/2.

Condition (C6) allows the minimal signal of the active covariates converges to zero as the sample size diverges, yet it requires the minimum signal of active covariates be not too small. The sure screening property is stated below.

Theorem 3

(Sure Screening Property) Under condition (C6) and the conditions in Theorem 2, it follows that

$$Pr(A\subseteq \widehat{A})\ge 1-O\left(s_n[\exp \{-c_1{n}^{1-2(\kappa +\gamma )}\}+n\exp (-c_2{n}^{\gamma })]\right),$$ (3.7)

where s_n is the cardinality of A. Thus, $Pr(A\subseteq \widehat{A})\to 1$ as n → ∞.

4. Numerical Studies

In this section, we first conduct simulations to demonstrate the finite sample performance of our proposals. We further illustrate the proposed methodology through an empirical analysis of a real data example.

4.1 Simulations

In Example 1 we compare the performance of several independence screening procedures, and in Example 2 we assess the performance of penalized linear quantile regressions with different penalties and at different quantiles. Throughout the simulations we generate x = (X₁, X₂, · · · , X_p)^T from N (0, Σ), where $\Sigma ={\left({\sigma }_{ij}\right)}_{p\times p}\mathrm{with}{\sigma }_{ij}={0.5}^{\mid i-j\mid }$. The dimensionality p = 1, 000 and the sample size n = 200.

Example 1

This example is designed to compare the finite sample performance of our proposal DC-RoSIS with existing procedures including SIS (Fan and Lv, 2008), SIRS (Zhu, Li, Li and Zhu, 2011), RRCS (Li, Peng, Zhang and Zhu, 2012) and DC-SIS (Li, Zhong and Zhu, 2012). We repeat each experiment 500 times and evaluate the performance with the following three criteria.

1. S: The minimum model size to include all active covariates. We summarize the median of S with its associated robust estimate of the standard deviation (RSD = IQR/1.34). A smaller S value indicates a better performance.

2. P_sj : The empirical probability that the active covariate X_j is selected for a given model size d. We set d = 2[n/ log n] throughout.

3. P_a: The empirical probability that all active covariates are selected for the given model size d = 2[n/ log n]. If the sure screening property holds true, both P_sj and P_a values are close to one when the estimated model size d is reasonably large.

We consider the following four models:

Model (1): H(Y) = x^T β + ε,

Model (2): Y = exp(2 − x^T β/2) + (2 − x^T β/2)² + exp(x^T β/2)ε,

Model (3): Y = {1 + exp (−3x^T β)}⁻¹ ε,

Model (4): Y = β₁X₁ + β₂X₂ + ${\beta }_7{X}_7^2$ + ε,

where β = (3, 1.5, 0, 0, 0, 0, 2, 0, …, 0)^T. In the above four models, only X₁, X₂ and X₇ are truly important. The random error ε is independently generated from either standard normal or standard Cauchy distribution. In model (1), H(Y) = {|Y |^λsgn(Y) − 1}/λ is the Box-Cox transformation. This model was used in Li, Peng, Zhang and Zhu (2012). We set λ = 1 and λ = 0.25. Both models (2) and (3) are heteroscedastic single-index models. The single index (x^T β) contributes both the conditional mean and variance of the response in model (2), and are totally irrelevant to the mean regression function in model (3). The active covariate X₇ in model (4) is quadratically related to the response. Though it is not a special case of model (1.3) or (1.4), we use it here to show that our independence screening procedure works quite well for a variety of regressions even when the model assumptions are violated.

The results are summarized in Table 4.1. It can be seen that SIS does not perform well when ε follows Cauchy distribution. Even when ε follows standard normal distribution, SIS still fails to behave well in nonlinear models (3) and (4). SIRS performs very well for all single-index models. However, SIRS fails to identify X₇ as an important covariate in model (4) because it is not capable of detecting symmetric patterns. The performance of RRCS is generally favorable for models (1) and (2). However, RRCS hardly detects the active covariates that are only relevant to the conditional variance of the response in model (3) or X₇ that exhibits symmetric patterns with Y in model (4). DC-RoSIS and DC-SIS have similar performances when ε follows standard normal distribution. When ε follows Cauchy distribution, DC-RoSIS significantly improves DC-SIS. For example, in model (1) with λ = 0.25, DC-SIS fails to detect the true relationship between two random variables when very extreme values are present.

Table 4.1.

Performance comparison among different independence screening methods for four regression models with two different random errors.

		ε ~ N(0, 1)					ε ~ Cauchy Distribution
	Method	S	P s1	P s2	P s7	P a	Size	P s1	P s2	P s7	P a
	SIS	3.0(0.0)	1.00	1.00	1.00	1.00	220.0(483.4)	0.67	0.62	0.49	0.39
	DC-SIS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.7)	0.98	0.98	0.95	0.95
Model (1) (λ = 1)	SIRS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	RRCS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	DC-RoSIS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	SIS	3.0(0.7)	1.00	1.00	0.99	0.99	794.5(210.5)	0.10	0.07	0.09	0.00
	DC-SIS	3.0(0.0)	1.00	1.00	1.00	1.00	702.5(246.6)	0.17	0.14	0.13	0.05
Model (1) (λ = 0.25)	SIRS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	RRCS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	DC-RoSIS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.0)	1.00	1.00	1.00	1.00
	SIS	5.0(14.2)	0.99	0.99	0.90	0.90	29.0(122.4)	0.89	0.83	0.69	0.63
	DC-SIS	3.0(0.7)	1.00	1.00	0.99	0.99	3.0(2.9)	0.99	0.99	0.94	0.94
Model (2)	SIRS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.7)	1.00	1.00	1.00	1.00
	RRCS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.7)	1.00	1.00	1.00	1.00
	DC-RoSIS	3.0(0.0)	1.00	1.00	1.00	1.00	3.0(0.7)	1.00	1.00	1.00	1.00
	SIS	786.5(217.5)	0.08	0.06	0.07	0.00	791.0(213.2)	0.06	0.07	0.07	0.00
	DC-SIS	4.0(4.5)	1.00	1.00	0.97	0.97	70.0(130.8)	0.92	0.83	0.57	0.52
Model (3)	SIRS	7.0(8.2)	1.00	1.00	0.99	0.99	8.0(8.9)	1.00	1.00	0.98	0.98
	RRCS	796.0(222.8)	0.10	0.10	0.09	0.00	782.0(253.3)	0.12	0.09	0.06	0.00
	DC-RoSIS	8.0(9.7)	1.00	1.00	0.96	0.96	9.0(11.9)	1.00	1.00	0.96	0.96
	SIS	270.5(400.6)	1.00	1.00	0.25	0.25	594.0(358.0)	0.72	0.62	0.07	0.05
	DC-SIS	4.0(0.7)	1.00	1.00	1.00	1.00	8.0(18.7)	0.99	0.98	0.86	0.86
Model (4)	SIRS	427.0(419.6)	1.00	1.00	0.11	0.11	493.5(387.7)	1.00	1.00	0.09	0.09
	RRCS	434.0(391.1)	1.00	1.00	0.13	0.13	477.5(394.0)	1.00	1.00	0.10	0.10
	DC-RoSIS	4.0(1.5)	1.00	1.00	1.00	1.00	6.0(5.2)	1.00	1.00	0.99	0.99

Example 2

In this example, we examine the finite sample performance of the penalized linear quantile regression with different penalties including LASSO (Tibshirani, 1996), SCAD (Fan and Li, 2001) and MCP (Zhang, 2010). We first utilize our proposed screening procedure to select d = 2[n/ log(n)] top ranked covariates and then apply the penalized linear quantile regression to estimate the direction of β. For the conditional quantile regression, we consider three different quantiles τ = 0.25, 0.50 and 0.75, which correspond to the 1st quartile, the median and 3rd quartile of the response conditioning on the covariates. Following Wang, Wu and Li (2012), an additional independent data set of size 10n is generated to select the tuning parameter λ by minimizing the estimated prediction error based on the quantile check loss function.

We denote the final estimator by ${\widehat{\beta }}_{\tau }={({\widehat{\beta }}_1,{\widehat{\beta }}_2,\dots ,{\widehat{\beta }}_p)}^{\mathrm{T}}$. Note that the coefficients of covariates removed by the screening procedure are directly set to be zero in the final estimator. Based on 100 repetitions, we evaluate the performance in terms of the following criteria.

Size: The average number of non-zero estimated regression coefficients ${\widehat{\beta }}_j\ne 0\mathrm{for}1\le j\le p$;

C: The average number of truly non-zero coefficients correctly estimated to be non-zero;

IC: The average number of truly zero coefficients incorrectly estimated to be non-zero;

AE: The average of absolute estimation error of ${\widehat{\beta }}_{\tau }$, which is defined by ${\Sigma }_{j=1}^p\mid {\widehat{\beta }}_j\mathrm{sign}\left({\widehat{\beta }}_{j,1}\right)∕\Vert {\widehat{\beta }}_{\tau }\Vert -{\beta }_{0j}\mathrm{sign}\left({\beta }_{0j,1}\right)∕\Vert {\beta }_0\Vert \mid$.

We only report the results for model (2) in Example 1, which is a heteroscedastic single-index model, as the results for other models lead to similar conclusion. The simulation results are charted in Table 4.2. In each column, the value represents the mean of 100 replicates with its sample standard deviation in the parentheses. For two different random errors and different quantiles, the first three columns demonstrate that the LASSO is relatively conservative and tends to select larger models while the SCAD and the MCP are consistent to select the true model. The relatively small values in the column labeled “AE” shows that the proposed penalized linear quantile regression procedure can produce consistent estimators and support the theoretical findings in Theorem 1. In conclusion, the satisfactory simulation results demonstrate that the proposed robust two-stage procedure is indeed robust to the presence of heteroscedasticity and extreme values in the response.

Table 4.2.

Simulation Results for Penalized Linear Quantile Regression at difference quantile levels (25%; 50% and 75%) and with difference penalties (LASSO, SCAD, MCP).

ε ~ N(0, 1)
Method	Size	C	IC	AE
LASSO(τ = 0.25)	18.16(6.28)	3.00(0.00)	15.16(6.28)	0.47(0.22)
LASSO(τ = 0.50)	18.14(6.33)	3.00(0.00)	15.14(6.33)	0.93(0.36)
LASSO(τ = 0.75)	13.97(6.16)	2.96(0.20)	11.01(6.15)	1.33(0.57)
SCAD(τ = 0.25)	3.46(0.86)	3.00(0.00)	0.46(0.86)	0.11(0.07)
SCAD(τ = 0.50)	3.68(1.58)	2.96(0.20)	0.72(1.56)	0.28(0.23)
SCAD(τ = 0.75)	3.47(1.58)	2.68(0.55)	0.79(1.52)	0.62(0.36)
MCP(τ = 0.25)	3.36(0.73)	3.00(0.00)	0.36(0.73)	0.11(0.07)
MCP(τ = 0.50)	3.53(1.23)	2.96(0.20)	0.57(1.21)	0.28(0.20)
MCP(τ = 0.75)	3.50(1.68)	2.68(0.55)	0.82(1.62)	0.63(0.36)
ε ~ Cauchy Distribution
Method	Size	C	IC	AE
LASSO(τ = 0.25)	23.75(6.63)	3.00(0.00)	20.75(6.63)	0.66(0.25)
LASSO(τ = 0.50)	19.29(7.66)	3.00(0.00)	16.29(7.66)	0.97(0.42)
LASSO(τ = 0.75)	14.01(6.89)	2.88(0.33)	11.13(6.82)	1.34(0.63)
SCAD(τ = 0.25)	3.56(1.19)	3.00(0.00)	0.56(1.19)	0.12(0.08)
SCAD(τ = 0.50)	3.66(1.36)	2.94(0.24)	0.72(1.36)	0.27(0.22)
SCAD(τ = 0.75)	3.33(1.91)	2.60(0.61)	0.73(1.84)	0.63(0.38)
MCP(τ = 0.25)	3.43(0.83)	3.00(0.00)	0.43(0.83)	0.11(0.07)
MCP(τ = 0.50)	3.70(1.67)	2.94(0.24)	0.76(1.66)	0.28(0.24)
MCP(τ = 0.75)	3.57(2.23)	2.64(0.53)	0.93(2.18)	0.65(0.42)

4.2 An Application

In this section we conduct an empirical study of the Cardiomyopathy microarray dataset. This dataset was analyzed by Segal, Dahlquist and Conklin (2003), Hall and Miller (2009) and Li, Zhong and Zhu (2012). The response variable is the genetic overexpression level of a G protein-coupled receptor (Ro1) in mice, which can sense molecules outside the cell and activate inside signal transduction pathways and cellular responses. The covariates are 6, 319 genetic expression levels. Only 30 specimens are observed. The main goal of this analysis is to determine the most influential genes for the response.

We display both the boxplot and the histogram of Y in Figure 4.1. Both indicate that the response distribution may be heavy-tailed and contains outliers. We first implement independence screening procedures to reduce the covariate dimension to the size of 2[n/ log n] = 16. The performances of SIS and DC-SIS are similar to that of DC-RoSIS in this real data analysis. Thus, we only present results of DC-RoSIS with regularized quantile regression with different penalties in this example. The DC-RoSIS selects the two genes, labeled Msa.2877.0 and Msa.2134.0, in the top, which are same as the DC-SIS (Li, Zhong and Zhu, 2012). The gene, Msa.1166.0, identified by generalized correlation ranking (Hall and Miller, 2009) is also ranked in the top 10 by our screening procedure.

Figure 4.1.

Exploratory Data Analysis: Histogram and Boxplot of Ro1.

We further apply our proposed penalized linear quantile regression to the reduced model to estimate the direction of the index parameter and to simultaneously select important variables at different quantiles of the response. We choose the quantile levels τ = 0.25, 0.50 and 0.75, and three different penalties, LASSO, SCAD and MCP. We use BIC to select the tuning parameters for each method. With the estimated single index, denoted (${\mathbf{x}}^{\mathrm{T}}{\widehat{\beta }}_{\tau }$), we apply the cubic splines to estimate the quantile functions ${\widehat{q}}_{\tau }(\cdot )$ of model (1.3), or equivalently, model (1.4). Figure 4.2 depicts the estimated curves of ${\widehat{q}}_{\tau }\left({\mathbf{x}}^{\mathrm{T}}{\widehat{\beta }}_{\tau }\right)$ at different quantiles and for different penalties, which demonstrate the computational effectiveness of our proposals.

Figure 4.2.

The estimated curves of ${\widehat{q}}_{\tau }\left({\mathbf{x}}^{\mathrm{T}}{\widehat{\beta }}_{\tau }\right)$ (the vertical axis) versus (${\mathbf{x}}^{\mathrm{T}}{\widehat{\beta }}_{\tau }$) (the horizontal axis) at different quantiles for different penalties. From left to right, τ = 0.25, 0.50 and 075; From up to down, LASSO, SCAD and MCP.

To compare the finite sample performances of different methods with different quantiles, we report the number of nonzero coefficients selected by each method, denoted by “Size” in Table 4.3. In addition, to evaluate the goodness of fit for each model, we follow the idea of R² for the linear model and define the quantile-adjusted R² (“Q-R²”) as follows,

$$\mathrm{Q}-R^2=[1-\frac{{\Sigma }_{i=1}^n{\rho }_{\tau }^2\{Y_i-{\widehat{q}}_{\tau }\left(X_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }\right)\}}{{\Sigma }_{i=1}^n{\rho }_{\tau }^2(Y_i-{\widehat{Y}}_{\tau })}]\times 100\%,$$ (4.1)

where ρ_τ (·) is the τ th quantile check loss function, ${\widehat{q}}_{\tau }(\cdot )$ is the cubic-spline estimate of $q_{\tau }(\cdot ),{\widehat{q}}_{\tau }\left({\mathbf{x}}^{\mathrm{T}}{\widehat{\beta }}_{\tau }\right)$ is the τ -th quantile function of Y_i, and ${\widehat{Y}}_{\tau }$ is the sample τ th quantile of Y. The larger Q-R² is, the better the model fit is. For example, for τ = 0.75, SCAD selected 3 covariates, which can explain 93.0% variance of the response in terms of the defined Q-R². As a benchmark, we also report the model with all 16 selected genes by our screening procedure, denoted by SCREEN in Table 4.3. In addition, we conduct 100 random partitions to examine the prediction performance. For each partition, we randomly select 90% of the data (27 observations) as the training set and the rest 10% (3 observations) as the test set. The average of the model sizes selected by each method with its standard error across 100 partitions in the parenthesis are reported in the third column (“Ave Size”) of Table 4.3. In this table, we also report the average of quantile-adjusted R² for each method on the training set and its associated standard error, denoted by “Ave Q-R²”. The column labeled by “PE” denotes the median of prediction errors based on the quantile check loss function and its associated robust estimate of the standard deviation (i.e. interquartile range/1.34) in the parentheses. In conclusion, the penalized linear quantile regression improves both the model interpretability in terms of the model size and the model predictability in terms of the prediction errors.

Table 4.3.

Empirical analysis of Cardiomyopathy microarray dataset.

	All Data		Partitioned Data
Method	Size	Q-R2	Ave Size	Ave Q-R2	PE
SCREEN(τ = 0.25)	16	97.6	16.00(0.00)	94.41(3.66)	0.48(0.21)
SCREEN(τ = 0.50)	16	93.1	16.00(0.00)	92.96(2.67)	0.67(0.27)
SCREEN(τ = 0.75)	16	94.4	16.00(0.00)	94.99(1.31)	0.59(0.33)
LASSO(τ = 0.25)	12	87.8	8.56(1.55)	78.59(10.75)	0.44(0.20)
LASSO(τ = 0.50)	8	89.1	7.21(1.54)	90.71(3.49)	0.55(0.17)
LASSO(τ = 0.75)	5	91.2	5.64(1.25)	94.54(1.71)	0.44(0.22)
SCAD(τ = 0.25)	10	96.9	8.29(2.81)	90.88(6.46)	0.44(0.18)
SCAD(τ = 0.50)	6	92.3	6.52(3.13)	92.33(3.24)	0.58(0.20)
SCAD(τ = 0.75)	3	93.0	3.82(1.46)	94.04(1.45)	0.50(0.25)
MCP(τ = 0.25)	10	96.9	8.69(2.64)	91.71(5.92)	0.43(0.14)
MCP(τ = 0.50)	5	89.3	6.91(2.81)	92.58(3.09)	0.55(0.26)
MCP(τ = 0.75)	4	92.8	4.13(1.64)	94.15(1.46)	0.51(0.28)

5. Discussions

In this paper, we first study the regularized quantile regression for ultrahigh dimensional single-index models, in which the conditional distribution of the response depends on the covariates via a single-index structure. The consistency and the oracle property for the penalized linear quantile regression estimator have been established under the sparsity condition. Then, we propose a robust independence screening based on the distance correlation between the distribution function of the response variable and each covariate, called as DCRoSIS. The new DC-RoSIS enjoys the sure screening property under even milder conditions than the existing alternative methods. It can be applied before the regularized quantile regression to reduce the covariate dimension from ultrahigh dimensionality to a moderate scale. The numerical studies show that the proposed methodology has the excellent finite-sample performance compared with other methods.

The proposed method has reliable performance when the distribution of the response variable is heavily tailed or response realizations contain extreme values. An interesting point raised by a referee is concerned with the performance of proposed procedure in the presence of heavy-tail predictors or extreme outliers contained in the predictors. In this case, Condition (C1) will be violated and the proposed method may fail. However, we may simply use F_k (X_k), the distribution function of X_k, in place of X_k in the proposed screening procedure. This replacement will help us remove condition (C1) and achieve the robustness feature in the x-direction. Please see Appendix A in the Supplement for more details. However, implementing penalized linear quantile regression when x contains outliers is not straightforward. How to remove condition (C1) in the penalized linear quantile regression would be an interesting topic for future research.

Theorem 1 implies that the oracle estimator is a local minimizer of the objective function (2.2) with the probability approaching one as n → ∞. Thus, the proposed method can detect the non-zero components of the true coefficient and simultaneously estimate its direction. For the statistical inference purpose, if one may be interested in the asymptotical distribution of the regularized quantile estimator, we can adapt the idea of Theorem 2 in Wu and Liu (2009). They proved that the SCAD and Adaptive-LASSO penalized linear quantile estimator is asymptotically normal if the number of important covariates is a fixed number. If the number of important covariates diverges to infinity, it becomes much more challenging to derive the asymptotic normality. But it is an interesting and potential research direction.

Appendix. Proof of Theorems

Appendix A: Proof of Lemma 2

We require the following lemma to prove Lemma 2.

Lemma 3

According to (2.4), $E\{I(Y-{\mathbf{x}}_A^T{\beta }_{\tau 1}^o\le u_{\tau }^o)\mid {\mathbf{x}}_A\}=\tau$. That is, the oracle estimator $u_{\tau }^o$ of u is the τ th quantile of $Y-{\mathbf{x}}_A^{\tau }{\beta }_{\tau 1}^o$ conditional on x_A.

Proof of Lemma 3

Let ξ_τ be the τ th quantile of $Y-{\mathbf{x}}_A^T{\beta }_{\tau 1}^o$ conditional on x_A. By definition, we have that $E\{I(Y-{\mathbf{x}}_A^T{\beta }_{\tau 1}^o\le {\xi }_{\tau })\mid {\mathbf{x}}_A\}=\tau$. It suffices to show ${ℒ}_{\tau }({\xi }_{\tau },{\beta }_{\tau 1}^o)\le {ℒ}_{\tau }(u,{\beta }_{\tau 1}^o)$ holds for any u. To be specific,

$$\begin{array}{cc}\hfill & {ℒ}_{\tau }(u,{\beta }_{\tau 1}^o)-{ℒ}_{\tau }({\xi }_{\tau },{\beta }_{\tau 1}^o)=E\left\{{\rho }_{\tau }(Y-u-{\mathbf{x}}_A^{\tau }{\beta }_{\tau 1}^o)\right\}-E\left\{{\rho }_{\tau }(Y-{\xi }_{\tau }-{\mathbf{x}}_A^{\tau }{\beta }_{\tau 1}^o)\right\}\hfill \\ \hfill =& E\left[(u-{\xi }_{\tau })\{I(Y-{\xi }_{\tau }-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)-\tau \}\right]\hfill \\ \hfill & +E\left[{\int }_0^{u-{\xi }_{\tau }}\{I(Y-{\xi }_{\tau }-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o\le t)-I(Y-{\xi }_{\tau }-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}dt\right]\ge 0,\hfill \end{array}$$

where the second equality follows from Knight (1998). In the second equality, the first term is zero and the second is nonnegative. Thus ${\xi }_{\tau }=u_{\tau }^o$ and the desired conclusion follows.

Proof of Lemma 2

To prove Lemma 2, we borrow the idea of He and Shao (2000) on M-estimation. It suffices to show that for any fixe η > 0, there exists two constants Δ₁ and Δ₂ such that for all sufficiently large n,

$$\mathrm{Pr}\{\underset{\underset{\mid u\mid ={\Delta }_2}{\overset{\Vert \gamma \Vert ={\Delta }_1}{}}}{\inf }{ℒ}_{\tau n}(u_{\tau }^o+n^{-1∕2}{q}_n^{1∕2}u,{\beta }_{\tau 1}^o+n^{-1∕2}{q}_n^{1∕2}\gamma )>{ℒ}_{\tau n}(u_{\tau }^o,{\beta }_{\tau 1}^o)\}\ge 1-\eta .$$

We define that

$$\begin{array}{cc}\hfill & G_n(u,\gamma )≕n{q}_n^{-1}\{{ℒ}_{\tau n}(u_{\tau }^o+n^{-1∕2}{q}_n^{1∕2}u,{\beta }_{\tau 1}^o+n^{-1∕2}{q}_n^{1∕2}\gamma )-{ℒ}_{\tau n}(u_{\tau }^o,{\beta }_{\tau 1}^o)\}\hfill \\ \hfill =& q_n^{-1}\sum _{i=1}^n{n}^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le u_{\tau }^0)-\tau \}\hfill \\ \hfill & +q_n^{-1}\sum _{i=1}^n{\int }_0^{n^{-1∕2q_n^{1∕2}(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )}}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le u_{\tau }^o+s)-I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le u_{\tau }^o)\}ds\hfill \\ \hfill ≕& I_{n1}+I_{n2},\hfill \end{array}$$

where the second equality follows from Knight (1998)’s identity.

Note that $E\{I(Y-{\mathbf{x}}_A^T{\beta }_{\tau 1}^o\le u_{\tau }^o)\mid {\mathbf{x}}_A\}=\tau$ by Lemma 3 and hence E(I_{n1) = 0.}

$$\begin{array}{cc}\hfill \mathrm{var}\left(I_{n1}\right)& =E\left\{\mathrm{var}(I_{n1}\mid {\mathbf{x}}_A)\right\}=\mathrm{var}\left\{E(I_{n1}\mid {\mathbf{x}}_A)\right\}\hfill \\ \hfill & =\tau (1-\tau )q_n^{-1}E\left\{n^{-1}\sum _{i=1}^n{(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )}^2\right\}\hfill \\ \hfill & \le 2\tau (1-\tau )q_n^{-1}[u^2+{\lambda }_{\max }\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^{\mathrm{T}}\right)\right\}\Vert \gamma {\Vert }^2]\le C{q}_n^{-1}({\Delta }_1^2+{\Delta }_2^2),\hfill \end{array}$$

where the last inequality follows by Condition (C2). Thus, $I_{n1}=O_p\left({q_n}^{-1∕2}\right){({\Delta }_1^2+{\Delta }_2^2)}^{1∕2}$.

Next we evaluate I_n2. Denote by F (· | x_A) and f (· | x_A) the conditional distribution and density of ($(Y-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o)$) given x_A, respectively

$$\begin{array}{cc}\hfill E\left(I_{n2}\right)& =q_n^{-1}E\left[\sum _{i=1}^n{\int }_0^{n^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )}\{F(u_{\tau }^o+s\mid {\mathbf{x}}_{i,A})-F(u_{\tau }^o\mid {\mathbf{x}}_{i,A})\}ds\right]\hfill \\ \hfill & =q_n^{-1}E\left[\sum _{i=1}^n{\int }_0^{n^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )}f(u_{\tau }^o+s^{\prime }\mid {\mathbf{x}}_{i,A})sds\right]\hfill \\ \hfill & \ge C{q}_n^{-1}E\left[\sum _{i=1}^n{\left\{n^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_{i,A}^{\mathrm{T}}\gamma )\right\}}^2\right]\hfill \\ \hfill & =CE{(u+{\mathbf{x}}_A^{\mathrm{T}}\gamma )}^2\ge C[1+{\lambda }_{\min }\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^{\mathrm{T}}\right)\right\}](u^2+\Vert \gamma {\Vert }^2)\ge C({\Delta }_1^2+{\Delta }_2^2),\hfill \end{array}$$

where the first inequality follows by Condition (C3) and the last inequality follows by Condition (C2). Therefore, $E\left(I_{n2}\right)=O\left(1\right)({\Delta }_1^2+{\Delta }_2^2)$. Next we consider the variance of I_n2,

$$\begin{array}{cc}\hfill & \mathrm{var}\left(I_{n2}\right)\hfill \\ \hfill \le & n{q}_n^{-2}E{\left[{\int }_0^{n^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_A^{\mathrm{T}}\gamma )}\{I(Y-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o\le u_{\tau }^o+s)-I(Y-{\mathbf{x}}_A^{\mathrm{T}}{\beta }_{\tau 1}^o\le u_{\tau }^o)\}ds\right]}^2\hfill \\ \hfill \le & n{q}_n^{-2}E{\left\{n^{-1∕2}{q}_n^{1∕2}(u+{\mathbf{x}}_A^{\mathrm{T}}\gamma )\right\}}^2\le q_n^{-1}[1+{\lambda }_{\min }\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^{\mathrm{T}}\right)\right\}](u^2+\Vert \gamma {\Vert }^2)\hfill \\ \hfill \le & O\left(q_n^{-1}({\Delta }_1^2+{\Delta }_2^2)\right),\hfill \end{array}$$

which converges to zero as n → ∞ because q_n = O(n^c1). This indicates that |I_n2 −E(I_n2)| = o_p(1) by Chebyshev’s inequality. Since I_n2 is always nonnegative,

$$I_{n2}=E\left(I_{n2}\right)+o_p\left(1\right)\ge C({\Delta }_1^2+{\Delta }_2^2)+o_p\left(1\right).$$

For sufficiently large Δ₁ and Δ₂, I_n2 dominates I_n1 asymptotically as n → ∞. Therefore, for any fixed η > 0, there exists two constants Δ₁ and Δ₂ such that for all sufficiently large n, we have G_n(u, γ) > 0 with probability at least 1 − η.

Appendix B: Proof of Theorem 1

We follow the idea of the proof of Theorem 2.4 in Wang, Wu and Li (2012) to demonstrate the technical proof of Theorem 1. Note that their moment conditions on x are different. With slightly notational abuse, we write ${\mathbf{x}}_A={(1,{\mathbf{x}}_A)}^{\mathrm{T}},{\beta }_{\tau }^o={(u_{\tau }^o,{\beta }_{\tau }^{o\mathrm{T}})}^{\mathrm{T}}$ defined in (2.4), ${\widehat{\beta }}_{\tau }={({\widehat{u}}_{\tau },{\widehat{\beta }}_{\tau }^{\mathrm{T}})}^{\mathrm{T}}\mathrm{and}{\widehat{\beta }}_{\tau }^o={({\widehat{u}}_{\tau }^o,{\beta }_{\tau }^{o\mathrm{T}})}^{\mathrm{T}},\mathrm{where}{\widehat{\beta }}_{\tau }$ denotes the penalized linear quantile estimator defined in (2.3) and ${\widehat{\beta }}_{\tau }^o={({\widehat{\beta }}_{\tau 1}^{o\mathrm{T}},0^{\mathrm{T}})}^{\mathrm{T}}$ is the oracle estimator defined in (2.5). Accordingly, we write ${\beta }_{\tau 1}^o={(u_{\tau }^o,{\beta }_{\tau 1}^{o\mathrm{T}})}^{\mathrm{T}}\mathrm{and}{\widehat{\beta }}_{\tau 1}^o={({\widehat{u}}_{\tau }^o,{\widehat{\beta }}_{\tau 1}^{o\mathrm{T}})}^{\mathrm{T}}$.

We first write the objective function (2.2) of the penalized linear quantile regression as the difference of two convex functions in β. Here, we only consider the proof for the SCAD penalty, and the proof for the MCP penalty can be achieved by the similar arguments. To be precise, Q(β) = g(β) − h(β), where $g\left(\beta \right)=n^{-1}{\Sigma }_{i=1}^n{\rho }_{\tau }(Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta )+\lambda {\Sigma }_{j=1}^{p_n}\mid {\beta }_j\mid ,\mathrm{and}h\left(\beta \right)={\Sigma }_{j=1}^{p_n}{H}_{\lambda }\left({\beta }_j\right)$, with

$$H_{\lambda }\left({\beta }_j\right)=\{\begin{array}{cc}0,\hfill & 0\le \mid {\beta }_j\mid <\lambda ;\hfill \\ ({\beta }_j^2-2\lambda \mid {\beta }_j\mid +{\lambda }^2)∕\left\{2(a-1)\right\},\hfill & \lambda \le \mid {\beta }_j\mid \le a\lambda ;\hfill \\ \lambda \mid {\beta }_j\mid -(a+1){\lambda }^2∕2,\hfill & \mid {\beta }_j\mid >a\lambda .\hfill \end{array}\phantom{\}}$$

Thus, the subdifferential of h(β) at any β is

$$\partial h\left(\beta \right)=\{\mu ={({\mu }_0,{\mu }_1,\dots ,{\mu }_{p_n})}^{\mathrm{T}}\in {\mathbb{ℝ}}^{p_n+1}:{\mu }_0=0,{\mu }_j=\frac{\partial h\left(\beta \right)}{\partial {\beta }_j},j=1,2,\dots ,p_n\}.$$

The subdifferential of g(β) at any β is

$$\begin{array}{cc}\hfill {\partial }_g\left(\beta \right)=& \{\xi ={({\xi }_0,{\xi }_1,\dots ,{\xi }_{p_n})}^{\mathrm{T}}\in {\mathbb{ℝ}}^{p_n+1}:{\xi }_j=(1-\tau )n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta <0)\hfill \\ \hfill & -\tau n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta >0)-n^{-1}\sum _{i=1}^n{X}_{ij}{v}_i+\lambda l_j\},\hfill \end{array}$$

where v_i = 0 if $Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta \ne 0$ and v_i ∈ [τ − 1, τ ] otherwise; l₀ = 0; l_j = sgn(β_j) if β_j ≠ 0 and l_j ∈ [−1, 1] otherwise, for 1 ≤ j ≤ p_n.

Let $s\left(\widehat{\beta }\right)={\{s_0\left(\widehat{\beta }\right),s_1\left(\widehat{\beta }\right),\dots ,s_{p_n}\left(\widehat{\beta }\right)\}}^{\mathrm{T}}$ be the set of the subgradient functions for the unpenalized quantile regression, where

$$s_j\left(\beta \right)=(1-\tau )n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta <0)-\tau n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}\beta >0)-n^{-1}\sum _{i=1}^n{X}_{ij}{v}_i,$$

where v_i = 0 if $Y_i-{\mathbf{x}}_i^{\mathrm{T}}\widehat{\beta }\ne 0$ and v_i ∈ [τ − 1, τ ] otherwise.

Next we present Lemmas 4, 5 and 6 to facilitate the proof of Theorem 1. Tao and An (1997) proposed the numerical algorithm based on the convex difference representation, which is stated in Lemma 4. Lemmas 5 and 6 characterize the properties of the oracle estimator ${\widehat{\beta }}_{\tau }^o$ and the associated subgradient functions s(${\widehat{\beta }}_{\tau }^o$) respectively.

Lemma 4

(Difference Convex Program) g(x) and h(x) are two convex functions. Let x^∗ be a point that admits a neighborhood U such that ∂h(x) ∩ ∂g(x^∗) ≠ ∅, ∀x ∈ U ∩ dom(g). Then x^∗ is a local minimizer of g(x) − h(x).

Lemma 5

Assume the conditions (C4)-(C5) holds and λ = o(n^{−(1−c₂)/2}). For the oracle estimator ${\widehat{\beta }}_{\tau }^o$, there exist $v_i^{\ast }$ which satisfies $v_i^{\ast }=0if{Y}_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o\ne 0and{v}_i^{\ast }\in [\tau -1,\tau ]$ otherwise, such that, with probability approaching one, we have

$$s_j\left({\widehat{\beta }}_{\tau }^o\right)=0,j=0,1,\dots ,q_n,and\mid {\widehat{\beta }}_j^o\mid \ge (a+1∕2)\lambda ,j=1,\dots ,q_n.$$

Proof of Lemma 5

This lemma is parallel to Lemma 2.2 in Wang, Wu and Li (2012). The unpenalized quantile loss objective function is convex. By the convex optimization theory, $0\in \partial {\Sigma }_{i=1}^n{\rho }_{\tau }(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o)$. Therefore, there exists $v_i^{\ast }$ such that $s_j\left({\widehat{\beta }}_{\tau }^o\right)=0$ with $v_i=v_i^{\ast }$ for j = 0, 1, …, q_n. On the other hand,

$$\underset{1\le j\le q_n}{\min }\mid {\widehat{\beta }}_j^o\mid \ge \underset{1\le j\le q_n}{\min }\mid {\beta }_{\tau ,j}^o\mid -\underset{1\le j\le q_n}{\max }\mid {\widehat{\beta }}_j^o-{\beta }_{\tau ,j}^o\mid .$$

Condition (C5) requires that $\underset{1\le j\le q_n}{\min }\mid {\beta }_{\tau ,j}^o\mid \ge {C_n}^{-(1-c_2)∕2}$. In addition, $\underset{1\le j\le q_n}{\max }\mid {\widehat{\beta }}_j^o-{\beta }_{\tau ,j}^o\mid \le \Vert {\widehat{\beta }}_{\tau }^o-{\beta }_{\tau 1}^o\Vert =O_p\left(\sqrt{q_n∕n}\right)=O_p\left(n^{-(1-c_1)∕2}\right)=o_p\left(n^{-(1-c_2)∕2}\right)$. Therefore, $\underset{1\le j\le q_n}{\min }\mid {\widehat{\beta }}_j^o\mid \ge {C_n}^{-(1-c_2)∕2}-o_p\left(n^{-(1-c_2)∕2}\right)$, where c₁ and c₂ are defined in conditions (C4) and (C5) respectively. For λ = o{n^{−(1−c₂)/2}}, we have that, with probability approaching one, $\mid {\widehat{\beta }}_j^o\mid \ge (a+1∕2)\lambda ,j=1,\dots ,q_n$, which completes the proof.

Lemma 6

Assume the conditions (C1)-(C5) hold and λ = o(n^{−(1−c₂)/2}), log p_n = o(n^{min{c₂−2θ,θ}}) with some constant 0 < θ < (c₂ − c₁)/2. For the oracle estimator ${\widehat{\beta }}_{\tau 1}^o$ and the s_j(${\widehat{\beta }}_{\tau 1}^o$), with probability approaching one, we have

$$\mid s_j\left({\widehat{\beta }}_{\tau }^o\right)\mid \le \lambda ,and\mid {\widehat{\beta }}_j^o\mid =0,j=q_n+1,\dots ,p_n.$$

Proof of Lemma 6

This lemma is parallel to Lemma 2.3 in Wang, Wu and Li (2012). Since the ${\widehat{\beta }}_{\tau }^o$ is the oracle estimator, |${\widehat{\beta }}_j^o$| = 0, j = q_n + 1, …, p_n. It remains to show that

$$\mathrm{Pr}(\mid s_j\left({\widehat{\beta }}_{\tau }^o\right)\mid >\lambda ,\mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n)\to 0,\mathrm{as}n\to \infty .$$

Let $D=\{i:Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o=0\}=\{i:Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\widehat{\beta }}_{\tau 1}^o=0\}$, then for j = q_n + 1, …, p_n,

$$\begin{array}{cc}\hfill & s_j\left({\widehat{\beta }}_{\tau }^o\right)\hfill \\ \hfill =& (1-\tau )n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o<0)-\tau n^{-1}\sum _{i=1}^n{X}_{ij}I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o>0)-n^{-1}\sum _{i=1}^n{X}_{ij}{v}_i,\hfill \\ \hfill =& n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o\le 0)-\tau \}-n^{-1}\sum _{i=1}^n{X}_{ij}\{v_i+(1-\tau )I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o=0)\}\hfill \\ \hfill =& n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\widehat{\beta }}_{\tau 1}^o\le 0)-\tau \}-n^{-1}\sum _{i\in D}{X}_{ij}[v_i^{\ast }+(1-\tau )],\hfill \end{array}$$

where $v_i^{\ast }\in [\tau -1,\tau ]$ with i ∈ D satisfies s_j(${\widehat{\beta }}_{\tau }^o$) with $v_i=v_i^{\ast }$, for j = 1, …, q_n by Lemma 5.

$$\begin{array}{cc}\hfill & \mathrm{Pr}(\mid s_j\left({\widehat{\beta }}_{\tau }^o\right)\mid >2\lambda ,\mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n)\hfill \\ \hfill \le & \mathrm{Pr}(\mid n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\widehat{\beta }}_{\tau 1}^o\le 0)-\tau \}\mid >\lambda ,\mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n)\hfill \\ \hfill & +\mathrm{Pr}(\mid n^{-1}\sum _{i\in D}{X}_{ij}\{v_i^{\ast }+(1-\tau )\}\mid >\lambda ,\mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n)\hfill \\ \hfill ≕& T_{n1}+T_{n2}.\hfill \end{array}$$

First, we deal with T_n2. Let M = O(n^θ) with some constant 0 < θ < (c₂ − c₁)/2, we have that

$$\begin{array}{cc}\hfill T_{n2}& \le \mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i\in D}{X}_{ij}1\{\mid X_{ij}\mid \le M\}\{v_i^{\ast }+(1-\tau )\}\mid >\lambda ∕2)\hfill \\ \hfill & +\mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i\in D}{X}_{ij}1\{\mid X_{ij}\mid >M\}[v_i^{\ast }+(1-\tau )]\mid >\lambda ∕2)\hfill \\ \hfill & ≕T_{n21}+T_{n22}\hfill \end{array}$$

Since (x_i,A, Y_i) are in general positions (Koenker, 2005, Section 2.2), with probability tending to one there exists exactly q_n + 1 elements in D. Thus, with probability tending to one,

$$\begin{array}{cc}\hfill & \underset{q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i\in D}{X}_{ij}1\{\mid X_{ij}\mid \le M\}\{v_i^{\ast }+(1-\tau )\}\mid \hfill \\ \hfill & \le M(q_n+1)n^{-1}=O\left(n^{\theta +c_1-1}\right)=o\left(\lambda \right),\hfill \end{array}$$

where the last equality holds for λ = o(n^{−(1−c₂)/2}) and 0 < θ < (c₂ − c₁)/2. Therefore, T_n21 → 0 as n → ∞. Next, we deal with T_n22. Note that the events satisfy

$$\{\mid n^{-1}\sum _{i\in D}{X}_{ij}1\{\mid X_{ij}\mid >M\}[v_i^{\ast }+(1-\tau )]\mid >\lambda ∕2\}\subseteq \{\mid X_{ij}\mid >M,\mathrm{for}\mathrm{some}i\in D\},$$

because that if |X_ij | ≤ M for all i ∈ D, then $n^{-1}{\Sigma }_{i\in D}^{}{X}_{ij}1\{\mid X_{ij}\mid <M\}=0$.

Therefore,

$$\begin{array}{cc}\hfill T_{n22}& \le p_n\underset{j=q_n+1,\dots ,p_n}{\max }\mathrm{Pr}(\mid n^{-1}\sum _{i\in D}{X}_{ij}1\{\mid X_{ij}\mid >M\}[v_i^{\ast }+(1-\tau )]\mid >\lambda ∕2)\hfill \\ \hfill & \le p_n(q_n+1)\underset{i\in D,q_n+1\le j\le p_n}{\max }\mathrm{Pr}(\mid X_{ij}\mid >M)\le p_n(q_n+1)\exp (-tM)E\left\{\exp (t\mid X_{ij}\mid )\right\}\hfill \\ \hfill & \le C{p}_n(q_n+1)\exp (-tM)=C{p}_{n}O\left(n^{c_1}\right)\exp (-t{n}^{\theta })\to 0,\hfill \end{array}$$

as n → ∞, where log p_n = o(n^{min{c₂−2θ,θ}}) with some constant 0 < θ < (c₂ − c₁)/2, 0 < t ≤ t₀, the third inequality holds from Markov’s inequality and the fourth inequality follows from Condition (C1). Therefore, T_n2 = T_n21 + T_n22 → 0, as n → ∞.

It remains to show that

$$\mathrm{Pr}(\mid n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_i^{\mathrm{T}}{\widehat{\beta }}_{\tau }^o<0)-\tau \}\mid >\lambda ,\mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n)\to 0,$$

as n → ∞. We consider

$$\begin{array}{cc}\hfill & T_{n1}\hfill \\ \hfill & \le \mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)-\tau \}\mid >\frac{\lambda }{2})\hfill \\ \hfill & +\mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid n^{-1}\sum _{i=1}^n{X}_{ij}[I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\hfill \\ \hfill & -\{\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}]\mid >\frac{\lambda }{4})\hfill \\ \hfill & +\mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid n^{-1}\sum _{i=1}^n{X}_{ij}\hfill \\ \hfill & \{\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}\mid >\frac{\lambda }{4})=:J_{n1}+J_{n2}+J_{n3}.\hfill \end{array}$$

First, let us consider J_n1. We choose a M = O(n^θ) with 0 < θ < (c₂ − c₁)/2, then

$$\begin{array}{cc}\hfill & J_{n1}\hfill \\ \hfill \le & \mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid \le M\}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o<0)-\tau \}\mid >\frac{\lambda }{4})\hfill \\ \hfill & +\mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid >M\}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)-\tau \}\mid >\frac{\lambda }{4})\hfill \\ \hfill ≕& J_{n11}+J_{n12}.\hfill \end{array}$$

By Hoeffding’s inequality, we have that

$$\mathrm{Pr}(\mid n^{-1}\sum _{i=1}^n{X}_{ij}1(\mid X_{ij}\mid \le M)\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)-\tau \}\mid >\lambda ∕4)\le 2\exp (-\frac{n{\lambda }^2}{8M^2}).$$

Thus, J_n11 ≤ 2p_n exp{−nλ²/(8M ²)} = 2p_n exp(−n^1−2θ λ²/8) → 0, as n → ∞, because log p_n = o(n^{min{c₂−2θ,θ}}) with some constant 0 < θ < (c₂ − c₁)/2 and λ = o{n^{−(1−c₂)/2}}. On the other hand, we can similarly follow the arguments that deal with T_n22 and have that

$$\begin{array}{cc}\hfill J_{n12}& \le p_n\underset{j=q_n+1,\dots ,p_n}{\max }\mathrm{Pr}(\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid >M\}\mid >\frac{\lambda }{4})\hfill \\ \hfill & \le p_{n}n\underset{1\le i\le n,j=q_n+1,\dots ,p_n}{\max }\mathrm{Pr}(\mid X_{ij}\mid >M)=O\left(p_{n}n\right)\exp (-t{n}^{\theta })\to 0,\hfill \end{array}$$

as n → ∞, because log p_n = o(n^{min{c₂−2θ,θ}}). Therefore, J_n1 = J_n11 + J_n12 = o(1).

Following similar arguments for proving Lemma 4.3 of Wang, Wu and Li (2012) and the arguments that deal with T_n22 and J_n12, we can show that J_n2 = o(1). It remains to deal with J_n3. For a fixed M = O(n^θ) with 0 < θ < (c₂ −c₁)/2,

$$\begin{array}{cc}\hfill J_{n3}& \le \mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid \le M\}\hfill \\ \hfill & \{\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}\mid >\frac{\lambda }{8})\hfill \\ \hfill & +\mathrm{Pr}(\underset{j=q_n+1,\dots ,p_n}{\max }\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid >M\}\hfill \\ \hfill & \{\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}\mid >\frac{\lambda }{8})\hfill \\ \hfill & ≕J_{n31}+J_{n32}.\hfill \end{array}$$

To handle J_n31, we observe that

$$\begin{array}{cc}\hfill & \underset{j=q_n+1,\dots ,p_n}{\max }\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid n^{-1}\sum _{i=1}^n{X}_{ij}1\{\mid X_{ij}\mid >M\}\hfill \\ \hfill & \{\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_1\le 0)-\mathrm{Pr}(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\beta }_{\tau 1}^o\le 0)\}\mid \hfill \\ \hfill & \le M\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }\mid E\left\{f(\zeta \mid {\mathbf{x}}_A){\mathbf{x}}_A^T({\beta }_1-{\beta }_{\tau 1}^o)\right\}\mid \hfill \\ \hfill & \le M\underset{\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \le \Delta \sqrt{q_n∕n}}{\sup }{\lambda }_{\max }^{1∕2}\left\{E\left({\mathbf{x}}_A{\mathbf{x}}_A^T\right)\right\}\Vert {\beta }_1-{\beta }_{\tau 1}^o\Vert \hfill \\ \hfill & \le O\left\{n^{\theta }{(q_n∕n)}^{1∕2}\right\}=O\left\{n^{-(1-c_1-2\theta )∕2}\right\},\hfill \end{array}$$

where f (· | x_A) is defined in Condition (C3) with ζ is between $u_{\tau }^o+{\mathbf{x}}_A^{\mathrm{T}}({\beta }_1-{\beta }_{\tau 1}^o)$ and $u_{\tau }^o$ and thus the second inequality follows Condition (C3) and Cauchy-Schwartz inequality, and the third inequality follows Condition (C2). Consequently, together with λ = o{n^{−(1−c₂)/2}}, we have that J_n31 ≤ Pr{O(n^{−(1−c₁−2θ)/2}) > λ/8} = o(1) if 0 < θ < (c₂ − c₁)/2. We can also follow similar arguments for handling J_n12 and obtain that J_n32 = o(1). Therefore, J_n3 = J_n31 + J_n32 = o(1). Consequently,

$$\mathrm{Pr}\{\underset{q_n+1,\dots ,p_n}{\max }\mid n^{-1}\sum _{i=1}^n{X}_{ij}\{I(Y_i-{\mathbf{x}}_{i,A}^{\mathrm{T}}{\widehat{\beta }}_{\tau 1}^o<0)-\tau \}\mid >\lambda \}\le J_{n1}+J_{n2}+J_{n3}=o\left(1\right),$$

which implies that $\mathrm{Pr}\{\mid s_j\left({\widehat{\beta }}_{\tau }^o\right)\mid <\lambda \mathrm{for}\mathrm{some}j=q_n+1,\dots ,p_n\}\to 0$. This completes the proof of Lemma 6.

With Lemmas 5 and 6 for the random x with sub-exponential tail probability Condition (C1), we can follow the technical proof of Theorem 2.4 of Wang, Wu and Li (2012) to obtain the oracle property and complete the proof.

Proof of Theorem 2

For notational clarity, we use c₁ and c₂ to denote two different generic positive constants. First we assume F (y) is known. That is, ${\widehat{\mathrm{dcov}}}^{\ast 2}\{X_k,F\left(Y\right)\}={\widehat{S}}_{k1}^{\ast }+{\widehat{S}}_{k2}^{\ast }-2{\widehat{S}}_{k3}^{\ast }$, where

$$\begin{array}{cc}\hfill & {\widehat{S}}_{k,1}^{\ast }=\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid X_{ik}-X_{jk}\mid \mid F\left(Y_i\right)-F\left(Y_j\right)\mid ,\hfill \\ \hfill & {\widehat{S}}_{k,2}^{\ast }=\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid X_{ik}-X_{jk}\mid \frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\mid F\left(Y_i\right)-F\left(Y_j\right)\mid ,\mathrm{and}\hfill \\ \hfill & {\widehat{S}}_{k,3}^{\ast }=\frac{1}{n^3}\sum _{i=1}^n\sum _{j=1}^n\sum _{l=1}^n\mid X_{ik}-X_{jk}\mid \mid F\left(Y_i\right)-F\left(Y_j\right)\mid .\hfill \end{array}$$

Similarly we define ${\widehat{\omega }}_k^{\ast }={\widehat{\mathrm{dcorr}}}^{\ast 2}\{X_k,F\left(Y\right)\}$. Theorem 1 of Li, Zhong and Zhu (2012) stated that, for any 0 < γ < 1/2 − κ, there exist positive constants c₁ > 0 and c₂ > 0 such that

$$\mathrm{Pr}(\underset{1\le k\le p}{\max }\mid {\widehat{\omega }}_k^{\ast }-{\omega }_k\mid \ge c{n}^{-\kappa })\le O\left(p[\exp \{-c_1{n}^{1-2(\kappa +\gamma )}\}+n\exp (-c_2{n}^{\gamma })]\right).$$ (C.1)

To prove Theorem 2, it thus suffices to show the difference between ${\widehat{\omega }}_k^{\ast }\mathrm{and}{\widehat{\omega }}_k$ defined in (3.5) is ignorable when size n is large enough, which amounts to studying the differences between ${\widehat{S}}_{km}^{\ast }\mathrm{and}{\widehat{S}}_{km}$ for m = 1, 2, 3. Next, we sketch the proof for the case m = 1 only because the proof of the other two cases is in spirit the same. We recall that ${\widehat{S}}_{k1}^{\ast }=\frac{1}{n^2}{\Sigma }_{i=1}^n{\Sigma }_{j=1}^n\mid X_{ik}-X_{jk}\mid \mid F\left(Y_1\right)-F\left(Y_j\right)\mid \mathrm{and}{\widehat{S}}_{k1}^{\ast }=\frac{1}{n^2}{\Sigma }_{i=1}^n{\Sigma }_{j=1}^n\mid X_{ik}-X_{jk}\mid \mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid$.

$$\begin{array}{cc}\hfill & \mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}\mid {\widehat{S}}_{k1}^{\ast }-{\widehat{S}}_{k1}\ge \epsilon \right)\hfill \\ \hfill =& \mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}{n}^{-2}\sum _{i=1}^n\sum _{j=1}^n\mid X_{ik}-X_{jk}\mid \mid \mid F\left(Y_i\right)-F\left(Y_j\right)-\mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid \mid \ge \epsilon \right)\hfill \\ \hfill \le & \mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}{\left(A_n{B}_n\right)}^{1∕2}\ge \epsilon \right)\hfill \\ \hfill \le & \mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}{\left(A_n{B}_n\right)}^{1∕2}\ge \epsilon ,\mid X_k\mid \le M\right)+\mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}{\left(A_n{B}_n\right)}^{1∕2}\ge \epsilon ,\mid X_k\mid >M\right)\hfill \\ \hfill ≕& T_1+T_2,\hfill \end{array}$$

where M is a positive constant specified later, $A_n=n^{-2}{\Sigma }_{i=1}^n{\Sigma }_{j=1}^n{(X_{ik}-X_{jk})}^2,\mathrm{and}{B}_n=n^{-2}{\Sigma }_{i=1}^n{\Sigma }_{j=1}^n{\{\mid F\left(Y_i\right)-F\left(Y_j\right)\mid -\mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid \}}^2$.

Using the argument that | |x| − |y| |≤ |x − y| ≤ |x| + |y|, we obtain that

$$\begin{array}{cc}\hfill & \mid \mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid -\mid F\left(Y_i\right)-F\left(Y_j\right)\mid \mid \hfill \\ \hfill \le & \mid F_n\left(Y_i\right)-F\left(Y_i\right)\mid +\mid F_n\left(Y_j\right)-F\left(Y_j\right)\mid \le 2\underset{1\le i\le n}{\text{max}}\mid F_n\left(Y_i\right)-F\left(Y_i\right)\mid .\hfill \end{array}$$

Also because $\underset{1\le k\le p_n}{\max }{A}_n\le \underset{1\le k\le p_n}{\max }{n}^{-2}{\Sigma }_{i=1}^n{\Sigma }_{j=1}^{n}2(X_{ik}^2+X_{jk}^2)\le 4M^2$, we have

$$\begin{array}{cc}\hfill T_1& \le \mathrm{Pr}\left[\underset{1\le k\le p_n}{\text{max}}2M{n}^{-1}{\left\{\sum _{i=1}^n\sum _{j=1}^n{(\mid F\left(Y_i\right)-F\left(Y_j\right)\mid -\mid F_n\left(Y_i\right)-F_n\left(Y_j\right)\mid )}^2\right\}}^{1∕2}\ge \epsilon \right]\hfill \\ \hfill & \le \mathrm{Pr}\left\{4M\underset{1\le i\le n}{\text{max}}\mid F_n\left(Y_i\right)-F\left(Y_i\right)\mid \ge \epsilon \right\}\le \mathrm{Pr}\left\{\underset{y\in R}{\text{max}}\mid F_n\left(y\right)-F\left(y\right)\mid \ge \epsilon ∕4M\right\}\hfill \\ \hfill & \le 2\text{exp}\{-2n{(\epsilon ∕4M)}^2\}=2\text{exp}(-n{\epsilon }^2∕8M^2),\hfill \end{array}$$ (C.2)

where the last inequality follows by Dvoretzky-Kiefer-Wolfowitz inequality.

For the second term, for all 0 < s ≤ 2s₀, where s₀ is defined in Condition (C1),

$$\begin{array}{cc}\hfill T_2& \le \mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}\mid X_k\mid >M\right)=\mathrm{Pr}\left(\underset{1\le k\le p_n}{\text{max}}\text{exp}(s\mid X_k\mid )>\text{exp}\left(sM\right)\right)\hfill \\ \hfill & \le \underset{1\le k\le p_n}{\text{max}}E\left\{\text{exp}(s\mid X_k\mid )\right\}\text{exp}(-sM)\le C\text{exp}(-sM),\hfill \end{array}$$ (C.3)

where C is a positive constant, the second inequality follows Markov’s inequality and the last inequality is applied under Condition (C1).

Then, by choosing M = O(n^γ) for 0 < γ < 1/2 − κ, (C.2) and (C.3) together imply that, for some positive constants c₁ and c₂,

$$\begin{array}{cc}\hfill \text{Pr}\left(\underset{1\le k\le p_n}{\text{max}}\mid {\widehat{S}}_{k1}^{\ast }-{\widehat{S}}_{k1}\mid \ge \epsilon \right)& \le 2\text{exp}(-n{\epsilon }^2∕8M^2)+C\text{exp}(-sM)\hfill \\ \hfill & \le 2\text{exp}(-c_1{\epsilon }^2{n}^{1-2\gamma })+C\text{exp}(-c_2{n}^{\gamma }).\hfill \end{array}$$ (C.4)

Thus, it is not difficult to show that

$$\text{Pr}\left(\underset{1\le k\le p}{\text{max}}\mid {\widehat{\omega }}_k^{\ast }-{\widehat{\omega }}_k\mid \ge c{n}^{-\kappa }\right)\le O\left(\text{exp}\left\{-c_1{n}^{1-2(\kappa +\gamma )}\right\}+\text{exp}(-c_2{n}^{\gamma })\right).$$ (C.5)

Therefore, (C.1) and (C.5) together completes the proof of Theorem 2.