Optimal Feature Selection in High-Dimensional Discriminant Analysis

Abstract

We consider the high-dimensional discriminant analysis problem. For this problem, different methods have been proposed and justified by establishing exact convergence rates for the classification risk, as well as the ℓ₂ convergence results to the discriminative rule. However, sharp theoretical analysis for the variable selection performance of these procedures have not been established, even though model interpretation is of fundamental importance in scientific data analysis. This paper bridges the gap by providing sharp sufficient conditions for consistent variable selection using the sparse discriminant analysis (Mai et al., 2012). Through careful analysis, we establish rates of convergence that are significantly faster than the best known results and admit an optimal scaling of the sample size n, dimensionality p, and sparsity level s in the high-dimensional setting. Sufficient conditions are complemented by the necessary information theoretic limits on the variable selection problem in the context of high-dimensional discriminant analysis. Exploiting a numerical equivalence result, our method also establish the optimal results for the ROAD estimator (Fan et al., 2012) and the sparse optimal scaling estimator (Clemmensen et al., 2011). Furthermore, we analyze an exhaustive search procedure, whose performance serves as a benchmark, and show that it is variable selection consistent under weaker conditions. Extensive simulations demonstrating the sharpness of the bounds are also provided.

1 Introduction

We consider the problem of binary classification with high-dimensional features. More specifically, given n data points, {(x_i, y_i), i = 1, …, n}, sampled from a joint distribution of (X, Y) ∈ ℝ^p × {1, 2}, we want to determine the class label y for a new data point x ∈ ℝ^p.

Let p₁(x) and p₂(x) be the density functions of X given Y = 1 (class 1) and Y = 2 (class 2) respectively, and the prior probabilities π₁ = (Y = 1), π₂ = (Y = 2). Classical multivariate analysis theory shows that the Bayes rule classifies a new data point x to class 2 if and only if

$$log\left(\frac{p_2(\mathbf{x})}{p_1(\mathbf{x})}\right)+log\left(\frac{{\pi }_2}{{\pi }_1}\right)>0.$$ (1.1)

The Bayes rule usually serves as an oracle benchmark, since, in practical data analysis, the class conditional densities p₂(x) and p₁(x) are unknown and need to be estimated from the data.

Throughout the paper, we assume that the class conditional densities p₁(x) and p₂(x) are Gaussian. That is, we assume that

$$\mathbf{X}\mid Y=1~\mathcal{N}({\mathit{\mu }}_1,\mathbf{\sum })\text{and}\mathbf{X}\mid Y=2~\mathcal{N}({\mathit{\mu }}_2,\mathbf{\sum }).$$ (1.2)

This assumption leads us to linear discriminant analysis (LDA) and the Bayes rule in (1.1) becomes

$$g(\mathbf{x};{\mathit{\mu }}_1,{\mathit{\mu }}_2,\mathbf{\sum }):=\{\begin{array}{ll}2\hfill & \text{if}{\left(\mathbf{x}-({\mathit{\mu }}_1+{\mathit{\mu }}_2)/2\right)}^{\prime }{\mathbf{\sum }}^{-1}\mathit{\mu }+log({\pi }_2/{\pi }_1)>0\hfill \\ 1\hfill & \text{otherwise}\hfill \end{array}$$ (1.3)

where μ = μ₂ − μ₁. Theoretical properties of the plug-in rule g(x; μ̂₁, μ̂₂, Σ̂), where (μ̂₁, μ̂₂, Σ̂) are sample estimates of (μ₁, μ₂, Σ), have been well studied when the dimension p is low (Anderson, 2003).

In high-dimensions, the standard plug-in rule works poorly and may even fail completely. For example, Bickel and Levina (2004) show that the classical low dimensional normal-based linear discriminant analysis is asymptotically equivalent to random guessing when the dimension p increases at a rate comparable to the sample size n. To overcome this curse of dimensionality, it is common to impose certain sparsity assumptions on the model and then estimate the high-dimensional discriminant rule using plug-in estimators. The most popular approach is to assume that both Σ and μ are sparse. Under this assumption, Shao et al. (2011) propose to use a thresholding procedure to estimate Σ and μ and then plug them into the Bayes rule. In a more extreme case, Tibshirani et al. (2003), Wang and Zhu (2007), Fan and Fan (2008) assume that Σ = I and estimate μ using a shrinkage method. Another common approach is to assume that Σ⁻¹ and μ are sparse. Under this assumption, Witten and Tibshirani (2009) propose the scout method which estimates Σ⁻¹ using a shrunken estimator. Though these plug-in approaches are simple, they are not appropriate for conducting variable selection in the discriminant analysis setting. As has been elaborated in Cai et al. (2011) and Mai et al. (2012), for variable selection in high-dimensional discriminant analysis, we need to directly impose sparsity assumptions on the Bayes discriminant direction β = Σ⁻¹μ instead of separately on Σ and μ. In particular, it is assumed that $\mathit{\beta }={({\mathit{\beta }}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ for T = {1,…, s}. Their key observation comes from the fact that the Fisher’s discriminant rule depends on Σ and μ only through the product Σ⁻¹μ. Furthermore, in the high-dimensional setting, it is scientifically meaningful that only a small set of variables are relevant to classification, which is equivalent to the assumption that β is sparse. On a simple example of tumor classification, Mai et al. (2012) elaborate why it is scientifically more informative to directly impose sparsity assumption on β instead of on μ (For more details, see Section 2 of their paper). In addition, Cai et al. (2011) point out that the sparsity assumption on β is much weaker than imposing sparsity assumptions Σ⁻¹ and μ separately. A number of authors have also studied classification in this setting (Wu et al., 2009, Fan et al., 2012, Witten and Tibshirani, 2011, Clemmensen et al., 2011, Cai et al., 2011, Mai et al., 2012).

In this paper, we adopt the same assumption that β is sparse and focus on analyzing the SDA (Sparse Discriminant Analysis) proposed by Mai et al. (2012). This method estimates the discriminant direction β (More precisely, they estimate a quantity that is proportional to β.) and our focus will be on variable selection consistency, that is, whether this method can recover the set T with high probability. In a recent work, Mai and Zou (2012) prove that the SDA estimator is numerically equivalent to the ROAD estimator (Fan et al., 2012) and the sparse optimal scaling estimator (Clemmensen et al., 2011). By exploiting this result, our theoretical analysis provides a unified theoretical justification for all these three methods.

1.1 Main Results

Let n₁ = |{i : y_i = 1}| and n₂ = n − n₁. The SDA estimator is obtained by solving the following least squares optimization problem

$$\underset{\mathbf{v}\in {ℝ}^p}{min}\frac{1}{2(n-2)}\sum _{i\in [n]}{(z_i-{\mathbf{v}}^{\prime }({\mathbf{x}}_i-\overline{\mathbf{x}}))}^2+\lambda {\Vert \mathbf{v}\Vert }_1,$$ (1.4)

where [n] denotes the set {1,…, n}, x̄ = n⁻¹Σ_i x_i and the vector z ∈ ℝⁿ encodes the class labels as z_i = n₂/n if y_i = 1 and z_i = −n₁/n if y_i = 2. Here λ > 0 is a regularization parameter.

The SDA estimator in (1.4) uses an ℓ₁-norm penalty to estimate a sparse v and avoid the curse of dimensionality. Mai et al. (2012) studied its variable selection property under a different encoding scheme of the response z_i. However, as we show later, different coding schemes do not affect the results. When the regularization parameter λ is set to zero, the SDA estimator reduces to the classical Fisher’s discriminant rule.

The main focus of the paper is to sharply characterize the variable selection performance of the SDA estimator. From a theoretical perspective, unlike the high dimensional regression setting where sharp theoretical results exist for prediction, estimation, and variable selection consistency, most existing theories for high-dimensional discriminant analysis are either on estimation consistency or risk consistency, but not on variable selection consistency (see, for example, Fan et al., 2012, Cai et al., 2011, Shao et al., 2011). Mai et al. (2012) provide a variable selection consistency result for the SDA estimator in (1.4). However, as we will show later, their obtained scaling in terms of (n, p, s) is not optimal. Though some theoretical analysis of the ℓ₁-norm penalized M-estimators exists (see Wainwright (2009a), Negahban et al. (2012)), these techniques are not applicable to analyze the estimator given in (1.4). In high-dimensional discriminant analysis the underlying statistical model is fundamentally different from that of the regression analysis. At a high level, to establish variable selection consistency of the SDA estimator, we characterize the Karush-Kuhn-Tucker (KKT) conditions for the optimization problem in (1.4). Unlike the ℓ₁-norm penalized least squares regression, which directly estimates the regression coefficients, the solution to (1.4) is a quantity that is only proportional to the Bayes rule’s direction $\mathit{\beta }={\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T$. To analyze such scaled estimators, we need to resort to different techniques and utilize sophisticated multivariate analysis results to characterize the sampling distributions of the estimated quantities. More specifically, we provide sufficient conditions under which the SDA estimator is variable selection consistent with a significantly improved scaling compared to that obtained by Mai et al. (2012). In addition, we complement these sufficient conditions with information theoretic limitations on recovery of the feature set T. In particular, we provide lower bounds on the sample size and the signal level needed to recover the set of relevant variables by any procedure. We identify the family of problems for which the estimator (1.4) is variable selection optimal. To provide more insights into the problem, we analyze an exhaustive search procedure, which requires weaker conditions to consistently select relevant variables. This estimator, however, is not practical and serves only as a benchmark. The obtained variable selection consistency result also enables us to establish risk consistency for the SDA estimator. In addition, Mai and Zou (2012) show that the SDA estimator is numerically equivalent to the ROAD estimator proposed by Wu et al. (2009), Fan et al. (2012) and the sparse optimal scaling estimator proposed by Clemmensen et al. (2011). Therefore, the results provided in this paper also apply to those estimators. Some of the main results of this paper are summarized below.

Let v̂^SDA denote the minimizer of (1.4). We show that if the sample size

$$n\ge C\left(\underset{a\in N}{max}{\mathrm{\sum }}_{a\mid T}\right){\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})slog((p-s)log(n)),$$ (1.5)

where C is a fixed constant which does not scale with n, p and s, ${\sigma }_{a\mid T}={\sigma }_{aa}-{\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{Ta}$, and Λ_min(Σ) denotes the minimum eigenvalue of Σ, then the estimated vector v̂^SDA has the same sparsity pattern as the true β, thus establishing variable selection consistency (or sparsistency) for the SDA estimator. This is the first result that proves that consistent variable selection in the discriminant analysis can be done under a similar theoretical scaling as variable selection in the regression setting (in terms of n, p and s). To prove (1.5), we impose conditions that min_j∈T |β_j| is not too small and ${\Vert {\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }\le 1-\alpha$ with α ∈ (0, 1), where N = [p]\T. The latter one is the irrepresentable condition, which is commonly used in the ℓ₁-norm penalized least squares regression problem (Zou, 2006, Meinshausen and Bühlmann, 2006, Zhao and Yu, 2006, Wainwright, 2009a). Let β_min be the magnitude of the smallest absolute value of the non-zero component of β. Our analysis of information theoretic limitations reveals that, whenever $n<C_1{\beta }_{min}^{-2}log(p-s)$, no procedure can reliably recover the set T. In particular, under certain regimes, we establish that the SDA estimator is optimal for the purpose of variable selection. The analysis of the exhaustive search decoder reveals a similar result. However, the exhaustive search decoder does not need the irrepresentable condition to be satisfied by the covariance matrix. Thorough numerical simulations are provided to demonstrate the sharpness of our theoretical results.

In a preliminary work, Kolar and Liu (2013) present some variable selection consistency results related to the ROAD estimator under the assumption that π₁ = π₂ = 1/2. However, it is hard to directly compare their analysis with that of Mai et al. (2012) to understand why an improved scaling is achievable, since the ROAD estimator is the solution to a constrained optimization while the SDA estimator is the solution to an unconstrained optimization. This paper analyzes the SDA estimator and is directly comparable with the result of Mai et al. (2012). As we will discuss later, our analysis attains better scaling due to a more careful characterization of the sampling distributions of several scaled statistics. In contrast, the analysis in Mai et al. (2012) hinges on the sup-norm control of the deviation of the sample mean and covariance to their population quantities, which is not sufficient to obtain the optimal rate. Using the numerical equivalence between the SDA and the ROAD estimator, the theoretical results of this paper also apply on the ROAD estimator. In addition, we also study an exhaustive search decoder and information theoretic limits on the variable selection in high-dimensional discriminant analysis. Furthermore, we provide discussions on risk consistency and approximate sparsity, which shed light on future investigations.

The rest of this paper is organized as follows. In the rest of this section, we introduce some more notation. In §2, we study sparsistency of the SDA estimator. An information theoretic lower bound is given in §3. We characterize the behavior of the exhaustive search procedure in §4. Consequences of our results are discussed in more details in §5. Numerical simulations that illustrate our theoretical findings are given in §6. We conclude the paper with a discussion and some results on the risk consistency and approximate sparsity in §7. Technical results and proofs are deferred to the appendix.

1.2 Notation

We denote [n] to be the set {1,…, n}. Let T ⊆ [p] be an index set, we denote β_T to be the subvector containing the entries of the vector β indexed by the set T, and X_T denotes the submatrix containing the columns of X indexed by T. Similarly, we denote A_TT to be the submatrix of A with rows and columns indexed by T. For a vector a ∈ ℝⁿ, we denote supp(a) = {j : a_j ≠ 0} to be the support set. We also use ||a||_q, q ∈ [1, ∞), to be the ℓ_q-norm defined as ||a||_q = (Σ_i∈[n] |a_i|^q)^1/q with the usual extensions for q ∈ {0, ∞}, that is, ||a||₀ = |supp(a)| and ||a||_∞ = max_i∈[n] |a_i|. For a matrix A ∈ ℝ^n×p, we denote |||A|||_∞ = max_i∈[n]Σ_j∈[p] |a_ij| the ℓ_∞ operator norm. For a symmetric positive definite matrix A ∈ ℝ^p×p we denote Λ_min(A) and Λ_max(A) to be the smallest and largest eigenvalues, respectively. We also represent the quadratic form ${\Vert \mathbf{a}\Vert }_{\mathbf{A}}^2={\mathbf{a}}^{\prime }\mathbf{Aa}$ for a symmetric positive definite matrix A. We denote I_n to be the n × n identity matrix and 1_n to be the n × 1 vector with all components equal to 1. For two sequences {a_n} and {b_n}, we use a_n = (b_n) to denote that a_n < Cb_n for some finite positive constant C. We also denote a_n = (b_n) to be b_n ≳ a_n. If a_n = (b_n) and b_n = (a_n), we denote it to be a_n ≍ b_n. The notation a_n = o(b_n) is used to denote that $a_n{b}_n^{-1}\to 0$.

2 Sparsistency of the SDA Estimator

In this section, we provide sharp sparsistency analysis for the SDA estimator defined in (1.4). Our analysis decomposes into two parts: (i) We first analyze the population version of the SDA estimator in which we assume that Σ, μ₁, and μ₂ are known. The solution to the population problem provides us insights on the variable selection problem and allows us to write down sufficient conditions for consistent variable selection. (ii) We then extend the analysis from the population problem to the sample version of the problem in (1.4). For this, we need to replace Σ, μ₁, and μ₂ by their corresponding sample estimates Σ̂, μ̂₁, and μ̂₂. The statement of the main result is provided in §2.2 with an outline of the proof in §2.3.

2.1 Population Version Analysis of the SDA Estimator

We first lay out conditions that characterize the solution to the population version of the SDA optimization problem.

Let X₁ ∈ ℝ^n₁×p be the matrix with rows containing data points from the first class and similarly define X₂ ∈ ℝ^n₂×p to be the matrix with rows containing data points from the second class. We denote ${\mathbf{H}}_1={\mathbf{I}}_{n_1}-n_1^{-1}{\mathbf{1}}_{n_1}{\mathbf{1}}_{n_1}^{\prime }$ and ${\mathbf{H}}_2={\mathbf{I}}_{n_2}-n_2^{-1}{\mathbf{1}}_{n_2}{\mathbf{1}}_{n_2}^{\prime }$ to be the centering matrices. We define the following quantities

$$\begin{array}{c}{\widehat{\mathit{\mu }}}_1=n_1^{-1}\sum _{i:y_i=1}{\mathbf{x}}_i=n_1^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{1}}_{n_1},{\widehat{\mathit{\mu }}}_2=n_2^{-1}\sum _{i:y_i=2}{\mathbf{x}}_i=n_2^{-1}{\mathbf{X}}_2^{\prime }{\mathbf{1}}_{n_2},\\ \widehat{\mathit{\mu }}={\widehat{\mathit{\mu }}}_2-{\widehat{\mathit{\mu }}}_1,\\ {\mathbf{S}}_1={(n_1-1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{H}}_1{\mathbf{X}}_1,{\mathbf{S}}_2={(n_2-1)}^{-1}{\mathbf{X}}_2^{\prime }{\mathbf{H}}_2{\mathbf{X}}_2,\\ \mathbf{S}={(n-2)}^{-1}((n_1-1){\mathbf{S}}_1+(n_2-1){\mathbf{S}}_2).\end{array}$$

With this notation, observe that the optimization problem in (1.4) can be rewritten as

$$\underset{\mathbf{v}\in {ℝ}^p}{min}\frac{1}{2}{\mathbf{v}}^{\prime }\left(\mathbf{S}+\frac{n_1{n}_2}{n(n-2)}\widehat{\mathit{\mu }}{\widehat{\mathit{\mu }}}^{\prime }\right)\mathbf{v}-\frac{n_1{n}_2}{n(n-2)}{\mathbf{v}}^{\prime }\widehat{\mathit{\mu }}+\lambda {\Vert \mathbf{v}\Vert }_1,$$

where we have dropped terms that do not depend on v. Therefore, we define the population version of the SDA optimization problem as

$$\underset{\mathbf{w}}{min}\frac{1}{2}{\mathbf{w}}^{\prime }(\mathbf{\sum }+{\pi }_1{\pi }_2\mathit{\mu }{\mathit{\mu }}^{\prime })\mathbf{w}-{\pi }_1{\pi }_2{\mathbf{w}}^{\prime }\mathit{\mu }+\lambda {\Vert \mathbf{w}\Vert }_1,$$ (2.1)

Let ŵ be the solution of (2.1). We are aiming to characterize conditions under which the solution ŵ recovers the sparsity pattern of β = Σ⁻¹μ. Recall that T = supp(β) = {1,…, s} denotes the true support set and N = [p]\T, under the sparsity assumption, we have

$${\mathit{\beta }}_T={\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\text{and}{\mathit{\mu }}_N={\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T.$$ (2.2)

We define β_min as

$${\beta }_{min}=\underset{a\in T}{min}\mid {\beta }_a\mid .$$ (2.3)

The following theorem characterizes the solution to the population version of the SDA optimization problem in (2.1).

Theorem 1

Let α ∈ (0, 1] be a constant and ŵ be the solution to the problem in (2.1). Under the assumptions that

$${\Vert {\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }\le 1-\alpha ,$$ (2.4)

$${\pi }_1{\pi }_2\frac{1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{\beta }_{min}>\lambda {\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty },$$ (2.5)

we have $\widehat{\mathbf{w}}=({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })$ with

$${\widehat{\mathbf{w}}}_T={\pi }_1{\pi }_2\frac{1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{\mathit{\beta }}_T-\lambda {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T).$$ (2.6)

Furthermore, we have sign(ŵ_T) = sign(β_T).

Equations (2.4) and (2.5) provide sufficient conditions under which the solution to (2.1) recovers the true support. The condition in (2.4) takes the same form as the irrepresentable condition commonly used in the ℓ₁-penalized least squares regression problem (Zou, 2006, Meinshausen and Bühlmann, 2006, Zhao and Yu, 2006, Wainwright, 2009a). Equation (2.5) specifies that the smallest component of β_T should not be too small compared to the regularization parameter λ. In particular, let $\lambda ={\lambda }_0/(1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2)$ for some λ₀. Then (2.5) suggests that ŵ_T recovers the true support of β as long as ${\beta }_{min}\ge {\lambda }_0{\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }$. Equation (2.6) provides an explicit form for the solution ŵ, from which we see that the SDA optimization procedure estimates a scaled version of the optimal discriminant direction. Whenever λ ≠ 0, ŵ is a biased estimator. However, such estimation bias does not affect the recovery of the support set T of β when λ is small enough.

We present the proof of Theorem 1, as the analysis of the sample version of the SDA estimator will follow the same lines. We start with the Karush-Kuhn-Tucker (KKT) conditions for the optimization problem in (2.1):

$$(\mathbf{\sum }+{\pi }_1{\pi }_2\mathit{\mu }{\mathit{\mu }}^{\prime })\widehat{\mathbf{w}}-{\pi }_1{\pi }_2\mathit{\mu }+\lambda \widehat{\mathbf{z}}=\mathbf{0}$$ (2.7)

where ẑ ∈ ∂||ŵ||₁ is an element of the subdifferential of ||·||₁.

Let ŵ_T be defined in (2.6). We need to show that there exists a ẑ such that the vector $\widehat{\mathbf{w}}={({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$, paired with ẑ, satisfies the KKT conditions and sign(ŵ_T) = sign(β_T).

The explicit form of ŵ_T is obtained as the solution to an oracle optimization problem, specified in (2.8), where the solution is forced to be non-zero only on the set T. Under the assumptions of Theorem 1, the solution ŵ_T to the oracle optimization problem satisfies sign(ŵ_T) = sign(β_T). We complete the proof by showing that the vector ${({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ satisfies the KKT conditions for the full optimization procedure.

We define the oracle optimization problem to be

$$\underset{{\mathbf{w}}_T}{min}\frac{1}{2}{\mathbf{w}}_T^{\prime }({\mathbf{\sum }}_{TT}+{\pi }_1{\pi }_2{\mathit{\mu }}_T{\mathit{\mu }}_T^{\prime }){\mathbf{w}}_T-{\pi }_1{\pi }_2{\mathbf{w}}_T^{\prime }{\mathit{\mu }}_T+\lambda {\mathbf{w}}_T^{\prime }\text{sign}({\mathit{\beta }}_T).$$ (2.8)

The solution w̃_T to the oracle optimization problem (2.8) satisfies w̃_T = ŵ_T where ŵ_T is given in (2.6). It is immediately clear that under the conditions of Theorem 1, sign (w̃_T) = sign(β_T).

The next lemma shows that the vector ${({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is the solution to the optimization problem in (2.1) under the assumptions of Theorem 1.

Lemma 2

Under the conditions of Theorem 1, we have that $\widehat{\mathbf{w}}=({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })$ is the solution to the problem in (2.1), where ŵ_T is defined in (2.6).

This completes the proof of Theorem 1.

The next theorem shows that the irrepresentable condition in (2.4) is almost necessary for sign consistency, even if the population quantities Σ and μ are known.

Theorem 3

Let ŵ be the solution to the problem in (2.1). If we have sign(ŵ_T) = sign(β_T), Then, there must be

$${\Vert {\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }\le 1.$$ (2.9)

The proof of this theorem follows similar argument as in the regression settings in Zhao and Yu (2006), Zou (2006).

2.2 Sample Version Analysis of the SDA Estimator

In this section, we analyze the variable selection performance of the sample version of the SDA estimator v̂ = v̂^SDA defined in (1.4). In particular, we will establish sufficient conditions under which v̂ correctly recovers the support set of β (i.e., we will derive conditions under which $\widehat{\mathbf{v}}={({\widehat{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ and sign(v̂_T) = sign(β_T)). The proof construction follows the same line of reasoning as the population version analysis. However, proving analogous results in the sample version of the problem is much more challenging and requires careful analysis of the sampling distribution of the scaled functionals of Gaussian random vectors.

The following theorem is the main result that characterizes the variable selection consistency of the SDA estimator.

Theorem 4

We assume that the condition in (2.4) holds. Let the penalty parameter be $\lambda ={\left(1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)}^{-1}{\lambda }_0$ with

$${\lambda }_0=K_{{\lambda }_0}\sqrt{{\pi }_1{\pi }_2\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right)\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log((p-s)log(n))}{n}}$$ (2.10)

where K_λ0 is a sufficiently large constant. Suppose that β_min = min_a∈T |β_a| satisfies

$${\beta }_{min}\ge K_{\beta }(\sqrt{\left(\underset{a\in T}{max}{({\mathbf{\sum }}_{TT}^{-1})}_{aa}\right)\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log(slog(n))}{n}}\bigvee {\lambda }_0{\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty })$$ (2.11)

for a sufficiently large constant K_β. If

$$n\ge K{\pi }_1{\pi }_2\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right){\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})slog((p-s)log(n))$$ (2.12)

for some constant K, then $\widehat{\mathbf{w}}={({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is the solution to the optimization problem in (1.4), where

$$\begin{array}{l}{\widehat{\mathbf{v}}}_T=\frac{n_1{n}_2}{n(n-2)}\frac{1+\lambda {\Vert \widehat{{\mathit{\beta }}_T}\Vert }_1}{1+\frac{n_1{n}_2}{n(n-2)}{\Vert \widehat{{\mathit{\beta }}_T}\Vert }_{{\mathbf{S}}_{TT}}^2}\widehat{{\mathit{\beta }}_T}-\lambda {\mathbf{S}}_{TT}^{-1}\text{sign}(\widehat{{\mathit{\beta }}_T})\mathit{and}\\ {\widehat{\mathit{\beta }}}_T={\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T,\end{array}$$ (2.13)

with probability at least 1 − (log⁻¹(n)). Furthermore, sign(v̂_T) = sign(β_T).

Theorem 4 is a sample version of Theorem 1 given in the previous section. Compared to the population version result, in addition to the irrepresentable condition and a lower bound on β_min, we also need the sample size n to be large enough for the SDA procedure to recover the true support set T with high probability.

At the first sight, the conditions of the theorem look complicated. To highlight the main result, we consider a case where 0 < c ≤ Λ_min(Σ_TT) and $\left({\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\bigvee {\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }\right)\le \overline{C}<\infty$ for some constants c, C̄. In this case, it is sufficient that the sample size scales as n ≍ s log(p − s) and β_min ≳ s^−1/2. This scaling is of the same order as for the Lasso procedure, where n ≳ s log(p − s) is needed for correct recovery of the relevant variables under the same assumptions (see Theorem 3 in Wainwright, 2009a). In §5, we provide more detailed explanation of this theorem and complement it with the necessary conditions given by the information theoretic limits.

Variable selection consistency of the SDA estimator was studied by Mai et al. (2012). Let C = Var(X) denote the marginal covariance matrix. Under the assumption that ${\mid \mid \mid {\mathbf{C}}_{NT}{\mathbf{C}}_{TT}^{-1}\mid \mid \mid }_{\infty },{\mid \mid \mid {\mathbf{C}}_{TT}^{-1}\mid \mid \mid }_{\infty }$ and ||μ||_∞ are bounded, Mai et al. (2012) show that the following conditions

$$i)\underset{n\to \infty }{lim}\frac{s^{2}logp}{n}=0,\text{and}ii){\beta }_{min}\gg \sqrt{\frac{s^{2}log(ps)}{n}}$$

are sufficient for consistent support recovery of β. This is suboptimal compared to our results. Inspection of the proof given in Mai et al. (2012) reveals that their result hinges on uniform control of the elementwise deviation of Ĉ from C and μ̂ from μ. These uniform deviation controls are too rough to establish sharp results given in Theorem 4. In our proofs, we use more sophisticated multivariate analysis tools to control the deviation of $\widehat{{\mathit{\beta }}_T}$ from β_T, that is, we focus on analyzing the quantity ${\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T$ but instead of studying S_TT and μ̂_T separately.

The optimization problem in (1.4) uses a particular scheme to encode class labels in the vector z, though other choices are possible as well. For example, suppose that we choose z_i = z⁽¹⁾ if y_i = 1 and z_i = z⁽²⁾ if y_i = 2, with z⁽¹⁾ and z⁽²⁾ such that n₁z⁽¹⁾ + n₂z⁽²⁾ = 0. The optimality conditions for the vector $\stackrel{\sim }{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_{\stackrel{\sim }{\mathbf{T}}}^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ to be a solution to (1.4) with the alternative coding are

$$\left({\mathbf{S}}_{\stackrel{\sim }{T}\stackrel{\sim }{T}}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{T}}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{T}}^{\prime }\right){\stackrel{\sim }{\mathbf{w}}}_{\stackrel{\sim }{T}}=\frac{n_1{z}^{(1)}}{n-2}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{T}}-\stackrel{\sim }{\lambda }\text{sign}({\stackrel{\sim }{\mathbf{w}}}_{\stackrel{\sim }{T}})$$ (2.14)

$${\Vert \left({\mathbf{S}}_{\stackrel{\sim }{N}\stackrel{\sim }{T}}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{N}}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{T}}^{\prime }\right){\stackrel{\sim }{\mathbf{w}}}_{\stackrel{\sim }{T}}-\frac{n_1{z}^{(1)}}{n-2}{\widehat{\mathit{\mu }}}_{\stackrel{\sim }{N}}\Vert }_{\infty }\le \stackrel{\sim }{\lambda }.$$ (2.15)

Now, choosing $\stackrel{\sim }{\lambda }=\frac{z^{(1)}n}{n_2}\lambda$, we obtain that w̃_T̃, which satisfies (2.14) and (2.15), is proportional to ŵ_T with T̃ = T. Therefore, the choice of different coding schemes of the response variable z_i does not effect the result.

The proof of Theorem 4 is outlined in the next subsection.

2.3 Proof of Sparsistency of the SDA Estimator

The proof of Theorem 4 follows the same strategy as the proof of Theorem 1. More specifically, we only need to show that there exists a subdifferential of ||·||₁ such that the solution v̂ to the optimization problem in (1.4) satisfies the sample version KKT condition with high probability. For this, we proceed in two steps. In the first step, we assume that the true support set T is known and solve an oracle optimization problem to get ṽ_T which exploits the knowledge of T. In the second step, we show that there exists a dual variable from the subdifferential of ||·||₁ such that the vector ${({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$, paired with ${({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$, satisfies the KKT conditions for the original optimization problem given in (1.4). This proves that $\widehat{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is a global minimizer of the problem in (1.4). Finally, we show that v̂ is a unique solution to the optimization problem in (1.4) with high probability.

Let T̂ = supp(v̂) be the support of a solution v̂ to the optimization problem in (1.4) and T̂ = [p]\T̂. Any solution to (1.4) needs to satisfy the following Karush-Kuhn-Tucker (KKT) conditions

$$\left({\mathbf{S}}_{\widehat{T}\widehat{T}}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\widehat{T}}{\widehat{\mathit{\mu }}}_{\widehat{T}}^{\prime }\right){\widehat{\mathbf{v}}}_{\widehat{T}}=\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\widehat{T}}-\lambda \text{sign}({\widehat{\mathbf{v}}}_{\widehat{T}}),$$ (2.16)

$${\Vert \left({\mathbf{S}}_{\widehat{N}\widehat{T}}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\widehat{N}}{\widehat{\mathit{\mu }}}_{\widehat{T}}^{\prime }\right){\widehat{\mathbf{v}}}_{\widehat{T}}-\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_{\widehat{N}}\Vert }_{\infty }\le \lambda .$$ (2.17)

We construct a solution $\widehat{\mathbf{v}}={({\widehat{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ to (1.4) and show that it is unique with high probability.

First, we consider the following oracle optimization problem

$${\stackrel{\sim }{\mathbf{v}}}_T=arg\underset{\mathbf{v}\in {ℝ}^s}{min}\frac{1}{2(n-2)}\sum _{i\in [n]}{(z_i-{\mathbf{v}}^{\prime }({\mathbf{x}}_{i,T}-{\overline{\mathbf{x}}}_T))}^2+\lambda {\mathbf{v}}^{\prime }\text{sign}({\mathit{\beta }}_T).$$ (2.18)

The optimization problem in (2.18) is related to the one in (1.4), however, the solution is calculated only over the subset T and ||v_T ||₁ is replaced with ${\mathbf{v}}_T^{\prime }\text{sign}({\mathit{\beta }}_T)$. The solution can be computed in a closed form as

$$\begin{array}{l}{\stackrel{\sim }{\mathbf{v}}}_T={\left({\mathbf{S}}_{TT}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T{\widehat{\mathit{\mu }}}_T^{\prime }\right)}^{-1}\left(\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T-\lambda \text{sign}({\mathit{\beta }}_T)\right)\\ =\left({\mathbf{S}}_{TT}^{-1}-\frac{n_1{n}_2}{n(n-2)}\frac{{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}}{1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\right)\left(\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T-\lambda \text{sign}({\mathit{\beta }}_T)\right)\\ =\frac{n_1{n}_2}{n(n-2)}\frac{1+\lambda {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T).\end{array}$$ (2.19)

The solution ṽ_T is unique, since the matrix S_TT is positive definite with probability 1.

The following result establishes that the solution to the auxiliary oracle optimization problem (2.18) satisfies sign(ṽ_T) = sign(β_T) with high probability, under the conditions of Theorem 4.

Lemma 5

Under the assumption that the conditions of Theorem 4 are satisfied, sign(ṽ_T) = sign(β_T) and $\text{sign}(\widehat{{\mathit{\beta }}_T})=\text{sign}({\mathit{\beta }}_T)$ with probability at least 1 − (log⁻¹(n)).

The proof Lemma 5 relies on a careful characterization of the deviation of the following quantities ${\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T,{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}(\widehat{{\mathit{\beta }}_T}),{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T$ and ${\mathbf{S}}_{TT}^{-1}\text{sign}(\widehat{{\mathit{\beta }}_T})$ from their expected values.

Using Lemma 5, we have that ṽ_T defined in (2.19) satisfies ṽ_T = v̂_T. Next, we show that $\widehat{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is a solution to (1.4) under the conditions of Theorem 4.

Lemma 6

Assuming that the conditions of Theorem 4 are satisfied, we have that $\widehat{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is a solution to (1.4) with probability at least 1 − (log⁻¹(n)).

The proof of Theorem 4 will be complete once we show that $\widehat{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is the unique solution. We proceed as in the proof of Lemma 1 in Wainwright (2009a). Let v̌ be another solution to the optimization problem in (1.4) satisfying the KKT condition

$$\left(\mathbf{S}+\frac{n_1{n}_2}{n(n-2)}\widehat{\mathit{\mu }}{\widehat{\mathit{\mu }}}^{\prime }\right)\check{\mathbf{v}}-\frac{n_1{n}_2}{n(n-2)}\widehat{\mathit{\mu }}+\lambda \widehat{\mathbf{q}}=\mathbf{0}$$

for some subgradient q̂ ∈ ∂||v̌||₁. Given the subgradient q̂, any optimal solution needs to satisfy the complementary slackness condition q̂′v̌ = ||v̌||₁, which holds only if v̌_j = 0 for all j such that |q̂_j| < 1. In the proof of Lemma 6, we established that |q̂_j| < 1 for j ∈ N. Therefore, any solution to (1.4) has the same sparsity pattern as v̂. Uniqueness now follows since ṽ_T is the unique solution of (2.18) when constrained on the support set T.

3 Lower Bound

Theorem 4 provides sufficient conditions for the SDA estimator to reliably recover the true set T of nonzero elements of the discriminant direction β. In this section, we provide results that are of complementary nature. More specifically, we provide necessary conditions that must be satisfied for any procedure to succeed in reliable estimation of the support set T. Thus, we focus on the information theoretic limits in the context of high-dimensional discriminant analysis.

We denote Ψ to be an estimator of the support set T, that is, any measurable function that maps the data {x_i, y_i}_i∈[n] to a subset of {1, …, p}. Let θ = (μ₁, μ₂, Σ) be the problem parameters and Θ be the parameter space. We define the maximum risk, corresponding to the 0/1 loss, as

$$R(\mathrm{\Psi },\mathrm{\Theta })=\underset{\mathit{\theta }\in \mathrm{\Theta }}{sup}{\mathbb{P}}_{\mathit{\theta }}[\mathrm{\Psi }({\{{\mathbf{x}}_i,y_i\}}_{i\in [n]})\ne T(\mathit{\theta })]$$

where denotes the joint distribution of {x_i, y_i}_i∈[n] under the assumption that ${\pi }_1={\pi }_2=\frac{1}{2}$, and T(θ) = supp(β) (recall that β = Σ⁻¹ (μ₂ − μ₁)). Let (s, ) be the class of all subsets of the set of cardinality s. We consider the parameter space

$$\mathrm{\Theta }(\mathbf{\sum },\tau ,s)=\underset{\omega \in ℳ(s,[p])}{\cup }\left\{\mathit{\theta }=({\mathit{\mu }}_1,{\mathit{\mu }}_2,\mathbf{\sum }):\begin{array}{l}\mathit{\beta }={\mathbf{\sum }}^{-1}({\mathit{\mu }}_2-{\mathit{\mu }}_1),\hfill \\ \mid {\beta }_a\mid \ge \tau \text{if}a\in \omega ,{\beta }_a=0\text{if}a\notin \omega \hfill \end{array}\right\},$$ (3.1)

where τ > 0 determines the signal strength. The minimax risk is defined as

$$\underset{\mathrm{\Psi }}{inf}R(\mathrm{\Psi },\mathrm{\Theta }(\mathbf{\sum },\tau ,s)).$$

In what follows we provide a lower bound on the minimax risk. Before stating the result, we introduce the following three quantities that will be used to state Theorem 7

$${\phi }_{\text{close}}(\mathbf{\sum })=\underset{T\in ℳ(s,[p])}{min}\underset{u\in T}{min}\frac{1}{p-s}\sum _{v\in [p]\backslash T}({\mathrm{\sum }}_{uu}+{\mathrm{\sum }}_{vv}-2{\mathrm{\sum }}_{uv}),$$ (3.2)

$${\phi }_{\text{far}}(\mathbf{\sum })=\underset{T\in ℳ(s,[p])}{min}\frac{1}{\left(\begin{array}{c}p-s\\ s\end{array}\right)}\sum _{T^{\prime }\in ℳ(s,[p]\backslash T)}{\mathbf{1}}^{\prime }{\mathbf{\sum }}_{T\cup T^{\prime },T\cup T^{\prime }}\mathbf{1},$$ (3.3)

and

$${\tau }_{min}=2·max\left(\sqrt{\frac{log\left(\begin{array}{c}p-s\\ s\end{array}\right)}{n{\phi }_{\text{far}}(\mathbf{\sum })}},\sqrt{\frac{log(p-s+1)}{n{\phi }_{\text{close}}(\mathbf{\sum })}}\right).$$ (3.4)

The first quantity measures the difficulty of distinguishing two close support sets T₁ and T₂ that differ in only one position. The second quantity measures the effect of a large number of support sets that are far from the support set T. The quantity τ_min is a threshold for the signal strength. Our main result on minimax lower bound is presented in Theorem 7.

Theorem 7

For any τ < τ_min, there exists some constant C > 0, such that

$$\underset{\mathrm{\Psi }}{inf}\underset{\mathit{\theta }\in \mathrm{\Theta }(\mathbf{\sum },\tau ,s)}{sup}{\mathbb{P}}_{\mathit{\theta }}[\mathrm{\Psi }({\{{\mathbf{x}}_i,y_i\}}_{i\in [n]})\ne T(\mathit{\theta })]\ge C>0.$$

Theorem 7 implies that for any estimating procedure, whenever τ < τ_min, there exists some distribution parametrized by θ ∈ Θ(Σ, τ, s) such that the probability of incorrectly identifying the set T(θ) is strictly bounded away from zero. To better understand the quantities φ_close(Σ) and φ_far(Σ), we consider a special case when Σ = I. In this case both quantities simplify a lot and we have φ_close(I) = 2 and φ_far(I) = 2s. From Theorem 7 and Theorem 4, we see that the SDA estimator is able to recover the true support set T using the optimal number of samples (up to an absolute constant) over the parameter space

$$\mathrm{\Theta }(\mathbf{\sum },{\tau }_{min},s)\cap \{\mathit{\theta }:{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\le M\}$$

where M is a fixed constant and Λ_min(Σ_TT) is bounded from below. This result will be further illustrated by numerical simulations in §6.

4 Exhaustive Search Decoder

In this section, we analyze an exhaustive search procedure, which evaluates every subset T′ of size s and outputs the one with the best score. Even though the procedure cannot be implemented in practice, it is a useful benchmark to compare against and it provides deeper theoretical insights into the problem.

For any subset T′ ⊂ [p], we define

$$f(T^{\prime })=\underset{\mathbf{u}\in {ℝ}^{\mid T^{\prime }\mid }}{min}\{{\mathbf{u}}^{\prime }{\widehat{\mathbf{S}}}_{T^{\prime }{T}^{\prime }}\mathbf{u}:{\mathbf{u}}^{\prime }{\widehat{\mathit{\mu }}}_{T^{\prime }}=1\}=\underset{T^{\prime }\subset [p]:\mid T^{\prime }\mid =s}{min}\frac{1}{{\widehat{\mathit{\mu }}}_{T^{\prime }}^{\prime }{\mathbf{S}}_{T^{\prime }{T}^{\prime }}^{-1}{\widehat{\mathit{\mu }}}_{T^{\prime }}^{\prime }}.$$

The exhaustive search procedure outputs the support set T̂ that minimizes f(T′) over all subsets T′ of size s,

$$\widehat{T}=\underset{T^{\prime }\subset [p]:\mid T^{\prime }\mid =s}{argmin}f(T^{\prime })=\underset{T^{\prime }\subset [p]:\mid T^{\prime }\mid =s}{argmax}{\widehat{\mathit{\mu }}}_{T^{\prime }}^{\prime }{\mathbf{S}}_{T^{\prime }{T}^{\prime }}^{-1}{\widehat{\mathit{\mu }}}_{T^{\prime }}.$$

Define $g(T^{\prime })={\widehat{\mathit{\mu }}}_{T^{\prime }}^{\prime }{\mathbf{S}}_{T^{\prime }{T}^{\prime }}^{-1}{\widehat{\mathit{\mu }}}_{T^{\prime }}$. In order to show that the exhaustive search procedure identifies the correct support set T, we need to show that with high probability g(T) > g(T′) for any other set T′ of size s. The next result gives sufficient conditions for this to happen. We first introduce some additional notation. Let A₁ = T ∩ T′, A₂ = T\T′ and A₃ = T′\T. We define the following quantities

$$\begin{array}{l}{a}_1(T^{\prime })={\mathit{\mu }}_{A_1}^{\prime }{\mathbf{\sum }}_{A_1{A}_1}^{-1}{\mathit{\mu }}_{A_1},\\ a_2(T^{\prime })={\mathit{\mu }}_{A_2\mid A_1}^{\prime }{\mathbf{\sum }}_{A_2{A}_2\mid A_1}^{-1}{\mathit{\mu }}_{A_2\mid A_1},\\ a_3(T^{\prime })={\mathit{\mu }}_{A_3\mid A_1}^{\prime }{\mathbf{\sum }}_{A_3{A}_3\mid A_1}^{-1}{\mathit{\mu }}_{A_3\mid A_1},\end{array}$$

where ${\mathit{\mu }}_{A_2\mid A_1}={\mathit{\mu }}_{A_2}-{\mathbf{\sum }}_{A_2{A}_1}{\mathbf{\sum }}_{A_1{A}_1}^{-1}{\mathit{\mu }}_{A_1}$ and ${\mathbf{\sum }}_{A_2{A}_2\mid A_1}={\mathbf{\sum }}_{A_2{A}_2}-{\mathbf{\sum }}_{A_2{A}_1}{\mathbf{\sum }}_{A_1{A}_1}^{-1}{\mathbf{\sum }}_{A_2{A}_1}$. The quantities μA₃|A₁ and ΣA₃A₃|A₁are defined similarly.

Theorem 8

Assuming that for all T′ ⊆ [p] with |T′| = s and T′ ≠ T the following holds

$$a_2(T^{\prime })-\left(1+C_1\sqrt{{\mathrm{\Gamma }}_{n,p,s,k}}\right)a_3(T^{\prime })\ge C_2\sqrt{(1\vee a_1(T^{\prime }))a_2(T^{\prime }){\mathrm{\Gamma }}_{n,p,s,k}}+C_3(1\vee a_1(T^{\prime })){\mathrm{\Gamma }}_{n,p,s,k},$$ (4.1)

where |T′ ∩ T| = k, ${\mathrm{\Gamma }}_{n,p,s,k}=n^{-1}log\left(\left(\begin{array}{c}p-s\\ s-k\end{array}\right)\left(\begin{array}{c}s\\ k\end{array}\right)slog(n)\right)$ and C₁, C₂, C₃ are constants independent of the problem parameters, we have [T̂ ≠ T] = (log⁻¹(n)).

The condition in (4.1) allows the exhaustive search decoder to distinguish between the sets T and T′ with high probability. Note that the Mahalanobis distance decomposes as $g(T)={\widehat{\mathit{\mu }}}_{A_1}^{\prime }{\mathbf{S}}_{A_1{A}_1}^{-1}{\widehat{\mathit{\mu }}}_{A_1}+{\stackrel{\sim }{\mathit{\mu }}}_{A_2\mid A_1}^{\prime }{\mathbf{S}}_{A_2{A}_2\mid A_1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{A_2\mid A_1}$ where ${\stackrel{\sim }{\mathit{\mu }}}_{A_2\mid A_1}={\widehat{\mathit{\mu }}}_{A_2}-{\mathbf{S}}_{A_2{A}_1}{\mathbf{S}}_{A_1{A}_1}^{-1}{\widehat{\mathit{\mu }}}_{A_1}$ and ${\mathbf{S}}_{A_2{A}_2\mid A_1}={\mathbf{S}}_{A_2{A}_2}-{\mathbf{S}}_{A_2{A}_1}{\mathbf{S}}_{A_1{A}_1}^{-1}{\mathbf{S}}_{A_1{A}_2}$, and similarly $g(T^{\prime })={\widehat{\mathit{\mu }}}_{A_1}^{\prime }{\mathbf{S}}_{A_1{A}_1}^{-1}{\widehat{\mathit{\mu }}}_{A_1}+{\stackrel{\sim }{\mathit{\mu }}}_{A_3\mid A_1}^{\prime }{\mathbf{S}}_{A_3{A}_3\mid A_1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{A_3\mid A_1}$. Therefore g(T) > g(T′) if ${\stackrel{\sim }{\mathit{\mu }}}_{A_2\mid A_1}^{\prime }{\mathbf{S}}_{A_2{A}_2\mid A_1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{A_2\mid A_1}>{\stackrel{\sim }{\mathit{\mu }}}_{A_3\mid A_1}^{\prime }{\mathbf{S}}_{A_3{A}_3\mid A_1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{A_3\mid A_1}$. With infinite amount of data, it would be sufficient that a₂(T′) > a₃(T′). However, in the finite-sample setting, condition (4.1) ensures that the separation is big enough. If X_T and X_N are independent, then the expression (4.1) can be simplified by dropping the second term on the left hand side.

Compared to the result of Theorem 4, the exhaustive search procedure does not require the covariance matrix to satisfy the irrepresentable condition given in (2.4).

5 Implications of Our Results

In this section, we give some implications of our results. We start with the case when the covariance matrix Σ = I. The same implications hold for other covariance matrices that satisfy Λ_min(Σ) ≥ C > 0 for some constant C independent of (n, p, s). We first illustrate a regime where the SDA estimator is optimal for the problem of identifying the relevant variables. This is done by comparing the results in Theorem 4 to those of Theorem 7. Next, we point out a regime where there exists a gap between the sufficient and necessary conditions of Theorem 7 for both the exhaustive search decoder and the SDA estimator. Throughout the section, we assume that s = o(min(n, p)).

When Σ = I, we have that β_T = μ_T. Let

$${\mu }_{min}=\underset{a\in T}{min}\mid {\mathit{\mu }}_T\mid .$$

Theorem 7 gives a lower bound on μ_min as

$${\mu }_{min}\gtrsim \sqrt{\frac{log(p-s)}{n}}.$$

If some components of the vector μ_T are smaller in absolute value than μ_min, no procedure can reliably recover the support. We will compare this bound with sufficient conditions given in Theorems 4 and 8.

First, we assume that ${\Vert {\mathit{\mu }}_T\Vert }_2^2=C$ for some constant C. Theorem 4 gives that ${\mu }_{min}\gtrsim \sqrt{\frac{log(p-s)}{n}}$ is sufficient for the SDA estimator to consistently recover the relevant variables when n ≳ s log(p − s). This effectively gives μ_min ≳ s^−1/2, which is the same as the necessary condition of Theorem 7.

Next, we investigate the condition in (4.1), which is sufficient for the exhaustive search procedure to identify the set T. Let T′ ⊂ [p] be a subset of size s. Then, using the notation of Section 4,

$$a_1(T^{\prime })={\Vert {\mathit{\mu }}_{A_1}\Vert }_2^2,a_2(T^{\prime })={\Vert {\mathit{\mu }}_{A_2}\Vert }_2^2,\text{and}{a}_3(T^{\prime })=0.$$

Now, if |T′ ∩ T| = s − 1 and T′ does not contain a smallest component of μ_T, (4.1) simplifies to ${\mu }_{min}\gtrsim \sqrt{\frac{log(p-s)}{n}}$, since ${\Vert {\mathit{\mu }}_{A_1}\Vert }_2^2\le {\Vert {\mathit{\mu }}_T\Vert }_2^2=C$. This shows that both the SDA estimator and the exhaustive search procedure can reliably detect signals at the information theoretic limit in the case when the norm of the vector μ_T is bounded and μ_min ≳ s^−1/2. However, when the norm of the vector μ_T is not bounded by a constant, for example, μ_min = C′ for some constant C′, Theorem 7 gives that at least n ≳ log(p − s) data points are needed, while n ≳ s log(p − s) is sufficient for correct recovery of the support set T. This situation is analogous to the known bounds on the support recovery in the sparse linear regression setting (Wainwright, 2009b).

Next, we show that the largest eigenvalue of a covariance matrix Σ can diverge, without affecting the sample size required for successful recovery of the support set T. Let $\mathbf{\sum }=(1-\gamma ){\mathbf{I}}_p+\gamma {\mathbf{1}}_p{\mathbf{1}}_p^{\prime }$ for γ ∈ [0, 1). We have Λ_max(Σ) = 1 + (p − 1)γ, which diverges to infinity for any fixed γ as p → ∞. Let T = [s] and set β_T = β1_T. This gives μ_T = β(1 + γ(s − 1))1_T and μ_N = γβs1_N. A simple application of the matrix inversion formula gives

$${\mathbf{\sum }}_{TT}^{-1}={(1-\gamma )}^{-1}{\mathbf{I}}_s-\frac{\gamma }{(1-\gamma )(1+\gamma (s-1))}{\mathbf{1}}_T{\mathbf{1}}_T^{\prime }.$$

A lower bound on β is obtained from Theorem 7 as $\beta \ge \sqrt{\frac{2}{1-\gamma }\frac{log(p-s)}{n}}$. This follows from a simple calculation that establishes φ_close(Σ) = 2(1 − γ) and φ_far(Σ) = 2s(1 − γ) + (2s)²γ.

Sufficient conditions for the SDA estimator follow from Theorem 4. A straightforward calculation shows that

$${\sigma }_{a\mid T}=\frac{(1-\gamma )(1+\gamma s)}{1+\gamma (s-1)},{\mathrm{\Lambda }}_{min}(\mathbf{\sum })=1-\gamma ,{\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }=\frac{1}{1+\gamma (s-1)}.$$

This gives that $\beta \ge K\sqrt{\frac{log(p-s)}{(1-\gamma )n}}$ (for K large enough) is sufficient for recovering the set T, assuming that ${\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2=\mathcal{O}(1)$. This matches the lower bound, showing that the maximum eigenvalue of the covariance matrix Σ does not play a role in characterizing the behavior of the SDA estimator.

6 Simulation Results

In this section, we conduct several simulations to illustrate the finite-sample performance of our results. Theorem 4 describes the sample size needed for the SDA estimator to recover the set of relevant variables. We consider the following three scalings for the size of the set T:

1. fractional power sparsity, where s = ⌈2p^0.45⌉

2. sublinear sparsity, where s = ⌈0.4p/log(0.4p)⌉, and

3. linear sparsity, where s = ⌈0.4p⌉.

For all three scaling regimes, we set the sample size as

$$n=\theta slog(p)$$

where θ is a control parameter that is varied. We investigate how well can the SDA estimator recovers the true support set T as the control parameter θ varies.

We set $\mathbb{P}[Y=1]=\mathbb{P}[Y=2]=\frac{1}{2}$, X|Y = 1 ~ (μ, Σ) and without loss of generality X|Y = 2 ~ (0, Σ). We specify the vector μ by choosing the set T of size |T| = s randomly, and for each a ∈ T setting μ_a equal to +1 or −1 with equal probability, and μ_a = 0 for all components a ∉ T. We specify the covariance matrix Σ as

$$\mathbf{\sum }=\left(\begin{array}{cc}{\mathbf{\sum }}_{TT}& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{p-s}\end{array}\right)$$

so that $\mathit{\beta }={\mathbf{\sum }}^{-1}\mathit{\mu }={({\mathit{\beta }}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$. We consider three cases for the block component Σ_TT:

1. identity matrix, where Σ_TT = I_s,

2. Toeplitz matrix, where Σ_TT = [Σ_ab]_a,b∈T and Σ_ab = ρ^|a−b| with ρ = 0.1, and

3. equal correlation matrix, where Σ_ab = ρ when a ≠ b and σ_aa = 1.

Finally, we set the penalty parameter λ = λ_SDA as

$${\lambda }_{\text{SDA}}=0.3\times {(1+{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2/4)}^{-1}\sqrt{\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log(p-s)}{n}}$$

for all cases. We also tried several different constants and found that our main results on high dimensional scalings are insensitive to the choice of this constant. For this choice of λ, Theorem 4 predicts that the set T will be recovered correctly. For each setting, we report the Hamming distance between the estimated set T̂ and the true set T,

$$h(\widehat{T},T)=\mid (\widehat{T}\backslash T)\cup (T\backslash \widehat{T})\mid ,$$

averaged over 200 independent simulation runs.

Figure 1 plots the Hamming distance against the control parameter θ, or the rescaled number of samples. Here the Hamming distance between T̂ and T is calculated by averaging 200 independent simulation runs. There are three subfigures corresponding to different sparsity regimes (fractional power, sublinear and linear sparsity), each of them containing three curves for different problem sizes p ∈ {100, 200, 300}. Vertical line indicates a threshold parameter θ at which the set T is correctly recovered. If the parameter is smaller than the threshold value, the recovery is poor. Figure 2 and Figure 3 show results for two other cases, with Σ_TT being a Toeplitz matrix with parameter ρ = 0.1 and the equal correlation matrix with ρ = 0.1. To illustrate the effect of correlation, we set p = 100 and generate the equal correlation matrices with ρ ∈ {0, 0.1, 0.3, 0.5, 0.7, 0.9}. Results are given in Figure 4.

Figure 1.

(The SDA Estimator) Plots of the rescaled sample size n/(s log(p)) versus the Hamming distance between T̂ and T for identity covariance matrix Σ = I_p (averaged over 200 simulation runs). Each subfigure shows three curves, corresponding to the problem sizes p ∈ {100, 200, 300}. The first subfigure corresponds to the fractional power sparsity regime, s = 2p^0.45, the second subfigure corresponds to the sublinear sparsity regime s = 0.4p/log(0.4p), and the third ssubfigure corresponds to the linear sparsity regime s = 0.4p. Vertical lines denote a scaled sample size at which the support set T is recovered correctly.

Figure 2.

(The SDA Estimator) Plots of the rescaled sample size n/(s log(p)) versus the Hamming distance between T̂ and T for the Toeplitz covariance matrix Σ_TT with ρ = 0.1 (averaged over 200 simulation runs). Each subfigure shows three curves, corresponding to the problem sizes p ∈ {100, 200, 300}. The first subfigure corresponds to the fractional power sparsity regime, s = 2p^0.45, the second subfigure corresponds to the sublinear sparsity regime s = 0.4p/log(0.4p), and the third subfiguren corresponds to the linear sparsity regime s = 0.4p. Vertical lines denote a scaled sample size at which the support set T is recovered correctly.

Figure 3.

(The SDA Estimator) Plots of the rescaled sample size n/(s log(p)) versus the Hamming distance between T̂ and T for equal correlation matrix Σ_TT with ρ = 0.1 (averaged over 200 simulation runs). Each subfigure shows three curves, corresponding to the problem sizes p ∈ {100, 200, 300}. The first subfigure corresponds to the fractional power sparsity regime, s = 2p^0.45, the second subfigure corresponds to the sublinear sparsity regime s = 0.4p/log(0.4p), and the third subfigure corresponds to the linear sparsity regime s = 0.4p. Vertical lines denote a scaled sample size at which the support set T is recovered correctly.

Figure 4.

(The SDA Estimator) Plots of the rescaled sample size n/(s log(p)) versus the Hamming distance between T̂ and T for equal correlation matrix Σ_TT with ρ ∈ {0, 0.1, 0.3, 0.5, 0.7, 0.9} (averaged over 200 simulation runs). The ambient dimension is set as p = 100. The first subfigure corresponds to the fractional power sparsity regime, s = 2p^0.45 and the second subfigure corresponds to the sublinear sparsity regime s = 0.4p/log(0.4p).

7 Discussion

In this paper, we address the problem of variable selection in high-dimensional discriminant analysis problem. The problem of reliable variable selection is important in many scientific areas where simple models are needed to provide insights into complex systems. Existing research has focused primarily on establishing results for prediction consistency, ignoring feature selection. We bridge this gap, by analyzing the variable selection performance of the SDA estimator and an exhaustive search decoder. We establish sufficient conditions required for successful recovery of the set of relevant variables for these procedures. This analysis is complemented by analyzing the information theoretic limits, which provide necessary conditions for variable selection in discriminant analysis. From these results, we are able to identify the class of problems for which the computationally tractable procedures are optimal. In this section, we discuss some implications and possible extensions of our results.

7.1 Theoretical Justification of the ROAD and Sparse Optimal Scaling Estimators

In a recent work, Mai and Zou (2012) show that the SDA estimator is numerically equivalent to the ROAD estimator proposed by Wu et al. (2009), Fan et al. (2012) and the sparse optimal scaling estimator proposed by Clemmensen et al. (2011). More specifically, all these three methods have the same regularization paths up to a constant scaling. This result allows us to apply the theoretical results in this paper to simultaneously justify the optimal variable selection performance of the ROAD and sparse optimal scaling estimators.

7.2 Risk Consistency

The results of Theorem 4 can be used to establish risk consistency of the SDA estimator. Consider the following classification rule

$$\widehat{y}(\mathbf{x})=\{\begin{array}{ll}1\hfill & \text{if}g(\mathbf{x};\widehat{\mathbf{v}})=1\hfill \\ 2\hfill & \text{otherwise}\hfill \end{array}$$ (7.1)

where g(x; v̂) = I[v̂′ (x − (μ̂₁ + μ̂₂)/2) > 0] with v̂ = v̂^SDA. Under the assumption that β = (β_T′, 0′)′, the risk (or the error rate) of the Bayes rule defined in (1.1) is $R_{\text{opt}}=\mathrm{\Phi }\left(-\sqrt{{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T}/2\right)$, where Φ is the cumulative distribution function of a standard Normal distribution. We will compare the risk of the SDA estimator against this Bayes risk.

Recall the setting introduced in §1.1, conditioning on the data points {x_i, y_i}_i∈[n], the conditional error rate is

$$R(\widehat{\mathbf{w}})=\frac{1}{2}\sum _{i\in \{1,2\}}\mathrm{\Phi }\left(\frac{-{\widehat{\mathbf{v}}}^{\prime }({\mathit{\mu }}_i-{\widehat{\mathit{\mu }}}_i)-{\widehat{\mathbf{v}}}^{\prime }\widehat{\mathit{\mu }}/2}{\sqrt{{\widehat{\mathbf{v}}}^{\prime }\mathbf{\sum }\widehat{\mathbf{v}}}}\right).$$ (7.2)

Let r_n = λ||β_T||₁ and q_n = sign(β_T)′Σ_TT sign(β_T). We have the following result on risk consistency.

Corollary 9

Let v̂ = v̂^SDA. We assume that the conditions of Theorem 4 hold with

$$n\asymp K(n)\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right){\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})slog((p-s)log(n)),$$ (7.3)

where K(n) could potentially scale with n, and ${\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\ge C>0$. Furthermore, we assume that $r_n\stackrel{n\to \infty }{\to }0$. Then

$$R(\widehat{\mathbf{w}})=\mathrm{\Phi }\left(-\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}}{2}\frac{(1+{\mathcal{O}}_P(r_n))}{\sqrt{1+{\mathcal{O}}_P\left(r_n\vee \frac{{\lambda }_0^2{q}_n}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\right)}}\right).$$

First, note that ${\Vert {\mathit{\beta }}_T\Vert }_1/{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}=o\left(\sqrt{K(n)s/{\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT})}\right)$ is sufficient for $r_n\stackrel{n\to \infty }{\to }0$. Under the conditions of Theorem 9, we have that ${\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2/({\lambda }_0^2{q}_n)=\mathcal{O}(K(n)s/({\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT})q_n))=\mathcal{O}(K(n))$. Therefore, if $K(n)\stackrel{n\to \infty }{\to }\infty$ and $K(n)\ge Cs{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2/({\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT}){\Vert {\mathit{\beta }}_T\Vert }_1^2)$ we have

$$R(\widehat{\mathbf{w}})=\mathrm{\Phi }\left(-\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}}{2}(1+{\mathcal{O}}_P(r_n))\right)$$

and R(ŵ) − R_opt →_P 0. If in addition

$${\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}{\Vert {\mathit{\beta }}_T\Vert }_1=o\left(\sqrt{K(n)s/{\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT})}\right),$$

then R(ŵ)/R_opt →_P 1, using Lemma 1 in Shao et al. (2011).

The above discussion shows that the conditions of Theorem 4 are sufficient for establishing risk consistency. We conjecture that substantially less restrictive conditions are needed to establish risk consistency results. Exploring such weaker conditions is beyond the scope of this paper.

7.3 Approximate sparsity

Thus far, we were discussing estimation of discriminant directions that are exactly sparse. However, in many applications it may be the case that the discriminant direction $\mathit{\beta }={({{\mathit{\beta }}_T}^{\prime },{\mathit{\beta }}_N^{\prime })}^{\prime }={\mathbf{\sum }}^{-1}\mathit{\mu }$ is only approximately sparse, that is, β_N is not equal to zero, but is small. In this section, we briefly discuss the issue of variable selection in this context.

In the approximately sparse setting, since β_N ≠ 0, a simple calculation gives

$${\mathit{\beta }}_T={\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T-{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{TN}{\mathit{\beta }}_N$$ (7.4)

and

$${\mathit{\mu }}_N={\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T+({\mathbf{\sum }}_{NN}-{\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{TN}){\mathit{\beta }}_N.$$ (7.5)

In what follows, we provide conditions under which the solution to the population version of the SDA estimator, given in (2.1), correctly recovers the support of large entries T. Let $\widehat{\mathbf{w}}={({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ where ŵ_T is given as

$${\widehat{\mathbf{w}}}_T={\pi }_1{\pi }_2\frac{1+\lambda {\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_{{\mathbf{\sum }}_{TT}}^2}\stackrel{\sim }{{\mathit{\beta }}_T}-\lambda {\mathbf{\sum }}_{TT}^{-1}\text{sign}(\stackrel{\sim }{{\mathit{\beta }}_T})$$

with $\stackrel{\sim }{{\mathit{\beta }}_T}={\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T$. We will show that ŵ is the solution to (2.1).

We again define β_min = min_a∈T|β_a|. Following a similar argument as the proof of Theorem 1, we have that $\text{sign}({\widehat{\mathbf{w}}}_T)=\text{sign}\left(\stackrel{\sim }{{\mathit{\beta }}_T}\right)$ holds if $\stackrel{\sim }{{\mathit{\beta }}_T}$ satisfies

$${\pi }_1{\pi }_2\frac{1+\lambda {\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_{{\mathbf{\sum }}_{TT}}^2}{\beta }_{min}>\lambda {\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}\left(\stackrel{\sim }{{\mathit{\beta }}_T}\right)\Vert }_{\infty }.$$ (7.6)

In the approximate sparsity setting, it is reasonable to assume that ${\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{TN}{\mathit{\beta }}_N$ is small compared to $\stackrel{\sim }{{\mathit{\beta }}_T}$, which would imply that $\text{sign}({\mathit{\beta }}_T)=\text{sign}\left(\stackrel{\sim }{{\mathit{\beta }}_T}\right)$ using (7.4). Therefore, under suitable assumptions we have sign(ŵ_T) = sign(β_T). Next, we need conditions under which ŵ is the solution to (2.1).

Following a similar analysis as in Lemma 2, the optimality condition

$${\Vert ({\mathbf{\sum }}_{NT}+{\pi }_1{\pi }_2{\mathit{\mu }}_N{\mathit{\mu }}_T^{\prime }){\widehat{\mathbf{w}}}_T-{\pi }_1{\pi }_2{\mathit{\mu }}_N\Vert }_{\infty }\le \lambda$$

needs to hold. Let $\widehat{\gamma }=\frac{1+\lambda {\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert \stackrel{\sim }{{\mathit{\beta }}_T}\Vert }_{{\mathbf{\sum }}_{TT}}^2}$. Using (7.5), the above display becomes

$${\Vert -\lambda {\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\pi }_1{\pi }_2\widehat{\gamma }({\mathbf{\sum }}_{NN}-{\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{TN}){\mathit{\beta }}_N\Vert }_{\infty }<\lambda .$$

Therefore, using the triangle inequality, the following assumption

$${\pi }_1{\pi }_2\widehat{\gamma }·{\Vert ({\mathbf{\sum }}_{NN}-{\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{TN}){\mathit{\beta }}_N\Vert }_{\infty }<\alpha \lambda ,$$

in addition to (2.4) and (7.6), is sufficient for ŵ to recover the set of important variables T.

The above discussion could be made more precise and extended to the sample SDA estimator in (1.4), by following the proof of Theorem 4. This is beyond the scope of the current paper and will be left as a future investigation.

A Proofs Of Main Results

In this section, we collect proofs of results given in the main text. We will use C, C₁, C₂, … to denote generic constants that do not depend on problem parameters. Their values may change from line to line.

Let

$$\mathcal{A}={ℰ}_n\cap {ℰ}_1({log}^{-1}(n))\cap {ℰ}_2({log}^{-1}(n))\cap {ℰ}_3({log}^{-1}(n))\cap {ℰ}_4({log}^{-1}(n)),$$ (A.1)

where is defined in (C.1), in Lemma 10, in Lemma 11, in (C.8), and in (C.9). We have that [ ] ≥ 1 − log⁻¹(n)).

A.1 Proofs of Results in Section 2

Proof of Lemma 2

From the KKT conditions given in (2.7), we have that $\widehat{\mathbf{w}}={({\widehat{\mathbf{w}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ is a solution to the problem in (2.1) if and only if

$$\begin{array}{l}({\mathbf{\sum }}_{TT}+{\pi }_1{\pi }_2{\mathit{\mu }}_T{\mathit{\mu }}_T^{\prime }){\widehat{\mathbf{w}}}_T-{\pi }_1{\pi }_2{\mathit{\mu }}_T+\lambda \text{sign}({\widehat{\mathbf{w}}}_T)=\mathbf{0}\\ {\Vert ({\mathbf{\sum }}_{NT}+{\pi }_1{\pi }_2{\mathit{\mu }}_N{\mathit{\mu }}_T^{\prime }){\widehat{\mathbf{w}}}_T-{\pi }_1{\pi }_2{\mathit{\mu }}_N\Vert }_{\infty }\le \lambda \end{array}$$ (A.2)

By construction, ŵ_T satisfy the first equation. Therefore, we need to show that the second one is also satisfied. Plugging in the explicit form of ŵ_T into the second equation and using (2.2), after some algebra we obtain that

$${\Vert {\mathbf{\sum }}_{NT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }\le 1$$

needs to be satisfied. The above display is satisfied with strict inequality under the assumption in (2.4).

Proof of Lemma 5

Throughout the proof, we will work on the event defined in (A.1).

Let a ∈ T be such that ṽ_a > 0, noting that the case when ṽ_a < 0 can be handled in a similar way. Let

$$\begin{array}{c}{\delta }_1={\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T),{\delta }_2={\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T,\\ {\delta }_3={\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T),{\delta }_4={\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2,\text{and}\\ {\delta }_5=\frac{n_1{n}_2}{n(n-2)}-{\pi }_1{\pi }_2.\end{array}$$

Furthermore, let

$$\widehat{\gamma }=\frac{n_1{n}_2}{n(n-2)}\frac{1+\lambda {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\text{and}\gamma =\frac{{\pi }_1{\pi }_2(1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1)}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}$$

For sufficiently large n, on the event , together with Lemma 12, Lemma 16, and Lemma 13, we have that γ̂ ≥ γ(1 − o(1)) > γ/2 and ${\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)={\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)(1+o(1))\le \frac{3}{2}{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)$ with probability at least 1 − (log⁻¹(n). Then

$$\begin{array}{l}{\stackrel{\sim }{v}}_a\ge \frac{\gamma }{2}({\beta }_a+{\delta }_2)-\frac{3}{2}\lambda {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ \ge \frac{{\pi }_1{\pi }_2(1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1)({\beta }_a-\mid {\delta }_2\mid )-3{\lambda }_0{\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }}{2(1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2)},\end{array}$$

so that sign(ṽ_a) = sign(β_a) if

$${\pi }_1{\pi }_2(1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1)({\beta }_a-\mid {\delta }_2\mid )-3{\lambda }_0{\Vert {\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\Vert }_{\infty }>0.$$ (A.3)

Lemma 15 gives a bound on |δ₂|, for each fixed a ∈ T, as

$$\mid {\delta }_2\mid \le C_1\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log(slog(n))}{n}}+C_2\mid {\beta }_a\mid \sqrt{\frac{log(slog(n))}{n}}.$$

Therefore assumption (2.11), with K_β sufficiently large, and a union bound over all a ∈ T implies (A.3).

Lemma 15 gives sign(β_T) = sign(β_T) with probability 1 − (log⁻¹(n)).

Proof of Lemma 6

Throughout the proof, we will work on the event defined in (A.1). By construction, the vector $\widehat{\mathbf{v}}={({\stackrel{\sim }{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ satisfies the condition in (2.16). Therefore, to show that it is a solution to (1.4), we need to show that it also satisfies (2.17).

To simplify notation, let

$$\begin{array}{r}\mathbf{C}=\mathbf{S}+\frac{n_1{n}_2}{n(n-2)}\widehat{\mathit{\mu }}{\widehat{\mathit{\mu }}}^{\prime },\widehat{\gamma }=\frac{n_1{n}_2}{n(n-2)}\frac{1+\lambda {\Vert \widehat{{\mathit{\beta }}_T}\Vert }_1}{1+\frac{n_1{n}_2}{n(n-2)}{\Vert \widehat{{\mathit{\beta }}_T}\Vert }_{{\mathbf{S}}_{TT}}^2},\\ \text{and}\gamma =\frac{{\pi }_1{\pi }_2(1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1)}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}.\end{array}$$

Recall that ${\stackrel{\sim }{\mathbf{v}}}_T=\widehat{\gamma }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)$.

Let U ∈ ℝ^(n−2)×p be a matrix with each row ${\mathbf{u}}_i\stackrel{\mathit{iid}}{~}\mathcal{N}(0,\mathbf{\sum })$ such that (n − 2)S = U′U. For a ∈ N, we have

$$(n-2){\mathbf{S}}_{aT}={({\mathbf{U}}_T{\mathbf{\sum }}_{TT}^{-1}{\mathbf{\sum }}_{Ta}+{\mathbf{U}}_{a·T})}^{\prime }{\mathbf{U}}_T={\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{U}}_T^{\prime }{\mathbf{U}}_T+{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T$$

where ${\mathbf{U}}_{a·T}~\mathcal{N}\left(0,\frac{n_1{n}_2}{n(n-2)}{\sigma }_{a\mid T}{\mathbf{I}}_{n-2}\right)$ is independent of U_T, and

$${\widehat{\mu }}_a={\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T+{\widehat{\mu }}_{a·T}$$

where ${\widehat{\mu }}_{a·T}~\mathcal{N}\left(0,\frac{n}{n_1{n}_2}{\sigma }_{a\mid T}\right)$ is independent of μ̂_T. Therefore,

$$\begin{array}{l}{\mathbf{C}}_{aT}={\mathbf{S}}_{aT}+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mu }}_a{\widehat{\mathit{\mu }}}_T^{\prime }\\ ={\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\mathbf{S}}_{TT}+{(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mu }}_a{\widehat{\mathit{\mu }}}_T^{\prime },\end{array}$$

$$\begin{array}{l}{\mathbf{C}}_{aT}{\stackrel{\sim }{\mathbf{v}}}_T=\widehat{\gamma }{\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T+\widehat{\gamma }\frac{n_1{n}_2}{n(n-2)}({\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T){\widehat{\mu }}_a\\ -\lambda \left({\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)+\frac{n_1{n}_2}{n(n-2)}{\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1·{\widehat{\mathit{\mu }}}_a\right)\\ +{(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\stackrel{\sim }{\mathbf{v}}}_T\\ =\left(\widehat{\gamma }+\widehat{\gamma }\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda \frac{n_1{n}_2}{n(n-2)}{\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1\right){\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\\ -\lambda {\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ +{(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\stackrel{\sim }{\mathbf{v}}}_T+\widehat{\gamma }\frac{n_1{n}_2}{n(n-2)}({\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T){\widehat{\mu }}_{a·T}\\ -\lambda \frac{n_1{n}_2}{n(n-2)}{\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1·{\widehat{\mu }}_{a.T}\\ =\frac{n_1{n}_2}{n(n-2)}{\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ +{(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\stackrel{\sim }{\mathbf{v}}}_T+\widehat{\gamma }\frac{n_1{n}_2}{n(n-2)}({\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T){\widehat{\mu }}_{a.T}\\ -\lambda \frac{n_1{n}_2}{n(n-2)}{\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1·{\widehat{\mu }}_{a·T},\end{array}$$

and finally

$${\mathbf{C}}_{aT}{\stackrel{\sim }{\mathbf{v}}}_T-\frac{n_1{n}_2}{n(n-2)}{\widehat{\mu }}_a=-\lambda {\mathbf{\sum }}_{aT}{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)+{(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\stackrel{\sim }{\mathbf{v}}}_T+\frac{n_1{n}_2}{n(n-2)}\left(\widehat{\gamma }{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1-1\right){\widehat{\mu }}_{a·T}.$$

First, we deal with the term

$${(n-2)}^{-1}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\stackrel{\sim }{\mathbf{v}}}_T=\underset{T_{1,a}}{\underbrace{\frac{\widehat{\gamma }}{n-2}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}}-\underset{T_{2,a}}{\underbrace{\frac{\lambda }{n-2}{\mathbf{U}}_{a·T}^{\prime }{\mathbf{U}}_T{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}}.$$

Conditional on {y_i}_i∈[n] and X_T, we have that

$$T_{1,a}~\mathcal{N}\left(0,\frac{n_1{n}_2}{n(n-2)}{\sigma }_{a\mid T}\frac{{\widehat{\gamma }}^2}{n-2}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\right)$$

and

$$\underset{a\in N}{max}\mid T_{1,a}\mid \le \sqrt{2\frac{n_1{n}_2}{n(n-2)}\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right)\frac{{\widehat{\gamma }}^2}{n-2}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_{T}log((p-s)log(n))}$$

with probability at least 1 − log⁻¹(n). On the event , we have that

$$\begin{array}{l}\underset{a\in N}{max}\mid T_{1,a}\mid \le (1+o(1))\sqrt{2{\pi }_1{\pi }_2{\gamma }^2\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\frac{log((p-s)log(n))}{n}}\\ =(1+o(1))\sqrt{2}{\pi }_1{\pi }_2(1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1)\frac{\lambda }{K_{{\lambda }_0}}.\end{array}$$

Since

$${\Vert {\mathit{\beta }}_T\Vert }_1\le \sqrt{s}{\Vert {\mathit{\beta }}_T\Vert }_2=\sqrt{s}{\Vert {\mathbf{\sum }}_{TT}^{-1/2}{\mathbf{\sum }}_{TT}^{1/2}{\mathit{\beta }}_T\Vert }_2\le \sqrt{s{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT}){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}$$

and

$$\begin{array}{l}\lambda {\Vert {\mathit{\beta }}_T\Vert }_1=\frac{{\lambda }_0{\Vert {\mathit{\beta }}_T\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\le \frac{{\lambda }_0\sqrt{s{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT}){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\\ \le \frac{K_{{\lambda }_0}}{\sqrt{K}}\frac{\sqrt{\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\le \frac{K_{{\lambda }_0}}{{\pi }_1{\pi }_2\sqrt{K}},\end{array}$$

we have that

$$\underset{a\in N}{max}\mid T_{1,a}\mid \le (1+o(1))\sqrt{2}{\pi }_1{\pi }_2\left(K_{{\lambda }_0}^{-1}+{\left({\pi }_1{\pi }_2\sqrt{K}\right)}^{-1}\right)\lambda <(\alpha /3)\lambda$$

by taking both K_λ0 and K sufficiently large.

Similarly, conditional on {y_i}_i∈[n] and X_T, we have that

$$T_{2,a}~\mathcal{N}\left(0,\frac{n_1{n}_2}{n(n-2)}{\sigma }_{a\mid T}\frac{{\lambda }^2}{n-2}\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\right),$$

which, on the event , gives

$$\begin{array}{l}\underset{a\in N}{max}\mid T_{2,a}\mid \le (1+o(1))\lambda \sqrt{2{\pi }_1{\pi }_2\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right){\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\frac{slog((p-s)log(n))}{n}}\\ \le (1+o(1))\sqrt{\frac{2}{K}}\lambda <(\alpha /3)\lambda \end{array}$$

with probability at least 1 − log⁻¹(n) for sufficiently large K.

Next, let

$$T_{3,a}=\frac{n_1{n}_2}{n(n-2)}\left(\widehat{\gamma }{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1-1\right){\widehat{\mu }}_{a·T}.$$

Simple algebra shows that

$$\widehat{\gamma }{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-\lambda {\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1-1=-\frac{1+\lambda {\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1}{1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}.$$

Therefore conditional on {y_i}_i∈[n] and X_T, we have that

$$T_{3,a}~\mathcal{N}\left(0,{\left(\frac{n_1{n}_2}{n(n-2)}\frac{1+\lambda {\Vert {\widehat{\mathit{\beta }}}_T\Vert }_1}{1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\right)}^2\frac{n}{n_1{n}_2}{\sigma }_{a\mid T}\right),$$

which, on the event , gives

$$\begin{array}{l}\underset{a\in N}{max}\mid T_{3,a}\mid \le (1+o(1))\frac{1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\sqrt{2{\pi }_1{\pi }_2\left(\underset{a\in N}{max}{\sigma }_{a\mid T}\right)\frac{log((p-s)log(n))}{n}}\\ \le (1+o(1))\sqrt{2}\frac{1+\lambda {\Vert {\mathit{\beta }}_T\Vert }_1}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\frac{1}{K_{{\lambda }_0}\sqrt{1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}}\lambda <(\alpha /3)\lambda \end{array}$$

with probability at least 1 − log⁻¹(n) when K_λ0 and K are chosen sufficiently large.

Piecing all these results together, we have that

$$\underset{a\in N}{max}\mid {\mathbf{C}}_{aT}{\stackrel{\sim }{\mathbf{v}}}_T-\frac{n_1{n}_2}{n(n-2)}{\widehat{\mu }}_a\mid <1.$$

A.2 Proof of Theorem 7

The theorem will be shown using standard tools described in Tsybakov (2009). First, in order to provide a lower bound on the minimax risk, we will construct a finite subset of Θ(Σ, τ, s), which contains the most difficult instances of the estimation problem so that estimation over the subset is as difficult as estimation over the whole family. Let Θ₁ ⊂ Θ(Σ, τ, s), be a set with finite number of elements, so that

$$\underset{\mathrm{\Psi }}{inf}R(\mathrm{\Psi },\mathrm{\Theta }(\mathbf{\sum },\tau ,s)\ge \underset{\mathrm{\Psi }}{inf}\underset{\mathit{\theta }\in {\mathrm{\Theta }}_1}{max}{\mathbb{P}}_{\mathit{\theta }}[\mathrm{\Psi }({\{{\mathbf{x}}_i,y_i\}}_{i\in [n]})\ne T(\mathit{\theta })].$$

To further lower bound the right hand side of the display above, we will use Theorem 2.5 in Tsybakov (2009). Suppose that Θ₁ = {θ₀, θ₁, …, θ_M} where T (θ_a) ≠ T (θ_b) and

$$\frac{1}{M}\sum _{a=1}^{M}KL({\mathbb{P}}_{{\mathit{\theta }}_0}\mid {\mathbb{P}}_{{\mathit{\theta }}_a})\le \alpha log(M),\alpha \in (0,1/8)$$ (A.4)

then

$$\underset{\mathrm{\Psi }}{inf}R(\mathrm{\Psi },\mathrm{\Theta }(\beta ,s)\ge \frac{\sqrt{M}}{1+\sqrt{M}}\left(1-2\alpha -\sqrt{\frac{2\alpha }{log(M)}}\right).$$

Without loss of generality, we will consider θ_a = (μ_a, 0, Σ). Denote the joint distributions of {X_i, Y_i}_i∈[n]. Under , we have ${\mathbb{P}}_{{\mathit{\theta }}_a}(Y_i=1)={\mathbb{P}}_{{\mathit{\theta }}_a}(Y_i=2)=\frac{1}{2}$, X_i|Y_i = 1 ~ (0, Σ) and X_i|Y_i = 2 ~ (μ_a, Σ). Denote f (x; μ, Σ) the density function of a multivariate Normal distribution. With this we have

$$\begin{array}{l}KL({\mathbb{P}}_{{\mathit{\theta }}_0}\mid {\mathbb{P}}_{{\mathit{\theta }}_a})={\mathbb{E}}_{{\mathit{\theta }}_0}log\frac{d{\mathbb{P}}_{{\mathit{\theta }}_0}}{d{\mathbb{P}}_{{\mathit{\theta }}_a}}\\ ={\mathbb{E}}_{{\mathit{\theta }}_0}log\frac{{\prod }_{i\in [n]}d{\mathbb{P}}_{{\mathit{\theta }}_0}[{\mathbf{X}}_i\mid Y_i]{\mathbb{P}}_{{\mathit{\theta }}_0}[Y_i]}{{\prod }_{i\in [n]}d{\mathbb{P}}_{{\mathit{\theta }}_a}[{\mathbf{X}}_i\mid Y_i]{\mathbb{P}}_{{\mathit{\theta }}_a}[Y_i]}\\ ={\mathbb{E}}_{{\mathit{\theta }}_0}\sum _{i:y_i=2}log\frac{f({\mathbf{X}}_i;{\mathit{\mu }}_0,\mathbf{\sum })}{f({\mathbf{X}}_i;{\mathit{\mu }}_a,\mathbf{\sum })}\\ =\frac{{\mathbb{E}}_{{\mathit{\theta }}_0}{n}_2}{2}{({\mathit{\mu }}_0-{\mathit{\mu }}_a)}^{\prime }{\mathbf{\sum }}^{-1}({\mathit{\mu }}_0-{\mathit{\mu }}_a)\\ =\frac{n}{4}{({\mathit{\beta }}_0-{\mathit{\beta }}_a)}^{\prime }\mathbf{\sum }({\mathit{\beta }}_0-{\mathit{\beta }}_a)\end{array}$$ (A.5)

where β_a = Σ⁻¹μ_a. We proceed to construct different finite collections for which (A.4) holds.

Consider a collection Θ₁ = {θ₀, θ₁, …, θ_p−s}, with θ_a = (μ_a, 0), that contains instances whose supports differ in only one component. Vectors ${\{{\mathit{\mu }}_a\}}_{a=0}^{p-s}$ are constructed indirectly through ${\{{\mathit{\beta }}_a\}}_{a=0}^{p-s}$, using the relationship β_a = Σ⁻¹μ_a. Note that this construction is possible, since Σ is a full rank matrix. For every a, all s non-zero elements of the vector β_a are equal to τ. Let T be the support and u(T) an element of the support T for which (3.2) is minimized. Set β₀ so that supp(β₀) = T. The remaining p–s parameter vectors ${\{{\mathit{\beta }}_a\}}_{a=1}^{p-s}$ are constructed so that the support of β_a contains all s − 1 element in T\u(T) and then one more element from [p]\T. With this, (A.5) gives

$$KL({\mathbb{P}}_{{\mathit{\theta }}_0}\mid {\mathbb{P}}_{{\mathit{\theta }}_a})=\frac{n{\tau }^2}{4}({\mathrm{\sum }}_{uu}+{\mathrm{\sum }}_{vv}-2{\mathrm{\sum }}_{uv})$$

and (3.2) gives

$$\frac{1}{p-s}\sum _{a=1}^{p-s}KL({\mathbb{P}}_{{\mathit{\theta }}_0}\mid {\mathbb{P}}_{{\mathit{\theta }}_a})=\frac{n{\tau }^2}{4}{\phi }_{\text{close}}(\mathbf{\sum }).$$

It follows from the display above that if

$$\tau <\sqrt{\frac{4}{{\phi }_{\text{close}}(\mathbf{\sum })}\frac{log(p-s+1)}{n}},$$ (A.6)

then (A.4) holds with α = 1/16.

Next, we consider another collection Θ₂ = {θ₀, θ₁, …, θ_M}, where $M=\left(\begin{array}{c}p-s\\ s\end{array}\right)$, and the Hamming distance between T (θ₀) and T (θ_a) is equal to 2s. As before, θ_a = (μ_a, 0) and vectors ${\{{\mathit{\mu }}_a\}}_{a=0}^M$ are constructed so that β_a = Σ⁻¹ μ_a with s non-zero components equal to τ. Let T be the support set for which the minimum in (3.3) is attained. Set the vector β₀ so that supp(β₀) = T. The remaining vectors ${\{{\mathit{\beta }}_a\}}_{a=1}^M$ are set so that their support contains s elements from the set [p]\T. Now, (A.5) gives

$$KL({\mathbb{P}}_{{\mathit{\theta }}_0}\mid {\mathbb{P}}_{{\mathit{\theta }}_a})=\frac{n{\tau }^2}{4}{\mathbf{1}}^{\prime }{\mathbf{\sum }}_{T({\mathit{\theta }}_0)\cup T({\mathit{\theta }}_a),T({\mathit{\theta }}_0)\cup T({\mathit{\theta }}_a)}\mathbf{1}.$$

Using (3.3), if

$$\tau <\sqrt{\frac{4}{{\phi }_{\text{far}}(\mathbf{\sum })}\frac{log\left(\begin{array}{c}p-s\\ s\end{array}\right)}{n}},$$ (A.7)

then (A.4) holds with α = 1/16.

Combining (A.6) and (A.7), by taking the larger β between the two, we obtain the result.

A.3 Proof of Theorem 8

For a fixed T, let Δ(T′) = f(T) − f(T′) and = {T′ ⊂ [p]: |T′| = s, T′ ≠ T}. Then

$${\mathbb{P}}_T[\widehat{T}\ne T]={\mathbb{P}}_T[\underset{T\in \mathcal{T}}{\cup }\{\mathrm{\Delta }(T^{\prime })<0\}]\le \sum _{T^{\prime }\in \mathcal{T}}{\mathbb{P}}_T[\mathrm{\Delta }(T^{\prime })<0].$$

Partition ${\widehat{\mathit{\mu }}}_T={({\widehat{\mathit{\mu }}}_1^{\prime },{\widehat{\mathit{\mu }}}_2^{\prime })}^{\prime }$, where μ̂₁ contains the variables in T ∩ T′, and ${\widehat{\mathit{\mu }}}_{T^{\prime }}={({\widehat{\mathit{\mu }}}_1^{\prime },{\widehat{\mathit{\mu }}}_3^{\prime })}^{\prime }$. Similarly, we can partition the covariance matrix S_TT and S_T′T′. Then,

$$g(T)={\widehat{\mathit{\mu }}}_1^{\prime }{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1+{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}$$

where ${\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}={\widehat{\mathit{\mu }}}_2-{\mathbf{S}}_{21}{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1$ and ${\mathbf{S}}_{22\mid 1}={\mathbf{S}}_{22}-{\mathbf{S}}_{21}{\mathbf{S}}_{11}^{-1}{\mathbf{S}}_{12}$ (see Section 3.6.2 in Mardia et al., 1979). Furthermore, we have that

$$\mathrm{\Delta }(T^{\prime })={\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}-{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}^{\prime }{\mathbf{S}}_{33\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}.$$ (A.8)

The two terms are correlated, but we will ignore this correlation and use the union bound to lower bound the first term and upper bound the second term. We start with analyzing ${\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}$, noting that the result for the second term will follow in the same way. By Theorem 3.4.5 in Mardia et al. (1979), we have that

$${\mathbf{S}}_{22\mid 1}~{\mathcal{W}}_{s-\mid T\cap T^{\prime }\mid }({(n-2)}^{-1}{\mathbf{\sum }}_{22\mid 1},n-2-\mid T\cap T^{\prime }\mid )$$

and independent of (S₁₂, S₁₁, μ̂). Therefore S_22|1 is independent of μ̃_2|1 and Theorem 3.2.12 in Muirhead (1982) gives us that

$$(n-2)\frac{{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}}{{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}}~{\chi }_{n-1-s}^2.$$

As in Lemma 10, we can show that

$$1-C_1\sqrt{\frac{log({\eta }^{-1})}{n}}\le \frac{{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}}{{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}}\le 1+C_2\sqrt{\frac{log({\eta }^{-1})}{n}}.$$

For μ̃_2|1, we have

$${\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}={\widehat{\mathit{\mu }}}_2-{\mathbf{S}}_{21}{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1={\widehat{\mathit{\mu }}}_{2\mid 1}+{\mathbf{\sum }}_{21}{\mathbf{\sum }}_{11}^{-1}{\widehat{\mathit{\mu }}}_1-{\mathbf{S}}_{21}{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1,$$

where ${\widehat{\mathit{\mu }}}_{2\mid 1}~\mathcal{N}({\mathit{\mu }}_{2\mid 1},\frac{n}{n_1{n}_2}{\mathbf{\sum }}_{22\mid 1})$, independent of μ̂₁, and ${\mathit{\mu }}_{2\mid 1}={\mathit{\mu }}_2-{\mathbf{\sum }}_{21}{\mathbf{\sum }}_{11}^{-1}{\mathit{\mu }}_1$. Conditioning on μ̂₁ and S₁, we have that

$${\mathbf{S}}_{21}{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1={\mathbf{\sum }}_{21}{\mathbf{\sum }}_{11}^{-1}{\widehat{\mathit{\mu }}}_1+\mathbf{Z},$$

where $\mathbf{Z}~\mathcal{N}(\mathbf{0},{(n-2)}^{-1}{\widehat{\mathit{\mu }}}_1^{\prime }{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1{\mathbf{\sum }}_{22\mid 1})$. Since μ̂_2|1 is independent of (S₁₂, S₁₁, μ̂₁), we have that

$${\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}\mid {\widehat{\mathit{\mu }}}_1,{\mathbf{S}}_{11}~\mathcal{N}\left({\mathit{\mu }}_{2\mid 1},\left(\frac{n}{n_1{n}_2}+{(n-2)}^{-1}{\widehat{\mathit{\mu }}}_1^{\prime }{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1\right){\mathbf{\sum }}_{22\mid 1}\right).$$

Let $a=\frac{n}{n_1{n}_2}+{(n-2)}^{-1}{\widehat{\mathit{\mu }}}_1^{\prime }{\mathbf{S}}_{11}^{-1}{\widehat{\mathit{\mu }}}_1$. Then

$${\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}\mid {\widehat{\mathit{\mu }}}_1,{\mathbf{S}}_{11}~a{\chi }_{\mid T\backslash T^{\prime }\mid }^2\left(a^{-1}{\mathit{\mu }}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\mathit{\mu }}_{2\mid 1}\right).$$

Therefore, conditioned on (μ̂₁, S₁₁),

$$\begin{array}{l}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}\\ \ge \left(1-C_1\sqrt{\frac{log({\eta }^{-1})}{n}}\right)\\ \times \left(\left({\mathit{\mu }}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\mathit{\mu }}_{2\mid 1}+a\mid T\backslash T^{\prime }\mid \right)-2\sqrt{\left(2a{\mathit{\mu }}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\mathit{\mu }}_{2\mid 1}+a^2\mid T\backslash T^{\prime }\right)log({\eta }^{-1})}\right)\end{array}$$

with probability 1 − 2η. Similarly,

$$\begin{array}{l}{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}^{\prime }{\mathbf{S}}_{33\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}\\ \le \left(1+C_2\sqrt{\frac{log({\eta }^{-1})}{n}}\right)\\ \times \left(\left({\mathit{\mu }}_{3\mid 1}^{\prime }{\mathbf{\sum }}_{33\mid 1}^{-1}{\mathit{\mu }}_{3\mid 1}+a\mid T^{\prime }\backslash T\mid \right)+2\sqrt{\left(2a{\mathit{\mu }}_{3\mid 1}^{\prime }{\mathbf{\sum }}_{33\mid 1}^{-1}{\mathit{\mu }}_{3\mid 1}+a^2\mid T^{\prime }\backslash T\right)log({\eta }^{-1})}+2alog({\eta }^{-1})\right)\end{array}$$

with probability 1 − 2η. Finally, Lemma 12 gives that $\mid a\mid \le C(1\vee {\mathit{\mu }}_1^{\prime }{\mathbf{\sum }}_{11}^{-1}{\mathit{\mu }}_1)n^{-1}$ with probability 1 − 2η.

Set ${\eta }_k={\left(\left(\begin{array}{c}p-s\\ s-k\end{array}\right)\left(\begin{array}{c}s\\ k\end{array}\right)slog(n)\right)}^{-1}$. For any T′ ⊂ [p], where |T′| = s and |T′ ∩ T | = k, we have that

$$\begin{array}{l}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}^{\prime }{\mathbf{S}}_{22\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{2\mid 1}-{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}^{\prime }{\mathbf{S}}_{33\mid 1}^{-1}{\stackrel{\sim }{\mathit{\mu }}}_{3\mid 1}\ge (1-o(1)){\mathit{\mu }}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\mathit{\mu }}_{2\mid 1}-(1+o(1)){\mathit{\mu }}_{3\mid 1}^{\prime }{\mathbf{\sum }}_{33\mid 1}^{-1}{\mathit{\mu }}_{3\mid 1}\\ -C\sqrt{(1\vee {\mathit{\mu }}_1^{\prime }{\mathbf{\sum }}_{11}^{-1}{\mathit{\mu }}_1){\mathit{\mu }}_{2\mid 1}^{\prime }{\mathbf{\sum }}_{22\mid 1}^{-1}{\mathit{\mu }}_{2\mid 1}{\mathrm{\Gamma }}_{n,p,s,k}}\\ -C(1\vee {\mathit{\mu }}_1^{\prime }{\mathbf{\sum }}_{11}^{-1}{\mathit{\mu }}_1){\mathrm{\Gamma }}_{n,p,s,k}.\end{array}$$

The right hand side in the above display is bounded away from zero with probability $1-\mathcal{O}\left({\left(\left(\begin{array}{c}p-s\\ s-k\end{array}\right)\left(\begin{array}{c}s\\ k\end{array}\right)slog(n)\right)}^{-1}\right)$ under the assumptions. Therefore,

$${\mathbb{P}}_T[\widehat{T}\ne T]\le \sum _{k=0}^{s-1}\sum _{T^{\prime }\in \mathcal{T}:\mid T\cap T^{\prime }\mid =k}{\mathbb{P}}_T[\mathrm{\Delta }(T^{\prime })<0]\le \frac{C}{log(n)},$$

which completes the proof.

B Proof of Risk Consistency

In this section, we give a proof of Corollary 9. From Theorem 4 we have that $\widehat{\mathbf{v}}={({\widehat{\mathbf{v}}}_T^{\prime },{\mathbf{0}}^{\prime })}^{\prime }$ with v̂_T defined in (2.13). Define

$${\stackrel{\sim }{\mathbf{v}}}_T=\frac{n(n-2)}{n_1{n}_2}\left(1+\frac{n_1{n}_2}{n(n-2)}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\right){\widehat{\mathbf{v}}}_T.$$

To obtain a bound on the risk, we need to control

$$\frac{-{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }({\mathit{\mu }}_{i,T}-{\widehat{\mathit{\mu }}}_{i,T})-{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\widehat{\mathit{\mu }}}_T/2}{\sqrt{{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\mathbf{\sum }}_{TT}{\stackrel{\sim }{\mathbf{v}}}_T}}$$ (B.1)

for i ∈ {1, 2}. Define the following quantities

$$\begin{array}{c}{\delta }_1={\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\Vert {\mathit{\beta }}_T\Vert }_1,{\stackrel{\sim }{\delta }}_1={\delta }_1/{\Vert {\mathit{\beta }}_T\Vert }_1,\\ {\delta }_2={\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1},{\widehat{\mathit{\mu }}}_T-{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2,\text{and}{\stackrel{\sim }{\delta }}_2={\delta }_2/{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2.\end{array}$$

Under the assumptions, we have that

$$\begin{array}{l}{\lambda }_0=\mathcal{O}\left(\sqrt{\frac{{\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT}){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{K(n)s}}\right),\\ r_n=\mathcal{O}\left(\frac{{\lambda }_0{\Vert {\mathit{\beta }}_T\Vert }_1}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\right)=\mathcal{O}\left(\frac{{\Vert {\mathit{\beta }}_T\Vert }_1}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}}\sqrt{\frac{{\mathrm{\Lambda }}_{min}({\mathbf{\sum }}_{TT})}{K(n)s}}\right),\text{and}\\ {\stackrel{\sim }{\delta }}_2={\mathcal{O}}_P\left(\sqrt{\frac{loglog(n)}{n}}\vee \frac{s\vee loglog(n)}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}n}\right).\end{array}$$

The last equation follows from Lemma 12. Note that δ̃₂ = (r_n). From Lemma 13, we have that δ̃₁ = o_p(1).

We have $\lambda {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)=\lambda {\Vert {\mathit{\beta }}_T\Vert }_1(1+{\mathcal{O}}_P({\stackrel{\sim }{\delta }}_1))={\mathcal{O}}_P(r_n)$, since Lemma 13 gives δ̃₁ = o_p(1), and

$$\frac{\frac{n(n-2)}{n_1{n}_2}+{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{1+{\pi }_1{\pi }_2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}={\mathcal{O}}_P(1).$$

Therefore ${\stackrel{\sim }{\mathbf{v}}}_T=(1+{\mathcal{O}}_P(r_n)){\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathcal{O}}_P(1){\lambda }_0{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)$. With this, we have

$$\begin{array}{l}{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\widehat{\mathit{\mu }}}_T=(1+{\mathcal{O}}_P(r_n))(1+{\mathcal{O}}_P({\stackrel{\sim }{\delta }}_2)){\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2-{\mathcal{O}}_P(1)(1+{\mathcal{O}}_P({\stackrel{\sim }{\delta }}_1)){\lambda }_0{\Vert {\mathit{\beta }}_T\Vert }_1\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\left[1+{\mathcal{O}}_P(r_n)-{\mathcal{O}}_P(1)\frac{{\lambda }_0{\Vert {\mathit{\beta }}_T\Vert }_1}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\right]\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2[1+{\mathcal{O}}_p(r_n)],\end{array}$$ (B.2)

where the last line follows from $\lambda {\Vert {\mathit{\beta }}_T\Vert }_1\asymp {\lambda }_0{\Vert {\mathit{\beta }}_T\Vert }_1/{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2$. Next

$$\begin{array}{l}{({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T})}^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\\ \le {\Vert {\mathbf{S}}_{TT}^{-1/2}({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T})\Vert }_2{\Vert {\mathbf{S}}_{TT}^{-1/2}{\widehat{\mathit{\mu }}}_T\Vert }_2\\ \le (1+{\mathcal{O}}_P(\sqrt{s/n})){\mathrm{\Lambda }}_{min}^{-1/2}({\mathbf{\sum }}_{TT})\sqrt{s}{\Vert {\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T}\Vert }_{\infty }{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}\sqrt{1+{\mathcal{O}}_P({\stackrel{\sim }{\delta }}_2)}\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}{\mathcal{O}}_P\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\mathit{s}loglog(n)/n}\right)\end{array}$$

and similarly

$${({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T})}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)=\sqrt{q_n}{\mathcal{O}}_P\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\mathit{s}loglog(n)/n}\right).$$

Combining these two estimates, we have

$$\begin{array}{l}\mid {\stackrel{\sim }{\mathbf{v}}}_T^{\prime }({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T})\mid \\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}(1-{\lambda }_0\sqrt{q_n}/{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}){\mathcal{O}}_P\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\mathit{s}loglog(n)/n}\right)\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}{\mathcal{O}}_P\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\mathit{s}loglog(n)/n}\right).\end{array}$$ (B.3)

From (B.2) and (B.3), we have that

$$-({\stackrel{\sim }{\mathbf{v}}}_T^{\prime }({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T}))-{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\widehat{\mathit{\mu }}}_T/2=-\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{2}(1+{\mathcal{O}}_P(r_n)).$$ (B.4)

Finally, a simple calculation gives,

$$\begin{array}{l}{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\mathbf{\sum }}_{TT}{\stackrel{\sim }{\mathbf{v}}}_T\\ \le {\mathrm{\Lambda }}_{max}\left({\mathbf{S}}_{TT}^{-1/2}{\mathbf{\sum }}_{TT}{\mathbf{S}}_{TT}^{-1/2}\right)\\ \times ({(1+{\mathcal{O}}_P(r_n))}^2{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T+{\mathcal{O}}_P(1){\lambda }_0^2\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ -{\mathcal{O}}_P(1){\lambda }_0{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T))\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\left(1+{\mathcal{O}}_P\left(r_n\vee {\stackrel{\sim }{\delta }}_2\vee \frac{{\lambda }_0^2{q}_n}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\vee \sqrt{\frac{s}{n}}\right)\right)\\ ={\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\left(1+{\mathcal{O}}_P\left(r_n\vee \frac{{\lambda }_0^2{q}_n}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\right)\right).\end{array}$$ (B.5)

Combining the equation (B.4) and (B.5), we have that

$$\frac{-{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }({\mathit{\mu }}_{1,T}-{\widehat{\mathit{\mu }}}_{1,T})-{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\widehat{\mathit{\mu }}}_T/2}{\sqrt{{\stackrel{\sim }{\mathbf{v}}}_T^{\prime }{\mathbf{\sum }}_{TT}{\stackrel{\sim }{\mathbf{v}}}_T}}=-\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}}{2}\frac{(1+{\mathcal{O}}_P(r_n))}{\sqrt{1+{\mathcal{O}}_P\left(r_n\vee \frac{{\lambda }_0^2{q}_n}{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\right)}}.$$

This completes the proof.

C Technical Results

We provide some technical lemmas which are useful for proving the main results. Without loss of generality, π₁ = π₂ = 1/2 in model (1.2). Define

$${ℰ}_n=\left\{\frac{n}{4}\le n_1\le \frac{3n}{4}\right\}\cap \left\{\frac{n}{4}\le n_2\le \frac{3n}{4}\right\},$$ (C.1)

where n₁, n₂ are defined in §1. Observe that n₁ ~ Binomial(n, 1/2), which gives [{n₁ ≤ n/4}] ≤ exp(−3n/64) and [{n₁ ≥ 3n/4}] ≤ exp(−3n/64) using standard tail bound for binomial random variable (Devroye et al., 1996, p. 130). Therefore

$$\mathbb{P}[{ℰ}_n]\ge 1-4exp(-3n/64).$$ (C.2)

The analysis is performed by conditioning on y and, in particular, we will perform analysis on the event . Note that on , 16/9n⁻¹ ≤ n/(n₁n₂) ≤ 16n⁻¹. In our analysis, we do not strive to obtain the sharpest possible constants.

C.1 Deviation of the Quadratic Scaling Term

In this section, we collect lemmas that will help us deal with bounding the deviation of ${\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T$ from ${\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T$.

Lemma 10

Define the event

$${ℰ}_1(\eta )=\left\{1-C_1\sqrt{\frac{log({\eta }^{-1})}{n}}\le \frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\le 1+C_2\sqrt{\frac{log({\eta }^{-1})}{n}}\right\},$$ (C.3)

for some constants C₁, C₂ > 0. Assume that s = o(n), then [ (η)] ≥ 1 − η for n sufficiently large.

Proof of Lemma 10

Using Theorem 3.2.12 in Muirhead (1982)

$$(n-2)\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}~{\chi }_{n-1-s}^2$$

(D.4) gives

$$\frac{n-2}{n-1-s}\frac{1}{1+\sqrt{\frac{16log({\eta }^{-1})}{3(n-1-s)}}}\le \frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\le \frac{n-2}{n-1-s}\frac{1}{1-\sqrt{\frac{16log({\eta }^{-1})}{3(n-1-s)}}}$$

with probability at least 1 − η. Since s = o(n), the above display becomes

$$1-C_1\sqrt{\frac{log({\eta }^{-1})}{n}}\le \frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}\le 1+C_2\sqrt{\frac{log({\eta }^{-1})}{n}}$$ (C.4)

for n sufficiently large.

Lemma 11

Define the event

$${ℰ}_2(\eta )=\left\{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\le {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+C_1\left(\frac{s\vee log({\eta }^{-1})}{n}\vee \sqrt{\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}log({\eta }^{-1})}{n}}\right)\right\}\cap \left\{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\ge {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2-C_2\left(\frac{s}{n}\vee \sqrt{\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}log({\eta }^{-1})}{n}}\right)\right\}.$$ (C.5)

Assume that β_min ≥ cn^−1/2, then [ (η)] ≥ 1 − 2η for n sufficiently large.

Proof of Lemma 11

Recall that ${\widehat{\mathit{\mu }}}_T~\mathcal{N}({\mathit{\mu }}_T,\frac{n}{n_1{n}_2}{\mathbf{\sum }}_{TT})$. Therefore

$$\frac{n_1{n}_2}{n}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T~{\chi }_s^2\left(\frac{n_1{n}_2}{n}{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\right).$$

Using (D.5), we have that

$$\begin{array}{l}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\le {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+\frac{ns}{n_1{n}_2}+\frac{2n}{n_1{n}_2}\sqrt{\left(s+2\frac{n_1{n}_2}{n}{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)log({\eta }^{-1})}+\frac{2n}{n_1{n}_2}log({\eta }^{-1})\\ \le {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+\frac{16s}{n}+32\sqrt{\left(\frac{s}{n^2}+\frac{2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{n}\right)log({\eta }^{-1})}+\frac{32log({\eta }^{-1})}{n}\\ \le {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+C_1\left(\frac{s}{n}\vee \sqrt{\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}log({\eta }^{-1})}{n}}\vee \frac{log({\eta }^{-1})}{n}\right),\end{array}$$ (C.6)

with probability 1 − η. The second inequality follows since we are working on the event , and the third inequality follows from the fact that β_min ≥ cn^−1/2. A lower bound follows from (D.6),

$$\begin{array}{l}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\ge {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+\frac{ns}{n_1{n}_2}-\frac{2n}{n_1{n}_2}\sqrt{\left(s+2\frac{n_1{n}_2}{n}{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)log({\eta }^{-1})}\\ \ge {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+\frac{16s}{n}-32\sqrt{\left(\frac{s}{n^2}+\frac{2{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}{n}\right)log({\eta }^{-1})}\\ \ge {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2-C_2\left(\frac{s}{n}\vee \sqrt{\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}log({\eta }^{-1})}{n}}\right)\end{array}$$ (C.7)

with probability 1 − η.

Lemma 12

On the event (η) ∩ (η) the following holds

$$\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\mid \le C\left(({\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}})\sqrt{\frac{log({\eta }^{-1})}{n}}\vee \frac{s\vee log({\eta }^{-1})}{n}\right).$$

Proof of Lemma 12

On the event (η) ∩ (η), using Lemma 10 and Lemma 11, we have that

$$\begin{array}{l}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\\ =\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\\ =\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T+\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}({\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T)\\ \le {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2+C_1{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\sqrt{\frac{log({\eta }^{-1})}{n}}+C_2\left(\frac{s\vee log({\eta }^{-1})}{n}\vee \sqrt{\frac{{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^{2}log({\eta }^{-1})}{n}}\right).\end{array}$$

A lower bound is obtained in the same way.

C.2 Other Results

Let the event (η) be defined as

$${ℰ}_3(\eta )=\underset{a\in [s]}{\cap }\left\{\mid {\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \le \sqrt{32{({\mathbf{\sum }}_{TT}^{-1})}_{aa}\frac{log(s{\eta }^{-1})}{n}}\right\}.$$ (C.8)

Since ${\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T$ is a multivariate normal with mean ${\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T$ and variance $\frac{n}{n_1{n}_2}{\mathbf{\sum }}_{TT}^{-1}$, we have [ (η)] ≥ 1 − η.

Furthermore, define the event (η) as

$${ℰ}_4(\eta )=\left\{\mid {({\widehat{\mathit{\mu }}}_T-{\mathit{\mu }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \le \sqrt{32{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})\frac{slog({\eta }^{-1})}{n}}\right\}.$$ (C.9)

Since ${\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)~\mathcal{N}\left({\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T),\frac{n}{n_1{n}_2}\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\right)$, we have [ (η)] ≥ 1 − η.

The next result gives a deviation of ${\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)$ from ${\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)$.

Lemma 13

The following inequality

$$\begin{array}{l}\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ \le C\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)}\vee {\Vert {\beta }_T\Vert }_1\right)\sqrt{\frac{loglog(n)}{n}}\end{array}$$ (C.10)

holds with probability at least 1 − (log⁻¹(n)).

Proof

Using the triangle inequality

$$\begin{array}{l}\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ \le \mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ +\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathit{\mu }}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid .\end{array}$$ (C.11)

For the first term, we write

$$\begin{array}{l}\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ \le \text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\left|\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}-\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\right|\\ +\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\left|\frac{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}-1\right|\left|\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}.\right|\end{array}$$ (C.12)

Let

$$\mathbf{G}=\left(\begin{array}{cc}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T& {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ \text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T& \text{sign}({\mathit{\beta }}_T){\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\end{array}\right),$$

and

$$\widehat{\mathbf{G}}=\left(\begin{array}{cc}{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T& {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\\ \text{sign}{({\mathit{\beta }}_T)}^{\prime }{S}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T& \text{sign}({\mathit{\beta }}_T)S_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\end{array}\right).$$

Using Theorem 3 of Bodnar and Okhrin (2008), we compute the density of ${\widehat{z}}_a={\widehat{\mathbf{G}}}_{12}{\widehat{\mathbf{G}}}_{22}^{-1}$ conditional on μ̂_T and obtain that

$$\sqrt{\frac{n-s}{q}}\left(\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}-\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\right)\mid {\widehat{\mathit{\mu }}}_T~t_{n-s},$$

where

$$q=\frac{{\widehat{\mathit{\mu }}}_T^{\prime }\left({\mathbf{\sum }}_{TT}^{-1}-\frac{{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\right){\widehat{\mathit{\mu }}}_T}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\le \frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}.$$

Lemma 20 gives

$$\left|\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}-\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\right|\le C\sqrt{\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}\frac{loglog(n)}{n}},$$

with probability at least 1 − log⁻¹(n). Combining with Lemma 14, Lemma 11, and (C.9), we obtain an upper bound on the RHS of (C.12) as

$$\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \le C\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2}\vee {\Vert {\beta }_T\Vert }_1\right)\sqrt{\frac{loglog(n)}{n}}$$ (C.13)

with probability at least 1 − (log⁻¹(n)).

The second term in (C.11) can be bounded using (C.9) with η = log⁻¹(n). Therefore, combining with (C.13), we obtain

$$\mid {\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \le C\left(\sqrt{{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)}\vee {\Vert {\beta }_T\Vert }_1\right)\sqrt{\frac{loglog(n)}{n}}$$ (C.14)

with probability at least 1 − (log⁻¹(n)), as desired.

Lemma 14

There exist constants C₁, C₂, C₃, and C₄ such that each of the following inequalities hold with probability at least 1 − log⁻¹(n):

$${\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\le C_1{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a\left(1+\mathcal{O}\left(\sqrt{\frac{log(slog(n))}{n}}\right)\right),\forall a\in T$$ (C.15)

$$\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-1\right|\le C_2\sqrt{\frac{log(slog(n))}{n}},\forall a\in T$$ (C.16)

$$\left|\frac{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}-1\right|\le C_3\sqrt{\frac{loglog(n)}{n}},\mathit{and}$$ (C.17)

$$\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\le C_4\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\left(1+\mathcal{O}\left(\sqrt{\frac{loglog(n)}{n}}\right)\right).$$ (C.18)

Proof

Theorem 3.2.12. in Muirhead (1982) states that

$$(n-2)\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}~{\chi }_{n-s-1}^2.$$

Using Equation (D.4),

$$\left|\frac{n-2}{n-s-1}\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-1\right|\le \sqrt{\frac{16log(2slog(n))}{3(n-s-1)}}$$

with probability 1 − (2s log(n))⁻¹. Rearranging terms in the display above, we have that

$${\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\le C{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a\left(1+\mathcal{O}\left(\sqrt{\frac{log(slog(n))}{n}}\right)\right)$$

and

$$\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-1\right|\le C\sqrt{\frac{log(slog(n))}{n}}.$$

A union bound gives (C.15) and (C.16).

Equations (C.17) and (C.18) are shown similarly.

Lemma 15

There exist constants C₁, C₂ > 0 such that the following inequality

$$\forall a\in T:\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \le C_1\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log(slog(n))}{n}}+C_2\mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \sqrt{\frac{log(slog(n))}{n}}$$ (C.19)

holds with probability at least 1 − (log⁻¹(n)).

Proof

Using the triangle inequality, we have

$$\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \le \mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\mid +\mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid .$$ (C.20)

For the first term, we write

$$\begin{array}{l}\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\mid \le {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|\\ +{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-1\right|\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|.\end{array}$$ (C.21)

As in the proof of Lemma C.10, we can show that

$$\sqrt{\frac{n-s}{q_a}}\left(\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right)\mid {\widehat{\mathit{\mu }}}_T~t_{n-s},$$

where

$$q_a=\frac{{\widehat{\mathit{\mu }}}_T^{\prime }\left({\mathbf{\sum }}_{TT}^{-1}-\frac{{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right){\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\le \frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}.$$

Lemma 20 and an application of union bound gives

$$\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|\le C\sqrt{\frac{{\widehat{\mathit{\mu }}}_T^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\frac{log(slog(n))}{n}},\forall a\in T,$$

with probability at least 1 − (log⁻¹(n)). Combining Lemma 11, Lemma 14 and Equation (C.8) with η = log⁻¹(n), we can bound the right hand side of (C.21) as

$$\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T\mid \le C_1\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}{\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\frac{log(slog(n))}{n}}+C_2\mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \sqrt{\frac{log(slog(n))}{n}}$$ (C.22)

with probability at least 1 − (log⁻¹(n)).

The second term in (C.20) is handled by (C.8) with η = log⁻¹(n). Combining with (C.22), we obtain

$$\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\widehat{\mathit{\mu }}}_T-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \le C_1\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}\left(1\vee {\Vert {\mathit{\beta }}_T\Vert }_{{\mathbf{\sum }}_{TT}}^2\right)\frac{log(slog(n))}{n}}+C_2\mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathit{\mu }}_T\mid \sqrt{\frac{log(slog(n))}{n}}$$ (C.23)

with probability at least 1 − (log⁻¹(n)). This completes the proof.

Lemma 16

The probability of the event

$$\underset{a\in [s]}{\cap }\left\{\mid {\mathbf{e}}_a^{\prime }({\mathbf{S}}_{TT}^{-1}-{\mathbf{\sum }}_{TT}^{-1})\text{sign}({\mathit{\beta }}_T)\mid \le C\left(\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s}\vee \mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \right)\sqrt{\frac{log(slog(n))}{n}}\right\}$$

is at least 1 − 2log⁻¹(n) for n sufficiently large.

Proof

Write

$$\begin{array}{l}\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ \le {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|\\ +{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-1\right|\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|.\end{array}$$ (C.24)

As in the proof of Lemma C.10, we can show that

$$\sqrt{\frac{n-s}{q_a}}\left(\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right)~t_{n-s},$$

where

$$q_a=\frac{\text{sign}{({\mathit{\beta }}_T)}^{\prime }\left({\mathbf{\sum }}_{TT}^{-1}-\frac{{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right)\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}.$$

Therefore,

$$\left|\frac{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}{\mathbf{e}}_a}-\frac{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)}{{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}{\mathbf{e}}_a}\right|\le C\sqrt{q_a\frac{log(slog(n))}{n}}$$ (C.25)

with probability 1 − (s log(n))⁻¹. Combining Lemma 14 and (C.25), we can bound the right hand side of Equation (C.24) as

$$\begin{array}{l}\mid {\mathbf{e}}_a^{\prime }{\mathbf{S}}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)-{\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \\ \le C({({\mathbf{\sum }}_{TT}^{-1})}_{aa}\sqrt{q_a}\vee \mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid )\sqrt{\frac{log(slog(n))}{n}}\\ \le C\left(\sqrt{{({\mathbf{\sum }}_{TT}^{-1})}_{aa}^{-1}{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s}\vee \mid {\mathbf{e}}_a^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\mid \right)\sqrt{\frac{log(\mathit{s}logn)}{n}}\end{array}$$

where the second inequality follows from

$$q_a\le {({\mathbf{\sum }}_{TT}^{-1})}_{aa}^{-1}\text{sign}{({\mathit{\beta }}_T)}^{\prime }{\mathbf{\sum }}_{TT}^{-1}\text{sign}({\mathit{\beta }}_T)\le {({\mathbf{\sum }}_{TT}^{-1})}_{aa}^{-1}{\mathrm{\Lambda }}_{min}^{-1}({\mathbf{\sum }}_{TT})s.$$

An application of the union bound gives the desired result.

D Tail Bounds For Certain Random Variables

In this section, we collect useful results on tail bounds of various random quantities used throughout the paper. We start by stating a lower and upper bound on the survival function of the standard normal random variable. Let Z ~ (0, 1) be a standard normal random variable. Then for t > 0

$$\frac{1}{\sqrt{2\pi }}\frac{t}{t^2+1}exp(-t^2/2)\le \mathbb{P}(Z>t)\le \frac{1}{\sqrt{2\pi }}\frac{1}{t}exp(-t^2/2).$$ (D.1)

Next, we collect results concerning tail bounds for central χ² random variables.

Lemma 17 (Laurent and Massart (2000))

Let $X~{\chi }_d^2$. For all x ≥ 0,

$$\mathbb{P}[X-d\ge 2\sqrt{dx}+2x]\le exp(-x)$$ (D.2)

$$\mathbb{P}[X-d\le -2\sqrt{dx}]\le exp(-x).$$ (D.3)

Lemma 18 (Johnstone and Lu (2009))

Let $X~{\chi }_d^2$, then

$$\mathbb{P}[\mid d^{-1}X-1\mid \ge x]\le exp(-\frac{3}{16}d{x}^2),x\in [0,\frac{1}{2}).$$ (D.4)

The following result provides a tail bound for non-central χ² random variable with non-centrality parameter ν.

Lemma 19 (Birgé (2001))

Let $X~{\chi }_d^2(\nu )$, then for all x > 0

$$\mathbb{P}[X\ge (d+\nu )+2\sqrt{(d+2\nu )x}+2x]\le exp(-x)$$ (D.5)

$$\mathbb{P}[X\le (d+\nu )-2\sqrt{(d+2\nu )x}]\le exp(-x).$$ (D.6)

The following Lemma gives a tail bound for a t-distributed random variable.

Lemma 20

Let X be a random variable distributed as

$$X~\sigma d^{-1/2}{t}_d,$$

where t_d denotes a t-distribution with d degrees of freedom. Then

$$\mid X\mid \le C\sqrt{{\sigma }^2{d}^{-1}log(4{\eta }^{-1})}$$

with probability at least 1 − η.

Proof

Let Y ~ (0, 1) and $Z~{\chi }_d^2$ be two independent random variables. Then X is equal in distribution to

$$\frac{\sigma d^{-1/2}Y}{\sqrt{d^{-1}Z}}.$$

Using (D.1),

$$\mid \sigma d^{-1/2}Y\mid \le \sigma d^{-1/2}\sqrt{log(4{\eta }^{-1})}$$

with probability at least 1 − η/2. (D.4) gives

$$d^{-1}X\ge 1-\sqrt{\frac{16}{3d}log(2{\eta }^{-1})}$$

with probability at least 1 − η/2. Therefore, for sufficiently large d,

$$\mid X\mid \le C\sqrt{{\sigma }^2{d}^{-1}log(4{\eta }^{-1})}.$$