Sparse Group Lasso: Optimal Sample Complexity, Convergence Rate, and Statistical Inference

Abstract

We study sparse group Lasso for high-dimensional double sparse linear regression, where the parameter of interest is simultaneously element-wise and group-wise sparse. This problem is an important instance of the simultaneously structured model – an actively studied topic in statistics and machine learning. In the noiseless case, matching upper and lower bounds on sample complexity are established for the exact recovery of sparse vectors and for stable estimation of approximately sparse vectors, respectively. In the noisy case, upper and matching minimax lower bounds for estimation error are obtained. We also consider the debiased sparse group Lasso and investigate its asymptotic property for the purpose of statistical inference. Finally, numerical studies are provided to support the theoretical results.

I. Introduction

Consider the high-dimensional double sparse regression with simultaneously group-wise and element-wise sparsity structures

$$\begin{gathered} y=X{\beta }^{*}+\epsilon , \\ \text{or\hspace{0.17em}equivalently }{y}_i=X_i^{\top }{\beta }^{*}+{\epsilon }_i,i=1,\dots ,n. \end{gathered}$$ (1)

Here, the covariates $X\in {ℝ}^{n\times p}$ and parameter β* are divided into d known groups, where the jth group contains b_j variables,

$$\begin{gathered} X=\left[X_{(1)}\cdots X_{(d)}\right],{\beta }^{*}={\left({\left({\beta }_{(1)}^{*}\right)}^{\top },\cdots {\left({\beta }_{(d)}^{*}\right)}^{\top }\right)}^{\top }, \\ {X}_{(j)}\in {ℝ}^{n\times b_j},{\beta }_{(j)}^{*}\in {ℝ}^{b_j}; \end{gathered}$$ (2)

β* is a (s, s_g)-sparse vector in the sense that

$$\begin{gathered} {\Vert {\beta }^{*}\Vert }_{0,2}≔\sum _{j=1}^d{1}_{\left\{{\beta }_{(j)}^{*}\ne 0\right\}}\le s_g \\ \text{and }{\Vert {\beta }^{*}\Vert }_0≔\sum _{i=1}^p{1}_{\left\{{\beta }_i^{*}\ne 0\right\}}\le s, \end{gathered}$$ (3)

The focus of this paper is on the estimation of and inference for β* based on (y, X). This problem has great importance in a variety of applications. For example in genome-wide association studies (GWAS) [1], the genes can be grouped into pathways and it is believed that only a small portion of the pathways contain causal single nucleotide polymorphisms (SNPs), and the number of causal SNPs is much less than the one of non-causal SNPs in a causal pathway. The sparse group Lasso has been applied to identify causal genes or SNPs associated with a certain trait [1]. Other examples include cancer diagnosis and therapy [2], [3], classification [4], and climate prediction [5] among many others. The problem can also be viewed as a prototype of various problems in statistics and machine learning, such as the sparse multiple response regression [6] and multiple task learning [7], [8], [9].

The sparse group Lasso [10], [11], [12] provides a classic and straightforward estimator for β*:

$$\widehat{\beta }=\underset{\beta }{\text{arg min}}{\Vert y-X\beta \Vert }_2^2+\lambda {\Vert \beta \Vert }_1+{\lambda }_g{\Vert \beta \Vert }_{1,2}.$$ (4)

Here, ${\Vert \beta \Vert }_1={\sum }_{i=1}^p\left|{\beta }_i\right|$ and ‖β‖_1,2 = Σ_j ‖β_(j)‖₂ are ℓ₁ and ℓ_1,2 convex regularizers to account for element-wise and group-wise sparsity structures, respectively. λ, λ_g ≥ 0 are tuning parameters. In the noiseless setting that ε = 0, one can apply the constrained ℓ₁ + ℓ_1,2 minimization instead to estimate β*:

$$\begin{gathered} \widehat{\beta }=\text{arg min\hspace{1em}}\lambda {\Vert \beta \Vert }_1+{\lambda }_g{\Vert \beta \Vert }_{1,2} \\ \text{subject to }y=X\beta . \end{gathered}$$ (5)

In fact, when λ, λ_g tend to zero while λ/λ_g is fixed as a constant, the sparse group Lasso (4) tends to the ℓ₁ + ℓ_1,2 minimization (5).

When β* is only element-wise sparse, the regular Lasso [13]

$${\widehat{\beta }}^L=\underset{\beta }{\text{arg min }}{\Vert y-X\beta \Vert }_2^2+\lambda {\Vert \beta \Vert }_1$$ (6)

can be applied and its theoretical properties have been well studied. See, for example, [14], [15]. When β* is only group-wise sparse, the group Lasso

$${\widehat{\beta }}^{GL}=\underset{\beta }{\text{arg min }}{\Vert y-X\beta \Vert }_2^2+{\lambda }_g{\Vert \beta \Vert }_{1,2}$$ (7)

and its variations have been widely investigated [16], [17], [18]. However, to estimate the simultaneously element-wise and group-wise sparse vector β*, despite many empirical successes of sparse group Lasso in practice, the theoretical properties, including optimal rate of convergence and sample complexity, are still unclear so far to the best of our knowledge.

A. Simultaneously Structured Models

More broadly speaking, the simultaneously structured models, i.e., the parameter of interest has multiple structures at the same time, have attracted enormous attention in many fields including statistics, applied mathematics, and machine learning. In addition to the high-dimensional double sparse regression, other simultaneously structured models include sparse principal component analysis [19], [20], tensor singular value decomposition [21], [22], simultaneously sparse and low-rank matrix/tensor recovery [23], [24], sparse matrix/tensor SVD [25], and sparse phase retrieval [26], [27], [28]. As shown in [29], [23], by minimizing multi-objective regularizers with norms associated with these structures (such as ℓ₁ norm for element-wise sparsity, nuclear norm for low-rankness, and total variation norm for piecewise constant structures), one usually cannot do better than applying an algorithm that only exploits one structure. They particularly illustrated that simultaneously sparse and low-rank structured matrix cannot be well estimated by penalizing ℓ₁ and nuclear norm regularizers. Instead, non-convex methods were proposed and shown to achieve better performance.

However based on their results, it remains an open question whether the convex regularization, such as sparse group Lasso or ℓ₁ + ℓ_1,2 minimization, can achieve good performance in estimation of parameter with two types of sparsity structures, such as the aforementioned high-dimensional double sparse regression. Specifically, as illustrated in Section II-B, a direct application of [23] does not provide a sample complexity lower bound for exact recovery that matches our upper bound.

B. Optimality and Related Literature

This paper fills the void of statistical limits of sparse group Lasso and provides an affirmative answer to the aforementioned question: by exploiting both element-wise and group-wise sparsity structures, the ℓ₁ + ℓ_1,2 regularization does provide better performance in high-dimensional double sparse regression. Particularly in the noiseless case, it is shown that (s, s_g)-sparse vectors can be exactly recovered and approximately (s, s_g)-sparse vectors can be stably estimated with high probability whenever the sample size satisfies n ≳ s_g log(d/s_g) + s log(es_gb), where b = max_1≤i≤d b_i. On the other hand, we prove that exact recovery cannot be achieved by ℓ₁ + ℓ_1,2 regularization and stable estimation of approximately (s, s_g)-sparse vectors is impossible in general unless n ≳ s_g log(d/s_g) + s log(es_gb/s). We then consider the noisy case and develop the matching upper and lower bounds on the convergence rate for the estimation error. Simulation studies are carried out and the results support our theoretical findings. In addition, statistical inference for the individual coordinates of β* is studied. A confidence interval is constructed based on the debiased sparse group Lasso estimator and its asymptotic property. The results show that by exploring the simultaneously element-wise and group-wise sparsity structures, the debiased sparse group Lasso requires less sample size than the debiased Lasso and debiased group Lasso in the literature [30], [31], [32], [33].

The theoretical analysis of sparse group Lasso and ℓ₁ + ℓ_1,2 minimization is highly non-trivial. First, the regularizer λ‖ · ‖₁ + λ_g‖ · ‖_1,2 is not decomposable with respect to the support of β* so that the classic techniques of decomposable regularizers [34] and null space property [35] may not be suitable here. Despite a substantial body of literature on high-dimensional element-wise sparse vector estimation based on restricted isometry property (RIP) [36], [37], [38], [39], [40] and restricted eigenvalue [14], these techniques cannot provide nearly optimal results for sparse group Lasso here as it is technically difficult to partition general vectors into simultaneously element-wise and group-wise ones that preserves some ordering structures. Departing from the previous literature, our theoretical analysis relies on a novel construction of approximate dual certificate. See Section II-C for further details. Although our results mostly focus on the performance of sparse group Lasso and ℓ₁ + ℓ_1,2 estimators, the techniques of approximate dual certificate on multi-norm structures here can also be of independent interest.

The statistical properties of sparse group Lasso and related estimators have been studied previously. For example, [5] developed consistency results for estimators with a general tree-structured norm regularizers, of which the sparse group Lasso is a special case. [41] analyzed the asymptotic behaviors of the adaptive sparse group Lasso estimator. [4], [42] studied the multi-task learning and classification problems based on a variant of sparse group Lasso estimator. [12] studied multivariate linear regression via sparse group Lasso. [43] provided a theoretical framework for developing error bounds of the group Lasso, sparse group Lasso, and group Lasso with tree structured overlapping groups. Specifically, their results imply that the group-wise sparse signal can be exactly recovered with high probability by solving (5) if the sample size satisfies n ≳ s_g (b + log d). Different from previous results, this paper focused on both the required sample size and convergence rate of estimation error of sparse group Lasso. To the best of our knowledge, this is the first paper that provides optimal theoretical guarantees for both the sample complexity and estimation error of sparse group Lasso.

C. Organization of the Paper

The rest of the article is organized as follows. After a brief introduction to notation and preliminaries in Section II-A, the main theoretical results on constrained ℓ₁ + ℓ_1,2 minimization in the noiseless setting is presented in Section II-B and the key proof ideas are explained in Section II-C. Results for sparse group Lasso in the noisy setting are discussed in Section III. In particular, the optimal rate of estimation error and statistical inference are studied in Sections III-A and III-B, respectively. In Section IV-A, we introduce a practical scheme to select tuning parameters. In Section IV-B, we provide simulation results in both noiseless and noisy cases to justify our theoretical findings. The proofs of technical results are given in Section VI. All technical lemmas and their proofs can be found in Appendix A.

II. ℓ₁ + ℓ_1,2 Minimization in Noiseless Case

A. Notation and Preliminaries

The following notation will be used throughout the paper. We denote a⋀b = min{a, b}, a⋁b = max{a, b}. Let sgn(·) be the sign function, i.e., sgn(x) = 1, 0, or −1, if x > 0, x = 0, or x < 0, respectively. H_α(·) is the soft-thresholding function such that H_α(x) = sgn(x) · {(|x| − α) ⋁ 0} for any $x\in ℝ$. We say a ≲ b and a ≳ b if a ≤ Cb and b ≤ Ca for some uniform constant C > 0, respectively. a ≍ b means a ≲ b and a ≳ b both hold. Let the uppercase C, C₁, C₀, … and lowercase c, c₁, c₀, … denote large and small positive constants respectively, whose actual values vary from time to time. Throughout the paper, we focus on the parameter index set {1, …, p} partitioned into d groups. Denote (1), …, (d) ⊆ {1, … , p} as the index sets belonging to each group. Additionally, for any group index subset G ⊆ {1, …, d}, define (G) = ∪_j∈G(j), (G^c) = ∪_j∉G(j). For any vector γ and index subset T, ${\gamma }_T\in {ℝ}^{\left|T\right|}$ represents the sub-vector of γ with index set T. In particular, γ(G) represents the sub-vector of γ in the union of Groups j ∈ G. Define the ℓq norm of any vector γ as ‖y‖_q = (∑_i|γ_i|^q)^1/q. For any vector $\beta \in {ℝ}^p$ with group structures, we also define the ℓ_q1,q2 norm for any 0 ≤ q₁, q₂ ≤ ∞ as

$$\begin{gathered} \Vert \gamma {\Vert }_{q_1,q_2}={\left(\sum _{j=1}^d{\Vert {\gamma }_{(j)}\Vert }_{q_2}^{q_1}\right)}^{1/q_1} \\ ={\left\{\sum _{j=1}^d{\left(\sum _{i\in (j)}{\left|{\gamma }_i\right|}^{q_2}\right)}^{q_1/q_2}\right\}}^{1/q_1}. \end{gathered}$$

In particular, ${\Vert \gamma \Vert }_{0,2}={\sum }_{j=1}^d{1}_{\left\{{\gamma }_{(j)}\ne 0\right\}}$ is the number of non-zero groups of γ, ‖γ‖_∞,2 = max_j ‖γ_(j)‖₂ is the maximum ℓ₂ norm among all groups of γ, and ${\Vert \gamma \Vert }_{1,2}={\sum }_{j=1}^d{\Vert {\gamma }_{(j)}\Vert }_2$ is the group-wise ℓ₁ penalty. With a slight abuse of notation, we simply denote ‖γ_T‖_q1,q2 = ‖u‖_q1,q2 if $u\in {ℝ}^p$, u restricted on subset T is γ_T and u restricted on T^c is 0.

The focus of this paper is on simultaneously element-wise and group-wise sparse vectors defined as follows.

Definition 1 (Simultaneous element-wise and group-wise sparsity): Assume ${\beta }^{*}\in {ℝ}^p$ is associated with group partition (1), …, (d). For positive integers s, s_g satisfying s_g ≤ d and $s_g\le s\le {\text{max}}_{\Omega \subseteq \{1,\dots ,d\},|\Omega |=s_g}{\sum }_{i\in \Omega }{b}_i$, we say β* is (s, s_g)- sparse if

$${\Vert {\beta }^{*}\Vert }_{0,2}=\sum _{j=1}^d{1}_{\left\{{\beta }_{(j)}^{*}\ne 0\right\}}\le s_g,{\Vert {\beta }^{*}\Vert }_0=\sum _i{1}_{\left\{{\beta }_i^{*}\ne 0\right\}}\le s.$$

B. Noiseless Case and Sample Complexity

To analyze the performance of sparse group Lasso and ℓ₁ + ℓ_1,2 minimization, we first introduce the following assumption on the design matrix X.

Assumption 1 (Sub-Gaussian assumption): Suppose all rows of X are i.i.d. centered sub-Gaussian distributed. Specifically, $\mathbb{E}{X}_{i\cdot }=0$, $\text{Var}\left(X_{i\cdot }^{\top }\right)=\Sigma$, and for any $\alpha \in {ℝ}^p$, we have $\mathbb{E}\text{\hspace{0.17em}exp}\left({\alpha }^{\top }{\Sigma }^{-1/2}{X}_{i\cdot }^{\top }\right)\le \text{exp}\left({\kappa }^2{\Vert \alpha \Vert }_2^2/2\right)$ for constant κ > 0. We also assume there exist two constants C_max ≥ c_min > 0 such that c_min ≤ σ_min (Σ) ≤ σ_max(Σ) ≤ C_max, where σ_max(Σ) and σ_min(Σ) are the largest and smallest eigenvalues of Σ, respectively.

Clear, a random matrix X with i.i.d. standard normal entries satisfies this assumption – this design is referred to as the Gaussian ensemble and has been considered as a benchmark setting in compressed sensing and high-dimensional regression literature [44], [45].

The following theorem shows that the ℓ₁ + ℓ_1,2 minimization achieves the exact recovery with high probability when β* is simultaneously element-wise and group-wise sparse, X is weakly dependent, and Assumption 1 holds. The theorem also provides a more general upper bound on estimation error if β* is approximately element-wise and group-wise sparse.

Theorem 1 (ℓ₁ + ℓ_1,2 minimization in noiseless case): Suppose one observes y = Xβ*, where X has the group structure (2) and satisfies Assumption 1, β* is (s, s_g)-sparse, and b = max_1≤i≤d b_i. Let T be the support of β*. Suppose there exist uniform constants C, c > 0 such that

$$n\ge C\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)\right),$$ (8)

$$\underset{i\in T^c}{\text{max\hspace{0.17em}}}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\le c/\sqrt{s},$$ (9)

then the constrained ℓ₁ + ℓ_1,2 minimization (5) with ${\lambda }_g=\sqrt{s/s_g}\lambda$ achieves the exact recovery with probability at least 1 − C exp(−cn/s).

Moreover, if ${\beta }^{*}\in {ℝ}^p$ is a general vector and $\widehat{\beta }$ is the solution to the constrained ℓ₁ + ℓ_1,2 minimization (5) with ${\lambda }_g=\sqrt{s/s_g}\lambda$, then

$$\begin{gathered} {\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2 \\ \lesssim \underset{S\in 𝒮}{\text{min}}\left(\frac{1}{\sqrt{s}}{\Vert {\beta }_{S^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{S^c}^{*}\Vert }_{1,2}\right). \end{gathered}$$ (10)

with probability at least 1 − C exp(−cn/s). Here,

$$𝒮=\left\{S:{\Vert {\beta }_S^{*}\Vert }_0\le s,{\Vert {\beta }_S^{*}\Vert }_{0,2}\le s_g\underset{i\in S^c}{\text{ max\hspace{0.17em}}}{\Vert {\Sigma }_{i,S}{\Sigma }_{S,S}^{-1}\Vert }_2\le c/\sqrt{s}\right\}.$$

Remark 1 (Interpretation and comparison): In Theorem 1, the required sample size for achieving exact recovery contains two terms: s_g log(d/s_g) and s log(es_gb). Intuitively speaking, s_g log(d/s_g) corresponds to the complexity of identifying s_g non-zero groups and s log(es_gb) corresponds to the complexity of estimating s non-zero elements of β in s_g known groups.

When β* is only element-wise or group-wise sparse, one can apply respectively the classic ℓ₁ or ℓ_1,2 minimization to recover β*,

$${\widehat{\beta }}^{{\ell }_1}=\underset{\beta }{\text{arg min }}{\Vert \beta \Vert }_1\text{\hspace{1em} subject to \hspace{1em}}y=X\beta \text{,}$$ (11)

$${\widehat{\beta }}^{{\ell }_{1,2}}=\underset{\beta }{\text{arg min }}{\Vert \beta \Vert }_{1,2}\text{\hspace{1em} subject to \hspace{1em}}y=X\beta .$$ (12)

The ℓ₁ minimization and ℓ_1,2 minimization here are respectively the special form of the regular Lasso and group Lasso (if λ, λ_g = 0₊ in (6) and (7)), respectively. Especially if the group size b₁ ≍ ⋯ ≍ b_d ≍ b, to ensure exact recovery in the noiseless setting with high probability, (11) requires n ≳ Cs log(ebd/s) [46] and group Lasso requires n ≳ s_g(b + log(ed/s_g)). The ℓ₁ + ℓ_1,2 minimization (5) has provable advantages over both regular and group Lasso when b ≫ log(d) ≫ log(es_gb) and s_gb/ log(es_gb) ≫ s ≫ s_g. In particular, when s_g = s, the double sparse regression reduces to the vanilla sparse linear regression, and the upper bound (10) matches the classic upper bound for ℓ₁ minimization [44].

In addition, Condition (9) is an important technical condition we used in our theoretical analysis.

Next, we consider the sample complexity lower bound. Suppose b₁ = b₂ = ⋯ = b_d and d ≥ 2s_g. Recall that one observes y = Xβ* without noise and aims to estimate the (s, s_g)-sparse vector β* based on y and X. As indicated by classic results in compressed sensing [47], with sufficient computing power, the ℓ₀ minimization below achieves exact recovery of β*

$${\widehat{\beta }}^{{\ell }_0}=\text{arg min }{\Vert \beta \Vert }_0\text{\hspace{1em} subject to \hspace{1em}}X\beta =y$$ (13)

as along as X is non-degenerate and n ≥ 2s. This bound is actually sharp: when n < 2s, for any set T ⊆ {1, …, db} with cardinality 2s, one can find a vector γ such that supp(γ) ⊆ T and Xγ = 0. By choosing an appropriate T, we can split the support γ to obtain two (s, s_g)-sparse vectors β₁, β₂ satisfying β₁ + β₂ = γ. Then, Xβ₁ = X(−β₂) but there is no way to distinguish β₁ and β₂ merely based on X and y = Xβ₁ = X(−β₂).

However, the ℓ₀ minimization (13) is computational infeasible in practice while a larger sample size is required for applying more practical methods. The following theorem shows that by performing the convex ℓ₁ regularization, ℓ_1,2 regularization, or any weighted combination of them, one requires at least Ω(s_g log(d/s_g) + s log(es_gb/s)) observations to ensure exact recovery of (s, s_g)-sparse vectors.

Theorem 2 (Sample complexity lower bound for exact recovery): Suppose b₁ = ⋯ = b_d = b, d, b ≥ 3. Suppose X is an n-by-(db) matrix. If every (2s, 2s_g)-sparse vector $\beta \in {ℝ}^{db}$ is a minimizer of the following programming for some (λ, λ_g) ∈ {(λ, λ_g) : λ, λ_g ≥ 0, λ + λ_g > 0}:

$$\underset{z}{\text{min\hspace{0.17em}}}\lambda {\Vert z\Vert }_1+{\lambda }_g{\Vert z\Vert }_{1,2}\text{\hspace{1em} subject to \hspace{1em}}Xz=y=X\beta \text{. }$$

In other words, if the ℓ₁ + ℓ_1,2 minimization exactly recover all (2s, 2s_g)-sparse vector β, then we must have n ≳ s_g log(d/s_g) + s log(es_gb/s).

The following sample complexity lower bound shows that for arbitrary methods, to ensure stable estimation of all approximately sparse vectors, one requires at least Ω(s_g log(d/s_g) + s log(es_gb/s)) observations.

Theorem 3 (Sample complexity lower bound for stable estimation): Suppose b₁ = ⋯ = b_d = b, b, d ≥ 3. Assume there exists a matrix $X\in {ℝ}^{n\times (bd)}$, a map $\Delta :{ℝ}^n\to {ℝ}^{bd}$ (Δ may depend on X), and a constant C > 0 satisfying

$${\Vert \beta -\Delta (X\beta )\Vert }_2\le C\left(\frac{{\Vert \beta \Vert }_1}{\sqrt{s}}+\frac{{\Vert \beta \Vert }_{1,2}}{\sqrt{s_g}}\right)$$ (14)

for all $\beta \in {ℝ}^p$ and some s, s_g satisfying d ≥ s_g, s_gb ≥ s ≥ s_g. There exists constants C₀ and c₀ that depend only on C such that whenever s_g ≥ C₀, we must have

$$n\ge c_0\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right).$$

Remark 2 (Optimality and comparison with previous results): Theorems 2 and 3 show that the sample complexity upper bound in Theorem 1 is rate-optimal under a weak condition: log(es_gb) ≍ log(es_gb)−log(s) or log(d) ≥ 2s log(s)/s_g. Oymak, et al. [23] provided a general analysis for convex regularization of simultaneously structured parameter estimation. Specifically for the high-dimensional double sparse regression, a direct application of their Theorem 3.2 and Corollary 3.1 implies that if ℓ₁ + ℓ_1,2 minimization can exactly recover (s, s_g)-sparse vector β* with a constant probability, one must have n ≳ s. We can see that Theorem 2 provides a sharper lower bound on sample complexity.

In addition, by setting s_g = s, the lower bound in Theorems 2 and 3 reduces to n ≳ s log(p/s), which matches the optimal sample complexity lower bound for exact recovery of s-sparse vectors [46, Theorem10.11, Proposition 10.7]. By setting s = s_gb, we obtain a sample complexity lower bound n ≳ s_g(b + log(d/s_g)) for (approximate) s_g-group-wise sparse vector recovery and stable estimation. To the best of our knowledge, this is the first sample complexity lower bound for group Lasso.

C. Proof Sketches

We briefly discuss the proof sketches of the main technical results in this section. The detailed proofs are postponed to Section VI.

The proof of Theorem 1 is based on a novel dual certificate scheme. The dual certificate [48] has been used in the theoretical analysis for various convex optimization methods in high-dimensional problems, such as matrix completion [49], [50], compressed sensing [44], robust PCA [51], tensor completion [52], etc. The high-dimensional double sparse linear regression exhibits different aspects from these previous works due to the simultaneous sparsity structure. In particular, we can show that if the u_et defined below is in the row space of X, it can be used as an exact dual certificate for recovery of (s, s_g)-sparse vector β*:

$$\begin{gathered} u_{et}=v_{et}+w_{et}\in {ℝ}^p, \\ \{\begin{array}{ll}{\left(v_{et}\right)}_{(j)}=\sqrt{s/s_g}{\beta }_{(j)}^{*}/{\Vert {\beta }_{(j)}^{*}\Vert }_2,\hfill & j\in G;\hfill \\ {\Vert {\left(v_{et}\right)}_{(j)}\Vert }_2<\sqrt{s/s_g},\hfill & j\in G^c;\hfill \end{array} \\ \{\begin{array}{l}{\left(w_{et}\right)}_T=\text{sgn}\left({\beta }_T^{*}\right)\hfill \\ {\Vert {\left(w_{et}\right)}_{T^c}\Vert }_{\infty }<1.\hfill \end{array} \end{gathered}$$ (15)

Here, T and G are the element-wise and group-wise supports of β*:

$$\begin{gathered} T=\left\{i:{\beta }_i\ne 0\right\}\subseteq \left\{1,\dots ,p\right\}, \\ G=\left\{j:{\beta }_{(j)}\ne 0\right\}\subseteq \left\{1,\dots ,d\right\}. \end{gathered}$$

Roughly speaking, u_et is the sub-gradient of objective function (5) evaluated at β = β*. If u_et is in the row space of X, the sub-gradient will be perpendicular to the feasible set of (5), which implies that β* is the unique minimizer of ℓ₁ + ℓ_1,2 minimization (5).

For more general vector β* that does not necessarily have a sparse support T or G, we consider the following (s, s_g)-sparse approximation:

$$\begin{gathered} {\beta }^{ap}=\underset{S}{\text{arg min }}\frac{1}{\sqrt{s}}{\Vert {\beta }_{S^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{S^c}^{*}\Vert }_{1,2} \\ \text{subject to }{\Vert {\beta }_S^{*}\Vert }_0\le s.{\Vert {\beta }_S^{*}\Vert }_{0,2}\le s_g, \\ \underset{i\in S^c}{\text{max }}{\Vert {\Sigma }_{i,S}{\Sigma }_{S,S}^{-1}\Vert }_2\le c/\sqrt{s}. \end{gathered}$$ (16)

Let $T=\left\{i:{\beta }_i^{ap}\ne 0\right\}$ and G = {j : (β^ap)_(j) ≠ 0} be the element-wise and group-wise supports of β^ap. Define

$$\begin{gathered} {\tilde{u}}_0={\tilde{v}}_0+{\tilde{w}}_0\in {ℝ}^p, \\ \{\begin{array}{ll}{\left({\tilde{v}}_0\right)}_{(j)}=\sqrt{s/s_g}{\beta }_{T,(j)}^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2,\hfill & j\in G;\hfill \\ {\Vert {\left({\tilde{v}}_0\right)}_{(j)}\Vert }_2<\sqrt{s/s_g},\hfill & j\in G^c;\hfill \end{array} \\ \{\begin{array}{l}{\left({\tilde{w}}_0\right)}_T=\text{sgn}\left({\beta }_T^{*}\right)\hfill \\ {\Vert {\left({\tilde{w}}_0\right)}_{T^c}\Vert }_{\infty }<1.\hfill \end{array} \end{gathered}$$ (17)

Here ${\beta }_{T,(j)}^{*}\in {ℝ}^{b_j}$ is the subvector β* restricted on the j-th group with all entries in T^c set to zero. Similarly to the exactly sparse case, if ũ₀ is in the row space of X and the true β* is approximately (s, s_g)-sparse, the minimizer of (5) will be close to β*.

However, it is often difficult to find an exact dual certificate that lies in the row space of X and satisfies stringent conditions in (15) or (17). We instead propose to analyze via the approximate dual certificate defined as (18) in the following lemma.

Lemma 1 (Approximate dual certificate for sparse group Lasso): Suppose T, G are element-wise and group-wise support defined in (16). ũ₀ is defined in (17). Assume X satisfies ${\sigma }_{\text{min}}\left(X_T^{\top }{X}_T/n\right)\ge c_{\text{min}}/2$. If there exists $u\in {ℝ}^p$ in the row span of X satisfying

$$\begin{gathered} {\Vert u_T-{\left({\tilde{u}}_0\right)}_T\Vert }_2\cdot \underset{i\in T^c}{\text{max\hspace{0.17em}}}{\Vert X_T^{\top }{X}_i/n\Vert }_2\le c_{\text{min}}/8, \\ {\Vert H_{1/2}\left(u_{\left(G^c\right)}\right)\Vert }_{\infty ,2}\le \sqrt{s_0}/2,{\Vert u_{(G)\backslash T}\Vert }_{\infty }\le 1/2, \end{gathered}$$ (18)

Then the conclusion of Theorem 1 (10) holds with probability at least 1 − 2e^−cn. Here, H_1/2(·) is the soft-thresholding operator defined at the beginning of Section II.

If we additionally assume β* is (s, s_g)-sparse, then β* is the unique solution to the sparse group ℓ₁ +ℓ_1,2 minimization (5) with probability at least 1 – 2e^−cn.

Lemma 1 shows that the conclusion of Theorem 1 holds if there exists an approximate dual certificate u satisfying the condition (18). The following lemma shows that, under the assumptions in Theorem 1, one can find such an approximate dual certificate with high probability.

Lemma 2: Suppose X has group structure (2) and satisfies Assumption 1. Recall ${\sigma }_{\text{min}}\left(X_T^{\top }{X}_T/n\right)$ is the least eigenvalue of $X_T^{\top }{X}_T/n$. Then ${\sigma }_{\text{min}}\left(X_T^{\top }{X}_T/n\right)\ge 1/2$ and (18) holds with probability at least 1 − Ce^−cn/s, where T is defined in (16).

Another key technical tool to the proof of Theorem 1 is the following Lemma, which shows that X satisfies the restricted isometry property for all simultaneously element-wise and group-wise sparse vectors with high probability when there are enough samples.

Lemma 3: If n ≥ C(s_g log(d/s_g) + s log(es_gb)),

$$\begin{gathered} \frac{c_{\text{min}}}{2}{\Vert \gamma \Vert }_2^2\le \frac{1}{n}{\Vert X\gamma \Vert }_2^2\le \left(C_{\text{max}}+\frac{c_{\text{min}}}{2}\right){\Vert \gamma \Vert }_2^2, \\ \forall \gamma \in \left\{\gamma \in {ℝ}^p:{\Vert \gamma \Vert }_0\le 2s,{\Vert \gamma \Vert }_{0,2}\le 2s_g\right\} \end{gathered}$$ (19)

with probability at least 1 – 2e^−cn.

Next we briefly discuss the proof of Theorem 2. Consider the quotient space ${ℝ}^{db}/\text{ker}(X)=\left\{[\gamma ]≔x+\text{ker}(X),\gamma \in {ℝ}^{db}\right\}$ and define an associated norm as ‖[γ]‖ = inf_v∈ker(X) {λ‖γ − v‖₁ + λ_g‖γ − v‖_1,2}. We show that there exist N different (s, s_g)-sparse vectors β⁽¹⁾, …, β^(N) such that log(N) ≍ s log(es_gb/s) + s_g log(d/s_g) and ‖[β⁽ⁱ⁾]‖ = 1, ‖[β⁽ⁱ⁾] − [β^(j)]‖ ≥ 2/9 for all 1 ≤ i ≠ j ≤ N. By a property of the packing number and the fact that $\text{dim}\left({ℝ}^{db}/\text{ker}(X)\right)\le n$, we must have N ≤ 10ⁿ. Thus n ≳ log(N) ≍ s log(es_gb/s) + s_g log(d/s_g).

We prove Theorem 3 by contradiction. Assume that

$$n<c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)$$ (20)

for a sufficiently small constant c₀. Let $\Vert \cdot \Vert ={\Vert \cdot \Vert }_1+\sqrt{s/s_g}{\Vert \cdot \Vert }_{1,2}$ and $B=\left\{x\in {ℝ}^{db}:\Vert x\Vert \le 1\right\}$ be the unit ball associated with ‖·‖. Define

$$d^n\left(B,{ℝ}^p\right)=\underset{L^n\text{ is a subspace of }{ℝ}^p\text{ with dim}\left({ℝ}^p/L^n\right)\le n}{\text{inf}}\left\{\underset{\beta \in B\cap L^n}{\text{sup}}{\Vert \beta \Vert }_2\right\},$$

We have $d^n\left(B,{ℝ}^p\right)\le \frac{C}{\sqrt{s}}$ by the assumption of this theorem. We can also show that there exists a uniform constant c > 0 such that

$$d^n\left(B,{ℝ}^p\right)\ge c\text{\hspace{0.17em}min}\left\{\frac{1}{\sqrt{s_0}},[(\frac{s_g}{s}\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right){+\text{log}\left(e{s}_{g}b/s\right))/n]}^{1/2}\right\}.$$

The previous two inequalities and (20) together imply that

$$\begin{gathered} n\ge c\left(s_g\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right) \\ \ge c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)>n. \end{gathered}$$

This contradiction shows that

$$n\ge c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right).$$

III. Sparse Group Lasso in Noisy Case

We now turn to the noisy case.

A. Optimal Rate of Estimation Error of Sparse Group Lasso

When observations are noisy, we have the following theoretical guarantee for the sparse group Lasso.

Theorem 4 (Upper bound of estimation error): Suppose y = Xβ* + ε, X satisfies Assumption 1, n ≥ C (s_g log(d/s_g) + s log(es_gb)) for some uniform constant C > 0, $\epsilon \stackrel{iid}{~}N\left(0,{\sigma }^2\right)$, and b = max_1≤i≤d b_i. Then the sparse group Lasso estimator (4) with

$$\lambda =C\sigma \sqrt{\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)\right)n/s}$$

and

$${\lambda }_g=\sqrt{s/s_g}\lambda$$

satisfies

$${\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2\lesssim {\text{min}}_{S\in 𝒮}\left\{\sqrt{\frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)\right)}{n}}+\frac{{\Vert {\beta }_{S^c}^{*}\Vert }_1}{\sqrt{s}}+\frac{{\Vert {\beta }_{S^c}^{*}\Vert }_{1,2}}{\sqrt{s_g}}\right\}$$

with probability at least

$$1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right).$$

Here,

$$𝒮=\left\{S:{\Vert {\beta }_S^{*}\Vert }_0\le s,{\Vert {\beta }_S^{*}\Vert }_{0,2}\le s_g,\underset{i\in S^c}{\text{max\hspace{0.17em}}}{\Vert {\Sigma }_{i,S}{\Sigma }_{S,S}^{-1}\Vert }_2\le c/\sqrt{s}\right\}.$$

Especially, if β* is exactly (s, s_g)-sparse and

$$\underset{i\in T^c}{\text{max}}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\le c/\sqrt{s}$$

holds, then

$${\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2^2\lesssim \frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)\right)}{n}$$ (21)

with probability at least

$$1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right).$$

In addition, we focus on the following class of simultaneously element-wise and group-wise sparse vectors,

$${ℱ}_{s,s_g}=\left\{\beta :{\Vert \beta \Vert }_0\le s,{\Vert \beta \Vert }_{0,2}\le s_g\right\}.$$

The following minimax lower bound of estimation error holds.

Theorem 5 (Lower bound of estimation error): Suppose X satisfies Assumption 1, b₁ = ⋯ = b_d = b, and d, b ≥ 3. Then we have

$$\underset{\widehat{\beta }}{\text{inf }}\underset{\beta \in {ℱ}_{s,s_g}}{\text{sup }}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2^2\gtrsim \frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right)}{n}.$$

Remark 3: Theorems 4 and 5 together show that the sparse group Lasso yields the minimax optimal rate of convergence as long as the following condition holds: log(es_gb) ≍ log(es_gb) − log(s) or log(d) ≳ s log(s)/s_g.

Remark 4: We briefly discuss the main proof ideas of Theorem 5 here. First, we randomly generate a series of subsets Ω⁽ⁱ⁾ ⊆ {1, …, p} as feasible supports of (s, s_g)-sparse vectors. Then, we prove by a probabilistic argument that there exist N ≳ (s_g log(d/s_g) + s log(es_gb/s)) subsets ${\left\{{\Omega }^{(i)}\right\}}_{i=1}^N$ such that |Ω⁽ⁱ⁾ ∩ Ω^(j)| < 8s_g⌊s/s_g⌋/9 for any i < j. Next, we construct a series of candidate (s, s_g)-sparse vectors β⁽ⁱ⁾ such that ${\beta }_k^{(i)}=\tau 1_{\left\{k\in {\Omega }^{(i)}\right\}}$. Intuitively speaking, ${\left\{{\beta }^{(i)}\right\}}_{i=1}^N$ are non-distinguishable based only on observations (y, X) by such a construction. Theorem 5 then follows by choosing an appropriate τ and the generalized Fano’s lemma.

B. Statistical Inference via Debiased Sparse Group Lasso

We further consider the statistical inference for β* under the double sparse linear regression model. First, let $\widehat{\beta }$ be the sparse group Lasso estimator given by (4). Inspired by the recent advances in inference for high-dimensional linear regression [30], [53], [31], [33], we propose the following debiased sparse group Lasso estimator,

$${\widehat{\beta }}^u=\widehat{\beta }+\frac{1}{n}\widehat{M}{X}^{\top }(Y-X\widehat{\beta }).$$ (22)

Here, $\widehat{\Sigma }=\frac{1}{n}{\sum }_{k=1}^n{X}_k{X}_k^{\top }$ is the sample covariance matrix and $\widehat{M}={\left[\begin{array}{lll}{\widehat{m}}_1\hfill & \cdots \hfill & {\widehat{m}}_p\hfill \end{array}\right]}^{\top }$ is an approximation of the inverse covariance matrix Σ⁻¹, where ${\widehat{m}}_i$ is the solution to the following convex optimization,

$$\begin{gathered} \text{minimize\hspace{1em}}{m}^{\top }\widehat{\Sigma }m \\ \text{subject to }{\Vert H_{\alpha }\left(\widehat{\Sigma }m-e_i\right)\Vert }_{\infty ,2}\le \gamma . \end{gathered}$$ (23)

Here, H_α is the soft-thresholding operator with thresholding level α defined at the beginning of Section II and e_i is the i-th vector in the canonical basis of ${ℝ}^p$. The following theorem establishes an asymptotic result for debiased sparse group Lasso.

Theorem 6 (Asymptotic distribution of debiased sparse group Lasso): Suppose ${\beta }^{*}\in {ℝ}^p$ is (s, s_g)-sparse, $X\in {ℝ}^{n\times p}$ satisfies Assumption 1, and ${\text{max}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\le c/\sqrt{s}$. Set $\lambda =C\sigma \sqrt{\frac{\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)n}{s}}$ and ${\lambda }_g=\sqrt{\frac{s}{s_g}}\lambda$ in (4), $\alpha =\frac{\lambda }{n\sigma }$, $\gamma =\sqrt{\frac{s}{s_g}}\frac{\lambda }{n\sigma }$ in (23). Then with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$, the debiased sparse group Lasso estimator ${\widehat{\beta }}^u$ can be decomposed as $\sqrt{n}\left({\widehat{\beta }}^u-{\beta }^{*}\right)=\Delta +w$, where

$$\begin{gathered} {\Vert \Delta \Vert }_{\infty }\le \frac{C\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)\right)}{\sqrt{n}}\sigma , \\ w\mid X~N\left(0,{\sigma }^2\widehat{M}\widehat{\Sigma }{\widehat{M}}^{\top }\right). \end{gathered}$$ (24)

In particular, if $\sqrt{n}\gg s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)$, for any 1 ≤ i ≤ p,

$$\frac{\sqrt{n}\left({\widehat{\beta }}_i^u-{\beta }_i^{*}\right)}{\sqrt{{\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i}}\to N\left(0,{\sigma }^2\right).$$ (25)

Remark 5: (25) provides a method to construct confidence intervals for β*. Specifically if $\widehat{\sigma }$ is a consistent estimator of σ, such as the scaled sparse group Lasso to be discussed in Section V,

$$\left[{\widehat{\beta }}_i^u-{\Phi }^{-1}(1-\alpha /2)\widehat{\sigma }\sqrt{\frac{{\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i}{n}},{\widehat{\beta }}_i^u+{\Phi }^{-1}(1-\alpha /2)\widehat{\sigma }\sqrt{\frac{{\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i}{n}}\right]$$

would be an asymptotic (1 – α)-confidence interval for ${\beta }_i^{*}$. We can see that the debiased sparse group Lasso estimator has the provably advantage on sample complexity (n ≫ (s log(es_gb) + s_g log(ed/s_g))²) over the ones via debiased Lasso (n ≫ s log p, see [30], [31], [33]) or debiased group Lasso (n ≫ (s_gb + s_g log p)², see [32]) for constructing asymptotic confidence intervals of β*.

IV. Simulation Studies

In this section, we investigate the numerical performance of the sparse group Lasso and ℓ₁ +ℓ_1,2 minimization for double sparse regression. The results support our theoretical findings in Sections II and III. We first discuss the practical choice for the tuning parameters used in the proposed algorithms.

A. Practical Selection of Tuning Parameters

By introducing τ as a surrogate for (λ_g/λ)², we can rewrite the ℓ₁ + ℓ_1,2 minimization and the sparse group Lasso as

$$\widehat{\beta }=\text{arg min }{\Vert \beta \Vert }_1+\sqrt{\tau }{\Vert \beta \Vert }_{1,2}\text{\hspace{1em} subject to \hspace{1em}}y=X\beta ,$$ (26)

$$\widehat{\beta }=\underset{\beta }{\text{arg min }}{\Vert y-X\beta \Vert }_2^2+\lambda {\Vert \beta \Vert }_1+\lambda \sqrt{\tau }{\Vert \beta \Vert }_{1,2}.$$ (27)

As suggested by Theorems 1 and 4, the theoretical choice of the tuning parameters (λ, τ) relies on σ, s, and s_g in sparse group Lasso and ℓ₁ + ℓ_1,2 minimization for double sparse regression. These values, however, are usually unknown in practice. In addition, those theoretical values of tuning parameters may not achieve the best finite-sample numerical performance. We thus introduce in this section a data-driven approach to tuning parameter selection using K-fold cross-validation.

We first discuss how to select τ in the ℓ₁ + ℓ_1,2 minimization (26). Recall n is the sample size, p is the total number of covariates, d is the number of groups, b₁, …, b_d are the number of covariates in each group, and b = max_j b_j. Since the theoretical value τ = s/s_g and s/s_g must satisfy 1 ≤ s/s_g ≤ b, for a given integer L ≥ 1, we introduce a grid

$$S_0=\left\{b^{(l-1)/(L-1)}:1\le l\le L\right\}$$ (28)

as a set of candidate values for τ. Here, the grid size L can be set to a typical value of 10, or a larger value if more computing power is available. We split the data ${\left\{X_i,y_i\right\}}_{i=1}^n$ into K groups. For 1 ≤ k ≤ K, let J_k ⊂ {1, …, n} be the index set of the kth group and $J_k^c=\left\{1,\dots ,n\right\}\backslash J_k$. For each τ ∈ S₀, we solve

$$\begin{gathered} {\widehat{\beta }}^{(k)}(\tau )=\text{arg min }{\Vert \beta \Vert }_1+\sqrt{\tau }{\Vert \beta \Vert }_{1,2} \\ \text{subject to }y{J}_k^c=X_{\left[J_k^c,:\right]}\beta \end{gathered}$$

and calculate the prediction error

$$\widehat{R}(\tau )=\sum _{k=1}^K\sum _{j\in J_k}{\left(y_j-X_{[j,:]}{\widehat{\beta }}^{(k)}(\tau )\right)}^2.$$

Let τ_* be the minimizer of the prediction error: ${\tau }_{*}={\text{arg min }}_{\tau \in S_0}\widehat{R}(\tau )$. Then, the final estimator $\widehat{\beta }$ is calculated using (26) with τ_*.

Then we consider the sparse group Lasso (27), which includes two tuning parameters (τ, λ). We still define S₀ in (28) as a grid of candidate values of τ. Following the idea in [11, Section 3.3], for each τ ∈ S₀, we begin with a large value of λ_max(τ) so that $\widehat{\beta }$, the outcome of sparse group Lasso (27) with tuning parameters (τ, λ_max(τ)), is zero (this can be achieved by the SGL package¹). Let λ_min(τ) be a small fraction of λ_max(τ) (e.g., λ_min = 0.1λ_max as suggested in [11, Section 5]). Then we define

$$\Lambda (\tau )=\left\{{\left\{{\lambda }_{\text{min}}(\tau )\right\}}^{(L-l)/(L-1)}\cdot {\left\{{\lambda }_{\text{max}}(\tau )\right\}}^{(l-1)/(L-1)}:l=1,\dots ,L\right\}.$$

Next, we split the data ${\left\{X_i,y_i\right\}}_{i=1}^n$ into K groups. For 1 ≤ k ≤ K, let J_k ⊂ {1, …, n} be the index set of the kth group and $J_k^c=\left\{1,\dots ,n\right\}\backslash J_k$. For each τ ∈ S₀, λ ∈ Λ(τ), and k ∈ {1, …, K}, we solve

$${\widehat{\beta }}^{(k)}(\tau ,\lambda )=\underset{\beta }{\text{arg min }}\left\{{\Vert y_{J_k^c}-X_{\left[J_k^c,\right]]}\beta \Vert }_2^2+\lambda {\Vert \beta \Vert }_1+\lambda \sqrt{\tau }{\Vert \beta \Vert }_{1,2}\right\}$$

and calculate the prediction error

$$\widehat{R}(\tau ,\lambda )=\sum _{k=1}^K\sum _{j\in J_k}{\left(y_j-X_{[j,:]}{\widehat{\beta }}^{(k)}(\tau ,\lambda )\right)}^2.$$

Let (τ_*, λ_*) be the minimizer of the prediction error:

$$\left({\tau }_{*},{\lambda }_{*}\right)=\underset{\tau \in S_0,\lambda \in \Lambda (\tau )}{\text{arg min }}\widehat{R}(\tau ,\lambda ).$$

The final estimator $\widehat{\beta }$ is calculated using (27) with (τ_*, λ_*).

In our simulation studies next, we will examine the performance of this cross-validation scheme with K = L = 10, λ_min = 0.1λ_max.

B. Numerical Results

We begin by considering the sample complexity for the exact recovery in the noiseless case. Suppose all group sizes are equal (b₁ = ⋯ = b_d = b) and the number of observations n varies from 5 to 200. We consider four simulation designs with (1) d = 60, b = 20, s_g = 1; (2) d = 100, b = 30, s_g = 2; (3) d = b = 20, s_g = 1; and (4) d = b = 40, s_g = 1. For each setting, we randomly draw $X\in {ℝ}^{n\times db}$ with i.i.d. standard normal entries, construct the fixed vector ${\beta }^{*}\in {ℝ}^{db}$ satisfying

$${\beta }_{(j)}^{*}=\{\begin{array}{ll}(1,2,3,4,5,0,\dots ,0)\in {ℝ}^b\hfill & j=1,\dots ,s_g;\hfill \\ 0\hfill & j=s_g+1,\dots ,d,\hfill \end{array}$$

and generate $y=X{\beta }^{*}={\sum }_{j=1}^{s_g}{X}_{(j)}{\beta }_{(j)}^{*}$. We implement the ℓ₁ + ℓ_1,2 minimization (5) with ${\lambda }_g=\sqrt{s/s_g}\lambda$ (SGL), ℓ₁ minimization (11) (Lasso), and ℓ_1,2 minimization (12) (Group Lasso), and ℓ₁ + ℓ_1,2 minimization (5) with the tuning parameter λ_g/λ selected using cross validation discussed in Section IV-A (SGL_CV). An exact recovery of β* is considered to be successful if ${\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2\le {10}^{-4}$. The successful recovery rate based on 100 replicates is shown in Figure 1. It can be seen that SGL and SGL_CV have comparable performance and both methods have significantly better performance than Lasso and Group Lasso. This is in line with our theoretical results.

Fig. 1.

Exact recovery rate in the noiseless case

Then we consider the noisy case and focus on average estimation errors of different methods. We generate

$$y=X{\beta }^{*}+\epsilon =\sum _{j=1}^{s_g}{X}_{(j)}{\beta }_{(j)}^{*}+\epsilon ,$$

where X, β* are drawn in the same way as the previous setting and $\epsilon \stackrel{iid}{~}N\left(0,{0.1}^2\right)$. We consider four designs: i. d = 60, b = 20, s_g = 1; ii. d = 100, b = 30, s_g = 2; iii. d = b = 20, s_g = 1; and iv. d = b = 40, s_g = 2. For each case, the number of observations n is chosen from an equally spaced sequence from 5 to 200 and the simulation is replicated for 500 times. We compare the average estimation error of (a) SGL_CV1: sparse group Lasso with theoretical value ${\lambda }_g=\sqrt{s/s_g}\lambda$ and λ selected via cross validation; (b) SGL_package: sparse group Lasso via SGL package² in R with the option of automatic tuning parameter selection; (c) Lasso: regular Lasso with tuning parameter selected via cross validation; (d) group Lasso: group Lasso with tuning parameter selected via cross validation; (e) SGL_CV2: sparse group Lasso with both λ and λ_g selected using the proposed cross validation scheme. We can see the proposed method SGL_CV2 achieves smaller estimation error than all other methods, including SGL_CV1, the focus of our theory. These experimental results demonstrate our theory and the applicability of the proposed cross-validation scheme.

V. Discussions

In this paper, we study the high-dimensional double sparse regression and investigate the theoretical properties of the sparse group Lasso and ℓ₁ + ℓ_1,2 minimization. Particularly, we develop the matching upper and lower bounds on the sample complexity for ℓ₁ + ℓ_1,2 minimization in the noiseless case. We also prove that the sparse group Lasso achieves minimax optimal rate of convergence in a range of settings in the noisy case. Our results give an affirmative answer to the open question for high-dimensional statistical inference for simultaneously structured model: by introducing both ℓ₁ and ℓ_1,2 penalties, one can achieve better performance on estimation and statistical inference for simultaneously element-wise and group-wise sparse vectors.

In addition to β*, the estimation and inference for noise level σ is another importance task in high-dimensional double sparse regression. Motivated by the recent development of scaled Lasso [54], one may consider the following scaled sparse group Lasso estimator:

$$\left\{{\widehat{\beta }}^s,\widehat{\sigma }\right\}=\underset{\beta \in {ℝ}^p,\sigma >0}{\text{arg min }}\left\{\frac{{\Vert y-X\beta \Vert }_2^2}{\sigma }+n\sigma +\tilde{\lambda }{\Vert \beta \Vert }_1+{\tilde{\lambda }}_g{\Vert \beta \Vert }_2\right\},$$

where $\tilde{\lambda }$. and ${\tilde{\lambda }}_g$ are tuning parameters that do not rely on σ. The consistency of $\widehat{\sigma }$ can be established based on similar ideas of scaled Lasso in the literature [54], [31] and the approximate dual certificate in this work.

Moreover, our technical results can be useful in a variety of other problems with simultaneous sparsity structures. For example, [55], [56] considered the estimation of piece-wise constant sparse signals, i.e., both the signal vector and the difference between successive entries of the signal vector are sparse. [57], [58] discussed the estimation of structured parameters where both the number of non-zero elements and the number of distinct values of the parameter vectors are small. [59] considered the estimation of matrices with simultaneous sparsity structures within each block and among different blocks. It is interesting to further study the statistical limits, including the sample complexity and minimax optimal rate of convergence for these problems. In particular, based on the specific sparsity structures of each problem, we can introduce corresponding multi-objective regularizers and the convex regularization methods. The corresponding approximate dual certificates can be proposed, constructed, and analyzed to provide strong theoretical guarantees.

VI. Proofs

We collect the proofs of technical results in this section.

A. Proof of Lemma 1

Let T satisfy (16). For convenience, we denote s₀ = s/s_g and decompose u as

$$\begin{gathered} u=v+w, \\ {v}_i=\{\begin{array}{ll}{u}_i-\sqrt{s_0}{\beta }_i^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2,\hfill & i\in T,i\in (j);\hfill \\ u_i,\hfill & i\in (G)\backslash T;\hfill \\ u_i-H_{1/2}\left(u_i\right),\hfill & i\in \left(G^c\right).\hfill \end{array} \\ {w}_{(j)}=\{\begin{array}{ll}\sqrt{s_0}{\beta }_{T,(j)}^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2,\hfill & j\in G;\hfill \\ H_{1/2}\left(u_{(j)}\right),\hfill & j\notin G.\hfill \end{array} \end{gathered}$$ (29)

Note that |H_1/2(x) − x| ≤ 1/2 for any $x\in ℝ$. Based on the property of (18), ‖u_(G)\T‖_∞ ≤ 1/2, then

$$\begin{gathered} \underset{i\in T^c}{\text{max\hspace{0.17em}}}\left|v_i\right|\le 1/2, \\ {\Vert v_T-\text{sgn}\left({\beta }_T^{*}\right)\Vert }_2={\Vert u_T-{\left({\tilde{u}}_0\right)}_T\Vert }_2\le \frac{c_{\text{min}}}{8{\text{\hspace{0.17em}max}}_{i\in T^c}{\Vert X_T^{\top }{X}_i/n\Vert }_2}; \end{gathered}$$ (30)

$$\begin{gathered} w_{(j)}=\sqrt{s_0}{\beta }_{T,(j)}^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2,\text{ if }j\in G; \\ {\Vert w_{(j)}\Vert }_2\le \sqrt{s_0}/2\text{, if }j\notin G. \end{gathered}$$ (31)

Suppose $\widehat{\beta }$ is the minimizer to (5), $h=\widehat{\beta }-{\beta }^{*}$, then based on the sub-differential of ‖β‖₁ and ‖β‖_1,2, we have

$$\begin{gathered} 𝒫(\widehat{\beta })={\Vert \widehat{\beta }\Vert }_1+\sqrt{s_0}{\Vert \widehat{\beta }\Vert }_{1,2}={\Vert {\beta }^{*}+h\Vert }_1+\sqrt{s_0}{\Vert {\beta }^{*}+h\Vert }_{1,2} \\ \ge {\Vert {\beta }_T^{*}\Vert }_1+\text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+{\Vert h_{T^c}\Vert }_1+\sqrt{s_0}\left({\Vert {\beta }_T^{*}\Vert }_{1,2}+\sum _{j\in G}\frac{{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}+\sum _{j\notin G}{\Vert h_{(j)}\Vert }_2\right)-{\Vert {\beta }_{T^c}^{*}\Vert }_1-\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2} \\ \ge 𝒫\left({\beta }^{*}\right)+{\Vert h_{T^c}\Vert }_1+\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}+\text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}-2{\Vert {\beta }_{T^c}^{*}\Vert }_1-2\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}. \end{gathered}$$ (32)

The last inequality comes from ${\Vert {\beta }^{*}\Vert }_1={\Vert {\beta }_T^{*}\Vert }_1+{\Vert {\beta }_{T^c}^{*}\Vert }_1$ and ${\Vert {\beta }^{*}\Vert }_{1,2}\le {\Vert {\beta }_T^{*}\Vert }_{1,2}+{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}$.

In particular, given Xh = 0 and that u lies in the row span of X, we have v^⊤h + w^⊤h = u^⊤h = 0. Therefore,

$$\begin{gathered} \text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2} \\ =\text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T-v^{\top }h+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}-w^{\top }h \\ =-{\left(v_T-\text{sgn}\left({\beta }_T^{*}\right)\right)}^{\top }{h}_T-v_{T^c}^{\top }{h}_{T^c}-\sum _{j\in G}{\left(w_{(j)}-\sqrt{s_0}{\beta }_{T,(j)}^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2\right)}^{\top }{h}_{(j)}-{\left(w_{\left(G^c\right)}\right)}^{\top }{h}_{\left(G^c\right)} \\ \ge -{\Vert v_T-\text{sgn}\left({\beta }_T^{*}\right)\Vert }_2{\Vert h_T\Vert }_2-{\Vert v_{T^c}\Vert }_{\infty }\cdot {\Vert h_{T^c}\Vert }_1-\underset{j\in G}{\text{max\hspace{0.17em}}}\left\{{\Vert w_{(j)}-\sqrt{s_0}{\beta }_{T,(j)}^{*}/{\Vert {\beta }_{T,(j)}^{*}\Vert }_2\Vert }_2\cdot {\Vert h_{(G)}\Vert }_{1,2}\right\}-{\Vert w_{\left(G^c\right)}\Vert }_{\infty ,2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \stackrel{(30)(31)}{\ge }-{\Vert v_T-\text{sgn}\left({\beta }_T^{*}\right)\Vert }_2\cdot {\Vert h_T\Vert }_2-{\Vert h_{T^c}\Vert }_1/2-\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2. \end{gathered}$$ (33)

Next note that $h=h_T+h_{T^c}$, we must have $X_T{h}_T=-X_{T^c}+h_{T^c}$, then

$$\begin{gathered} {\Vert h_T\Vert }_2={\Vert {\left(X_T^{\top }{X}_T/n\right)}^{-1}{X}_T^{\top }{X}_T{h}_T/n\Vert }_2 \\ \le {\sigma }_{\text{min}}^{-1}\left(X_T^{\top }{X}_T/n\right){\Vert X_T{X}_{T^c}{h}_{T^c}/n\Vert }_2 \\ \le \frac{2}{c_{\text{min}}}\cdot \underset{i\in T^c}{\text{max\hspace{0.17em}}}{\Vert X_T^{\top }{X}_i/n\Vert }_2\cdot {\Vert h_{T^c}\Vert }_1. \end{gathered}$$ (34)

Combining (30), (33), and (34), one obtains

$$\text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}\ge -3/4\cdot {\Vert h_{T^c}\Vert }_1-\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2.$$

Plug this inequality to (32), we finally have

$$𝒫(\widehat{\beta })\ge 𝒫\left({\beta }^{*}\right)+{\Vert h_{T^c}\Vert }_1/4+\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2-2{\Vert {\beta }_{T^c}^{*}\Vert }_1-2\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}.$$

Since $\widehat{\beta }$ is the minimizer to (5), we must have $𝒫(\widehat{\beta })\le 𝒫\left({\beta }^{*}\right)$, then

$${\Vert h_{T^c}\Vert }_1/4+\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2\le 2{\Vert {\beta }_{T^c}^{*}\Vert }_1+2\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}.$$ (35)

If β* is (s, s_g)-sparse, immediately we have $h_{T^c}=0$. Then $0=X_T^{\top }Xh=\left(X_T^{\top }{X}_T\right)h_T$. By ${\sigma }_{\text{min}}\left(X_T^{\top }{X}_T/n\right)\ge c_{\text{min}}/2$, we know $X_T^{\top }{X}_T/n$ is non-singular, then h_T = 0.

Now, we consider the general case. Without loss of generality, suppose G = {1, …, g}, where g ≤ s_g. Denote T₁ as the indices of the s largest entries of h_(G)\T, T₂ as the indices of the s largest entries of $h_{(G)\backslash \left[T\cup T_1\right]}$, and so on. For s_g + 1 ≤ i ≤ d, denote S_i,1 as the indices of the ⌊s/s_g⌋ largest entries of h_(i), S_i,2 as the indices of the ⌊s/s_g⌋ largest entries of $h_{(i)\backslash S_{i,1}}$, and so on. Let ${\tilde{S}}_1,\dots ,{\tilde{S}}_{{\sum }_{i=g+1}^d\lceil b_i/\lfloor s/s_g\rfloor \rceil }$ be an arrangement of S_i,j(1 ≤ j ≤ ⌈b_i/⌊s/s_g⌋⌉, g + 1 ≤ i ≤ d) such that ${\Vert h_{{\tilde{S}}_1}\Vert }_2^2\ge \cdots \ge {\Vert h_{{\tilde{S}}_{{\sum }_{i=g+1}^d\lceil b_i/\lfloor s/s_g\rfloor \rceil }}\Vert }_2^2$. Let $R_1={\cup }_{i=1}^{s_g}{\tilde{S}}_i$, $R_2={\cup }_{i=s_g+1}^{2s_g}{\tilde{S}}_i$, and so on. Then (T₁, T₂, …, R₁, R₂,…) is a partition of T^c, and |T_i|, |R_j| ≤ s, |g(T_i)|, |g(R_j)| ≤ s_g, where g(S) = {i₁, …, i_k} if $S\subseteq {\cup }_{j=1}^k\left(i_j\right)$ and S ∩ (i_j) are not empty for all 1 ≤ j ≤ k. Let T = T ∪ T₁ ∪ R₁. If (19) holds, then

$$\begin{gathered} \frac{c_{\text{min}}}{2}{\Vert h_{\tilde{T}}\Vert }_2^2\le \frac{1}{n}{\Vert X_{\tilde{T}}{h}_{\tilde{T}}\Vert }_2^2 \\ =\frac{1}{n}\langle X_{\tilde{T}}{h}_{\tilde{T}},Xh\rangle -\frac{1}{n}\langle X_{\tilde{T}}{h}_{\tilde{T}},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle . \end{gathered}$$ (36)

Since Xh = 0, we have

$$\langle X_{\tilde{T}}{h}_{\tilde{T}},Xh\rangle =0.$$ (37)

Now, we consider $\left|\langle X_{\tilde{T}}{h}_{\tilde{T}},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right|$. By triangle inequality,

$$\begin{gathered} \left|\langle X_{\tilde{T}}{h}_{\tilde{T}},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le \left|\langle X_T{h}_T,X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right|+\left|\langle X_{T_1}{h}_{T_1},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right|+\left|\langle X_{R_1}{h}_{R_1},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right|. \end{gathered}$$

The triangle inequality shows that

$$\begin{gathered} \left|\langle X_T{h}_T,X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le \sum _{i\ge 2}\left|\langle X_T{h}_T,X_{T_i}{h}_{T_i}\rangle \right|+\sum _{j\ge 2}\left|\langle X_T{h}_T,X_{R_j}{h}_{R_j}\rangle \right|. \end{gathered}$$

Combine the parallelogram identity and (19) together, we have

$$\begin{gathered} \left|\langle X_T{h}_T,X_{T_i}{h}_{T_i}\rangle \right|\le C_{\text{max}}n{\Vert h_T\Vert }_2{\Vert h_{T_i}\Vert }_2, \\ \left|\langle X_T{h}_T,X_{R_j}{h}_{R_j}\rangle \right|\le C_{\text{max}}n{\Vert h_T\Vert }_2{\Vert h_{R_j}\Vert }_2. \end{gathered}$$

Thus,

$$\begin{gathered} \left|\langle X_T{h}_T,X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le C_{\text{max}}n{\Vert h_T\Vert }_2\left(\sum _{i\ge 2}{\Vert h_{T_i}\Vert }_2+\sum _{j\ge 2}{\Vert h_{R_j}\Vert }_2\right). \end{gathered}$$ (38)

By (3.10) in [37], we have

$$\sum _{i\ge 2}{\Vert h_{T_i}\Vert }_2\le s^{-1/2}{\Vert h_{(G)\backslash T}\Vert }_1,$$ (39)

and

$$\begin{gathered} \sum _{j\ge 2}{\Vert h_{R_j}\Vert }_2=\sum _{j\ge 2}{\left(\sum _{i=(j-1)s_g+1}^{j{s}_g}{\Vert h_{{\tilde{S}}_i}\Vert }_2^2\right)}^{1/2} \\ \le \sum _{j\ge 2}\sqrt{s_g}{\Vert h_{{\tilde{S}}_{(j-1)s_g}}\Vert }_2 \\ \le \sum _{j\ge 2}\sqrt{s_g}\sum _{i=(j-2)s_g+1}^{(j-1)s_g}{\Vert h_{{\tilde{S}}_i}\Vert }_2/s_g \\ =s_g^{-1/2}\sum _k{\Vert h_{{\tilde{S}}_k}\Vert }_2=s_g^{-1/2}\sum _{i=g+1}^d\sum _j{\Vert h_{S_{i,j}}\Vert }_2. \end{gathered}$$

For all g +1 ≤ i ≤ d, apply (3.10) in [37] again,

$$\begin{gathered} \sum _{j\ge 2}{\Vert h_{S_{i,j}}\Vert }_2\le {\left(\lfloor s/s_g\rfloor \right)}^{-1/2}{\Vert h_{(i)}\Vert }_1 \\ \le \sqrt{2}{\left(s/s_g\right)}^{-1/2}{\Vert h_{(i)}\Vert }_1. \end{gathered}$$

Moreover, by the definition of S_i,1,

$$\sum _{i=g+1}^d{\Vert h_{S_{i,1}}\Vert }_2\le \sum _{i=g+1}^d{\Vert h_{(i)}\Vert }_2={\Vert h_{\left(G^c\right)}\Vert }_{1,2}.$$

Therefore,

$$\begin{gathered} \sum _{j\ge 2}{\Vert h_{R_j}\Vert }_2\le s_g^{-1/2}\left(\sum _{i=g+1}^d\sqrt{2}{\left(s/s_g\right)}^{-1/2}{\Vert h_{(i)}\Vert }_1\right)+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ =\sqrt{2}{s}^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}. \end{gathered}$$ (40)

Combine (38), (39) and (40) together, if (19) holds, we have

$$\begin{gathered} \left|\langle X_T{h}_T,X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le \left(s^{-1/2}{\Vert h_{(G)\backslash T}\Vert }_1+\sqrt{2}{s}^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right)\cdot C_{\text{max}}n{\Vert h_T\Vert }_2 \\ \le C_{\text{max}}n{\Vert h_T\Vert }_2\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right). \end{gathered}$$

Similarly, if (19) holds, then

$$\begin{gathered} \left|\langle X_{T_1}{h}_{T_1},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le C_{\text{max}}n{\Vert h_{T_1}\Vert }_2\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right) \end{gathered}$$

and

$$\begin{gathered} \left|\langle X_{R_1}{h}_{R_1},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le C_{\text{max}}n{\Vert h_{R_1}\Vert }_2\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right). \end{gathered}$$

Thus, with probability at least 1 – 2e^−cn,

$$\begin{gathered} \left|\langle X_{\tilde{T}}{h}_{\tilde{T}},X_{{\tilde{T}}^c}{h}_{{\tilde{T}}^c}\rangle \right| \\ \le C_{\text{max}}n\left({\Vert h_T\Vert }_2+{\Vert h_{T_1}\Vert }_2+{\Vert h_{R_1}\Vert }_2\right)\cdot \left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right) \\ \le \sqrt{3}{C}_{\text{max}}n{\Vert h_{\tilde{T}}\Vert }_2\cdot \left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right). \end{gathered}$$ (41)

The last inequality holds due to Cauchy-Schwarz inequality. Combine (36), (37), (41) and Lemma 3 together, we know that with probability at least 1 – 2e^−cn,

$$\begin{gathered} \frac{c_{\text{min}}}{2}{\Vert h_{\tilde{T}}\Vert }_2^2 \\ \le \sqrt{3}{C}_{\text{max}}{\Vert h_{\tilde{T}}\Vert }_2\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right), \end{gathered}$$

i.e., with probability at least 1 – 2e^−cn,

$${\Vert h_{\tilde{T}}\Vert }_2\le 2\sqrt{3}\frac{C_{\text{max}}}{c_{\text{min}}}\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right).$$

Finally, by (35), (39), (40) and the previous inequality, with probability at least 1 – 2e^−cn,

$$\begin{gathered} \Vert h{\Vert }_2\le {\Vert h_{\tilde{T}}\Vert }_2+\sum _{i\ge 2}{\Vert h_{T_i}\Vert }_2+\sum _{j\ge 2}{\Vert h_{R_j}\Vert }_2 \\ \le 2\sqrt{3}\frac{C_{\text{max}}}{c_{\text{min}}}\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right)+\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_2+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le C\left(\frac{1}{\sqrt{s}}{\Vert {\beta }_{T^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right). \end{gathered}$$

In summary, we have finished the proof of this lemma. □

B. Proof of Lemma 2

Let T satisfy (16). Given ${\Vert {\beta }_T^{*}\Vert }_{0,2}\le s_g$, without loss of generally we assume that

$${\beta }_{T,\left(s_g+1\right)}^{*},\cdots ,{\beta }_{T,(d)}^{*}=0.$$

We also denote T_(j) as the support of ${\beta }_{T,(j)}^{*}$. First by Lemma 6 Part 3 with

$$\begin{gathered} v\in {ℝ}^p,v_k=\{\begin{array}{ll}1,\hfill & k=i;\hfill \\ 0,\hfill & k\ne i;\hfill \end{array} \\ U\in {ℝ}^{p\times \left|T\right|}={ℝ}^{\left({\sum }_{i=1}^d{b}_i\right)\times \left|T\right|},U_{[T,:]}=I;U_{\left[T^c,:\right]}=0, \end{gathered}$$

and notice that x log(eu/x) ≥ log(eu) for all 1 ≤ x ≤ u, we have

$$\begin{gathered} \mathbb{P}\left(\underset{i\in T^c}{\text{max}}{\Vert X_T^{\top }{X}_i/n\Vert }_2\ge 1/2\right) \\ \le \sum _{i\in T^c}\mathbb{P}\left({\Vert X_T^{\top }{X}_i/n\Vert }_2\ge 1/2\right) \\ \le \sum _{i\in T^c}\mathbb{P}\left({\Vert X_T^{\top }{X}_i/n-\mathbb{E}{X}_T^{\top }{X}_i/n\Vert }_2+{\Vert \mathbb{E}{X}_T^{\top }{X}_i/n\Vert }_2\ge 1/2\right) \\ \le \sum _{i\in T^c}\mathbb{P}({\Vert X_T^{\top }{X}_i/n-\mathbb{E}{X}_T^{\top }{X}_i/n\Vert }_2\ge 1/2-\Vert {\Sigma }_{T,T}\Vert {\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2) \\ \le \sum _{i\in T^c}\mathbb{P}\left({\Vert X_T^{\top }{X}_i/n-\mathbb{E}{X}_T^{\top }{X}_i/n\Vert }_2\ge 1/4\right) \\ \le db\cdot C\text{\hspace{0.17em}exp}(Cs-n) \\ \le C\text{\hspace{0.17em}exp}(\text{\hspace{0.17em}log}(d)+\text{log}(b)+Cs-n) \\ \le C\text{\hspace{0.17em}exp}\left(s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+Cs-n\right) \\ \le C\text{\hspace{0.17em}exp}(-cn) \end{gathered}$$ (42)

provided that n ≥ C (s log(es_gb/s) + s_g log(d/s_g)) for some large constant C > 0. Note that the fourth inequality comes from the facts that ‖Σ_T,_T‖ ≤ ‖Σ‖ ≤ C_max and ${\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\le c/\sqrt{s}\le 1/\left(4C_{\text{max}}\right)$. By Lemma 7 Part 1, we also know

$$\begin{gathered} \mathbb{P}\left({\sigma }_{\text{min}}\left(X_T^{\top }{X}_T/n\right)\le c_{\text{min}}/2\right) \\ \le \mathbb{P}\left(\Vert X_T^{\top }{X}_T/n-{\Sigma }_{T,T}\Vert \ge c_{\text{min}}/2\right) \\ \le \mathbb{P}\left(\Vert X_T^{\top }{X}_T{\Sigma }_{T,T}^{-1}/n-I_{\left|T\right|}\Vert \Vert {\Sigma }_{T,T}\Vert \ge c_{\text{min}}/2\right) \\ \le \mathbb{P}\left(\Vert X_T^{\top }{X}_T{\Sigma }_{T,T}^{-1}/n-I_{\left|T\right|}\Vert \ge c_{\text{min}}/\left(2C_{\text{max}}\right)\right) \\ \le C\text{\hspace{0.17em}exp}(Cs-cn)\le C\text{\hspace{0.17em}exp}(-cn) \end{gathered}$$

Next, we apply the well-regarded golfing scheme [50], [44] to find an approximate dual certificate u that satisfies (18). Let $u_0\in {ℝ}^p$ such that

$${\left(u_0\right)}_{(j)}=\{\begin{array}{ll}\frac{\sqrt{s/s_g}{\beta }_{T,(j)}^{*}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}+\text{sgn}\left({\beta }_{T,(j)}^{*}\right),\hfill & j\in G;\hfill \\ 0,\hfill & j\in G^c.\hfill \end{array}$$ (43)

Immediately we have ${\left(u_0\right)}_T={\left({\tilde{u}}_0\right)}_T$. We divide n rows of X into non-overlapping batches, say $X_{\left[I_1,:\right]}$, $X_{\left[I_2,:\right]}$, …, with |I_l| = n_l. Here, n₁, n₂,… will be specified a little while later. Consider the following sequences

$$\begin{gathered} {\alpha }_0=u_0, \\ {\gamma }_l=X_{\left[I_l,:\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot {\left({\alpha }_{l-1}\right)}_T, \\ {\alpha }_l={\alpha }_{l-1}-{\gamma }_l,l=1,2,\dots ,l_{\text{max}}. \end{gathered}$$ (44)

Finally the approximate dual certificate is defined as

$$u=\sum _{l=1}^{l_{\text{max}}}{\gamma }_l=\sum _{l=1}^{l_{\text{max}}}{X}_{\left[I_l,:\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot {\left({\alpha }_{l-1}\right)}_T.$$ (45)

From the inductive definition we can see

$$\begin{gathered} {\left({\alpha }_l\right)}_T=\left(I-X_{\left[I_l,T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\right){\left({\alpha }_{l-1}\right)}_T, \\ {\left({\gamma }_l\right)}_{T^c}=X_{\left[I_l,T^c\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot {\left({\alpha }_{l-1}\right)}_T,l=1,2,\dots . \end{gathered}$$

Next, we apply the random matrix results (Lemmas 7 and 6) and obtain the following tail probabilities.

• if n_l ≥ C st_l for large constant C > 0 and t_l ≥ C, by Part 1 of Lemma 7,

  $$\begin{gathered} \mathbb{P}\left(\Vert X_{\left[I_l,T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l-I_{\left|T\right|}\Vert \ge C\sqrt{\frac{s{t}_l}{n_l}}\right) \\ \le C\text{\hspace{0.17em}exp}\left(Cs-n_l\text{ min}\left\{\frac{s{t}_l}{n_l},{\left(\frac{s{t}_l}{n_l}\right)}^{1/2}\right\}\right) \\ \le C\text{\hspace{0.17em}exp}\left(-cs{t}_l\right); \end{gathered}$$ (46)
• Suppose $q_{l-1}={\left({\alpha }_{l-1}\right)}_T\in {ℝ}^{\left|T\right|}$ is independent of $X_{\left[I_l,:\right]}$. If $n_l\ge \frac{C\left(s_0\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+\text{log}d\right)}{\text{min}\left\{s_0{\delta }_l^2,\sqrt{s_0}{\delta }_l\right\}}$ for ${\delta }_l\ge C{\text{ max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\ge C\left({\text{max}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\right){\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2/{\Vert q_{l-1}\Vert }_2$, by Lemma 7 Part 2,

  $$\begin{gathered} \mathbb{P}\left(\underset{j\in G^c}{\text{max\hspace{0.17em}}}{\Vert H_{{\Vert q_{l-1}\Vert }_2{\delta }_l}\cdot \left(X_{\left[I_l,(j)\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\right)\Vert }_2\ge \sqrt{s_0}{\Vert q_{l-1}\Vert }_2{\delta }_l\right) \\ \le \sum _{j\in G^c}\mathbb{P}\left({\Vert H_{{\Vert q_{l-1}\Vert }_2{\delta }_l}\cdot \left(X_{\left[I_l,(j)\right]}^{\top }{X}_{\left[I_l,T\right]}\left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)/n_l\right)\Vert }_2\ge \sqrt{s_0}{\Vert q_{l-1}\Vert }_2{\delta }_l\right) \\ \le d\cdot \left(\begin{array}{c}b\\ \lceil s_0\rceil \end{array}\right)\cdot \text{exp}\left(C{s}_0-c{n}_l\text{ min}\left\{\frac{s_0{\Vert q_{l-1}\Vert }_2^2{\delta }_l^2}{{\kappa }^4{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2^2},\frac{\sqrt{s_0}{\Vert q_{l-1}\Vert }_2{\delta }_l}{{\kappa }^2{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2}\right\}\right)+d\cdot \left(\begin{array}{c}b\\ \lfloor s_0\rfloor \end{array}\right)\cdot \text{exp}\left(C{s}_0-c{n}_l\text{\hspace{0.17em}min}\left\{\frac{s_0{\Vert q_{l-1}\Vert }_2^2{\delta }_l^2}{{\kappa }^4{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2^2},\frac{\sqrt{s_0}{\Vert q_{l-1}\Vert }_2{\delta }_l}{{\kappa }^2{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2}\right\}\right) \\ \le 2d\cdot {\left(\frac{eb}{\lfloor s_0\rfloor }\right)}^{\lceil s_0\rceil }\cdot \text{exp}\left(C{s}_0-c{n}_l\text{\hspace{0.17em}min}\left\{s_0{\delta }_l^2,\sqrt{s_0}{\delta }_l\right\}\right) \\ \le C\text{\hspace{0.17em}exp}(\text{\hspace{0.17em}log}(d)+C{s}_0\text{\hspace{0.17em}log}\left(2e{s}_{g}b/s\right)+C{s}_0-c{n}_l\text{\hspace{0.17em}min}\left\{s_0{\delta }_l^2,\sqrt{s_0}{\delta }_l\right\}) \\ \le C\text{\hspace{0.17em}exp}\left(-c{n}_l\text{\hspace{0.17em}min}\left\{s_0{\delta }_l^2,\sqrt{s_0}{\delta }_l\right\}\right); \end{gathered}$$ (47)

  The third inequality comes from $\Vert {\sum }_{T,T}^{-1}\Vert \le \frac{1}{c_{\text{min}}}$.

• Suppose $q_{l-1}={\left({\alpha }_{l-1}\right)}_T\in {ℝ}^{\left|T\right|}$ is fixed. If $n_l\text{\hspace{0.17em}min}\left\{{\theta }_l^2,{\theta }_l\right\}\ge C\text{\hspace{0.17em}log}\left(e{s}_{g}b\right),{\theta }_l\ge 2{\text{ max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2$, by Lemma 6 Part 2,

  $$\begin{gathered} \mathbb{P}\left({\Vert X_{\left[I_l,(G)\backslash T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\Vert }_{\infty }\ge {\theta }_l{\Vert q_{l-1}\Vert }_2\right) \\ \le \sum _{i\in (G)\backslash T}\mathbb{P}\left(\left|X_{\left[I_l,i\right]}^{\top }{X}_{\left[I_l,T\right]}/n_l\cdot \left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)\right|\ge {\theta }_l{\Vert q_{l-1}\Vert }_2\right) \\ \le \sum _{i\in (G)\backslash T}\mathbb{P}\left(\left|X_{\left[I_l,i\right]}^{\top }{X}_{\left[I_l,T\right]}/n_l\cdot \left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)-{\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\right|\ge {\theta }_l{\Vert q_{l-1}\Vert }_2-\left|{\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\right|\right) \\ \le \sum _{i\in (G)\backslash T}\mathbb{P}\left(\left|X_{\left[I_l,i\right]}^{\top }{X}_{\left[I_l,T\right]}/n_l\cdot \left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)-{\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\right|\ge {\theta }_l{\Vert q_{l-1}\Vert }_2-{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2{\Vert q_{l-1}\Vert }_2\right) \\ \le \sum _{i\in (G)\backslash T}\mathbb{P}\left(\left|X_{\left[I_l,i\right]}^{\top }{X}_{\left[I_l,T\right]}/n_l\cdot \left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)-{\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\right|\ge \frac{1}{2}{\theta }_l{\Vert q_{l-1}\Vert }_2\right) \\ \le \sum _{i\in (G)\backslash T}\mathbb{P}\left(\left|X_{\left[I_l,i\right]}^{\top }{X}_{\left[I_l,T\right]}/n_l\cdot \left({\Sigma }_{T,T}^{-1}{q}_{l-1}\right)-{\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\right|\ge \frac{c_{\text{min}}}{2}{\theta }_l{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2\right) \\ \le s_{g}b\cdot C\text{\hspace{0.17em}exp}\left(-c{n}_l\text{\hspace{0.17em}min}\left\{{\theta }_l^2,{\theta }_l\right\}\right) \\ =C\text{\hspace{0.17em}exp}\left(\text{\hspace{0.17em}log}\left(s_{g}b\right)-c{n}_l\text{\hspace{0.17em}min}\left\{{\theta }_l^2,{\theta }_l\right\}\right) \\ \le C\text{\hspace{0.17em}exp}\left(-c{n}_l\text{\hspace{0.17em}min}\left\{{\theta }_l^2,{\theta }_l\right\}\right). \end{gathered}$$ (48)

Then we specify {n_l, t_l, δ_l, θ_l}_{l ≥ 1} as follows,

• n₁ = n₂ ≥ C(s log(es_gb) + s_g log(d/s_g)), t₁ = t₂ = cn₁/(s log(es)) ≥ C, ${\delta }_1={\delta }_2=1/(16\sqrt{s})$, ${\theta }_1={\theta }_2=1/(16\sqrt{s})$;

• $n_3=\cdots =n_{l_{\text{max}}}\asymp \frac{n_1}{l_{\text{max}}-2}\ge C\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)/\text{\hspace{0.17em}log}(es)$, $t_3=\cdots =t_{l_{\text{max}}}=c{n}_3/s\ge C$, ${\delta }_3=\cdots ={\delta }_{l_{\text{max}}}=\text{\hspace{0.17em}log}(es)/(16\sqrt{s})\ge \text{max\hspace{0.17em}}\left\{{(\text{\hspace{0.17em}log}(es)/s)}^{1/2}/16,\text{\hspace{0.17em}log}(es)\sqrt{s_0}/(16s)\right\}$, ${\theta }_3=\cdots ={\theta }_{l_{\text{max}}}={\left(\text{\hspace{0.17em}log}(es)/s\right)}^{1/2}/16$, with l_max = ⌈C log(es)⌉ + 2.

We can see the following events happen

$$\begin{gathered} \Vert X_{\left[I_l,T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l-I_{\left|T\right|}\Vert \\ \le C\sqrt{s{t}_l/n_l}\le \sqrt{1/\text{\hspace{0.17em}log}(es)},l=1,2;\Vert X_{\left[I_l,T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l-I_{\left|T\right|}\Vert \\ \le C\sqrt{s{t}_l/n_l}\le 1/2,l=3,\dots ,l_{\text{max}}; \end{gathered}$$ (49)

$$\begin{gathered} \underset{j\in G^c}{\text{max}}{\Vert H_{{\Vert q_{l-1}\Vert }_2/(16\sqrt{s})}\cdot \left(X_{\left[I_l,(j)\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\right)\Vert }_2\le \sqrt{s_0}{\Vert q_{l-1}\Vert }_2/(16\sqrt{s}),l=1,2; \\ \underset{j\in G^c}{\text{max}}{\Vert H_{{\Vert q_{l-1}\Vert }_2\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s})}\cdot \left(X_{\left[I_l,(j)\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\right)\Vert }_2\le \sqrt{s_0}{\Vert q_{l-1}\Vert }_2\text{\hspace{0.17em}log}(es)/(16\sqrt{s}),l=3,\dots ,l_{\text{max}}; \end{gathered}$$ (50)

$$\begin{gathered} {\Vert X_{\left[I_l,(G)\backslash T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\Vert }_{\infty }\le {\Vert q_{l-1}\Vert }_2/(16\sqrt{s}),l=1,2 \\ {\Vert X_{\left[I_l,(G)\backslash T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot q_{l-1}\Vert }_{\infty }\le {\Vert q_{l-1}\Vert }_2\cdot {(\text{\hspace{0.17em}log}(es)/s)}^{1/2}/16,l=3,\dots ,l_{\text{max}}. \end{gathered}$$ (51)

with probability at least $1-C\text{\hspace{0.17em}log}(es)\text{exp}\left(-c\frac{n}{\text{\hspace{0.17em}log}(es)}\right)-C\text{\hspace{0.17em}log}(es)\text{exp}\left(-c\frac{n}{s_g}\right)-C\text{\hspace{0.17em}log}(es)\text{exp}\left(-c\frac{n}{s}\right)$. By triangle inequality, u₀ satisfies

$$\begin{gathered} {\Vert u_0\Vert }_2\le \sqrt{s/s_g}{\left(\sum _{j\in G}{\Vert \frac{{\beta }_{T,(j)}^{*}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}\Vert }_2^2\right)}^{1/2}+{\Vert \text{sgn}\left({\beta }_T^{*}\right)\Vert }_2 \\ \le 2\sqrt{s}. \end{gathered}$$ (52)

When ${\text{max}}_{i\in T^c}{\Vert X_T^{\top }{X}_i/n\Vert }_2\le \frac{1}{2}$ and (49)–(52) hold, we have

$$\begin{gathered} {\Vert q_0\Vert }_2\le 2\sqrt{s}, \\ {\Vert q_1\Vert }_2=\Vert \left(I_{\left|T\right|}-X_{I_1,T}^{\top }{X}_{I_1,T}{\Sigma }_{T,T}^{-1}/n_1\right)q_0\Vert \\ \le \Vert I_{\left|T\right|}-X_{I_1,T}^{\top }{X}_{I_1,T}{\Sigma }_{T,T}^{-1}/n_1\Vert \cdot {\Vert q_0\Vert }_2 \\ \le 2\sqrt{s/\text{\hspace{0.17em}log}(es)}; \\ \text{similarly, }{\Vert q_2\Vert }_2\le {\Vert q_1\Vert }_2/\sqrt{\text{\hspace{0.17em}log}(es)}\le 2\sqrt{s}/(\text{\hspace{0.17em}log}(es)); \\ {\Vert q_l\Vert }_2\le {\Vert q_{l-1}\Vert }_2/2\le \cdots \le \Vert q_2\Vert /2^{l-2} \\ \le 2^{3-l}\sqrt{s}/(\text{\hspace{0.17em}log}(es)),l\ge 3. \end{gathered}$$ (53)

For large constant $C>0,{\Vert q_{l_{\text{max}}}\Vert }_2\le 2^{3-C\text{\hspace{0.17em}log}(es)}\sqrt{s}/\text{\hspace{0.17em}log}(es)\le c_{\text{min}}/8$. Notice that

$$\begin{gathered} u_T={\left(\sum _{l=1}^{l_{\text{max}}}{\gamma }_l\right)}_T={\left(\sum _{l=1}^{l_{\text{max}}}\left({\alpha }_{l-1}-{\alpha }_l\right)\right)}_T={\left({\alpha }_0-{\alpha }_{l_{\text{max}}}\right)}_T \\ ={\left({\tilde{u}}_0\right)}_T-{\left(q_{l_{\text{max}}}\right)}_T, \end{gathered}$$

we know that

$$\begin{gathered} {\Vert u_T-{\left({\tilde{u}}_0\right)}_T\Vert }_2\cdot \underset{i\in T^c}{\text{max\hspace{0.17em}}}{\Vert X_T^{\top }{X}_i/n\Vert }_2 \\ ={\Vert q_{l_{\text{max}}}\Vert }_2\cdot \underset{i\in T^c}{\text{max\hspace{0.17em}}}{\Vert X_T^{\top }{X}_i/n\Vert }_2\le \frac{c_{\text{min}}}{8}\cdot \frac{1}{2}<\frac{c_{\text{min}}}{8}. \end{gathered}$$

In addition,

$$\begin{gathered} {\Vert u_{(G)\backslash T}\Vert }_{\infty } \\ \le \sum _{l=1}^{l_{\text{max}}}{\Vert X_{\left[I_l,(G)\backslash T\right]}^{\top }{X}_{\left[I_l,T\right]}{\Sigma }_{T,T}^{-1}/n_l\cdot {\left({\alpha }_{l-1}\right)}_T\Vert }_{\infty } \\ \le {\Vert q_0\Vert }_2/(16\sqrt{s})+{\Vert q_1\Vert }_2/(16\sqrt{s})+\sum _{l=3}^{l_{\text{max}}}{\Vert q_{l-1}\Vert }_2\cdot {(\text{\hspace{0.17em}log}(es)/s)}^{1/2}/16 \\ \le 1/8+1/8+\sum _{l=3}^{\infty }{2}^{4-l}/16\le 1/2. \end{gathered}$$

Since

$$\begin{gathered} {\Vert q_0\Vert }_2/(16\sqrt{s})+{\Vert q_1\Vert }_2/(16\sqrt{s}) \\ +\sum _{l=3}^{l_{\text{max}}}{\Vert q_{l-1}\Vert }_2\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s}) \\ \le \frac{1}{8}+\frac{1}{8}+\sum _{l=3}^{l_{\text{max}}}{2}^{4-l}\sqrt{s}/(\text{\hspace{0.17em}log}(es))\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s})\le \frac{1}{2}, \end{gathered}$$

$$\begin{gathered} {\Vert H_{1/2}\left(u_{\left(G^c\right)}\right)\Vert }_{\infty ,2} \\ \le {\Vert H_{\nu }\left(u_{\left(G^c\right)}\right)\Vert }_{\infty ,2} \\ \le \sum _{l=3}^{l_{\text{max}}}{\Vert H_{{\Vert q_{l-1}\Vert }_2\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s})}\left(X_{\left[I_l,\left(G^c\right)\right]}^{\top }{X}_{\left[I_l,T^c\right]}{q}_{l-1}\right)\Vert }_{\infty ,2}+\sum _{l=1}^2{\Vert H_{{\Vert q_{l-1}\Vert }_2/(16\sqrt{s})}\left(X_{\left[I_l,\left(G^c\right)\right]}^{\top }{X}_{\left[I_l,T^c\right]}{q}_{l-1}\right)\Vert }_{\infty ,2} \\ \le \sum _{l=1}^2\sqrt{s_0}{\Vert q_{l-1}\Vert }_2/(16\sqrt{s})+\sum _{l=3}^{l_{\text{max}}}\sqrt{s_0}{\Vert q_{l-1}\Vert }_2\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s}) \\ \le \sqrt{s_0}/2, \end{gathered}$$

where

$$\nu ={\Vert q_0\Vert }_2/(16\sqrt{s})+{\Vert q_1\Vert }_2/(16\sqrt{s})+\sum _{l=3}^{l_{\text{max}}}{\Vert q_{l-1}\Vert }_2\cdot \text{\hspace{0.17em}log}(es)/(16\sqrt{s}).$$

Thus, the construction of u satisfies all required condition in Lemma 1 with probability at least $1-C\text{\hspace{0.17em}exp}\left(-c\frac{n}{s}\right)$. This has finished the proof of this lemma. □

C. Proof of Lemma 3

Let g(S) be the group support of set S, that is, g(S) = {i₁, …, i_k} if $S\subset {\cup }_{j=1}^k\left(i_j\right)$ and S ∩ (i_j) are not empty for all 1 ≤ j ≤ k. Lemma 7 Part 1 and the union bound show that

$$\begin{gathered} \mathbb{P}\left(\exists \gamma \in {ℝ}^p,{\Vert \gamma \Vert }_0\le 2s,{\Vert \gamma \Vert }_{0,2}\le 2s_g\frac{1}{n}{\Vert X\gamma \Vert }_2^2\notin \left[\frac{c_{\text{min}}}{2}{\Vert \gamma \Vert }_2^2,\left(C_{\text{min}}+\frac{c_{\text{min}}}{2}\right){\Vert \gamma \Vert }_2^2\right]\right) \\ =\mathbb{P}\left(\exists x\in {ℝ}^{2s\wedge p},S\subseteq 1,\cdots ,p,\left|S\right|=2s\wedge p,|g(S)|\le 2s_g\frac{1}{n}{\Vert X_{S}x\Vert }_2^2\notin \left[\frac{c_{\text{min}}}{2}{\Vert \gamma \Vert }_2^2,\left(C_{\text{min}}+\frac{c_{\text{min}}}{2}\right){\Vert \gamma \Vert }_2^2\right]\right) \\ \le \sum _{S\subseteq \{1,\dots ,p\},\left|S\right|=2s\wedge p,|g(S)|\le 2s_g}\mathbb{P}\left(\forall x\in {ℝ}^{2s\wedge p},\frac{1}{n}{\Vert X_{S}x\Vert }_2^2\notin \left[\frac{c_{\text{min}}}{2}{\Vert \gamma \Vert }_2^2\left(C_{\text{min}}+\frac{c_{\text{min}}}{2}\right){\Vert \gamma \Vert }_2^2\right]\right) \\ \le \sum _{S\subseteq \{1,\dots ,p\},\left|S\right|=2s\wedge p,|g(S)|\le 2s_g}\mathbb{P}\left(\Vert \frac{1}{n}{X}_S^{\top }{X}_S-{\Sigma }_{S,S}\Vert \ge \frac{c_{\text{min}}}{2}\right) \\ \le \sum _{S\subseteq \{1,\dots ,p\},\left|S\right|=2s\wedge p,|g(S)|\le 2s_g}\mathbb{P}\left(\Vert \frac{1}{n}{X}_S^{\top }{X}_S{\Sigma }_{S,S}^{-1}-I_{\left|S\right|}\Vert \ge \frac{c_{\text{min}}}{2C_{\text{max}}}\right) \\ \le \left[\left(\begin{array}{c}d\\ 2s_g\end{array}\right)\vee 1\right]\left(\begin{array}{c}2s_{g}b\\ 2s\end{array}\right)\cdot 2\text{\hspace{0.17em}exp}(Cs-cn) \\ \le {\left(\frac{ed}{2s_g}\right)}^{2s_g}{\left(\frac{e\cdot 2s_{g}b}{2s}\right)}^{2s}\cdot 2\text{\hspace{0.17em}exp}(Cs-cn) \\ \le 2\text{\hspace{0.17em}exp}\left(2s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+2s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)+Cs-cn\right) \\ \le 2e^{-cn}. \end{gathered}$$

□

D. Proof of Theorem 2

If d ≥ 3s_g and b ≥ 3s/s_g, by (80), we can find Ω⁽¹⁾, …, Ω^(N) ⊂ {1, …, db} such that |Ω⁽ⁱ⁾| = s_g⌊s/s_g⌋, $\left|{\text{Ω}}_{(k)}^{(i)}\right|=\lfloor s/s_g\rfloor 1_{\left\{{\text{Ω}}_{(k)}^{(i)}\text{is not empty}\right\}}$ for all 1 ≤ i ≤ N, ≤ k ≤ d, and

$$\left|{\Omega }^{(i)}\cap {\Omega }^{(j)}\right|\le 8s_g\lfloor s/s_g\rfloor /9,1\le i\ne j\le N,$$ (54)

$$\begin{gathered} \left|\left\{k\left|{\Omega }_{(k)}^{(i)}\cap {\Omega }_{(k)}^{(j)}\right|\ge 2\lfloor s/s_g\rfloor /3\right\}\right|\le 2s_g/3, \\ 1\le i\ne j\le N, \end{gathered}$$ (55)

where $N=\left|{\left(\frac{d}{2\sqrt{2}{s}_g}\right)}^{s_g/3}{\left(\frac{b}{2\sqrt{2}\lfloor s/s_g\rfloor }\right)}^{s/9}\right|$. For any 1 ≤ j ≤ db, 1 ≤ i ≤ N, define

$${\beta }_j^{(i)}=\{\begin{array}{ll}\frac{1}{\lambda s_g\lfloor s/s_g\rfloor +{\lambda }_g{s}_g\sqrt{\lfloor s/s_g\rfloor }},\hfill & j\in {\Omega }^{(i)}\hfill \\ 0\hfill & j\notin {\Omega }^{(i)},\hfill \end{array}$$

then ∥β⁽ⁱ⁾∥₀ ≤ s, ∥β⁽ⁱ⁾∥_0,2 ≤ s_g. We consider the quotient space

$${ℝ}^{db}/\text{ker}(X)=\left\{[x]≔x+\text{ker}(X),x\in {ℝ}^{db}\right\}.$$

Then the dimension of ${ℝ}^{db}/\text{ker}(X)$ is rank(X) ≤ n. Define the norm ∥[x]∥ = inf_v∈ker(X){λ∥x − v∥₁ + λ_g ∥x − v∥_1,2}. For any vector $x\in {ℝ}^{db}$ satisfying ∥x∥₀ ≤ 2s, ∥x∥_0,2 ≤ 2s_g, note that x − v with v ∈ ker(X) satisfies X(x − v) = Xx, by our assumption, we have ∥[x]∥ = λ∥x∥₁ +λ_g∥x∥_1,2. Thus ∥[β⁽¹⁾]∥ = ⋯ = ∥[β^(N)]∥ = 1. Moreover, by (54) and (55),

$$\begin{gathered} {\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_1=\frac{1}{\lambda s_g\lfloor s/s_g\rfloor +{\lambda }_g{s}_g\sqrt{\lfloor s/s_g\rfloor }}\cdot \left(\left|{\Omega }^{(i)}\right|+\left|{\Omega }^{(j)}\right|-2\left|{\Omega }^{(i)}\cap {\Omega }^{(j)}\right|\right) \\ \ge \frac{2s_g\lfloor s/s_g\rfloor }{9\left(\lambda s_g\lfloor s/s_g\rfloor +{\lambda }_g{s}_g\sqrt{\lfloor s/s_g\rfloor }\right)}, \end{gathered}$$

and

$$\begin{gathered} {\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_{1,2} \\ =\sum _{k=1}^d{\Vert {\beta }_{(k)}^{(i)}-{\beta }_{(k)}^{(j)}\Vert }_2\ge \sum _{k\in S_{i,j}}{\Vert {\beta }_{(k)}^{(i)}-{\beta }_{(k)}^{(j)}\Vert }_2 \\ \ge \frac{1}{\lambda s_g\lfloor s/s_g\rfloor +{\lambda }_g{s}_g\sqrt{\lfloor s/s_g\rfloor }}\sqrt{\frac{2\lfloor s/s_g\rfloor }{3}}\cdot \left|S_{i,j}\right| \\ \ge \frac{1}{\lambda s_g\lfloor s/s_g\rfloor +{\lambda }_g{s}_g\sqrt{\lfloor s/s_g\rfloor }}\sqrt{\frac{2\lfloor s/s_g\rfloor }{3}}\cdot \frac{s_g}{3}, \end{gathered}$$

where

$$S_{i,j}=\left\{k\left|{\Omega }_{(k)}^{(i)},{\Omega }_{(k)}^{(j)}\text{ are not empty sets},\left|{\Omega }_{(k)}^{(i)}\cap {\Omega }_{(k)}^{(j)}\right|<2\lfloor s/s_g\rfloor /3\right|\right\}\text{. }$$

Since β⁽ⁱ⁾ − β^(j) is (2s, 2s_g)-sparse,

$$\begin{gathered} \Vert \left[{\beta }^{(i)}\right]-\left[{\beta }^{(j)}\right]\Vert =\Vert \left[{\beta }^{(i)}-{\beta }^{(j)}\right]\Vert \\ =\lambda {\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_1+{\lambda }_g{\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_{1,2} \\ \ge 2/9. \end{gathered}$$

By [46, Proposition C.3], we have N ≤ 10^rank(X) ≤ 10ⁿ. Therefore we have

$$\lfloor {\left(\frac{d}{2\sqrt{2}{s}_g}\right)}^{s_g/3}{\left(\frac{b}{2\sqrt{2}\lfloor s/s_g\rfloor }\right)}^{s/9}\rfloor \le {10}^n,$$

which means that n ≥ c(s_g log(d/s_g) + s log(es_gb/s)).

If d < 3s_g or b < 3s/s_g, let $s_g^{\prime }=\left[s_g/3\right]\vee 1\ge s_g/5$, $s^{\prime }=[s/15]\vee s_g^{\prime }$, then $d\ge 3s_g^{\prime }$ and $b\ge 3s^{\prime }/s_g^{\prime }$. Since all (2s, 2s_g)-sparse vectors can be exactly recovered by the ℓ₁ + ℓ_1,2 minimization and s′ ≤ s, $s_g^{\prime }\le s_g$, we know that the ℓ₁ + ℓ_1,2 minimization exactly recover all (2s′, $2s_g^{\prime }$)-sparse vectors. Therefore, we have

$$\begin{gathered} n\ge c\left(s_g^{\prime }\text{\hspace{0.17em}log}\left(d/s_g^{\prime }\right)+s^{\prime }\text{\hspace{0.17em}log}\left(e{s}_g^{\prime }b/s^{\prime }\right)\right) \\ \ge c\left(\frac{s_g}{5}\cdot \text{\hspace{0.17em}log}\left(\frac{d}{s_g}\right)+\frac{s}{15}\cdot \text{\hspace{0.17em}log}\left(\frac{eb\left(s_g/5\right)}{s/15}\vee eb\right)\right) \\ \ge c^{\prime }\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right). \end{gathered}$$ (56)

□

E. Proof of Theorem 3

We would like prove Theorem 3 by contradiction. Let

$$\begin{gathered} c=\text{min}\left\{\frac{1}{8},c^{\prime },\sqrt{\frac{c^{\prime }}{256}}\right\},c_0=\text{min}\left\{\frac{c}{2e},\frac{c^2}{2C^2},16c^2\right\}, \\ {C}_0=\text{max\hspace{0.17em}}\left\{\frac{C^2}{c^2},\frac{1}{32c^2}\right\}, \end{gathered}$$

where c′ is a uniform constant such that n ≥ c′(s log(es_gb/s) + s_g log(d/s_g)) if the conditions in Theorem 2 are satisfied. Assume for contradiction that

$$n<c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right).$$ (57)

Let s₀ = s/s_g, define the norm $\Vert \cdot \Vert ={\Vert \cdot \Vert }_1+\sqrt{s_0}{\Vert \cdot \Vert }_{1,2}$. Let $B=\left\{x\in {ℝ}^p\mid \Vert x\Vert \le 1\right\}$,

$$d^n\left(B,{ℝ}^p\right)=\underset{L^n\text{ is a subspace of }{ℝ}^p\text{ with dim}\left({ℝ}^p/L^n\right)\le n}{\text{inf}}\left\{\underset{\beta \in B\cap L^n}{\text{sup}}{\Vert \beta \Vert }_2\right\}.$$

By [46, Theorem 10.4], we have

$$\begin{gathered} d^n\left(B,{ℝ}^p\right)\le \underset{\beta \in B}{\text{sup}}{\Vert \beta -\Delta (X\beta )\Vert }_2 \\ \le \frac{C}{\sqrt{s}}\underset{\beta \in B}{\text{sup}}\left({\Vert \beta \Vert }_1+\sqrt{s_0}{\Vert \beta \Vert }_{1,2}\right)=\frac{C}{\sqrt{s}}. \end{gathered}$$ (58)

If

$$d^n\left(B,{ℝ}^p\right)\ge c\text{\hspace{0.17em}min}\left\{\frac{1}{\sqrt{s_0}},{\left[\left(\frac{s_g}{s}\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)+\text{log}\left(e{s}_{g}b/s\right)\right)/n\right]}^{1/2}\right\},$$ (59)

since

$$\frac{C}{\sqrt{s}}\le \frac{c\sqrt{C_0}}{\sqrt{s}}\le \frac{c\sqrt{s_g}}{\sqrt{s}}=\frac{c}{\sqrt{s_0}},$$

(58) and (59) together imply that

$$n\ge \frac{c^2}{C^2}\left(s_g\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right).$$ (60)

By (57),

$$\begin{gathered} \frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n} \\ >\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)} \\ \ge 2e\frac{\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)} \\ \ge \text{min}\left\{e\frac{\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)},\frac{e\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{s_g\text{\hspace{0.17em}log}\left(ed/s_g\right)}\right\} \\ \ge \text{min}\left\{\frac{ed}{s_g},\frac{\frac{ed}{s_g}}{\text{\hspace{0.17em}log}\left(\frac{ed}{s_g}\right)}\right\}\ge {\left(\frac{ed}{s_g}\right)}^{1/2}. \end{gathered}$$ (61)

In the last inequality, we used x^1/2 ≥ log(x)/2 for all x ≥ 1. Combine (60) and (61) together, we have

$$\begin{gathered} n\ge \frac{c^2}{2C^2}\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right) \\ \ge c_0\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)>n, \end{gathered}$$

contradiction!

Thus, we only need to prove (59) based on (57). We still use the proof of contradiction. If

$$\begin{gathered} d^n\left(B,{ℝ}^p\right)<c\text{\hspace{0.17em}min}\left\{\frac{1}{\sqrt{s_0}},{\left[\left(\frac{s_g}{s}\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)+\text{log}\left(e{s}_{g}b/s\right)\right)/n\right]}^{1/2}\right\} \\ =:\mu , \end{gathered}$$

then there exists a subspace Lⁿ of ${ℝ}^p$ with $\text{dim}\left({ℝ}^p/L^n\right)\le n$ such that for all v ∈ Lⁿ\{0},

$${\Vert v\Vert }_2<\mu \left({\Vert v\Vert }_1+\sqrt{s_0}{\Vert v\Vert }_{1,2}\right).$$

Let $B\in {ℝ}^{n\times p}$ satisfying ker(B) = Lⁿ. Let $s^{\prime }=\lfloor \frac{1}{32{\mu }^2}\rfloor$, $s_g^{\prime }=\lfloor s^{\prime }/s_0\rfloor$, by (57) and (61),

$$\begin{gathered} \frac{1}{8}{s}_0^{-1/2}\ge c{s}_0^{-1/2}\ge \mu \ge c\text{\hspace{0.17em}min}\left\{\sqrt{\frac{C_0}{s}},{\left(\frac{\frac{s_g}{2s}\text{\hspace{0.17em}log}\left(d/s_g\right)+\text{log}\left(e{s}_{g}b/s\right)}{c_0\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right)}\right)}^{1/2}\right\} \\ \ge \frac{1}{4\sqrt{2}}{s}^{-1/2}, \end{gathered}$$

which means that

$$1\le s^{\prime }\le s,1\le s_g^{\prime }\le s_g.$$

Moreover, we have $\frac{1}{64{\mu }^2}<s^{\prime }\le \frac{1}{32{\mu }^2}$. For any (2s′, $2s_g^{\prime }$)-sparse β with support set T and group support set G, and v ∈ ker(A), by Cauchy-Schwarz inequality,

$$\begin{gathered} {\Vert v_T\Vert }_1+\sqrt{s_0}{\Vert v_{(G)}\Vert }_{1,2}\le \sqrt{2s^{\prime }}{\Vert v_T\Vert }_2+\sqrt{s_0}\sqrt{2s_g^{\prime }}{\Vert v_T\Vert }_2 \\ \le 2\sqrt{2s^{\prime }}{\Vert v_T\Vert }_2 \\ <2\sqrt{2}\frac{1}{4\sqrt{2}\mu }\mu \left({\Vert v\Vert }_1+\sqrt{s_0}{\Vert v\Vert }_{1,2}\right) \\ =\frac{1}{2}\left({\Vert v\Vert }_1+\sqrt{s_0}{\Vert v\Vert }_{1,2}\right), \end{gathered}$$

i.e.,

$${\Vert v_T\Vert }_1+\sqrt{s_0}{\Vert v_{(G)}\Vert }_{1,2}<{\Vert v_{T^c}\Vert }_1+\sqrt{s_0}{\Vert v_{\left(G^c\right)}\Vert }_{1,2}.$$

Based on Cauchy-Schwarz inequality and the sub-differential of ∥β∥₁ and ∥β∥_1,2, we have

$$\begin{gathered} {\Vert \beta +v\Vert }_1+\sqrt{s_0}{\Vert \beta +v\Vert }_{1,2}\ge {\Vert \beta \Vert }_1+\text{sgn}{(\beta )}^{\top }{v}_T+{\Vert v_{T^c}\Vert }_1+\sqrt{s_0}\left({\Vert \beta \Vert }_{1,2}+\sum _{j\in G}\frac{{\beta }_{(j)}^{\top }{v}_{(j)}}{{\Vert {\beta }_{(j)}\Vert }_2}+\sum _{j\in G^c}{\Vert v_j\Vert }_2\right) \\ \ge {\Vert \beta \Vert }_1-{\Vert v_T\Vert }_1+{\Vert v_{T^c}\Vert }_1+\sqrt{s_0}\left({\Vert \beta \Vert }_{1,2}-{\Vert v_{(G)}\Vert }_2+{\Vert v_{\left(G^c\right)}\Vert }_2\right) \\ >{\Vert \beta \Vert }_1+\sqrt{s_0}{\Vert \beta \Vert }_{1,2}. \end{gathered}$$

By Theorem 2,

$$\begin{gathered} n\ge c^{\prime }\left(s^{\prime }\text{\hspace{0.17em}log}\left(e{s}_g^{\prime }b/s^{\prime }\right)+s_g^{\prime }\text{\hspace{0.17em}log}\left(d/s_g^{\prime }\right)\right) \\ \ge c^{\prime }{s}^{\prime }\left\{\text{\hspace{0.17em}log}\left(\frac{e{s}_{g}b}{2s}\right)+\frac{1}{2s_0}\text{\hspace{0.17em}log}\left(s_{0}d/s^{\prime }\right)\right\} \\ \ge c^{\prime }{s}^{\prime }\text{\hspace{0.17em}log}\left(\frac{e{s}_{g}b}{2s}\right). \end{gathered}$$

Thus

$$\begin{gathered} n\ge c^{\prime }{s}^{\prime }\left(\text{\hspace{0.17em}log}\left(\frac{e{s}_{g}b}{2s}\right)+\frac{s_g}{s}\text{\hspace{0.17em}log}\left(\frac{c^{\prime }\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)\right) \\ >\frac{c^{\prime }}{64{\mu }^2}\left(\frac{1}{4}\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)+\frac{s_g}{s}\text{\hspace{0.17em}log}\left(\frac{c\frac{s}{s_g}d\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)}{n}\right)\right) \\ \ge n \end{gathered}$$

provided that $c=\text{min}\left\{\frac{1}{8},c^{\prime },\sqrt{\frac{c^{\prime }}{256}}\right\}$, contradiction! This means that (59) holds if (57) is true.

Therefore, we have finished the proof of Theorem 3. □

F. Proof of Theorem 4

Let $\lambda =C\sigma \sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}n}$, ${\lambda }_g=\sqrt{s/s_g}\lambda$. By (86) in Lemma 5 and (101), one has

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\frac{1}{10}\lambda }\left(X^{\top }\epsilon \right)\Vert }_{\infty ,2}\ge \frac{1}{10}{\lambda }_g\right)\le \mathbb{P}\left(\exists 1\le j\le d,{\Vert H_{\frac{1}{10}\lambda }\left(X_{(j)}^{\top }\epsilon \right)\Vert }_2\ge \frac{1}{10}{\lambda }_g,\Vert \epsilon {\Vert }_2\ge 5\sqrt{n{\sigma }^2}\right)+\mathbb{P}\left(\Vert \epsilon {\Vert }_2\ge 5\sqrt{n{\sigma }^2}\right) \\ \le \sum _{j=1}^d\mathbb{P}\left({\Vert H_{\frac{1}{10}\lambda }\left(X_{(j)}^{\top }\epsilon \right)\Vert }_2\ge \left|\frac{1}{10}{\lambda }_g\right|\Vert \epsilon {\Vert }_2\ge 5\sqrt{n{\sigma }^2}\right)+\mathbb{P}\left(\Vert \epsilon {\Vert }_2\ge 5\sqrt{n{\sigma }^2}\right) \\ \le d\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)+e^{-n} \\ =\text{exp}\left(\text{\hspace{0.17em}log}\left(s_g\right)+\text{log}\left(d/s_g\right)-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)+e^{-n} \\ \le \text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)+e^{-n}. \end{gathered}$$ (62)

By the definition of $\widehat{\beta }$ and KKT condition, we have

$$X^{\top }(y-X\widehat{\beta })+\lambda z_1+{\lambda }_g{z}_2=0,$$

where

$$\{\begin{array}{ll}{\left(z_1\right)}_i=\text{sgn}\left({\widehat{\beta }}_i\right),\hfill & {\widehat{\beta }}_i\ne 0;\hfill \\ \left|{\left(z_1\right)}_i\right|\le 1,\hfill & {\widehat{\beta }}_i=0;\hfill \end{array}$$

$$\{\begin{array}{ll}{\left(z_2\right)}_{(j)}=\frac{{\widehat{\beta }}_{(j)}}{{\Vert {\widehat{\beta }}_{(j)}\Vert }_2},\hfill & {\widehat{\beta }}_{(j)}\ne 0;\hfill \\ {\Vert {\left(z_2\right)}_{(j)}\Vert }_2\le 1,\hfill & {\widehat{\beta }}_{(j)}=0.\hfill \end{array}$$

Therefore,

$${\Vert H_{\lambda }\left(X^{\top }(X\widehat{\beta }-y)\right)\Vert }_{\infty ,2}\le {\lambda }_g.$$

(62), Lemma 8 Part 1 and the previous inequality together imply that

$$\mathbb{P}\left({\Vert H_{\left(1+\frac{1}{10}\right)\lambda }\left(X^{\top }Xh\right)\Vert }_{\infty ,2}\le \left(1+\frac{1}{10}\right){\lambda }_g\right)\ge 1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)-e^{-n},$$ (63)

where $h=\widehat{\beta }-{\beta }^{*}$. By the definition of $\widehat{\beta }$, we have

$${\Vert y-X\widehat{\beta }\Vert }_2^2+\lambda {\Vert \widehat{\beta }\Vert }_1+{\lambda }_g{\Vert \widehat{\beta }\Vert }_{1,2}\le {\Vert y-X{\beta }^{*}\Vert }_2^2+\lambda {\Vert {\beta }^{*}\Vert }_1+{\lambda }_g{\Vert {\beta }^{*}\Vert }_{1,2}.$$

(32) and the previous inequality show that

$$\begin{gathered} {\Vert Xh\Vert }_2^2+\lambda {\Vert h_{T^c}\Vert }_1+{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le 2\langle Xh,\epsilon \rangle -\lambda \cdot \text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T-{\lambda }_g\sum _{j\in G}\frac{{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2} \\ +2\lambda {\Vert {\beta }_{T^c}^{*}\Vert }_1+2{\lambda }_g{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}. \end{gathered}$$ (64)

First, we consider 〈Xh, ε〉. Denote $P=X_T{\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }$, since $Xh=X_T{h}_T+X_{T^c}{h}_{T^c}$ and (I_n − P)X_T = 0,

$$\begin{gathered} |\langle Xh,\epsilon \rangle |\le |\langle PXh,\epsilon \rangle |+\left|\langle \left(I_n-P\right)Xh,\epsilon \rangle \right| \\ =\left|\langle X_T^{\top }Xh,{\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \rangle \right|+\left|\langle \left(I_n-P\right)X_{T^c}{h}_{T^c},\epsilon \rangle \right| \\ =\left|\langle X_T^{\top }Xh,{\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \rangle \right|+\left|\langle X_{T^c}{h}_{T^c},\left(I_n-P\right)\epsilon \rangle \right|. \end{gathered}$$ (65)

Therefore, to give an upper bound of |〈Xh, ε〉|, we only need to bound $\left|\langle X_T^{\top }Xh,{\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \rangle \right|$ and $\left|\langle X_{T^c}{h}_{T^c},\left(I_n-P\right)\epsilon \rangle \right|$, respectively. By Part 1 of Lemma 7 and also notice that c_min ≤ σ_min (Σ) ≤ σ_max (Σ) ≤ C_max,

$$\begin{gathered} \mathbb{P}\left(\Vert {\left(\frac{1}{n}{X}_T^{\top }{X}_T\right)}^{-1}\Vert \ge \frac{2}{c_{\text{min}}}\right) \\ \le \mathbb{P}\left(\Vert \frac{1}{n}{X}_T^{\top }{X}_T-{\Sigma }_{T,T}\Vert \ge \frac{c_{\text{min}}}{2}\right) \\ \le \mathbb{P}\left(\Vert \frac{1}{n}{X}_T^{\top }{X}_T{\Sigma }_{T,T}^{-1}-I_s\Vert \ge \frac{c_{\text{min}}}{2C_{\text{max}}}\right) \\ \le 2\text{\hspace{0.17em}exp}(Cs-cn)\le 2\text{\hspace{0.17em}exp}(-cn). \end{gathered}$$ (66)

(66), Lemma 9 and Cauchy-Schwarz inequality together imply that with probability at least $1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-2\text{\hspace{0.17em}exp}(-cn)$,

$$\begin{gathered} {\Vert {\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \Vert }_1\le \frac{2}{c_{\text{min}}}\frac{\sqrt{s}}{n}{\Vert X_T^{\top }\epsilon \Vert }_2\le \frac{2}{c_{\text{min}}}\frac{s}{n}{\Vert X_T^{\top }\epsilon \Vert }_{\infty } \\ \le C\frac{s}{n}\sqrt{n\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}}{\sigma }^2 \\ \le C\frac{s}{n}\lambda , \end{gathered}$$

$${\Vert {\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \Vert }_{1,2}\le \sqrt{s_g}{\Vert {\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \Vert }_2\le \frac{2}{c_{\text{min}}}\frac{\sqrt{s_g}}{n}{\Vert X_T^{\top }\epsilon \Vert }_2\le C\frac{\sqrt{s\cdot s_g}}{n}\lambda .$$

Combine Lemma 8 Part 2, (63) and the previous two inequalities together, with probability at least $1-2\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-3e^{-cn}$,

$$\begin{gathered} \left|\langle X_T^{\top }Xh,{\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \rangle \right|\le \frac{11}{10}\lambda {\Vert {\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \Vert }_1+\frac{11}{10}{\lambda }_g{\Vert {\left(X_T^{\top }{X}_T\right)}^{-1}{X}_T^{\top }\epsilon \Vert }_{1,2} \\ \le C\frac{s}{n}{\lambda }^2. \end{gathered}$$ (67)

Similarly to the proof of (62), also notice that ∥(I_n − P)ε∥₂ ≤ ∥ε∥₂ and $X_{\left(G^c\right)}$ is independent of I_n − P, we have

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\frac{1}{10}\lambda }\left(X_{\left(G^c\right)}^{\top }\left(I_n-P\right)\epsilon \right)\Vert }_{\infty ,2}\ge \frac{1}{10}{\lambda }_g\right) \\ \le \mathbb{P}\left(\exists j\in G^c,{\Vert H_{\frac{1}{10}\lambda }\left(X_{(j)}^{\top }\left(I_n-P\right)\epsilon \right)\Vert }_2\ge \frac{1}{10}{\lambda }_g|{\Vert \left(I_n-P\right)\epsilon \Vert }_2\ge 5\sqrt{n{\sigma }^2}\right)+\mathbb{P}\left(\Vert \epsilon {\Vert }_2\ge 5\sqrt{n{\sigma }^2}\right) \\ \le \text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)+e^{-n}. \end{gathered}$$

By Lemma 8 Part 2 and (62), with probability at least $1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)-e^{-\eta }$,

$$\begin{gathered} \left|\langle X_{\left(G^c\right)}{h}_{\left(G^c\right)},\left(I_n-P\right)\epsilon \rangle \right|=\left|\langle h_{\left(G^c\right)},X_{\left(G^c\right)}^{\top }\left(I_n-P\right)\epsilon \rangle \right| \\ \le \frac{1}{10}\lambda {\Vert h_{\left(G^c\right)}\Vert }_1+\frac{1}{10}{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2}. \end{gathered}$$

Notice that $X_{T^c\backslash \left(G^c\right)}$ and I_n − P are independent and |T^c\(G^c)| ≤ |G| ≤ s_gb, by Lemma 9, with probability at least $1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-e^{-n}$,

$$\begin{gathered} \left|\langle X_{T^c\backslash \left(G^c\right)}{h}_{T^c\backslash \left(G^c\right)},\left(I_n-P\right)\epsilon \rangle \right|\le {\Vert h_{T^c\backslash \left(G^c\right)}\Vert }_1{\Vert X_{T^c\backslash \left(G^c\right)}^{\top }\left(I_n-P\right)\epsilon \Vert }_{\infty } \\ \le C\sqrt{n\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}{\sigma }^2}{\Vert h_{T^c\backslash \left(G^c\right)}\Vert }_1 \\ \le \frac{1}{10}\lambda {\Vert h_{T^c\backslash \left(G^c\right)}\Vert }_1. \end{gathered}$$

Combine the previous two inequalities together, we have

$$\begin{gathered} \left|\langle X_{T^c}{h}_{T^c},\left(I_n-P\right)\epsilon \rangle \right|\le \left|\langle X_{\left(G^c\right)}{h}_{\left(G^c\right)},\left(I_n-P\right)\epsilon \rangle \right|+\left|\langle X_{T^c\backslash \left(G^c\right)}{h}_{T^c\backslash \left(G^c\right)},\left(I_n-P\right)\epsilon \rangle \right| \\ \le \frac{1}{10}\lambda {\Vert h_{T^c}\Vert }_1+\frac{1}{10}{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \end{gathered}$$ (68)

with probability $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn}$. Combine (65), (67) and (68) together, we know that with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn}$,

$$\left|\langle Xh,\epsilon \rangle \right|\le C\frac{s}{n}{\lambda }^2+\frac{1}{10}\lambda {\Vert h_{T^c}\Vert }_1+\frac{1}{10}{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2}.$$ (69)

Moreover, by the proof of Theorem 1, with probability at least 1 − C exp(−cn/s), there exists an approximate dual certificate $u\in {ℝ}^p$ in the row span of X satisfying (18), and ${\Vert v_T-\text{sgn}\left({\beta }_T^{*}\right)\Vert }_2\le \frac{1}{8}$, where v is defined in (29). Similarly to (33), we have

$$\begin{gathered} \text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2}\ge -{\Vert v_T-\text{sgn}\left({\beta }_T^{*}\right)\Vert }_2\cdot {\Vert h_T\Vert }_2-{\Vert h_{T^c}\Vert }_1/2-\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2+\langle h,u\rangle \\ \ge -\frac{c_{\text{min}}}{8}\cdot {\Vert h_T\Vert }_2-{\Vert h_{T^c}\Vert }_1/2-\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2+\langle h,u\rangle . \end{gathered}$$

By Lemma 10, with probability at least 1 − Ce^−cn/s, u = X^⊤ w with ${\Vert w\Vert }_2\le C\sqrt{s/n}$. Therefore, with probability at least 1 − Ce^−cn/s,

$$\left|\langle h,u\rangle \right|=\left|\langle Xh,w\rangle \right|\le {\Vert Xh\Vert }_2{\Vert w\Vert }_2\le C\sqrt{s/n}{\Vert Xh\Vert }_2.$$

The two previous inequalities together imply that

$$\begin{gathered} \text{sgn}{\left({\beta }_T^{*}\right)}^{\top }{h}_T+\sum _{j\in G}\frac{\sqrt{s_0}{\beta }_{T,(j)}^{*\top }{h}_{(j)}}{{\Vert {\beta }_{T,(j)}^{*}\Vert }_2} \\ \ge -\frac{c_{\text{min}}}{8}\cdot {\Vert h_T\Vert }_2-{\Vert h_{T^c}\Vert }_1/2-\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}/2-C\sqrt{s/n}{\Vert Xh\Vert }_2 \end{gathered}$$ (70)

with probability at least 1 − Ce^−cn/s.

Combine (64), (69) and (70) together, with probability at least $1-C{e}^{-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}}-C{e}^{-cn/s}$,

$$\begin{gathered} {\Vert Xh\Vert }_2^2+\frac{3}{10}\lambda {\Vert h_{T^c}\Vert }_1+\frac{3}{10}{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le C\frac{s}{n}{\lambda }^2+\frac{c_{\text{min}}}{8}\lambda {\Vert h_T\Vert }_2+C\sqrt{s/n}\lambda {\Vert Xh\Vert }_2+2\lambda {\Vert {\beta }_{T^c}^{*}\Vert }_1+2{\lambda }_g{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}. \end{gathered}$$ (71)

By (42), (63) and (66), with probability at least $1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)-C{e}^{-cn}$,

$$\begin{gathered} {\Vert h_T\Vert }_2\le \Vert {\left(X_T^{\top }{X}_T\right)}^{-1}\Vert {\Vert X_T^{\top }{X}_T{h}_T\Vert }_2 \\ \le \frac{2}{c_{\text{min}}n}{\Vert X_T^{\top }Xh-X_T^{\top }{X}_{T^c}{h}_{T^c}\Vert }_2 \\ \le \frac{2}{c_{\text{min}}n}\left({\Vert X_T^{\top }Xh\Vert }_2+{\Vert X_T^{\top }{X}_{T^c}{h}_{T^c}\Vert }_2\right) \\ \le \frac{2}{c_{\text{min}}n}\left({\Vert H_{\frac{11}{10}\lambda }\left(X_T^{\top }Xh\right)\Vert }_2+\frac{11}{10}\sqrt{s}\lambda +n\sum _{i\in T^c}{\Vert X_T^{\top }{X}_i/n\Vert }_2\left|h_i\right|\right) \\ \le \frac{2}{c_{\text{min}}n}\left(\sqrt{s_g}{\Vert H_{\frac{11}{10}\lambda }\left(X_T^{\top }Xh\right)\Vert }_{\infty ,2}+\frac{11}{10}\sqrt{s}\lambda +n\underset{i\in T^c}{\text{max}}{\Vert X_T^{\top }{X}_i/n\Vert }_2{\Vert h_{T^c}\Vert }_1\right) \\ \le \frac{2}{c_{\text{min}}n}\left(\sqrt{s_g}\frac{11}{10}{\lambda }_g+\frac{11}{10}\sqrt{s}\lambda +\frac{n}{2}{\Vert h_{T^c}\Vert }_1\right) \\ \le \frac{5}{c_{\text{min}}}\frac{\sqrt{s}}{n}\lambda +\frac{1}{c_{\text{min}}}{\Vert h_{T^c}\Vert }_1. \end{gathered}$$ (72)

The fourth inequality comes from ${\Vert x\Vert }_2\le {\Vert H_{\alpha }(x)\Vert }_2+\sqrt{s}\alpha$ for $x\in {ℝ}^s$; the fifth inequality holds since ${\Vert X_T^{\top }Xh\Vert }_{0,2}\le s_g$. (71) and (72) together imply that

$$\begin{gathered} {\Vert Xh\Vert }_2^2+\frac{7}{40}\lambda {\Vert h_{T^c}\Vert }_1+\frac{3}{10}{\lambda }_g{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le C\frac{s}{n}{\lambda }^2+C\sqrt{s/n}\lambda {\Vert Xh\Vert }_2+2\lambda {\Vert {\beta }_{T^c}^{*}\Vert }_1+2{\lambda }_g{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2} \end{gathered}$$

with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn/s}$. Also notice that

$$C\sqrt{s/n}\lambda {\Vert Xh\Vert }_2\le {\Vert Xh\Vert }_2^2+C\frac{s}{n}{\lambda }^2,$$

with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn/s}$,

$${\Vert h_{T^c}\Vert }_1+\sqrt{s_0}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\le C\left(\frac{s}{n}\lambda +{\Vert {\beta }_{T^c}^{*}\Vert }_1+\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right).$$ (73)

From the proof of Lemma 1, we know that (36) and (41) hold with probability at least 1 − 2e^−cn. By Lemma 8 Part 2 and (63), with probability at least $1-\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right)-e^{-n}$,

$$\begin{gathered} \left|\langle X_{\tilde{T}}{h}_{\tilde{T}},Xh\rangle \right|=\left|\langle h_{\tilde{T}},X_{\tilde{T}}^{\top }Xh\rangle \right|\le \frac{11}{10}\left(\lambda {\Vert h_{\tilde{T}}\Vert }_1+{\lambda }_g{\Vert h_{\tilde{T}}\Vert }_{1,2}\right) \\ \le \frac{11}{10}\left(\lambda \cdot \sqrt{3s}{\Vert h_{\tilde{T}}\Vert }_2+{\lambda }_g\sqrt{2s_g}{\Vert h_{\tilde{T}}\Vert }_2\right) \\ \le 4\lambda \sqrt{s}{\Vert h_{\tilde{T}}\Vert }_2. \end{gathered}$$ (74)

The second inequality is due to ${\Vert h_{\tilde{T}}\Vert }_0\le 3s$, ${\Vert h_{\tilde{T}}\Vert }_{0,2}\le 2s_g$ and Cauchy-Schwarz inequality.

Combine (36), (41), (73) and (74) together, with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn/s}$, we have

$$\begin{gathered} \frac{c_{\text{min}}}{2}{\Vert h_{\tilde{T}}\Vert }_2^2\le \frac{1}{n}4\lambda \sqrt{s}{\Vert h_{\tilde{T}}\Vert }_2+\sqrt{3}{C}_{\text{max}}{\Vert h_{\tilde{T}}\Vert }_2\left(\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_1+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\right) \\ \le \frac{1}{n}4\lambda \sqrt{s}{\Vert h_{\tilde{T}}\Vert }_2+\sqrt{3}{C}_{\text{max}}{\Vert h_{\tilde{T}}\Vert }_2\cdot \frac{C}{\sqrt{s}}\cdot \left(\frac{s}{n}\lambda +{\Vert {\beta }_{T^c}^{*}\Vert }_1+\sqrt{s_0}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right) \\ \le C\left(\frac{\sqrt{s}}{n}\lambda +\frac{1}{\sqrt{s}}{\Vert {\beta }_{T^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right){\Vert h_{\tilde{T}}\Vert }_2. \end{gathered}$$

Therefore, with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)-C{e}^{-cn/s}$,

$${\Vert h_{\tilde{T}}\Vert }_2\le C\left(\frac{\sqrt{s}}{n}\lambda +\frac{1}{\sqrt{s}}{\Vert {\beta }_{T^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right).$$ (75)

By (39), (40), (73) and the previous inequality, also notice that $e^{-cn/s}\le e^{-C\frac{s\text{\hspace{0.17em}log}(e{s}_{g}b)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}}$, with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$,

$$\begin{gathered} \Vert h{\Vert }_2\le {\Vert h_{\tilde{T}}\Vert }_2+\sum _{i\ge 2}{\Vert h_{T_i}\Vert }_2+\sum _{j\ge 2}{\Vert h_{R_j}\Vert }_2 \\ \le {\Vert h_{\tilde{T}}\Vert }_2+\sqrt{2}{s}^{-1/2}{\Vert h_{T^c}\Vert }_2+s_g^{-1/2}{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le C\left(\frac{\sqrt{s}}{n}\lambda +\frac{1}{\sqrt{s}}{\Vert {\beta }_{T^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right), \end{gathered}$$ (76)

i.e., with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$,

$${\Vert h\Vert }_2\le C\left(\sqrt{\frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)\right)}{n}}+\frac{1}{\sqrt{s}}{\Vert {\beta }_{T^c}^{*}\Vert }_1+\frac{1}{\sqrt{s_g}}{\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}\right).$$

Moreover, if β* is (s, s_g)-sparse, then ${\Vert {\beta }_{T^c}^{*}\Vert }_1={\Vert {\beta }_{T^c}^{*}\Vert }_{1,2}=0$. Therefore, with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$,

$${\Vert h\Vert }_2^2\le \frac{C{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)\right)}{n}.$$

□

G. Proof of Theorem 5

First, we consider the case that d ≥ 3s_g and b ≥ 3s/s_g. Let ω⁽¹⁾, …, ω^(N) be uniformly randomly vectors from

$$\begin{gathered} A=\left\{\omega \in {\{0,1\}}^{db}\mid \sum _j{1}_{\left\{{\omega }_{(j)}\ne 0\right\}}=s_g, \\ {\Vert {\omega }_{(j)}\Vert }_0=\lfloor s/s_g\rfloor \text{ if }{\omega }_{(j)}\ne 0\right\}. \end{gathered}$$

Denote ${\text{Ω}}^{(i)}=\left\{j\mid {\omega }_j^{(i)}\ne 0\right\}$, ${\text{Ω}}_{(k)}^{(i)}=\left\{j\mid j\in (k),{\omega }_j^{(i)}\ne 0\right\}$ and β⁽ⁱ⁾ = τω⁽ⁱ⁾, for all 1 ≤ i ≤ N, 1 ≤ k ≤ d, where τ is a parameter that will be specified later. Obviously, ∥β⁽ⁱ⁾∥₀ = s_g⌊s/s_g⌋ ≤ s, therefore ${\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2\le 2s_g\lfloor s/s_g\rfloor {\tau }^2\le 2s{\tau }^2$.

Moreover, if |Ω⁽ⁱ⁾ ∩ Ω^(j)| ≥ 8s_g⌊s/s_g⌋/9, then we must have

$$\left|\left\{k|{\omega }_{(k)}^{(i)},{\omega }_{(k)}^{(j)}\ne 0,\left|{\Omega }_{(k)}^{(i)}\cap {\Omega }_{(k)}^{(j)}\right|\ge 2\lfloor s/s_g\rfloor /3\right\}\right|\ge 2s_g/3,$$

otherwise $\left|{\text{Ω}}^{(i)}\cap {\text{Ω}}^{(j)}\right|\le \frac{2s_g}{3}\lfloor s/s_g\rfloor +\frac{s_g}{3}2\lfloor s/s_g\rfloor /3\le 8s_g\lfloor s/s_g\rfloor /9$, which is a contradiction.

Therefore,

$$\begin{gathered} \mathbb{P}\left({\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2\le 2s_g\lfloor s/s_g\rfloor {\tau }^2/9\right) \\ =\mathbb{P}\left(\left|{\Omega }^{(i)}\cap {\Omega }^{(j)}\right|\ge 8s_g\lfloor s/s_g\rfloor /9\right) \\ \le \mathbb{P}\left(\left|\left\{k|{\omega }_{(k)}^{(i)},{\omega }_{(k)}^{(j)}\ne 0,\left|{\Omega }_{(k)}^{(i)}\cap {\Omega }_{(k)}^{(j)}\right|\ge 2\lfloor s/s_g\rfloor /3\right\}\right|\ge 2s_g/3\right) \\ \le \sum _{l=\lceil s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right)\cdot [\sum _{t=\lceil 2\lfloor s/s_g\rfloor /3\rceil }^{\lfloor s/s_g\rfloor }\left(\begin{array}{c}\lfloor s/s_g\rfloor \\ t\end{array}\right){\left(\begin{array}{c}b-\lfloor s/s_g\rfloor \\ \lfloor s/s_g\rfloor -t\end{array}\right]}^l\cdot {\left(\begin{array}{c}b\\ \lfloor s/s_g\rfloor \end{array}\right)}^{s_g-l}\left(\begin{array}{c}d-l\\ s_g-l\end{array}\right)/\left(\begin{array}{c}d\\ s_g\end{array}\right){\left(\begin{array}{c}b\\ \lfloor s/s_g\rfloor \end{array}\right)}^{s_g} \\ =\sum _{l=\lceil 2s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right)\frac{\left(\begin{array}{c}d-l\\ s_g-l\end{array}\right)}{\left(\begin{array}{c}d\\ s_g\end{array}\right)}\cdot {\left[\sum _{t=\lceil 2\lfloor s/s_g\rfloor /3\rceil }^{\lfloor s/s_g\rfloor }\left(\begin{array}{c}\lfloor s/s_g\rfloor \\ t\end{array}\right)\frac{\left(\begin{array}{c}b-\lfloor s/s_g\rfloor \\ \lfloor s/s_g\rfloor -t\end{array}\right)}{\left(\begin{array}{l}b\hfill \\ \lfloor s/s_g\rfloor \hfill \end{array}\right)}\right]}^l. \end{gathered}$$ (77)

Note that

$$\begin{gathered} \frac{\left(\begin{array}{l}d-l\hfill \\ s_g-l\hfill \end{array}\right)}{\left(\begin{array}{l}d\hfill \\ s_g\hfill \end{array}\right)}=\frac{\frac{(d-l)\cdots \left(d-s_g+1\right)}{\left(s_g-l\right)!}}{\frac{d(d-1)\cdots \left(d-s_g+1\right)}{s_g!}}=\frac{s_g\left(s_g-1\right)\cdots \left(s_g-l+1\right)}{d(d-1)\cdots (d-l+1)} \\ \le {\left(\frac{s_g}{d}\right)}^l, \end{gathered}$$

The inequality holds since $\frac{s_g-i}{d-i}\le \frac{s_g}{d}$, for all 1 ≤ i ≤ s_g. Similarly, for 1 ≤ t ≤ ⌊s/s_g⌋,

$$\frac{\left(\begin{array}{c}b-\lfloor s/s_g\rfloor \\ \lfloor s/s_g\rfloor -t\end{array}\right)}{\left(\begin{array}{c}b\\ \lfloor s/s_g\rfloor \end{array}\right)}\le \frac{\left(\begin{array}{c}b-t\\ \lfloor s/s_g\rfloor -t\end{array}\right)}{\left(\begin{array}{c}b\\ \lfloor s/s_g\rfloor \end{array}\right)}\le {\left(\frac{\lfloor s/s_g\rfloor }{b}\right)}^t.$$

Combine (77) and the previous two inequalities together, we have

$$\begin{gathered} \mathbb{P}\left({\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2\le 2s_g\lfloor s/s_g\rfloor {\tau }^2/9\right)\le \sum _{l=\lceil 2s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right){\left(\frac{s_g}{d}\right)}^l\cdot {\left[\sum _{t=\lceil 2\lfloor s/s_g\rfloor /3\rceil }^{\lfloor s/s_g\rfloor }\left(\begin{array}{c}\lfloor s/s_g\rfloor \\ t\end{array}\right){\left(\frac{\lfloor s/s_g\rfloor }{b}\right)}^t\right]}^l \\ \le \sum _{l=\lceil 2s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right){\left(\frac{s_g}{d}\right)}^l\cdot {\left[\sum _{t=\lceil 2\lfloor s/s_g\rfloor /3\rceil }^{\lfloor s/s_g\rfloor }\left(\begin{array}{c}\lfloor s/s_g\rfloor \\ t\end{array}\right){\left(\frac{\lfloor s/s_g\rfloor }{b}\right)}^{2\lfloor s/s_g\rfloor /3}\right]}^l \\ \le \sum _{l=\lceil 2s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right){\left(\frac{s_g}{d}\right)}^l\cdot {\left[2^{\lfloor s/s_g\rfloor }{\left(\frac{\lfloor s/s_g\rfloor }{b}\right)}^{2\lfloor s/s_g\rfloor /3}\right]}^l \\ \le \sum _{l=\lceil 2s_g/3\rceil }^{s_g}\left(\begin{array}{c}{s}_g\\ l\end{array}\right){\left(\frac{s_g}{d}\right)}^{2s_g/3}\cdot {\left[{\left(\frac{2\sqrt{2}\lfloor s/s_g\rfloor }{b}\right)}^{2\lfloor s/s_g\rfloor /3}\right]}^{2s_g/3} \\ \le {\left(\frac{2\sqrt{2}{s}_g}{d}\right)}^{2s_g/3}\cdot {\left(\frac{2\sqrt{2}\lfloor s/s_g\rfloor }{b}\right)}^{2s/9}. \end{gathered}$$ (78)

Set $N=\left[{\left(\frac{d}{2\sqrt{2}{s}_g}\right)}^{s_g/3}{\left(\frac{b}{2\sqrt{2}\lfloor s/s_g\rfloor }\right)}^{s/9}\right]$, then

$$\begin{gathered} \mathbb{P}\left(\forall 1\le i\ne j\le N,{\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2>2s_g\lfloor s/s_g\rfloor {\tau }^2/9\right) \\ \ge 1-\frac{N(N-1)}{2}{\left(\frac{2\sqrt{2}{s}_g}{d}\right)}^{2s_g/3}\cdot {\left(\frac{2\sqrt{2}\lfloor s/s_g\rfloor }{b}\right)}^{2s/9} \\ >0. \end{gathered}$$

i.e., the probability that β⁽¹⁾, …, β^(N); Ω⁽¹⁾, ⋯, Ω^(N) satisfy

$$\frac{s}{9}{\tau }^2<2s_g\lfloor s/s_g\rfloor {\tau }^2/9<\underset{i\ne j}{\text{min\hspace{0.17em}}}{\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2\le 2s{\tau }^2,$$ (79)

$$\left|{\Omega }^{(i)}\cap {\Omega }^{(j)}\right|<8s_g\lfloor s/s_g\rfloor /9,\forall 1\le i<j\le N$$ (80)

is positive. For convenience, we fix β⁽¹⁾, …, β^(N) to be the vectors satisfying (79).

Denote y⁽ⁱ⁾ = Xβ⁽ⁱ⁾ + ε for all 1 ≤ i ≤ N. We consider the Kullback-Leibler divergence between different distribution pairs:

$$D_{KL}\left(\left(y^{(i)},X\right),\left(y^{(j)},X\right)\right)={\mathbb{E}}_{\left(y^{(j)},X\right)}\left[\text{\hspace{0.17em}log}\left(\frac{p\left(y^{(i)},X\right)}{p\left(y^{(j)},X\right)}\right)\right],$$

where p(y⁽ⁱ⁾, X) is the probability density of (y⁽ⁱ⁾, X). Conditioning on X, we have

$${\mathbb{E}}_{\left(y^{(j)},X\right)}\left[\text{\hspace{0.17em}log}\left(\frac{p\left(y^{(i)},X\right)}{p\left(y^{(j)},X\right)}\right)\mid X\right]=\frac{{\Vert X\left({\beta }^{(i)}-{\beta }^{(j)}\right)\Vert }_2^2}{2{\sigma }^2}.$$

Thus for 1 ≤ i ≠ j ≤ N,

$$\begin{gathered} D_{KL}\left(\left(y^{(i)},X\right),\left(y^{(j)},X\right)\right) \\ ={\mathbb{E}}_X\frac{{\Vert X\left({\beta }^{(i)}-{\beta }^{(j)}\right)\Vert }_2^2}{2{\sigma }^2} \\ =\frac{n{\left({\beta }^{(i)}-{\beta }^{(j)}\right)}^{\top }\Sigma \left({\beta }^{(i)}-{\beta }^{(j)}\right)}{2{\sigma }^2} \\ \le \frac{3n{\Vert {\beta }^{(i)}-{\beta }^{(j)}\Vert }_2^2}{4{\sigma }^2}\le \frac{3ns{\tau }^2}{2{\sigma }^2}. \end{gathered}$$ (81)

In the first inequality, we used ${\sigma }_{\text{max}}(\Sigma )\le \frac{3}{2}$. By generalized Fano’s Lemma,

$$\underset{\widehat{\beta }}{\text{inf}}\underset{\beta \in {\mathbb{F}}_{s,s_g}}{\text{sup}}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2\ge \frac{\sqrt{s{\tau }^2/9}}{2}\left(1-\frac{\frac{3ns{\tau }^2}{2{\sigma }^2}+\text{log\hspace{0.17em}}2}{\text{\hspace{0.17em}log\hspace{0.17em}}N}\right).$$

Since $\text{\hspace{0.17em}log\hspace{0.17em}}N\asymp s_g\text{\hspace{0.17em}log}\left(\frac{d}{s_g}\right)+s\text{\hspace{0.17em}log}\left(\frac{e{s}_{g}b}{s}\right)$, by setting $\tau =c\sqrt{\frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(\frac{d}{s_g}\right)+s\text{\hspace{0.17em}log}\left(\frac{e{s}_{g}b}{s}\right)\right)}{ns}}$, we have

$$\underset{\widehat{\beta }}{\text{inf}}\underset{\beta \in {\mathbb{F}}_{s,s_g}}{\text{sup}}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2^2\ge {\left(\underset{\widehat{\beta }}{\text{inf}}\underset{\beta \in {\mathbb{F}}_{s,s_g}}{\text{sup}}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2\right)}^2\ge c\frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right)}{n}.$$

If d < 3s_g or b < 3s/s_g, let $s_g^{\prime }=\left[s_g/3\right]\vee 1\ge s_g/5$, $s^{\prime }=[s/15]\vee s_g^{\prime }$, then $d\ge 3s_g^{\prime }$ and $b\ge 3s^{\prime }/s_g^{\prime }$. Similarly to (56), we have

$$\begin{gathered} \underset{\widehat{\beta }}{\text{inf}}\underset{\beta \in {\mathbb{F}}_{s,s_g}}{\text{sup}}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2^2\ge \underset{\widehat{\beta }}{\text{inf}}\underset{\beta \in {\mathbb{F}}_{s^{\prime },s_g^{\prime }}}{\text{sup}}\mathbb{E}{\Vert \widehat{\beta }-\beta \Vert }_2^2 \\ \ge c\frac{{\sigma }^2\left(s_g^{\prime }\text{\hspace{0.17em}log}\left(d/s_g^{\prime }\right)+s^{\prime }\text{\hspace{0.17em}log}\left(e{s}_g^{\prime }b/s^{\prime }\right)\right)}{n} \\ \ge c^{\prime }\frac{{\sigma }^2\left(s_g\text{\hspace{0.17em}log}\left(d/s_g\right)+s\text{\hspace{0.17em}log}\left(e{s}_{g}b/s\right)\right)}{n}. \end{gathered}$$

□

H. Proof of Theorem 6

The proof of Theorem 6 relies on the following key lemma, which shows that Σ⁻¹ is in the feasible set of the optimization problem (23) with high probability by choosing appropriate α and γ.

Lemma 4: By setting $\alpha =C\sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{sn}}$, $\gamma =\sqrt{\frac{s}{s_g}}\alpha$ in (23), we have

$$\mathbb{P}\left({\text{max\hspace{0.17em}}}_{1\le i\le p}{\Vert H_{\alpha }\left(e_i-\frac{1}{n}{X}^{\top }X{\Sigma }^{-1}{e}_i\right)\Vert }_{\infty ,2}\le \gamma \right)\ge 1-4\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right).$$

Note that Y = Xβ* + ε, we have

$$\begin{gathered} \sqrt{n}\left({\widehat{\beta }}^u-{\beta }^{*}\right)=\sqrt{n}\left(\widehat{\beta }-{\beta }^{*}+\frac{1}{n}\widehat{M}{X}^{\top }(Y-X\widehat{\beta })\right) \\ =\sqrt{n}\left(I-\frac{1}{n}\widehat{M}{X}^{\top }X\right)\left(\widehat{\beta }-{\beta }^{*}\right)+\frac{1}{\sqrt{n}}\widehat{M}{X}^{\top }\epsilon . \end{gathered}$$

Since ${\epsilon }_i\stackrel{i.i.d.}{~}N\left(0,{\sigma }^2\right)$, we know that

$$\frac{1}{\sqrt{n}}\widehat{M}{X}^{\top }\epsilon \mid X~N\left(0,\widehat{M}\widehat{\Sigma }{\widehat{M}}^{\top }\right).$$

Denote $h=\widehat{\beta }-{\beta }^{*}$. Since β* is (s, s_g)-sparse, by (73), (76) and Cauchy-Schwarz inequality, with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$,

$$\begin{gathered} {\Vert h\Vert }_1\le {\Vert h_T\Vert }_1+{\Vert h_{T^c}\Vert }_1\le \sqrt{s}{\Vert h_T\Vert }_2+{\Vert h_{T^c}\Vert }_1 \\ \le \sqrt{s}{\Vert h\Vert }_2+{\Vert h_{T^c}\Vert }_1\le C\frac{s}{n}\lambda . \end{gathered}$$

$$\begin{gathered} {\Vert h\Vert }_{1,2}\le {\Vert h_{(G)}\Vert }_{1,2}+{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\le \sqrt{s_g}{\Vert h_{(G)}\Vert }_2+{\Vert h_{\left(G^c\right)}\Vert }_{1,2} \\ \le \sqrt{s_g}{\Vert h\Vert }_2+{\Vert h_{\left(G^c\right)}\Vert }_{1,2}\le C\frac{\sqrt{s\cdot s_g}}{n}\lambda . \end{gathered}$$

In addition, Lemma 4 shows that Σ⁻¹ is in the feasible set of (23) with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$. By the definition of $\widehat{M}$,

$$\begin{gathered} \underset{i}{\text{max\hspace{0.17em}}}{\Vert H_{\alpha }\left(e_i-\widehat{\Sigma }{\widehat{M}}^{\top }{e}_i\right)\Vert }_{\infty ,2}=\underset{i}{\text{max\hspace{0.17em}}}{\Vert H_{\alpha }\left(e_i-\widehat{\Sigma }{\widehat{m}}_i\right)\Vert }_{\infty ,2} \\ \le \gamma . \end{gathered}$$ (82)

Combining these facts, by Lemma 8 Part 2, we must have

$$\begin{gathered} {\Vert \left(I-\frac{1}{n}\widehat{M}X{X}^{\top }\right)\left(\widehat{\beta }-{\beta }^{*}\right)\Vert }_{\infty } \\ ={\text{max\hspace{0.17em}}}_i\left|\langle e_i-\widehat{\Sigma }{\widehat{M}}^{\top }{e}_i,h\rangle \right| \\ \le \alpha {\Vert h\Vert }_1+\gamma {\Vert h\Vert }_{1,2} \\ \le C\frac{s}{n}\alpha \lambda +C\frac{\sqrt{s\cdot s_g}}{n}\gamma \lambda \\ =\frac{C\left(s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)\right)}{n}\sigma \end{gathered}$$

with probability at least $1-C\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}\right)$. This has finished the proof of (24).

Next, we consider ${\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i$. By (82) and Lemma 8 Part 2, we have

$$1-\langle e_i,\widehat{\Sigma }{\widehat{m}}_i\rangle =\langle e_i,e_i-\widehat{\Sigma }{\widehat{m}}_i\rangle \le \alpha {\Vert e_i\Vert }_1+\gamma {\Vert e_i\Vert }_{1,2}=\alpha +\gamma .$$

Therefore, for any c ≥ 0,

$$\begin{gathered} {\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i\ge {\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i+c(1-\alpha -\gamma )-c\langle e_i,\widehat{\Sigma }{\widehat{m}}_i\rangle \\ \ge {\text{min}}_m\left\{m^{\top }\widehat{\Sigma }m+c(1-\alpha -\gamma )-c\langle e_i,\widehat{\Sigma }m\rangle \right\}. \end{gathered}$$

Since m = ce_i/2 achieves the minimum of the right hand side, we have

$${\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i\ge c(1-\alpha -\gamma )-\frac{c^2}{4}{\widehat{\Sigma }}_{i,i}.$$

If ${\widehat{\Sigma }}_{ii}>0$ for all 1 ≤ i ≤ p, by setting $c=2(1-\alpha -\gamma )/{\widehat{\Sigma }}_{i,i}$, we have

$${\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i\ge \frac{{(1-\alpha -\gamma )}^2}{{\widehat{\Sigma }}_{i,i}},\forall 1\le i\le p.$$ (83)

Moreover, by Lemma 6 Part 2 with u = v = e_i, we have

$$\mathbb{P}\left(\left|{\widehat{\Sigma }}_{i,i}-{\Sigma }_{i,i}\right|\ge \frac{c_{\text{min}}}{2}\right)\le 2\text{\hspace{0.17em}exp}(-cn).$$

By the union bound,

$$\begin{gathered} \mathbb{P}\left(\exists 1\le i\le p,\left|{\widehat{\Sigma }}_{i,i}-{\Sigma }_{i,i}\right|\ge \frac{c_{\text{min}}}{2}\right) \\ \le \sum _{i=1}^p\mathbb{P}\left(\left|{\widehat{\Sigma }}_{i,i}-{\Sigma }_{i,i}\right|\ge \frac{c_{\text{min}}}{2}\right) \\ \le db\cdot 2\text{\hspace{0.17em}exp}(-cn) \\ \le 2\text{\hspace{0.17em}exp}(-cn). \end{gathered}$$

Therefore, with probability at least 1 – 2 exp (−cn),

$$\frac{c_{\text{min}}}{2}\le {\widehat{\Sigma }}_{i,i}\le C_{\text{max}}+\frac{c_{\text{min}}}{2},\forall 1\le i\le p.$$

(83) and the previous inequality together imply that with probability at least 1 – 2 exp (−cn),

$${\widehat{m}}_i^{\top }\widehat{\Sigma }{\widehat{m}}_i\ge \frac{1}{2C_{\text{max}}},\forall 1\le i\le p.$$

(24) and the previous inequality together imply (25). □

Fig. 2.

Average estimation error in the noisy case

Biographies

T. Tony Cai received the Ph.D. degree from Cornell University, Ithaca, NY, in 1996. His research interests include high-dimensional statistics, machine learning, large- scale inference, nonparametric function estimation, functional data analysis, and statistical decision theory. He is the Daniel H. Silberberg Professor of Statistics and data science at the Wharton School of the University of Pennsylvania, Philadelphia, PA, USA. Dr. Cai is the recipient of the 2008 COPSS Presidents Award and a fellow of the Institute of Mathematical Statistics. He is a past editor of the Annals of Statistics.

Anru R. Zhang is the Eugene Anson Stead, Jr. M.D. Associate Professor in the Department of Biostatistics & Bioinformatics and Associate Professor in the Departments of Computer Science, Mathematics, and Statistical Science at Duke University. He was an assistant professor of statistics at the University of Wisconsin-Madison in 2015–2021. He obtained his bachelor’s degree from Peking University in 2010 and his Ph.D. from the University of Pennsylvania in 2015. His work focuses on high-dimensional statistical inference, non-convex optimization, statistical tensor analysis, computational complexity, and applications in genomics, microbiome, electronic health records, and computational imaging. He received the IMS Tweedie Award (2022), ASA Gottfried E. Noether Junior Award (2021), Bernoulli Society New Researcher Award (2021), ICSA Outstanding Young Researcher Award (2021), and NSF CAREER Award (2020).

Yuchen Zhou is a postdoctoral researcher in the Department of Statistics and Data Science, The Wharton School, University of Pennsylvania. He received the B.E. degree from Peking University in 2016 and the Ph.D. degree in statistics from the University of Wisconsin-Madison in 2021. His research interests include high-dimensional statistical inference, tensor data analysis, reinforcement learning and statistical learning theory.

Appendix

We collect all additional technical lemmas and their proofs in this section.

Lemma 5 (Bernstein-type Inequality for Soft-thresholded Sub-Gaussian Vectors):

Suppose the rows of $X\in {ℝ}^{n\times p}$ are independent sub-Gaussian vectors satisfying Assumption 1. $w\in {ℝ}^n$ is a fixed vector, Ω is a subset of {1, …, p} with |Ω| = r. Then

$$\mathbb{P}\left({\Vert \sum _{k=1}^n{w}_k{X}_{k,\Omega }\Vert }_2\ge \sqrt{C_{\text{max}}}\kappa {\Vert w\Vert }_2\cdot (\sqrt{r}+\sqrt{2t})\right)\le \text{exp}(-t).$$ (84)

For any fixed vector $w\in {ℝ}^n$ and fixed index subset Ω ⊆ {1, …, p} with |Ω| = r,

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\left(\delta \Vert w{\Vert }_2\right)}\left(w^{\top }{X}_{\Omega }\right)\Vert }_2\ge t\Vert w{\Vert }_2\right) \\ \le \left(\begin{array}{c}r\\ \lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}\right)\cdot \text{exp}\left(-{\left(t/\left(\kappa \sqrt{C_{\text{max}}}\right)-(t/\delta )\wedge \sqrt{r}\right)}_{+}^2/2\right)+\left(\begin{array}{c}r\\ \lceil {\left(t/\delta \right)}^2\rceil \end{array}\right)\cdot \text{exp}\left(-{\left(t/\left(\kappa \sqrt{C_{\text{max}}}\right)-\sqrt{\lceil {(t/\delta )}^2\rceil }\right)}_{+}^2/2\right). \end{gathered}$$ (85)

In particular, for any b ≥ r, if $\overline{\lambda }=C{\Vert w\Vert }_2\sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}}$, ${\overline{\lambda }}_g=\sqrt{s/s_g}\overline{\lambda }$, we have

$$\mathbb{P}\left({\Vert H_{\overline{\lambda }}\left(w^{\top }{X}_{\Omega }\right)\Vert }_2\ge {\overline{\lambda }}_g\right)\le \text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right).$$ (86)

Proof of Lemma 5. We only need to focus on the case where ∥w∥₂ = 1. Let $W_{\text{Ω}}=X_{\text{Ω}}{\Sigma }_{\text{Ω},\text{Ω}}^{-1/2}$, immediately we know that W_1,Ω, …, W_n,Ω are isotropic sub-Gaussian distributed. Then for any fixed w, w^⊤W_Ω is also an isotropic sub-Gaussian vector such that for any $\alpha \in {ℝ}^r$,

$$\begin{gathered} \mathbb{E}\text{\hspace{0.17em}exp}\left(w^{\top }{W}_{\Omega }\alpha \right)=\mathbb{E}\text{\hspace{0.17em}exp}\left(w^{\top }{X}_{\Omega }{\Sigma }_{\Omega ,\Omega }^{-1/2}\alpha \right) \\ =\mathbb{E}\text{\hspace{0.17em}exp}\left(w^{\top }X{\Sigma }^{-1/2}\left({\Sigma }^{1/2}\right).{}_{,\Omega }{\Sigma }_{\Omega ,\Omega }^{-1/2}\alpha \right) \\ \le \text{exp}\left({\kappa }^2{\Vert \left({\Sigma }^{1/2}\right).{}_{,\Omega }{\Sigma }_{\Omega ,\Omega }^{-1/2}\alpha \Vert }_2^2/2\right) \\ =\text{exp}\left({\kappa }^2\Vert \alpha {\Vert }_2^2/2\right). \end{gathered}$$

The last equation holds since (Σ^1/2)_Ω,·(Σ^1/2)·,_Ω = (Σ^1/2 Σ^1/2)_Ω,Ω = Σ_Ω,Ω.

By the tail inequality of sub-Gaussian quadratic form ([60, Theorem 2.1]),

$$\mathbb{P}\left({\Vert w^{\top }{W}_{\Omega }\Vert }_2^2\ge {\kappa }^2\left(r+2\sqrt{rt}+2t\right)\right)\le \text{exp}(-t).$$

By taking square-root of the previous inequality, we have

$$\mathbb{P}\left({\Vert w^{\top }{W}_{\Omega }\Vert }_2\ge \kappa {\Vert w\Vert }_2\cdot \left(\sqrt{r}+\sqrt{2t}\right)\right)\le \text{exp}(-t).$$

Also note that

$$\begin{gathered} {\Vert w^{\top }{X}_{\Omega }\Vert }_2={\Vert w^{\top }{W}_{\Omega }{\Sigma }_{\Omega ,\Omega }^{1/2}\Vert }_2\le \Vert {\Sigma }_{\Omega ,\Omega }^{1/2}\Vert {\Vert w^{\top }{W}_{\Omega }\Vert }_2 \\ \le {\Vert \Sigma \Vert }^{1/2}{\Vert w^{\top }{W}_{\Omega }\Vert }_2\le \sqrt{C_{\text{max}}}{\Vert w^{\top }{W}_{\Omega }\Vert }_2, \end{gathered}$$

we obtain (84).

For the second part of proof, note that

$$\begin{gathered} \mathbb{P}\left(\Vert H_{\delta }\left(w^{\top }{X}_{\Omega }\right)\Vert \ge t\right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq \Omega \text{, such that all entries of }\left|w^{\top }{X}_{\Lambda }\right|\ge \delta \text{ and }{\Vert w^{\top }{X}_{\Lambda }\Vert }_2\ge t\right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq \Omega ,\sqrt{\left|\Lambda \right|}\delta \le t,{\Vert w^{\top }{X}_{\Lambda }\Vert }_2\ge t\right)+\mathbb{P}\left(\exists \Lambda \subseteq \Omega ,\sqrt{\left|\Lambda \right|}\delta >t,\text{ all entries of }\left|w^{\top }{X}_{\Lambda }\right|\ge \delta \right) \\ \le \sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}}\mathbb{P}\left({\Vert w^{\top }{X}_{\Lambda }\Vert }_2\ge t\right)+\sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lceil {(t/\delta )}^2\rceil \end{array}}\mathbb{P}\left({\Vert w^{\top }{X}_{\Lambda }\Vert }_2\ge t\right). \end{gathered}$$

By the first part of this lemma,

$$\mathbb{P}\left({\Vert w^{\top }{X}_{\Lambda }\Vert }_2\ge \sqrt{C_{\text{max}}}\kappa {\Vert w\Vert }_{2}t\right)\le \text{exp}\left(-{(t-\sqrt{\left|\Lambda \right|})}_{+}^2/2\right).$$

Plug in this to the previous inequality, one has

$$\begin{gathered} \mathbb{P}\left(\Vert H_{\delta }\left(w^{\top }{X}_{\Omega }\right)\Vert \ge t\right) \\ \le \left(\begin{array}{c}r\\ \lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}\right)\cdot \text{exp}\left(-{\left(t/\left(\kappa \sqrt{C_{\text{max}}}\right)-(t/\delta )\wedge \sqrt{r}\right)}_{+}^2/2\right)+\left(\begin{array}{c}r\\ \left[{(t/\delta )}^2\right]\end{array}\right)\cdot \text{exp}\left(-{\left(t/\left(\kappa \sqrt{C_{\text{max}}}\right)-\sqrt{\lceil {(t/\delta )}^2\rceil }\right)}_{+}^2/2\right). \end{gathered}$$

Specifically, if $\delta =C\sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s}}$, $t=\sqrt{s/s_g}\delta$,

$$\begin{gathered} t/\left(\kappa \sqrt{C_{\text{max}}}\right)-\sqrt{\lceil {(t/\delta )}^2\rceil } \\ \ge C\sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}}-\sqrt{2\frac{s}{s_g}} \\ \ge C\sqrt{\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}}. \end{gathered}$$

Therefore, (85) shows that

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\overline{\lambda }}\left(w^{\top }{X}_{\Omega }\right)\Vert }_2\ge {\overline{\lambda }}_g\right) \\ \le r\lfloor {(t/\delta )}^2\rfloor \text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_q}\right)+r^{\lceil {(t/\delta )}^2\rceil }\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le 2r^{2s/s_g}\text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le \text{exp}\left(\text{\hspace{0.17em}log}2+\frac{2s\text{\hspace{0.17em}log}(eb)}{s_g}-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le \text{exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right). \end{gathered}$$

□

Lemma 6 (sub-Gaussian quadratic form concentrations): Suppose $Z\in {ℝ}^p$ is a sub-Gaussian vector satisfying Assumption 1.

1. For any fixed $u,v\in {ℝ}^p$, u, v ≠ 0, u^⊤ZZ^⊤v is sub-exponential such that for every t > 0,

   $$\mathbb{P}\left(\left|u^{\top }Z{Z}^{\top }v-\mathbb{E}{u}^{\top }Z{Z}^{\top }v\right|\ge t{\Vert u\Vert }_2{\Vert v\Vert }_2\right)\le C\text{\hspace{0.17em}exp}\left(-ct/{\kappa }^2\right).$$ (87)
2. In addition, suppose $X={\left[X_1^{\top },\dots ,X_n^{\top }\right]}^{\top }\in {ℝ}^{n\times p}$ is a random matrix with independent random sub-Gaussian rows satisfying Assumption 1,

   $$\mathbb{P}\left(\left|\frac{1}{n}\sum _{k=1}^n{u}^{\top }{X}_k{X}_k^{\top }v-u^{\top }\Sigma v\right|\ge t{\Vert u\Vert }_2{\Vert v\Vert }_2\right)\le 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$ (88)
3. More generally, for any fixed matrix $U\in {ℝ}^{p\times r}$, the following concentration inequality in spectral norm holds,

   $$\mathbb{P}\left({\Vert \frac{1}{n}\sum _{k=1}^n{U}^{\top }{X}_k{X}_k^{\top }v-U^{\top }\Sigma v\Vert }_2\ge t\Vert U\Vert {\Vert v\Vert }_2\right)\le 2\text{\hspace{0.17em}exp}\left(Cr-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$ (89)

Proof of Lemma 6. Since we can rescale u and v without essentially changing the problem, without loss of generality we assume ∥u∥₂ = ∥v∥₂ = 1. Let A = uv^⊤, then u^⊤ZZ^⊤v = Z^⊤uv^⊤Z = Z^⊤AZ. By Assumption 1, $\mathbb{E}Z=0$ and ${\Vert \langle Z,e_i\rangle \Vert }_{{\psi }_2}\le C\kappa$. By Hanson-Wright inequality ([61, Theorem 1.1]),

$$\begin{gathered} \mathbb{P}\left(\left|u^{\top }Z{Z}^{\top }v-\mathbb{E}{u}^{\top }Z{Z}^{\top }v\right|\ge t\right) \\ =\mathbb{P}\left(\left|Z^{\top }AZ-\mathbb{E}{Z}^{\top }AZ\right|\ge t\right) \\ \le 2\text{\hspace{0.17em}exp}\left[-c\text{\hspace{0.17em}min}\left(\frac{t^2}{{\kappa }^4{\Vert A\Vert }_{HS}^2},\frac{t}{{\kappa }^2\Vert A\Vert }\right)\right] \\ \le 2\text{\hspace{0.17em}exp}\left[-c\text{\hspace{0.17em}min}\left(\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right)\right], \end{gathered}$$

where

$$\begin{gathered} {\Vert A\Vert }_{HS}={\left(\sum _{i,j}{\left|a_{i,j}\right|}^2\right)}^{1/2}={\left(\sum _{i,j}{\left|u_i{v}_j\right|}^2\right)}^{1/2} \\ ={\Vert \text{u}\Vert }_{\text{2}}{\Vert \text{v}\Vert }_{\text{2}}=1, \\ \Vert A\Vert =\underset{{\Vert x\Vert }_2\le 1}{\text{max}}{\Vert Ax\Vert }_2=\underset{{\Vert x\Vert }_2\le 1}{\text{max}}{\Vert u{v}^{\top }x\Vert }_2 \\ ={\Vert u\Vert }_2\underset{{\Vert x\Vert }_2\le 1}{\text{max}}\left|v^{\top }x\right|={\Vert u\Vert }_2{\Vert v\Vert }_2=1. \end{gathered}$$

Therefore, for every t ≥ κ²,

$$\mathbb{P}\left(\left|u^{\top }Z{Z}^{\top }v-\mathbb{E}{u}^{\top }Z{Z}^{\top }v\right|\ge t\right)\le 2\text{\hspace{0.17em}exp}\left(-ct/{\kappa }^2\right).$$

Thus, there exists a constant c < log 2, for every t ≥ 0,

$$\mathbb{P}\left(\left|u^{\top }Z{Z}^{\top }v-\mathbb{E}{u}^{\top }Z{Z}^{\top }v\right|\ge t\right)\le 2\text{\hspace{0.17em}exp}\left(-ct/{\kappa }^2\right).$$

Notice that $\mathbb{E}{u}^{\top }{X}_k{X}_k^{\top }v=u^{\top }\Sigma v$, for all 1 ≤ k ≤ n, by Bernstein-type concentration inequality (c.f., [62, Proposition 5.16]),

$$\mathbb{P}\left(\left|\frac{1}{n}\sum _{k=1}^n{u}^{\top }{X}_k{X}_k^{\top }v-u^{\top }\Sigma v\right|\ge t\right)\le 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$

This has finished the proof of (88).

Finally, we consider (89), which can be done by an ε-net argument and the result in (88). For any $w\in {ℝ}^r$, ∥w∥₂ = 1, set u = Uw in (88), we have

$$\mathbb{P}\left(\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|\ge \frac{t}{2}{\Vert Uw\Vert }_2{\Vert v\Vert }_2\right)\le 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$

By [62, Lemma 5.3], we can find a $\frac{1}{2}$-net ${𝒩}_{\frac{1}{2}}$ of $S^{r-1}=\left\{x\mid x\in {ℝ}^r,{\Vert x\Vert }_2=1\right\}$ with $\left|{𝒩}_{\frac{1}{2}}\right|\le 5^r$. By the union bound,

$$\begin{gathered} \mathbb{P}\left(\forall w\in {𝒩}_{\frac{1}{2}},\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|\ge \frac{t}{2}{\Vert Uw\Vert }_2{\Vert v\Vert }_2\right) \\ \le 5^r\cdot 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right). \end{gathered}$$ (90)

For any $g\in {ℝ}^r$, g ≠ 0, set $x=\frac{g}{{\Vert g\Vert }_2}\in {\text{ arg max\hspace{0.17em}}}_{w\in {ℝ}^r,{\Vert w\Vert }_2=1}\left|w^{\top }g\right|$, we can find $y\in {𝒩}_{\frac{1}{2}}$ such that ${\Vert x-y\Vert }_2\le \frac{1}{2}$. By triangle inequality,

$$\begin{gathered} {\Vert g\Vert }_2-\left|y^{\top }g\right|=\left|x^{\top }g\right|-\left|y^{\top }g\right|\le \left|x^{\top }g-y^{\top }g\right| \\ \le {\Vert x-y\Vert }_2{\Vert g\Vert }_2\le \frac{1}{2}{\Vert g\Vert }_2. \end{gathered}$$

Therefore,

$$\begin{gathered} \underset{w\in {ℝ}^r,{\Vert w\Vert }_2=1}{\text{sup}}\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right| \\ \le 2\underset{w\in {𝒩}_{\frac{1}{2}}}{\text{sup}}\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|. \end{gathered}$$

The (90) and the previous inequality together, also notice that $\Vert U\Vert ={\text{sup}}_{w\in {ℝ}^r,{\Vert w\Vert }_2=1}{\Vert Uw\Vert }_2$, we have

$$\begin{gathered} \mathbb{P}\left(\underset{w\in {ℝ}^r,{\Vert w\Vert }_2=1}{\text{sup}}\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|\ge t\Vert U\Vert {\Vert v\Vert }_2\right) \\ \le \mathbb{P}\left(\underset{w\in {𝒩}_{\frac{1}{2}}}{\text{sup}}\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|\ge \frac{t}{2}\Vert U\Vert {\Vert v\Vert }_2\right) \\ \le \mathbb{P}\left(\forall w\in {𝒩}_{\frac{1}{2}},\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|\ge \frac{t}{2}{\Vert Uw\Vert }_2{\Vert v\Vert }_2\right) \\ \le 5^r\cdot 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right). \end{gathered}$$ (91)

Finally, note that

$${\Vert \frac{1}{n}\sum _{k=1}^n{U}^{\top }{X}_k{X}_k^{\top }v-U^{\top }\Sigma v\Vert }_2=\underset{w\in {ℝ}^r,{\Vert w\Vert }_2=1}{\text{sup}}\left|\frac{1}{n}\sum _{k=1}^n{w}^{\top }{U}^{\top }{X}_k{X}_k^{\top }v-w^{\top }{U}^{\top }\Sigma v\right|,$$

we have proved (89). □

We collect the random matrix properties of X in the following lemma. These properties will be extensively used in the main content of the paper.

Lemma 7: Suppose $X={\left[X_1^{\top },\dots ,X_n^{\top }\right]}^{\top }\in {ℝ}^{n\times p}$ is a random matrix with independent random sub-Gaussian rows satisfying Assumption 1.

1. Suppose T ⊆ {1, …, p} is with cardinality s. Then,

   $$\mathbb{P}\left(\Vert \frac{1}{n}{X}_T^{\top }{X}_T{\Sigma }_{T,T}^{-1}-I_s\Vert \ge t\right)\le 2\text{\hspace{0.17em}exp}\left(Cs-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right);$$ (92)
2. For any fixed vector $\alpha \in {ℝ}^s$, δ > 0, and fixed index subset Ω ⊆ T^c satisfying |Ω| = r, $t\ge \delta \ge C\left({\text{max}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\right){\Vert \alpha \Vert }_2$,

   $$\begin{gathered} \mathbb{P}\left({\Vert H_{\delta }\left({\alpha }^{\top }{X}_T^{\top }{X}_{\Omega }/n\right)\Vert }_2\ge t\right) \\ \le \left(\begin{array}{c}r\\ \lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}\right)\cdot \text{exp}\left(C\lfloor {(t/\delta )}^2\rfloor \wedge r-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4{\Vert \alpha \Vert }_2^2},\frac{t}{{\kappa }^2{\Vert \alpha \Vert }_2}\right\}\right)+{\left(\begin{array}{c}r\\ \lceil {(t/\delta )}^2\rceil \end{array}\right)}_{+}\cdot \text{exp}\left(C\lceil {(t/\delta )}^2\rceil -cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4{\Vert \alpha \Vert }_2^2},\frac{t}{{\kappa }^2{\Vert \alpha \Vert }_2}\right\}\right). \end{gathered}$$ (93)

Here, H_λ(·) is the soft-thresholding estimator at level λ.

Proof of Lemma 7.

1. The first statement is via ε-net. Denote $W_T=X_T{\Sigma }_{T,T}^{-1/2}$, then the rows of W_T are independent isotropic sub-Gaussian distributed. For any fixed vector $x\in S^{s-1}=\left\{x:x\in {ℝ}^s,{\Vert x\Vert }_2=1\right\}$, by [62, Lemma 5.5], $Z_i=\langle {\left(W_T\right)}_{i.}^{\top },x\rangle$ are independent sub-Gaussian random variables with $\mathbb{E}{Z}_i^2=1$ and ${\Vert Z_i\Vert }_{{\psi }_2}\le C\kappa$. Therefore, by Remark 5.18 and Lemma 5.14 in [62], ${\Vert Z_i^2-1\Vert }_{{\psi }_1}\le 2{\Vert Z_i^2\Vert }_{{\psi }_1}\le 4{\Vert Z_i\Vert }_{{\psi }_2}^2\le C{\kappa }^2$. Bernstein-type inequality shows that

   $$\begin{gathered} \mathbb{P}\left(\left|\frac{1}{n}{\Vert W_{T}x\Vert }_2^2-1\right|\ge \frac{t}{2}\right) \\ =\mathbb{P}\left(\left|\frac{1}{n}\sum _{i=1}^n\left(Z_i^2-1\right)\right|\ge \frac{t}{2}\right) \\ \le 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right). \end{gathered}$$
   By [62, Lemma 5.2], we can find a $\frac{1}{4}$-net ${𝒩}_{\frac{1}{4}}$ of $S^{s-1}=\left\{x:x\in {ℝ}^s,{\Vert x\Vert }_2=1\right\}$ with $\left|{𝒩}_{\frac{1}{4}}\right|\le 9^s$. The union bound tells us

   $$\mathbb{P}\left(\underset{x\in {𝒩}_{\frac{1}{4}}}{\text{max}}\left|\frac{1}{n}{\Vert W_{T}x\Vert }_2^2-1\right|\ge \frac{t}{2}\right)\le 9^s\cdot 2\text{\hspace{0.17em}exp}\left(-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$ (94)
   By [62, Lemma 5.4],

   $$\begin{gathered} \Vert \frac{1}{n}{W}_T^{\top }{W}_T-I_s\Vert \\ \le 2\underset{x\in {𝒩}_{\frac{1}{4}}}{\text{ max\hspace{0.17em}}}\left|\langle \left(\frac{1}{n}{W}_T^{\top }{W}_T-I_s\right)x,x\rangle \right| \\ =2\underset{x\in {𝒩}_{\frac{1}{4}}}{\text{ max\hspace{0.17em}}}\left|\frac{1}{n}{\Vert W_{T}x\Vert }_2^2-1\right|. \end{gathered}$$ (95)
   Since c_min ≤ σ_min(Σ) ≤ σ_max(Σ) ≤ C_max, we have $\Vert {\Sigma }_{T,T}^{1/2}\Vert \le \sqrt{C_{\text{max}}}$ and $\Vert {\Sigma }_{T,T}^{-1/2}\Vert \le 1/\sqrt{c_{\text{min}}}$. Therefore,

   $$\begin{gathered} \Vert \frac{1}{n}{X}_T^{\top }{X}_T^{{\Sigma }_{T,T}^{-1}}-I_s\Vert \\ =\Vert {\Sigma }_{T,T}^{1/2}\left(\frac{1}{n}{W}_T^{\top }{W}_T-I_s\right){\Sigma }_{T,T}^{-1/2}\Vert \\ \le \Vert {\Sigma }_{T,T}^{1/2}\Vert \Vert \frac{1}{n}{W}_T^{\top }{W}_T-I_s\Vert \Vert {\Sigma }_{T,T}^{-1/2}\Vert \\ \le \sqrt{\frac{C_{\text{max}}}{c_{\text{min}}}}\Vert \frac{1}{n}{W}_T^{\top }{W}_T-I_s\Vert . \end{gathered}$$ (96)

   Combine (94), (95) and (96) together, we have arrived at the conclusion.

2. Now we consider the proof for (93). Note that ${\Vert H_{\delta }\left({\alpha }^{\top }{X}_T^{\top }{X}_{\text{Ω}}\right)\Vert }_2\ge t$ implies that there exists Λ ⊂ Ω such that all entry of $\left|{\alpha }^{\top }{X}_T^{\top }{X}_{\text{Λ}}\right|$ are greater than δ, and ${\Vert \left|{\alpha }^{\top }{X}_T^{\top }{X}_{\text{Λ}}\right|-\delta \Vert }_2\ge t$. Thus,

   $$\begin{gathered} \mathbb{P}\left({\Vert H_{\delta }\left({\alpha }^{\top }{X}_T^{\top }{X}_{\Omega }/n\right)\Vert }_2\ge t\right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq \Omega ,\text{ such that all entries of }\left|{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\right|\ge \delta ,\text{ and }{\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq \Omega ,\sqrt{\left|\Lambda \right|}\delta \le t,{\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right)+\mathbb{P}\left(\exists \Lambda \subseteq \Omega ,\sqrt{\left|\Lambda \right|}\delta >t,\text{ all entries of }\left|{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\right|\ge \delta \right) \\ \le \sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lceil {(t/\delta )}^2\rceil \end{array}}\mathbb{P}\left(\text{ all entries of }\left|{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\right|\ge \delta \right)++\sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}}\mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right) \\ \le \sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lceil {(t/\delta )}^2\rceil \end{array}}\mathbb{P}\left({\Vert {\alpha }^{\top }{X}_{\mid T}^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right)+\sum _{\begin{array}{c}\Lambda \subseteq \Omega \\ \left|\Lambda \right|=\lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}}\mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right). \end{gathered}$$ (97)
   Since $t\ge \delta \ge C{\text{ max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2{\Vert \alpha \Vert }_2$, we know that no matter |Λ| = ⌊(t/δ)²⌋ ∧ r or ⌈(t/δ)²⌉,

   $$\begin{gathered} 2C_{\text{max}}\sqrt{\lceil {(t/\delta )}^2\rceil }\left({\text{max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\right)\Vert \alpha {\Vert }_2 \\ \le 2C_{\text{max}}\sqrt{2}(t/\delta )\left({\text{max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\right)\Vert \alpha {\Vert }_2\le t. \end{gathered}$$
   By Part 3 of Lemma 6, for any Λ ⊆ Ω, $t\ge 2C_{\text{max}}\sqrt{|\text{Λ}|}{\text{ max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2{\Vert \alpha \Vert }_2$, we have

   $$\begin{gathered} \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t\right) \\ \le \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t-{\Vert \mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\right) \\ \le \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t-{\Vert {\Sigma }_{\Lambda ,T}\alpha \Vert }_2\right) \\ =\mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t-{\left(\sum _{i\in \Lambda }{\left({\Sigma }_{i,T}\alpha \right)}^2\right)}^{1/2}\right) \\ \le \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t-\sqrt{\left|\Lambda \right|}\underset{i\in T^c}{\text{\hspace{0.17em}max\hspace{0.17em}}}\left|{\Sigma }_{i,T}\alpha \right|\right) \\ \le \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t-\sqrt{\left|\Lambda \right|}{\text{max\hspace{0.17em}}}_{i\in T^c}{\Vert {\Sigma }_{i,T}{\Sigma }_{T,T}^{-1}\Vert }_2\Vert {\Sigma }_{T,T}\Vert {\Vert \alpha \Vert }_2\right) \\ \le \mathbb{P}\left({\Vert {\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n-\mathbb{E}{\alpha }^{\top }{X}_T^{\top }{X}_{\Lambda }/n\Vert }_2\ge t/2\right) \\ \le 2\text{\hspace{0.17em}exp}\left(C\left|\Lambda \right|-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4{\Vert \alpha \Vert }_2^2},\frac{t}{{\kappa }^2{\Vert \alpha \Vert }_2}\right\}\right). \end{gathered}$$
   Combine (97) and the previous inequality, one obtains

   $$\begin{gathered} \mathbb{P}\left({\Vert H_{\delta }\left({\alpha }^{\top }{X}_T^{\top }{X}_{\Omega }/n\right)\Vert }_2\ge t\right) \\ \le \left(\begin{array}{c}r\\ \lfloor {(t/\delta )}^2\rfloor \wedge r\end{array}\right)\cdot \text{exp}\left(C\lfloor {(t/\delta )}^2\rfloor \wedge r-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4{\Vert \alpha \Vert }_2^2},\frac{t}{{\kappa }^2{\Vert \alpha \Vert }_2}\right\}\right)+{\left(\begin{array}{c}r\\ \lceil {(t/\delta )}^2\rceil \end{array}\right)}_{+}\cdot \text{exp}\left(C\lceil {(t/\delta )}^2\rceil -cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4{\Vert \alpha \Vert }_2^2},\frac{t}{{\kappa }^2{\Vert \alpha \Vert }_2}\right\}\right). \end{gathered}$$

   □

Lemma 8 (Properties of Soft-thresholding):

1. Suppose a, b > 0, $x,y\in ℝ$, H. (·) is the soft-thresholding operator satisfying H_a(x) = sgn(x) · (|x| − a)₊. Then the following triangular inequality holds,

   $$\left|H_{a+b}(x+y)\right|\le \left|H_a(x)\right|+\left|H_b(y)\right|.$$ (98)
2. Suppose a, b > 0, $x,y\in {ℝ}^p$, if ∥H_a(x)∥_∞,2 ≤ b, then

   $$\left|\langle x,y\rangle \right|\le a{\Vert y\Vert }_1+b{\Vert y\Vert }_{1,2}.$$ (99)

Proof of Lemma 8.

1)

$$\begin{gathered} \left|H_{a+b}(x+y)\right|={\left(|x+y|-a-b\right)}_{+} \\ \le {\left(\left|x\right|-a+\left|y\right|-b\right)}_{+} \\ \le {\left(\left|x\right|-a\right)}_{+}+{\left(\left|y\right|-b\right)}_{+} \\ =\left|H_a(x)\right|+\left|H_b(y)\right|. \end{gathered}$$

2)

$$\begin{gathered} \left|\langle x,y\rangle \right|\le \left|\langle H_a(x),y\rangle \right|+\left|\langle x-H_a(x),y\rangle \right| \\ =\left|\sum _{j=1}^d\langle {\left[H_a(x)\right]}_{(j)},y_{(j)}\rangle \right|+\left|\langle x-H_a(x),y\rangle \right| \\ \le \sum _{j=1}^d{\Vert {\left[H_a(x)\right]}_{(j)}\Vert }_2{\Vert y_{(j)}\Vert }_2+{\Vert x-H_a(x)\Vert }_{\infty }{\Vert y\Vert }_1 \\ \le {\Vert H_a(x)\Vert }_{\infty ,2}{\Vert y\Vert }_{1,2}+\Vert x-a{\Vert y\Vert }_1 \\ \le b{\Vert y\Vert }_{1,2}+a{\Vert y\Vert }_1. \end{gathered}$$

□

Lemma 9: Suppose $X={\left[X_1^{\top },\dots ,X_n^{\top }\right]}^{\top }$ is a random matrix with independent random sub-Gaussian rows satisfying Assumption 1, ${\epsilon }_i\stackrel{i.i.d.}{~}N\left(0,{\sigma }^2\right)$ Suppose T ⊆ {1, …, p} is with cardinality s, $P\in {ℝ}^{n\times n}$ is a projection matrix and independent of X_T. Then, for any t ≥ log(es),

$$\mathbb{P}\left({\Vert X_T^{\top }P\epsilon \Vert }_{\infty }\ge C\kappa \sqrt{nt}{\sigma }^2\right)\le e^{-n}+e^{-Ct}.$$

Proof of Lemma 9. For fixed vector $w\in {ℝ}^n$, since Assumption 2 is satisfied, for i ∈ T, X_1i, …, X_ni are independent sub-Gaussian distributed such that

$$\begin{gathered} \mathbb{E}\text{\hspace{0.17em}exp}\left(t{X}_{ji}\right)=\mathbb{E}\text{\hspace{0.17em}exp}\left(t{e}_i^{\top }{\Sigma }^{1/2}{\Sigma }^{-1/2}{X}_{j\cdot }^{\top }\right) \\ \le \text{exp}\left(\frac{{\kappa }^2{\Vert {\Sigma }^{1/2}{e}_i\Vert }_2^2{t}^2}{2}\right) \\ \le \text{exp}\left(\frac{{\kappa }^2{\Sigma }_{i,i}{t}^2}{2}\right) \\ \le \text{exp}\left(\frac{C_{\text{max}}{\kappa }^2{t}^2}{2}\right). \end{gathered}$$

By Hoeffding-type inequality,

$$\mathbb{P}\left(\left|X_{\cdot i}^{\top }w\right|\ge t{\Vert w\Vert }_2\right)\le 2\text{\hspace{0.17em}exp}\left(-c\frac{t^2}{{\kappa }^2}\right).$$ (100)

Moreover, by [63, Lemma 1], for any x ≥ 0,

$$\mathbb{P}\left(\sum _{i=1}^n{\epsilon }_i^2\ge (n+2\sqrt{nx}+2x){\sigma }^2\right)\le e^{-x}.$$

Set x = n in the last inequality, we have

$$\mathbb{P}\left({\Vert \epsilon \Vert }_2\ge \sqrt{5n{\sigma }^2}\right)\le e^{-n}.$$ (101)

Combine (100) and (101) together and notice that ∥Pε∥₂ ≤ ∥ε∥₂, we have

$$\begin{gathered} \mathbb{P}\left({\Vert X_T^{\top }P\epsilon \Vert }_{\infty }\ge C\kappa \sqrt{nt{\sigma }^2}\right) \\ \le \sum _{i\in T}\mathbb{P}\left(\left|X_{\cdot i}^{\top }P\epsilon \right|\ge C\kappa \sqrt{nt{\sigma }^2}\right) \\ \le \mathbb{P}\left({\Vert P\epsilon \Vert }_2\ge \sqrt{5n{\sigma }^2}\right)+\sum _{i\in T}\mathbb{P}\left(\left|X_{\cdot i}^{\top }P\epsilon \right|\ge C\kappa \sqrt{nt{\sigma }^2},{\Vert P\epsilon \Vert }_2\le \sqrt{5n{\sigma }^2}\right) \\ \le \mathbb{P}\left({\Vert \epsilon \Vert }_2\ge \sqrt{5n{\sigma }^2}\right)+\sum _{i\in T}\mathbb{P}\left(\left|X_{.i}^{\top }P\epsilon \right|\ge C\kappa \sqrt{nt{\sigma }^2}\mid {\Vert P\epsilon \Vert }_2\le \sqrt{5n{\sigma }^2}\right) \\ \le e^{-n}+s\cdot 2\text{\hspace{0.17em}exp}(-Ct)\le e^{-n}+e^{-Ct}. \end{gathered}$$

□

Lemma 10: With probability at least 1 − Ce^−cn/s, the approximate dual certificate defined in (45) can be written as u = X^⊤w, where ${\Vert w\Vert }_2\le C\sqrt{s/n}$.

Proof of Lemma 10. By (45), we have u = X^⊤w, where $w={\left(w_1^{\top },\dots ,w_{l_{\text{max}}}^{\top }\right)}^{\top }$ and $w_l=\frac{1}{n_l}{X}_{I_l,T}^{{\Sigma }_{T,T}^{-1}}{q}_{l-1}$. Thus ${\Vert w\Vert }_2^2={\sum }_{l=1}^{l_{\text{max}}}{\Vert w_l\Vert }_2^2$. Also note that

$$\begin{gathered} \frac{1}{n_l}{\Vert X_{I_l,T}{\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2^2 \\ =\langle \frac{1}{n_l}{X}_{I_l,T}^{\top }{X}_{I_l,T}{\Sigma }_{T,T}^{-1}{q}_{l-1},{\Sigma }_{T,T}^{-1}{q}_{l-1}\rangle \\ =\langle \left(\frac{1}{n_l}{X}_{I_l,T}^{\top }{X}_{I_l,T}{\Sigma }_{T,T}^{-1}-I_{\left|T\right|}\right)q_{l-1},{\Sigma }_{T,T}^{-1}{q}_{l-1}\rangle +{\Vert {\Sigma }_{T,T}^{-1/2}{q}_{l-1}\Vert }_2^2 \\ =\langle -q_l,{\Sigma }_{T,T}^{-1}{q}_{l-1}\rangle +{\Vert {\Sigma }_{T,T}^{-1/2}{q}_{l-1}\Vert }_2^2 \\ \le {\Vert q_l\Vert }_2{\Vert {\Sigma }_{T,T}^{-1}{q}_{l-1}\Vert }_2+{\Vert {\Sigma }_{T,T}^{-1/2}{q}_{l-1}\Vert }_2^2 \\ \le \frac{1}{c_{\text{min}}}{\Vert q_l\Vert }_2{\Vert q_{l-1}\Vert }_2+\frac{1}{c_{\text{min}}}{\Vert q_{l-1}\Vert }_2^2\le \frac{2}{c_{\text{min}}}{\Vert q_{l-1}\Vert }_2^2 \end{gathered}$$

By (53), with probability at least 1 − C exp(−cn/s),

$$\begin{gathered} {\Vert w\Vert }_2^2\le \sum _{l=1}^{l_{\text{max}}}\frac{C}{n_l}{\Vert q_{l-1}\Vert }_2^2 \\ \le \frac{C}{n}{(2\sqrt{s})}^2+\frac{C}{n}{(2\sqrt{s/\text{\hspace{0.17em}log}(es)})}^2+\frac{C\text{\hspace{0.17em}log}(es)}{n}\sum _{l=3}^{l_{\text{max}}}{\left(2^{4-l}\sqrt{s}/\text{\hspace{0.17em}log}(es)\right)}^2 \\ \le C\frac{s}{n}. \end{gathered}$$

□

Proof of Lemma 4. For any 1 ≤ i ≤ p, 1 ≤ j ≤ d, ,Λ ⊆ (j), |Λ| = k, by Lemma 6 with

$$v={\Sigma }^{-1}{e}_i,U\in {ℝ}^{p\times k},U_{\left[\Lambda ,:\right]}=I,U_{\left[{\Lambda }^c,:\right]}=0,$$

we have

$$\mathbb{P}\left({\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge t\right)\le 2\text{\hspace{0.17em}exp}\left(Ck-cn\text{\hspace{0.17em}min}\left\{\frac{t^2}{{\kappa }^4},\frac{t}{{\kappa }^2}\right\}\right).$$ (102)

By the same method in Lemma 5 Part 2,

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\alpha }\left({\left(e_i\right)}_{(j)}-\frac{1}{n}{X}_{(j)}^{\top }X{\Sigma }^{-1}{e}_i\right)\Vert }_2\ge \gamma \right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq (j),\text{ all entries of }\left|{\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\right|\ge \alpha \text{ and }{\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge \gamma \right) \\ \le \mathbb{P}\left(\exists \Lambda \subseteq (j),\sqrt{\left|\Lambda \right|}\alpha \le \gamma ,{\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge \gamma \right)+\mathbb{P}\left(\exists \Lambda \subseteq (j),\sqrt{\left|\Lambda \right|}\alpha >\gamma ,\text{ all entries of }\left|{\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\right|\ge \alpha \right) \\ \le \sum _{\begin{array}{c}\Lambda \subseteq (j)\\ \left|\Lambda \right|=\lceil s/s_g\rceil \end{array}}\mathbb{P}\left(\text{all entries of }\left|{\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\right|\ge \alpha \right)+\sum _{\begin{array}{c}\Lambda \subseteq (j)\\ \left|\Lambda \right|=\lfloor s/s_g\rfloor \end{array}}\mathbb{P}\left({\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge \gamma \right) \\ \le \sum _{\begin{array}{c}\Lambda \subseteq (j)\\ \left|\Lambda \right|=\lfloor s/s_g\rfloor \end{array}}\mathbb{P}\left({\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge \gamma \right)+\sum _{\begin{array}{c}\Lambda \subseteq (j)\\ \left|\Lambda \right|=\lceil s/s_g\rceil \end{array}}\mathbb{P}\left({\Vert {\left(e_i\right)}_{\Lambda }-\frac{1}{n}{X}_{\Lambda }^{\top }X{\Sigma }^{-1}{e}_i\Vert }_2\ge \gamma \right). \end{gathered}$$

Combine (102) and the previous inequality together, we have

$$\begin{gathered} \mathbb{P}\left({\Vert H_{\alpha }\left({\left(e_i\right)}_{(j)}-\frac{1}{n}{X}_{(j)}^{\top }X{\Sigma }^{-1}{e}_i\right)\Vert }_2\ge \gamma \right) \\ \le \left(\begin{array}{c}{b}_j\\ \lfloor s/s_g\rfloor \end{array}\right)\cdot 2\text{\hspace{0.17em}exp}\left(C\lfloor s/s_g\rfloor -cn\cdot C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_{g}n}\right)+\left(\begin{array}{c}{b}_j\\ \lceil s/s_g\rceil \end{array}\right)\cdot 2\text{\hspace{0.17em}exp}\left(C\lceil s/s_g\rceil -cn\cdot C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_{g}n}\right) \\ \le 4{\left(\frac{2e{s}_{g}b}{s}\right)}^{2s/s_g}\cdot \text{exp}\left(Cs/s_g-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le 4\text{\hspace{0.17em}exp}\left(\frac{2s}{s_g}\text{\hspace{0.17em}log}\left(\frac{2e{s}_{g}b}{s}\right)+Cs/s_g-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right). \end{gathered}$$ (103)

By (103) and the union bound, we have

$$\begin{gathered} \mathbb{P}\left({\text{max\hspace{0.17em}}}_{1\le i\le p}{\Vert H_{\alpha }\left(e_i-\frac{1}{n}{X}^{\top }X{\Sigma }^{-1}{e}_i\right)\Vert }_{\infty ,2}\le \gamma \right) \\ \le \sum _{i=1}^p\sum _{j=1}^d\mathbb{P}\left({\Vert H_{\alpha }\left({\left(e_i\right)}_{(j)}-\frac{1}{n}{X}_{(j)}^{\top }X{\Sigma }^{-1}{e}_i\right)\Vert }_2\ge \gamma \right) \\ \le d^{2}b\cdot 4\text{\hspace{0.17em}exp}\left(\frac{2s}{s_g}\text{\hspace{0.17em}log}\left(\frac{2e{s}_{g}b}{s}\right)+Cs/s_g-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le 4\text{\hspace{0.17em}exp}\left(2\text{\hspace{0.17em}log}\left(s_g\right)+2\text{\hspace{0.17em}log}\left(d/s_g\right)+\frac{3s}{s_g}\text{\hspace{0.17em}log}(2eb)+Cs/s_g-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right) \\ \le 4\text{\hspace{0.17em}exp}\left(-C\frac{s\text{\hspace{0.17em}log}\left(e{s}_{g}b\right)+s_g\text{\hspace{0.17em}log}\left(d/s_g\right)}{s_g}\right). \end{gathered}$$

□