Graph-based sparse linear discriminant analysis for high-dimensional classification

Abstract

Linear discriminant analysis (LDA) is a well-known classification technique that enjoyed great success in practical applications. Despite its effectiveness for traditional low-dimensional problems, extensions of LDA are necessary in order to classify high-dimensional data. Many variants of LDA have been proposed in the literature. However, most of these methods do not fully incorporate the structure information among predictors when such information is available. In this paper, we introduce a new high-dimensional LDA technique, namely graph-based sparse LDA (GSLDA), that utilizes the graph structure among the features. In particular, we use the regularized regression formulation for penalized LDA techniques, and propose to impose a structure-based sparse penalty on the discriminant vector β. The graph structure can be either given or estimated from the training data. Moreover, we explore the relationship between the within-class feature structure and the overall feature structure. Based on this relationship, we further propose a variant of our proposed GSLDA to utilize effectively unlabeled data, which can be abundant in the semi-supervised learning setting. With the new regularization, we can obtain a sparse estimate of β and more accurate and interpretable classifiers than many existing methods. Both the selection consistency of β estimation and the convergence rate of the classifier are established, and the resulting classifier has an asymptotic Bayes error rate. Finally, we demonstrate the competitive performance of the proposed GSLDA on both simulated and real data studies.

1. Introduction

Classification problems are commonly seen in practice. There are many existing classification techniques in the literature; see [2, 17] for a comprehensive review. Among various existing methods, linear discriminant analysis (LDA) has a long history and remains an important tool in the standard classification toolbox. LDA can be viewed as a rule for a classification problem of two Gaussian populations with a common covariance matrix. Despite its seemingly strong assumptions, LDA often works well in practice, especially for low-dimensional problems [15]. It mimics Bayes’ rule and has a simple closed form which only involves the within-class sample covariance matrix and group averages. Given these estimates, the original formulation for the discriminant vector of LDA is computed as the product of the inverse within-class sample covariance matrix and the mean difference vector. Thus, standard LDA can be computed and implemented easily in the traditional low-dimensional setting. LDA also has interpretations beyond the Gaussian model. In particular, the same formulation can be obtained from Fisher’s discriminant analysis problem [13], the optimal scoring problem [16], and linear regression [17].

Despite the usefulness of LDA, it needs to be adapted when the dimension of features is high. For example, the form of standard LDA is only valid when the sample covariance matrix is invertible. Moreover, as the dimension grows, the errors in the sample covariance and group means accumulate and consequently LDA can become increasingly unstable [11, 35]. To address this problem, a number of LDA extensions have been proposed for high-dimensional scenarios.

The existing high-dimensional LDA methods in the literature can be roughly divided into two categories, plugin approaches and direct approaches. A plug-in approach tackles high-dimensional problems by using regularized estimates for the within-class covariance matrix and group means. For example, the naive Bayes method, or the independence rule, treats the covariance matrix as diagonal. Bickel and Levina [1] showed that it outperforms LDA with Moore–Penrose pseudoinverse covariance matrix when the dimension grows faster than the sample size. To further reduce the instability of LDA, Tibshirani et al. [36] additionally used shrunken estimates of group means. Fan and Fan [11] showed that, even under the independence feature assumption, naive Bayes can be as bad as random guessing due to error accumulation in group means. They resolved this issue by reducing the dimension via feature screening. In contrast to these independence rules, Shao et al. [35] assumed sparsity of the covariance matrix and the mean difference vector, and used thresholded estimates to construct a sparse LDA classifier. It was shown to be asymptotically optimal under certain conditions. All of these methods adopt the original formulation of LDA by calculating some improved estimates of the covariance matrix and group means. Thus, some strong assumptions on the covariance matrix and the group means need to be imposed for the resulting LDA rule.

In contrast to the plug-in methods, direct approaches aim at estimating the discriminant vector β directly. Since LDA can also be obtained from some risk minimization problems, it can be extended to high-dimensional scenarios via these formulations with regularization on β. For example, Wu et al. [41] considered Fisher’s discriminant analysis and proposed an ℓ₁ -penalized version for dimension reduction. The corresponding problem has a piece-wise linear solution path which can be computed efficiently. Witten and Tibshirani [39] also used Fisher’s discriminant analysis formulation for a general K-class problem with a general regularization. Clemmensen et al. [10] proposed the optimal scoring formulation with the ℓ₁ penalty. Following the idea of minimizing the misclassification rates, Fan et al. [12] proposed a method closely related to the method by Wu et al. [41] and directly computed the misclassification rate of the classifier. Mai et al. [26] took advantage of the regression formulation and estimated the discriminant vector of LDA by solving a Lasso-type problem, which was shown to have the same solution path as the method of Wu et al. [41] and the method of Clemmensen et al. [10] when K = 2; see [25]. Using a different idea of direct estimation, Cai and Liu [6] formed a linear programming problem to estimate β and showed that the error rate of the estimated classifier is close to the Bayes rule under certain conditions. Compared to plug-in approaches, these methods estimate LDA directly and the assumptions can be less stringent since only the sparsity of the discriminant vector of LDA is assumed [6].

Both plug-in and direct methods can work well for certain practical problems. However, these methods do not utilize the feature structure information when available. In practice, features are often correlated with some structure. Such structure can usually be represented by an undirected graph $\mathcal{G}$. Connected features may work together and thus be effective or not effective simultaneously for classification. For instance, in the diagnosis of a disease using genetic information, genes are naturally grouped by their functions or gene pathways. Relevant genes tend to contribute or not contribute to the disease together. Moreover, when the population in consideration is Gaussian, the conditional independence graph, or Gaussian graphical model, often represents a natural structure. By considering such structure information, we are likely to be able to construct a better classifier. For regression problems, there are some methods that utilize the graph structure in the literature; see, e.g., [3, 18, 33, 53]. For example, Li and Li [19] proposed a penalty on the coefficient difference of each pair of connected features. Yang et al. [42] used pairwise ℓ_∞ penalties on relevant features to encourage simultaneous inclusion and exclusion. Based on the decomposition of the regression coefficient vector, Yu and Liu [44] proposed a node-wise penalty. In particular, the regularization term is the summation of penalties over all nodes rather than all edges. Compared to pairwise penalties, the node-wise penalty is better motivated and computationally efficient. More recently, Zhao and Shojaie [51] proposed new inference methods for such graph-constrained estimation.

Despite great progress for regression problems, much less research has been done for classification problems. Structured penalties such as group Lasso and fused Lasso have been employed in classification methods [27, 39], but they are not applicable to a general sparse graph structure among predictors. Zhang et al. [49] considered logistic regression with a combination of ℓ₁ penalty and pairwise ℓ₂ difference penalty. Min et al. [29] generalized the regularization and provided a unified algorithm. However, both methods may also suffer from too much computational burden in high dimensions. Very recently, Wu et al. [40] proposed an unsupervised graph-based variable screening method for general problems.

In this paper, we propose a new method, called graph-based sparse LDA (GSLDA), that exploits the graphical structure of features. GSLDA estimates LDA in high dimensions directly by solving a convex optimization problem. Similar to the sparse regression method in [44], we incorporate the graph structure through a node-wise penalty. In the presence of an underlying feature structure, the new method outperforms existing high-dimensional LDA methods by utilizing the structure directly. As a key component, the graphical structure can be either given or estimated from the training data. In addition, we investigate the relationship between the within-class inverse covariance matrix and overall inverse covariance matrix. Based on these findings, we propose a variant of GSLDA that can utilize unlabeled data, which are often much more accessible than labeled data. We name this variant as the semi-supervised GSLDA. Selection consistency is shown for the estimated discriminant vector. Moreover, we show that the misclassification rate of our classifier converges to the Bayes error rate at a fast rate under certain conditions. Numerical studies are used to demonstrate the performance of this method. In particular, the semi-supervised GSLDA enjoys higher classification accuracy than the original GSLDA method in most cases. This reveals the potential advantages of using unlabeled data in classification problems.

The rest of the paper is organized as follows. In Section 2, we review some existing high-dimensional LDA methods, and introduce our motivations and formulations of our proposed methods. Section 3 focuses on graph estimation and the implementation of GSLDA. In particular, graph estimation methods are discussed for both GSLDA and its variant. In Section 4, theoretical justification is provided for our method. Sections 5 and 6 demonstrate the performance of GSLDA by simulated examples and real data studies respectively. We conclude this paper with some discussion in Section 7. Proofs of the theoretical results are provided in the Appendix.

2. Methodology

In this section, we first review LDA and construct a relationship between β and the graph structure of features in Section 2.1, based on which GSLDA is proposed. We also explain how to estimate the graph structure when it is not directly available and discuss the connections of our methods with several existing classification methods. In Section 2.2, we investigate the overall graph structure of the features and consider a variant of GSLDA which can efficiently utilize unlabeled data.

2.1. Motivation and formulation of GSLDA

We first discuss the problem setting and introduce some notations. Given the training dataset {(x₁, g₁), … , (x_n, g_n)} where for each i ∈ {1, … , n}, ${\mathbf{x}}_i\in {\mathbb{R}}^p$ is the feature vector and g_i ∈ {1, 2} is the class label. A linear classifier g_β0,β is defined as follows. For any $\mathbf{x}\in {\mathbb{R}}^p$, g_β0,β(x) = 1 if β₀ + x^⊺β > 0 and 2 otherwise. In particular, we consider the standard setting of the two-class LDA. That is, the binary label G takes 1 with probability π₁ and 2 with probability π₂ = 1 − π₁ and the feature vector X has a conditional Gaussian distribution, i.e., $\mathbf{X}\mid (G=k)\sim \mathcal{N}({\mu }^{(k)},\Sigma )$ for k ∈ {1,2}. Under this setting, the Bayes classifier $g_{{\beta }_0^{\ast },{\beta }^{\ast }}$ is specified by

$${\beta }^{\ast }={\Sigma }^{-1}\delta \text{and}{\beta }_0^{\ast }=-{({\mu }^{(1)}+{\mu }^{(2)})}^{⊺}{\beta }^{\ast }∕2+\ln ({\pi }_1∕{\pi }_2),$$ (1)

where δ = μ⁽¹⁾ − μ⁽²⁾. By replacing Σ and δ in (1) with their sample estimates, we have the LDA classifier with $\widehat{\beta }={\widehat{\Sigma }}^{-1}\widehat{\delta }$. Typically, we take $\widehat{\Sigma }=(n_1{\mathbf{S}}^{(1)}+n_2{\mathbf{S}}^{(2)})∕(n-2)$ and $\widehat{\delta }={\stackrel{‒}{\mathbf{x}}}^{(1)}-{\stackrel{‒}{\mathbf{x}}}^{(2)}$, where n_k, ${\stackrel{‒}{\mathbf{x}}}^{(k)}$ and S^(k) denote respectively the sample size, mean and covariance matrix for group k. Note that this formulation is valid only when n > p. In high-dimensional problems or when n ≤ p, there are various extensions of LDA that either use the formulation with shrunken estimates of Σ and δ or find a direct estimation of β; see [7, 12, 26, 35, 36]. Here we focus on the direct estimation approach.

Inspired by the regression formulation of LDA [17], Mai et al. [26] proposed the direct sparse discriminant analysis (DSDA) method to estimate β by solving the Lasso problem

$$\widehat{\beta }=\underset{\beta }{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta )}^2+\lambda {\Vert \beta \Vert }_1,$$

where y_i = n/n₁ if g_i = 1 and −n/n₂ if g_i = 2. It was shown that DSDA gives the same solution path as the method in [25, 41]. Compared to plug-in approaches, the DSDA estimates β directly in high dimensions and the assumptions are less stringent. However, it is unclear how we can utilize any structure information among features with the method or other high-dimensional LDA methods.

Assume that there is some structure among the features. In particular, we consider the case where the structure can be represented by a graph, denoted as $\mathcal{G}$. There are methods that effectively use the graph structure in regression problems. For example, Li and Li [19] used the penalty

$$\sum _{(j,\ell )\in \mathcal{G}}{\left({\beta }_j∕\sqrt{d_j}-{\beta }_{\ell }∕\sqrt{d_{\ell }}\right)}^2,$$

where d_j denotes the neighborhood size of feature j, to encourage close coefficients for connected features. Yang et al. [42] employed pairwise ℓ_∞ penalty for connected features, i.e., ${\Sigma }_{(j,\ell )\in \mathcal{G}}\max \{\mid {\beta }_j\mid ,\mid {\beta }_{\ell }\mid \}$, so their coefficients can be estimated zero or nonzero simultaneously. Recently, Yu and Liu [44] proposed a node-wise penalty

$$P_{\mathcal{G},\tau }(\beta )=\underset{\sum _{j=1}^p{\mathbf{v}}^{(j)}=\beta ,\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)}}{\min }\sum _{j=1}^p{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2$$

based on the decomposition of regression coefficient vector β = var(X)⁻¹cov(X, Y). In contrast to these developments for regression problems, little work has been done for classification problems.

We propose our method formulation based on a decomposition of β*, the discriminant vector of Bayes’ rule. Denote Ω = Σ⁻¹ the within-class precision matrix and δ = μ⁽¹⁾ − μ⁽²⁾ the group mean difference. We can decompose the discriminant vector β* in (1) as

$${\beta }^{\ast }=\Omega \delta =\sum _{j=1}^p{\delta }_j{\omega }_j,$$ (2)

where ω_j is the jth column of Ω. Recall that the support of Ω in fact forms a conditional correlation graph of features X. In this way, the optimal discriminant vector is linked to the Gaussian graph structure of the features. We use a toy example for demonstration. In a 3-dimensional LDA setting, assume ω₂₃ = ω₃₂ = 0, then β* = Ωδ = (δ₁ω₁₁ + δ₂ω₂₁ + δ₃ω₃₁, δ₁ω₁₂ + δ₂ω₂₂, δ₁ω₁₃ + δ₃ω₃₃)^⊺. See Figure A.1 in the Appendix for a graphical demonstration of the decomposition.

Denote the graph corresponding to Ω as $\mathcal{G}$, and the neighborhood of feature j ∈ {1, … , p} as ${\mathcal{N}}^{(j)}$. Replacing δ_jω_j by v^(j), then β* = v⁽¹⁾ + ⋯ + v^(p), where v^(j) is either 0 (when δ_j = 0) or with a support $\mathrm{supp}({\mathbf{v}}^{(j)})={\mathcal{N}}^{(j)}$ when δ_j ≠ 0. Instead of estimating β* itself, we can estimate v^(j)’s. Moreover, the decomposition (2) motivates a natural regularization on {v⁽¹⁾, … , v^(p)}, viz.

$$\sum _{j=1}^p{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2,$$

in which $\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)}=\mathrm{supp}({\omega }_j)$ and the τ_js are positive weights. Note that the group ℓ₂ penalty on v^(j) encourages a group sparsity effect, i.e., v^(j) is estimated as 0 or a sparse vector with support ${\mathcal{N}}^{(j)}$, which matches the decomposition (2). In the formulations, the τ_js are weights for the group regularization. In particular, the larger τ_j is, the more likely v^(j) is estimated as 0. Similar to the group Lasso [45], we can take

$${\tau }_j=\sqrt{\mid {\mathcal{N}}^{(j)}\mid }∕\mid {\widehat{\delta }}_j\mid ,$$

where ${\widehat{\delta }}_j={\stackrel{‒}{x}}_j^{(1)}-{\stackrel{‒}{x}}_j^{(2)}$.

We need to apply this regularization to a risk minimization framework of LDA to formulate our method. The regression formulation is an appropriate one due to its simplicity and convenience for theoretical analysis. By combining the formulation with the group regularization, we can estimate {v₁, … , v_p} by

$$({\widehat{\beta }}_0,{\widehat{\mathbf{v}}}_1,\dots ,{\widehat{\mathbf{v}}}_p)=\underset{{\beta }_0,{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{\left(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\sum _{j=1}^p{\mathbf{v}}^{(j)}\right)}^2+\lambda \sum _{j=1}^p{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2,$$ (3)

where $\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)}$ for all j ∈ {1, … , p}. Then β is estimated as ${\widehat{\mathbf{v}}}_1+\cdots +{\widehat{\mathbf{v}}}_p$. Furthermore, from the perspective of β estimation, the formulation is equivalent to

$$({\widehat{\beta }}_0,\widehat{\beta })=\underset{{\beta }_0,\beta }{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta )}^2+\lambda {\Vert \beta \Vert }_{\mathcal{G},\tau },$$ (4)

where

$${\Vert \beta \Vert }_{\mathcal{G},\tau }=\underset{\sum _{j=1}^p{\mathbf{v}}^{(j)}=\beta ,\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)}}{\min }\sum _{j=1}^p{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2$$ (5)

can be viewed as a structured regularization on β; see [31]. Since the regularization is specified by the graph $\mathcal{G}$, we call the method graph-based sparse LDA (GSLDA). Although we use the same squared loss function as in [26], our method focuses on utilizing the graph structure of features in β* estimation. We use the estimator $\widehat{\beta }$ from (4) for the discriminant vector β. With respect to β₀, the estimator from (4) may not be a good choice for the classification problem due to the regression formulation. To solve this problem, we adopt a similar approach by [26] and estimate it by

$${\widehat{\beta }}_0=-{({\stackrel{‒}{\mathbf{x}}}^{(1)}+{\stackrel{‒}{\mathbf{x}}}^{(2)})}^{⊺}\widehat{\beta }∕2+\ln (n_1∕n_2){\widehat{\beta }}^{⊺}\widehat{\Sigma }\widehat{\beta }∕({\widehat{\delta }}^{⊺}\widehat{\beta }).$$

While the GSLDA method is motivated from the discriminant vector decomposition (2), the decomposition of β* is not restricted to this form only. Therefore, the graph structure $\mathcal{G}$ used in our method is not restricted to the conditional independence graph. We will present another decomposition of β* in Section 2.2. In fact, any graph structure of features satisfying our assumptions in Section 4.1 can be possibly used. When the structure information is available, e.g., the gene pathways in genetic studies, we can construct a graph $\mathcal{G}$ using the gene pathway information. If the graph is not available, we can estimate it based on the training data. There are many methods for estimation of Gaussian graphical models, including the neighborhood selection [28], the graphical Lasso [14, 46], and the CLIME [7]. We will discuss them further in Section 3. In summary, GSLDA can be implemented in two steps: (i) graph construction and (ii) direct estimation of β via solving formulation (4).

The formulation (4) is closely related to the regression method proposed in [44]. However, both the problem setting and the motivation of our paper are different. In our problem, the response y is a binary variable and the features are from a mixed population. Although our formulation also uses the squared loss as in regression, the “error” $y_i-{\mathbf{x}}_i^{⊺}\beta$ has a very different interpretation and distribution. In particular, the distribution of $y_i-{\mathbf{x}}_i^{⊺}\beta$ depends on x_i. These issues bring unique challenges for the theoretical analysis of GSLDA. Although there are some classification methods that also utilize predictor structure, such as logistic regression with group Lasso penalty [27] and LDA with fused Lasso penalty [39], these methods do not utilize a general graph structure.

Depending on the feature structure, there are special cases in which GSLDA is closely connected with existing sparse LDA methods. For example, if we use an empty graph $\mathcal{G}$ with no edge at all, the regularization (5) simplifies to τ₁|β₁| + ⋯ + τ_p|β_p|. Then, formulation (4) becomes an adaptive Lasso type problem, viz.

$$\underset{{\beta }_0,\mathit{\beta }}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta )}^2+\lambda \sum _{j=1}^p{\tau }_j\mid {\beta }_j\mid .$$

When all penalty weights τ_j take value 1, the GSLDA is equivalent to the DSDA method in [26]. When the graph $\mathcal{G}$ consists of K disjoint complete subgraphs, denoted as ${\mathcal{G}}^{(1)},\dots ,{\mathcal{G}}^{(K)}$, then the regularization (5) simplifies to ${\tau }^{(1)}{\Vert {\beta }_{G^{(1)}}\Vert }_2+\cdots +{\tau }^{(K)}{\Vert {\beta }_{G^{(K)}}\Vert }_2$ where τ^(k) = min_j∈G^(k) τ_j and G^(k) is the index set of predictors involved in the subgraph ${\mathcal{G}}^{(k)}$. In this case, GSLDA becomes a variant of DSDA with the group Lasso penalty, i.e.,

$$\underset{{\beta }_0,\mathit{\beta }}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta )}^2+\lambda \sum _{k=1}^K{\tau }^{(k)}{\Vert {\beta }_{G^{(k)}}\Vert }_2.$$

For a general graph $\mathcal{G}$, our method is different from the existing ones.

Remark 1. While we are mainly concerned with binary classification in this paper, there are many scenarios with more than two classes [21, 47, 48]. Our GSLDA method can also be extended to the multi-class case. For example, consider a formulation of K-class sparse LDA proposed in [24], viz.

$$({\widehat{\theta }}_2,\dots ,{\widehat{\theta }}_K)=\underset{{\theta }_2,\dots ,{\theta }_K}{\mathrm{argmin}}\sum _{k=2}^K\{{\theta }_k^{⊺}\widehat{\Sigma }{\theta }_k∕2-{({\stackrel{‒}{\mathbf{x}}}^{(k)}-{\stackrel{‒}{\mathbf{x}}}^{(1)})}^{⊺}{\theta }_k\}+\lambda \sum _{j=1}^p{\Vert {\theta }_{\cdot j}\Vert }_2,$$

where θ₂, … , θ_K are discriminant vectors and ${\theta }_{\cdot j}={({\theta }_{2j},\dots ,{\theta }_{Kj})}^{⊺}$ for j ∈ {1, … , p}. The resulting discriminant rule is $\widehat{g}={\mathrm{argmax}}_k\{{\widehat{\theta }}_k^{⊺}(\mathbf{x}-{\stackrel{‒}{\mathbf{x}}}^{(k)}∕2)+\ln {\widehat{\pi }}_k\}$ where ${\widehat{\theta }}_1=0$ and ${\widehat{\pi }}_k$ is the proportion of class k in the sample. We can take advantage of a similar formulation with the graph-based regularization $\lambda {\Vert {\theta }_2,\dots ,{\theta }_K\Vert }_{\mathcal{G},\tau ,\text{grouped}}$, where

$${\Vert {\theta }_2,\dots ,{\theta }_K\Vert }_{\mathcal{G},\tau ,\text{grouped}}=\underset{\sum _{j=1}^p{\mathbf{v}}_k^{(j)}={\theta }_k,\mathrm{supp}({\mathbf{v}}_k^{(j)})\subseteq {\mathcal{N}}^{(j)}}{\mathrm{argmin}}\sum _{j=1}^p{\tau }_j{\Vert {({\mathbf{v}}_2^{(j)⊺},\dots ,{\mathbf{v}}_K^{(j)⊺})}^{⊺}\Vert }_2.$$

This formulation can be solved in a way similar to the binary GSLDA. Nevertheless, we do not pursue this direction in the paper so we can focus on core ideas of the GSLDA.

2.2. Semi-supervised GSLDA

With recent advances in graphical estimation [7, 28, 46], we can estimate $\mathcal{G}$ for the GSLDA based on the training data when the graph structure is unknown. However, as the dimension p increases, we expect the selection error to accumulate. When the dimension is much larger than the sample size, the graph estimate of GSLDA can be almost random. We use a toy example in Figure 1 to illustrate this phenomenon. In the setting of standard LDA, we set weights π₁ = π₂ = 0.5, and group means μ⁽¹⁾ = (0.5, … , 0.5, 0, … , 0)^⊺ and μ⁽²⁾ = (−0.5, … , −0.5, 0, … , 0)^⊺, which only differ in the first 10 features. To specify the graph structure, Ω is generated from an AR(5) model, i.e., Ω_jj = c, Ω_jℓ = −0.5 if 1 ≤ |j − ℓ| ≤ 5 and 0 otherwise, where c > 0 is a scalar such that the eigenvalues of Ω are between 0 and 1. We standardize Ω so that diag(Ω) = 1 and define in-class covariance matrix Σ = Ω⁻¹. Let the sample size n be 50 and p vary from 10 to 200. We estimate the graph by SR-SLasso [22] with extended BIC for tuning. For each setting, we repeat the procedure 100 times and evaluate the accuracy of graph estimation by false positive rate (FPR) and false negative rate (FNR). Figure 1 summarizes the performance of graph estimation for varying dimensions.

Figure 1:

Performance evaluation of graph estimation for varying dimensions. The black solid lines are for graph estimation based on a labeled dataset of size 50; the red dashed lines are for graph estimation based on an unlabeled dataset of size 1000; vertical segments indicate the standard deviations of FPR or FNR of 100 repetitions.

As shown in Figure 1, the graph estimation using only labeled data deteriorates quickly as the dimension increases. Note that the structured penalty in (5) encourages the coefficients of all features in a neighborhood to be nonzero together as long as some of them is useful for classification. Inaccurate graph estimation can reduce the accuracy and the interpretability of GSLDA.

Compared to labeled data, unlabeled data can be more accessible in many applications. For example, in the handwritten digit recognition problem discussed in Section 6.2, we can easily obtain a large number of images of different digits. However, it can be expensive to label these images by corresponding digits. As a result, many semi-supervised methods try to utilize the unlabeled data to improve the classification accuracy [5, 32]. In this paper, we focus on using unlabeled data for the graph construction when available. The following proposition studies the relationship between the within-class inverse covariance matrix and the overall one.

Proposition 1. Assume X comes from a mixture of two populations with a common covariance matrix Σ. The weight and the expectation of population k ∈ {1, 2} is π_k and μ^(k). Denote the mean difference of the two populations μ⁽¹⁾−μ⁽²⁾ as δ. We denote $\tilde{\Sigma }=\mathrm{var}(X)$ the overall covariance matrix of the population mixture and $\tilde{\Omega }={\tilde{\Sigma }}^{-1}$ the overall precision matrix. Then $\tilde{\Sigma }=\Sigma +{\pi }_1{\pi }_2\delta {\delta }^{⊺}$ and $\tilde{\Omega }=\Omega -c{\beta }^{\ast }{{\beta }^{\ast }}^{⊺}$, where β* = Ωδ and $c=1∕\{{({\pi }_1{\pi }_2)}^{-1}+{\delta }^{⊺}\Omega \delta \}$.

As a remark, we do not require any specific distribution for the populations in Proposition 1, while β* is the optimal discriminant vector if both classes are Gaussian populations. The overall precision matrix $\tilde{\Omega }$ is sparse if both Ω and β* are sparse, and its support forms the conditional correlation graph of the mixed population. Moreover, we have $\tilde{\Omega }\delta =(1-c{\beta }^{\ast ⊺}\delta ){\beta }^{\ast }\propto {\beta }^{\ast }$. In our problem, a decomposition of the optimal discriminant vector analogous to (2) using $\tilde{\Omega }$ can be written as

$${\beta }^{\ast }=\xi \sum _{j=1}^p{\delta }_j{\tilde{\mathbf{w}}}_j,$$

where ξ is a positive scalar and ${\tilde{\mathbf{w}}}_j$ is the jth column of $\tilde{\Omega }$. Therefore, the Bayes classifier can be connected to the graph structure of the mixed population through the new decomposition. Define the graph corresponding to the support of $\tilde{\Omega }$ as $\tilde{\mathcal{G}}$. Following the same rationale of GSLDA, we can formulate another estimator of β based on the overall graph structure, viz.

$$({\widehat{\beta }}_0,\widehat{\beta })=\underset{{\beta }_0,\beta }{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta {)}^2+\lambda {\Vert \beta \Vert }_{\tilde{\mathcal{G}},\tilde{\tau }},$$ (6)

where ${\Vert \beta \Vert }_{\tilde{\mathcal{G}},\tilde{\tau }}$ is defined in (5) and $\tilde{\tau }$ adapts to $\tilde{\mathcal{G}}$ as in (4). The only difference between (6) and (4) is which graph structure we use. When unlabeled data are abundant, the estimated graph $\tilde{\mathcal{G}}$ can be more accurate and thus the new formulation may provide better classification. We name the formulation (6) as semi-supervised GSLDA. Similar to the original GSLDA, the semi-supervised variant also has two step: (i) graph estimation based on all available data and (ii) direct estimation of β by solving formulation (6).

Both versions of GSLDA need to estimate a graph when no prior graph structure is given. But there is a major difference: unlike $\mathcal{G}$ in (4), the graph $\tilde{\mathcal{G}}$ in (6) is not for a Gaussian population but a Gaussian mixture. As we will see in Section 3, likelihood-based estimation such as graphical Lasso would be too complicated to implement. Instead, we can still use neighborhood selection. In fact, in regressing the feature X_j on the other features X_−j, the coefficient vector corresponds to the conditional correlations between X_j and other features regardless of the distribution of the features, as stated by the following lemma.

Lemma 1. For any random vector X = (X₁, … , X_p)^⊺ ~ F, assume we have finite second-order moments and denote $\tilde{\mu }=E_F(X)$, $\tilde{\Sigma }=E_F\{(X-\tilde{\mu }){(X-\tilde{\mu })}^{⊺}\}$ and $\tilde{\Omega }={\tilde{\Sigma }}^{-1}$. Then for any j, ℓ ∈ {1, … , p},

• (i)${\tilde{\omega }}_{j\ell }$, the (j, ℓ)th element of $\tilde{\Omega }$, is 0 if and only if X_j and X_ℓ are conditionally uncorrelated, i.e., cov(X_j, X_ℓ|X_−{j,ℓ}) = 0, where X_−{j,ℓ} denotes all features other than X_j and X_ℓ;
• (ii)${\tilde{\omega }}_{j\ell }$ is 0 if and only if ${\gamma }_{\ell }^{(j)}=0$, where ${\gamma }_{\ell }^{(j)}$ is the coefficient of X_ℓ in the regression of X_j on X_−j.

This lemma is closely related to the results in [28]. According to Lemma 1, the graph based on the inverse covariance matrix always corresponds to the conditional correlation structure. As long as variable selection consistency of the regression is guaranteed, neighborhood selection methods are valid for graph estimation. Figure 1 also shows the performance of graph estimation based on a large unlabeled dataset under the same settings. We can observe that the estimation still performs well when the dimension increases.

Remark 2. In practice, we generally use all available data, including both unlabeled and labeled data, in the first step of semi-supervised GSLDA. Note that even without unlabeled data, the method is still applicable. If we use neighborhood selection for graph estimation, then the error variance of the jth node-wise regression is $\mathrm{var}(X_j\mid X_{-j})=1∕{\tilde{\omega }}_{\mathit{jj}}=1∕({\omega }_{\mathit{jj}}-c{\beta }_j^{\ast 2})$ by Proposition 1. In contrast, when using the labels as in the original GSLDA, the error variance is $\mathrm{var}(X_j\mid X_{-j},G)=1∕{\omega }_{\mathit{jj}}<1∕({\omega }_{\mathit{jj}}-c{\beta }_j^{\ast 2})$. Therefore, the semi-supervised GSLDA has better graph estimation only when unlabeled data are abundant. When there are relatively little unlabeled data, the original GSLDA is more advantageous.

3. Graph estimation and method implementation

If the feature structure is given from prior knowledge, the graph can be directly constructed by assigning edges between related features. Otherwise, we need to estimate the graph based on training data. In particular, when unlabeled data are available, we can also use that to estimate the graph and implement semi-supervised GSLDA. In this section, we first discuss specific graph estimation methods for GSLDA. Then we introduce algorithms to solve formulation (4) as well as some strategies for efficient implementation.

3.1. Graph estimation

There have been extensive studies on graphical model estimation [7, 9, 14, 28, 38, 46]. As we discussed in Section 2.2, the graph estimation based on labeled and unlabeled data are different to some extent. Next we discuss them separately. Given labeled data, the likelihood conditional on the labels becomes

$$(2\pi {)}^{-pn∕2}\mid \Omega {\mid }^{n∕2}\exp \left\{-\frac{1}{2}\sum _{g_i=1}({\mathbf{x}}_i-{\mu }^{(1)}{)}^{⊺}\Omega ({\mathbf{x}}_i-{\mu }^{(1)})-\frac{1}{2}\sum _{g_i=2}({\mathbf{x}}_i-{\mu }^{(2)}{)}^{⊺}\Omega ({\mathbf{x}}_i-{\mu }^{(2)})\right\}.$$

Similar to the graphical Lasso, we can estimate Ω by minimizing ℓ₁ penalized log-likelihood, i.e.,

$$\underset{{\mu }^{(1)},{\mu }^{(2)},\Omega \in {\mathbb{S}}_{++}}{\mathrm{argmin}}\frac{n}{2}\ln \mid \Omega \mid -\frac{1}{2}\sum _{g_i=1}{({\mathbf{x}}_i-{\mu }^{(1)})}^{⊺}\Omega ({\mathbf{x}}_i-{\mu }^{(1)})-\frac{1}{2}\sum _{g_i=2}{({\mathbf{x}}_i-{\mu }^{(2)})}^{⊺}\Omega ({\mathbf{x}}_i-{\mu }^{(2)})+\lambda {\Vert \Omega \Vert }_1,$$

where ${\mathbb{S}}_{++}$ denotes the set of p-dimensional positive definite matrices and ||Ω||₁ = Σ_j≠ℓ|ω_jℓ|. It results in ${\widehat{\mu }}^{(1)}={\stackrel{‒}{\mathbf{x}}}^{(1)},{\widehat{\mu }}^{(2)}={\stackrel{‒}{\mathbf{x}}}^{(2)}$, and

$$\widehat{\Omega }=\underset{\Omega \in {\mathbb{S}}_{++}}{\mathrm{argmin}}\frac{n}{2}\ln \mid \Omega \mid -\frac{1}{2}\sum _{i=1}^n{({\mathbf{x}}_i-{\stackrel{‒}{\mathbf{x}}}^{(g_i)})}^{⊺}\Omega ({\mathbf{x}}_i-{\stackrel{‒}{\mathbf{x}}}^{(g_i)})+\lambda {\Vert \Omega \Vert }_1.$$ (7)

This is equivalent to the graphical Lasso for the centered data ${\mathbf{x}}_1-{\stackrel{‒}{\mathbf{x}}}^{(g_1)},\dots ,{\mathbf{x}}_n-{\stackrel{‒}{\mathbf{x}}}^{(g_n)}$.

Instead of solving (7), we can also estimate the graph by neighborhood selection as proposed by [28]. This method solves p node-wise regularized regressions, viz.

$$\underset{{\gamma }_0^{(1j)},{\gamma }_0^{(2j)},{\gamma }^{(j)}}{\mathrm{argmin}}\frac{1}{2n}{\Vert {\mathbf{X}}_j^{(1)}-{\gamma }_0^{(1j)}-{\mathbf{X}}_{-j}^{(1)}{\gamma }^{(j)}\Vert }_2^2+\frac{1}{2n}{\Vert {\mathbf{X}}_j^{(2)}-{\gamma }_0^{(2j)}-{\mathbf{X}}_{-j}^{(2)}{\gamma }^{(j)}\Vert }_2^2+\lambda {\Vert {\gamma }^{(j)}\Vert }_1,$$

where ${\mathbf{X}}_j^{(k)}$ denotes the jth feature of sample from group k and ${\mathbf{X}}_{-j}^{(k)}$ represents the other features. One can verify that

$${\widehat{\gamma }}^{(j)}=\underset{{\gamma }^{(j)}}{\mathrm{argmin}}\frac{1}{2n}{\Vert {\stackrel{.}{\mathbf{X}}}_j-{\gamma }_0-{\stackrel{.}{\mathbf{X}}}_{-j}{\gamma }^{(j)}\Vert }_2^2+\lambda {\Vert {\gamma }^{(j)}\Vert }_1,$$ (8)

where $\stackrel{.}{\mathbf{X}}$ denotes the data centered by subtracting corresponding group means. We can also use sequential Lasso [23] for computational efficiency. The graph $\mathcal{G}$ is constructed by connecting nodes j and ℓ if ${\widehat{\gamma }}_{\ell }^{(j)}\ne 0$ and/or ${\widehat{\gamma }}_j^{(\ell )}\ne 0$.

Both approaches for estimating $\mathcal{G}$ have been justified theoretically [28, 46]. In this paper, we recommend to use neighborhood selection approaches for GSLDA. The main reason is that the former approaches, such as graphical Lasso, usually run through many iterations and can be slow for high-dimensional data (p > 1000). In contrast, neighborhood selection approaches only require p penalized regressions. Moreover, our direct interest is not Ω but the graph $\mathcal{G}$ on which neighborhood selection focuses. We use the extended BIC (EBIC) [8] to select λ in (8). As suggested in [8], we choose 1 − 1/(2 log_n p) as the EBIC tuning parameter.

When we have an extra unlabeled dataset, denoted as x_n+1, … , x_n+m, the likelihood becomes complicated because of the Gaussian mixture distribution of the unlabeled data. Thus it is difficult to estimate the parameters via likelihood. Moreover, the graph we need is directly related to $\tilde{\Omega }=\mathrm{var}{(X)}^{-1}$ rather than Ω. Thus, a penalized likelihood approach is not suitable. Nevertheless, the neighborhood selection approaches are still valid by Lemma 1, because we are concerned with conditional correlation. In particular, we estimate the neighborhoods by

$${\widehat{\tilde{\gamma }}}^{(j)}=\underset{{\tilde{\gamma }}^{(j)}}{\mathrm{argmin}}\frac{1}{2(n+m)}{\Vert {\tilde{\mathbf{X}}}_j-{\tilde{\gamma }}_0-{\tilde{\mathbf{X}}}_{-j}{\tilde{\gamma }}^{(j)}\Vert }_2^2+\lambda {\Vert {\tilde{\gamma }}^{(j)}\Vert }_1,$$

where $\tilde{\mathbf{X}}={({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n,{\mathbf{x}}_{n+1},\dots ,{\mathbf{x}}_{n+m})}^{⊺}$ denotes the combined feature matrix. Similarly, we use EBIC to select the tuning parameter λ.

3.2. Parameter estimation and tuning parameter selection

Given the graph $\mathcal{G}$, formulation (4) is a latent group Lasso problem [31]. It can be transformed to an ordinary group Lasso problem as stated in Problem (3). There are many efficient algorithms to solve group Lasso problems, for example, groupwise majorization descent [43]. For very high-dimensional data, we use an iterative proximal algorithm as in [44]. For implementation, we use cross validation for tuning parameter selection.

3.3. Pre-screening

Suppose that there are some entries of δ being zero. Then β* can be a linear combination of only a few column vectors,

$${\beta }^{\ast }=\sum _{j\in J}{\delta }_j{\mathbf{w}}_j,$$

where J = {j : δ_j ≠ 0}. Using two-sample t tests for screening, we can specify J′ ⊂ {1, … , p}, which is a superset of J with a large probability. In particular, we have the following lemma.

Lemma 2. Define the t-statistic $T_j={\widehat{\delta }}_j∕{\{s_j^{(1)2}∕n_1+s_j^{(2)2}∕n_2\}}^{1∕2}$, where $s_j^{(k)2}$ is the sample variance of feature j ∈ {1, … , p} in group k ∈ {1,2}. Assume ln p = o(n^γ), ln |J| = o(n^1/2−γ B_n), and ${\min }_{j\in J}\mid {\delta }_j\mid \sqrt{2{\Sigma }_{\mathit{jj}}}=B_n∕n^{\gamma }$ for some γ ∈ (0,1/3) and B_n → ∞. Then there exists C > 0 such that

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left\{\underset{j\in J}{\min }\mid T_j\mid \ge {\mathit{Cn}}^{\gamma ∕2},\underset{j\notin J}{\max }\mid T_j\mid <{\mathit{Cn}}^{\gamma ∕2}\right\}=1.$$

The result in Lemma 2 was previously obtained by Fan and Fan [11] and the corresponding proof is omitted. Lemma 2 guarantees the accuracy of our pre-screening procedure.

After feature screening, the proposed regularization can be simplified as follows:

$${\Vert \beta \Vert }_{{\mathcal{G}}_{j^{\prime }},\tau }=\underset{{\Sigma }_{j\in J^{\prime }}{\mathbf{v}}^{(j)}=\beta ,\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)}}{\min }\sum _{j\in J^{\prime }}{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2.$$ (9)

Compared with the original regularization (5), the new one in (9) is often simpler and enjoys computational advantages. Moreover, the new regularization (9) only requires part of the graph, i.e., the part corresponding to the support of {ω_j : j ∈ J′}. Graph estimation methods based on neighborhood selection fit into this idea naturally. When δ is approximately sparse and |J′| ⪡ p, the computational cost can be reduced substantially. Unlike the feature screening in [11], features outside J′ are not necessarily excluded. Instead, they can be introduced into the model via connection with other features in J′.

4. Theoretical properties

In this section, we study the theoretical properties of GSLDA. In particular, the original GSLDA in (4) with a known graph $\mathcal{G}$ is considered. Since the semi-supervised GSLDA only differs from GSLDA in the graph used, we do not consider it separately. In Section 4.1, we show the selection consistency of GSLDA. In Section 4.2, we study the misclassification rate of the GSLDA and compare it with the Bayes error.

Before diving into the theoretical analysis, we first introduce some notations for our setting. We define, for an n-dimensional vector a, ||a||_∞ = max(|a₁|, … , |a_n|); for an n × m matrix A, ||A||_∞ = max_i{|A_i1| + ⋯ + |A_im|} and |||A|||_∞ = max_i,j |A_ij|. We consider the problem setting of standard LDA, in which both within-class populations are Gaussian, i.e., $\mathcal{N}({\mu }^{(1)},\Sigma )$ and $\mathcal{N}({\mu }^{(2)},\Sigma )$. The discriminant vector of the Bayes rule, denoted as β*, is given in (1). Denote $A=\{j:{\beta }_j^{\ast }\ne 0\}$ the active set, and s = |A|. Define ${\beta }^{†}=\tilde{\Omega }\delta$, then β^† is proportional to β* (Proposition 1) and thus defines an equivalent classifier.

4.1. Selection consistency

Assume that the feature vectors are centralized, thus $\widehat{\beta }={\mathrm{argmin}}_{\beta }{\Vert \mathrm{y}-\mathbf{X}\beta \Vert }_2^2∕n+\lambda {\Vert \beta \Vert }_{\mathcal{G},\tau }$. Denote $\tilde{\mathbf{S}}={\mathbf{X}}^{⊺}\mathbf{X}∕n$ and $\kappa ={\Vert {\tilde{\Sigma }}_{A^{\mathrm{C}}A}{\tilde{\Sigma }}_{\mathit{AA}}^{-1}\Vert }_{\infty }$. Define

$${\tilde{\tau }}_j=\underset{\ell }{\min }\{{\tau }_{\ell }:j\in {\mathcal{N}}^{(\ell )}\},{\tau }^{\ast }=\underset{j\in A}{\max }{\tilde{\tau }}_j,{\tau }_{\ast }=\underset{j\in A^{\mathrm{C}}}{\min }{\tau }_j{\mid {\mathcal{N}}^{(j)}\mid }^{-1∕2}.$$

We present several assumptions to be used as follows.

• (A1)p = O{exp(n^γ)}, s = o(n^a), for some γ ∈ (0,1), a ∈ (0, (1 − γ)/2).
• (A2)For every j ∈ {1, … , p}, either ${\mathcal{N}}^{(j)}\subseteq A$ or ${\mathcal{N}}^{(j)}\subseteq A^{\mathrm{C}}$.
• (A3)${\Vert {\tilde{\Sigma }}_{\mathit{AA}}^{-1}\Vert }_{\infty }$ is bounded by φ < ∞.
• (A4)${\Vert {\tilde{\Sigma }}_{A^{\mathrm{C}}A}{\tilde{\Sigma }}_{\mathit{AA}}^{-1}\Vert }_{\infty }<{\tau }_{\ast }∕{\tau }^{\ast }$.
• (A5)$b={\min }_{j\in A}\mid {\beta }_j^{†}\mid ⪢\sqrt{\ln p∕n}$.

Here (A1) specifies the order of feature dimension as well as the number of discriminating features. By Assumption (A2), a discriminative feature can only be connected with other discriminative features. This is a reasonable condition in reality since a feature is often relevant for classification if it is related to another useful feature. Condition (A3) ensures that there is no extreme collinearity among discriminative features. Assumption (A4) is an irrepresentability condition that is often employed in showing the selection consistency of regularized estimators [28, 50].

It may not be immediately clear why we impose the irrepresentability condition (A4) on $\tilde{\Omega }$ rather than Ω. Note that the more similarity between predictive and non-predictive features, the more difficult it is to achieve selection consistency. While Ω encodes the within-class feature dependence, the relationship among features in the whole dataset is determined by the overall covariance. Thus we impose the condition on $\tilde{\Omega }$. The main theoretical result on the selection consistency of the GSLDA is given in the following theorem.

Theorem 1 (Selection consistency). Under conditions (A1)-(A5), let $\sqrt{\ln p∕n}\le \lambda {\tau }^{\ast }\le O(b)$ and n be sufficiently large, then the GSLDA recovers the active set A and ${\Vert {\widehat{\beta }}_A-{\beta }_A^{†}\Vert }_{\infty }=O(\sqrt{\ln p∕n})$ with probability at least 1 − O(p^−C₁) for some C₁ > 0.

When we use an empty graph $\mathcal{G}$ and set τ_j = 1 for all j, our GSLDA is equivalent to the DSDA method. In this special case, τ* = τ_* = 1, and the selection consistency conditions are similar to those for DSDA [26].

4.2. Convergence rate

With respect to a classifier, the error rate is one of the most important performance measures. In this section, we investigate the misclassification rate of GSLDA. We first present some basic results on the classification problem. For a linear classifier ĝ, denote its classification error under our settings as $Q_{{\beta }_0,\beta }=\mathrm{Pr}\{g_{{\beta }_0,\beta }(X)\ne G\}$. Then we have the following results from [6].

Lemma 3 (Classification error rate in LDA setting). Under our setting,

$$Q_{{\beta }_0,\beta }=\frac{1}{2}\Phi \left(\frac{-{\beta }_0-{\beta }^{⊺}{\mu }^{(1)}}{\sqrt{{\beta }^{⊺}\Sigma \beta }}\right)+\frac{1}{2}\Phi \left(\frac{{\beta }_0+{\beta }^{⊺}{\mu }^{(2)}}{\sqrt{{\beta }^{⊺}\Sigma \beta }}\right),$$

where Φ denotes the cumulative distribution function of $\mathcal{N}(0,1)$. The misclassification rate of the Bayes classifier $g_{{\beta }_0^{\ast },{\beta }^{\ast }}$ is $Q_{{\beta }_0^{\ast },{\beta }^{\ast }}=\Phi (-{\Delta }^{1∕2}∕2)$, where Δ = δ^⊺Ωδ.

Since Q is a continuous function of β₀ and β, the misclassification rate of the GSLDA classifier is asymptotically the same as the Bayes error rate, i.e., $Q_{{\widehat{\beta }}_0,\widehat{\beta }}\stackrel{p}{\to }{Q}_{{\beta }_0^{\ast },{\beta }^{\ast }}$, as long as $\widehat{\beta }\stackrel{p}{\to }{\beta }^{\ast }$. A more interesting problem is the order of the misclassification rate of the GSLDA when $Q_{{\beta }_0^{\ast },{\beta }^{\ast }}\to 0$. To investigate this, we first introduce a new condition, under which we can construct an ℓ₂ error bound for the GSLDA estimator.

• (A6)Denote $\mathcal{C}(A)=\{\Delta \in {\mathbb{R}}^p:{\Vert {\Delta }_{A^{\mathrm{C}}}\Vert }_{\mathcal{G},\tau }\le 3{\Vert {\Delta }_A\Vert }_{G,\tau }\}$, where ${\Delta }_A={({\Delta }_{j}1(j\in A))}_{p\times 1}$ and ${\Delta }_{A^{\mathrm{C}}}={({\Delta }_{j}1(j\notin A))}_{p\times 1}$. For all $\Delta \in \mathcal{C}(A)$, ${\Delta }^{⊺}\tilde{\Sigma }\Delta ∕{\Delta }^{⊺}\Delta \ge \sigma >0$.

This is actually a restricted eigenvalue condition, which is often used in showing the error bound for regularized estimators [30]. Compared to the irrepresentability condition (A4), this is much less stringent. With the new condition, we have the following ℓ₂ error bound for the GSLDA estimator.

Theorem 2 (ℓ₂-error bound). Under conditions (A1)–(A2) and (A6), let $\lambda \ge 4C_2(1+{\Vert {\beta }^{†}\Vert }_1)\sqrt{\ln p∕n}$ for some C₂ > 0 and n be sufficiently large, then ${\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2^2\le 9{\lambda }^{2}s{\tau }^{\ast 2}∕{\sigma }^2$ with probability at least 1 − sp^−C₃ for some C₃ > 0.

Based on Theorem 2 above, we can establish the asymptotic error rate of the GSLDA classifier as follows.

Theorem 3 (Convergence rate). Under conditions (A1)–(A2) and (A6), as n, p → ∞, if Δ → ∞, we have

$$Q_{{\beta }_0^{\ast },{\beta }^{\ast }}\to 0\text{and}{Q}_{{\widehat{\beta }}_0,\widehat{\beta }}∕Q_{{\beta }_0^{\ast },{\beta }^{\ast }}\stackrel{p}{\to }1,$$

given $\lambda {\tau }^{\ast }=o[\min \{{\lambda }_{\max }{(\Sigma )}^{-1}{\Delta }^{-2}{s}^{-1∕2}{\Vert {\beta }^{†}\Vert }_2^{-1},{\Delta }^{-1}{s}^{-1∕2}{\Vert \delta \Vert }_2^{-1}\}]$ and ${\Vert {\beta }^{†}\Vert }_1=o(n^{1-\gamma }{\Delta }^{-1})$, where Δ is defined as in Lemma 3 and λ_max(Σ) denotes the largest eigenvalue of Σ.

That is, under mild conditions, the misclassification rate of the GSLDA classifier is of the same order as the Bayes error rate in this case.

5. Simulation study

To demonstrate the performance of the GSLDA methods, we compare them with several existing high-dimensional LDA extensions and other classification methods. The methods in comparison include the naive Bayes rule (NB), nearest shrunken centroids (NSC), sparse LDA (SLDA) [35], ℓ₁ penalized Logistic regression (PLR), penalized Fisher’s discriminant analysis (PLDA) [39], direct sparse discriminant analysis (DSDA) [26], linear programming discriminant (LPD) [6], and the ROAD [12]. In particular, the methods NSC, PLR, PLDA and DSDA are implemented with R packages pamr, glmnet, penalizedLDA and dsda, respectively. We implement the LPD method via the parametric simplex algorithm [37] as suggested in [34].

Besides the above supervised methods, there are many semi-supervised clustering (or classification) methods; see, e.g., [20, 32, 52]. We have implemented the semi-supervised spectral clustering (SSSC) method proposed in [20]. Both the original and the semi-supervised GSLDA are implemented, and the latter is denoted as GSLDA-S. We also include the GSLDA methods with the true graph, denoted as GSLDA-O (with $\mathcal{G}$) and GSLDA-SO (with $\tilde{\mathcal{G}}$), in the comparison. To make a fair comparison, pre-screening is not employed in the numerical studies. The Bayes rule, denoted as Oracle, is used as a benchmark.

In the simulation, we fix the dimension p = 200 and the sample size n = 200. The labels g₁, … , g_n are generated with π₁ = π₂ = 1/2 and the features are sampled from $\mathcal{N}({\mu }^{(g_i)},{\Omega }^{-1})$ based on the labels. Moreover, we generate an independent dataset of sample size 2000 and remove the labels, for the semi-supervised methods. All tuning parameters are selected by 10-fold cross validation. We consider four different feature structures as follows.

Example 1. Blockwise sparse model. In this example, Σ^B is a 5 × 5 matrix with 1 for the diagonal and 0.7 for off-diagonal elements. We use 20 such blocks for the diagonal of the covariance matrix Σ and 0 for the rest, and let Ω = Σ. The group means are generated such that ${\mu }_j^{(1)}=0.5$ for j ∈ {5, 10, … , 25} and ${\mu }_j^{(1)}=0$ otherwise; and μ⁽²⁾ = −μ⁽¹⁾.

Example 2. AR(3) model. The precision matrix Ω is generated such that ω_jj = 1, and ω_jℓ = −2/3 if 1 ≤ |j − ℓ| ≤ 3 and 0 otherwise. The group means are generated such that ${\mu }_j^{(1)}=0.75$ for j ∈ {5, 10, … , 25} and ${\mu }_j^{(1)}=0$ otherwise; and μ⁽²⁾ = −μ⁽¹⁾.

Example 3. Random sparse model. The graph $\mathcal{G}$ is generated in such a way that any two nodes are connected with probability 0.05. Based on $\mathcal{G}$, we generate the precision matrix Ω by setting ω_jℓ = −0.5 for all connected j and ℓ in the graph and 0 otherwise. We add c I_p, where c > 0 and I_p is an identity matrix, to Ω such that the eigenvalues are between 0 and 1. We standardize Ω so that its diagonal elements are all 1. The group means are generated in such a way that ${\mu }_j^{(1)}=0.75$ for all j ∈ S and 0 otherwise; and μ⁽²⁾ = −μ⁽¹⁾.

Example 4. Scale-free random graph. The graph is generated in a way similar to the Barabasi-Albert (BA) model. Starting from an identity matrix $\mathbf{L}\in {\mathbb{R}}^{p\times p}$, at step i we randomly assign −0.5 to min{⌊0.05 p⌋, i − 1} entries in row i with probability Pr(i, j) ∝ #{L_ℓj ≠ 0 : 1 ≤ ℓ ≤ p}, j < i. Repeat the procedure until i = p. Then we get a lower triangular matrix. We construct Ω = L^⊺L and standardize it such that the eigenvalues are between 0 and 1. Denote the 6th to 10th most connected nodes as J. The group means are generated such that ${\mu }_j^{(1)}=0.75$ for all j ∈ J and 0 otherwise; and μ⁽²⁾ = −μ⁽¹⁾.

All four graph structures are displayed in Figure 2. The first two examples are fixed while the last two produce random graphs. Compared with the random sparse model, the scale-free random graphs are featured with hubs. For each graph structure, we repeat the simulation for 100 times and evaluate the performance, both prediction and selection accuracy, of all classification methods. Table B.1 in the Appendix displays the graph estimation accuracy for all examples.

Figure 2:

The graph structures used in the simulation study. From left to right: the blockwise sparse model, the AR(3) model, the random sparse model, the scale-free model. The last two plots use one realization for demonstration, and the graphs may vary among different realizations.

Tables 1–4 give a summary of the performance comparison of all methods in Examples 1 and 4. In particular, misclassification rates in percentage (Error), false positives (FP) and false negatives (FN) of β estimation are computed. The misclassification rate is evaluated based on an independent test dataset of size 20,000. All metrics are averaged over 100 simulations and the numbers within parentheses are the standard errors. Both the NB and the SSSC are not considered in the comparison of variable selection, since these methods do not perform variable selection.

Table 1:

Performance comparisons of different classification methods for Example 1.

	Error	FP	FN	Size
NB	27.01 (0.18)	—	—	—
NSC	14.17 (0.11)	0.71 (0.54)	20.27 (0.13)	5.44 (0.62)
SLDA	10.28 (0.16)	5.71 (1.29)	12.53 (0.31)	18.18 (1.61)
PLR	7.17 (0.13)	14.73 (0.56)	8.1 (0.24)	31.63 (0.69)
DSDA	6.76 (0.13)	23.26 (1.53)	6.79 (0.27)	41.47 (1.71)
LPD	7.80 (0.38)	37.20 (1.97)	5.73 (0.29)	56.47 (2.17)
ROAD	6.54 (0.12)	23.45 (1.24)	6.01 (0.24)	42.44 (1.37)
PLDA	14.16 (0.10)	3.62 (1.16)	19.53 (0.16)	9.09 (1.29)
SSSC	8.11 (0.10)	—	—	—
GSLDA	5.57 (0.07)	20.48 (2.17)	7.31 (0.25)	38.17 (2.33)
GSLDA-S	4.53 (0.06)	18.79 (2.26)	0.74 (0.11)	43.05 (2.29)
GSLDA-O	4.86 (0.08)	18.55 (2.63)	0 (0)	43.55 (2.63)
GSLDA-SO	4.52 (0.07)	16.43 (1.93)	0 (0)	41.43 (1.93)
Oracle	3.27 (0.01)	0 (0)	0 (0)	25 (0)

Table 4:

Performance comparisons of different classification methods for Example 4.

	Error	FP	FN	Size
NB	32.84 (0.28)	—	—	—
NSC	22.78 (0.12)	8.62 (1.76)	48.87 (0.76)	18.75 (2.49)
SLDA	17.53 (0.27)	19.83 (1.29)	38.23 (0.67)	40.60 (1.98)
PLR	14.60 (0.21)	16.51 (0.66)	35.5 (0.5)	40.01 (1.06)
DSDA	13.48 (0.17)	33.71 (1.9)	28.18 (0.65)	64.53 (2.47)
LPD	16.87 (0.36)	46.86 (1.66)	29.17 (0.71)	76.69 (2.31)
ROAD	13.95 (0.19)	36.9 (2.01)	27.71 (0.77)	68.19 (2.69)
PLDA	22.64 (0.12)	6.6 (1.06)	51.58 (0.45)	14.02 (1.48)
SSSC	12.08 (0.21)	—	—	—
GSLDA	10.46 (0.11)	21.53 (1.7)	15.69 (0.55)	64.84 (2.14)
GSLDA-S	9.15 (0.12)	12.87 (1.29)	5.03 (0.54)	66.84 (1.57)
GSLDA-O	10.39 (0.18)	28.29 (1.73)	19.44 (0.8)	67.85 (2.43)
GSLDA-SO	9.36 (0.17)	19.87 (1.69)	5.47 (0.71)	73.05 (2.31)
Oracle	4.62 (0.02)	0 (0)	0 (0)	59 (0)

From Tables 1–4, we observe that the two plug-in extensions of LDA, namely the naive Bayes and the NSC, perform worse than ℓ₁ penalized logistic regression and other direct LDA methods under these settings. This is expected because there is substantial correlation among the features while both the plug-in extensions of LDA use diagonal estimates of Σ. In contrast, the performance of the direct LDA methods varies across the settings. For example, the DSDA has lower misclassification rates than the ROAD in most cases, while ROAD has better classification accuracy in Example 1. Utilizing the graph structures, high-dimensional LDA is further improved in GSLDA. As we can see from the results, GSLDA methods have the best performance among all methods in these four settings. In particular, the GSLDA method has lower misclassification rates than all other methods except its semi-supervised variant. Since the DSDA is the special case of the GSLDA with an empty graph, it is a good benchmark to quantify the benefit of using graph structures. In most cases, the GSLDA provides better model selection than the DSDA. Therefore, utilizing the graph structure does help us to improve the LDA classifier in high dimensions.

With respect to the semi-supervised GSLDA, due to the large amount of unlabeled data, it often has better graph estimation and yields more accurate classifiers. In fact, the semi-supervised GSLDA has the lowest misclassification rates among all methods in all cases. Furthermore, the semi-supervised GSLDA has superior model selection over the original GSLDA in most cases. This demonstrates the advantages of using unlabeled data.

We notice that models estimated by the semi-supervised GSLDA often have larger sizes, sometimes more false positives in coefficient vectors, than the original GSLDA classifiers. This is probably because the graph used in the semi-supervised GSLDA often has more edges. There are two possible reasons: (i) the true graph $\tilde{\mathcal{G}}$ corresponding to $\tilde{\Omega }$ has more edges than $\mathcal{G}$, and (ii) graph estimation based on unlabeled data uses a much larger training dataset which often leads to denser graphs estimate. While a denser graph estimate may recover more connections among the features, it can also result in more false edges. This effect is enhanced by the difficulty of graph estimation with unlabeled data. As a consequence, the semi-supervised GSLDA may suffer from more false positives, as shown in Examples 2 and 3. To resolve this issue, we may consider to use more conservative graph estimation for the semi-supervised GSLDA.

6. Real data analysis

In this section, we implement our methods and several other existing classifiers on two real datasets. The first dataset is a genetic dataset with very high dimensions, and the second one consists of images of handwritten digits. We estimate the graphs from labeled training data and unlabeled data. We find that GSLDA methods have a good performance in both datasets and utilizing the feature structure is beneficial.

6.1. Arcene cancer data

Nowadays, genetic diagnosis is an important tool in the clinical study and medical practice. By using the genetic information, we can estimate the potential risk of cancer for healthy people or determine cancer subtypes for patients. The Arcene dataset is a gene dataset of 88 cancer patients and 112 healthy individuals. The dataset contains 10,000 features and was originally used in the NIPS 2003 feature selection challenge (https://archive.ics.uci.edu/ml/data\-sets/Arcene). Out of the 10,000 features, 7000 are real genes while the other 3000 are noise features that have no predictive power and make the prediction harder. Besides the labeled data, there is an unlabeled dataset of 700 individuals, which is used to construct a graph for GSLDA-S. As in the previous simulation studies, we apply the GSLDA and other methods on the dataset.

The labeled data are randomly split into a training set and a test set, of sizes 150 and 50, respectively. All methods except the naive Bayes are tuned by 10-fold cross validation. The experiment is repeated 100 times and the results are summarized in Table 5.

Table 5:

Comparison of GSLDA and other methods on the Arcene dataset.

	Error	Size
NB	35.50 (0.62)	—
NSC	36.05 (0.61)	9934.46 (9.06)
SLDA	34.64 (0.73)	297 (4.17)
PLR	28.36 (0.65)	16.57 (0.90)
DSDA	28.29 (0.72)	30.96 (2.69)
LPD	31.59 (1.33)	10.95 (3.58)
ROAD	29.29 (0.64)	31.86 (3.43)
PLDA	34.36 (0.61)	9.39 (1.63)
SSSC	27.93 (0.83)	—
GSLDA	22.57 (0.70)	229.36 (6.39)
GSLDA-S	24.50 (0.68)	319.57 (8.37)

From Table 5, we can see that both GSLDA and semi-supervised GSLDA outperform other methods in prediction. Although semi-supervised GSLDA uses more data for graph estimation, its performance is inferior to GSLDA for this application, possibly due to the difficulty of graph estimation based on unlabeled data. In addition, the size of the unlabeled dataset is not substantially larger than that of the labeled dataset. Compared with PLR, DSDA and ROAD, our methods have significantly larger model sizes. This may indicate that many genes are related to each other. It is likely that those genes contribute to cancer together, and including all of them in modeling can potentially make the classifier more robust. This characteristic may also contribute to the good performance of the proposed two GSLDA methods.

6.2. Semeion handwritten digits dataset

The Semeion dataset (https://archive.ics.uci.edu/ml/datasets/Semeion+Handwritten+Digit) consists of 1593 images of handwritten digits. Each digit is in the form of a 16 × 16 grayscale image and saved as a vector of 256 features. We take a subset of the dataset that only contains digits 1 and 7, which are generally difficult to distinguish. We randomly choose 40 images for training, and 80 for graph estimation of the semi-supervised GSLDA after removing labels. The remaining 200 images are used for testing. Other settings are the same as the cancer example. Table 6 gives a summary of the results.

Table 6:

Comparison of GSLDA and other methods on the Semeion dataset.

	Error	Size
NB	13.81 (0.34)	—
NSC	15.21 (0.44)	84.74 (11.31)
SLDA	14.43 (0.67)	20.23 (2.80)
PLR	18.69 (0.88)	9.46 (0.40)
DSDA	13.76 (0.66)	16.76 (1.01)
LPD	17.15 (0.86)	15.32 (0.91)
ROAD	19.73 (0.98)	15.38 (1.25)
SSSC	13.97 (0.75)	—
GSLDA	12.65 (0.61)	28.46 (1.45)
GSLDA-S	11.23 (0.56)	33.28 (1.32)

As shown in Table 6, the semi-supervised GSLDA has excellent performance for this problem. It has the lowest misclassification rate among all methods in comparison. The original GSLDA method also has good classification accuracy for this problem. Moreover, we can see that both GSLDA methods have larger model sizes than other direct LDA methods, as in the previous analysis in Section 6.1.

7. Discussion

With many extensions in the literature, LDA can be readily applied to high-dimensional classification problems. In particular, the direct approaches of high-dimensional LDA are attractive due to their simplicity and good performance. Under the standard setting of LDA problems, we explore the relationship between the graph structure of features and the optimal discriminant vector β*. Our study shows that, by taking advantage of such structure, we can get better LDA classifiers in high dimensions. Based on this idea, we propose the GSLDA method. After investigating the overall graph structure of the Gaussian mixture population for unlabeled data, we further propose the semi-supervised GSLDA that can utilize unlabeled data. Both GSLDA methods have been evaluated on simulated and real data, which demonstrate the advantages of utilizing the graph structures. Moreover, we conclude that the performance of semi-supervised GSLDA depends on both the size of the unlabeled dataset and the graph complexity. When the graph structure is very complex, it is better to consider a conservative graph estimate for GSLDA. Finally, our focus in this paper is on binary problems. It will be useful to extend the methods for multicategory problems.

Table 2:

Performance comparisons of different classification methods for Example 2.

	Error	FP	FN	Size
NB	36.59 (0.43)	—	—	—
NSC	17.46 (0.14)	42.75 (2.16)	25.96 (0.45)	55.79 (2.48)
SLDA	14.39 (0.12)	19.28 (1.72)	17.59 (0.43)	40.69 (2.17)
PLR	7.86 (0.11)	15.83 (0.42)	20.58 (0.29)	34.25 (0.54)
DSDA	6.96 (0.09)	25.13 (1.22)	17.21 (0.38)	46.92 (1.46)
LPD	8.84 (0.69)	34.48 (1.56)	17.98 (0.48)	55.50 (1.97)
ROAD	7.42 (0.12)	25.16 (0.98)	17.36 (0.35)	46.80 (1.17)
PLDA	16.48 (0.12)	2.26 (0.48)	32.69 (0.14)	8.57 (0.57)
SSSC	9.27 (0.17)	—	—	—
GSLDA	6.60 (0.10)	25.48 (1.83)	15.41 (0.43)	49.07 (2.19)
GSLDA-S	5.56 (0.07)	34.43 (2.52)	3.33 (0.41)	70.1 (2.77)
GSLDA-O	6.19 (0.09)	27.26 (1.72)	7.37 (0.47)	58.89 (2.08)
GSLDA-SO	5.79 (0.07)	30.78 (1.94)	2.16 (0.39)	67.62 (2.31)
Oracle	3.32 (0.01)	0 (0)	0 (0)	39 (0)

Table 3:

Performance comparisons of different classification methods for Example 3.

	Error	FP	FN	Size
NB	36.86 (0.80)	—	—	—
NSC	24.16 (0.84)	29.15 (3.35)	44.78 (1.62)	50.37 (4.93)
SLDA	13.28 (0.72)	21.07 (2.29)	40.59 (1.57)	46.48 (3.87)
PLR	11.09 (0.12)	21.44 (0.56)	42.08 (0.39)	45.36 (0.75)
DSDA	10.94 (0.15)	30.32 (1.49)	38.12 (0.63)	58.20 (2.01)
LPD	13.19 (0.73)	41.67 (1.52)	39.84 (0.82)	67.83 (2.25)
ROAD	11.25 (0.15)	33.14 (1.46)	37.53 (0.55)	61.61 (1.92)
PLDA	26.31 (0.68)	22.34 (2.33)	50.89 (1.12)	37.45 (3.41)
SSSC	13.57 (0.91)	—	—	—
GSLDA	10.53 (0.10)	27.34 (1.91)	36.67 (0.85)	56.67 (2.67)
GSLDA-S	8.77 (0.08)	34.08 (2.77)	18.2 (0.72)	81.88 (3.37)
GSLDA-O	9.77 (0.08)	36.87 (2.54)	26.22 (0.78)	76.65 (3.27)
GSLDA-SO	8.91 (0.08)	35.17 (2.37)	16.31 (0.63)	84.86 (3.01)
Oracle	5.36 (0.02)	0 (0)	0 (0)	66 (0)

Appendix A. Some comments on the GSLDA method

Appendix A.2. Connection between GSLDA and existing methods

We first consider the case when $\mathcal{G}$ is a complete graph. Without loss of generality, we assume that there is a unique minimum weight, i.e., there exists an ℓ such τ_ℓ < τ_j for all j ≠ ℓ. In this case, for any $\beta \in {\mathbb{R}}^p$ and v⁽¹⁾ + ⋯ + v^(p) = β, we have

$$\sum _{j=1}^p{\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2\ge {\tau }_{\ell }\sum _{j=1}^p{\Vert {\mathbf{v}}^{(j)}\Vert }_2\ge {\tau }_{\ell }{\Vert \beta \Vert }_2.$$

By taking v^(l) = β and v^(j) = 0 for all j ≠ ℓ, the regularization (5) becomes ${\Vert \beta \Vert }_{\mathcal{G},\tau }={\tau }_{\ell }{\Vert \beta \Vert }_2$. Similarly, we can show the equivalence in the case where $\mathcal{G}$ consists of K disjoint complete subgraphs.

Figure A.1:

A 3-dimensional LDA example demonstrating how marginal differences of the three features (δ₁, δ₂, δ₃) contribute to the predictive power of all features. Here ω₂₃ = ω₃₂ = 0. The terms around each node represent a decomposition of the corresponding coefficient. The gray scale of each term and the edge direction together indicate the source of the marginal differences.

Appendix B. Numerical results

Appendix B.1. Graph estimation results

To better understand the performance of our proposed GSLDA methods, we also present the graph estimation results for the methods. In particular, we compare the graph estimation based on both labeled data (for supervised GSLDA) and unlabeled data (for semi-supervised GSLDA) with the true graphs, within-class graph $\mathcal{G}$ and overall graph $\tilde{\mathcal{G}}$. The accuracy metrics include false positives (TP) and false positives (FP).

Table B.1:

Graph estimation accuracy for all examples in the simulations. The graphs are estimated with labeled data (L) after centering, or with unlabeled data (U). The former estimation is compared with $\mathcal{G}$, and the latter is compared with both $\mathcal{G}$ and $\tilde{\mathcal{G}}$. The results are averaged over 100 repetitions and the standard errors are provided in the parentheses.

Graph Type	Data	TP	FP	Size	True Size
Block Sparse	L	51.54 (0.25)	6.86 (0.48)	58.4 (0.55)	$\mathcal{G}:100(0)$
	U	100 (0)	82.96 (0.66)	182.96 (0.66)	$\mathcal{G}:100(0)$
	U	176.48 (0.46)	6.48 (0.38)	182.96 (0.66)	$\tilde{\mathcal{G}}:600(0)$
AR(3)	L	468.02 (1.31)	69.64 (1.24)	537.66 (1.34)	$\mathcal{G}:1188(0)$
	U	1178.04 (0.38)	76.72 (0.87)	1254.76 (1.01)	$\mathcal{G}:1188(0)$
	U	1235.76 (0.75)	19 (0.61)	1254.76 (1.01)	$\tilde{\mathcal{G}}:2508(0)$
Random Sparse	L	353.14 (2.02)	69.34 (1.25)	422.48 (1.73)	$\mathcal{G}:818(0)$
	U	814.52 (0.18)	72.74 (1.04)	887.26 (1.12)	$\mathcal{G}:818(0)$
	U	866.14 (0.87)	21.12 (0.70)	887.26 (1.12)	$\tilde{\mathcal{G}}:2426(0)$
Scale-Free	L	374.92 (1.37)	32.44 (0.92)	407.36 (1.48)	$\mathcal{G}:776(0)$
	U	709.88 (0.73)	103.5 (1.00)	813.38 (1.18)	$\mathcal{G}:776(0)$
	U	799.08 (1.05)	14.3 (0.53)	813.38 (1.18)	$\tilde{\mathcal{G}}:3564(0)$

Appendix B.2. Additional simulation results

The misclassification rates may not reflect the comprehensive performance of classification models, especially when the classes are unbalanced. Thus we present the receiver operating characteristic (ROC) curve for the classification models. Besides the balanced class setting as in the main text, we also consider an unbalanced class setting in which Class-0 accounts for 80% of the whole dataset. As we can see from Figures B.1 and B.2, our methods still outperforms other methods in terms of higher sensitivities at each specificity level.

Figure B.1:

ROC Curve under the balanced setting for the four examples. The proportion of Class-0 sample is 50%. The ROC curve is computed based on 100 repetitions.

Figure B.2:

ROC Curve under the unbalanced setting for the four examples. In particular, the proportion of Class-0 sample is 80%. The ROC curve is computed based on 100 repetitions.

Appendix C. Proofs to the theoretical results

Appendix C.1. Proof of Proposition 1

The random variable X can be represented as X = ξZ₁ + (1 − ξ)Z₂, where (i) ξ ~ Bin(1, π₁) is a Bernoulli random variable and (ii) Z₁ and Z₂ are from the two population components, respectively. Moreover, ξ, Z₁, and Z₂ are mutually independent. We have var(Z₁) = var(Z₂) = Σ, EZ₁ = μ⁽¹⁾, and EZ₂ = μ⁽²⁾. Then E(X) = π₁μ⁽¹⁾ + π₂μ⁽²⁾ and

$$\mathrm{E}({\mathbf{XX}}^{⊺})=\mathrm{E}\{{\xi }^2{\mathbf{Z}}_1{\mathbf{Z}}_1^{⊺}+{(1-\xi )}^2{\mathbf{Z}}_2{\mathbf{Z}}_2^{⊺}+\xi (1-\xi ){\mathbf{Z}}_1{\mathbf{Z}}_2^{⊺}+\xi (1-\xi ){\mathbf{Z}}_2{\mathbf{Z}}_1^{⊺}\}={\pi }_1\mathrm{E}({\mathbf{Z}}_1{\mathbf{Z}}_1^{⊺})+{\pi }_2\mathrm{E}({\mathbf{Z}}_2{\mathbf{Z}}_2^{⊺}).$$

Thus the overall covariance matrix is var(X) = Σ + π₁π₂δδ^⊺, where δ = μ⁽¹⁾ − μ⁽²⁾.

Now we verify the inverse matrix of var(X), i.e., the overall precision matrix of the mixture distribution. By setting c = π₁π₂/(1 + π₁π₂δ^⊺Σ⁻¹δ), we have

$$({\Sigma }^{-1}-c{\Sigma }^{-1}{\delta \delta }^{⊺}{\Sigma }^{-1})(\Sigma +{\pi }_1{\pi }_2{\delta \delta }^{⊺})=\mathbf{I}+{\pi }_1{\pi }_2{\Sigma }^{-1}{\delta \delta }^{⊺}-c{\Sigma }^{-1}{\delta \delta }^{⊺}-{\pi }_1{\pi }_{2}c{\Sigma }^{-1}{\delta \delta }^{⊺}{\Sigma }^{-1}{\delta \delta }^{⊺}=\mathbf{I}.$$

Denote β* = Σ⁻¹δ. Then we have var(X)⁻¹ = Σ⁻¹ − cβ*β*^⊺. □

Appendix C.2. Proof of Theorem 1

Before the proof, we introduce a lemma from [7]. The proof is omitted.

Lemma 4. Let ξ₁, … , ξ_n be independent random variables with mean zero. Suppose that there exists some t > 0 and ${\stackrel{‒}{B}}_n$ such that ${\Sigma }_{k=1}^n\mathrm{E}({\xi }_k^2{e}^{t\mid {\xi }_k\mid })\le {\stackrel{‒}{B}}_n^2$. Set C_t = t + t⁻¹. Then uniformly for $x\in (0,{\stackrel{‒}{B}}_n]$,

$$\mathrm{Pr}\left(\sum _{k=1}^n{\xi }_k\ge C_t{\stackrel{‒}{B}}_{n}x\right)\le \exp (-x^2).$$

Denote ${\xi }_1={\Vert {\tilde{\mathbf{S}}}_{\mathit{AA}}-{\tilde{\Sigma }}_{\mathit{AA}}\Vert }_{\infty }$, ${\xi }_2={\Vert {\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}-{\tilde{\Sigma }}_{\mathit{AA}}^{-1}\Vert }_{\infty }$, and $\xi ={\Vert {\tilde{\mathbf{S}}}_{A^{\mathrm{C}}A}{\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}-{\tilde{\Sigma }}_{A^{\mathrm{C}}A}{\tilde{\Sigma }}_{\mathit{AA}}^{-1}\Vert }_{\infty }$. With simple calculations, one can show that for all $\beta \in {\mathbb{R}}^p$,

$$\underset{{\beta }_0\in \mathbb{R}}{\min }\sum _{i=1}^n{(y_i-{\beta }_0-{\mathbf{x}}_i^{⊺}\beta )}^2=\sum _{i=1}^n{(y_i-{\stackrel{.}{\mathbf{x}}}_i^{⊺}\beta )}^2,$$

where ${\stackrel{.}{\mathbf{x}}}_i={\mathbf{x}}_i-({\mathbf{x}}_1+\cdots +{\mathbf{x}}_n)∕n$ is the centralized feature vector. Thus the loss function of GSLDA in (4) is equivalent to ${\Sigma }_{i=1}^n{(y_i-{{\stackrel{.}{\mathbf{x}}}_i}^{⊺}\beta )}^2∕n+\lambda {\Vert \beta \Vert }_{\mathcal{G},\tau }$. In the rest of our proof, we assume the sample X has been centralized. Then the GSLDA formulation (4) becomes

$$\widehat{\beta }=\underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}{\Vert Y-\mathbf{X}\beta \Vert }_2^2∕n+\lambda {\Vert \beta \Vert }_{\mathcal{G},\tau }.$$ (C.1)

Under the assumption (A1), we can define

$$\widehat{\gamma }=\underset{\gamma \in {\mathbb{R}}^s}{\mathrm{argmin}}{\Vert Y-{\mathbf{X}}_A\gamma \Vert }_2^2∕n+\lambda {\Vert \gamma \Vert }_{{\mathcal{G}}_A,{\tau }_A},$$ (C.2)

where ${\mathcal{G}}_A$ denotes the subgraph of $\mathcal{G}$ corresponding to A. If we can show that (i) all elements of $\widehat{\gamma }$ are non-zero; and (ii) $\widehat{\beta }$ with ${\widehat{\beta }}_A=\widehat{\gamma }$ and ${\widehat{\beta }}_{A^{\mathrm{C}}}=0$ solves (C.1); then, GSLDA estimation recovers all significant features accurately.

We first show statement (i). By Section 4.6 of [31], the formulation (C.2) is equivalent to $\widehat{\gamma }={\Sigma }_{j\in A}{\widehat{\mathbf{u}}}^{(j)}$ where

$$\{{\widehat{\mathbf{u}}}^{(j)}:j\in A\}\underset{{\mathbf{u}}^{(j)}\in {\mathbb{R}}^s:\mathrm{supp}({\mathbf{u}}^{(j)})\subseteq {\mathcal{N}}^{(j)}\cap A,j\in A}{\mathrm{argmin}}{\Vert Y-\sum _{j\in A}{\mathbf{X}}_A{\mathbf{u}}^{(j)}\Vert }_2^2∕n+\lambda \sum _{j\in A}{\tau }_j{\Vert {\mathbf{u}}^{(j)}\Vert }_2.$$

Since this is a convex optimization problem, any solution {u^(j) : j ∈ A} satisfies the KKT conditions [4], which are for all j ∈ A, either

$${\mathbf{u}}^{(j)}\ne 0\text{and}2{\mathbf{X}}_{\mathcal{N}(j)}^{⊺}(Y-{\mathbf{X}}_A\gamma )∕n=\lambda {\tau }_j{\mathbf{u}}_{{\mathcal{N}}^{(j)}}^{(j)}∕{\Vert {\mathbf{u}}^{(j)}\Vert }_2,$$

or

$${\mathbf{u}}^{(j)}=0\text{and}2{\Vert {\mathbf{X}}_{{\mathcal{N}}^{(j)}}^{⊺}(Y-{\mathbf{X}}_A\gamma )\Vert }_2∕n\le \lambda {\tau }_j,$$

where γ = Σ_j∈A u^(j). Thus we have ${\Vert {\mathbf{X}}_A^{⊺}(Y-{\mathbf{X}}_A\widehat{\gamma })\Vert }_{\infty }∕n\le \lambda {\tau }^{\ast }∕2$, and we can write $\widehat{\gamma }$ as $\widehat{\gamma }={\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}({\widehat{\delta }}_A+\lambda {\tau }^{\ast }{\mathbf{t}}_A∕2)$, where ${\mathbf{t}}_A\in {\mathbb{R}}^s$ satisfies ||t_A||_∞ ≤ 1. We have

$$\begin{gathered} {\Vert \widehat{\gamma }-{\beta }_A^{†}\Vert }_{\infty }={\Vert ({\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}-{\tilde{\Sigma }}_{\mathit{AA}}^{-1}){\delta }_A-{\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}({\delta }_A-{\widehat{\delta }}_A)+{\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}\lambda {\tau }^{\ast }{\mathbf{t}}_A∕2\Vert }_{\infty } \\ \le {\Vert {\delta }_A\Vert }_{\infty }{\xi }_2+(\phi +{\xi }_2){\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }+\lambda {\tau }^{\ast }(\phi +{\xi }_2)∕2 \\ \le {\xi }_2({\Vert {\delta }_A\Vert }_{\infty }+{\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }+\lambda {\tau }^{\ast }∕2)+\phi ({\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }+\lambda {\tau }^{\ast }∕2) \\ \le \frac{{\phi }^2{\xi }_1}{1-\phi {\xi }_1}({\Vert {\delta }_A\Vert }_{\infty }+{\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }+\lambda {\tau }^{\ast }∕2)+\phi ({\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }+\lambda {\tau }^{\ast }∕2)\equiv L_1, \end{gathered}$$

in which the second inequality holds for sufficiently large n because φξ₁ ≤ 1 and ξ₂ ≤ (1 − φξ₁)⁻¹ φ²ξ₁. If ξ₁ ≤ ϵ and ${\Vert {\widehat{\delta }}_A-{\delta }_A\Vert }_{\infty }\le ϵ$, then L₁ = O(ϵ) + λτ*φ/2 > 0, which provides (i). By Lemma 4, the statement (i) is true with probability at least 1 − 2s² exp(−a₁nϵ²/s²) − 2sexp(−a₂nϵ²), for some positive a₁ and a₂.

Now we prove statement (ii). The formulation (C.1) is equivalent to $\widehat{\beta }={\Sigma }_{j=1}^p{\widehat{\mathbf{v}}}^{(j)}$, where

$$\{{\widehat{\mathbf{v}}}^{(1)},\dots ,{\widehat{\mathbf{v}}}^{(p)}\}=\underset{{\mathbf{v}}^{(j)}:\mathrm{supp}({\mathbf{v}}^{(j)})\subseteq {\mathcal{N}}^{(j)},1\le j\le p}{\mathrm{argmin}}\frac{1}{n}{\Vert Y-\sum _{j=1}^p{\mathbf{Xv}}^{(j)}\Vert }_2^2+\lambda \sum _{j=1}^p{\Vert {\mathbf{v}}^{(j)}\Vert }_2.$$ (C.3)

This is also a convex optimization problem and the KKT conditions of formulation (C.3) are for all j ∈ {1, … , p}, either

$${\mathbf{v}}^{(j)}\ne 0\text{and}2{\mathbf{X}}_{{\mathcal{N}}^{(j)}}^{⊺}(Y-\mathbf{X}\beta )∕n=\lambda {\tau }_j{\mathbf{v}}_{{\mathcal{N}}^{(j)}}^{(j)}∕{\Vert {\mathbf{v}}^{(j)}\Vert }_2,$$ (C.4)

or

$${\mathbf{v}}^{(j)}=0\text{and}2{\Vert {\mathbf{X}}_{{\mathcal{N}}^{(j)}}^{⊺}(Y-\mathbf{X}\beta )\Vert }_2∕n\le \lambda {\tau }_j,$$ (C.5)

where β = v⁽¹⁾ + ⋯ + v^(p). Let v^(j) = 0 for all j ∈ A^C, and ${\mathbf{v}}_A^{(j)}={\mathbf{u}}^{(j)},{\mathbf{v}}_{A^{\mathrm{C}}}^{(j)}=0$ for all j ∈ A. Then ${\beta }_A=\widehat{\gamma }$ and β_A^C = 0.

For j ∈ A, (C.4) holds owing to the definition of $\widehat{\gamma }$. For j ∈ A^C, $2{\Vert {\mathbf{X}}_{{\mathcal{N}}^{(j)}}^{⊺}(Y-\mathbf{X}\beta )\Vert }_2∕n\le 2\sqrt{\mid {\mathcal{N}}^{(j)}\mid }{\Vert {\mathbf{X}}_{{\mathcal{N}}^{(j)}}^{⊺}(Y-\mathbf{X}\beta )\Vert }_{\infty }∕n$. Denote $\eta ={\Vert \widehat{\delta }-\delta \Vert }_{\infty }$ if ${\Vert \widehat{\delta }-\delta \Vert }_{\infty }\le ϵ$ and ${\Vert {\tilde{\mathbf{S}}}_{A^{\mathrm{C}}A}-{\tilde{\Sigma }}_{A^{\mathrm{C}}A}\Vert }_{\infty }\le ϵ$, then

$${\Vert {\mathbf{X}}_{A^{\mathrm{C}}}^{⊺}(Y-{\mathbf{X}}_A\widehat{\gamma })\Vert }_{\infty }∕n={\Vert {\tilde{\mathbf{S}}}_{A^{\mathrm{C}}A}{\tilde{\mathbf{S}}}_{\mathit{AA}}^{-1}({\widehat{\delta }}_A+\lambda {\tau }^{\ast }{\mathbf{t}}_A∕2)-{\widehat{\delta }}_{A^{\mathrm{C}}}\Vert }_{\infty }\le ({\Vert {\delta }_A\Vert }_{\infty }+1+ϵ+\kappa +\lambda {\tau }^{\ast }∕2)ϵ+\lambda {\tau }^{\ast }\kappa ∕2\le O(ϵ)+\lambda {\tau }_{\ast }∕2.$$

By Lemma 4, the statement (ii) is true with probability at least 1 − 2ps exp(−a₁nϵ²/s²)−2p exp(−a₂nϵ²). By taking $ϵ=\sqrt{\ln p∕n}$, the active set is recovered and ${\Vert {\widehat{\beta }}_A-{\beta }_A^{†}\Vert }_{\infty }\le O(\sqrt{\ln p∕n})$ with probability at least 1 − 2s² exp(−a₁nϵ²/s²)−2ps exp(−a₁nϵ²/s²) − 2p exp(−a₂nϵ²) = 1 − O(p^−C₁) for some C₁ > 0. □

Appendix C.3. Proof of Theorem 2

The proof uses the following lemma from [30].

Lemma 5. Denote $\mathcal{M}$ a subspace of ${\mathbb{R}}^p$ and ${\mathcal{M}}^{\perp }$ its orthogonal complement. For a regularized estimation problem

$$\widehat{\theta }=\underset{\theta \in {\mathbb{R}}^p}{\arg \min }L(\theta )+\lambda R(\theta ),$$

where

• (i)R is a norm and is decomposable with respect to ($\mathcal{M},{\mathcal{M}}^{\perp }$), i.e., R(θ + η) = R(θ) + R(η) for all $\theta \in \mathcal{M},\mathit{\eta }\in {\mathcal{M}}^{\perp }$;
• (ii)L is convex and differentiable, and satisfies restricted strong convex condition with curvature κ_L, i.e., $\delta L({\mathit{\theta }}^{\ast },\Delta )=L({\mathit{\theta }}^{\ast }+\Delta )-L({\mathit{\theta }}^{\ast })-\nabla L({\mathit{\theta }}^{\ast }{)}^{⊺}\Delta \ge {\kappa }_L\parallel \Delta {\parallel }_2^2$ for some θ*, for all Δ such that $R({\Delta }_{{\mathcal{M}}^{\perp }})\le 3R({\Delta }_{\mathcal{M}})+4R({\theta }_{{\mathcal{M}}^{\perp }}^{\ast })$.

Let λ ≥ 2R*{∇L(θ*)}, where R* denote the dual norm of R, then any solution $\widehat{\mathit{\theta }}$ to the problem satisfies

$${\Vert \widehat{\theta }-{\theta }^{\ast }\Vert }_2\le 9{\lambda }^2\psi (\mathcal{M})∕{\kappa }_L^2+4\lambda R^{\ast }({\theta }_{{\mathcal{M}}^{\perp }}^{\ast })∕{\kappa }_L,$$

where $\psi (\mathcal{M})={\text{sup}}_{\mathbf{u}\in \mathcal{M}∕\{0\}}R(\mathbf{u})∕\parallel \mathbf{u}{\parallel }_2$.

Proof. In the GSLDA formulation, the loss function is $L({\beta }_0,\beta )=\parallel \mathbf{y}-{\beta }_{0}1-\mathbf{X}\beta {\parallel }_2^2∕n$, and the regularization is $R(\beta )=\parallel \beta {\parallel }_{\mathcal{G},\tau }$. It has been shown in [31] that R is a norm and its dual norm is $R^{\ast }(\mathbf{u})={\text{max}}_{1\le j\le p}{\tau }_j^{-1}\parallel {\mathbf{u}}_{\mathcal{N}(j)}{\parallel }_2$. When we take ${\tau }_j=\sqrt{\mid {\mathcal{N}}^{(j)}\mid },R^{\ast }(\mathbf{u})\le {\text{max}}_j\parallel {\mathbf{u}}_{{\mathcal{N}}^{(j)}}{\parallel }_{\infty }=\parallel \mathbf{u}{\parallel }_{\infty }$.

For some ϵ > 0 denote the event $\mathcal{X}=\{\parallel \widehat{\delta }-\delta {\parallel }_{\infty }\le \in ,\parallel \mid \tilde{\mathbf{S}}{.}_{,A}-\tilde{\Sigma }{.}_{,A}\mid {\parallel }_{\infty }\le \in \}$. Then by Lemma 4, $\text{Pr}(\mathcal{X})\ge 1-2p\text{exp}(-a_{2}n{ϵ}^2)-2ps\text{exp}(-a_{1}n{ϵ}^2)$. Under the event $\mathcal{X}$,

$$\begin{gathered} {\Vert \nabla L({\beta }^{†})\Vert }_{\infty }={\Vert 2n^{-1}{\mathbf{X}}^{⊺}(\mathbf{y}-\mathbf{X}{\beta }^{†})\Vert }_{\infty } \\ ={\Vert 2(\widehat{\delta }-\tilde{\mathbf{S}}{\beta }^{†}\Vert }_{\infty }={\Vert 2(\widehat{\delta }-\delta )-2(\tilde{\mathbf{S}}-\tilde{\Sigma }){\beta }^{†}\Vert }_{\infty }\le 2{\Vert \widehat{\delta }-\delta \Vert }_{\infty }+2{\Vert {(\tilde{\mathbf{S}}-\tilde{\Sigma })}_{\cdot ,A}{\beta }_A^{†}\Vert }_{\infty } \\ \le 2{\Vert \widehat{\delta }-\delta \Vert }_{\infty }+2{\Vert {(\tilde{\mathbf{S}}-\tilde{\Sigma })}_{\cdot ,A}\Vert }_{\infty }{\Vert {\beta }_A^{†}\Vert }_1\le 2ϵ+2{\Vert {\beta }_A^{†}\Vert }_1ϵ. \end{gathered}$$

We take $\in =C_2\sqrt{\text{ln}p∕n}$ where C₂ > (a₁ ∧ a₂)⁻¹. Then $\lambda \ge 2R^{\ast }(\nabla L({\beta }^{†}))$. Under the event $\mathcal{X}$ with ϵ ≤ cσ for some c > 0, for sufficiently large n, we have ${\Delta }^{⊺}\tilde{\mathbf{S}}\Delta \ge {\Delta }^{⊺}\tilde{\Sigma }\Delta -\mid {\Delta }^{⊺}(\tilde{\mathbf{S}}-\tilde{\Sigma })\Delta \mid \ge \sigma \parallel \Delta {\parallel }_2^2∕2$ for $\Delta \in \mathcal{C}(A)$. Thus $\delta L({\beta }^{\ast },\Delta )\ge 2{\Delta }^{⊺}\tilde{\mathbf{S}}\Delta \ge \sigma {\Vert \Delta \Vert }_2^2$ for all $\Delta \in \mathcal{C}(A)$.

We take $\mathcal{M}=\{\beta \in {\mathbb{R}}^p:{\beta }_{A^{\mathrm{C}}}=0\}$. Then ${\beta }^{†}\in \mathcal{M}$ and ${\beta }_{{\mathcal{M}}^{\perp }}^{†}=0$. Moreover,

$$\psi (\mathcal{M})=\underset{\beta \in \mathcal{M}}{\sup }\frac{R(\beta )}{{\Vert \beta \Vert }_2}\underset{{\beta }_A\mathrm{C}=0}{\sup }\frac{{\min }_{\Sigma {\mathbf{v}}^{(j)}=\beta }\sum {\tau }_j{\Vert {\mathbf{v}}^{(j)}\Vert }_2}{{\Vert \beta \Vert }_2}\le \underset{{\beta }_A\mathrm{C}=0}{\sup }\frac{\sum _{j\in A}{\tilde{\tau }}_j\mid {\beta }_j\mid }{{\Vert {\beta }_A\Vert }_2}\le {\tau }^{\ast }\sqrt{s}.$$

Therefore, by Lemma 5, we have

$${\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2^2\le 9{\lambda }^{2}s{\tau }^{\ast 2}∕{\sigma }^2,$$

with probability at least 1 − 2ps exp(−a₁nϵ²) − 2p exp(−a₂nϵ²) ≥ 1 − sp^−C₃ where C₃ = C₂(a₁ ∨ a₂) − 1 > 0. □

Appendix C.4. Proof of Theorem 3

We use the same notations as in the proof above. Without loss of generality, we assume μ⁽¹⁾ + μ⁽²⁾ = 0, then μ⁽¹⁾ = δ/2, μ⁽²⁾ = −δ/2, and ${\beta }_0^{†}=0$. According to Proposition 2, we have

$$\begin{gathered} Q({\widehat{\beta }}_0,\widehat{\beta })∕Q({\beta }_0^{†},{\beta }^{†})-1=\left[\left\{\Phi \left(\frac{-{\widehat{\beta }}_0-{\widehat{\beta }}^{⊺}{\mu }^{(1)}}{\sqrt{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }}}\right)-\Phi \left(\frac{-{\beta }^{†T}{\mu }^{(1)}}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)\right\}+\left\{\Phi \left(\frac{{\widehat{\beta }}_0-{\widehat{\beta }}^{⊺}{\mu }^{(2)}}{\sqrt{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }}}\right)-\Phi \left(\frac{{\beta }^{†T}{\mu }^{(2)}}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)\right\}\right] \\ \times {\left\{\Phi \left(\frac{-{\beta }^{†T}{\mu }^{(1)}}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)+\Phi \left(\frac{{\beta }^{†T}{\mu }^{(2)}}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)\right\}}^{-1} \\ \le \max \left\{\Phi \left(\frac{-{\widehat{\beta }}_0-{\widehat{\beta }}^{⊺}\delta ∕2}{\sqrt{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }}}\right)∕\Phi \left(\frac{-{\beta }^{†T}\delta ∕2}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)-1,\Phi \left(\frac{{\widehat{\beta }}_0-{\widehat{\beta }}^{⊺}\delta ∕2}{\sqrt{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }}}\right)∕\Phi \left(\frac{-{\beta }^{†T}\delta ∕2}{\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}}\right)-1\right\} \\ \equiv \max \{R^{(1)},R^{(2)}\}. \end{gathered}$$

We will use the following property of standard Gaussian distribution function [6]:

$$\mid \Phi (x_0+r)∕\Phi (x_0)-1\mid \le c_1\mid r\mid (\mid x_0\mid +1)\exp (c_2\mid x_{0}r\mid ).$$ (C.6)

For k ∈ {1, 2}, let

$$r^{(k)}=\mid ({\widehat{\beta }}_0+{\widehat{\beta }}^{⊺}{\mu }^{(k)})∕\sqrt{{\beta }^{⊺}\Sigma \widehat{\beta }}-{\beta }^{†T}{\mu }^{(k)}∕\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}\mid .$$

Since ${\beta }^{†T}\delta ∕\sqrt{{\beta }^{†T}\Sigma {\beta }^{†}}={\Delta }^{1∕2}$, it suffices to verify the orders of r^(k) and Δ.

According to Theorem 2, $\parallel \widehat{\beta }-{\beta }^{†}{\parallel }_2\le 3\lambda {\tau }^{\ast }\sqrt{s}∕\sigma$ with probability going to 1. Moreover, by the definition of β^†, we have β^† = 4/(4 + Δ)β*, and β^†T Σβ^† = 16Δ/(4 + Δ)². Since

$$\begin{gathered} \mid {\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }-{\beta }^{†T}\Sigma {\beta }^{†}\mid \le \mid {(\widehat{\beta }-{\beta }^{†})}^{⊺}\Sigma (\widehat{\beta }-{\beta }^{†})\mid +2\mid {(\widehat{\beta }-{\beta }^{†})}^{⊺}\Sigma {\beta }^{†}\mid \\ \le {\lambda }_{\max }(\Sigma ){\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2^2+2{\lambda }_{\max }(\Sigma ){\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2{\Vert {\beta }^{†}\Vert }_2\le 3{\lambda }_{\max }(\Sigma ){\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2{\Vert {\beta }^{†}\Vert }_2, \\ \le 9{\lambda }_{\max }(\Sigma )\lambda {\tau }^{\ast }\sqrt{s}{\Vert {\beta }^{†}\Vert }_2∕\sigma . \end{gathered}$$

for sufficiently large n, we have

$$\begin{gathered} \mid {({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{-1∕2}-{({\beta }^{†T}\Sigma {\beta }^{†})}^{-1∕2}\mid =\mid \frac{{({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{1∕2}-{({\beta }^{†T}\Sigma {\beta }^{†})}^{1∕2}}{{({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{1∕2}{({\beta }^{†T}\Sigma {\beta }^{†})}^{1∕2}}\mid \\ =\mid \frac{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }-{\beta }^{†T}\Sigma {\beta }^{†}}{{({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{1∕2}\cdot {({\beta }^{†T}\Sigma {\beta }^{†})}^{1∕2}[{({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{1∕2}+{({\beta }^{†T}\Sigma {\beta }^{†})}^{1∕2}]}\mid \\ \le \mid \frac{{\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }-{\beta }^{†T}\Sigma {\beta }^{†}}{3∕4{({\beta }^{†T}\Sigma {\beta }^{†})}^{3∕2}}\mid \le \frac{{(4+\Delta )}^3}{4{\Delta }^{3∕2}}{\lambda }_{\max }(\Sigma )\lambda {\tau }^{\ast }\sqrt{s}{\Vert {\beta }^{†}\Vert }_2∕\sigma , \end{gathered}$$

in which the first inequality holds because ${\widehat{\beta }}^{⊺}\Sigma \widehat{\beta }\ge {\beta }^{†T}\Sigma {\beta }^{†}∕2$ for sufficiently large n. Moreover, under $\mathcal{X}$ we have,

$$\begin{gathered} \mid ({\widehat{\beta }}_0+{\widehat{\beta }}^{⊺}\delta ∕2)-({\beta }^{†T}\delta ∕2)\mid \le \mid {(\widehat{\beta }-{\beta }^{†})}^{⊺}\delta \mid ∕2+\mid -{\stackrel{‒}{\mathbf{x}}}^{⊺}\widehat{\beta }\mid \\ \le {\Vert \widehat{\beta }-{\beta }^{†}\Vert }_2{\Vert \delta \Vert }_2∕2+\mid {\stackrel{‒}{\mathbf{x}}}^{⊺}{\beta }^{†}\mid +\mid {\stackrel{‒}{\mathbf{x}}}^{⊺}(\widehat{\beta }-{\beta }^{†})\mid \le 3\lambda {\tau }^{\ast }\sqrt{s}{\Vert \delta \Vert }_2∕(2\sigma )+{\Vert {\beta }^{†}\Vert }_1ϵ. \end{gathered}$$

Therefore,

$$\begin{gathered} r^{(1)}\le \mid ({\widehat{\beta }}_0+{\widehat{\beta }}^{⊺}\delta ∕2)\{{({\widehat{\beta }}^{⊺}\Sigma \widehat{\beta })}^{-1∕2}-{({\beta }^{†}\Sigma {\beta }^{†})}^{-1∕2}\}\mid +\mid ({\widehat{\beta }}_0+{\widehat{\beta }}^{⊺}\delta ∕2)-{\beta }^{†T}\delta ∕2\mid ∕{({\beta }^{†T}\Sigma {\beta }^{†})}^{1∕2} \\ \le \frac{{(4+\Delta )}^2}{2{\Delta }^{1∕2}}\lambda {\tau }^{\ast }{\lambda }_{\max }(\Sigma )\sqrt{s}{\Vert {\beta }^{†}\Vert }_2∕\sigma +\frac{4+\Delta }{4{\Delta }^{1∕2}}\{2\lambda {\tau }^{\ast }\sqrt{s}{\Vert \delta \Vert }_2∕\sigma +{\Vert {\beta }^{†}\Vert }_1ϵ\}. \end{gathered}$$

Using the property (C.6), then we have

$$R^{(1)}=\mid \frac{\Phi (-{\Delta }^{1∕2}∕2+r^{(1)}}{\Phi (-{\Delta }^{1∕2}∕2)}-1\mid \le c_1\mid r^{(1)}\mid ({\Delta }^{1∕2}∕2+1)\exp (c_2{r}^{(1)}{\Delta }^{1∕2}∕2)=O\{r^{(1)}{\Delta }^{1∕2}\exp (c_2{r}^{(1)}{\Delta }^{1∕2}∕2)\}.$$

Since ${\Delta }^2\lambda {\tau }^{\ast }{\lambda }_{\max }(\Sigma )\sqrt{s}{\Vert {\beta }^{†}\Vert }_2∕\sigma \to 0$, $\Delta \lambda {\tau }^{\ast }\sqrt{s}{\Vert \delta \Vert }_2∕\sigma \to 0$, and $\Delta n^{\gamma -1}{\Vert {\beta }^{†}\Vert }_1\to 0$, we have r⁽¹⁾Δ^1/2 → 0 and thus $R^{(1)}\stackrel{p}{\to }0$. Similarly, we can show $R^{(2)}\stackrel{p}{\to }0$, which proves the theorem.