Feature Grouping and Selection Over an Undirected Graph

Abstract

High-dimensional regression/classification continues to be an important and challenging problem, especially when features are highly correlated. Feature selection, combined with additional structure information on the features has been considered to be promising in promoting regression/classification performance. Graph-guided fused lasso (GFlasso) has recently been proposed to facilitate feature selection and graph structure exploitation, when features exhibit certain graph structures. However, the formulation in GFlasso relies on pairwise sample correlations to perform feature grouping, which could introduce additional estimation bias. In this paper, we propose three new feature grouping and selection methods to resolve this issue. The first method employs a convex function to penalize the pairwise l_∞ norm of connected regression/classification coefficients, achieving simultaneous feature grouping and selection. The second method improves the first one by utilizing a non-convex function to reduce the estimation bias. The third one is the extension of the second method using a truncated l₁ regularization to further reduce the estimation bias. The proposed methods combine feature grouping and feature selection to enhance estimation accuracy. We employ the alternating direction method of multipliers (ADMM) and difference of convex functions (DC) programming to solve the proposed formulations. Our experimental results on synthetic data and two real datasets demonstrate the effectiveness of the proposed methods.

1. INTRODUCTION

High-dimensional regression/classification is challenging due to the curse of dimensionality. Lasso [18] and its various extensions, which can simultaneously perform feature selection and regression/classification, have received increasing attention in this situation. However, in the presence of highly correlated features lasso tends to only select one of those features resulting in suboptimal performance [27]. Several methods have been proposed to address this issue in the literature. Shen and Ye [15] introduce an adaptive model selection procedure that corrects the estimation bias through a data-driven penalty based on generalized degrees of freedom. The Elastic Net [27] uses an additional l₂ regularizer to encourage highly correlated features to stay together. However, these methods do not incorporate prior knowledge into the regression/classification process, which is critical in many applications. As an example, many biological studies have suggested that genes tend to work in groups according to their biological functions, and there are some regulatory relationships between genes [10]. This biological knowledge can be represented as a graph, where the nodes represent the genes, and the edges imply the regulatory relationships between genes. Therefore, we want to study how estimation accuracy can be improved using dependency information encoded as a graph.

Given feature grouping information, the group lasso [1, 7, 22, 11] yields a solution with grouped sparsity using l₁/l₂ penalty. The orignal group lasso does not consider the overlaps between groups. Zhao et al. [24] extend the group lasso to the case of overlapping groups. Jacob et al. [7] introduce a new penalty function leading to a grouped sparse solution with overlapping groups. Yuan et al. [21] propose an efficient method to solve the overlapping group lasso. Other extensions of group lasso with tree structured regularization include [11, 8]. Prior works have demonstrated the benefit of using feature grouping information for high-dimensional regression/classification. However, these methods need the feature groups to be pre-specified. In other words, they only utilize the grouping information to obtain solutions with grouped sparsity, but lack the capability of identifying groups.

There are also a number of existing methods for feature grouping. Fused lasso [19] introduces an l₁ regularization method for estimating subgroups in a certain serial order, but pre-ordering features is required before using fused lasso. A study about parameter estimation of the fused lasso can be found in [12]; Shen et al. [13] propose a non-convex method to select all possible homogenous subgroups, but it fails to obtain sparse solutions. OSCAR [2] employs an l₁ regularizer and a pairwise l_∞ regularizer to perform feature selection and automatic feature grouping. Li and Li [10] suggest a grouping penalty using a Laplacian matrix to force the co-efficients to be similar, which can be considered as a graph version of the Elastic Net. When the Laplacian matrix is an identity matrix, Laplacian lasso [10, 5] is identical to the Elastic Net. GFlasso employs an l₁ regularization over a graph, which penalizes the difference |β_i − sign(r_ij)β_j|, to encourage the coefficients β_i, β_j for features i, j connected by an edge in the graph to be similar when r_ij > 0, but dissimilar when r_ij < 0, where r_ij is the sample correlation between two features [9]. Although these grouping penalties can improve the performance, they would introduce additional estimation bias due to strict convexity of the penalties or due to possible graph misspecification. For example, additional bias may occur when the signs of coefficients for two features connected by an edge in the graph are different in Laplacian lasso [10, 5], or when the sign of r_ij is inaccurate in GFlasso [9].

In this paper, we focus on simultaneous estimation of grouping and sparseness structures over a given undirected graph. Features tend to be grouped when they are connected by an edge in a graph. When features are connected by an edge in a graph, the absolute values of the model coefficients for these two features should be similar or identical. We propose one convex and two non-convex penalties to encourage both sparsity and equality of absolute values of coefficients for connected features. The convex penalty includes a pairwise l_∞ regularizer over a graph. The first non-convex penalty improves the convex penalty by penalizing the difference of absolute values of coefficients for connected features. The other one is the extension of the first non-convex penalty using a truncated l₁ regularization to further reduce the estimation bias. These penalties are designed to resolve the aforementioned issues of Laplacian lasso and GFlasso. The non-convex penalties shrink only small differences in absolute values so that estimation bias can be reduced; several recent works analyze their theoretical properties [26, 14]. Through ADMM and DC programming, we develop computational methods to solve the proposed formulations. The proposed methods can combine the benefit of feature selection and that of feature grouping to improve regression/classification performance. Due to the equality of absolute values of coefficients, the model complexity of the learned model can be reduced. We have performed experiments on synthetic data and two real datasets. The results demonstrate the effectiveness of the proposed methods.

The rest of the paper is organized as follows. We introduce the proposed convex method in Section 2, and the two proposed non-convex methods in Section 3. Experimental results are given in Section 4. We conclude the paper in Section 5.

2. A CONVEX FORMULATION

Consider a linear model in which response y_i depends on a vector of p features:

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },$$ (1)

where β ∈ ^p is a vector of coefficients, X ∈ ^n×p is the data matrix, and ε is random noise. Given an undirected graph, we try to build a prediction model (regression or classification) incorporating the graph structure information to estimate the nonzero coefficients of β and to identify the feature groups when the number of features p is larger than the sample size n. Let (N, E) be the given undirected graph, where N = {1, 2, …, p} is a set of nodes, and E is the set of edges. Node i corresponds to feature x_i. If nodes i and j are connected by an edge in E, then features x_i and x_j tend to be grouped. The formulation of graph OSCAR (GOSCAR) is given by

$$\underset{\mathit{\beta }}{min}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathit{\beta }\Vert }_1+{\lambda }_2\sum _{(i,j)\in E}max\{\mid {\beta }_i\mid ,\mid {\beta }_j\mid \}$$ (2)

where λ₁, λ₂ are regularization parameters. We use a pairwise l_∞ regularizer to encourage the coefficients to be equal [2], but we only put grouping constraints over the nodes connected over the given graph. The l₁ regularizer encourages sparseness. The pairwise l_∞ regularizer puts more penalty on the larger coefficients. Note that max{|β_i|, |β_j|} can be decomposed as

$$max\{\mid {\beta }_i\mid ,\mid {\beta }_j\mid \}=\frac{1}{2}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid ).$$

$\frac{1}{2}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid )$ can be represented by

$$\mid {\mathbf{u}}^T\mathit{\beta }\mid +\mid {\mathbf{v}}^T\mathit{\beta }\mid ,$$

where u, v are sparse vectors, each with only two non-zero entries $u_i=u_j=\frac{1}{2},v_i=-v_j=\frac{1}{2}$. Thus Eq. (2) can be rewritten in a matrix form as

$$\underset{\mathit{\beta }}{min}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathit{\beta }\Vert }_1+{\lambda }_2{\Vert \mathbf{T}\mathit{\beta }\Vert }_1,$$ (3)

where T is a sparse matrix constructed from the edge set E.

The proposed formulation is closely related to OSCAR [2]. The penalty of OSCAR is λ₁||β||₁ + λ₂ Σ_i<j max{|β_i|, |β_j|}. The l₁ regularizer leads to a sparse solution, and the l_∞ regularizer encourages the coefficients to be equal. OSCAR can be efficiently solved by accelerated gradient methods, whose key projection can be solved by a simple iterative group merging algorithm [25]. However, OSCAR assumes each node is connected to all the other nodes, which is not sufficient for many applications. Note that OSCAR is a special case of GOSCAR when the graph is complete. GOSCAR, incorporating an arbitrary undirected graph, is much more challenging to solve.

2.1 Algorithm

We propose to solve GOSCAR using the alternating direction method of multipliers (ADMM) [3]. ADMM decomposes a large global problem into a series of smaller local subproblems and coordinates the local solutions to identify the globally optimal solution. ADMM attempts to combine the benefits of dual decomposition and augmented Lagrangian methods for constrained optimization [3]. The problem solved by ADMM takes the form of

$$\begin{array}{l}{min}_{\mathbf{x},\mathbf{z}}f(\mathbf{x})+g(\mathbf{z})\\ s.t.\mathbf{Ax}+\mathbf{Bz}=\mathbf{c}.\end{array}$$

ADMM uses a variant of the augmented Lagrangian method and reformulates the problem as follows:

$$L_{\rho }(\mathbf{x},\mathbf{z},\mathit{\mu })=f(\mathbf{x})+g(\mathbf{z})+{\mu }^T(\mathbf{Ax}+\mathbf{Bz}-\mathbf{c})+\frac{\rho }{z}{\Vert \mathbf{Ax}+\mathbf{Bz}-\mathbf{c}\Vert }^2,$$

with μ being the augmented Lagrangian multiplier, and ρ being the non-negative dual update step length. ADMM solves this problem by iteratively minimizing L_ρ(x, z, μ) over x, z, and μ. The update rule for ADMM is given by

$$\begin{array}{l}{\mathbf{x}}^{k+1}:arg\underset{\mathbf{x}}{min}{L}_{\rho }(\mathbf{x},{\mathbf{z}}^k,{\mathit{\mu }}^k),\\ {\mathbf{z}}^{k+1}:arg\underset{\mathbf{z}}{min}{L}_{\rho }({\mathbf{x}}^{k+1},\mathbf{z},{\mathit{\mu }}^k),\\ {\mathit{\mu }}^{k+1}:={\mathit{\mu }}^k+\rho ({\mathbf{Ax}}^{k+1}+{\mathbf{Bz}}^{k+1}-\mathbf{c}).\end{array}$$

Consider the unconstrained optimization problem in Eq. (3), which is equivalent to the following constrained optimization problem:

$$\begin{array}{l}{min}_{\mathit{\beta },\mathbf{q},\mathbf{p}}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathbf{q}\Vert }_1+{\lambda }_2{\Vert \mathbf{p}\Vert }_1\\ s.t.\mathit{\beta }-\mathbf{q}=\mathbf{0},\mathbf{T}\mathit{\beta }-\mathbf{p}=\mathbf{0},\end{array}$$ (4)

where q, p are slack variables. Eq. (4) can then be solved by ADMM. The augmented Lagrangian is

$$L_{\rho }(\mathit{\beta },\mathbf{q},\mathbf{p},\mathit{\mu },\mathit{\upsilon })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathbf{q}\Vert }_1+{\lambda }_2{\Vert \mathbf{p}\Vert }_1+{\mathit{\mu }}^T(\mathit{\beta }-\mathbf{q})+{\mathit{\upsilon }}^T(\mathbf{T}\mathit{\beta }-\mathbf{p})+\frac{\rho }{2}{\Vert \mathit{\beta }-\mathbf{q}\Vert }^2+\frac{\rho }{2}{\Vert \mathbf{T}\mathit{\beta }-\mathbf{p}\Vert }^2,$$

where μ, υ are augmented Lagrangian multipliers.

Update β

In the (k + 1)-th iteration, β^k+1 can be updated by minimizing L_ρ with q, p, μ, υ fixed:

$${\mathit{\beta }}^{k+1}=arg{min}_{\mathit{\beta }}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{({\mathit{\mu }}^k+{\mathbf{T}}^T{\mathit{\upsilon }}^k)}^T\mathit{\beta }+\frac{\rho }{2}{\Vert \mathit{\beta }-{\mathbf{q}}^k\Vert }^2+\frac{\rho }{2}{\Vert \mathbf{T}\mathit{\beta }-{\mathbf{p}}^k\Vert }^2.$$ (5)

The above optimization problem is quadratic. The optimal solution is given by β^k+1 = F⁻¹b^k, where

$$\begin{array}{l}\mathbf{F}={\mathbf{X}}^T\mathbf{X}+\rho (\mathbf{I}+{\mathbf{T}}^T\mathbf{T}),\\ {\mathbf{b}}^k={\mathbf{X}}^T\mathbf{y}-{\mathit{\mu }}^k-{\mathbf{T}}^T{\mathit{\upsilon }}^k+\rho {\mathbf{T}}^T{\mathbf{p}}^k+\rho {\mathbf{q}}^k.\end{array}$$

The computation of β^k+1 involves solving a linear system, which is the most time-consuming part in the whole algorithm. To compute β^k+1 efficiently, we compute the Cholesky factorization of F at the beginning of the algorithm:

$$\mathbf{F}={\mathbf{R}}^T\mathbf{R}.$$

Note that F is a constant and positive definite matrix. Using the Cholesky factorization we only need to solve the following two linear systems at each iteration:

$${\mathbf{R}}^T\widehat{\mathit{\beta }}={\mathbf{b}}^k,\mathbf{R}\mathit{\beta }=\widehat{\mathit{\beta }}.$$ (6)

Since R is an upper triangular matrix, solving these two linear systems is very efficient.

Update q

q^k+1 can be obtained by solving

$${\mathbf{q}}^{k+1}=arg\underset{\mathbf{q}}{min}\frac{\rho }{2}{\Vert \mathbf{q}-{\mathit{\beta }}^{k+1}\Vert }^2+{\lambda }_1{\Vert \mathbf{q}\Vert }_1-{({\mathit{\mu }}^k)}^T\mathbf{q}$$

which is equivalent to the following problem:

$${\mathbf{q}}^{k+1}=arg\underset{\mathbf{q}}{min}\frac{1}{2}{\Vert \mathbf{q}-{\mathit{\beta }}^{k+1}-\frac{1}{\rho }{\mathit{\mu }}^k\Vert }^2+\frac{{\lambda }_1}{\rho }{\Vert \mathbf{q}\Vert }_1$$ (7)

Eq. (7) has a closed-form solution, known as soft-thresholding:

$${\mathbf{q}}^{k+1}=S_{{\lambda }_1/\rho }({\mathit{\beta }}^{k+1}+\frac{1}{\rho }{\mathit{\mu }}^k),$$ (8)

where the soft-thresholding operator is defined as:

$$S_{\lambda }(x)=\text{sign}(x)max(\mid x\mid -\lambda ,0).$$

Update p

Similar to updating q, p^k+1 can also be obtained by soft-thresholding:

$${\mathbf{p}}_i^{k+1}=S_{{\lambda }_2/\rho }(\mathbf{T}{\mathit{\beta }}^{K+1}+\frac{1}{\rho }{\mathit{\upsilon }}^k).$$ (9)

Update μ, υ

$$\begin{array}{l}{\mathit{\mu }}^{k+1}={\mathit{\mu }}^k+\rho ({\mathit{\beta }}^{k+1}-{\mathbf{q}}^{k+1}),\\ {\mathit{\upsilon }}^{k+1}={\mathit{\upsilon }}^k+\rho (\mathbf{T}{\mathit{\beta }}^{k+1}-{\mathbf{p}}^{k+1}).\end{array}$$ (10)

A summary of GOSCAR is shown in Algorithm 1.

Algorithm 1.

The GOSCAR algorithm

In Algorithm 1, the Cholesky factorization only needs to be computed once, and each iteration involves solving one linear system and two soft-thresholding operations. The time complexity of the soft-thresholding operation in Eq. (8) is O(p). The other one in Eq. (9) involves a matrix-vector multiplication. Due to the sparsity of T, its time complexity is O(n_e), where n_e is the number of edges. Solving the linear system involves computing b^k and solving Eq. (6), whose total time complexity is O(p(p + n) + n_e). Thus the time complexity of each iteration is O(p(p + n) + n_e).

3. TWO NON-CONVEX FORMULATIONS

The grouping penalty of GOSCAR overcomes the limitation of Laplacian lasso that the different signs of coefficients can introduce additional penalty. However, under the l_∞ regularizer, even if |β_i| and |β_j| are close to each other, the penalty on this pair may still be large due to the property of the max operator, resulting in the coefficient β_i or β_j being over penalized. The additional penalty would result in biased estimation, especially for large coefficient, as in the lasso case [18]. Another related grouping penalty is GFlasso, |β_i − sign(r_ij)β_j|, where r_ij is the pairwise sample correlation. GFlasso relies on the pairwise sample correlation to decide whether β_i and β_j are enforced to be close or not. When the pairwise sample correlation wrongly estimates the sign between β_i and β_j, an additional penalty on β_i and β_j would occur, introducing estimation bias. This motivates our non-convex grouping penalty, ||β_i| − |β_j||, that shrinks only small differences in absolutes values. As a result, estimation bias is reduced as compared to these convex grouping penalties. The proposed non-convex methods perform well even when the graph is wrongly specified, unlike GFlasso. Note that the proposed non-convex grouping penalty does not assume the sign of an edge is given; it only relies on the graph structure.

3.1 Non-Convex Formulation I: ncFGS

The proposed non-convex formulation (ncFGS) solves the following optimization problem:

$$\underset{\mathit{\beta }}{min}f(\mathit{\beta })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathit{\beta }\Vert }_1+{\lambda }_2\sum _{(i,j)\in E}\mid \mid {\beta }_i\mid -\mid {\beta }_j\mid \mid ,$$ (11)

where the grouping penalty Σ_(i;j)∈E ||β_i| − |β_j|| controls only magnitudes of differences of coefficients ignoring their signs over the graph. Through the l₁ regularizer and grouping penalty, simultaneous feature grouping and selection are performed, where only large coefficients as well as pairwise differences are shrunk.

A computational method for the non-convex optimization in Eq. (11) is through DC programming. We will first give a brief review of DC programming.

A particular DC program on ^p takes the form of

$$f(\mathit{\beta })=f_1(\mathit{\beta })-f_2(\mathit{\beta })$$

with f₁(β) and f₂(β) being convex on ^p. Algorithms to solve DC programming based on the duality and local optimality conditions have been introduced in [17]. Due to their local characteristic and the non-convexity of DC programming, these algorithms cannot guarantee the computed solution to be globally optimal. In general, these DC algorithms converge to a local solution, but some researchers observed that they converge quite often to a global one [16].

To apply DC programming to our problem we need to decompose the objective function into the difference of two convex functions. We propose to use:

$$\begin{array}{l}{f}_1(\mathit{\beta })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathit{\beta }\Vert }_1+{\lambda }_2{\sum }_{(i,j)\in E}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid ),\\ f_2(\mathit{\beta })={\lambda }_2{\sum }_{(i,j)\in E}(\mid {\beta }_i\mid +\mid {\beta }_j\mid ).\end{array}$$

The above DC decomposition is based on the following identity: ||β_i| − |β_j|| = |β_i + β_j| + |β_i − β_j| − (|β_i| + |β_j|). Note that both f₁(β) and f₂(β) are convex functions.

Denote $f_2^k(\mathit{\beta })=f_2({\mathit{\beta }}^k)+\langle \mathit{\beta }-{\mathit{\beta }}^k,\partial f_2({\mathit{\beta }}^k)\rangle$ as the affine minorization of f₂(β), where 〈·, ·〉 is the inner product. Then DC programming solves Eq. (11) by iteratively solving a sub-problem as follows:

$$\underset{\beta }{min}{f}_1(\mathit{\beta })-f_2^k(\mathit{\beta }).$$ (12)

Since 〈β^k, ∂f₂(β^k)〉 is constant, Eq. (12) can be rewritten as

$$\underset{\mathit{\beta }}{min}{f}_1(\mathit{\beta })-\langle \mathit{\beta },\partial f_2({\mathit{\beta }}^k)\rangle .$$ (13)

Let c^k = ∂f₂(β^k). Note that

$$c_i^k={\lambda }_2{d}_i\text{sign}({\beta }_i^k)\mathbb{I}({\beta }_i^k\ne 0),$$ (14)

where d_i is the degree of node i, and (·) is the indicator function. Hence, the formulation in Eq. (13) is

$${min}_{\mathit{\beta }}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{\Vert \mathit{\beta }\Vert }_1-{({\mathbf{c}}^k)}^T\mathit{\beta }+{\lambda }_2{\sum }_{(i,j)\in E}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid ),$$ (15)

which is convex. Note that the only differences between the problems in Eq. (2) and Eq. (15) are the linear term (c^k)^Tβand the second regularization parameter. Similar to GOSCAR, we can solve Eq. (15) using ADMM, which is equivalent to the following optimization problem:

$$\begin{array}{l}{min}_{\mathit{\beta },\mathbf{q},\mathbf{p}}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2-{({\mathbf{c}}^k)}^T\mathit{\beta }+{\lambda }_1{\Vert \mathbf{q}\Vert }_1+2{\lambda }_2{\Vert \mathbf{p}\Vert }_1\\ s.t\mathit{\beta }-\mathbf{q}=\mathbf{0},\mathbf{T}\mathit{\beta }-\mathbf{p}=\mathbf{0}.\end{array}$$ (16)

There is an additional linear term (c^k)^Tβ in updating β compared to Algorithm 1. Hence, we can use Algorithm 1 to solve Eq. (15) with a small change in updating β:

$$\mathbf{F}\mathit{\beta }-{\mathbf{b}}^s-{\mathbf{c}}^k=0.$$

where s represents the iteration number in Algorithm 1.

The key steps of ncFGS are shown in Algorithm 2.

Algorithm 2.

The ncFGS algorithm

3.2 Non-Convex Formulation II: ncTFGS

It is known that the bias of lasso is due to the looseness of convex relaxation of l₀ regularization. The truncated l₁ regularizer, a non-convex regularizer close to the l₀ regularizer, has been proposed to resolve the bias issue [23]. The truncated l₁ regularizer can recover the exact set of nonzero coefficients under a weaker condition, and has a smaller upper error bound than lasso [23]. Therefore, we propose a truncated grouping penalty to further reduce the estimation bias. The proposed formulation based on the truncated grouping penalty is

$${min}_{\mathit{\beta }}{f}_T(\mathit{\beta })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_1{p}_1(\mathit{\beta })+{\lambda }_2{p}_2(\mathit{\beta })$$ (17)

where

$$\begin{array}{l}{p}_1(\mathit{\beta })={\sum }_i{J}_{\tau }(\mid {\beta }_i\mid ),\\ p_2(\mathit{\beta })={\sum }_{(i,j)\in E}{J}_{\tau }(\mid \mid {\beta }_i\mid -\mid {\beta }_j\mid \mid ),\end{array}$$

and $J_{\tau }(x)=min(\frac{x}{\tau },1)$ is the truncated l₁regularizer, a surrogate of the l₀ function; τ is a non-negative tuning parameter. Figure 1 shows the difference between l₀ norm, l₁ norm and J_τ (|x|). When τ → 0, J_τ (|x|) is equivalent to the l₀ norm given by the number of nonzero entries of a vector. When τ ≥ |x|, τJ_τ (|x|) is equivalent to the l₁ norm of x.

Figure 1.

Example for l₀ norm (left), l₁ norm (middle), and J_τ (|x|) with $\tau =\frac{1}{8}$ (right).

Note that J_τ(||β_i| − |β_j||) can be decomposed as

$$J_{\tau }(\mid \mid {\beta }_i\mid -\mid {\beta }_j\mid \mid )=\frac{1}{\tau }(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid )-\frac{1}{\tau }max(2\mid {\beta }_i\mid -\tau ,2\mid {\beta }_j\mid -\tau ,\mid {\beta }_i\mid +\mid {\beta }_j\mid ),$$

and a DC decomposition of J_τ(|β_i|) is

$$J_{\tau }(\mid {\beta }_i\mid )=\frac{1}{\tau }\mid {\beta }_i\mid -\frac{1}{\tau }max(\mid {\beta }_i\mid -\tau ,0).$$

Hence, the DC decomposition of f_T (β) can be written as

$$f_T(\mathit{\beta })=f_{T,1}(\mathit{\beta })-f_{T,2}(\mathit{\beta }),$$

where

$$\begin{array}{l}{f}_{T,1}(\mathit{\beta })=\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+\frac{{\lambda }_1}{\tau }{\Vert \mathit{\beta }\Vert }_1+\frac{{\lambda }_2}{\tau }{\sum }_{(i,j)\in E}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid ),\\ f_{T,2}(\mathit{\beta })=\frac{{\lambda }_1}{\tau }{\sum }_{i}max(\mid {\beta }_i\mid -\tau ,0)+\frac{{\lambda }_2}{\tau }{\sum }_{(i,j)\in E}max(2\mid {\beta }_i\mid -\tau ,2\mid {\beta }_j\mid -\tau ,\mid {\beta }_i\mid +\mid {\beta }_j\mid ).\end{array}$$

Let ${\mathbf{c}}_T^k=\partial f_{T,2}({\mathit{\beta }}^k)$ be the subgradient of f_T;2 in the (k +1)-th iteration. We have

$$c_{T,i}^k=\text{sign}({\beta }_i^k)\left(\frac{{\lambda }_1}{\tau }\mathbb{I}(\mid {\beta }_i^k\mid >\tau )+\frac{{\lambda }_2}{\tau }{\sum }_{j:(i,j)\in E}(2\mathbb{I}(\mid {\beta }_j^k\mid <\mid {\beta }_i^k\mid -\tau )+\mathbb{I}(\mid \mid {\beta }_i^k\mid -\mid {\beta }_j^k\mid \mid <\tau ))\right).$$ (18)

Then the subproblem of ncTFGS is

$${min}_{\mathit{\beta }}\frac{1}{2}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+\frac{{\lambda }_1}{\tau }{\Vert \mathit{\beta }\Vert }_1-{({\mathbf{c}}_T^k)}^T\mathit{\beta }+\frac{{\lambda }_2}{\tau }{\sum }_{(i,j)\in E}(\mid {\beta }_i+{\beta }_j\mid +\mid {\beta }_i-{\beta }_j\mid ),$$ (19)

which can be solved using Algorithm 1 as in ncFGS.

The key steps of ncTFGS are summarized in Algorithm 3.

Algorithm 3.

The ncTFGS algorithm

ncTFGS is an extension of ncFGS. When τ ≥ |β_i|, ∀i, ncT-FGS with regularization parameters τλ₁ and τλ₂ is identical to ncFGS (see Figure 3). ncFGS and ncTFGS have the same time complexity. The subproblems of ncFGS and ncTFGS are solved by Algorithm 1. In our experiments, we observed ncFGS and ncTFGS usually converge in less than 10 iterations.

Figure 3.

MSEs (left), s₀ (middle), and s (right) of ncFGS and ncTFGS on dataset 1 for fixed λ₁ and λ₂. The regularization parameters for ncTFGS are τλ₁ and τλ₂. τ ranges from 0.04 to 4.

4. NUMERICAL RESULTS

We examine the performance of the proposed methods and compare them against lasso, GFlasso, and OSCAR on synthetic datasets and two real datasets: FDG-PET images¹ and Breast Cancer². The experiments are performed on a PC with dual-core Intel 3.0GHz CPU and 4GB memory. The code is written in MATLAB. The algorithms and their associated penalties are:

• Lasso: λ₁||β||₁;

• OSCAR: λ₁||β||₁ + λ₂ Σ_i<j max{|β_i|, |β_j|};

• GFlasso: λ₁||β||₁ + λ₂ Σ _(i;j)∈E |β_i − sign(r_ij)β_j|;

• GOSCAR: λ₁||β||₁ + λ₂ Σ _(i;j)∈E max{|β_i|, |β_j|};

• ncFGS: λ₁||β||₁ + λ₂ Σ _{(i;j) ∈E} ||β_i| − |β_j||;

• ncTFGS: λ₁ Σ_i J_τ(|β_i|) + λ₂ Σ_{(i;j) ∈E} J_τ(||β_i| − |β_j||);

4.1 Efficiency

To evaluate the efficiency of the proposed methods, we conduct experiments on a synthetic dataset with a sample size of 100 and dimensions varying from 100 to 3000. The regression model is y = Xβ+ ε, where X ~ (0, I_p×p), β_i ~ (0, 1), and ε_i ~ (0, 0.01²). The graph is randomly generated. The number of edges n_e varies from 100 to 3000. The regularization parameters are set as λ₁ = λ₂ = 0.8 max{|β_i|} with n_e fixed. Since the graph size affects the penalty, λ₁ and λ₂ are scaled by $\frac{1}{n_e}$ to avoid trivial solutions with dimension p fixed. The average computational time based on 30 repetitions is reported in Figure 2. As can be seen in Figure 2, GOSCAR can achieve 1e-4 precision in less than 10s when the dimension and the number of edges are 1000. The computational time of ncTFGS is about 7 times higher than that of GOSCAR in this experiment. The computational time of ncFGS is the same as that of ncTFGS when τ = 100, and very close to that of ncTFGS when τ = 0.15. We can also observe that the proposed methods scale very well to the number of edges. The computational time of the proposed method increases less than 4 times when the number of edges increases from 100 to 3000. It is not surprising because the time complexity of each iteration in Algorithm 1 is linear with respect to n_e, and the sparsity of T makes the algorithm much more efficient. The increase of dimension is more costly than that of the number of edges, as the complexity of each iteration is quadratic with respect to p.

Figure 2.

Comparison of GOSCAR, ncFGS, ncT-FGS (τ = 0.15), and ncTFGS (τ = 100) in terms of computation time with different dimensions, precisions and the numbers of edges (in seconds and in logarithmic scale).

4.2 Simulations

We use five synthetic problems that have been commonly used in the sparse learning literature [2, 10] to compare the performance of different methods. The data is generated from the regression model y = Xβ+ ε, ε_i ~ (0, σ²). The five problems are given by:

1. n = 100, p = 40, and σ = 2, 5, 10. The true parameter is given by

   $$\mathit{\beta }={(\underset{10}{\underbrace{0,\dots ,0}},\underset{10}{\underbrace{2,\dots ,2}},\underset{10}{\underbrace{0,\dots ,0}},\underset{10}{\underbrace{2,\dots ,2}})}^T.$$

   X ~ (0, S_p×p) with s_ii = 1, ∀i and s_ij = 0.5 for i ≠ j.

2. n = 50, p = 40, $\mathit{\beta }={(\underset{15}{\underbrace{3,\dots ,3}},\underset{25}{\underbrace{0,\dots ,0}})}^T$, and σ = 2, 5, 10. The features are generated as

   $$\begin{array}{l}{\mathbf{x}}_i=Z_1+{\epsilon }_i^x,Z_1~\mathcal{N}(0,1),i=1,\dots ,5\\ {\mathbf{x}}_i=Z_2+{\epsilon }_i^x,Z_2~\mathcal{N}(0,1),i=6,\dots ,10\\ {\mathbf{x}}_i=Z_3+{\epsilon }_i^x,Z_3~\mathcal{N}(0,1),i=11,\dots ,15\\ {\mathbf{x}}_i~\mathcal{N}(0,1)i=16,\dots ,40\end{array}$$

   with ${\epsilon }_i^x~\mathcal{N}(0,0.16)$, and X = [x₁, …, x₄₀].

3. Consider a regulatory gene network [10], where an entire network consists of n_TF subnetworks, each with one transcription factor (TF) and its 10 regulatory target genes. The data for each subnetwork can be generated as ${\mathbf{X}}_i^{TF}~\mathcal{N}(0,{\mathbf{S}}_{11\times 11})$ with s_ii = 1, s_1i = s_i1 = 0.7, ∀i, i ≠ 1 and s_ij = 0 for i ≠ j, j ≠ 1, i ≠ 1. Then $\mathbf{X}=[{\mathbf{X}}_1^{TF},\dots ,{\mathbf{X}}_{n_{TF}}^{TF}]$, n = 100, p = 110, and σ = 5. The true parameters are

   $$\mathit{\beta }={(\underset{11}{\underbrace{\frac{5}{\sqrt{11}},\dots ,\frac{5}{\sqrt{11}}}},\underset{11}{\underbrace{\frac{-3}{\sqrt{11}},\dots ,\frac{-3}{\sqrt{11}}}},\underset{p-22}{\underbrace{0,\dots ,0}})}^T.$$
4. Same as 3 except that

   $$\mathit{\beta }={(5,\underset{10}{\underbrace{\frac{5}{\sqrt{10}},\dots ,\frac{5}{\sqrt{10}}}},-3,\underset{10}{\underbrace{\frac{-3}{\sqrt{10}},\dots ,\frac{-3}{\sqrt{10}}}},\underset{p-22}{\underbrace{0,\dots ,0}})}^T$$
5. Same as 3 except that

   $$\mathit{\beta }={(5,\underset{10}{\underbrace{\frac{5}{\sqrt{10}},\dots ,\frac{5}{\sqrt{10}}}},-5,\underset{10}{\underbrace{\frac{-5}{\sqrt{10}},\dots ,\frac{-5}{\sqrt{10}}}},3,\underset{10}{\underbrace{\frac{3}{\sqrt{10}},\dots ,\frac{3}{\sqrt{10}}}},-3,\underset{10}{\underbrace{\frac{-3}{\sqrt{10}},\dots ,\frac{-3}{\sqrt{10}}}}\underset{p-44}{\underbrace{0,\dots ,0}})}^T$$

We assume that the features in the same group are connected in a graph, and those in different groups are not connected. We use MSE to measure the performance of estimation of β, which is defined as

$$\mathit{MSE}(\mathit{\beta })={(\mathit{\beta }-{\mathit{\beta }}^{\ast })}^T{\mathbf{X}}^T\mathbf{X}(\mathit{\beta }-{\mathit{\beta }}^{\ast }).$$

For feature grouping and selection, we introduce two separate metrics to measure the accuracy of feature grouping and selection. Denote I_i, i = 0, 1, 2, …, K as the index of different groups, where I₀ is the index of zero coefficients. Then the metric for feature selection is defined as

$$s_0=\frac{{\sum }_{i\in I_0}\mathbb{I}({\beta }_i=0)+{\sum }_{i\notin I_0}\mathbb{I}({\beta }_i\ne 0)}{p},$$

and the metric for feature grouping is defined as

$$s=\frac{{\sum }_{i=1}^K{s}_i+s_0}{K+1},$$

where

$$s_i=\frac{{\sum }_{i\ne j,i,j\in I_i}\mathbb{I}(\mid {\beta }_i\mid =\mid {\beta }_j\mid )+{\sum }_{i\ne j,i\in I_i,j\notin I_i}\mathbb{I}(\mid {\beta }_i\mid \ne \mid {\beta }_j\mid )}{\mid I_i\mid (p-1)}.$$

s_i measures the grouping accuracy of group i under the assumption that the absolute values of entries in the same group should be the same, but different from those in different groups. s₀ measures the accuracy of feature selection. It is clear that 0 ≤ s₀, s_i, s ≤ 1.

For each dataset, we generate n samples for training, as well as n samples for testing. To make the synthetic datasets more challenging, we first randomly select ⌊n/2⌋ coefficients, and change their signs, as well as those of the corresponding features. Denote β̃ and X̃ as the coefficients and features after changing signs. Then β̃_i = −β_i, x̃_i = −x_i, if the i-th coefficient is selected; otherwise, β̃_i = β_i, x̃_i = x_i. So that X̃β̃ = Xβ. We apply different approaches on X̃. The covariance matrix of X is used in GFlasso to simulate the graph misspecification. The results of β converted from β̃ are reported.

Figure 3 shows that ncFGS obtains the same results as ncTFGS on dataset 1 with σ = 2 when τ is larger than |β_i|. The regularization parameters are τλ₁ and τλ₂ for ncTFGS, and λ₁ and λ₂ for ncFGS. Figure 4 shows the average nonzero coefficients obtained on dataset 1 with σ = 2. As can be seen in Figure 4, GOSCAR, ncFGS, and ncTFGS are able to utilize the graph information, and achieve good parameter estimation. Although GFlasso can use the graph information, it performs worse than GOSCAR, ncFGS, and ncTFGS due to the graph misspecification.

Figure 4.

The average nonzero coefficients obtained on dataset 1 with σ = 2: (a) Lasso; (b) GFlasso; (c) OSCAR; (d) GOSCAR; (e); ncFGS; (f) ncTGS

The performance in terms of MSEs averaged over 30 simulations is shown in Table 1. As indicated in Table 1, among existing methods (Lasso, GFlasso, OSCAR), GFlasso is the best, except in the two cases where OSCAR is better. GOSCAR is better than the best existing method in all cases except for two, and ncFGS and ncTFGS outperform all the other methods.

Table 1.

Comparison of performance in terms of MSEs and estimated standard deviations (in parentheses) for different methods based on 30 simulations on different synthetic datasets.

Datasets	Lasso	OSCAR	GFlasso	GOSCAR	ncFGS	ncTFGS
Data1 (σ = 2)	1.807 (0.331)	1.441 (0.318)	1.080 (0.276)	0.315 (0.157)	0.123 (0.075)	0.116 (0.075)
Data1 (σ = 5)	5.294 (0.983)	5.328 (1.080)	3.480 (1.072)	1.262 (0.764)	0.356 (0.395)	0.356 (0.395)
Data1 (σ = 10)	12.628 (3.931)	13.880 (4.031)	13.411 (4.540)	6.061 (4.022)	1.963 (1.600)	1.958 (1.593)
Data2 (σ = 2)	1.308 (0.435)	1.084 (0.439)	0.623 (0.250)	0.291 (0.208)	0.226 (0.175)	0.223 (0.135)
Data2 (σ = 5)	4.907 (1.496)	4.868 (1.625)	2.538 (0.656)	0.781 (0.598)	0.721 (0.532)	0.705 (0.535)
Data2 (σ = 10)	18.175 (6.611)	18.353 (6.611)	6.930 (2.858)	4.601 (2.623)	4.232 (2.561)	4.196 (2.577)
Data3 (σ = 5)	5.163 (1.708)	4.503 (1.677)	4.236 (1.476)	3.336 (1.725)	0.349 (0.282)	0.348 (0.283)
Data4 (σ = 5)	7.664 (2.502)	7.167 (2.492)	7.516 (2.441)	7.527 (2.434)	5.097 (0.780)	4.943 (0.764)
Data5 (σ = 5)	9.893 (1.965)	7.907 (2.194)	9.622 (2.025)	9.810 (2.068)	7.684 (1.1191)	7.601 (1.038)

Table 2 shows the results in terms of accuracy of feature grouping and selection. Since Lasso does not perform feature grouping, we only report the results of the other five methods: OSCAR, GFlasso, GOSCAR, ncFGS, and ncT-FGS. Table 2 shows that ncFGS and ncTFGS achieve higher accuracy than other methods.

Table 2.

Accuracy of feature grouping and selection based on 30 simulations for five feature grouping methods: the first row for each dataset corresponds to the accuracy of feature selection; the second row corresponds to the accuracy of feature grouping. The numbers in parentheses are the standard deviations.

Datasets	OSCAR	GFlasso	GOSCAR	ncFGS	ncTFGS
Data1 (σ = 2)	0.675 (0.098)	0.553 (0.064)	0.513 (0.036)	0.983 (0.063)	1.000 (0.000)
	0.708 (0.021)	0.709 (0.017)	0.702 (0.009)	0.994 (0.022)	1.000 (0.000)
Data1 (σ = 5)	0.565 (0.084)	0.502 (0.009)	0.585 (0.085)	1.000 (0.000)	1.000 (0.000)
	0.691 (0.011)	0.709 (0.016)	0.708 (0.017)	1.000 (0.000)	1.000 (0.000)
Data1 (σ = 10)	0.532 (0.069)	0.568 (0.088)	0.577 (0.061)	0.983 (0.063)	1.000 (0.000)
	0.675 (0.031)	0.725 (0.022)	0.708 (0.020)	0.994 (0.022)	0.999 (0.001)
Data2 (σ = 2)	0.739 (0.108)	0.544 (0.272)	1.000 (0.000)	0.958 (0.159)	0.958 (0.159)
	0.625 (0.052)	0.823 (0.029)	0.837 (0.014)	0.831 (0.052)	0.846 (0.041)
Data2 (σ = 5)	0.763 (0.114)	0.717 (0.275)	0.999 (0.005)	0.979 (0.114)	0.975 (0.115)
	0.650 (0.066)	0.741 (0.062)	0.833 (0.011)	0.845 (0.030)	0.842 (0.037)
Data2 (σ = 10)	0.726 (0.101)	0.818 (0.149)	0.993 (0.024)	1.000 (0.000)	1.000 (0.000)
	0.597 (0.058)	0.680 (0.049)	0.829 (0.025)	0.851 (0.015)	0.856 (0.014)
Data3 (σ = 5)	0.886 (0.135)	0.736 (0.103)	0.382 (0.084)	0.992 (0.026)	0.996 (0.014)
	0.841 (0.056)	0.739 (0.041)	0.689 (0.013)	0.995 (0.017)	0.978 (0.028)
Data4 (σ = 5)	0.875 (0.033)	0.881 (0.026)	0.882 (0.037)	0.796 (0.245)	0.950 (0.012)
	0.834 (0.030)	0.805 (0.035)	0.805 (0.036)	0.895 (0.114)	0.890 (0.074)
Data5 (σ = 5)	0.760 (0.203)	0.802 (0.153)	0.861 (0.051)	0.881 (0.174)	0.894 (0.132)
	0.858 (0.031)	0.821 (0.037)	0.805 (0.037)	0.920 (0.056)	0.919 (0.057)

Table 3 shows the comparison of feature selection alone (λ₂ = 0), feature grouping alone (λ₁ = 0), and simultaneous feature grouping and selection using ncTFGS. From Table 3, we can observe that simultaneous feature grouping and selection outperforms either feature grouping or feature selection, demonstrating the benefit of joint feature grouping and selection in the proposed non-convex method.

Table 3.

Comparison of feature selection alone (FS), feature grouping alone (FG), and simultaneous feature grouping and feature selection (Both). The average results based on 30 replications of three datasets with σ = 5: Data3 (top), Data4 (middle), and Data5 (bottom) are reported. The numbers in parentheses are the standard deviations.

Meth.	MSE	s0	s
FG	2.774 (0.967)	0.252 (0.156)	0.696 (0.006)
FS	6.005 (1.410)	0.945 (0.012)	0.773 (0.037)
Both	0.348 (0.283)	0.996 (0.014)	0.978 (0.028)
FG	9.4930 (1.810)	0.613 (0.115)	0.770 (0.038)
FS	6.437 (1.803)	0.947 (0.016)	0.782 (0.046)
Both	4.944 (0.764)	0.951 (0.166)	0.890 (0.074)
FG	10.830 (2.161)	0.434 (0.043)	0.847 (0.014)
FS	10.276 (1.438)	0.891 (0.018)	0.768 (0.026)
Both	7.601 (1.038)	0.894 (0.132)	0.919 (0.057)

4.3 Real Data

We conduct experiments on two real datasets: FDG-PET and Breast Cancer. The metrics to measure the performance of different algorithms include accuracy (acc.), sensitivity (sen.), specificity (spe.), degrees of freedom (dof.), and the number of nonzero coefficients (nonzero coeff.). The dof. of lasso is the number of nonzero coefficients [18]. For the algorithms capable of feature grouping, we use the same definition of dof. in [2], which is the number of estimated groups.

4.3.1 FDG-PET

In this experiment, we use FDG-PET 3D images from 74 Alzheimer’s disease (AD), 172 mild cognitive impairment (MCI), and 81 normal control (NC) subjects downloaded from the Alzheimer’s disease neuroimaging initiative (ADNI) database. The different regions of whole brain volume can be represented by 116 anatomical volumes of interest (AVOI), defined by Automated Anatomical Labeling (AAL) [20]. Then we extracted data from each of the 116 AVOIs, and derived average of each AVOI for each subject.

In our study, we compare different methods in distinguishing AD and NC subjects, which is a two-class classification problem over a dataset with 155 samples and 116 features. The dataset is randomly split into two subset, one training set consisting of 104 samples, and one testing set consisting of the remaining 51 samples. The tuning of the parameter is achieved by 5-fold cross validation. Sparse inverse covariance estimation (SICE) has been recognized as an effective tool for identifying the structure of the inverse covariance matrix. We use SICE developed in [6] to model the connectivity of brain regions. Figure 5 shows sample subgraphs built by SICE consisting of 115 nodes and 265 edges.

Figure 5.

Subgraphs of the graph built by SICE on FDG-PET dataset, which consists of 265 edges.

The results based on 20 replications are shown in Table 4. From Table 4, we can see that ncTFGS achieves more accurate classification while obtaining smaller degrees of freedom. ncFGS and GOSCAR achieve similar classification, while ncFGS selects more features than GOSCAR.

Table 4.

Comparison of classification accuracy, sensitivity, specificity, degrees of freedom, and the number of nonzero coefficients averaged over 20 replications for different methods on FDG-PET dataset.

Metrics	Lasso	OSCAR	GFlasso	GOSCAR	ncFGS	ncTFGS
acc.	0.886 (0.026)	0.891 (0.026)	0.901 (0.029)	0.909 (0.026)	0.909 (0.031)	0.920 (0.024)
sen.	0.870 (0.041)	0.876 (0.038)	0.904 (0.038)	0.909 (0.041)	0.915 (0.043)	0.933 (0.047)
pec.	0.913 (0.0446)	0.917 (0.050)	0.902 (0.046)	0.915 (0.046)	0.908 (0.047)	0.915 (0.052)
dof.	22.150	29.150	24.350	21.300	31.250	18.250
nonzero coeff.	22.150	38.000	41.900	24.250	37.350	19.350

Figure 6 shows the comparison of accuracy with either λ₁ or λ₂ fixed. The λ₁ and λ₂ values range from 1e-4 to 100. As we can see, the performance of ncTFGS is slightly better than that of the other competitors. Since the regularization parameters of subproblems in ncTFGS are $\frac{{\lambda }_1}{\tau }$ and $\frac{2{\lambda }_2}{\tau }$, the solution of ncTFGS is more sparse than those of other competitors when λ₁ and λ₂ are large and τ is small (τ = 0.15 in this case).

Figure 6.

Comparison of accuracies for various methods with λ₁ fixed (left) and λ₂ fixed (right) on FDT-PET dataset.

4.3.2 Breast Cancer

We conduct experiments on the breast cancer dataset, which consists of gene expression data for 8141 genes in 295 breast cancer tumors (78 metastatic and 217 non-metastatic). The network described in [4] is used as the input graph in this experiment. Figure 7 shows a subgraph consisting of 80 nodes of the used graph. We restrict our analysis to the 566 genes most correlated to the output, but also connected in the graph. 2/3 data is randomly chosen as training data, and the remaining 1/3 data is used as testing data. The tuning parameter is estimated by 5-fold cross validation. Table 5 shows the results averaged over 30 replications. As indicated in Table 5, GOSCAR, ncFGS and ncTFGS outperform the other three methods, and ncTFGS achieves the best performance.

Figure 7.

A subgraph of the network in Breast Cancer dataset [4]. The subgraph consists of 80 nodes.

Table 5.

Comparison of classification accuracy, sensitivity, specificity, degrees of freedom, and the number of nonzero coefficients averaged over 30 replications for various methods on Breast Cancer dataset.

Metrics	Lasso	OSCAR	GFlasso	GOSCAR	ncFGS	ncTFGS
acc.	0.739 (0.054)	0.755 (0.055)	0.771 (0.050)	0.783 (0.042)	0.779 (0.041)	0.790 (0.036)
sen.	0.707 (0.056)	0.720 (0.060)	0.749 (0.060)	0.755 (0.050)	0.755 (0.055)	0.762 (0.044)
pec.	0.794 (0.071)	0.810 (0.068)	0.805 (0.056)	0.827 (0.061)	0.819 (0.058)	0.834 (0.060)
dof.	239.267	165.633	108.633	70.267	57.233	45.600
nonzero coeff.	239.267	243.867	144.867	140.667	79.833	116.567

5. CONCLUSION

In this paper, we consider simultaneous feature grouping and selection over a given undirected graph. We propose one convex and two non-convex penalties to encourage both sparsity and equality of absolute values of coefficients for features connected in the graph. We employ ADMM and DC programming to solve the proposed formulations. Numerical experiments on synthetic and real data demonstrate the effectiveness of the proposed methods. Our results also demonstrate the benefit of simultaneous feature grouping and feature selection through the proposed non-convex methods. In this paper, we focus on undirected graphs. A possible future direction is to extend the formulations to directed graphs. In addition, we plan to study the generalization performance of the proposed formulations.