Grouping pursuit through a regularization solution surface

Summary

Extracting grouping structure or identifying homogenous subgroups of predictors in regression is crucial for high-dimensional data analysis. A low-dimensional structure in particular–grouping, when captured in a regression model, enables to enhance predictive performance and to facilitate a model's interpretability Grouping pursuit extracts homogenous subgroups of predictors most responsible for outcomes of a response. This is the case in gene network analysis, where grouping reveals gene functionalities with regard to progression of a disease. To address challenges in grouping pursuit, we introduce a novel homotopy method for computing an entire solution surface through regularization involving a piecewise linear penalty. This nonconvex and overcomplete penalty permits adaptive grouping and nearly unbiased estimation, which is treated with a novel concept of grouped subdifferentials and difference convex programming for efficient computation. Finally, the proposed method not only achieves high performance as suggested by numerical analysis, but also has the desired optimality with regard to grouping pursuit and prediction as showed by our theoretical results.

1 Introduction

Essential to high-dimensional data analysis is seeking a certain lower-dimensional structure in knowledge discovery, as in web mining. Extracting one-kind of lower-dimensional structure–grouping, remains largely unexplored in regression. In gene network analysis, a large amount of current genetic knowledge has been organized in terms of networks, for instance, the Kyoto Encyclopedia of Genes and Genomes (KEGG), a collection of manually drawn pathway maps representing the knowledge about molecular interactions and reactions. In situations as such, extracting homogenous subnetworks from a network of dependent predictors, most responsible for predicating outcomes of a response, has been one key challenge of biomedical research. There homogenous subnetworks of genes are usually estimated for understanding a disease's progression. The central issue this article addresses is automatic identification of homogenous subgroups in regression, what we call grouping pursuit.

Now consider a linear model in which response Y_i depends on a vector of p predictors:

$$Y_i\equiv \mu ({\mathit{\text{x}}}_i)+{\mathit{\epsilon }}_i,\mu (\mathit{\text{x}})\equiv {\mathit{\text{x}}}^T\mathit{\beta }=\sum _{j=1}^p{x}_j{\beta }_j,E({\epsilon }_i)=0,\mathit{\text{Var}}({\epsilon }_i)={\sigma }^2;i=1,\dots ,n,$$ (1)

where β ≡ (β₁, …, β_p)^T is a vector of regression coefficients, x_i is independent of ε_i, and μ(x) is in a generic form, including linear, and nonlinear predictors expressed in terms of linear combinations of known bases. Our objective is to identify all possible homogenous subgroups of predictors, for optimal prediction of the outcome of Y. Here homogeneity means that regression coefficients are of similar (same) values, that is, β_j1 ≈ ⋯ ≈ β_jK within each group {j₁, …, j_K} ⊂ {1, …, p}. In (1), grouping pursuit estimates all distinct values of β as well as all corresponding subgroups of homogenous predictors.

Grouping pursuit seeks variance reduction of estimation while retaining roughly the same amount of bias, which is advantageous in high-dimensional analysis. First, it collapses predictors whose sample covariances between the residual and predictors are of similar values, for best predicting outcomes of Y; c.f., Theorem 4. Moreover, it goes beyond the notion of feature selection. This is because it seeks not only a set of redundant predictors, or a single group of zero-coefficient predictors, but also additional homogenous subgroups for further variance reduction. As a result, it yields higher predictive performance. These aspects are confirmed by numerical and theoretical results in Sections 4 and 5. Second, the price to be paid for adaptive grouping pursuit is estimation of tuning parameters, which is small as compared to its potential gain of a simpler model with higher predictive accuracy.

Grouping pursuit considered in here is one kind of supervised clustering. Papers that investigate groping pursuit are those of Tibshirani et al. (2005), where the Fused Lasso is proposed using an L₁-penalty with respect to a certain serial order; Bondell and Reich (2008), where the OSCAR penalty involves pairwise L_∞-penalties for grouping variables in terms of absolute values, in addition to variable selection. Grouping pursuit dramatically differs from feature selection for grouped predictors. This is in the sense that the former only groups predictors without removing redundancy, whereas the latter removes redundancy by encouraging grouped predictors stay together in selection; see Yuan and Lin (2006), and Zhao, Rocha and Yu (2009).

Our primary objective is achieving high accuracy in both grouping and prediction through a computationally efficient method, which seems to be difficult, if not impossible, with existing methods, especially those through enumeration. To achieve our objective, we employ the regularized least squares method with a piecewise linear nonconvex penalty. The penalty to be introduced in (2) involves one thresholding parameter determining which pairs to be shrunk towards a common group, which works jointly with one regularization parameter for shrinkage towards unknown location. These two tuning parameters combine thresholding with shrinkage for achieving adaptive grouping, which is otherwise not possible with shrinkage alone. The penalty is overcomplete in that the number of individual penalty terms in the penalty may be redundant with regard to certain grouping structures, and is continuous but with three nondifferentiable points, leading to significant computational advantage, in addition to the desired optimality for grouping pursuit (Theorems 3 and Corollary 1).

Computationally, the proposed penalty imposes great challenges in two aspects: (a) potential discontinuities and (b) overcompleteness of the penalty, where an effective treatment does not seem to exist in the literature; see Friedman et al. (2007) about computational challenges for a pathwise coordinate method in this type of situation. To meet the challenges, we design a novel homotopy algorithm to compute the regularization solution surface. The algorithm uses a novel concept of grouped subdifferentials to deal with overcompleteness for tracking the process of grouping, and difference convex (DC) programming to treat discontinuities due to nonconvex minimization. This, together with a model selection routine for estimators that can be discontinuous, permits adaptive grouping pursuit.

Theoretically, we derive a finite-sample probability error bound of our DC estimator, what we call DCE, computed from the homotopy algorithm for grouping pursuit. On this basis, we prove that DCE is consistent with regard to grouping pursuit as well as reconstructing the unbiased least squares estimate under the true grouping, roughly for nearly exponentially many predictors in n as long as $\frac{logp}{n}\to 0$, c.f., Theorem 3 for details.

For subnetwork analysis, we apply our proposed method to study predictability of a protein-protein interaction (PPI) network of genes on the time to breast cancer metastasis through gene expression profiles. In Section 5.2, 27 homogenous subnetworks are identified through a Laplacian network weight vector, which surround three tumor suppressor genes TP53, BRACA1 and BRACA2 for metastasis. There 17 disease genes that were identified in the study of Wang et al. (2005) belong to 5 groups containing 1, 1, 1, 1, and 13 disease genes, indicating gene functionalities with regard to breast cancer survivability.

This article is organized in seven sections. Section 2 introduces the proposed method and the homotopy algorithm. Section 3 is devoted to selection of tuning parameter. Section 4 presents a theory concerning optimal properties of DCE in grouping pursuit and prediction, followed by some numerical examples and an application to breast cancer data in Section 5. Section 6 discusses the proposed method. Finally, the appendix contains technical proofs.

2 Grouping pursuit

In (1), let the true coefficient vector ${\mathit{\beta }}^0={({\beta }_1^0,\dots ,{\beta }_p^0)}^T$ be ${({\alpha }_1^0{\mathbf{1}}_{|{\mathcal{G}}_1^0|}^T,\dots ,{\alpha }_{K^0}^0{\mathbf{1}}_{|{\mathcal{G}}_{K^0}^0|}^T)}^T$, where K⁰ is the number of distinct groups, ${\alpha }_1^0<\cdots <{\alpha }_{K^0}^0$, and ${\mathbf{1}}_{|{\mathcal{G}}_1^0|}$ denotes a vector of 1's with length $|{\mathcal{G}}_1^0|$. Grouping pursuit, as defined early, estimates true grouping ${\mathcal{G}}^0=({\mathcal{G}}_1^0,\dots ,{\mathcal{G}}_{K^0}^0)$ as well as ${\mathit{\alpha }}^0={({\alpha }_1^0,\dots ,{\alpha }_{K^0}^0)}^T$. Without loss of generality, assume that the response and predictors are centered, that is, Y^T1 = 0 and (x_1j, …, x_nj)1 = 0; j = 1, …, p.

Ideally, one may enumerate over all possible least squares regressions for identifying the best grouping. However, the total number of all possible groupings, which is the pth Bell number (Rota, 1964), is much larger than that of all possible subsets in feature selection, hence that it is computationally infeasible even for moderate p. For instance, the 10th order Bell number is 115975. To circumvent this difficulty, we develop an automatic nonconvex regularization method to obtain (1) accurate grouping, (2) the least squares estimate based on the true grouping, (3) an efficient homotopy algorithm, and (4) high predictive performance.

Our approach utilizes a penalty involving pairwise comparisons: {β_j − β_j′ : 1 ≤ j < j′ ≤ p}. When β_j − β_j′ = 0, X_j and X_j′ are grouped. By transitivity, that is, β_j1 − β_j2 = 0 and β_j2 − β_j3 = 0 imply that β_j1 = β_j2 = β_j3, we identify all homogenous groups through p(p − 1)/2 comparisons. Naturally, these comparisons can be conducted through penalized least squares with penalty Σ_j<j′ |β_j − β_j′|. However, this convex penalty is not desirable for predictive performance, because it is not adaptive for discriminating large from small pairwise differences. As a result, overpenalizing large differences due to shrinking small differences towards zero impedes predictive performance. We thus introduce its nonconvex counterpart J(β) = Σ_j<j′ G(β_j − β_j′) for adaptive grouping pursuit, where G(z) = λ₂ if |z| > λ₂ and G(z) = |z| otherwise, with λ₂ > 0 being the thresholding parameter. For G(z), one locally convex and two locally concave points at z = 0, ±λ₂ enable us to achieve computational advantage, as well as to realize sharp statistical properties. First, the piecewise linearity and the two locally concave points of G(z) yield an efficient method (Algorithm 1), and fast finite-step convergence of the surface algorithm (Algorithm 2). Second, they yield a sharp finite-sample error bound in Theorem 3. These aspects are unique for G(z), which may not be shared by other penalties such as SCAD (Fan and Li, 2001); see the discussion after Theorem 2. A function like G(z) was considered in other contexts such as wavelet denoising (Fan, 1997) and a combined L₀ and L₁ penalty via integer programming (Liu and Wu, 2007).

We now propose our penalized least squares criterion for automatic grouping pursuit:

$$S(\mathit{\beta })=\frac{1}{2n}\sum _{i=1}^n{(Y_i-{\mathit{\text{x}}}_i^T\mathit{\beta })}^2+{\mathrm{\lambda }}_{1}J(\mathit{\beta }),J(\mathit{\beta })=\sum _{j<j\prime }G({\beta }_j-{\beta }_{j\prime }),$$ (2)

where λ₁ > 0 is the regularization parameter controlling the degree of grouping. For (2), any local/global minimizer can not attain at any of non-smooth locally concave points of J(β).

Lemma 1 Let h(·) be any differentiable function in ℝ^p and $\mathit{\beta }\ast ={({\beta }_1^{\ast },\dots ,{\beta }_p^{\ast })}^T$ be a local minimizer of f(β) = h(β) + λ₁J(β) with J(·) given in (2). Then $|{\beta }_j^{\ast }-{\beta }_{j\prime }^{\ast }|\ne {\mathrm{\lambda }}_2$ for j ≠ j′.

2.1 Grouped subdifferentials

We now introduce a novel concept of grouped subdifferentials for a convex function, which constitutes a basis of our homotopy algorithm for tracking the process of grouping.

A subgradient of a convex function f(β) with respect to β at β is any vector b ∈ ℝ^p satisfying f(β*) ≥ f(β) + b^T(β* − β) for any β* with sufficiently small β* − β, and reduces to the derivative at a smooth point. The subdifferential of S(β) at any β is the set of all such b's, which is either a singleton or a non-singleton compact set. Let β̂ be a local minimizer of (2) and (₁, …, _K) be the corresponding grouping, where K is the number of distinct groups. The subgradient of |β_j − β_j′| with respect to β_j at β = β̂ is given by b_jj′ = Sign(β̂_j − β̂_j′) if |β̂_j − β̂_j′| > 0, and |b_jj′| ≤ 1 otherwise. Here b_jj′ is a singleton everywhere except at β̂_j − β̂_j′ = 0.

To proceed, write β̂ as ${({\widehat{\alpha }}_1{\mathbf{1}}_{|{\mathcal{G}}_1|}^T,\dots ,{\widehat{\alpha }}_K{\mathbf{1}}_{|{\mathcal{G}}_K|}^T)}^T$, where the index in each group is arranged increasingly, α̂₁ < ⋯ < α̂_K. Define g(j) ≡ k if β̂_j = α̂_k; j = 1, …, p, mapping indices from β̂ to α̂. Then grouping (₁, …, _K) partitions index set {1, …, p}, with _k ≡ {j : g(j) = k}; k = 1, …, K.

Ordinarily, group splitting can be tracked through certain transition conditions for {b_jj′ : j ≠ j′}, c.f., Rosset and Zhu (2007). However, {b_jj′ : j ≠ j′} are not estimable from data when an overcomplete penalty is used. To overcome this difficulty, we define the grouped subgradient of index j ∈ _k at β = β̂ as B_j ≡ Σ_j′∈k\{j} b_jj′ if |_k| > 1, and B_j ≡ 0 if |_k| = 1; j = 1, …, p. Note that Σ_j∈k B_j = 0; k = 1, …, K, because b_jj′ = −b_j′j; j ≠ j′. Moreover, we define the grouped subgradient of a subset A ⊂ _k at β = β̂ as B_A ≡ Σ_j∈A B_j = Σ_{(j,j′)∈A×(k\A)} b_jj′. Then

$$|B_A|\le |A|(|{\mathcal{G}}_k|-|A|).$$ (3)

Subsequently, we work with {B_A : A ⊂ {1, …, p}} that can be uniquely determined; see Theorem 1.

2.2 Difference convex programming

This section treats non-differentiable nonconvex minimization (2) through DC programming, which is a principle for nonconvex minimization, relying on decomposing an objective function into a difference of two convex functions. The reader may consult An and Tao (1997) for DC programming. Through this DC method, we will design a novel homotopy algorithm for a DC solution of (2) in Section 2.3, which is a solution through DC programming.

First, we decompose S(β) in (2) into a difference of two convex functions $S_1(\mathit{\beta })=\frac{1}{2n}{\sum }_{i=1}^n{(Y_i-{\mathit{\text{x}}}_i^T\mathit{\beta })}^2+{\mathrm{\lambda }}_1{\sum }_{j<j\prime }|{\beta }_j-{\beta }_{j\prime }|$ and S₂(β) = λ₁ Σ_j<j′ G₂(β_j − β_j′), through a DC decomposition of G(·) = G₁(·) − G₂(·) with G₁(z) = |z| and G₂(z) = (|z| − λ₂)₊, where z₊ is the positive part of z. This DC decomposition is interpretable in that S₂(·) corrects the estimation bias due to use of convex penalty λ₁ Σ_j<j′ |β_j − β_j′| for a nonconvex problem (2).

Second, we construct a sequence of upper approximations by successively replacing S₂(β) at iteration m = 0, 1, …, by its affine minorization based on iteration m − 1, leading to an upper convex approximating function at iteration m:

$$S_1\left(\mathit{\beta }\right)-S_2\left({\widehat{\widehat{\mathit{\beta }}}}^{(m-1)}\left({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2\right)\right)-{\left(\mathit{\beta }-{\widehat{\widehat{\mathit{\beta }}}}^{(m-1)}\left({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2\right)\right)}^T\nabla S_2\left({\widehat{\widehat{\mathit{\beta }}}}^{(m-1)}\left({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2\right)\right),$$ (4)

where ∇ is the subgradient operator, ^(m−1) (λ₁, λ₂) is the minimizer of (4) at iteration m − 1, and ⁽⁻¹⁾ (λ₁, λ₂) ≡ 0. The last term in (4) becomes ${\mathrm{\lambda }}_1{\sum }_{j=1}^p\left({\beta }_j-{\widehat{\widehat{\beta }}}_j^{(m-1)}({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2)\right)\times {\sum }_{j\prime :j\prime \ne j}\nabla G_2\left({\widehat{\widehat{\beta }}}_j^{(m-1)}({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2)-{\widehat{\widehat{\beta }}}_{j\prime }^{(m-1)}({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2)\right)$ with ∇G₂(z) = Sign(z)I(|z| > λ₂) being a subgradient of G₂ at z.

Third, we utilize the grouped subdifferentials to track the entire solution surface iteratively. One technical difficulty is that ^(m) (λ₁, λ₂) would have jumps in λ₁ if ⁽⁰⁾ (λ₁, λ₂) were piecewise linear in λ₁ given λ₂, in view of (6) and (7) in Theorem 1. This is undesirable for tracking by continuity through homotopy. For grouping pursuit, we therefore replace ⁽⁰⁾ (λ₁, λ₂) in (4) by ⁽⁰⁾ (λ₀, λ₂), where rough tuning for λ₀ suffices because a DC algorithm is not sensitive to an initial value (An and Tao, 1997). This choice leads to a piecewise linear and continuous minimizer β̂⁽¹⁾(λ) of (4) in λ₁ given (λ₀, λ₂), where λ = (λ₀, λ₁, λ₂)^T. Successively replacing ^(m−1) (λ₁, λ₂) in (4) by β̂^(m−1) (λ₀, λ₀, λ₂) for m ∈ ℕ, we obtain a modified version of (4):

$$S^{(m)}(\mathit{\beta })=S_1(\mathit{\beta })-S_2\left({\widehat{\mathit{\beta }}}^{(m-1)}\left({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2\right)\right)-{\left(\mathit{\beta }-{\widehat{\mathit{\beta }}}^{(m-1)}\left({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2\right)\right)}^T\nabla S_2\left({\widehat{\mathit{\beta }}}^{(m-1)}\left({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2\right)\right),$$ (5)

which yields its minimizer β̂^(m)(λ) and the estimated grouping ^(m)(λ). As suggested by Theorems 1 and 2, β̂^(m)(λ) converges in finite steps. Most importantly, the iterative scheme yields an estimator having the desired properties of a global minimizer, c.f., Theorem 3.

Given grouping = (₁, …, _K), let Z = (z₁, …, z_K) be an n × K matrix with z_k = X_k 1, and X_k be the design matrix spanned by the predictors of _k; k = 1, …, K.

Theorem 1 Assume that ${\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}$ is invertible. Then β̂^(m)(λ) defined by (5) is piecewise linear in (Y, λ) and continuous in λ₁. In addition,

$${\widehat{\mathit{\alpha }}}^{\left(m\right)}(\mathbf{\lambda })\equiv {\left({\widehat{\alpha }}_1^{\left(m\right)}(\mathbf{\lambda }),\dots ,{\widehat{\alpha }}_{K^{\left(m\right)}(\mathbf{\lambda })}^{\left(m\right)}(\mathbf{\lambda })\right)}^T={\left({\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}\right)}^{-1}\left({\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T\mathit{\text{Y}}-n{\mathrm{\lambda }}_1{\mathit{\delta }}^{(m)}(\mathbf{\lambda })\right),$$ (6)

where ${\mathit{\delta }}^{(m)}(\mathbf{\lambda })\equiv {\left({\delta }_1^{(m)}(\mathbf{\lambda }),\dots ,{\delta }_{K^{(m)}(\mathbf{\lambda })}^{(m)}(\mathbf{\lambda })\right)}^T,{\delta }_k^{(m)}(\mathbf{\lambda })\equiv {\sum }_{j\in {\mathcal{G}}_k^{(m)}(\mathbf{\lambda })}{\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda })$, and

$${\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda })\equiv \sum _{j^{\prime }:j^{\prime }\ne j}\left\{\text{sign}\left({\widehat{\beta }}_j^{(m)}(\mathbf{\lambda })-{\widehat{\beta }}_{j^{\prime }}^{(m)}(\mathbf{\lambda })\right)-\nabla G_2\left({\widehat{\beta }}_j^{(m-1)}({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2)-{\widehat{\beta }}_{j^{\prime }}^{(m-1)}({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2)\right)\right\}.$$ (7)

Moreover, for $j\in {\mathcal{G}}_k^{(m)}(\mathbf{\lambda })$ with $|{\mathcal{G}}_k^{(m)}(\mathbf{\lambda })|\ge 2$, and k = 1, …, K^(m)(λ),

$$B_j^{(m)}(\mathbf{\lambda })=\frac{1}{n{\mathrm{\lambda }}_1}{\mathit{\text{x}}}_j^T\left(\mathit{\text{I}}-{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}{\left({\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}\right)}^{-1}{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T\right)\mathit{\text{Y}}+{\mathit{\text{x}}}_j^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}{\left({\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}\right)}^{-1}{\mathit{\delta }}^{(m)}(\mathbf{\lambda })-{\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda }).$$ (8)

Theorem 1 reveals two important aspects of β̂^(m)(λ) from (5). First, α̂^(m)(λ) and $B_j^{(m)}(\mathbf{\lambda })$ are continuous and piecewise linear in λ₁ and ${\mathrm{\lambda }}_1^{-1}$, piecewise constant in λ₂ with possible jumps, and piecewise linear in Y with possible jumps. In other words, $({\widehat{\mathit{\alpha }}}^{(m)}(\mathbf{\lambda }),B_j^{(m)}(\mathbf{\lambda }))$ is continuous in λ₁ given (Y, λ₀, λ₂), but may contain jumps with respect to (Y, λ₂). Second, J(·) shrinks β̂^(m)(λ) towards the least squares estimate ${\left({\widehat{\mathit{\alpha }}}_1^0,\dots ,{\widehat{\mathit{\alpha }}}_{K^{(m)}(\mathbf{\lambda })}^0\right)}^T$, with the amount of shrinkage controlled by λ₁ > 0. This occurs only when a pairwise difference stays below the thresholding value of λ₂. As λ₂ → 0, the bias due to penalization becomes ignorable, yielding a nearly unbiased estimate for grouping and parameter estimation.

2.3 Algorithms for difference convex solution surface

One efficient computational tool is a homotopy method (Allgower and Georg, 2003; Wu et al., 2009), which utilizes continuity of a solution in λ to compute the entire solution surface simultaneously. To our knowledge, homotopy methods for nonconvex problems have not yet received attention in the literature. This section develops a homotopy method for a regularization solution surface for nonconvex minimization (2) through DC programming. One major computational challenge is that the solution may be piecewise linear with jumps in (Y, λ), which is difficult to treat with homotopy. To overcome this difficulty, we design a DC algorithm to obtain an easily computed solution β̂(λ), which could be local or global. Note that a DC method guarantees a global solution when it is combined with the branch-and-bound method, c.f., Liu, Shen and Wong (2005). However, seeking a global minimizer of (2) is unnecessary, because the DC solution has the desired statistical properties for grouping (Theorem 3), and can be computed more efficiently (Theorem 2).

The main gradients of our DC homotopy algorithm are (1) iterating the entire DC solution surfaces, (2) utilizing the piecewise linear continuity of $({\widehat{\mathit{\alpha }}}^{(m)}(\mathbf{\lambda }),B_j^{(m)}(\mathbf{\lambda }))$ in $({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_1^{-1})$ for given (λ₀, λ₂, m), and (3) tracking transition points (joints for a piecewise linear function) through the grouped subdifferentials. This algorithm permits efficient computation of a DC solution for nonconvex minimization (2) with an overcomplete penalty, which is otherwise difficult to treat. To compute β̂^(m)(λ), we proceed as follows. First, we fix at one evaluation point of (λ₀, λ₂), then move along path from λ₁ = ∞ towards λ₁ = 0 given (λ₀, λ₂). By Theorem 1, β̂^(m)(λ) is piecewise linear and continuous in λ₁ given (λ₀, λ₂). Along this path, we compute transition points at which the derivative of β̂^(m)(λ) with respect to λ₁ changes. Second, we move to other evaluation points of (λ₀, λ₂) and repeat the above process.

For given (λ₀, λ₂), transition in λ₁ occurs when either of the following conditions is met:

1. Merging: Groups ${\mathcal{G}}_l^{(m)}(\mathbf{\lambda })$ and ${\mathcal{G}}_k^{(m)}(\mathbf{\lambda })$ are combined at λ, when ${\widehat{\alpha }}_l^{(m)}(\mathbf{\lambda })={\widehat{\alpha }}_k^{(m)}(\mathbf{\lambda })$;

2. Splitting: Group ${\mathcal{G}}_k^{(m)}(\mathbf{\lambda })$ is split into two disjoint sets A₁ and A₂ with $A_1\cup A_2={\mathcal{G}}_k^{(m)}(\mathbf{\lambda })$ at λ, when $|B_{A_1}^{(m)}(\mathbf{\lambda })|=|B_{A_2}^{(m)}(\mathbf{\lambda })|=|A_1||A_2|$, according to (3).

In (A) and (B), at a transition point, two or more groups may be merged, and a single group may be split into two or multiple subgroups.

We now describe the basic idea for computing {β̂^(m)(λ) : λ₁ > 0} given (λ₀, λ₂, m). From (6), we track $({\mathcal{G}}^{(m)}(\mathbf{\lambda }),{\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda }))$ along the path from λ₁ = ∞ towards λ₁ = 0 for given (λ₀, λ₂). Let ${\mathrm{\lambda }}_1^{\ast }>0$ be the current transition point. Our algorithm successively identifies the next transition point ${\mathrm{\lambda }}_1^{\ast \ast }$ along the path. For notational ease, we use ( = (₁, …, _K), Δ_j) to denote the current $({\mathcal{G}}^{(m)}(\mathbf{\lambda }),{\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda }))$ after the current transition; j = 1, …, p. Note that $({\mathcal{G}}^{(m)}(\mathbf{\lambda }),{\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda }))$; j = 1, …, p, remain unchanged before the next transition is reached.

For merging in (A), we compute potential merge points:

$$m_{kl}\equiv \frac{{\left({\mathit{\text{e}}}_k-{\mathit{\text{e}}}_l\right)}^T{\left({\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}\right)}^{-1}{\mathit{\text{Z}}}_{\mathcal{G}}\mathit{\text{Y}}}{n{\left({\mathit{\text{e}}}_k-{\mathit{\text{e}}}_l\right)}^T{\left({\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}\right)}^{-1}\mathit{\delta }},\mathit{\delta }={\left({\delta }_1,\dots ,{\delta }_K\right)}^T,{\delta }_k=\sum _{j:j\in {\mathcal{G}}_k}{\mathrm{\Delta }}_j;$$ (9)

1 ≤ k < l ≤ K, where e_k is the kth column of I_p. Then

$${\mathrm{\lambda }}_{1,A}=max\left\{m_{kl}\in (0,{\mathrm{\lambda }}_1^{\ast }]:1\le k<l\le K\right\}$$ (10)

is a potential transition point at which _k′ and _l′ are combined into one group. Define ${\mathrm{\lambda }}_{1,A}=0\text{if}\left\{m_{kl}\in (0,{\mathrm{\lambda }}_1^{\ast }):1\le k<l\le K\right\}=\varnothing$, with ∅ denoting the empty set.

For splitting in (B), we utilize (3) and the subdifferentials in (8), to compute, for each k = 1, …, K, the largest ${\mathrm{\lambda }}_1\in ({\mathrm{\lambda }}_{1,A},{\mathrm{\lambda }}_1^{\ast }]$ and A ⊂ _k with |A| < |_k|/2 such that

$$L_k^{+}({\mathrm{\lambda }}_1,A)L_k^{-}({\mathrm{\lambda }}_1,A)=0,$$ (11)

where $L_k^{\pm }({\mathrm{\lambda }}_1,A)\equiv {\sum }_{j\in A}\left\{\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j\right\}\mp |A|(|{\mathcal{G}}_k|-|A|)$, ${\eta }_j\equiv {\mathit{\text{x}}}_j^T{\mathit{\text{Z}}}_{\mathcal{G}}{\left({\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}\right)}^{-1}\mathit{\delta }-{\mathrm{\Delta }}_j$, and ${\xi }_j\equiv {\mathit{\text{x}}}_j^T\left(I-{\mathit{\text{Z}}}_{\mathcal{G}}{\left({\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}\right)}^{-1}{\mathit{\text{Z}}}_{\mathcal{G}}^T\right)\mathit{\text{Y}}$. It follows from (8) that $B_j^{(m)}(\mathbf{\lambda })=\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j$ before the next transition occurs, and hence that $L_k^{\pm }({\mathrm{\lambda }}_1,{\mathcal{G}}_k)={\sum }_{j\in {\mathcal{G}}_k}{B}_j^{(m)}(\mathbf{\lambda })=0$ for any λ₁ ∈ ℝ. Unfortunately, solving (11) through enumeration is infeasible over all subsets A ⊂ _k. In Algorithm 1 below, we develop an efficient strategy utilizing piecewise linearity of $L_k^{\pm }({\mathrm{\lambda }}_1,A)$ in ${\mathrm{\lambda }}_1^{-1}$ for computing the potential transition point ${\mathrm{\lambda }}_{1,B}\equiv max\left\{s_k\in \left({\mathrm{\lambda }}_{1,A},{\mathrm{\lambda }}_1^{\ast }\right]:k=1,\dots ,K\right\}$, as well as its corresponding grouping, where s_k is the solution of (11); k = 1, …, K. This strategy requires roughly O(p² log p) operations, c.f. Proposition 1.

To describe our strategy for solving (11), let $A_{k\ell }^{+}({\mathrm{\lambda }}_1)$ and $A_{k\ell }^{-}({\mathrm{\lambda }}_1)$ be the two subsets of _k of size ℓ corresponding to the ℓ largest and smallest values of $D_{k\ell }^{+}({\mathrm{\lambda }}_1)\equiv \left\{\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j:{\xi }_j>0,j\in {\mathcal{G}}_k\right\}$ and $D_{k\ell }^{-}({\mathrm{\lambda }}_1)\equiv \left\{\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j:{\xi }_j<0,j\in {\mathcal{G}}_k\right\}$, for ℓ = 1, …, |_k| and k = 1, …, K, where $A_{k\ell }^{\pm }({\mathrm{\lambda }}_1)\equiv A_{k,\ell -1}^{\pm }({\mathrm{\lambda }}_1)$ if $\left|D_{k\ell }^{\pm }({\mathrm{\lambda }}_1)\right|<\ell$. Since $L_k^{\pm }({\mathrm{\lambda }}_1^{\ast },{\mathcal{G}}_k)=0$, we can refine our search to $\left\{A_{k\ell }^{\pm }({\mathrm{\lambda }}_1):\ell =1,\dots ,[{\mathcal{G}}_k/2],{\mathrm{\lambda }}_1\in ({\mathrm{\lambda }}_{1,A},{\mathrm{\lambda }}_1^{\ast }]\right\}$ for solving (11). Note that $L_k^{+}\left({\mathrm{\lambda }}_1,A_{k\ell }^{+}({\mathrm{\lambda }}_1)\right)\le 0$ and $L_k^{-}\left({\mathrm{\lambda }}_1,A_{k\ell }^{-}({\mathrm{\lambda }}_1)\right)\ge 0$ by the definition of $B_j^{(m)}(\mathbf{\lambda })$'s for ${\mathrm{\lambda }}_1\in \left[{\mathrm{\lambda }}_1^{\ast \ast },{\mathrm{\lambda }}_1^{\ast }\right]$. For k = 1, …, K and ℓ = 1, …, [_k/2], we seek the first zero-crossing λ₁ values for $L_k^{\pm }({\mathrm{\lambda }}_1,A_{k\ell }^{\pm }({\mathrm{\lambda }}_1))$, denoted as $s_{k\ell }^{\pm }$, as λ₁ decreases from ${\mathrm{\lambda }}_1^{\ast }$. For each k = 1, …, K, we start with ℓ = 1 and compute

$$s_{k1}^{\pm }=max\left\{\frac{{\xi }_j}{\pm (|{\mathcal{G}}_k|-1)-{\eta }_j}\in \left({\mathrm{\lambda }}_{1,A},{\mathrm{\lambda }}_1^{\ast }\right]:j=1,\dots ,p\right\}.$$ (12)

For ℓ = 2, …, [_k/2], we observe that elements in $A_{k\ell }^{\pm }({\mathrm{\lambda }}_1)$ need to be updated as λ₁ decreases due to rank changes in $\left\{\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j:j\in {\mathcal{G}}_k\right\}$. This occurs at switching points:

$$h_{j{j}^{\prime }}\equiv \frac{{\xi }_j-{\xi }_{j^{\prime }}}{{\eta }_{j^{\prime }}-{\eta }_j};\text{at which}\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_j+{\eta }_j=\frac{1}{n{\mathrm{\lambda }}_1}{\xi }_{j^{\prime }}+{\eta }_{j^{\prime }};1\le j<j^{\prime }\le p.$$ (13)

On this basis, $s_{k\ell }^{\pm }$ can be computed. First, calculate the largest λ₁ ∈ {λ_1,A} ∪ {h_jj′ : 1 ≤ j < j′ ≤ p} at which $L_k^{+}({\mathrm{\lambda }}_1,A_{k\ell }^{+}({\mathrm{\lambda }}_1))\ge 0$ or $L_k^{-}({\mathrm{\lambda }}_1,A_{k\ell }^{-}({\mathrm{\lambda }}_1))\le 0$. Second, compute the exact crossing point for $L_k^{\pm }({\mathrm{\lambda }}_1,A_{k\ell }^{\pm }({\mathrm{\lambda }}_1))=0$ through linear interpolation due to the fact that $L_k^{\pm }({\mathrm{\lambda }}_1,A_{k\ell }^{\pm }({\mathrm{\lambda }}_1))$ is piecewise linear and continuous in ${\mathrm{\lambda }}_1^{-1}$ with joints at h_jj′, 1 ≤ j < j′ ≤ p.

Algorithm 1 computes the next transition point, and Δ_j's.

Algorithm 1: Computation of next transition point

Given the current grouping and Δ_j's with invertible ${\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}$,

• Step 1 (Potential transition for merging) Compute λ_1,A as defined in (10) as well as the corresponding grouping.

• Step 2 Compute $s_{k1}^{\pm }$; k = 1, …, K, as defined in (12), and H ≡ {λ_1,A} ∪ {h_jj′ : 1 ≤ j < j′ ≤ p} based on (13).

• Step 3 (Splitting points) Starting with ℓ = 1, and for k = 1, …, K, we

  • compute the largest λ₁ ∈ H such that $L_k^{+}({\mathrm{\lambda }}_1,A_{k\ell }^{+}({\mathrm{\lambda }}_1))\ge 0$, and the largest λ₁ ∈ H such that $L_k^{-}({\mathrm{\lambda }}_1,A_{k\ell }^{-}({\mathrm{\lambda }}_1))\le 0$, where bisection or Fibonacci search (e.g., Gill, Murray and Wright, 1981) may be applied;

  • interpolate $L_k^{+}({\mathrm{\lambda }}_1,A_{k\ell }^{+}({\mathrm{\lambda }}_1))(\text{resp.}{L}_k^{-}({\mathrm{\lambda }}_1,A_{k\ell }^{-}({\mathrm{\lambda }}_1)))$ linearly to obtain $s_{k\ell }^{+}(\text{resp.}{s}_{k\ell }^{-})$ satisfying $L_k^{+}\left(s_{k\ell }^{+},A_{k\ell }^{+}(s_{k\ell }^{+})\right)=0(\text{resp.}{L}_k^{-}\left(s_{k\ell }^{-},A_{k\ell }^{-}(s_{k\ell }^{-})\right)=0)$; set $s_{k\ell }^{+}=0(\text{resp.}{s}_{k\ell }^{-}=0)$ if ${max}_{{\mathrm{\lambda }}_1\in H}{L}_k^{+}({\mathrm{\lambda }}_1,A_{k\ell }^{+}({\mathrm{\lambda }}_1))<0(\text{resp.}{max}_{{\mathrm{\lambda }}_1\in H}{L}_k^{-}({\mathrm{\lambda }}_1,A_{k\ell }^{-}({\mathrm{\lambda }}_1))>0)$;

  • compute $s_{k\ell }=max\left\{s_{k\ell }^{+},s_{k\ell }^{-}\right\}$ and the corresponding index set;

  • if s_k,ℓ−1 > s_k,ℓ and ℓ ≤ [_k/2] − 1, then go to Step 3 with ℓ replaced by ℓ + 1. Otherwise, set s_k,ℓ+1 = ⋯ = s_k,[k/2] = 0.

• Step 4 (Potential transition for splitting) Compute ${\mathrm{\lambda }}_{1,B}=max\left\{s_{k\ell }\in \left[0,{\mathrm{\lambda }}_1^{\ast }\right]:1<k<K,l=1,\dots ,\left[{\mathcal{G}}_k/2\right]\right\}$, as well as the corresponding grouping.

• Step 5 (Transition) Compute ${\mathrm{\lambda }}_1^{\ast \ast }=max\{{\mathrm{\lambda }}_{1,A},{\mathrm{\lambda }}_{1,B}\}$. If ${\mathrm{\lambda }}_1^{\ast \ast }={\mathrm{\lambda }}_{1,A}>0$, two groups are merged at ${\mathrm{\lambda }}_1^{\ast \ast }$. If ${\mathrm{\lambda }}_1^{\ast \ast }={\mathrm{\lambda }}_{1,B}>0$, a group is split into two at ${\mathrm{\lambda }}_1^{\ast \ast }$. Update and Δ_j's. If ${\mathrm{\lambda }}_1^{\ast \ast }=0$, no further transition can be obtained.

Algorithm 2: Main algorithm

• Step 1 (Parameter initialization): Specify the upper bound parameter K*, and evaluation points of (λ₀, λ₂), where K* ≤ min{n, p}.

• Step 2 (Initialization for DC iterations): Given (λ₀, λ₂), compute β̂⁽⁰⁾(λ₀, +∞, λ₂), the corresponding and Δ's by solving (5) with m = 0. Compute β̂⁽⁰⁾(λ) along the path from λ₁ = ∞ to λ₁ = 0 until |⁽⁰⁾(λ)| = K* using Algorithm 1 while holding (λ₀, λ₂) fixed.

• Step 3 (DC iterations): Starting from m = 1, compute β̂^(m)(λ₀, +∞, λ₂), the corresponding and Δ's by solving (5); then successively compute β̂^(m)(λ) along the path from λ₁ = ∞ to λ₁ = 0 until || = K* using Algorithm 1.

• Step 4 (Stopping rule): If S(β̂^(m−1)(λ)) − S(β̂^(m)(λ)) = 0, then go to Step 3 with m replaced by m + 1. Otherwise, move to next evaluation point of (λ₀, λ₂) and go to Step 2 until all the evaluation points have been computed.

Denote by m* the termination step of Algorithm 2, which may depend on (λ₀, λ₂). Our estimate DCE of β is β̂(λ) ≡ β̂^(m*)(λ). The corresponding grouping of β̂(λ) is (λ). In practice, Algorithm 2 is applicable to unstandardized or standardized predictors.

Proposition 1 (Computational properties). For Algorithm 1, $L_k^{\pm }({\mathrm{\lambda }}_1,A_{k\ell }^{\pm }({\mathrm{\lambda }}_1))$ is continuous, piecewise linear, and strictly monotone in λ₁; k = 1, …, K, ℓ = 1, …, [_k/2]. Moreover, the computational complexities of Algorithms 1 and 2 are no greater than p² (log p + n) and O(m*n*p²(log p + n)), where n* is the number of transition points.

In general, it is difficult to bound n* precisely. However, an application of a heuristic argument similar to that of Rosset and Zhu (2007, Section 3.2, p. 1019) suggests that n* is O(min{n, p}) on average for group combining and splitting.

Theorem 2 (Computation). Assume that ${\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}$ is invertible for m ≤ m*. Then the solution of Algorithm 2 is unique, and sequence S(β̂^(m)(λ₀, λ₀, λ₂)) decreases strictly in m unless β̂^(m)(λ₀, λ₀, λ₂) = β̂^(m−1)(λ₀, λ₀, λ₂). In addition, Algorithm 2 terminates in finite steps, i.e., m* < ∞ with

$${\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })={\widehat{\mathit{\beta }}}^{(m^{\ast }-1)}(\mathbf{\lambda })$$ (14)

for all λ over the evaluation region of λ and all m ≥ m*.

Two distinctive properties of β̂^(m*)(λ) are revealed by (14), leading to fast convergence and a sharp result for consistency of β̂(λ). First, the stopping rule of Algorithm 2, which is reinforced at one fixed λ₀, controls the entire surface for all (λ₁, λ₂) simultaneously, owing to the replacement of ^(m−1)(λ₀, λ₂) by β̂^(m−1)(λ₀, λ₀, λ₂) for iteration m in (4). Second, the iteration process terminates finitely with β̂^(m*)(λ) satisfying (14), because of the step function of ∇S₂(β̂^(m−1)(λ)) resulted from the locally concave points z = ±λ₂ of G(z). Most critically, (14) is not expected for any penalty that is not piecewise linear with non-differentiable but continuous points.

3 Estimation of tuning parameters and σ²

Selection of tuning parameters is important for DCE, which involves λ = (λ₀, λ₁, λ₂). In (1), predictive performance of estimator β̂(λ) is measured by MSE(β̂(λ)), defined as $\frac{1}{n}EL\left(\widehat{\mathit{\beta }}(\mathbf{\lambda }),{\mathit{\beta }}^0\right)$, where $L\left(\widehat{\mathit{\beta }}(\mathbf{\lambda }),{\mathit{\beta }}^0\right)={\sum }_{i=1}^n{\left(\widehat{\mu }(\mathbf{\lambda },{\mathit{\text{x}}}_i)-{\mu }^0({\mathit{\text{x}}}_i)\right)}^2$, μ̂(λ, x) = β̂^T(λ)x, and μ⁰(x) = (β⁰)^Tx.

One critical aspect of tuning is that DCE is a piecewise continuous estimator with jumps in Y. Therefore, any model selection routine may be used for tuning of DCE, which allows for estimators with discontinuities. For instance, cross-validation and the generalized degrees of freedom (GDF, Shen and Huang, 2006) are applicable, but Stein's unbiased risk estimator (Stein, 1981) is not suited because of the requirement of continuity. Then the tuning parameters are estimated by minimizing the model selection criterion.

In practice, σ² needs to be estimated when it is unknown. In the literature, there have been many proposals for the case of p < n, for instance, σ² can be estimated by the residual sum squares over (n − p). In general, estimation of σ² in the case of p > n has not yet received much attention. In our case, we propose a simple estimator ${\widehat{\sigma }}^2=\frac{1}{n-K^{\ast }/2}{\sum }_{i=1}^n{(Y_i-{\widehat{\mu }}_i({\mathbf{\lambda }}^{\ast }))}^2$, where λ* = (λ₀, λ̃₁, ∞), λ̃₁ is the smallest λ₁ reaching the upper bound K(λ₀, λ₁, ∞) = K*/2, and K* is defined in Step 2 of Algorithm 2. Note that when λ₂ = ∞, μ̂_i(λ*) is independent of λ₀. The quality of estimation depends on the bias of μ̂_i(λ*) as well as the variance. By choosing a tight value of K* ≥ K₀, one may achieve good performance.

4 Theory

This section derives a finite-sample probability error bound, based on which we prove that β̂(λ) is consistent with regard to grouping pursuit and predictive optimality simultaneously for the same set of values of λ. As a result, the true grouping ⁰ is reconstructed, as well as the unbiased least squares estimate ${\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})}\equiv {\left({\widehat{\beta }}_1^{(\mathit{\text{ols}})},\dots ,{\widehat{\beta }}_p^{(\mathit{\text{ols}})}\right)}^T={\left({\widehat{\alpha }}_1^{(\mathit{\text{ols}})}{\mathbf{1}}_{|{\mathcal{G}}_1^0|},\dots ,{\widehat{\alpha }}_{K^0}^{(\mathit{\text{ols}})}{\mathbf{1}}_{|{\mathcal{G}}_{K^0}^0|}\right)}^T$ given ⁰. Here ${\widehat{\mathit{\alpha }}}^{(\mathit{\text{ols}})}\equiv {\left({\widehat{\alpha }}_1^{(\mathit{\text{ols}})},\dots ,{\widehat{\alpha }}_{K^0}^{(\mathit{\text{ols}})}\right)}^T={\left({\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^0}\right)}^{-1}{\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T\mathit{\text{Y}}$ with ${\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^0}$ being invertible.

Denote by c_min() > 0 the smallest eigenvalue of ${\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}/n$, where Z is the design matrix based on grouping Denote by ${\gamma }_{min}\equiv min\{|{\alpha }_k^0-{\alpha }_l^0|>0:1\le k<l\le K^0\}$, the resolution level that may depend on (p, n), or the level of difficulty of grouping pursuit, with a small value of γ_min being difficult. The following result is established for β̂(λ) from Algorithm 2.

Theorem 3 (Error bounds for grouping pursuit and consistency). Under the model assumptions of (1) with ε_i ∼ N(0, σ²), assume that λ₀ = λ₁, (2K* + 1)λ₁/λ₂ < min_||≤(K*)² c_min(), where $K_0<K^{\ast }\le min\{\sqrt{n},p\}$. Then for any n and p, we have

$$\begin{array}{l}P(\mathcal{G}(\mathbf{\lambda })\ne {\mathcal{G}}^0)\le P(\widehat{\mathit{\beta }}(\mathbf{\lambda })\ne {\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})})\\ \le \frac{K^0(K^0-1)}{2}\mathrm{\Phi }\left(\frac{-n^{1/2}({\gamma }_{min}-3{\mathrm{\lambda }}_2/2)}{2\sigma c_{min}^{-1/2}({\mathcal{G}}^0)}\right)+p\mathrm{\Phi }\left(\frac{-n{\mathrm{\lambda }}_1}{\sigma {max}_{1\le j\le p}\Vert {\mathit{\text{x}}}_j\Vert }\right),\end{array}$$ (15)

where $\mathrm{\Phi }(z)={\int }_{-\infty }^{z}exp(-u^2/2)du$ is the cumulative distribution function of N(0, 1), and ‖x_j‖ is the L₂-norm of x_j ∈ ⁿ. Moreover, as p, n → +∞, if

1. $\frac{n{({\gamma }_{min}-3{\mathrm{\lambda }}_2/2)}^2}{8c_{min}({\mathcal{G}}^0){\sigma }^2}-2log{K}^0\to \infty ,0<{\mathrm{\lambda }}_2<\frac{2}{3}{\gamma }_{min}$,

2. $\frac{n{\mathrm{\lambda }}_1^2}{2{\sigma }^2{max}_{1\le j\le p}{\Vert {\mathit{\text{x}}}_j\Vert }^2/n}-logp\to \infty$,

then P((λ) ≠ (⁰) ≤ P(β̂(λ) ≠ β̂^(ols)) → 0. In other words, (λ) = ⁰ and β̂(λ) = β̂^(ols) with probability tending to 1.

Corollary 1 (Predictive performance) Under the assumption of Theorem 3, $\frac{L(\widehat{\mathit{\beta }}(\mathbf{\lambda }),{\mathit{\beta }}^0)}{L({\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})},{\mathit{\beta }}^0)}\to 1$, and L(β̂(λ), β⁰) = O_p(K₀/n), as p, n → ∞.

Theorem 3 and Corollary 1 say that DCE consistently identifies the true grouping ⁰ and reconstructs the unbiased least squares estimator β̂^(ols) based on ⁰ when p, n → ∞. They also confirm the assertion made in the Introduction for consistency with regard to nearly exponentially many predictors in n. Specifically, consistency occurs when (a) $p=O(exp(n{\mathrm{\lambda }}_1^2))$ (Condition (ii)) or $\frac{logp}{n}\to 0$, (b) λ₁(2K* + 1) < λ₂min_||≤(K*)² c_min(), (c) $0<{\mathrm{\lambda }}_2<\frac{2}{3}{\gamma }_{min}$ and $n{c}_{min}^{-1}({\mathcal{G}}^0){({\gamma }_{min}-3{\mathrm{\lambda }}_2/2)}^2-16{\sigma }^{2}log{K}^0\to \infty$ (Condition (i)), provided that max_j:1≤j≤p ‖x_j‖²/n is bounded. Note that c_min(⁰) may tend to zero as p, n → ∞ even if the number of true groups K⁰ is independent of (p, n). To understand the conditions (a)-(c), we examine the simplest case in which min_||≤(K*)² c_min(), K⁰ and K* are independent of (p, n). Then (a)-(c) reduce to that $p=O(exp(n{\mathrm{\lambda }}_1^2))$, n^1/2λ₁ → ∞, and λ₂ > c₁λ₁ but ${\mathrm{\lambda }}_2\le \frac{3}{2}{\gamma }_{min}-d_n$ for some constant c₁ > 0 and some sequence d_n > 0 with n^1/2d_n → ∞, where the resolution level γ_min is required to be not too low in that n^1/2γ_min → ∞. Interestingly, there is a trade-off between p and the resolution level γ_min. Note that p = O(exp(n^2δ)) when λ is tuned: λ₁ = c₂λ₂ = c₃γ_min, given that γ_min = O(n^−1/2+δ), for some positive constants c₂, c₃ > 0 and 0 < δ ≤ 1/2. Depending on the value of δ or γ_min, p can be nearly exponentially many for high-resolution regression functions with δ = 1/2, whereas p can be ΔO(exp(n^2δ)) for low-resolution functions when δ is close to 0. The resolution level for DCE can be as low as nearly O(n^−1/2), which, to our knowledge, compares favorably with existing penalties for feature selection.

We now describe characteristics of grouping, in particular—how predictors are grouped. Denote by ${\rho }_j(\mathbf{\lambda })\equiv {\mathit{\text{x}}}_j^T\left(\mathit{\text{Y}}-{\mathit{\text{X}}}^T\widehat{\mathit{\beta }}(\mathbf{\lambda })\right)$, the sample covariance between x_j and the residual.

Theorem 4 (Grouping). Let ${\mathrm{\Delta }}_j(\mathbf{\lambda })={\mathrm{\Delta }}_j^{(m^{\ast })}(\mathbf{\lambda })$ be defined in Theorem 1, where Δ_j(λ) = Δ_j′ (λ) if j, j′ ∈ _k(λ). Let E_k(λ) ≡ [Δ_j(λ) − (|_k(λ)| − 1), Δ_j(λ) + (|_k(λ)| − 1)] be an interval or a point for any j ∈ _k(λ). Then E₁(λ), …, E_K(λ)(λ) are disjoint. Finally, j belongs to _k(λ) if and only if $\frac{1}{n{\mathrm{\lambda }}_1}{\rho }_j(\mathbf{\lambda })\in E_k(\mathbf{\lambda })$; k = 1, …, K(λ).

Theorem 4 says that predictors are grouped according to if their sample covariance values fall into the same intervals, where these disjoint intervals E₁(λ), …, E_K(λ)(λ) characterize grouping. As λ varies, group splitting or combining may take place when these intervals split or combine.

5 Numerical examples

This section examines effectiveness of the proposed method on three simulated examples and one real application to gene network analysis.

For a fair comparison, we compare DCE with the estimator obtained from its convex counterpart–an ultra-fused version of the fused Lasso based on convex penalty Σ_j<j′ |β_j− β_j′|. This allows us to understand the role λ₂ plays in grouping. In addition, we examine the Lasso to investigate the connection between grouping pursuit and feature selection, to confirm our intuition in the foregoing discussion. For reference, least squares estimators based on the full model, and the true grouping, are reported as well, in addition to the average number of iterations in Algorithm 2. Finally, we compare DCE with OSCAR using two examples from Bondell and Reich (2008) in Example 3.

5.1 Benchmarks

We perform simulations in several scenarios, including correlated predictors, different noise levels and situations of “small p but large n” and “small n but large p”. Note that a decrease in the value of σ² implies an increase in the sample size in this case. We therefore fix n = 50 and vary the value of σ² in Examples 1 and 2 and use different sample sizes in Example 3.

For estimating λ for DCE, we generate an independent tuning set ${({\mathit{\text{x}}}_i,{\stackrel{\sim }{y}}_i)}_{i=1}^n$ of size n in each example. Specially, the estimated λ, denoted by λ̂, is obtained by minimizing tuning error $n^{-1}{\sum }_{i=1}^n{({\stackrel{\sim }{y}}_i-\widehat{\mu }(\mathbf{\lambda },{\mathit{\text{x}}}_i))}^2$ over the tuning set with respect to λ over the path of λ₁ > 0, ${\mathrm{\lambda }}_0\in \{i{\mathrm{\lambda }}_1^{\ast }/10:i=1,\dots ,10\}$ and λ₂ ∈ {0.5, 1, 1.5, 2, 2.5, 3, 5, 10, ∞} in Examples 1-3, where ${\mathrm{\lambda }}_1^{\ast }$ is the largest transition point corresponding to λ₂ = ∞. The predictive performance is evaluated by MSE(β̂(λ̂)) as defined in Section 3. The accuracy of grouping is measured by the percentage of matchings between estimated and true pairs of indices (j, j′), for j ≠ j′. Similarly, the tuning parameter of Lasso and the convex counterpart of DCE is estimated over λ₁ > 0 based on the same tuning set for a fair comparison. All numerical analyses are conducted in R 2.9.1.

Example 1 (Sparse Grouping)

This example was used previously for feature selection in Zou and Hastie (2005). A random sample of {(x_i, Y_i) : i = 1, …, n} is obtained with n = 50, where Y_i follows (1) with ε_i ∼ N(0, σ²) and σ² = 2, 1, .5, and x_i is sampled from N(0, Σ_40×40) with p = 20, having the diagonal and off-diagonal elements 1 and 0.5. Here $\mathit{\beta }={(\underset{5}{\underbrace{0,\dots ,0}},\underset{5}{\underbrace{2,\dots ,2}},\underset{5}{\underbrace{0,\dots ,0}},\underset{5}{\underbrace{2,\dots ,2}})}^T$.

As suggested in Table 1, DCE outperforms its convex counterpart and the least squares estimate based on estimated grouping across three levels of σ². This is due primarily to the nonconvex penalty, which corrects the estimation bias due to use of its convex counterpart. This is evident from the fact that the convex counterpart of DCE (m = 0) performs worst than the least square estimates based on the estimated grouping. Furthermore, grouping indeed offers additional improvement in predictive performance in view of the result of the Lasso. Most importantly, DCE reconstructs the least square estimates based on true grouping well, confirming the asymptotic results in Theorem 3 and Corollary 1. The reconstruction is nearly perfectly when σ² = .5 but is less so when the value of σ² increases towards 2, indicating that the noise level does impact the accuracy of reconstruction. Overall, grouping identification and reconstruction appear to be accurate, agreeing with the theoretical results.

Table 1.

MSEs as well as estimated standard errors (in parentheses) of grouping pursuit for various methods based on 100 simulation replications in Example 1. Here Full, True, Lasso, Convex, DCE, denote the least squares estimates based on the full model, the true model, the Lasso estimate, our convex counterpart based on iteration m = 0, and our estimate.

n	σ	Full	True model		Lasso	Our		Ave # Iter	Ave match proportion
			Grouping	Variable		Convex	DCE
50	2.0	1.607 (0.0446)	0.235 (0.0150)	0.845 (0.0310)	1.318 (0.0449)	1.418 (0.0483)	0.837 (0.0721)	4.31 (0.17)	0.633
	1.0	0.402 (0.0111)	0.059 (0.0037)	0.211 (0.0077)	0.330 (0.0112)	0.362 (0.0128)	0.070 (0.0050)	4.08 (0.14)	0.716
	0.5	0.100 (0.0028)	0.015 (0.0009)	0.053 (0.0019)	0.083 (0.0029)	0.091 (0.0032)	0.019 (0.0016)	3.71 (0.10)	0.733
100	2.0	0.833 (0.0244)	0.129 (0.0083)	0.456 (0.0176)	0.658 (0.0229)	0.699 (0.0234)	0.183 (0.0157)	4.32 (0.16)	0.788
	1.0	0.208 (0.0061)	0.032 (0.0021)	0.114 (0.0044)	0.164 (0.0058)	0.175 (0.0059)	0.040 (0.0035)	3.94 (0.12)	0.867
	0.5	0.052 (0.0015)	0.008 (0.0005)	0.028 (0.0011)	0.041 (0.0014)	0.044 (0.0015)	0.010 (0.0008)	3.16 (0.06)	0.887
200	2.0	0.411 (0.0123)	0.060 (0.0041)	0.223 (0.0092)	0.335 (0.0117)	0.364 (0.0129)	0.080 (0.0057)	4.03 (0.13)	0.926
	1.0	0.103 (0.0031)	0.015 (0.0010)	0.056 (0.0023)	0.084 (0.0029)	0.091 (0.0032)	0.020 (0.0014)	3.33 (0.06)	0.950
	0.5	0.026 (0.0008)	0.004 (0.0003)	0.014 (0.0006)	0.022 (0.0008)	0.023 (0.0008)	0.005 (0.0003)	2.98 (0.01)	0.960

Figure 1 displays the paths in λ₁ given various values of λ₂ with λ₀ = .2. Clearly, β̂(λ) is continuous in λ₁ given (λ₀, λ₂) and has jumps in λ₂ given (λ₀, λ₁), as discussed early. Figure 2 shows four two-dimensional DCE solution surfaces for (β̂₁(λ), β̂₂(λ), β̂₁₉(λ), β̂₂₀(λ)) with respect to λ. In Figure 2, the four estimates are close to their corresponding least squares estimates when either λ₁ or λ₂ becomes small. On the other hand, they tend to be close to each other when both λ₁ and λ₂ become large. Note that some jumps in λ₂ are visible for β̂₁(λ) around λ = (0.2, 2.5, 2.5).

Figure 1.

Plots of β̂(λ) as a function of λ₁ for various λ₂ values with λ₀ = .2 and σ = 1.2 in Example 1. Different components of β̂(λ) are represented by different types of lines and colors.

Figure 2.

Image plots of regularization solution surfaces of four components β̂₁(λ), β̂₂(λ), β̂₁₉(λ), and β̂₂₀(λ) as a function of (λ₁, λ₂) for λ₀ = 0.2 and σ = 1.2 in Example 1.

Example 2 (Large p but small n)

A random sample of {(x_i, Y_i) : i = 1, …, n} with n = 50, is obtained, where Y_i follows (2) with ε_i ∼ N(0, σ²), σ = .41, .58, p = 50, 100, and x_i is sampled from N(0, Σ) with the (j, k)th element of Σ being 0.5^|j−k|. Here $\mathit{\beta }={(\underset{5}{\underbrace{3,\dots ,3}},\underset{5}{\underbrace{-1.5,\dots ,-1.5}},\underset{5}{\underbrace{1,\dots ,1}},\underset{5}{\underbrace{2,\dots ,2}},\underset{p-20}{\underbrace{0,\dots ,0}})}^T$.

In this “large p but small n” example, DCE outperforms its convex counterpart with regard to predictive performance in all the cases, but the amounts of improvement vary. Here, the Lasso performs slightly better in some cases, which is expected because of the large group size of zero-coefficient predictors. Interestingly, the average number of iterations for Algorithm 2 is about 3 as compared to 4 in Example 1. Finally, the matching proportion for grouping is reasonably high.

Example 3 (Small p but large n)

Consider Examples 4 and 5 of Bondell and Reich (2008), which are low-dimensional with relatively large dimensions. Their Example 4 is the same as Example 1 except that n = 100, p = 40, σ² = 15² and $\mathit{\beta }={(\underset{10}{\underbrace{0,\dots ,0}},\underset{10}{\underbrace{2,\dots ,2}},\underset{10}{\underbrace{0,\dots ,0}},\underset{10}{\underbrace{2,\dots ,2}})}^T$. Their Example 5 has a similar setting, but with n = 50, p = 40, σ² = 15², and $\mathit{\beta }={(\underset{15}{\underbrace{3,\dots ,3}},\underset{25}{\underbrace{0,\dots ,0}})}^T$, where x_i ∼ N(0, V), and V is a block-diagonal matrix with diagonal blocks ${\mathbf{1}}_5\mathbf{1}\frac{T}{5}$, ${\mathbf{1}}_5\mathbf{1}\frac{T}{5}$, ${\mathbf{1}}_5\mathbf{1}\frac{T}{5}$ and I_p−15. In addition, the first 15 components of x_i are added to independent noise distributed as N(0, .16) to generate three equally important groups having pairwise correlations being around 0.85.

Overall DCE performs comparably with OSCAR in these noisy situations with σ² = 15², which performs better but worst, respectively in Examples 4 and 5 of Bondell and Reich (2008). This is mainly due to the fact that DCE does not select variables beyond grouping. This aspect was also evident from Table 2, where DCE performs worse than Lasso in some cases. Most noticeably, DCE performs slightly worse than its convex counterpart when σ is very large. This is expected because an adaptive method tends to perform worse than its non-adaptive counterpart in a noisy situation.

Table 2.

MSEs as well as estimated standard errors (in parentheses) for various methods based on 100 simulation replications in Example 2. Here Full, True, Lasso, Convex, DCE denote the least squares estimates based on the full model, the true model, the Lasso estimate, our convex counterpart based on iteration m = 0, and our estimate.

p	n	σ	Full	True model		Lasso	Our		Ave # Iter	Ave match proportion
				Grouping	Variable		Convex	DCE
50	50	0.58	0.327 (0.007)	0.041 (0.002)	0.142 (0.004)	0.227 (0.006)	0.276 (0.006)	0.123 (0.010)	3.01 (0.07)	0.799
50	100	0.41	0.089 (0.002)	0.010 (0.001)	0.037 (0.001)	0.056 (0.002)	0.075 (0.002)	0.011 (0.001)	3.12 (0.03)	0.862
100	50	0.58	0.325 (0.007)	0.037 (0.002)	0.136 (0.004)	0.311 (0.007)	0.327 (0.007)	0.325 (0.007)	2.17 (0.04)	0.778
100	100	0.41	0.165 (0.003)	0.010 (0.001)	0.034 (0.001)	0.073 (0.002)	0.116 (0.003)	0.110 (0.003)	2.06 (0.02)	0.722

5.2 Breast cancer metastasis and gene network

Mapping the pathways giving rise to metastasis is important in breast cancer research. Recent studies suggest that gene expression profiles are useful in identifying gene subnetworks correlated with metastasis. Here we apply our proposed method to understand the functionality of subnetworks of genes for predicting the time to metastasis, which may provide novel hypotheses and confirm the existing theory for pathways involved in tumor progression.

The breast cancer metastasis data (Wang et al., 2005; Chuang et al., 2007) contain gene expression levels of 8141 genes for 286 patients, 107 of whom were detected to develop metastasis within a five year follow-up after surgery. To utilize the present gene network knowledge, we explore the PPI network previously constructed in Chuang et al. (2007).

For breast metastasis, three tumor suppressor genes–TP53, BRACA1 and BRACA2, are known to be crucial in preventing uncontrolled cell proliferation and repairing the chromosomal damage. Certain mutations of these genes increase risk of breast center, c.f., Soussi (2003). In our analysis, we construct a subnetwork of Chuang et al. (2007), consisting of genes TP53, BRCA1 and BRCA2, as well as genes that were regulated by them. This leads to 294 expressed genes for 107 patients who developed metastasis.

For subnetwork analysis, consider a vector of p predictors, each corresponding to one node in an undirected graph together with edges connecting nodes. Also available is a vector of network weights w ≡ (w₁, …, w_p)^T, indicating relative importance of the predictors. The weight vector reflects the biological importance of a “hub” gene. Given predictors ${({\stackrel{\sim }{\mathit{\text{x}}}}_i={({\stackrel{\sim }{x}}_{i1},\dots ,{\stackrel{\sim }{x}}_{ip})}^T)}_{i=1}^n$,

$$Y_i\equiv \mu ({\stackrel{\sim }{\mathit{\text{x}}}}_i)+{\epsilon }_i,\mu (\stackrel{\sim }{\mathit{\text{x}}})\equiv {\stackrel{\sim }{x}}^T\stackrel{\sim }{\mathit{\beta }},E({\epsilon }_i)=0,\text{Var}({\epsilon }_i)={\sigma }^2;i=1,\dots ,n,$$ (16)

where β̃ ≡ (β̃₁, …, β̃_p)^T is a vector of regression coefficients, and x̃_i is independent of ε_i. In (16), we aim to identify all possible homogenous subnetworks of predictors with respect to w. That is, $w_{j_1}^{-1}{\stackrel{\sim }{\beta }}_{j_1}\approx \cdots \approx w_{j_K}^{-1}{\stackrel{\sim }{\beta }}_{j_K}$ within each group {j₁, …, j_K} ⊂ {1, …, p}. Model (16) reduces to (1) by letting x_i = (w₁x̃_i1, …, w_px̃_ip)^T and $\mathit{\beta }={(w_1^{-1}{\stackrel{\sim }{\beta }}_1,\dots ,w_p^{-1}{\stackrel{\sim }{\beta }}_p)}^T$. Note that existence of a path between nodes j and j′ in the undirected graph indicates if predictors x_j and x_j′ can be grouped. However, our network is a complete graph in this application.

For data analysis, the 107 patients are divided randomly into two groups with 70 and 37 patients, respectively for model-building and validation. For model building, an estimated MSE based on the generalized degrees of freedom is minimized with regard to a set of grid points over the path of λ₁ > 0, ${\mathrm{\lambda }}_0\in \{i{\mathrm{\lambda }}_1^{\ast }/10:i=1,\dots ,10\}$ and λ₂ ∈ {0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, ∞}, where ${\mathrm{\lambda }}_1^{\ast }$ is the largest transition point corresponding to λ₂ = ∞, to obtain the optimal tuning parameters based on the data perturbation method, c.f., Shen and Huang (2007). Moreover, we take Y_i as the log time to metastasis (in months) and x_i to be the expression levels with dimension p = 294, together with a Laplacian weight vector $\{w_j=d_j^{1/2},:j=1,\dots ,294\}$, where d_j is the number of directed nodes connecting to node j, c.f., Li and Li (2008).

Table 3 summarizes the estimated grouped regression coefficients based on 27 estimated groups with the corresponding subnetworks displayed by different colors in Figure 3. Interestingly TP53 and BRACA1 are similar but they differ from BRACA2, as evident by the corresponding color intensities in Figure 3, indicating the roles that they play in the process of the metastasis. To make a sense of the estimated grouping, we examine the disease genes causing the metastasis, which were identified in Wang et al. (2005). Among the 17 disease genes expressed in our network, 1, 1, 1, 1 and 13 genes belong to the second, 13th, 14th, 17th and 18th groups, respectively. In fact, three disease genes form single groups, and one disease gene belongs to a small group, and 13 of them are in a large group, indicating that genes work in groups according to their functionalities with regard to survivability of breast cancer. In our analysis, the mean squared prediction error is .81 based on the testing data set of 37 observations, which yields the MSE of .3. This is reasonably good relative to the estimated σ² = .51.

Table 3.

Median MSEs based on 100 simulation replications in Example 3. Here Full, True, Lasso, OSCAR, E-NET, Convex, DCE denote the least squares estimates based on the full model, the true model, the Lasso estimate, the elastic net estimate, the OSCAR estimate, our convex counterpart based on iteration m = 0, and our estimate. Note that the results for OSCAR and E-NET in Examples 4 and 5 of Bondell and Reich (2008) with σ = 15 are taken from Table 1 there.

Bondell & Reich	σ	Full	True model		Lasso	E-NET	OSCAR	Our		Ave # Iter	Ave match proportion
			Group	Variable				Conv	DCE
Ex 4	15	95.1	6.2	47.4	45.4	34.4	25.9	21.4	22.0	2	0.516
Ex 5	15	174.8	11.6	67.4	64.7	40.7	51.8	67.6	70.0	4	0.535
	10	77.7	5.1	30.0	33.1			35.3	38.0	4	0.575
	5	19.4	1.3	7.5	10.7			10.4	6.0	4.5	0.626
	1	0.86	0.05	0.30	0.47			0.43	0.06	3	0.703

Figure 3.

Plot of the PPI subnetwork for the metastasis data, as described by an undirected graph with 294 nodes and 326 edges. Three regulating genes TP53, BRACA1 and BRACA2 are represented by large nodes. There are 27 estimated groups colored over the color spectrum with dark color corresponding to small estimated group regression coefficients. The plot is produced in Cytoscape.

6 Discussion

This article proposes a novel grouping pursuit method for high-dimensional least squares regression. The proposed method is placed in the framework of likelihood estimation and model selection. It offers a general treatment to a continuous but non-differentiable nonconvex likelihood problem, where the global penalized maximum likelihood estimate is difficult to obtain. Remarkably, our DC treatment yields desired statistical properties expected from the global penalized maximum likelihood estimate. This is mainly because the DC score equation defined through subdifferentials equals to that of the original nonconvex problem, when the termination criterion is met, c.f., Theorem 2. In this process, the continuous but non-differentiable penalty is essential, At present the proposed method is not designed for feature selection. To generalize, one may replace J(β) by ${\sum }_{j=1}^{p}G({\beta }_j)+{\sum }_{j<j^{\prime }}G({\beta }_j-{\beta }_{j^{\prime }})$ in (2). Moreover, estimation of the tuning parameters needs to be further investigated with regard to the accuracy of selection. Further research is therefore necessary.

7 Technical proofs

Proof of Lemma 1

We prove by contradiction. Without loss of generality, assume that ${\mathit{\beta }}^{\ast }={\left({\beta }_1^{\ast },\dots {\beta }_m^{\ast }\right)}^T$ is a local minimum of f(β) = h(β) + λ₁J(β) attaining at the locally non-smooth concave point λ₂ for pair (1, 2) in that ${\beta }_1^{\ast }-{\beta }_2^{\ast }={\mathrm{\lambda }}_2$. Let $f_1({\beta }_1)=f({\beta }_1,{\beta }_2^{\ast },\dots ,{\beta }_p^{\ast })$, $h_1({\beta }_1)=h({\beta }_1,{\beta }_2^{\ast },\dots ,{\beta }_p^{\ast })$ and $J_1({\beta }_1)=J({\beta }_1,{\beta }_2^{\ast },\dots ,{\beta }_p^{\ast })$. Denote by the right derivative of J₁(β₁) at ${\beta }_1^{\ast }$ to be b. By assumption, its left derivative at ${\beta }_1^{\ast }$ must be b + λ₁. Consider the derivative of h₁(β₁) at ${\beta }_1^{\ast }$. Note that f₁(β₁) achieves a local minimum at ${\beta }_1^{\ast }$, implying that its right and left derivatives at ${\beta }_1^{\ast }$ are larger than 0 and less than 0. Hence the right and left derivatives of h₁(β₁) at ${\beta }_1^{\ast }$ is larger than −b and smaller than −b − λ₁. This contradicts to the fact that h₁(β₁) is differentiable in β₁ for any λ₂ > 0. This completes the proof.

Proof of Theorem 1

To prove continuity in λ₁, note that the derivative of (5) with respect to β is continuous in λ₁ because ${\sum }_{j^{\prime }:j^{\prime }\ne j}\nabla G_2({\widehat{\beta }}_j^{(m-1)}({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2)-{\widehat{\beta }}_{j^{\prime }}^{(m-1)}({\mathrm{\lambda }}_0,{\mathrm{\lambda }}_0,{\mathrm{\lambda }}_2))$ is independent of λ₁. By convexity of (5) in β and uniqueness of β̂^(m)(λ), β̂^(m)(λ) is continuous in λ₁ for each m.

Next we derive an expression for α̂^(m)(λ). If K^(m)(λ) = 1 and $|{\mathcal{G}}_k^{(m)}(\mathbf{\lambda })|\ge 2$, then the result follows from (5). Now consider the case of K^(m)(λ) ≥ 2. For any j = 1, …, p, write ${\sum }_{j^{\prime }\ne j}{b}_{j{j}^{\prime }}^{(m)}(\mathbf{\lambda })={\sum }_{j^{\prime }:j^{\prime }\sim j}{b}_{j{j}^{\prime }}^{(m)}(\mathbf{\lambda })+{\sum }_{j^{\prime }:j^{\prime }\nsim j}{b}_{j{j}^{\prime }}^{(m)}(\mathbf{\lambda })$ as $B_j^{(m)}(\mathbf{\lambda })+{\sum }_{k:k\ne g(\mathbf{\lambda },j)}|{\mathcal{G}}_{g(\mathbf{\lambda },j)}^{(m)}(\mathbf{\lambda })|$ Sign $\left({\widehat{\alpha }}_k^{(m)}(\mathbf{\lambda })-{\widehat{\alpha }}_{g(\mathbf{\lambda },j)}^{(m)}(\mathbf{\lambda })\right)$, where j′ ∼ j if j′ and j are in the same group and j′ ≁ j otherwise. Differentiating (5) with respect to β, we obtain

$$-{\mathit{\text{x}}}_j^T(\mathit{\text{Y}}-{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}{\widehat{\mathit{\alpha }}}^{(m)}(\mathbf{\lambda }))+n{\mathrm{\lambda }}_1\left({\mathrm{\Delta }}_j^{(m)}(\mathbf{\lambda })+B_j^{(m)}(\mathbf{\lambda })\right)=0,j=1,\dots ,p,$$ (17)

which is an optimality condition (Rockafellar and Wets, 2003). For k = 1, …, K^(m)(λ), invoking the sum-to-zero constraint ${\sum }_{j\in {\mathcal{G}}_k^{(m)}(\mathbf{\lambda })}{B}_j^{(m)}(\mathbf{\lambda })=0$, we have $-{\mathit{\text{z}}}_{k,{\mathcal{G}}^{(m)}(\mathbf{\lambda })}(\mathit{\text{Y}}-{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}{\widehat{\mathit{\alpha }}}^{(m)}(\mathbf{\lambda }))+n{\mathrm{\lambda }}_1{\delta }_k^{(m)}(\mathbf{\lambda })=0$, implying (6). Thus (8) follows from (17).

Proof of Proposition 1

By definition, $L_k^{\pm }\left({\mathrm{\lambda }}_1,A_{k\ell }^{\pm }({\mathrm{\lambda }}_1)\right)$ is piecewise linear and continuous in λ₁, and is strictly monotone because ${\sum }_{j\in A_{k\ell }^{+}({\mathrm{\lambda }}_1)}{\xi }_j>0$ and ${\sum }_{j\in A_{k\ell }^{-}({\mathrm{\lambda }}_1)}{\xi }_j<0$.

Lets g_k = |_k| − 1. For Algorithm 1, the complexities for Steps 1 and 2 are O(np²). In Step 3, sorting for computing search points is no greater than $O(g_k^{2}log{g}_k)$ for group _k. Hence the complexity for Step 3 is O(p² log p) using the fact that ${\sum }_{k=1}^K{g}_k<p$, because the complexity of search in Step 3 is no great than log p for the bisection (Fibonacci) search. Then the complexity for Algorithm 2 is O(m*n*p²(log p + n)). This completes the proof.

Proof of Theorem 2

Uniqueness of the solution follows from strict convexity of S^(m)(β) in β for each m under the assumption that ${\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}$ is invertible.

Our plan is to prove the result for λ₁ = λ₀ with λ = (λ₀, λ₀, λ₂)^T. Then controlling at this point implies the desirable result for all λ. In what follows, we set λ₁ = λ₀ unless indicated otherwise. For convergence of Algorithm 2, it follows from (2) and (5) that for m ∈ ℕ, 0 ≤ S(β̂^(m)(λ)) = S^(m+1)(β̂^(m)(λ)) ≤ S^(m)(β̂^(m)(λ)) ≤ S^(m)(β̂^(m−1)(λ)) = S(β̂^(m−1)(λ)). This implies that lim_m→∞ S(β̂^(m)(λ)) exists, thus leading to convergence. To study the number of steps to termination, note that S^(m)(β̂^(m−1)(λ)) − S^(m)(β̂^(m)(λ)) can be written as

$$\frac{1}{2n}{\Vert \mathit{\text{X}}\left({\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })-{\widehat{\mathit{\beta }}}^{(m-1)}(\mathbf{\lambda })\right)\Vert }^2+\frac{1}{n}{\left(\mathit{\text{Y}}-\mathit{\text{X}}{\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })\right)}^T\mathit{\text{X}}\left({\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })-{\widehat{\mathit{\beta }}}^{(m-1)}(\mathbf{\lambda })\right)+{\mathrm{\lambda }}_1\left\{\sum _{j=1}^p({\widehat{\beta }}_j^{(m)}(\mathbf{\lambda })-{\widehat{\beta }}_j^{(m-1)}(\mathbf{\lambda }))\sum _{j^{\prime }\ne j}\nabla G_2({\widehat{\beta }}_j^{(m-1)}(\mathbf{\lambda })-{\widehat{\beta }}_{j^{\prime }}^{(m-1)}(\mathbf{\lambda }))\right\}+{\mathrm{\lambda }}_1\{\sum _{j<j^{\prime }}(\left|{\widehat{\beta }}_j^{(m-1)}(\mathbf{\lambda })-{\widehat{\beta }}_{j^{\prime }}^{(m-1)}(\mathbf{\lambda })\right|-\left|{\widehat{\beta }}_j^{(m)}(\widehat{\mathbf{\lambda }})-{\widehat{\beta }}_{j^{\prime }}^{(m-1)}(\mathbf{\lambda })\right|)\},$$

which can be simplified, using the following equality from (17), $-{\mathit{\text{x}}}_j^T(\mathit{\text{Y}}-\mathit{\text{X}}{\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda }))=n{\mathrm{\lambda }}_1{\sum }_{j\prime :j\prime \ne j}\left\{\nabla G_2\left({\widehat{\beta }}_j^{(m-1)}(\mathbf{\lambda })-{\widehat{\beta }}_{j\prime }^{(m-1)}(\mathbf{\lambda })\right)-b_{jj\prime }^{(m)}(\mathbf{\lambda })\right\}$, as $\frac{1}{2n}{\Vert \mathit{\text{X}}\left({\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })-{\widehat{\mathit{\beta }}}^{(m-1)}(\mathbf{\lambda })\right)\Vert }^2+{\mathrm{\lambda }}_1{\sum }_{j<j\prime }\left\{\left|z_{jj\prime }^{(m-1)}(\mathbf{\lambda })\right|-\left|z_{jj\prime }^{(m)}(\mathbf{\lambda })\right|-b_{jj\prime }^{(m)}(\mathbf{\lambda })\left(z_{jj\prime }^{(m-1)}(\mathbf{\lambda })-z_{jj\prime }^{(m)}(\mathbf{\lambda })\right)\right\}$, where $z_{jj\prime }^{(m)}(\mathbf{\lambda })={\widehat{\beta }}_j^{(m)}(\mathbf{\lambda })-{\widehat{\beta }}_{j\prime }^{(m)}(\mathbf{\lambda })$. By convexity of |z|, $\left|z_{jj\prime }^{(m-1)}(\mathbf{\lambda })\right|-\left|z_{jj\prime }^{(m)}(\mathbf{\lambda })\right|-b_{jj\prime }^{(m)}(\widehat{\mathbf{\lambda }})\left(z_{jj\prime }^{(m-1)}(\mathbf{\lambda })-z_{jj\prime }^{(m)}(\mathbf{\lambda })\right)\ge 0$, implying S(β̂^(m−1)(λ)) − S(β̂^(m) (λ)) ≥ S^(m) (β̂^(m−1) (λ)) −S^(m) (β̂^(m) (λ)) is bounded below by $\frac{1}{2n}{\Vert \mathit{\text{X}}\left({\widehat{\mathit{\beta }}}^{(m)}(\mathbf{\lambda })-{\widehat{\mathit{\beta }}}^{(m-1)}(\mathbf{\lambda })\right)\Vert }^2$. That is, $\frac{1}{2n}{\left({\widehat{\mathit{\alpha }}}^{(m)}(\mathbf{\lambda })-{\mathit{\alpha }}^{(m-1)}(\widehat{\mathbf{\lambda }})\right)}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^{(m)}(\mathbf{\lambda })}\left({\widehat{\mathit{\alpha }}}^{(m)}(\mathit{\lambda })-{\widehat{\mathit{\alpha }}}^{(m-1)}(\mathbf{\lambda })\right)$ is greater than zero unless α̂^(m) (λ) = α̂^(m−1) (λ).

Finally, finite step convergence follows from strict decreasingness of S^(m)(β̂ (λ)) in m and finite possible values of ∇S₂(β̂^(m−1) (λ)) in (5). For (14), note that when termination, ∇S₂(β̂^(m−1) (λ)) remains unchanged for m ≥ m*, so does the cost function (5) for m ≥ m*. This implies termination for all λ = (λ₀, λ₁, λ₂)^T in (14). This completes the proof.

Proof of Theorem 3

Define event

$$F\equiv \left\{\underset{k<l}{min}\left|{\widehat{\alpha }}_k^{(\mathit{\text{ols}})}-{\widehat{\alpha }}_l^{(\mathit{\text{ols}})}\right|>3{\mathrm{\lambda }}_2/2\right\}{\cap }_{k:|{\mathcal{G}}_k^0|>1}\left\{\underset{j:j\in {\mathcal{G}}_k^0}{max}\left|{\mathit{\text{x}}}_j^T\left(\mathit{\text{Y}}-{\mathit{\text{X}}}^T{\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})}\right)\right|\le n{\mathrm{\lambda }}_1(|{\mathcal{G}}_k^0|-1)\right\}.$$

By (17) with m = m* and (14), for k = 1, …, K, β̂(λ) = β̂^(m)(λ) satisfies

$$\{\begin{array}{l}-{\left({\sum }_{j\in {\mathcal{G}}_k}{\mathit{\text{x}}}_j\right)}^T(\mathit{\text{Y}}-\mathit{\text{X}}\mathit{\beta })-n{\mathrm{\lambda }}_1\left({\sum }_{j\in {\mathcal{G}}_k}{\mathrm{\Delta }}_j(\mathit{\beta })\right)=0;\\ |{\mathit{\text{x}}}_j^T(\mathit{\text{Y}}-\mathit{\text{X}}\mathit{\beta })+n{\mathrm{\lambda }}_1{\mathrm{\Delta }}_j(\mathit{\beta })|\le n{\mathrm{\lambda }}_1(|{\mathcal{G}}_k|-1);j\in {\mathcal{G}}_k,|{\mathcal{G}}_k|>1,\end{array}$$ (18)

for some partition (₁, …, _K) of {1, …, p} with K ≤ min {n, p}, where Δ_j(β) ≡ Σ_{j′:j′≠j} {Sign(β_j − β_j′) − ∇G₂(β_j − β_j′)}; j = 1, …, p.

Note that the first event in F, together with the grouped subdifferentials, yields that ${\sum }_{j\in {\mathcal{G}}_k^0}{\mathrm{\Delta }}_j({\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})})=0$; k = 1, …, K⁰. This, together with the least squares property that ${\left({\sum }_{j\in {\mathcal{G}}_k^0}{\mathit{\text{x}}}_j\right)}^T(\mathit{\text{Y}}-\mathit{\text{X}}{\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})})=0$, implies that the first equation of (18) is fulfilled with β = β̂^(ols). Moreover, the events in F imply the second equation of (18) with β = β̂^(ols). Consequently β̂^(ols) is a solution of (18) on F.

It remains to show that (18) yields the unique minimizer on F. Define

$$\stackrel{\sim }{G}(z)=\{\begin{array}{ll}G(z);& \text{if}\left|z\right|\le {\mathrm{\lambda }}_2(1-\nu )\text{or}\left|z\right|\ge {\mathrm{\lambda }}_2(1+\nu ),\\ -\frac{1}{4{\mathrm{\lambda }}_2\nu }{(z-{\mathrm{\lambda }}_2)}^2+\frac{1}{2}(z-{\mathrm{\lambda }}_2)+{\mathrm{\lambda }}_2(1-\frac{\nu }{4});& \text{if}\left|z-{\mathrm{\lambda }}_2\right|<{\mathrm{\lambda }}_2\nu ,\\ -\frac{1}{4{\mathrm{\lambda }}_2\nu }{(z+{\mathrm{\lambda }}_2)}^2-\frac{1}{2}(z+{\mathrm{\lambda }}_2)+{\mathrm{\lambda }}_2(1-\frac{\nu }{4});& \text{if}\left|z+{\mathrm{\lambda }}_2\right|<\nu {\mathrm{\lambda }}_2,\end{array}$$

for ν = 1/2. Given any grouping with || ≤ K*, S̃(β) is a function of α with β = (α₁1_|1|, …, α_K1_|K|)^T. Then $\stackrel{\sim }{S}(\mathit{\beta })=\frac{1}{2n}{\sum }_{i=1}^n{(Y_i-{\mathit{\text{x}}}_i^T\mathit{\beta })}^2+{\mathrm{\lambda }}_1{\sum }_{j<j\prime }\stackrel{\sim }{G}({\beta }_j-{\beta }_{j\prime })$ is strictly convex in α ∈ ℝ^|| when $\frac{1}{n}{\mathit{\text{Z}}}_{\mathcal{G}}^T{\mathit{\text{Z}}}_{\mathcal{G}}>\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}\left\{(|\mathcal{G}|+1){\mathit{\text{I}}}_{|\mathcal{G}|}-{\mathbf{1}}_{|\mathcal{G}|}{\mathbf{1}}_{|\mathcal{G}|}^T\right\}$, which occurs when $c_{min}(\mathcal{G})>\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(|\mathcal{G}|+1)$. To prove that β̂^(ols) is the unique minimizer of S̃(β), suppose β̃ is another minimizer of S̃(β) with the corresponding grouping and || < K*. Because ${min}_{|\mathcal{G}|\le {(K^{\ast })}^2}{c}_{min}\left(\mathcal{G}\right)>\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(K^{\ast }+1)$ and (K*)² ≤ n, it follows that S̃(β) is strictly convex in α_⁰∨ ∈ ℝ^|0∨&|, implying β̃ = β̂^(ols), where ⁰ ∨ is the coarsest common refinement of ⁰ and with |⁰ ∨ | ≤ min{n, p}.

To prove that β̂^(ols) is the unique minimizer of S(β) on F, we let * = ⁰ ∨ with || ≤ K*. Let ${\widehat{\mathit{\alpha }}}_{\mathcal{G}\ast }^{(\mathit{\text{ols}})}$ be the estimate corresponding to β̂^(ols). By the mean value theorem, $\Vert \frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta })-\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta }){|}_{{\mathit{\alpha }}_{\mathcal{G}\ast }={\widehat{\mathit{\alpha }}}_{\mathcal{G}\ast }^{(\mathit{\text{ols}})}}\Vert$ is lower bounded by

$$\left\{\underset{|\mathcal{G}|\le {(K\ast )}^2}{min}{c}_{min}(\mathcal{G})-\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(K^{\ast }+1)\right\}\Vert {\mathit{\alpha }}_{\mathcal{G}\ast }-{\widehat{\mathit{\alpha }}}_{\mathcal{G}\ast }^{(\mathit{\text{ols}})}\Vert >0.$$ (19)

Note that S̃(β) = S(β) over E = {β : ||α_k − α_l| − λ₂| > λ₂/2 : 1 ≤ k < l ≤ ||}. Moreover, by construction, ${sup}_{{\mathit{\alpha }}_{\mathcal{G}\ast }}\Vert \frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}S(\mathit{\beta })-\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta })\Vert \le \frac{{\mathrm{\lambda }}_1}{2}{K}^{\ast }$ on F, which implies, together with (19), for any β ≠ β̂^(ols) ∈ E^c, $\Vert \frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}S(\mathit{\beta })\Vert$ is

$$\Vert \left(\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta })-\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta }){|}_{{\mathit{\alpha }}_{\mathcal{G}\ast }={\widehat{\mathit{\alpha }}}_{\mathcal{G}\ast }^{(\mathit{\text{ols}})}}\right)+\left(\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}S(\mathit{\beta })-\frac{\partial }{\partial {\mathit{\alpha }}_{\mathcal{G}\ast }}\stackrel{\sim }{S}(\mathit{\beta })\right)\Vert \ge \left(\underset{|\mathcal{G}|\le {(K^{\ast })}^2}{min}{c}_{min}(\mathcal{G})-\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(K^{\ast }+1)\right)\Vert {\mathit{\alpha }}_{\mathcal{G}\ast }-{\widehat{\mathit{\alpha }}}_{\mathcal{G}\ast }^{(\mathit{\text{ols}})}\Vert -\frac{{\mathrm{\lambda }}_1}{2}{K}^{\ast },$$

which is lower bounded by $\left({min}_{|\mathcal{G}|\le {(K^{\ast })}^2}{c}_{min}(\mathcal{G})-\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(2K^{\ast }+1)\right)\frac{{\mathrm{\lambda }}_2}{2}>0$, because ${min}_{|\mathcal{G}|\le {(K^{\ast })}^2}{c}_{min}(\mathcal{G})>\frac{{\mathrm{\lambda }}_1}{{\mathrm{\lambda }}_2}(2K^{\ast }+1)$. This, together with Lemma 1, implies that S(β) has no local minimal in E^c on F, and hence it has the unique local minimal on F. On the other hand, β̂(λ) is a local minimizer of S(β) on F. Consequently, β̂^(ols) = β̂(λ) on F.

Note that ${\widehat{\alpha }}_k^{(\mathit{\text{ols}})}-{\widehat{\alpha }}_l^{(\mathit{\text{ols}})}\sim N({\alpha }_k^0-{\alpha }_l^0,\mathit{\text{Var}}({\widehat{\alpha }}_k^{(\mathit{\text{ols}})}-{\widehat{\alpha }}_l^{(\mathit{\text{ols}})}))$ with $\mathit{\text{Var}}({\widehat{\alpha }}_k^{(\mathit{\text{ols}})}-{\widehat{\alpha }}_l^{(\mathit{\text{ols}})})\le 4c_{min}^{-1}({\mathcal{G}}^0){\sigma }^2/n$, and ${\mathit{\text{x}}}_j^T\left(\mathit{\text{Y}}-{\mathit{\text{X}}}^T{\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})}\right)\sim N\left(0,{\sigma }^2{\Vert \left(\mathit{\text{I}}-{\mathit{\text{Z}}}_{{\mathcal{G}}^0}{\left({\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^0}\right)}^{-1}{\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T\right){\mathit{\text{x}}}_j\Vert }^2\right)$ with ${\Vert \left(\mathit{\text{I}}-{\mathit{\text{Z}}}_{{\mathcal{G}}^0}{\left({\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^0}\right)}^{-1}{\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T\right){\mathit{\text{x}}}_j\Vert }^2\le {\Vert x_j\Vert }^2$. It follows that P((λ) ≠ ⁰) ≤ P(β̂(λ) ≠ β̂^(ols)) ≤ P(F^c), which is upper bounded by

$$\sum _{k<l}P\left(\left|{\alpha }_k^0-{\alpha }_l^0\right|-{\mathit{\text{q}}}_{kl}^T\mathit{\epsilon }\le 3{\mathrm{\lambda }}_2/2\right)+\sum _{k=1}^{K^0}\sum _{j\in {\mathcal{G}}_k^0}P(|{\mathit{\text{x}}}_j^T(\mathit{\text{Y}}-{\mathit{\text{X}}}^T{\widehat{\mathit{\beta }}}^{(\mathit{\text{ols}})})|>n{\mathrm{\lambda }}_1(|{\mathcal{G}}_k^0|-1)\}\le \frac{K^0(K^0-1)}{2}\mathrm{\Phi }\left(\frac{n^{1/2}(3{\mathrm{\lambda }}_2/2-{\gamma }_{min}}{2\sigma c_{min}^{-1/2}({\mathcal{G}}^0)}\right)+p\mathrm{\Phi }\left(\frac{-n{\mathrm{\lambda }}_1}{\sigma {max}_{1\le j\le p}\Vert x_j\Vert }\right),$$

where ${\mathit{\text{q}}}_{kl}={\mathit{\text{Z}}}_{{\mathcal{G}}^0}{\left({\mathit{\text{Z}}}_{{\mathcal{G}}^0}^T{\mathit{\text{Z}}}_{{\mathcal{G}}^0}\right)}^{-1}({\mathit{\text{e}}}_k-{\mathit{\text{e}}}_l)$ and e_k is the kth column of I_p. Using an inequality that $\mathrm{\Phi }(-|z|)\le \sqrt{2/\pi }{|z|}^{-1}exp(-z^2/2)$, we obtain the desired bound.

Proof of Corollary 1

It is a direct consequence of Theorem 3 and the least squares property. The proof is thus omitted.

Proof of Theorem 4

By Theorem 2, β̂(λ) = β̂^(m*)(λ). From (3), for any j ∈ _k(λ); k = 1, …, K(λ), we have |B_j(λ)| ≤ |_k(λ)| − 1. Note further that for j ∈ _k(λ), Δ_j(λ) can be rewritten as Δ_j(λ) = Σ_{k′:k′≠k} {|_k′(λ)|(Sign (α̂_k(λ) − α̂_k′(λ)) − ∇G₂(α̂_k(λ̂⁽⁰⁾) − α̂_k′(λ̂⁽⁰⁾)))}. By (17), |r_j(β̂(λ)) − nλ₁Δ_j(λ)| ≤ nλ₁|B_j(λ)|, implying $\frac{1}{n{\mathrm{\lambda }}_1}{\rho }_j(\mathbf{\lambda })\in E_k(\mathbf{\lambda })$.

For disjointness of E_k(λ)'s, assume, without loss of generality, that α̂₁(λ) < ⋯ < α̂_K(λ)(λ). For any j ∈ _k(λ), j′ ∈ _k′(λ), and k < k′, Δ_j(λ) − Δ_j′(λ) = (|_k′(λ)| + |_k′(λ)|)(k′ − k) > (|_k′(λ)| − 1) + (|_k(λ)| − 1), implying disjointness.

Table 4.

Estimated group coefficients and group sizes for breast cancer data in Section 5.3.

k	1	2	3	4	5	6	7	8	9
α̂k	−0.634	−0.507	−0.449	−0.416	−0.381	−0.370	−0.244	−0.225	−0.205
|k|	1	1	1	2	1	1	2	1	4
k	10	11	12	13	14	15	16	17	18
α̂k	−0.116	−0.059	−0.041	−0.018	−0.017	−0.017	0.006	0.039	0.048
|k|	1	2	1	1	1	3	1	6	237
k	19	20	21	22	23	24	25	26	27
α̂k	0.060	0.105	0.110	0.127	0.140	0.247	0.301	0.361	0.392
|k|	2	10	3	5	2	1	1	1	2