Homogeneity Pursuit

Abstract

This paper explores the homogeneity of coefficients in high-dimensional regression, which extends the sparsity concept and is more general and suitable for many applications. Homogeneity arises when regression coefficients corresponding to neighboring geographical regions or a similar cluster of covariates are expected to be approximately the same. Sparsity corresponds to a special case of homogeneity with a large cluster of known atom zero. In this article, we propose a new method called clustering algorithm in regression via data-driven segmentation (CARDS) to explore homogeneity. New mathematics are provided on the gain that can be achieved by exploring homogeneity. Statistical properties of two versions of CARDS are analyzed. In particular, the asymptotic normality of our proposed CARDS estimator is established, which reveals better estimation accuracy for homogeneous parameters than that without homogeneity exploration. When our methods are combined with sparsity exploration, further efficiency can be achieved beyond the exploration of sparsity alone. This provides additional insights into the power of exploring low-dimensional structures in high-dimensional regression: homogeneity and sparsity. Our results also shed lights on the properties of the fussed Lasso. The newly developed method is further illustrated by simulation studies and applications to real data. Supplementary materials for this article are available online.

1 Introduction

Driven by applications in genomics, image processing, etc., high dimensionality has become one of the major themes in statistics. See Bühlmann and van de Geer (2011) and references therein for an overview of recent developments in this area. To overcome the difficulty of fitting high dimensional models, one usually assumes that the true parameters lie in a low dimensional subspace. For example, many papers focus on sparsity, i.e., only a small fraction of coefficients are nonzero (Chen, Donoho, and Saunders 1998; Tibshirani 1996). In this article, we consider a more general type of low dimensional structure: homogeneity, i.e., the regression coefficients share the same values in their unknown clusters. A motivating example is the gene network analysis, where it is assumed that genes cluster into groups which play similar roles in molecular processes (Kim and Xing 2009; Li and Li 2010). It can be modeled as a linear regression problem with groups of homogeneous coeffficients. Similarly, in diagnostic lab tests, one often counts the number of positive results in a battery of medical tests, which implicitly assumes that their regression coefficients (impact) in the joint models are approximately the same. In spatial-temporal studies, it is not unreasonable to assume that the dynamics of neighboring geographical regions are similar, namely, their regression coefficients are clustered (Fan, Lv, and Qi 2011; Huang, Hsu, Theobald, and Breidt 2010). In the same vein, financial returns of similar sectors of industry share similar loadings on risk factors.

Homogeneity is a more general assumption than sparsity, where the latter can be viewed as a special case of the former with a large group of 0-value coefficients. In addition, the atom 0 is known to data analysts. One advantage of assuming homogeneity rather than sparsity is that it enables us to possibly select more than n variables (n is the sample size). It is well known that the sparsity-based techniques, such as the lasso, can select at most n variables. Moreover, identifying the homogeneous groups naturally provides a structure in the covariates, which can be helpful in scientific discoveries.

Regression under the homogeneity setting has been previously studied in the literature. Park, Hastie, and Tibshirani (2007) propose a two-step method. Their method performs hierarchical clustering on the predictors, cuts the obtained dendrogram at an appropriate level, and treats the cluster averages as new predictors. The fused lasso (Friedman, Hastie, Höfling, and Tibshirani 2007; Tibshirani, Saunders, Rosset, Zhu, and Knight 2005) can also be regarded as an effort of exploring homogeneity, with the assistance of neighborhoods defined according to either time or location. In this sense, our newly proposed methods are different since we do not know such a neighborhood a priori. The clustering of homogeneous coefficients is completely data-driven. For example, in the fused Lasso, where a complete ordering of the covariates is given, Tibshirani et al. (2005) use the L₁ penalty to penalize the pairwise differences of adjacent coordinates; in the case without a complete ordering, they suggest penalizing the pairs of ‘neighboring’ nodes in the sense of a general distance measure. Bondell and Reich (2008) propose the method OSCAR where a special octagonal shrinkage penalty is applied to each pair of coordinates to promote equal-value solutions. Shen and Huang (2010) develop an algorithm called Grouping Pursuit, where they use the truncated L₁ penalty to penalize the pairwise differences for all pairs of coordinates. In an extension, Zhu, Shen, and Pan (in press) consider simultaneous grouping pursuit and feature selection by including additional truncated L₁ penalties on the individual coefficients. Yang et al. (2012) explore simultaneous feature grouping and selection with the assistance of an undirected graph by penalizing the pairwise difference for each pair of coordinates that are connected by an edge in the graph. All the aforementioned methods either depend on a known ordering or graph of the covariates, which is sometimes not available, or use exhaustive pairwise penalties, which increase the computational complexity. Yang and He (2012) consider the homogeneity across coefficients of different percentile levels in quantile regression, and propose a Bayesian framework by using shrinkage priors to promote homogeneity. Although similar ideas may be applied to regression models, their settings are very different from ours, and there are no existing results on feature grouping for their method.

In this article, we propose a new method called Clustering Algorithm in Regression via Data-driven Segmentation (CARDS) to explore homogeneity. The main idea of CARDS is to take advantage of a preliminary estimate without homogeneity structure and to shrink those coefficients, that are estimated “closely”, further towards each other to achieve homogeneity. In the basic version of CARDS, we first build an ordering of covariates from the preliminary estimate and run a penalized least squares afterwards with fused penalties in the new ordering. The number of penalty terms is only (p − 1), compared to p(p − 1)/2 in the exhaustive pairwise penalties. On the other hand, an advanced version of CARDS builds an “ordered segmentation” on the covariates, which can be viewed as a generalized ordering, and imposes “hybrid pairwise penalties”, which can be viewed as a generalization of fused penalties. This version of CARDS tolerates possible misorderings in the preliminary estimate better and is thus more robust. Compared with other existing methods for homogeneity exploration, CARDS can successfully deal with the case of unordered covariates. At the same time, it avoids using exhaustive pairwise penalties and can be computationally more efficient than the Grouping Pursuit and OSCAR.

We study CARDS in details by providing some theoretical analysis. It reveals that the sum of squared errors of estimated coefficients is O_p(K/n), where K is the number of true homogeneous groups. Therefore, the smaller the number of true groups is, the better precision it can achieve. In particular, when K = p, there is no homogeneity to explore and the result reduces to the case without grouping. Moreover, in order to exactly recover the true groups with high probability, the minimum signal strength (the gaps between different groups) is of the order ${max}_k\{\sqrt{\mid A_k\mid log(p)/n}\}$, where |A_k|’s are sizes of true groups. In addition, the asymptotic normality of our proposed CARDS estimator is established, which reveals better estimation accuracy than that without homogeneity exploration. Furthermore, our results can be combined with the sparsity-based results to provide additional insights into the power of exploring low-dimensional structure in high-dimensional regression: homogeneity and sparsity. As a byproduct, our analysis on the basic version of CARDS also establishes a framework for analyzing the fused type of penalties, which is new to our knowledge.

Throughout this paper, we consider the following linear regression setting

$$\mathit{y}=\mathbf{X}{\mathit{\beta }}^0+\mathit{\epsilon },$$ (1)

where X = (x₁, · · ·, x_p) is an n × p design matrix, y = (y₁, · · ·, y_n)^T is an n × 1 vector of response, ${\mathit{\beta }}^0={({\beta }_1^0,\cdots ,{\beta }_p^0)}^T$ denotes the true parameters of interest, and ε = (ε₁, · · ·, ε_n)^T contains independently and identically distributed noises with E(ε_i) = 0 and $E({\epsilon }_i^2)={\sigma }^2$. We assume further that there is a partition of {1, 2, · · ·, p} denoted as = (A₀, A₁, · · ·, A_K) such that

$${\beta }_j^0={\beta }_{A,k}^0\text{for}\text{all}j\in A_k,$$ (2)

where ${\beta }_{A,k}^0$ is the common value shared by all regression coefficients whose indices are in A_k. By default, ${\beta }_{A,0}^0=0$, so A₀ is the group of 0-value coefficients. This allows us to explore homogeneity and sparsity simultaneously. Write ${\mathit{\beta }}_A^0={({\beta }_{A,1}^0,\cdots ,{\beta }_{A,K}^0)}^T$. Without loss of generality, we assume ${\beta }_{A,1}^0<{\beta }_{A,2}^0<\cdots <{\beta }_{A,K}^0$.

Our theory and methods are stated for the standard least-squares problem although they can be adapted to other more sophisticated models. For example, when forecasting housing appreciation in the United States (Fan et al. 2011), one builds the spatial-temporal model

$$Y_{it}={\mathbf{X}}_{it}^T{\mathit{\beta }}_i+{\epsilon }_{it},$$ (3)

in which i indicates a spatial location and t indicates time. It is expected that ${\mathit{\beta }}_i^{\prime }$s are approximately the same for neighboring zip codes i and this type of homogeneity can be explored in a similar fashion. Similarly, when Y_it represents the return of a stock and X_it = X_t stands for common market risk factors, one can assume a certain degree of homogeneity within each sector of industry; namely, the factor loading vector β_i is approximately the same for stocks belonging to the same sector of industry.

Throughout this paper, ℝ denotes the set of real numbers and ℝ^p denotes the p-dimensional real Euclidean space. For any a, b ∈ ℝ, a ∨ b denotes the maximum between a and b. For any positive sequences {a_n} and {b_n}, we write a_n ≫ b_n if a_n/b_n → ∞ as n → ∞. Given 1 ≤ q < ∞, for any vector x, ||x||_q = (Σ_j |x_j|^q)^1/q denotes the L_q-norm of x and ||x||_∞ = max_j{|x_j|}. For any matrix M, ||M||_q = max_x:||x||q=1 ||Mx||_q denotes the matrix L_q-norm of M. In particular, ||M||_∞ is the maximum absolute row sum of M. We omit the subscript q when q = 2. ||M||_max = max_i,j{|M_ij|} denotes the matrix max norm. When M is symmetric, λ_max(M) and λ_min(M) denote the maximum and minimum eigenvalues of M, respectively.

The rest of the paper is organized as follows. Section 2 describes CARDS, including the basic, advanced and shrinkage versions. Section 3 studies theoretical properties of the basic version of CARDS, and Section 4 analyzes the advanced and shrinkage versions. Sections 5 and 6 present the results of simulation studies and real data analysis, respectively. Some concluding remarks are given in Section 7. Proofs can be found in the Appendix and supplemental materials.

2 CARDS: a data-driven pairwise shrinkage procedure

2.1 Basic version of CARDS

Without considering the homogeneity assumption (2), there are many methods available for fitting model (1). Let β̃ be such a preliminary estimator. A simple idea to generate homogeneity is as follows: first, rearrange the coefficients in β̃ in the ascending order; second, group together those adjacent indices whose coefficients in β̃ are close to each other; finally, force indices in each estimated group to share a common coefficient and refit model (1). A problem of this naive procedure is how to group the indices. Alternatively, we can run a penalized least squares to simultaneously extract the grouping structure and estimate coefficients. To shrink coefficients of adjacent indices (after reordering) towards homogeneity, we can add fused penalties, i.e., {|β_i+1 − β_i|, i = 1, · · ·, p − 1} are penalized. This leads to the following two-stage procedure:

• Preordering: Construct the rank statistics {τ(j) : 1 ≤ j ≤ p} such that β̃_τ(j) is the j-th smallest value in {β̃_i, 1 ≤ i ≤ p}, i.e.,

  $${\stackrel{\sim }{\beta }}_{\tau (1)}\le {\stackrel{\sim }{\beta }}_{(2)}\le \cdots \le {\stackrel{\sim }{\beta }}_{\tau (p)}.$$ (4)
• Estimation: Given a folded concave penalty function p_λ(·) (Fan and Li 2001) with a regularization parameter λ, the final estimate is given by

  $$\widehat{\mathit{\beta }}=arg\underset{\mathit{\beta }}{min}\left\{\frac{1}{2n}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+\sum _{j=1}^{p-1}{p}_{\lambda }(\mid {\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}\mid )\right\}.$$ (5)

We call this two-stage procedure bCARDS (basic CARDS).

At the first stage, bCARDS establishes a data-driven rank mapping τ(·) based on the preliminary estimator β̃. At the second stage, only “adjacent” coefficient pairs in the order τ are penalized, resulting in only (p −1) penalty terms in total. In addition, (5) does not require that β_τ(j) ≤ βτ_(j+1). This allows coordinates in β̂ to have a different order of increasing values from that in β̃.

With an appropriately large tuning parameter λ, β̂ is a piecewise constant vector in the order τ and consequently its elements have homogeneous groups. In Section 3, we shall show that, if τ is consistent with the order of β⁰, that is,

$${\beta }_{\tau (1)}^0\le {\beta }_{\tau (2)}^0\le \cdots \le {\beta }_{\tau (p)}^0,$$ (6)

then under some regularity conditions, β̂ can consistently estimate the true coefficient groups of β⁰ with high probability.

When p_λ(·) is a folded-concave penalty function (e.g. SCAD (Fan and Li 2001), MCP (Zhang 2010)), (5) is a non-convex optimization problem. It is generally challenging to find the global minimizer. The local linear approximation (LLA) algorithm can be applied to find a local minimizer for any fixed initial solution; see Zou and Li (2008); Fan, Xue, and Zou (2012) and references therein for details. The coupling of the concave convex procedure (CCCP) can also be applied to produce a local minimizer; see Kim, Choi, and Oh (2008); Wang, Kim, and Li (in press) for a detailed explanation of CCCP.

2.2 Advanced version of CARDS

To guarantee the success of bCARDS, (6) is an essential condition. It requires that whenever ${\beta }_i^0<{\beta }_j^0$, τ (i) < τ(j) must hold. This imposes fairly strong conditions on the preliminary estimator β̃. For example, (6) can be violated if ||β̃ − β⁰||_∞ is larger than the minimum gap between groups. To relax such a restrictive requirement, we now introduce an advanced version of CARDS, where the main idea is to use less information from β̃ and to add more penalty terms in (5).

We first introduce the ordered segmentation, which can be viewed as a generalized ordering. Note that each rank mapping τ in bCARDS actually defines a partition of {1, · · ·, p} into p disjoint sets B₁, · · ·, B_p with B_j = {τ(j)} being a singleton. Similarly, we may divide {1, · · ·, p} into L(≤p) disjoint sets B₁, · · ·, B_L, where B_l’s are not necessarily singletons. We call such B_l’s segments. The segments B₁, · · ·, B_L are ordered, but the ordering of coordinates within each segment is not defined. This is similar to letter grades assigned to a course. A formal definition is as follows:

Definition 2.1

For an integer 1 ≤ L ≤ p, the mapping ϒ : {1, · · ·, p} → {1, · · ·, L} is called an ordered segmentation if the sets B_l ≡ {1 ≤ j ≤ p : ϒ(j) = l}, 1 ≤ l ≤ L, form a partition of {1, · · ·, p}.

When L = p, ϒ is a one-to-one mapping and it defines a complete ordering.

Note that, in the basic version of CARDS, the preliminary estimator β̃ produces a complete rank mapping τ. Now in the advanced version of CARDS, instead of extracting a complete ordering, we only extract an ordered segmentation ϒ from β̃. The analogue is similar to grading an exam: overall score rank (percentile rank) versus letter grade. Let δ > 0 be a predetermined parameter. First, obtain the rank mapping τ as in (4) and find all indices i₂ < i₃ < · · ·< i_L such that the gaps

$${\stackrel{\sim }{\beta }}_{\tau (j)}-{\stackrel{\sim }{\beta }}_{\tau (j-1)}>\delta ,j=i_2,\cdots ,i_L.$$

Then, construct the segments

$$B_l=\{\tau (i_l),\tau (i_l+1),\cdots ,\tau (i_{l+1}-1)\},l=1,\cdots ,L,$$ (7)

where i₁ = 1 and i_L+1 = p + 1. This process is indeed similar to the letter grades that we assign to a course. The intuition behind this construction is that when β̃_τ(j+1) ≤ β̃_τ(j) + δ, i.e., the estimated coefficients of two “adjacent coordinates” differ by only a small amount, we do not trust the ordering between them and group them into a same segment. Compared to the complete ordering τ, the ordered segments {B₁, · · ·, B_L} utilize less information from β̃ and hence are less sensitive to the estimation error in β̃.

Given an ordered segmentation τ, we wish to take advantage of the order of segments B₁, · · ·, B_L and at the same time allow flexibility of order shuffling within each segment. Towards this goal, we introduce the hybrid pairwise penalty.

Definition 2.2

Given a penalty function p_λ(·) and tuning parameters λ₁ and λ₂, the hybrid pairwise penalty corresponding to an ordered segmentation ϒ is

$$P_{\mathrm{\Upsilon },{\lambda }_1{\lambda }_2}(\mathit{\beta })=\sum _{l=1}^{L-1}\sum _{i\in B_l,j\in B_{l+1}}{p}_{{\lambda }_1}(\mid {\beta }_i-{\beta }_j\mid )+\sum _{l=1}^L\sum _{i,j\in B_l}{p}_{{\lambda }_2}(\mid {\beta }_i-{\beta }_j\mid ).$$ (8)

In (8), we call the first part between-segment penalty and the second part within-segment penalty. The within-segment penalty penalizes all pairs of indices in each segment, hence, it does not rely on any ordering within the segment. The between-segment penalty penalizes pairs of indices from two adjacent segments, and it can be viewed as a “generalized” fused penalty on segments.

When L = p, each B_l is a singleton and (8) reduces to the fused penalty in (5). On the other hand, when L = 1, namely, no prior information about β, there is only one segment B₁ = {1, · · ·, p}, and (8) reduces to the exhaustive pairwise penalty

$$P_{\lambda }^{TV}(\mathit{\beta })=\sum _{1\le i,j\le p}{p}_{\lambda }(\mid {\beta }_i-{\beta }_j\mid ).$$ (9)

It is also called the total variation penalty (Harchaoui and Lévy-Leduc 2010), and the case with p_λ(·) being a truncated L₁ penalty is studied in Shen and Huang (2010). Thus, the penalty (8) is a generalization of both the fused penalty and the total variation penalty, which explains the name “hybrid”.

The main motivation of introducing the hybrid pairwise penalty is to provide a set of intermediate versions between the fused penalty and the total variation penalty. When using pairwise penalties to promote homogeneity, we need to penalize “enough” pairs to guarantee that all true groups can be exactly recovered when the signal-to-noise ratio is sufficiently large. Given a consistent ordering, the fused penalty contains “just enough” pairs; but when the ordering is inconsistent, we have to penalize more pairs to achieve the aforementioned exact-group-recovery (see Section 2.3 for a numerical example). However, it may not be a good choice to include all pairs, i.e., using the total variation penalty, as the large number of redundant pairs can result in statistical and computational inefficiency. The hybrid penalty is designed aiming to include “just enough” pairs that adapt to the available “partial” ordering information of an ordered segmentation.

Now, we discuss how the requirement (6) can be relaxed. If we let B_j = {τ(j)}, then (6) can be written equivalently as ${max}_{i\in B_j}{\beta }_i^0\le {min}_{i\in B_{j+1}}{\beta }_i^0$, for 1 ≤ j ≤ p −1. This definition can be generalized to the case B_j’s are not singletons.

Definition 2.3

An ordered segmentation ϒ preserves the order of β⁰ if ${max}_{j\in B_l}{\beta }_j^0\le {min}_{j\in B_{l+1}}{\beta }_j^0$, for l = 1, · · ·, L − 1.

In the construction (7), even if (6) does not hold, it is still possible that the resulting ϒ preserves the order of β⁰. Consider a toy example where p = 4, and ${\beta }_{\tau (1)}^0={\beta }_{\tau (2)}^0={\beta }_{\tau (4)}^0<{\beta }_{\tau (3)}^0$ so that {τ (1), τ (2), τ (4)} and {τ (3)} are two true homogeneous groups in β⁰. Here τ ranks the coordinate τ(3) ahead of τ(4) based on the preliminary estimator β̃, but ${\beta }_{\tau (3)}^0>{\beta }_{\tau (4)}^0$. So, τ fails to give a consistent ordering. However, as long as β̃_τ(4) ≤ β̃_τ(3)+δ, τ(3) and τ(4) are grouped into the same segment in (7), say, B₁ = {τ (1), τ (2)} and B₂ = {τ (3), τ (4)}. Then ϒ still preserves the order of β⁰ according to Definition 2.3.

Now we formally introduce the advanced version of Clustering Algorithm in Regression via Data-driven Segmentation (aCARDS). It consists of three steps, where the first two steps are very similar to the way that we assign letter grades based on scores of an exam.

• Preliminary Ranking: Given a preliminary estimate β̃, generate the rank statistics {τ(j) : 1 ≤ j ≤ p} such that β̃_τ(1) ≤ β̃_τ(2) ≤ · · · ≤ β̃_τ(p).

• Segmentation: For a tuning parameter δ > 0, construct an ordered segmentation ϒ as described in (7).

• Estimation: For tuning parameters λ₁ and λ₂, compute the solution β̂ that minimizes

$$Q_n(\mathit{\beta })=\frac{1}{2n}{\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2+P_{\mathrm{\Upsilon },{\lambda }_1,{\lambda }_2}(\mathit{\beta }).$$ (10)

We call this procedure aCARDS (advanced CARDS).

In Section 4, we shall show that if ϒ preserves the order of β⁰, under certain conditions, β̂ recovers the true homogeneous groups of β⁰ with high probability. Therefore, to guarantee the success of aCARDS, we need the existence of a δ > 0 for the preliminary estimator β̃ such that the associated ϒ preserves the order of β⁰. The above toy example shows that even when (6) fails, this condition can still hold. So aCARDS requires weaker conditions on β̃ than bCARDS. This is due to that the hybrid penalty contains penalty terms corresponding to more pairs of indices, hence, it is more robust to mis-ordering in τ. In fact, bCARDS is a special case of aCARDS with δ = 0.

2.3 Comparison of two versions of CARDS

In this subsection, we first use a numerical example to compare bCARDS and aCARDS. It reveals how the ordered segmentation and hybrid pairwise penalty (8) play a role in aCARDS. We then discuss how to choose between two versions of CARDS in real data analysis.

We generate a data set with p = 40 predictors and n = 100 samples. The predictors are divided into two homogeneous groups, each of size 20. Let ${\beta }_j^0=-0.2$ for j in Group 1 and ${\beta }_j^0=0.2$ for j in Group 2. X₁, · · ·, X_n are generated independently and identically from N_p(0, I), and $Y_i={\mathbf{X}}_i^T{\mathit{\beta }}^0+{\epsilon }_i$ for 1 ≤ i ≤ n, where ε₁, ···, ε_n are independent noises with a standard normal distribution. In aCARDS, we take the ordinary least squares (OLS) estimator as the preliminary estimator. Figure 1 plots the sorted OLS coefficients for a realization. The estimated rank is not exactly consistent with the order of β⁰ since the predictors τ(17) and τ(18), which belong to Group 2, are mistakenly ranked ahead of some predictors in Group 1. If we use only the fused penalty, the terms that involve τ(17) and τ(18) are

Figure 1.

Illustration of the hybrid pairwise penalty and the aCARDS algorithm. Top panel: OLS coefficients and the associated ordered segmentation. Red dots and blue crosses represent predictors from Group 1 and Group 2, respectively. Bottom panel: Solution paths of bCARDS (left) and aCARDS (right) under misranking. The ranking and ordered segmentation are the same as in the top panel. For bCARDS, the horizontal axis represents the parameter λ. For aCARDS, the horizontal axis represents the between-segment parameter λ₁, where we fix the within-segment parameter λ₂ = 0.02. The vertical axis represents the estimated 40 regression coefficients, which are shrunk towards homogeneity (as the figures do not start from the smallest λ, we do not see initial 40 regression coefficients).

$$p_{\lambda }(\mid {\beta }_{\tau (16)}-{\beta }_{\tau (17)}\mid )+p_{\lambda }(\mid {\beta }_{\tau (17)}-{\beta }_{\tau (18)}\mid )+p_{\lambda }(\mid {\beta }_{\tau (18)}-{\beta }_{\tau (19)}\mid ).$$

There are no penalty terms to shrink the coefficients of τ(17) and τ(18) towards being equal to the coefficients of other predictors in Group 2. Now, suppose that we extract an ordered segmentation from the OLS coefficients by taking δ = 0.3; see Figure 1. Since it allows for arbitrary order reshuffling within the segment B₄, this ordered segmentation preserves the order of β⁰, i.e., Definition 2.3 is satisfied. The hybrid pairwise penalty associated with this segmentation includes terms

$$p_{{\lambda }_1}(\mid {\beta }_{\tau (17)}-{\beta }_{\tau (23)}\mid )+p_{{\lambda }_1}(\mid {\beta }_{\tau (18)}-{\beta }_{\tau (23)}\mid )$$

between segments B₄ and B₅. So aCARDS will shrink the coefficients of τ(17) and τ(18) towards being equal to the coefficient of τ(23), a predictor in Group 2. Moreover, there are terms such as

$$p_{{\lambda }_1}(\mid {\beta }_{\tau (23)}-{\beta }_{\tau (24)}\mid )+p_{{\lambda }_1}(\mid {\beta }_{\tau (23)}-{\beta }_{\tau (25)}\mid )+p_{{\lambda }_1}(\mid {\beta }_{\tau (23)}-{\beta }_{\tau (26)}\mid )$$

between segments B₅ and B₆. So aCARDS will also shrink the coefficient of τ(23) towards being equal to the coefficients of other predictors in Group 2. Eventually, aCARDS will shrink the coefficients of τ(17) and τ(18) towards being equal to the coefficients of many other predictors in Group 2. This example explains how the ordered segmentation and hybrid penalty help overcome issues caused by misranking in the preliminary estimator.

To better illustrate the effects of fused penalty and hybrid penalty under misranking, we fix the estimated rank and ordered segmentation from above, and compute the solution paths of both bCARDS and aCARDS. Note that the penalty terms in both (5) and (8) are now fixed (hence we do not need the parameter δ in aCARDS). For bCARDS, we let λ vary. For aCARDS, we set the within-segment parameter λ₂ = 0.02 and let the between-segment parameter λ₁ vary. Figure 1 displays the solution paths. We see that although bCARDS does not include the true grouping in the solution path owing to misranking, aCARDS still achieves the true grouping, which is a benefit of the hybrid penalty.

In practical data analysis, we need not differentiate between two versions of CARDS, but the tuning parameter selection process automatically tells us which version to use. This is because bCARDS is a special case of aCARDS with δ = 0. We only need to include 0 in the candidates of the parameter δ and select δ in a data-driven manner (e.g., AIC, BIC and GCV). We call the resulting method CARDS, which involves a data-driven selection between bCARDS and aCARDS.

2.4 CARDS under sparsity

In applications, we may need to explore homogeneity and sparsity simultaneously. Often the preliminary estimator β̃ takes into account the sparsity, namely it is obtained with a penalized least-squares method (Fan and Li 2001; Tibshirani et al. 2005) or sure independence screening (Fan and Lv 2008). Suppose β̃ has the sure screening property, i.e., S₀ ⊂ S̃ with high probability, where S̃ and S₀ denote the support of β̃ and β⁰, respectively. We modify CARDS as follows: in the first two steps, using the non-zero elements of β̃, we can similarly construct a data-driven hybrid penalty only on coefficients of variables in S̃; in the third step, we fix β̂_S̃^c = 0 and obtain β̂_S̃ by minimizing the following penalized least squares

$$Q_n^{\mathit{sparse}}(\mathit{\beta })=\frac{1}{2n}{\Vert \mathbf{y}-{\mathbf{X}}_{\stackrel{\sim }{S}}{\mathit{\beta }}_{\stackrel{\sim }{S}}\Vert }^2+P_{\mathrm{\Upsilon },{\lambda }_1,{\lambda }_2}({\mathit{\beta }}_{\stackrel{\sim }{S}})+\sum _{j\in \stackrel{\sim }{S}}{p}_{\lambda }(\mid {\beta }_j\mid ),$$ (11)

where X_S̃ is the submatrix of X restricted to columns in S̃. In (11), the second term is the hybrid penalty to encourage homogeneity among coefficients of variables already selected in β̃, and the third term is the element-wise penalty to help further filter out falsely selected variables. We call this modified version sCARDS (shrinkage CARDS).

3 Analysis of the basic CARDS

In this section, we analyze theoretical properties of bCARDS. Due to the limited space, we state the results here and only prove Theorems 3.1–3.3 in the Appendix, leaving the rest of the proofs to the online supplemental materials of this paper.

3.1 Heuristics

We first provide some heuristics on why taking advantage of the homogeneity helps reduce the estimation error ||β̂ − β⁰||. Consider an ideal case of orthogonal design X^TX = nI_p (necessarily p ≤ n). The ordinary least-squares estimator β̂^ols = (X^TX)⁻¹X^Ty has the decomposition

$${\widehat{\beta }}_j^{\mathit{ols}}={\beta }_j^0+{\epsilon }_j,{\epsilon }_j\stackrel{i.i.d.}{\sim }N(0,n^{-1}),j=1,\cdots ,p.$$

It is clear by the square-root law that $\Vert {\widehat{\mathit{\beta }}}^{\mathit{ols}}-{\mathit{\beta }}^0\Vert =O_P(\sqrt{p/n})$. Now, if there are K homogeneous groups in β⁰ and that we know the true groups, the original model (1) can be rewritten as

$$\mathit{y}={\mathbf{X}}_A{\mathit{\beta }}_A^0+\mathit{\epsilon },$$

where ${\mathit{\beta }}_A^0={({\beta }_{A,1}^0,\cdots ,{\beta }_{A,K}^0)}^T$ contains distinct values in β⁰, and X_A = (x_A,1, · · ·, x_A,K) is such that x_A,k = Σ_j∈Ak x_j. The corresponding ordinary least-squares estimator ${\widehat{\mathit{\beta }}}_A^{\mathit{ols}}={({\mathbf{X}}_A^T{\mathbf{X}}_A)}^{-1}{\mathbf{X}}_A^T\mathit{y}$ has the decomposition

$${\widehat{\beta }}_{A,k}^{\mathit{ols}}={\beta }_{A,k}^0+{\overline{\epsilon }}_k,{\overline{\epsilon }}_k’\mathrm{s}\text{are}\text{independent},{\overline{\epsilon }}_k~N(0,n^{-1}{\mid A_k\mid }^{-1}).$$ (12)

Here ${\overline{\epsilon }}_k=\frac{1}{\mid A_k\mid }{\sum }_{j\in A_k}{\epsilon }_j$ is the noise averaged over group k. The oracle estimator β̂^oracle is defined such that ${\widehat{\beta }}_j^{\mathit{oracle}}={\widehat{\beta }}_{A,k}^{\mathit{ols}}$ for all j ∈ A_k. Then, by the square-root law,

$$\begin{array}{l}{\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert }^2=\sum _{k=1}^K\mid A_k\mid {\mid {\widehat{\beta }}_{A,k}^{\mathit{ols}}-{\beta }_{A,k}^0\mid }^2\\ =O_p\left(\sum _{k=1}^K\mid A_k\mid ·n^{-1}{\mid A_k\mid }^{-1}\right)=O_p(K/n),\end{array}$$

which immediately implies that $\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n})$.

The surprises of the results are two fold. First, ||β̂^oracle − β⁰|| has the rate convergence $\sqrt{K/n}$ instead of $\sqrt{p/n}$. The point is that in (12) the noises are averaged, thanks to exploiting homogeneity, and consequently ${\beta }_{A,k}^0$ is estimated more accurately. The second surprise is that the rate has nothing to do with the sizes of true homogeneous groups. No matter whether we have K groups of equal size, or one dominating group with other (K − 1) small groups, the rate is always the same for the oracle estimator.

3.2 Notations and regularity conditions

Let be the subspace of ℝ^p defined by

$${\mathcal{M}}_A=\{\mathit{\beta }\in {ℝ}^p:{\beta }_i={\beta }_j,\text{for}\text{any}i,j\in A_k,1\le k\le K\}.$$

For each β ∈ , we can always write Xβ = X_Aβ_A, where X_A = (x_A,1, · · ·, x_A,K) is an n × K matrix such that its k-th column x_A,k = Σ_j∈Ak x_j, and β_A is a K-dimensional vector with its k-th component β_A,k being the common coefficient in group A_k. Define the matrix D = diag(|A₁|^1/2, · · ·, |A_K|^1/2). We introduce the following conditions on the design matrix X:

Condition 3.1

$\Vert {\mathbf{x}}_j\Vert =\sqrt{n}$, for 1 ≤ j ≤ p. The eigenvalues of the K × K matrix $\frac{1}{n}{\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1}$ are bounded below by c₁ > 0 and bounded above by c₂ > 0.

In the case of orthogonal designs, i.e., X^TX = nI_p, the matrix $\frac{1}{n}{\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1}$ simplifies to I_K, and c₁ = c₂ = 1.

Let ρ(t) = λ⁻¹p_λ(t) and ρ̄(t) = ρ′(|t|)sgn(t). We assume that the penalty function p_λ(·) satisfies the following condition.

Condition 3.2

p_λ(·) is a symmetric function and it is non-decreasing and concave on [0, ∞). ρ′(t) exists and is continuous except for a finite number of t with ρ′(0+) = 1. There exists a constant a > 0 such that ρ(t) is a constant for all |t| ≥ aλ.

We also assume that the noise vector ε = (ε₁, · · ·, ε_p)^T has sub-Gaussian tails.

Condition 3.3

For any vector a ∈ ℝⁿ and x > 0, P(|a^Tε| > ||a||x) ≤ 2e^−c₃x₂, where c₃ is a positive constant.

Given the design matrix X, let X_k be its submatrix formed by including only columns in A_k, for 1 ≤ k ≤ K. For any vector v ∈ ℝ^q, let $\text{DC}(\mathbf{v})={max}_{1\le i\le q}\mid v_i-q^{-1}{\sum }_{j=1}^q{v}_j\mid$ be the “deviation from centrality”. Define

$${\sigma }_k={\lambda }_{max}(\frac{1}{n}{\mathbf{X}}_k^T{\mathbf{X}}_k)\text{and}{\nu }_k=\underset{\mathit{\mu }\in {\mathcal{M}}_A:\Vert \mathit{\mu }\Vert =1}{max}\text{DC}(\frac{1}{n}{\mathbf{X}}_k^T\mathbf{X}\mathit{\mu }),$$ (13)

where λ_max(·) denotes the largest eigenvalue operator. In the case of orthogonal design, σ_k = 1 and ν_k = 0. Let

$$b_n=\frac{1}{2}\underset{1\le k\le l\le K}{min}\mid {\beta }_{A,k}^0-{\beta }_{A,l}^0\mid$$

be half of the minimum gap between groups in β⁰, and λ = λ_n the tuning parameter in the penalty function.

3.3 Properties of bCARDS

When the true groups A₁, · · ·, A_K are known, the oracle estimator is

$${\widehat{\mathit{\beta }}}^{\mathit{oracle}}=arg\underset{\mathit{\beta }\in {\mathcal{M}}_A}{min}\left\{\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2\right\}.$$

Theorem 3.1

Suppose Conditions 3.1–3.3 hold, K = o(n), and the preliminary estimator β̃ generates a rank mapping τ that is consistent with the order of β⁰, i.e., (6) holds, with probability at least 1 − ε₀. If the half minimum gap between groups, b_n, satisfies that b_n > aλ_n, where a is the same as that in Condition 3.2, and

$${\lambda }_n\gg \underset{k}{max}\left\{\sqrt{{\sigma }_k\mid A_k\mid log(n\vee p)/n}+(1+{\nu }_k{\mid A_k\mid }^{1/2})\sqrt{Klog(n)/n}\right\},$$ (14)

then with probability at least 1 − ε₀ − n⁻¹K − (n ∨ p)⁻¹, the bCARDS objective function (5) has a strictly local minimizer β̂ such that

• β̂ = β̂^oracle,

• $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n})$.

Theorem 3.1 shows that bCARDS includes the oracle estimator as a strictly local minimizer, with overwhelming probability. This strong oracle property is a stronger result than the oracle property in Fan and Li (2001).

The objective function (5) in bCARDS is non-convex and may have multiple local minimizers. In practice, we apply the Local Linear Approximation algorithm (LLA) (Zou and Li 2008) to solve it: start from an initial solution β̂⁽⁰⁾ = β̂^initial; at step m, update the solution by

$${\widehat{\mathit{\beta }}}^{(m)}=arg\underset{\mathit{\beta }}{min}\left\{\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2+\sum _{j=1}^{p-1}{p}_{\lambda }^{\prime }(\mid {\widehat{\beta }}_{\tau (j+1)}^{(m-1)}-{\widehat{\beta }}_{\tau (j)}^{(m-1)}\mid )·\mid {\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}\mid \right\}.$$

Given β̂^initial, this algorithm produces a unique sequence of estimators which converge to a certain local minimizer. Next, Theorem 3.2 shows that under certain conditions, the sequence of estimators produced by the LLA algorithm converge to the oracle estimator.

Theorem 3.2

Under conditions of Theorem 3.1, suppose ρ′(λ_n) ∈ a₀ for some constant a₀ > 0, and that there exists an initial solution β̂^initial of (5) satisfying ||β̂^initial − β⁰||_∞ ≤ λ_n/2. Then with probability at least 1 − ε₀ − n⁻¹K − (n ∨ p)⁻¹, the LLA algorithm yields β̂^oracle after one iteration, and it converges to β̂^oracle after two iterations.

From Theorems 3.1 and 3.2, we conclude that bCARDS combined with the LLA algorithm yields the oracle estimator with overwhelming probability, provided that we have a good preliminary estimator β̃. Next, we discuss the choice of β̃.

Since we focus on dense problems in this section, the usual sparsity is not explicitly explored and the ordinary least squares estimator

$${\widehat{\mathit{\beta }}}^{\mathit{ols}}=arg\underset{\mathit{\beta }\in {ℝ}^p}{min}\left\{\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2\right\}$$

can be used as the preliminary estimator. The following theorem shows that it induces a rank consistent mapping with high probability.

Theorem 3.3

Suppose Condition 3.3 holds, p = O(n^α) and ${\lambda }_{min}(\frac{1}{n}{\mathbf{X}}^T\mathbf{X})\ge c_4$, where 0 < α < 1 and c₄ > 0 are constants. If $b_n>\sqrt{(2a{c}_4/c_3)log(n)/n}$ where b_n is the half of the minimum gap between groups in β⁰, then with probability at least 1 − O(n^−α), the rank mapping τ generated from β̂^ols is consistent with the order of β⁰.

When the rank mapping τ extracted from β̃ does not give a consistent order, i.e., (6) does not hold, the penalty in (5) is no longer a “correct” penalty for promoting the true grouping structure. There is no hope that local minimizers of (5) exactly recover the true groups. However, if there is not too much misranking in τ, it is still possible to control ||β̂ − β̂⁰||. Given a rank mapping τ, define

$$K^{\ast }(\tau )=\sum _{j=1}^{p-1}1\{{\beta }_{\tau (j)}^0\ne {\beta }_{\tau (j+1)}^0\}.$$

It is the number of coefficient “jumps” in β⁰ under the order given by τ. These “jumps” define subgroups $A_1^{\prime },A_2^{\prime },\cdots ,A_{K^{\ast }}^{\prime }$, each being a subset of one true group. Although different subgroups may share the same true coefficient, any two consecutive subgroups $A_k^{\prime }$ and $A_{k+1}^{\prime }$ have a gap in coefficient values. As a result, the above results apply to this subgrouping structure. The following theorem is a generalization and a direct application of the proof of Theorem 3.1 and its details are omitted.

Theorem 3.4

Suppose Conditions 3.1–3.3 hold, K^*(τ) = o(n), the half minimum gap b_n > aλ_n, and λ_n satisfies (14) with K replaced by K*(τ). Then with probability at least 1−ε₀ −n⁻¹K −(n ∨ p)⁻¹, the bCARDS objective function (5) has a strictly local minimizer β̂ such that $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K^{\ast }(\tau )/n})$.

3.4 bCARDS with the L₁ penalty

In the bCARDS formulation (5), ρ(t) can also be the L₁ penalty function ρ(t) = |t|. It can be useful to get the initial solution β̂^initial for the LLA algorithm. However, ρ(t) = |t| does not satisfy Condition 3.2. Hence, Theorem 3.1 does not apply and its associated properties requires additional studies.

We first relax the requirement that τ is consistent with the order of β⁰. Instead, we consider the case that “τ is consistent with groups in β⁰”: there exists a permutation μ on {1, · · ·, K} and 1 = i₁ < i₂ < · · · < i_K < i_K+1 = p + 1 such that for k = 1, · · ·, K,

$${\beta }_{\tau (i)}^0={\beta }_{A,\mu (k)}^0,i_k\le i\le i_{k+1}-1.$$ (15)

When μ is the identical permutation, i.e., μ(k) = k, (15) is equivalent to (6) and τ is consistent with the order of β⁰. Under the condition (15), recovering the true groups is equivalent to locating coefficient jumps in β⁰ under the order given by τ, and these jumps can have positive or negative magnitudes.

To guarantee the exact recovery of jumps, we need a joint condition on X and β⁰, it is in the same spirit of the “irrepresentability” condition in Zhao and Yu (2006) but is specifically designed for the homogeneity setting. For notation simplicity, we change the indices of groups to let μ(k) = k for all k. Note that ${\beta }_1^0<{\beta }_2^0<\cdots <{\beta }_K^0$ does not hold with these new group indices.

For k = 1, · · ·, K − 1, write $d{\beta }_{A,k}^0={\beta }_{A,k+1}^0-{\beta }_{A,k}^0$. Define the K-dimensional vector d₀ by $d_1^0=sgn(d{\beta }_{A,1}^0),d_K^0=-sgn(d{\beta }_{A,K-1}^0)$ and

$$d_k^0=sgn(d{\beta }_{A,k}^0)-sgn(d{\beta }_{A,k-1}^0),2\le k\le K-1.$$

Here d⁰ is the adjacent difference of the sign vector of jumps in β⁰. For example, suppose K = 4 and the common coefficients in 4 groups satisfy ${\beta }_{A,2}^0-{\beta }_{A,1}^0>0,{\beta }_{A,3}^0-{\beta }_{A,2}^0<0$ and ${\beta }_{A,4}^0-{\beta }_{A,3}^0>0$. Then d = (1, −2, 2, −1). Also, define the p-dimensional vector

$${\mathbf{b}}^0={\mathbf{X}}^T{\mathbf{X}}_A{({\mathbf{X}}_A^T{\mathbf{X}}_A)}^{-1}{\mathbf{d}}^0.$$

In the case of orthogonal design X^TX = nI_p, b⁰ ∈ and it has the form $b_j^0=d_k^0/\mid A_k\mid$ for j ∈ A_k. For each j ∈ A_k, let

$$A_{kj}^1=\{\tau (i)\in A_k:i\le j\},A_{kj}^2=\{\tau (i)\in A_k:i>j\}.$$

Namely, $A_{kj}^1$ and $A_{kj}^2$ contain indices in group k that are ranked above and below τ(j), respectively. Let ${\theta }_{kj}=\mid A_{kj}^1\mid /\mid A_k\mid$ be the proportion of indices in group k that are ranked above τ(j). Denote by ${\overline{b}}_{kj}=\frac{1}{\mid A_{kj}^1\mid }{\sum }_{\tau (i)\in A_{kj}^1}{b}_{\tau (i)}^0$ the average of elements in b⁰ over the indices in $A_{kj}^1$, and ${\underset{\_}{b}}_{kj}=\frac{1}{\mid A_{kj}^2\mid }{\sum }_{\tau (i)\in A_{kj}^2}{b}_{\tau (i)}^0$ the average over the indices in $A_{kj}^2$.

Condition 3.4

There exists a positive sequence {ω_n}, which can go to 0, such that for 1 ≤ k ≤ K, j ∈ A_k and j ≠ j_k+1 − 1,

$$1-{\omega }_n\ge \{\begin{array}{l}\mid {\theta }_{1j}\text{sgn}(d{\beta }_{A,1}^0)+{\mid A_1\mid }^2{\theta }_{1j}(1-{\theta }_{1j})({\overline{b}}_{1j}-{\underset{\_}{b}}_{1j})\mid ,\hfill \\ \mid (1-{\theta }_{kj})\text{sgn}(d{\beta }_{A,k-1}^0)+{\theta }_{kj}\text{sgn}(d{\beta }_{A,k}^0)+{\mid A_k\mid }^2{\theta }_{kj}(1-{\theta }_{kj})({\overline{b}}_{kj}-{\underset{\_}{b}}_{kj})\mid ,2\le k\le K-1,\hfill \\ \mid (1-{\theta }_{Kj})\text{sgn}(d{\beta }_{A,K-1}^0)+{\mid A_K\mid }^2{\theta }_{Kj}(1-{\theta }_{Kj})({\overline{b}}_{Kj}-{\underset{\_}{b}}_{Kj})\mid .\hfill \end{array}$$ (16)

In the case of orthogonal design X^TX = nI_p, b⁰ ∈ and b̄_kj − b_kj = 0 holds for all k and j ∈ A_k. Condition 3.4 reduces to

$$1-{\omega }_n\ge \{\begin{array}{l}\mid {\theta }_{1j}\text{sgn}(d{\beta }_{A,1}^0)\mid ,\hfill \\ \mid (1-{\theta }_{kj})\text{sgn}(d{\beta }_{A,k-1}^0)+{\theta }_{kj}\text{sgn}(d{\beta }_{A,k}^0)\mid ,2\le k\le K-1,\hfill \\ \mid (1-{\theta }_{Kj})\text{sgn}(d{\beta }_{A,K-1}^0)\mid .\hfill \end{array}$$

This is possible only when

$$sgn(d{\beta }_{A,k-1}^0)\ne sgn(d{\beta }_{A,k}^0),2\le k\le K-1.$$ (17)

Noting that 1/|A_k| ≤ θ_kj ≤ 1 − 1/|A_k|, the associated ω_n can be chosen as min_k{1/|A_k|} when (17) holds.

Theorem 3.5

Suppose Conditions 3.1, 3.3 and 3.4 hold, K = o(n), and the preliminary estimator β̃ generates an order τ that is consistent with groups in β⁰, i.e., (15) holds, with probability at least 1 − ε₀. If the half minimum gap b_n and the tuning parameter λ_n satisfy

$$b_n\gg \sqrt{Klog(n)/n}+{\lambda }_n{\left(\sum _{k=1}^K\frac{1}{{\mid A_k\mid }^2}\right)}^{1/2}\mathit{and}{\lambda }_n\gg {\omega }_n^{-1}\underset{k}{max}\left\{\sqrt{{\sigma }_k\mid A_k\mid log(n\vee p)/n}\right\},$$ (18)

then with probability at least 1 − ε₀ − n⁻¹K − (n ∨ p)⁻¹, the bCARDS objective function (5) with ρ(t) = |t| has a unique global minimizer β̂ such that

• β̂ ∈ ;

• $sgn({\widehat{\beta }}_{A,k+1}-{\widehat{\beta }}_{A,k})=sgn({\beta }_{A,k+1}^0-{\beta }_{A,k}^0)$, k = 1, · · ·, K − 1;

• $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n}+{\gamma }_n)$, where ${\gamma }_n={\lambda }_n{({\sum }_{k=1}^K\frac{1}{\mid A_k\mid })}^{1/2}$.

Compared to Theorem 3.1, there is an extra bias term in the estimation error ||β̂− β⁰||, p which is of order $\sqrt{Klog(n\vee p)/n}$. Moreover, to achieve the exact recovery, it requires Condition 3.4, which is restrictive. For example, in the case of orthogonal designs, it is required that all consecutive jumps (under the order given by τ) have oppositive signs.

4 Analysis of the advanced CARDS

In this section, we analyze aCARDS and its variant sCARDS. The proofs can be found in the online supplemental materials.

4.1 Properties of aCARDS

To guarantee the success of aCARDS, a key condition is that the ordered segmentation ϒ = {B₁, · · ·, B_L} defined in (7) preserves the order of β⁰ in the sense of Definition 2.3. This allows the ranking of coefficients in β̃ to deviate from that in β⁰, but not too much: for some δ > 0, whenever ${\beta }_i^0<{\beta }_j^0$, β̃_i ≤ β̃_j + δ must hold.

For given A₁, · · ·, A_K and a segmentation ϒ = {B₁, · · ·, B_L}, define

$${\varphi }_k=\mid A_k\mid /min\{{\mid A_k\mid }^3,\underset{l:B_l\cap A_k\ne \varnothing }{min}\{{\mid B_l\mid }^2\}\}.$$

Here 1/|A_k|² ≤ ϕ_k ≤ |A_k| for 1 ≤ k ≤ K.

Theorem 4.1

Suppose Conditions 3.1–3.3 hold, K = o(n), and the preliminary estimator β̃ and the tuning parameter δ_n together generate an ordered segmentation ϒ that preserves the order of β⁰, with probability at least 1−ε₀. If the half minimum gap b_n and the tuning parameters (λ_1n, λ_2n) in (10) satisfy that b_n > a max{λ_1n, λ_2n}, where a is the same as that in Condition 3.2, and

$${\lambda }_{1n}\gg \underset{k}{max}\left\{\sqrt{{\sigma }_k{\varphi }_{k}log(n\vee p)/n}+(1-{\nu }_k{\varphi }_k^{1/2})\sqrt{Klog(n)/n}\right\},$$ (19)

and

$${\lambda }_{2n}\gg \underset{k}{max}\left\{{\mid A_k\mid }^{-1}\sqrt{log(n\vee p)/n}+(1+{\nu }_k{\mid A_k\mid }^{-1})\sqrt{Klog(n)/n}\right\},$$ (20)

then with probability at least 1 − ε₀ − n⁻¹K − 2(n ∨ p)⁻¹, the aCARDS objective function (10) has a strictly local minimizer β̂ such that

• β̂ = β̂^oracle,

• $\Vert \widehat{\beta }-{\beta }^0\Vert =O_p(\sqrt{K/n})$.

Compared to Theorem 3.1, aCARDS not only imposes less restrictive conditions on β̃, but also requires a smaller minimum gap between true coefficients.

Next, we establish the asymptotic normality of the CARDS estimator. By Theorem 4.1, with overwhelming probability, aCARDS performs just like the oracle. In the oracle situation, for example, if p = 5 and there are three true groups {β₁, β₄}, {β₂} and {β₃, β₅}, the accuracy of estimating β is the same as if we know the model:

$$Y={\beta }_1(X_1+X_4)+{\beta }_2{X}_2+{\beta }_3(X_3+X_5)+\epsilon .$$

Theorem 4.2

Given a positive integer q, let {B_n} be a sequence of matrices such that B_n ∈ ℝ^q×K, ${max}_{\mathbf{v}\in {ℝ}^q:\Vert \mathbf{v}\Vert =1}{\Vert {\mathbf{X}}_A^T{({\mathbf{X}}_A^T{\mathbf{X}}_A)}^{-1}{\mathbf{B}}_n^T\mathbf{v}\Vert }_4=o(1)$ and ${\mathbf{B}}_n{\mathbf{B}}_n^T\to \mathbf{H}$, where H is a fixed q × q positive definite matrix. Suppose conditions of Theorem 4.1 hold and let β̂ be the local minimizer of the aCARDS objective function (10) given in Theorem 4.1. Then

$${\mathbf{B}}_n{({\mathbf{X}}_A^T{\mathbf{X}}_A)}^{1/2}({\widehat{\mathit{\beta }}}_A-{\mathit{\beta }}_A^0)\stackrel{d}{\to }N(\mathbf{0},\mathbf{H}),$$

where β̂_A is the K-dimensional vector of distinct values in β̂.

Theorem 4.2 states the asymptotic normality of β̂_A. Note that β̂ duplicates elements in β̂_A. We introduce the following corollary to compare the asymptotic covariance of β̂ to that of β̂^ols.

Corollary 4.1

Suppose conditions of Theorems 4.1 and 4.2 hold. Let β̂^ols and β̂ be the ordinary least squares estimator and CARDS estimator respectively. Let M_n be the p × K matrix with M_n(j, k) = (1/|A_k|^1/2)1{j ∈ A_k}. For any sequence of p-dimensional vectors {a_n},

• $v_{1n}^{-1/2}{\mathbf{a}}_n^T({\widehat{\mathit{\beta }}}^{\mathit{ols}}-{\mathit{\beta }}^0)\stackrel{d}{\to }N(0,1)$ with $v_{1n}={\mathbf{a}}_n^T{({\mathbf{X}}^T\mathbf{X})}^{-1}{\mathbf{a}}_n$;

• $v_{2n}^{-1/2}{\mathbf{a}}_n^T(\widehat{\mathit{\beta }}-{\mathit{\beta }}^0)\stackrel{d}{\to }N(0,1)$ with $v_{2n}={\mathbf{a}}_n^T{\mathbf{M}}_n{({\mathbf{M}}_n^T{\mathbf{X}}^T{\mathbf{XM}}_n)}^{-1}{\mathbf{M}}_n^T{\mathbf{a}}_n$.

Moreover, v_1n ≥ v_2n.

4.2 Properties of sCARDS

In Section 2.4, we introduced sCARDS to explore both homogeneity and sparsity. In sCARDS, given a preliminary estimator β̃ and a parameter δ, we extract segments B₁, · · ·, B_L such that ${\cup }_{l=1}^L{B}_l=\stackrel{\sim }{S}$, where S̃ is the support of β̃. Denote B₀ = {j : β̃_j = 0}. In this case, we say ϒ = {B₀, B₁, · · ·, B_L} preserves the order of β⁰ if

$$\underset{j\in B_0}{max}\mid {\beta }_j^0\mid =0\text{and}\underset{j\in B_l}{max}{\beta }_j^0\le \underset{j\in B_{l+1}}{min}{\beta }_j^0,l=1,\cdots ,L-1.$$ (21)

This implies that β̃ has the sure screening property, and on those preliminarily selected variables, the data-driven segments preserve the order of true coefficients.

Suppose there is a group of zero coefficients in β⁰, namely, = (A₀, A₁, · · ·, A_K). Let ${\mathcal{M}}_A^{\ast }$ be the subspace of ℝ^p defined by

$${\mathcal{M}}_A^{\ast }=\{\mathit{\beta }\in {ℝ}^p:{\beta }_i=0,\text{for}\text{any}i\in A_0;{\beta }_i={\beta }_j,\text{for}\text{any}i,j\in A_k,1\le k\le K\}.$$

The oracle estimator is

$${\widehat{\mathit{\beta }}}^{\mathit{oracle}}=arg\underset{\mathit{\beta }\in {\mathcal{M}}_A^{\ast }}{min}\left\{\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2\right\}.$$

Denote by S the support of β⁰, and write s = |S| and s̃ = |S̃|. Define

$$b_n^{\prime }=\frac{1}{2}min\{\mid {\beta }_j^0\mid :{\beta }_j^0\ne 0\}$$

to be the half minimum signal strength.

Theorem 4.3

Suppose Conditions 3.1–3.3 hold, s = o(n), log(p) = o(n), and the preliminary estimator β̃ and the tuning parameter δ_n together generate an ordered segmentation ϒ that preserves the order of β⁰, i.e., (21) holds, with probability at least 1 − ε₀. If $b_n^{\prime }$, b_n and the tuning parameters (λ_1n, λ_2n, λ_n) satisfy that $b_n^{\prime }=a{\lambda }_n$, b_n > a max{λ_1n, λ_2n} and

$$\begin{array}{l}{\lambda }_n\gg \sqrt{log(n\vee \stackrel{\sim }{s})/n}+{\Vert {\mathbf{X}}_{\stackrel{\sim }{S}}^T{\mathbf{X}}_S\Vert }_{2,\infty }\sqrt{Klog(n)/n},\\ {\lambda }_{1n}\gg \underset{k}{max}\left\{\sqrt{{\sigma }_k{\varphi }_{k}log(n\vee s)/n}+(1+{\nu }_k{\varphi }_k^{1/2})\sqrt{Klog(n)/n}\right\},\\ {\lambda }_{2n}\gg \underset{k}{max}\left\{{\mid A_k\mid }^{-1}\sqrt{log(n\vee s)/n}+(1+{\nu }_k{\mid A_k\mid }^{-1})\sqrt{Klog(n)/n}\right\}.\end{array}$$

Then with probability at least 1−ε₀ − n⁻¹K − (n ∨ s)⁻¹ − (n ∨ s̃)⁻¹, the sCARDS objective function (11) has a strictly local minimizer β̂ such that

• β̂ = β̂^oracle,

• $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n})$.

The preliminary estimator β̃ can be chosen, for example, as the SCAD estimator

$${\widehat{\mathit{\beta }}}^{\mathit{scad}}=argmin\left\{\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2+\sum _{j=1}^p{p}_{{\lambda }^{\prime }}(\mid {\beta }_j\mid )\right\},$$ (22)

where p_λ′(·) is the SCAD penalty function (Fan and Li 2001). The following theorem is a direct result of Theorem 2 in Fan and Lv (2011), and the proof is omitted.

Theorem 4.4

Under Conditions 3.1 and 3.3, if s = o(n), ${\lambda }_n^{\prime }\gg n^{-1/2}{[log(n)]}^2$ and $min\{\mid {\beta }_j^0\mid :{\beta }_j^0\ne 0\}\gg n^{-1/2}max\{\sqrt{logp},{\Vert \frac{1}{n}{\mathbf{X}}_{S^c}^T{\mathbf{X}}_S\Vert }_{\infty }\sqrt{logn}\}$, then with probability at least 1 − o(1), there exists a strictly local minimizer β̂^scad and δ_n = O(log(n)/n) which together generate a segmentation preserving the order of β⁰.

5 Simulation studies

We conduct numerical experiments to study the performance of two versions of CARDS, bCARDS and aCARDS, and their variant sCARDS. The goal is to investigate the performance of CARDS under different situations. Experiments 1–4 are based on the linear regression model $Y_i={\mathbf{X}}_i^T{\mathit{\beta }}^0+{\epsilon }_i$, with Experiments 1–3 exploring the homogeneity only and Experiment 4 exploring the homogeneity and sparsity simultaneously. Experiment 5 is based on the spatial-temporal model $Y_{it}={\mathbf{X}}_t^T{\mathit{\beta }}_i^0+{\epsilon }_{it}$.

In all experiments, {X_i : 1 ≤ i ≤ n} or {X_t : 1 ≤ t ≤ T} are generated independently and identically from the multivariate standard Gaussian distributions, and {ε_i : 1 ≤ i ≤ n} or {ε_it: 1 ≤ i ≤ p, 1 ≤ t ≤ T} are IID samples of N(0, 1). All results are based on 100 repetitions.

Experiment 1

Consider the linear regression model with p = 60 and n = 100. Predictors are divided into four groups, each of size 15. The true regression coefficients shared within each group are −2r, −r, r and 2r, respectively. Different values of r > 0 lead to various signal-to-noise ratios. Here we let r take values in {1, 0.8, 0.5}, corresponding to high, moderate and low signal-to-noise ratio, respectively.

We compare the performance of six different methods: Oracle, ordinary least squares (OLS), bCARDS, aCARDS, total variation (TV), fused Lasso (fLasso). Oracle is the least squares estimator knowing the true groups. aCARDS and bCARDS are described in Section 2; here we let the penalty function p_λ(·) be the SCAD penalty with a = 3.7, and take the OLS estimator as the preliminary estimator. TV uses the exhaustive pairwise penalty (9), where p_λ(·) is also the SCAD penalty with a = 3.7. The fused Lasso is based on an order generated from ranking the OLS coefficients. In this sense, the fused Lasso is essentially bCARDS with the Lasso penalty p_λ(t) = λ|t|. Tuning parameters of all these methods are selected via Bayesian information criteria (BIC).

Prediction performance of different methods is evaluated in terms of the average model error over an independent test set of size 10, 000. The model error is the prediction error subtracted by the variance of ε_it, and it better reflects the performance of statistical methods. In addition, to measure how close the estimated grouping structure approaches the true one, we introduce the normalized mutual information (NMI), which is a common measure for similarity between clusterings (Fred and Jain 2003). Suppose ℂ = {C₁, C₂, · · ·} and ⅅ= {D₁, D₂, · · ·,} are two sets of disjoint clusters of {1, · · ·, p}, define

$$\text{NMI}(ℂ,\mathbb{D})=\frac{I(ℂ;\mathbb{D})}{[H(ℂ)+H(\mathbb{D})]/2},$$

where I(ℂ; ⅅ) = Σ_k,j(|C_k ∩ D_j |/p) log(p|C_k ∩ D_j |/|C_k||D_j |) is the mutual information between ℂ and ⅅ, and H(ℂ) = Σ_k(|C_k|/p) log(|C_k|/p) is the entropy of ℂ. NMI(ℂ,ⅅ) takes values on [0, 1], and larger NMI implies the two groupings are closer. In particular, NMI = 1 means that the two groupings are exactly the same.

Figure 2 shows boxplots of the average model error and NMI for six different methods. We observe that except for the case of weak signals (r = 0.5), two versions of CARDS outperform other methods since they lead to smaller average model error and larger NMI. bCARDS is performing especially well in achieving low model errors, even in the case r = 0.5. aCARDS has a better performance in terms of NMI, which indicates that it is better in recovering the true grouping structure. The possible reason that aCARDS does not perform as well as bCARDS in model errors is that aCARDS has more tuning parameters and selection of these tuning parameters in simulations may be non-optimal.

Figure 2.

The average model error and normalized mutual information in Experiment 1, where p = 60, n = 100, and there are four equal-size coefficient groups.

Experiment 2

The setting of this experiment is the same as in Experiment 1, except that the homogeneous groups have non-equal sizes. In Experiment 2a, the predictors are divided into four groups of size 1, 15, 15 and 29. The four distinct regression coefficients are −4r, −r, r and 2r, respectively. Here, the first group is a singleton. In Experiment 2b, there is one dominating group of size 50 with a common regression coefficient −2r. The other 10 predictors have the regression coefficients $0,\frac{2}{9},\frac{4}{9},\frac{6}{9},\cdots ,2$, respectively. In both subexperiments, we take r = 1 and 0.7 to represent two levels of signal-to-noise ratio. Besides the six methods compared in Experiment 1, we also implement a data-driven selection between bCARDS and aCARDS, as described in Section 2.3. In detail, we select the parameter δ via BIC (for each value of δ, the other parameters are also selected via BIC). The resulting method is called CARDS.

Figure 3 displays results for Experiment 2a. It suggests that both bCARDS and aCARDS outperform other methods, so does CARDS. Figure 4 displays results for Experiment 2b. We see that the total variation and fused Lasso can not improve the OLS estimator in terms of the average model error. bCARDS also performs unsatisfactorily, possibly due to misranking in the preliminary estimate. But aCARDS performs much better than OLS and other methods. After a data-driven selection between bCARDS and aCARDS, the resulting method CARDS also outperforms other methods.

Figure 3.

The average model error and normalized mutual information in Experiment 2a, where p = 60, n = 100, and there are four coefficient groups of size 1, 15, 15 and 29.

Figure 4.

The average model error and normalized mutual information in Experiment 2b, where p = 60, n = 100, and there is one dominating group of size 50.

Experiment 3

We use this experiment to investigate how the performance of two versions of CARDS is affected by misranking in the preliminary estimate. The setting is the same as Experiment 1 and we fix r = 1 (so n = 100, p = 60, and there are 4 equal-size groups with true regression coefficients equal to −2, −1, 1 and 2 respectively). For each data set (X, y), we generate 11 different preliminary ranks as follows: for each σ in {1, 1.2, 1.4, · · ·, 3}, we generate z ~ N(Xβ⁰, σ²I_n) independently of y conditional on X, and then use the OLS estimator associated with z to get a preliminary rank. A larger value of σ tends to yield a “worse” preliminary rank. We use K^* defined in Section 3.3 to quantify the level of misranking. Recall that K^* is the total number of jumps in true regression coefficients under the preliminary rank, and K^* > 3 means there exists misordering in the preliminary rank. We generated 100 data sets and 11 preliminary ranks for each data set so that the results are based on 1100 repetitions. We compare the performance of bCARDS and aCARDS with that of the total variation (TV), where TV does not use any information from the preliminary rank. Figure 5 contains boxplots of the average model error as K^* changes. First, we see that two versions of CARDS are quite robust to the increase of K^* and always outperform the total variation. Second, the model error of bCARDS increases as K^* increases, which provides empirical evidence for the claim in Theorem 3.4 about the effect of misranking to bCARDS. Third, when K^* is large, aCARDS has a better performance than bCARDS, due to the fact that the hybrid pairwise penalty can tolerate a higher level of misranking than the fused penalty.

Figure 5.

The average model error in Experiment 3. The horizontal axis represents K^*, and “b”, “a” and “TV” are short for bCARDS, aCARDS and total variation.

Experiment 4

This experiment explores the homogeneity and sparsity simultaneously. Consider the linear regression model with p = 100 and n = 150. Among the 100 predictors, 60 are important ones and their coefficients are the same as those in Experiment 1. Besides, there are 40 unimportant predictors whose coefficients are all equal to 0.

We implemented sCARDS and compared its performance with different oracle estimators, Oracle, Oracle0 and OracleG, as well as ordinary least squares (OLS), SCAD, shrinkage total variation (sTV) and fused Lasso (fLasso). The three oracles are defined with different levels of prior information: the Oracle knows both the important predictors and the true groups, the Oracle0 only knows which are important predictors; and the OracleG only knows the true groups (it treats all unimportant predictors as one group with unknown coefficients). sCARDS is as described in Section 2; while implementing it, we take the SCAD estimator as the preliminary estimator. The shrinkage total variation is an extension of TV by adding both the elementwise SCAD penalty and exhaustive pairwise SCAD penalty. The fused Lasso used here has both the elementwise L₁ penalty and fused pairwise L₁ penalty.

Figure 6 displays the boxplots of average model errors for r = 1 and r = 0.7. First, by comparing model errors of the three oracles, we see a significant advantage of taking into account both homogeneity and sparsity over pure sparsity. Moreover, the results of Oracle0 and OracleG show that exploring the group structure is more important than sparsity. Second, sCARDS achieves a smaller model error than OLS, SCAD and fused Lasso. The performance of sCARDS and sTV in terms of model error are comparable. However, they are different in feature selection. Figure 7 contains frequency histograms of the number of falsely selected features for two methods. In about 17% of the repetitions, sTV fails to shrink coefficients of the 40 unimportant predictors to 0.

Figure 6.

The average model error and normalized mutual information in Experiment 4, where p = 100, n = 100, and there are 60 important predictors divided into four equal-size groups.

Figure 7.

Feature selection of sCARDS and shrinkage total variation (sTV) in Experiment 4 (left: r = 1; right: r = 0.7), where there are 40 unimportant predictors.

Experiment 5

We consider a special case of the spatial-temporal model (3), $Y_{it}={\mathbf{X}}_t^T{\mathit{\beta }}_i+{\epsilon }_{it}$, i = 1, · · ·, p, i.e., the predictors are the same for all spatial locations. There are p = 100 different locations and k = 5 common predictors (so each β_i has a dimension 5). We assume only the spatial homogeneity in the regression coefficients, that is, for each predictor j (j = 1, · · ·, 5), the coefficients, {β_i,j, 1 ≤ i ≤ 100}, are divided into four groups of equal size 25, where coefficients in the same group share a same value. In simulations, we let

$${\beta }_{i,j}={\omega }_j+(-2)I_{1\le i\le 25}+(-1)I_{26\le i\le 50}+I_{51\le i\le 75}+2I_{76\le i\le 100},1\le j\le 5,$$

where {ω_j = 0.1 × (j −1), 1 ≤ j ≤ 5} are location-independent constants. In this experiment, instead of varying the signal-to-noise ratio directly, we equivalently change T, the total number of time points.

We extend aCARDS to this model by adding the hybrid pairwise penalty on coefficients of the same predictor at different locations, and still call the method aCARDS. The total variation (TV) and fused Lasso (fLasso) can be extended to this model in a similar way. The Oracle is the maximum likelihood estimator which knows the true groups for each predictor. We aim to compare the performance of Oracle, OLS, aCARDS, total variation and fused Lasso.

Figure 8 displays the results. We see that aCARDS achieves significantly lower model errors than OLS, due to exploring homogeneity. Moreover, it has a better performance than the total variation and fused Lasso, particularly when T = 50, 80. aCARDS also estimates well the true grouping structure; when T = 50, 80, the normalized mutual information is larger than 0.95 in most repetitions.

Figure 8.

The average model error and normalized mutual information in Experiment 5, where the model is a spatial-temporal regression model with p = 100 locations and k = 5 common predictors.

6 Real data analysis

6.1 S&P500 returns

In this study, we fit a homogeneous Fama-French model for stock returns: $Y_{it}={\alpha }_i+{\mathbf{X}}_t^T{\mathit{\beta }}_i^0+{\epsilon }_{it}$, where X_t contains three Fama-French factors at time t, Y_it is the excess return of stocks and ε_it are idiosyncratic noises. We collected daily returns of 410 stocks, which were always included in the components of the S&P500 index in the period December 1, 2010 to December 1, 2011 (T = 254). We applied CARDS as in Experiment 5, except that the intercepts α_j ‘s were also penalized for sparsity. The sparsity of α_j ‘s is supported by the capital asset pricing model (CAPM) and its extension, multifactor pricing model, in financial econometric theories. The tuning parameters were chosen via generalized cross validation (GCV). Table 1 shows the number of fitted coefficient groups on three factors and the number of non-zero intercepts. We then used daily returns of the same stocks in the period December 1, 2011 to July 1, 2012 (T = 146) to evaluate the prediction error. Let ŷ_it and y_it be the fitted and observed excess returns of stock i at time t = 1, · · ·, 146, respectively. Define the discounted cumulative sum of squared estimation errors (cRSS) at time t as ${\text{cRSS}}_t={\sum }_{s=1}^t{\rho }^{\lfloor s/10\rfloor }{\sum }_i{({\widehat{y}}_{it}-y_{it})}^2$, where we take ρ = 0.95. Figure 9 shows the percentage improvement in cRSS_t of the CARDS estimator over the OLS estimator. We see that CARDS achieves a smaller discounted cRSS compared to OLS at most time points, especially in the “very-close” and “far-away” future. The North American Industry Classification System (NAICS) classifies these 410 companies into 18 different industry sectors. Figure 10(a) shows the OLS coefficients on the factor “book-to-market ratio”. We can see that stocks belonging to Sector 3 “Utilities” (29 stocks in total) have very close OLS coefficients, and 17 stocks in this sector were clustered into one group in CARDS estimator. Figure 10(b) shows the percentage improvement in cRSS_t only for stocks in this sector, where the improvement is more significant.

Table 1.

Number of groups in fitting the S&P500 data.

Fama-French factors	No. of coef. groups
“market return”	41
“market capitalization”	32
“book-to-market ratio”	56
intercept	60

Figure 9.

Comparison of the cumulative sum of squared prediction errors of the S&P500 data from December 1, 2011 to July 1, 2012. The vertical axis is the percent of relative errors between the predictions made by OLS and CARDS, defined by $100({\text{cRSS}}_t^{\mathit{ols}}-{\text{cRSS}}_t^{\mathit{cards}})/{\text{cRSS}}_t^{\mathit{ols}}$. The right panel is a zoom-in of the results for CARDS.

Figure 10.

(a)OLS coefficients on the “book-to-market ratio” factor. The x axis represents different sectors. (b)Percentage improvement of the discount cumulative sum of squared estimation errors for stocks in the sector “Utilities” (Sector 2).

6.2 Polyadenylation signals

CARDS can be easily extended to more general settings such as generalized linear models (McCullagh and Nelder 1989) although we have focused on the linear regression model so far. In this subsection, we apples CARDS to a logistic regression example. This study tried to predict polyadenylation signals (PASes) in human DNA and mRNA sequences by analyzing features around them. The data set was first used in Legendre and Gautheret (2003) and later analyzed by Liu et al. (2003), and it is available at http://datam.i2r.a-star.edu.sg/datasets/krbd/SequenceData/Polya.html. There is one training data set and five testing data sets. To avoid any platform bias, we use the training data set only. It has 4418 observations each with 170 predictors and a binary response. The binary response indicates whether a terminal sequence is classified as a “strong” or “weak” polyA site, and the predictors are features from the upstream (USE) and downstream (DSE) sequence elements. We randomly select 2000 observations to perform model estimation and use the rest to evaluate performance. Our numerical analysis consists of the following steps. Step 1 is to apply the L₁-penalized logistic regression to these 2000 observations with all 170 predictors and use AIC to select an appropriate regularization parameter. In Step 2, we use the logistic regression coefficients obtained in Step 1 as our preliminary estimate and apply sCARDS accordingly. Average prediction error (and standard error in parentheses) over 40 random splitting are reported in Table 2. We also report the average number of non-zero coefficient groups and the average number of selected features. It shows that sCARDS lead to a smaller prediction error when compared with the shrinkage total variation (sTV). In addition, the sCARDS has fewer groups of non-zero coefficients but more selected features, which implies that we can include more predictors while fixing the degrees of freedom at a small value. Note that in this example, the fused Lasso (fLasso) has a similar performance as the sCARDS. In Section 5, we remarked that the fused Lasso is essentially bCARDS with the lasso penalty p_λ(t) = λ|t|.

Table 2.

Results of the PASes data.

	sCARDS	SCAD	sTV	fLasso
Prediction Error	0.2449 (.0015)	0.2458 (.0014)	0.2757 (.0026)	0.2445 (0.0010)
No. of non-zero coef. groups	5.5000	21.6250	5.7500	5.5000
No. of selected features	73.2750	21.6250	40.3500	86.8750

7 Conclusion

In this paper, we explored homogeneity of coefficients in high-dimensional regression. We proposed a new method called Clustering Algorithm in Regression via Data-driven Segmentation (CARDS) to estimate regression coefficients and to detect homogeneous groups. The implementation of CARDS does not need any geographical information (neighborhoods, distance, graphs, etc.) a priori, which distinguishes it from other methods in similar settings and makes it more general for applications. A modification of CARDS, sCARDS, can be used to explore homogeneity and sparsity simultaneously. Our theoretical results show that better estimation accuracy can be achieved by exploring homogeneity. In particular, when the number of homogeneous groups is small, the power of exploring homogeneity and sparsity simultaneously is much larger than that of exploring sparsity only, which is also confirmed in our simulation studies.

Methodologically, CARDS has two main innovations. First, it takes advantage of a preliminary estimate by extracting from which either an estimated ranking or an estimated ordered segmentation. Second, it introduces the so-called “hybrid pairwise penalty” to adapt to available partial ordering information. The hybrid pairwise penalty not only is robust to misordering, but also avoids statistical and computational inefficiency due to penalizing too many pairs. These ideas about handling homogeneity can be applied to much broader situations than linear regression, if we combine the hybrid pairwise penalty with appropriate loss functions. For example, CARDS can be extended to generalized linear models (GLM) when homogeneity appears.

To promote homogeneity, CARDS takes advantage of a preliminary estimate. Such idea can be generalized. Instead of extracting a complete raking or an ordered segmentation, we may also apply clustering methods to coefficients of the preliminary estimate, such as k-mean algorithm or hierarchical clustering algorithm, to help construct datadriven penalties and further promote homogeneity.

This paper only considers the case where predictors in one homogeneous group have equal coefficients. In a more general situation, coefficients of predictors in the same group are close but not exactly equal. The idea of data-driven pairwise penalties still applies, but instead of using the class of folded concave penalty functions, we may need to use penalty functions which are smooth at the origin, e.g., the L₂ penalty function. Another possible approach is to use posterior-type estimators combined with, say, a Gaussian prior on the coefficients. These are beyond the scope of this paper and we leave them as future work.

A Proofs

A.1 Proof of Theorem 3.1

Introduce the mapping T : → ℝ^K, where T(β) is the K-dimensional vector whose k-th coordinate equals to the common value of β_j for j ∈ A_k. Note that T is a bijection and T⁻¹ is well-defined for any μ ∈ ℝ^K. Also, introduce the mapping T^* : ℝ^p → ℝ^K, where $T^{\ast }{(\mathit{\beta })}_k=\frac{1}{\mid A_k\mid }{\sum }_{j\in A_k}{\beta }_j$. We see that T^* = T on , and T⁻¹ ∘ T^* is the orthogonal projection from ℝ^p to . Denote μ⁰ = T (β⁰) and μ̂^oracle = T(β̂^oracle).

Denote $L_n(\mathit{\beta })=\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2$ and $P_n(\mathit{\beta })={\lambda }_n{\sum }_{j=1}^{p-1}\rho ({\beta }_{\tau (j+1)}-{\beta }_{\tau (j)})$, so that we can write Q_n(β) = L_n(β) + P_n(β). For any μ ∈ ℝ^K, let

$$L_n^A(\mathit{\mu })=\frac{1}{2n}{\Vert \mathit{y}-{\mathbf{X}}_{A\mathit{\mu }}\Vert }^2,P_n^A(\mathit{\mu })={\lambda }_n\sum _{k=1}^{K-1}\rho ({\mu }_{k+1}-{\mu }_k),$$

and define $Q_n^A(\mathit{\mu })=L_n^A(\mathit{\mu })+P_n^A(\mathit{\mu })$. Note that when τ is consistent with the order of β⁰, there exist 1 = j₁ < j₂ < · · · < j_K < j_K+1 = p + 1 such that A_k = {τ(j_k), τ(j_k + 1), · · ·, τ(j_k+1 − 1) for 1 ≤ k ≤ K. Then $Q_n(\mathit{\beta })=Q_n^A(T(\mathit{\beta }))$ and $Q_n^A(\mathit{\mu })=Q_n(T^{-1}(\mathit{\mu }))$ for any β ∈ and μ ∈ ℝ^K.

In the first part of the proof, we show $\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n})$. By definition and direct calculations,

$$\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert =\Vert \mathbf{D}({\widehat{\mathit{\mu }}}^{\mathit{oracle}}-{\mathit{\mu }}^0)\Vert ,{\widehat{\mathit{\mu }}}^{\mathit{oracle}}-{\mathit{\mu }}^0={({\mathbf{X}}_A^T{\mathbf{X}}_A)}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }.$$

Therefore, we can write

$$\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert =\Vert {({\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1})}^{-1}{\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert .$$ (23)

From Condition 3.1, $\Vert {({\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1})}^{-1}\Vert \le {(c_{1}n)}^{-1}$ and $\text{tr}({\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1})\le c_{2}nK$. By the Markov inequality, for any δ > 0,

$$P\left(\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert >\sqrt{\frac{c_{2}nK}{\delta }}\right)\le \frac{E{\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert }^2}{c_{2}nK/\delta }=\frac{\text{tr}({\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1})}{c_{2}nK/\delta }\le \delta .$$ (24)

Combining the above, we have shown that with probability at least 1 − δ, $\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert \le C{\delta }^{-1/2}\sqrt{K/n}$. This proves $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert =O_p(\sqrt{K/n})$.

Furthermore, we show a result that will be frequently used in later proofs:

$$\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert \le C\sqrt{Klog(n)/n},\text{with}\text{probability}\ge 1-n^{-1}K.$$ (25)

Write ${\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }={({\mathbf{v}}_1^T\mathit{\epsilon },\cdots ,{\mathbf{v}}_k^T\mathit{\epsilon })}^T$, where v_k = X_AD⁻¹e_k and e_k is the unit vector with 1 on the k-th coordinate and 0 elsewhere. Observing that ||v_k||² is the k-th diagonal of the matrix ${\mathbf{D}}^{-1}{\mathbf{X}}_A^T{\mathbf{X}}_A{\mathbf{D}}^{-1}$, we have $\Vert {\mathbf{v}}_k\Vert \le \sqrt{c_{2}n}$. It follows from Condition 3.3 and the union bound that

$$P\left({\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert }_{\infty }>\sqrt{c_2{c}_3^{-1}nlog(2n)}\right)\le \sum _{k=1}^{K}P\left(\Vert {\mathbf{v}}_k^T\mathit{\epsilon }\Vert >\Vert {\mathbf{v}}_k\Vert \sqrt{c_3^{-1}log(2n)}\right)\le n^{-1}K.$$

Since $\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert \le K^{1/2}{\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert }_{\infty }$,

$$\Vert {\mathbf{D}}^{-1}{\mathbf{X}}_A^T\mathit{\epsilon }\Vert \le \sqrt{c_2{c}_3^{-1}\mathit{Kn}log(2n)},\text{with}\text{probability}\ge 1-n^{-1}K.$$ (26)

Then (25) follows by combining (23) and (26).

In the second part of the proof, we show that β̂^oracle is a strictly local minimizer of Q_n(β) with probability at least 1 − ε₀ − n⁻¹ K − (n ∨ p)⁻¹. By assumption, there is an event E₁ such that $P(E_1^c)\le {\epsilon }_0$ and over the event E₁, τ is consistent with the order of β⁰. Consider the neighborhood of β⁰:

$$\mathcal{B}=\left\{\mathit{\beta }\in {ℝ}^p:\Vert \mathit{\beta }-{\mathit{\beta }}^0\Vert <2C\sqrt{Klog(n)/n}\right\}.$$

By (25), there is an event E₂ such that $P(E_2^c)\le n^{-1}K$ and over the event E₂, $\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert \le C\sqrt{Klog(n)/n}$. Hence, β̂^oracle ∈ over the event E₂.

For any β ∈ , write β^* as its orthogonal projection to . We aim to show

1. Over the eventE₁ ∩ E₂,

   $$Q_n({\mathit{\beta }}^{\ast })\ge Q_n({\widehat{\mathit{\beta }}}^{\mathit{oracle}}),\text{for}\text{any}\mathit{\beta }\in \mathcal{B},$$ (27)

   and the inequality is strict whenever β^* ≠ β̂^oracle.

2. There is an event E₃ such that $P(E_3^c)\le {(n\vee p)}^{-1}$. Over the event E₁ ∩ E₂ ∩ E₃, there exists ( ⊂ ), a neighborhood of β̂^oracle, such that

   $$Q_n(\mathit{\beta })\ge Q_n({\mathit{\beta }}^{\ast }),\text{for}\text{any}\mathit{\beta }\in {\mathcal{B}}_n,$$ (28)

   and the inequality is strict whenever β ≠ β^*.

Combining (a) and (b), Q_n(β) ≥ Q_n(β̂^oracle) for any β ∈ , a neighborhood of β̂^oracle, and the inequality is strict whenever β ≠ β̂^oracle. This proves that β̂^oracle is a strictly local minimizer of Q_n, over the event E₁ ∩ E₂ ∩ E₃.

It remains to show (a) and (b). Consider (a) first. We claim that

$$P_n^A(T^{\ast }(\mathit{\beta }))=0\text{for}\text{any}\mathit{\beta }\in \mathcal{B}.$$ (29)

To see this, for a given β ∈ , write μ = T^* (β). It suffices to check |μ_k+1 − μ_k| > aλ_n for k = 1, · · ·, K − 1. Note that $\mid {\mu }_{k+1}-{\mu }_k\mid \ge {min}_{i\in A_k,j\in A_{k+1}}\mid {\beta }_i-{\beta }_j\mid \ge {min}_{i,j}\mid {\beta }_i^0-{\beta }_j^0\mid -2{\Vert \mathit{\beta }-{\mathit{\beta }}^0\Vert }_{\infty }\ge 2b_n-2C\sqrt{Klog(n)/n}$. Since $b_n>a{\lambda }_n\gg \sqrt{Klog(n)/n}$, it is easy to see that |μ_k+1 − μ_k| > aλ_n.

Using (29), we see that $Q_n^A(T^{\ast }(\mathit{\beta }))=L_n^A(T^{\ast }(\mathit{\beta }))$, for all β ∈ . Since $Q_n^A=Q_n\circ T^{-1}$ and T⁻¹ ∘ T^* is the orthogonal projection from ℝ^p to , for any β ∈ ,

$$Q_n({\mathit{\beta }}^{\ast })=Q_n(T^{-1}\circ T^{\ast }(\mathit{\beta }))=Q_n^A(T^{\ast }(\mathit{\beta }))=L_n^A(T^{\ast }(\mathit{\beta })).$$ (30)

In particular, noting that β̂^oracle ∈ and its orthogonal projection to is itself, the above further implies

$$Q_n({\widehat{\mathit{\beta }}}^{\mathit{oracle}})=L_n^A({\widehat{\mathit{\mu }}}^{\mathit{oracle}}).$$ (31)

By definition and the fact that $\frac{{\partial }^2{L}_n^A(\mathit{\mu })}{\partial \mathit{\mu }\partial {\mathit{\mu }}^T}=\frac{1}{n}{\mathbf{X}}_A^T{\mathbf{X}}_A$ is positive definite, μ̂^oracle is the unique global minimizer of $L_n^A(\mathit{\mu })$. As a result,

$$L_n^A(T^{\ast }(\mathit{\beta }))\ge L_n^A({\widehat{\mathit{\mu }}}^{\mathit{oracle}}),$$ (32)

and the inequality is strict whenever T^*(β) ≠ μ̂^oracle, i.e., β^* ≠ T⁻¹(μ̂^oracle) = β̂^oracle. Combining (30)–(32) gives (a).

Second, consider (b). For a positive sequence {t_n} to be determined, let

$${\mathcal{B}}_n=\mathcal{B}\cap \{\mathit{\beta }:\Vert \mathit{\beta }-{\widehat{\mathit{\beta }}}^{\mathit{oracle}}\Vert \le t_n\}.$$

Since β^* is the orthogonal projection of β to , ||β − β^* || ≤ ||β − β′|| for any β′ ∈ . In particular, ||β − β^*|| ≤ ||β − β̂^oracle||. As a result, to show (28), it suffices to show

$$Q_n(\mathit{\beta })\ge Q_n({\mathit{\beta }}^{\ast }),\text{for}\text{any}\mathit{\beta }\text{such}\text{that}\Vert \mathit{\beta }-{\mathit{\beta }}^{\ast }\Vert \le t_n,$$ (33)

and the inequality is strict whenever β ≠ β^*.

To show (33), write μ = T^*(β) so that β^* = T⁻¹(μ). By Taylor expansion,

$$\begin{array}{l}{Q}_n(\mathit{\beta })-Q_n({\mathit{\beta }}^{\ast })=-\frac{1}{n}{(\mathit{y}-\mathbf{X}{\mathit{\beta }}^m)}^T\mathbf{X}(\mathit{\beta }-{\mathit{\beta }}^{\ast })+\sum _{j=1}^p\frac{\partial P_n({\mathit{\beta }}^m)}{\partial {\beta }_{\tau (j)}}({\beta }_{\tau (j)}-{\beta }_{\tau (j)}^{\ast })\\ \equiv I_1+I_2,\end{array}$$

where β^m is in the line between β and β^*. Consider I₂ first. Direct calculations yield

$$\frac{\partial P_n(\mathit{\beta })}{\partial {\beta }_{\tau (j)}}=\{\begin{array}{ll}-{\lambda }_n\overline{\rho }({\beta }_{\tau (2)}-{\beta }_{\tau (1)}),\hfill & \hfill j=1\\ {\lambda }_n\overline{\rho }({\beta }_{\tau (j)}-{\beta }_{\tau (j-1)})-{\lambda }_n\overline{\rho }({\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}),\hfill & \hfill 2\le j\le p-1\\ {\lambda }_n\overline{\rho }({\beta }_{\tau (p)}-{\beta }_{\tau (p-1)}),\hfill & \hfill j=p,\end{array}$$

where ρ̄(t) = ρ′(|t|)sgn(t) and ρ(t) = λ⁻¹p_λ(t). Plugging it into I₂ and rearranging the sum, we obtain

$$I_2={\lambda }_n\sum _{j=1}^{p-1}\overline{\rho }({\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m)[({\beta }_{\tau (j+1)}-{\beta }_{\tau (j)})-({\beta }_{\tau (j+1)}^{\ast }-{\beta }_{\tau (j)}^{\ast })].$$ (34)

When τ(j) and τ(j + 1) belong to the same group, ${\beta }_{\tau (j)}^{\ast }={\beta }_{\tau (j+1)}^{\ast }$, and hence the sign of ( ${\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m$) is the same as the sign of (β_τ(j+1) − β_τ(j)) if neither of them is 0. In addition, recall that A_k = {τ(j_k), τ(j_k + 1), · · ·, τ(j_k+1 − 1)} for all 1 ≤ k ≤ K, for some indices 1 = j₁ < j₂ < · · · < j_K < j_K+1 = p + 1. Combining the above, we can rewrite

$$I_2={\lambda }_n\sum _{k=1}^K\sum _{j=j_k}^{j_{k+1}-2}{\rho }^{\prime }(\mid {\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m\mid )\mid {\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}\mid +{\lambda }_n\sum _{k=2}^K\overline{\rho }(\mid {\beta }_{\tau (jk)}^m-{\beta }_{\tau (j_k-1)}^m\mid )[({\beta }_{\tau (j_k)}-{\beta }_{\tau (j_k-1)})-({\beta }_{\tau (j_k)}^{\ast }-{\beta }_{(j_k-1)}^{\ast })].$$

First, since β⁰ ∈ and β^* is the orthogonal projection of β to , ||β^* − β⁰|| ≤ ||β − β⁰||. Hence, β ∈ implies β^*, β^m ∈ . By repeating the proof of (29), we can show $\overline{\rho }(\mid {\beta }_{\tau (j_k)}^m-{\beta }_{\tau (j_k-1)}^m\mid )=0$ for 2 ≤ k ≤ K. So the second term in I₂ disappears. Second, in the first term of I₂, since $\mid {\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m\mid \le 2{\Vert {\mathit{\beta }}^m-{\mathit{\beta }}^{\ast }\Vert }_{\infty }\le 2{\Vert \mathit{\beta }-{\mathit{\beta }}^{\ast }\Vert }_{\infty }\le 2t_n$, it follows by concavity that ${\rho }^{\prime }(\mid {\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m\mid )\ge {\rho }^{\prime }(2t_n)$. Together, we have

$$I_2\ge {\lambda }_n\sum _{k=1}^K\sum _{j=j_k}^{j_{k+1}-2}{\rho }^{\prime }(2t_n)\mid {\beta }_{\tau (j+1)}-{\beta }_{(j)}\mid .$$ (35)

Next, we simplify I₁. Let z = z(β^m) = X^T(y − Xβ^m) and write $I_1=-\frac{1}{n}{\mathbf{z}}^T(\mathit{\beta }-{\mathit{\beta }}^{\ast })$. For any fixed k and l such that τ(l) ∈ A_k and l ≠ j_k+1 −1, let $A_{kl}^1=\{\tau (j)\in A_k:j\le l\}$ and $A_{kl}^2=\{\tau (j)\in A_k:j>l\}$. Regarding that ${\beta }_{\tau (i)}^{\ast }=\frac{1}{\mid A_k\mid }{\sum }_{j=j_k}^{j_{k+1}-1}{\beta }_{\tau (j)}$ for i ∈ A_k, we can reexpress I₁ as

$$\begin{array}{l}{I}_1=-\frac{1}{2}\sum _{k=1}^K\sum _{i=j_k}^{j_{k+1}-1}\frac{1}{n}{z}_{\tau (i)}[{\beta }_{\tau (i)}-{\beta }_{\tau (i)}^{\ast }]-\frac{1}{2}\sum _{k=1}^K\sum _{j=j_k}^{j_{k+1}-1}\frac{1}{n}{z}_{\tau (j)}[{\beta }_{\tau (j)}-{\beta }_{\tau (j)}^{\ast }]\\ =-\sum _{k=1}^K\frac{1}{2n\mid A_k\mid }\sum _{i,j=j_k}^{j_{k+1}-1}{z}_{\tau (i)}[{\beta }_{\tau (i)}-{\beta }_{\tau (j)}]-\sum _{k=1}^K\frac{1}{2n\mid A_k\mid }\sum _{i,j=j_k}^{j_{k+1}-1}{z}_{\tau (j)}[{\beta }_{\tau (j)}-{\beta }_{\tau (i)}]\\ =-\sum _{k=1}^K\frac{1}{2n\mid A_k\mid }\sum _{i,j=j_k}^{j_{k+1}-1}[z_{\tau (j)}-z_{\tau (i)}][{\beta }_{\tau (j)}-{\beta }_{\tau (i)}]\\ =-\sum _{k=1}^K\frac{1}{n\mid A_k\mid }\sum _{j_k\le i<j=j_{k+1}-1}[z_{\tau (j)}-z_{\tau (i)}]\sum _{i\le l<j}[{\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}]\\ =-\sum _{k=1}^K\frac{1}{n\mid A_k\mid }\sum _{l=j_k}^{j_{k+1}-2}[{\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}]\left[\mid A_{kl}^1\mid \sum _{j\in A_{kl}^2}{z}_{\tau (j)}-\mid A_{kl}^2\mid \sum _{i\in A_{kl}^1}{z}_{\tau (i)}\right]\\ \equiv \sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}{w}_{\tau (l)}(\mathbf{z})[{\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}],\end{array}$$ (36)

where for any vector v ∊ ℝ^p,

$$w_{\tau (l)}(\mathbf{v})=n^{-1}\left[\frac{\mid A_{kl}^2\mid }{\mid A_k\mid }\sum _{j\in A_{kl}^1}{v}_{\tau (j)}-\frac{\mid A_{kl}^1\mid }{\mid A_k\mid }\sum _{j\in A_{kl}^2}{v}_{\tau (j)}\right].$$

We aim to bound |w_τ(l)(z)|. Let η = X^TX(β^* − β⁰), η^m = X^TX(β^m − β^*) and write z = X^Tε − η − η^m. First, w_τ(l)(v) is a linear function of v. Second, since β^m lies between β and β^*, we have ||β^* − β^m|| ≤ ||β^* − β|| ≤ t_n. It follows that ||η^m|| ≤ λ_max(X^TX)t_n. Moreover, |w_τ(l)(v)| ≤ (|A_k|/n)||v||_∞ ≤ (p/n)||v|| for all v. Combining the above yields

$$\begin{array}{l}\mid w_{\tau (l)}(\mathbf{z})\mid \le \mid w_{\tau (l)}({\mathbf{X}}^T\mathit{\epsilon })\mid +\mid w_{\tau (l)}(\mathit{\eta })\mid +\underset{\mathbf{v}:\Vert \mathbf{v}\Vert \le {\lambda }_{max}({\mathbf{X}}^T\mathbf{X})t_n}{sup}\mid w_{\tau (l)}(\mathbf{v})\mid \\ \le \mid w_{\tau (l)}({\mathbf{X}}^T\mathit{\epsilon })\mid +\mid w_{\tau (l)}(\mathit{\eta })\mid +(p/n){\lambda }_{max}({\mathbf{X}}^T\mathbf{X})·t_n.\end{array}$$ (37)

First, we bound the term w_τ(l)(X^Tε). Let E₃ be the event that

$$\underset{\tau (l)\in A_k}{max}\mid w_{\tau (l)}({\mathbf{X}}^T\mathit{\epsilon })\mid \le n^{-1/2}\sqrt{{\sigma }_k\mid A_k\mid log(2(n\vee p))/c_3},k=1,\cdots ,K,$$ (38)

where we recall σ_k is the maximum eigenvalue of n⁻¹X^TX restricted to the (A_k, A_k)-block. Given τ(l), we can express w_τ(l)(X^Tε) as

$$w_{\tau (l)}({\mathbf{X}}^T\mathit{\epsilon })={\mathbf{a}}_{\tau (l)}^T\mathit{\epsilon },\text{where}{\mathbf{a}}_{\tau (l)}=n^{-1}\left(\frac{\mid A_{kl}^2\mid }{\mid A_k\mid }{\mathbf{X}}_{A_{kl}^1}{\mathbf{1}}_{A_{kl}^1}-\frac{\mid A_{kl}^1\mid }{\mid A_k\mid }{\mathbf{X}}_{A_{kl}^2}{\mathbf{1}}_{A_{kl}^2}\right).$$

Write $L_1=\mid A_{kl}^1\mid$ and $L_2=\mid A_{kl}^2\mid$, so that |A_k| = L₁+L₂. It is observed that ${\Vert {\mathbf{X}}_{A_{kl}^1}{\mathbf{1}}_{A_{kl}^1}\Vert }^2\le n{\sigma }_k{\Vert {\mathbf{1}}_{A_{kl}^1}\Vert }^2\le n{\sigma }_k{L}_1$. Using the fact that (a + b)² ≤ 2(a² + b²) for any real values a, b, we have ${\Vert {\mathbf{a}}_{\tau (l)}\Vert }^2\le 2n^{-1}{\sigma }_k(L_2^2{L}_1/{\mid A_k\mid }^2+L_1^2{L}_2/{\mid A_k\mid }^2)=2{\sigma }_k{L}_1{L}_2/(n\mid A_k\mid )\le {\sigma }_k\mid A_k\mid /(2n)$. Applying Condition 3.3 and the probability union bound,

$$\begin{array}{l}P(E_3^c)\le \sum _{k=1}^K\sum _{\tau (l)\in A_k}P\left(\mid w_{\tau (l)}({\mathbf{X}}^T\mathit{\epsilon })\mid >n^{-1/2}\sqrt{{\sigma }_k\mid A_k\mid log(2(n\vee p))/c_3}\right)\\ \le \sum _{1\le j\le p}P\left(\mid {\mathbf{a}}_j^T\mathit{\epsilon }\mid >\Vert {\mathbf{a}}_j\Vert \sqrt{2log(2(n\vee p))/c_3}\right)<{(n\vee p)}^{-1}.\end{array}$$ (39)

Second, we bound the term w_τ(l)(η). Observing that for any vector v, w_τ(l)(v) = w_τ(l)(v − v̄_k1), where v̄_k is the mean of {v_j, j ∈ A_k}, we have

$${\mid w_{\tau (l)}(\mathbf{v})\mid }^2\le 2\left(\frac{{\mid A_{kl}^2\mid }^2\mid A_{kl}^1\mid }{n^2{\mid A_k\mid }^2}+\frac{{\mid A_{kl}^1\mid }^2\mid A_{kl}^2\mid }{n^2{\mid A_k\mid }^2}\right){(\underset{j\in A_k}{max}\mid v_j-{\overline{v}}_k\mid )}^2\le \frac{\mid A_k\mid }{2n^2}{(\underset{j\in A_k}{max}\mid v_j-{\overline{v}}_k\mid )}^2.$$

Since η = X^TX(β^* − β⁰) and β^* − β⁰ ∈ , we have max_j∈Ak|η_j − η̄_k| ≤ nν_k||β^* − β⁰||, where ν_k is defined in (13). As a result,

$$\underset{\tau (l)\in A_k}{max}\mid w_{\tau (l)}(\mathit{\eta })\mid \le \frac{{\nu }_k}{\sqrt{2}}{\mid A_k\mid }^{1/2}·\Vert {\mathit{\beta }}^{\ast }-{\mathit{\beta }}^0\Vert \le C{\nu }_k\sqrt{K\mid A_k\mid log(n)/n},$$ (40)

where the last inequality is because we consider β ∈ in (28), and ||β^* − β⁰|| ≤ ||β − β⁰|| (noticing that β^* is the orthogonal projection of β onto ).

Combining (36)–(40), we find that over the event E₁ ∩ E₂ ∩ E₃,

$$\begin{array}{l}\mid I_1\mid \le \sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}\left[C\left(\sqrt{\frac{{\sigma }_k\mid A_k\mid log(n\vee p)}{n}}+{\nu }_k\sqrt{\frac{K\mid A_k\mid log(n)}{n}}\right)+\frac{p{\lambda }_{max}({\mathbf{X}}^T\mathbf{X})}{n}{t}_n\right]\mid {\beta }_{\tau (l+1)}-{\beta }_{(l)}\mid \\ \le \sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}\left(\frac{{\lambda }_n}{2}+\frac{p{\lambda }_{max}({\mathbf{X}}^T\mathbf{X})}{n}{t}_n\right)\mid {\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}\mid ,\end{array}$$ (41)

where we have used condition (14) on λ_n.

From (35) and (41), over the event E₁ ∩ E₂ ∩ E₃,

$$\underset{\mathit{\beta }\in \mathcal{B}:\Vert \mathit{\beta }-{\mathit{\beta }}^{\ast }\Vert \le t_n}{inf}[Q_n(\mathit{\beta })-Q_n({\mathit{\beta }}^{\ast })]\ge \sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}\left[\frac{{\lambda }_n}{2}-g_n(t_n)\right]\mid {\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}\mid ,$$

where g_n(t_n) = n⁻¹pλ_max(X^TX)t_n − λ_n[1 − ρ′(2t_n)]. Since ρ′(0+) = 1, g_n(0+) = 0. So we can always choose t_n sufficiently small to make sure |g_n(t_n)| < λ_n/2; consequently, the right hand side is non-negative, and strictly positive when ${\sum }_{k=1}^K{\sum }_{l=j_k}^{j_{k+1}-2}\mid {\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}\mid >0$, i.e., β ≠ β^*. This proves (b).

A.2 Proof of Theorem 3.2

First, we show that the LLA algorithm yields β̂^oracle after one iteration. Let E₁ be the event that the ranking τ is consistent with the order of β⁰, E₂ the event that $\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^0\Vert \le C\sqrt{Klog(n)/n}$ and E₃ the event that (38) holds. We have shown that P (E₁ ∩E₂ ∩E₃) ≥ 1 − ε₀ − n⁻¹K − (n ∨ p)⁻¹. It suffices to show that over the event E₁ ∩ E₂ ∩ E₃, the LLA algorithm gives β̂^oracle after the first iteration.

Let $w_j={\rho }^{\prime }(\mid {\widehat{\beta }}_{\tau (j+1)}^{\mathit{initial}}-{\widehat{\beta }}_{\tau (j)}^{\mathit{initial}}\mid )$. At the first iteration, the algorithm minimizes

$$Q_n^{\mathit{initial}}(\mathit{\beta })\equiv \frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2+{\lambda }_n\sum _{j=1}^{p-1}{w}_j\mid {\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}\mid .$$

This is a convex function, hence it suffices to show that β̂^oracle is a strictly local minimizer of $Q_n^{\mathit{initial}}$. Using the same notations as in the proof of Theorem 3.1, for any β ∈ ℝ^p, write β^* = T⁻¹ ∘ T^*(β) as its orthogonal projection to . Let $\mathcal{B}=\{\mathit{\beta }\in {ℝ}^p:\Vert \mathit{\beta }-{\mathit{\beta }}^0\Vert \le C\sqrt{Klog(n)/n}\}$, and for a sequence {t_n} to be determined, consider the neighborhood of β̂^oralce defined by = {β ∈ : ||β − β̂^oracle|| ≤ t_n}. It suffices to show

$$Q_n^{\mathit{initial}}(\mathit{\beta })\ge Q_n^{\mathit{initial}}({\mathit{\beta }}^{\ast })\ge Q_n^{\mathit{initial}}({\widehat{\mathit{\beta }}}^{\mathit{oracle}}),\text{for}\text{any}\mathit{\beta }\in {\mathcal{B}}_n,$$ (42)

and the first inequality is strict whenever β ≠ β^*, and the second inequality is also strict whenever β ≠ β̂^oracle.

We first show the second inequality in (42). For τ(j) and τ(j + 1) in different groups, $\mid {\beta }_{\tau (j+1)}^0-{\beta }_{\tau (j)}^0\mid >2b_n$. In addition, ||β̂^initial − β⁰||_∞ ≤ λ_n/2. Hence, $\mid {\widehat{\beta }}_{\tau (j+1)}^{\mathit{initial}}-{\widehat{\beta }}_{\tau (j)}^{\mathit{initial}}\mid \ge 2b_n-{\lambda }_n>a{\lambda }_n$, and it follows that w_j = 0. On the other hand, for τ(j) and τ(j + 1) in the same group, β_τ(j+1) − β_τ(j) = 0 whenever β ∈ . Consequently,

$$Q_n^{\mathit{initial}}(\mathit{\beta })=\frac{1}{2n}{\Vert \mathit{y}-\mathbf{X}\mathit{\beta }\Vert }^2=L_n(\mathit{\beta }),\text{for}\mathit{\beta }\in {\mathcal{M}}_A.$$

It is easy to see that β̂^oracle is the unique global minimizer of L_n constrained on . So the second inequality in (42) holds.

Next, consider the first inequality in (42). We apply Taylor expansion to $Q_n^{\mathit{initial}}(\beta )-Q_n^{\mathit{initial}}({\mathit{\beta }}^{\ast })$, and rearrange the sums as in (34). Then, for some β^m that lies in the line between β and β^*,

$$Q_n^{\mathit{initial}}(\mathit{\beta })-Q_n^{\mathit{initial}}({\mathit{\beta }}^{\ast })={\lambda }_n\sum _{j=1}^{p-1}{w}_j·sgn({\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m)[({\beta }_{\tau (j+1)}-{\beta }_{\tau (j)})-({\beta }_{\tau (j+1)}^{\ast }-{\beta }_{\tau (j)}^{\ast })]-\frac{1}{n}{(\mathit{y}-\mathbf{X}{\mathit{\beta }}^m)}^T\mathbf{X}(\mathit{\beta }-{\mathit{\beta }}^{\ast })\equiv J_1+J_2.$$

We first simplify J₁. Note that w_j = 0 when τ(j) and τ(j + 1) are in different groups. When τ(j) and τ(j + 1) are in the same A_k, first, ${\beta }_{\tau (j+1)}^{\ast }={\beta }_{\tau (j)}^{\ast }$, and [ ${\beta }_{\tau (j+1)}^m-{\beta }_{\tau (j)}^m$] has the same sign as [β_τ(j+1) − β_τ(j)]; second, $\mid {\widehat{\beta }}_{\tau (j+1)}^{\mathit{initial}}-{\widehat{\beta }}_{\tau (j)}^{\mathit{initial}}\mid \le 2{\Vert {\widehat{\mathit{\beta }}}^{\mathit{initial}}-{\mathit{\beta }}^0\Vert }_{\infty }\le {\lambda }_n$, and hence w_j ≥ ρ′(λ_n) ≥ a₀. Combining the above yields

$$J_1={\lambda }_n\sum _{k=1}^K\sum _{j=j_k}^{j_{k+1}-2}{w}_j\mid {\beta }_{\tau (j+1)}-{\beta }_{(j)}\mid \ge a_0{\lambda }_n\sum _{k=1}^K\sum _{j=j_k}^{j_{k+1}-2}\mid {\beta }_{\tau (j+1)}-{\beta }_{\tau (j)}\mid$$ (43)

Next, we simplify J₂. Denote z = X^T(y − Xβ^m). Similarly to (36)–(41), we find that

$$J_2=-\sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}{w}_{\tau (l)}(\mathbf{z})[{\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}],$$

where over the event E₃, for any j_k ≤ l ≤ j_k+1 − 2,

$$\mid w_{\tau (l)}(\mathbf{z})\mid \le C\left(\sqrt{\frac{{\sigma }_k\mid A_k\mid log(n\vee p)}{n}}+{\nu }_k\sqrt{\frac{K\mid A_k\mid log(n)}{n}}\right)+\frac{p{\lambda }_{max}({\mathbf{X}}^T\mathbf{X})}{n}{t}_n.$$

From the condition on λ_n, the sum of the first two terms is upper bounded by a₀λ_n/3 for large n. We choose t_n = a₀nλ_n/(3pλ_max(X^TX)). It follows that

$$\mid J_2\mid \le \sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}\frac{2a_0{\lambda }_n}{3}\mid {\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}\mid .$$ (44)

Combining (43) and (44), over the event E₁ ∩ E₂ ∩ E₃,

$$Q_n^{\mathit{initial}}(\mathit{\beta })-Q_n^{\mathit{initial}}({\mathit{\beta }}^{\ast })\ge \frac{a_0{\lambda }_n}{3}\sum _{k=1}^K\sum _{l=j_k}^{j_{k+1}-2}\mid {\beta }_{\tau (l+1)}-{\beta }_{\tau (l)}\mid \ge 0.$$

This proves the first inequality in (42).

Second, we show that over the event E₁ ∩ E₂ ∩ E₃, at the second iteration, the LLA algorithm still yields β̂^oracle and therefore it converges to β̂^oracle. We have shown that after the first iteration, the algorithm outputs β̂^oracle. It then treats β̂^oracle as the initial solution for the second iteration. So it suffices to check

$${\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert }_{\infty }\le {\lambda }_n/2.$$

This is true because over the event E₁, $\Vert {\widehat{\mathit{\beta }}}^{\mathit{oracle}}-{\mathit{\beta }}^0\Vert \le C\sqrt{Klog(n)/n}\ll {\lambda }_n$.

A.3 Proof of Theorem 3.3

It suffices to show that, with probability at least 1 − O(n^−α), ${\beta }_i^0<{\beta }_j^0$ implies ${\widehat{\beta }}_i^{\mathit{ols}}\le {\widehat{\beta }}_j^{\mathit{ols}}$ for any 1 ≤ i, j ≤ p. When ${\beta }_i^0<{\beta }_j^0$, necessarily ${\beta }_j^0-{\beta }_i^0\ge 2b_n$. Moreover, ${\widehat{\beta }}_j^{\mathit{ols}}-{\widehat{\beta }}_i^{\mathit{ols}}\ge ({\beta }_j^0-{\beta }_i^0)-2{\Vert {\widehat{\mathit{\beta }}}^{\mathit{ols}}-{\mathit{\beta }}^0\Vert }_{\infty }$. So it suffices to show that ||β̂^ols − β⁰||_∞ ≤ b_n with probability at least 1 − O(n^−α).

From direct calculations, β^ols = β⁰ + (X^TX)⁻¹X^Tε. Write ${\widehat{\beta }}_j^{\mathit{ols}}-{\beta }_j^0={\mathbf{a}}_j^T\mathit{\epsilon }$, where a_j = X(X^TX)⁻¹e_j for j = 1, · · ·, p. Then ${\Vert {\mathbf{a}}_j\Vert }^2={\mathbf{e}}_j^T{({\mathbf{X}}^T\mathbf{X})}^{-1}{\mathbf{e}}_j\le c_4{n}^{-1}$. By Condition 3.3 and applying the union bound,

$$\begin{array}{l}P\left({\Vert {\widehat{\mathit{\beta }}}^{\mathit{ols}}-{\mathit{\beta }}^0\Vert }_{\infty }>b_n\right)\le P\left({\Vert {\widehat{\mathit{\beta }}}^{\mathit{ols}}-{\mathit{\beta }}^0\Vert }_{\infty }>\sqrt{(2\alpha c_4/c_3)log(n)/n}\right)\\ \le \sum _{j=1}^{p}P\left(\mid {\mathbf{a}}_j^T\mathit{\epsilon }\mid >\Vert \mathbf{a}\Vert \sqrt{2\alpha log(n)/c_3}\right)\le 2{pn}^{-2\alpha }.\end{array}$$

Since p = O(n^α), 2pn^−2α = O(n^−α). This completes the proof.