Network Granger Causality with Inherent Grouping Structure

Abstract

The problem of estimating high-dimensional network models arises naturally in the analysis of many biological and socio-economic systems. In this work, we aim to learn a network structure from temporal panel data, employing the framework of Granger causal models under the assumptions of sparsity of its edges and inherent grouping structure among its nodes. To that end, we introduce a group lasso regression regularization framework, and also examine a thresholded variant to address the issue of group misspecification. Further, the norm consistency and variable selection consistency of the estimates are established, the latter under the novel concept of direction consistency. The performance of the proposed methodology is assessed through an extensive set of simulation studies and comparisons with existing techniques. The study is illustrated on two motivating examples coming from functional genomics and financial econometrics.

1. Introduction

We consider the problem of learning a directed network of interactions among a number of entities from time course data. A natural framework to analyze this problem uses the notion of Granger causality (Granger, 1969). Originally proposed by C.W. Granger this notion provides a statistical framework for determining whether a time series X is useful in forecasting another one Y, through a series of statistical tests. It has found wide applicability in economics, including testing relationships between money and income (Sims, 1972), government spending and taxes on economic output (Blanchard and Perotti, 2002), stock price and volume (Hiemstra and Jones, 1994), etc. More recently the Granger causal framework has found diverse applications in biological sciences including functional genomics, systems biology and neurosciences to understand the structure of gene regulation, protein-protein interactions and brain circuitry, respectively.

It should be noted that the concept of Granger causality is based on associations between time series, and only under very stringent conditions, true causal relationships can be inferred (Pearl, 2000). Nonetheless, this framework provides a powerful tool for understanding the interactions among random variables based on time course data.

Network Granger causality (NGC) extends the notion of Granger causality among two variables to a wider class of p variables. Such extensions involving multiple time series are handled through the analysis of vector autoregressive processes (VAR) (Lütkepohl, 2005). Specifically, for p stationary time series $X_1^t,\dots ,X_p^t$, with $X^t=(X_1^t,\dots ,X_p^t{)}^{\prime }$, one considers the class of models

$$X^t=A^1{X}^{t-1}+\dots +A^d{X}^{t-d}+{ϵ}^t,$$ (1)

where A¹, A², … , A^d are p × p real-valued matrices, d is the unknown order of the VAR model and the innovation process satisfies ϵ^t ~ N(0, σ²I). In this model, the time series $\{X_j^t\}$ is said to be Granger causal for the time series $\{X_i^t\}$ if $A_{i,j}^h\ne 0$ for some h = 1, … , d. Equivalently we can say that there exists an edge $X_j^{t-h}\to X_i^t$ in the underlying network model comprising of (d + 1) × p nodes (see Figure 1). We call A¹, … , A^d the adjacency matrices from lags 1, … , d. Note that the entries $A_{ij}^h$ of the adjacency matrices are not binary indicators of presence/absence of edges between two nodes $X_i^t$ and $X_j^{t-h}$. Rather, they represent the direction and strength of influence from one node to the other.

Figure 1:

An example of a Network Granger causal model with two non-overlapping groups observed over T = 4 time points

The temporal structure induces a natural partial order among the nodes of this network, which in turn simplifies significantly the corresponding estimation problem (Shojaie and Michailidis, 2010a) of a directed acyclic graph. Nevertheless, one still has to deal with estimating a high-dimensional network (e.g., hundreds of genes) from a limited number of samples.

The traditional asymptotic framework of estimating VAR models requires observing a long, stationary realization {X¹, … , X^T, T → ∞, p, d fixed} of the p-dimensional time series. This is not appropriate in many biological applications for the following reasons. First, long stationary time series are rarely observed in these contexts. Second, the number of time series (p) being large compared to T, the task of consistent order (d) selection using standard criteria (e.g., AIC or BIC) becomes challenging. Similar issues arise in many econometric applications where empirical evidence suggests lack of stationarity over a long time horizon, although the multivariate time series exhibits locally stable distributional properties.

A more suitable framework comes from the study of panel data, where one observes several replicates of the time series, with possibly short T, across a panel of n subjects. In biological applications replicates are obtained from test subjects. In the analysis of macroeconomic variables, households or firms typically serve as replicates. After removing panel specific fixed effects one treats the replicates as independent samples, performs regression analysis under the assumption of common slope structure and studies the asymptotic properties under the regime n → ∞. Recent works of Cao and Sun (2011) and Binder et al. (2005) analyze theoretical properties of short panel VARs in the low-dimensional setting (n → ∞, T, p fixed).

The focus of this work is on estimating a high-dimensional NGC model in the panel data context (p, n large, T small to moderate). This work is motivated by two application domains, functional genomics and financial econometrics. In the first application (presented in Section 6) one is interested in reconstructing a gene regulatory network structure from time course data, a canonical problem in functional genomics (Michailidis, 2012). The second motivating example examines the composition of balance sheets of the n = 50 largest US banks by size, over T = 9 quarterly periods, which provides insight into their risk profile.

The nature of high-dimensionality in these two examples comes from both estimation of p² coefficients for each of the adjacency matrices A¹, … , A^d, but also from the fact that the order of the time series d is often unknown. Thus, in practice, one must either “guess” the order of the time series (often times, it is assumed that the data is generated from a VAR(1) model, which can result in significant loss of information), or include all of the past time points, resulting in significant increase in the number of variables in cases where d ⪡ T. Thus, efficient estimation of the order of the time series becomes crucial.

Latent variable based dimension reduction techniques like principal component analysis or factor models are not very useful in this context since our goal is to reconstruct a network among the observed variables. To achieve dimension reduction we impose a group sparsity assumption on the structure of the adjacency matrices A₁, … , A_d. In many applications, structural grouping information about the variables exists. For example, genes can be naturally grouped according to their function or chromosomal location, stocks according to their industry sectors, assets/liabilities according to their class, etc. This information can be incorporated to the Granger causality framework through a group lasso penalty. If the group specification is correct it enables estimation of denser networks with limited sample sizes (Bach, 2008; Huang and Zhang, 2010; Lounici et al., 2011). However, the group lasso penalty can achieve model selection consistency only at a group level. In other words, if the groups are misspecified, this procedure can not perform within group variable selection (Huang et al., 2009), an important feature in many applications.

Over the past few years, several authors have adopted the framework of network Granger causality to analyze multivariate temporal data. For example, Fujita et al. (2007) and Lozano et al. (2009) employed NGC models coupled with penalized ℓ₁ regression methods to learn gene regulatory mechanisms from time course microarray data. Specifically, Lozano et al. (2009) proposed to group all the past observations, using a variant of group lasso penalty, in order to construct a relatively simple Granger network model. This penalty takes into account the average effect of the covariates over different time lags and connects Granger causality to this average effect being significant. However, it suffers from significant loss of information and makes the consistent estimation of the signs of the edges difficult (due to averaging). Shojaie and Michailidis (2010b) proposed a truncating lasso approach by introducing a truncation factor in the penalty term, which strongly penalizes the edges from a particular time lag, if it corresponds to a highly sparse adjacency matrix.

Despite recent use of NGC in applications involving high dimensional data, theoretical properties of the resulting estimators have not been fully investigated. For example, Lozano et al. (2009) and Shojaie and Michailidis (2010b) discuss asymptotic properties of the resulting estimators, but neither addresses in depth norm consistency properties, nor do they examine under what vector autoregressive structures the obtained results hold.

In this paper, we develop a general framework that accommodates different variants of group lasso penalties for NGC models. It allows for the simultaneous estimation of the order of the times series and the Granger causal effects; further, it allows for variable selection even when the groups are misspecified. In summary, the key contributions of this work are: (i) investigate in depth sufficient conditions that explicitly take into consideration the structure of the VAR(d) model to establish norm consistency, (ii) introduce the novel notion of direction consistency, which generalizes the concept of sign consistency and provides insight into the properties of group lasso estimates within a group, and (iii) use the latter notion to introduce an easy to compute thresholded variant of group lasso, that performs within group variable selection in addition to group sparsity pattern selection even when the group structure is misspecified.

All the obtained results are non-asymptotic in nature, and hence help provide insight into the properties of the estimates under different asymptotic regimes arising from varying growth rates of T, p, n, group sizes and the number of groups.

2. Model and Framework

Notation.

Consider a VAR model

$$\underset{p\times 1}{\underbrace{X^t}}=\underset{p\times p}{\underbrace{A^1}}{X}^{t-1}+\dots +A^d{X}^{t-d}+{ϵ}^t,{ϵ}^t\sim N(0_{p\times 1},{\sigma }^2{I}_{p\times p}),$$ (2)

observed over T time points t = 1, … , T, across n panels. The index set of the variables ${\mathbb{N}}_p=\{1,2,\dots ,p\}$ can be partitioned into G non-overlapping groups ${\mathcal{G}}_g$, i.e., ${\mathbb{N}}_p={\cup }_{g=1}^G{\mathcal{G}}_g$ and ${\mathcal{G}}_g\cap {\mathcal{G}}_{g^{\prime }}=\varphi$ if g ≠ g′. Also $k_g=\mid {\mathcal{G}}_g\mid$ denotes the size of the g^th group with $k_{max}=\underset{1\le g\le G}{\max }{k}_g$. In general, we use λ_min and λ_max to denote the minimum and maximum of a finite collection of numbers λ₁, … , λ_m.

For any matrix A, we denote the i^th row by A_i:, j^th column by A_:j and the collection of rows (columns) corresponding to the g^th group by A_[g]: (A_:[g]). The transpose of a matrix A is denoted by A′ and its Frobenius norm by ∥A∥_F. For a symmetric/Hermitian matrix Σ, its maximum and minimum eigenvalues are denoted by Λ_min(Σ) and Λ_max(Σ), respectively. The symbol A^1:h is used to denote the concatenated matrix [A¹ : ⋯ : A^h], for any h > 0. For any matrix or vector D, ∥D∥₀ denotes the number of non-zero coordinates in D. For notational convenience, we reserve the symbol ∥.∥ to denote the ℓ₂ norm of a vector and/or the spectral norm of a matrix. For a pre-defined set of non-overlapping groups ${\mathcal{G}}_1,\dots ,{\mathcal{G}}_G$ on {1, … , p}, the mixed norms of vectors $v\in {\mathbb{R}}^p$ are defined as $\Vert v{\Vert }_{2,1}={\sum }_{g=1}^G\Vert v_{[g]}\Vert$ and ∥v∥_2,∞ = max_1≤g≤G∥v_[g]∥. Also for any vector β, we use β_j to denote its j^th coordinate and β_[g] to denote the coordinates corresponding to the g^th group. We also use supp(v) to denote the support of v, i.e., supp(v) = {j ∈ {1, … , p}∣v_j ≠ 0}.

Network Granger causal (NGC) estimates with group sparsity.

Consider n replicates from the NGC model (2), and denote the n × p observation matrix at time t by ${\mathcal{X}}^t$. In econometric applications the data on p economic variables across n panels (firms, households etc.) can be observed over T time points. For time course microarray data one typically observes the expression levels of p genes across n subjects over T time points. After removing the panel specific fixed effects one assumes the common slope structure and independence across the panels. The data are high-dimensional if either T or p is large compared to n. In such a scenario, we assume the existence of an underlying group sparse structure, i.e., for every i = 1, … , p, the support of the i^th row of A^1:T−1 = [A¹ : ⋯ : A^T−1] in the model (2) can be covered by a small number of groups s_i, where s_i ⪡ (T − 1)G. Note that the groups can be misspecified in the sense that the coordinates of a group covering the support need not be all non-zero. Hence, for a properly specified group structure we shall expect $s_i\ll \Vert A_{i:}^{1:T}{\Vert }_0$. On the contrary, with many misspecified groups, s_i can be of the same order, or even larger than $\Vert A_{i:}^{1:T}{\Vert }_0$.

Learning the network of Granger causal effects {(i, j) ∈ {1, … , p} : $A_{ij}^t\ne 0$ for some t} is equivalent to recovering the correct sparsity pattern in A^1:(T−1) and consistently estimating the non-zero effects $A_{ij}^t$. In the high-dimensional regression problems this is achieved by simultaneous regularization and selection operators like lasso and group lasso. The group Granger causal estimates of the adjacency matrices A¹, … , A^T−1 are obtained by solving the following optimization problem

$${\widehat{A}}^{1:T-1}=\underset{A^1,\cdots ,A^{T-1}}{\text{argmin}}\frac{1}{2n}{\Vert {\mathcal{X}}^T-\sum _{t=1}^{T-1}{\mathcal{X}}^{T-t}(A^t{)}^{\prime }\Vert }_F^2+\lambda \sum _{t=1}^{T-1}\sum _{i=1}^p\sum _{g=1}^G{w}_{i,g}^t\Vert A_{i:[g]}^t\Vert ,$$ (3)

where ${\mathcal{X}}^t$ is the n × p observation matrix at time t, constructed by stacking n replicates from the model (2), w^t is a p × G matrix of suitably chosen weights and λ is a common regularization parameter. The optimization problem can be separated into the following p penalized regression problems:

$${\widehat{A}}_{i:}^{1:T-1}=\underset{{\theta }^1,\cdots ,{\theta }^{T-1}\in {\mathbb{R}}^p}{\text{argmin}}\frac{1}{2n}\Vert {\mathcal{X}}_{:i}^T-\sum _{t=1}^{T-1}{\mathcal{X}}^{T-t}{\theta }^t{\Vert }^2+\lambda \sum _{t=1}^{T-1}\sum _{g=1}^G{w}_{i,g}^t\Vert {\theta }_{[g]}^t\Vert ,i=1,\cdots ,p.$$ (4)

The order d of the VAR model is estimated as $\widehat{d}=\underset{1\le t\le T-1}{\max }\{t:{\widehat{A}}^t\ne 0\}$.

Different choices of weights $w_{i:g}^t$ lead to different variants of NGC estimates. The regular NGC estimates correspond to the choices $w_{i,g}^t=1$ or $\sqrt{k_g}$, while for adaptive group NGC estimates the weights are chosen as $w_{i,g}^t={\Vert {\widehat{A}}_{i:[g]}^t\Vert }^{-1}$, where ${\widehat{A}}^t$ are obtained from a regular NGC estimation. For ${\widehat{A}}_{i:[g]}^t=0$, the weight $w_{i,g}^t$ is infinite, which is interpreted as discarding the variables in group g from the optimization problem.

Thresholded NGC estimates are calculated by a two-stage procedure. The first stage involves a regular NGC estimation procedure. The second stage uses a bi-level thresholding strategy on the estimates ${\widehat{A}}^t$. First, the estimated groups with ℓ₂ norm less than a threshold (δ_grp = cλ, c > 0) are set to zero. The second level of thresholding (within group) is applied if the a priori available grouping information is not entirely reliable. ${\widehat{A}}_{ij}^t$ within an estimated group ${\widehat{A}}_{i:[g]}^t$ is thresholded to zero if $\mid {\widehat{A}}_{ij}^t\mid ∕\Vert {\widehat{A}}_{i:[g]}^t\Vert$ is less than a threshold δ_misspec ∈ (0,1). So, for every t = 1, … , T − 1, if $j\in {\mathcal{G}}_g$, the thresholded NGC estimates are

$${\tilde{A}}_{ij}^t={\widehat{A}}_{ij}^{t}I\left\{\mid {\widehat{A}}_{ij}^t\mid \ge {\delta }_{misspec}\Vert {\widehat{A}}_{i:[g]}^t\Vert \right\}I\left\{\Vert {\widehat{A}}_{i:[g]}^t\Vert \ge {\delta }_{grp}\right\}.$$

The tuning parameters λ_grp and δ_misspec are chosen via cross-validation. The rationale behind this thresholding strategy is discussed in Section 4.

3. Estimation Consistency of NGC estimates

In this section we establish the norm consistency of regular group NGC estimates. The regular NGC estimates in (3) are obtained by solving p separate group lasso programs with a common design matrix ${\mathbf{X}}_{n\times p(T-1)}=[{\mathcal{X}}^1:\cdots :{\mathcal{X}}^{T-1}]$. This design matrix has $\overline{p}=(T-1)p$ columns which can be partitioned into $\overline{G}=(T-1)G$ groups $\{{\mathcal{G}}_1,\dots ,{\mathcal{G}}_{\overline{G}}\}$. We denote the sample Gram matrix by $C={\mathcal{X}}^{\prime }\mathcal{X}∕n$. For the i^th optimization problem, these $\overline{G}=(T-1)G$ groups are penalized by ${\lambda }_{(t-1)G+g}≔\lambda w_{i,g}^t$, 1 ≤ t ≤ T − 1, 1 ≤ g ≤ G, with the choice of weights $w_{i,g}^t$ described in Section 2. Following Lounici et al. (2011) one can establish a non-asymptotic upper bound on the ℓ₂ estimation error of the NGC estimates ${\widehat{A}}^t$ under certain restricted eigenvalue (RE) assumptions. These assumptions are common in the literature of high-dimensional regression (Lounici et al., 2011; Bickel et al., 2009; van de Geer and Bühlmann, 2009) and are known to be sufficient to guarantee consistent estimation of the regression coefficients even when the design matrix is singular. Of main interest, however, is to investigate the validity of these assumptions in the context of NGC models. This issue is addressed in Proposition 3.2.

For L > 0, we say that a Restricted Eigenvalue (RE) assumption RE(s, L) is satisfied if there exists a positive number ϕ_RE = ϕ_RE(s) > 0 such that

$$\underset{\underset{\Delta \in {\mathbb{R}}^{\stackrel{‒}{p}}\setminus \{0\}}{J\subset {\mathbb{N}}_{\stackrel{‒}{G}},\mid J\mid \le s}}{\text{min}}\left\{\frac{\Vert \mathbf{X}\Delta \Vert }{\sqrt{n}\Vert {\Delta }_{[J]}\Vert }:\sum _{g\in J^c}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \le L\sum _{g\in J}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \right\}\ge {\varphi }_{RE}.$$ (5)

The following proposition provides a non-asymptotic upper bound on the ℓ₂-estimation error of the group NGC estimates under RE assumptions. The proof follows along the lines of Lounici et al. (2011) and is delegated to Appendix C.

Proposition 3.1 Consider a regular NGC estimation problem (4) with s_max = max_1≤i≤p s_i and $s={\sum }_{i=1}^p{s}_i$. Suppose λ in (3) is chosen large enough so that for some α > 0,

$${\lambda }_g\ge \frac{2\sigma }{\sqrt{n}}\sqrt{\Vert C_{[g][g]}\Vert }\left(\sqrt{k_g}+\frac{\pi }{\sqrt{2}}\sqrt{\alpha \log \overline{G}}\right)\mathrm{for}\mathrm{every}g\in {\mathbb{N}}_{\overline{G}},$$ (6)

Also assume that the common design matrix $\mathbf{X}=[{\mathcal{X}}^1:\cdots :{\mathcal{X}}^{T-1}]$ in the p regression problems (4) satisfy RE(2s_max, 3). Then, with probability at least $1-2p{\overline{G}}^{1-\alpha }$,

$${\Vert {\widehat{A}}^{1:T-1}-A^{1:T-1}\Vert }_F\le \frac{4\sqrt{10}}{{\varphi }_{RE}^2(2s_{\max })}\frac{{\lambda }_{\max }^2}{{\lambda }_{\text{min}}}\sqrt{s}.$$ (7)

Remark. Consider a high-dimensional asymptotic regime where $\overline{G}\asymp n^B$ for some B > 0, k_max/k_min = O(1), s = O(n^a₁) and k_max = O(n^a₂) with 0 < a₁, a₂ < a₁ + a₂ < 1 so that the total number of non-zero effects is o(n). If $\{\Vert C_{[g][g]}\Vert ,g\in {\mathbb{N}}_{\overline{G}}\}$ are bounded above (often accomplished by standardizing the data) and ${\varphi }_{RE}^2(2s_{\max })$ is bounded away from zero (see Proposition 3.2 for more details), then the NGC estimates are norm consistent for any choice of α > 2 + a₂/B.

Note that group lasso achieves faster convergence rate (in terms of estimation and prediction error) than lasso if the groups are appropriately specified. For example, if all the groups are of equal size k and λ_g = λ for all g, then group lasso can achieve an ℓ₂ estimation error of order $O\left(\sqrt{s}(\sqrt{k}+\sqrt{\log \overline{G}})∕\sqrt{n}\right)$. In contrast, lasso’s error is known to be of the order $O\left(\sqrt{\Vert A^{1:d}{\Vert }_0\log \overline{p}∕n}\right)$, which establishes that group lasso has a lower error bound if s ⪡ ∥A^1:d∥₀. On the other hand, lasso will have a lower error bound if s ≍ ∥A^1:d∥₀, i.e., if the groups are highly misspecified.

Validity of RE assumption in Group NGC problems.

In view of Theorem 3.1, it is important to understand how stringent the RE condition is in the context of NGC problems. It is also important to find a lower bound on the RE coefficient ϕ_RE, as it affects the convergence rate of the NGC estimates. For the panel-VAR setting, we can rigorously establish that the RE condition holds with overwhelming probability, as long as n, p grow at the same rate required for ℓ₂-consistency.

The following proposition achieves this objective in two steps. Note that each row of the design matrix X (common across the p regressions) is independently distributed as N(0, Σ) where Σ is the variance-covariance matrix of the (T − 1)p-dimensional random variable ((X¹)′, … , (X^T−1)′)′. First, we exploit the spectral representation of the stationary VAR process to provide a lower bound on the minimum eigenvalue of Σ. In the next step, we establish a suitable deviation bound on X − Σ to prove that X satisfies RE condition with high probability for sufficiently large n.

Proposition 3.2 (a) Suppose the VAR(d) model of (2) is stable, stationary. Let Σ be the variance-covariance matrix of the (T−1)p-dimensional random variable ((X¹)′, … , (X^T−1)′)′. Then the minimum eigenvalue of Σ satisfies

$${\Lambda }_{min}(\Sigma )\ge {\sigma }^2{\left[\underset{\theta \in [-\pi ,\pi ]}{\max }\Vert \mathcal{A}(e^{-i\theta })\Vert \right]}^{-2}\ge {\sigma }^2{\left[1+\sum _{t=1}^d\Vert A^t\Vert \right]}^{-2}\ge {\sigma }^2{\left[1+\frac{1}{2}({\mathbf{v}}_{in}+{\mathbf{v}}_{out})\right]}^{-2},$$

where $\mathcal{A}(z)≔I-A^{1}z-A^2{z}^2-\dots -A^d{z}^d$ is the reverse characteristic polynomial of the VAR(d) process, and v_in, v_out are the maximum incoming and outgoing effects at a node, cumulated across different lags

$${\mathbf{v}}_{in}=\sum _{t=1}^d\underset{1\le i\le p}{\max }\sum _{j=1}^p\mid A_{ij}^t\mid ,{\mathbf{v}}_{out}=\sum _{t=1}^d\underset{1\le j\le p}{\max }\sum _{j=1}^p\mid A_{ij}^t\mid .$$

(b) In addition, suppose the replicates from different panels are i.i.d. Then, for any s > 0, there exist universal positive constants c_i such that if the sample size n satisfies

$$n>\frac{{\Lambda }_{\max }^2(\Sigma )}{{\Lambda }_{\text{min}}^2(\Sigma )}(2+L{\lambda }_{\max }∕{\lambda }_{\text{min}}{)}^4{c}_{0}s(k_{\max }+c_1\log (e\overline{G}∕2s)),$$

then X satisfies RE(s, L) with ${\varphi }_{RE}^2\ge {\Lambda }_{\text{min}}(\Sigma )∕2$ with probability at least 1 − c₂ exp(−c₃ n).

Remark. Proposition 3.2 has two interesting consequences. First, it provides a lower bound on the RE constant ϕ_RE which is independent of T. So if the high dimensionality in the Granger causal network arises only from the time domain and not the cross-section (T → ∞, p, G fixed), the stationarity of the VAR process guarantees that the rate of convergence depends only on the true order (d), and not T. Second, this result shows that the NGC estimates are consistent even if the node capacities v_in and v_out grow with n, p at an appropriate rate.

4. Variable Selection Consistency of NGC estimates

In view of (4), to study the variable selection properties of NGC estimates it suffices to analyze the variable selection properties of p generic group lasso estimates with a common design matrix.

The problem of group sparsity selection has been thoroughly investigated in the literature (Wei and Huang, 2010; Lounici et al., 2011). The issue of selection and sign consistency within a group, however, is still unclear. Since group lasso does not impose sparsity within a group, all the group members are selected together (Huang et al., 2009) and it is not clear which ones are recovered with correct signs. This also leads to inconsistent variable selection if a group is misspecified, i.e., not all the members within a group have non-zero effect. Several alternate penalized regression procedures have been proposed to overcome this shortcoming (Breheny and Huang, 2009; Huang et al., 2009). The main idea behind these procedures is to combine ℓ₂ and ℓ₁ norms in the penalty to encourage sparsity at both group and variable level. These estimators involve nonconvex optimization problems and are computationally expensive. Also their theoretical properties in a high dimensional regime are not well studied.

We take a different approach to deal with the issue of group misspecification. Although the group lasso penalty does not perform exact variable selection within groups, it performs regularization and shrinks the individual coefficients. We utilize this regularization to detect misspecification within a group. To this end, we formulate a generalized notion of sign consistency, henceforth referred as “direction consistency”, that provides insight into the properties of group lasso estimates within a single group. Subsequently, these properties are used to develop a simple, easy to compute, thresholded variant of group lasso which, in addition to group selection, achieves variable selection and sign consistency within groups.

We consider a generic group lasso regression problem of the linear model y = Xβ⁰ + ϵ with p variables partitioned into G non-overlapping groups $\{{\mathcal{G}}_1,\dots ,{\mathcal{G}}_G\}$ of size k_g, g = 1, … , G. Without loss of generality, we assume ${\beta }_{[g]}^0\ne 0$ for g ∈ S = {1, 2, … , s} and ${\beta }_{[g]}^0=0$ for all g ∉ S and consider the following group lasso estimate of β⁰:

$$\widehat{\beta }=\underset{\beta \in {\mathbb{R}}^p}{\text{argmin}}\frac{1}{2n}\Vert \mathbf{Y}-\mathbf{X}\beta {\Vert }^2+\sum _{g=1}^G{\lambda }_g\Vert {\beta }_{[g]}\Vert ,$$ (8)

$$\underset{p\times 1}{\underbrace{{\beta }^0}}=[\underset{k_1+\dots +k_s=q}{\underbrace{{\beta }_{[1]}^0,\dots ,{\beta }_{[s]}^0}},\underset{p-q}{\underbrace{0,\dots ,0}}]=[{\beta }_{(1)}^0:{\beta }_{(2)}^0],$$ (9)

$$\underset{n\times p}{\underbrace{\mathbf{X}}}=[\underset{n\times q}{\underbrace{{\mathbf{X}}_{(1)}}}:\underset{n\times (p-q)}{\underbrace{{\mathbf{X}}_{(2)}}}],C=\frac{1}{n}{\mathbf{X}}^{\prime }\mathbf{X}=\left[\begin{array}{cc}\hfill C_{11}\hfill & \hfill C_{12}\hfill \\ \hfill C_{21}\hfill & \hfill C_{22}\hfill \end{array}\right].$$ (10)

Direction Consistency.

For an m-dimensional vector $\tau \in {\mathbb{R}}^m\setminus \{0\}$ define its direction vector D(τ) = τ/∥τ∥ , D(0) = 0. In the context of a generic group lasso regression (10), for a group g ∈ S of size k_g, $D({\beta }_{[g]}^0)$) indicates the direction of influence of ${\beta }_{[g]}^0$ at a group level in the sense that it reflects the relative importance of the influential members within the group. Note that for k_g = 1 the function D(·) simplifies to the usual sgn(·) function.

Definition.

An estimate $\widehat{\beta }$ of a generic group lasso problem (8) is direction consistent at a rate δ_n, if there exists a sequence of positive real numbers δ_n → 0 such that

$$\mathbb{P}\left(\Vert D({\widehat{\beta }}_{[g]})-D({\beta }_{[g]}^0)\Vert <{\delta }_n,\forall g\in S,{\widehat{\beta }}_{[g]}=0,\forall g\notin S\right)\to 1\text{as}n,p\to \infty .$$ (11)

Now suppose $\widehat{\beta }$ is a direction consistent estimator. Consider the set ${\tilde{S}}_g^n≔\{j\in {\mathcal{G}}_g:\mid {\beta }_j^0\mid ∕\Vert {\beta }_{[g]}^0\Vert >{\delta }_n\}$. ${\tilde{S}}_g^n$ can be viewed as a collection of influential group members within a group ${\mathcal{G}}_g$, which are “detectable” with a sample of size n. Then, it readily follows from the definition that

$$\mathbb{P}(sgn({\widehat{\beta }}_j)=sgn({\beta }_j),\forall j\in {\tilde{S}}_g^n,\forall g\in \{1,\dots ,s\})\to 1\text{as}n,p\to \infty .$$ (12)

The latter observation connects the precision of group lasso estimates to the accuracy of a priori available grouping information. In particular, if the pre-specified grouping structure is correct, i.e., all the members within a group have non-zero effects, then for a sufficiently large sample size we have ${\tilde{S}}_g^n={\mathcal{G}}_g$ for all g ∈ S. Hence, if the group lasso estimate is direction consistent, it will correctly estimate the sign of all the variables in the support. On the other hand, in case of a misspecified a priori grouping structure (numerous zero coordinates in β_g for g ∈ S), group lasso will correctly estimate only the signs of the influential group members. This argument on zero vs. non-zero effects can be generalized to strong vs. weak effects, as well.

Example. We demonstrate the property of direction consistency using a small example. Consider a linear model with 8 predictors

$$y=0.5x_1-3x_2+3x_3+x_4-2x_5+3x_8+e,e\sim N(0,1).$$

The coefficient vector β⁰ is partitioned into four groups of size 2, viz., (0.5, −3), (3, 1), (−2, 0) and (0, 3). The last two groups are misspecified. We generated n = 25 samples from this model and ran group lasso regression with the above group structure. Figure 2 shows the true coefficient vectors (solid) and their estimates (dashed) from five iterations of the above exercise. Note that even though the ℓ₂ errors between ${\beta }_{[g]}^0$ and ${\widehat{\beta }}_{[g]}$ vary largely across the four groups, the distance between their projections on the unit circle, $\Vert D({\beta }_{[g]}^0)-D({\widehat{\beta }}_{[g]})\Vert$, are comparatively stable across groups. In fact, Theorem 4.1 shows that under certain irrepresentable conditions (IC) on the design matrix, it is possible to find a uniform (over all g ∈ S) upper bound δ_n on the ℓ₂ gap of these direction vectors. This motivates a natural thresholding strategy to correct for the misspecification in groups (cf. Proposition 4.2). Even though a group ${\beta }_{[g]}^0$ is misspecified (i.e., lies on a coordinate axis), direction consistency ensures, with high probability, that the corresponding coordinate in $D({\widehat{\beta }}_{[g]})$ will be smaller than a threshold δ_n which is common across all groups in the support.

Figure 2:

Example demonstrating direction consistency

Group Irrepresentable Conditions (IC).

Next, we define the IC required for direction consistency of group lasso estimates. Irrepresentable conditions are common in the literature of high-dimensional regression problems (Zhao and Yu, 2006; van de Geer and Bühlmann, 2009) and are shown to be sufficient (and essentially necessary) for selection consistency of the lasso estimates. Further these conditions are known to be satisfied with high probability, if the population analogue of the Gram matrix belongs to the Toeplitz family (Zhao and Yu, 2006; Wainwright, 2009). In NGC estimation the population analogue of the Gram matrix Σ = Var(X^1:(T−1)) is block Toeplitz, so the irrepresentable assumptions are natural candidates for studying selection consistency of the estimates. Consider the notations of (8) and (10). Define K = diag (λ₁I_k1, λ₂I_k2, … , λ_sI_ks).

Uniform Irrepresentable Condition (IC) is satisfied if there exists 0 < η < 1 such that for all $\tau \in {\mathbb{R}}^q$ with $\Vert \tau {\Vert }_{2,\infty }=\underset{1\le g\le s}{\max }\Vert {\tau }_{[g]}{\Vert }_2\le 1$,

$$\frac{1}{{\lambda }_g}\Vert {\left[C_{21}(C_{11}{)}^{-1}K\tau \right]}_{[g]}\Vert <1-\eta ,\forall g\notin S=\{1,\dots ,s\}.$$ (13)

Note that the definition reverts to the usual IC for lasso when all groups correspond are singletons.

The IC is more stringent than the RE condition and is rarely met if the underlying model is not sparse. It can be shown that a slightly weaker version of this condition is necessary for direction consistency. We refer the readers to Appendix D for further discussion on the different irrepresentable assumptions and their properties. Numerical evidence suggests that the group IC tends to be less stringent than the IC required for the selection consistency of lasso. We illustrate this using three small simulated examples.

Simulation 1. We constructed group sparse NGC models with T = 5, p = 21, G = 7, k_g = 3 and different levels of network densities, where the network edges were selected at random and scaled so that ∥A¹∥ = 0.1. For each of these models, we generated 100 samples of size n = 150 and calculated the proportions of times the two types of irrepresentable conditions were met. The results are displayed in Figure 3a.

Figure 3:

Comparison of lasso and group irrepresentable conditions in the context of group sparse NGC models. (a) group ICs tend to be met for dense networks where lasso IC fails to meet. (b) For the same network group IC is met with smaller sample size than required by lasso. (c) For longer time series group IC is satisfied more often than lasso IC.

Simulation 2. We selected a VAR(1) model from the above class and generated samples of size n = 20, 50, … , 250. Figure 3b displays the proportions of times (based on 100 simulations) the two ICs were met.

Simulation 3. We generated n = 200 samples from the VAR(1) model of example 2 for T = 2, 3, 4, 5, 10, … , 40. Figure 3c displays the proportions of times (based on 100 simulations) the two ICs were met.

Selection consistency for generic group lasso estimates.

For simplicity, we discuss the selection consistency properties of a generic group lasso regression problem with a common tuning parameter across groups, i.e., λ_g = λ for every $g\in {\mathbb{N}}_G$. Similar results can be obtained for more general choices of the tuning parameters.

Theorem 4.1 Assume that the group uniform IC holds with 1 − η for some η > 0. Then, for any choice of α > 0,

$$\begin{gathered} \lambda \ge \underset{g\notin S}{\max }\frac{1}{\eta }\frac{\sigma }{\sqrt{n}}\sqrt{\Vert (C_{22}{)}_{[g][g]}\Vert }\left(\sqrt{k_g}+\frac{\pi }{\sqrt{2}}\sqrt{\alpha \log G}\right)and \\ {\delta }_n\ge \underset{g\in S}{\max }\frac{1}{\Vert {\beta }_{[g]}^0\Vert }\left(\lambda \sqrt{s}\Vert (C_{11}{)}^{-1}\Vert +\frac{\sigma }{\sqrt{n}}\sqrt{\Vert (C_{11}{)}_{[g][g]}^{-1}\Vert }\left(\sqrt{k_g}+\frac{\pi }{\sqrt{2}}\sqrt{\alpha \log G}\right)\right), \end{gathered}$$

with probability greater than 1 − 4G^1−α, there exists a solution $\widehat{\beta }$ satisfying

1. ${\widehat{\beta }}_{[g]}=0$ for all g ∉ S,

2. $\Vert {\widehat{\beta }}_{[g]}-{\beta }_{[g]}^0\Vert <{\delta }_n\Vert {\beta }_{[g]}^0\Vert$, and hence $\Vert D({\widehat{\beta }}_{[g]})-D({\beta }_{[g]}^0)\Vert <2{\delta }_n$, for all g ∈ S. If δ_n < 1, then ${\widehat{\beta }}_{[g]}\ne 0$ for all g ∈ S.

Remark. The tuning parameter λ can be chosen of the same order as required for ℓ₂ consistency to achieve selection consistency within groups in the sense of (12). Further, with the above choice of λ, δ_n can be chosen of the order of $O(\sqrt{s}(\sqrt{k_{max}}+\sqrt{\log G})∕\sqrt{n})$. Thus, group lasso correctly identifies the group sparsity pattern and is direction consistent if $\sqrt{s}(\sqrt{k_{max}}+\sqrt{\log G})∕\sqrt{n}\to 0$, the same scaling required for ℓ₂ consistency.

Thresholding in Group NGC estimators.

As described in Section 2, regular group NGC estimates can be thresholded both at the group and coordinate levels. The first level of thresholding is motivated by the fact that lasso can select too many false positives [cf. van de Geer et al. (2011), Zhou (2010) and the references therein]. The second level of thresholding employs the direction consistency of regular group NGC estimates to perform within group variable selection with high probability. The following proposition demonstrates the benefit of these two types of thresholding. The second result is an immediate corollary of Theorem 4.1. Proof of the first result (thresholding at group level) requires some additional notations and is delegated to Appendix E.

Theorem 4.2 Consider a generic group lasso regression problem (8) with common tuning parameter λ_g = λ.

(i) Assume the RE(s, 3) condition of (5) holds with a constant ϕ_RE and define ${\widehat{\beta }}_{[g]}^{thgrp}={\widehat{\beta }}_{[g]}{1}_{\Vert {\widehat{\beta }}_{[g]}\Vert >4\lambda }$. If $\widehat{S}=\{g\in {\mathbb{N}}_G:{\widehat{\beta }}_{[g]}^{thgrp}\ne 0\}$, then $\mid \widehat{S}\setminus S\mid \le \frac{s}{{\varphi }_{RE}^2∕12}$, with probability at least 1 − 2G^1−α.

(ii) Assume that uniform IC holds with 1 − η for some η > 0. Choose λ and δ_n as in Theorem 4.1 and define

$${\widehat{\beta }}_j^{thgrp}={\widehat{\beta }}_{j}1\{\mid {\widehat{\beta }}_j\mid ∕\Vert {\widehat{\beta }}_{[g]}\Vert >2{\delta }_n\}\mathrm{for}\mathrm{all}j\in {\mathcal{G}}_g.$$

Then $sgn({\beta }_j^0)=sgn({\widehat{\beta }}_j^{thgrp})\forall j\in {\mathbb{N}}_p$ with probability at least 1 − 4G^1−α, if $\underset{j\in supp({\beta }^0)}{\text{min}}\mid {\beta }_j^0\mid >2{\delta }_n\Vert {\beta }_{[g]}^0\Vert$ for all $j\in {\mathcal{G}}_g$, i.e., if the effect of every non-zero member in a group is “visible” relative to the total effect from the group.

5. Performance Evaluation

We evaluate the performances of regular, adaptive and thresholded variants of the group NGC estimators through an extensive simulation study, and compare the results to those obtained from lasso estimates. The R package grpreg (Breheny and Huang, 2009) was used to obtain the group lasso estimates. The settings considered are:

(a) Balanced groups of equal size: i.i.d samples of size n = 60, 110, 160 are generated from lag-2 (d = 2) VAR models on T = 5 time points, comprising of p = 60, 120, 200 nodes partitioned into groups of equal size in the range 3-5.

(b) Unbalanced groups: We retain the same setting as before, however the corresponding node set is partitioned into one larger group of size 10 and many groups of size 5.

(c) Misspecified balanced groups: i.i.d samples of size n = 60, 110, 160 are generated from lag-2 (d = 2) VAR models on T = 10 time points, comprising of p = 60, 120 nodes partitioned into groups of size 6. Further, for each group there is a 30% misspecification rate, namely that for every parent group of a downstream node, 30% of the group members do not exert any effect on it.

Using a 19 : 1 sample-splitting, the tuning parameter λ is chosen from an interval of the form [C₁λ_e, C₂λ_e], C₁, C₂ > 0, where ${\lambda }_e=\sqrt{2\log p∕n}$ for lasso and $\sqrt{2\log G∕n}$ for group lasso. The thresholding parameters are selected as δ_grp = 0.7λσ at the group level and δ_misspec = n^−0.2 within groups. These parameters are chosen by conducting a 20-fold cross-validation on independent tuning data sets of same sizes, using intervals of the form [C₃λ, C₄λ] for δ_grp and {n^−δ, δ ∈ [0, 1]} for δ_misspec. Finally, within group thresholding is applied only when the group structure is misspecified.

The following performance metrics were used for comparison purposes: (i) Precision = TP/(TP + FP), (ii) Recall = TP/(TP + FN) and (iii) Matthew’s Correlation coefficient (MCC) defined as

$$\frac{(TP\times TN)-(FP\times FN)}{((TP+FP)\times (TP+FN)\times (TN+FP)\times (TN+FN){)}^{1∕2}},$$

where TP, TN, FP and FN correspond to true positives, true negatives, false positives and false negatives in the estimated network, respectively. The average and standard deviations (over 100 replicates) of the performance metrics are presented for each setup.

The results for the balanced settings are given in Table 1. The Recall for p = 60 shows that even for a network with 60 × (5 − 1) = 240 nodes and ∣E∣ = 351 true edges, the group NGC estimators recover about 71% of the true edges with a sample size as low as n = 60, while lasso based NGC estimates recover only 31% of the true edges. The three group NGC estimates have comparable performances in all the cases. However thresholded lasso shows slightly higher precision than the other group NGC variants for smaller sample sizes (e.g., n = 60, p = 200). The results for p = 60, n = 110 also display that lower precision of lasso is caused partially by its inability to estimate the order of the VAR model correctly, as measured by ERR LAG=Number of falsely connected edges from lags beyond the true order of the VAR model divided by the number of edges in the network (∣E∣). This finding is nicely illustrated in Figure 4 and Table 1. The group penalty encourages edges from the nodes of the same group to be picked up together. Since the nodes of the same group are also from the same time lag, the group variants have substantially lower ERR LAG. For example, average ERR LAG of lasso for p = 200, n = 160 is 19.79% while the average ERR LAGs for the group lasso variants are in the range 3.06% – 4.21%.

Table 1:

Performance of different regularization methods in estimating graphical Granger causality with balanced group sizes and no misspecification; d = 2, T = 5, SNR = 1.8. Precision (P), Recall (R), MCC are given in percentages (numbers in parentheses give standard deviations). ERR LAG gives the error associated with incorrect estimation of VAR order.

		p = 60, ∣E∣ =351 Group Size=3			p = 120, ∣E∣ =1404 Group Size=3			p = 200, ∣E∣ =3900 Group Size=5
	n	160	110	60	160	110	60	160	110	60
P	Lasso	80(2)	75(2)	66(4)	69(1)	62(2)	52(2)	52(1)	47(1)	38(1)
	Grp	95(2)	91(4)	83(7)	91(3)	80(5)	68(7)	78(4)	72(3)	59(6)
	Thgrp	96(1)	92(3)	86(6)	93(3)	83(5)	70(7)	82(4)	76(3)	64(6)
	Agrp	96(2)	92(4)	83(7)	92(3)	82(5)	69(7)	81(3)	74(3)	60(6)
R	Lasso	71(2)	54(2)	31(2)	54(1)	40(1)	22(1)	38(1)	28(1)	15(1)
	Grp	99(1)	93(3)	71(7)	91(2)	81(2)	48(8)	84(1)	70(2)	41(4)
	Thgrp	99(1)	93(3)	71(7)	91(2)	81(2)	48(8)	84(2)	69(2)	41(3)
	Agrp	99(1)	93(3)	71(7)	91(2)	81(2)	47(8)	84(1)	69(2)	40(4)
MCC	Lasso	75(2)	63(2)	45(3)	60(1)	49(1)	33(1)	43(1)	35(1)	23(1)
	Grp	97(1)	92(3)	76(5)	91(1)	80(2)	56(2)	81(2)	70(2)	48(2)
	Thgrp	98(1)	93(2)	78(5)	92(1)	81(2)	57(3)	83(2)	72(2)	50(3)
	Agrp	97(1)	92(3)	76(5)	91(1)	81(2)	56(3)	82(2)	71(2)	48(2)
ERR	Lasso	10.5	11.3	13.9	16.63	17.37	16.69	19.79	20	18.52
LAG	Grp	3.19	6.95	12.76	4.86	10.77	12.65	4.21	5.27	7.8
	Thgrp	2.83	5.87	10.01	3.98	9.03	11.19	3.06	3.91	5.68
	Agrp	3.13	6.89	12.59	4.63	10.37	12.34	3.58	4.87	7.59

Figure 4:

Estimated adjacency matrices of a misspecified NGC model with p = 60, T = 10, n = 60: (a) True, (b) Lasso, (c) Group Lasso, (d) Thresholded Group Lasso. The grayscale represents the proportion of times an edge was detected in 100 simulations.

The results for the unbalanced networks are given in Table 2. As in the balanced group setup, in almost all the simulation settings the group NGC variants outperform the lasso estimates with respect to all three performance metrics. However the performances of the different variants of group NGC are comparable and tend to have higher standard deviations than the lasso estimates. Also the average ERR LAGs for the group NGC variants are substantially lower than the average ERR LAG for lasso demonstrating the advantage of group penalty. Although the conclusions regarding the comparisons of lasso and group NGC estimates remain unchanged it is evident that the performances of all the estimators are affected by the presence of one large group, skewing the uniform nature of the network. For example the MCC measures of group NGC estimates in a balanced network with p = 60 and ∣E∣ = 351 vary around 97 – 98% which lowers to 89% – 90% when the groups are unbalanced.

Table 2:

Performance of different regularization methods in estimating graphical Granger causality with unbalanced group sizes and no misspecification; d = 2, T = 5, SNR = 1.8. Precision (P), Recall (R), MCC are given in percentages (numbers in parentheses give standard deviations). ERR LAG gives the error associated with incorrect estimation of VAR order.

		p = 60, ∣E∣ = 450 Groups=1 × 10, 11 × 5			p = 120, ∣E∣ = 1575 Groups=1 × 10, 23 × 5			p = 200, ∣E∣ = 4150 Groups=1 × 10, 39 X× 5
	n	160	110	60	160	110	60	160	110	60
P	Lasso	72(2)	69(3)	62(2)	51(1)	48(1)	41(1)	61(1)	53(1)	42(2)
	Grp	84(4)	79(6)	76(9)	55(5)	47(5)	40(6)	86(3)	77(5)	66(7)
	Thgrp	86(4)	82(7)	78(11)	60(6)	50(7)	40(5)	88(2)	79(6)	69(6)
	Agrp	85(3)	81(5)	77(9)	59(5)	51(5)	42(6)	88(2)	78(5)	67(6)
R	Lasso	45(2)	35(2)	22(2)	43(1)	34(1)	22(1)	23(1)	15(0)	7(0)
	Grp	94(3)	87(5)	61(8)	88(2)	75(5)	48(6)	73(3)	49(6)	22(5)
	Thgrp	95(2)	88(4)	62(8)	89(3)	77(4)	50(5)	73(3)	50(6)	21(5)
	Agrp	94(3)	87(5)	61(8)	88(2)	75(5)	48(6)	73(3)	49(6)	22(5)
MCC	Lasso	56(2)	48(2)	35(2)	46(1)	39(1)	29(1)	36(1)	28(1)	17(1)
	Grp	89(3)	82(4)	67(5)	68(3)	58(3)	42(3)	79(1)	61(3)	37(3)
	Thgrp	90(3)	84(4)	68(6)	72(4)	61(4)	43(2)	80(1)	62(3)	37(3)
	Agrp	89(3)	83(4)	67(6)	71(3)	60(3)	43(3)	79(1)	61(3)	37(3)
ERR	Lasso	10.59	10.74	11.76	18.3	18.72	18.76	11.54	10.93	9.29
LAG	Grp	7.04	9.85	13.04	12.53	14.71	13.06	4.8	6.41	6.85
	Thgrp	6.58	8.98	11.1	9.6	11.9	10.9	4.06	5.65	5.7
	Agrp	6.74	9.19	12.96	10.81	12.78	11.79	4.55	6.2	6.81

The results for misspecified groups are given in Table 3. Note that for higher sample size n, the MCC of lasso and regular group lasso are comparable. However, the thresholded version of group lasso achieves significantly higher MCC than the rest. This demonstrates the advantage of using the directional consistency of group lasso estimators to perform within group variable selection. We would like to mention here that a careful choice of the thresholding parameters δ_grp and δ_misspec via cross-validation improves the performance of thresholded group lasso; however, we do not pursue these methods here as they require grid search over many tuning parameters or an efficient estimator of the degree of freedom of group lasso.

Table 3:

Performance of different regularization methods in estimating graphical Granger causality with misspecified groups (30% misspecification); d = 2, T = 10, SNR = 2. Precision (P), Recall (R), MCC are given in percentages (numbers in parentheses give standard deviations). ERR LAG gives the error associated with incorrect estimation of VAR order.

		p = 60, ∣E∣ = 246 Group Size=6			p = 120, ∣E∣ = 968 Group Size=6
	n	160	110	60	160	110	60
P	Lasso	88(2)	85(3)	77(5)	59(1)	55(1)	49(2)
	Grp	65(2)	66(2)	66(3)	43(3)	44(4)	38(4)
	Thgrp	87(3)	88(3)	85(3)	56(6)	56(6)	51(7)
	Agrp	65(2)	66(2)	66(3)	45(2)	45(4)	39(4)
R	Lasso	80(3)	63(3)	37(2)	66(1)	54(1)	35(1)
	Grp	100(0)	98(2)	82(6)	87(2)	78(3)	59(4)
	Thgrp	100(0)	98(2)	79(6)	86(2)	79(3)	57(4)
	Agrp	100(0)	98(2)	82(6)	86(2)	78(3)	58(3)
MCC	Lasso	84(2)	73(2)	53(3)	62(1)	54(1)	41(1)
	Grp	81(1)	80(2)	74(4)	61(2)	58(3)	47(2)
	Thgrp	93(2)	93(2)	82(4)	69(4)	66(4)	53(3)
	Agrp	81(1)	80(2)	74(4)	62(2)	59(2)	47(2)
ERR	Lasso	12.63	17.05	22.41	45.09	49.68	53.4
LAG	Grp	9.43	8.78	15.12	18.22	18.43	29.26
	Thgrp	6.45	5.34	8.02	11.81	12.84	15.57
	Agrp	9.11	8.78	14.96	16.32	16.9	27.69

In summary, the results clearly show that all variants of group lasso NGC outperform the lasso-based ones, whenever the grouping structure of the variables is known and correctly specified. Further, their performance depends on the composition of group sizes. On the other hand, if the a priori known group structure is moderately misspecified lasso estimates produce comparable results to regular and adaptive group NGC ones, while thresholded group estimates outperform all other methods, as expected.

6. Application

Example: T-cell activation.

Estimation of gene regulatory networks from expression data is a fundamental problem in functional genomics (Friedman, 2004). Time course data coupled with NGC models are informationally rich enough for the task at hand. The data for this application come from Rangel et al. (2004), where expression patterns of genes involved in T-cell activation were studied with the goal of discovering regulatory mechanisms that govern them in response to external stimuli. Activated T-cells are involved in regulation of effector cells (e.g., B-cells) and play a central role in mediating immune response. The available data comprising of n = 44 samples of p = 58 genes, measure the cells response at 10 time points, t = 0, 2, 4, 6, 8, 18, 24, 32, 48, 72 hours after their stimulation with a T-cell receptor independent activation mechanism. We concentrate on data from the first 5 time points, that correspond to early response mechanisms in the cells.

Genes are often grouped based on their function and activity patterns into biological pathways. Thus, the knowledge of gene functions and their membership in biological pathways can be used as inherent grouping structures in the proposed group lasso estimates of NGC. Towards this, we used available biological knowledge to define groups of genes based on their biological function. Reliable information for biological functions were found from the literature for 38 genes, which were retained for further analysis. These 38 genes were grouped into 13 groups with the number of genes in different groups ranging from 1 to 5.

Figure 5 shows the estimated networks based on lasso and thresholded group lasso estimates, where for ease of representation the nodes of the network correspond to groups of genes. In this case, estimates from variants of group NGC estimator were all similar, and included a number of known regulatory mechanisms in T-cell activation, not present in the regular lasso estimate. For instance, Waterman et al. (1990) suggest that TCF plays a significant role in activation of T-cells, which may describe the dominant role of this group of genes in the activation mechanism. On the other hand, Kim et al. (2005) suggest that activated T-cells exhibit high levels of osteoclast-associated receptor activity which may attribute the large number of associations between member of osteoclast differentiation and other groups. Finally, the estimated networks based on variants of group lasso estimator also offer improved estimation accuracy in terms of mean squared error (MSE) despite having having comparable complexities to their regular lasso counterpart (Table 4), which further confirms the findings of other numerical studies in that paper.

Figure 5:

Estimated Gene Regulatory Networks of T-cell activation. Width of edges represent the number of effects between two groups, and the network represents the aggregated regulatory network over 3 time points.

Table 4:

Mean and standard deviation of MSE for different NGC estimates

	Lasso	Grp	Agrp	Thgrp
mean	0.649	0.456	0.457	0.456
stdev	0.340	0.252	0.251	0.252

Example: Banking balance sheets application.

In this application, we examine the structure of the balance sheets in terms of assets and liabilities of the n = 50 largest (in terms of total balance sheet size) US banking corporations. The data cover 9 quarters (September 2009-September 2011) and were directly obtained from the Federal Deposit Insurance Corporation (FDIC) database (available at www.fdic.gov). The p = 21 variables correspond to different assets (US and foreign government debt securities, equities, loans (commercial, mortgages), leases, etc.) and liabilities (domestic and foreign deposits from households and businesses, deposits from the Federal Reserve Board, deposits of other financial institutions, non-interest bearing liabilities, etc.) We have organized them into four categories: two for the assets (loans and securities) and two for the liabilities (Balances Due and Deposits, based on a $250K reporting FDIC threshold). Amongst the 50 banks examined, one discerns large integrated ones with significant retail, commercial and investment activities (e.g., Citibank, JP Morgan, Bank of America, Wells Fargo), banks primarily focused on investment business (e.g., Goldman Sachs, Morgan Stanley, American Express, E-Trade, Charles Schwab), regional banks (e.g., Banco Popular de Puerto Rico, Comerica Bank, Bank of the West).

The raw data are reported in thousands of dollars. The few missing values were imputed using a nearest neighbor imputation method with k = 5, by clustering them according to their total assets in the most recent quarter in the data collection period (September 2011) and subsequently every missing observation for a particular bank was imputed by the median observation on its five nearest neighbors. The data were log-transformed to reduce nonstationarity issues. The data set was restructured as a panel with p = 21 variables and n = 50 replicates observed over T = 9 time points. Every column of replicates was scaled to have unit variance.

We applied the proposed variants of NGC estimates on the first T = 6 time points (Sep 2009 - Dec 2010) of the above panel data set. The parameters λ and δ_grp were chosen using a 19 : 1 sample-splitting method and the misspecification threshold δ_misspec was set to zero as the grouping structure was reliable. We calculated the MSE of the fitted model in predicting the outcomes in the four quarters (December 2010 - September 2011). The Predicted MSE (MSE for Dec 2010) are listed in Table 5. The estimated network structures are shown in Figure 6.

Table 5:

Mean and standard deviation (in parentheses) of PMSE (MSE in case of Dec 2010) for prediction of banking balance sheet variables.

Quarter	Lasso	Grp	Agrp	Thgrp
Dec 2010	1.59 (0.29)	0.36 (0.05)	0.36 (0.05)	0.37 (0.05)
Mar 2011	1.46 (0.30)	0.47 (0.23)	0.47 (0.23)	0.46 (0.22)
Jun 2011	1.33 (0.26)	0.36 (0.11)	0.36 (0.11)	0.35 (0.11)
Sep 2011	1.72 (0.32)	0.50 (0.18)	0.50 (0.18)	0.47 (0.16)

Figure 6:

Estimated Networks of banking balance sheet variables using (a) lasso and (b) group lasso. The networks represent the aggregated network over 5 time points.

It can be seen that the lasso estimates recover a very simple temporal structure amongst the variables; namely, that past values (in this case lag-1) influence present ones. Given the structure of the balance sheet of large banks, this is an anticipated result, since it can not be radically altered over a short time period due to business relationships and past commitments to customers of the bank. However, the (adaptive) group lasso estimates reveal a richer and more nuanced structure. Examining the fitted values of the adjacency matrices A^t, we notice that the dominant effects remain those discovered by the lasso estimates. However, fairly strong effects are also estimated within each group, but also between the groups of the assets (loans and securities) on the balance sheet. This suggests rebalancing of the balance sheet for risk management purposes between relatively low risk securities and potentially more risky loans. Given the period covered by the data (post financial crisis starting in September 2009) when credit risk management became of paramount importance, the analysis picks up interesting patterns. On the other hand, significant fewer associations are discovered between the liabilities side of the balance sheet. Finally, there exist relationships between deposits and securities such as US Treasuries and other domestic ones (primarily municipal bonds); the latter indicates that an effort on behalf of the banks to manage the credit risk of their balance sheets, namely allocating to low risk assets as opposed to more risky loans.

It is also worth noting that the group lasso model exhibits superior predictive performance over the lasso estimates, even 4 quarters into the future. Finally, in this case the thresholded estimates did not provide any additional benefits over the regular and adaptive variants, given that the specification of the groups was based on accounting principles and hence correctly structured.

7. Discussion

In this paper, the problem of estimating Network Granger Causal (NGC) models with inherent grouping structure is studied when replicates are available. Norm, and both group level and within group variable selection consistency are established under fairly mild assumptions on the structure of the underlying time series. To achieve the second objective the novel concept of direction consistency is introduced.

The type of NGC models discussed in this study have wide applicability in different areas, including genomics and economics. However, in many contexts the availability of replicates at each time point is not feasible (e.g., in rate of returns for stocks or other macroeconomic variables), while grouping structure is still present (e.g., grouping of stocks according to industry sector). Hence, it is of interest to study the behavior of group lasso estimates in such a setting and address the technical challenges emanating from such a pure time series (dependent) data structure.

Appendix

Appendix A. Auxiliary Lemmas

Lemma A.1 (Characterization of the Group lasso estimate) A vector $\widehat{\beta }\in {\mathbb{R}}^p$ is a solution to the convex optimization problem

$$\underset{\beta \in {\mathbb{R}}^p}{\text{argmin}}\frac{1}{2n}\Vert Y-X\beta {\Vert }^2+\sum _{g=1}^G{\lambda }_g\Vert {\beta }_{[g]}\Vert$$ (14)

if and only if $\widehat{\beta }$ satisfies, for some $\tau \in {\mathbb{R}}^p$ with max_1≤g≤G ∥τ_[g]∥ ≤ 1, $\frac{1}{n}{\left[X^{\prime }(Y-X\widehat{\beta })\right]}_{[g]}={\lambda }_g{\tau }_{[g]}{\forall }_g$. Further, ${\tau }_{[g]}=D\left({\widehat{\beta }}_{[g]}\right)$ whenever ${\widehat{\beta }}_{[g]}\ne 0$.

Proof Follows directly from the KKT conditions for the optimization problem (14). ■

Lemma A.2 (Concentration bound for multivariate Gaussian) Let Z_k×1 ~ N(0, Σ). Then, for any t > 0, the following inequalities hold:

$$\mathbb{P}(\mid \Vert Z\Vert -\mathbb{E}\Vert Z\Vert \mid >t)\le 2\text{exp}\left(-\frac{2t^2}{{\pi }^2\Vert \Sigma \Vert }\right),\mathbb{E}\Vert Z\Vert \le \sqrt{k}\sqrt{\Vert \Sigma \Vert }.$$

Proof The first inequality can be found in Ledoux and Talagrand (1991) (equation (3.2). To establish the second inequality note that,

$$\mathbb{E}\Vert Z\Vert \le \sqrt{\mathbb{E}\Vert Z{\Vert }^2}=\sqrt{\mathbb{E}[\text{tr}(Z{Z}^{\prime })]}=\sqrt{\text{tr}(\Sigma )}\le \sqrt{k}\sqrt{\Vert \Sigma \Vert }.$$

■

Lemma A.3 Let β, $\widehat{\beta }\in {\mathbb{R}}^m\{0\}$. Let $\widehat{u}=\widehat{\beta }-\beta$ and $r=D(\widehat{\beta })-D(\beta )$. Then ∥r∥ < 2δ whenever $\Vert \widehat{u}\Vert <\delta \Vert \beta \Vert$.

Proof It follows from $\Vert \widehat{u}\Vert <\delta \Vert \beta \Vert$ that

$$(1-\delta )\Vert \beta \Vert <\Vert \beta \Vert -\Vert \widehat{u}\Vert \le \Vert \widehat{\beta }\Vert \le \Vert \widehat{u}\Vert +\Vert \beta \Vert <(1+\delta )\Vert \beta \Vert ,$$

which implies that $\mid \Vert \beta \Vert -\Vert \widehat{\beta }\Vert \mid <\delta \Vert \beta \Vert$. Now,

$$\Vert \widehat{\beta }\Vert \Vert \beta \Vert \Vert r\Vert =\Vert \widehat{\beta }\Vert \beta \Vert +(\widehat{u}-\widehat{\beta })\Vert \widehat{\beta }\Vert \Vert \le \Vert \widehat{\beta }\left(\Vert \beta \Vert -\Vert \widehat{\beta }\Vert \right)+\Vert \widehat{\beta }\Vert \widehat{u}\Vert <\Vert \widehat{\beta }\Vert \Vert \beta \Vert (\delta +\delta ),$$

since $\mid \Vert \beta \Vert -\Vert \widehat{\beta }\Vert \mid <\delta \Vert \beta \Vert$ and $\Vert \widehat{u}\Vert <\delta \Vert \beta \Vert$. ■

Lemma A.4 Let ${\mathcal{G}}_1$, … , ${\mathcal{G}}_G$ be any partition of {1, …, p} into G non-overlapping groups and λ₁, … , λ_G be positive real numbers. Define the cone sets $\mathcal{C}(J,L)=\{v\in {\mathbb{R}}^p:{\sum }_{g\notin J}{\lambda }_g\Vert v_{[g]}\Vert \le L{\sum }_{g\in J}{\lambda }_g\Vert v_{[g]}\Vert \}$ for any subset of groups $J\subseteq {\mathbb{N}}_G$. Also define the set of group s-sparse vectors $\mathbb{D}(s)≔\{v\in {\mathbb{R}}^p:\Vert v\Vert \le 1\}$, $\mathit{supp}(v)\subseteq {\mathcal{G}}_J$ for some $J\subseteq {\mathbb{N}}_G$, ∣J∣ ≤ s}. Then

$$\bigcup _{J\subseteq {\mathbb{N}}_G,\mid J\mid \le s}\mathcal{C}(J,L)\cap {\mathbb{S}}^{p-1}\subseteq (2+L^{\prime })cl\{conv\{\mathbb{D}(s)\}\},$$ (15)

where L′ = Lλ_max/λ_min, ${\mathbb{S}}^{p-1}=\{v\in {\mathbb{R}}^p:\Vert v\Vert =1\}$ is the ball of unit norm vectors in ${\mathbb{R}}^p$ and cl{.}, conv{.} respectively denote the closure and convex hull of a set.

Proof Note that for any $J\subseteq {\mathbb{N}}_G$, ∣J∣ ≤ s, and $v\in \mathcal{C}(J,L)\cap {\mathbb{S}}^{p-1}$, we have

$$\sum _{g\notin J}\Vert v_{[g]}\Vert \le L\frac{{\lambda }_{\max }}{{\lambda }_{\text{min}}}\sum _{g\in J}\Vert v_{[g]}\Vert ,$$

which implies

$$\Vert v{\Vert }_{2,1}\le (L^{\prime }+1)\sum _{g\in J}\Vert v_{[g]}\Vert \le (L^{\prime }+1)\sqrt{s}\Vert v_{[J]}\Vert \le (L^{\prime }+1)\sqrt{s}.$$

Hence the union of the cone sets on the left hand side of (15) is a subset of $A≔\{v\in {\mathbb{R}}^p:\Vert v\Vert \le 1,\Vert v{\Vert }_{2,1}\le (L^{\prime }+1)\sqrt{s}\}$.

We will show that the set A is a subset of $B≔(2+L^{\prime })\mathit{cl}\{\mathit{conv}\{\mathbb{D}(s)\}\}$, the closed convex hull on the right hand side of (15). Since both sets A and B are closed convex, it is enough to show that the support function of A is dominated by the support function of B.

The support function of A is given by ϕ_A(z) = sup_θ∈A〈θ, z〉. For any $z\in {\mathbb{R}}^p$, let S ⊆ {1, … , G} be a subset of top s groups in terms of the ℓ₂ norm of z_[g]. Thus, ∥z_[S^c]∥_{2, ∞} ≤ ∥z_[g]∥ for all g ∈ S. This implies $\Vert z_{[S^c]}{\Vert }_{2,\infty }\le (1∕s)\Vert z_{[S]}{\Vert }_{2,1}\le (1∕\sqrt{s})\Vert z_{[S]}\Vert$. So, we have

$${\varphi }_A(z)=\underset{\theta \in A}{\text{sup}}\langle \theta ,z\rangle \le \underset{\Vert {\theta }_{[S]}\Vert \le 1}{\text{sup}}\langle {\theta }_{[S]},z_{[S]}\rangle +\underset{\Vert {\theta }_{[S^c]}{\Vert }_{2,1}\le \sqrt{s}(L^{\prime }+1)}{\text{sup}}\langle {\theta }_{[S^c]},z_{[S^c]}\rangle$$ (16)

$$\le \Vert z_{[S]}\Vert +(L^{\prime }+1)\sqrt{s}\Vert z_{[S^c]}{\Vert }_{2,\infty }\le (L^{\prime }+2)\Vert z_{[S]}\Vert .$$ (17)

On the other hand, support function of $B≔(L\prime +2^{})\mathit{cl}\{\mathit{conv}\{\mathbb{D}(s)\}\}$ is given by

$${\varphi }_B(z)=\underset{\theta \in B}{\text{sup}}\langle \theta ,z\rangle =(L^{\prime }+2)\underset{\mid U\mid =s,U\subseteq {\mathbb{N}}_G}{\max }\underset{\Vert {\theta }_{[U]}\Vert \le 1}{\text{sup}}\langle {\theta }_{[U]},z_{[U]}\rangle =(L^{\prime }+2)\Vert z_{[S]}\Vert .$$

This concludes the proof. ■

Lemma A.5 Consider a matrix X_n×p with rows independently distributed as N(0, Σ), Λ_min(Σ) > 0. Let ${\mathcal{G}}_1$, … , ${\mathcal{G}}_G$ be any partition of {1, … , p} into G non-overlapping groups of size k₁, … , k_g, respectively. Let C = X′X/n denote the sample Gram matrix and $\mathbb{D}(s)$ denote the set of group s-sparse vectors defined in Lemma A.4. Then, for any integer s ≥ 1 and any η > 0, we have

$$\begin{gathered} \mathbb{P}\left[\underset{v\in cl\{conv\{\mathbb{D}(s)\}\}}{\text{sup}}\mid v^{\prime }(C-\Sigma )v\mid >6\eta \Vert \Sigma \Vert \right] \\ \le c_0\text{exp}[-n\text{min}\{\eta ,{\eta }^2\}+c_{1}s(k_{\max }+c_2\log (eG∕2s))] \end{gathered}$$ (18)

for some universal positive constants C_i.

Proof We consider a fixed vector $v\in {\mathbb{R}}^p$ with ∥v∥ ≤ 1, the support of which can be covered by a set J of at most s groups, i.e., $\mathit{supp}(v)\subseteq {\mathcal{G}}_J$, $J\subseteq {\mathbb{N}}_G$, ∣J∣ ≤ s. Define Y = Xv. Then each coordinate of Y is independently distributed as $N(0,{\sigma }_y^2)$, where ${\sigma }_y^2=v^{\prime }\Sigma v\le \Vert \Sigma \Vert$.

Then, for any η > 0, Hanson-Wright inequality of Rudelson and Vershynin (2013) ensures

$$\mathbb{P}\left[\mid v^{\prime }(C-\Sigma )v\mid >\eta \Vert \Sigma \Vert \right]\le \mathbb{P}\left[\frac{1}{n}\mid Y^{\prime }Y-\mathbb{E}{Y}^{\prime }Y\mid >\eta {\sigma }_y^2\right]\le 2\text{exp}\left[-cn\text{min}\{\eta ,{\eta }^2\}\right].$$

Next, we extend this deviation bound on all vectors v in the sparse set

$$\mathbb{D}(2s)=\{v\in {\mathbb{R}}^p:\Vert v\Vert \le 1,supp(v)\subseteq {\mathcal{G}}_J\text{for some}J\subseteq {\mathbb{N}}_G,\mid J\mid \le 2s\}.$$ (19)

For a given $J\subseteq {\mathbb{N}}_G$, ∣J∣ = 2s, we define ${\mathbb{D}}_J=\{v\in {\mathbb{R}}^p:\Vert v\Vert \le 1,\mathit{supp}(v)\subseteq {\mathcal{G}}_J\}$ and note that $\mathbb{D}(2s)={\cup }_{\mid J\mid =2s}{\mathbb{D}}_J$. For an ϵ > 0 to be specified later, we construct an ϵ-net $\mathcal{A}$ of ${\mathbb{D}}_J$. Since ∑_g∈J k_g ≤, 2s k_max it is possible to construct such a net $\mathcal{A}$ with cardinality at most (1 + 2/ϵ)^2sk_max (Vershynin, 2009).

We want a tail inequality for $M≔{\text{sup}}_{v\in {\mathbb{D}}_J}\mid v^{\prime }\Delta v\mid$, where Δ = C − Σ. Since $\mathcal{A}$ is an ϵ-cover of ${\mathbb{D}}_J$, for any $v\in {\mathbb{D}}_J$, there exists $v_0\in \mathcal{A}$ such that w = v − v₀ satisfies ∥w∥ ≤ ϵ. Then

$$\mid v^{\prime }\Delta v\mid =\mid (w+v_0{)}^{\prime }\Delta (w+v_0)\mid \le \mid w^{\prime }\Delta w\mid +\mid v_0^{\prime }\Delta v_0\mid +2\mid v_0^{\prime }\Delta w\mid .$$

Taking supremum over all $v\in {\mathbb{D}}_J$, and noting that $w∕ϵ\in {\mathbb{D}}_J$, we obtain

$$M\le {ϵ}^{2}M+\underset{v_0\in \mathcal{A}}{\max }\mid v_0^{\prime }\Delta v_0\mid +\underset{u,v\in {\mathbb{D}}_J}{\text{sup}}2ϵ\mid u^{\prime }\Delta v\mid .$$ (20)

To upper bound the third term, note that $(u+v)∕2\in {\mathbb{D}}_J$, and

$$2\mid u^{\prime }\Delta v\mid \le \mid (u+v{)}^{\prime }\Delta (u+v)\mid +\mid u^{\prime }\Delta u\mid +\mid v^{\prime }\Delta v\mid .$$

Hence

$$\underset{u,v\in {\mathbb{D}}_J}{\text{sup}}2ϵ\mid u^{\prime }\Delta v\mid \le 4ϵM+ϵM+ϵM=6ϵM.$$

From equation (20), we now have

$$M\le (1-6ϵ-{ϵ}^2{)}^{-1}\underset{v_0\in \mathcal{A}}{\max }\mid v_0^{\prime }\Delta v_0\mid .$$

Choosing ϵ > 0 small enough so that (1 − 6ϵ − ϵ²) > 1/2, we obtain

$$\begin{gathered} \mathbb{P}\left[\underset{v\in {\mathbb{D}}_J}{\text{sup}}\mid v^{\prime }\Delta v\mid >2\eta \Vert \Sigma \Vert \right]\le \mathbb{P}\left[\underset{v_0\in \mathcal{A}}{\max }\mid v_0^{\prime }\Delta v_0\mid >\eta \Vert \Sigma \Vert \right] \\ \le 2(1+2∕ϵ{)}^{2s{k}_{\max }}\text{exp}[-cn\text{min}\{\eta ,{\eta }^2\}]. \end{gathered}$$

Taking supremum over $\left(\begin{array}{c}\hfill G\hfill \\ \hfill 2s\hfill \end{array}\right)\le (eG∕2s{)}^{2s}$ choices of J, we get

$$\mathbb{P}\left[\underset{v\in \mathbb{D}(2s)}{\text{sup}}\mid v^{\prime }\Delta v\mid >2\eta \Vert \Sigma \Vert \right]\le 2\text{exp}\left[-cn\text{min}\{\eta ,{\eta }^2\}+2s\log \left(\frac{eG}{2s}\right)+2s{k}_{\max }\log \left(1+\frac{2}{ϵ}\right)\right].$$

In order to extend this deviation inequality to $cl\{\mathit{conv}\{\mathbb{D}(s)\}\}$, we note that any v in the convex hull of $\mathbb{D}(s)$ can be expressed as $v={\sum }_{i=1}^m{\alpha }_i{v}_i$, where v₁, … , v_m are in $\mathbb{D}(s)$ and 0 ≤ α_i ≤ 1, ∑ α_i = 1. Then

$$\mid v^{\prime }\Delta v\mid \le \sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\mid v_i^{\prime }\Delta v_j\mid .$$

Also, for every i, j, $(v_i+v_j)∕2\in \mathbb{D}(2s)$, and

$$\mid v_i^{\prime }\Delta v_j\mid \le \frac{1}{2}\left[\mid (v_i+v_j{)}^{\prime }\Delta (v_i+v_j)\mid +\mid v_i^{\prime }\Delta v_i\mid +\mid v_j^{\prime }\Delta v_j\mid \right].$$

Hence

$$\underset{v\in conv\{\mathbb{D}(s)\}}{\text{sup}}\mid v^{\prime }\Delta v\mid \le \sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j\frac{1}{2}[4+1+1]\underset{v\in \mathbb{D}(2s)}{\text{sup}}\mid v^{\prime }\Delta v\mid .$$

Together with the continuity of quadratic forms, this implies

$$\underset{v\in cl\{conv\{\mathbb{D}(s)\}\}}{\text{sup}}\mid v^{\prime }\Delta v\mid \le 3\underset{v\in \mathbb{D}(2s)}{\text{sup}}\mid v^{\prime }\Delta v\mid .$$

The result then readily follows from the above deviation inequality. ■

Appendix B. Proof of Main Results

Proof [Proof of Proposition 3.2] (a) Note that Σ is a p(T − 1) × p(T − 1) block Toeplitz matrix with (i,j)^th block (Σ_ij)_{1≤i,j≤(T−1)} := Γ(i − j), where Γ(ℓ)_p×p is the autocovariance function of lag ℓ for the zero-mean VAR(d) process (2), defined as $\Gamma (\ell )=\mathbb{E}[{\mathbf{X}}^t({\mathbf{X}}^{\mathbf{t}-\ell }{)}^{\prime }]$.

We consider the cross spectral density of the VAR(d) process (2)

$$f(\theta )=\frac{1}{2\pi }\sum _{\ell =-\infty }^{\infty }\Gamma (\ell )e^{-i\ell \theta },\theta \in [-\pi ,\pi ].$$ (21)

From standard results of spectral theory we know that $\Gamma (\ell )={\int }_{-\pi }^{\pi }{e}^{i\ell \theta }f(\theta )d\theta$, for every ℓ.

We want to find a lower bound on the minimum eigenvalue of Σ, i.e., inf_∥x∥=1 x′Σx. Consider an arbitrary p(T − 1)-variate unit norm vector x, formed by stacking the p-tuples x¹, … , x^T−1.

For every θ ∈ [−π, π], define $G(\theta )={\sum }_{t=1}^{T-1}{x}^t{e}^{-\mathit{it}\theta }$ and note that

$$\begin{gathered} {\int }_{-\pi }^{\pi }{G}^{\ast }(\theta )G(\theta )d\theta =\sum _{t=1}^{T-1}\sum _{\tau =1}^{T-1}(x^t{)}^{\prime }(x^{\tau }){\int }_{-\pi }^{\pi }{e}^{i(t-\tau )\theta }d\theta \\ =\sum _{t=1}^{T-1}\sum _{\tau =1}^{T-1}(x^t{)}^{\prime }(x^{\tau })(2\pi 1_{\{t=\tau \}})=2\pi \sum _{t=1}^{T-1}(x^t{)}^{\prime }(x^t)=2\pi \Vert x{\Vert }^2=2\pi . \end{gathered}$$

Also let μ(θ) be the minimum eigenvalue of the Hermitian matrix f(θ). Following Parter (1961) we have the result

$$\begin{gathered} x^{\prime }\Sigma x=\sum _{t=1}^{T-1}\sum _{\tau =1}^{T-1}(x^t{)}^{\prime }\Gamma (t-\tau )x^{\tau }=\sum _{t=1}^{T-1}\sum _{\tau =1}^{T-1}(x^t{)}^{\prime }\left({\int }_{-\pi }^{\pi }{e}^{i(t-\tau )\theta }f(\theta )d\theta \right)x^{\tau } \\ ={\int }_{-\pi }^{\pi }\left(\sum _{t=1}^{T-1}(x^t{)}^{\prime }{e}^{it\theta }\right)f(\theta )\left(\sum _{\tau =1}^{T-1}{x}^{\tau }{e}^{-i\tau \theta }\right)d\theta ={\int }_{-\pi }^{\pi }{G}^{\ast }(\theta )f(\theta )G(\theta )d\theta \\ \ge {\int }_{-\pi }^{\pi }\mu (\theta )(G^{\ast }(\theta )G(\theta ))d\theta \ge \left(\underset{\theta \in (-\pi ,\pi )}{\text{min}}\mu (\theta )\right){\int }_{-\pi }^{\pi }{G}^{\ast }(\theta )G(\theta )d\theta =2\pi \underset{\theta \in (-\pi ,\pi )}{\text{min}}\mu (\theta ). \end{gathered}$$

So ${\Lambda }_{\mathit{min}}(\Sigma )\ge 2\pi \underset{\theta \in (-\pi ,\pi )}{\min }\mu (\theta )$. Since $\mathcal{A}(z)=I-A^{1}z-A^2{z}^2-\dots -A^d{z}^d$ is the (matrix-valued) characteristic polynomial of the VAR(d) model (2), we have the following representation of the spectral density (see Priestley, 1981, eqn 9.4.23):

$$f(\theta )=\frac{1}{2\pi }{\sigma }^2(\mathcal{A}(e^{-i\theta }){)}^{-1}({\mathcal{A}}^{\ast }(e^{-i\theta }){)}^{-1}.$$

Thus, $2\pi \mu (\theta )=2\pi {\Lambda }_{\mathit{min}}(f(\theta ))=2\pi ∕{\Lambda }_{\mathit{max}}(f(\theta {)}^{-1})\ge {\sigma }^2∕{\Vert \mathcal{A}(e^{-i\theta })\Vert }^2$. But $\Vert \mathcal{A}(e^{-i\theta })\Vert \le 1+{\sum }_{t=1}^d\Vert A^t\Vert$ for every θ ∈ [−π, π]. The result then follows at once from the standard matrix norm inequality (see e.g., Golub and Van Loan, 1996, Cor 2.3.2)

$$\Vert A^t{\Vert }_2\le \sqrt{\Vert A^t{\Vert }_1\Vert A^t{\Vert }_{\infty }}\le \frac{\Vert A^t{\Vert }_1+\Vert A^t{\Vert }_{\infty }}{2}t=1,\dots ,d,$$

where

$$\Vert A^t{\Vert }_1=\underset{1\le i\le p}{\max }\sum _{j=1}^p\mid A_{ij}^t\mid ,\Vert A^t{\Vert }_{\infty }=\underset{1\le j\le p}{\max }\sum _{i=1}^p\mid A_{ij}^t\mid .$$

(b) The first part of the proposition ensures that ${\Lambda }_{\mathit{min}}(\Sigma )\ge {\sigma }^2[1+\frac{1}{2}({\mathbf{v}}_{in}+{\mathbf{v}}_{\mathit{out}}){]}^{-2}$. If the replicates available from different panels are i.i.d, each row of the design matrix is independently and identically distributed according to a N(0, Σ) distribution.

To show that RE(s, L) of (5) holds with high probability for sufficiently large n, it is enough to show that

$$\underset{\underset{J\subset {\mathbb{N}}_{\overline{G}},\mid J\mid \le s}{v\in \mathcal{C}(J,L)\setminus \{0\}}}{\text{min}}\frac{1}{n}\frac{\Vert \mathbf{X}v{\Vert }^2}{\Vert v{\Vert }^2}\ge {\varphi }_{RE}^2$$ (22)

holds with high probability, where the cone sets $\mathcal{C}(J,L)$ are defined as

$$\mathcal{C}(J,L)≔\{v\in {\mathbb{R}}^{\overline{p}}:\sum _{g\notin J}{\lambda }_g\Vert v_{[g]}\Vert \le L\sum _{g\in J}{\lambda }_g\Vert v_{[g]}\Vert \}$$ (23)

for all $J\subset {\mathbb{N}}_{\overline{G}}$ with ∣J∣ ≤ s. Denote the ball of unit norm vectors in ${\mathbb{R}}^{\overline{p}}$ by ${\mathbb{S}}^{\overline{p}-1}$. By scale invariance of ∥Xv∥²/n∥v∥², it is enough to show that with high probability

$$\underset{\underset{J\subset {\mathbb{N}}_{\overline{G}},\mid J\mid \le s}{v\in {\mathbb{S}}^{\overline{p}-1}\cap \mathcal{C}(J,L)}}{\text{min}}{v}^{\prime }Cv\ge {\varphi }_{RE}^2,$$ (24)

where C = X′X/n is the sample Gram matrix.

By part (a), we already know that v′Σv ≥ Λ_min(Σ) > 0 for all $v\in {\mathbb{S}}^{\overline{p}-1}$. So we only need to show that ∣v′ (C − Σ) v∣ ≤ Λ_min(Σ)/2 with high probability, uniformly on the set

$$\bigcup _{J\subseteq {\mathbb{N}}_{\overline{G}},\mid J\mid \le s}\mathcal{C}(J,L)\cap {\mathbb{S}}^{\overline{p}-1}.$$ (25)

The proof relies on two key parts. In the first part, we use an extremal representation to show that the above union of the cone sets sits within the closed convex hull of a suitably defined set of group s-sparse vectors. In particular, it follows from Lemma A.4 that

$$\bigcup _{J\subseteq {\mathbb{N}}_{\overline{G}},\mid J\mid \le s}\mathcal{C}(J,L)\cap {\mathbb{S}}^{\overline{p}-1}\subseteq (L^{\prime }+2)cl\{conv\{\mathbb{D}(s)\}\},$$ (26)

where $\mathbb{D}(s)=\{v\in {\mathbb{R}}^{\overline{p}}:\Vert v\Vert \le 1,$, $\mathit{supp}(v)\subseteq {\mathcal{G}}_J$ for some $J\subseteq {\mathbb{N}}_{\overline{G}}$, ∣J∣ ≤ s}, L′ = Lλ_max/λ_min and cl{.}, conv{.} respectively denote the closure and convex hull of a set.

The next part of the proof is an upper bound on the tail probability of v′(C − Σ)v, uniformly over all $v\in cl\{\mathit{conv}\{\mathbb{D}(s)\}\}$, presented in Lemma A.5. In particular, setting η = Λ_min(Σ)/12∥Σ∥(2 + L′)² in the above lemma yields

$$\mathbb{P}\left[\underset{v\in (2+L^{\prime })cl\{conv\{\mathbb{D}(s)\}\}}{\text{sup}}\mid v^{\prime }(C-\Sigma )v\mid >{\Lambda }_{\text{min}}(\Sigma )∕2\right]\le c_0\text{exp}[-c_{1}n]$$ (27)

for the proposed choice of n. Together with the lower bound on Λ_min(Σ) established in part (a), this concludes the proof. ■

Proof [Proof of Theorem 4.1] Consider any solution ${\widehat{\beta }}_R\in {\mathbb{R}}^q$ of the restricted regression

$$\underset{\beta \in {\mathbb{R}}^q}{\text{argmin}}\frac{1}{2n}{\Vert \mathbf{Y}-X_{(1)}\beta \Vert }_2^2+\lambda \sum _{g=1}^s{\Vert {\beta }_{[g]}\Vert }_2$$ (28)

and set $\widehat{\beta }={\left[{\widehat{\beta }}_R^{\prime }:0_{1\times (p-q)}\right]}^{\prime }$. We show that such an augmented vector $\widehat{\beta }$ satisfies the statements of Theorem 4.1 with high probability.

Let $\widehat{u}={\widehat{\beta }}_{(1)}-{\beta }_{(1)}^0={\widehat{\beta }}_R-{\beta }_{(1)}^0$. In view of lemmas A.1 and A.3, it suffices to show that the following events happen with probability at least 1 − 4 G^1−α:

$$\Vert {\widehat{u}}_{[g]}\Vert <{\delta }_n\Vert {\beta }_{[g]}^0\Vert ,\text{for all}g\in S,$$ (29)

$$\frac{1}{n}\Vert {\left[X^{\prime }\left(ϵ-X_{(1)}\widehat{u}\right)\right]}_{[g]}\Vert \le \lambda ,\text{for all}g\notin S.$$ (30)

Note that, in view of Lemma A.1, $\widehat{u}=(C_{11}{)}^{-1}\left(\frac{1}{\sqrt{n}}{Z}_{(1)}-\lambda \tau \right)$ for some $\tau \in {\mathbb{R}}^q$ with ∥τ_[g]∥ ≤ 1 for all g ∈ S, and $Z=\frac{1}{\sqrt{n}}{X}^{\prime }ϵ={\left[Z_{(1)}^{\prime }:Z_{(2)}^{\prime }\right]}^{\prime }$. Thus, for any g ∈ S,

$$\begin{gathered} \mathbb{P}\left(\Vert {\widehat{u}}_{[g]}\Vert >{\delta }_n\Vert {\beta }_{[g]}^0\Vert \right)\le \mathbb{P}\left(\Vert {\left[(C_{11}{)}^{-1}\left(\frac{1}{\sqrt{n}}{Z}_{(1)}-\lambda \tau \right)\right]}_{[g]}\Vert >{\delta }_n\Vert {\beta }_{[g]}^0\Vert \right) \\ \le \mathbb{P}\left(\Vert {\left[(C_{11}{)}^{-1}{Z}_{(1)}\right]}_{[g]}\Vert >\sqrt{n}\left[{\delta }_n\Vert {\beta }_{[g]}^0\Vert -\lambda \Vert {\left[(C_{11}{)}^{-1}\tau \right]}_{[g]}\Vert \right]\right). \end{gathered}$$

Note that V = (C₁₁)⁻¹ Z₍₁₎ ~ N(0, σ² (C₁₁)⁻¹). So $V_{[g]}~N(0,{\sigma }^2{C}_{11}^{[g][g]})$, where Σ^[g][g] := (Σ⁻¹)_[g][g]]. Also, by the second statement of lemma A.2 we have $\mathbb{E}\Vert V_{[g]}\Vert \le \sigma \sqrt{k_g}\sqrt{\Vert C_{11}^{[g][g]}\Vert }$. Therefore $\mathbb{P}\left(\Vert {\widehat{u}}_{[g]}\Vert >{\delta }_n\Vert {\beta }_{[g]}^0\Vert \right)$ is bounded above by

$$\begin{gathered} \mathbb{P}\left(\mid \Vert V_{[g]}\Vert -\mathbb{E}\Vert V_{[g]}\Vert \mid >\sqrt{n}\left[{\delta }_n\Vert {\beta }_{[g]}^0\Vert -\lambda \Vert (C_{11}{)}^{-1}\Vert \sqrt{s}\right]-\sigma \sqrt{k_g\Vert C_{11}^{[g][g]}\Vert }\right) \\ \le 2\text{exp}\left[-\frac{2}{{\pi }^2{\sigma }^2\Vert C_{11}^{[g][g]}\Vert }{\left(\sqrt{n}{\delta }_n\Vert {\beta }_{[g]}^0\Vert -\sqrt{n}\lambda \Vert C_{11}^{-1}\Vert \sqrt{s}-\sigma \sqrt{k_g\Vert C_{11}^{[g][g]}\Vert }\right)}^2\right]. \end{gathered}$$

For the proposed choice of δ_n, this expression is bounded above by 2G^−α. Next, for any g ∉ S, we get

$$\begin{gathered} \mathbb{P}\left(\frac{1}{n}\Vert {\left[X^{\prime }\left(ϵ-X_{(1)}\widehat{u}\right)\right]}_{[g]}\Vert >\lambda \right) \\ \le \mathbb{P}\left(\Vert {\left[Z_{(2)}-C_{21}{C}_{11}^{-1}{Z}_{(1)}\right]}_{[g]}\Vert >\sqrt{n}\lambda \left(1-\Vert {\left[C_{21}{C}_{11}^{-1}\tau \right]}_{[g]}\Vert \right)\right). \end{gathered}$$

Defining $W=Z_{(2)}-C_{21}{C}_{11}^{-1}{Z}_{(1)}~N(0,{\sigma }^2(C_{22}-C_{21}{C}_{11}^{-1}{C}_{12}))$, the uniform irrepresentable condition implies that the above probability is bounded above by $\mathbb{P}(\Vert W_{[g]}\Vert >\sqrt{n}\lambda \eta )$.

It can then be seen that $W_{[g]}\sim N(0,{\sigma }^2{\overline{C}}_{[g][g]})$, where $\overline{C}=C_{22}-C_{21}{C}_{11}^{-1}{C}_{12}$ denotes the Schur complement of C₂₂. As before, lemma A.2 establishes that

$$\begin{gathered} \mathbb{P}\left(\Vert W_{[g]}\Vert >\sqrt{n}\lambda \eta \right)\le \mathbb{P}\left(\mid \Vert W_{[g]}\Vert -\mathbb{E}\Vert W_{[g]}\Vert \mid >\sqrt{n}\lambda \eta -\sigma \sqrt{k_g\Vert {\overline{C}}_{[g][g]}\Vert }\right) \\ \le 2\text{exp}\left[-\frac{2}{{\pi }^2\Vert {\sigma }^2{\overline{C}}_{[g][g]}\Vert }{\left(\sqrt{n}\lambda \eta -\sigma \sqrt{k_g\Vert {\overline{C}}_{[g][g]}\Vert }\right)}^2\right], \end{gathered}$$

and the last probability is bounded above by 2G^−α for the proposed choice of λ.

The results in the proposition follow by considering the union bound on the two sets of the probability statements made across all $g\in {\mathbb{N}}_G$. ■

Appendix C. Proof of results on ℓ₂-consistency

We first note that each of the p optimization problems in (4) is essentially a generic group lasso regression on n independent samples from a linear model Y = Xβ⁰ + ϵ, ϵ ~ N(0, σ²):

$$\widehat{\beta }=\underset{\beta \in {\mathbb{R}}^p}{\text{argmin}}\frac{1}{2n}\Vert \mathbf{Y}-\mathbf{X}\beta {\Vert }^2+\sum _{g=1}^{\overline{G}}{\lambda }_g\Vert {\beta }_{[g]}\Vert ,$$ (31)

where ${\mathbf{Y}}_{n\times 1}={\mathcal{X}}_i^T$, ${\mathbf{X}}_{n\times \overline{p}}=[{\mathcal{X}}^1:\cdots :{\mathcal{X}}^{T-1}]$, ${\beta }_{\overline{p}\times 1}^0=vec(A_{i:}^{1:(T-1)})$, $\{1,\dots ,\overline{p}\}={\cup }_{g=1}^{\overline{G}}{\mathcal{G}}_g$,$\stackrel{‒}{p}=(T-1)p$,$\overline{G}=(T-1)G$ and ${\lambda }_g=\lambda w_{i,g}^t$. In Proposition C.1, we first establish the upper bounds on estimation error in the context of a generic group lasso penalized regression problem. The results for regular group NGC then readily follows by applying the above Proposition on the p separate regressions.

Recall the Restricted Eigenvalue assumption required for the derivation of ℓ₂ estimation and prediction error. Following van de Geer and Bühlmann (2009), we introduce a slightly weaker notion called Group Compatibility (GC). For a constant L > 0 we say that GC(S, L) condition holds, if there exists a constant ϕ_compatible = ϕ_compatible(S, L) > 0 such that

$$\underset{\Delta \in {\mathbb{R}}^p\setminus \{0\}}{\text{min}}\left\{\frac{{\left({\sum }_{g\in S}{\lambda }_g^2\right)}^{1∕2}\Vert X\Delta \Vert }{\sqrt{n}\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert }:\sum _{g\notin S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \le L\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \right\}\ge {\varphi }_{compatible}.$$ (32)

The fact that GC(S, L) holds whenever RE(s, L) is satisfied (and ϕ_RE ≤ ϕ_compatible) follows at once from Cauchy Schwarz inequality. We shall derive upper bounds on the prediction and ℓ_2,1 estimation error of group lasso estimates involving the compatibility constant. This notion will also be used later to connect the irrepresentable conditions to the consistency results of group lasso estimators.

Proposition C.1 Suppose the GC condition (32) holds with L = 3. Choose α > 0 and denote λ_min = min_1≤g≤Gλ_g. If

$${\lambda }_g\ge \frac{2\sigma }{\sqrt{n}}\sqrt{\Vert C_{[g][g]}\Vert }\left(\sqrt{k_g}+\frac{\pi }{\sqrt{2}}\sqrt{\alpha \log G}\right)$$

for every $g\in {\mathbb{N}}_G$, then, the following statements hold with probability at least 1 − 2G^1−α,

$$\frac{1}{n}{\Vert X\left(\widehat{\beta }-{\beta }^0\right)\Vert }^2\le \frac{16}{{\varphi }_{compatible}^2}\sum _{g=1}^s{\lambda }_g^2,$$ (33)

$$\Vert \widehat{\beta }-{\beta }^0{\Vert }_{2,1}\le \frac{16}{{\varphi }_{compatible}^2}\frac{{\sum }_{g=1}^s{\lambda }_g^2}{{\lambda }_{min}}.$$ (34)

If, in addition, RE(2s, 3) holds, then, with the same probability we get

$$\Vert \widehat{\beta }-{\beta }^0\Vert \le \frac{4\sqrt{10}}{{\varphi }_{RE}^2(2s)}\frac{{\sum }_{g=1}^s{\lambda }_g^2}{{\lambda }_{min}\sqrt{s}}.$$ (35)

Proof [Proof of Proposition (C.1)] Since $\widehat{\beta }$ is a solution of the optimization problem (31), for all $\beta \in {\mathbb{R}}^p$, we have

$$\frac{1}{n}\Vert Y-X\widehat{\beta }{\Vert }^2+2\sum _{g=1}^G{\lambda }_g\Vert {\widehat{\beta }}_{[g]}\Vert \le \frac{1}{n}\Vert Y-X\beta {\Vert }^2+2\sum _{g=1}^G{\lambda }_g\Vert {\beta }_{[g]}\Vert .$$

Plugging in Y = Xβ⁰ + ϵ, and simplifying the resulting equation, we get

$$\begin{gathered} \frac{1}{n}\Vert X(\widehat{\beta }-{\beta }^0){\Vert }^2\le \frac{1}{n}\Vert X(\beta -{\beta }^0){\Vert }^2+\frac{2}{n}\sum _{g=1}^G\Vert (X^{\prime }ϵ{)}_{[g]}\Vert \Vert (\widehat{\beta }-\beta {)}_{[g]}\Vert \\ +2\sum _{g=1}^G{\lambda }_g\left(\Vert {\beta }_{[g]}\Vert -\Vert {\widehat{\beta }}_{[g]}\Vert \right). \end{gathered}$$

Fix $g\in {\mathbb{N}}_G$ and consider the event ${\mathcal{A}}_g\left\{ϵ\in {\mathbb{R}}^n:\frac{2}{n}\Vert (X^{\prime }ϵ{)}_{[g]}\Vert \le {\lambda }_g\right\}$. Note that $Z=\frac{1}{\sqrt{n}}{X}^{\prime }ϵ\sim N(0,{\sigma }^{2}C)$. So Z_[g] ~ N(0, σ²C_[g][g]). Then,

$$\begin{gathered} \mathbb{P}({\mathcal{A}}_g^c)=\mathbb{P}\left(\Vert Z_{[g]}\Vert >\frac{1}{2}{\lambda }_g\sqrt{n}\right) \\ \le \mathbb{P}\left(\mid Z_{[g]}-\mathbb{E}\Vert Z_{[g]}\Vert \mid >\frac{{\lambda }_g\sqrt{n}}{2}-\sigma \sqrt{k_g}\sqrt{\Vert C_{[g][g]}\Vert }\right), \end{gathered}$$

where the last inequality follows from the second statement of Lemma A.2. Now, let $x_g=\frac{{\lambda }_g\sqrt{n}}{2}-\sigma \sqrt{k_g}\sqrt{\Vert C_{[g][g]}\Vert }$. Then, for x_g > 0, if

$$2\text{exp}\left(-\frac{2x_g^2}{{\pi }^2{\sigma }^2\Vert C_{[g][g]}\Vert }\right)\le 2G^{-\alpha },$$

we get

$$\mathbb{P}({\mathcal{A}}_g^c)\le 2G^{-\alpha }.$$

But this happens if,

$$\sqrt{2}{x}_g\ge \sqrt{\alpha \log G}\pi \sigma \sqrt{\Vert C_{[g][g]}\Vert },$$

which is ensured by the proposed choice of λ_g.

Next, define $\mathcal{A}≔{\cap }_{g=1}^G{\mathcal{A}}_g$. Then, $\mathbb{P}(\mathcal{A})\ge 1-2G^{1-\alpha }$, and on the event $\mathcal{A}$, we have, for all $\beta \in {\mathbb{R}}^p$,

$$\begin{gathered} \frac{1}{n}\Vert X(\widehat{\beta }-{\beta }^0){\Vert }^2+\sum _{g=1}^G{\lambda }_g\Vert {\widehat{\beta }}_{[g]}-{\beta }_{[g]}\Vert \le \frac{1}{n}\Vert X(\beta -{\beta }^0){\Vert }^2 \\ +2\sum _{g=1}^G{\lambda }_g\left(\Vert {\widehat{\beta }}_{[g]}-{\beta }_{[g]}\Vert +\Vert {\beta }_{[g]}\Vert -\Vert {\widehat{\beta }}_{[g]}\Vert \right). \end{gathered}$$

Note that $\left(\Vert {\widehat{\beta }}_{[g]}-{\beta }_{[g]}\Vert +\Vert {\beta }_{[g]}\Vert -\Vert {\widehat{\beta }}_{[g]}\Vert \right)$ vanishes if g ∉ S and is bounded above by $\text{min}\{2\Vert {\beta }_{[g]}\Vert ,2\left(\Vert {\beta }_{[g]}-{\widehat{\beta }}_{[g]}\Vert \right)\}$ if g ∈ S.

This leads to the following sparsity oracle inequality, for all $\beta \in {\mathbb{R}}^p$,

$$\begin{gathered} \frac{1}{n}\Vert X(\widehat{\beta }-{\beta }^0){\Vert }^2+\sum _{g=1}^G{\lambda }_g\Vert {\widehat{\beta }}_{[g]}-{\beta }_{[g]}\Vert \le \frac{1}{n}\Vert X(\beta -{\beta }^0){\Vert }^2 \\ +4\sum _{g\in S}{\lambda }_g\text{min}\left\{\Vert {\beta }_{[g]}\Vert ,\Vert {\beta }_{[g]}-{\widehat{\beta }}_{[g]}\Vert \right\}. \end{gathered}$$ (36)

The sparsity oracle inequality (36) with β = β⁰, and $\Delta ≔\widehat{\beta }-{\beta }^0$ leads to the following two useful bounds on the prediction and ℓ_2,1-estimation errors:

$$\frac{1}{n}\Vert X\Delta {\Vert }^2\le 4\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert ,$$ (37)

$$\sum _{g\notin S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \le 3\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert .$$ (38)

Now, assume the group compatibility condition 32 holds. Then,

$$\frac{1}{n}\Vert X\Delta {\Vert }^2\le 4\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \le \sqrt{\sum _{g\in S}{\lambda }_g^2}\frac{\Vert X\Delta \Vert }{\sqrt{n}}\frac{4}{{\varphi }_{compatible}},$$ (39)

which implies the first inequality of proposition C.1. The second inequality follows from

$$\begin{gathered} {\lambda }_{min}{\Vert \widehat{\beta }-\beta \Vert }_{2,1}\le \sum _{g=1}^G{\lambda }_g\Vert {\Delta }_{[g]}\Vert \le 4\sum _{g\in S}{\lambda }_g\Vert {\Delta }_{[g]}\Vert \\ \le 4\sqrt{\sum _{g\in S}{\lambda }_g^2}\frac{\Vert X\Delta \Vert }{\sqrt{n}}\frac{1}{{\varphi }_{compatible}}\le \frac{16}{{\varphi }_{compatible}^2}\sum _{g\in S}{\lambda }_g^2, \end{gathered}$$

where the last step uses (39).

The proof of the last inequality of proposition C.1, i.e., the upper bound on ℓ₂ estimation error under RE(2s), is the same as in Theorem 3.1 in Lounici et al. (2011) and is omitted. ■

Proof [Proof of Proposition 3.1] Applying the ℓ₂-estimation error of (35) on the i^th group lasso regression problem of regular group NGC, we have

$$\Vert {\widehat{A}}_{i:}^{1:T-1}-A_{i:}^{1:T-1}\Vert \le \frac{4\sqrt{10}}{{\varphi }_{RE}^2(2s_i)}\frac{{\sum }_{g=1}^{s_i}{\lambda }_g^2}{{\lambda }_{\text{min}}\sqrt{s_i}}\le \frac{4\sqrt{10}}{{\varphi }_{RE}^2(2s_{\max })}\frac{{\lambda }_{\max }}{{\lambda }_{\text{min}}}\sqrt{s_i}$$

with probability at least $1-2{\overline{G}}^{1-\alpha }$. Combining the bounds for all i = 1, … , p and noting that $s={\sum }_{i=1}^p{s}_i$, we have the required result. ■

Appendix D. Irrepresentable assumptions and consistency

In this section, we discuss two results involving the compatibility and irrepresentable conditions for group lasso. We first show that a stronger version of the uniform irrepresentable assumption implies the group compatibility (32), and hence, consistency in ℓ_2,1 norm. Next we argue that a weaker version of the irrepresentable assumption is indeed necessary for the direction consistency of the group lasso estimates. These results generalize analogous properties of lasso (van de Geer and Bühlmann, 2009; Zhao and Yu, 2006) to the group penalization framework. The proofs are given under a special choice of tuning parameter ${\lambda }_g=\lambda \sqrt{k_g}$. Similar results can be derived for the general choice of λ_g, although their presentation is more involved.

Proposition D.1 Assume uniform irrepresentable condition (13) holds with η ∈ (0,1), and Λ_min(C₁₁) > 0. Then group compatibility(S, L) (32) condition holds whenever $L<\frac{1}{1-\eta }$.

Proof First note that with the above choice of λ_g the Group Compatibility (S, L) condition simplifies to

$${\varphi }_{compatible}≔\underset{\Delta \in {\mathbb{R}}^p\setminus \{0\}}{\text{min}}\left\{\frac{\sqrt{q}\Vert X\Delta \Vert }{\sqrt{n}\sum _{g\in S}\sqrt{k_g}\Vert {\Delta }_{[g]}\Vert }:\sum _{g\notin S}\sqrt{k_g}\Vert {\Delta }_{[g]}\Vert \le L\sum _{g\in S}\sqrt{k_g}\Vert {\Delta }_{[g]}\Vert \right\}>0.$$ (40)

Also, the uniform irrepresentable condition guarantees that there exists 0 < η < 1 such that $\forall \tau \in {\mathbb{R}}^q$ with $\Vert \tau {\Vert }_{2,\infty }=\underset{1\le g\le s}{\max }\Vert {\tau }_{[g]}{\Vert }_2\le 1$, we have,

$$\frac{1}{\sqrt{k_g}}{\Vert {\left[C_{21}(C_{11}{)}^{-1}{K}_{\tau }^0\right]}_{[g]}\Vert }_2<1-\eta \forall g\notin S.$$

Here K⁰ = K/λ is a q × q block diagonal matrix with diagonal blocks $\sqrt{k_1}{\mathbf{I}}_{k_1\times k_1},\dots ,\sqrt{k_s}{\mathbf{I}}_{k_s\times k_s}$. Define

$${\Delta }^0≔\underset{\Delta \in {\mathbb{R}}^p}{\text{argmin}}\left\{\frac{1}{2n}\Vert \mathbf{X}\Delta {\Vert }_2^2:\sum _{g\in S}\sqrt{k_g}\Vert {\Delta }_{[g]}{\Vert }_2=1,\sum _{g\notin S}\sqrt{k_g}\Vert {\Delta }_{[g]}{\Vert }_2\le L\right\}.$$ (41)

Note that $\frac{1}{n}\Vert \mathbf{X}{\Delta }^0{\Vert }_2^2={\varphi }_{compatilble}^2∕q$, and introduce two Lagrange multipliers λ and λ′ corresponding to the equality and inequality constraints for solving the optimization problem in (41). Also, partition ${\Delta }^0=\left[{\Delta }_{(1)}^0:{\Delta }_{(2)}^0\right]$ and X = [X₍₁₎ : X₍₂₎] into signal and nonsignal parts as in (10). The first q linear equations of the KKT conditions imply that there exists ${\tau }^0\in {\mathbb{R}}^q$ such that

$$C_{11}{\Delta }_{(1)}^0+C_{12}{\Delta }_{(2)}^0=\lambda K^0{\tau }^0$$ (42)

and, for every g ∈ S,

$$\begin{gathered} {\tau }_{[g]}^0=D({\Delta }_{[g]}^0)\text{if}{\Delta }_{[g]}^0\ne 0, \\ \Vert {\tau }_{[g]}^0{\Vert }_2\le 1\text{if}{\Delta }_{[g]}^0=0. \end{gathered}$$

It readily follows that $({\tau }^0{)}^T{K}^0{\Delta }_{(1)}^0=\sum _{g\in S}\sqrt{k_g}\Vert {\Delta }_{[g]}^0{\Vert }_2=1$.

Multiplying both sides of (42) by $({\Delta }_{(1)}^0{)}^T$ we get

$${\left({\Delta }_{(1)}^0\right)}^T{C}_{11}{\Delta }_{(1)}^0+{\left({\Delta }_{(1)}^0\right)}^T{C}_{12}{\Delta }_{(2)}^0=\lambda .$$ (43)

Also, (42) implies

$${\Delta }_{(1)}^0+(C_{11}{)}^{-1}{C}_{12}{\Delta }_{(2)}^0=\lambda (C_{11}{)}^{-1}{K}^0{\tau }^0.$$ (44)

Multiplying both sides of the equation by (K⁰τ⁰)^T = (τ⁰)^T K⁰ we obtain

$$1=-({\tau }^0{)}^T{K}^0(C_{11}{)}^{-1}{C}_{12}{\Delta }_{(2)}^0+\lambda (K^0{\tau }^0{)}^T(C_{11}{)}^{-1}(K^0{\tau }^0).$$ (45)

Note that the absolute value of the first term,

$$\mid \sum _{g\notin S}{\left({\Delta }_{[g]}^0\right)}^T{\left[C_{21}(C_{11}{)}^{-1}{K}^0{\tau }^0\right]}_{[g]}\mid ,$$ (46)

is bounded above by

$$(1-\eta )\left(\sum _{g\notin S}\sqrt{k_g}\Vert {\Delta }_{[g]}^0{\Vert }_2\right)\le (1-\eta )L$$ (47)

by virtue of the uniform irrepresentable condition and the Cauchy-Schwartz inequality. Assuming the minimum eigenvalue of C₁₁, i.e., Λ_min (C₁₁), is positive and considering $\Vert K^0{\tau }^0{\Vert }_2\le \sqrt{q}$, the second term is at most λq/Λ_min (C₁₁). So (45) implies

$$1\le (1-\eta )L+\frac{\lambda q}{{\Lambda }_{min}(C_{11})}.$$ (48)

In particular, λ ≥ Λ_min (C₁₁) (1 − (1 − η)L)/q is positive whenever L < 1/(1 − η). Next, multiply both sides of (44) by $({\Delta }_{(2)}^0{)}^T{C}_{21}$ to get

$${\left({\Delta }_{(2)}^0\right)}^T{C}_{21}{\Delta }_{(1)}^0+{\left({\Delta }_{(2)}^0\right)}^T{C}_{21}(C_{11}{)}^{-1}{C}_{(12)}{\Delta }_{(2)}^0=\lambda {\left({\Delta }_{(2)}^0\right)}^T{C}_{21}(C_{11}{)}^{-1}{K}^0{\tau }^0.$$ (49)

Using the upper bound in (47), the right hand side is at least −λ(1 − η)L.

Also a simple consequence of the block inversion formula of the non-negative definite matrix C guarantees that the matrix C₂₂ − C₂₂ (C₁₁)⁻¹ C₁₂ is non-negative definite. Hence,

$$\begin{gathered} {\left({\Delta }_{(2)}^0\right)}^T\left[C_{22}-C_{21}(C_{11}{)}^{-1}{C}_{12}\right]{\Delta }_{(2)}^0\ge 0 \\ \text{and}{\left({\Delta }_{(2)}^0\right)}^T{C}_{22}{\Delta }_{(2)}^0\ge {\left({\Delta }_{(2)}^0\right)}^T{C}_{21}(C_{11}{)}^{-1}{C}_{12}{\Delta }_{(2)}^0. \end{gathered}$$

Putting all the pieces together we get

$$\begin{gathered} {\varphi }_{compatible}^2∕q=\frac{1}{n}\Vert \mathbf{X}{\Delta }^0{\Vert }_2^2 \\ ={{\Delta }_{(1)}^0}^T{C}_{11}{\Delta }_{(1)}^0+2{{\Delta }_{(2)}^0}^T{C}_{21}{\Delta }_{(1)}^0+{{\Delta }_{(2)}^0}^T{C}_{22}{\Delta }_{(2)}^0 \\ =\lambda +{{\Delta }_{(2)}^0}^T{C}_{21}{\Delta }_{(1)}^0+{{\Delta }_{(2)}^0}^T{C}_{22}{\Delta }_{(2)}^0,\text{by}(43) \\ \ge \lambda -\lambda (1-\eta )L,\text{by}(49) \\ =\lambda (1-(1-\eta )L). \end{gathered}$$

Plugging in the lower bound for λ we obtain the result; namely,

$${\varphi }_{compatible}^2={\Lambda }_{min}(C_{11})(1-(1-\eta )L{)}^2>0$$

for any $L<\frac{1}{1-\eta }$. ■

In this section we investigate the necessity of irrepresentable assumptions for direction consistency of group lasso estimates. To this end we first introduce the notion of weak irrepresentability.

For a q-dimensional vector τ define the stacked direction vector

$$\underset{q\times 1}{\underbrace{\tilde{D}(\tau )}}=[\underset{k_1\times 1}{\underbrace{D({\tau }_{[1]}{)}^{\prime }}},\dots ,\underset{k_s\times 1}{\underbrace{D({\tau }_{[s]}{)}^{\prime }}}{]}^{\prime }.$$

Weak Irrepresentable Condition is satisfied if

$$\frac{1}{{\lambda }_g}\Vert {\left[C_{21}(C_{11}{)}^{-1}K\tilde{D}({\beta }_{(1)}^0)\right]}_{[g]}\Vert \le 1,\forall g\notin S=\{1,\dots ,s\}.$$ (50)

We argue the necessity of weak irrepresentable condition for group sparsity selection and direction consistency under two regularity conditions on the design matrix, as n, p → ∞:

(A1) The minimum eigenvalue of the signal part of the Gram matrix, viz. Λ_min(C₁₁), is bounded away from zero.

(A2) The matrices C₂₁ and C₂₂ are bounded above in spectral norm.

As in the last proposition, we set ${\lambda }_g=\lambda \sqrt{k_g}$ and K⁰ = K/λ. Suppose that the weak irrepresentable condition does not hold, i.e., for some g ∉ S and ξ > 0, we have,

$$\frac{1}{\sqrt{k_g}}\Vert {\left[C_{21}(C_{11}{)}^{-1}{K}^0\tilde{D}({\beta }_{(1)}^0)\right]}_{[g]}\Vert >1+\xi$$

for infinitely many n. Also suppose that there exists a sequence of positive reals δ_n → 0 such that the event

$$E_n≔\{\Vert D({\widehat{\beta }}_{[g]})-D({\beta }_{[g]}){\Vert }_2<{\delta }_n,\forall g\in S,\text{and}{\widehat{\beta }}_{[g]}=0\forall g\notin S\}$$

satisfies $\mathbb{P}(E_n)\to 1$ as p, n → ∞.

Note that for large enough n so that δ_n < min_g ∥D(β_[g])∥, we have ${\widehat{\beta }}_{[g]}\ne 0$, ∀_g ∈ S on the event E_n.

Then, as in the proof of Theorem 4.1, we have, on the event E_n,

$$\widehat{\mathbf{u}}=(C_{11}{)}^{-1}\left[\frac{1}{\sqrt{n}}{Z}_{(1)}-\lambda K^0\tilde{D}({\widehat{\beta }}_{(1)})\right]$$ (51)

$$\text{and}\frac{1}{n}\Vert [{{\mathbf{X}}_{(2)}}^T(ϵ-{\mathbf{X}}_{(1)}\widehat{u}){]}_{[g]}\Vert \le \lambda \sqrt{k_g},\forall g\notin S.$$ (52)

Substituting the value of $\widehat{u}$ from (51) in (52), we have, on the event E_n,

$$\frac{1}{\sqrt{n}}\Vert {\left[{\mathbf{Z}}_{(2)}-C_{21}(C_{11}{)}^{-1}{\mathbf{Z}}_{(1)}+\lambda \sqrt{n}{C}_{21}(C_{11}{)}^{-1}{K}^0\tilde{D}({\widehat{\beta }}_{(1)})\right]}_{[g]}\Vert \le \lambda \sqrt{k_g},$$

which implies that

$$\begin{gathered} \Vert {\left[Z_{(2)}-C_{21}(C_{11}{)}^{-1}{Z}_{(1)}\right]}_{[g]}\Vert \\ \ge \lambda \sqrt{n}\sqrt{k_g}\left[\frac{1}{\sqrt{k_g}}\Vert {\left[C_{21}(C_{11}{)}^{-1}{K}^0\tilde{D}({\widehat{\beta }}_{(1)})\right]}_{[g]}\Vert -1\right]. \end{gathered}$$ (53)

Now note that for large enough n, if ∥C₂₁∥ is bounded above, direction consistency guarantees that the expression on the right is larger than

$$\frac{1}{2}\lambda \sqrt{n}\sqrt{k_g}\left[\frac{1}{\sqrt{k_g}}\Vert {\left[C_{21}(C_{11}{)}^{-1}{K}^0\tilde{D}({\beta }_{(1)})\right]}_{[g]}\Vert -1\right],$$

which in turn is larger than $\frac{1}{2}\lambda \sqrt{n}\sqrt{k_g}\xi$, in view of the weak irrepresentable condition.

This contradicts $\mathbb{P}(E_n)\to 1$, since the left-hand side of (53) corresponds to the norm of a centered Gaussian random variable with bounded variance structure ${\left[C_{22}-C_{21}{C}_{11}^{-1}{C}_{12}\right]}_{[g][g]}$ while $\lambda \sqrt{n}\sqrt{k_g}$ diverges with $\sqrt{\log G}$.

Appendix E. Thresholding Group Lasso Estimates.

Proof [Proof of Theorem 4.2] We use the notations developed in the proof of Proposition C.1. First note that, (ii) follows directly from Theorem 4.1. For (i), since the falsely selected groups are present after the initial thresholding, we get $\Vert {\widehat{\beta }}_{[g]}\Vert >4\lambda$ for every such group. Next, we obtain an upper bound for the number of such groups. Specifically, denoting $\Delta =\widehat{\beta }-{\beta }^0$, we get

$$\mid \widehat{S}\setminus S\mid \le \frac{\Vert {\widehat{\beta }}_{S^c}{\Vert }_{2,1}}{4\lambda }=\frac{{\sum }_{g\notin S}\Vert {\Delta }_{[g]}\Vert }{4\lambda }.$$ (54)

Next, note that from the sparsity oracle inequality (37), the following holds on the event $\mathcal{A}$,

$$\sum _{g\notin S}\Vert {\Delta }_{[g]}\Vert \le 3\sum _{g\in S}\Vert {\Delta }_{[g]}\Vert .$$

It readily follows that

$$4\sum _{g\notin S}\Vert {\Delta }_{[g]}\Vert \le 3\Vert \Delta {\Vert }_{2,1}\le \frac{48}{{\varphi }^2}s\lambda ,$$

where the last inequality follows from the ℓ_2,1-error bound of (34). Using this inequality together with (54) gives the result. ■