Estimating networks with jumps

Abstract

We study the problem of estimating a temporally varying coefficient and varying structure (VCVS) graphical model underlying data collected over a period of time, such as social states of interacting individuals or microarray expression profiles of gene networks, as opposed to i.i.d. data from an invariant model widely considered in current literature of structural estimation. In particular, we consider the scenario in which the model evolves in a piece-wise constant fashion. We propose a procedure that estimates the structure of a graphical model by minimizing the temporally smoothed L1 penalized regression, which allows jointly estimating the partition boundaries of the VCVS model and the coefficient of the sparse precision matrix on each block of the partition. A highly scalable proximal gradient method is proposed to solve the resultant convex optimization problem; and the conditions for sparsistent estimation and the convergence rate of both the partition boundaries and the network structure are established for the first time for such estimators.

1. Introduction

Networks are a fundamental form of representation of relational information underlying large, noisy data from various domains. For example, in a biological study, nodes of a network can represent genes in one organism and edges can represent associations or regulatory dependencies among genes. In a social analysis, nodes of a network can represent actors and edges can represent interactions or friendships between actors. Exploring the statistical properties and hidden characteristics of network entities, and the stochastic processes behind temporal evolution of network topologies is essential for computational knowledge discovery and prediction based on network data.

In many dynamical environments, such as a developing biological system, it is often technically impossible to experimentally determine the network topologies specific to every time point in a certain time period. Resorting to computational inference methods, such as extant structural learning algorithms, is also difficult because for every model unique to a single time point, there exist as few as only a single snapshot of the nodal states distributed accordingly to the model in question. In this paper, we consider an estimation problem under a particular dynamic context, where the model evolves piecewise constantly, i.e., staying structurally invariant during unknown segments of time, and then jump to a different structure.

Approximately piecewise constantly evolving networks can be found underlying many natural dynamic systems of intellectual and practical interest. For example, in a biological developmental system such as the fruit fly, the entire life cycle of the fly consists of 4 discrete developmental stages, namely, embryo, larva, pupa, and adult; across the stages, one expect to see dramatical rewiring of the regulatory network to realize very different regulation functions due to different developmental needs, whereas within each stage, the change of the network topology are expected to be relatively more mild as revealed by the smoother trajectories of the gene expression activities, because a largely stable regulatory machinery is employed to control stage-specific developmental processes. Such phenomena are not uncommon in social systems. For example, in an underlying social network between the senators, even it is not visible to outsiders, we would imagine the network structure being more stable between the elections but more volatile when the campaigns start. Although it is legitimate to use a completely unconstrained time-evolving network model to describe or analysis such systems, an approximately piecewise constantly evolving network model is better at capturing the different amount of network dynamics during different phases of a entire life cycle, and detecting boundaries between different phases when desirable.

A popular technique for deriving the network structure from iid sample is to estimate a sparse precision matrix. The importance of estimating precision matrices with zeros was recognized by Depmster [11] who coined the term covariance selection. The elements of the precision matrix represent the associations or conditional covariances between corresponding variables. Once a sparse precision matrix is estimated, a network can be drawn by connecting variables whose corresponding elements of the precision matrix are non-zero. Recent studies have shown that covariance selection methods based on the penalized likelihood maximization can lead to a consistent estimate of the network structure underlying a Gaussian Markov Random Fields [12, 32]. Moreover, a particular procedure for covariance selection known as neighborhood selection, which is built on ℓ₁ norm regularized regression, can produce a consistent estimate of the network structure when the sample is assumed to follow a general Markov Random Field distribution whose structure corresponds to the network in question [33, 28, 31]. Specifically, a Markov Random Field (MRF) is a probabilistic graphical model defined on a graph G = (V, E), where V = {1, …, p} is a vertex set corresponding to the set of random variables to be modeled (in this paper we call them nodes and variables interchangeably), and E ⊆ V × V is the edge set capturing conditional independencies among these nodes. Let X = (X₁, …, X_p)′ denote a p-dimensional random vector, whose elements are indexed by the nodes of the graph G. Under the MRF, a pair (a, b) is not an element of the edge set E if and only if the variable X_a is conditionally independent of X_b given all the rest of variables X_V\{a,b}, X_a ⊥ X_b|X_V\{a,b}. A distribution over X can be defined by taking the following log linear form that makes explicit use of the (presence and absence of edges in the) edge set: p(X) ∝ exp{Σ_(a,b)∈V θ_abX_aX_b}. When the elements of the random vector X are discrete, e.g., X ∈ {0, 1}^p, the model is referred to as a discrete MRF, sometimes known as an Ising model in statistics physics community; whereas when X is a continuous vector, the model is referred to as a Gaussian graphical model (GGM) because one can easily show that the p(X) above is actually a multivariate Gaussian. The MRF have been widely used for modeling data with graphical relational structures over a fixed set of entities [39, 14]. The vertices can describe entities such as genes in a biological regulatory network, stocks in the market, or people in society; while the edges can describe relationships between vertices, for example, interaction, correlation or influence.

The statistical problem we concern in this paper is to estimate the structure of the Gaussian graphical model from observed samples of nodal states in a dynamic world. Traditional methods handle this problem with the assumption that the samples are iid. Let = {x₁, …, x_n} be an independent and identically distributed sample according to a p-dimensional multivariate normal distribution (0, Σ), where Σ is the covariance matrix. Let Ω := Σ⁻¹ denote the precision matrix, with elements (ω_ab), 1 ≤ a, b ≤ p. Then one can obtain an estimator of the Ω from via optimizing a proper statistical loss function, such as likelihood or penalized likelihood. As mentioned earlier, the precision matrix Ω encodes the conditional independence structure of the distribution and the pattern of the zero elements in the precision matrix define the structure of the associated graph G. There has been a dramatic growth of interest in recent literature in the problem of covariance selection, which deals with the graph estimation problem above. Existing works range from algorithmic development focusing on efficient estimation procedures, to theoretical analysis focusing on statistical guarantees of different estimators. We do not intend to give an extensive overview of the literature here, but interested readers can follow the pointers bellow. In the classical literature (e.g., [22]), procedures are developed for small dimensional graphs and commonly involve hypothesis testing with greedy selection of edges. More recent literature estimates the sparse precision matrix by optimizing penalized likelihood [42, 12, 4, 35, 13, 32, 16, 44] or through neighborhood selection [28, 31, 15, 40], where the structure of the graph is estimated by estimating the neighborhood of each node. Both of these approaches are suitable for high-dimensional problems, even when p ≫ n, and can be efficiently implemented using scalable convex program solvers.

Most of the above mentioned work assumes that a single invariant network model is sufficient to describe the dependencies in the observed data. However, when the observed data are not iid, such an assumption is not justifiable. For example, when data consist of microarray measurements of the gene expression levels collected throughout the cell cycle or development of an organism, different genes can be active during different stages. This suggests that different distributions and hence different networks should be used to describe the dependencies between measured variables at different time intervals. In this paper, we are going to tackle the problem of estimating the structure of the GGM when the structure is allowed to change over time. By assuming that the parameters of the precision matrix change with time, we obtain extra flexibility to model a larger class of distributions while still retaining the interpretability of the static GGM. In particular, as the coefficients of the precision matrix change over time, we also allow the structure of the underlying graph to change as well. This semi-parametric generalization of the parametric model is referred to as a varying coefficient varying structure (VCVS) model.

Now, let {x_i}_i∈[n] ∈ ℝ^p be a sequence of n conditionally independent observations ¹ (we use [n] to denote the set {1, …, n}) from some p-dimensional multivariate normal distributions, not necessarily the same for every observation. Let { }_j∈[B] be a disjoint partitioning of the set [n] where each block of the partition consists of consecutive elements, that is, ∩ = ∅ for j ≠ j′ and ⋃_j = [n] and = [T_j−1 : T_j] := {T_j−1, T_j−1 + 1, …, T_j − 1}. Let := {T₀ = 1 < T₁ < ··· < T_B = n + 1} denote the set of partition boundaries.

We consider the following model

$${\mathbf{x}}_i~{\mathcal{N}}_p(\mathbf{0},{\mathbf{\sum }}^j),i\in {\mathcal{B}}^j,$$ (1.1)

such that observations indexed by elements in are p-dimensional realizations of a multivariate normal distribution with zero mean and the covariance matrix ${\mathbf{\sum }}^j={({\sigma }_{ab}^j)}_{a,b\in [p]}$, which suggest that it is only unique to segment j the time series. Let Ω^j := (Σ^j)⁻¹ denote the precision matrix with elements ${({\omega }_{ab}^j)}_{a,b\in [p]}$. With the number of partitions, B, and the boundaries of partitions, , unknown, we study the problem of estimating both the partition set { } and the non-zero elements of the precision matrices {Ω^j}_j∈[B] from the sample {x_i}_i∈[n]. Note that in this work we study a particular case of the VCVS model, where the coefficients are piece-wise constant functions of time. Although this model does not yet entirely agree with how a real world time series data would behave, as none existing model does, this instantiation of the VCVS model come one step closer in some sense to the real world scenario than other popular approaches for time series modeling, such as Hidden Markov Models or state space models, where stationary emission models such as linear Gaussian are usually employed to relate observation at different time points to simple latent states. Here, instead of positing an observation at time t to be derived from a latent state that transitions stationarily from a previous time point, we assume that such an observation is generated from a latent network model that are related to the network models active at the previous and subsequent time points nonparametrically. As suggested in the introduction, many real world dynamic systems, such as the stage-specific development of multi-cellular organisms like the fruit fly, and the evolving network of latent relatedness between politicians, are likely to behave approximately piecewise constantly; therefore time series data from such systems, such as the continuous-valued gene expression microarray time series, and the discrete-valued voting records, are suitable examples where our proposed models can be applied to [1]. A scenario where the coefficients are smoothly varying functions of time has been considered in [44] for the GGM and in [21] and [19] for an Ising model, which complements the model studied in this paper, whose asymptotic properties are somewhat easier to analyze as we have shown earlier.

If the partitions { }_j were known, the problem would be trivially reduced to the setting analyzed in the previous work. Dealing with the unknown partitions, together with the structure estimation of the model, calls for new methods. We propose and analyze a method based on time-coupled neighborhood selection, where the model estimates are forced to stay similar across time using a fusion-type total variation penalty and the sparsity of each neighborhood is obtained through the ℓ₁ penalty. Details of the approach are given in §2.

The model in Eq. (1.1) is related to the varying-coefficient models (e.g., [18]) with the coefficients being piece-wise constant functions. Varying coefficient regression models with piece-wise constant coefficients are also known as segmented multivariate regression models [24] or linear models with structural changes [2]. The structural changes are commonly determined through hypothesis testing and a separate linear model is fit to each of the estimated segments. In our work, we use the penalized model selection approach to jointly estimate the partition boundaries and the model parameters.

Little work has been done so far towards modeling dynamic networks and estimating changing precision matrices. [44] develops a nonparametric method for estimation of time-varying GGM, where x^t ~ (0, Σ(t)) and Σ(t) is smoothly changing over time. The procedure is based on the penalized likelihood approach of [42] with the empirical covariance matrix obtained using a kernel smoother. Our work is very different from the one of [44], since under our assumptions the network changes abruptly rather than smoothly. Furthermore, as we outline in §2, our estimation procedure is not based on the penalized likelihood approach. Estimation of time-varying Ising models has been discussed in [1] and [21]. [41] and [20] studied nonparametric ways to estimate the conditional covariance matrix. The work of [1] is most similar to our setting, where they also use a fused-type penalty combined with an ℓ₁ penalty to estimate the structure of the varying Ising model. Here, in addition to focusing on GGMs, there is an additional subtle, but important, difference to [1]. In this work, we use a modification of the fusion penalty (formally described in §2) which allows us to characterize the model selection consistency of our estimates and the convergence properties of the estimated partition boundaries, which is not available in the earlier work.

The remaining of the paper is organized as follows. In §2, we describe our estimation procedure and provide an efficient first-order optimization procedure capable of estimating large graphs. The optimization procedure is based on the smoothing procedure of [29] and converges in (1/ε) iterations, where ε is the desired accuracy. Our main theoretical results are presented in §3. In particular, we show that the partition boundaries are estimated consistently. Furthermore, the graph structure is consistently estimated on every block of the partition that contains enough samples. In §4, we discuss alternative estimation procedures based on penalized maximum likelihood estimation, instead of the neighborhood selection. Numerical studies showing the finite sample performance of our procedure are given in §5. The proofs of the main results are relegated to §7, with some technical details presented in Appendix.

Notation schemes

For clarity, we end the introduction with a summary of the notations used in the paper. We use [n] to denote the set {1, …, n} and [l : r] to denote the set {l, l + 1, …, r − 1}. We use to denote j-th block of the partition . With some abuse of notation, we also use to denote the set [T_j−1 : T_j]. The number of samples in the block is denoted as | |. For a set S ⊂ V, we use the notation X_S to denote the set {X_a : a ∈ S} of random variables. We use X to denote the n × p matrix whose rows consist of observations. The vector X_a = (x_1,a, …, x_n,a)′ denotes a column of matrix X and, similarly, X_S = (X_b : b ∈ S) denotes the n × |S| sub-matrix of X whose columns are indexed by the set S and X denotes the sub-matrix | | × p whose rows are indexed by the set . For simplicity of notation, we will use \a to denote the index set [p]\{a}, X_\a = (X_b : b ∈ [p]\{a}). For a vector a ∈ ℝ^p, we let S(a) denote the set of non-zero components of a. Throughout the paper, we use c₁, c₂, … to denote positive constants whose value may change from line to line. For a vector a ∈ ℝⁿ, define ||a||₁ = Σ_i∈[n] |a_i|, ${\Vert \mathbf{a}\Vert }_2=\sqrt{{\sum }_{i\in [n]}{a}_i^2}$ and ||a||_∞ = max_i |a_i|. For a symmetric matrix A, Λ_min(A) denotes the smallest and Λ_max(A) the largest eigenvalue. For a matrix A (not necessarily symmetric), we use |||A|||_∞ = max_i Σ_j |A_ij|. For two vectors a, b ∈ ℝⁿ, the dot product is denoted 〈a, b〉 = Σ_i∈[n] a_ib_i. For two matrices A, B ∈ ℝ^n×m, the dot product is denoted as 〈〈A, B〉〉 = tr(A′B). Given two sequences {a_n} and {b_n}, the notation a_n = (b_n) means that there exists a constant c₁ such that a_n ≤ c₁b_n; the notation a_n = Ω(b_n) means that there exists a constant c₂ such that a_n ≥ c₂b_n and the notation a_n ≍ b_n means that a_n = (b_n) and b_n = (a_n). Similarly, we will use the notation a_n = o_p(b_n) to denote that $b_n^{-1}{a}_n$ converges to 0 in probability.

2. Graph estimation via Temporal-Difference Lasso

In this section, we introduce our time-varying covariance selection procedure, which is based on the time-coupled neighborhood selection using the fused-type penalty. We call the proposed procedure Temporal-Difference Lasso (TD-Lasso). We start by reviewing the basic neighborhood selection procedure, which has previously been used to estimate graphs in, e.g., [31, 28, 33, 15].

We start by relating the elements of the precision matrix Ω to a regression problem. Let the set S_a to denote the neighborhood of the node a. Denote S̄_a the closure of S_a, S̄_a: = S_a ∪ {a}, and N_a the set of nodes not in the neighborhood of the node a, N_a = [p]\S̄_a. It holds that X_a ⊥ X_Na|X_Sa. The neighborhood of the node a can be easily seen from the non-zero pattern of the elements in the precision matrix Ω, S_a = {b ∈ [p]\{a} : ω_ab ≠ 0}. See [22] for more details. It is a well known result for Gaussian graphical models that the elements of

$${\mathit{\theta }}^a=\underset{\mathit{\theta }\in {ℝ}^{p-1}}{\text{argmin}}\mathbb{E}{\left(X_a-\sum _{b\in \backslash a}{X}_b{\theta }_b\right)}^2$$

are given by ${\theta }_b^a=-{\omega }_{ab}/{\omega }_{aa}$. Therefore, the neighborhood of a node a, S_a, is equal to the set of non-zero coefficients of θ^a. Using the expression for θ^a, we can write $X_a={\sum }_{b\in S_a}{X}_b{\theta }_b^a+\epsilon$, where ε is independent of X_\a.

The neighborhood selection procedure was motivated by the above relationship between the regression coefficients and the elements of the precision matrix. [28] proposed to solve the following optimization procedure

$${\widehat{\mathit{\theta }}}^a=\underset{\mathit{\theta }\in {ℝ}^{p-1}}{\text{argmin}}\frac{1}{n}{\Vert {\mathbf{X}}_a-{\mathbf{X}}_{\backslash a}\mathit{\theta }\Vert }_2^2+\lambda {\Vert \mathit{\theta }\Vert }_1$$ (2.1)

and proved that for iid sample the non-zero coefficients of θ̂^a consistently estimate the neighborhood of the node a, under a suitably chosen penalty parameter λ.

In this paper, we build on the neighbourhood selection procedure to estimate the changing graph structure in model (1.1). We use $S_a^j$ to denote the neighborhood of the node a on the block and $N_a^j$ to denote nodes not in the neighborhood of the node a on the j-th block, $N_a^j=V\backslash S_a^j$. Consider the following estimation procedure

$${\widehat{\mathit{\beta }}}^a=\underset{\mathit{\beta }\in {ℝ}^{p-1\times n}}{\text{argmin}}\mathcal{L}(\mathit{\beta })+{\text{pen}}_{{\lambda }_1,{\lambda }_2}(\mathit{\beta })$$ (2.2)

where the loss is defined for β = (β_b,i)_{b∈[p−1],i∈[n]} as

$$\mathcal{L}(\mathit{\beta }):=\sum _{i\in [n]}{\left(x_{i,a}-\sum _{b\in \backslash a}{x}_{i,b}{\beta }_{b,i}\right)}^2$$ (2.3)

and the penalty is defined as

$${\text{pen}}_{{\lambda }_i,{\lambda }_2}(\mathit{\beta }):=2{\lambda }_1\sum _{i=2}^n{\Vert {\mathit{\beta }}_{·,i}-{\mathit{\beta }}_{·,i-1}\Vert }_2+2{\lambda }_2\sum _{i=1}^n\sum _{b\in \backslash a}\mid {\beta }_{b,i}\mid .$$ (2.4)

The penalty term is constructed from two terms. The first term ensures that the solution is going to be piecewise constant for some partition of [n] (possibly a trivial one). The first term can be seen as a sparsity inducing term in the temporal domain, since it penalizes the difference between the coefficients β_·,i and β_·,i+1 at successive time-points. The second term results in estimates that have many zero coefficients within each block of the partition. The estimated set of partition boundaries

$$\widehat{\mathcal{T}}=\{{\widehat{T}}_0=1\}\cup \{{\widehat{T}}_j\in [2:n]:{\widehat{\mathit{\beta }}}_{·,{\widehat{T}}_j}^a\ne {\widehat{\mathit{\beta }}}_{·,{\widehat{T}}_j-1}^a\}\cup \{{\widehat{T}}_{\widehat{B}}=n+1\}$$

contains indices of points at which a change is estimated, with B̂ being an estimate of the number of blocks B. The estimated number of the block B̂ is controlled through the user defined penalty parameter λ₁, while the sparsity of the neighborhood is controlled through the penalty parameter λ₂.

Based on the estimated set of partition boundaries , we can define the neighborhood estimate of the node a for each estimated block. Let ${\widehat{\mathit{\theta }}}^{a,j}={\widehat{\mathit{\beta }}}_{·,i}^a$, ∀i ∈ [T̂_j−1 : T̂_j] be the estimated coefficient vector for the block = [T̂_j−1 : T̂_j]. Using the estimated vector θ̂^a,j, we define the neighborhood estimate of the node a for the block as

$${\widehat{S}}_a^j:=S({\widehat{\mathit{\theta }}}^{a,j}):=\{b\in \backslash a:{\widehat{\theta }}_b^{a,j}\ne 0\}.$$

Solving (2.2) for each node a ∈ V gives us a neighborhood estimate for each node. Combining the neighborhood estimates we can obtain an estimate of the graph structure for each point i ∈ [n].

The choice of the penalty term is motivated by the work on penalization using total variation [34, 27], which results in a piece-wise constant approximation of an unknown regression function. The fusion-penalty has also been applied in the context of multivariate linear regression [36], where the coefficients that are spatially close, are also biased to have similar values. As a result, nearby coefficients are fused to the same estimated value. Instead of penalizing the ℓ₁ norm on the difference between coefficients, we use the ℓ₂ norm in order to enforce that all the changes occur at the same point.

The objective (2.2) estimates the neighborhood of one node in a graph for all time-points. After solving the objective (2.2) for all nodes a ∈ V, we need to combine them to obtain the graph structure. We will use the following procedure to combine {β̂^a}_a∈V,

$${\widehat{E}}_i=\{(a,b):max(\mid {\beta }_{b,i}^a\mid ,\mid {\beta }_{a,i}^b\mid )>0\},i\in [n].$$

That is, an edge between nodes a and b is included in the graph if at least one of the nodes a or b is included in the neighborhood of the other node. We use the max operator to combine different neighborhoods as we believe that for the purpose of network exploration it is more important to occasionally include spurious edges than to omit relevant ones. For further discussion on the differences between the min and the max combination, we refer an interested reader to [4].

2.1. Numerical procedure

Finding a minimizer β̂^a of (2.2) can be a computationally challenging task for an off-the-shelf convex optimization procedure. We propose to use an accelerated gradient method with a smoothing technique [29], which converges in (1/ε) iterations where ε is the desired accuracy.

We start by defining a smooth approximation of the fused penalty term. Let H ∈ ℝ^n×n−1 be a matrix with elements

$$H_{ij}=\{\begin{array}{ll}\hfill -1\hfill & \text{if}i=j\hfill \\ \hfill 1\hfill & \text{if}i=j+1\hfill \\ \hfill 0\hfill & \text{otherwise}.\hfill \end{array}$$

With the matrix H we can rewrite the fused penalty term as $2{\lambda }_1{{\sum }_{i=1}^{n-1}\Vert {(\mathit{\beta }\mathbf{H})}_{·,i}\Vert }_2$ and using the fact that the ℓ₂ norm is self dual (e.g., see [7]) we have the following representation

$$2{\lambda }_1\sum _{i=2}^n{\Vert {\mathit{\beta }}_{·,i}-{\mathit{\beta }}_{·,i-1}\Vert }_2=\underset{\mathbf{U}\in \mathcal{Q}}{max}\langle \langle \mathbf{U},2{\lambda }_1\mathit{\beta }\mathbf{H}\rangle \rangle$$ (2.5)

where := {U ∈ ℝ^p−1×n−1 : ||U_·,i||₂ ≤ 1, ∀i ∈ [n − 1]}. The following function is defined as a smooth approximation to the fused penalty,

$${\mathrm{\Psi }}_{\mu }(\mathit{\beta }):=\underset{\mathbf{U}\in \mathcal{Q}}{max}\langle \langle \mathbf{U},2{\lambda }_1\mathit{\beta }\mathbf{H}\rangle \rangle -\mu {\Vert \mathbf{U}\Vert }_F^2$$ (2.6)

where μ > 0 is the smoothness parameter. It is easy to see that

$${\mathrm{\Psi }}_{\mu }(\mathit{\beta })\le {\mathrm{\Psi }}_0(\mathit{\beta })\le {\mathrm{\Psi }}_{\mu }(\mathit{\beta })+\mu (n-1).$$

Setting the smoothness parameter to $\mu =\frac{\epsilon }{2(n-1)}$, the correct rate of convergence is ensured. Let U_μ(β) be the optimal solution of the maximization problem in (2.6), which can be obtained analytically as

$${\mathbf{U}}_{\mu }(\mathit{\beta })={\mathrm{\Pi }}_{\mathcal{Q}}\left(\frac{\lambda \mathit{\beta }\mathbf{H}}{\mu }\right)$$ (2.7)

where Π(·) is the projection operator onto the set . From Theorem 1 in [29], we have that Ψ_μ(β) is continuously differentiable and convex, with the gradient

$$\nabla {\mathrm{\Psi }}_{\mu }(\mathit{\beta })=2{\lambda }_1{\mathbf{U}}_{\mu }(\mathit{\beta }){\mathbf{H}}^{\prime }$$ (2.8)

that is Lipschitz continuous.

With the above defined smooth approximation, we focus on minimizing the following objective

$$\underset{\mathit{\beta }\in {ℝ}^{p-1\times n}}{min}F(\mathit{\beta }):\underset{\mathit{\beta }\in {ℝ}^{p-1\times n}}{min}\mathcal{L}(\mathit{\beta })+{\mathrm{\Psi }}_{\mu }(\mathit{\beta })+2{\lambda }_2{\Vert \mathit{\beta }\Vert }_1.$$

Following [5] (see also [30]), we define the following quadratic approximation of F(β) at a point β₀

$$Q_L(\mathit{\beta },{\mathit{\beta }}_0):=\mathcal{L}({\mathit{\beta }}_0)+{\mathrm{\Psi }}_{\mu }({\mathit{\beta }}_0)+\langle \langle \mathit{\beta }-{\mathit{\beta }}_0,\nabla \mathcal{L}({\mathit{\beta }}_0)+\nabla \mathrm{\Psi }({\mathit{\beta }}_0)\rangle \rangle +\frac{L}{2}{\Vert \mathit{\beta }-{\mathit{\beta }}_0\Vert }_F^2+2{\lambda }_2{\Vert \mathit{\beta }\Vert }_1$$ (2.9)

where L > 0 is the parameter chosen as an upper bounds for the Lipschitz constant of ∇ +∇Ψ. Let p_L(β₀) be a minimizer of Q_L(β, β₀). Ignoring constant terms, p_L(β₀) can be obtained as

$$p_L({\mathit{\beta }}_0)=\underset{\mathit{\beta }\in {ℝ}^{p-1\times n}}{\text{argmin}}\frac{1}{2}{\Vert \mathit{\beta }-\left({\mathit{\beta }}_0-\frac{1}{L}(\nabla \mathcal{L}+\nabla \mathrm{\Psi })({\mathit{\beta }}_0)\right)\Vert }_F^2+\frac{2{\lambda }_2}{L}{\Vert \mathit{\beta }\Vert }_1.$$

It is clear that p_L(β₀) is the unique minimizer, which can be obtained in a closed form, as a result of the soft-thresholding,

$$p_L({\mathit{\beta }}_0)=T\left({\mathit{\beta }}_0-\frac{1}{L}(\nabla \mathcal{L}+\nabla \mathrm{\Psi })({\mathit{\beta }}_0),\frac{2{\lambda }_2}{L}\right)$$ (2.10)

where T(x, λ) = sign(x) max(0, |x| − λ) is the soft-thresholding operator that is applied element-wise.

In practice, an upper bound on the Lipschitz constant of ∇ + ∇Ψ can be expensive to compute, so the parameter L is going to be determined iteratively. Combining all of the above, we arrive at Algorithm 1. In the algorithm, β₀ is set to zero or, if the optimization problem is solved for a sequence of tuning parameters, it can be set to the solution β̂ obtained for the previous set of tuning parameters. The parameter γ is a constant used to increase the estimate of the Lipschitz constant L and we set it to γ = 1.5 in our experiments, while L = 1 initially. Compared to the gradient descent method (which can be obtain by iterating β_k+1 = p_L(β_k)), the accelerated gradient method updates two sequences {β_k} and {z_k} recursively. Instead of performing the gradient step from the latest approximate solution β_k, the gradient step is performed from the search point z_k that is obtained as a linear combination of the last two approximate solutions β_k−1 and β_k. Since the condition F(p_L(z_k)) ≤ Q_L(p_L(z_k), z_k) is satisfied in every iteration, we have the algorithm converges in (1/ε) iterations following [5]. As the convergence criterion, we stop iterating once the relative change in the objective value is below some threshold value (e.g., we use 10⁻⁴).

Algorithm 1.

Accelerated Gradient Method for Equation (2.2)

2.2. Tuning parameter selection

The penalty parameters λ₁ and λ₂ control the complexity of the estimated model. In this work, we propose to use the BIC score to select the tuning parameters. Define the BIC score for each node a ∈ V as

$${\text{BIC}}_a({\lambda }_1,{\lambda }_2):=log\frac{\mathcal{L}({\widehat{\mathit{\beta }}}^a)}{n}+\frac{logn}{n}\sum _{j\in [\widehat{B}]}\mid S({\widehat{\mathit{\theta }}}^{a,j})\mid$$ (2.11)

where (·) is defined in (2.3) and β̂^a = β̂^a(λ₁, λ₂) is a solution of (2.2). The penalty parameters can now be chosen as

$$\{{\widehat{\lambda }}_1,{\widehat{\lambda }}_2\}=\underset{{\lambda }_1,{\lambda }_2}{\text{argmin}}\sum _{a\in V}{\text{BIC}}_a({\lambda }_1,{\lambda }_2).$$ (2.12)

We will use the above formula to select the tuning parameters in our simulations, where we are going to search for the best choice of parameters over a grid.

3. Theoretical results

This section is going to address the statistical properties of the estimation procedure presented in Section 2. The properties are addressed in an asymptotic framework by letting the sample size n grow, while keeping the other parameters fixed. For the asymptotic framework to make sense, we assume that there exists a fixed unknown sequence of numbers {τ_j} that defines the partition boundaries as T_j = ⌊nτ_j⌋, where ⌊a⌋ denotes the largest integer smaller that a. This assures that as the number of samples grow, the same fraction of samples falls into every partition. We call {τ_j} the boundary fractions.

We give sufficient conditions under which the sequence {τ_j} is consistently estimated. In particular, if the number of partition blocks is estimated correctly, then we show that max_j∈[B] |T̂_j − T_j| ≤ nδ_n with probability tending to 1, where {δ_n}_n is a non-increasing sequence of positive numbers that tends to zero. If the number of partition segments is over estimated, then we show that for a distance defined for two sets A and B as

$$h(A,B):=\underset{b\in B}{sup}\underset{a\in A}{inf}\mid a-b\mid ,$$ (3.1)

we have h( , ) ≤ nδ_n with probability tending to 1. With the boundary segments consistently estimated, we further show that under suitable conditions for each node a ∈ V the correct neighborhood is selected on all estimated block partitions that are sufficiently large.

The proof technique employed in this section is quite involved, so we briefly describe the steps used. Our analysis is based on careful inspection of the optimality conditions that a solution β̂^a of the optimization problem (2.2) need to satisfy. The optimality conditions for β̂^a to be a solution of (2.2) are given in §3.2. Using the optimality conditions, we establish the rate of convergence for the partition boundaries. This is done by proof by contradiction. Suppose that there is a solution with the partition boundary that satisfies h( , ) ≥ nδ_n. Then we show that, with high-probability, all such solutions will not satisfy the KKT conditions and therefore cannot be optimal. This shows that all the solutions to the optimization problem (2.2) result in partition boundaries that are “close” to the true partition boundaries, with high-probability. Once it is established that and satisfy h( , ) ≤ nδ_n, we can further show that the neighborhood estimates are consistently estimated, under the assumption that the estimated blocks of the partition have enough samples. This part of the analysis follows the commonly used strategy to prove that the Lasso is sparsistent (e.g., see [9, 38, 28]), however important modifications are required due to the fact that position of the partition boundaries are being estimated.

Our analysis is going to focus on one node a ∈ V and its neighborhood. However, using the union bound over all nodes in V, we will be able to carry over conclusions to the whole graph. To simplify our notation, when it is clear from the context, we will omit the superscript a and write β̂, θ̂ and S, etc., to denote β̂^a, θ̂^a and S_a, etc.

3.1. Assumptions

Before presenting our theoretical results, we give some definitions and assumptions that are going to be used in this section. Let Δ_min := min_j∈[B] |T_j − T_j−1| denote the minimum length between change points, ξ_min := min_a∈V min_j∈[B−1] × ||θ^a,j+1 − θ^a,j||₂ denote the minimum jump size and ${\theta }_{min}={min}_{a\in V}{min}_{j\in [B]}\times {min}_{b\in S^j}\mid {\theta }_b^{a,j}\mid$ the minimum coefficient size. Throughout the section, we assume that the following holds.

A1 There exist two constants φ_min > 0 and φ_max < ∞ such that

$${\phi }_{min}=min\{{\mathrm{\Lambda }}_{min}({\mathbf{\sum }}^j):j\in [B],a\in V\}$$

and

$${\phi }_{max}=max\{{\mathrm{\Lambda }}_{max}({\mathbf{\sum }}^j):j\in [B],a\in V\}.$$

A2 Variables are scaled so that ${\sigma }_{aa}^j=1$ for all j ∈ [B] and all a ∈ V.

The assumption A1 is commonly used to ensure that the model is identifiable. If the population covariance matrix is ill-conditioned, the question of the correct model identification if not well defined, as a neighborhood of a node may not be uniquely defined. The assumption A2 is assumed for the simplicity of the presentation. The common variance can be obtained through scaling.

A3 There exists a constant M > 0 such that

$$\underset{a\in V}{max}\underset{j,k\in [B]}{max}{\Vert {\mathit{\theta }}^{a,k}-{\mathit{\theta }}^{a,j}\Vert }_2\le M.$$

The assumption A3 states that the difference between coefficients on two different blocks, ||θ^a,k − θ^a,j||₂, is bounded for all j, k ∈ [B]. This assumption is simply satisfied if the coefficients θ^a were bounded in the ℓ₂ norm.

A4 There exist a constant α ∈ (0, 1], such that the following holds

$$\underset{j\in [B]}{max}\mid \mid \mid {\mathbf{\sum }}_{N_a^j{S}_a^j}{({\mathbf{\sum }}_{S_a^j{S}_a^j})}^{-1}\Vert {\mid }_{\infty }\le 1-\alpha ,\forall a\in V.$$

The assumption A4 states that the variables in the neighborhood of the node a, $S_a^j$, are not too correlated with the variables in the set $N_a^j$. This assumption is necessary and sufficient for correct identification of the relevant variables in the Lasso regression problems (e.g., see [43, 37]). Note that this condition is sufficient also in our case when the correct partition boundaries are not known.

A5 The minimum coefficient size θ_min satisfies ${\theta }_{min}=\mathrm{\Omega }(\sqrt{log(n)/n})$.

The lower bound on the minimum coefficient size θ_min is necessary, since if a partial correlation coefficient is too close to zero the edge in the graph would not be detectable.

A6 The sequence of partition boundaries {T_j} satisfy T_j = ⌊nτ_j⌋, where {τ_j} is a fixed, unknown sequence of the boundary fractions belonging to [0, 1].

The assumption is needed for the asymptotic setting. As n → ∞, there will be enough sample points in each of the blocks to estimate the neighborhood of nodes correctly.

3.2. Convergence of the partition boundaries

In this subsection we establish the rate of convergence of the boundary partitions for the estimator (2.2). We start by giving a lemma that characterizes solutions of the optimization problem given in (2.2). Note that the optimization problem in (2.2) is convex, however, there may be multiple solutions to it, since it is not strictly convex.

Lemma 1

Let $x_{i,a}={\mathbf{x}}_{i,\backslash a}^{\prime }{\mathit{\theta }}_a+{\epsilon }_i$. A matrix β̂ is optimal for the optimization problem (2.2) if and only if there exist a collection of subgradient vectors {ẑ_i}_i∈[2:n] and {ŷ_i}_i∈[n], with ẑ_i ∈ ∂||β̂_·,i − β̂_·,i−1||₂ and ŷ_i ∈ ∂||β̂_·,i||₁, that satisfies

$$\sum _{i=k}^n{\mathbf{x}}_{i,\backslash a}\langle {\mathbf{x}}_{i,\backslash a},{\widehat{\mathit{\beta }}}_{·,i}-{\mathit{\beta }}_{·,i}\rangle -\sum _{i=k}^n{\mathbf{x}}_{i,\backslash a}{\epsilon }_i+{\lambda }_1{\widehat{\mathbf{z}}}_k+{\lambda }_2\sum _{i=k}^n{\widehat{\mathbf{y}}}_i=0$$ (3.2)

for all k ∈ [n] and ẑ₁ = ẑ_n+1 = 0.

The following theorem provides the convergence rate of the estimated boundaries of , under the assumption that the correct number of blocks is known.

Theorem 2

Let {x_i}_i∈[n] be a sequence of observation according to the model in (1.1). Assume that A1–A3 and A5–A6 hold. Suppose that the penalty parameters λ₁ and λ₂ satisfy

$${\lambda }_1\asymp {\lambda }_2=\mathcal{O}(\sqrt{log(n)/n}).$$ (3.3)

Let {β̂_·,i}_i∈[n] be any solution of (2.2) and let be the associated estimate of the block partition. Let {δ_n}_n≥1 be a non-increasing positive sequence that converges to zero as n → ∞ and satisfies Δ_min ≥ nδ_n for all n ≥ 1. Furthermore, suppose that (nδ_nξ_min)⁻¹λ₁ → 0, ${\xi }_{min}^{-1}\sqrt{p}{\lambda }_2\to 0$ and ${({\xi }_{min}\sqrt{n{\delta }_n})}^{-1}\sqrt{\mathit{p}logn}\to 0$, then if | | = B + 1 the following holds

$$\mathbb{P}[\underset{j\in [B]}{max}\mid T_j-{\widehat{T}}_j\mid \le n{\delta }_n]\stackrel{n\to \infty }{\to }1.$$

The proof builds on techniques developed in [17] and is presented in §7.

Suppose that δ_n = (log n)^γ/n for some γ > 1 and ${\xi }_{min}=\mathrm{\Omega }(\sqrt{logn/{(logn)}^{\gamma }})$, the conditions of Theorem 2 are satisfied, and we have that the sequence of boundary fractions {τ_j} is consistently estimated. Since the boundary fractions are consistently estimated, we will see below that the estimated neighborhood S(θ̂^j) on the block consistently recovers the true neighborhood S^j.

Unfortunately, the correct bound on the number of block B may not be known. However, a conservative upper bound B_max on the number of blocks B may be known. Suppose that the sequence of observation is over segmented, with the number of estimated blocks bounded by B_max. Then the following proposition gives an upper bound on h( , ) where h(·, ·) is defined in (3.1).

Proposition 3

Let {x_i}_i∈[n] be a sequence of observation according to the model in (1.1). Assume that the conditions of Theorem 2 are satisfied. Let β̂ be a solution of (2.2) and the corresponding set of partition boundaries, with B̂ blocks. If the number of blocks satisfy B ≤ B̂ ≤ B_max, then

$$\mathbb{P}[h(\widehat{\mathcal{T}},\mathcal{T})\le n{\delta }_n]\stackrel{n\to \infty }{\to }1.$$

The proof of the proposition follows the same ideas of Theorem 2 and its sketch is given in the appendix.

The above proposition assures us that even if the number of blocks is overestimated, there will be a partition boundary close to every true unknown partition boundary. In many cases it is reasonable to assume that a practitioner would have an idea about the number of blocks that she wishes to discover. In that way, our procedure can be used to explore and visualize the data. It is still an open question to pick the tuning parameters in a data dependent way so that the number of blocks are estimated consistently.

3.3. Correct neighborhood selection

In this section, we give a result on the consistency of the neighborhood estimation. We will show that whenever the estimated block is large enough, say | | ≥ r_n where {r_n}_n≥1 is an increasing sequence of numbers that satisfy (r_nλ₂)⁻¹λ₁ → 0 and $r_n{\lambda }_2^2\to \infty$ as n → ∞, we have that S(θ̂^j) = S(β^k), where β^k is the true parameter on the true block that overlaps the most. Figure 1 illustrates this idea. The blue region in the figure denotes the overlap between the true block and the estimated block of the partition. The orange region corresponds to the overlap of the estimated block with a different true block. If the blue region is considerably larger than the orange region, the bias coming from the sample from the orange region will not be strong enough to disable us from selecting the correct neighborhood. On the other hand, since the orange region is small, as seen from Theorem 2, there is little hope of estimating the neighborhood correctly on that portion of the sample.

Fig 1.

The figure illustrates where we expect to estimate a neighborhood of a node consistently. The blue region corresponds to the overlap between the true block (bounded by gray lines) and the estimated block (bounded by black lines). If the blue region is much larger than the orange regions, the additional bias introduced from the samples from the orange region will not considerably affect the estimation of the neighborhood of a node on the blue region. However, we cannot hope to consistently estimate the neighborhood of a node on the orange region.

Suppose that we know that there is a solution to the optimization problem (2.2) with the partition boundary . Then that solution is also a minimizer of the following objective

$$\underset{{\mathit{\theta }}^1,\dots ,{\mathit{\theta }}^{\widehat{B}}}{min}\sum _{j\in \widehat{B}}{\Vert {\mathbf{X}}_a^{{\widehat{\mathcal{B}}}^j}-{\mathbf{X}}_{\backslash a}^{{\widehat{\mathcal{B}}}^j}{\mathit{\theta }}^j\Vert }_2^2+2{\lambda }_1\sum _{j=2}^{\widehat{B}}{\Vert {\mathit{\theta }}^j-{\mathit{\theta }}^{j-1}\Vert }_2+2{\lambda }_2\sum _{j=1}^{\widehat{B}}\mid {\widehat{\mathcal{B}}}^j{\mid \Vert {\mathit{\theta }}^j\Vert }_1.$$ (3.4)

Note that the problem (3.4) does not give a practical way of solving (2.2), but will help us to reason about the solutions of (2.2). In particular, while there may be multiple solutions to the problem (2.2), under some conditions, we can characterize the sparsity pattern of any solution that has specified partition boundaries .

Lemma 4

Let β̂ be a solution to (2.2), with being an associated estimate of the partition boundaries. Suppose that the subgradient vectors satisfy |ŷ_i,b| < 1 for all b ∉ S(β̂_·,i), then any other solution β̂ with the partition boundaries satisfy β̂_b,i = 0 for all b ∉ S(β̂_·,i).

The above Lemma states sufficient conditions under which the sparsity pattern of a solution with the partition boundary is unique. Note, however, that there may other solutions to (2.2) that have different partition boundaries.

Now, we are ready to state the following theorem, which establishes that the correct neighborhood is selected on every sufficiently large estimated block of the partition.

Theorem 5

Let {x_i}_i∈[n] be a sequence of observation according to the model in (1.1). Assume that the conditions of theorem 2 are satisfied. In addition, suppose that A4 also holds. Then, if | | = B + 1, it holds that

$$\mathbb{P}[S^k=S({\widehat{\mathit{\theta }}}^k)]\stackrel{n\to \infty }{\to }1,\forall k\in [B].$$

Under the assumptions of theorem 2 each estimated block is of size (n). As a result, there are enough samples in each block to consistently estimate the underlying neighborhood structure. Observe that the neighborhood is consistently estimated at each i ∈ ∩ for all j ∈ [B] and the error is made only on the small fraction of samples, when i ∉ ∩ , which is of order (nδ_n).

Using proposition 3 in place of theorem 2, it can be similarly shown that, for a large fraction of samples, the neighborhood is consistently estimated even in the case of over-segmentation. In particular, whenever there is a sufficiently large estimated block, with | ∩ | = (r_n), it holds that S( ) = S^j with probability tending to one.

4. Alternative estimation procedures

In this section, we discuss some alternative estimation methods to the neighborhood selection detailed in §2. We start describing how to solve the objective (2.2) for different penalties than the one given in (2.4). In particular, we describe how to minimize the objective when the ℓ₂ is replaced with the ℓ_q (q ∈ {1, ∞}) norm in (2.4). Next, we describe how to solve the penalized maximum likelihood objective with the temporal difference penalty. We do not provide statistical guarantees for solutions of these objective functions.

4.1. Neighborhood selection with modified penalty

We consider the optimization problem given in (2.2) with the following penalty

$${\text{pen}}_{{\lambda }_1,{\lambda }_2}(\mathit{\beta }):=2{\lambda }_1\sum _{i=2}^n{\Vert {\mathit{\beta }}_{·,i}-{\mathit{\beta }}_{·,i-1}\Vert }_q+2{\lambda }_2\sum _{i=1}^n\sum _{b\in \backslash a}\mid {\beta }_{b,i}\mid ,q\in \{1,\infty \}.$$ (4.1)

We call the penalty in (4.1) the TD_q penalty. As in §2.1, we apply the smoothing procedure to the first term in (4.1). Using the dual norm representation, we have

$$2{\lambda }_1\sum _{i=2}^n{\Vert {\mathit{\beta }}_{·,i}-{\mathit{\beta }}_{·,i-1}\Vert }_q=\underset{\mathbf{U}\in {\mathcal{Q}}^q}{max}\langle \langle \mathbf{U},2{\lambda }_1\mathit{\beta }\mathbf{H}\rangle \rangle$$

where

$${\mathcal{Q}}^1:=\{\mathbf{U}\in {ℝ}^{p-1\times n-1}:{\Vert {\mathbf{U}}_{·,i}\Vert }_{\infty }\le 1,\forall i\in [n-1]\}$$

and

$${\mathcal{Q}}^{\infty }:=\{\mathbf{U}\in {ℝ}^{p-1\times n-1}:{\Vert {\mathbf{U}}_{·,i}\Vert }_1\le 1,\forall i\in [n-1]\}.$$

Next, we define smooth approximation to the norm as

$${\mathrm{\Psi }}_{\mu }^q(\mathit{\beta }):=\underset{\mathbf{U}\in {\mathcal{Q}}^q}{max}\langle \langle \mathbf{U},2{\lambda }_1\mathit{\beta }\mathbf{H}\rangle \rangle -\mu {\Vert \mathbf{U}\Vert }_F^2$$ (4.2)

where μ > 0 is the smoothness parameter. Let

$${\mathbf{U}}_{\mu }^q(\mathit{\beta })={\mathrm{\Pi }}_{{\mathcal{Q}}^q}\left(\frac{\lambda \mathit{\beta }\mathbf{H}}{\mu }\right)$$ (4.3)

be the optimal solution of the maximization problem in (4.2), where Π (·) is the projection operator onto the set . We observe that the projection on the ℓ_∞ unit ball can be easily obtained, while a fast algorithm for projection on the ℓ₁ unit ball can be found in [8]. The gradient can now be obtained as

$$\nabla {\mathrm{\Psi }}_{\mu }^q(\mathit{\beta })=2{\lambda }_1{\mathbf{U}}_{\mu }^q(\mathit{\beta }){\mathbf{H}}^{\prime },$$ (4.4)

and we can proceed as in § 2.1 to arrive at the update (2.10).

We have described how to optimize (2.2) with the TD_q penalty for q ∈ {1, 2, ∞}. Other ℓ_q norms are not commonly used in practice. We also note that a different procedure for q = 1 can be found in [26].

4.2. Penalized maximum likelihood estimation

In §2, we have related the problem of estimating zero elements of a precision matrix to a penalized regression procedure. Now, we consider estimating a sparse precision matrix using a penalized maximum likelihood approach. That is, we consider the following optimization procedure

$$\underset{{\{{\mathbf{\Omega }}_i\succ \mathbf{0}\}}_{i\in [n]}}{min}\sum _{i\in [n]}(\text{tr}{\mathbf{\Omega }}_i{\mathbf{x}}_i{\mathbf{x}}_i^{\prime }-log\mid {\mathbf{\Omega }}_i\mid )+{\text{pen}}_{{\lambda }_1,{\lambda }_2}({\{{\mathbf{\Omega }}_t\}}_{t\in [n]})$$ (4.5)

where

$${\text{pen}}_{{\lambda }_1,{\lambda }_2}({\{{\mathbf{\Omega }}_i\}}_{i\in [n]}):=2{\lambda }_1\sum _{i=2}^n{\Vert {\mathbf{\Omega }}_i-{\mathbf{\Omega }}_{i-1}\Vert }_F+2{\lambda }_2\sum _{i=1}^n{\mid {\mathbf{\Omega }}_i\mid }_1.$$ (4.6)

In order to optimize (4.5) using the smoothing technique described in §2.1, we need to show that the gradient of the log-likelihood is Lipschitz continuous. The following Lemma establishes the desired result.

Lemma 6

The function f(A) = tr SA − log |A| has Lipschitz continuous gradient on the set {A ∈ : Λ_min(A) ≥ γ}, with Lipschitz constant L = γ⁻².

Following [3], we can show that a solution to the optimization problem (4.5), on each estimated block, is indeed positive definite matrix with smallest eigenvalue bounded away from zero. This allows us to use the Nesterov’s smoothing technique to solve (4.5).

Penalized maximum likelihood approach for estimating sparse precision matrix was proposed by [42]. Here, we have modified the penalty to perform estimation under the model (1.1). Although the parameters of the precision matrix can be estimated consistently using the penalized maximum likelihood approach, a number of theoretical results have shown that the neighborhood selection procedure requires lest stringent assumptions in order to estimate the underlying network consistently [28, 32]. We observe this phenomena in our simulation studies as well.

5. Numerical studies

In this section, we present a small numerical study on simulated networks. A full performance test and application on real world data is beyond the scope of this paper which mainly focuses on the theory of time-varying model estimation. In all of our simulations studies we set p = 30 and B = 3 with | | = 80, | | = 130 and | | = 90, so that in total we have n = 300 samples. We consider two types of random networks: a chain and a nearest neighbor network. We measure the performance of the estimation procedure outlined in §2 on the following metrics: average precision of estimated edges, average recall of estimated edges and average F₁ score which combines the precision and recall score. The precision, recall and F₁ score are respectively defined as

$$\begin{array}{l}\mathit{precision}=\frac{1}{n}\sum _{i\in [n]}\frac{{\sum }_{a\in [p]}{\sum }_{b=a+1}^{p}1\mathrm{I}\{(a,b)\in {\widehat{E}}_i\wedge (a,b)\in E_i\}}{{\sum }_{a\in [p]}{\sum }_{b=a+1}^{p}1\mathrm{I}\{(a,b)\in {\widehat{E}}_i\}}\\ \mathit{recall}=\frac{1}{n}\sum _{i\in [n]}\frac{{\sum }_{a\in [p]}{\sum }_{b=a+1}^{p}1\mathrm{I}\{(a,b)\in {\widehat{E}}_i\wedge (a,b)\in E_i\}}{{\sum }_{a\in [p]}{\sum }_{b=a+1}^{p}1\mathrm{I}\{(a,b)\in E_i\}}\\ F_1=\frac{2\ast \mathit{precision}\ast \mathit{recall}}{\mathit{precision}+\mathit{recall}}.\end{array}$$

Furthermore, we report results on estimating the partition boundaries using n⁻¹h( , ), where h( , ) is defined in (3.1). Results are averaged over 50 simulation runs. We compare the TD-Lasso algorithm introduced in §2.1 against an oracle algorithm which exactly knows the true partition boundaries. In this case, it is only needed to run the algorithm of [28] on each block of the partition independently. We use a BIC criterion to select the tuning parameter for this oracle procedure as described in [31]. Furthermore, we report results using neighborhood selection procedures introduced in §4, which are denoted TD₁-Lasso and TD_∞-Lasso, as well as the penalized maximum likelihood procedure, which is denoted as LL_max. We choose the tuning parameters for the penalized maximum likelihood procedure using the BIC procedure.

Chain networks

We follow the simulation in [12] to generate a chain network (see Figure 2). This network corresponds to a tridiagonal precision matrix (after an appropriate permutation of nodes). The network is generated as follows. First, we choose to generate a random permutation π of [n]. Next, the covariance matrix is generated as follows: the element at position (a, b) is chosen as σ_ab = exp(−|t_π(a) − t_π(b)|/2) where t₁ < t₂ < ··· < t_p and t_i − t_i−1 ~ Unif(0.5, 1) for i = 2, …, p. This processes is repeated three times to obtain three different covariance matrices, from which we sample 80, 130 and 90 samples respectively.

Fig 2.

A chain graph

For illustrative purposes, Figure 3 plots the precision, recall and F₁ score computed for different values of the penalty parameters λ₁ and λ₂. Table 2 shows the precision, recall and F₁ score for the parameters chosen using the BIC score described in 2.2, as well as the error in estimating the partition boundaries. The numbers in parentheses correspond to standard deviation. Due to the fact that there is some error in estimating the partition boundaries, we observe a decrease in performance compared to the oracle procedure that knows the correct position of the partition boundaries. Further, we observe that the neighborhood selection procedure estimate the graph structure more accurately than the maximum likelihood procedure. For TD₁-Lasso we do not report n⁻¹h( , ), as the procedure does not estimate the partition boundaries.

Fig 3.

Plots of the precision, recall and F₁ scores as functions of the penalty parameters λ₁ and λ₂ for chain networks estimated using the TD-Lasso. The parameter λ₁ is obtained as 100 * 0.98⁵⁰⁺ⁱ, where i indexes y-axis. The parameter λ₂ is computed as 285 * 0.98^230+j, where j indexes x-axis. Black dot represents the selected tuning parameters. The white region of each plot corresponds to a region of the parameter space that we did not explore.

Table 2.

Performance of different procedures when estimating chain networks

Method name	Precision	Recall	F1 score	n−1h( , )
TD-Lasso	0.84 (0.04)	0.80 (0.04)	0.82 (0.04)	0.03 (0.01)
TD1-Lasso	0.78 (0.05)	0.70 (0.03)	0.74 (0.04)	N/A
TD∞-Lasso	0.83 (0.03)	0.80 (0.03)	0.81 (0.03)	0.03 (0.01)
LLmax	0.72 (0.03)	0.65 (0.03)	0.68 (0.04)	0.06 (0.02)
Oracle procedure	0.97 (0.02)	0.89 (0.02)	0.93 (0.02)	0 (0)

Nearest neighbors networks

We generate nearest neighbor networks following the procedure outlined in [23]. For each node, we draw a point uniformly at random on a unit square and compute the pairwise distances between nodes. Each node is then connected to 4 closest neighbors (see Figure 4). Since some of nodes will have more than 4 adjacent edges, we remove randomly edges from nodes that have degree larger than 4 until the maximum degree of a node in a network is 4. Each edge (a, b) in this network corresponds to a nonzero element in the precision matrix Ω, whose value is generated uniformly on [−1, −0.5] ∪ [0.5, 1]. The diagonal elements of the precision matrix are set to a smallest positive number that makes the matrix positive definite. Next, we scale the corresponding covariance matrix Σ = Ω⁻¹ to have diagonal elements equal to 1. This processes is repeated three times to obtain three different covariance matrices, from which we sample 80, 130 and 90 samples respectively.

Fig 4.

An instance of a random neighborhood graph with 30 nodes.

For illustrative purposes, Figure 5 plots the precision, recall and F₁ score computed for different values of the penalty parameters λ₁ and λ₂. Table 3 shows the precision, recall, F₁ score and n⁻¹h( , ) for the parameters chosen using the BIC score, together with their standard deviations. The results obtained for nearest neighbor networks are qualitatively similar to the results obtain for chain networks.

Fig 5.

Plots of the precision, recall and F₁ scores as functions of the penalty parameters λ₁ and λ₂ for nearest neighbor networks estimated using the TD-Lasso. The parameter λ₁ is obtained as 100 * 0.98⁵⁰⁺ⁱ, where i indexes y-axis. The parameter λ₂ is computed as 285 * 0.98^230+j, where j indexes x-axis. Black dot represents the selected tuning parameters. The white region of each plot corresponds to a region of the parameter space that we did not explore.

Table 3.

Performance of different procedure when estimating random nearest neighbor networks

Method name	Precision	Recall	F1 score	n−1h( , )
TD-Lasso	0.79 (0.06)	0.76 (0.05)	0.77 (0.05)	0.04 (0.02)
TD1-Lasso	0.70 (0.05)	0.68 (0.07)	0.69 (0.06)	N/A
TD∞-Lasso	0.80 (0.06)	0.75 (0.06)	0.77 (0.06)	0.04 (0.02)
LLmax	0.62 (0.08)	0.60 (0.06)	0.61 (0.06)	0.06 (0.02)
Oracle procedure	0.87 (0.05)	0.82 (0.05)	0.84 (0.04)	0 (0)

6. Conclusion

We have addressed the problem of time-varying covariance selection when the underlying probability distribution changes abruptly at some unknown points in time. Using a penalized neighborhood selection approach with the fused-type penalty, we are able to consistently estimate times when the distribution changes and the network structure underlying the sample. The proof technique used to establish the convergence of the boundary fractions using the fused-type penalty is novel and constitutes an important contribution of the paper. Furthermore, our procedure estimates the network structure consistently whenever there is a large overlap between the estimated blocks and the unknown true blocks of samples coming from the same distribution. The proof technique used to establish the consistency of the network structure builds on the proof for consistency of the neighborhood selection procedure, however, important modifications are necessary since the times of distribution changes are not known in advance. Applications of the proposed approach range from cognitive neuroscience, where the problem is to identify changing associations between different parts of a brain when presented with different stimuli, to system biology studies, where the task is to identify changing patterns of interactions between genes involved in different cellular processes. We conjecture that our estimation procedure is also valid in the high-dimensional setting when the number of variables p is much larger than the sample size n. We leave the investigations of the rate of convergence in the high-dimensional setting for a future work.

7. Proofs

7.1. Proof of Lemma 1

For each i ∈ [n], introduce a (p − 1)-dimensional vector γ_i defined as

$${\mathit{\gamma }}_i=\{\begin{array}{ll}{\mathit{\beta }}_{·,i}\hfill & \text{for}i=1\hfill \\ {\mathit{\beta }}_{·,i}-{\mathit{\beta }}_{·,i-1}\hfill & \text{otherwise}\hfill \end{array}$$

and rewrite the objective (2.2) as

$${\{{\widehat{\mathit{\gamma }}}^i\}}_{i\in [n]}=\underset{\mathit{\gamma }\in {ℝ}^{n\times p-1}}{\text{argmin}}\sum _{i=1}^n{\left(x_{i,a}-\sum _{b\in \backslash a}{x}_{i,b}\sum _{j\le i}{\gamma }_{j,b}\right)}^2+2{\lambda }_1\sum _{i=2}^n{\Vert {\mathit{\gamma }}_i\Vert }_2+2{\lambda }_2\sum _{i=1}^n\sum _{b\in \backslash a}\left|\sum _{j\le i}{\gamma }_{j,b}\right|.$$ (7.1)

A necessary and sufficient condition for {γ̂_i}_i∈[n] to be a solution of (7.1), is that for each k ∈ [n] the (p − 1)-dimensional zero vector, 0, belongs to the subdifferential of (7.1) with respect to γ_k evaluated at {γ̂_i}_i∈[n], that is,

$$\mathbf{0}=2\sum _{i=k}^n(-{\mathbf{x}}_{i,\backslash a})\left(x_{i,a}-\sum _{b\in \backslash a}{x}_{i,b}{\widehat{\beta }}_{b,i}^a\right)+2{\lambda }_1{\widehat{\mathbf{z}}}_k+2{\lambda }_2\sum _{i=k}^n{\widehat{\mathbf{y}}}_i,$$ (7.2)

where ẑ_k ∈ ∂||·||₂(γ̂_k), that is,

$${\stackrel{\sim }{\mathbf{z}}}_k=\{\begin{array}{ll}\frac{{\stackrel{\sim }{\mathit{\gamma }}}_k}{{\Vert {\stackrel{\sim }{\mathit{\gamma }}}_k\Vert }_2}\hfill & \text{if}{\stackrel{\sim }{\mathit{\gamma }}}_k\ne 0\hfill \\ \in {\mathcal{B}}_2(0,1)\hfill & \text{otherwise}\hfill \end{array}$$

and for k ≤ i, ŷ_i ∈ ∂|Σ_j≤i γ̂_j|, that is, y_i = sign(Σ_j≤i γ̂_j) with sign(0) ∈ [−1, 1]. The Lemma now simply follows from (7.2).

7.2. Proof of Theorem 2

We build on the ideas presented in the proof of Proposition 5 in [17]. Using the union bound,

$$\mathbb{P}[\underset{j\in [B]}{max}\mid T_j-{\widehat{T}}_j\mid >n{\delta }_n]\le \sum _{j\in [B]}\mathbb{P}[\mid T_j-{\widehat{T}}_j\mid >n{\delta }_n]$$

and it is enough to show that ℙ[|T_j − T̃_j| > nδ_n] → 0 for all j ∈ [B]. Define the event A_n,j as

$$A_{n,j}:=\{\mid T_j-{\widehat{T}}_j\mid >n{\delta }_n\}$$

and the event C_n as

$$C_n:=\left\{\underset{j\in [B]}{max}\mid {\widehat{T}}_j-T_j\mid <\frac{{\mathrm{\Delta }}_{min}}{2}\right\}.$$

We show that ℙ[A_n,j] → 0 by showing that both ℙ[A_n,j ∩ C_n] → 0 and $\mathbb{P}[A_{n,j}\cap C_n^c]\to 0$ as n → ∞. The idea here is that, in some sense, the event C_n is a good event on which the estimated boundary partitions and the true boundary partitions are not too far from each other. Considering the two cases will make the analysis simpler.

First, we show that ℙ[A_n,j ∩ C_n] → 0. Without loss of generality, we assume that T̂_j < T_j, since the other case follows using the same reasoning. Using (3.2) twice with k = T̂_j and with k = T_j and then applying the triangle inequality we have

$$2{\lambda }_1\ge {\Vert \sum _{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}\langle {\mathbf{x}}_{i,\backslash a},{\widehat{\mathit{\beta }}}_{·,i}-{\mathit{\beta }}_{·,i}\rangle -\sum _{i={\widehat{T}}_j}^{{\widehat{T}}_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i+{\lambda }_2\sum _{i={\widehat{T}}_j}^{T_j-1}{\widehat{\mathbf{y}}}_i\Vert }_2.$$ (7.3)

Some algebra on the above display gives

$$\begin{array}{l}2{\lambda }_1+(T_j-{\widehat{T}}_j)\sqrt{p}{\lambda }_2\ge {\Vert \sum _{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}\langle {\mathbf{x}}_{i,\backslash a},{\mathit{\theta }}^j-{\mathit{\theta }}^{j+1}\rangle \Vert }_2-{\Vert \sum _{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}\langle {\mathbf{x}}_{i,\backslash a},{\mathit{\theta }}^{j+1}-{\widehat{\mathit{\theta }}}^{j+1}\rangle \Vert }_2-{\Vert \sum _{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2\\ =:{\Vert R_1\Vert }_2-{\Vert R_2\Vert }_2-{\Vert R_3\Vert }_2.\end{array}$$

The above display occurs with probability one, so that the event $\{2{\lambda }_1+(T_j-{\widehat{T}}_j)\sqrt{p}{\lambda }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}\cup \{{\Vert R_2\Vert }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}\cup \{{\Vert R_3\Vert }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}$ also occurs with probability one, which gives us the following bound

$$\begin{array}{l}\mathbb{P}[A_{n,j}\cap C_n]\le \mathbb{P}[A_{n,j}\cap C_n\cap \{2{\lambda }_1+(T_j-{\widehat{T}}_j)\sqrt{p}{\lambda }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}]\\ +\mathbb{P}[A_{n,j}\cap C_n\cap \{{\Vert R_2\Vert }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}]\\ +\mathbb{P}[A_{n,j}\cap C_n\cap \{{\Vert R_3\Vert }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2\}]\\ =:\mathbb{P}[A_{n,j,1}]+\mathbb{P}[A_{n,j,2}]+\mathbb{P}[A_{n,j,3}].\end{array}$$

First, we focus on the event A_n,j,1. Using lemma 9, we can upper bound ℙ[A_n,j,1] with

$$\mathbb{P}[2{\lambda }_1+(T_j-{\widehat{T}}_j)\sqrt{p}{\lambda }_2\ge \frac{{\phi }_{min}}{27}(T_j-{\widehat{T}}_j){\xi }_{min}]+2exp(-n{\delta }_n/2+2logn).$$

Since under the assumptions of the theorem (nδ_nξ_min)⁻¹ λ₁ → 0 and ${\xi }_{min}^{-1}\sqrt{p}{\lambda }_2\to 0$ as n → ∞, we have that ℙ[A_n,j,1] → 0 as n → ∞.

Next, we show that the probability of the event A_n,j,2 converges to zero. Let T̄_j:= ⌊2⁻¹(T_j + T_j+1)⌋. Observe that on the event C_n, T̂_j+1 > T̄_j so that β̂_·,i = θ̂^j+1 for all i ∈ [T_j, T̄_j]. Using (3.2) with k = T_j and k = T̄_j we have that

$$2{\lambda }_1+({\overline{T}}_j-T_j)\sqrt{p}{\lambda }_2\ge {\Vert \sum _{i=T_j}^{{\overline{T}}_j-1}{\mathbf{x}}_{i.\backslash a}\langle {\mathbf{x}}_{i,\backslash a},{\mathit{\theta }}^{j+1}-{\widehat{\mathit{\theta }}}^{j+1}\rangle \Vert }_2-{\Vert \sum _{i=T_j}^{{\overline{T}}_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2.$$

Using lemma 9 on the display above we have

$${\Vert {\mathit{\theta }}^{j+1}-{\widehat{\mathit{\theta }}}^{j+1}\Vert }_2\le \frac{36{\lambda }_1+18({\overline{T}}_j-T_j)\sqrt{p}{\lambda }_2+18{\Vert {\sum }_{i=T_j}^{{\overline{T}}_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2}{(T_{j+1}-T_j){\phi }_{min}},$$ (7.4)

which holds with probability at least 1–2 exp(−Δ_min/4+2 log n). We will use the above bound to deal with the event { ${\Vert R_2\Vert }_2\ge \frac{1}{3}{\Vert R_1\Vert }_2$}. Using lemma 9, we have that φ_min(T_j − T̂_j)ξ_min/9 ≤ ||R₁||₂ and ||R₂||₂ ≤ (T_j − T̂_j)9φ_max||θ^j+1 − θ̂^j+1||₂ with probability at least 1 – 4 exp(−nδ_n/2 + 2 log n). Combining with (7.4), the probability ℙ[A_n,j,2] is upper bounded by

$$\begin{array}{l}\mathbb{P}[c_1{\phi }_{min}^2{\phi }_{max}^{-1}{\mathrm{\Delta }}_{min}{\xi }_{min}\le {\lambda }_1]+\mathbb{P}[c_2{\phi }_{min}^2{\phi }_{max}^{-1}{\xi }_{min}\le \sqrt{p}{\lambda }_2]\\ +\mathbb{P}\left[c_3{\phi }_{min}^2{\phi }_{max}^{-1}{\xi }_{min}\le {({\overline{T}}_j-T_j)}^{-1}{\Vert \sum _{i=T_j}^{{\overline{T}}_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2\right]+c_{4}exp(-n{\delta }_n/2+2logn).\end{array}$$

Under the conditions of the theorem, the first term above converges to zero, since Δ_min > nδ_n and (nδ_nξ_min)⁻¹ λ₁ → 0. The second term also converges to zero, since ${\xi }_{min}^{-1}\sqrt{p}{\lambda }_2\to 0$. Using lemma 8, the third term converges to zero with the rate exp(−c₆ log n), since ${({\xi }_{min}\sqrt{{\mathrm{\Delta }}_{min}})}^{-1}\sqrt{\mathit{p}logn}\to 0$. Combining all the bounds, we have that ℙ[A_n,j,2] → 0 as n → ∞.

Finally, we upper bound the probability of the event A_n,j,3. As before, φ_min(T_j−T̂_j)ξ_min/9 ≤ ||R₁||₂ with probability at least 1 – 2 exp(−nδ_n/2 + 2 log n). This gives us an upper bound on ℙ[A_n,j,3] as

$$\mathbb{P}\left[\frac{{\phi }_{min}{\xi }_{min}}{27}\le \frac{{\Vert {\sum }_{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2}{T_j-{\widehat{T}}_j}\right]+2exp(-n{\delta }_n/2+2logn),$$

which, using lemma 8, converges to zero as under the conditions of the theorem ${({\xi }_{min}\sqrt{n{\delta }_n})}^{-1}\sqrt{\mathit{p}logn}\to 0$. Thus we have shown that ℙ[A_n,j,3] → 0. Since the case when T̂_j > T_j is shown similarly, we have proved that ℙ[A_n,j ∩ C_n] → 0 as n → ∞.

We proceed to show that $\mathbb{P}[A_{n,j}\cap C_n^c]\to 0$ as n → ∞. Recall that $C_n^c=\{{max}_{j\in [B]}\mid {\widehat{T}}_j-T_j\mid \ge {\mathrm{\Delta }}_{min}/2\}$. Define the following events

$$\begin{array}{l}{D}_n^{(l)}:=\left\{\exists j\in [B],{\widehat{T}}_j\le T_{j-1}\right\}\cap C_n^c,\\ D_n^{(m)}:=\left\{\forall j\in [B],T_{j-1}<{\widehat{T}}_j<T_{j+1}\right\}\cap C_n^c,\\ D_n^{(r)}:=\left\{\exists j\in [B],{\widehat{T}}_j\ge T_{j+1}\right\}\cap C_n^c\end{array}$$

and write $\mathbb{P}[A_{n,j}\cap C_n^c]=\mathbb{P}[A_{n,j}\cap D_n^{(l)}]+\mathbb{P}[A_{n,j}\cap D_n^{(m)}]+\mathbb{P}[A_{n,j}\cap D_n^{(r)}]$. First, consider the event $A_{n,j}\cap D_n^{(m)}$ under the assumption that T̂_j ≤ T_j. Due to symmetry, the other case will follow in a similar way. Observe that

$$\begin{array}{l}\mathbb{P}[A_{n,j}\cap D_n^{(m)}]\le \mathbb{P}[A_{n,j}\cap \{({\widehat{T}}_{j+1}-T_j)\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap D_n^{(m)}]+\mathbb{P}[\{(T_{j+1}-{\widehat{T}}_{j+1})\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap D_n^{(m)}]\\ \le \mathbb{P}[A_{n,j}\cap \{({\widehat{T}}_{j+1}-T_j)\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap D_n^{(m)}]+\sum _{k=j+1}^{B-1}\mathbb{P}[\{(T_k-{\widehat{T}}_k)\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap \{({\widehat{T}}_{k+1}-T_k)\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap D_n^{(m)}].\end{array}$$ (7.5)

We bound the first term in (7.5) and note that the other terms can be bounded in the same way. The following analysis is performed on the event $A_{n,j}\cap \{({\widehat{T}}_{j+1}-T_j)\ge {\mathrm{\Delta }}_{min}/2\}\cap D_n^{(m)}$. Using (3.2) with k = T̂_j and k = T_j, after some algebra (similar to the derivation of (7.3)) the following holds

$${\Vert {\mathit{\theta }}^j-{\widehat{\mathit{\theta }}}^{j+1}\Vert }_2\le \frac{18{\lambda }_1+9(T_j-{\widehat{T}}_j)\sqrt{p}{\lambda }_2+9\Vert {\sum }_{i={\widehat{T}}_j}^{T_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }{{\phi }_{min}(T_j-{\widehat{T}}_j)},$$

with probability at least 1–2 exp(−nδ_n/2+2 log n), where we have used lemma 9. Let T̄_j = ⌊2⁻¹(T_j + T_j+1)⌋. Using (3.2) with k = T̄_j and k = T_j after some algebra (similar to the derivation of (7.4)) we obtain the following bound

$${\Vert {\mathit{\theta }}^j-{\mathit{\theta }}^{j+1}\Vert }_2\le \frac{18{\lambda }_1+9({\overline{T}}_j-T_j)\sqrt{p}{\lambda }_2+9{\Vert {\sum }_{i=T_j}^{{\overline{T}}_j-1}{\mathbf{x}}_{i,\backslash a}{\epsilon }_i\Vert }_2}{{\phi }_{min}({\overline{T}}_j-T_j)}+81{\phi }_{max}{\phi }_{min}^{-1}{\Vert {\mathit{\theta }}^j-{\widehat{\mathit{\theta }}}^{j+1}\Vert }_2,$$

which holds with probability at least 1 − c₁ exp(−nδ_n/2 + 2 log n), where we have used lemma 9 twice. Combining the last two displays, we can upper bound the first term in (7.5) with

$$\mathbb{P}[{\xi }_{min}n{\delta }_n\le c_1{\lambda }_1]+\mathbb{P}[{\xi }_{min}\le c_2\sqrt{p}{\lambda }_2]+\mathbb{P}[{\xi }_{min}\sqrt{n{\delta }_n}\le c_3\sqrt{\mathit{p}logn}]+c_{4}exp(-c_{5}logn),$$

where we have used lemma 8 to obtain the third term. Under the conditions of the theorem, all terms converge to zero. Reasoning similar about the other terms in (7.5), we can conclude that $\mathbb{P}[A_{n,j}\cap D_n^{(m)}]\to 0$ as n → ∞.

Next, we bound the probability of the event $A_{n,j}\cap D_n^{(l)}$, which is upper bounded by

$$\mathbb{P}[D_n^{(l)}]\le \sum _{j=1}^B{2}^{j-1}\mathbb{P}[max\{l\in [B]:{\widehat{T}}_l\le T_{l-1}\}=j].$$

Observe that

$$\{max\{l\in [B]:{\widehat{T}}_l\le T_{l-1}\}=j\}\subseteq \underset{l=j}{\overset{B}{\cup }}\{T_j-{\widehat{T}}_j\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap \{{\widehat{T}}_{j+1}-T_j\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}$$

so that we have

$$\mathbb{P}[D_n^{(l)}]\le 2^{B-1}\sum _{j=1}^{B-1}\sum _{l>j}\mathbb{P}[\{T_l-{\widehat{T}}_l\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}\cap \{{\widehat{T}}_{l+1}-T_l\ge \frac{{\mathrm{\Delta }}_{min}}{2}\}].$$

Using the same arguments as those used to bound terms in (7.5), we have that $\mathbb{P}[D_n^{(l)}]\to 0$ as n → ∞ under the conditions of the theorem. Similarly, we can show that the term $\mathbb{P}[D_n^{(r)}]\to 0$ as n → ∞. Thus, we have shown that $\mathbb{P}[A_{n,j}\cap C_n^c]\to 0$, which concludes the proof.

7.3. Proof of Lemma 4

Consider fixed. The lemma is a simple consequence of the duality theory, which states that given the subdifferential ŷ_i (which is constant for all i ∈ , being an estimated block of the partition ), all solutions {β̌_·,i}_i∈[n] of (2.2) need to satisfy the complementary slackness condition Σ_b∈\a ŷ_i,bβ̌_b,i = ||β̌_·,i||₁, which holds only if β̌_b,i = 0 for all b ∈ \a for which |ŷ_i,b| < 1.

7.4. Proof of Theorem 5

Since the assumptions of theorem 2 are satisfied, we are going to work on the event

$$\mathcal{E}:=\{\underset{j\in [B]}{max}\mid {\widehat{T}}_j-T_j\mid \le n{\delta }_n\}.$$

In this case, | | = (n). For i ∈ , we write

$$x_{i,a}=\sum _{b\in S^j}{x}_{i,b}{\theta }_b^k+e_i+{\epsilon }_i$$ (7.6)

where $e_i={\sum }_{b\in S}{x}_{i,b}({\beta }_{b,i}-{\theta }_b^k)$ is the bias. Observe that ∀i ∈ ∩ , the bias e_i = 0, while for i ∉ ∩ , the bias e_i is normally distributed with variance bounded by M²φ_max under the assumption A1 and A3.

We proceed to show that S(θ̂^k) ⊂ S^k. Since θ̂^k is an optimal solution of (2.2), it needs to satisfy

$${({\mathbf{X}}_{\backslash a}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{\backslash a}^{{\widehat{\mathcal{B}}}^k}({\widehat{\mathit{\theta }}}^k-{\mathit{\theta }}^k)-{({\mathbf{X}}_{\backslash a}^{{\widehat{\mathcal{B}}}^k})}^{\prime }({\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k})+{\lambda }_1({\widehat{\mathbf{z}}}_{{\widehat{T}}_{k-1}}-{\widehat{\mathbf{z}}}_{{\widehat{T}}_k})+{\lambda }_2\mid {\widehat{\mathcal{B}}}^k\mid {\widehat{\mathbf{y}}}_{{\widehat{T}}_{k-1}}=0.$$ (7.7)

Now, we will construct the vectors θ̌^k, ž_T̂k−1, ž_T̂k and y̌_T̂k−1 that satisfy (7.7) and verify that the subdifferential vectors are dual feasible. Consider the following restricted optimization problem

$$\underset{{\mathit{\theta }}^1,\dots ,{\mathit{\theta }}^{\widehat{B}};{\mathit{\theta }}_{N^k}^k=\mathbf{0}}{min}\sum _{j\in [\widehat{B}]}{\Vert {\mathbf{X}}_a^{{\widehat{\mathcal{B}}}^j}-{\mathbf{X}}_{\backslash a}^{{\widehat{\mathcal{B}}}^j}{\mathit{\theta }}^j\Vert }_2^2+2{\lambda }_1\sum _{j=2}^{\widehat{B}}{\Vert {\mathit{\theta }}^j-{\mathit{\theta }}^{j-1}\Vert }_2+2{\lambda }_2\sum _{j=1}^{\widehat{B}}\mid {\widehat{\mathcal{B}}}^j{\mid \Vert {\mathit{\theta }}^j\Vert }_1,$$ (7.8)

where the vector ${\mathit{\theta }}_{N^k}^k$ is constrained to be 0. Let {θ̌^j}_{j∈ [B̂]} be a solution to the restricted optimization problem (7.8). Set the subgradient vectors as ž_T̂k−1 ∈ ∂||θ̌^k − θ̌^k−1||, ž_Tk ∈ ∂||θ̌^k+1 − θ̌^k|| and ${\check{\mathbf{y}}}_{{\widehat{T}}_{k-1},S^k}=\text{sign}({\check{\mathit{\theta }}}_{S^k}^k)$. Solve (7.7) for y̌_T̂k−1,N^k. By construction, the vectors θ̌^k, ž_T̂k−1, ž_T̂k and y̌_T̂k−1 satisfy (7.7). Furthermore, the vectors ž_T̂k−1 and ž_T̂k are elements of the subdifferential, and hence dual feasible. To show that θ̌^k is also a solution to (3.4), we need to show that ||y̌_T̂k−1,N^k||_∞ ≤ 1, that is, that y̌^T̂_k−1 is also dual feasible variable. Using lemma 4, if we show that y̌_T̂k−1,N^k is strict dual feasible, ||y̌_T̂k−1,N^k||_∞ < 1, then any other solution ${\widehat{\check{\mathit{\theta }}}}^k$ to (3.4) will satisfy ${\widehat{\check{\mathit{\theta }}}}_N^k=\mathbf{0}$.

From (7.7) we can obtain an explicit formula for θ̌_S^k

$${\check{\mathit{\theta }}}_{S^k}^k={\mathit{\theta }}_{S^k}^k+{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}{({\mathbf{X}}_{S^k}^{\widehat{\mathcal{B}}})}^{\prime }({\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k})-{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\left({\lambda }_1({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k})+{\lambda }_2\mid {\widehat{\mathcal{B}}}^k\mid {\check{\mathbf{y}}}_{{\widehat{T}}_{k-1},S^k}\right).$$ (7.9)

Recall that for large enough n we have that | | > p, so that the matrix ${({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}$ is invertible with probability one. Plugging (7.9) into (7.7), we have that ||y̌_T̂k−1,N^k||_∞ < 1 if max_b∈N^k |Y_b| < 1, where Y_b is defined to be

$$Y_b:={\left({\mathbf{X}}_b^{{\widehat{\mathcal{B}}}^k}\right)}^{\prime }[{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\left({\check{\mathbf{y}}}_{{\widehat{T}}_{k-1},S^k}+\frac{{\lambda }_1({\widehat{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\widehat{\mathbf{z}}}_{{\widehat{T}}_k,S^k})}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)+{\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }\left(\frac{{\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k}}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)]-\frac{{\lambda }_1({\check{z}}_{{\widehat{T}}_{k-1},b}-{\check{z}}_{{\widehat{T}}_k,b})}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2},$$ (7.10)

where ${\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }$ is the projection matrix

$${\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }=\mathbf{I}-{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}{\left({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{\prime }.$$

Let Σ̃^k and ${\widehat{\mathbf{\sum }^{\sim }}}^k$ be defined as

$${\mathbf{\sum }^{\sim }}^k=\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }\sum _{i\in {\widehat{\mathcal{B}}}^k}\mathbb{E}[{\mathbf{x}}_{\backslash a}^i{({\mathbf{x}}_{\backslash a}^i)}^{\prime }]\text{and}{\widehat{\mathbf{\sum }^{\sim }}}^k=\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }\sum _{i\in {\widehat{\mathcal{B}}}^k}{\mathbf{x}}_{\backslash a}^i{({\mathbf{x}}_{\backslash a}^i)}^{\prime }.$$

For i ∈ [n], we let (i) index the block to which the sample i belongs to. Now, for any b ∈ N^k, we can write $x_b^i={\mathbf{\sum }}_{{bS}^k}^{\mathcal{B}(i)}{({\mathbf{\sum }}_{S^k{S}^k}^{{\mathcal{B}}^{(i)}})}^{-1}{\mathbf{x}}_{S^k}^i+w_b^i$ where $w_b^i$ is normally distributed with variance ${\sigma }_b^2<1$ and independent of ${\mathbf{x}}_{S^k}^i$. Let F_b ∈ ℝ^{| |} be the vector whose components are equal to ${\mathbf{\sum }}_{{bS}^k}^{\widehat{\mathcal{B}}(i)}{({\mathbf{\sum }}_{S^k{S}^k}^{\widehat{\mathcal{B}}(i)})}^{-1}{\mathbf{x}}_{S^k}^i$, i ∈ , and W_b ∈ ℝ^{| |} be the vector with components equal to $w_b^i$. Using this notation, we write $Y_b=T_b^1+T_b^2+T_b^3+T_b^4$ where

$$T_b^1={\mathbf{F}}_b^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\left({\check{\mathbf{y}}}_{{\widehat{T}}_{k-1}}+\frac{{\lambda }_1({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k})}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)$$ (7.11)

$$T_b^2={\mathbf{F}}_b^{\prime }{\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }\left(\frac{{\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k}}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)$$ (7.12)

$$T_b^3={\left({\stackrel{\sim }{\mathbf{W}}}_b\right)}^{\prime }\left[{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\left({\check{\mathbf{y}}}_{{\widehat{T}}_{k-1}}+\frac{{\lambda }_1({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k})}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)+{\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }\left(\frac{{\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k}}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)\right]$$ (7.13)

and

$$T_b^4=-\frac{{\lambda }_1({\check{z}}_{{\widehat{T}}_{k-1},b}-{\check{z}}_{{\widehat{T}}_k,b})}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}.$$ (7.14)

We analyze each of the terms separately. Starting with the term $T_b^1$, after some algebra, we obtain that

$$\begin{array}{l}{\mathbf{F}}_b^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}=\sum _{j:{\widehat{\mathcal{B}}}^k\cap {\mathcal{B}}^j\ne \varnothing }\frac{\mid {\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k\mid }{\mid {\widehat{\mathcal{B}}}^k\mid }{\mathbf{\sum }}_{{bS}^k}^j{({\mathbf{\sum }}_{S^k{S}^k}^j)}^{-1}({\widehat{\mathbf{\sum }}}_{S^k{S}^k}^{{\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k}-{\mathbf{\sum }}_{S^k{S}^k}^j){\left({\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k\right)}^{-1}\\ +{\mathbf{\sum }^{\sim }}_{{bS}^k}^k({({\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k)}^{-1}-{({\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k)}^{-1})\\ +{\mathbf{\sum }^{\sim }}_{{bS}^k}^k{({\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k)}^{-1}.\end{array}$$ (7.15)

Recall that we are working on the event , so that $\mid \mid \mid {\mathbf{\sum }^{\sim }}_{N^k{S}^k}^k{({\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k)}^{-1}\Vert {\mid }_{\infty }\stackrel{n\to \infty }{\to }\mid \mid \mid {\mathbf{\sum }}_{N^k{S}^k}^k{({\mathbf{\sum }}_{S^k{S}^k}^k)}^{-1}\Vert {\mid }_{\infty }$ and ${(\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2)}^{-1}{\lambda }_1({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k})\stackrel{n\to \infty }{\to }0$ element-wise. Using (7.20) we bound the first two terms in the equation above. We bound the first term by observing that for any j and any b ∈ N^k and n sufficiently large

$$\begin{array}{l}\frac{\mid {\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k\mid }{\mid {\widehat{\mathcal{B}}}^k\mid }{\Vert {\mathbf{\sum }}_{{bS}^k}^j{({\mathbf{\sum }}_{S^k{S}^k}^j)}^{-1}({\widehat{\mathbf{\sum }}}_{S^k{S}^k}^{{\widehat{\mathcal{B}}}^j\cap {\widehat{\mathcal{B}}}^k}-{\mathbf{\sum }}_{S^k{S}^k}^j)\Vert }_{\infty }\le \frac{\mid {\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k\mid }{\mid {\widehat{\mathcal{B}}}^k\mid }{\Vert {\mathbf{\sum }}_{{bS}^k}^j{({\mathbf{\sum }}_{S^k{S}^k}^j)}^{-1}\Vert }_1{\Vert {\widehat{\mathbf{\sum }}}_{S^k{S}^k}^{{\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k}-{\mathbf{\sum }}_{S^k{S}^k}^j\Vert }_{\infty }\\ \le C_1\frac{\mid {\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k\mid }{\mid {\widehat{\mathcal{B}}}^k\mid }{\Vert {\widehat{\mathbf{\sum }}}_{S^k{S}^k}^{{\mathcal{B}}^j\cap {\widehat{\mathcal{B}}}^k}-{\mathbf{\sum }}_{S^k{S}^k}^j\Vert }_{\infty }\le {\epsilon }_1\end{array}$$

with probability 1−c₁ exp(−c₂ log n). Next, for any b ∈ N^k we bound the second term as

$$\begin{array}{l}{\Vert {\mathbf{\sum }^{\sim }}_{{bS}^k}^k({({\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k)}^{-1}-{({\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k)}^{-1})\Vert }_1\le C_2{\Vert {({\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k)}^{-1}-{({\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k)}^{-1})\Vert }_F\\ \le C_2{\Vert {\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k\Vert }_F^2{\Vert {\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k-{\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k\Vert }_F+\mathcal{O}({\Vert {\widehat{\mathbf{\sum }^{\sim }}}_{S^k{S}^k}^k-{\mathbf{\sum }^{\sim }}_{S^k{S}^k}^k\Vert }_F^2)\\ \le {\epsilon }_2\end{array}$$

with probability 1−c₁ exp(−c₂ log n). Choosing ε₁, ε₂ sufficiently small and for n large enough, we have that ${max}_b\mid T_b^1\mid \le 1-\alpha +o_p(1)$ under the assumption A4.

We proceed with the term $T_b^2$, which can be written as

$$\begin{array}{l}{T}_b^2={(\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2)}^{-1}\left({\mathbf{\sum }}_{{bS}^k}^k{({\mathbf{\sum }}_{S^k{S}^k}^k)}^{-1}-{\mathbf{F}}_b^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\right)\sum _{i\in {\mathcal{B}}^k\cap {\widehat{\mathcal{B}}}^k}{\mathbf{x}}_{S^k}^i{\epsilon }^i\\ +{(\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2)}^{-1}\sum _{i\notin {\mathcal{B}}^k\cap {\widehat{\mathcal{B}}}^k}\left({\mathbf{\sum }}_{{bS}^k}^{\mathcal{B}(i)}{\left({\mathbf{\sum }}_{S^k{S}^k}^{\mathcal{B}(i)}\right)}^{-1}-{\mathbf{F}}_b^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\right){\mathbf{x}}_{S^k}^i(e^i+{\epsilon }^i).\end{array}$$

Since we are working on the event the second term in the above equation is dominated by the first term. Next, using (7.15) together with (7.20), we have that for all b ∈ N^k

$${\Vert {\mathbf{\sum }}_{{bS}^k}^k{({\mathbf{\sum }}_{S^k{S}^k}^k)}^{-1}-{\mathbf{F}}_b^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\Vert }_2=o_p(1).$$

Combining with Lemma 8, we have that under the assumptions of the theorem

$$\underset{b}{max}\mid T_b^2\mid =o_p(1).$$

We deal with the term $T_b^3$ by conditioning on ${\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}$ and ε, we have that W_b is independent of the terms in the squared bracket in $T_b^3$ , since all ž_T̂k−1,S, ž_T̂k,S and ŷ_T̂k−1,S are determined from the solution to the restricted optimization problem. To bound the second term, we observe that conditional on ${\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}$ and ε, the variance of $T_b^3$ can be bounded as

$$\begin{array}{l}\text{Var}(T_b^3)\le {\Vert {\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}{\check{\eta }}_{S^k}+{\mathbf{H}}_{S^k}^{{\widehat{\mathcal{B}}}^k,\perp }\left(\frac{{\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k}}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\right)\Vert }_2^2\\ \le {\check{\eta }}_{S^k}^{\prime }{\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}{\check{\eta }}_{S^k}+{\Vert \frac{{\mathbf{e}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k}}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\Vert }_2^2,\end{array}$$ (7.16)

where

$${\check{\eta }}_{S^k}=\left({\check{\mathbf{y}}}_{{\widehat{T}}_{k-1},S^k}+\frac{{\lambda }_1({\check{\mathbf{z}}}_{T_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S})}{\mid \widehat{\mathcal{B}}\mid {\lambda }_2}\right).$$

Using lemma 9 and Young’s inequality, the first term in (7.16) is upper bounded by

$$\frac{18}{\mid \widehat{\mathcal{B}}\mid {\phi }_{min}}\left(s+\frac{2{\lambda }_1^2}{{\mid \widehat{\mathcal{B}}\mid }^2{\lambda }_2^2}\right)$$

with probability at least 1 − 2 exp(−| |/2 + 2 log n). Using lemma 7 we have that the second term is upper bounded by

$$\frac{(1+{\delta }^{\prime })(1+M^2{\phi }_{max})}{\mid \widehat{\mathcal{B}}\mid {\lambda }_2^2}$$

with probability at least 1−exp(−c₁| |δ′²+2 log n). Combining the two bounds, we have that $\text{Var}(T_b^3)\le c_{1}s{(\mid {\widehat{\mathcal{B}}}^k\mid )}^{-1}$ with high probability, using the fact that (| |λ₂)⁻¹λ₁ → 0 and | |λ₂ → ∞ as n → ∞. Using the bound on the variance of the term $T_b^3$ and the Gaussian tail bound, we have that

$$\underset{b\in N}{max}\mid T_b^3\mid =o_p(1).$$

Combining the results, we have that max_b∈N^k |Y_b| ≤ 1 − α + o_p(1). For a sufficiently large n, under the conditions of the theorem, we have shown that max_b∈N |Y_b| < 1 which implies that $\mathbb{P}[S({\widehat{\mathit{\theta }}}^k)\subset S^k]\stackrel{n\to \infty }{\to }1$.

Next, we proceed to show that $\mathbb{P}[S^k\subset S({\widehat{\mathit{\theta }}}^k)]\stackrel{n\to \infty }{\to }1$. Observe that

$$\mathbb{P}[S^k\not\subset S({\widehat{\mathit{\theta }}}^k)]\le \mathbb{P}[{\Vert {\widehat{\mathit{\theta }}}_{S^k}^k-{\mathit{\theta }}_{S^k}^k\Vert }_{\infty }\ge {\theta }_{min}].$$

From (7.7) we have that ${\Vert {\widehat{\mathit{\theta }}}_{S^k}^k-{\mathit{\theta }}_{S^k}^k\Vert }_{\infty }$ is upper bounded by

$$\begin{array}{l}{\Vert {\left(\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }({\stackrel{\sim }{\mathbf{e}}}^{{\widehat{\mathcal{B}}}^k}+{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k})\Vert }_{\infty }\\ +{\Vert {\left({({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\left({\lambda }_1({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k})-{\lambda }_2\mid {\widehat{\mathcal{B}}}^{{\widehat{\mathcal{B}}}^k}\mid {\check{\mathbf{y}}}_{T_{k-1},S^k}\right)\Vert }_{\infty }.\end{array}$$

Since ẽ_i ≠ 0 only on i ∈ \ and nδ_n/| | → 0, the term involving ẽ is stochastically dominated by the term involving ∊ and can be ignored. Define the following terms

$$\begin{array}{l}{T}_1={\left(\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathit{\epsilon }}^{{\widehat{\mathcal{B}}}^k},\\ T_2={\left(\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}\frac{{\lambda }_1}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}({\check{\mathbf{z}}}_{{\widehat{T}}_{k-1},S^k}-{\check{\mathbf{z}}}_{{\widehat{T}}_k,S^k}),\\ T_3={\left(\frac{1}{\mid {\widehat{\mathcal{B}}}^k\mid }{({\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k})}^{\prime }{\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}\right)}^{-1}{\check{\mathbf{y}}}_{{\widehat{T}}_{k-1},S^k}.\end{array}$$

Conditioning on ${\mathbf{X}}_{S^k}^{{\widehat{\mathcal{B}}}^k}$, the term T₁ is a |S^k| dimensional Gaussian with variance bounded by c₁/n with probability at least 1 − c₁ exp(−c₂ log n) using lemma 9. Combining with the Gaussian tail bound, the term ||T₁||_∞ can be upper bounded as

$$\mathbb{P}\left[{\Vert T_1\Vert }_{\infty }\ge c_1\sqrt{\frac{logs}{n}}\right]\le c_{2}exp(-c_{3}logn).$$ (7.17)

Using lemma 9, we have that with probability greater than 1 − c₁ exp(−c₂ log n)

$${\Vert T_2\Vert }_{\infty }\le {\Vert T_2\Vert }_2\le c_3\frac{{\lambda }_1}{\mid {\widehat{\mathcal{B}}}^k\mid {\lambda }_2}\to 0$$

under the conditions of theorem. Similarly ${\Vert T_3\Vert }_{\infty }\le c_1\sqrt{s}$, with probability greater than 1 − c₁ exp(−c₂ log n). Combining the terms, we have that

$${\Vert {\mathit{\theta }}^k-{\widehat{\mathit{\theta }}}^k\Vert }_{\infty }\le c_1\sqrt{\frac{logs}{n}}+c_2\sqrt{s}{\lambda }_2$$

with probability at least 1 − c₃ exp(−c₄ log n). Since ${\theta }_{min}=\mathrm{\Omega }(\sqrt{log(n)/n})$, we have shown that S^k ⊆ S(θ̂^k). Combining with the first part, it follows that S(θ̂^k) = S^k with probability tending to one.

7.5. Proof of Lemma 6

We have that ∇ f(A) = A⁻¹. Then

$$\begin{array}{l}{\Vert \nabla f(\mathbf{A})-\nabla f({\mathbf{A}}^{\prime })\Vert }_F={\Vert {\mathbf{A}}^{-1}-{({\mathbf{A}}^{\prime })}^{-1}\Vert }_F\\ \le {\mathrm{\Lambda }}_{max}{\mathbf{A}}^{-1}{\Vert \mathbf{A}-{\mathbf{A}}^{\prime }\Vert }_F{\mathrm{\Lambda }}_{max}{\mathbf{A}}^{-1}\\ \le {\gamma }^{-2}{\Vert \mathbf{A}-{\mathbf{A}}^{\prime }\Vert }_F.\end{array}$$

Table 1.

Summary of symbols used throughout the paper

Symbol	Meaning	Example
[n]	used to denote the set {1, …, n}
[t1 : t2]	used to denote the set {t1, t1 + 1, …, t2 − 1}
i	used for indexing related to samples	xi or ${\beta }_{·,i}^a$
j, k	used for indexing related to block	θa,j or $S_a^k$
A, b	used for indexing nodes in a graph	a, b ∈ V
G	the graph consisting of vertices and edges	G = (V, E)
V	the set of nodes in a graph	V = [p]
Ei	the set of edges at time i
Xa	the component of a random vector X indexed by the vertex a
${\mathit{\beta }}_{·,i}^a$	the vector of regression coefficients for sample i
θa,j	the vector of regression coefficients for block j
	the set of partition boundaries
{τj}j	the set of boundary fractions	Tj = ⌊nτj⌋
	an index set for the samples in the partition j	⊂ [n]
B	denotes the number of partitions
$S_a^j$	the set of neighbors of node a in block j
S(θa,j)	the set of non-zero elements of θa,j
${\overline{S}}_a^j$	the closure of $S_a^j$	${\overline{S}}_a^j=S_a^j\cup \{a\}$
$N_a^j$	nodes not in the neighborhood of the node a in block j	$N_a^j=[p]\backslash {\overline{S}}_a^j$
\a	the set of all vertices excluding the vertex a	\a = [p]\{a}
|·|	cardinality of a set or absolute value
Σ	the covariance matrix
σab	an element of the covariance matrix
Ω	the precision matrix
ωab	an element of the precision matrix
〈·, ·〉	the dot product	〈a, b〉 = a′b
〈〈·, ·〉〉	the dot product between matrices	〈〈A, B〉〉 = tr(A′B)
ξmin	the minimum change between regression coefficient	||θa,j − θa,j−1||2≥ξmin
θmin	the minimum size of a coefficient	$\mid {\theta }_b^{a,j}\mid \ge {\theta }_{min}$
Δmin	the minimum size of a block	| | ≥ Δmin

Appendix

Technical results

In this section we collect some technical results needed for the proves presented in §7.

Lemma 7

Let {ζⁱ}_{i ∈ [n]} be a sequence of iid (0, 1) random variables. If v_n ≥ C log n, for some constant C > 16, then

$$\mathbb{P}\left[\underset{\begin{array}{c}1\le l<r\le n\\ r-l>r_n\end{array}}{\cap }\left\{\sum _{i=l}^r{({\zeta }^i)}^2\le (1+C)(r-l+1)\right\}\right]\ge 1-exp(-c_{1}logn)$$

for some constant c₁ > 0.

Proof

For any 1 ≤ l < r ≤ n, with r − l > v_n we have

$$\begin{array}{l}\mathbb{P}\left[\sum _{i=l}^r{({\zeta }^i)}^2\ge (1+C)(r-l+1)\right]\le exp(-C(r-l+1)/8)\\ \le exp(-\mathit{C}logn/8)\end{array}$$

using (7.21). The lemma follows from an application of the union bound.

Lemma 8

Let {x_i}_{i ∈ [n]} be independent observations from (1.1) and let {ε_i}_{i ∈ [n]} be independent (0, 1). Assume that A1 holds. If v_n ≥ C log n for some constant C > 16, then

$$\mathbb{P}\left[\underset{j\in [B]}{\cap }\underset{\begin{array}{c}l,r\in {\mathcal{B}}^j\\ r-l>v_n\end{array}}{\cap }\left\{\frac{1}{r-l+1}{\Vert \sum _{i=l}^r{\mathbf{x}}_i{\epsilon }_i\Vert }_2\le \frac{{\phi }_{max}^{1/2}\sqrt{1+C}}{\sqrt{r-l+1}}\sqrt{p(1+\mathit{C}logn)}\right\}\right]\ge 1-c_{1}exp(-c_{2}logn),$$

for some constants c₁, c₂ > 0.

Proof

Let Σ^1/2 denote the symmetric square root of the covariance matrix Σ_SS and let (i) denote the block of the true partition such that i ∈ . With this notation, we can write x_i = (Σ⁽ⁱ⁾)^1/2 u_i where u_i ~ (0, I). For any l ≤ r ∈ we have

$${\Vert \sum _{i=l}^r{\mathbf{x}}_i{\epsilon }_i\Vert }_2={\Vert \sum _{i=l}^r{({\mathbf{\sum }}^j)}^{1/2}{\mathbf{u}}_i{\epsilon }_i\Vert }_2\le {\phi }_{max}^{1/2}{\Vert \sum _{i=l}^r{\mathbf{u}}_i{\epsilon }_i\Vert }_2.$$

Conditioning on {∊_i}_i, for each b ∈ [p], ${\sum }_{i=l}^r{u}_{i,b}{\epsilon }_i$ is a normal random variable with variance ${\sum }_{i=l}^r{({\epsilon }_i)}^2$. Hence, ${\Vert {\sum }_{i=l}^r{\mathbf{u}}_i{\epsilon }_i\Vert }_2^2/({\sum }_{i=l}^r{({\epsilon }_i)}^2)$ conditioned on {ε_i}_i is distributed according to ${\chi }_p^2$ and

$$\begin{array}{c}\mathbb{P}\left[\frac{1}{r-l+1}{\Vert \sum _{i=l}^r{\mathbf{x}}_i{\epsilon }_i\Vert }_2\ge \frac{{\phi }_{max}^{1/2}\sqrt{{\sum }_{i=l}^r{({\epsilon }_i)}^2}}{r-l+1}\sqrt{p(1+\mathit{C}logn)}|{\{{\epsilon }_i\}}_{i=l}^r\right]\\ \le \mathbb{P}[{\chi }_p^2\ge p(1+\mathit{C}logn)]\le exp(-\mathit{C}logn/8),\end{array}$$

where the last inequality follows from (7.21). Using lemma 7, for all l, r ∈ with r − l > v_n the quantity ${\sum }_{i=l}^r{({\epsilon }_i)}^2$ is bounded by (1 + C)(r − l + 1) with probability at least 1 − exp(−c₁ log n), which gives us the following bound

$$\mathbb{P}\left[\underset{j\in [B]}{\cap }\underset{\begin{array}{c}l,r\in {\mathcal{B}}^j\\ r-l>v_n\end{array}}{\cap }\left\{\frac{1}{r-l+1}{\Vert \sum _{i=l}^r{\mathbf{x}}_i{\epsilon }_i\Vert }_2\le \frac{{\phi }_{max}^{1/2}\sqrt{1+C}}{\sqrt{r-l+1}}\sqrt{p(1+\mathit{C}logn)}\right\}\right]\ge 1-c_{1}exp(-c_{2}logn).$$

Lemma 9

Let {x_i}_{i ∈ [n]} be independent observations from (1.1). Assume that A1 holds. Then for any v_n > p,

$$\mathbb{P}\left[\underset{\begin{array}{c}1\le l<r\le n\\ r-l>v_n\end{array}}{max}{\mathrm{\Lambda }}_{max}\left(\frac{1}{r-l+1}\sum _{i=l}^r{\mathbf{x}}_i{({\mathbf{x}}_i)}^{\prime }\right)\ge 9{\phi }_{max}\right]\le 2n^{2}exp(-v_n/2)$$

and

$$\mathbb{P}\left[\underset{\begin{array}{c}1\le l<r\le n\\ r-l>v_n\end{array}}{min}{\mathrm{\Lambda }}_{min}\left(\frac{1}{r-l+1}\sum _{i=l}^r{\mathbf{x}}_i{({\mathbf{x}}_i)}^{\prime }\right)\le {\phi }_{min}/9\right]\le 2n^{2}exp(-v_n/2).$$

Proof

For any 1 ≤ l < r ≤ n, with r − l ≥ v_n we have

$$\begin{array}{l}\mathbb{P}\left[{\mathrm{\Lambda }}_{max}\left(\frac{1}{r-l+1}\sum _{i=l}^r{\mathbf{x}}_i{({\mathbf{x}}_i)}^{\prime }\right)\ge 9{\phi }_{max}\right]\le 2exp(-(r-l+1)/2)\\ \le 2exp(-v_n/2)\end{array}$$

using (7.18), convexity of Λ_max(·) and A1. The lemma follows from an application of the union bound. The other inequality follows using a similar argument.

Proof of Proposition 3

The following proof follows main ideas already given in theorem 2. We provide only a sketch.

Given an upper bound on the number of partitions B_max, we are going to perform the analysis on the event {B̂ ≤ B_max}. Since

$$\mathbb{P}[h(\widehat{\mathcal{T}},\mathcal{T})\ge n{\delta }_n\mid \{\widehat{B}\le B_{max}\}]\le \sum _{B^{\prime }=B}^{B_{max}}\mathbb{P}[h(\widehat{\mathcal{T}},\mathcal{T})\ge n{\delta }_n\mid \{\mid \widehat{\mathcal{T}}\mid =B^{\prime }+1\}],$$

we are going to focus on ℙ [h( , ) ≥ nδ_n|{| | = B′+ 1}] for B′> B (for B′ = B it follows from theorem 2 that h( , ) < nδ_n with high probability). Let us define the following events

$$\begin{array}{l}{\mathcal{E}}_{j,1}=\{\exists l\in [B^{\prime }]:\mid {\widehat{T}}_l-T_j\mid \ge n{\delta }_n,\mid {\widehat{T}}_{l+1}-T_j\mid \ge n{\delta }_n\text{and}{\widehat{T}}_l<T_j<{\widehat{T}}_{l+1}\}\\ {\mathcal{E}}_{j,2}=\{\forall l\in [B^{\prime }]:\mid {\widehat{T}}_l-T_j\mid \ge n{\delta }_n\text{and}{\widehat{T}}_l<T_j\}\\ {\mathcal{E}}_{j,3}=\{\forall l\in [B^{\prime }]:\mid {\widehat{T}}_l-T_j\mid \ge n{\delta }_n\text{and}{\widehat{T}}_l>T_j\}.\end{array}$$

Using the above events, we have the following bound

$$\mathbb{P}[h(\widehat{\mathcal{T}},\mathcal{T})\ge n{\delta }_n\mid \{\mid \widehat{\mathcal{T}}\mid =B^{\prime }+1\}]\le \sum _{j\in [B]}\mathbb{P}[{\mathcal{E}}_{j,1}]+\mathbb{P}[{\mathcal{E}}_{j,2}]+\mathbb{P}[{\mathcal{E}}_{j,3}].$$

The probabilities of the above events can be bounded using the same reasoning as in the proof of theorem 2, by repeatedly using the KKT conditions given in (3.2). In particular, we can use the strategy used to bound the event A_n,j,2. Since the proof is technical and does not reveal any new insight, we omit the details.

A collection of known results

This section collects some known results that we have used in the paper. We start by collecting some results on the eigenvalues of random matrices. Let $\mathbf{x}\stackrel{\mathit{iid}}{~}\mathcal{N}(0,\mathbf{\sum })$, i ∈ [n], and Σ̂ = n⁻¹ Σ x_i(x_i)′ be the empirical covariance matrix. Denote the elements of the covariance matrix Σ as [σ_ab] and of the empirical covariance matrix Σ̂ as [σ̂_ab].

Using standard results on concentration of spectral norms and eigenvalues [10], [38] derives the following two crude bounds that can be very useful. Under the assumption that p < n,

$$\mathbb{P}[{\mathrm{\Lambda }}_{max}(\widehat{\mathbf{\sum }})\ge 9{\phi }_{max}]\le 2exp(-n/2)$$ (7.18)

$$\mathbb{P}[{\mathrm{\Lambda }}_{min}(\widehat{\mathbf{\sum }})\le {\phi }_{min}/9]\le 2exp(-n/2).$$ (7.19)

From Lemma A.3. in [6] we have the following bound on the elements of the covariance matrix

$$\mathbb{P}[\mid {\widehat{\sigma }}_{ab}-{\sigma }_{ab}\mid \ge \epsilon ]\le c_{1}exp(-c_{2}n{\epsilon }^2),\mid \epsilon \mid \le {\epsilon }_0$$ (7.20)

where c₁ and c₂ are positive constants that depend only on Λ_max(Σ) and ε₀.

Next, we use the following tail bound for χ² distribution from [25], which holds for all ε > 0,

$$\mathbb{P}[{\chi }_n^2>n+\epsilon ]\le exp(-\frac{1}{8}min(\epsilon ,\frac{{\epsilon }^2}{n})).$$ (7.21)