The graphical lasso: New insights and alternatives

Abstract

The graphical lasso [5] is an algorithm for learning the structure in an undirected Gaussian graphical model, using ℓ₁ regularization to control the number of zeros in the precision matrix Θ = Σ⁻¹ [2, 11]. The R package GLASSO [5] is popular, fast, and allows one to efficiently build a path of models for different values of the tuning parameter. Convergence of GLASSO can be tricky; the converged precision matrix might not be the inverse of the estimated covariance, and occasionally it fails to converge with warm starts. In this paper we explain this behavior, and propose new algorithms that appear to outperform GLASSO.

By studying the “normal equations” we see that, GLASSO is solving the dual of the graphical lasso penalized likelihood, by block coordinate ascent; a result which can also be found in [2]. In this dual, the target of estimation is Σ, the covariance matrix, rather than the precision matrix Θ. We propose similar primal algorithms P-GLASSO and DP-GLASSO, that also operate by block-coordinate descent, where Θ is the optimization target. We study all of these algorithms, and in particular different approaches to solving their coordinate sub-problems. We conclude that DP-GLASSO is superior from several points of view.

1. Introduction

Consider a data matrix X_n×p, a sample of n realizations from a p-dimensional Gaussian distribution with zero mean and positive definite covariance matrix Σ. The task is to estimate the unknown Σ based on the n samples — a challenging problem especially when n ≪ p, when the ordinary maximum likelihood estimate does not exist. Even if it does exist (for p ≤ n), the MLE is often poorly behaved, and regularization is called for. The Graphical Lasso [5] is a regularization framework for estimating the covariance matrix Σ, under the assumption that its inverse Θ = Σ⁻¹ is sparse [2, 11, 8]. Θ is called the precision matrix; if an element θ_jk = 0, this implies that the corresponding variables X_j and X_k are conditionally independent, given the rest. Our algorithms focus either on the restricted version of Θ or its inverse W = Θ⁻¹. The graphical lasso problem minimizes a ℓ₁-regularized negative log-likelihood:

$$\underset{\mathbf{\Theta }\succ 0}{\text{minimize}}f(\mathbf{\Theta }):=-\mathrm{log det}(\mathbf{\Theta })+\mathrm{tr}(\mathbf{S}\mathbf{\Theta })+\mathrm{\lambda }{\Vert \mathbf{\Theta }\Vert }_1.$$ (1.1)

Here S is the sample covariance matrix, ${\Vert \mathbf{\Theta }\Vert }_1$ denotes the sum of the absolute values of Θ, and λ is a tuning parameter controlling the amount of ℓ₁ shrinkage. This is a semidefinite programming problem (SDP) in the variable Θ [4].

In this paper we revisit the GLASSO algorithm proposed by Friedman, Hastie and Tibshirani [5] for solving (1.1); we analyze its properties, expose problems and issues, and propose alternative algorithms more suitable for the task.

Some of the results and conclusions of this paper can be found in [2], both explicitly and implicitly. We re-derive some of the results and derive new results, insights and algorithms, using a unified and more elementary framework.

Notation

We denote the entries of a matrix A_n×n by a_ij. ‖A‖₁ denotes the sum of its absolute values, ‖A‖_∞ the maximum absolute value of its entries, ‖A‖_F is its Frobenius norm, and abs(A) is the matrix with elements |a_ij|. For a vector u ∈ ℜ^q, ‖u‖₁ denotes the ℓ₁ norm, and so on.

From now on, unless otherwise specified, we will assume that λ > 0.

2. Review of the GLASSO algorithm

We use the frame-work of “normal equations” as in [6, 5]. Using sub-gradient notation, we can write the optimality conditions (aka “normal equations”) for a solution to (1.1) as

$$-{\mathbf{\Theta }}^{-1}+\mathbf{S}+\mathrm{\lambda }\mathbf{\Gamma }=\mathbf{0},$$ (2.1)

where Γ is a matrix of component-wise signs of Θ:

$$\begin{array}{l}{\mathbf{\gamma }}_{\mathit{jk}}=\mathrm{sign}({\theta }_{\mathit{jk}})\text{if}{\theta }_{\mathit{jk}}\ne 0\\ {\mathbf{\gamma }}_{\mathit{jk}}\in [-1,1]\text{if}{\theta }_{\mathit{jk}}=0\end{array}$$ (2.2)

(we use the notation γ_jk ∈ Sign(θ_jk)). Since the global stationary conditions of (2.1) require θ_jj to be positive, this implies that

$$w_{\mathit{ii}}=s_{\mathit{ii}}+\mathrm{\lambda },i=1,\dots ,p,$$ (2.3)

where W = Θ⁻¹.

GLASSO uses a block-coordinate method for solving (2.1). Consider a partitioning of Θ and Γ:

$$\mathbf{\Theta }=\left(\begin{array}{cc}{\mathbf{\Theta }}_{11}& {\mathit{\theta }}_{12}\\ {\mathit{\theta }}_{21}& {\theta }_{22}\end{array}\right),\mathbf{\Gamma }=\left(\begin{array}{cc}{\mathbf{\Gamma }}_{11}& {\mathbf{\gamma }}_{12}\\ {\mathbf{\gamma }}_{21}& {\mathbf{\gamma }}_{22}\end{array}\right)$$ (2.4)

where Θ₁₁ is (p − 1) × (p − 1), θ₁₂ is (p − 1) × 1 and θ₂₂ is scalar. W and S are partitioned the same way. Using properties of inverses of block-partitioned matrices, observe that W = Θ⁻¹ can be written in two equivalent forms:

$$\left(\begin{array}{cc}{\mathbf{W}}_{11}& {\mathbf{w}}_{12}\\ {\mathbf{w}}_{21}& w_{22}\end{array}\right)=\left(\begin{array}{cc}{({\mathbf{\Theta }}_{11}-\frac{{\mathit{\theta }}_{12}{\mathit{\theta }}_{21}}{{\mathit{\theta }}_{22}})}^{-1}& -{\mathbf{W}}_{11}\frac{{\mathit{\theta }}_{12}}{{\mathit{\theta }}_{22}}\\ ·& \frac{1}{{\mathit{\theta }}_{22}}-\frac{{\mathit{\theta }}_{21}{\mathbf{W}}_{11}{\mathit{\theta }}_{12}}{{\mathit{\theta }}_{22}^2}\end{array}\right)$$ (2.5)

$$=\left(\begin{array}{cc}{\mathbf{\Theta }}_{11}^{-1}+\frac{{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}& -\frac{{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}\\ ·& \frac{1}{{\theta }_{22}-{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}\end{array}\right)$$ (2.6)

GLASSO solves for a row/column of (2.1) at a time, holding the rest fixed. Considering the pth column of (2.1), we get

$$-{\mathbf{w}}_{12}+{\mathbf{s}}_{12}+\mathrm{\lambda }{\mathbf{\gamma }}_{12}=\mathbf{0}.$$ (2.7)

Reading off w₁₂ from (2.5) we have

$${\mathbf{w}}_{12}=-{\mathbf{W}}_{11}{\mathit{\theta }}_{12}/{\theta }_{22}$$ (2.8)

and plugging into (2.7), we have:

$${\mathbf{W}}_{11}\frac{{\mathit{\theta }}_{12}}{{\theta }_{22}}+{\mathbf{s}}_{12}+\mathrm{\lambda }{\mathbf{\gamma }}_{12}=\mathbf{0}.$$ (2.9)

GLASSO operates on the above gradient equation, as described below.

As a variation consider reading off w₁₂ from (2.6):

$$\frac{{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}}{({\theta }_{22}-{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12})}+{\mathbf{s}}_{12}+\mathrm{\lambda }{\mathbf{\gamma }}_{12}=\mathbf{0}.$$ (2.10)

The above simplifies to

$${\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12}{w}_{22}+{\mathbf{s}}_{12}+\mathrm{\lambda }{\mathbf{\gamma }}_{12}=\mathbf{0},$$ (2.11)

where $w_{22}=1/({\theta }_{22}-{\mathit{\theta }}_{21}{\mathbf{\Theta }}_{11}^{-1}{\mathit{\theta }}_{12})$ is fixed (by the global stationary conditions (2.3)). We will see that these two apparently similar estimating equations (2.9) and (2.11) lead to very different algorithms.

The GLASSO algorithm solves (2.9) for β = θ₁₂/θ₂₂, that is

$${\mathbf{W}}_{11}\mathit{\beta }+{\mathbf{s}}_{12}+\mathrm{\lambda }{\mathbf{\gamma }}_{12}=\mathbf{0},$$ (2.12)

where γ₁₂ ∈ Sign(β), since θ₂₂ > 0. (2.12) is the stationarity equation for the following ℓ₁ regularized quadratic program:

$$\underset{\mathit{\beta }\in {\mathrm{\Re }}^{p-1}}{\text{minimize}}\{\frac{1}{2}\mathit{\beta }\prime {\mathbf{W}}_{11}\mathit{\beta }+\mathit{\beta }\prime {\mathbf{s}}_{12}+\mathrm{\lambda }{\Vert \mathit{\beta }\Vert }_1\},$$ (2.13)

where W₁₁ ≻ 0 is assumed to be fixed. This is analogous to a lasso regression problem of the last variable on the rest, except the cross-product matrix S₁₁ is replaced by its current estimate W₁₁. This problem itself can be solved efficiently using elementwise coordinate descent, exploiting the sparsity in β. From $\widehat{\mathit{\beta }}$, it is easy to obtain ${\widehat{\mathbf{w}}}_{12}$ from (2.8). Using the lower-right element of (2.5), ${\widehat{\theta }}_{22}$ is obtained by

$$\frac{1}{{\widehat{\theta }}_{22}}=w_{22}-\widehat{\mathit{\beta }}\prime {\widehat{\mathbf{w}}}_{12}.$$ (2.14)

Finally, ${\widehat{\mathit{\theta }}}_{12}$ can now be recovered from $\widehat{\mathit{\beta }}$ and ${\widehat{\theta }}_{22}$. Notice, however, that having solved for β and updated w₁₂, GLASSO can move onto the next block; disentangling θ₁₂ and θ₂₂ can be done at the end, when the algorithm over all blocks has converged. The GLASSO algorithm is outlined in Algorithm 1. We show in Lemma 3 in Section 8 that the successive updates in GLASSO keep W positive definite.

Figure 1 (left panel, black curve) plots the objective f(Θ^(k)) for the sequence of solutions produced by GLASSO on an example. Surprisingly, the curve is not monotone decreasing, as confirmed by the middle plot. If GLASSO were solving (1.1) by block coordinate-descent, we would not anticipate this behavior.

FIG 1.

[Left panel] The objective values of the primal criterion (1.1) and the dual criterion (4.1) corresponding to the covariance matrix W produced by GLASSO algorithm as a function of the iteration index (each column/row update). [Middle Panel] The successive differences of the primal objective values — the zero crossings indicate non-monotonicity. [Right Panel] The successive differences in the dual objective values — there are no zero crossings, indicating that GLASSO produces a monotone sequence of dual objective values.

A closer look at steps (2.8) and (2.9) of the GLASSO algorithm leads to the following observations:

Algorithm 1 GLASSO algorithm [5]

Initialize W = S + λI. Cycle around the columns repeatedly, performing the following steps till convergence: Rearrange the rows/columns so that the target column is last (implicitly). Solve the lasso problem (2.13), using as warm starts the solution from the previous round for this column. Update the row/column (off-diagonal) of the covariance using ${\widehat{\mathbf{w}}}_{12}$ (2.8). Save $\widehat{\beta }$ for this column in the matrix B. Finally, for every row/column, compute the diagonal entries ${\widehat{\theta }}_{\mathit{jj}}$ using (2.14), and convert the B matrix to Θ.

1. We wish to solve (2.7) for θ₁₂. However θ₁₂ is entangled in W₁₁, which is (incorrectly) treated as a constant.

2. After updating θ₁₂, we see from (2.6) that the entire (working) covariance matrix W changes. GLASSO however updates only w₁₂ and w₂₁.

These two observations explain the non-monotone behavior of GLASSO in minimizing f(Θ). Section 3 shows a corrected block-coordinate descent algorithm for Θ, and Section 4 shows that the GLASSO algorithm is actually optimizing the dual of problem (1.1), with the optimization variable being W.

3. A corrected GLASSO block coordinate-descent algorithm

Recall that (2.11) is a variant of (2.9), where the dependence of the covariance sub-matrix W₁₁ on θ₁₂ is explicit. With α = θ₁₂w₂₂ (with w₂₂ ≥ 0 fixed), Θ₁₁ ≻ 0, (2.11) is equivalent to the stationary condition for

Algorithm 2 P-GLASSO Algorithm

Initialize W = diag(S) + λI, and Θ = W−1. Cycle around the columns repeatedly, performing the following steps till convergence: Rearrange the rows/columns so that the target column is last (implicitly). Compute ${\mathbf{\Theta }}_{11}^{-1}$ using (3.3). Solve (3.1) for α, using as warm starts the solution from the previous round of row/column updates. Update ${\widehat{\mathit{\theta }}}_{12}=\widehat{\mathit{\alpha }}/w_{22}$, and ${\widehat{\theta }}_{22}$ using (3.2). Update Θ and W using (2.6), ensuring that ΘW = Ip. Output the solution Θ (precision) and its exact inverse W (covariance).

$$\underset{\mathit{\alpha }\in {\mathrm{\Re }}^{p-1}}{\text{minimize}}\{\frac{1}{2}\mathit{\alpha }\prime {\mathbf{\Theta }}_{11}^{-1}\mathit{\alpha }+\mathit{\alpha }\prime {\mathbf{s}}_{12}+\mathrm{\lambda }{\Vert \mathit{\alpha }\Vert }_1\}.$$ (3.1)

If $\widehat{\mathit{\alpha }}$ is the minimizer of (3.1), then ${\widehat{\mathit{\theta }}}_{12}=\widehat{\mathit{\alpha }}/w_{22}$. To complete the optimization for the entire row/column we need to update θ₂₂. This follows simply from (2.6)

$${\widehat{\theta }}_{22}=\frac{1}{w_{22}}+{\widehat{\mathit{\theta }}}_{21}{\mathbf{\Theta }}_{11}^{-1}{\widehat{\mathit{\theta }}}_{12},$$ (3.2)

with w₂₂ = s₂₂ + λ.

To solve (3.1) we need ${\mathbf{\Theta }}_{11}^{-1}$ for each block update. We achieve this by maintaining W = Θ⁻¹ as the iterations proceed. Then for each block

• we obtain ${\mathbf{\Theta }}_{11}^{-1}$ from

  $${\mathbf{\Theta }}_{11}^{-1}={\mathbf{W}}_{11}-{\mathbf{w}}_{12}{\mathbf{w}}_{21}/w_{22};$$ (3.3)

• once θ₁₂ is updated, the entire working covariance matrix W is updated (in particular the portions W₁₁ and w₁₂), via the identities in (2.6), using the known ${\mathbf{\Theta }}_{11}^{-1}$.

Both these steps are simple rank-one updates with a total cost of O(p²) operations.

We refer to this as the primal graphical lasso or P-GLASSO, which we present in Algorithm 2.

The P-GLASSO algorithm requires slightly more work than GLASSO, since an additional O(p²) operations have to be performed before and after each block update. In return we have that after every row/column update, Θ and W are positive definite (for λ > 0) and ΘW = I_p.

4. What is GLASSO actually solving?

Building upon the framework developed in Section 2, we now proceed to establish that GLASSO solves the convex dual of problem (1.1), by block coordinate ascent. We reach this conclusion via elementary arguments, closely aligned with the framework we develop in Section 2. The approach we present here is intended for an audience without much of a familiarity with convex duality theory [4].

Figure 1 illustrates that GLASSO is an ascent algorithm on the dual of the problem 1.1. The red curve in the left plot shows the dual objective rising monotonely, and the rightmost plot shows that the increments are indeed positive. There is an added twist though: in solving the block-coordinate update, GLASSO solves instead the dual of that subproblem.

4.1. Dual of the ℓ₁ regularized log-likelihood

We present below the following lemma, the conclusion of which also appears in [2], but we use the framework developed in Section 2.

Lemma 1

Consider the primal problem (1.1) and its stationarity conditions (2.1). These are equivalent to the stationarity conditions for the box-constrained SDP

$$\underset{\tilde{\mathbf{\Gamma }}:{\Vert \tilde{\mathbf{\Gamma }}\Vert }_{\infty }\le \mathrm{\lambda }}{\text{maximize}}g(\tilde{\mathbf{\Gamma }}):=\mathrm{log det}(\mathbf{S}+\tilde{\mathbf{\Gamma }})+p$$ (4.1)

under the transformation $\mathbf{S}+\tilde{\mathbf{\Gamma }}={\mathbf{\Theta }}^{-1}.$

Proof

The (sub)gradient conditions (2.1) can be rewritten as:

$$-{(\mathbf{S}+\mathrm{\lambda }\mathbf{\Gamma })}^{-1}+\mathbf{\Theta }=\mathbf{0}$$ (4.2)

where Γ = sgn(Θ). We write $\tilde{\mathbf{\Gamma }}=\mathrm{\lambda }\mathbf{\Gamma }$ and observe that ${\Vert \tilde{\mathbf{\Gamma }}\Vert }_{\infty }\le \mathrm{\lambda }$. Denote by abs(Θ) the matrix with element-wise absolute values.

Hence if (Θ, Γ) satisfy (4.2), the substitutions

$$\tilde{\mathbf{\Gamma }}=\mathrm{\lambda }\mathbf{\Gamma };\mathbf{P}=\mathrm{abs}(\mathbf{\Theta })$$ (4.3)

satisfy the following set of equations:

$$\begin{array}{rrr}\hfill -{(\mathbf{S}+\tilde{\mathbf{\Gamma }})}^{-1}+\mathbf{P}\ast \mathrm{sgn}(\tilde{\mathbf{\Gamma }})& \hfill =& \hfill \mathbf{0}\\ \hfill \mathbf{P}\ast (\mathrm{abs}(\tilde{\mathbf{\Gamma }})-\mathrm{\lambda }{\mathbf{1}}_p{\mathbf{1}}_p^{\prime })& \hfill =& \hfill \mathbf{0}\\ \hfill \Vert \tilde{\mathbf{\Gamma }}{\Vert }_{\infty }& \hfill \le & \hfill \mathrm{\lambda }.\end{array}$$ (4.4)

In the above, P is a symmetric p × p matrix with non-negative entries, ${\mathbf{1}}_p{\mathbf{1}}_p^{\prime }$ denotes a p×p matrix of ones, and the operator ‘*’ denotes element-wise product. We observe that (4.4) are the KKT optimality conditions for the box-constrained SDP (4.1). Similarly, the transformations $\mathbf{\Theta }=\mathbf{P}\ast \mathrm{sgn}(\tilde{\mathbf{\Gamma }})$ and $\mathbf{\Gamma }=\tilde{\mathbf{\Gamma }}/\mathrm{\lambda }$ show that conditions (4.4) imply condition (4.2). Based on (4.2) the optimal solutions of the two problems (1.1) and (4.1) are related by $\mathbf{S}+\tilde{\mathbf{\Gamma }}={\mathbf{\Theta }}^{-1}$.  □

Notice that for the dual, the optimization variable is $\tilde{\mathbf{\Gamma }}$, with $\mathbf{S}+\tilde{\mathbf{\Gamma }}={\mathbf{\Theta }}^{-1}=\mathbf{W}.$ In other words, the dual problem solves for W rather than Θ, a fact that is suggested by the GLASSO algorithm.

Remark 1

The equivalence of the solutions to problems (4.1) and (1.1) as described above can also be derived via convex duality theory [4], which shows that (4.1) is a dual function of the ℓ₁ regularized negative log-likelihood (1.1). Strong duality holds, hence the optimal solutions of the two problems coincide [2].

We now consider solving (4.4) for the last block ${\tilde{\mathbf{\gamma }}}_{12}$ (excluding diagonal), holding the rest of $\tilde{\mathbf{\Gamma }}$ fixed. The corresponding equations are

$$\begin{array}{rrr}\hfill -{\mathit{\theta }}_{12}+{\mathbf{p}}_{12}\ast \mathrm{sgn}({\tilde{\mathbf{\gamma }}}_{12})& \hfill =& \hfill \mathbf{0}\\ \hfill {\mathbf{p}}_{12}\ast (\mathrm{abs}({\tilde{\mathbf{\gamma }}}_{12})-\mathrm{\lambda }{\mathbf{1}}_{p-1})& \hfill =& \hfill \mathbf{0}\\ \hfill \Vert {\tilde{\mathbf{\gamma }}}_{12}{\Vert }_{\infty }& \hfill \le & \hfill \mathrm{\lambda }.\end{array}$$ (4.5)

The only non-trivial translation is the θ₁₂ in the first equation. We must express this in terms of the optimization variable ${\tilde{\mathbf{\gamma }}}_{12}$. Since ${\mathbf{s}}_{12}+{\tilde{\mathbf{\gamma }}}_{12}={\mathbf{w}}_{12}$, using the identities in (2.5), we have ${\mathbf{W}}_{11}^{-1}({\mathbf{s}}_{12}+{\tilde{\mathbf{\gamma }}}_{12})=-{\mathit{\theta }}_{12}/{\theta }_{22}$. Since θ₂₂ > 0, we can redefine ${\tilde{\mathbf{p}}}_{12}={\mathbf{p}}_{12}/{\theta }_{22}$, to get

$$\begin{array}{rrr}\hfill {\mathbf{W}}_{11}^{-1}({\mathbf{s}}_{12}+{\tilde{\mathbf{\gamma }}}_{12})+{\tilde{\mathbf{p}}}_{12}\ast \mathrm{sgn}({\tilde{\mathbf{\gamma }}}_{12})& \hfill =& \hfill \mathbf{0}\\ \hfill {\tilde{\mathbf{p}}}_{12}\ast (\mathrm{abs}({\tilde{\mathbf{\gamma }}}_{12})-\mathrm{\lambda }{\mathbf{1}}_{p-1})& \hfill =& \hfill \mathbf{0}\\ \hfill \Vert {\tilde{\mathbf{\gamma }}}_{12}{\Vert }_{\infty }& \hfill \ge & \hfill \mathrm{\lambda }.\end{array}$$ (4.6)

The following lemma shows that a block update of GLASSO solves (4.6) (and hence (4.5)), a block of stationary conditions for the dual of the graphical lasso problem. Curiously, GLASSO does this not directly, but by solving the dual of the QP corresponding to this block of equations.

Lemma 2

Assume W₁₁ ≻ 0. The stationarity equations

$${\mathbf{W}}_{11}\widehat{\mathit{\beta }}+{\mathbf{s}}_{12}+\mathrm{\lambda }{\widehat{\mathbf{\gamma }}}_{12}=0,$$ (4.7)

where ${\widehat{\mathbf{\gamma }}}_{12}\in \mathrm{Sign}(\widehat{\mathit{\beta }})$, correspond to the solution of the ℓ₁-regularized QP:

$$\underset{\mathit{\beta }\in {\mathrm{\Re }}^{p-1}}{\text{minimize}}\frac{1}{2}\mathit{\beta }\prime {\mathbf{W}}_{11}\mathit{\beta }+\mathit{\beta }\prime {\mathbf{s}}_{12}+\mathrm{\lambda }{\Vert \mathit{\beta }\Vert }_1.$$ (4.8)

Solving (4.8) is equivalent to solving the following box-constrained QP:

$$\underset{\mathbf{\gamma }\in {\mathrm{\Re }}^{p-1}}{\text{minimize}}\frac{1}{2}({\mathbf{s}}_{12}+\mathbf{\gamma })\prime {\mathbf{W}}_{11}^{-1}({\mathbf{s}}_{12}+\mathbf{\gamma })\text{subject to}{\Vert \mathbf{\gamma }\Vert }_{\infty }\le \mathrm{\lambda },$$ (4.9)

with stationarity conditions given by (4.6), where the $\widehat{\mathit{\beta }}$ and ${\tilde{\mathbf{\gamma }}}_{12}$ are related by

$$\widehat{\mathit{\beta }}=-{\mathbf{W}}_{11}^{-1}({\mathbf{s}}_{12}+{\tilde{\mathbf{\gamma }}}_{12}).$$ (4.10)

Proof

(4.7) is the KKT optimality condition for the ℓ₁ regularized QP (4.8). We rewrite (4.7) as

$$\widehat{\mathit{\beta }}=-{\mathbf{W}}_{11}^{-1}({\mathbf{s}}_{12}+\mathrm{\lambda }{\tilde{\mathbf{\gamma }}}_{12})=0.$$ (4.11)

Observe that ${\widehat{\beta }}_i=\mathrm{sgn}(\widehat{\beta })\left|{\beta }_i\right|\forall i$ and ${\Vert {\widehat{\mathbf{\gamma }}}_{12}\Vert }_{\infty }\le 1$. Suppose $\widehat{\mathit{\beta }},{\widehat{\mathbf{\gamma }}}_{12}$ satisfy (4.11), then the substitutions

$${\tilde{\mathbf{\gamma }}}_{12}=\mathrm{\lambda }{\widehat{\mathbf{\gamma }}}_{12},{\tilde{\mathbf{p}}}_{12}=\mathrm{abs}(\widehat{\mathit{\beta }})$$ (4.12)

in (4.11) satisfy the stationarity conditions (4.6). It turns out that (4.6) is equivalent to the KKT optimality conditions of the box-constrained QP (4.9). Similarly, we note that if ${\tilde{\mathbf{\gamma }}}_{12}$, ${\tilde{\mathbf{p}}}_{12}$ satisfy (4.6), then the substitution

$${\widehat{\mathbf{\gamma }}}_{12}={\tilde{\mathbf{\gamma }}}_{12}/\mathrm{\lambda };\widehat{\mathit{\beta }}={\tilde{\mathbf{p}}}_{12}\ast \mathrm{sgn}({\tilde{\mathbf{\gamma }}}_{12})$$

satisfies (4.11). Hence the $\widehat{\mathit{\beta }}$ and ${\widehat{\mathbf{\gamma }}}_{12}$ are related by (4.10).  □

Remark 2

The above result can also be derived via convex duality theory[4], where (4.9) is actually the Lagrange dual of the ℓ₁ regularized QP (4.8), with (4.10) denoting the primal-dual relationship. [2, Section 3.3] interpret (4.9) as an ℓ₁ penalized regression problem (using convex duality theory) and explore connections with the set up of [8].

Note that the QP (4.9) is a (partial) optimization over the variable w₁₂ only (since s₁₂ is fixed); the sub-matrix W₁₁ remains fixed in the QP. Exactly one row/column of W changes when the block-coordinate algorithm of GLASSO moves to a new row/column, unlike an explicit full matrix update in W₁₁, which is required if θ₁₂ is updated. This again emphasizes that GLASSO is operating on the covariance matrix instead of Θ. We thus arrive at the following conclusion:

Theorem 4.1

GLASSO performs block-coordinate ascent on the box-constrained SDP (4.1), the Lagrange dual of the primal problem (1.1). Each of the block steps are themselves box-constrained QPs, which GLASSO optimizes via their Lagrange duals.

In our annotation perhaps GLASSO should be called DD-GLASSO, since it performs dual block updates for the dual of the graphical lasso problem. Banerjee, Ghaoui and d’Aspremont [2], the paper that inspired the original GLASSO article [5], also operates on the dual. They however solve the block-updates directly (which are box constrained QPs) using interior-point methods.

5. A new algorithm — DP-GLASSO

In Section 3, we described P-GLASSO, a primal coordinate-descent method. For every row/column we need to solve a lasso problem (3.1), which operates on a quadratic form corresponding to the square matrix ${\mathbf{\Theta }}_{11}^{-1}$. There are two problems with this approach:

• the matrix ${\mathbf{\Theta }}_{11}^{-1}$ needs to be constructed at every row/column update with complexity O(p²);

• ${\mathbf{\Theta }}_{11}^{-1}$ is dense.

We now show how a simple modification of the ℓ₁-regularized QP leads to a box-constrained QP with attractive computational properties.

The KKT optimality conditions for (3.1), following (2.11), can be written as:

$${\mathbf{\Theta }}_{11}^{-1}\mathit{\alpha }+{\mathbf{s}}_{12}+\mathrm{\lambda }\mathrm{sgn}(\mathit{\alpha })=0.$$ (5.1)

Algorithm 3 DP-GLASSO algorithm

Initialize Θ = diag(S + λI)−1. Cycle around the columns repeatedly, performing the following steps till convergence: Rearrange the rows/columns so that the target column is last (implicitly). Solve (5.3) for $\tilde{\mathbf{\gamma }}$ and update $${\widehat{\mathit{\theta }}}_{12}=-{\mathbf{\Theta }}_{11}({\mathbf{s}}_{12}+\tilde{\mathbf{\gamma }})/w_{22}$$ Solve for θ22 using (5.5). Update the working covariance ${\mathbf{w}}_{12}={\mathbf{s}}_{12}+\tilde{\mathbf{\gamma }}$.

Along the same lines of the derivations used in Lemma 2, the condition above is equivalent to

$$\begin{array}{r}{\tilde{\mathbf{q}}}_{12}\ast \mathrm{sgn}(\tilde{\mathbf{\gamma }})+{\mathbf{\Theta }}_{11}({\mathbf{s}}_{12}+\tilde{\mathbf{\gamma }})=\mathbf{0}\\ {\tilde{\mathbf{q}}}_{12}\ast (\mathrm{abs}(\tilde{\mathbf{\gamma }})-\mathrm{\lambda }{\mathbf{1}}_{p-1})=0\\ {\Vert \tilde{\mathbf{\gamma }}\Vert }_{\infty }\le \mathrm{\lambda }\end{array}$$ (5.2)

for some vector (with non-negative entries) ${\tilde{\mathbf{q}}}_{12}$. (5.2) are the KKT optimality conditions for the following box-constrained QP:

$$\underset{\mathbf{\gamma }\in {\mathrm{\Re }}^{p-1}}{\text{minimize}}\frac{1}{2}({\mathbf{s}}_{12}+\mathbf{\gamma })\prime {\mathbf{\Theta }}_{11}({\mathbf{s}}_{12}+\mathbf{\gamma })\text{subject to}{\Vert \mathbf{\gamma }\Vert }_{\infty }\le \mathrm{\lambda }.$$ (5.3)

The optimal solutions of (5.3) and (5.1) are related by

$$\widehat{\mathit{\alpha }}=-{\mathbf{\Theta }}_{11}({\mathbf{s}}_{12}+\tilde{\mathbf{\gamma }}),$$ (5.4)

a consequence of (5.1), with $\widehat{\mathit{\alpha }}={\widehat{\mathit{\theta }}}_{12}·w_{22}$ and w₂₂ = s₂₂ + λ. The diagonal θ₂₂ of the precision matrix is updated via (2.6):

$${\widehat{\theta }}_{22}=\frac{1-({\mathbf{s}}_{12}+\tilde{\mathbf{\gamma }})\prime {\widehat{\mathit{\theta }}}_{12}}{w_{22}}$$ (5.5)

By strong duality, the box-constrained QP (5.3) with its optimality conditions (5.2) is equivalent to the lasso problem (3.1). Now both the problems listed at the beginning of the section are removed. The problem matrix Θ₁₁ is sparse, and no O(p²) updating is required after each block.

The solutions returned at step 2(b) for ${\widehat{\mathit{\theta }}}_{12}$ need not be exactly sparse, even though it purports to produce the solution to the primal block problem (3.1), which is sparse. One needs to use a tight convergence criterion when solving (5.3). In addition, one can threshold those elements of ${\widehat{\mathit{\theta }}}_{12}$ for which $\tilde{\mathbf{\gamma }}$ is away from the box boundary, since those values are known to be zero.

Note that DP-GLASSO does to the primal formulation (1.1) what GLASSO does to the dual. DP-GLASSO operates on the precision matrix, whereas GLASSO operates on the covariance matrix.

6. Computational costs in solving the block QPs

The ℓ₁ regularized QPs appearing in (2.13) and (3.1) are of the generic form

$$\underset{\mathbf{u}\in {\mathrm{\Re }}^q}{\text{minimize}}\frac{1}{2}\mathbf{u}\prime \mathbf{Au}+\mathbf{a}\prime \mathbf{u}+\mathrm{\lambda }{\Vert \mathbf{u}\Vert }_1,$$ (6.1)

for A ≻ 0. In this paper, we choose to use cyclical coordinate descent for solving (6.1), as it is used in the GLASSO algorithm implementation of Friedman, Hastie and Tibshirani [5]. Moreover, cyclical coordinate descent methods perform well with good warm-starts. These are available for both (2.13) and (3.1), since they both maintain working copies of the precision matrix, updated after every row/column update. There are other efficient ways for solving (6.1), capable of scaling to large problems — for example first-order proximal methods [3, 9], but we do not pursue them in this paper.

The box-constrained QPs appearing in (4.9) and (5.3) are of the generic form:

$$\underset{\mathbf{v}\in {\mathrm{\Re }}^q}{\text{minimize}}\frac{1}{2}(\mathbf{v}+\mathbf{b})\prime \tilde{\mathbf{A}}(\mathbf{v}+\mathbf{b})\text{subject to}{\Vert \mathbf{v}\Vert }_{\infty }\le \mathrm{\lambda }$$ (6.2)

for some à ≻ 0. As in the case above, we will use cyclical coordinate-descent for optimizing (6.2).

In general it is more efficient to solve (6.1) than (6.2) for larger values of λ. This is because a large value of λ in (6.1) results in sparse solutions û; the coordinate descent algorithm can easily detect when a zero stays zero, and no further work gets done for that coordinate on that pass. If the solution to (6.1) has κ non-zeros, then on average κ coordinates need to be updated. This leads to a cost of O(qκ), for one full sweep across all the q coordinates.

On the other hand, a large λ for (6.2) corresponds to a weakly-regularized solution. Cyclical coordinate procedures for this task are not as effective. Every coordinate update of v results in updating the gradient, which requires adding a scalar multiple of a column of Ã. If à is dense, this leads to a cost of O(q), and for one full cycle across all the coordinates this costs O(q²), rather than the O(qκ) for (6.1).

However, our experimental results show that DP-GLASSO is more efficient than GLASSO, so there are some other factors in play. When à is sparse, there are computational savings. If à has κq non-zeros, the cost per column reduces on average to O(κq) from O(q²). For the formulation (5.3) à is Θ₁₁, which is sparse for large λ. Hence for large λ, GLASSO and DP-GLASSO have similar costs.

For smaller values of λ, the box-constrained QP (6.2) is particularly attractive. Most of the coordinates in the optimal solution $\widehat{\mathbf{v}}$ will pile up at the boundary points {−λ, λ}, which means that the coordinates need not be updated frequently. For problem (5.3) this number is also κ, the number of non-zero coefficients in the corresponding column of the precision matrix. If κ of the coordinates pile up at the boundary, then one full sweep of cyclical coordinate descent across all the coordinates will require updating gradients corresponding to the remaining q −κ coordinates. Using similar calculations as before, this will cost O(q(q − κ)) operations per full cycle (since for small λ, Ã will be dense). For the ℓ₁ regularized problem (6.1), no such saving is achieved, and the cost is O(q²) per cycle.

Note that to solve problem (1.1), we need to solve a QP of a particular type (6.1) or (6.2) for a certain number of outer cycles (ie full sweeps across rows/columns). For every row/column update, the associated QP requires varying number of iterations to converge. It is hard to characterize all these factors and come up with precise estimates of convergence rates of the overall algorithm. However, we have observed that with warm-starts, on a relatively dense grid of λs, the complexities given above are pretty much accurate for DP-GLASSO (with warmstarts) specially when one is interested in solutions with small / moderate accuracy. Our experimental results in Section 9.1 and Appendix Section B support our observation.

We will now have a more critical look at the updates of the GLASSO algorithm and study their properties.

7. GLASSO: Positive definiteness, sparsity and exact inversion

As noted earlier, GLASSO operates on W — it does not explicitly compute the inverse W⁻¹. It does however keep track of the estimates for θ₁₂ after every row/column update. The copy of Θ retained by GLASSO along the row/column updates is not the exact inverse of the optimization variable W. Figure 2 illustrates this by plotting the squared-norm $\Vert (\mathbf{\Theta }-{\mathbf{W}}^{-1}){\Vert }_F^2$ as a function of the iteration index. Only upon (asymptotic) convergence, will Θ be equal to W⁻¹. This can have important consequences.

FIG 2.

Figure illustrating some negative properties of GLASSO using a typical numerical example. [Left Panel] The precision matrix produced after every row/column update need not be the exact inverse of the working covariance matrix — the squared Frobenius norm of the error is being plotted across iterations. [Right Panel] The estimated precision matrix Θ produced by GLASSO need not be positive definite along iterations; plot shows minimal eigen-value.

In many real-life problems one only needs an approximate solution to (1.1):

• for computational reasons it might be impractical to obtain a solution of high accuracy;

• from a statistical viewpoint it might be sufficient to obtain an approximate solution for Θ that is both sparse and positive definite

It turns out that the GLASSO algorithm is not suited to this purpose.

Since the GLASSO is a block coordinate procedure on the covariance matrix, it maintains a positive definite covariance matrix at every row/column update. However, since the estimated precision matrix is not the exact inverse of W, it need not be positive definite. Although it is relatively straightforward to maintain an exact inverse of W along the row/column updates (via simple rank-one updates as before), this inverse W⁻¹ need not be sparse. Arbitrary thresholding rules may be used to set some of the entries to zero, but that might destroy the positive-definiteness of the matrix. Since a principal motivation of solving (1.1) is to obtain a sparse precision matrix (which is also positive definite), returning a dense W⁻¹ to (1.1) is not desirable.

Figures 2 illustrates the above observations on a typical example.

The DP-GLASSO algorithm operates on the primal (1.1). Instead of optimizing the ℓ₁ regularized QP (3.1), which requires computing ${\mathbf{\Theta }}_{11}^{-1}$, DP-GLASSO optimizes (5.3). After every row/column update the precision matrix Θ is positive definite. The working covariance matrix maintained by DP-GLASSO via ${\mathbf{w}}_{12}:={\mathbf{s}}_{12}+\widehat{\mathbf{\gamma }}$ need not be the exact inverse of Θ. Exact covariance matrix estimates, if required, can be obtained by tracking Θ⁻¹ via simple rank-one updates, as described earlier.

Unlike GLASSO, DP- GLASSO (and P- GLASSO) return a sparse and positive definite precision matrix even if the row/column iterations are terminated prematurely.

8. Warm starts and path-seeking strategies

Since we seldom know in advance a good value of λ, we often compute a sequence of solutions to (1.1) for a (typically) decreasing sequence of values λ₁ > λ₂ > ⋯ > λ_K. Warm-start or continuation methods use the solution at λ_i+1, as an initial guess for the solution at λ_i+1, and often yield great efficiency. It turns out that for algorithms like GLASSO which operate on the dual problem, not all warm-starts necessarily lead to a convergent algorithm. We address this aspect in detail in this section.

The following lemma states the conditions under which the row/column updates of the GLASSO algorithm will maintain positive definiteness of the covariance matrix W.

Lemma 3

Suppose Z is used as a warm-start for the GLASSO algorithm. If Z ≻ 0 and ‖Z−S‖_∞ ≤ λ, then every row/column update of GLASSO maintains positive definiteness of the working covariance matrix W.

Proof

Recall that the GLASSO solves the dual (4.1). Assume Z is partitioned as in (2.4), and the pth row/column is being updated. Since Z ≻ 0, we have both

$${\mathbf{Z}}_{11}\succ \mathbf{0}\text{and}(z_{22}-{\mathbf{z}}_{21}{({\mathbf{Z}}_{11})}^{-1}{\mathbf{z}}_{12})>0.$$ (8.1)

Since Z₁₁ remains fixed, it suffices to show that after the row/column update, the expression $({\widehat{w}}_{22}-{\widehat{\mathbf{w}}}_{21}{({\mathbf{Z}}_{11})}^{-1}{\widehat{\mathbf{w}}}_{12})$ remains positive. Recall that, via standard optimality conditions we have ${\widehat{w}}_{22}=s_{22}+\mathrm{\lambda }$, which makes ${\widehat{w}}_{22}\ge z_{22}$ (since by assumption, |z₂₂−s₂₂| ≤ λ and z₂₂ > 0). Furthermore, ${\widehat{\mathbf{w}}}_{21}={\mathbf{s}}_{21}+\widehat{\mathbf{\gamma }}$, where $\widehat{\mathbf{\gamma }}$ is the optimal solution to the corresponding box-QP (4.9). Since the starting solution z₂₁ satisfies the box-constraint (4.9) ‖z₂₁−s₂₁‖_∞ ≤ λ, the optimal solution of the QP (4.9) improves the objective:

$${\widehat{\mathbf{w}}}_{21}{({\mathbf{Z}}_{11})}^{-1}{\widehat{\mathbf{w}}}_{12}\le {\mathbf{z}}_{21}{({\mathbf{Z}}_{11})}^{-1}{\mathbf{z}}_{12}$$

Combining the above along with the fact that ${\widehat{w}}_{22}\ge z_{22}$ we see

$${\widehat{w}}_{22}-{\widehat{\mathbf{w}}}_{21}{({\mathbf{Z}}_{11})}^{-1}{\widehat{\mathbf{w}}}_{12}>0,$$ (8.2)

which implies that the new covariance estimate $\widehat{\mathbf{W}}\succ \mathbf{0}.$  □

Remark 3

If the condition ‖Z−S‖_∞ ≤ λ appearing in Lemma 3 is violated, then the row/column update of GLASSO need not maintain PD of the covariance matrix W.

We have encountered many counter-examples that show this to be true, see the discussion below.

The R package implementation of GLASSO allows the user to specify a warm-start as a tuple (Θ₀, W₀). This option is typically used in the construction of a path algorithm.

If $({\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }},{\widehat{\mathbf{W}}}_{\mathrm{\lambda }})$ is provided as a warm-start for λ′ < λ, then the GLASSO algorithm is not guaranteed to converge. It is easy to find numerical examples by choosing the gap λ − λ′ to be large enough. Among the various examples we encountered, we briefly describe one here. Details of the experiment/data and other examples can be found in the Appendix A.1. We generated a data-matrix X_n×p, with n = 2, p = 5 with iid standard Gaussian entries. S is the sample covariance matrix. We solved problem (1.1) using GLASSO for λ = 0.9 × max_i≠j|s_ij|. We took the estimated covariance and precision matrices: ${\widehat{\mathbf{W}}}_{\mathrm{\lambda }}\text{and}{\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}$ as a warm-start for the GLASSO algorithm with λ′ = λ× 0.01. The GLASSO algorithm failed to converge with this warm-start. We note that ${\Vert {\widehat{\mathbf{W}}}_{\mathrm{\lambda }}-\mathbf{S}\Vert }_{\infty }=0.0402\nleqq \mathrm{\lambda }\prime$ (hence violating the sufficient condition in Lemma 4) and after updating the first row/column via the GLASSO algorithm we observed that “covariance matrix” W has negative eigen-values — leading to a non-convergent algorithm. The above phenomenon is not surprising and easy to explain and generalize. If the warm-start fails to satisfy ${\Vert {\widehat{\mathbf{W}}}_{\mathrm{\lambda }}-\mathbf{S}\Vert }_{\infty }$ then during the course of the row/column updates the working covariance matrix may lose positive definiteness. In such a case, the block problems (QPs) may not correspond to valid convex programs (due to the lack of the postive-definiteness of the quadratic forms). This seems to be the fundamental reason behind the non-convergence of the algorithm. Since ${\widehat{\mathbf{W}}}_{\mathrm{\lambda }}$ solves the dual (4.1), it is necessarily of the form ${\widehat{\mathbf{W}}}_{\mathrm{\lambda }}=\mathbf{S}+\tilde{\mathbf{\Gamma }},\text{for}{\Vert \tilde{\mathbf{\Gamma }}\Vert }_{\infty }\le \mathrm{\lambda }.$ In the light of Lemma 3 and also Remark 3, the warm-start needs to be dual-feasible in order to guarantee that the iterates $\widehat{\mathbf{W}}$ remain PD and hence for the sub-problems to be well defined convex programs. Clearly ${\widehat{\mathbf{W}}}_{\mathrm{\lambda }}$ does not satisfy the box-constraint ${\Vert {\widehat{\mathbf{W}}}_{\mathrm{\lambda }}-\mathbf{S}\Vert }_{\infty }\le \mathrm{\lambda }\prime ,$ for λ′ < λ However, in practice the GLASSO algorithm is usually seen to converge (numerically) when λ′ is quite close to λ. This is probably because the working covariance matrix remains positive definite and the block QPs are valid convex programs. If the difference between λ′ and λ is large then the algorithm may very likely get into trouble.

The following lemma establishes that any PD matrix can be taken as a warm-start for P-GLASSO or DP- GLASSOto ensure a convergent algorithm.

Lemma 4

Suppose Φ ≻ 0 is a used as a warm-start for the P-GLASSO (or DP-GLASSO) algorithm. Then every row/column update of P-GLASSO (or DP-GLASSO) maintains positive definiteness of the working precision matrix Θ.

Proof

Consider updating the pth row/column of the precision matrix. The condition Φ ≻ 0 is equivalent to both

$${\mathbf{\Phi }}_{11}\succ \mathbf{0}\text{and}({\varphi }_{22}-{\mathbf{\Phi }}_{21}{({\mathbf{\Phi }}_{11})}^{-1}{\mathbf{\Phi }}_{12})>0.$$

Note that the block Φ₁₁ remains fixed; only the pth row/column of Θ changes ϕ₂₁. gets updated to ${\widehat{\mathit{\theta }}}_{21}$, as does ${\widehat{\mathit{\theta }}}_{12}$. From (2.6) the updated diagonal entry ${\widehat{\theta }}_{22}$ satisfies:

$${\widehat{\theta }}_{22}-{\widehat{\mathit{\theta }}}_{21}{({\mathbf{\Phi }}_{11})}^{-1}{\widehat{\mathit{\theta }}}_{12}=\frac{1}{(s_{22}+\mathrm{\lambda })}>0.$$

Thus the updated matrix $\widehat{\mathbf{\Theta }}$ remains PD. The result for the DP-GLASSO algorithm follows, since both the versions P-GLASSO and DP-GLASSO solve the same block coordinate problem.  □

Remark 4

A simple consequence of Lemmas 3 and 4 is that the QPs arising in the process, namely the ℓ₁ regularized QPs (2.13), (3.1) and the box-constrained QPs (4.9) and (5.3) are all valid convex programs, since all the respective matrices ${\mathbf{W}}_{11},{\mathbf{\Theta }}_{11}^{-1}\text{and}{\mathbf{W}}_{11}^{-1},{\mathbf{\Theta }}_{11}$ appearing in the quadratic forms are PD.

As exhibited in Lemma 4, both the algorithms DP-GLASSO and P-GLASSO are guaranteed to converge from any positive-definite warm start. This is due to the unconstrained formulation of the primal problem (1.1).

GLASSO really only requires an initialization for W, since it constructs Θ on the fly. Likewise DP-GLASSO only requires an initialization for Θ. Having the other half of the tuple assists in the block-updating algorithms. For example, GLASSO solves a series of lasso problems, where Θ play the role as parameters. By supplying Θ along with W, the block-wise lasso problems can be given starting values close to the solutions. The same applies to DP-GLASSO. In neither case do the pairs have to be inverses of each other to serve this purpose.

If we wish to start with inverse pairs, and maintain such a relationship, we have described earlier how O(p²) updates after each block optimization can achieve this. One caveat for GLASSO is that starting with an inverse pair costs O(p³) operations, since we typically start with W = S + λI. For DP-GLASSO, we typically start with a diagonal matrix, which is trivial to invert.

9. Experimental results & timing comparisons

We compared the performances of algorithms GLASSO and DP-GLASSO (both with and without warm-starts) on different examples with varying (n, p) values. While most of the results are presented in this section, some are relegated to the Appendix B. Section 9.1 describes some synthetic examples and Section 9.2 presents comparisons on a real-life micro-array data-set.

9.1. Synthetic experiments

In this section we present examples generated from two different covariance models — as characterized by the covariance matrix Σ or equivalently the precision matrix Θ. We create a data matrix X_n×p by drawing n independent samples from a p dimensional normal distribution MVN(0, Σ). The sample covariance matrix is taken as the input S to problem (1.1). The two covariance models are described below:

• Type-1 The population concentration matrix Θ = Σ⁻¹ has uniform sparsity with approximately 77 % of the entries zero.

  We created the covariance matrix as follows. We generated a matrix B with iid standard Gaussian entries, symmetrized it via $\frac{1}{2}(\mathbf{B}+{\mathbf{B}}^{\prime })$ and set approximately 77% of the entries of this matrix to zero, to obtain $\tilde{\mathbf{B}}$ (say). We added a scalar multiple of the p dimensional identity matrix to $\tilde{\mathbf{B}}$ to get the precision matrix $\mathbf{\Theta }=\tilde{\mathbf{B}}+\eta {\mathbf{I}}_{p\times p},$ with η chosen such that the minimum eigen value of Θ is one.

• Type-2 This example, taken from [11], is an auto-regressive process of order two — the precision matrix being tri-diagonal:

  $${\theta }_{\mathit{ij}}=\{\begin{array}{ll}0.5,\hfill & \text{if}|j-i|=1,i=2,\dots ,(p-1);\hfill \\ 0.25,\hfill & \text{if}|j-i|=2,i=3,\dots ,(p-2);\hfill \\ 1,\hfill & \text{if}i=j,i=1,\dots ,p;\text{and}\hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}$$

For each of the two set-ups Type-1 and Type-2 we consider twelve different combinations of (n, p):

1. p = 1000, n ∈ {1500, 1000, 500}.

2. p = 800, n ∈ {1000, 800, 500}.

3. p = 500, n ∈ {800, 500, 200}.

4. p = 200, n ∈ {500, 200, 50}.

For every (n, p) we solved (1.1) on a grid of twenty λ values linearly spaced in the log-scale, with λ_i = 0.8ⁱ × {0.9λ_max}, i = 1, ⋯, 20, where λ_max = max_i≠j |s_ij|, is the off-diagonal entry of S with largest absolute value. λ_max is the smallest value of λ for which the solution to (1.1) is a diagonal matrix.

Since this article focuses on the GLASSO algorithm, its properties and alternatives that stem from the main idea of block-coordinate optimization, we present here the performances of the following algorithms:

• Dual-Cold GLASSO with initialization W = S + λI_p×p, as suggested in [5].

• Dual-Warm The path-wise version of GLASSO with warm-starts, as suggested in [5]. Although this path-wise version need not converge in general, this was not a problem in our experiments, probably due to the fine-grid of λ values.

• Primal-Cold DP-GLASSO with diagonal initialization Θ = (diag(_S) + λ_I)⁻¹.

• Primal-Warm The path-wise version of DP-GLASSO with warm-starts.

We did not include P-GLASSO in the comparisons above since P-GLASSO requires additional matrix rank-one updates after every row/column update, which makes it more expensive. None of the above listed algorithms require matrix inversions (via rank one updates). Furthermore, DP-GLASSO and P-GLASSO are quite similar as both are doing a block coordinate optimization on the dual. Hence we only included DP-GLASSO in our comparisons. We used our own implementation of the GLASSO and DP-GLASSO algorithm in R. The entire program is written in R, except the inner block-update solvers, which are the real work-horses:

• For GLASSO we used the lasso code crossProdLasso written in FORTRAN by [5];

• For DP-GLASSO we wrote our own FORTRAN code to solve the box QP.

An R package dpglasso that implements DP-GLASSO is available on CRAN.

In the figure and tables that follow below, for every algorithm, at a fixed λ we report the total time taken by all the QPs — the ℓ₁ regularized QP for GLASSO and the box constrained QP for DP-GLASSO till convergence All computations were done on a Linux machine with model specs: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz.

Convergence Criterion

Since DP-GLASSO operates on the the primal formulation and GLASSO operates on the dual — to make the convergence criteria comparable across examples we based it on the relative change in the primal objective values i.e. f (Θ) (1.1) across two successive iterations:

$$\frac{f({\mathbf{\Theta }}_k)-f({\mathbf{\Theta }}_{k-1})}{|f({\mathbf{\Theta }}_{k-1})|}\le \mathrm{TOL},$$ (9.1)

where one iteration refers to a full sweep across p rows/columns of the precision matrix (for DP-GLASSO) and covariance matrix (for GLASSO); and TOL denotes the tolerance level or level of accuracy of the solution. To compute the primal objective value for the GLASSO algorithm, the precision matrix is computed from $\widehat{\mathbf{W}}$ via direct inversion (the time taken for inversion and objective value computation is not included in the timing comparisons).

Computing the objective function is quite expensive relative to the computational cost of the iterations. In our experience convergence criteria based on a relative change in the precision matrix for DP-GLASSO and the covariance matrix for GLASSO seemed to be a practical choice for the examples we considered. However, for reasons we described above, we used criterion 9.1 in the experiments.

Observations

Figure 3 presents the times taken by the algorithms to converge to an accuracy of TOL = 10⁻⁴ on a grid of λ values.

FIG 3.

The timings in seconds for the four different algorithmic versions: GLASSO (with and without warm-starts) and DP-GLASSO (with and without warm-starts) for a grid of λ values on the log-scale. [Left Panel] Covariance model for Type-1, [Right Panel] Covariance model for Type-2. The horizontal axis is indexed by the proportion of zeros in the solution. The vertical dashed lines correspond to the optimal λ values for which the estimated errors ${\Vert {\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}-\mathbf{\Theta }\Vert }_1$ (green) and ${\Vert {\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}-\mathbf{\Theta }\Vert }_F$ (blue) are minimum.

The figure shows eight different scenarios with p > n, corresponding to the two different covariance models Type-1 (left panel) and Type-2 (right panel). It is quite evident that DP-GLASSO with warm-starts (Primal-Warm) outperforms all the other algorithms across all the different examples. All the algorithms converge quickly for large values of λ (typically high sparsity) and become slower with decreasing λ. For large p and small λ, convergence is slow; however for p > n, the non-sparse end of the regularization path is really not that interesting from a statistical viewpoint. Warm-starts apparently do not always help in speeding up the convergence of GLASSO; for example see Figure 3 with (n, p) = (500, 1000) (Type 1) and (n, p) = (500, 800) (Type 2). This probably further validates the fact that warm-starts in the case of GLASSO need to be carefully designed, in order for them to speed-up convergence. Note however, that GLASSO with the warm-starts prescribed is not even guaranteed to converge — we however did not come across any such instance among the experiments presented in this section.

Based on the suggestion of a referee we annotated the plots in Figure 3 with locations in the regularization path that are of interest. For each plot, two vertical dotted lines are drawn which correspond to the λs at which the distance of the estimated precision matrix ${\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}$ from the population precision matrix is minimized wrt to the ‖·‖₁ norm (green) and ‖·‖_F norm (blue). The optimal λ corresponding to the ‖·‖₁ metric chooses sparser models than those chosen by ‖·‖_F; the performance gains achieved by DP-GLASSO seem to be more prominent for the latter λ.

Table 1 presents the timings for all the four algorithmic variants on the twelve different (n, p) combinations listed above for Type 1. For every example, we report the total time till convergence on a grid of twenty λ values for two different tolerance levels: TOL ∈ {10⁻⁴, 10⁻⁵}. Note that the DP-GLASSO returns positive definite and sparse precision matrices even if the algorithm is terminated at a relatively small/moderate accuracy level — this is not the case in GLASSO. The rightmost column presents the proportion of non-zeros averaged across the entire path of solutions ${\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}$, where ${\widehat{\mathbf{\Theta }}}_{\mathrm{\lambda }}$ is obtained by solving (1.1) to a high precision i.e. 10⁻⁶, by algorithms GLASSO and DP-GLASSO and averaging the results.

TABLE 1.

Table showing the performances of the four algorithms GLASSO (Dual-Warm/Cold) and DP-GLASSO (Primal-Warm/Cold) for the covariance model Type-1. We present the times (in seconds) required to compute a path of solutions to (1.1) (on a grid of twenty λ values) for different (n,p) combinations and relative errors (as in (9.1)). The rightmost column gives the averaged sparsity level across the grid of λ values. DP-GLASSO with warm-starts is consistently the winner across all the examples

p/n	relative error (TOL)	Total time (secs) to compute a path of solutions				Average % Zeros in path
		Dual-Cold	Dual-Warm	Primal-Cold	Primal-Warm
1000 / 500	10−4 10−5	3550.71 4706.22	6592.63 8835.59	2558.83 3234.97	2005.25 2832.15	80.2
1000 / 1000	10−4 10−5	2788.30 3597.21	3158.71 4232.92	2206.95 2710.34	1347.05 1865.57	83.0
1000 / 1500	10−4 10−5	2447.19 2764.23	4505.02 6426.49	1813.61 2199.53	932.34 1382.64	85.6
800 / 500	10−4 10−5	1216.30 1776.72	2284.56 3010.15	928.37 1173.76	541.66 798.93	78.8
800 / 800	10−4 10−5	1135.73 1481.36	1049.16 1397.25	788.12 986.19	438.46 614.98	80.0
800 / 1000	10−4 10−5	1129.01 1430.77	1146.63 1618.41	786.02 992.13	453.06 642.90	80.2
500 / 200	10−4 10−5	605.45 811.58	559.14 795.43	395.11 520.98	191.88 282.65	75.9
500 / 500	10−4 10−5	427.85 551.11	241.90 315.86	252.83 319.89	123.35 182.81	75.2
500 / 800	10−4 10−5	359.78 416.87	279.67 402.61	207.28 257.06	111.92 157.13	80.9
200 / 50	10−4 10−5	65.87 92.04	50.99 75.06	37.40 45.88	23.32 35.81	75.6
200 / 200	10−4 10−5	35.29 45.90	25.70 33.23	17.32 22.41	11.72 17.16	66.8
200 / 300	10−4 10−5	32.29 38.37	23.60 33.95	16.30 20.12	10.77 15.12	66.0

Again we see that in all the examples DP-GLASSO with warm-starts is the clear winner among its competitors. For a fixed p, the total time to trace out the path generally decreases with increasing n. There is no clear winner between GLASSO with warm-starts and GLASSO without warm-starts. It is often seen that DP-GLASSO without warm-starts converges faster than both the variants of GLASSO (with and without warm-starts).

Table 2 reports the timing comparisons for Type 2. Once again we see that in all the examples Primal-Warm turns out to be the clear winner.

TABLE 2.

Table showing comparative timings of the four algorithmic variants of glasso and DP-GLASSO for the covariance model in Type-2. This table is similar to Table 1, displaying results for Type-1. DP-GLASSO with warm-starts consistently outperforms all its competitors

p/n	relative error (TOL)	Total time (secs) to compute a path of solutions				Average % Zeros in path
		Dual-Cold	Dual-Warm	Primal-Cold	Primal-Warm
1000 / 500	10−4 10−5	6093.11 7707.24	5483.03 7923.80	3495.67 4401.28	1661.93 2358.08	75.6
1000 / 1000	10−4 10−5	4773.98 6054.21	3582.28 4714.80	2697.38 3444.79	1015.84 1593.54	76.70
1000 / 1500	10−4 10−5	4786.28 6171.01	5175.16 6958.29	2693.39 3432.33	1062.06 1679.16	78.5
800 / 500	10−4 10−5	2914.63 3674.73	3466.49 4572.97	1685.41 2083.20	1293.18 1893.22	74.3
800 / 800	10−4 10−5	2021.55 2521.06	1995.90 2639.62	1131.35 1415.95	618.06 922.93	74.4
800 / 1000	10−4 10−5	3674.36 4599.59	2551.06 3353.78	1834.86 2260.58	885.79 1353.28	75.9
500 / 200	10−4 10−5	1200.24 1574.62	885.76 1219.12	718.75 876.45	291.61 408.41	70.5
500 / 500	10−4 10−5	575.53 730.54	386.20 535.58	323.30 421.91	130.59 193.08	72.2
500 / 800	10−4 10−5	666.75 852.54	474.12 659.58	373.60 485.47	115.75 185.60	73.7
200 / 50	10−4 10−5	110.18 142.77	98.23 133.67	48.98 55.27	26.97 33.95	73.0
200 / 200	10−4 10−5	50.63 66.63	40.68 56.71	23.94 31.57	9.97 14.70	63.7
200 / 300	10−4 10−5	47.63 60.98	36.18 50.52	21.24 27.41	8.19 12.22	65.0

For n ≤ p = 1000, we observe that Primal-Warm is generally faster for Type-2 than Type-1. This however, is reversed for smaller values of p ∈ {800, 500}. Primal-Cold is has a smaller overall computation time for Type-1 over Type-2. In some cases (for example n ≤ p = 1000), we see that Primal-Warm in Type-2 converges much faster than its competitors on a relative scale than in Type-1 — this difference is due to the variations in the structure of the covariance matrix.

9.2. Micro-array example

We consider the data-set introduced in [1] and further studied in [10, 7]. In this experiment, tissue samples were analyzed using an Affymetrix Oligonucleotide array. The data was processed, filtered and reduced to a subset of 2000 gene expression values. The number of Colon Adenocarcinoma tissue samples is n = 62. For the purpose of the experiments presented in this section, we pre-screened the genes to a size of p = 725. We obtained this subset of genes using the idea of exact covariance thresholding introduced in our paper [7]. We thresholded the sample correlation matrix obtained from the 62 × 2000 microarray data-matrix into connected components with a threshold of 0.00364¹ — the genes belonging to the largest connected component formed our pre-screened gene pool of size p = 725. This (subset) data-matrix of size (n, p) = (62, 725) is used for our experiments.

The results presented below in Table 3 show timing comparisons of the four different algorithms: Primal-Warm/Cold and Dual-Warm/Cold on a grid of fifteen λ values in the log-scale. Once again we see that the Primal-Warm outperforms the others in terms of speed and accuracy. Dual-Warm performs quite well in this example.

TABLE 3.

Comparisons among algorithms for a microarray dataset with n = 62 and p = 725, for different tolerance levels (TOL). We took a grid of fifteen λ values, the average % of zeros along the whole path is 90.8

relative error (TOL)	Total time (secs) to compute a path of solutions
	Dual-Cold	Dual-Warm	Primal-Cold	Primal-Warm
10−3	515.15	406.57	462.58	334.56
10−4	976.16	677.76	709.83	521.44

10. Conclusions

This paper explores some of the apparent mysteries in the behavior of the GLASSO algorithm introduced in [5]. These have been explained by leveraging the fact that the GLASSO algorithm is solving the dual of the graphical lasso problem (1.1), by block coordinate ascent. Each block update, itself the solution to a convex program, is solved via its own dual, which is equivalent to a lasso problem. The optimization variable is W, the covariance matrix, rather than the target precision matrix Θ. During the course of the iterations, a working version of Θ is maintained, but it may not be positive definite, and its inverse is not W. Tight convergence is therefore essential, for the solution $\widehat{\mathbf{\Theta }}$ to be a proper inverse covariance. There are issues using warm starts with GLASSO, when computing a path of solutions. Unless the sequence of λs are sufficiently close, since the “warm start”s are not dual feasible, the algorithm can get into trouble.

We have also developed two primal algorithms P-GLASSO and DP-GLASSO. The former is more expensive, since it maintains the relationship W = Θ⁻¹ at every step, an O(p³) operation per sweep across all row/columns. DP-GLASSO is similar in flavor to GLASSO except its optimization variable is Θ. It also solves the dual problem when computing its block update, in this case a box-QP. This box-QP has attractive sparsity properties at both ends of the regularization path, as evidenced in some of our experiments. It maintains a positive definite Θ throughout its iterations, and can be started at any positive definite matrix. Our experiments show in addition that DP-GLASSO is faster than GLASSO.

An R package dpglasso that implements DP-GLASSO is available on CRAN.

Appendix A: Additional numerical illustrations and examples

This section complements the examples provided in the paper with further experiments and illustrations.

A.1. Examples: Non-convergence of GLASSO with warm-starts

This section illustrates with examples that warm-starts for the GLASSO need not converge. This is a continuation of examples presented in Section 8.

Example 1

We took (n, p) = (2, 5) and setting the seed of the random number generator in R as set.seed(2008) we generated a data-matrix X_n×p with iid standard Gaussian entries. The sample covariance matrix S is given below:

$$\left(\begin{array}{ccccc}\hfill 0.03597652\hfill & \hfill 0.03792221\hfill & \hfill 0.1058585\hfill & \hfill -0.08360659\hfill & \hfill 0.1366725\hfill \\ \hfill 0.03597652\hfill & \hfill 0.03792221\hfill & \hfill 0.1058585\hfill & \hfill -0.08360659\hfill & \hfill 0.1366725\hfill \\ \hfill 0.10585853\hfill & \hfill 0.11158361\hfill & \hfill 0.3114814\hfill & \hfill -0.24600689\hfill & \hfill 0.4021497\hfill \\ \hfill -0.08360659\hfill & \hfill -0.08812823\hfill & \hfill -0.2460069\hfill & \hfill 0.19429514\hfill & \hfill -0.3176160\hfill \\ \hfill 0.13667246\hfill & \hfill 0.14406402\hfill & \hfill 0.4021497\hfill & \hfill -0.31761603\hfill & \hfill 0.5192098\hfill \end{array}\right)$$

With q denoting the maximum off-diagonal entry of S (in absolute value), we solved (1.1) using GLASSO at λ = 0.9 × q. The covariance matrix for this λ was taken as a warm-start for the GLASSO algorithm with λ′ = λ × 0.01. The smallest eigen-value of the working covariance matrix W produced by the GLASSO algorithm, upon updating the first row/column was: −0.002896128, which is clearly undesirable for the convergence of the algorithm GLASSO. This is why the algorithm GLASSO breaks down.

Example 2

The example is similar to above, with (n, p) = (10, 50), the seed of random number generator in R being set to set.seed(2008) and X_n×p is the data-matrix with iid Gaussian entries. If the covariance matrix ${\widehat{\mathbf{W}}}_{\mathrm{\lambda }}$ which solves problem (1.1) with λ = 0.9 × max_i≠j |s_ij| is taken as a warm-start to the GLASSO algorithm with λ′ = λ×0.1 — the algorithm fails to converge. Like the previous example, after the first row/column update, the working covariance matrix has negative eigen-values.

Appendix B: More examples and comparisons

This section is a continuation to Section 9, in that it provides further examples comparing the performance of algorithms GLASSO and DP-GLASSO. The experimental data is generated as follows. For a fixed value of p, we generate a matrix A_p×p with random Gaussian entries. The matrix is symmetrized by A ← (A + A′)/2. Approximately half of the off-diagonal entries of the matrix are set to zero, uniformly at random. All the eigen-values of the matrix A are lifted so that the smallest eigen-value is zero. The noiseless version of the precision matrix is given by Θ = A + τI_p×p. We generated the sample covariance matrix S by adding symmetric positive semi-definite random noise N to Θ⁻¹; i.e. s = Θ⁻¹ + N, where this noise is generated in the same manner as A. We considered four different values of p ∈ {300, 500, 800, 1000} and two different values of τ ∈ {1, 4}.

For every p, τ combination we considered a path of twenty λ values on the geometric scale. For every such case four experiments were performed: Primal-Cold, Primal-Warm, Dual-Cold and Dual-Warm (as described in Section 9). Each combination was run 5 times, and the results averaged, to avoid dependencies on machine loads. Figure 4 shows the results. Overall, DP-GLASSO with warm starts performs the best, especially at the extremes of the path. We gave some explanation for this in Section 6. For the largest problems (p = 1000) their performances are comparable in the central part of the path (though DP-GLASSO dominates), but at the extremes DP-GLASSO dominates by a large margin.

FIG 4.

The timings in seconds for the four different algorithmic versions GLASSO (with and without warm-starts) and DP-GLASSO (with and without warm-starts) for a grid of twenty λ values on the log-scale. The horizontal axis is indexed by the proportion of zeros in the solution.