An Expectation Conditional Maximization approach for Gaussian graphical models

Abstract

Bayesian graphical models are a useful tool for understanding dependence relationships among many variables, particularly in situations with external prior information. In high-dimensional settings, the space of possible graphs becomes enormous, rendering even state-of-the-art Bayesian stochastic search computationally infeasible. We propose a deterministic alternative to estimate Gaussian and Gaussian copula graphical models using an Expectation Conditional Maximization (ECM) algorithm, extending the EM approach from Bayesian variable selection to graphical model estimation. We show that the ECM approach enables fast posterior exploration under a sequence of mixture priors, and can incorporate multiple sources of information.

1. Introduction

For high dimensional data, graphical models (Lauritzen, 1996) provide a convenient characterization of the conditional independence structure amongst variables. In settings where the rows in the data matrix $X\in {\mathbb{R}}^{n\times p}$ follow an i.i.d multivariate Gaussian distribution, Normal(0, Σ), the zeros in off-diagonal elements of the precision matrix Ω = Σ⁻¹ correspond to pairs of variables that are conditionally independent. Standard maximum likelihood estimators of the sparse precision matrix behave poorly and do not exist when n < p , leading to extensive work on algorithms (and their properties) for estimating Ω (e.g., Meinshausen and Bühlmann, 2006; Yuan and Lin, 2007; Friedman et al., 2008; Rothman et al., 2008; Friedman et al., 2010; Cai et al., 2010; Witten et al., 2011; Mazumder and Hastie, 2012, etc.).

In the Bayesian literature, structure learning in high-dimensional Gaussian graphical models has also gained popularity in the past decade. Broadly speaking, two main classes of priors have been studied for inference of the precision matrix in Gaussian graphical models, namely the G-Wishart prior, and shrinkage priors. The G-Wishart prior (Roverato, 2002) extends the Wishart distribution by restricting its support to the space of positive definite matrices with zeros specified by a graph. It is attractive in Bayesian modeling due to its conjugacy with the Gaussian likelihood. Posterior inference under the G-Wishart distribution, though computationally challenging, can be carried out via various algorithms, including shotgun stochastic search (Jones et al., 2005), reversible jump MCMC (Lenkoski and Dobra, 2011; Dobra et al., 2011; Wang and Li, 2012), and birth-death MCMC (Mohammadi et al., 2017), etc. More recently, shrinkage priors for precision matrices have gained much popularity, as they provide Bayesian interpretations to some of the widely used penalized likelihood estimators. As a Bayesian analogy to graphical lasso (Yin and Li, 2011; Witten et al., 2011; Mazumder and Hastie, 2012), Bayesian graphical lasso has been proposed in Wang et al. (2012) and Peterson et al. (2013). Wang (2015) later draws the connection between the Bayesian variable selection (George and McCulloch, 1993) and Bayesian graphical model estimation, and proposed a new class of spike-and-slab prior for precision and covariance matrices. This class of priors was also later explored in Peterson et al. (2015) to estimate the dependence structures among regression coefficients, and in Lukemire et al. (2017) to estimate multiple networks. This type of spike-and-slab prior enables a fast block Gibbs sampler that significantly improves the scalability of the model, but such flexibility is at the cost of prior interpretability since the implied marginal distribution of each elements in the precision matrix is intractable due to the positive definiteness constraint. Wang (2015) provides some heuristics and discussions on prior choices, but it is still not clear how to choose the hyperparameters for practical problems or how those choices affect parameter estimation.

In this paper, we introduce a new algorithm to estimate sparse precision matrices with spike-and-slab priors (Wang, 2015) using a deterministic approach, EMGS (EM graph selection), based on the Expectation Conditional Maximization (ECM) algorithm (Meng and Rubin, 1993). We also show that a stochastic variation of the EMGS approach can be extended to copula graphical model estimation. Our work extends the EM approach to variable selection (EMVS) (Ročková and George, 2014) to general graphical model estimation.

The proposed ECM algorithm is closely connected to frequentist penalized likelihood methods. Similar to the algorithms with concave penalized regularization, such as SCAD (Fan et al., 2009), the spike-and-slab prior used in our method yields sparse inverse covariance matrix where large values are estimated with less bias (see Figure 1). Similar work has been concurrently developed by Deshpande et al. (2017) using spike-and-slab lasso prior in the multivariate linear regression models. The proposed approach in this paper differs from Deshpande et al. (2017) in two ways: First, we use a mixture of Gaussian distributions instead of the Laplace distributions as the prior on the off-diagonal elements of the precision matrix, which allows us to construct a closed-form conditional maximization step using coordinate descent, rather than relying on additional algorithms solving a graphical lasso problem at each iteration. Second, and more importantly, our work also differs in scope, as we extended the algorithm to non-Gaussian outcomes, the scenarios where informative priors exist, and to incorporate the imputation of missing values.

Fig. 1.

Comparing partial correlation path using EMGS and graphical lasso, on a 10-node graph. The red dashed line at 0.5 is the true value for the non-zero negative partial correlations. The non-zero off-diagonal elements are plotted with blue solid lines. The vertical line indicates the tuning parameter selected with cross-validation.

The rest of the paper is organized as follows: In Section 2, we describe the spike-and-slab prior we use for the precision matrix. Section 3 presents the main ECM framework and algorithms for Gaussian graphical model estimation, and Section 4 proposes the extension to the copula graphical model and the modified stochastic ECM algorithm. Then in Section 5 we explore the incorporation of informative prior knowledge into the model. We discuss briefly about single model selection in Section 6. Section 7 examines the performance of our method through numerical simulations. Section 8 and 9 further illustrate our model using two examples from scientific settings. Section 8 compares our method and alternatives in terms of structure learning and prediction of missing values in a dataset of hourly bike/pedestrian traffic volumes along a busy trail in Seattle. Section 9 discusses our method in the context of learning latent structures among binary symptoms from a dataset of Verbal Autopsy (VA) surveys, which are used to estimate a likely cause of death in places where most deaths occur outside of medical facilities. Finally, in Section 10 we discuss the limitations of the approach and provide some future directions for improvements.

2. Spike-and-slab prior for Gaussian graphical model

First, we review the Stochastic Search Structure Learning (SSSL) prior proposed in Wang (2015) for sparse precision matrices. Consider the standard Gaussian graphical model setting, with observed data $X\in {\mathbb{R}}^{n\times p}$. Each observation follows a multivariate Gaussian distribution, i.e., x_i ~ Normal(0, Ω⁻¹) , where x_i is the i-th row of the X, and Ω is the precision matrix. Given hyperparameter V₀, V₁, and π_δ, the prior on Ω is defined as:

$$p(\Omega \mid \delta )=C_{{}_{\delta }}^{{}^{-1}}\prod _{j<k}\text{Normal}({\omega }_{{}_{jk}}\mid 0,v_{{}_{{\delta }_{jk}}}^{{}^2})\prod _j\text{Exp}({\omega }_{{}_{jj}}\mid \lambda ∕2)1_{\Omega \in M^{{}^{+}}}$$ (1)

$$p(\delta \mid {\pi }_{{}_{\delta }})\propto C_{{}_{\delta }}\prod _{j<k}{\pi }_{{}_{\delta }}^{{}^{{\delta }_{jk}}}(1-{\pi }_{{}_{\delta }}{)}^{{}^{1-{\delta }_{jk}}}$$ (2)

where δ_jk are latent indicator variables, and π_δ is the prior sparsity parameter. The C_δ term is the normalizing constant that ensures the integration of p(Ω ∣ δ) on M⁺ is one. This formulation places a Gaussian mixture prior on the off-diagonal elements of Ω , similar to the spike-and-slab prior used in the Bayesian variable selection literature. By setting ν₁ ≫ ν₀, the mixture prior imposes a different strength of shrinkage for elements drawn from the “slab” (V₁) and “spike” (V₀) respectively. This representation allows us to shrink elements in Ω to 0 if they are small in scale, while not biasing the large elements significantly.

The spike-and-slab formulation of Ω provides an efficient computation strategy via block Gibbs sampling. However, a main limitation is that parameter estimation can be sensitive to the choice of prior parameters. Unlike the variable selection problem in regression, information on the scale of the elements in the precision matrix typically cannot be easily solicited from domain knowledge. As shown in Wang (2015), there is no analytical relationship between the prior sparsity parameter π_δ and the induced sparsity from the joint distribution. This complexity results from the positive definiteness constraint on the precision matrix. Thus even if the sparsity of the precision matrix is known before fitting the model, additional heuristics and explorations are required to properly select the prior π_δ. Similarly, the induced marginal distribution of the elements in Ω is intractable as well. The supplementary material contains an simple illustration of such differences. Thus although the fully Gibbs sampler is attractive for high dimensional problems, in practice researchers will usually need to evaluate the model fit under multiple prior choices, adding substantially to the computational burden.

3. Fast deterministic algorithm for graph selection

Consider spike-and-slab priors on Ω as described in the previous section and let the hyperprior on the sparsity parameter to be π_δ ~ Beta (a, b) , the complete-data posterior distribution can be expressed as

$$p(\Omega ,\delta ,{\pi }_{{}_{\delta }}\mid \mathit{X})=p(\mathit{X}\mid \Omega )p(\Omega \mid \delta ,v_{{}_0},v_{{}_1},\lambda )p(\delta \mid {\pi }_{{}_{\delta }})p({\pi }_{{}_{\delta }}\mid a,b),$$

In order to perform posterior sampling in the fully Bayesian fashion, the block Gibbs algorithm in Wang (2015) reduces the problem to iteratively sampling from (p − 1) -dimensional multivariate Gaussian distributions for each column of Ω , which can still be computationally expansive for large p or if the sampling needs to be repeated for multiple prior setups. Inspired by the EM approach for variable selection proposed in Ročková and George (2014), we propose a EMGS algorithm to identify the posterior mode of p(Ω , π_δ ∣ X) directly without the full stochastic search. We iteratively maximize the following objective function

$$\begin{gathered} Q(\Omega ,{\pi }_{{}_{\delta }}\mid \Omega {}^{{}^{(l)}},{\pi }_{{}_{\delta }}^{{}^{(l)}})=E_{{}_{\delta \mid {\Omega }^{{}^{(l)}},{\pi }_{{}_{\delta }}^{(l)},\mathit{X}}}(\log p(\Omega ,\delta ,{\pi }_{{}_{\delta }}\mid \mathit{X})\mid {\Omega }^{{}^{(l)}},{\pi }_{{}_{\delta }}^{{}^{(l)}},\mathit{X}) \\ =\text{constant +}\frac{n}{2}\log \mid \Omega \mid -\frac{1}{2}tr({\mathit{X}}^T\mathit{X}\Omega ) \\ -\frac{1}{2}\sum _{j<k}{\omega }_{{}_{jk}}^{{}^2}{E}_{{}_{\cdot \mid \cdot }}[\frac{1}{v_{{}_0}^{{}^2}(1-{\delta }_{{}_{jk}})+v_{{}_1}^{{}^2}{\delta }_{{}_{jk}}}]-\frac{\lambda }{2}\sum _j{\omega }_{{}_{jj}} \\ +\sum _{j<k}\log (\frac{{\pi }_{{}_{\delta }}}{1-{\pi }_{{}_{\delta }}}{E}_{{}_{\cdot \mid \cdot }}[{\delta }_{{}_{jk}}])+\frac{p(p-1)}{2}\log (1-{\pi }_{{}_{\delta }}) \\ +(a-1)\log ({\pi }_{{}_{\delta }})+(b-1)\log (1-{\pi }_{{}_{\delta }}) \end{gathered}$$

where E_·∣·[·] denotes $E_{\delta \mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}[\cdot ]$. This objective function can be easily estimated using ECM algorithm, and the algorithm can naturally handle missing values in the E-step. We present the details of the proposed algorithm in the next subsection and then compare the algorithm with the coordinate ascent algorithm for solving graphical lasso problem in Section 3.2.

3.1. The ECM algorithm

The E-step

We start by computing the conditional expectations $E_{\delta \mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}[{\delta }_{jk}]$ and $E_{\delta \mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}[\frac{1}{{\nu }_0^2(1-{\delta }_{jk})+{\nu }_1^2{\delta }_{jk}}]$. This proceeds in the similar fashion as the standard EMVS,

$$E_{{\delta }_{jk}\mid {\Omega }^{{}^{(l)}},{\pi }_{{}_{\delta }}^{(l)},\mathit{X}}[{\delta }_{{}_{jk}}]=p_{{}_{jk}}^{{}^{\ast }}\equiv \frac{a_{{}_{jk}}}{a_{{}_{jk}}+b_{{}_{jk}}},$$ (3)

where $a_{jk}=p({\omega }_{jk}\mid {\delta }_{jk}=1){\pi }_{\delta }^{(l)}$ and $b_{jk}=p({\omega }_{jk}\mid {\delta }_{jk}=0)(1-{\pi }_{\delta }^{(l)})$, and

$$E_{\delta \mid {\Omega }^{{}^{(l)}},{\pi }_{{}_{\delta }}^{(l)},\mathit{X}}[\frac{1}{v_{{}_0}^{{}^2}(1-{\delta }_{{}_{jk}})+v_{{}_1}^{{}^2}{\delta }_{{}_{jk}}}]=\frac{1-p_{{}_{jk}}^{{}^{\ast }}}{v_{{}_0}^{{}^2}}+\frac{p_{{}_{jk}}^{{}^{\ast }}}{v_{{}_1}^{{}^2}}\equiv d_{{}_{jk}}^{{}^{\ast }}.$$ (4)

Modified E-step with missing data

When missing data exists in the data matrix X, the E-step can be easily extended to find the expectation of the missing values as well. In that case, the conditional expectations of δ remains unaffected, and we only need to additionally obtain the expectation for the X^T XΩ term as

$$E_{{}_{\delta ,\mathit{X}\mid \Omega }}({\mathit{X}}^T\mathit{X}\Omega )=E_{{}_{\delta ,\mathit{X}\mid \Omega }}((\sum _i^n{\mathit{x}}_{{}_i}{\mathit{x}}_{{}_i}^{{}^T})\Omega )=(\sum _i^n{E}_{{\mathit{x}}_{{}_{i,m}}\mid {\mathit{x}}_{{}_{i,o}},\Omega }({\mathit{x}}_{{}_i}{\mathit{x}}_{{}_i}^{{}^T}))\Omega .$$

where x_i,o and x_i,m denote the observed and missing cells in x_i respectively. Without loss of generality, if we let ${\mathit{x}}_i^T=[{\mathit{x}}_{i,o}^T,{\mathit{x}}_{i,m}^T]$ , we know

$$\begin{gathered} E_{{}_{{\mathit{x}}_{{}_{i,m}}\mid {\mathit{x}}_{{}_{i,o}},\Omega }}({\mathit{x}}_{{}_{i,m}})=-{\Omega }_{{}_{oo}}^{{}^{-1}}{\Omega }_{{}_{mo}}{\mathit{x}}_{{}_{i,o}} \\ {E}_{{\mathit{x}}_{{}_{i,m}}\mid {\mathit{x}}_{{}_{i,o}},\Omega }({\mathit{x}}_{{}_i}{\mathit{x}}_{{}_i}^{{}^T})=E_{{}_{\cdot \mid \cdot }}({\mathit{x}}_{{}_i})E_{{}_{\cdot \mid \cdot }}({\mathit{x}}_{{}_i}{)}^T+\left(\begin{array}{cc}\hfill 0_{{}_{oo}}\hfill & \hfill 0_{{}_{om}}\hfill \\ \hfill 0_{{}_{mo}}\hfill & \hfill {\Omega }_{{}_{mm}}^{{}^{-1}}\hfill \end{array}\right) \end{gathered}$$

where Ω_{o o}, Ω_{m o} and Ω_{m m} are the corresponding submatrices of Ω .

The CM-step

After the E-step is performed, the CM-step performs the maximization of (Ω, π_δ) in a coordinate ascent fashion. First, the maximization of π_δ has the close-form solution

$${\pi }_{{}_{\delta }}^{{}^{(l+1)}}=(a+\sum _{j<k}{\delta }_{{}_{jk}}-1)∕(a+b+p(p-1)∕2-2).$$ (5)

The joint maximization of Ω has no closed-form solution, but if we denote

$$\Omega =\left(\begin{array}{cc}\hfill {\Omega }_{{}_{11}}\hfill & \hfill {\omega }_{{}_{12}}\hfill \\ \hfill {\omega }_{{}_{12}}^{{}^T}\hfill & \hfill {\omega }_{{}_{22}}\hfill \end{array}\right){\mathit{X}}^T\mathit{X}=\left(\begin{array}{cc}\hfill {\mathit{S}}_{{}_{11}}\hfill & \hfill {\mathit{s}}_{{}_{12}}\hfill \\ \hfill {\mathit{s}}_{{}_{12}}^{{}^T}\hfill & \hfill s_{{}_{22}}\hfill \end{array}\right),$$

Wang (2015) showed that the conditional distribution of the last column satisfies

$${\omega }_{{}_{12}}\sim \text{Normal}(-\mathit{C}{\mathit{s}}_{{}_{12}},\mathit{C}),\mathit{C}=((s_{{}_{22}}+\lambda ){\Omega }^{{}^{-1}}+\text{diag}(v_{{}_{{\delta }_{12}}}){)}^{{}^{-1}},$$

where ν_δ12 are the inclusion indicators for ω₁₂ and

$${\omega }_{{}_{22}}-{\omega }_{{}_{12}}^{{}^T}{\Omega }_{{}_{11}}^{{}^{-1}}{\omega }_{{}_{12}}\sim \text{Gamma}(1+\frac{n}{2},\frac{\lambda +s_{{}_{22}}}{2}).$$

This enables us to perform conditional maximization (Meng and Rubin, 1993) for the last column holding the rest of Ω fixed. That is, starting with Ω^(l+1) = Ω^(l) , we iteratively permute each column to the last and update it with

$${\omega }_{{}_{12}}^{{}^{(l+1)}}=((s_{{}_{22}}+\lambda )({\Omega }_{{}_{11}}^{{}^{(l+1)}}{)}^{{}^{-1}}+\text{diag}(d_{{}_{jk}}^{{}^{\ast }}){)}^{{}^{-1}}{\mathit{s}}_{{}_{12}}$$ (6)

and

$${\omega }_{{}_{22}}^{{}^{(l+1)}}=({\omega }_{{}_{12}}^{{}^{(l+1)}}{)}^T({\Omega }_{{}_{11}}^{{}^{(l+1)}}{)}^{{}^{-1}}{\omega }_{{}_{12}}^{{}^{(l+1)}}+\frac{n}{\lambda +s_{{}_{22}}}.$$ (7)

Finally, be iterating between the E-step and the CM-steps until convergence, we obtain our estimator of the posterior mode Ω and ${\widehat{\pi }}_{\delta }$ .

3.2. Connection to the graphical lasso

This column-wise update resembles the penalized likelihood approach in frequentist settings. In the graphical lasso algorithm (Mazumder and Hastie, 2012) for example, the goal is to minimize the l₁-penalized negative log-likelihood:

$$f(\Omega )=-\log \mid \Omega \mid +\text{tr}(\mathit{S}\Omega )+{\Vert \Omega \Vert }_{{}_1},$$

which can be solved via a block coordinate descent that iteratively solves the lasso problem

$${\omega }_{{}_{12}}=\underset{\alpha \in R^{{}^{m-1}}}{\text{arg}\text{min}}{\alpha }^T{\Omega }_{{}_{11}}^{{}^{-1}}\alpha +{\alpha }^T{\mathit{s}}_{{}_{12}}+\lambda {\Vert \alpha \Vert }_{{}_1}.$$

The updates at each iteration in the EMGS framework solve the optimization problem for ω₁₂ under an adaptive ridge penalty

$${\omega }_{12}=\underset{\alpha \in R^{m-1}}{\text{arg min}}{\alpha }^T{\Omega }_{11}^{-1}\alpha +{\alpha }^T{\mathit{s}}_{12}+\sum _{j=1}^{m-1}{d}_j^{\ast }{\alpha }_j^2.$$

The penalty parameters $d_j^{\ast }$ are the corresponding $d_{jk}^{\ast }$ estimated from the E-step and are informed by data. That is, instead of choosing a fixed penalty parameter for all precision matrix elements, the EMGS approach learns the element-wise penalization parameter at each iteration based on the magnitude of the current estimated Ω and the hyperpriors placed on θ. Thus, as long as the signal from data is not too weak, the EMGS procedure can estimate large elements in the precision matrix with much lower bias than graphical lasso, as the adaptive penalties associated with large ω _jk are small. To illustrate the diminished bias, we fit the EMGS algorithm to a simple simulated example, where n = 100, p = 10 and Ω is constructed by ω _jj = 1, and ω _jk = 0.5 if ∣ j − k ∣= 1 . We fix ν₁ = 100 and compare the regularization path with various V₀ values with graphical lasso, as shown in Figure 1. This simple example illustrates two main advantages of EMGS. First, it identifies the set of non-zero elements quickly and estimates the partial correlations correctly around 0.5 under all values of V₀. The clear separation of the truly non-zero edges regardless of V₀ also makes it straightforward to threshold ∣ ω _jk ∣ to recover the true graph structures. Graphical lasso, on the other hand, shrinks the non-zero partial correlations significantly under large penalties, and thus lead to worse graph selection if the tuning parameter is not properly chosen. Second, In order to select and compare a single model, we also identified the optimal tuning parameter using 5-fold cross validation for both methods, and it can be seen that the graphical lasso estimator suffers from the weak penalty and contains more noise than using EMGS.

4. ECM algorithm for copula graphical models

In this section, we extend the framework to non-Gaussian data with Gaussian copulas (Nelsen, 1999). Denote the observed data $\mathit{X}\in {\mathbb{R}}^{n\times p}$ , and each of the p variables could be either continuous, ordinal, or binary. We model each observation as following a Gaussian copula model, i.e., there exists a set of monotonically increasing transformations f = {f₁, . . . , f_p} such that Z = f (X) ~ Normal(0, R), where R is a correlation matrix. Following the same setup as before, we let R be the induced correlation matrix from Ω with the spike-and-slab prior defined as before, i.e.,

$${\mathit{R}}_{{}_{[j,k]}}={\Omega }_{{}_{[j,k]}}^{{}^{-1}}∕\sqrt{{\Omega }_{{}_{[j,j]}}^{{}^{-1}}{\Omega }_{{}_{[k,k]}}^{{}^{-1}}}.$$

The explicit form of f is typically unknown, thus we impose no restrictions on the class of marginal transformations. Instead, we follow the extended rank likelihood method proposed in Hoff (2007), decomposing the complete data likelihood into

$$p(\mathit{X}\mid \mathit{R},f)=\mathit{Pr}(\mathit{Z}\in \mathit{S}\mid \mathit{R})p(\mathit{X}\mid \mathit{Z}\in \mathit{S},\mathit{R},f),$$ (8)

where S is the support of Z induced by the ranking of X defined by

$${\mathit{S}}_{{}_{ij}}=[\max \{z_{{}_{i^{\prime }{j}^{\prime }}}:x_{{}_{i^{\prime }{j}^{\prime }}}<x_{{}_{ij}}\},\min \{z_{{}_{i^{\prime }{j}^{\prime }}}:x_{{}_{i^{\prime }{j}^{\prime }}}>x_{{}_{ij}}\}].$$

Since our goal is to recover the structure in Ω , we can estimate the parameters using only the first part of (8) without estimating the nuisance parameter f. Moreover, since the latent Gaussian variable Z is constructed to be centered at 0 , the rank likelihood remains unchanged when multiplying columns of X by any constant. Thus, inference could be performed without restricting R to be an correlation matrix (Hoff, 2007). In this way, the target function to maximize is the extended rank likelihood function:

$$p(\Omega ,\delta ,{\pi }_{\delta },\mathit{Z}\mid \mathit{X})=p(\mathit{Z}\in \mathit{S}\mid \Omega ,\mathit{S})p(\Omega \mid \delta )p(\delta \mid {\pi }_{\delta }).$$

This is immediately analogous to the EMGS framework with latent Gaussian variable Z as additional missing data. That is, we maximize the objective function defined as

$$\begin{gathered} Q(\Omega ,{\pi }_{\delta }\mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)})=E_{\delta ,\mathit{Z}\mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}(\log p(\Omega ,\delta ,{\pi }_{\delta },\mathit{Z}\mid \mathit{X})\mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}) \\ =\text{constant}+Q_1-\frac{1}{2}\sum _{j<k}{\omega }_{jk}^2{E}_{\cdot \mid \cdot }[\frac{1}{{\nu }_0^2(1-{\delta }_{jk})+{\nu }_1^2{\delta }_{jk}}]-\frac{\lambda }{2}\sum _i{\omega }_{ii} \\ +\sum _{j<k}\log (\frac{{\pi }_{\delta }}{1-{\pi }_{\delta }}{E}_{\cdot \mid \cdot }[{\delta }_{jk}])+\frac{p(p-1)}{2}\log (1-{\pi }_{\delta }) \\ +(a-1)\log ({\pi }_{\delta })+(b-1)\log (1-{\pi }_{\delta }) \end{gathered}$$

where E_·∣·[·] denotes $E_{\delta ,\mathit{Z}\mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}[\cdot ]$, and the only term different from the standard EMGS objective function is

$$\begin{gathered} Q_1=E_{\mathit{Z}\mid {\Omega }^{(l)},{\pi }_{\delta }^{(l)},\mathit{X}}(\log p(\mathit{Z}\mid \Omega ,\mathit{S})) \\ =\text{constant}+\frac{n}{2}\log \mid \Omega \mid -\frac{1}{2}{E}_{\mathit{Z}\mid {\Omega }^{(l)},\mathit{X}}[\mathit{tr}({\mathit{Z}}^T\mathit{Z}\Omega )]. \end{gathered}$$

Exact computation for this expectation is intractable as z ∣ x is a Gaussian random matrix where each row is conditionally Gaussian and the within column ranks are fixed by S. Alternatively, posterior samples of Z are easy to obtain from the conditional truncated Gaussian distribution (Hoff, 2007), so we can adopt stochastic variants of the EM algorithm (Wei and Tanner, 1990; Delyon et al., 1999; Nielsen, 2000; Levine and Casella, 2001). We present one such algorithm in the subsequent subsection.

The SAE-step for non-Gaussian variables

Among the many variations of the EM with stochastic approximation, we discuss estimation steps using stochastic approximation EM (SAEM) algorithm (Delyon et al., 1999). SAEM calculates the E-step at each iteration as a weighted average of the current objective function and new stochastic samples using a decreasing sequence of weights for the stochastic averages, in a similar fashion as simulated annealing. In the stochastic E-step, we compute an additional term Q(Ω^(l)) = E_Z∣Ω^(l),X[Z^T Z] as

$$Q({\Omega }^{{}^{(l)}})=(1-t_{{}_k})Q({\Omega }^{{}^{(l)}})+\frac{t_{{}_k}}{B_{{}_k}}\sum _{b=1}^{B_k}{\mathit{Z}}_{{}_{(b)}}^{{}^T}{\mathit{Z}}_{{}_{(b)}}$$

where t_k is an decreasing step-size sequence such that $\sum t_k=\infty ,\sum t_k^2<\infty$ ,and B_k is the number of stochastic samples drawn at each iteration. The rank constrained Gaussian variables can be drawn using the same procedure described in Hoff (2007).

The CM-step then proceeds as before, except that the empirical cross-product matrix S is replaced by its expectation Q(Ω^k). For the numerical examples in this paper, we set fixed B_k and t_k = 1/k. Other weighting schemes could also be explored and may yield different rates of convergence.

5. Incorporating edge-wise informative priors

The exchangeable beta-binomial prior discussed so far assumes no prior structure on Ω and prior sparsity controlled by a single parameter for all off-diagonal elements. For many problems in practice, informative priors may exist for pairwise interactions of the variables. For example, Peterson et al. (2013) infers cellular metabolic networks based on prior information in the form of reference network structures. Bu and Lederer (2017) improve estimation of brain connectivity network by incorporating the distance between regions of the brain. In problems with small sample sizes, such prior information can help algorithms identify the high probability edges more quickly and provide more interpretable model. More generally, we can consider a situation where certain groupings exist among variables. For example, when the variables represent log sales of p products on the market, one might expect that the products within the same brand are more likely to be more strongly correlated. If we define a fixed index function g_j ∈ {1, . . . , G}, j ∈ {1, . . . , p}, where G denotes the total number of groups, we can modify the prior into

$$\begin{gathered} p(\Omega \mid \delta )=C_{\delta }^{-1}\prod _{j<k}\text{Normal}({\omega }_{jk}\mid 0,\frac{{\nu }_{{\delta }_{jk}}^2}{{\tau }_{g_j{g}_k}})\prod _j\text{Exp}({\omega }_{jj}\mid \lambda ∕2)1_{\Omega \in M^{+}} \\ p(\delta \mid {\pi }_{\delta })\propto C_{\delta }\prod _{j<k}{\pi }_{\delta }^{{\delta }_{jk}}(1-{\pi }_{\delta }{)}^{1-{\delta }_{jk}} \\ p(\tau )=\prod _{g<g^{\prime }}\text{Gamma}(a_{\tau },b_{\tau }) \end{gathered}$$

The block-wise rescaling parameter τ_gjgk of the variance parameter allows us to model within- and between-block elements of Ω adaptively with different scales. This is particularly useful in applications where block dependence structures have different strengths. Take the example of sales of products for example. Products within the same brand or category are more likely to be conditional dependent, yet the within group sparsity and the scale of the off-diagonal elements may differ for different brands. In the special case where the full edge-level prior probabilities of connection are known, as considered by Peterson et al. (2013) and Bu and Lederer (2017), we can also equivalently let G = P and parameterize p(τ) with the edge-specific priors.

The ECM algorithm discussed above only requires minor modifications to include the additional scale parameter so that the penalties for each block are allowed to vary (e.g., Ishwaran and Rao, 2003; Wakefield et al., 2010). The new objective function could be similarly estimated with ECM algorithm by including this additional update in the CM-step:

$${\tau }_{g{g}^{\prime }}^{(l+1)}=\frac{a_{\tau }-1+\frac{1}{2}\sum _{j<k}{1}_{j,k,g,g^{\prime }}}{b_{\tau }+\frac{1}{2}\sum _{j<k}{\omega }_{jk}^2{d}_{jk}^{\ast }{1}_{j,k,g,g^{\prime }}},$$ (9)

where 1_j,k,g,g′ = 1 if g_j = g,g_k = g′, or g_j = g′,g_k = g. To illustrate the behavior of this block rescaled prior, we simulate data with n = 200, p = 60, with the precision matrix to be block diagonal with three equal-sized blocks. We simulate the three block sub-matrices of Ω to correspond to random graphs with sparsity 0.4, as described in Section 7. Figure 2 shows the effect of the structured prior. It can be seen that the estimated $1∕{\widehat{\tau }}_{g{g}^{\prime }}$, are much larger where g = g′, which leads to weaker shrinkage effects for within cluster cells. Accordingly the resulting graph using the structured prior shows fewer false positives for the off-diagonal blocks, and better discovery of the true positives within blocks.

Fig. 2.

Comparing the estimated and true precision matrix using graphical lasso, EMGS with exchangeable prior, and with structured prior for block-wise rescaling. In each plot of the precision matrix comparison, the upper triangle shows the estimated matrix and the lower triangle shows the true precision matrix. All the tuning parameters are selected by cross-validation. The presented edges are further thresholded to have the same number of edges compared to the true graph. The forth plot shows the change of $1∕{\widehat{\tau }}_{gg}$, over different choices of V₀. The blocks are labeled 1 to 3 from top left to bottom right.

6. Posterior summary of the ECM output

One of the main computational advantage of the ECM approach over stochastic search is that the posterior mode is fast to obtain. Thus it provides a more efficient alternative to experimenting multiple choices of priors with full MCMC, as discussed before. In practice, we fix V₁ to be a large constant and vary the choice of V₀ to reflect different levels of shrinkage on the off-diagonal elements of Ω that are close to 0. Intuitively, a larger V₀ increases the probability of small parameters being drawn form the spike distribution and thus leads to sparse models. By fitting a sequence of V₀, we can create regularization plots, e.g., Figure 1, similar to that used in penalized regression literature to visually examine the influence of the prior choices. Choosing a single tuning parameter V₀ is possible with standard model selection criterion, such as AIC (Akaike, 1998), BIC (Schwarz et al., 1978), RIC (Lysen, 2009), StARS (Liu et al., 2010), etc., or K-fold cross validation using the average log-likelihood of the validation sets. In the rest of the paper, we select a single tuning parameter V₀ using 5-fold cross validation. In the case of non-Gaussian data or data with missing values, the likelihood on test data can be evaluated by the average of the expected covariance $\frac{1}{m}\sum _{{\nu }_0}{E}_{{\mathit{X}}_{\mathit{test}}\mid {\mathit{X}}_{\mathit{train}},{\nu }_0}({\mathit{X}}_{\mathit{test}}^T{\mathit{X}}_{\mathit{test}})$ under the sequence of m tuning parameters. This term can be easily calculated by plugging in the test data in the E-step of the algorithm. It is worth noting that since the mixture of Gaussian prior does not lead to exact sparsity, in scenarios where graph structure is of direct interest, we further determining the graph structure by thresholding the off-diagonal elements, ∣ ω _jk ∣ , as the posterior inclusion probability $p_{jk}^{\ast }$ conditional on ω_jk is a monotone function of ω_jk.

7. Simulation

We follow a similar simulation setup to Mohammadi et al. (2017) with different graph structures. We compare the performance of our method with graphical lasso for Gaussian data and graphical lasso with nonparanormal transformation (Liu et al., 2009), and the rank-based extension proposed in Xue et al. (2012) for non-Gaussian data. We consider the following sparsity patterns in our simulation:

• AR(1): A graph with σ _jk = 0.7 ^∣j−k∣.

• AR(2): A graph with ω _jj = 1, ω _j,j−1 = ω _j−1,j = 0.5, and ω _j,j−2 = ω _j−2,j = 0.25, and ω _jk = 0 otherwise.

• Random: A graph in which the edge set E is randomly generated from independent Bernoulli distributions with probability 0.2 and the corresponding precision matrix is generated from Ω ~ W_G (3, I _p).

• Cluster: A graph in which the number of clusters is max{2,[p / 20]}. Each cluster has the same structure as a random graph. The corresponding precision matrix is generated from Ω ~ W_G (3, I _p).

We simulate data with sample size n ∈ {100, 200, 500}, and of dimension p ∈ {50, 100, 200}, using the each types of precision matrices above that are rescaled to have unit variances. We generate both Gaussian and non-Gaussian data for each configuration. For the non-Gaussian case, we perform the marginal transformation of the latent Gaussian variables so that the variables follow a marginal distribution of Poisson(θ), with θ = 10 or 2.

For each generated graph, we fit our ECM algorithm with a sequence of 40 increasing V₀’s, and fix ν₁ = 100, λ = 1, and a = b = 1. We select the final V₀ using 5-fold cross validation. We also select the tuning parameter for graphical lasso using cross-validation (GL-CV). We then evaluate the bias of EMGS and graphical lasso estimator of precision matrices compared to the truth in terms of the matrix Frobenius norm, $\Vert \Omega -\Omega {\Vert }_F=\sqrt{\sum _j\sum _k\mid {\widehat{\omega }}_{jk}-{\omega }_{jk}{\mid }^2}$. Because of the excess biased induced by a single penalty parameter, cross-validation tend to choose small penalties for graphical lasso, leading to massive false positives in edge discovery. Thus to allow a fair comparison, we compare the area under the ROC curve (AUC) by increasingly thresholding elements in Ω obtained by cross-validation for both EMGS and graphical lasso. Besides selecting tuning parameter by cross-validation for graphical lasso and the nonparanormal transformed estimator, we also consider Ω selected using two popular model selection criterion: rotation information criterion (GL-RIC) (Lysen, 2009), and stability approach (GL-StARS) (Liu et al., 2010). For the copula graphical model, we also compare the rank-based extension proposed in Xue et al. (2012) of graphical lasso (GL-rank) with the tuning parameter selected with cross validation.

The simulation results are summarized in Figure 3 and 4. Less bias in parameter estimation are indicated by smaller F-norm values and better graph learning is indicated by larger AUC values. In almost all cases of our simulation study, we observe significantly reduced biases in the estimator from EMGS estimators, as well as better graph selection performance in most cases. We also include additional comparisons in the supplementary material that examine the bias in matrix spectral norms, the F₁-score for graphical lasso estimators at the selected penalty levels, as well as the F₁-score when all estimators are thresholded to have the correct number of edges.

Fig. 3.

Comparing estimation of the precision matrix for both the Gaussian and Gaussian copula case under different simulation setups. Five estimators are considered: the proposed method (EMGS), Gaussian and nonparanormal graphical lasso with penalty selected by cross validation (GL-CV), RIC (GL-RIC), stability approach (GL-StARS), and rank-based extension of graphical lasso proposed in Xue (2012) selected by cross validation for the copula case (GL-rank). EMGS shows lower bias in almost all cases.

Fig. 4.

Comparing estimation of the graph structure for both the Gaussian and Gaussian copula case under different simulation setups. EMGS shows higher AUC in almost all cases.

All computation are conducted in the R statistical programming environment (R Core Team, 2018). The graphs are simulated using the R packages BDgraph (Mohammadi and Wit, 2015) and tmvtnorm (Wilhelm and G, 2015). EMGS is implemented with the Rcpp package (Eddelbuettel and François, 2011). The graphical lasso estimation are implemented with the R package huge (Zhao et al., 2012) and glasso (Friedman et al., 2018). Visualizations are created with ggplot2 (Wickham, 2016) and corrplot (Wei and Simko, 2017). The AUC values are calculated with the ROCR package (Sing et al., 2005).

8. Traffic on the Burke Gilman Trail

In this section we consider graph estimation and prediction for the hourly traffic on the Burke Gilman Trail in Seattle. We use the hourly counts of bikes and pedestrians traveling on the trail through north of NE 70th Street using data from the Seattle Open Data program¹.

The data are captured by sensors that detect both bikes and pedestrians, and their directions of travel. At each hour, the sensors record four counts of travelers: by bike or foot, and towards north or south. We used all the data from 2014 that contain n = 365 observations of 24 × 4 = 96 measurements. We first performed a log transformation on the raw counts, and subtracted the hourly average from the log counts. The data are reformatted from the original format with the R package reshape2 (Wickham, 2007).

We estimated the joint distribution of the 96 measurements using EMGS with both the beta-binomial prior and the group-wise structured priors, with 4 groups defined by the mode of travel/direction pairs. Figure 5 shows the estimated graphs and the induced covariance matrices. Graphical lasso estimates many edges with small ω_jk, while EMGS allows us to pick out large ω_jk, especially those that correspond to the edges between the number of pedestrians traveling within the same hour in opposite directions, and the number of bikes traveling in adjacent hours in the same direction during morning and afternoon commute hours. In this analysis, the structured priors lead to a slightly more concentrated set of entires, but both priors lead to similar graph estimation for EMGS. We also compare the performance of predicting missing values using $\widehat{\Omega }$, by randomly removing half of the measurements on half of the days. The missing observations can be imputed by the EMGS algorithm described in Section3, and similarly we can estimate $\widehat{\Omega }$ by either the empirical covariance matrix or from graphical lasso using only the observed variables. We compare the predictive performance by the Mean Squared Error defined as $\frac{1}{n_{\mathit{miss}}}\sum _{i,j}(X_{ij}-{\widehat{X}}_{ij}{)}^2$.

Fig. 5.

Comparing the estimated precision matrices from cross validation. The blocks correspond to travel mode and direction pairs. From upper left to lower right: southbound pedestrians, northbound pedestrians, southbound bikes, and northbound bikes. Within each block, the entries correspond to 24 hourly intervals starting from midnight. Top row: estimated covariance matrix. Edges with less than 0.5 probability of being from the slab distributions in EMGS output, and exact zeros in graphical lasso output are marked with gray color. Bottom row: estimated precision matrix with highlighted graph selection.

Intuitively, predictions based on penalized estimators that are over shrunk towards zero is likely to increase bias, while with little penalization, the estimated covariance matrix is more likely to be noisy, as shown in Figure 5. Table 1 shows the average MSE and their standard deviations using different estimators over 100 replications, and it confirms the improved prediction performance from EMGS compared to graphical lasso.

Table 1.

Average and standard deviation of the mean squared errors from 100 cross-validation experiments.

	EMGS
	exchangeable	structured	GLasso	Empirical
Average MSE	0.2828	0.2809	0.4262	0.4602
Standard deviation of the MSEs	0.0052	0.0050	0.0064	0.0096

9. Symptom structure in Verbal Autopsies

In this section, we use EMGS to learn the latent dependence structure among symptoms reported on verbal autopsy (VA) surveys. VA surveys collect information about a deceased person’s health history through an interview with caregivers or family members of the decedent. VAs are widely used in countries without full-coverage civil registration and vital statistics systems. About 2/3 of deaths worldwide occur in such settings (Horton, 2007). VA data consist primarily of binary indicators of symptoms and conditions leading to the death (e.g. Did the decedent have a fever? Was there pain in the lower belly?).

Several algorithms have been proposed to assign causes of death using such binary input (Byass et al., 2012; Serina et al., 2015; McCormick et al., 2016), but these algorithms typically assume that the binary indicators are independent. We use data from the Physicians Health Metrics Research Consortium (Murray et al., 2011). We created 107 variables from the binary questions in the dataset of 7, 841 adults using the R package openVA (Li et al., 2019), and removed the variables with more than 50% of values missing, leaving us with 90 indicators. There are many missing values even after reducing the number of indicators, so there is only one observation with answers for all 90 indicators. This high proportion of missing data makes it difficult to directly apply different types of rank-based estimators for the latent precision matrix that only uses complete observations. Instead, we focus on exploration of the joint distribution of the binary variables under the latent Gaussian framework described in Section 4. We first rescale the dataset by the marginal means of the indicators to remove the different levels of prevalence among the symptoms. We then apply the EMGS algorithm to the rescaled dataset with the same hyperpriors used in Section 7, and select the final V₀ using cross validation. The resulting conditional dependence graph with 46 indicators and 42 edges is shown in Figure 6, where several main symptom pairs (e.g., fever and sweating, stroke and paralysis, etc.) and symptom groups (e.g., indicators related to pregnancy) are discovered, indicating the existence of some symptom clusters that are strongly dependent in the dataset. Further incorporation of the ECM framework into a classification framework could improve accuracy over existing methods for automatic cause-of-death assignment. The visualization of the symptom network is made with R package network (Butts, 2008).

Fig. 6.

Estimated edges between the indicators in the VA dataset. The width of the edges are proportional to the value of ∣ω_jk∣. Red edges correspond to negative values of ω_jk, or positive partial correlations. Black edges correspond to positive values of ω_jk, or negative partial correlations.

10. Discussion

We propose a deterministic approach for graphical model estimation that builds upon the recently proposed class of spike-and-slab prior for precision matrices. By drawing the connection between the conditional maximization updates under the spike-and-slab prior and the graphical lasso algorithm, we illustrate that EM type algorithm can be used to efficiently obtain posterior modes of the precision matrix under adaptive penalization. It also allows us to build richer class of models that incorporate prior information and extend to copula graphical models. The computational speed of the EGMS algorithm allows us to explore multiple prior choices without fitting many time-consuming MCMC chains. However, it also comes at the price of two potential limitations. First, characterization of posterior uncertainty is nontrivial due to the deterministic nature of the algorithm. As in Ročková and George (2014), one may choose to fit a Bayesian model “locally” from the posterior mode obtained by the ECM procedure, though this may still be challenging in high-dimensional problems. Another limitation is that like the EM algorithm, ECM algorithm also converges only to local modes, thus the precision matrix initialization is critical. In this paper, we used the same initialization as the P-Glasso algorithm described in Mazumder and Hastie (2012). Other heuristics for initialization and warm start may also be explored. Finally, multimodal posteriors are common with spike-and-slab priors. The proposed method could be extended to introduce perturbations in the algorithm, possibly drawing from the variable selection literature (see, e.g., Ročková and George, 2014; Rocková, 2016).

Replication code for the numerical examples in this article is available at https://github.com/richardli/EMGS.