A STATISTICAL FRAMEWORK FOR DATA INTEGRATION THROUGH GRAPHICAL MODELS WITH APPLICATION TO CANCER GENOMICS

Abstract

Recent advances in high-throughput biotechnologies have generated var-ious types of genetic, genomic, epigenetic, transcriptomic and proteomic data across different biological conditions. It is likely that integrating data from diverse experiments may lead to a more unified and global view of biolog-ical systems and complex diseases. We present a coherent statistical frame-work for integrating various types of data from distinct but related biological conditions through graphical models. Specifically, our statistical framework is designed for modeling multiple networks with shared regulatory mech-anisms from heterogeneous high-dimensional datasets. The performance of our approach is illustrated through simulations and its applications to cancer genomics.

1. Introduction.

Recent advances in high-throughput technologies have generated unprecedented types and amounts of data for biomedical research. Examples include genome-wide characterizations of DNA variations (e.g., genotyping arrays, whole exome or genome sequencing), gene expression variations (e.g., gene expression microarrays, RNA sequencing), epigenetic variations and protein expression variations. Each data type, for example, genomic, transcriptomic or proteomic data, provides a comprehensive, but one-layer, view of the biological system being studied. Integrating data of diverse types is likely to lead to a more unified and global view. Thus, increasing research attention is being paid to the integrative analysis and modeling of various types of biomedical data. For instance, Varambally et al. (2005) reported the signatures of metastatic progression through integrative genomic and proteomic analysis of prostate cancer. Ouyang, Zhou and Wong (2009) proposed a predictive model to integrate ChIP-Seq and RNA-Seq to capture cooperation among regulators. Chen, Slack and Zhao (2013) developed a statistical framework for joint analysis of expression profiles of microRNA and messenger RNAs from multiple cancers. Troyanskaya et al. (2003) proposed a Bayesian framework for combining heterogeneous data sources for gene function prediction in Saccharomyces cerevisiae. Myers and Troyanskaya (2007) developed a network prediction approach to leveraging biological context information, and applied it to Saccharomyces cerevisiae. Myers et al. (2005) proposed a Bayesian approach to identifying biological networks from diverse functional genomic data. Myers et al. (2006) evaluated several evaluation methods, and suggested a new approach to evaluation in functional genomics. Shen and Tseng (2010) developed a new meta-analysis approach to pathway enrichment analysis when combining multiple genomic studies. For more literature review, see Ge, Walhout and Vidal (2003), Hawkins, Hon and Ren (2010), Hecker et al. (2009), Joyce and Palsson (2006), Ritchie et al. (2015).

In this paper, we focus on the problem of discovering regulatory relationships among heterogeneous genomic variables from biological conditions with potentially shared regulation mechanisms. In this scenario, genomic variables can be genomic variants (for instance, mutations and copy number alterations), epigenetic states (for instance, methylation status) and gene expression profiles. Biological conditions can be different tissue types or different cancer types, etc. The heterogeneous genomic variables can be binary, categorical or continuous. Different biological systems have both shared regulations and tissue or disease specific regulations. Thus, we need a statistical method to jointly learn conditional independence among a set of discrete or continuous variables across a set of distinct but related conditions. Conditional independence among variables can be represented by a graphical model in which nodes represent variables and the absence of an edge between two variables implies conditional independence.

In recent years, many efforts have been devoted to estimating undirected graphical models, especially in the high-dimensional setting under the assumption that the underlying graph is sparse. In most of the published work, the nodes in the graphical models represent either continuous or discrete variables, but not both. In the case of continuous variables, much interest has been focused on estimating Gaussian graphical models of the relationships among a set of random variables with a joint multivariate normal distribution, where zero entries in the precision (or concentration) matrix correspond to conditional independence. Meinshausen and Bühlmann (2006) proposed to estimate the precision matrix via a marginal penalized regression approach. Peng, Zhou and Zhu (2009) extended this approach to estimate partial correlations of Gaussian random variables by joint sparse regression models. Instead of performing regressions, Yuan and Lin (2006), Friedman, Hastie and Tibshirani (2008) and others took a penalized log-likelihood approach. This approach has been extended by Guo et al. (2011) and Danaher, Wang and Witten (2013) to infer multiple Gaussian graphical models based on data collected from distinct but related conditions such as different cancer types. Yin and Li (2011) and Li, Chun and Zhao (2012) considered external effects on the inferred edges through modeling conditional Gaussian graphical models. Chun et al. (2013) proposed joint conditional Gaussian graphical models with multiple sources of genomic data. In the case of discrete variables, the Ising model can be used to model conditional independence. Höfling and Tibshirani (2009) proposed a pseu-dolikelihood approach to estimating the sparse binary pairwise Markov networks. Ravikumar, Wainwright and Lafferty (2010) and Guo et al. (2010) formulated the model selection methods for high-dimensional Ising models under a penalized logistic regression framework. In the case that both discrete and continuous variables are considered, Lauritzen (1996) proposed a mixed graphical model in the low-dimensional setting. Recently, several methods have been proposed to estimate a mixed graphical model in the high-dimensional setting. Lee and Hastie (2012) proposed a pairwise graphical model over continuous and discrete variables using a group lasso penalty. Cheng, Levina and Zhu (2013) provided an approach that substitutes the l₁ penalty for the group lasso penalty to reduce computation. Fellinghauer et al. (2013) took a random forests approach to mixed variables. Chen, Witten and Shojaie (2015) and Yang et al. (2013) investigated the pairwise graphical model in which the conditional distribution of the nodes belong to an exponential family.

However, in the scenario of multiple networks with mixed types of measurements, for instance, multiple cancer types with copy number variations and mutation measurements, the methods mentioned above are not suitable to be applied directly to gain biologically interpretable results. For instance, if we simply treat biological conditions (cancer types in the example) as categorical variables with equal roles as mutations, we may end up with a network modeling interactions among cancer types. These interactions are not as biologically meaningful as the interactions among mutations and/or copy number variations. Moreover, if we ignore the similarities among biological conditions and estimate the networks separately, we may get less accurate networks. Thus, there is a need to treat biological conditions and genomic measurements differently, and to discover multiple related mixed graphical models in the high-dimensional setting to represent distinct but related relationships under different conditions. We will use cancer genomic data as a motivating example to illustrate our method.

Cancers are complex diseases involving many different mechanisms. High-throughput technologies applied to human cancers have generated large genomic datasets, such as The Cancer Genome Atlas (TCGA) [Tomczak, Czerwińska and Wiznerowicz (2015)]. TCGA provides molecular landscapes of thousands of human cancers at multiple layers, including mutations and copy number alterations. It facilitates the study of regulatory mechanisms underlying various cancers. For example, Ciriello et al. (2013) identified distinct oncogenic processes as well as unexpected similarities among tumors originating from different tissues. However, the molecular regulatory networks underlying cancers are still largely unknown, impeding our understanding of cancer classifications and patient stratification, an important issue in precision medicine.

In this paper, we consider statistical learning of multiple graphical models that consist of both continuous and discrete variables, and develop a method named as Data Integration through Graphical models (DIG). We formally introduce our model in Section 2, and propose appropriate penalty schemes in Section 3 to handle high-dimensional data. Then the problem of estimating multiple mixed graphical model is formulated into an optimization problem. We propose an algorithm for parameter estimation in Section 4 and tuning parameter selection in Section 5. We then illustrate our method through simulations in Section 6 and real application to cancer genomic data in Section 7. We conclude our paper with discussion in Section 8.

2. Model.

We assume that there are a total of K groups where we have observations consisting of both continuous and discrete variables from each group. Let (x_p×1, y_q×1)_k denote a mixed (i.e., having both continuous and discrete variables) random vector, where k is the group label (such as tissue or disease), x_p×1 is a p-dimensional vector of discrete variables, and y_q×1 is a q-dimensional vector of continuous variables. We assume that the density function f (x, y) has the following form proposed by Lauritzen (1996), with k omitted for simplicity:

$$f(\mathbf{\text{x}},\mathbf{\text{y}})=\exp \left(g_{\mathbf{\text{x}}}+{\mathbf{\text{h}}}_{\mathbf{\text{x}}}^{\mathbf{\top }}\mathbf{\text{y}}-\frac{1}{2}{\mathbf{\text{y}}}^{\mathbf{\top }}{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}\mathbf{\text{y}}\right),$$ (2.1)

where g_x is a real-valued function of x, h_x is a q-vector-valued function of x taking discrete values, and Ω_x is a q × q positive definite symmetric matrix of x.

One can note that equation (2.1) can be rewritten as

$$f(\mathbf{\text{x}},\mathbf{\text{y}})=\exp \left(g_{\mathbf{\text{x}}}+\frac{1}{2}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}^{\mathbf{\top }}{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}-\frac{1}{2}{\left(\mathbf{\text{y}}-{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}\right)}^{\top }{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}\left(\mathbf{\text{y}}-{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}\right)\right).$$

Thus, the density defined in equation (2.1) implies a conditional Gaussian distribution of y|x with mean ${\mathit{\mu }}_{\mathit{x}}={\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}$ and variance ${\mathbf{\text{Σ}}}_{\mathit{x}}={\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}$. The marginal distribution of the discrete variables x has the following form:

$$\mathit{P}(\mathbf{\text{x}})={(2\pi )}^{q/2}{\left(\det ({\mathbf{\text{Ω}}}_{\mathit{x}})\right)}^{-1/2}\exp \left(g_{\mathbf{\text{x}}}+\frac{1}{2}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}^{\mathbf{\top }}{\mathbf{\text{Ω}}}_{\mathbf{\text{x}}}^{-1}{\mathbf{\text{h}}}_{\mathbf{\text{x}}}\right).$$

We further simplify equation (2.1) by ignoring all interaction terms between discrete variables of order higher than two, and assuming that discrete variables affect continuous variables in a linear form, that is, the conditional covariance matrix and the canonical mean vector of the continuous (Gaussian) variables is modeled as a linear function of the discrete variables. Therefore, we have the following specifications for the functional forms of g_x, h_x and Ω_x:

$$\begin{gathered} g_{\mathit{x}}={\lambda }_0+\sum _{j=1}^p{\lambda }_j(x_j)+\sum _{j\ne m}{\lambda }_{jm}(x_j,x_m), \\ {\mathbf{h}}_{\mathbf{x}}={\mathbf{\eta }}_0+\sum _{j=1}^p{\mathbf{\eta }}_j(x_j), \\ {\mathbf{\Omega }}_{\mathit{x}}={\mathbf{\Phi }}_0+\sum _{j=1}^p{\mathbf{\Phi }}_j(x_j), \end{gathered}$$

where λ₀ is the normalizing constant,

$$\begin{gathered} {\text{λ}}_0=-\frac{q}{2}\ln (2\pi ) \\ -\ln \left\{\sum _x\det {({\mathbf{\text{Ω}}}_{\mathbf{\text{x}}})}^{-1/2}\exp \left(\sum _j{\text{λ}}_j(x_j)+\sum _{j\ne m}{\text{λ}}_{jm}(x_j,x_m)+\frac{1}{2}{\mathbf{h}}_{\mathbf{x}}^{\mathbf{\top }}{\mathbf{\text{Ω}}}_{\mathbf{x}}^{-1}{\mathbf{h}}_{\mathbf{x}}\right)\right\}; \end{gathered}$$

each x_j takes integer values 1 to L_j; λ_j (·) is a discrete function taking on L_j possible values; λ_jm(·, ·) is a bi-variate function with L_j × L_m possible values, and λ_jm(x_j, x_m) = λ_mj (x_m, x_j); η₀ is a q-dimensional vector; η_j(·) is a q-dimensional function with L_j possible values for each dimension; Φ₀ is a q × q matrix; and Φ_j (·) is a q × q matrix with each element having L_j possible values, which ${\{\text{diag}({\mathrm{\Phi }}_j)\}}_{j=1}^p=\{{\mathrm{\Phi }}_{jrr}:j=1,\dots ,p,r=1,\dots ,q\}$ are all 0. For identifiability, we set λ_j(1) = 0, η_jr(1) = 0, Φ_jrs(1) = 0 and λ_jm(1, ·) = λ_jm(·, 1) = 0, for j ∈ {1, … , p}, m ∈ {1, … , p}, r ∈ {1, … , q} and s ∈ {1, … , q}.

Now, the model is parametrized by λ_j, λ_jm, η₀, η_j, Φ₀ and Φ_j, where j ∈ {1, … , p}, m ∈ {1, … , p}. For simplicity, we use Θ to denote the collection of the parameters mentioned above. Among these parameters, λ_j and η₀ are nuisance parameters, which refer to discrete and continuous node potentials, respectively. The rest are responsible for edge potentials, which are the parameters of interest. Specifically, ${\mathrm{\Phi }}_{0rs}+{\sum }_{j=1}^p{\mathrm{\Phi }}_{jrs}(x_j)$ is the continuous-continuous edge potential; λ_jm is the discrete–discrete edge potential, which is a bivariate function taking on L_j × L_m values; η_jr (x_j) is the continuous–discrete edge potential, which takes L_j values.

The model covers the two special situations when there are only discrete or continuous variables naturally. In the case of only discrete variables, the mixed model reduces to a discrete Markov random field,

$$p(\mathbf{\text{x}})\propto \exp \left\{\sum _{j=1}^p{\text{λ}}_j(x_j)+\sum _{j=1}^p\sum _{m=1}^p{\text{λ}}_{jm}(x_j,x_m)\right\};$$

while in the case of only continuous variables, the mixed model reduces to a multivariate Gaussian graphical model,

$$p(\mathbf{\text{y}})\propto \exp \left\{-\frac{1}{2}{(\mathbf{\text{y}}-{{\mathbf{\Phi }}_{\mathbf{0}}}^{-1}{\mathit{\eta }}_{\mathbf{0}})}^{\mathbf{\top }}{\mathbf{\Phi }}_{\mathbf{0}}(\mathbf{\text{y}}-{{\mathbf{\Phi }}_{\mathbf{0}}}^{-1}{\mathit{\eta }}_{\mathbf{0}})\right\}.$$

For a graphical model, the conditional distributions are important because they characterize the conditional dependence among the variables. The conditional distributions of the proposed model are as follows:

1. The conditional distribution of x_j given the rest is multinomial,

   $$\begin{gathered} p(x_j|\mathbf{\text{x}}\backslash j,\mathbf{\text{y}};\mathbf{\text{Θ}}) \\ =\frac{\exp \left\{{\text{λ}}_j(x_j)+{\sum }_{m=1}^p{\text{λ}}_{jm}(x_j,x_m)+{\sum }_{r=1}^q{\eta }_{jr}(x_j)y_r-\frac{1}{2}{\sum }_{r=1}^q{\sum }_{s=1}^q{\mathrm{\Phi }}_{jrs}(x_j)y_r{y}_s\right\}}{{\sum }_{l=1}^{L_j}\exp \left\{{\text{λ}}_j(l)+{\sum }_{m=1}^p{\text{λ}}_{jm}(l,x_m)+{\sum }_{r=1}^q{\eta }_{jr}(l)y_r-\frac{1}{2}{\sum }_{r=1}^q{\sum }_{s=1}^q{\mathrm{\Phi }}_{jrs}(l)y_r{y}_s\right\}}. \end{gathered}$$
2. The conditional distribution of y_r given the rest is Gaussian,

   $$\begin{gathered} p(y_r|\mathbf{\text{y}}\backslash r,\mathbf{\text{x}};\mathbf{\text{Θ}}) \\ =\sqrt{\frac{{\mathrm{\Omega }}_{xrr}}{2\pi }}\exp \left\{-\frac{{\mathrm{\Omega }}_{xrr}}{2}{\left[\frac{{\eta }_{0r}+{\sum }_{j=1}^p{\eta }_{jr}(x_j)-{\sum }_{s\ne r}{\mathrm{\Omega }}_{xrs}{y}_s}{{\mathrm{\Omega }}_{xrr}}-y_r\right]}^2\right\}, \end{gathered}$$

   where ${\mathrm{\Omega }}_{xrs}={\mathrm{\Phi }}_{0rs}+{\sum }_{j=1}^p{\mathrm{\Phi }}_{jrs}(x_j)$. This implies a regression model as

   $$\begin{gathered} y_r=\frac{1}{{\mathrm{\Omega }}_{xrr}}({\eta }_{0r}+\sum _{j=1}^p{\eta }_{jr}(x_j) \\ -\sum _{s\ne r}\left({\mathrm{\Phi }}_{0rs}+\sum _{j=1}^p{\mathrm{\Phi }}_{jrs}(x_j)\right)y_s)+e_r,\text{\hspace{1em}\hspace{1em}\hspace{1em}where\hspace{0.17em}}{e}_r~N\left(0,{\mathrm{\Omega }}_{xrr}^{-1}\right). \end{gathered}$$

We assume that the observations from different classes, labeled by k where k varies from 1 to K, are independent. Given the observed data ${\left\{{\mathbf{\text{x}}}^{i(k)},{\mathbf{\text{y}}}^{i(k)}\right\}}_{i=1}^{n_k}$ for class k with n_k samples, the negative log-likelihood for class k is

$$\tilde{\ell }\left({\mathbf{\text{Θ}}}^{(k)}\right)=-\sum _{i=1}^{n_k}\log f({\mathbf{\text{x}}}^{i(k)},{\mathbf{\text{y}}}^{i(k)};{\mathbf{\text{Θ}}}^{(k)}).$$

The minimization of the negative log-likelihood incorporates the calculation of the normalization scalar. Because the mixed model includes the discrete model as a special case which is computationally intractable, directly minimizing (2.2) is challenging and impractical. Instead, we propose to use a computationally efficient and consistent estimation approach, the pseudolikelihood method, which is formed by the product of all conditional distributions as below:

$$\begin{gathered} \ell \left({\mathbf{\text{Θ}}}^{(k)}|{\mathbf{\text{x}}}^{(k)},{\mathbf{\text{y}}}^{(k)}\right)=-\sum _{i=1}^{n_k}(\sum _{j=1}^p\log p\left(x_j^{i(k)}|{\mathbf{\text{x}}}_{\backslash j}^{i(k)},{\mathbf{\text{y}}}^{i(k)};{\mathbf{\text{Θ}}}^{(k)}\right) \\ +\sum _{r=1}^q\log p\left(y_r^{i(k)}|{\mathbf{\text{x}}}^{i(k)},{\mathbf{\text{y}}}_{\backslash r}^{i(k)};{\mathbf{\text{Θ}}}^{(k)}\right)). \end{gathered}$$ (2.3)

The model above treats different biological conditions differently from categorical biological measurements such as SNPs, and estimates multiple biological networks from different biological conditions jointly. Mathematically, one may think of a possibility as treating the biological conditions and the categorical biological measurements such as SNPs equally, and make estimation through one mixed graphical model. This approach may result in edges among biological conditions. Such a network is biologically hard to interpret. Thus, we propose to use a framework of joint mixed graphical models instead of treating biological conditions equal to categorical biological measurements and estimating one mixed graphical model.

Another possibility one may think of is to estimate the networks from different biological conditions separately instead of jointly. As illustrated through a toy example that consists of observations from two classes following two normal distributions with distinct covariance matrices in Danaher, Wang and Witten (2013), estimating networks separately in each class results in less accurate estimates than estimating networks jointly. Thus, we adapt a joint graphical model approach in our application scenario.

It is easy to show that the additive negative log-pseudolikelihood is jointly convex in all the parameters {Θ^(k)} over the region $\left\{{\mathrm{\Omega }}_{xrr}^{(k)}>0\right\}$ [see the Supplementary Material, Zhang, Ouyang and Zhao (2017)].

3. Penalty terms.

In the graphical representation of probability distributions, the absence of an edge between two variables corresponds to conditional independence between the two variables. In the proposed mixed model for each class (with k omitted for simplicity), there are three types of edges:

1. Discrete–discrete: If λ_jm(x_j, x_m) = 0 for all values of x_j and x_m, then there is no edge between nodes x_j and x_m. The corresponding edge potential is in either $p(x_j|{\mathbf{\text{x}}}_{\backslash j},\mathbf{\text{y}},\mathbf{\text{Θ}})$ or $p(x_m|\mathbf{\text{x}}{\text{}}_m,\mathbf{\text{y}};\mathbf{\text{Θ}})$ of equation (2.3).

2. Discrete–continuous: If η_jr = 0 for all values of x_j and Φ_jrs = 0 for all values of y_s , s ∈ {1, … , q}, then there is no edge between nodes x_j and y_r. The corresponding edge potential parameter is in either $p(x_j|{\mathbf{\text{x}}}_{\backslash j},\mathbf{\text{y}},\mathbf{\text{Θ}})$ or $p(y_r|\mathbf{\text{x}},{\mathbf{\text{y}}}_{\backslash r};\mathbf{\text{Θ}})$ of equation (2.3).

3. Continuous–continuous: If Φ_0rs = 0 for all values of y_r and y_s , and Φ_jrs = 0 for all values of y_r, y_s and x_j, then there is no edge between nodes y_r and y_s. These parameters are in either $p(y_r|\mathbf{\text{x}},{\mathbf{\text{y}}}_{\backslash r};\mathbf{\text{Θ}})$ or $p(y_s|\mathbf{\text{x}},{\mathbf{\text{y}}}_{\backslash s};\mathbf{\text{Θ}})$ of equation (2.3).

In summary, we have the following equivalences:

1. $x_j\perp x_m|(\mathbf{\text{x}}{\text{}}_j,\mathbf{\text{x}}{\text{}}_m,\mathbf{\text{y}})\iff {\text{λ}}_{jm}=0,$

2. $x_j\perp y_r|\{\mathbf{\text{x}}{\text{}}_j,\mathbf{\text{y}}{\text{}}_r\}\iff {\mathbf{\eta }}_{jr}=0$ and $\forall s\in \left\{1,\dots ,q\right\},{\mathbf{\Phi }}_{jrs}=0,$

3. $y_r\perp y_s|\{\mathbf{\text{y}}{\text{}}_r,\mathbf{\text{y}}{\text{}}_s,\mathbf{\text{x}}\}\iff {\mathrm{\Phi }}_{0rs}=0$ and $\forall j\in \left\{1,\dots ,p\right\},{\mathbf{\Phi }}_{jrs}=0.$

To estimate the parameters for high-dimensional data, we assume that the underlying true graph is sparse, and incorporate penalization on the number of edges in the minimization of the additive negative log-pseudolikelihood to obtain a sparse graphical model. For the l₀-norm, which is defined as the number of nonzero elements in a vector, that is, ${\Vert \mathbf{\text{u}}\Vert }_0:=\{\#i,\text{\hspace{0.17em}s}.\text{t}.{\mathbf{\text{u}}}_i\ne 0\}={\lim }_{q\to 0^{+}}({\sum }_{i=1}^n{\left|{\mathbf{\text{x}}}_i\right|}^q)$, we may use the l₀ penalty to infer the graphical model for each class separately. Also, we notice that, for edges involving discrete variables, the absence of that edge requires the entire matrix λ_jm (for discrete–discrete), or matrix Φ_jrs and vector η_jr (for discrete–continuous) to be 0. Specifically, we set up the following optimization problem (with k omitted for simplicity):

$$\begin{gathered} \underset{\mathbf{\text{Θ}}}{\min }\ell (\mathbf{\text{Θ}})+\rho (\sum _{j\ne m}\mathbb{I}({\mathbf{\text{λ}}}_{jm}\ne \mathbf{0})+\sum _{j=1}^p\sum _{r=1}^q\mathbb{I}({\mathit{\eta }}_{jr}\ne \mathbf{0}) \\ +\sum _{r\ne s}\mathbb{I}({\mathrm{\Phi }}_{0rs}\ne 0)+\sum _{j=1}^p\sum _{r\ne s}\mathbb{I}({\mathbf{\Phi }}_{jrs}\ne \mathbf{0})), \end{gathered}$$ (3.1)

where λ_jm is an L_j × L_m matrix, η_jr is a vector with length L_j, Φ_jrs is a vector with length L_j, Φ_0rs is a scalar, and ρ is a tuning parameter. The function involved is integer valued and nonconvex, and it is generally hard to solve the optimization. In the machine learning literature, such kinds of difficult optimization problems are usually solved through appropriate relaxation, for example, Cheng, Levina and Zhu (2013). Notice that, for any vector b, $\mathbb{I}(\text{b}\ne \mathbf{0})=0\iff {\Vert \mathbf{\text{b}}\Vert }_2=0$ and for any matrix B, $\mathbb{I}(\mathbf{\text{B}}\ne \text{0})=0\iff {\Vert \mathbf{\text{B}}\Vert }_F=0$. Thus, we can replace the l₀ norm in optimizing (3.1) with an appropriate convex relation, for example,

$$\underset{\mathbf{\text{Θ}}}{\min }\ell (\mathbf{\text{Θ}})+\rho \left(\sum _{j\ne m}{\Vert {\text{λ}}_{jm}\Vert }_F+\sum _{j=1}^p\sum _{r=1}^q{\Vert {\mathit{\eta }}_{jr}\Vert }_2+\sum _{r\ne s}\left|{\mathrm{\Phi }}_{0rs}\right|+\sum _{j=1}^p\sum _{r\ne s}{\Vert {\mathrm{\Phi }}_{jrs}(x_j)\Vert }_2\right).$$

We also notice that, for any vector b, ${\Vert \mathbf{\text{b}}\Vert }_2\le {\Vert \mathbf{\text{b}}\Vert }_1$, and for any matrix B = (b_ij), ${\Vert \mathbf{\text{B}}\Vert }_F\le {\sum }_{ij}\left|b_{ij}\right|$. Thus, we can replace ${\Vert \cdot \Vert }_2$ and ${\Vert \cdot \Vert }_F$ with the corresponding upper bound penalties, leading to the following optimization problem:

$$\begin{gathered} \underset{\mathbf{\text{Θ}}}{\min }\ell (\mathbf{\text{Θ}})+\rho (\sum _{j\ne m}\sum _{x_j=1}^{L_j}\sum _{x_m=1}^{L_m}\left|{\text{λ}}_{jm}(x_j,x_m)\right| \\ +\sum _{j=1}^p\sum _{r=1}^q\sum _{x_j=1}^{L_j}\left|{\eta }_{jr}(x_j)\right|+\sum _{r\ne s}\left|{\mathrm{\Phi }}_{0rs}\right|+\sum _{j=1}^p\sum _{r\ne s}\sum _{x_j=1}^{L_j}\left|{\mathrm{\Phi }}_{jrs}(x_j)\right|). \end{gathered}$$ (3.2)

For simplicity, we let C denote the indices such that (θ_uv)_{u∈C,v ∈C} contain parameters {λ_jm}, {η_j }, {Φ₀} and {Φ_j} only. Then problem (3.2) can be written as

$$\underset{\mathbf{\text{Θ}}}{\min }\ell (\mathbf{\text{Θ}})+\rho \sum _{u\in C}\sum _{v\in C}\left|{\theta }_{uv}\right|.$$

After discussing the optimization function for a single class, we now formulate our problem for joint analysis of multiple classes. The basic assumption for joint graphical model analysis across different biological conditions is that there are commonalities shared among multiple classes. We propose two penalization approaches (fused lasso and group lasso) to encourage borrowing information from multiple biological conditions for the estimation of the joint mixed graphical models:

$$\underset{{\mathbf{\text{Θ}}}^{(1)},\dots ,{\mathbf{\text{Θ}}}^{(K)}}{\arg \min }\sum _{k=1}^K\ell ({\mathbf{\text{Θ}}}^{(k)})+P({\mathbf{\text{Θ}}}^{(1)},\dots ,{\mathbf{\text{Θ}}}^{(K)}).$$ (3.3)

Specifically, we define the penalty terms as follows.

In the case of the fused graphical lasso,

$$\begin{gathered} P\left(\left\{{\mathbf{\text{Θ}}}^{(k)}\right\}\right)={\rho }_1\sum _{k=1}^K(\sum _{j\ne m}\sum _{x_j=1}^{L_j}\sum _{x_m=1}^{L_m}\left|{\text{λ}}_{jm}^{(k)}(x_j,x_m)\right| \\ +\sum _{j=1}^p\sum _{r=1}^q\sum _{x_j=1}^{L_j}\left|{\eta }_{jr}^{(k)}(x_j)\right|+\sum _{r\ne s}\left|{\mathrm{\Phi }}_{0rs}^{(k)}\right|+\sum _{j=1}^p\sum _{r\ne s}\sum _{x_j=1}^{L_j}\left|{\mathrm{\Phi }}_{jrs}^{(k)}(x_j)\right|) \\ +{\rho }_2\sum _{k<k\prime }(\sum _{j\ne m}\sum _{x_j=1}^{L_j}\sum _{x_m=1}^{L_m}\left|{\text{λ}}_{jm}^{(k)}(x_j,x_m)-{\text{λ}}_{jm}^{(k\prime )}(x_j,x_m)\right| \\ +\sum _{r,s}\left|{\mathrm{\Phi }}_{0rs}^{(k)}-{\mathrm{\Phi }}_{0rs}^{(k\prime )}\right|+\sum _{j=1}^p\sum _{r=1}^q\sum _{x_j=1}^{L_j}\left|{\eta }_{jr}^{(k)}(x_j)-{\eta }_{jr}^{(k\prime )}(x_j)\right| \\ +\sum _{j=1}^p\sum _{r\ne s}\sum _{x_j=1}^{L_j}\left|{\mathrm{\Phi }}_{jrs}^{(k)}(x_j)-{\mathrm{\Phi }}_{jrs}^{(k\prime )}(x_j)\right|) \\ ={\rho }_1\sum _{k=1}^K\sum _{u\in C}\sum _{v\in C}\left|{\theta }_{uv}^{(k)}\right|+{\rho }_2\sum _{k<k\prime }\sum _{u\in C}\sum _{v\in C}\left|{\theta }_{uv}^{(k)}-{\theta }_{uv}^{(k\prime )}\right|, \end{gathered}$$

where ρ₁ and ρ₂ are tuning parameters. The fused graphical lasso penalty implies that graphs from multiple biological conditions are the same except for a few edges.

In the case of the group graphical lasso,

$$\begin{gathered} P\left(\left\{{\mathbf{\text{Θ}}}^{(k)}\right\}\right)={\rho }_1\sum _{k=1}^K(\sum _{j\ne m}\sum _{x_j=1}^{L_j}\sum _{x_m=1}^{L_m}\left|{\text{λ}}_{jm}^{(k)}(x_j,x_m)\right| \\ +\sum _{j=1}^p\sum _{r=1}^q\sum _{x_j=1}^{L_j}\left|{\eta }_{jr}^{(k)}(x_j)\right|+\sum _{r\ne s}\left|{\mathrm{\Phi }}_{0rs}^{(k)}\right|+\sum _{j=1}^p\sum _{r\ne s}\sum _{x_j=1}^{L_j}\left|{\mathrm{\Phi }}_{jrs}^{(k)}(x_j)\right|) \\ +{\rho }_2(\sum _{j\ne m}\sum _{x_j=1}^{L_j}\sum _{x_m=1}^{L_m}\sqrt{\sum _{k=1}^K{({\text{λ}}_{jm}^{(k)}(x_j,x_m))}^2}+\sum _{r,s}\sqrt{\sum _{k=1}^K{({\mathrm{\Phi }}_{jrs}^{(k)})}^2} \\ +\sum _{j=1}^p\sum _{r=1}^q\sum _{x_j=1}^{L_j}\sqrt{\sum _{k=1}^K{({\eta }_{jr}^{(k)}(x_j))}^2}+\sum _{j=1}^p\sum _{r\ne s}\sum _{x_j=1}^{L_j}\sqrt{\sum _{k=1}^K{({\mathrm{\Phi }}_{jrs}^{(k)}(x_j))}^2}) \\ ={\rho }_1\sum _{k=1}^K\sum _{u\in C}\sum _{v\in C}\left|{\theta }_{uv}^{(k)}\right|+{\rho }_2\sum _{u\in C}\sum _{v\in C}\sqrt{\sum _{k=1}^K{\theta }_{uv}^{{(k)}^2}}, \end{gathered}$$

where ρ₁ and ρ₂ are tuning parameters. The group graphical lasso penalty treats the related biological conditions as one group, and implies that the underlying multiple graphs are the same.

4. Algorithm.

In this section, we focus on the numerical algorithms to solve the optimization problem proposed above. This constrained optimization problem can be simplified and solved by replacing it with a series of distributed problems through an augmented Lagrangian scheme. We first make the objective function separable by rewriting (3.3) as

$$\underset{\text{{}{\mathbf{\text{Θ}}}^{(k)}\},\{{\mathbf{\text{Z}}}^{(k)}\}}{\arg \min }\sum _{k=1}^K\ell ({\mathbf{\text{Θ}}}^{(k)})+P(\mathbf{\text{Z}}),$$ (4.1)

subject to the constraint that Z^(k) = Θ^(k) for k = 1, … , K , where {Z} = {Z⁽¹⁾, … , Z^(K)}. Then we carry out the function optimization and regularization locally and coordinate them globally via constraints by further rewriting problem (4.1) using the scaled augmented Lagrangian [Boyd et al. (2011), Hestenes (1969)] as

$$L_{\rho }\left(\left\{\mathbf{\text{Θ}}\right\},\left\{\mathbf{\text{Z}}\right\},\left\{\mathbf{\text{U}}\right\}\right)=\sum _{k=1}^K\ell ({\mathbf{\text{Θ}}}^{(k)})+P(\mathbf{\text{Z}})+\frac{d}{2}\sum _{k=1}^K{\Vert {\mathbf{\text{Θ}}}^{(k)}-{\mathbf{\text{Z}}}^{(k)}+{\mathbf{\text{U}}}^{(k)}\Vert }_F^2,$$ (4.2)

where U = {U⁽¹⁾, … , U^(K)} are the dual feasibility-tolerance variables, and d is a scalar. The augmented Lagrangian optimization problem (4.2) can be solved by the alternating direction method of multipliers (ADMM), which guarantees to con-verge to the global optimum [Boyd et al. (2011)]. The skeleton of the algorithm at the ith iteration includes the following three steps:

1. $\left\{{\mathbf{\text{Θ}}}_{(i)}\right\}\leftarrow \arg {\min }_{\left\{\mathbf{\text{Θ}}\right\}}{L}_d(\{\mathbf{\text{Θ}}\},\{{\mathbf{\text{Z}}}_{(i-1)}\},\{{\mathbf{\text{U}}}_{(i-1)}\}),$

2. $\left\{{\mathbf{\text{Z}}}_{(i)}\right\}\leftarrow \arg {\min }_{\left\{\mathbf{\text{Z}}\right\}}{L}_d(\{{\mathbf{\text{Θ}}}_{(i)}\},\{\mathbf{\text{Z}}\},\{{\mathbf{\text{U}}}_{(i-1)}\}),$

3. $\left\{{\mathbf{\text{U}}}_{(i)}\right\}\leftarrow \left\{{\mathbf{\text{U}}}_{(i-1)}\right\}+(\{{\mathbf{\text{Θ}}}_{(i)}\}-\{{\mathbf{\text{Z}}}_{(i)}\}).$

Please refer to Supplementary Material for details [Zhang, Ouyang and Zhao (2017)]. Briefly, to estimate Θ, we use a coordinate-wise descent approach to obtain each parameter in Θ, and directly apply a well-suited proximal gradient algorithm, which can achieve ε-optimality within $O(1/\sqrt{\epsilon })$ iterations. The convergence rates and properties of proximal gradient algorithms and their accelerated variants have been well studied [Auslender and Teboulle (2006), Beck and Teboulle (2009)]. To update Z, the optimization problem is separable with respect to each pair of elements in the matrix, and thus can be solved using the fused lasso signal approximator in Hoefling (2010) or the group lasso operator in Friedman, Hastie and Tibshirani (2010) depending on the choice of penalty P.

We note that separate regressions were used in estimating a single graphi-cal model, including the Gaussian graphical model [Meinshausen and Bühlmann (2006)], the Ising model [Ravikumar, Wainwright and Lafferty (2010)] and the mixed graphical model [Chen, Witten and Shojaie (2015), Cheng, Levina and Zhu (2013), Yang et al. (2013)]. The regression-type approach is computationally convenient by virtue of effective regression tools, such as glmnet [Friedman, Hastie and Tibshirani (2009)]. However, node-wise regression yields asymmetric estimates of edge potentials for an undirected graph, which results in an arbitrary or ad hoc selection of estimates of parameters. The computational diagram employed in our approach can yield symmetric estimates of edge potentials for the undirected graphs. In this respect, the proposed optimization method outperforms the simple approach to parameter estimation via separate node-wise regression. Moreover, as discussed in Section 2, it is not appropriate to treat biological conditions as discrete measurements in the modeling and estimate one mixed graph. Furthermore, estimating networks separately in each class can result in less accurate estimates than estimating networks jointly [Danaher, Wang and Witten (2013)]. Thus, we use the proposed symmetric pseudolikelihood method to jointly estimate mixed graphical models.

It is also notable that our model covers a special situation when there are only continuous variables across multiple biological conditions, which was studied by joint graphical lasso (JGL) [Danaher, Wang and Witten (2013)]. In this simple case, one only needs to estimate precision matrices of multivariate Gaussian random variables, which results in estimated concentration graphs. It has been shown that the thresholded sample covariance graph induces the same connected components as those induced by the estimated concentration graph under the same regularization parameter [Mazumder and Hastie (2012), Witten, Friedman and Simon (2011)]. With this nice property of a path of graphical lasso solutions, it can result in a faster computation by employing screenings of empirical covariance matrices to determine whether the solution to concentration graphs is block diagonal upon feature permutation, and by performing the JGL algorithm on the features within each block separately [Danaher, Wang and Witten (2013)].

5. Tuning parameter selection.

For the selection of tuning parameters, we propose the following Bayesian information criterion (BIC) type of approach:

$$\text{BIC(}{\rho }_1\text{,\hspace{0.17em}}{\rho }_2\text{)=}-2\sum _{k=1}^K\ln l\left({\mathbf{\text{Θ}}}_{{\rho }_1\text{,\hspace{0.17em}}{\rho }_2}^{(k)}\right)+E_k\ln (n_k),$$

where $l\left({\mathbf{\text{Θ}}}_{{\rho }_1\text{,\hspace{0.17em}}{\rho }_2}^{(k)}\right)$ is the pseudolikelihood for the observations from the kth class with the tuning parameters ρ₁ and ρ₂, and E_k is the number of edges in the kth mixed graphical model. It is notable that the proposed BIC-type approach departures from classical BIC approaches by using the pseduolikelihood rather than likelihoods.

We notice that the Akaike information criterion (AIC) approach has been used for the selection of Gaussian graphical models [Danaher, Wang and Witten (2013)]. Analogously, one may use the following AIC-type approach for model selection in our research scenario using the pseudolikelihood:

$$\text{AIC(}{\rho }_1\text{,\hspace{0.17em}}{\rho }_2\text{)=}-2\sum _{k=1}^K\ln l\left({\mathbf{\text{Θ}}}_{{\rho }_1\text{,\hspace{0.17em}}{\rho }_2}^{(k)}\right)+2E_k.$$

We compared the two model selection criteria through simulation studies as shown in Section 6. Our analysis suggests that the AIC-type approach tends to choose too large but less accurate models compared to the proposed BIC-type criterion. Thus, we use the proposed BIC-type approach in the real application as illustrated in Section 7.

6. Simulations.

We demonstrate the performance of our approach through simulations. Without loss of generality, in the following and for the simplicity of simulation, we focus on Ω_x = Ω, that is, Φ_jrs(x_j) = 0 for any j ∈ {1, … , p}, r ∈ {1, … , q}, s ∈ {1, … , q} and x_j ∈ {1, … , L_j }. With this assumption, the covariance matrix for the continuous variables is independent of the values of the discrete variables for the same class.

6.1. Random networks.

We first considered two mixed graphical models (representing two classes), each consisting of 10 categorical (with two levels, 1 and 2) and 10 Gaussian variables. The topologies for the two simulated networks are shown in Figure 1. The left panel of Figure 1 is the adjacent matrix of the mixed graphical model of the first class, while the right panel represents the second class. The first 10 rows/columns correspond to discrete variables, while the second 10 rows/columns correspond to continuous variables. The degree distributions are al-most uniform, which are similar to those in the synthetic experiments in Lee and Hastie (2012). Based on the edge sets defined in Figure 1, we assigned nonzero potentials on the edges for each mixed graphical model as follows.

Fig. 1.

The true adjacency matrices of simulated random networks in Section 6.1.

First, we generated a p × p edge potential matrix connecting discrete variables as below:

$${\text{λ}}_{jm}(x_j,x_m)=\{\begin{array}{l}0.5\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=1\text{\hspace{0.17em}and\hspace{0.17em}}{x}_m=1,\hfill \\ -0.5\text{\hspace{1em}\hspace{0.5em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=2\text{\hspace{0.17em}and\hspace{0.17em}}{x}_m=1,\hfill \\ -0.5\text{\hspace{1em}\hspace{0.5em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=1\text{\hspace{0.17em}and\hspace{0.17em}}{x}_m=2,\hfill \\ 0.5\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=2\text{\hspace{0.17em}and\hspace{0.17em}}{x}_m=2,\hfill \\ 0\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.5em}if }(j,m)\notin E.\hfill \end{array}$$ (6.1)

Second, we assigned 2p × q elements for an edge potential matrix connecting discrete and continuous random variables as follows:

$${\eta }_{jr}(x_j)=\{\begin{array}{l}1\text{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=1,\hfill \\ -1\text{\hspace{1em}\hspace{1em}\hspace{0.5em}if }(j,m)\in E\text{\hspace{0.17em}and\hspace{0.17em}}{x}_j=2,\hfill \\ 0\text{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(j,m)\notin E.\hfill \end{array}$$ (6.2)

Third, we generated a precision matrix Ω = (ω_rs) of continuous variables as below:

$${\omega }_{rs}=\{\begin{array}{l}0.25\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(r,s)\in E\text{\hspace{0.17em}and\hspace{0.17em}}r\ne s,\hfill \\ 1\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if \hspace{0.17em}}r=s,\hfill \\ 0\text{\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}if }(r,s)\notin E\text{\hspace{0.17em}and\hspace{0.17em}}r\ne s.\hfill \end{array}$$

To draw samples (x, y) from the joint density f (x, y), we first drew samples x ~ f (x) of the following form:

$$f(\mathbf{\text{x}})\propto \exp \left(\sum _{j=1}^p\sum _{m=1}^p{\text{λ}}_{jm}(x_j,x_m)+\frac{1}{2}{\eta }^{⊺}(\mathbf{\text{x}}){\mathbf{\text{Ω}}}^{-1}\eta (\mathbf{\text{x}})\right)$$

with

$${(\eta (\mathbf{\text{x}}))}_r=\sum _{j=1}^p{\eta }_{jr}(x_j).$$

To overcome the difficulty with direct sampling from f (x), we adapted the Gibbs sampling approach in Lee and Hastie (2012). We drew 202,000 samples in total for discrete random variables of each mixed graphical model, and discarded the first 2000 samples which were generated in a burn-in period. Then we took one sample every 100 draws to preserve independence. After sampling x, we sampled y from the conditional distribution f (y|x), which is N (Ω⁻¹(η₀ + η(x)), Ω⁻¹) with η₀ = 0.

Using the proposed method DIG, we discovered the network structures for two classes over a range of tuning parameters. We recorded the total number of identified edges for each pair of tuning parameters and calculated the number of true positive edges and the number of false positive edges. It took 52 seconds to obtain the results using the proposed algorithm using a 3-GHz Intel Core i7 processor. For comparison, we also applied the JGL proposed by Danaher, Wang and Witten (2013) by treating the two values (1 and 2) of discrete random variables as continuous variables. Similarly, we calculated the number of true positive edges and the number of false positive edges for a range of tuning parameters of JGL. The results are shown in Figure 2. One can see that our method has better performance with both group lasso and fused lasso penalty schemes. This shows the benefit of explicitly modeling discrete and continuous variables in discovering the underlying networks.

Fig. 2.

Comparison of DIG and JGL on random networks using the simulated data in Section 6.1.

Then we investigate the performance of the tuning parameter selection procedure by checking the sensitivity and specificity of the selected model. The sensitivity and specificity are defined as below, respectively,

$$\text{sensitivity}=\frac{\text{TP}}{\text{FN}+\text{TP}},$$

$$\text{specificity}=\frac{\text{TN}}{\text{FP}+\text{TN}},$$

where TP refers to true positives, FP refers to false positives, TN refers to true negatives and FN refers to false negatives. For each pair of tuning parameters, we calculated the corresponding sensitivity, specificity and score for the proposed BIC-type model selection criterion as shown in Figure 3. Figure 3(a) shows the sensitivities of DIG with the group lasso penalty over a range of tuning parameters, while Figure 3(b) shows the sensitivities of DIG with the fused lasso penalty. Figure 3(c) shows the specificities of DIG with the group lasso penalty over a range of tuning parameters, while Figure 3(d) shows the specificities of DIG with the fused lasso penalty. Figures 3(e) and (f) show the corresponding BIC-type scores for the group lasso and fused lasso, respectively, with the selected model for each type of penalty indicated by a purple diamond. The results show that DIG achieves high sensitivities (1 for both group lasso and fused lasso penalties) and specificities (0.93 for group lasso penalty, 0.96 for fused lasso penalty) with the proposed BIC-type model selection approach. To compare, we also investigated model selection performance through the AIC-type approach. The orange circles in Figure 3 indicate the corresponding selected models for the group lasso penalty and the fused lasso penalty. In this simulation study, although the graphical models selected by the AIC-type approach can achieve as high sensitivities as the BIC-type approach, the corresponding specificities are lower (0.78 for group lasso penalty, 0.49 for fused lasso penalty) than those of the BIC-type approach. The results suggest that the AIC-type approach tends to choose a larger but less accurate model than the BIC-type approach.

Fig. 3.

Performance of DIG with a range of tuning parameter pairs on random networks using the simulated data in Section 6.1. Purple diamonds indicate the model selected by the proposed BIC procedure. Orange circles indicate the model selected by the AIC-type criterion.

6.2. Scale-free networks.

It has been shown that many real networks are scale-free, of which degree distributions follow power law. In this simulation, we investigate the performance of our approach for scale-free networks where the probability that a node has a connectivity of d is proportional to d^–γ. It has been found that the γ values for real-world networks usually vary from 2 to 3 [Albert, Jeong and Barabási (2000), Barabási and Albert (1999), Govindan and Tangmunarunkit (2000), Jeong et al. (2001), Yook, Oltvai and Barabási (2004)]. Thus, we randomly generated two networks for two classes where γ is 2.433 and 2.317, respectively. The nodes in each network consist of 10 discrete variables and 10 continuous variables. The simulated structures of the two scale-free networks guided us to assign edge potentials. We used the formula of equation 6.1 for the potentials of the edges connecting discrete variables. We used the formula of equation 6.2 for the potentials of the edges between discrete and continuous variables. For the potentials of the edges connecting continuous variables, we drew random values using the following approach such that the precision matrix Ω is a positive definite matrix. For each class, we first created a q × q matrix with ones on the diagonal and zeros on elements not corresponding to network edges. Then we drew nonzero random values on elements corresponding to edges. To obtain each of these nonzero random values, we first drew a random number a from a uniform distribution U (0, 1), then we randomly picked a number from {0.1, –0.4} with equal probability. The nonzero random values assigned to the elements corresponding to edges are 0.3a + b. Then we divided each off-diagonal element by 1.5 times the sum of the absolute values of the off-diagonal elements in its row. Finally, we added the transpose of the matrix to the matrix itself to achieve a symmetric and positive definite matrix Ω. To draw samples for each graphical model, we adapted the sample generation procedure in Section 6.1. We set the burn-in threshold as 2000 in the Gibbs sampler. For each graphical model, we took one observation every 100 draws to preserve independence, and obtained 2000 samples in total for the following investigation.

Using the proposed DIG approach, we discovered network topologies using the simulated samples over a range of tuning parameters. Figures 4(a) and (b) show the sensitivities of DIG with group lasso penalty and fused lasso penalty, respectively, while Figures 4(c) and (d) show the corresponding specificities. Figures 4(e) and (f) show the corresponding scores of the proposed BIC-type approach, with the selected model for each type of penalty indicated by a purple diamond. The results show that DIG can achieve high sensitivity (1 for both group lasso and fused lasso penalties) and specificity (0.87 for group lasso penalty, 0.86 for fused lasso penalty) with the proposed BIC-type of approach. It suggests that the proposed BIC-type model selection approach can select suitable tuning parameters. We also investigated the effects of sample sizes on the performance of DIG. With a smaller sample size of 800, we obtained sensitivity as high as 1 for both group lasso and fused lasso, with a considerable specificity of 0.61 for group lasso and 0.62 for fused lasso.

Fig. 4.

Performance of DIG with a range of tuning parameter pairs on scale-free networks using the simulated data in Section 6.2. Purple diamonds indicate the models (group lasso penalty: ρ₁ = 0.0001, ρ₂ = 0.1; fused lasso penalty: ρ₁ = 0.05, ρ₂ = 0.1) selected by the proposed BIC procedure.

We also compared our approach with JGL in Danaher, Wang and Witten (2013). The results are shown in Figure 5. In the scenario of scale-free mixed networks, our method outperforms JGL with higher true positive discovery rates and lower false positive discovery rates in both group lasso and fused lasso penalty schemes.

Fig. 5.

Comparison of DIG and JGL on scale-free networks using the simulated data in Section 6.2.

7. Application to cancer genomic data.

We applied our DIG approach to TCGA datasets of two cancer types: colorectal carcinoma (coadread) and breast invasive carcinoma (brca). We obtained the mutation and copy number variation (CNV) data compiled by Ciriello et al. (2013) for 491 coadread subjects and 488 brca subjects, respectively. We used the PI3K-mTOR-AKT pathway to illustrate our method. Our data includes 62 genes with mutation information and their corresponding copy number measurements for each subject. We treated mutations as discrete variables with two levels representing the presence and absence of mutations, and copy number variations as continuous variables. We used the fused lasso penalty to the datasets from the two cancer types, and the proposed BIC approach to choose the tuning parameters (ρ₁ = 0.3, ρ₂ = 0.5). The selected optimum tuning parameter ρ₂ = 0.5 for fused lasso indicates similarities exist in our data between coadread and brca. It took 134 seconds to obtain the results using a 3-GHz Intel Core i7 processor with our current DIG implementation. We have identified 1660 edges for the coadread class and 1632 edges for the brca class. Among the interactions of coadread, 16% of them are mutation-mutation interactions, 42.3% of them are mutation-CNV interactions, and the rest are CNV-CNV interactions. For brca, 16.2% of edges are mutation-mutation interactions, 43% of them are mutation-CNV interactions, and the rest are CNV–CNV interactions. The two tumor networks share 1584 edges. We studied the community structures or modules in the identified networks through the eigenspectrum decomposition of the modularity matrices described in Newman (2006). We found four modules in coadread with sizes of 16, 66, 22 and 20. We also found two modules in brca with sizes 58 and 66. Interestingly, the second module is shared in both coadread and brca. This module is shown in the left panel of Figure 6 with common interactions in coad-read and brca plotted. The nodes with the highest degrees are indicated by their names. The hubs in this common module for both coadread and brca are important oncogenes including TP53 that play important roles in many cancers. We also performed enrichment analysis using INGENUITY (www.ingenuity.com) on the genes evolved in this common module. As shown in the right panel of Figure 6, we found that the identified functions are very relevant to the studied biological context and critical to tumor development, for example, melanoma signaling and cell-cycle events. The rest of the tumor-specific communities for coadread and brca are presented in Supplementary Figure 1, and indicated by different colors. The representative nodes for each module with the highest degrees are indicated by their names as well. We also plotted the detailed tumor-specific interactions in Supplementary Figure 2. There are 76 coadread-specific interactions and 48 brca-specific interactions. Nodes are shown in different colors corresponding to degree differences between the two types of tumors. Specifically, red nodes have the same degrees in the two conditions, while blue nodes have higher degrees in the coad-read condition, and orange nodes have higher degrees in the brca condition. We also zoomed in some known oncogenes with their names shown in the pictures. We identified genetic variants that are known to be implicated in individual cancer types. For example, the MTOR copy number has a higher degree in coadread ver-sus brca, which is consistent with the activation of the PI3K-mTOR-AKT pathway in coadread [Ciriello et al. (2013)]. Also, the BRCA1 mutation has a higher de-gree in brca versus coadread, which corresponds to the inactivation of BRCA1 in breast tumors that leads to defective cell cycle arrest in response to DNA damage [Network et al. (2012)].

Fig. 6.

Common module and functions for networks identified by DIG. (a) The shared module with common interactions presented in both coadread and brca. (b) Identified functions by enrichment analysis of genes evolved in the common module. P-values are obtained by Fisher’s exact tests.

To compare, we also performed the analysis without considering the similarities of coadread and brca, for which ρ₂ = 0. We selected the optimal value for the tuning parameter controlling sparsity as ρ₁ = 0.5. We found 1222 edges for coadread and 981 edges for brca, respectively. Among them, there are 1000 coadread-specific interactions, 759 brca-specific interactions and 222 common interactions. The number of common interactions in this analysis is much less than that in the joint analysis above. We also performed modularity analysis for the resulting networks. We found three modules for caodread with sizes of 67, 1 and 56, as well as four modules for brca with sizes of 36, 45, 22 and 21. Moreover, the results show that the shared edges cannot form a common community. Furthermore, we also applied JGL [Danaher, Wang and Witten (2013)] to this cancer dataset by treating mutations as continuous variables of CNVs. It ended up with 890 interactions for coadread and 1218 interactions for brca. Among these interactions, for coadread, 1.69% are mutation-mutation interactions and 21.5% are mutation-CNV interactions. For brca, 0.41% are mutation-mutation interaction and 14.4% are mutation-CNV interactions. The results suggest that JGL is less capable of identifying interactions related to mutations. Specifically, JGL identified only one interaction (NRG1) associated with the mTOR mutation in coadread, and no interaction was identified with the mTOR mutation in brca. However, DIG identified 20 and 19 interactions associated with the mTOR mutation in coadread and brca, respectively. In coadread, DIG suggests that the mTOR mutation is associ-ated with the mutations of TP53, RB1, MAP3K1, COL4A5 and PTEN, as well as CNVs of MTOR, ARID1A, DNMT3A, TET2, FBXW7, SDK1, NRG1, CDKN1B, HCN4, CTCF, CDH1, MAP2K4, SMAD4, PHLPP1 and EP300. In brca, DIG indicates that the mTOR mutation is connected with the mutations of TP53, RB1, MAP3K1, COL4A5 and PTEN, as well as MTOR, DNMT3A, TET2, FBXW7, SDK1, NRG1, CDKN1B, HCN4, CTCF, CDH1, MAP2K4, SMAD4, PHLPP1 and EP300. As the PI3K-mTOR-AKT pathway is the biological context considered in our application, JGL is likely to miss important interactions relevant to the mTOR mutation compared to DIG. One of the evidences is that activation of p53 inhibits mTOR activity, and inhibited mTOR also affects p53 activity [Feng et al. (2005)].

8. Discussion.

In this paper, we have proposed a coherent statistical frame-work, DIG, for the problem of estimating multiple related mixed graphical models from high-dimensional data with both discrete and continuous variables and with observations belonging to distinct but related biological conditions. The application has been illustrated using cancer studies. DIG is a general statistical framework that can be applied to the genomics of other diseases. For future work, it is natural to extend the proposed framework employing exponential families for a mixed graphical model [Chen, Witten and Shojaie (2015), Yang et al. (2013)]. Furthermore, it would be interesting to develop hypothesis testing methods such that the final mixed graphical models are accompanied by a p-value on each edge and an overall estimate of edge false discovery rate. A systematic investigation of model selections and hypothesis testing for the components of the mixed graphical models would be important future work.