PenPC: A Two-step Approach to Estimate the Skeletons of High Dimensional Directed Acyclic Graphs

Summary

Estimation of the skeleton of a directed acyclic graph (DAG) is of great importance for understanding the underlying DAG and causal e ects can be assessed from the skeleton when the DAG is not identifiable. We propose a novel method named PenPC to estimate the skeleton of a high-dimensional DAG by a two-step approach. We first estimate the non-zero entries of a concentration matrix using penalized regression, and then fix the difference between the concentration matrix and the skeleton by evaluating a set of conditional independence hypotheses. For high dimensional problems where the number of vertices p is in polynomial or exponential scale of sample size n, we study the asymptotic property of PenPC on two types of graphs: traditional random graphs where all the vertices have the same expected number of neighbors, and scale-free graphs where a few vertices may have a large number of neighbors. As illustrated by extensive simulations and applications on gene expression data of cancer patients, PenPC has higher sensitivity and specificity than the state-of-the-art method, the PC-stable algorithm.

1. Introduction

Many statistical methods have been developed to identify the associations between genomic features and disease outcomes or cancer subtypes. However, such association results are descriptive in their nature, and they cannot deliver “actionable” conclusions for disease treatment. Many recently developed cancer drugs are so-called “targeted drugs” that target particular (mutated) proteins in cancer cells, and the mechanism of such drugs can be understood as direct interventions on tumor cells. To characterize or predict the consequences such drug interventions, statical methods that allow causal inference based on high dimensional genomic data are urgently needed.

One of the most commonly used tools for causal inference among a large number of random variables is the probabilistic directed acyclic graph (DAG) (also known as Bayesian Network) (Lauritzen, 1996; Pearl, 2009). In a DAG, all the edges are directed, and the direction of an edge implies a direct causal relation. There is no loop in a DAG. Such “acyclic” property is necessary to study causal relations (Spirtes et al., 2000). When we remove the directions of all the edges in a DAG, the resulting undirected graph is the skeleton of the DAG.

Estimation of the skeleton of a DAG is of great importance because it is a crucial step towards estimating the underlying DAG and skeleton itself may provide a limited amount of information for causal inference (Maathuis et al., 2009, 2010). Several methods have been developed to estimate DAGs or their skeletons from observational data (Heckerman et al., 1995; Spirtes et al., 2000; Chickering, 2003; Kalisch and Bühlmann, 2007), however most of them are not suitable for the high dimensional genomic problems that motivate our study. In this paper, we proposed a new method named PenPC to address this challenging problem. We proved the estimation consistency of PenPC for high dimensional settings of p = O (exp{n^a}) for 0 ≤ a < 1, and we also derived the conditions for estimation consistency for two types of graphs: random graphs where all the vertices have the same expected number of neighbors, and scale-free graphs where a few vertices have much larger number of neighbors than other vertices. As verified by both simulation and real data analyses, PenPC provides more accurate estimates of DAG skeletons than existing methods. In addition to skeleton, PenPC can further estimate the complete partially directed acyclic graph (CPDAG), which can used to estimate causal effects (Maathuis et al., 2009).

The remaining parts of this paper are organized as follows. In Section 2, we give a brief review of DAG estimation methods and the conceptual advantages of our PenPC algorithm. We present the details of the PenPC algorithm and its theoretical properties in Sections 3 and 4, followed by simulations and real data analyses in Section 5 and Section 6, respectively. Finally, we conclude with some discussions in Section 7.

2. Review of DAG Estimation

2.1 Directed Acyclic Graph (DAG)

A DAG of random variables X₁, ..., X_p can be denoted by G = (V, E), where V contains p vertices 1, 2, ...., p that correspond to X₁, ..., X_p, and E contains all the directed edges. In a DAG, a chain of length n from i to j is a sequence i = i₀ – i₁ – · · · – i_n–1 – i_n = j of distinct vertices such that i_l–1 → i_l ∈ E or i_l i_l–1 ∈ E for l = 1, . . . , n; and a path of length n from i to j is a sequence i = i₀ → i₁ → · · · → i_n = j of distinct vertices such that i_l–1 → i_l ∈ E for l = 1, ..., n. Given this path, i_l–1 is a parent of i_l, i_l is a child of i_l–1, i₀, i₁, ..., i_l–1 are ancestors of i_l, and i_l+1, ..., i_n are descendants of i_l.

Given a DAG G for random variables X₁, . . . , X_p and assume that $\mathit{X}={(X_1,\dots ,X_p)}^{\mathrm{T}}\in {\mathbb{R}}^p\sim P_{\mathit{X}}$ with density f_X. Let X_pai be the parents of X_i. We say that the distribution P_X is Markov to G if the joint density f_X satisfies the recursive factorization: $f(X_1,\dots ,X_p)={\prod }_{i=1}^{p}f(X_i\mid X_{{\mathrm{pa}}_i})$ The factorization naturally implies acyclic restriction of the graph structure. Equivalently P_X is Markov to G if every variable is conditionally independent of its non-descendants given its parents. A related concept is the so-called faithfulness:

Definition 1

Let P_X be Markov to G. < G, P_X > satisfies the faithfulness condtiontion if and only if every conditional independence relation true in P_X is entailed by the Markov property applied to G (Spirtes et al., 2000).

This means that if a distribution P_X is faithful to a DAG G, all conditional independences can be read o from the DAG G using d-separation defined in the following definition 2, and thus the faithfulness assumption requires stronger relationship between the distribution P_X and the DAG G than the Markov property.

Definition 2

(d-separation). A vertex set S block a chain p if either (i) p contains at least one arrow-emitting vertex belonging to S, or (ii) p contains at least one collision vertex (j is a collision vertex if the chain includes i → j ← k) that is outside S and no descendant of the collision vertex belongs to S. If S blocks all the chains between two sets of random variables X and Y , we say “S d-separates X and Y” (Pearl, 2009).

Not all the distributions can be faithfully represented by a DAG. In this paper, we assume the random variables follow multivariate Gaussian distribution, then the faithfulness assumption can be justified by the fact that among all the multivariate Gaussian distributions associated with G, the non-faithful ones form a Lebesgue null set (Meek, 1995).

Given multivariate Gaussian distribution assumption, a commonly used graphical model is Gaussian Graphic Model (GGM), where two vertices are connected if the corresponding two variables are dependent, given all the other variables. A GGM can be constructed by a concentration matrix (i.e., precision matrix or inverse of covariance matrix) in that two vertices are connected if the corresponding elements in the concentration matrix is non-zero. The skeleton of a DAG is di erent from its GGM because of v-structures. In a v-structure X → W ← Z, co-parent X and Z are marginally independent or conditionally independent given their parents, but given every vertex set that contains W (a collision vertex) or any descendant of W , X and Z are dependent with each other. Note that by the definition of v-structure, the co-parents X and Z are not connected. A few examples are shown in Figure 1, and instances of the covariance and concentration matrices of the GGM in Figure 1(a) are shown in the Supplementary Materials, Section 1.

Figure 1.

Four DAGs where X and Z are not connected in the skeleton, but are connected in the corresponding GGMs. The true relation between X and Z can be revealed by appropriate conditional independence testing. For example, X ⊥ Z in Figure 1(a), X ⊥ Z|Y in Figure 1(b), X ⊥ Z|(Y, U) in Figure and 1(c), and X ⊥ Z|Y in Figure 1(d).

2.2 DAG estimation using observational data

In this paper we focus on DAG skeleton estimation using observational data instead of interventional data. When the p variables have a nature ordering (i.e., all the parents or ancestors of X_i are among the vertices X₁, ..., X_i–1, and all the children or descendants of X_i are among vertices X_i+1, ..., X_p), the problem of skeleton estimation is greatly simplified because a regression of X_i versus X₁, ..., X_i–1 can be used to identify the true skeleton (Shojaie and Michailidis, 2010). However, in many high-dimensional problems, such a nature ordering is not available. Throughout this paper, we assume no knowledge of nature ordering. Then the underlying DAG is not identifiable from observational data, because conditional dependencies implied by the Markov property on the observational distribution P_X only determine the skeleton and v-structures of the graph (Pearl, 2009). All the DAGs with the same skeleton and v-structures correspond to the same probability distribution and they form a Markov equivalence class. After estimating skeleton, the v-structures can be identified by a set of deterministic rules, and thus we do not distinguish the estimation of a DAG skeleton and a Markov equivalence class.

In general, there are two approaches for DAG or DAG skeleton estimation. The first one is the search-and-score approach that searches for the DAG that maximizes or minimizes a pre-defined score, such as BIC (Bayesian Information Criterion) or L₀-penalized maximum likelihood estimates (van de Geer and Bühlmann, 2013). Instead of searching across all the DAGs, which is often computationally infeasible, elegant methods have been developed to search across Markov equivalence classes (Chickering, 2003) or the nature orderings of the variables (Teyssier and Koller, 2005). However, these methods are still computationally very challenging for genomic applications with thousands of variables.

The second approach for DAG (skeleton) estimation is constraint-based approach that constructs DAGs by assessing conditional independence of random variables. One representative method is the PC algorithm (named after its authors, Peter Sprites and Clark Glymour) (Spirtes et al., 2000). Starting with a complete undirected graph where any two vertices are connected with each other, the PC algorithm first thins the complete graph by removing edges between vertices that are marginally independent. Then it removes edges by assessing conditional independence given one vertex, two vertices, and so on. Kalisch and Bühlmann (2007) proved the consistency of the PC-algorithm in high-dimensional settings where p = O(n^a) for a > 0. The results of the PC algorithm depend on the order of the edges to be assessed. Colombo and Maathuis (2012) proposed PC-stable algorithm, which modified the PC algorithm to remove such order dependency and substantially improve the performance of the PC algorithm. We consider the PC-stable algorithm as the state-of-theart method and compare our method with the PC-stable algorithm.

The Independence Graph (IG) algorithm (Chapter 5.4.3 of Spirtes et al. (2000)) modifies the PC algorithm by using a different initial graph: an undirected independence graph where two vertices are connected if the corresponding two variables are conditionally dependent given all the other variables, i.e., a GGM under multivariate Gaussian distribution assumption. In such an independence graph, the neighbors of a vertex Y_j include its parents, children, and co-parents of v-structures in the underlying DAG, which constitute the so-called Markov blanket of Y_j such that Y_j is independent of all the other vertices given its Markov blanket.

The Max-Min Hill-Climbing (MMHC) algorithm is a popular hybrid method that first estimates DAG skeleton using a constraint-based method (the Max-Min part of the algorithm), and then orient the edges using a search-and-score technique (the Hill-Climbing part of the algorithm) (Tsamardinos et al., 2006). Schmidt et al. (2007) proposed to replace the Max-Min part of the MMHC algorithm by a penalized regression with Lasso (l₁) penalty, i.e., neighborhood selection (Meinshausen and Bühlmann, 2006). Variable selection consistency of Lasso requires the irrepresentable condition (Zhao and Yu, 2006): there is weak correlation between the variables within and outside a Markov blanket. This is a strong condition and it generally does not hold for the genomic problems that motivate this study.

We propose a PenPC algorithm for DAG skeleton estimation in two steps. It first adapts neighborhood selection method to select Markov blanket of each vertex, and then it applies a modified PC-stable algorithm to remove false positive edges between co-parents of v-structures. Although the two-step approach of the PenPC algorithm shares similar spirit to the IG algorithm (Spirtes et al., 2000) and the modified MMHC algorithm (Schmidt et al., 2007), we have made the following novel contributions. First, we employ the log penalty p(|b|; λ, τ)= λlog(|b|+τ) (Mazumder et al., 2011) for neighborhood selection, which significantly improves the accuracy of Markov blanket search for higher dimensional problems, e.g., n = 30 and p = 100, or n = 300 and p = 1000. In contrast, Schmidt et al. (2007) explicitly assume n $\gg$ p in their paper. The resulting PenPC algorithm outperforms the state-of-the-art PC-stable algorithm and also enjoys some advantage in terms of computational efficiency in high dimensional settings. Second, we provide theoretical justifications of the estimation consistency of the PenPC algorithm in high dimensional settings where p = O (exp{n^a}) for 0 ≤ a < 1. We also discuss the implications for estimation consistency for two types of graphs: traditional random graph where all the vertexes have the same expected number of connections, and scale-free graph where a few vertices can have much larger number of neighbors than the other vertices. Whereas non-scale-free graph is often assumed in previous studies (Kalisch and Bühlmann, 2007), scale-free graph is more frequently observed in gene networks as well as many other applications (Barabási and Albert, 1999).

3. Methods

We adopt a multivariate Gaussian distribution assumption: X = (X₁, . . . , X_p)^T ~ N(0,Σ). Let X = (x₁, ..., x_p) be the n × p observed data matrix. Our PenPC algorithm proceeds in two steps: (1) neighborhood selection, and (2) application of a modified PC-stable algorithm to remove false connections. Theoretical justification of our algorithm is presented in Section 4.

Step 1. (Neighborhood Selection)

We first select the neighborhood of vertex i by a penalized regression with X_i as response variable and all the other variables corresponding to vertices V \ {i} as covariates:

$${\widehat{\mathbf{b}}}_i=\underset{{\mathbf{b}}_i\in {\mathbb{R}}^{p-1}}{\arg \min }\frac{1}{2}{({\mathbf{x}}_i-{\mathbf{X}}_{-i}{\mathbf{b}}_i)}^{\mathrm{T}}({\mathbf{x}}_i-{\mathbf{X}}_{-i}{\mathbf{b}}_i)+n\sum _{j\ne i}p(\mid b_{i,j}\mid ;{\varpi }_i),$$ (1)

where X_–i is an n × (p – 1) matrix for n measurements of the remaining p – 1 covariates, b_i = (b_i,1, ..., b_i,i–1, b_i,i+1, ..., b_i,p)^T, and $p(\mid b_{i,j}\mid ;{\varpi }_i)$ denotes a penalty function with one or more tuning parameters, denoted by ${\varpi }_i$. We consider a class of folded concave penalty functions satisfying the following condition:

Condition 1

The penalty function $p(\beta ;\varpi )$ is concave in β ∈ [0, ∞), with continuous derivative $p^{\prime }(\beta ;\varpi )\ge 0$, and $p^{\prime }(0+;\varpi )>0$.

This is a generalization of the Condition 1 in Fan and Lv (2011). In this study, we employ the log penalty p(|b|; λ τ) = λlog(|b| + τ), which has been demonstrated to have good performance in high-dimensional genetic studies (Sun et al., 2010). We solve penalized regression with log penalty using a coordinate descent algorithm (Sun et al., 2010), and the two tuning parameters λ and τ are selected by two-grid search to minimize extended BIC (Chen and Chen, 2008). After p penalized regressions for each of the p variables, we construct the GGM by adding an edge between vertices i and j if ${\widehat{b}}_{ij}\ne 0$ or ${\widehat{b}}_{ji}\ne 0$.

Step 2. (Modified PC-stable algorithm)

We apply a modified PC-stable algorithm to remove the false edges between parents of v-structures. For each edge i – j, we first assess marginal association between vertices i and j. If they remain dependent, we test whether they are conditionally dependent. The conditional set should be selected from the Markov blanket of i and j, after excluding i and j's common children or descendants. Specifically. we use the following strategy to search for candidate separation sets. Let A_i,j be the Markov blanket of i and j, and let C_i,j be the set of vertices that could be common children or descendants of i and j. Then the candidate conditional sets are

$${\Pi }_{i,j}=\{{\mathit{A}}_{i,j}\backslash {\mathit{D}}_{i,j},{\mathit{D}}_{i,j}\subseteq {\mathit{C}}_{i,j}\}.$$ (2)

Each element of Π_i,j is a set A_i,j \ D_i,j, where D_i,j is exhaustively searched across all subsets of C_i,j. More details are described in the Supplementary Materials, Section 2.

We test the conditional independence of X_i and X_j given K ∈ Π_i,j using Fisher transformation of partial correlation. Specifically, denote the partial correlation between X_i and X_j given K ∈ Π_i,j by ${\rho }_{i,j\mid \mathcal{K}}$. With the significance level α, we reject the null hypothesis H₀ : ${\rho }_{i,j\mid \mathcal{K}}=0$ against the alternative hypothesis H_a : ${\rho }_{i,j\mid \mathcal{K}}\ne 0$ if $\sqrt{n-\mid \mathcal{K}\mid -3}{\widehat{z}}_{i,k\mid \mathcal{K}}>{\Phi }^{-1}(1-\alpha ∕2)$ , where ${\widehat{z}}_{i,j\mid \mathcal{K}}=0.5\log ((1+{\widehat{\rho }}_{i,j\mid \mathcal{K}})∕(1-{\widehat{\rho }}_{i,j\mid \mathcal{K}}))$ and ϕ(·) is the cdf of N(0, 1).

The final output of PenPC algorithm is the estimated skeleton and separation sets S(i, j) for all (i, j). If vertices i and j are connected in the skeleton, the separate set is an empty set, otherwise X_i and X_j are independent given S(i, j), hence the name separation set. Given the skeleton and the separation sets, one can estimate CPDAG (Complete Partially Directed Acyclic Graphs) (Supplementary Materials Section 3) and then apply the idaFast or ida functions of R package pcalg (Kalisch et al., 2012) to estimate multi-set of possible causal effects.

4. Theoretical Properties

4.1 Fixed Graphs

We first introduce the following notations. For an m × n matrix A, denote the matrix L_b norm of A by ${\Vert \mathbf{A}\Vert }_b={\sup }_{\mathbf{x}\ne 0}({\Vert \mathbf{Ax}\Vert }_b∕{\Vert \mathbf{x}\Vert }_b)$, where x is a vector of length n, and ${\Vert \mathbf{x}\Vert }_b={\left({\sum }_{i=1}^n{x}_i^b\right)}^{1∕b}$. In particular, ${\Vert \mathbf{A}\Vert }_2$ is the spectral norm of A, ${\Vert \mathbf{A}\Vert }_1={\max }_{1\le j\le n}{\sum }_{i=1}^m\mid a_{ij}\mid$ is the maximum absolute column summation, and ${\Vert \mathbf{A}\Vert }_{\infty }={\max }_{1\le i\le m}{\sum }_{j=1}^n\mid a_{ij}\mid$ is the maximum absolute row summation. Denote the vector L_b norm of A by |A|b. In particular, ${\mid \mathbf{A}\mid }_2={\left({\sum }_{1\le i\le m,1\le j\le n}{a}_{ij}^2\right)}^{1∕2}$, and ${\mid \mathbf{A}\mid }_{\infty }={\max }_{1\le i\le m,1\le j\le n}\mid a_{ij}\mid$. Denote by A_i,–i the submatrix of A that includes the i-th row and excludes the i-th column of A. A_i,i, A_–i,i and A_–i,–i are defined similarly. For any compatible subsets S₁ and S₂, A_S1S2 is the submatrix that contains all the rows with indices in S₁ and all the columns with indices in S₂.

We denote p as p_n to emphasize it is a function of sample size n. Let a DAG and the corresponding GGM be G_n = (V_n, E_n) and C_Gn = (V_n, F_n), respectively. We further denote the skeleton of G_n by ${\mathcal{G}}_n^u=(V_n,E_n^u)$ where$a-b\in E_n^u\iff a\to b\in E_n$ or b → a ∈ E_n. For any vertex i, denote the observed centralized data of the variables within and outside of the neighbors of i in C_Gn (denoted by adj(i, C_Gn)), but not including X_i, by ${\mathcal{X}}_{i1}$ and ${\mathcal{X}}_{i2}$, respectively, i.e., ${\mathcal{X}}_{i1}={\mathbf{X}}_{{\mathcal{S}}_0,{\mathcal{S}}_i}$ and ${\mathcal{X}}_{i2}-{\mathbf{X}}_{{\mathcal{S}}_0,-\{i,{\mathcal{S}}_i\}}$ where S₀ = {1, 2, ..., n} and S_i = {j : j ∈ adj (i,C_Gn)}.

The following Lemma 1 is a well-known conclusion that gives the relation between concentrate matrix of multivariate Gaussian distribution and the regression coe cients when we regress one variable versus all the other variables (Anderson, 2003).

Lemma 1

Suppose $\mathit{X}={(X_1,\dots ,X_p)}^{\mathrm{T}}\sim {\mathcal{N}}_p(0,\Sigma )$ and $\Omega ={\Sigma }^{-1}={\left({\omega }_{ij}\right)}_{p\times p}$. Then $X_i={\mathit{X}}_{-i}^{\mathrm{T}}{\mathbf{b}}_i+{∊}_i$ where ${\mathbf{b}}_i=-{\Omega }_{-i,i}∕{\omega }_{ii}$, and ${∊}_i\sim \mathrm{N}(0,1∕{\omega }_{ii})$ independent of X_–i.

With the aforementioned notations and definitions, we can state the following conditions that are needed for the consistency of the PenPC algorithm.

• (A1)Dimensionality of the problem. p_n = O (exp{n^a}) with a ∈ [0, 1).
• (A2)Sparseness assumption. Let $q_n={\max }_{1\le j\le p_n}\mid \mathrm{adj}(j,{\mathcal{C}}_{{\mathcal{G}}_n})\mid$, i.e., the maximum degree of the Gaussian graphical model C_Gn. q_n = O(n^b) for some 0 ≤ b < (1 – a)/2. Let $M_n={\max }_{1\le j\le p_n}\mid \mathrm{adj}(j,{\mathcal{G}}_n)\mid$. By the following Lemma 2, M ≤ q_n = O(n^b).
• (A3)
  Minimum e ect size for neighborhood selection.

  $${\delta }_n\equiv (1∕2)\underset{i,j}{\inf }\left\{\mid \frac{{\omega }_{ij}}{{\omega }_{ii}}\mid :{\omega }_{ij}\ne 0\right\}\ge O\left(n^{-d_1}\right),0<d_1<(1-a-b)∕2.$$
• (A4)Conditions for the population covariance matrix $\Sigma =\left\{{\sigma }_{ij}\right\}$. Let λ_min(Σ_S,S) be the minimum eigen-value of a sub matrix Σ_S,S. For any S with $\mid \mathcal{S}\mid \le q_n,{\lambda }_{\min }\left({\Sigma }_{\mathcal{S},\mathcal{S}}\right)>C_1$. We also assume max_i σ_ii < C₂. Here C₁ and C₂ are two positive constants. Consequently C₂ is also an upper bound of all the off-diagonal elements of Σ because ${\sigma }_{ij}\le {\left({\sigma }_{ii}{\sigma }_{jj}\right)}^{1∕2}$
• (A5)Conditions for penalty function. Let $\kappa (\mathit{v};\varpi )={\lim }_{∊\to 0+}{\sup }_{t_1\le t_2\in (\mid v_j\mid -∊,\mid v_j\mid +∊)}\frac{p^{\prime }(t_2;\varpi )-p^{\prime }(t_1;\varpi )}{t_2-t_1}$ for $\mathit{v}={(v_1,\dots ,v_r)}^{\mathrm{T}}\in {\mathbb{R}}^r$. Thus $\kappa (\mathit{v};\varpi )={\max }_{1\le j\le r}-p^{″}(\mid v_j\mid ;\varpi )$ if the penalty function has continuous second derivative. Let ${\kappa }_{0,i}={\max }_{{\beta }_1\in {\mathcal{N}}_i}\kappa ({\beta }_1;\varpi )$ and N_i is a hypercube around the vector ${\mathbf{b}}_{i1}=({\omega }_{ij}∕{\omega }_{ii}:{\omega }_{ij}\ne 0)$ such that ${\mathcal{N}}_i=\{{\beta }_1:{\mid {\beta }_1-{\mathbf{b}}_{i1}\mid }_{\infty }\le C{n}^{-d_1}\}$. We assume ${\max }_i{\kappa }_{0,i}\le C_1$, where C₁ is defined in (A4) , $p^{\prime }({\delta }_n;{\varpi }_i)=o\left(q_n^{-1∕2}{n}^{-d_1}\right)$, and $q_n^{1∕2}\max \{{(\log p∕n)}^{1∕2},p^{\prime }({\delta }_n;{\varpi }_i)\}=o\left(p^{\prime }(0+;{\varpi }_i)\right)$.
• (A6)
  Restriction on the size of conditional partial correlation. Denote the partial correlations between X_i and X_j given a set of variables $\{X_r:r\in \mathcal{K}\}$ for $\mathcal{K}\subseteq V_n\backslash \{i,j\}$ by ${\varrho }_{i,j\mid \mathcal{K}}$. For K ∈ Π_ij (Π_i,j was defined in equation (2)), the absolute values of ${\varrho }_{i,j\mid \mathcal{K}}$'s are bounded:

  $$\underset{i,j,\mathcal{K}}{\inf }\{\mid {\varrho }_{i,j\mid \mathcal{K}}\mid :{\rho }_{i,j\mid \mathcal{K}}\ne 0,\mathcal{K}\in {\Pi }_{i,j}\}\ge c_n,\text{and}\underset{i,j,\mathcal{K}}{\sup }\{\mid {\varrho }_{i,j\mid \mathcal{K}}\mid :\mathcal{K}\in {\Pi }_{ij}\}\le M<1.$$

  where c_n = O(n^−d2) for some 0 < d₂ < min{(1 – a)/2, (1 – b)/2}.

The sparseness assumption (A2) will be replaced by tighter assumptions for two specific random graph models later. Assumptions (A3)-(A5) ensure that the step 1 of PenPC can recover the partial correlation graph. There are fairly reasonable conditions to ensure the identifiability of the problem. Assumption (A3) requires the minimum e ect size is larger than noise level (e.g., larger than O(n^−1/2) when p = O(1)). Assumption (A4) requires the covariance matrix for those important covariates in a neighborhood selection problem is not singular. Assumption (A5) are conditions for the penalty function, which can be easily satisfied by adjusting the two tuning parameters of the Log penalty (Chen et al., 2014). Assumption (A6) ensures the summation of the mistaken probabilities of the step 2 of the PenPC algorithm goes to 0 asymptotically. The condition $q_n^{1∕2}{p}^{\prime }({\delta }_n;{\varpi }_i)=o\left(p^{\prime }(0+;\varpi )\right)$ of assumption (A5) deserves more discussions because it corresponds to the irrepresentable condition that limits the performance of Lasso regression. Specifically, in Supplementary Materials, we show that ${\Vert {\mathcal{X}}_{i2}^{\mathrm{T}}{\mathcal{X}}_{i1}{\left({\mathcal{X}}_{i1}^{\mathrm{T}}{\mathcal{X}}_{i1}\right)}^{-1}\Vert }_{\infty }=O\left(q_n^{1∕2}\right)$ with probability approaching to 1. The assumption $q_n^{1∕2}{p}^{\prime }({\delta }_n;{\varpi }_i)=o\left(p^{\prime }(0+;\varpi )\right)$ is needed so that ${\Vert {\mathcal{X}}_{i2}^{\mathrm{T}}{\mathcal{X}}_{i1}{\left({\mathcal{X}}_{i1}^{\mathrm{T}}{\mathcal{X}}_{i1}\right)}^{-1}\Vert }_{\infty }=o(p^{\prime }(0+;\varpi )∕p^{\prime }({\delta }_n;\varpi ))$. For the Lasso, $p^{\prime }(0+;\varpi )∕p^{\prime }({\delta }_n;\varpi )$ = 1, and thus this is a very strong assumption for the size of $\Vert {\mathcal{X}}_{i2}^{\mathrm{T}}{\mathcal{X}}_{i1}{\left({\mathcal{X}}_{i1}^{\mathrm{T}}{\mathcal{X}}_{i1}\right)}^{-1}\Vert$ . In contrast, for the log penalty, $p^{\prime }(t;{\lambda }_i,{\tau }_i)={\lambda }_i\mathrm{sgn}\left(t\right)∕(\mid t\mid +{\tau }_i)$, and thus $p^{\prime }(0+;{\tau }_i)∕p^{\prime }({\delta }_n;{\tau }_i)\to ({\delta }_n+{\tau }_i)∕{\tau }_i$, which can goes to infinity if τ_i = o(δ_n). We can show that the log penalty satisfies other assumptions and refer the readers to Chen et al. (2014) for details.

Consider the neighborhood selection problem for the i-th variable versus all the other variables. Recall that ${\mathcal{S}}_i=\{j:j\in \mathrm{adj}(i,{\mathcal{C}}_{{\mathcal{G}}_n})\}=\mathrm{supp}\left({\mathbf{b}}_i\right)$ is the support of the true regression coe cient b_i with size $\mid {\mathcal{S}}_i\mid =s_i$. Let b_i1 and ${\widehat{\mathbf{b}}}_{i1}$ be respectively the sub-vectors of b_i and ${\widehat{\mathbf{b}}}_i$ corresponding to S_i.

Theorem 1

Given Assumptions (A1) - (A5), with probability at least 1 – C exp{–n^a} for a constant 0 < C < ∞, there exists a local minimizer ${\widehat{\mathbf{b}}}_i={({\widehat{\mathbf{b}}}_{i1},{\widehat{\mathbf{b}}}_{i2})}^{\mathrm{T}}$ that satisfies the following conditions for any i = 1, . . . , p_n,

• (a)Sparsity: ${\widehat{\mathbf{b}}}_{i2}=0$
• (b)L_∞ loss: ${\Vert {\widehat{\mathbf{b}}}_{i1}-{\mathbf{b}}_{i1}\Vert }_{\infty }=o\left(n^{-d_1}\right)$, where d₁ is defined in (A3).

Therefore, if we denote the estimate of ${\mathcal{C}}_{{\mathcal{G}}_n}$ by the neighborhood selection as ${\widehat{\mathcal{C}}}_{{\mathcal{G}}_n}\left(\varpi \right)$, where $\varpi$ are tuning parameters of the penalty function, $\mathbb{P}({\widehat{\mathcal{C}}}_{{\mathcal{G}}_n}\left(\varpi \right)={\mathcal{C}}_{{\mathcal{G}}_n})\ge 1-C\exp \{-n^a\}$.

The proof is in the Supplementary Materials. The following Lemma 2 and 3 provide the theoretical justifications for using GGM as a starting point of our modified PC-algorithm.

Lemma 2

If the distribution P_X is Markov to G, i.e., if the joint density f_X satisfies the recursive factorization, the set of edges F_n of C_Gn includes all edges $E_n^u$ of ${\mathcal{G}}_n^u$ plus the edges between co-parents of v-structures in G_n.

Lemma 3

Assume (A1). If (i, j) ∈ F_n of C_Gn but (i, j) $\notin E_n^u$ of ${\mathcal{G}}_n^u$, the conditioning set Π_i,j in (2) includes at least one set which d-separates vertices i and j in G.

Lemma 2 has been proved in Lemma 3.21 of Lauritzen (1996). The proof of Lemma 3 is presented in the Supplementary Materials. Lemma 2 shows that the concentration matrix recovers all the edges in the skeleton with no false negatives, but some false positives between the co-parents of v-structures. Lemma 3 shows that we can remove such false positives by examining partial correlation conditioning on some set in Π_i,j.

Next we discuss the theoretical property of the modified PC-stable algorithm given a perfect estimation of GGM.

Theorem 2

Let α_n be the p-value threshold for testing whether a partial correlation is 0. Let ${\widehat{\mathcal{G}}}_n^u\left({\alpha }_n\right)$ be the estimates of ${\mathcal{G}}_n^u$ from the second step of the PenPC algorithm given a perfect estimation of GGM from the first step of the PenPC algorithm. Assume (A1), (A2) and (A6), then there exists α_n → 0, such that $\mathbb{P}[{\widehat{\mathcal{G}}}_n^u\left({\alpha }_n\right)={\mathcal{G}}_n^u]=1-O\left(\exp \{-C{n}^{1-2d_2}\}\right)\to 1$ where 0 < C < ∞ is a constant.

The proof is in the Supplementary Materials. Similar theorem has been proved in Kalisch and Bühlmann (2007) with p_n at polynomial order of n. By starting with GGM, we extend the theorem to p_n = O (exp{n^a}) case. Combining the results of Theorem 1 and Theorem 2, corollary 1 show that the summation of mistaken probabilities of GGM estimation and skeleton estimation given GGM goes to 0 as n → ∞.

Corollary 1

Let ${\widehat{\mathcal{G}}}_n^u(\theta ,{\alpha }_n)$ be the estimates of ${\mathcal{G}}_n^u$ from the two-step approach PenPC algorithm. Assume (A1)-(A6), then there exists an α_n → 0, such that $\mathbb{P}[{\widehat{\mathcal{G}}}_n^u(\theta ,{\alpha }_n)={\mathcal{G}}_n^u]=1-O\left(\exp \{-n^a\}\right)\to 1$, where 0 < C < ∞ is a constant.

4.2 Random Graphs

Next we extend our theoretical results to two commonly used models for random graphs: Erdös and Rényi (ER) Model (Erdös and Rényi, 1960) and Barabási and Albert (BA) Model (Barabási and Albert, 1999). Let $q_n={\max }_{1\le j\le p_n}\mid \mathrm{adj}(j,{\mathcal{G}}_{{\mathcal{G}}_n})\mid$ and $M_n={\max }_{1\le j\le p_n}\mid \mathrm{adj}(j,{\mathcal{G}}_n)\mid$ In general, assumption (A2) no longer holds for random graphs. It is easy to see that assumption (A2) can be relaxed to (A2’) and we introduced an additional assumption (A7)

• (A2')$\mathbb{P}\{q_n\le O\left(n^b\right)\}=1$ for some $0\le b<1$.
• (A7)$M_n\ge C{q}_n$, where 0 < C < 1 is a constant.

Assumption (A7) says that the maximal degree in the GGM is not dominated by the edges induced by co-parents of v-structures as n goes to infinity, which is a reasonable assumption. Given this assumption, M_n and q_n are on the same scale.

4.2.1 Erdös and Rényi (ER) Model

The ER model constructs a graph G(p_n, p_E) of p_n vertices by connecting vertices randomly. Each edge is included in the graph with probability p_E independent from all other edges. By law of large numbers, such vertex is almost surely connected to (p_n – 1)p_E edges. Erdös and Rényi (1960) proved the following results about M_n, the maximal degree of the graph.

Lemma 4

In the graph G(p_n, p_E) following the ER model, the maximal degree M_n almost surely converges to m_n, where $m_n=O\left(\log p_n\right)$ if p_np_E < 1, $m_n=p_n^{2∕3}$ if p_np_E = 1, and m_n = O(p_n) if ${\lim }_{p_n\to \infty }{p}_n{p}_E=c>1$.

When p_n = O{exp(n^a)}, by Lemma 4 and assumption (A7), assumption (A2’) holds if p_np_E < 1 and b ≥ a. When p_np_E ≥ 1, our proof cannot handle the general case p_n = O{exp(n^a)}. However, when the number of vertices is of the polynomial order of n, assumption (A2’) may still hold. In particular, suppose p_n = O(n^r). When p_np_E < 1, assumption (A2’) holds for any b ∈ [0, ∞). When p_np_E = 1, assumption (A2’) holds if b ≥ 2r/3. When p_np_E → c > 1, assumption (A2’) holds if r < 1 and b ≥ r.

4.2.2 Barabási and Albert (BA) Model

The BA model is used to generate scale free graphs whose degree distribution follows a power law: $\mathbb{P}\left(\nu \right)={\gamma }_0{\nu }^{-{\gamma }_1}$, with a normalizing constant γ₀ and a exponent γ₁. Specifically, BA model generates a graph by adding vertices into the graph over time and when each new vertex is introduced into the graph, it is connected with larger probability to the existing vertices with larger number of connections. Since the distribution does not depend on the size of the network (or time), the graph organizes itself into a scale free state (Barabási and Albert, 1999). Móri (2005) showed that M_n (the maximal degree of the graph) almost surely converges to O(p^1/2). Thus, assumption (A2’) holds for the case p_n = O(n^r) with b ≤ r/2.

5. Simulation Studies

We evaluated the performance of the PenPC algorithm and the PC-stable algorithm in terms of sensitivity and specificity of skeleton estimation using DAGs simulated by the ER model or the BA model. In both simulations and real data analysis, we used the implementation of the PC-stable algorithm by function skeleton in R package pcalg (version 1.1-6), and we have implemented PenPC algorithm in R package PenPC.

Following Kalisch and Bühlmann (2007), we simulated DAGs of p vertices by the ER model as follows. For any vertex pair (i, j) where i < j, we added an edge i → j with probability p_E. For the BA model, the DAGs were simulated following Barabási and Albert (1999). The initial graph had one vertex and no edge. In the (t + 1)-th step, e edges were proposed. For each edge, the new vertex was connected to the i-th (1 ≤ i ≤ t) existing vertex with probability ${\nu }_i^{\left(t\right)}∕{\sum }_j{\nu }_j^{\left(t\right)}$, where ${\nu }_i^{\left(t\right)}=\mid \mathrm{adj}(i,{\mathcal{G}}^{\left(t\right)})\mid$, and ${\mathcal{G}}^{\left(t\right)}$ was the DAG at the t-th step. The distribution of the degrees ν from simulated DAGs under ER model (p = 1000 and p_E = 2/p) and BA model (p = 1000 and e = 1) are shown in Figure 2 and similar graph for BA model (p = 1000 and e = 2) is shown in Figure S2 of Supplementary Materials.

Figure 2.

Histograms of the degree ν. (a) ER model with p = 1000 and p_E = 2/p. (b) BA model with p = 1000 and e = 1 and the log₁₀ scale density of log₁₀ν in its subplot.

The probability of finding a highly connected vertex decreases exponentially with ν for the graphs generated by the ER model (Figure 2(a)). However, for the graphs generated by the BA model, there is a linear relation between degree and degree probability in log-log scale, confirming its scale-free property (Figure 2(b)).

After constructing the DAGs, the observed data were simulated by structure equations under multivariate Gaussian assumption. For example, denote the parents of X_j by pa_j, then ${\mathbf{x}}_j={\sum }_{k\in {\mathrm{pa}}_j}{b}_{jk}{\mathbf{x}}_k+{∊}_j$ where ${∊}_j\sim \mathcal{N}(0,{\sigma }^2{I}_{n\times n})$ In our simulations, all b_jk's and σ₂ were set to be 1. For either ER or BA model, we considered low dimension setting where p = 11, n = 100 and high-dimension settings where p = 100, n = 30 and p = 1000, n = 300 with various sparsity levels determined by p_E for ER model and e for BA model (Table 1).

Table 1.

Simulation Setting

p	n	pE (ER)	e (BA)
11	100	0.2	1,2
100	30	0.02, 0.03, 0.04, 0.05	1,2
1000	300	0.002, 0.005, 0.01	1,2

Due to limited space, here we only show the results for the simulation setups using ER model with p=1000, n=300, and p_E=0.005; and BA model with p=1000, n=300, and e=1. The remaining results are presented in Figure S3 - S14 of the Supplementary Materials.

First consider the results from the ER model when α = 0.01. Recall that α is the p-value threshold for conditional independence testing. The penalized regression (step 1 of the PenPC algorithm) identifies more true positives than the PC-stable algorithm, but also introduce more false positives (Figure 3 (a-b)), while PenPC algorithm significantly reduces the number of false positives, though some true positives are also removed. At the end, the PenPC has the lowest number of false positives plus false negatives, as measured by Hamming distance (HD) (Figure 3 (c)). Figures 3(d-f) show that across various values of α, PenPC consistently has better performance than the PC-stable algorithm. Finally, Figure 3(g) shows the ROC curves for the PenPC and the PC-stable algorithms, which illustrate that PenPC has better sensitivity and specificity than the PC-stable algorithm regardless of the cuto α. Similar conclusions can be drawn for the simulation results shown in Figure 4, where the DAGs are simulated by the BA model. We note that although PenPC performs well for the BA model, further improvement is possible by incorporating special consideration for the scale-free structure of BA graphs (Liu and Ihler, 2011).

Figure 3.

Performance of ER model (p = 1000, n = 300, p_E = 0.005). The upper panels are box plots (in log₁₀ scale) of true positive rate (TPR) (a), false positive rate (FPR) (b) and hamming distance (HD) (c) from 100 replications at α = 0.01. The lower panels are average true positive rate (d), false positive rate (e), and Hamming distance (f) from 100 replications when the tuning parameter α is changed from 0 to 0.1 (the grey vertical line are at α = 0.01). ROC curves are shown in panel (g). This figure appears in color in the electronic version of this article.

Figure 4.

Performance of BA model (p=1000,n=300,e=1). The upper panels are box plots (in log₁₀ scale) of true positive rate (TPR) (a), false positive rate (FPR) (b) and hamming distance (HD) (c) from 100 replications at α = 0.01. The lower panels are average true positive rate (d), false positive rate (e), and Hamming distance (f) from 100 replications when the tuning parameter α is changed from 0 to 0.1 (the grey vertical line are at α = 0.01). ROC curves are shown in panel (g). This figure appears in color in the electronic version of this article.

6. Application

We applied the PC-stable algorithm and the PenPC algorithm to study gene-gene network using gene expression data from tumor tissue of 550 TCGA (The Cancer Genome Atlas) breast cancer patients (Cancer Genome Atlas Network, 2012). Gene expression were measured by RNA-seq. We quantified the expression of each gene within each sample by log(total read count) (logTReC). After removing genes with low expression across most samples, we ended up with 18,827 genes. We first removed the e ects of several covariates by taking residuals of logTReC for each gene using a linear regression with the following covariates: 75 percentile of logTReC per sample, which captures read depth, plate, institution, age, and six PCs from the corresponding germline genotype data.

The computational cost of the PC-stable algorithm increases quickly as the number of vertices or the p-value cuto increases. When we study 410 genes, the computational time of the PC-stable and PenPC algorithms are both within an hour. When we expand the number of genes to 8,261. The step 1 in PenPC algorithm took 3 hours in total while searching for 1000 combinations of tuning parameters for each gene. Given the GGM, the 2nd step of the PenPC is computationally much more e cient than the PC-stable algorithm (Figure 5 (a)). For example, with p-value threshold varies from 10⁻⁷ to 10⁻⁵, the computational time of the PC-stable algorithm increases from 20 to 50 hours. In contrast, the computation time of the PenPC remains below 10 hours even for p-value cuto 5 × 10⁻³. All the computation are done in Linux server with an 2.93 GHz Intel processor and 48GB RAM.

Figure 5.

Comparing PenPC algorithm with PC-stable algorithm in terms of skeleton estimation by changing the significance levels for partial correlation testings: α=0.0001, 0.0005, 0.001, 0.005, 0.01 and 0.05. (a) The computational time at different α values when we consider 8,261 genes. (b) The number of detected edges vs. the number of edges in the PPI (protein-protein interaction) data when we consider 410 genes. This figure appears in color in the electronic version of this article.

Since PC-stable algorithm is not computationally feasible for larger gene set, we first discussed the results on 410 genes from the cancer Gene Census in http://cancer.sanger.ac.uk/cancergenome/projects/census/. For α = 0.0001 to 0.05, we estimated the skeleton by the PC-stable and PenPC algorithms. The estimated skeletons were evaluated by comparing the estimated edge sets with protein-protein interaction (PPI) database at http://www.pathwaycommons.org/pc2/downloads.html. PPI is a reasonable resource to evaluate graphical model estimates because genes with PPI tend to co-express (Rhodes et al., 2005). There were 3315 PPIs where both proteins belong to the 410 genes. Figure 5(a) shows the total number of detected edges versus the number of edges in PPI data. For both methods, the total number of detected edges increase monotonically as increases. The PenPC results have higher sensitivity to detect PPI given the number of edges discovered.

Next we applied PenPC to 8,261 genes with PPI annotation. Using α = 0.001, we detected 12,150 edges, that is 0.03% of the total number of edges. We arbitrarily define the genes with more than 7 neighbors as hub genes and there are 46 hub genes (Supplementary Table 1). Interestingly, many of the hub genes are cancer-related. For example, all the hub genes with more than 9 neighbors, MYC, ELF3, and RAB15, are cancer-related. MYC encodes Myc proto-oncogene protein, which are associated with multiple types of human cancers including breast cancer. ELF3 is one of the ETS transcription factor and it modulates breast cancer-associated gene expression. RAB15 is a member RAS oncogene family.

7. Discussions

The seminal works of Kalisch and Bühlmann (2007) have greatly advanced our understating of the PC-algorithm and provided well-designed and user-friendly software packages (Kalisch et al., 2012). Our PenPC algorithm provides some helpful improvements, especially in high dimensional settings. The PenPC algorithm has three tuning parameters, two for step 1 (tuning parameters of the Log penalty for neighborhood selection) and one for step 2 (p-value cutoff) of the PenPC algorithm. The selection of tuning parameters for step 1 and step 2 of PenPC are two independent procedures. For step 1, it is a classical problem of tuning parameter selection for penalized regression. We chose to use extended BIC as it delivers the best performance and it has sound theoretical justifications (Chen and Chen, 2008). Choosing the best combination of the two tuning parameters for log penalty using extended BIC does not induce heavy computational cost. For example, we use 1,000 combinations of λ and τ for each of the penalized regressions with Log penalty, and for our real data analysis with n = 550 and p = 8, 261, it takes about 3 hours for all the p = 8,261 penalized regressions. In the second step of PenPC, we need to choose a p-value threshold for conditional independence tests, similar to the tuning parameter for the PC algorithm. How to choose this p-value cuto is an open problem that warrants further research.

We have compared PenPC with the approach of replacing the Log penalty with the Lasso penalty. As expected, the Lasso penalty leads to much worse performance in high dimensional settings (Figures S15-S16 in Supplementary Materials). Following Kalisch and Bühlmann (2007), we assume a multivariate Gaussian assumption so that we may test conditional independence by assessing conditional correlation. The first step of the PenPC algorithm (neighborhood selection) does not require this assumption. Our method is robust to this multivariate Gaussian assumption. For example, Figures S15-S16 show that the performance of PenPC algorithm is comparable when the data are simulated from multivariate Gaussian and multivariate t-distribution (df=5). Assuming the observed data are generated by linear, potentially non-Gaussian structural equation model (SEM), Loh and Bühlmann (2014) proved that the moral graph of a DAG (i.e., the DAG skeleton plus the edges that connect the co-parents of v-structures) can be estimated by the support of the inverse covariance matrix and they estimated inverse covariance matrix using graphical Lasso. Borrowing their theoretical justifications, we can extend PenPC to non-Gaussian cases. The first step of PenPC is similar to graphical Lasso, but with log penalty instead of Lasso penalty. The second step of PenPC can be modified by using a conditional independent test that does not rely on Gaussian assumption. Finally, our work assumes no hidden confounders or latent variables, which may be justified by the fact that we examine the expression of all the genes and the e ects of confounders or latent variables may be manifested by the expression of certain genes.