Reconstruction of a directed acyclic graph with intervention

Abstract

Identification of causal relations among variables is central to many scientific investigations, as in regulatory network analysis of gene interactions and brain network analysis of effective connectivity of causal relations between regions of interest. Statistically, causal relations are often modeled by a directed acyclic graph (DAG), and hence that reconstruction of a DAG’s structure leads to the discovery of causal relations. Yet, reconstruction of a DAG’s structure from observational data is impossible because a DAG Gaussian model is usually not identifiable with unequal error variances. In this article, we reconstruct a DAG’s structure with the help of interventional data. Particularly, we construct a constrained likelihood to regularize intervention in addition to adjacency matrices to identify a DAG’s structure, subject to an error variance constraint to further reinforce the model identifiability. Theoretically, we show that the proposed constrained likelihood leads to identifiable models, thus correct reconstruction of a DAG’s structure through parameter estimation even with unequal error variances. Computationally, we design efficient algorithms for the proposed method. In simulations, we show that the proposed method enables to produce a higher accuracy of reconstruction with the help of interventional observations.

1. Introduction

Directed acyclic graph (DAG) models are useful to describe pairwise causal relations between random variables, defined by a certain Markov property [5], with each node representing one variable and each directed edge representing the corresponding pairwise causal relation. DAG models have been widely used in gene and social networks [7, 19]. To identify causal relations, intervention observations are usually collected in addition to observational attributes [16]. The central topic this article addresses is the reconstruction of a DAG model based on interventional data and pertinent issues with respect to the effect of the intervention on the reconstruction of a DAG’s structure.

In the literature, it is generally believed that interventions may help the reconstruction of a DAG’s structure, particularly when a DAG model is not identifiable from data, that is, DAGs in a Markov equivalence class are not distinguishable based on observational data alone [16]. In biological experiments, for instance, intervention occurs in a form of randomized treatments in a clinical trial or a form of gene knockdown or knockout experiments in systems biology. In such a situation, some or all system variables are controlled, permitting direction estimation of ambiguous edges connecting to these controlled variables. Yet exactly how intervention impacts reconstruction of a DAG’s structure remains unknown. Consequently, it is practically important to design a reconstruction method for interventional data, permitting the identification of a DAG’s structure. Most existing methods for intervention [6, 10, 8] assume known intervention, that is, affected variables of the intervention are known a priori before data collection; see [9] for references therein. However, assuming known intervention is impractical, as in system biology, where the effect of various chemicals intervening a system cannot be precisely known. To our knowledge, one exception is a Bayesian method of [4], which is designed for a low-dimensional problem without theoretical guarantee, due to the super-exponential complexity in the number of nodes.

In this article, we propose a novel variance constraint on interventions to fully identify the DAG structure in the framework of unknown intervention. Theoretically, we show in Theorem 1 that a DAG structure is fully identifiable under the constraint, which is otherwise only possible when all the error variances are not the same [18]. Moreover, we propose a constrained maximum likelihood to seek the most efficient interventions by sparsity pursuit, leading to an identifiable reconstruction of a DAG’s structure. Computationally, we develop an efficient algorithm to solve nonconvex minimization subject to the quadratic variance constraint based on the alternating direction method of multipliers (ADMM) [2]. In simulations, we investigate the impact of the intervention on reconstruction and compare the proposed method with its counterpart without intervention. Overall, the proposed method performs well.

This article is organized into seven sections. Section 2 introduces the proposed method and discusses the issue of identifiability due to intervention, followed by the computational development in Section 3. Section 4 establishes the consistency theorems of the proposed method. Section 5 performs some simulations to study the intervention effect, and two real datasets are analyzed. Section 6 summarizes the results. Finally, the Appendix A contains technical details and proofs.

2. Method

Consider a causal model consisting of p random variables Y = (Y₁, … , Y_p)^⊤ described by a DAG, with each node representing one variable and directed edges encoding causal relations between any two variables, where ^⊤ denotes the transpose. This model factorizes the joint distribution of Y , P(Y) into a product of conditional distributions of each variable given its parents: $P(Y)={\prod }_{j=1}^{p}p\left(Y_j\mid p{a}_j\right)$, where pa_j denotes the parent set of Y_j and is defined to be empty if Y_j has no parents. This factorization is known as the local Markov property [5].

A DAG is modeled by structural equations as

$$Y=AY+\mathit{ϵ},\mathit{ϵ}~N(0,D),$$ (1)

where ϵ = (ϵ₁, ϵ₂, … , ϵ_p)^⊤ represents the latent or unexplained error, $D=\text{Diag}\left({\sigma }_1^2,\cdots ,{\sigma }_p^2\right)$ is the error covariance matrix and A = (A_ij)_p×p is an adjacency matrix that uniquely determines a DAG. Here A_ij ≠ 0 encodes an edge from node j to node i. In (1), the inverse covariance matrix of Y is Ω = (I – A)^⊤D⁻¹(I − A), where I is the identity matrix.

When σ₁ = ⋯ = σ_p, (1) is identifiable [18].Then a DAG’s structure can be reconstructed by estimating A. However, when σ₁ = ⋯ = σ_p breaks down, (1) is usually not identifiable, which means that A is not estimable.

2.1. Intervention or covariate models and variance constraint

To treat non-identifiability in an observational study, consider a model consisting of W intervention variables X = (X₁, X₂, … , X_W)^⊤, where the outcome of Y is observed with intervention variables X that may be non-informative.

After incorporating the intervention variables into (1), we obtain that

$$Y=AY+BX+\mathit{ϵ},\mathit{ϵ}~N(0,D),$$ (2)

where A, ϵ and D are defined as in (1), B = (B_jw)_p×W is an unknown intervention coefficient matrix, whose jwth entry B_jw indicates the directional strength of the intervention of X_w on Y_j. When B_jw = 0; j = 1, ⋯ , p, there is no intervention of X_w on Y_i, and thus X_w is non-informative. Note that (2) becomes a causal model with covariates X.

In the situation of unequal error variances, with the help of the intervention, we may impose constraints to achieve model identifiability, which otherwise is impossible [18]. Assume that X ~ N(0, Σ_X), which is independent of ϵ. Without loss of generality, assume that Σ_X = I subsequently because we can reparametrize X as ${\Sigma }_X^{-1/2}X$. Then under (2), Y ~ N(0, Ω⁻¹), where Ω = (I − A)^⊤ (BB^⊤ + D)⁻¹(I − A). In (2), we impose the variance constraint:

$$\text{Var}(BX+\mathit{ϵ})=\theta I,\text{ or }B{B}^{\top }+D=\theta I,$$ (3)

where θ > 0 is a parameter to be estimated.

More details are deferred to Sections 3 and 4.

Theorem 1 (Identifiability). Assume that X ~ N(0, Σ_X) is independent of ϵ in (2), and A_jk ≠ 0 for all k which is a parent of j; j = 1, 2, ⋯ , p. Under (3), A is identifiable from the distribution of (Y , X).

As suggested by Theorem 1, A in (2) becomes identifiable when (3) is imposed on B, which is otherwise impossible. Note, however, that interventions B leading to identifiable A may not be unique. In what is to follow, we impose a sparsity constraint to identify a most sparse B in terms of the number of nonzero elements of B.

2.2. Constrained maximum likelihood

This section estimates A subject to the DAG requirement while seeking a most sparse B subject to (3). Consequently, the smallest set of informative intervention variables can be identified through B.

Under (2), two data matrices (y_ij)_n×p and (x_iw)_n×W are observed, with n representing the sample size. Then the negative log likelihood is

$$l(A,B,D)=\sum _{j=1}^p\left[-\frac{n}{2}\text{log }{\sigma }_j^2+\frac{1}{2{\sigma }_j^2}\sum _{i=1}^n{\left(y_{ij}-\sum _{k\ne j}{A}_{jk}{y}_{ik}-\sum _{w=1}^W{B}_{jw}{x}_{iw}\right)}^2\right].$$ (4)

To identify nonzero entries of A and B, we impose sparsity constraints to regularize:

$$\sum _{1\le j\ne l\le p}I\left(A_{jl}\ne 0\right)\le K_1,\sum _{1\le j\le p,1\le l\le W}I\left(B_{jl}\ne 0\right)\le K_2,$$ (5)

where K₁ and K₂ are nonnegative integer-valued tuning parameters. Note that the constraint on B removes zero entries thus zero-columns of B, which can be regarded as selection of intervention variables. To reinforce the DAG requirement, we impose acyclic constraints [27] to reinforce the DAG requirement to ensure no loops to occur:

$${\lambda }_{ik}+I(j\ne k)-{\lambda }_{jk}\ge I\left(A_{ij}\ne 0\right);i,j,k=1,\cdots ,p,i\ne j,$$ (6)

where λ = {λ_jl}_p×p is a dual variable matrix.

For computation, we replace the indicator functions in (5) and (6) by its computational surrogate $J_{\tau }(z)=\text{min}\left(\frac{|z|}{\tau },1\right)$ [20] to circumvent the difficulty of non-discontinuity in optimization. This yields

$$\sum _{1\le j<l\le p} {\text{J}}_{\tau }\left(A_{jl}\right)\le K_1,\sum _{1\le j\le p,1\le l\le W} {\text{J}}_{\tau }\left(B_{jl}\right)\le K_2,$$ (7)

$${\lambda }_{ik}+I(j\ne k)-{\lambda }_{jk}\ge J_{\tau }\left(A_{ij}\right);i,j,k=1,\cdots ,p,i\ne j,$$ (8)

where J_τ (z) approximates the indicator function as τ → 0⁺.

Minimizing (4) in (A, B, D) subject to (7), (8), and (3) yields the constrained maximum likelihood estimate (CMLE):

$${\text{min}}_{A,B,D}l(A,B,D)={\sum }_{j=1}^p\left[-\frac{n}{2}\text{log }{\sigma }_j^2+\frac{1}{2{\sigma }_j^2}{\sum }_{i=1}^n{\left(y_{ij}-{\sum }_{k\ne j}{A}_{jk}{y}_{ik}-{\sum }_{w=1}^W{B}_{jw}{x}_{iw}\right)}^2\right],$$

$$\text{subj to }{\sum }_{1\le j<l\le p} {\text{J}}_{\tau }\left(A_{jl}\right)\le K_1,{\sum }_{1\le j\le p,1\le l\le W} {\text{J}}_{\tau }\left(B_{jl}\right)\le K_2,$$

$${\lambda }_{ik}+I(j\ne k)-{\lambda }_{jk}\ge J_{\tau }\left(A_{ij}\right);i,j,k=1,\cdots ,p,i\ne j,$$

$$B{B}^{\top }+D=\theta I,$$ (9)

where (K₁, K₂, τ) are tuning parameters.

3. Computation

This section develops a computational strategy to solve (9). First, θ is estimated by $\widehat{\theta }$ through an estimate $\widehat{\mathit{A}}$ of A from the method in [27], that is,

$$\widehat{\theta }=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{\left(y_{ij}-\sum _{k\ne j,k=1}^p{\widehat{A}}_{jk}{y}_{ik}\right)}^2,$$ (10)

where ${\left\{{\widehat{\mathit{A}}}_{jk}\right\}}_{j,k=1,\cdots ,p}$ are obtained by a constrained maximum likelihood estimate with the sparsity constraint and acyclic constraint, based on the structural equation model (1) with observational data Y alone. Then we solve (9) with θ replaced by $\widehat{\theta }$ using a blockwise coordinate descent alternating between two blocks (A, B) and D until convergence. In particular, the (A, B)-block is solved via a difference convex (DC) programming followed by an alternating direction method of multipliers (ADMM), while the D-block is updated by gradient descent. More details are further discussed subsequently.

3.1. Optimization subject to the variance constraint

To deal with the variance constraint in (9), we first consider a general constrained minimization subject to the variance constraint as follows:

$${\text{min}}_{B}f(B),\text{ subj to }B{B}^{\top }=\Lambda ,$$ (11)

where f(B) is a cost function and Λ is a diagonal matrix.

For (11), we work with its equivalent form by introducing a dual matrix C to decouple B and the constraint: min_{(B, C)} f(B), subject to CC^⊤ = Λ and C − B = 0. Then we apply the alternating direction method of multipliers (ADMM) [2] to obtain its augmented Lagrangian: $L_{\rho }(B,C,y)=f(B)+y^{\top }\text{ vec}\left(C-B\right)+\frac{\rho }{2}\Vert C-B{\Vert }_F^2$, where $y\in {ℝ}^{pW}$ is the Lagrangian multiplier for constraint C−B = 0, and ρ > 0 is the augmented Lagrangian parameter. Then it is further simplified by introducing a dual variable matrix V = {V_jl}_p×W to incorporate y^⊤ vec(C − B) into the quadratic form

$${\text{min}}_{(B,C,V)}{L}_{\rho }(B,C,V)=f(B)+\frac{\rho }{2}\Vert C-B+V{\Vert }_F^2,\text{ subj to }C{C}^{\top }=\Lambda .$$ (12)

Now we apply ADMM to iterate through three steps to solve (12) until convergence. In the kth iteration,

$$C^{(k+1)}={\text{argmin}}_C\frac{\rho }{2}{\Vert C-B^{(k)}+V^{(k)}\Vert }_F^2,\text{ subj to }C{C}^{\top }=\Lambda ,$$ (13)

$$B^{(k+1)}={\text{argmin}}_{B}f(B)+\frac{\rho }{2}{\Vert C^{(k+1)}-B+V^{(k)}\Vert }_F^2,$$ (14)

$$V^{(k+1)}=V^{(k)}+C^{(k+1)}-B^{(k+1)}.$$ (15)

In (13)–(15), the variance constraint CC^⊤ = Λ enters only in (13). Next we provide a closed-form solution of (13) in Lemma 1.

Lemma 1. The solution of (13) can be written as C^(k+1) = Λ^1/2PO^⊤, where a singular value decomposition of (B^(k) − V^(k))^⊤Λ^1/2 gives OEP^⊤, $O\in {ℝ}^{W\times p}$, E, $P\in {ℝ}^{p\times p}$ are the left, diagonal, and right matrices in the decomposition.

3.2. Algorithm for solving (9)

After plugging $\widehat{\theta }$ into (9), we begin with the update of the (A, B)-block by fixing D at an initial value D₀ and optimize (9) with regard to (A, B). A good estimate of D₀ can be obtained by solving (9) without the variance constraint, details are given in the Appendix.

To solve (9) with a fixed D, we follow [27] to convert (9) to its equivalent dual form. The procedure contains two steps. First, we apply a DC programming method and decompose the nonconvex constraint function of the nonconvex constraints (7) and (8) into a difference of two convex functions, based on which we construct a sequence of convex approximation of nonconvex constrained sets iteratively, the details are given in the Appendix. Then at the mth iteration, we solve a relaxed subproblem (16). The iteration process continues until a termination criterion is met.

The mth subproblem amounts to

$${\text{min}}_{(A,B,\lambda ,\xi )}l(A,B)+\frac{{\mu }_1}{\tau }{\sum }_{1\le j\ne l\le p}|A_{jl}|w_{jl}^{(m-1)}+\frac{{\mu }_2}{\tau }{\sum }_{1\le j\le p,1\le l\le W}|B_{jl}|v_{jl}^{(m-1)},$$

$$\text{subj to}{\lambda }_{js}+\tau I(l\ne s)-{\lambda }_{ls}={|A_{jl}|}_1{w}_{jl}^{(m-1)}+\tau \left(1-w_{jl}^{(m-1)}\right)+{\xi }_{jls};$$

$$j,l,s=1,\dots ,p,j\ne l,{\xi }_{jls}\ge 0,$$

$$B{B}^{\top }+D=\widehat{\theta }I,$$ (16)

where ξ = {ξ_jls}_p×p×p, ξ_jls ≥ 0 is a slack variable tensor, and $w_{jl}^{(m-1)}=I\left(|{\widehat{A}}_{jl}^{(m-1)}|\le \tau \right)$ and $v_{jl}^{(m-1)}=I\left(|{\widehat{B}}_{jl}^{(m-1)}|\le \tau \right)$ are obtained from the (m−1)th iteration.

For (16), we apply ADMM method by decoupling (A, B) in the likelihood from the rest part of the cost function and the acyclic constraint. As in Section 3.1, we introduce dual variable tensor y = {y_jls}_p×p×p, dual variable matrix U = {U_jl}_p×(p+W) and V = {V_jl}_p×W. Then we minimize the augmented Lagrangian under the variance constraint:

$${\text{min}}_{(A,C,B,F,\lambda ,\xi ,y,U,V)}{L}_{\rho }(A,C,B,F,\lambda ,\xi ,y,U,V)=l(A,B)+\frac{{\mu }_1}{\tau }{\sum }_{1\le j\ne l\le p}|F_{jl}|w_{jl}^{(m-1)}+\frac{{\mu }_2}{\tau }{\sum }_{1\le j\le p,1\le l\le W}|B_{jl}|v_{jl}^{(m-1)}+{\sum }_{1\le s\le p}{\sum }_{1\le j\ne l\le p}\frac{\rho }{2}{\left(|F_{jl}|w_{jl}^{(m-1)}+\tau \left(1-w_{jl}^{(m-1)}\right)+{\xi }_{jls}-{\lambda }_{jl}-\tau I(l\ne s)+{\lambda }_{ls}+y_{jls}\right)}^2+\frac{\rho }{2}{\sum }_{1\le j,l\le p}{\left(A_{jl}-F_{jl}+U_{jl}\right)}^2+\frac{\rho }{2}{\sum }_{1\le j\le p,1\le l\le W}{\left(B_{jl}-F_{j,l+p}+U_{j,l+p}\right)}^2+\frac{\rho }{2}{\sum }_{1\le j\le p,1\le l\le W}{\left(C_{jl}-B_{jl}+V_{jl}\right)}^2,$$

$$\text{subj to }C{C}^{\top }+D=\widehat{\theta }I.$$ (17)

Again, we solve (17) over blocks (A, C, B, F, λ, ξ, y, U, V) iteratively until convergence, where analytic updating formulas are given in the Appendix. After ADMM iterations converge, we continue the DC iterations until converge, then the current (A, B)-block is updated.

When updating the D-block, we conduct a gradient descent update for each of $\left({\sigma }_1^2,\cdots ,{\sigma }_p^2\right)$ based on the current values of $(\widehat{\mathit{A}},\widehat{\mathit{B}})$. The gradients for the jth dimension is $l_j^{\prime }=-\frac{n}{2{\sigma }_j^2}-\frac{1}{2{\sigma }_j^4}{\sum }_{i=1}^n{\left(y_{ij}-{\sum }_{k\ne j}{\widehat{A}}_{jk}{y}_{ik}-{\sum }_{w=1}^W{\widehat{B}}_{jw}{x}_{iw}\right)}^2$. Then ${\sigma }_j^2$ is updated by

$${\sigma }_j^2={\sigma }_j^2-\alpha l_j^{\prime },$$ (18)

where α > 0 is the step size.

The computational strategy is summarized in Algorithm 1.

Algorithm 1:

Constrained maximum likelihood

Step 1. Obtain an estimate $\widehat{\theta }$ of θ by (10), then plug $\widehat{\theta }$ into (9).

Step 2. Fix D at an initial value D0. Set pre-specified error tolerance ϵ0 and the maximum number of iterations M0 for blockwise coordinate descent.

Step 3. ((A, B)-block) Initialize A and B. Set pre-specified error tolerance ϵ1 and the maximum number of DC iterations M1.

Step 3.1. At the mth DC iteration, compute ${\widehat{\mathit{A}}}^{(m)}$ and ${\widehat{\mathit{B}}}^{(m)}$ cycling through the ADMM updating steps until convergence.

Step 3.2. When $|l\left({\widehat{A}}^{(m+1)},{\widehat{B}}^{(m+1)},\widehat{D}\right)-l\left({\widehat{A}}^{(m)},{\widehat{B}}^{(m)},\widehat{D}\right)|\le {ϵ}_1$ or m = M1, stop the DC loop and output $(\widehat{A},\widehat{B})=\left({\widehat{A}}^{(m)},{\widehat{B}}^{(m)}\right)$.

Step 4. (D-block) Update D according to (18).

Step 5. Iterate through steps 3 through 4 until ${\sum }_{j=1}^p|l_j^{\prime }|\le {ϵ}_0$ or the number of iterations equals to M0, stop and output $(\widehat{\mathit{A}},\widehat{\mathit{B}})$.

The computation complexity of Algorithm 1 is of order O(M₀M₁M₂p²(p + W) + n(p + W)² + p(p + W)³), where M₀, M₁ and M₂ are the maximum number of iterations for (A, B)-step, DC and ADMM, respectively. For each ADMM iteration, the computation complexity is O(p²(p + W)). The preparation phase has O(n(p + W)² + p(p + W)³) complexity. In practice, the DC loop usually converges in a few iterations, which has finite termination property, c.f., Lemma 2 in [21].

4. Theory

This section develops a theory quantifying the reconstruction error of the CMLE defined in (9). In particular, we first show the estimated $\widehat{\theta }$ recovers the optimal parameter estimation of the oracle estimator ${\widehat{\theta }}_O$, which is defined as the maximum likelihood estimate in [27] provided that the true non-zero pattern of the DAG is given. Then we establish reconstruction consistency of the true DAG’s structure defined by adjacency matrix A in the presence of intervention variables X in (2).

Let A, B, and D represent model parameters under (2). Let E = {(i, j) : A_ij ≠ 0} be the set of non-zero edges in the graph $\mathcal{G}$, and |E| denote the size of the set. Let Ω = (I − A)^⊤ (BB^⊤ + D)⁻¹(I − A) = (I − A)^⊤ (I − A)/θ be the precision matrix of Y. In what follows, we will assume the variance constraint (3), and by Theorem 1, A is identifiable from the distribution of (Y, X), and thus the graph structure of $\mathcal{G}$ is identifiable. Let G⁰, A⁰, B⁰, R⁰, E⁰, Ω⁰, θ⁰ denote the truth. Let ${\widehat{\mathit{A}}}_O$ and ${\widehat{\theta }}_O$ denote the oracle estimator, or the maximum likelihood estimate provided that the true set E⁰ of non-zero edges is given. Let ${\widehat{\mathbf{\Omega }}}_O={\left(I-{\widehat{\mathit{A}}}_O\right)}^{\top }\left(I-{\widehat{\mathit{A}}}_O\right)/{\widehat{\theta }}_O$ be the oracle estimator for Ω.

For the observational model, let A^obs be the model parameter, then the precision matrix is Ω^obs = (I − A^obs)^⊤ (I − A^obs)/θ, whose oracle estimator is ${\widehat{\mathbf{\Omega }}}_O^{obs}={\left(I-{\widehat{\mathit{A}}}_O^{obs}\right)}^{\top }\left(I-{\widehat{\mathit{A}}}_O^{obs}\right)/{\widehat{\theta }}_O$ where ${\widehat{\mathit{A}}}_O^{obs}$ is the oracle estimator of A^obs.

The degree of reconstructability is defined as

$$C_{\text{min}}\left({\mathbf{\Omega }}^0\right)=\underset{\left\{A\ne A^0,|E|\le |E^0|,A\text{\hspace{0.17em}\hspace{0.17em}induces\hspace{0.17em}a\hspace{0.17em}DAG}\right\}}{\text{inf}}\frac{-\text{log }\left(1-h^2\left(\mathbf{\Omega },{\mathbf{\Omega }}^0\right)\right)}{\text{max}\left(|E^0\backslash E|,1\right)},$$

where h²(Ω, Ω⁰) is the Hellinger distance between Ω and Ω⁰ under (2) and the variance constraint (3), and E₁ \ E₂ denote the set difference between E₁ and E₂. The degree of reconstructability measures the difficulty of reconstructing the graph, and we require it to be larger than a certain level in order for our proposed method to be consistent in reconstructing the graph structure. For a detailed discussion about the degree of reconstructability of a graph, c.f., [27].

Assumption A.1 (Boundedness). For some positive constants M₁ and M₂, inf_Ω c_min(Ω) ≥ M₁, sup_1≤k≤p |Ω_kk| ≤ M₂, where c_min(Ω) is the smallest eigenvalue of Ω and Ω_kk is the kth diagonal element of Ω.

Assumption A.2 (Boundedness). For some positive constants M₃ and M₄, inf_Γ c_min(Γ) ≥ M₃ and sup_1≤k≤p+W, |Γ_kk)| ≤ M₄, where Π is the covariance matrix of the joint distribution of (Y, X) and Γ = Π⁻¹.

Assumption B (Degree of reconstructability). $C_{\text{min}}\left({\mathbf{\Omega }}^0\right)\ge 4d_0^{-1}{n}^{-1}\times \text{max}\left(\text{log }p,|E^0|\right)$, for some positive constant d₀ > 0, say $d_0=\frac{2}{27}\frac{1}{963}$.

Assumption C. For some positive constants d₁, d₂ and d₃,

$$h^2\left(\mathbf{\Omega },{\mathbf{\Omega }}^0\right)\ge d_1{h}^2\left({\mathbf{\Omega }}_{\tau },{\mathbf{\Omega }}^0\right)-d_{3}p{\tau }^{d_2},$$

where Ω_τ = (I − A_τ)^⊤ (I − A_τ)/θ and A_τ is a truncated version of A with its ijth element A_ijI(|A_ij| ≥ τ).

Assumptions A.1 and A.2 concern the smallest eigenvalues and the maximum diagonal element of Ω. Under Assumption A.1, the likelihood function becomes bounded. Assumption B is a key condition for the consistency of reconstructed graph structure, we require the degree of reconstructability to be no less than a lower bound, which is related to the size of p or |E⁰|. Assumption C requires the Hellinger distance to be smooth so that we approximate L₀ function with its computational surrogate, the TLP function [21] to the desired level by tuning τ.

First, we show that θ is uniquely defined regardless of the value of B.

Lemma 2. Under (2), θ uniquely satisfies (3).

Then we show the optimal parameter estimation of θ achieved by utilizing the observational data in the Theorem 2.

Theorem 2 (Optimal parameter estimation). Under Assumptions A.1 and C, if K₁ = |E⁰| and τ ≤ C_min(Ω⁰)M₁/4p, then there exists a constant c₂ > 0, say $c_2=\frac{4}{27}\frac{1}{1926}$, such that for any (n, |E⁰|, p),

$$P\left(\widehat{\theta }\ne {\widehat{\theta }}_O\right)\le P\left({\widehat{\mathbf{\Omega }}}^{obs}\ne {\widehat{\mathbf{\Omega }}}_O^{obs}\right)\le \text{exp}\left(-c_{2}n{C}_{\text{min}}\left({\mathbf{\Omega }}^0\right)+2\text{\hspace{0.17em}log}(p(p-1)+1)+3\right).$$

Under Assumption B, $P\left(\widehat{\theta }\ne {\widehat{\theta }}_O\right)\to 0$ as n, p, |E⁰| → ∞.

The next theorem gives a reconstruction error bound, under which we obtain reconstruction consistency of the CMLE as well as its optimal parameter estimation.

Theorem 3 (Error bound and oracle properties). Under Assumptions A.1, A.2 and C, if K₁ = |E⁰|, τ ≤ C_min(Ω⁰)M₁/4p, then there exists a constant c₂ > 0, say $c_2=\frac{4}{27}\frac{1}{1926}$, such that for any (n, |E⁰|, p),

$$P\left(\widehat{G}\ne G^0\right)\le P\left(\widehat{\mathbf{\Omega }}\ne {\widehat{\mathbf{\Omega }}}_O\right)\le \text{exp}\left(-c_{2}n{C}_{\text{min}}\left({\mathbf{\Omega }}^0\right)+2\text{\hspace{0.17em}log}(p(p-1)+1)+3\right).$$

Under Assumption B, $P\left(\widehat{G}\ne G^0\right)\to 0$, $\frac{E{h}^2\left(\widehat{\mathbf{\Omega }},{\mathbf{\Omega }}^0\right)}{E{h}^2\left({\widehat{\mathbf{\Omega }}}_O,{\mathbf{\Omega }}^0\right)}\to 1$ as n, p, |E⁰| → ∞.

5. Numerical study

5.1. Simulations

This section examines the performance of the proposed method and demonstrates how intervention, as well as the variance constraint, improves the reconstructability of a DAG’s structure. Seven methods are compared, including the proposed method with intervention, that without the variance constraint (3), the observational method without intervention [27], the constraint-based PC algorithm [22], the score-and-search method GES [3], and two hybrid methods, Max-Min Hill-Climbing (MMHC) [24] and ARGES [14]. For PC algorithm and GES, we use R package pcalg, while for MMHC we use R package bnlearn. For ARGES, we use the ARGES-CIG version [14] by first conducting a neighborhood selection using R package huge and then apply the greedy search using pcalg. For the other three methods, we implement in R with the main algorithm written in C, which is also available in the R package intdag [17] https://cran.r-project.org/web/packages/intdag/index.html.

Several performance metrics are used to measure the accuracy of reconstruction of a graph’s skeleton as well as directionality. With respect to the skeleton of a graph, we use the false discovery rate (FDR) and false negative rate (FNR), defined as FDR = FP/(TP + FP) and FNR = FN/(TP + FN), where TP, FP, TN, and FN denote the true positives, false positives, true negatives and false negatives, respectively. These two metrics together measure the abilities to control false discoveries, as well as false negatives. With regard to directionality, we employ the Structural Hamming Distance (SHD), defined as the minimal number of operations required to transform one DAG to the other, including edge insertions, deletions or flips, c.f., [24]. Note that a smaller SHD value indicates two DAGs are closer to each other. To compute the SHD, we use the R-package pcalg. All the metrics are used on the estimation of adjacency matrix A, since we focus on the reconstructability of DAG.

For tuning, PC algorithm and MMHC require one tuning parameter α controlling the significance level for independence tests, yet there is no practical tuning way via a separate tuning set. In this simulation, the significance level is fixed at 0.05. This choice of α seems sensible as the number of estimated edges is roughly the same as the number of edges in the true graph, as shown in the simulation. For ARGES, the first stage of neighborhood selection needs one tuning parameter corresponding to the LASSO penalization, we use the functions huge.path() in the R package huge to select the tuning range based on the data, and the function huge.fit() to select the tuning parameter. For the observational DAG method [27], τ is chosen from a set {0.1, 0.05, 0.01}, and the sparsity regularization parameter μ₁ is chosen so that the number of estimated edges roughly ranges from 0 to 100. For our methods, τ and μ₁ are selected similarly, and μ₂ = r × μ₁ with the ratio r selected from {1, 2, 4, 8}. For each method, the optimal tuning parameters are obtained by maximizing the predicted log-likelihood (4) based on an independent tuning set of size 1000 over a set pre-specified grid points.

In simulations, we examine a sparse neighborhood graph and a sparse graph with non-sparse neighborhoods in Examples 1 and 2, respectively. A sparse neighborhood requires each node to have sparse links, but a sparse graph does not necessarily have sparse neighborhoods. To further investigate the operating characteristics of the methods, we consider two additional situations in Examples 3 and 4, in which the true graphs satisfy the variance constraint while other settings resemble Examples 1 and 2, respectively. Finally, to compare the performances of the methods in non-sparse situations, Examples 5 and 6 are added by increasing the sampling probabilities when generating the edges. More details are given in the example settings.

Example 1 (Sparse neighborhood). A DAG with 50 nodes is generated with a random generation mechanism as described in [11]. First, the partial ordering of these nodes is randomly generated. Second, we sample edges according to a binomial distribution with probability 0.02, where the edges are assigned to a weight 0.5. The intervention matrix B is a diagonal matrix with diagonal values being 0.5s, describing a situation that each node in DAG is intervened by exactly one intervention covariate. Third, we set the error variances ${\left({\sigma }_1^2,\cdots {\sigma }_p^2\right)}^{\top }$ to be a sequence from 1.5 to 1 with equally spaced points. Finally, we sample X from a p-dimensional normal distribution N(0, I) and generate Y is generated according to (2).

Example 2 (Non-sparse neighborhood). This example is modified from the previous example to generate a DAG of 50 nodes with a special structure of so-called “one-control-all”, where all directed edges are connected from the first node to the other nodes with connection strength of 0.5. Evidently, the neighborhood of the first node is not sparse, but the overall graph remains sparse. The intervention matrix B and the error variances are the same as in Example 1.

Example 3 (Sparse neighborhood with a causal model satisfying (3)). This example is modified from Example 1, in which the intervention matrix is diagonal, and the squared diagonals are generated from a sequence from 0.5 to 1 with equally spaced points so that the variance constraint is satisfied. The true θ is 2 in (3). Other settings remain the same as Example 1.

Example 4 (Non-sparse neighborhood with a causal model satisfying (3)). This example is modified from Example 2, in which the intervention matrix and θ = 2 are set in the same fashion as in Example 3.

Example 5 (Non-sparse case). This example is modified from Example 1, in which the sampling probability of the binomial distribution is increased to 0.1 when generating the edges, the rest are in the same fashion as in Example 1.

Example 6 (Non-sparse case with a causal model satisfying (3)). This example is modified from Example 3, in which the sampling probability of the binomial distribution is increased to 0.1 when generating the edges, the rest are in the same fashion as in Example 3.

As indicated in Tables 1–4, the proposed method with the variance constraint (2) performs favorably against the other methods. In Examples 1 and 3, it performs the best in all the cases in terms of all the three evaluation metrics. In Example 2, it achieves the top performances in terms of FDR and FNR when n = 100, in terms of FNR when n = 200 and performs the best in terms of all the three metrics when n = 500. In Example 4, it also achieves the top performances in terms of FDR when n = 100, in terms of FDR and SHD when n = 200 and in terms of FNR when n = 500. Interestingly, in Examples 2 and 4, PC algorithm and MMHC do not handle “non-sparse neighborhood” structures well as they fail to return results within 24 hours in the case of n = 500. Also, when n is small, ARGES performs poorly, almost identifying no directed edges, such as in the cases of n = 100 of Examples 1–4 and n = 200 of Examples 3 and 4. One explanation is that when n is small the tuning process of the neighborhood selection step tends to select a large penalty, resulting in a very sparse conditional independent graph (CIG).

Table 1.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(100,50)	Our	0.17(0.03)	0.01(0.01)	10.60(2.32)
(100,50)	Int	0.28(0.06)	0.02(0.02)	20.60(6.04)
(100,50)	Obs	0.39(0.20)	0.23(0.07)	39.10(22.90)
(100,50)	PC	0.29(0.06)	0.20(0.07)	21.10(5.20)
(100,50)	GES	0.62(0.05)	0.24(0.06)	64.50(10.28)
(100,50)	MMHC	0.23(0.08)	0.19(0.09)	22.60(6.08)
(100,50)	ARGES	0.50(0.00)	1.00(0.01)	50.00(0.00)
(200,50)	Our	0.05(0.03)	0.00(0.00)	2.50(1.84)
(200,50)	Int	0.14(0.04)	0.00(0.01)	8.20(2.74)
(200,50)	Obs	0.37(0.16)	0.13(0.06)	29.80(13.12)
(200,50)	PC	0.29(0.05)	0.11(0.06)	19.10(3.63)
(200,50)	GES	0.51(0.05)	0.15(0.08)	43.40(5.25)
(200,50)	MMHC	0.18(0.05)	0.09(0.05)	16.30(5.14)
(200,50)	ARGES	0.49(0.04)	0.73(0.15)	48.70(2.41)
(500,50)	Our	0.01(0.02)	0.00(0.00)	0.50(0.97)
(500,50)	Int	0.05(0.05)	0.00(0.00)	2.50(2.80)
(500,50)	Obs	0.18(0.10)	0.00(0.01)	11.70(7.60)
(500,50)	PC	0.28(0.05)	0.04(0.05)	19.00(3.89)
(500,50)	GES	0.45(0.06)	0.13(0.06)	35.50(8.40)
(500,50)	MMHC	0.19(0.07)	0.08(0.06)	13.90(5.53)
(500,50)	ARGES	0.25(0.06)	0.10(0.07)	18.40(4.40)

Table 4.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 4. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘N/A’ means the method did not return a result after 24 hours. The best performers are in bold. ‘NaN’ in the FDR column means the ARGES method does not have any discoveries in all the replications.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(100,50)	Our	0.65(0.09)	0.16(0.09)	93.40(41.51)
(100,50)	Int	0.66(0.12)	0.15(0.05)	100.20(49.61)
(100,50)	Obs	0.76(0.02)	0.22(0.08)	133.60(20.30)
(100,50)	PC	0.87(0.05)	0.91(0.04)	65.70(3.86)
(100,50)	GES	0.63(0.04)	0.24(0.06)	68.70(9.83)
(100,50)	MMHC	0.68(0.03)	0.79(0.02)	60.20(2.70)
(100,50)	ARGES	NaN	1.00(0.00)	49.00(0.00)
(200,50)	Our	0.27(0.15)	0.20(0.03)	21.30(14.74)
(200,50)	Int	0.33(0.13)	0.02(0.04)	28.00(15.24)
(200,50)	Obs	0.61(0.09)	0.05(0.05)	81.90(29.97)
(200,50)	PC	0.82(0.09)	0.86(0.06)	63.60(5.72)
(200,50)	GES	0.46(0.04)	0.06(0.01)	34.60(7.18)
(200,50)	MMHC	0.61(0.04)	0.70(0.02)	57.30(3.71)
(200,50)	ARGES	0.50(0.00)	0.97(0.04)	49.00(0.00)
(500,50)	Our	0.14(0.05)	0.00(0.00)	8.10(2.88)
(500,50)	Int	0.29(0.24)	0.00(0.01)	27.10(23.13)
(500,50)	Obs	0.38(0.05)	0.00(0.00)	29.80(5.85)
(500,50)	PC	N/A	N/A	N/A
(500,50)	GES	0.35(0.06)	0.04(0.00)	17.90(4.95)
(500,50)	MMHC	N/A	N/A	N/A
(500,50)	ARGES	0.05(0.02)	0.05(0.02)	2.80(0.92)

Overall, both the variance constraint (3) and intervention lead to improvements of the reconstruction accuracy across all the situations. Specifically, the proposed method has an average amount of improvement (10.00, 10.50, 8.20, 6.80) in terms of SHD over the intervention method without the variance constraint when n = 100 in all the four examples, respectively. The amount of improvement becomes (5.70, 4.20, 8.30, 6.70) when n increases to 200 and (2.00, 14.50, 2.90, 19.00) when n = 500. The improvement remains noticeable in Examples 2 and 4 even when n is large. Interestingly, a DAG’s structure can be well-reconstructed even without the variance constraint, particularly when n is large, for example, n = 500 in Examples 1 and 3. Moreover, the proposed method has an average improvement of (28.50, 33.80, 29.10, 40.20) in SHD over the observational method when n = 100, (27.30, 1.40, 14.50, 60.60) when n = 200 and (11.20, 16.10, 9.60, 21.70) when n = 500. The amount of improvement is significant across all the cases except the case of n = 200 in Example 2.

In Examples 2 and 4, with a non-sparse neighborhood structure, the estimation becomes more challenging. In such situations, we added the simulations with n = 1000, as shown in Tables 5–6, the proposed method with the variance constraint (2) outperforms all the competitors, with SHD very close to 0 in Example 2 and exactly 0 in Example 4. As a comparison, PC and MMHC fail to return results within 24 hours, GES and ARGES perform even worse than the cases with n = 500. For large sample sizes, GES and ARGES tend to have more false positives, with estimated graph structure failing to determine the directions of the edges between the first node and the rest.

Table 5.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 2. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘N/A’ means the method did not return a result after 24 hours. The best performers are in bold.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(1000,50)	Our	0.02(0.02)	0.00(0.00)	1.20(1.23)
(1000,50)	Int	0.09(0.06)	0.00(0.00)	5.00(3.40)
(1000,50)	Obs	0.17(0.12)	0.00(0.00)	11.10(8.37)
(1000,50)	PC	N/A	N/A	N/A
(1000,50)	GES	0.58(0.01)	0.00(0.00)	58.80(2.20)
(1000,50)	MMHC	N/A	N/A	N/A
(1000,50)	ARGES	0.46(0.15)	0.00(0.01)	44.40(14.90)

Table 6.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 4. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘N/A’ means the method did not return a result after 24 hours. The best performers are in bold.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(1000,50)	Our	0.00(0.00)	0.00(0.00)	0.00(0.00)
(1000,50)	Int	0.04(0.03)	0.00(0.01)	2.10(1.79)
(1000,50)	Obs	0.07(0.14)	0.00(0.01)	5.20(10.00)
(1000,50)	PC	N/A	N/A	N/A
(1000,50)	GES	0.57(0.01)	0.00(0.00)	58.00(2.36)
(1000,50)	MMHC	N/A	N/A	N/A
(1000,50)	ARGES	0.50(0.00)	0.00(0.0)	49.10(0.32)

As demonstrated in Tables 7–8, the benefits of the proposed methods are evident against the competitors in the non-sparse cases of Examples 5 and 6, where each node in the graph has 10 edges instead of 2 edges on average. The proposed method with the variance constraint continues to perform the best in all cases in the three metrics, except the case of n = 200 in which PC performs better in FDR. When the sample size increases from 200 to 500 and then to 1000, our methods exhibit a larger amount of improvement, as the SHD reduces from 135.80 to 61.70 and to 10.60 in Example 5 while from 164.20 to 41.50 and to 8.60 in Example 6. By comparison, ARGES fails to produce a meaningful graph in all cases, GES yields very large values of FDR, FNR, and SHD. Whereas PC and MMHC do perform better than GES, their performances do not improve much at all when the sample size increases.

Table 7.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 5. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘NaN’ in the FDR column means the ARGES method does not have any discoveries in all the replications.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(200,50)	Our	0.34(0.11)	0.02(0.21)	135.80(66.63)
(200,50)	Int	0.47(0.14)	0.21(0.34)	211.20(74.60)
(200,50)	Obs	0.53(0.01)	0.04(0.02)	277.30(9.79)
(200,50)	PC	0.27(0.07)	0.78(0.03)	203.40(7.37)
(200,50)	GES	0.86(0.02)	0.63(0.06)	604.10(23.94)
(200,50)	MMHC	0.60(0.07)	0.90(0.02)	236.60(4.84)
(200,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)
(500,50)	Our	0.18(0.11)	0.01(0.02)	61.70(43.24)
(500,50)	Int	0.36(0.28)	0.21(0.32)	175.20(198.45)
(500,50)	Obs	0.44(0.07)	0.11(0.21)	194.40(41.48)
(500,50)	PC	0.19(0.04)	0.71(0.02)	187.50(5.70)
(500,50)	GES	0.83(0.03)	0.64(0.05)	505.80(31.71)
(500,50)	MMHC	0.55(0.06)	0.87(0.02)	225.70(6.58)
(500,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)
(1000,50)	Our	0.02(0.01)	0.03(0.04)	10.60(10.51)
(1000,50)	Int	0.09(0.06)	0.03(0.02)	29.20(19.47)
(1000,50)	Obs	0.10(0.11)	0.04(0.04)	39.30(40.73)
(1000,50)	PC	0.18(0.04)	0.69(0.02)	181.30(7.24)
(1000,50)	GES	0.80(0.02)	0.71(0.03)	394.40(15.90)
(1000,50)	MMHC	0.55(0.05)	0.85(0.02)	222.20(6.92)
(1000,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)

Table 8.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 6. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘NaN’ in the FDR column means the ARGES method does not have any discoveries in all the replications.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(200,50)	Our	0.39(0.10)	0.04(0.03)	164.20(69.71)
(200,50)	Int	0.50(0.06)	0.04(0.02)	243.60(50.96)
(200,50)	Obs	0.51(0.15)	0.16(0.21)	273.10(126.29)
(200,50)	PC	0.26(0.08)	0.78(0.02)	202.30(6.77)
(200,50)	GES	0.86(0.03)	0.63(0.06)	588.30(27.56)
(200,50)	MMHC	0.60(0.07)	0.90(0.02)	237.70(6.13)
(200,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)
(500,50)	Our	0.14(0.04)	0.00(0.01)	41.50(15.35)
(500,50)	Int	0.27(0.16)	0.10(0.26)	104.80(83.12)
(500,50)	Obs	0.36(0.10)	0.02(0.02)	148.80(56.29)
(500,50)	PC	0.16(0.03)	0.69(0.02)	178.80(5.25)
(500,50)	GES	0.83(0.02)	0.65(0.04)	485.20(26.17)
(500,50)	MMHC	0.59(0.07)	0.88(0.02)	231.30(5.42)
(500,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)
(1000,50)	Our	0.02(0.01)	0.02(0.03)	8.60(6.50)
(1000,50)	Int	0.10(0.04)	0.04(0.02)	31.00(14.91)
(1000,50)	Obs	0.13(0.06)	0.04(0.03)	47.40(20.37)
(1000,50)	PC	0.18(0.05)	0.68(0.02)	180.00(7.39)
(1000,50)	GES	0.80(0.04)	0.71(0.06)	382.80(23.86)
(1000,50)	MMHC	0.57(0.07)	0.85(0.03)	223.50(7.15)
(1000,50)	ARGES	NaN	1.00(0.00)	250.00(0.00)

In conclusion, intervention clearly has a positive impact on the reconstructability of a DAG through interventional covariates. With the variance constraint, the accuracy of reconstruction can be further enhanced.

5.2. Analysis of Alzheimer’s disease dataset

This section applies our proposed method to analyze the Alzheimer’s disease dataset [25], where 8560 gene expressions were collected for 176 Alzheimer’s disease patients and 187 healthy participants. Our primary goal is to reconstruct a causal network of Alzheimer’s disease-related genes with the help of intervention, and compare the DAG structures for the patient and control groups, for identifying regulatory gene-gene interactions that differentiate these two patient groups.

Biologically, transcription factors (TFs) are proteins controlling the transcription process. By regulating genes, TFs ensure that target genes have right expressions in a cell and during a biological organism. In fact, a TF binds to its target DNA sequence and thus can be mapped to specific genes. For our purpose, we use the database [23] to extract a list of human DNA binding TFs and map them to genes in the dataset, resulting in 1031 mapped TF genes. Then we treat TF genes as intervention covariates to facilitate reconstruction of causal relations encoded by the gene networks.

To identify TF gene expressions associated with Alzheimer’s disease in our dataset, we examine the KEGG database [12]. There, 168 genes in the Alzheimer’s disease pathway, among which the expressions of 99 genes are mapped to the pathway. Among the mapped genes, we perform a two-sample t-test to obtain the significant ones between the patient and control groups, resulting in 43 selected genes at the significance level 0.05.

After pre-processing, we apply the proposed method to both the groups to reconstruct two DAG networks for the disease and control groups with 176 and 187 subjects, involving p = 43 genes as causal variables and W = 1031 TFs as intervention variables, where the tuning parameters of the method are estimated by a five-fold cross validation. As shown in Figure 1, there are 29 and 33 estimated directed connections in the patient and control groups, respectively, with 11 shared common directed connections. Moreover, the two networks share some common sub-structures, particularly, we can find common directed connections from NDUFAB1 to ATP5A1, ATP5G3, and COX7A2. However, different sub-structures are also revealed. In the patient group, several genes, including SDHD, NDUFA1, NDUFAB1, ATP5G3, ATP5A1 and ATP5C1 have directed connections to SDHB, whereas in the control group, only two of them are connected to SDHB. This suggests that the differences in the two DAG networks reflect the disparity in the gene regulatory relations between an Alzheimer’s disease subject and a healthy subject.

Fig 1.

Top: DAG network for Alzheimer’s disease patient group. There are 38 directed connections. Bottom: DAG network for healthy participant group. There are 30 directed connections, 11 of which are shared by the DAG network of patient group.

Biologically, our estimated directed connections match the known biological pathway of Alzheimer’s disease in the KEGG database. For example, the estimated directed connections between NDU-genes, SDH-genes, ATP-genes and COX-genes match the pathway of the electron transport chain in mitochondria. Also, the estimated connection from CALM3 to PPP3CB matches the biological pathway from calcium-modulated protein(CaM) to protein phosphatase 3 catalytic subunit alpha(PP3CA).

5.3. Analysis of cytometry data

This section applies the proposed method to analyze the flow cytometry data in [19] to reconstruct causal relations between phosphorylated proteins and phospholipids in human primary naive CD4+ T cells of the immune system. These cells were perturbed with molecular interventions to drive the ordering of connections in an intracellular signaling network. The original flow cytometry data contains n = 7466 cell measurements, each consisting of the amount of p = 11 proteins and phospholipids. The data is collected from nine different experiments in which different components in the network are intervened, either by stimulatory cues or inhibitory interventions. For our analysis, we use the continuous version of the original data [19]. Our objective is to reconstruct this network while the consensus network [19] is used as a benchmark for discovery.

For interventional data, we pre-process by selecting nine out of the eleven components, four of which form a DAG with three directed links while remaining five serve as intervention nodes; see Figure 2 for a display of an enlarge network with the nine nodes, known as consensus network, where links between the five intervention nodes are purposely removed for the intervention purpose. Note that the removal of these links does not affect our analysis, because each node is independent of its non-descendants given its direct parents by the local Markov property. Among the five intervention nodes, only three of them are informative with the other two Plcg and PIP2 having no effects on the four nodes to be intervened.

Fig 2.

Consensus intracellular signaling network as benchmark. Four nodes in brown are intervened by five intervention nodes marked in blue, with directed links indicating their intervention directions by solid red lines. Note that no links are present between intervention nodes.

We fit our proposed method with and without interventions. For tuning, we use one-tenth of the samples for training, and the rest for tuning. As displayed in Figures 3 and 4, the proposed method with intervention correctly identifies the directed link from Erk to Akt, while the observational method alters the direction oppositely. Both the methods enable to reconstruct the directed link from Praf to Pmek, whereas they both miss the link from Pmek to Erk. Consequently, the proposed method with intervention identifies more correct directed links than that without intervention. With respect to the intervention effects, our method identifies three intervention links in the network, which rule out the true non-informative intervention nodes Plcg and PIP2. PIP3 is also non-informative in our estimation, and a possible reason is that Akt is already intervened by PKA, and thus the intervention effect of PIP3 on Akt is unnecessary. Similarly, the intervention from PKC to Praf is identified by the proposed method, while bothPKA and PKC have interventions on Praf in the consensus network. Finally, the proposed method yields a sparse intervention pattern.

Fig 3.

Intracellular signaling network reconstructed by the proposed method with intervention. Two directed links in solid black are correctly identified, denoted by solid black lines, while the directed link from Pmek to Erk is missed, denoted by a dashed black line. Three intervention links are identified by our model (denoted by solid red lines), all of which are present in the consensus network.

Fig 4.

Intracellular signaling network estimated by the model without intervention. One directed connection is correctly identified, denoted by the solid black line, while another connection is reversed, denoted by dotted black line, and the directed connection from Pmek to Erk is missed, denoted by dashed black line.

One plausible explanation of missing the link from Pmek to Erk is that the linear causal model fails to capture nonlinear functional relations among cytometry measurements of proteins Praf, Pmek, Erk, and Akt, as evident from nonlinear patterns revealed by their residual plots of the structural equation model in Figure 5. Another possibility is that the five nodes on the bottom-left in Figure 3 are not originally designed as intervention nodes in the experiments.

Fig 5.

Residual plots from our intervention model. Each subplot corresponds to one node in the DAG network.

6. Discussion

In this paper, a constrained maximum likelihood method is proposed to reconstruct the structure of a DAG using interventional data and efficient ADMM algorithms are developed to solve the optimization problems. In particular, a novel variance constraint is introduced and leverages the information in the interventional data to improve identifiability. Theories are established and it is shown that with the introduction of our variance constraint, the graphical structure is shown to be fully identifiable under some mild assumptions. The theoretical results are also demonstrated in our simulation results, as our proposed method performs well against competitors and can reconstruct the DAG more accurately than the observational method.

Table 2.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 2. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘N/A’ means the method did not return a result after 24 hours. The best performers are in bold.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(100,50)	Our	0.59(0.19)	0.05(0.05)	81.40(29.54)
(100,50)	Int	0.63(0.13)	0.07(0.07)	91.90(35.95)
(100,50)	Obs	0.67(0.16)	0.21(0.09)	115.20(65.37)
(100,50)	PC	0.84(0.06)	0.90(0.04)	63.20(2.57)
(100,50)	GES	0.61(0.05)	0.17(0.06)	66.00(10.80)
(100,50)	MMHC	0.68(0.04)	0.80(0.02)	60.40(3.47)
(100,50)	ARGES	0.50(0.00)	0.96(0.04)	49.00(0.00)
(200,50)	Our	0.38(0.18)	0.01(0.02)	35.20(18.56)
(200,50)	Int	0.37(0.24)	0.00(0.01)	39.40(29.05)
(200,50)	Obs	0.29(0.25)	0.06(0.06)	36.60(51.85)
(200,50)	PC	0.78(0.07)	0.82(0.06)	63.10(5.11)
(200,50)	GES	0.44(0.05)	0.06(0.01)	32.80(7.15)
(200,50)	MMHC	0.60(0.05)	0.69(0.02)	57.30(4.22)
(200,50)	ARGES	0.07(0.02)	0.23(0.09)	11.90(4.12)
(500,50)	Our	0.07(0.16)	0.00(0.01)	6.40(16.78)
(500,50)	Int	0.29(0.10)	0.00(0.00)	20.90(8.27)
(500,50)	Obs	0.28(0.18)	0.00(0.00)	22.50(15.20)
(500,50)	PC	N/A	N/A	N/A
(500,50)	GES	0.38(0.13)	0.03(0.02)	25.90(20.59)
(500,50)	MMHC	N/A	N/A	N/A
(500,50)	ARGES	0.11(0.14)	0.04(0.01)	7.30(15.02)

Table 3.

Averaged false discovery rate (FDR), false negative rate (FNR), and Structural Hamming Distance (SHD), as well as their standard errors for four competing methods based on 10 simulation replications in Example 3. Here “Our”, “Int”, “Obs”, “PC”, “GES”, “MMHC”, “ARGES” denote the proposed method subject to the variance constraint, the proposed method without the variance constraint, the proposed method without intervention, PC method, GES method, MMHC method and ARGES method, respectively. The best performers are in bold. ‘NaN’ in the FDR column means the ARGES method does not have any discoveries in all the replications.

(n,p)	Method	FDR(A)	FNR(A)	SHD
(100,50)	Our	0.11(0.05)	0.04(0.04)	7.60(3.41)
(100,50)	Int	0.22(0.08)	0.05(0.04)	15.80(5.63)
(100,50)	Obs	0.44(0.08)	0.14(0.05)	36.70(9.48)
(100,50)	PC	0.30(0.05)	0.21(0.07)	22.80(4.54)
(100,50)	GES	0.61(0.05)	0.21(0.09)	63.30(7.97)
(100,50)	MMHC	0.27(0.09)	0.22(0.11)	26.50(8.61)
(100,50)	ARGES	NaN	1.00(0.00)	50.00(0.00)
(200,50)	Our	0.04(0.06)	0.00(0.00)	2.30(3.40)
(200,50)	Int	0.16(0.11)	0.00(0.01)	10.60(9.07)
(200,50)	Obs	0.25(0.06)	0.03(0.04)	16.80(4.92)
(200,50)	PC	0.27(0.06)	0.12(0.05)	18.00(4.14)
(200,50)	GES	0.53(0.04)	0.17(0.05)	44.70(6.43)
(200,50)	MMHC	0.22(0.07)	0.08(0.06)	18.10(5.34)
(200,50)	ARGES	0.50(0.00)	1.00(0.00)	50.00(0.00)
(500,50)	Our	0.01(0.01)	0.00(0.00)	0.40(0.52)
(500,50)	Int	0.06(0.02)	0.00(0.00)	3.30(1.34)
(500,50)	Obs	0.17(0.05)	0.02(0.02)	10.00(3.40)
(500,50)	PC	0.27(0.04)	0.05(0.03)	18.10(3.31)
(500,50)	GES	0.42(0.08)	0.10(0.06)	32.00(8.51)
(500,50)	MMHC	0.14(0.04)	0.03(0.03)	8.80(3.01)
(500,50)	ARGES	0.43(0.06)	0.46(0.16)	41.80(6.07)

Appendix A: Technical Details

A.1. Computation details for solving (9)

For (9), after plugging in $\widehat{\theta }$ and fix D, we proceed in two steps. First, we relax nonconvex constraints (7) and (8) using a sequence of convex approximations involving 2^p + 1 linear constraints, where each approximation is refined iteratively. Then we solve each subproblem by employing a constrained alternating direction method of multipliers to estimate. The underlying process iterates until convergence.

For convex relaxation of nonconvex constraints (7) and (8) in (9), we employ difference convex (DC) programming similar to [27]. In particular, we decompose J_τ into a difference of two convex functions: ${\text{J}}_{\tau }(z)=S_1(z)-S_2(z)\equiv \text{min}\left(\frac{|z|}{\tau },1\right)=\frac{|z|}{\tau }-\text{max}\left(\frac{|z|}{\tau }-1,0\right)$. On this ground, we construct a sequence of convex approximating sets iteratively by replacing S₂ in the decomposition at iteration m by its affine majorization at iteration m − 1. (9) becomes

$${\text{min}}_{(A,B,\lambda )}l(A,B,\lambda )$$

$$\text{subj to }\frac{1}{\tau }{\sum }_{1\le j\ne l\le p}|A_{jl}|w_{jl}^{(m-1)}\le K_1-{\sum }_{1\le j<l\le p}\left(1-w_{jl}^{(m-1)}\right),$$

$$\frac{1}{\tau }{\sum }_{1\le j\le p,1\le l\le W}|B_{jl}|v_{jl}^{(m-1)}\le K_2-{\sum }_{1\le j\le p,1\le l\le W}\left(1-v_{jl}^{(k,m-1)}\right),$$

$${\lambda }_{js}+\tau I(l\ne s)-{\lambda }_{ls}\ge {|A_{jl}|}_1{w}_{jl}^{(m-1)}+\tau \left(1-w_{jl}^{(m-1)}\right);$$

$$j,l,s=1,\dots ,p,j\ne l,$$

$$B{B}^{\top }+D=\widehat{\theta }I,$$ (19)

where $w_{jl}^{(m-1)}=I\left({\Vert {\widehat{\mathit{A}}}_{jl}^{\left(m-1\right)}\Vert }_1\le \tau \right)$, $v_{jl}^{(m-1)}=I\left(|{\widehat{\mathit{B}}}_{jl}^{(m-1)}|\le \tau \right);1\le i,j\le p$ and $\left({\widehat{\mathit{A}}}^{(m-1)},{\widehat{\mathit{B}}}^{(m-1)}\right)$ is the solution at iteration m − 1.

Now consider a regularization version of (19), with a slack variable ξ added to the inequality constraint, yielding (16).

A.2. Analytic updating expressions for ADMM in (20)

At ADMM iteration step s + 1, the updating formula are

$$A^{(s+1)}={\text{argmin}}_A{L}_{\rho }\left(A,C^{(s)},B^{(s)},{\lambda }^{(s)},{\xi }^{(s)},y^{(s)},U^{(s)},V^{(s)}\right),$$

$$\begin{gathered} C^{(s+1)}={\text{argmin}}_C{L}_{\rho }\left(A^{(s+1)},C,B^{(s)},{\lambda }^{(s)},{\xi }^{(s)},y^{(s)},U^{(s)},V^{(s)}\right) \\ \text{ subj to }C{C}^{\top }+\widehat{\mathit{D}}={\sigma }^{2}I, \end{gathered}$$

$$B^{(s+1)}={\text{argmin}}_B{L}_{\rho }\left(A^{(s+1)},C^{(s+1)},B,{\lambda }^{(s)},{\xi }^{(s)},y^{(s)},U^{(s)},V^{(s)}\right),$$

$${\lambda }^{(s+1)}={\text{argmin}}_{\lambda }{L}_{\rho }\left(A^{(s+1)},C^{(s+1)},B^{(s+1)},\lambda ,{\xi }^{(s)},y^{(s)},U^{(s)},V^{(s)}\right),$$

$$\begin{gathered} {\xi }^{(s+1)}={\text{argmin}}_{\xi }{L}_{\rho }\left(A^{(s+1)},C^{(s+1)},B^{(s+1)},{\lambda }^{(s+1)},\xi ,y^{(s)},U^{(s)},V^{(s)}\right), \\ \text{ subj to }{\xi }_{ijk}\ge 0;i,j,k=1,\dots ,p,j\ne k, \end{gathered}$$

$$y_{ijk}^{(s+1)}=y_{ijk}^{(s)}+\left(|F_{ij}^{(s+1)}|+{\xi }_{ijk}^{(s+1)}-\tau {\lambda }_{ik}^{(s+1)}-\tau I(j\ne k)+\tau {\lambda }_{jk}^{(s+1)}\right),$$

$$U_{jl}^{(s+1)}=U^{(s)}+\left(A_{jl}^{(s+1)}-F_{jl}^{(s+1)}\right),1\le j,l\le p,$$

$$U_{j,l+p}^{(s+1)}=U^{(s)}+\left(B_{jl}^{(s+1)}-F_{j,l+p}^{(s+1)}\right),1\le j\le p,1\le l\le W,$$

$$V_{jl}^{(s+1)}=V_{jl}^{(s)}+\left(C_{jl}^{(s+1)}-B_{jl}^{(s+1)}\right),1\le j\le p,1\le l\le W.$$ (20)

A.2.1. A-step and B-step

For simplicity, denote H = (A, B) be the concatenation of adjacency matrix and intervention matrix, let Z = (Y, X) be the concatenated data matrices. For each row of H, the optimization problem is summarized as ${\text{min}}_{H_{j,j}-}\frac{1}{2}{\Vert z_j-Z_{j^{-}}{H}_{j,j^{-}}^{\top }\Vert }^2+\frac{\rho }{2}{\Vert H_{j,j^{-}}-F_{j,j^{-}}^{(s)}+U_{j,j^{-}}^{(s)}\Vert }_F^2$, where H_{j, j−} is the jth row of H with H_jj excluded, Z_j− is Z with its jth column removed and x_j is the jth column of X. The minimizer is the solution to $\left(Z_{j^{-}}^{\top }{Z}_{j^{-}}+\rho \mathit{I}\right)H_{j,j^{-}}=Z_{j^{-}}^{\top }{z}_j+\rho \left(F_{j,j^{-}}^{(s)}-U_{j,j^{-}}^{(s)}\right)$, where the factorization of $Z_{j^{-}}^{\top }{Z}_{j^{-}}+\rho I$ can be cached to speed up subsequent updates.

A.2.2. C-step

By Lemma 1, the updating formula for C is C^(s+1) = Λ^1/2PO^⊤, where O and P are obtained from singular value decomposition of (B^(s) − V ^(s))^⊤ Λ^1/2 = OEP^⊤.

A.2.3. F-step

F-step updates two parts of the matrix, one is the adjacency matrix, one is the intervention matrix.

Part I: adjacency matrix For i = 1, ⋯ , p and j = 1, ⋯ , p, we solve the following problem:

$$\underset{F}{\text{min}}{\mu }_1\sum _{i,j}|F_{ij}|w_{ij}^{m-1}+\frac{\rho }{2}\sum _{ijk}\left(|F_{ij}|w_{ij}^{m-1}+\tau \left(1-w_{ij}^{m-1}\right)-L_{ijk}^s\right)+\frac{\rho }{2}\sum _{i,j}{\left(A_{ij}^{s+1}-F_{ij}+U_{ij}^s\right)}^2,$$

where $L_{ijk}^s={\lambda }_{ik}^s+\tau I(j\ne k)-{\lambda }_{jk}^s-{\xi }_{ijk}^s-y_{ijk}^s$.

We can solve F_ij elementwise: $F_{ij}^{s+1}=\{\begin{array}{ll}S\left(\frac{A_{ij}^{s+1}+U_{ij}^s}{1+p},\frac{{\mu }_1-p\rho \sum _k{L}_{ijk}^s}{\rho (1+p)}\right)\hfill & \text{ if }{w}_{ij}^{m-1}=1,\hfill \\ A_{ij}^{s+1}+U_{ij}^s\hfill & \text{ if }{w}_{ij}^{m-1}=0,\hfill \end{array}$ where $S(b,\lambda )=\{\begin{array}{ll}b-0.5\lambda \hfill & \text{ if }b>0.5\lambda ,\hfill \\ b+0.5\lambda \hfill & \text{ if }b<-0.5\lambda ,\hfill \\ 0\hfill & \text{ otherwise,}\hfill \end{array}$ is the soft-thresholding operator.

Part II: intervention matrix For i = 1, ⋯ , p and j = p + 1, ⋯ , p + W, we solve the following problem:

$$\underset{F}{\text{min}}{\mu }_2\sum _{i,j}|F_{ij}|v_{ij}^{m-1}+\frac{\rho }{2}\sum _{i,j}{\left(A_{ij}^{s+1}-F_{ij}+U_{ij}^s\right)}^2+\frac{\rho }{2}\sum _{i,j}{\left(C_{i,j-p}^s-F_{ij}+Z_{i,j-p}^s\right)}^2.$$

We can solve F_ij elementwise:

$$F_{ij}^{s+1}=\{\begin{array}{ll}S\left(\frac{1}{2}\left(A_{ij}^{s+1}+U_{ij}^s+C_{i,j-p}^s+Z_{i,j-p}^s\right),\frac{{\mu }_2}{2\rho }\right)\hfill & \text{ if }{v}_{ij}^{m-1}=1,\hfill \\ \frac{1}{2}\left(A_{ij}^{s+1}+U_{ij}^s+C_{i,j-p}^s+Z_{i,j-p}^s\right)\hfill & \text{ if }{v}_{ij}^{m-1}=0.\hfill \end{array}$$

A.2.4. λ-step and ξ-step

(λ^s+1, ξ^s+1) is updated by ${\lambda }^{s+1}=M_{p\times p}{W}_{p\times p}^{s+1}$,

where

$$M_{p\times p}=\frac{1}{\tau }\left(\begin{array}{ccccc}1& 0& 0& \dots & 0\\ 1& \frac{2}{p}& \frac{1}{p}& \dots & \frac{1}{p}\\ 1& \frac{1}{p}& \frac{2}{p}& \dots & \frac{1}{p}\\ ⋮& ⋮& & \ddots & ⋮\\ 1& \frac{1}{p}& \dots & \frac{1}{p}& \frac{2}{p}\end{array}\right)$$

$$W_{1j}^{s+1}=1,$$

$$\begin{gathered} W_{ik}^{s+1}=\frac{1}{2}(\tau +{\sum }_j\left(|B_{kj}^{s+1}|w_{ij}^{(m-1)}+\tau \left(1-w_{ij}^{(m-1)}\right)+{\xi }_{ijk}^{s+1}+y_{ijk}^s\right) \\ -{\sum }_j\left(|B_{kj}^{s+1}|w_{ij}^{(m-1)}+\tau \left(1-w_{ij}^{(m-1)}\right)+{\xi }_{jik}^{s+1}+y_{jik}^s\right));i\ne k, \end{gathered}$$

$$\begin{gathered} W_{kk}^{s+1}=\frac{1}{2}(-(p-1)\tau +{\sum }_j\left(|B_{kj}^{s+1}|w_{ij}^{(m-1)}+\tau \left(1-w_{ij}^{(m-1)}\right)+{\xi }_{kjk}^{s+1}+y_{kjk}^s\right) \\ -{\sum }_j\left(|B_{kj}^{s+1}|w_{ij}^{(m-1)}+\tau \left(1-w_{ij}^{(m-1)}\right)+{\xi }_{jkk}^{s+1}+y_{jkk}^s\right)), \end{gathered}$$

for i, j, k = 1, … , p.

$${\xi }_{ijk}^{s+1}≔\text{max}\left(0,\left(\tau {\lambda }_{ik}^s+\tau I(j\ne k)-\tau {\lambda }_{jk}^s-|B_{ij}^{s+1}|-y_{ijk}^s\right)\right);i,j,k=1,\dots ,p.$$

A.3. Computation details for estimating D₀

A good estimate of D₀ is obtained by solving (9) without the variance constraint (3). First, we introduce a re-parametrization to exploit the convexity. Let R = D^−1/2, Φ = D^−1/2A and Ψ = D^−1/2B, then (2) becomes Y = R⁻¹ΦY + R⁻¹ΨX + R⁻¹ϵ, ϵ ~ N(0, I) and Φ = (ϕ_jk)_p×p, Ψ = (ψ_jw)_p×W are scaled versions of adjacency matrix A and intervention matrix B and R = (r₁, ⋯ , r_p). Under the new parametrization, the likelihood (4) becomes $l(\mathbf{\Phi },\mathbf{\Psi },\mathit{R})={\sum }_{j=1}^p\left[-n\text{ log }{r}_j+\frac{1}{2}{\sum }_{i=1}^n{\left(r_j{y}_{ij}-{\sum }_{k\ne j}{\varphi }_{jk}{y}_{ik}-{\sum }_{w=1}^W{\psi }_{jw}{x}_{iw}\right)}^2\right]$, an easy check will confirm that l(Φ, Ψ, R) is convex in (Φ, Ψ, R). Then (9) without variance constraint (3) is written as

$${\text{min}}_{(\Phi ,\Psi ,R)}l(\Phi ,\Psi ,R)={\sum }_{j=1}^p\left[-n\text{ log }{r}_j+\frac{1}{2}{\sum }_{i=1}^n{\left(r_j{y}_{ij}-{\sum }_{k\ne j}{\varphi }_{jk}{y}_{ik}-{\sum }_{w=1}^W{\psi }_{jw}{x}_{iw}\right)}^2\right],$$

$$\text{subj to }{\sum }_{1\le j<l\le p} {\text{J}}_{\tau }\left({\varphi }_{jl}\right)\le K_1,{\sum }_{1\le j\le p,1\le l\le W} {\text{J}}_{\tau }\left({\psi }_{jl}\right)\le K_2,$$

$${\sum }_{j_1=j_{L+1}:1\le k\le L} {\text{J}}_{\tau }\left({\varphi }_{j_k{j}_{k+1}}\right)\le L-1;\text{ any }\left(j_1,\dots ,J_L\right),L=2,\dots ,p.$$ (21)

To solve (21), following the computation strategy illustrated in Appendix 7.1 and 7.2, after transformation, in the mth DC step, we solve

$${\text{min}}_{(\mathbf{\Phi },\mathbf{\Psi },\mathit{F},\mathit{\lambda },\mathit{\xi },\mathit{y},\mathit{U},\mathit{R})}{L}_{\rho }(\mathbf{\Phi },\mathbf{\Psi },\mathit{F},\mathit{\lambda },\mathit{\xi },\mathit{y},\mathit{U},\mathit{R})=l(\mathbf{\Phi },\mathbf{\Psi },\mathit{R})+\frac{{\mu }_1}{\tau }{\sum }_{1\le j\ne l\le p}|F_{jl}|w_{jl}^{(m-1)}+\frac{{\mu }_2}{\tau }{\sum }_{1\le j\le p,1\le l\le W}|{\psi }_{jl}|v_{jl}^{(m-1)}+{\sum }_{1\le s\le p}{\sum }_{1\le j\ne l\le p}\frac{\rho }{2}{\left(|F_{jl}|w_{jl}^{(m-1)}+\tau \left(1-w_{jl}^{(m-1)}\right)+{\xi }_{jls}-{\lambda }_{jl}-\tau I(l\ne s)+{\lambda }_{ls}+y_{jls}\right)}^2+\frac{\rho }{2}{\sum }_{1\le j,l\le p}{\left({\varphi }_{jl}-F_{jl}+U_{jl}\right)}^2+\frac{\rho }{2}{\sum }_{1\le j\le p,1\le l\le W}{\left({\psi }_{jl}-F_{j,l+p}+U_{j,l+p}\right)}^2,$$ (22)

which is solved over blocks (Φ, Ψ, F, λ, ξ, y, U, R). The updating formula are similar to those in Appendix 7.2, with one exception, which is R-step, which is illustrated in the following section.

After solving (21), estimates ($\widehat{\mathbf{\Phi }}$, $\widehat{\mathbf{\Psi }}$, $\widehat{\mathit{R}}$) are obtained, then ${\mathit{D}}_0={\widehat{\mathit{R}}}^{-2}$.

A.3.1. R-step

Denote T = (Φ, Ψ) be the concatenation of Φ and Ψ, let Z = (Y , X) be the concatenated data matrices we solve the minimization problem: ${\text{min}}_{r_j}-n\text{ log }{r}_j+\frac{1}{2}{\Vert r_j{z}_j-{\mathit{Z}}_{j^{-}}{\mathit{T}}_{j{j}^{-}}^T\Vert }^2$ The derivative equation becomes $-\frac{n}{r_j}+{\sum }_{i=1}^n\left(r_j{z}_{ij}-{\sum }_{k\ne j}{T}_{jk}{z}_{ik}\right)z_{ij}=0$ , yielding a solution $r_j=\frac{b+\sqrt{b^2+4na}}{2a}$, where $a={\sum }_{i=1}^n{z}_{ij}^2$ and $b={\sum }_{i=1}^n{z}_{ij}{\sum }_{k\ne j}{T}_{jk}{z}_{ik}$.

A.4. Technical proofs

Proof of Lemma 2. Let ϵ′ = BX + ϵ, under (3), (2) reduces to

$$Y=AY+{\mathit{ϵ}}^{\prime },$$ (23)

where ϵ′ ~ N(0, θI). In (23), Ω = (I − A)^⊤ (I − A)/θ . Given A and the distribution of Y , the value of θ is unique. This completes the proof. □

Proof of Theorem 1. Let ϵ′= BX + ϵ. Note that X ~ N(0, Σ_X). Hence ϵ′ ~ N(0, θI), which is independent of any specific value of B. Now (2) is written as

$$Y=AY+{\mathit{ϵ}}^{\prime }.$$ (24)

By (3), ϵ′~ N(0, θI), (24) reduces to an observational model with equal error variance. By Theorem 1 of [18], A is identifiable from the distribution of Y , which in turn is identifiable from the joint distribution (Y , X) that is proportional to the distribution of Y . This completes the proof.□

Proof of Lemma 1. The update step for C in ADMM is

$$C^{(k+1)}={\text{argmin}}_C{\Vert C-B^{(k)}+V^{(k)}\Vert }_F^2,\text{ s.t. }C{C}^{\top }=\Lambda ,$$ (25)

(25) is equivalent to max_C Tr [C^⊤ (B^(k) − V^(k))], s.t. CC^⊤ = Λ , we can transform it into a standard form by introducing Q = C^⊤ Λ^−1/2 and S = (B^(k) − V ^(k+1))^⊤ Λ^1/2. Then (25) becomes

$$\underset{Q}{\text{max}}\text{ Tr }\left[Q^{\top }S\right],\text{ s.t. }{Q}^{\top }Q=I.$$ (26)

To solve (26), we can do a singular value decomposition S = OEP^⊤, where $O\in {ℝ}^{w\times p}$, E, $P\in {ℝ}^{p\times p}$, O^⊤O = P^⊤P = PP^⊤ = I. Then the cost function in (26) can be written in the form $\text{Tr }\left[Q^{\top }OE{P}^{\top }\right]=\text{Tr }\left[{(QP)}^{\top }OE\right]={\sum }_{j=1}^p{\left[M^{\top }O\right]}_{jj}{E}_{jj}$, where $M=QP\in {ℝ}^{w\times p}$ and [M^⊤O]_jj denotes the jth diagonal element of the cross product M^⊤O. Note that the constraint Q^⊤Q = I is equivalent to M^⊤M = I since P is an orthogonal matrix. Since E_jj ≥ 0, if we can maximize each of [M^⊤O]_jj; j = 1, ⋯ , p at the same time, then the problem is solved.

Since ${\left[M^{\top }O\right]}_{jj}={\sum }_{i=1}^W{M}_{ij}{O}_{ij}$, by the Cauchy-Schwartz inequality and the fact that M^⊤M = O^⊤O = I , we have ${\sum }_{i=1}^W{M}_{ij}{O}_{ij}\le \sqrt{{\sum }_{i=1}^W{M}_{ij}^2\sum _{i=1}^W{O}_{ij}^2}=\sqrt{1\times 1}=1$, where the equality holds if and only if M_ij = O_ij for i = 1, 2, ⋯ , W. If we maximize each [M^⊤O]_jj; j = 1, ⋯ , p at the same time, we need M = O, which leads to Q = OP^⊤. Then the solution of (25) is C^(k+1) = Λ^1/2PO^⊤. This completes the proof. □

Proof of Theorem 2. When we ignore the intervention covariates X and pull them into the error terms, under the variance constraint (3), (2) reduces to (24) with equal error variances, under which A is identifiable from Ω. Note that Ω^obs = (I − A^obs)^⊤(I − A^obs)/θ is a function of (A^obs, θ), it follows that $P\left(\widehat{\theta }\ne {\widehat{\theta }}_O\right)\le P\left({\widehat{\mathbf{\Omega }}}^{obs}\ne {\widehat{\mathbf{\Omega }}}_O^{obs}\right)$. The rest of the proof follows from Theorem 3 of [27]. This completes the proof. □

Proof of Theorem 3. First, we define a complexity measure for the size of a space $\mathcal{F}$. The bracketing Hellinger metric entropy of $\mathcal{F}$, denoted by $H(\cdot ,\mathcal{F})$, is the logarithm of the cardinality of the u-bracketing of $\mathcal{F}$ of the smallest size. That is, for a bracket covering $S(ϵ,m)=\left\{f_1^l,f_1^u,\cdots ,f_m^l,f_m^u\right\}\subset {\mathcal{L}}_2$ satisfying ${\text{max}}_{1\le j\le m}{\Vert f_j^u-f_j^l\Vert }_2\le ϵ$ and for any $f\in \mathcal{F}$, there exists a j such that $f_j^l\le f\le f_j^u$ a.e. P , then $H(u,\mathcal{F})=\text{log }(\text{min}\left\{m:S(u,m)\right\})$, where ∥f∥₂ = ∫ f² (z) dμ, with μ the dominating measure.

Denote E^τ = {(i, j) : |A_ij ≥ τ}. When K = |E⁰|, ${\sum }_{1\le i\ne j\le p}{J}_{\tau }\left(A_{ij}\right)\le |E^0|$, so $|{\widehat{E}}^{\tau }|\le |E^0|$. If ${\widehat{E}}^{\tau }=E^0$, then ${\sum }_{1\le i\ne j\le p}|A_{ij}|I\left(|A_{ij}|<\tau \right)=0$, then $\widehat{\mathit{A}}={\widehat{\mathit{A}}}_O$. Therefore, it suffices to prove the case when ${\widehat{E}}^{\tau }\ne E^0$.

Define ${\mathbf{\Omega }}_{E^{\tau }}={\left(\mathit{I}-{\mathit{A}}_{E^{\tau }}\right)}^{\top }\left(\mathit{I}-{\mathit{A}}_{E^{\tau }}\right)/{\theta }_{E^{\tau }}$ for any E^τ ⊂ {(i, j) : 1 ≤ i ≠ j ≤ p}. We can partition E^τ as E^τ = (E^τ \ E⁰) ∩ (E ∪ E⁰). Let $B_{kj}=\{{\mathbf{\Omega }}_{E^{\tau }}:E^{\tau }\ne E^0,|E^{\tau }\cap E^0|=k,|E^{\tau }\backslash E^0|=j,\left(d_1\left(|E^0|-k\right)C_{\text{min}}\left({\mathbf{\Omega }}^0\right)-d_{3}q{\tau }^{d_2}\right)\le h^2\left({\mathbf{\Omega }}_{E^{\tau }},{\mathbf{\Omega }}^0\right)\};k=0,\dots ,|E^0|-1,j=1,\dots ,|E^0|-k$. Then B_kj has $\left(\begin{array}{c}|E^0|\\ k\end{array}\right)\left(\begin{array}{c}p(p-1)-|E^0|\\ j\end{array}\right)$ different elements E^τ’s of sizes |E^τ ∩ E⁰| = k, |E^τ \ E⁰| = j. By definition $\left\{{\mathbf{\Omega }}_{E^{\tau }}:E^{\tau }\ne E^0,|E^{\tau }\ne E^0|\le |E^0|,C_{\text{min}}\left({\mathbf{\Omega }}^0\right)\le h^2\left({\mathbf{\Omega }}_{E^{\tau }},{\mathbf{\Omega }}^0\right)\right\}\subset {\cup }_{k=0}^{|E^0|-1}{\cup }_{j=1}^{|E^0|-k}{B}_{kj}$. Let L(Ω) = logf(Ω, y, x) where f(Ω, y, x) = f(Ω, y|x)f_X(x) is the joint density. Then

$$\begin{gathered} P\left(\widehat{G}\ne G^0\right) \\ \le P\left(\widehat{\mathbf{\Omega }}\ne {\widehat{\mathbf{\Omega }}}_O\right) \\ \le P^{*}\left(\underset{{\mathbf{\Omega }}_E\tau :E^{\tau }\ne E^0,|E^{\tau }|\le |E^0|}{\text{sup}}\left(L\left({\mathbf{\Omega }}_{E^{\tau }}\right)-L\left({\widehat{\mathbf{\Omega }}}_O\right)\right)\ge 0\right) \\ \le P^{*}\left(\underset{{\mathbf{\Omega }}_E\tau :E^{\tau }\ne E^0,|E^{\tau }|\le |E^0|}{\text{sup}}\left(L\left({\mathbf{\Omega }}_{E^{\tau }}\right)-L\left({\mathbf{\Omega }}^0\right)\right)\ge 0\right) \\ \le \sum _{E^{\tau }\subset \{(i,j):1\le i<j\le p\}:E^{\tau }\ne E^0,|E^{\tau }|\le |E^0|}{P}^{*}\left(\underset{{\mathbf{\Omega }}_E\tau \in B_{kj}}{\text{sup}}\left(L\left({\mathbf{\Omega }}_{E^{\tau }}\right)-L\left({\widehat{\mathbf{\Omega }}}_O\right)\right)\ge 0\right)\equiv I, \end{gathered}$$

where P* is the outer measure.

For I, we apply Theorem 1 of [26] to bound each term in the sum. We verify the entropy condition (3.1) there for the bracketing entropy over B_kj. Let Π denote the covariance matrix of the joint distribution of (Y , X) and Γ = Π⁻¹, then Ω is the upper p×p diagonal block of Γ. Let ${\mathcal{F}}_{kj}=\left\{f^{1/2}(\Gamma ,\cdot ,\cdot ):\mathbf{\Omega }\in B_{kj}\right\}$ be the class of square-root densities. Define $\Delta =\tilde{\mathbf{\Gamma }}-\mathbf{\Gamma }$ and let λ₁, …, λ_p+W be the eigenvalues of $\sqrt{\mathbf{\Pi }}\Delta \sqrt{\mathbf{\Pi }},z={\left(y^{\top },x^{\top }\right)}^{\top }$. Then max_1≤i≤p+W λ_i ≤ λ_max(Δ) × λ_max(Π) ≤ c(p + W)∥Π∥_max following Prob.III.6.14 [1]. By Lemma 6.5 of [28], it can be shown

$$\begin{gathered} |\text{log }f(\tilde{\Gamma },y,x)-\text{log }f(\Gamma ,y,x)|\le \underset{1\le i\le p+W}{\text{max}}{\lambda }_i(D(z)+p+W) \\ \le c(p+W)\Vert \mathbf{\Delta }{\Vert }_{\text{max}}(D(z)+p+W), \end{gathered}$$

where λ_max denotes the largest eigenvalue, D(z) = λ_max(Γ) tr(zz^⊤) ≤ M₄ × tr(zz^⊤). Note that E tr(zz^⊤) ≤ c(p + W) for some constant c > 0. By Assumption A.2, ${\lambda }_{\text{max}}^2(\sqrt{\mathbf{\Pi }})\le 1/M_3$. Then

$$\begin{gathered} I^{\prime }\equiv \int \underset{\tilde{\Gamma }\in B_{\delta }(\Gamma )}{\text{sup}}{\left(f^{1/2}(\tilde{\Gamma },y,x)-f^{1/2}(\Gamma ,y,x)\right)}^{2}d\mu \\ \le \underset{\tilde{\Gamma }\in B_{\delta }(\Gamma )}{\text{sup}}{c}^{\prime }(p+W)\Vert \mathbf{\Delta }{\Vert }_{\text{max}}^{2}E{(D(z)+p+W)}^2, \end{gathered}$$

for some constant c′ > 0.

Then for some positive constant Q, I′ ≤ Q(p + W)⁴δ². By Lemma 1 of [15], it suffices to bound the entropy of B_ij. There are |E| nonzero entries of A with $\left(\begin{array}{c}p(p-1)\\ |E|\end{array}\right)$ possible locations. By [13], for u ≥ ϵ²,

$$\begin{gathered} H\left(u,{\mathcal{F}}_{ij}\right)\le c_0\left(\text{log }\left(\begin{array}{c}p(p-1)\\ |E|\end{array}\right)+|E|\text{log }\left(\frac{\text{min}\left(M_2^{1/2},1\right)}{u}\right)\right) \\ \le c_0\left(|E|\text{log }\left(e\frac{p(p-1)}{|E|}\right)+|E|\text{log }\left(\frac{\text{min}\left(M_2^{1/2},1\right)}{u}\right)\right) \\ \le c_0(|E|\text{log }p\text{log }(1/u)) \end{gathered}$$

Then $ϵ={ϵ}_{n,p,|E^0|}=\text{min}\left(1,{\left(2c_0\right)}^{1/2}{c}_4^{-1}\text{log }\left(2^{1/2}/c_3\right)\text{log }p{\left(|E^0|/n\right)}^{1/2}\right)$ satisfies

$$\underset{0\le |E^{\tau }|\le |E^0|}{\text{sup}}{\int }_{2^{-8}{ϵ}^2}^{2^{1/2}ϵ}H\left(t/c_3,{\mathcal{F}}_{ij}\right)dt\le {\left(2|E^0|\right)}^{1/2}ϵ\text{log }\left(2^{1/2}/c_3\right)\le c_4{n}^{1/2}{ϵ}^2.$$ (27)

for some constants c₃, c₄ > 0, say c₃ = 10, $c_4=\frac{{(2/3)}^{5/2}}{512}$. By Assumption B, $C_{\text{min}}\left({\mathbf{\Omega }}^0\right)\ge {ϵ}_{n,p,|E^0|}$ implies (27), provided that $2d_0^{-1}>{\left(2c_0\right)}^{1/2}{c}_4^{-1}\text{log }\left(2^{1/2}/c_3\right)$. Using the facts about binomial coefficients: ${\sum }_{j=1}^{|E^0|-k}\left(\begin{array}{c}p(p-1)-|E^0|\\ j\end{array}\right)\le {\left(p(p-1)-|E^0|+1\right)}^{|E^0|-k}$ and $\left(\begin{array}{c}|E^0|\\ k\end{array}\right)\le {|E^0|}^k$, we have, by Theorem 1 of [26], that for a constant c₂ > 0, say $c_2=\frac{4}{27}\frac{1}{1926}$,

$$I\le \sum _{k=0}^{|E^0|-1}\sum _{j=1}^{|E^0|-k}{P}^{*}\left(\underset{{\Omega }_E\tau \in B_{kj}}{\text{sup}}\left(L\left({\Omega }_{E^{\tau }}\right)-L\left({\widehat{\mathbf{\Omega }}}_O\right)\right)\ge 0\right)$$

$$\le 4\sum _{k=0}^{|E^0|-1}\left(\begin{array}{c}|E^0|\\ k\end{array}\right)\text{ exp }\left(-c_{2}n\left(d_1{C}_{\text{min}}\left({\mathbf{\Omega }}^0\right)-d_{3}q{\tau }^{d_2}\right)\right)\sum _{j=1}^{|E^0|-k}\left(\begin{array}{c}p(p-1)-|E^0|\\ j\end{array}\right)$$

$$\le 4\sum _{i=1}^{|E^0|}\text{exp}\left(-i\left(\left(c_2{d}_1/2\right)C_{\text{min}}\left({\mathbf{\Omega }}^0\right)-\text{log }\left(p(p-1)-|E^0|+1\right)-\text{log }|E^0|\right)\right)$$

$$\le R\left(\text{exp}\left(-\left(c_2{d}_1/2\right)C_{\text{min}}\left({\mathbf{\Omega }}^0\right)-\text{log }\left(p(p-1)-|E^0|+1\right)-\text{log }|E^0|\right)\right),$$

provided that τ ≤ C_min(Ω⁰)M₁/4p, where R(x) = x/(1 − x). Moreover, since I ≤ 1 and log(p(p − 1) − |E⁰| + 1) + log|E⁰|) ≤ 2log((p(p − 1) + 1)/2),

$$\begin{gathered} I\le 5\text{exp}\left(-\left(c_2{d}_1/2\right)n{C}_{\text{min}}\left({\mathbf{\Omega }}^0\right)+2\text{log }((p(p-1)+1)/2)\right) \\ \le \text{exp}\left(-c_{2}n{C}_{\text{min}}+2\text{log }(p(p-1)+1)+3\right). \end{gathered}$$

Under Assumption B, $P\left(\widehat{G}\ne G^0\right)\to 0$ as n, p, |E⁰| → ∞. For parameter estimation,

$$\begin{gathered} E{h}^2\left(\widehat{\mathbf{\Omega }},{\mathbf{\Omega }}^0\right)\le E\left[h^2\left({\widehat{\mathbf{\Omega }}}_O,{\mathbf{\Omega }}^0\right)I\left(\widehat{\mathbf{\Omega }}={\widehat{\mathbf{\Omega }}}_O\right)\right]+P\left(\widehat{\mathbf{\Omega }}\ne {\widehat{\mathbf{\Omega }}}_O\right) \\ \le (1+o(1)){Eh}^2\left({\widehat{\mathbf{\Omega }}}_O,{\mathbf{\Omega }}^0\right), \end{gathered}$$

then $\frac{E{h}^2\left(\mathbf{\Omega },{\mathbf{\Omega }}^0\right)}{E{h}^2\left({\mathbf{\Omega }}_O,{\mathbf{\Omega }}^0\right)}\to 1$ as n, p, |E⁰| → ∞.

This completes the proof. □