Differential Analysis of Directed Networks

Abstract

We developed a novel statistical method to identify structural differences between networks characterized by structural equation models. We propose to reparameterize the model to separate the differential structures from common structures, and then design an algorithm with calibration and construction stages to identify these differential structures. The calibration stage serves to obtain consistent prediction by building the ℓ₂ regularized regression of each endogenous variables against pre-screened exogenous variables, correcting for potential endogeneity issue. The construction stage consistently selects and estimates both common and differential effects by undertaking ℓ₁ regularized regression of each endogenous variable against the predicts of other endogenous variables as well as its anchoring exogenous variables. Our method allows easy parallel computation at each stage. Theoretical results are obtained to establish non-asymptotic error bounds of predictions and estimates at both stages, as well as the consistency of identified common and differential effects. Our studies on synthetic data demonstrated that our proposed method performed much better than independently constructing the networks. A real data set is analyzed to illustrate the applicability of our method.

1. INTRODUCTION

It is of great importance and interest to detect sparse structural differences or differential structures between two cognate networks. For instance, the gene regulatory networks of diseased and healthy individuals may differ slightly from each other [West et al., 2012], and identifying the subtle difference between them helps design specific drugs. Social networks evolve over times, and monitoring their abrupt changes may serve as surveillance to economic stability or disease epidemics [Pianese et al., 2013, Berkman and Syme, 1979]. However, addressing such practical problems demands differential analysis of large networks, calling for development of efficient statistical method to infer and compare complex structures from high dimensional data. In this paper, we focus on differential analysis of directed acyclic or even cyclic networks which can be described by structural equation models (SEMs).

Many efforts have been made towards construction of a single network via SEM. For example, both Xiong et al. [2004] and Liu et al. [2008] employed a genetic algorithm to search for the best SEM using different information criteria. Most recently, Ni et al. [2017, 2018] employed a hierarchical Bayes approach to construct the SEM based networks. However, these approaches were designed for small or medium scale networks. For large-scale networks that the number of endogenous variables $p$ exceeds the sample size $n$, Cai et al. [2013] proposed a regularization approach to fit a sparse model. Because this method suffers from incapability of parallel computation, it may not be feasible for large networks. Logsdon and Mezey [2010] proposed another penalization approach to fit the model in a node-wise fashion which alleviates the computational burden. Most recently, Lin et al. [2015], Zhu [2018], and Chen et al. [2017] each proposed a two-stage approach to construct SEMs, with different algorithms designed at different stages. As shown by Chen et al. [2017], such a two-stage approach can have superior performance compared to other methods.

To the best of our knowledge, no algorithm has been proposed to conduct differential analysis of directed networks characterized by SEM. While a naive approach would separately construct each individual network and identify common and differential structures, this approach fails to take advantage of the commonality as well as sparse differential structures of the paired networks, leading to higher false positive rate or lower power. In this light, we introduce a novel statistical method, specially in the directed network regime, to conduct differential analysis of two networks via appropriate reparameterization of the corresponding models. There are two major features of our method. Firstly, we jointly model the commonality and difference between two networks explicitly. This helps us to gain dramatic performance improvements over the naive construction method. Secondly, benefiting from the flexible framework of SEMs, we are able to conduct differential analysis of directed networks. Most importantly, our method allow for both acyclic and cyclic networks. Compared to the other methods, directionality and allowing for cyclicity are crucial for many network studies, especially in constructing gene regulatory networks. As far as we know, our method is the first work on differential analysis of directed networks that enjoys the two promising features.

The rest of this paper is organized as follows. We first introduce the model and its identifiability condition in Section 2.1 and Section 2.2, respectively. Then, we present our proposed method of Reparameterization-based Differential analysis of directed Networks, termed as ReDNet, in Section 2.3. The theoretical justification of the proposed method is described in Section 2.4. Section 3 includes our studies on synthetic data showing the superior performance of our method, as well as an analysis of the Genotype-Tissue Expression (GTEx) data sets. We conclude our paper with brief discussion in Section 4.

2. METHODS

Here we first introduce the model and its identification condition, and then describe our proposed ReDNet method for identifying common and differential structures between two directed networks, followed with its theoretical justification.

2.1. THE MODEL

We consider two networks, each describing the dependencies among a common set of variables or nodes in a unique population. For each node $i\in \{1,2,\dots ,p\}$ in network $k\in \{1,2\}$, its regulation structure can be represented by the following equation,

$$\underset{\text{node}i}{\underbrace{{\mathbf{Y}}_i^{\left(k\right)}}}=\underset{\text{regulation}\text{by}\text{others}}{\underbrace{{\mathbf{Y}}_{-i}^{\left(k\right)}{\mathit{\gamma }}_i^{\left(k\right)}}}+\underset{\text{anchoring}\text{regulation}}{\underbrace{{\mathbf{X}}^{\left(k\right)}{\mathit{\varphi }}_i^{\left(k\right)}}}+\underset{\text{error}}{\underbrace{{\mathit{ϵ}}_i^{\left(k\right)}}},$$ (1)

where ${\mathbf{Y}}_i^{\left(k\right)}$ is the $i$-th column of ${\mathbf{Y}}^{\left(k\right)}$ and ${\mathbf{Y}}_{-i}^{\left(k\right)}$ is the submatrix of ${\mathbf{Y}}^{\left(k\right)}$ by excluding ${\mathbf{Y}}_i^{\left(k\right)}$, with ${\mathbf{Y}}^{\left(k\right)}$ a $n^{\left(k\right)}\times p$ matrix. ${\mathbf{X}}^{\left(k\right)}$ is a $n^{\left(k\right)}\times q$ matrix with each column standardized to have ${\ell }_2$ norm $\sqrt{n^{\left(k\right)}}$. The vectors ${\mathit{\gamma }}_i^{\left(k\right)}$ and ${\mathit{\varphi }}_i^{\left(k\right)}$ encode the inter-nodes and anchoring regulatory effects, respectively. The index set of non-zeros of ${\mathit{\varphi }}_i^{\left(k\right)}$ is known and denoted by ${𝒜}_i^{\left(k\right)}$, in other words, ${𝒜}_i^{\left(k\right)}=\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}({\varphi }_i^{\left(k\right)})$. The support set ${𝒜}_i^{\left(k\right)}$ indexes the direct causal effects for the $i$-th node, and can be prespecified based on the domain knowledge. However, the size of nonzero effect ${\mathit{\varphi }}_i^{\left(k\right)}$ is unknown and can be estimated. Further property of ${𝒜}_i^{\left(k\right)}$ will be discussed in Section 2.2. All elements of the error term are independently distributed following a normal distribution with mean zero and standard deviation ${\sigma }_i^{\left(k\right)}$. We assume that the matrix ${\mathbf{X}}^{\left(k\right)}$ and the error term ${\mathit{ϵ}}_i^{\left(k\right)}$ are independent of each other. However ${\mathbf{Y}}_{-i}^{\left(k\right)}$ and ${\mathit{ϵ}}_i^{\left(k\right)}$ may correlate with each other. ${\mathbf{Y}}^{\left(k\right)}$ and ${\mathbf{X}}^{\left(k\right)}$ include observed endogenous variables and exogenous variables, respectively.

By combining the $p$ linear equations in (1), we can rewrite the two sets of linear equations in a systematic fashion as two structural equation models below,

$$\left\{\begin{array}{l}{\mathbf{Y}}^{\left(1\right)}={\mathbf{Y}}^{\left(1\right)}{\mathbf{\Gamma }}^{\left(1\right)}+{\mathbf{X}}^{\left(1\right)}{\mathbf{\Phi }}^{\left(1\right)}+{ℰ}^{\left(1\right)},\\ {\mathbf{Y}}^{\left(2\right)}={\mathbf{Y}}^{\left(2\right)}{\mathbf{\Gamma }}^{\left(2\right)}+{\mathbf{X}}^{\left(2\right)}{\mathbf{\Phi }}^{\left(2\right)}+{ℰ}^{\left(2\right)},\end{array}\right.$$ (2)

where each matrix ${\mathbf{\Gamma }}^{\left(k\right)}$ is $p\times p$ with zero diagonal elements and represents the inter-nodes regulatory effects in the corresponding network. Specifically, excluding the $i$-th element (which is zero) from the $i$-th column of ${\mathbf{\Gamma }}^{\left(k\right)}$ leads to ${\mathit{\gamma }}_i^{\left(k\right)}$. The $q\times p$ matrix ${\mathbf{\Phi }}^{\left(k\right)}$ contains the anchoring regulatory effects and its $i$-th column is ${\mathit{\varphi }}_i^{\left(k\right)}$. Each error term ${ℰ}^{\left(k\right)}$ is $n^{\left(k\right)}\times p$ and has the error term ${\mathit{ϵ}}_i^{\left(k\right)}$ as its $i$-th column.

Figure 1 gives an illustrative example of networks with three nodes and one anchoring regulation per node for the structural equations in (2). For example, with anchoring regulation on node $Y_1,X_1$ has a direct effect on node $Y_1$ but indirect effects on node $Y_2$ and $Y_3$ via $Y_1$.

Figure 1:

An Illustrative Example of Differential Network Between Two Directed Networks. The error term for each node is not shown for simplicity.

For each network $k$, its full model in (2) can be further transformed into the reduced form as follows,

$${\mathbf{Y}}^{\left(k\right)}={\mathbf{X}}^{\left(k\right)}{\mathit{\pi }}^{\left(k\right)}+{\mathit{\xi }}^{\left(k\right)},$$ (3)

where the $q\times p$ matrix ${\mathit{\pi }}^{\left(k\right)}={\mathbf{\Phi }}^{\left(k\right)}{(\mathbf{I}-{\mathbf{\Gamma }}^{\left(k\right)})}^{-1}$ and the transformed error term ${\mathit{\xi }}^{\left(k\right)}={ℰ}^{\left(k\right)}{(\mathbf{I}-{\mathbf{\Gamma }}^{\left(k\right)})}^{-1}$. The reduced model (3) reveals variables observed in ${\mathbf{X}}^{\left(k\right)}$ as instrumental variables which will be used later to correct for the endogeneity issue. Otherwise, directly applying any regularization based regression to equation (1) will result in non-consistent or suboptimal estimation of model parameters [Fan and Liao, 2014, Chen et al., 2017, Lin et al., 2015, Zhu, 2018].

2.2. THE MODEL IDENTIFIABILITY

Here we introduce an identifiability assumption which helps to infer an identifiable system (2) from available data. We assume that each endogenous variable is directly regulated by a unique set of exogenous variables as long as it regulates other endogenous variables. That is, any regulatory node needs at least one anchoring exogenous variable to distinguish the corresponding regulatory effects from association. Explicitly let ${ℳ}_{i0}^{\left(k\right)}$ denote the index set of endogenous variables which either directly or indirectly regulate the $i$-th endogenous variable in the $k$-th network. Thus, ${𝒜}_i^{\left(k\right)}\subseteq {ℳ}_{i0}^{\left(k\right)}$. The model identification condition can be stated in the below.

Assumption 1.

For any $i=1,\cdots ,p$, ${𝒜}_i^{\left(k\right)}\ne \mathrm{\varnothing }$ if there exists $j$ such that $i\in {ℳ}_{j0}^{\left(k\right)}$. Furthermore, ${𝒜}_i^{\left(k\right)}\cap {𝒜}_j^{\left(k\right)}=\mathrm{\varnothing }$ as long as $i\ne j$.

This assumption is slightly less restrictive than the one employed by Chen et al. [2017], and is a sufficient condition for model identifiability as it satisfies the rank condition in Schmidt [1976]. It can be further relaxed to allow nonempty ${𝒜}_i^{\left(k\right)}\cap {𝒜}_j^{\left(k\right)}$ as long as each regulatory node has its own unique anchoring exogenous variables.

The above identifiability assumption not only identifies ${\mathit{\gamma }}_i^{\left(k\right)}$ in model (1) from ${\mathit{\pi }}^{\left(k\right)}$ in model (3) but also helps reveal regulatory directionality of the networks. As illustrated in Figure 2, we can not recover the directionality between nodes $Y_1$ and $Y_2$ without the extra information provided by the direct causal factors $X_1$ and $X_2$ because all four sub-networks consisting of $Y_1$ and $Y_2$ (without $X_1$ and $X_2$) will be Markov equivalent. The known set ${𝒜}_j^{\left(k\right)}$ serves as external prior knowledge which helps recover the directionality. In our two-stage construction of the differential network, the additional anchors $X_1$ and $X_2$ serve as instrumental variables in the calibration stage, since both $X_1$ and $X_2$ are independent of the error terms. The present direct causal effects from ${\mathbf{X}}^{\left(k\right)}$ together with Assumption 1 differentiates our approach from the classical graphical models [Meinshausen and Bühlmann, 2006, Yuan and Lin, 2007] or the PC algorithm approaches [Spirtes et al., 2000, Kalisch and Bühlmann, 2007], since those methods either cannot recover edge directions or do not allow for cyclic structures due to lack of additional direct causal effects from ${\mathbf{X}}^{\left(k\right)}$.

Figure 2:

An Illustrative Example of Networks Which Are Not Markov Equivalent. However, without X₁ and X₂, sub-networks consisting of only node Y₁ and Y₂ will be Markov equivalent.

2.3. TWO-STAGE DIFFERENTIAL ANALYSIS OF NETWORKS

Here we intend to develop a regularized version of the two-stage least squares. We first screen for exogenous variables and conduct ${\ell }_2$ regularized regression of each endogenous variable against screened exogenous variables to obtain its good prediction which helps address the endogeneity issue in the following stage. At the second stage, we reparametrize the model to explicitly model the common and differential regulatory effects and identify them via the adaptive lasso method.

2.3.1. The Calibration Stage

To address the endogeneity issue, we aim for good prediction of each endogenous variable following the reduced model in (3). However, in the high-dimensional setting, the dimension $q$ of ${\mathbf{X}}^{\left(k\right)}$ can be much larger than the sample size $n^{\left(k\right)}$, and any direct prediction with all exogenous variables may not produce consistent prediction. Note that both Lin et al. [2015] and Zhu [2018] proposed to conduct variable selection with lasso or its variants and predict with selected exogenous variables. We here instead propose to first screen for exogenous variables with ISIS [Fan and Lv, 2008], and then apply ridge regression to predict the endogenous variables with screened exogenous variables. While variable screening is more robust and provides higher coverage of true variables than variable selection, its combination with ridge regression puts less computational burden. Furthermore, as shown by Chen et al. [2017], ridge regression performs well in predicting the endogenous variables.

Let ${ℳ}_i^{\left(k\right)}$ denotes the selected index set for $i$-th node in $k$-th network from the variable screening which reduces the dimension from $q$ to $d=|{ℳ}_i^{\left(k\right)}|$. The Sure Independence Screening Property in Fan and Lv [2008] can be directly applied in our case to guarantee that ${ℳ}_i^{\left(k\right)}$ covers the true set ${ℳ}_{i0}^{\left(k\right)}$ with a large probability.

Assumption 2.

$n^{\left(1\right)}$ and $n^{\left(2\right)}$ are at the same order, i.e., $n_{\text{min}}=\mathrm{m}\mathrm{i}\mathrm{n}\left(n^{\left(1\right)},n^{\left(2\right)}\right)\asymp n^{\left(1\right)}\asymp n^{\left(2\right)}$, and $p\asymp q$.

Theorem 1.

Assuming Conditions 1–4 in the supplemental materials which restrict positive $\tilde{\tau }$ and $\tilde{\kappa }$, under Assumption 2, there exists some $\theta \in (0,1-2\tilde{\kappa }-\tilde{\tau })$ such that, when $d=|{ℳ}_i^{\left(k\right)}|=O({\left(n_{\text{min}}\right)}^{1-\theta })$, we have, for some constant $C>0$,

$$\mathbb{P}({ℳ}_{i0}^{\left(k\right)}\subseteq {ℳ}_i^{\left(k\right)})=1-𝒪\left(\exp \left\{-\frac{C{(n^{\left(k\right)})}^{1-2\tilde{\kappa }}}{\log \left(n^{\left(k\right)}\right)}\right\}\right).$$

Hereafter we assume that ${ℳ}_i^{\left(k\right)}$ successfully covers the true set ${ℳ}_{i0}^{\left(k\right)}$ for convenience of stating the following assumptions and theorems. That is, the probability of successful screening is not incorporated into our assumptions or theorems in the below.

For node $i$ in network $k$, let ${\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ denotes the submatrix of ${\mathbf{X}}^{\left(k\right)}$ with prescreened columns which are indexed by ${ℳ}_i^{\left(k\right)}$. With ${\mathit{\pi }}_i^{\left(k\right)}$ denoting the $i$-th column of ${\mathit{\pi }}^{\left(k\right)}$, the subvector of ${\mathit{\pi }}_i^{\left(k\right)}$ indexed by ${ℳ}_i^{\left(k\right)}$ will be simply denoted by ${\mathit{\pi }}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ without confusion. Such simplified notations will apply to other vectors and matrices in the rest of this paper.

With $d$ pre-screened exogenous variables, we can apply ridge regression to the model

$${\mathbf{Y}}_i^{\left(k\right)}={\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}{\mathit{\pi }}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}+{\mathit{\xi }}_i^{\left(k\right)},$$ (4)

to obtain the estimates ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ of ${\mathit{\pi }}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$, and predict ${\mathbf{Y}}_i^{\left(k\right)}$ with ${\stackrel{\mathbf{ˆ}}{\mathbf{Y}}}_i^{\left(k\right)}={\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}{\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$.

2.3.2. The Construction Stage

With known ${𝒜}_i^{\left(k\right)}$, we can rewrite model (1) as,

$${\mathbf{Y}}_i^{\left(k\right)}={\mathbf{Y}}_{-i}^{\left(k\right)}{\gamma }_i^{\left(k\right)}+{\mathbf{X}}_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)}{\varphi }_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)}+{ϵ}_i^{\left(k\right)}.$$ (5)

Before we use the predicted ${\mathbf{Y}}^{\left(k\right)}$ to identify both common and differential regulatory effects across the two networks, we first reparametrize the model so as to define differential regulatory effects explicitly,

$${\mathit{\beta }}_i^{-}=\frac{{\mathit{\gamma }}_i^{\left(1\right)}-{\mathit{\gamma }}_i^{\left(2\right)}}{2},{\mathit{\beta }}_i^{+}=\frac{{\mathit{\gamma }}_i^{\left(1\right)}+{\mathit{\gamma }}_i^{\left(2\right)}}{2}.$$ (6)

Here ${\mathit{\beta }}_i^{-}$ and ${\mathit{\beta }}_i^{+}$ represent the differential and average regulatory effects between the two networks, respectively. We need compare ${\mathit{\beta }}_i^{+}$ with ${\mathit{\beta }}_i^{-}$ to identify the common regulatory effects, that is, effects of all regulations with nonzero values in ${\mathit{\beta }}_i^{+}$ but zero values in ${\mathit{\beta }}_i^{-}$.

Note that other differential analysis of networks may suggest a different reparametrization to identify common and differential regulatory effects. For example, in a typical case-control study, we may expect few structures in the case network mutated from the control network. While we are interested in identifying differential structures in the case network, we may be also interested in identifying baseline network structures in the control network. Therefore we may reparametrize the model with the regulatory effects in the control network, as well as the differential regulatory effects defined as the difference of regulatory effects between case and control networks. We want to point out that the method described here still applies and we can also derive similar theoretical results as follows.

Following the reparametrization in (6), we can rewrite model (5) as follows,

$$\begin{array}{l}\left(\genfrac{}{}{0pt}{}{{\mathbf{Y}}_i^{\left(1\right)}}{{\mathbf{Y}}_i^{\left(2\right)}}\right)=\left(\begin{array}{cc}{\mathbf{Y}}_{-i}^{\left(1\right)}& {\mathbf{Y}}_{-i}^{\left(1\right)}\\ {\mathbf{Y}}_{-i}^{\left(2\right)}& -{\mathbf{Y}}_{-i}^{\left(2\right)}\end{array}\right)\left(\genfrac{}{}{0pt}{}{{\mathit{\beta }}_i^{+}}{{\mathit{\beta }}_i^{-}}\right)+\\ \left(\begin{array}{cc}{\mathbf{X}}_{{𝒜}_i^{\left(1\right)}}^{\left(1\right)}& 0\\ 0& {\mathbf{X}}_{{𝒜}_i^{\left(2\right)}}^{\left(2\right)}\end{array}\right)\left(\genfrac{}{}{0pt}{}{{\mathit{\varphi }}_{{𝒜}_i^{\left(1\right)}}^{\left(1\right)}}{{\mathit{\varphi }}_{{𝒜}_i^{\left(2\right)}}^{\left(2\right)}}\right)+\left(\genfrac{}{}{0pt}{}{{\mathit{ϵ}}_i^{\left(1\right)}}{{\mathit{ϵ}}_i^{\left(2\right)}}\right).\end{array}$$ (7)

Denote

$$\begin{array}{ll}{\mathbf{Y}}_i=\left(\genfrac{}{}{0pt}{}{{\mathbf{Y}}_i^{\left(1\right)}}{{\mathbf{Y}}_i^{\left(2\right)}}\right),& {\mathbf{Z}}_{-i}=\left(\begin{array}{cc}{\mathbf{Y}}_{-i}^{\left(1\right)}& {\mathbf{Y}}_{-i}^{\left(1\right)}\\ {\mathbf{Y}}_{-i}^{\left(2\right)}& -{\mathbf{Y}}_{-i}^{\left(2\right)}\end{array}\right),\\ {\mathit{\beta }}_i=\left(\genfrac{}{}{0pt}{}{{\mathit{\beta }}_i^{+}}{{\mathit{\beta }}_i^{-}}\right),& {\mathit{ϵ}}_i=\left(\genfrac{}{}{0pt}{}{{\mathit{ϵ}}_i^{\left(1\right)}}{{\mathit{ϵ}}_i^{\left(2\right)}}\right).\end{array}$$

Further define the projection matrix for each network,

$${\mathbf{H}}_i^{\left(k\right)}=I_{n^{\left(k\right)}}-{\mathbf{X}}_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)}{\left({\mathbf{X}}_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)T}{\mathbf{X}}_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)}\right)}^{-1}{\mathbf{X}}_{{𝒜}_i^{\left(k\right)}}^{\left(k\right)T}.$$

Applying the projection matrix ${\mathbf{H}}_i=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{{\mathbf{H}}_i^{\left(1\right)},{\mathbf{H}}_i^{\left(2\right)}\}$ to both sides of model (7), we can remove the exogenous variables from the model and obtain,

$${\mathbf{H}}_i{\mathbf{Y}}_i={\mathbf{H}}_i{\mathbf{Z}}_{-i}{\mathit{\beta }}_i+{\mathbf{H}}_i{\mathit{ϵ}}_i.$$ (8)

Algorithm 1.

Reparameterization-Based Differential Analysis of Network (ReDNet)

Input: For $k\in \{1,2\}$, ${\mathbf{Y}}^{\left(k\right)},{\mathbf{X}}^{\left(k\right)}$, index set ${𝒜}_i^{\left(k\right)}$ for each $i\in \{1,2,...,p\}$. Set $d=O\left(n_{\mathrm{m}\mathrm{i}\mathrm{n}}^{1-\theta }\right)$.

for $i\to 1$ to $p$ do

Stage 1.a. Screen for a submatrix ${\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ of ${\mathbf{X}}^{\left(k\right)}$ for ${\mathbf{Y}}_i^{\left(k\right)}$ versus ${\mathbf{X}}^{\left(k\right)}$ and set ${\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}={\mathbf{X}}^{\left(k\right)}$ if $q\le n^{\left(k\right)}$

Stage 1.b. Apply ridge regression to regress ${\mathbf{Y}}_i^{\left(k\right)}$ against ${\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ to obtain prediction ${\widehat{\mathbf{Y}}}_i^{\left(k\right)}$.

end for

for $\mathit{i}\to 1$ to $p$ do

Stage 2. Apply adaptive lasso to regress ${\mathbf{H}}_i{\mathbf{Y}}_i$ against ${\mathbf{H}}_i{\stackrel{\mathbf{ˆ}}{\mathbf{Z}}}_{-i}$ to obtain coefficients estimate ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i$.

end for

Output: The common and differential regulatory effects in ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_1,...,{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_p$.

To address the endogeneity issue, we predict ${\mathbf{Z}}_{-i}$ by replacing its component ${\mathbf{Y}}_{-i}^{\left(k\right)}$ with the predicted value ${\stackrel{\mathbf{ˆ}}{\mathbf{Y}}}_{-i}^{\left(k\right)}$ from the previous stage, and then regressing ${\mathbf{H}}_i{\mathbf{Y}}_i$ against ${\mathbf{H}}_i{\stackrel{\mathbf{ˆ}}{\mathbf{Z}}}_{-i}$ with the adaptive lasso to consistently estimate ${\mathit{\beta }}_i$. That is, an optimal ${\mathit{\beta }}_i$ can be obtained as,

$${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i=\underset{{\mathit{\beta }}_i}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\frac{1}{n}{‖{\mathbf{H}}_i{\mathbf{Y}}_i-{\mathbf{H}}_i{\stackrel{\mathbf{ˆ}}{\mathbf{Z}}}_{-i}{\mathit{\beta }}_i‖}_2^2+{\lambda }_i{\mathit{\omega }}_i^T{\left|{\mathit{\beta }}_i\right|}_1\right\},$$

where ${\left|{\mathit{\beta }}_i\right|}_1$ is a vector taking elementwise absolute values of ${\mathit{\beta }}_i$, ${\mathit{\omega }}_i$ is the adaptive weights whose components are inversely proportional to the components of an initial estimator of ${\mathit{\beta }}_i$, and ${\lambda }_i$ is the adaptive tuning parameter.

The two-stages algorithm is summarized in Algorithm 1. With the estimator ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i$ from the second stage, we can accordingly obtain estimators ${\stackrel{\mathit{ˆ}}{\mathit{\gamma }}}_i^{\left(1\right)}={\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i^{+}+{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i^{-}$ and ${\stackrel{\mathit{ˆ}}{\mathit{\gamma }}}_i^{\left(2\right)}={\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i^{+}-{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i^{-}$.

2.4. THEORETICAL ANALYSIS

As shown in Theorem 1, a screening method like ISIS [Fan and Lv, 2008] can identify ${ℳ}_i^{\left(k\right)}$ with size $d=O\left(n_{\mathrm{m}\mathrm{i}\mathrm{n}}^{1-\theta }\right)$ which covers the true set ${ℳ}_{i0}^{\left(k\right)}$ with a sufficiently large probability. For the sake of simplicity and without loss of generality, in the following we assume ${ℳ}_{i0}^{\left(k\right)}\subseteq {ℳ}_i^{\left(k\right)}$.

We first investigate the consistency of predictions from the first stage. The consistency properties will be characterized by prespecified sequences $f^{\left(k\right)}=o\left(n^{\left(k\right)}\right)$ but $f^{\left(k\right)}\to \mathrm{\infty }$ as $n^{\left(k\right)}\to \mathrm{\infty }$. We also denote $f_{\text{max}}=f^{\left(1\right)}\vee f^{\left(2\right)}$, i.e., $\mathrm{m}\mathrm{a}\mathrm{x}\{f^{\left(1\right)},f^{\left(2\right)}\}$.

The following assumption is required for the consistency properties.

Assumption 3.

For each network $k$, the singular values of $\mathbf{I}-{\mathbf{\Gamma }}^{\left(k\right)}$ are positively bounded from below, and there exist some positive constants $c_1^{\left(k\right)}$ and $c_2^{\left(k\right)}$ such that, for each node $i$, ${\mathrm{m}\mathrm{a}\mathrm{x}}_{{\Vert \delta \Vert }_2=1}{\left(n^{\left(k\right)}\right)}^{-1/2}{\Vert {\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}\delta \Vert }_2\le c_1^{\left(k\right)}$ and ${\mathrm{m}\mathrm{i}\mathrm{n}}_{\Vert \delta {\Vert }_2=1}{\left(n^{\left(k\right)}\right)}^{-1/2}{\Vert {\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}\delta \Vert }_2\ge c_2^{\left(k\right)}$. Furthermore, the ridge parameter ${\lambda }_i^{\left(k\right)}=o\left(n_{\text{min}}\right)$.

For the ease of exposition, we will omit the subscript ${ℳ}_i^{\left(k\right)}$ from ${\mathbf{X}}_{{ℳ}_i^{\left(k\right)}}^{\left(k\right)}$ henceforth, and accordingly use ${\mathit{\pi }}_i^{\left(k\right)}$ and ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_i^{\left(k\right)}$ which include the zero components of excluded predictors.

Denote $\mathbf{X}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{{\mathbf{X}}^{\left(1\right)},{\mathbf{X}}^{\left(2\right)}\}$, and

$$\mathbf{Z}=\left(\begin{array}{cc}{\mathbf{Y}}^{\left(1\right)}& {\mathbf{Y}}^{\left(1\right)}\\ {\mathbf{Y}}^{\left(2\right)}& -{\mathbf{Y}}^{\left(2\right)}\end{array}\right),\mathbf{\Pi }=\left(\begin{array}{cc}{\mathit{\pi }}^{\left(1\right)}& {\mathit{\pi }}^{\left(1\right)}\\ {\mathit{\pi }}^{\left(2\right)}& -{\mathit{\pi }}^{\left(2\right)}\end{array}\right).$$

We use ${\mathbf{\Pi }}_j$ to denote the $j$-th column of the matrix $\mathbf{\Pi }$ and ${\mathit{\pi }}_j^{\left(k\right)}$ to denote the $j$-th column of the matrix ${\mathit{\pi }}^{\left(k\right)}$. We also use $\stackrel{\mathbf{ˆ}}{\mathbf{Z}}$ and $\stackrel{\mathbf{ˆ}}{\mathbf{\Pi }}$ to denote the prediction of $\mathbf{Z}$ and estimate of $\mathbf{\Pi }$, respectively. Note that, with the ridge parameter ${\lambda }_i^{\left(k\right)}$ for the ridge regression taken on node $i$ in network $k$, we have $r_i^{\left(k\right)}={({\lambda }_i^{\left(k\right)})}^2{\Vert {\mathit{\pi }}_i^{\left(k\right)}\Vert }_2^2/n^{\left(k\right)}$ and hence define $r_{\text{max}}=\underset{1\le i\le p}{\mathrm{m}\mathrm{a}\mathrm{x}}[r_i^{\left(1\right)}\vee r_i^{\left(2\right)}]$. Then the estimation and prediction losses at the first stage can be summarized in the following theorem.

Theorem 2.

Under Assumptions 1–3, for each $j\in \{1,2,\dots ,2p\}$, there will exist some constant $C_1$ and $C_2$ such that, with probability at least $1-e^{-f^{\left(1\right)}}-e^{-f^{\left(2\right)}}$,

1. ${\Vert {\stackrel{\mathbf{ˆ}}{\mathbf{\Pi }}}_j-{\mathbf{\Pi }}_j\Vert }_2^2\le C_1\left(d\vee r_{\mathrm{m}\mathrm{a}\mathrm{x}}\vee f_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)/n_{\mathrm{m}\mathrm{i}\mathrm{n}}$;

2. ${\Vert \mathbf{X}({\stackrel{\mathbf{ˆ}}{\mathbf{\Pi }}}_j-{\mathbf{\Pi }}_j)\Vert }_2^2\le C_2\left(d\vee r_{\mathrm{m}\mathrm{a}\mathrm{x}}\vee f_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)$.

The proof is detailed in the supplemental materials.

Note that these two sets of losses can be controlled by the same upper bounds across the two networks with probability at least $1-e^{-f^{(1)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}-e^{-f^{(2)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}$. Therefore, $f^{(k)}$ can be selected such that $f^{(k)}-log(p)\to \mathrm{\infty }$, which will provide a probability approaching one to have the network-wide losses approaching zero.

Furthermore, the dimension $p$ can be divergent up to an exponential order, say $p=e^{n_{\text{min}}^c}$ for some $c\in (0,1)$. We can set $f^{(1)}=f^{(2)}=n_{\mathrm{m}\mathrm{i}\mathrm{n}}^{(1+c)/2}$ and, apparently, $f^{(k)}=o\left(n_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)$ but $f^{(k)}-log(p)=n_{\mathrm{m}\mathrm{i}\mathrm{n}}^{(1+c)/2}-n_{\mathrm{m}\mathrm{i}\mathrm{n}}^c\to \mathrm{\infty }$.

Since the ridge parameter ${\lambda }_i^{(k)}=o\left(n_{\text{min}}\right),r_i^{(k)}={\Vert {\mathit{\pi }}_i^{(k)}\Vert }_2^2\times o\left(n_{\text{min}}\right)$. Therefore, when all ${\Vert {\mathit{\pi }}_i^{(k)}\Vert }_2$ are uniformly bounded, we have $r_{\text{max}}=o\left(n_{\text{min}}\right)$. Otherwise, the ridge parameter ${\lambda }_i^{(k)}$ should be adjusted accordingly to control both estimation and prediction losses.

Before we characterize the consistency of estimated regulatory effects on the second stage, we first introduce the following concept of restricted eigenvalue which is used to present an assumption.

Definition 2.1.

The restricted eigenvalue of a matrix $\mathbf{A}$ on an index set $𝒮$ is defined as

$${\varphi }_{re}(\mathbf{A},𝒮)=\underset{{‖{\delta }_{{𝒮}^c}‖}_1\le 3{‖{\delta }_{𝒮}‖}_1}{\mathrm{m}\mathrm{i}\mathrm{n}}\frac{\Vert \mathbf{A}\delta {\Vert }_2}{\sqrt{n}{‖{\delta }_{𝒮}‖}_2}.$$ (9)

For the $i$-th node, we use ${𝒮}_i$ to denote the non-zero indices of ${\mathit{\beta }}_i$, i.e., ${𝒮}_i=\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left({\mathit{\beta }}_i\right)$. Further denote

$${\mathbf{\Pi }}_{-i}=\left(\begin{array}{cc}{\mathit{\pi }}_{-i}^{(1)}& {\mathit{\pi }}_{-i}^{(1)}\\ {\mathit{\pi }}_{-i}^{(2)}& -{\mathit{\pi }}_{-i}^{(2)}\end{array}\right).$$

As inBickel et al. [2009], we impose the following restricted eigenvalue condition on the design matrix in (8).

Assumption 4.

There exists a constant ${\varphi }_0>0$ such that ${\varphi }_{\mathrm{r}\mathrm{e}}\left({\mathbf{H}}_i\mathbf{X}{\mathbf{\Pi }}_{-i},{𝒮}_i\right)\ge {\mathit{\varphi }}_0$. Furthermore, ${‖{\mathit{\omega }}_{{𝒮}_i}‖}_{\mathrm{\infty }}\le {‖{\mathit{\omega }}_{{𝒮}_i^c}‖}_{-\mathrm{\infty }}$.

Let $n=n^{(1)}+n^{(2)}$, $c_{\text{max}}=c_1^{(1)}\vee c_1^{(2)}$, and $\mathbf{B}=\left[{\mathit{\beta }}_1,{\mathit{\beta }}_2,\dots ,{\mathit{\beta }}_p\right]$. The matrix norms $\Vert \cdot {\Vert }_1$ and $\Vert \cdot {\Vert }_{\mathrm{\infty }}$ are the maximum of column and row sums of absolute values of the matrix, respectively. For a vector, we define $\Vert \cdot {\Vert }_{\mathrm{\infty }}$ and $\Vert \cdot {\Vert }_{-\mathrm{\infty }}$ to be the maximum and minimum absolute values of its components. Then, we can derive the following loss bounds for the estimation and prediction at the second stage on the basis of Theorem 2.

Theorem 3.

Suppose that, for node $i$, the adaptive lasso at the second stage takes the tuning parameter ${\lambda }_i\asymp {‖{\mathit{\omega }}_i‖}_{-\mathrm{\infty }}^{-1}\Vert \mathbf{B}{\Vert }_1\Vert \mathbf{\Pi }{\Vert }_1\sqrt{\left(d\vee r_{\text{max}}\vee f_{\text{max}}\right)\mathrm{l}\mathrm{o}\mathrm{g}(p)/n_{\mathit{min}}}$, and $\sqrt{\left(d\vee r_{\text{max}}\vee f_{\text{max}}\right)/n}+c_{\text{max}}{\Vert \mathbf{\Pi }\Vert }_1\le \sqrt{c_{\text{max}}^2\Vert \mathbf{\Pi }{\Vert }_1^2+{\varphi }_0^2/\left(64C_2\left|{𝒮}_i\right|\right)}$. Let $h_n=(\Vert \mathbf{B}{\Vert }_1^2\wedge 1)\times ((n\Vert \mathbf{\Pi }{\Vert }_1^2/d)\wedge \left(d\vee r_{\text{max}}\vee f_{\text{max}}\right))\mathrm{l}\mathrm{o}\mathrm{g}(p)$. Under Assumptions 1–4, there exist positive constants $C_3$ and $C_4$ such that, with probability at least $1-3e^{-C_3{h}_n+\mathrm{l}\mathrm{o}\mathrm{g}(4pq)}-e^{-f^{(1)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}-e^{-f^{(2)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}$,

1. Estimation Loss:

   $$\begin{array}{l}{\Vert {\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i-{\mathit{\beta }}_i\Vert }_1\le 8C_4\left|{𝒮}_i\right|\times \\ \frac{{‖{\mathit{\omega }}_{{𝒮}_i}‖}_{\mathrm{\infty }}\Vert \mathbf{B}{\Vert }_1\Vert \mathbf{\Pi }{\Vert }_1}{{\mathit{\varphi }}_0^2{‖{\mathit{\omega }}_i‖}_{-\mathrm{\infty }}}\sqrt{\frac{\left(d\vee r_{max}\vee f_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)\mathrm{l}\mathrm{o}\mathrm{g}(p)}{n_{\mathrm{m}\mathrm{i}\mathrm{n}}};}\end{array}$$
2. Prediction Loss:

   $$\begin{array}{l}\frac{1}{n}{\Vert {\mathbf{H}}_i{\stackrel{\mathbf{ˆ}}{\mathbf{Z}}}_{-i}({\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_i-{\mathit{\beta }}_i)\Vert }_2^2\le C_4^2\left|{𝒮}_i\right|\times \\ & \frac{{\Vert {\mathit{\omega }}_{{𝒮}_i}\Vert }_{\mathrm{\infty }}^2\Vert \mathbf{B}{\Vert }_1^2\Vert \mathbf{\Pi }{\Vert }_1^2}{{\varphi }_0^2{‖{\mathit{\omega }}_i‖}_{-\mathrm{\infty }}^2}\frac{\left(d\vee r_{max}\vee f_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)\mathrm{l}\mathrm{o}\mathrm{g}(p)}{n_{\mathrm{m}\mathrm{i}\mathrm{n}}}.\end{array}$$

The main idea of the proof is to take advantage of the commonly used restricted eigenvalue condition and irrepresentable condition for lasso-type estimator. However, the design matrix in our case includes predicted values instead of the original one, which complicates the proof. We claim that the restricted eigenvalue and irrepresentable condition still hold for the predicted design matrix as long as the estimation and prediction losses are well controlled at the calibration stage. The proof is detailed in the supplemental materials.

The available anchoring regulators as required by Assumption 1 implies that both $\Vert \mathbf{B}{\Vert }_1>0$ and $\Vert \mathbf{\Pi }{\Vert }_1>0$, so $h_n/\mathrm{l}\mathrm{o}\mathrm{g}(p)\to \mathrm{\infty }$. That is, these loss bounds hold with a sufficient large probability with properly chosen $f^{(k)}$.

The two sets of losses in Theorem 3 can also be controlled across the whole system by the same upper bounds defined by replacing $\left|{𝒮}_i\right|$ with $s_{\text{max}}=\underset{i}{\mathrm{m}\mathrm{a}\mathrm{x}}\left|{𝒮}_i\right|$, with probability at least $1-3e^{-C_3{h}_n+\log \left(4q\right)+2\mathrm{l}\mathrm{o}\mathrm{g}(p)}-e^{-f^{\left(1\right)}+2\mathrm{l}\mathrm{o}\mathrm{g}(p)}-e^{-f^{\left(2\right)}+2\mathrm{l}\mathrm{o}\mathrm{g}(p)}$. When both $p$ and $q$ are divergent up to an exponential order, say $p\asymp q\asymp e^{n_{\text{min}}^c}$ for some $c\in (0,1)$, we can set $f^{(1)}=f^{(2)}=n_{\text{min}}^{(1+c)/2}$ to guarantee the bounds at a sufficient large probability. However, the bounds are determined by $\left(d\vee r_{\text{max}}\vee f_{\text{max}}\right)\mathrm{l}\mathrm{o}\mathrm{g}(p)$ which is $o\left(n_{\text{min}}\right)$ only when $c<\mathrm{m}\mathrm{i}\mathrm{n}(1/3,\theta )$. Therefore, if $s_{\text{max}}$ also diverges up to $n_{\text{min}}^{\tilde{c}}$ with $\tilde{c}<\mathrm{m}\mathrm{i}\mathrm{n}(1/4,\theta /\mathrm{2,1}-\theta )$, the losses can be well controlled for $c<\mathrm{m}\mathrm{i}\mathrm{n}((1-4\tilde{c})/3,\theta -2\tilde{c})$.

Note that, with properly chosen $f^{(1)}$ and $f^{(2)}$, these losses are well controlled at $o\left(n_{\text{min}}\right)$, revealing the fact that we need to have sufficient observations for each network for consistent differential analysis of the two networks.

Let $W_i=diag\left\{{\mathit{\omega }}_i\right\}$. Denote ${ℐ}_i=\frac{1}{n}{\mathbf{\Pi }}_{-i}^T{\mathbf{X}}^T{\mathbf{H}}_i\mathbf{X}{\mathbf{\Pi }}_{-i}$ and ${\widehat{ℐ}}_i=\frac{1}{n}{\hat{\mathbf{n}}}_{-i}^T{\mathbf{X}}^T{\mathbf{H}}_i\mathbf{X}{\hat{\mathbf{\Pi }}}_{-i}$. Let ${ℐ}_{i,11}$ be a submatrix of ${ℐ}_i$ with rows and columns both indexed by ${𝒮}_i$, and ${ℐ}_{i,21}$ be a submatrix of ${ℐ}_i$ with rows and columns indexed by ${𝒮}_i^c$ and ${𝒮}_i$, respectively. ${\widehat{ℐ}}_{i,11}$ and ${\widehat{ℐ}}_{i,21}$ are similarly defined from ${\widehat{ℐ}}_i$. We further define the minimal signal strength $b_i=\underset{j\in {𝒮}_i}{\mathrm{m}\mathrm{a}\mathrm{x}}|{\mathit{\beta }}_{ij}|$ and ${\psi }_i={\Vert {ℐ}_{i,11}^{-1}{W}_{{𝒮}_i}\Vert }_{\mathrm{\infty }}$.

The following assumption, reminiscent of the adaptive irrepresentable condition in Huang et al. [2008], helps investigate the selection consistency of regulatory effects.

Assumption 5.

(Weighted Irrepresentable Condition) There exists a constant $\tau \in (0,1)$ such that ${\Vert W_{{𝒮}_i^c}^{-1}{ℐ}_{i,21}^{-1}{ℐ}_{i,11}{W}_{{𝒮}_i}\Vert }_{\mathrm{\infty }}<1-\tau$.

Theorem 4.

(Variable Selection Consistency) Denote ${\widehat{𝒮}}_i=\{j:{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_{ij}\ne 0\}$. Suppose that, for each node $i$, ${\widehat{ℐ}}_{i,11}$ is invertible, $b_i>{\lambda }_i{\psi }_i/(2-\tau )$, and $\sqrt{\left(d\vee r_{\text{max}}\vee f_{\text{max}}\right)/n}+c_{\text{max}}\Vert \mathbf{\Pi }{\Vert }_1\le \sqrt{c_{\text{max}}^2\Vert \mathbf{\Pi }{\Vert }_1^2+\mathrm{m}\mathrm{i}\mathrm{n}({\varphi }_0^2/64,\tau (4-\tau {)}^{-1}{‖{\mathit{\omega }}_i‖}_{-\mathrm{\infty }}/{\psi }_i)/\left(C_2\left|{𝒮}_i\right|\right)}$. Under Assumptions 1–5, there exists some constant $C_5>0$ such that ${\widehat{𝒮}}_i={𝒮}_i$ with probability at least $1-3e^{-C_5{h}_n+\mathrm{l}\mathrm{o}\mathrm{g}(4pq)}-e^{-f^{(1)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}-e^{-f^{(2)}+\mathrm{l}\mathrm{o}\mathrm{g}(p)}$.

This theorem implies that our proposed method can identify both common and differential regulatory effects between the two networks with a sufficiently large probability. On the other hand, the assumed weighted irrepresentable condition means that the true signal should not correlate too much with irrelevant predictors so as to conduct a successful differential analysis. The corresponding proof is detailed in the supplemental materials.

3. EXPERIMENTS

3.1. SYNTHETIC DATA EVALUATION

Here we report on experiments with synthetic data to show the superior performance of our method. We compare the method ReDNet to a naive differential analysis which employs the 2SPLS method proposed by [Chen et al., 2017] to construct each network separately. Note that the 2SPLS method is modified here by applying ISIS to screen exogenous variables before conducting ridge regression to predict endogenous variables, making the naive differential analysis comparable to ReDNet.

Synthetic data are generated from both acyclic and cyclic networks involving 1000 endogenous variables, with the sample size from 200 to 300. Each network includes a subnetwork of 50 endogenous variables, whose shared and differential structures will be investigated against its pair. On average, each endogenous variable has one regulatory effect in a sparse subnetwork, and three regulatory effects in a dense network. While each pair of subnetworks in comparison share many identical regulatory effects, they also share five regulatory effects but with opposite signs, and each network has five unique regulatory effects (so the total number of differential regulatory effects is 15). The nonzero regulatory effects were independently sampled from a uniform distribution over the range [−0.8,−0.3]∪[0.3,8]. While assuming each node is directly regulated by one exogenous variable, each exogenous variable was sampled from discrete values 0,1 and 2 with probabilities 0.25, 05 and 0.25, respectively. All of the noise terms were independently sampled from the normal distribution $N(0,{0.1}^2)$. We also conducted differential analysis between two networks with both ${\mathbf{X}}^{(1)}\ne {\mathbf{X}}^{(2)}$ and ${\mathbf{X}}^{(1)}={\mathbf{X}}^{(2)}$ as in practice the paired networks may or may not share identically valued exogenous variables.

We evaluate the the performance in terms of the false discovery rate (FDR), power and Matthews correlation co-efficient ($\mathrm{M}\mathrm{C}\mathrm{C}$) [Matthews, 1975]. Let $\mathrm{T}\mathrm{P},\mathrm{T}\mathrm{N},\mathrm{F}\mathrm{P}$ and $\mathrm{F}\mathrm{N}$ denote the numbers of true positives, true negatives, false positives, and false negatives, respectively. $\mathrm{M}\mathrm{C}\mathrm{C}$ is defined as,

$$\mathrm{M}\mathrm{C}\mathrm{C}=\frac{\mathrm{T}\mathrm{P}\times \mathrm{T}\mathrm{N}-\mathrm{F}\mathrm{P}\times \mathrm{F}\mathrm{N}}{\sqrt{(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P})(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N})(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P})(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N})}}.$$

Here we refer nonzero effects as positives and zero effects as negatives. The $\mathrm{M}\mathrm{C}\mathrm{C}$ varies from 0 to 1 with larger values implying better variable selection.

In each differential analysis, the ridge regression employed the generalized cross validation [Golub et al., 1979] to select the ridge parameter, and the adaptive lasso used 10-fold cross-validation to choose its tuning parameter. Following the recommendation by Fan and Lv [2008], (n^(k))^0.9 variables are screened by ISIS.

For each type of networks, 100 synthetic data sets were generated, and the differential analysis results are summarized in Figure 3. Overall, both ReDNet and the naive approach maintain high power in identifying differential regulatory effects. However, the naive approach fails to identify common regulatory effects and tends to report FDR over 80% on differential regulatory effects. Such a tendency to report false positives by the naive approach results in lower $\mathrm{M}\mathrm{C}\mathrm{C}$, with dramatic decrease in identifying differential regulatory effects.

Figure 3:

Performance of ReDNet Versus the Naive Approach which Independently Constructs Two Networks. The results average over 100 synthetic data sets for different types of networks, with letters A, C, S, D in the x-axis denoting Acyclic, Cyclic, Sparse and Dense networks, respectively. “Diff”, “Common” and “Average” summarize the performance on differential, common and average regulatory effects, respectively. FDR and $\mathrm{M}\mathrm{C}\mathrm{C}$ of the naive approach are undefined due to its failure to identify common effects. The sample size $n^{(2)}=n^{(2)}$ is either 200 or 300.

While both methods performed stably across networks with ${\mathbf{X}}^{(1)}\ne {\mathbf{X}}^{(2)}$ and ${\mathbf{X}}^{(1)}={\mathbf{X}}^{(2)}$, ReDNet performed better in identifying differential regulatory effects from dense networks than sparse networks in terms of FDR and $\mathrm{M}\mathrm{C}\mathrm{C}$. However, the naive approach tends to report even higher FDR and so much lower $\mathrm{M}\mathrm{C}\mathrm{C}$ when identifying differential regulatory effects from dense networks. Nonetheless, the naive approach fails to identify common regulatory effects for each type of networks so the corresponding FDR and $\mathrm{M}\mathrm{C}\mathrm{C}$ are undefined.

We also calculated the standard errors (SE) of the reported FDR, power, and $\mathrm{M}\mathrm{C}\mathrm{C}$ over 100 synthetic data sets (the results are not shown). They are all small with most at the scale of thousandth and others at the scale of hundredth. Therefore, ReDNet performed robustly in differential analysis of networks, and the 2SPLS approach by Chen et al. [2017] performed also robustly in constructing single networks.

3.2. THE GENOTYPE-TISSUE EXPRESSION DATA

We performed differential analysis of gene regulatory networks on two sets of genetic genomics data from the Genotype-Tissue Expression (GTEx) project [Carithers et al., 2015], with one collected from human whole blood (WB) and another one from human muscle skeletal (MS). The WB and MS data included genome-wide genetic and genotypic values from 350 and 367 healthy subjects, respectively. Both data sets were preprocessed followingCarithers et al. [2015] and Stegle et al. [2010], resulting in a total of 15,899 genes and 1,083,917 single nucleotide polymorphisms (SNPs) being shared by WB and MS.

Expression quantitative trait loci (eQTL) mapping [Gilad et al., 2008] was conducted and identified 9875 genes with at least one marginally significant cis-eQTL (with p-value< 0.05). For each gene, we further filtered its set of cis-eQTL by controlling the pairwise correlation under 0.9 and keeping up to three cis-eQTL which have the strongest association with the corresponding gene expression. These cis-eQTL serve as anchoring exogenous variables for the genes, and expression levels of different genes are endogenous variables. At completion of preprocessing data, we have 9,875 endogenous variables and 23,920 exogenous variables.

We applied ReDNet to infer the differential gene regulation on a set of eighty target genes, which had largest changes on gene-gene correlation between the two tissues. We identified a total of 640 common and 572 differential regulations on the eighty target genes. To evaluate the significance of identified regulations, we bootstrapped 100 data sets, and conducted differential analysis on each bootstrap data set. As summarized in Table 1, 50, 43 and 34 differential regulatory effects were identified in over 70%, 80% and 90% of the bootstrap data sets, respectively.

Table 1:

Summary of Regulations Identified in Over 70%, 80%, 90% of the Bootstrap Data Sets by ReDNet From GTEx Data. Shown under “Original” are for those identified from the original data.

	Original	70%	80%	90%
Common	640	49	40	34
Differential	572	50	43	34

The top five subnetworks bearing differential regulations on some of the eighty target genes were shown in Figure 4. We also constructed the differential networks using the naive approach (the results are not shown), and reported more regulations which cover the reported ones by ReDNet. This concurs with our observation in the synthetic data evaluation that the naive approach tends to report higher false positives, especially for differential regulatory effects.

Figure 4:

The Top Five Differential Subnetworks of Gene Regulation Identified by ReDNet From GTEx Data. The dotted, dashed, and solid lines imply regulations constructed in over 70%, 80%, and 90% of the bootstrap data sets, respectively. Highlighted in yellow are the target genes whose regulatory genes are focused in this study. The differential regulations are in red while common regulations are in black. The arrow head implies up regulation in both networks or no regulation in at most one network; the circle head implies down regulation in the whole blood but up regulation in muscle skeletal; and the diamond head implies up regulation in whole blood but down regulation muscle skeletal.

4. CONCLUSION

We have developed a novel two-stage differential analysis method named ReDNet. The first stage, i.e., the calibration stage, aims for good prediction of the endogenous variables, and the second stage, i.e., the construction stage, identifies both common and differential network structures in a node-wise fashion. The key idea of ReDNet method is to appropriately reparametrize the independent models into a joint model so as to estimate differential and common effects directly. This approach can dramatically reduce the false discovery rate. In the experiments with synthetic data, we demonstrated the effectiveness of our method, which outperformed the naive approach with a large margin. Note that ReDNet allows independently conducting all ℓ₂ regularized regressions at the same time at the first stage, and all ℓ₁ regularized regressions at the same time at the second stage. Therefore, ReDNet not only permits parallel computation but also allows for fast subnetwork construction to avoid potential huge computational demands from differential analysis of large networks.

There are some interesting directions for future research. Firstly, it is worthwhile to explore other re-parametrization approaches such as baseline reparametrizaiton in a case-control study. Secondly, while we only consider differential analysis of two networks, it is possible to generalize our method to compare multiple networks, demanding more complex reparametrization. Finally, applying the proposed method for fully differential analysis of 53 tissues in the GTEx project still provides challenging computational and methodological issues.