Reconstructing transcriptional regulatory networks through genomics data

Abstract

One central problem in biology is to understand how gene expression is regulated under different conditions. Microarray gene expression data and other high throughput data have made it possible to dissect transcriptional regulatory networks at the genomics level. Owing to the very large number of genes that need to be studied, the relatively small number of data sets available, the noise in the data and the different natures of the distinct data types, network inference presents great challenges. In this article, we review statistical and computational methods that have been developed in the last decade in response to genomics data for inferring transcriptional regulatory networks.

1 Introduction

The central dogma of molecular biology is DNA→RNA→Protein, where DNA is transcribed to RNA, which is subsequently translated into protein to perform diverse functions. Transcription regulation is a fundamental biological process and much effort has been made to identify important players in this process and to understand how they work with each other to regulate gene expression levels. Before transcription of a specific gene is initiated, its regulatory region is bound by one or more transcription factors (TFs), which recognise their targets through specific sequences, often called binding motifs. After the binding of TFs to the regulatory region of a gene, they recruit the transcriptional machinery to initiate the transcription process. After a gene is transcribed, the product messenger RNA (mRNA) is transported to the cytoplasm of a cell, where it is used as a template for translation. Chromatin structure also plays a key role in transcription regulation and is under many levels of tight regulation.^1,2 After transcription, mRNA decay and silencing are examples of yet further regulation of gene expression.³ Therefore, the transcript abundance in a cell is the result of joint actions of many regulators. Because of its central role in molecular biology, delineating transcriptional regulatory networks (TRNs) has been a very active research area and much progress has been made in recent years as a result of the availability of genome-wide gene expression data and other high throughput data. In this review, we discuss statistical and computational methods that have been developed to infer TRNs. We will primarily focus on inferring the regulatory targets of different TFs through the analysis of different types of genomics data.

In the simplest case, a TRN can be represented by a G × R matrix, where G is the number of genes considered and R is the number of TFs involved in regulation. An entry of 1 in the (i, j) position indicates a regulatory relationship between the j-th TF and the i-th gene, and an entry 0 indicates no regulation between them. Under this formulation, the goal of a TRN inference is to infer which elements in this matrix are 1. This matrix can be made more general to allow distinction between activation and repression and the joint action from multiple TFs. Furthermore, the strength of regulation may also be included in this matrix, that is, a 0 element indicates regulation relationship, but non-0 elements can take on real values that quantify the regulatory strength of a TF on its target genes.

For the purpose of inferring TRNs, the most relevant type of genomics data are location data including ChIP-chip data using either gene-based microarrays^4,5 or tiling arrays,^6,7 and ChIP-seq data.^8,9 ChIP stands for chromatin immunoprecipitation, which is a protocol to separate the truncated DNA sequences that bind to specific protein from DNA suspensions. The DNA sequence samples are then amplified and measured through microarray or sequencing technique with different genome coverage. Each ChIP-chip or ChIP-seq experiment involves one TF and the results give information about the binding locations in the genome for this specific TF either in vivo^4,5 or in vitro.^10,11 The binding locations can then be associated with genes according to genome annotation. From the experimental results, for each location in the genome, the data are either summarised as a statistical significance level for binding between a TF and this location, or an intensity level that quantifies the strength of binding. In their analysis of nine TFs involved in the yeast cell cycle, Simon et al.¹² showed that the observed binding patterns for these nine TFs can largely explain the expression profiles of their target genes. For example, the binding targets of TFs that are active in the early phase of the cell cycle also tend to have their peak expression levels in the early phase. Therefore, it may be possible to infer TRNs directly from ChIP-chip data. For example, for a given TF, we can select a statistical significance threshold, say 0.001, and view all genes that have an observed statistical significance for binding for this TF less than the threshold as regulatory targets for this TF. It is apparent that such a thresholding rule will lead to both false-positive and false-negative results due to the inherent noises in ChIP-chip data collection. If the threshold is set too stringently, we will miss too many regulatory targets, whereas there will be many false-positive results if the threshold is set too high. The newly developed ChIP-seq has the potential to outperform ChIP-chip to provide more accurate results on TF-DNA binding locations. Additionally, three other concerns make this naïve TRN method less appealing. First, the observed binding information does not imply regulatory relationship between a TF and a gene that it is binding to, even when the results are highly significant. Gao et al.¹³ estimated that only 58% of the genes whose promoters were bound by a TF are its true regulatory targets. Second, TF binding is a dynamic process, and a TF can have different targets at different time points in a time course experiment and/or under different conditions. If we draw conclusions on the regulatory targets for a TF based on one or a few ChIP-chip experiments, we will miss many true targets and also include many false targets for this TF under other conditions even if the experiments are done perfectly. Third, TRNs involve combinatorial effects of multiple TFs whereas ChIP-chip data only involve one TF per chip, so joint regulation from multiple TFs needs to be inferred indirectly.¹⁴

In addition to location data, other types of genomics data offer valuable information on TRNs, with the most commonly available and studied type being microarray gene expression data. Compared to the relatively limited amount of ChIP-chip or ChIP-seq data, there is a very large amount of gene expression data in the literature,^15–18 and there are many public repositories for expression data. It is easy to understand how gene expression data can help delineate TRNs. If a set of genes are under the control of the same TFs, they should exhibit similar expression profiles over time or across different conditions. If we can identify a gene cluster based on their similar expression profiles, we may hypothesise that they should share similar regulation patterns and use this rationale to improve TRN inference. Alternatively, if we can either observe or infer the activity levels of a TF across a set of conditions, we may infer its regulatory targets by selecting those genes showing similar (maybe time delayed) expression profiles. These ideas have been formalised mathematically in the literature and we will discuss these methods below.

Another type of data commonly used in inferring TRNs are DNA sequences. As TFs recognise their targets through specific sequences, genes sharing common sequences in their regulatory regions are more likely to be under similar regulation. This logic has been extensively used to infer TF binding motifs. On the other hand, if the binding motif for a TF is known, a gene whose regulatory region contains one or more instance of this motif is more likely to be the regulatory target of this TF. Putative regulatory targets only identified through motif matching are known to lead to many false-positive and false-negative results as many genes whose regulatory regions contain the motifs are not the regulatory targets and many regulatory targets do not have known binding motifs in their regulatory regions. This lack of sensitivity and specificity through sequence and motif analysis alone is the result of many unknown factors and mechanisms for gene expression regulation. Nevertheless, sequence information does provide valuable information complementary to ChIP-chip and expression data for TRN inference.

Apart from the above three data types commonly used in TRN inference, other high throughput data also offer useful information, such as mRNA decay data¹⁹ and chromatin modification data.² In principle, joint analysis of all data types under a single unified framework should be the most informative way for TRN inference. In this article, we review various approaches that have been developed for reconstructing TRNs based on one or more data types discussed above. We will focus on the basic modelling assumption and main ideas in inferential strategies without going to detailed implementations and parameter inference. The technical details can be found in the original articles. Because TRN inference research is highly inter-disciplinary and fast evolving, we only provide a partial review of what are available with a focus on those methods with a strong statistical flavour.

2 Methods based only on gene expression data

Since microarray technology was first developed for expression analysis, gene expression microarrays were the only genomic array type available before other platforms, e.g. ChIP-chip data. Even today, most genomics data are collected from gene expression microarrays. Therefore, considerable efforts were made to deduce genetic networks from gene expression data alone. In this review, we begin with the methods that use only gene expression data for network inference. In this context, most methods consider general relationships among genes, without specific reference to TFs. Therefore, the inferred network should be more appropriately described as gene regulatory networks (GRNs).²⁰

For gene expression data, there are broadly two types of experiments. In the first type, gene expression profiles are followed across a number of time points. For the second type, a number of perturbations or a number of individual samples are studied but only one observation is made for each perturbation/individual. Some published methods can be applied to both types of gene expression data, whereas specific methods have been developed to deal with only one type, e.g. time course experiments. In our following discussion, we call these types time course and steady state data.

2.1 Relevance networks

One intuitive method for network reconstruction is to examine the degree of association of expression profiles between each pair of genes. A strong association suggests either two genes are under similar regulatory control or one gene is involved in the regulation of the other gene. A network can thus be built based on pairwise dependencies,²¹ and this network is sometimes called relevance network (RN). Practical issues involve (1) the choice of association measures, which can be based on standard Pearson correlation coefficient, its various transformations, or more robust mutual information measures, and (2) the labelling of the network, whether to keep all the edges or only focus on those passing a specific threshold. In this context, a sub-network where all gene pairs have high association values is sometimes called a module,²² and it has been found that these modules studied as a unit can lead to interesting biological findings. One drawback of this approach is that pairwise dependency may be due to either direct regulatory relationship or indirect relationship through other genes. Therefore, if the primary objective is to identify genes that directly interact with each other, the networks constructed purely from pairwise analysis may contain many false-positive results. One simple way to reduce the number of false positives is to study association between two genes in the presence of one or more other genes. The basic idea is that if both genes are regulated through a third gene, then conditional on the third gene, the first two genes will no longer be associated. This idea can be realised in different forms. ARACNE developed by Margolin et al.²³ uses the data processing inequality

$$I(g_1,g_3)\le \text{min}[I(g_1,g_2),I(g_2,g_3)],$$

where I(g₁, g₃) is the mutual information between g₁ and g₃ to remove dependency due to other genes such as g₂. A similar idea using conditional correlation coefficients was used by Rice et al.²⁴ and Wille and Buhlmann.²⁵

2.2 Gaussian graphical models

If we consider all the genes simultaneously instead of two or three at a time, we may model the joint distribution as a multivariate normal with its mean and covariance matrix. Although the multivariate normality assumption may be violated for real data, it does provide a compromise between real data complexity and statistical and computational feasibility. In this formulation called Gaussian graphical models (GGMs), the associations between the genes are described by a covariance matrix Σ = {σ_ij}, where σ_ij is the covariance between genes i and j. One attractive property of GGMs is that it is straightforward to calculate the conditional correlation coefficient between genes i and j conditional on all other genes. This conditional association is defined as partial correlation coefficient ρ_ij = −σ^ij/(σⁱσ^jj)^1/2, where σ^ij is the (i, j)-th element in Σ⁻¹, the inverse of the covariance matrix Σ. Σ⁻¹ is often called the precision or concentration matrix. Two genes in a GGM are connected by an edge if ρ_ij is not equal to 0, and the edges are undirected. This is in contrast to RNs, which can be built from the Σ matrix. Therefore, if we have access to the true covariance matrix describing pairwise associations between gene pairs, the GGM formulation will allow us to distinguish gene pairs that directly interact from those that indirectly interact through other genes. However, the matrix Σ is unknown and has to be estimated from the observed data. In typical microarray studies, G is usually in the order of thousands whereas the number of experiments is usually much smaller. Therefore, the rank of Σ is much smaller than its dimension, and this matrix is singular. Early on, Toh and Horimoto²⁶ proposed to reduce the dimensionality of the matrix Σ by first clustering genes into clusters and then only evaluating the relationships among gene clusters. It is apparent that there will be significant information loss in this process and we can no longer refer to individual genes, which are the primary targets of network inference.

Under a GGM, we can consider one gene at a time and model the dependency between this gene and other genes through regression, where the dependent variable is the expression level of a specific gene and the independent variables are the expression levels of other genes. More specifically, we have the following regression model:

$$m_i={\sum }_{j=1,j\ne i}^G{\beta }_{ij}{m}_j+{\epsilon }_i,$$ (1)

where m_i is the expression level for the i-th gene, β_ij is the regression coefficient of the j-th gene on the i-th gene and ε_i is the noise term. This regression model is over-parameterised in our context as the number of predictors, i.e. genes, is generally much larger than the number of observations. However, because a biological network is usually sparse, that is, the number of regulators for a gene is limited, this sparsity assumption can be used for statistical inference.^27–34 For example, Meinshausen and Buhlmann²⁹ proposed to consider one gene at a time and perform regression through regularised regression methods such as LASSO.³⁵ The objective function has the following form:

$${\mathit{\beta ̂}}_i^{\lambda }=\underset{{\mathit{\beta }}_i}{\text{arg min}}\{{\Vert m_i-{\sum }_{j=1,j\ne i}^G{\beta }_{ij}{m}_j\Vert }_2^2+\lambda {\sum }_{j=1,j\ne i}^G|{\beta }_{ij}\left|\right\},$$

where ${\mathit{\beta ̂}}_i^{\lambda }$ is the regression coefficient vector for gene i, and λ is the penalty term that balances between model fit and model complexity. This procedure often leads to a model with a few non-zero regression coefficients. The interpretation of these parameters is that those genes with non-zero regression coefficients have an impact on the i-th gene. This regression analysis can be repeated for every single gene treated as the dependent variable to construct the network. Because each gene is fitted separately, the resulting network structure may not be symmetric, e.g. it may happen that β_ij = 0 but β_ji ≠ 0, leading to interpretation problems. The sparsity regression can also be accomplished through specifying sparse priors within the Bayesian setting.^27,31 Other methods based on the regression setting include approaches from iterative greedy algorithms and combinatorial optimisation algorithms.^36,37 Gardner et al.³⁸ addressed the sparsity problem by considering a fixed number of TFs at a time and then selecting the subset with the best fit to the experimental data.

Amore direct way of imposing sparsity constraint on the whole network was proposed by Peng et al.³⁰ Their objective function is:

$$\frac{1}{2}\left({\sum }_{i=1}^G{w}_i{\Vert m_i-{\sum }_{j=1,j\ne i}^G{\rho }_{ij}\sqrt{\frac{{\sigma }^{jj}}{{\sigma }^{ii}}}{m}_j\Vert }^2\right)+\lambda \sum _{1\le i<j\le G}\left|{\rho }_{ij}\right|,$$

where β_ij = ρ_ij(σ^jj/σⁱⁱ)^1/2 is the regression coefficient, w_i is the weight for gene i, which may be related to the variance of the noise for the i-th gene, and λ is again the penalty term that balances between model fit and model complexity across all the genes. The resulting matrix should retain symmetry under this approach. The authors argued that the general sparsity constraint on the whole network leaves room for specific genes to have many connections, an attractive feature to accommodate the presence of so-called hub genes that have many interactions. Similarly, a threshold gradient descent method was proposed by Li and Gui²⁸ to estimate the precision matrix.

Yeung et al.³⁹ proposed to use singular value decomposition (SVD) in this context for TRN inference. In the case that external perturbations are used, they wrote their system in the following matrix form:

$$\mathrm{\Delta }{\mathit{M}}_{G\times K}={\mathit{B}}_{G\times G}{\mathit{M}}_{G\times K}+{\mathit{E}}_{G\times K}+{\mathit{\epsilon }}_{G\times K},$$

where ΔM_G×K is the change of expression levels for all the genes, represented by the G rows, after each of the K perturbations represented by the columns, M_G×K is the collection of all the expression levels, B_G×G is the regression coefficient matrix, E_G×K summarises the external perturbations and ε_G×K is the noise matrix. Standard regression does not apply here because more genes than conditions are studied. With SVD, M^T can be written as ${\mathit{M}}_{K\times G}^T={\mathit{U}}_{K\times G}{\mathit{D}}_{G\times G}{\mathit{V}}_{G\times G}^T$, where U_K×G and V_G×G are orthogonal and D_G×G is a diagonal matrix. The pseudo-inverse of M^T can be calculated as ${\mathit{V}}_{G\times G}{\mathit{D}}_{G\times G}^{-1}{\mathit{U}}_{G\times K}^T$. The family of solutions for B_G×G has the form of $(\mathrm{\Delta }{\mathit{M}}_{G\times K}-{\mathit{E}}_{G\times K}){\mathit{U}}_{K\times G}{\mathit{D}}_{G\times G}^{-1}{\mathit{V}}_{G\times G}^T+{\mathit{C}}_{G\times G}{\mathit{V}}_{G\times G}^T$, where C_G×G is an arbitrary G × G matrix. Yeung et al.³⁴ proposed to select among these solutions the sparsest one as the inferred network.

2.3 Bayesian networks

One extensively studied class of models in causal inference is Bayesian networks (BNs) and it is not surprising that they were proposed very early on to infer gene networks based on gene expression data.^40,41 Within a BN, the relationships among the genes are described by a directed acyclic graph (DAG), where the state of a gene is jointly determined by its parental nodes in a probabilistic manner. For a given DAG, the joint distribution of all the genes can be factored into a product of conditional probabilities. The balance between model fit and model complexity is integrated into the learning for the structure of the underlying DAG for a gene network. BN analysis itself is a very broad and active research field.⁴² Because most published articles simply adopted existing methods to analyse gene expression data, we will not attempt to review the general methods related to BNs here, but note a number of limitations in the context of gene network inference. First, different DAGs may give rise to the same likelihood for the observed data, leading to equivalent classes. Therefore, it is impossible to distinguish these DAGs purely from the observed data. Second, different perturbation experiments are treated exchangeably in the analysis although the underlying biological states are clearly different. Third, BNs do not allow feedback loops, a common phenomenon in biological networks. Dynamic Bayesian networks (DBNs), which can alleviate this problem to some extent, have been proposed.^43,44 However, a time homogeneous transition model is normally assumed in DBNs, an oversimplification of the dynamics of the biological process. Fourth, continuous observations need to be discretised for BN analysis. It is often a non-trivial task to decide on the number of levels and the thresholds used for discretisation. Finally, although methods have been developed to incorporate prior information in BN analysis,⁴⁵ it is difficult to incorporate some aspects of biological knowledge explicitly, e.g. the kinetic models relating transcription rates with TF activities to be discussed below. The performance of BNs and DBNs was evaluated through simulations.⁴⁶ It was found that some local structures of the networks may be recovered and the author also showed the importance of incorporating prior biological knowledge and having the systems studied at appropriate time points, preferentially right after the system is perturbed.

The relative performance of RNs, GGMs and BNs has been compared in the literature.^23,47,48 Both BNs and GGMs had better performance than RNs. It was found that structural perturbations are more informative than dynamic perturbations, e.g. time course experiments. In addition, when a small system was considered, BNs outperformed GGMs.

2.4 Time course microarray gene expression data

The methods discussed above can be applied to both perturbation data and time course data. However, the above methods treat each time point separately, so the time dependencies are not utilised. This may potentially lead to loss of information for TRN inference. Many statistical methods have been developed to analyse time course gene expression data, and here we focus on those whose primary goal is to infer regulatory relationships among the genes.

Similar to the regression setting discussed above, the following model can be used to describe the dynamics of a gene:

$$\frac{m_i(t+\mathrm{\Delta }t)-m_i(t)}{\mathrm{\Delta }t}={\sum }_{j=1,j\ne i}^G{\beta }_{ij}{m}_j(t)-d_i{m}_i(t)+{\epsilon }_i(t+\mathrm{\Delta }t),$$ (2)

where the dependent variable is the expression level change between two observation points instead of the expression level itself, and there is an additional term d_i that characterises the decay rate of the i-th gene. If we ignore the decay rate in this model, the methods discussed in the previous section can be applied here for inference. To take advantage of the fact that expression levels at nearby points tend to be similar, the TSNI algorithm developed by Bansal et al.⁴⁹ uses splines to smooth the observed time course data first and then apply SVD to the gene expression matrix to reduce the dimensionality of the predictors.

In contrast to linear models analysed in most published articles, Wang⁵⁰ considered the following non-linear regression model:

$$\frac{m_i(t+\mathrm{\Delta }t)-m_i(t)}{\mathrm{\Delta }t}=[{\alpha }_i\prod _j{m}_j^{w_{ji}}(t)\prod _k{m}_k^{w_{ki}}(t-\mathrm{\Delta }t)\prod _l{m}_l^{w_{li}}(t-2\mathrm{\Delta }t)-d_i{m}_i(t)]+{\epsilon }_i(t+\mathrm{\Delta }t),$$

where the joint effect from multiple genes are modelled through a multiplicative function and time-delayed responses are allowed by incorporating gene expression levels at previous time points, t − Δt and t − 2Δt. The exponents in the equation, w_ji, w_ki and w_li, quantify the effects of the corresponding genes, and d_i is the mRNA decay rate. This model can be considered a variant of the S-system.^51,52 It is apparent that the model fitting can be challenging for this setup. Indeed, the author only considered a limited set of genes when this model was applied and used the genetic algorithm coupled with Bayesian information criterion (BIC) for model selection.

When data from samples in steady state and time course experiments are both available, Bonneau et al.⁵³ proposed to use a single regression model of equation form (1) to analyse them together. Here is a brief justification. Consider the time course dynamic model of equation form (2). At the steady date, there is no change of expression levels, and we have

$$0={\sum }_{j=1,j\ne i}^G{\beta }_{ij}{m}_j-d_i{m}_i.$$

So the regression model can be written as

$$m_i={\sum }_{j=1,j\ne i}^G{\beta }_{ij}{m}_j/d_i,$$

which has similar form to the dynamic model of equation form (2). Bonneau et al. also considered alternative forms of the regulation function that may incorporate interactions, e.g. adding another term min(m_i, m_j), in the model.

Only gene expression data are used for network inference in the methods discussed so far. In this setting, there is an implicit assumption that the observed expression level for a gene can be used as a surrogate for the TF activity level of this gene so that it can be used as a predictor in regression models. However, it is well known that there is poor correlation between gene expression levels and protein activity levels due to many steps involved from transcription to translation, and then to post-translational modification. Although the above methods have the advantage that they are generic, easily understood, and adapted from established methods, it would seem intuitive that more can be gained by appropriately modelling the underlying biological process and incorporating prior knowledge and different data types in TRN inference.

3 Methods based on combined data analysis

3.1 Clustering-based analysis

Clustering analysis is the most commonly used method for visualising gene expression patterns.⁵⁴ Many statistical methods have been proposed to cluster genes. Sets of genes with highly similar expression profiles are often called modules.²² Genes in the same cluster tend to have similar biological functions, so these clusters can be used to infer the functional roles of un-annotated genes. As TFs are a major regulator of transcription, genes in the same cluster or module are more likely to be under the control of the same or similar TFs. Because TFs recognise their targets through sequence specific binding motifs, the regulatory regions of genes in the same cluster should be enriched for some binding motifs. The discovery of these binding motifs can facilitate the inference of relevant TFs if the binding motifs of the TFs are known from prior experiments. In fact, this was done early on in microarray data analysis, e.g. Spellman et al.¹⁸ Since a set of genes may be co-regulated only under a subset of the experimental conditions, a number of methods have been developed to cluster genes under specific conditions,^55–57 and the gene clusters thus identified are sometimes called conditional specific expression modules. The methodologies developed in this context can also be extended to combine data from diverse data sources.⁵⁸

For a given gene cluster or module, it is natural to relate it to the activities of a set of TFs. In the absence of information for TF activities, the corresponding gene expression levels may be used as surrogates. Segal et al.⁵⁹ developed an iterative procedure for combining gene expression data with regulator activities (estimated by gene expression levels) for both improved cluster analysis and the interpretation of how each cluster is regulated. The basic idea is to iterate between two steps. The first step involves inferring key regulators for each cluster by correlating regulator levels with expression patterns of genes in the cluster across a large number of experimental conditions. In the second step, regulator activities and gene expression data are jointly used to more accurately assign each gene to its corresponding cluster. However, this procedure is limited by the pre-specified number of clusters and the possibly poor correspondence between gene expression levels and regulator activities. A similar approach to incorporate gene expression data and sequence data was introduced by the same authors.⁶⁰ In the context of biclustering, Reiss et al.⁶¹ developed an iterative algorithm to incorporate sequence information into building biclusters.

Bar-Joseph et al.⁶² proposed a procedure called GRAM for TRN inference by combining gene expression data and ChIP-chip data. In the first step, they used a stringent criterion to infer the binding targets only for those TF-gene pairs that are highly statistically significant, e.g. p-value less than 0.001. Then, gene expression data are used to define a core expression profile for a set of genes sharing a common set of TFs as their regulators. After the core expression profiles are defined, other genes are included in a transcription module if their expression profile is similar to the core profile and there is some marginal evidence of binding. A conceptually similar approach was proposed by Lemmens et al.⁶³ that also incorporates motif data in the analysis.

Model-based clustering methods have also been proposed to combine gene expression data with other information. A two-component mixture model was proposed by Wang et al.⁶⁴ to infer true regulatory sites from combined analysis of gene expression data with either sequence data or ChIP-chip data. A sparse regression mixture model was developed by Li et al.⁶⁵ where the mean expression profile in each cluster is determined by the additive effects of a set of regulators. Liu et al.⁶⁶ proposed an infinite mixture model to jointly analyse gene expression data and ChIP-chip data. In this work, the expression information for a specific gene i is represented by a vector m_i = (m_i1, …, m_iK) across a set of K experiments. The ChIP-chip binding data between R TFs and this gene is summarised by γ_i = (γ_i1, …, γ_iR), where γ_ij indicates statistical evidence for binding from ChIP-chip data. A gene in each cluster is assumed to have a multivariate normal component describing expression levels and a binary vector component modelling binding patterns.

Compared to the methods that rely only on gene expression data for network inference, these cluster-based methods aim to integrate other information, gene annotations, sequence information, or ChIP-chip data, to both make more accurate gene groupings and interpret each cluster as a result of the joint action of a set of regulators. Therefore, not only can we learn which genes belong to the same cluster, we can also explain the regulation mechanism that makes these genes in the same cluster. One major drawback of these methods is that gene expression data are often used as surrogates for TF activities, which may lead to poor performance due to the lack of correspondence between them. In the following, we will review kinetic models relating transcription levels/rates with TF activities, and discuss various methods that have been proposed to infer TF activities from gene expression data.

3.2 Kinetic models for mRNA synthesis

We first consider the Michaelis–Menten kinetics model, which has been used⁶⁷ to relate transcription (mRNA synthesis) rate s_i for the i-th gene with a single TF activity r in the following form:

$$s_i=b_i\frac{r}{c_i+r}+a_i,$$ (3)

where b_i is the maximum transcription rate, c_i is the half-saturation constant and a_i is the basal transcription rate. The parameters used in this formulation, a, b and c, are TF-gene specific. The value of 1/c, denoted by γ, can also be interpreted as the ratio of association constant (k_a) and dissociation constant (k_d) between TF and DNA. Let PD denote the concentration of TF-DNA complexes and D denote the concentration of non-occupied sites. DNA occupancy is defined as O = PD/(PD + D). At equilibrium, k_a/k_d = PD/(D × P), where P is the free TF concentration. Therefore, O = 1/[1 + (k_a/k_d)/P] = P/[k_a/k_d + P]. As O is the proportion of the sites occupied by the TF-DNA complexes, it is reasonable to assume that the transcription rate is a linear function of the occupancy, leading to a functional form of Equation (3).⁶⁸ The value of O may be estimated from high-throughput TF-DNA interaction data^4,5 or from sequence data⁶⁸ based on the binding motif of the TF. To allow for the saturation effect, we can use the following model:

$$s_i=b_i\frac{e^r}{c_i+e^r}+a_i.$$

When there are two TFs, under different assumptions on the cooperativity of the two factors, e.g. independent binding, Nachman et al.⁶⁹ derived the binding pattern probabilities under the equilibrium state. Then the overall transcription rate can be calculated as the weighted average of these four probabilities, where the weights correspond to the transcription rate at a specific binding pattern. More general forms are given in Nachman et al.⁶⁹ Consider the scenario that transcription can only start when a set of R regulators are all present in the promoter region of a gene,

$$s_i=\alpha \frac{\prod _{j=1}^R{(\frac{r_j}{c_{ij}})}^{{\beta }_{ij}}}{1+\prod _{j=1}^R{(\frac{r_j}{c_{ij}})}^{{\beta }_{ij}}},$$

where c_ij is an association constant for binding between gene i and TF j and β_ij measures the cooperativity in binding. Pan et al.⁷⁰ modelled binding as a function of sequences.

Chen et al.⁷¹ considered a simpler additive model of the following form:

$$s_i=c_0+{\sum }_{j=1}^R{c}_{ij}\frac{e^{r_j}}{1+e^{r_j}}.$$

A similar functional form was used by Chen et al.⁷² who also included a time delay term from gene expression level to activity level.

Based on equilibrium assumption, Sun et al.⁷³ considered a similar model of the form

$$s_i=\alpha \prod _{j=1}^R{r}_j^{{\beta }_{ij}},$$

where β_ij is the binding affinity between gene i and TF j which can be approximated by the binding intensity measured from ChIP-chip data.

When different TFs can either activate or repress a gene, Yeung et al.³⁹ considered the following model:

$$s_i=\frac{a_i+{\sum }_{j\in A_i}{r}_j^{{\beta }_{ij}}}{1+{\sum }_{j\in A_i}{r}_j^{{\beta }_{ij}}+{\sum }_{k\in R_i}{r}_k^{{\gamma }_{ik}}},$$

where β_ij is the activation cooperativity of the j-th TF on the i-th gene, and γ_ik is the repression cooperativity of the k-th TF on the i-th gene. When there are no repressors, this model is very similar to the additive model of Chen et al.⁷¹

Porreca et al.⁷⁴ considered piecewise linear models as a compromise between linear models, which are easy to handle computationally and statistically but may only provide an approximation to real models, and non-linear models, which are more relevant biologically but present significant computationally challenges.

In summary, a variety of kinetics models have been considered in the literature, some based on physical-chemical principles whereas others purely from statistical convenience. A more detailed review on various kinetics models for the synthesis rate of the i-th gene, s_i, can be found in Ben-Tabou de-Leon and Davidson.⁷⁵

The kinetics of mRNA degradation is usually modelled as a first-order differential equation. The combination of mRNA synthesis and degradation depicts the rate of transcription level of a given gene (or dm_i/dt).

3.3 Regulation models

For a given kinetics model, we have the following differential equation describing the dynamics of the gene expression level for the i-th gene:

$$\frac{\mathrm{d}{m}_i}{\mathrm{d}t}=s_i-d_i{m}_i+{\epsilon }_i,$$

where m_i is the mRNA level of the i-th gene, s_i is the transcriptional synthesis rate, d_i is the mRNA decay rate and ε_i is the noise associated with the system and measurements. This model can be used to describe time course gene expression patterns. If the cells have reached an equilibrium state, we have s_i − d_im_i = 0. Hence the transcript abundance at the equilibrium state m_i is proportional to s_i/d_i. Therefore, we can relate the expression level with TF activities through the kinetics models discussed in the previous section. When a TRN is known, the above models can be used to both estimate TF activities and kinetics parameters^67,69,76,77 from the observed expression data. For example, when the regulatory targets are assumed to be known for a TF, the Michaelis-Menten kinetics model can be used to infer TF activities as well as the kinetic parameters either through maximum likelihood⁶⁷ or Bayesian approach.⁷⁶ However, care needs to be taken because there may be identifiability problems for some systems.⁷⁸

Because linear models are most commonly used in relating expression levels with TF binding and TF activities, we will focus on linear statistical approaches in the following discussion. In this case, the following model has been used by a number of groups.^13,73,79,80:

$$\text{log}(m_i)={\sum }_{r=1}^R{a}_{ij}\text{log}(r_j)+{\epsilon }_i,$$

where m_i is either the absolute or relative observed expression level for the i-th gene, a_ij represents the regulatory strength between the j-th TF and the i-th gene (which can be measured from location data) and r_j represents the activity of the j-th TF. When the ChIP-chip data are used as an estimate for the a_ij, e.g. the binding intensity is used as a surrogate, the estimates of the TF activities reduce to a regular regression problem.^13,80 Cokus et al.⁸¹ studied the dynamics of the inferred TF activities from linear regression models. For time course data, Wang et al.⁸² proposed to borrow information across time points to have improved TF activity estimates. This is achieved through the following model:

$$m_i(t)=\mu (t)+{\sum }_{j=1}^G{r}_j(t)x_{ij}+{\epsilon }_i(t),$$

where r_j(t) is modelled by natural cubic B-splines, and x_ij is the estimated regulation strength between TF j and gene i. These studies found that the inferred TF activities can be substantially different from the expression levels of the same gene, leading to concerns on potential biases in using expression levels as surrogates for TF activities in many of the published studies.

Because physical binding does not imply regulation,¹³ TRN inference can be improved by combining ChIP-chip data with gene expression data. Sun et al.⁷³ proposed BEAM, a Bayesian method for integrating ChIP-chip data and expression data, based on the following model:

$$\text{log}(m_i/m_i^0)={\sum }_{r=1}^R{a}_{ij}\text{log}(r_j/r_j^0)+{\epsilon }_i,$$ (4)

where the log-ratio of the expression levels (m_i and $m_i^0$) between two conditions is related to the additive effects from a set of R TFs through the log-ratio of the TF activities (r_j and $r_j^0$) and the regulatory strength a_ij. Instead of directly using the binding intensities from ChIP-chip data,^13,80 Sun et al. assumed that a_ij = b_ijc_ij, where c_ij is the unknown but desired regulatory relationship between TF j and gene i. With this formulation, model (4) becomes

$$\text{log}(m_i/m_i^0)={\sum }_{r=1}^R{b}_{ij}{c}_{ij}\text{log}(r_j/r_j^0)+{\epsilon }_i.$$

Under this set-up, gene expression levels m_i and binding intensities b_ij are observed, and there is some partial knowledge on c_ij. The objective is to infer the true regulatory matrix c_ij. For a set of K experiments, the above model can be written in the following matrix form (ignoring the noise terms):

$${\left(\begin{array}{ccc}\hfill \text{log}\left(\frac{m_1^1}{m_1^0}\right)\hfill & \hfill \cdots \hfill & \hfill \text{log}\left(\frac{m_1^K}{m_1^0}\right)\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill \text{log}\left(\frac{m_G^1}{m_G^0}\right)\hfill & \hfill \cdots \hfill & \hfill \text{log}\left(\frac{m_G^K}{m_G^0}\right)\hfill \end{array}\right)}_{G\times K}={\left(\begin{array}{ccc}\hfill b_{11}{c}_{11}\hfill & \hfill \cdots \hfill & \hfill b_{1R}{c}_{1R}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill b_{G1}{c}_{G1}\hfill & \hfill \cdots \hfill & \hfill b_{GR}{c}_{GR}\hfill \end{array}\right)}_{G\times R}{\left(\begin{array}{ccc}\hfill \text{log}\left(\frac{r_1^1}{r_1^0}\right)\hfill & \hfill \cdots \hfill & \hfill \text{log}\left(\frac{r_1^K}{r_1^0}\right)\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill \\ \hfill \text{log}\left(\frac{r_R^1}{r_R^0}\right)\hfill & \hfill \cdots \hfill & \hfill \text{log}\left(\frac{r_R^K}{r_R^0}\right)\hfill \end{array}\right)}_{R\times K}{\mathit{M}}_{G\times K}={\mathit{A}}_{G\times R}{\mathrm{\Gamma }}_{R\times K}$$ (5)

In this form, the same reference sample is used and the same regulatory matrix A_G×R is assumed across all K experiments. The K columns in Γ_R×K summarise the activities of the R TFs in the K experiments.

BEAM assumes that some information is available on the TFs involved in regulation and the availability of ChIP-chip data to provide useful information on the general structure of the TRN. In addition to ChIP-chip data, sequence data can be used to estimate regulatory potential between a TF and a gene. Xing and van der Laan⁸³ used regression analysis to identify active TFs based on expression data and motifs from known TFs. However, they did not estimate TF activities or regulatory strengths. With sequence information as input for predicting TF binding, an approach similar to BEAM was proposed by Sabatti and James⁸⁴ to infer TRNs.

It is apparent that, if only gene expression data are available, it is generally impossible to factor the observed expression matrix M_G×K in the form of Equation (5) without imposing any constraints on A_G×R and Γ_R×K. Liao et al.⁷⁹ discussed the constraints on the structure of A_G×R, essentially the distribution of the 0 elements in this matrix, needed under which the decomposition is unique and called the analysis as Network Component Analysis (NCA). Compared to Bayesian approaches,^73,84 NCA requires prior knowledge on the connectivity between TFs and genes but does not need inputs on regulatory strengths or TF activities. To accommodate uncertainties in the connectivities, Yu and Li⁸⁵ proposed a two-stage constrained space factor analysis where the constraints are derived from ChIP-chip or other data and the algorithm iterates between network configuration estimation and regulation strength estimation. When the number of TFs is large with possible interactions among them, Boulesteix and Strimmer⁸⁶ proposed to use partial least squares in this regression setting to reduce the dimensionality of the predictor space, i.e. the space spanned by the TFs.

Despite somewhat different motivations, these methods all share the same general modelling form, involving three sets of variables: the gene expression levels, the regulation strengths between TFs and their target genes, and the TF activities. Statistical estimation of TF activities is part of the inference procedure. Pournara and Wernisch⁸⁷ compared the performance of some methods in this context and concluded that most of the tested algorithms are successful in reconstructing the connectivity structure as well as the TF profile.

Tanay and Shamir⁸⁸ considered a different functional form, called a dose-affinity-response function, that relates the response (transcription rate) with the TF doses (activities) and the affinities of TFs to the regulatory region of a gene (regulatory strength). The affinity information can be gathered from the ChIP-chip data and the doses are iteratively estimated from motif information. It appears that only single TFs can be analysed under this approach for practical data and all the variables need to be discretised.

3.4 State-space models

We discussed time course data methods above where only gene expression data are used for inferring relationships among genes. If we consider each gene being regulated by a set of TFs whose activities are unknown, this naturally leads to classical state space models (SSMs).⁹⁰ In an SSM, the gene expression levels, m_t = (m₁(t), m₂(t), …, m_G(t)), are assumed to be generated from a set of R hidden state variables, h_t = (h₁(t), h₂(t), …, h_R(t)), and that the h_t follow the first-order Markov process as follows:

$${\mathit{h}}_t=\mathit{T}{\mathit{h}}_{t-1}+{\mathit{w}}_t,{\mathit{w}}_t~N(0,{\mathrm{\Sigma }}_w){\mathit{m}}_t=\mathit{A}{\mathit{h}}_t+{\mathit{v}}_t,{\mathit{v}}_t~N(0,{\mathrm{\Sigma }}_v)$$

where T is the state dynamics matrix, A is the observation matrix and Σ_w and Σ_v are the covariance matrices for errors associated with the state and observed variables w_t and v_t, respectively. This formulation fits naturally to our understanding of transcription regulation because the hidden variables can be interpreted as TFs, the A matrix defines how TFs regulate gene expression levels, and the T matrix defines how TFs regulate among each other over time. The second component of this SSM has the same form as Equation (5), with the difference being that the first component is used to describe the dynamics of the hidden state variables. Because T is time independent, this model implicitly assumes a time invariant regulation pattern among the TFs. If this assumption largely holds, SSMs may allow us to borrow information across time points to make more efficient TRN inference. However, the downside of this model is that if this time homogeneity assumption is seriously violated, the TRN inference can be biased. In this case, we may use non-parametric methods to model TF activities across time,⁸² but statistical methods are lacking for model inference in this more general case. A more general model form (called an input driven model⁸⁹) has also been considered in the literature:

$${\mathit{h}}_t=\mathit{T}{\mathit{h}}_{t-1}+\mathit{E}{\mathit{m}}_{t-1}+{\mathit{w}}_t,{\mathit{w}}_t~N(0,{\mathrm{\Sigma }}_w),{\mathit{m}}_t=\mathit{A}{\mathit{h}}_t+\mathit{F}{\mathit{m}}_{t-1}+{\mathit{v}}_t,{\mathit{v}}_t~N(0,{\mathrm{\Sigma }}_v),$$

where gene expression data are allowed to affect the hidden state variables. As SSMs have been extensively studied in the literature, the model inference can be estimated using established maximum likelihood methods or Bayesian methods. For example, the methods that can be applied to estimate the number of state variables include BIC,⁹⁰ cross validation⁹¹ and Bayesian approaches.⁹²

In the specific context of TRN inference, the hidden variables have clear biological interpretation as TFs and possibly other regulators. Therefore, with ChIP-chip and sequence data, the structure of the regulation network reflected in A and possibly the T matrix may be partially known. Assuming knowledge on the structure, Xiong and Choe⁹³ proposed a constrained SSM approach for TRN inference. Similarly, when the connectivity matrix is assumed to be known, Sanguinetti et al.⁹⁴ used an SSM to model the dynamics of the observed expression data. In this case, the sparsity constraint is directly provided by the connectivity matrix. With prior knowledge on the TFs involved in regulation, SSMs have also been used to infer TF activities.⁹⁵

3.5 mRNA decay data

Although mRNA decay rate, denoted by d_i, in the above discussion is a key component in the kinetics and regulation model, it is not used in most statistical methods. This is partly due to the limited information on the decay rate at the genome level¹⁹ that can be used in the analysis. In addition, at the steady state, the decay rate appears as a factor in the model for the mean gene expression level, so ignoring it does not change the general model form. However, in the analysis of time course data, the decay rate does play an important role and it is conceivable that more accurate TRN inference may be achieved if each gene’s approximate decay rate can be incorporated in statistical analysis.

In the context of time course gene expression data analysis, Chen and Zhao⁹⁶ used gene expression data and mRNA decay date to estimate transcription rate through simple differential equations. They found that the estimated transcription rates may be more informative on co-regulation and DNA binding motif discoveries. In TRN inference, mRNA decay rates were used by Nachman et al.⁶⁹ to estimate transcription rate. One caveat in using experimentally derived decay rates is that their values are also context dependent,⁹⁷ so it may not add to TRN inference if the expression data and decay data were observed under very different experimental conditions.

4 Discussion

We have reviewed many statistical and computational methods that have been developed in recent years to reconstruct transcriptional regulatory networks from gene expression data, ChIP-chip data, sequence data, and mRNA decay data. Although we have primarily focused on TF-gene interactions in this review, there are many other important regulators involved in chromatin modification and post-transcription regulation. It would be ideal to consider them all under a consistent framework to reconstruct TRNs. For example, Whitington et al.⁹⁸ found that chromatin information offers accurate prediction of tissue-specific binding sites and such data will undoubtedly lead to more accurate TRN inference.

Because the focus of this current review is the inference of TF targets, a thorough review of the extensive literature on inferring TF binding motifs is beyond the scope of this article. We simply note that a large number of methods have been developed to infer TF binding motifs based on gene expression data, sequence data and ChIP-chip data. Some of these motif-finding algorithms are closely tied to TRN inference. For example, REDUCE⁹⁹ regresses gene expression levels on the presence or absence of a putative motif in the promoter regions of all the genes. A functional motif can be identified if there is an association between the observed expression levels and the presence or absence of this motif. In essence, this motif occurrence serves as a surrogate for TF occupancy. The same idea was used to infer binding motifs from ChIP-chip data¹⁰⁰ and combined analysis of gene expression and ChIP-chip data for motif discovery.¹⁰¹ More accurate motif knowledge can provide more informative prior as well as more effective filtering for TRN inference.

Most TRN work relies on expression studies conducted under controlled experiments. However, the combination of naturally occurring variations observed in individuals in a population sample does represent a rich set of perturbation experiments, though not as well controlled. Such samples have been used to study gene expression profiles across individuals and to identify genetic regions controlling gene expression patterns.^102–104 These studies also yield valuable information on TRNs. However, early results did not show an enrichment for TFs in the identified regions.¹⁰⁵ More recent studies do suggest that more informative regulation results can be derived if different data types are systematically integrated, including binding and protein interaction information.¹⁰⁶

In addition to collecting more data types, more samples under different conditions are required to reconstruct a global network. How to best perturb the system to facilitate TRN inference is an area that has not received much attention, but see Barrett and Palsson,¹⁰⁷ Tegner et al.¹⁰⁸ and Bonneau.¹⁰⁹ This may be due to the limited knowledge on TRNs and the fact that the optimal experiments depend on a good understanding of the true underlying networks. In fact, even when a TF is perturbed, it is often not a trivial task to select the optimal experimental condition to perturb this TF to observe the system’s responses.¹¹⁰

Essentially, all the methods proposed to date for TRN inference are statistical in nature without invoking detailed mechanistic models for specific genes involved in a TRN. However, with more research done and more knowledge accumulated, there is no question that the field is moving towards a more detailed and mechanism-driven approach to describing the system. For certain pathways, large differential equation systems have been analyzed in the literature.¹¹¹ One major limitation in analyzing such systems is that many of the kinetic parameters are either unknown or known only to be within a wide range. The system behaviour may be quite sensitive to the choice of these parameter values, and it is often difficult to infer their values because there may be very limited amount of information in gene expression data.^78,112 Currently, there is still a large gap between statistical (descriptive) and mathematical (mechanistic) approaches to systems modelling and analysis. Although they can be considered complementary at this point, these two approaches will certainly converge in the future, but much effort is needed to make this happen.

One factor that will help to achieve this goal is technology, which keeps evolving at a very rapid pace. Although we have focused on microarray gene expression data, sequence-based expression profiling will provide more comprehensive and accurate assessment of the expression patterns for an organism under a given condition.¹¹³ Due to the nature of sequencing data, the above discussed statistical methods need to be modified even if the same general approach is pursued for TRN inference. In addition, we can obtain data at ever-finer levels, e.g. cell-based assays, and monitor expression patterns both in space and time.

Finally, given the many methods available for TRN inference, it is important to compare the performance of these methods based on both simulated and real data.¹¹⁴ But very limited studies have been done to date. Although it may well be that different methods suit different systems and data types, it would be helpful to biologists if some general principles and guidelines can emerge from such studies so that they can indeed take advantage of the advances in statistical and computational methods instead of being inundated with different approaches.