High Dimensional ODEs Coupled with Mixed-Effects Modeling Techniques for Dynamic Gene Regulatory Network Identification

Abstract

Gene regulation is a complicated process. The interaction of many genes and their products forms an intricate biological network. Identification of this dynamic network will help us understand the biological process in a systematic way. However, the construction of such a dynamic network is very challenging for a high-dimensional system. In this article we propose to use a set of ordinary differential equations (ODE), coupled with dimensional reduction by clustering and mixed-effects modeling techniques, to model the dynamic gene regulatory network (GRN). The ODE models allow us to quantify both positive and negative gene regulations as well as feedback effects of one set of genes in a functional module on the dynamic expression changes of the genes in another functional module, which results in a directed graph network. A five-step procedure, Clustering, Smoothing, regulation Identification, parameter Estimates refining and Function enrichment analysis (CSIEF) is developed to identify the ODE-based dynamic GRN. In the proposed CSIEF procedure, a series of cutting-edge statistical methods and techniques are employed, that include non-parametric mixed-effects models with a mixture distribution for clustering, nonparametric mixed-effects smoothing-based methods for ODE models, the smoothly clipped absolute deviation (SCAD)-based variable selection, and stochastic approximation EM (SAEM) approach for mixed-effects ODE model parameter estimation. The key step, the SCAD-based variable selection of the proposed procedure is justified by investigating its asymptotic properties and validated by Monte Carlo simulations. We apply the proposed method to identify the dynamic GRN for yeast cell cycle progression data. We are able to annotate the identified modules through function enrichment analyses. Some interesting biological findings are discussed. The proposed procedure is a promising tool for constructing a general dynamic GRN and more complicated dynamic networks.

1. Introduction

Gene regulation plays a fundamental role in biological activities, such as cell growth, division, development, response to environmental stimulus and so on. Genes carry out their functions through DNA transcription and protein synthesis. Reversely, some proteins, like transcription factors, regulate the activity of genes by binding to their promoters or even encoding the proteins themselves. In addition, some noncoding RNAs, the direct transcript replicates of certain DNA segments regulate genes at some important levels of genome function, including chromatin structure, chromosome segregation, transcription, RNA processing, RNA stability, and translation (Carthew and Sontheimer, 2009). The gene regulatory network (GRN) is a complex system, which should be more appropriately modeled in a dynamic way (Hecker et al., 2009).

Owing to the advancement of new high-throughput technologies such as DNA microarray, we are able to observe the dynamic feature of gene expression profiles on a genome scale. This type of data is composed of experimental measurements of all interested genes simultaneously at different time points. Statistical methods for analyzing high-dimensional time course data are subject to active research currently. For example, clustering genes with similar expression profiles based on time course microarray data is particularly useful for identifying co-expressed genes as well as predicting functions of gene products with unknown functions. Several clustering methods have been developed to account for the temporal order of time course data including curve-based nonparametric clustering methods (James and Sugar, 2003; Luan and Li, 2004; Heard et al., 2006) and functional ANOVA methods (Ma et al., 2006; Ma and Zhong, 2008). The identification of differentially expressed genes under different conditions or treatments is another important research topic of time course gene expression data. For example, Yuan and Kendziorski (2006) employed a hidden Markov model, Storey et al. (2005) introduced a curve-based method, Tai and Speed (2009) used a multisample multivariate empirical Bayes method to rank microarray time course data, Chi et al. (2007) applied a Bayesian hierarchical modeling approach and Yao et al. (2005); Hong and Li (2006); Müller et al. (2008); Liu and Yang (2009) proposed functional data analysis approaches to identify temporally differentially expressed genes.

The time course gene expression data also allow investigators to study gene regulatory networks from a dynamic point of view. Currently, several models have been proposed for GRN construction, such as information theory models (Steuer et al., 2002; Stuart et al., 2003); Boolean networks (Kauffman, 1969; Thomas, 1973; Bornholdt, 2008); Bayesian Networks (Heckerman, 1996; Friedman et al., 2000; Hartemink et al., 2001; Imoto et al., 2003; Zou and Conzen, 2005; Needham et al., 2007; Werhli and Husmeier, 2007); vector autoregressive (VAR) and state space models (SSM) (Hirose et al., 2008; Shimamura et al., 2009; Kojima et al., 2009); and differential equation models (Voit, 2000; Holter et al., 2001; DeJong, 2002; Yeung et al., 2002). The information theory model is basically a correlation network and too simple to model a complex network. In addition, it does not accommodate multiple genes with dynamic expressions. The Boolean model is a discrete dynamic model in which the state of a gene is represented by only two states: on or off, so that the time course nature of gene expression profiles is not taken into account efficiently. The Bayesian network makes use of the Bayes’ rule by combining different types of data and prior knowledge. However, the challenge for establishing a Bayesian network comes from the high computational cost to optimize the network from a large number of combinatory candidates. The VAR and SSM models are discrete dynamic models which usually require equally-spaced and intensive time-series data in order to obtain reliable inference results for model parameters. In most cases, only the first-order and linear autoregressive model is used in practice due to the complications in computations and theoretical justifications although it can be theoretically extended to more general cases. Another problem is the identifiability problem embedded in the state space model. Hirose et al. (2008) made the assumption that the covariance matrix of the state equation is fixed as an identity matrix in order to make the model identifiable.

The focus of this paper is on the ordinary differential equation (ODE) modeling approach for the GRN. The ODE approach models the dynamic change of a gene expression (the derivative of the expression) as a function of expression levels of all related genes. So it is a directed network graph model and the dynamic feature of the GRN is automatically and naturally quantified. Both positive and negative as well as the feedback effects of gene regulations can be appropriately captured by the ODE model. A general ODE model for GRN can be written as:

$$\frac{d\mathit{X}}{dt}=\mathit{F}(\mathit{X},\mathit{\theta },t)$$ (1.1)

where X(t) = (x₁(t), …, x_n(t))^T is a vector representing the gene expression level of gene 1, …, n at time t. F serves as the link function that quantifies the regulatory effects of other genes on the expression change of gene i which depends on parameter θ. In general F can take any linear or non-linear function forms. However, the non-linear specification of F usually needs prior information on biological mechanisms and requires high computational cost, so that the nonlinear ODE model is only feasible for a small-scale network containing only a few to dozens of genes (Weaver et al., 1999; Sakamoto and Iba, 2001; Spieth et al., 2006). Many GRN models are based on linear ODEs due to its simplicity and usefulness in practical applications. However, we also recognize that the dynamics of gene expression may exhibit complex patterns, which may not be completely captured by a linear model. A simple linear ODE model can be written as

$$\frac{{dx}_i}{dt}=\sum _{j=1}^n{\theta }_{ij}{x}_j,i=1,\cdots ,n,$$ (1.2)

where parameters θ = {θ_ij}_{i,j=1,· · ·,n} quantify the regulation effects of the genes in the network. For a small-scale ODE-based GRN model (i.e. n is small), some standard statistical methods such as the standard least squares method or likelihood-based method can be used to perform statistical inference for the dynamic parameters θ from time course gene expression data. However, for a large-scale GRN ODE model that involves hundreds or even thousands of genes, the standard statistical methods may fail due to the curse-of-dimensionality.

We employ two methods to deal with the high-dimensional problem. The first one is the dimension reduction by clustering. We notice that many genes usually behave similarly during the experimental period, which makes it difficult to distinguish the expression patterns of these genes based on the time course microarray data. In this case, investigators have proposed clustering methods to group these similarly behaved genes (co-expressed genes) into functional modules (Luan and Li, 2004; Ma et al., 2006; Ma and Zhong, 2008). Consequently, our GRN model can be based on the functional modules instead of individual genes. Thus, the dimension of our ODE model can be significantly reduced. We can write the ODE model for functional modules as

$$\frac{{dM}_k}{dt}=\sum _{j=1}^p{\beta }_{kj}{M}_j,k=1,\cdots ,p$$ (1.3)

where p is the number of functional modules from clustering. In addition, from the sparseness principle (Arnone and Davidson, 1997), each gene or module may be regulated by only a few other genes or modules, i.e., most coefficients β_kj, that quantify the regulation effects, are zeros. The model selection approaches for ODE models can be used to determine the significance of parameters so that the dimension of the parameter space can be reduced significantly. We emphasize that the clustering and ODE model selection based on the sparseness principle and regularization are the two key steps for dimension reduction in the proposed GRN construction.

Another important feature of the proposed procedure is the use of mixed-effects modeling approach to deal with the data sparseness in several steps. In the first step, to cluster time course microarray data into functional modules, we adopt the nonparametric mixed-effects smoothing spline model under the framework of a mixture distribution proposed by Ma and Zhong (2008). In the second step, we employ the nonparametric mixed-effects model (NPME) smoothing approach to estimate the mean function and its derivative for each functional module. Then, we use a nonlinear mixed-effects modeling approach to refine the parameter estimates in the ODE models. The mixed-effects modeling approach allows us to borrow information across genes with similar expression patterns, which is more efficient.

In this article, we propose a roadmap to identify dynamic GRNs based on time course microarray data by combining a series of cutting-edge statistical techniques for ODE models and mixed-effects modeling approaches. In Section 2, we provide details of the proposed five-step procedure to identify the dynamic GRN. We applied the proposed method to the cell cycle gene regulatory network identification for yeast in Section 3. In Section 4 we present a simulation study to validate the key steps of the proposed procedure. We conclude the paper with some remarks in Section 5. The theoretical properties and a sketch of the proof for the SCAD estimator of the ODE model are given in the Appendix.

2. A Five-Step Procedure: CSIEF

In this section, we propose a roadmap for dynamic GRN identification with five steps. Step I) Clustering individual genes into functional modules: clustering time course microarray data into functional modules using mixed-effects nonparametric models with a mixture distribution (Luan and Li, 2004; Ma et al., 2006; Ma and Zhong, 2008). Step II) Smoothing the time course data to obtain the population (mean) expression pattern and its derivative for each of the functional modules independently. Step III) Identifying significant connections (regulations) among different functional modules: the two-stage estimation method for ODE models (Varah, 1982; Chen and Wu, 2008b,a; Liang and Wu, 2008), coupled with the smoothly clipped absolute deviation (SCAD) technique for high-dimensional variable selection (Fan and Li, 2001; Kim et al., 2008) is employed. Step IV) Refining the parameter estimates for the functional module-based GRN using stochastic approximation EM (SAEM) for mixed-effects ODE models (Delyon et al., 1999; Kuhn and Lavielle, 2005; Samson et al., 2006; Donnet and Samson, 2007). Step V) Performing function enrichment analyses for the identified GRN using Gene Ontology (GO). We coin this five-step procedure as the Clustering-Smoothing-Identification-Estimation-Function enrichment analysis (CSIEF) method. The roadmap for the CSIEF procedure is provided in Figure 1. The details for each step are given below. For convenience, we also provide a list of all notations in Table S1 of the supplementary materials.

Figure 1.

A roadmap for the CSIEF approach of dynamic GRN Construction

2.1. Step I: Clustering

It is important to cluster individual genes into groups or functional modules if the expression patterns of genes are similar during the experimental period. The concept of functional modules is crucial for understanding the proposed GRN construction strategy. It is not only that the expression profiles of the genes in the same module cannot be distinguished from the time course microarray data, but also the genes in the same module may share particular biological functions during the biological process. Thus, clustering individual genes into functional modules based on their expression patterns can help to reduce the dimension of the model, and is also biologically justifiable (Luan and Li, 2004; Ma et al., 2006; Ma and Zhong, 2008).

Several methods have been proposed to cluster time course data which include nonparametric smoothing-based approaches (James and Sugar, 2003; Luan and Li, 2004). Recently Ma et al. (2006) and Ma and Zhong (2008) successfully applied the nonparametric mixed-effects smoothing spline model under the framework of a mixture distribution to cluster time course gene expression data. We adopt this approach for the first step of our proposed procedure. Denote G_i = (g_i(t₁), · · ·, g_i(t_ni))^T as the expression data vector for the ith gene. We assume the following mixture normal distribution with p components (clusters) for G_i,

$${\mathit{G}}_{\mathit{i}}~w_1\mathcal{N}({\mu }_1({\mathit{T}}_{\mathit{i}}),{\mathrm{\sum }}_1)+w_2\mathcal{N}({\mu }_2({\mathit{T}}_{\mathit{i}}),{\mathrm{\sum }}_2)+\cdots ,+w_p\mathcal{N}({\mu }_p({\mathit{T}}_{\mathit{i}}),{\mathrm{\sum }}_p),i=1,\cdots ,n$$ (2.4)

where n is the total number of genes; w_k is the proportion of cluster k; μ_k(.) is the mean curve of cluster k; T_i = (t_i1, · · ·, t_i,ni)^T is the vector of measurement times for gene i and usually all genes share the same measurement times in the same experiment. Since different genes may have different expression patterns and because it is difficult to find a common parametric model for the time course expression profiles for all genes, a mixed-effects nonparametric smoothing splines approach is employed, i.e., a measurement model for G_i for the kth cluster can be written as

$${\mathit{G}}_{\mathit{i}}={\mu }_k({\mathit{T}}_{\mathit{i}})+{\mathit{Z}}_{\mathit{i}}{\mathit{b}}_{\mathit{i}}+{\mathit{\epsilon }}_{\mathit{i}},$$ (2.5)

where the random effect b_i quantifies the deviation of gene i from the mean curve μ_k of the kth cluster and is r × 1 vector following a mean zero normal distribution, (0, Ω_k). Z_i is n_i × r design matrix for the random effects. Measurement error ε_i is assumed to follow (0, σ²I) and independent of b_i. Thus, we have Σ_k = Z_iΩ_kZ_i^T + σ²I.

The maximum penalized likelihood approach can be used to estimate the parameters in models (2.4) and (2.5). Ma et al. (2006) also proposed a variation of the EM algorithm to estimate the cluster proportion parameters w_k and assign genes to appropriate clusters. To alleviate the computational cost and to stabilize the algorithm, the rejection-controlled EM algorithm is used to implement the proposed method. The number of clusters is determined by the BIC criterion. See more details for the implementation of this clustering procedure in Ma et al. (2006) and Ma and Zhong (2008).

2.2. Step II: Smoothing

From Step I, we can cluster all genes into functional modules. The expression pattern for all the genes within each module is similar. Thus, we can treat the time course data for all genes within the same module as longitudinal measurements of the expression level for the functional module. To facilitate the identification of significant regulations using the variable selection approach for ODE models in the next step, we need to estimate the mean curve and its derivative for each module independently. The mixed-effects modeling approach for the longitudinal data can be applied for this purpose. In this step, we employ a nonparametric mixed-effects (NPME) model as introduced in Wu and Zhang (2002, 2006) and by others to the data for each functional module to obtain the estimates of the mean expression and its derivative for the module.

Denote t_ij, i = 1, · · ·, n, j = 1, · · ·, n_i as the design time points and g_ki(t_ij) is the gene expression level at t_ij for gene i within the kth functional module, then we can write a nonparametric mixed-effects (NPME) model as

$$g_{ki}(t_{ij})=M_k(t_{ij})+V_{ki}(t_{ij})+{\epsilon }_k(t_{ij}),$$ (2.6)

where M_k(t) is the mean expression curve for the kth module and its derivative is denoted by ${dM}_k(t)/dt=M_k^{\prime }(t)$, and V_ki(t) are random-effects functions that quantify the departure of the ith gene expression level from the mean expression of the module M_k(t).

Over the past decade, many nonparametric smoothing techniques have been proposed to fit the NPME model, which include smoothing spline, regression spline, penalized spline, and local polynomial smoothing (Brumback and Rice, 1998; Wang, 1998; Gu, 2002; Gu and Ma, 2005; Wu and Zhang, 2002). We omit the details of these standard smoothing methods for the NPME model. A good review on these methods can be found in Wu and Zhang (2006). We can apply any of these smoothing techniques to obtain the estimates of M_k(t) and $M_k^{\prime }(t)$, k = 1, …, p.

2.3. Step III: Identification of Significant Regulations

In this step, we propose a method to identify significant regulations (coefficients) in model (1.3) using a high-dimensional variable selection technique, the smoothly clipped absolute deviation (SCAD) method (Fan and Li, 2001; Kim et al., 2008). First, following the similar idea from Chen and Wu (2008b,a) and Liang and Wu (2008), we plug the estimated mean expression curves and their derivatives, M̂_k(t) and ${\widehat{M}}_k^{\prime }(t)$, into the ODE model (1.3) to form a regression model,

$$y_k(t)=\sum _{j=1}^p{\beta }_{kj}{x}_j(t)+{\epsilon }_k(t),k=1,\cdots ,p;t=t_1,t_2,\dots ,t_N,$$ (2.7)

where $y_k(t)={\widehat{M}}_k^{\prime }(t)$ and x_j(t) = M̂_j(t). Error term ε_k(t) is the model approximation error due to the substitutions of the differential equation variables by their estimates from Step II. Since M̂_k(t) and ${\widehat{M}}_k^{\prime }(t)$ in the above model are estimated continuously at any time point t from Step II, we may not need to take t = t₁, t₂, …, t_N as the same time points as the original observation times. In fact, investigators (D’Haeseleer et al., 1999; Wessels et al., 2001; Bansal et al., 2006) have augmented the data for the above ODE-based regression model (2.7) in order to better estimate the ODE coefficients β_kj, where N is the number of time points for augmented data. We adopt a similar data augmentation strategy in this step.

If we treat model (2.7) as a standard high-dimensional linear regression model, many model selection methods are available for use, such as, stepwise and subset selection, penalized approaches including bridge selection (Frank and Friedman, 1993), least absolute shrinkage and selection operator (LASSO) (Tibshirani, 1996), smoothly clipped absolute deviation (SCAD) (Fan and Li, 2001) and so on. However, traditional approaches such as stepwise and subset selection methods estimate parameters separately from the model selection stage, which ignore stochastic errors inherited in the stage of model selection and their sampling properties are difficult to derive (Breiman, 1996). On the other hand, the recent work on penalized variable selection approaches such as LASSO and SCAD have demonstrated good properties and advantages to handle high-dimensional models.

The SCAD procedure has been paid great attention because of its desirable properties: unbiasedness, sparsity, and continuity (Fan and Li, 2001). Recently Kim et al. (2008) developed a new computational algorithm, coupled with the concave convex procedure (CCCP) (An and Tao, 1997), to implement the SCAD in an efficient way. This algorithm was confirmed to be less sensitive to the initial value, faster, more stable, and always guaranteed to converge to a minimum, and in addition the oracle property is still ensured in the case where the dimension of covariates is allowed to grow at a certain polynomial rate provided that the true model is sparse. These properties are not attainable for other methods such as LASSO (Meinshausen and Bühlmann, 2006; Zou, 2006). In this paper, we employ Kim et al.’s (2008) idea in our ODE model selection, although it is not trivial to establish the theoretical properties for our ODE model.

Consider the penalized objective function

$$C(\mathit{\beta })=\frac{1}{2N}\sum _{i=1}^N{\{y_k(t_i)-{\mathit{x}}_i^{\text{T}}\mathit{\beta }\}}^2+\sum _{j=1}^p{J}_{\lambda }(\mid {\beta }_j\mid ),$$ (2.8)

where J_λ(|β|) is the SCAD penalty proposed by Fan and Li (2001), which can be expressed as:

$$J_{\lambda }(\mid \beta \mid )=\{\begin{array}{ll}\lambda \mid \beta \mid ,\hfill & \text{if}\mid \beta \mid <\lambda ;\hfill \\ \frac{(a^2-1){\lambda }^2-{(\mid \beta \mid -a\lambda )}^2}{2(a-1)},\hfill & \text{if}\lambda \le \mid \beta \mid <a\lambda ;\hfill \\ \frac{(a+1){\lambda }^2}{2},\hfill & \text{if}\mid \beta \mid \ge a\lambda ,\hfill \end{array}$$ (2.9)

where a = 3.7. Note that here β = (β₁, β₂, …, β_p)^T and we drop the subscript k for notational simplicity, since we will apply the variable selection technique to each of the p modules separately.

The main idea of the CCCP-SCAD algorithm is to decompose (2.8) as the sum of the convex and concave functions, and then use the CCCP algorithm (An and Tao, 1997) for implementation. The key idea of the CCCP algorithm is to update the solution with the minimizer of the tight convex upper bound of the objective function obtained at the current solution. Thus, even though the objective function given in (2.8) is not convex, it can be decomposed into the sum of a convex function and a concave function, and then the CCCP algorithm can be applied. The most attractive feature of the CCCP-SCAD is that this algorithm is guaranteed to converge to the local minimum and has been shown to have the oracle property for high-dimensional linear models.

Note that J_λ(|β|) = J̃_λ(|β|) + λ|β| and |β| is a convex function, where

$${\stackrel{\sim }{J}}_{\lambda }(\mid \beta \mid )=-\lambda \mid \beta \mid I(\mid \beta \mid \ge \lambda )+\frac{(a^2-1){\lambda }^2-{(\mid \beta \mid -a\lambda )}^2}{2(a-1)}I(\lambda \le \mid \beta \mid <a\lambda )+\frac{(a+1){\lambda }^2}{2}I(\mid \beta \mid \ge a\lambda )$$

and J̃_λ(|β|) is a differentiable concave function. Thus C(β) in (2.8) can be rewritten as

$$\frac{1}{2N}\sum _{i=1}^N{\{y_k(t_i)-{\mathit{x}}_i^{\text{T}}\mathit{\beta }\}}^2+\sum _{j=1}^p{\stackrel{\sim }{J}}_{\lambda }(\mid {\beta }_j\mid )+\lambda \sum _{j=1}^p\mid {\beta }_j\mid ,$$

which is the sum of convex and concave functions. We apply the CCCP algorithm as follows. Given a current solution β^c, the tight convex upper bound is given as

$$Q(\mathit{\beta })=\frac{1}{2N}\sum _{i=1}^N{\{y_k(t_i)-{\mathit{x}}_i^{\text{T}}\mathit{\beta }\}}^2+\sum _{j=1}^p\nabla {\stackrel{\sim }{J}}_{\lambda }(\mid {\beta }_j^c\mid ){\beta }_j+\lambda \sum _{j=1}^p\mid {\beta }_j\mid .$$ (2.10)

We then update the current solution of β^c with the minimizer of (2.10). We minimize (2.10) by using the algorithm proposed by Rosset and Zhu (2007) as Kim et al. (2008) suggested, because Q(β) is a piecewise quadratic function.

Note that there is a closed-form solution to a linear ODE model, which is a summation of exponential terms (a nonlinear model). However, it will be difficult to estimate the parameters in a high-dimensional nonlinear model and very challenging if not impossible to perform model selection for a high-dimensional nonlinear model. The proposed two-step smoothing-based method transforms this nonlinear regression model into a linear regression model so that standard variable selection approaches for high-dimensional linear regression models can be applied. The computation is easier for the linear model and we can also show that the proposed SCAD estimator has an oracle property under our ODE setting. The theoretical results and the proof are provided in the Appendix.

2.4. Step IV: SAEM for Refining Parameter Estimates

In Step III, we can obtain the estimates of coefficients β_kj from model (2.7) while the variable selection is performed using the SCAD approach, which is another advantage of the penalized variable selection methods. However, these estimates are the so-called pseudo-least squares (PsLS) estimates which were investigated by Liang and Wu (2008). As noted by Liang and Wu (2008), the PsLS estimation method is computationally efficient and easy to implement for our variable selection purpose in Step III, but the PsLS estimates are not the most efficient in terms of estimation accuracy. This is because the estimates of the mean expression curves, in particular, their derivatives from Step II may inherit a large error if the data are sparse with a large noise. To overcome this problem, we invoke a nonlinear mixed-effects (NLME) modeling approach to refine the parameter estimates based on the selected model from Step III. The PsLS estimates from the previous step can be used as the initial estimates for the NLME estimates.

The genes in each of the functional module can be treated as “subjects”. Thus, the gene expression data for each module can be considered as a longitudinal data set. The mixed-effects ODE model for module k can be written as

$$\frac{{dx}_{ki}}{dt}=\sum _{j=1}^{m_k}{\beta }_{\mathit{kij}}{M}_{[kj]}(t),i=1,\cdots ,n_k;k=1,\dots ,p,$$ (2.11)

where x_ki is the gene expression level for the ith gene in the kth module, n_k is the number of genes in the kth module, M_[kj](t) are mean expression levels of the modules that have significant effects (non-zero coefficients determined by the SCAD method in Step III) on the kth module, m_k is the number of modules that have significant effect on the kth module. That is, we assume that the ith gene in the kth module is regulated by the mean effects of genes in other modules. To fit model (2.11) and obtain the maximum likelihood estimates of non-zero coefficients β_kij, we suggest to use the Stochastic Approximation EM (SAEM) algorithm (Delyon et al., 1999; Kuhn and Lavielle, 2005; Donnet and Samson, 2007).

Denote g_ki(t) as the expression measurement of gene i at time t in module k; the population parameters β_k = (β_k1, · · ·, β_kmk)^T for the kth module and individual parameters β_ki = (β_ki1, · · ·, β_kimk)^T. We have a longitudinal measurement model:

$$g_{ki}(t)=x_{ki}(t)+{\mathit{\epsilon }}_{ki}(t),$$ (2.12)

where the measurement error ε_ki(t) is assumed to follow an independent normal distribution with mean zero and variance σ². The between-gene variation can be modeled by random effects of parameters:

$${\mathit{\beta }}_{ki}={\mathit{\beta }}_k+{\mathbf{b}}_{ki},$$ (2.13)

where random effects b_ki characterize the deviation of individual genes from the population level of the module. We assume that b_ki ~ (0, D_k). Note that x_ki(t) is a solution after integrating ODE (2.11) separately for each module k = 1, · · ·, p, which requires M_[kj](t) to be known. We use the nonparametric estimates of M_[kj](t) from Step II. Since x_ki(t) in (2.12) is the solution to ODE (2.11) which is a nonlinear function of β_kij and M_[kj](t), the response variable g_ki(t) is nonlinearly related to β_kij and M_[kj](t). Thus, we are dealing with a nonlinear mixed effects (NLME) model in this step.

The SAEM approach is very efficient and generally applicable to obtain the maximum likelihood estimates (MLE) for the unknown parameters in mixed-effects ODE models from longitudinal data. Delyon et al. (1999) has shown that this algorithm converges very well under very general conditions. When the conditional distribution of random effects has no closed-form expression, a Markov chain Monte Carlo (MCMC) method can be used. Kuhn and Lavielle (2005) also established the convergence of the SAEM estimates to the MLE. Donnet and Samson (2007) applied this method to a dynamic ODE model where ODEs are solved by a local linearization scheme. Samson et al. (2006) extended the method to deal with censored data. However, the SAEM approach, similar to other likelihood-based methods for nonlinear models, requires supply of the initial values of the unknown parameters. If these initial values are too far from the true values, the computational algorithm may suffer a convergence problem or may converge to a local maximum, in particular for a high-dimensional parameter space. As aforementioned, the rough estimates of the unknown parameters in the model selection step (Step III) can be served as good starting values for our SAEM algorithm.

2.5. Step V: Function Enrichment Analysis

Based on the variable selection results from Step III and the refined estimates of non-zero coefficients in ODE models from Step IV, we can build the gene regulatory network (GRN) and interpret the results using function enrichment analyses (FEA). The FEA allows us to study what functions of the genes in each module typically perform. Annotating modules appropriately is critical for explaining the constructed GRN. Fortunately, the biological functions of many genes are known and documented in publicly accessible databases such as Gene Ontology Consortium (2000). Genes within a module may perform multiple biological functions. However, a certain function may be over-represented in a module compared to the population of genes in an organism or a biological process which consists of thousands of genes. Therefore, a module can be annotated by enriched function(s).The process of detecting over-represented functions in a module is called function enrichment analysis (Gene Ontology Consortium, 2000). A great deal of genes’ functions have been discovered for many model organisms. Such information provides a platform for measuring the degree of enrichment of certain function(s) in a module. The over-representation of a function in the module can be tested by a hypergeometric distribution (Sokal and Rohlf, 1995),

$$Pr(z\mid T,S,n_k\mid )=\left(\begin{array}{c}S\\ z\end{array}\right)\left(\begin{array}{c}T-S\\ n_k-z\end{array}\right)/\left(\begin{array}{c}T\\ n_k\end{array}\right)$$ (2.14)

where T is the population size, i.e., the total number of genes in an organism or an interested biological process; S is the number of genes that share the same function in the population; n_k is the number of genes clustered in the kth module; and z is the number of genes that share the same function in the module. This formula quantifies the probability of finding the over-represented function in a functional module. In fact, the enrichment test is similar to Fisher’s exact test in a 2 × 2 contingency table where the null hypothesis is that a particular function is not represented in a module more than that in the population. By summing over all the probabilities of the extreme or more extreme than the observed data from (2.14), the p-value can be obtained. Since all categories of gene functions are tested for the same set of genes, a correction is necessary to alleviate the false discovery rate in multiple tests. Bonferroni correction is a conservative way to control the error in this case. The over-represented function is then annotated for each module and the biological interactions of gene functions in a GRN can be interpreted.

3. Identification of Dynamic Gene Regulatory Network for Yeast Cell Cycle

3.1. Time Course Microarray Data for Yeast Cell Cycle-Regulated Genes

The first time course gene expression data set for yeast cell cycle regulated genes were collected by Spellman et al. (1998) and this data set has been analyzed using different approaches by many investigators (Friedman et al., 2000; Holter et al., 2001; DeJong, 2002; Yeung et al., 2002; Yao et al., 2005; Ma et al., 2006; Werhli and Husmeier, 2007). Cell cycle progression in Saccharomyces cerevisiae (yeast) is controlled in part by the sequential transcription of specific gene clusters whose expression peaks at different times during the cell cycle. Such regulation might be required for the proper functioning of mechanisms that maintain orders during cell divisions. Spellman et al. (1998) carried out a series of experiments aimed at objectively identifying all protein-encoding gene transcripts in the genome of Saccharomyces cerevisiae that are cell cycle regulated. They used DNA microarrays to analyze gene expression levels in cell cultures that had been synchronized by three independent methods. They analyzed the data by deriving a numerical score based on a Fourier algorithm (testing periodicity) and using a correlation function, and they found that 800 genes were cell cycle regulated which constitutes > 10% of all protein-coding genes in yeast genome. The gene expression levels were measured at 18 equally spaced time points during two cell cycles. In this paper, we intend to apply differential equation models and the proposed methods in the previous section to establish a dynamic regulatory network among functional modules (instead of individual genes) for the yeast cell cycle.

3.2. Application of the CSIEF Approach

We consider the 800 genes that were identified as cell cycle-regulated by Spellman et al. (1998). Among these 800 genes, many of them showed a similar expression pattern during the two-cell-cycle experiments. It is legitimate to apply the clustering method in Step I of the proposed CSIEF procedure to group these genes into functional modules. The nonparametric mixed-effects smoothing splines method with a mixture distribution (Ma et al., 2006; Ma and Zhong, 2008) was applied to the data and 41 functional modules were identified. The raw data for the 41 modules along with the fitted mean expression curves using the proposed approaches are presented in Figure 2. We can see that these functional modules differ from each other in terms of gene expression patterns, but all genes in the same module behave similarly.

Figure 2.

Yeast cell cycle-regulated gene expression data (dots) for individual modules with nonparametric smoothing estimates of mean expression curves (dashed lines) and the ODE fitted curves (solid lines) by the SAEM approach.

In the second step, we smoothed the data for each of the modules using the nonparametric mixed-effects smoothing splines technique again to obtain the estimates of the mean expression curves and their derivative curves for each module, separately. These estimates were plugged in model (2.7) to formulate a regression-like model in the third step, and then the SCAD approach (Step III of the CSIEF) was used to identify significant regulations (connections) among the 41 functional modules. The SAEM approach was applied to the identified sparse ODE model to refine the parameter estimates for each of the modules in Step IV. We obtained the fitted curves for all modules by integrating 41 ODEs concurrently with parameters estimated from the SAEM approach in Step IV. We overlayed the fitted mean expression curves with the raw data and the smoothed mean curves from Step II in Figure 2 (note that the initial values for the fitted ODE models are taken as M̂_k(0) from Step II which are smoothing estimates of M_k(0)). From this figure, we can see that the proposed method fits the data quite well and the fitted mean curves from ODE models (Step IV) for each module also coincide well with the smoothed mean curves from the smoothing splines approach in Step II. We notice that the proposed ODE models not only can fit the simple monotonic curves well, but also can fit complex and periodical curves quite reasonably without invoking any periodical functions explicitly.

Finally we applied the function enrichment analysis (Step V) to each of the 41 functional modules based on Gene Ontology Consortium (2000). The function enrichment analysis is implemented with the whole yeast genome as a population for annotating the modules. These results are reported in Table 1.

Table 1.

Yeast cell cycle gene regulatory network. For a particular module listed in Column 1, Column 4 provides the modules that have a significant effect (inward influence) on this module and Column 5 provides the modules that this particular module has a significant effect on (outward influence).

Module	# gene	Function Annotation	Inward Influence Modules	Outward Influence Modules
1	17	Nucleotide, nucleic acid transportation	9−, 15−, 18−, 30−
2	13	Translational elongation in cellular protein metabolic process	8−, 25−, 30−, 32, 41
3	8	Microtubule cytoskeleton organization	7, 9−, 14−, 18−, 25
4	34	DNA replication, repair, maintenance	9, 14, 32, 39, 41−
5	22	Carbohydrate transmembrane transportation	8, 9, 14, 18−, 32, 41
6	7	Ammonia-lyase catalytic activity	18−, 25, 30, 32−, 41−
7	7	Cell budding	18, 25−, 30−, 32, 41	3
8*	9	N/A	18−, 32, 41−	2−, 5, 18, 19, 24, 25−, 26, 27, 28,32, 34−,39, 40−
9*	4	Cell wall organization	14, 18−, 20, 32−, 41	1−, 3−, 4, 5, 10, 11, 13−, 18, 19, 20−, 21, 24, 25−, 26−, 27−, 28−, 31, 32−, 35, 37−, 40−, 41
10	37	Microtubule based process	9, 14, 18, 25−, 32, 41
11	18	Cell budding neck formation	9, 14, 18−, 41−
12	19	N/A	14−, 15, 41−
13	38	L-serine ammonia-lyase activity	9−, 14−, 25, 32, 41−
14*	1	Mating pheromone activity	14−, 15, 25−, 30−, 38, 41−	1−, 3−, 4, 5, 9, 10, 11, 12−, 13−, 14−, 15, 16, 17−, 19, 20−, 21−, 22−, 23−, 24, 26−, 29, 30, 31, 32, 33−, 35, 36−, 37−, 38, 39, 40−, 41
15*	11	Conjugation with cell fusion	14, 15−, 30, 32−, 41	1−, 12, 14, 15−, 17, 27, 29, 31−, 33, 34, 35−, 37−, 38−
16	8	Iron ion transmembrane transportation	14, 30−, 32−, 38−, 41
17	25	Intramolecular transferase activity, phosphotransferases	14−, 15, 25−, 30−, 41−
18*	11	Meiotic mismatch repair	8, 9, 38, 41−	1−, 3−, 5−, 6−, 7, 8−, 9−, 10, 11−, 13−, 20−, 21−, 22, 29−, 31−, 32, 33, 34−, 35, 37, 39−, 41
19	18	Manosyltransferase activity	8, 9, 14, 25−, 32,41−
20*	27	DNA replication	9−, 18−, 25, 41−	9, 22−, 25−, 36,
21	31	Telomere maintenance via DNA recombination; DNA helicase activity	9, 14−, 18−, 25, 34, 41−
22	24	Actomyosin contractile ring	14−, 18, 20−, 32−, 41−
23	6	Protein amino acid glycosylation	25−, 30−, 38, 41−
24	23	ER-associated protein catabolic process; cellular response to heat	8, 9, 14, 25−, 32, 41−
25*	25	Contract ring contraction involved in cell division	9−, 20−, 32, 41	2−, 3, 6, 7−, 10−, 12−, 13, 14−, 17−, 19−, 20, 21, 23−, 24−, 27−, 38, 40−, 41−
26	24	Chronological cell aging; response to pheromone	8, 9−, 14−, 30−, 32, 41
27	25	Sulfur amino acid biosynthesis, metabolism, transportation	8, 9−, 15, 25−, 41
28	39	Chromosome segregation; sister chromatid cohesion	8, 9−, 32, 38, 41−
29	19	Response to abiotic stimulus	14, 15, 18−, 30, 32
30*	15	Amino acid myristoylation in protein modification process	14, 32−, 38−	1−, 2−, 6, 7−, 12−, 14−, 15, 16−, 17−, 23−, 26−, 29, 31−, 32, 33−, 35, 36−, 37−, 38, 39−
31	18	Fungal-type cell wall biogenesis	9, 14, 15−, 18−, 30−, 41
32*	3	N/A	8, 9−, 14, 18, 30	2, 4, 5, 6−, 7, 8, 9−, 10, 13, 15−, 16−, 19, 20, 22−, 24, 25, 26, 28, 29, 30−, 33−, 34, 36, 40−
33	23	N/A	14−, 15, 18, 30−, 32−, 41
34	26	Glycine metabolism	9−, 15, 18−, 32, 38	21
35	29	Glucanosyltransferase activity	9, 14, 15−, 18, 30, 41
36	31	Methionine metabolism	14−, 20−, 30−, 32, 38
37	28	Nuclear migration along microtubule; chromatin silencing; negative regulation of transcription	9−, 14−, 15−, 18, 30−, 41
38*	17	Cell separation during cytokinesis	14, 15−, 25, 30, 41	4, 14, 16−, 18, 23, 28, 30−, 34, 36, 39
39	19	Response to DNA damage stimulus; DNA repair; DNA strand elongation during DNA replication	8, 14, 18−, 30−, 38
40	21	N/A	8−, 9−, 14−, 25−, 32−, 41−
41*	15	Nuclear chromatin assembly/disassembly	9, 14, 18, 25−, 41−	2, 4−, 5, 6−, 7, 8−, 9, 10, 11−, 12−, 13−, 14−, 15, 16, 17−, 18−, 19−, 20−, 21−, 22−, 23−, 24−, 25, 26, 27, 28−, 31, 33, 35, 37, 38, 40−, 41−

The superscript ‘−’ on the module numbers indicates the negative (inhibitory) effect (a negative coefficient in the ODE model), otherwise the effect is positive (activatory).

The superscript ^* indicates the modules that have an effect on multiple other modules.

3.3. Results and Biological Implications

First we notice that 41 functional modules were identified from Step I of the CSIEF procedure (Figure 2 and Table 1). The number of genes in each functional module ranges from 1 to 39 (Column 2 in Table 1). The function annotation results are listed in Column 3 in Table 1. The fourth and fifth columns in Table 1 list all the modules, identified by the SCAD method from Step III, that have significant impact on (inward influence) or are impacted by (outward influence) respectively. If a module effect is negative (inhibitory), we indicate it by a superscript ‘-’ in Columns 4 and 5. A list of the regulation parameter (ODE coefficient) estimates can be found in Table S2 of the supplementary materials.

Figure 2 shows that some of the 41 modules, such as Modules 3 and 14, exhibited a monotonic expression pattern although most of them exhibited a periodical pattern during the two cell cycles. As shown in Table 1, the function annotation suggests that the genes in the identified functional modules may participate in broad activities in a cell cycle such as DNA replication, repair and maintenance, protein synthesis and modification, nucleic acid transportation, nuclear chromatin assembly, cytoskeleton and cell wall organization, cell fusion and division etc..

We also note that the functions of five modules cannot be identified from the function annotation analysis(denoted as ‘N/A’ in ‘Function Annotation’ column in Table 1). But the genes in each of these five modules exhibited similar expression patterns, which indicate that the genes in these modules may share some common functions that have not been discovered to date. We would suggest that experimental biologists explore the genes in these five modules which may lead to new discoveries in cell cycle progression.

The findings from our module-based network modeling are biologically interesting. For instance, the dynamic expression of Module 1 was inhibited (negatively affected) by Modules 9, 15, 18 and 30. We noticed that Module 1’s function is related to nucleic acid transportation, which seems to be inhibited by modules with functions of cell wall organization (the function of Module 9), conjugation with cell fusion (the function of Module 15), meiotic mismatch repair (the function of Module 18), and amino acid myristoylation in protein modification process (the function of Module 30). This seems reasonable since the nucleic acid is transported for DNA synthesis. When a cell is ready to undergo fusion or division, DNA does not participate in such a process. In contrast, the pause of DNA synthesis may help alleviate the burden of accumulating materials and energy for cell fusion or division. In addition, the expression level of Module 1 is almost flat (Figure 2), which seems to be inhibited by some modules such as those identified from Step III of our CSIEF procedure (i.e., Modules 9, 15, 18 and 30). Biological interpretations of function annotations and module interactions for other modules are also interesting, but are omitted here due to space limitation.

One important feature of the identified GRN is that most modules are regulated by only a few other modules (ranging from 3 to 6 modules), which reflects the fact of sparseness of the network connections. However, all the 41 modules are connected via a few important modules that appear to be key players (‘hubs’) in the network. We found that 13 out of 41 modules had the function to regulate other modules. Among them, two modules (Modules 14 and 41) had effects on about 80% of all other modules; five modules (Modules 9, 18, 25, 30, and 32) regulated about 50% of all other modules; and three modules (Modules 8, 15, and 38) regulated about 25% of all other modules. In fact, all 41 modules were regulated and connected by these 10 key modules. In addition, one module (Module 20) regulated 4 other modules and two modules (Modules 7 and 34) regulated one other module by each. We also noticed that very few modules had a feedback loop (the module is regulated by itself). In fact, only two big ‘hubs’, Modules 14 and 41 as well as Module 15 had a negative feedback loop (Table 1).

For the two biggest ‘hubs’, Module 14 only contains one gene with a function of mating pheromone activity and Module 41 contains 15 genes with an annotated function of nuclear chromatin assembly/disassembly. This may suggest that knocking out the gene in Module 14 or some genes in Module 41 may have a significant effect on the yeast cell cycle-regulated gene network, which can be validated via future experiments. Also notice that, among the five modules whose functions were not identified from the GO function annotation (labeled as ‘N/A’ in Column 3 of Table 1), two of them (Modules 32 and 8) are among the second and third largest ‘hubs’ which had an effect on 50% and 25% of all other modules, respectively. This finding provides additional evidence for our suggestion that these 5 modules with unidentified functions are worth more attention from experimental biologists. Recent work published in Nature (Orlando1 et al., 2008) did knock out one gene, cyclin, in a similar yeast cell cycle-regulated gene study. They showed that cells lacking the gene cyclin are blocked at the G1/S border, however, nearly 70% of periodic genes continued to be expressed periodically and on schedule. This suggests that, although the cyclin gene has a function in the regulation of cell-cycle transcription, it is not solely responsible for establishing the global periodic transcription system. This result agrees well with our findings that the gene cyclin was clustered in Module 20 in our analysis and Module 20 is not a big hub although it affects 4 other modules in the network.

Transcription factors(TF) play key roles in regulating yeast gene transcription (Simon et al., 2001). By employing genome-wide binding/location analysis, investigators have already identified the interactions among TFs as well as interactions between TFs and other genes. Lee et al. (2002) identified the binding capability of all 106 TFs of the complete yeast genome which contains 6270 genes. They found that 37% of these genes are bound by one or more of the 106 TFs. We explored the interactions between the 21 cell cycle related TFs and the 800 cell cycle-regulated genes (41 modules). We found, on average, the 21 cell cycle related TFs are more likely to bind to the 13 key modules (‘hubs’) as we have identified, compared to other modules. In particular, among the identified biggest ‘hubs’, Module 41, was affected by >50% of the 21 TFs, followed by Module 14 regulated by more than 40% cell cycle related TFs. This fact suggests that the capability of genes in key modules to regulate other genes is probably due to active transcription binding of cell cycle-related TFs.

It is helpful to provide an intuitive visualization for the established GRN. We used R function ‘igraph’ to plot the yeast GRN in Figure 3. From this figure, we can clearly see that the large hubs, Modules 9, 14, 18, 25, and 41 are located in the center of the network. Some other important hubs including Modules 30, 8, 15, and 38 can also be easily identified from the graph.

Figure 3.

Graph of yeast cell cycle GRN formed by 41 modules. Each module is represented as a node. Arrow stands for the direction of influence.

4. Simulation Study: Evaluation of ODE Variable Selection and Parameter Estimation

The proposed dynamic GRN construction procedure (CSIEF) is a complicated process involving five steps. In this section, we design simulation experiments to validate some of the key steps such as the ODE variable selection step (Step III) and ODE parameter estimation step (Step IV). The statistical methods in other steps have been well studied in statistical literature and will not be investigated here. We design our simulation study based on the yeast data application in the previous section. We consider a ODE network model with 41 variables (equations) which is the same as the module-based ODE model (1.3). The estimated ODE model parameters (coefficients) are used as the true values in our simulation study (see Table S2 in the supplementary materials). Note that for each of the 41 ODEs, there are 3 to 6 non-zero coefficients and 35 to 38 zero coefficients so that we are mimicking the scenarios of the yeast data application in the previous section. The initial values of the ODE model are taken from the nonparametric smoothing estimates of the mean expression curve M_k(t) at t = 0 for the kth module from Step II in the yeast data analysis. We simulated the experimental data, g_ki(t) at different time points by solving the ODEs numerically and then adding measurement errors using the following observation model,

$$g_{ki}(t_j)=M_k(t_j)+b_{ki}+e_{ki}(t_j)k=1,\cdots ,41;j=1,\cdots ,N;i=1,\cdots ,n_k,$$ (4.15)

where n_k is the number of genes in the kth module which is set according to Table 1 (Column 2); b_ki is the random coefficient of gene i in the kth module to quantify the departure of gene i from the mean expression of Module k, which is assumed to follow a standard normal distribution. The measurement error e_ki(t_j) is assumed to follow a normal distribution with mean zero and variance δ² and we simulated the data for two variance levels, δ² = 0.05 and 0.5. In order to compare the performance of the proposed method for different sample sizes, we took the number of measurements N = 20, 100 and 300 with equally spaced time points for each gene respectively. The number of simulation runs is taken as N_r = 100.

We applied the SCAD method (Step III) to perform variable selection for the simulated data. Table 2 shows that the SCAD method can correctly select the variables for about 32% of the worst cases (the largest variance and the smallest sample size) and 79% of the best case scenarios (the smallest variance and largest sample size). For other cases, the performance of the SCAD is in between the above two scenarios as we expected. In general, the SCAD approach tends to select more variables (over-fitted). We believe that this problem can be fixed using the adaptive LASSO or other related approaches that penalize more on over-fitted models, but the detailed exploration of this is beyond the scope of this paper.

Table 2.

Simulation results: the SCAD variable selection and comparisons of parameter estimation between the SCAD estimates (Step III) and the SAEM estimates (Step IV), N_r = 100 simulation runs.

δ2	N	Under-fitted(%)	Correctly fitted(%)	Over-fitted(%)	SCAD Estimates		SAEM Estimates
					ARE(%)	SD	ARE(%)	SD
0.5	20	13	32	55	27.7	9.4	15.2	10.3
	100	20	48	32	16.9	8.3	11.4	8.1
	300	0	61	39	11.4	6.2	6.9	4.5
0.05	20	8	44	48	16.4	10.3	10.9	12.0
	100	0	57	43	9.9	10.1	6.4	4.2
	300	0	79	21	6.4	5.9	3.3	2.7

To evaluate the performance of the ODE parameter estimation from Step III and Step IV, we calculated the average relative error (ARE) for all the parameter estimates in model (1.3), which is defined as

$$\text{ARE}=\frac{1}{{qN}_r}\sum _{i=1}^{N_r}\sum _{j=1}^q\frac{\mid {\beta }_{kj}-{\widehat{\beta }}_{kj}^{(i)}\mid }{\mid {\beta }_{kj}\mid }\times 100\%$$ (4.16)

where ${\widehat{\beta }}_{kj}^{(i)}$ is the estimate of parameter β_kj in the kth module at the ith simulation run, q is the number of non-zero coefficients in the model, and N_r is the number of simulation replicates (N_r = 100 in our simulation study). We take the average of the AREs for all the parameter estimates in all 41 equations to compare the estimates from Step III and Step IV.

From Step III of the proposed CSIEF method, rough estimates of ODE parameters can be obtained via the smoothing-based estimation method (Liang and Wu, 2008) while the variable selection is performed simultaneously. In Step IV, these estimates are refined using the SAEM method to fit a nonlinear mixed-effects (NLME) model. We applied these methods to the simulated data to evaluate their performance. Table 2 shows the average estimation error for all parameters in model (1.3). The AREs range from 6% to 28% for the smoothing-based SCAD estimates (Step 3) while the AREs are reduced to 3–15% for the SAEM estimates (Step IV). This demonstrates that the SAEM parameter estimation refinement step is necessary. In summary, the SCAD-based variable selection in Step III works quite well for ODE models. The ODE parameter estimates are reasonable and the SAEM estimation refinement in Step IV is necessary.

5. Concluding Remarks

We proposed a roadmap to construct a dynamic gene regulatory network (GRN) based on ordinary differential equation (ODE) models. Statistical inference methods for ODE models and mixed-effects modeling approaches as well as nonparametric smoothing and clustering techniques for repeated measurement data were used to deal with the high-dimensional problem and complex GRN. We successfully applied the proposed method to construct a dynamic GRN for yeast cell cycle-regulated genes. The 800 cell cycle-regulated genes were clustered into 41 functional modules. The functional module-based regulatory network was established. With the help from functional enrichment analysis, we were able to annotate the modules with specific functions. We described the extensive interactions among 41 functional modules and we suggest that some biological implications from our findings are worth further exploration by experimental biologists. We also justified and validated the key methods of the proposed procedure via investigating their theoretical properties and via Monte Carlo simulations.

One limitation of the proposed ODE models for a dynamic GRN is that only linear ODEs are considered here due to the high computational cost and complexity of the variable selection for nonlinear ODEs. The extension of linear ODE models to nonlinear ODE models for dynamic GRN is conceptually straightforward, but the difficulties in computation and variable selection as well as the nonlinear ODE parameter estimation may not be easily overcome. More work in this direction is warranted. Alternative models for dynamic GRN include boolean network, Bayesian network, and hidden Markov models among others. It is interesting to compare these modeling approaches with the proposed ODE model method from both perspectives of computational cost and likelihood to recover the true network. But the computational cost for this comparison is beyond the limit of our current computational power. In Step III, we employed a smoothing-based two-stage estimation method for ODE model parameters (Liang and Wu, 2008) in order to simplify the SCAD-based variable selection for ODE models. Some other more efficient estimation approaches for ODE models such as the nonlinear least squares method (Xue et al., 2010) and generalized profiling approach (Ramsay et al., 2007) are good for ODE parameter estimation, but these methods may not be easily able to handle the variable selection, and thus were not used here. We believe that it is worth future exploration along a similar line.

Gene regulation is a complex dynamic process. In addition, not only genes but also proteins, RNAs, macro or micro molecules play important roles in this process. In theory, the proposed ODE-based network modeling techniques are applicable to the complex network involving more and different biological elements and entities if these data are available, which may help us to reveal more detailed mechanisms of the gene regulatory network in the future.

APPENDIX: ASYMPTOTIC RESULTS AND PROOF

For ease of presentation, we consider a local polynomial estimation for M(t) and M′(t) from Step II. It is obvious that the same results apply for splines as well because splines and kernels are asymptotically equivalent in some sense.

Let T_p,t be an n × (p + 1) design matrix, whose (i, j) entry is (t_i − t)^j−1 for p = 1 or 2, and W_t = diag{K_h(t₁ − t), …, K_h(t_N − t)} be an N × N diagonal matrix of kernel weights, where K_h(· ) = K(·/h)/h with K(·) being a symmetric kernel function, and h being a proper bandwidth. As a consequence, the estimators M̂(t) and M̂′(t) can be expressed as

$$\begin{array}{l}\widehat{M}(t)={\mathit{\xi }}_1^{\text{T}}{({\mathbf{T}}_{1,t}^{\text{T}}{\mathbf{W}}_t^{1/2}{\mathrm{\sum }}_1^{-1}{\mathbf{W}}_t^{1/2}{\mathbf{T}}_{1,t})}^{-1}{\mathbf{T}}_{1,t}^{\text{T}}{\mathbf{W}}_t^{1/2}{\mathrm{\sum }}_1^{-1}{\mathbf{W}}_t^{1/2}\mathbb{Y}\\ {\widehat{M}}^{\prime }(t)={\mathit{\xi }}_2^T{({\mathbf{T}}_{2,t}^{\text{T}}{\mathbf{W}}_t^{1/2}{\mathrm{\sum }}_2^{-1}{\mathbf{W}}_t^{1/2}{\mathbf{T}}_{2,t})}^{-1}{\mathbf{T}}_{2,t}^{\text{T}}{\mathbf{W}}_t^{1/2}{\mathrm{\sum }}_2^{-1}{\mathbf{W}}_t^{1/2}\mathbb{Y},\end{array}$$

respectively, where ξ₁ is the 2 × 1 vector having 1 in the first entry and zero in the 2nd entry, while ξ₂ is the 3 × 1 vector having 1 in the 2nd entry and zeros in the other entries; ${\mathrm{\sum }}_j={\mathbf{W}}_t^{1/2}{\mathbf{T}}_{j,t}{\mathbf{DT}}_{j,t}^{\text{T}}{\mathbf{W}}_t^{1/2}+\mathit{diag}({\mathbf{R}}_1,\cdots {\mathbf{R}}_n)$ for j = 1, 2, with D and R_i being the covariance matrices of the random-effect and measurement error terms (See Wu and Zhang (2006) for a detailed description), and is defined below.

Note that M̂′(t) is actually the slope of the local quadratic fit. Since model (2.7) is not a standard linear regression model, the properties of the SCAD estimator may need to be studied carefully for this ODE-based regression model. Here we study the oracle property of the SCAD estimator for model (2.7).

Let = (x₁, …, x_N )^T for x_j = {x₁(t_j), …, x_p(t_j)}^T, = (Y₁, …, Y_N)^T for Y_j = {y₁(t_j), …, y_p(t_j)}^T, = (M₁, …, M_N )^T for M_j = {M₁(t_j), …, M_p(t_j)}^T, j = 1, …, N. For the sparse model, we consider a situation where most of the regression coefficients are exactly 0. Without loss of generality, we assume that the first q regression coefficients are nonzero and the remaining p – q regression coefficients are 0. Let = ( , ), where is the first N × q submatrix and is the last N × (p − q) submatrix of . Similarly, we write β = (β₍₁₎, β₍₂₎). Let C = /N, C = /N, $C_{\mathbb{M}}^{(i,j)}={\mathbb{M}}_{(i)}^{\text{T}}{\mathbb{M}}_{(j)}/N$ and $C_{\mathbb{X}}^{(i,j)}={\mathbb{X}}_{(i)}^{\text{T}}{\mathbb{X}}_{(j)}/N$ for i, j = 1, 2. In what follows, we denote λ_max(A) and λ_min(A) as the largest and smallest eigenvalues of matrix A, respectively, and A_,j and A_j, the jth column and row of a given matrix A. In addition, in the following theoretical development we express p_N = p and q_N = q to emphasize that p and q depend on sample size N. We now prove that the oracle estimator is asymptotically a local minimum of C(β). Here the oracle estimator is defined by β̂_o = (β̂_(1)o, 0₍₂₎), where β̂_(1)o is the minimizer of ${\left|\right|{\widehat{\mathbb{M}}}^{\prime }-{\mathbb{X}}_{(1)}{\beta }_{(1)}\left|\right|}_2^2$, 0₍₂₎ is (p_N – q_N )-dimensional 0 vector, and ||·||₂ is the standard Euclidean norm on R^N. We assume the following regularity conditions:

Assumption A

1. The function M⁽³⁾(t) is continuous on [0, 1].

2. The kernel function K is symmetric about zero and is supported on [−1, 1].

3. The bandwidth h = h_N = N^−1/7a_N is a sequence satisfying h → 0 as N → ∞, where a_N is a sequence tending to 0 slower than log⁻¹ N.

4. t_i are iid and have a common compact support and their density function, f (t), is bounded away from zero and has bounded and continuous second derivatives.

Assumption B

1. There exists a positive constant D₁ such that ${\mathbb{M}}_{,k}^{\text{T}}{\mathbb{M}}_{,k}/N\le D_1$ for all k = 1, …, p_N.

   There exists a positive constant D₂ such that λ_min(C) ≥ D₂.

   q_N = O(N^c₁) for some 0 < c₁ < 1.

   There exist positive constants c₂ and D₃ such that c₁ < c₂ ≤ 1 and N^(1−c₂)/2 min_1≤j≤qN|β_j| ≥ D₃.

The foregoing regularity assumptions (B) were used by Zhao and Yu (2006) to prove the model selection consistency of the LASSO estimator. The following proposition is the main result.

Proposition 1

Assume that E(e_i)^2κ < ∞ for an integer κ > 0. Let (λ_N ) be the set of local minima of (2.8) with the SCAD penalty and a regularization parameter λ_N. Under Assumptions A and B, we have Pr{β̂_o ∈ (λ_N )} → 1 as N → ∞ provided that λ_N = o(N^{−(1−c₂+c₁)/2}) and $p_N/{(\sqrt{N}{\lambda }_N)}^{2\kappa }\to 0$.

Before we prove the main result, we state the following preliminary fact under Assumptions A and B and its proof can be found in Wu and Zhang (2006):

$$\underset{t}{sup}\mid {\widehat{M}}^{\prime }(t)-M^{\prime }(t)\mid =O_p(b_N)\text{and}\underset{t}{sup}\mid \widehat{M}(t)-M(t)\mid =O_p(b_N),$$ (A.17)

where b_N = o(N^−1/4a_N ) with a_N given in Assumption A (iii). Note that (A.17) and Assumption B(ii) indicate that, for large enough N, λ_min(C) ≥ .5D₂.

Proof of Proposition 1

The proof of Proposition 1 can be completed following the spirit of Theorem 1 of Kim et al. (2008). However, the proof for our case is more complex due to the fact that the SCAD estimator for the ODE model is not based on the observed iid data, instead it is derived from nonparametric smoothing estimates of the functional curves and their derivatives. Here only the key steps that are different from Kim et al.’s (2008) proof are given. Details are omitted due to space limit, but can be obtained from the authors.

Let $L_{\lambda }(\mathit{\beta })=\frac{1}{2N}{\sum }_{i=1}^N{\{y_k(t_i)-{\mathit{x}}_i^{\text{T}}\mathit{\beta }\}}^2+{\sum }_{j=1}^p{\stackrel{\sim }{J}}_{\lambda }(\mid {\beta }_j\mid )$. Note that ∂L_λ(β)/∂β_j is

$$\{\begin{array}{ll}{S}_j(\mathit{\beta })-\lambda \text{sign}({\beta }_j)\hfill & \text{if}\mid {\beta }_j\mid \ge a\lambda ,\hfill \\ S_j(\mathit{\beta })+\frac{a\lambda -\mid {\beta }_j\mid }{(a-1)}\text{sign}({\beta }_j)-\lambda \text{sign}({\beta }_j)\hfill & \text{if}\lambda \le \mid {\beta }_j\mid <a\lambda ,\hfill \\ S_j(\mathit{\beta })\hfill & \text{if}\mid {\beta }_j\mid \le \lambda ,\hfill \end{array}$$

where $S_j(\mathit{\beta })=-\frac{1}{N}{\sum }_{i=1}^N\{y_k(t_i)-{\mathit{x}}_i^{\text{T}}\mathit{\beta }\}{x}_j(t_i)$. By the second-order sufficiency of the Karush-Kuhn-Tucker condition (see, e.g., Bertsekas, 1999, pp.320), any β to satisfy

$$S_j(\mathit{\beta })=0\text{and}\mid {\beta }_j\mid \ge a\lambda \text{for}j=1,\dots ,q_N,\text{and}$$ (A.18)

$$\mid S_j(\mathit{\beta })\mid \le \lambda \text{and}\mid {\beta }_j\mid \le a\lambda \text{for}j=q_N+1,\dots ,p_N$$ (A.19)

is an element of A(λ). Thus it suffices to show that β̂_o satisfies (A.18) and (A.19) with λ = λ_N.

Note that S_j(β̂_o) = 0 for j ≤ q_N holds trivially by the definition of the oracle estimator. So for proving (A.18), it suffices to prove that as N → ∞,

$$Pr(\mid {\widehat{\beta }}_{jo}\mid \ge a{\lambda }_N\text{for}j=1,\dots ,q_N)\to 1$$ (A.20)

Let ${\mathit{\xi }}_{2,t}={\mathit{\xi }}_2^{\text{T}}{({\mathbf{T}}_{2,t}^{\text{T}}{\mathbf{W}}_t^{1/2}{\mathrm{\sum }}_2^{-1}{\mathbf{W}}_t^{1/2}{\mathbf{T}}_{2,t})}^{-1}{\mathbf{T}}_{2,t}^{\text{T}}{\mathbf{W}}_1^{1/2}{\mathrm{\sum }}_2^{-1}{\mathbf{W}}_t^{1/2}$ and ${\mathbf{\Xi }}_2={({\mathit{\xi }}_{2,t_1}^{\text{T}},\dots ,{\mathit{\xi }}_{2,t_N}^{\text{T}})}^{\text{T}}$. Note that {M̂′(t₁) − M′(t₁), · · ·, M̂′(t_N ) − M′(t_N )}^T = Ξ₂ − + Ξ₂e and ${\widehat{\mathit{\beta }}}_{(1)o}=\frac{1}{N}{(C_{\mathbb{X}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}{\widehat{\mathbb{M}}}^{\prime }$. A direct calculation yields ${\mathbb{X}}_1^{\text{T}}{\widehat{\mathbb{M}}}^{\prime }={\mathbb{X}}_{(1)}^{\text{T}}({\mathbf{\Xi }}_2\mathbb{M}-{\mathbb{M}}^{\prime })+{\mathbb{X}}_{(1)}^{\text{T}}({\mathbb{M}}_{(1)}-{\mathbb{X}}_{(1)}){\mathit{\beta }}_{(1)}+{\mathbb{X}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2\mathit{\epsilon }+{\mathbb{X}}_{(1)}^{\text{T}}{\mathbb{X}}_{(1)}{\mathit{\beta }}_{(1)}$. It follows that

$${\widehat{\mathit{\beta }}}_{(1)o}-{\mathit{\beta }}_{(1)}=\frac{1}{N}{(C_{\mathbb{X}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}({\mathbf{\Xi }}_2\mathbb{M}-{\mathbb{M}}^{\prime })+\frac{1}{N}{(C_{\mathbb{X}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}({\mathbb{M}}_{(1)}-{\mathbb{X}}_{(1)}){\mathit{\beta }}_{(1)}+\frac{1}{N}{(C_{\mathbb{X}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2\mathit{\epsilon }.$$ (A.21)

We now deal with the three terms on the right-hand side of (A.21) separately. Let ${\mathbf{H}}_{(1)}^{\text{T}}={({\mathbf{h}}_{(1)1},\dots ,{\mathbf{h}}_{(1)q_N})}^T={(C_{\mathbb{X}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}/\sqrt{N}$ and ${(S_{(1)1},\cdots ,S_{(1)q_N})}^{\text{T}}={\mathbf{H}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2$. It follows that

$$\begin{array}{l}{\left|\right|S_{(1)j}\left|\right|}^2=S_{(1)j}^{\text{T}}{S}_{(1)j}={(C_{\mathbb{X}}^{(1,1)})}_{j,}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2{\mathbf{\Xi }}_2^{\text{T}}{\mathbb{X}}_{(1)}{(C_{\mathbb{X}}^{(1,1)})}_{,j}^{-1}/N\\ \le {\lambda }_{max}({\mathbf{\Xi }}_2{\mathbf{\Xi }}_2^{\text{T}}){\lambda }_{max}\{{(C_{\mathbb{X}}^{(1,1)})}^{-1}\}.\end{array}$$

Recall λ_min(C) ≥ .5D₂. We know that ${\lambda }_{max}\{{(C_{\mathbb{X}}^{(1,1)})}^{-1}\}\le 2/D_2$. On the other hand, using the arguments similar to the proof of Theorem 1 in Liang and Wu (2008) yields that ${\lambda }_{max}({\mathbf{\Xi }}_2{\mathbf{\Xi }}_2^{\text{T}})=O_p(1)$. These statements indicate that

$${\left|\right|S_{(1)j}\left|\right|}^2<C$$ (A.22)

for some positive constant C. Note that ${\mathbf{H}}_{(1)}^{\text{T}}{\mathbf{H}}_{(1)}={(C_{\mathbb{X}}^{(1,1)})}^{-1}$. In addition, ${\left|\right|{\mathbf{H}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2({\mathbb{X}}_{(1)}-{\mathbb{M}}_{(1)})\left|\right|}^2={({\mathbb{X}}_{(1)}-{\mathbb{M}}_{(1)})}^{\text{T}}{\mathbf{\Xi }}_2^{\text{T}}{\mathbf{H}}_{(1)}{\mathbf{H}}_{(1)}^{\text{T}}{\mathbf{\Xi }}_2({\mathbb{X}}_{(1)}-{\mathbb{M}}_{(1)})$. (A.17) indicates that each element of − is o(N^−1/4). An argument similar to the proof of (A.22) yields that each element of ${\mathbf{H}}_{(1)}^{\text{T}}({\mathbb{M}}_{(1)}-{\mathbb{X}}_{(1)})$ is of order o(N^−1/4). In the same way, we can prove that each element of ${\mathbf{H}}_{(1)}^{\text{T}}({\mathbf{\Xi }}_2\mathbb{M}-{\mathbb{M}}^{\prime })$ is of order o(N^−1/4) as well. So the first two terms on the right-hand side of (A.21) are order o(N^−1/4). Note that |β̂_jo| ≥ |β_j| − |β̂_jo − β_j|. Because min_j≤qN|β_j | = O(N^{−(1−c₂)/2}) and λ_N = o(N^{−(1−c₂+c₁)/2}), it suffices to show that max_1≤j≤qN|β̂_jo − β_j| = o_p(N^{−(1−c₂)/2}). Let $z_j=\sqrt{N}({\widehat{\beta }}_{jo}-{\beta }_j)$. It is equivalent to proving max_j≤qN|z_j | = o_p(N^c₂)/2). Using the arguments similar to the proof of Theorem 1 in Kim et al. (2008), we can easily show that, for any η > 0,

$$Pr(\mid z_j\mid >\eta N^{c_2/2}\text{for}\text{some}j=1,\dots ,q_N)\to 0,$$

and we complete the proof of (A.18) for the oracle estimator.

To show (A.19), we have |β̂_jo| ≤ λ_N for j > q_N, because β̂_jo = 0 by definition. Thus it suffices to show that

$$Pr(\mid S_j({\widehat{\mathit{\beta }}}_o)\mid >{\lambda }_N\text{for}\text{some}j=1+q_N,\dots p_N)\to 0.$$ (A.23)

Note that {S_j(β̂_o), for some j = 1 + q_N, …, n} can be expressed as

$$-\frac{1}{N}{\mathbb{X}}_{(2)}^{\text{T}}\left\{\mathbb{I}-{\mathbb{X}}_{(1)}\frac{1}{N}{({C_{\mathbb{X}}}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}\right\}\{({\mathbf{\Xi }}_2\mathbb{M}-{\mathbb{M}}^{\prime })+({\mathbb{X}}_{(1)}-{\mathbb{M}}_{(1)}){\mathit{\beta }}_{(1)}+{\mathbf{\Xi }}_2\mathit{\epsilon }\}.$$

Using the arguments similar to the proof for (A.18) yields that the first two terms are of order o(N^−1/4). Thus we are going to deal with

$$-\frac{1}{N}{\mathbb{X}}_{(2)}^{\text{T}}\left\{\mathbb{I}-{\mathbb{X}}_{(1)}\frac{1}{N}{({C_X}^{(1,1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}\right\}{\mathbf{\Xi }}_2\mathit{\epsilon },$$

which is denoted by ${\mathbf{h}}_{(2)j}^{\text{T}}{\mathbf{\Xi }}_2\mathit{\epsilon }$, for some j = 1 + q_N, …, N, where h_(2)j is the j – q_N column vector of H₍₂₎ and ${\mathbf{H}}_{(2)}^{\text{T}}=C_{\mathbb{X}}^{(2,1)}{(C_{\mathbb{X}}^{(1,1)})}^{-1}\frac{1}{\sqrt{N}}{\mathbb{X}}_{(1)}^{\text{T}}-\frac{1}{\sqrt{N}}{\mathbb{X}}_{(2)}^{\text{T}}$. Note that ${\mathbf{H}}_{(2)}^{\text{T}}{\mathbf{H}}_{(2)}={\mathbb{X}}_{(2)}^{\text{T}}\{\mathbb{I}-{\mathbb{X}}_{(1)}{({\mathbb{X}}_{(1)}^{\text{T}}{\mathbb{X}}_{(1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}\}{\mathbb{X}}_{(2)}/N$, and all eigenvalues of $\mathbb{I}-{\mathbb{X}}_{(1)}{({\mathbb{X}}_{(1)}^{\text{T}}{\mathbb{X}}_{(1)})}^{-1}{\mathbb{X}}_{(1)}^{\text{T}}$ are between 0 and 1, we have that ||h_(2)j|| ≤ D₁ for all j = q_N + 1, …, p_N. Recall that ${\lambda }_{max}({\mathbf{\Xi }}_2{\mathbf{\Xi }}_2^{\text{T}})=O_p(1)$. Thus $E{\{\sqrt{N}{S}_j({\widehat{\mathit{\beta }}}_o)\}}^{2\kappa }<\infty$ and $Pr\{\mid \sqrt{N}{S}_j({\widehat{\mathit{\beta }}}_o)\mid >t\}=O(t^{-2\kappa })$. As a consequence, we obtain (A.23), and thus complete the proof of Proposition 1.