Independence screening for high dimensional nonlinear additive ODE models with applications to dynamic gene regulatory networks

Abstract

Mechanism-driven low-dimensional ordinary differential equation (ODE) models are often used to model viral dynamics at cellular levels and epidemics of infectious diseases. However, low-dimensional mechanism-based ODE models are limited for modeling infectious diseases at molecular levels such as transcriptomic or proteomic levels, which is critical to understand pathogenesis of diseases. Although linear ODE models have been proposed for gene regulatory networks (GRNs), nonlinear regulations are common in GRNs. The reconstruction of large-scale nonlinear networks from time-course gene expression data remains an unresolved issue. Here, we use high-dimensional nonlinear additive ODEs to model GRNs and propose a 4-step procedure to efficiently perform variable selection for nonlinear ODEs. To tackle the challenge of high dimensionality, we couple the 2-stage smoothing-based estimation method for ODEs and a nonlinear independence screening method to perform variable selection for the nonlinear ODE models. We have shown that our method possesses the sure screening property and it can handle problems with non-polynomial dimensionality. Numerical performance of the proposed method is illustrated with simulated data and a real data example for identifying the dynamic GRN of Saccharomyces cerevisiae.

1 |. INTRODUCTION

Although mechanism-based ordinary differential equation (ODE) models have been widely used to describe dynamics of infectious agents in a host and epidemics of transmission between hosts for infectious diseases, data-driven high-dimensional ODE models for infectious diseases, in particular, at molecular levels, are still sparse. For example, the gene regulatory network (GRN), governing the gene transcriptions, is a collection of interacting genes and their products in the cell. It plays a fundamental role in the development of living cells and host response to infectious agents. Recent advancements in modern high-throughput technologies, such as DNA microarray and next generation RNA-Seq, have allowed the collection of time course gene expression profiles in an affordable cost. It is of increasing interest to reconstruct the GRNs from such experimental data using mathematical models, especially dynamic network models that aim to capture the complex phenomena of biological systems and host response to infections by modeling the time course gene expression data. Commonly used network models include information theory models,^1,2 Boolean networks,^3,4 Bayesian networks,^5–8 vector autoregressive and state space models,^9–11 latent variable models,¹² and differential equation models.^13–17 Recently, Xing and his associates have introduced temporal exponential random graph models and time-varying networks to capture dynamics of networks.^18–20 Excellent overviews on diverse data-driven modeling schemes and related topics can be found in De Jong,²¹ Filkov,²² and Hecker et al.²³

We focus on the ODE modeling approach, which offers a description of the gene network as a continuous time dynamical system. Gene regulation is modeled by rate equations, quantifying the change rate (derivative) of the expression of one gene in the system as a function of expression levels of all related genes. The general form of the equations can be written as

$${\mathit{X}}^{\prime }(t)=\mathit{F}(t,\mathit{X}(t),\mathit{\theta }),$$ (1)

Where X(t) = {X ₁(t),…, X_p(t)}^T is a vector representing the gene expression level of the genes 1,…,p at time t, t ∈ [t₀, T], 0 ≤ t₀ < T < ∞, and X^′(t) is the first-order derivative of X(t). The link function F describes the regulatory effects of other genes on the expression change of gene i, i = 1,…,p, with a vector of parameters θ.

The identification of the parameters in (1) requires some constraints on the model structure, for instance, specification of the form of the function F. Many previous works have assumed that F is linear, due to the simplicity of linear functions.^13,14,16,17 However, complex dynamic behaviors and regulations in a GRN usually cannot be explained by simple linear systems.²⁴ A number of nonlinear ODE models have been proposed for GRNs, such as the sigmoid function model¹⁵ and S-systems.²⁵ Compared with linear models, the structure identification and parameter estimation of the nonlinear differential equation models are computationally more intensive and may require more data. Thus, existing efforts on reconstructing GRNs based on nonlinear ODE models have mostly been limited to small-scale systems.

In this article, we consider the following high-dimensional nonlinear additive ODE model for reconstructing GRNs:

$$X_k^{\prime }(t)={\beta }_{k0}+\sum _{j=1}^p{\beta }_{kj}{f}_{kj}\left\{X_j(t),{\mathit{\alpha }}_{kj}\right\},k=1,\dots ,p,$$ (2)

where β_k0 is the intercept, f_kj is known nonlinear functions with parameters α_kj, and coefficients β_kj represent the regulation effects of the genes in the network. The nonlinear parameter α_kj usually controls the shape of the function f_kj. For example, in the sigmoid function,

$$f(x)=\frac{1}{1+e^{-\alpha x}},$$ (3)

which is usually used to model nonlinear systems that exhibit “saturation” phenomena, the parameter α>0 indicates the transition rate or how fast the regulation effect of a gene saturates. For a small α, the sigmoid function (3) is close to a linear function, while for a large α, the function (3) is close to a step function. Similar sigmoid functions were used to model nonlinear gene regulatory effects in literature.^14,15 For different application scenarios, different nonlinear functions of f(x) can be chosen based on biological mechanisms or knowledge, or through initial data exploration, but the results under the different assumption of nonlinear functions are likely to be different, which is a limitation of nonlinear models. Some sensitivity analysis may be performed under different assumptions. Notice that this nonlinear additive model is different from that of the additive nonparametric model,²⁶ which can be converted into a linear model with spline-basis approximation. Thus, the proposed nonlinear additive model is more challenging.

Since biological systems are seldom fully connected and most nodes are only directly connected to a small number of other nodes resulting in sparse networks,²⁷ we assume that the number of significant nonlinear effects, ie, nonzero β_kj in model (2), is small, although the total number of genes p may be large. In practice, the gene expression profiles X_k(t) are never measured continuously, but rather at some discretized time points, and possibly with measurement errors. Denote

$$Y_{ki}=X_k\left(t_i\right)+{\epsilon }_{ki},i=1,\dots ,n_k,k=1,\dots ,p,$$ (4)

where ε_ki is assumed to be i.i.d r.v.’s with mean 0 and variance ${\sigma }_k^2$. Without loss of generality, we assume that all n_k’s are the same and referred as n hereafter. The challenging question is how to perform variable selection to identify the significant edges ${\mathcal{M}}_{*}$ = {1 ≤ k, j ≤ p : β_kj ≠ 0} in the GRN from the noisy experimental data.

Very little statistical research has been dedicated to parameter estimation and variable selection for nonlinear ODE models. Because nonlinear ODEs have no close-form solutions in general, the standard statistical methodologies developed for nonlinear regression²⁸ and generalized nonlinear regression models²⁹ cannot be applied directly. The existing parameter estimation methods for ODE models include the nonlinear least-squares (NLS) method,^30,31 the 2-stage smoothing-based estimation method,^32–35 the Bayesian approaches,^36,37 the principal differential analysis,³⁸ and the generalized profiling approach.^39,40 Among these methods, we are more interested in the 2-stage smoothing-based estimation method, which decouples the system of differential equations to a set of pseudo-regression models by substituting the differentials with estimated derivatives from observed data. This method not only reduces the computational burden dramatically but also has some desirable statistical properties.^33,34 Moreover, this parameter estimation method also conveniently facilitates the incorporation of variable selection techniques for ODEs. For each pseudo-regression model, we can apply variable selection methods, such as the adaptive lasso,⁴¹ SCAD,⁴² and the nonnegative garrote (NNG),⁴³ to determine the important network edges.

The number of genes (covariates) p in the nonlinear ODE system (2) usually grows with the sample size n, even p ≫ n. Moreover, since the total number of unknown parameters in model (2) is 2p²+p, the model complexity and parameter dimensionality grow a lot faster than the sample size n as both p and n increase. Because of the nonlinearity of model (2), it is very challenging, if possible at all, to directly apply the variable selection methods developed for small or moderate numbers of covariates to model (2). To overcome this challenge, we adopt the idea of independence screening introduced by⁴⁴ and extend it to the nonlinear additive ODE models. The sure independence screening method performs efficiently for dimension reduction via marginal correlation learning for high-dimensional feature selection. It has been shown to have a sure screening property; that is, with probability tending to 1, the independence screening technique retains all of the important features in the model. The sure independence screening was later extended to high-dimensional generalized linear models,⁴⁵ nonparametric additive models,⁴⁶ varying coefficient models,⁴⁷ and single index hazard rate models.⁴⁸ The idea of using marginal information to deal with high dimensionality was also adopted in other works.^49–54

We propose a 4-step variable selection procedure for the high-dimensional nonlinear additive ODE model (2). In step I, we use a nonparametric smoothing-based estimation approach to obtain the estimates for both the state variables and their derivatives based on the measurement model (4), and then in step II, we substitute the estimated state variables ${\widehat{X}}_k(t)$ and their derivatives ${\widehat{X}}_k^{\prime }(t)$ from step I into the ODE model (2) to form a “pseudo” nonlinear additive model. In step III, we propose a nonlinear independence screening (NLIS) method based on the pseudo-residual sum of squares of the marginal modeling. For every fixed 1≤ k ≤ p, we fit p marginal pseudo nonlinear regressions of the response ${\widehat{X}}_k^{\prime }(t)$ against each covariate ${\widehat{X}}_j(t)$ separately and select covariates with the pseudo-residual sum of squares smaller than a prespecified threshold value. Finally in step IV, we use the NNG to select important covariates in the reduced nonlinear additive model conditional on those selected covariates from Step III. The NLIS procedure in step III can significantly reduce the dimension of covariates, and more importantly, under some mild conditions, it has a sure screening property with the same parametric sure screening rate as that for generalized linear models derived by Fan and Song,⁴⁵ which ensures that all the important covariates are retained in step III with probability tending to 1. To the best of our knowledge, this is the first attempt to propose a variable selection method for a high-dimensional nonlinear ODE model and to establish the screening properties of the NLIS for ODE models under the “large p, small n” setting. We expect that the proposed nonlinear ODE model and method can be widely used to identify nonlinear dynamic GRNs for different biological systems with significant impacts in understanding pathogenesis of various diseases.

2 |. DYNAMIC GRN RECONSTRUCTION VIA A 4-STEP VARIABLE SELECTION PROCEDURE

The proposed variable selection procedure for the high-dimensional nonlinear ODE model (2) can be summarized into the following 4 steps. To simplify the presentation, we assume that for all $k,j=1,\cdots ,p,\text{E}\left[f_{kj}|X_j(t),{\mathit{\alpha }}_{kj}\right]]=0$ and ${\beta }_{k0}=0$. We use the GRN modeling as an example to present the proposed methodologies.

2.1 |. Step I: nonparametric smoothing to estimate state variables and their derivatives

First, we can apply one of nonparametric smoothing approaches such as smoothing splines, regression splines, penalized splines, or local polynomial to estimate the state variables and their derivatives, X_k(t) and $X_k^{\prime }(t)$. In this article, we adopt the penalized splines^34,55 to obtain the estimates, ${\widehat{X}}_k(t)$ and ${\widehat{X}}_k^{\prime }(t)$, $k=1,\cdots ,p$. That is, we approximate X_k(t) 1 by 1 for 1 ≤ k ≤ p by $X_k(t)\approx {\sum }_{j=-\nu }^{K_k}{\delta }_{k,j}{\mathbf{N}}_{k,j,\nu +1}(t)={\mathbf{N}}_{k,\nu +1}^T(t){\mathit{\delta }}_k$, where ${\mathit{\delta }}_k={\left({\delta }_{k,-\nu },\cdots ,{\delta }_{k,K_k}\right)}^T$ is the unknown coefficient vector to be estimated from the data and ${\mathbf{N}}_{k,v+1}(t)={\left\{N_{k,-v,v+1}(t),\cdots ,N_{k,K_k,v+1}(t)\right\}}^T$ is the B-spline basis function vector of degree v and dimension K_k+v+1 at a sequence of knots $t_0={\tau }_{k,-v}={\tau }_{k,-v+1}=\cdots ={\tau }_{k,-1}={\tau }_{k,0}<{\tau }_{k,1}<\cdots <{\tau }_{k,K_k}<{\tau }_{k,K_k+1}={\tau }_{k,K_k+2}=\cdots ={\tau }_{k,K_k+\nu +1}=T$ on [t₀, T]. Further, let $S_{{\lambda }_k}$ be the “hat” matrix that maps the observation ${\mathbf{Y}}_k={\left(Y_{k1},\cdots ,Y_{kn}\right)}^T$ into ${\widehat{\mathbf{X}}}_k:{\widehat{\mathbf{X}}}_k=S_{{\lambda }_k}{\mathbf{Y}}_k$. Usually, the knots of the P-splines are generally placed on a grid of equally spaced sample quantiles, then the number of knots and the penalty parameter are jointly determined by generalized cross-validation (GCV) described below,⁵⁵ although more complicated adaptive approaches are also proposed to place the knots at the sharp change points.

The penalized spline (P-spline) objective function contains a penalized sum of squared differences with a penalized term. To determine the penalization (smoothing) parameter, λ_k and the number of knots, we can use the standard GCV method,⁵⁶ ie, λ_k is chosen by minimizing the GCV score:

$$GCV\left({\lambda }_k\right)=\frac{1}{n}\frac{{\sum }_{i=1}^n{\left\{Y_{ki}-{\widehat{X}}_k\left(t_i\right)\right\}}^2}{{\left\{1-n^{-1}\mathit{\text{tr}}\left(S_{{\lambda }_k}\right)\right\}}^2}.$$

It has been shown in the literature^55,57 that the GCV performs similar to the classic cross-validation but is much faster in computation in nonparametric smoothing parameter selection. Some other methods such as Mallows C_p criterion and AIC perform similarly to the cross-validation and GCV methods.⁵⁷ Note that other nonparametric smoothing approaches such as regression spline and local polynomial smoothing can also be used in this step. Each of these smoothing methods has its pros and cons, but it should produce similar results if the appropriate smoothing parameter is chosen.

2.2 |. Step II: pseudo nonlinear additive models

Similar to Brunel³² and Liang and Wu,³³ we substitute the estimated state variables ${\widehat{X}}_k(t)$ and their derivatives ${\widehat{X}}_k^{\prime }(t)$ from step I into the ODE model (2) to form a set of pseudo nonlinear additive models:

$$Z_{ki}=\sum _{j=1}^p{\beta }_{kj}{f}_{kj}\left({\widehat{X}}_{ji},{\mathit{\alpha }}_{kj}\right)+e_{ki},k=1,\dots ,p;i=1,\cdots ,n,$$ (5)

where $Z_{ki}={\widehat{X}}_k^{\prime }\left(t_i\right)$ and ${\widehat{X}}_{ji}={\widehat{X}}_j\left(t_i\right);e_{ki}$ represents the aggregated measurement error and estimation error of ${\widehat{X}}_k(t)$ and ${\widehat{X}}_k^{\prime }(t)$ from step I. The p-dimensional ODE model (2) is then decoupled into p pseudo nonlinear regression models, which can be fitted using the least squares approach. For nonlinear ODE models, usually p cannot be too big; otherwise, it may encounter the computing and convergence problems. In model (5), the response variables and the covariates are derived from the nonparametric smoothing estimates of the state variables and their derivatives, respectively. Moreover, the resulting error terms e_ki are not i.i.d., but dependent. Thus, this is not a standard nonlinear model or generalized nonlinear model studied in the literature.^28,29 That is why we refer it as a pseudo nonlinear model. Since ${\widehat{X}}_k(t)$ and ${\widehat{X}}_k^{\prime }(t)$ are estimated continuously at every time point t from step I, we may augment more time points than the original observation times for the next step analysis. This data augmentation strategy has also been used by other investigators for ODE parameter estimation.^16,17

2.3 |. Step III: NLIS

We extend the idea of independence screening for linear and generalized linear regression models^44,45 to the nonlinear additive ODE models. Now, we rewrite p as p_n to emphasize that p may increase as n increases. For every fixed 1 ≤ k ≤ p_n, we fit p_n marginal pseudo nonlinear regression models for the response Z_k = (Z_k1,…,Z_kn)^T against each covariate ${\widehat{X}}_j={\left({\widehat{X}}_{j1},\dots ,{\widehat{X}}_{jn}\right)}^T$ separately and rank their importance to the joint model according to a measure of the goodness of fit of the marginal model. One screening strategy is to use the magnitude of the marginal least squares estimate (MLSE) ${\widehat{\beta }}_{kj}^M$ as the marginal utility for the independence screening. We refer to this strategy as MLSE screening, where we select a set of variables with the magnitude of ${\widehat{\beta }}_{kj}^M$ greater than a prespecified threshold ζ_k:

$${\widehat{\mathcal{N}}}_k=\left\{1\le j\le p_n:\left|{\widehat{\beta }}_{kj}^M\right|\ge {\zeta }_k\right\}.$$ (6)

Another natural screening strategy is to rank the variables according to the pseudo residual sum of squares of the component-wise nonlinear regression, where we select a set of variables:

$${\widehat{\mathcal{M}}}_k=\left\{1\le j\le p_n:{\text{RSS}}_{kj}\le {\xi }_k\right\},$$ (7)

with ${\text{RSS}}_{kj}={\min }_{{\beta }_{kj},{\mathit{\alpha }}_{kj}}{\sum }_{i=1}^n{w}_k\left(t_i\right){\left[Z_{ki}-{\beta }_{kj}{f}_{kj}\left({\widehat{X}}_{ji},{\mathit{\alpha }}_{kj}\right)\right]}^2$ as the pseudo residual sum of squares of the jth marginal fit and ξ_k as a prespecified threshold value, and w_k(t) is a prescribed weight function on [t₀, T] with boundary constraints w_k(t₀) = w_k(T) = 0. More discussions on how to select the weight function can be found in Brunel³² and Wu et al.³⁴ A data-driven threshold ξ_k can be determined by using random permutations as in Fan et al.⁴⁶ An alternative thresholding scheme is to choose d covariates with smallest RSS_kj. This strategy is analogous to the likelihood ratio screening proposed in Fan et al,⁵⁸ and we refer to it as the RSS screening. We show that it is asymptotically equivalent to the MLSE screening in preserving the important variables of the joint model (see Supporting Information). We refer to the aforementioned 2 feature screening approaches as NLIS approaches. In the case of large p_n, this NLIS approach can save a lot of computing efforts. In this article, we adopt the RSS screening.

To establish sure screening properties, we need to characterize the minimum distinguishable signal level.^44,45 In our case, the minimum distinguishable signal in the screening step is related not only to the stochastic error in estimating the nonlinear parameters but also to the approximation error in smoothing the state variables X_k(t) and their derivatives X_k’(t). We show that our screening procedure has a sure screening property, as defined by Fan and Lv⁴⁴ (see Supporting Information). The NLIS reduces the dimensionality of covariates from p_n to a possibly much smaller space with model size $\left|{\widehat{\mathcal{M}}}_k\right|$, to which the NNG for nonlinear regressions can be applied to perform more refined variable selection for the submodel conditional on the covariates selected by the NLIS.

2.4 |. Step IV: NNG for refining variable selection

Given initial estimates ${\widehat{\mathit{\alpha }}}_{kj}^{(init)}$, the nonlinear additive regression (5) reduces to a linear additive regression model:

$$Z_{ki}=\sum _{j=1}^p{\beta }_{kj}{\tilde{f}}_{kji}+e_{ki},$$ (8)

with the predictors being ${\tilde{f}}_{kji}=f_{kj}\left({\widehat{X}}_{ji},{\widehat{\mathit{\alpha }}}_{kj}^{(init)}\right)$. One can then use variable selection techniques for linear regressions to shrink unnecessary β_kj to zeros. In this article, we adopt the NNG⁴³ to find a set of nonnegative scaling factors c_kj to minimize

$$\frac{1}{2}\sum _{i=1}^n{w}_k\left(t_i\right){\left[Z_{ki}-\sum _{j=1}^p{c}_{kj}{\widehat{\beta }}_{kj}^{(init)}{\tilde{f}}_{kji}\right]}^2+n{\lambda }_k^{*}\sum _{j=0}^p{c}_{kj},\text{s.t.}{c}_{\mathrm{kj}}\ge 0,$$ (9)

where ${\widehat{\beta }}_{kj}^{(\mathit{init})}$ is initial estimates of β_kj and w_k(t) is a prescribed weight function on [t₀, T] with boundary constraints w_k(t₀) = w_k(T) = 0. The garrote estimates are given by ${\widehat{\beta }}_{kj}={\widehat{c}}_{kj}{\widehat{\beta }}_{kj}^{(\mathit{\text{init}})}$. An appropriately chosen ${\lambda }_k^{*}$ can shrink some c_kj to exact 0 and thus produces a more parsimonious model. We use the GCV to select ${\lambda }_k^{*}$ in our simulations and real data analysis.

Suppose the NNG selects covariates ${\tilde{\mathit{f}}}_{kj}={\left({\tilde{f}}_{kj1},\dots ,{\tilde{f}}_{kjn}\right)}^T,j\in {\mathcal{A}}_k=\left\{1\le j\le p:{\widehat{c}}_{kj}\ne 0\right\}$. We can then update the parameters α_kj conditional on the NNG estimates ${\widehat{\beta }}_{kj},j\in {\mathcal{A}}_k$. These procedures iterate until a convergence criterion is met; for instance, the residual sum of squares of the fitted model does not change up to a tolerance. The initial estimates need to be carefully chosen, because the solution path consistency of the NNG depends on the consistency of the initial estimators.^41,59 Here, we use the NLS estimates of (5) as the initial estimates for both α_kj and β_kj, which satisfy the consistency and asymptotical normality.^33,34

We have established the sure independence property for the proposed procedure. The detailed theoretical results are provided in Supporting Information.

3 |. MPLEMENTATION ALGORITHM

We propose to iteratively use a large-scale screening, the NLIS procedure and a moderate-scale variable selection technique, the NNG, to enhance the performance of the proposed variable selection method in terms of false selection errors. For every 1 ≤ k ≤ p_n, the algorithm is detailed as follows.

1. For every j ∈ {1,…,p_n}, compute the marginal fit by solving

   $$\underset{{\beta }_{\mathit{kj}},{\mathit{\alpha }}_{\mathit{kj}}}{\min }\sum _{l=1}^m{w}_k\left(t_l^{*}\right){\left[Z_{kl}-{\beta }_{kj}{f}_{kj}\left({\widehat{X}}_{jl},{\mathit{\alpha }}_{kj}\right)\right]}^2,t_l^{*}\in \left[t_0,T\right],$$ (10)

   where the data argumentation size m ≥ n and the weight function w_k(t) = sin{(t-t₀)π/(T-t₀)}. Rank the covariates according to the marginal pseudo residual sum of squares:

   $${\text{RSS}}_{kj}=\underset{{\beta }_{\mathit{kj}},{\mathit{\alpha }}_{\mathit{kj}}}{\min }\sum _{l=1}^m{w}_k\left(t_l^{*}\right){\left[Z_{kl}-{\beta }_{kj}{f}_{kj}\left({\widehat{\mathit{X}}}_{jl},{\mathit{\alpha }}_{kj}\right)\right]}^2.$$

   Select the top d covariates with the smallest RSS_kj or covariates with RSS_kj smaller than a threshold ξ_k estimated from random permutations. The set of selected covariates is denoted by S_k,1.

2. Apply the NNG for the nonlinear additive model introduced in section 2.1 on the set S_k,1 to select a subset $\mathcal{M}$_k,1. The BIC score of the model with covariates in $\mathcal{M}$_k,1 is computed, denoted as BIC_k(1).

3. For every $j\in {\mathcal{M}}_{k,1}^c=\left\{1,\dots ,p_n\right\}\backslash {\mathcal{M}}_{k,1}$, minimize

   $$\sum _{l=1}^m{w}_k\left(t_l^{*}\right){\left[Z_{kl}-\sum _{j^{\prime }\in {\mathcal{M}}_{k,1}}{\beta }_{k{j}^{\prime }}{f}_{k{j}^{\prime }}\left({\widehat{X}}_{j^{\prime }l},{\mathit{\alpha }}_{k{j}^{\prime }}\right)-{\beta }_{kj}{f}_{kj}\left({\widehat{X}}_{jl},{\mathit{\alpha }}_{kj}\right)\right]}^2,$$ (11)

   with respect to ${\beta }_{{kj}^{\prime }},{\mathit{\alpha }}_{{kj}^{\prime }},j^{\prime }\in {\mathcal{M}}_{k,1}$ and β_kj, ${\mathit{\alpha }}_{kj}\in {\mathcal{M}}_{k,1}^c$. This regression reflects the additional contribution of the jth covariate conditioning on the existence of the variable set $\mathcal{M}$_k,1. After marginal screening similar as in step 1 by ranking the pseudo residual sum of squares of model (11), choose a set of covariates ${\mathcal{S}}_{k,2}\subset {\mathcal{M}}_{k,1}^c$. The NNG procedure is then applied on the set ${\mathcal{M}}_{k,1}\cup {\mathcal{S}}_{k,2}$ to select a subset ${\mathcal{M}}_{k,2}$. Notice that we should keep model (11) as small as possible. Otherwise, we may be trapped in local solutions or encounter the convergence and computational problems.

4. Repeat step 3 until $\mathcal{M}$_k,ι = $\mathcal{M}$_k,ι+1, or the size of $\mathcal{M}$_k,ι, reaches a prespecified threshold d*. The set of selected covariates is ${\mathcal{M}}_{k,{\mathcal{K}}_k}$, where ${\mathcal{K}}_k=\arg {\min }_{\iota }{\text{BIC}}_k(\iota )$.

   For the numerical examples in the following sections, we recruit a fixed number of covariates at each round of marginal screening; that is, each S_k,l includes d covariates with the smallest marginal residual sum of squares. We experimented with different choices of d and found d =1 provides the smallest false selection rate and therefore is adopted in our numerical examples (including both simulation studies and real data application). We keep d as small as possible in order to avoid local solutions and convergence problems while fitting nonlinear ODEs.

4 |. SIMULATION STUDIES

Simulation experiments are designed to study and evaluate numerical performance of the proposed method. We consider a dynamic GRN described by the ODE model (2) with regulatory functions f_kj as centered sigmoid functions:

$$f_{kj}\left(x,{\alpha }_{kj}\right)=f\left(x,{\alpha }_{kj}\right)=\frac{1}{1+\exp \left\{-{\alpha }_{kj}x\right\}}-0.5,k,j=1,\dots ,p.$$ (12)

The sigmoid function is usually used to model nonlinear systems that exhibit saturation phenomena, and the parameter α_kj > 0 indicates the transition rate.¹⁵ For a small α, the sigmoid function is close to a linear function, while for a large α, the function is close to a step function. This model is also used for the application example in Section 5.

First, we consider a dynamic gene network with p=10 genes. We set the true nonlinear ODE model as

$$X_1^{\prime }=0.9f\left(X_1,{\alpha }_1\right)+1.5f\left(X_2,{\alpha }_2\right),$$

$$X_2^{\prime }=-1.7f\left(X_1,{\alpha }_1\right)+0.95f\left(X_3,{\alpha }_3\right),$$

$$X_3^{\prime }=-1.45f\left(X_2,{\alpha }_2\right)+2f\left(X_4,{\alpha }_4\right)+1.55f\left(X_6,{\alpha }_6\right),$$

$$X_4^{\prime }=-1.2f\left(X_3,{\alpha }_3\right)+1.35f\left(X_5,{\alpha }_5\right),$$

$$X_5^{\prime }=-2f\left(X_4,{\alpha }_4\right)-1.2f\left(X_5,{\alpha }_5\right),$$

$$X_6^{\prime }=-1.23f\left(X_7,{\alpha }_7\right)+1.58f\left(X_{10},{\alpha }_{10}\right),$$

$$X_7^{\prime }=1.94f\left(X_6,{\alpha }_6\right)-0.98f\left(X_8,{\alpha }_8\right),$$

$$X_8^{\prime }=f\left(X_6,{\alpha }_6\right)+1.45f\left(X_7,{\alpha }_7\right),$$

$$X_9^{\prime }=-1.24f\left(X_8,{\alpha }_8\right)+1.35f\left(X_{10},{\alpha }_{10}\right),$$

$$X_{10}^{\prime }=-1.21f\left(X_6,{\alpha }_6\right)-1.38f\left(X_9,{\alpha }_9\right),$$

where (α₁,…,α₁₀) are equally spaced in [2, 4] and t ∈ [0,10]. The initial conditions are X(0) = (0.6, −0.3, −0.75,−0.6,1.2, 1.4, −0.8, −1.2, −0.6, −1). We solve the above ODEs numerically for X_j(t), j = 1,…,10 and simulate the experimental data Y_ki at time points t₁,…t_n by model (4), where the measurement errors ${\in }_{ki}$ are normally distributed r.v.’s with mean 0 and variance σ². We consider 2 variance levels, σ = 0.05 and 0.2, and 2 sample sizes, n = 20 and 50. The observation time points are taken equidistantly in [0, 10]. We use the penalized splines to obtain the estimates of all state variables and their derivatives, where the smoothing parameters λ_k are chosen by GCV. The data augmentation size is set to be m = 100, and the weight function is chosen to be $w_k(t)=\sin (\pi t/10)$.

Next, we consider a sparse GRN consisting of p=30 genes. The nonlinear ODE system is randomly generated, and we need to make sure that it is identifiable and has stable solutions. There are several identifiability analysis approaches such as structural identifiability, practical identifiability, and numerical identifiability analyses that can be used to evaluate the identifiability of nonlinear ODE models. A good survey of ODE model identifiability can be found in Miao et al.⁶⁰ Notice that, among these p = 30 coupled nonlinear ODEs, the total number of parameters is 2p² = 1800, which is quite challenging in variable selection and parameter estimation. The total number of nonzero coefficients is 77, and there are 2 to 4 nonzero coefficients for each differential equation. The nonlinear parameter ${\alpha }_{kj}$’s are randomly generated similar to that in [2, 4]. Examples of generated parameters for p = 30 nonlinear additive ODEs are provided in Supporting Information. The number of observed time points for each X_j(t), j = 1,…,30 is set to be n = 50 or 100, and the standard deviation for the measurement error is σ = 0.01 or 0.05.

The accuracy of our variable selection approach is measured by sensitivity and specificity, which are defined as follows:

$$\text{sensitivity}=\frac{\#\text{of}\text{correctly}\text{estimated}\text{edges}}{\#\text{of}\text{all}\text{edges}\text{in}\text{the}\text{true}\text{network}},$$

$$\text{specificity}=\frac{\#\text{of}\text{correctly}\text{estimated}\text{edges}}{\#\text{of}\text{all}\text{estimated}\text{edges}}.$$

The simulations are repeated by 100 times, and results are presented in Table 1. Both sensitivity and specificity of the fitted models increase with the increase of sample size and decrease of measurement error variance. For the small scale network with 10 genes, our method can correctly identify over 70% of the nonzero coefficients under all simulation scenarios. For the larger scale network with 30 genes, our method can correctly select about 60% of the variables under the worse scenario (the largest measurement error and smallest sample size) and about 66% under the best scenario (the smallest measurement error and largest sample size). As suggested by one referee, we also performed the comparison simulations with the existing linear ODE model selection method¹⁷; that is, although the data are generated from nonlinear ODE models, we apply the linear ODE model selection method¹⁷ to the data and see whether we can still recover the correct variables in the model. As expected, the sensitivity and specificity of the linear ODE model selection method perform poorly, in particular, for higher dimensional cases with a large noise (Table 1). On the other hand, we also tried to perform simulation comparisons with the data generated from the linear ODE model, but we failed to obtain the simulation results for most cases from the nonlinear ODE model approach in this case. This is expected since the nonlinear model estimation algorithm often fails to converge if the model is misspecified. This limitation is true for any nonlinear modeling approaches.

TABLE 1.

The average values of the total number of estimated edges, and the sensitivity and specificity of the fitted models^a

			Proposed Nonlinear ODE Method		Existing Linear ODE Method
p	n	σ	Sensitivity	Specificity	Sensitivity	Specificity
10	20	0.2	0.7143 (0.0864)	0.7187 (0.0841)	0.5971 (0.0704)	0.3006 (0.0312)
		0.05	0.8743 (0.0724)	0.8516 (0.0721)	0.5490 (0.0446)	0.3236 (0.0200)
	50	0.2	0.8638 (0.0602)	0.8691 (0.0656)	0.5828 (0.0582)	0.3151 (0.0279)
		0.05	0.9648 (0.0250)	0.9610 (0.0363)	0.5347 (0.0375)	0.3240 (0.0197)
30	50	0.05	0.6074 (0.0260)	0.7454 (0.0249)	0.0861 (0.0283)	0.0850 (0.0262)
		0.01	0.6351 (0.0178)	0.7800 (0.0163)	0.0857 (0.0318)	0.0856 (0.0315)
	100	0.05	0.6455 (0.0138)	0.7979 (0.0198)	0.0984 (0.0299)	0.0852 (0.0240)
		0.01	0.6610 (0.0198)	0.8055 (0.0188)	0.0949 (0.0275)	0.0854 (0.0247)

^aStandard deviations in parentheses.

In general, the proposed method works well in identifying the structure of nonlinear ODE networks. The estimated parameters from the variable selection step can be used as the initial estimates for more refined ODE estimation procedures, such as the NLS method.^30,31 Since the focus of this article is on the variable selection, we did not perform the parameter estimate refinement of the selected network.

5 |. APPLICATION TO DYNAMIC GRN

In this section, we apply the proposed method to infer the GRN of Saccharomyces cerevisiae, based on the time course microarray data collected by Orlando et al.⁶¹ The global transcription dynamics were examined in synchronized populations of both wild-type and cyclin-mutant yeast cells. The gene expression was measured every 8 minutes for a total of 30 time points, covering about 2 cell cycles in the wild type and about 1.5 cell cycles in the cyclin mutant. Here, we only use the data for the wild-type cells to illustrate our method. We focus on the cell cycle pathway compiled in KEGG database (http://www.genome.jp/kegg/), which is around CDC28 (YBR160w), the catalytic subunit of the main cell cycle cyclin-dependent kinase (CDK). We select genes contained in this pathway that are also identified as cell cycle-regulated genes in Orlando et al.⁶¹ We also include 9 transcription factors that have been identified to have roles in regulating cell cycle-dependent yeast genes⁶²; these include MBP1, SWI4, SWI6, MCM1, FKH1, FKH2, NDD1, SWI5, and ACE2. The total number of genes in this network is 60; that is, p=60.

Following Chen et al,¹⁵ we model the dynamics of gene expressions by the nonlinear ODE model (2) with the regulatory function f_kj as the centered sigmoid function (12). The sigmoid function is chosen because of its good properties for modeling “dose-response” relationship. Similar sigmoid functions are also popularly used in modeling nonlinear gene regulatory effects in literature.^14,15 We first standardize the gene expression profiles of the 60 selected genes and obtain the estimates of the gene expression curves and their first derivatives by penalized smoothing splines as described in section 2.1. Figure 2 shows the smoothed trajectories (solid line) of 9 randomly selected genes. We can see that these genes show periodic patterns with 2 peaks, corresponding to 2 cell cycles. The estimates of the gene expression curves and their derivatives are then plugged into model (2) to form the pseudo nonlinear additive regressions. The variable selection method proposed in Section 2 is applied to identify significant regulations (connections) among the 60 genes, and the results are shown in Table 2. One important feature of the identified GRN is that each of these 60 genes is regulated by only a few other genes (ranging from 2 to 5 genes), reflecting the fact that the GRN is sparsely connected. The complete network is visualized in Figure 1. The fitted expression curves from the nonlinear model (dashed line) for 9 randomly selected genes are plotted and overlaid with the raw data and the smoothed expression curves (solid line) in Figure 2. We can see that these estimates fit the data quite well.

FIGURE 2.

Gene expression data for 9 randomly selected genes overlaid with the smoothed curves (solid line) and fitted curves by the nonlinear ODE model (dashed line)

TABLE 2.

Yeast cell cycle gene regulatory network. For a particular gene listed in Column 1, Column 2 provides the list of genes that have significant regulation effects (inward influence) on this gene, and Column 3 provides the list of genes that this particular gene has a significant regulation effect on (outward influence)

Gene	Inward Influence Genes	Outward Influence Genes
ACE2	CDC14, CDC20, FKH1, SPO12	CLN3
APC9	CDC28, CDC5
BUB1	NDD1, PCL1
CDC14	DBF2, MCM7	ACE2, CLB1, CLB4, FKH1, YHP1
CDC20	CDC20, MCM5, SPO12	ACE2, CDC20, FKH1
CDC28	CDC7, CLB5, CLN3, FAR1,	APC9, CDC6, CDC7, CLB5, CLB6, CLN3, FKH2, MBP1,
	SWI4	MCM1, MCM2, MCM3, MCM6, MPS1, MRC1, SWI6, WHI5
CDC45	MCM3, MCM7, NDD1, YHP1	CDC6, MCM4, MCM5
CDC5	DBF2, ESP1, WHI5	APC9, DBF2
CDC6	CDC28, CDC45, DBF2, SIC1	CLB2, MCM1, ORC1
CDC7	CDC28, MCM2	CDC28, MAD3, RAD17, SWI4
CLB1	CDC14, MAD3, SPO12	CLB4, ESP1, YCS4
CLB2	CDC6, HSL1, SIC1, SWI6	FKH1
CLB3	DBF2, MCM3, WHI5	MOB1, SWI5
CLB4	CDC14, CLB1	SLK19, WHI5
CLB5	CDC28, MCM5	CDC28, FAR1, MCM2, MCM3, MCM4, MCM7, ORC1
CLB6	CDC28, MCM3	MBP1
CLN1	CLN3, FKH1, SPO12	SWI4
CLN2	CLN3, FKH2, MCM4, SPO12
CLN3	ACE2, CDC28, MBP1, YCS4	CDC28, CLN1, CLN2, MCD1, RAD17, SWI4, SWI5, SWI6, YOX1
DBF2	CDC5, MCM3, SIC1, YCS4	CDC14, CDC5, CDC6, CLB3, FKH2, MPS1
DUN1	MCM3, SWE1, SWI6, WHI5
ESP1	CLB1, SWI6	CDC5
FAR1	CLB5, SLK19	CDC28
FKH1	CDC14, CDC20, CLB2, SWI5	ACE2, CLN1
FKH2	CDC28, DBF2	CLN2, NDD1, YCS4
GIN4	MCM2, SWI6
HSL1	HSL1, MCM2, WHI5	CLB2, HSL1
LTE1	MCM2, MOB1
MAD3	CDC7, SPO12	CLB1, SCC2, SMC1
MBP1	CDC28, CLB6	CLN3
MCD1	CLN3, MCM3
MCM1	CDC28, CDC6, SIC1	MOB1, PCL1
MCM2	CDC28, CLB5	CDC7, GIN4, HSL1, LTE1, RAD53, SMC3
MCM3	CDC28, CLB5	CDC45, CLB3, CLB6, DBF2, DUN1, MCD1, MCM5, YOX1
MCM4	CDC45, CLB5	CLN2, PCL1
MCM5	CDC45, MCM3, SCC2, YCS4	CDC20, CLB5, MRC1, RAD53
MCM6	CDC28, SIC1
MCM7	CLB5, SIC1	CDC14, CDC45
MOB1	CLB3, MCM1, SIC1, SWI6	LTE1, SPO12
MPS1	CDC28, DBF2
MRC1	CDC28, MCM5	NDD1
NDD1	FKH2, MRC1	BUB1, CDC45, RAD53, SMC3, SWE1
ORC1	CDC6, CLB5	SCC2
PCL1	MCM1, MCM4	BUB1
RAD17	CDC7, CLN3
RAD53	MCM2, MCM5, NDD1	WHI5
SCC2	MAD3, ORC1, SWI6	MCM5
SIC1	SIC1, SWI5	CDC6, CLB2, DBF2, MCM1, MCM6, MCM7, MOB1, SIC1, SPO12
SLK19	CLB4, SLK19	FAR1, SLK19, SWI6
SMC1	MAD3, WHI5
SMC3	MCM2, NDD1
SPO12	MOB1, SIC1, SWI6	ACE2, CDC20, CLB1, CLN1, CLN2, MAD3
SWE1	NDD1, SWI4	DUN1
SWI4	CDC7, CLN1, CLN3	CDC28, SWE1, SWI6
SWI5	CLB3, CLN3	FKH1, SIC1, YHP1
SWI6	CDC28, CLN3, SLK19, SWI4	CLB2, DUN1, ESP1, GIN4, MOB1, SCC2, SPO12
WHI5	CDC28, CLB4, RAD53	CDC5, CLB3, DUN1, HSL1, SMC1
YCS4	CLB1, FKH2	CLN3, DBF2, MCM5
YHP1	CDC14, SWI5	CDC45
YOX1	CLN3, MCM3

FIGURE 1.

Graph of yeast cell cycle GRN formed by 60 genes

We find that gene CDC28 regulates the most genes in this network, demonstrating its central role in yeast cell cycle process. This is consistent with the fact that CDC28 is essential for the completion of the start and the controlling event in the cell cycle, and it associates with different regulators throughout the cell cycle to accomplish the waves of CDK activity that drive events of the cell cycle through phosphorylation of key substrates.^63,64 The identified network also includes several important pathways in the cell cycle process. For example, in our network, CDC28 and the G1 cyclin CLN3 regulate each other, and CDC28 regulates the repressor of G1 transcription WHI5, and CLN3 regulates the other two G1 cyclins CLN1 and CLN2 as well as the SBF complex (SWI4-SWI6). This is in line with the first wave of CDK activity where CDC28 associates with CLN3 and inactivates WHI5, which in turn leads to active SBF, transcription factors that promote transcription of CLN1 and CLN2.^65,66 The MCM2–7 proteins are components of a DNA helicase that plays an essential role in DNA replication and cell proliferation. They form a ring-shaped heterohexamer that assembles as part of the prereplicative complex (pre-RC) during the G1 phase of the cell cycle and is activated at the G1/S transition by the DNA replication initiation factor, CDC45.^67,68 As cells enter the S phase, CLB/CDC28 kinases trigger the initiation of DNA replication, and this results in disassembly of pre-RCs.⁶⁷ In addition, CLB/CDC28 kinases can prevent the reassembly of pre-RCs until the end of mitosis, and this is done by promoting the net nuclear export of MCM proteins.⁶⁹ In our network, genes MCM2–7 are regulated by either CDC28 or CDC45, consistent with previous biological findings.

6 |. DISCUSSION

In this article, we considered variable selection for high-dimensional nonlinear additive ODE models. To tackle the challenge of high dimensionality, we proposed a 4-step variable selection procedure, where we coupled the 2-stage smoothing-based estimation method for ODEs and an NLIS method based on the pseudo-residual sum of squares of the marginal modeling to perform variable selection. We extended the independence screening approach from linear and generalized linear models^44,45 to the nonlinear additive ODE model setting. We have shown that our NLIS method possesses the sure screening property, and it can handle problems with non-polynomial dimensionality. After reducing the dimension of covariates to a moderate size through the NLIS procedure, we used the NNG for nonlinear additive regressions to perform more refined variable selection for the submodel conditional on the covariates selected by the NLIS. The proposed method was applied to simulated data and an real data example for identifying the dynamic GRN of S cerevisiae to illustrate its numerical performance.

This is not a trivial combination of the 2-stage smoothing-based estimation method for ODE models and the high-dimensional variable selection techniques. For the pseudo nonlinear additive model, the minimum distinguishable signal of the marginal screening is closely related to the measurement errors and estimation errors of the state variables and their derivatives as well as the stochastic and numerical errors in the optimization of the nonlinear parameters. In addition, the objective function derived from the pseudo nonlinear additive model often has multiple local minima, making the estimation and marginal screening more challenging in computation. There are also several critical challenges in theoretical development. First, the decoupled pseudo nonlinear additive models are based on the nonparametric smoothing estimates of the ODE state variables and their derivatives, rather than the observed data. Therefore, the errors in these regressions are not i.i.d., but dependent. Moreover, the variable selections are also performed with respect to the smoothing estimates, so the theoretical properties are not trivial to establish. Specifically, our results improve the uniform convergence results of the marginal nonparametric estimation obtained by Fan et al⁴⁶ from nonparametric rates to parametric rates. This results in the improvement of the convergence rate of the nonparametric independence screening.

In our nonlinear additive ODE model (2), the nonlinear functions f_kj are assumed to be known, which requires some prior knowledge about the relations between the covariates and the response. If we have no prior information about such relations, then applying our model may result in a bias. In this case, we recommend to use a nonparametric ODE model to replace the model (2), but its methodologies and theories need to be carefully studied. In model (2), we only considered the additive structure for nonlinear functions. The proposed 4-step variable selection procedure may be extended to more complex structure, such as ODE models with interactions, which is left for further research in the future. We would like to point out that, for any nonlinear modeling approaches, a critical limitation is that it is sensitive to model misspecification. If the nonlinear ODE model is misspecified, the computational algorithm often fails to converge.

We used the 2-stage smoothing-based ODE estimation method for the nonlinear ODE model in order to reduce the computational cost and simplify the implementation of variable selection methods. Some other more efficient estimation approaches for ODE models such as the generalized profiling approach³⁹ may also be considered for high-dimensional ODE variable selection, but it may be computationally challenging. In addition, we adopt the data argumentation strategy for ODE parameter estimation in step II, which may improve the numerical performance of the proposed method. Intuitively, the bigger the data argumentation size is, the better the estimation is. However, a bigger data argumentation size will increase the computational cost, and this may become a serious problem when the ODE system is large. It is important to study the trade-off between the data argumentation size and the computational cost, which is still an open problem.

In this article, we have focused on variable selection, however, the proposed approach could also produce parameter estimation simultaneously. But usually, the parameter estimates from a variable selection procedure may not be efficient and accurate. A parameter estimation refinement step based on the selected model is required in order to obtain good parameter estimates. For high-dimensional nonlinear ODE models, the parameter estimation refinement is not trivial due to local solution and computational convergence problems in high-dimensional nonlinear optimizations. The output of parameter estimation from the variable selection procedure could be used as initial values for the parameter refinement step. But this is beyond the scope of this paper. We refer the readers to Li et al³⁰ and Xue et al. ³¹

Although we have used the time course gene expression data from S cerevisiae, a popular experimental organism to study gene regulation networks and biological mechanisms at molecular levels, the proposed model and methodology could be used to model any high-dimensional and nonlinear systems, in particular, within-host dynamic responses to infectious diseases at gene and protein levels. We are currently applying these methods to time course gene expression and proteomics data from HIV and influenza infections. The biological findings will be reported elsewhere in the near future.