Model Discrimination in Dynamic Molecular Systems: Application to Parotid De-differentiation Network

Abstract

In modern systems biology the modeling of longitudinal data, such as changes in mRNA concentrations, is often of interest. Fully parametric, ordinary differential equations (ODE)-based models are typically developed for the purpose, but their lack of fit in some examples indicates that more flexible Bayesian models may be beneficial, particularly when there are relatively few data points available. However, under such sparse data scenarios it is often difficult to identify the most suitable model. The process of falsifying inappropriate candidate models is called model discrimination. We propose here a formal method of discrimination between competing Bayesian mixture-type longitudinal models that is both sensitive and sufficiently flexible to account for the complex variability of the longitudinal molecular data. The ideas from the field of Bayesian analysis of computer model validation are applied, along with modern Markov Chain Monte Carlo (MCMC) algorithms, in order to derive an appropriate Bayes discriminant rule. We restrict attention to the two-model comparison problem and present the application of the proposed rule to the mRNA data in the de-differentiation network of three mRNA concentrations in mammalian salivary glands as well as to a large synthetic dataset derived from the model used in the recent DREAM6 competition.

1. Introduction

The process of falsifying inappropriate candidate models in order to find the best-suited model among different hypotheses is called model discrimination and is often based on discriminant analysis—a statistical technique going back to Fisher (1936) and applied in the fields of statistics, pattern recognition, and machine learning for the purpose of objects classification. The application of discriminant-like methods to modern biological data seems also useful in the current era of systems biology. Indeed, in most modern approaches to modeling biological phenomena via systems methods, one faces the challenge of evaluating multiple competing mathematical models of complicated reaction networks describing interactions among the biochemical species, such as various RNA molecules, proteins, enzymes, metabolites, and so on (see, e.g., Liao et al., 2005, or Carrillo et al., 2010, for specific examples in enzyme dynamics). Typically, before the correct model of a biochemical system is found, different hypothetical models might be reasonable and consistent with previous knowledge and available data. Whereas the full validation and fine-tuning of such models may require single cell-level data obtainable only through complex experiments, the initial model discrimination may sometimes be achieved by the analysis of aggregated data obtained from multiple cells (see, e.g., Hecker et al., 2009, for a recent review). Such data is often much more easily obtained with standard (hence cheaper) experimental methods, such as reverse transcriptase–polymerase chain reaction (RT-PCR). The dynamics of such aggregated data models are often amenable to analysis via a system of ordinary differential equations (ODEs) approximating the average stochastic effects of the cellular-level behavior (see, e.g., El Samad et al., 2005). Under the assumption that such a representation is meaningful, that is, the aggregated system indeed averages out the stochastic fluctuations around a single cellular-level steady state, the mathematical models are deterministic in nature and their sole sources of uncertainty come from the biased representation of reality and the data collection methodology (e.g., measurement bias and errors). This in turn allows one to invoke the general statistical methodology of (posterior) likelihood-based model discrimination, reminiscent of the classical linear and quadratic discriminant methods (LDA and QDA) (see, e.g., Hastie et al., 2001, Chapter 4), as well as the general methodology of the statistical validation of kinetic models (see, e.g., Zhan et al., 2011).

In a related problem of validating a given ODE model of a biological system with the aggregated time-course RT-PCR data, Rempala et al. (2007) proposed a likelihood-based procedure applicable to simple kinetic models, like, for instance, a simple transcription network (e.g., L1-gene retrotransposition). The procedure described in Rempala et al. (2007) was based on the ideas of Bayarri et al. (2007) on statistical validation in computer models, and was seen to work well for simple biological networks but not necessarily for more complicated ones such as complex signaling pathways. For these more complicated settings, a more comprehensive approach seems to be required, involving both the model validation and selection (see, e.g., Neff et al., 2011; Morton et al., 2010; Steimer et al., 2010, for applications to various molecular systems). For the latter, the ability to perform a discriminant-like analysis between competing models based on available time-course data is essential.

The aim of the current work is to propose a relatively general procedure for discriminating between different ODE-based kinetic models of a biological phenomenon under the assumption that the time-course data from the trajectory is available and that basic sources of both experimental and instrumental uncertainty and bias are accounted for. The proposed procedure extends Rempala et al.'s (2007) approach to multivariate time-course data and nonlinear interactions and is illustrated with two examples of data from time-course measurements. The first one is based on examining mRNA expression in acinar cells, obtained by RT-PCR with the purpose of analyzing transcription factor (TF) regulation of genes, which are indicators of differentiation. This was the original biological problem that motivated the methodological study. However, since the resulting network of the first example is relatively simple due to limited data availability, we have also considered another example in which two much larger synthetic models are compared. The second example is based on the ODE models considered in the recent Dialogue for Reverse Engineering Assessments and Methods (DREAM6) competition.

In order to motivate our biological example, we briefly recall some basic facts of the salivary glands biology relevant to our modeling efforts discussed below. (For a more general introduction to the topic, see, for instance, Harunaga et al., 2011 and Gorr et al., 2005.) Salivary glands are important for maintaining a healthy oral cavity in most mammals by production of saliva, which is necessary for lubrication and digestion initiation and which also contains proteins with antibacterial and antifungal properties. However, in humans salivary glands are often damaged or destroyed by radiation therapy, surgery for head and neck cancers, or by advanced Sjögren's syndrome. Secretion from salivary glands originates in clusters of acinar cells, which are classified as either serous (protein-secreting) or mucous (mucin-secreting). The parotid is the largest salivary gland, composed of serous acinar cells, and therefore is responsible for the secretion of the majority of salivary proteins. The most abundant proteins in parotid saliva are amylase and parotid secretory protein (PSP). The serous acinar cell is the salivary cell type that is most sensitive to radiation (Grundmann et al., 2009), and understanding its differentiation process is a necessary step in enabling the restoration or regeneration of diseased or destroyed parotid salivary tissue. Unfortunately, unlike in some differentiated tissues, the molecular mechanisms that drive differentiation of parotid acinar cells are not well characterized. Deciphering how the genes are transcriptionally regulated during de-differentiation is therefore important, as it might provide molecular insights into the reverse process of differentiation. Acinar cells, as with certain other cell types, lose their specialized characteristics rapidly when they are removed from the animal and placed in cell culture (Gorr et al., 2005). The synthesis and secretion of amylase and PSP are markers of terminal differentiation, which are rapidly lost under the conditions of culture (Venkatesh et al., 2012), providing for an excellent experimental system to study the loss of differentiation.

As the first step in analyzing the de-differentiation process, one would attempt to relate the mRNA levels of the terminal markers of differentiation (amylase and PSP) with that of exocrine-specific transcription factors such as Mist1, which is known to contribute to differentiation of pancreatic and salivary acinar cells (Luo et al., 2005). In Figure 1, we present a portion of a hypothesized parotid molecular differentiation model, expressing the idea that Mist1 mRNA generates Mist1 protein, which acts as a dimer at both the PSP gene and the amylase gene to stimulate the expression of both PSP and amylase mRNAs. On the basis of the hypothesized structure summarized in Figure 1, in order to help develop formal ODE-based models of their interactions, the longitudinal mRNA expression levels (measured via RT-PCR) were collected for the three genes. Similar ODE models have recently been used to characterize cell differentiation during hematopoiesis (Wang 2007; Soneji et al., 2007), but have been used to model differentiation of few if any other cell types. Nonetheless, they have been used successfully in other systems. For example, Gin et al. (2007) developed an ODE model to study the fluid secretion from adult parotid acinar cells. In addition, ODE modeling has been used extensively to study critical aspects of the cell cycle (Sveiczer et al., 2000; Tyson et al., 2001; Novak and Tyson, 2004) and the molecular interactions that cause circadian rhythms (Brown et al., 2000; Baker et al., 2011; Jolma et al., 2010).

FIG. 1.

Hypothesized interactions driving differentiation. A minimal hypothesized network consisting of the Mist1 transcription factor driving expression of both the parotid secretory protein (PSP) gene and the amylase gene. During differentiation of the cell, Mist1 would increase the amount of PSP mRNA and amylase mRNA. During the loss of differentiation in vitro, the decrease of Mist1 would lead to a decrease of PSP and amylase mRNAs.

2. Methods and Results

We start by providing some further description of the time-course mRNA expression data and the RT-PCR procedure in Section 2.1 below. In the following Section 2.2, two possible ODE models for the given data are proposed; the procedures for validation and selection between them are presented in Section 2.3, with the results discussed in Section 2.4. The additional results from synthetic large-scale DREAM6 model comparisons are summarized in Section 2.5.

2.1. Time-course mRNA data

The procedure of collecting the data may be described as follows. For each experiment, parotid glands from 3–5 rats were collected and pooled, and the acinar cells isolated (see Venkatesh et al., 2012, for a detailed description of the experiment). This single pool of acinar cells was then split into eight cultures (in 6-well plates) and incubated for 0, 1, 6, or 24 hours. At each time point, the cells in two different wells were separately used to make RNA samples, therefore creating technical replicates at each time point. The expression of amylase, Mist1, and PSP mRNAs were measured by TaqMan RT-PCR assays and normalized to glyceraldehydes 3-phosphate dehydrogenase (GAPDH) mRNA, which served as the internal experimental control. For each timepoint expression measured, the corresponding values of the two technical replicates were averaged and expressed relative to their initial value at the experiment inception (0-hour timepoint). The measurements from distinct wells were treated as independent. Mist1 and either PSP or amylase (due to limiting amounts of RNA) was measured in a total of seven independent experiments, as shown in Table 1.

Table 1.

Time-Course mRNA Data for Amylase, Mist1, and PSP

	Amylase			Mist1
Time (ti)	A1	A2	A3	M1	M2	M3	M4
0	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
1	0.7020	2.1320	0.8440	0.7385	0.6083	0.7135	0.6322
6	0.0540	0.2420	0.2055	0.0565	0.2176	0.2400	0.3197
24	0.0060	0.0875	0.0375	0.0035	0.0202	0.1005	0.0444

	Mist1			PSP
Time (ti)	M5	M6	M7	P1	P2	P3	P4
0	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
1	0.8840	0.7435	0.7110	0.6310	1.0170	1.0745	1.0435
6	0.2645	0.1160	0.0640	0.0520	0.0825	0.1000	0.0995
24	0.0225	0.0105	0.0030	0.0040	0.0260	0.0095	0.0220

The average mRNA expression data for each gene are plotted in Figure 2. For better visualization, in this as well as in all subsequent time-course plots presented in this article, we follow the convention of extending all graphs beyond the observation time points via the linear interpolation. As seen from the plot in Figure 2, mRNA expression of amylase and PSP reach a saturation point at approximately 1 hour and then exponentially decrease. In contrast, Mist1 expression does not seem to have a saturation point and is seen to rapidly decrease in acinar cell cultures. As expected, the variations for each species also decrease with the time of culture.

FIG. 2.

mRNA expression time-course data. Colored solid lines are linearly interpolated average values for the three genes across time points. Each interval for each gene and time point shows the minimum and maximum mRNA expression obtained from the RT-PCR experiment.

2.2. ODE models

Based on the data presented in Figure 2, we may derive the following qualitative chemical “mass transfer” model (see, e.g., Chang et al., 2011; Likhoshvai and Ratushny, 2007) represented, in our case, by the coupled biochemical reactions

(1)

While this is a very simple network, it represents a realistic, testable example, as it is well documented that Mist1 transcription factor helps drive terminal differentiation of acinar cells, and expression of both PSP and amylase are markers of acinar differentiation. To quantify the model above, it was hypothesized that the action of Mist1 was carried out by the dimerization of Mist1 molecules and that the saturation of PSP and amylase levels were attained while Mist1 was abundant. Two ODE representations of these interactions were considered. The first one (denoted here ) employed the standard Hill-function ODE mass transfer model (see, e.g., Likhoshvai and Ratushny, 2007) as follows

(2)

Here A(t), M(t), P(t) are mRNA expressions of amylase, Mist1, and PSP, respectively, at time t, with A(0) = 1, M(0) = 1, and P(0) = 1, and are nonnegative coefficients parametrizing . In particular, v₃, v₄, and v₇ are seen as the respective degradation rates, v₁ and v₅ as the maximum reaction rates, and v₂ and v₆ as the half-saturation points.

The particular form of the ODE model is based on some earlier work (see, e.g., Venkatesh et al., 2012, and references therein) stipulating the presence of the trajectories extrema for all three species (maxima for amylase and PSP and a minimum for Mist1) at some t₀ > 0. The reason for the Hill-function form of is the consideration of the saturation effect of the molecules of Mist1. However, the Hill–type growth does not take directly into account the “mass transfer” aspect of the reactions (1), which require that M(t) → 0 as t→∞. The simplified version of the model (denoted ) may be obtained by employing the Taylor series approximation x²/(a+x²)≈x² valid for small x, where “small” may be defined more precisely by considering non-zero intersection points of both curves and depends on the value of a > 0 (cf., also Sorribas et al., 2007). The new model is

(3)

where A(0) = 1, M(0) = 1, and P(0) = 1 and are, as before, assumed to be nonnegative system parameters. The main difference between and is the lack of the denominator terms in , which allows it to be parametrized by two less coefficients v. The additional advantage of this simplified ODE model is that (3) follows the so-called law of mass-action, a popular general empirical law of molecular dynamics (Hornos et al., 2005).

Given the two models and , a natural question arises as to whether either one of them may meaningfully describe the data in Table 1. In order to answer it, we need the assessment of the accuracy of the model predictions associated with and , which we discuss next.

2.3. Model validation and discrimination

To describe the relationship between the solution of ODE models (2) and (3), and the actual mRNA expressions for the genes of interest summarized in Table 1, we follow Rempala et al. (2007) and employ the additive data model as follows.

Denote the time-course vectors of mRNA expression for each of the TFs by , with . Here K = 3 with the time-points (in hourly units) t₁ = 1, t₂ = 6, t₃ = 24; A, M, and P stand for amylase, Mist1, and PSP respectively, with the corresponding number of experiments (see Table 1), r_A = 3, r_M = 7, and r_P = 4. Let v_i, i = {A, M, P} be the vector of parameters of the ODE models for each gene (i.e., for , v_A = (v₁, v₂, v₃)′, v_M = v₄, and v_P = (v₅, v₆, v₇)′ and for , v_A = (v₁, v₂)′, v_M = v₃, and v_P = (v₄, v₅)′). If is the solution of the ODE model (either under or ) corresponding to the i-th TF, then is assumed to be related to as follows

(4)

where b_i(t) (≡ b_i) is the vector of parameters accounting for the model bias, and is the vector of random errors. Since, after the initial measurement, we have three time points for each mRNA, both models and have the same number of nine bias parameters. Since and are parametrized by seven and five ODE coefficients respectively, for the model (4) has a total of 16 parameters while for it has 14.

For the purpose of model validation, we need to provide the credibility bounds in (4) and this can be done, for instance, by considering the joint posterior distribution of the model parameters. The Bayesian framework is particularly appropriate in our case due to the limited number of observations available and hence potential problems with the identifiability of the standard maximum likelihood estimators (see, e.g., Wilkinson, 2009). As in Rempala et al. (2007) and Bayarri et al. (2007), we assume the form of the likelihood function for as follows:

(5)

where MVN(μ_i, diag(δ)) is a multivariate normal distribution with a mean vector , and the variance–covariance matrix with diagonal elements δ_i, where , and off-diagonal elements zero due to the assumed independence of each culture well (see Section 2.1). The variance terms for the longitudinal levels of mRNA are taken as decreasing exponentially with time—the assumption that seems consistent with the data pattern observed in Figure 2. The form of the likelihood (5) may be justified by the fact that in many kinetic biochemical systems (including those considered here) the fluctuations around ODE models are approximately Gaussian (see, e.g., Ge and Qian, 2009; Andersson and Britton, 2000, Chapter 5).

To complete the statistical model setup, we need to define flexible prior distributions for the ODE parameters v = (v_A, v_M, v_P)′ as well as the bias terms b = (b_A, b_M, b_P)′. Due to the parameter values support and the covariance structure considerations, the convenience choices here are a multivariate log-normal prior for v and a multivariate normal one for b. Whereas the appropriateness of these particular distributions may be debated, their form does not seem to particularly effect the results of our analysis presented below. Define

(6)

where ν is a mean vector and Λ is a variance–covariance matrix, empirically estimated from the data, as well as

(7)

where the variance–covariance matrix Δ is a block diagonal matrix with the same diagonal blocks . Thus, in our case (K = 3), Δ is a 9 × 9 matrix. The exponential, block-diagonal form of Δ takes into account the possible dependancies among biases between and within the trajectory points of the three mRNAs. From the forms of the likelihood function of (5) and the priors (6) and (7), we see that the posterior distribution is proportional to

(8)

To obtain this posterior distribution, the usual Markov Chain Monte Carlo (MCMC) method based on the Gibbs sampler as in Bayarri et al. (2007) was used, but other alternatives could have been also entertained (see, for instance, Zhan et al., 2011, or Jones, 2011). Regardless of the detail of the particular MCMC technique, consider now v_il and as the collection of posterior samples obtained from the converged chain. Using these posterior samples, we can approximate the posterior means of the ODE parameter v_i, i = {A, M, P}, (cf, Eq. (A.1) in Appendix A), which in turn allows us to approximate the mean of the pure model prediction and the bias-corrected prediction (cf, Eq. (A.3) in Appendix A) for the i-th mRNA (i = {A, M, P}). The model prediction (1 − α)% credibility bounds for the i-th mRNA (i = {A, M, P}) may be obtained by calculating α/2 and (1 − α/2) quantiles of the empirical distribution based on . Further details and relevant formulas are provided in Appendix A.

2.4. Results

The results of the analysis based on (8) are summarized in Table 2 and illustrated in Figure 3, which shows, for each mRNA, the pure model and the bias-corrected predictions, credibility envelopes for both and ODE models given by (2) and (3), as well as the actual data. From the plots in Figure 3, it appears that both and have validity for the mRNA expression data from Table 1, as for all three mRNAs in respective models, their 95% credibility envelopes include both the pure and the bias-corrected model predictions (cf., Rempala et al., 2007; Bayarri et al., 2007). Moreover, the width of the credibility envelopes in both models seems similar for each of the respective genes. However, when comparing the differences between the pure and the bias-corrected model predictions, it is clear that those under are considerably smaller, especially for amylase and PSP mRNAs. This seems to indicate that, albeit the uncertainties of the prediction in both models are similar (which is also seen more directly by comparing the variances of the model predictions computed via the formula (A.5) in Appendix A), the simplified model , despite its sparser parametrization, has smaller bias than . Plots in Figures 4 and 5 further support this conclusion. In Figure 4, variances of each gene for both models across times are seen as not very different. However, in Figure 5, appears to have small biases relative to , with the similar overall longitudinal bias pattern for both models.

Table 2.

Parameter Estimates (Posterior Means) and Their Standard Deviations Under and

Model	Mean & SD	v1	v2	v3	v4	v5	v6	v7
		21.4868	4.0750	1.6180	0.2990	20.2618	2.1828	4.5004
		16.0030	2.8402	1.5623	0.0557	16.6145	2.3336	2.1067
		1.7622	0.9648	0.3237	1.4291	1.2157
		2.0204	0.7260	0.0526	1.4561	0.7214

FIG. 3.

The 95% posterior credibility bounds for and . Actual data (circle) versus the pure model prediction (red line) and the bias-corrected model prediction (black line) along with the 95% credibility bounds (blue dotted line): (a) amylase for ; (b) Mist1 for ; (c) PSP for ; (d) amylase for ; (e) Mist1 for ; and (f) PSP for .

FIG. 4.

Variance of the model prediction for mRNA data: (a) and (b) .

FIG. 5.

Bias of the model prediction for mRNA data: (a) (b) .

Denote for brevity the observed data by and let θ_i be the parameter vector (v,b)_i for the model (i = 1,2), with the respective prior . In order to quantitatively discriminate between the two models, we again utilize the results of the MCMC analysis based on Equation (8), in the spirit of the Fisher discriminant principle alluded to in the introduction. To this end, we consider the Bayes factor (BF) criterion (see, e.g., Hastie et al., 2001, Chapter 7; for a possible alternative based on Bayesian information (see, e.g., Jones, 2011, and Skanda and Lebiedz, 2010)

(9)

where both the numerator and the denominator integrals are now approximated by the numerically averaging across the posterior sample obtained from the MCMC algorithm under respective models. The numerical value BF≈0.0001 indicates that under the equal prior preference for both and the posterior probability of is considerably larger (10,000-fold) than that of and thus the simplified, mass action-based ODE system (2) is strongly preferred over the originally proposed Hill-function-based ODE system (3).

2.5. Additional results with synthetic data

In order to assess the scalability of the method to more realistic biological networks, in addition to (2) and (3), a larger synthetic system was also considered in the model discrimination analysis. This synthetic system model is described by an ODE set with 12 species and 24 parameters (see Appendix B) and comes from the simplified version of the “Model1” of the most recent DREAM6 Estimation of Model Parameters Challenge. The synthetic data was generated from this ODE model under the assumptions (4). Both the correct () and slightly altered () ODE systems were compared as above, in an effort to discriminate between the two alternatives. The complete descriptions of and are provided in (B.1) and (B.2), in Appendix B. Both sets of ODE are seen to have the same number of parameters, with the difference introduced in the growth models of the species Y₉ and Y₁₁ in both sets. The details on data generation are provided in Appendix B. The generated data and fitted models are presented in Figure 6, which shows little difference in terms of the model fit. In order to analyze the fit in more detail, the same MCMC method was applied to obtain 95% credibility bounds for models and . The results are plotted in Figures 7 and 8 and include both the pure and the bias-corrected model predictions. From the plots it appears that both models have similar patterns of predictions for all species, with no obvious differences between the pure and the bias-correct versions. Some prediction bias is visible in both, for instance, at time-point 1 for species Y₁₂ in and for species Y₉, Y₁₀, and Y₁₁ in . Overall, it seems hard to discriminate between two models on the basis of Figures 7 and 8 alone, and the BF criterion has to be invoked. The large BF value in (9) (BF = 2348.4) allows us to identify the model as having considerably larger posterior probability of being correct than the partially altered model ₂ of (B.2) and, consequently, to select the correct one.

FIG. 6.

Synthetic time-course data and fitted lines for and . Circles are synthetic data points; a blue solid line represents the correct model fitted to the data using the least-square method, and a red dotted line represents the fitted partially incorrect model .

FIG. 7.

The 95% posterior credibility bounds for . Simulated data (circle) versus the pure model prediction (red line) and the bias-corrected model prediction (black line) along with the 95% credibility bounds (blue dotted line).

FIG. 8.

The 95% posterior credibility bounds for . Simulated data (circle) versus the pure model prediction (red line) and the bias-corrected model prediction (black line) along with the 95% credibility bounds (blue dotted line).

3. Discussion

Selecting the model that best describes a given biological system is increasingly relevant in the modern era of systems biology and ever-improving molecular data collection methods. In the context of discriminating between competing kinetic ODE-based models of interactions among several biomolecular species, we presented here a possible solution to the problem with a new, conceptually simple and flexibly applicable data-based discrimination rule. The rule is based on analyzing the appropriate Bayes factor and draws from the fields of Bayesian analysis, computer model validation, and modern MCMC analysis. Specifically, the proposed rule is based on the pairwise comparisons of the model-specific posterior distributions induced by the data against those predicted by the corresponding ODE models and is similar in spirit to a longitudinal version of the celebrated Fisher's discriminant analysis. Due to its fully Bayesian implementation (as opposed to an approximation, like, e.g., the Bayesian information criterion), the method allows us to take properly into account the competing models' complexity, and hence, automatically penalizes models that attempt to over-fit the data. In two particular examples considered in this article—de-differentiation process of a mammalian salivary gland and a large synthetic model of nonlinear dynamics from the DREAM6 competition—the proposed method worked well, despite the small size of the longitudinal data set, multiple noise sources, a relatively high experimental variation between technical replicates, and parametric similarity of the competing ODE models. In spite of these challenges, the Bayesian criterion in both examples clearly identified the models with less bias and with biochemically more realistic dynamics as the more likely ones.

Whereas the immediate goal of the proposed method is to discriminate between the two competing, predefined models, one could possibly envision the use of a similar criterion sequentially in order to build up, in a stepwise process, the most likely posterior model among a set of dynamically proposed competitors. The potential candidates for comparisons could be generated based on the current model and either the available biological knowledge or a random walk across the model space. However, such model-building considerations, being computationally challenging, would with no doubt require further refinements of the currently proposed discrimination method.

The empirically demonstrated relative robustness and sensitivity of the proposed selection criterion indicates that it might be appropriate for use with other molecular models, particularly in sparse-data scenarios. While in our example we used a specific parametric form of the likelihood function underlying the data, both the parametric assumptions as well as the MCMC algorithms applied could be easily modified to accommodate the specifics of other systems. In our settings, the Gaussian likelihood model seemed justified by the approximation results from biochemical stochastic dynamics of aggregated cellular systems. However, when considering, for instance, data from next-generation (high-throughput) sequencing, the use of Poisson likelihood might be more appropriate. In general, with suitable adjustments due to the nature of various experimental data, the methodology presented here should be broadly applicable to a wide variety of biological settings and experimental designs.

4. Appendix A

Throughout this section, we use the terminology developed within the computer validation framework as described, for instance, in Bayarri et al. (2007). For further discussions and other recent alternatives, see, e.g., Zhan et al. (2011). Let v_il and be n posterior samples obtained by the converged MCMC algorithm as discussed in the article. The quantities of interest may be approximately estimated from these samples as follows. The estimated posterior means for v_i and b_i may be approximated by, respectively,

(A.1)

and

(A.2)

Similarly, the bias-corrected model prediction for may be approximated by

(A.3)

where is the vector of mean pure model predictions. The bias of the pure model prediction may be approximated by

(A.4)

Finally, the variance of the bias-corrected predictor (A.3) may be approximately calculated as

(A.5)

5. Appendix B

For the synthetic data analysis of Section 2.5, the correct ODE model is given by the following ODE system (“Model1” in DREAM6).

(B.1)

A competing model is defined by the following set of partially altered ODE, where the alterations (in the growth rates equations for Y₉ and Y₁₁) were made so as to preserve the overall model dynamics.

(B.2)

In order to generate a synthetic time-course dataset for 12 species (), we used the correct set of ODE models of (B.1), which is a simplified version of the Model1 from DREAM6. The parameters of both ODE models, as well as the initial conditions, followed the setting of Model1 (i.e., and Y₁(0) = Y₃(0) = Y₅(0) = Y₇(0) = Y₉(0) = Y₁₁(0) = 0 and Y₂(0) = Y₄(0) = Y₆(0) = Y₈(0) = Y₁₀(0) = Y₁₂(0) = 1). In the experiment described in the current article, we considered 4 time-points and 10 experiments for each species (which is consistent with typical DREAM6 scenarios). The time-course vector for each species is therefore , with , where t₁ = 1, t₂ = 2, t₃ = 4, and t₄ = 9. The bias b_i(t) was generated from the multivariate normal distribution with the mean vector 0 and the variance–covariance matrix C = [exp{−((t_m−t_n)/10)²}]_mn, , and the random error ε_ij (t) was randomly sampled from the multivariate normal distribution with the mean vector 0 and the diagonal variance–covariance matrix with diagonal element δ_i, where δ_i = [exp{−t_k/3}]_k, . The data for 10 time-course experiments we generated from (B.1) and used in the analysis under the parametric noise (4) model. Both the generated data and the least-squares fitted ODE model trajectories are presented in Figure 6.

Acknowledgments

The research of all the authors was partially funded by the U.S. National Institutes of Health under grant R01DE019243 to D.S.D. and G.A.R. The authors are grateful to the reviewers for their helpful comments.

Author Disclosure Statement

No competing financial interests exist.