Fitting thermodynamic-based models: Incorporating parameter sensitivity improves the performance of an evolutionary algorithm

Abstract

A detailed comprehension of transcriptional regulation is critical to understanding the genetic control of development and disease across many different organisms. To more fully investigate the complex molecular interactions controlling the precise expression of genes, many groups have constructed mathematical models to complement their experimental approaches. A critical step in such studies is choosing the most appropriate parameter estimation algorithm to enable detailed analysis of the parameters that contribute to the models.

In this study, we develop a novel set of evolutionary algorithms that use a pseudo-random Sobol Set to construct the initial population and incorporate parameter sensitivities into the adaptation of mutation rates in the model, using local, global, and hybrid strategies. Comparison of the performance of these new algorithms to a number of current state-of-the-art global parameter estimation algorithms on a range of continuous test functions, as well as synthetic biological data representing models of gene regulatory systems, reveals improved performance of the new algorithms in terms of runtime, error and reproducibility. In addition, by analyzing the ability of these algorithms to fit datasets of varying quality, we provide the experimentalist with a guide to how the algorithms perform across a range of noisy data.

These results demonstrate the improved performance of the new set of parameter estimation algorithms and facilitate meaningful integration of model parameters and predictions in our understanding of the molecular mechanisms of gene regulation.

1. Introduction

The process of gene regulation, and understanding the intricate machinery involved in controlling it, poses a very important question in modern biology: how do genomic DNA sequences direct complex interactions with proteins leading to the precise expression of genes? All living organisms have genes that must be turned on and off to regulate critical cellular processes. In many cases, ranging from bacterial growth to human disease pathways, the expression of genes must be finely tuned at precise levels, spatial locations and times [1, 7, 74]. The genomic regions of DNA that control gene expression are referred to as enhancers (see Figure 1) and can be highly divergent in different organisms [37, 64, 32, 34]. In eukaryotes, many of the rules underlying gene regulation remain unclear due to the complexity of eukaryotic gene regulatory systems, involving the concerted occupancy and interaction of many trans-factors; including multiple transcription factor (TF) proteins, cofactors, and nucleosomes on enhancers [54, 39, 5]. One would like to understand not only the basic properties of protein-binding events involved in gene regulation (i.e., the quantitative change in mRNA expression levels as one protein represses or activates transcription), but also gain the ability to model the complex nature of the molecular interactions driving gene expression.

Fig. 1.

Illustration of an Enhancer Region. The horizontal lines represent sequences of genomic DNA. The vertical lines correspond to the transcription start site, the nucleotide at which the messenger RNA coding region, or gene, begins. The blue region, just upstream of the TSS, depicts the region of DNA where regulatory proteins bind to control gene expression, the enhancer. Green circles (activators) and red squares (repressors) depict transcription factor proteins. Three different scenarios are depicted, representing the gene as ‘On’, ‘Off’, or ‘Partially On’ due to the different combinations of activators and repressors bound to the enhancer.

Experimentalists use diverse techniques to gather information regarding transcriptional regulation. Some studies identify the developmental consequences linked to a particular gene expression pattern, some determine what proteins are physically interacting to control a single gene or group of genes’ expression in a gene regulatory network, and others classify the environmental conditions under which gene expression patterns are altered. These studies typically address questions about specific genes using only experimentation in the lab [66, 2, 36, 4, 51, 41, 61, 53, 75, 26]. Other studies, addressing wider questions of gene regulatory circuitry begin in the lab and then further test hypotheses through the integration of statistical, mathematical or computational modeling tools [16, 80, 29, 27, 73, 12, 13, 28, 62]. In addition to choosing a suitable model for their particular system and obtaining high-quality experimental data, modelers are faced with other key considerations in such studies; including: which parameter estimation technique should be used [3, 31, 65], how to analyze performance regarding the limits of the model or sensitivity of the model to parameter perturbations [3, 18, 10, 45], and how much information the model can provide experimental biologists [16, 13, 62]. These three critically important considerations, and their relationships to different modeling approaches, are the main foci of this study. In particular, we aim to develop a parameter estimation algorithm to be used when fitting thermodynamic-based models, with the goal of: a) reducing model prediction error with respect to the experimental data, b) reducing runtime, and c) maximizing reproducibility. The overarching goal is to improve model performance to ultimately provide a better understanding of gene regulation.

2. Background

2.1. Mathematical Modeling of Gene Regulation

A variety of different mathematical models have been implemented to explore the molecular basis for the expression of individual genes, as well as large- and small-scale gene regulatory networks in Drosophila melanogaster [70, 55, 29, 21, 9, 12, 28]. Drosophila, more commonly known as the ‘fruit fly,’ offers particular advantages for studying gene regulation; their basal transcriptional machinery closely resembles that of all other animal species, including humans, they have very short life spans and high birth rates, enabling the acquisition of a large amount of high quality quantitative data. In addition, early development in the Drosophila embryo is very rapid, with nuclear divisions occurring approximately every nine minutes in the syncytial stages. As a result, static mathematical models that can capture a snapshot of the complex regulatory interactions at distinct time points in development are particularly valuable.

One particular class of static mathematical models that has been extensively integrated with experimental data are referred to as thermodynamic-based (or fractional occupancy) models [16, 10, 9, 65, 11, 13, 62]. Thermodynamic-based models are used to predict gene expression output at a particular developmental timepoint based on the regulatory DNA sequence within an enhancer and the transcription factor (TF) concentrations present. For a complete derivation of these models, please see [9] or [11].

Thermodynamic-based models have been successfully implemented by different research groups to study a variety of TFs and genes crucial to the proper development of Drosophila melanogaster [30, 80, 16, 27, 73, 13, 62]. Many of these studies have illustrated the importance of protein-protein interactions amongst TFs, such as cooperativity and short-range repression (often referred to as ‘quenching’), by analyzing enhancers in the early Drosophila embryo [30, 80, 16, 13, 62]. There have also been studies focusing on the inclusion of binding sites with different affinities for the TF that binds, allowing for a range of weak and strong binding sites, as well as binding sites with affinities that are modified through non DNA-binding events [63, 27, 60]. Additional thermodynamic-based modeling efforts have focused on the dual role of particular TFs as both activators and repressors [73, 35]. Taken together, all of these studies have improved our understanding of transcriptional regulation in Drosophila development by building upon our existing knowledge and the model framework.

It should also be noted that thermodynamic-based models have demonstrated parameter non-identifiability (i.e. two or more parametrizations producing equivalent results) in studies that have found multiple distinct parameter sets to be consistent (up to some given threshold) with a single experimental dataset [10, 20, 80, 60]. It has, however, become standard practice in thermodynamic-based modeling studies to report a single best parameter set, often after many iterative rounds of parameter estimation [30, 63, 27, 73, 35]. This possible lack of uniqueness while reporting of a single best parameter set highlights the importance of both the choice of parameter estimation strategy and the testing of a given strategy on synthetic data constructed from known parameter values.

2.2. Sensitivity Analysis

Analyzing how a particular model’s output is affected by varying parameter values is an extremely important aspect of modeling. This is particularly critical when dealing with high quality quantitative data and trying to interpret parameter values in terms of their relationship to specific biophysical and biochemical properties. This is achieved using methods collectively referred to as sensitivity analysis. Sensitivity analysis has been performed using many different methods, each falling into one of two broad categories, a local or global parameter sensitivity approach. Local approaches focus on a particular region in parameter space to obtain a local response of the model to each parameter. Global approaches aim to capture the response of the model to small changes in each parameter, but over the entire parameter space.

Local parameter sensitivity analysis is extremely intuitive, typically involving a simple calculation or approximation of the partial derivative of the model output with respect to the parameter of interest. The local sensitivity coefficient for an objective function, f , with respect to a parameter, x_i, is defined by:

$$\text{sensitivity}\hspace{0.33em}\text{coefficient}=\sigma =\frac{x_i}{f}\left(\frac{\partial f}{\partial x_i}\right)$$ (1)

The major advantage to using local parameter sensitivity analysis lies in the simple formulation and speed of computation [67]. The disadvantages of this simple approach are that 1) the analysis is only valid in a small region around a particular point in parameter space [15, 43, 77], and 2) the formula does not account for any parameter interactions (or compensation) [19, 67]. In many biological systems, including those that use thermodynamic-based modeling in Drosophila, the parameter space is quite large, as we frequently don’t have tight bounds on our parameter values; so the first disadvantage of local approaches can be quite costly in terms of overall performance. Additionally, most models implemented on biological systems are non-linear, thus the second disadvantage highlights the fact that the local sensitivity may underestimate the true model sensitivities. However, if the objective is to understand how small changes in a single specific parameter will affect the model output at a particular point in parameter space, this approach can be extremely informative.

Global parameter sensitivity analysis overcomes many of the disadvantages of local analyses, as its primary goal is to capture the sensitivity of the model to parameters over the entire parameter space and often incorporates parameter dependencies by calculating higher order sensitivity coefficients [43, 77, 19, 67]. Its major disadvantages lie in the fact that there are multiple different approaches for calculating global sensitivity indices, global approaches are often computationally more expensive than calculating local sensitivity coefficients, and global approaches rely on some sampling of parameter space [68]. Two common approaches for calculating global parameter sensitivity are the Fourier Amplitude Sensitivity Test (FAST) and ANalysis Of VAriance (ANOVA), also referred to as the High Dimensional Model Representation (HDMR), methods. Each of these methods relies on the same basic idea: to decompose the model output function into a summation of terms of increasing dimensionality and approximate sensitivity coefficients for each model parameter by exploring parameter space. This is done using a decomposition such as the Fourier decomposition or one constructed from a set of orthonormal polynomials. Here, we give the formulation used in the GUI-HDMR developed by Ziehn [77, 78]. This algorithm uses the Monte Carlo integration method to decompose the model output, f (x):

$$f\left(x\right)=f_0+\sum _{i=1}^n{f}_i\left(x_i\right)+\sum _{1\le i<j\le n}{f}_{ij}\left(x_i,x_j\right)+\cdots +f_{12\dots n}\left(x_1,x_2,\dots ,x_n\right)$$ (2)

where f₀ is the main effect (the overall mean model output), f_i (x_i) is a first-order term representing the effect of the parameter x_i acting independently on the model output, and f_ij (x_i, x_j) is a second-order term representing the effect of the parameters x_i and x_j on the model output. These f_ij terms represent the impact pairwise parameter interactions have on determining the model output. Higher-order terms represent higher-order interactions.

The HDMR algorithm first takes a pseudo-random generated sampling of parameter space using a Sobol Set. It then approximates each term in the expansion using orthonormal polynomial approximations. Although this method has the ability to calculate higher-order sensitivities, it has been shown that using only first- and second-order terms is often sufficient to approximate the total sensitivity in the system [38, 40, 10]. Thus, first- and second-order terms are typically normalized by the total variance to obtain the main effect of each parameter and the effects of pair-wise parameter interactions, referred to as first- and second-order sensitivity indices respectively [10, 45].

2.3. Parameter Estimation

When fitting a model to experimental data, one of the most important decisions to make is in choosing a parameter estimation algorithm. The major challenge is that no single parameter estimation algorithm has been found to show optimal performance on all problems or in all modeling contexts. Further, in many areas of mathematical and computational biology, such as thermodynamic-based modeling, many different parameter estimation algorithms have been implemented with little comparison or exploration to determine the best strategy for the given problem [16, 27, 30, 63, 80]. However, most mathematical modeling groups studying transcriptional regulation have turned to global approaches for parameter estimation [16, 35, 62].

Global parameter estimation techniques have demonstrated great success in finding global minima of an objective function when no a priori information on parameters is available [42, 72]. Both deterministic and stochastic global techniques have been developed, although deterministic methods are computationally very expensive and thus impractical for many nonlinear problems in which the problem of parameter estimation is inherently ill-posed. For this reason, stochastic methods such as genetic algorithms, simulated annealing, and evolutionary strategies have been implemented on thermodynamic-based models [16, 35, 62]. These methods move through parameter space with some stochasticity to avoid converging on local minima. Global convergence has been proven for evolutionary algorithms provided a unique global minimum exists, and a nonzero probability of reaching the neighborhood of that minimum from any initial starting population in a single evolutionary time step [56].

Evolutionary strategies have demonstrated excellent performance in many studies of continuous models dealing with high-dimensional parameter spaces [16, 31, 47, 48, 8, 65, 62]. These strategies were shown to be the only algorithms capable of efficiently fitting parameters in early investigations, and are derived using the basic biological principles of evolution, including recombination, mutation and selection [47](Figure 2). They evolve a population of parameter sets by ensuring that the parameter sets with the lowest value for the objective function have a higher probability of surviving within the population. Along with selection criteria, the use of mutation rates and recombination allows populations of parameter sets to leave local minima to reach a global minimum.

Fig. 2.

Flow chart of an evolutionary algorithm.

Multiple different modifications have been made to evolutionary algorithms in attempts to improve the algorithm’s ability to discover global minima as well as its computational speed [17, 52, 25, 23, 31, 46, 79, 76]. Some of these modifications include pruning parameters and introducing new termination criteria for fine-tuning evolutionary algorithms to specific parameter estimation problems [79, 76]. For use more broadly, many of these modifications have focused on self-adaptation of the mutation distribution, either by simply considering a decay in the mutation rate over time or by considering the distribution of mutation rates over many generations, including the cumulative step size adaptation, generating set adaptation and covariance matrix adaptation methods [52, 25, 23]. All of these methods using alternative mutation steps have been implemented on test functions and shown to be faster and more reliable in finding global minima of these test functions than previous methods. It should be noted, however, that none of these algorithms have incorporated any problem-specific knowledge other than that revealed by selection.

There have also been a number of recent studies, using specific biochemical systems and dynamic mathematical models, that have shown alternatives, such as Particle Swarm Optimization and Differential Evolution algorithms, outperform even the state-of-the-art Covariance Matrix Adaptation-Evolutionary Strategy (CMAES) method on their particular parameter estimation problem [8, 6, 71, 49, 50]. In [50], it is hypothesized that the poor performance by CMAES may be due to the ‘peculiar’ distribution of kinetic parameters. Further, in both [8] and [50], the authors suggest that modifications to evolutionary algorithms could result in highly competitive performance in terms of convergence speed and quality of results.

In this study we develop a novel set of evolutionary algorithms that use a Sobol Set to construct the initial population and incorporate model parameter sensitivities into the adaptation of mutation rates using three distinct methods (Local, Global and Hybrid). Comparison of the performance of these new algorithms to a number of existing global parameter estimation algorithms on; 1) four different test functions over a range of parameter dimensionalities and 2) synthetic datasets of varying quality generated by static mathematical models of gene regulation, reveals improved performance in many instances in terms of runtime, error and reproducibility.

3. Materials and Methods

3.1. Implementing a General Parameter Estimation Algorithm

Parameter estimation in this context refers to finding the global minimum of an objective function on a finite domain that the user provides [65]. Fitness for continuous or piecewise functions is calculated by substituting the estimated parameter values back into the equation and obtaining the function value at these parameters. The best parameter set would yield the absolute minimum function value. For fitting model parameters to a set of data points, the objective function used is the sum of the square difference between the model output and the data over all data points. The parameter estimation algorithm minimizes this Sum of Square Error (SSE), or fitness value, to find the best parameter set. Since this is the sum of squared numbers, the absolute minimum fitness value is zero [65]. Unfortunately, in biological systems it is very rare to have experimental data that fits a model exactly, so the algorithm searches to find a fitness as close to zero as possible.

In a basic evolutionary algorithm, the user typically inputs a population size, the number or percentage of the population that will be used as ‘parents’ in subsequent generations, the upper and lower bounds of parameter space, a fixed mutation rate, and stop criteria [59]. Additionally, the user specifies an objective function to use when calculating fitness. Some algorithms require the user to input an initial ‘guess point’ and the algorithm starts its search in a neighborhood of this starting point. The algorithm, when executed, starts by creating a random population of parameters or parameter vectors, and evaluates them relative to the objective function. The most ‘fit’ individuals are selected as parents and are used to breed offspring parameter sets for the next generation. The algorithm randomly selects two parents and creates an offspring that has some parameter vector entries the same as the ‘mother’ and the rest of the entries are those of the ‘father’; this is what is often referred to as ‘discrete breeding’ [57]. Each of these offspring then have all of these vector entries randomly mutated. For each vector entry, the algorithm generates a random number between −1 and 1 and multiplies it by the user-specified mutation rate, and then adds this mutation to the particular vector entry. In this way, offspring are created and this process repeats from generation to generation until stop criteria are met. When one of these criteria are met, the algorithm outputs the most ‘fit’ parameter set.

3.1.1. Stop Criteria

Evolutionary algorithms use a number of different stop criteria. The criteria we implement are:

• the maximum number of generations to run

• the number of generations the algorithm should check back for convergence (i.e. if the best fitness value has not improved, on average, by more than the user-specified error tolerance after ‘check’ generations, the algorithm stops).

3.1.2. Initial Population

As stated above, most evolutionary algorithms to date have used a randomly generated initial population [47, 58, 31, 44, 22]. However, in an attempt to improve both runtime and accuracy of solutions, all of our new algorithms described in the next section have been run using an initial population constructed from a Sobol Set. The Sobol Set consists of a pseudo-random sampling of parameter space, thus ensuring the algorithm at least initially explores all areas of parameter space, avoiding the possibility of a randomly initialized population only exploring a small subset of parameter space. The population size is calculated as a function of the number of parameters, n:

$$\text{Population}\left(n\right)=10⌈\frac{\left(121451\right)\left(1.721\right)e^{1.148n}}{121451+1.721\left(e^{1.148n}-1\right)}⌉$$

This function was optimized, as described in Section 1 of the Supporting Information.

3.2. Incorporating Parameter Sensitivities

All of our methods and functions have been written in MATLAB and are available publicly on GitHub. To incorporate parameter sensitivities into an evolutionary algorithm, we construct mutation rates as a function of parameter sensitivity. In general, parameters with high sensitivity are assigned lower mutation rates, and parameters with low sensitivity are assigned high mutation rates. This allows the algorithm to more efficiently explore parameter space, not spending too much time (using larger mutations) optimizing less sensitive parameters, while carefully searching (using smaller mutations) parameter space near more sensitive parameters. We apply two main approaches for calculating sensitivities, and for both methods our mutation rate is set to $\text{Mutation}=\frac{1}{\left|\text{SC}\right|}$ where SC is the sensitivity coefficient. This simple absolute value of the reciprocal mathematically has the characteristics we desire for mutation rates, and is computationally inexpensive. We tested a variety of similar monotonically decreasing functions, but found that this simple reciprocal was the least computationally expensive, and was generally the most effective in terms of accuracy.

We additionally scale our mutation function by$\frac{1}{n}$, where n is the number of parameters, making our final mutation function: $\text{Mutation}=\frac{1}{n\left|\text{SC}\right|}$This scaling is incorporated to avoid unrealistically high mutation rates. This would occur because when calculating global sensitivities, all coefficients are between zero and one, and as the coefficient approaches zero, the mutation rate approaches infinity. Thus, an offspring could be over-mutated in these circumstances instead of helping to fine tune the parameter value. To ensure that our mutation rates are still not too large, the program also checks that the mutation will not cause an offspring to be located outside the parameter space given by the upper and lower bounds. After mutation, if any vector entry is out of the parameter boundaries for that particular parameter, it is set to the closest boundary value (either the upper or lower bound for that particular parameter).

3.2.1. Local method

The first new method we develop is referred to as the ‘Local method’. This method calculates local sensitivities for each parameter in each offspring at every generation and assigns each offspring its own mutation rates. The local sensitivity coefficient is calculated using the limit definition of a partial derivative, with h = 10⁻⁶ [10].

3.2.2. Global method

The second new method is referred to as the ‘Global method’. This method calculates global sensitivities using a modified script version of the HDMR software created by Ziehn and Tomlin [78]. The algorithm uses the function CoefOrthPolyLeg15.m initially and sets max_1st to 10 and max_2nd to 5 in the HDMR software package. The sample population used for the HDMR algorithm is a Sobol Set of size max (2ⁿ⁻¹, 1024). The Global method calculates global sensitivities for each parameter, and then uses them to calculate mutation rates. These mutation rates are fixed and do not change from generation to generation.

3.2.3. Hybrid method

In developing a method with the goal of combining the accuracy of the Local method and the speed of the Global method, the third new method is referred to as the ‘Hybrid method’. This method initially calculates global sensitivities and corresponding mutation rates using the HDMR software and fixes mutation rates for 100 generations. For every additional 100 generations the algorithm runs, the Hybrid method takes the current population and calculates local sensitivities for each of the individuals of that generation. The absolute value of each of the sensitivities for each parameter is then averaged over all individuals to obtain new sensitivity coefficients and corresponding mutation rates to be used for the next 100 generations. This process is repeated every 100 generations until one of the stop criteria is met.

3.3. Existing Methods for Comparison

3.3.1. Genetic Algorithm

Since we are using MATLAB to write and test all of our algorithms, we compare our methods to MATLAB’s own Genetic Algorithm (GA), ga.m from MATLAB’s Global Optimization Toolbox [44]. MATLAB’s GA uses similar stop criteria and we can input our own parameter bounds, population sizes, etc. into this algorithm. Comparing our new algorithms to this algorithm serves as a performance benchmark.

3.3.2. Covariance Matrix Adaptation Evolutionary Strategy

The Covariance Matrix Adaptation Evolutionary Strategy (CMAES), which was first developed by Nikolaus Hansen in 1996 [23], has been used extensively in previous modeling studies [24, 65, 62]. The MATLAB code for this algorithm is available online, and we use it to compare our own algorithms in terms of speed and accuracy [22]. CMAES already has its own default inputs, including a population sizing function [22]. We left these defaults unchanged, but provided inputs for the upper and lower bounds of parameter space, the error tolerance, and the maximum number of generations. We kept these input values the same across all of our evolutionary algorithms. CMAES further requires an initial guess point. In all of our simulations, we used a random initial point from a normal distribution on the parameter space as the initial guess point each time the method was executed [22]. One should also note that CMAES implements a number of additional ‘built-in’ stop criteria compared to our algorithms.

3.3.3. Particle Swarm Optimization

The Particle Swarm Optimization (PSO), developed by Kennedy and Eberhart in 1995 [33], is similar to a genetic algorithm in the sense that it is purely stochastic, but is a distinct approach that is meant to mimic the behavior of insects swarming by moving ‘particles’ through space after evaluating an objective function at each time step and using that information to change the velocity of the particles. It has been shown to be a competitive parameter estimation algorithm on a number of problems in systems biology [69], and is available (particleswarm.m) as part of MATLAB’s Global Optimization Toolbox [44]. We ran PSO with all the default settings in MATLAB, except for maximum iterations, which we set to 10⁵, and function tolerance, which we set to 10⁻⁶.

3.3.4. Enhanced Scatter Search

The Enhanced Scatter Search (eSS) metaheuristic method, developed in 2009 [14] is a hybrid global-local method. It combines a global stochastic search algorithm with local searches in an attempt to explore large parameter spaces with an accelerated convergence time. The MATLAB code for this algorithm is available online [69]. To compare the performance of eSS to our algorithms, we used ess kernal.m with all the default settings, except for max function evaluations, which we set to 10⁵, and tolc, which we set to 10⁻⁶.

3.4. Evaluating the Performance on Thermodynamic-Based Models: Synthetic datasets

Synthetic datasets were constructed using three simple thermodynamic-based models [9]. These models represent three different very important potential biological interactions taking place at an enhancer: quenching (or short-range repression) between a bound repressor and a bound activator, protein-protein cooperativity between two activators, and competition between an activator and repressor for a single TF binding site. Table 1 describes the three models and their characteristics in detail.

Table 1.

Simple thermodynamic-based models

Model	Equation	Parameters	Values (for Synthetic datasets)	Parameter Bounds
Quenching	$\frac{K_A\left[A\right]+Q{K}_A{K}_R\left[A\right]\left[R\right]}{1+K_A\left[A\right]+K_R\left[R\right]+K_A{K}_R\left[A\right]\left[R\right]}$	KA = activator BA KR = repressor BA Q = probability of activation	KA =5 KR =10 Q ={0.0, 0.1, … , 0.8, 0.9}	KA ∈ [0, 20] KR ∈ [0, 20] Q ∈ [0, 1]
Cooperativity	$\frac{2K_A\left[A\right]+C{K}_A^2{\left[A\right]}^2}{1+2K_A\left[A\right]+C{K}_A^2{\left[A\right]}^2}$	KA = activator BA C = cooperativity coefficient	KA =5 C ={0.0, 0.25, … , 2.25, 2.5}	KA ∈ [0, 20] C ∈ [0, 10]
Competition	$\frac{K_A\left[A\right]}{1+K_A\left[A\right]+K_R\left[R\right]}$	KA = activator BA KR = repressor BA	KA ={5, 6, … , 9, 10} KR ={5, 6, … , 9, 10}	KA ∈ [0, 20] KR ∈ [0, 20]

3.4.1. Construction of Continuous, Perturbed, Binary, and Interpolated Data

The quality of experimental data has varied greatly in different modeling studies that have focused on transcriptional regulation [63, 16, 27, 35, 62]. Experimental data can be extremely noisy, and in many cases, qualitative at best. To compare the parameter estimation algorithms’ performance on datasets of varying quality, we constructed continuous, perturbed, binary, and interpolated synthetic datasets using the same process as in [65]. The thermodynamic-based models described above were used to construct 57 continuous synthetic datasets. 10 datasets were created using the Quenching model (fixing K_A = 5, K_R = 10, and varying Q from 0.0 to 0.9), 11 datasets were created using the Cooperativity model (fixing K_A = 5 and varying C from 0.0 to 2.5), and the remaining 36 datasets were created using the Competition model (varying both K_A and K_R from 5 to 10). The parameter values and combinations chosen provided a variety of expression patterns and avoided saturation in predicted gene expression. Perturbed data was generated from these datasets by adding to each data point, x, a perturbation drawn from a uniform distribution on the interval [−rx, rx], where r represents the perturbation percentage. We used r = 0.025, 0.05, … , 0.225, 0.25. Results shown in Figure 5 were obtained with r = 0.025. Binary data was created from the continuous data using a threshold of 0.5, and corresponding interpolated data was then created by replacing data points near a discontinuity with values such as 0.89 or 0.11 to smooth out the discontinuity using the same process as in [65, 27, 63]. An example (created using the Quenching Model with Q = 0.5) is shown in Figure 3.

Fig. 5.

Performance on Thermodynamic-based Models. x-axes represent the 57 synthetic datasets created using the Quenching (1–10), Cooperativity (11–21) and Competition (22–57) models. (A)-(D) show the average Parameter SSEs obtained from datasets of varying quality: (A) continuous, (B) perturbed, (C) binary, and (D) interpolated.

Fig. 3.

Example of Synthetic Data. This is an example of a set of synthetic data, created using the Quenching Model with K_A = 5, K_R = 10, and Q = 0.5. The perturbed data was created using r = 0.20 and binary data was created from the continuous data using a threshold of 0.5. Interpolated data was created as in [65, 27, 63].

3.4.2. Objective / Fitness Function

For each test run on the synthetic datasets, the objective function used was the Sum of Square Error (SSE), the sum of the square difference between the synthetic dataset generated using the parameter values and bounds given in Table 1 and the corresponding model predictions (i.e. Quenching, Cooperativity and Competition) over all data points in that dataset.

3.4.3. Parameter SSE

For all of the continuous functions as well as synthetic datasets we use to test our algorithms, the ‘true’ parameter values are known; for continuous functions these refer to the point in parameter space where the global minimum of the function occurs, and for synthetic datasets these refer to the parameter values used to generate the synthetic data. To measure each algorithm’s ability to find the true parameter values, we calculate the Parameter SSE, the sum of the square difference between the true parameters and the parameters found by the algorithm over all n parameters. All data shown in Figures 4–5, labeled ‘SSE’, refers to the Parameter SSE.

Fig. 4.

Random vs. Sobol Initial Population. All methods were run on the Griewank function in one-dimensional parameter space. (A) shows the Parameter SSEs obtained from a random initial population and (B) shows the Parameter SSEs obtained from an initial population selected using a Sobol Set.

3.4.4. Success Rate

The Success Rate (SR) of an algorithm is defined as the fraction of searches that reached some predefined objective function value or ‘value to reach (VTR)’ within the maximum computational time allowed [69]. On the synthetic datasets, the SR is calculated for each individual dataset. All data shown in Figures 6–7 were generated using a VTR = 10⁻¹⁰, averaging 1/SR over datasets created by the same model (Cooperativity, Competition or Quenching) in Figure 6 and averaging 1/SR over all 57 datasets in Figure 7.

Fig. 6.

Performance Evaluation on Thermodynamic-based Models with Clean Data. x-axes represent the average computational time (minimum time required to reach the VTR) and y-axes represent the average 1/SR. In all panels, the results shown were generated using a VTR = 10⁻¹⁰. The average computational times and 1/SR values were calculated using datasets generated from the same thermodynamic-based model type: (A and D) Quenching (datasets 1–10), (B and E) Cooperativity (datasets 11–21), and (C and F) Competition (datasets 22–57). Panels (A)-(C) show values generated from all seven parameter estimation algorithms. Panels (D)-(F) restrict the x-axis to better illustrate the different performance of those models with significantly lower runtime, removing the values generated using eSS from the figure.

Fig. 7.

Overall Performance Evaluation on Thermodynamic-based Models with Clean Data. x-axes represent the average computational time (minimum time required to reach the VTR) and y-axes represent the average 1/SR. In both panels, the results shown were generated using a VTR = 10⁻¹⁰. The average computational times and 1/SR values were calculated using all 57 synthetic datasets. Panel (A) shows values generated from all seven parameter estimation algorithms. Panel (B) has restricted the x-axis to better illustrate the different performance of those models with significantly lower runtime, removing the values generated using eSS from the figure.

4. Results

4.1. Generating the Initial Population

Most global optimization algorithms currently implemented for parameter estimation use a randomly generated initial population, either completely random within the bounded parameter space or randomly chosen in a neighborhood of some initial guess [44, 22]. In all of our newly designed algorithms, we first tested whether a randomly generated initial population or a pseudo-random initial population would result in differences in these algorithms’ ability to reproducibly find global optima. The results were striking. In almost all functions tested, random initial populations resulted in a clear bimodality of solutions (Figure 4A). In contrast, when we used a pseudo-random initial population, chosen using a Sobol Set, the bimodal distribution of solutions for the algorithms was avoided and the algorithms robustly performed as well or better than the random initial population in every case (Figure 4B). Note that in Figure 4, all three new algorithms (Local, Global and Hybrid) were run on the Griewank function in one-dimensional parameter space (see Additional File 1 for description of Griewank function). All inputs to the algorithm (maximum number of generations, error tolerance, and number of generations to check back for convergence) were kept constant for all runs. The only difference in the runs in Figure 4A and Figure 4B was the initial population. To test for consistency, each run was repeated 20 times. When a randomly generated initial population was used, one can observe clear bimodality in the results; the Par ameter sum of square error (SSE) obtained fell into the range [10⁻¹¹, 10⁻⁴] or [10, 10⁴] (Figure 4A). In contrast, when a Sobol Set was used to generate the initial population, the Parameter SSE obtained from all sixty runs fell into the range [10⁻¹², 10⁻⁶] (Figure 4B). Thus, the reproducibility seems to be greatly improved when the algorithm starts with an initial population generated from a Sobol Set. Therefore, for the remainder of our study, and all of the results shown in Figures 5–8, our newly developed algorithms were run using an initial population generated from a Sobol Set.

Fig. 8.

Impact of Noise on Fitting Thermodynamic-based Models. (A)-(C) show the results from fitting parameter values using the Hybrid method on synthetic datasets created using the Quenching (A), Cooperativity (B), and Competition(C) models.

4.2. Analysis of Synthetic datasets

To directly address the question of whether the quality of data affects the relative performance of all seven parameter estimation algorithms, we created synthetic datasets of varying quality using thermodynamic-based models (see Methods section for details and Figure 3 for an example of the synthetic data). Our results demonstrate that, in general, our Local and Hybrid methods outperform all other methods on continuous non-noisy data, with the only exception being the comparable SSE values found using the eSS algorithm (see Figure 5A). However, one should note the significantly longer runtime of the eSS algorithm (see Figure S4A, eSS runtimes > 60 seconds on all 57 datasets, compared to the Local and Hybrid methods with runtimes < 6 seconds on all 57 datasets). In addition, as we look at lower quality perturbed, binary, and interpolated datasets, the performance of the seven algorithms becomes almost indistinguishable (see Figure 5B-D).

To further evaluate the performance of all seven parameter estimation algorithms on continuous non-noisy data, we calculated the average computational time and the inverse of its success rate on each of the 57 datasets. We then averaged these values over datasets generated from the same thermodynamic-based model type (Quenching, Cooperativity, and Competition), shown in Figure 6, and over all 57 datasets, shown in Figure 7. When interpreting these figures, the methods illustrating the best performance are those located toward the bottom (i.e. high success rate) and left (i.e. low computational time) of the plot. Our results support the well-known result that performance of methods is dependent on the specific problem considered, as the results vary from model to model (see Figure 6). However, in every model type the eSS algorithm has a significantly longer mean computational time (see Figure 6A-C) and our new Hybrid method results in the highest success rate out of the six algorithms with comparable mean computational times (see Figure 6D-F). Overall, this analysis again shows the superior performance of our new Hybrid method when fitting thermodynamic-based models to continuous non-noisy data (see Figure 7).

We performed the Mann Whitney U Test on the parameter SSE values to validate our comparison of the parameter estimation algorithms. We found that all pairwise parameter SSE values were significantly different except our Local and Hybrid methods on the Quenching Models (Tables S21-S23 - Additional File 2).

5. Discussion

Our results demonstrate that the selection of the initial starting point(s) in parameter space in an evolutionary algorithm can profoundly impact the subsequent performance of the algorithm. For each of the three new Local, Global and Hybrid algorithms we developed in this study, a randomly generated initial population resulted in a distinct bimodality of final Parameter SSE scores (Figure 4A). In contrast, a reproducibly improved performance, in terms of the tighter clustering of lower SSE scores, could be achieved by using a pseudo-random starting population chosen using a Sobol Set (Figure 4B). This phenomenon was echoed in the results obtained from the pre-existing GA, CMAES, and PSO algorithms, all of which utilize a randomly generated initial starting population [44, 22] and demonstrated varying degrees of bimodality of their SSE scores on different test functions (Figure S3). In combination, these results therefore suggest that evolutionary algorithm performance can be improved by using a pseudo-randomly selected starting population. One likely explanation for this result is that as the pseudo-random population more fully explores the initial parameter space, it prevents the algorithm from becoming trapped in local minima and consequently enhances the probability of converging on the true global minimum for any given function.

One should also note that we were limited in the dimensionality we could test due to the high computational cost of all algorithms tested. A potential alternative approach would be to consider modifying the backbone evolutionary strategy to run in a parallel manner. This has been done, and shown successful in modeling gene regulatory networks, using a ‘parallel island evolutionary strategy’ [31]. Using a similar strategy could greatly reduce the computational time, and allow us to test the algorithms on much higher dimensional models; however, this is beyond the scope of this study and will be considered as a future direction.

6. Conclusions

The goal of testing the parameter estimation algorithms was to determine their performance on different test functions, and consider the trade-off between Parameter SSE, runtime, and reproducibility. For modeling groups working in mathematical and computational biology, choosing the best strategy when fitting experimental data can be extremely important, yet is a difficult decision. For example, most mathematical biologists studying transcriptional regulation using static models have implemented global approaches for parameter estimation [16, 35, 62]. However, to date there is not a single, universal approach used in all such studies that has proved to be optimal. In our current study, taking into account all three important factors: minimizing Parameter SSE, minimizing runtime, and maximizing reproducibility, our new Hybrid method emerges as the leading algorithm when compared to the six other algorithms (Figures 5–7 and S2-S4). Not surprisingly, the Hybrid method always runs faster than the Local method, but frequently doesn’t sacrifice performance, as measured by Parameter SSE score and Success Rate, when compared to the Global method. In many cases, particularly for the more complex test functions and in higher parameter dimension space, the Hybrid method also reproducibly outperforms the GA and CMAES algorithms, performing at a level comparable to that of the PSO algorithm, without exhibiting any of the bimodality in results (Figure S3). Taken together, these results indicate that our novel Hybrid method, incorporating calculated parameter sensitivities, may be well suited to approaches that utilize evolutionary algorithms to investigate parameters in a range of models.

We directly addressed this question in the context of thermodynamic-based models of gene regulation by testing the performance of the seven different algorithms on 228 different synthetic datasets of varying quality (Figure 5). The Local and Hybrid methods outperform all other methods on high quality datasets with only few exceptions in which the eSS algorithm resulted in lower Parameter SSE values and higher Success Rates but much longer runtimes (as represented by continuous data, Figures 5A, 6, 7, and S4A). However, this was not the case for the perturbed, binary or interpolated datasets (Figure 5B-D). As expected, the Hybrid method has the added advantage of running approximately 2.7 times more quickly than the Local method on average over these 57 datasets (Figure S4A). This result suggests that the Hybrid method should ideally be implemented on complex datasets of high quality (i.e. low levels of noise) and that the choice of parameter estimation method may not be as important on lower quality datasets. One should note that all of the approaches we have implemented here are global parameter estimation approaches, although we have not exhaustively tested all parameter estimation algorithms available. This conclusion, however, regarding the importance of the choice of parameter estimation method on low quality datasets is in-line with the results from Suleimenov et al, comparing the performance of local and global approaches [65].

In order to provide the biologist with some insight into how experimental noise may impact modeling studies, we also investigated the performance of each algorithm as noise was added to each of the 57 synthetic datasets in our thermodynamic-based gene regulation models. The results were consistent across all seven parameter estimation algorithms on all models; as the data becomes noisier, the ability of the algorithm to fit the data accurately decreases and the range of scores becomes broader (Figure 8 and Supporting Information, Figures S2-S7). A comprehension of this reproducible pattern is therefore informative for the integration of modeling and experimental data, irrespective of the specific evolutionary algorithm being utilized. Essentially, what it provides is a guide to how ‘clean’ the experimental dataset needs to be to result in meaningful model fits and predictions. Conversely, in the case where experimental data has already been collected, one could use the noise/fitness relationship to better understand the weight that should be placed on parameter values fit by a particular algorithm in the associated model and the interpretation of the predictions from the model.

Highlights.

• A critical step in modeling transcriptional regulation is choosing the most appropriate parameter estimation algorithm.

• Evolutionary algorithms that use a pseudo-random Sobol Set to construct the initial population outperform those that use a random initial population.

• Evolutionary algorithms that incorporate parameter sensitivities into mutation rates exhibit improved performance compared to existing algorithms in terms of runtime, error and reproducibility.

• Analyzing the ability of new evolutionary algorithms to fit datasets of varying quality provides the experimentalist with insight into applying these methods across a range of noisy data.