Programmatic modeling for biological systems

Abstract

Computational modeling has become an established technique to encode mathematical representations of cellular processes and gain mechanistic insights that drive testable predictions. These models are often constructed using graphical user interfaces or domain-specific languages, with community standards used for interchange. Models undergo steady state or dynamic analysis, which can include simulation and calibration within a single application, or transfer across various tools. Here, we describe a novel programmatic modeling paradigm, whereby modeling is augmented with software engineering best practices. We focus on Python – a popular programming language with a large scientific package ecosystem. Models can be encoded as programs, adding benefits such as modularity, testing, and automated documentation generators, while still being extensible and exportable to standardized formats for use with external tools if desired. Programmatic modeling is a key technology to enable collaborative model development and enhance dissemination, transparency, and reproducibility.

Introduction

Mathematical modeling of cellular processes for mechanism exploration is becoming widespread [1–6], but challenges remain as to how models should be built, calibrated, analyzed, and interpreted to extract much-needed mechanistic knowledge from experimental data. Historically, methods and techniques from other fields have been directly imported to systems biology with varying success. For example, early interpretations of cellular processes as circuits provided insights about basic regulatory motifs that could explain cellular behaviors [7] and insights from chemical engineering provided the foundations of flux analysis and metabolic modeling [8–11]. Similarly, techniques from chemistry [12], physics [13,14], and various engineering disciplines [15,16] have been used to model cellular processes, but due to their associated spatiotemporal complexity, from femtosecond/nanometer electron transfer reactions to years and meter scales in tumor growth, no established paradigm has emerged to capture the full complexity of a cell. Multiple tools have been developed with differing modeling objectives, fields, and target audiences. For example, COPASI [5], RuleMonkey [17], Simmune [18], and StochSS [19] all provide graphical user interfaces that cater to non-expert modelers wishing to encode mechanistic representations of biological processes. More abstract approaches such as BioNetGen [20] and Kappa [21], employ a domain-specific language (DSL) to describe and simulate models. However, most tools are self-contained platforms with a small set of included methods and analyses, limiting access to other standalone simulation tools such as StochKit [4], SciML [22], URDME [23], and SmolDyn [24] to name a few. Similarly, optimization techniques ranging from vector-based optimization methods [25,26] to probabilistic-based methods [27,28] exist in yet another isolated domain. Therefore, the current modeling and simulation ecosystem is compartmentalized and fractured, and thus, unification and intercompatibility efforts are sorely needed.

Valuable efforts toward unification have been put forth to create standards for model instantiation, simulation, analysis and dissemination [29–35]. Of these, Systems Biology Markup Language (SBML) is perhaps the most successful to date. While there are many useful models and frameworks that do not use or predate SBML (for a kinetic metabolic modeling review, see [36]), SBML compatibility has emerged as a valuable standard for reproducibility and interchange. Despite these efforts, mathematical modeling for cell biology remains challenging to scale; both vertically (larger, more complex models) and horizontally (more active collaborators). While mathematical tools are the obvious way forward to describe cellular processes, the community knowledge base is highly domain specific due to the substantial complexity involved, with some notable exceptions [37].

A novel, more flexible approach to encode knowledge about biological processes as computer programs is slowly emerging and gaining momentum [3,38,39]. In this approach, biological models are no longer static information, but computer code that is executable, modular, and aggregates community knowledge towards crowd-driven mathematical models. Although computer languages like Lisp [40] and proprietary packages such as MATLAB have been proposed toward this goal, we believe Python provides the largest open-source ecosystem, myriad learning resources, and large applicable base to unify modeling practices in the field. Adopting a programmatic modeling paradigm for systems biology automatically accrues decades of computer science practices including structured documentation, integrated development environments (IDEs), (model) version control, code-sharing platforms, code testing frameworks, and importantly, literate programming/computational notebook dissemination. Here, we review the recent developments in programming-based approaches for systems biology. The structure of the manuscript is motivated by the model specification, simulation, calibration, analysis, and visualization paradigm/pipeline, commonly practiced in systems biology. To guide the reader, we provide a list of major Python modeling frameworks and their features (Table 1) and useful Python packages and services to support programmatic modeling (Table 2). Throughout the manuscript, we note how this approach could be supplemented and improved by incorporating best practices from software engineering (Figure 1).

Table 1:

List of major modeling frameworks in Python. BSD and MIT are permissive software licenses. GPL is a “copyleft” software license.

Frame-work	License	Field/Focus	Model Format(s)	Programmatic modeling	ODE simulations	Spatial modeling	Stochastic modeling	GPU acceleration	Steady-state analysis
PySB [3]	BSD	Cell signaling	PySB1 SBML BNGL2	✔	✔	✔3	✔	✔	✖
Tellurium [39]	Apache	Cell signaling	Antimony4 SBML	✔5	✔	✔3	✔	✖	✔
PySCeS [86]	BSD	Metabolic	Custom DSL, SBML	✖	✔	✔3	✖	✖	✔
ScrumPy [87]	GPL	Metabolic	Custom DSL	✖	✔	✖	✖	✖	✔
COBRAPy [6]	GPL	Metabolic	COBRAPy SBML	✔	✖	✔3	✖	✖	✔

¹Python-embedded, general purpose programming language (GPPL) approach

²BioNetGen language

³Compartment-based only

⁴Domain-specific language (DSL)

⁵Function support within DSL

Table 2:

List of key tools and services to support programmatic modeling in Python. BSD, MIT, and PSF are permissive software licenses. GPL is a “copyleft” software license.

Tool or service	License	Notes
Testing
PyTest	MIT	Testing framework; pytest.org
GitHub Actions	Free* service	Continuous Integration; github.com/features/actions
Circle CI	Free* service	Continuous Integration; circleci.com
Calibration
PyBioNetFit [61]	BSD	BNGL and SBML models
PyPESTO [62]	BSD	SBML and PEtab support
PyDREAM [27]	GPL	PySB interface
Analysis & Visualization
Matplotlib	PSF	Plotting library; matplotlib.org
Jupyter Notebooks	BSD	Computational notebooks; jupyter-notebook.readthedocs.io
PyVIPR [73]	MIT	PySB, Tellurium interfaces
Sharing and modification
GitHub	Free* service	Code hosting and collaboration suite; github.com
Sphinx	BSD	Documentation framework; sphinx-doc.org
Readthedocs	Free* service	Automated documentation compiler and hosting; readthedocs.io

^*Services are free for open-source projects.

Figure 1:

The traditional modeling paradigm in systems biology entails model building, simulation, calibration, and analysis (left column), which is carried out with myriad tools and practices. Software engineering practices can add much needed structure to the practice through maintenance, testing, documentation and sharing paradigms (right column), vetted by the software community.

Model specification

Traditionally, encoding a model of biochemical reactions would require the user to write each equation by hand, encode these into a solver, and run the simulations [41]. Although this is still common practice for smaller model systems, these lists of equations often lead to a model dead-end as the biological context is completely lost in the mathematical representation, which hinders model reuse. Reaction-based modeling formats add one layer of abstraction where the user instead writes chemical reactions of the form □+ □ ↔ □ in a program-specific notation and the computer parses this information into a mathematical representation [42]. These DSLs can operate either through a GUI that generates the code in the background, or directly through a text editor [20,21,42]. However, signaling pathways often comprise a large number of molecular complexes, which can assemble in multiple orders, leading to a large number of reactions and intermediate species during complex assembly and degradation. Therefore, traditional approaches become unwieldy as model systems become larger, leading to model dead-ends. A level of abstraction to alleviate the problem of manual reaction enumeration is presented by rule-based modeling formalisms whereby reaction rules rather than explicit reactions (or equations) are used to encode the system [3,20,21]. A reaction rule is a template for reaction patterns to be enumerated and instantiated recursively, starting from a defined list of initial species, thereby saving the user time and reducing error-prone repetition. In rule-based approaches, the reaction center (the relevant molecular components for a given reaction) is separated from the context (attached molecular components which have minimal or no effect on the reaction). These approaches often require a pre-processing step to generate the network of nodes (chemical species) and edges (chemical reactions) from the initial pool of chemical species and a set of reaction rules. However, network-free methodologies have been proposed to bypass the network generation step [43].

Model specification has also been embedded into General Purpose Programming Languages (GPPL) to provide a more powerful approach to biological modeling. In this programmatic modeling paradigm, the model is encoded as executable computer code, thereby offering all the advantages of a full-fledged computer programming language (Figure 2) for all aspects of model manipulation. Modularity, in which a model can be split into smaller, reusable code objects, is perhaps the most useful aspect for cell biology modeling. For example, PySB currently includes a library of 25 macros (small modules or functions) that encode reaction patterns commonly found in biology such as catalysis, inhibition, or oligomerization. From a user perspective, GPPLs have greater integration with Integrated Development Environments (IDEs) than DSLs, thus allowing syntax highlighting and checking, as well as navigation between functions. The model is also inspectable at runtime, allowing searching and filtering of model components. For example, a user could check whether certain species or reactions are present before simulation commences. Currently, the most used modeling frameworks to model non-equilibrium processes that use the programmatic modeling paradigm in Python are PySB [3], written in and using Python, and Tellurium [39], which is written in Python but uses Antimony [42], a DSL with function support, for model specification. For constraint-based metabolic modeling, COBRAPy [6] models are encoded in Python with a broad array of analysis types, including access to the COBRA Toolbox [44].

Figure 2:

Levels of abstraction in programmatic modeling. Models are composed of modules and macros, which are handled by the programming language interpreter/compiler; rules encode sets of reactions using structured pattern templates; reactions specify biochemical species’ transformations; and finally equations are handled by an ODE integrator or simulation algorithm directly.

Model simulation

Model simulation involves numerically solving the model equations to obtain trajectories for dynamically controlled species. Concentrations or molecule counts of chemical species in the model are the most commonly simulated quantities. Integration of systems of ordinary differential equations (ODEs) for deterministic simulations is the most common model simulation approach. Many ODE integrators are available and the best choice depends on model stiffness, desired integrator tolerances, and other requirements. In Python, a family of integrators is available through SciPy [45] including VODE and LSODA, but many other solvers have been proposed. Other commonly used solver suites include StochKit (Stochastic Simulation Algorithm) [4,46], BioNetGen (CVODE, SSA, tau-leaping algorithm, partition-leaping algorithm) [20], cupSODA (GPU ODE) [47], GPU_SSA (GPU SSA) [48], SUNDIALS [49], and Libroadrunner (CVODE, SSA) [50]. Within the Python ecosystem, PySB provides a simulation class that enables users to use many of these simulation tools or to connect new tools as needed. In addition, users of other Python-based tools such as Tellurium can also take advantage of these resources. We note that steady-state simulation tools can also be used, such as flux analysis using COBRApy [6] and pyTFA [51].

Model calibration

Model calibration is the process of adjusting model parameters to match experimental data, also known as parameter estimation/optimization when applied to parametric models. The most common form of model calibration involves a process of running many simulations (thousands to millions or more) and checking the distance between model and experimental data error using an objective function, which gives a measure of the model’s simulation “error” versus experiment; for a review see [52]. Since dynamic data for signaling models are hard to come by, the modeler often only has data for a few species, and thus model calibration often leaves a model underdetermined - multiple parameter sets fit the data equally well [53]. The concept of parameter “sloppiness” states that only a few “stiff” combinations of parameters are important in determining model outcomes, and others are “sloppy” and have little effect. Thus, an undetermined model can still be useful in predicting biological properties [54]. However, the interpretation of large, underdetermined models in the context of limited data is still up for debate. Lessons from e.g. hydrology and climate modeling have been highly influential toward addressing these issues [27,55,56].

The landscape of model parameters is often envisioned as a multidimensional surface with “height” representing the objective function, where the (ideally global) minimum or minima (representing the best fit(s)) must be found. SciPy [45], for example, includes gradient descent and simplex-based methods. However, the curse of dimensionality means that local optimization can give far-from-globally optimal results as the number of model parameters increases. Finding the global minimum of a multivariate nonlinear model is NP-hard [57], however several methods can make statistically good approximations. Markov Chain Monte Carlo sampling methods are among the most popular algorithms [58]. General purpose optimization toolkits for Python include SciPy.optimize [45] and Pyomo [59]. We have found that DEAP [26] provides excellent support for particle swarm optimization (PSO) and genetic algorithm-based optimization.

Given the dearth of data available for biological model calibration, conditional probability (Bayesian) approaches are gaining traction. These approaches provide a probabilistic interpretation of model parameters [60], including uncertainty quantification, at the cost of increased computer time. However, new GPU-based integrators mitigate this problem. Excellent tools for Bayesian parameter inference include PyDREAM (which can readily take PySB models) [27], PyBioNetFit [61], PyPESTO [62], PyMC3 [63], and PySTAN [64], although popular data-science tools such as TensorFlow [65] and PyTorch [66] also provide Bayesian inference capabilities. ABC-SysBio [67] provides a hybrid solution using Approximate Bayesian Computation (ABC), which has also been applied to metabolic models [68].

Model analysis and visualization

Model analysis and visualization is likely the least developed area in systems biology as no clear standards have been proposed. In general, modelers explore the chemical species concentration trajectories in their model to infer mechanistic behaviors and properties. Although well-defined methods exist for solving biochemical flux in steady-state systems, calculating flux through non-equilibrium reactions is highly challenging with some notable attempts toward this goal in the literature [13,60]. For visualization, perhaps the most useful tool in Python is matplotlib [69], which provides flexible graphing capabilities. Other Python tools include Seaborn (https://seaborn.pydata.org/), Plotly [70], Bokeh [71], and Mayavi [72]. Network visualization is perhaps the other major area of model analysis that is addressed in various ways in Python. For example, PyVIPR [73] is a visualization tool built on Cytoscape.js [74] for rule- and reaction-based models which animates model dynamics over time, overlaid on a graph. MASSPy [75] also provides some visualization capabilities for metabolic models. Escher visualizes pathways in a web interface, and has a Python package [76]. Additionally, we note that excellent tools for graph manipulation in Python exist, such as NetworkX [77].

Model sharing and modification

Perhaps the most appealing benefit for the systems biology community from program-based paradigm is the use of literate programming for model and results dissemination. Introduced by Donald Knuth, literate programming is a paradigm whereby the code and the document coexist in an interactive format [78]. Jupyter Notebook, a popular format, has been described as “data scientists’ computational notebook of choice” [79]. Jupyter Notebooks allow analyses to be run in a web browser, checked into version control, and include documentation alongside analyses, in turn improving transparency and reproducibility. We believe that Jupyter notebooks are a highly desirable step forward in systems biology as it greatly contributes to model transparency, revision, and dissemination, and should be included in paper submissions where computational simulation and analysis are involved.

Programmatic models’ code can be managed using existing version control tools. Git has emerged as the de facto standard for version control, providing powerful capabilities for decentralized editing, branching, and merging, with online platforms such as GitHub adding a collaborative interface for change management, commenting, and other functions. In PySB, models are Python programs, and so can be imported like other Python modules and extended or modified. The code can be inspected, for example the model can be searched for species or reactions using pattern matching. Tellurium’s antimony language has an import function, but previous model definitions are currently not programmatically searchable or modifiable.

Good documentation can be vital to ensure model reproducibility and interpretability by others. Sphinx (sphinx-doc.org) is a de facto documentation standard for Python code, which allows code comments to be compiled into multiple formats including website (HTML) and PDF. The former can be combined with continuous integration, for always up-to-date documentation (readthedocs.io).

Model checking and testing

Complex biochemical models present challenges in both ensuring they are correctly encoded, and ensuring their dynamics meet a given specification. In software engineering, it has become common practice to build an accompanying test suite while developing code, which runs the code under scrutiny to test that works as expected. Subtle errors can be introduced as models grow larger. In our opinion, the field should establish minimum standards to ensure software is runnable, reproducible, and meets basic quality standards [80]. In the context of models-as-programs, unit and integration tests can be borrowed from software engineering to ensure code correctness. Unit tests refers to code which checks the functionality of other, minimal units of code; integration tests check that units work as expected when combined. Python has several frameworks for testing such as PyTests, a popular option with a plugin for Jupyter Notebooks [81]. PySB introduces a framework for testing static properties of rule-based models after network generation; for example, checking that certain species are produced by the reaction network, or that certain reactions are present. MEMOTE [34] provides testing and difference-checking capabilities for metabolic models. Using continuous integration (CI), these tests can be run automatically when changes are made and checked into version control, and/or on a regular basis. Running tests regularly is recommended because, even if a model itself does not change, changes to software dependencies could lead to unexpected errors. The importance of this is emphasized by a recent review, which found a majority of Jupyter Notebooks were not automatically reproducible, often due to dependency errors [82]. For open-source models, these tests can be run for free using services such as Github Actions, Travis, and Circle CI (Table 2). Finally, we recommend containerization technologies such as Docker [83] and Singularity [84], which bundle model and software dependencies together in a self-contained environment, further aiding reproducibility and cross-platform compatibility.

Conclusions

Python has recently turned 30 years old and is now one of the most popular programming languages in the world. There are many reasons for its success, but a key insight of its creator is that code is read much more often than it’s written [85]. The same principle applies to biological models, which emphasizes the importance of clear documentation, transparency of approach, and the separation of model specification from simulation and downstream analysis code. These efforts are central to improving reproducibility, code maintenance, and model extensions, by original authors and third parties. We believe the programmatic modeling approach provides a foundation to address the challenges of growing model complexity via well-tested, documented, modular models. Models should ideally be compatible with a range of simulation engines and analysis types; Python-based frameworks enable extensibility with new methods, and standards such as SBML additionally enable cross-framework compatibility. We additionally envisage large, composite models may require multi-modal simulations, e.g. spatial and non-spatial components, or connected gene regulatory, signaling, and metabolic modules. Further development on programmatic modeling frameworks will be needed to fully address these challenges.

For beginners interested in modeling cell signaling, we recommend either the PySB or Tellurium frameworks, both of which have high quality documentation and active communities for support. We expect the Python modeling ecosystem will continue to grow, and efforts for framework and package interoperability to increase.

Highlights.

• Programmatic modeling combines computational modeling with software engineering best practices.

• An executable model enables users to leverage all available resources from the language.

• Community benefits include improved collaboration, reusability, and reproducibility.

• Python has multiple modeling frameworks with a broad, active scientific ecosystem.