To the Editor — Reconstructing metabolic reaction networks enables the development of testable hypotheses of an organism’s metabolism under different conditions1. State-of-the-art genome-scale metabolic models (GEMs) can include thousands of metabolites and reactions that are assigned to subcellular locations. Gene–protein–reaction (GPR) rules and annotations using database information can add meta-information to GEMs. GEMs with metadata can be built using standard reconstruction protocols2, and guidelines have been put in place for tracking provenance and enabling interoperability, but a standardized means of quality control for GEMs is lacking3. Here we report a community effort to develop a test suite named MEMOTE (for metabolic model tests) to assess GEM quality.
Incompatible description formats and missing annotations4 limit GEM reuse. Moreover, numerical errors5 and omission of essential cofactors6 in a single biomass objective function can have substantial impact on the predictive performance of a GEM. Failure to make checks for flux cycles and imbalances can render model predictions untrustworthy7.
Every year, increasing numbers of manually curated and automatically generated GEMs are published, including those for human and cancer tissue models8. We believe that it is essential to optimize GEM reproducibility and reuse. Researchers need models that are software-agnostic, with components that have standardized, database-independent identifiers. Default conditions and mathematically specified modeling formulations must be precisely defined to allow reproduction of the original model predictions. Models must produce feasible phenotypes under various conditions. Finally, data used to build any model must be made available in a reusable format.
A dual approach could be used to improve GEM reuse and reproducibility. First, we advocate adoption of the latest version of the Systems Biology Markup Language (SBML) level 3 flux balance constraints (SBML3FBC) package9 as the primary description and exchange format. The SBML3FBC package adds structured, semantic descriptions for domain-specific model components such as flux bounds, multiple linear objective functions, GPR rules, metabolite chemical formulas, charge and annotations. The SBML and constraint-based modeling communities collaboratively develop this package, updating it based on user input. It has been adopted by a wide range of constraint-based modeling software and public model repositories (http://cbmpy.sourceforge.net/ and refs.10–15), and should therefore be considered the standard for encoding GEMs.
Second, we present MEMOTE (/’mi:moʊt/ in international phonetic alphabet notation), an open-source Python software that represents a unified approach to ensure the formally correct definition of SBML3FBC and provides quality control and continuous quality assurance of metabolic models with tools and best practices already used in software development16,17. MEMOTE accepts stoichiometric models encoded in SBML3FBC and previous versions as input. In addition to structural validation analogous to the SBML validator18, MEMOTE benchmarks metabolic models using consensus tests from four general areas: annotation, basic tests, biomass reaction and stoichiometry.
Annotation tests check that a model is annotated according to community standards with minimum information required in annotation of models (MIRIAM)-compliant cross-references19, that all primary identifiers belong to the same namespace rather than being fractured across several namespaces, and that components are described using Systems Biology Ontology (SBO) terms20. A lack of explicit, standardized annotations complicates the use, comparison and extension of GEMs, and thus strongly hampers collaboration3,4.
Basic tests check the formal correctness of a model and verify the presence of components such as metabolites, compartments, reactions and genes. These tests also check for metabolite formula and charge information, and GPR rules. General quality metrics, such as the degree of metabolic coverage representing the ratio of reactions and genes21, are also checked.
A model is tested for production of biomass precursors in different conditions, for biomass consistency, for nonzero growth rate and for direct precursors. The biomass reaction is based on the biomass composition of the modeled organism and expresses its ability to produce the necessary precursors for in silico cell growth and maintenance. Thus, an extensive, well-formed biomass reaction is crucial for accurate predictions with a GEM6.
Stoichiometric inconsistency, erroneously produced energy metabolites7 and permanently blocked reactions are identified by MEMOTE. Errors in stoichiometries may result in the production of ATP or redox cofactors from nothing2 and are detrimental to the performance of the model when using flux-based analysis4.
MEMOTE enables a quick comparison of any two given models, in which individual test results are quantified and condensed to calculate an overall score (Supplementary Note 1). In addition to these consensus tests, researchers can supply experimental data from growth and gene perturbation studies in a range of input formats (.csv, .tsv, .xls or .xslx) in MEMOTE. To support reproducibility, researchers can configure MEMOTE to recognize specific data types as input to predefined experimental tests for model validation (Supplementary Note 2).
There are two main workflows for MEMOTE (Fig. 1a and Supplementary Figs. 1–3). For peer review, MEMOTE can produce either a ‘snapshot report’ or a ‘diff report’ that display MEMOTE test results of one single or multiple models, respectively. For model reconstruction, MEMOTE helps users to create a version-controlled repository of the model and to activate continuous integration toward building a ‘history report’ that records the results of each tracked edit of the model. Although a model repository can be used offline, we encourage community collaboration via distributed version control development platforms, such as GitHub (https://github.com), GitLab (https://gitlab.com/) or BioModels12 (http://wwwdev.ebi.ac.uk/biomodels/). MEMOTE is tightly integrated with GitHub. Models generated and versioned in MEMOTE can easily be uploaded to GitLab and BioModels. Collaborative model reconstruction with MEMOTE as benchmark can occur using all three software platforms (Fig. 1b).
We validated MEMOTE using models from seven GEM collections (Fig. 2, Supplementary Table 1 and Supplementary Methods), that comprise manually and (semi)-automatically reconstructed GEMs (10,780 models in total). Most GEM collections have already made models available in SMBL format. A nonlinear dimensional reduction of the normalized test results (Supplementary Methods) using t-distributed stochastic neighbor embedding (t-SNE; Fig. 2a) indicates that models from the same source are generally more similar to each other than to models from other sources. Nevertheless, several model sources reveal internal subgroupings (Fig. 2a). With the exception of Path2Models22, which relies on pathway resources that contain problematic reaction information on stoichiometry and directionality23, automatically reconstructed GEMs were stoichiometrically consistent (Fig. 2b) and mass-balanced (Supplementary Fig. 4). Of the manually reconstructed GEMs we tested, most models in BiGG13 are stoichiometrically consistent, but there is wide variation among published models, with ~70% of models having at least one stoichiometrically unbalanced metabolite. Stoichiometrically inconsistent models cannot be mass-balanced, but missing formula annotations, from which molecular masses are calculated, further contribute to reactions being counted as unbalanced. The problems that we identified in published models underpin the need for application of MEMOTE during peer-review process (but ideally before submission) of GEMs.
During GEM reconstruction, metabolic reactions are defined based on functional gene annotations, and this information is output as GPR rules. We found that ~15% of reactions in models we tested are not annotated with GPR rules (Fig. 2c). For published models, subgroups of models contain up to 85% of reactions without GPR rules. This could be due to a large number of modeling-specific reactions, spontaneous reactions24 and known reactions with undiscovered genes, or if GPR rules were annotated in nonstandard ways.
CarveMe25 and Path2Models22 have a very low fraction of universally blocked reactions, whereas models from AGORA26 and KBase14 contain ~30% blocked reactions, and BiGG13 models and OptFlux15 models contain ~20% blocked reactions (Fig. 2d). Similarly, orphan and dead-end metabolites (Supplementary Figs. 5 and 6) are also present in all of these published collections. We note that blocked reactions and dead-end metabolites are not indicators of low-quality models but that a large proportion (for example, >50%) of universally blocked reactions can indicate problems in reconstruction that need solving.
AGORA, KBase and BiGG are the only collections with SBML-compliant metabolite and reaction annotations. Gene annotations are only present in KBase models and selected BiGG models (Supplementary Figs. 7–9). Each collection uses its own system of identifiers for each model component, but there is some overlap between all three (Supplementary Figs. 10 and 11), and partial overlaps for models from KBase and BiGG (Supplementary Figs. 12–16), or AGORA and BiGG (Supplementary Figs. 17 and 18), but not KBase and AGORA. BiGG is the only collection with models using MetaNetX27 annotations (Supplementary Fig. 19). MetaNetX consolidates biochemical namespaces by establishing a mapping between them through a set of unique identifiers. Hence, knowing the MetaNetX identifier for a given entity often means also knowing the identifiers for other databases (Supplementary Methods).
MEMOTE tests cover semantic and conceptual requirements, which are fundamental to SBML3FBC and constraint-based modeling, respectively. They are extensible to allow the validation of a model’s performance against experimental data and can be executed as a stand-alone tool or integrated into existing reconstruction pipelines. Capitalizing on robust workflows established in modern software development, MEMOTE promotes openness and collaboration by granting the community tangible metrics to support their research and to discuss assumptions or limitations openly.
Application of a set of defined metabolic model tests is not dependent on implementation in MEMOTE, and for some users it may be more desirable to implement each test separately to streamline the user experience.
We propose that an independent, central library of tests and a tool to run them offers an unbiased approach to quality control because the tests are continuously reviewed by the community. This resource will be maintained under stewardship of Nikolaus Sonnenschein by the openCOBRA consortium (https://github.com/opencobra). To encourage integration as opposed to duplication, MEMOTE provides a Python application programing interface (API) as well as being available as a web service. MEMOTE has already been integrated in several services and tools (Supplementary Note 3). We discuss alternatives and future perspectives of MEMOTE in Supplementary Notes 4 and 5, respectively.
We recommend that MEMOTE users reach out to GEM authors to report any errors and thereby enable community improvement of models as resources. Using inconsistent GEMs for hypothesis generation could lead researchers down blind alleys, so we weighed the influence of ‘consistency’ and ‘stoichiometric consistency’ and SBO terms higher than tests for metabolite, reaction and gene annotations.
We are committed to keeping MEMOTE open to support community principles. Robust benchmarking will only work if it is actively supported by the whole community, and we call on any interested experts to join this endeavor and enable its continual improvement.
Supplementary Material
Acknowledgements
We acknowledge D. Dannaher and A. Lopez for their supporting work on the Angular parts of MEMOTE; resources and support from the DTU Computing Center; J. Cardoso, S. Gudmundsson, K. Jensen and D. Lappa for their feedback on conceptual details; and P. D. Karp and I. Thiele for critically reviewing the manuscript. We thank J. Daniel, T. Kristjánsdóttir, J. Saez-Saez, S. Sulheim, and P. Tubergen for being early adopters of MEMOTE and for providing written testimonials. J.O.V. received the Research Council of Norway grants 244164 (GenoSysFat), 248792 (DigiSal) and 248810 (Digital Life Norway); M.Z. received the Research Council of Norway grant 244164 (GenoSysFat); C.L. received funding from the Innovation Fund Denmark (project “Environmentally Friendly Protein Production (EFPro2)”); C.L., A.K., N. S., M.B., M.A., D.M., P.M, B.J.S., P.V., K.R.P. and M.H. received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement 686070 (DD-DeCaF); B.G.O., F.T.B. and A.D. acknowledge funding from the US National Institutes of Health (NIH, grant number 2R01GM070923-13); A.D. was supported by infrastructural funding from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Cluster of Excellence EXC 2124 Controlling Microbes to Fight Infections; N.E.L. received funding from NIGMS R35 GM119850, Novo Nordisk Foundation NNF10CC1016517 and the Keck Foundation; A.R. received a Lilly Innovation Fellowship Award; B.G.-J. and J. Nogales received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no 686585 for the project LIAR, and the Spanish Ministry of Economy and Competitivity through the RobDcode grant (BIO2014-59528-JIN); L.M.B. has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement 633962 for project P4SB; R.F. received funding from the US Department of Energy, Offices of Advanced Scientific Computing Research and the Biological and Environmental Research as part of the Scientific Discovery Through Advanced Computing program, grant DE-SC0010429; A.M., C.Z., S.L. and J. Nielsen received funding from The Knut and Alice Wallenberg Foundation, Advanced Computing program, grant #DE-SC0010429; S.K.’s work was in part supported by the German Federal Ministry of Education and Research (de.NBI partner project “ModSim” (FKZ: 031L104B)); E.K. and J.A.H.W. were supported by the German Federal Ministry of Education and Research (project “SysToxChip”, FKZ 031A303A); M.K. is supported by the Federal Ministry of Education and Research (BMBF, Germany) within the research network Systems Medicine of the Liver (LiSyM, grant number 031L0054); J.A.P. and G.L.M. acknowledge funding from US National Institutes of Health (T32-LM012416, R01-AT010253, R01-GM108501) and the Wagner Foundation; G.L.M. acknowledges funding from a Grand Challenges Exploration Phase I grant (OPP1211869) from the Bill & Melinda Gates Foundation; H.H. and R.S.M.S. received funding from the Biotechnology and Biological Sciences Research Council MultiMod (BB/N019482/1); H.U.K. and S.Y.L. received funding from the Technology Development Program to Solve Climate Changes on Systems Metabolic Engineering for Biorefineries (grants NRF-2012M1A2A2026556 and NRF-2012M1A2A2026557) from the Ministry of Science and ICT through the National Research Foundation (NRF) of Korea; H.U.K. received funding from the Bio & Medical Technology Development Program of the NRF, the Ministry of Science and ICT (NRF-2018M3A9H3020459); P.B., B.J.S., Z.K., B.O.P., C.L., M.B., N.S., M.H. and A.F. received funding through Novo Nordisk Foundation through the Center for Biosustainability at the Technical University of Denmark (NNF10CC1016517); D.-Y.L. received funding from the Next-Generation BioGreen 21 Program (SSAC, PJ01334605), Rural Development Administration, Republic of Korea; G.F. was supported by the RobustYeast within ERA net project via SystemsX. ch; V.H. received funding from the ETH Domain and Swiss National Science Foundation; M.P. acknowledges Oxford Brookes University; J.C.X. received support via European Research Council (666053) to W.F. Martin; B.E.E. acknowledges funding through the CSIRO-UQ Synthetic Biology Alliance; C.D. is supported by a Washington Research Foundation Distinguished Investigator Award. I.N. received funding from National Institutes of Health (NIH)/National Institute of General Medical Sciences (NIGMS) (grant P20GM125503).
Footnotes
Reporting Summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Data availability
The model collection is available at https://doi.org/10.5281/zenodo.2636858. Individual results and aggregated tables, as well as analysis code, are available at https://doi.org/10.5281/zenodo.2638234.
Code availability
MEMOTE source code is available at https://github.com/opencobra/memote under the Apache license, version 2.0. Supporting documentation is available at https://memote.readthedocs.io/en/latest/. The MEMOTE web interface is hosted at https://memote.io. A detailed list of all tests in MEMOTE is available at https://memote.readthedocs.io/en/latest/autoapi/index.html.
Competing interests
The authors declare no competing interests.
Supplementary information is available for this paper at https://doi.org/10.1038/s41587-020-0446-y.
References
- 1.Palsson BØ Systems Biology: Constraint-based Reconstruction and Analysis (Cambridge Univ. Press, 2015). [Google Scholar]
- 2.Thiele I & Palsson BØ Nat. Protoc 5, 93–121 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Heavner BD & Price ND Curr. Opin. Biotechnol 34, 105–109 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ravikrishnan A & Raman K Brief. Bioinform 16, 1057–1068 (2015). [DOI] [PubMed] [Google Scholar]
- 5.Chan SHJ, Cai J, Wang L, Simons-Senftle MN & Maranas CD Bioinformatics 10.1093/bioinformatics/btx453 (2017). [DOI] [PubMed] [Google Scholar]
- 6.Xavier JC, Patil KR & Rocha I Metab. Eng 39, 200–208 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Fritzemeier CJ, Hartleb D, Szappanos B, Papp B & Lercher MJ PLoS Comput. Biol 13, e1005494 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Jerby L & Ruppin E Clin. Cancer Res 18, 5572–5584 (2012). [DOI] [PubMed] [Google Scholar]
- 9.Olivier BG & Bergmann FT J. Integr. Bioinform 15, 20170082 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Heirendt L et al. Nat. Protoc 14, 639–702 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ebrahim A, Lerman JA, Palsson BO & Hyduke DR BMC Syst. Biol 7, 74 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Chelliah V et al. Nucleic Acids Res 43, D542–D548 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.King ZA et al. Nucleic Acids Res 44, D515–D522 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Arkin AP et al. Nat. Biotechnol 36, 566–569 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rocha I et al. BMC Syst. Biol 4, 45 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Cooper J, Vik JO & Waltemath D Prog. Biophys. Mol. Biol 117, 99–106 (2015). [DOI] [PubMed] [Google Scholar]
- 17.Beaulieu-Jones BK & Greene CS Nat. Biotechnol 35, 342–346 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Bornstein BJ, Keating SM, Jouraku A & Hucka M Bioinformatics 24, 880–881 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Le Novère N et al. Nat. Biotechnol 23, 1509–1515 (2005). [DOI] [PubMed] [Google Scholar]
- 20.Courtot M et al. Mol. Syst. Biol 7, 543 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Monk J, Nogales J & Palsson BO Nat. Biotechnol 32, 447–452 (2014). [DOI] [PubMed] [Google Scholar]
- 22.Büchel F et al. BMC Syst. Biol 7, 116 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Yuan Q et al. PLoS One 12, e0169437 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Keller MA, Piedrafita G & Ralser M Curr. Opin. Biotechnol 34, 153–161 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Machado D, Andrejev S, Tramontano M & Patil KR Nucleic Acids Res 46, 7542–7553 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Magnúsdóttir S et al. Nat. Biotechnol 35, 81–89 (2017). [DOI] [PubMed] [Google Scholar]
- 27.Moretti S et al. Nucleic Acids Res 44, D523–D526 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Steffensen JL, Dufault-Thompson K, Zhang Y & Dandekar T PSAMM: a portable system for the analysis of metabolic models. PLOS Comput. Biol 12, e1004732 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sidiropoulos N et al. SinaPlot: an enhanced chart for simple and truthful representation of single observations over multiple classes. J. Comput. Graph. Stat 27, 673–676 (2018). [Google Scholar]
- 30.Ebrahim A et al. Do genome-scale models need exact solvers or clearer standards? Mol. Syst. Biol 11, 831 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.