Skip to main content
Nature Portfolio logoLink to Nature Portfolio
letter
. 2020 Mar 2;38(3):272–276. doi: 10.1038/s41587-020-0446-y

MEMOTE for standardized genome-scale metabolic model testing

Christian Lieven 1,#, Moritz E Beber 1,#, Brett G Olivier 2, Frank T Bergmann 3, Meric Ataman 4, Parizad Babaei 1, Jennifer A Bartell 1, Lars M Blank 5, Siddharth Chauhan 6, Kevin Correia 7, Christian Diener 8,9, Andreas Dräger 10,11,12, Birgitta E Ebert 5,13, Janaka N Edirisinghe 14, José P Faria 14, Adam M Feist 1,6, Georgios Fengos 4, Ronan M T Fleming 15, Beatriz García-Jiménez 16,40, Vassily Hatzimanikatis 4, Wout van Helvoirt 17,18, Christopher S Henry 14, Henning Hermjakob 19, Markus J Herrgård 1, Ali Kaafarani 1, Hyun Uk Kim 20, Zachary King 6, Steffen Klamt 21, Edda Klipp 22, Jasper J Koehorst 23, Matthias König 22, Meiyappan Lakshmanan 24, Dong-Yup Lee 24,25, Sang Yup Lee 1,20, Sunjae Lee 26,27, Nathan E Lewis 6,28, Filipe Liu 14, Hongwu Ma 29, Daniel Machado 30, Radhakrishnan Mahadevan 7,31, Paulo Maia 32, Adil Mardinoglu 26,27, Gregory L Medlock 33, Jonathan M Monk 6, Jens Nielsen 1,34, Lars Keld Nielsen 1,13, Juan Nogales 16, Intawat Nookaew 34,35, Bernhard O Palsson 1,6, Jason A Papin 33, Kiran R Patil 30, Mark Poolman 36, Nathan D Price 9, Osbaldo Resendis-Antonio 8, Anne Richelle 28, Isabel Rocha 37,38, Benjamín J Sánchez 1,34, Peter J Schaap 23, Rahuman S Malik Sheriff 19, Saeed Shoaie 26,27, Nikolaus Sonnenschein 1,, Bas Teusink 2, Paulo Vilaça 32, Jon Olav Vik 17, Judith A H Wodke 22, Joana C Xavier 39, Qianqian Yuan 29, Maksim Zakhartsev 17, Cheng Zhang 26
PMCID: PMC7082222  NIHMSID: NIHMS1572784  PMID: 32123384

To the Editor — Reconstructing metabolic reaction networks enables the development of testable hypotheses of an organism’s metabolism under different conditions1. State-of-the-art genome-scale metabolic models (GEMs) can include thousands of metabolites and reactions that are assigned to subcellular locations. Gene–protein–reaction (GPR) rules and annotations using database information can add meta-information to GEMs. GEMs with metadata can be built using standard reconstruction protocols2, and guidelines have been put in place for tracking provenance and enabling interoperability, but a standardized means of quality control for GEMs is lacking3. Here we report a community effort to develop a test suite named MEMOTE (for metabolic model tests) to assess GEM quality.

Incompatible description formats and missing annotations4 limit GEM reuse. Moreover, numerical errors5 and omission of essential cofactors6 in a single biomass objective function can have substantial impact on the predictive performance of a GEM. Failure to make checks for flux cycles and imbalances can render model predictions untrustworthy7.

Every year, increasing numbers of manually curated and automatically generated GEMs are published, including those for human and cancer tissue models8. We believe that it is essential to optimize GEM reproducibility and reuse. Researchers need models that are software-agnostic, with components that have standardized, database-independent identifiers. Default conditions and mathematically specified modeling formulations must be precisely defined to allow reproduction of the original model predictions. Models must produce feasible phenotypes under various conditions. Finally, data used to build any model must be made available in a reusable format.

A dual approach could be used to improve GEM reuse and reproducibility. First, we advocate adoption of the latest version of the Systems Biology Markup Language (SBML) level 3 flux balance constraints (SBML3FBC) package9 as the primary description and exchange format. The SBML3FBC package adds structured, semantic descriptions for domain-specific model components such as flux bounds, multiple linear objective functions, GPR rules, metabolite chemical formulas, charge and annotations. The SBML and constraint-based modeling communities collaboratively develop this package, updating it based on user input. It has been adopted by a wide range of constraint-based modeling software and public model repositories (http://cbmpy.sourceforge.net/ and refs. 1015), and should therefore be considered the standard for encoding GEMs.

Second, we present MEMOTE (/’mi:moʊt/ in international phonetic alphabet notation), an open-source Python software that represents a unified approach to ensure the formally correct definition of SBML3FBC and provides quality control and continuous quality assurance of metabolic models with tools and best practices already used in software development16,17. MEMOTE accepts stoichiometric models encoded in SBML3FBC and previous versions as input. In addition to structural validation analogous to the SBML validator18, MEMOTE benchmarks metabolic models using consensus tests from four general areas: annotation, basic tests, biomass reaction and stoichiometry.

Annotation tests check that a model is annotated according to community standards with minimum information required in annotation of models (MIRIAM)-compliant cross-references19, that all primary identifiers belong to the same namespace rather than being fractured across several namespaces, and that components are described using Systems Biology Ontology (SBO) terms20. A lack of explicit, standardized annotations complicates the use, comparison and extension of GEMs, and thus strongly hampers collaboration3,4.

Basic tests check the formal correctness of a model and verify the presence of components such as metabolites, compartments, reactions and genes. These tests also check for metabolite formula and charge information, and GPR rules. General quality metrics, such as the degree of metabolic coverage representing the ratio of reactions and genes21, are also checked.

A model is tested for production of biomass precursors in different conditions, for biomass consistency, for nonzero growth rate and for direct precursors. The biomass reaction is based on the biomass composition of the modeled organism and expresses its ability to produce the necessary precursors for in silico cell growth and maintenance. Thus, an extensive, well-formed biomass reaction is crucial for accurate predictions with a GEM6.

Stoichiometric inconsistency, erroneously produced energy metabolites7 and permanently blocked reactions are identified by MEMOTE. Errors in stoichiometries may result in the production of ATP or redox cofactors from nothing2 and are detrimental to the performance of the model when using flux-based analysis4.

MEMOTE enables a quick comparison of any two given models, in which individual test results are quantified and condensed to calculate an overall score (Supplementary Note 1). In addition to these consensus tests, researchers can supply experimental data from growth and gene perturbation studies in a range of input formats (.csv, .tsv, .xls or .xslx) in MEMOTE. To support reproducibility, researchers can configure MEMOTE to recognize specific data types as input to predefined experimental tests for model validation (Supplementary Note 2).

There are two main workflows for MEMOTE (Fig. 1a and Supplementary Figs. 13). For peer review, MEMOTE can produce either a ‘snapshot report’ or a ‘diff report’ that display MEMOTE test results of one single or multiple models, respectively. For model reconstruction, MEMOTE helps users to create a version-controlled repository of the model and to activate continuous integration toward building a ‘history report’ that records the results of each tracked edit of the model. Although a model repository can be used offline, we encourage community collaboration via distributed version control development platforms, such as GitHub (https://github.com), GitLab (https://gitlab.com/) or BioModels12 (http://wwwdev.ebi.ac.uk/biomodels/). MEMOTE is tightly integrated with GitHub. Models generated and versioned in MEMOTE can easily be uploaded to GitLab and BioModels. Collaborative model reconstruction with MEMOTE as benchmark can occur using all three software platforms (Fig. 1b).

Fig. 1. Graphical summary of MEMOTE.

Fig. 1

a, Graphical representation of the two principal workflows in detail. For peer review, MEMOTE serves as a benchmark tool generating a comprehensive, human-readable report, which quantifies the model’s performance (Supplementary Figs. 1 and 2). With this information, a definitive assessment of model quality can be made by editors, reviewers and users. This workflow is accessible through a web interface (https://memote.io) or locally through a command line interface. For model reconstruction, MEMOTE helps users to create a version-controlled repository for the model (indicated by the blue asterisk), and to activate continuous integration. The model is tested using MEMOTE’s library of test cases, the results are saved, and an initial report of the model is generated. This constitutes the first iteration of the development cycle. Now, users may edit the model using their preferred reconstruction tool and subsequently export it to SBML3FBC, thus creating a new version (indicated by +n). This will restart the cycle by running the tests automatically, saving the results for each version and including them incrementally in a report on the entire history of results. This serves as a guide toward a functional, high-quality GEM (Supplementary Fig. 3). This workflow is accessible through the command line only. b, Both, GitHub and GitLab support a branching strategy, which model builders could use to curate different parts of the model simultaneously or to invite external experts to improve specific model features. MEMOTE further enables model authors to act as gatekeepers, choosing to accept only high-quality contributions. Identification of functional differences happens in the form of a comparative ‘diff’ report, whereas for file-based discrepancies MEMOTE capitalizes on the platform’s ability to show the line-by-line changes between different versions of a model. For this purpose, the model is written in a sorted YAML format28 after every change. Bold blue text denotes actions performed by MEMOTE.

We validated MEMOTE using models from seven GEM collections (Fig. 2, Supplementary Table 1 and Supplementary Methods), that comprise manually and (semi)-automatically reconstructed GEMs (10,780 models in total). Most GEM collections have already made models available in SMBL format. A nonlinear dimensional reduction of the normalized test results (Supplementary Methods) using t-distributed stochastic neighbor embedding (t-SNE; Fig. 2a) indicates that models from the same source are generally more similar to each other than to models from other sources. Nevertheless, several model sources reveal internal subgroupings (Fig. 2a). With the exception of Path2Models22, which relies on pathway resources that contain problematic reaction information on stoichiometry and directionality23, automatically reconstructed GEMs were stoichiometrically consistent (Fig. 2b) and mass-balanced (Supplementary Fig. 4). Of the manually reconstructed GEMs we tested, most models in BiGG13 are stoichiometrically consistent, but there is wide variation among published models, with ~70% of models having at least one stoichiometrically unbalanced metabolite. Stoichiometrically inconsistent models cannot be mass-balanced, but missing formula annotations, from which molecular masses are calculated, further contribute to reactions being counted as unbalanced. The problems that we identified in published models underpin the need for application of MEMOTE during peer-review process (but ideally before submission) of GEMs.

Fig. 2. Quality of manually reconstructed GEMs from collections without quality control or quality assurance.

Fig. 2

a, Depicted is a t-SNE two-dimensional reduction of models using normalized test features as input. Only GEMs from the BiGG collection form a single albeit small cluster. Models from all other collections are grouped in several fragmented but distinct clusters. bd, SinaPlots29 of each collection overlaid with box and whisker plots to indicate 25%, 50% (median) and 75% quantiles. GEMs from collections built in a modern automated pipeline (AGORA, CarveMe, KBase) are stoichiometrically consistent, whereas models from the older Path2Models collection are up to 50% stoichiometrically inconsistent (b). Manually reconstructed models (BiGG, Ebrahim et al.30, OptFlux models) contain varying degrees of inconsistent GEMs. GPR rules are essential for in silico knockout studies, but also serve to justify the presence of a reaction (c). Generally, the fraction of reactions without GPR rules is low (~15%). Yet a distinct group of models from the collections of Ebrahim et al. and OptFlux lack GPR rules for >75% of their reactions. Most models from the CarveMe and Path2Models collections contain very few blocked reactions, whereas for models from the other collections the number of blocked reactions lies mostly between 10% and 30% (d). Again, models from the collections of Ebrahim et al. and the Optflux models show the largest variance.

Source Data

During GEM reconstruction, metabolic reactions are defined based on functional gene annotations, and this information is output as GPR rules. We found that ~15% of reactions in models we tested are not annotated with GPR rules (Fig. 2c). For published models, subgroups of models contain up to 85% of reactions without GPR rules. This could be due to a large number of modeling-specific reactions, spontaneous reactions24 and known reactions with undiscovered genes, or if GPR rules were annotated in nonstandard ways.

CarveMe25 and Path2Models22 have a very low fraction of universally blocked reactions, whereas models from AGORA26 and KBase14 contain ~30% blocked reactions, and BiGG13 models and OptFlux15 models contain ~20% blocked reactions (Fig. 2d). Similarly, orphan and dead-end metabolites (Supplementary Figs. 5 and 6) are also present in all of these published collections. We note that blocked reactions and dead-end metabolites are not indicators of low-quality models but that a large proportion (for example, >50%) of universally blocked reactions can indicate problems in reconstruction that need solving.

AGORA, KBase and BiGG are the only collections with SBML-compliant metabolite and reaction annotations. Gene annotations are only present in KBase models and selected BiGG models (Supplementary Figs. 79). Each collection uses its own system of identifiers for each model component, but there is some overlap between all three (Supplementary Figs. 10 and 11), and partial overlaps for models from KBase and BiGG (Supplementary Figs. 1216), or AGORA and BiGG (Supplementary Figs. 17 and 18), but not KBase and AGORA. BiGG is the only collection with models using MetaNetX27 annotations (Supplementary Fig. 19). MetaNetX consolidates biochemical namespaces by establishing a mapping between them through a set of unique identifiers. Hence, knowing the MetaNetX identifier for a given entity often means also knowing the identifiers for other databases (Supplementary Methods).

MEMOTE tests cover semantic and conceptual requirements, which are fundamental to SBML3FBC and constraint-based modeling, respectively. They are extensible to allow the validation of a model’s performance against experimental data and can be executed as a stand-alone tool or integrated into existing reconstruction pipelines. Capitalizing on robust workflows established in modern software development, MEMOTE promotes openness and collaboration by granting the community tangible metrics to support their research and to discuss assumptions or limitations openly.

Application of a set of defined metabolic model tests is not dependent on implementation in MEMOTE, and for some users it may be more desirable to implement each test separately to streamline the user experience.

We propose that an independent, central library of tests and a tool to run them offers an unbiased approach to quality control because the tests are continuously reviewed by the community. This resource will be maintained under stewardship of Nikolaus Sonnenschein by the openCOBRA consortium (https://github.com/opencobra). To encourage integration as opposed to duplication, MEMOTE provides a Python application programing interface (API) as well as being available as a web service. MEMOTE has already been integrated in several services and tools (Supplementary Note 3). We discuss alternatives and future perspectives of MEMOTE in Supplementary Notes 4 and 5, respectively.

We recommend that MEMOTE users reach out to GEM authors to report any errors and thereby enable community improvement of models as resources. Using inconsistent GEMs for hypothesis generation could lead researchers down blind alleys, so we weighed the influence of ‘consistency’ and ‘stoichiometric consistency’ and SBO terms higher than tests for metabolite, reaction and gene annotations.

We are committed to keeping MEMOTE open to support community principles. Robust benchmarking will only work if it is actively supported by the whole community, and we call on any interested experts to join this endeavor and enable its continual improvement.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Supplementary information

Supplementary Materials (28.6MB, pdf)

Supplementary Figs. 1–162, Notes 1–5, Methods, and Tables 1 and 2

Reporting Summary (67KB, pdf)

Acknowledgements

We acknowledge D. Dannaher and A. Lopez for their supporting work on the Angular parts of MEMOTE; resources and support from the DTU Computing Center; J. Cardoso, S. Gudmundsson, K. Jensen and D. Lappa for their feedback on conceptual details; and P. D. Karp and I. Thiele for critically reviewing the manuscript. We thank J. Daniel, T. Kristjánsdóttir, J. Saez-Saez, S. Sulheim, and P. Tubergen for being early adopters of MEMOTE and for providing written testimonials. J.O.V. received the Research Council of Norway grants 244164 (GenoSysFat), 248792 (DigiSal) and 248810 (Digital Life Norway); M.Z. received the Research Council of Norway grant 244164 (GenoSysFat); C.L. received funding from the Innovation Fund Denmark (project “Environmentally Friendly Protein Production (EFPro2)”); C.L., A.K., N. S., M.B., M.A., D.M., P.M, B.J.S., P.V., K.R.P. and M.H. received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement 686070 (DD-DeCaF); B.G.O., F.T.B. and A.D. acknowledge funding from the US National Institutes of Health (NIH, grant number 2R01GM070923-13); A.D. was supported by infrastructural funding from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Cluster of Excellence EXC 2124 Controlling Microbes to Fight Infections; N.E.L. received funding from NIGMS R35 GM119850, Novo Nordisk Foundation NNF10CC1016517 and the Keck Foundation; A.R. received a Lilly Innovation Fellowship Award; B.G.-J. and J. Nogales received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no 686585 for the project LIAR, and the Spanish Ministry of Economy and Competitivity through the RobDcode grant (BIO2014-59528-JIN); L.M.B. has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement 633962 for project P4SB; R.F. received funding from the US Department of Energy, Offices of Advanced Scientific Computing Research and the Biological and Environmental Research as part of the Scientific Discovery Through Advanced Computing program, grant DE-SC0010429; A.M., C.Z., S.L. and J. Nielsen received funding from The Knut and Alice Wallenberg Foundation, Advanced Computing program, grant #DE-SC0010429; S.K.’s work was in part supported by the German Federal Ministry of Education and Research (de.NBI partner project “ModSim” (FKZ: 031L104B)); E.K. and J.A.H.W. were supported by the German Federal Ministry of Education and Research (project “SysToxChip”, FKZ 031A303A); M.K. is supported by the Federal Ministry of Education and Research (BMBF, Germany) within the research network Systems Medicine of the Liver (LiSyM, grant number 031L0054); J.A.P. and G.L.M. acknowledge funding from US National Institutes of Health (T32-LM012416, R01-AT010253, R01-GM108501) and the Wagner Foundation; G.L.M. acknowledges funding from a Grand Challenges Exploration Phase I grant (OPP1211869) from the Bill & Melinda Gates Foundation; H.H. and R.S.M.S. received funding from the Biotechnology and Biological Sciences Research Council MultiMod (BB/N019482/1); H.U.K. and S.Y.L. received funding from the Technology Development Program to Solve Climate Changes on Systems Metabolic Engineering for Biorefineries (grants NRF-2012M1A2A2026556 and NRF-2012M1A2A2026557) from the Ministry of Science and ICT through the National Research Foundation (NRF) of Korea; H.U.K. received funding from the Bio & Medical Technology Development Program of the NRF, the Ministry of Science and ICT (NRF-2018M3A9H3020459); P.B., B.J.S., Z.K., B.O.P., C.L., M.B., N.S., M.H. and A.F. received funding through Novo Nordisk Foundation through the Center for Biosustainability at the Technical University of Denmark (NNF10CC1016517); D.-Y.L. received funding from the Next-Generation BioGreen 21 Program (SSAC, PJ01334605), Rural Development Administration, Republic of Korea; G.F. was supported by the RobustYeast within ERA net project via SystemsX.ch; V.H. received funding from the ETH Domain and Swiss National Science Foundation; M.P. acknowledges Oxford Brookes University; J.C.X. received support via European Research Council (666053) to W.F. Martin; B.E.E. acknowledges funding through the CSIRO-UQ Synthetic Biology Alliance; C.D. is supported by a Washington Research Foundation Distinguished Investigator Award. I.N. received funding from National Institutes of Health (NIH)/National Institute of General Medical Sciences (NIGMS) (grant P20GM125503).

Source data

Source Data Fig. 2 (1.4MB, xlsx)

Data availability

The model collection is available at 10.5281/zenodo.2636858. Individual results and aggregated tables, as well as analysis code, are available at 10.5281/zenodo.2638234.

Code availability

MEMOTE source code is available at https://github.com/opencobra/memote under the Apache license, version 2.0. Supporting documentation is available at https://memote.readthedocs.io/en/latest/. The MEMOTE web interface is hosted at https://memote.io. A detailed list of all tests in MEMOTE is available at https://memote.readthedocs.io/en/latest/autoapi/index.html.

Competing interests

The authors declare no competing interests.

Footnotes

These authors contributed equally: Christian Lieven, Moritz E. Beber.

Change history

3/19/2020

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Supplementary information

Supplementary information is available for this paper at 10.1038/s41587-020-0446-y.

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Materials (28.6MB, pdf)

Supplementary Figs. 1–162, Notes 1–5, Methods, and Tables 1 and 2

Reporting Summary (67KB, pdf)

Data Availability Statement

The model collection is available at 10.5281/zenodo.2636858. Individual results and aggregated tables, as well as analysis code, are available at 10.5281/zenodo.2638234.

MEMOTE source code is available at https://github.com/opencobra/memote under the Apache license, version 2.0. Supporting documentation is available at https://memote.readthedocs.io/en/latest/. The MEMOTE web interface is hosted at https://memote.io. A detailed list of all tests in MEMOTE is available at https://memote.readthedocs.io/en/latest/autoapi/index.html.


Articles from Nature Biotechnology are provided here courtesy of Nature Publishing Group

RESOURCES