MetSim: Integrated Programmatic Access and Pathway Management for Xenobiotic Metabolism Simulators

Abstract

Xenobiotic metabolism is a key consideration in evaluating the hazards and risks posed by environmental chemicals. A number of software tools exist that are capable of simulating metabolites but each reports out its predictions in a different format and with varying levels of detail. This makes comparing the performance and coverage of tools a practical challenge. To address this shortcoming, we developed a metabolic simulation framework called MetSim, comprising three main components. A graph-based schema was developed to allow metabolism information to be harmonized. The schema has been implemented in MongoDB to store and retrieve metabolic graphs for subsequent analysis. MetSim currently includes an application programming interface for four metabolic simulators: BioTransformer, the OECD Toolbox, EPA’s Chemical Transformation Simulator (CTS), and Tissue Metabolism Simulator (TIMES). Lastly, MetSim provides functions to help evaluate simulator performance for specific datasets. In this study, a set of 112 drugs with 432 reported metabolites were compiled and predictions were made using the 4 simulators. Fifty-nine of the 112 drugs were taken from the Small Molecule Pathway Database with the remainder sourced from the literature. The human models within BioTransformer and CTS (Phase I only) and the rat models within TIMES and the OECD Toolbox (Phase I only) were used to make predictions for the chemicals in the dataset. The recall and precision (recall, precision) ranked in order of highest recall for each individual tool were CTS (0.54, 0.017), BioTransformer (0.50, 0.008), Toolbox in vitro (0.40, 0.144), TIMES in vivo (0.40, 0.133), Toolbox in vivo (0.40, 0.118) and TIMES in vitro (0.39, 0.128). Combining all the model predictions together increased the overall recall (0.73, 0.008). MetSim enabled insights on the performance and coverage of in silico metabolic simulators to be more efficiently derived, which in turn should aid future efforts to evaluate other datasets.

Graphical Abstract

INTRODUCTION

Background

Regulatory programs globally are resulting in more chemicals requiring safety assessment. There is a need for these assessments to be conducted faster and with fewer animals.^{1, 2} Such considerations have contributed to a shift in how toxicity testing are being undertaken, moving from testing based on phenotypic responses in animals towards pathway approaches reliant on physiological mechanisms and dose-dependent biological changes in exposed organisms in addition to in silico models such as (quantitative) structure-activity relationships ((Q)SARs) and read-across. The CompTox BluePrint outlined by Thomas et al (2019) set the strategic and operational direction for the CompTox program within the US EPA to obtain a broader acceptance of CompTox approaches for application in higher-tier regulatory decisions. The recently initiated New Chemicals Collaborative Research Program (NCCRP) within the US EPA marks a significant step in bringing innovative approaches to address the requirements of the Toxic Substances Control Act (TSCA) for the review of new chemicals.³ That said, there remains a significant gap between the large number of chemicals in commerce that result in human exposure and the small number of well-studied chemicals with available animal or human data. This is particularly pertinent for metabolism since such information is not routinely generated despite its importance to risk assessment.⁴

Whilst information about compounds with the metabolites formed in species other than humans and rats exist in different databases,⁵ they are primarily focused on pharmaceuticals and pesticides.^{6, 7} Publicly available databases containing endogenous and xenobiotic metabolic information include Reactome,⁸, the Kyoto Encyclopedia of Genes and Genomes (KEGG),^{9, 10} the Human Metabolome Database (HMDB),^11,12 the University of Minnesota Biocatalysis/Biodegradation Database (UMBBD)¹³ and the Encyclopedia of Human Genes and Metabolism (HumanCyc).^{14, 15} Reactome (Version 86, September 2023) contains 14,516 human reactions organized into 2,615 pathways with 19,435 pathways for 14 other non-human organisms for chemical classes including endogenous metabolic compounds, and pharmaceutical compounds. KEGG (Release 108.0, October 1, 2023) contains databases of 563 metabolic pathways, ~12,000 metabolic reactions, ~8,000 metabolically active enzymes, and ~19,000 metabolite compound structures for chemical classes including endogenous metabolic compounds, lipids, proteins, phytochemicals, nucleotide sugars, glycosides, endocrine disruptors, pesticides, narcotics, and psychotropics.^{9, 10} UMBBD (January 2016 Release), contains 219 metabolic pathways, 1,503 reactions, 1,396 compounds, 993 enzymes, and 249 transformation rules primarily encompassing microbial catabolism and biotransformations for environmental pollutants.¹³ HumanCyc (Version 27.1, August 2023), contains 3,105 metabolic pathways, 18,566 reactions and 18,973 metabolites within its MetaCyc library for chemical classes including endogenous metabolites, dyes, bisphenols, and pharmaceuticals.^{14, 15} HMDB (Version 5.0, January 2022) contains ~50,000 metabolic pathways, ~18,000 reactions, and ~248,000 metabolites (with exception of UMBBD, this includes chemical class data from the aforementioned databases, plus common toxins and environmental pollutants, and food components and additives).^{11, 12} Mining these resources of metabolic information can shed light on the metabolites of environmental chemicals, drugs, and biotransformation pathways. Computational analysis of pathways is facilitated by formal standards that harmonize how metabolic pathways and their associated chemical reactions are represented and exchanged. These include ontologies such as Biological Pathway Exchange (BioPAX),¹⁶ the Systems Biology Markup Language (SBML),¹⁷ and pathway analysis tools for integration and knowledge acquisition (PATIKA).¹⁸

In Silico Metabolism Tools

In silico metabolism tools are a time and resource efficient means of filling data gaps about the metabolites of environmental chemicals and drugs. A number of commercial and freely available in silico metabolism prediction tools are available which rely on a chemical structure as an input. There are several excellent reviews that describe the different types of in silico metabolism prediction tools.^19–21 Briefly, there are three main types of metabolic simulators: 1) those that are rule-based and rely on pattern matching structural features relative to a knowledge base of biotransformation rules that simulate the metabolites; 2) those that predict Sites of Metabolism (SOM), and 3) those that are hybrid in scope. Rule-based tools include BioTransformer,²² TIssue MEtabolism Simulator (TIMES),^{23, 24} the EPA Chemical Transformation Simulator (CTS),²⁵ the OECD QSAR Toolbox (OECD Toolbox),²⁶ METEOR Nexus,^{27, 28} Systematic Generation of Metabolites (SyGMa),²⁹ GLORY/GLORYx,^{30, 31} MetaSense,³², and MetaTox.³³ Tools that predict probable sites of metabolism (SoM) based on the lability or reactivity of structural moieties include SmartCyp,³⁴ MetaSite,³⁵ XenoSite,³⁶, FAME 2/FAME 3,^{37, 38} the CypBoM module within the CyProduct model of BioTransformer, ³⁹ RS-Predictor ⁴⁰, and SOMP.^{19, 32, 41} Hybrid tools such as the CypReact model within BioTransformer,⁴² ADMET Predictor,⁴³ SmartCyp,³⁴ and OpenVirtualToxLab⁴⁴ aim to predict whether parent substances are probable reactants for cytochrome P450 (Cyp450) or other metabolically competent enzymes in addition to simulating metabolites. Another hybrid tool includes enviPath, the Environmental Contaminant Biotransformation Pathway resource which is both a database and prediction system for microbial biotransformations of environmental contaminants (envipath.org).

Performance Assessments

There have been comparative studies for several of these metabolic simulation tools to evaluate their ability to faithfully reproduce reported metabolites. Comparisons of Phase I metabolism on a dataset of ~30 well-known human CYP450 reactant chemicals via GLORY (includes FAME 2 SoM), SyGMa (recall 0.72), and BioTransformer (phase I via CypReact, recall 0.74) showed that depending on the mode of execution of GLORY, it performed less favorably relative to the other 2 tools if its parameters were chosen to optimize for precision (recall 0.64), but performed better when parameters were chosen to optimize for recall (0.83).^{31, 37} When applied to a dataset of 37 top-selling drugs, GLORYx (Phase I and Phase II with FAME 3 SoM, recall 0.77) outperformed SyGMa (recall 0.68).^{30, 38} A 2019 BioTransformer publication compared single-step metabolism on two different datasets to two different tools.²² Single-step human metabolism in BioTransformer (all human models except gut metabolism, recall 0.88) outperformed Meteor Nexus (recall 0.71, 0.45, and 0.13, from lowest to highest confidence levels) when applied to a dataset of 40 pharmaceuticals and pesticides. A second comparison was made on a dataset of 60 drugs, pesticides, phytochemicals, and endobiotics (e.g., lipids), where BioTransformer (CypReact, recall 0.90) outperformed ADMET Predictor (recall 0.61). Few evaluations have considered the performance of TIMES beyond Boyce et al.⁴⁵ The training sets underpinning the TIMES simulators contain between ~440 and ~700 empirically validated chemicals from the literature containing drugs, pharmaceuticals, and pesticides. The rat liver models have high recall for the training data (0.81 for in vitro model, 0.77 for in vivo) as reported in the model documentation available in the software platform itself.^46–49 Many of the metabolic simulators that focus on the estimation of toxicity endpoints have not performed comparisons with other tools of similar function to predict in silico metabolites. Rather, they discuss their ability to predict the endpoints of interest for the metabolites generated accurately.^{26, 33, 44, 50} Given that these tools will often be used for toxicity endpoint estimation,^{51, 52} working with tools that integrate metabolite simulation in conjunction with toxicity endpoint estimates,^{53, 54} can be an advantage. Open-source tools that allow for the addition of customized transformation rules (e.g., SyGMa, BioTransformer) can provide a route towards improving performance should a given tool underperform for a given set of chemistries or particular chemical classes outside of those initially included in their respective training sets.

In this study, a metabolic simulation framework called MetSim was developed to facilitate a performance comparison of different metabolism prediction tools for a dataset of chemicals with reported metabolism. The framework comprises three main components: 1) a graph-based representation to harmonize the storage of xenobiotic metabolism information, 2) an application programming interface (API) to enable predictions to be generated from different simulators, and 3) functions to systematically evaluate simulator performance on datasets. The performance (precision and recall) of four tools namely BioTransformer (version 3), the OECD Toolbox (version 4.5), TIMES (version 2.31.2.82) and CTS (version 2.0) was evaluated using a dataset comprising 112 drugs with reported human metabolism information. Finally, the performance of each simulator was summarized by chemical class making use of the chemistry taxonomy ClassyFire (http://classyfire.wishartlab.com/).

Methods

Data Sources: Metabolism Data

Two datasets with reported in vivo human metabolism data were compiled. The first was a collection of 59 drugs and non-steroidal anti-inflammatory drugs (NSAIDs) with a total of 179 observed Phase I metabolites by Piechota et al.⁶ The second dataset comprised 59 drugs and their 259 Phase I and Phase II metabolites stored within SMPDB.^{63, 64} Six of the parent drug chemicals were common to both datasets (acetaminophen/paracetamol, celecoxib, clopidogrel, esomeprazole, venlafaxine, and ibuprofen). All analyses were performed on this combined dataset consisting of 112 unique parent chemicals and 427 unique metabolites. All predicted and empirical metabolism data were represented using the MetSim schema and are provided as Supplementary Files 3–10.

Chemical Information

Chemicals were characterized by their Simplified Molecular Input Line Entry System (SMILES) structure encoding format,⁵⁵ and lookup identifiers such as International Chemical Identifier Key (InChIKey)⁵⁶ and/or Distributed Structure-Searchable Toxicity (DSSTox) Substance Identifiers (DTXSID). Chemical structural information was retrieved programmatically through web API calls using the EPA’s CompTox Chemicals Dashboard (hereafter referred to as the CCD),^{57, 58} (see https://www.epa.gov/comptox-tools/comptox-data-and-apis) or the EPA’s Cheminformatics Modules Standardizer (hereafter referred to as the Standardizer).⁵⁹ (see https://www.epa.gov/comptox-tools/cheminformatics) All SMILES were converted into their QSAR-Ready equivalents using the Standardizer to ensure uniformity of input structural data across the metabolic simulators being used. The standardization involves stripping any stereochemistry, removing any isotopically labeled atoms as well as salt counterions. The Standardizer also returns DTXSIDs, CASRNs, InChIKeys, etc.,^{60, 61} if the chemicals are already registered within the DSSTox database. If a SMILES for a parent chemical was not provided but the DTXSID was known, CCD was first queried to retrieve the associated SMILES. This would then be queried by the Standardizer to return a QSAR-Ready SMILES representation.

Figure 1 outlines the main workflow for the study. Firstly, a dataset of chemicals with reported metabolism information was compiled. Chemical identity and structural information was retrieved using CCD and the Standardizer. Structures represented as SMILES were converted into QSAR-Ready formats. The MetSim framework was then utilized to generate predictions for each chemical using 4 different metabolic simulators. Relevant information associated with the simulators to contextualize the predictions generated was captured and stored in a MongoDB. Performance was assessed using recall and precision metrics.

Figure 1.

Workflow describing the main components of the study. Firstly a dataset was compiled, from which structural information was gathered. Predictions were generated in 4 tools making use of the MetSim framework to facilitate subsequent processing.

Metabolic Simulation (MetSim)

The Metabolic Simulation (MetSim) framework was developed to facilitate a more efficient means of comparing the performance of different metabolic simulators. The MetSim framework comprises: 1) a graph-based representation to harmonize the storage of metabolism information for effective search and retrieval, 2) an application programming interface (API) for selected metabolic simulators, and 3) functions to systematically evaluate their performance.

Graph-based representations of Metabolism

A wide variety of chemical properties and processes may be modelled as graphs. Each metabolism prediction could be represented as a directed metabolic graph comprising chemicals as nodes and transformations as edges. The edges would contain attributes corresponding to the reaction mechanism, metabolizing enzymes involved (if available), and genes associated with the enzymes (if available). Edges from the parent chemicals would link to the first generation of metabolites. These would then form the input nodes for a subsequent set of transformations thus forming successive generations making up the entire metabolic graph.

Augmenting Chemical information for metabolites

All four metabolic simulators used SMILES to represent parent chemicals (as inputs) and their metabolites (as predicted outputs).The Standardizer was used to query each metabolite SMILES to return DSSTox and Chemical Abstract Services (CAS) Registry Numbers (CASRNs) records if available. This would supplement the metabolite SMILES with other identifier information such as systematic names (e.g., International Union of Pure and Applied Chemistry (IUPAC) names) or InChIKeys.

Standardizing Transformations

Specific transformation data such as the type of reaction mechanism occurring, and available enzyme information was captured in the MetSim hierarchy. TIMES outputs yield specific information regarding types of reaction mechanisms (e.g., “O-Glucuronidation|Alcohol, Phenol, Carboxylic Acid Glucuronidation”, “Quinone Imine Formation|4-Aminophenol Quinone Imine Formation”, etc.), and high-level, generalized information about enzymes as either “[Phase I]” or “[Phase II]”. It should be noted that the latest version of TIMES (version 2.32.1) now provides specific enzyme information. CTS provides specific Phase I reaction mechanism information (e.g., “AminophenolOxidation”, “N-Hydroxylation”, etc.), but no enzyme information. Given that the ChemAxon transformer within CTS is limited to Phase I interactions, it was assumed that most enzymes would be of the Cytochrome P450 (CYP450) class. BioTransformer provides specific data for the type of reaction mechanism (e.g., “N-Dealkylation of mixed tertiary amine”, “Sulfation of secondary alcohol”, etc.), and specific enzyme information in terms of genes or Enzyme Commission (EC) numbers (e.g., “EC 2.8.2.2” corresponding to Alcohol Sulfotransferase, “CYP3A4” for Cyp450 3A4, etc.). No attempt was made to harmonize the transformation information across the different tools. The output information was captured in its raw form from each tool.

Storing the Metabolic Graphs

Metabolic graphs were converted into a hierarchical document-oriented format, called the MetSim schema, shown in Figure 2 for storage and retrieval purposes. The MetSim schema describes the parent chemical, the provenance of the metabolic information (empirical or in silico), and the entire graph as the set of transformations. Each transformation comprised a precursor metabolite, a successor metabolite and the enzyme catalyzing the reaction. The root level of the hierarchy contains the date and time of record generation, the software tool used, the software version, as well as the input parameters. These parameters comprised such aspects as the transformation depth (i.e., how many cycles of metabolism were being simulated), whether site of metabolism information was incorporated within the predictions, the specific model(s) being used to simulate metabolism, as well as the organism the model was based on. The parent chemical data was stored as input data by SMILES and the other identifiers already described. The output data consisted of the collection of metabolite precursor and successor information. The data stored for each precursor consisted of the same types of chemical structural information and chemical identifier information as described for the input. If the enzyme or reaction mechanism associated with a given transformation was known, that information was stored for each successor, along with the metabolic generation expressed in the current successor level (e.g., parent input precursor to primary successor metabolite is generation 1, etc.). Lastly, the metabolite chemical structures and identifying data were stored for each successor, which consisted of the same information types as were stored for the parent input and each precursor.

Figure 2.

Illustration of the hierarchical schema of a metabolic graph.

Empirically observed metabolic data were organized in a similar manner, with a few minor differences. The root level of the hierarchy contains relevant reference data such as journal title, digital object identifier, publication date, and title of the article. Further details of how experimental data were processed by relevant pieces of the schema are described in the Supporting Information along with a truncated example output for the chemical Aripiprazole (Figure S1).

Tool Outputs

The output obtained from the Toolbox was simply the SMILES for all metabolites generated from a given input chemical. Each run of BioTransformer produced precursor information for generations beyond the first from the parent. The BioTransformer output “Precursor ID” numbers and “Metabolite ID” numbers were utilized to add metabolic graph directionality to the hierarchy to track generations. Metabolic graph directionality was also incorporated from TIMES output using the first four columns namely:“#”, “ID of metabolite”, “Level of generation” and “Predecessor ID”.

Metabolism Prediction Tool Selection

The four tools used in this study were selected based on the following criteria: 1) ability to batch process large numbers of chemicals, 2) command line capability vs. a graphical user interface, 3) transformation filters based on confidence or probability thresholds (which affects the total number of predicted metabolites), 4) the average simulation processing time, 5) generational tracking information for metabolites at different steps, 6) transformation filters based on mechanisms (e.g., Phase I, Phase II, etc.), and 7) choice of organisms. The four tools in the study were the Toolbox, BioTransformer, EPA’s Chemical Transformation Simulator (CTS), and TIMES. BioTransformer and CTS were used to simulate human metabolism, whereas the Toolbox and TIMES were used to simulate in vivo and in vitro rat liver metabolism. All four tools could be run in batch mode. With exception of TIMES, the other three simulators could be used on the command line. TIMES was included in addition to the Toolbox to have a cross-reference between the full versions of the rat liver metabolism models included within TIMES compared to the Phase I parameter-restricted models that exist as part of the Toolbox. The models from the TIMES/Toolbox contain probability threshold parameters to truncate metabolic maps for metabolites that are below a set threshold of observing a given metabolite, and accounts for phase II elimination routes to truncate a metabolic pathway (TIMES only). CTS returns a qualitative “likelihood” parameter along with each predicted metabolite as part of the Human Phase I transformer (ChemAxon) used in this study. BioTransformer does not have probability thresholding for the models used in this study. The ability to track which transformations corresponded to each sequential cycle of metabolism was available in the output of BioTransformer, CTS, and TIMES. No such information was available in the output from the Toolbox, such that all transformations were taken as parent to first metabolic generation. This assumption may not be true in all cases, all generated metabolites were all phase I. To estimate simulation runtime, each tool was run on the 37 parent chemicals from our earlier study (PoC dataset),⁴⁵ where both the total runtime and number of metabolites were assessed for each tool. Their execution times, requirements for installation, necessary inputs, and outputs given are tabulated in the Supplementary Material. Table 1 provides a summary.

Table 1:

Abbreviated Tool Feature Summary Table and Runtime Results for the PoC Dataset

Tool	Batch Executable?	Programmatic access?	Generational Tracking from Tool Output?	Model	Organism	Total Runtime (seconds)	Total Number of Unique Predicted Metabolites
OECD Toolbox WebAPI	Yes	Yes	No	Rat Liver S9 (Phase I Restricted)	Rat	70*	207
BioTransformer	Yes	Yes	Yes	1x ecbased, 2x Cyp450, 1x Phase II	Human	255*	825
TIMES	Yes	No**	Yes	Rat Liver S9 (Full model)	Rat	10*	492
CTS	Yes	Yes	Yes	ChemAxon Human Phase I	Human	110	434

^*Runtimes are based on parallel processing with 16 CPU cores.

^**Although TIMES does not expose a public API, its models have been available in a proprietary SaaS solution offered by LMC. It features a public WebAPI that allows integration of the models in automated systems providing results in a machine-interpretable way, both synchronously and asynchronously.

Chemical Class Data

Chemical class information was generated using the Wishart Lab ClassyFire Web API ((http://classyfire.wishartlab.com/entities/) for each parent chemical on the basis of their INChIKey.⁶⁵ All parent chemicals in the dataset were successfully assigned a chemical class with ClassyFire.^{56, 65} ClassyFire provides a hierarchical chemical classification of chemical entities (mostly small molecules and short peptide sequences), as well as a structure-based textual description, based on a chemical taxonomy named ChemOnt, which covers 4825 chemical classes of organic and inorganic compounds. Deriving a class from ClassyFire enabled a summarized perspective of the performance and coverage of the different simulator tools e.g. whether certain classes of chemicals were better or more poorly predicted with different tools or whether certain rules were missing for specific classes.

Parameter Settings

The parameters utilized to simulate metabolism for each tool are provide in brief herein with a summary in Table 2. The rat liver in vitro and in vivo models were used within the Toolbox and TIMES. The BioTransformer models used consisted of sequentially run Enzyme-Commission Based (ECBased), CypReact, and Phase II models. CTS was run using its ChemAxon Human Phase I metabolizer.

Table 2.

Model settings for each tool and each dataset.

Tool	BioTransformer (Wishart Lab)	Chemical Transformation Simulator (CTS, US EPA)	Tissue Metabolism Simulator (TIMES, OASIS-LMC)	OECD QSAR Toolbox WebAPI (OECD, models sourced from OASIS-LMC)
Model Details	• 4 generations (trackable) • Enzyme Commision Based (ecbased), Phase I (CYP450), and Phase II (phaseII) models • Base Cyp450 model (CypReact)	• 3 generations (trackable) • ChemAxon Human Phase I model	• 6 generations (trackable) • In Vitro Rat Liver S9 and In Vivo Rat Simulator models • Thresholded at 10 metabolites per cycle or 0.1 transformation probability	• 3–6 generations (no tracking) • TIMES Default In Vitro Rat Liver S9 and In Vivo Rat Simulator models • Phase I only
Model Abbreviation	• bt.ec.1.cyp450.2.phase2.1	• cts.chemaxon.3	• tm.vitro_rat.6 • tm.vivo_rat.6	• tb.vitro_rat.3 • tb.vivo_rat.3

Toolbox models could not be adjusted from the defaults of 3 cycles of Phase I metabolism for in vitro rat and 999 cycles of phase I metabolism (with no transformation depths greater than six cycles observed given the base transformation probability rules from TIMES) for in vivo rat. TIMES was thresholded to six cycles of mixed Phase I and Phase II metabolism since six generations was the highest transformation depth observed in the dataset under study. CTS was thresholded to three cycles of human Phase I metabolism since the scale of the metabolic trees produced at three cycles would occasionally cause the API to crash due to out-of-memory errors. BioTransformer was set to run a sequence of four cycles of metabolism in the following sequence: one cycle of ECBased, two cycles of CYP450, and finally one cycle of Phase II, yielding four cycles of metabolism. The ECBased model was chosen to initiate the sequence since it possesses both Phase I and Phase II transformation rules to better capture any primary Phase II metabolites where they occur, and the sequence was set to four total cycles given practical limitations in terms of execution time (1 day vs multiple days). This transformation sequence successfully completed over the span of 24 hours for the dataset under study.

Other input parameters used to perform alternate metabolism simulations with the Toolbox are discussed in further detail in the Supplemental Information.

Metabolic Simulator Performance

The ability of each simulator to accurately predict metabolites for each parent was assessed over all experimentally observed metabolites for the parent, and across all generational levels simultaneously, rather than assessing predictive performance for each generational level. When assessing overall simulator performance (see Table 3), performance metrics were summed across all parent chemicals. The enumerated metrics are the number of true positive (TP) predictions (i.e., predicted metabolites in the simulated dataset that match observed metabolites in the empirical dataset), false negative (FN) predictions (i.e., observed metabolites in the empirical dataset that were not predicted in the simulated dataset), false positive (FP) predictions (i.e., predicted metabolites in the simulated dataset that were not observed in the empirical dataset), the total number of metabolites (also represented as the sum of TP+FN), and the total number of predictions (also represented as the sum of TP+FP). The recall (i.e., ratio of correctly identified metabolites to the total number of observed metabolites) is calculated as per Equation 1:

$$Recall=\frac{TP}{TotalReportedMetabolites}=\frac{TP}{TP+FN},$$ (Eq. 1)

and the precision (i.e., ratio of correct predictions to the overall number of predicted metabolites) is calculated as per Equation 2:

$$Precision=\frac{TP}{TotalPredictedMetabolites}=\frac{TP}{TP+FN},$$ (Eq. 2)

Table 3.

Predictive performance characteristics for the 432 observed Phase I and Phase II metabolite relationships in the combined literature and SMPDB Drug Dataset.

SMPDB/JCIM 112 Unique Drugs/NSAIDs	CTS ChemAxon Phase I 3 cycles	BioTransformer 1x EC-Based, 2x Cyp450, 1x Phase II 4 cycles	Toolbox API In Vivo Rat Simulator Phase I	Toolbox API In Vitro Rat Liver S9 Phase I	TIMES In Vivo Rat Simulator 6 cycles	TIMES In Vitro Rat Liver S9 6 cycles	Aggregate of All Models
True Positives	234	216	173	173	172	170	313
False Positives	13230	27296	1289	1030	1120	1162	38303
False Negatives	198	216	259	259	260	262	119
Total No. of Predictions	13464	27512	1462	1203	1292	1332	38616
Total Precision	0.017	0.008	0.118	0.144	0.133	0.128	0.008
Total Recall	0.54	0.50	0.40	0.40	0.40	0.39	0.73

An example case of determining simulator performance metrics is given in Figure 3 for Aripiprazole.

Figure 3.

Comparison of predicted metabolism map generated from TIMES against experimental metabolism map for Aripiprazole (DTXSID3046083) to illustrate how recall was derived.

InChIKey data were preferred over SMILES when making comparisons for TP, FP, and FN prediction assessments because InChIKey structural data are less variable than SMILES for a given chemical structure (due to variations in SMILES canonicalization between tools), and thus it was easier to ensure that the assignment of TP, FP, or FN for a prediction was reliable based on structural data.

Further performance assessment was accomplished via hierarchical clustering of the recall across chemical classes and grouped by model choice using chemical class. In these instances, the mean recall across a parent chemical class was given for each choice of model individually, as well as for all models together.

DATA AVAILABILITY AND CODE

With exception of metabolic simulations performed using TIMES (version 2.31.2.82), all work was performed using Python (version 3.10.4) run with IPython (version 8.4.0) in JupyterLab (version 3.3.2).^66–68 The Toolbox API (OECD Toolbox version 4.5 with Service Pack 1 update, API version 6), and BioTransformer (Wishart Lab, version 3.0, executable Java Archive, June 15, 2022 release) were used for automated metabolic simulations. Efficient batch execution of metabolism simulations was handled via parallel processing multiple individual calls to either BioTransformer or the Toolbox API via the “multiprocess” package.^{69, 70} The command line interface (CLI) calls needed to interact with BioTransformer were executed via the “subprocess” package, and the Toolbox API was queried via its Swagger user interface hosted on a locally running Windows Desktop instance of the Toolbox Server. The data generated from the MetSim hierarchical schema were translated into JavaScript Object Notation (JSON) format using Python. The resulting data were inserted into a Mongo Database (MongoDB) using the “pymongo” package for efficient storage, and retrieval. The code repository including all Jupyter Notebooks documenting the analysis performed and the MetSim framework are available at https://github.com/patlewig/metsim. Data files needed to reproduce the analysis are provided at https://doi.org/10.23645/epacomptox.25463926 and as Supplementary Information.

RESULTS

All tools finished their execution on the order of tens of seconds to slightly more than four minutes for the model parameters used in this study for each simulator. BioTransformer produced the most metabolites whereas the Toolbox produced the least.

Drug Dataset Performance

A summary of the performance metrics is given in Table 3. BioTransformer generated a total of 27512 metabolite predictions. Of those predictions, 216 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 216 FN predictions of the 432 reported metabolites, yielding a recall of 0.50. There were a total of 27296 FP predictions, yielding a precision of 0.008.

TIMES predictions using the in vitro rat model generated a total of 1332 metabolite predictions. Of those predictions, 170 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 262 FN predictions of the 432 reported metabolites, yielding a recall of 0.39, which was the lowest individual recall across all four tools. There were a total of 1162 FP predictions, yielding a precision of 0.128. The in vivo rat model generated a total of 1292 metabolite predictions. Of those predictions, 172 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 260 FN predictions of the 432 reported metabolites, yielding a recall of 0.40. There were a total of 1120 FP predictions, yielding a precision of 0.133.

Toolbox predictions with the in vitro rat model generated a total of 1203 metabolites. 173 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 259 FN predictions of the 432 reported metabolites, yielding a recall of 0.40. There were a total of 1030 FP predictions, yielding a precision of 0.144. The in vivo rat model generated a total of 1462 metabolite predictions. Of those predictions, 173 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 259 FN predictions of the 432 reported metabolites, yielding a recall of 0.40. There were a total of 1289 FP predictions, yielding a precision of 0.118.

CTS generated a total of 13464 metabolite predictions. Of those predictions, 234 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 198 FN predictions of the 432 reported metabolites, yielding a recall of 0.54, which was the highest individual recall across all four tools. There were a total of 13230 FP predictions, yielding a precision of 0.017.

When the predictions from all four of the tools were combined, as an ensemble approach, a total of 38616 unique metabolite predictions were generated. Of those, 313 TP predictions of the 432 total reported metabolites were correctly identified. Conversely, there were 117 FN predictions of the 432 reported metabolites, yielding a recall of 0.73, a 19% increase in recall compared to CTS, the highest performing individual tool. There were a total of 38303 FP predictions, yielding a precision of 0.008.

The lower recall values for TIMES and the Toolbox relative to CTS and BioTransformer could be in part attributed to the fact that rat simulators were being used to predict human metabolism information. The lack of quantification of the metabolic transformations in most the tools in terms of not considering the quantity of the metabolites or the feasibility of transformation occurrence (particularly in CTS and BioTransformer) also contributes to the generation of a large number of metabolites resulting in a biased estimation of performance due to the amount of simulator noise. Often experimental metabolism studies miss or are not able to detect intermediate metabolites due to them being too reactive or too small in size, such that the constraining the number of simulated metabolism levels to match the number of documented metabolism levels might also have contributed to the low recall values observed.

Analysis of Metabolic Predictions

Given that none of the models exhibited a high (>70%) recall, the underlying causes were explored, such as whether the choice of simulator, or the types of chemicals being simulated, or both may have played a role. Hierarchical clustering analysis of recall against the chemical classes within both datasets was performed and compared across all tools (see Figure 4). The highest recall rates across all tools were observed for carboxylic acids, phenanthrenes, lactams, benzothiazenes, and piperidines, comprising ~18% of the dataset (20 chemicals), where all tools resulted in high recalls. Conversely, all tools performed poorly on nucleoside and nucleotide analogues, 5’-deoxyribonucleosides, imidazopyrimidines, benzimidazoles, pyrrolines, and fatty acyls, comprising ~13% of the dataset (14 chemicals). The remaining ~69% of the dataset (78 chemicals) varied greatly in their recall, depending on the choice of tool and model. Examples of ensemble model recall rate improvement compared to the highest individual model recall for specific chemical class groupings included pyrrolines (+20% recall), fatty acyls (+13% recall), benzazepines (+11% recall), anthracyclines (+33% recall), and carboxylic acids (+4% recall).

Figure 4.

Hierarchically clustered heatmap of Recall clustered by chemical class for each metabolism simulator and model applied to the dataset. Model selections are indicated at the bottom of each column of the clustered heatmap as BioTransformer, CTS, Toolbox Vitro or Toolbox Vivo, and TIMES Vitro, TIMES Vivo or All Tools. Average recall for a given chemical class is illustrated by an increasingly dark gradient from zero (white) to unity (dark red) with the actual mean recall rate given in each cell. (Right) Bar chart of parent chemical class versus log2 scaled occurrence frequency in the dataset.

Further Evaluation of Selected Simulators

The training sets for BioTransfomer and TIMES were available which enabled an assessment of whether the performance characteristics reported could be reproduced. Single-step human metabolism in absence of gut predictions for BioTransformer was performed against the dataset of 40 drugs reported in their 2019 paper comparing BioTransformer against ADMET Predictor. Comparing the version of BioTransformer used in this study (version 3.0, June 2022 release) to the version used in the 2019 BioTransformer publication (version 1.1.5) (see Table S1 in the Supplementary Information), the highest recall (0.74) was produced using the sequential model of one cycle of the EC Based model, one cycle of CYP430 and one cycle of Phase II (precision 0.11). compared to the literature recall value of 0.88 (precision 0.49). Single-step phase I metabolism was also evaluated against a dataset of 60 drugs from the same paper, where recall with the current version of BioTransformer was measured to be 0.76 (precision 0.43) compared to the reported value of 0.90 (precision 0.46).²²

TIMES models were compared to their training set observed maps for each of the chosen simulators using functionality within the TIMES application. The training sets for each model (438 chemicals for In Vitro Rat Liver S9 and 701 chemicals for In Vivo Rat Simulator) were evaluated using both their default parameters and the parameter adjustments selected for uniformity across tools. The documented recall of 0.81 for the In Vitro Rat Liver S9 was improved in the version of TIMES used in this study with an observed increase in recall to 0.88 whether defaults were chosen or whether the current parameter threshold adjustments from 5 metabolites per cycle to 10 metabolites per cycle were used. The In Vivo Rat Simulator had a documented recall of 0.77, and in the version of TIMES used, recall was measured to be 0.76 with default parameters, and 0.69 when parameters were matched to those mentioned above for In Vitro Rat Liver S9 parameters with 10 metabolites per cycle.

Overall, the two simulators with available training data were generally capable of reproducing their own published results sufficiently such that the low recall observed was not attributable to parameter settings or systematic error in simulator execution.

DISCUSSION

A framework known as MetSim has been developed to aid the comparison of different metabolic simulator tools in terms of their performance. Toolbox and TIMES successfully yielded predicted metabolites for both in vitro and in vivo rat liver models. Both simulators could make batch predictions with faster execution times and higher precision as compared to CTS and BioTransformer. The main limitation of the Toolbox was its limitation to phase I predictions, which motivated the comparison with TIMES. BioTransformer and CTS successfully produced metabolite predictions for human tissues. The models used here were restricted to phase I and phase II metabolism only, and those that did not include gut microbiome predictions. While execution times were slower for BioTransformer, a richer data output was yielded from BioTransformer that included enzymatic transformation data, particularly in terms of the specific information content related to the enzymes involved, and the reaction mechanisms required to yield the transformation from a parent to its metabolite, though TIMES (at least the version used) and CTS also provided the latter. For all four tools, the data inputs and outputs were harmonized so that a unified hierarchical schema could be used to facilitate data storage of prediction datasets and experimental datasets within MongoDB, and for side-by-side comparisons of simulator performance.

One potential explanation for the lower recall with individual tools compared to the ensemble results from all tools on the dataset was that transformation rules were not encoded for every potential SMILES representation of a given parent. An example case of this was discovered in the June 2022 release of BioTransformer, where the simulator produced fewer accurate predictions for structures containing amide bonds with SMILES representations $RC\left(O\right)=N{R}^{\prime }$, as compared to one alternate SMILES representation for an amide $RC\left(O\right)N{R}^{\prime }$, where $R$ and $R^{\prime }$ represent the other branches of the chemical structure connected to the amide functional group. This was a unexpected result that the transformation rules embedded in either of these individual tools should have any difficulty reproducing metabolites for the dataset used in this study. The chemical class analysis in Figure 4 provides a means to diagnose where transformation rules could be improved.

An examination of whether there was any correlation between the log octanol-water partition coefficient logK_ow, related to water solubility and hydrophobic character of a parent chemical, and the observed recall for each individual model, as well as the aggregate recall over the union of all models was made. Results of Spearman correlation tests at a 95% confidence level, found that there were weak positive correlations between logK_ow and recall from each set of MetSim model parameters (correlation coefficients ranging 0.19 – 0.31, p-values < 0.05), with exception of the two TIMES models that showed no statistical significance between logK_ow and recall at the given confidence level (p-values of 0.059 for In Vitro Rat Liver S9, 0.144 for In Vivo Rat Simulator). When the mean logK_ow values per chemical class were aligned with the sorting of the hierarchical clustering analysis from Figure 4, there was no consistent trend in logK_ow with recall. The detailed results of this analysis are given in the Supplemental Material (Figure S6, and Table S6).

Model Selection

Choice of model had some impact on simulator performance—as observed in Table 3 and Figure 4 for the Toolbox and TIMES models. However, it is possible that other models could be selected for any choice of simulator. TIMES, the Toolbox, and BioTransformer contain a larger variety of metabolic models than the ones used here. BioTransformer phase I and phase II liver simulators were selected in absence of human gut microbiome transformations (“hgut” model). Another option could have been to run all four human models sequentially via the “allHuman” aggregate model. TIMES has simulators incorporating gut, skin as well as lung tissue metabolism, as does the Toolbox.

Inherent Trade-Offs Between Recall Rate and Precision

BioTransformer and CTS tended to produce substantially more metabolite predictions compared with the Toolbox and TIMES, often between one to two orders of magnitude more than Toolbox or TIMES. Consequently, BioTransformer has the lowest precision of any of the individual tools studied. The ensemble of all model predictions suffered from the same low precision, but with improvement in recall. Figure 5 shows a ROC Curve to summarize the performance across all the simulators.

Figure 5.

ROC Curve showing the trade-off between TPR and FPR across the different simulators. TB(TIMES)_iv and TB(TIMES)_ivt refers to the Toolbox or TIMES in vivo and in vitro models.

On the other hand, while precision might be low for BioTransformer compared to other tools, the lack of probabilistic thresholding does provide a broader picture of all possible transformations that could result from a parent chemical. This functionality has uses in applications such as Non-Targeted Analysis (NTA), where previously unobserved or otherwise data-poor chemicals are often discovered in environmental samples via methods such as high-resolution mass-spectrometry. One could query lists of observed candidate structures for chemicals of emerging concern (CECs) from a NTA experiment against existing predictions from BioTransformer as one approach to correlate potential precursors to metabolite CECs discovered within an environmental sample.

Future Directions

Metabolic similarity is a critical consideration in improving analogue identification and evaluation methods for read-across. In the absence of empirical metabolism data, reliance on robust metabolism predictions is key. For read-across tools, the aggregate of all models may hold most promise in building metabolic similarity profiles from in silico predictions. Though it should be noted that the simple aggregation performed in this study was a pragmatic ensemble approach in pooling predictions together given no single tool appeared to perform particularly well. The applicability domain of each model should be carefully considered before aggregating predictions from different species for more broader use. Adding these and other predictions into a larger database of predicted metabolites for each simulator is the subject of ongoing work.

CONCLUSIONS

A metabolic simulation framework was developed called MetSim that was applied to a dataset of 112 chemicals using 4 metabolic simulator tools. The framework facilitated a comparison of performance across the 4 tools. CTS yielded the highest individual recall for the dataset with a recall of 0.54 using three generations of Phase I metabolism via its ChemAxon Human Phase I model, whereas the TIMES In Vitro Rat Liver S9 model yielded the lowest individual recall at 0.39. Aggregating the results of all six models increased the total recall by 18% to 0.72. There was a tendency for tools with high recall (i.e., higher true positive rates) to be associated with low precision values (i.e., high false positive rates). The Toolbox and TIMES models yielded a fairly consistent recall of 0.39–0.40, with precision ranging between 0.118–0.144. BioTransformer and CTS had higher recall values between 0.50–0.54, but precision ranged between 0.008–0.017. The ensemble model yielded the lowest precision of 0.008. Hierarchical clustering analysis on the basis of chemical class offered insights for where transformation rules/models could be improved and which types of chemicals performed poorly depending on each tool.

Data Availability Statement

The code has been made available at https://github.com/patlewig/metsim and the processed data will be downloadable from figshare.