Machine Learning Models for Mycobacterium tuberculosis in vitro Activity: Prediction and Target Visualization

Abstract

Tuberculosis is a major global health challenge, with approximately 1.4 million deaths per year. There is still a need to develop novel treatments for patients infected with Mycobacterium tuberculosis (Mtb). There have been many large-scale phenotypic screens that have led to the identification of thousands of new compounds. Yet, there is very limited investment in TB drug discovery which points to the need for new methods to increase the efficiency of drug discovery against Mtb. We have used machine learning approaches to learn from the public Mtb data, resulting in many datasets and models with robust enrichment and hit rates leading to discovery of new active compounds. Recently we have curated predominantly small molecule Mtb data and developed new machine learning classification models with 18,886 molecules at different activity cut-offs. We now describe the further validation of these Bayesian models using a library of over 1000 molecules synthesized as part of the EU-funded New Medicines for TB and More Medicines for TB programs. We highlight molecular features which are enriched in these active compounds. In addition, we provide new regression and classification models which can be used for scoring compound libraries or used to design new molecules. We have also visualized these molecules in the context of known molecular targets and identified clusters in chemical property space which may aid in future target identification efforts. Finally, we are also making these datasets publicly available, representing a significant increase to the available Mtb inhibition data in the public domain.

Graphical Abstract

INTRODUCTION

Tuberculosis (TB) represents a major world health challenge for which the search for new drug treatments continues. In 2019, the WHO annual TB report described ~1.4 million fatalities due to TB and 10.4 million new TB cases¹. Drug-sensitive TB treatment has an 85% success rate; however, current TB treatments for multi-drug-resistant (MDR) Mycobacterium tuberculosis (Mtb) is a challenge as the available treatments suffer from lower efficacy, longer treatment duration, and debilitating side effects that range from liver toxicity, hearing loss, and psychosis^2–4. Streptomycin was discovered in 1944 which is considered the ‘golden age’ of antibiotic drug discovery, which led to the TB drugs isoniazid and pyrazinamide (INH and PZA, 1952), ethambutol (EMB, 1961) and rifampicin (RIF, 1963), and remain the cornerstones of drug susceptible Mtb therapy today. There has been extensive research to develop molecules that overcome Mtb drug resistance while shortening the duration of TB therapy^5–8. Relapse and treatment failure are observed with ca. 5% of drug susceptible cases^{9, 10} and in up to 50% of multidrug-resistant tuberculosis (MDR-TB) cases,¹¹ despite prolonged treatment. Extended therapy and potentially limited access to therapy (requirement for daily injectable drugs and IV infusions) heightens the probability of partial non-compliance^{2, 4}, which leads to both treatment failure and the emergence of new drug resistance¹². Of major concern are reports of totally drug resistant (TDR) Mtb infections that are now resistant to all known drugs^{13, 14}. Development of new treatments has been slow with only four new approved drugs in the last 40 years for TB (bedaquiline, delamanid, linezolid and pretomanid). However, these molecules are not widely used due to cost and concern regarding MDR-TB^{15, 16}.

While the last 20 years have seen an increase in phenotypic and target-based screening for new drugs, these approaches have proven to be inefficient. Something must be done quickly to gain control of TB and provide any hope of achieving the WHO End TB goals of reducing TB deaths by 95% and cutting new cases by 90% between 2015 and 2035¹⁷ especially as from 2015–2019 there was only a 9% reduction in TB incidence and 14% reduction in deaths¹. Due to the global COVID-19 pandemic it is likely that the incidence of TB and associated deaths will increase^18–20. To achieve these ambitious goals, shorter and simpler treatments are needed for both drug susceptible and resistant TB. There is therefore a dire need to develop and apply alternative approaches to Mtb drug discovery.

With a lack of funding for future research²¹ there is a need for enhancing the efficiency of TB drug discovery efforts. A faster path to new TB treatments may be to revisit and optimize preexisting drugs²². This could take the form of rescuing activity on drug resistant TB or improving activity or drug delivery/distribution to shorten treatment times, prevent reoccurrence and improve compliance to halt further drug resistance. Gopal and Dick²³ proposed to go back to old TB drugs that are more like fragments (low molecular weight ~200–250 Da, e.g. INH (137.14 Da) and PZA (123.11 Da) or even smaller)²³. There is increasing interest in drug repurposing, taking drugs that are already approved for another use and applying them for additional diseases²⁴. This can be facilitated using high-throughput screening of libraries of approved drugs from the FDA or other regulatory agencies, computational or virtual screening using docking or pharmacophores, machine learning or combinations of methods, knowledge-based methods, gene-expression data or serendipity^24–29. These various computational or experimental approaches are particularly important for addressing neglected and rare diseases where funding for drug discovery²¹ may be negligible or even non-existent or where an already approved drug would shorten the path to the clinic or shorten the treatment regimen.

There are 3 barriers to drug delivery including penetration of the granuloma, macrophage and Mtb cell wall³⁰. For example, recent observations suggest that first line drugs may have much lower concentrations inside the granuloma than in the blood^31–33. Ideally the most desirable TB drug properties include the inhibition of a druggable target, synergistic with other therapies, and a rapidly sterilizing action. A new TB drug should also ideally be cidal to clinical drug-sensitive as well as drug-resistant strains currently circulating. This may be achieved by potently inhibiting an essential gene product not modulated by currently approved TB therapies. To date the phenotypic screens providing whole-cell efficacy of molecules has been a primary driver in drug discovery efforts, but this does not provide information on primary target nor molecular interactions of the small molecule and the protein involved. Many years of drug development have uncovered several bacterial processes that contain excellent new targets for antibacterials^34–37. These pathways include cell wall biosynthesis which may be important for Mtb. Drugs that can function by reducing the treatment duration for both drug susceptible and MDR-TB would also have high impact on the field. Antitubercular agents are therefore needed which demonstrate cidality in vitro and in vivo to reduce the bacterial infection. Drugs and combinations that are rapidly sterilizing in vivo would also be highly desirable and are more likely to lead to shortened therapy. Subsequently, the bacterial cell wall has been one of the most successful targets for Mtb drug discovery with molecules such as isoniazid (INH), ethambutol, and ethionamide. INH is a front-line TB therapy which is cidal in vitro and in vivo inhibiting the enoyl-acyl carrier protein reductase (inhA)^{36, 38} within the FAS-II system³⁹. INH demonstrates biphasic kill kinetics in vitro, with resistance and persistence developing within a week of exposure which decreases cidality but this may be overcome by use of drugs that are synergistic with it. As mycolic acids are structural elements of the cell wall and essential for Mtb in vivo pathogenesis^40–42 the type I fatty acid synthase (FAS-I) and FAS-II systems represent important targets⁴². Two different fatty acid biosynthetic steps have previously been described representing additional targets, namely Pks13⁴³ and KasA^44–46. Molecules that could potentiate the activity of INH as well as eradicate INH-induced persisters, may provide some substantial benefit for the treatment of TB patients.

For over a decade, we have focused our research on using machine learning models for in vitro Mtb datasets, initially using Bayesian approaches⁴⁷, which we have also used widely for modeling ADME/Tox properties^{48, 49}. We have modeled large Mtb high throughput screening (HTS) datasets that were made available by SRI/NIAID^{50, 51}, as well as smaller datasets from Novartis⁵². Such Bayesian models were initially used to predict and rank molecules identified with pharmacophore methods before testing in vitro⁵³, leading directly to subsequent prospective tests of models built with in vitro data. The first such study used Bayesian models to identify five active molecules out of just 7 tested⁵⁴. A second study tested 550 molecules in vitro, identifying 124 actives⁵⁵. A final study found 11 actives out of 48 compounds tested⁵⁶. We also performed a retrospective validation with 1,924 molecules as a test and showed enrichments greater than ten-fold⁵⁷. We also explored combining Mtb data sets followed by testing the machine learning models on data from other groups^{58, 59}, which pointed to the need for multiple Mtb models as well as proposing important molecular features for actives⁵⁹. These molecular features have in turn been utilized to create new β-lactam antibacterials⁶⁰. Our use of machine learning for Mtb has also expanded to models for multiple targets^{61, 62} such as ThyX²⁷ and Topoisomerase I⁶³. We have also developed machine learning models using mouse models of Mtb infection⁶⁴ where there is data built up over 70 years. This allowed us to collate nearly 800 molecules⁶⁴ which we have also extended with additional data⁶⁵. More recently we have described using multiple machine learning approaches with 18,886 molecules (with activity cut-offs of 10 μM, 1 μM and 100 nM)⁶⁶. These models were both internally and externally validated. The Bayesian model at the 100 nM activity cut-off yielded promising metrics for 5-fold cross validation (Accuracy = 0.88, Precision = 0.22, Recall = 0.91, Specificity = 0.88, Kappa = 0.31, and MCC = 0.41) which resulted in comparable statistics for an external evaluation set (n = 153 compounds). We also compared these models with multiple machine learning algorithms demonstrating Bayesian models were equivalent to or outperformed other machine learning methods with external test sets. In recent years several diverse classes of antitubercular compounds have been identified with activity in vitro or in vivo as summarized by our recent analyses of over 100 active leads. We illustrated that researchers are repeatedly identifying compounds that are similar to molecules that have previously been described⁶⁷. This led us to suggest that the field needs to dramatically increase the diversity of chemical libraries tested in order to explore beyond the historic Mtb property space. Machine learning approaches may represent one way to do this.

We have recently participated in the EU funded public-private consortia New Medicines for Tuberculosis (NM4TB) and More Medicines for Tuberculosis (MM4TB) which ran from 2005–2016^{68, 69}. In the process of this work > 1000 molecules were synthesized and tested and some of these molecules were described in earlier publications^70–95. We have now curated this data and used it as a test set for our machine learning models as well as a unique training set for regression model building and for prediction of the potential targets for these compounds. In the course of this work, we are now making this data publicly available.

EXPERIMENTAL SECTION

Computing

All computing was done on three servers: (1) 2x Intel(R) Xeon(R) CPU E5–2620 v4, 64GB DDR4, 2x GeForce GTX 1080 Ti, (2) 2x Intel(R) Xeon(R) CPU E5–2630 v4, 128GB DDR4, 4x GeForce GTX 1080 Ti, (3) 2x Intel(R) Xeon(R) CPU E5–2630 v4, 128GB DDR4, 4x GeForce GTX 1080 Ti. Each server is running CentOS 7 with recent updates. Docker Engine Community v19.03.8 was installed on all three servers and was used to host all parts of the application. MongoDB 4.2.5 cluster is used to support all persistent operations. RabbitMQ 3.8.3 cluster is used to orchestrate asynchronous fail-safe long-running operations. Anaconda 4.8.3 with Python 3.7.6 was used to create the following environment: connexion, html2text, matplotlib, numpy, pandas, rdkit, scikit-learn, mongoengine, jinja2, pika, xgboost=1.3.3, docker-py, XlsxWriter, seaborn, gevent, epam.indigo, flask-cors, tf-nightly-gpu.

Data Curation

Our proprietary software “E-Clean” was used to “clean” and average activities for datasets prior to model building in Assay Central. “E-Clean” handles duplicate compounds by either averaging, removing, or keeping duplicates based on InChIKey. For these data, duplicate molecules with continuous activity data were first converted to -logM and then were averaged. “E-Clean” logs the SMILES strings of the duplicate compounds along with their activities and indices for inspection by the user. If needed, compounds are also subjected to charge neutralization, salt removal, and standardization via custom software using open-source RDkit functions.

The standardization within Assay Central was done as follows: A simple standardization workflow consisting of the following steps and using the Indigo Toolkit⁹⁶ is applied: read molecule from the string representation (e.g. SMILES or MOL), generate InChI and InChIKey, use InChIKey to find and remove duplicates, dearomatize, remove enhanced stereo, remove unknown stereo, standardize and reposition, if necessary, stereo bonds (e.g., wedged bonds), standardize or flag erroneous charges, flag erroneous valences, remove isotopes, remove dative and hydrogen bonds, remove smaller component if multicomponent chemical, flag multicomponent chemicals, neutralize. All chemicals which are duplicates or flagged with errors (e.g. erroneous valences or charges) are then excluded from the result, but all erroneous or duplicate records are included into a Protocol associated with a given Dataset and available for review in the user interface.

The literature Mtb data was compiled from four distinct sources described previously⁶⁶. A dataset that had not previously been used for machine learning, referred to as the “RCB” dataset, consisted of 1193 unique compounds. These compounds were synthesized during the last 12 years as part of the antituberculosis consortiums NM4TB and MM4TB. About 15 % of these compounds are related to Macozinone which was one of the main achievements of these two consortiums. Antituberculosis testing of these compounds with MIC₉₉ determination was made under the same conditions by standard resazurin assay⁹⁷ therefore they can be compared with each other. The synthesis and activity against Mtb for many of these compounds has been previously published^70–95. The activity data were both in MIC and MIC₉₉ values, which were defined as equivalent based on the criteria for Mtb activity from our previous published work. An additional step was used to quantify activity values that were ill defined (i.e. did not have an equal qualifier). Activity values that were reported as “<” were given a value of 0.5x the lowest non-ambiguous value and those that were “>” were assigned a value of 2x the highest defined value, but this was only used for classification models. For the continuous models all values that were “>” were removed from dataset for purposes of model building or when used as a test set for a regression model. In addition, there were 21 compounds from the RCB set that were in the previously published training set (classification). There were also 9 compounds that overlapped with the training set for the large regression Mtb model, which were removed from the test set to reduce bias.

Model Building with Assay Central (Legacy version)

We previously described how we generated Bayesian machine learning models with the literature Mtb data⁶⁶. Three different TB Bayesian models of 18,886 molecules with cutoffs at 100 nM, 1 μM or 10 μM models were used from our Assay Central (AC) legacy software. Each model was previously internally validated by a five-fold cross-validation, where 20% of the dataset is left out and used as a test set five times and Receiver operator characteristic (ROC) plots were generated along with other model statistics⁶⁶. These previously published models were used to score the RCB external test set for comparative purposes.

Model Building with Assay Central

Our updated proprietary Assay Central (AC) software uses multiple algorithms that are integrated in our web-based software to build classification models as described previously⁶⁶ along with the addition of four new algorithms: linear regression, graph convolutional network (GCN), and end-to-end Convolutional Long-Short Term Memory Neural network (ConvLSTM-net), and XGboost. Furthermore, we have implemented multiple regression algorithms for model building of continuous data which include adaboost regression, Bayesian regression, elastic net regression, knn regression, random forest regression, support vector machine regression and XGboost regression. We removed 20% of the training set, in a stratified manner for the classification models, and these were used as external test sets for models trained on the remainder of the data. The same method was also used for the continuous data model validation.

Compounds and in vitro testing

The majority of compounds have been previously published^70–95. The determination of minimum inhibitory concentration (MIC) against M. tuberculosis strain H₃₇Rv and M. smegmatis was determined by the resazurin reduction assay methods which have been described previously⁹⁷.

Model Validation (General)

Machine learning model validation was performed in several ways: a 20% leave out set, 5-fold cross validation and with external test sets. For the “20% leave out (LO)” classification model validation a random, stratified 20% is removed from the training set prior to model building. The model is then built with the remaining 80% of the training data and the hyperparameters (if applicable) are optimized using a grid search using five-fold dataset splits. Using the optimized hyperparameters a final model is built, which is then used to predict the activity of the LO set as an external test set. 5-fold cross validation is done using the Legacy software and since Bayesian does not use hyperparameters these cross-validation statistics are not biased. The validation statistics are generated for each of the 20% holdout sets and are then averaged together. These differences are visualized in Figure S1. External test sets are independent of the training data and are true external evaluations of the models. For regression model validation the data is split randomly and is chosen independent of actual activity.

Model Validation (Comparison)

For a direct comparison of our legacy and updated AC software we built Mtb inhibition models using the same training data as was previously published (18,886 molecules) at various thresholds (100nM, 1μm, 10μM). Due to a slightly different automated compound cleanup process our latest machine learning models built in the updated AC software included 18,738 compounds. Both the original test set (258 compounds)⁶⁶ and RCB dataset (1196 compounds) were used as external test sets for these models.

Descriptors.

FCFP and ECFP fingerprints as generated by RDKit (http://www.rdkit.org/) are used with the options of specifying the radius of 2–4 (which corresponds to [EF]CFP4–8) and bits 512, 1024 and 2048. The particular type of the fingerprints is chosen before running the model training from the UI with an option to specify a combination of fingerprints. For the GCN, molecules were first translated into a graph G(V,E), with each atom comprising a node in V and each molecular bond comprising an edge in E. Each node was featurized with the default canonical features described in the DGL-LifeSci⁹⁸, documentation, including Atom type, valence states, hybridization, chirality, whether the atom is in a ring, etc. For the ConvLSTM-net input, molecules are represented as tokenized SMILES strings. Briefly, each SMILES is tokenized, and each character is represented in a vocabulary (e.g. “c” [nH]”, “1”, =”). Each token in the vocabulary has a corresponding numerical representation (e.g., all “c” are represented by 1, all “=” are represented by the number 2, etc). SMILES are encoded by their integer vocabulary representation and padded to the longest sequence length with zeroes which were masked during training.

Machine learning.

The methods utilized have been previously published by our group in most cases⁶⁶ and further details are described in the Supplemental Methods.

t-SNE visualization

t-SNE⁹⁹ embeds data into a lower-dimensional space. 1024-bit ECFP6 fingerprints were generated for all compounds. The ECFP6 fingerprints were then embedded into a 2-dimensional vector using t-SNE. All t-SNE values were generated using the scikit-learn library in python with default hyperparameters (n_components = 2, perplexity = 30, early exaggeration = 12.0, learning rate = 200, n_iter = 1000).

Statistical analysis

In this study, several traditional measurements of model performance were used as described previously⁶⁶ and further details are provided in the Supplemental Methods.

Rank Normalized Score and ΔRNS

External test set validation metrics between algorithms for 3 datasets (100 nM, 1 μM and 10 μM) were compared with a rank normalized score, which was also used in previous investigations^{100 101, 102} of various machine learning methods. Rank normalized scores can be evaluated in either a pairwise or independent fashion: the former is informative for a comparison of algorithms per training set, while the latter provides a more generalized comparison of algorithms overall. Additionally, a “difference from the top” (ΔRNS) metric, where the rank normalized score for each algorithm is subtracted from the highest corresponding score of a specific dataset¹⁰². This metric maintains the pairwise comparison results and enables a direct assessment of the performance of multiple machine learning algorithms. Statistical analyses were performed in Graphpad Prism 9.2.0.

Atomic Contribution to Model Prediction Similarity Maps

Coloring of atomic contributions to model predictions was performed using RDkit as previously described¹⁰³. Briefly, model prediction scores were calculated for each molecule using ECFP6 fingerprints. We used a random forest model (described in methods above) to make model predictions. A weight is determined for each atom in the molecules by removing the circular fingerprint bits associated with the atom (each bit in an ECFP fingerprint has an associated set of atoms which can be determined in RDkit) followed by making a prediction on the molecule with the removed associated bits. The difference in prediction score probabilities between the original molecule fingerprint and the removed fingerprint bits is the atomic weight for each atom. The atomic weights are normalized by dividing all weights by the maximum absolute weight. Bivariate gaussian distributions are then fit over each atom, with the atomic weight corresponding to the peak of each atom gaussian (the variance remains the same among all atoms). This similarity map is then superimposed on an image of the molecule, with green indicating positive atomic contributions to prediction score (i.e., when bits containing the presence of the atom is removed, the prediction scores decrease) and red indicating negative atomic contributions to prediction scores. White or non-gaussians represent atoms which contribute little-to-none to the prediction of the models.

RESULTS

Analysis of RCB data

We calculated and analyzed the active (≤100nM) and inactive molecules from the RCB dataset as separate groups using simple molecular descriptors from Chemaxon software (Budapest, Hungary) to discern significant differences between these groups. The molecular descriptors used to describe each dataset were molecular weight, AlogP, molecular fraction polar surface area, logD (pH 7.4) as well as the number of aromatic rings, hydrogen bond acceptor and donor, rings and rotatable bonds. Normality tests were done for each group and all but the LogD active group failed the normality tests (Prism 9.2) therefore the median was used to represent these data and all follow-up comparison tests assumed non-parametric data distributions (Mann Whitney and Kolmogorov-Smirnov tests). Surprisingly, all characteristics examined except for the distribution of aromatic rings were statistically distinct between the active and inactive groups (Figure 1). The three most notable differences were in the molecular weight (Median active, 451.9; inactive, 319.3) and number of hydrogen bond acceptors (active, 6; inactive, 4) and donors (active, 0; inactive, 1). The distribution of the hydrogen bond donors was particularly different, where ~90% of the active (<100nM) compounds had none. The active compounds have a higher median AlogP and logD (pH 7.4) suggesting that on average these compounds are more hydrophobic, a trend that we had previously underscored^{66, 104, 105}.

Figure 1.

Comparison of the distribution of molecular properties for ~1200 unique compounds in the RCB dataset split into active (≤100nM; 112 compounds) and inactive (1081 compounds) groups. Median values are highlighted along with statistical analyses. Noted, is that gaussians are meant to visually highlight each group and are not meant to imply a normal distribution.

Internal Validation for the models built with RCB data

For comparison with our previously published work on Mtb⁶⁶, the RCB dataset was binarized and analyzed using the same thresholds for the previously generated classification models (100nM, 1μM and 10μM). The total number of unique compounds for the RCB dataset is 1193 with the number of actives for the 100nM, 1μM and 10μM datasets of 112, 190 and 309, respectively (Figure S1). All the classification models had excellent validation statistics for the 20% leave-out set, with average cross validation ROC AUC values (across 8 algorithms) of 0.91–0.97 and an average F1 score range of 0.73–0.78 (Table 1).

Table 1.

Validation statistics of a 20% leave-out set for RCB Mtb models at various activity thresholds.

Mtb inhibition (RCB, MIC); 100 nM Threshold 112 (Active) / 1091 (Inactive)
	AUC	F1-Score	Precision	Recall	Accuracy	Specificity	Cohen’s Kappa	MCC
DL	0.987	0.737	0.875	0.636	0.958	0.991	0.715	0.725
ADA	0.972	0.700	0.778	0.636	0.950	0.982	0.673	0.677
BNB	0.963	0.816	0.741	0.909	0.962	0.968	0.796	0.801
KNN	0.938	0.744	0.762	0.727	0.954	0.977	0.719	0.719
LREG	0.990	0.837	0.857	0.818	0.971	0.986	0.821	0.821
RF	0.977	0.792	0.677	0.955	0.954	0.954	0.767	0.782
SVC	0.990	0.809	0.760	0.864	0.962	0.972	0.788	0.790
XGB	0.980	0.791	0.810	0.773	0.962	0.982	0.770	0.770
Mtb inhibition (RCB, MIC); 1 μM Threshold 190 (Active) / 1003 (Inactive)
	AUC	F1-Score	Precision	Recall	Accuracy	Specificity	Cohen’s Kappa	MCC
DL	0.966	0.769	0.750	0.789	0.925	0.950	0.724	0.725
ADA	0.958	0.746	0.862	0.658	0.929	0.980	0.706	0.714
BNB	0.949	0.750	0.714	0.789	0.916	0.940	0.700	0.701
KNN	0.954	0.762	0.696	0.842	0.916	0.930	0.712	0.716
LREG	0.963	0.775	0.738	0.816	0.925	0.945	0.730	0.731
RF	0.972	0.821	0.800	0.842	0.941	0.960	0.786	0.786
SVC	0.969	0.829	0.773	0.895	0.941	0.950	0.794	0.797
XGB	0.952	0.722	0.765	0.684	0.916	0.960	0.673	0.675
Mtb inhibition (RCB, MIC); 10 μM Threshold 309 (Active) / 884 (Inactive)
	AUC	F1-Score	Precision	Recall	Accuracy	Specificity	Cohen’s Kappa	MCC
DL	0.920	0.735	0.782	0.694	0.870	0.932	0.650	0.652
ADA	0.868	0.615	0.762	0.516	0.833	0.944	0.513	0.529
BNB	0.901	0.699	0.705	0.694	0.845	0.898	0.595	0.595
KNN	0.918	0.774	0.707	0.855	0.870	0.876	0.684	0.690
LREG	0.919	0.729	0.768	0.694	0.866	0.927	0.640	0.642
RF	0.928	0.762	0.750	0.774	0.874	0.910	0.677	0.677
SVC	0.925	0.758	0.758	0.758	0.874	0.915	0.673	0.673
XGB	0.920	0.740	0.723	0.758	0.862	0.898	0.646	0.647

RCB Test set analysis using existing classification models

Since we have added additional algorithms to our updated software, we have also rebuilt the classification models from our previous 2018 publication⁶⁶ using the same previously manually curated datasets. One purpose of this was to analyze the predictive power of these models using both cross and external validation and to do a head-to-head comparison to our legacy AC software.

To compare between our legacy (Bayesian only) and updated software (8 classification algorithms and a consensus model for external testing only) we first compared the cross-validation statistics. It should be noted that the cross-validation comparison has a caveat, with the legacy software scores being averages from 5-fold cross validations and the updated software is assessing a single 20% leave-out set only to avoid bias due to data leakage with algorithms that use hyperparameters (Figure S2). The average validation scores for the 20% leave-out set were very good at each threshold with AUC of 0.77 – 0.92 and F1 scores having a range of 0.41 – 0.61 (Figure S3). The best individual machine learning algorithms for the leave-out set, based on their rank normalized scores¹⁰⁰, were XGboost for 100 nM and svc for both the 1 μM and 10 μM thresholds (Figure 2A–B). This analysis is expanded for each algorithm by metric (Figure S4).

Figure 2.

Mtb machine learning algorithm comparisons for an internal leave-out (A-B) and external test set (RCB) set (C-D) at three thresholds (100 nM, 1 μM and 10 μM) based on the rank normalized scores. Rank normalized score (A, C) and ΔRNS (B,D) distributions. Each floating bar represents the entire range of the scores and individual points are shown in yellow. The solid central line represents the median. Legacy AC = Legacy Assay Central (Bayesian), rf = Random Forest, knn = k-Nearest Neighbors, svc = Support Vector Classification, bnb = Naïve Bayesian, ada = AdaBoosted Decision Trees, DL = Deep Learning, xgb = xgboost, lreg = linear regression.

In addition to cross-validation analysis we also used the Bayesian models built with the published datasets⁶⁶ to score the RCB dataset as an external test set. The external test set was very well predicted with a consensus AUC range of 0.75 – 0.76 and an average F1 score of 0.61 – 0.65 (Table 2). Accuracy was particularly high with the 100 nM and 1 μM thresholds where the average accuracy was 0.93 and 0.89, respectively (Table 2). The best individual ML algorithms for this external test set, based on their rank normalized scores¹⁰⁰, were XGboost, random forest and naïve Bayesian for the 100 nM, 1 μM and 10 μM thresholds, respectively (Figure 2C–D). These differences were not statistically different from each other, but this is likely due to the small sample size. This analysis is expanded for each algorithm by metric in (Figure S5).

Table 2.

External test set statistics using the RCB dataset with Mtb Bayesian classification models built with a previously published dataset⁶⁶ at various thresholds.

The validation statistics for the individual models are broken down further in Table 3C. For the updated software the scores for the consensus model are highlighted, but the individual statistics are also shown. Both models were able to predict these new data similarly, with a tendency to have a higher accuracy/specificity but a lower recall between the legacy and updated software, respectively (Table 3). Across all these models there was an increase to metrics developed to account for imbalanced dataset (F1 score, MCC and Cohen’s kappa) suggesting these have higher predictive power.

Table 3.

Head-to-head comparison of legacy and updated Assay Central using the RCB Mtb dataset as an external test set. (A) Validation statistics for Legacy Assay Central, (B) Assay Central consensus models and (C) full statistical breakdown by ML algorithm.

RCB Test set analysis using GCN and ConvLSTM net

In addition, we have also built a GCN model and a ConvLSTM neural network model to compare against the classical models described above. Our goal was to compare these newer algorithms to determine if they were significantly better or worse than the classical algorithms (Table S1). Our results suggest that the GCN is competitive with the classic machine learning algorithms at the 100nM cutoff and in the case of the 1 μM and 10 μM is one of the top-performing models. At the 1 μM threshold, the GCN displays the highest MCC and Cohen’s Kappa, while at the 10 μM threshold it maintains the highest AUC, albeit having a tied or slightly lower MCC and Cohen’s Kappa compared to the top classical models (BNB, RF, and DL).

ConvLSTM-net shows a significant improvement over DL at the 1 μM and 100nM threshold, showing a significant boost in the MCC, Cohen’s Kappa, and F1-scores, while lagging behind the DL model at the 10 μM threshold. This suggests that different architectures may excel under different scenarios, however ConvLSTM-net may serve as a better architecture over DL with ECFP6 fingerprints for investigating structure activity relationships, at least in the case of building models for Mtb inhibition with 2/3 threshold performing significantly better than DL. However, ConvLSTM-net did not improve significantly over most of the existing classical algorithms, which may be related to the challenge of hyperparameter optimization or end-to-end learning.

Regression models and test set

We have also built regression models for both the RCB data and for the previously published Mtb inhibition datasets⁶⁶. We used a 20% leave out set to emulate an external test set for model internal validation for the Mtb inhibition models. In the case of the previously published Mtb dataset, we also used the RCB dataset as an external test set to access its predictive ability. Only compounds that had an actual MIC were used to train or test these model (i.e. we removed those with > a value).

The internal validation statistics for the RCB data continuous model were favorable, with svr (MAE, RMSE and R² of 0.67, 0.90 and 0.69, respectively) and rfr (MAE, RMSE and R² of 0.65, 0.92 and 0.68, respectively) performing the best for the 20% leave-out set, though all algorithms tested performed well except for adar (Table 4, Figure S6).

Table 4.

Cross validation statistics for the regression models built with the RCB Mtb inhibition data. Note that all compounds that did not have an exact MIC (i.e. > 100μg/ml) were removed from the regression training set. Regression model statistics are calculated with -logM units.

Mtb Inhibition (RCB, MIC); Regression 628 Unique Compounds Range: 0.59 nM = 0.65 mM
20 % Leave-Out Set
Method	MAE	RMSE	R2
adar	0.79	0.99	0.62
br	0.73	0.92	0.67
enr	0.76	0.94	0.66
knnr	0.67	0.91	0.68
rfr	0.65	0.92	0.68
svr	0.67	0.9	0.69
xgbr	0.77	1.12	0.52

The regression model built with the previously published data (Table 5 and Figure S7) was chemically diverse and after all duplicates and ambiguous molecules were removed the training set included 15,618 unique molecules and had an activity span of 0.1 nM – 15.85 mM. The svr algorithm performed better with the 20% holdout set than the other tested algorithms with an MAE = 0.42 and RMSE = 0.56 (Table 5). Using an average prediction score, the model predicted the previously published test set well with and MAE and RMSE of 0.53 and 0.67, respectively (Figure 3A). The model performed similarly with the RCB dataset with R², MAE and RMSE of 0.46, 1.00 and 1.21, respectively (Figure 3B). While svr performed better than average, it was a small variation, with MAE and RMSE of 0.98 and 1.20, respectively.

Table 5.

Regression model validation statistics for the previously published Mtb data. This was done by internal validation of a 20% leave-out set.

Mtb Inhibition (Published, MIC); Regression 15,618 Unique Compounds Range: 0.1 nM = 15.85 mM
20 % Leave-Out Set
Method	MAE	RMSE	R2
adar	0.61	0.78	0.11
br	0.53	0.68	0.3
enr	0.54	0.7	0.28
knnr	0.45	0.6	0.46
rfr	0.46	0.6	0.47
svr	0.42	0.56	0.54
xgbr	0.5	0.67	0.34

Figure 3.

Regression model validation using the RCB Mtb dataset and published test set as external test sets for the 15,618 compound Mtb regression model.

t-SNE Plot for Model Property Space Analysis

t-SNE plots were used to visualize the difference in the chemical space explored between the RCB dataset and our previously published Mtb dataset⁶⁶. As both datasets are large and have some overlap, we also took additional steps to help to differentiate these libraries based on their activity. We visualized only the active compounds at the same thresholds used for the classification models: 100 nM, 1 μM and 10 μM. The t-SNE plot coordinates were not recalculated for each activity threshold, but the inactive compounds were filtered visually. As the activity thresholds became more stringent the separation between the two libraries became more apparent (Figure 4). Two example groupings highlighted by the t-SNE plot at 1 μM are a series of novel likely DprE1 inhibitors (Figure 4C–D) and the likely cell-wall synthesis inhibitors (nitroimidazoles), showing the ability of these plots to partition compounds by classes. In addition, we also generated t-SNE plots for the RCB dataset alone to assess if compound class separation was also apparent using a smaller dataset. As shown in Figure S8, the DprE1 inhibitors are still distinct from the other compounds in this dataset.

Figure 4.

t-SNE plot data at different thresholds. (A) Full comparison of the RCB Mtb data and previously published Mtb dataset⁶⁶. For clarity, an activity filter was also added for different thresholds (B,10μM; C,1μM; D,100nM) to highlight the difference in the active compounds visualized by the t-SNE plot.

Mtb Target Property Space

We have previously described the curation of molecules with both Mtb data and known targets as TB Mobile^{106, 107}. We have now used this dataset along with the t-SNE approach to visualize the clustering of these molecules and predict potential targets for molecules in the RCB Mtb dataset as well as visualize molecules from a recent review⁶⁷ (Figure 5).

Figure 5.

t-SNE plot of TB Mobile dataset^{106, 107} of Mtb inhibitors with annotated targets showing A. clustering of molecules with known Mtb targets B. overlap of RCB Mtb dataset C. overlap of over 100 recently reviewed Mtb inhibitors⁶⁷.

DISCUSSION

Whole-cell HTS approaches have been widely used in both academia and industry to identify compounds with antitubercular activity, albeit with very low hit rates^108–110. Machine learning has been used in many areas of biomedical and environmental research to assist in learning from patterns in the data^111–116. We have advocated for the use of machine learning approaches which learn from earlier Mtb data from these screens as well as smaller scale studies in order to improve the efficiency of drug discovery which builds on our earlier work^{50–52, 54–59} that demonstrated an enhancement in hit rates. Having previously described the curation of a large (18,886 molecule), chemically diverse Mtb dataset using the primary literature, ChEMBL, and in vivo data⁶⁶ followed by building and validation of machine learning models using extended connectivity fingerprints ECFP6 with Assay Central, we now provide further validation of these models using the RCB Mtb dataset of > 1000 molecules synthesized and tested by a single group.

The RCB Mtb dataset has a range of activities, with MICs that range from 0.59 nM – 0.65 mM as well as over 600 compounds showing no inhibition in the range tested. We have built a variety of classification models using these data, with 9.4, 15.9 and 25.9% actives for 100 nM, 1 μM and 10 μM thresholds, respectively. Analysis of molecular properties showed a significant difference between the highly active (≤ 100nM) and the low / inactive compounds in this library for the majority of descriptors generated (logD (pH=7.4), logP, MW, aromatic ring count, hydrogen-bond acceptor or donor count, ring count, rotatable bond count and polar surface area). One difference that was anticipated is that the logP (median active: 4.17; inactive: 3.06) and logD (active: 3.94; inactive: 2.91) have a higher median in the active compounds, which suggests that, on average, hydrophobicity is important for activity which is a trend that has previously been noted^{66, 104, 105}. An interesting property difference was also in the number of hydrogen bond donors, as over 90% of the active compounds had zero, suggesting this may also be important for active compounds.

As we had previously built a series of classification models using a ~19,000 compound dataset we used these Bayesian models based on our legacy AC software to score the RCB Mtb data as an external test set. Overall, these models predicted the test set well (Figure 4), with high AUC (0.84 – 0.95) and recall (0.70 – 0.97) but with low precision (0.30 – 0.50). We now describe an expanded series of Mtb inhibition models, built with this previously compiled dataset⁶⁶, including classification models with additional algorithms as well as a regression models and compared the predictive ability of these new models. We have extensively compared classification algorithms using our previously established methodology^{66, 100} using the RCB Mtb dataset as an external test set and found that there was a trend of the consensus and rf outperforming overall (based on rank normalized score and ΔRNS), but, possibly due to the small dataset (100 nM, 1 μM and 10 μM models), this was not statistically significant.

The GCN model is also a top-performing model, suggesting it is a promising algorithm to investigate for future QSAR model building. The ConvLSTM-net model outperformed DL and was comparable to the other classic models tested. One benefit of an end-to-end model such as ConvLSTM-net is the speed of preprocessing and predictions. As technologies such as DNA encoded libraries allow billions of molecules to be virtually screened, the computational cost of feature-generation and storage becomes a bottleneck in the screening process. ConvLSTM-net is comparable in performance to most of the classic models, while allowing datasets to be stored and entered into the model purely as SMILES. The slightly lower predictive capability compared to a top-performing model may be an acceptable tradeoff when considering the need to generate and store 1024 or more fingerprint bits for a billion molecules.

The Mtb regression model covered a large range of activities (MIC) and was of comparable size to the classification models. We validated this model by performing a leave-out test set approach as well as scoring two external test sets: one independently curated previously⁶⁶ and the RCB dataset. A series of t-SNE plots were used to visualize the separation of these datasets, suggesting that these are distinct based on ECFP6 molecular fingerprints. These model validations suggest that this continuous model is able to perform well on novel compounds previously not seen by the models. We have also implemented the ability to identify the top enriched ECFP6 fingerprints in a dataset. As an example, we can observe the top 6 fingerprints at the 100nM in our large dataset as well as examples of molecules containing these fingerprints (Figure S9). Further we can assess the atomic contributions for the top molecules in the RCB dataset compared with those that are inactive (Figure 6). This illustrates the benzothiazinone molecules as the most active (DprE1 inhibitors).

Figure 6.

Atomic contributions for the top molecules in the RCB Mtb dataset A. Actives, dominated by benzothiazinones B. Inactives, of different compound classes. Green = contributes to predicting “active”, Red = contributes to predicting “inactive”, no color/close to nothing/white: no contribution to predictions.

Our t-SNE visualization of the distribution of Mtb data with known targets^{106, 107} illustrates how closely related molecules cluster in space and sometimes the same target may have multiple clusters (Figure 5A). This may have utility for predicting targets for molecules from phenotypic screens or as we have illustrated for the RCB Mtb dataset (Figure 5B). A recent review summarized over 100 molecules for which very few had targets identified⁶⁷ and these molecules seem to be within the space covered by this annotated target dataset which may also allow us to propose potential targets (Figure 5C).

In conclusion, we previously proposed further prospective testing of our machine learning models⁶⁶ and have now generated and tested numerous models for Mtb as suggested. There is continually new in vitro data that is generated by research groups and much of it is not published and therefore not accessible for machine learning modeling. We now describe how data generated in the course of a long-term project can be utilized to help in the design of new molecules in the future. We have also described how the initial three Mtb models can be used either together or in consensus as described for other models⁵⁹. We can also utilize our t-SNE visualizations to predict potential targets for molecules derived from phenotypic testing by comparison to annotated datasets. With the RCB Mtb dataset we have illustrated that the DprE1 inhibitors are clustered in a specific region of chemical property space that appears distinct from the location of other Mtb targets. Clearly there are many other regions of this Mtb property / target space that need to be explored as we have previously suggested⁶⁷ and these machine learning models may enable this in conjunction with methods such as generative approaches to design new molecules^117–119.

ABBREVIATIONS USED

ABDT

AdaBoost

ADME

absorption, distribution, metabolism, and excretion

ANN

artificial neural networks

AUC

area under the curve

BNB

Bernoulli Naive Bayes

ConvLSTM-net

Convolutional Long-Short Term Memory Neural Network

DT

Decision Tree

DNN

Deep Neural Networks

ECFP6

extended connectivity fingerprints of maximum diameter 6

GCN

Graph Convolutional Network

HTS

high throughput screening

kNN

k-Nearest Neighbors

QSAR

quantitative structure activity relationships

MIC

minimum inhibitory concentration

Mtb

Mycobacterium tuberculosis

NIAID

NIAID, National Institute of Allergy and Infectious Diseases

QSAR

quantitative structure activity relationships

RF

Random forest

ROC

receiver operating characteristic

RP

Recursive partitioning

SI

selectivity index

SVM

support vector machines

TB

Tuberculosis

XV ROC AUC

cross-validated receiver operator characteristic curve’s area under the curve