Fusing Dual-Event Datasets for Mycobacterium Tuberculosis Machine Learning Models and their Evaluation

Abstract

The search for new tuberculosis treatments continues as we need to find molecules that can act more quickly, be accommodated in multi-drug regimens, and overcome ever increasing levels of drug resistance. Multiple large scale phenotypic high-throughput screens against Mycobacterium tuberculosis (Mtb) have generated dose response data, enabling the generation of machine learning models. These models also incorporated cytotoxicity data and were recently validated with a large external dataset.

A cheminformatics data-fusion approach followed by Bayesian machine learning, Support Vector Machine or Recursive Partitioning model development (based on publicly available Mtb screening data) was used to compare individual datasets and subsequent combined models. A set of 1924 commercially available molecules with promising antitubercular activity (and lack of relative cytotoxicity to Vero cells) were used to evaluate the predictive nature of the models. We demonstrate that combining three datasets incorporating antitubercular and cytotoxicity data in Vero cells from our previous screens results in external validation receiver operator curve (ROC) of 0.83 (Bayesian or RP Forest). Models that do not have the highest five-fold cross validation ROC scores can outperform other models in a test set dependent manner.

We demonstrate with predictions for a recently published set of Mtb leads from GlaxoSmithKline that no single machine learning model may be enough to identify compounds of interest. Dataset fusion represents a further useful strategy for machine learning construction as illustrated with Mtb. Coverage of chemistry and Mtb target spaces may also be limiting factors for the whole-cell screening data generated to date.

INTRODUCTION

Mycobacterium tuberculosis (Mtb), the causative agent of tuberculosis (TB), infects approximately one third of the world’s population, and 1.7–1.8 million people die from this disease annually ¹. Agents active against Mtb are urgently needed to overcome resistance to the available regimen of drugs, shorten a lengthy treatment (that is at a minimum six months in duration), and address drug-drug interactions that may arise during the treatment of TB/HIV co-infections ^{2, 3}. Efforts to leverage sequencing and partial annotation of the Mtb genome ⁴ and pursue specific small molecule modulators of the function of essential gene products have proven more challenging than expected ^{5, 6} in part due to a suggested disconnect between inhibition of protein function and a no-growth whole-cell phenotype ⁷. Thus, a target-agnostic approach has gained favor in recent years, focusing on whole-cell phenotypic highthroughput screens (HTS) of commercial vendor libraries ^{3, 8–10}. This random approach has afforded the clinical-stage SQ109 ¹¹ and a diarylquinoline hit that was optimized to afford the drug bedaquiline ¹². However, Mtb screening hit rates tend to be in the low single digits, if not below 1% as seen elsewhere in drug discovery ¹³.

One can, however, learn from both the active and inactive samples arising from these screens. Leveraging this prior knowledge to produce computational models is an approach we have taken to improve screening efficiency both in terms of cost and relative hit rates. Machine learning and classification methods have been used in TB drug discovery ¹⁴, and have enabled rapid virtual screening of compound libraries for novel inhibitors ^{15, 16}. Specifically, Novartis examined the application of Bayesian models, relying on conditional probabilities ¹⁷. Our work has built on this early contribution to examine significantly larger screening libraries (individually in excess of 200,000 compounds) utilizing commercially available model construction software with molecular function class fingerprints of maximum diameter 6 (FCFP_6) ¹⁸ to model recent tuberculosis screening datasets ^19–21. Single- (predicting whole-cell antitubercular activity) and dual-event (predicting both efficacy and lack of model mammalian cell line cytotoxicity where: IC₉₀ < 10 µg/ml or 10 µM and a selectivity index (SI) greater than ten where the SI is calculated from SI = CC₅₀/IC₉₀) have been created ⁹. The models were demonstrated to be statistically robust ¹⁷ and validated retrospectively through enrichment studies (in excess of 10-fold as compared to random HTS) ²⁰. Most significantly, the Bayesian models were harnessed to predict novel actives through experimental validation with hit rates up to ~20%. ^{22, 23}. Most recently we examined 1924 molecules with three dual-event dose response and cytotoxicity models (these are called MLSMR (derived from Molecular Libraries Screening Center Network), TAACF-CB2, and TAACF kinase) ²⁴. The molecules were ranked using the Bayesian score (which scales with the probability of activity) from all three different dual-event models. Then a receiver operator curve (ROC) plot was generated and we found the MLSMR dose response and cytotoxicity model appeared to perform the best at identifying the active compounds (11.8 fold enrichment in the top 1%). The TAACF kinase dose response and cytotoxicity model showed a similar enrichment (11.1 fold) while the TAACF-CB2 dose response and cytotoxicity model consistently performed poorly. These results highlighted the influence of model training set on performance, suggesting the utility of using multiple models as it is not known a priori which model may perform the best. We now evaluate the effect of combination of datasets and use of different machine learning algorithms (Support Vector Machines, Recursive Partitioning (RP) Forests, RP Single Trees and Bayesian) and their impact on model predictions (internal and external validation) using data from the same laboratory (to minimize inter-laboratory variability ²⁵) and the literature. The knowledge gained from these studies will aid in the further development of machine-learning methods with tuberculosis drug discovery.

MATERIALS AND METHODS

CDD Database and SRI Datasets

The development of the CDD TB database (Collaborative Drug Discovery Inc. Burlingame, CA) has been previously described ²¹. The Tuberculosis Antimicrobial Acquisition and Coordinating Facility (TAACF) and Molecular Libraries Small Molecule Repository (MLSMR) screening datasets ^8–10 were collected and uploaded in CDD TB from sdf files and mapped to custom protocols ²⁶. All of these Mtb datasets used in model building are available for free public read-only access and mining upon registration in the CDD database ^{20, 26–28}, making them a valuable molecule resource for researchers along with available contextual data on these samples from other non Mtb assays. These datasets used previously for modeling are also publically available in PubChem ²⁹. The TB: ARRA dataset used as a test set is available in the CDD TB database (Collaborative Drug Discovery, Burlingame, CA) ^{24, 26}.

Building and Validating Dual-Event Machine Learning Models with Novel Bioactivity and Cytotoxicity Data

We have previously described the generation and validation of the Laplacian-corrected Bayesian classifier models developed with cytotoxicity data to create dual-event models ^{22, 23} using Discovery Studio 3.5 (San Diego, CA) ^{17, 30–33}. These models were developed based on: a. MLSMR dose response and cytotoxicity; b. TAACF-CB2 dose response and cytotoxicity; and c. TAACF kinase dose response and cytotoxicity, where cytotoxicity was determined in Vero cells for each set. All three models were generated using standard protocols with the following molecular descriptors: molecular function class fingerprints of maximum diameter 6 (FCFP_6) ¹⁸, AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area were calculated from input sdf files. Models were validated using leave-one-out cross-validation in which each sample was left out one at a time, a model was built using the remaining samples, and that model utilized to predict the left-out sample. Each model was internally validated, ROC plots were generated, and the crossvalidated ROC area under the curve (XV ROC AUC) calculated. All Bayesian models generated were additionally evaluated by leaving out 50% of the data and rebuilding the model 100 times using a custom protocol for validation, to generate the ROC AUC, concordance, specificity and selectivity as described previously ^{22, 23}. The three models were used to score a set of 1924 commercial analogs previously in the ARRA dataset ²⁴. In addition we used the ARRA dataset to create a separate dual-event model. The prediction data were evaluated using a receiver operator characteristic (ROC) plot. In the current study, as well as using the datasets individually, we also combined the three previously generated datasets (MLSMR, TAACF-CB2, TAACF-kinase) and compared Bayesian, SVM and RP Forest and single tree models built with the same molecular descriptors in Discovery Studio. For SVM models we calculated interpretable descriptors in Discovery Studio then used Pipeline Pilot to generate the FCFP_6 descriptors followed by integration with R ³⁴. RP Forest and RP Single Tree models used the standard protocol in Discovery Studio. In the case of RP Forest models 10 trees were created with bagging. Bagging is short for “Bootstrap AGgregation”. For each tree, a bootstrap sample of the original data is taken, and this sample is used to grow the tree. A bootstrap sample is a data set of the same size as the original one, but in which the same data record can be included multiple times. RP Single Trees had a minimum of 10 samples per node and a maximum tree depth of 20. In all cases, 5-fold cross validation (leave out 20% of the database) was used to calculate the ROC for the models generated. In the case of the combined datasets, predictions were evaluated using binary classification as well as the continuous probability score calculated where possible (e.g. Bayesian Score) followed by ROC plot calculation.

Testing Machine Learning Models with Additional Previously Published Data and Assessing Chemistry Space

177 Mtb leads were recently disclosed by GlaxoSmithKline (GSK) ³⁵ and represent a promising set of small molecules for further exploration as potential antitubercular drug candidates. The GSK set was scored with all of the combined models generated in this study. As the 177 compounds can be classed as actives, our goal was to ascertain which models were able to predict the most as actives. In addition, we compared the 177 compounds to the four datasets used in this study (including actives and inactives) as to their relative placement in chemistry space. We generated a Principal Component Analysis (PCA) using Discovery Studio with the interpretable descriptors chosen previously (AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area). The mean closest distance to training set was also calculated for the 177 compounds for each of the five models to provide an idea of similarity of the test set to the training set. These data were calculated from the outputs of each of the Bayesian models. For each test set molecule a score for closest distance to training set was calculated using Discovery Studio. We averaged this number across the 177 molecules. The smaller the value, the closer a compound is to the training set. In the past we had used mean-maximal similarity value which provides a value of the opposite magnitude.

Understanding the Mtb Target Space Using Known Inhibitors

745 compounds with known Mtb targets collated from the literature ³⁶ and available in TB Mobile ³⁷ were utilized to generate a PCA plot with the interpretable descriptors selected previously (AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area) for machine learning. This PCA model represents essentially the published target-chemistry space for Mtb. We also compared 1429 Mtb hits (active and non-toxic only, from the SRI screens where: IC₉₀ < 10 µg/ml or 10 µM and a selectivity index (SI) greater than ten where the SI is calculated from SI = CC₅₀/IC₉₀) to show how they covered the target-chemistry space. In addition the 177 GSK Mtb leads published by GSK recently ³⁵ were also compared to this target-chemistry space using PCA. The overlaps in data sets were qualitatively compared.

RESULTS

Effect of Training Set and Approach on Prediction of ARRA data

Following on from a previous study in which a large external set of 1924 molecules (ARRA) was used to evaluate three Bayesian models by assessing the enrichment in finding active compounds, we calculated ROC AUC values using the Bayesian score for ranking compounds (Table 1) ²⁴. The MLSMR dose response and cytotoxicity model had the best value (0.82) followed by the TAACF kinase dose response and cytotoxicity model (0.74) and these data are in line with the enrichments we observed previously ²⁴ (Table 1). In addition, these values were similar if not identical to the ROC AUC values for leave out 50% × 100 cross validation performed previously ^{22, 23}. This comparison of models stimulated us to explore different machine learning models and combining data sets as well as suggested that leave out cross validation provided similar results to using a single external test set. The TAACF-CB2 models performed poorly as described previously ²⁴.

Table 1.

Bayesian models predicting the ARRA dose response and cytotoxicity data. Where: IC₉₀ < 10 µg/ml (TAACF-CB2) or 10 µM and a selectivity index (SI) greater than ten were the SI is calculated from SI = CC₅₀/IC_90. Receiver Operator Curve Statistics were calculated for previously published data ^{22, 23}.

Mtb Models (training set N)	Bayesian (Leave out 50% × 100 ROC)	Predicting ‘ARRA dose response and cytotoxicity’ dataset (N = 1924) ROC	Enrichment observed in top 20 ranked ‘ARRA dose response and cytotoxicity’ dataset molecules (Vero, THP-1 and HepG2 cell data) 24
MLSMR dose response and cytotoxicity (2273)	0.82 22	0.82	10.7 – 11.8 fold
TAACF-CB2 dose response and cytotoxicity (1783)	0.64 23	0.54	Poor – random
TAACF Kinase dose response and cytotoxicity (1248)	0.74 22	0.74	6.7–11.1 fold

Comparing SVM, Trees and Bayesian Dual Event Machine Learning Models

Ligand based screening studies traditionally use one or more machine learning approach to build models and predict new compounds, with individual groups having their own preferred methods. Previously we have reported the use of one such approach applied to Mtb, namely, Bayesian models. To insure that our studies of training set effects are more broadly applicable, we now report the examination of SVM, RP Single Tree and RP forest models to compare with Bayesian models. These types of models (Bayesian, SVM, and RP) are the most commonly used of machine learning methods and offer documented differences in terms of their approach and ability to fit the training set data versus offer predictive capability outside of the training set’s chemical space ³⁸. RP models are easily interpretable, while also providing a high degree of predictive accuracy. Single Tree models can be influenced by small changes in the training data resulting in a large change in the tree, and, hence, poorer resulting predictions. An RP forest model resamples the training data randomly multiple times and then grows a tree from each resampled dataset. When making predictions the sample is sent down each tree until it reaches a leaf node then the leaf node probabilities are averaged together to yield a prediction for the forest. SVMs have been widely described in the literature and at their core is the use of a kernel function which converts a scalar product into a higher dimensional space to attempt a linear separation (summarized previously ³⁹). SVMs are generally used for binary data and ranking.

The new machine learning models were generated with all three original datasets (MLSMR, TAACF-CB2, and TAACF kinase; dose-response and cytotoxicity) as well as the more recent ARRA dataset. The Bayesian model statistics were generated by leaving out 50% of the data and rebuilding the model 100 times using a custom protocol for validation to generate the ROC AUC, concordance, specificity and sensitivity as described previously ^{22, 23}, are shown in Supplemental Table 1. Using the FCFP-6 descriptors, we can identify those substructure descriptors consistent with both activity and lack of cytotoxicity, namely alkyl-2-aryloxyacetate and 2,4-disubstituted 1,3,4-oxadiazole (Figure S1), and features of inactives such as 2,5-disubstituted furan, oxepane, tetrasubstituted pyrazole/pyrazolidine, 5-substituted 1,3,4-oxadiazole 2-amide and 2-substituted thiazole/thiazolidine (Figure S2).

For comparison of all the machine learning models we used a slightly less aggressive cross validation (5 fold, e.g. leave out 20%) as this is readily implemented in the machine learning methods. The models provide almost identical ROC AUC results with the leave out 50% × 100 when performed with the datasets (Tables 1 and 2). The RP Forest method used an out-of-bag ROC (in which 20% of the compounds are left out from model building). All four machine learning methods show comparable ROC AUC values across the four datasets using this method of internal validation. The Bayesian method has the best statistics based on the 5-fold cross validation with ROC values slightly higher across all models.

Table 2.

Individual machine learning model cross validation Receiver Operator Curve Statistics. Where: IC₉₀ < 10µg/ml (CB2-TAACF) or 10µM and a selectivity index (SI) greater than ten were the SI is calculated from SI = CC₅₀/IC_90.

Mtb Models (training set N)	RP Forest (Out of bag ROC)	RP Single Tree (With 5 fold cross validation ROC)	SVM (with 5 fold cross validation ROC)	Bayesian (with 5 fold cross validation ROC)	Bayesian (leave out 50% × 100 ROC)
MLSMR dose response and cytotoxicity (2273)	0.78	0.77	0.80	0.83	0.82
TAACF-CB2 dose response and cytotoxicity (1783)	0.57	0.57	0.58	0.60	0.64
TAACF Kinase dose response and cytotoxicity (1248)	0.73	0.72	0.75	0.76	0.74
ARRA dose response and cytotoxicity (1924)	0.82	0.80	0.83	0.86	0.81

The three original data sets (MLSMR, TAACF-CB2, and TAACF kinase; dose-response and cytotoxicity) were combined to build SVM, RP Forest, RP Single Tree and Bayesian models that were then used to predict the ARRA dataset. The Bayesian model statistics for the combined model were generated by leaving out 50% of the data and rebuilding the model 100 times, using a custom protocol for validation. The ROC AUC, concordance, specificity and sensitivity, described previously ^{22, 23}, are shown in Supplemental Table 1. Using the FCFP-6 descriptors, we can identify those substructure descriptors consistent with both activity and lack of cytotoxicity including 3,5-disubstituted thienopyrimidinone, 1-adamantane and acylthiourea (Figure S3) and features of inactives such as isothiazole/isothiazolidine, benzoisoxazole and pyrazoloquinoline (Figure S4).

The external testing ROC AUC for combined models using the ARRA dataset with Bayesian, RP Forest and RP Single Tree methods ranged from 0.65–0.83 for probability (Trees) or Bayesian scores data (Table 3). The SVM method used did not output a continuous probability in the implementation used and so was excluded from this comparison. While using the predicted classification data for the ARRA dataset for all 4 machine learning methods was more instructive (Table 4). For example the Bayesian method had the worst concordance and specificity but the best sensitivity (92.7%) while the SVM had the best concordance and specificity. The RP Single Tree had the lowest sensitivity (58.5%) (Table 4).

Table 3.

Combined MLSMR, TAACF-CB2 and TAACF Kinase dose response and cytotoxicity dataset models created with RP Forest models (Out of bag testing ROC = 0.71), RP Single Tree (Out of bag testing ROC = 0.74) and Bayesian (5 fold cross validation ROC = 0.75) used to predict the ARRA dose response and cytotoxicity data, reporting Receiver Operator Curve statistics using probability (Trees) or Bayesian scores. Note SVM model did not out put a probability value.

Mtb Models	ROC AUC
RP Forest	0.83
RP Single Tree	0.65
Bayesian	0.83

Table 4.

Combined MLSMR, TAACF-CB2 and TAACF Kinase dose response and cytotoxicity dataset models created with SVM (5 fold cross validation ROC = 0.73), RP Forest models (Out of bag testing ROC = 0.71), RP Single Tree (Out of bag testing ROC = 0.74) and Bayesian (5 fold cross validation ROC = 0.75) used to predict the ARRA dose response and cytotoxicity data, reporting contingency table statistics for classification data.

Mtb Models	Concordance (%)	Specificity (%)	Sensitivity (%)
SVM	76.7	77.1	67.1
RP Forest	63.1	61.9	89.0
RP Single Tree	69.1	69.5	58.5
Bayesian	47.2	45.2	92.7

The Effect of Training Set Selection on Prediction of GSK Data and Assessment of Mtb Chemistry Space

The 177 Mtb leads published by GSK recently ³⁵ were scored with the combined models generated in this study (Supplemental Table 2). As all of the 177 compounds can be classed as actives, our goal was to ascertain which models were able to predict the most as actives. We found the TAACF-CB2 dose response and cytotoxicity models performed best, correctly identifying between 48–67.8% of the compounds (Table 5). The SVM model performed optimally with this test set. It is important to note that out of the 177 GSK compounds only a small number were in the models (MLSMR N = 5, TAACF-CB2 N = 2, TAACF-Kinase N = 3, ARRA N = 4, and combined N = 10).

Table 5.

The number of molecules predicted as active out of 177 GSK ³⁵ lead compounds (%). Mean-closest distance = smaller is more similar to training set. Out of the 177 GSK compounds only a small number were in the models corresponding to MLSMR (N = 5), TAACF=CB2 (N = 2), SRI-Kinase (N = 3), ARRA (N = 4) and combined (N = 10). These were included in the table above for ease of comparison.

Mtb Models (training set N)	Random Forest	SVM	Bayesian	Mean–closest distance of training set to test set
MLSMR dose response and cytotoxicity (2273)	17 (9.6)	12 (6.8)	66 (37.3)	0.50
TAACF-CB2 dose response and cytotoxicity (1783)	97 (54.8)	120 (67.8)	85 (48.0)	0.58
TAACF Kinase dose response and cytotoxicity (1248)	36 (20.3)	1 (0.5)	33 (18.6)	0.62
ARRA dose response and cytotoxicity (1924)	7 (3.9)	0 (0)	17 (9.6)	0.59
Combined MLSMR, TAACF-CB2 and TAACF Kinase dose response and cytotoxicity	34 (19.2)	23 (13)	65 (36.7)	0.46

A comparison was made of the 177 compounds to all four datasets used in this study with a Principal Component Analysis. The GSK leads appear distributed within the chemistry space of the >7000 compounds (Figure 1). Next we calculated the mean closest distance to the model training set for each of the 177 compounds to provide an idea of similarity of the test set to the training set. All datasets have roughly similar values but the test set was closest to the combined dataset based on this measure of similarity, while the TAACF-CB2 dose response and cytotoxicity dataset was third closest to the GSK hits. This may suggest such similarity predictors are not a valid measure of model success alone.

Figure 1.

A. Principal Component Analysis of all Mtb datasets (7728 active and inactive compounds) used in this study and overlap of 177 GSK published leads. 3 principal components explain 73% of the variance. B inset to show some of the GSK leads (yellow) widely dispersed and within the chemistry space of the Mtb datasets used for modeling.

Understanding the Mtb target Space Using Known Ligands

We previously created a collection of molecules with their Mtb target/s from published data ²⁸ collated in the course of a previous study ³⁶. This dataset was made available in the Collaborative Drug Discovery (CDD) database ²⁸ and most recently the TB Mobile app ³⁷. We have recently updated the content such that we have 745 small molecules. Following PCA these compounds can give us an approximation of target chemistry space covered in the literature for known antituberculars (Figure 2). When we overlap the 1429 SRI (active and non-cytotoxic compounds) obtained from the 4 different datasets (based on the previously described methods) they overlap approximately half of the compounds with target data (Figure 2B). The 177 GSK hits overlap partially the same area as the SRI hits, but they cover less space in the plot. The GSK hits were also clustered with the 745 compounds with known Mtb targets as a method to infer their potential targets (Supplemental Table 3). Clustering used the MDL fingerprints and created 100 clusters. Examples of compounds clustering near molecules with known targets in Mtb are shown in Figure S5. These include compounds clustering near known QcrB inhibitors (Figure S5A), PanC inhibitors (Figure S5B), Alr or IlvG (Figure S5C), MmpL3 (Figure S5D), Alr (Figure S5E) and InhA (Figure S5F).

Figure 2.

Clustering and PCA of TB Mobile data. A. Examination of 745 TB Mobile molecules with interpretable descriptors results in a PCA with 3 PCs, which explain 88% variability. Outlier compounds represent macrocycles (bottom right) and long lipid-like molecules (bottom left). B. 1429 SRI hits from four datasets (active and non-toxic only, from the SRI screens where: IC₉₀ < 10 µg/ml or 10 µM and a selectivity index (SI) greater than ten where the SI is calculated from SI = CC₅₀/IC₉₀) and 745 TB Mobile compounds results in a PCA with 3 PCs explaining 83% variability; SRI compounds are clustered (yellow). C. Examination of 177 GSK leads (yellow) and the TB Mobile compounds results in a PCA with 3 PCs, which explain 88 % of variance.

DISCUSSION

There is a resurgence in whole cell HTS for Mtb and this has resulted in low hit rates ^{35, 40–42}. Utilizing past screening data with machine learning methods could improve the efficiency of such screens. Our prior machine learning studies have demonstrated that single and dual-event Bayesian machine learning models based on public data can enrich hit discovery using retrospective and prospective testing ^{22, 23}. While we have focused on Bayesian machine learning due to their processing speed and ease of use, many other algorithms exist that can be used for machine learning. SVM ^43–52 and Random Forests ^53–55 like Bayesian classification methods ^56–60 have also been used extensively for drug discovery and ADME/Tox models ^{31, 57, 61, 62}. For example, extensive evaluations of different machine learning methods and descriptors have been performed by Broccatelli et al. ⁶³ using SVM, Random Forest, Partial Least Squares, Linear Discriminant Analysis, Random Forests (RF) and Genetic Algorithm-kNN models with MOE, MACCS, CDK, Dragon descriptors and 545 literature compounds with the ion channel hERG activity. The best models were RF MOE2D, RF-MACCS and PLSD-VS+ with consensus accuracy 90%, specificity 93% and sensitivity 89%. A set of 7617 compounds with genotoxicity (Ames) data were used to compare five machine learning methods (SVM, kNN, Naïve Bayes, Artificial Neural networks and C4.5 decision trees) each using five fingerprint descriptor methods (PubChem, E-state, MACCS, CDK fingerprints and substructure fingerprints) ⁶⁴. Using a test set of 831 diverse molecules, the accuracy ranged from 90–98% with three combinations of descriptors and algorithms proving equally accurate (PubChem-kNN, MAACS-kNN and PubChem SVM). Although we have analyzed the Mtb literature extensively ^{65, 66} we are not aware of similar exhaustive analyses of machine learning methods used to prospectively predict whole cell Mtb activity. Predominantly the focus has been retrospective or leave out testing ^67,68

Frequently, we have seen multiple Bayesian models perform differently with varying datasets ^19–24 and with the current test set we see a wide range in the ROC values for the ARRA dataset of 1924 molecules, with ROC AUC values of 0.54 – 0.82 (Table 1, not previously reported). Interestingly, combining the datasets only slightly improves the Bayesian model ROC value to 0.83 (Table 1 versus Table 3). However, this model also has the lowest concordance when compared to the other methods at binary classification of the 1924 compounds (Table 4). Using an external dataset of 177 recently published Mtb leads from GSK ³⁵ we found a wide variability between models and datasets in identifying leads from this set (Table 4). It should also be noted that all these molecules can be classed as actives while only a small number of compounds overlapped between the training and test sets. The best models at evaluating this GSK test set, identifying approximately 48–68% of the actives, were the TAACF-CB2 dose response and cytotoxicity RP Forest, SVM, and Bayesian models. These highlight the value of using such models to select compounds for testing without extensive HTS. We had previously used the Bayesian model successfully to screen a larger set of 13,533 GSK compounds found to have antimalarial activity ⁶⁹. We had scored these molecules ⁷⁰, which enabled us to identify several with potent antitubercular activity upon empirical testing ²³. Yet, this present work also suggests using the ROC value for 5-fold validation alone is not likely to be a single reliable measure (or predictor) of the utility of a model as this TAACF-CB2 dose response and cytotoxicity model also had the lowest ROC scores (below 0.6, Table 2). Conversely, we have also shown that the similarity of molecules in the test and training sets is also not a reliable measure of likely correct predictions as the TAACF-CB2 training set was not the closest to the test set of the GSK leads (Table 4). This result may also suggest the need for a deeper analysis of FCFP_6 descriptors between training and test sets, or more simply a further investigation as to which molecular substructures are important for Mtb activity (that are present in the training and test set molecules). Overlap of certain molecular features between datasets may be a better predictor of the ROC value and model performance (Figure S1–4) and this hypothesis remains to be tested. Ultimately in comparing predictions across datasets one also should consider experimental variability in Mtb screening ²⁵, so it is at least reassuring that models from one laboratory can be used to predict data from another to a reasonable degree. Of course we have relied in this study on the ROC metric (Tables 1–3) and contingency table statistics (Table 4) as measures for comparing models. This may not be enough. Future studies could explore whether other measures commonly used for assessment of virtual screening provide more insight into why there are model and dataset dependencies (e.g. concentrated ROC (CROC), Boltzmann-enhanced discrimination of ROC (BEDROC), Guner Henry Score etc.) ^71–74 and whether consensus scoring could overcome these.

This study continues our efforts to build and validate machine learning models for ​Mtb. ^19–24 It extends recent externally validated dual-event models to consider the fusion of datasets as a method to increase coverage of chemistry space and simplify the number of models required. Although, it should be noted that the MLSMR, TAACF-CB2 and TAACF kinase datasets have a fair degree of overlap, and the ARRA dataset overlaps with some of these ²⁴, which may explain why the ROC AUC values for this dataset vary from 0.54 – 0.83 when looking at individual models (Table 1) and there is not a great deal of improvement when datasets are combined. There is also some variation in ROC AUC values across machine learning models when the datasets are combined (Table 3) and across contingency table statistics (Table 4).

Our PCA in this study using molecules with annotated targets (covering over 70 to date with identified inhibitors ³⁷) suggests the hits from SRI and GSK overlap and are only exploring a fraction of the Mtb chemistry target space. So this might indicate that any machine learning models derived from such HTS data are only going to be useful for predictions in a relatively small segment of Mtb chemistry target space. Conversely, this type of analysis may also be useful for predicting potential targets for the training set actives. The opportunity also exists to extend our initial approach based on molecule similarity ³⁷ to one predicated on multiple physicochemical descriptors. The potential targets for some of the 177 GSK compounds are suggested based on clustering with compounds with known annotated Mtb targets which could be useful for further future experimental verification. Similarly one could pursue this approach with the active subset of compounds in the ARRA or other datasets. Our approach in this study using machine learning models to predict compounds with activity could also be combined with inhibitors of known targets and clustering to suggest their potential targets in a single workflow. Such a process may lead to more rapid target identification efforts. Verification of such predictions is however time consuming and costly and whole cell phenotypic screening will also identify compounds that act through more than one mechanism.

In conclusion, the choice of Bayesian models would appear to be acceptable for predicting whole-cell antitubercular efficacy under the current conditions when compared to SVM and RP approaches. Each of the methods has their strengths and weaknesses and it would appear that no one method stands out as best for Mtb active prediction. Others have previously shown SVM and Random Forest approaches to outperform Bayesian models in different areas ⁶⁴. Additional researchers have used ensembles of models rather than rely on a single model ⁷⁵. To date none of these ensemble machine learning approaches had been tested with Mtb datasets. A major advantage of dataset fusion is that a single model can be created that covers the sum chemical space of individual models and may be more likely to be used rather than multiple individual smaller models. This is distinct from the fusion of predictions and consensus scoring with individual machine learning or similarity methods ⁷⁶. Future efforts may explore using other machine learning methods, e.g. k–Nearest Neighbors ⁷⁷, K-Partial Least Squares ⁷⁸, Self Organizing Maps and Kohonen maps ⁷⁹ for Mtb model building with this combined dataset. In addition, efforts to make Mtb models more readily available may also be evaluated using free or open source resources like Bioclipse ^80–82, Chembench ⁸³ and others ^{84, 85}. This would then make the models globally accessible ⁸⁶ and perhaps increase the speed and efficiency of screening efforts in vitro.