Comparison of Deep Learning With Multiple Machine Learning Methods and Metrics Using Diverse Drug Discovery Datasets

Abstract

Machine learning methods have been applied to many datasets in pharmaceutical research for several decades. The relative ease and availability of fingerprint type molecular descriptors paired with Bayesian methods resulted in the widespread use of this approach for a diverse array of endpoints relevant to drug discovery. Deep learning is the latest machine learning algorithm attracting attention for many of pharmaceutical applications from docking to virtual screening. Deep learning is based on an artificial neural network with multiple hidden layers and has found considerable traction for many artificial intelligence applications. We have previously suggested the need for a comparison of different machine learning methods with deep learning across an array of varying datasets that is applicable to pharmaceutical research. Endpoints relevant to pharmaceutical research include absorption, distribution, metabolism, excretion and toxicity (ADME/Tox) properties, as well as activity against pathogens and drug discovery datasets. In this study, we have used datasets for solubility, probe-likeness, hERG, KCNQ1, bubonic plague, Chagas, tuberculosis and malaria to compare different machine learning methods using FCFP6 fingerprints. These datasets represent whole cell screens, individual proteins, physicochemical properties as well as a dataset with a complex endpoint. Our aim was to assess whether deep learning offered any improvement in testing when assessed using an array of metrics including AUC, F1 score, Cohen’s kappa, Matthews correlation coefficient and others. Based on ranked normalized scores for the metrics or datasets Deep Neural Networks (DNN) ranked higher than SVM, which in turn was ranked higher than all the other machine learning methods. Visualizing these properties for training and test sets using radar type plots indicates when models are inferior or perhaps over trained. These results also suggest the need for assessing deep learning further using multiple metrics with much larger scale comparisons, prospective testing as well as assessment of different fingerprints and DNN architectures beyond those used.

Graphical abstract

INTRODUCTION

Drug discovery is now at the point where the increasing amount of public data in PubChem^{1, 2}, ChEMBL³, and a growing list of other databases created from high throughput screens and high throughput biology in general, (including whole cell phenotypic screens, enzymes, receptors etc.) has placed it squarely in the realm of ‘Big data’⁴. We are faced with a significant challenge. No longer can we be limited to a small number of molecules and their properties, we now have thousands of molecules and scores of properties to consider. How do we mine this data, use it and hope to learn from it such that we can make drug discovery more efficient and successful?

One approach which we propose is to use machine learning which can deal with this big data used in cheminformatics with methods such as support vector machines (SVM)^5–11, k-Nearest Neighbors (kNN)¹², Naïve Bayesian^13–17, Decision Trees¹⁸ and others¹⁹ which have been increasingly used^{4, 20–22}. These methods can be used for binary classification, multiple classes classification or values prediction.

In recent years, deep artificial neural networks (including convolutional and recurrent networks) have won numerous contests in pattern recognition and machine learning^23–25. Deep learning solves the central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. N-layer neural networks are shown in Figure 1. It is worth noting that a single-layer neural network describes a network with no hidden layers where the input is directly mapped to the output layer. In that sense, the logistic regression or SVMs are simply a special case of single-layer Neural Networks. In our work for simplicity of Deep Neural Networks (DNN) representation, we will be counting hidden layers only. Quite often 1–2 hidden layers NN are called shallow neural networks and 3 or more hidden layers NN, are called deep neural networks. A recent review addressed the development and application of deep learning in pharmaceutical research²⁶, a method that has proven very successful in learning images and languages elsewhere²⁷. Previously deep learning has been used mostly for unsupervised learning and noisy data^28–34. Limited efforts to use deep learning for pharmaceutical applications suggest a need for further exploration to access its utility for cheminformatics compared with other methods³⁵. Deep Learning has been relatively widely used for bioinformatics³⁶ and computational biology³⁷. Deep Learning has also been used to predict properties such as aqueous solubility using four published datasets and was shown to compare favorably to other machine learning methods using 10-fold cross validation³³. Merck have performed a comparison of deep neural networks to date and have in turn compared them to random forests for use with large quantitative structure activity relationships (QSAR) datasets. They showed they out-performed random forests for 11 out of 15 datasets and 13 of 15 datasets in a second evaluation using time-split test sets³⁸. Merck did not however look at other machine learning methods. One of the largest examples of validation of Deep learning models alongside other machine learning approaches is in the case of the Tox21 Challenge. Deep learning with multitask learning³⁹ slightly outperformed the closest consensus ANN method⁴⁰ across nuclear receptor and stress response datasets. Most recently, one group has suggested some datasets for molecular machine learning and used these for comparison with selected machine learning methods⁴¹. A second group has assessed several machine learning methods with 7 ChEMBL datasets but only focused on a single metric to assess performance⁴². Very frequently deep learning is applied to a single dataset in isolation and not compared to many of the available alternative methods. There are likely many more datasets that could benefit from deep learning even though they may be smaller⁴³.

Figure 1.

A 4-layer neural network with four inputs, two hidden layers of 4 neurons each and one output layer. Notice that connections are between neurons across layers, but not within a layer.

These machine learning methods are increasingly used for virtual screening of compounds which can enable more efficient utilization of high throughput screening (HTS) resources, by enriching the set of compounds screened with compounds that are active^44–47. In addition, such machine learning approaches can also be used for absorption, distribution, metabolism, excretion and toxicity (ADME/Tox) properties as these factors can impact the success of drug discovery processes and their early assessment can later prevent failure^48–54. Big pharma companies have access to these computational technologies but it is generally expensive to produce the data and models in house using commercial software as well as run the models and screening on project teams. Ideally, what if we could augment these resources such that a single program manager or chemist could use the models to prioritize the compounds they made and tested in a combined workflow to find “hits” that can then be reconfirmed and optimized⁵⁵. Past collaborations^{56, 57} suggested these computational approaches could greatly impact drug discovery efficiency.

Over the last decade we and others^{15–17, 58–61} have increasingly focused on Bayesian approaches because of their ease of use and general applicability^{56, 62–78} using molecular function class fingerprints^{79, 80} of maximum diameter 6 and several other simple descriptors^{81, 82}. Much of this work was centered on models for Mycobacterium tuberculosis^83–85 taking account of cytotoxicity and prospectively evaluating them to show high hit rates compared to random screening^85–87. We have since followed this with datasets for Chagas disease⁸⁸ and Ebola⁸⁹ to repurpose approved drugs as well as model ADME properties such as aqueous solubility, mouse liver microsomal stability⁹⁰, Caco-2 cell permeability⁶², toxicology datasets⁹¹ and transporters^{66, 92–97}. By making the fingerprints⁹⁸, and Bayesian model building algorithm open source^{21, 62} there is the potential to further expand on this work.

The main aim of this study was to evaluate whether deep learning offered any improvement in testing when assessed using an array of metrics over other computational methods for drug discovery and ADME/Tox datasets. In the process, we have developed an approach to making Deep learning models more accessible.

EXPERIMENTAL SECTION

Computing

All computing was done on a single dual-processor, quad-core (Intel E5640) server running CentOS 7 with 96GB memory and two Tesla K20c GPU. The following software modules were installed: nltk 3.2.2, scikit-learn 0.18.1, Python 3.5.2, Anaconda 4.2.0 (64-bit), Keras 1.2.1, Tensorflow 0.12.1, Jupyter Notebook 4.3.1.

Datasets and Descriptors

Diverse drug discovery datasets that are publicly available for different type of activity prediction were used to develop prediction pipelines (Table 1). The same datasets have been used in Clark et al.,⁶² for exploring applicability of an array of Bayesian models for ADME/Tox and other physicochemical properties prediction. In the current study the FCFP6 fingerprints, 1024 bins datasets were computed from SDF files using RDKit (http://www.rdkit.org/). A typical frequency of fingerprints occurrence in the 1024 bins compounds representation in a dataset shown in Figure 2.

Table 1.

Binary classification datasets used for evaluating multiple computational methods for activity prediction. Note, that some datasets have heavily imbalanced (This table was adapted from Clark et al, Table #2). Note the active / inactive ratios for hERG and KCNQ1 are reversed as we are trying to obtain compounds that are more desirable (active = non-inhibitors). Models for solubility and hERG are important for drug discovery because they represent an important physicochemical property and a toxicological target, respectively. The probe-like model represents small drug-like molecules that have been scored for probe-likeness by a medicinal chemist. KCNQ1 represents a very large dataset from a screen which could be relevant to a large target-based high throughput screening campaign. Currently this dataset is on the upper end of the size scale in the public domain for drug discovery. The remaining 4 datasets (bubonic plague, Chagas disease, Tuberculosis and malaria) represent whole cell phenotypic screens of various sizes and with differing ratios of active to inactive compounds.

Model	Datasets used and references	Cutoff for active	Number of molecules and ratio
solubility	119	Log solubility = −5	1144 active, 155 inactive, ratio 7.38
probe-like	120	described in120	253 active, 69 inactive, ratio 3.67
hERG	121	described in ref121	373 active, 433 inactive, ratio 0.86
KCNQ1	PubChem BioAssay: AID 2642122	using actives assigned in PubChem	301,737 active, 3878 inactive, ratio 77.81
Bubonic plague (Yersina pestis)	PubChem single-point screen BioAssay: AID 898	active when inhibition ≥50%	223 active, 139, 710 inactive, ratio 0.0016
Chagas disease (Typanosoma cruzi)	Pubchem BioAssay: AID 2044	with EC50 <1 μM, >10-fold difference in cytotoxicity as active as described in88	1692 active, 2363 inactive, ratio 0.72
TB (Mycobacterium tuberculosis)	in vitro bioactivity and cytotoxicity data from MLSMR, CB2, kinase, and ARRA datasets22	Mtb activity and acceptable Vero cell cytotoxicity selectivity index = (MIC or IC90)/CC50 ≥10	1434 active, 5789 inactive, ratio 0.25
malaria (Plasmodium falciparum)	CDD Public datasets (MMV, St. Jude, Novartis, and TCAMS)123–125	3D7 EC50 <10 nM	175 active, 19,604 inactive, ratio 0.0089

Figure 2.

Typical frequency of fingerprints occurrence in the 1024 bins compounds representation in a dataset.

Machine learning

Two general prediction pipelines were developed. The first pipeline solely built using only classic Machine Learning (CML) methods, such as Bernoulli Naive Bayes, Linear Logistic Regression, AdaBoost Decision Tree, Random Forest, and Support Vector Machine. Open source Scikit-learn (http://scikit-learn.org/stable/, CPU for training and prediction) ML python library was used for building, tuning, and validating all CML models included in this pipeline. The second pipeline was built using Deep Neural Networks (DNN) learning models of different complexity using Keras (https://keras.io/), a deep learning library, and Tensorflow (www.tensorflow.org, GPU training and CPU for prediction) as a backend. The developed pipeline consists of a random splitting of the input dataset into training (80%) and test (20%) datasets, while maintaining equal proportions of active to inactive class ratios in each split (stratified splitting). Thus, all the models’ tuning and hyper parameters search were conducted solely through 4-fold cross validation on training data for better model generalization. An example Jupyter notebook is provided (Supplemental information 1).

Bernoulli Naive Bayes

The Naive Bayes method is a supervised learning algorithm based on applying Bayes’ theorem with the “naive” assumption of independence between every pair of features. Bernoulli Naive Bayes implements the naive Bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions; i.e., there may be multiple features but each one is assumed to be a binary-valued (Bernoulli, boolean) variable.

Naive Bayes learners and classifiers can be extremely fast compared to more sophisticated methods and have been widely used¹⁰². The decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one-dimensional distribution. On the other hand, although naive Bayes is known as a decent classifier, it is known to be not a very good estimator, so the class probability outputs are not very accurate.

Our Bernoulli Naive Bayes (BNB) models were tuned and trained using BernoulliNB() method from the Naïve Bayes module of Scikit-learn. The 4-fold stratified cross-validation with a non-parametric approach based on isotonic regression for balancing classes (most of datasets are heavily imbalanced) have been used. The cross-validation generator estimates the model parameter on the train portions of cross-validation split for each split, and the calibration is done on the test cross-validation split of the train dataset, then the probabilities predicted for the folds are then averaged. AUC, F1-score and other metrics listed in data analysis section were computed using those probabilities.

Linear Logistic Regression with regularization

Logistic regression measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function, which is the cumulative logistic distribution, thus predicting the probability of particular outcomes. The L2 binominal regularized logistic regression method was used to classify the activities. A stochastic average gradient optimizer was used in the LogisticRegressionCV() method from the Linear Module of Scikit-learn. A 4-fold stratified cross-validation method was used in grid search of the best regularization parameter (L2 penalties were in logarithmic scale between 1e⁻⁵ and 1e⁻¹). The AUC of ROC was used for scoring the classification (maximizing AUC) performance for each fold of balanced classes’ classification task.

AdaBoost Decision Tree

AdaBoost is a type of “Ensemble Learning” where multiple learners are employed to build a stronger learning algorithm by conjugating many week classifiers. The Decision Tree (DT) was chosen as a base algorithm in our implementation of the AdaBoost method (ABDT). AdaBoostClassifier() method with 100 estimators and 0.9 learning rate from Scikit-learn ensemble methods was used. Similarly to Naïve Bayes, the ABDT models were tuned using isotonic calibration for the imbalanced classes with 4-fold stratified cross-validation method.

Random Forest

Random forest (RF) method is another ensemble method, which fits a number of decision tree classifiers on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting. The RandomForestClassifier() method with maximum depth of tree 5 and balanced classes weights were was used to build the model. The 4-fold stratified cross-validation grid search was done using 5, 10, 25, and 50 estimators with the AUC of ROC as a scoring function of the estimator.

Support Vector Machine

Support Vector Machine (SVM) is one of the most popular supervised machine learning algorithms mostly used in classification problems and it is quite effective in high dimensional spaces⁷. The learning of the hyperplane in the SVM algorithm can be done using different kernel functions for the decision function. C SVM classification with libsvm implementation method from Scikit-learn was used (svm.SVC()). The 4-fold stratified cross-validation grid search using weighted classes was done for 2 kernels (linear, rbf), C (1, 10, 100), and gamma values (1e−2, 1e−3, 1e−4). The parameter C, common to all SVM kernels, trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly. Gamma defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected. Our implementation of SVM automatically finds the best parameters and save the best SVM model for activity predictions.

Deep Neural Networks

Two basic approaches to avoid DNN model overfitting are used in training including L2 norm and drop out regularization for all hidden layers. The following hyperparameters optimization was performed using a 3 hidden layers DNN (Keras with Tensorflow backend) and the grid search method from Scikit-learn. The following parameters were optimized prior to final model training: optimization algorithm (SGD, Adam, Nadam), learning rate (0.05, 0.025, 0.01, 0.001), network weight initialization (uniform, lecun_uniform, normal, glorot_normal, he_normal, he_normal), hidden layers activation function (relu, tanh, LeakyReLU, SReLU), output function (softmax, softplus, sigmoid), L2 regularization (0.05, 0.01, 0.005, 0.001, 0.0001), dropout regularization (0.2, 0.3, 0.5, 0.8) and number of nodes all hidden layers (512, 1024, 2048, 4096).

The following hyperparameters were used for further DNN training: SGD, learning rate 0.01 (automatically 10% reduced on plateu of 50 epochs), weight initialization he_normal, hidden layers activation SReLU, output layer function sigmoid, L2 regularization 0.001, dropout is 0.5. The binary crossentropy was used as a loss function. In order to save training time, an early training termination were implemented by stopping training if no change in loss were observed after 200 epochs. The number of hidden nodes in all hidden layers were set equal to number of input features (number of bins in fingerprints). The DNN model performance was evaluated on up to 8 hidden layers DNNs.

Data analysis

In this study, several traditional measurements of model performance were used, including recall, precision, F1-Score, accuracy, ROC curve and the area under it (AUC), Cohen’s Kappa^{103, 104}, and the Matthews correlation¹⁰⁵. For the metric definitions, we will use the following abbreviations: the number of true positives (TP), the number of false positives (FP), the number of true negatives (TN), and the number of false negatives (FN). Model Recall (also known as the True Positive Rate or Sensitivity) can be thought of percentage of true class labels correctly identified by the model as true and is defined: $\mathit{Recall}= \frac{TP}{TP+FN }$. Similarly, model precision (also known as the Positive Predictive Value) is the probability a predicted true label is indeed true and is defined: $\mathit{Precision}= \frac{TP}{TP+FP }$. The F1-Score is simply the harmonic mean of the Recall and Precision: $F_1\mathit{Score}=2\frac{\mathit{Precision} ·\mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}$. Accuracy is another measure of all-around model robustness, and is the percentage of correctly identified labels out of the entire population. $\mathit{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$. The ROC curve can be computed by plotting the Recall vs the False Positive Rate (FPR) at various decision thresholds T, where $\mathit{FPR}= \frac{FP}{FP+TP }$. In this study, all constructed models are capable of assigning a probability estimate of a sample belonging to the true class. Thus, we can construct an ROC curve by measuring the Recall and FPR performance when we considered a sample with a probability estimate > T as being True for various intervals between 0 and 1. The AUC can be constructed from this plot and can be thought of as the ability of the model to separate classes, where 1 denotes perfect separation and 0.5 is random classification. Another measure of overall model classification performance is the Matthew’s Correlation Coefficient (MCC), which is not subject to heavily imbalanced classes and is defined as $\mathit{MCC}=\frac{TP·TN-FP·FN}{\surd \left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}$. As a measure of correlation, its value can be between −1 and 1. Cohen’s Kappa (CK), another metric estimating overall model performance, attempts to leverage the Accuracy by normalizing it to the probability that the classification would agree by chance (p_e) and is calculated by $CK=\frac{\mathit{Accuracy}-p_e}{1-p_e }$, where $p_e=p_{\mathit{True}}+p_{\mathit{False}}$, $p_{\mathit{True}}=\frac{TP+FN}{TP+TN+FP+FN} ·\frac{TP+FP}{TP+TN+FP+FN}$, $p_{\mathit{False}}=\frac{TN+FN}{TP+TN+FP+FN} ·\frac{TN+FP}{TP+TN+FP+FN}$.

Experimental solubility determination

For one of our drug discovery projects the measurement of the aqueous solubility of 3 test compounds in pH 7.4 PBS was performed by Bioduro (San Diego, CA) utilizing the published modified shake flask method^{106, 107}. These molecules were used as an external test set for the solubility models developed in this study.

RESULTS

The AUC, F1-score, accuracy, Kappa, Matthews correlation, precision and recall values of the all trained models for compounds represented as FCFP6 fingerprints in 1024 bins have been summarized (Table 2 and Supplemental Tables 1–16) while individual model files are also provided (Supplemental Information 2). For clarity, we have grouped all the metrics by each dataset training and test set and presented them as radar plots. A perfect score on all metrics would be represented by a circle the size of the complete plot. Based on the ROC score alone, the Naïve Bayes models are comparable to those described previously by Clark et al.⁶² (Table 2). In this case however, we are using a different source of descriptors and modeling algorithms FCFP6 vs ECFP6 and the RDKit vs CDK¹⁰⁸. In many cases the test sets for the SVM models were generally better than any other models in the same pipeline.

Table 2.

AUC values for all machine learning models and tested compounds (ECFP6, bin=1024) implemented in our pipelines.

Model	BNB	LLR	ABDT	RF	SVM	DNN-2	DNN-3	DNN-4	DNN-5	Published Naïve Bayesian 5 fold cross validation with ECFP662
solubility train	0.96	0.99	1.00	0.93	0.98	1.00	1.00	1.00	1.00	0.87
solubility test	0.86	0.94	0.93	0.87	0.93	0.93	0.93	0.93	0.93
probe-like train	0.99	0.93	1.00	0.98	1.00	1.00	1.00	1.00	1.00	0.76
probe-like test	0.64	0.66	0.66	0.57	0.66	0.56	0.56	0.56	0.56
hERG train	0.93	0.92	0.99	0.92	0.96	1.00	1.00	1.00	1.00	0.85
hERG test	0.84	0.85	0.84	0.83	0.86	0.84	0.84	0.84	0.84
KCNQ1 train	0.80	0.86	0.81	0.76	0.86	1.00	1.00	1.00	1.00	0.84
KCNQ1 test	0.79	0.83	0.80	0.73	0.83	0.86	0.86	0.85	0.85
Bubonic plague train	0.96	0.95	0.98	0.90	0.99	1.00	1.00	1.00	1.00	0.81
Bubonic plague test	0.68	0.77	0.64	0.71	0.76	0.75	0.75	0.75	0.75
Chagas disease train	0.81	0.85	0.87	0.82	0.93	1.00	1.00	1.00	1.00	0.80
Chagas disease test	0.73	0.76	0.77	0.73	0.79	0.79	0.79	0.79	0.79
Tuberculosis train	0.72	0.74	0.76	0.74	0.80	1.00	1.00	1.00	1.00	0.73
Tuberculosis test	0.67	0.68	0.68	0.68	0.70	0.69	0.68	0.69	0.69
Malaria train	0.99	0.99	1.00	0.98	1.00	1.00	1.00	1.00	1.00	0.98
Malaria test	0.98	0.98	0.97	0.95	0.98	0.97	0.97	0.97	0.97

BNB - Bernoulli Naive Bayes, LLR - Logistic linear regression, ABDT - AdaBoost Decision Trees, RF - Random Forest, SVM - Support Vector Machines, DNN-N - DNN with X hidden layers. For comparison, we added previously published 5-fold cross validation ROC data.

When the radar plots are analyzed it is readily apparent which models are likely over-trained. In this case the model training sets have large scores across all the metrics for the training set and have much lower scores for the test set. The shape of the plots can also be indicative of the quality of the models. The larger the circle for the test set, the better the model. The solubility model (Figure 3) is a good example of a model that is well balanced. The training and test sets are virtually both represented by similar circular plots and it is clear that the BNB method performs worst across most of the metrics. The Probe-like model (Figure 4) has an irregular arrangement of scores for the test set which shows it performs very poorly for the Cohen’s Kappa across all methods. The test set for the hERG model (Figure 5) shows that most methods are comparable across the metrics and ABDT stands out as performing the worst with the test set. Overall, the Cohen’s Kappa is the most sensitive metric for this dataset. The KCNQ1 model (Figure 6) shows that DNN and SVM outperform other methods for training and testing and the Matthews correlation and Cohen’s Kappa scores are dramatically lower than all other metrics. The Bubonic plague model (Figure 7) is a difficult example, DNN easily out performs all methods in training and testing (AUC, Matthews correlation and accuracy metric perform best). The Chagas disease dataset (Figure 8) again shows that the DNN has preferable training and testing performance with Cohen’s kappa as the most sensitive metric. The tuberculosis dataset (Figure 9) is another example where DNN outperforms all methods on training by a large margin and by a small margin on the test set, except for the recall statistics. Precision, F1-score, and Cohen’s Kappa are poor for all methods with the test set. The Malaria dataset (Figure 10) shows the impact of DNN with big improvements in precision, F1-score and Cohen’s Kappa for the training and test set when compared to the other machine learning methods.

Figure 3.

Radar plot for the Solubility training and testing data.

Figure 4.

Radar plot for the Probe-like training and testing data.

Figure 5.

Radar plot for hERG training and testing data.

Figure 6.

Radar plot for KCNQ1 training and testing data.

Figure 7.

Radar plot for Bubonic plague training and testing data.

Figure 8.

Radar plot for Chagas training and testing data.

Figure 9.

Radar plot for tuberculosis training and testing data.

Figure 10.

Radar plot for Malaria training and testing data.

In general, the DNN models performed well for external test set predictions except for the AUC performance of the probe-like dataset. For AUC DNN-3 outperforms BNB on 6 of 8 datasets (Table 2). For F1 score the DNN-3 model outperforms BNB on 6 of 8 datasets (Supplemental Table 1). For Accuracy DNN-3 outperforms BNB on 8 datasets (Supplemental Table 2). For Kappa it outperforms BNB on 7 datasets (Supplemental Table 3). For Matthews correlation DNN-3 outperforms or equals BNB on 6 of 8 datasets (Supplemental Table 4). For precision, DNN-3 outperforms BNB on 6 of 8 datasets (Supplemental Table 5). For recall it outperforms BNB on 5 of 8 datasets (Supplemental Table 6). 4 of 8 datasets had AUC values higher for DNN-3 than SVM while 7 of the 8 datasets showed the same of greater F1 values for DNN-3. For accuracy DNN-3 was equal or better than SVM in all 8 datasets. For Kappa, Matthews correlation and recall 5 datasets were equal or better than SVM. For precision 6 datasets were equal or better than SVM.

In order to further understand which models performed the best we have used ranked normalized scores for each machine learning algorithm ranked by metric (Table 3) and by dataset (Table 4). This approach has been previously used by others¹⁰⁹ to compare multiple machine learning methods and performance criteria. When the models are ranked by both metric or dataset, Deep learning DNN-5 and DNN-4) ranks above SVM and all other methods are below this (Table 3 and 4).

Table 3.

Ranked normalized scores for each machine learning algorithm by metric (average over eight datasets for the test dataset).

MODEL	AUC	F1-score	ACC	Cohen-Kappa	Matthews	Precision	Recall	MEAN	RANK
DNN-5	0.798	0.688	0.874	0.382	0.387	0.683	0.700	0.6448	1
DNN-4	0.799	0.684	0.872	0.383	0.387	0.682	0.705	0.6446	2
DNN-2	0.800	0.685	0.868	0.378	0.382	0.664	0.723	0.6431	3
DNN-3	0.799	0.689	0.871	0.378	0.381	0.670	0.711	0.6428	4
SVM	0.813	0.620	0.832	0.341	0.366	0.599	0.744	0.6164	5
LLR	0.809	0.604	0.789	0.286	0.316	0.584	0.746	0.5907	6
RF	0.760	0.612	0.795	0.277	0.297	0.594	0.700	0.5765	7
ABDT	0.786	0.575	0.763	0.260	0.294	0.570	0.741	0.5698	8
BNB	0.774	0.587	0.760	0.251	0.282	0.574	0.733	0.5659	9

BNB - Bernoulli Naive Bayes, LLR - Logistic linear regression, ABDT - AdaBoost Decision Trees, RF - Random Forest, SVM - Support Vector Machines, DNN-N - DNN with X hidden layers. For comparison, we added previously published 5-fold cross validation ROC data.

Table 4.

Ranked normalized scores of each machine learning algorithm by datasets (averaged over seven metrics for test datasets).

MODEL	Solubility	Probe-like	hERG	KCNQ	Bubonic	Chagas	Tuberculosis	Malaria	MEAN	RANK
DNN-5	0.870	0.582	0.740	0.746	0.408	0.649	0.412	0.752	0.6448	1
DNN-4	0.870	0.582	0.739	0.748	0.428	0.640	0.405	0.747	0.6446	2
DNN-2	0.865	0.564	0.731	0.753	0.401	0.640	0.418	0.773	0.6431	3
DNN-3	0.870	0.566	0.739	0.749	0.414	0.649	0.409	0.746	0.6428	4
SVM	0.831	0.624	0.742	0.721	0.320	0.633	0.456	0.604	0.6164	5
LLR	0.839	0.578	0.750	0.626	0.311	0.601	0.437	0.583	0.5907	6
RF	0.823	0.541	0.729	0.590	0.296	0.566	0.430	0.637	0.5765	7
ABDT	0.857	0.608	0.665	0.592	0.290	0.597	0.440	0.509	0.5698	8
BNB	0.763	0.590	0.747	0.574	0.298	0.564	0.413	0.578	0.5659	9

The solubility of 3 compounds from one of our drug discovery projects was assessed using all the different solubility machine learning models developed in this study. The cut off for a soluble molecule was LogS = −5 (10 μM/L). The experimental solubility for the 3 compounds evaluated ranged from 80.8 μM to 465 μM. In virtually all cases the molecules were correctly predicted as soluble (Table 5).

Table 5.

Observed and predicted solubility for 3 novel external test compounds as part of a drug discovery project (Solubility class probabilities in parenthesis).

Compound	BNB	LLR	ABDT	RF	SVM	DNN-2	DNN-3	DNN-4	DNN-5	Experimental
1	Soluble (0.886)	Soluble (0.799)	Insoluble (0.348)	Soluble (0.622)	Soluble (0.930)	Soluble (0.999)	Soluble (0.999)	Soluble (0.999)	Soluble (0.999)	168 μM at pH 7.4
2	Soluble (0.799)	Soluble (0.709)	Insoluble (0.154)	Soluble (0.540)	Soluble (0.926)	Soluble (0.998)	Soluble (0.998)	Soluble (0.999)	Soluble (0.999)	80.8 μM at pH 7.4
3	Soluble (0.799)	Soluble (0.782)	Soluble (0.590)	Soluble (0.590)	Soluble (0.973)	Soluble (0.996)	Soluble (0.998)	Soluble (0.998)	Soluble (0.998)	465 μM at pH 7.4

DISCUSSION

To date there have been relatively few studies that have made comparisons of deep learning to the wide array of classical machine learning methods or have discussed this methods application in pharmaceutical research^{41, 110, 111} or even used the models for actual predictions for ongoing projects. This study therefore fills a void related to drug discovery applications of these methods. For all the computational modeling approaches which we may want to consider they are dependent upon the model applicability domain¹¹² and are affected by the quality of the underlying data,^{113, 114} which may in turn determine the utility and relevance of any model and prediction¹¹⁵. Comparisons of deep learning with other machine learning algorithms across a range of applications suggests this method frequently improves when using predominantly internal cross-validation as the form of evaluation²⁶. In this study, we used an external test set in all cases for comparison of the different algorithms. In addition, we compared several metrics for assessing performance, which is unlike most published studies in this field which rely on one or a narrow range. Our approach for comparison uses a rank normalized score approach which has been used by others to compare machine learning algorithms and performance metrics in other areas¹⁰⁹.

It is likely that the lack of a commercial machine learning platform that is accessible for users to build, validate and test machine learning models based on their own data, is holding back smaller organizations and academic groups from using these approaches. Making these machine learning models something that can be created and used without the need for an expert in cheminformatics would be a considerable achievement. We are far from there yet but are using the Jupyter Notebook to seamlessly integrate chemical operations, dataset manipulation, and machine learning models (e.g., DL, as well as Bayesian, Trees, etc.) within one framework. Deep learning methods have not been widely assessed using prospective validation. Making such models accessible will allow us to take previously published and novel data, enable building of models, and evaluate them for internal quality, before validating them using predictions on vendor libraries, purchase and testing.

The results of this study show there is still work to be done to improve the deep learning models overall due to the differences across all the metrics for training and test set evaluation. Also it is apparent that some of the metrics are less sensitive than others. For example, AUC is far less sensitive than Cohen’s Kappa. This might therefore represent a more useful test set metric than AUC. However, using AUC alone we can discern differences in the models with external testing (Table 2). At least 7 out of 8 datasets show an improvement from Bayesian to SVM and 5 out of the 8 datasets show an improvement in the Test set ROC from Bayesian to DNN-3. This represents results from a true external test set. In 8 of 8 cases the training ROC increases from Bayesian to DNN3. The rank normalized score approach indicates that DNN outperforms SVM, which in turn then outperforms all other algorithms (Table 3 and 4). These results suggest SVM and DNN should perhaps be further embraced by scientists and more rigorously evaluated in different scenarios for drug discovery.

Regardless of the individual models’ shortcomings, we investigated the ‘real world’ applicability of one of the models, namely solubility, to a recent drug discovery project. Solubility is a physicochemical property which is usually determined for many compounds in the process of a drug discovery project and any efforts that could help accurately predict this property might ultimately help in the drug design process. Here, we use classification models and regression models for solubility to identify the predicted solubility for three in-house compounds. 8 of the 9 models (including all DNN models) correctly identified all three compounds as being soluble (Solubility > 10 μM/L). The only misclassifications came from the AdaBoost Decision Tree model, labeling the two less soluble compounds as insoluble. Notably, the DNN models all labeled the compounds as having > 99% probability of being soluble. We then evaluated regression performance on predicting solubility using a linear classifier (Elastic Net) and the DNN models. All models achieved accuracy within approximately 1 log unit of solubility, with a tendency to under-predict solubility (Supplemental Table 7). This is admittedly a very small test set but represents compounds which are advanced leads and therefore of considerable interest. Further prospective assessment of these solubility machine learning models requires much larger numbers of molecules, but this represents a starting point.

While we did not evaluate different descriptors in this study, FCFP6 does quite well with the datasets in this study. However future studies may evaluate additional descriptors such as other non-fingerprint descriptors with deep learning. A recent paper described molecular graph convolutions which represents a simpler encoding of molecules as undirected graphs of atoms for machine learning applications¹¹⁶. The development of additional descriptors and their assessment with different machine learning methods would go some way towards finding the best combination of descriptors and machine learning algorithm. We did not observe any obvious effect of dataset size or balance and this could be due to the limited number of datasets assessed.

Increasingly there are efforts to develop open platforms for connecting scientists and sharing data for many types of projects relevant to drug discovery, these will often include various prediction algorithms e.g. qsardb.org and ochem.eu^{99, 100} or private sharing and development tools such as Chembench¹⁰¹ and CDD Vault⁶². It should be noted that up until recently⁴³ the open source deep learning toolkits for cheminformatics have been relatively inaccessible to the average scientist to use for either building models or generating predictions. Facilitating making machine learning accessible to non-expert users will help in increasing the potential impact of these methods for drug discovery. Ideally these and other machine learning models need to be simple to use and transparent.

As our previous prospective testing of Bayesian models for various drug discovery projects (built with commercial software) has led to several promising leads and candidates for in vivo animal testing^{85–87, 117, 118}, a meaningful improvement in model statistics could lead to improved hit rates in prospective testing. We also propose the need for actually applying the models to real world drug discovery tasks like assessing solubility (Table 5). A key finding of this study is that a wider array of model metrics is likely essential for model comparison and reliance on AUC may not be ideal. In future, it is likely we will also use these and other metrics to evaluate how Deep Learning performs for prospective virtual screening as well as assess a much larger array of datasets. Such case studies are ongoing and will be of value to understand the scope and limitations of deep learning and other machine learning methods for drug discovery.

ABBREVIATIONS

ADME/Tox

Absorption, Distribution, Metabolism, Excretion/Toxicology

ABDT

AdaBoost

ANN

artificial neural networks

AUC

area under the curve

BNB

Bernoulli Naive Bayes

CML

classic Machine Learning

DT

Decision Tree

DNN

Deep Neural Networks

hERG

human ether a-go-go related gene

HTS

high throughput screening

kNN

k-Nearest Neighbors

QSAR

quantitative structure activity relationships

RF

Random forest

ROC

receiver operating characteristic

SVM

support vector machines