Computational Prediction and Validation of an Expert's Evaluation of Chemical Probes

Abstract

In a decade with over half a billion dollars of investment, more than 300 chemical probes have been identified to have biological activity through NIH funded screening efforts. We have collected the evaluations of an experienced medicinal chemist on the likely chemistry quality of these probes based on a number of criteria including literature related to the probe and potential chemical reactivity. Over 20% of these probes were found to be undesirable. Analysis of the molecular properties of these compounds scored as desirable suggested higher pKa, molecular weight, heavy atom count and rotatable bond number. We were particularly interested whether the human evaluation aspect of medicinal chemistry due diligence could be computationally predicted. We used a process of sequential Bayesian model building and iterative testing as we included additional probes. Following external validation of these methods and comparing different machine learning methods we identified Bayesian models with accuracy comparable to other measures of drug-likeness and filtering rules created to date.

INTRODUCTION

In the past decade the National Institutes of Health (NIH) has funded extensive high throughput screening (HTS) efforts in both intra-mural and academic centers to identify small molecule chemical probes or tool compounds via the Molecular Libraries Screening Center Network (MLSCN) and the Molecular Library Probe Production Center Network (MLPCN). By 2009 it was estimated to have cost $385 million¹ and by 2010 $576.6 million² but since then funding was dramatically scaled back³. Various definitions for compounds to become probes based on a combination of potency, selectivity, solubility, and availability, have been used¹. To date the NIH-funded academic screening centers have discovered just over 300 chemical probes (at the time of writing) and some of the groups have demonstrated sophisticated drug discovery capabilities⁴. The NIH has subsequently made publically accessible an extensive compilation of chemical and biology data on these probes. From medicinal and computational chemistry and biology perspectives, we thought this an opportune time to perform a subjective analysis in order to learn from this massive effort. We posit that medicinal chemistry and computational considerations are relevant and complementary to biological activity considerations. Presenting our probe analysis in the context of medicinal chemistry due diligence is timely given the emerging understanding of the relationship of chemistry ligand structure to biology target topology^5–7. Moreover we used our analysis to test whether it was possible to computationally predict the evaluations and eventual decisions of an experienced medicinal chemist.

In the same period that these probes were generated, there have been extensive assessments of medicinal chemists' appraisal of drug- or lead-likeness. For example, Lajiness et al., have evaluated the ability of medicinal chemists to assess the drug- or lead-likeness of molecules⁸. Thirteen medicinal chemists assessed approximately 22,000 compounds, broken into 11 lists of approximately 2000 compounds each. It was found that they were not very consistent in the compounds they rejected as being undesirable⁸. Cheshire described how modern medicinal chemists are often “over-productive” synthesizing many more compounds than are required to achieve the objectives of the project⁹. In contrast Hack et al., used the wisdom of crowds to fill holes in a screening library by leveraging 145 global medicinal chemists at J&J¹⁰. A recent study by Kutchukian et al., investigated medicinal chemists behavior using surveys in which they selected chemical fragments for development into a lead compound from a set of ~4,000 available fragments¹¹. Computational Bayesian Classifiers were also built for each chemist to model their selection strategy. These models were not used to prospectively predict compounds. The results suggested the chemists greatly simplified the problem, using only 1–2 of many possible parameters. Overall there was a lack of consensus in compound selections. Cumming et al., have proposed that the `quality' of small-molecule drug candidates, are under the control of chemists during the identification and optimization of lead compounds¹². The fusion of all of these studies suggests that decision making by the medicinal chemist while variable, is still widely regarded and critical to drug discovery. However, since the development of the Rule of 5¹³ there has been an apparent focus on similar rules to filter undesirable compounds and influence synthetic decisions.

Previously others have suggested that marketed drugs contain a high percentage of such undesirable groups (277 out of 1070 compounds in one study)¹⁴. Filters or rules are widely used by pharmaceutical companies to flag molecules that may be false positives and frequent hitters from HTS screening libraries as well as select compounds from commercial vendors¹⁵. Some examples of widely used substructure filters include REOS from Vertex¹⁶, as well as filters from GlaxoSmithKline¹⁷, BristolMyersSquibb¹⁸ and Abbott^19–21. These pick up a range of chemical substructures such as thiol traps and redox-active compounds, epoxides, anhydrides, and Michael acceptors. Another group has developed a series of over 400 substructural features for removal of Pan Assay INterference compoundS (PAINS) from screening libraries that is likely considered a definitive rule set²². Bruns and Watson have also described a set of 275 rules they developed at Eli Lilly over an 18-year period, that were used to identify compounds that may interfere with biological assays²³. The structural queries were profiled for frequency of occurrence in drug-like and non-drug-like compound sets and were extensively reviewed by a panel of experienced medicinal chemists. Drug-likeness itself has been suggested to not reflect adequately compound quality and rules may lead to undesirable molecular property inflation. Bickerton et al., proposed a measure of drug-likeness based on the concept of desirability called the quantitative estimate of drug-likeness (QED)²⁴. A recently developed computational method termed Bioactivity Data Associative Promiscuity pattern Learning Engine (BadApple) uses public data to predict promiscuity based on comparison of scaffolds^{25, 26}. Finally, a retrospective analysis has shown that recently approved oral drugs are highly optimized for ligand efficiency, a metric that integrates binding affinity with molecular properties²⁷.

The only critical evaluation to date of the NIH probes, was a study in 2009 that took a crowdsourcing approach to evaluation of 64 NIH chemical probes¹. Eleven experts subjectively scored the probes and found 48 of 64 (75%) probes were of medium or high confidence. We proposed we could use this type of expert-derived classification data to learn what had been classed as desirable and then using this prospectively to help score additional molecules computationally. One of the chemists involved in the 2009 crowdsourcing study was enlisted to test the approach of whether an algorithm could learn from his selections. We have taken the approach of using several machine learning methods and have focused on Naïve Bayesian classification which has been used for modeling in vitro and in vivo data²⁸ by us and many others²⁹. In addition we have compared the results of this effort with PAINS²², QED²⁴, BadApple²⁵ and ligand efficiency^{27, 30}.

Experimental

Dataset

With just a few exceptions NIH probe compounds were identified from NIH's Pubchem web based book²⁵ summarizing five years of probe discovery efforts. Probes are identified by ML number and by PubChem CID number. NIH probe compounds were compiled using the NIH PubChem Compound Identifier (CID) as the defining field for associating chemical structure. For chiral compounds, two dimensional depictions were searched in CAS SciFinder™ (CAS, Columbus OH) and associated references were used to define the intended structure.

A medicinal chemist,with more than 40 years of experience (C.A.L.) followed a consistent protocol for determining if compounds should be considered undesirable or desirable. The number of biological literature references associated with each compound was determined. Probes with more than 150 references to biological activity were considered unlikely to be selective despite any PubChem HTS assay data. Alternatively, probes with zero references were considered to have uncertain biological quality if the probe was not of recent vintage. The idea is that if a probe report is at least several years old and if neither the probe originator nor anyone else has published on the probe then one might conclude there was some sort of problem. The idea that the presence of a given probe in a US patent application³¹ across many probes might be a pointer to promiscuous activity was shown on detailed examination to be incorrect. CAS SciFinder™ was used to determine the CAS RegNo. Probes with no CAS RegNo were considered problematic if associated with chemistry generally unexplored in drugs. Finally, probes with predicted chemical reactivity were considered to lead to uncertainty about the cause of the biological effect. Judgment was used for this criterion. Only the most flagrant offenders were flagged and then only with caution often in the face of considerable HTS experimental data. The reactivity criterion is likely the “softest” since this is a criteria where different experts clearly often disagree. Probes that met any of these criteria were considered undesirable and given a score of 0. A summary of the percentage of undesirable compounds that fell within each criterion is shown in Fig. 1. All other probes were given a score of 1 for desirable. These are binary choices with no degree of gradation and thus carry the biases inherent of any binary yes no methodology³² The data and molecular structures have been made publically available in the CDD Public database (Collaborative Drug Discovery Inc. Burlingame, CA)³³.

Figure 1.

Relative contribution of each criteria for considering compounds as undersirable.

Molecular properties and filtering methods

Salts were removed from molecules prior to calculation of molecular properties and computational modeling. One compound that was a complex, ML134, was also removed from analysis. Several descriptors were calculated in the CDD database using the Marvin suite (ChemAxon, Budapest, Hungary) namely: molecular weight, logP, H bond donors, H bond acceptors, Lipinski score, pKa, heavy atom count, polar surface area, rotatable bond number. The JChem suite (ChemAxon) was also used to generate two values for pKa namely first the average charge at pH 7.4, where negative values are acidic, positive basic, and close to zero neutral, and second the distribution of the major microspecies.

Similarly descriptors were calculated including AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area from input SD files using Discovery Studio 3.5 (Biovia, San Diego, CA).

BadApple learns from “frequent hitters” found in the Molecular Libraries Screening program, and flags promiscuous compounds. This method was used to score the compounds in this study at the website^25,26.

PAINS is a set of filters determined by identifying compounds that were frequent hitters in numerous high throughput screens. PAINS filters were determined using the FAFDrugs² program^{34, 35}.

The desirability of the NIH chemical probes was also compared with QED²⁴ which was calculated using open source software from SilicosIt (Schilde, Belgium).

For determination of ligand efficiency, the IC₅₀, AC₅₀, and EC₅₀ values associated with each chemical probe were accessed from PubChem using a java script. The calculations function in CDD Vault software was used to determine ligand efficiency with the formula LE= 1.4*−log(EC₅₀ or IC₅₀ or AC₅₀)/(Heavy atom count).^{30, 36}

Analysis of the 307 compounds was performed with Principal Component Analysis (PCA) using Discovery Studio was generated with the interpretable descriptors chosen previously (AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area). The desirable and undesirable compounds were also treated as unique “libraries” that were also compared through the “compare libraries” protocol in Discovery Studio via the use of assemblies (Murcko Assemblies)³⁷.

Machine learning models

We have previously described the generation and validation of the Laplacian-corrected Bayesian classifier models developed for various datasets using Discovery Studio 3.5²⁸. This approach was utilized with the literature probe data from PubChem. A set of simple molecular descriptors were used: molecular function class fingerprints of maximum diameter 6 (FCFP_6)³⁸, AlogP, molecular weight, number of rotatable bonds, number of rings, number of aromatic rings, number of hydrogen bond acceptors, number of hydrogen bond donors, and molecular fractional polar surface area were calculated from input SD files. Models were validated using leave-one-out cross-validation in which each sample was left out one at a time, a model was built using the remaining samples, and that model utilized to predict the left-out sample. Each of the models was internally validated, receiver operator (ROC) plots were generated, and the cross validated (XV) ROC area under the curve (AUC) calculated. The Bayesian model was additionally evaluated by performing 5 fold cross validation in which 20% of the data set is left out 5 times. Additionally leaving out 50% of the data and rebuilding the model 100 times using a custom protocol was used for validation, to generate the ROC AUC, concordance, specificity and selectivity as described previously^{39, 40}. The Internal ROC value represents the training set value while the external ROC represents the test set molecules left out. For the largest model created we also compared the Bayesian model with SVM and RP Forest and single tree models built with the same molecular descriptors. For SVM models we calculated interpretable descriptors in Discovery Studio then used Pipeline Pilot to generate the FCFP_6 descriptors followed by integration with R⁴¹. RP Forest and RP Single Tree models used the standard protocol in Discovery Studio. In the case of RP Forest models ten trees were created with bagging. RP Single Trees had a minimum of ten samples per node and a maximum tree depth of 20. In all cases, 5-fold cross validation (leave out 20% of the database 5 times) was used to calculate the ROC for the models generated and for comparison.

Model predictions for additional compounds identified after model building

After each model was built additional compounds were identified and these were used as an external test set for prospective analysis. These compounds were then later combined to enable model rebuilding. This approach was repeated 3 times. Finally a further 15 compounds initially not scored were identified and these were used as a test set for all models created. For each molecule, the closest distance to the training set was also calculated (a value of zero represents a molecule in the training set).

Statistical analysis

The mean calculated molecular property values for compounds were compared using two tailed t-test with JMP v. 8.0.1 (SAS Institute, Cary, NC). For external test set prediction evaluation, the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) were determined and used to generate ROC plots and determine accuracy ((TP+TN)/total), specificity (TN/(TN+FP)), sensitivity (TP/(TP+FN), and precision (TP/(TP+FP) using the standard indicated formula. Mean ligand efficiency, was assessed using a student's t-test, and PAINS and BadApple predicted scores were assessed using Fisher's exact test.

RESULTS

Determining NIH chemical probe quality

For a medicinal chemist to accurately assess the quality of probes identified through HTS, it is necessary to assess previous findings in the literature and in patent applications. Unfortunately, this type of due diligence is not easily performed and requires labor intensive mixing of publically available data and the use of commercial software. In the process of this study we have noted a number of areas where optimizing integration of this information would benefit all participants in the drug discovery community and these will be detailed elsewhere. Eventually, it should be possible to integrate and automate this type of due diligence analysis. With just a few exceptions (ML032, ML049 and ML287) NIH probe compounds were identified from NIH's PubChem web based book. ML032, ML049, ML287 are identified in a spreadsheet available on the MLP website⁴². ML049 is associated with a probe report “Fluorescent Cross-Reactive FPR/FPRL1 Hexapeptide Ligand”. Neither ML032 and ML287 is associated with probe reports as listed on the MLP website⁴².

We have noticed that in terms of selectivity, the probes fall into two main categories: 1) the probe is selective in a drug discovery sense, i.e. one or two orders of magnitude more selective for the target versus all salient anti-targets 2) the probe is 3 to 10 times more selective for the target than for any single anti-target, is novel for the target and the breadth of activity against all anti-targets is experimentally very well characterized. Probes in this second category are in the majority.

Molecular properties

The 307 molecules in the complete dataset (excluding the final 15 molecule test set) used for the largest model consisted of 240 scored as desirable and 67 as undesirable. Using 9 descriptors from the CDD Vault software, the molecular weight, rotatable bond number and heavy atom count were all statistically significantly larger in the desirable compound set (Table 1). When we focused on just the acidic or basic compounds we found a statistically significant higher pKa value for acidic compounds only (Table 1). Using 8 partially overlapping descriptors calculated in Discovery Studio the number of molecular weight and rotatable bonds was also greater in desirable compounds while the molecular fractional polar surface area was lower (Table 2). Other descriptors showed no statistically significant difference between groups. Diversity analysis of the desirable and undesirable compounds using FCFP_6 fingerprints, Murcko assemblies, Tanimoto similarity, and default properties indicated few differences apart from undesirable molecules displaying a higher diversity of Fingerprint features (Table S1). Library analysis using Murcko Assemblies showed little similarity (Table S2).

Table 1.

Mean ± SD molecular properties calculated in CDD Vault using ChemAxon software for the 307 molecules used in model building,

	Molecular weight	LogP	H-bond donors	H-bond acceptors	Lipinski score	pKa (6 cpds missing)	Heavy atom count	Polar Surface Area	Rotatable bond number
Undesirable	359.87 ± 83.26	3.43 ± 1.27	1.07 ± 1.13	4.14 ± 1.52	0.18 ± 0.39	5.14 ± 5.13	24.70 ± 5.69	73.03 ± 29.3	4.20 ± 2.25
Desirable	383.53 ± 87.17*	3.43 ± 1.36	1.23 ± 1.01	4.11 ± 1.61	0.21 ± 0.49	6.52 ± 4.94	26.92 ± 6.15*	72.61 ± 28.10	4.92 ± 2.40*

^*statistically significant p <0.05 two sided t-test.

For acidic compounds (n = 142) the mean pKa was 8.12 ± 3.95 for undesirable and 9.71 ± 3.84 for desirable compounds (p <0.05 two sided t-test). For basic compounds (n = 160) the mean pKa was 2.25 ± 4.47 for undesirable and 3.75 ± 4.04 for desirable compounds,

Table 2.

Mean ± SD molecular properties calculated in with Discovery Studio 3.5 software. ± SD for the 307 molecules used in model building,

	Molecular weight	ALogP	Number of Rings	Number Aromatic rings	Number H-bonds donors	Number H-bond acceptors	Molecular fractional polar surface area	Number rotatable bonds
Undesirable	359.87 ± 83.26	3.55 ± 1.31	3.29 ± 1.00	2.35± 1.06	1.07 ± 1.13	4.77 ± 1.75	0.28 ± 0.11	4.30 ± 2.31
Desirable	383.53 ± 87.16*	3.50 ± 1.33	3.47 ± 1.02	2.55 ± 0.98	1.23 ± 1.00	4.36 ± 1.63	0.23 ± 0.08*	5.13 ± 2.41 *

^*statistically significant p <0.05 two sided t-test.

PCA analysis showed the desirable and non-desirable molecules overlapped based on these molecular properties (Fig. 2A) and that these were within the more diverse chemical property space of the Microsource spectrum commercial library of drugs and natural products (Fig. 2B). Library analysis using Murcko Assemblies showed higher similarity to the Microsource library than between the NIH undesirable and undesirable compounds. In contrast the use of a global fingerprint suggested the NIH probes and Microsource libraries were less similar than between the NIH desirable and undesirable compounds (Table S2).

Figure 2. Visualizing chemical property space of desirable and undesirable probes.

A. Principal component analysis (PCA) of the 307 desirable (yellow) and undesirable (blue) NIH probes. 76.6% of the variance was explained with three principal components. B. PCA of the NIH chemical probes (yellow) and 2320 Microsource spectrum compounds (blue). 81.9% of variance was explained by 3 principal components.

The desirable/ and undesirable scores were compared to several other rules and tools used to predict drug-likeness. None of the molecular probes characterized had more than 2 violations of the Rule of Five¹², and we found no statistical difference between the desirable and undesirable molecules¹³. BadApple revealed that the desirable compounds are less likely to be promiscuous, predicting 12% of desirable and 33% of undesirable probes fall into this category (Fisher's exact test, p=0.04). In total, 34 of 322 probes were flagged by the PAINS filters, representing 6.7% of desirable and 25% of undesirable compounds (Fisher's exact test, p>0.0001) (Fig. 3). There was no significant difference in the QED. The mean ligand efficiency of both desirable and undesirable probes was in the “ideal” range defined as greater than 0.3 kcal per mole heavy atom³⁰ (desirable = 0.308 vs. undesirable =0.327, p= 0.000003). All the various scores were used to create a heatmap for ease of comparison (Fig. 4).

Figure 3. Comparison of the expert's evaluation of the NIH chemical probes with PAINS and BadApple Filters.

Desirable NIH chemical probes are less likely to be filtered by PAINS or BadApple as promiscuous than those scored as undesirable. (Fisher's exact test, p>0.0001 for PAINS and p=0.04 for BadApple).

Figure 4. Comparison of desirability scores with Bayesian learning predicted scores for each test set, QED, BadApple, and ligand efficiency metrics.

The colors on the heat map correspond to the value of the indicated metric for each probe, listed vertically. The scale was normalized internally with green corresponding to the optimal condition within each metric.

Bayesian Model 1

Using 57 molecules from the original dataset from 2009. The model had a Receiver Operator Characteristic (ROC) value for leave one out (optimistic) of 0.654, 5 fold cross validation ROC = 0.613 (Table S3) and leave out 50% × 100 ROC of 0.63 (Table 3). The top twenty substructure descriptors consistent with compounds classed as desirable probes contain well known drug like substructures like thiazole, pyridine, and morpholine fragments^{43, 44} to name just a few representative examples (Fig. S1). The top 20 features in undesirable probes include seleno-organic, compounds prone to facile oxidation such as phenol rich aromatic rings, aniline-rich functionality, and thio-ethers^{45, 46} (Fig. S2).

Table 3.

Leave out 50% × 100 cross validation analysis of Bayesian models of an expert's evaluation of the NIH chemical probes.

Dataset size	External ROC Score	Internal ROC Score	Concordance	Specificity	Sensitivity
N57	0.63 ± 0.08	0.65 ± 0.14	54.30 ± 10.03	62.04 ± 29.46	51.40 ± 21.98
N170	0.59 ± 0.05	0.69 ± 0.07	53.54 ± 7.62	59.30 ± 19.65	51.31 ± 17.48
N191	0.60 ± 0.05	0.69 ± 0.07	55.06 ± 8.00	57.83 ± 17.81	54.10 ± 16.98
N307	0.69 ± 0.04	0.75 ±0.05	64.57 ± 8.06	61.89 ± 14.83	65.25 ± 13.02

ROC = receiver operator characteristic.

Bayesian Model 2

Using 170 molecules the model had an ROC value for leave one out of 0.710, 5 fold cross validation ROC = 0.654 (Table S4) and leave out 50% × 100 ROC of 0.59 (Table 1). In this model, the top twenty substructure descriptors consistent with compounds classed as desirable probes contain drug-like heterocycles, biomimetic amino acid analogs, and charged functionality likely to help with compound solubility at physiologically relevant pH levels (Fig. S3), the top 20 features in undesirable probes include highly conjugated systems with potential Michael acceptors, relatively unstable hydrozones, and easily oxidized heterocycles like the furan derivatives (Fig. S4).

Bayesian Model 3

Using 191 molecules the model had an ROC value for leave one out of 0.707, 5 fold cross validation ROC = 0.671 (Table S5) and leave out 50% × 100 ROC of 0.60. The top twenty substructure descriptors consistent with compounds classed as desirable probes contain drug-like features such as Nitrogen-rich heterocycles, biomimetic amino acid analogs, thioamides, and charged functionality (Fig. S5). The top 20 features in undesirable probes include relatively unstable hydrazones, seleno compounds, heterocycles known to oxidize and a variety of potential electrophiles (Fig. S6).

Bayesian Model 4

Using 307 molecules the model had an ROC value for leave one out of 0.796, 5 fold cross validation ROC = 0.735 (Table S6) and leave out 50% × 100 ROC of 0.69. The top twenty substructure descriptors consistent with compounds classed as desirables probes contain drug-like heterocycles, aromatic fragments with electron withdrawing functionality, and charged functionality (Fig. S7), and the top 20 features in undesirable probes include electron rich ring systems prone to facile oxidation, compounds with adjacent functionality likely to increase functionality (such as two adjacent carbonyl groups or a variety of Michael acceptors), and potentially unstable ring systems (Fig. S8).

Differences are observed between 5 fold cross validation (Tables S1–4) and leave out 50% × 100 cross validation (Table 3). The N307 model has statistics that are slightly higher than the 191 molecule model.

Comparison of machine learning methods

For the n307 dataset a Forest model, a single tree model and an SVM model were all created to compare with the Bayesian model. The 5 fold cross validation ROC values were highest for the Bayesian model followed by the SVM (Table 4).

Table 4.

5 fold cross validation results for n307 Bayesian model of an expert's evaluation of the NIH chemical probes.

Algorithm	Receiver Operator Characteristic
Bayesian	0.735
Support Vector Machine	0.638
Recursive partitioning – Single tree	0.630
Recursive partitioning – Forest	0.607

Rebuilding the Bayesian model in secure CDD Vault with Models

Rebuilding the n307 Bayesian model in CDD Models software with just FCFP_6 fingerprints and using 3 fold cross validation ROC (0.69) suggested that it was comparable to the model developed previously which included additional molecular descriptors (Table 3, 4 Table S4).

External test set predictions

Table 5 summarizes model external testing as additional data were discovered. All four models were compared using the prediction of the final test set of 15 molecules. The largest model with 307 molecules has the best ROC AUC of 0.78 while the model built with 191 molecules comes a close second with an ROC AUC of 0.75. Comparison of the various models and their prediction of test set compounds was enabled via a heatmap (Table 3).

Table 5.

Test set assessment of Bayesian models of an expert's evaluation of the NIH chemical probes, a value of 1 = ideal. Each statistic is colored by the value, where blue is desirable and red is undesirable.

Model	Test molecules	AUC	Sensitivity	Specificity	Precision	Accuracy
N57	114	0.58	0.7	0.42	0.75	0.62
N57	120	0.55	0.69	0.42	0.91	0.66
N57	15	0.72	0.67	0.33	0.8	0.6
N170	21	0.55	0.38	0.6	0.75	0.43
N170	120	0.78	0.62	0.67	0.94	0.63
N170	15	0.58	0.67	0.33	0.8	0.6
N191	120	0.8	0.67	0.75	0.96	0.68
N191	15	0.75	0.83	0.33	0.83	0.73
N307	15	0.78	0.75	0.33	0.82	0.67

We also compared the ability of other tools for predicting the medicinal chemist's desirability scores for the same set of 15 compounds. We found neither the QED, BadApple, or ligand efficiency metrics to be as predictive with ROC AUC of 0.58, 0.36, and 0.29 respectively. Therefore these drug likeness methods do not agree with the medicinal chemist's desirability scores.

DISCUSSION

This work was directly influenced by the earlier study scoring 64 NIH chemical probes by 11 experts¹. We hypothesized that we could learn from one of these medicinal chemistry experts (C.A.L.) and use the resultant computational models to predict newer chemical probe scores. While our aim is not to replace the medicinal chemist, we have shown in this study that a machine learning algorithm has the potential to learn from them, and could be used to filter compounds for assessment either alone or alongside other approaches such as rules or filters recently described^{24, 25,27, 30}. Interestingly, ligand efficiency was higher in compounds scored as undesirable, this would echo recent suggestions that ligand efficiency is not a replacement for considering in vitro and in vivo properties of molecules⁴⁷. Therefore using methods that can learn from the medicinal chemist expert's decisions based on this data may be advantageous.

For accessing and evaluating the public chemical probe data, we followed a standardized process for obtaining prior references. This process had severe limitations caused by a mismatch of privately (commercial) and publicly accessible data, and navigating between these two spheres was problematic which we will discuss elsewhere in due course. In addition, lack of information due to publication bias is another hurdle in medicinal chemistry due diligence. Much of the process of due diligence relies on “soft” skills – such as appropriately combining the literature and making subjective determinations. These aspects of decision making likely will never go away but arguably it would be an advantage if the “soft” aspects of a medicinal chemist's choices could be captured computationally. Much of chemical biology focuses on discovery of tools and probes and sometimes in an environment with considerably more biology than chemistry expertise. We posit that the medicinal chemistry aspect in tool and probe discovery is important and that learning from the thorough documentation of a set of probes that have gone through this process could encode these medicinal chemistry “insights” in an algorithm.

What may have been an early probe learning process could have colored a 2009 published probe assessment¹ in the sense that the earlier assessment included compounds with more dubious credentials that would have not been chosen in more recent probe reports (in 2009, for 64 compounds 75% were acceptable, for 307 compounds in this study 78% were acceptable). Surprising to us was the probe use of trifluoroacetate (TFA) salts given the known deleterious effects of TFA on long term cell culture⁴⁸, the frequent contamination of TFA salts with excess TFA⁴⁹ and the known biological activity of TFA per se⁵⁰.

A cheminformatics analysis of the data collected showed that high confidence probe compounds have statistically higher numbers of rotatable bonds, more heavy atoms and a higher molecular weight (Table 1). These and other readily interpretable calculated properties were used as descriptors alongside FCFP_6 fingerprints to develop machine learning models. These models were both internally and externally validated and went through a prospective iterative learning process as we uncovered data on more probes (Table 3–5) which were scored by the medicinal chemist. The results of this study indicate that Naïve Bayesian models represent a potential way to surmount the issues with bridging the public and commercial data sources required to perform the chemical probe due diligence process. They also perform similarly to recently developed heuristics and may be complimentary. Such models may serve as selection criteria for compounds for screening in order to identify future probes that would be acceptable to medicinal chemists.

In a decade and after over half a billion dollars of investment, over 300 chemical probes have come out of NIH funded screening efforts to date. A thorough due diligence process by an experienced medicinal chemist with over 40 years of experience, suggests that out of a total of 322 probes 79% are considered to have desirable qualities. The outputs from this time- consuming process are a moving target but can be effectively used to build algorithms that learn from the data. A comparison versus other molecule quality metrics or filters such as QED, PAINS, BadApple and ligand efficiency indicates that a Bayesian model based on a single medicinal chemist's decisions for a small set of probes not surprisingly can make decisions that are preferable in classifying desirable compounds (based on the expert's a priori definition of desirability). The use of such machine learning methods may be an approach to increase the probability that a chemical probe would be considered useful by a medicinal chemist, and avoid much wasted time and money invested in poor compounds. Our integration of a Bayesian model based on open source FCFP_6 descriptors alone and the 307 NIH chemical probes in the CDD database (Fig. S9) suggests that it could be used readily to score vendor libraries before screening as well as evaluate future chemical probes prior to the extensive due diligence process being performed with commercial tools. This set of NIH chemical probes could also be scored by other in-house medicinal chemistry experts to come up with a customized score that in turn could be used to tailor the algorithm to their own preferences. For example this could be tailored towards CNS or anticancer compounds.

Abbreviations Used

(AUC)

Area under the curve

(BadApple)

Bioactivity Data Associative Promiscuity pattern Learning Engine

(CDD)

Collaborative Drug Discovery

(CID)

Compound Identifier

(FCFP)

molecular function class fingerprints

(MLSCN)

Molecular Libraries Screening Center Network

(MLPCN)

Molecular Library Probe Production Center Network

(PAINS)

Pan Assay INterference compoundS

(PCA)

Principal component analysis

(SVM)

Support vector machine

(TFA)

Trifluoroacetate

(QED)

Quantitative estimate of drug-likeness

(XV)

cross validated