Skip to main content
ACS AuthorChoice logoLink to ACS AuthorChoice
. 2024 Nov 22;64(23):8684–8704. doi: 10.1021/acs.jcim.4c01116

Improved Prediction of Ligand–Protein Binding Affinities by Meta-modeling

Ho-Joon Lee †,¶,*, Prashant S Emani §,*, Mark B Gerstein §,∥,⊥,#,∇,*
PMCID: PMC11632770  PMID: 39576762

Abstract

graphic file with name ci4c01116_0013.jpg

The accurate screening of candidate drug ligands against target proteins through computational approaches is of prime interest to drug development efforts. Such virtual screening depends in part on methods to predict the binding affinity between ligands and proteins. Many computational models for binding affinity prediction have been developed, but with varying results across targets. Given that ensembling or meta-modeling approaches have shown great promise in reducing model-specific biases, we develop a framework to integrate published force-field-based empirical docking and sequence-based deep learning models. In building this framework, we evaluate many combinations of individual base models, training databases, and several meta-modeling approaches. We show that many of our meta-models significantly improve affinity predictions over base models. Our best meta-models achieve comparable performance to state-of-the-art deep learning tools exclusively based on 3D structures while allowing for improved database scalability and flexibility through the explicit inclusion of features such as physicochemical properties or molecular descriptors. We further demonstrate improved generalization capability by our models using a large-scale benchmark of affinity prediction as well as a virtual screening application benchmark. Overall, we demonstrate that diverse modeling approaches can be ensembled together to gain meaningful improvement in binding affinity prediction.

Introduction

Molecular interactions such as protein–protein or ligand–protein interactions are fundamental processes in biology as well as biophysics and biochemistry. Knowledge of interaction energy or binding affinity is essential in understanding biological functions and biomedical applications. In drug discovery or development, accurate prediction of binding affinity of a drug ligand bound to a target molecule (typically a protein) is of central importance in identifying the most stable structure of a drug–target complex, i.e., the best 3-dimensional pose (geometry) of a drug in the binding pocket of a target molecule, known as molecular docking. Along with development of molecular docking algorithms, there have been extensive computational studies with decades of efforts to achieve better prediction of ligand–protein binding affinity with applications to high throughput drug screening or virtual screening.1−10 This has been achieved by various scoring functions that represent the binding free energy or stability of a ligand–protein complex and select the best docking pose. Docking tools typically generate multiple candidate poses of a ligand and rank them by binding affinity scores. Therefore, accurate prediction of binding affinity is critical in drug discovery and development. Scoring functions were developed traditionally by ab initio physical force field-based, knowledge-based, or empirical methods, or more recently by advanced machine learning or deep learning approaches.

The predictive power of traditional or classical scoring functions is known to be very limited due to rigid or naive underlying assumptions,1,5 e.g. reliance on linear regression11,12 or oversimplification or ignorance of desolvation and entropy contributions because of computational cost.13−15 In addition, previous evaluation studies of more than a dozen scoring functions reported that no scoring function was consistently better than others in all conducted tests, showing both strengths and weaknesses for different purposes.16−18 For example, the best scoring function in terms of a prediction correlation with experimental binding affinities was not necessarily best in terms of predicting correct ligand poses (i.e., not the lowest prediction errors). Despite those limitations, classical scoring functions (especially empirical ones) have been widely used and implemented in several software tools such as Autodock Vina19 or smina.20 Furthermore, consensus scoring strategies from multiple scoring functions have shown improved performances.16,21−24

In continued efforts to improve prediction accuracy, advanced machine or statistical learning-based regression methods such as Random Forests, support vector machines, or neural networks have been successfully used to develop alternative scoring functions by leveraging large-scale databases such as PDBbind25 for more diverse physicochemical and geometric features of ligand–protein complexes and their experimental binding affinities.1,3,11,26−28 Improved performance of machine learning (ML) methods is mainly due to their ability to leverage big data and engineered features to capture nonlinear or nonadditive relationships in binding or docking that are likely to be missed by classical scoring functions.29 Furthermore, with the groundbreaking advancement of deep learning (DL) in computer science, its applications in biomedicine, including protein structure prediction, binding affinity prediction, and drug discovery, have been very successful and promising.1,27,30−39 Deep learning methods for binding affinity prediction are advanced regression methods that typically generate sophisticated abstract representations or embeddings for ligands, proteins, or ligand–protein complexes via various neural network architectures to learn hidden patterns. Inspired by the breakthroughs in computer vision,40,41 various DL models based on convolutional neural networks (CNNs) have been developed using 3-dimensional (3D) structural information on ligand–protein complexes with high prediction accuracies superior to those advanced ML models mentioned above.42−50 In addition, methods based on graph neural networks have been successful as well.42,47,51−56 On the other hand, DL methods with ligand and protein sequences alone, i.e., without structural information, have also shown comparable performances.57,58 In general, DL architectures are highly diverse with a number of hyperparameters such as the network depth and width, loss and activation functions, and a learning rate, which makes model optimization very challenging due to high computational cost. More importantly, fundamental issues with DL approaches have been interpretability or explainability of their nonintuitive excellent performances.59−63 On another note, mathematically advanced methods such as differential geometry and/or algebraic topology-based analyses in combination with ML or DL regression have been also developed showing notably improved performances in ligand–protein binding affinity prediction,64−71 as well as physics-based hybrid models.72,73

In this work, we aim to improve ligand–protein binding affinity prediction by combining both classical scoring functions, which are built upon physical force fields, and sequence-based DL models into a single integrative framework of meta-models (also called ensemble, fusion, or hybrid models). Meta-modeling is a type of ensemble learning that combines diverse algorithms and has been successful in machine or statistical learning for improved results with more robustness and less bias than individual algorithms (known as stacked generalization or super learning).74,75 Meta-models, termed blending in associated publications, were also very successful in addressing the Netflix Prize challenges.76−79 In predicting ligand–protein binding affinity, ensemble models of multiple 3D CNNs have been shown to outperform single CNNs and previous models.50 On the other hand, theory-driven physical models and data-driven ML or statistical models are complementary to each other. The former offers physical interpretability and constraints, whereas the latter is better at complex pattern discovery from massive data. For this reason, hybrid approaches have been favored in recent years.72,73,80−83 In general, the intuition behind meta-modeling is that different solutions to a problem could be modeling nonoverlapping aspects of the ground truth. These different solutions may be trained on data with their own systematic biases, or might have architectures that access some limited subspace of the possible model space. When combined, an ensemble of such solutions might serve to better represent the ground truth. We have designed a modeling framework that allows the inclusion of structure-based classical scoring functions and sequence-based DL models, as well as explicit inclusion of covariates such as physicochemical features of ligand, proteins, or ligand–protein complexes (Figures 1 and S1). In this study, we first focus on 2 empirical scoring functions (ESFs), SMINA and Vinardo (implemented in smina)20 together with consensus docking strategies based on experimental or predicted ligand poses. For DL models, we use sequence-based DL models of 10 different architectures with multiple training strategies, employing an open-source DL library for drug-target interaction prediction called DeepPurpose57 and two ligand–protein binding databases, BindingDB84 and PDBbind. We investigate both pretrained models from the DeepPurpose library and our own fine-tuned or de novo-trained models using PDBbind, which is the main data used by the ESF models. In addition, by merging all prediction scores from repeated cross-validations for all DL models under study, we investigate top principal components from principal component analysis (PCA) as additional prediction scores. Finally, by taking all prediction scores from both ESF and DL models as well as molecular weights of ligands, we build meta-models using 4 machine learning algorithms and 11 feature sets as our final tools (44 in total) for ligand–protein binding affinity prediction.

Figure 1.

Figure 1

Overview of our workflow. The inputs are training datasets (BindingDB and PDBbind) and a cross-validation strategy. The baseline models consist of docking tools (SMINA and Vinardo) and deep-learning (DL) models (from the DeepPurpose library). The DL models were either pretrained or de novo trained, while the docking tools were pretrained. The output ligand poses of the docking tools are filtered by structural comparisons with the experimental docked poses, or by comparisons between the poses predicted by the two docking tools. The BindingDB-trained DL models are fine-tuned on the PDBbind dataset, and their prediction scores are further transformed through principal component analysis (PCA). Additional features, such as molecular weight, can be added, before various combinations of all these meta-features (filtered or unfiltered docking scores, fine-tuned and/or PCA-transformed DL scores) are fed into several machine-learning meta-models for final training on the PDBbind dataset. The meta-models include linear regression, LASSO, ElasticNet, and XGBoost.

Using 3 diverse benchmark evaluations of CASF-2016,7 the large-scale PDBbind v2020 general set, and a virtual screening dataset of LIT-PCBA,85 we demonstrate that our meta-models outperform individual base models and exhibit competitive performance to more sophisticated structure-based DL models, serving as a potentially useful general framework for continual future improvement of ligand–protein binding affinity prediction. The advantages offered by such an approach over existing methods include: (1) scalability by utilizing sequence-based databases for DL models without the need for 3D structures; and (2) flexibility by allowing different base models and physicochemical or molecular properties to be incorporated into meta-models. We believe that such an extensive exploration of these aspects of binding affinity prediction makes a meaningful contribution to the discussion of model generalization.

Methods

Dataset Selection

BindingDB

We used the BindingDB ver. 2020m2 database (http://www.bindingdb.org) for the purpose of DL model training. A total of 66,444 ligand–protein complexes from the Ligand-Target-Affinity dataset were filtered with Kd ≤ 0.01M, valid PubChem IDs or InChI keys for ligands, and valid UniProt IDs for proteins.

PDBbind

We downloaded the PDBbind v2020 refined set from http://www.pdbbind.org.cn/. The dataset consisted of 5,316 target protein–drug ligand complexes. The refined set (labeled RefinedSet) has been filtered by the PDBbind creators from a larger dataset by the removal of complexes that have “obvious problems in 3D structure, binding data or other aspects”.86 Of these, 285 ligand–protein complexes are in the so-called core set (labeled CoreSet), which are high-quality data that are used as the primary test set for the Comparative Assessment of Scoring Functions (CASF) benchmark.7 This CoreSet was separated from the remainder of the PDBbind RefinedSet to serve as an external test set (i.e., a benchmark set) for evaluating model performance. Thus, we considered a training set of 5031 ligand–protein complexes and an external test set of 285 complexes. As 266 of 285 affinities were properly predicted upon running the docking tools (see below), we took those 266 complexes as our primary benchmark set. The target binding affinity values we used in this study were the natural logarithms of Kd or Ki values. We did not distinguish Kd and Ki from each other. We denote “Kd or Ki” as “Kd/Ki” hereafter.

Molecular Docking

Implementation of Ligand Docking

We employ the smina implementation20 of the AutoDock Vina method19 for the docking component. The smina implementation is designed to make AutoDock easier to use, with larger support for ligand molecular formats and improved minimization algorithms. We used both the default SMINA scoring function and another previously published scoring function, Vinardo,87 provided in smina. We note that the italicized smina describes the program while the capitalized SMINA describes the scoring function. In the meta-models below, we refer to this class of empirical docking methods as E. The tandem use of both scoring functions is meant to address possible idiosyncrasies in the parametrizations of the scoring functions. smina is run as a command-line program using a receptor structure in PDBQT format and a ligand in 3D SDF format. Our in-house Python script includes steps for the preparation of the receptor structures, the creation of the configuration file, and the running of smina (Supporting Information). The output for each scoring function consists of the binding affinities for the 9 (the default option) best docking poses. Seven complexes with multiple ligands (whose names were separated with “&” or “/”) in the PDBbind dataset were not docked as there was some ambiguity as to which ligand to consider in the docking. These complexes were excluded from further analyses.

Processing of Docking Scores

Previous studies have demonstrated the value of (a) filtering the docking poses output by empirical docking programs based on the degree to which they agree with experimental scores20 and (b) comparing poses from two or more docking methods, in so-called “consensus” approaches, and filtering based on the degree of agreement to yield a higher quality output.23,24,88 Accordingly, we carried out preprocessing of the docking scores by filtering out certain complexes based on the quality of the associated poses assessed by these two criteria. For each complex out of the available 9 docking poses, we calculated the root-mean-square deviation (RMSD) between docking poses in two different ways (described below), and utilized the RMSD value as a cutoff in determining the best set of binding affinities to choose in the final dataset to the meta-models. The goal was to determine if pose quality, either relative to an available experimental gold-standard or relative to the results of another scoring function, would impact both the matching of the predicted binding affinity as well as the downstream prediction process.

We choose to implement the symmetric RMSD employed by Trott and Olson19 using PyMOL’s Python API (The PyMOL Molecular Graphics System, Version 1.2r3pre, Schrödinger, LLC). In brief, this RMSD is calculated between two different ligand structures as RMSDab = max(Ra|b, Rb|a) where Ra|b describes an asymmetric measure of distance:

graphic file with name ci4c01116_m001.jpg

where the summation is over all N heavy atoms in structure a, the minimum is over all atoms (j) in structure b of the same element type as atom i in structure a.

We calculate RMSDab in two ways, depending on the choice of docking poses: (1) Experimental pose-RMSD filtering, where the RMSD is calculated for a ligand pose predicted by a docking method relative to the experimental structure of the ligand (when available); and (2) Consensus pose-RMSD filtering, where the RMSD is calculated between the docking poses for the same ligand–target pair as predicted by two different docking scoring methods, Vinardo and SMINA. Details of these filtering approaches are provided in the Supporting Information. We also chose a cutoff of 3 Å as representing a reasonably good quality match between poses based on our manual inspection of RMSD distributions (Figure S2A). This is a more lenient cutoff relative to other analyses20,23 that suggest a 2 Å cutoff. Our approach is a compromise between the need for higher quality matches, and the need to have sufficient numbers of data points to train our models as stricter cutoffs lead to smaller dataset sizes.

An important aspect of these parallel approaches to applying the RMSD filters on the data is that the scores extracted from the empirical docking runs are different when considering the Experimental RMSD filter and the Consensus RMSD filter. This is even the case when considering the unfiltered dataset, where the set of complexes are identical, but the choice of poses used to extract the best docking scores differ between the two RMSD filter approaches. Another aspect we note is that experimental structures are not necessarily available for protein–ligand complexes used in many downstream use cases. It is therefore not our intention that users will apply these filters on the prediction dataset. Rather, we utilize the filters on our training data to produce what is hopefully a more informative dataset for the meta-model training. Our evaluations of the relative merits of different filters on the training set are provided in Results. Finally, we note that, in the naming of the model files, we use the shorthand “VvS” to indicate the Consensus RMSD filtering as the comparison involves Vinardo versus SMINA poses.

Deep Learning

DeepPurpose Library

We used the Python library, DeepPurpose, for deep learning of binding affinities of ligand–target complexes.57 It offers a framework for various models that combine ligand-encoding and protein-encoding neural networks. We modified and customized the version 0.0.1 of the library (as of July 24, 2020) for development of our work, while all 6 pretrained models with BindingDB-2020m2-Kd were downloaded from the version 0.1.5 (as of June 29, 2022). The 6 pretrained models are CNN-CNN, Daylight-AAC, Morgan-AAC, Morgan-CNN, MPNN-CNN, and Transformer-CNN. We refer to this class of BindingDB pretrained models as D1.

Training of 12 DL Models from the DeepPurpose Library Using BindingDB

The DeepPurpose library offers various embedding methods for both ligands and proteins to build regression models to predict binding affinities of ligand–protein complexes. After initial pilot experiments, we chose the following 12 models of ligand–protein embedding methods for downstream analyses: CNN-CNN, Daylight-AAC, Morgan-AAC, Morgan-CNN, MPNN-CNN, Transformer-CNN, MPNN-Quasi-seq, CNN/RNN-CNN/RNN, CNN-AAC, MPNN-Transformer, MPNN-PseudoAAC, MPNN-AAC, MPNN-CNN/RNN, Transformer-Quasi-seq, and Transformer-Transformer. We applied the 12 models to BindingDB Kd data, which were used by the DeepPurpose library. Their 6 pretrained models mentioned above were reported to achieve performances of 5-fold cross-validated mean squared error (MSE) < 0.6 or Pearson correlation coefficient (PCC) < 0.8 on average, using the target binding affinity unit of −log10(Kd).57 Given the published results of their best models, we retrained or de novo-trained all 12 models from scratch to either reproduce their best models or identify more models with similar performances for our downstream model development. We used all default hyperparameters for each model in DeepPurpose and did not perform cross validation given the target performance of MSE = 0.6 or PCC = 0.8. The data split for training, validation, and testing was 70% (46,511 complexes), 10% (6644 complexes), and 20% (13,289 complexes), respectively, with random seed = 1. Those top models selected with the target performance are considered as crude or low-confidence deep learning models, being a starting point in our entire pipeline for the sake of computational efficiency given limited resources. We refer to this class of BindingDB de novo-trained models as D2.

Training of DL Models Using PDBbind

We also trained 10 DL models using the PDBbind RefinedSet binding affinity data (Kd/Ki) that contain ligand–target structural data and were used by the docking tools. The 10 models come from the 6 pretrained models and the top 6 de novo-trained models selected from above (two models in common). This is to compare with the BindingDB-trained DL models. We excluded the CASF2016 CoreSet within the PDBbind RefinedSet, which we use for external validation, and performed 5-fold cross validation, repeated 10 times, with random seed = 1701. We refer to this class of PDBbind-trained models as D3.

Fine-Tuning of the BDB-Trained Models Using PDBbind

In order to leverage the respectable performance of the BindingDB-trained models (D1 and D2), we finetuned them using the PDBbind data excluding the CoreSet (referred as D1F and D2F, respectively). To increase analysis robustness, we performed iterative 5-fold CV with 10 iterations, i.e., 50 validations (using the scikit-learn function in Python, RepeatedKFold or KFold, with random_state = 1701) for each of the 6 D1F, 6 D2F, and 10 D3 models to yield 300, 300, 500 model variants or predictions, respectively. We also performed PCA on those predictions for the 3 model classes (referred as D1FP, D2FP, D3P, respectively) as well as on the merged 1,100 predictions (referred as DAP) for ML meta-models below for dimensionality reduction of the meta-features (predictors).

Machine Learning Meta-models

The fact that we have two independent approaches of quantifying the binding affinities (i.e., docking and deep learning tools) means that we potentially have complementary sources of information in the data. The docking tools are based on empirical scoring functions trained to match experimental structural data, while the deep learning tools capture information on the sequences of ligands and proteins. The exact partition of dual sequence/structure impacts on the results from these two approaches is not entirely clear, but the expectation is that there is likely nonoverlapping information contained in the predictions of these approaches. Accordingly, we have sought to exploit any complementary information by feeding the docking and deep learning scores into a set of linear and nonlinear machine learning models to gain improvements in prediction of binding affinities. We term these machine learning models as meta-models since they incorporate predictions from independent tools as base models. We used linear regression, ElasticNet, and LASSO algorithms for linear meta-models and the XGBoost algorithm for a nonlinear meta-model implemented in scikit-learn in Python. Hyperparameter optimization was done along with multiple 5-fold cross validations. Details are given in Supporting Information.

The features for the meta-models are SMINA and Vinardo predictions with different RMSD filtering cutoffs, ligand molecular weights (MWs), and DL predictions. We experimented with various combinations of those features: (1) SMINA + Vinardo (with/without the filters) (2) SMINA + Vinardo (with/without the filters) + MW (3) SMINA + Vinardo + MW + DL mean predictions (BindingDB or PDBbind) and (4) SMINA + Vinardo + MW + DL PCA projections (BindingDB or PDBbind). For the BindingDB-trained scores, we used the top 6 models as described above. For the DL models fine-tuned on the PDBbind dataset, we take the mean of the 50 CV predictions for each of the 6 models (i.e., 6 meta-model features). For the PCA of all DL scores, we also ran an optimization over the number of principal components (PCs) to include as features. We allowed for between 1 and 22 top principal components to be included (i.e., PC 1, PCs 1–2, PCs 1–3, ..., PCs 1–22), and the selection of the best model among the 22 options was done by maximizing the Pearson correlation for each of the 4 meta-models separately using 20% of the training data as a leave-out validation set for hyperparameter optimization. We chose to optimize up to the top 22 PCs to match the number of models used in all the deep learning runs in D1, D2 and D3 combined.

Each of the optimized meta-models generated above is tested against benchmark sets using the relevant meta-model-specific feature sets. For all meta-models, PCC, the Spearman rank correlation coefficient, MSE, and the root-mean-square error (RMSE) were calculated and reported for model evaluation. A detailed description of these meta-model features is shown in Table 1.

Table 1. Descriptions of Meta-model Feature Groups.

Feature Group ID Description of features included Feature list
E Smina and Vinardo empirical (E) docking scores only Smina score, Vinardo score
EW E + molecular weight (MW) Smina score, Vinardo score, MW
ED1 E + MW + Deep Learning (D) average scores from 6 BindingDB-pretrained models provided in DeepPurpose v0.1.5 Smina score, Vinardo score, MW, CNN_CNN, Daylight_AAC, Morgan_AAC, Morgan_CNN, MPNN_CNN, Transformer_CNN
ED2 E + MW + D average scores from 6 models from DeepPurpose v0.0.1 trained de novo on BindingDB Smina score, Vinardo score, MW, CNN_AAC, MPNN_CNN, CNN_CNN, Daylight_CNN, MPNN_AAC, MPNN_Transformer
ED3 E + MW + D average scores from 10 models from DeepPurpose v0.0.1 trained de novo on PDBbind Smina score, Vinardo score, MW, CNN_AAC, MPNN_CNN, CNN_CNN, Daylight_CNN, MPNN_AAC, MPNN_Transformer, Daylight_AAC, Morgan_AAC, Morgan_CNN, Transformer_CNN
ED1-F E + MW + D average of scores obtained from fine-tuning the ED1 models, using PDBBind v2020 (the D scores are averaged for each model across 5 folds of cross-validation X 10 repetitions) Smina score, Vinardo score, MW, CNN_CNN, Daylight_AAC, Morgan_AAC, Morgan_CNN, MPNN_CNN, Transformer_CNN
ED2-F E + MW + D average of scores obtained from fine-tuning the ED2 models, using PDBBind v2020 (the D scores are averaged for each model across 5 folds of cross-validation X 10 repetitions) Smina score, Vinardo score, MW, CNN_AAC, MPNN_CNN, CNN_CNN, Daylight_CNN, MPNN_AAC, MPNN_Transformer
ED1–F-P E + MW + D, where the reported scores are obtained by applying PCA to all scores from the 5 folds of cross-validation X 10 repetitions for the six models in ED1-F Smina score, Vinardo score, MW, Up to 22 top PCs (determined for each machine learning method through hyperparameter optimization)
ED2–F-P E + MW + D, where the reported scores are obtained by applying PCA to all scores from the 5 folds of cross-validation X 10 repetitions for the six models in ED2-F Smina score, Vinardo score, MW, Up to 22 top PCs (determined for each machine learning method through hyperparameter optimization)
ED3-P E + MW + D, where the reported scores are obtained by applying PCA to all scores from the 5 folds of cross-validation X 10 repetitions for the six models in ED3 Smina score, Vinardo score, MW, Up to 22 top PCs (determined for each machine learning method through hyperparameter optimization)
ED-A-P E + MW + D, where the reported scores are obtained by applying PCA to all scores from the 5 folds of cross-validation X 10 repetitions for the models in ED1-F, ED2-F and ED3 Smina score, Vinardo score, MW, Up to 22 top PCs (determined for each machine learning method through hyperparameter optimization)

SwissADME and UniProt Analysis

To understand the contributions of ligand properties to benchmark performances of all the models, we used the web tool SwissADME(89) to extract the properties of the PDBbind CoreSet ligands from their SMILES strings. Also, we investigated the SwissADME features that might be associated with better or worse performance for subsets of complexes that were best predicted by the different groups of tools (docking, DL, and meta-models). To evaluate the contributions of protein features, we extracted the relevant subsets of PDBbind CoreSet PDB IDs from our analyses and fed them into the “Retrieve/ID mapping” tool of the UniProt database.90 We studied the resulting protein properties (e.g., pathway information) for over- or under-representation in our predictions. Details are provided in the Supporting Information.

Comparison to Other Tools

We compared our models to 4 recently published structure-based tools. For fair comparisons, we retrained those tools using our training dataset as long as there were no technical difficulties. Kyro et al. developed a new structure-based DL ensemble or meta model, HAC-Net,47 by combining MP-GNN (message passing graph neural networks) and 3D-CNN (convolutional neural networks) trained on the PDBbind-2020 general set of >18,800 ligand–protein complexes. For a fairer, but limited, comparison, we retrained HAC-Net (v1.3.1) with our training set of the PDBbind-2020 RefinedSet\CoreSet by subsetting from their train/validation/test data splits (4942/82/265 out of 18,818/300/290 complexes from the general set, respectively). Jones et al. developed meta or fusion models, FAST,42 by combining 3D-CNN and SG-CNN (spatial graph CNN). Due to technical difficulties, we were able to only run their pretrained SG-CNN model trained on the PDBbind-2016 general set excluding the CoreSet. KDeep46 was one of the earlier structure-based DL models with 3D-CNN to predict ligand–protein binding affinity. The KDeep model available on the public web server91 could be retrained with limited data size (KdeepTrainer, https://playmolecule.com/KdeepTrainer/). While we were not able to use our full training data due to the limitation, we were able to retrain the KDeep model for 250 epochs with each of 3 training sets of 1800 random complexes sampled from our training set of the RefinedSet\CoreSet. PerSpect ML69 is one of the best performing tools in recent years based on topological data analysis and spectral theory for multiscale molecular featurization, which may generate more than 100,000 features from 3D structures or binding poses for the gradient boosting tree-based regression. Due to incomplete code availability, it was not possible to reproduce or implement PerSpect ML for fair comparisons with the data we used. We compare our results to their published results with that caveat.

GeneralSet Benchmark

To assess model generalizability on a large scale, we performed an external validation of binding affinities for complexes in the PDBbind v2020 general set as another benchmark in addition to the CoreSet. The full general set consists of 19,443 complexes and is not as carefully curated as the RefinedSet. Hence, we filtered complexes according to the following rules:

  • 1.

    Filter out all complexes belonging to the RefinedSet and CoreSet.

  • 2.

    Remove all complexes assayed by NMR.

  • 3.

    Remove all complexes with an IC50 dissociation measure (as opposed to Kd or Ki).

  • 4.

    Only include complexes whose PDB structures were published after the year 2000.

  • 5.

    Finally, not all complexes had successful docking calculations for both Vinardo and SMINA. We left out complexes with any predicted binding affinity of zero.

The resulting subset included 5261 complexes, which is larger than our training set of 5031 complexes from the RefinedSet. Therefore, it is a more challenging benchmark than the CoreSet. We ran the docking and our model predictions using the same settings as for the CoreSet. We label this filtered subset as the ‘GeneralSet’, to differentiate it from the full PDBbind general set.

Ranking Benchmark

To test the possibility of certain tools performing the prediction task with differing degrees of accuracy for certain proteins or even families of proteins, we devised a virtual screening test by selecting 3 most frequently targeted proteins by multiple ligands from the PDBbind general set excluding the RefinedSet we used for model training. We considered only those proteins in the general set whose sequence similarity to those in RefinedSet was found to be less than 30% by the NCBI Blastp package.92 We also filtered out complexes in the general set that had only IC50 experimental measurements for consistency with this study. The resulting top 3 proteins are O60341, O15151, and P0C6U8 (UniProt Accession IDs), which are complexed with 20, 8, and 7 ligands, respectively. The resolution of the 35 complexes ranges from 1.33 to 3.50 Å with mean of 2.58 and standard deviation of 0.69.

Virtual Screening

For a more general applicability, we also performed a virtual screening using an external benchmark dataset, LIT-PCBA, of active and inactive ligands for 15 target proteins.85 Although our models were neither explicitly designed nor trained to distinguish active and inactive ligands from each other (i.e., binary classification for functional activity), our assumption was that active ligands tend to have higher binding affinities (i.e., lower Kd/Ki) for target proteins than inactive ligands. This assumption is weak because there is an unclear causal relationship between binding affinity (Kd, Ki) and functional activity (IC50, EC50). As mentioned above, we used ligand–protein complexes with Kd or Ki from both BindingDB and PDBbind for consistency. LIT-PCBA is a recently developed virtual screening dataset which was reported to be less biased than previous datasets. It is based on PubChem BioAssays which includes 4 potency types of IC50, EC50, Kd, and Ki. We point out that LIT-PCBA did not distinguish the 4 potency types from each other. As a proof-of-concept, with the aforementioned caveat in mind, we randomly selected active and inactive ligands maintaining the ratio 1:2 (number of active ligands:number of inactive ligands) for each of the 15 proteins in LIT-PCBA to predict their binding affinities by our models. Specifically, if the number of active ligands was <100, we included them all; if greater than or equal to 100, we randomly sampled 100 active ligands. The inactive ligand numbers were then selected to maintain the aforementioned 1:2 ratio. This gave us a total of 1,050 active ligand–target protein pairs and 2,100 inactive ligand–target protein pairs. The active and inactive ligands were chosen from the combined training and validation sets in LIT-PCBA. We generated three-dimensional SDF files for each of the ligands from the SMILES strings. We also randomly selected only one of the PDB structures (in mol2 format) for each target protein in this virtual screening exercise (further details in the Supporting Information). The evaluation was done in 4 ways: (1) comparisons of predicted binding affinity distributions between active and inactive ligands for each protein with one-sided Welch’s t test and the Mann–Whitney U test for significant differences; (2) top5 recall: an enrichment or fraction of active ligands (hits) in the top 5 highest-affinity ligands predicted for each protein; (3) similarly, top10 recall for hits in the top 10 ligands; and (4) precision for hits in the number of the top ligands equal to that of all active ligands for each protein.

Results

Docking Tools

We ran smina with two different scoring functions, SMINA and Vinardo (model class E), on the PDBbind dataset. We distinguish the PDBbind “RefinedSet” excluding the “CoreSet” (designated as “RefinedSet\CoreSet”) from the “CoreSet” which is used as the external validation (benchmark) set in this study. Additionally, as described in the Methods, we considered two formulations of the docking pose-based RMSD for the 9 poses generated by the docking tools for each protein–ligand complex: an “Experimental” RMSD relative to the experimental pose, and a “Consensus” RMSD. We indicate the unfiltered dataset by RMSD < 101 Å, as all structures will satisfy this condition based on our setup (see Methods and Supplementary Methods for details). We compare SMINA and Vinardo scores against the experimental binding affinities using both RMSD filter types for the two scoring functions and cutoffs of 101 Å, 100 Å and 3 Å (Figure 2). The sizes of the RefinedSet\CoreSet data satisfying the filters are provided in Table S1.

Figure 2.

Figure 2

Performances of empirical scoring functions and consensus filtering. The outputs of the docking tools, SMINA and Vinardo, are shown with the binding affinities for the RefinedSet\CoreSet in black and those for the CoreSet in red. The columns indicate the different choices of filtering approaches and pose-based RMSD thresholds. The Pearson correlation values for the two different sets of predictions are shown in each panel in their respective colors.

The results indicate that the outputs of the docking tools consistently have a moderate correlation with the experimental results (on the order of 0.5), for both the RefinedSet\CoreSet, and the CoreSet. Upon restricting the Experimental RMSD cutoff, the Pearson correlation with the experimental docking scores (both SMINA and Vinardo) improves to 0.58–0.59 for the RefinedSet\CoreSet at RMSD < 100 Å, while the correlation improves for the CoreSet only for RMSD < 3 Å. The latter effect could be a result of either a legitimate increase in prediction quality when restricting the Experimental RMSD for the CoreSet, or could simply be the stochastic impact of a reduction in the dataset size. To test this, we ran Monte Carlo simulations randomly subsetting 197 out of 266 complexes a thousand times, and calculating the resulting correlation with experimental affinities. The empirical null distributions demonstrate that the improvement is unlikely to be a statistical artifact of subsampling (Figure S2B). Overall, this indicates that constraining the pose to be similar to the experimental pose has benefits for the quality of the docking score prediction. The results for the Consensus RMSD do not suggest any significant improvement by lowering the RMSD cutoff (only a slight improvement is observed for the RefinedSet\CoreSet).

Additionally, it is interesting to note that the prediction on the CoreSet is consistently better than that of the RefinedSet\CoreSet. The most obvious explanation for this is that the scoring function for Vinardo was trained on the PDBbind 2013 CoreSet. This dataset overlaps with the PDBbind 2016 CoreSet we are using. The SMINA scoring function was trained on the CSAR-NRC HiQ 2010 dataset,93 which includes some structures from PDBbind RefinedSet from 2007. These overlaps will result in better performance on the CoreSet. Another possibility is that the complexes in the CoreSet are, by design, of a higher quality in terms of the removal of steric clashes and other structural irregularities relative to the RefinedSet\CoreSet. Working with such protein–ligand pairs may result in a better fit to the experimental binding affinities. We also note that the SMINA and Vinardo scores are highly correlated with each other, a fact that impacts meta-modeling later, as two strongly collinear features compete with each other when input into the same model.

We conclude that the overall performances of the docking results are not strong enough to make reliable predictions. The results motivated us to explore alternative methods of binding affinity prediction or scoring, especially deep learning tools.

Deep Learning Models

Trained Models Using BindingDB or PDBbind-RefinedSet

As our first group of DL models, we investigated either pretrained (i.e., published) or de novo-trained models using the BindingDB (BDB) or PDBbind-RefinedSet (PDBb) data (model class D1, D2, or D3, respectively) (Table 2). Based on mean-squared-error (MSE), we selected the following top 6 out of the 12 de novo-trained models for subsequent analyses, which show comparable performance to the DeepPurpose pretrained models with MSE < 0.6: CNN-CNN, Daylight-AAC, Morgan-AAC, Morgan-CNN, MPNN-CNN, and Transformer-CNN (Figure S3). As the overlap between the published and de novo-trained models is only two, CNN-CNN and MPNN-CNN, this expanded the family of BDB-trained base models to a total of 10 with competitive performances in this study (Table 2). In terms of mean cross-validation (from 10× 5-fold CV), the best performing BDB-pretrained, BDB-de novo-trained, and PDBb-trained models were Daylight-AAC, Daylight-CNN, and CNN-AAC (Table 2). The model average Pearson correlation coefficients were 0.468, 0.475, and 0.746 for the CoreSet, respectively (Figure 3A; D1, D2, and D3). The two BDB-trained models perform worse than the docking tools (Figure 2). Full performance results are provided in Tables S2 and S3. We note that the PDBb-trained models performed poorly for BDB-Kd predictions with poor correlations (Figure S4).

Table 2. Performances of DL Models.
    Validation
    PDBbind2020 Refined Set
CASF-2016 CoreSet
Model Training Pearson MSE Pearson MSE
CNN_CNN BindingDB pretrained (D1) 0.424 18.734 0.358 25.116
Daylight_AAC 0.407 19.852 0.383 24.327
Morgan_AAC 0.465 17.859 0.343 25.274
Morgan_CNN 0.44 20.083 0.292 27.301
MPNN_CNN 0.47 21.167 0.333 27.685
Transformer_CNN 0.324 20.098 0.301 26.094
CNN-AAC BindingDB de novo-trained (D2) 0.417 19.434 0.392 23.621
CNN-CNN 0.399 19.505 0.337 24.735
Daylight-CNN 0.382 21.756 0.461 24.559
MPNN-AAC 0.439 20.300 0.280 29.255
MPNN-CNN 0.417 22.031 0.286 30.418
MPNN-Transformer 0.237 26.710 0.297 27.936
CNN/RNN-CNN/RNN 0.235 24.322 0.261 28.996
MPNN-CNN/RNN 0.303 24.030 0.135 34.107
MPNN-PseudoAAC 0.351 20.376 0.333 26.823
MPNN-Quasi_seq 0.366 19.763 0.344 25.803
Transformer-Quasi_seq 0.002 23.456 –0.098 30.534
Transformer-Transformer 0.220 21.101 0.100 28.555
CNN-AAC PDBbind2020 trained (D3) 0.929 2.913 0.76 10.581
CNN-CNN 0.91 3.671 0.733 11.624
Daylight_AAC 0.895 4.161 0.685 13.132
Daylight-CNN 0.875 4.933 0.599 15.767
Morgan_AAC 0.886 4.465 0.718 12.077
Morgan_CNN 0.926 3.078 0.644 14.375
MPNN-AAC 0.916 3.371 0.731 11.471
MPNN-CNN 0.897 4.121 0.727 11.76
MPNN-Transformer 0.922 3.161 0.737 11.29
Transformer_CNN 0.824 6.681 0.553 17.155
CNN_CNN BindingDB pretrained and PDBbind fine-tuned (D1F) 0.844 5.846 0.691 12.891
Daylight_AAC 0.861 6.615 0.628 16.728
Morgan_AAC 0.874 5.154 0.677 13.98
Morgan_CNN 0.887 4.398 0.637 14.739
MPNN_CNN 0.873 5.04 0.634 15.042
Transformer_CNN 0.751 8.84 0.489 19.17
PC1 NA NA 0.715 NA
CNN-AAC BindingDB de novo-trained and PDBbind fine-tuned (D2F) 0.841 6.618 0.627 16.344
CNN-CNN 0.872 5.907 0.654 15.773
Daylight-CNN 0.829 6.5 0.613 15.694
MPNN-AAC 0.894 4.162 0.663 13.882
MPNN-CNN 0.895 4.113 0.654 14.2
MPNN-Transformer 0.876 5.465 0.637 15.822
PC1 NA NA 0.749 NA

Pearson ≥ 0.700 and MSE < 12.000 in bold italic.

Figure 3.

Figure 3

Performances of deep learning models. (A) Scatter plots of experimental binding affinities (x axis) and model predictions (y axis) for the full 5,305 complexes in the PDBbind RefinedSet (266 complexes in the CoreSet benchmark). Each plot shows predictions by the average of the best models or PC1 in terms of Pearson correlation coefficients for the CoreSet in each training strategy. (B) Box-whisker plots of Pearson correlation coefficients (left panel) and mean squared errors (MSE; right panel) for the CoreSet by a total of 22 PDBbind-finetuned (M1 and M2) and PDBbind-trained (M3) models and PC1 models of M1, M2, and M3 along with SMINA and Vinardo models. Predictions of the 22 models are mean values from 10 repeated 5-fold CVs. M1, M2, and M3 correspond to D1F, D2F, and D3 in (A).

Fine-Tuned Models Using PDBbind-RefinedSet

As our second group of models, we investigated fine-tuned models using PDBb for both BDB-pretrained and BDB-de novo-trained models (D1F and D2F, respectively). In terms of mean cross-validation (from 10× 5-fold CV), the best performing PDBb-finetuned models for BDB-pretrained and BDB-de novo-trained models were CNN-CNN and MPNN-AAC. The model average Pearson correlation coefficients were 0.716 and 0.749 for the CoreSet, respectively (Figure 3A). Full performance results are provided in Table S3. As for BDB-Kd prediction, all PDBb-finetuned models maintained good performances with Pearson correlations of 0.6–0.7, compared to predictions by the BDB-trained models (Figures S4 and S5).

PCA of Predictions

As our third or final group of DL models, we investigated PCA projections of all predictions from those fine-tuned models for the BDB-pretrained (D1F) or BDB-de novo-trained (D2F) in the second model group (6·5·10 = 300 D1F or D2F model instances from 10× 5-fold CV for each model architecture) or those PDBb-trained models (D3) in the first model group (10·5·10 = 500 model instances). The first principal component (PC1) shows the best performance in all PCAs, Pearson correlation coefficients for D1F, D2F, and D3 being 0.715, 0.749, and 0.747, respectively (Figure 3A; D1FP, D2FP, and D3P). We also performed PCA on the merged set of all predictions from D1F, D2F, and D3 (300 + 300 + 500 = 1100 model instances), the Pearson correlation coefficient for PC1 being 0.744 (Figure 3A; DAP). As expected, the PC1 is highly correlated with the mean prediction with the largest Pearson correlation coefficient of >0.99 in all cases, while the second most correlated PC was not the second PC (PC2). By looking at performances of all cross-validated fine-tuned or PCA models, we observe that PC1 and D3 models tend to perform better than D1 or D2 models for the CoreSet in terms of Pearson correlation coefficients or mean squared errors (Figure 3B).

Comparisons among DL and Docking Models

Given that we have investigated the 38 DL models (6 D1, 6 D2, 10 D3, 6 D1F, 6 D2F, 1 D1FP, 1 D2FP, 1 D3P, and 1 DAP) as well as the 4 docking models, we first compared their predictions by Pearson correlation values among themselves. Figure 4A shows a heatmap of Pearson correlation values for all pairs of predictions for the CoreSet. As expected, high correlations (i.e., similar predictions) are observed among (1) the 4 docking models and (2) the fine-tuned or PC1 models (D1F, D2F, D1FP, D2FP, D3P). The BindingDB-based models (D1 and D2) are less correlated among themselves. Correlations between the experimental binding affinity and those predictions show that the top model is D1F|Daylight_AAC (pretrained and fine-tuned) and the next top 3 models are PC1 models (Figure 4B). We also performed another comparison analysis by prediction errors or deviations from the experimental binding affinity (Table S3). Figure 4C,D shows a heatmap and a box-whisker plot of prediction deviations or absolute errors for the 42 models. The fine-tuned or PC1 models tend to show lower errors than the BindingDB models, which in turn tend to show lower errors than the docking models. The top 5 models in terms of median errors are the 4 PC1 models and D2F|MPNN_Transformer (de novo-trained and fine-tuned). Similar but more pronounced correlations and deviations are observed for the training set, as expected (Figure S6).

Figure 4.

Figure 4

Model comparison of CoreSet predictions. (A) Heatmap of Pearson correlations between all pairs of CoreSet predictions by the 42 models from the docking and deep learning tools. The dendrograms are results of hierarchical clustering with Euclidean distance and the complete linkage method. (B) Bar plot of Pearson correlations between the experimental binding affinity and predictions by the 42 models for the CoreSet complexes. The models are ordered by the correlation values. (C) Heatmap of absolute errors or deviations between the experimental binding affinity and predictions by the 42 models (columns) for each of the CoreSet complexes (rows). The dendrograms are as in (A). (D) Box-whisker plot of the data used in (B). The models are ordered by the median values.

Meta-models of Docking and DL Tools

We ran several versions of meta-models to cover the spectrum of input features discussed in Methods (Table 1), with the goal of parsing the contributions of particular components of the models. The broad feature groups included docking scores, untransformed DL scores, and PC-transformed DL scores. Early exploratory analyses with SwissADME-derived ligand properties suggested the inclusion of molecular weight to contend with possible biases in the docking scores. Furthermore, we examined the impact of different DL training sets and models.

We present representative meta-model results using XGBoost in Table 3 and Figure 5, which showed the best performance among the 4 algorithms (Tables S4–S6). The different panels show the scores for the XGBoost meta-models trained on RefinedSet\CoreSet in black and the corresponding test scores for the CoreSet in red. The panels of Figure 5 are arranged in a manner that reflects the different sets of features and training data: starting with meta-models constructed entirely based on the docking methods, SMINA and Vinardo; adding in MW; the inclusion of DL scores based on BindingDB or PDBbind trainings; and the fine-tuning and PCA. Note that, in the PC-based models, the optimal number of PCs was obtained by maximizing the respective machine-learning scores for a leave-out validation set (the RefinedSet\CoreSet was split 80%–20% into a training-validation partition).

Table 3. Meta-model Performances by XGBoosta.

  Pearson
MSE
  Consensus
RelExpt
Consensus
RelExpt
Feature groups 101 3 101 100 3 101 3 101 100 3
E 0.527 0.527 0.547 0.536 0.628 18.192 16.756 17.761 17.459 12.529
EW 0.587 0.568 0.586 0.605 0.639 16.997 15.907 16.842 15.804 12.226
ED1 0.600 0.598 0.620 0.636 0.671 16.000 14.765 15.504 14.914 11.235
ED2 0.639 0.626 0.645 0.657 0.702 15.030 14.125 14.753 14.249 10.426
ED3 0.753 0.743 0.750 0.758 0.717 10.587 10.163 10.728 10.340 9.717
ED1-F 0.722 0.694 0.722 0.714 0.703 11.730 11.779 11.742 11.902 10.175
ED2-F 0.750 0.739 0.750 0.752 0.704 10.748 10.339 10.757 10.579 10.139
ED1–F-P 0.747 0.731 0.755 0.749 0.735 10.951 10.650 10.664 10.763 9.149
ED2–F-P 0.768 0.750 0.768 0.770 0.732 10.164 9.967 10.198 9.973 9.267
ED3-P 0.777 0.764 0.778 0.773 0.769 9.920 9.602 9.879 10.013 8.202
ED-A-P 0.763 0.759 0.764 0.776 0.765 10.420 9.804 10.437 9.951 8.368
a

Consensus = Consensus RMSD filtering; RelExpt = Experimental RMSD filtering. The values, 101, 100, and 3, represent RMSD cutoffs in Å. Pearson ≥ 0.775 and MSE < 10.000 in bold italic.

Figure 5.

Figure 5

Performances of meta-models. Predictions by XGBoost for the unfiltered training set are shown. The first row represents the meta-models constructed entirely based on the docking methods, SMINA and Vinardo. The first panel shows results for the two scores alone (E), while the second panel shows the results also including the molecular weight parameter (EW). The remaining rows show results for meta-models that include DL scores. The second, third and fourth rows represent successive processing of the deep learning model scores, with the second row showing the average scores (ED1, ED2, ED3), the third row (if a panel is present) showing the average after fine-tuning (F) using the RefinedSet\CoreSet (ED1-F, ED2-F), and the fourth row showing the scores using the optimal number of PCs from a PCA (P) on the previous level (either fine-tuned or not) (ED1–F-P, ED2–F-P, ED3-P). The columns of the second, third and fourth rows show the three different base training sets used for the DL scores, where the leftmost column is based on the pretrained models on BindingDB (ED1 series), the central column is based on the de novo-trained models on BindingDB (ED2 series) and the rightmost column is based on the de novo-trained models on PDBbind (ED3 series). The single panel in the last fifth row shows the results for the meta-model ED-A-P, which is based on a PCA on all the training scores from ED1-F, ED2-F, and ED3.

We first note that the docking results alone (E) produce a moderate Pearson correlation for the test set (0.527). Unexpectedly, the E correlation for the training set (0.487) is lower than that for the test set. This might be due to the more careful curation and filtering of complexes in the CoreSet, relative to the broader RefinedSet\CoreSet. Moreover, adding in the molecular weight of the ligands (EW) showed significant improvements for both training and test sets. The original motivation for including the MW was the fact that the absolute deviation between the SMINA and Vinardo predictions and the experimental binding affinity is relatively highly correlated (R ∼ 0.3) with the MW and a few related ADME features (such as the number of heavy atoms and molecular refractivity) than others (Table S7). Including MW was an attempt at addressing this bias in the docking scores. We also noticed that the MW was correlated with the absolute deviation in the scores for some of the deep learning features, including the top principal component. When we added molecular weight as a meta-model feature (in all the meta-models except for E), we found that the deviations of the final predictions are much less correlated with molecular weight (Table S8).

In the next rows, it can be seen that the meta-models including DL scores significantly increase the Pearson correlation relative to the docking-based meta-models. The fine-tuning of the BindingDB-trained models on the PDBbind-RefinedSet\CoreSet improves the prediction performance. PCA on any of the DL scores improves performance even more. We note here that D3 was trained on the RefinedSet\CoreSet, while the D1F and D2F models were fine-tuned on the same set. This means that the training set for the DL training/fine-tuning process is identical to those for the associated meta-models (ED1-F, ED1-F-P, ED2-F, ED2-F-P, ED3, ED3-P, and ED-A-P). This also means that the leave-out validation split for the PC number optimization is not an independent dataset. The consequence of the most overlap between the DL and meta-model training sets is that we observe a very high correlation in the training set for those meta-models. Nonetheless, the high performances of these models on the external validation (CoreSet) signal good overall model robustness. The only caveat to this conclusion would stem from any systematic bias of the PDBbind dataset as a whole: if all the complexes in the full dataset were biased, then the use of a common training set would enable the models to learn the bias in the CoreSet as well, artificially inflating the performance. We are, however, not aware of any such bias in the PDBbind dataset, given that the complexes are sourced from a large number of independent studies.

Beneficial effects observed in the meta-models by performing PCA on the DL outputs are possibly due to the denoising or debiasing of outputs. The denoising is achieved by first running multiple parallel cross-validations, and then combining them into a lower-dimensional representation. However, unlike the DL results in Figure 3, the performance improvement in the meta-models is not observed with the first PC alone, but rather requires the combination of a few PCs. We also note that, among the 4 PCA-based models (ED1-F-P, ED2-F-P, ED3-P, and ED-A-P), there are no large differences in the performance, i.e., different meta-model algorithms for PCA on different DL training processes show similar performances (PCC = 0.76–0.78). This suggests that we have maximized the information content of the constituent docking and DL model predictions.

To understand the contributions of the component features, we extracted the feature importance scores for each meta-model (Figure S7). We find that, while the contributions of the docking scores dominate for the EW, ED1, and ED2, these contributions drop to very low values for the other DL-containing meta-models. In all meta-models, SMINA scores contribute more than Vinardo scores. However, given the aforementioned similarity between the SMINA and Vinardo scores, this difference in the contributions between the two docking scores is more likely to be an artifact of the modeling process (and how it deals with nearly collinear features). The other noteworthy aspect of the importance scores is that the performance of ED3, fine-tuned, and PC-based meta-models is dominated by one, or at most two significant features. ED1-F, ED2-F and ED3 are impacted mainly by two DL scores, while the PC models are dominated by PC1. We observe again that PC1 alone is not sufficient to improve the predictions, but requires the assistance of other PCs. The patterns of feature importance scores, however, are not entirely consistent across the different ML meta-models. The top features of the LASSO and Linear Regression meta-models are distinct from the XGBoost top features. The signed feature importance scores in the linear models show less domination of single scores, especially in the PC-based models. In particular, PC1, which is individually highly correlated with experimental binding affinities, is less important in the predictions by the linear meta-models. Given the similar performances of the ML meta-models, it appears that there are several paths to attaining nearly equivalent overall correlation values.

Given the improvement in the docking predictions by filtering out certain ligand–protein pairs (Figures 2 and S1), we explored the effect of the Experimental and Consensus RMSD filters with 3 and 100 Å cutoffs on the meta-models (Tables S4 and S5). We found that the E and EW meta-models are affected by the different filters, with the Experimental RMSD filters producing slightly better results than the Consensus filters. This reinforces the idea that the docking affinity predictions benefit from constraining the predictions based on the pose. However, as the influence of the docking predictions decreases in the meta-models, it appears as if the impact of the RMSD filters also goes down and the best-performing meta-models show negligible difference in their performance with the various RMSD filters.

Prediction Synergy and Complementarity of Docking and DL Tools

As we obtained better overall performance by meta-modeling of docking and DL tools, we further investigated the degree of prediction synergy and complementarity of the two groups of tools, i.e., whether the meta-models utilize the strengths of both tool groups and the degree to which the two provide nonoverlapping information. To do this, we focused on absolute errors of the mean predictions for the benchmark set by the 4 meta-models with ED-A-P, by the 4 docking tools, and by the DAP|PC1 model (Figure 4D). When mutually compared, the mean predictions by the meta-models achieved the lowest errors for 123 of the 266 complexes (46.2%), the DAP|PC1 for 104 complexes (39.1%), and the docking tools for 39 complexes (14.7%) (Figure 6A–C). If we only compare the docking and DAP|PC1 tools, the former showed better mean predictions for 48 of the 266 complexes (18.0%) than the latter (Figure 6D). This suggests that, beyond the globally better performance of the meta-models, prediction synergy is realized for 46.2% of the benchmark set by our meta-modeling due to complementary information of the docking and DL tools (even though each group of tools is better at predicting a subset of complexes by itself). Similarly, although the DL tools are generally better than the docking tools, there exists a small subset of complexes that are better predicted by the latter.

Figure 6.

Figure 6

Prediction synergy and complementarity of docking and DL tools by meta-modeling. (A) Absolute errors (AEs) of the mean predictions for the CASF-2016 benchmark set (266 complexes) by the 4 meta-models with ED-A-P (open circles; ordered by AEs). The blue circles correspond to 104 complexes (39.1%) whose predictions by the DL tools (DAP|PC1 in Figure 4D) are better than the mean predictions by the 4 meta-models or the 4 docking tools (i.e., lower AEs). The red circles correspond to 39 complexes (14.7%) whose mean predictions by the 4 docking tools are better than those by the meta-models or the DAP|PC1. (B-D) Scatter plots of AEs of the (mean) predictions for the CASF-2016 benchmark set by the DAP|PC1 vs the 4 meta-models (B), by the 4 docking tools vs the 4 meta-models (C), and by the 4 docking tools vs the DAP|PC1 (D). The blue and red dots in (B) and (C) correspond to those in (A). In (D), the 48 complexes (18.0%) in red indicate better predictions (i.e., lower AEs) by the docking tools on average, whereas the remaining 218 complexes (82.0%) in blue indicate better predictions by the DAP|PC1.

To further parse the differences between the outputs of the prediction tools, we assessed whether certain properties of the proteins or ligands are over- or under-represented in the complexes which were: (1) Best modeled by empirical docking; (2) Best modeled by deep learning; (3) Best modeled by meta-modeling. We also explored the top 50 complexes as predicted by the XGBoost Meta-model for ED-A-P to see if the performance of the meta-models relative to the experimental data was contingent on protein or ligand properties. We extracted protein features from the UniProt database and ligand features from the SwissADME database. We tested the significance of the occurrence of certain annotations (using two-sided Fisher’s exact test) and of the distributions of certain quantitative features (using a two-sided Wilcoxon test) for each of the four subsets of complexes. Specifically, we considered whether several UniProt features were annotated more or less frequently, and whether certain Gene Ontology (GO),94 domain and pathway annotations occurred with different probabilities in the subsets. We also compared the distributions of the ligand SwissADME features. The results (Table S9) indicate that there are practically no significant (nominal p value ≤0.05) protein features prioritized in the four cases considered. This may be due to the fact that several proteins (and their associated UniProt IDs) are duplicated in the CoreSet. The number of reviewed proteins with UniProtIDs drops from 243 to 60 after removal of all duplicates. The highly significant results with nominal p values ≤0.01 (both Wilcoxon and t tests) are the distributions of several ligand features including MW, lipophilicity, and solubility measures in the structures for which the empirical docking tools perform the best. There is also some hint of significance for MW, the “#Rotatable bonds” and “#H-bond acceptors” in the structures for which the meta-models perform the best.

Comparison with Structure-Based Tools

Having built our meta-models based on sequence-based DL models, we were interested in comparisons with structure-based tools through voxelization of binding pockets. As described in Methods, we focused on the following 3 DL tools by retraining them with our training data for fair comparisons as much as possible, as well as one recent tool based on advanced mathematics. HAC-Net47 was trained on the PDBbind-2020 general set of >18,800 ligand–protein complexes. Their best performance was PCC = 0.846 for the PDBbind-2016 CoreSet of 290 complexes. We note that we focused on the PDBbind-2020 RefinedSet rather than the general set because of two reasons: (1) higher quality and (2) limited resources. The retrained HAC-Net with our training set showed PCC = 0.756 for the CoreSet we used, which is worse than our best meta-model performance of PCC = 0.777 or our average DL performance of PCC = 0.763 (Table 4). Although their RMSE value is better than ours, this performance metric is not a fair metric for comparison in this study because they used −log10(Kd/Ki) (also known as pKd/Ki) while we used ln(Kd/Ki), which show different distributions of binding affinities for model training. We note that the HAC-Net performance based on ln(Kd/Ki) notably degraded to PCC = 0.152 and RMSE = 18.85. FAST42 was trained on the PDBbind-2016 general set excluding the CoreSet. We tested the pretrained SG-CNN of FAST against the CoreSet we used, which resulted in PCC = 0.716, worse than HAC-Net (Table 4). KDeep46 was one of the earlier structure-based 3D-CNN models. Their reported Pearson correlation on the PDBbind-2016 CoreSet was 0.82, which we were not able to verify because their code is not open-source. With 3 retrained KDeep models using 3 limited training sets from KdeepTrainer (see Methods), we obtained PCC = 0.718, 0.730, and 0.754 (mean = 0.734) for the CoreSet we used (Table 4), which are similar to those reported values of 0.701–0.738 by Kwon et al. using the PDBbind-2016 RefinedSet as a training set and 4 different learning rates.50 PerSpect ML69 recently reported the highest Pearson correlation of 0.84 in the case of the PDBbind 2016 CoreSet. While we were not able to confirm their result due to incomplete code availability, those recent methods based on advanced mathematics from the Xia group67−70 showed largely better performances than ours and others. We note, however, that they may generate more than 100,000 features requiring 3D structures or binding poses, which are often unavailable in real-world drug discovery. Despite these limited comparisons, we conclude that our meta-models and our sequence-based DL models show competitive performances or advantages to more sophisticated structure-based models.

Table 4. Performance Comparisons with Structure-based Toolsa.

        CASF-2016 Benchmark
GeneralSet Benchmark
Models Data Type Prediction Unit Training set PCC RMSE PCC RMSE
E (docking) Structures ln(Kd/Ki) PDBbind2016 RefinedSet\CoreSet 0.530 7.68 0.349 6.35
D (deep learning) Sequences ln(Kd/Ki) BindingDB2022 and PDBbind2020 RefinedSet\CoreSet 0.763 3.56 0.591 3.41
Meta-models Sequences or Structures ln(Kd/Ki) 0.777 3.16 0.598 3.35
HAC-Net Structures –log10(Kd/Ki) PDBbind2020 RefinedSet\CoreSet 0.756 1.45 0.008 1150.70
FAST (SG-CNN) Structures –log10(Kd/Ki) PDBbind2016 GeneralSet 0.716 1.53 0.526 1.59
KDeepb Structures –log10(Kd/Ki) Customized or N/A 0.734 3.40 0.318 6.01
a

PCC = Pearson correlation coefficient. RMSE = Root Mean Square Error.

b

Retraining for the CASF-2016 benchmark was done using 3 customized sets of PDBbind2020 RefinedSet\CoreSet. The GeneralSet benchmark was based on the default KDeep model of ACEBind-2023 Graph-Net where the training set is not available (see the main text). Our models are in bold italic. The best performances in terms of PCC are in bold. Kd/Ki implies Kd or Ki.

GeneralSet Benchmark

In addition to the CASF-2016 benchmark, we also tested our meta-models against the PBDbind2020 GeneralSet of 5,261 complexes (Methods; Table 4). Although the size of the GeneralSet benchmark is as large as our training data, we observe performance improvements of our meta-models over the individual base models (Figure S8). As expected, the performance for this benchmark is worse than that for CASF-2016 (PCC < 0.60 vs >0.76) because of its lower data quality and the data size being similar to the training data. As for the structure-based tools mentioned above, the Pearson correlation by the retrained HAC-Net was R = 0.008 (excluding 22 complexes that failed in preprocessing) and that by KDeep (the Web server default model of ACEBind2023-GraphNet as of Sept. 2, 2024) was R = 0.318 (excluding 71 complexes that failed in preprocessing), which performed worse than our meta-models or DL models and similar to the docking tools. We note that the huge RMSE by HAC-Net is due to poor performances of the 3D-CNN component of HAC-Net. A similarly poor performance was observed by the published pretrained HAC-Net with the PDBbind2020 general set (R = 0.009 and RMSE = 1138.51). The PDBbind2016-GeneralSet pretrained SG-CNN of FAST achieved R = 0.526 (excluding 124 complexes that failed in preprocessing), which is also a worse performance than our meta-models and DL models. Although their RMSE is better than ours, it is not a fair metric for comparison as explained in the previous section. Also, we note that 3314 out of 5261 complexes in our benchmark set are part of the training set of PDBbind2016 general set by FAST.

Ranking Benchmark

To further explore the performance of the models on specific target proteins with multiple ligands, we designed a binding affinity ranking benchmark against 3 target proteins for 35 complexes from the PBDbind v2020 general set: O60341, O15151, and P0C6U8 (see Methods). The performances of our models for the 3 target proteins are shown in Table 5 in comparison with the 3 tools described above, HAC-Net (retrained), SG-CNN (FAST; pretrained), and KDeep (the up-to-date default version on their web server). The pretrained SG-CNN performed best for O60341 and O15151 in terms of RMSE (2.10 and 1.37, respectively). This was expected because their training data were the PDBbind v2016 general set, which includes 30 out of the 35 complexes in our benchmark set except 5 complexes for O60341. Otherwise, our meta-models or fine-tuned DL models performed best for all 3 target proteins in terms of both RMSE and Pearson’s correlation. Our best performing models are ED2-F_VvS_101.0_LinReg (meta-model; docking + MW + de novo-trained and fined-tuned DL by linear regression) and D1F|Daylight_AAC (pretrained and fine-tuned DL model) for O60341 (RMSE = 2.34), EW_VvS_101.0_LinReg (meta-model; docking + MW by linear regression) for O15151 (RMSE = 1.44), and D2F|MPNN-AAC (de novo-trained and fine-tuned DL model) for P0C6U8 (RMSE = 1.01). We consider the Pearson correlation metric less reliable due to the small sample size.

Table 5. Performance Comparisons of a Ranking Benchmarka.

  RMSE PCC
Target protein (UniProt ID) O60341 O15151 P0C6U8 O60341 O15151 P0C6U8
Number of ligands 20 8 7 20 8 7
E (docking) 5.04 10.45 5.82 –0.18 –0.07 0.57
D (deep learning) 2.34 2.20 1.01 0.72 0.59 0.94
Meta-models 2.34 1.44 2.12 0.64 0.95 0.92
HAC-Net 5.13 7.16 5.58 0.33 0.23 0.80
FAST (SG-CNN)b 2.10 1.37 1.80 –0.38 0.25 –0.59
KDeep 4.99 3.68 6.01 –0.17 0.60 0.14
a

The best performance value for each target protein is in bold, as well as performance values for our models (in bold italic). RMSE and PCC are defined as in Table 4.

b

Pretrained on PDBbind v2016 GeneralSet.

As noted in the previous section on prediction synergy, different meta-models or component DL/docking models may perform best for different complexes or proteins. In the case of this virtual screening, we noticed that while some of the same meta-models would often predict the binding affinities for O60341 and P0C6U8 with reasonable accuracy, they would not predict the affinities for O15151 as well, and vice versa. For example, ED2-F_VvS_101.0_LinReg showed low RMSEs for O60341 (RMSE = 2.34) and P0C6U8 (RMSE = 2.54) but higher RMSEs for O15151 (RMSE = 5.88). In particular, we observe that 6 out of the 8 ligands in the complexes for O15151 were k-mers (k = 10, 12, or 15) with relatively high MW of >1000. Among the 35 complexes there are 18 k-mers with mean MW of 927.2 compared to 762.7 for 17 non-k-mers. We note that about 5% of the RefinedSet (we used for model training) and about 10% of the general set contain k-mers (up to 10-mer and 19-mer, respectively). Given that k-mers are not representative and tend to have high MW, we aimed to assess whether high or low MW causes differences in the performance of the meta-models independently from the choice of a target protein by dividing ligands into two groups of low MW ≤ 900 and high MW > 900 (the median threshold for the 35 complexes). By focusing on ED2-F_VvS_101.0_LinReg from above, we find that the accuracy for low MW ligands is better than high MW ligands (RMSE = 2.45 vs 4.36). This is expected given the fact that the ligands in the RefinedSet we used for training have mean MW of 384.3 and standard deviation of 172.6 and that the Pearson correlations between predictions and MW are negative ranging from −0.62 to −0.29. While far from conclusive for generalization, these results suggest that there might be identifiable features of ligands and/or proteins which would explain why certain meta-models perform variably for different target proteins.

Application to Virtual Screening

As the above results suggest that top-ranking high-affinity ligands predicted by our models could be prioritized as hits in virtual screening, we also tested our models using the virtual screening benchmark dataset of LIT-PCBA to distinguish active ligands from inactive ligands for 15 proteins (see Methods; Table S10). As our models were not trained for binary classification of active and inactive ligands, our hypothesis was that active ligands tend to have higher affinity scores than inactive ligands. To test the hypothesis, we evaluated our models and the 2 docking tools using 3 performance metrics (top5 recall, top10 recall, and precision) for each protein (Methods; Table 6) and the distribution difference of predicted affinity scores between all active and all inactive ligands for each protein (one-sided Welch’s t test and the Mann–Whitney U test; Table S11). The 3 performance metrics show that our averaged DL models and meta-models tend to be better at predicting active ligands than the 2 docking tools (i.e., > 50% in blue in Table 6). The docking tools performed better for 3 proteins, but their performance values are mostly 20% or less otherwise (in red in Table 6). As expected, active ligands tend to have higher affinity scores than inactive ligands for most of the proteins individually, and when aggregated across all proteins (Table S11). We emphasize that, as in the section on prediction synergy, there is variability among the target proteins in terms of the best-performing model. Different protein targets may benefit from the use of different modeling techniques, conditional on their physicochemical properties. This is also observed in terms of the target-specific behavior of the p values (Table S11). All those structure-based models mentioned above, HAC-Net, FAST, KDeep, and the models from the Xia group (e.g., PerSpect ML), require input data of docked poses and hence cannot be used in the LIT-PCBA virtual screening benchmark or are not generally applicable to identification of active vs inactive ligands.

Table 6. Enrichment of Active Ligands in the Top Predicted Ligands for LIT-PCBA Virtual Screeninga.

graphic file with name ci4c01116_0012.jpg

a

≥0.50 in blue; ≤0.20 in red.

Overall, we conclude that families of our models are competitive to several state-of-the-art structure-based models and also useful for virtual screening applications.

Discussion

The framework we present here, inspired by stacked generalization and super learner,74,75 is based on the central concept that different models designed and trained for the same prediction task often carry model-specific biases. Hence, simple approaches to combining the results of their independent predictions may yield superior results if the biases are even partially independent and the models are complementary to some degree. Indeed, we find this to be borne out in the ligand–protein binding affinity prediction in several ways. First, we find that combining the predictions from empirical docking methods with those from the pretrained or de novo-trained deep learning tools can yield gains in predictive power over the individual approaches (ED1–3 vs E or D1–2). Second, utilizing models trained on the BindingDB as the base and refining them on the PDBbind resulted in significant improvements (D1 and D2 vs D1F and D2F, respectively), demonstrating that fine-tuning pretrained models could retain good predictive quality on both primary (BindingDB) and secondary (PDBbind) datasets (Figure S4). Third, even within the suite of deep learning tools, transforming their prediction scores through principal component analysis could improve the performance on the test set (i.e., principal component regression). This may be due to a denoising of individual DL scores. Finally, in addition to performance improvement or synergy by the meta-models over the docking and DL models for a subset of complexes, the use of meta-models by canonical machine learning enables us to explicitly add features that may be associated with prediction deviations from the experimental affinities in the individual models. For example, we found that, for both empirical docking tools and some deep learning models, the deviations of the predictions from the experimental affinities were correlated with molecular weight. Explicitly adding in molecular weight as a meta-model feature resulted in the deviations of the final predictions being much less correlated with molecular weight. Moreover, our DL models and meta-models demonstrated their competitive power in comparison to other recently developed structure-based models in a large-scale affinity prediction benchmark and a virtual screening application benchmark.

Overall, our framework corroborates existing work on ensemble learning, stacked generalization, super learner, and consensus-based improvements in predictions. Our machine-learning-based meta-models allow for the flexibility of adding or removing any component tools, physicochemical properties, or molecular descriptors as part of feature engineering. Models trained on multiple datasets (e.g., BindingDB and PDBbind) and on multiple data types (e.g., sequence and/or structure) may be unified to reduce individual biases and improve predictions. Furthermore, we posit that the DL models from DeepPurpose considered herein, being sequence-based, offer better scalability. Given that only ligand and protein sequences are required for the prediction, and not all-atom 3D structures, we expect that increasingly large databases could be used for training in future iterations by running high-throughput experimental binding affinity assays (without the need for concurrent X-ray or NMR experiments) as a filtering strategy in early drug discovery. Considering that the observed performances of these models for all our benchmarks are better or on par with more sophisticated structure-based DL models, there is an argument to be made in favor of more scalable approaches. Additionally, we note that the public and private experimental binding affinity databases are highly fragmented, in that they are produced separately, with potentially different types of assays, or at different scales. While efforts such as the PDBbind database manage to carefully curate a high-quality single dataset, not all data sources can be incorporated because of the type of data (e.g., experimental data without associated structures being available) or because of data access (e.g., behind a private paywall). Our meta-modeling may address some of these issues by pooling only prediction outputs from models trained on diverse public or private databases with or without available structures, protecting the databases themselves if necessary and maintaining performance quality. Subsequent in-depth analysis could also identify prediction synergy and complementarity of all available tools as we demonstrated in Figure 6. Finally, there might be value in expanding families of meta-models with different input data, each family being optimized to target a particular protein or class of proteins and/or ligands as demonstrated in Tables 5 and 6.

We acknowledge some limitations of our study. First, we primarily focused on a single dataset, PDBbind RefinedSet, for model training. In fact, in light of recent positive results from training models on the full PDBbind general set of ∼19,000 complexes,42,47 there is a scope for further scale-up of the model training process with larger data. It is worth mentioning, however, that when Jones et al. used the PDBbind general or refined set for model training, they observed the trade-offs between data quality and quantity in performance of different models: SG-CNN performed better with the general set (in favor of data quantity); 3D-CNN performed better with the refined set (in favor of data quality). On the other hand, we started with DL models pretrained on BindingDB of ∼66,000 complexes and either employed them “out of the box” or fine-tuned them on the PDBbind RefinedSet (excluding the CoreSet). The two BindingDB-based meta-models, ED1 and ED2, perform better than any of the stand-alone docking or DL tools (PCC = 0.62−0.64 vs 0.47−0.59). We also tested the two PDBbind fine-tuned models, D1F and D2F, on the primary training set of BindingDB, to ascertain the degree to which the models retained their “out of the box” performance on the primary training set. We found reasonably good correlations for D1F and D2F (R = 0.636 to 0.711) (Figure S4) compared to the pretrained D1 (R = 0.800).57 However, the PDBbind-trained D3 showed poor prediction on BindingDB (R < 0.200). More importantly, while the CASF-2016 benchmark set has been well accepted, more benchmark sets of high quality would be needed to test and compare models in a more objective manner and also to challenge model generalization. Our second benchmark using the large-scale GeneralSet, although lower quality than CASF-2016, was such an effort to assess model generalization. Our virtual screening benchmark is also very limited, although our models outperformed the docking tools and tend to perform well for ligands with low molecular weight. In particular, the binary classification problem for screening of active or functional ligands would require a new model development for higher accuracy rather than repurposing exclusively affinity-based regression models as in this study. In this sense, the good performances of our fine-tuned DL models or meta-models do not imply model generalizability in an absolute sense. We also note that we neither fine-tuned the docking tools nor conducted hyperparameter optimization for the DL models, which might help performance as well.

Our proof-of-concept study of meta-modeling can be readily extended to integrate with (i.e., to add more meta-features of) predictions from any other tools, such as HAC-Net, FAST, KDeep, and/or PerSpect ML, molecular properties or embeddings, or even predictions from the same base models trained on different orthogonal datasets, to address model bias and robustness, which is the key conceptual advantage of any ensemble or fusion model. In parallel to this type of horizontal extension (i.e., feature engineering), it is conceivable that meta-models may be expanded or stacked vertically, leading to deep meta-models. Our meta-modeling approach offers promising and flexible strategies for future model iterations to further improve ligand–protein binding affinity prediction.

Data and Software Availability

The BindingDB and PDBbind databases are publicly available. The code and our pre-trained models are partially available, due to our pending patent, on our GitHub page at https://github.com/Lee1701/Lee2023a.

Acknowledgments

This work was in part supported by the National Institutes of Health under the award R01DA051906-04 and the Albert L. Williams award to MBG. We thank the Yale Center for Research Computing for their high-performance-computing resources and help. We also thank Drs. Jeffrey Brock and Sarah Miller for their support in the initiation of this project through the COVID HASTE community at the Yale School of Engineering & Applied Science.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.jcim.4c01116.

  • Supplementary methods with additional details on models and datasets, including meta-model parameters, and Figures S1–S8 (PDF)

  • Tables S1–S11 (XLSX)

Author Contributions

‡ H.-J.L. and P.S.E. contributed equally. H.-J.L. and P.S.E. conceived, designed, and performed the study, analyzed the data, and drafted the manuscript. H.-J.L., P.S.E., and M.G.B. interpreted the data and critically reviewed and approved the manuscript. H.-J.L. supervised the study.

The authors declare the following competing financial interest(s): H.-J.L. reports a consulting role at Guidepoint outside of this submitted work.

Supplementary Material

ci4c01116_si_001.pdf (1.7MB, pdf)
ci4c01116_si_002.xlsx (13.4MB, xlsx)

References

  1. Shen C.; Ding J.; Wang Z.; Cao D.; Ding X.; Hou T. From machine learning to deep learning: Advances in scoring functions for protein–ligand docking. Wiley Interdiscip. Rev.: Comput. Mol. Sci. 2020, 10, e1429 10.1002/wcms.1429. [DOI] [Google Scholar]
  2. Ain Q. U.; Aleksandrova A.; Roessler F. D.; Ballester P. J. Machine-learning scoring functions to improve structure-based binding affinity prediction and virtual screening. Wiley Interdiscip. Rev.: Comput. Mol. Sci. 2015, 5, 405–424. 10.1002/wcms.1225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Khamis M. A.; Gomaa W.; Ahmed W. F. Machine learning in computational docking. Artif. Intell. Med. 2015, 63, 135–52. 10.1016/j.artmed.2015.02.002. [DOI] [PubMed] [Google Scholar]
  4. Kitchen D. B.; Decornez H.; Furr J. R.; Bajorath J. Docking and scoring in virtual screening for drug discovery: methods and applications. Nat. Rev. Drug Discovery 2004, 3, 935–49. 10.1038/nrd1549. [DOI] [PubMed] [Google Scholar]
  5. Moitessier N.; Englebienne P.; Lee D.; Lawandi J.; Corbeil C. R. Towards the development of universal, fast and highly accurate docking/scoring methods: a long way to go. Br. J. Pharmacol. 2008, 153 (S1), S7–S26. 10.1038/sj.bjp.0707515. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Grinter S. Z.; Zou X. Challenges, applications, and recent advances of protein-ligand docking in structure-based drug design. Molecules 2014, 19, 10150–76. 10.3390/molecules190710150. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Su M.; Yang Q.; Du Y.; Feng G.; Liu Z.; Li Y.; Wang R. Comparative Assessment of Scoring Functions: The CASF-2016 Update. J. Chem. Inf. Model. 2019, 59, 895–913. 10.1021/acs.jcim.8b00545. [DOI] [PubMed] [Google Scholar]
  8. Ferreira L. G.; Dos Santos R. N.; Oliva G.; Andricopulo A. D. Molecular docking and structure-based drug design strategies. Molecules 2015, 20, 13384–421. 10.3390/molecules200713384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Sousa S. F.; Fernandes P. A.; Ramos M. J. Protein-ligand docking: current status and future challenges. Proteins 2006, 65, 15–26. 10.1002/prot.21082. [DOI] [PubMed] [Google Scholar]
  10. Krovat M. E.; Steindl T.; Langer T. Recent Advances in Docking and Scoring. Curr. Comput.-Aided Drug Des. 2005, 1, 93–102. 10.2174/1573409052952314. [DOI] [Google Scholar]
  11. Ballester P. J.; Mitchell J. B. O. A machine learning approach to predicting protein–ligand binding affinity with applications to molecular docking. Bioinformatics 2010, 26, 1169–1175. 10.1093/bioinformatics/btq112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Böhm H. J. The development of a simple empirical scoring function to estimate the binding constant for a protein-ligand complex of known three-dimensional structure. J. Comput.-Aided Mol. Des. 1994, 8, 243–56. 10.1007/BF00126743. [DOI] [PubMed] [Google Scholar]
  13. Ruvinsky A. M. Role of binding entropy in the refinement of protein-ligand docking predictions: analysis based on the use of 11 scoring functions. J. Comput. Chem. 2007, 28, 1364–72. 10.1002/jcc.20580. [DOI] [PubMed] [Google Scholar]
  14. Mysinger M. M.; Shoichet B. K. Rapid context-dependent ligand desolvation in molecular docking. J. Chem. Inf. Model. 2010, 50, 1561–73. 10.1021/ci100214a. [DOI] [PubMed] [Google Scholar]
  15. Lill M. A. Efficient incorporation of protein flexibility and dynamics into molecular docking simulations. Biochemistry 2011, 50, 6157–69. 10.1021/bi2004558. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Cheng T.; Li X.; Li Y.; Liu Z.; Wang R. Comparative Assessment of Scoring Functions on a Diverse Test Set. J. Chem. Inf. Model. 2009, 49, 1079–1093. 10.1021/ci9000053. [DOI] [PubMed] [Google Scholar]
  17. Wang R.; Lu Y.; Fang X.; Wang S. An Extensive Test of 14 Scoring Functions Using the PDBbind Refined Set of 800 Protein–Ligand Complexes. J. Chem. Inf. Comput. Sci. 2004, 44, 2114–2125. 10.1021/ci049733j. [DOI] [PubMed] [Google Scholar]
  18. Warren G. L.; Andrews C. W.; Capelli A. M.; Clarke B.; LaLonde J.; Lambert M. H.; Lindvall M.; Nevins N.; Semus S. F.; Senger S.; Tedesco G.; Wall I. D.; Woolven J. M.; Peishoff C. E.; Head M. S. A critical assessment of docking programs and scoring functions. J. Med. Chem. 2006, 49, 5912–31. 10.1021/jm050362n. [DOI] [PubMed] [Google Scholar]
  19. Trott O.; Olson A. J. AutoDock Vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. J. Comput. Chem. 2010, 31, 455–61. 10.1002/jcc.21334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Koes D. R.; Baumgartner M. P.; Camacho C. J. Lessons learned in empirical scoring with smina from the CSAR 2011 benchmarking exercise. J. Chem. Inf Model 2013, 53, 1893–904. 10.1021/ci300604z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Wang R.; Wang S. How does consensus scoring work for virtual library screening? An idealized computer experiment. J. Chem. Inf. Comput. Sci. 2001, 41, 1422–6. 10.1021/ci010025x. [DOI] [PubMed] [Google Scholar]
  22. Charifson P. S.; Corkery J. J.; Murcko M. A.; Walters W. P. Consensus scoring: A method for obtaining improved hit rates from docking databases of three-dimensional structures into proteins. J. Med. Chem. 1999, 42, 5100–9. 10.1021/jm990352k. [DOI] [PubMed] [Google Scholar]
  23. Houston D. R.; Walkinshaw M. D. Consensus Docking: Improving the Reliability of Docking in a Virtual Screening Context. J. Chem. Inf. Model. 2013, 53, 384–390. 10.1021/ci300399w. [DOI] [PubMed] [Google Scholar]
  24. Palacio-Rodríguez K.; Lans I.; Cavasotto C. N.; Cossio P. Exponential consensus ranking improves the outcome in docking and receptor ensemble docking. Sci. Rep. 2019, 9, 5142. 10.1038/s41598-019-41594-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Liu Z.; Su M.; Han L.; Liu J.; Yang Q.; Li Y.; Wang R. Forging the Basis for Developing Protein–Ligand Interaction Scoring Functions. Acc. Chem. Res. 2017, 50, 302–309. 10.1021/acs.accounts.6b00491. [DOI] [PubMed] [Google Scholar]
  26. Ashtawy H. M.; Mahapatra N. R. A Comparative Assessment of Ranking Accuracies of Conventional and Machine-Learning-Based Scoring Functions for Protein-Ligand Binding Affinity Prediction. IEEE/ACM Trans. Comput. Biol. Bioinf. 2012, 9, 1301–1313. 10.1109/TCBB.2012.36. [DOI] [PubMed] [Google Scholar]
  27. Patel L.; Shukla T.; Huang X.; Ussery D. W.; Wang S. Machine Learning Methods in Drug Discovery. Molecules 2020, 25, 5277 10.3390/molecules25225277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Wójcikowski M.; Ballester P. J.; Siedlecki P. Performance of machine-learning scoring functions in structure-based virtual screening. Sci. Rep. 2017, 7, 46710. 10.1038/srep46710. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Baum B.; Muley L.; Smolinski M.; Heine A.; Hangauer D.; Klebe G. Non-additivity of functional group contributions in protein-ligand binding: a comprehensive study by crystallography and isothermal titration calorimetry. J. Mol. Biol. 2010, 397, 1042–54. 10.1016/j.jmb.2010.02.007. [DOI] [PubMed] [Google Scholar]
  30. Baskin I. I.; Winkler D.; Tetko I. V. A renaissance of neural networks in drug discovery. Expert Opin. Drug Discovery 2016, 11, 785–95. 10.1080/17460441.2016.1201262. [DOI] [PubMed] [Google Scholar]
  31. Dana D.; Gadhiya S. V.; St. Surin L. G.; Li D.; Naaz F.; Ali Q.; Paka L.; Yamin M. A.; Narayan M.; Goldberg I. D.; Narayan P. Deep Learning in Drug Discovery and Medicine; Scratching the Surface. Molecules 2018, 23, 2384 10.3390/molecules23092384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Sapoval N.; Aghazadeh A.; Nute M. G.; Antunes D. A.; Balaji A.; Baraniuk R.; Barberan C. J.; Dannenfelser R.; Dun C.; Edrisi M.; Elworth R. A. L.; Kille B.; Kyrillidis A.; Nakhleh L.; Wolfe C. R.; Yan Z.; Yao V.; Treangen T. J. Current progress and open challenges for applying deep learning across the biosciences. Nat. Commun. 2022, 13, 1728. 10.1038/s41467-022-29268-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Yang Z.; Zeng X.; Zhao Y.; Chen R. AlphaFold2 and its applications in the fields of biology and medicine. Signal Transduction Targeted Ther. 2023, 8, 115. 10.1038/s41392-023-01381-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Akdel M.; Pires D. E. V.; Pardo E. P.; Jänes J.; Zalevsky A. O.; Mészáros B.; Bryant P.; Good L. L.; Laskowski R. A.; Pozzati G.; Shenoy A.; Zhu W.; Kundrotas P.; Serra V. R.; Rodrigues C. H. M.; Dunham A. S.; Burke D.; Borkakoti N.; Velankar S.; Frost A.; Basquin J.; Lindorff-Larsen K.; Bateman A.; Kajava A. V.; Valencia A.; Ovchinnikov S.; Durairaj J.; Ascher D. B.; Thornton J. M.; Davey N. E.; Stein A.; Elofsson A.; Croll T. I.; Beltrao P. A structural biology community assessment of AlphaFold2 applications. Nat. Struct. Mol. Biol. 2022, 29, 1056–1067. 10.1038/s41594-022-00849-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Abbasi K.; Razzaghi P.; Poso A.; Ghanbari-Ara S.; Masoudi-Nejad A. Deep Learning in Drug Target Interaction Prediction: Current and Future Perspectives. Curr. Med. Chem. 2021, 28, 2100–2113. 10.2174/0929867327666200907141016. [DOI] [PubMed] [Google Scholar]
  36. Li H.; Zou L.; Kowah J. A. H.; He D.; Liu Z.; Ding X.; Wen H.; Wang L.; Yuan M.; Liu X. A compact review of progress and prospects of deep learning in drug discovery. J. Mol. Model. 2023, 29, 117. 10.1007/s00894-023-05492-w. [DOI] [PubMed] [Google Scholar]
  37. Kim J.; Park S.; Min D.; Kim W. Comprehensive Survey of Recent Drug Discovery Using Deep Learning. Int. J. Mol. Sci. 2021, 22, 9983 10.3390/ijms22189983. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lavecchia A. Deep learning in drug discovery: opportunities, challenges and future prospects. Drug Discovery Today 2019, 24, 2017–2032. 10.1016/j.drudis.2019.07.006. [DOI] [PubMed] [Google Scholar]
  39. Ma J.; Sheridan R. P.; Liaw A.; Dahl G. E.; Svetnik V. Deep Neural Nets as a Method for Quantitative Structure–Activity Relationships. J. Chem. Inf. Model. 2015, 55, 263–274. 10.1021/ci500747n. [DOI] [PubMed] [Google Scholar]
  40. Krizhevsky A.; Sutskever I.; Hinton G. E.. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems; Curran Associates: Lake Tahoe, NV, 2012; Vol. 1, pp 1097–1105. [Google Scholar]
  41. Szegedy C.; Liu W.; Jia Y.; Sermanet P.; Reed S.; Anguelov D.; Erhan D.; Vanhoucke V.; Rabinovich A.. Going Deeper with Convolutions. arXiv (SSSS), September 17, 2014, 1409.4842, ver. 1. https://arxiv.org/abs/1409.4842.
  42. Jones D.; Kim H.; Zhang X.; Zemla A.; Stevenson G.; Bennett W. F. D.; Kirshner D.; Wong S. E.; Lightstone F. C.; Allen J. E. Improved Protein–Ligand Binding Affinity Prediction with Structure-Based Deep Fusion Inference. J. Chem. Inf. Model. 2021, 61, 1583–1592. 10.1021/acs.jcim.0c01306. [DOI] [PubMed] [Google Scholar]
  43. Gomes J.; Ramsundar B.; Feinberg E. N.; Pande V. S.. Atomic Convolutional Networks for Predicting Protein-Ligand Binding Affinity. arXiv (SSSS), March 30, 2017, 1703.10603, ver. 1. https://arxiv.org/abs/1703.10603.
  44. Wallach I.; Dzamba M.; Heifets A.. AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery. arXiv (Computer Science.Machine Learning), October 10, 2015, 1510.02855, ver. 1. https://arxiv.org/abs/1510.02855.
  45. Ragoza M.; Hochuli J.; Idrobo E.; Sunseri J.; Koes D. R. Protein–Ligand Scoring with Convolutional Neural Networks. J. Chem. Inf. Model. 2017, 57, 942–957. 10.1021/acs.jcim.6b00740. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Jiménez J.; Škalič M.; Martínez-Rosell G.; De Fabritiis G. KDEEP: Protein–Ligand Absolute Binding Affinity Prediction via 3D-Convolutional Neural Networks. J. Chem. Inf. Model. 2018, 58, 287–296. 10.1021/acs.jcim.7b00650. [DOI] [PubMed] [Google Scholar]
  47. Kyro G. W.; Brent R. I.; Batista V. S. HAC-Net: A Hybrid Attention-Based Convolutional Neural Network for Highly Accurate Protein–Ligand Binding Affinity Prediction. J. Chem. Inf. Model. 2023, 63, 1947–1960. 10.1021/acs.jcim.3c00251. [DOI] [PubMed] [Google Scholar]
  48. Wang Z.; Zheng L.; Liu Y.; Qu Y.; Li Y.-Q.; Zhao M.; Mu Y.; Li W. OnionNet-2: A Convolutional Neural Network Model for Predicting Protein-Ligand Binding Affinity Based on Residue-Atom Contacting Shells.. Front. Chem. 2021, 9, 753002 10.3389/fchem.2021.753002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Shan W.; Li X.; Yao H.; Lin K. Convolutional Neural Network-based Virtual Screening. Curr. Med. Chem. 2021, 28, 2033–2047. 10.2174/0929867327666200526142958. [DOI] [PubMed] [Google Scholar]
  50. Kwon Y.; Shin W.-H.; Ko J.; Lee J. AK-Score: Accurate Protein-Ligand Binding Affinity Prediction Using an Ensemble of 3D-Convolutional Neural Networks. Int. J. Mol. Sci. 2020, 21, 8424. 10.3390/ijms21228424. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Jiang H.; Wang J.; Cong W.; Huang Y.; Ramezani M.; Sarma A.; Dokholyan N. V.; Mahdavi M.; Kandemir M. T. Predicting Protein-Ligand Docking Structure with Graph Neural Network. J. Chem. Inf. Model. 2022, 62, 2923–2932. 10.1021/acs.jcim.2c00127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Knutson C.; Bontha M.; Bilbrey J. A.; Kumar N. Decoding the protein-ligand interactions using parallel graph neural networks. Sci. Rep. 2022, 12, 7624. 10.1038/s41598-022-10418-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Nikolaienko T.; Gurbych O.; Druchok M. Complex machine learning model needs complex testing: Examining predictability of molecular binding affinity by a graph neural network. J. Comput. Chem. 2022, 43, 728–739. 10.1002/jcc.26831. [DOI] [PubMed] [Google Scholar]
  54. Yang Z.; Zhong W.; Lv Q.; Dong T.; Yu-Chian Chen C. Geometric Interaction Graph Neural Network for Predicting Protein-Ligand Binding Affinities from 3D Structures (GIGN). J. Phys. Chem. Lett. 2023, 14, 2020–2033. 10.1021/acs.jpclett.2c03906. [DOI] [PubMed] [Google Scholar]
  55. Feinberg E. N.; Sur D.; Wu Z.; Husic B. E.; Mai H.; Li Y.; Sun S.; Yang J.; Ramsundar B.; Pande V. S. PotentialNet for Molecular Property Prediction. ACS Cent Sci. 2018, 4, 1520–1530. 10.1021/acscentsci.8b00507. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Karlov D. S.; Sosnin S.; Fedorov M. V.; Popov P. graphDelta: MPNN Scoring Function for the Affinity Prediction of Protein-Ligand Complexes. ACS Omega 2020, 5, 5150–5159. 10.1021/acsomega.9b04162. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Huang K.; Fu T.; Glass L.; Zitnik M.; Xiao C.; Sun J., DeepPurpose: a Deep Learning Library for Drug-Target Interaction Prediction and Applications to Repurposing and Screening. arXiv (Computer Science.Machine Learning), December 9, 2020, 2004.08919, ver. 3. https://arxiv.org/abs/2004.08919.
  58. Öztürk H.; Özgür A.; Ozkirimli E. DeepDTA: deep drug–target binding affinity prediction. Bioinformatics 2018, 34, i821–i829. 10.1093/bioinformatics/bty593. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Castelvecchi D. Can we open the black box of AI?. Nature 2016, 538, 20–23. 10.1038/538020a. [DOI] [PubMed] [Google Scholar]
  60. Doshi-Velez F.; Kim B.. Towards a rigorous science of interpretable machine learning. arXiv (Statistics.Machine Learning), March 2, 2017, 1702.08608, ver. 2. https://arxiv.org/abs/1702.08608.
  61. Murdoch W. J.; Singh C.; Kumbier K.; Abbasi-Asl R.; Yu B. Definitions, methods, and applications in interpretable machine learning. Proc. Natl. Acad. Sci. U. S. A. 2019, 116, 22071–22080. 10.1073/pnas.1900654116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Barredo Arrieta A.; Díaz-Rodríguez N.; Del Ser J.; Bennetot A.; Tabik S.; Barbado A.; Garcia S.; Gil-Lopez S.; Molina D.; Benjamins R.; Chatila R.; Herrera F. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 2020, 58, 82–115. 10.1016/j.inffus.2019.12.012. [DOI] [Google Scholar]
  63. Roscher R.; Bohn B.; Duarte M. F.; Garcke J. Explainable Machine Learning for Scientific Insights and Discoveries. IEEE Access 2020, 8, 42200–42216. 10.1109/ACCESS.2020.2976199. [DOI] [Google Scholar]
  64. Cang Z.; Wei G. W. Integration of element specific persistent homology and machine learning for protein-ligand binding affinity prediction.. Int. J. Numer. Methods Biomed. Eng. 2018, 34, e2914 10.1002/cnm.2914. [DOI] [PubMed] [Google Scholar]
  65. Cang Z.; Mu L.; Wei G. W. Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening. PLoS Comput. Biol. 2018, 14, e1005929 10.1371/journal.pcbi.1005929. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Wu J.; Chen H.; Cheng M.; Xiong H. CurvAGN: Curvature-based Adaptive Graph Neural Networks for Predicting Protein-Ligand Binding Affinity. BMC Bioinf. 2023, 24, 378. 10.1186/s12859-023-05503-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Wee J.; Xia K. Ollivier Persistent Ricci Curvature-Based Machine Learning for the Protein–Ligand Binding Affinity Prediction. J. Chem. Inf. Model. 2021, 61, 1617–1626. 10.1021/acs.jcim.0c01415. [DOI] [PubMed] [Google Scholar]
  68. Wee J.; Xia K. Forman persistent Ricci curvature (FPRC)-based machine learning models for protein–ligand binding affinity prediction. Briefings Bioinf. 2021, 22, bbab136 10.1093/bib/bbab136. [DOI] [PubMed] [Google Scholar]
  69. Meng Z.; Xia K. Persistent spectral–based machine learning (PerSpect ML) for protein-ligand binding affinity prediction.. Sci. Adv. 2021, 7, eabc5329 10.1126/sciadv.abc5329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Liu X.; Feng H.; Wu J.; Xia K. Persistent spectral hypergraph based machine learning (PSH-ML) for protein-ligand binding affinity prediction. Briefings Bioinf. 2021, 22, bbab127 10.1093/bib/bbab127. [DOI] [PubMed] [Google Scholar]
  71. Nguyen D. D.; Gao K.; Wang M.; Wei G.-W. MathDL: mathematical deep learning for D3R Grand Challenge 4. J. Comput.-Aided Mol. Des. 2020, 34, 131–147. 10.1007/s10822-019-00237-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  72. Moon S.; Zhung W.; Yang S.; Lim J.; Kim W. Y. PIGNet: a physics-informed deep learning model toward generalized drug–target interaction predictions. Chem. Sci. 2022, 13, 3661–3673. 10.1039/D1SC06946B. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Guedes I. A.; Barreto A. M. S.; Marinho D.; Krempser E.; Kuenemann M. A.; Sperandio O.; Dardenne L. E.; Miteva M. A. New machine learning and physics-based scoring functions for drug discovery. Sci. Rep. 2021, 11, 3198. 10.1038/s41598-021-82410-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Wolpert D. H. Stacked generalization. Neural Networks 1992, 5, 241–259. 10.1016/S0893-6080(05)80023-1. [DOI] [Google Scholar]
  75. van der Laan M. J.; Polley E. C.; Hubbard A. E. Super Learner. Stat. Appl. Genet. Mol. Biol. 2007, 10.2202/1544-6115.1309. [DOI] [PubMed] [Google Scholar]
  76. Bennett J.; Elkan C.; Liu B.; Smyth P.; Tikk D. KDD Cup and workshop 2007. SIGKDD Explor. Newsl. 2007, 9, 51–52. 10.1145/1345448.1345459. [DOI] [Google Scholar]
  77. Feuerverger A.; He Y.; Khatri S. Statistical Significance of the Netflix Challenge. Stat. Sci. 2012, 27 (2), 202–231. 10.1214/11-STS368. [DOI] [Google Scholar]
  78. Hallinan B.; Striphas T. Recommended for you: The Netflix Prize and the production of algorithmic culture. New Media Soc. 2016, 18, 117–137. 10.1177/1461444814538646. [DOI] [Google Scholar]
  79. Bell R. M.; Koren Y.; Volinsky C. All Together Now: A Perspective on the Netflix Prize. CHANCE 2010, 23, 24–29. 10.1080/09332480.2010.10739787. [DOI] [Google Scholar]
  80. Reichstein M.; Camps-Valls G.; Stevens B.; Jung M.; Denzler J.; Carvalhais N.; Prabhat Prabhat, Deep learning and process understanding for data-driven Earth system science. Nature 2019, 566, 195–204. 10.1038/s41586-019-0912-1. [DOI] [PubMed] [Google Scholar]
  81. Karniadakis G. E.; Kevrekidis I. G.; Lu L.; Perdikaris P.; Wang S.; Yang L. Physics-informed machine learning. Nat. Rev. Phys. 2021, 3, 422–440. 10.1038/s42254-021-00314-5. [DOI] [Google Scholar]
  82. Thuerey N.; Holl P.; Mueller M.; Schnell P.; Trost F.; Um K.. Physics-based Deep Learning. arXiv (Computer Science.Machine Learning), April 25, 2022, 2109.05237, ver. 3. https://arxiv.org/abs/2109.05237.
  83. Wang B.; Yan C.; Lou S.; Emani P.; Li B.; Xu M.; Kong X.; Meyerson W.; Yang Y. T.; Lee D.; Gerstein M. Building a Hybrid Physical-Statistical Classifier for Predicting the Effect of Variants Related to Protein-Drug Interactions. Structure 2019, 27, 1469–1481. 10.1016/j.str.2019.06.001. [DOI] [PubMed] [Google Scholar]
  84. Gilson M. K.; Liu T.; Baitaluk M.; Nicola G.; Hwang L.; Chong J. BindingDB in 2015: A public database for medicinal chemistry, computational chemistry and systems pharmacology. Nucleic Acids Res. 2016, 44, D1045–D1053. 10.1093/nar/gkv1072. [DOI] [PMC free article] [PubMed] [Google Scholar]
  85. Tran-Nguyen V.-K.; Jacquemard C.; Rognan D. LIT-PCBA: An Unbiased Data Set for Machine Learning and Virtual Screening. J. Chem. Inf. Model. 2020, 60, 4263–4273. 10.1021/acs.jcim.0c00155. [DOI] [PubMed] [Google Scholar]
  86. Li Y.; Han L.; Liu Z.; Wang R. Comparative Assessment of Scoring Functions on an Updated Benchmark: 2. Evaluation Methods and General Results. J. Chem. Inf. Model. 2014, 54, 1717–1736. 10.1021/ci500081m. [DOI] [PubMed] [Google Scholar]
  87. Quiroga R.; Villarreal M. A. Vinardo: A Scoring Function Based on Autodock Vina Improves Scoring, Docking, and Virtual Screening. PLoS One 2016, 11, e0155183 10.1371/journal.pone.0155183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  88. Blanes-Mira C.; Fernández-Aguado P.; de Andrés-López J.; Fernández-Carvajal A.; Ferrer-Montiel A.; Fernández-Ballester G. Comprehensive Survey of Consensus Docking for High-Throughput Virtual Screening. Molecules 2023, 28, 175. 10.3390/molecules28010175. [DOI] [PMC free article] [PubMed] [Google Scholar]
  89. Daina A.; Michielin O.; Zoete V. SwissADME: a free web tool to evaluate pharmacokinetics, drug-likeness and medicinal chemistry friendliness of small molecules. Sci. Rep. 2017, 7, 42717. 10.1038/srep42717. [DOI] [PMC free article] [PubMed] [Google Scholar]
  90. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Res. 2023, 51, D523–D531. 10.1093/nar/gkac1052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  91. Torrens-Fontanals M.; Tourlas P.; Doerr S.; De Fabritiis G. PlayMolecule Viewer: A Toolkit for the Visualization of Molecules and Other Data. J. Chem. Inf. Model. 2024, 64 (3), 584–589. 10.1021/acs.jcim.3c01776. [DOI] [PubMed] [Google Scholar]
  92. Camacho C.; Coulouris G.; Avagyan V.; Ma N.; Papadopoulos J.; Bealer K.; Madden T. L. BLAST+: architecture and applications. BMC Bioinf. 2009, 10, 421. 10.1186/1471-2105-10-421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  93. Dunbar J. B. Jr.; Smith R. D.; Yang C. Y.; Ung P. M.; Lexa K. W.; Khazanov N. A.; Stuckey J. A.; Wang S.; Carlson H. A. CSAR benchmark exercise of 2010: selection of the protein-ligand complexes. J. Chem. Inf. Model. 2011, 51, 2036–46. 10.1021/ci200082t. [DOI] [PMC free article] [PubMed] [Google Scholar]
  94. Aleksander S. A.; Balhoff J.; Carbon S.; Cherry J. M.; Drabkin H. J.; Ebert D.; Feuermann M.; Gaudet P.; Harris N. L.; Hill D. P.; et al. The Gene Ontology knowledgebase in 2023. Genetics 2023, 224, iyad031 10.1093/genetics/iyad031. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ci4c01116_si_001.pdf (1.7MB, pdf)
ci4c01116_si_002.xlsx (13.4MB, xlsx)

Data Availability Statement

The BindingDB and PDBbind databases are publicly available. The code and our pre-trained models are partially available, due to our pending patent, on our GitHub page at https://github.com/Lee1701/Lee2023a.


Articles from Journal of Chemical Information and Modeling are provided here courtesy of American Chemical Society

RESOURCES