The Need for continuing blinded Pose- and Activity Prediction Benchmarks

Abstract

Computational tools for structure-based drug design (SBDD) are widely used in drug discovery and can provide valuable insights to advance projects in an efficient and cost-effective manner. However, despite the importance of SBDD to the field, the underlying methodologies and techniques have many limitations. In particular, binding pose and activity predictions (P-AP) are still not consistently reliable. We strongly believe that a limiting factor is the lack of a widely accepted and established community benchmarking process that independently assesses the performance and drives the development of methods, similar to the CASP benchmarking challenge for protein structure prediction. Here, we provide an overview of P-AP, unblinded benchmarking datasets and blinded benchmarking initiatives (concluded and ongoing) and offer a perspective on learnings and the future of the field. To accelerate a breakthrough on the development of novel P-AP methods, it is necessary for the community to establish and support a long-term benchmark challenge that provides non-biased training/test/validation sets, a systematic independent validation, and a forum for scientific discussions.

Introduction

Structure-based drug design (SBDD) has become a core method in drug discovery. It has been applied to many different protein families.^1,2 SBDD presumes that a small molecule binds to its target protein in a fixed pose around which it fluctuates due to thermal motion. While SBDD has contributed to numerous clinical candidates, many of the underlying techniques still have limitations. Chief among these is binding pose prediction for a small molecule in a target binding pocket, based on a variety of algorithms. Determining this pose is a key component of all structure-based design techniques, as it enables drug discovery teams to evaluate and optimize the interactions between a proposed molecule and a protein binding site.

The pose-finding workflow has two components: sampling, which generates a pose for a molecule in a binding site, and scoring, which ranks the poses. For a manageably small number of compounds, one may use detailed, more accurate (but costlier) simulation-based methods to estimate affinities.^3–5 For virtual screening of large compound libraries, physical simplifications, such as neglecting protein flexibility, entropy and solvation, are typically employed to reduce the computational time. The recent advent of purchasable ultra-large chemical spaces has further magnified the need for fast and efficient ways to virtually screen large datasets of compounds, such as in ultra-large virtual screening (ULVS) applications. Classical docking methods based on physical principles have been combined with machine learning/deep learning (ML/DL) models. The result can be an active learning (AL) approach in which a small fraction (1–10%) of the overall dataset is docked and used to build an ML model that predicts the docking score. The ML model, which is typically several orders of magnitude faster than docking, is then used to score the entire dataset.⁶ This process is typically repeated for 3–5 cycles to obtain a final set of molecules for purchase or synthesis.

Both pose- and activity prediction (P-AP) are relevant across all preclinical drug design phases. Based on predicted poses, binding affinities to on- and off-targets can be calculated during lead optimization using molecular dynamics simulation techniques, such as relative (RBFE) or absolute (ABFE) binding free energy calculations.^3,4,7 The emergence of generative algorithms that can optimize a chemical structure with respect to a particular scoring function are highly dependent on P-AP approaches to direct the chemical modifications. Small changes need to be accurately scored, thus the accuracy of the activity prediction function needs to be sufficiently high. All SBDD, artificial intelligence (AI) and ULVS applications critically depend on reliable P-AP methods.

Despite its importance to drug design and decades of development, P-AP is still not a solved challenge. P-AP fails too often, in particular for novel targets where the structures of co-complexes have yet to be solved. Even when the structures of co-complexes are known, P-AP does not always perform well. Zev et al. showed in a study of Mpro X-ray complexes⁸ that the experimental pose could be regenerated within 2Å RMSD for only 26% of non-covalently bound ligands and 46% of covalent inhibitors. It is very difficult to quantify the current state of the art, as there is no independent, continuous benchmarking effort. The protein-ligand (P-L) structure and affinity prediction field has seen several benchmarking initiatives^9–13 (for more detail see section “Past and Present Protein-Ligand Binding Benchmarks” below). Unfortunately, most of these have been discontinued, usually due to lack of sustained funding. Currently, there is no established community benchmark that independently assesses and thus drives development of novel P-AP methods.

The protein structure prediction field, which recently experienced its “AlphaFold moment”, was able to achieve positive progress primarily due to the CASP benchmarking challenge which has now been run through 15 iterations over 30 years. Although the latest CASP15 included ligand-containing structures, only a small fraction of the complexes were relevant for drug discovery.¹⁴ CASP16 will continue to feature a section on protein-ligand predictions, but CASP has not so far become central to the P-AP community, as it has to the protein structure prediction community, and stable community commitment would be needed for this to occur.

With this perspective, we want to bring attention to the need for a widely accepted, ongoing, benchmarking process for P-AP methods. We are united in the belief that P-AP approaches, with all their physical approximations, can and need to be substantially better than they are today. Novel physical approximations and AI/ML algorithms, together with more and better data to train and validate activity prediction approaches, will provide substantial improvements. The potential benefit for drug discovery, and ultimately patients, will be huge, as P-AP is still a key technology for boosting the efficiency of drug design. Developing activity prediction approaches will be a long-term effort that requires continuous enablement, not only monetarily but also by supplying diverse high-quality structures and affinity data to enable rigorous benchmarking and feedback on methodological improvements. This is especially true for testing of AI/ML models which make increasing use of available data for training; such models need new, diverse data for fair benchmarking.

In what follows, we briefly review various algorithmic approaches to P-AP. Then, we provide an overview of evaluation datasets and benchmarks related to P-AP that have either concluded or are still ongoing and discuss learnings from them. Finally, we discuss open issues in P-AP and requirements for the next generation of benchmarks and conclude with a perspective on the future of the field.

Approaches to Pose & Activity Prediction

P-AP approaches are regularly reviewed,^15–17 so we only give a short overview of the general directions.

Pose Prediction Overview

Pose prediction algorithms attempt to predict the most stable pose of a small molecule bound to a target protein. Classic algorithms typically consist of two components: a sampling method that establishes poses and a scoring method that ranks the stabilities of the poses. In many cases, the sampling and scoring components work in concert. An initial set of poses is sampled and scored, then the most promising poses are subjected to additional sampling and scoring. It should be noted that scores obtained from docking programs only provide a rough estimate of the “fit” of a small molecule in a protein binding pocket and should not be considered “activity predictions”. The prediction of binding affinity is typically a separate procedure that is performed on a docked pose.

Pose generation algorithms have been developed in two waves. The first wave, which we term classical methods,^{18 19 20 21 22 23 24 25} use a combination of physics-based, knowledge-based, and empirical terms to define and evaluate the poses of ligands in a protein binding site. The second wave is based on deep learning architectures like DiffDock²⁶ and EquiBind²⁷. These networks are directly trained on known 3D structures without any human-imposed constraints on preferred geometries. While these methods show promise, they are not mature and generated structures can be problematic.²⁸

Additional criteria are often used together with activity prediction approaches to filter out physically unrealistic poses. These include terms such as electrostatic and van der Waals energies, rotatable bond penalties, lipophilic contacts, binding surface area, ligand strain energy, as well as interaction fingerprints, shape similarity, and pharmacophore points on known interaction hotspots.

Activity Prediction Overview

Many approaches for activity prediction—which typically is approached by affinity prediction -- have been developed. Standard scoring functions have primarily been designed for the initial screening of extensive drug candidate libraries. Here, the total number of P-L complexes to be evaluated grows extremely rapidly, considering that the number of compounds must be multiplied by their numerous protonation and tautomer states, as well as flexibility in the protein binding site. Therefore, these scoring functions were designed to highlight plausible ligand binding modes rather than rank them accurately by their binding free energies. Their efficiency relies on severe approximations, which limit their accuracy in predicting binding affinities.²⁹ Structure-based deep-learning-based scoring functions augment standard scoring functions with additional machine-learning corrections or utilize standalone machine-learning algorithms to estimate affinity directly from the structure of the P-L complex. These deep-learning-based approaches are fast but not necessarily as reliable.

Computationally less efficient but more rigorous methods include MM-GB/PBSA³⁰ and the linear interaction energy (LIE) method.³¹ These are typically based on molecular dynamics simulations and so are intermediate in both accuracy and the computational cost between standard scoring functions and rigorous perturbation methods. The MM/GBSA method, for example offers the decomposition of binding free energies³² to individual terms and residues, but its predictive power is limited³⁰. A more rigorous approach is the direct computation of binding free energies from MD simulation, represented by thermodynamic integration³³ (TI) and free energy perturbation (FEP) methods. These methods offer the advantage of a theoretically sound derivation of the computed quantities based on statistical mechanics, which blur the strict separation between pose prediction and activity prediction, as many local poses are sampled during a simulation. Calculating simulation-based free energies can take up to tens of GPU hours per molecule. The accuracy to be expected is highest for congeneric series of small molecules that only differ by a small part of the molecule and may be compromised by the use of approximate force fields.³⁴ Poor 3D starting points for binding free energy calculations may have a substantial negative effect on the predicted affinities. Mining Minima provides an alternate, rigorously grounded approach that uses a force field and an implicit solvent model and falls between docking and simulation-based free energy methods in terms of speed and accuracy.³⁵

Quantum Mechanics-based (QM) methods have been recently utilized to avoid the limitations imposed by the simplified formalism of both the conventional scoring functions and simulation force fields.³⁶ QM methods are currently too computationally demanding to be routinely applied to P-L systems. Hybrid QM/MM schemes (where only a small part of the system is treated at the QM level), fragmentation schemes and semiempirical methods have been utilized in the physics-based end-point scoring functions.^37,38

Consensus scoring, the application of different activity prediction and combination approaches to the same complex, has repeatedly been attempted. This is based on the assumption that a single scoring function is not optimal, but different scoring functions can compensate for weakness of the individual ones.^39,40

Judging P-AP approaches

There are three classic criteria by which P-AP approaches are judged: Pose prediction, ranking/early enrichment, and correlation of predicted and measured binding affinities. For pose prediction, geometries of predicted poses are compared to experimentally determined poses, typically from CryoEM or X-ray. The identification of the ligand near-native or native pose within all generated poses as the minimum-free-energy structure is an unambiguous criterion for the performance of any scoring/sampling method. The difficulty of classic pose prediction generally varies between self-docking (e.g. docking the ligand into a protein binding pocket perfectly pre-formed) and cross-docking into (1) experimental holo structures, (2) experimental apo structures and (3) structures obtained through structure prediction techniques that start with an amino-acid sequence. When comparing predictions to experimentally determined poses, it is important to note that the experimental pose is also a model which has been fit into calculated electron density. The quality of the experimental pose fit should always be confirmed when utilizing this criterion.

Ranking and early enrichment is typically used to examine the virtual screening power of a P-AP setup. Here, the goal is to assign a higher score to known active ligands and lower scores to inactive decoy compounds. Success can be measured as the area under the curve (AUC) of the receiver operator characteristics (ROC) curve, or as the enrichment of known active ligands within the top X% of a rank-ordered list. Success can also be measured as the reduction of the number of false negatives and false positives. However, in lead optimization campaigns, most or all of the compounds of interest are likely to be active, so the goal is to rank candidate ligands by their affinities: performance is evaluated by calculating the correlation between the experimental and calculated binding affinity, or the difference (e.g. RMSD) between the predicted and true values. All three classic criteria, i.e. posing, early enrichment, and affinity prediction, come with their own practical challenges for the evaluation, as will be discussed in later sections.

When setting up protein systems for P-AP predictions, structural approximations need to be made: Which protein structure should be docked into? Which rotamers, tautomers, and protomers should be used? Which water molecules are important to consider? Water is well known to mediate crucial interactions for binding, but should the water be considered as a part of the ligand or the protein? Those questions often do not have clear scientific answers, and answers may be system- and method-dependent. Covalent binding, metal coordination, or cofactors can introduce additional layers of complexity that need careful consideration. The ultimate performance depends on the individual user bringing in their expert understanding of the system.⁴¹ While commercial tools typically come with a pipeline to prepare protein-ligand systems, there is a lack of good open-source tools for doing so. Additionally, the quality of system preparation in isolation is difficult to benchmark, making it challenging to assess the utility of tools in this space.

Deep Learning-based P-AP approaches

The recent surge in deep learning (DL) has led to numerous publications with novel DL-based methods for pose prediction. They can be differentiated into pocket-based docking with pre-defined pockets, such as DeepDock⁴² or Uni-Mol,⁴³ as well as blind pose prediction, e.g. DiffDock²⁶, EquiBind²⁷, or TankBind,⁴⁴ where the binding site is identified by the method itself. A recent publication by Deane and coworkers pointed out that many evaluations of these are flawed because evaluation criteria are inadequate for these methods.²⁶ DL-based methods need a lot of data to learn the physics-based principles that are hard-coded in the conventional methods. Most pose prediction evaluation criteria developed for conventional methods do not test for physical correctness (e.g. the flatness of aromatic systems, or the absence of intramolecular clashes). Deane et al. recently showed that those algorithms are immature, as the local small molecule geometries generated can be physically highly implausible, and when presented with unbiased test, none of the current DL-based pose prediction algorithms perform better than conventional algorithms.²⁸ This is particularly true if the DL-based methods are applied to structures that they have never seen before.^28,50,51

More recently, the field has seen the emergence of a new docking approach referred to as “co-folding”. This method, which has its origins in protein structure prediction, simultaneously generates the structure of the protein while predicting a ligand binding pose. Such models generate the protein structure based on the amino acid sequence information and include the ligand during this process. This way, protein structure generation and ligand pose generation are tackled simultaneously, which would solve current induced fit binding issues. Perhaps the most noteworthy example of co-folding is AlphaFold3,⁴⁵ a successor to the widely acclaimed AlphaFold2.⁴⁶ Co-folding approaches learn the relationships between proteins and ligands from structures in the PDB. Some of these methods can reproduce the poses of covalently bound ligands. The published methods include NeuralPLexer,⁴⁷ RoseTTAFold All-Atom,⁴⁸ and AlphaFold3,⁴⁵ some of which have performed well on the PoseBusters²⁸ benchmark. However, the limited availability of program code and a suitable shared benchmark has prevented broader benchmarking by the community.

Blinded vs unblinded benchmarking

It is necessary to differentiate between blinded and unblinded benchmarking when judging pose prediction and activity prediction approaches. Unblinded benchmarking is typically done by the authors of new approaches, using a publicly available dataset. Those datasets often have some relationship to the model training set, for example by being split out of a ground truth pool before training (such as by a random partition into train/validation/test sets, with the test set only used for evaluation of the final model). Thus, validation and test sets may have the same basic distribution as the training dataset, which can result in an unfair evaluation if the ultimate goal is to predict an unseen new protein-ligand system (the core point of discovery and lead optimization in Pharma is to discover and/or optimize new chemical matter). If standard datasets are used for the evaluation, there is a real risk that samples overlap between training and evaluation sets. Beyond this, judging what is inside and outside of the distribution is very hard, and novel proposals for how to best do this are still being published. Overlap between training and evaluation sets are a particular concern for AI/ML-based scoring functions that can potentially memorize training samples. ^28,49,50

In contrast, blinded benchmarking is performed on completely independent, new data that has not been previously available to the creators of a novel P-AP approach. Blinded benchmarking sets a higher bar than unblinded benchmarking, as it seeks to make it impossible for models to cheat or accidentally give artificially inflated performance. To guarantee that benchmark data has not been used to train a model, blinded benchmarking either has to be done by an independent organization on unreleased or secret data, or P-AP approaches have to be locked and containerized at a certain point in time and then tested on new data. For an evolving discipline, this means that blinded benchmark datasets must regularly be refreshed or updated with the latest data. There are examples for both approaches, and we will discuss them below.

Benchmarking datasets

Although we favor blinded benchmarking as final assessment, unblinded benchmarking datasets are very useful for P-AP development. Unblinded benchmarking datasets for pose prediction are typically curated directly from the PDB. These sets differ in the years of publication, size, and criteria used to measure experimental quality.^51–54

Datasets for ranking include decoys^55–58 whereas later datasets include property-matching of decoys,^59,60 and ensure that decoys are experimentally validated (i.e. measured inactives). Negative data is published less often and is thus more difficult to compile for a dataset.

Binding free energy prediction has benchmarking datasets of different kinds. There are large datasets comprising protein-ligand systems with experimentally determined cocrystal structures and measured Ki or IC50 values. Structures typically come from the PDB database. The PDBbind database, underlying the CASF benchmark (evaluation according to our nomenclature), and BindingMOAD⁶¹ are such datasets. There are also much larger datasets of binding free energies that do not include crystal structures, notably BindingDB⁶², and ChEMBL⁶³. One challenge with all of these datasets is that Ki and IC50 values are not really transferable across assays.^64,65 It is safest to evaluate ranking only within each assay series, for example as in a recent publication introducing the PL-REX benchmark dataset consisting of 10 diverse protein-ligand series with consistently measured affinities and high-resolution X-ray crystal structures.³⁸ There are also collections of congeneric series of ligands crystallized or template-docked into the same target structure, with activity values that have been measured in the same assay setup.^5,34,66,67 and the OpenFF benchmark set (in process).

Issues with evaluation datasets

Apart from the risk that evaluation datasets overlap with the training data, there are a number of problems with current benchmarking datasets:

• Decoys are often not experimentally confirmed (may be true actives).

• Pose Prediction: Native structure is overly simple, cross-docking is a more realistic scenario.

• Docking evaluations based on X-ray structures do not evaluate how activity prediction approaches account for nonphysical poses, i.e. clashes (see comment on DL-based scoring function above).

• Public benchmarking datasets based on compilations from mixed assays can have a huge spread in measured values (PDBbind has an activity range from 2 – 12 log units). While this is not a problem per se, evaluators need to be aware that the R² values observed from such datasets do not translate to R² values observed on datasets obtained for a single assay which typically has a spread of 4 log units (10 μM to 1 nM). Also, affinity values need to be log-transformed, as the fold factor difference linearly corresponds to a change in Gibbs’ free energy of binding.

• Public benchmarking datasets (as opposed to blinded benchmarks) do not allow comparing different P-AP approaches, since it is very hard to control the overlap of training and evaluation data. Note that overlap does not necessarily mean that the identical protein-ligand structure is part of both datasets. A setup where representatives of the same protein family with very similar ligands are part of both datasets also renders the evaluation much less meaningful. In addition, different methods usually are tested against different subsets of the publicly available data, and this inconsistency introduces uncontrolled variation into any comparison of methods.

Past and Present Protein-Ligand Binding Benchmarks

The need for rigorous blinded benchmarking motivated the organization of community-wide prediction challenges in this space since the first SAMPL challenge in 2008.⁶⁸ The SAMPL challenges, most recently SAMPL9,⁶⁹ have maintained a distinctive focus on the calculation of physical properties simpler than protein-ligand affinities as tests of the fundamental reliability of physics-based methods, though they have also included some protein-ligand binding challenges.^70–72 For about a decade, the U.S. National Institutes of Health supported the CSAR series of protein-ligand pose- and affinity-prediction challenges,^73–75 followed by the D3R challenges.^9,76–78 The TDT Challenge, sponsored by the American Chemical Society combined the generation of tutorials with a prospective prediction component where selected molecules were screened experimentally.¹⁰ With the sunsetting of CSAR, D3R, and TDT, the CASP challenge series, which has traditionally focused on protein structure prediction, has branched into protein-ligand pose prediction, starting with CASP15⁷⁹ and heading into CASP16. The recently launched CACHE challenges are benchmarking hit finding methods where the predicted compounds are experimentally tested. Therefore CACHE is complementary to benchmarking initiatives focused on P-AP.^12,80

The protein-ligand challenges led to some specific conclusions, though many of these have concerned the validation process as much as the computational methods themselves. At the last D3R workshop, it was noted that every D3R challenge included some methods with accurate pose predictions, and that most affinity predictions did correlate positively with the experimental “ground truth”, but that the results did not make it possible to identify a single best method or to rank methods in terms of accuracy. Such results highlight the importance of parameter settings and protein preparation. Similarly, methods that incorporated machine learning/artificial intelligence in some cases underperformed and in other cases outperformed more structure- and physics-based methods. Overall, it was not feasible to track the performance of a given method through multiple challenges across the years, because methods and their applications changed over time.

Two efforts, CELPP¹³ and CAMEO,¹¹ addressed this limitation by establishing continuous, high-throughput benchmarking of protein-ligand pose predictions. These took advantage of the fact that the RCSB Protein Data Bank⁸¹ releases a tranche of new structures every week, and several days before the structures are released, it provides a file with a list of the forthcoming structures. Thus, an up-tempo, weekly, blinded prediction challenge can be held in the days between releases of the listing files and release of the structures. In about a year of operation, CELPP generated over 2,000 pose-prediction challenges and used them to benchmark methods of external participants. This reflects a data flow about 100 times that of D3R, and it demonstrated the ability to provide statistically significant comparisons among methods.

The CAMEO project provides a decentralized resource for model evaluation. In a manner similar to CELPP, unpublished structures are transmitted to user’s servers for predictions. The predictions are then returned to the central CAMEO servers that evaluate the predicted structures. While CAMEO was originally intended for protein structure prediction, it had for some time expanded its capabilities to include the prediction of liganded structures but discontinued that in 2016. Although CELPP and CAMEO are not currently operational for predicting protein-ligand structures, this powerful approach gives a strong direction for future work.

There is no analog of the PDB to enable an exactly analogous high-throughput benchmark of protein-ligand affinity calculations. Something very similar could be implemented by taking advantage of the ongoing flow of new protein-ligand affinity data available in newly published articles and patents. We envision that challenge participants would provide fully automated affinity-calculation workflow software to a trusted evaluation group, which would curate newly published protein-ligand affinity data and use them to benchmark the workflows in an effectively blinded manner. Given that BindingDB and ChEMBL currently curate on the order of 10⁵ protein-ligand affinity data per year, it is likely that a robust benchmarking flow could be established.

Protein-ligand binding benchmarks may also emerge from large open-science initiatives dedicated to generating experimental screening data to train ML models. The Structural Genomics Consortium (SGC) is using DNA-encoded-library and affinity selection mass spectrometry screening platforms to screen dozens (and soon hundreds) of proteins against billions and thousands of compounds respectively. Computational predictions from ML models trained on this set are then tested experimentally at the SGC and data shared openly.

Open Issues for P-AP prediction and benchmarking

In the field of P-AP development and benchmarking, there are some issues beyond individual P-AP approaches that are rarely addressed:

P-AP performance is highly dependent on the individual user bringing in their expert understanding of the system when setting up the system and evaluating the output.⁴¹ Errors will propagate and poorly modelled poses can affect the performance of more rigorous methods such as FEP.⁸² Additionally, users who are experts on various target families will often be able to determine the most appropriate pose by simple visual inspection.⁸³ To make benchmarks fully comprehensible, we propose that participants deposit exact input files used in the calculations.

The limits of the experimental data against which P-AP predictions are compared also requires a better understanding. Studies have demonstrated that mixing Ki or IC50 data from different assays leads to increased levels of noise.^64,65,84 In reality, assays measure more than the idealized protein-ligand binding event that P-AP models assume. In fact, various idiosyncrasies of each individual assay contribute to the experimental outcome. This not only leads to a parallel shift of measured activities, but also a substantial amount of random noise. Thus, it would be important to understand how good a general P-AP model can become when compared to a typical Ki/IC50 assay - even with a perfect model, the upper limit is much lower than the reproducibility from an individual assay.

Classical evaluation metrics for predicted poses, such as RMSD between docked and crystal pose, are not sufficient to evaluate the quality of a pose prediction tool, especially for ML/DL based pose prediction methods, as demonstrated in the PoseBusters benchmark. Here, additional metrics are necessary to get a real estimate of performance. In addition, for ML/DL based pose prediction methods, a classical non-DL method should be included to test advantages over the baseline state-of-the-art. Similarly, it is important to include comparisons to null hypotheses. As a null hypothesis for activity prediction, we recommend comparing to correlations with simple descriptors like molecular weight or ClogP. These descriptors can be calculated via many open source tools and are fully transparent. For pose prediction, simple comparator models include predicting pose of a new compound based on the most similar compound-target pair in public databases or running a docking using an established approach with a defined version.

Setting Up Future Benchmarks

The requirements for a good benchmark set have been articulated,^52,85 but it is worthwhile to look at them in the context of advancing the field.

Relevance

Benchmarks should aim to contain relevant drug targets with drug-like molecules. As this is the key use case for pose prediction, it would help if the dataset is drawn from the druglike space and macromolecules that are representative drug targets. This space is expanding to include macrocyclic compounds, radioligand therapies, harder protein targets⁸⁶, as well as nucleic acid targets.⁷⁹ Pose prediction benchmarks should also seek to expand their domain. This implies sets of complexes with ligands that are dissimilar to currently known ligands in public benchmark sets, not just in terms of structure but also in terms of the protein-ligand interactions, e.g. covering rare ligand functional groups and protein co-factors. It should also be recognized that we need to drug targets previously considered undruggable, as discovery projects are being challenged with multimers as well as monomers, with highly flexible and poorly defined pockets, with cryptic pockets, and with other types of macromolecules. Target diversity is important, not just from the point of view of biological function, but also the properties of the pocket.

The data sets used should also include a range of scenarios that are challenging to current methods. For example, they should include activity cliffs, where similar ligands have very different affinities or binding modes. In addition, docking calculations typically yield highly similar poses for similar ligands, and cases like those described in development of BAY069,⁸⁷ where there are two flipped binding modes for very similar structures, are beyond the current state–of–the–art. These emphasize the need for experimental validation of any pose hypothesis. This situation would be a very hard test for pose prediction tools and demonstrate that even with prior knowledge of a similar system in the training set, predictions would be flawed. It would also be good to have sets split by differing protonation states and whether the water network is a key factor, as currently this is only poorly dealt with.

Interpretability

An important component of a benchmark test is not just to numerically assess a particular approach but also to understand why it did well or had issues. P-AP methods have many components that can be tuned and modified; several have been discussed above. A given docking method may perform well or poorly, depending on the methods used to set up the proteins and ligands. A recent review of DL-based activity prediction approaches aimed at introducing measures to compare different types of DL-based activity prediction approaches and also their explainability.⁸⁸ All evaluated methods exhibited a lack of interpretability if applied to real-world data sets. The authors of the programs should be afforded the opportunity to present their results and freely discuss these topics. It is important in benchmarking challenges to capture as much information as possible on employed parameters, set-up, etc. to identify if trends exist.

Structure quality

To be a good benchmark, the experimental ground truth should be clearly defined. In terms of a protein-ligand complex Xray structure, this would mean clear electron density maps for the binding pocket. The quality of the pose prediction should be assessed by eventual fit to these maps in addition to using the typical RMSD criterion of < 2A. If the binding site is well defined and some other parts of the structure not involved in binding are not, this criterion can be relaxed. By the same token, NMR structures and cryo-EM structures offer important challenges of several binding modes/protein conformers with near equivalent energies. Pose prediction programs should be able to deal with this energy landscape with several minima.

Affinity data quality

In tandem with structure, the experimental ground truth of measured affinity needs to be reliable and consistent, measured using a single method under same conditions (ideally in the same laboratory), with enough information to provide K_d values. Individually measured Kd data for enantiomeric pairs with a substantial difference in pK_d (> 2 log units, the more the better) will help to highlight approaches that can distinguish strong and weak binding based on modelled 3D interactions. With all benchmarks, there needs to be enough data to draw statistically significant conclusions.

“Fairness”

How do we ensure people don’t train on highly related data to optimize performance on the benchmarks? If benchmarks are very kinase-heavy then training on kinase data will improve performance, presumably, even if it is not exactly the benchmark data. A target with a novel pocket architecture and varying binding modes would be much harder. We recognize that pose prediction programs have been monetized and more spectacular results can be published in higher ranking journals, so there are incentives to appear to do better. To make matters fairer, it behooves scientists to disclose their training sets, so that a benchmark can be assessed in this context.

All of these issues can be addressed by running a diverse array of truly blind benchmark tests, where either containerized methods are tested on newly released, and hence effectively blind data, or on data released only after predictions are made. In both cases, high quality new datasets are needed on a recurring basis. Datasets with consistent measurement of affinity, with a good range of affinity, are most available in the industrial setting. It would be useful if Pharma/Biotech had a clear incentive to disclose these data sets as part of the publication strategy for projects that have come to an end. The incentive for those participating in a challenge may seem clearer (publications, status of best-in-class), but may not be compelling for existing industry leaders that may feel there is more to lose than to gain. In these cases, the incentive must come from industrial customers insisting on seeing the results of commercial software participation in blinded challenges. For the end user community, the tests need to have a realistic perspective to run long enough and be accepted by the whole community. (CASP is an inspiring example, as it is now in iteration 16 and has a diverse international board). The resulting insights will enable well-informed purchasing decisions and may even help determine which tools merit further investment and development.

The Future of the Field

Throughout the last few years, virtual libraries have increased dramatically in size, such as ENAMINE’s REadily AvaiLable for synthesis (REAL) combinatorial library,⁸⁹ which nowadays contains 48 billion molecules. The appearance of ultra-large on-demand chemical libraries reinvigorated the docking and scoring field. Screening such ultra-large virtual libraries demands novel approaches for P-AP. Fast docking approaches, e.g. based on DOCK3.7^90–92, or heavily parallelized setups like VirtualFlow⁹³ were developed to perform P-AP of these ultra-large libraries. Other approaches such as VSYNTHES,⁸⁹ SpaceDocking,⁹⁴ and others⁹⁵ make use of approximations, such as a combinatorial approach, modeling first minimum scaffolds and then expanding on the best modelled scaffolds. Hybrid models and deep docking methods were built, where a subset of molecules is docked and a DL model is built based on these docking scores to predict docking scores for the remaining library. Subsequently, active learning iterations improve the model by taking more actual docking scores into account during training.^96–99 As ultra-large libraries continue to grow, the interest in novel and better performant P-AP approaches will continue to grow.

Even though P-AP is not a new problem, open questions and challenges still abound as described above. Due to the frequent application of P-AP during drug development campaigns, novel methods and approaches have the potential for high impact. To advance the field, blinded benchmarks are needed to guide developers and continuously reveal the evolving state of the art. As the field advances, our understanding of how to best run those benchmarks also evolves. To fully leverage the potential of P-AP and develop novel approaches that will further reduce cost and cycle times of preclinical drug discovery, ongoing benchmarking initiatives need a continuous supply of novel blinded structures and other relevant data, an independent cross-institutional community that facilitates the evaluation, and a community of method developers that engages in the blinded benchmarks.