Assessing the potential of deep learning for protein-ligand docking

Abstract

The effects of ligand binding on protein structures and their in vivo functions carry numerous implications for modern biomedical research and biotechnology development efforts such as drug discovery. Although several deep learning (DL) methods and benchmarks designed for protein-ligand docking have recently been introduced, to date no prior works have systematically studied the behavior of the latest docking and structure prediction methods within the broadly applicable context of (1) using predicted (apo) protein structures for docking (e.g., for applicability to new proteins); (2) binding multiple (cofactor) ligands concurrently to a given target protein (e.g., for enzyme design); and (3) having no prior knowledge of binding pockets (e.g., for generalization to unknown pockets). To enable a deeper understanding of docking methods’ real-world utility, we introduce PoseBench, the first comprehensive benchmark for broadly applicable protein-ligand docking. PoseBench enables researchers to rigorously and systematically evaluate DL methods for apo-to-holo protein-ligand docking and protein-ligand structure prediction using both primary ligand and multi-ligand benchmark datasets, the latter of which we introduce for the first time to the DL community. Empirically, using PoseBench, we find that (1) DL co-folding methods generally outperform comparable conventional and DL docking baseline algorithms, yet popular methods such as AlphaFold 3 are still challenged by prediction targets with novel protein-ligand binding poses; (2) certain DL co-folding methods are highly sensitive to their input multiple sequence alignments, while others are not; and (3) DL methods struggle to strike a balance between structural accuracy and chemical specificity when predicting novel or multi-ligand protein targets. Code, data, tutorials, and benchmark results are available at https://github.com/BioinfoMachineLearning/PoseBench.

1. Introduction

The field of drug discovery has long been challenged with a critical task: determining the structure of ligand molecules in complex with proteins and other key biomolecules [1]. As accurately identifying such complex structures (in particular multi-ligand structures) can yield advanced insights into the binding dynamics and functional characteristics (and thereby, the medicinal potential) of numerous protein complexes in vivo, in recent years, significant resources have been spent developing new experimental and computational techniques for protein-ligand structure determination [2]. Over the last decade, machine learning (ML) methods for structure prediction have become indispensable components of modern structure determination at scale, with AlphaFold 2 for protein structure prediction being a hallmark example [3, 4].

As the field has gradually begun to investigate whether proteins in complex with other types of molecules can faithfully be modeled with ML (and particularly deep learning (DL)) techniques [5–7], several new works in this direction have suggested the promising potential of such approaches to protein-ligand structure determination [8–11]. Nonetheless, it remains to be shown the extent to which the latest of such (docking and co-folding-based) DL methods can adequately generalize to the context of binding novel or uncommon protein-ligand interaction (PLI) pockets and multiple interacting ligand molecules (e.g., which can alter the chemical functions of various enzymes) as well as whether such methods can faithfully model amino acid-specific types of PLIs natively found in crystallized biomolecular structures.

To bridge this knowledge gap, our contributions in this work are as follows:

• We introduce the first unified benchmark for protein-ligand docking and structure prediction that evaluates the performance of several recent DL-based baseline methods (DiffDock-L, DynamicBind, NeuralPLexer, RoseTTAFold-All-Atom, Chai-1, Boltz-1, and AlphaFold 3) as well as conventional algorithms (P2Rank + AutoDock Vina) for primary and multi-ligand docking, which suggests that DL co-folding methods generally outperform conventional algorithms yet remain challenged by novel or uncommon prediction targets.

• In contrast to several recent works using crystal protein structures for protein-ligand docking [12, 13], the docking benchmark results we present in this work are all within the context of standardized input multiple sequence alignments (MSAs) and high accuracy apo-like (i.e., AlphaFold 3-predicted) protein structures (see Appendix D) without specifying known binding pockets, which notably enhances the broad applicability of this study’s findings.

• Our newly proposed benchmark, PoseBench, enables specific insights into necessary areas of future work for accurate and generalizable biomolecular structure prediction, including that DL methods struggle to balance faithful modeling of native PLI fingerprints (PLIFs) with structural accuracy during pose prediction and that some DL co-folding methods (AlphaFold 3) are more dependent than others (Boltz-1, Chai-1) on the availability of input MSAs.

• Our benchmark results also highlight the importance of including challenging (out-of-distribution) datasets when evaluating future DL methods while measuring their ability to recapitulate amino acid-specific PLIFs with an appropriate new metric that we introduce in this work.

2. Results and discussion

In this section, we present PoseBench’s results for primary and multi-ligand protein-ligand docking and structure prediction and discuss their implications for future work, as succinctly illustrated in Figure 1. Note that across all experiments, for generative methods, we report their performance metrics in terms of the mean and standard deviation across three independent runs of each method to gain insights into their inter-run stability and consistency. Key metrics include a method’s percentage of structurally accurate ligand pose predictions with a (heavy atom centroid) root mean square deviation (RMSD) less than 2 (1) Å (i.e., (c)RMSD ≤ 2 (1) Å); its percentage of structurally accurate pose predictions that are also chemically valid according to the PoseBusters software suite (i.e., RMSD ≤ 2 Å & PB-Valid), which can be affected by the post-hoc application of structural relaxation driven by computationally expensive molecular dynamics (MD) simulations [14] (i.e., with relaxation); and our newly proposed Wasserstein matching score of its amino acid-specific predicted PLIFs (PLIF-WM). We formally define these metrics in Section 4.4. For interested readers, in Appendix C, we report the average runtime and memory usage of each baseline method to determine which methods are the most efficient for real-world structure-based applications, and in Appendix G we present supplementary results.

Fig. 1:

Overview of PoseBench, our comprehensive benchmark for broadly applicable DL modeling of primary and multi-ligand protein complex structures. Baseline algorithms within the benchmark include a range of the latest DL docking and co-folding methods, both open-source and commercially restrictive, as well as conventional algorithms for docking. Key observations derived using PoseBench include the discontinuity between structure and interaction modeling performance for novel or uncommon prediction targets and the heavy reliance of key DL co-folding methods on MSA-based input features to achieve high structural accuracy.

2.1. Astex Diverse results

Containing PLI structures deposited in the RCSB Protein Data Bank (PDB) [15] up until 2007, most of the well-known Astex Diverse dataset’s structures [16] are present in the training data of each baseline method, yet benchmarking results for this dataset (n=85), shown in Figure 2, indicate that only DL co-folding methods achieve higher structural and chemical accuracy rates (RMSD ≤ 2 Å & PB-Valid) than the conventional docking baseline AutoDock Vina combined with P2Rank for PLI binding site prediction to facilitate blind molecular docking. Interestingly, nearly all baseline methods identify the correct PLI binding pocket approximately 90% of the time, yet only the DL co-folding methods Chai-1 [17], Boltz-1 [18], and AlphaFold 3 (AF3) [11] achieve a reasonable balance between their rates of structural and chemical accuracy and chemical specificity (PLIF-WM), with the single-sequence (i.e., MSA-ablated) version of AF3 being a notable exception. These results suggest that DL co-folding methods have learned the most comprehensive representations of this dataset’s input sequences, yet only the DL co-folding method Chai-1 maintains strong performance without the availability of diverse input MSAs. One likely explanation for this phenomenon is that Chai-1’s training relied upon the availability of amino acid sequence embeddings generated by the protein language model ESM2 [19] in addition to features derived from input MSAs, which may have imbued the model with rich MSA-independent representations for biomolecular structure prediction.

Fig. 2:

Astex Diverse primary ligand docking success rates (n=85).

2.2. DockGen-E results

As visualized in Figure 3, results with our new DockGen-E dataset of biologically relevant PLI complexes deposited in the PDB up to 2019 (n=122) demonstrate that only the latest DL co-folding methods can locate a sizable fraction of structurally accurate PLI binding poses represented in this dataset. As such methods may have previously seen these PLI structures in their respective training data, it is surprising that even the latest AF3 model fails to identify a structurally and chemically accurate pose for more than 75% of the dataset’s complexes. Further, for Chai-1, Boltz-1, and AF3, their single-sequence variants achieve higher chemical specificity than their MSA-based versions, which may indicate that for these methods MSA features obfuscate primary sequence knowledge in favor of evolution-averaged (i.e., amino acid-generic) representations. The overall lower range of PLIF-WM values achieved by each method for this dataset further suggests the increased chemical modeling difficulty of this dataset’s complexes compared to those presented by the Astex Diverse dataset. A potential source of these difficulties is that each of this dataset’s complexes represents a functionally distinct PLI binding pocket (as codified by ECOD domains [20], see [21] for more details) compared to data deposited in the PDB before 2019. As such, it is likely that Chai-1, Boltz-1, and AF3 are “overfitted” to the most common types of PLI structures in the PDB and may overlook several uncommon types of PLI binding pockets present in nature.

Fig. 3:

DockGen-E primary ligand docking success rates (n=122).

2.3. PoseBusters Benchmark results

With approximately half of its PLI structures deposited in the PDB after AF3 and Boltz-1’s maximum-possible training data cutoff of September 30, 2021 (n=308 total, filtered to n=130 for subsequent analyses), the PoseBusters Benchmark dataset’s results, presented in Figure 4, indicate once again that DL co-folding methods achieve top performance compared to conventional and DL docking baseline methods. Nonetheless, we observe an interesting phenomenon whereby Chai-1 strikes a balance of structural and chemical accuracy and chemical specificity comparable to that of the best-performing AF3, even without input MSAs, potentially suggesting that Chai-1 achieves stronger binding pose generalization for this dataset than AF3. Moreover, with the single-sequence version of AF3, we again observe significant degradations in its overall performance, whereas running Chai-1 with input MSAs achieves higher chemical specificity at the cost of marginal structural accuracy compared to running it in single-sequence mode. These observations highlight the importance in future work of carefully studying why and how the training of biomolecular structure generative models can be influenced to varying degrees by the availability and composition of diverse input MSAs.

Fig. 4:

PoseBusters Benchmark primary ligand docking success rates (n=130/308).

2.4. CASP15 results

As a new dataset of novel and challenging PLI complexes on which no method has been trained, the CASP15 dataset’s multi-ligand results (n=13), illustrated in Figure 5, indicate that most methods fail to adequately generalize to multi-ligand prediction targets, yet AF3 stands out in this regard (only) when provided input MSAs. As many of these CASP15 multi-ligand targets represent large, highly symmetric protein complexes, it is likely that additional evolutionary information in the form of MSAs has improved AF3’s ability to predict higher-order protein-protein interactions for these targets, yet interestingly its improved rate of structural accuracy comes at the cost of its protein-ligand chemical specificity (in comparison to its single-sequence results). For the CASP15 dataset’s single-ligand (i.e., primary ligand) results (n=6) presented in Figure 6, this trend is subverted in that conventional docking and open-source DL co-folding methods such as AutoDock Vina, NeuralPLexer, and Boltz-1 outperform all other recent DL co-folding methods in modeling crystalized PLIFs while achieving comparable rates of structural accuracy. Given the small size of the CASP15 dataset, it is reasonable to conclude that DL methods, in particular some of the latest co-folding methods, may be challenged to predict protein-ligand complexes containing novel PLIs. In the following Section 2.5, we will explore this latter point in greater detail by analyzing the protein-ligand binding similarities between common PDB training data and this benchmark’s evaluation datasets.

Fig. 5:

CASP15 multi-ligand docking success rates (n=13).

2.5. Exploratory analyses of results

In this section, we explore a range of questions to study the common “failure” modes of the baseline methods included in this work, to outline new directions for future research and development efforts in drug discovery.

Research Question 1:

What are the most common types of protein-ligand complexes that all baseline methods fail to predict?

→ To address this query, we first collect all ligand pose predictions that no method could predict with structural and chemical accuracy (according to the metric RMSD ≤ 2 Å & PB-Valid). For each of these “failed” ligand poses, we retrieve the PDB’s functional annotation of the protein in complex with this ligand and construct a histogram to visualize the frequency of these (failed complex) annotations. The results of this analysis are presented in Figure 7, in which we see that metal transport proteins, flavoproteins, biosynthetic proteins, RNA binding proteins, immune system proteins, and oxidoreductases are commonly mispredicted by all baseline methods such as Chai-1 and RoseTTAFold-All-Atom (RFAA) [7], suggesting these classes of proteins may be largely unaddressed by the most recent DL methods for PLI structure prediction. To illuminate potential future research directions, in the next analysis, we investigate whether this pattern persists specifically for AF3, the most accurate DL co-folding method according to our benchmarking results.

Research Question 2:

What are the most common types of protein-ligand complexes that highly-accurate DL co-folding methods such as AF3 fail to predict?

→ For this follow-up question, we link all of AF3’s failed ligand predictions with corresponding protein function annotations available in the PDB to understand which types of PLI complexes AF3 finds the most difficult to predict (to understand its predictive coverage of important molecular functions). Similar to the answer to our first research question, Figure 8 shows that, in order of difficulty, AF3 is most challenged to produce ligand poses of high structural and chemical accuracy for ligand-bound RNA binding proteins, immune system proteins, metal transport proteins, biosynthetic proteins, flavoproteins, lyases, and oxidoreductases. As several of these classes of proteins have not been well represented in the PDB over the last 50 years (e.g., immune system and biosynthetic proteins), in future work, it will be important to ensure that either the performance of new DL methods for PLI structure prediction is expanded to support accurate modeling of these uncommon types of ligand-bound proteins or a broadly applicable fine-tuning method for uncommon types of interactions is proposed.

Research Question 3:

Is lack of protein-ligand binding pose homology to PDB training data (inversely) correlated with the prediction accuracy of each method?

→ To understand the impact of protein-ligand binding pose similarity on the performance of each baseline method, we use the PLINDER data resource [22] to identify (n=41/130) cluster representatives of the PoseBusters Benchmark dataset based on the product of each complex’s ligand (structure and feature-based) Combined Overlap Score (SuCOS) [23] and its protein binding pocket’s (structure and sequence-based) similarity [24], as none of this subset’s prediction targets are contained in any method’s training dataset. For these cluster representatives, we then calculate the Pearson and Spearman correlations (and p-values) between each method’s complex prediction accuracy (i.e., ligand pose RMSD) and the complex’s maximum SuCOS-binding pocket-based similarity to any complex deposited in the PDB before AF3’s training dataset cutoff of September 30, 2021. Figure 9 reveals that all DL methods’ performance is correlated with a complex’s similarity to common PDB training data, with (MSA-based) Boltz-1, AF3, and Chai-1 exhibiting the strongest and most statistically significant (p < 0.05) correlations. Like concurrent work assessing the performance of co-folding methods in novel prediction settings [24], our findings suggest that although the current generation of deep learning models (both docking and co-folding methods) for protein-ligand docking and structure prediction can occasionally make accurate predictions for truly novel (SuCOS-pocket similarity < 30) protein-ligand complexes, such methods rely (at least in large part) on recapitulating protein-ligand binding patterns seen during training to make accurate predictions for unseen complexes (n.b., while interestingly for conventional docking methods such as P2Rank-Vina, this trend is not observed). We conclude our quantitative analyses with an illustration of the different failure modes of each baseline method, as depicted in Figure 10. In this figure, we illustrate that DL methods such as RFAA and AF3 commonly struggle to accurately predict the structure of ligand-binding biosynthetic and immune system proteins, suggesting that these (uncommon) types of PLIs are not well addressed by the current generation of DL-based structure prediction methods, suggesting future research opportunities for interaction-specific modeling (e.g., through the use of fine-tuning or preference optimization).

3. Conclusions

In this work, we have introduced PoseBench, a unified, broadly applicable benchmark and toolkit for studying the performance of methods for protein-ligand docking and structure prediction. Benchmarking results with PoseBench, summarized in Table 1, suggest that DL co-folding methods generally outperform conventional and DL docking baselines yet remain challenged to predict complexes containing novel protein-ligand binding poses, with AF3 performing best overall when deep MSAs are available for a target protein (Chai-1 otherwise), regardless of the availability of homologous proteins. Further, we find that several DL methods face difficulties balancing the structural accuracy of their predicted poses with the chemical specificity of their induced protein-ligand interactions, highlighting that future methods may benefit from the introduction of physico-chemical loss functions or sampling techniques to bridge this performance gap. Lastly, we observe that some (but not all) DL co-folding methods are highly dependent on the availability of diverse input MSAs to achieve high structural prediction accuracy (e.g., AF3 but not Chai-1 or Boltz-1), underscoring the need in future work to elucidate the impact of the availability of MSAs and protein language model embeddings on the training dynamics of biomolecular structure prediction methods. As a publicly available resource, PoseBench is flexible to accommodate new datasets, methods, and analyses for protein-ligand docking and structure prediction.

Table 1:

PoseBench evaluation datasets of protein-(multi-)ligand structures.

	Astex Diverse	DockGen-E	PoseBusters Benchmark	CASP15
Ligand Type	Primary	Primary	Primary	Multi
Source	[16]		[12]
Size (Total # Ligands)	85	122	130/308	102 (across 19 complexes) → 6 (13) single (multi)-ligand complexes
Training Data Homology	High	Moderate	Low	Low
Top-Ranked Method (w/ MSAs)	AF3	AF3	AF3	AF3
Top-Ranked Method (w/o MSAs)	Chai-1	Chai-1	Chai-1	NeuralPLexer

4. Methods

4.1. PoseBench

The overall goal of PoseBench, our newly designed benchmark for protein-ligand docking and structure prediction, is to provide the research community with a centralized resource with which one can systematically measure, in a variety of macromolecular contexts, the methodological advancements of new conventional and DL methods proposed for this domain. In the following sections, we describe PoseBench’s design and composition (as portrayed in Figure 1) and how we have used PoseBench to evaluate several recent DL docking and co-folding methods (as well as a strong conventional baseline algorithm) for protein-ligand structure modeling.

4.2. Benchmark datasets

As shown in Table 1, PoseBench provides users with broadly applicable, preprocessed versions of four datasets with which to evaluate existing or new protein-ligand structure prediction methods: Astex Diverse [16], PoseBusters Benchmark [12], and the new DockGen-E and CASP15 PLI datasets that we have manually curated in this work.

Astex Diverse dataset.

The Astex Diverse dataset is a collection of 85 PLI complexes composed of various drug-like molecules and cofactors known to be of pharmaceutical or agrochemical interest, where a primary (representative) ligand is annotated for each complex. This dataset can be considered an easy benchmarking dataset for methods trained on recent data contained in the PDB in that most of its complexes (deposited in the PDB up to 2007) are known to overlap with the commonly used PDBBind 2020 (time-split) training dataset [25, 26] containing complexes deposited in the PDB before 2019. As such, including this dataset for benchmarking allows one to estimate the breadth of a method’s structure prediction capabilities for important primary ligand protein complexes represented in the PDB.

To perform unbound (apo) protein-ligand docking with this dataset, we used AF3 to predict the structure of each of its protein complexes, with all ligands and cofactors excluded. We then optimally aligned these predicted protein structures to the corresponding crystal (holo) PLI complex structures using a PLI binding site-focused structural alignment performed using PyMOL [27], where each binding site is defined as all amino acid residues containing crystallized heavy atoms that are within 10 Å of any crystallized ligand heavy atom. To enable the broad availability of PoseBench’s benchmark datasets in both commercial and academic settings, we also provide unbound (apo) protein structures predicted using the MIT-licensed ESMFold model [19], although in Section 2 we report results using AF3’s predicted structures as the default data source. We further note that on average across all benchmark datasets and methods, AF3’s predicted structures improve baseline docking methods’ structural accuracy rates by 5–10%.

PoseBusters Benchmark dataset.

Version 2 of the the popular PoseBusters Benchmark dataset [12], which we adopt in this work, contains 308 recent primary ligand protein complexes deposited in the PDB from 2019 onwards. Accordingly, in contrast to Astex Diverse, this dataset can be considered a moderately difficult benchmark dataset for baseline methods, since many of its complexes do not directly overlap with the most commonly used PDB-based training data. Important to note is that, among all baseline methods, AF3 and Boltz-1 used the most recent PDB training data cutoff of September 30, 2021, which motivated us to report the results in Section 2.3 for only the subset of PoseBusters Benchmark complexes (n=130) deposited in the PDB after this date. Like Astex Diverse, for the PoseBusters Benchmark dataset, we used AF3 (and ESMFold) to predict the apo protein structures of each of its complexes and then performed our PyMOL-based structural binding site alignments.

DockGen-E dataset.

The original DockGen dataset [13] contains 189 diverse primary ligand protein complexes, each representing a functionally distinct type of PLI binding pocket according to ECOD domain partitioning [20, 21]. Consequently, this dataset can be considered PoseBench’s most difficult primary ligand dataset to model since its PLI binding sites are distinctly uncommon compared to those frequently found in the training datasets of all baseline methods, though it is important to note that these original DockGen complexes were deposited in the PDB from 2019 onward, making this benchmarking dataset partially overlap with the training datasets of baseline DL co-folding methods such as Chai-1, Boltz-1, and AF3. Nonetheless, in line with our initial hypotheses, the benchmarking results in Section 2 demonstrate that no baseline method can adequately predict the PLI binding sites and ligand poses represented by this bespoke subset of the PDB, suggesting that all baseline DL methods have yet to learn broadly applicable representations of protein-ligand binding.

Unfortunately, the original DockGen dataset contains only the primary protein chains representing each novel binding pocket after filtering out all non-interacting chains and cofactors in a given biological assembly (bioassembly), which considerably reduces the biophysical context provided to baseline methods to make reasonable predictions. As such, we argue for the need to construct a new dataset that challenges baseline methods (in particular DL co-folding methods) to predict full bioassemblies containing novel PLI binding pockets, which we address with our enhanced version of DockGen called DockGen-E.

To construct DockGen-E, we collected the original DockGen dataset’s PLI binding pocket annotations for each complex. We then retrieved the corresponding first bioassembly listed in the PDB to obtain each PDB entry’s biologically relevant complex, filtering out DockGen complexes for which the first bioassembly could not be mapped to its original PLI binding pocket annotation (which indicates these original DockGen PLI binding pockets were initially not derived from the PDB’s corresponding first bioassembly). This procedure left 122 biologically relevant assemblies remaining for benchmarking. Like Astex Diverse and PoseBusters Benchmark, for DockGen-E, we then used AF3 (and ESMFold) to predict the unbound (apo) protein structures of each complex in the dataset and structurally aligned the predicted protein structures to their corresponding crystallized PLI binding sites using PyMOL.

CASP15 dataset.

To assess the multi-primary ligand (i.e., multi-ligand) modeling capabilities of recent methods for protein-ligand docking and structure prediction, with PoseBench, we introduce a preprocessed, DL-ready version of the CASP15 PLI dataset debuted as a first-of-its-kind prediction category in the 15th Critical Assessment of Techniques for Structure Prediction (CASP) competition held in 2022 [28]. The CASP15 PLI dataset is originally comprised of 23 protein-ligand complexes released in the PDB from 2022 onward, where we subsequently filter out 4 complexes based on (1) whether the CASP organizers ultimately assessed predictions for the complex and (2) whether they are nucleic acid-ligand complexes with no interacting protein chains. The 19 remaining PLI complexes, which contain a total of 102 (fragment) ligands, consist of a variety of ligand types including single-atom (metal) ions and large drug-sized molecules with up to 92 atoms in each (fragment) ligand. As such, this dataset is appropriate for assessing how well structure prediction methods can model interactions between different (fragment) ligands in the same complex, which can yield insights into the inter-ligand steric clash rates of each method. As with all other benchmark datasets, we used AF3 (and ESMFold) to predict the unbound (apo) structure of each protein complex in the dataset and then performed a PyMOL-based structural alignment of the corresponding PLI binding sites.

PLI similarity analysis between datasets.

For an investigation of the similarity of PLIs represented in each dataset, in Appendix E, we analyze the different types and frequencies of common, ProLIF-annotated protein-ligand binding pocket interactions [29] natively found within the common PDBBind 2020 training dataset and the Astex Diverse, PoseBusters Benchmark, DockGen-E, and CASP15 datasets, respectively, to quantify the diversity of the (predicted) interactions each dataset can be used to evaluate. In short, we find that the DockGen-E and CASP15 benchmark datasets are the most dissimilar compared to the common PDBBind 2020 training dataset, further illustrating the unique PLI modeling challenges offered by these evaluation datasets.

4.3. Formulated tasks

In this work, we developed PoseBench to focus our analysis on the behavior of different conventional and DL methods for protein-ligand structure prediction in a variety of macromolecular contexts (e.g., with or without inorganic cofactors present). With this goal in mind, below we formalize the structure prediction tasks currently available with PoseBench, with its source code flexibly designed to accommodate new tasks in future work.

Primary ligand blind docking.

For primary ligand blind docking, each baseline method is provided with a complex’s (multi-chain) protein sequence and an optional predicted (apo) protein structure as input along with its corresponding (fragment) ligand SMILES strings, where fragment ligands include the primary binding ligand to be scored as well as all cofactors present in the corresponding crystal structure. In particular, no knowledge of the complex’s PLI binding pocket is provided to evaluate how well each method can (1) identify the correct PLI binding pockets and (2) correct ligand poses within each pocket (3) with high chemical validity and (4) specificity for the pockets’ amino acid residues. After all fragment ligands are predicted, PoseBench extracts each method’s prediction of the primary binding ligand and reports evaluation results for these primary predictions.

Multi-ligand blind docking.

For multi-ligand blind docking, each baseline method is provided with a complex’s (multi-chain) protein sequence and an optional predicted (apo) protein structure as input along with its corresponding (fragment) ligand SMILES strings. As in primary ligand blind docking, no knowledge of the PLI binding pockets is provided, which offers the opportunity to evaluate not only PLI binding pocket and conformation prediction accuracy but, in the context of multi-binding ligands, also inter-ligand steric clash rates.

4.4. Metrics

Traditional metrics.

For PoseBench, we reference two key metrics in the field of structural bioinformatics: the root-mean-square deviation (RMSD) and local Distance Difference Test (lDDT) [30]. The RMSD between a predicted 3D conformation (with atomic positions ${\widehat{x}}_i$ for each of the molecule’s $n$ heavy atoms) and the ground-truth (crystal structure) conformation $\left(x_i\right)$ is defined as:

$$\text{RMSD}=\sqrt{\frac{1}{n}\sum _{i=1}^n{‖{\widehat{x}}_i-x_i‖}^2}.$$ (1)

The lDDT score, which is commonly used to compare predicted and ground-truth protein 3D structures, is defined as:

$$\mathrm{lDDT}=\frac{1}{N}\sum _{i=1}^N\frac{1}{4}\sum _{k=1}^4\left(\frac{1}{\left|{\mathcal{N}}_i\right|}\sum _{j\in {\mathcal{N}}_i}\text{Θ}\left(\left|{\widehat{d}}_{ij}-d_{ij}\right|<{\text{Δ}}_k\right)\right),$$ (2)

where $N$ is the total number of heavy atoms in the ground-truth structure; ${\mathcal{N}}_i$ is the set of neighboring atoms of atom $i$ within the inclusion radius $R_o=15\AA$ in the ground-truth structure, excluding atoms from the same residue; ${\widehat{d}}_{ij}\left(d_{ij}\right)$ is the distance between atoms $i$ and $j$ in the predicted (ground-truth) structure; ${\text{Δ}}_k$ are the distance tolerance thresholds (i.e., $0.5\text{Å}$ $1\text{Å}$ $2\text{Å}$ and $4\text{Å}$); $\text{Θ}(x)$ is a step function that equals 1 if x is true, and 0 otherwise; and $\left|{\mathcal{N}}_i\right|$ is the number of neighboring atoms for atom $i$. As originally proposed by Robin et al. [28], in this study, we adopt the PLI-specific variant of lDDT for scoring multi-ligand complexes, which calculates lDDT scores to compare predicted and ground-truth protein-(multi-)ligand complex structures following optimal (chain-wise and residue-wise) structural alignment of the predicted and ground-truth PLI binding pockets.

Lastly, we also measure the molecule validity rates of each predicted PLI complex pose using the PoseBusters software suite (i.e., PB-Valid) [12]. This suite runs several important chemical and structural sanity checks for each predicted pose including energy ratio inspection and geometric (e.g., flat aliphatic ring) assertions which provide a secondary filter of accurate poses that are also chemically and structurally meaningful.

New metrics.

The RMSD, lDDT, and PB-Valid metrics of a protein-ligand binding structure provide useful characterizations of how accurate and reasonable a predicted pose is. However, a key limitation of these metrics is that they do not measure how well a predicted pose resembles a native pose when comparing their induced PLIFs. Recently, [31] introduced a complementary benchmarking metric, PLIF-valid, assessing DL methods’ recovery rates of known PLIs. However, this metric only reports a strict recall rate of each method’s interaction types rather than a continuous measure of how well each method’s interactions match the distribution of crystalized PLIs. Moreover, in drug discovery, a primary concern when designing new drug candidates is ensuring they produce amino acid-specific types of interactions (and not others), hence we desire each baseline method to recall the correct types of PLIs for each pose and to avoid predicting (i.e., hallucinating) types of interactions that are not natively present. Consequently, we argue that an ideal PLI-aware benchmarking metric is a single continuous metric that assesses the recall and precision of a method’s predicted distribution of amino acid-specific PLIFs. To this end, we propose two new benchmarking metrics, PLIF-EMD and PLIF-WM.

For each PLI complex, PLIF-EMD measures the Earth mover’s distance (EMD) [32] between a method’s predicted histogram of PLI type counts $u$ (specific to each type of interaction) and the corresponding native histogram $v$, where each histogram of interaction type counts is represented as a 1D discrete distribution. Formally, this equates to computing the Wasserstein distance between these two 1D distributions $u$ and $v$ as

$$\text{PLIF-EMD}:=l_1(u,v)=\underset{\pi \text{∈Π}(u,v)}{\text{inf}}{\int }_{\text{ℝ}\times \text{ℝ}}|x-y|d\pi (x,y),$$ (3)

where $\text{Π}(u,v)$ denotes the set of distributions on $ℝ\times ℝ$ whose marginals, $u$ and $v$, are on the first and second factors, respectively. To penalize a baseline method for producing non-native interaction types, we unify the bins in each histogram before converting them into 1D discrete representations. Namely, to perform this calculation, each PLI is first represented as a fingerprint tuple of <ligand type, amino acid type, interaction type> as determined by the software tool ProLIF [29] and then grouped to count each type of tuple to form a histogram. As such, a lower PLIF-EMD value implies a better continuous agreement between predicted and native interaction histograms. PLIF-WM, derived from PLIF-EMD, assesses the Wasserstein matching (WM) score of a pair of PLIF histograms. Specifically, to obtain a more benchmarking-friendly score ranging from 0 to 1 (higher is better), we define PLIF-WM as

$$\text{PLIF-WM}:=1-\frac{\text{PLIF-EMD}-{\text{PLIF-EMD}}_{\text{min}}}{{\text{PLIF-EMD}}_{\max }-{\text{PLIF-EMD}}_{\min }},$$ (4)

where ${\text{PLIF-EMD}}_{\min }$ and ${\text{PLIF-EMD}}_{\max }$ denote the minimum (best) and maximum (worst) values of PLIF-EMD, respectively. As a metric normalized relative to each collection of the latest baseline methods, PLIF-WM allows one to quickly identify which of the latest methods has the greatest capacity to produce realistic distributions of PLIs. As a practical note, we use SciPy 1.15.1 [33] to provide users of PoseBench with an optimized implementation of PLIF-EMD and thereby PLIF-WM.

4.5. Baseline methods and experimental setup

Overview.

We designed PoseBench to answer specific modeling questions for PLI complexes such as (1) which types of methods (if any) can predict both common and uncommon PLI complexes with high structural and chemical accuracy and (2) which most accurately predict multi-ligand structures without steric clashes? In the following sections, we discuss which types of methods we evaluate in our benchmark and how we evaluate each method’s predictions for PLI complex targets.

Method categories.

As illustrated in Figure 1, to explore a range of the most well-known or recent methods to date, we divide PoseBench’s baseline methods into one of three categories: (1) conventional algorithms, (2) DL docking algorithms, and (3) DL co-folding algorithms.

As a representative algorithm for conventional protein-ligand docking, we pair AutoDock Vina (v1.2.5) [34] for molecular docking with P2Rank for protein-ligand binding site prediction [35] to form a strong conventional (blind) docking baseline (P2Rank-Vina) for comparison with DL methods. To represent DL docking methods, we include DiffDock-L [13] for docking with static protein structures and DynamicBind [9] for flexible docking. Lastly, to represent some of the latest DL co-folding methods, we include NeuralPLexer [10], RFAA [7], Chai-1 [17], Boltz-1 [18] (versus Boltz-2 [36] for sake of time-split benchmarking validity), and AF3 [11]. For interested readers, each method’s input and output data formats are described in Appendix F.

Prediction and evaluation procedures.

The PLI complex structures each method predicts are subsequently evaluated using different structural and chemical accuracy and molecule validity metrics depending on whether the targets are primary or multi-ligand complexes. In Section 4.4, we provide formal definitions of PoseBench’s evaluation metrics. Note that if a method’s prediction raises any errors in subsequent scoring stages (e.g., due to missing entities or formatting violations), the prediction is excluded from the evaluation.

Primary ligand evaluation.

For primary ligand targets, we report each method’s percentage of (top-1) ligand conformations within 2 Å of the corresponding crystal ligand structure (RMSD ≤ 2 Å), using 1 Å to instead assess whether the predicted ligand’s heavy atom centroid (i.e., binding pocket) was correct (cRMSD ≤ 1 Å), as well as the percentage of such ”correct” ligand conformations that are also considered to be chemically and structurally valid according to the PoseBusters software suite [12] (RMSD ≤ 2 Å & PB-Valid). Importantly, as described in Section 4.4, we also report each method’s new PLIF-WM scores to study the relationship between its structural accuracy and chemical specificity.

Multi-ligand evaluation.

Similar to the protein-ligand scoring procedure employed in the CASP15 competition [28], for multi-ligand targets, we report each method’s (top-1) percentage of ”correct” (binding site-superimposed) ligand conformations (RMSD ≤ 2 Å) as well as violin plots of the RMSD and PLI-specific lDDT scores of its protein-ligand conformations across all (fragment) ligands within the benchmark’s multi-ligand complexes (see Appendix G for these plots). Notably, this latter metric, referred to as lDDT-PLI, allows one to evaluate specifically how well each method can model protein-ligand structural interfaces. Additionally, we report each method’s PB-Valid rates (calculated once for each multi-ligand complex) and PLIF-WM scores.

5. Related work

Structure prediction of PLI complexes.

The field of DL-driven protein-ligand structure determination was largely sparked with the development of geometric deep learning methods such as EquiBind [26] and TANKBind [37] for direct (i.e., regression-based) prediction of bound ligand structures in protein complexes. Notably, these predictive methods could estimate localized ligand structures in complex with multiple protein chains as well as the associated complexes’ binding affinities. However, in addition to their limited predictive accuracy, they have more recently been found to frequently produce steric clashes between protein and ligand atoms, notably hindering their widespread adoption in modern drug discovery pipelines.

Protein-ligand structure prediction and docking.

Shortly following the first wave of predictive methods for protein-ligand structure determination, DL methods such as DiffDock [8] demonstrated the utility of a new approach to this problem by reframing protein-ligand docking as a generative modeling task, whereby multiple ligand conformations can be generated for a particular protein target and rank-ordered using a predicted confidence score [38]. This approach has inspired many follow-up works offering alternative formulations of this generative approach to the problem [7, 9–11, 13, 17, 18, 36, 39–52], with some of such follow-up works also being capable of accurately modeling protein flexibility upon ligand binding or predicting binding affinities to a high degree of accuracy.

Benchmarking efforts for protein-ligand complexes.

In response to the large number of new methods that have been developed for protein-ligand structure prediction, recent works have introduced several new datasets and metrics with which to evaluate newly developed methods [24], with some of such benchmarking efforts focusing on modeling single-ligand protein interactions [12, 22, 31, 53–56] and others specializing in the assessment of multi-ligand protein interactions [28]. One of the motivations for introducing PoseBench in this work is to bridge this gap by systematically assessing a selection of the latest (pocket-blind) structure prediction methods within both interaction regimes, using unbound (apo) protein structures with docking methods and challenging DL co-folding methods to predict full bioassemblies from primary sequences. Our benchmarking results in Section 2 demonstrate the relevance and utility of this comprehensive new evaluation suite for the future of protein-ligand modeling.

Extended Data

Fig. 6: Extended Data Figure 1 —

CASP15 single-ligand docking success rates (n=6).

Fig. 7: Extended Data Figure 2 —

Function annotations of the PLI complexes all methods mispredicted (n=129).

Fig. 8: Extended Data Figure 3 —

Function annotations of the PLI complexes AF3 mispredicted (n=171).

Fig. 9: Extended Data Figure 4 —

Relationship between methods’ PoseBusters Benchmark prediction accuracy (RMSD) and each complex’s training set similarity (SuCOS-pocket similarity) (n=41).

Fig. 10: Extended Data Figure 5 —

Examples of baseline methods’ three failure modes discussed in this work.

6. Data availability

The PoseBench datasets and benchmark results are available at https://zenodo.org/records/16791095 under a Creative Commons Attribution 4.0 International Public License, with further licensing discussed in Appendix A and detailed dataset documentation (e.g., of AF3’s predicted protein structure accuracy) provided in Appendix D.