Assessing structural prediction accuracy for nanobody–small molecule complexes

Abstract

Generative models have transformed structural bioinformatics, enabling antibody and nanobody design against protein epitopes; however, nanobody engineering for small molecule sensing remains largely experimental. In this work, we evaluate seven state-of-the-art structure predictors, AlphaFold3, Chai-1, Boltz-2x, RoseTTAFold3, Protenix, FlowDock and OmegaFold, on nanobody–small molecule complexes. Most predictors accurately reproduced nanobody and CDR geometries but struggled with ligand placement and orientation, although co-folding improved overall accuracy. Contact analysis revealed that CDR1, rather than CDR3, was predominantly involved in ligand binding. Intrinsic confidence scores correlated poorly with experimental accuracy and showed limited power in distinguishing binders from non-binders. Increasing the number of samples and seeds yielded modest gains in accuracy, whereas additional recycles did not. These findings highlight both the strengths and limitations of structure prediction methods for nanobody–small molecule complexes.

Introduction

Protein prediction models have evolved at a high speed in recent years. From the first monomer predictions of AlphaFold in 2021 (Jumper et al., 2021) to the recent development of AlphaFold3 (Abramson et al., 2024a) and other diffusion-based deep learning softwares (Chai Discovery et al., 2024; Passaro et al., 2025), only three years have elapsed. Now, predicting structures containing combinations of proteins, nucleic acids, and small molecules is possible. Structure prediction algorithms have shaped the way we study structural biology, which led to the bestowal of the 2024 Chemistry Nobel Prize to their main contributors (Callaway, 2024). The structure prediction revolution triggered a paradigm shift in protein design (Listov et al., 2024; Khakzad et al., 2023). Once experimentally costly, this task is now greatly streamlined by computational methods, though experimental validation remains indispensable.

The design of protein binders for small molecules has flourished as a result of the protein design revolution. It has led to the creation of biosensors (Quijano-Rubio et al., 2021; An et al., 2024), therapeutic compounds (Feng et al., 2017; Lu et al., 2024) and even enzymes (Mann et al., 2025; Völler, 2023; Lauko et al., 2025). Most of these new protein structures are based on artificial folds developed by protein design groups. However, to design highly selective and affine binders, nature relies on antibodies and their smaller ‘cousins’, nanobodies.

Nanobodies (Nbs) were discovered in 1993 in the laboratory of Richard Hammers, where they observed camelid antibodies containing only a heavy chain, and not the light and heavy chains found in conventional antibodies (Hamers-Casterman et al., 1993). Conventional antibodies are composed of four polypeptidic chains: two light and two heavy chains, which have a total molecular weight of 150 kDa. In contrast, nanobodies, or also so-called single-chain antibodies, are the recombinant part of the variable fraction of the heavy chain antibodies. The fact that Nbs are composed by only one polypeptidic chain confers additional stability while maintaining the affinity constant against their corresponding antigen. According to their low molecular weight (12–15 kDa), Nbs can be internalized in cells (Bruce et al., 2016), and, due to their recombinant nature, can be engineered in different fusion proteins (He et al., 2025).

In comparison to conventional antibodies, they only contain half of the complementary determining regions (CDRs) that are responsible for the specificity and selectivity against the target analyte. CDRs are disordered regions composed by 7–17 amino acids that form the high affinity binding region and source of variability of both antibodies and nanobodies. The other more rigid and conserved regions are named framework regions (FWRs). Despite their small size, nanobodies are on par with antibodies in selectivity and specificity, reaching nanomolar or even picomolar ranges of affinity (Salvador et al., 2019). Although the vast majority of nanobodies are used to target proteins, they have also been employed to create immunosensors to target small molecules thanks to their outstanding properties (He et al., 2022; Tabares-da Rosa et al., 2019a; Li et al., 2021; Zuo et al., 2023). Despite the computational advances in the last years, most of the efforts in the field have been experimental and the few computational advances were more focused on sequence-based approaches (Li et al., 2024; Zhang & Tsuda, 2025; Hacisuleyman & Erman, 2020), with only one structural work published (Li et al., 2025) in 2025.

Meanwhile, there have been attempts in the field of computational structural biology to benchmark structure predictors for protein-ligand compounds (Leemann et al., 2023; Kroll & Rousset, 2025; Škrinjar et al., 2026), albeit not focused on nanobody or antibody usage. Nanobody and antibody benchmarks have shown these proteins are challenging (Valdés-Tresanco et al., 2023; Yin & Pierce, 2024; Fromm et al., 2025; Hitawala & Gray, 2025; Sánchez-Marín et al., 2026), something that was also noted by AlphaFold 3 authors (Abramson et al., 2024a), but these works have overlooked small molecule binding. This might be due to the low availability of structures to produce a benchmarking: in SabDab database (Dunbar et al., 2014) only 3% of antibodies and 1% of nanobodies are bound to small molecules. However, establishing the accuracy of current structure predictors on small molecule–nanobody complexes could yield significant advancements in structure-based nanobody drug discovery and optimization.

In this study we present a benchmarking on the performance of different state-of-the-art structure prediction softwares on different nanobody-small molecule analyte systems. We analyse their accuracy on protein prediction and ligand placement, as well as CDR prediction and their relationship with binding contacts. We also assess how this accuracy is related to the reported confidence scores for the best couple of methods and the dependence of the results in the number of recycles and diffusion samples.

Methods

Nanobody–small molecule dataset

The Nb-analyte dataset has been curated using the following approach: First, SAbDab (Dunbar et al., 2014) was queried using the keywords ‘Nanobody + hapten’, prompting the search for all the structures containing nanobody strands targeting a small molecule analyte. From there, 69 protein chains were obtained, with their CDRs. They were then filtered using pandas (pandas, n.d.) to procure only the chains that contained unique CDRs, as the complementary regions have higher amino acid variability and are tailored for binding. Mutations in framework regions were not considered significant for the binding process and thus would probably not lead to changes in structure worth analysing.

After that filtering, 13 Nb-SM systems were left and upon further visual inspection with PyMOL the structure corresponding to 7TE8 (Cao et al., 2022) was discarded. It corresponded to a dual biosensor, where two nanobodies with different paratopes bound to a small molecule. This was not the geometry we were searching for and differs enough from the other nanobodies in the dataset to be analysed and processed differently. Hence, the new dataset consisted in 11 structures, omitting the 2 sequences that formed 7TE8.

From the final dataset, 5 structures were published in the PDB prior to 2021, and thus would be included in most of the training sets for the structure predictors analysed in this paper. Those complexes are 1QD0 (Spinelli et al., 2000), 5VL2, 5VM0 and 5VM6 (Tabares-da Rosa et al., 2019a); and 6ITQ (Ding et al., 2019). The other remaining structures that were not included in the training sets were 7TJC (Swofford et al., 2022), 8FTG (Smith et al., 2023), 8V9W (Gallant et al., 2024), 8V9Z (Gallant et al., 2024), 7RGA (Guo et al., 2021), and 8H7I (Li et al., 2023) (Table S1).

Input generation and structure pre-processing

To generate the inputs for the structure predictions, an automated pipeline was used, available at https://github.com/QTC-IQAC/predictorVerse. It allowed us to generate the necessary inputs with just the protein sequence and ligand SMILES. The protein sequences were obtained from SAbDab, and the ligand SMILES were obtained by loading the PDB structure with MDTraj (McGibbon et al., 2015), separating the protein and the ligand chains and extracting the SMILES code from those small molecules using RDKit (Landrum et al., 2025). However, some automatically generated SMILES were confusing for some structure predictors and gave erroneous results, be it not interpreting the molecule at all or providing the wrong stereochemistry. Thus, we manually replaced those SMILES for the ones defined in PubChem for the associated analytes. This has been done for the structures 7RGA and 8H7I. For 1QD0, the copper atoms have been left out of the predictions.

Structure predictions

The results have been obtained using the aforementioned structure predictors: AF3 (Abramson et al., 2024a), Chai-1 (Chai Discovery et al., 2024), Boltz-2x (Passaro et al., 2025), (Protenix Team et al., 2026), RF3 (Corley et al., 2025), FlowDock (Morehead & Cheng, 2025) and OmegaFold (Wu et al., 2022). We decided to include FlowDock, with and without using a template from Chai-1, as its affinity predictions could provide reliable insight in a protein design pipeline. OmegaFold was included as a previous benchmark reported it yielded good results for the modelling of antibodies and nanobodies (Valdés-Tresanco et al., 2023). However, it is not capable of producing co-folding predictions (see Table 1). Although some specific tools exist to predict antibody structures, such as IgFold (Ruffolo et al., 2023) or ABlooper (Abanades et al., 2022), they cannot co-fold the sequences in the presence of a small molecule. Although they could be combined with a docking algorithm, we do not consider this approach in the present work.

Table 1.

Information about the protein folding capabilities of the analysed structure predictors.

	AF3	Chai-1	Boltz-2x	Protenix	RF3	FlowDock	OmegaFold
Predicts only protein?	YES	YES	YES	YES	YES	NO	YES
Predicts protein + ligand (co-folding)?	YES	YES	YES	YES	YES	YES	NO
Training set cut-off date	Sept 2021	Sept 2021	June 2023	Sept 2021 (protenix_base_default_v1.0.0 model)	Sept 2021 (rf3_foundry_09_21_preprint model)	Sept 2021	Sept 2021

Results have been run with the default hyper-parameters of the predictors to obtain 10 samples. For the case of OmegaFold, only 1 sample per nanobody structure was considered as there were no specific options to sample more structures. Predictions were run on several computational resources, using Nvidia GeForce RTX™ 4070 SUPER, Nvidia GeForce RTX™ 4090 and Nvidia H100 GPUs.

Analysis

Prior to the analysis, the original PDBs were cleaned to be easier to manipulate. Any extra chains were removed to only leave one instance of nanobody with its corresponding small molecule. The water atoms were also removed.

Analysis of the structures was performed using MDTraj (McGibbon et al., 2015), pandas (pandas, n.d.), seaborn (Waskom, 2021; Hunter, 2007) and NumPy (Harris et al., 2020).

RMSD and B-factor

In order to take into account the experimental uncertainty, we devised a method to downweight distance contributions from residues with high B-factor to RMSD. We weighted the atom distances by B-factor of the Cα carbon:

Where B is the B-factor of the Cα carbon in the CDR and thr is a threshold that determines if the B-factor is considered high. In our case, we considered that threshold to be 50. Then, we added those weights in the CDR as:

where d is the distance.

Contacts

For the analysis of contacts, we added hydrogens to the ligands with RDKit (Landrum et al., 2025) and Biotite (Kunzmann et al., 2023) and to the protein with PDBFixer (openmm/pdbfixer, 2025; Eastman et al., 2024). Then, the contacts were assessed using PLIP (Adasme et al., 2021). We used the default distance parameters of PLIP to define the contacts. Then, we classified the contacting residues as pertaining to a CDR, its neighbourhood or another part of the protein. We considered a residue as being part of a CDR neighbourhood if any of its heavy atoms was at less than 8 Å from a residue pertaining to the corresponding CDR. We only considered the neighbourhoods for CDR1 and CDR3 as they are the largest CDRs and their surrounding residues might overlap with the neighbour regions of both CDR2 and the CDR4 region.

Classifier analysis

To assess the classifying power of the predictors, we formed 88 pairs of Nb-SMs, combining the unique Nbs and ligands in the dataset. All 11 Nbs from all complexes are unique in structure but complexes 5VM0, 5VM6 and 5VL2 all contain triclocarban and 8V9W and 8V9Z contain fentanyl. This leaves us with 8 unique ligands and 88 protein-ligand pairs. 11 of the pairs corresponded to true binders and 77 corresponded to decoy complexes, assumed to not have binding.

As this new dataset is rather small (<100 samples) we used Leave-One-Out (LOO) cross-validation (LOOCV). This strategy consists in training a classifier with data points and predicting the remaining one. For each of the possible training sets then we chose a threshold that provided the best Matthews Correlation Coefficient score. That threshold was then used to evaluate the test point. Statistics only consider the classification of the different left out points.

In addition, as the dataset is imbalanced (13% of true cases), we did not use the standard Receiver Operating Characteristic Area Under the Curve (ROC-AUC) score to assess the classifier performance. We chose to use Matthews Correlation Coefficient (MCC), Precision-Recall AUC (PR-AUC) and F1 score, as all of them use the number of false negatives in their scores.

Number of samples, seeds and recycles optimization

To assess the optimal number of samples, seeds and recycles for the predictors, we made extra predictions, constraining the following parameters.

• For the optimal number of samples we made predictions with 50 samples, 5 recycles and 1 seed.

• For the optimal number of seeds we made predictions with 5 samples and 5 recycles for 20 different seeds.

• For the optimal number of recycles we made predictions with 10 samples and 1 seed for 3, 5, 10, 15, 20, 25, 30 and 35 recycles.

Results

The softwares used in this benchmark include folding programs like AlphaFold3 (AF3) (Abramson et al., 2024a), Chai-1 (Chai Discovery et al., 2024), Boltz-2x (Passaro et al., 2025), (Protenix Team et al., 2026), as well as RosettaFold3 (RF3) (Corley et al., 2025) and OmegaFold (Wu et al., 2022). We also added the docking software FlowDock (Morehead & Cheng, 2025) (see Methods).

In the benchmark, we decided to test the capabilities of all the aforementioned softwares for the prediction of the nanobody-only as well as the co-folded complex (nanobody and small molecule) structures for eleven complexes retrieved from SAbDab (Dunbar et al., 2014) (see Methods). We also compare how the quality of the predictions changes when the predicted sequences are in the training sets of the predictors.

Co-folding increases nanobody structure accuracy

To assess the overall accuracy of the predicted nanobody structures we calculated the RMSD of the Cα between the crystal structures and the predictions, both with and without co-folding. In Fig. 1A and Fig. S1 we observe that all predictors achieve satisfactory results, with most RMSDs below 2 Å. The best predictor for this dataset is Boltz-2x when co-folding the nanobody and the small molecule. It is closely followed by Protenix and RF3 and then by AF3 and Chai-1. As the former are more recent predictors, the field seems to show a slow systematic improvement. FlowDock provides the worst results among the compared softwares. Even when prompted with a structure from Chai-1 as template, it is not able to reproduce the same accuracy as Chai-1. This could be due to the focus of FlowDock on blind docking and not structure prediction. OmegaFold produces better results than FlowDock but worse than the more recent co-folding methods, rendering OmegaFold less recommendable for this task.

Figure 1.

Comparison of protein structure accuracy for different predictors. (A) Comparison between nanobody Cα RMSD in co-folding predictions versus only protein ones. (B) Effect of the presence of the complexes in the training set of the predictors to the RMSD of co-folded structures. The scatter points in this figure and subsequent ones show the values obtained for every predicted sample for each complex.

Ligand co-folding significantly increases the predicted structure quality for all the methods capable of performing it. This phenomenon has also been found by Hitawala and Gray (Hitawala & Gray, 2025) in AF3 predicting antibody-protein antigen systems.

Because the benchmarking dataset contains structures present in the training sets of several of these deep learning predictors, we assessed whether prediction accuracy is inflated for sequences seen during training. When co-folding is enabled (Fig. 1B), all predictors except RF3 yield significantly more accurate structures for training-set sequences than for held-out ones, consistent with memorization effects previously reported by Škrinjar et al (Škrinjar et al., 2026). In contrast, when structures are predicted without co-folding (Fig. S2), training-set membership does not significantly improve accuracy; notably, FlowDock performs worse on sequences included in its training set. Together, these results indicate that the apparent gain from training-set exposure is driven by the co-folding regime, and imply that performance on novel sequences, such as those generated in protein design, can be lower.

Accurate binding site location, but poor ligand orientation

Having studied the accuracy of the nanobody structure, the next analysis assesses the accuracy of ligand positioning. For that, we calculated the centre of mass distance between the predicted and crystal ligands, and the RMSD of ligand atoms to account for ligand orientation. In Fig. 2A, we observe that the predictors place the ligands centre of mass with a deviation of 5 Å, which indicates a moderate accuracy in pose predictions. However, some complexes have ligand distance as high as 20 Å. For the ligand RMSD, the overall results show mean values ranging from 2 Å to 10 Å, reflecting poor ligand orientation capabilities. Regarding the wide range of accuracy values, we did not find any clear ligand-dependent chemical descriptor that correlated with binding site distance or ligand RMSD. Analyzing the specific performance of the predictors, Protenix provides the most accurate results, outperforming the other methods, especially in ligand RMSD. It is closely followed by Boltz-2x, AF3 and RF3. FlowDock performs the worst compared to other predictors in ligand placement. Although a better performance might be expected since it is a dedicated docking program, its accuracy might be limited due to the poor accuracy in the predicted protein backbone.

Figure 2.

Ligand placement precision assessed by binding site distance (left column) and ligand RMSD (right column) for different predictors. (A) Comparison of ligand placement for different predictors. (B) Effect of the presence of the complexes in the training set of the predictors on ligand placement. Binding site distance is defined as the distance between the center of mass of the experimental and predicted ligands.

Results in Fig. 2B clearly indicate a relationship between the inclusion of a nanobody in the training set and the accuracy of the ligand placement, consistent with the discussion in the previous section. It is interesting to note that AF3 has the lowest dispersion of ligand placement and orientation for data outside the training set, showing that the memorization effect is heavier on post-AF3 methods. Maybe, efforts of the predictors in surpassing AF3 led to an overfitting of the models. Analysing individual nanobody cases in Fig. 3 and S3, we observe some interesting behaviours. 8FTG, 8V9W and 8V9Z trick some predictors into placing the ligand in the wrong position. In 8FTG, the ligand is bound by two tyrosine residues through pi-stacking, placed next to CDR3. Only AF3 and Boltz-2x place the ligand in the correct spot, the other softwares position it outside of, but still near, CDR3. Similarly, for 8V9W and 8V9Z all predictors except for AF3 place the ligand near CDR3 while its correct position is below CDR1. We tested if the protonation state of fentanyl was relevant in that regard, and for prediction accuracy in general (Fig. S4). The results showed no significant difference when that information was added and hence, we use the unprotonated version of fentanyl throughout this study.

Figure 3.

Visualization of the best predicted structures for a subset of nanobodies. Predictors shown (from left to right): AF3, Chai-1, Boltz-2x, Protenix and RF3. Experimental structures are shown with transparency for comparison. CDR1 in blue, CDR2 in pink, CDR3 in orange and CDR4 in grey.

Other interesting complexes are 7RGA and 7TJC. For the former, it is worth mentioning that it is the best predicted complex from those outside the training set of the predictors. Its centre of mass distance is identical to the experimental, and the only difference in the ligand is in the orientation of a carboxylic tail from methotrexate. This slight change in ligand orientation affects CDR1 by opening it slightly compared to the experimental structure. For 7TJC, an enhanced version of this phenomenon is observed, where chloramphenicol ligand is placed in the correct binding site but its orientation is wrong. The most exaggerated case happens with AF3, where CDR2 and CDR3 predicted structures are also affected. Protenix and RF3, although positioning the ligand mostly correctly, still show notable differences in the CDRs. This showcases that ligand placement has a profound effect on CDR prediction, which remains a long-standing challenge in antibody and nanobody structure modeling (Valdés-Tresanco et al., 2023; Yin & Pierce, 2024; Fromm et al., 2025; Hitawala & Gray, 2025; Sánchez-Marín et al., 2026).

CDR length determines accuracy

The previous results, specifically the Cα RMSD, provide an insight into the accuracy of the predictors for the protein structure. However, as seen in the last section, the binding process occurs mainly near the CDRs and profoundly affects their structure. Moreover, the framework regions (FWRs) follow a rigid β-sheet pattern and high sequence conservation. Therefore, one might expect to see less accuracy in the CDR RMSD. In Fig. 4 we show the results for that analysis.

Figure 4.

CDR RMSD for different structure predictors. (A) Effect of co-folding on CDR predictions. (B) Effect of inclusion in the training set on CDR predictions.

In Fig. 4A and Fig. S5, CDR3 has the highest RMSD score out of the 3 CDRs. This is to be expected since it commonly has the largest sequence, and thus it is more flexible and difficult to model. CDR2 has the lowest RMSD, as it is also the shortest CDR.

Ligand co-folding significantly decreases the RMSD of CDR1, and of CDR2 to a lesser extent, but not of CDR3. In contrast, in Fig. 4B the inclusion in the training set modulates the accuracy of CDR3. As CDR3 is not the binding site for the presented small molecules, it is not affected by ligand presence. However, as it is the biggest disordered region of the nanobody it is also the most difficult to predict correctly (Valdés-Tresanco et al., 2023; Yin & Pierce, 2024; Fromm et al., 2025; Hitawala & Gray, 2025; Sánchez-Marín et al., 2026).

CDR1 is the main contributor to binding

To assess how much each CDR contributed to ligand binding, we calculated the number of contacts that each CDR and CDR neighbourhood had with the ligand (see Methods). We do not report results for FlowDock because, as it has been shown, its results are not on par with the co-folding predictors.

The results in Fig. 5A and S6 indicate that CDR1 has significantly more interactions with the ligand than CDR2 and CDR3. This is consistent across predictors and the crystal structures, with no significant differences between these three CDRs.

Figure 5.

Analysis of the number of CDR and CDR neighbourhood (nbr) contacts. (A) Number of contacts in predictions from the best co-folding predictors and crystal PDB structures. (B) Comparison of the contacts predicted with the experimental structures. We report the recovery for hydrophobic contacts and hydrogen bonds, and the difference in hydrogen contact number. (C) Representations of the contacts of four nanobodies. Ligands in green, CDR1 contact residues in blue, CDR2 in magenta, CDR3 in yellow and CDR4 region in grey.

The CDR4 region plays a role as important as CDR2 in binding small molecules. It is a loop situated in FWR3, between CDR2 and CDR3 (Tabares-da Rosa et al., 2019b; Fanning & Horn, 2011). Its importance was already noticed by Fanning and Horn (Fanning & Horn, 2011) for nanobody 7RGA in its role to binding methotrexate and even aiding in the stabilization of CDR1. Figure S7 indicates that 6ITQ, 8V9W and 8V9Z, and 8H7I also have important contributions from the CDR4 region in binding cortisol, fentanyl and quinalphos, respectively.

In addition to the amino acids surrounding the CDRs, there are more distant residues that also interact with the ligand (‘other’ column in Fig. 5A). To check to which amino acids this field refers to, some structures with high contributions of ‘other’ residues have been visually inspected using PyMOL (Fig. 5C). Surprisingly, most of the ‘other’ residues correspond to Val78, just below the CDR4 region. This residue forms the bed of the binding site and has been described as a binding residue in quinalphos and fentanyl targeting nanobodies (Gallant et al., 2024; Li et al., 2023). This is also the case for Arg73 in 7RGA, which is not mentioned in the original publication. Furthermore, in some of the structures, we have found terminal residues that contribute to ligand binding, such as residue Val4 in 7RGA (light blue) and triclocarban-targeting nanobodies (5VL2, 5VM6, 5VM0) or Leu4 in 8H7I (light yellow). These interactions were also described in the articles describing triclocarban and quinalphos (8H7I) targeting nanobodies (Li et al., 2023; Tabares-da Rosa et al., 2019b). This reinforces the idea that, in addition to the CDRs, other residues surrounding the ligand also play important roles in nanobody–epitope binding.

In Fig. 5B, we analyse which interactions are predicted compared to the ones in the experimental structures. Hydrophobic interactions are recovered roughly 20% of the time, with great variation across complexes and predictors. Hydrogen bonds are also not recovered in the majority of nanobodies. This indicates that current prediction methods are able to recover the general contact landscape, but fail at the nuances of interactions. The wide spread in the distributions showing the difference in the number of predicted hydrogen bonds indicates individual case performance is inconsistent across predictors.

Confidence scores do not correlate with structure accuracy, but might discern binders from non-binders

We next wondered if the predictor confidence parameters could inform about the quality of the model. These parameters, often measured with pLDDT, iPTM and PAE, are of great interest when making blind predictions, as protein designs can be scored by them, if they are reliable.

First, we analysed 5 metrics from AF3, Boltz-2x, Protenix and RF3. We do not analyse the results from Chai-1, as it does not report pLDDTs nor PAE values and it underperforms in accuracy compared to the other predictors. We analysed the general score provided by each predictor, the CDR pLDDT and iPTM to measure the confidence of binding residues and ligand pLDDT and ligand PAE for the confidence in the ligand. In Figs. S8-S11 we report the correlation of these confidence scores with the accuracy parameters measured in the previous sections. Surprisingly, the results show low to null correlation for Nb-SM complexes.

We also tested the accuracy of Boltz-2x’s affinity predictions. In Figs. 6A and 6B we compare the correlations of the affinity predictions for each Nb-SM complex against the number of ligand atoms and the experimental K_d values, respectively. We show that Botlz-2x affinity predictions depend mostly on the size of the predicted molecule, with null correlation with experimental results. Furthermore, given that the predicted affinities should approximate the log10 of IC50 values and that nanobodies generally display high affinities, negative values would be expected; nevertheless, many predictions are positive, which is incompatible with the nanomolar affinities characteristic of this class. Hence, this metric is not appropriate for Nb-SM complexes. We speculate this is because of the broad ranges of IC50 values in the training set.

Figure 6.

Confidence scores across predictors. (A) Correlation of Boltz-2x affinity predictions and ligand atom number. (B) Null correlation between Boltz-2x and experimental Kd values for the predicted complexes. (C) Matthews correlation coefficient (MCC) for different predictors and metrics when classifying the 11 binding complexes from 77 non-binding decoy complexes. Values for MCC range from −1 to 1, where −1 indicates a completely incorrect classification; 0, a random one; and 1, a perfect one. Score corresponds to the global score provided by each predictor. (D) Distribution of affinity binary metric from Boltz-2x for 11 binding (WT true) and 77 non-binding (WT false) complexes.

Even if confidence or affinity measures did not predict the properties of Nb-SMs, we decided to test the classifying power of the predictors at discerning true from false binders, as similarly done by Teufel et al (Teufel et al., 2023). Hence, we formed 88 pairs of Nb-SMs, combining all the different Nb structures with all the unique ligands in the dataset (see Methods). These pairs contained the 11 true binders and 77 new decoy complexes. Given the selectivity and specificity of nanobodies, we assume that these new decoys are not binders. We used a Leave-One-Out cross-validation and the determination of an optimal threshold for the training set (see Methods) to assess the classifying capabilities of the different confidence metrics. We report the Matthews Correlation Coefficient (MCC) in Fig. 6C and the Precision-Recall AUC and F1 score in Fig. S12, with all the classifying scores leading to the same conclusions. In Fig. 6C we observe modest results, with not much classifying power. Still, some metrics do better than random (MCC = 0) and might be able to aid in the hit-to-lead design strategies. These metrics are Protenix ligand PAE and pLDDT, Boltz-2x CDR and ligand pLDDTs and RF3 iPTM and general score. We also tested Boltz-2x’s affinity binary metric (Fig. 6D) and affinity (Fig. S13) in this classification task, but we found no difference between the values of binding and non-binding complexes.

Finally, we tested how the decoy structures differ from the experimental ones in terms of ligand placement and CDR accuracy. Figure S14 reveals that there is a significant ligand displacement in decoy structures, with less notable changes in CDR1 and CDR2. Due to these results, it is not surprising that ligand related confidence metrics are the best classifiers to discern binders from non-binders. In addition, the differences shown in CDRs indicate that structure-related thresholds could be useful in design. This approach has already been used in some works, like NISE from Fry et al (Fry et al., 2025a)., where they use ligand and protein RMSD criterias, jointly with pLDDT metrics, to design small molecule binders.

Increasing number of samples or seeds improves accuracy but recycles doesn’t

Finally, we assessed how the accuracy of Boltz-2x and Protenix evolves with the number of samples, seed and recycles. These indicate which are the best parameters to obtain the most cost-efficient results. For this analysis, we chose Boltz-2x and Protenix as they provided the best structural results and the best classifier capabilities. Figures 7 and S15-S17 show the effect of the number of samples, seeds and recycles in the predictions of ligand binding site distance and RMSD and CDR1 RMSD.

Figure 7.

Effect of diffusion sample size on Boltz-2x and Protenix accuracy on a subset of complexes. Results are obtained using the perfect oracle approach, where the best sample for each PDB is considered. from left to right, the plots indicate the effect of sample size on ligand binding site distance, ligand RMSD and CDR1 RMSD.

Ideally, results would be reported by selecting the most accurate structure according to the predictor’s own confidence score, reflecting a realistic usage scenario. However, the confidence scores lack correlation with structural accuracy (Figs. S8-S11), and hence cannot be used as reliable selection criterion. We therefore adopted a ‘perfect oracle’ approach, in which the most accurate structure is identified by direct comparison with the crystallographic complex. In Fig. 7 and Fig. S15 we observe that all of the reported metrics stabilize at around the sampling of 10–15 structures for both Boltz-2x and Protenix, with the exception of 1QD0 and 5VM0 under Protenix, for which ligand distance and ligand RMSD, respectively, continue to decrease across the full sampling range. Accordingly, 10 samples represent a reasonable trade-off between accuracy and computational cost.

In Fig. S16, the number of seeds also increases the accuracy of the predictions, as mentioned in the literature (Hitawala & Gray, 2025; Abramson et al., 2024b). We observe that the number of seeds does not affect the accuracy as much as the number of samples across complexes. However, ligand RMSD is the metric that benefits most from the increase in the number of seeds. In that regard, we recommend using 5 different seeds.

Regarding the number of recycles, previous literature (Abramson et al., 2024a; Fry et al., 2025b) already discussed that a high number of recycles increases the accuracy of the results. However, Fig. S17 shows that none of the studied accuracy metrics improves with the number of recycles, sometimes even achieving worse results. Hence, increasing the number of recycles above 5 is not recommended, as the computational cost rises without any benefit in structure precision.

Discussion

Across this study we have tested the performance of state-of-the-art protein structure predictors to a small dataset of Nb-Small Molecule complexes. The results demonstrate that current models predict the general nanobody scaffold and CDRs with considerable accuracy, especially in co-folding predictions. In contrast, ligand placement and orientation remain a significant challenge. In addition, the predicted structures recapitulated correctly the residues contributing to binding both from CDR and non-CDR regions. Despite that, the recovery of specific binding interactions was minimal.

A pronounced memorization effect was also identified in these tasks. This effect likely contributes to the errors observed in ligand pose prediction and highlights the limitations of current models to reason about binding configurations. In addition, memorization discourages the use of these models in design tasks to predict binding poses for novel binders. Other types of computational modeling or experiments should be done before determining a final binding mode.

Confidence metrics were occasionally able to discriminate native structures from decoy configurations. This is useful in design pipelines, as this is a fast scoring that can be done already with the prediction of a structure, which includes the ligand effect on the binder and cannot be appreciated with current docking methods. However, the same metrics did not correlate with prediction accuracy. In the same line of thought, the binding affinity metrics of Boltz-2x were unable to reproduce the known affinities of the complexes. We attribute this, in part, to the fact that nanobodies typically operate in the nanomolar affinity range, a set of values that may be poorly sampled in the heterogeneous datasets used for training. Additionally, the binding mode of nanobodies differs substantially from that of conventional globular proteins, further hindering affinity estimation.

All these results lead to the interpretation that prediction models are able to reproduce the broad characteristics of nanobody-small molecule recognition, capturing general binding geometries and CDR loop conformations but failing to recapitulate the specific chemical interactions. This is most likely a lack of refined chemical and physical knowledge in the current model architectures. The limitations of the models are made more patent in this study because of the difficulties of predicting binding to flexible regions and because our dataset is underrepresented in the training set of the predictors.

A notable limitation of this study lies in the small size of the analyzed dataset, constrained by the low availability of nanobody-ligand crystal structures. Despite this, our findings lay important groundwork and highlight the potential of computational structure prediction for nanobody design in the context of small molecule binding.

Editor: Professor Whitehead

Author contributions

Berta Bori-Bru (Conceptualization [supporting], Data curation [lead], Formal analysis [lead], Investigation [equal], Methodology [equal], Writing - original draft [lead], Writing - review & editing [supporting]), Juan-Pablo Salvador (Conceptualization [equal], Funding acquisition [equal], Investigation [supporting], Supervision [equal], Writing - review & editing [equal]), Ramon Crehuet (Conceptualization [equal], Formal analysis [supporting], Funding acquisition [lead], Investigation [equal], Methodology [equal], Supervision [lead], Writing - original draft [supporting], Writing - review & editing [lead]).

Conflict of interest

None declared.

Funding

This work was supported by the Ministerio de Ciencia, Innovación y Universidades under grant FPU23/2698, and projects PID2022-138040OB-I00 and PID2024-155678OB-C21; by European Union’s Horizon 2020 research and innovation programme under project TAME (HORIZON-MSCA-DN-101119596). J.P.S. and R.C. belong to a consolidated research group (Grup de Recerca) of the Generalitat de Catalunya and has support from the Departament d’Universitats, Recerca i Societat de la Informació de la Generalitat de Catalunya (Reference: 2021 SGR 00408, and 2021 SGR 00476). CIBER-BBN is an initiative funded by the Spanish National Plan for Scientific and Technical Research and Innovation 2013–2016, Iniciativa Ingenio 2010, Consolider Program, CIBER Actions are financed by the Instituto de Salud Carlos III with assistance from the European Regional Development Fund. Some calculations described in this work were carried out at the Centre de Supercomputació de Catalunya (CSUC).

Data availability

The data underlying this article are available in Zenodo, at https://doi.org/10.5281/zenodo.18184470. This data was generated using an automated pipeline available at https://github.com/QTC-IQAC/predictorVerse.