What does AlphaFold3 learn about antibody and nanobody docking, and what remains unsolved?

ABSTRACT

Antibody therapeutic development is a major focus in healthcare. To accelerate drug development, significant efforts have been directed toward the in silico design and screening of antibodies for which high modeling accuracy is necessary. To probe AlphaFold3’s (AF3) capabilities and limitations, we tested AF3’s ability to capture the fine details and interplay between antibody structure prediction and antigen docking accuracy. With one seed, AF3 achieves a 10.2% and 13.3% high-accuracy docking success rate for antibodies and nanobodies, respectively. AF3-like models Boltz-1 and Chai-1 achieve 4.08% and 0% high-accuracy rates for antibodies, and 5% and 3.33% for nanobodies, respectively. With twenty seeds, AF3 achieves a median unbound CDR H3 RMSD accuracy of 2.9 Å … and 2.2 Å … for antibodies and nanobodies, respectively. Both AF3-like models Boltz-1 and Chai-1 improve further on antibodies (2.08 Å … and 2.71 Å …, respectively), but do poorly on nanobodies (3.78 Å … , 3.63 Å …). CDR H3 accuracy boosts AF3 complex prediction accuracy, with antigen context improving CDR H3 accuracy, particularly for loops longer than 15 residues. Combining ipTM-HA and I-pLDDT with $\mathrm{\Delta }{G}_B$ improves discriminative power for correctly docked antibody and nanobody complexes. However, AF3’s 65% failure rate for antibody and nanobody docking (with single seed sampling) demonstrates a need to further improve antibody modeling tools.

Introduction

Antibodies (Abs) play a critical role in the immune system, and the development of antibody and nanobody therapeutics is of major interest due to their ability to target cancer, autoimmune, cardiovascular, and infectious diseases. Therapeutic advantages include their soluble nature, tunable affinity, high tolerance by the human body, and manufacturability.¹ The antigen (Ag) binding interface of an antibody (nanobody) is composed of six (three) hypervariable loops, called the complementarity determining region (CDR) loops. The third loop on the heavy chain of the antibody (CDR H3) is particularly diverse and typically has the highest number of contacts with the epitope.² CDR loops sometimes undergo conformational changes upon binding to an antigen.³ Designing antibodies is challenging, primarily due to potential off-target effects⁴ and the substantial time and resources required for developability testing.¹ Due to the flexibility and importance of antibody CDR loops, modeling structural movement and docking is highly valuable, and significant effort has been put into developing antibody and antibody-antigen complex structure predictors.^5,6

Traditional Rosetta-based antibody-antigen docking algorithms use ensembles of homology models of antibodies, and sampling of rigid backbones, loop conformation, and ${\mathrm{V}}_H$-${\mathrm{V}}_L$ relative orientations.⁵ This general protocol had a 20% rate of successfully docking antibody-antigen complexes.⁷ (Docking accuracy is quantified via binning the fraction of native residue-residue contacts, interface and ligand RMSDs to the native structure into incorrect, acceptable, medium and high-accuracy categories by standards at the Critical Assessment of Protein Interactions (CAPRI) blind challenge, and have since been condensed into a single correlated score: DockQ, where a higher score defines better accuracy.)^8,9 Several other physics- and structure-based Ab docking methods have been published with similar performance.^10–13 While these methods are generalizable, the calculations are time consuming. Success rates are also limited by the ability to accurately model CDR H3.

Machine learning methods are faster and have significantly improved antibody-antigen and nanobody-antigen complex prediction. There are a variety of methods, with some focusing on only structure prediction or docking,^14–18 while others combine the tasks.^19–21 Tested architectures have included convolutional neural networks, transformers, diffusion models, and normalizing flow models for structure and complex prediction.^{14,16,17,20–24} While these methods have improved protein-protein docking success rates, AlphaFold2.3-Multimer (AF2.3-M) still had a poor 20% success rate for antibody-antigen docking.^7,25

While AF2 and the sub-series of models that were based on the AF2 architecture (AF2.x-M) established a robust algorithm for predicting structure from processed sequence context, the limitations in docking and structure prediction of some protein families, e.g. antibodies and nanobodies to antigens, have led to developments focused on improving the processed sequence context and sample diversity. Multiple Sequence Alignment (MSA) sub-sampling can extract conformational change information from sequence data,²³ while massive sampling with increased diversity via tuned dropout rates performed the best in CASP15.²⁶ AF3 is a culmination of many of these methods.

Until AlphaFold3 (AF3), the highest reported success rate for antibody docking was 43% by AlphaRED,²⁵ a hybrid model using AlphaFold2-Multimer (AF2-M) predicted complexes and confidence measures with Rosetta-based replica exchange docking. Then in May 2024, with AF3 being trained on the same antibody dataset as AF2-M,^19,27 DeepMind reported a notable 60% success rate for AF3 when 1,000 seeds were sampled.

To understand the source of improvement and where AF3 still has limitations, here we thoroughly assess AF3’s ability to dock antibody-antigen and nanobody-antigen complexes and predict unbound antibody and nanobody structures. To discern the effects of the limited experimental structures provided in the PDB, we study the interplay of the CDR H3 loop and Ab-Ag (and Nb-Ag) docking using structures from a redundancy-filtered bespoke dataset after the 2021 AF3 training cutoff. With this dataset we similarly benchmark newer AF3-like models Boltz-1²⁸ and Chai-1.²⁹ Using the DockQ score, we conduct an uncertainty quantification analysis to determine the best combination of confidence metrics for antibody and nanobody complexes. For reproducibility and standardized benchmarking, we make the benchmark data and code available at https://github.com/NooriFatima/AF3_AbNb_Benchmark.

Results

AF3 outperforms previous state-of-the-art antibody docking methods

To compare AF3 and AF3-like models (Boltz-1, and Chai-1) to previous state-of-the-art models, we first curated a benchmark set of bound and unbound antibodies and nanobodies from SAbDab structures filtered by AF3’s training set cutoff date followed by quality and sequence and structure redundancy filtering, resulting in 49 (bound), 13 (unbound) antibodies, and 60 (bound), and 10 (unbound) nanobodies (details in Methods). We limited our analysis to Boltz-1 as Boltz-2’s training cutoff date (June 1, 2023) is after AF3 (September 30, 2021). For every target sequence, we ran three seeds in AF3 to account for the variance generated by the diffusion model. The AF3 paper reports an overall success rate of 60% for Ab-Ag docking, however this is with using 1000 seeds.²⁷ On the AF3 server we were limited by the number of jobs per day from running a greater number of seeds.

We sought to compare AF3 to its predecessors (AF2.3-M and AF2-M) AlphaRED (which previously achieved the highest reported Ab-Ag docking success rate), and newly released models with similar architectures: Boltz-1 and Chai-1. We compared AF3, AF2.3-M, Boltz-1, and Chai-1 using our curated benchmark using the top-ranked decoy from the first seed, as AlphaRED is run on a single decoy generated from AF2.3-M (ranking protocol for AF3, Boltz-1, and Chai-1 in Methods). As seen in Figure 1(A), AF3 improves over AF2.3-M in both the percentage of high-accuracy docking (DockQ $\ge$ 0.80) and overall docking accuracy (DockQ $>$ 0.23). For high-accuracy docking, AF2.3-M has a low success rate of 2.4%, while AF3 has a considerably high accuracy success rate of 10.2%, AF3’s overall success (DockQ $>$ 0.23) rate of 34.7% also improves on AF2.3-M’s 23.4% success rate. Surprisingly, Boltz-1 and Chai-1 perform poorly compared to both AF3 and AF2.3-M on our docking benchmark, with overall success rates of 20.4% for both, and high-accuracy success rates of 4.1% and 0% respectively. The Boltz-1²⁸ and Chai-1²⁹ papers report higher success rates; besides the different test sets, we used the default setting of three recycles, while the paper evaluations used 10 and 4, respectively. Increased recycling has been found to improve performance,¹⁹ although the ablations in the Boltz-1 paper show minimal impact above three recycles.²⁸ Further, the three models vary in MSA depth, also shown to be a key factor in docking prediction quality in antibody-antigen docking.³⁰

Figure 1.

The success rates in antibody (Ab) and nanobody (Nb) docking for state-of-the-art models. (A) Performance of AF3 on antibody-antigen docking (N = 49) and nanobody-antigen docking (N = 60) against AF2.3-M, Boltz-1, and Chai-1 with curated dataset of recent novel complex structures. DockQ scores for the bound antibodies and nanobodies are binned into incorrect, acceptable, medium, and high categories based on CAPRI classifications, and represented in a stacked column plot.⁹ (B, C, D) Ab protein complex structures of example incorrect, acceptable, and high accuracy predictions. Experimental crystal structures are gray, predicted antibodies are blue, the predicted antigen is seagreen; crystal CDR H3 loop (defined by Chothia numbering) is dark gray and predicted CDR H3 loop is dark blue.

In Figure 1(B-D), we show docked structures with highlighted CDR H3 loops with increasing docking accuracy to illustrate failed and correct docking predictions in different CAPRI quality categories. In the acceptable docking accuracy example, the CDR H3 loop structure is incorrect (5.3 Å … RMSD). In the high-accuracy case, the CDR H3 loop structure is almost exactly correct (0.76 Å …). To understand the difference between docking performance of AF3 and the related models (Boltz-1, Chai-1), we inspected an example complex that AF3 docks with high accuracy. In Figure S1A, B, C, D we show one example docked complex by AF3, AF2.3-M, Boltz-1, and Chai-1 respectively against the ground truth structure. AF2.3-M, Boltz-1 and Chai-1 all dock the complex inaccurately, despite Boltz-1 and Chai-1 having sub-angstrom accuracy on the CDR H3 loop. In these cases, we find that all three methods struggle with accurately predicting the antigen, making docking difficult.

For nanobodies, AF3 achieves a 13.3% success rate for highly accurate complexes, with a lower overall success rate (31.6%). Boltz-1 has a better overall success rate (23.3%) and high-accuracy success rate (5.0%) for nanobodies than on antibodies, while Chai-1’s nanobody performance decreases to 15.0% overall and 3.3% for high-accuracy docking. Docked Nb-Ag complexes shown in Figure S2 illustrate examples of incorrect and highly accurate nanobody-binding modes and the resulting structures. Again in the nanobody examples, the high-accuracy docking case has a sub-angstrom CDR H3 RMSD, while the medium and acceptable cases show more error (over 2 Å … RMSD).

We then compared AF3 to AlphaRED and AF2-M on the DB5.5 docking dataset. This alternate test set was chosen (acknowledging that there is contamination of targets in the AF training sets) because it would be time-consuming to run the computationally intensive AlphaRED refinement stages on the new benchmark set. Figure S3 shows that while the overall success rates of AlphaRED and AF3 are close (44.3% and 50.8% respectively), AF3 has more high-accuracy successes compared to AlphaRED. The fact that AF3’s overall success rate improves on the DB5.5 docking dataset (50.8%) over our curated, independent benchmark set (34.7%) shows that AF3 perhaps unsurprisingly achieves a higher success rate on targets that may have been in its training set.

High docking accuracy requires high CDR H3 accuracy, but not vice-versa

The antibody maturation process improves binding affinity for an expressed antigen³¹ so that an unbound antibody can target the antigen by complementing the epitope.³² As the hypervariable H3 loop often makes the majority of contacts between the antibody and antigen,² in traditional methods, correctly modeling the CDR H3 loop has been pivotal in improving docking quality.³³ This effect may be stronger for nanobodies, as they do not have a light chain with which to split the contacts to the antigen.

To understand the correlation between modeling the CDR H3 loop and docking accuracy in AF3 predictions, we measured the CDR H3 RMSDs and DockQ scores for antibody and nanobody complexes across three seeds and five diffusion samples (decoys) per target from our curated benchmark and visualized their joint and marginal distributions (Figure 2). The marginal DockQ distribution ($\mathit{y}$-axis on right) is bimodal, implying that AF3 often docks a complex with high accuracy, or incorrectly. A CDR H3 loop RMSD threshold of 1 Å … has previously been found to be sufficient to support rigid-body docking of Ab-Ag complexes.³³ In the joint distribution, the predictions that are docked with high accuracy (DockQ $\ge$ 0.8, corresponding to the green box) are heavily clustered within the bounds of CDR H3 RMSD $\le 1$ Å (the pink box). However predictions with CDR H3 RMSD $\le 1.0$ Å (pink box) are scattered over the full range of possible DockQ scores, implying that a correct CDR H3 loop is insufficient to push the prediction to sample the correct antigen-binding interface.

Figure 2.

Joint distribution of DockQ scores and CDR H3 loop RMSD of predicted antibody-antigen complexes with marginal distributions of both variables. CAPRI classification zones separated by dashed lines, with the high-accuracy docking complex region (DockQ $\ge$ 0.8) shaded in green, the sub-angstrom CDR H3 loop RMSD region shaded in pink, and the intersection shaded in purple. The conditional probability of the CDR H3 loop RMSD being less than 1.0 Å … given a highly accurate complex is the number of points in the intersection of both events (purple) over the number of total points with highly accurate docking (green). The conditional probability of a highly accurate complex given a less than 1.0 Å … H3 loop RMSD is the number of points in the intersection (purple) over the number of points in the sub-angstrom H3 loop RMSD region (pink).

To quantify this observation, we calculate the conditional probabilities. To examine the effect of one variable (either CDR H3 RMSD, or DockQ score) on the joint concentration of points defined by both variables (both CDR H3 RMSD and DockQ score), we begin by examining conditional probabilities above two specific thresholds, CDR H3 RMSD $\le 1.0$ Å (which has previously been observed to be critical for traditional antibody docking protocols) and a DockQ score $\ge$ 0.8 (high-accuracy CAPRI cutoff). The $\mathrm{p}(\mathrm{CDRH}3\mathrm{RMSD}\le 1.0$Å $|\mathrm{DockQ}\ge 0.8)$ is 86.0%, implying that a correct CDR H3 loop is also critical for high-quality docking predictions for AF3. Similarly for nanobodies, $\mathrm{p}(\mathrm{CDRH}3\mathrm{RMSD}\le 1.0$Å $|\mathrm{DockQ}\ge 0.8)$ is 84.7%. The converse does not appear to be true: the $\mathrm{p}(\mathrm{DockQ}\ge 0.8|\mathrm{CDRH}3\mathrm{RMSD}\le 1.0$ Å) is 55.8% for antibodies and 36.2% for nanobodies, showing that a correct CDR H3 loop does not guarantee high-accuracy docking.

To determine the effect of different thresholds of CDR H3 RMSD on docking success rate, we plot $\mathrm{p}(\mathrm{DockQ}\ge \mathit{D}|\mathrm{CDRH}3\mathrm{RMSD}\le \mathit{T}$ Å) where D takes the values of each CAPRI quality category cutoff (0.23, 0.49, 0.8) and T is varied along the $\mathit{x}$-axis over the range of all CDR H3 RMSDs in the dataset (Figure 3). For the incorrect cases, we plot $\mathrm{p}(\mathrm{DockQ}<\mathit{D}|\mathrm{CDRH}3\mathrm{RMSD}\le \mathit{T}$ Å), where $\mathit{D}=0.23$ is the DockQ cutoff for incorrect docking per CAPRI standards. The probabilities of successful docking drop sharply as CDR H3 RMSD increases over the first few angstroms. The probability of an acceptable-accuracy or better complex drops below that of the probability of an incorrect complex at a CDR H3 RMSD value of 3.1 Å … (antibodies), and 2.1 Å … (nanobodies). For nanobodies, Figure S4B shows the joint and marginal distributions, where the DockQ marginal distribution is also bimodal. Figure 3 shows similar trends to antibodies for the conditionals. Supplemental files (Files. S1, S2) contains the calculated probability distributions for users to be able to determine stringency of the CDR H3 RMSD cutoff needed for the acceptable probability of docking and dock score desired from their datasets.

Figure 3.

Conditional probabilities of docking accuracy per CDR H3 RMSD threshold for antibodies and nanobodies. the probabilities of correctly docked antibody and nanobody complexes are calculated for DockQ scores corresponding to acceptable ($\mathit{D}\ge 0.23$), medium ($\mathit{D}\ge 0.49$), and high-accuracy ($\mathit{D}\ge 0.80$) CAPRI categories, while the probabilities of incorrect complexes are calculated via DockQ score $<$ 0.23, covering a CDR H3 RMSD range from 0–10 Å … for antibodies and nanobodies.

AF3 outperforms AF2.3-M, AF2-M, and IgFold in predicting unbound Fv structures

Considering the impact of CDR H3 loop accuracy on overall docking success, we sought to evaluate AF3’s predictive accuracy for unbound CDR H3 loops. That is, we examined the accuracy when predicting antibodies alone, without antigen. To compare AF3 against previous state-of-the-art structure prediction models, we filtered IgFold’s curated benchmark of 196 unbound antibody variable fragments and 70 unbound nanobodies¹⁴ by the new training cutoff date used in AF2.3-M and AF3 (September 30, 2021), resulting in 63 antibodies and 23 nanobodies. For the comparison we used one seed with AF3, Boltz-1, and Chai-1 and used the top-ranked decoy (ranking methodology for each model in Methods) from the five decoys predicted, staying consistent with IgFold’s evaluation method.

Figure 4 shows CDR H3 accuracies. While AF2.3-M has the lowest median CDR H3 RMSD of the three previous state-of-the-art models at 2.83 Å … , for antibodies AF3 achieves 2.52 Å … . Boltz-1 improves the median further, to 2.08 Å … , while Chai-1 does not perform as well as the other diffusion architectures, achieving a median of 2.71 Å … . IgFold and AF2.3-M perform similarly (3.0 Å … for IgFold), with AF2.3-M performing better (2.83 Å …). Thus, AF3 has improved CDR H3 loop prediction by 0.31 Å … , and Boltz-1 by an impressive 0.75 Å … . AF3 does not significantly improve upon its predecessors for nanobody CDR H3 loops, decreasing median RMSD slightly from AF2-M’s 3.05 Å … performance to 3.01 Å … . Boltz-1 (3.78 Å …) and Chai-1 (3.63 Å …) fail to reach AF3’s level of performance, having CDR H3 median RMSDs closer to IgFold, albeit the statistical test’s ability to detect differences between model performance is limited by the small sample sizes for antibody and nanobody groups.

Figure 4.

Performance of AF3, Boltz-1, and Chai-1 on predicting unbound CDR H3 loop structures of 63 antibodies and 23 nanobodies compared to previous models. The letter-value plot represents the traditional quartiles of the data, with additional boxes representing additional density. (A) AF3 and Boltz-1 improve upon the average median to 2.52 Å … and 2.08 Å … , respectively. For reference, the RMSD between experimental unbound and antigen-bound antibody CDR H3 loops has a median of 0.5 Å … . CDR loops are all defined via the Chothia numbering scheme. (B) AF3 minimally improves the median of AF2-M (the lowest of IgFold, AF2-M and AF2.3-M) to 3.01 Å … . All CDRs are defined using the Chothia numbering scheme, and p $\le$ 0.05 corresponds to *, p $\le$ 0.01 corresponds to **, p $\le$ 0.001 corresponds to ***, and p $\le$ 0.0001 corresponds to ****.

To probe the effect of target similarity to the training set, we plotted CDR H3 RMSDs as a function of edit distance (substitutions plus indels, details in Methods) of concatenated CDRs from closest training point sample to the evaluated data point, including 133 additional antibody, and 47 nanobody structures solved between the AF3 and IgFold cutoff dates (July-September 2021) with very close or identical sequences to the training set. We observe a positive trend between edit distance and CDR H3 loop RMSD for antibodies for post-AF3-cutoff structures, but not for nanobodies (Figure S5A). There is a much heavier density of sub-angstrom CDR H3 RMSD nanobodies in the pre-cutoff group (20.3%) compared to the post-cutoff group (4.2%). Binning pre- and post-cutoff sets (i.e., small and large edit distances) (Figure S5B), there is a wide range of CDR H3 RMSDs for AF3 predictions on pre-cutoff proteins. The medians of pre-cutoff samples are lower (2.02 Å … (Abs), 1.80 Å … (Nbs)) compared to post-cutoff evaluation data points (2.52 Å … (Abs), 3.01 Å … (Nbs)).

The accuracy plateau reached by IgFold, AF2-M, and AF2.3-M suggests questions about the limit of accuracy possible for the loop, considering that it may simply be mobile in solution. AF3 and Boltz-1’s results prove that better predictions are possible. Further, a recent survey of 177 pairs of bound-unbound antibody complexes reported that in 70.6% of antibody CDR H3 loops, binding-induced conformational changes are under 1 Å.³ Figure 4 also shows the CDR H3 RMSD between antigen-bound and unbound antibodies (Xtal B-U column); these measurements suggest that antibody loop prediction may still be able to be far more accurate.

Antigen context affects antibody CDR H3 loop prediction accuracy

Using our curated benchmark set to observe how the CDR H3 loop’s structure is affected by antigen context, we compared the RMSD of all bound CDR H3 loops against all unbound CDR H3 loops, separately analyzing those of antibodies versus nanobodies. To further determine whether there are learned biases toward bound structures due to their over-representation in the training set, we also predict the experimentally bound structures without providing antigen context and then calculate their RMSDs with respect to the experimental bound conformation. We divide the RMSD into the global H3 RMSD, calculated by superposing heavy chains, and the local H3 RMSD, calculated by superposing only the CDR H3 residues. The global RMSD captures the loop shape and placement, while the local RMSD represents the loop’s shape only. As the generative nature of the model introduces high amounts of variation in the structures, the median CDR H3 loop RMSD can change from one seed to the next, so we first determine how many seeds are necessary for the median CDR H3 loop RMSD to minimally vary. We generated 100 seeds per target, calculated their RMSD, then generated a plot comparing the number of seeds to the variance in the medians as the number of seeds increase. We found that 20 seeds are sufficient (Figure S6). Upon visualization of per-seed RMSD distributions of all CDR loops and framework regions for antibodies and nanobodies (Figs. S7,S8,S9,S10), we found that unbound antibodies and nanobodies predictions have greater RMSD variation per-seed compared to their bound counterparts.

Figure 5(A) shows that providing antigen context improves CDR H3 loop shape and position prediction accuracy for antibodies. The local (1.99 Å …) and global (3.05 Å …) CDR H3 RMSD of predicted unbound structures are higher than that of local (1.67 Å …) and global (2.51 Å …) predicted bound structures, with significance of p = $3.50\times {10}^{-4}$, and p = $3.47\times {10}^{-6}$, respectively.

Figure 5.

Effect of antigen context and loop length on antibody CDR H3 loop prediction accuracy. (A) Top-ranked known unbound structures, bound structures predicted with antigen, known bound structures predicted without antigen with number of decoys annotated above letter-value plots. Global RMSD calculated after superposition of the ${\mathrm{V}}_{\mathrm{H}}$ domain; local RMSD calculated by superposing the loop residues only. (B, C) Effect of loop length and antigen context on antibody CDR H3 loop prediction accuracy. Top-ranked known unbound structures, bound structures predicted with antigen, known bound structures predicted without antigen. Short loops are defined as less than 10 residues, medium loops between 10 to 15 residues, and long loops longer than 15 residues, with number of decoys above each letter-value plot. p $\le 0.05$ corresponds to *, p $\le 0.01$ corresponds to **, p $\le 0.001$ corresponds to ***, and p $\le 0.0001$ corresponds to ****.

For nanobodies, Figure S11A shows that the local CDR H3 RMSD is improved by the antigen context when comparing median RMSDs of the known bound structures (1.13 Å …) to the predicted unbound structures (0.80 Å …) (p = $1.55\times {10}^{-5}$). The loop’s position (global RMSD) also improves when provided antigen context when comparing the known bound structures to both the known (p = $1.5\times {10}^{-3}$) and predicted unbound structures (p = $2.34\times {10}^{-5}$). The predicted unbound local CDR H3 RMSD decreases from 1.13 Å … to the known bound structure’s RMSD of 0.80 Å … , but the global CDR H3 RMSD only decreases from 1.88 Å … to 1.70 Å … ; demonstrating that for nanobodies, antigen context improves the loop’s shape to a greater extent than the position of the loop.

Previous studies report CDR H3 loop RMSD increasing as loop length increases due to the degrees of freedom granted to the loop.¹⁴ Locally, short antibody CDR H3 loops are minimally affected by the addition of antigen context, as the CDR H3 RMSD is already sub-angstrom (Figure 5(B)). For long loops however, bound structures (1.57 Å …) have lower local CDR H3 loop RMSDs compared to both known (3.34 Å …) and predicted unbound (2.78 Å …) structures with significances of p = $3.47\times {10}^{-6}$, and p = $3.47\times {10}^{-6}$, respectively. The difference in global CDR H3 loop RMSDs is more drastic; known bound structures have a lower median CDR H3 RMSD of 3.17 Å … , compared to known unbound structures (5.71 Å …) and predicted unbound structures (4.85 Å …), with significances of p = $6.74\times {10}^{-6}$, and p = $7.40\times {10}^{-3}$, respectively.

Nanobodies show no effect on short loops, as known bound and predicted unbound have sub-angstrom RMSDs locally and globally (Figure S11B). However, the medium-length loops surprisingly show a significantly higher (p = $8.75\times {10}^{-4}$) global CDR H3 RMSD for known bound structures (2.65 Å …) compared to known unbound structures (2.12 Å …), while long loops show improvement given antigen context. Upon investigation, we found that seven out of the 36 structures in the medium length group bind to the spike glycoprotein, an antigen seen during training. However, the common paratope for these structures is inside the receptor binding domain (RBD), which is accessible only in its open conformation.^34,35 AF3 repeatedly predicts the closed conformation, obstructing access to the correct spatial constraints to aid in CDR H3 prediction accuracy. Further, the spike protein has multiple sites amenable to antibody and nanobody binding.³⁴ Supporting this, we see that for antibodies and nanobodies, predicted structures whose antigens were seen during training have larger global and local CDR H3 RMSDs, as a large portion of the overlapping antigens are the spike glycoprotein (Figure S12). Thus we find that long loops benefit the most from added antigen context so long as the antigen is correctly predicted. Consistent improvement in CDR H3 loop accuracy in known bound structures compared to known bound structures predicted without antigen context implies that AF3 leverages biophysical context for better predictions.

A combination of predicted confidence interface metrics and Rosetta energies refines blind prediction scoring

Researchers need to know when to trust AF3 Ab-Ag and Nb-Ag structural models. Thus, we investigated confidence metrics and their respective cutoff values for scoring blind Ab-Ag and Nb-Ag docking using our curated benchmark set. We use the top-ranked prediction for twenty seeds for all targets to simulate the confidence filtration common in blind prediction ranking. Using twenty seeds helps ensure a sufficient dataset size for analysis.

We first analyzed the ipTM score between the heavy chain and antigen (ipTM-HA) for both antibodies and nanobodies (output by AF3), because we have found that AF3 has a strong calibration of the ipTM (Figure S13) as it is part of the training loss.²⁷ To effectively compare the discriminative power of single and combined metrics, we use the average precision metric (see Methods), with precision-recall (PR) curves and the average precision (AP) metric for antibodies and nanobodies (Figure 6). To determine the best threshold per confidence metric, we find where precision and recall are jointly maximized. For antibodies and nanobodies, ipTM-HA is the best single discriminatory metric, with an average precision of 0.458 (threshold of ipTM-HA $\ge$ 0.49) and 0.450 (threshold of ipTM-HA $\ge$ 0.40) respectively.

Figure 6.

Precision-recall curves comparing single and combined scoring metrics for Ab-Ag and Nb-Ag docking using AF3.

We conducted a similar analysis using Rosetta-based binding energies ($\mathrm{\Delta }{G}_B$) (calculations in Methods), as it has been used to reliably rank protein structures.³⁶ Due to the high structural accuracy of AF3, we calculated $\mathrm{\Delta }{G}_B$ directly on the predicted complexes. For both antibodies and nanobodies, $\mathrm{\Delta }{G}_B$ has low average precision scores of 0.188 and 0.178 respectively. Using precision and recall values, we determined the best $\mathrm{\Delta }{G}_B$ cutoff threshold to be −30 REU and 4 REU respectively.

In Figure 6, we show the same analysis for the interface-predicted local distance test (I-pLDDT) as I-pLDDT is part of the training loss for the full model,²⁷ and it has been reported to be a stronger discriminator than ipTM for AlphaFold2-Multimer^25,30 (Methods). We found that I-pLDDT performs the best after ipTM-HA for both antibodies and nanobodies, with average precisions of 0.368 and 0.336 and I-pLDDT cutoffs of 85 and 87, respectively.

We next analyzed the averaged pLDDT of the CDR H3 loop residues (avg. H3-pLDDT) (Methods), due to the correlation we determined between CDR H3 accuracy and docking for antibodies and nanobodies. This metric is the third best discriminatory metric, performing better for nanobodies with average precision scores of 0.287 and 0.316 for antibodies and nanobodies, respectively. The optimal cutoff is an avg. H3-pLDDT of 86 for both.

Combining scoring metrics provides the chance to tighten filtration and optimize the likelihood of sampling a correctly predicted complex. We determined the best metric combination for antibodies and nanobodies by conducting a joint optimization procedure (Methods). The resulting AP score is unaffected by shorter curves, so we directly compared single and combined metrics with the raw score.

The metric combination with the highest average precision score for antibodies is ipTM-HA and $\mathrm{\Delta }{G}_B$ (AP = 0.412) and I-pLDDT and $\mathrm{\Delta }{G}_B$ (AP = 0.743) for nanobodies, demonstrating that $\mathrm{\Delta }{G}_B$ provides useful complementary data to AF3’s confidence metrics to enhance discrimination. For antibodies, the optimal cutoff of ipTM-HA is 0.49 combined with a $\mathrm{\Delta }{G}_B$ cutoff of −47 REU. For nanobodies the combination metric cutoffs for I-pLDDT and $\mathrm{\Delta }{G}_B$ are 87 and −4 REU respectively. While the AP score of the best combined metric for antibodies is lower than that of ipTM-HA by itself, the optimal operating point of the combination ($\mathit{R}=0.8$, $\mathit{P}=1.0$ at $\mathrm{ipTM}-\mathrm{HA}=0.49$ and $\mathrm{\Delta }{G}_B=-47$) has higher precision (fewer false positives) than that of ipTM-HA alone ($\mathit{R}=0.88$, $\mathit{P}=0.94$ at $\mathrm{ipTM}-\mathrm{HA}=0.49$). For nanobodies, the AP score of the combined metric is higher than that of all other metrics. Other tested thresholds for combined metrics are shown in Figure S14, where discriminative power improves as threshold stringency increases. Datafiles with all tested single and combined metrics along with their precision-recall values and average precision scores have been provided so that users can determine optimal thresholds specific to their needs (Files S3, S4).

Discussion

AF3 builds on AF2’s performance by expanding capabilities toward general chemical structures by reducing emphasis on MSA processing and replacing the residue frame-based structure module with an atomic-precision diffusion model. We demonstrate that these architectural changes help AF3 dramatically improve the success rate of high-quality docked antibody-antigen from 2.4% (AF2.3-M) to 10.2%; however, AF3 (with one seed) still leaves 65% of the targets incorrect. Further, new AF3-like models, Boltz-1 and Chai-1, perform poorly on antibody and nanobody docking compared to AF3. From the bimodal distribution of DockQ scores seen in the marginal distribution for antibodies in Figure 2 and for nanobodies in Figure S4, we see that AF3 either has difficulty sampling the correct binding interfaces or generates high-quality complexes, however AF3-like models face difficulty in accurate antigen prediction, leading to poorer performance despite accurate CDR H3 loops (Figure S1). The AF3 paper notes that during training local structures are learned quickly, while global structures are learned at a slower pace.²⁷ The success rate of the model has been reported to increase to 60% when evaluating 1,000 seeds,²⁷ a finding similar to the AF2 massive sampling study where artificial diversity was injected by tuning dropout rates²⁶ The impressive rate of high-accuracy docked complex predictions coincides with the lower CDR H3 loop RMSD of unbound and bound antibodies and nanobodies as well as the observed improvement of long loop CDR H3 prediction accuracy when provided with antigen context, implying that AF3 has learned better docking biophysics.

Despite the improvement in CDR H3 loop RMSD, according to the findings of,³ the loop prediction accuracy should be able to improve further. The conformational ensembles in³ were derived from crystal structures which provide a limited view of protein flexibility as compared to NMR or molecular dynamics-simulated structures. However, bound crystal structures have been found to predominantly express CDR H3 loop conformations that optimize antigen binding³⁷ and unbound antibodies affected by crystal packing effects have been studied using molecular-dynamics to find that the predominant conformation is the bound conformation.^37,38 This result supports the theory of conformational selection, where a binding event results in the selection of one CDR H3 loop conformation out of a preexisting set.³⁷ Thus, these crystal-structure ensembles can approximate conformational ensembles. A limitation of our work is that after removing contaminated data with respect to AF3’s training set, an insufficient number of conformational ensembles remained, barring us from assessing AF3’s ability to generate accurate bound-unbound conformational ensembles of antibodies and nanobodies and thus the limits of its understanding of Ab-Ag and Nb-Ag binding.

AF3 outputs a well-calibrated bespoke rank-score in the prediction’s confidence files, but often decoys will have the same rank score, causing difficulty in evaluation. We evaluated the robustness of the confidence metrics generated by AF3 against a benchmark of sequences and structures nonhomologous to the model’s training data to see if prediction ranking could be improved using additional metrics of CDR H3 loops and biophysical interface energies. We find that combining Rosetta-based binding energies with ipTM-HA improves discrimination.

When we started this study, we were limited by the number of jobs available per day on the AF3 server and the small dataset size due to data contamination concerns. Our results show that while AF3 is considerably more accurate in modeling antibody and nanobody structures and docked complexes than previous approaches, there remains room for improvement of the 65% failure rate for both antibody and nanobody docking for single seed predictions; even greater improvement is needed in AF3-like open-source models. Researchers can now improve upon this baseline by running more seeds on a local AF3 setup the authors of AF3 have made generously available,³⁹ however we note that this improvement may not be replicable in Boltz-1 and Chai-1, as recent evaluations on antibody-antigen show conflicting results in improvements from multi-seed-based sampling for these models.^40,41 Our curated benchmark set (with predicted structures for AF3, Boltz-1, and Chai-1) is available so that new methods replicating AF3,^28,29,42 and additional seed sampling in local AF3 can be tested. We now have more powerful tools than ever for Ab engineering, which opens promising avenues to design and engineer Ab therapeutics and understand immunology better.

Methods and materials

Individual structure evaluation set

To create an immunoglobulin structure prediction dataset, we pulled structures from SAbDab (May 31, 2024 for Abs, June 4, 2024 for Nbs),⁴³ and temporally separated evaluation structures using the Sept. 30, 2021 training dataset date cutoff by AF3. We separated all antibody copies in the remaining PDBs, then conducted a quality filtration similar to,³ where we removed PDBs with a resolution $\ge$ 2.8 Å … , or had missing residues in the CDR loops by comparing atomic sequence and sequence residues using the Bio.Seq python package.⁴⁴ We generated ensembles of sequences using a 90% sequence identity cutoff and clustering using MMSeqs2.⁴⁵ Using these ensembles, we then used the Kabsch alignment algorithm to calculate the structural redundancy of variable heavy and light chains between pairs of structures.¹⁴ We kept pairs of structures that had heavy and light chain RMSD $\ge$ 1 Å … , but only one representative out of a pair of redundant structures. To ensure that we did not lose structures with high CDR H3 loop diversity despite similar heavy chain structures, we recovered structures from pairs with heavy chain RMSD $<$ 1 Å … , but CDR H3 RMSD $\ge$ 1 Å … . Finally, we filtered the remaining structures based on sequence redundancy against each other and AF3’s training set with a sequence identity cutoff of 99% and 95% respectively. We conducted MSA alignment on the heavy and light chains separately via Abalign,⁴⁶ and then a custom Python function to calculate the sequence identities. To ensure that we did not retain structures that had identical CDRs to the training set, we concatenated all CDRs of the structure, conducted an MMSeqs search against concatenated CDRs from the AF3 training set, and removed any benchmark sequences that were identical to training data CDR combinations. The number of structures at each step of this process is shown in Figure 7. We characterize the benchmark sequences via sequence identity of both concatenated CDRs and the CDR H3 sequence against the benchmark set and AF3’s training set, as well as antigen overlap and the distribution of light chain isotypes in the benchmark and training sets in Figure 8. We did not remove identical CDR H3 loops to the training data in order to retain structural differences granted by non-CDR H3 loops.

Figure 7.

(A) Dataset curation for evaluating antibody structure prediction, number of structures at each step. (B) Bins breaking down the number of bound and unbound structures in antibodies and nanobodies.

Figure 8.

Dataset characterization of benchmark set. (A) Distribution of benchmark concatenated CDRs sequence identity to closest match in training set. (B) Distributions comparing the sequence identity of benchmark CDR H3 sequences against the benchmark set and AF3’s training set. (C) Count-plot comparing the percent of kappa and lambda light chain subtypes in the benchmark set and training set. (D) Histogram comparing the sequence identity of antigens in the benchmark set against those in AF3’s training set for antibodies and nanobodies respectively.

To prevent the RMSD calculations from being confounded by small hinge movements in the loop connecting the variable and constant regions, we cropped to the variable fragment region using a custom function. Finally, we renumbered the structures using the AbNum webserver⁴⁷ using the Chothia scheme.

AF2.3-M predictions

We used a local ColabFold installation with AlphaFold-Multimer version 2.3 downloaded from https://github.com/YoshitakaMo/localcolabfold. We predicted a single decoy for each target and did not use templates. Similar to the curated benchmark, we cropped the predictions to the variable fragment region using a custom function and renumbered the structures using the AbNum webserver⁴⁷ using the Chothia scheme.

AF3 predictions

We used the AF3 server (https://alphafoldserver.com) to generate decoys. The server generates five decoys (diffusion samples) per seed. To test the diversity produced by seeds, we predicted three seeds per target, with the seed number pre-set to either one, two, or three by using the JSON file upload option. We cropped the predictions to the variable fragment region using a custom function and renumbered the structures using the AbNum webserver⁴⁷ using the Chothia scheme. By generating AF3 predictions through the server, it is possible that model weights may have changed during data collection, making it difficult to reproduce our results for the first three seeds.

As CDR H3 RMSD varies by seed, to ensure stable and reproducible data analysis we used the local AF3 set up (https://github.com/google-deepmind/alphafold3) to generate another 97 seeds per target on April 8th, 2025 using JSON files. The predicted structures were then cropped and renumbered with the same method mentioned above.

Boltz-1 predictions

We use the local Boltz-1 installed via PyPi (version 2.2.0)²⁸ to generate five diffusion samples using a single seed (random). We use the default three recycling rounds (which is also consistent with AF3), the default 200 diffusion steps, and auto-generated MSAs using the ColabFold server.⁴⁸ We renumber the structures using the AbNum webserver⁴⁷ using Chothia scheme.

Chai-1 predictions

We use a local Chai-1 setup installed via PyPi (version 0.6.1)²⁹ to generate five decoys across a single seed (random). We use three recycling rounds, 200 diffusion steps and auto-generate MSAs using the ColabFold server.⁴⁸ Chai-1 has no flag to select MSA depth. Using pre-cropped sequences from crystal structures to match the length of the ground-truth and AF3 structures, then renumber using the AbNum webserver to the Chothia scheme.⁴⁷

Selecting the top-ranked decoy

We choose the top-ranked AF3 prediction per seed via the calculated rank score (a bespoke combination of ipTM, pTM, and disorder confidence measures) that AF3 provides in the confidence summary files with its predictions. If the rank score is equivalent for multiple predictions, we select the prediction with the highest ipTM-HA (as it is applicable to both antibodies and nanobodies). If multiple predictions share a high rank score and ipTM-HA, we select one from the pool randomly, as we aim to simulate blind prediction scoring.

For predictions from the local AF3 setup, AF3 specifies a single decoy out of the five predicted via ranking_score in the score file, so we forego the secondary filtration step using ipTM-HA.

The top-ranked decoys for Boltz-1 and Chai-1 are similarly chosen through the decoy with the highest overall rank score in the provided score files, using confidence_score for Boltz-1 and aggregate_score for Chai-1. The ranking scores are a similar combination of confidences to AF3.

RMSD calculations

To calculate RMSD, we used PyRosetta with its AntibodyInfo class.⁴⁹ For calculating the CDR H3 global RMSD, we used the CDR backbone function, and for CDR H3 local RMSD, we extracted sub-poses for the CDR H3 loops also using the AntibodyInfo class, then calculated the C$\mathit{\alpha }$ loop RMSD using the Rosetta Scoring class.

ipTM extraction

The TM-score measures topological similarity of two proteins, such that incorrect loop movements do not result in a severe scoring penalty, and is not dependent on protein size, making it more robust than the RMSD. The ipTM score is the predicted TM-score between two specified chains which estimates the difference between the predicted complex and the unknown ground truth.⁵⁰ AF3 outputs an ipTM score per chain. For our work we use either the score between the heavy chain and antigen, or light chain and antigen.

Binding energy calculations

To calculate binding energies, we used PyRosetta’s InterfaceAnalyzer with the REF2015 score function.⁴⁹ We set the InterfaceAnalyzer to pack the complex before and after separating the binding partners to ensure that steric violations do not affect the rigid and flexible docking signals.

Figures and statistical analysis

We conducted Mann-Whitney-U and Pearson correlation statistical analyses using the Scipy package⁵¹ in Python, and generated letter-value plots, scatter plots, strip plots, and regression plots using the Seaborn package⁵² in Python. We used PyMol 3.0 (Schrodinger, Inc.)⁵³ for protein structure visualization images.

Edit distance calculations

We calculate edit distances between concatenated CDR sequences using the Levenshtein python package⁵⁴ via the distance function, which calculates the number of substitutions and indels required to convert one sequence to the other.

Precision-recall (PR) curve

The PR curve is used to compare the ability of the confidence metrics to discriminate between correct (DockQ $\ge$ 0.23) versus incorrectly docked (DockQ $<$ 0.23) complexes. We generate operating points by comparing the recall at a particular threshold for the predictive metric to the precision. Both values are calculated via counts of true positive ($T_p$), false positive ($F_p$), and false negative ($F_n$) data points. The recall value is defined by $\frac{T_p}{T_p+F_n}$, while the precision value is defined by $\frac{T_p}{T_p+F_p}$. Discriminative power increases with increasing recall and precision.

As we vary the thresholds separating correct and incorrect cases for each metric, we generate a characteristic curve. The strength of the metric’s discrimination is measured via the average precision score. The average precision (AP) score is calculated through the equation ${\sum }_n(R_n-R_{\mathit{n}-1})P_n$, where $R_n$ and $P_n$ are the recall and precision values at each threshold $\mathit{n}$.

Combined metrics are equivalent to a double filtering strategy. We use a union operator to combine an interface metric (ipTM or I-pLDDT) set at the optimized cutoff value with another measure, such as $\mathrm{\Delta }{G}_B$ or avg. H3-pLDDT. We generate a range of values of interface metrics to test based on the data, then find the range of values of the second measure by determining what range provides a true positive, false negative and false positive value. Thus each combination metric PR curve is generated via a single interface metric threshold with a range of values for the second threshold. We use the raw AP score to determine the best metric and do not normalize it. We are able to use the PR curves to investigate why a metric is poorer than another.

Interface and average H3 pLDDT calculations

We calculate the I-pLDDT by averaging the residue pLDDTs for all residues within a 10 Å … radius from the binding interface.

We extracted pLDDTs of C$\mathit{\alpha }$ residues for each H3 loop using the bfactor column through PyRosetta. We then averaged them in Python.

Funding Statement

This work was supported by National Institutes of Health(NIH) [R35 GM141881].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data and code availability

We make the curated benchmark dataset, PDB IDs used in prediction and their sequences, and the AF3 predictions available here: https://zenodo.org/records/16426003. The code we created for quick and standardized calculations and to visualize our results can be found here: https://github.com/NooriFatima/AF3_AbNb_Benchmark.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/19420862.2025.2545601