Fitness Landscape for Antibodies 2: Benchmarking Reveals That Protein AI Models Cannot Yet Consistently Predict Developability Properties

Abstract

A prominent application of machine learning in therapeutic antibody design is the development of models that can generate or screen antibody candidates with a high probability of success in manufacturing and clinical trials. These models must accurately represent sequence-structure-function relationships, also known as the fitness landscape. Previous protein function benchmarks examine fitness landscapes across diverse protein families, but they exclude antibody data. Here, we introduce the second iteration of the Fitness Landscape for Antibodies (FLAb2), the largest public therapeutic antibody design benchmark to date. The datasets collected in FLAb2 contain developability assay data for over 4M antibodies across 32 studies, encompassing seven properties of therapeutic antibodies: thermostability, expression, aggregation, binding affinity, pharmacokinetics, polyreactivity, and immunogenicity. Using the curated data, we evaluate the performance of 30 artificial intelligence (AI) and biophysical models in learning these properties. Protein AI models on average do not produce statistically significant correlations for most (80%) of developability datasets. No models correlate with all properties or across multiple datasets of similar properties. Zero-shot predictions from pretrained models are incapable of accurately predicting all developability properties, although several models (IgLM, ProGen2, Chai-1, ESM2, ISM, IgFold) produce statistically significant correlations for multiple datasets for thermostability, expression, binding, or immunogenicity. Fine-tuning with at least 10² points improves performance on thermostability, aggregation, and binding, but polyreactivity and pharmacokinetics lack enough data for significance. Yet it is humbling to observe that given enough developability data (10³ points), a fine-tuned one-hot encoding model can match the performance of fine-tuned billion-parameter pretrained models. Training data composition influences performance more than model architecture, and intrinsic biophysical properties (thermostability) are more readily learned than extrinsic properties (immunogenicity, pharmacokinetics). Controlling for germline distance with partial correlation reveals that protein language models draw substantially on evolutionary signal; on average, germline edit distance accounts for 40% of their apparent predictive power. FLAb2 data are accessible at https://github.com/Graylab/FLAb, together with scripts that allow researchers to benchmark, compare, and iteratively improve new AI-based developability prediction models.

2. Introduction

The innate and adaptive immune system is pivotal for safeguarding the human body, with antibodies employed as specialized proteins evolved to combat diseases. Antibody engineering exploits their therapeutic potential, resulting in over 200 marketed antibody therapeutics and a pipeline of nearly 1,400 investigational product candidates currently undergoing evaluation in clinical studies as treatments for a wide variety of diseases¹. The efficacy of therapeutic antibody candidates hinges on achieving a delicate balance of drug-like biophysical properties, also known as developability properties, characterized by intricate tradeoffs where enhancing one property may compromise others.

The rapidly advancing field of artificial intelligence is increasingly being applied to antibody design, enabling the generation of novel and diverse therapeutic candidates with favorable developability profiles in significantly reduced timeframes. Generative models can create antibodies in various formats - including VHHs, scFvs, and Fvs - that target a broad range of antigen epitopes, such as those from infectious agents like intestinal pathogens, influenza, RSV, SARS-CoV-2; or endogenous molecular targets like IL-7, TNFα and GPCRs^2–5. In parallel, pretrained protein language models have been used in an unsupervised, zero-shot manner to guide mutational selection, enhancing the stability and antigen-binding affinity of wild-type antibodies^6,7. Protein language model embeddings have also been leveraged to fine-tune downstream models for improved prediction of key developability properties, including clearance⁸, viscosity⁹, and in vitro assays^10–12.

As the diversity of deep learning approaches increases^13,14, a systematic benchmark is needed for evaluating performance. Current antibody design methods are typically evaluated with native sequence or structure recovery^15–19, which does not provide a complete indication of therapeutic potential. Other antibody design studies rely on sequence- or structure-based predictors of biophysical properties that have not yet been robustly validated against experimental antibody developability data^20,21.

Here, we provide a therapeutic antibody design benchmark dataset by curating experimental fitness data from over 30 studies spanning antibody thermostability, expression, aggregation, binding affinity, pharmacokinetics, polyreactivity, and immunogenicity. Our dataset collection, the Fitness Landscape for Antibodies (FLAb2), is the largest publicly available database of diverse developability data covering many relevant therapeutic properties. With FLAb2, we contribute (1) diverse datasets of therapeutic antibody assay data, (2) an assessment of the performance of widely adopted, pretrained and fine-tuned deep learning models for proteins, (3) an interpretation for what protein AI models learn and how model characteristics influence performance, and (4) scripts for zero-shot scoring and fine-tuning pretrained models. After introducing the data, we demonstrate its value by benchmarking a collection of models relevant to antibodies for their ability to correlate likelihoods to fitness properties. Our long-term vision is that FLAb2 will help the development of models that can generate or filter new antibody design candidates more efficiently than what is often done experimentally.

3. Related work

3.1. Several databases provide antibody sequence and structure data, but not developability

The Observed Antibody Space (OAS)²² and NaturalAntibody AbNGS²³ collectively provide 5B annotated sequences from immune repertoires. The Structural Antibody Database (SAbDab)²⁴ contains 10K antibody structures from the Protein Data Bank (PDB) and is continually updated. SAbDab provides over 500 experimental binding affinity data points associated with some of the antibody-antigen crystal structures, but does not include other developability properties.

3.2. Most protein fitness benchmarks exclude antibodies

Previous endeavors to establish benchmarks for function prediction have laid a foundation for protein engineers to assess new models. The Critical Assessment of Functional Annotation (CAFA)²⁵ tests the assignment of gene ontology classes to proteins. The Task Assessing Protein Embeddings (TAPE)²⁶ evaluates different pretrained models in predicting three protein structure properties (remote homology, secondary structure, residue contacts), as well as two fitness properties (fluorescence and stability). The Fitness Landscape for Proteins (FLIP)²⁷ examines the performance of masked language models, convolutional networks, and evolutionary models at expression, binding, and thermostability prediction. ProteinGYM²⁸ tests zero-shot and few-shot benchmarking of protein language models from deep mutational scans. Garcia et al.²⁹ curated a dataset of 614 experimentally characterized de novo designed monomers and benchmarked the ability of AlphaFold, ProteinMPNN, and ESMFold to zero-shot predict expression and solubility. RemoteFoldSet benchmarks the structural awareness of structure-informed language models³⁰. These benchmarks provide insight in the capabilities of AI protein models for protein function prediction, but they all exclude antibody data. The absence of a publicly accessible benchmark dataset for therapeutic antibodies hinders the ability to identify which model properties contribute to optimal performance in predicting antibody fitness landscapes.

3.3. Existing antibody benchmarks focus on binding prediction

Antibody binding datasets have been curated for benchmarking. AstraZeneca used publicly available binding data from AbSci³¹, Porebski et al.³², and 85 data points generated in-house to benchmark their antibody sequence ranking model DiffAbXL against other approaches³³. AbRank³⁴ benchmarked five deep learning models at antibody binding affinity prediction, using a database of 300K antibody-antigen pairs curated from nine studies. NaturalAntibody curated AbDesign³⁵, a dataset of 14 antibody-antigen complexes, each paired with mutational data totaling over 600 affinities. AbAgym³⁶ curated 67 deep mutational scan datasets to benchmark ML models at predicting the change in antibody-antigen binding affinity upon mutation. AbBiBench³⁷ curated 180K experimental binding measurements and benchmarked protein AI models at antibody binding affinity maturation. These benchmarks assess binding affinity, a key step in therapeutic design, but an antibody’s success also hinges on other developability factors. Our preliminary version of FLAb³⁸ provided 30K datapoints across six developability properties to benchmark six deep learning models at zero-shot fitness prediction. Here, our updated release, FLAb2, integrates the previously described datasets with additional datasets mined from publications, yielding a database of 4M antibodies, and we benchmark 30 protein AI models on both zero-shot and few-shot prediction.

4. Fitness landscape data

Jain et al.³⁹ defines desirable developability characteristics as (1) high conformational stability, (2) high-level expression, (3) low propensity for aggregation, (4) high binding affinity towards the target antigen, (5) low polyreactivity, and (6) low immunogenicity (Fig. 1). FLAb2 includes sequences and measurements from Jain et al. and 31 other studies, which we use to assess the efficacy of protein AI models in capturing essential characteristics of therapeutic antibodies (Fig. 2, Table S1). We collected in vitro and in vivo assays commonly used in therapeutic antibody discovery campaigns and summarized their counts by developability category and define each fitness landscape in Fig. S1 and Supp. Section 10.2, respectively. Each sequence is associated with at least one fitness label pertaining to the developability properties. Most of the collected datasets contain measurements from a single assay for a relatively small set of antibody sequences, with only a few datasets providing multiple assays per-sequence and large sequence sets (Fig. 2).

Figure 1: Seven classes of biophysical data are relevant to antibody developability.

Left to right: Increasing machine learning prediction complexity. Provided below each category is an example assay used to quantify the property, with desirable values shown in green and undesirable values in red. (a) Antibody thermostability (Tm) is assessed through differential scanning calorimetry (DSC), which measures the heat capacity change of a sample with temperature; higher Tm values are considered better. (b) Expression is often assessed using enzyme-linked immunosorbent assay (ELISA) fluorescence with high optical density indicating desired high expression. (c) Aggregation can be measured using affinity-capture self-interaction nanoparticle spectroscopy (AC-SINS), where no change in the measured plasmon wavelength shift is ideal. (d) Binding affinity to a target antigen is assessed using the equilibrium dissociation constant (K_d), with a desirable K_d typically falling in the low nanomolar to picomolar range; smaller K_d values indicate a desired tighter binding. (e) Pharmacokinetics is measured by timing the clearance of a drug in vivo through an animal, typically mice or cynomolgus monkeys, where a low clearance rate is preferred. (f) Polyreactivity can be measured with cross-interaction chromatography (CIC) retention time, where therapeutic antibodies are expected to have a low retention time in the column indicating minimal non-specific interactions. (g) Immunogenicity is quantified as the percentage of patients developing anti-drug antibodies (ADAs) after therapeutic administration, where an ideal, non-immunogenic antibody results in 0% ADA response within a patient population.

Figure 2: FLAb2 is composed of developability data collected from publications, competitions, and collaborators.

(a) Most of the collected data is for binding, and data for pharmacokinetics is scarce. (b) Binding data is gathered from 22 publications. Except for binding, in each category, we were only able to identify 8 or fewer studies with open data. (c) Most of the datasets are either large (>1,000 sequences) but closely related (e.g. from a deep mutational scan of single point mutations) or small (<200 sequences) and diverse (antibodies from unique clonotypes).

5. Pipeline for model evaluation

Protein models that produce statistically significant correlations with the antibody fitness landscapes can be reliable predictors for new therapeutic antibody design candidates. To benchmark protein models, we created a pipeline to run prediction models and correlate them to each experimental dataset (Fig. 3).

Figure 3: Pipeline for benchmarking protein AI models.

All fitness datasets contain columns for antibody heavy chain sequence, antibody light chain sequence, and an associated fitness metric. (a) Zero-shot benchmarking: For each protein language model, we separately input the heavy and light sequence to return two perplexity scores, and we tabulate the average perplexity between the two sequences. For structure-conditioned language models, we first predict the antibody structure with Chai-1³⁹, and then tabulate the single perplexity scored from the model. Spearman’s correlations are calculated between average perplexity and the fitness measure. (b) Few-shot benchmarking: For each protein language model, we separately input the heavy and light chain sequences to return two embedded representations. The pairs of embeddings are concatenated, pooled, and used to train a two-layer fully connected network for regression with an 80/10/10 train/validation/test split. Spearman’s correlations are calculated between predicted fitness and true fitness. No antigen information is provided for any benchmarked models. We use Spearman’s correlation between model outputs and true fitness labels to capture any monotonic relationship between variables and to account for the non-normally distributed data we use to benchmark. Some assay labels were inverted to ensure that, in all cases, a positive Spearman’s correlation indicates better alignment between the model’s confidence and the assay values. We provide examples for (c) zero-shot and (d) few-shot benchmarking for antibody binding prediction.

5.1. Zero-shot benchmarking

Zero-shot antibody fitness prediction is the task of using a model pretrained on protein sequence and structure to score a new set of antibodies, where the score is assumed to be descriptive of the antibody’s fitness, even though the model is not explicitly trained on this task. For zero-shot benchmarking (Fig. 3a), we use the antibody variable region sequence or structure as inputs for each model and output perplexity scores (averaged over all residues in the heavy and light chains). We use perplexity because it represents the confidence that the new antibody is similar to the learned distribution of high-fitness proteins from the pretraining dataset. If a protein language model implicitly captures the biophysical landscapes of antibodies, it should assign high confidence (low perplexity) to previously unseen high-fitness antibodies and low confidence (high perplexity) to low-fitness antibodies (examples in Fig. 3c). In zero-shot testing, all models were previously trained in their respective studies - we performed no additional fine-tuning prior to calculating perplexities.

We report the Spearman’s correlations and associated p-value to establish the connection between the model uncertainty values and the fitness metrics associated with the sequences in the dataset. An optimal zero-shot predictor will display a Spearman’s correlation close to 1.0 between model perplexity and developability labels. In contrast, a Spearman’s correlation of 0.0 suggests no relationship between these metrics, while a correlation of −1.0 indicates an inverse relationship between the model’s predictions and the developability label.

Fig. 3c provides an example for zero-shot benchmarking. Ideally, the model should be more confident in the higher binding affinity antibody than the variants with lower binding affinity. This correct downward trend is observed for several candidates (MEDIUCA H1-Solomon, ρ = −0.53; MEDIUCA H4-Hubei, ρ = −0.47; mab114 Ebola, ρ = −0.28; MEDI H7-HK16, ρ = −0.20), yet the model incorrectly displays more confidence with the weaker affinity antibodies for S309 SARS-CoV2 (ρ = 0.57) and REGN10987 SARS-CoV2-Omicron (ρ = 0.57).

5.2. Few-shot benchmarking

Few-shot antibody fitness prediction is the task of supervised fine-tuning a protein AI model on a developability dataset for predicting that property. For few-shot benchmarking (Fig. 3b), we explore an embedding-based fine-tuning approach. We use each protein language model to generate an embedded representation for each antibody and train a regression model on part of the developability dataset to predict developability of the holdout set. Predictions on the test set are evaluated based on the Spearman’s rank order correlation between the predicted and true assay values. Like the zero-shot setting, few-shot benchmarking is performed per dataset, and statistical significance is evaluated based on the associated p-value of the Spearman correlation. If a protein language model captures the biophysical landscapes of a pretraining set of antibodies or general proteins, the rich features in extracted embeddings should allow a learnable mapping between embeddings and developability properties. Like zero-shot benchmarking, an ideal predictor will have a Spearman’s correlation approaching 1.0 between predicted developability and true developability.

In the few-shot benchmarking example in Fig. 3d, there is a significant (p-value = 0.00) positive correlation (ρ = 0.93) between the predicted and true fitness label for the strong binders of the broadly neutralizing antibody CR6261 in the upper right-hand quadrant, while showing relatively more noise for predicting weak binders in the lower left-hand quadrant.

5.3. Models benchmarked

We benchmark a diverse set of AI models associated with antibody design: (1) Decoder-only language models are trained using next-token prediction, and we investigate IgLM⁴⁰ and ProGen2⁴¹; (2) encoder-only language models capture continuous representations of sequences, and we investigate AntiBERTy⁴², ESM-2⁴³, and ISM⁴⁴; (3) inverse folding models predict protein sequences from structure, and we investigate AbMPNN⁴⁵, ESM-IF⁴⁶, and ProteinMPNN⁴⁷; (4) structure prediction models predict protein structures from sequence, and we investigate IgFold⁴⁸ and Chai-1⁴⁹. As a baseline for physics-based approaches, we calculate PyRosetta⁵⁰ total energy and BioPython⁵¹ charge terms. We provide a more in-depth description of each model in Table S2 and further details about the categories of benchmarked models in Supp. Section 10.3.

6. Zero-shot results

We evaluated whether model likelihoods, expressed as average perplexities, correlate with experimentally measured antibody developability properties.

6.1. Most models do not have statistically significant correlations with most datasets

We first investigate how many statistically significant (p-value < 0.05) Spearman’s correlations each zero-shot model has to each developability dataset. The structure-informed masked language model ISM 650M displayed the highest count of statistically significant correlations, with 94 correlations or 39% of the total developability data sets (Fig. 4a). In order of decreasing count, protein AI models on average displayed significant correlations to 51% of immunogenicity datasets (Fig. 4h), 35% of expression datasets (Fig. 4c), 24% of thermostability datasets (Fig. 4b), 20% of binding datasets (Fig. 4e), 17% of aggregation datasets (Fig. 4d), 16% of polyreactivity datasets (Fig. 4g), and 14% of pharmacokinetics datasets (Fig. 4f). The observed variation in the number of statistically significant correlations across developability categories likely reflects a combination of factors that are difficult to disentangle, including whether a model captures the underlying biophysical property, whether a data set is simply too small, and whether the biological assay readouts are too noisy. Some datasets exhibit statistically significant correlations across multiple models, suggesting that some data sets are clean and large enough to be used to fairly benchmark models. Accordingly, we designated a subset of datasets that show at least five statistically significant model–dataset correlations for further comparison (Table S3); the corresponding distributions of Spearman’s correlation coefficients for each model are shown in Fig. 5.

Figure 4: Count of significant correlations per zero-shot model.

Count of significant (p-value < 0.05) correlations each protein AI and physics-based model had within each developability category. The “all” column within each subplot represents the total number of datasets within that category.

Figure 5: Distribution of zero-shot prediction performances for each model.

Models are colored based on their architecture, and the y-axis displays the range of Spearman’s correlations between model confidence and developability label, where a correlation approaching 1.0 is ideal. Some of the assay labels are inverted so that a positive correlation between fitness and assay value always indicates improved model performance. We only report Spearman’s correlations with datasets that exhibit at least five significant dataset-model p-values to ensure our conclusions are statistically sound.

6.2. No single model predicts all developability properties with high accuracy

Within each developability category, we investigate the average performance across all protein AI models and identify the best performing model. Expression was the least challenging task (average Spearman’s rank correlation ρ = 0.22), whereas aggregation was the most difficult (ρ = −0.07) (Table S4). From easiest to hardest, the average task ranking across all models was: expression > immunogenicity > thermostability > binding > polyreactivity > pharmacokinetics > aggregation.

IgFold was the best overall performer (ρ = 0.12) (Fig. 5a), and most developability categories had a unique top-performing model (Fig. 5b–g, Table S5). The structure prediction model Chai-1 best predicted thermostability (ρ = 0.45) (Fig. 5b); the masked language model ESM2 150M best predicted expression (ρ = 0.44) (Fig. 5c); BioPython charge at pH 7.4 best predicted aggregation (ρ = 0.29) (Fig. 5d), pharmacokinetics (ρ = 0.59) (Fig. 5f), and polyreactivity (ρ = 0.18) (Fig. 5g); and the autoregressive language model ProGen2 Medium best predicted immunogenicity (ρ = 0.34) (Fig. 5h, Table S5). Top performance in one category did not generalize to others. For instance, while Chai-1 excelled at thermostability prediction, it performed poorly in aggregation prediction (ρ = −0.17). Similarly, ESM2 150M, despite being the best model for expression, ranked as one of the lowest for pharmacokinetics (ρ = −0.15). An ideal antibody developability oracle would exhibit a ρ = 1.0 correlation between model confidence and experimental label for all developability categories, accurately predicting properties across all seven fitness domains. However, no single model achieves uniformly high performance.

Developability properties vary in their predictability using the antibody alone, because some are intrinsic to the antibody sequence and structure, while others depend on extrinsic biological or environmental factors that are not provided to the models. Intrinsic properties such as thermostability and aggregation can, in principle, be inferred from the sequence or fold of the antibody variable domain, whereas extrinsic properties—such as polyreactivity, binding, immunogenicity, expression, and pharmacokinetics—depend on contextual information beyond the sequence. Binding depends on the antigen sequence and structure; immunogenicity depends on the patient’s immune system response; expression depends on host cell machinery; and pharmacokinetics depends on how the body traffics and metabolizes the administered drug. The unexpected difficulty of thermostability prediction may be explained by complications in identifying which portion of the antibody format melts first and is reported in melting curves⁵². These results for antibodies are similar to the results of the general protein benchmark ProteinGYM²⁸ in that there is a unique top performing model depending on assay function type, yet different in that some FLAb2 categories had few good predictors (i.e. few models presented a ρ < 0.3), whereas all assay function types in ProteinGYM had multiple models with ρ > 0.4.

6.3. Architecture does not significantly impact zero-shot performance

We next ask whether model architecture influences zero-shot performance. Across all developability categories, autoregressive language models (ρ = 0.01), masked language models (ρ = 0.01), inverse folding models (ρ = 0.02), and structure predictors (ρ = 0.07) performed similarly (Table S6). Variation in performance is seen between individual developability categories. Autoregressive models performed best at binding and immunogenicity; masked language models performed best at expression; inverse folding models performed best at polyreactivity (tied with structure predictors); and structure predictors performed best at thermostability and polyreactivity (Table S6). When comparing two models trained on the same 550M antibody sequence dataset from OAS, AntiBERTy (a masked language model) and IgLM (an autoregressive language model), we found both performed equally in most categories (Table S5). Similarly, when comparing the ESM2 suite of masked language models with the ProGen2 suite of autoregressive language models, both performed equally in most categories (Table S5). Thus, regardless of whether next-token or masked prediction schemes are used to learn a fitness landscape, models of similar size with identical input representation and training set learn the same data distribution.

6.4. Incorporating structure improves zero-shot performance over sequence-only models

We next asked whether learned protein representation influences zero-shot performance. Structure-informed masked language models, inverse folding models, and structure predictors–all of which learn over the protein sequence and structure space–dominated the developability prediction categories (Table S7). Comparing the ESM2 suite with the ISM suite allows us to compare two masked models with the same architecture, differing only in the tokenizer: ESM2 tokenizes sequence, while ISM also distills structural representations of the input protein. For thermostability prediction, ISM 3B (ρ = 0.38) outperformed ESM2 3B (ρ = 0.34), however ESM2 650M (ρ = 0.37) outperformed ISM 3B (ρ = 0.34) (Table S5). On average, across all categories, the ISM models outperformed the ESM models (Table S5). Structural reasoning allows these models to explicitly capture geometric and energetic constraints that sequence-only models overlook. Our findings align with those of Notin et al.²⁸, who reported that incorporating structure substantially improves zero-shot prediction relative to sequence-only representations.

6.5. Antibody-specific and general protein models excel in different developability domains

We next assessed how models trained specifically on antibody sequences compare to general protein models. General protein models were better at predicting expression (ρ = 0.32), while antibody specific models were better at predicting immunogenicity (ρ = 0.11) (Table S8). A direct comparison reveals a slightly different story: Between ProGen2 Medium (trained on UniRef) and ProGen2 OAS (trained exclusively on antibody sequences), both of which are the same autoregressive architecture with 764M parameters, ProGen2 Medium was the better performer for expression (ρ = 0.36), binding (ρ = 0.18), and immunogenicity (ρ = 0.34) (Table S5). The different trend when comparing ProGen2 Medium and ProGen2 OAS may be particular to the ProGen2 suite–the original ProGen2 paper also noted that the general protein ProGen2 models were better than ProGen2 OAS for antibody binding, thermostability, and expression prediction⁴¹. Antibody-specific and general protein models appear to specialize in different aspects of developability. Antibody-specific models perform better at predicting binding and immunogenicity, reflecting their learned understanding of antibody-specific sequence patterns and human-like framework regions. In contrast, general protein models, trained on broader sequence and structural diversity, perform better at predicting expression and thermostability—properties governed by general protein folding and biophysical behavior.

6.6. Physics-based and AI-based models capture orthogonal properties of proteins

We then examined the performance of physics-based models for developability prediction and compared it to that of zero-shot protein AI models. PyRosetta was not a top performer in any developability category and generally had no strong correlations to the developability categories (Table S5). Several sequence-based BioPython models performed exceptionally well: Across all physics-based and AI-based approaches, BioPython charge at pH 7.4 performed best at aggregation (ρ = 0.29), pharmacokinetics (ρ = 0.59), and polyreactivity (ρ = 0.18) prediction (Table S5). Physics-based models exhibited statistically significant correlations across the largest number of models for aggregation, pharmacokinetics, and polyreactivity prediction (Fig. 5d, f, g). While several physics-based models performed well at aggregation, pharmacokinetics, and polyreactivity prediction, protein AI models generally performed better at binding, expression, thermostability, and immunogenicity prediction. A charge-based model performs well at aggregation and pharmacokinetic prediction, consistent with the fact that these properties are driven by charged surface patches^53,54.

6.7. Larger protein AI models do not outperform medium-sized models

We next asked whether the size of the model, determined by the number of parameters, influences zero-shot prediction performance. Fig. S2 shows that increasing model size did not systematically improve zero-shot performance across categories. For aggregation, binding, and pharmacokinetics, performance remained flat. Thermostability and immunogenicity showed marginal improvement with model scale, while polyreactivity displayed high variability. A common claim with large language models is that scale improves zero-shot performance^55–57, yet we do not observe this with therapeutic antibody engineering tasks. Notin et al.²⁸ similarly found that billion parameter protein language models underperformed million parameter models. This trend may reflect the balance between model capacity and data coverage: overly large models risk memorization, while smaller models generalize more efficiently. The ratio of training tokens to model parameters, as noted by Hou et al.⁵⁸, may provide a more meaningful predictor of generalizability.

6.8. Protein AI models are biased towards antibody germline signal rather than developability

Intrigued by the observation of Olsen et al.⁵⁹ that antibody language models have a germline bias, we investigated to what extent protein AI models are correlated to germline signals, rather than the actual developability label. We calculated the Spearman’s correlation between model confidence and the Levenstein distance of the antibody from its respective germline sequence (see Supp. Section 10.3 for calculating germline distance). Fig. S3 and Table S9 report the Spearman’s correlation between model confidence and germline distance for datasets within each developability category. ProGen2 BFD90 had the strongest correlation across all categories (ρ = 0.80), and IgLM perplexities were most correlated to germline for sequences with thermostability, binding, polyreactivity, and immunogenicity labels (Table S9). Sequence-only models also had significantly higher correlation to germline than any models that learned over sequence and structure (Table S10). BioPython and PyRosetta displayed little to no correlation with germline. Increasing the number of model parameters increased the correlation between model confidence and germline signal (Fig. S4). The correlations between model confidence and edit distance to germline were significantly higher than the correlations we reported above between model confidence and fitness value. Additionally, AI-based developability prediction performance was significantly higher for datasets where germline sequences had relatively higher fitness, and lower for datasets where germline sequences had relatively lower fitness (Fig. S5a)—a trend not observed for physics-based models (Fig. S5b).

We next controlled for germline distance using partial correlation⁶⁰, which measures the relationship between two variables while eliminating influence of a third variable (Supp. Section 10.4). Upon eliminating the influence of germline distance, protein language models lost, on average, 40% of their apparent fitness-predictive power, whereas physics-based models were significantly less affected (14%) (Fig. 6, Table S12). Our results suggest that the dominance of germline residues in the antibody pretraining data introduces a strong bias toward them in zero-shot prediction. Previous work^61–64 similarly observed that protein language models capture and exploit phylogenetic distances more readily than biophysical characteristics. The reduction of germline bias we observe with the introduction of structure in the learned representation indicates that this bias can be reduced by expanding the learned representation beyond sequence.

Figure 6: Partial correlation between model confidence and fitness when accounting for germline signal.

The x-axis shows the correlation between model confidence and fitness, and the y-axis presents the partial correlation between model confidence and fitness after accounting for germline signal (see Supp. Section 10.4). AI models are displayed as red circles and physics models as blue triangles. To focus on statistically robust results, we include only datasets for which at least five models show statistically significant correlations and for which the mean Spearman correlation across models is at least 0.3 (see datasets in Table S11).

7. Few-shot results

In the zero-shot setting, we evaluated model uncertainty (perplexity) as a proxy for developability. However, this metric may not fully capture the knowledge encoded during pretraining. To test whether models implicitly learned developability-related features, we used their latent embeddings in a supervised few-shot regression framework. The datasets that satisfy the cutoff of five statistically significant dataset-models and are used in this section are listed in Table S13. Results for pharmacokinetics and polyreactivity are omitted due to a lack of at least five statistically significant dataset-model correlations and big enough training datasets. The distribution of Spearman’s correlation coefficients for each fine-tuned model is displayed in Fig. 7, and the count of statistically significant correlations per model is displayed in Fig. S6.

Figure 7: Distribution of few-shot prediction performances for each model.

Models are colored based on their architecture, and the y-axis displays the range of Spearman’s correlations between predicted developability and true developability label, where a correlation approaching 1.0 is ideal. We only report Spearman’s correlations with datasets that exhibit at least five significant dataset-model p-values to ensure our conclusions are statistically sound.

7.1. Few-shot models yield fewer significant correlations than zero-shot

As in the zero-shot setting, we first investigate how many statistically significant Spearman’s correlations each few-shot model has to each developability dataset. The masked language model ESM2 8M displayed the highest count of statistically significant correlations, with 30 correlations or 13% of the total developability datasets (Fig. S6a). In order of decreasing count, protein AI models on average displayed significant correlations to 73% of immunogenicity datasets (Fig. S6h), 46% of expression datasets (Fig. S6c), 9% of binding datasets (Fig. 6e), 4% of aggregation datasets (Fig. S6d), 6% of thermostability datasets (Fig. S6b), 5% of polyreactivity datasets (Fig. S6g), and 2% of pharmacokinetic datasets (Fig. S6f). This trend is similar to the zero-shot setting, where immunogenicity and expression had the highest count of significant correlations compared to the other developability categories. The overall lower count of significant correlations is in part due to the small size of the developability datasets – most datasets are less than 100 datapoints (Fig. 2c) leading to small train/val/test splits and poor performance. To enable a more equitable comparison of model performance, we identified the subset of the datasets that exhibit statistically significant correlations across at least five models (Table S13). The corresponding distributions of Spearman’s correlation coefficients for each model with the selected datasets are shown in Fig. 7.

7.2. Few-shot learning improves performance in all developability categories

Within each developability category, we investigate the average performance of MLPs trained on embeddings from each protein AI model, identify the best performing model, and compare it to the top performing zero-shot model. Thermostability was the least challenging to predict (ρ = 0.61), while aggregation was the most difficult (ρ = 0.32) (Table S14). Like the zero-shot setting, expression and immunogenicity showed statistically significant correlations across the largest fraction of models (Fig. S6). From the easiest to hardest, the average developability prediction performance across all models was: thermostability > expression > immunogenicity > binding > aggregation.

Embeddings from two protein AI models tied for best performer across all developability properties (ρ = 0.56): ESM2 3B and ISM 3B (Fig. 7a, Table S15). Most developability categories had a unique top-performing model (Fig. 7b–g, Table S15). Embeddings from the structure-informed masked language model ISM uc30 650M best predicted thermostability (ρ = 0.75) (Fig. 7b), the masked language models ESM2 35M and ISM 3B best predicted expression (ρ = 0.56) (Fig. 7c), ISM uc30pdb 650M best predicted aggregation (ρ = 0.66) (Fig. 7d), ISM 3B best predicted binding (ρ = 0.61) (Fig. 7e), and ESM2 150M best predicted immunogenicity (ρ = 0.49) (Fig. 7h, Table S15). High performance in one category did not generalize to others. For instance, while the IgFold structure embeddings model was among the best predictors for expression (ρ = 0.54), it was a poor predictor of aggregation (ρ = −0.03). Across all categories, the best few-shot models perform better than the best zero-shot models. In both the zero-shot and few-shot setting, aggregation was among the most challenging. However, there was a change in the other categories, as thermostability became the easiest to predict after finetuning, over expression (Table S15).

7.3. Architecture impacts few-shot more than zero-shot performance

We next ask whether model architecture influences few-shot performance. Across each developability category, embeddings from masked language models (ρ = 0.55) performed the best (Table S16). One hot encoding models performed similarly to inverse folding models and structure predictors (Table S16). These results are different to the zero-shot setting, where we found that architecture did not significantly impact performance.

7.4. Learned representation does not significantly impact few-shot performance

We next asked whether learned protein representation influences few-shot performance. Embeddings from structure-informed masked language models generally performed best (Table S17). Across all developability properties, both ESM2 3B (sequence-only) and ISM 3B (structure-informed) performed equally (ρ = 0.56). ISM uc30pdb 650M (ρ = 0.54) and ESM2 650M (ρ = 0.51) also performed similarly. Some models, such as IgFold, produce multiple learned representations throughout their pipeline. From IgFold, we extract sequence-only AntiBERTy embeddings, structure-informed embeddings after the graph transformer, and further structure-informed embeddings following template incorporation. Performance varied across these representations: AntiBERTy embeddings performed best across all properties (ρ = 0.48), followed by IPA (ρ = 0.41) and the graph transformer (ρ = 0.40). Whereas in the zero-shot setting adding structure improved performance, fine-tuning sequence-only embeddings were sufficient to obtain competitive developability prediction performance (Table S15).

7.5. General protein embeddings outperform antibody-specific embeddings in few-shot setting

We next assessed how embeddings from models trained specifically on antibody sequences compare to embeddings from general protein models. Embeddings from general protein models dominated at predicting each developability property (Table S18). These results differ from the zero-shot setting, where there was more variation in what general protein and antibody-specific models were effective at predicting.

7.6. One-hot encodings perform comparably to billion-parameter models in few-shot prediction

We next asked whether the size of the model influences few-shot prediction performance. Fig. 8 shows that increasing model size did not significantly improve few-shot performance across all categories. For thermostability (Fig. 8b), expression (Fig. 8c), aggregation (Fig. 8d), binding (Fig. 8e), and immunogenicity (Fig. 8f), one hot encoding performed equally to million and billion parameter models, indicating that simple models given sufficient developability data can approach the accuracy of large pretrained protein models. Developability prediction does not require billion parameter models, neither in zero-shot nor in few-shot.

Figure 8: Distribution of few-shot prediction performances for each model compared to parameter size.

Models are colored based on their architecture, and the y-axis displays the range of Spearman’s correlations between predicted developability and true developability label, where a correlation approaching 1.0 is ideal. We only report Spearman’s correlations with datasets that exhibit at least five significant dataset-model p-values to ensure our conclusions are statistically sound.

7.7. Few-shot models lose their germline bias compared to zero-shot models

We next investigated whether models trained on embeddings of pretrained protein AI models are correlated to germline signals, rather than the actual developability label. We calculated the Spearman’s correlation between model’s predicted developability and the Levenstein distance of the antibody from its respective germline sequence (see Supp. Section 10.3 for calculating germline distance). Fig. S7 and Table S19 report the Spearman’s correlation between predicted developability and germline distance. Across all developability properties, the correlation to germline varied significantly, with some exhibiting an inverse correlation and others showing a moderate correlation of ρ < 0.3 (Table S19). The germline bias shows no sign of increasing or decreasing with parameter size (Fig. S8). Compared to the zero shot models presented earlier, the germline correlations exhibited by few shot models were significantly less.

In the zero-shot setting described earlier, all pretrained protein AI models exhibited a clear germline bias, reflecting their reliance on evolutionary signals encoded during pretraining. In contrast, this bias substantially diminishes in the few-shot setting. Here, model predictions show little to no correlation with germline distance, suggesting that the models are no longer depending on evolutionary similarity to infer developability. Instead, they appear to have learned features more directly related to the experimental labels, resulting in improved predictive performance. The only developability property that consistently retained a notable correlation with germline distance was immunogenicity. This finding aligns with prior reports^65,66 that antibodies diverging further from their germline sequences tend to have higher immunogenic potential, and so germline distance is inherently correlated with immunogenicity.

8. Conclusion

This work benchmarked the zero-shot and few-shot performance of diverse protein AI models in predicting antibody developability properties. We highlighted the models that performed best in each developability category; evaluate how model architecture, learned protein representation, pretraining data composition, and parameter size influence performance; and identified what properties of therapeutic antibodies protein AI models learn.

Based on these observations, we suggest a set of recommendations to consider when developing zero-shot and few-shot antibody developability oracles:

1. Developability properties should be examined separately, as models that perform well in one fitness task may not perform well in any of the others.

2. Incorporating protein structure into the learned representation improves zero-shot and few-shot performance and reduces germline bias.

3. Antibody-specific models generally perform better for binding affinity and immunogenicity prediction, while general protein models perform better at thermostability and expression prediction.

4. Model size does not significantly impact zero-shot prediction.

5. When presented with little fine-tuning data (< 300 sequences), train an MLP on embeddings from a large, billion parameter model. When presented with a lot of fine-tuning data (> 1000 sequences), training an MLP on one-hot encodings may suffice.

The task of generalizable zero-shot and few-shot developability prediction remains unsolved. The protein AI models we evaluated rely on simplified heuristics—distance from germline and evolutionary patterns—rather than capturing the biophysical mechanisms underlying antibody developability. These heuristics do not generalize across diverse developability property landscapes, reflecting an incomplete internal representation. These models learn statistical patterns observed in the training data rather than the physical laws governing folding or function. Recognizing these limitations highlights the need to design models that can integrate richer sources of information to improve generalization.

The current benchmark has several constraints that limit broad conclusions. Not all model architectures are equally represented, with fewer structure-informed masked language models and a predominance of general protein models over antibody-specific models. Additionally, some datasets are small (2 – 20 sequences), resulting in weak statistical power and potential spurious correlations. To ensure robustness, we chose to interpret our results by looking only at datasets that had at least five significant dataset-model correlations from our set of 30 models and 240 datasets. Ideally, all correlations between models and datasets would be significant for a robust comparison to draw conclusions, but this is not the case for a database composed of datasets with varying size and biological assays with varying noise in their readouts. The next iteration of an antibody fitness benchmark could extend FLIP²⁷ by evaluating model generalization across diverse mutational regimes—including single, double, and triple mutants—to capture nonlinear epistatic effects⁶⁷. Investigating models trained on low-fitness variants and tested on high-fitness regions could further assess their capacity to guide directed evolution. Comparing machine learning–designed sequences with experimentally evolved variants would provide insight into whether a protein AI model interprets each differently. Pairwise variant ranking frameworks like that used in AbRank³⁴ could enhance robustness to assay noise and distribution shifts. Finally, exploring homolog density in training data⁵⁸ and adopting fine-tuning strategies like PLMFit⁶⁸, domain adaptation⁶⁹, and calibration⁷⁰ could improve zero-shot and few-shot prediction. Future improvements to developability oracles may involve integrating developability-labeled data⁷¹, experimental feedback⁷², in-silico developability data⁷³, and physics-based information⁷⁴ to enhance model generalization. Incorporating antigen or environmental context, along with domain adaptation strategies, may help account for covariate shifts⁷⁵ across heterogeneous datasets. Mixing representations, such as combining language model embeddings with one-hot encodings⁷⁶, has shown promise for improving predictive performance, and should be further explored.

Improvements to both benchmarking and developability oracles requires more developability data, as we have found that sequence and structure alone cannot capture all the fitness landscapes of therapeutic antibodies. This ideal developability dataset includes (1) diversity in antibody sequence space with many clonotypes and many types of mutations like substitution and indels; (2) diversity in developability space by including antibodies with good or poor developability properties, as we have limited data of antibodies with poor developability; (3) antibodies with low or high edit distances from germline; and (4) data replicated in multiple labs or environments to estimate the effects of covariates in combining datasets. Hummer et al.⁷⁷ estimated at least 90K computational ΔΔG values must be obtained to train an accurate binding affinity oracle. Properties that we found challenging to predict in our benchmark (immunogenicity, aggregation, pharmacokinetics) may require more data than this estimate, and properties we found easier to predict (expression) may require less data. Although FLAb2 contains millions of datapoints for antibody binding affinity, most are for the same 2 – 3 antigen sequences, making it difficult for us to reproduce the results of Hummer et al. which trained on a more diverse set of antibody-antigen complexes across the entire PDB. Expanding the benchmark with larger, more balanced datasets in the bottom right regime of Fig. 2 c) will improve the robustness and interpretability of observed trends.

Open data initiatives such as FLAb2 can accelerate the development of robust computational oracles, provide reproducible benchmarks, and ultimately guide the design of therapeutic antibodies with improved developability. However, more publicly available antibody developability data than what is currently available is necessary to benchmark and train the developability oracles of tomorrow. The continual release of antibody developability data will lead to major improvements in performance of antibody design and screening by leveraging the ingenuity of the global antibody engineering community.