Accelerating antibody development: sequence and structure-based models for predicting developability properties via size exclusion chromatography

ABSTRACT

Experimental screening for biopharmaceutical developability properties typically relies on resource-intensive, and time-consuming assays such as size exclusion chromatography (SEC). This study highlights the potential of in silico models to accelerate the screening process by exploring sequence and structure-based machine learning techniques. Specifically, we compared surrogate models based on pre-computed features extracted from sequence and predicted structure with sequence-based approaches using protein language models (PLMs) like ESM-2. In addition to different end-to-end fine-tuning strategies for PLM, we have also investigated the integration of the structural information of the antibodies into the prediction pipeline through graph neural networks (GNN). We applied these different methods for predicting protein aggregation propensity using a dataset of approximately 1200 Immunoglobulin G (IgG1) molecules. Through this empirical evaluation, our study identifies the most effective in silico approach for predicting developability properties for SEC assays, thereby adding insights to existing screening efforts for accelerating the antibody development process.

1. Introduction

Monoclonal antibodies (mAbs) are effective therapeutic proteins due to their high specificity, versatility, and efficacy in targeting a wide range of diseases. They enable targeted therapy with minimal off-target effects, applicable in oncology, autoimmune, infectious, cardiovascular, and metabolic diseases.¹ Advances in antibody engineering, such as humanization and affinity maturation, have enhanced their clinical efficacy and reduced immunogenicity, making mAbs safer and therapeutically effective for long-term use.² With over 145 antibody-based drugs approved by the FDA and many more in clinical trials, their impact on modern medicine is substantial and continues to grow.^3,4

The developability properties of antibodies are crucial for their transition from early-stage discovery to large-scale manufacturing. Since the antibody design process prioritizes the binding activity in neutralizing the target protein, the resulting antibodies may end up with unfavorable biophysical attributes hindering its progression into the next development stages.^5,6 Hence, it is extremely beneficial to screen antibodies not only for target specificity and binding affinity but also for physicochemical and biopharmaceutical properties such as solubility, stability, aggregation propensity, and manufacturability.⁷ These properties – which quantifies the developability of antibodies – significantly influence the antibody’s success in later development stages, including clinical trials and commercialization.⁸ Poor developability can lead to high attrition rates, increased costs, and extended timelines due to formulation challenges, instability, or immunogenicity.⁹ Systematic early evaluation of these properties can streamline development, advancing only the most promising candidates.¹⁰ This proactive approach also aids in designing robust manufacturing processes to produce high-quality therapeutic antibodies at scale, reducing late-stage failures and ensuring a more efficient path to commercialization.^2,11

Various assays and methods, such as size exclusion chromatography (SEC)¹² – which is the focus of this paper – dynamic light scattering (DLS),¹³ differential scanning calorimetry (DSC),¹⁴ and isoelectric focusing (IEF),¹⁵ are employed to measure the physicochemical attributes of antibodies, serving as proxy evaluations for their developability characteristics. These measurements are utilized to detect aggregates, assess conformational stability, evaluate chemical degradation, and determine both the isoelectric point and charge isoform content, all of which impact the molecule’s efficacy, solubility, and overall chemical stability.^2,5,7 In this work, we have considered the SEC assay which offers unique advantages in the purification and characterization of biomolecules. This chromatographic method, which separates analytes based on molecular size, plays a critical role in ensuring the purity, efficacy, and safety of therapeutic proteins. SEC allows for the effective separation of desired therapeutic proteins from aggregates and impurities that may affect the drug’s safety and efficacy. Furthermore, SEC is crucial for the detailed characterization of protein biotherapeutics, providing insights into their molecular weight distribution, aggregation state, and stability, which are essential parameters for regulatory approval and clinical success.^16,17 The technique’s ability to operate under mild conditions without altering the biological activity of the molecules makes it particularly valuable for the analysis of sensitive biologics.^18,19

While the experimental assays like SEC provide valuable insights into the development of biologics, they are often time-consuming and costly. The high-throughput screening methods required to evaluate a large number of candidates can be resource-intensive, requiring significant investment in both equipment and materials.²⁰ Machine learning and in silico approaches can significantly enhance the prediction of antibody developability by leveraging medium to high-throughput datasets. These methods aid in early-stage drug development, enabling the rapid selection of lead candidates and potentially reducing the need for extensive high-throughput analytical measurements. However, the limited amount of experimental assay datapoints poses a critical challenge in building a reliable surrogate model for predicting the developability property of interest. Existing efforts^21–25 primarily involve processing the sequences (with or without 3D protein structure) to compute the protein descriptors that are utilized by machine learning models for prediction. In addition to the computational burden of generating the features from structures, the performance of this approach is sensitive to the feature selection process. The protein language models (PLMs) offer a faster and more efficient alternative to utilize the structure information of the proteins. Leveraging the large pool of protein sequences, the PLMs learn to incorporate the structural information implicitly into the sequence embedding.²⁶ With the advancement of protein language models, there have been efforts^27,28 to predict the protein property from the sequence embedding learned by the pre-trained PLM. Since the PLMs work on the sequence representations of the antibodies, this approach has the potential for an accurate and faster screening pipeline by removing the need for prediction of structure and processing of structural features. On the other hand, several works^28–30 explicitly leverage the 3D protein structures along with the protein language model for the prediction of protein properties. In the absence of experimental structure data, this approach relies on either the protein folding tools like AlphaFold2,³¹ or homology modeling to build the structure from the protein sequence.

The performance of the above-mentioned approaches for predicting the protein properties varies across different types of assays. In this work, we have investigated the application of these different prediction approaches for the SEC assay. Specifically, we have considered the classification task of two developability properties – monomer content and difference in retention time of the IgG1 molecules to a reference sample. To summarize our work:

• We performed experiments with four prediction pipelines – one based on sequence and structure-based features, three others leveraging the protein language model and graph neural network – to select the best performing configuration (e.g. which features/PLM/GNN to use) under each pipeline.

• We have identified the best prediction strategy for two SEC properties of interest based on the hold-out test set performance by these four pipelines.

• Furthermore, we have assessed the impact of two different protein structure prediction tools on the developability prediction pipeline.

2. Methodology

2.1. High-performance size exclusion chromatography assay

HPSEC assays help determine the purity of protein samples by measuring the percentage of protein monomer (main product), HMWFs (higher molecular weight forms) and LMWFs (lower molecular weight forms). The chromatogram results from SEC assays typically display several peaks corresponding to detected species. Peaks appearing before the expected monomer peak are classified as HMWFs, while those appearing after are identified as LMWFs. This interpretation is crucial for understanding the composition and quality of the protein sample. The chromatograms also provide the retention time of peaks, which indicate the elution time of the protein in the chromatography columns. The SEC retention time of the monomer is primarily influenced by its molecular weight, but it is also affected by other molecular properties, including charge, hydrophobicity, and self-association,³² suggesting that SEC can offer a multifaceted view of the molecule’s behavior.^5,33 To normalize retention times of monomer content across multiple studies utilized in this paper, $\Delta \mathit{RT}$ is considered, which compares the retention time of the monomer protein to a reference protein, NIP228, with a known SEC retention time of approximately 8.47 minutes. This normalization is essential for consistent analysis across different studies.^19,34 In this work, we focus on predicting the percentage peak area for the monomer product and the delta retention time of the monomer peak. Since these assays are primarily used to distinguish problematic molecules from developable ones using well-defined thresholds, we concentrate mainly on a classification problem to differentiate between the two.

2.2. Problem statement

Experimental assays for screening antibodies, such as SEC, can characterize multiple attributes of the molecule under consideration (Section 2.1). For one of such properties, we assume to have a dataset $\mathcal{D}=\{{\mathit{s}}^{(\mathit{i})},o^{(\mathit{i})}\}$ where $\mathit{s}$ denotes the sequence representation of the antibody, i.e. both heavy and light chains. The attribute $\mathit{o}\in \mathbb{R}$ corresponds to the observation value directly obtained through the assay. During the screening stage, the sample is classified as desirable or problematic by comparison with a pre-defined specification, usually based on developability requirements. By denoting this process as $f_{\mathit{screen}}$ we can get the binary label, $\mathit{y}\in \{0,1\}$, i.e. desirable/problematic for each sample as:

$$y^{(\mathit{i})}=f_{\mathit{screen}}(o^{(\mathit{i})})={\mathbf{1}}_{{\mathit{o}}^{(\mathit{i})}\in {\mathcal{I}}_{\mathrm{problematic}}}$$ (1)

1 is an indicator function which labels the sample $1$ (problematic) when the corresponding observation falls within the interval ${\mathcal{I}}_{\mathrm{problematic}}$. This interval expectedly varies across different properties (Figure 1).

Figure 1.

Data distribution of SEC monomer % and $\mathbf{\Delta RT}$. Problematic antibodies are characterized by lower monomer content and $\Delta \mathit{RT}$ values further from zero, as indicated by the red dashed lines in the figure.

Given the data with binary labels, ${\mathcal{D}}_{\mathit{bin}}=\{{\mathit{s}}^{(\mathit{i})},y^{(\mathit{i})}\}$, our goal is to build a classifier to identify an Immunoglobulin G (IgG1) molecule as a desirable (developable) or problematic sample from information embedded in $\mathbf{s}$. Once we have such a classifier trained, this can serve as a surrogate for the experimental assay in the in silico screening process.

2.3. Dataset and train-test split

The SEC dataset, consisting of around 1200 IgG1 molecules, was collected from multiple internal studies. We considered the monomer content percentage and the $\Delta \mathit{RT}$ of the antibodies relative to a reference antibody, NIP228, as previously described. Duplicate samples, defined as multiple assay observations for an identical antibody sequence, were removed for each property. The processed dataset was then divided into training ($90\mathrm{\%}$) and test ($10\mathrm{\%}$) partitions. Due to the dataset’s collection from various studies, the sequences exhibit considerable diversity, with a median of over 100 mutations between pairs. Further details regarding sequence diversity are provided in the supplementary section. Despite this diversity, we still specifically chose the test set to remain diverse, based on clusters formed according to sequence similarities (Levenshtein distance with representative members of clusters chosen randomly^35,36). This selection also ensured a similar stratified distribution of the two classes as observed in the train set which has $71\mathrm{\%}$ and $69\mathrm{\%}$ samples from class $0$ for monomer % and $\Delta \mathit{RT}$ respectively. The test split serves as a hold-out set for evaluating the efficacy of the prediction pipelines (Section 3.2.2).

2.4. Prediction pipeline

The primary data modality in our study is the sequence representation. $\mathit{s}=({\mathit{s}}_{\mathrm{heavy}},{\mathit{s}}_{\mathrm{light}})$ of the antibody. All molecules in our dataset belong to the IgG1 subclass and share a similar constant region. This bias to similar constant region is introduced by the antibody design approach where the variable fragment sequences are lead optimized, followed by grafting to the known constant framework. Consequently, our predictions concentrate exclusively on the impact of sequences within the variable fragments (Fvs). Through the application of the AlphaFold2 (AF2)³¹ or similar protein folding tools, one can also predict the 3D structure folded from the sequence information alone, adding another modality to our prediction pipeline. While explicit incorporation of structure information provides richer information than the sequence for predicting protein properties, one also needs to consider possible errors propagated from the protein structure prediction tool’s inaccuracy.

In this work, we have considered the following four approaches (illustrated in Figure 2) for building the prediction network for the developability property of interest by leveraging different combinations of both modalities. Details of each pipeline are discussed in the subsequent sections.

Figure 2.

Developability property prediction workflow. All four pipelines except PLM (b) leverage the predicted 3D protein structure into their prediction network by either protein descriptor generator (a) or graph neural network (c and d). Note that GNN pipeline (c) utilizes the fixed node attributes which can include the embeddings from a pre-trained protein language model. In the PLM+GNN (d), we allow the PLM to be updated jointly with the GNN during training of the prediction network.

• Sequence and Structure-based Features: An ML classifier predicts the target from the selected protein features, processed by a protein descriptor generator from the antibody sequence and its predicted structure.

• PLM Pipeline: The pipeline utilizes only the sequence information through a combination of a protein language model (PLM) and a prediction head. The latter component is a shallow multilayer perceptron (MLP) network that predicts the property from concatenated heavy and light chain sequences embedding projected by the PLM.

• GNN Pipeline: The graph neural network (GNN) predicts the property from the amino acid (AA) graph constructed from the predicted structure.

• PLM + GNN Pipeline: This incorporates the residue embedding from the protein language model into the GNN pipeline as the node attributes of the AA graph and the combined network is trained jointly.

2.4.1. Prediction utilizing sequence and structural features

We used Schrödinger software to extract the molecular properties from the predicted AF2 static structure https://www.schrodinger.com/(Schrödinger, LLC). Schrödinger software offers an extensive suite of computational tools for molecular modeling, simulation, and extraction of protein descriptors. In our workflow, we utilized AF2 instead of Schrödinger’s homology modeling and employed Schrödinger solely for extracting molecular properties and biophysical features. Schrödinger provides an exhaustive list of patch-level sequence- and structure-based protein properties, including positive and negative charges, hydrophobicity, and aggregation propensities. To ensure we selected the most informative features for our final model, we utilized a combination of unsupervised and supervised feature selection Section 3.2.1. Finally, an Extra Trees classifier³⁷ was applied to the selected features for predicting the developability properties.

2.4.2. PLM-based prediction network from sequence representation

The core idea of the PLM pipeline is to utilize the sequence embedding of an antibody sequence from the protein language model in combination with a simple prediction head network to classify the developability property. For the heavy chain and light chain in the antibody sequence, we consider separate instances of the protein language model, ${\varphi }_{\mathrm{PLM},\mathrm{H}}$ and ${\varphi }_{\mathrm{PLM},\mathrm{L}}$ respectively. We first compute the hidden states of each chain using corresponding instance of PLM. Specifically, we denote residue embedding matrix ${\mathit{E}}_{\mathrm{H}},{\mathit{E}}_{\mathrm{L}}$ as the last hidden state of all tokens of each chain. Note these embedding matrices can include the hidden state of special tokens added for the PLM. Next, we compute the sequence-level representation, i.e. ${\mathit{e}}_{\mathrm{H}},{\mathit{e}}_{\mathrm{L}}$ for each chain by pooling its embedding matrix. We have explored two aggregation techniques to do this – mean pooling where the residue embeddings (excluding any special tokens) are averaged and CLS pooling where embedding for the special token [CLS] is considered as the sequence-level embedding. We have considered [CLS] token since it is often used in end-to-end fine-tuning²⁸ of the language models. Finally, we concatenate the sequence embeddings ${\mathit{e}}_{\mathrm{H}}$ and ${\mathit{e}}_{\mathrm{L}}$ and pass it to a shallow MLP network ${\varphi }_{\mathrm{MLP}}$ to predict the class label (i.e., the logits) for the sequence $\mathbf{s}$.

$${\mathit{E}}_{\mathrm{H}}={\varphi }_{\mathrm{PLM},\mathrm{H}}({\mathit{s}}_{\mathrm{heavy}}){\mathit{E}}_{\mathrm{L}}={\varphi }_{\mathrm{PLM},\mathrm{L}}({\mathit{s}}_{\mathrm{light}})$$ (2)

$${\mathit{e}}_{\mathrm{H}}=\mathrm{Pooling}({\mathit{E}}_{\mathrm{H}}){\mathit{e}}_{\mathrm{L}}=\mathrm{Pooling}({\mathit{E}}_{\mathrm{L}})$$ (3)

$$\hat{y}={\varphi }_{\mathrm{MLP}}({\mathit{e}}_{\mathrm{H}}\oplus {\mathit{e}}_{\mathrm{L}})$$ (4)

In case of general protein language model like ESM2,^38,39 ${\varphi }_{\mathrm{PLM},\mathrm{H}}$ and ${\varphi }_{\mathrm{PLM},\mathrm{L}}$ are initialized at the pre-trained weights. One can also consider the antibody chain-specific pre-trained models like AbLang-1.⁴⁰ When both instances are frozen at their pre-trained model weights, and we learn the prediction head ${\varphi }_{\mathrm{MLP}}$, we denote this as “fixed PLM” approach in our work. On the other hand, we can also fine-tune those PLM instances while training the prediction head by allowing the gradient information backpropagated to the layers of the PLMs. We have investigated two ways of performing this fine-tuning – full parameter fine-tuning and low rank adaptation (LoRA)⁴¹ technique. The latter approach learns low-rank transformation matrices (for the attention networks in our work) that work in conjunction with pre-trained weights to adapt the model for a specific task.

2.4.3. GNN-Based prediction network leveraging structure

To leverage the protein structural information explicitly in the prediction pipeline, the GNN pipeline begins with the construction of the amino acid graph for the antibody sequence $\mathbf{s}$ from the predicted 3D structure. For each residue in the sequence $\mathit{s}$, the top $\mathit{k}$ nearest residues are considered to be its neighbors based on the coordinates of $\mathit{C}-\mathit{\alpha }$ of the residues. Up to this step, the procedure is similar to the approaches of.^28,30,42,43 In our work, however, we add a refinement step by pruning residues that lie outside a predefined local region. Hence, in the resulting AA graph $\mathcal{G}$, the edge from node $\mathit{j}$ to node $\mathit{i}$ exists if residue $\mathit{j}$ is in the $\mathit{k}$-nearest neighbors of residue $\mathit{i}$ and distance of the edge, $||{\mathit{e}}_{\mathit{j}\to \mathit{i}}|{|}_2$ is lower than $r_{\mathrm{thr}}$ Å. In our work, we set $r_{\mathrm{thr}}=9$ which results in around $13$ neighbors on average for the nodes of AA graphs in our dataset.

The $\mathit{m}$ layers of GNN, denoted as ${\varphi }_{\mathrm{GNN}}$ projects the node attribute matrix $\mathit{X}\in {\mathbb{R}}^{L\times F_i}$ of AA graph $\mathcal{G}$ for sequence $\mathbf{s}$ with $\mathit{L}$ residues into the hidden embeddings matrix $\mathit{H}\in {\mathbb{R}}^{L\times F_h}$. $F_i$ and $F_h$ are the dimension of node attributes and hidden representation respectively. The prediction head ${\varphi }_{\mathrm{MLP}}$ transforms the pooled embedding from $\mathit{H}$ into the class probability for the antibody sample.

$$\mathit{H}={\varphi }_{\mathrm{GNN}}\left(\mathit{X},\mathcal{G}\right)$$ (5)

$$\hat{y}={\varphi }_{\mathrm{MLP}}(\mathrm{Pooling}(\mathit{H}))$$ (6)

For the node attributes of the graph, we have considered the one-hot encoding, amino acid properties from,^44,45 node embedding from the pre-trained variational graph autoencoder (VGAE) of⁴⁶ and residue embedding from pre-trained ESM2 (8 M) model. In case of the VGAE and ESM2 (8 M), the weights of these models are frozen at the pre-trained values in the entire training process. To evaluate the effectiveness of GNN in processing the AA graph, three different GNNs are considered – geometric vector perception (GVP),⁴³ graph attention network (GAT)⁴⁷ and graph isomorphism network (GIN).⁴⁸ The main difference between these three is that the GVP leverages the edge features in the message-passing updates while the other two do not. Also, the GVP considers the scalar and vector features for the nodes and edges where the scalar features of the nodes are based on dihedral angles and one of the four node attributes mentioned earlier. In case of other two GNNs, only the node attributes are utilized.

After $3$ layers of GNN, the updated node embeddings are aggregated to get the global embedding for the entire graph. Here, we have considered the mean pooling technique which takes an average of node embeddings across all nodes of the graph. In summary, we have explored 4 choices of node attributes and 3 choices of GNN for each developability property.

2.4.4. Prediction network leveraging PLM and GNN

In terms of architectural similarity, the PLM+GNN pipeline is the same as the GNN-based prediction network where the node attributes of the amino acid residue graph are assigned by the pre-trained protein language model. While in the GNN pipeline, we only learn the GNN modules during training, the PLM+GNN pipeline additionally facilitates the adaptation of the parameters of the PLM leveraging the gradients at the node attributes backpropagated from the GNN modules as in.²⁸ For an antibody sequence $\mathit{s}$, first the PLM module ${\varphi }_{\mathrm{PLM}}$ computes the node attribute matrix $\mathit{X}$. The rest of the steps are similar to the GNN pipeline where $\mathit{X}$ is transformed to the prediction through ${\varphi }_{\mathrm{GNN}}$ and ${\varphi }_{\mathrm{MLP}}$ by using the corresponding amino acid graph $\mathcal{G}$.

$$\mathit{H}={\varphi }_{\mathrm{GNN}}\left({\varphi }_{\mathrm{PLM}}(\mathit{s}),\mathcal{G}\right)$$ (7)

$$\hat{y}={\varphi }_{\mathrm{MLP}}(\mathrm{Pooling}(\mathit{H}))$$ (8)

For this approach, we have considered 3 PLMs – (ESM2 (8 M),³⁸ AbLang-1⁴⁰ and AbLang-2⁴⁹). For the ESM2 (8 M), we allow the gradient flow through its all layers excluding the embedding layer. In the case of fine-tuning of AbLang-1 and AbLang-2, the first $6$ layers are frozen at their pre-trained weights while the rest of the layers are updated based on the gradient backpropagated from the GNN module. In addition to the global mean pooling of the GNN pipeline, we have investigated the effectiveness of universal pooling^30,50 over the mean pooling technique. Although the sequences in our work are not processed using the multiple sequence alignment (MSA) technique, our exploration attempts to analyze the robustness of this pooling technique for three different choices of GNN.

3. Result and discussion

3.1. Processing of 3D protein structure

In this work, we utilized two state-of-the-art computational tools for predicting protein structures: AlphaFold 2 and ImmuneBuilder. AlphaFold 2, developed by DeepMind, has revolutionized the field by accurately predicting protein structures from amino acid sequences, applicable across a wide range of proteins.³¹ ImmuneBuilder, on the other hand, is tailored for rapid and accurate predictions of immune protein structures, such as antibodies and T-cell receptors. Noted for its speed, ImmuneBuilder is significantly faster than AlphaFold 2⁵¹ and more suitable for high-throughput screening of large numbers of antibodies. Therefore, we conducted a comprehensive comparison of the structure prediction models in Section 3.2.3.

3.2. Experiments

3.2.1. Selecting the best combination for each pipeline

We have explored different combinations for each of the four prediction pipelines discussed in Section 2.4. In each pipeline, we have run 10 trials of 10-fold cross-validation on the SEC dataset using different combinations and selected the best configuration based on the average prediction performance on the validation splits. In the Supplementary Information, we have provided pipeline-specific procedures and explained the corresponding cross-validation results in detail.

3.2.2. Results on test set

From the cross-validation experiments of Section 3.2.1, we have only identified the best combination for each pipeline. Next, we have trained each best combination with 10-fold cross-validation with the training split and evaluated the trained models’ performance on the hold-out test set. Specifically, we split the training data (90% of the SEC dataset) into 10-fold cross-validation splits, and we pick the model with best performance (accuracy) on the validation split in each of 10 folds. These 10 trained models for each pipeline are applied on the hold-out test set (Section 2.3) to measure the predictive performance. Tables 1 and 2 show the average and standard deviation of performance metrics across these 10 models for each pipeline corresponding to monomer percentage and $\Delta \mathit{RT}$ respectively.

Table 1.

Prediction performance on the hold-out test set for SEC monomer %. The majority class (label $0$) predictor has an accuracy of $71\mathrm{\%}$. ESM2 (8 M) model fine-tuned via LoRA technique produces the sequence embedding for “PLM” pipeline. For “GNN” pipeline, the embedding from pre-trained VGAE is used with 3 layers of GVPs. For “PLM+GNN,” a combination of AbLang1 and GVP is trained in an end-to-end fashion. The best-performing pipeline is highlighted for each metric based on their average performance over the hold-out test set for $10$ models from $10$-Fold cross-validation.

Prediction Pipeline	Accuracy	F1	Sensitivity	Precision
PLM	0.75 (0.03)	0.54 (0.04)	0.52 (0.05)	0.58 (0.13)
GNN	0.74 (0.02)	0.49 (0.03)	0.44 (0.04)	0.58 (0.12)
PLM + GNN	0.75 (0.01)	0.50 (0.04)	0.43 (0.06)	0.61 (0.14)
Sequence + Structure based features	0.77 (0.01)	0.49 (0.04)	0.38 (0.05)	0.66 (0.15)

Table 2.

Prediction performance on the hold-out test set for SEC $\Delta \mathit{RT}$. The majority class (label $0$) predictor has an accuracy of $69\mathrm{\%}$. The “PLM” pipeline utilizes the ESM2 (8 M) model obtained after the full parameter fine-tuning with the training set. For “GNN” pipeline, the embedding from pre-trained ESM2 (8 M) is used with 3 layers of GVPs. For “PLM+GNN,” a combination of AbLang1 and GVP is trained in an end-to-end fashion. The best-performing pipeline is highlighted for each metric based on their average performance over the hold-out test set for $10$ models from $10$-Fold cross-validation.

Prediction Pipeline	Accuracy	F1	Sensitivity	Precision
PLM	0.77 (0.02)	0.60 (0.04)	0.56 (0.06)	0.66 (0.12)
GNN	0.80 (0.03)	0.64 (0.07)	0.58 (0.10)	0.68 (0.15)
PLM + GNN	0.78 (0.04)	0.61 (0.13)	0.57 (0.15)	0.58 (0.19)
Sequence + Structure based features	0.80 (0.01)	0.59 (0.03)	0.46 (0.05)	0.77 (0.12)

For both properties, the pipeline utilizing sequence and structure-based features achieves the highest accuracy while performing poorly in selecting problematic molecules (class label 1) as evidenced by the low value in sensitivity. The other three pipelines show slightly lower accuracy in SEC monomer percentage and out of these three, the PLM pipeline produces better F1 score and sensitivity. This result is particularly interesting for high-throughput screening since this PLM pipeline predicts the developability property from the sequences, making it much faster than “sequence + structure-based features” which requires computationally time-consuming feature processing by the Schrödinger suite. In the case of the SEC $\Delta \mathit{RT}$, the GNN pipeline and the feature-based pipeline have similar accuracy, but the former shows better sensitivity. When the structural information is combined with the PLM approach resulting in PLM+GNN, the average performance stays similar to the PLM approach (within the standard deviation) but this causes larger variation in performance across different models learned in each of the 10 folds.

All four pipelines have a similar standard deviation in F1 score (and sensitivity) for the SEC monomer percentage property. On the other hand for SEC $\Delta \mathit{RT}$, the GNN and PLM+GNN pipelines show comparatively larger fluctuations in similar performance metrics. For example, the standard deviations in sensitivity for these two pipelines are $0.10$ and $0.15$ respectively (Table 2) which are 2–3 times larger than the other two pipelines. Given that the PLM approach shows similar fluctuations as the pipeline with sequence and structure-based features, this higher variation in the performance metrics possibly originates from the sensitivity of GNN components to AA graphs seen in the 10-fold cross-validation training. Additionally, the difference in this trend between the monomer percentage and $\Delta \mathit{RT}$ indicates that the structural information of the antibody molecules may have more importance in the predictive performance of the latter property.

3.2.3. Ablation study for different structure prediction tools

The results in Tables 1 and 2 are for the experiments utilizing the predicted structure from AlphaFold2. Despite having a high structure prediction performance, AlphaFold2 can be a source of bottleneck in the high-throughput screening process due to its longer structure prediction time. In this section, we used a faster structure prediction tool, ImmuneBuilder⁵¹ for the three pipelines that use antibody structure in predicting the developability properties. By comparing with the previous results (from AlphaFold2), we assessed the robustness of the developability prediction pipeline to the predicted structures.

For three developability prediction pipelines – Sequence + Structure based features, GNN, PLM+GNN – we have repeated the hold-out test set experiment from Section 3.2.2. The models under each pipeline have the same architecture as in Section 3.2.2 but are trained with the datapoints where the structures are predicted via ImmuneBuilder. Tables 3 and 4 have these hold-out set performance metrics for monomer % and $\Delta \mathit{RT}$ respectively. We have also shown the results with AlphaFold2 (from Tables 1 and 2) for reference. For both properties, we observe a decline in the sensitivity (and F1 score) for Sequence + Structure based features and GNN pipelines when we replace AlphaFold2 with ImmuneBuilder as the antibody structure prediction tools. The performance of PLM+GNN pipeline remains similar for SEC monomer % while we see a more stable performance trend for SEC $\Delta \mathit{RT}$. With AlphaFold2, the GNN pipeline is the best performing pipeline for $\Delta \mathit{RT}$ but has a relatively large performance variation. In the PLM+GNN pipeline for $\Delta \mathit{RT}$, the ImmuneBuilder not only works as a faster structure prediction tool but provides a comparable performance consistently ($0.62(0.04)$ vs $0.61(0.13)$ with AlphaFold2).

Table 3.

Variation in predictive performance for SEC monomer % due to different structure prediction tools. The neural network architectures for three prediction pipelines are the same as in Table 1.

Prediction Pipeline	Structure Prediction Tool	Accuracy	F1	Sensitivity	Precision
GNN	AlphaFold2	0.74 (0.02)	0.49 (0.03)	0.44 (0.04)	0.58 (0.12)
	ImmuneBuilder	0.75 (0.02)	0.46 (0.04)	0.38 (0.04)	0.59 (0.15)
PLM + GNN	AlphaFold2	0.75 (0.01)	0.50 (0.04)	0.43 (0.06)	0.61 (0.14)
	ImmuneBuilder	0.74 (0.02)	0.49 (0.05)	0.44 (0.08)	0.56 (0.15)
Sequence + Structure based features	AlphaFold2	0.77 (0.01)	0.49 (0.04)	0.38 (0.05)	0.66 (0.15)
	ImmuneBuilder	0.76 (0.01)	0.45 (0.04)	0.34 (0.04)	0.64 (0.16)

Table 4.

Variation in predictive performance for SEC $\Delta \mathit{RT}$ due to different structure prediction tools. The neural network architectures for three prediction pipelines are the same as in Table 2.

Prediction Pipeline	Structure Prediction Tool	Accuracy	F1	Sensitivity	Precision
GNN	AlphaFold2	0.80 (0.03)	0.64 (0.07)	0.58 (0.10)	0.68 (0.15)
	ImmuneBuilder	0.78 (0.03)	0.59 (0.07)	0.52 (0.10)	0.64 (0.16)
PLM + GNN	AlphaFold2	0.78 (0.04)	0.61 (0.13)	0.57 (0.15)	0.58 (0.19)
	ImmuneBuilder	0.79 (0.01)	0.62 (0.04)	0.55 (0.06)	0.71 (0.12)
Sequence + Structure based features	AlphaFold2	0.80 (0.01)	0.59 (0.03)	0.46 (0.05)	0.77 (0.12)
	ImmuneBuilder	0.79 (0.01)	0.54 (0.02)	0.40 (0.02)	0.81 (0.09)

4. Conclusion

Our work has explored four different prediction approaches for two developability properties from the SEC assays: monomer percentage and $\Delta \mathit{RT}$. We aimed to provide promising in silico models that can be used to screen less developable molecules at an early stage without the need for time- and material-intensive experimental methods. We investigated high-throughput models that require only sequences (protein language models) as well as models that require protein structure prediction and feature extraction from predicted structures, which are computationally more demanding. Specifically, we examined whether protein language models and graph neural networks can effectively utilize sequence and structural information to achieve performance comparable to methods employing explicit structural and sequence features calculated by Schrödinger software. For both properties, the performance on the hold-out test set shows that the latter approach achieves higher accuracy but suffers from considerable degradation in its ability to identify molecules violating the developability constraint. The best-performing pipeline (in terms of F1 score) for monomer percentage is the protein language model (PLM) approach, which offers a promising opportunity for high-throughput screening due to its faster inference time from the sequence of the antibody chains. Further comparison of variations in the performance metrics for $\Delta \mathit{RT}$ and monomer percentage indicates the relative importance of structural information in predicting these two properties. Finally, we have identified (in Section 3.2.3) the potential of the PLM+GNN pipeline as a high-throughput screening tool for $\Delta \mathit{RT}$, with ImmuneBuilder (replacing AlphaFold2) as the protein structure prediction tool.

Since the dataset used in our work was compiled from multiple projects, it naturally encompasses a diverse range of sequences. Nonetheless, in Lead Optimization (LO) projects, new antibodies may exhibit mutations in regions that were previously unmutated in the training data, and Lead Identification (LI) projects might yield sequences that are significantly different. Specifically, the prediction pipelines that use the predicted structure, e.g., sequence and structure-based features, PLM+GNN and GNN, is likely to be negatively impacted by the structure prediction model’s low sensitivity to the small number of mutations, which are shared by the similar sequences in the LO projects. However, the PLM-based pipeline may be more robust in such scenario, as PLMs are shown to be more effective in encoding the mutational change in protein fitness. Assessing the robustness of the prediction pipelines in handling this challenge of out-of-distribution data is an important area for further exploration. Given the cost of measuring developability properties, our dataset of approximately 1,200 antibodies is considered moderately large. For other new or expensive low-throughput assays, we might not have access to such a large dataset, which makes it challenging to directly leverage the protein language model and graph neural networks. In those cases, it would be interesting to see whether the trained models using the relatively large dataset can give an edge to learning the prediction networks for low-data assay through transfer learning technique.⁵² A few recent works^53–55 explicitly incorporate the structural information into the training of the protein language models. Exploration of their applicability in developability prediction can provide further insights into the importance of explicit structural information in protein property prediction.

Funding Statement

The author(s) reported there is no funding associated with the work featured in this article.

Disclosure statement

M.B., I.S., J.G.C, V.S., N.M., G.K., J.S., R.C.W. and M.P. are employees of AstraZeneca. A.N.M.N.A. was employed as an intern at AstraZeneca.

Author contributions

Conceptualization: M.P., A.N.M.N.A., M.B., V.S., J.G.C., N.M., G.K., R.C.W.; Methodology: A.N.M.N.A., M.P.; Data Processing: I.S., G.K., A.N.M.N.A., M.P.; Visualization: A.N.M.N.A., M.P.; Supervision: M.P., J.S., R.C.W.; Writing – Original Draft: A.N.M.N.A., M.P.; Writing – Review and Editing: A.N.M.N.A., M.P., N.M., J.G.C., V.S., M.B.

Supplementary Information

Supplemental data for this article can be accessed online at https://doi.org/10.1080/19420862.2025.2562997