Beyond native sequence recovery: Improved modeling of the sequence-energy landscape of protein structures

Abstract

Computational protein design using machine learning models has advanced rapidly since the introduction of AlphaFold2. There is now a suite of tools that enable in silico design of proteins with desired structures and properties. Most design workflows require fitting a designed backbone with a sequence that stabilizes it, and many machine learning sequence design models have been proposed. These models are trained to recover the native sequence paired with a known structure, a task known as native sequence recovery (NSR). Here, we demonstrate the limitations of optimizing a sequence design model only for NSR. We show that NSR is often misaligned with more important metrics of model performance: the compatibility of the generated sequence with the desired fold and the ability of the model to predict the energetic effects of mutations. We introduce PottsMPNN, which is trained to generate a Potts energy function consisting of single-residue and residue-pair terms from a protein backbone, and we demonstrate that learning a Potts model reduces NSR but improves sequence generation and energy prediction. We also trained PottsMPNN with noised backbone structures and multiple sequence alignments. In tests on held-out data, NSR decreased, but the quality of the designed sequences and energy predictions improved. By demonstrating the limitations of optimizing for NSR and the effectiveness of strategies that avoid NSR overoptimization, our work advances sequence design and highlights future directions for the broader protein design field.

Introduction

Computational protein design can address outstanding needs in many areas, including human health and sustainability (1). A common design approach is to generate a protein backbone structure with desired properties (e.g., shape complementarity to a desired binding target) and then design a sequence that is consistent with that backbone (1). Deep learning methods that can be used for this task have advanced rapidly (2). Broadly, such techniques involve optimizing a mathematical model defined by a specific functional form and a set of parameters. The parameters are determined through a training process that involves learning from examples. In the field of protein modeling and design, AlphaFold (versions 1 through 3) is a deep learning method that can predict a protein structure from a protein sequence (3–5), and RFdiffusion is a deep learning method that can generate novel, realistic protein structure backbones (6). Both models were trained using large numbers of experimentally determined or predicted protein structures.

Several deep learning methods have been trained to predict amino-acid sequences compatible with a given protein backbone structure (7–15). These models learn the probability distribution over possible sequences conditioned on the input backbone, $P_{\theta }(s\mid f)$, where $s$ is the sequence, $f$ is the backbone, and $\theta$ represents the learned parameters. Of these models, ProteinMPNN is the most widely used for protein design (10). ProteinMPNN was built on foundational work that established that using a graph neural network (GNN), with residues as graph nodes and model parameters $\theta$ describing the properties of nodes and relationships between them, facilitates learning sequence-structure relationships (7). In this framework, for a protein of length $L,P_{\theta }(s\mid f)$ is represented as $L$ single-site probability distributions over the 20 standard amino acids that are extracted from the node embeddings in the graph.

Li et al. developed a model, COORDinator, that is based on an alternative framework where $P_{\theta }(s\mid f)$ is represented as a function that decomposes the sequence-energy landscape into a sum of self energies and pair energies, expressed as $H\left(s\right)=\sum _{i=1}^{\left|s\right|}{h}_i\left(s\right[i\left]\right)+\sum _{i=1}^{\left|s\right|}\sum _{j>i}^{\left|s\right|}{J}_{ij}\left(s\right[i],s[j\left]\right)$ for self energies $h$ and pair energies $J$ (see Potts model for details) (11). This functional form is called a Potts model. Such models, along with other pairwise interaction analyses such as double-mutant cycles, have been used to model protein energies for decades because protein folding and stability emerge from networks of cooperative pairwise interactions, such as hydrogen bonds, hydrophobic packing, and salt bridges (16–20). Li et al. and several other groups have shown that learning a pairwise-decomposable energy function from a backbone structure can outperform learning single-site probability distributions on some tasks (11, 21–23).

One way to assess the success of a sequence design model is to determine whether the sequences it generates fold into the desired structures. A definitive assessment requires experimental testing, but as a computational proxy, and to determine the sequences most worth testing, tools like AlphaFold and the structure-modeling method Rosetta (24) are frequently used to assess the quality of sequence designs. Because designed sequences lack homologs, AlphaFold is run without multiple sequence alignments (MSAs) or templates when used as a design filter (10). Design pipelines that generate large numbers of sequences with ProteinMPNN, score them with AlphaFold or Rosetta, and retain only the high-scoring designs have generated sequences that adopt desired structures with high experimental success rates (25–28).

Because sequence design models learn the probability distribution of residues given a protein backbone, they can also predict the effect of mutations (9, 11, 29, 30). This task relies on the learned probability distribution $P_{\theta }(s\mid f)$ matching the real sequence-energy landscape for a given structure: i.e., if a model evaluates one sequence to be more likely than another, that should reflect greater protein stability. Because sequence design models are trained to predict the probability of a sequence given a structure, they should be evaluated using physical properties that are closely tied to structural features (e.g., folding and binding energies) as opposed to molecular or cellular functions (e.g., effects on protein expression or cell growth) that are indirect readouts of such thermodynamic properties (31).

The two use cases described above provide two optimization objectives for sequence design models: (1) sequence-structure self-consistency, which assesses the compatibility of the designed sequence with the desired structure, and (2) mutation-effect prediction, which assesses the accuracy of the learned sequence-energy landscape and is quantified by the correlation between experimentally observed and predicted mutation energies. Unfortunately, there are impediments to training on either objective. The sequence-structure self-consistency scores are either nondifferentiable with respect to sequence (Rosetta) or slow to compute and difficult to optimize in sequence space (AlphaFold). And although energy prediction is an effective objective on which to fine-tune the weights of a previously trained model (29, 30), there are insufficient experimental energy data to train a generalizable model from scratch.

Because of these limitations, sequence design models are trained on the task of native sequence recovery (NSR). Given a pairing of backbone structure, $f_{\mathit{\text{nat}}}$, and native sequence, $s_{nat}$, from the Protein Data Bank (PDB) (32) or AlphaFoldDB (9, 33, 34), the task is to maximize $P_{\theta }\left(s_{nat}\mid f_{nat}\right)$. Training on NSR is empirically effective. The reference energies for amino acids in the Rosetta energy function were set by optimizing for NSR (24, 35), and Rosetta has been successfully used to design many proteins (36). ProteinMPNN was trained on NSR and can design sequences that fold to intended structures and can be fine-tuned to predict energies with state-of-the-art performance (29, 30).

However, there are several reasons why optimizing for continual improvement in NSR may fail to produce corresponding improvements in sequence-structure self-consistency and energy prediction. A fundamental challenge is that the relationship between protein sequence and structure is many-to-many, not one-to-one. For any given fold $f_{nat},s_{nat}$ is not the only sequence that will adopt that fold; many proteins that share only 40% sequence identity adopt highly similar structures (37, 38). Furthermore, the native sequence is usually not the sequence that maximizes the stability of a structure. Deep mutational scanning experiments show that although most mutations in native proteins are energetically unfavorable, many positions have at least one favorable substitution relative to wildtype (39–41). Furthermore, a protein sequence $s_{\mathit{\text{nat}}}$ does not adopt a single structure: the structures of proteins are over-specified by the atomic coordinates deposited in the PDB, which capture a single conformation that lies in a local optimum of the force field used for structure refinement. Optimizing for similarity to $s_{nat}$ ignores sequence-structure degeneracy and the fact that evolution may select for sequences that are good enough to function and not necessarily optimal in any way. Continuously maximizing $P_{\theta }\left(s_{nat}\mid f_{nat}\right)$ may therefore result in an unrealistic model of the sequence-energy landscape.

We benchmarked state-of-the-art sequence design models on sequence-structure self-consistency and energy prediction. We found that ProteinMPNN (10), Frame2Seq (12), and Caliby (23) achieved the best performance, even though they generate sequences with lower NSR compared to other models. We then tested several ways of altering the training task to better reflect principles underlying protein sequence-structure relationships. First, we developed PottsMPNN, a sequence design model that is trained to learn a Potts model and, simultaneously, single-site amino-acid probability distributions. PottsMPNN achieved lower NSR but outperformed all models on sequence-structure self-consistency and energy prediction. Second, consistent with an observation by Dauparas et al., we found that training with coordinate noise improves sequence-structure self-consistency (10). We show that introducing noise during training also improves energy prediction and that training with noise is beneficial because it prevents overfitting to NSR during training. Third, and most importantly for reducing reliance on NSR, we trained on MSAs to directly provide the model with examples of related sequences that adopt a shared fold. Doing so improved both sequence-structure self-consistency and energy prediction. Our results demonstrate that NSR is not the right objective for state-of-the-art sequence design models and that optimizing for more biologically appropriate objectives increases model performance.

Results

PottsMPNN architecture.

PottsMPNN is a GNN that represents the protein backbone structure as a graph. Nodes in the graph represent residues and are used to predict single-site amino acid probability distributions, and edges in the graph represent interactions between residues and are used to predict energies. PottsMPNN generates sequences autoregressively by iterating over the node representations and selecting high-probability amino acids at each site. PottsMPNN generates a Potts model by using the self edges in the graph to predict the self energies $\left(h\right)$ and using the pair edges to predict the pair energies $\left(J\right)$ (Fig. 1; see Model architecture for details). The Potts model is used to score sequence variants, such as those resulting from point mutations.

Fig. 1.

PottsMPNN is a graph neural network that learns a Potts model energy function and single-site residue probabilities conditioned on protein structure. The model input is a protein structure backbone that is used to define a $k$-NN graph. The nodes and edges of the graph are encoded using a message-passing neural network. The edges, including self edges, are supervised to learn single and pairwise residue interaction energies in the form of a Potts model $\left(H\right(s\left)\right)$, which is used to compute the energies of sequences conditioned on structure. The Potts model is supervised using a negative log composite pseudo-likelihood loss $\left({ℒ}_E\right)$ that maximizes the probabilities of native-residue pairs. The nodes are decoded autoregressively to generate single-site amino-acid probabilities and are supervised by a negative log-likelihood loss $\left({ℒ}_V\right)$ that maximizes the likelihood of the native sequence; during inference, the nodes are used to generate a sequence $\left(s_{\mathit{\text{gen}}}\right)$. See Methods for details.

PottsMPNN integrates two models: COORDinator (11) and ProteinMPNN (10). PottsMPNN uses the same architecture as ProteinMPNN, with the addition of a single neural network layer that applies a linear function to reshape the edge embeddings into the form of a Potts model. The Potts model is supervised using the negative log composite pseudo-likelihood loss developed for COORDinator, which rewards the model for assigning a high probability to the native pair of residues at a given pair of positions (graph edge loss ${ℒ}_E$). As observed by Li et al., the Potts model learned using this loss function can predict energies effectively, but the sequences derived from using Markov chain Monte Carlo (MCMC) to sample from the Potts model are often low complexity, and running MCMC sampling takes a long time (11). Accordingly, PottsMPNN retains the autoregressive sequence generation method from ProteinMPNN to generate single-site probability distributions at each graph node that are supervised to reward the model for assigning a high probability to the native residues at each site (graph node loss ${ℒ}_V$). The model is trained using a composite loss that is the sum of ${ℒ}_E$ and ${ℒ}_V$, equally weighted.

Model benchmarking.

We compared PottsMPNN with many other sequence design models. Because the data on which models were trained can have a significant effect on performance, we focused on models that were trained—or that we retrained—on the structurally nonredundant dataset curated by Ingraham et al. (7) from the CATH 4.2 database (42) with no CATH topology classification overlap between the training, validation, and test sets (we refer to this nonredundant dataset as the CATH 4.2 dataset; see Datasets for details). These models are ProteinMPNN (10), GVP (8), Frame2Seq (12), KW-Design (13), and MapDiff (15). We also tested UniIF (14), which was trained on a nonredundant dataset curated from the CATH 4.3 database. We tested ProteinMPNN because it provided the model architecture that we used to train PottsMPNN and because it is the most widely used model for protein design; we retrained ProteinMPNN on the CATH 4.2 dataset to facilitate direct comparisons and, because ProteinMPNN was originally trained on the larger PDB-clust dataset (10) (see Datasets for details) that includes multichain proteins, we also trained versions of PottsMPNN on the PDB-clust dataset. We tested GVP because it introduced the Geometric Vector Perceptron graph neural network layer, an architecture used by several other sequence design models, including ESM-IF (9). We selected the other four models to include those with among the highest reported NSR (MapDiff and UniIF), one model with intermediate NSR (KW-Design), and one model shown to be experimentally successful even at low NSR (Frame2Seq).

Sequence-structure self-consistency was evaluated on the CATH 4.2 or PDB-clust test sets in two ways. First, we predicted structures of generated sequences with AlphaFold2 and calculated the TM-score (43) between predicted and native structures. TM-score is a metric of structural similarity that ranges from 0 to 1, with 0 indicating no structural similarity and 1 indicating structural identity. Second, we calculated length-normalized Rosetta energies for proteins constructed by fitting generated sequences onto the native structures. Rosetta provides an assessment of biophysical structural plausibility that is orthogonal to AlphaFold-derived metrics and is frequently used to assess designed protein quality (25, 26). See Sequence-structure self-consistency for details.

To validate model performance using experimental data, we evaluated energy prediction on three large datasets of point mutation effects: the Megascale dataset (202,804 mutations across 232 proteins) (40), FireProt (2,301 mutations across 64 proteins) (44), and SARS-CoV-2 RBD (~ 4,000 mutations) (41) (see Energy prediction for details). Notably, many Megascale proteins are de novo designs, providing a stringent test of generalization to non-natural sequences.

Figure 2 shows the results of PottsMPNN and six sequence design models trained without noise on NSR, sequence-structure self-consistency, and energy prediction. Fig. 2A shows that the NSR of ProteinMPNN and PottsMPNN are very low relative to other models: the best model, MapDiff, achieves ~58% NSR, while PottsMPNN achieves only ~44% NSR. However, Figs. 2B–C show a very different trend for sequence-structure self-consistency. PottsMPNN achieves the highest sequence-structure self-consistency: PottsMPNN sequences folded by AlphaFold2 have a significantly higher average TM-score to the native structures compared to the sequences generated by all other models (Fig. 2B), and PottsMPNN sequences threaded onto the native structure have significantly better Rosetta scores than the sequences generated by all other models except Frame2Seq (Fig. 2C). The improved sequence-structure self-consistency of PottsMPNN sequences is also apparent when examining the fraction of sequences with TM-scores above various thresholds (SI Appendix, Fig. S1) and the confidence of the predicted structures (SI Appendix, Fig. S2).

Fig. 2.

NSR does not correlate with performance on sequence-structure self-consistency or energy prediction for models trained on the CATH 4.2 dataset (except for UniIF, which was trained on the CATH 4.3 dataset) and tested on the benchmarking subset of the CATH 4.2 test set. (A) NSR results for each model. (B) TM-scores between native structures and AlphaFold2 predicted structures for sequences generated using each model. (C) Rosetta scores (multiplied by −1 and normalized for sequence length) after threading generated sequences onto the native backbone and relaxing for each model. (D) Pearson r for using each model to predict the effect of single-site mutations on protein stability for three datasets. For (A - C), error bars show SEM over the proteins in the test set after averaging results over three retrained model replicates for ProteinMPNN and PottsMPNN; for (D), error bars show SEM over three retrained model replicates for ProteinMPNN and PottsMPNN. Stars indicate statistical significance relative to PottsMPNN, assessed using a two-tailed paired $t$ test over per-protein values for (A - C) and a two-tailed unpaired t-test over average Pearson r values for (D) (*P < 0.05, **P < 0.01, ***P < 0.001, ****P < 0.0001; for (A - C), no star indicates nonsignificance).

For stability prediction, PottsMPNN outperforms the other models (Fig. 2D). A case study of 2KXD (45, 46), a protein in the Megascale protein stability test set, shows that PottsMPNN predictions are more correlated with experimental values than are ProteinMPNN predictions (SI Appendix, Figs. S3A–B). Coloring the structure by persite Pearson r values shows that PottsMPNN is especially good at predicting the effects of mutations in the core of the protein (SI Appendix, Figs. S3C–D).

We observed the same trend in expanded benchmarking that included additional models and models trained on the significantly larger PDB (32) or AlphaFoldDB (9, 33, 34) datasets: models with the highest NSR do not have the best sequence-structure self-consistency or energy prediction performance (SI Appendix, Table S1).

Local sequence optimization using energy predictions.

Given that the Potts model performs well when predicting the effects of single-site mutations, we hypothesized that it could improve the sequences generated autoregressively using the single-site probability distributions. To test this, we visited each position in the sequence and used the Potts model to find favorable substitutions (see Sequence generation for details). We tracked how sequence-structure self-consistency changed during the local optimization process, and we found that both TM-scores and Rosetta scores consistently improved, with notable alignment between the change in energies predicted by the Potts model and the change in Rosetta scores (SI Appendix, Fig. S4). Energies predicted using single-site probabilities did not reflect that the sequences improve in quality as the local optimization progresses (SI Appendix, Fig. S4). Iterative optimization to convergence did not result in continued improvement in sequence quality after each site had been visited once (SI Appendix, Fig. S5). Because the local optimization requires scoring every possible mutation at all positions in the sequence, it requires more inference time, but sequence generation remains fast at 0.32 ± 0.16 seconds per sequence (SI Appendix, Fig. S6).

The importance of explicitly modeling pairwise interactions.

We explored why PottsMPNN is superior to ProteinMPNN for energy prediction. PottsMPNN learns a Potts model from the edges in the graph embedding of the protein—by supervising joint probability distributions at pairs of sites—and simultaneously learns single-site probability distributions. In contrast, ProteinMPNN only learns single-site probability distributions. To test if the Potts model learned by PottsMPNN is critical to its performance, we compared the sequence scoring ability of the Potts model from PottsMPNN with the sequence scoring ability of the single-site probability distributions from PottsMPNN. Although these two functions are learned simultaneously during training, subject to a joint loss function, using the single-site probabilities to predict energies resulted in significantly worse performance than using the Potts model; using the single-site probabilities from PottsMPNN performs equivalently to using the single-site probabilities from ProteinMPNN (SI Appendix, Fig. S7A). Consistent with this, when we used the single-site probabilities for local optimization of sequences generated using the autoregressive decoder, we observed significantly lower sequence-structure self-consistency scores than we did when using the Potts model for local optimization (SI Appendix, Figs. S7B–D). Indeed, the TM-scores for single-site locally optimized sequences were significantly worse than the TM-scores for the original sequences (SI Appendix, Fig. S7C). The Potts model energies recognize that the sequences do not improve in quality as the single-site local optimization progresses (SI Appendix, Fig. S8). These tests establish the utility of the Potts function as an output of the model.

To test whether supervising residue pairs during training is critical to learning an informative Potts model, we trained two simplified versions of PottsMPNN: PottsOnlyMPNN and PottsSingleMPNN (SI Appendix, Fig. S9; see Loss functions and optimization for details). Both models were trained only to learn a Potts model.

PottsOnlyMPNN was trained with the standard graph edge loss ${ℒ}_E$ that supervises the model predictions for pairs of positions based on the self and pair energies from the Potts model that involve each pair. PottsSingleMPNN, in contrast, was trained only on the model predictions for single sites, where each single-site probability distribution was calculated from the Potts model by summing the self energies at that site and the pair energies with the native residues at neighboring sites. PottsOnlyMPNN performs similarly to PottsMPNN at energy prediction, but PottsSingleMPNN performs significantly worse, supporting the importance of supervising residue pairs during training (SI Appendix, Fig. S7A).

We also tested whether certain pairwise interactions in the Potts model are more important than others by summing over different numbers of residue pairs at inference time. Progressively removing residue pairs substantially reduced energy prediction performance (SI Appendix, Fig. S10). Removing interactions between near neighbors had a greater effect than removing interactions between far neighbors for most tests.

Adding noise during training.

We tested the effect of adding increasing amounts of noise to the input structures during training. For these experiments, models were trained on the larger PDB-clust dataset. Following Dauparas et al., noise was added by perturbing the position of each atom by sampling a displacement in each coordinate axis independently from a Gaussian distribution of a set standard deviation, $\sigma$, which we refer to as the noise level (10). As the amount of noise added during training increased, the models generated sequences with lower NSR but significantly higher sequence-structure self-consistency scores, and they performed significantly better at energy prediction on the Megascale test set, up to 0.3 Å of noise (Fig. 3A–C).

Fig. 3.

Training PottsMPNN on the PDB-clust dataset with noise improves performance and prevents overfitting to NSR. (A) NSR decreases, whereas the TM-score between the native structure and AlphaFold2 structure predicted from generated sequences improves with increasing noise. (B) Rosetta scores from modeling generated sequences on native backbones improve with increasing noise up to 0.3 Å. (C) Pearson r for predicting the Megascale single-residue mutation energies improves with increasing noise up to 0.3 Å. For (A - B), error bars represent SEM over the proteins in the test set after averaging results over at least two retrained model replicates; for (C), error bars show SEM over at least two retrained model replicates. (D - F) TM-score (D), Rosetta score (E), and Megascale Pearson r values (F) for various epoch checkpoints when PottsMPNN was trained without noise. (G - I) TM-score (G), Rosetta score (H), and Megascale Pearson r values (I) for various epoch checkpoints when PottsMPNN was trained with 0.2 Å of noise. For (D), (E), (G), and (H), error bars show SEM over the proteins in the PDB-clust test set.

To investigate why adding noise during training improves performance, we compared the training and validation loss curves and sequence-structure self-consistency and energy prediction performance at selected epochs of training for two models: one trained without noise (Fig. 3D–F) and one trained with 0.2 Å of noise (Fig. 3G–I). The model trained without noise exhibits signs of overfitting to NSR: the model continually generates sequences that more closely match the native sequence, but these sequences are not more compatible with the target structure. However, the model trained with noise exhibits the opposite behavior: sequence-structure self-consistency either fluctuates around a constant value (AlphaFold2 TM-scores) or improves up to epoch 100 (Rosetta scores). Also, the model trained with noise shows greater improvement in energy prediction over the course of training. The train loss for the model with noise is higher than the validation loss because noise is only added during training.

Training using MSAs.

Because many protein sequences can adopt the same structure, we tested the effect of modifying the loss function by averaging the loss over sequences sampled from an MSA (see Loss functions and optimization and Datasets for details, including how the MSAs were constructed; SI Appendix, Fig. S11 and Fig. S12). We compared models trained on the CATH 4.2 dataset with and without noise and with and without MSAs. Training on MSAs significantly reduced NSR (Fig. 4A) but significantly increased sequence-structure self-consistency (Fig. 4B–C). There is a positive, albeit modest, effect of using MSAs on energy prediction performance (Fig. 4D). The trends in performance are consistent with or without noise, so the best model is trained with noise and MSAs. Using multiple sequences from the MSA during each training iteration is key: when a single random sequence is chosen for each protein for each iteration, performance does not improve (SI Appendix, Fig. S13).

Fig. 4.

Training on the CATH 4.2 dataset with an MSA loss function improves performance. (A) NSR for models trained with and without noise and with and without an MSA loss function. (B) TM-scores between native structures and AlphaFold2 structures predicted for sequences generated using each model. (C) Rosetta scores after modeling generated sequences on the native backbone. (D) Pearson r for using each model to predict the effect of single-site mutations on protein stability for three datasets. For (A - C), error bars show SEM over the proteins in the test set after averaging results over at least two retrained model replicates; for (D), error bars show SEM over at least two retrained model replicates. Stars indicate statistical significance comparing the model trained without MSAs to the model trained with MSAs in each condition, assessed using a two-tailed paired t-test over per-protein values for (A - C) and a two-tailed unpaired $t$ test over average Pearson r values for (D) (*P < 0.05, **P < 0.01, ***P < 0.001, ****P < 0.0001; n.s. indicates nonsignificance).

We compared PottsMPNN and ProteinMPNN trained on the larger PDB-clust dataset with and without noise and with and without MSAs. Both models improve when trained with noise and MSAs, and PottsMPNN performs better in both conditions (SI Appendix, Fig. S14). To investigate the difference between training on the CATH 4.2 and PDB-clust datasets, we compared PottsMPNN models trained on both datasets with and without noise and with and without MSAs. When evaluated on a subset of the PDB-clust test set made non-redundant with the CATH 4.2 train set, the PDB-clust models perform better on sequence-structure self-consistency and energy prediction (SI Appendix, Fig. S15).

We tested the performance of MSA-trained models in two other ways. First, we calculated TM-scores from AlphaFold3 structure predictions (5). These results confirm that PottsMPNN trained with noise and MSAs outperforms other PottsMPNN models and highlight the significant improvement over the standard ProteinMPNN model (i.e., trained with 0.2 Å of noise) (SI Appendix, Fig. S16). Second, we used ESM C, a large protein language model trained on native protein sequences (47–49), to evaluate the likelihood of sequences generated by several models, all trained on the CATH 4.2 or CATH 4.3 datasets. Of these, only PottsMPNN trained with noise and MSAs generated sequences with mean ESM C log probabilities comparable to those of native sequences (SI Appendix, Fig. S17).

Model interpretability.

We assessed the interpretability of the PottsMPNN Potts model trained with noise and MSAs. We examined how the energies change upon sequence mutations for two proteins randomly selected from the Megascale dataset. These case studies revealed biophysically interpretable self and pair energies. For example, PottsMPNN correctly predicts that the G56P mutation in 2KXD (located in a flexible loop) is highly stabilizing and that the R41P mutation in 2KXD (located in a rigid $\beta$-sheet) is highly destabilizing, and the predictions are driven by changes in the self energies at those positions (SI Appendix, Fig. S18A, D). PottsMPNN also made structurally reasonable pair-energy predictions: Q42K forms a destabilizing interaction with K35 and a stabilizing interaction with E37 (SI Appendix, Fig. S18B); K5Y forms a stabilizing interaction with K36 (SI Appendix, Fig. S19A); and F35K forms a destabilizing interaction with K30 (SI Appendix, Fig. S19D).

We further investigated the interpretability of the pair energies predicted by PottsMPNN. We calculated the Shannon entropy of the distribution of pair energies at all pairs of positions in the PDB-clust test set and found that pairs of positions only have low entropy when they are relatively close in the structure (SI Appendix, Fig. S20A). Among low entropy positions, native residues of opposite formal charge are scored favorably by the model relative to other possible pairs of residues at that position, while native residues of the same charge are scored unfavorably (SI Appendix, Fig. S20B). We examined how the pair energies of certain pairs of residues vary with distance and found biophysically reasonable trends, such as aspartate-lysine interactions having a shorter $\text{C}\alpha -\text{C}\alpha$ minimum than glutamate-lysine interactions and cysteine-cysteine interactions having very favorable energies at short distances (SI Appendix, Fig. S20C). We also found that PottsMPNN pair energies strongly correlate with experimental residue-residue coupling energies measured using double-mutant cycles in a model coiled-coil protein (50) (SI Appendix, Fig. S20D).

Finally, we tested the effect of training using MSAs on the learned single-site and pairwise amino acid probability distributions. We found that training with MSAs slightly but consistently improves the correlation between the entropy of the learned single-site probability distributions and the entropy of the associated site in the MSA (SI Appendix, Fig. S21A). The Jensen-Shannon divergence between the learned single-site probability distributions and the MSA single-site probability distributions also tended to decrease, indicating greater similarity, when the model was trained with an MSA (SI Appendix, Fig. S21B). The correlation between entropies of probability distributions at pairs of positions also improved upon training with MSAs, although the effect was less noticeable than for single sites (SI Appendix, Fig. S21C). Also, we used the learned Potts models to predict energies for sequences in the MSA and found that for models trained with and without MSAs, the energies increased as the sequences diverge from the native, but the model trained with MSAs predicted significantly lower energies for sequences across all distances from the native (SI Appendix, Fig. S21D).

Discussion

State-of-the-art sequence design models are trained on NSR, under the assumption that maximizing the similarity between a generated sequence and the native sequence also maximizes the likelihood that the generated sequence adopts the desired fold. Our results show that models trained on NSR—especially ProteinMPNN, Frame2Seq, and Caliby, which achieve NSR values that are low compared to other models—can achieve strong performance on sequence-structure self-consistency and show predictive capabilities on energy prediction tests. The high quality of Frame2Seq sequences is consistent with the finding of Akpinaroglu et al. that Frame2Seq can generate sequences with 0% NSR that fold to stable structures with secondary structures consistent with the input backbone structures (12). In general, however, our results demonstrate that optimizing solely for NSR is insufficient to drive further model improvement: compared to ProteinMPNN, Frame2Seq, and Caliby, models like MapDiff and UniIF achieve significantly higher NSR but perform poorly on the sequence-structure self-consistency and energy prediction tests (Fig. 2; SI Appendix, Table S1).

PottsMPNN achieves substantial improvements over ProteinMPNN trained under identical conditions: a 22.6% improvement in predicting Megascale mutation energies and an 8% improvement in AlphaFold2 TM-scores (Fig. 2B, D). When trained with noise and MSAs, PottsMPNN achieves a Pearson r of 0.639 for Megascale mutation energies, representing a 25% improvement over standard ProteinMPNN trained with 0.2 Å noise (Pearson r = 0.511) and a 7.8% improvement over the best non-PottsMPNN model (ESM-IF, Pearson r = 0.593) (SI Appendix, Table S1). This increased performance resulted from three strategies that we used to improve the training objective, motivated by better aligning the sequence design models with established principles of protein sequence-structure relationships.

First, learning a Potts model improved model performance, with especially large increases in energy prediction performance (Figs. 2; SI Appendix, Figs. S1–S3; SI Appendix, Table S1). The basis for our work on PottsMPNN is the COORDinator model developed by Li et al., which learns a Potts model and performs well on energy prediction tasks compared to contemporary models (11). We establish that the COORDinator framework can be improved by incorporating an additional loss based on single-site probabilities, ${ℒ}_V$, and using single-site probabilities for autoregressive sequence design, as is done by ProteinMPNN. PottsMPNN, a model trained with a joint objective, generates high-quality sequences as assessed by sequence-structure self-consistency. Sequence quality can be further improved by local optimization using the Potts model energies, indicating that Potts model energies are predictive of sequence quality in the local space around the initial sequence (SI Appendix, Fig. S4). The single-site probability distributions were unable to recognize the improved quality of the locally optimized sequences, further supporting the importance of the Potts model for improved energy prediction (SI Appendix, Fig. S4). Continuing the Potts model optimization until convergence did not improve sequence quality, indicating that the Potts model may not be as informative in the global sequence space (SI Appendix, Fig. S5).

Performing local optimization using the Potts model allows PottsMPNN to use the high-quality energetics information in the Potts model without resulting in low complexity sequences, as Li et al. observed when using MCMC to find low energy sequences in the Potts model sequence-energy landscape (11). We also observed low complexity sequences when using MCMC, even when using models trained with noise and MSAs. Another way to avoid low complexity sequences is to apply a complexity penalty term while sampling. Li et al., Ingraham et al., and Shuai et al. demonstrated this approach, but we found that even penalizing low-complexity sequences while sampling did not outperform autoregressive generation, as evaluated using AlphaFold2 TM-scores and Rosetta scores, and is much slower, even when autoregressive generation is followed by local energy optimization (11, 23, 51).

We postulated that the ability of PottsMPNN to optimize sequences and compute energies derives from the explicit residue-pair terms in the Potts model. Learning pairwise interactions allows PottsMPNN to capture physical constraints (e.g., electrostatic repulsion between proximal residues with the same charge) in a way that a model that only learns single-site probability distributions must learn implicitly through the message passing between sites that occurs in the decoder. This could allow PottsMPNN to capture more complex physical constraints. Although protein energetics involve many-body effects, approximating protein energetics using only self and pair energies has proven effective in many other studies (52, 53).

We performed experiments that withheld all or partial residue-pair information from the model during inference (SI Appendix, Fig. S7, S8, S10), and we compared models trained with a single-residue loss function to those trained with residue-pair loss functions (SI Appendix, Figs. S7, S9). In all cases, the performance of PottsMPNN degraded when the residue-pair information in the Potts model was limited. The near- and far-neighbor ablation series results suggest that model performance starts saturating when each residue has information about its nearest 24 or its furthest 32 of its 48 neighbors, which is consistent with previous observations on the effect of changing the number of nearest neighbors in the $k$-NN graph on the performance of ProteinMPNN (10).

Second, we found that adding noise improves model performance (Fig. 3A–C). This idea was introduced in the context of ProteinMPNN, and the most used version of ProteinMPNN was trained with 0.2 Å of noise. Dauparas et al. suggested that training with noise is beneficial because it may prevent the model from memorizing nongeneralizable artifacts of structural refinement (10). We found evidence consistent with this hypothesis by measuring trends in sequence-structure self-consistency and energy prediction over the duration of training with and without noise: training with noise prevents overoptimization to NSR (Fig. 3D–I).

Third, training using an MSA recovery objective instead of an NSR objective improves performance (Figs. 4; SI Appendix, Figs. S13–S17). Because many evolutionarily related proteins adopt closely related structures, we reasoned that including information about homologs would improve the ability of the model to capture meaningful sequence-structure relationships. MSA-based training rewards models that recognize that many sequences can adopt a given structure and prevents overoptimization to a native sequence that may not be optimal in any meaningful sense. We found that model performance decreases if the MSAs are filtered too stringently so that the sequences in the MSA do not provide diversity (SI Appendix, Fig. S12). We did not explore other ways to improve the use of information in the MSA, such as weighting sequences by their similarity to other sequences in the MSA to increase the effective sequence diversity seen by the model, so we expect that work in this area could result in further performance improvements. Similar to our work, InvMSAFold uses a Potts model parameterization and MSA-based training (54); however, our approach differs in that PottsMPNN jointly learns a Potts model and single-site distributions within a unified GNN architecture. Also, while InvMSAFold focused on improving sequence diversity and sampling speed, we show that MSA-based training improves sequence-structure self-consistency and energy prediction performance.

We trained models with and without noise and with and without MSAs on two different datasets, CATH 4.2 and PDB-clust, and observed consistent performance trends. CATH 4.2 is substantially smaller and has a more rigorous, structure-based train-test split, making it ideal for experimenting with different model architectures and optimization objectives. However, models trained on PDB-clust tend to perform better on shared held-out test sets (SI Appendix, Fig. S15). This is likely due to the increased structural diversity in the PDB-clust dataset. Because energy data are not used in training, limiting the possibility of data leakage, we recommend using large structure sets when training models for energy prediction. The energy data also show that models trained on both CATH 4.2 and PDB-clust generalize to de novo proteins: on the subset of the Megascale dataset consisting only of de novo designs, models trained with noise and MSAs on both datasets achieve a Pearson r of approximately 0.55.

In addition to the state-of-the-art sequence-structure compatibility and energy prediction performance, PottsMPNN produces energy decompositions that are biophysically interpretable. We used case studies of eight mutations in two structures to demonstrate that the Potts model energy predictions are structurally interpretable (SI Appendix, Figs. S18–19). The pair energies capture charge-charge interactions well and give a strong correlation with experimentally measured coiled-coil coupling energies (50) (SI Appendix, Fig. S20). Together, these results suggest that PottsMPNN has learned a physically meaningful model of sequence-structure compatibility rather than a superficial statistical mapping; this may partly explain its strong generalization across diverse sequence-structure self-consistency and energy prediction tests. We also found that training with MSAs makes the learned probability distributions and Potts model better match the variability observed in MSAs (SI Appendix, Fig. S21).

Learning residue-pair energies, introducing coordinate noise, and training on homologs of a target sequence do not improve NSR. Nevertheless, these modifications result in a model that is superior at the two tasks that sequence design models are most used for: generating a sequence that folds into a desired structure and predicting the energies of mutations given a structure. Thus, our results demonstrate that NSR should not be the primary metric of success for sequence design models.

There is substantial room for further progress on moving sequence design models away from optimizing for NSR. One approach is post-training alignment: several groups have demonstrated that pre-trained sequence design models can be fine-tuned on energy data (29, 30). These approaches are effective but operate as a second stage on top of a model that was initially trained on NSR; the improvements they achieve may be limited by the representations learned during pretraining. Our results suggest that improving the pre-training objective itself, through pairwise energy terms, noise, and MSA-based training, could provide a stronger foundation for such downstream alignment strategies. Additionally, the generalization of post-training alignment methods remains underexplored. Another way to improve the pre-training objective is to optimize in structure space instead of sequence space. Optimizing only in sequence space leaves the model unaware of potential off-target folding. DivPro incorporates a structure prediction model during training to guide a VAE-like latent sequence space toward structural fidelity, resulting in improved sequence diversity without compromising sequence-structure self-consistency (55). In an orthogonal approach, Pacesa et al. developed BindCraft (56) to directly optimize sequences iteratively according to a gradient derived from AlphaFold predictions. BindCraft generates sequences with high experimental success rates—after redesigning part of the generated sequence with ProteinMPNN and filtering for structure quality using AlphaFold2—indicating the promise of supervising sequence design models in structure space instead of sequence space. Accordingly, future work should explore training or fine-tuning a sequence design model directly on sequence-structure self-consistency or other desired tasks.

Methods

Potts model.

A Potts model is a function $H\left(s\right)$ that decomposes the sequence-energy landscape for a sequence $s$ of length $n$ into a sum of self energies and pair energies: $H\left(s\right)=\sum _{i=1}^n{h}_i\left(s\right[i\left]\right)+\sum _{i=1}^n\sum _{j>i}^n{J}_{ij}\left(s\right[i],s[j\left]\right)$, where $h=\left\{h_i\left(a\right)\mid 1\le i\le n,a\in 𝒜\right\}$ is a lookup table for self energies, $J=\left\{J_{ij}(a,b)\mid 1\le i<j\le n,a,b\in 𝒜\right\}$ is a lookup table for pair energies, and $𝒜$ is the set of all 20 amino acids. Several machine learning sequence design models have successfully parameterized a Potts model with learned weights $\theta$: they learn $H_{\theta }\left(s\mid f_{\mathit{\text{nat}}}\right)$ by maximizing $P\left(s_{\mathit{\text{nat}}}\mid H_{\theta }\left(s\mid f_{\mathit{\text{nat}}}\right)\right)=\mathtt{\text{softmax}}{\left[H_{\theta }\left(s\mid f_{\mathit{\text{nat}}}\right)\right]}_{s=s_{\mathit{\text{nat}}}}$ (see Loss functions and optimization for details) (11, 23, 51).

Model architecture.

We use the same GNN architecture and hyperparameters as ProteinMPNN (10). In brief, the protein backbone is encoded as a $k$-nearest neighbors ($k$-NN) graph, with $k=48$. Nodes $V$ in the graph are residues and are initialized with null vectors. Edges $E$ are interactions between residues. For each pair of residues, the interaction representation is initialized using radial basis functions to parameterize the interatomic distances for all 25 pairs of backbone heavy atoms, including virtual $\text{C}\beta$ atoms. The node and edge embeddings are then updated using a three-layer message passing neural network (MPNN) encoder. Finally, the node embeddings are updated using a three-layer MPNN autoregressive decoder that selects residues incrementally, conditioning on previously selected residues. The encoder and decoder use a hidden dimensionality of 128. The final edge embeddings (including self edges) are converted to the Potts model $H_{(\theta ,E)}(s\mid f)$ using a single linear layer. (Fig. 1)

Datasets.

To benchmark against other models, we used the dataset that Ingraham et al. curated from the CATH 4.2 database. We refer to this dataset as the CATH 4.2 dataset, and it consists of 19,700 single-chain structures split 80/10/10 on CATH protein structure classification codes (7, 42). We also trained on the larger set of structures that was used to train ProteinMPNN. This dataset was created by clustering chains from the PDB at 30% sequence identity, creating 25,361 clusters split 90/5/5 such that no chain in the training set is in a complex with chains in the validation or test clusters (10, 32). When using this dataset during training, following Dauparas et al., a new member from each cluster was randomly selected every two epochs. We refer to this dataset as the PDB-clust dataset. Because training on CATH 4.2 is much faster, we generally experimented with models trained on that dataset. We note when models are trained on the PDB-clust dataset.

MSAs for 140,000 unique protein chains were downloaded from the OpenFold OpenProteinSet database (57). These MSAs were generated using HHblits (-n3)(58) and JackHMMER (59) and searching against the MGnify (60), BFD (4), UniRef90 (61, 62), and UniClust30 (63) databases. For each biological assembly in the PDB-clust dataset, we create a paired MSA from the OpenFold chain MSAs following the AlphaFold-Multimer species-pairing methodology (64, 65). For each chain in the CATH 4.2 dataset, we generated MSAs using the ColabFold MSA server (66).

Because PottsMPNN relies on a graph representation of the protein backbone with a fixed number of nodes and edges, and because an assumption of the MSA approach is that $P\left(f_{\mathit{\text{nat}}}\mid s\right)$ is high for all $s\in MSA$, sequences in the MSA with gaps or insertions present a difficulty to the model. For a gap, we masked that position and all pairwise interactions that involve that position from affecting the loss calculation. We ignored insertions. We tested various filtering hyperparameters: the minimum sequence identity to the native sequence, the maximum gap percentage, and the maximum insertion percentage. The minimum sequence identity hyperparameter has the largest effect on performance, as models trained with a 70% minimum perform worse across all tasks compared to models trained with a 50% minimum and the same noise level (SI Appendix, Fig. S12). Changing the other hyperparameters did not have large effects, and while no set of hyperparameters resulted in the best performance across all tests, using a 50% sequence identity minimum, a 20% gap maximum, and a 20% insertion maximum resulted in strong performance across all tests (SI Appendix, Fig. S12). MSAs were only used during training.

Structural noise.

To add noise during training, we moved each backbone atom by independently sampling a displacement in all three axes from a Gaussian distribution with mean 0 Å and standard deviation $\sigma$ Å, as done by Dauparas et al. (10). We tested $\sigma \in [0,0.02,0.05,0.2,0.3,0.5]$. When training with noise, we apply noise only in training, not in validation. This creates a deliberately more challenging training task that improves model generalization.

Loss functions and optimization.

We trained PottsMPNN using an equally weighted composite of two loss functions. First, we supervised the nodes using the negative log likelihood loss used to train ProteinMPNN (10), shown in Equation 1:

$${ℒ}_v\left(s_{nat},f_{nat}\right)=-\frac{1}{2000}\sum _{i=1}^n\text{log}{P}_{\theta ,V}\left(s_{nat}\left[i\right]\mid f_{nat},s_{nat}\left[\prec i\right]\right),$$ [1]

where $n$ is the sequence length, $P_{\theta ,V}\left(s\left[i\right]\mid f_{\mathit{\text{nat}}},s_{nat}[\prec i]\right)$ is the model’s predicted probability distribution for all 20 amino acids at position $i$ conditioned on the structure $\left(f_{nat}\right)$ and the native residues prior in the decoding order $\left(s_{nat}\right[\prec i\left]\right)$, and $s_{nat}\left[i\right]$ is the native residue at position $i$.

Second, we supervised the edges using the negative log composite pseudo-likelihood loss used to train COORDinator (11), shown in Equation 2:

$${ℒ}_E\left(s_{nat},f_{nat}\right)=-\frac{1}{n_{\mathit{\text{edges}}}}\sum _{(i,j)\in E}\text{log}{P}_{\theta ,E}\left(s_{nat}\left[i\right],s_{nat}\left[j\right]\mid f_{nat},s_{nat\backslash \left(i,j\right)}\right),$$ [2]

where $n_{\mathit{\text{edges}}}$ is the number of edges in the graph, $i$ and $j$ are the indices of the residues connected by an edge, and $P_{\theta ,E}\left(s\left[i\right],s\left[j\right]\mid f_{\mathit{\text{nat}}},s_{\mathit{\text{nat}}\backslash (i,j)}\right)$ is the model’s predicted probability distribution for all 400 pairs of amino acids at the pair of positions $(i,j)$ conditioned on the structure $\left(f_{nat}\right)$ and the native sequence at all other sites $\left(s_{nat\backslash (i,j)}\right)$. This distribution is defined according the energies in the Potts model as in Equation 3:

$$P_{\theta ,E}\left(s[i],s[j]\mid f_{nat},s_{nat\backslash (i,j)}\right)=\mathtt{\text{softmax}}\left[h_i(s[i])+h_j(s[j])+J_{ij}(s[i],s[j])\left.+\sum _{\begin{array}{c}k\in \mathcal{N}(i)\\ k\ne j\end{array}}{J}_{ik}\left(s[i],s_{nat}[k]\right)+\sum _{\begin{array}{c}k\in \mathcal{N}(j)\\ k\ne i\end{array}}{J}_{jk}\left(s[j],s_{nat}[k]\right)\right],\right.$$ [3]

where $𝒩\left(i\right)$ denotes the set of all neighbors of the residue at site $i$. That is, the composite pseudo-likelihood loss rewards the model for assigning a high probability to the native pair of residues at each pair of positions in the sequence, given the native sequence at all other positions.

To train with MSAs, we used the loss function shown in Equation 4:

$${ℒ}_{\text{MSA}}\left(f_{nat}\right)=\frac{1}{\left|M_{\mathit{\text{batch}}}\right|}\sum _{s_M\in M_{\mathit{\text{batch}}}}{ℒ}_V\left(s_M,f_{\mathit{\text{nat}}}\right)+{ℒ}_E\left(s_M,f_{\mathit{\text{nat}}}\right),$$ [4]

where $M_{\mathit{\text{batch}}}$ is the set of sequences in the filtered MSA subsampled to fit in memory for the current training batch (i.e., such that the total number of residues does not exceed 10,000).

We trained a single-site version of PottsMPNN called PottsSingleMPNN that extracts single-site residue probabilities from the conditional distributions of the Potts model as shown in Equation 5:

$$P_{\theta ,E,ss}\left(s\left[i\right]\mid f_{nat},s_{nat\backslash i}\right)=\mathtt{\text{softmax}}\left[h_i(s[i])+\sum _{j\in 𝒩\left(i\right)}{J}_{ij}\left(s\left[i\right],s_{nat}\left[j\right]\right)\right].$$ [5]

That is, the conditional probability distribution at site $i$ is defined by summing the energies of each possible residue at site $i$ interacting with all its neighbors given the identities of those residues but not given the identity at site $i$. These probabilities were supervised using the negative log likelihood loss function shown in Equation 6:

$${ℒ}_{E,ss}\left(s_{nat},f_{nat}\right)=-\frac{1}{2000}\sum _{i=1}^n\text{log}{P}_{\theta ,E,ss}\left(s_{nat}\left[i\right]\mid f_{nat},s_{nat\backslash i}\right).$$ [6]

We used the same optimization hyperparameters used to originally train ProteinMPNN (10): an Adam optimizer with ${\beta }_1=0.9,{\beta }_2=0.98,ϵ={10}^{-9}$, the standard attention learning rate schedule with 4000 warm-up steps, and a dropout rate of 10%. We used a batch size of 10,000 tokens. When training without MSAs, we included as many different proteins as possible until the total number of sequence tokens exceeded 10,000. With MSAs, each batch consisted of a single protein, and we included as many randomly sampled sequences from the protein’s MSA as possible given the 10,000 token limit. For each model, we trained for 200 epochs, evaluated the validation loss after each epoch, and tested the model checkpoint with the best validation loss.

Sequence generation.

We used the autoregressive decoding process from ProteinMPNN (10) to iteratively generate a sequence from $P_{\theta ,V}(s\mid f)$. We used a fixed, randomly chosen decoding order for each protein to control for the effect of the decoding order on model performance by ensuring that every model variant and replicate uses the same order. We used the default ProteinMPNN temperature of 0.1. We experimented with optimizing the autoregressively generated sequences by searching the local space around $s_{gen}$. In the local optimization protocol, we iterate over the positions in sequence, evaluate the energy of all amino-acid substitutions at that position according to the Potts model (with the current identity at that position masked), and update the sequence with the best-scoring residue. We iterate in the same decoding order used to generate the sequence. We refer to sequences generated using local optimization with the Potts energies as “optimized.” We experimented with continuing to iterate over the sequence, in the same order used to generate the sequence, until the Potts model finds no favorable substitution at any position (i.e., until convergence). Finally, we experimented with scoring the substitutions using the single-site probabilities. See Energy prediction for details on how energies are calculated. To estimate the feasibility of generated sequences, we calculated sequence log probabilities using ESM C 300M, a state-of-the-art protein language model (47–49).

Sequence-structure self-consistency.

We mainly used two methods for evaluating $P\left(f_{\mathit{\text{nat}}}\mid s_{\mathit{\text{gen}}}\right)$. First, we used the monomer_ptm version of AlphaFold2 to predict a structure $f_{\mathit{\text{pred}}}$ from $s_{\mathit{\text{gen}}}$ (4). We ran AlphaFold2 without MSAs or templates. We then calculated the TM-score, a metric of structure similarity, between $f_{\mathit{\text{pred}}}$ and $f_{\mathit{\text{nat}}}$ (43). We also examined AlphaFold2 confidence using pLDDT. Second, we used Rosetta to evaluate the compatibility between $s_{\mathit{\text{gen}}}$ and $f_{\mathit{\text{nat}}}$. To do so, we threaded $s_{\mathit{\text{gen}}}$ onto $f_{\mathit{\text{nat}}}$ and relaxed using the FastRelax protocol with the standard full-atom energy function to minimize steric clashes and optimize side-chain packing (67–69). Following relaxation, the total all-atom energy was evaluated with the same scoring function. To attain the final Rosetta score, we normalize the energy by the protein length. For a few models, we also used AlphaFold3 to predict $f_{\mathit{\text{pred}}}$ and calculate TM-scores (5). For AlphaFold2 and AlphaFold3, we generated five structures for each sequence and scored the highest-confidence structure.

Because some models cannot predict residue identities for positions that lack structural information, for the benchmarking experiments, we removed structures with gaps, leaving around 700 structures. We refer to these structures as the CATH 4.2 test set benchmarking subset. For PDB-clust, to facilitate running AlphaFold2 and Rosetta, we limited the length of proteins to 500 residues, resulting in 702 single- and multi-chain protein structures.

Energy prediction.

To compute energies from single-site probabilities $P_{\theta ,V}(s\mid f)$— from PottsMPNN, ProteinMPNN, or a benchmark model—we compare the mutant and wild-type probabilities:

$$\Delta \Delta G_{\mathit{\text{pred,}}V}\left(s_{wt},s_{mut}\right)=\text{log}{P}_{\theta ,V}\left(s_{mut}\mid f\right)-\text{log}{P}_{\theta ,V}\left(s_{wt}\mid f\right).$$

To generate energy predictions from the Potts model, we compute the energy of a mutation as follows:

$$\Delta \Delta G_{\mathit{\text{pred}},E}\left(s_{wt},s_{mut}\right)=H_{\theta ,E}\left(s_{mut}\mid f\right)-H_{\theta ,E}\left(s_{wt}\mid f\right).$$

Unless otherwise specified, PottsMPNN energy predictions use the Potts model.

For the benchmarking comparisons, we used the scoring functions for each model as described in the literature to score the wildtype and mutant sequences. For example, most models predict single-site probabilities, so for those models we used the log probability approach described above. Because Caliby predicts a Potts model (23), we used the approach that we used for the PottsMPNN Potts models.

We evaluated energy prediction performance by calculating the Pearson correlation between predicted and observed energies for three datasets. First, we used 238,661 point mutations in the Megascale dataset, which contains stability energy measurements for 298 single-chain proteins (40). Second, we used 2,542 point mutations in the FireProt dataset, which contains stability energy measurements for 88 proteins curated from the literature; energies were measured using various methods (44). Third, we used a dataset of ~4,000 single mutants from a DMS screen over the SARS-CoV-2 receptor binding domain (41). Each mutation has expression data, which we used as a proxy for stability. We removed 200 mutations that affected binding but not stability according to thresholds used by Starr et al. During training, the models never saw any energy data, but out of an abundance of caution we took steps to prevent data leakage. For the Megascale and FireProt datasets, we removed any proteins that are in the CATH 4.2 and PDB-clust training sets, leaving 202,804 point mutations from 232 proteins for Megascale and 2,301 point mutations from 64 proteins for FireProt. For the SARS-CoV-2 dataset, the structure, 6M0J (70, 71), is in the PDB-clust training set and is not in the CATH 4.2 training set. To facilitate comparison across proteins, we centered both predicted and observed energies to zero mean for each protein before computing the Pearson correlation over all mutations in a dataset; this per-protein centering had little effect on PottsMPNN correlations but noticeably improved results for some other models.

Model interpretability.

All interpretability experiments were conducted with PottsMPNN trained with noise and MSAs on the PDB-clust dataset, except for the MSA interpretability experiments, which involved PottsMPNN trained with noise on the PDB-clust dataset with and without MSAs.

Using mutations from two randomly selected proteins from the Megascale dataset, 2KXD (45, 46) and the de novo designed protein EHEE_rd3_0053 (72), we performed case studies on the interpretability of the Potts model energies. For each protein, we selected the following mutations: the most favorable, the 33^rd percentile most favorable, the 66^th percentile most favorable, and the least favorable. For each mutation, we visualized the change in self energy, the total change in pair energy, and the five individual changes in pair energy with the largest magnitudes. We used PyMOL to view the structures, perform mutagenesis to structurally model the mutation, and manually select rotamers of the mutated and neighboring residues to minimize the PyMOL strain score (73).

We performed the pair energy interpretability analysis on the proteins in the PDB-clust test set. To account for variance in the scale of the energy values, we used z-scores to compare energy values across different positions. Also, we compared the Potts model pair energies to coupling energies, measured using double-mutant cycles, between residues at the $\mathbf{g}\leftrightarrow {\mathbf{e}}^{\prime }$ interface in the leucine zipper coiled coil (50).

For the MSA interpretability analyses, we computed single-site probability distributions for a given site as follows: for each sequence sampled from the MSA, we obtained the distribution output by the decoder when conditioned on the residue identities in the sampled sequence at all positions except the given site and averaged these distributions across all sampled sequences. We computed pairwise distributions for a given pair of sites analogously, obtaining for each sampled sequence the pseudolikelihood distribution calculated from the Potts model by conditioning on the residue identities at all positions except the given pair of sites and averaging across all sampled sequences. We filtered the MSAs to exclude sequences with more than 20% gaps or insertions. For the plots comparing predicted and observed distributions, we analyzed 50 proteins randomly selected from the CATH 4.2 test set with lengths under 100 residues. For the energy comparison plot, we used the entire CATH 4.2 test set.

Data, Materials, and Software Availability

The code for PottsMPNN is available at (DOI 10.5281/zenodo.18274667). The repository includes training and inference scripts, as well as Colab implementations of the sequence design, sequence optimization, and energy prediction tasks discussed in the paper. The repository also includes the experimental energy data used to evaluate the models, which are also available at their respective literature sources: Megascale (40), FireProt (44), and SARS-CoV-2 (41).