Protein Design Using Structure-Prediction Networks: AlphaFold and RoseTTAFold as Protein Structure Foundation Models

Abstract

Designing proteins with tailored structures and functions is a long-standing goal in bioengineering. Recently, deep learning advances have enabled protein structure prediction at near-experimental accuracy, which has catalyzed progress in protein design as well. We review recent studies that use structure-prediction neural networks to design proteins, via approaches such as activation maximization, inpainting, or denoising diffusion. These methods have led to major improvements over previous methods in wet-lab success rates for designing protein binders, metalloproteins, enzymes, and oligomeric assemblies. These results show that structure-prediction models are a powerful foundation for developing protein-design tools and suggest that continued improvement of their accuracy and generality will be key to unlocking the full potential of protein design.

Proteins are composed of linear chains of 20 possible amino acids, yet are able to perform an amazing array of natural functions such as harvesting energy from the sun (Nelson and Yocum 2006), producing chemicals for self-defense (Christianson 2017) and performing logical computations inside the cell (Antebi et al. 2017). Drawing upon these functions and molding them through directed evolution, protein engineers have developed countless useful medicines and catalysts to address human problems (Arnold 2018; Wang et al. 2021). However, natural proteins occupy only a small part of the combinatorial vastness of sequence space (Huang et al. 2016). Therefore, a long-standing goal in protein science is to design “de novo” proteins unrelated to any natural starting point, whose functions go beyond what has been explored by evolution (Kuhlman and Bradley 2019).

Traditionally, de novo protein design aimed to generate proteins that minimize an energy function based on physics and natural protein statistics (Alford et al. 2017). This approach has successfully created a variety of structural scaffolds and functional proteins but suffers from low success rates due to the inaccuracy of the energy function (Huang et al. 2016). Recently, neural networks trained on the Protein Data Bank (PDB) have made it possible to predict protein structure from sequence at near-experimental accuracy (Baek et al. 2021; Jumper et al. 2021). This was not possible with energy-function-based methods, which raised the possibility that structure-prediction networks could advance the state-of-the-art in protein design as well.

Here, we review recent studies where models such as AlphaFold2 (AF) and RoseTTAFold (RF) are used directly or fine-tuned to design proteins. In particular, the approach of fine-tuning a structure-prediction network has led to one of the most powerful current methods for de novo protein design (Watson et al. 2023). This suggests that structure-prediction models may serve as “foundation models” for protein-structure-centric tasks, analogous to large language models for natural language understanding and generation (Bommasani et al. 2021; OpenAI 2023). Where relevant, we will mention machine-learning protein design methods that do not explicitly train on structure prediction, but we defer to other sources for comprehensive reviews of traditional or machine-learning-based protein design (Huang et al. 2016; Kuhlman and Bradley 2019; Gao et al. 2020; Ding et al. 2022; Strokach and Kim 2022; Ferruz et al. 2023; Winnifrith et al. 2023; Yang 2023).

SINGLE-SEQUENCE STRUCTURE PREDICTION

To achieve high accuracy, AF and RF require not only an input sequence but also a multiple-sequence alignment (MSA) of homologs for inferring evolutionary covariation between residue pairs (Fig. 1B). Covarying residues tend to be near each other in 3D space. This knowledge drastically reduces the search space for structure prediction (Marks et al. 2012). Accordingly, AF and RF perform poorly when predicting natural protein structures from only a single input sequence (Dauparas et al. 2022; Wang et al. 2022a). Recent methods based on protein language models such as ESMFold (Lin et al. 2023), OmegaFold (Wu et al. 2022), and others (Chowdhury et al. 2022; Wang et al. 2022b) do not require an alignment at prediction time, but their accuracy on natural proteins is nevertheless correlated to the number of homologs in the training set (Lin et al. 2023). This suggests that evolutionary covariation information has been “stored” in the parameters of the language model. There are no current methods capable of predicting natural protein structures to high accuracy from single sequences without using some form of homolog information. Achieving this ability would likely require advances in incorporating physical principles into structure prediction and is an important open research problem.

Figure 1.

Schematic of de novo protein design methods. (A) Typical pipeline for de novo protein design. Initially, the design problem is specified by choosing, for example, a desired set of secondary structures (green), a motif to be scaffolded (orange), or a protein (blue) or ligand (purple) target to bind. Then, step 1 is to generate a protein backbone that is compatible with the design goals. Step 2 is to design a sequence that is predicted to fold into the backbone (sequence and sidechains shown in red). Step 3 is to verify that the designed sequence indeed folds into the desired backbone, using an independent structure-prediction method (predicted structure shown in red). Finally, designs are selected based on various in silico metrics and tested in the wet lab. (B) Input and outputs of structure-prediction networks such as AlphaFold and RoseTTAFold. Random residues in the input multiple-sequence alignment (MSA) are masked (replaced with a special token) and re-predicted during training, thereby enabling sequence information to be output by the network for fine-tuning tasks. Structures of “template” proteins homologous to the query sequence are used to aid structure prediction, thereby allowing structure information to be input during fine-tuning. (C–E) Protein design methods based on structure-prediction networks: (C) hallucination, (D) inpainting with RF_joint, (E) structure diffusion with RF_diffusion, and (F) sequence diffusion with ProteinGenerator. An example of a motif-scaffolding task is shown, where green represents a prespecified structural motif and gray represents what is being generated by the design method. In principle, hallucination and diffusion are inherently iterative while RF_joint inpainting can perform one-shot design. However, iteration or “recycling” of outputs is also needed to obtain good design in practice (gray dotted arrow in D). In F, sequence is represented schematically as a matrix of logits, where the region to be generated is initialized to random real-numbered values and denoised until it becomes 1-hot.

It is surprising then that AF, RF, ESMFold, and other models can predict the structures of designed proteins—proteins unrelated to natural sequences, generated by an energy-function or machine-learning-based method—with very high accuracy from only a single sequence (Yang et al. 2020; Wang et al. 2022a; Watson et al. 2023). This is fortunate, as these proteins lack evolutionary homologs by construction. This empirical observation is the original motivation (and justification) for using structure-prediction networks in the protein-design methods described below. It is not clear why these networks have this ability, but one possible explanation comes from the observation that natural proteins are marginally stable, or only a few mutations away from misfolding (Dill 1990; Thomas et al. 2010; Goldenzweig and Fleishman 2018). Therefore, natural proteins probably do not have the lowest-energy (or most probable) sequence for their structures, obscuring the sequence–structure relationship and necessitating homolog information to increase “signal-to-noise” during structure prediction. On the other hand, de novo protein sequences have been designed to have the most probable amino acids for their structure and a large energy gap between unfolded and folded states (Fleishman and Baker 2012; Baker 2019), making their structure prediction easier.

STRUCTURE PREDICTION AS DESIGN FILTER

Given that structure-prediction networks can accurately predict de novo proteins, the simplest way to use them in protein design is for in silico validation. The pipeline for structure-based protein design typically follows three steps (Fig. 1A; Huang et al. 2016). First, a protein backbone is designed that is thought to be capable of carrying out a particular function, such as scaffolding a functional motif or forming a binding interface. Second, a sequence is generated that is designed to fold to the desired structure, and therefore carry out the desired function. This second step is known as “fixed backbone sequence design,” “inverse folding,” or simply “sequence design.” Finally, the designed sequence is input to an independent structure-prediction calculation to assess whether it will fold—and therefore function—as intended. This step, called “forward folding,” is required because methods for sequence design aim only to find the lowest-energy sequence for a given structure, which does not guarantee that there is not another structure for which the chosen sequence is even lower energy (Norn et al. 2021). In other words, once a sequence has been designed, it is necessary to predict whether it will fold to the designed backbone structure. Forward folding was traditionally done using energy-function methods (Shortle et al. 1998), which are substantially less accurate than deep learning. Replacing these low-accuracy methods with AF structure prediction has provided, for example, a 10-fold increase in experimental success rates for de novo binder design (Bennett et al. 2023). Similar increases in success rate have been observed in most other design problems to which AF filtering has been applied.

INVERTING STRUCTURE-PREDICTION NETWORKS

Traditional protein design tools make heavy use of the Markov Chain Monte Carlo (MCMC) algorithm: Starting from an initial structure and/or sequence, a small change is proposed by a sampling method and then accepted or rejected based on its energy according to an energy function (Kuhlman and Baker 2000). This is repeated for thousands of iterations until a low-energy protein is found. In an analogous approach called “hallucination,” a structure-prediction network is used instead of an energy function to evaluate incremental designs (Fig. 1C). This is similar to the “activation maximization” method in machine learning (Erhan et al. 2009) (but unrelated to the “hallucination problem” of language models generating nonfactual outputs).

Many different hallucination methods have been developed, but they all share the same basic principle (Anishchenko et al. 2021; Jendrusch et al. 2021; Moffat et al. 2021; Norn et al. 2021; Hie et al. 2022; Verkuil et al. 2022; Wang et al. 2022a; Wicky et al. 2022; An et al. 2023a; Frank et al. 2023; Goverde et al. 2023; Jeliazkov et al. 2023). A random starting sequence is given to a structure-prediction network to predict an initial, usually poorly folded, structure. Using this structure, as well as other outputs from the network such as prediction confidence, a user-defined “loss” function (analogous to the traditional energy function) is computed that represents the design objective. Subsequently, the sequence is mutated, given to the network again to predict a new structure, and the loss function recomputed. The new loss value is compared to the previous loss and some rule (typically the Metropolis–Hastings condition) is used to decide whether to “accept” or “reject” the mutated sequence. This process is repeated until the loss has converged, and the final sequence and its predicted structure are taken as outputs.

Hallucination's biggest advantage is simplicity and flexibility: novel tasks can be addressed by formulating new loss functions on top of otherwise unmodified structure-prediction models (Hie et al. 2022; Wang et al. 2022a). This can be a practical benefit for users who do not have the resources or expertise to train models. Early hallucination studies used a simple loss function based on model confidence—for example, the predicted error from the network (Jendrusch et al. 2021) or the entropy of predicted distributions of residue–residue distances (Anishchenko et al. 2021). This type of loss can be seen as promoting “free hallucination” of random but well-structured proteins, also known as “unconditional structure generation.” Early studies demonstrated this idea in silico (Jendrusch et al. 2021), and Anishchenko et al. (2021) showed in wet-lab experiments that free hallucinations with diverse secondary structures fold to their designed structures (Fig. 2B). Elaborating on this idea, Wicky et al. (2022) performed free hallucination with a chain break input feature to create assemblies of multiple protein chains (i.e., oligomers), which could be homo- or hetero-oligomeric (composed of identical or different subunits) depending on whether different chains were constrained to have the same sequences. Adding a cyclic symmetry loss term allowed the design of C33 assemblies with up to 1550 total residues (Fig. 2F). Using symmetry losses, sequence repetition, but no chain break, An et al. (2023a) hallucinated pseudo symmetric proteins (i.e., a single chain consisting of identical repeating units) with pockets for ligand binding or catalysis (Fig. 2D). Both studies experimentally tested their designs and found them to fold to the intended structures. These design problems would have been extremely difficult to address using traditional methods.

Figure 2.

Experimentally validated proteins designed by structure-prediction networks. In all panels, experimentally determined structures (crystal or cryo-EM) are shown in gray, design models are shown in blue or other colors, and AlphaFold2 (AF) models are shown in pink. (A) A fixed-backbone sequence design to the Top7 protein backbone using AF hallucination (Goverde et al. 2023). (B) Unconditionally generated monomer from Markov Chain Monte Carlo (MCMC) trRosetta hallucination (Anishchenko et al. 2021). (C,D) Repeat protein monomers (An et al. 2023a; Lisanza et al. 2023). (C) Repeat protein with secondary structure (all-β) conditioning from ProteinGenerator sequence diffusion (An et al. 2023a). (D) Repeat cyclic protein with pocket for potential ligand binding, from AF hallucination (An et al. 2023a). (E–G) Homo- and hetero-oligomers. (B) Hetero-oligomer from continuous-sequence AF hallucination (Frank et al. 2023). (F) Large C33-symmetric oligomer from MCMC AF hallucination (Wicky et al. 2022). (G) Icosahedral cage from RF_diffusion structure diffusion (Watson et al. 2023). (H–J) Motif-scaffolding. (H) RF hallucination presenting site V epitope from respiratory syncytial virus (Wang et al. 2022a). (I) C4-symmetric oligomer scaffolding a Ni²⁺-binding motif from RF_diffusion (Watson et al. 2023). (J) Di-iron binding protein from RF_joint inpainting, scaffolding iron-coordinating motif (Wang et al. 2022a). (K) Protein binder against influenza H1 hemagglutinin from RF_diffusion (Watson et al. 2023). (L) De novo luciferase, from fold-constrained trRosetta hallucination and Rosetta (Yeh et al. 2023). (M) De novo heme protein, from RF_diffusion all-atom (Krishna et al. 2023).

Although free hallucination can create useful building block proteins, practical applications usually require directly specifying the structure (or function) of a protein. To achieve this, a “structure recapitulation” loss can be defined as the deviation between the predicted structure and a prespecified structure, either in terms of 3D coordinates (e.g., frame-aligned point error or FAPE; Goverde et al. 2023) or residue–residue distance and orientation distributions (e.g., cross-entropy; Norn et al. 2021; Wang et al. 2022a). When this loss is applied over an entire protein, hallucination becomes a method for fixed-backbone sequence design (Fig. 2A; Norn et al. 2021; Verkuil et al. 2022; Goverde et al. 2023). If this loss is combined with a confidence-based loss—i.e., freely hallucinating part of a protein but constraining another part to recapitulate a predefined structural motif—then hallucination can create de novo proteins that scaffold functional motifs such as binding interfaces and catalytic sites (Tischer et al. 2020). Using this approach, Wang et al. (2022a) designed and experimentally validated metal- and protein-binding proteins, including scaffolding an epitope from the respiratory syncytial virus (Fig. 2H) that could not be tackled with previous methods. The method also created promising de novo enzymes in silico by scaffolding catalytic sites. However, these were not tested experimentally because AF and RF cannot model small molecules and therefore cannot hallucinate substrate-binding pockets (see below). To overcome this, Yeh et al. (2023) used hallucination to design a library of pocket-containing proteins constrained to the natural NTF2 fold family. Then they used Rosetta, an energy-function-based method, to install de novo luciferase active sites into the hallucinated scaffolds and design pocket residues for binding nonnatural luciferin analogs (Fig. 2L). This pipeline, combined with three rounds of directed evolution, resulted in de novo luciferases with near-native activity. Key to success was the ability of hallucination to create pocket geometries better suited to the nonnatural substrates than those of native proteins from the PDB.

In principle, hallucination can jointly design sequence and structure (i.e., perform backbone and sequence design steps simultaneously) (Anishchenko et al. 2021; Wang et al. 2022a). Unfortunately, sequences from standard hallucination are poorly expressed and insoluble (Dauparas et al. 2022; Verkuil et al. 2022; Wicky et al. 2022). This is likely due to the phenomenon of adversarial examples, where applying activation maximization to deep neural networks leads to sampling from outside the training distribution (Goodfellow et al. 2015; Stern et al. 2023). The most practical solution, which dramatically increases wet-lab success rates, is to input hallucinated backbones to a dedicated sequence design method to generate new sequences. Traditionally, sequence design was done using Rosetta; however, recently, the ProteinMPNN neural network, which is trained to predict the amino acid at each residue position given the local structural context, has become a state-of-the-art widely adopted method for this step (Dauparas et al. 2022; Wicky et al. 2022; An et al. 2023a; Frank et al. 2023; Glasscock et al. 2023; Goverde et al. 2023; Kim et al. 2023; Krishna et al. 2023; Sumida et al. 2023; Torres et al. 2023; Yeh et al. 2023). However, because joint sequence-structure design has the potential to improve overall design quality, various strategies have been pursued to improve hallucinated sequences. One idea is to simply add loss terms to control the amino acid composition and properties such as solvent-exposed hydrophobic residues (a cause of poor expression), but this has only modest benefits (Anishchenko et al. 2021; Wang et al. 2022a; Goverde et al. 2023). A more principled strategy is to constrain sequence updates by another model that has learned the distribution of plausible sequences (Moffat et al. 2021; Verkuil et al. 2022; Jeliazkov et al. 2023). One example used the ESM-2 protein language model to propose MCMC mutations and achieved protein expression success rates qualitatively similar to redesign by ProteinMPNN (Verkuil et al. 2022). However, this method is impractical, requiring more than 100,000 MCMC steps per trajectory. Given the above, hallucination is most useful as a backbone-generation tool and hallucinated sequences are usually discarded.

The biggest practical limitation of conventional hallucination is its high computational cost, often requiring minutes to hours of GPU time per design and sampling of 10³-10⁴ designs to obtain experimental success (Anishchenko et al. 2021; Wang et al. 2022a). This is partly because MCMC requires many steps to converge, and partly because discrete amino acids form a rugged, difficult-to-optimize loss landscape. To avoid MCMC, one can take advantage of the differentiability of structure-prediction networks and use backpropagation to calculate the derivative of the loss function with respect to the input sequence (Norn et al. 2021; Wang et al. 2022a; Frank et al. 2023). This allows mutational steps to be made in the direction of most improved loss, greatly accelerating convergence. However, this comes at the cost of increased GPU memory usage and leads to worse losses than MCMC, likely due to the need for gradient approximations when backpropagating through discrete amino acids (Anishchenko et al. 2021; Linder and Seelig 2021; Wang et al. 2022a). To address this last issue, Frank et al. (2023) hallucinated proteins with a continuous (real-valued) sequence representation, completely dispensing with the goal of generating a usable sequence. This avoids the need to approximate gradients and the resulting loss landscape is smoother, leading to faster convergence. This also allows structures to be realized more precisely, improving the success rate and reducing the number of trajectories needed. Using this approach for backbone generation and ProteinMPNN for sequence design, the authors achieved the highest in silico design quality per compute time out of all current hallucination methods, in some cases rivaling more sophisticated generative models (see below). They created a number of oligomeric assemblies that successfully expressed in the wet lab and validated the structures of a subset of designs by cryo-EM (Fig. 2E).

STRUCTURE PREDICTION AS PRETRAINING

The most direct solution to the drawbacks of hallucination is to train a model explicitly for protein design. For fixed-backbone sequence design, a number of effective methods already exist (Alford et al. 2017), with performance improving even more as modern machine learning has been applied to the problem (Dauparas et al. 2022). By contrast, most early neural networks for backbone generation gave poor results (Anand and Huang 2018; Anand et al. 2019; Guo et al. 2020; Lin et al. 2021; Harteveld et al. 2022; Lai et al. 2022) unless the sampling space was constrained to a specific protein fold (Eguchi et al. 2020). As these networks typically used convolutional layers and were trained as variational autoencoders (VAEs) or generative adversarial networks (GANs), they probably lacked the inductive bias and expressive capacity to fully model the distribution of high-quality protein structures.

AF and RF showed that equivariant, transformer-based models excel at reasoning about protein structures, inspiring a new wave of generative methods. Wang et al. (2022a) noticed that RF itself could represent arbitrary design tasks because it already had sequence and structure as both inputs and outputs (Fig. 1B). This is because, in addition to the usual sequence input and structure output used for structure prediction, the network also receives homology template structures as input and outputs sequence logits to perform auxiliary masked-token prediction. Therefore, motif scaffolding or unconditional structure generation could be performed simply by masking out portions of the input sequence and structure and asking the network to “inpaint” or predict the missing regions (Fig. 1D). (For simplicity, we use “inpainting” even when the majority of the input is masked, which is sometimes called “outpainting.”) By starting with RF weights pretrained for structure prediction and fine-tuning them on inpainting and sequence design, Wang et al. obtained a model, RF_joint, that could design metalloproteins (Fig. 2J) and protein binders with higher wet-lab success rates than hallucination. Success rates increased even further as subsequent versions of RF_joint were fine-tuned on top of improved variants of the base structure-prediction model.

Compared to hallucination, inpainting scaffolds motifs more accurately and requires less compute at design time (Wang et al. 2022a; Watson et al. 2023). This comes at the cost of an initial fine-tuning run, but it becomes negligible when amortized over all downstream design work. Like hallucination, inpainting can in theory design sequence and structure jointly, and, in fact, inpainted sequences have reasonably high in silico and wet-lab success rates (Wang et al. 2022a). However, success rates are still higher when sequences are redesigned using ProteinMPNN (Sumida et al. 2023; Watson et al. 2023). Inpainting excels at “idealizing” local backbone structure, or replacing loopy regions with α helices or β sheets (Glasscock et al. 2023; Sumida et al. 2023; Torres et al. 2023). This is a common “polishing” step applied to designs generated by any method, because it improves the quality of sequences from ProteinMPNN and wet-lab success rates. On the other hand, inpainting's weaknesses are low design diversity and poor performance on unconditional generation of full structures. This is not surprising given that the training task defines one specific solution, rather than a distribution of solutions, for any given input. A better formulation would pose protein design as a probabilistic modeling problem, but traditional approaches like VAEs and GANs have shown limited success in generating proteins with high quality and diversity.

COMBINING STRUCTURE PREDICTION AND DIFFUSION GENERATIVE MODELS

Recently, denoising diffusion probabilistic models (DDPMs or “diffusion models”) (Sohl-Dickstein et al. 2015; Ho et al. 2020; Song and Ermon 2020; Yang et al. 2023) have achieved state-of-the-art in image generation (Dhariwal and Nichol 2021; Ramesh et al. 2022; Rombach et al. 2022). These models are trained to take a data sample to which noise has been added and predict either the noise or the original un-noised sample. At inference time, the model is used to transform a sample of pure noise into an example from the data distribution by many steps of iterative denoising. In the limit of infinitesimally small step sizes, a correctly trained diffusion model can theoretically sample from the true data distribution. This framework is also highly amenable to adding conditioning information to control properties of the generated samples (Ramesh et al. 2022; Rombach et al. 2022).

For proteins, the key challenge is to represent structure and sequence as quantities that can be diffused. To date, the majority of protein diffusion models have focused on diffusing protein structure (Anand and Achim 2022; Trippe et al. 2022; Wu et al. 2022; Chu et al. 2023; Ingraham et al. 2023; Lin and AlQruaishi 2023; Watson et al. 2023; Yim et al. 2023). Protein structures are inherently three dimensional, and their biological function is determined by the position of atoms relative to one another, rather than their absolute coordinates in some arbitrary reference frame. As such, protein structure diffusion models are all SE(3)-equivariant, and the majority have reused architectural components directly from structure-prediction networks. The challenges in formulating the diffusion process in three dimensions has commonly led to practical compromises, such as suboptimal protein representations or unprincipled “diffusion-like” processes that are not theoretically guaranteed to learn the true data distribution.

ProtDiff, one of the earliest protein diffusion models, simplified the problem of backbone generation to C-α coordinate denoising, which allowed a principled formulation of diffusion with a mean-squared error loss (Trippe et al. 2022). However, the simplistic backbone representation and an architecture invariant to reflections led to incorrect chirality in some of the designed backbones. FoldingDiff performed diffusion over backbone torsion angles, but this representation is known to be vulnerable to error propagation through lever-arm effects (Wu et al. 2022). Anand and Achim (2022) trained a protein diffusion model that adopted the frame-based representation of protein backbones used in AF and RF (treating backbones as elements of the SE(3)^N Lie group). Like ProtDiff, they also diffused C-α atoms, but further used spherical interpolation to define a diffusion-like process over residue backbone orientations. This permitted the use of AlphaFold's FAPE loss to jointly train denoising of the backbone position and orientation, enforcing the chirality of natural proteins. This model generated quite realistic protein structures when the samples were conditioned on fold information (secondary structure blocks and their relative adjacency), but unconditional sampling was not explored. The above studies demonstrated the promise of diffusion for protein design, but used relatively small neural networks and did not pursue wet-lab testing of designs.

Currently, the state of the art for protein diffusion models (and perhaps de novo protein design in general) is RF_diffusion, a fine-tuned version of the RF structure-prediction network (Fig. 1E). Watson et al. (2023) noticed that inpainting using RF_joint was almost identical to one step of denoising, and therefore proposed that structure-prediction weights could serve as the foundation for a diffusion model. Diffusion fine-tuning was done with noised inputs, an input feature representing the denoising step (to specify the expected level of noise), mean-squared error loss on coordinates as well as frame orientations, and self-conditioning, where the network is given its own outputs from a previous step. To perform specific design tasks such as motif scaffolding or specifying the overall fold, additional fine-tuning was performed with task-specific conditioning inputs. The resulting RF_diffusion model proved to be extremely capable on a wide variety of protein design tasks. Simply by adding a symmetrization step to inference and leveraging the SE(3) equivariance of the RF architecture, RF_diffusion was able to generate large, homo-oligomeric assemblies with cyclic and point group symmetry that adopted the intended quaternary structure by cryo-EM (Fig. 2G). Additionally, RF_diffusion was able to scaffold symmetric motifs (Fig. 2I) and very small motifs such as single residues from an enzyme active site that were almost impossible with hallucination or RF_joint. Finally, the model designed protein binders conditioned only on the target structure with wet-lab success rates of up to 50% on some targets (Fig. 2K). For comparison, this is around an order of magnitude higher success rate than binder design with energy-based methods (combined with improved AF-based filtering) (Bennett et al. 2023; Watson et al. 2023). For two different peptide-binder design problems, one out of 96 RF_diffusion designs tested experimentally had picomolar affinity for the target without any experimental optimization (Torres et al. 2023). These are the highest-affinity de novo protein binders ever designed.

Alongside and after RF_diffusion, several other general-purpose protein diffusion models have been reported (as well as application-specific diffusion models such as for antibody design (Luo et al. 2022), which will not be detailed here. Chroma, a model based on a “polymer diffusion” approach, is trained only unconditionally, relying on classifier-based guidance at inference time to condition on design objectives such as functional motifs, text descriptions, and arbitrary 3D shapes (Ingraham et al. 2023). Genie uses C-α diffusion and an AF-like architecture to perform unconditional generation (Lin and AlQuraishi 2023). FrameDiff uses a similar frame representation as RF_diffusion but within a simpler, lighter-weight network architecture and a more-exact score-matching objective (Yim et al. 2023). Finally, Protpardelle diffuses coordinates of all atoms corresponding to all possible sidechains at each residue, and then uses these “superposed atom” positions to predict amino acid identities (Chu et al. 2023). This avoids the complexity of residue orientation diffusion and provides a framework for joint sequence-structure design. All of these models are trained “from scratch” rather than fine-tuned from a structure-prediction network. Currently, however, in silico performance of such models lags behind RF_diffusion, and only one has reported results from wet-lab testing.

Interestingly, RF_joint or RF_diffusion performance is substantially worse when protein design fine-tuning (i.e., a relatively short training run) is performed from randomly initialized weights instead of pretrained structure-prediction weights (Wang et al. 2022a; Watson et al. 2023). Given this, it is tempting to conclude that something special about structure-prediction pretraining makes it necessary for generating high-quality designs. However, since RF_diffusion fine-tuning typically uses <2% of the GPU time used to train RF structure prediction, it is not surprising that omitting the pretraining step reduces design quality (a full RF_diffusion training run “from scratch” with the same amount of compute as structure-prediction pretraining was not attempted). Similarly, the lower performance of other diffusion models could simply be due to lower model capacity. For example, FrameDiff, which has a similar diffusion scheme as RF_diffusion, only has 1/4 as many parameters, yet already has reasonable performance (Yim et al. 2023). There is no obvious reason in principle why the performance of RF_diffusion cannot be matched or exceeded by training on protein design from scratch, given sufficient model capacity and training time. However, the fact that structure-prediction pretraining provides such a strong starting point for design has important practical benefits, allowing rapid experimentation with different fine-tuning strategies at relatively low computational expense.

An interesting alternative to protein structure diffusion is to diffuse protein sequences instead. Diffusion of discrete variables (such as amino acids) currently has worse performance than autoregressive or masked-token models for natural language modeling (Austin et al. 2021; Hoogeboom et al. 2021; Dieleman et al. 2022). However, for proteins, sequence diffusion can be much simpler than structure diffusion, and a wealth of protein sequence-function data (binding or enzymatic activity) exists that can potentially be used to train oracles to perform classifier guidance on sequences. This is the motivation behind ProteinGenerator, a diffusion model fine-tuned from RF that performs diffusion in sequence space (Fig. 1F; Lisanza et al. 2023). Amino acids are represented as one-hot encodings which have Gaussian noise added, and the network is trained to denoise them via a cross-entropy loss. The model was used to design proteins conditioned on sequence-centric design objectives such as enrichment for certain amino acids, masked bioactive peptide scaffolding, and repeat protein design (Fig. 2C). Sequences directly from ProteinGenerator (without additional sequence design) expressed well and had the expected secondary structure content in the wet lab. Because ProteinGenerator is based on RF, it can also predict structure (which was maintained during fine-tuning by continuing to apply a FAPE loss), and therefore can perform structure-conditioned tasks such as motif scaffolding and multistate design. However, overall, in silico metrics and wet-lab success rates were lower than in the RF_diffusion-ProteinMPNN pipeline. Other sequence diffusion models (Alamdari et al. 2023) have also recently been reported. The true potential of this approach will be realized when it can design functional proteins guided by a sequence-to-function oracle.

PROTEIN DESIGN IN THE CONTEXT OF NONPROTEIN MOLECULES

Many important protein-design problems require modeling the interactions between proteins and DNA, RNA, and small molecules (Tinberg et al. 2013; Huang et al. 2016; Dou et al. 2018; Glasscock et al. 2023; Kalvet et al. 2023). Classical methods handle these problems naturally because their force fields are defined on explicitly represented atoms (Alford et al. 2017). However, these methods do not have the ability of hallucination and diffusion to generate full proteins from scratch, and so require complex, multistep design pipelines with an initial input of a library of predefined protein “scaffold” structures for docking against the intended binding target (Tinberg et al. 2013; Dou et al. 2018; An et al. 2023b; Lee et al. 2023; Lu et al. 2023). Machine-learning-based design methods are simpler and more flexible, but AF and RF cannot model nonprotein molecules. Therefore, until recently, protein-design tools based on these models could only design protein monomers and oligomers, with any interactions to nonprotein molecules being modeled in a separate step by classical methods (Glasscock et al. 2023; Kalvet et al. 2023; Yeh et al. 2023) or machine learning (Dauparas et al. 2023). This means that overall protein shape (and therefore the binding pocket) cannot be defined in a ligand- or nucleic-acid-aware manner, which limits the degree of achievable shape and chemical complementarity between the protein and the nonprotein molecule (Wang et al. 2022a; An et al. 2023a; Watson et al. 2023). This is likely a key reason why existing methods have struggled to obtain high binding affinity “out of the computer,” before doing any experimental optimization via directed evolution (Tinberg et al. 2013; Dou et al. 2018; Yeh et al. 2023).

Recently, RF was generalized to model protein-nucleic-acid complexes (RF nucleic acid [RFNA]; Baek et al. 2022) as well as protein-small-molecule complexes and covalently modified amino acids (RF all-atom [RFAA]; Krishna et al. 2023). The latter model, RFAA, can model arbitrary biomolecular assemblies and has been trained on all protein and nonprotein structures in the PDB. Unlike traditional docking methods that take as input a known protein and ligand structure and predict how they will bind as a complex (Trott and Olson 2010; Corso et al. 2023), RFAA predicts the structure of a protein along with any binding partners, effectively performing folding and docking simultaneously. For structure modeling, this potentially allows the prediction of binding-induced conformational changes. For design, this potentially enables designing proteins with highly tailored binding interfaces to small molecules and nucleic acids.

To test this idea, Krishna et al. fine-tuned their RFAA model to create RF_diffusion AA (Krishna et al. 2023), a diffusion model that can generate proteins conditioned on small molecule ligands (Fig. 1A, purple input). This was used to design protein binders for a number of biological and nonbiological small molecules, and designs against three targets­—digoxigenin, heme (Fig. 2M), and bilin—were tested and confirmed to have binding activity in wet-lab experiments. Notably, one of the tested digoxigenin binders had a K_d of 10 nM, comparable to a previously designed binder that had been optimized by directed evolution to have three mutations relative to its original design (Tinberg et al. 2013). This demonstrates that the “folding and docking” capability of RFAA translates, after fine-tuning, to an ability to generate binding pockets highly tailored to specific binding partners, thus eliminating (and improving on) the traditional step of docking the target against a library of scaffolds. These results open up several directions for future work. DNA- (or RNA)-binding proteins could be designed by generating proteins in the presence of a nucleic acid, potentially improving on similar recent efforts using more traditional tools (Glasscock et al. 2023). Enzymes could be designed by combining ligand-conditioned protein generation with motif scaffolding of catalytic residues (Huang et al. 2016; Yeh et al. 2023). These problems would pose a number of additional scientific challenges, but the key technical obstacle of generating novel proteins with highly complementary binding pockets now seems to have been overcome.

AlphaFold AND RoseTTAFold AS PROTEIN STRUCTURE FOUNDATION MODELS

In the domains of natural language and computer vision, a common practice is to train high-capacity models on large data sets using a simple self-supervised objective, and then use these “foundation” models, either directly or with fine-tuning, on a variety of downstream tasks (Bommasani et al. 2021; OpenAI 2023). Although AF and RF were originally trained to solve the supervised task of protein structure prediction, they have proven useful in unanticipated applications such as model accuracy estimation (Roney and Ovchinnikov 2022) and mutation effect prediction (Mansoor et al. 2023). Given the above and the success of hallucination, RF_joint and RF_diffusion, ProteinGenerator, it is clear that AF and RF function practically as foundation models for protein structure analysis and generation.

Therefore, to improve design success rates and expand the range of addressable problems, a key overarching challenge is improving the structural accuracy of foundation models. Applications such as enzyme design and binder design against polar targets (whether protein, DNA, or RNA) require subangstrom positioning of sidechain atoms. Although tools like RF all-atom have made it possible to model arbitrary biomolecular assemblies, their accuracy on protein-small-molecule and protein-nucleic-acid complexes lags behind that of protein monomers and oligomers (Linder and Seelig 2021; Krishna et al. 2023). Addressing this will require improving machine learning methods as well as increasing the size and diversity of training data sets.

The field of de novo protein design has only recently begun to succeed in practically important problems, partly due to the advances reviewed here. A key theme for future work will be identifying novel design problems and devising new hallucination losses or diffusion conditioning mechanisms to address them. At a technical level, even current applications often require designing degrees of freedom for which optimal values are not known a priori. A simple example of this is the length of a designed protein and those of linker regions between discontinuous motifs (Tischer et al. 2020; Wang et al. 2022a; Watson et al. 2023). A more complex example is the fact that infinitely many motifs can be consistent with the optimal geometric constraints between residues in a given enzyme active site (as computed by quantum chemistry) (Tantillo et al. 1998; Zanghellini et al. 2006). For both problems, there are currently no methods better than exhaustive enumeration, blind sampling, or human intuition (Wang et al. 2022a). If these variables can be chosen “automatically” by neural networks based on optimal patterns learned during training, design quality could improve substantially.

Joint sequence-structure design is a long-sought goal in protein design because it could potentially improve design quality by letting backbone coordinates depend on the sequence and vice versa (Huang et al. 2011; Loshbaugh and Kortemme 2020; Maguire et al. 2021; Bennett et al. 2023). The most widely used traditional method for this is “flexible backbone” sequence design using Rosetta, where multiple rounds of sequence design are alternated with relaxation of the backbone conformation (Huang et al. 2011; Maguire et al. 2021; Bennett et al. 2023). However, this requires an already-designed backbone, or a coarse-grained description of one, as an input. The few non-deep-learning methods that could design a sequence and structure from scratch, given a minimal description of the design problem (such as a motif to be scaffolded), have had poor success rates (Bonet et al. 2018; Sesterhenn et al. 2020; Yang et al. 2021). Hallucination, inpainting, and diffusion represent the state-of-the-art in solving this problem. However, the fact that a separate ProteinMPNN sequence-design step increases wet-lab success rates suggests that current generative models still cannot perfectly sample from the joint distribution of sequences and structures. Given that sequence-only (e.g., ProGen; Madani et al. 2023), structure-only (e.g., RF_diffusion), and structure-conditioned sequence generation (e.g., ProteinMPNN) all perform quite well, the key unsolved problem seems to be the way in which simultaneous sequence- and structure-generative processes are coupled. This will be an important near-term research direction and a likely source of additional gains in success rate.

Finally, a major underexplored area in de novo design is the modeling of protein function as part of the design process. The field of directed evolution has long used machine learning models trained on protein activity data to guide in silico design efforts (Yang et al. 2019; Johnston et al. 2023). To date, there are relatively few applications of these ideas to de novo design (Rocklin et al. 2017; Gligorijević et al. 2021). However, many practically important protein characteristics, such as immunogenicity or catalytic activity, are not understood well enough to be easily formulated as structure or sequence conditioning. Therefore, learning from and conditioning on direct functional measurements may be the most reliable way to design these functions. In the short term, functional measurements can be most easily incorporated during fine-tuning or inference and applied to design problems with a narrow sequence space (such as a specific fold type or enzyme class) (Madani et al. 2023; Yeh et al. 2023). However, a potentially even more powerful approach could be training a general foundation model on all existing function measurements, provided that the data can be standardized across different sources. The representations and capabilities of sequence- and structure-centric protein models are already converging (Yang et al. 2022; Heinzinger et al. 2023; Lin et al. 2023; Mansoor et al. 2023; Su et al. 2023). We may soon begin to see the application of multimodal sequence-structure-function models to the most difficult protein design problems.