Generative design of de novo proteins based on secondary structure constraints using an attention-based diffusion model

SUMMARY:

We report two generative deep learning models that predict amino acid sequences and 3D protein structures based on secondary structure design objectives via either overall content or per-residue structure. Both models are robust regarding imperfect inputs and offer de novo design capacity as they can discover new protein sequences not yet discovered from natural mechanisms or systems. The residue-level secondary structure design model generally yields higher accuracy and more diverse sequences. These findings suggest unexplored opportunities for protein designs and functional outcomes within the vast amino acid sequences beyond known proteins. Our models, based on an attention-based diffusion model and trained on a dataset extracted from experimentally known 3D protein structures, offer numerous downstream applications in conditional generative design of various biological or engineering systems. Future work may include additional conditioning, and an exploration of other functional properties of the generated proteins for various properties beyond structural objectives.

eTOC blurb

Designing proteins beyond naturally existing ones is of great potential for numerous scientific and engineering applications; however, to date, it remains prohibitively expensive. Here, we leverage attention-based diffusion models to efficiently generate novel protein sequences with prescribed secondary structures. Our models handle variegated design objectives robustly and predict de novo protein sequences that are not yet discovered in nature, which open many avenues for discovering superior protein materials and systems and can be extended beyond structural objectives.

Graphical Abstract

1. Introduction

Proteins are critical biological building blocks that constitute fundamental functions of all life, as well as significant biomaterials emerging from natural evolution, including silks, collagens, complex assemblies such as cells, and tissue assemblies such as skin.^1–4 These various structures and associated outstanding properties depend on the underlying sequences of amino acids (AAs) and the subsequent folded three-dimensional (3D) structures. For millennia, it has been a popular and fruitful approach to take inspiration from nature about how to design materials,^2,5 including recent lessons learned from proteins (e.g., designing synthetic materials inspired by nacre^6,7 and silks^8,9). Furthermore, considering that there exist 20¹⁰⁰ possible AA sequences for a 100-residue protein and that natural evolution has only sampled a small fraction of these, there remains a broad unexplored design space and hence significant potential for discovering de novo proteins with potentially unprecedented properties and functions.¹⁰ However, it is also due to this enormous design space and costs associated with experimental testing, that great challenges remain in finding appropriate tools to design de novo protein sequences that yield a set of targeted structural features or properties.^8,9,11–13

In the present work, we are particularly focused on mechanical properties of proteins, for which secondary structure content is key (previous studies have demonstrated that, elementary units of secondary structures and their interactions within one chain can still play key roles in determining the mechanical properties of protein materials). For instance, alpha-helical proteins¹⁴ tend to yield stretchy materials, beta-sheet rich ones rather rigid materials,^15–17 and combinations as in silks provide simultaneously rigidity, strength, and toughness via a nanocomposite design strategy.^15,17,18 Further, using protein molecular building blocks as means to construct higher-level hierarchical materials offers high degrees of design and flexibility to generate targeted mechanical properties, such as flaw-tolerance,^19–21 or tunable properties like stiffness or toughness^22–24 among many others. For example, for tissue repairs and regenerative medicine, the durability of protein-based implants can be tuned by designing the content and types of the secondary structures.^25–31 For silk proteins, by combining sequences with different ratios of beta-sheet, beta-turn and random coil content, the mechanical properties of the resulting materials can vary in an expanded range.^26,32,33 Therefore, the significant role played by the secondary structure content on influencing the mechanical properties of protein materials and complex tissue systems suggest that this feature can be a valuable target or condition for de novo protein design of mechanical properties.

The application of machine learning (ML) approaches to protein studies in recent years has provided effective avenues to predict structure and properties/functions based on protein sequences. Taking the folding problem of predicting 3D structures for given sequences as an example, the deep learning-based tool AlphaFold 2 represented a breakthrough in achieving competitive accuracy while bypassing expensive and time-consuming measurements in conventional experimental methods.³⁴ Using the latest AlphaFold model, the structures of ~200 million proteins in the human proteome, as well as that of several other organisms, have been predicted, going far beyond the pace of experimental methods.³⁵ Furthermore, end-to-end models based on Deep Learning (DL) that predicts the secondary structures and properties for given sequences have also been developed.^36,37 For instance, from their primary sequences, the secondary structure type and contents can be predicted with good accuracy using various ML models, including feed forward neural networks,³⁸ recurrent neural networks,³⁹ deep convolutional networks,⁴⁰ and transformer-based language-models.^41,42 The development of these deep learning models greatly reduces the cost of screening large numbers of protein sequences.

In contrast, the inverse design of de novo proteins that satisfy targeted features presents unique challenges and remains largely open for exploration, even with deep learning models. On the one hand, stochastic search algorithms constructed using handcrafted optimization functions and sampling approaches are often adopted for such inverse designs.^43,44 For example, a combination of genetic algorithms and deep learning-based predictors is required to search for protein sequences that yield the desired ratio of secondary structure content.⁴⁵ However, even with the efficient deep learning-based predictor, the iterative process of searching can potentially still be time consuming, while the convergence of the iterations and the quality and varieties of the discovered sequences are not necessarily correlated.

On the other hand, compared to applications of image generation, generative models have not yet been broadly generalized for protein structures and their potentials in solving such problems remain largely unexplored. Various deep learning methods have been used to generate images and image-like field data, including autoencoders,^46,47 generative adversarial nets (GANs)^48–50 and transformer-based diffusion models.^51,52 For example, recently diffusion model architectures,⁵¹ such as DALL-E2,⁵³ Imagen⁵⁴ and Latent/Stable Diffusion,⁵⁵ have produced state-of-the-art performance in text-to-image generation tasks for an unprecedented degree of photorealism and language understanding. For engineering applications, conditional GAN models have been demonstrated to be capable of generating stress/strain fields from modulus fields^56,57 and vice versa;^58,59 and progressive transformer diffusion models can learn and predict the various behaviors during dynamic fracture processes.⁶⁰ In comparison, protein sequences and structures show different formats and features from those of images, graphs, or image-like fields. Attempts to bridge such generative models for protein studies remain few but grow rapidly. For example, variational autoencoders have been adjusted to generate diverse new protein structures without designed conditions;⁶¹ equivariant denoising diffusion probabilistic models have been developed to generate proteins following a given topology and constraints;⁶² another diffusion model has been applied to construct scaffold structures that support a desired motif in proteins.⁶³ With these recent successes, it is promising to explore how to leverage these generative models in image domains and adjust them to handle de novo protein design effectively and efficiently.

In this paper, we propose an attention-based diffusion model for proteins and report a generative deep learning model that predicts AA sequences and 3D protein structures based on secondary structure design objectives (Figure 1). By proposing a U-Net architecture that handles one-dimensional (1D) protein data, we construct a model that takes a conditioning description of the desired secondary structures as input and produces various sequences of AAs from random noise vector sources (akin to the schematic shown in Figure 2a). This is achieved via an attention-based diffusion model that is trained by minimizing the L2 difference between the actual and predicted noise level (eq. (4)), realizing a step-wise denoising strategy, using a U-net architecture with intersecting convolutional and attention layers as depicted in Figure 2b-c.

Figure 1:

Overview of the model developed here. The model takes a conditioning description as input and produces, via a conditional attention-based diffusion model (Figure 2), an amino acid sequence. We use OmegaFold⁶⁴ and AlphaFold^34,65 to then predict 3D structures of the resulting proteins. Two models are trained; first, a model that takes fractional input of secondary structure and then predicts sequences (model A). Second, a model that takes per-residue secondary structure information as input and predicts sequences (model B). The generative model can be used to produce a number of samples that can be analyzed for further down selection (e.g., we can select samples that meet the target conditioning input the best, or that are the most novel, etc.).

Figure 2:

Overview of the attention-based diffusion model. a, Visualization of how noise and conditioning data is transformed into a solution. During training pairs of conditioning data and a resulting output are used. b, Illustration of the Markov chain of noising (top) and denoising (bottom). c, Depiction of the 1D U-net architecture that translates an input I_𝑖 into an output O_i under a condition set C_i. The model features 1D conv layers, as well as self-/cross-attention layers as shown on the right.

We then integrate the model with folding prediction methods to determine 3D structures of the resulting sequences, classify their secondary structures and compare these outputs with the input conditions. Finally, the designed sequences are checked with known proteins to analyze their novelty. After training the model on a set of Protein Data Bank (PDB) proteins, we demonstrated that the models are able to generate various de novo proteins sequences of stable structures that closely follow the given secondary structure conditions, thus bypassing the iterative search process in previous optimization methods.^43–45 Built upon the capability of our model and the known significance of secondary structures on mechanical properties of proteins, we expect our model can be useful in numerous applications in conditional generative design of various scientific and engineering protein systems.

2. Results and Discussion

Figure 1 depicts an overview of the model developed here, generating novel protein structures based on conditioning parameters. In this process, the model takes a conditioning description as input and produces, via a conditional attention-based diffusion model (Figure 2), an amino acid sequence that is then used to construct a 3D protein model. We use OmegaFold⁶⁴ and AlphaFold^34,65 to predict 3D structures of the resulting proteins. Two models are developed, trained and applied. First, a model that takes fractional input of secondary structure and then predicts sequences (model A). Second, a model that takes per-residue secondary structure information as input and predicts sequences (model B). In both models, the predictions/sequences are constructed from random signals, under conditioning, by reversing the diffusion process in a step-by-step fashion, as summarized in Figure 2a-b. The deep neural network is tasked with identifying the added noise at each step so that it can be removed successfully. Details of the models, training procedure and related details are included in the Experimental Procedures section, and Table 1 shows a summary of the eight secondary structure parameters used here, following the Define Secondary Structure of Proteins (DSSP) convention.⁶⁶ Additional aspects including an analysis of the distribution of the data in the training set are included in the Supplementary Material, specifically Figures S1, S2 and S3.

Table 1:

Secondary structure code, following the DSSP convention.⁶⁶

DSSP code	Description
H	Alpha-helix (AH)
E	Extended parallel and/or anti-parallel beta-sheet (BS) conformation
T	Hydrogen bonded turns (3, 4 or 5 turn)
~	Unstructured
B	Beta-bridge (single pair beta-sheet hydrogen bond formation)
G	3/310 helix
I	Pi-helix
S	Bend

Figure 3 shows results for protein generation based on fractional secondary structure content (model A). Figure 3a-f show a variety of representative cases, including high beta-sheet content (a), a mix of alpha-helix and beta-sheet (b, f), pure alpha-helical content (c), alpha-helical content with significant disorder (d), and a completely disordered protein (e). The left column shows the conditioning vector (eight length reflecting the eight different types of secondary structure content as shown in Table 1) and the resulting amino acid sequences. The columns in the middle depict the resulting protein structures as predicted by OmegaFold in two renderings. The right column shows a comparison of secondary structure fractions between the input conditions and those reconstructed from the generated sequences.

Figure 3:

Results for protein generation based on fractional secondary structure content (model A). Panels a-f show a variety of representative cases, including high beta-sheet content (a), a mix of alpha-helix and beta-sheet (b, f), pure alpha-helical content (c), alpha-helical content with significant disorder (d), and a completely disordered protein (e). The fractional patterns used here are similar to those that have been observed in known proteins and relate to mechanical properties. The left column shows the conditioning vector (eight length reflecting the eight different types of secondary structure content as shown in Table 1) and the resulting amino acid sequences. The two columns in the middle depict the resulting protein structures as predicted by OmegaFold in two renderings (middle left: pLDDT score, middle right: secondary structure analysis via PyMol; yellow=beta-sheet, red=helix, green=disordered). The right column shows the comparison of secondary structure fractions between the input conditions and those reconstructed from the generated sequences. Additional designs are depicted in Figure S7.

We now focus on one of the predictions, the beta-sheet structure in Figure 3a, and explore the assembly of this protein into higher-order arrangements. These assemblies are not predicted by our model; rather, we explore whether the predicted beta-strands would assemble into higher-order filamentous structures. Figure 4 shows the analysis of the results from Figure 3a in greater detail. Figure 4a-b shows a comparison of the predictions between OmegaFold and AlphaFold. The results are comparable and indicate that both folding methods yield similar results. Figure 4c shows the structure prediction of an assembly of three of these beta-sheet building blocks, predicted using AlphaFold-Multimer.⁶⁷ Figure 4d shows, similarly, the assembly geometry of eight beta-sheets. It is notable that the model arranges three sub-parts consisting of two mini beta-barrels and one bivalent assembly in the middle while for the individual chains, the designed secondary structures are preserved.

Figure 4:

Analysis of the results from Figure 3a in greater detail, and exploring how beta-strand – which are known to assemble into higher-scale structural assemblies such as amyloid filaments or other fibrous structures – yield such emerging structures. a-b, Comparison of the predictions between OmegaFold and AlphaFold. The results are comparable. Panel c shows the structure prediction of an assembly of three of these beta-sheet building blocks, predicted using AlphaFold. Panel d shows, similarly, the assembly geometry of eight beta-sheets. It is notable that the model arranges in three sub-parts consisting of two mini beta-barrels and one bivalent assembly in the middle. The designed protein fragments are preserved well when multiple chains aggregate, which indicates, for some cases, the current design models may be applied even for multimer cases in such cases, especially for beta-sheet rich protein assemblies. We use the AlphaFold-Multimer model for folding prediction of protein complexes (in c-d).

Next, we analyze the predicted amino acid sequences to assess whether, and to what extent, they represent novel sequences or closely related forms of existing and/or known proteins. This is done using basic local alignment search tool (BLAST) analysis.⁶⁸ Table 2 shows the results of the BLAST analysis for the various cases, for the results from model A. We find that generally, the model predicts structures that are similar to existing protein sequences, as can be seen from the BLAST results. However, some generated sequences (as for the 2^nd and 3^rd case) do not exist in the PDB-based training set. While there is some novelty in the sequences, and there are measures one could take to drive further variation, we focused on ways to ensure a greater diversity of sequence predictions (strategies to enhance this could be to increase the conditioning probability dropout or to add noise to the conditioning vector during training).

Table 2:

Results of the BLAST⁶⁸ analysis for the various cases, for the results from model A. Generally, the model predicts structures that are similar to existing protein sequences as can be seen from the BLAST results. However, some generated sequences (as for the 2^nd and 3^rd case) do not exist in the PDB-based training set.

Conditioning	Sequence	BLAST result: the sequence producing the most significant alignment
		among PDB proteins	beyond PDB proteins
[0, 0.7, 0.07, 0.1, 0.01, 0.02, 0.01, 0.11]	GFCRCLCRRGICRCICTR	100% query cover, 94.44% identical with PDB ID 1HVZ	N/A
[0.2, 0.2, 0.07, 0.3, 0.01, 0.02, 0.01, 0.11]	IPCACFRNYVPVCGSDGKVYGNPCMLNLAAYVKVTGLKLRFSGRLPVSNREYQ	--	94% query cover, 76.00% identical with GenBank: AHW57452.1
[0.8, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]	RDQSLEARLVELETRLSFFEQALTELSEALADARLTGARNAELIRWLLEDL	--	100% query cover, 94.12% with NCBI Reference Sequence: WP_011036690.1
[0.5, 0.0, 0.0, 0.2, 0.0, 0.0, 0.1, 0.1]	GSSGSSGVIRQMLQELVNAGCDQEMAGRALKQTGSRSIEAALEYIAKMSGPSSG	100% query cover, 96.30% identical with PDB ID 2COS	N/A
[0.01, 0.1, 0.0, 0.5, 0.0, 0.01, 0.1, 0.5 ]	RQARWELAFDLD	91% query cover, 90.91% identical with PDB ID 1ID6	N/A
[0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11]	GSSGSSGTGEKPYICIECGKVFVVNSFLARHGKTHTGEKPSGPSSG	80% query cover, 80.43% identical with PDB ID 2ENE	N/A

This is accomplished via the second model, Model B, where we use residue-level conditioning. In this scenario, each residue is conditioned based on one of the eight DSSP secondary structure codes. Figure 5 depicts results for protein generation based on residue-level secondary structure content implemented in model B. We consider five cases with different distributions of secondary structure, including predominant beta-sheet (a), a long alpha-helix with a breaker in the center (b), a small alpha-helix (c), a sandwich alpha-helix/beta-sheet structure (a beta-sheet centered between two helical domains) (d), and a partially disordered-helical protein (e). The folded results (left column in Figure 5) reveal generally good agreement with the design objectives specified in the right (blue font), and confirms that the model enables us to design specific geometric details and localizations of structure. Even though these proteins are de novo sequences (see BLAST analysis in Table 3), OmegaFold (and AlphaFold) reach relatively high pLDDT scores (denoting a per-residue estimate of prediction confidence in a range from 0 to 100); and it is typically the largest for the alpha-helical structures.

Figure 5:

Results for protein generation based on residue-level secondary structure content (model B). We consider five cases with different distributions of secondary structure, including predominant beta-sheet (a), a long alpha-helix with a breaker in the center (b), a small alpha-helix (c), a sandwich alpha-helix/beta-sheet structure (a beta-sheet centered between two helical domains) (d), and a partially disordered-helical protein (e). The right column shows the input secondary structure sequences (in blue), generated amino acid sequences (in black) and secondary structure contents reconstructed from the generated proteins (in red). The left column depicts the resulting protein structures in the rendering of pLDDT score. The secondary structures of the folded results show good agreement with the design objectives and the per-residue accuracy (defined in eq. (6)) of those cases are 44% (a), 79% (b), 81% (c), 60% (d) and 73% (d), confirming that the model enables us to design specific geometric details and localizations of structure. Even though these proteins are de novo sequences (see analysis in Table 3), OmegaFold (and AlphaFold) reach relatively high pLDDT scores; typically, the largest for the alpha-helical structures. Several additional designs are depicted in Figures S4–S6.

Table 3:

Results of the BLAST analysis for the various cases for model B. In the interest of space, the secondary structure design objective is not shown but rather, the case ID (from a to e) is referred to corresponding to the cases shown in Figure 5. The analysis shows that the model generates protein structures that reflect the design objectives well (see analysis in Figure 5), and which are also de novo sequences that have little similarity with existing amino acid sequences. The BLAST results indicate similarities in around 50–60% for most cases, with one case reaching 85.71% case for 66% of the query cover. Most other cases have much smaller similarities and small query covers.

Case	Sequence	BLAST result: the sequence producing the most significant alignment
		among PDB proteins	beyond PDB proteins
a	ACICANYDYLGRKCVCIRCGTPGLFSVKCYVPRL	--	82% query cover, 43.24% identical with GenBank: MBR4744867.1
b	ASTFEEAIEIKRVLQIIGSKLARSGETSPEEIEEILIIFEVIKAVLKA	--	60% query cover, 57.58% identical with GenBank: MBR3311276.1
c	GPDAVLFAGFKTGLAKMLKNR	--	66% query cover, 85.71% identical with GenBank: MCI9553838.1
d	ASNETKLEEQKQDAIKECESEVTCIKIIKANGEDGIGVILCDSGGDVVADKVKIAQDGDNEGFAKSSQLHARIAELN	--	45% query cover, 44.68% identical with NCBI Reference Sequence: WP_257531070.1
e	ASNETKLEEQKQDAIKECESEVTCIKIIKANGEDGIGVILCDSGGDVVADKVKIAQDGDNEGFAKSSQLHARIAELN	--	41% query cover, 61.54% identity with GenBank: TBR12876.1

Built on these predictions we now conduct a more detailed analysis of one of the designs. Figure 6 reveals a detailed analysis of two designs (Figure 5a and e), including a comparison between OmegaFold and AlphaFold predictions. The indication of the residue number (C- and N-terminus, respectively) and the localization of specific secondary structure shows excellent agreement with the design objective. The top row in each case depicts the structure color coded by residue number (rainbow plot), and the bottom row a secondary structure color coding.

Figure 6:

Detailed analysis of two designs (Figure 5a and e), including a comparison between OmegaFold and AlphaFold predictions (the design objective is noted at the top of each subplot). The indication of the residue number (C- and N-terminus, respectively) and the localization of specific secondary structure shows excellent agreement with the design objective. The top row in each case depicts the structure color coded by residue number (rainbow plot), and the bottom row a secondary structure color coding.

Table 3 summarizes the results of the BLAST analysis for the various cases for model B. The analysis shows that the model generates protein structures that reflect the design objectives well (see analysis in Figure 5), and which are also de novo sequences that have little similarity with existing amino acid sequences. We find that the BLAST results indicate similarities of around 50–60% for most cases, with one case reaching 85.71% and one for 66% of the query cover. It is further noted that most of the other cases have much smaller similarities and small query covers. This result is noteworthy as it provides strong evidence that Model B is capable of discovering new protein designs. The difference in this explorative potential is likely due to the way by which novel sequences are generated, using condition dropouts. Since the conditioning sequences are short, there is limited variability as the model explores new designs that conform less strongly to the conditioning task.

Several deep learning-based protein design systems have emerged recently.^69–71 Complementing these alternative approaches, our model serves a unique design perspective focused on secondary structure content (model A) or residue-specific secondary structure content (model B). We briefly discuss the differences with regard to other generative models. For example, to generate desired sequences, proteinMPNN⁶⁹ requires atomic details of the protein backbone design as input, including distances between Cα-Cα atoms, relative Cα-Cα-Cα frame orientations and rotations and backbone dihedral angles. In contrast, our model only conditions the secondary structure types of the generated sequences in the global (model A) or residual (model B) level, leaving the detailed atomic coordinates information unspecified. Indeed, for one pick of secondary structure contents, in terms of either global fractions or secondary structure type sequence, there could exist many different backbone atomic structures. So, these two are protein design tools working on different levels with different respective advantages.

Another model is Chroma,⁷⁰ which adopts a graph neural network representation of proteins, and is based on a diffusion process between protein backbone and the noisy structure of a collapsed polymer system, instead of Gaussian noise. In the RFdiffusion model,⁷¹ the folding tool, RoseTTAFold,⁷² is fined-tuned as the denoising network. Both models pay particular attention to the backbone structures of the designed protein. Thus, various conditions regarding the atomic structure of the backbone can be implemented for both monomer and protein complex designs, making them general-purpose structure-focused protein design tools. In contrast, our model specifically focuses on the mapping between secondary structure and primary sequences, bypassing the construction of the atomic details of the backbone. The rules of what sequences can be generated are learned implicitly by our model. Correspondingly, users can directly work on the residual level (model B) as well as monomer level (model A) and skip the detailed choice of conditions on the atomic level. One can also combine Model A as a way to get a first protein sequence estimate, and then refine the design using Model B by specifying residue-level secondary structure detail.

To test and apply our models, we note that there is a lack of clear/explicit rules on which secondary structure distributions are physically possible and which are not, given the finite amino acids as the building blocks. Therefore, it remains unclear and challenging to directly construct an exhausted set of physically possible input conditions to test our models. Instead, in the current work, we focus on testing our models with secondary structure patterns that have been observed in known proteins and relate to mechanical properties. In Figures 3 and 5, we manually constructed the input conditions so that they resemble some typical secondary structure patterns in known PDB proteins, especially those that are known for affecting the deformation process and mechanical properties, such as beta-sheet and alpha-helix. Here, we intentionally included noises or even errors in the input conditions to test and demonstrate the behaviors and robustness of our model. For example, for Model A, the sum of the secondary structure fractions in some of the tested conditions in Figure 3 slightly deviates from 1. However, even with those imperfect inputs, the model is still able to generate sequences which respect the relative concentrations of different secondary structure types as shown in Figure 3 (the right column). Moreover, the generated proteins have a correct fractional content that does add up to one. To expand on this, Figure S8 shows an analysis of the rigor of input conditioning in Model A. We examine here the difference of predictions that do not necessarily add up to one (${\sum }_{i=1\dots 8}{n}_i=1$, Figure S8a) and conditioning that add up to one (${\sum }_{i=1\dots 8}{n}_i=1$, Figure S8b). Similar predictions are obtained in most cases, except for the second to last from the bottom. This result shows that the model has a certain degree of robustness and can deal with unphysical input (e.g. fractional secondary structure content that does not add up to one) and still produce reliable results.

While the rules of protein construction are complex, especially for secondary structures that yield long-range interactions (such as beta-sheets, where close or distant parts of the protein combine to form the 3D structure), primarily local-focused structures such as alpha-helices do not have such constraints. Hence, we conducted a case study to explore whether the model can solve systematic variations of the input. Applying Model B, we generate a series of alpha-helix (AH) sequences of increasing length. As shown in Figure S4, as we systematically control the length of helical residue in the inputs, the model predicts a length-dependent conformation transition, from an unconstructed chain, to a straight segment of AH, to multiple AH segments with kinks, and to a slightly curved continuous AH segment. This transition indicates that the trained model not only seeks to respect the input conditions but also yield to the underline constraints of physically possible secondary structures learned implicitly during training.

In another test, Figure S5 depicts designed sequences based on secondary structure sequences identified from recently deposited PDBs, taken from the CASP15 (a, PDB ID 7ROA) and CASP14 (b, PDB ID 7JTL) target set. The generated proteins, shown on the right panel, are de novo sequences with no significant overlap with any existing known sequences. This shows that the model can generate new protein sequences – different from those generated via evolutionary mechanisms – and offer alternative, or candidate sequence options for structural templates. In addition to this result, Figure S6 shows results for designed sequences based on several design targets with different complexity (hard to easier, from top to bottom). The left panels show the error value over the iterations (we repeat generation until a maximum number of iterations is reached, or until the design error is below a critical value). The critical error is set to 0.1 in all cases, only the alpha-helix structures reach that goal. Generally, error values fluctuate around a mean value, reflecting the general level of challenge to solve the design problem. Furthermore, since we trained our model using a classifier-free strategy, we can explore how removing conditioning information affects the results. In Figure S7 we show the results of these experiments, which demonstrate that increasing the parameter $\xi$ from 1 to 30 offers additional tuning of the predictions (for $\xi =1$ the model is fully conditioned, and becomes less conditioned as the parameter increases).

3. Conclusions

We report a method to generate novel protein designs, based on an attention-based diffusion model. Two variants of the model are presented, one that conditions predictions based on overall fractional content of secondary structure (Model A) and a second model based on per-residue level secondary structure conditioning. The results show that Model B is more effective in generating de novo sequences that are not found in nature, or that have not been discovered yet (see, analysis in Table 3). The model is capable of predicting a variety of new sequences; while they reveal some similarities with existing sequences (on the order of 50–60% similarity) they introduce significant amounts of new design cues. This level of variation can be regarded as an interesting measure to add to the diversity of natural protein designs.

Future work could include further refinement of sequences to meet additional criteria, for instance biological activity. Other critical steps include experimental validation of the designs, especially for cases that have little similarities with known proteins. Also, it may be interesting to add more explorative capacity to Model A, to achieve greater sequence diverse much more distinct from existing proteins at levels similar to what we achieved in Model B. One way to achieve this could be to use Model B to generate a greater variety of protein sequences, then fold the proteins and use these new data points to expand the existing dataset. There are already millions of new protein structures that were predicted by computational methods available, which could also be added to the training data set. So far, we excluded these predicted proteins as we wanted to focus the training procedure on experimental data. One may also use integrated optimization algorithms where sequence predictions are accepted, rejected or altered based on performance criteria, such as invoking structure prediction tools or methods to directly assess secondary structure content in an end-to-end fashion. These strategies may offer interesting research avenues and the possibility of achieving multiple objective functions.

There are other future research avenues in which this work can be taken, including via the use of inpainting strategies where part of the sequence is provided as a boundary condition, and the model is asked to fill in a missing part. If conditioning strategies are used as proposed in this paper, such models would likely be able to solve protein design tasks of the kind where a domain is sought to be altered, for instance for mechanical or other (e.g. biological, and other) property optimization. A useful application of this strategy can be, for instance, the design of silk materials where key domains could be strengthened by adding greater beta-sheet forming domains, whereas others can be rendered stretchier via the use of helical domains. The sandwich design shown in Figure 5d is an example that indicates the feasibility of such design strategies. An important next step is the experimental synthesis of such protein structures and the use of mechanical analysis tools like optical tweezers to assess outcomes. The tool reported here may also be combined with other generative protein models,^69–71 including those focused on predicting sequences to meet a certain geometric/shape demand.

Our model can generate de novo proteins following desired secondary structures. As another future research topic, the novelty of these new sequences may lead to superior mechanical properties and related functions that go beyond what has been observed in natural proteins. For example, studies of natural proteins have demonstrated that the mechanical stiffness, strength and toughness of silk show size effect and length dependence on the beta-strands in silk proteins.⁷³ Combining this mechanism knowledge with our generative model, one can systematically generate new sequences that yields the secondary structures of rationally optimized designs and verify the performances via simulations and experiments.

While we have explored the potential to study how beta-strand monomers assemble into larger assemblies (Figure 4), the models and scope of this work is strictly limited to single chain proteins. It has been demonstrated that the individual pieces of secondary structures and their interactions can play important roles in determining the mechanical proteins of protein and protein materials. For example, the unfolding of individual alpha-helices in vimentin in intermediate filament contributes mainly to its great extensibility;¹⁹ the interaction between parallel pieces of beta strands in beta-sheet-rich proteins governs their rupture strength.⁷⁴ Our models have generated sequences with similar patterns (e.g., Figure 3a and 3c, or Figure 5c and 5a) effectively under different formats of conditioning.

We also identified ways to increase the de novo nature of predicted sequences, especially using Model A that tends to synthesize less diverse sequences compared to Model B. With strategies such as classifier-free guidance and early stopping during training, we can generate significantly more diverse sequences (another option, not yet explored here, can be to vary the number of sampling steps, which can be powerful to generate greater variety of generative results). A visual summary of the sequence alignments, as examples for some of the designs, are shown in Figures S9 and S10. Figure S11 shows sample alignments to explain the nature of the variations of the generated sequences to provide visual depictions of the results and the novel nature of the sequences. As evidenced from these analyses, the generative algorithm realized in Model A has a capacity to discover sequence designs from a deep reservoir of patterns; some of which have also been discovered via natural processes.

As another future direction, it may be interesting to generalize our models towards the design of multimers under similar secondary structure conditions. One straightforward strategy to do so is to introduce trivial linkers that represent the break between individual chains and further train the model with available data of protein complexes as similar scheme has been experimented to generalize AlphaFold2 for multimer tasks.^75,76 A more systematic way to generalize our current models for multimer design would be to include more conditions, including those specifying the cross-chain geometry in protein complexes, like binding residues and residue ties for symmetric or repeat protein designs.⁶⁹ Next, we could envision combining the current model and their possible generalizations for multimer tasks to generate large numbers of sequences and screen for those that undergo conformational changes in terms of secondary structure when forming multimer complexes. As those require significant additional research (including a development of a proper dataset), we leave them to future studies.

4. Experimental Procedures

Resource Availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Professor Markus Buehler (mbuehler@MIT.EDU).

Materials availability

This study did not generate new unique reagents.

Data availability

Data and codes, as well as trained weights, are either available on GitHub at https://github.com/lamm-mit/ProteinDiffusionGenerator or will be provided by the lead contact based on reasonable request.

4.1. Dataset

We use the same dataset utilized in the earlier work, see⁴⁵. Figure S1 shows a statistical analysis of the dataset used to train model B. Figure S1a summarizes the sequence length distributions for lengths up to 128 amino acids. Figure S1b depicts the secondary structure coding statistics. Figure S1c shows the amino acid residue statistics.

The right panel in Figure S1 shows the tokenizer dictionary for both secondary structure and amino acid codes. The associations listed in these is used to translate string characters (for both amino acid codes and secondary structure code) into a numerical integer value. We then normalize the data to lay between 0 and 1, resulting in float numbers for input and output.

For Model A, the input vector is the fractional content of the 8 different types of DSSP secondary structures:

$$C=\frac{1}{N}\left[n_{\text{H}},n_{\text{E}},n_{\text{T}},n_{~},n_{\text{B}},n_{\text{G}},n_{\text{I}},n_{\text{S}}\right]$$ (1)

where $n_i$ is the number of residues with type $i$. secondary structure, and $N$ the total number of amino acids in a proteins.

Unused tokens in the predicted sequence (both models) and in the conditioning sequence (model B) are identified with token 0 for padding.

Figure S2 shows the distribution of the fractional contents of the eight types of secondary structure in the training set used for Model A. Note that, for many types of secondary structures, the distribution is not uniform and quite skewed with respective to the fraction between 0 and 1. These distributions are likely a reflection of chemical and/or biological/evolutionary, principles that resulted in the protein designs). There exist relatively few sequences with higher fraction of beta-bridge (>0.4), beta-sheet (>0.9), 3/3₁₀ helix (>0.8), hydrogen bonded turn (>0.8), pi-helix (>0.5), or bend (>0.7). Figure S3 depicts plots of the fractional content distributions of different pairs of secondary structures in the training set. These pair correlation plots are useful to understand naturally occurring combinations of secondary structure contents.

4.2. Design of the neural network architectures

All code is developed in PyTorch,⁷⁷ except for the tokenizer that is developed based on, and trained, using TensorFlow Keras.⁷⁸

The model is built on a one-dimensional U-Net architecture composed of convolutional and transformer layers with skip connections (Figure 2c). U-Nets are a type of neural network that features the same input dimension as output dimension, commonly used in problems such as image segmentation. The U-Net implemented here feature a more complex architecture as described in ^54,79, including the use of ResNet blocks, attention modules, and skip connections (see Table 4 for details). As depicted in Figure 2c, the U-net features self- and cross-attention blocks. These are used to contextualize the denoising process with both, the conditioning data and the diffusion time step. The use of Transformer-type architectures provides a meaningful way to also learn long-range relationships in the construction of amino acid sequences and how they interact to satisfy various physical/chemical constraints. During training, these are all learned.

Table 4:

Parameters used in the progressive transformer diffusion model (parameters for 1D U-net, the integrated architecture, and additional parameters are provided).

Neural network component	Parameter	Value
U-net	U-net dimension	128/512/768 (ratio-based model, Model A) 256 (residue-level model, Model B)
	Dimension multipliers	1, 2, 4, 8
	Resnet blocks	1
	Attention heads	8
	Feed forward multiplier	2.0
Overall architecture	Sequence length	64/128
	Cond. drop probability	0.2/0.1
	Sample steps	96/64
	σ min	0.002
	σ max	80, 160
	σ data	0.5
	ρ	7
	P mean	−1.2
	P std	1.2
	P churn	80
	S t,min	0.05
	S t,max	50.
	S noise	1.003
	Optimizer and parameters	Adam Learning rate=1E-4, epsilon=1e-8, betas=(0.9,0.99)
Additional parameters	Batch size	256
Additional parameters, Model A:
Condition embedding dimension	512
Positional encoding depth	128
Signal embedding depth	368

The U-Net is used to translate the secondary structure fraction vector (model A) or input sequence (model B) into the final field output via a learned denoising process. Figure 2b visualizes the denoising process, where the top defines a Markov chain operator q that adds Gaussian noise step-by-step (according to a defined noise schedule that defines how much noise 𝜀_𝑖 is added at each step i), translating the original sequence X₀ (left) into pure noise, X_F (right) via:

$$X_{i+1}=\text{q}\left(X_i\right)$$ (2)

The deep neural network is then trained to reverse this process, identifying an operator p that maximizes the likelihood of the training data, thereby offering a means to translate noise to solutions and thereby realizing the transition illustrated in Figure 2a (noise to solution), in a step-by-step fashion as indicated in the lower row of Figure 2b:

$$X_{i-1}=\text{p}\left(X_i\right)$$ (3)

Specifically, the U-Net is tasked with predicting the added noise and correspondingly, the L2 distance of the actual added noise ε_i and the predicted added noise ${\epsilon }_i^{\prime }$ is adopted as the loss function for the training process.

$$ℒ={‖{\epsilon }_i-{\epsilon }_i^{\prime }‖}_{L_2}$$ (4)

That way, the trained model can predict the added noise. Then, knowing this quantity then allows us to realize a numerical solution to the problem stated in eq. (3), used to generate the next iteration of the denoised sequence:

$$X_{i-1}=X_i-{\epsilon }_i^{\prime }$$ (5)

In eq. (2), the sequence $X_i$ at step $i$ is transformed by removing the noise ${\epsilon }_i\prime$. This process is performed iteratively; whereas the neural network predicts, given the current state $X_i$, the noise to be removed at a given time step in the denoising process (see Figure 2b).

We use an improved noise schedule, sampling and training processes proposed in proposed in ⁸⁰ since it provides us with enhanced and computationally efficient predictions, specifically obtaining results within just 64 (model A) or 96 (mode B) denoising steps. Table 4 provides details about the model architecture parameters. The implementation is based on the code published at ⁷⁹, but extended to feature a new U-Net architecture to feature one-dimensional sequence data with higher-order embeddings.

The conditional encoder scales the sequence data values to be between 0 and 1, and feeds each in the embedding dimension (and unscaled for reverse tokenization and analysis).

In Model A, the conditioning is fed via embeddings that are used for cross-attention with the input, after being expanded into a higher-order embedding dimensions through linear layers. Trainable positional encoding is used (ordinal ordering of each conditioning parameter, from 1 to 8 as defined in eq. (1) is designated and then encoded via a fully connected embedding layer). In Model B, the conditioning is provided as sequence conditioning, where the conditioning sequence of secondary structure encodings is concatenated to the input, i.e., the noise vector. We find the latter strategy to work better when the conditioning signal has similar dimensionality as the output (and this is the case in Model B since the conditioning and prediction are of the same dimension). In both cases, this yields a conditional algorithm where

$$X_{i-1}=\text{p}\left(X_i,C_i\right),$$ (6)

So that the model can produce samples that meet a certain target features defined by $C_i$.

Model A tends to generate less novel, diverse sequences. We explored ways to address this limitation. One possible way is to use early stopping to avoid overfitting, as had also been proposed in ⁸¹. The results shown in Figure S8 are obtained, for instance, using a model with a relatively large 768-dimensional U-net and early stopping. This strategy, combined with classifier-free guidance⁸² parameter variations as presented in Figure S7, can be means to increase the creative capacity of this model. Classifier-free guidance is achieved by training the model with conditional dropout; in our case, 20% of the time the conditioning information is removed so that we can make predictions both, with conditioning $\left(\text{p}\left(X_i,C_i\right)\right)$ and without conditioning $\left(\text{p}\left(X_i\right)\right)$. During sampling, the predictions are then combined according to $p_{\text{CFG}}\left(X_i,C_i;\xi \right)=\text{p}\left(X_i\right)+\left(\text{p}\left(X_i,C_i\right)-\text{p}\left(X_i\right)\right)\xi$, where $\xi$ is a conditioning parameter that determines how the conditional and unconditional solutions are mixed ($\xi =1$ yields the fully conditioned model, and for larger values, less conditioning is obtained). Other than in Figure S7, all results in this paper are obtained using $\xi =1$.

The entire prediction pipeline involves all of the steps shown in Figure 1; first, taking a conditioning parameter, using the diffusion model to make conditional predictions, and then predicting the folded 3D protein structure using OmegaFold.⁶⁴ For validation we also implement folding predictions using alternative methods, including AlphaFold 2 via ColabFold and some testing with trRosetta. For details regarding the primary folding strategies used, see Section 4.4.

As indicated in Figure 1, both models sample solutions, and are hence capable of generating a set of possible solutions to the same design problem. As a systematic way to obtain the best possible solution, we implement an iterative algorithm as outlined in Figure 1. We repeat generation until we reach a maximum number of iterations, or until we are below a critical error value (see Figures S6 for an analysis of errors over iterations). Alternative approaches may target those designs that are the most novel, or that show a compromise between novelty and meeting the design demand.

4.3. Training and validation

We use an Adam optimizer,⁸³ with Table 4 including a variety of key model and hyperparameters. Figure 7 shows several validation cases, comparing ground truth and predictions, for two cases (a and b), for Model B (results are similar for Model A). These are protein sequences taken from the validation set (10% of the total data available); hence, they are not de novo sequences and do not merit further analysis. Both cases show that the model can accurately predict sequences according to specific secondary structure content. For cases where multiple sequences exist that correspond to the same or similar secondary structure input, multiple, varied predictions are made.

Figure 7:

Validation cases, comparing ground truth and predictions, for two cases (a and b). These are protein sequences taken from the validation set (10% of the total data available); hence, they are not de novo sequences. Both cases show that the model can accurately predict sequences according to specific secondary structure content. For cases where multiple sequences exist that correspond to the same or similar secondary structure input, multiple, varied predictions are made.

To measure the conditioning capability of model B, we define per-residue accuracy, 𝐴_pr, as the fraction of residues with the designed secondary structure types for the generated protein sequence, i.e.,

$$A_{\mathrm{pr}}=\frac{n}{N}$$ (7)

where $n$ is the number of residues with the same secondary structure types as the input condition, and $N$ is the total number of amino acids in the protein. $A_{\mathrm{p}\mathrm{r}}$ is between 0 and 1, and the error is defined as $E_{\mathrm{p}\mathrm{r}}=1-A_{\mathrm{p}\mathrm{r}}$.

4.4. Protein folding

We implement OmegaFold⁶⁴ directly in our model architecture for rapid prediction of protein structures from the sequence. OmegaFold offers a rapid alternative as it does not require Multiple Sequence Alignment (MSA), yet produces results of similar accuracy as AlphaFold 2 and trRosetta (and similar, related state of the art methods).

To check the results, we further use AlphaFold 2³⁴ for monomers and AlphaFold-Multimer⁶⁷ for multimers via ColabFold⁶⁵ to conduct these experiments, using the ColabFold implementation. We use the pdb70 template set, and MMseqs2 (UniRef and Environmental), using 6 cycles. The highest prediction is used for the analysis in this paper. Additional comparisons are conducted using trRosetta⁸⁴ for validation.

The inclusion of folding tools and the comparison of several of such prediction strategies here are aimed at validating whether the generated sequences will likely fulfill the designed secondary structure conditioning parameters. While the ultimate validation of our model may require producing and examining those protein sequences experimentally, the applications of the state-of-the-art in silico folding tools, like AlphaFold2, provide a useful benchmarking pathway of higher efficiency and lower costs, which has been shown to be a viable strategy in recent works. As shown in Figures 3 and 5, the majority parts of the folded structures achieve a relatively high pLDDT score (~70), which indicates that the predicted structures are expected to be modelled well and remain stable. Furthermore, to fold de novo sequences with potentially limited multiple sequence alignments (MSAs) information, we use OmegaFold, which is designed to make prediction directly from the primary sequence, accurately and efficiently without MSAs. We find good agreement between high-confidence folded structures predicted by these two methods for the sequences we generated, including the de novo designs. This gives us confidence that the novel sequences generated by our model are likely to deliver the desired secondary structures.

4.5. BLAST analysis

The basic local alignment search tool (BLAST)⁶⁸ analysis for the various cases is conducted using the blastp (protein-protein BLAST) algorithm, and the non-redundant protein sequences (nr) database. Summary results are shown in Tables 2 and 3, and detailed results are provided in Figures S9–S11.

4.6. Visualization

We use PyMol⁸⁵ and Py3DMol⁸⁶ for visualization of the protein structures.

4.7. Software versions and hardware

We use Python 3.8.12, PyTorch 1.10 ⁷⁷ with CUDA (CUDA version 11.6), and a NVIDIA RTX A6000 with 48 GB VRAM for training and inference.

Highlights.

Diffusion models can efficiently generate proteins with desired secondary structures.

De novo protein sequences not yet discovered in nature can be generated.

The models remain robust regarding imperfect or even unrealistic design goals.

The models can be extended to generate de novo proteins for other properties.

THE BIGGER PICTURE.

The design of de novo protein sequences has great potential in achieving superior combinations of novel functions and mechanical properties beyond known, natural proteins. However, the tremendous number of possible sequences and the cost of experimental testing make the effective search and validation of superior de novo protein candidates extremely challenging. Here, we leverage a diffusion model-based deep learning framework to efficiently generate novel protein sequences that meet a desired overall secondary structure fractional content or per-residue type of secondary structure. The generated sequences show novelty beyond existing, natural ones. By robustly generating various novel sequences with the desired structural features, our model provides rapid strategies for target-guided de novo protein design that leads to novel discoveries of superior protein materials for various biological and engineering applications and can be extended for other design objectives in future work.