End-to-End Deep Learning Model to Predict and Design Secondary Structure Content of Structural Proteins

Abstract

Structural proteins are the basis of many biomaterials and key construction and functional components of all life. Further, it is well-known that the diversity of proteins’ function relies on their local structures derived from their primary amino acid sequences. Here we report a deep learning model to predict the secondary structure content of proteins directly from primary sequences. Understanding the secondary structure content of proteins is crucial to designing proteins with targeted material functions, especially mechanical properties. Using convolutional and recurrent architectures and natural language models, our deep learning model predicts the content of two essential types of secondary structures, alpha helix and beta sheet. The training data is collected from the Protein Data Bank and contains many existing protein geometries. We find that our model can learn the hidden features as patterns of input sequences that can then be directly related to secondary structure content. The alpha helix and beta sheet content predictions show excellent agreement with training data and newly deposited protein structures that were recently identified and which were not included in the original training set. We further demonstrate the features of the model by a search for de novo protein sequences that optimize max/min alpha-helix/beta-sheet content and compare the predictions with folded models of these sequences based on AlphaFold2. Excellent agreement is found, underscoring that our model has predictive potential for designing proteins with specific secondary structures and could be widely applied to biomedical industries, including protein biomaterial designs and regenerative medicine applications.

1. Introduction

Proteins form the basis for one of the most abundant biomaterials occurring in nature, as well as essential building blocks that constitute fundamental functions of all living organisms [1]. Their function depends mainly on secondary structures [2], which can be challenging to identify using traditional experimental or computational methods [3,4]. Deep learning now provides a novel tool to elucidate the relationships between input sequences and predicted secondary structures, and can hence be used as a means to predict other properties for these proteins or to optimize sequences in a materials-by-design paradigm [5-8]. For example, the secondary structure has been directly linked to mechanical properties as well as rates of degradation of protein materials [9-21].

The use of deep learning for predicting secondary structures has become a vital research area in recent years. Along with powerful computational ability and the emergence of Graphical Processing Unit (GPU) acceleration (the use of GPU compute power to significantly accelerate computational tasks), deep learning models are now capable of finding hidden patterns in complex datasets. Several models have been applied to predict secondary structures from their primary sequences, such as support vector machines, feed forward neural networks [22,23], recurrent neural networks [24,25], as well as deep convolutional networks [26,27]. However, previous studies have focused on determining the secondary structure feature of each residue, that is, typically three general states (Q3) or the more general 8 states (Q8) predicted by conventional Dictionary of Secondary Structure of Proteins (DSSP) methods that operate based on full molecular geometric models of proteins. The average accuracy for the Q3 model, for instance, is about 80%, with the highest accuracy of about 85% [28]. In contrast to these earlier models that focus on residue-level predictions, our approach aims to predict an overall features of the folded protein, and estimates the amount of alpha-helix vs. beta-sheet content a protein has. Such overarching predictions can be powerful in protein design, where biochemists are interested in screening a very large number of protein designs against a target function.

For biomaterials, the design and control of secondary structures can be used to tune or predict the durability of protein-based implants for tissue repairs or regenerative medicine [11,16,29-34]. For example, increased content of beta sheet secondary structure in silk protein material systems is directly linked to stronger materials with slower rates of enzymatic digestion; thus, more durable materials with a longer lifetime in vivo ([9,10,14-16]). Similarly, collagen-based protein biomaterials are inherently longer lasting in vivo and more mechanically robust when a higher content of alpha helix is present, reflective of organized hierarchical structures, in contrast to the denatured, gelatin-like formats of collagen, where rapid degradation and low mechanics are inherent limitations ([17]). It is also well understood that beta-sheet structures such as those found in amyloid fibrils play an important role in generating very strong and tough biomaterials [35-39].

Furthermore, recent progress in the study of structure-function relationship of synthetic polypeptide polymers paves the way to obtain materials to further develop platforms for drug delivery with controllable, precise, and directed self-assembly/release properties. As an example, the secondary structure content of silk fibroin nanoparticles has been shown to govern the release of small model molecule drugs: more crystalline (beta sheet-rich) structures demonstrated a faster release rate [40]. Another example of how control of secondary structure can be applied to biomaterial design includes fine tuning of the materia´s mechanical properties. Both the mechanical strength and extensibility of silk proteins are determined by the total amount of both β-sheet nanocrystals and non-crystalline amorphous regions within the protein, respectively [41,42]. By combining sequences from different silk types with varying ratios of β-sheet, β-turn and random coil content, the mechanical properties of the resulting materials were expanded [43].

In this paper, we report a deep learning model to predict proteins' overall secondary structure content directly from the sequence of amino acids, focusing on the overall content in a given sequence. Based on training data collected from the Protein Data Bank [44] (125,955 structures, details see Materials and Methods section), our model learns the hidden features as patterns of secondary structures. The predictions of alpha helix and beta sheet ratio show good agreement with the training and testing data as well as with newly deposited protein structures. Hence, our model shows potential for the design of fibrous proteins with specific secondary structures, and hence could be widely applied to needs in medicine and more broadly where protein-based material applications are relevant (e.g., food technologies, environmental monitoring and devices, energy storage and utility, and others) [45,46]. Moreover, due to the known significance of secondary structures for the mechanical properties of proteins, the algorithm may be useful for the design of structural and mechanically relevant proteins [11-13,18-21]. A few examples are provided in this paper.

2. Results

The details of the model, training and validation process are summarized in the Materials and Methods section. The model uses data of existing protein structures (more than 120,000 proteins), and their secondary structure content, for training. A summary of the input data is shown in Figure 2. Figure 3 shows the overall process used here for training and testing the end-to-end deep learning model. Figure 3(a) shows the flowchart of the training process. The input data is the protein sequence, and the structure of the deep learning model includes convolution blocks, bidirectional Long short-term memory (LSTM) blocks, and the fully connected layers towards the end. The outputs of the deep learning model are two scalars, corresponding to the alpha helix and beta sheet concent, respectively. Figure 3(b) depicts steps taken for testing the prediction ability of the deep learning model. The protein sequences are fed through the deep learning model, and the final output are alpha helix and beta sheet content, corresponding to the each of protein sequences, which can then be compared to the ground truth (experimental data or computational predictions using other methods).

Figure 2.

Summary of the data distribution of alpha helix and beta sheet ratios as derived from the Protein Data Bank (PDB) database [44]. The histogram shows different numbers of alpha helix and beta sheet ratios. In panel A, we can see that most of the sequences are between 30 % and 50 % of alpha helix ratio, and there is a small part of the sequences even more than 80 %. In panel B, we show that most sequences are less than 30 % of beta sheet ratio. The distribution maps of alpha helix and beta sheet are almost identical, overall. Panels (C) and (D) depict the ratio distributions of each component.

Figure 3:

The overall processes to training and testing the end-to-end deep learning model. Panel A shows the flowchart of the training process. The input data is the sequence of the protein, and the structure of the deep learning model includes convolution blocks, bidirectional LSTM blocks, and fully connected layers. The outputs of the deep learning model are alpha helix and beta sheet content, respectively, characterizing a global protein property. Panel B depicts steps for testing the prediction ability of the deep learning model (by comparing the predictions against proteins not included in the training). The protein sequences flow through the deep learning model, and the final output are alpha helix and beta sheet ratios corresponding to the input protein sequences. In future work we anticipate coupling the predictive model with an optimization algorithm such as genetic evolution or simulated evolution, to identify new sequences that meet certain design criteria.

Training results

The history of the training process is depicted in (a), which shows values of the MSE loss for the training and validation dataset. Both training error and validation decrease significantly and converge with towards a mean square error (MSE) loss lower than 0.005 after 50 epochs. Furthermore, the R² scores of alpha-helix and beta-sheet ratio in training data are 0.99 and 0.96, respectively, close to 1. Hence, the difference between the ground truth label and the prediction value is slight and the model is considered reliable.

Prediction capacity

To visualize the prediction ability of the deep learning model, scatter plots are chosen to demonstrate a direct comparison between prediction and ground truth. As can be seen in Figure 4(b-c), the testing and training points roughly overlap. This result supports that our labels of training data conform to our predictions from the testing. It is noted that the comparison for beta sheet content does not overlap as much as the testing points for alpha helix, but still has a high coincidence rate. Overall, the distribution of both training and testing points are concentrated in the diagonal zone of the diagrams, which means that the gaps between labels and ratios predicted by the model are narrow. Also, the R² scores of alpha-helix and beta-sheet in the testing dataset are up to 0.99 and 0.98; thus, the model has a strong ability of predicting alpha helix and beta sheet ratios.

Figure 4.

Panel A: Mean square error (MSE) during the whole training process. The data shows that the MSE loss decreased rapidly at the beginning of the training process and then converged to a quite low loss after 60 epochs. The prediction results by the deep learning model. Panels B-C depict alpha helix, beta-sheet, respectively, and show a comparison of the predictions against ground truth. Blue dots denote the training data and red dots indicate the testing data. The distribution of regression results is close to the diagonal line, suggesting good agreement. The evolution of the error during the training process reveals that the MSE loss decreased extremely at the beginning of the training process and then converged to a rather low loss after 60 epochs, indicating convergence.

Predictions and validation

To test the end-to-end model against other protein sequences out of our dataset, we chose several large SARS and COVID-19 proteins that were recently deposited in the Protein Data Bank and fed these protein sequences to the deep learning model [47-51]. Table 1 shows several protein sequences from PDB sources, the proportion of alpha helix and beta sheet ratios predicted by our deep learning model. The results demonstrate that the proportion of various secondary structures calculated using other tools, such as PredictProtein, agrees reasonably well with our model and the PDB results.

Table 1:

The table reports an analysis of various different protein structures in the Protein Data Bank (PDB) with a range of sequence lengths and secondary structures. The charts in the right column report the alpha helix and beta sheet ratios (PDB ground truth vs. earlier model PredictProtein [54] vs. our new deep learning model, marked ‘ML’). The table includes newly deposited proteins that were not yet available during training, assessing the predictive power against new experimental data.

Systematic sequence variation for protein design

As a simple case study, we take the sequence of hen egg-white lysozyme (PDB ID: 194L):

KVFGRCELAAAMKRHGLDNYRGYSLGNWVCAAKFESNFNTQATNRNTDGSTDYGILQINSRWWCNDGRTPGSRNLCNIPCSALLSSDITASVNCAKKIVSDGNGMNAWVAWRNRCKGTDVQAWIRGCRL

This starting sample sequence is predicted to feature 29.99% alpha helical and 6.7% beta-sheet. The PDF entry features 38% alpha helix, and 3.1% beta-sheet, which is close. Using a replacement algorithm, we now systematically replace all amino acids of a certain type in the sequence with all other possible amino acids. For example, we start with A - and replace all residues with that amino acid type with other 20 natural amino acids (in the order AGILPVFWYDERHKSTCMNQ where A=0, Q=19 I the associated plots depicted in Figure 5). This results in 20 new sequences where in the first one all A residues were replaced with A (same sequence as original since we are replacing A with A), in the second one all A residues were replaced with G, in the third one where all A residues were replaced with I, and so on. This process is repeated in a nested loop for all amino acid types, resulting in 20×20=400 new sequence candidates (noting that the diagonal sequences are identical but included for completeness, so that there are actually only 380 new sequences).

Figure 5:

Protein design example, showing the effect of systematic variation of point mutations on alpha-helix and beta-sheet contents. Panel A shows the original sequence of lysozyme (PDB ID: 194L), and an image of the molecular structure of the wildtype protein as found in nature. Panel B shows the effect of systematically substituting amino acids of a certain type in the entire sequence (left: AH content, right: BS content). In the plot, the substation numbers range from 0-19 and reflect the sequence of substitutions from AGILPVFWYDERHKSTCMNQ (i.e., A=0…Q=19). As we go from top to bottom in each column, the plot indicates how the secondary structure content changes if all A are replaced with A, then G, then I, and so on. As we vary the columns the residue type that is replaced is varied. In the first column all A residue types are replaced, in the second column all G residue types are replaced, then I, and so on. As the plots show, while the protein remains largely alpha-helical for most changes, there are a few sequence mutations that lead to significant changes in the protein secondary structure content. These max/min results are extracted using a min/max algorithm and then folded using AlphaFold2, and depicted in panel C. The changes in secondary structure is clearly visible, confirming the predictions from our model and the optimization scheme used here.

Figure 5 plots the results of how these replacements affect the alpha-helix (AH) and beta-sheet (BS) content. We then analyze this data and identify mutations that lead to the greatest variations, e.g., the highest/lowest alpha-helix or beta-sheet content.

The lowest alpha-helix content is achieved by replacing A residues with I, which yields an alpha-helix content of 8.8% and a beta-sheet content of 33.97%. The sequence is:

KVFGRCELIIIMKRHGLDNYRGYSLGNWVCIIKFESNFNTQITNRNTDGSTDYGILQINSRWWCNDGRTPGSRNLCNIPCSILLSSDITISVNCIKKIVSDGNGMNIWVIWRNRCKGTDVQIWIRGCRL

The highest alpha-helix content is achieved by replacing D residues with M, which yields an alpha-helix content of 56.73% and a beta-sheet content of 35%. The sequence is:

KVFGRCELAAAMKRHGLMNYRGYSLGNWVCAAKFESNFNTQATNRNTMGSTMYGILQINSRWWCNMGRTPGSRNLCNIPCSALLSSMITASVNCAKKIVSMGNGMNAWVAWRNRCKGTMVQAWIRGCRL

The lowest beta-sheet content is achieved by replacing D residues with L, which yields an alpha-helix content of 51.45% and a beta-sheet content of 3.05%. The sequence is:

KVFGRCELAAAMKRHGLLNYRGYSLGNWVCAAKFESNFNTQATNRNTLGSTLYGILQINSRWWCNLGRTPGSRNLCNIPCSALLSSLITASVNCAKKIVSLGNGMNAWVAWRNRCKGTLVQAWIRGCRL

The highest beta-sheet content is achieved by replacing A residues with V, which yields an alpha-helix content of 10.9%% and a beta-sheet content of 36/38%. The sequence is:

KVFGRCELVVVMKRHGLDNYRGYSLGNWVCVVKFESNFNTQVTNRNTDGSTDYGILQINSRWWCNDGRTPGSRNLCNIPCSVLLSSDITVSVNCVKKIVSDGNGMNVWVVWRNRCKGTDVQVWIRGCRL

Figure 5C shows folded predictions of these new sequences based on AlphaFold2 [5,52], confirming the changes in secondary structures predicted by our model, and obtained from the optimization steps sought out here. As we seek to minimize the alpha-helix content, the resulting proteins feature more beta-sheet content and alpha-helices disappear. A similar behavior is observed for the case where a maximum beta-sheet content is sought, where alpha-helices disappear as well. When we search for variations with maximum alpha-helix content, the beta-sheets disappear, and the protein resembles a closely packed alpha-helix design. Similar behavior is seen when minimizing beta-sheet content. This example demonstrates the utility of our model to provide design suggestions for novel proteins and illustrates that the new designs are anticipated to fold as predicted.

The potential of the method for explorative design, other than looking just for min/max cases, is demonstrated in Figure 6, plotting the distribution of alpha-helix and beta-sheet content in the lysozyme-based protein variations described in the previous paragraphs. This plot shows that the variations explored in that case study result in a variety of potential target secondary structure content levels that can be explored in protein design. The mean values of the two peaks align well with the wildtype structure of lysozyme, as expected.

Figure 6:

Distribution of alpha-helix and beta-sheet content in the lysozyme-based protein variation results shown in Figure 5. This plot shows that the variations explored in that case study result in a variety of potential target secondary structure content levels that can be explored in protein design. The mean values of the two peaks align well with the wildtype structure of lysozyme.

Future work could explore experimental synthesis of such protein designs, and further analysis of the resulting properties – which can include mechanical, or enzymatic. We also anticipate that tool developed here could be used in conjunction with other protein property prediction methods to assess a range of potential target functions.

3. Discussion and Conclusion

Our results show that the end-to-end approach reported here is a powerful tool to relate primary sequences to secondary structure content, and ultimately, function. Our demonstrations of performance assessments of small and large proteins, as shown in Table 1, reveal excellent agreement with experimental data. The additional explorations shown in Figure 5 illustrate how the method can quickly explore vast search spaces and identify new sequence candidates to evolve proteins towards a desired property space. We hence envision that our tool, given its simplicity and effectiveness, can be adequately applied in an optimization scheme, for instance, using genetic algorithms to simulate evolutionary changes [53].

Generally, the problem solved in protein folding is that the primary structure is used to predict the folded protein structure. There are several existing software or structural analysis tools to predict protein structures. For comparison and validation, we used one of the prediction tools, PredictProtein [54] by Rostlab et al., as an alignment template for comparison. This approach incorporates over 30 tools to predict protein structures and derive information such as secondary structure content. In the procedure implemented in this prediction server, homologous sequences are searched in the sequence library Swiss-Prot [55]and TrEMBL [56] first, then multiple sequence pairings are performed, and finally, the paired results for structure prediction are attained through deep learning. Our model is different from this approach. We do not need to use a set of different tools and databases. Instead, we only feed sequences into our model and directly predict the content of the secondary protein structure, offering a novel and direct way to predict structural features and other properties, from a de novo end-to-end perspective. This enables a quick search for novel sequences and associated properties, as outlined in Figure 5. As the results depicted in that figure show, the model is capable of predicting the secondary structure content of completely new proteins (for which no templates exist) quite well, as evidenced by comparing with the de novo folding predictions, and can explore a wider array of secondary structure variations (Figure 6). A mode detailed analysis of such predictions and general validity, including experimental validation, is left to future work.

Predicting structural and mechanical features of proteins based on the sequence is an essential step in rational protein design. Utilizing an end-to-end approach to solve this challenge, our tool offers a rapid computational means for design iteration and target assessments. Furthermore, since secondary structures are particularly critical for mechanical functions [11,13,38], the approach reported here offers novel means to predict estimates of mechanically relevant proteins. Such predictions are beneficial in the designs of materials with improved mechanical properties, changes in degradability in vitro and in vivo, binding efficacy for molecular recognition or diagnostics, and many related applications.

4. Materials and methods

Data statistics

The dataset used for this work was developed from the Protein Data Bank (PDB) [57], following previous work from our group [6]. This database contained 125,955 protein sequences, recorded protein ID, length, primary structure, and secondary structure which was labeled afterward. Protein primary structures contain linear sequences of amino acids composed of different letters, which represent different natural amino acids and the other residues. The segments of secondary structure and folding structure can be predicted by the array of amino acids. The distribution of the input data can be observed in Figure 2. In the dataset, the average length of the protein sequence is 644 amino acids. The shortest protein is composed of only 11 amino acids, along with the longest one that features up to 19,350 amino acids. The standard deviation of the length is 855 amino acids.

The data distribution of alpha-helix and beta-sheet ratios is depicted in Figure 2(b) and (c). The beta sheet content is less than 30% in most sequences and even features about 20,000 sequences under 10%. On the other hand, the alpha helix ratio is typically higher than the beta sheet ratio. Most of the sequences feature an alpha helix content between 30% and 50%, but a small part of the sequences has even more than 80% alpha helix content. Also, of note is the fact the data shows a high proportion of sequences with a low ratio under 5% of both alpha helix and beta sheet. Because there are many sequences in this database, each has a different length, structural organization, and secondary structure content, so we can quickly analyze the relationships between primary structure and the secondary structures of the different sequences.

Data preprocessing

In order to analyze protein sequences effectively, tokenization was adopted in the present study by using the Keras [58] Tokenizer module [59] to facilitate segmentation for each amino acid. The primary structure of a protein is represented by 20 different single letter code for numerical convenience. Tokenization is a common way to pre-process sequences before feeding them into a ML/DL model. The tokenizer can separate the entire protein sequence into letters and transform each letter into a numeric digit for further computation.

Data labeling

The Protein Data Bank data was labeled using the widely used Define Secondary Structure of Protein (DSSP) [60,61] in order to determine the ratio of alpha helix and beta sheet, to determine ground truth data for training. DSSP is a quasi-standard algorithm for classifying secondary structure conformation of amino acid residues in protein structure, which predicts the secondary structure ration by directly obtaining residues of protein sequences. The DSSP algorithm uses the atomic-level resolution of protein in three-dimensional structure coordinates in the PDB format. It relies on hydrogen bond recognition based on the electrostatic definition and the calculation of the main chain and side-chain dihedral angles to obtain the information of each amino acid residue conformational parameters of certain secondary structure. The data was labeled before they were fed into our model for training or testing.

Deep Learning Model

As summarized in Table 2, the deep learning model is an end-to-end regression model to predict protein secondary structure ratio. Moreover, both language data and protein sequence have the same characterization, which is sequential order. As a result, we adopt an NLP-like structure [62] to construct our model, which includes an embedding layer, a batch normalization layer, 1D convolutional (Conv1D) layers, Bi-directional Long Short-Term Memory Units (BiLSTM)[63] units, a GaussianNoise layer, and several dropout layers and dense layers.

Table 2:

Overview of the deep neural network model proposed here, featuring an embedding layer to process the amino acid letters as a natural language model, a batch normalization layer, 1D convolutional (Conv1D) layers, Bi-directional Long Short-Term Memory Units (BiLSTM) units, a GaussianNoise layer, and several dropout layers and dense layers for the final prediction of the secondary structure content.

Layer	Parameter
Embedding	345
Batch Normalization	60
Convolution 1D	38528
Convolution 1D	81984
Bidirectional LSTM	66048
Bidirectional LSTM	41216
Dense	33280
Dropout	0
Dense	65664
Dropout	0
Dense	8256
Dropout	0
Dense	1040
Gaussian Noise	0
Dense	34

Total parameters: 336,455

Trainable parameters:336,425

Non-trainable parameters:30

First, we process the sequence data through a Tokenizer into an embedding layer with 15 embedding dimensions to change our data from characters to vectors, as common in natural language models. Then the data pass through a batch normalization layer for numerical convergence. Next, the data flows into two Conv1D layers to capture a series of protein sequential features and associated hierarchical patterning. After passing through Conv1D layers, data is feeds into two BiLSTM units to extract sequential relations of proteins bidirectionally. Next, we use a GaussianNoise layer for denoising. Finally, all data flows through several fully connected and dropout layers to prevent our model from overfitting. The last layer of the model is a dense layer with scalar values as output that represents the two content of the two secondary structures predicted here.

Training and Testing

The training and testing dataset is split into 80% of training and 20% of testing sets. All training and testing processes are conducted with one GeForce RTX 2080Ti GPU. We implement the deep learning model in TensorFlow [64] with Keras, a high-level application interface (API). We set a batch size of 512 and adopted an Adam optimizer to train the model and measure the performance of the model prediction by the mean square error (MSE) loss.

Evaluation and Validation

To compare the difference between labels and ratios that we use the model to predict, the coefficient of determination (R² score) was picked to measure the model's performance. The R² score, which represents the prediction capacity of a regression model, is determined by:

$$R^2=1-\frac{{\sum }_i(y_i-\widehat{y}{)}^2}{{\sum }_i(y_i-\overline{y}{)}^2}$$ (1)

where y_i is the label of each data as the ground truth, $\overline{y}$ denotes the average of y_i, and $\widehat{y}$ is the value of prediction.

Protein folding

Newly designed sequences are folded using AlphaFold2[5,52] , a state-of-the-art folding method. AMBER is used to relax the predicted structures, and the algorithm is executed using ColabFold [65]. No templates are used to assess de novo predictions. Instead, the single sequence MSA mode is used as it is most suitable for de novo proteins and allows for a relevant comparison of the wild type vs. mutants. Images of proteins are visualized using PyMol [66].

Figure 1.

Hierarchical architecture of proteins and relevant overarching features of a protein molecule, here focusng on alpha-helix and beta-sheet content. In the formation of proteins, the primary structure controls the folding of secondary structures or proteins (panel A). In this work we use the primary structure to predict secondary structure content, specifically the alpha helix and beta sheet content, as depicted in panel B, towards engineering functions of proteins and protein materials. One particular example examined in this paper is the optimization of secondary structure content – e.g., maximum/minimum alpha-helix or beta-sheet content.