NMR Spectroscopy for the Validation of AlphaFold2 Structures

Abstract

The introduction of AlphaFold has fundamentally changed our ability to predict the structure of proteins from their primary sequence of amino acids. As machine learning (ML) and artificial intelligence (AI) based protein prediction continues to advance, we examine the potential of hybrid techniques that combine experiment and computation that may yield more accurate structures than AI alone with significantly reduced experimental burden. We have developed heuristics comparing N-edited NOESY spectra and AlphaFold predicted structures that seek to determine whether the predicted structure reasonably describes the structure of the protein which generated the NOESY. We present a large collection of data connecting entries across the BMRB, PDB and AlphaFold Database that includes experimentally derived structures and corresponding spectra, establishing it as a means to develop and test hybrid methods utilizing AlphaFold and NMR spectra to perform structure determination. These data test the new heuristics’ ability to identify inaccurate AlphaFold structures. A support vector machine was developed to test the consistency of NMR data with predicted structure and we show its application to the structure of an unsolved engineered protein, LoTOP.

1. Introduction

The rise of big data and deep learning methods have fundamentally changed the way in which we pursue science. For example, it has long been a goal in biochemistry to predict the 3D structure, or fold, of a protein from its primary amino acid sequence. [1, 2]. AlphaFold [3], a deep learning model for exactly this, has revolutionized our ability to make such predictions. This was first demonstrated in a community experiment called the Critical Assessment of Structure Prediction (CASP), which compares computational methods for determining protein structures from their amino acid sequences. AlphaFold2 competed in CASP-14 where it was found to be ‘competitive with experimental accuracy’ [4].

While AlphaFold2 is a powerful system, the protein folding problem is not completely solved, nor has deep learning been shown to be entirely self-sufficient. For example, AlphaFold2 is known to fail on certain point mutations [5] and unusual sequences [6, 7], cannot model proteins that have multiple stable structures [8], does not model the dynamics of systems in solution or reactions [9], and struggles with complexes and multimers [10]. AlphaFold2 does work to mitigate its own potential pitfalls by augmenting its predictions with a model of the uncertainty involved in those predictions [3]. In many of these areas, new systems have been developed from the AlphaFold framework. Recently, AlphaFold-Multimer [11] and AlphaFold3 [12] were introduced to tackle the challenges involved in predicting protein complexes. The success and predictive power of AlphaFold2 have led to a range of competitors. For example, RoseTTAFold is a competing neural network based method that has shown similar success [13]. We expect that the methods presented here using AlphaFold2 will be relevant to all future prediction methods.

Understanding the structures predicted by AlphaFold, including when they are accurate or inaccurate, has become a focal point for those studying protein structure determination. We seek to generate hybrid methods that combine traditional experimental methods and current deep learning methods to balance the trade offs between the convenience of deep learning models and time consuming and expensive experimental structural determination. We propose to combine nuclear magnetic resonance (NMR) experiments with deep-learning models to check and refine the modeled protein structures. This paper concentrates on using Nuclear Overhauser Effect Spectroscopy (NOESY) to validate structures through the pairwise atomic distance restraints it imposes [14]. Even in settings with reduced information about the nature of the peaks generated in a NOESY spectrum, the experimental data can be used to determine likelihood of structure quality [15], and these methods inspire our work. Other NMR experiments which probe characteristics such as rigidity [16], bond torsion angles[17], and residual dipolar couplings can also be used to validate structures and will be the subject of future work.

This paper focuses on experimentally validating predicted protein structures with constraints derived from easily acquired NMR spectra; i.e., the ¹⁵N edited NOESY spectrum. In order to quantify the agreement between predicted structure and NMR data, two heuristics have been developed; the Contact and Distance Scores, CS and DS, respectively. These heuristics are based on the idea of ‘potential contacts’, as described fully in section 2.2. These heuristics are tested on a few example structures to demonstrate their feasibility. We have mined data from multiple sources to compile a dataset that contains proteins with experimentally derived structures, experimental NMR parameters, and AlphaFold2 predicted structures. This data collection is an important tool for creating and testing hybrid methods and forms the basis of this and future work. It allows us to test our heuristics over our current state of knowledge. We introduce a predictive model, called Structural Prediction Assessment by NMR (SPANR), that was trained on our dataset and tests the accuracy of the AlphaFold predicted structures relative to the experimentally derived structures. We demonstrate SPANR’s ability to distinguish high-quality predictions and use SPANR to test the AlphaFold structure of a novel engineered protein, LoTOP.

2. Computational Methods

2.1. Collecting Existing Data

Publicly available archived data were collected and used in this analysis. There were three primary sources of these data: the Biological Magnetic Resonance Data Bank (BMRB), the AlphaFold DB, and Protein Data Bank (PDB). These sources provided NMR assignments, predicted protein structures, and NMR solved protein structures, respectively, and are the primary inputs to our data pipeline, shown as the top row in figure 1. We explicitly eliminate membrane proteins, where high-quality NMR data are difficult to acquire, and proteins measured in 8M urea solution.

Fig. 1:

Visualization of the flow of data and analysis. Our model compares spectra from the BMRB to structures predicted by AlphaFold using the heuristics described in section 2.2. These heuristics and the predicted uncertainty from AlphaFold feed a small support vector machine which predicts the quality of the AlphaFold structure. The true quality of the AlphaFold structure is determined by the TM-score between the AlphaFold structure and the experimentally derived structure in the PDB.

2.1.1. BMRB

The Biological Magnetic Resonance Data Bank (BMRB) supplied our experimental NMR data [18]. The BMRB is a repository for spectroscopic results obtained in magnetic resonance experiments, and contains experimental NMR spectra, lists of assigned chemical shift values for individual atoms, as well as some lists of peaks directly from the spectra. Data are archived in STAR format, which is a dictionary-based method for data exchange commonly used in molecular structure sciences [19, 20].

These files were screened for lists of assigned chemical shifts and NOESY peaks; however, the majority of data reported do not contain the NOESY data. Excluding files without NOESY data would have produced a dataset that was too small to produce reliable results. For this reason, we opted to simulate all NOESY spectra using the assigned chemical shifts and the PDB structure. This process is described in section 2.1.3.

2.1.2. PDB

The NMR data acquired from the BMRB are associated with experimentally derived structures in the Protein Data Bank (PDB), the primary repository for all 3D structures of proteins, nucleic acids, and complexes [21]. AlphaFold also obtained its training data from the PDB.

We have focused on protein structures that were solved by NMR spectroscopy. Thus, if the BMRB file lists only PDB structures that were solved using crystallography or another non-NMR method, we discard this structure from our dataset. Even though AlphaFold was primarily trained on protein structures derived from X-ray diffraction data, we decided to compare its ability to find the structure in solution as determined by NMR spectroscopy where artifacts due to effects such as crystal packing are absent. PDB files provide the atomic coordinates of each atom in an XYZ format. When the positions of hydrogen atoms were excluded from the pdb files, RDKit was used to calculate their positions [22]. We assume that the NMR determined PDB structure is the ground truth for each protein in our dataset.

2.1.3. NOESY Simulation

From the list of assigned chemical shifts and an experimentally determined structure, we simulate the peaks from an ¹⁵N-edited NOESY experiment, Algorithm 1. The algorithm loops over all backbone amide hydrogens as the ‘base’ hydrogens and calculates the distance to all other hydrogens. We add a peak to the spectrum with a probability that is related to the inverse of the distance to the 6th power, as shown in line 12. The inverse 6th power is related to the intensity of the NOE as described in SIMPSON[23].

This simulation algorithm allows us to approximate the list of peaks we might see in a 3D ¹⁵N-edited NOESY spectrum. However, it does not require the actual list of NOESY peaks or any parsing of the BMRB file beyond the assigned chemical shifts, which are the most reliably available and easily parsed section of the BMRB files. We are able to do these simulations because we have identified a ground truth 3D structure from the PDB. We use this simulation as a final check of the validity of the experimental data, removing any structures for which less than half of the residues have generated a potential contact, as this data is not informative enough to make conclusions about the full structure.

2.1.4. AlphaFold DB

The predicted structures were collected from the AlphaFold Database, [24], which contains over 200 million predicted structures. Each entry in the database consists of a single predicted conformer for its amino acid sequence. Each record in the database also contains the ”predicted aligned error” (PAE), which is a matrix that describes the uncertainty in each pairwise distance between residues.[3]

We correlate the AlphaFold DB entries with the BMRB files using the UniProt accession code, which is the protein sequence identifier used by AlphaFold DB. For each structure, we obtain the AlphaFold2 best predicted structure and the PAE. We verify that the amino acid sequence in the AlphaFold DB matches the amino acid sequence in the BMRB file. Using the AlphaFold DB to collect predicted structures allows us to examine larger amounts of data than would otherwise be possible.

2.1.5. Data Collection Summary

Our final dataset has 3085 structures, each with an experimentally derived ground-truth PDB structure, an AlphaFold predicted structure, and assigned NMR chemical shifts from which we simulate a list of NOESY peaks. In the rest of this paper we will refer to this dataset as the ”original” dataset, as it hasn’t been augmented with other data.

2.2. Analysis Techniques

We have developed a set of heuristics that take as input a structure and a list of NOESY peaks and output a single value intended to quantify the likelihood that the structure could have generated the NOESY peaks. The input structure can derive from any source: structurally determined by X-ray crystallography, NMR, or cryo-EM, or predicted by AlphaFold or any other tool. The heuristics can be applied to almost any type of protein for which experimental NMR data exist, and even unstructured regions of proteins can give useful insight for distinguishing high-quality from lower-quality structures by providing negative examples.

The quality of the predicted structure is quantified by calculating the TM-score[25] between it and the PDB structure. We judge our heuristics to be useful if they correlate well with the TM-score. Most powerfully in terms of efficiency, these heuristics are calculated without explicitly assigning the NOESY peaks. This is done using a set of ‘potential contacts’, which is described in section 2.2.1. Potential contacts feed into the two heuristics, CS and DS, described in sections 2.2.2 and 2.2.3, respectively.

2.2.1. Potential Contacts

Potential contacts are derived from the ¹⁵N-edited NOESY spectrum and the ¹H assignments. The ¹⁵N-edited NOESY spectrum provides ¹H chemical shifts of hydrogens that are near the hydrogen of the assigned amide, HN. Due to the extensive overlap of the ¹H chemical shifts and the limited resolution of the spectra, only potential contacts between HN and the other hydrogen atoms can be determined. A potential contact is determined if the chemical shift difference between the NOESY peak and its possible assignment is less than ${\Delta }_{shift}$ ppm. Thus, each off-diagonl NOESY peak provides a list of potential contacts to the residue assigned to the given HN peak, Figure 2.

Fig. 2:

Potential Contacts are generated by comparing the chemical shifts of the assigned H’s with the positions of the off-diagonal peaks in the NOESY spectrum. a) shows the chemical shifts of the HN and N of V59 of a protein. b) shows an ${\delta }_N=127.08$ ppm plane of an ¹⁵N-edited NOESY spectrum. ¹H peaks along the red line at 9.34 ppm are either the HN peak of V59 or due to H’s near to it. One such peak is ta 2.05 ppm. c) Several H’s have shifts near to 2.05 and are potential contacts to V59.

This concept has been used in a similar fashion for previous validation methods. Although these previous methods do not create the potential contacts in the exact same way, we test their metrics using our own potential contacts. In particular, we use the recall, precision, and F-measure (RPF) from Huang et al..[15] These metrics were included as RPF inputs in the results shown in section 3.3.2. In the next two sections, we will describe the algorithms for our own methods of using these potential contacts to assess the quality of a predicted structure.

2.2.2. Contact Score

CS compares the interresidue contacts in the predicted structure to those predicted by the NMR data. It concentrates only on long-range contacts; i.e., those whose primary sequence separation is greater than 4 residues. As described above, NOESY peaks in the N-edited NOESY spectrum provide potential contacts which are compared to those seen the predicted structure. Scoring these potential contacts both in terms of the number of contacts and their agreement with their assignment determines CS.

This procedure is illustrated in Figure 3A and detailed in Algorithm 2. The spatial distance between ${\text{C}}_{\alpha }$ of residues identified by the potential contacts is determined from the proposed structure and compared to an upper cutoff threshold ${ϵ}_{CS}$. ${ϵ}_{CS}$ was treated as a hyperparameter and optimized by a grid search, which determined that for two residues to be ”in contact” their ${\text{C}}_{\alpha }\text{s}$ had to be within 12 Å. The grid search optimized the correlation between CS and the TM-score, with full details in the SI. If the spatial distance between two hydrogen atoms in residues i and j is less than ${ϵ}_{CS}$ and these residues are separated by at least 5 amino acids, are deemed to be ”in contact” and a score is calculated. This score measures the interresidue contact weighted by the chemical shift agreement. It is calculated from the chemical shift difference, ${\delta }_{PC}$, and the chemical shift tolerance, ${\delta }_{measure}$, such that a score between 0 and 1 is assigned for every contact.

Fig. 3:

Calculation of Contact and Distance Scores. The HN of residue $i$ is seen to have a NOESY cross peak at a shift of 2.05 ppm. All potential contacts with a residue separation of 4 or more are shown as residues $j$,$k$,$l$, and $m$. A. The CS is calculated by considering all residues whose ${\text{C}}_{\alpha }$ distance is less than the cutoff ${ϵ}_{CS}$. In the final CS, the $ik$ NOE will contribute the most as its chemical shift agreement is best, as indicated by the bolder blue arrow. B. In the DS, only the potential contact with the shortest HN to H distance is considered, indicated by the blue arrow between $i$ and $j$. If the interresidue distance is greater than ${ϵ}_{DS}$, it is counted towards the final DS.

The scores for individual interresidue contacts are total and normalized by the maximum number of contacts with a separation of at least five residues, given by $n_{resi}^2+-9n+20$, where $n_{resi}$ is the number of residues compared. CS has been designed to be high when the structure and data are good matches. In this case, most of the off-diagonal peaks in the NOESY spectrum due to long-range contacts are assigned to contacts seen in the proposed structure and contribute to CS. Higher confidence in the assignments and more observed NOESY crosspeaks lead to a higher value of this score. Therefore, a high CS should correspond to a good structure with a large number of reliably assigned NOESY crosspeaks.

2.2.3. Distance Score

The calculation of DS is illustrated in Figure 3B and is detailed in Algorithm 3. DS is the fraction of NOESY crosspeaks that cannot be assigned by the proposed structure. To calculate DS, all potential contacts for a single peak are reduced to the single HNH pair that has the smallest separation according to the proposed structure. If this separation is greater than the threshold ${ϵ}_D$, the peak is counted in the DS score. This indicates that that peak cannot be reasonably explained by the proposed structure. As for CS, only long-range contacts are considered. The final DS is the ratio of crosspeaks that cannot be explained by the proposed structure to the total number of cross peaks. The closer DS is to 1, the worse the agreement between the NMR data and the proposed structure.

The threshold ${ϵ}_D$ is selected separately from the contact distance threshold ${ϵ}_{CS}$ used in CS. ${ϵ}_{CS}$ is the maximum distance between the $\alpha$ carbons for two residues to be considered ‘in contact’ while ${ϵ}_D$ is the maximum $\text{C}\alpha$ distance between two residues that can give rise to an observable NOESY peak. ${ϵ}_D$ was determined via a grid search to optimize the correlation between the DS and TM-score, as detailed in the SI. It was found to be 10 Å.

Filtering the potential contacts to a single contact distinguishes DS from CS. CS considers all potential contacts weighed by their assignment certainty, while DS reduces each peak to a single potential contact that best represents the peak. Because of the r⁻⁶ dependence of the NOE intensity, the chosen potential contact used in DS is expected to make the primary contribution to the cross peak.

3. Results

3.1. Testing CS and DS on example structures: Aq1974 and pro-IL-18

To highlight the usefulness of CS and DS, we discuss their application to two example proteins, both of which are known to be difficult to model even with AlphaFold, but their structures have been determined by NMR spectroscopy. By comparing the values of CS and DS on proteins of known difficulty for AlphaFold prediction, we can begin to gain some insight into their value.

Aq1974 (PDB ID: 5SYQ) is a small protein of unknown function (puf) from Aquifex aeolicus with an unusual amino acid sequence that has an exceptionally high aromatic content, 14% of the folded region of the protein. Due to its unusual primary sequence, the CASP-10 competition found that the structure of Aq1974 is difficult to predict.[7] It is also challenging for AlphaFold2. AlphaFold2’s internal reliability testing algorithms(MSA coverage, alignment error, and LDDT) all warn that this prediction is probably not reliable. Another protein that has been found to be a difficult target for AlphaFold2 is Pro-interleukin-18 (pro-IL-18, PDB ID: 8urv), which is the precursor form of the cytokine IL-18. Kay and coworkers have shown that its 3D structure as determined by NMR disagrees significantly from that predicted by AlphaFold2[6]. We use these structures to test our new heuristics.

To examine the scaling of CS and DS with prediction accuracy, we chose one of the NMR models of both Aq1974 and pro-IL-18 as their reference model. TM-scores were then calculated with respect to these reference models. The average value of the TM-score in the NMR structural ensemble for Aq1974 is 0.860 ± 0.037 while that for pro-IL-18 is 0.905 ± 0.010. In the AlphaFold2 prediction of the structure of Aq1974, there are two classes in the top five models. The first class, colored orange in Figure 4, contains two members and gave an average TM-score of 0.587 ± 0.005. The remaining 3 predicted structures, colored red, gave an average TM-score of 0.327 ± 0.017. AlphaFold2 performed equally poorly for all its predictions of pro-IL-18, where the average TM-score with the reference structure of the five predicted structures is 0.510 ± 0.001.

Fig. 4:

Contact Score (CS) and Distance Score (DS) as functions of TM-score for Aq1974 and pro-IL-18. A) CS and DS as a function of TM-score for Aq1974. Blue points are from the PDB structure where the first model structure was used a the reference model. Also included are the first model from the PDB structure (blue) and two AlphaFold models: orange was the best alpha fold model while while the red models is an example where the TM-score was about 0.35. The dashed yellow lines are the assigned long range interresidue connections seen in the NMR data and identified by comparing with the structure. Red and orange points correspond to red and orange models. B) CS and DS as a function of TM-score for pro-IL-18. AlphaFold2 only predicted one class of incorrect models labeled red.

Figure 4 shows example models of Aq1974: NMR determined structure colored blue, the best AlphaFold model colored orange, and an example of the second group of poor AlphaFold models colored red. The dashed yellow lines on the structural models indicate the identified long-range interresidue contacts from the comparison of the NMR data with the proposed structure. The number of identified contacts decreases as the quality of the model decreases. This correlates directly with CS and DS. For both Aq1974 and pro-IL-18 show monotonic relationships between CS, DS, and TM. CS increases with increasing values of TM while DS decreases in both of these structures. This indicates that these scores are useful in comparing structures to NMR data.

3.2. Testing Heuristics on the Compiled Dataset

Although examples like Aq1974 and pro-IL-18 demonstrate that CS and DS correlate with TM-score for these cases, we need to demonstrate that they are useful in a broader context. CS and DS were calculated over the entire compiled dataset of 3085 structures described in section 2.1. To increase the number of poorly predicted structures, the structure and NMR data of each member of the original dataset were associated with an AlphaFold prediction taken from another randomly chosen member of the dataset. These randomly chosen predictions are usually quite different from the experimentally determined structure, increasing the number of poorly predicted structures. We call the dataset that includes these ‘mismatched’ structures the ‘augmented dataset’ and it contains 6170 entries. It provides a useful way of examining the behavior of CS and DS on a wider variety of predicted structures than AlphaFold could typically provide.

Figure 5 shows CS and DS as a function of TM-score on both the original and mismatched datasets. Each point in the scatter plots is colored by whether the sample comes from the mismatched (red) or the original (blue) data. The augmented dataset is the union of these two sets. This figure shows that in the original dataset contains more ‘good’ structures, while the mismatched experiment introduces more ‘poor’ structures, as indicated by the shift of the distribution between the blue and red points. This is expected from the success of AlphaFold; however, there are still a significant number of structures in the original dataset (blue) whose TM-scores are less than 0.5, indicating problematic predictions. While the observed correlation between CS and DS with the TM-score is noisy in Figure 5, it is clearly present, demonstrating the power of these heuristics.

Fig. 5:

CS and DS as a function of the TM-score over the entire compiled dataset. Blue data are from the original dataset, while red are from intentionally mismatched data. The number of structures in a region is indicated by the shade of red and blue: the darker the color, the more data represented. While there is considerable scatter in the data, it is clear that there is a positive correlation between CS and TM-score and a negative correlation between DS and TM-score over the entire dataset, indicated by the black curves.

Other measures of prediction quality and agreement between NOESY data and protein structure exist besides those defined here. For instance Recall, Precision and F-measure (RPF) for structure validation by NMR [15] generates sets of proton pairs from both an input structure and an input NOESY spectrum, then applies the notions of recall, precision and F-measure from information retrieval theory. In particular, recall is the percent of true HH pairs identified by the spectrum, precision is the distance-weighted percent of identified HH pairs that are true, and F-measure divides the product of the two by the sum. AlphaFold’s own internal uncertainty measure, known as the predicted aligned error (PAE), also predicts structure quality.

Figure 6 shows the Spearman’s rank correlation coefficients between all input parameters and the TM-score for the original and augmented datasets, upper and lower triangles, respectively. Spearman’s rank correlation coefficients are a nonparametric method of determining whether two parameters are monotonically dependent upon each other. The more typically used Pearson’s correlation coefficients assume that the parameters are linearly related and the data are Gaussian distributed, both of which are not necessarily true for our dataset. These assumptions are removed when using Spearman’s rank correlations.[26] Figure 6 shows that Recall, Precision, and F-measure are highly correlated with each other as one would expect from their definition. The CS is strongly correlated with Recall and F-measure and more weakly correlated with Precision. DS is primarily correlated with Precision and negatively correlated with CS, as was shown in the example cases above. While Recall and F-measure essentially measure the same characteristics of our dataset as indicated by a correlation coefficient of 0.99, the other heuristics manage to capture other relevant aspects of the data. These correlation data show the power of using the augmented dataset. RPF and CS are all expected to be positively correlated with the TM-score; however, in the original dataset, these heuristics are anticorrelated with the TM-score and in the case of CS the correlation is not statistically significant. Upon adding the mismatched data, these correlations become positive and, in the case of CS, highly statistically significant, p << 0.0001. These results indicate that the RPF scores perform as expected when identifying very good versus very bad structures but struggle when needing to distinguish structures that have fold similarity with the true structure but are not accurate. The addition of mismatched structures increases our confidence in correlations between our new heuristics and the TM-score by increasing the number of poor structures in the dataset. This demonstrates the power of adding these mismatched data to our analysis.

Fig. 6:

Spearman’s rank correlations between heuristics and TM-score. Large positive values are in red and large negative values are in blue. Correlations presented in the upper triangle are from the original dataset, while the lower triangle comes from the dataset augmented with mismatched experiments. The final columns and the final row give the correlations with the TM-score from the regular experiments and the mismatched experiments, the ground truth measure of quality in each experiment. The predicted aligned error (PAE) and distance heuristic have the highest correlations with the ground truth in the regular experiment, while both of our heuristics have a strong correlation in the mismatched experiment.

3.3. Structural Prediction Assessment by NMR (SPANR)

We developed a small machine learning method, specifically a support vector machine (SVM), to predict the quality of the AlphaFold structure based on our heuristics. This is shown in the left two columns of Figure 1. An SVM is a simple machine learning algorithm that, in this case, we have trained to place AlphaFold predicted structures into one of two categories: consistent or inconsistent with the NMR data. We call our SVM Structural Prediction Assessment by NMR (SPANR).

3.3.1. Training

Developing multiple SVMs allows us to determine our heuristics’ ability to experimentally test the consistency of AlphaFold2 predicted structures with experimentally determined data. Predicted structures were classified as ”consistent” if the TM-score between predicted and experimentally determined structures was greater than 0.5. Multiple metrics were used as inputs to the SVMs which were split into three groups RPF, PAE, and CS-DS. We trained seven SVMs on each of the seven possible combinations of these groups on the original dataset, see Table-section 3.3.2, allowing us to determine the effect of each combination on the accuracy of the SVM to categorize the predicted structure. To perform the training, we split each dataset into training and test sets with an 80–20 split, then used scikit-learn to fit the SVM on the training data [27]. We also trained an SVM on the augmented dataset using only the CS and DS as inputs.

3.3.2. Accuracy

Each of the seven SVMs that were trained on the original dataset was scored on the test set. Due to presence of many more well predicted structures in the original dataset, we consider the balanced accuracy to be the best metric for comparing SVMs. We are particularly interested in each classifier’s ability to identify models inconsistent with the NMR data, because false negatives are more tolerable in most structure determination cases; i.e., verifying a well-predicted structure is preferable to treating a poorly predicted structure as correct. The results are collected in Table 1. Of each single group input, CS-DS perform the best at identifying inconsistent structures. PAE performs best at identifying consistent structures, succeeding 87% on the test dataset, but only succeeding 40% in predicting inconsistent structures. This suggests that the predicted aligned error overestimates AlphaFold’s accuracy. CS and DS are capable of correcting these inaccuracies. The rate of identifying consistent structures dropped to 82% using just the DS and CS scores, but the rate of identifying poor structures increased to 67%. Using all inputs together gives the best results, resulting in a balanced accuracy of 84%, with consistent structures identified at a rate of 89% and inconsistent structures at a rate of 79% in the test dataset.

Table 1:

Comparison of accuracy of SVMs with different inputs on the binary classification task of identifying ”consistent” (TM-score > 0.5) and ”inconsistent” (TM-score < 0.5) structures using the original data set; i.e., those without the mismatched data. Negative and Positive ID rates refer to the percent of ”inconsistent” or ”consistent” structures correctly identified. The SVMs were all trained with balanced class weights, therefore we include the balanced accuracy to demonstrate how well each model has achieved the goal of equally identifying positive and negative samples. The types of inputs used are labeled in the first column. The bottom line shows the results from training on the augmented data set in which the SVM was given only the CS and DS as inputs.

Inputs	Accuracy (%)
	Neg ID	Pos ID	Total	Balanced
RPF only [15]	61.43	83.33	81.10	72.38
PAE only	40.00	86.73	81.98	63.37
CS-DS only	67.14	81.88	80.38	74.51
RPF-PAE	65.71	87.86	85.61	76.79
CS-DS-RPF	71.43	88.35	86.63	79.89
CS-DS-PAE	75.71	81.07	80.52	78.39
All Inputs	78.57	89.00	87.94	83.78
Augmented Data	92.07	90.23	91.21	91.15

Sci-kit learn has the option to predict the confidence of each class categorized by the SVM, rather than predicting the single most likely class. To calculate the confidence of the classes, a slightly different model is used than in the baseline SVM. Figure 7 shows the classification confidence of the SVM compared to the TM-score for every member of the test dataset. Figure 7A uses the CS-DS-PAE SVM over the original dataset while Figure 7B shows the results from the CS-DS SVM on the augmented dataset. The augmented data cannot use PAE as input due to the inclusion of mismatched data. The red dotted lines indicate predictions above 60% confidence in either class, consistent or inconsistent; greater than the red line at 0.6 indicates greater than 60% confidence the predicted structure is consistent with the NMR data and less than that at 0.4 means greater than 60% confidence that the structure is inconsistent. The SVMs show very high accuracy in these regions with improvements to above 90% for inconsistent results using the augmented data. In fact when the model is over 80% confident in a positive sample, it is 94% accurate. If the SVM produces a confidence in the range 40%−60%, the model is only 50%−60% accurate.

Fig. 7:

TM-score as a function of the confidence the data are consistent from our SVM models. Blue dots are TM-Score/confidence pair for each structural prediction. The horizontal blue line indicates a TM-score of 0.5 which is the value we chose for a good prediction of the true structure. The vertical red lines are at 0.4 and 0.6 confidence levels. A) SVM trained and calculated on the original dataset. B) Trained and calculated on the augmented dataset.

To test our SVMs ability to detect inconsistent structures, the CS-DS SVM trained on the original dataset was applied to a test set taken from just the mismatched data and found to have a total accuracy of 95%. A new CS-DS SVM trained on the augmented dataset using the same train-test split then analyzed the same test set from the mismatched data and was found to have an accuracy of 94%. Thus, training on the augmented dataset did not improve the SVMs ability to identify inconsistent models over the original SVM using the dataset without the mismatched data. We conclude that while mismatched samples are important to verify correct performance of the CS and DS on very poor samples, they do not help the SVM learn to separate consistent from inconsistent structures more than the original dataset.

3.4. Applications of SPANR

To demonstrate the power of SPANR, predicted structures and NMR data of two known structures, prot-IL-18 and Aq1974 which are not part of our training or test sets, were analyzed by SPANR. Additionally, we examined a designed protein, LoTOP, whose structure has not been previously determined with SPANR. These examples demonstrate the power of our SVM.

3.4.1. pro-IL-18 and Aq1974

SPANR analyzed the consistency of the predicted structures with NMR data of the proteins pro-IL-18 and Aq1974 discussed above. Using the best AlphaFold2 model for pro-IL-18 and entering its NOESY peak data, chemical shift assignments, and AlphaFold’s PAE file, SPANR finds that this structure is consistent with the NMR data with a 93% certainty. This means that with 93% certainty the TM-score between this structure and the ”true” structure is greater than 0.5. The TM-score between this predicted structure and model 1 of the NMR determined structure was found to be 0.51.

In the case of Aq1974, we can examine predicted structures from both classes shown in Figure 4. We find that SPANR identifies AlphaFold model 1 as consistent with a confidence of 88% and that of model 3 as inconsistent with a confidence of 69%. The TM-scores of the first model of the NMR determined ensemble to the first and third predicted structures were 0.58 and 0.31, respectively.

3.4.2. LoTOP

LoTOP is a fast folding variant of Top7 that was designed by permuting the secondary structure elements of Top7. Circular permutation is a typical method for creating a new version of an existing protein. This method alters the protein’s topology by linking the N and C termini and nicking one of the existing loops; unfortunately, the N and C termini are too far apart in Top7 to apply this method. Hence, two new archetypical beta-turn sequences[28] were introduced into Top7 to rewire its secondary structures into a new topology.[29] Instead of Top7’s ${\left(\beta \right)}_{\text{2}}\text{-H-}\beta \text{-H-}{\left(\beta \right)}_2$ order, the permutant “LoTOP” has a $\text{H-}{\left(\beta \right)}_5\text{-H}$ order, a simpler topology with a significantly lower relative contact order (RCO) (SI Figure 4). Although LoTOP is not strictly a new fold (e.g., bacterial, yeast and human Frataxins have the same gross fold but are $\text{H-}{\left(\beta \right)}_7\text{-H}$), LoTOP can be considered a new superfamily. Our first design attempt was successful. LoTOP retains Top7’s high stability (12 kcal mol⁻¹, SI Figure 5).

¹H¹⁵N HSQC spectra of LoTOP gave a high-quality spectrum with most HN peaks observable (SI Figure 6), indicating a stable folded structure. The spectrum was assigned by routine methods, resulting in 97% of the backbone shifts and 80% of all ¹Hs assigned. The product map between the potential contact map of possible assignments of the NOESY peaks and the AlphaFold predicted contact map for the structure was calculated. Figure 8A shows this product map for the highest confidence AlphaFold2 structure, where the color scale represents the score for a long-range interresidue contact as defined in section 2.2.2. CS is the sum of the off-diagonal elements of this matrix, shown as the yellow region of the map, normalized to the area of this region. The average CS over the 5 best AlphaFold structures is 0.117±0.001 while the average DS is 0.316±0.002. Comparing these values with the distribution of the scores over the augmented dataset, Figure 4, the TM-scores to the ”true” structure is expected to be greater than 0.8, indicating that the AlphaFold structures are a good representation of the ”true” structure. These data were input into our SVM that uses CS, DS and PAE and trained on the original dataset, where it finds that the NMR data are consistent with the proposed structure with a confidence of 97%.

Fig. 8:

Agreement between NMR data and AlphaFold2 predicted structure of LoTOP. A. Product of the potential contact map and the AlphaFold2 predicted contact map. Sum of the long range off-diagonal elements of this map (yellow region) gives the CS. B. The AlphaFold2 predicted structure with the NMR determined long range contacts.

4. Discussion

These results demonstrate the power of simple NMR experiments in testing AI-derived structural models. By measuring a limited set of NOESY constraints, just from the ¹⁵N-edited NOESY experiment, SPANR can distinguish structural models that are consistent or inconsistent with the data. To train SPANR, we developed CS and DS to quantify the agreement between the NMR data and the predicted structure.

Both CS and DS are highly correlated to the TM-score between the NMR-etermined and the predicted structure. Examining CS in more detail, we did find that its value is highly dependent upon the chemical shift uncertainty,${\Delta }_{shift}$ used in its calculation, making it a reliable measure of agreement between NMR data and structure for a single sequence, but noisy over a large dataset where different chemical shift uncertainties are expected. We have shown that our SVM has higher accuracy when trained with CS, demonstrating its usefulness. DS does not suffer from this dependence on chemical shift uncertainty. These quick and simple methods validate AlphaFold structures, and will serve as the basis for more complex validation and refinement methods.

CS and DS are valuble in inputs into SVMs to determine the consistence of NMR data with proposed structures. SPANR has been shown to be useful in performing this categoriztion. When SPANR determines the certainty of these categories is above 70–80%, the categorization accuracy is well above 90%. This allows us to use NMR data to decide if we need to further refine the proposed structure or not.

The results in this paper rely on the large dataset of NMR determined and AlphaFold2 predicted structures, and corresponding NMR data that we compiled. This dataset will be key in the development of novel tools to accelerate protein structure determination through the use of hybrid AI-experiment methods; however, the dataset does need improvement, which will occur by increasing both its size and the quality of its structural constraints. One improvement we made to this dataset was to included intentionally mismatched data. This significantly increased the number of poorly predicted structures in the augmented dataset produced. One major issue with the compiled dataset is its lack of NOESY constraints for many of its members. Due to this limitation, we had to calculate the expected NOESY peaks from the true structure and the assigned chemical shifts. An ideal dataset would include a comprehensive set of experimentally determined NMR structural constraints which could be used to train hybrid AI-experimental models. The main bottleneck to this is finding reported experimental NMR results, for which the BMRB is the largest source, but not necessarily the only. Besides finding other sources of data, adding new experimental results to the dataset as they are obtained will be important. The BioNMR community is developing tools that will help homogenize and collect newly captured experimental data [30] which will be crucial in this.

Ultimately, we hope that the recent advances in AI and machine-learning will increase our ability to solve structures by NMR quickly and accurately. The future of structural NMR will refining predicted structures based upon NMR constraints. We also foresee using these predicted structures to aid in assigning proteins without ¹³C labeling. Ultimately, the goal is to glean biological information in as efficient way as possible, which we see as hybrid methods combining AI with experiment.

5. Methods

5.1. LoTOP Expression and Purification

A single colony of E. coli BL21(DE3) C+ that carries the recombinant plasmid vector pET-21–8xHis-DesG-LoTOP was inoculated in 5 mL LB broth containing 1:1000 (Ampicillin 100µg/mL) overnight at 37°C on shaker at 225 rpm. This culture was added into 1L of M9 minimal medium containing 100ug/mL Ampicillin and supplemented with ¹⁵NH₄Cl (1g/ L) and incubated at 37°C until an OD600 of 0.6–0.8 was reached. The temperature was then lowered to 20°C, and the flask was cooled for 30 minutes. Following this, protein induction was carried out with 1mM IPTG, and the culture was incubated overnight at 225 rpm. The next morning, the induced expression culture was harvested by centrifugation at 5500rpm for 15 minutes at 4°C and the cell pellet frozen at −80°C.

The frozen cell pellet was resuspended in 50mM Tris-HCl pH 7.5, 150mM NaCl, containing a cocktail of protease inhibitors (Abcam ab27005). The cells were then lysed by sonication (Branson Sonifier 450) on ice. The cell debris was removed by centrifugation at 12,500 rpm, and 4°C for 30 minutes. The protein was then purified using metal affinity chromatography. The binding and washing steps were carried out in 50mM Tris-HCl pH 7.5, 150mM NaCl, and 20mM imidazole, and the protein was eluted with 250mM imidazole. The His-tag was removed by incubating overnight at 4°C with 6xHis-TEV (Tobacco Etch Virus) protease. Metal affinity chromatography was then used to remove the 8xHis-DesG tag and 6xHis-TEV protease from the solution. The protein was buffer exchange for long-term storage using a Superdex 200 column into 10mM Phosphate buffer at pH 7.0. Finally, for NMR studies, the buffer was exchanged, when needed, into different buffers depending on the pH using a G-25 Sephadex spin column.

5.2. NMR Spectroscopy

NMR spectra were acquired on Bruker AVANCE IIIHD 600 MHz NMR spectrometer equipped with a room temperature TXI probe. Spectra were acquired at a sample temperature of 37° C in 20 mM MES and 50 mM NaCl buffer at a pH 6.5. Assignments of U-[¹³C, ¹⁵N]-LoTOP were obtained by analysis of standard triple resonance NMR spectra. This resulted in the assignment of 96.8% of the backbone atoms and 79.6% of all ¹H atoms.

A. Appendix

$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathbf{\text{Algorithm}}\mathbf{1}\text{NOESY}\text{Simulation}}\\ 1:shifts=\left\{{\delta }_a\forall a\in atoms\right\}\\ 2:struct=\left\{\left(x_a,y_a,z_a\right)\forall a\in atoms\right\}\\ 3:\mathbf{\text{procedure}}\text{SIMULATE}\text{NOESY}\left(shifts,struct\right)\\ 4:peaks\Leftarrow \left\{\right\}\\ 5:\mathbf{\text{for}}a\in atoms\mathbf{\text{do}}\\ 6:\mathbf{\text{if}}a{\text{is}}^1\text{H}\text{\&}a\text{bonded}\text{to}{\text{a}}^{\text{15}}\text{N}\text{atom}{\mathbf{\text{then}}}^a\\ 7:{\delta }_a,{\delta }_N\Leftarrow \text{Chemical}\text{Shift}{\text{of}}^{\text{1}}{\text{H,}}^{\text{15}}\text{N}\text{atoms}\\ 8:\mathbf{\text{for}}b\in atoms\mathbf{\text{do}}\\ 9:\mathbf{\text{if}}b{\text{is}}^{\text{1}}\text{H}\text{and}\text{not}a=b\mathbf{\text{then}}\\ 10:{\delta }_b\Leftarrow \text{Chemical}\text{Shift}\text{of}{\text{second}}^1\text{H}\\ 11:d\Leftarrow ‖a-b‖\\ 12:P\Leftarrow 1-{\exp \left(-\frac{4^6\text{*In}\text{2}}{d^6}\right)}^b\\ 13:\text{With}\text{probability}P,peaks.\text{append}\left(\left({\delta }_N,{\delta }_a,{\delta }_b\right)\right)\\ 14:\mathbf{\text{end}}\mathbf{\text{if}}\\ 15:\mathbf{\text{end}}\mathbf{\text{for}}\\ 16:\mathbf{\text{end}}\mathbf{\text{if}}\\ 17:\mathbf{\text{end}}\mathbf{\text{for}}\\ 18:\text{return}peaks\\ 19:\mathbf{\text{end}}\mathbf{\text{procedure}}\end{array}}}$

^aWe simulate a ¹⁵N HSQC-NOESY, for the reasons described in section 2.2.1, which means that we are interested in the hydrogen atoms that are bonded to a nitrogen atom and nearby in space to another hydrogen.

^bThe distance between two hydrogens is used to determine the probability of a peak in the spectrum according to the inverse of the distance to the 6th power, as used in SIMPSON[23].

$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathbf{\text{Algorithm}}\mathbf{2}\text{Contact Score}}\\ 1:peaks=\left\{\left({\delta }_{i,N},{\delta }_{i,H1},{\delta }_{i,H2}\right)\forall i\in 1,\dots ,N_p\right\}\\ 2:struct=\left\{\left(x_a,y_a,z_a\right)\forall a\in atoms\right\}\\ 3:\mathbf{\text{procedure}}\text{CONTACTSCORE}\left(peaks,struct\right)\\ 4:contact\text{\_}scores\Leftarrow \left\{\right\}\\ 5:\mathbf{\text{for}}p\in peaks\mathbf{\text{do}}\\ 6:potential\text{\_}contacts\Leftarrow PC{\left(p,,,,s,t,r,u,c,t\right)}^a\\ 7:\mathbf{\text{for}}c1,c2\in potential\text{\_}contacts\mathbf{\text{do}}\\ 8:\mathbf{\text{if}}{d}_{struct}\left(c1,c2\right)\text{<}{{ϵ}_{CS}}^b\&\left|c1-c2\right|{\text{>=5}}^c\mathbf{\text{then}}\\ 9:score\Leftarrow 1-{\left({\delta }_{PC}/{\Delta }_{shift}\right)}^d\\ 10:contact\text{\_}scores.\text{append}\left(score\right)\\ 11:\mathbf{\text{end}}\mathbf{\text{if}}\\ 12:\mathbf{\text{end}}\mathbf{\text{for}}\\ 13:\mathbf{\text{end}}\mathbf{\text{for}}\\ 14:\text{ContactScore}\Leftarrow \text{total}\left(contact\text{\_}scores\right)/\left(n_{resi}^2-9n_{resi}+20\right)\\ 15:\mathbf{\text{end}}\mathbf{\text{procedure}}\end{array}}}$

^aGets all potential contacts for the peak, as described in section 2.2.1, returning a list of pairs of residues.

^b${ϵ}_{\mathit{CS}}$ is the ${\text{C}}_{\alpha }$ to ${\text{C}}_{\alpha }$ distance threshold chosen to determine whether two residues are in contact.

^cResidues which are not in contact or are short/medium range contacts do not contribute to the score.

^dScores are determined by comparing the difference in chemical shift of the peak and its potential assignment relative to the error in the peak assignments, ${\Delta }_{shift}$.

$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathbf{\text{Algorithm}}\mathbf{3}\text{Distance}\text{Score}}\\ 1:peaks=\left\{\left({\delta }_{i,N},{\delta }_{i,H1},{\delta }_{i,H2}\right)\forall i\in 1,\dots ,N_p\right\}\\ 2:struct=\left\{\left(x_a,y_a,z_a\right)\forall a\in atoms\right\}\\ 3:\mathbf{\text{procedure}}\text{DISTANCE}\left(peaks,struct\right)\\ 4:peak\text{\_}distances\Leftarrow \left\{\right\}\\ 5:\mathbf{\text{for}}p\in peaks\mathbf{\text{do}}\\ 6:potential\text{\_}contacts\Leftarrow PC{\left(p,struct\right)}^a\\ 7:minimum\text{\_}distance\Leftarrow \infty \\ 8:contact\text{\_}distance\Leftarrow \infty \\ 9:\mathbf{\text{for}}c1,c2\in potential\text{\_}contacts\mathbf{\text{do}}\\ 10:\mathbf{\text{if}}{d}_{struct}\left(c1,c2\right)<minimum\text{\_}distance\mathbf{\text{then}}\\ 10:minimum\text{\_}distance\Leftarrow d_{struct}\left(c1,c2\right)\\ 11:contact\text{\_}distance\Leftarrow \left|c1-c2\right|\\ 12:\mathbf{\text{end}}\mathbf{\text{if}}\\ 13:\mathbf{\text{end}}\mathbf{\text{for}}\\ 14:\mathbf{\text{if}}contact\text{\_}distance>=5^b\mathbf{\text{then}}\\ 15:peak\text{\_}distances.\text{append}\left(minimum\text{\_}distance\right)\\ 16:\mathbf{\text{end}}\mathbf{\text{if}}\\ 17:\mathbf{\text{end}}\mathbf{\text{for}}\\ 18:\text{Distance}\Leftarrow \text{len}\left(\{d>{ϵ}_D|d\in peak\text{\_}distances\}\right)/\text{len}{\left(peak\text{\_}distances\right)}^c\\ 19:\mathbf{\text{end}}\mathbf{\text{procedure}}\end{array}}}$

^aGets all potential contacts for the peak, as described in section 2.2.1, returning a list of pairs of residues.

^bIf a peak is best explained by a short/medium range contact, it does not contribute to the distance.

^cDS is the fraction of minimal distance peaks that are greater than the interresidue ${\text{C}}_{\alpha }$ C $\alpha$ distance threshold ${ϵ}_D$.

Data Availability

Code and data references will be made available at the time of publication via GitHub. Structures and data downloaded from the PDB (https://www.wwpdb.org), BMRB (https://bmrb.io/), and AlphaFold DB (https://alphafold.ebi.ac.uk).