Structural heterogeneity and precision of implications drawn from cryo-electron microscopy structures: SARS-CoV-2 spike-protein mutations as a test case

Abstract

Protein structures may be used to draw functional implications at the residue level, but how sensitive are these implications to the exact structure used? Calculation of the effects of SARS-CoV-2 S-protein mutations based on experimental cryo-electron microscopy structures have been abundant during the pandemic. To understand the precision of such estimates, we studied three distinct methods to estimate stability changes for all possible mutations in 23 different S-protein structures (3.69 million ΔΔG values in total) and explored how random and systematic errors can be remedied by structure-averaged mutation group comparisons. We show that computational estimates have low precision, due to method and structure heterogeneity making results for single mutations uninformative. However, structure-averaged differences in mean effects for groups of substitutions can yield significant results. Illustrating this protocol, functionally important natural mutations, despite individual variations, average to a smaller stability impact compared to other possible mutations, independent of conformational state (open, closed). In summary, we document substantial issues with precision in structure-based protein modeling and recommend sensitivity tests to quantify these effects, but also suggest partial solutions to the problem in the form of structure-averaged “ensemble” estimates for groups of residues when multiple structures are available.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00249-022-01619-8.

Introduction

Protein structures are commonly used as input to draw mechanistic implications at the residue level, either by inspection or more often by some form of computation/calculation. This has been much the case during the pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The spike glycoprotein (S-protein) binds to the human receptor angiotensin-converting enzyme 2 (ACE2) during cell infection and is targeted by the immune system, which is the mode of action of many vaccines. (Fehr and Perlman 2015; Letko et al. 2020; Liu et al. 2020; Wang et al. 2020; Forni and Mantovani 2021; van Dorp et al. 2021) Antigenic drift of new S-protein mutations is an urgent challenge, (McCallum et al. 2021; van Dorp et al. 2021) with the recent omicron variant being a hallmark example. (Liu et al. 2022; Planas et al. 2022) Accordingly, S-protein structures are of high interest. (Alsulami et al. 2021; Mehra and Kepp 2022) Made possible by the recent technical breakthroughs in cryo-electron microscopy (cryo-EM) of macromolecules, (Fernandez-Leiro and Scheres 2016; Murata and Wolf 2018; Danev et al. 2019; Blundell and Chaplin 2021) an unprecedentedly rich structural biology of the S-protein was induced by healthy competition during the pandemic, with hundreds of structures published. (Mehra and Kepp 2022) While important of course in themselves, this richness of data also enables new options for testing the sensitivity of computational protein modeling to the exact choice of input structure.

The S-protein is under selection for antigenic drift and plausibly also for maintenance of overall fold stability, (Mehra and Kepp 2022) as seen in other cases of protein evolution. (Bershtein et al. 2008; Tokuriki and Tawfik 2009; Goldstein 2011; Liberles et al. 2012; Kepp 2020) The prefusion S-protein is in a conformationally variable metastable state that evades antibodies and contributes to ACE2 binding. (Berger and Schaffitzel 2020) An accurate understanding and prediction of future SARS-CoV-2 evolution (Maher et al. 2022) requires insight into the potential impact of mutations, and experimental cryo-EM structures are essential for such insight. (Mehra and Kepp 2022).

Computer models are often applied to understand the role of specific residues, typically by studying mutations. However such models have limitations due to noise and biases in the training data and dependencies on a folded wild-type structure for extrapolating the impacts, if the mutant structure is not available. (Khan and Vihinen 2010; Christensen and Kepp 2012; Kepp 2015; Pucci et al. 2018, 2022; Caldararu et al. 2020, 2021a; Iqbal et al. 2021; Louis and Abriata 2021) Computational methods carry bias toward some mutation types. (Pucci et al. 2018; Caldararu et al. 2020) In addition to the heterogeneity within each separate cryo-EM map, which is informative of the ensemble dynamics within the settings of that study and increasingly handled with new techniques, (Scheres 2016; Zhong et al. 2021) “extra” heterogeneity arising from e.g., different chemical conditions and sample and data collection protocols can also be important but has not been studied much, and only for crystal structures obtained by X-ray diffraction. (Mehra et al. 2020; Caldararu et al. 2021a) We are not aware of any studies of the precision of cryo-EM structures in a functional context, i.e., how differences between cryo-EM structures of the supposedly same protein state affect functional deductions from computation. Understanding the precision of such protein modeling requires comparing multiple structures. More than 200 structures published during the pandemic (Alsulami et al. 2021; Mehra and Kepp 2022) makes the SARS-CoV-2 S-protein an ideal study case for exploring this problem with a sufficiently large number of structures.

Our work was partly motivated by many well-cited computational studies published during the pandemic using single methods and a few structures to deduce functional implications for single residues. (Delgado Blanco et al. 2020; Hadi-Alijanvand and Rouhani 2020; Laha et al. 2020; Othman et al. 2020; Shorthouse and Hall 2021; Xue et al. 2021; Teng et al. 2021; Kumar et al. 2022; Rochman et al. 2022) In our initial studies, we found that results varied with method and structure choice, requiring us to investigate much more data intensively many different structures, as we report below.

Here, we address this problem of precision (i.e., how much the choice of cryo-EM structure affects computationally deduced functional implications) via the structure-guided impact of residue substitutions on S-protein stability. Computational methods to estimate these effects from experimental wild-type structures have been developed and tested over several decades and their strengths and weaknesses are fairly well understood. (Sanavia et al. 2020; Casadio et al. 2022; Pucci et al. 2022) We show that even without experimental data available to validate accuracy, the precision of such computational estimates is a major challenge. However, we demonstrate that we can utilize the rich structural biology of the S-protein by explicitly considering several experimental structures as a simple ensemble to improve the estimates.

Methods

Strategy

How much does the choice of cryo-EM structure affect functional deductions at the residue level? This question of inter-structure heterogeneity should not be confused with heterogeneity within a specific cryo-EM map, which gives important information on the ensemble in a specific study’s context. Computational estimates of the impact of mutating a residue based on a wild-type structure provide a metric for answering this question, which is essentially about precision and does not require the thermodynamic stabilities of hundreds of S-protein mutants to be known. The unique richness of cryo-EM structures of the S-protein (Mehra and Kepp 2022) offers an unprecedented possibility to test this question in the resolution regime of 2.5 − 3.5 Å.

Stability impacts of mutating a residue are arguably the most straightforward way to study this problem, as methods have been developed for several decades for this purpose. (Louis and Abriata 2021; Casadio et al. 2022; Pucci et al. 2022) To estimate the effects of S-protein mutations, we

• 1) use several methods for sensitivity analysis, including some that are more or less sensitive to structural variations,

• 2) account for heterogeneity in the experimental S-protein structures by using 23 structures as input,

• 3) compare both heterogeneity between structures of the supposedly same conformational state and between different conformational states,

• 4) illustrate that interpretations at the residue level can be very structure-dependent, and

• 5) propose protocols to handle this, by comparing mutation groups rather than single mutations and sites, averaged over an “ensemble” of experimental structures.

Protein structures studied

The initial part of our work emphasized S-protein structures without antibodies or ACE2 bound in an evaluation of structural sensitivity. An example of the prefusion closed state is shown in Fig. 1a. We computed all mutations possible in the protein and illustrated results for a group of natural mutations (Fig. 1b) known to have large impact on function. (Tegally et al. 2020; Li et al. 2020; Chen et al. 2021; Yuan et al. 2021; Greaney et al. 2021; McCallum et al. 2021; Starr et al. 2021; Wang et al. 2021; Thomson et al. 2021; Dejnirattisai et al. 2021; Ferreira et al. 2021) We used six different S-protein structures all representing the S-protein closed state (Fig. 1c), selected based on resolution and residue coverage. These include 6X6P by Herrera et al. 2021, 7DF3 by Xu et al. 2021, 7CAB by Lv et al. 2020, 6X79 by McCallum et al. 2020, 7DDD by Zhang et al. 2021, and 6VXX by Walls et al. 2020.

Fig. 1.

SARS CoV-2 S-protein structures studied in this work. a Trimer colored by chain (PDB code: 7DF3). b Prominent mutation sites studied shown as red balls on monomer unit (PDB code: 7DF3). c Six structurally aligned S-protein structures investigated in this work (each structure represented in a single color, RMSD 0.5−2.9 Å)

The number of residues (N), the percentage of outliers from a Ramachandran analysis (% Outl.), and the resolutions (in Å) of these structures are given in Table 1. We note that the Ramachandran outliers partly reflect molecular mechanics force-field optimization and thus, while indicating deviations from expected realistic backbone conformations, do not inform directly on the empirical quality of the structure (in fact, very accurate structures may have some real outliers not observed upon force field optimization, which is biased toward small variation from ideal torsion angles). We also note that the ambient temperature ensemble may differ somewhat from the cryo-temperature ensemble obtained upon rapid cooling, in addition to chemical composition and cell effects (Mehra et al. 2020). We finally note that the cryo-EM structure deposited in the PDB is usually just one structure out of many that could be derived from the particle data, and it is possible to address the heterogeneity and ensemble effects of a specific experiment by other techniques. Our interest here is to quantify the effect of the choice of deposited structure on results, as it is standard to use a single structure sometimes rather arbitrarily if more good structures are available. However, we expect the structures to provide good information on backbone conformation and secondary and tertiary structure.

Table 1.

Structures used in the present study to compute all mutations

PDB	N	% Outl	Res (Å)	Reference
6X6P	1017	0.0	3.2	Herrera et al. (2021)
7DF3	1088	0.3	2.7	Xu et al. (2021)
7CAB	1029	0.1	3.5	Lv et al. (2020)
6X79	950	0.1	2.9	McCallum et al. (2020)
7DDD	1088	0.1	3.0	Zhang et al. (2021)
6VXX	972	0.2	2.8	Walls et al. (2020)

To understand conformation-state-specific mutation stability effects, we also studied all possible mutations in an additional 17 S-protein structures representing different states: closed, open, locked and active (Table S1). To minimize confounding effects (chemical conditions, lab protocol, protein modifications) we only compared conformational states reported by the same study (not one state from one study with another state from another study). These structures include 6VYB, (Walls et al. 2020) 7DWY, (Yan et al. 2021) 7DWZ, (Yan et al. 2021) 6XF5, (Zhou et al. 2020) 6XF6, (Zhou et al. 2020) 7A4N, (Juraszek et al. 2021) 7AD1, (Juraszek et al. 2021) 6X2C, (Henderson et al. 2020) 6X2A, (Henderson et al. 2020) 6X2B, (Henderson et al. 2020) 6ZGE, (Wrobel et al. 2020) 6ZGI, (Wrobel et al. 2020) 6ZGG, (Wrobel et al. 2020) 7KDG, (Gobeil et al. 2021) 7KDH, (Gobeil et al. 2021) 6Z97, (Huo et al. 2020) and 6ZB4 (Toelzer et al. 2020) having N = 963−1099, 0−0.5% outliers from Ramachandran analysis, and resolutions of 2.6−4.0 Å. These data were analyzed as pair comparisons of closed and open structures.

Methods to estimate stability effects of mutations

Protein mutations typically tend to more often be destabilizing than stabilizing, because of the usual evolutionary optimization of structures, giving a skewed fold stability (ΔΔG) distribution of arising mutations. (Tokuriki et al. 2007) Datasets accordingly also tend to be skewed toward destabilization and biased toward small stability effects, and also carry a bias in mutation type, as some mutations have been more commonly studied experimentally, e.g. alanine mutations, and these imbalances also affect method performance. (Pucci et al. 2018; Caldararu et al. 2020) The output must thus be seen in the context of other methods using different approaches, in addition to with same method for other structural inputs.

To study the impact of method choice on the results for individual mutations, we used three methods, SimBa-2 (or SimBa-IB), (Caldararu et al. 2021b; Bæk and Kepp 2022a) I-Mutant 3.0, (Capriotti et al. 2005, 2006) and CUPSAT, (Parthiban et al. 2006) to compute the change in fold stability (ΔΔG, in units of kcal/mol) for all possible mutations of all the 23 studied structures (69 chains). The methods represent three different design types–linear regression, machine learning, and force field-potential based–and are thus expected to provide a good indication of the maximal sensitivity toward method choice. SimBa has been tested both by us (Caldararu et al. 2021b) and in an independent benchmarking study (Pucci et al. 2022) where it performed slightly above average in terms of error and R², despite its simplicity. SimBa reduces biases by design, but was run in the nonsymmetric mode (IB) which typically has slightly higher accuracy for random mutations. (Bæk and Kepp 2022a) I-Mutant (Capriotti et al. 2005, 2006) is a robust Support Vector Machine trained on ProTherm data which is not mutation-type or stability-balanced (Bæk and Kepp 2022a; Pucci et al. 2022) but has shown good general performance in many benchmarks, (Potapov et al. 2009; Kepp 2014, 2015; Pucci et al. 2022) and has low structural sensitivity. (Caldararu et al. 2021a) CUPSAT (Parthiban et al. 2006) uses environment-specific force fields to predict protein stability that is more sensitive to structure input. (Kepp 2015; Caldararu et al. 2021a) In addition, relative solvent accessibility (RSA) was calculated using each method for all the mutated sites using all 23 structures.

Computations and group comparisons

We used a group comparison protocol to illustrate how a hypothetical study of a few mutations of interest will be sensitive to method and structure choices, and to illustrate the statistical advantages of comparing mutations in groups against each other. By comparing groups of mutations, we can determine if a group of mutations has unusual properties relative to another group by e.g., a t-test. This protocol can produce meaningful results if output for single sites is uncertain. The grouping protocol works because random errors are reduced with larger N of the sample size, and systematic errors are reduced by comparison of the average effects in a t-test (i.e., the systematic errors in a method will exist in both groups and partly cancels when comparing only the mean output of the groups). To this end, we computed the stability effect of all possible N × 19 mutations (about 18,000 to 21,000 data points per chain) for each of the three chains of the closed six S-protein structures with all three methods, corresponding to approximately 3 × 19,000 ΔΔG values for each structure (N typically ~ 1000).

To illustrate the heterogeneity in estimates of functional impact at the individual residue level as would often feature in a study, we separated out results for a small group of mutations seen in prominent SARS-CoV-2 variants and known to impact virus function: K417T, K417N, N439K, N440K, Y449H, L452R, Y453F, S477N, T478K, E484K, E484Q, S494P, N501Y, and D614G. For example, K417T and K417N found in gamma (P.1) and beta (B.1.351) (Yuan et al. 2021) and L452R and N439K evade some antibodies (Li et al. 2020; McCallum et al. 2021; Starr et al. 2021; Thomson et al. 2021) as do mutations in the E484 site in e.g. alpha (B.1.117) and beta (B.1.351). (Tegally et al. 2020; Chen et al. 2021; Greaney et al. 2021; Wang et al. 2021; Yuan et al. 2021; Dejnirattisai et al. 2021; Ferreira et al. 2021) We also performed complete saturated mutagenesis for additional structures in partially open states to estimate if the effects were state-dependent, such that the total number of protein structures studied was 23, or 69 chains, and the number of mutation effects computed was thus 69 × N × 19, about 1.23 million ΔΔG values in total for each of the three methods (3.69 million ΔΔG values).

Analysis of data and statistics

The percentage of outliers from Ramachandran plots (torsion angles of the peptide backbone) indicate the realism of the experimentally obtained structure’s backbone conformations (Wlodawer et al. 2008) and were calculated using the Procheck program (Laskowski et al. 1993) (version 3.6.2) available via the PDBsum server. (Laskowski et al. 2018) The average ΔΔG, its standard deviation and average RSA values were calculated and compared. Mutation groups were compared for their stability effects using two-tailed student t-tests for the same mean, using 95% confidence intervals. We also studied the relationship between ΔΔG and RSA values using linear regression. Statistical analysis and plotting were performed using Matplotlib and Sklearn libraries of Python 3 and Microsoft Excel.

Results and discussion

Computing the stability effects of SARS-CoV-2 mutations

The main aim of our study was to investigate how much the choice of cryo-EM structure affects functional implications at the single residue level, illustrated by the stability effect of mutations. The rich S-protein structural biology has made such a study feasible and arguably also relevant to some biological conclusions specifically regarding S-protein mutations in the literature.

First, to quantify how structure choice affects computed estimates broadly, scatter plots of the ΔΔG values of all possible mutations in chain A of six similar prefusion S-protein structures (RMSD 0.5–2.9 Å) are shown in Figures S1, S2 and S3 for the three methods. Despite the correlations, the result for a specific mutation commonly depends on the structure used, with cases of several kcal/mol differences and sign changes. SimBa and I-Mutant3.0 show less structural sensitivity (Figures S1 and S2), with R² > 0.9 between different structures, whereas CUPSAT shows large variation (Figure S3) with R² ranging from 0.30 to 0.61, consistent with its larger emphasis on the local site geometry. (Kepp 2015; Caldararu et al. 2021a) These plots illustrate how the structural variations in the cryo-EM structures affect the precision of functional implications for methods that emphasize such local site structural variations more or less.

To understand how this heterogeneity affects estimates for specific groups of mutations in an envisioned study, Fig. 2 shows the mutation effects estimated using the SimBa method for the 18 chains of these six structures. Results are shown for all possible mutations (blue), for a selection of mutations observed in the wild (green), and for all possible mutations in the sites harboring these selected mutations (orange) (negative values represent destabilization). The selected mutations in this example (green) could have been studied with any of these structures as the green data in one of the panels. However, Fig. 2 shows that results depend on structure used, with the average for all possible mutations varying from − 0.55 to − 0.58 kcal/mol (standard deviation (SD) 1.32 to 1.38), from − 0.13 to − 0.55 (SD 0.95 to 1.18) for all mutations in the selected sites, and from − 0.12 to − 0.50 (SD 0.66 to 0.84) for the selected mutations in the example. The groups, by increasing N, reduce random errors in estimates, i.e., the central estimate interval is smaller for a larger batch of mutations than if studying one or a few mutations, as commonly done.

Fig. 2.

Violin plots of stability effects (ΔΔG, kcal/mol) calculated using the SimBa method (six structures, chains A, B, and C). Blue violins represent all possible mutations in the chain; orange represent all possible mutations at selected mutated positions; green represents selected natural mutations. The values are the average and standard deviation (in brackets) of the datasets

Figure 2 indicates structure-dependent variations even for a group comparison and a single method. Such dependencies are seen for all three methods (results for the other two methods in Figure S4 and Figure S5) and reflect differences in the cryo-EM structures that affect the model estimates. The group of all possible mutations provides a global comparison against which one can compare the mutations selected for study. Only the differences between averages are then relevant, not the direct values, even if structure-averaged, due to systematic and random errors in methods used to extract the information from the structures. We note that the three chains give similar results here because they are all in the down conformation.

The results suggest that random mutations are more likely to destabilize the S-protein, as expected since proteins are optimized for some fold stability. (Tokuriki et al. 2007) Therefore, stability effects tend to be skewed toward destabilization, and when trained on such data the methods develop a destabilization tendency that may be too small or too large. (Christensen and Kepp 2012; Pucci et al. 2018; Sanavia et al. 2020; Casadio et al. 2022) Most mutations have small, nearly neutral effects, but a few are predicted to be very destabilizing or stabilizing.

While individual mutation effects vary substantially even for the same mutation across different structures, the group comparisons for six structures reduce this noise considerably. In all 18 comparisons, the naturally occurring mutations (green) are as a group less destabilizing than both control sets (blue and orange), and the site-specific control (orange) produces less destabilization than the full control group (blue), consistent with surface residues having less impact on stability, all-else-being equal. Thus, group comparisons can make computational studies of mutations meaningful despite the noise in structure and method input. We also note that the natural mutations (green) have consistently smaller destabilization effects than all possible mutations in the same naturally evolved sites (orange) by 0.08 kcal/mol for 6X6P, 0.15 kcal/mol for 7CAB, and 0.12 kcal/mol for 6VXX, but only 0.01 kcal/mol for 7DF3, 0.05 kcal/mol for 6X79, and 0.03 kcal/mol for 7DDD.

The corresponding computations with I-Mutant and CUPSAT show similar results vs. the total group (Figure S4 and Figure S5). However, I-Mutant has a natural mutation group slightly more destabilizing than the same-site mutation group, whereas CUPSAT has the same behavior for both control sets as SimBa, although producing larger values. There are clear differences in the magnitude of the effects both absolutely and relative to the compared groups.

For C3 symmetrized structures, the symmetry constraints generate identical chain conformations. However, the comparison of C1 (no symmetrisation) and C3 EM maps is interesting: for the structures based on C1 EM maps, we observed clear differences in averages (e.g., 7DDD, 6ZB4 in Figure S5 and S8c) for individual chains for some methods. The cryo-EM chain structures, if not symmetrized, reflect relevant noise in the experiment or real dynamics but can give different results if one focuses on a single chain as basis for modeling or interpretation. These effects depend on the method used to map structure to function: if one puts a lot of emphasis on structural variations, such as CUPSAT, then the C1 differences between chains become larger.

The results document the importance of using multiple methods and structures and shows why the direct numbers cannot be used, only the differences in averages between mutation groups. Given the precision effects, we suggest that this type of analysis is done in future studies. We do not wish to discuss specific examples in the literature based on single mutations and structures, but our results suggest that studies based on single structures and methods or emphasizing the direct numbers of the computed effects for a few mutations (as the orange data in any of the panels in Fig. 2) should be supplemented by a broader sensitivity analysis of the type shown in the combined Fig. 2. (Laha et al. 2020; Walls et al. 2020; Shorthouse and Hall 2021; Teng et al. 2021) While we did study several million ΔΔG values with this protocol, we stress that computer power is not a limiting factor for these methods, and the structures are available, so it is mainly a matter of scripting and automating the analysis.

Implied mutation effects: structure and method dependencies

To illustrate how this heterogeneity affects computed results, Fig. 3 summarizes results for 14 natural S-protein mutations observed in variants of concern, computed using the same six prefusion S-protein structures as in Fig. 2. The simple linear regression model SimBa (Fig. 3a) and the vector machine model I-Mutant (Fig. 3b) produce results relatively less sensitive to structure input, consistent with previous findings (Caldararu et al. 2021a, b; Bæk and Kepp 2022a) and more similar tendencies of destabilization, except for S477N and N501Y where SimBa predicts high stabilization and I-Mutant a small impact. They tend to provide the same sign regardless of structure used, but magnitudes vary. However, I-Mutant predicts all mutations to be destabilizing, which the other methods do not. Since this comparison is for the prefusion S-protein, the chains are nearly identical and the methods produce similar results for all chains except a few cases for CUPSAT (Fig. 3c).

Fig. 3.

Change in stability (kcal/mol) of studied natural mutations for six prefusion structures of the SARS-CoV-2 S-protein. a SimBa. b I-Mutant 3.0. c CUPSAT

Figure 3 shows that the methods disagree on the magnitude and sometimes sign, i.e., one cannot draw conclusions for a single mutation based on a single method. The obvious solution to such errors is to only consider differences between mutations for each method separately. For CUPSAT, an example of a statistical force-field based method, the conclusions sometimes depend substantially and even qualitatively on the structure used, with e.g., S494P being very destabilizing when using 6VXX but very stabilizing when using 7CAB. This shows that results from single structures can be very imprecise for methods that emphasize local site geometry. Opposite effects to those reported could have been seen if using another structure. As far as we know, most (if not all) studies published used mainly one method and one or a few structures of the S-protein, and we suggest that the structure-averaged group differences, as illustrated in Fig. 2, will solve some of the precision issues seen in Fig. 3.

Since all structures selected here are of reasonable resolution from a cryo-EM perspective and are of similar residual coverage, no structure is clearly preferable to any other. Thus, in an ensemble of structures, differences are likely due to variations in chemical conditions and sample preparation and data collection. For this reason, we weighted all six structures equally as an “ensemble”, simply averaging out the structural heterogeneity to get a more compressed indication of the stability effects of natural mutations (in blue) in Figure S6. The standard deviations across the structures (in orange) can be taken as a measure of the “precision” of the method for the studied mutation group (Caldararu et al. 2021a). This comparison yields in general good agreement between I-Mutant and SimBa, but very different results for CUPSAT even after averaging out structural heterogeneity and considering the standard deviations.

Reducing heterogeneity by structure-averaged group comparisons

To reduce structural noise, we compared the structure-averaged results only, and additionally only considered differences in averages for groups of mutations, rather than direct values or differences in values of individual mutations. Figure 4 shows that even though local heterogeneity affects individual mutation estimates, comparison of mutation groups is statistically meaningful (i.e., has precision high enough to give statistically significant results in comparisons), as confirmed by a two-tailed t-test for the same mean (Table S2). The difference between the studied natural mutations and all possible mutations in the protein was significant at > 99% confidence for all three methods, as was all possible mutations in the natural mutation sites vs. all possible sites in the S-protein. The similar comparative behavior predicted by the three methods after incorporating structural heterogeneity differs from the analysis of individual mutations in Fig. 3. In the comparison of the natural mutations vs. all possible mutations in the naturally evolving sites, results are significant for SimBa (Fig. 4a) and I-Mutant (Fig. 4b), but not for CUPSAT (Fig. 4c), and the direction of effect differs between I-Mutant and SimBa, making these results inconclusive. In other words, conclusions based on a single cryo-EM structure can sometimes be misleading, and a sensitivity test to the use of other equally reasonable cryo-EM structures as input is therefore recommended.

Fig. 4.

ΔΔG values of some important natural mutations compared to full-protein averaged mutation backgrounds, averaged over six closed S-protein structures (6X6P, 7DF3, 7CAB, 6X79, 7DDD and 6VXX). a SimBa. b I-Mutant3.0. c CUPSAT

State-specific effects

The SARS-CoV-2 S-protein undergoes a conformational change when fusing with host cells, from the closed prefusion state studied above to a partially open state interacting with ACE2, with one, two, or three of its receptor binding domains (RBD) in an upwards conformation (1-up, 2-up, 3-up). To test whether the conformational change affects the stability trends discussed above, or if the stability effects were more or less pronounced in the open states, we computed the mutation effects for all-possible mutations for 17 additional S-protein structures (for each of the three chains) in the partially open or locked states. Consistently, the overall tendency of natural mutations to produce less destabilization as a group than the group of random mutations was seen not just for the closed structures (Figure S7, as discussed above) but also for these additional structures, with an overview for all 23 structures provided in the Supporting Information Figures S8-S9.

To explore whether these conformational changes affect the estimated stability effects, we analyzed the pairs of closed and open structures published from the same studies to reduce noise from confounders (such as different protocols and conditions of structure preparation and analysis) in Figures S10–S15, with one example shown in Fig. 5, for 6ZGE (closed uncleaved, locked) vs. 6ZGI (closed and cleaved) and 6ZGG (1-up). For the mutations as a whole, the state did not substantially affect the stability effects, irrespectively of using SimBa (Fig. 5a), I-Mutant (Fig. 5b), or CUPSAT (Fig. 5c), due to the group effect of large N averaging out the heterogeneity at the site level.

Fig. 5.

Comparison of the ΔΔG values of important natural mutations in 6ZGE (closed uncleaved, locked), 6ZGI (closed cleaved) and 6ZGG (cleaved 1-up) structures using three methods: a SimBa. b I-Mutant3.0. c CUPSAT

This finding was also confirmed by other pair comparisons given in the Supporting Information, such as for 6VYB (open), 7DWY (locked) and 7DWZ (active) (Figure S10), 6XF5 (prefusion closed) and 6XF6 (prefusion 1-up) (Figure S11), 7A4N (closed) and 7AD1 (1-up) structures (Figure S12), 6X2C (closed), 6X2A (1-up) and 6X2B (2-up) (Figure S13), 6VXX (closed) and 6VYB (open/1-up) (Figure S14), and 7KDG (closed) and 7KDH (1-up) structures (Figure S15). Numerical data for these plots are available in Tables S3-S9. All these pair comparisons suggest that group tendencies are not much affected by the state of the protein, although some individual sites (and thus mutations) are disordered and thus highly sensitive to conformational state and the structure used, a notable example being the D614 site (Mehra and Kepp 2022) that harbors the D614G mutation that was fixated early in the pandemic.(Yurkovetskiy et al. 2020; Mansbach et al. 2021).

Relationship between solvent accessibility and stability effects

Since we expect that the more solvent-exposed sites have milder impact on the S-protein’s fold stability than random mutations, (Caldararu et al. 2021b) regardless of the protein state (open, closed), we mapped this relationship in Fig. 6. The blue circles represent the stability effects of the studied natural mutations averaged over the six S-protein structures. The larger green and orange circles represent the average of the natural mutation effects and the average of all possible mutation effects, respectively, with stability changes shown vertically and solvent exposure of the site shown horizontally.

Fig. 6.

Scatter plots of ΔΔG vs. relative solvent accessibility (RSA) for natural mutations (blue circles) (structure-averaged on 6X6P, 7DF3, 7CAB, 6X79, 7DDD and 6VXX). Green circles represent the average for the natural mutations and orange represents the average for all possible chain mutations. a SimBa. b I-Mutant-3.0. c CUPSAT

As confirmed by the aggregate results of SimBa (Fig. 6a), I-Mutant (Fig. 6b), and CUPSAT (Fig. 6c), the natural mutations (as a group shown in green) tend to be more solvent-exposed and less destabilizing than expected for a random mutation as a group (orange circle) in the S-protein. The similarly decent correlations (R = 0.36 − 0.75, with same direction) of the average RSA with average ΔΔG also exist while using different methods. The composite results considering structural heterogeneity by averaging data and group comparisons yield similar tendencies for the three methods that increases significance, which could be different if analyzing a single method and structure, and a few mutations. Thus, while the average site in the SARS-CoV-2 S-protein is only 25% solvent-exposed, the studied natural mutations are on average 40% exposed, although there is substantial variation.

Conclusions

The SARS-CoV-2 pandemic has motivated publication of hundreds of cryo-EM structures of the S-protein. (Mehra and Kepp 2022) While of value for understanding SARS-CoV-2, this unprecedented effort also enables investigation of a more fundamental topic: how precise are computed implications using cryo-EM structures? The precision of such estimates is as important as the accuracy, but only the former can be studied meaningfully if many structures of the same protein state are available, whereas assessment of accuracy requires large unbiased experimental data sets. It has now become possible to study this problem at high data coverage due to the SARS-CoV-2 pandemic producing many structures even for the same states of a single protein.

We show that precision is a major challenge already before any accuracy assessment. Since results can be very dependent on structure used, one cannot simply rely on one method and structure. To handle this, the protein modeling community is recommended to use several methods and structures of presumed similar quality to sensitivity test conclusions. A more robust protocol involves analysis at four levels:

• 1) structure-averaging to reduce noise in the residue coordinates,

• 2) comparing several methods to understand errors and obtain consensus estimates,

• 3) using groups of mutation averages to reduce noise and improve significance (large N effect), and

• 4) comparing only differences between averages of mutation groups, to reduce systematic errors.

We can thereby utilize the structural information more broadly and take advantage of the law of large numbers to reduce noise for individual residues. The issues are probably aggravated when having additional heterogeneity from protein–protein or protein–ligand interactions. Our study may have some general relevance to the computational structural biology field by quantifying how method and structure choices affect deduced structure–function relationships at the residue level of cryo-EM structures. Variations in published structures, partly due to some arbitrariness in the structure taken as representative from each cryoEM map and partly due to real variations in protocols, reflect a real additional uncertainty that we need to deal with, in addition to the uncertainty seen within a single experiment.

As a note, similar issues of precision and heterogeneity may in principle exist also for crystal structures. However, even with a statistically meaningful analysis of many available structures of the supposedly same protein state, as was possible for S-protein cryo-EM structures due to the pandemic, the distinct protocol for fitting and modeling X-ray reflections will lead to heterogeneity and resolution not being directly comparable. For example, the crystal packing could affect the solvent-exposed sites differently from the frozen protein obtained from cryo-EM microscopy, and this could also affect both precision and accuracy, i.e., even if precision would be higher for exposed crystal-structure sites, their realism could be lower, although these effects will take substantial work to understand fully.

Structure-averaged group comparisons offer a partial solution, utilizing the published data broadly. In cases where several (same-state) structures are available, if there is no reason to favor one over the other, several structures should be used to produce the chemical insight and a sensitivity estimate provided. In the absence of several structures, the heterogeneity could be generated computationally. Molecular dynamics may, with some limitations due to sampling, force field, and realism of the chemical composition, be used to estimate conformational heterogeneity, e.g., under different conditions of temperature and ionic strength and pH. (Bottaro and Lindorff-Larsen 2018; Mehra et al. 2020), and machine-learning methods could be expected to help to represent the heterogeneity of an ensemble beyond the single experiment itself, if the more problematic e.g., polar disordered residues can be described by ensembles. (Chen and Shukla 2022; Bæk and Kepp 2022b) If the protein is not very flexible and condition-dependent, a few structures might be sufficient for the analysis.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

RM: Conceptualization and computation, visualization, analysis and writing the paper. KK: Conceptualization and computation, analysis and writing the paper.

Data availability

The Supporting Information file contains additional data relevant to the study (tables and figures). The methods used are available online: CUPSAT: http://cupsat.tu-bs.de/. SimBa: https://github.com/kasperplaneta/SimBa2. I-Mutant: https://folding.biofold.org/i-mutant/.

Declarations

Conflicts of interest

The authors declare that they have no conflicts of interest related to this work.