Driven to Near-Experimental Accuracy by Refinement via Molecular Dynamics Simulations

Abstract

Protein model refinement has been an essential part of successful protein structure prediction. Molecular dynamics simulation-based refinement methods have shown consistent improvement of protein models. There had been progress in the extent of refinement for a few years since the idea of ensemble averaging of sampled conformations emerged. There was little progress in CASP12 because conformational sampling was not sufficiently diverse due to harmonic restraints. During CASP13, a new refinement method was tested that achieved significant improvements over CASP12. The new method intended to address previous bottlenecks in the refinement problem by introducing new features. Flat-bottom harmonic restraints replaced harmonic restraints, sampling was performed iteratively, and a new scoring function and selection criteria were used. The new protocol expanded conformational sampling at reduced computational costs. In addition to overall improvements, some models were refined significantly to near-experimental accuracy.

INTRODUCTION

Protein structure prediction has become an essential tool for understanding biological processes at the molecular level after decades of advances.¹ Due to an increasing number of experimentally resolved protein structures and progress in template-based modeling methods, it has become possible to predict protein structures with high accuracy.² Increased coverage of the protein fold space means that it is more likely that close homolog structures suitable for template-based modeling can be found. At the same time, there has been significant progress in the prediction without structural templates.³ Enabled by the rapid growth in protein sequence databases from next-generation sequencing (NGS), residue-residue contacts can often be predicted based on co-evolution analysis.^4–6 Machine learning techniques and especially deep neural network variants have been most successful in making reliable contact predictions based on sequence alignments.^7–11 If sufficient residue contact information is available, high-resolution models can be built even when there is no homologous structure available as a template.

Despite many successes with protein structure prediction, the resulting models often retain inaccuracies and fail to reach the accuracy of true experimental structures. Consequently, there have been increased efforts on improving predicted models via refinement as a final touch.¹² In the early 2010s, a few successful protein model refinement protocols began to emerge.^13–15 These methods were able to refine initial models consistently but delivered only moderate improvements. One of the methods, developed by our group, is based on ensemble averaging of sampled conformations via molecular dynamics (MD) simulations.¹⁶ The method was able to improve both global and local model quality including streochemical properties. The approach has been adopted as part of many structure prediction protocols as an endgame player in structure prediction.^17,18 This approach improved for a few years due to advances in force fields and benefited from an ability to carry out longer simulations.¹⁹ However, there was little progress in recent years even with much longer simulations.²⁰ During CASP12, as much as a few microsecond MD simulations were applied per target, but the increased sampling did not improve refinement success much. Post-CASP12 analysis revealed that the sampling was trapped in the vicinity of the initial model and most of the conformational space remained undiscovered. A main issue was the use of restraints that successfully prevented deterioration of the initial model away from the native stute but also hindered the conformational sampling towards the native state. Other studies sought to overcome the sampling problem in refinement by devising enhanced sampling strategies. Della Corte et al. proposed to use homologous structures to make energy landscapes smoother.²¹ GalaxyRefine sampled structures by perturbing confromations in various ways focusing on rearranging side chains, seconday structure elements, or even more global structural changes in order to cross energy barriers believed to hinder refinement.^13,22 However, these strategies were not successful so far in fully addressing the refinement challenge.

Recently, we investigated in more depth whether it is possible at all to refine models to experimental accuracies via MD simulations and what it would take in terms of sampling to accomplish that goal.²³ Eight small refinement targets from previous rounds of CASP were selected with different folds and different types of modeling errors in the initial models. Sets of extensive MD simulations were carried out to explore the conformational landscape between initial models and the known experimental structures of the respective native states. In all cases, numerous conformational states were identified and Markov-state modeling was applied to construct kinetic networks that connect the initial models to the native states. For all systems, the state closest to the native structure had the lowest free energy and deviated by less than 1 Å in most cases from the experimental structure. While this suggests that refinement via MD to near-experimental accuracy is in principle possible, the analysis also revealed that the transition from the initial models to the native states required the crossing of multiple kinetic barriers with time scales of a few to hundreds of microseconds. These large barriers were found to correspond to partial unfolding and refolding transitions along the refinement pathways that may be difficult to sample without a biasing potential or by using a single short simulations. The detailed analysis of the refinement pathways in these targets also informed how wide of a restraint is needed so that refinement to the native structure is possible, while still preventing larger unfolding transitions from which it would be difficult to recover.

Based on the analysis of refinement described above, an improved refinement protocol with several new features was devised and tested in the CASP13 experiment. A new type of restraint function was used to admit structural transitions by making it possible to partially unfold and refold. Moreover, we sampled iteratively to expand the discovery of conformational space by learning from sampling in a previous iteration. We also included putative bound ligands for some targets if there was evidence in homologous proteins. Finally, the Rosetta scoring function²⁴ was adopted with the goal to better identify native states if and when they were generated. Other more minor changes involved force field modifications to enable longer simulations with less computational cost and achieve lower energy barriers for dihedral transitions. We also applied an optimized version of our previous protocol in CASP13 in addition to the new protocol to produce more conservatively refined models as a reference.

Overall, our new approach succeeded in extending the conformational sampling, and this resulted in an enhancement of refinement. It was possible to significantly refine a few targets very close to their native structures while still achieving at least moderate refinement for most other target.

In the remainder of this paper, we are describing our refinement protocol applied in CASP13 in more detail and present a comprehensive analysis of its performance.

METHODS

Overview of refinement protocol

The overall method for protein structure refinement used in CASP13 is based on our earlier refinement strategy of averaging MD-generated structure ensembles^{14,16,19,20,25,26} but the protocol has been updated by incorporating new features. As outlined in Figure 1, the CASP13 refinement protocol consisted of three stages: pre-sampling, sampling, and post-sampling. In the pre-sampling stage, initial models were subjected to locPREFMD²⁷, our local refinement method, to quickly remedy stereochemical errors that can potentially cause abnormal MD sampling afterward. We conducted protein refinement with ligands if there were putative binding ligands. Therefore, an additional pre-sampling step involved the prediction and docking of ligand conformations where appropriate. Starting from the processed initial model, two protocols were followed in the sampling stage. The iterative protocol consisted of three iterations of MD simulations with flat-bottom harmonic restraints, followed by clustering, and selection of new initial models for subsequent rounds of sampling. Altneratively, we also applied a conservative protocol analogous to our CASP12 refinement method^20,26, but with reduced simulation time. The refined model from the iterative protocol was submitted as “Model 1” when we had confidence that it was successful. Otherwise, the refined model from the conservative protocol replaced it, and the iterative protocol was terminated after the first iteration. Based on benchmark tests on CASP10–12 targets, we did not have confidence that our new, more aggressive protocol would succeed in the case of bigger targets (with a radius of gyration of the initial model greater than 17 Å), targets with highly unstable initial models, or with targets involving bound ligands. The initial model stability was determined from the ratio of snapshots deviating less than 1 Å Cα-RMSD from the initial model with the conservative protocol. A model was considered highly unstable if the fraction of structures remaining close to the initial model was less than 0.1%. After the sampling stage, the generated conformations were evaluated with the Rosetta score (ref2015).²⁴ Based on the score, a subset of selected structures were selected for each protocol and subjected to averaging and final stereochemical refinement. We submitted five models for each target including local error estimations as described below. The refined models from the main and the conservative protocol were submitted as either “Model 1” or “2” as described above, and ensemble averaged structures of the three largest clusters from the iterative protocol comprised the remainder of the submitted models. Further details of the refinement protocol are described in the following.

Figure 1.

Overall refinement protocol applied by FEIGLAB group in CASP13.

Prediction of bound ligands

Putative bound ligands were predicted by manually considering sequence and structural similarity of homologous proteins with biologically relevant ligands. We searched for homologs with HHsearch²⁸ against the PDB70 database, which has a maximum mutual sequence identity of 70% between proteins deposited in PDB, released on May 23, 2018. Sequence profiles were generated with HHblits²⁹ by searching homologous sequences with the default options against Uniclust30³⁰, which is a clustered UniProtKB database³¹ at the level of 30% pairwise sequence identity, released in September 2016. We predicted bound ligands by considering the structural similarity of detected homologs. We determined a ligand to be bound if it was found to be a biologically relevant molecule and considered to be an essential role in maintaining its bound protein structure. Topology and parameter files for ligands were generated by using CGenFF^32,33. For metal ions, additional distance restraints were applied to enforce their predicted coordination geometry with conserved nearby amino acids.

Iterative sampling via MD simulations with flat-bottom restraints

For the iterative protocol, MD simulations were conducted with flat-bottom harmonic restraints and a modified version of the CHARMM 36m force field³⁴ with explicit water molecules. The force field modifications involved an alternative CMAP term^35,36 where energy barriers in transitions of backbone dihedral angles were lowered manually between left-, right-handed helix and beta-sheet regions in order to accelerate conformational transitions. The free energy landscapes for the backbone dihedral angles in alanine dipeptide with the original and modified CMAP terms are shown in Figure S1 for comparison. Apart from the modified CMAP term, we also re-distributed atomic masses so that hydrogen atoms became heavier (3 a.m.u.)³⁷. This allowed MD simulations with a 4-fs integration time step, thereby cutting computational costs in half for the same simulation time compared to unmodified masses. A flat-bottom harmonic restraint was applied to every Cα atom with respect to their position in a reference structure. In contrast to the harmonic restraints used in our earlier refinement protocols, we did not restrain any sampling up to 4 Å (d₀) from the reference positions. Once the flat-bottom potentials became active, a force constant (k₀) of 0.025 kcal/mol/Ų was applied (see Eq. 1).

$$f\left(\mathbf{r};{\mathbf{r}}_0\right)=\{\begin{array}{cc}0& ,d\le d_0\\ k_0{\left(d-d_0\right)}^2& ,d>d_0\end{array},\text{where}d=\left|\mathbf{r}-{\mathbf{r}}_0\right|$$ (1)

During the first iteration, the initial model was used as a reference structure, and new initial models selected from the previous iteration were used as reference structures for subsequent iterations. MD simulations were carried out by using OpenMM³⁸ to take advantages of GPU acceleration.

The initial protein models were solvated with TIP3P water molecules³⁹ in a periodic rectangular box with at least 9 Å of solvent to the box edge. Either sodium or chloride ions were added to the simulation box to neutralize the system as needed. The Lennard-Jones potential and the electrostatic energy direct term were switched to zero with a switching function between 8 and 10 Å, while particle-mesh Ewald summation was used to calculate the full electrostatic energy in a periodic system.⁴⁰ Bonds involving hydrogen were constrained to be kept rigid with SHAKE method.⁴¹ Langevin dynamics simulations were carried out with a 4-fs time step and a 0.01/ps friction coefficient at 298 K in the NVT ensemble after preparation of the system by local energy minimization and heating up to the simulation temperature. During CASP13, three iterations of MD simulations were conducted with the iterative protocol, and five 100 ns, ten 50 ns, and twenty 50 ns-long MD trajectories were obtained during the first, second, and third iteration. In total, 2 μs of MD simulations were performed and 40,000 conformations were generated per refinement target.

The sampled conformations were clustered by considering both structural similarity and transition kinetics via Markov-state modeling. Initial clustering was done based on Cα-RMSD by using hierarchical clustering with complete linkage. The number of clusters was determined by a 1 Å Cα-RMSD cutoff or 0.05 × the number of conformations, whichever was smaller. A Markov-state model was built by using MSMbuilder based on clustering with a lag time of 5 ns which showed good Markovian behavior for refinement sampling. A transition matrix was calculated by symmetrizing transition frequencies between states. The model was lumped further by considering transition rates between states that have a minimum relaxation time longer than 20 ns. This resulted in states that were kinetically separated but structurally dissimilar only in details. The structure ensemble for each cluster was averaged to obtain a representative structure. In descending order of cluster population, the representative structures were compared with previously used initial models, and a structure was selected as a new initial model and reference where it had more than 1 Å Cα-RMSD from all of the previously used initial models. We selected new initial models from up to five and ten clusters for the second and third iterations, respectively.

Conservative sampling protocol

The conservative sampling protocol followed our previous protocol applied in CASP12^20,26 and involved MD simulations under weak harmonic restraints. Each system was solvated with explicit water molecules and modeled with the CHARMM 36m force field.³⁴ Harmonic restraints with a 0.025 kcal/mol/Ų force constant were applied to every Cα atom to prevent significant structural changes. Langevin dynamics simulations were carried out with a 2-fs time step, while other simulation details were identical to the iterative protocol. Five independent 50 ns-long trajectories were generated, and 5,000 conformations were obtained for a given refinement target.

Ensemble selection and averaging

Sampled structures were scored after the sampling stage. The ensemble selection method was updated with a different scoring scheme compared to previous protocols. During CASP13, we selected conformations by using the Rosetta scoring function (ref2015)²⁴. Sampled snapshots were locally minimized in torsional space by using Rosetta⁴², and the scores were evaluated for the minimized structures. For the iterative and conservative protocol, the lowest 25% and 75% of structures, respectively, were selected to be averaged. Since sidechains in the averaged structure were distorted upon averaging, the averaged structure was locally relaxed via a very short MD simulation with strong harmonic restraints applied to Cα atoms. Averaged structures were first locally minimized, subsequently heated to 298 K, and relaxed for 10 ps with harmonic restraints on Cα atoms relative to the averaged structure with a 1.0 kcal/mol/Ų force constant. Sidechains were reconstructed by using SCWRL4.⁴³ Finally, the overall stereochemical properties were refined with locPREFMD²⁷ before the submission.

Local error estimation

Residue-wise local errors were also estimated via MD simulations and submitted as part of each model. We found that local errors have a high correlation with residue-wise root mean square fluctuation (RMSF) as in previous studies by other groups.^22,44 In CASP13, we ran short MD simulations to obtain RMSFs for residues and directly assigned those values as the predicted local errors. For this prediction, two independent 5 ns-long Langevin dynamics were carried out with the CHARMM 36m force field without any restraints. Other simulation details were identical to the simulation procedures for the conservative protocol.

Unrestrained simulations

To provide a better perspective of how the sampling during CASP13 compared to the full energy landscape between the initial models and native states, additional unrestrained MD simulations were carried out after CASP13 for selected targets. These simulations were started from the experimental structure, the initial model, and states extracted from the iterative protocol. The simulations were performed iteratively and MSMs were built subsequently as described in Heo et al.²³ Detailed information about the additional simulations and the MSMs is given in Table S1. Structure features were extracted from the trajectories after discarding the first 5 ns. We used either residue-wise Cα deviations from the experimental and the initial model structures after superposition or the Cα-Cα distance matrix. Time-structure independent component analysis (tICA)⁴⁵ with a lag time was applied to the extracted features to define reaction coordinates. Clustering in the reaction coordinates system followed to define micro-states by using hierarchical clustering based on the Ward distance. A Markov-state model was built with micro-states, and it was further lumped by the Perron Cluster Cluster Analysis+ (PCCA+) algorithm⁴⁶ to obtain a Markov-state model with macro-states according to a desired transition time-scale. We tested a range of lag times for building Markov-state models and selected 5 ns because it showed a good balance between Markovian behavior and statistics (Figure S2).

Software

Preparation for the MD simulations, pre-/post-processing of PDB files, locPREFMD were performed by the MMTSB Tool Set⁴⁷ with CHARMM⁴⁸. The equilibration and production runs of MD simulations were performed by using the OpenMM³⁸ Python library. Clustering and Markov-state modeling were carried out usingMSMBuilder⁴⁹ and the MDTraj⁵⁰ Python libraries.

The conservative protocol is freely available at PREFMD web server (http://feiglab.org/prefmd). The scripts for the both protocols can be downloaded at https://github.com/feiglab/prefmd, but only the flat-bottom harmonic restraints were included to reflect the iterative protocol since other protocol changes tested in CASP13 were not found to be crucial.

RESULTS AND DISCUSSION

CASP13 refinement performance

Overview

The results of the CASP13 refinement are summarized in Figure 2 and Table S2. “Model 1” submissions were made from selecting refined models either from the iterative or conservative protocol as described in the Methods section. To further compare the performance between the iterative and conservative protocol, we re-generated models with the iterative protocol with the full set of three sampling iterations after CASP13 for targets where the results from the iterative protocol were not used as the “Model 1” submission and where the iterative protocol was terminated after the first iteration during the CASP competition. Considering just “Model 1” predictions, the average improvement in GDT-HA⁵¹ scores was 3.99 units, and 69% of the targets (20 out of 29) were refined. In terms of Cα-RMSD, the average improvement was 0.22 Å and 72% (21 out of 29 targets) were refined. Moreover, side-chains qualities were also improved on average by 4.67 units in GDC-SC⁵² score, and 83% of the targets (24 out of 29) had an improved GDC-SC score after refinement. Twelve out of 29 targets (41%) were significantly improved (> 5 GDT-HA units), while only one target (3%) deteriorated significantly (< −5 GDT-HA units). Since there were only 29 targets in CASP13, we carried out a similar analyses for CASP10–12 targets (102 targets) as a benchmark set with better statistics (see Figure S3). We found similar results for the CASP10–12 targets suggesting that the CASP13 targets and our refinement performance during CASP13 is representative of a large and diverse set of targets.

Figure 2.

Overall performance in CASP13 based on GDT-HA, Cα-RMSD, GDC-SC metrics. “Model 1” submissions (A–C), refined models only with the iterative protocol (D–F), and refined models only with the conservative protocol (G–I) are shown for comparison between each protocol. Solid diagonal lines and dashed diagonal lines with offsets (± 5 for GDT-HA and GDC-SC and ± 0.5 Å for Cα-RMSD) are shown to provide visual guidance.

Overall, we find that it was more likely to refine targets that had an initial GDT-HA between 40 and 70 or an initial Cα-RMSD between 1 and 5 Å (Figure 3). Targets in these ranges of initial quality occasionally improved significantly via refinement. This implies that our refinement protocol has the capability of reaching near-experimental accuracy. Refined models for three CASP13 targets (R0974s1, R0986s1, R1004-D2) reached 0.8, 1.0, and 1.1 Å in Cα-RMSD, and they were improved from GDT-HA scores of 65.6, 59.2, and 60.4 to 89.5, 78.5, and 79.9. This reflects gains of around 20 GDT-HA units. As the backbone structures in these targets reached near-experimental accuracy, side-chains were also simultaneously folded to native conformations., The resulting models have GDC-SC scores of 59.0, 47.6, and 48.5. Similar examples of significant refinement were also found in the CASP10–12 target sets when we re-applied our new protocol to those targets (see Figure S3).

Figure 3.

Refinement performance as a function of the initial model quality for the CASP13 targets (A) and benchmark targets (B). Boxplots show the statistics for each initial model quality bins. For the boxplots, boxes represent the first and the third quartiles, medians are marked with red horizontal lines, whiskers range between the minimum and the maximum data within 1.5 interquartile from the first and the third quartiles. Raw data points are overlaid as grey Xs. Linear regressions of raw data points are shown as black dashed lines, and the corresponding Pearson’s correlations coefficients are shown at the bottom of each box.

Successes and failures

Three out of 29 refinement targets were significantly refined and reached near-experimental accuracies below 1 Å RMSD in CASP13. As shown in Figure 4A, the backbone of the refined model for R0974s1 is almost indistinguishable from the experimental structure. The model had 0.8 Å Cα-RMSD to the experimental structure, and the GDT-HA score for the model was 89.5, an increase of 23.9 GDT-HA units from the initial model. The C-terminal region initially had one more helical turn with poor hydrophobic packings between the helices. With our refinement method, the overpredicted helical turn was resolved, and all of the helices relaxed simultaneously to the native conformation. Similarly, for target R0986s1, the refined model had high similarity to the experimental structure as shown in Figure 4B, with 1.0 Å Cα-RMSD and a GDT-HA score of 78.5. For this target, our refinement method improved the β-sheet region by correcting its curvature and its orientation relative to other parts of the structure. In summary, successful refinement was achieved via translational movement, partial unfolding of secondary structure elements, and overall relaxation.

Figure 4.

Refinement examples for targets R0974s1 (A), R0986s1 (B), R1004-D2 (C), and R1002-D1 (D). Native structures, initial models, and refined models are depicted in yellow, sky blue, and magenta, respectively. Incorrect regions in the initial models that were significantly refined are indicated with blue arrows, while regions that could not be resolved are indicated with red arrows. Buried side-chains are also shown as sticks in (A and B). The structure quality before and after refinement in terms of GDT-HA CαRMSD, and GDC-SC is given below each model.

Although several targets were refined remarkably, more than half of the refinement targets were improved only moderately, and a few targets deteriorated after “refinement”. For example, for target R0986s2 (Figure 4C), the model was improved by 8.0 GDT-HA units and 0.2 Å in Cα-RMSD. However, even though it was the best model in GDT-HA among the “Model 1” submissions, a few errors in the initial models still remained in the refined model. For instance, the C-terminal β-strands were not formed and a helix turn was also missing. The refined model for R1002-D2 (Figure 4D) became actually significantly worse by 13.1 GDT-HA units and increased by 0.4 Å in Cα-RMSD over the initial model. The initial model had high structural similarity to the native structure, but there was a register-shift error in the N-terminal β-strand. During refinement, the hydrogen bonds between the β-strands broke as a first step to correct the error. However, further structural transitions that would have led to the correct β-strand paring did not occur. As a result, models with broken hydrogen bonds that still exhibited the register-shift error were generated. In summary, the examples where refinement did ntot succeed or resulted in structures that were worse than the initial models involved structural errors that require complicated structural changes that could not be accomplished within the limited simulation time.

Iterative vs. conservative protocol

The selection of “Model 1” prediction from either the iterative or conservative protocol provided a better overall performance tthan using either protocol alone. If we had used a single protocol on CASP13 targets, the average improvements in GDT-HA and Cα-RMSD would have been 3.66 and 0.20 Å with the iterative protocol and 2.98 and 0.04 Å with the conservative protocol. Both values are less than the combination of both protocols, 3.99 and 0.22 Å. The gaps between the protocols were more significant on the benchmark test, where we found average improvements of 2.29, 2.14, and 3.11 in GDT-HA and 0.09, 0.02, and 0.08 Å in Cα-RMSD values for the iterative, conservative, and combined protocol, respectively. Generally, the iterative protocol was more aggressive and presented more variable results. On the other hand, the conservative protocol showed more consistent, but moderate improvements. The iterative protocol typically refined better than the conservative protcol, however, it also failed in some cases. Often, when the iterative protocol did not perform well, there was also less coinfidence in its ability to refine based on the criteria of a target being too large, the initial model being poor, or ligands predicted to be present. In those cases, the conservative protocol was applied instead, thereby improving the the overall performance during CASP13 for “Model 1” submissions.

According to the benchmark test, the performance of the iterative protocol depends on the target size and the initial model stability, and these tendencies were generally reproduced with CASP13 targets (Figure 5). The iterative protocol usually worked better than the conservative one. However, it did not perform better for bigger proteins, since the conservative protocol showed very little dependency on the target size. As we observed in CASP12, refinement with harmonic restraints hardly samples diverse conformations. The restraints increase the energy barriers, so they hinder transitions to other conformational states. Therefore, refined models are effectively obtained as a result of only local relaxation of the initial models which can be accomplished even for large targets within 100’s of nanoseconds. In contrast, the iterative protocol uses flat-bottom harmonic restraints that allow much wider sampling so that a number of conformational states can be reached. For smaller proteins, at least some of these states can be sampled effectively and there is a chance that the native state or a state close to it is found. However, the number of conformational states increases exponentially as a protein gets bigger, and the required simulation time grows as well. This makes it difficult to progress significantly from an initial model of a large protein and consequently, the iterative protocol could not outperform the conservative protocol for bigger targets.

Figure 5.

Comparison of improvements in GDT-HA or RMSD values as a function of protein size in terms of the radius of gyration (A, B), initial model stability (C, D), and the number of clusters when simulated with the conservative protocol (E, F). Results for the iterative and conservative protocols are colored in blue and magenta, respectively. Benchmark results are shown as boxplots, and CASP13 results as scattered circles. For details about boxplots, see Figure 3. Outliers are indicated by Xs.

It was better to use the conservative protocol for targets with highly unstable initial models. Those unstable initial models substantially deviated even although they were biased to the initial model by harmonic restraints. Not surprisingly, the deviations were even larger in the iterative protocol with flat-bottom restraints were nothing would keep models from opening up until a distance of 4 Å RMSD from the initial model was reached. Targets with unstable initial models usually had critical structure problems such as severe steric clashes that led to rapid unfolding without restraints. During CASP13, we measured initial model stabilities by using the ratio of sampled structures deviating more than 1 Å Cα-RMSD from the initial model. However, this criterion sometimes prevented actual improvements that initially required larger changes. During post-analysis of CASP13 results, we determined that the number of clusters sampled with the conservative protocol may be a better metric for measuring initial model stabilities instead of a rigid RMSD criterion. The rationale is that a protein will typically visit diverse conformational states when it unfolds. For the benchmark tests, unstable initial models could be discriminated well when the number of clusters exceeds five under the current parameters of the conservative protocol. If a criterion based on the number of clusters was applied to determine initial model stability, the corresponding improvements in the CASP10–12 benchmark set increased from 3.11 and 0.08 Å to 3.30 and 0.09 Å in GDT-HA and Cα-RMSD, respectively. Similarly, if we had used this criterion for CASP13 targets, the iterative protocol would have been applied to three more targets (22 out of 29 targets), and the average improvements in GDT-HA and Cα-RMSD would have become 4.26 and 0.19 Å, respectively.

Key elements of the iterative CASP13 refinement protocol

Wider sampling with flat-bottom restraints

A major change from our previous protocols was the use of flat-bottom harmonic restraints instead of simple harmonic restraints. The impact of the different restraints is illustrated most clearly by comparing the energy landscapes generated with the iterative protocol (that used the flat-bottom restraints), the conservative protocol (with harmonic restraints), and unrestrained simulations started from both the initial model and the native structure (see Methods). An example of this analysis for R0974s1 is shown in Figure 6. Additional targets are shown in Figures S4–S6. The choice of restraints resulted in significantly different free energy landscapes. With harmonic restraints, structural transitions from the initial state to other states did not occur and sampling remained near the initial model (Figure 6C). In contrast, the flat-bottom restraints allowed for more extensive sampling already after the first iteration (Figure 6B), that included the native state for R0974s1 and also R0986s1 (see Figure S4). The energy landscapes with the flat-bottom restraints overlap with parts of the full energy landscape from the unrestrained simulations (Figure 6A). Moreover, the parts that were not sampled with the flat-bottom restraints were generally further away from either the initial model or the native state in the successful refinement cases. Therefore, it is clear that the flat-bottom restraints permitted structure transitions necessary for refinement that were previously prevented by the harmonic restraints, while still restricting sampling far away from either the initial model or the native state.

Figure 6.

Free energy landscapes for target R0974s1 from MSM analysis as a function of the first two tIC coordinates from unrestrained simulations (A), simulations according to the iterative protocol after each iteration (B) and according to the conservative protocol (C). Contour maps are shown for every 0.5 kcal/mol up to 6.0 kcal/mol. Projections of the native and the initial models are marked with blue and black Xs, respectively. States identified from MSM analysis are further characterized in terms of their populations (colored bars) and Cα-RMSD from the native structure (black line) (D). Population bars are colored as follows: blue - unrestrained simulation; green/yellow/orange - iterative protocol after first/second/third iteration; red - conservative protocol.

For R0986s2, which was only partially improved, the conformational sampling with the flat-bottom restraints was in fact not very different than the sampling from the unrestrained simulations (see Figure S5). However, for other targts, such as R1002-D2 (see Figure S6) sampling was as restricted as with the harmonic restraints in the conservative protocol. Significant kinetic barriers may be one reason for why more extensive sampling was not seen, but the fact that the unrestrained simulations sampled much more broadly in some targets (see Figure S6) suggest that the restraints may still be discouraging essential refinement transitions. In the case of R1002-D2, it was necessary to unfold a β-strand in order to correct a register-shift error. Since the β-strand was bound between an adjacent β-strand and C-terminal residues via backbone hydrogen bonds, it would have taken significant structural changes to refine this structure. We tried to resolve this issue via adaptive iterative sampling where intermediate states could serve as new initial models for subsequent simulations, but this failed, apparently because no intermediate state significantly closer to the native state was visited. Nevertheless, in general, the use of flat-bottom restraints greatly improved the balance between allowing refinement while still preventing the unproductive sampling of regions far away from the native state.

Iterative sampling

Sampling via MD simulations was conducted iteratively in CASP13. As mentioned above, the intent was to extend the sampling by reinitiating simulations from intermediate states. Table S3 summarizes how additional iterations contributed to refinement and how conformational sampling expanded away from the initial model. Overall, the additional iterations did not make a significant difference. Sampling did expand relative to the initial model with every iteration and the very best structure that was sampled improved moderately with additional iterations. However, the top 5% structures remained essentially unchanged while the ensemble-averaged conformations after filtering improved only slightly in RMSD and became actually slightly worse in terms of GDT-HA scores after additional iterations. The MSM analysis (see Figure 6 and Figures S4–S6) shows that there is only a slight expansion of the sampled conformational space with additional iterations, both for targets where refinement was successful and unsuccessful. This is consistent with the limited benefit of additional iterations in overall refinement. The lack of further sampling progress during iterations may be related to significant kinetic barriers that would need to be overcome to explore other regions of conformational space closer to the native state.²³ However, it remains unclear whether a better strategy for selecting new starting structures for additional iterations could increase the likelihood of crossing such barriers. We expect that this would be especially important for achieving refinement for the targets that were not significantly refined because sampling remained close to the initial model as in R1002-D2 (see Figure S6). Clearly, this part of the refinement protocol could benefit from further improvements.

Scoring and filtering scheme

We showed previously that the force field used for sampling is in principle accurate enough to identify the native state at the lowest free energy and with the highest population. In fact, the native state was the most populated state for R0974s1 where the sampling reached the native state. For R0986s1, the native state was the second most populated state with only a small difference from the most populated state. However, if the native state is not reached or if the sampling is not sufficient to estimate relative populations between different states, a more practical approach is the use of scoring functions. Previously, we established a protocol where a subset of conformations was selected via scoring and subsequently ensemble-averaged to obtain refined models. In CASP12, the filtering scheme involved RWplus⁵³ and the deviation from the initial models as a conservative strategy to avoid selecting conformations that deviated further not just from the initial model but also the native state.¹⁶ However, as the sampling protocol used in CASP13 allowed for wider sampling, the deviation from the initial model became a less useful criterion for judging whether a sampled conformation is closer to the native state. In CASP13 we used only the Rosetta energy function²⁴ as a filter and selected the 25% and 75% of structures with the lowest energy for the iterative and conservative protocol, respectively. During post-analysis of CASP13, we also assessed other scoring and filtering schemes to compare what may have worked better. We tested five filtering strategies: CASP13 protocol, CASP12 protocol²⁰, CASP13 protocol but using RWplus⁵³ or dDFIRE⁵⁴ instead of the Rosetta score²⁴, and averaging all sampled conformation without any filtering. The results of this analysis is summarized in Tables S4 and S5. For snapshots extracted from the iterative protocol, all of the score-based filtering schemes clearly offered better results than simply averaging all of the sampled conformations, however, the benefit of filtering was not as significant when snapthosts were taken from the conservative protocol, presumably because the distribution of structures was narrower.

The Rosetta score was used in CASP13 based on previous benchmark tests, but the actual performance was very similar to other scoring functions. It appears that the use of dDFIRE or even the use of the CASP12 protocol, where the deviation from the initial model is included as well, could have offered a slight advantage (see Table S4). However, rigorous statistical analysis suggests that all of the approaches are essentially equivalent in terms of generating refined models within our protocol. We further compared the evaluated scores with Rosetta, RWplus, and dDFIRE for the entire set of conformations in comparison to the more complete set of structures obtained via unrestrained simulations for the four targets R0974s1, R0986s1, R0986s2, and R1002-D2 (see Figure S7). We find that all of the scoring functions largely reproduce a funnel-like energy landscape with the native state at the lowest score. Therefore, scoring was effective when the conformations sampled with the iterative protocol extended to the native state as for R0974s1 and R0986s1. The Rosetta scores appear to exhibit a slightly sharper “funnel-like” energy landscape close to the native state that could be expected to make a difference when structures very close to the native state are compared. However, this has little impact when the generated structures are not close enough to the native state (as for R0986s2 and R1002-D2). Moreover, for the cases where native states were sampled (R0974s1 and R0986s1), the use of the score as a filter to select structures to be ensemble-averaged structures made the exact performance of the scores near the native state less critical. In fact, ensemble-averaged structures for R0974s1 had very similar Cα-RMSDs of 0.82, 0.84, and 0.81 Å with Rosetta, RWplus, and dDFIRE, respectively, and for R0986s1, they were 1.06, 1.05, and 1.06 Å, respectively. In conclusion, filtering and ensemble-averaging of sampled conformations remain a key part of our refinement protocol, but the exact choice of scoring function is not critical.

Refinement with ligands

In CASP13, we included ligands during refinement for three targets. Only one target, R0949, actually bound a copper ion (Cu²⁺), and this information was also provided by the organizers. For the other two targets, R0999-D3 and R1016, we predicted shikimate and a phosphoserine ligands, respectively. However, it turned out that there were no biologically relevant ligands in the experimental structures. Since we were not confident that the inclusion of ligands would be beneficial with the iterative protocol, we chose predictions from the conservative protocol as “Model 1” submissions. To further analyze whether the inclusion of ligands could help with refinement, we also revisited the CASP10–12 benchmark test set that included 15 ligand-binding targets. Prediction results on these targets are summarized in Tables S6 and S7. We find that the conservative protocol worked slightly better in terms of GDT-HA improvements, while the iterative protocol did better in reducing Cα-RMSD values. Predictions were also made without putative ligands on some targets. However, there was little difference between predictions with and without binding ligands. This may not be surprising since the initial models were predicted by using template-based modeling and, therefore, ligand binding information was transferred implicitly from the templates. At the same time, the restraints in the conservative protocol did not allow significant rearrangements of the initial model that already reflected the presence of a ligand. One concern is that we used predicted ligands and their binding poses and their accuracy may not be high enough as a starting conformation for the iterative protocol to succeed. Therefore, the conclusion so far is that there is little benefit to model ligands as part of the refinement protocol.

CASP13 vs. CASP12

We already discussed that the new iterative protocol mostly outperformed the more conservative protocol employed in previous rounds of CASP^14,19,20, with additional improvements when both protocols were combined. Simply in terms of numbers, our performance during CASP13 was also better than during CASP12. However, a more rigorous comparison involves the application of the CASP13 method to the CASP12 refinement targets. We find that while the CASP12 protocol applied to CASP12 targets resulted in an average GDT-HA improvement of 1.61, our new protocol would have given 3.54 units of improvement (see also Figure S8). Furthermore, three targets would have been refined significantly more with the new protocol than what we achieved in CASP12. The new CASP13 method would have been slightly less consistent, improving 28 out of 36 targets (78%) compared to our CASP12 result where 30 targets (83%) were improved. However, there would have been a dramatic increase in the number of targets that significantly improved (by more than 5 GDT-HA units), from 4 to 13 with the new protocol. Using only the conservative protocol, that was similar to the actual protocol applied in CASP12 but used shorter simulations, an average improvement of 1.84 GDT-HA units is found which essentially reproduces the CASP12 results (with a P-value of 0.30 from Student’s paired T-test for the null hypothesis that both methods are equal). It was possible to obtain equivalent performance with short MD simulations because the sampled conformational space was quickly saturated at the beginning of the simulations with harmonic restraints, as we found after CASP12.²⁰ We simulated only 2.25 μs in total for a target during CASP13, and it took around 120 GPU hours on GTX1080Ti for a typical 129 residue target (R0949). In comparison with CASP12, we used 40% less simulation time and only 22% of the computation time that we used during CASP12. Therefore, it can be concluded that there was significant progress in refinement after CASP12 both in terms of refinement progress and computational efficiency due to the new iterative protocol introduced in CASP13.

Initial model dependency on refinement

During CASP12, we found a strong dependence of refinement success on the group that generated the initial model.²⁰ In particular, it was difficult to refine models from the Lee group.¹⁸ This was attributed to the use of a refinement stage very similar to our protocol as part of their prediction pipeline so that further refinement using essentially the same protocol again would not have been productive. We were especially curious to see if the iterative protocol introduced in CASP13 would allow further refinement of these models. It turns out, the CASP13 protocol also could not significantly refine the Lee predictions from CASP12 (see Table S8) although CASP12 models from other groups were refined more significantly with the CASP13 protocol than with the protocol that was applied during CASP12. One reason may be related to the size of the targets. The targets where initial models came from the Lee group tended to be on the larger side whereas the iterative CASP13 protocol offers the most significant advantages for smaller targets.

In CASP13, 16 out of 29 targets came from predictions by the Baker and RaptorX groups. Other targets came from various predictors, each contributing only one or two models. In CASP13, we did not find a significant difference in refinement performance between initial models from RaptorX, Baker, or other groups (see Table S9). However, we note that the two targets (R0986s1 and R0986s2) that came from A7D (DeepMind’s AlphaFold method) were improved significantly with increases in GDT-HA scores by 19.3 and 8.1 units, respectively.

Residue-wise error estimation

Knowledge about how the structural accuracy may vary across a given protein model provides valuable information for directing refinment but is also important when using such models in applications. We devised a very simple approach for estimating residue-wise errors by using the RMSF from short MD simulations that were started from the refined models. During CASP13, the residue-wise errors estimated in this fashion showed good agreement in terms of the ASE (Accuracy Self Estimation) measure.⁵⁵ The average ASE was 87.27 for the Model 1s compared with random error estimates between 0 and 5 Å that give an averaged value of 77.97 (P-value of 8.9×10⁻⁸). Because incorrect regions tend to have poor interactions with neighboring residues, they are more likely to move. We also tested the method using the initial models instead of the refined models and found a slightly lower average ASE value of 86.91, compared with 77.95 for random errors (P-value of 1.0×10⁻¹⁵). We further assessed our method by using measures that are used in the evaluation of the model accuracy category.⁵⁶ In terms of a binary classification, identifying residues deviating more than 3.8 Å, F1, precision, and recall were 0.539, 55.9%, and 64.5% with a cutoff 2.4 Å for predictions, respectively, and the area under the curve (AUC) was 0.867. If the goal is to predict unreliable local regions (ULR)⁵⁷ as a stretch of residues with a tolerance of 2 residues, F1, precision, and recall were 0.202, 18.9%, and 23.0%, respectively. Therefore, while we could distinguish mispredicted residues well, it was more challenging to assign the correct boundaries of ULRs. In general, the predicted ULRs were shorter than the actual ones. Therefore, the RMSF-based error estimates are more useful as local error estimates than identifying overall regions of a protein that require further refinement.

CONCLUSIONS

Refinement via MD simulations continues to be a successful approach. A new protocol introduced during CASP13 led to significant improvements. The main advance was the introduction of a more permissive flat-bottom restraint potential that allowed the sampling of various conformational states instead of simple local relaxations around the initial model. This allowed the native states to be reached for three targets, where the refined models approached near-experimental accuracy. Markov-state model analysis after CASP13 showed that the new restraining procedure better balanced allowing transitions to accessible states while still preventing transitions to states that would involve significant unfolding of the structure. However, the advantage of the flat-bottom restraint potential was realized mostly with smaller proteins as sufficient conformational sampling remains a challenge for bigger proteins. Another advance was the use of an iterative sampling approach in order to assist in overcoming kinetic barriers. However, this part of the protocol did not contribute much to the overall refinement success. Enhanced sampling methods such as replica-exchange MD simulations⁵⁸ and metadynamics⁵⁹ could be used in the future to overcome the remaining significant kinetic energy barriers. However, it is unclear how to choose reaction coordinates that would be most effective for accelerating refinement in the absence of knowing the native structure. Finally, a modified scoring and filtering scheme worked well although post-CASP13 analysis suggests that the exact nature of the scoring function is not so critical. The much more important feature was whether sampling could reach native-like conformations or not. While the improvements in MD-based refinement realized in CASP13 are encouraging, there continues to be room for improvement. It is becoming increasingly clear that conformational sampling is the main limitation and future efforts will need to focus on better sampling strategies to solve the refinement problem.