Quality Assessment of Protein NMR Structures

Abstract

Biomolecular NMR structures are now routinely used in biology, chemistry, and bioinformatics. Methods and metrics for assessing the accuracy and precision of protein NMR structures are beginning to be standardized across the biological NMR community. These include both knowledge-based assessment metrics, parameterized from the database of protein structures, and model vs. data assessment metrics. On line servers are available that provide comprehensive protein structure quality assessment reports, and efforts are in progress by the world-wide Protein Data Bank (wwPDB) to develop a biomolecular NMR structure quality assessment pipeline as part of the structure deposition process. These quality assessment metrics and standards will aid NMR spectroscopists in determining more accurate structures, and increase the value and utility of these structures for the broad scientific community.

Introduction

Nuclear Magnetic Resonance (NMR) spectroscopy, along with X-ray diffraction and cryo-electron microscopy, is one of the three major experimental techniques providing three-dimensional (3D) structures of biological macromolecules. In addition to its unique role in characterizing biomolecular dynamics, NMR is routinely used for structure determinations of small (< 20 kDa) proteins [1-3] and is beginning to be used more routinely for determining structures of larger (20 – 50 kDa) soluble and membrane proteins (e.g. refs [4-7]). NMR-derived structure models can be used interchangeably with models generated by X-ray crystallography in many biological applications. It is therefore natural that many quality assessment metrics are common between the two techniques. In addition, there is a portfolio of metrics that are specific to NMR, which take into account the distinctive features of the NMR data used in the structure determination process.

In this review, we outline some of the metrics in common use for protein NMR structure quality assessment. A large part of our review reflects recently published recommendations of the world-wide Protein Data Base (wwPDB) task forces on validation of biomolecular structures determined by X-ray crystallography [8] and NMR methods [9].

Knowledge-based measures

Knowledge-based (KB) metrics describe how well the structure model conforms to expectations with respect to selected features that can be assessed by comparison with the extensive data base of experimental structures. These include bond length and bond angle distributions, dihedral angle distributions, atomic packing, hydrogen bond geometries, and other geometric features. Ideal values or value distributions are derived from statistical analyses of high-resolution X-ray structures, and are generally consistent with basic principles of biophysical chemistry.

There has been some debate regarding the use of KB information derived from X-ray crystal structures of biomolecules in assessing solution NMR structures. There are often differences in the sample conditions used in determining NMR and X-ray structures, and particular conformations from the distribution present in solution may be selected by the requirements of the crystal lattice. Nevertheless, there is no cogent reason to adopt different KB parameter distributions for assessing solution or solid-state protein NMR structures with respect to those used to assess structures determined by X-ray crystallography. Problems indicated by KB assessments can also be mapped onto the 3D structure to identify local hot spots of structural inaccuracy [10,11].

The most general protein model assessment tools look at residue pair-distribution functions (e.g. PROSA2 [12]) or distributions of hydrophobic and hydrophilic residues (e.g. Verify3D [13]) which are characteristic of native protein structures. These analyses are important first steps in structure validation. However, their value is primarily in identifying severely incorrect folds [14], which rarely result from NMR structure determinations done with high restraint densities. These scores may, however, be important when assessing structures determined from sparse restraint networks. Non-globular protein folds (e.g. coil-coil structures) may also exhibit poor PROSA2 or Verify3D scores even when the models are accurate.

Dihedral angle distributions are the most prominent KB statistic used in assessing protein NMR structures. They are generally reported as Z-scores relative to distributions observed in high-resolution crystal structures, or across all structures that have been deposited in the wwPDB. Dihedral angle distributions are generally reported separately for amino-acid residue backbone and side chains. Backbone ϕ, ψ distributions are generally assessed based on compliance with the Ramachandran plot. Historically, the program ProCheck [15] has been used for this analysis, but recent work using a larger set of X-ray structures determined at high resolution suggests that more accurate assessments can be made using improved backbone ϕ, ψ distribution statistics [8,16]. The assessment of side chain dihedral angle distributions (also referred to as rotamer normality) can be more subtle. Protein side chains have been observed to largely adopt standard rotamer states (g-, t, g+) even when buried in the cores of protein crystal structures. While NMR data can in principle determine accurate side-chain conformations, surface side chains are often dynamically-averaged complicating the interpretation of the corresponding NMR data.

Several tools have been used to assess core atom packing, including the Molprobity [11] program for assessment of overpacking, and both the Molprobity and RosettaHoles [17] programs for assessment of underpacking. Severe atomic overlaps are rarely an issue in NMR structures, unless there are errors in the restraints, because of the use of lower-distance bounds. High-energy contacts can occur due to simplified treatments of van der Waals and hydrogen-bonded interactions, but are generally relieved by energy refinement. Underpacking, however, may be a more general problem for NMR-derived structures than generally appreciated, and methods like RosettaHoles [17] should be a key component of protein NMR structure assessment.

The role of KB-driven energy refinement in determining protein NMR structures, such as KB potentials or fragment libraries [18,19], is also somewhat controversial. In our opinion and experience, appropriate protocols for energy refinement, including the use of KB potentials and fragment libraries, can significantly improve the accuracy of NMR structures (see e.g., [1,3,20,21]), particularly for larger proteins determined with sparse restraint networks [4,5]. Approaches based on molecular dynamics can also often provide appreciable improvements in structure quality [22].

Model vs data measures

Model vs data (MvD) metrics describe how well the NMR structure model matches experimental data. MvD quality assessments include data that have been used in the structure generation process and, where feasible, cross-validation using data that have not been used in structure generation calculations.

NMR Restraint Analysis

The most general form of MvD validation involves comparison of distances and dihedral angles in models with the corresponding experimental restraints. Table 1 provides a standard format used to report such restraint violations [9,20], although other formats are also common in the NMR literature. These metrics provide an overview of global (or average) restraint violation statistics, as well as information on the most significant outliers. Clusters of restraint violations in regions of the 3D structure may indicate errors in the local structure. Methods have been described to convert between restraint formats [20,23,24], allowing initial restraint lists generated for one structure generation program to be used with alternative structure generation programs. The NOE completeness score [20,25] is a useful metric of structural accuracy, assessing the fraction of short distances in the model structure that are consistent with restraint data set.

Table 1.

Summary of KB and MvS structural statistics for bacterial protein Alr2454. This analysis was generated by the PSVS protein nmr structure quality assessment server [10, 14]. These results are adopted from ref. [67].

		Alr2454a
Completeness of resonance assignmentsb:
Backbone (%)		99.4
Side chain (%)		98.3
Aromatic (%)		96.6
Stereospecific methyl (%)		100
Conformationally-restricting restraintsc:
Distance restraints
Total		2478
intra-residue (i = j)		688
sequential (i-j| = 1)		619
medium range (1 < |i - j| < 5)		462
long range (|i - j| ≥ 5)		709
Dihedral angle restraints		162
Hydrogen bond restraints		0
Disulfide bond restraints		0
No. of restraints per residue		25.5
No. of long range restraints per residue		6.8
Residual restraint violationsc:
Average no. of distance violations per structure:
0.1 - 0.2 Å		8.75
0.2 - 0.5 Å		1.85
> 0.5 Å		0
Largest distance violation (Å)		0.35
Average no. of dihedral angle violations per structure:
1 - 10°		8.75
° 10°		0
Largest dihedral angle violation (°)		3.8
NOE Completeness Score		0.692
Model Qualityc:
RMSD backbone atoms (Å)d		0.6
RMSD heavy atoms (Å)d		0.9
RMSD bond lengths (Å)		0.018
RMSD bond angles (°)		1.1
MolProbity Ramachandran statisticsc,d
most favored regions (%)		96.8
allowed regions (%)		3.1
disallowed regions (%)		0.1
Global quality scores (Raw / Z-score)c
Verify3D	0.40	−0.96
ProsaII	0.66	0.04
ProCheck (phi-psi)d	−0.15	−0.28
ProCheck (all)d	−0.03	−0.18
MolProbity clash score	12.51	−0.62
RPF Scorese
Recall / Precision	0.976	0.934
F-measure / DP-score	0.955	0.817
Model Contents:
Ordered residue ranged	1-100
Total no. of residues	108
BMRB accession number:	17965
PDB ID:	2LJW a

^aStructural statistics computed for an ensemble of 20 structures.

^bComputed using AVS software [68] from the expected number of assignable resonances, excluding: highly exchangeable protons (N-terminal and Lys amino groups, Arg guanido groups, hydroxyls of Ser, Thr, Tyr), carboxyls of Asp and Glu, non-protonated aromatic carbons, and the C-terminal His₆ tag. Methyl protons are counted as a single assignable resonance.

^cCalculated using PSVS 1.4 [10]. Ramachandran statistics were calculated by Molprobity [11]. Average distance violations were calculated using the sum over r⁻⁶ for degenerate protons and stereochemically-distinct protons lacking stereospecific assignments.

^dBased on “well defined” residue ranges [S(phi) + S(psi) > 1.8].

NOESY Data

Restraint analysis has the significant shortcoming that the restraints are themselves interpretations of NOESY and other NMR data. Accordingly, NMR structure quality assessment should also include some metrics validating models against uninterpreted spectral data. In the case of NOESY data, several methods have been developed for back-calculating NOESY spectral data (e.g. refs [26-28]), although to date none of these has come into general use. A more rapid, though approximate, approach is to compare models with unassigned NOESY peak lists. The RPF program [29] quantifies the agreement between the NOESY peak list, chemical shift assignments, and NMR models by calculating recall, precision, and F measures, as well as a normalized F-measure (called a Discriminating Power DP score). The normalized DP score is highly correlated with the accuracy of the NMR model [30,31].

RDC, Scalar Coupling, Paramagnetic, and SAXS Data

MvD metrics could also include assessment of scalar coupling, residual dipolar coupling (RDC), chemical shift anisotropy (CSA), paramagnetic resonance enhancement (PRE), paramagnetic pseudo-contact shift (PCS), solid-state dipolar coupling, and small angle X-ray or neutron scattering (SAXS or SANS ) data. Several tools for validating structures against these data are available, including methods for validation of protein structures against RDC data [32-34], CSA data [35] and SAXS data [36]. Residual dipolar coupling data provide information about the orientations of internuclear bond vectors with respect to the molecular orientation tensor, and hence provide long-range information including, for example, the relative orientations of secondary structure elements, such as helix tilt angles, which are sometimes inaccurate in protein NMR structures. The most commonly used MvD metric for these data is the RDC Q-factor [37]. Aromatic RDC's also have been found to provide an important approach for validating accuracy in core structures of proteins [38].

Free R Factors

In spite of extensive discussions and of various formulations that have been proposed [25,27-29,37,39], a cross-validation metric analogous to the free R-factor of X-ray crystallography has not yet been broadly adopted by the biological NMR community. This is attributable, at least in part, to the sparseness of NMR data, and the need to manually evaluate each individual peak in NOESY spectra when applying truly quantitative methods. The RDC free Q-factor [37] is potentially more accessible, as quantifying RDC data is generally more straight forward than the quantification of large numbers of NOESY peaks.

Chemical Shift Data

Chemical shift data have great potential for NMR structure validation. The most straightforward validation involves comparing experimental chemical shift data with values predicted from the 3D model structure [40-47]. Extensive chemical shift data must be obtained at the onset of any protein structure determination, and chemical shift data often used indirectly as restraints on dihedral angles [48]. Several recent papers have described important progress in calculating chemical shifts from molecular models using both empirical [49-52] and quantum chemical [44,47,53,54] approaches. However, systematic, large scale tests using chemical shifts as a standard metric for protein NMR structure validation are not yet available.

Which parts of the structure should be validated?

Protein NMR structures are generally represented by ensembles of conformers having the same level of agreement with the experimental data. Different regions of the structure are often converged to different degrees among conformers. In common practice, a distinction is made between well-defined and ill-defined (i.e., not-well-defined) regions [9,20]. In single-domain proteins, the backbone or all-heavy-atom root mean square deviation of coordinates (RMSD) computed after superposition of all the well-defined regions is taken as a measured of structure precision. Similarly, the per-residue RMSD is taken as a measure of local precision. Alternative measures of local or global precision include dihedral angle order parameters (DAOP) [55-57] and distance variance matrix methods [57-60] ( Fig. 1). Often, there is a good, albeit qualitative, correlation between local restraint density and local precision, implying that the ill-defined regions of the structure result from the experimental data providing insufficient information. This variability can be due to local protein dynamics (so that locally a single conformation in solution in fact does not exist) or to experimental factors limiting the information that can be extracted from spectra (e.g., extensive resonance degeneracy).

Fig. 1.

Comparison of DOAP and variance distance matrix results for identifying well-defined atom sets. (A) NMR structure ensemble superimposition showing residues defined as “well defined” by the PDBStat DAOP analysis [20] and those identified as “well defined” by the FindCore [20,58,59] variance matrix analysis for protein SgR42 (PDB_id 2jz2). Residues identified as “well defined” by both methods are shown in dark blue, those identified as “well defined” by variance matrix but not by DAOP in light blue, and those identified as “well defined’ by DAOP but not by variance matrix in green. Residues that identified as “ill defined” by both methods are shown in yellow. The backbone atoms of the corresponding X-ray crystal structure (PDB id 3c4s) are shown in red. (B) Expansion showing atom-specific “well defined” (dark blue) and “ill defined” (yellow) designations for the sidechains of residues Phe20, Phe36, and Phe47 in protein SgR42. This image demonstrates the value of atom-specific designations of well-defined regions of a protein NMR structure. Adopted from ref [20].

The precision of NMR structures indicated by the convergence across the ensemble of NMR conformers is operational, and does not provide a true representation of the Boltzmann distribution of conformations actually present in the NMR sample. Indeed, NMR structure ensembles do not even provide a statistically meaningful description of the true precision of coordinates given the experimental uncertainties in deriving distance restraints. For example, fast exchange may give rise to inconsistent restraints, which when simultaneously satisfied can pin the local conformational distribution into an unrealistically narrow range of conformations [61].

Because NMR experiments do not provide enough data to characterize them, the conformations observed in ill-defined regions largely result from of the combination of random initial conformations with the potential energy functions. Hence, these regions should not be included in global structure validation. However, even such ill-defined regions may be restrained to some degree by sparse experimental data, which may be biologically relevant. Ill-defined regions may also contribute to the back-calculation of NMR observables (e.g. NOESY peak lists). Hence, in our opinion all atoms for which experimental data are available (including only chemical shift data) should be included in the atomic coordinates that are deposited in the PDB, consistent with recommendations of the wwPDB NMR VTF [9].

Protein structure quality assessment servers

Recently, several on-line servers have become available which integrate both KB and MvD assessment metrics to provide a comprehensive NMR structure quality assessment report. Web-based tools described in Table 2 include the CiNG [62,63], Molprobity [11], PSVS [10,14], Vivaldi [64], and ResProx [43] servers. These, as well as other programs for NMR structure quality assessment, have been critically reviewed in a recent publication [57]. The wwPDB NMR VTF has also recommended metrics and standards for biomolecular structure quality assessment [8,9], and these guidelines are being used in developing software pipelines that will generate standardized structure quality reports for all NMR and X-ray crystal structures submitted to the wwPDB.

Table 2.

Web accessible servers providing Knowledge-Based (KB) and/or Model vs Data (MvD) protein NMR structure quality assessment reports.

Server	url	Validation statistics provided
CING [62]v.1.0: Common Interface for NMR Structure Generation	http://nmr.cmbi.ru.nl/cing/Home.html	KB: Bond lengths and bond angles, backbone Ramachandran distributions, peptide omegas, packing, sidechain rotomer normalilty, disulfides, salt-bridges, chemical shift validations. Provides residue-specific ROC scores [69]. Uses DSSP [70,71], WHATIF [72], PROCHECK [15,73], and BMRB chemical shift validation.
		MvD: DOAP analysis. Restraint violation analysis, redundancy and duplicated restraints, back calculation of chemical shift from structure. Uses SHIFTX [49] and VASCO [74].
Molprobity [11]	http://molprobity.biochem.duke.edu/	KB: Backbone Ramachandran distributions, peptide omegas, packing, H-bond satisfaction, sidechain rotomer normalilty.
		MvD: RDC analysis with RDCvis[75].
PSVS [10,14] v.1.5. Protein Structure Validation Software suite.	http://psvs.nesg.org/	KB: Bond lengths and bond angles, backbone Ramachandran distributions, peptide omegas, packing, sidechain rotomer normality, chemical shift validation and completeness using AVS [68]. Provides Z scores relative to high-resolution crystal strutures for ProsaII [12], Verify3D [13], Procheck_bb [15,73], Procheck_all [15,73], and MolProbity [11]. Also uses DSSP [70,71], PDBStat [20], , LACS [76], and PDB Clash score (deposit.rcsb.org/validate).
		MvD: DAOP or Variance Matrix analysis. Restraint violation analysis, redundancy and duplicated restraints, NOE completeness, NOE DP scores, RDC Q scores, and GLM-RMSD [31] “equivalent resolution” score. Provides mapping of Ramachanran outliers and RPF violations onto 3D structure. Also uses PDBStat [20], FindCore [20,58,59], and RPF [29,30] software.
ResProx [43] Resolution by Proxy	http://www.resprox.ca/	KB: Assesses “equivalent resolution” based on 25 protein features including backbone Ramachandran distributions, peptide omegas, H-bond geometry, over and underpacking. Uses Vader [40], PROSESS [42], Molprobity [11], RosettaHoles [17], and GeNMR [41] software.
		KB: Bond lengths and bond angles, peptide omegas, chemical shift validations, backbone Ramachandran distributions, CING [62] ROC scores. Uses WHATIF [72], CING [62], and VASCO [74].
Vivaldi [64] v.1: Visualization and validation of biomacromolecular NMR structures from the PDB	http://www.ebi.ac.uk/pdbe/vivaldi/	MvD: DAOP and Variance Matrix analysis. Restraint violation and RDC analysis. Uses NMRCore [77]

Quality measures vs. structure accuracy

Satisfaction of KB metrics is a necessary but not sufficient criteria for validating the accuracy of protein NMR structures. This conclusion was illustrated by the recent Critical Assessment of Protein Structure Determination by NMR (CASD-NMR) [1,2], in which NMR data was publicly released, and the results of automated NOE assignment and structure generation methods were compared in a blind fashion with manually-refined protein NMR structures. In many cases, models with excellent energetics were in fact quite different from the manually-refined “gold standard” structures. However, energetic considerations can identify inaccurate restraint data, and thus guide the experimentalist to more accurate interpretations of the raw NMR data in terms of experimental restraints [20,21].

The CASD-NMR experiment also compared correlations between several KB and MvD metrics with structural accuracy. In this analysis, the DP score was observed to provide the highest correlation between submitted models and the manually-refined target structure (Fig. 2). Although the DP score has significant shortcomings to the extent that it uses an interpreted NOESY peak list rather than raw NOESY spectral data, these results demonstrate the importance of combining KB and MvD validation scores in structure quality assessment. For example, the DP score has recently been combined with several KB scores by linear regression analysis to provide an “equivalent resolution” metric [31,65].

Fig. 2.

Correlation between accuracy measures (backbone RMSD to the reference structure and GDT_TS score) and the DP-score. These results demonstrate the discriminating power of the DP score in distinguishing accurate from less-accurate protein NMR models. The horizontal and vertical lines indicate empirical thresholds for accurate structure models. Adopted from ref [30].

Areas of current research

The trend towards determining larger biomolecular structures using sparse NMR data and hybrid methods incorporating small angle X-ray scattering, chemical cross links, and other data, requires increasing use of KB information and potential energy functions in the structure determination process. Some solid-state NMR structure determinations may also rely on relatively sparse networks of experimental data. These methods provide less independent information for use in cross-validation, creating important challenges to the field in terms of robust structure quality assessment.

Quality assessment considerations for NMR structures of nucleic acids and carbohydrates, while generally similar to those for proteins, has not yet been extensively investigated. Methods also need to be developed for generating Boltzmann ensembles of conformers that best satisfy the experimental data (e.g., [37,66]), rather than fitting all the data to a single conformer multiple times, as is the common practice. Ensemble-averaging interpretations are particularly important in highly dynamic regions of a biomolecular structure, and new methods are needed for quality assessment of the ensembles of models proposed for such disordered regions.

Conclusions

A high quality NMR-derived structure should meet minimal standards based on a wide range of KB and MvD validation assessment metrics. No single metric score is sufficient to validate a protein NMR structure. Users of biomolecular NMR structures also need to be informed about which parts of the structure are well-defined by the data, and which are less-reliably defined in terms of a unique structure. This information can be critical in interpreting structure-function relationships.

Structure determinations by NMR are often marginally underdetermined, and rely in either subtle or substantial ways on KB information, including at the very least standard values of bond lengths and bond angles. Although they are useful for biological studies, and in some cases quite accurate, the relatively low density of data constituting a typical protein NMR structure presents challenges to cross validation, using part of the data to generate the structure and another part of the data to validate the structure. In this regard, chemical shift data, which are available for most atoms reported in modern biomolecular NMR structures, hold promise for providing a robust and general approach for data-based protein structure quality assessments.

Highlights – Montelione.

• - Model quality is assessed with Knowledge-Based (KB) and Model-vs-Data (MvD) metrics

• - KB metrics compare structural features with the database of protein structures

• - MvD metrics compare NMR-derived models against experimental NMR data

• - Assessment requires annotation of well-defined regions the protein structure model

• - NMR structure quality assessment provides new opportunities for methods development