On the relationship between cumulative correlation coefficients and the quality of crystallographic data sets

Abstract

In 2012, Karplus and Diederichs demonstrated that the Pearson correlation coefficient CC_1/2 is a far better indicator of the quality and resolution of crystallographic data sets than more traditional measures like merging R‐factor or signal‐to‐noise ratio. More specifically, they proposed that CC_1/2 be computed for data sets in thin shells of increasing resolution so that the resolution dependence of that quantity can be examined. Recently, however, the CC_1/2 values of entire data sets, i.e., cumulative correlation coefficients, have been used as a measure of data quality. Here, we show that the difference in cumulative CC_1/2 value between a data set that has been accurately measured and a data set that has not is likely to be small. Furthermore, structures obtained by molecular replacement from poorly measured data sets are likely to suffer from extreme model bias.

Introduction

In his classic note on regression and the theory of evolution in 1896,1 Karl Pearson introduced the statistic now known as the Pearson correlation coefficient (CC), and it has been widely used in the social and behavioral sciences ever since.2 The Pearson CC of the data obtained when the same set of observations are made twice (x_j,y_j) is given by the following expression:

$$CC=\frac{{\sum }_j\left(x_j-〈x〉\right)\left(y_j-〈y〉\right)}{\sqrt{{\sum }_j{\left(x_j-〈x〉\right)}^2{\sum }_j{\left(y_j-〈y〉\right)}^2}}$$ (1)

The reflective CC, or CC₀, is obtained when the same expression is evaluated with the mean values replaced by zero [Eq. (2)].

$${CC}_0=\frac{{\sum }_j\left({x_{j}y}_j\right)}{\sqrt{{\sum }_j{\left(x_j\right)}^2{\sum }_j{\left(y_j\right)}^2}}$$ (2)

Both Pearson and reflective CCs can provide useful insights into the information content of experimental data sets and the reproducibility of measurements.

Although some crystallographic applications of reflective CC₀ were reported in the 1960s,3, 4 until recently, they were seldom used. Those early studies demonstrated that CC₀ of the amplitudes of crystallographic data sets are not good measures of overall data quality because the range of values possible for that statistic is so small. If two non‐centrosymmetric data sets are being compared, the value of CC₀ will be 100% if the two are identical, but fall only to 78.5% (= π/4) if the two are completely unrelated (see Supporting Information Materials). These limits hold for the CC₀ values obtained within thin shells of constant resolution, and for cumulative (or overall) CC₀ values. Another reason correlation coefficients were slow to gain a foothold is that data quality is seldom an issue in small molecule crystallography. The data are usually accurately measured, and the models derived from them often account well for them [e.g., Ref. 5].

In structural biology the situation can be quite different. The data are often harder to measure well, and the models inferred from them commonly fail to explain the data as accurately as they were measured.6 As Karplus and Diederichs pointed out a few years ago, the Pearson CCs can play as useful a role in this arena,7 as they do in electron microscopy (EM), where CC₀ (i.e., Fourier Shell Correlation) have been used for years to estimate the resolution of the EM maps.4, 8, 9

The CC_1/2 is particularly useful for detecting the presence of weak signals in the high‐resolution shells of crystallographic data sets. The measurements used to estimate the intensities of each reflection are partitioned into two non‐overlapping sets of equal size that are then used to obtain two independent sets of intensity estimates. CC_1/2 is the Pearson correlation coefficient obtained by comparing these two sets of intensities. As mentioned earlier, the calculations are usually done after the two sets of intensities have been divided into thin shells of increasing resolution so that the dependence of CC_1/2 on resolution can be determined. At low resolution, the CC_1/2 of crystallographic data sets is usually close to 100%, and it tends to fall off quite rapidly as the resolution limit of the data is reached.7, 10 Strong reflections in a data set, which are almost always better measured than weak reflections, tend to dominate correlation coefficients, and that is why CC_1/2 is a better tool for detecting signals in noisy data than data quality indexes that treat each Bragg reflection equally, such as the fractional intensity R(diff), which is the average fractional intensity difference between the two half‐data sets. More important, CC_1/2 can identify very weak signals in noisy data because it is reliable and does not involve a scaling issue (and even when scaling is involved, it is scaling independent). In contrast, traditional R‐factors such as R_sym, R_merge, and R_meas (see Ref. 11 for definitions) may fail to do so because their values are sensitive to the linear and Wilson B scale factors applied to the individual images that contributed to the two sets of intensities that are being compared, and they can be hard to determine accurately when the data are noisy.

Results and Discussion

The R(diff) and Pearson CC_1/2 values of crystallographic data sets are related to one another. If the distribution of intensities in an X‐ray diffraction data set obeys Wilson statistics,12 which it will if properly measured, and the noise in those data is Gaussian [OSM & Refs. 7, 11], it can be shown that:

$${CC}_{1/2}=\frac{1}{1+a{R}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}}^2},$$ (3)

where a is a constant that depends only on the symmetry‐related multiplicity, and that, typically, has a value between 1.0 and 2.0. This relationship is independent of both the average resolution of the reflections in each resolution shell and the thickness of those shells, which means that it is as applicable to the CC_1/2 values of entire data sets (i.e., cumulative CC_1/2′s) as it is to the CC_1/2 values of individual resolution shells. R(diff) in this equation is a lower‐bound estimate because it assumes that the intensities being compared can be, and have already been, properly scaled.

The data released recently for four X‐ray free‐electron laser (XFEL) structures of photosystem II (PSII) [5WS5, 5WS6, 5WS0, and 5GTI]13 are complete enough so that one can see whether Eq. (3) applies to real data. As Figure 1 shows, both the CC_1/2 values for thin shells of resolution from these data sets and their cumulative CC_1/2 values conform to the upper bound curve predicted by Eq. (3) (i.e., the a=1.0 curve). Analyses done on other data sets (data not shown), as well as studies published by Karplus and Diederichs11 further support the conclusion that Eq. (3) is generally valid.

Figure 1.

The relationship between intensity CC_1/2 and R(diff) values. The two theoretical limits for the relationship between these two quantities are shown using cyan and magenta solid lines. CC_1/2 and R(diff) values have been computed for four XFEL experimental data sets for PSII (5WS5, black spheres; 5WS6, red; 5WS0, green; 5GTI, blue) as a function of resolution. Those data follow the magenta curve, as do their cumulative CC_1/2 values (see the horizontal and vertical dotted lines having the same colors)

When cumulative CC_1/2 values are compared to the dependence of CC_1/2 on resolution from the same data sets, it becomes obvious that cumulative CC_1/2 values are insensitive measures of data quality. The cumulative CC_1/2 values of the four PSII data sets mentioned above range between 99.4% and 99.7%, even though CC_1/2 falls to ∼ 50% in the highest resolution shell in each of them. The cumulative CC_1/2 value of another, unrelated data set (4LR3),10 which was obtained using conventional synchrotron methods, is only one percent smaller, 98.5%, even though the CC_1/2 value of that data set in its highest resolution shell is much worse, 13.5%.

Problems can arise when cumulative CC_1/2 values fall below ∼ 95%, as is the case for some of the other XFEL data sets reported for PSII.14, 15, 16, 17 For example, the 3.0‐Å resolution data set for 5KAF has a cumulative CC_1/2 value of only 53.2%, and the cumulative CC_1/2 value for the 2.8‐Å resolution data set that corresponds to 5KAI is only 54.2%.17 These cumulative CC_1/2 values imply that the lower‐bound value of R(diff) for these two data sets, as a whole, is likely to be 92% to 93% (Fig. 1)! In two of the other XFEL data sets (4IXR and 4IXQ) that have been reported for PSII the cumulative CC_1/2 values are 66.5% and 79.1%, respectively,15 which suggests R(diff) values that are only slightly better, ∼ 71% and 51%. Thus by normal crystallographic standards, the quality of all of these data sets is very poor.

The four data sets just discussed are similar in quality to many of the other XFEL data sets that have been deposited in the PDB. Diederichs and colleagues have expressed concern about the quality of some of them, and about the wisdom of using cumulative CC_1/2 values as measure of data set quality.7, 18, 19 It has been recommended that individual values at both low and high resolution bins should be reported, instead of just an overall value.11 Nevertheless, cumulative CC_1/2 values are sometimes the only data quality statistics reported [e.g., Refs. 17, 20]. It should be noted that the overall CC value is also being used as a measure of the correspondence between cryo‐EM maps and the atomic models derived from them [e.g., Ref. 21]. This practice too is suspect.

How much structural information can there really be in the data sets that have quality statistics like those of the four data sets mentioned above? Rossmann has asked the same question about other XFEL data sets.22, 23 The reply sometimes given is that the quality of data cannot be as bad as it seems because the electron density (ED) maps obtained from them by molecular replacement “look good,” and the model R‐factors appear acceptable. We have explored this issue computationally in two different ways. First, starting with the 5KAI model for PSII,17 a 2.80‐Å resolution ED map was calculated using model phases, and a set of amplitudes obtained by Fourier transformation of a molecular model that was generated from 5KAI by changing the positions of all protein atoms at random so that the distribution of their displacements is a Gaussian function with σ_{|Δ r |} = 1.038 Å (see OSM). This shifted‐coordinate model is unphysical, and its transform differs from that of its parent structure, on average, by 42.5% in amplitude and by 67.6% in intensity, which is a lot. Nevertheless, the cumulative intensity Pearson CC between these two data sets is 75%, consistent with the arguments made above about the insensitivity of that statistic to data quality. Furthermore, the ED map computed using these coordinate‐shifted amplitudes and model phases is strikingly similar to the one obtained using the measured amplitudes and model phases [Fig. 2(A,B), top panels]. In fact, the fit of the original protein model to the ED map calculated using both the amplitudes and phases obtained from the coordinate‐shifted model is reasonably good, and not obviously worse than its fit to the ED map computed using coordinate‐shifted amplitudes but the original model phases (data not shown).

Figure 2.

Electron density (ED) maps. (A) A portion of the ED map computed for 5KAI using observed amplitudes and model phases with the 5KAI model superimposed. (B) The ED map for the same region as (A) computed by combining amplitudes obtained from a model that was derived from 5KAI by altering the locations of all protein atoms at random by the standard deviation of 1.038 Å with (unaltered) 5KAI phases. The position of the atoms in the partially randomized model used to compute the amplitudes used are shown as colored spheres. (C) The ED obtained for the same region shown in (A) using random permutated amplitudes and phases from the original model. The 5KAI model is again superimposed. The first row shows map features contoured at +1.5σ for an α‐helix next to the oxygen‐evolving complex (OEC) of photosystem II. The lower two rows contoured at +0.5σ for and +1.0σ show map features in solvent channels

In the second test, the 5KAI experimental data were divided into 100 thin shells of resolution, each containing the same number of reflections. A second data set was obtained from the first by assigning measured amplitudes to Bragg indices at random within each shell (see OSM). This procedure is called random permutation, and it has used in the past to investigate the statistical properties of small molecule crystallographic data.24 The average Pearson CC between the parent data and the randomized data in each shell was zero for all intents and purposes: ∼ −0.001 ± 0.021. Furthermore, on average, amplitudes had changed by 55.7% and intensities by 101.9%, which is close to what would be seen if diffraction data sets were compared that had been obtained from two unrelated, non‐centrosymmetric crystals that happened to have the same symmetry and unit cell dimensions (58.6% and 100%).25, 26 Nevertheless, the cumulative Pearson CC between the permuted data set and its parent was 22%, rather than the 0% one might have anticipated (see below). Figure 2(A,C) (top panels) show the same small region of the ED map (ρ_obs) one computed using the original amplitudes and model phases [Fig. 2(A), Supporting Information Fig. S1A], and the other computed using these randomized amplitudes and the original model phases (ρ_chimeric) [Fig. 2(C), Supporting Information Fig. S1B]. Once again, the similarity of the two is striking. This observation should not come as a surprise because similar results were reported for computations carried out using small molecule data over half a century ago.27, 28, 29

What these computations illustrate is the impact that model bias can have on the outcomes of a structure determination that depend on molecular replacement, a problem that has long been recognized, but may still be underappreciated. It should also serve as a warning that the claim that an ED map “looks good” is no guarantee that the data on which it is based are meaningful.

The similarity between ρ _obs and ρ _chimeric maps is easy to understand.

$${\rho }_{\mathrm{o}\mathrm{b}\mathrm{s}}\left(\mathit{r}\right)=FT\left( F_{\mathrm{o}\mathrm{b}\mathrm{s}},{\alpha }_{\mathrm{o}\mathrm{b}\mathrm{s}}\right);$$ (4)

$${\rho }_{\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{c}}\left(\mathit{r}\right)=FT\left( F_{\mathrm{a}\mathrm{l}\mathrm{t}},{\alpha }_{\mathrm{o}\mathrm{b}\mathrm{s}}\right)$$

$$=FT\left( F_{\mathrm{o}\mathrm{b}\mathrm{s}},{\alpha }_{\mathrm{o}\mathrm{b}\mathrm{s}}\right)+FT\hspace{0.17em}[\left(F_{\mathrm{a}\mathrm{l}\mathrm{t}}-F_{\mathrm{o}\mathrm{b}\mathrm{s}}),{\alpha }_{\mathrm{o}\mathrm{b}\mathrm{s}}\right]$$

$$={\rho }_{\mathrm{o}\mathrm{b}\mathrm{s}}\left(\mathit{r}\right)+FT\left[\left(\Delta F\right),{\alpha }_{\mathrm{o}\mathrm{b}\mathrm{s}}\right];$$ (5)

where FT indicates Fourier transformation, experimentally observed structure factors are F _obs and α_obs, F _alt are the randomly altered amplitudes, and ΔF are differences between the experimental and altered amplitudes. The second term in the last line of Eq. (5) is a difference Fourier map. It is well known, of course, that if the amplitudes identified here as F _alt derive from crystals that are isomorphous to those used to produce the F _obs data, and the structures of the molecules in the two crystals are closely related, but not identical, the FT[ΔF,α_obs] map obtained will reveal the specific differences between the two structures.30, 31, 32 However, for the computations just described, the amplitude differences are completely uncorrelated with the structure that gave rise to the model phases. Thus, this component of the chimeric maps should consist of random features that extend throughout the unit cell, including its solvent channels, and bear no relationship to ρ_obs(r) (Fig. 2). That this is in fact the case is evident in the lower panels in Figure 2 that are centered on the solvent region in the crystal of interest, which show a much larger portion of the unit cell than the top panels. The solvent region is almost featureless in the map computed using model phases and the observed amplitudes [Fig. 2(A), lower panels]. It is significantly noisier in the map computed using model phases, but amplitudes derived from a randomly distorted model [Fig. 2(B), lower panels], and it is as feature‐filled as the parts of the unit cell that contain protein in the map obtained with random permutated intensities and model phases [Fig. 2(C), lower panels]. There are other indications that all is not well with the two randomized data sets. First, the R factors that measure the correspondence between the observed amplitudes and the computed data are poor: 42.5% and 55.7%, and the density histograms within the protein‐containing regions of the two maps are obviously abnormal.

It is straightforward to convert intensity R(diff) estimates into estimates of intensity and amplitude R(sigma) values (OSM),4 and when this is done one finds that the intensity R(sigma) values for the 5KAF and 5KAI data sets17 should be 65.1% to 65.8%, and their amplitude R(sigma) values should be 37.6% to 38.0%. In light of what has just been said, it is quite surprising that the amplitude free R‐factors reported for the models derived from these two data sets are significantly lower than these lower bound estimates of amplitude R(sigma) values of the data from which they derive: 30.30% (5KAF) and 29.97% (5KAI).17 How can a model derived from a set of data explain those data more accurately than the data explain themselves? These observations suggest that both models are over‐fitted, and one has to ask why the cross validation procedures used, which are supposed to prevent over‐fitting,33 failed to do so in both cases.

Examples of poor quality data sets can also be found in the electron crystallographic literature. For example, the overall data merging R‐factor34 reported for a data set obtained from proteinase K crystals at a resolution between 21.91 and 1.30 Å was 62.9%. Yet, the authors were able to produce what appears to be an outstanding electrostatic potential (ESP) map of that molecule by molecular replacement, even though the model they used for molecular replacement took no account of the impact that partial charges have on electron scattering factors, which are known to be large.35, 36, 37, 38 In addition, a recent analysis done by Spence and colleagues has demonstrated that a number of electron crystallographic structures are computational artifacts because the influence of model bias had on the ESP density functions on which they are based.39

In the past, structural biologists were more concerned about the quality of their diffraction data than the quantity, and often adopted a very conservative approach40 to this problem by, for example, rejecting all the data they had measured beyond the resolution at which the average signal‐to‐noise ratio dropped to 2.0. The introduction of the CC_1/2 method for evaluating resolution limits has encouraged structural biologists to use high resolution data they would otherwise have ignored, and by so doing, they have been able to arrive at better models.7, 10, 11, 41

Nevertheless, the fact that it is a good idea to use high‐resolution data that have a CC_1/2 value of, say, 50%, does not mean that it is a good idea to use data sets that have a cumulative CC_1/2 value of 50%. The cumulative Pearson CC_1/2 statistic is not very sensitive to the overall quality, as we have demonstrated above, and while we have not found a rigorous way to estimate a lower bound value for this parameter, that bound is clearly greater than zero, and it is easy to understand why. If the average intensities of the reflections in two data sets have the same dependence on resolution, then, even if the two data sets are completely unrelated, their cumulative Pearson CC will be greater than zero because (i) strong reflections predominate at low resolution and weak reflections predominate at high resolution, and (ii) the two uncorrelated data sets are likely to have similar mean intensity values within each resolution shell. It appears from the results described above that the cumulative CC_1/2 of a data set that has been well measured by traditional reproducibility criteria, such as R(diff), R(split), R(sigma), R(merging), R(meas), or R(pim) (see Ref. 11, for definitions), will certainly exceed 90%.

Over the last few decades, molecular replacement has become the preferred method for solving crystal structures in structural biology. It is usually the case that the molecular model used is not a fully accurate representation of the molecules in the crystals of interest, and the hope is that the ED map that ultimately emerges will not be identical to that anticipated for the starting model, and that new insights will emerge when those differences are interpreted. This expectation is unlikely to be satisfied if the quality of the data set under consideration is low because of the overwhelming impact that model‐phase bias will have on outcomes, as discussed above. Indeed, the recent literature provides an example of a failure of just this kind that resulted when an XFEL data set having a resolution of 1.35 Å, and a cumulative CC_1/2 of 81.3% was used to compute a difference map [Figure 6A in Ref. 20].

In conclusion, the CC_1/2 method Karplus and Diederichs devised for assessing data quality was an important advance in macromolecular crystallography.11 By contrast, although related, the cumulative CC_1/2, is a statistic best avoided because it is ill behaved, and less informative than more traditional statistics like R(merge).11 When it is used, it must be supplemented with other relevant statistics.

Conflict of Interest Statement

The authors declare no conflict of interest in publishing results of this study.

Supporting information

Supporting Information