What is the best reference state for designing statistical atomic potentials in protein structure prediction?

Abstract

Many statistical potentials were developed in last two decades for protein folding and protein structure recognition. The major difference of these potentials is on the selection of reference states to offset sampling bias. However, since these potentials used different databases and parameter cutoffs, it is difficult to judge what the best reference states are by examining the original programs. In this work, we aim to address this issue and evaluate the reference states by a unified database and programming environment. We constructed distance-specific atomic potentials using six widely-used reference states based on 1,022 high-resolution protein structures, which are applied to rank modeling in six sets of structure decoys. The reference state on random-walk chain outperforms others in three decoy sets while those using ideal-gas, quasi-chemical approximation and averaging sample stand out in one set separately. Nevertheless, the performance of the potentials relies on the origin of decoy generations and no reference state can clearly outperform others in all decoy sets. Further analysis reveals that the statistical potentials have a contradiction between the universality and pertinence, and optimal reference states should be extracted based on specific application environments and decoy spaces.

Introduction

The design and construction of energy function that has a global minimum in the native state are essential for protein folding and protein structure prediction ^1-3. Since Anfinsen's hypothesis ⁴ was put forward in the 1970s, different types of knowledge-based empirical potentials have developed like mushrooms ^5-7, by virtue of the rapid increase of structure data in the PDB library ⁸. Any aspects of structural features that differ substantially between the set of native and non-native conformations can be used to construct statistical potential ⁹, e.g. the strength of electrostatic interactions, the torsion angle, the exposure of non-polar groups to solvent, and etc. In particular, following the idea of Sippl ^6,10, a variety of atomic-level distance-dependent contact potentials have been recently developed ^9,11-17, and successfully applied to many molecular modeling areas, including fold recognition ^18-20, ab initio folding ^21-26, protein structure refinement ^27-28, 3D model assessment ^12,17,29, protein stability analysis ^15,30, and protein-protein docking ^11,31.

Most of the knowledge-based potentials were derived based on the Boltzmann or Bayesian formulations. For the atomic distance-specific contact potentials, the potential can be written as:

$${\overline{u}}_{i,j}\left(r\right)=-\mathit{\text{RT}}\text{ln}\left[\frac{f_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}{f_{i,j}^{\mathit{\text{ERF}}}\left(r\right)}\right]$$ (1)

where R and T are Boltzmann constant and Kelvin temperature, respectively. $f_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$ is the observed probability of atomic pairs (i, j) within a distance bin r to r+Δr in experimental protein conformations. $f_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ is the expected probability of atomic pairs (i, j) in the corresponding distance from random conformations without atomic interactions, which is so-called reference state. Since most existing statistical potentials use the same size of datasets to calculate $f_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$ and $f_{i,j}^{\mathit{\text{REF}}}\left(r\right)$, the probabilities in Eq. (1) can be replaced by the frequency counts of atomic pairs:

$${\overline{u}}_{i,j}\left(r\right)\approx -\mathit{\text{RT}}\text{ln}\left[\frac{N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)/N_{i,j}^{\mathit{\text{OBS}}}}{N_{i,j}^{\mathit{\text{ERF}}}\left(r\right)/N_{i,j}^{\mathit{\text{ERF}}}}\right]=-\mathit{\text{RT}}\text{ln}\left[\frac{N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}{N_{i,j}^{\mathit{\text{ERF}}}\left(r\right)}\right].$$ (2)

Here, $N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$ is the observed number of atom pairs (i, j) at the distance r in experimental protein structures. $N_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ is the expected number of atomic pairs (i, j) if there were no interactions between atoms. $N_{i,j}^{\mathit{\text{OBS}}}\equiv N_{i,j}^{\mathit{\text{REF}}}={\sum }_r^{r_{\mathit{\text{cut}}}}{N}_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$ is the total number of atomic pairs (i, j) in the structure samples, where r_cut is the cutoff distance.

The statistical potential in Eqs. (1-2) is also known as the potential of mean force. In specific derivations, it needs a clear delineation of distance interval and bin splitting scheme. Meanwhile, it should be clearly defined on what kinds of atoms to be considered, and which set of experimental structures to be used. The most critical step for statistical potentials is the selection of reference states ². In principle, the reference state should be obtained from the statistics of random conformations which lacks of inherent atomic interactions and has the ability to offset the statistical biases from specific sample selections and parameter cutoffs.

There is however no universal way as for the construction of the reference states. Common disposal methods for the reference state calculation can be divided into two categories: one is by analytical assumptions, the other is by statistics but the statistical samples are from native protein conformations or their decoys. Due to the importance, a number of studies have been conducted for assessing the performance of different reference states ^{2,14-15,17,32}. However, because these studies exploited the potentials from the original programs which had been constructed using different databases and programming environments, it remains unclear whether the observed differences in performance is due to the selection of reference state, or due to the technical details of training databases, programming and parameter cutoffs.

Meanwhile, most of the previous assessment studies were focused on the selection of native structures. Since the native structures can never been generated by computer simulations, a more realistic and challenging task is to prioritize the best near-native computer models from the structural decoys. Another critical criterion of the potential development is to examine the correlations of the potential with the similarity to the native (e.g. RMSD, TM-score and GDT_TS)³³, because a better long-range correlation is essential to guide the protein folding simulations from non-native states to the native ones ²⁸.

In this paper, we made a systematical examination of six most-often used reference states, including averaging ⁹, quasi-chemical approximation ¹², finite ideal-gas ¹⁵, spherical non-interacting ¹⁷, atom-shuffled ¹⁶ and random-walk chain ¹⁴. To rule out the dependence of training databases and technical details from original potentials, we reconstructed all the potentials using a uniform dataset by the same programming environment. To establish the generality of the analyses, we applied the potentials to six independent decoy sets, from various resources of template reassembly and ab initio folding, with a comprehensive assessment of both native, near-native structure prioritization and energy-TM-score correlation.

Methods

We constructed six statistical potentials using Eqs. (1-2). As in most of previous potential developments, 167 residue-specific heavy atom types are employed ⁹. The distance cutoff is set to 15Ǻ with a bin width 0.5Ǻ, which results in 30 bins. Atom pairs from the same residue are ignored in our pair-wise potential counting. The constructed potential can be written as a 30×27,889 matrix. In the cases where certain atom pairs are not observed at specific distance bin, the potentials are set to a score corresponding to the least favorable one in the whole potential.

A unified, non-redundant set of experimental protein structures was collected for the construction of various potentials in this study. The protein list is generated from the PISCES server ³⁴, with a resolution cutoff 1.6 Ǻ, R-factor cutoff 0.25 Ǻ, and sequence identity cutoff 20%. Only the structures determined by X-ray crystallography were considered. In addition, protein structures with incomplete, missing or nonstandard residues were excluded, except for the structures that missed residues only in the terminals. The final sample of the experimental structures contains 1,022 protein chains, including 165 α, 100 β, and 713 αβ proteins (others 44 have little secondary structure), which are publicly available at http://zhanglab.ccmb.med.umich.edu/potential/assessment.

The total energy score of a given protein sequence S_q with conformation C_p is calculated by

$$\mathit{\text{Score}}\left(S_q,C_p\right)=\sum _{m=1}\sum _{n=m+1}{\overline{u}}_{i_m,i_n}\left(r_{m,n}\right)$$ (3)

where r_m,n is the distance between mth and nth atoms, and i_m and i_n are the residue-specific atom types, respectively. m and n runs through all the atoms in the protein chain except for those pairs from the same residues.

1. Averaging reference state (RAPDF-REF)

The RAPDF potential was proposed by Samudrala and Moult, which uses an average over different atom types in the experimental conformations to represent the random reference states ⁹. Therefore, $N_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ can be calculated as follows:

$$N_{i,j}^{\mathit{\text{REF}}}\left(r\right)=f^{\mathit{\text{REF}}}\left(r\right)N_{i,j}^{\mathit{\text{REF}}}=\frac{{\sum }_{i,j}{N}_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}{{\sum }_{i,j}{\sum }_r{N}_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}\sum _r{N}_{i,j}^{\mathit{\text{OBS}}}\left(r\right)=\frac{N^{\mathit{\text{OBS}}}\left(r\right)}{N_{\mathit{\text{total}}}^{\mathit{\text{OBS}}}}{N}_{i,j}^{\mathit{\text{OBS}}}$$ (4)

Here N^OBS (r) is the number of observed contacts between all pairs of atom types at a particular distance r. ${\mathit{\text{N}}}_{\mathit{\text{total}}}^{\mathit{\text{OBS}}}$ is the total number of contacts between all pairs of atom types summed over all distance r. The contact numbers from different proteins in the dataset are pooled together to calculate N^OBS (r), ${\mathit{\text{N}}}_{\mathit{\text{total}}}^{\mathit{\text{OBS}}}$ and $N_{i,j}^{\mathit{\text{OBS}}}$. Here, an assumption of N^REF (r)= N^OBS (r) has been taken by the authors. Thus, $N_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ and $N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$ can be derived from the same protein dataset. Although the averaging reference state is easy to calculate, a weakness of the potential is that the contact density distribution for all pairs of atom types is assumed to be the same, which deviates from the reality.

2. Quasi-chemical approximation reference state (KBP-REF)

In the quasi-chemical approach of Lu and Skolnick ¹², $N_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ was defined as:

$$N_{i,j}^{\mathit{\text{REF}}}\left(r\right)=x_i{x}_j{N}^{\mathit{\text{OBS}}}\left(r\right)$$ (5)

where x_k is the mole fraction of atom type k, which is calculated based on the whole dataset. Here it has also the assumption N^REF (r)= N^OBS (r). As a reasonable approximation for reference state, the referential number of atomic pairs (i, j) within certain distance bin is proportional to the mole fraction of atom type i and atom type j. The atomic potential using Eq. 5 was named KBP ¹².

3. Finite ideal-gas reference state (Dfire-REF)

In Dfire potential ¹⁵, Zhou and Zhou exploited a ideal-gas system to simulate the reference state. The number of atom pairs in the system was calculated by:

$$N_{i,j}^{\mathit{\text{REF}},p}\left(r\right)=N_{i,j}^{\mathit{\text{REF}},p}\frac{4\pi r^2\Delta r}{V^p}=n_i^p{n}_j^p\frac{4\pi r^2\Delta r}{V^p}$$ (6)

where V^P is the volume of protein P, $n_i^p$ and $n_j^p$ are the number of atoms of type i and j in the protein, respectively. Since Eq. (6) is from liquid-state statistical mechanics of infinite systems but protein chains are finite systems, to remedy the conflict, the authors assumed that $N_{i,j}^{\mathit{\text{REF}},p}\left(r\right)$ increases in r^α with a to-be-determined constant α. Supposing that ū_i,j (r) = 0 for r ≥ r_cut and $N_{i,j}^{\mathit{\text{REF}},p}\left(r_{\mathit{\text{cut}}}\right)=N_{i,j}^{\mathit{\text{OBS}},p}\left(r_{\mathit{\text{cut}}}\right)$, $N_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ can be written as:

$$N_{i,j}^{\mathit{\text{REF}}}\left(r\right)={\left(\frac{r}{r_{\mathit{\text{cut}}}}\right)}^{\alpha }\frac{\Delta r}{\Delta r_{\mathit{\text{cut}}}}{N}_{i,j}^{\mathit{\text{OBS}}}\left(r_{\mathit{\text{cut}}}\right)$$ (7)

where $N_{i,j}^{\mathit{\text{OBS}}}\left(r_{\mathit{\text{cut}}}\right)={\sum }_p{N}_{i,j}^{\mathit{\text{OBS}},p}\left(r_{\mathit{\text{cut}}}\right)$, and the summation is over all protein structures in the dataset. In Zhou and Zhou's training, α was set to 1.57 and r_cut to 14.5 Å.

4. Spherical non-interacting reference state (Dope-REF)

The Dope potential developed by Shen and Sali used a spherical non-interacting reference state ¹⁷, which considered a sphere with a uniform uncorrelated atom density:

$$f^{\mathit{\text{REF}},p}\left(r,a\right)=\{\begin{array}{cc}\frac{3r^2{\left(r-2a\right)}^2\left(r+4a\right)}{r_{\mathit{\text{cut}}}^3\left(r_{\mathit{\text{cut}}}^3-18a^2{r}_{\mathit{\text{cut}}}+32a^3\right)}& r_{\mathit{\text{cut}}}\le 2a\\ \frac{6r^2{\left(r-2a\right)}^2\left(r+4a\right)}{16a^6}& r_{\mathit{\text{cut}}}>2a\end{array}$$ (8)

where a is the size of the experimental structure sample p. Although protein structure is usually not a sphere, the size a can be defined as the radius of an effective sphere which has the same radius of gyration R_g as the sampled experimental structure, i.e. $a={\sqrt{5/3R}}_g$. We can thus calculate the potential by:

$${\overline{u}}_{i,j}\left(r\right)=-\mathit{\text{RT}}\text{ln}\left[\sum _p{w}_p\frac{N_{i,j}^{\mathit{\text{OBS}},p}\left(r\right)}{f^{\mathit{\text{REF}},p}\left(r,a\right)N_{i,j}^{\mathit{\text{OBS}},p}}\right]$$ (9)

where $N_{i,j}^{\mathit{\text{OBS}},p}={\sum }_r^{r_{\mathit{\text{cut}}}}{N}_{i,j}^{\mathit{\text{OBS}},p}\left(r\right)$, and w_p is the weight of the sampled experimental structure p which is calculated as the ratio between the number of atom pairs in this structure and the number of atom pairs in all sampled experimental structures, irrespective of the pair type. In some extent, the spherical non-interacting reference state can be regarded as an extended version of finite ideal-gas reference state with more theoretical details.

5. Atom-shuffled reference state (SRS-REF)

Unlike the above reference states which are either based on the sampled experimental structures or derived from certain analytical assumption, in the atom-shuffled reference state, all atomic positions were preserved while atom identities were shuffled within each of the experimental structures. $f_{i,j}^{\mathit{\text{REF}}}\left(r\right)$ can be calculated from these shuffled structures, and we wrote it as $f_{i,j}^{\mathit{\text{shuffled}}}\left(r\right)$.

$${\overline{u}}_{i,j}\left(r\right)=-\mathit{\text{RT}}ln\left[\frac{f_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}{f_{i,j}^{\mathit{\text{shuffled}}}\left(r\right)}\right]\approx -\mathit{\text{RT}}\text{ln}\left[\frac{N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)}{N_{i,j}^{\mathit{\text{shuffled}}}\left(r\right)}\right]$$ (10)

The HA_SRS potential developed by Rykunov and Fiser used this reference state ¹⁶. The authors presented three shuffle patterns including residue-shuffled, sequence-shuffled and atom-shuffled. Here, we implemented the last one. The dataset used to generate the shuffled structures is the same as that used to calculate $N_{i,j}^{\mathit{\text{OBS}}}\left(r\right)$. We shuffled every experimental structure more than one million times by randomly exchanging the identity of two atoms.

6. Random-walk chain reference state (RW-REF)

Since the starting point of protein folding is the amino acid sequence, the RW potential developed by Zhang and Zhang used an ideal random-walk (RW) chain of a rigid step length as the reference state ¹⁴. This RW model mimics well the generic entropic elasticity and inherent connectivity of polymer protein molecules and yet ignores the atomic interactions of amino acids. According to the polymer theory in the freely-jointed chain model, the reference probability can be written as:

$$f^{\mathit{\text{REF}},p}\left(r\right)=\int f^{\mathit{\text{REF}},p}\left(r,n\right)\text{d}n={\sum }_{n=1}^{N}4\pi r^2{\left(\frac{3}{2\pi n{l}^2}\right)}^{3/2}exp\left(-\frac{3r^2}{2n{l}^2}\right)\Delta r$$ (11)

where N is the number of residues in the sample protein p, and l is the Kohn length. As is done in finite ideal-gas reference state, given a cutoff distance and assuming $N_{i,j}^{\mathit{\text{REF}},p}\left(r_{\mathit{\text{cut}}}\right)=N_{i,j}^{\mathit{\text{OBS}},p}\left(r_{\mathit{\text{cut}}}\right)$, we can get:

$$N_{i,j}^{\mathit{\text{REF}}}\left(r\right)=\sum _p{\left(\frac{r}{r_{\mathit{\text{cut}}}}\right)}^2\frac{{\sum }_{n=1}^{N}exp\left(-3r^2/2n{l}^2\right)/n^{3/2}}{{\sum }_{n=1}^{N}exp\left(-3r_{\mathit{\text{cut}}}^2/2n{l}^2\right)/n^{3/2}}{N}_{i,j}^{\mathit{\text{OBS}},p}\left(r_{\mathit{\text{cut}}}\right)$$ (12)

The value of l² was set to 460 in the RW potential, under which the potential had the best performance ¹⁴.

Results

We constructed the six potentials based on the same dataset of 1,022 protein structures using the reference models as formulated in Eqs. (4-12). Our evaluations are focused on the ability of prioritization of the native and near-native structures, as well as the energy-TM-score correlations. To establish the generality of the analysis, we apply the potentials to various decoy sets generated from different methods.

1. CASP Decoy Set

First, we evaluate the potentials in the structural models generated in CASP5-CASP8 experiments as collected by Rykunov and Fiser ¹³, which include 143 targets and 2,628 models. Since these structural models were predicted blindly by all CASP participants using the state-of-the-art methods, this set represents the most diverse decoys and the selection of best decoy models has practical use.

Table 1 summarizes the performance results of all six potentials on prioritizing the CASP models. Here and after, we denote the potential based on certain reference as “xxx-REF”. We can see that KBP-REF outperforms all other potentials on most evaluation criteria except for the average RMSD and TM-score of the first ranked models which are slightly lower than Dfire-REF and RW-REF. RAPDF-REF, Dope-REF and SRS-REF have similar performances, and select about 10 more native structures than Dfire-REF and RW-REF; however, these potentials have generally lower correlations than Dfire-REF and RW-REF. The performances of Dfire-REF and RW-REF are similar which have the best average RMSD and TM-score of the first ranked models.

Table 1.

Performance of six potentials in CASP decoys.

Potentiala	Nnatb	Ranknatc	Z-scored	R/TMe	Corrf
RAPDF-REF	90/143	2.0/19.4	1.46	10.88/0.581	-0.46
KBP-REF	107/143	1.6/19.4	1.65	7.60/0.613	-0.63
Dfire-REF	80/143	2.9/19.4	1.23	6.60/0.644	-0.57
Dope-REF	93/143	1.9/19.4	1.50	10.71/0.584	-0.46
SRS-REF	92/143	2.1/19.4	1.44	7.49/0.607	-0.49
RW-REF	79/143	3.2/19.4	1.19	6.56/0.646	-0.56

^aPotentials that we reconstructed from a unified structure dataset by using corresponding reference state models from Eqs. (4-12).

^bThe number of targets with the native structure ranked as first versus the total number of test proteins.

^cThe average rank of the native structures versus the average number of conformations per target.

^dZ-score=(E_average-E_native)/σ, where E_native is the energy of the native structure, and E_average is the average energy of all decoys. σ is the energy deviation of all decoys.

^eThe average RMSD and TM-score to the native of the first ranked models.

^fThe average Pearson correlation between energy and TM-score of decoys.

Take T0233 as a typical example, the correlations from different potentials are varied (Figure 1). Dfire-REF and RW-REF fail to select the native structure while their correlation coefficients are relatively better (but still lower than that of KBP), which demonstrates the potential usefulness of the potentials to guide the folding simulations. In Figure S1-S3 in the Supporting Materials, we show three additional examples from T0137, T0211, and T0423, which have three level of high, medium, and low potential-TM-score correlations, respectively. They have a similar tendency in the energy-TM-score correlations as what we have seen in Figure 1 and Table 1.

Fig 1.

A typical example of energy-TM-score correlation from T0233 in the CASP decoy set, where the energy of each decoy conformation is calculated by 6 different potentials. The native structure is highlighted by the open circles.

2. Ig_structal_hires Decoy Set

Next we applied the potentials to three target decoy sets from the Decoys ‘R’ Us ³⁵, including ig_structal_hires, fisa_casp3 and lattice_ssfit. The ig_structal-_hires decoy set contains 20 immunoglobulin proteins and the decoy structures were built by comparative modeling program segmod ³⁶ using other immunoglobulins as templates. As shown in Table 2, RAPDF-REF performs the best on selecting native structures, while KBP-REF has the highest energy-TM-score correlation with a typical example shown in Figure 2. The average RMSD and TM-score of the first ranked models from RW-REF is slightly better than other potentials. In Figure S4-S6, we present three additional examples of this decoy set with the decoy structures from 1mfa, 1vge, and 7fab, respectively.

Table 2.

Performance of potentials in ig_structal_hires of Decoys ‘R’ Us.^*

Potential	Nnat	Ranknat	Z-score	R/TM	Corr
RAPDF-REF	11/20	5.4/20.0	1.05	2.21/0.945	-0.77
KBP-REF	6/20	6.1/20.0	0.69	2.16/0.945	-0.86
Dfire-REF	2/20	11.1/20.0	0.15	2.14/0.948	-0.81
Dope-REF	10/20	6.3/20.0	0.82	2.32/0.945	-0.80
SRS-REF	10/20	5.8/20.0	0.94	2.20/0.946	-0.79
RW-REF	1/20	11.9/20.0	-0.04	2.11/0.949	-0.80

^*Notations are the same as that in Table 1

Fig 2.

A typical example of energy-TM-score correlation from 1fgv in ig_structal_hires of Decoys ‘R’ Us, where the energy of each decoy conformation is calculated by 6 different potentials. The native structure is highlighted by the open circles.

3. Fisa_casp3 Decoy Set

There are 5 decoy sets in fisa_casp3, and each set contains about 1,400 decoy conformations. The backbone conformations of these decoys were generated by Rosetta program ²¹ which assembled the models using fragments of other solved protein structures; side–chain atoms were then added by SCWRL ³⁷. Since the decoy conformations were from ab initio modeling, most structures have a low TM-score (<0.5). In this low-resolution region, all potentials have an almost negligible correlation with the TM-score. Figure 3 shows four proteins by RW-REF, where the energy-TM-score correlation coefficient is below 0.4 for all protein targets. A similar tendency is seen in all other potentials on this decoy set (see Figures S7-S11).

Fig 3.

Examples of energy-TM-score correlation by RW-REF in fisa_casp3 of Decoys ‘R’ Us. The native structure is highlighted by the open circles.

Probably because the decoys are mainly distributed at low TM-score (far from the native), the native structures in this set are relatively easy to recognize by most potentials. As shown in Table 3, all potentials, except for KBP-REF, can correctly recognize the native in 4 out of 5 targets. The remaining target is from 1b0nB whose native structure has an irregular topology of the extended two-helix bundle which is stabled only when intertwined with the Chain A of the protein. All potentials, ranking on the isolated domain without counting the interaction with Chain A, failed to recognize the native state. The overall ranking and correlation results of fisa_casp3 are listed in Table 3, where the RW-REF performs relatively better than other potentials on every aspect.

Table 3.

Performance of six potentials in fisa_casp3 of Decoys ‘R’ Us.^*

Potential	Nnat	Ranknat	Z-score	R/TM	Corr
RAPDF-REF	4/5	203.8/1439.0	3.45	10.97/0.299	-0.11
KBP-REF	2/5	108.4/1439.0	2.20	11.99/0.294	-0.17
Dfire-REF	4/5	7.6/1439.0	4.62	11.00/0.298	-0.26
Dope-REF	4/5	104.8/1439.0	3.88	11.46/0.265	-0.12
SRS-REF	4/5	133.6/1439.0	3.98	10.97/0.299	-0.14
RW-REF	4/5	4.8/1439.0	4.78	10.70/0.310	-0.28

^*Notations are the same as that in Table 1

4. Lattice_ssfit Decoy Set

The lattice_ssfit decoy set contains 8 small proteins generated by ab initio enumerations of possible conformations in a lattice system ³⁸. Similar to the fisa_casp3, most of the decoy structures have a low TM-score. Thus, the recognition of the native structure is relatively easy and all potentials could recognize the native state of all targets with a high Z-score. Accordingly, there is almost no correlation between energy and TM-score as shown in Figure 4, which was based on RW-REF that has the highest average correlation coefficient. In Figure S12-16, we present examples from other five potentials on the same set of proteins, where a similar correlation range is seen in these potentials. Again, as shown in Table 4, RW-REF outperforms all potentials in all the criteria in this decoy set.

Fig 4.

Examples of energy-TM-score correlation by RW-REF in lattice_ssfit of Decoys ‘R’ Us. The native structure is highlighted by the open circles.

Table 4.

Performance of six potentials in lattice_ssfit of Decoys ‘R’ Us.^*

Potential	Nnat	Ranknat	Z-score	R/TM	Corr
RAPDF-REF	8/8	1/1999.6	5.30	10.42/0.241	-0.07
KBP-REF	8/8	1/1999.6	5.53	10.98/0.237	-0.12
Dfire-REF	8/8	1/1999.6	7.65	9.77/0.248	-0.15
Dope-REF	8/8	1/1999.6	5.39	10.00/0.245	-0.08
SRS-REF	8/8	1/1999.6	5.40	10.52/0.243	-0.08
RW-REF	8/8	1/1999.6	8.30	10.08/0.250	-0.17

^*Notations are the same as that in Table 1

5. MOULDER Decoy Set

We also tested the potentials in the MOULDER decoy sets which were generated by the comparative modeling program MODELLER where close homologous templates have been used to guide the model generations ³⁹. To cover a wider RMSD range, the authors have selected templates with alignments ranging from 0 to 100% of the native overlaps. As shown in Table 5, all six potentials can easily select the native structures for the majority of targets with an appreciable Z-score. The averages of the energy-TM-score correlation also reach to a high level with coefficient >0.75 for all potentials.

Table 5.

Performance of six potentials in the MOULDER decoy sets.^*

Potential	Nnat	Ranknat	Z-score	R/TM	Corr
RAPDF-REF	19/20	5.3/301.0	3.05	4.60/0.746	-0.78
KBP-REF	19/20	2.4/301.0	2.42	4.57/0.750	-0.87
Dfire-REF	19/20	6.0/301.0	2.98	3.98/0.771	-0.88
Dope-REF	19/20	5.3/301.0	3.23	4.34/0.761	-0.79
SRS-REF	19/20	4.3/301.0	3.18	4.32/0.750	-0.81
RW-REF	19/20	4.5/301.0	2.94	4.45/0.752	-0.88

^*Notations are the same as that in Table 1

This high correlation value is partly due to the wider range of the decoy distributions because by definition the correlation coefficient can achieve a higher value in the wider distributed decoys than the narrow distributed ones even with a similar level of decoy fluctuations. Second, the decoy structures in MOULDER were generated by comparative modeling which keeps most of the template structural unchanged. These are different from the decoys generated by ab initio folding which have all structure regions reassembled from scratch. Thus, the statistical potentials, which are all developed from the PDB structure library, may tend to have a better discrimination power on the homology-based decoys due to some level of memory effects.

Among all the potentials, Dfire-REF has a relatively stronger energy-TM-score correlation and recognition accuracy for near-native structures according Table 1, but its performance to recognize the native structures is slightly worse than other potentials. Figure 5 shows four typical examples by Dfire-REF. Indeed, the decoys have a quite uniformed distribution spanning a much larger range than the ab initio folding decoys. The correlation is consequently higher than that in other decoy sets. The illustrated examples for other five potentials are shown in Figures S17-S21.

Fig 5.

Examples of energy-TM-score correlation by Dfire-REF in the MOULDER decoy sets. The native structure is highlighted by the open circles.

6. I-TASSER Decoy Set-II

Finally, we used the I-TASSER Decoy Set-II which have the coarse-grained models first generated by iterative Monte Carlo fragment assembly and then refined by GROMACS4.0 MD simulation ¹⁴. This set represents a typical procedure of protein structure predictions combining template-based modeling and atomic-level structure refinements. As shown in Table 6, the six potentials can select the majority of native structures with discrepancies less than 9. RW-REF outperforms others on all criteria, and Dfire-REF takes second place. The gap between the best and worst performing potentials on energy-TM-score correlation is as high as twenty percent.

Table 6.

Performance of six potentials in I-TASSER Decoy Set-II.^*

Potential	Nnat	Ranknat	Z-score	R/TM	Corr
RAPDF-REF	49/56	23.48/441.2	5.28	6.20/0.545	-0.34
KBP-REF	45/56	34.43/441.2	3.82	5.38/0.549	-0.42
Dfire-REF	53/56	6.79/441.2	5.08	5.23/0.561	-0.52
Dope-REF	50/56	18.54/441.2	5.43	6.12/0.548	-0.35
SRS-REF	49/56	25.68/441.2	5.11	5.72/0.552	-0.38
RW-REF	53/56	2.48/441.2	5.45	5.14/0.568	-0.54

^*Notations are the same as that in Table 1

Figure 6 presents four typical examples of I-TASSER Decoy Set-II by RW-REF. The decoy conformations from 1abv_, 1gjxA and 1vcc_ have low TM-score, which have accordingly a low energy-TM-score correlation value. But in decoy set of 1thx_, the decoy conformations gather into two clusters, one cluster is with TM-score around 0.8 and the other is with TM-score around 0.5. The correlation value for this target is much stronger (-0.84). This data demonstrates again that as a necessary condition, the decoys should cover a broad range of resolution in order to have a high apparent value of correlation coefficient (see Figure 5). In Figure S22-S26, we present the examples of other five potentials on the same set of proteins, where they all have a higher correlation coefficient on 1thx_. This example again demonstrates that a wider range of decoy distribution is necessary to assess the whole range of correlations of the potentials with the quality of the decoys.

Fig 6.

Examples of energy-TM-score correlation by RW-REF in I-TSAAER Decoy Set-II. The native structure is highlighted by the open circles.

Discussion

The importance of reference state

The characteristics of native conformations can clearly show up only when comparing with non-native ones, where the non-native conformations serve as the reference state. Our brains can subconsciously set a reference state for every judgment or evaluation with powerful inertia and intelligence, while computer-based statistical potentials cannot do so. We must design a reference state in advance and integrate it into the formula of potential. The usefulness of statistical potential depends on its ability to distinguish native conformations or find best models from non-native conformations. So the key task for the potential construction is to explore and utilize the structural differences between native and non-native conformations ⁹. As to atomic distance-dependent pair-wise contact potential, what we concern are the differences of atom pair distribution between native and non-native conformations. The distribution of native conformations can be obtained through statistics on the PDB library. The problem is how to get the distribution of non-native conformations, or in other words, how to describe the reference state. Any reference state can only cover a specific conformation space, thus the potential should better be applied to the structures that the reference state can suitably cover. The diverse performances of the potentials on different reference states imply that the potentials are strongly shaped by its reference state.

Statistical reference state versus theoretical reference state

In the six reference states considered here, the averaging, quasi-chemical approximation and atom-shuffled reference states are primarily base on statistics of experimental structures ¹⁴. The statistical samples for averaging and quasi-chemical approximation reference states are directly from experimental protein structures, and atom-shuffled reference state uses a set of shuffled experimental conformations. Since there is no proper non-native dataset exploited, the reference state derived from the native protein structures may not appropriately reflect the conformational sampling of non-native states encountered in real folding simulations. On the contrary, the finite ideal-gas, spherical non-interacting and random-walk chain reference states are from theoretical reference state for they are mainly based on theoretical assumptions and effectively circumvent concrete statistical processes, which are often oversimplified for real modeling procedure. In this context, a reference state considering the statistics of realistic computational simulation decoys is probably essential.

The universality and pertinence of statistical potential

The results presented here show that no potential can always outperform others in different decoy sets. Even in the same decoy set they often rank inconsistently in different evaluation criteria. As described above, the distinction among the six potentials merely reflects in their different reference states, from which their diverse performances consequently arise. No matter how to deal with the reference state, the conformation spaces that different reference states can cover are different. For example, the averaging reference state was based on native structures and can be a suitable representation of near-native conformations; while the finite ideal-gas reference state is based on the assumption of finite ideal-gas and thus can roughly cover a broad conformation space. But what method is the more suitable? If we want the potential to be efficient under a broader application environment, namely that the universality of potential is emphasized, we should calculate the reference state basing on a more general conformation space. However, the pertinence of potential would be compromised while enhancing its universality, and too much emphasis on universality is likely to make the potential perform poorly in any application environment. As for the six reference states we used here, the conformation spaces they can cover are obviously different, which consequently makes the potentials based on them have respective universality and pertinence. It is the distinction on universality and pertinence that makes the potentials perform diversely in different decoy sets. To further enhance the performance of statistical potential, we can envisage the range of application at the beginning of potential construction while not being keen on its universal validity, and calculate reference state based on the specific application environment. For example, if the potential is designed mainly for assessing and refining the conformations produced by certain prediction method, we should probably take a non-redundant conformation set produced by this method as the statistical samples of reference state, and both expanded and narrowed conformation space of the sample structures would have negative impact on its performance.

Calculation procedure of statistical potential

There are two ways that we can choose in the calculation procedure of statistical potential. One is to divide the observed contact numbers in the entire sample dataset by the referential contact numbers first and then take its negative logarithm; the other is to divide the observed contact numbers in a single sample protein by corresponding referential contact numbers and then combine the results over the entire dataset, and finally take its negative logarithm. While the observed contact numbers in a single sample protein would likely be too sparse to allow an effective statistics ¹². We tested the above two ways in the calculation procedure of Dope-REF and RW-REF potentials since both of their reference states are related to the protein size. The result shows that the Dope-REF potentials calculated in two ways perform similarly, but the RW-REF potential calculated in the first way performs much better than that calculated in the second way (detail data not shown). With a view to the conventional calculation procedures related to different reference states, in this paper we used the first way to calculate all the potentials except for the Dope-REF potential, which is calculated in the second way.

Effects of TM-score (or RMSD) distribution of decoy set to evaluation criteria

All decoy sets we used here include the native structures. There are often large gaps on TM-score between the native structures and their decoy conformations, which may partly make the native structure selection much easier than the discrimination of decoys in different accuracy. As shown in the previous section, the criteria related to the native structure selection (N_nat, Rank_nat and Z-score) generally get better values than those related to the discrimination of decoys in different accuracy (R/TM and CORR). We investigated into the TM-score and RMSD distributions of decoy sets and found there are large discrepancies among different decoy sets. When the distributions are narrow and concentrated, R/TM and CORR might be poor. For instance, energy-TM-score correlation calculated in decoy set 1thx_ is much better than that calculated in the other three set in Fig 6, which is clearly linked to the particular TM-score distribution of decoy set 1thx_. Overall, these data indicated that the potentials are merely able to distinguish the decoys in a coarse level and their discriminatory powers remain to be enhanced.

Here, it is important to note that our assessment criteria are more practices-oriented rather than physics-based, although it is important to have the correct reference state that is as close as possible to physics. One reason is that most of the reference states are based on some aspect of physical rules in their original developments, but we do not have an objective criterion to quantitatively assess how close the potentials are to physics. The energy-TM-score correlations and the Z-score of the native structures over decoys, on the other hand, can give a quantitative assessment of the potentials in their ability of assisting protein folding and decoy recognition. These criteria have been widely used in the development and assessment of various statistical potentials ^{9,12,14-15,17}. Second, due to the limit size of the current structural databases, the “physically correct” reference states do not always work the best in practical uses. Although the ideal potential should be both physically and practically sound, here we prefer to choose those that can have best performance in practical applications, when a compromise has to make between them and especially when we do not have a clear criterion to assess the physical correctness of the potentials.

Conclusion

Starting with different reference states, we constructed six atomic distance-dependent pair-wise contact potentials based on a uniform sampling dataset and bin-width procedure. These potentials were assessed by virtue of six independent decoy sets. Overall, the random-walk chain model outperformed others in three sets of decoy sets, while reference states based on ideal-gas, quasi-chemical approximation and averaging sample did so in one decoy set separately. Nevertheless, the performance of the potentials fluctuated depending on the decoy sets. No potential could dominate the structural selection and energy-TM-score correlation in all the cases. Our analyses demonstrate that statistical potential has its universality and pertinence which is decided by the reference state and the decoy sets. The optimal reference state should probably be derived by the consideration of the conformational sampling of specific modeling simulations.

The somewhat contradictory assessment results and especially the performance dependence on decoy distributions indicate that the current mean-force statistical potential developments are far from the true solution (if it exists at all). This result is consistent with the well-established agreement in the community that the single-model based quality assessment method cannot compete with the consensus-based approaches in near-native structure recognitions ^40-44. However, the performance of statistical potentials is still significantly better than the random model selections based on our unpublished data. Recent studies showed that a combination of the single-model potentials with structural clustering can outperform that based on consensus ^45-46, which may represent another promising avenue to the improvement of the single-model statistical potentials.