Two New Heuristic Methods for Protein Model Quality Assessment

Abstract

Protein tertiary structure prediction is an important open challenge in bioinformatics and requires effective methods to accurately evaluate the quality of protein 3-D models generated computationally. Many quality assessment (QA) methods have been proposed over the past three decades. However, the accuracy or robustness is unsatisfactory for practical applications. In this paper, two new heuristic QA methods are proposed: MUfoldQA_S and MUfoldQA_C. The MUfoldQA_S is a quasi-single-model QA method that assesses the model quality based on the known protein structures with similar sequences. This algorithm can be directly applied to protein fragments without the necessity of building a full structural model. A BLOSUM-based heuristic is also introduced to help differentiate accurate templates from poor ones. In MUfoldQA_C, the ideas from MUfoldQA_S were combined with the consensus approach to create a multi-model QA method that could also utilize information from existing reference models and have demonstrated improved performance. Extensive experimental results of these two methods have shown significant improvement over existing methods. In addition, both methods have been blindly tested in the CASP12 world-wide competition in the protein structure prediction field and ranked as top performers in their respective categories.

1. INTRODUCTION

Proteins are macromolecules performing varieties of functions within organisms [1]. The biological functionality of a protein is highly relevant to its 3D conformation [2]. Therefore, knowing the 3D structure of a protein is important for the analysis of its functionality [3]. Current experimental methods to acquire protein 3D structure include electron microscopy, protein crystallography and nuclear magnetic resonance [4]. The experimental approach of determining protein structure is cost prohibitive and laborious [5], which results in a small fraction of known protein sequences to have experimental 3D structures. As of Dec 4, 2017, the UniProtKB/TrEMBL database¹ contains 98,705,220 protein sequence entries, but only 135,519 of which have their structures determined by experiments². Predicting protein structure computationally is essential for obtaining important structural information based on protein sequence data [6]. There has been a steady progress during the last few decades [7] in generating high-quality protein structural models, but these models are often buried in the large number of predictions been generated. How to select good models from the prediction pool is often a bottleneck of the protein structure prediction [8], and the development of model assessment methods has not been as successful as model generation methods [9].

The problem of evaluating the quality of computationally generated models, also known as quality assessment (QA), has received extensive attention [9] and many methods have been developed [10]. Currently there exist three major approaches to tackle this issue: single-model methods, quasi-single-model methods, and multi-model methods [11]. The single-model methods only use the predicted model itself to generate its quality score. Common single-model approaches take advantage of protein physical statistics and machine-learning methods [12]. Well-known methods include QAcon [13], dDFIRE [14], Qprob [15], DOPE [16], SMOQ [17], RW [18], RAPDF [19], OPCU-Cα [20], ProQ2 [21] and DeepQA [22].

Quasi-single-model QA methods use information from known protein structures and/or generate some reference 3D models as the basis for scoring the query model. For example, the MQAPsingleA method [23] first utilizes GeneSilico fold prediction meta-server [24] to generate about one hundred reference models and then scores the query model by calculating the average GDT-TS (Global Distance Test Total Score) value [25], the most commonly used protein structure similarity metric, between the model and reference models. These methods may also take advantage of the information from other single-model QA methods. For instance, the MQAPsingleB score is a combination of 80% MQAPsingleA score and 20% MQAPsingleC score, where MQAPsingleC is a single-model QA method based on linear regression [23].

Finally, multi-model QA methods use information from a set of provided models as the basis for scoring the query model. For example, the well-known naïve consensus algorithm compares a query model with a given model set and outputs the average quality score as the final score [26]. Other methods like MULTICOM [27], FDUbio [28] have also achieved great success in the past CASP competition.

There were some earlier attempts to directly use known 3D structures of similar proteins, commonly called templates, to evaluate a predicted model. The similar proteins are found from the PDB (Protein Data Bank) database³ via sequence-based search. The fragments of the proteins that match well with the predicted model in terms of amino acid sequence information are extracted, and their corresponding 3D structures are used as templates to assess the quality of the predicted model. A well-known method is TASSER-QA [29] that used a sliding window of 9 residues to scan and compare the templates with query model. One problem with this approach of using short fragments is that GDT-TS values are not sensitive enough for such short protein fragments. So TASSER-QA had to use RMSD (rootmean-square deviation) instead. However, even though RMSD can be useful in its own way (e.g. finding the better model in the pool when all of them have GDT-TS greater than 0.95), RMSD values are not strongly correlated with GDT-TS values (In CASP11 stage 2 150-decoy QA dataset, the average of Pearson correlation between GDT-TS and log₂(RMSD) of decoys of each target is just shy of −0.5, which is far worse than mainstream QA algorithms) and they have very different ranges and meanings. RMSD is not a good starting point for estimating the GDT-TS value. Compared to TASSER-QA, the methods proposed in this paper use different techniques in utilizing templates, allow variable template lengths, provide GDT-TS style score calculation, and employ a non-linear score combination.

In this paper, based on the idea of directly using known 3D structures of similar proteins, we have developed two new heuristic QA methods under two different catagories: MUfoldQA_S and MUfoldQA_C. MUfoldQA_S is a quasisingle-model QA method that uses fragments of known proteins with similar amino acid sequences to evaluate the quality of a protein model. MUfoldQA_C is a multi-model QA method that ultilizes the intermediate scores from MUfoldQA_S as weights and the top models from a given input set of models as the reference models to calculate a weighted consensus score for a query model.

The contributions of this work include:

1. Proposed an innovative technique to directly and effectively utilize known protein fragments’ 3D structural information to assess protein model quality, without the necessity of building a full model. Also, this technique addressed the normalization problem caused by the differences in protein fragment lengths and produced GDT-TS based scores without using sliding windows.

2. Proposed a new technique to combine both sequential and structural information of the templates to improve the quasi-single-model QA performance.

3. Proposed a new consensus-based technique that uses structural information from protein templates to differentiate high-quality segments of the reference models from low quality ones so that the method would be robust and performs well even when the size of reference model pool is small or the majority of the models are of low quality, which are the top causes of low performance for consensus-based methods.

4. Implemented two fully automatic QA software tools based on the two new methods MUfoldQA_S and MUfoldQA_C, which were ranked as top performers in their respective categories in CASP12 held in 2016 and achieved impressive performance as No. 1 and No. 2 in one overall QA category⁴.

In the rest of the paper, after a brief description of the QA problem, the two new methods, MUfoldQA_S and MUfoldQA_C are presented in detail. Then, extensive experimental results are presented to show the superior performance of the proposed methods over the best existing methods.

2. PROBLEM FORMULATION

The protein model QA problem is defined as following: for a given protein model (i.e., 3D structure or tertiary structure) predicted based on its amino acid sequence, return a quality score representing its similarity to the native structure. Usually the quality score has the range of [0, 1], with 1 being identical to the native structure and 0 being completely different. The ground-truth measurement of similarity to the native structure being used may vary in different situations, but the most common one is the global distance test total score (GDT-TS). This score is between 0 and 1 with 1 being almost identical. It is calculated by superimposing two protein 3D structures, and computing the percentage of corresponding C-alpha atom pairs whose distance falls within the cut-off value of 1, 2, 4, or 8 ångströms. The final score is the average of these four percentages [25].

3. METHODS

3.1. MUfoldQA_S - A New Quasi-Single-Model QA Method

Unlike existing traditional quasi-single-model QA methods, the new MUfoldQA_S method directly uses structural information from known protein fragments as the basis for scoring the query model without building any reference model. These protein fragments, called templates, are found from the PDB database by using sequence-based search. Given a predicted model to be scored and its corresponding protein sequence, MUfoldQA_S outputs a quality score of the model in the range of 0 to 1, with 1 being the highest quality.

The flowchart of MUfoldQA_S is shown in Fig. 1. It consists of the following five steps.

Fig. 1.

Flowchart of the new MUfoldQA_S method.

1. Acquire Templates. To find proper templates, the target protein sequence is queried in the PDB database to find proteins with similar sequences. Both Blast [30] and HHsearch [31], two of the leading tools in the field, are used to search for desirable protein fragments.

For Blast, first run 3 iterations on NR database with e-value threshold 0.001 to get the checkpoint file, then use this checkpoint file as input to another iteration with e-value threshold of 11,000 on our generated Blast database. The threshold was determined in our previous work to make sure at least some templates can be found even when the alignments are possibly inaccurate [32].

For HHSearch, the multiple sequence alignment (MSA) from Blast alignment result is used as input to search on our generated HHSearch database.

The two sets of search results are then sorted separately based on the following template quality score T:

$$T=\left(3-{\log }_{10}E\right)\cdot I\cdot C$$ (1)

where E is the E-value and I is the percentage of identical sequences returned by Blast or HHsearch, respectively, and C is the ratio of the length of template sequence to the length of target sequence.

The final set of templates is the union of the top templates (e.g., 10) from both result sets. Each template contains some fragments of native protein sequences that is similar to the target protein sequence. The 3D structures of the fragments are retrieved from the PDB database for use in the next step.

1. Calculate Template Pairwise GDT-TS. For each of the protein template (AKA fragment) found in step 1) (e.g. total 20), we extract the 3D coordinates of the matched region from decoy and the template. For example, given the sample Blast output shown in Fig. 2, we extract the 3D coordinates of positions 23–83 of the predicted protein model and positions 484–541 of the 4CY6_C, minus the gap indicated by “-“. Then, the GDT-TS score of the 3D structure of these two is calculated. This step is repeated between each decoy-template pair.

2. Calculate Sequence Based Weights. The weights are calculated based on BLOSUM (BLOcks SUbstitution Matrix) [33], a table containing statistical values describing the probability of a biologically meaningful substitution of one amino acid with another amino acid in protein sequences. For each template, we compare its C-alpha sequence with the C-alpha sequence of the target protein. For each pair of C-alpha atoms in the corresponding position of the two sequences, retrieve the BLOSUM value of them (B) and use the following formula to calculate a heuristic weight W_i,j

Fig. 2.

Example of Blast output

$$W_{i,j}=2^B$$ (2)

where i ∈ m_i, the set of indices of the templates that have valid value at that C-alpha position, and j=1..n, where n is the number of C-alpha atoms in the target protein. In our implementation, considering the target proteins are usually distantly related to the existing ones in the protein structure database, we used BLOSUM45. It is a matrix built using sequences with no more than 45% of similarity, with its values being integers between the range of [−5,15]. Table 1 is an illustration of the weight vector, in which each amino acid (C-alpha) of the template sequence (TS) is compared with that at the same position of the query sequence (QS) and the weight vectors (WV) are calculated based on the above formula.

TABLE 1.

AN EXAMPLE OF A SEGMENT OF A QUERY SEQUENCE AND 5 TEMPLATE SEQUENCES, TOGETHER WITH THEIR WEIGHT VECTORS.

QS	R	R	Q	G	K
TS1	K	E	R	G	Q
WV1	8	1	2	128	2
TS2	K	A	L	G	R
WV2	8	0.25	0.25	128	8
TS3	N	A	Q	G	E
WV3	1	0.25	64	128	2
TS4	F	Q	C	R	R
WV4	0.25	2	0.125	0.25	8
TS5	K	E	A	R	I
WV5	8	1	0.5	0.25	0.125

1. Calculate Local Score. For each C-alpha position in the predicted model, calculate the weighted average score (H_j) using all templates with valid value at that position.

$$H_j=\frac{{\mathrm{\Sigma }}_{k\in m_j}{W}_{kj}{S}_k}{{\mathrm{\Sigma }}_{k\in m_j}{W}_{kj}}(j\in [1,n])$$ (3)

where S_k is the pairwise score calculated in step 2) and m_j is the set of indices of the templates that have valid value at that C-alpha position. If there is not a single valid value at position j among all templates, H_j is set to zero.

1. Calculate Global Score. The global score is the average of local scores at all C-alpha positions.

3.2. MUfoldQA_C - A New Multi-Model QA Method

Multi-model QA methods use information from a set of provided reference models as the basis for scoring the query model. A well-known multi-model QA method is the naïve consensus algorithm that uses a simple average of GDT-TS value between the query model and a given model set. The weakness of this algorithm is that if the overall quality of the model pool is poor or the model set is small, then the consensus score is usually not a good indicator of the quality of a model. To overcome this problem, MUfoldQA_C is designed to first use real protein fragments to score the models and then use these scores to weight different segments in models when calculating the consensus score.

Given a predicted model to be scored, its corresponding protein sequence of length n, and a set of reference models R_q, q=1..r, MUfoldQA_C outputs a quality score of the model in the range of 0 to 1, with 1 being the highest quality.

The flowchart of MUfoldQA_C is shown in Fig. 3. It utilizes MUfoldQA_S (referred to as MQS in Fig. 3) to compute some intermediate scores and consists of the following four steps:

1. Run MUfoldQA_S to calculate Local and Global Scores. First, the quality of the reference models is estimated using MUfoldQA_S. For each reference model, MUfoldQA_S calculates the MQS global score as well as the MQS local scores for each C-alpha position in the model. The local scores H_qj (q=1..r, j=1..n) are used as weights for each C-alpha position in the model.

2. Select Top Reference Models. To find the ones with top quality, reference models are sorted by the MQS global score, and up to 100 top reference models are selected.

3. Calculate Pairwise GDT-TS. Calculate GDT-TS values (Gi, i=1,100) between the predicted model to be scored and each of the top reference models.

4. Calculate Weighted Consensus. For each C-alpha position in the predicted model, calculate the weighted average of G_i using weight H_ij. Then, the final model score is the average of all C-alpha position scores:

Fig. 3.

Flowchart of the new MUfoldQA_C method.

$$\mathit{Score}=\frac{1}{n}{\Sigma }_{j=1}^n\frac{{\Sigma }_i{H}_{ij}{G}_i}{{\Sigma }_i{H}_{ij}}$$ (4)

3.3. Comparison between MUfoldQA_S and MUfoldQA_C

Table 2 compares the similarities and differences between MUfoldQA_S and MUfoldQA_C. They both are the product of the idea that using weighted reference score to estimate the quality of predicted 3D protein models. However, there are a few key differences. The MUfoldQA_S uses raw protein fragments as references while MUfoldQA_C uses full predicted models. MUfoldQA_S uses BLOSUM-inspired score as weight while MUfoldQA_C uses MUfoldQA_S local score as weight.

TABLE 2.

Comparison between MUfoldQA_S and MUfoldQA_C

	MUfoldQA_S	MUfoldQA_C
Similarities	Both are weighted reference score
	Both use template fragments to improve QA results
	MUfoldQA_C uses MUfoldQA_S as one of its steps
Type	Quasi-Single-Model	Multi-Model
Reference Structure	Raw protein fragments	Other predicted models
Weight Calculation	BLOSUM-inspired score	MUfoldQA_S local score

From another point of view, both methods are the realization of the idea of taking advantage of raw protein fragments to improve the protein QA results. These two algorithms are the adaptation of this idea in two different QA algorithm categories: quasi-single-model and multimodel QA methods.

4. EXPERIMENTAL RESULTS

The Critical Assessment of Techniques for Protein Structure Prediction (CASP) experiment is a world-wide competition been held every two years since 1994. It is designed to provide objective evaluation of the state-of-the-art methods for protein structure prediction. In CASP12 of 2016, 188 unique groups from all over the world participated in the competition and 54970 predictions were submitted⁵. Multiple CASP datasets were used in our experiments. In recent CASP competitions, the quality assessment (QA) task consists of two stages. In stage 1, each QA group was given up to 20 selected predicted models ranging from good to bad. In the stage 2, each QA group was given up to 150 top models selected by the naïve consensus algorithm.

4.1. Preparing Database

Although our methods could use any database the Blast and HHsearch are compatible with, we have been maintaining an in-house database [4] and use it to perform all our experiments. It is generated in the following manner:

1. Get a copy of the whole PDB bank and parse their sequences from SEQRES record in the PDB file. The proteins with their sequences shorter than 30 or with too many ‘X’ in the sequences are discarded.

2. Extract PDB chain from each PDB file and save each chain into individual files. If two or more chains share the same sequence, only keep one of them.

3. Generate several features for each PDB chain. 1) Use PSIBlast to generate Blast alignment results and the Position Specific Scoring Matrix (PSSM). The e-value threshold used by PSIBlast is 1,000, which is selected based on out previous research [35]. The purpose of these searches is to get many hits as candidates of similar structural template fragments. Although the high e-values will no longer indicate the statistical significance of these hits, there are weak signals to reflect some similarity between the hits and the query. The data-base used in this step is an old version of the official NR database. 2) Use the Blast alignment results acquired from step 1) to build the multiple sequence alignments (MSA) for each PDB chain in A3M format. 3) Use HHMake to generate HHM file from the A3M file in step 2). 4) Use PSIPred to generate secondary structure prediction for each PDB chain. 5) Use DSSP to generate DSSP file for each PDB chain.

4. Use makeblastdb software to generate Blast database using all the sequences from step 1. The input is all the sequences and the output is the Blast database.

5. Aggregate the generated secondary structure predictions, the DSSP results, and the coordinates of each PDB chain in to one text file for each PDB chain. All those files are in DBInfo database.

6. Aggregate all the HHM files generated in step 3 into a single file and this file is the HHSearch database.

4.2. Effectiveness of Templates Generated by MUfoldQA_S

This set of experiments was done using CASP11 dataset with the PDB database generated on April 30, 2014 for searching the templates, before the start of CASP11 in May 2014, so that the target proteins do not exist in the PDB database. In CASP 11 dataset, there are 77 targets been evaluated.

An innovation of the new methods is to use 3D structure fragments of native proteins to evaluate a predicted model. For this method to work, the protein fragments should cover as much of the target sequence as possible and have sufficient overlap with each other. To verify its combined coverage, we generated templates using both Blast and HHsearch. For each target protein, we selected top 10 Blast results and top 10 HHsearch results. Then, for each C-alpha position on the target sequence, we searched through the combined top 20 templates and counted the number of templates that contained valid 3D structural information (hits) on that specific C-alpha position. For each target sequence, we calculated the percentage of C-alpha positions that had exactly n (0 to 20) hits and averaged them across all targets. The result is shown in Fig. 4.

Fig. 4.

Distribution of number of hits for each C-alpha based on the CASP11 dataset. In our program, the cut-off parameter was set to 20, meaning that even if there were more than 20 hits, we only used the top 20 and put all these cases in the 20-hit category, which is why the percentage is high.

Fig. 4 indicates that, on average, 24% of C-alpha positions were covered by all 20 templates and 57% of C-alpha positions had at least 10 hits. The number rose to 76% for at least 5 hits. Unfortunately, on average, 8% of C-alpha positions were not covered by any template. Without a hit on a particular C-alpha position, no structural information was available, which could negatively impact the accuracy of model quality evaluation. After further investigation into the positions without a hit, we discovered that it mostly happened on the end of the protein sequence, which would not impact the quality assessment results much.

To further investigate this issue, we designed an experiment to see if we can achieve better result when more templates are used. In here, we use the concept coverage to indicate the percentage of target sequence that can find corresponding template in Blast or HHsearch search result. And coverage improvement is calculated in the following manner: consider a target protein sequence of length 50. Template 1 contains a subsequence similar to the target sequence from position 1 to 10. Template 2 contains a subsequence similar to the target sequence from position 5 to 15. These two templates combined covers positions from 1 to 15, with coverage 15/50=30%. Now, if we add another template that covers position 8 to 20, then these three templates combined cover from position 1 to 20, with a new coverage 20/50=40%, which is an improvement from the original 30%.

Fig. 5 and Fig. 6 show the improvement of coverage of templates on each target protein when the number of templates was increased from 10 to 20 using Blast and HHsearch, respectively. Fig. 5 shows the result for Blast: no improvement for 29% target proteins, less than 5% improvement for about 41% targets, and more than 10% improvement for only 13% targets. For HHsearch, the result in Fig. 6 shows no improvement for 36% targets and less than 5% improvement for 39% targets.

Fig. 5.

Distribution of coverage improvement of Blast templates on target proteins when the number of templates was increased from 10 to 20.

Fig. 6.

Distribution of coverage improvement of HHsearch templates on target proteins when the number of templates was increased from 10 to 20.

Increased number of templates may lead to higher template coverage on target protein. However, since the templates have been ranked from best to worst, the quality of the additional templates decreases significantly when the number of templates is moderately larger, which in turn has negative impact on the performance.

With all these aspects considered, we used 10 templates from Blast and HHsearch respectively, which, in our experiments, is a good trade-off between the quantity and the quality of the templates used in our methods.

4.3. Performance Comparison on CASP11 Dataset

To verify if the theory behind the algorithms is sound and the proof-of-concept can produce reasonable results, we have run both of our algorithms on existing CASP 11 dataset (77 targets with up to 20 models in stage 1 QA task and up to 150 models in stage 2 QA task). The ground truth, true GDT-TS of each decoy was acquired from CASP official website (a few missing data points were filled by comparing the decoy against the true structure using TMscore software [36], [37]). The results were then compared with official data of top reference QA methods, including ModFOLD5_single [38] [39], VoroMQA [40], ProQ2, DAVIS-QAconsensus [9], MULTICOM-REFINE [41] and ModFOLDclust2 [42]. As in CASP 11, the performance measurement for a QA method is, given a pool of models of a target protein, the Pearson correlation coefficient between the QA scores of the models generated by the QA method and the GDT-TS values of the models to the target protein’s native structure. The Pearson correlation coefficient is in a range between −1 and 1, with 1 being perfectly correlated.

Fig. 7 shows the performances between MUfoldQA_S and other single-model and quasi-single-model QA methods on CASP stage 1 and stage 2 QA task on each target. In stage 1, MUfoldQA_S performs dominantly better than the other three methods with Pearson correlation coefficient of 0.8170, significantly better than ModFOLD5_single (0.7389), VoroMQA (0.5681) and ProQ2 (0.6589). It is also worth noting that MUfoldQA_S does not exhibit any negative correlation in stage 1. These two factors make it a great QA method when the size of candidate model pool is small. In stage 2, MUfoldQA_S scored 0.4691 in Pearson correlation, which outperformed ModFOLD5_single (0.4040), VoroMQA (0.4142) and ProQ2 (0.3827) by a huge margin.

Fig. 7.

Evaluation of MUfoldQA_S in terms correlation with true GDT-TS on CASP11 targets and its pairwise comparison with other single-model and quasi-single-model QA methods. The numbers in the brackets are average Pearson correlation coefficients toward true GDT-TS.

Fig. 8 shows the performances between MUfoldQA_C and other multi-model QA methods. In stage 1, MUfoldQA_C reached GDT-TS correlation of 0.8462, outperformed Davis-QAconsensus (0.8083), MULTICOM-REFINE (0.8139) and ModFOLDclust2 (0.7426) by a large margin. Similar phenomenon can also be observed on stage 2 results. Also, like MUfoldQA_S, MUfoldQA_C did not exhibit any negative correlation in stage 1.

Fig. 8.

Evaluation of MUfoldQA_C in terms of correlation with true GDT-TS on CASP11 targets and its pairwise comparison with other multimodel QA methods. The numbers in the brackets are average Pearson correlation coefficients toward true GDT-TS.

When comparing Fig. 7 and Fig. 8, it can be seen that Multi-model QA methods generally perform better than single-model and quasi-single model QA methods on larger dataset and MUfoldQA_C performs the best among all eight top methods.

4.4. Performance Comparison on CASP12 Dataset

To obtain an objective performance comparison with all state-of-the-art QA methods, the two proposed methods were implemented and participated in the CASP12 competition in the summer of 2016. All of the evaluation results in Tables 3 and 4 came from the official assessment of CASP12.

TABLE 3.

GDT-TS DIFFERENCES BETWEEN PREDICTED AND OBSERVED IN CASP12 QA STAGE 1, AVERAGE OVER ALL TARGETS (TOP 20 GROUPS)

GR	C	CR	Group Name	AD	#
1	Q	1	MUfoldQA_S	3.602	70
2	M	1	MUfoldQA_C	3.818	70
3	M	2	Davis-EMAconsensus	5.615	70
4	M	3	FDUBio	5.756	70
5	Q	2	ModFOLD6	5.883	70
6	Q	3	ModFOLD6_cor	6.697	70
7	M	4	ModFOLDclust2	6.878	70
8	S	1	Wang4	7.021	70
9	M	5	Wallner	7.272	70
10	S	2	Wang2	8.021	70
11	S	3	ProQ3_1_diso	8.155	70
12	S	4	VoroMQAsr	8.275	70
13	S	5	ProQ3_1	8.449	69
14	S	6	VoroMQA	8.488	70
15	M	6	naive	8.507	68
16	M	7	iFold_2	8.507	68
17	M	8	DeepFold-Boom	8.507	68
18	M	9	Deepfold-Contact	8.507	68
19	M	10	Pcomb-domain	8.560	70
20	M	11	QASproCL	9.107	70
			...

TABLE 4.

GDT-TS DIFFERENCES BETWEEN PREDICTED AND OBSERVED IN CASP12 QA STAGE 2, AVERAGE OVER ALL TARGETS (TOP 20 GROUPS)

GR	C	CR	Group Name	AD	#
1	M	1	FDUBio	5.173	70
2	M	2	MUfoldQA_C	5.512	70
3	Q	1	ModFOLD6_cor	6.748	70
4	M	3	Davis-EMAconsensus	6.781	70
5	Q	2	ModFOLD6	7.087	70
6	M	4	ModFOLDclust2	7.093	70
*	S	*	GOAL	7.910	4
7	M	5	iFold_2	8.373	70
8	M	6	Deepfold-Contact	8.373	70
9	M	7	DeepFold-Boom	8.373	70
10	Q	3	MUfoldQA_S	8.898	70
11	M	8	naive	9.210	70
12	M	9	QASproCL	9.664	70
13	M	10	Wallner	9.710	70
14	Q	4	ModFOLD6_rank	9.754	70
15	M	11	Pcomb-domain	9.839	70
*	-	*	COFOLD_QA	9.877	5
16	S	1	ProQ3_1	10.155	70
17	S	2	ProQ3_1_diso	10.159	70
18	S	3	ProQ3	11.418	70
19	S	4	MULTICOM-CLUSTER	11.445	70
20	Q	5	qSVMQA	11.608	70
			...

^*GOAL only submitted 4 predictions and COFOLD_QA only submitted 5 predictions, which makes it an unfair comparison when other groups have submitted 70 predictions. As the result, we removed these two groups from the ranking.

Table 3 shows the results for stage 1 (select-20) QA task of the top 20 groups in CASP12. In the table, GR denotes global ranking, the ranking among all participating methods; C stands for the category of a method, which could be either single-model (S), quasi-single-model (Q) or multi-model (M); CR is for the ranking of a method within its category. The last column contains the number of targets for which the group submitted predictions. All groups were evaluated based on their results on the same set of up to 70 targets. The methods were ranked by the average GDT-TS differences (AD) between predicted QA score and the ground truth. Impressively, MUfoldQA_S was the best overall, with a small AD value 3.602. Closely behind at the No. 2 place is MUfoldQA_C, with a slightly larger AD value of 3.818. Both of them were significantly better than the third-place method, Davis-EMAconsensus.

Table 4 shows the results for stage 2 (best-150) QA task in CASP12. MUfoldQA_C was ranked No. 2 overall, slightly worse than FDUBio. Although MUfoldQA_S ranked tenth among all methods, it ranked the third (after Mod-FOLD6_cor and ModFOLD6) among all quasisingle-model methods and outperformed all single-model methods.

Another QA performance measurement used in CASP12 was the top 1 model GDT-TS loss: the quality of the true best model in the pool (i.e., the best GDT-TS value among all models in the pool to the protein’s native structure) minus the quality of the predicted best model in the pool (i.e., the GDT-TS value of the top model selected by a QA method). In the top 1 model GDT-TS loss category on the CASP12 QA stage 1 dataset (select-20), MUfoldQA_C was superior again, ranked No. 1, outper forming the second best by a huge margin of 23.9% (0.82 vs. 1.077).

4.5. Robustness Evaluation

To evaluate the robustness of the new methods, we tested them on three problem attributes: difficulty of protein target, overall quality of QA model pool, and degree of similarity between protein models in the QA model pool. Difficulty of target is represented by the quality of the best predicted models in CASP12 submissions; overall quality of QA model pool is measured by the average quality in terms of GDT-TS of the subsets (set select-20 or set best-150); and degree of similarity between protein models in the QA pool is measured by the standard deviation of GDT-TS values of the models in the pool.

In the experiments, the two new methods were compared with the reference method Davis-EMAconsensus, which is also one of the best multi-model QA methods. Fig. 9 shows the correlation between the QA performance of each method and the three problem attributes of the model pool in QA stage 1 (select-20) in CSAP12. The results show that for this model pool, which was small (20 models) and had lower average quality, the DavisEMAconsensus performance was highly affected by all three attributes. MUfoldQA_C was less affected and MUfoldQA_S was the least affected, especially robust to the overall quality of the model pool.

Fig. 9.

Pearson correlation between QA performance of three methods and the three problem attributes of the QA stage 1 dataset (select-20) in CASP 12. DOT means Difficulty of Target; OQoP for Overall Quality of Pool and DoSBM for Degree of Similarity Between Models.

Fig. 10 shows the correlation between QA performance of the three methods and the three problem attributes of the stage 2 QA task dataset (best-150) in CASP12. These model pools were larger (150 models) and had better overall quality. In this case, both consensus-based algorithms were not as affected by the dataset attributes as in the select-20 set, while MUfoldQA_C was the most robust. When the model pool is small and quality is poor, MUfoldQA_C can take advantage of the information from templates to differentiate good models from poor ones. When the model pool is larger and of better quality, MUfoldQA_C can then take advantage of its quantity and diversity, and thus showing more robustness and better performance than previous multi-model methods.

Fig. 10.

Pearson correlation between QA performance of three methods and the three problem attributes of the QA stage 2 dataset (best-150) in CASP 12.

4.6. Execution Time

To calculate the time consumption of generating quality estimation, we performed the following experiment, using CASP 12 targets, given Blast and HHsearch results, test how long it would take MUfoldQA_S and MUfoldQA_C to estimate the quality for a pool of 150 decoys. The result is shown in Fig. 11.

Fig. 11.

CASP 12 stage 2 time consumption for targets of different numbers of residues.

The Fig. 11 indicates that, to estimate the quality of all 150 decoys in the pool, MUfoldQA_S took about 4 to 100 minutes for each target, and MUfoldQA_C took about 0.4 to 8.4 hours for each target. Both of which are well within the 48-hour limit of the CASP competition.

4.7. Discussion

Based on above results, MUfoldQA_S has shown its great advantage over other QA methods when only a small number of predictions have been generated or the model quality varies greatly, which is actually often the challenge that individual structure prediction servers face and make this method particularly useful. One of the major factors contribute to the superiority of MUfoldQA_S over other quasi-single-model QA method is its capability of directly running on raw protein fragments while others require full models, where lots of useful information in the raw fragments is lost. For example, merging the overlapping parts between different templates will lead to loss of information. And combined at wrong angle will create wrong information. Also, some of the rare templates have to be used far more often than others, which creates weighting issue. MUfoldQA_S avoids these problems introduced when creating full models.

MUfoldQA_C demonstrated strong robustness compared to all other state-of-art methods and consistent ultra-high performance. We attribute its success to the cleverly designed mechanism that takes advantage of MUfoldQA_S’ intermediate results and its ability to accurately weight the consensus scores generated by each reference model.

In real-world application, if the size of the predicted model pool is small, e.g., less than 30, use MUfoldQA_S; otherwise, use MUfoldQA_C.

5. CONCLUSION

In this paper, we proposed two new protein model quality assessment methods, MUfoldQA_S and MUfoldQA_C. The former is a quasi-single-model QA method that uses templates found by HHsearch and Blast to perform the evaluation, whereas the latter is a multi-model QA method that takes advantages of both template and reference model information. They obtained superior results on real-world datasets. In the well-known CASP12 competition, both methods were among the top performers. MUfoldQA_S ranked overall No. 1 in one QA category, and MUfoldQA_C ranked overall No. 1 in one category and No. 2 in another category.

Biography

Wenbo Wang received his B.S. degree in Automation from Huazhong University of Science and Technology in 2013 and M.S. degree in Computer Science from the University of Missouri-Columbia in 2016. He is currently pursuing the PhD degree in Electrical Engineering and Computer Science in University of Missouri-Columbia. His research interest is Deep learning and Bioinformatics.

Junlin Wang received his B.S. degree in Software Engineering from Beijing Jiaotong University in 2013 and M.S. degree in Computer Science from the University of Missouri-Columbia in 2015. He is currently working on the PhD degree in the Department of Electrical Engineering and Computer Science, University of Missouri-Columbia. His research direction is Deep learning and Bioinformatics.

Dong Xu is a James C. Dowell Professor in the Department of Electrical Engineering and Computer Science, with appointments in the Christopher S. Bond Life Sciences Center and the MU Informatics Institute at the University of Missouri. After earning his doctorate in 1995, Xu completed two years of postdoctoral work at the U.S. National Cancer Institute and then was a staff scientist at Oak Ridge National Laboratory until joining the university in 2003. His research includes protein structure prediction, high-throughput biological data analyses and in silico studies of plants, microbes and cancers. His work has been funded by the U.S. Department of Energy, the National Science Foundation, U.S. Department of Agriculture, National Institutes of Health, U.S. Army, United Soybean Board, Missouri Soybean Merchandising Council, Missouri Life Science Trust Fund, Monsanto Co., Cerner Corporation and National Center for Soybean Biotechnology. He has published more than 240 papers and is a recipient of the 2001 R&D 100 Award, 2003 Federal Laboratory Consortium’s Award of Excellence in Technology Transfer and 2010 Outstanding Achievement Award from International Society of Intelligent Biological Medicine. Xu is an editor in chief of International Journal of Functional Informatics and Personalized Medicine and associate editor in chief of IEEE/ACM Transactions on Computational Biology and Bioinformatics. He is an editorial board member of Current Protein and Peptide Science, Applied and Environmental Microbiology and International Journal of Data Mining and Bioinformatics.

Yi Shang is Professor and Director of Graduate Studies in the Electrical Engineering and Computer Science Department, University of Missouri, Columbia, Missouri. He received his Ph.D. degree in Computer Science from University of Illinois at Urbana-Champaign in 1997, M.S. degree from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, in 1991, and B.S. degree from the University of Science and Technology of China, Hefei, in 1988. He has published over 190 refereed papers in the fields of artificial intelligence, wireless sensor networks, mobile and distributed computing, and bioinformatics, and received 6 U.S. patents. He has worked as a researcher at the University of Illinois at Urbana-Champaign and the Xerox Palo Alto Research Center (PARC), Palo Alto, CA. He has received research funding from NSF, NIH, Army, DARPA, Microsoft, and Raytheon. He is a lifetime member of ACM and senior member of IEEE.