DeepUMQA-X: Comprehensive and insightful estimation of model accuracy for protein single-chain and complex

Abstract

An open protein model quality assessment server is essential for improving the accuracy of structure prediction and advancing the application of protein models in the biology community. In the post-AlphaFold2 era, protein complex structure prediction often relies on large-scale sampling for high-precision structures, while accurate scoring, ranking, and selection of protein models have become critical challenges that urgently need to be addressed. This work presents a comprehensive web server, DeepUMQA-X, which combines our single-model protocols for various evaluation metrics with a consensus strategy for protein model accuracy estimation (EMA). The server supports multiple protein single-chain or complex models as input, providing overall, interface, and residue accuracy estimates for each model. In the CASP16 EMA blind test, DeepUMQA-X achieved top performance across nearly all tracks, including QMODE1, QMODE2, QMODE3, and self-assessment. Remarkably, its single-model protocols outperformed all other single-model methods in accuracy assessment. Additionally, the server ranked first in a one-year (9 June 2023 to 1 June 2024) CAMEO-QE blind test. By integrating single-model approaches with a consensus-based strategy, DeepUMQA-X effectively bridges the performance gap between currently predominant consensus methods and the increasingly demanded single-model methods. The DeepUMQA-X server is freely available at http://zhanglab-bioinf.com/DeepUMQA-X.

Graphical Abstract

Graphical Abstract.

Introduction

With the breakthrough of AlphaFold2 [1] in protein tertiary structure modeling, the study of protein quaternary structures has garnered increasing attention within the biological community. In the post-AlphaFold era, most complex structure prediction methods, such as DMFold [2], MassiveFold [3], and AlphaFold3 [4], achieve high-precision modeling through extensive sampling approaches. Moreover, structure-based docking methods also depend heavily on large-scale sampling strategies [5–7]. The recently concluded CASP16 blind assessment has clearly shown that extensive sampling has emerged as one of the most critical elements underlying the success of top-performing methods. However, each quality assessment metric offers valuable insights within its specific facet; none alone can provide a comprehensive evaluation of protein models. Moreover, the limitations of the EMA method in assessment accuracy often led to unreliable evaluation results. Consequently, the development of high-performance protein model quality assessment (MQA), also known as estimation of model accuracy (EMA) methods for accurate scoring, ranking, and selection of models has become one of the most critical challenges that urgently need to be addressed in the field of protein complex structure prediction and EMA [8].

Protein EMA methods can generally be categorized into three types based on the number of input models: multi-model (i.e. consensus) methods, quasi-single-model methods, and single-model methods. Consensus methods evaluate model quality based on the structural similarity between input models, while their performance largely depends on the quality of the model pool. Representative methods include Pcons [9], MULTICOM series [10, 11], ModFOLD series [12, 13], VoroIF-jury [14], and MUfoldQA [15]. In particular, MULTICOM_qa [10], which recently combines structural similarity and deep learning interface contact prediction, achieved excellent performance in CASP15. To reduce reliance on model pool quality, quasi-single-model methods evaluate models based on internal structural modeling. Representative methods include ModFOLD series, MIEnsembles-Server, QMEANDisCo [16], and GOAP [17]. Among of them, the ModFOLD series from McGuffin’s has been the leading method for the past decade, providing high-performance protein structure prediction and EMA. In contrast, single-model methods do not rely on model pools or modeling techniques and directly assess individual model structures. These methods are the most challenging and widely studied in the community of EMA and CASP. Representative methods include the Voro series [18, 19], ProQ series [20, 21], DeepUMQA series [8, 22–25], and DeepAccNet [26]. The DeepUMQA series, a set of EMA methods continuously developed by our group over an extended period, focuses on improving the performance of single-model methods and providing innovative solutions to critical challenges. Each version has demonstrated outstanding performance in CASP and CAMEO blind tests, especially in local EMA. However, most existing EMA methods focus on a single evaluation metric, failing to provide comprehensive assessments of protein models. Moreover, only a few methods provide free and convenient online services. Additionally, the quality assessment of complex models remains in its early stages and urgently requires insightful approaches to drive progress in the field.

To address these challenges, we introduce the DeepUMQA-X server, an online platform for comprehensive and insightful protein EMA. DeepUMQA-X provides various protein model quality scores, including TM-score [27], QS-score [28], and lDDT [29], and supports input of multiple protein single-chain or complex models. Recent CASP16 blind test results showed that DeepUMQA-X (Group name: GuijunLab-QA & Human) performed well in scoring, ranking, and selecting of EMA, and is one of the top performing methods in nearly all tracks. The single-model protocols of DeepUMQA-X, GraphCPLMQA2 (Group name: GuijunLab-Assembly & PAthreader) perform best in overall (i.e. SCORE), interface (i.e. QSCORE), and local accuracy estimation among all single-model methods. Furthermore, one of the components of DeepUMQA-X, the residue-level single-model method GraphCPLMQA2L (server name: ZJUT-GraphCPLMQA), ranked first in the CAMEO quality assessment blind test (9 June 2023 to 1 June 2024). In addition, the DeepUMQA-X was independently tested on the CASP15 benchmark dataset and significantly outperformed the state-of-the-art (SOTA) methods at that time.

Materials and methods

The DeepUMQA-X server integrates multiple single-model protocols and a consensus strategy to comprehensively evaluate the accuracy of protein models. The single-model protocols are based on hybrid network architectures, including transformer networks, convolutional neural networks, and graph neural networks, which take input of multiple protein single-chain or complex structures and are trained on our previous work dataset [24]. In addition, the consensus strategy is based on structural similarities between protein models using structural alignment tools [30, 31] from the biology community or lightweight alignment methods developed by our group to derive consensus scores that reflect structural consistency. The workflow is shown in Fig. 1.

Figure 1.

Workflow of the DeepUMQA-X server. (A) The consensus method in DeepUMQA-X evaluates model quality using a consensus strategy based on two single-model approaches. (B) The key component of DeepUMQA-X, the GraphCPLMQA2 pipeline, is used for overall, interface, and local quality assessment of protein models. (C) A lightweight interface-based structural alignment process.

Workflow of DeepUMQA-X web server

DeepUMQA-X web server combines distinct single-model protocols for evaluation metrics with a consensus strategy for protein model accuracy estimation, where the single-model protocols include GraphCPLMQA2L for local accuracy assessment, GraphCPLMQA2Q for interface accuracy (i.e. QSCORE) assessment, GraphCPLMQA2S, and DeepUMQAS, which focuses more on ranking, for overall accuracy (i.e. SCORE) assessment, as shown in Fig. 1. The server offers two modes to cater to diverse user needs: consensus method mode (Fig. 1A) and single-model method mode (Fig. 1B).

Single-model method mode

In this mode, GraphCPLMQA2 (including S, L, and Q versions) is used to evaluate protein model accuracy, which is a comprehensive enhancement of our previous single-model method, DeepUMQA series [22–24]. Building upon features of previous methods, GraphCPLMQA2 further introduces critical evolutionary features, embeddings derived from protein language models ESM [32], and AlphaFold’s Evoformer [33] to improve the accuracy of the protein model evaluation. In addition, to more effectively capturing the relationships between features and structure accuracy, GraphCPLMQA2 improves the original convolutional network into a graph-coupled network architecture [24].

For given protein structure models, GraphCPLMQA2 first extracts sequence, structural, evolutionary, and statistical features (see Supplementary Text S1 for details), where the evolutionary features include residue-level embeddings from the 33rd layer of the ESM2 [32], along with multiple sequence alignment (MSA) and pairwise representations from the 48 Evoformer blocks of AlphaFold-Multimer [33]. These evolutionary features are refined through a graph Transformer module [24], enabling more effective modeling of the coupling between sequence and structure. Subsequently, the refined embeddings, with the protein model’s rotation and translation, are input into IPA modules [1] to generate new geometric constraint representations (i.e. reference structures) that better approximate the native protein structure. These geometric representations are combined with the extracted nonevolutionary features, which are input into a deep residual neural network (ResNet) [34] based on a Transformer strategy to predict the distance deviation and contact maps.

To achieve a comprehensive assessment of overall, interface, and local accuracy, different output network types, architectures, and information interaction ways were designed to train various network models for predicting TM-score (GraphCPLMQA2S output), QS-score (GraphCPLMQA2Q output), and lDDT (GraphCPLMQA2L output) in the fusion block module. Specifically, for local accuracy evaluation, the accuracy score for each residue is directly calculated using the lDDT formula based on predicted distance deviations and contact maps. For overall accuracy evaluation, the predicted maps and ResNet-derived embeddings are further refined by additional ResNet blocks to improve the model’s ability to capture the connection between local structures and geometric topology, with an overall accuracy score of protein structures predicted by convolutional global pooling. For interface accuracy evaluation, the sequence, structural, and statistical features from the interface regions of the complex structures are extracted to enhance sensitivity to the interchain orientation. These interface features are processed by a graph transformer [24] and combined with global accuracy scores, which contain the overall topology information, to predict the overall interface quality.

Consensus method mode

For input protein models, the DeepUMQAS (overall fold scoring version) newly introduces multiple view protein topology features and enhanced hierarchical network architecture to predict the overall quality score (TM-score) for each model. Meanwhile, GraphCPLMQA2L is used to infer the local quality (lDDT) of each model. Next, the models are ranked based on their overall and local quality scores to jointly select high-quality candidate structures, where those ranking within the top k (k= 25%) in both scores. These structures are subsequently compared with all input models using structure alignment tools to compute overall, interface, and local structure similarity scores. Finally, these scores are iteratively refined by re-evaluate and re-rank the models to output overall (TM-score), interface (QS-score), and local (lDDT) quality scores for each model. Moreover, we developed an interface-based, lightweight alignment algorithm for large proteins and mass model assessments (Fig. 1C). Specifically, for the input protein model, the interface structure is extracted using an interface contact threshold (8 Å), obtaining the corresponding residue sequence, with any missing parts represented as gaps. The interface residue sequences of all models are then collected to calculate the similarity scores through pairwise alignment (i.e. sequence consensus).

Results and discussion

Server input

The DeepUMQA-X web server provides a variety of open and free evaluation services, with a user-friendly web interface that supports more personalized customization, as shown in Fig. 2. The server is designed to evaluate both protein single-chain and complex models separately, ensuring more accurate results. To facilitate use by typical users, the DeepUMQA-X web server offers multiple input ways and supports the submission of multiple protein models. Users can upload protein models through text files, PDB files, or zip archives, and can add more models using the “Add model” button or upload multiple models within a zip archive. Meanwhile, users can remove incorrectly submitted models using the “Remove model” button. Furthermore, considering users’ different requirements for evaluation and assessment methods usage scenarios, the server provides two modes: single-model method and consensus method. The single-model method is suitable for the evaluation of one or a few models, while the consensus method is more appropriate for large batches of models (at least 5 models, with 50 or more models recommended). To simplify the result retrieval process, the server encourages users to provide an email address (though this is optional). Upon completion of the submission, the server will automatically send a notification email with a link to the results page. In addition, users can also optionally provide a job name for better task management.

Figure 2.

DeepUMQA-X web server job submission that includes (1) optional protein single-chain or protein complex assessment, (2) protein models upload (PDB format), (3) optional single-model or consensus quality assessment method, (4) optional user email address and job name, and (5) and the “submit” button.

Server output

The DeepUMQA-X web server provides model quality analysis and visualization results on various protein metrics. On the web server, after successfully submitting the job, the web page will be redirected to the job status page to report the running progress. Once the job is completed, the status page will automatically redirect to the results page, as shown in Fig. 3A (protein single-chain assessment) and 3B (complex assessment). Additionally, if the user fills in an email address, they will also receive a notification upon job completion. In the result page, each job ID is composed of a timestamp (year, month, day, hour, and minute) followed by six random numbers to help users track submission times. Moreover, the server also provides a download link for all results, which includes four types of files: (i) CSV files containing the predicted overall, interface (only for complexes), and local quality scores for the protein models; (ii) a predicted distance deviation image (.png, only for single-chain proteins) and a per-residue lDDT evaluation image (.png); (iii) the evaluation model (.pdb) with the B-factor column replaced by the predicted lDDT values; and (iv) text files containing the lDDT scores for each residue and interface, facilitating further processing. These data are also visually presented on the results page, where protein models can be interactively viewed using 3Dmol [35] software by control buttons. The job running time depends on the sequence length, the number of submitted models, and the server load. For single protein model evaluations, users typically receive results within 1 h, and sometimes as fast as minutes.

Figure 3.

DeepUMQA-X web server output page. Panels (A) and (B) show the output results for the protein single-chain and complex model evaluations, respectively, and include the following: (1) job result title; (2) a link to download all results; (3) predicted overall, interface (only for complexes), and local quality score; (4) Jmol visualization controlled by optional buttons; (5) predicted distance deviation map (only for single-chain proteins) and per-residue lDDT evaluation map; (6) predicted lDDT scores for interface residues (inter-chain distance cutoff <8 Å, only for complexes).

Performance of the DeepUMQA-X server

The DeepUMQA-X server and its single-model protocols have undergone multiple rounds of improvements, with critical blind testing in both CASP and CAMEO. In addition, it has been independently tested on the CASP15 benchmark dataset. Detailed test results are provided below.

Single-model performance

All CASP16 EMA methods can be divided into single-model methods, quasi-single-model methods and consensus methods, according to the CASP16 abstract for method description (Supplementary Text S2). The results show that the key component of the DeepUMQA-X server, GraphCPLMQA2 (S, Q, and L versions), outperformed other single-model methods in overall, interface, and local quality assessments, as shown in Fig. 4A–C and Supplementary Tables S1–S3. Notably, in local interface assessment, GraphCPLMQA2L (Z-score = 142.48) significantly outperformed the second-ranked Guijunlab-Complex method (Z-score = 121.38, from our group).

Figure 4.

Performance comparison of all single-model methods and example results from the DeepUMQA-X server (all data from CASP16 blind test results). (A) Overall accuracy assessment results for CASP16 protein models from all single-model methods. (B) Interchain interface accuracy assessment results for CASP16 protein models from all single-model methods. (C) Local interface accuracy assessment results for CASP16 protein models from all single-model methods. (D) The Top 1 model selected by the DeepUMQA-X server for the CASP16 target H1233 models. (E) Comparison of fold (bottom) and interchain interface (top) accuracy assessments between DeepUMQA-X and the Top 3 methods for the T1201o models. (F) Local interface accuracy of the ground truth (top) and DeepUMQA-X predictions (bottom) for the H1213 model (H1213TS198_2). The quality distribution ranges from 0 (low) to 1 (high), aligned with the bottom-to-top orientation of the scale bar.

Moreover, the CAMEO blind test results demonstrate that GraphCPLMQA2L, a key component of the DeepUMQA-X server, ranked first across “All,” “Low,” “Medium,” and “High” datasets, as shown in Table 1 and Supplementary Figs. S1–S4. The CAMEO official performance ranking is based on ROC AUC. In the evaluation of low-quality models, this method significantly outperforms others, achieving an AUC_0,0.2 = 0.84, a 13.5% improvement over the second-ranked DeepUMQA (AUC_0,0.2= 0.74, from our group), as shown in Supplementary Fig. S2. This performance gain is primarily attributed to the improvement of network architecture and features. Compared to DeepUMQA, which only utilizes features of the model itself and relies on convolutional neural networks focusing on local pattern learning, GraphCPLMQA2L introduces critical evolutionary features [32, 33] and graph-coupled architecture [24], which allows for effective modeling of long-range residue interactions and construction of representative geometric constraints. This change enhances its ability to capture deviations between the predicted model and the native structure, thereby significantly improving its evaluation performance on low-quality models.

Table 1.

The results are based on the CAMEO one-year blind test “All” data (9 June 2023 to 1 June 2024)

Rank	Methods	ROC AUC0,0.2	ROC AUC0,1	PR AUC0,0.2	PR AUC0,1
1	GraphCPLMQA2L	0.75	0.91	0.36	0.78
2	ModFOLD9	0.72	0.91	0.37	0.77
3	DeepUMQA	0.71	0.90	0.34	0.74
4	ProQ3D_LDDT	0.58	0.87	0.32	0.63
5	ProQ3	0.58	0.86	0.31	0.64
6	VoroMQA_v2	0.57	0.87	0.29	0.59
7	QMEAN 3	0.57	0.83	0.24	0.61
8	QMEANDiSCo 3	0.56	0.87	0.31	0.58
9	ProQ2	0.50	0.82	0.25	0.55
10	ProQ3D	0.49	0.82	0.27	0.55

CAMEO official website ranks methods by ROC AUC, and table only shows the top 10 groups.

In the evaluation of medium-quality and high-quality models, this method achieved AUC_0,0.2= 0.63 (medium) and AUC_0,0.2 = 0.60 (high), which represent improvements of 3.2% and 11.1%, respectively, over the second-ranked methods, ModFOLD9 [36] and ProQ3 [20], as shown in Supplementary Figs. S3 and S4. Compared to these representative methods, which mainly involve ensemble optimization of prediction results or rely on physics-based energy terms, GraphCPLMQA2L not only integrates physicochemical features of protein structures but also systematically incorporates evolutionary, geometric, and sequence-based features, providing a comprehensive representation of protein models from multiple perspectives. Moreover, by employing deep graph neural networks to describe the spatial relationships between residues in three-dimensional structures, the method can effectively capture local structural errors. These designs significantly enhance the evaluation accuracy for medium- and high-quality models, which typically exhibit overall correct folds but contain subtle local deviations.

Consensus strategy advantages

The DeepUMQA-X server with consensus strategy performed well across all four evaluation categories in CASP16 EMA, including QMODE1, QMODE2, QMODE3, and self-assessment. Specifically, QMODE1 focuses on the overall folding accuracy (Supplementary Fig. S5) and interface accuracy (Supplementary Fig. S6) of protein complex models; QMODE2 assesses the local interface accuracy (Supplementary Fig. S7) and local interface residue identification (Supplementary Fig. S8) of complex models; QMODE3 evaluates the model selection ability of complex models (Supplementary Fig. S9); while self-assessment involves the evaluating the residue quality of the models submitted by the server itself (Supplementary Fig. S10). The results demonstrate that DeepUMQA-X is one of the top-performing methods in nearly all tracks.

Furthermore, on the CASP16 dataset, we also conducted a performance and runtime comparison with docking-based assessment methods such as HADDOCK [37]. The test results show that the models selected by DeepUMQA-X achieve lower Top1Loss than those selected by HADDOCK on overall and interface evaluation metrics (Supplementary Fig. S11). This limitation may be attributed to the fact that the scoring function of HADDOCK is more suitable for docking protocols. In terms of runtime, HADDOCK demonstrates higher efficiency, with an average evaluation time of 1.57 h per target, while DeepUMQA-X requires a longer time of 9.21 h per target. This is mainly due to the longer time required to obtain consensus scores for conventional protein targets using traditional structural alignment tools, such as US-align [30] and MMalign [31].

Additionally, the models selected by DeepUMQA-X rank first in large assembly structure prediction, as reported in CASP16–CAPRI 57 summary (https://predictioncenter.org/casp16). This success is largely attributed to our proposed lightweight interface alignment method. For large assembly target, such as H1227 (5689 amino acids), from CASP16, the traditional structure alignment tools typically require several days to process ∼300 candidate models. In contrast, the interface alignment tool can complete the consensus alignment of all models in just ∼1 h, with minimal loss of evaluation accuracy. This way significantly improves the efficiency and practicality for large assembly protein evaluation tasks.

In the CASP15 benchmark dataset, DeepUMQA-X with the consensus strategy was compared with the CASP15 best methods in the global, interface, and local quality assessment. To ensure the fairness and reliability of results, the training set used by DeepUMQA-X was from before January 2022, while the CASP15 benchmark dataset was generated after May 2022, ensuring complete temporal independence between the datasets. The results show that DeepUMQA-X significantly outperforms all the SOTA methods in overall, interface, and local quality assessments (Supplementary Tables S4–S6). In local interface quality assessment, DeepUMQA-X achieved the best performance in predicting lDDT and PatchDockQ [38] metrics (lDDT: Pearson = 0.809, Spearman = 0.763, AUC = 0.847; PatchDockQ: Pearson = 0.615, Spearman = 0.551, AUC = 0.741), outperforming the best method, GuijunLab-RocketX (also from our group), by at least 10% in lDDT, 5% in PatchDockQ, and even up to 43.7% in Spearman (Supplementary Table S4). Similar improvements were observed in overall and interface quality assessments (Supplementary Tables S5 and S6). These results indicate that the CASP16 SOTA methods represent a significant improvement over the CASP15 SOTA methods, reflecting positive progress in the EMA. Furthermore, these findings are consistent with the conclusions in the CASP16 EMA assessor’s summary report (https://predictioncenter.org/casp16/doc/presentations).

Complex model evaluation

The following case studies further validate these advantages. Fig. 4D–F shows examples of DeepUMQA-X server assessment for CASP16 target models across different types of protein complexes. Fig. 4D displays the top 1 model (H1233TS028_1) selected from all predicted structures by DeepUMQA-X in target H1233 (antibody antigen), which is highly close to the native structure (TM-score = 0.990, QS-score = 0.947). Furthermore, Fig. 4E analyses the correlation between the true scores and predicted scores for overall and interface accuracy assessments in target T1201o (homopolymer), comparing DeepUMQA-X with other best methods (top 3 methods). In interface accuracy assessment, the Pearson correlation of this server achieves 0.986, significantly higher than other methods. In overall fold assessment, DeepUMQA-X also achieves the best correlation (Pearson = 0.992), indicating that the predicted values are almost linearly consistent with the true values. Fig. 4F shows the local interface evaluation results for the heteropolymer H1213 model (H1213TS198_2), where the quality distribution ranges from 0 (low) to 1 (high), aligned with the bottom-to-top orientation of the scale bar. The local interface quality predictions of DeepUMQA-X are basically consistent with the true quality distribution. Additionally, Supplementary Figs. S12–S15 provide more detailed results for these targets and the large assembly. The results demonstrate that, for these examples, the server’s evaluation of overall, interface, and local quality is highly consistent with the true quality distribution.

Conclusions

The DeepUMQA-X web server provides accurate assessments of protein single-chain and complex models across various quality metrics, including overall, interface (for complexes only), and local quality. It is freely available to the public in a visual, user-friendly way. In the recent CASP16 EMA and the one-year (9 June 2023 to 1 June 2024) CAMEO-QE blind test, DeepUMQA-X and its single-model protocol, GraphCPLMQA2, have demonstrated leading performance in EMA. With advancements in quaternary structure modeling, high-quality structures can be generated through large-scale sampling, while challenges remain in accurately identifying and selecting true interface regions. To facilitate the widespread use of protein models within the biological community, an open and comprehensive platform for protein quality analysis, such as the DeepUMQA-X server, may be essential for researchers to build and maintain confidence in protein structures.

Supplementary data

Supplementary data is available at NAR online.

Conflict of interest

None declared.

Funding

This work was supported by the National Key R & D Program of China (2022ZD0115103), the National Nature Science Foundation of China (62173304), the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2025C01190), Zhejiang Provincial Special Support Program for High-Level Talents (2023R5248). Funding to pay the Open Access publication charges for this article was provided by the support from foundation projects.

Data availability

The DeepUMQA-X web server is freely available at: http://zhanglab-bioinf.com/DeepUMQA-X/.