Comparative evaluation of the prediction accuracy of AlphaFold and ESMFold for monomeric and dimeric proteins

Abstract

We have evaluated the prediction accuracy of three different tools, deep-learning-based AlphaFold2, AlphaFold3, and large language model-based ESMFold, utilizing the experimentally derived structures deposited in the Protein Data Bank between 2022 and 2024, excluding those entries with close homologs in the structures released prior to 2022. Based on the criteria of sequence identity lower than 40% and query coverage <70%, 1666 monomeric and 994 dimeric proteins were selected as challenging targets for benchmarking. Our analysis showed that AlphaFold2 and AlphaFold3 correctly predicted 88% of monomeric structures and 77% of dimeric proteins. On the other hand, ESMFold accurately predicted 76% of the monomeric proteins and 41% of the dimeric proteins. Since most incorrect predictions involved nuclear magnetic resonance structures, benchmarking on X-ray and cryo-electron microscopy structures showed that the prediction accuracy of AlphaFold and ESMFold was 95% and 83%, respectively, for monomeric proteins. Overall, these findings demonstrate significant differences in the prediction accuracy of these machine learning (ML)-based tools for monomeric and dimeric proteins, highlighting the advantages and limitations of these tools. Finally, to facilitate easy access to benchmarking data, we developed ProModEv (Protein Model Evaluation portal), an interactive web portal for systematic analysis of these benchmarking results, and it is available at http://pdbi.nii.ac.in/ProModEv/.

Introduction

Proteins play a crucial role in various life processes and serve as the fundamental units for carrying out functions within organisms. Understanding the tertiary structure of proteins is essential for gaining insight into their biological functions. Conventionally, experimental techniques such as X-ray crystallography [1], nuclear magnetic resonance (NMR) [2], and cryo-electron microscopy (cryo-EM) [3] have been used to determine accurate protein structures, but these methods can be time-consuming and expensive. With decades of effort, around 238,000 structures [4, 5] have been determined using these labour-intensive approaches, representing only a minor fraction compared to millions (253 million entries in UniProtKB as of June 2025) of known protein sequences (uniprot.org) [6]. In recent years, the field experienced a breakthrough when computational biologists demonstrated the accurate protein structure prediction from amino acid sequences using artificial intelligence (AI) [7–9]. Notably, AlphaFold, a deep-learning-based method for predicting protein structure, has shown remarkable success in independent assessments of prediction accuracy. The advances in AlphaFold (version 2) were unveiled at CASP14, marking a turning point for the field where the best models of protein targets were found to be competitive with experimental structures in terms of their backbone accuracy, regardless of the modelling difficulty [10, 11]. In recent years, several other machine learning/deep learning/AI-based protein structure prediction methods have emerged, such as RoseTTAFold [12], ESMFold [13], OpenFold [14], and OmegaFold [15] that require only the protein sequences as input to obtain highly reliable 3D protein structures [16, 17]. In this study, we only evaluated the performance of deep-learning-based AlphaFold (AlphaFold2 and AlphaFold3) and large language-based model ESMFold in predicting protein structures. These methods offer a clear advantage by predicting protein–protein complexes directly from sequence [17–19]. AlphaFold has released predictions for 200 million individual protein sequences [20, 21]. Similarly, ESMFold has also developed the ESM Metagenomics Atlas, which is an open atlas of 772 million predicted metagenomics protein structures [13].

While the advancements made by these AI-based tools have been validated in specific contexts, their widespread adoption in structural biology applications remains to be comprehensively tested. Several earlier reports have evaluated the performance of machine-learning-based methods for protein structure prediction, namely, AlphaFold, RoseTTAFold, ESMFold, and OmegaFold [22–28]. In addition, platforms like CASP (Critical Assessment of Structure Prediction) [10, 29] and CAPRI (Critical Assessment of PRediction of Interactions) [30] evaluate the performance of protein structure prediction methods at periodic intervals. However, these studies often concentrate on specific sets of proteins released in a short period of time, limiting the generalizability of their findings. CAMEO (Continuous Automated Model EvaluatiOn) evaluates the groups of protein structure prediction servers/tools every week [31]. However, it does not have data on prediction results for software like ESMFold or AlphaFold2, although the monomer prediction results from AlphaFoldDB are evaluated by CAMEO.

In this work, we have compared the performance of the deep-learning-based AlphaFold2, AlphaFold3, and the large language model (LLM)-based ESMFold using experimentally determined protein structures released in the Protein Data Bank (PDB) over a three-year period, from 1^st January 2022 to 31^st December 2024. By applying stringent filtering criteria based on sequence identity and query coverage, we selected only those structures that lacked high-quality templates among entries released before 1^st January 2022. As a result, our benchmarking set consisted of 1666 monomers and 994 dimers, mainly challenging targets that are typically difficult to model using conventional template-based approaches. To make this benchmarking data accessible and systematic, we developed ProModEv (Protein Model Evaluation portal), an interactive web-based platform that hosts curated comparisons between predicted and experimental structures, allowing user-friendly exploration and analysis.

Materials and methods

Collection of benchmarking datasets and structure selection criteria

The PDB structures released between 1^st January 2022 and 31^st December 2024 were downloaded from the PDB [5], yielding a total of 39,298 protein structures across the three-year period. Corresponding sequences were retrieved from the PDB database, and a cross-BLAST analysis was conducted against all PDB sequences released up to 31^st December 2021 [32]. These cut-off dates were chosen to ensure that selected structures were not part of the training data for AlphaFold2, AlphaFold3, or ESMFold. The BLAST results were filtered at an e-value cut-off of 0.001. Sequences with <40% identity were retained regardless of query coverage, and those with <70% query coverage were retained irrespective of sequence identity. This strategy ensured the inclusion of challenging targets that are difficult to model using template-based methods. Subsequently, each entry was classified as a monomer, dimer, or higher-order oligomer using the PDBe biological assembly annotations as well as manual curation, and only structures whose biological assembly corresponded to a monomeric or dimeric state were retained for benchmarking. In the case of dimers, structures were included even if only one chain lacked a homologous structure. This filtering yielded a dataset of 2799 monomers and 2179 dimers. To eliminate redundancy, CD-HIT was used at a 90% sequence identity threshold [33]. For dimeric structures, the sequences of both chains were concatenated before clustering. Additionally, proteins with <20 residues or >2700 residues were excluded, resulting in the final benchmarking set of 1666 monomeric and 994 dimeric structures. A schematic pipeline outlining the sequence selection process is shown in Fig. 1.

Figure 1.

Workflow for sequence selection in benchmarking. A schematic flowchart outlining the selection of non-redundant monomeric and dimeric protein structures from PDB (2022–2024), excluding entries with close homologs (sequence identity >40%, query coverage >70%). The resulting dataset was used to evaluate AlphaFold2, AlphaFold3, and ESMFold predictions and evaluated.

Tertiary structure predictions using AlphaFold and ESMFold

The AlphaFold2 and ESMFold programs were installed locally by following the installation instructions in their respective GitHub repositories [11, 13]. All AlphaFold2 and ESMFold predictions were performed on a local HPC cluster equipped with NVIDIA A100 (40 GB) and (80 GB) GPUs, dual Intel Xeon CPUs, and 256 GB RAM. The AlphaFold3 was accessed as a web service and utilized for structure predictions [34]. Five structures for each sequence were generated using AlphaFold2 (v2.2.3) and AlphaFold3 server. ESMFold v1.0.3 was run with default parameters, resulting in single structures as output. The AlphaFold2 required approximately half to two hours on an A100 GPU, whereas complex or long proteins (>1000 aa) occasionally required >3–5 hours. Dimer predictions were more demanding, depending on interface size and multiple sequence alignment (MSA) depth. ESMFold was substantially faster, requiring 30 seconds to a few minutes on a single A100 80 GB GPU. AlphaFold3 server predictions showed faster run time (~1–5 min).

Residue alignment and quality assessment of predicted monomer structures

To evaluate the structures predicted by AlphaFold and ESMFold, each predicted monomeric model was aligned with its corresponding experimentally determined crystal structure from the PDB. For AlphaFold, the top-ranked model (ranked_0) was used, while ESMFold provides a single predicted model. To measure the prediction accuracy, two primary structural metrics were used for the evaluation, namely, root mean square deviation (RMSD) and the local distance difference test (LDDT).

RMSD is a global metric that measures the average deviation between equivalent atoms after optimal structural superposition [35]. Lower RMSD values indicate closer global similarity to the experimental structure. However, RMSD can be misleading for multidomain proteins, where flexible linkers or conformational changes may increase the global deviation even when individual domains are accurately predicted [36]. To complement this, we employed LDDT, a superposition-free local accuracy metric that evaluates the preservation of intramolecular distance patterns within defined cutoffs. LDDT values range from 0 to 1, with higher values reflecting better local structural agreement. Because LDDT does not require global alignment, it is particularly effective in identifying well-predicted regions within proteins that exhibit domain flexibility [37].

Based on a combined assessment of RMSD and LDDT, predicted monomer models were categorized into three groups: (i) Accurate prediction: RMSD ≤ 3.0 Å and LDDT ≥ 0.7, (ii) correct with domain movement: RMSD > 3.0 Å and LDDT ≥ 0.7, and (iii) incorrect prediction: RMSD > 3.0 Å and LDDT < 0.7. In this study, we define ‘domain movement’ as cases in which the global RMSD exceeds 3.0 Å while LDDT remains ≥0.7, indicating that individual domains are well folded but differ in their relative spatial arrangement compared to the experimental structure.

In addition, the predicted local distance difference test (pLDDT) was calculated as a per-residue confidence measure for all AlphaFold2, AlphaFold3, and ESMFold predictions. AlphaFold2 and AlphaFold3 compute per-residue confidence using lDDT-Cα, whereas ESMFold employs a similar transformer-based uncertainty score, which is calibrated differently. pLDDT is reported on a 0–100 scale and reflects the expected local accuracy of residue based on the predicted LDDT-Cα metric. Scores >90 indicate very high confidence in both backbone and side-chain geometry; values between 70 and 90 correspond to reliable backbone structure; 50–70 indicate low confidence; and values <50 typically mark flexible, intrinsically disordered, or weakly constrained regions. Although all three predictors output values on a 0–100 scale, thresholds such as ≥70 for ‘reliable backbone’ are equivalent across methods. Lin et al. have compared pLDDT values for ESMFold and AlphaFold and shown them to be comparable [13]. In addition, TMscore, which measures topological similarity between predicted and experimental structures, was also used as a structural similarity measure [38].

Quality assessment of predicted dimer structures

Dimeric predictions generated by AlphaFold2, AlphaFold3, and ESMFold were evaluated using the DockQ score, an established metric for assessing the accuracy of protein–protein complex predictions. DockQ integrates several interface-specific measures, including interface RMSD (iRMSD), ligand RMSD (LRMSD), and the fraction of native contacts preserved (Fnat) [39]. Because AlphaFold and ESMFold predictions do not incorporate ligands, all ligand molecules were removed from the experimental PDB files to maintain consistency with predicted models.

To ensure residue numbering consistency between PDB structures and predicted models, we used the fix_numbering.pl script from the DockQ package prior to scoring. This preprocessing step enables accurate interface alignment and reliable DockQ computation. For each dimeric target, DockQ scores were calculated for all five AlphaFold2/AlphaFold3 predictions, and the model with the highest DockQ score was selected for downstream analysis. A higher DockQ score indicates a more accurate reproduction of the experimental interface. Following CAPRI standards, predictions were classified as: incorrect (<0.23), acceptable (0.23–0.49), medium (0.49–0.80), and high quality (≥0.80). pLDDT was also calculated for dimer models to provide complementary residue-level confidence information.

Development of ProModEv

To provide public access to benchmarking results, we developed ProModEv, a web-based platform for interactive evaluation of structure prediction tools. The frontend was implemented using HTML5 and CSS, and PHP was used for server-side scripting. Benchmarking data was managed using MariaDB, and the portal was hosted on an Apache server running on a Linux system. Interactive tables were built using DataTables.js to enable sorting and filtering of entries. Mol* (MolStar) was integrated for in-browser visualization and comparison of predicted and experimental protein structures.

Statistical analysis, scripts and figures

All statistical analyses were performed in R using the stats package, and figures were generated with the ggplot2 package [40]. Spearman’s correlation and t-test analyses were performed to evaluate the metrics. All analyses in this study were performed using custom shell and Python scripts. The plots were created with Matplotlib24 [41] and seaborn2 [42], and protein structures were visualized using PyMOL2.5 (the PyMol Molecular Graphics System, version 2.5.2 Schrodinger, LLC).

Results

Sequence selection and model generation

We analysed protein structures released between 1^st January 2022 and 31^st December 2024. To ensure unbiased evaluation, only structures without homologous templates were selected, as described in the ‘Materials and methods’ section. This yielded a benchmarking dataset of 1666 monomeric and 994 dimeric structures. Among monomers, 1314 (79%) were resolved using X-ray diffraction (XRD), 160 (9.6%) by electron microscopy (EM), and 190 (11.4%) by NMR. The 994 dimers comprised 795 (80%) homodimers and 199 (20%) heterodimers. Of these, 760 (76.5%) were resolved via XRD, 219 (22%) by EM, and 15 (1.5%) by NMR. This distribution highlights the dominance of crystallographic methods while reflecting growing contributions from EM and NMR. We generated structure predictions for each selected target using their respective amino acid sequences, enabling subsequent benchmarking and comparative evaluation. Complete metadata, including PDB IDs, sequence lengths, resolution techniques, and evaluation scores, is accessible via the ProModEv portal.

Performance of alphafold for monomer structure prediction

A comprehensive evaluation of 1666 monomeric protein structures was performed using AlphaFold2, AlphaFold3, and ESMFold. Based on RMSD and LDDT metrics, predictions were categorized into three classes: accurate predictions, correct predictions with domain movements, and incorrect predictions. The ‘domain movement’ category captures proteins whose individual folded regions were predicted with high local accuracy (LDDT ≥ 0.7), but whose interdomain orientations differ from the experimental arrangement, resulting in global RMSD >3.0 Å. Representative examples of each category are shown in Fig. 2, and predictive classifications from AlphaFold2 are summarized alongside experimental methods in Table 1 and Fig. 3. A scatter plot of RMSD and LDDT scores for AlphaFold2 predictions is provided in Fig. 4A.

Figure 2.

Cartoon depictions of representative structures from each categories of monomers. (A) Accurate prediction structures with high prediction confidence and close alignment with experimental data. (B) Domain movements depict structures with significant domain rearrangements in multidomain proteins. (C) Incorrect prediction structures with low confidence and substantial deviations from experimental data.

Table 1.

Evaluation summary of AlphaFold2, AlphaFold3, and ESMFold predicted monomer structures

Tool	Predictive class	No.	pLDDT ≥ 70	XRD	EM	NMR	TQ 0	TQ 1–40	TQ > 40
AlphaFold2	Accurate prediction	1069	1067	985	52	31	399	637	33
	Domain movement	386	373	281	84	21	107	245	34
	Incorrect prediction	200	150	47	25	128	135	54	11
	Special cases	11	10	1	0	10	11	0	0
AlphaFold3	Accurate prediction	1076	1074	1001	46	28	405	640	31
	Domain movement	388	379	274	95	19	109	244	35
	Incorrect prediction	191	158	37	19	135	130	49	12
	Special cases	11	11	2	1	8	8	3	0
ESMFold	Accurate prediction	832	821	783	27	22	290	514	28
	Domain movement	430	404	332	77	20	143	257	30
	Incorrect prediction	391	137	195	140	56	212	160	19
	Special cases	13	9	4	0	9	8	5	0

The predictions were assessed based on the following parameters: accurate predictions (RMSD ≤ 3.0 and LDDT ≥ 0.7), correct predictions with domain movement (RMSD > 3.0 and LDDT ≥ 0.7), incorrect predictions (RMSD > 3.0 and LDDT < 0.7), and special cases (LDDT < 0.7 and RMSD < 3.0). Additionally, the table provides the proportion of structures with pLDDT scores indicating a backbone quality of ≥70, as well as the distribution of structures derived from different experimental methods (XRD, EM, and NMR). Details regarding the presence of homologous structures are included in the form of template quality (sequence identity × query coverage).

Figure 3.

Bar plot showing the distribution of monomeric protein structure predictions across AlphaFold2, AlphaFold3, and ESMFold, categorized into four predictive classes: accurate prediction (green), domain movement (blue), incorrect prediction (red), and special cases (grey). The classification is based on structural comparison with experimentally resolved PDB structures deposited between 2022 and 2024.

Figure 4.

Evaluation of 1666 monomeric structure predictions by AlphaFold2, AlphaFold3, and ESMFold. Predictions are classified as accurate (RMSD ≤ 3.0 Å, LDDT ≥ 0.7), domain movement (RMSD > 3.0 Å, LDDT ≥ 0.7), incorrect (RMSD > 3.0 Å, LDDT < 0.7), or special cases (RMSD ≤ 3.0 Å, LDDT < 0.7). Each point is colour-coded based on the experimental structure origin: XRD, NMR, or EM. The pLDDT score represents residue-wise confidence and was averaged across Cα atoms (LDDT-Cα) per structure. (A) Scatter plot of RMSD versus LDDT for AlphaFold2-predicted monomers. (B) Violin plot showing pLDDT distributions for AlphaFold2 across prediction categories. (C) Scatter plot of AlphaFold3 predictions. (D) Violin plot of pLDDT distributions for AlphaFold3. (E) Scatter plot of ESMFold predictions. (F) Corresponding violin plot of pLDDT distributions for ESMFold.

Out of 1666 monomeric structures evaluated with AlphaFold2, 1069 (64.2%) displayed high LDDT (≥0.70) and low RMSD (≤3), and were categorized as accurate predictions. An additional 386 (23.2%) predictions met the LDDT threshold but showed RMSD values >3, may be due to domain movements, and were also considered as correct predictions. Taken together, AlphaFold2 achieved 87.4% accuracy. In contrast, 200 structures (12%) showed low LDDT and high RMSD and were considered incorrect. A separate group of 11 structures exhibited low RMSD (<3) but also low LDDT (<0.7); all were derived from NMR experiments and were grouped as a special category.

AlphaFold2 also reports a pLDDT score as a per-residue confidence measure on a 0–100 scale, and pLDDT values above 70 correspond to reliable backbone predictions. In our evaluation, mean pLDDT values were calculated across the entire structures using Cα atoms. Among the 1069 accurately predicted structures, 1067 showed average pLDDT ≥70, with only two structures marginally lower at 67.3 and 69.9. For the 386 structures classified as domain movement cases, 373 had pLDDT ≥70, while 13 scored between 62 and 69. Among the 200 incorrect predictions, 150 (75%) still showed high confidence (pLDDT ≥ 70), while the remaining 50 ranged from 39 to 69 (Table 1 and Fig. 4B).

As noted in the ‘Materials and methods’ section, structures were selected to avoid close homologs, using a threshold of <40% identity and/or <70% query coverage. Homology was assessed using Template Quality (TQ), calculated as the product of sequence identity and query coverage (TQ = identity × coverage). Of the 1666 monomer structures, 652 (39%) had no homologous structure (TQ = 0), 936 (56%) had TQ between 1 and 40, and only 78 (5%) had TQ >40 (Table 1 and Supplementary Fig. S1). Among the 1069 accurate AlphaFold2 predictions, 399 (37.3%) lacked homologs, 637 (59.6%) had TQ <40, and 33 (3%) had TQ >40. Similarly, in the domain movement group, 107 (27.7%) had no homologs, 245 (63.5%) had TQ <40, and 34 (8.8%) had TQ >40. In the incorrect prediction set, 135 structures (67.5%) had no homologs, 54 (27%) had TQ <40, and only 11 had TQ >40. These results underscore the critical role of template availability in influencing prediction accuracy, with lower TQ correlating with higher misprediction rates.

Evaluation of the same 1666 structures using AlphaFold3 yielded highly consistent results. AlphaFold3 accurately predicted 1076 (64.6%) structures, while 388 (23.3%) showed correct topology with domain movements, and 191 (11.5%) were deemed incorrect based on low LDDT and high RMSD. As with AlphaFold2, 11 NMR-based structures were assigned to the special low-LDDT/low-RMSD category. Table 1 and Fig. 3 illustrate AlphaFold3 classification results, and Fig. 4C provides the distribution of LDDT and RMSD. AlphaFold3 also outputs pLDDT values. Of the 1076 accurately predicted structures, 1074 showed pLDDT ≥70, while two had scores of 61 and 68. Among the 388 domain-movement predictions, 379 had pLDDT ≥70, and the remaining nine scored between 65 and 69. For the 191 incorrect predictions, 158 (83%) had pLDDT ≥70, and for the remaining 33, pLDDT ranged between 43 and 69 (Table 1 and Fig. 4D). TQ distribution for AlphaFold3 predictions is shown in Supplementary Fig. S2.

Direct comparison between AlphaFold2 and AlphaFold3 reveals high consistency in performance, with both achieving overall prediction accuracies of 87.3% and 87.9%, respectively. Among the 1076 structures classified as accurate predictions by AlphaFold3, AlphaFold2 matched 1005, while reclassifying 56 as domain movement, 13 as incorrect predictions, and 2 as special cases. For the 388 domain movement predictions from AlphaFold3, 323 were similarly classified by AlphaFold2; 54 were reassigned as accurate and 11 as incorrect. Of the 191 incorrect predictions in AlphaFold3, AlphaFold2 agreed on 174 cases but reclassified 7 as accurate, 7 as domain movement, and 3 as special cases. For the special cases group, 6 structures were consistently labelled by both models, while AlphaFold2 reassigned 3 as accurate and 2 as incorrect. These results highlight the robust agreement between AlphaFold2 and AlphaFold3, with slight variations mainly arising in borderline and domain movement cases.

Performance of ESMFold for monomer structure prediction

Out of 1666 monomeric structures predicted by ESMFold, 832 (50%) were classified as accurate with high LDDT and low RMSD, while 430 (26%) showed domain-level movements despite acceptable local accuracy. A total of 391 structures (23.5%) were classified as incorrect, and 13 structures with low LDDT and low RMSD were grouped as special cases. The resulting classification outcomes are shown in Table 1 and Fig. 3, and the RMSD–LDDT distribution is presented in Fig. 4E. Like AlphaFold, ESMFold provides predicted pLDDT scores. Among the 832 accurate models, 821 had average pLDDT ≥70, and 11 ranged between 56 and 69. In the domain movement group, 404 structures had pLDDT ≥70 and 26 scored between 47–69. Among the 391 incorrect predictions, 137 (35%) showed pLDDT ≥70, while for the remaining, it ranged from 26 to 69 (Table 1 and Fig. 4F).

Assessment of homologous structure availability using TQ revealed similar trends. Of the 832 accurately predicted structures, 290 (35%) had no homologs (TQ = 0), while 514 had TQ <40, and only 28 had TQ >40. In domain movement cases, 143 (33.3%) had TQ = 0. In the incorrect prediction set, nearly half (46%) lacked homologs (Table 1 and Supplementary Fig. S3).

Performance of AlphaFold for dimeric structure prediction

As discussed previously, we also assessed AlphaFold’s performance in predicting 994 dimeric protein complexes, comprising 795 (80%) homodimers and 199 (20%) heterodimers. Prediction quality was evaluated using DockQ scores, as described in the ‘Materials and methods’ section. Out of the 994 dimers, 763 (76.8%) had DockQ scores above 0.23 and were classified as accurate, while 231 (23.2%) fell below this threshold and were considered incorrect. Representative structures with correct and incorrect interfaces are shown in Fig. 5. Among the 763 accurate structures, 361 were categorized as high quality, 297 as medium, and 105 as acceptable. The distribution patterns of predicted dimer classes are illustrated in Table 2 and Fig. 6, while detailed DockQ score distributions for AlphaFold2 are provided in Fig. 7A.

Figure 5.

Cartoon images of representative structures for protein–protein complexes (dimers). (A) Accurate predictions with high DockQ scores depicting structures with correct interface alignment. (B) Incorrect predictions with low DockQ scores, showing structures with poor interface alignment. The pLDDT plots for each category are displayed on the side of the respective panels, indicating the pLDDT scores for each residue.

Table 2.

Evaluation summary of AlphaFold2, AlphaFold3, and ESMFold predicted dimer structures

Tool	Predictive class	No.	pLDDT ≥ 70	Homodimer	Heterodimer	XRD	EM	NMR	TQ 0	TQ 1–40	TQ > 40
AlphaFold2	High quality	361	356	287	74	318	43	0	188	173	0
	Medium quality	297	283	224	73	199	91	7	165	130	2
	Acceptable quality	105	92	84	21	63	37	5	59	45	1
	Incorrect prediction	231	197	200	31	180	47	3	145	85	1
AlphaFold3	High quality	346	341	274	72	307	39	0	102	243	1
	Medium quality	306	293	231	75	208	91	7	101	202	2
	Acceptable quality	93	89	71	22	53	3	37	25	67	1
	Incorrect prediction	249	220	219	30	192	51	5	91	158	0
ESMFold	High quality	85	79	77	8	83	2	0	23	60	2
	Medium quality	201	192	163	38	168	31	2	59	139	3
	Acceptable quality	123	105	102	21	86	36	1	39	80	3
	Incorrect prediction	585	333	453	132	423	149	12	270	286	28

This table summarizes the results of 994 predicted dimer structures based on the following criteria: high quality (DockQ ≥ 0.80), medium quality (0.49 ≤ DockQ < 0.80), acceptable quality (0.23 ≤ DockQ < 0.49), and incorrect prediction (DockQ < 0.23). Additionally, it provides information on the proportion of structures with pLDDT ≥70, the distribution across experimental methods (XRD, EM, NMR), and TQ (sequence identity × query coverage).

Figure 6.

Bar plot depicting the distribution of dimeric protein structure predictions across AlphaFold2, AlphaFold3, and ESMFold. Predictions are grouped into four quality classes: high quality (green), medium quality (blue), acceptable quality (grey), and incorrect prediction (red), based on structural agreement with experimental PDB entries.

Figure 7.

Evaluation of 994 dimeric protein complex predictions by AlphaFold2, AlphaFold3, and ESMFold. Predicted dimer models were classified based on DockQ scores into high quality (DockQ ≥ 0.80), medium quality (0.49 ≤ DockQ < 0.80), acceptable quality (0.23 ≤ DockQ < 0.49), and incorrect (DockQ < 0.23) categories and plotted against the total sequence length of the dimeric complex. Corresponding violin plots show the distribution of average per-structure pLDDT values across these DockQ-based quality classes. Each point in the scatter plots is colour-coded according to the experimental structure determination method(XRD, NMR, EM). (A) Scatter plot of DockQ versus sequence length for AlphaFold2-predicted dimers. (B) Violin plot showing pLDDT distributions for AlphaFold2 predictions across quality categories. (C) Scatter plot of AlphaFold3-predicted dimer structures. (D) Violin plot of pLDDT distributions for AlphaFold3-predicted dimers. (E) Scatter plot of DockQ scores for ESMFold-predicted dimers. (F) Violin plot showing pLDDT distributions across quality classes for ESMFold predictions.

Analysis of pLDDT scores showed that 731 of the accurate predictions had average pLDDT ≥70, and the remaining 32 ranged between 30 and 69. Among the 231 incorrect predictions, 197 (85%) still showed pLDDT ≥70, with the rest between 32 and 69 (Table 2 and Fig. 7B). While pLDDT scores offer residue-level confidence, they may not fully capture inter-chain interface accuracy. We also analysed template availability using TQ. Of the 763 accurate predictions, 412 (54%) had no homologs (TQ = 0), 348 had TQ <40, and only 3 had TQ >40. Among the 231 incorrect predictions, 145 (62.8%) lacked homologs, 85 had TQ <40, and just 1 had TQ >40 (Table 2 and Supplementary Fig. S4).

Similar analysis was performed using AlphaFold3. Out of 994 dimers, 745 (75%) had DockQ scores >0.23 and were considered accurate, while 249 (25%) were incorrect. Among the accurate predictions, 346 were high quality, 306 medium, and 93 acceptable, as shown in Table 2 and Fig. 6. The distribution plot of DockQ score for AlphaFold3 is depicted in Fig. 7C. AlphaFold3 also provides pLDDT scores. Of the 745 accurate dimers, 723 had pLDDT ≥70, with 22 ranging from 56 to 69. Among the 249 incorrect predictions, 220 (88%) still had pLDDT ≥70, while the rest ranged from 34 to 69 (Table 2 and Fig. 7D). TQ analysis (Table 2 and Supplementary Fig. S5) showed that 414 (55.6%) of the accurate dimers had no homologs, 326 had TQ <40, and only 5 had TQ >40. Among the incorrect group, 153 (61.4%) had no homologs, 88 had TQ <40, and 8 had TQ >40.

We compared AlphaFold2 and AlphaFold3 predictions to assess consistency. Of the 745 accurate AlphaFold3 predictions, AlphaFold2 agreed on 713, while 32 were classified as incorrect by AlphaFold2. Among the 346 high-quality predictions by AlphaFold3, 294 remained the same in AlphaFold2, while 52 were downgraded. For the 306 medium-quality predictions, AlphaFold2 agreed in 219 cases and reassigned the remaining 87 across different categories (52 to high quality, 22 to acceptable quality, and 13 classified as incorrect prediction). Among 249 AlphaFold3 incorrect predictions, 199 were consistent with AlphaFold2, while 50 were reassigned to correct predictions (10 to high quality, 18 to medium quality, and 22 to acceptable quality). Overall, both versions showed strong concordance, especially in identifying incorrect predictions, with some variation in borderline or intermediate-quality cases.

Performance of ESMFold for dimeric structure prediction

We extended our evaluation to ESMFold by assessing its performance on the same 994 dimeric structures. Out of these, 409 (41.1%) dimers, including 342 homodimers and 67 heterodimers, showed DockQ scores above 0.23 and were classified as accurate, while 585 (58.9%) dimers had DockQ scores below 0.23 and were considered incorrect. Among the 409 accurate predictions, 85 were high quality, 201 medium quality, and 123 acceptable quality. The predictive classifications are shown in Table 2 and Fig. 6, and DockQ score distributions are detailed in Fig. 7E.

Similar to AlphaFold, ESMFold also provides pLDDT scores as an indicator of residue-level confidence. Among the 409 accurately predicted dimers, 376 had average pLDDT scores ≥70, while the remaining 33 ranged from 41 to 69. In the 585 incorrect predictions, 333 (57%) had pLDDT ≥70, while the remaining 252 ranged between 20 and 69 (Table 2 and Fig. 7F). As with AlphaFold, these findings suggest that pLDDT alone is not a sufficient indicator for predicting correct dimer interfaces.

To further understand the influence of homologous templates on prediction accuracy, we analysed the distribution of ESMFold dimer predictions based on TQ. Among the 409 accurate structures, 121 (29.6%) lacked homologous templates (TQ = 0), 279 (68.2%) had TQ <40, and only 9 (2.2%) had TQ >40. Of the 585 incorrect predictions, 270 (46.2%) had no homologs, 286 (49%) had TQ <40, and 29 (4.8%) had TQ >40 (Table 2 and Supplementary Fig. S6).

Detailed case study of representative structures

To better understand the systematic limitations of current structure predictors and to provide practical insight into failure cases, we performed a case study analysis of representative targets from our benchmark. Although representative examples of accurate and failed predictions were illustrated earlier (Fig 2 and 5), here we present case studies involving (i) NMR-ensemble and intrinsic disorder region (IDR) driven failures, (ii) X-ray (crystal) discrepancies for monomers, (iii) dimer interface failures and transient contacts, and (iv) monomer and dimer predictor discordance (one tool correct, others incorrect). For each case, we analysed the metrics described in the ‘Materials and methods’ section (RMSD, LDDT, pLDDT, and DockQ and interface metrics for dimers). Representative multipanel examples are provided in Supplementary Figs S7–S10.

Our analysis suggests that predicted models for many NMR targets fail to match any single deposited conformer: in our examples, the RMSD for individual ensemble members often exceeds 5 Å. When we compared predictions to the full NMR ensembles rather than to a single conformer (PDB ID 7LVG, 7QAL), the apparent accuracy improved (from ~27% to 37%). We therefore suggest that mapping against multiconformer ensembles can increase the apparent accuracy for NMR targets (Supplementary Fig. S7). In addition, presence of extensive IDR regions in NMR structures (PDB ID 7VZM) is also a likely contributor to the failure in predictions.

A minority of X-ray structures also show unexpectedly large deviations. Inspection of these cases implies that their predictors failed to capture local fold conformations or poorly resolved loops or presence of IDR regions (PDB ID 7FAV and 7XBJ). In some synthetic proteins with designed coiled-coil or extended helical architectures (8P4Y), predictors struggle to recover correct conformations (Supplementary Fig. S8). Since correlated mutation information from MSA will not be available for such synthetic proteins, the models failed to predict the correct inter-residue contacts. These cases demonstrate that even high-resolution crystal structures can diverge substantially from predictions when contextual structural information or information of stabilizing inter residues are missing. Another possible reason could be the presence of biologically relevant cofactors or metals crucial for folding that were omitted during preprocessing, because earlier predictor versions did not support explicit ligand/cofactor modelling, omission of bound partners can explain some mismatches; we note this limitation and recommend retaining biologically relevant ligands in future, ligand-aware benchmarks.

For many dimers, predictors produce accurate monomer folds (high per-chain pLDDT) but fail to predict the native assembly or interface (PDB ID 8HJF, 7BGS, and 8CQM). These failures are associated with low DockQ scores (<0.23) and loss of native contacts on contact maps, consistent with transient or weak interfaces (Supplementary Fig. S9). The native experimental structures also had a low interface area (<500). In such cases, high confidence (pLDDT) is not sufficient to infer a correct biologically relevant assembly; interface-focused analysis is required.

We also analysed predictor discordance. For monomers (PDB ID 7PAB and 7QLH) and dimers (PDB ID 7F4O and 8G9K), one predictor yields a substantially more accurate model than others. For example, in the case of monomers for 7PAB, AlphaFold2/3 predicted structures accurately while ESMFold failed. In the case of 7QLH, AlphaFold2 showed domain movement, while AlphaFold3 captured the native domain orientation, and ESMFold failed. Similarly, in cases of dimeric examples 8GNK, AlphaFold3 was able to place the chain correctly, while AlphaFold2 and ESMFold failed to predict the native assembly interface. These discordances often track differences in the model training and thus reflect systematic methodological differences. ESMFold, which does not use MSAs, shows a different error profile from AlphaFold2/3 and is often less reliable for interfaces; AlphaFold2 and AlphaFold3 are broadly concordant but can differ in domain placement for borderline cases (Supplementary Fig. S10).

Taken together, these focused case studies explain a representative fraction of the apparent failures in our benchmark and yield practical usage examples that users can apply to assess confidence in individual predictions (per-residue pLDDT, ensemble comparison, DockQ, and interface area).

Statistical analysis of correlation between matrices and significance of difference between tools

Statistical analyses were carried out to quantify the significance of differences between predictors, as well as correlations between metrics. Statistical significance test (t-test) was performed for monomer and dimer structures. RMSD, pLDDT, and LDDT metrics were used for the monomers, whereas pLDDT and DockQ metrics were used for the dimer structures. Supplementary Fig. S11 displays correlations between different metrics used for structural comparison of monomers. Similarly, for the monomers, Supplementary Fig. S12 shows the distributions of the values for these metrics as box plots for predictions by different tools, along with p-values to estimate statistical significance of the differences for predictions by different tools.

For monomers, RMSD showed a negative correlation with the other metrics (LDDT, pLDDT, and TM-score), whereas LDDT exhibited a positive correlation with both pLDDT and TM-score. In addition, pLDDT and TM-score were moderately positively correlated. These correlations were stronger for ESMFold than for AlphaFold, indicating that ESMFold’s confidence scores may more accurately reflect prediction quality. Statistical significance testing revealed significant differences (P-value <.05 for t-test) between the AlphaFold and ESMFold models across all metrics (RMSD, LDDT, and pLDDT). For the RMSD metric, no significance was observed between AlphaFold2 and AlphaFold3, while a highly significant difference was detected between the AlphaFold models and ESMFold. Moreover, both LDDT and pLDDT showed highly significant differences between the AlphaFold and ESM fold model, while AlphaFold2 and AlphaFold3 have moderate significance.

In the case of dimers, all three prediction tools, DockQ showed a strong positive correlation with TM-score (AF2/AF3 ≈ 0.90, ESM ≈ 0.84), indicating improved prediction quality with higher scores. A moderate positive correlation was observed between DockQ and pLDDT (0.49–0.54). Similarly, pLDDT and TM-score were moderately correlated (AF2 ≈ 0.54, AF3 ≈ 0.47, ESM ≈ 0.59) (Supplementary Fig. S13). Significance testing for DockQ and pLDDT revealed no significant difference between the AlphaFold2 and AlphaFold3 models, whereas significant differences were observed between the AlphaFold and ESMFold predictions. A high level of significance was detected when comparing AlphaFold models with ESMFold (Supplementary Fig. S14). This trend was consistently reproduced in the subsampling analysis of 100 datasets.

In addition, we conducted 100 times resampling of 50% of the data and performed a statistical significance test (Supplementary Fig. S15). For monomer structures, AF2 and AF3 showed no significant differences in RMSD (1% i.e. only 1 out of 100 subsamples showed differences with p-value <.05), whereas both differed consistently from ESM, with significance observed in all subsamples (100%). Differences in pLDDT between AF2 and AF3 were moderate (60%), while AlphaFold and ESMFold showed a 100% significant difference. In LDDT, all three tools showed highly significant differences (100%). For dimer structures, AF2 and AF3 again produced no significant differences in pLDDT (3%) and significant differences of 41% in the DockQ metric. In contrast, ESMFold differed from both AlphaFold models across DockQ and pLDDT metrics (100%).

Development of ProModEv

To facilitate user-friendly access to benchmarking results of protein structure prediction tools, we developed ProModEv, a web-based platform available at http://pdbi.nii.ac.in/ProModEv/. The portal provides an overview of performance evaluation matrices, including structural alignments and benchmarking statistics, as illustrated in Fig. 8. It features interactive dashboards and intuitive visualizations that support both global summaries and detailed assessments. Users can explore the performance of individual tools or perform side-by-side comparisons of AlphaFold2, AlphaFold3, and ESMFold across specific monomeric or dimeric proteins. The portal also supports visualization of structural alignments, aiding users in interpreting prediction outcomes in a structural context. To assist users in interpreting benchmarking results, the ProModEv portal provides an interpretation section that explains the recommended use of each metric (pLDDT, LDDT, RMSD, DockQ). In addition, it also provides the TM-score. TM-score reflects global topological similarity between structures (values >0.5 typically indicate similar overall fold, and values >0.7 suggest strong fold-level agreement), and therefore serves as a useful complementary metric for users who wish to explore global similarity. All analysis scripts and input metadata are publicly archived in a Zenodo repository to support transparency and reproducibility.

Figure 8.

Screenshot of the ProModEv web interface. The portal allows interactive exploration of benchmarking results for predicted protein structures. Users can filter, sort, and compare predictions from AlphaFold2, AlphaFold3, and ESMFold, and examine models in three dimensions directly within the browser.

Although the present study focuses on monomers and dimers, the benchmarking pipeline has been designed to accommodate higher-order oligomers, and we plan to extend the analysis to larger assemblies as well as newly resolved structures. The recent release of the standalone AlphaFold3 code further enables future benchmarking using local installations, including support for co-factors and small ligands, thereby broadening the scope of biologically relevant targets that can be evaluated. ProModEv is intended as an evolving resource, and we plan to update periodically to incorporate newly deposited PDB structures, additional predictors, and curated evaluation metrics.

Discussion

Recent advancements in computational techniques for predicting protein structures have reached a level of accuracy comparable to that of experimentally derived models. The release of AlphaFold2 in 2020 brought a transformative shift to the field of structural biology [8, 10, 18, 43]. In the face of its groundbreaking impact, the success of AlphaFold has sparked debate over whether experimental structural biology remains essential. However, critical evaluation of these deep learning models, particularly their limitations, remains necessary to understand their role in complementing, rather than replacing, experimental approaches. In this study, we benchmarked AlphaFold2, AlphaFold3, and ESMFold using experimentally resolved protein structures from 2022 to 2024. For monomeric proteins, AlphaFold2 and AlphaFold3 exhibited comparable performance, each accurately predicting 87%–88% of the structures, while ESMFold achieved 77% accuracy. When focusing specifically on X-ray and Cryo-EM structures, AlphaFold’s accuracy increased to 95%, and ESMFold to 83%, highlighting their strong performance for well-resolved targets. The most noticeable discrepancies in predictions were associated with structures solved by NMR, which display greater conformational variability. Among the 190 monomeric NMR structures, AlphaFold2, AlphaFold3, and ESMFold failed to accurately predict 128 (67.3%), 135 (71%), and 139 (73%) structures, respectively. This high misprediction rate likely results from the models’ limited training on NMR-derived data, which tends to be more flexible and ensemble-like compared to the crystalline rigidity of X-ray structures. Earlier benchmarking studies of AlphaFold2 against NMR structures have also observed significant differences between prediction results and solution conformations of proteins [23, 44, 45]. NMR-derived proteins are often small, flexible and partially disordered, and are deposited as ensembles of conformations rather than a single static structure. In contrast, AlphaFold and ESMFold generate a single ‘best’ conformation. Therefore, we carried out additional analysis involving comparison of the predicted structure with each model of the ensemble and counted it as a correct prediction if any of the models showed low RMSD or high LDDT with the predicted structure. This approach involving comparison of the AlphaFold predicted structures with each conformation of the NMR ensemble resulted in an increase in the accuracy by 10%, i.e. from 27% to 37%. This suggests that a higher degree of disagreement between AI/ML predicted structures and NMR structures may relate to conformational flexibility. It may also be noted that we have compared only the AlphaFold predicted model with the highest pLDDT with the NMR ensembles. Comparison of other models may further reduce the disagreement between NMR structures and AI/ML predicted structures. The other source of disagreement of NMR structures with AlphaFold or ESMFold predictions could arise from structures which have significant portions which are flexible or harbour IDR. Therefore, we calculated the IDR regions for each NMR structure using AIUPred software [46] and also calculated the residue-wise mean RMSF for the ensemble of conformations obtained from NMR. It was found that NMR structures with a higher percentage of intrinsically disordered residues also showed high RMSF values for the corresponding regions, thus suggesting predictions from AIUPred could be a true indicator of the extent of disorder or high flexibilities in the NMR structures. It was interesting to note that out of the 118 NMR structures, for which AlphaFold predictions were incorrect, 41 had IDR values above 50%. In Supplementary Fig. S16, we provide IDR values for the XRD, NMR, and EM structures, as well as the distribution of IDR values for the incorrectly predicted NMR structures. These results indicate that disagreement between AI/ML predictions and NMR could also arise from intrinsically disordered regions. These results suggest that global measures like RMSD and LDDT might not be appropriate for structures harbouring flexible regions. In such cases, residue-wise comparison of pLLDT with IDR score may be more appropriate.

Although our extensive analysis of AlphaFold’s performance on a non-redundant benchmark of monomeric structures indicates that AlphaFold is predominantly successful in predicting monomer structures, its accuracy decreases when predicting the structures of dimeric protein complexes. In the original AlphaFold-Multimer publication, the authors reported an accuracy of over 70% across a dataset of 4 446 dimeric protein complexes [47]. Additionally, in a benchmarking study by Yin et al. involving 149 protein complexes, acceptable or high structural accuracy was achieved in ~51% of the cases [27]. Bryant et al. reported a success rate of 63% for models with acceptable or higher accuracy across 216 protein–protein complexes using different MSA generation methods [48]. Similarly, Zhu et al. observed an accuracy of 60% in predicting acceptable dimer interface structures [28].

In our benchmark, AlphaFold2 and AlphaFold3 demonstrated nearly comparable accuracy in dimer structure prediction, with AlphaFold2 performing slightly better, achieving 76.8% accuracy compared to 75% for AlphaFold3. In contrast, EMSFold showed a significant drop in accuracy for dimeric protein–protein complexes, correctly predicting only 41% of the structures. ESMFold, although less accurate overall, offered unique advantages, such as faster runtime and less dependence on external MSA quality, making it a promising candidate in high-throughput scenarios. The challenges associated with dimer prediction are multifaceted, including modelling of interface regions, accommodating the conformational flexibility of individual monomers and capturing inter-protein coevolutionary signals, all of which complicate dimer structure prediction [27, 48–50]. ESMFold relies on protein large language model (pLLM) trained solely on sequences, without MSA or explicit co-evolutionary coupling signals. The training of ESMFold uses only pLLM embedding of the sequences of individual chains and structures in PDB to learn inter-residue contacts and oligomeric structures in PDB are also used as individual chains during this training process. Hence, it has a limited knowledge base on interface contacts present in oligomeric structures. Even though AlphaFold also uses very similar training module, by using MSAs of interacting chain pairs, it is able to get interface contacts from correlated mutations using attention network. While this design without MSA enables rapid predictions, it reduces accuracy for protein–protein interactions where co-evolutionary information is critical for interface modelling. Our analysis of dimers confirms this, showing that the majority of ESMFold failures occur at protein–protein interfaces with small or transient contacts. We further examined cases where ESMFold produced reasonable folds for monomers but failed to correctly orient interfaces, consistent with the lack of MSA-based evolutionary constraints.

Our analysis for evaluation of prediction accuracy of dimeric structures was carried out after verifying biologically relevant assemblies using the PDBe biological assembly annotations and non-biological interfaces/crystallographic artefacts were excluded prior to benchmarking. However, in the absence of information about true biological assemblies, one real-world difficulty often encountered involves annotation of the oligomeric status of entries containing two chains. To address this, we conducted a separate comparative analysis of the interface area for a subset of data consisting 483 actual dimers and a separate dataset of 285 misclassified monomeric assemblies (having two chains in PDB) and evaluated their respective DockQ scores to assess prediction quality (Supplementary Figs S17 and S18). In the actual dimers predicted by AlphaFold2 (Supplementary Fig. S17), DockQ scores generally increased with interface area. Among these, over 70% were classified as accurate or acceptable predictions (DockQ ≥ 0.23). The interface areas in these actual dimers ranged from 224.7 Ų to 8460.1 Ų, with a mean of ~1922.7 Ų. High and medium-quality predictions were characterized by well-defined and substantial interface regions, often exceeding 1200 Ų, highlighting a strong correlation between true biological dimer interfaces and reliable model quality. In contrast, inaccurately predicted dimers, those with DockQ <0.23, tend to have smaller interfaces, typically under 1000 Ų. This trend was particularly noticeable in monomers mistakenly annotated as dimers (Supplementary Fig. S18), where interface areas ranged from 0 to 1640.5 Ų, with a mean of 872.9 Ų. Notably, almost 90% of these misclassified dimers had DockQ scores below 0.23 and were classified as incorrect predictions where interface areas were also under 800 Ų, supporting the notion that they are unlikely to form stable biological interfaces. Such interpretation is further supported by previous structural studies, which have shown that biologically relevant protein–protein interfaces tend to exhibit significantly larger buried surface area compared to non-biological or crystallographic contacts. Lo Conte et al. reported a mean interface area of ~1600 ± 400 Ų for functional protein complexes [51], while Bahadur et al. observed that the range of interface areas extends from 500 to 7000 Ų, with a mean around 1970 Ų, depending on the protein’s functional classification [52]. Likewise, Caffrey et al. specifically analysed heterodimeric complexes and found their interface areas to span from 415 to 2361 Ų. These findings strongly align with our current results, where accurately predicted biological dimers exhibit significantly higher interface areas, while misclassified or incorrectly annotated dimers (actually monomeric in nature) cluster within lower interface ranges, often below 800 Ų. This further emphasises that the interface area can serve as a filter to identify biologically meaningful assemblies, particularly when used in conjunction with geometric tools like PISA. The slightly higher dimer prediction accuracy in our study compared to previous reports may be primarily due to the rigorous curation of biologically validated dimers in our benchmark dataset, excluding non-functional or non-biological assemblies.

All three structure prediction tools, i.e. AlphaFold2, AlphaFold3, and ESMFold, provide per-residue confidence estimates in the form of pLDDT scores. While these scores are widely used as indicators of model quality, our findings demonstrate that high pLDDT values do not necessarily signify structural accuracy. Among the monomeric predictions identified as incorrect based on structural comparisons, a significant proportion still exhibited high pLDDT values (Average pLDDT ≥70). Specifically, 75% (150 out of 200) of AlphaFold2, 83% (158 out of 191) of AlphaFold3, and 35% (137 out of 391) of ESMFold’s incorrect monomer predictions were associated with high pLDDT scores. A similar trend was observed in dimeric (protein–protein) predictions, where 85.3% (197 out of 231) of AlphaFold2, 88.3% (220 out of 249) of AlphaFold3, and 57% (333 out of 585) of ESMFold’s incorrect models still displayed high pLDDT scores. To evaluate how well pLDDT reflects actual prediction accuracy, we performed a calibration analysis for each predictor (AlphaFold2, AlphaFold3, and ESMFold) on both monomeric and dimeric benchmarks (Supplementary Figs S19 and S20). Out of the total number of structures predicted with a given pLDDT, we have calculated the fraction of structures having LDDT >0.7 and shown the fraction of correct predictions as a function of pLDDT. Calibration curves show that pLDDT is well aligned with accuracy only in the uppermost confidence bin (pLDDT > 90), while intermediate scores (70–90) contain a mixture of correct and incorrect structures, and the lower bin (<70) contains incorrect predictions. These findings underscore a critical limitation: a high pLDDT score reflects the model’s internal confidence but not necessarily its external correctness when compared to experimental structures. Overall, while pLDDT remains a valuable tool for quick evaluation of predicted models, our analysis clearly shows that it should not be used in isolation. Instead, it should be supplemented with external structural validation metrics such as RMSD, DockQ, and interface-specific lDDT scores. Additionally, visual inspection and context-specific analysis remain indispensable, especially when dealing with novel proteins, flexible regions, or complex assemblies.

In addition to accuracy metrics, our study evaluated the influence of homolog availability using the TQ metric. As expected, low TQ scores (especially TQ = 0) were significantly associated with mispredictions across all three tools. Structures with higher TQ generally fared better, indicating the persistent role of evolutionary information even in deep learning models. However, it was encouraging to observe that AlphaFold and ESMFold still succeeded in predicting a considerable proportion of structures with TQ = 0, demonstrating their capacity for generalization beyond template-based learning.

Overall, our study affirms that AI-based tools like AlphaFold and ESMFold represent a pivotal advancement in protein structural modelling. However, limitations remain, particularly in reliably predicting NMR-derived structures and intricate protein–protein interfaces. In addition, although AlphaFold remains the most accurate predictor for both monomers and dimers, its computational cost is substantial when run locally. For AlphaFold2, the runtime differs from half an hour to a few hours and increases with sequence length and MSA complexity A100 GPU (Supplementary Fig. S21). Our analysis shows that feature generation dominates total runtime, particularly for long sequences. This scaling behaviour is important for large-scale modelling projects or structural genomics pipelines. In contrast, ESMFold, being LLM-based and not requiring MSA generation, demonstrated significantly faster runtimes (30 s to 2 min per sequence when run on A100 GPUs) enabling faster predictions, though at the cost of lower dimer accuracy. We also note that AlphaFold3 prediction using the AlphaFold server is faster, and the results are produced in a few minutes based on the queue. These analyses underscore both the utility and the current boundaries of AI-powered prediction tools.

Our analysis indicates that the prediction accuracy of the structures varies depending on the structure prediction software used and the experimental method employed for structure determination. Hence, while using experimental workflows involving further experiments based on protein structure prediction, attention should be given to the extent of deviation between predicted and experimental structures of homologous proteins available in PDB. If prediction accuracy is suboptimal, either multiple models should be analysed or lower weightage should be given to structure-derived information. Family-wise analysis of prediction accuracy would also help in the design of suitable additional targets for structure determination for families that show poor agreement between experiment and prediction.

In conclusion, while AlphaFold2, AlphaFold3, and ESMFold have significantly advanced structural biology, continued benchmarking, interpretability research, and method refinement are crucial. The development of tools like our ProModEv platform further supports transparent evaluation, enabling the community to critically assess and compare different structure prediction models across diverse biological structures. Our dataset was cautiously restricted to structures released between January 2022 and December 2024 with <40% sequence identity and <70% query coverage relative to pre-2022 PDB entries. This allowed us to focus on challenging targets and reduce template bias. However, we acknowledge that this filtering also narrows the diversity of protein classes represented. We further note that future updates to ProModEv will incorporate higher-order oligomers and broader protein classes. We have designed ProModEv as an evolving resource, and future updates will incorporate newly deposited PDB structures, additional predictors, higher-order oligomers, and benchmarking support for the standalone AlphaFold3 implementation with ligands and other biomolecules.

Supplementary data

Supplementary data is available at NAR Genomics & Bioinformatics online.

Conflict of interest

None declared.

Funding

This work was supported by the Department of Biotechnology, Government of India grant to National Institute of Immunology, New Delhi. D.M. acknowledges financial support from Department of Biotechnology, India, under R&D project grants (BICB: BT/PR40325/BTIS/137/1/2020; DIAU: BT/BI/TCB/007/2021; and NNP: BT/PR40267/BTIS/137/67/2023 and BT/PR40160/BTIS/137/64/2023). The work was also supported by National Supercomputing Mission, MeiTY, India, under NSM-NPGDD project [MeitY/R&D/HPC/2(1)/2014/ CORP:DG:3191] grant to D.M. The S.K.M. acknowledges the MK Bhan Fellowship of Department of Biotechnology (HRD-16016/8/2025-HRD-DBT). S.V. also acknowledges the RCB-GSK PhD program in Bioinformatics and Biostatistics at RCB, Faridabad, for fellowship. The authors also acknowledge HPC resources provided by Indian Biological Data Centre.

Data availability

Publicly available datasets were analysed in this study. The complete protein sequences and native structure were retrieved from the PDB. All of the datasets supporting the results of this article are included within the article and its supplementary files. The detailed benchmarking matrices for individual structures are available at ProModEv and can be accessed using http://pdbi.nii.ac.in/ProModEv/ as well as deposited in Zenodo repository Zenodo DOI: 10.5281/zenodo.17749805.