Assessing the binding properties of CASP14 targets and models

Abstract

An important question is how well the models submitted to CASP retain the properties of target structures. We investigate several properties related to binding. First we explore the binding of small molecules as probes, and count the number of interactions between each residue and such probes, resulting in a binding fingerprint. The similarity between two fingerprints, one for the X-ray structure and the other for a model, is determined by calculating their correlation coefficient. The fingerprint similarity weakly correlates with global measures of accuracy, and GDT_TS higher than 80 is a necessary but not sufficient condition for the conservation of surface binding properties. The advantage of this approach is that it can be carried out without information on potential ligands and their binding sites. The latter information was available for a few targets, and we explored whether the CASP14 models can be used to predict binding sites and to dock small ligands. Finally, we tested the ability of models to reproduce protein–protein interactions by docking both the X-ray structures and the models to their interaction partners in complexes. The analysis showed that in CASP14 the quality of individual domain models is approaching that offered by X-ray crystallography, and hence such models can be successfully used for the identification of binding and regulatory sites, as well as for assembling obligatory protein–protein complexes. Success of ligand docking, however, often depends on fine details of the binding interface, and thus may require accounting for conformational changes by simulation methods.

1 |. INTRODUCTION

The Critical Assessment of Structure Prediction (CASP) is a bi-annual, world-wide prediction experiment where groups submit modeled structures for a number of different target protein sequences. Over the course of its 14 rounds the quality of predictions has improved steadily,^1–5 and the most recent rounds, CASP14, demonstrated a major step toward higher accuracy. In fact, the top models submitted to CASP14 were of such high quality that that they were often indistinguishable from the experimental structure, and in some cases were even used to help solve the coordinates. These unprecedented results have put a spotlight on protein structure prediction and its potential applications in drug discovery as it is generally understood that very high quality models are needed for tasks such as identification of binding sites, high throughput ligand docking, modeling lead compound specificity and binding affinity, and protein–protein docking.⁶ In this analysis we evaluate how well the models perform in these specific areas. We note that in recent CASP rounds the functional and biological evaluation of models has been heavily based on the notes of the experimentalist, indicating why they solved the structure and what biological insights they intended to find.^7–9 These notes have been used to assess ligand-binding pockets in some of the CASP targets depending on whether the X-ray structure was co-crystalized with a ligand, to examine if known motifs or critical patches existed in the models, or whether a site could be identified using a geometry-based site finding algorithm.⁸ The models have also been subjected to protein–ligand and protein–protein docking with limited success in the past. While these analyses provided insight into the functional relevance of protein models, the relatively small number of responses from the depositors limited the generality of this approach.

To overcome the above limitation, we have created a universally applicable method to assess the surface binding property conservation between the X-ray structures of targets and their models. Our analysis is motivated by solvent mapping, based on soaking protein crystals in aqueous solutions of organic solvents.^10,11 We utilize a computational analog of the method, implemented as the FTMap server, to determine the surface properties of a protein structure.^12,13 The distribution of small molecule probes across the protein surface are tabulated into a “binding fingerprint” consisting of the number of probes proximal to each residue in the protein structure. The surface property conservation is evaluated by calculating the Pearson correlation coefficient (PCC) between the binding fingerprints of the X-ray structure of a target and its models. Importantly, this assessment of surface binding properties can be applied universally, and requires no a priori information on the binding site. We have recently tested this approach on the CASP12 targets deposited in the Protein Data Bank¹⁴ (PDB) and found that the binding fingerprint PCC was correlated to the primary structure metrics in CASP, the Global Distance Test (GDT), and particularly noted that in order to capture the general surface binding properties of a protein, a GDT Total Score (GDT_TS) greater than 80 was necessary.¹⁵ However, while correctly modeling the backbone of the structure (GDT_TS > 80) was required to capture the surface properties, it did not guarantee accurate modeling of the surface, as in some models relatively small differences in particular side chain and loop conformations had a large impact. Our general observations aligned with a 2019 review of homology modeling for drug discovery that highlighted the remaining challenges, namely loop modeling, side chain modeling, and selection of the best model among many candidates.¹⁶ As will be discussed, the top models submitted to CASP14 have made significant advances in all aforementioned areas, therefore in this review we assess how the structural improvements in modeling translate to surface binding properties and binding site identification.

Modeling of loops and side chains can also have a large impact in both protein–ligand and protein–protein docking. In protein–ligand docking, it has been determined that docking into a receptor co-crystalized with a similar ligand doubles the success rate for poses with RMSD < 2 Å, relative to a receptor with no bound ligand.¹⁷ Moreover, docking into a model of the protein, created without any ligand-bound constraints proves even more challenging. In a large-scale study of ligand docking to homology models it was observed that GDT High accuracy (GDT_HA) correlated with the accuracy of docking poses, measured by distance-root mean square deviation (dRMSD), but the authors noted that orientations of side chains in the active site play an important role in current docking methods as well.¹⁸ In the light of the increased accuracy in CASP14 models, we ran ligand docking to models of three targets that had co-crystalized ligands in the X-ray structure. Due to the limited sample size, this part of the recent article provides limited general insight. As higher quality models become the new normal, an updated study on model performance in ligand docking would be more insightful.

Finally, protein–protein interactions (PPIs) play an important role in cellular signaling, and there is major interest in elucidating protein interaction networks. Protein–protein interfaces can also serve as targets of drugs to inhibit or modulate the interactions.¹⁹ Thus, it is important to correctly identify the structure of the oligomeric complexes, and protein–protein docking has proven a valuable addition to experimental methods.²⁰ In a recent study we considered the CASP12 targets that are subunits of multimeric structures, and docked both the subunits extracted from the complex and their models to the rest of complex.¹⁵ For CASP14 targets, we run the same analysis with one important addition, namely we docked models to models when both parts of a complex were considered as targets. This is clearly a more challenging test than docking models to X-ray structures. Nevertheless, we will show that the significant advancements in protein modeling substantially improved the success rates in protein–protein docking, opening the possibility of structure based construction or validation of protein interaction networks.

2 |. METHODS

2.1 |. Selection of CASP14 regular targets for the surface binding property analysis

For the analysis of surface binding properties, we considered all regular CASP14 target domains evaluated by the CASP14 assessors (Table S1). We focus on domain-specific mapping because many of the targets in CASP14 are large, multi-domain proteins, and domain-specific results provide more accurate comparisons between surface binding conservation and the structural properties. Among the 67 CASP14 regular targets, 48 targets have one domain, 11 targets have two domains, 6 targets have three domains, and 2 targets have four domains, resulting in 96 domains total. Each domain assessed is listed in Table S1. All canceled targets were excluded from this analysis (T1044, T1048, T1051, T1059, T1062, T1063, T1066s1/s2, T1069s1/s2, T1072s1/s2, T1075, T1077, and T1098). The X-ray structure of each target was submitted to the CASP14 assessors by experimentalists; some of these structures are now published in the Protein Data Bank (PDB), as shown on the CASP14 website.

For the functional analysis of the CASP14 targets, only the top five ranked models by GDT_TS were considered. We refer to these models as Top1, Top2, and so on, and if more than one target was given the same rank, we simply considered the top 5 models in the order they were presented in the CASP14 assessor results files so that only five models were assessed for each target. Finally, for the rare cases where identical models were submitted for a target, defined by equivalent GDT_TS, GDT_HA, and GDC_SC values to two significant digits, only one representative structure was considered, and subsequent structures were added until five unique models were selected. The GDT scores, along with all other structural scores mentioned in this article are published for each target on the CASP14 website.

2.2 |. Selection of CASP14 refinement targets for the surface binding property analysis

CASP14 organizers selected a number of targets and their initial models to determine if the models can be refined into higher quality models. In CASP14 this provided a unique challenge as the quality of the models submitted for most targets reached an unprecedented high. In an effort to select models that might be reasonably improved, the organizers selected lower quality models for 30 regular targets. All 30 CASP14 refinement targets are analyzed here for surface binding property conservation (Table S2). We consider them separately for two reasons: first, the refinement targets are duplicates of regular targets, and second, the refinement targets in CASP14 are typically of lower quality (by GDT_TS) than the best models for their respective regular targets. Note that refinement targets begin with an “R,” prior to the target number, whereas regular targets begin with a “T.” Many of the refinement targets were analyzed with many different parameters, such as specific domains (D1, D2), extended time (x1, x2) or with different starting models (v1, v2). Wherever possible, the domain, version 1 and no extended time were selected for simplicity; the specific targets analyzed are noted in Table S2. As with the regular targets, the X-ray structures for the refinement targets were provided to the CASP14 assessor team by experimentalists, and trimmed to their respective domains by the assessors.

2.3 |. Selection of targets for protein–protein docking analysis

For the analysis of PPIs, we considered the CASP14 regular targets characterized as multimers by the organizers, and then selected only those where a multimeric structure was provided or could be generated using PISA.²¹ We excluded three targets due to their large size: H1060 (26 subunits), H1047 (52 subunits), and T1099 (240 subunits). The 16 multimeric targets considered for analysis included seven homodimers, two heterodimers, three trimers, two homotetramers, a heterotetramer (A2B2) and a hetero-9-mer (A3B3C3). Both subunits of the heterotetramer were modeled separately and so we were able to consider two different interfaces for the target (chain A docked to chains BCD, and chain B docked to chains ACD). This resulted in 17 interfaces for the analysis.

2.4 |. Evaluation of surface binding properties and binding sites

Solvent mapping has been shown to reveal small molecule binding “hot spots” that contribute disproportionately toward a ligand’s binding free energy.^11,22,23 Here, we perform a computational analog of solvent mapping, where the free energy of small molecule probes are calculated across the protein surface using Fourier transforms.¹² Traditionally, the low energy conformations of these probes are clustered to form consensus clusters of multiple probe clusters that represent the binding hot spots.¹³ However, for the evaluation of protein models, instead of clustering the probes we have created a “binding fingerprint” to represent the distribution of probes bound across the protein surface, rather than a particular hot region. The binding fingerprint indicates the number of probes within 3 Å of each residue in the structure. The conservation of surface binding properties between two structures is easily compared by calculation of the Pearson correlation coefficient (PCC) between the X-ray and model binding fingerprints.

For the identification of the ligand binding sites in CASP14 targets, computational solvent mapping was again utilized, but rather than creating the binding fingerprints, we clustered the low energy probes to ultimately represent the ligand binding sites.²⁴ In this analysis, the binding sites were ranked by the number of probe clusters in each site, indicating the strength of the binding site. This method can essentially be broken down into two steps, first the use of computational solvent mapping to generate binding hot spots, and then joining of overlapping hot spots to create ligand binding sites. Both steps have been implemented in public servers, FTMap (ftmap.bu.edu)¹³ and FTSite (ftsite.bu.edu),²⁴ respectively. These methods of binding site detection have been used in proteins that bind various ligands, including traditional small-molecule drugs,²⁵ macrocycles,²⁶ and beyond rule-of-five drugs.²⁷

To assess the conservation of ligand-binding sites, the site overlapping with the ligand in the experimental structure is identified. If more than one site is overlapping, the strongest site is selected for simplicity. The top five models with the bound probes placed by FTMap are aligned by the PyMol²⁸ align function to the experimental structure, also with FTMap generated probes, and the overlap with the experimental ligand binding site is calculated as the percentage of probe clusters in the model within 2 Å of the probe clusters in the experimental binding sites.

The mapping results presented in this study were obtained by running a command-line version of FTMap and FTSite to accommodate the large number of mapping jobs required to assess the CASP14 targets. As a result, the specific strength and placement of the probes and binding sites presented here may vary slightly from the output of the publicly available servers. The method described here was also used to evaluate the surface properties of the CASP12 targets.¹⁵ We note that potential crystal contacts have not been taken into account in the mapping calculations. This decision was made for two reasons. First, in our many previous articles FTMap has been extensively tested in reference to experimental fragment binding and was able to reproduce important fragment binding profiles without any special consideration given to crystal contacts.^{13,25,29–32} Second, accounting for crystal contacts would require simultaneously mapping multiple copies of the target protein. FTMap is currently unable to perform such calculations, and developing and validating a modified version of the program would require work beyond the limits of the current article.

2.5 |. Selection of targets and evaluation with ligand docking methods

Among the CASP14 targets, 15 proteins were co-crystallized with some form of a ligand bound. For nine of these targets (T1025, T1028, T1034, T1038, T1052, H1065, T1074, T1079, and T1101) the ligand was a crystallization additive (i.e., ACT, CIT, EDO, FMT, GOL, IMD, NAG, MPD, MLI, PEGA, and TBU) and thus not suitable for ligand docking. Of the remaining six targets, T1058 was excluded from docking because it is bound to HEME, a large obligatory cofactor. T1024 was excluded from docking because the best model (T1024TS427_3) had a GDT_TS of 79.22, which is too poor quality for docking. Note, the ligands (LMU, HT1, and XP4) bind between the domains, so GDT_TS of structures with both domains were considered. Finally, T1049 was excluded from docking as the ligand, glutamic acid, is an amino acid. As a result, three targets were considered for ligand docking: T1053-ADP, T1057-SAM, and T1076-ADP. Note that T0176 has another ligand (TZD) bound between its two chains, but interchain binding was excluded from this study.

Protein-ligand docking was performed with two different methods: AutoDock Vina v1.12³³ and ClusPro LigTBM.³⁴ For docking with AutoDock Vina, preparation of the protein structures included removing all non-protein molecules, such as ligands, ions, and water molecules. Next, the protein structures were prepared using AutoDockTools (ADT) v1.5.7³⁵ to add missing hydrogen atoms and convert the structures to PDBQT files expected as inputs for AutoDock Vina. Ligand structures were constructed from the SMILES description given in the PDB, and missing hydrogen atoms were added with OpenBabel v2.4.0,³⁶ followed by a manual check to ensure pH 7. The resulting ligand structures were processed through ADT to create the PDFQT files needed as input for docking. All docking simulations were carried out using default AutoDock Vina parameters, except for the exhaustiveness, which was changed to 10 (the default setting is 8). In each case, the search space was confined to a box with 15 Å sides, centered at the geometric center of the ligand atoms in the crystal structure. Each ligand was docked to the X-ray structure of the CASP14 target and to the 30 top models using AutoDock Vina.

Ligand-docking with ClusPro LigTBM only requires the sequence of the protein and the SMILES string of the ligand, as the method identifies available templates in the PDB, and builds a protein model using MODELLER.³⁴ As a result, the CASP14 target sequence was input for the three targets assessed here (T0153, T1057, and T1076) along with the ligand SMILES string given in the PDB. The docking search space is not confined, as the ligand structure is placed based on available templates found in the PDB.³⁴ For both docking methods, the top 5 models output from each docking run were retained for RMSD assessment, computed with DockRMSD,³⁷ where the native ligand structure was used as a reference. Finally, the local distance difference test (LDDT) was calculated for all residues within 10 Angstroms of the bound ligand, for each of the 30 models. The docking method described here was also previously used to evaluate three CASP12 targets.¹⁵

2.6 |. Protein–protein docking of predicted structures

As noted earlier, we considered CASP14 regular targets that were known to be parts of multimers. We emphasize that here we investigated whether the models of such targets can replace the X-ray structures to form the complex, and hence used classical docking of subunits rather than any method to directly model the complex. The subunits were docked with the rigid-body docking program ClusPro,²⁰ a publicly available server (https://cluspro.bu.edu/), which is free for academic use. The engine of the ClusPro server is a program called PIPER that performs docking in 6D rotational and translational space.³⁸ The program efficiently samples billions of conformations using Fourier transforms, and outputs the low energy complexes to the user. A variety of coefficients are provided to the user, and the balanced set of coefficients are set as the default. For this analysis we used the electrostatically favored coefficients (or coefficient set 2), as they have been shown to produce equivalent or better docking results than the balanced coefficients.³⁹

Docking of targets can be submitted one of two ways: receptor–ligand docking or, in the case of homo-multimers, symmetry docking. In receptor–ligand docking, the receptor and ligand structures are input separately, the receptor is stationary and the ligand is mobile. In general, the larger subunit is selected as the receptor and the smaller subunit is selected as the ligand—this practice was maintained in this study, wherever possible. ClusPro samples essentially all orientations of the rigid subunits, produces 70 000 docked conformations, selects and clusters the 2000 lowest energy structures, and provides the centers of the largest clusters as predicted structures of the complex.²⁰ In symmetry docking, a monomer structure is input along with the predicted oligomeric state (i.e., dimer, trimer). ClusPro uses copies of the monomer for all the subunits and restricts conformation sampling to orientations where the rotational symmetry of the predicted oligomeric state is satisfied. Both methods were used to assess the homomultimeric CASP14 targets, and we indicate specific cases where symmetry docking was used.

The quality of the protein complex created via docking was assessed with a program called DockQ.⁴⁰ The continuous score output by DockQ was developed to encompass the three different RMSD scoring metrics used by Critical Assessment of Predicted Interactions (CAPRI)⁴¹, a community wide docking test similar to CASP, to assess accuracy of docked structures. The DockQ score ranges from 0 to 1, where >0.80 indicates high quality, >0.49 indicates medium quality, >0.23 is acceptable, and <0.23 is incorrect.

For evaluation of CASP14 targets, we performed three sets of docking. First, X-ray to X-ray docking was performed, where the X-ray subunits were submitted by experimentalists and prepared by the CASP assessment team. For all X-ray to X-ray docking cases, receptor-ligand docking was utilized. Second, X-ray to model docking was performed, where one subunit consisted of the model, and it was docked to the X-ray structure of the remaining oligomeric unit. For each target, five sets of docking were performed to analyze the top 5 models (ranked by GDT_TS) submitted for each target. Third, model-to-model docking was performed for CASP14 targets that were classified as homodimers or homotrimers, or for heterodimers with multiple chains selected as targets. Rather than assessing the top 5 models for each target, we focused on the groups that performed well in CASP14 (427, 403, 9, 420, 42, 324, 480, 129, 209, and 473), and the top model from each group was used for evaluation of model docking. For these cases, heterodimers and homodimers were docked with receptor–ligand docking, where the input structure for each subunit was the top model submitted by each group. For example, evaluation of docking group 427’s top models for heterodimer target H1065 consisted of docking T1065s1_427_1 (receptor) to T1065s2_427_1 (ligand). Likewise, evaluation of homodimer target T1054 consisted of docking T1054_427_1 to itself. Homotrimers were docked using symmetry docking such that the model of the subunit was the only input structure. Notably, one heterodimer (H1045) was excluded from model to model docking because models were missing for one subunit (T1045s1) from seven of the groups. For each type of docking, the top 10 complexes produced by ClusPro were retained for assessment with DockQ.

3 |. RESULTS

3.1 |. Surface binding property conservation for CASP14 targets

Comparison of the FTMap binding fingerprints allows us to assess the overall conservation of binding properties between two homologous structures without knowing any information about particular binding sites a priori. In CASP14, as in many of the preceding CASP rounds, only a handful of experimental structures (6 of the 67 regular targets) were submitted with a biological ligand co-crystalized with the protein. Thus, for many of the CASP14 targets, particular binding sites of interest have not been identified and assessment of a more general conservation of surface binding properties was necessary.

Here, we assess the general binding site properties by comparing the probe binding fingerprints of the predicted and experimental structures with a Pearson correlation. An example of structures with high conservation of surface properties is shown by target T1091-D1. One predicted structure for T1091-D1 has a binding fingerprint Pearson correlation coefficient (PCC) of 0.96 with the experimental structure (Figure 1A,B); this structure (T1091TS427_5-D1) was ranked 4 by CASP with GDT_TS = 92.63. With such high GDT_TS and binding fingerprint PCC, the three-dimensional protein representations and probe fingerprints (Figure 1C) appear almost identical. Alternatively, predictions for target T1029-D1 provide an example of very low correlation of probe placement relative to the experimental structure (see Figure 1D–F). Among the top 5 predictions for T1029-D1, the GDT_TS ranges from 45.80 to 46.40 and the binding fingerprint PCC ranges from 0.03 to 0.20. The model with the highest PCC (T1029TS364_1-D1, GDT_TS = 45.80) is shown in Figure 1E, and differs significantly from the experimental structure. While some of the main binding sites are captured in the models, many of the surrounding probe regions differ, resulting in poor correlation (PCC = 0.20) with the experimental structure.

FIGURE 1.

Examples of mapping for CASP14 targets. (A) Experimental structure of CASP14 target T1091-D1 with probes overlaid. The protein is shown as grey cartoon, and clusters of probes molecules as colored sticks. (B) Predicted structure for T1091-D1 (Model T1091TS247_5-D1, GDT_TS = 92.63, CASP Rank = 4) with the highest binding fingerprint PCC = 0.96. (C) Binding fingerprints for the T1091-D1 experimental structure (black) and the model shown in B (green). (D) Experimental structure of T1029-D1 with probes overlaid. (E) Predicted structure for T1029-D1 (Model T1029TS364_1-D1, GDT_TS = 45.80, CASP Rank = 5 [tie for 3rd]) with poor correlation to the experimental structure, PCC = 0.20. (F) The binding fingerprints for the T1029-D1 experimental structure (black) and modeled structure shown in E (red)

While it is clear from these examples that the top models for T1091-D1 represent highly correlated binding fingerprints and the models for T1029-D1 show poorly correlated fingerprints, we look to the correlation of experimental homolog structures to define a threshold between “good” and “bad” correlations. From our evaluation of CASP12 targets, assessment of 51 pairs of CASP12 homologs deposited in the PDB revealed the binding fingerprint PCC between homologs is generally greater than 0.5, and have an average PCC = 0.80 ± 0.16.¹⁵ We observed that some natural variation in the structures can create minor changes in the probe binding fingerprints, but that a PCC > 0.5 generally indicates good conservation of main binding regions. Therefore, in this analysis we consider models with PCC ≥ 0.50 to have good conservation of surface properties. We note that more significant variation of surface properties is possible among homologs with flexible binding pockets, however such cases were rare among CASP12 targets and do not seem common among CASP14 targets either, as will be shown by the conservation of the models.

The binding fingerprint PCC was calculated for the top models of all CASP14 regular target domains. For each of the 96 domains, the top 5 models submitted to CASP (ranked by GDT_TS) were assessed, and their binding fingerprints were compared with the experimental structure, see Figure 2. The targets are organized by their classification, indicating whether template-based-modeling (TBM) or free-modeling (FM) could be used. The classifications are: TBM-Easy, TBM-Hard, FM/TBM, and FM, in order of increasing difficulty, see Figure 2. The average PCC across all 96 domains is 0.70, with SD 0.20. Although this is lower than the PCC = 0.80 value observed between highly homologous X-ray structures, the majority of the CASP14 domains (81 of 96, or 84.4%) had an average PCC ≥ 0.50 across the top 5 CASP-ranked models, indicating that the differences between models and X-ray structures in terms of surface binding properties are within the range of the differences between X-ray structures of homologous proteins. Such high quality structures have never been seen before for so many targets in the history of CASP. For example, the average binding fingerprint PCC for published targets in CASP12 was 0.33 ± 0.24, markedly lower than 0.70 reported here.¹⁵ This improvement in overall conservation of binding site properties is accompanied by a drastic improvement in other CASP scoring metrics as well. The 15 domains with poor surface property conservation are spread across all prediction categories. Not surprisingly, the most challenging category (FM) had the most targets with poor binding fingerprint correlations (8 of the 23 targets with PCC < 0.50), see Figure 2. T1029-D1, discussed previously, is one of the eight FM targets with poor correlation.

FIGURE 2.

Average binding fingerprint PCC for the five top CASP-ranked models for 96 CASP14 regular target domains. The center-hash (black) indicates average PCC for the target domain and whiskers extend to the minimum and maximum PCC. Target domains are arranged in order of increasing difficulty as TBM-easy (green), TBM-hard (yellow), FM/TBM (orange), and FM (red)

3.2 |. Binding fingerprint correlation versus CASP scoring metrics

Ranking of models by the CASP assessors is based on either GDT_TS or an assessor’s formula that combines a number of metrics, defined based on the difficulty level of target. In this study we have considered the top 5 targets ranked by GDT_TS. In our previous study of CASP12 targets we found a correlation between binding fingerprint PCC versus GDT_TS (R² = 0.52), and observed that models with GDT_TS ≥ 80 had about a 50% chance of surface property correlation in the homolog range (PCC ≥ 0.50). In this study of CASP14 targets, the relationship between the binding fingerprint PCC and GDT metrics are shown in Figure 3. Due to the shift in CASP14 model quality toward higher GDT_TS scores, and thus reducing the spread of data, the correlation between binding fingerprint PCC and GDT_TS is weaker here (R² = 0.11) than was reported for CASP12 targets (see Figure 3A).¹⁵ The number of targets with models that have GDT_TS ≥ 90 is far higher in CASP14 than ever before. For CASP14 targets, we still see that targets with an average GDT_TS above 80 (but <90) have about a 50% chance of having homolog-level surface correlation. However, we now observe targets with an average GDT_TS ≥ 90 are almost guaranteed to have homolog-level surface property conservation, apart from a few exceptions. Thus, the unprecedented accuracy of models submitted to CASP14 is accompanied by remarkable surface binding property conservation, showing promise for meaningful use of these models in drug discovery applications. We note that GDT_TS separates models with high binding surface conservation well, as there is a high density of points in the upper right corner of the correlation (Figure 3A). We also compared the binding fingerprint PCC to GDT high accuracy (GDT_HA, see Figure 3B), Global Distance Calculation for sidechains (GDC_SC, see Figure 3C), and the global local distance difference test (LDDT, see Figure 3D). The correlations are similar across all these metrics, with slightly higher correlations for GDC_SC (R²_regular = 0.17) and LDDT (R²_regular = 0.25) due to the wider distribution of values. The binding fingerprint PCC and CASP metrics for regular targets are given in Tables S1 and S3.

FIGURE 3.

Binding Fingerprint PCC versus select CASP scores. The 96 CASP14 regular target domains are shown in black and the 29 CASP14 refinement targets are shown in green. Points indicate the average values across the top 5 CASP-ranked structures for each target, and SD is indicated by the light gray lines. Linear regression for regular and refinement targets are shown as black and green lines, respectively. Plots indicate binding fingerprint PCC versus (A) GDT_TS, (B) GDT_HA, (C) GDC_SC, and (D) LDDT

Figure 3 also shows the binding fingerprint PCC values for CASP14 refinement targets. Since the goal of refining models is to improve the accuracy of predictions, we expected somewhat higher average PCC value than for the regular targets. This was the case in CASP12, where the refinement slightly increased the average PCC, from 0.33 ± 0.24 to 0.40 ± 0.27. In CASP14, however, there was little room for refinement among the highest quality models, therefore lower quality models were selected for refinement, and as a result the average GDT_TS = 78.57 ± 11.08 of the refinement targets is significantly lower (P < .0001) than the average GDT_TS = 88.36 ± 10.8 of their regular counterparts. Accordingly, the average of binding fingerprint PCC values dropped from 0.70 ± 0.20 (Table S4) to 0.56 ± 0.21 (Table S2).

Three targets, T1053-D1, T1070-D4, and T1036s1-D1, appear as outliers by having very high GDT_TS scores but low binding fingerprint correlation coefficients (see Figure 3A). Target T1053 is a T4SS effector protein (or a PI3 kinase) from Legionella pneumophila with two domains. The surface properties of models are well conserved in the second domain (GDT_TS_avg = 91.96, PCC_avg = 0.87), but not in the first domain (GDT_TS_avg = 96.79, PCC_avg = 0.16), despite the high GDT_TS values. This difference can be attributed to the lack of two flexible loops (residues 19–28 and 175–185) in the experimental structure that surround the two strongest binding sites identified by FTMap (see Figure 4A). First, the strongest binding site in the X-ray structure is in a pocket near the loop of residues 175–185, and because the structure of this loop is unresolved in the X-ray, the probes are able to congregate in this area and appear as a strong binding site. In the models, the loop is present and its orientation closes off the binding site completely, inhibiting any probes from congregating in this region in any of the models. Based on these observations, we expect this loop may pose strong competition for a ligand binding in this region, and molecular dynamics simulations focusing on the movement of this loop may help reveal how frequently this site is actually accessible for ligand binding. Second, the loop of residues 19–28 near the known ATP/ADP binding site of this kinase is also unresolved in the X-ray structure. While the ATP/ADP binding site, with 18 probe clusters, is present in the X-ray structure, it is actually predicted to be stronger in the models (>30 probe clusters), which have coordinates for this loop modeled. The orientation of the loop appears to create a stronger pocket around the ADP binding site, enhancing the probe binding in this region. Since the ADP site is generally the strongest binding site in kinases, restoring the missing loops with modeling reflects the biological significance better than the X-ray structure. Another rather large flexible loop (residues 204–238) is also missing from the X-ray structure, but does not seem to impact any binding sites.

FIGURE 4.

FTMap predicted binding sites for structures of T1053-D1 (A and B) and T1070-D4 (C and D). Binding sites are ranked according to strength and colored as follows: Site 1 cyan, Site 2 magenta, Site 3 yellow, Site 4 salmon, and Site 5 pale-green. (A) The experimental structure of T1053-D1 bound to ADP (shown as pink sticks). (B) The top 1 ranked model (T1053TS427_2-D1) for T1053-D1. Flexible loops in the model that are not present in the experimental structure are colored, containing residues 19–28 (blue), 175–185 (red), and 204–238 (green). (C) The experimental structure of T1070-D4. (D) The top 5 ranked model (T1070TS427_2-D4) for T1070-D4

Target T1070 is a tailspike protein from Escherichia virus CBA120 with four domains (Uniprot ID G3M192). The fourth domain of residues 265–332 is classified as TBM Easy. The surface properties are generally conserved between the top 5 models and the experimental structure, but some slight variations put the average binding fingerprint PCC at 0.38 with the SD of 0.03, falling below the homolog range discussed above. These differences can be visualized by looking at the binding sites identified for the X-ray and model structures (see Figure 4C,D). The strongest binding site in the X-ray structure (Figure 4C, cyan site) is captured in most models (Figure 4D, magenta site), but the third strongest binding site (Figure 4C, yellow site) is very weak and may not even be seen in the top 5 models. The variation in the strength of this binding site may be attributed to the orientation of the ILE265 side chain, and ranges from a maximum strength of 24 probe clusters to a minimum of 0 probe clusters (see Figure S1). Group 427, or AlphaFold 2,⁴² submitted all five top-ranked models and while they are all incredibly similar, each model has a slightly different orientation of ILE265, ranging from completely closing off the binding site (Top 2 and Top 5 models), to slightly more open orientations (Top1 and Top 3 models), and to the most open orientation (Top 4 model). The latter binds 17 probe clusters, compared with 24 in the X-ray structure. The drastic change in binding site strength observed in these models is due to very small changes in side chain placement, which shows the orientation of key side chains can have substantial impact on ligand binding properties. However, ILE265 is at the amino end of the domain, and hence the site seen in the X-ray structure can be an artifact. More generally, a model can be of very high quality (i.e., GDT_TS > 90) but still miss (or overestimate) the local binding potential at a particular region due to subtle changes in the residue side chain orientations surrounding the site. Here, molecular dynamics studies of models and their respective X-ray structures may be useful in determining the potential of a given binding site.

Target T1036s1 is a human monoclonal antibody against the varicella-zoster virus, and domain one covers the entire antibody. None of the top 5 models submitted for this target has good prediction of surface binding properties (PCC_avg = 0.11), despite all having a GDT_TS greater than 87. In general, this structure is larger than proteins typically submitted for mapping, and further breakdown of domains may help to better distinguish the true differences in binding properties between the experimental and modeled structures. As it is, mapping of the entire antibody shows the binging fingerprints of the model are significantly different from the X-ray structure in essentially all areas.

An opposite scenario is observed in target T1047s1-D1, where the top 5 predictions have relatively low GDT_TS < 50, but all still have good conservation of binding properties (PCC_avg = 0.74). T047s1-D1 contains two beta sheets that cross over and then extend in different directions. According to the X-ray structure the crossover region contains all the binding sites, and these are predicted quite well in all models. Thus, the structural conservation of the binding region in the models explains the high binding fingerprint PCC. The regions of the structure that are less accurately predicted (and lower the GDT_TS score) have no binding sites, and therefore do not impact the overall binding surface properties (see Figure S2).

3.3 |. Binding site conservation in ligand-bound CASP14 targets

Six of the CASP14 targets were co-crystalized with one or more ligands bound to the protein, creating an opportunity to analyze conservation of these binding pockets in the predicted structures. The ligand-binding site in each X-ray structure, along with the strength of the site in the models, is defined as the number of probe clusters within 3 Å of the ligand. This measure was used to determine how well the sites were retained in the models, see Table 1. Additionally, the binding site residues for each target-ligand pair are listed in Table S5. Across all six targets, the models captured the ligand-binding sites very well, with the exception of the Top3 model for target T1024.

TABLE 1.

Number of FTMap probe clusters binding to CASP14 targets with known ligand binding sites

			X-ray	Number of probes within 3 Å of ligand
Target	PDB ID	Ligand	Site #|Probes	X-ray	TOP1	TOP2	TOP3	TOP4	TOP5
T1024	6T1Z_A	HT1	1|29	29	31	53	3	36	69
T1024	6T1Z_A	XP4	3|16, 4|14, 5|10	40	42	72	0	85	89
T1049-D1	6Y4F_A	GLU	3|12	12	15	14	17	16	23
T1053-D1	–	ADP	2|18	18	41	44	37	34	29
T1057-D1	–	SAM	1|41, 2|18	59	64	57	62	65	61
T1058-D1	–	HEMa	1|19, 4|12, 5|11	42	60	62	68	62	61
T1058-D1	–	HEMb	2|17, 3|16	33	19	23	19	22	17
T1076-D1	–	ADP	1|33, 6|2	35	42	47	34	37	33

^aHEM bound to T1058-D1 near HIS-198.

^bHEM bound to T1058-D1 near HIS-136.

Target T1024 encodes the protein LmrP, a prototypical multidrug transporter, from Lactococcus lactis. The structure has been deposited in the PDB (6T1Z) co-crystalized with three ligands: 1,2-dimyristoyl-SN-glycero-3-phosphate (XP4), Hoechst 33342 (HT1), and dodecyl-alpha-d-maltoside (LMU).⁴³ The latter is bound on the outside of the protein within its transmembrane region, and since mapping with FTMap is not optimized for such hydrophobic sites, we focus only on the inter-domain binding ligands XP4 and HT1. Debruycker and colleagues⁴³ noted that binding of HT1 seems to stabilize the protein in the “outward-open” conformation, and that binding of the lipid (XP4) nearby HT1 may provide a “hydrophobic counterpart” involved in ligand binding. In the experimental structure, the strongest binding site (Site 1) overlaps with ligand HT1, and sites 3, 4, and 5 overlap with the lipid, XP4 (Figure 5A). The Top1 model for T1024 (T1024TS427_3, GDT_TS = 79.22) has the binding fingerprint PCC = 0.55. The binding sites in the model are of similar placement as the sites in the experimental structure (see Figure 5B), and present with similar strengths of 31 probes clusters in the HT1 site and 42 probe clusters in the XP4 site, relative to 29 and 40 probe clusters in the X-ray structure, respectively (see Table 1). However, the HT1 and XP4 binding sites were not predicted in the Top3 model for T1024 (T1024TS226_5, GDT_TS = 65.73) (see Figure 5C). The models for target T1024 have lower GDT_TS values than for the other targets in Table 1, and the lack of binding sites is reflected in the poor binding fingerprint PCC of 0.06 (see Figure 5D).

FIGURE 5.

Experimental and predicted binding regions for target T1024 (LmrP) with ligands HT1 (green) and XP4 (pink) overlaid. Binding sites are shown in mesh, ranked according to their strength, and colored as follows: Site 1 cyan, Site 2 magenta, Site 3 yellow, Site 4 salmon, and Site 5 pale-green. (A) Experimental Structure of LmrP (PDB ID: 6T1Z) co-crystalized with ligands HT1 and XP4. (B) Model of target T1024, top 1 ranked by both GDT_TS and PCC (Model T1024TS427_3, GDT_TS = 79.22, PCC = 0.55). (C) Model of target T1024, top 3 ranked model by GDT_TS, but with the lowest binding fingerprint PCC (Model T1024TS226_5, GDT_TS = 65.73, PCC = 0.06). (D) Binding fingerprints for the X-ray structure (6T1Z), top 1 model with the highest PCC (T1024TS427_3, green), and model with the lowest PCC (T1024TS226_5, red)

The ligand-binding sites were predicted across all top 5 models for the other six targets with known ligand-binding sites, as shown in Table 1, albeit with some variations in strength. We note that based on the analysis of X-ray structures, sites with more than 13 probe clusters are generally capable of binding small molecules with micro-molar or better affinity, and sites with more than 16 probe clusters are predicted to be druggable.²⁵ Variations in binding site strength is also expected among homologs, depending on the orientation of the surrounding side chains. However, without dynamic simulations of the binding sites we cannot comment on the realistic minimum and maximum strength of each binding site. Therefore, we focus largely on the prediction of the location of the binding site, with less emphasis on its strength.¹³

Note that all top 5 models for five targets shown in Table 1 were submitted by group 427, AlphaFold 2 of Google DeepMind.⁴² The ligand binding sites for the experimental structures and the top ranked models are shown in Figure 6. For target T1049-D1, all five top models have GDT_TS > 90 and binding fingerprint PCC > 0.82. The Top 1 model is shown in Figure 6, indicating that the ligand-binding site is similar to the one in the experimental structure. For target T1053-D1 all five top models have GDT_TS > 96, but low binding fingerprint PCC ≤ 0.20 due to the lack of two flexible loops in the X-ray structure as discussed previously (Figure 4). Nevertheless, the ADP/ATP binding site is identified in all five top models, and it is predicted with higher strength, likely due to the presence of the nearby loop (residues 19–28) that creates a stronger pocket around the binding site (see Figure 4B). For target T1057-D1, the SAM binding site is observed in all five top models, with similar strength to the X-ray structure. These models have very high GDT_TS (>93) and high binding fingerprint PCC > 0.90. Hemoglobin (HEM) binds in two locations to target T1058-D1, and the top 5 models have average GDT_TS = 88.04 and PCC = 0.61. Both HEM binding sites are predicted well in the five top models, with similar but slightly more probe clusters than in the X-ray structure. Finally, the ADP binding site for target T1076-D1 is well predicted in all five top models that have very high GDT_TS (>98) and good binding fingerprints correlations (PCC > 0.67).

FIGURE 6.

Binding sites identified in CASP14 targets co-crystalized with a ligand bound to the protein. The experimental structure is shown on the left and the top 1 ranked (by GDT_TS) prediction is shown on the right. The ligand is shown as sticks and the binding sites found for each structure are ranked according to strength and shown as mesh pockets with the following colors: Site1—cyan, Site2—magenta, Site3—yellow, Site4—salmon, Site5—pale-green, and Site6—orange. The strength of the binding site is indicated by the number of probe clusters in parenthesis. Target T1049-D1 is bound to glutamic acid (GLU). In the experimental structure we have found Site3 (12) and Site6 (5). In the top predicted structure (model T1049TS427_3-D1) found are Site3 (13) and Site6 (2). Target T1053-D1 is bound to adenosine-5′-diphosphate (ADP). In the experimental structure we identified only Site2 (1). In the top predicted structure (model T1053TS427_2-D1) we found the well-populated Site1 (25). Target T1057-D1 is bound to S-adenosylmethionine (SAM). In the experimental structure FTMap identified Site1 (41), Site2 (18), Site5 (2), and Site6 (2). In the top predicted structure (model T1057TS427_1-D1) we found Site1 (64) and Site4 (2). Target T1058-D1 is bound to hemoglobin (HEM). For binding site 1, in the experimental structure we find Site1 (19), Site4 (12), and Site5 (11); in the top predicted structure (model T1058TS427_1-D1) we identified Site1 (47), Site2 (13). In binding site 2, in the experimental structure we find Site2 (17) and Site3 (16). In the top predicted structure (model T1058TS427_1-D1) the binding sites found are Site3 (11), Site4 (6), Site5 (1), and Site6 (1). Target T1076-D1 is bound to adenosine-5′-diphosphate (ADP). In the experimental structure we found Site1 (33), and in the top predicted structure (model T1076TS427_1-D1) Site1 (26)

The binding site analysis on the five top ranked models submitted for each target has overwhelmingly featured models submitted by AlphaFold 2 (group 427),⁴² due to their high quality predictions. In order to see the performance of other groups we analyzed the surface binding properties from the top models submitted by each of the following groups: Baker (groups 403 and 473), Feig (groups 335, 480, 013s, 334, and 314), TFold (groups 009, 488, and 368), and the DellaCorteLab (group 323). Histograms of the binding fingerprint correlations for each group are shown in Figure S3. Not surprisingly, AlphaFold 2 has the highest fraction of top models, with binding fingerprints in the homolog range (PCC > 0.5), followed by Baker, Feig, TFold, and then the DellaCorteLab. We also assessed how these groups were able to capture the ligand binding sites in the five ligand binding domain targets, see Table 2. For targets T1049-GLU, T1053-ADP, T1058-HEM^a, and T1076-ADP the models ranged from missing the site completely to predicting the site stronger than the X-ray structure. However, targets T0157-SAM and T1058-HEM^b had robust binding site prediction, with all groups having the ligand-binding site predicted with a strength similar to that of the X-ray structure.

TABLE 2.

Number of probe clusters within 3 Å of ligand for the top model of each predictor group

		T1049-D1	T1053-D1	T1057-D1	T1058-D1	T1058-D1	T1076-D1
Group name	Group number	GLU	ADP	SAM	HEMa	HEMb	ADP
X-ray	–	12	18	59	42	33	35
AlphaFold 2	427	15	41	64	60	19	44
Baker	403	19	54	50	0	21	19
	473	3	16	50	0	21	21
Feig	335	16	11	49	39	15	0
	480	12	20	41	65	17	39
	013s	35	17	31	41	31	14
	334	0	31	44	32	17	24
	314	0	0	42	8	38	17
Tfold	9	31	38	58	47	21	16
	488	16	26	34	0	21	0
	368	36	39	44	0	29	16
DellaCorteLab	323	0	4	72	34	40	78

^aHEM bound to T1058-D1 near HIS-198.

^bHEM bound to T1058-D1 near HIS-136.

3.4 |. Ligand docking to CASP14 regular targets co-crystallized with ligands

Three CASP14 targets were selected for ligand docking, T1053-ADP, T1057-SAM, and T1076-ADP. For all targets, the top model submitted by 30 different groups was considered for ligand docking with AutoDock Vina. Any correlation between GDT_TS and the best RMSD obtained for the top 5 models was observed only for one target, T1057-SAM. We note that the quality of models is not evenly distributed among the targets and the number of targets analyzed here is limited to three, therefore we do not intend to make any general conclusions with regards to CASP metrics and ligand RMSD. Rather, we discuss each target’s docking results separately, touching on the different factors that impact ligand docking in these cases.

Target T1053 is the T4SS effector protein, discussed previously, and was co-crystalized with ADP. Most top models submitted by each group have GDT_TS near 55, with the exception of group 427 whose top model had GDT_TS = 97.06. RMSD values range from approximately 3 to 7 Å RMSD for models with GDT_TS near 55, and docking of the model submitted by group 427 produced a ligand pose with 4.06 Å RMSD. Thus, GDT_TS is not the only driving factor for T1053-ADP docking success, see Figure 7A and Table S6. In fact, the top model by group 368 (GDT_TS = 53.73) had the lowest RMSD of 2.94 Å, next to the X-ray structure (RMSD = 2.20 Å). This counter-intuitive result may be explained by recalling that T1053 contains an unstructured loop in the vicinity of the binding site, reference Figure 4A,B. Because this loop (residues 19–28) is not resolved in the X-ray structure of the protein, the corresponding portion of the model does not affect the GDT_TS score. We noted previously that some poses of the loop appear to make this ADP/ATP binding site stronger, however, certain poses of this loop may also partially enclose the binding site, making the near-native docking poses highly unfavorable due to steric clashes, and ultimately resulting in poor models produced by the docking program.

FIGURE 7.

Best RMSD (Å) from the top 5 poses produced by ligand docking with AutoDock Vina of three targets: (A) T1053, (B) T1057, and (C) T1076

Target T1057 is a N4-cytosine methyltransferase from Caldicellulosiruptor bescii in complex with the ligand S-adenosylmethionine (SAM). There is some correlation between the best RMSD (among the top 5 docked cases) and the GDT_TS of the model (R² = 0.4203) (see Figure 7B and Table S7). The best model docking (RMSD 1.26 Å) is produced for the top model from group 427, with the GDT_TS = 93.9. In fact, the quality of the docked pose to this model is comparable to the quality of the pose obtained by redocking the ligand to the X-ray structure (RMSD = 1.11 Å). Thus, here it appears that GDT_TS is a sufficient predictor of the quality of ligand docking.

Finally, target T1076 encodes 2-hydroxyacyl-CoA lyase (HACL) from Rhodospirillales bacterium and was co-crystalized with ADP. Docking ADP to the highest quality model (group 427, GDT_TS = 99.07) resulted in a top 1 docking pose with 2.21 Å RMSD to native. However, similar, and even slightly higher quality (as low as 1.36 Å RMSD) docking poses were produced with lower quality models with GDT_TS near 85 (see Figure 7C and Table S8). Thus, there is no correlation between docking RMSD and GDT_TS (R² = 0.008) for T1076. In this case docking results seem independent of GDT_TS for values of 85 and greater, but without docking to lower quality models we cannot make any generalizations on what GDT quality is needed for reasonable docking performance.

In conclusion, the above results show that the relationship between the quality of protein receptor models and the success of ligand docking is very system dependent. For some systems, high-quality models of the protein receptor are essential for obtaining near-native ligand docking poses. Even with high quality models, however, docking can be sensitive to conformational changes that may be hard to adequately capture by modeling or even by with X-ray crystallography. For other systems, lower quality protein models are sufficient for successful docking runs. In fact, the success of docking heavily depend on the local conformation of the residues that determine the shape of the binding site, and these generally have moderate impact on global measures such as GDT_TS. For reference to alternate docking strategies, we have also performed ligand docking with ClusPro LigTBM, a template-based ligand docking method that uses the sequence of the protein and smiles string of the ligand to identify templates and build a protein–ligand model. ClusPro LigTBM performed well for all three cases: T1053-ADP (RMSD 1.09 Å, LDDT 0.33) and T1057-SAM (RMSD 0.75 Å, LDDT 0.56) and T1076-ADP (RMSD 1.24 Å, LDDT 0.55), outperforming AutoDock Vina docking to the X-ray structure in two cases (T1053 and T1057).

3.5 |. Protein docking of multimeric CASP14 targets

Protein docking was performed on 17 CASP14 targets classified as oligomers. We docked subunits of the CASP14 target proteins, rather than directly modeling multimeric targets, and hence our results differ from those reported by CASP14 for multimeric targets, and also from the results for the targets that were considered in CAPRI. Importantly, all CASP targets are so called “obligatory” complexes with subunits that exist only in presence of their partner proteins, and the majority of the complexes are homooligomers. Structure determination by X-ray crystallography is generally easier for obligatory complexes and homooligomers than for transient complexes that may be difficult to isolate. We note that since the X-ray structures of subunits in obligatory complexes are not determined separately, there is no need for docking. However, docking modeled subunits is an important problem if no good template for the complex is available.

To establish the potential docking quality for each case, first the X-ray subunits were docked (see Table 3). The largest subunit of the oligomer was selected as the receptor and the smaller subunit as the ligand. For large biological assemblies composed of more than four chains only the components that were observed to directly interact with the target domain were considered in the docking. The top 10 complexes produced by ClusPro were assessed for quality by the DockQ program,⁴⁰ and the best complex is presented in Table 3. Of the 17 unique X-ray docking cases, docking with ClusPro resulted in 8 high quality, 4 medium quality, 1 acceptable quality, and 4 incorrect complexes. Three of the incorrect complexes are trimers (T1052, T1070, and T1080), and the fourth complex (H1036) is a hetero-9-mer; in all four structures the chains are intertwined in the structures and rigid body docking by ClusPro cannot recreate the interaction (see Figure S4).

TABLE 3.

X-ray and model protein–protein docking of CASP14 regular targets, best DockQ scores of top 10 ClusPro models

		Docking of X-ray structures					Docking of X-ray to model structures
CASP ID	Complex	Receptor	Ligand	Rank	DockQ	Quality	Receptor	Ligand	Rank	DockQ	Quality
T1032	AB	B	A	4	0.908	High	B	TOP5_A (427)	9	0.225	Incorrect
T1034	ABCD	BCD	A	1	0.839	High	BCD	TOP2_A (427)	0	0.775	Medium
H1036s1a	ABCHL	BCHL	A	8	0.018	Incorrect	BCHL	TOP4_A (042)	9	0.023	Incorrect
T1038	AB	A	B	0	0.833	High	A	TOP2_B (427)	0	0.625	Medium
H1045s1b	AB	B	A	0	0.838	High	B	TOP1_A (050)	0	0.286	Acceptable
H1045s2b	AB	B	A	0	0.838	High	TOP5_B (427)	A	6	0.844	High
H1046s1	ABRT	BRT	A	0	0.812	High	BRT	TOP3_A (427)	0	0.686	Medium
H1046s2	ABRT	ART	B	0	0.537	Medium	ART	TOP4_B (427)	0	0.554	Medium
T1052c	ABC	BC	A	1	0.019	Incorrect	BC	TOP1_A (427)	7	0.035	Incorrect
T1054c	AB	B	A	1	0.856	High	B	TOP1_A (427)	9	0.497	Medium
H1065s1b	AC	C	A	2	0.867	High	TOP5_C (427)	A	1	0.797	Medium
H1065s2b	AC	C	A	2	0.867	High	C	TOP2_A (427)	1	0.737	Medium
T1070	XYZ	XY	Z	0	0.017	Incorrect	XY	TOP2_Z (427)	7	0.139	Incorrect
T1073	ABCD	BCD	A	4	0.891	High	BCD	TOP4_A (288)	5	0.537	Medium
T1078	AB	B	A	2	0.572	Medium	B	TOP2_A (427)	4	0.718	Medium
T1080	ABC	BC	A	0	0.060	Incorrect	BC	TOP2_A (427)	8	0.078	Incorrect
T1083	AB	B	A	0	0.723	Medium	B	TOP1_A (427)	4	0.605	Medium
T1084	AB	B	A	4	0.310	Acceptable	B	TOP1_A (427)	2	0.634	Medium
T1087	AB	B	A	2	0.650	Medium	B	TOP1_A (427)	3	0.705	Medium

^aOnly the subunits that were in contact with the ligand subunit in the crystal structure were submitted for docking.

^bHeterodimer targets were docked with the larger subunit as the receptor and smaller subunit as the ligand.

^cThe biological assembly for the target was generated using PISA.

Second, the target subunit was replaced with a model to assess the capabilities of model to X-ray docking for each target. To ensure best results, the top 5 models ranked by GDT_TS were evaluated for each target, and the best complex was recorded. There are 19 unique model to X-ray docking cases, as two X-ray cases are heterodimers and both subunits were given as CASP14 targets. All of the cases that produced acceptable or better complexes in X-ray docking also produced acceptable or better complexes in model to X-ray docking, with the exception of T1032. Docking chain A to chain B of the X-ray structure of T1032 produced a high quality complex (DockQ = 0.908), but replacing chain A with the any of the five top models, resulted in, at best, an incorrect complex with DockQ = 0.225 from the fifth ranked model (T1032TS427_3-D1, GDT_TS = 68.23). Incorrect docking for T1032 is not surprising due to the relatively poor quality of the model. In fact, for the other multimeric structures that resulted in acceptable or better complexes the models docked had GDT_TS ≥80. The quality of docking models to the rest of the X-ray structures showed some correlation with the GDT_TS (R² = 0.38) and GDT_HA (R² = 0.45) of the model (see Figure S5).

Third, both subunits were replaced with models for docking of dimer and trimer targets where all subunits were given as CASP14 targets. Rather than docking the top 5 models for each target and reporting the best solution, we considered the top model submitted by the 10 best performing predictor groups (groups 427, 403, 009, 420, 042s, 324s, 480, 129, 209s, and 473—where “s” designates a server group). This docking setup more closely follows the typical usage pattern of docking tools, where a single best available model of the subunit is used as input. We directly docked the structures to themselves for homodimers, and docked models of subunits to each other for heterodimers. The symmetry docking feature of ClusPro was used for homotrimers as the direct docking of the subunits produces an artificial dimeric state instead of the true trimeric state.²⁰ We report the interface with the highest DockQ score from the top 10 interfaces generated by ClusPro for each docking case.

Docking performance varied greatly within each target and quality scores of the best docked complex for each group are recorded in Table 4. One example is the homodimer target T1038, shown in Figure 8A. The top model submitted by group 427 has GDT_TS = 86.71 and produces a medium quality complex with DockQ = 0.539 (see Figure 8B). In contrast, the top model by group 403 has GDT_TS = 26.45 and produces an incorrect complex DockQ = 0.021 (see Figure 8C). For model to model docking, we observe a stronger correlation between model quality and the accuracy of the predicted complex, as DockQ is correlated with GDT_TS (R² = 0.55) and GDT_HA (R² = 0.60) (see Figure 8D,E). Similar to our observations in surface binding property correlations, we almost always observe acceptable or better quality docked complexes with models having GDT_TS ≥ 90. The models submitted by group 427 produced the highest quality complexes (see Figure 8F). This observation is in line with the high GDT_TS of models submitted by group 427, and our observation that the GDT metrics correlate with DockQ quality. The docking of models submitted by group 427 resulted in four medium quality and five acceptable quality complexes. Strikingly, this performance of the subunit models is comparable to that of native X-ray structures in terms of the number of acceptable or better quality docking poses. Group 403 also had good results, producing five targets with acceptable interfaces. The docking of models from the servers (042s, 324s, and 209s) resulted in acceptable or better quality docked structures only for two of the targets.

TABLE 4.

Model-to-model docking for CASP14 dimer and trimer targets, best DockQ score of top 10 ClusPro models^a

Target	X-ray	427	403	009	420	042s	324s	480	129	209s	473
T1032	0.908	0.400	0.364	0.095	0.065	0.057	0.063	0.047	0.057	0.069	0.074
T1038	0.833	0.539	0.021	0.050	0.019	0.045	0.045	0.033	0.034	0.048	0.040
T1052b	0.052	0.012	0.019	0.010	0.023	0.009	0.011	0.020	0.011	0.008	0.032
T1054	0.856	0.266	0.217	0.262	0.258	0.189	0.055	0.099	0.152	0.176	0.247
H1065c	0.867	0.710	0.291	0.428	0.320	0.539	0.516	0.554	0.394	0.277	0.146
T1070b	0.065	0.038	0.023	–	0.016	0.013	0.012	0.011	0.017	0.016	0.019
T1078	0.572	0.518	0.222	0.237	0.187	0.080	0.079	0.148	0.071	0.055	0.061
T1080b	0.507	0.448	0.057	0.043	0.059	0.037	0.023	0.033	0.032	0.034	0.049
T1083	0.723	0.361	0.369	0.179	0.205	0.126	0.194	0.250	0.185	0.213	0.175
T1084	0.436	0.323	0.451	0.355	0.396	0.329	0.248	0.403	0.283	0.359	0.371
T1087	0.650	0.505	0.460	0.142	0.151	0.146	0.147	0.040	0.126	0.049	0.042

^aThe DockQ score ranges from 0 to 1, where >0.80 indicates high quality, >0.49 indicates medium quality, >0.23 is acceptable, and <0.23 is incorrect.

^bSymmetry docking was performed on these homotrimer targets.

^cReceptor–ligand docking was performed on these heterodimer targets—the receptor was designated as the larger subunit.

FIGURE 8.

Model-to-model docking results for the top submitted models. (A) The crystal structure for target T1038. (B) The best X-ray to model docked conformation obtained for T1038 using the top 2 model (Group = 427, DockQ = 0.539, GDT_TS = 86.71, and GDT_HA = 69.74). (C) The X-ray to model docking for T1038 using the top model by group 403 (DockQ = 0.021, GDT_TS = 26.45, and GDT_HA = 17.89). (D) DockQ scores for group model to model docking versus GDT_TS, and (E) GDT_HA. Heterodimers are excluded from the plots DockQ versus GDT plots. (F) Group performance on frequency of targets with docked complexes of acceptable, medium, and high quality. Server groups are designated with an “s”

The docking of models from almost all groups performed well for the heterodimeric target H1065, with nine of the 10 groups considered producing at least an acceptable quality interface, see Table 4. This is a rather impressive result, since here we use “classical free docking” of the subunits modeled separately. It is even more impressive that there was no template available for this complex, and the modeling of individual subunits (T1065s1 and T1065s2) were classified as TBM/Hard and TBM/FM, respectively, which underscores the significance of this success.

4 |. DISCUSSION

We have focused this evaluation of CASP14 models on properties that are relevant to molecular interactions and to drug discovery, including how the models compare to their respective X-ray structures in terms of the overall binding surface, specific ligand-binding site detection, ligand docking, and protein docking. Previously we performed a similar evaluation of CASP12 targets, where top models covered a range of GDT_TS values, with an average of 61.31 among regular targets.¹⁵ However, because the quality of models submitted to CASP14 greatly surpassed the quality of previous CASP models, with an average GDT_TS of 87.72 among regular targets, this study provides an exciting analysis of how these higher quality structures would perform in the same drug discovery applications.

Analysis of surface binding property conservation among homologs revealed that binding fingerprint PCC values >0.50 represent good conservation of surface properties.¹⁵ In CASP14, the overwhelming majority of target domains (88 of 96) had at least one model submitted with binding fingerprint PCC > 0.50. Due to the high GDT values for the top 5 ranked targets the correlation between binding fingerprint PCC and GDT_TS is low (R² = 0.11); however, it is clear that apart from a couple exceptions, models with GDT_TS ≥ 90 are almost guaranteed to have good conservation of surface binding properties. This is an important observation as one potential use of structural models is the discovery of druggable targets, where assessment of surface properties and identification of suitable binding pockets is of high interest. We had eight targets in CASP14 with known ligand binding sites, and the location of these site was robustly detected in most models ranked in the top 5 by GDT_TS (Table 1). Ligand docking of three of these sites revealed that while overall structural quality, such as GDT_TS, can be an important indicator of ligand pose quality (i.e., T1057-SAM), other factors, such as the orientation of side chains in the binding pocket also play a critical role (i.e., T1053-ADP). Thus, good conservation of global binding properties as described in terms of binding fingerprint PCC is a necessary but not sufficient condition for successful ligand docking, as it may also depend on the local environment.

Docking of protein subunit models for “obligatory” complexes shows that the quality of subunit models in terms of GDT_TS is positively correlated (R² = 0.55) with the quality of docking poses, as measured by the DockQ score achieved when using this subunit as docking input. In practical terms, docking of CASP14 top models can result in good complex models. In fact, when docking subunit models from the top CASP14 group (group 427 from AlphaFold 2), the number of acceptable or better quality models is the same (9 out of 11 total targets) as when docking bound X-ray structures. The results for other top-performing groups are also quite good, with 5, 4, and 3 targets being reconstructed using models from groups 403, 009, and 009, respectively. Notably, the server models from groups 042s and 324s resulted into two correctly reconstructed complexes. Overall, this is a very impressive result, demonstrating major progress for modeling “obligatory” complexes straight from sequence. This also implies that machine learning methods used by the top groups are able to implicitly learn the protein environment, which substantially improves the docking of “obligatory” complexes. It would be interesting to see application of such high quality models for the docking of “transient” complexes.

5 |. CONCLUSIONS

The analysis of CASP14 results shows that the quality of individual domain modeling for the first time is approaching that offered by X-ray crystallography. As a result, state-of-the-art models can be successfully used for identification of binding and regulatory sites, as well as for docking to assemble obligatory complexes. It is likely that the major improvements in the accuracy of subunit models, once such will be available, will substantially improve protein-protein docking results as measured by CAPRI, an ongoing docking experiment. In contrast, the success of ligand docking, however, often depends on fine details of the transient binding interface, and thus may require further development of methodology, for example, simulation methods.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are openly available in the CASP website at https://predictioncenter.org/.