Multi-Level Analysis of Intrinsically Disordered Protein Docking Methods

Abstract

Intrinsically Disordered Proteins (IDPs) are a class of proteins in which at least some region of the protein does not possess any stable structure in solution in the physiological condition but may adopt an ordered structure upon binding to a globular receptor. These IDP-receptor complexes are thus subject to protein complex modeling in which computational techniques are applied to accurately reproduce the IDP ligand-receptor interactions. This often exists in the form of protein docking, in which the 3D structures of both the subunits are known, but the position of the ligand relative to the receptor is not. Here, we evaluate the performance of three IDP-receptor modeling tools with metrics that characterize the IDP-receptor interface at various resolutions. We show that all three methods are able to properly identify the general binding site, as identified by lower resolution metrics, but begin to struggle with higher resolution metrics that capture biophysical interactions.

Introduction

Intrinsically Disordered Proteins (IDPs) are a class of proteins in which at least some region of the protein does not possess any stable structure in solution in their physiological condition. Current predictions show that 33.0% of all proteins in the human proteome contain at least some residues in disordered conformations [1]. IDPs are prominent in liquid-liquid phase separation [2], [3], cell signaling and regulation [4], such as with the P53 Tumor Suppressor [5] , as well as other cellular functionality. This functionality is largely mediated through interactions that the disordered regions make with their receptor molecules, which may take a wide variety of forms.

Through both experimental [3] and computational [6] means, certain biophysical interactions have been shown to be prominent in the binding of IDPs to receptors. These receptors may be other IDPs, nucleic acids, or globular proteins. It is fairly common for these interactions to be non-specific or “fuzzy” to where, although the IDP may be interacting with specific regions on the receptor, a singular conformation between the IDP region and receptor does not exist [7]. Molecular Dynamics (MD) simulations of these interactions have shown that specific forms and designs are commonly responsible for these interactions [6]. So-called “patterning” of charged residues sequences of IDPs [8], [9] has also been attributed as a contributing factor in phase separation.

Despite frequent fuzziness, it is possible for IDPs to adopt a sufficiently stable conformation upon binding to a receptor, to the point that their complex structures may be experimentally determined by X-ray crystallography, Nuclear Magnetic Resonance (NMR), or Cryo-Electron Microscopy (Cryo-EM) [5], [10]. This brings IDP-receptor modeling into the domains of computational protein structure modeling and docking.

Due to their prevalence in therapeutic and cellular functionality, accurate modeling of IDP-receptor like complexes has been long sought after in the Protein docking community. Protein docking consists of finding a 3D complex structure from the known 3D structures of two subunit molecules, termed the ligand and receptor. The primary goal of this process is correct placement of the ligand subunit on the surface of the receptor subunit, while adhering to the chemical and biophysical constraints of protein structures. Protein docking is typically evaluated through the Critical Assessment of the Prediction of Interactions (CAPRI) [11] community, which has conducted over 50 rounds of assessment [12]. Due to their prevalence in cellular signaling and disease, highly accurate prediction of IDP-receptor complexes will thus largely aid in therapeutic design.

The disordered nature of IDPs alters the protein docking problem. Instead of placing a 3D structure of a ligand subunit onto a globular receptor surface, an unstructured sequence of an IDP (or region thereof) must have its structure modeled on the surface of the receptor. Previous software packages have attempted to model these IDP-receptor interactions [13], [14], and can often provide a low-precision view of potential ligand placement on the receptor where the general binding site may be identified (often in grooves or crevices in the protein surface). However, residue and atomic level contacts are largely ignored in the context of global IDP docking. The advent of AlphaFold [15], along with its variant AlphaFold Multimer [16], have propelled the field of protein structure prediction, as well as protein complex prediction, to a level of never before seen accuracy and consistency at an atomic level. Previous studies have shown that AlphaFold is capable of modeling peptide-protein complexes [17], [18]. It is unclear how well AlphaFold is able to handle fuzzy interactions between IDPs and globular receptors.

In this study, we investigate the abilities of docking and modeling of the disordered regions of IDPs, along with other sufficiently long peptides, to their respective receptors at various levels of meticulousness. We specifically test our own IDP-LZerD docking software, the coarse-grained CABS-Dock docking software, and the AlphaFold-Multimer docking software. We start with more low-resolution metrics that are primarily concerned with the ability to identify and dock the ligands to the proper binding side on the receptor. Low resolution metrics should not be immediately disregarded, as “fuzzy” interactions, in which the IDP retains some degree of disorder while still specifically binding to a region on the receptor, may still be identified by them. From here, we explore more moderate resolution metrics that focus on the orientation of the flexible ligand in the receptor, as well as proximity of residues between the ligand and the receptor. Finally, we explore the ability to model high-resolution interactions between the ligand and the receptor. These high-quality metrics are derived directly from the biophysical interactions that drive IDP binding in reality. All methods performed acceptably well with identifying binding sites on the receptor surface for many cases. However, as the resolution of the metrics increased, AlphaFold was able to outperform the other two methods, particularly in the residue and high-resolution metrics.

Materials and Methods

IDP Dataset

To create a dataset of IDP-receptor complexes to test modeling ability, the PEPtide Binding Database (pepBDB) [19] was clustered by any available sequences via a pre-computed MMseqs2 [20] from the Protein Data Bank (PDB). Structures that only contain protein chains, have an experimental resolution <=3.0 Å or are NMR structures, have a minimum receptor chain length of 100 residues, have a minimum IDP chain length of 20 residues, and only have a single ligand chain binding to the receptor were included. From this set, we cross-referenced the PDB-IDs against structures listed in the DisProt database [21]. We removed a case (PDB-ID: 1S6C) where the peptide chain was covalently bound to the receptor via a peptide bond, and another case (PDB-ID: 5KY7) where the ligand was determined to be globular, despite being included in the PEP-BDB. Two more entries were also removed (PDB-IDs: 3HPW and 5FMK) because models of these targets were unable to be refined through CHARMM [22], which is a part of the modeling procedure with IDP-LZerD. The resultant test set contains a total of 35 entries, with details in Supplementary Table 1.

Modeling Methods

We used the following three modeling methods in this study. Here we briefly describe their algorithms.

IDP-LZerD

IDP-LZerD [14], [23], which was specifically designed to handle longer IDP Sequences, works by first generating overlapping 9 residue fragments of the IDP using the Rosetta Fragment Picker [24], in which the predicted secondary structure is considered. Here, we consider secondary structure predictions from PSIPRED 4.0 [25], DeepCNF [26], DNSS [27], and SPOT1D [28]. Using the LZerD docking program [29], [30], several predicted conformations of each of these fragments are docked onto the receptor structure. These docked fragments are then stitched together into the full IDP using the GOAP [31], D-FIRE [32] and IT-SCORE [33] scoring functions, as well as the complementarity of overlapping regions between full fragments, producing up to 100 full length IDP-receptor structures. A restrained relaxation via CHARMM [22] is then used to clear any poorly modeled geometry. A custom scoring function that considers GOAP, D-FIRE, and IT-Score, as well as a molecular-mechanics scoring function commonly used with LZerD applications [34] is again used to rank the refined models. IDP-LZerD is available at https://kiharalab.org/proteindocking/idplzerd.php as a part of the LZerD protein docking suite.

CABS-Dock

CABS-Dock [13] is a coarse-grained peptide-protein docking program that has previously been used for docking of IDP fragments [14], as it allows for large flexibility of the ligand chain. CABS is a simplified representation of proteins in which each residue is modeled only as up to four beads [35]. CABS-Dock initially selects 1000 frames from 10 separate trajectories. These 10,000 frames are then filtered down to 1000 separate models, which are then subsequently clustered into 10 final models. These 10 models are then reconstructed to an all-atom representation via MODELLER [36]. We ran the stand-alone CABS-Dock program downloaded from https://bitbucket.org/lcbio/cabsdock/downloads/.

AlphaFold-Multimer

AlphaFold-Multimer [16] predicts protein complexes from amino acid sequences of component proteins. The algorithm is based on AlphaFold [15], which predicts single chain protein structures, with modifications made for handling multiple input proteins as input and for the scoring function to make accurate protein interactions. AlphaFold-Multimer first makes a Multiple Sequence Alignment (MSA) for each input protein sequence and concatenates them into a single MSA. The MSA is input to deep neural networks, which embed pairwise residue features and builds structure models of the protein complex. By default, five models will be constructed. The models are ranked by a model confidence score. Each target takes approximately 1.5 hours for full pipeline inference on a Titan RTX GPU with 24GB memory, dependent on the protein size. We used the AlphaFold-Multimer code that was downloaded from the Github repository, https://github.com/deepmind/alphafold, on February 21^st 2022. AlphaFold-Multimer needs two databases, a protein sequence database from which similar sequences to the input sequences are selected and aligned and a protein structure template database which includes known protein structures which will be used as templates in the modeling. For a protein sequence database, we used Uniref90 v.2021_03 [37], BFD, Uniclust30 v.2018_08 [38], and MGnify v.2018_12 [39]. All databases were downloaded on July 22^nd 2021. We used the same template database as used in the original paper [16]. For template search, we used structures from the Protein Data Bank (PDB), which are dated on 2022/02/01 or earlier. The default parameters of the program were used. Below, we refer to AlphaFold-Multimer as AlphaFold for simplicity.

Evaluation Metrics

We present a variety of evaluation metrics that span various levels of precision. We start with low-resolution metrics that only concern which residues of the interface are interacting with the IDP. These methods do not relate to any specific contacting pairs, nor the orientation of the IDP. We then investigate intermediate-resolution metrics, which penalize the incorrect orientation of the IDP, as well as receptor-ligand contacting residue pairs. Lastly, we evaluated high-level metrics, derived from the biophysical interactions that drive IDP Docking.

Interface Precision and Recall

The interface precision and recall are a group of metrics used to gauge the ability of docking methods to identify the binding interface on the receptor. We describe four related metrics that measure similarity between the interfaces of the true and modeled complexes. Receptor Interface Precision and Receptor Interface Recall measure the number of the true receptor interface residues that exist at the model complex interface as a proportion to either the total number of predicted receptor interface residues or true receptor interface residues. A receptor residue is considered an interface residue if it is within 5 Å of any ligand residues.

$$ReceptorPrecision=\frac{\left(\mathit{True}\mathit{Receptor}\mathit{Interface}\mathit{Resdiues}\right)\cap \left(\mathit{Predicted}\mathit{Receptor}\mathit{Interface}\mathit{Residues}\right)}{\left(\mathit{Predicted}\mathit{Receptor}\mathit{Interface}\mathit{Residues}\right)}$$ (1)

$$\mathit{Receptor}\mathit{Recall}=\frac{\left(\mathit{True}\mathit{Receptor}\mathit{Interface}\mathit{Resdiues}\right)\cap \left(\mathit{Predicted}\mathit{Receptor}\mathit{Interface}\mathit{Residues}\right)}{\left(\mathit{True}\mathit{Receptor}\mathit{Interface}\mathit{Residues}\right)}$$ (2)

Figure 1A illustrates these two metrics.

Figure 1:

Diagrams depicting the interface residue metrics. A, Visualization of interfaces used in the Receptor Precision and Receptor Recall metrics. B, Visualization of the ligand and receptor residues used in the Ligand Precision metric. C, Visualization of the ligand and receptor residues used in the Ligand Recall metric.

We also defined recall and precision on the ligand (IDP) side. Since most of ligand residues in a model interact with the receptor, we defined Ligand Precision and Recall considering if ligand residues interact with predicted or true interface residues of the receptor. Ligand Precision quantifies the proportion of the predicted ligand interface residues that are in interaction with any of the true receptor interface residues (Figure 1B). On the other hand, Ligand Recall quantifies the fraction of true ligand interface residues that are in contact with predicted receptor interface residues. Ligand residues are considered to be at interface if they are within 5 Å of any of the receptor interface residues. Particularly, ligand residues are considered as:

$$\mathit{Ligand}\mathit{Precision}=\frac{(\mathit{Predicted}\mathit{Ligand}\mathit{Interface}\mathit{Residues}\mathit{in}\mathit{Contact}\mathit{with}\mathit{True}\mathit{Receptor}\mathit{Interface}\mathit{Residues})}{(\mathit{Predicted}\mathit{Ligand}\mathit{Interface}\mathit{Residues})}$$ (3)

$$\mathit{Ligand}\mathit{Recall}=\frac{(\mathit{True}\mathit{Ligand}\mathit{Interface}\mathit{Residues}\mathit{in}\mathit{Contact}\mathit{with}\mathit{Predicted}\mathit{Receptor}\mathit{Interface}\mathit{Residues})}{(\mathit{True}\mathit{Ligand}\mathit{Interface}\mathit{Residues})}$$ (4)

Intermediate-Resolution Metrics

Ligand-RMSD (LRMSD)

The Ligand RMSD (LRMSD) is a CAPRI metric that measures the Cα RMSD of the ligand chain (in this case the docked IDP) when the receptor chains are superimposed [11].

F_nat

The Fraction of Native Contacts (F_nat) is a metric used to assess docking performance in CAPRI [11]. The metric measures the number of native contacts found in the computational model, divided by the total number of contacts. Here, a contact is defined as a ligand-receptor residue pair within 5 Å of each other.

High-Resolution Metrics

F_hbond

The criteria for defining a hydrogen bond via atom coordinates were derived from constraints described previously [40]. This definition considers both the distance and angle of a potential hydrogen bond, and thus requires the coordinates of the donor and acceptor heavy atoms, as well as the hydrogen atom. Hydrogens were added to target and model PDB files with PHENIX reduce [41].

From all modeled hydrogen bonds, we created an F_hbond metric, which follows the same functional form of the F_nat metric described above but considers pairs of intermolecular hydrogen bonding residues instead of contacts.

$$F_{\mathit{hbond}}=\frac{\mathit{Hydrogen}\mathit{Bonding}\mathit{Pairs}\mathit{Correctly}\mathit{Modeled}}{\mathit{Total}\mathit{Hydrogen}\mathit{Bonding}\mathit{Pairs}}$$ (5)

F_hphobe

The criteria for defining a hydrophobic interactions via atom coordinates were derived from constraints described previously [40], with definitions of hydrophobes being simplified to include only elements found naturally in proteins. This simple definition only considers distances between two hydrophobic atoms.

$$F_{\mathit{hphobe}}=\frac{\mathit{Hydrophobic}\mathit{Pairs}\mathit{Correctly}\mathit{Modeled}}{\mathit{Total}\mathit{Hydrophobic}\mathit{Pairs}}$$ (6)

F_ionic

The criteria for defining short range ionic interactions via atom coordinates were derived from constraints described previously [40], with definitions of cations and anions modified to fit proteins, and will thus only identify salt bridges. This definition only considers distances between the cation and anion.

$$F_{\mathit{ionic}}=\frac{\mathit{Ionic}\mathit{Pairs}\mathit{Correctly}\mathit{Modeled}}{\mathit{Total}\mathit{Ionic}\mathit{Pairs}}$$ (7)

Results

Global Positioning of an IDP Ligand on the Receptor Surface

First, we gauged the ability of the methods to identify the correct global location of the IDP-receptor interface (Figure 2). The modeled ligand at the interface can be evaluated between either the true and predicted receptor surfaces (Figure 2A–B), or the ligand residues (Figure 2C–D), as well as with respect to the predicted interface or the true interface, resulting in 4 possible precision and recall metrics. The top ranked, best within the top 5, and best of all models of IDP-LZerD were able to successfully model over 70% of the true interface for 13 (37.14%), 22 (62.86%), and 27 (77.14%) models, respectively (Figure 2A). These numbers drop to 8 (22.86%), 16 (45.71%), and 17 (48.57%) for top ranked, best within the top 5, and best of all models for CABS-Dock, and increased to 31 (88.57%) for both the top ranked and best within the top 5 for AlphaFold. Comparing Figure 2A and 2B, the precision was generally lower than the recall in Figure 2B for models by IDP-LZerD and CABS-Dock. A typical reason for this is that these two methods tend to bind all residues in the ligand at the receptor surface, making a larger interface region than the true one.

Figure 2:

Recall and precision distributions for modeled complexes. Swarm plots of recall and precision distributions of the top ranked model (blue), best of top 5 ranked models (orange), and best of all models (green) for each method. Each dot represents the result from a target complex in the IDP dataset. A, the recall of receptor residues in modeled complexes. B, the precision of receptor residues in the modeled complexes. C, the recall of the true ligand (IDP) residues on the predicted receptor surface. D, the precision of modeled ligand residues on the true receptor surface. For AlphaFold only results for the top ranked and best of the top 5 ranked models were shown because it only outputs five models in the default setting.

Evaluation from the ligand shows a similar trend (Figure 2C–D). Using the same 70% cutoff for the Ligand Precision (that is to say that 70% of the contacting ligand residues are contacting at the correct interface), the precision of the top ranked, best within the top 5, and best of all models for IDP-LZerD are 13 (37.14%), 25 (71.43%), and 31 (88.57%), respectively. These numbers drop to 11 (31.43%), 23 (65.71%), and 27 (77.14%) for top ranked, best within the top 5, and best of all models for CABS-Dock, and increase slightly to 34 (97.14%) and 35 (100.00%) for top ranked and best within the top 5 for AlphaFold.

We visually inspected four targets in Figure 3. In Figure 3A, (PDB ID: 1WLP), the Receptor Recall was well captured by all three methods (0.93, 0.89 and 0.93 for IDP-LZerD, CABS-Dock, and AlphaFold, respectively). The target with the longest IDP (49 residues) in the dataset, 1JPW, had an elongated IDP structure which all three methods were able to generally follow (Figure 3B). Within the resolved regions, it resulted in Receptor Recall values of 0.54, 0.46, and 0.98 for IDP-LZerD, CABS-Dock, and AlphaFold, respectively. It was noted that for the best ranking models of target 2PHG (Figure 3C), both IDP-LZerD and CABS-Dock were able to generally trace the ligand, which does not leave the receptor surface. However, AlphaFold begins to follow the same general path as IDP-LZerD did, but veers away from the receptor surface midway. This caused the Receptor Recall values for IDP-LZerD and CABS-Dock to be 0.73 and 0.77 respectively, and the Receptor Recall for AlphaFold to be 0.50, despite exceptional modeling in the contacting region. Lastly, target 3PK1 (Figure 3D), shows that both IDP-LZerD and CABS-Dock have Receptor Recall values (0.84 and 0.96, respectively) despite the CABS-Dock model being modeled in backwards. In contrast, AlphaFold modeled this target very well, resulting in a Receptor Recall value of 1.00.

Figure 3:

Modeling examples. The receptor and target IDP are colored green and translucent red respectively. IDP-LZerD, CABS, and AlphaFold models are colored blue, orange, and pink, respectively. The best in Top 5 by Receptor Recall are shown for A. The best in Top 5 by LRMSD of each method is shown in B, C, and D. A, Strong performance by Receptor Recall for target 1WLP (Cytochrome b-245 light chain and Neutrophil cytosol factor 1) by all three methods. The length of the IDP is 25 residues. B, Docking a long IDP of 49 residues for target 1JPW (Transcription factor 7-like 2 and Beta-catenin). In this target, the IDP structure was only resolved for 24 residues in the IDP (residues 13 to 25; and 40 to 50), which are shown as two fragments in red. C, Mis-modeling of target 2PHG (Alpha trans-inducing protein and Transcription initiation factor IIB) by AlphaFold, better modeled by CABS-Dock and IDP-LZerD. The length of the IDP is 26 residues. D, High performance of AlphaFold on target 3PK1 (Apoptosis regulator BAX and Induced myeloid leukemia cell differentiation protein Mcl-1). The length of the IDP is 34 residues.

Intermediate-Resolution Accuracy

Next, we evaluated the models in terms of LRMSD and F_nat. These two metrics concern the ability of the methods to model the IDP ligand in the correct overall orientation and binding site (Figure 4A). Out of all resultant models, AlphaFold was able to consistently identify the binding site of the ligand, and placed the ligands of all 5 models for any given target in the same general location on the receptor. AlphaFold models have 31 targets where all models have an LRMSD less than 10.0 Å, and 25 targets where all models have an LRMSD less than 5.0 Å. The average standard deviation of AlphaFold across all model LRMSDs is 0.59 Å.

Figure 4:

Intermediate Resolution Modeling Methods. A, Swarm plots of LRMSD distribution of the top ranked model, best of top 5 ranked models, and best of all models by LRMSD. B, Swarm plots of F_nat distribution of the top ranked model, best of top 5 ranked models, and best of all models by F_nat. C, Correlation between Receptor Recall and LRMSD of the best of top 5 models by Receptor Recall for all methods.

With CABS-Dock and IDP-LZerD, binding locations of the IDP ligand were sometimes localized or clustered in grooves on the receptor surface, but were generally more diverse in the placement of the IDP on the receptor surface compared to AlphaFold. As a result, CABS-Dock and IDP-LZerD have average LRMSD standard deviations of 5.67 Å and 5.98 Å within the top 5 ranked models respectively. In Figure 4C, we see that models that have a high interface residue accuracy often have a large LRMSD, which correspond to the models that have different binding poses of IDPs in the correct binding sites.

This tendency to place the ligand in grooves and concave places in the receptor was particularly problematic for IDP-LZerD on target 1IK9, in which IDP-LZerD placed all models in ridges near the globular domain, instead of on the coiled-coil stem where it was crystalized (Figure 5A). This resulted in a minimum LRMSD of 69.64 Å for this model. CABS-Dock had a similar problem with this target, but was able to place a single model near the binding site (albeit backwards, and distorting the receptor), resulting in a minimum LRMSD of 28.13 Å. AlphaFold was able to model this structure with a minimum LRMSD of 6.03 Å. It was noticed that in several cases where AlphaFold also reported a poor LRMSD, the ligand was placed in the correct receptor site, but backwards (Figure 5B). This has been noted previously with AlphaFold [18], as well as other IDP docking studies [23]. This is by no means specific to AlphaFold, as CABS-Dock also docked the ligand backwards in target 3PK1 (Figure 3D). This caused the LRMSD of the CABS-Dock model to be significantly higher than the IDP-LZerD model, which placed the ligand in the correct direction (20.10 Å vs 6.98 Å, respectively) despite having similar Receptor Precision and Recall.

Figure 5:

Examples of intermediate resolution modeling. A, IDP-LZerD models of 1IK9 (DNA ligase IV and DNA repair protein xrcc4). The receptor is colored gray, the 100 modeled ligands magenta, and the true ligand red. B, the top-ranked AlphaFold model of target 2B3G (Cellular tumor antigen p53 and Replication protein A 70 kDa DNA-binding subunit), in which the IDP was modeled backwards. Receptor coloring is in gray, and true and modeled IDPs have a blue-red spectrum from N-terminus to C-terminus. This model has a Receptor Recall of 0.57, but an LRMSD of 23.12 Å. The true model is modeled from left to right. C, the best in top 5 models for target 1JSP (Tumor protein p53 and CREB-binding protein). The receptor and target IDP are colored green and translucent red, respectively. IDP-LZerD, CABS-Dock, and AlphaFold models are colored blue, orange, and pink, respectively. These models have Receptor Recall values of 0.11, 0.11 and 0.78 respectively, and LRMSD values of 34.18, 33.80 and 19.01 respectively.

We further examined the F_nat across the best and top ranked models for each method, to determine if the methods are able to place residues within the proper vicinity on the receptor surface (Figure 4B). AlphaFold managed to maintain the highest average F_nat of 0.78 across all best within top 5 models, with 30 models recording an F_nat of above 0.5. CABS-Dock scored the lowest of the three methods with an average F_nat of 0.13, and 0 models with an F_nat above 0.5. IDP-LZerD had an average F_nat of 0.23 across all best within top 5 models, with 5 models recording an F_nat of above 0.5.

AlphaFold did score 5 of its top models below an F_nat of 0.5. As expected, these are largely the same models in which the LRMSD was relatively large, often due to the flipping of the ligand within the correct binding site. IDP-LZerD did score relatively high F_nat values for several targets (also targets where the LRMSD was high). However, it was typical for the ranking of generated models, particularly for IDP-LZerD and CABS-Dock, to have sub-optimal ranking of the models. Thus, the quality generally increases from the top ranked model, best in the top, and best of all models (Figure 4B). For IDP-LZerD, the average F_nat values increased from 0.12, 0.23, and 0.33 for the top ranked, best in the top 5, and best of all models, respectively. For CABS-Dock, the average F_nat values increased from 0.05, 0.13, and 0.14 for the top ranked, best in the top 5, and best of all models, respectively.

Several of the models available in the PepBDB are derived from NMR targets or have unresolved regions in which the IDP is not completely contacting the receptor. This is a recognized weakness in the methodology of IDP-LZerD and CABS, since these two methods tend to bind all the residues in an IDP to the receptor surface by their algorithm design. This is the case with target 1JSP, in which only the N-terminal region is making contact with the receptor, while the C-terminal region is not making any contacts (Figure 5C). Both the best CABS-Dock and IDP-LZerD models attempted to keep the entire fragment in contact with the receptor (resulting in LRMSDs of 34.18 Å and 33.79 Å for the best in top 5 respectively), whereas the AlphaFold model was able to leave certain portions of the ligand out of contact with the receptor. While AlphaFold’s LRMSD was still lower than average (19.01 Å), the models were still able to keep some proper contacts with the receptor (F_nat: 0.29).

Atomic Level Interactions

Due to the remarkable performance of AlphaFold in the traditional protein docking metrics as discussed above, we decided to re-evaluate the modeling performance using more stringent, biophysical interaction-based metrics only for AlphaFold models. These are derived from geometric constraints for hydrogen bonds, hydrophobic contacts, and short-range ionic interactions. We compared the ability of AlphaFold to model these interactions (in the form of fractions of true interactions) against the F_nat values of the same target (Figure 6A). Despite the overwhelmingly high quality F_nat values of AlphaFold in Figure 4, the fractions of individual interactions often fall short of their relative F_nat values, even for more prominent interaction types, such as hydrogen bonds. Top ranked and best AlphaFold Interactions have average F_nat values of 0.76 and 0.78 respectively, but have average F_hbond values of 0.59 and 0.62, F_hphobe values of 0.72 and 0.73, and F_ionic values of 0.75 and 0.77. The discrepancy between F_nat values and the interaction metrics is most apparent in the F_hbond values. In the plot in Figure 6A, we observe many cases where F_hbond is approximately half of F_nat.

Figure 6:

Interaction Fractions of AlphaFold Models. A, scatterplots of F_nat against the fractions of the three interaction types, F_hbond, F_hyrdophobic, and F_ionic. The top ranked AlphaFold Model (orange), as well as the best of all 5 models (blue) are shown. B, A missed hydrogen bond in the top ranked AlphaFold model of 3JUA (Yes-associated protein and TEF-3). Cyan and deepblue are the ligand chains of the ground truth and AlphaFold model, respectively, and lime and dark green are the receptor chains of the ground truth and AlphaFold model, respectively. C, AlphaFold model of 5OHG (Ribonuclease E and Enolase). Pink and magenta are the ligand chains of the ground truth and AlphaFold model respectively, and Cyan and deep blue are the receptor chains of the ground truth and AlphaFold model respectively. D, General sidechain mismodeling in target 5FCG (Protein X and Apoptosis regulator Bcl-2). Cyan and deepblue are the ligand chains of the ground truth and AlphaFold model, respectively, and lime and dark green are the receptor chains of the ground truth and AlphaFold model, respectively.

We visualize how AlphaFold sometimes misses making potential interactions in Figure 6. In Figure 6B, we see that in target 3JUA (F_nat: 0.88, F_hbond: 0.87) a missed hydrogen bond is purely due to sidechain rotamer modeling, in which the backbone of the receptor is well aligned, but the sidechains of both the donor and acceptor are rotated away from each other. In another example (Figure 6C), 5OHG (F_nat:0.87, F_ionic:0.50) a missed ionic interaction is caused by the terminal region of the ligand chain to continue forward in an alpha helix, pushing the cation away from the anion. In the last example in Figure 6D, we observed a large discrepancy between F_nat and values of the three atomic level metrics. This target (5FCG) manages to maintain an F_nat value of 0.63, while the F_hbond, F_hphobe, and F_ionic values are 0.17, 0.60, and 0.33, respectively. Many of the sidechains in this model have rotamers that place sidechains in wrong orientations with respect to the receptor. There errors are small enough to maintain a relatively large F_nat but fail to meet the criteria for the more stringent interaction based metrics.

Discussion

We evaluated models from three different modeling software packages for their ability to accurately model IDP-receptor interactions known from experimental crystal structures. We investigated modeling with metrics of various resolutions in ascending order: the Interface Precision and Recall, LRMSD, F_nat, and high-resolution fractions of interactions (F_hbond, , F_hphobe, , and F_ionic) for intermolecular hydrogen bonds, hydrophobic contacts, and ionic interactions. We evaluated the top ranked, best within the top 5, and best of all generated models for each method.

In general, IDP-LZerD, CABS-Dock, and AlphaFold were all able to perform acceptably with low resolution metrics such as the fractions at the interface. However, as the resolution of the analysis metrics increases, AlphaFold does manage to outperform IDP-LZerD and CABS-Dock, with the difference in performance being particularly noticeable in the F_nat metric. Despite the tremendous leap forward AlphaFold has made to protein structure and complex prediction, there is still progress to be made on the atomic-interaction level at the interface of IDP-receptor complexes, as the fraction of interactions are often below the F_nat values. Particularly, F_hbond are often substantially lower than F_nat values. Future AlphaFold derivative modeling methods may aim to take advantage of this. By explicitly considering these types of biophysical interactions, one may be able to better model these interactions, but also may aid in other modeling errors of AlphaFold, such as flipping of the IDP in the receptor site.

It should be kept in mind that many known IDP-receptor complexes are fuzzy in nature and deviate significantly on the receptor surface. IDP-LZerD and CABS both exhibit significant conformational deviations in their generated models, yet are often able to place their models in the general vicinity of the binding site, perhaps making their modeling techniques better suited for these types of interactions. Although it is clear that AlphaFold maintains a strong performance lead with the well-ordered crystal structures tested here, the fuzzy nature of IDPs may still prove traditional physics-based methods such as IDP-LZerD and CABS-Dock useful. Future approaches in IDP-receptor modeling, particularly with those that exhibit dynamic interactions, are likely to include hybrid approaches between machine learning and traditional physics and MD based methods.