AlphaFold Illuminates Half of the Dark Human Proteins

Abstract

We investigate the use of confidence scores to evaluate the accuracy of a given AlphaFold (AF2) protein model for drug discovery. Prediction of accuracy is improved by not considering confidence scores below 80 due to the effects of disorder. On a set of recent crystal structures, 95% are likely to have accurate folds. Conformational discordance in the training set has a much more significant effect on accuracy than sequence divergence. We propose criteria for models and residues that are possibly useful for virtual screening. Based on these criteria, AF2 provides models for half of understudied (dark) human proteins and two-thirds of residues in those models.

INTRODUCTION

About half of Americans answering a 2020 survey would not get in an AI-driven taxi, and about three-quarters of them believed AI (artificial intelligence) cars were “not ready for primetime” [1]. Whether driving a vehicle or discovering new medicines, trust in AI depends on accumulated community experience and the consequences of errors in specific cases. There are over 20,000 protein-coding genes in the human genome [2-4]. Of these, 7074 have experimentally determined structures deposited in the Protein Data Bank (PDB) as of July 2021 [5]. Only 670 human proteins are therapeutically targeted by medicines, comprising the "drugged genome" [6]. Significant areas of biology remain potentially amenable to drug discovery [7]. Initiatives like “Illuminating the Druggable Genome” [8], OpenTargets [9], and Target 2035 [10] are exploring novel therapeutic opportunities in the “druggable” genome.

DeepMind described [11,12] AF2 (AlphaFold version 2.0), an AI method that predicts overall 3D structures of proteins. More than 350,000 AF2 structural models (including models of nearly every human protein) are now publicly accessible [12]. DeepMind garnered worldwide attention with their decisive win of the Critical Assessment of Techniques for Protein Structure Prediction, CASP14 [13]. Currently, scientists are assessing the impact of AF2 on research, including how much AF2 models expand the druggable genome.

Winning CASP14 presents a set of challenges specific to protein folding. However, protein 3D models do not often play a crucial role in drug discovery. The notion of trust in AF2 models is illustrated with a histogram of atomic Root-Mean-Square Deviations (aRMSD) on C_α atoms for crystal structures deposited in the PDB since AF2 was trained (Fig. 2a in [11]). It shows that AF2 produces high-quality folds in two-thirds of cases. However, the overall accuracy of a given AF2 model was not discussed. Local confidence scores (predicted Local Distance Difference Test, pLDDT) show a 95% per-residue correlation with experimentally-derived LDDT values [14] over the same proteins. AF2 model confidence evaluation is needed in the drug discovery context, given the non-local nature of aRMSD, the inherent selection bias of recent PDB structures, and the lack of any overall confidence-in-accuracy measure that can be calculated for individual models. Here, we discuss the issue of trust in AF2 models by addressing disorder, divergence, discordance, and druggability.

Disorder dominates confidence scores below 80

More than 30% of eukaryotic proteins contain one or more intrinsically disordered regions, IDRs [15-21]. Disorder is reflected in confidence scores as regions with low pLDDT [12]. Figure 1 displays the distributions of the pLDDT scores reported by AF2 for resolved/ordered and unresolved/disordered regions of crystal structures deposited in the PDB since AF2 was trained (the “post-AF2 test set”, see Supplemental Information). On this set of structures, ordered regions most frequently show pLDDT scores greater than 80, while IDRs have a broad distribution of pLDDT scores, with about 40% of unresolved regions falling below a pLDDT score of 50. From this analysis, we conclude that confidence scores below 80 are more indicative of disorder than of confidence in the accuracy of ordered structures, therefore in calculations on ordered-model accuracy we employ a cutoff of pLDDT>80.

Figure 1. Distribution of AlphaFold confidence scores across ordered (blue) and disordered (red) regions.

Ordered and disordered regions correspond to resolved and unresolved parts, respectively, for the post-AF2 test set. Terminal regions were not included. Ordered regions most frequently show pLDDT scores >80%. Disordered regions show a broad distribution of pLDDT scores with comparable frequencies from pLDDT scores between 20% and 90%.

Divergence has a minor effect on model accuracy

A problem with the 6-bin histogram used to estimate the distribution of model accuracies (Fig. 2a in [11]) is that aRMSD is a non-local measure. If a model is incorrect at the fold level, the expectation value of aRMSD scales with the radius of gyration. Thus, a model with 30 Å aRMSD against the experimental structure could be consistent with an entirely misfolded domain of around 1000 residues in length [22] or simply with rotation of a smaller domain about a single residue. To characterize different effects on model accuracy, we down selected the post-AF2 test set to 1,779 models that can be aligned with a corresponding experimental structure (see Supplemental Information) and used them to evaluate all-atom and backbone measures. pLDDT correlates poorly with log(aRMSD) on this set: Spearman rank correlation coefficient is 0.43 on the median (Supplemental Figure S1A). Truncating the range of pLDDT over which the median is calculated with a floor of 80 slightly improves the coefficient to −0.48 (Supplemental Figure S1B). We refer to the per-model median value of pLDDT scores greater than 80 as pLDDT₈₀.

Next, we split this down-selected test set into two pairs of subsets. The first pair explored high (pLDDT₈₀ > 90) or low (pLDDT₈₀ < 88) confidence scores. The second pair explored high (in clusters at 100% identity for over 80% of the length) or low (out of clusters at 5% identity for over 80% of the length) sequence identity to structures previously in the PDB. Cutoff values in these pairings were chosen to give maximal differences while maintaining roughly comparable fractions of the test set in the two arms of each pairing. We calculated distributions on log(aRMSD) and on the all-atom LDDT [14] for each of the subsets (Figure 2).

Figure 2. Kernel-density estimates of the distribution of differences between AF2 models and crystal structures using atomic Root-Mean-Square Displacements on C_α atoms (top) and Global Local Distance Difference Test metrics (bottom).

These distributions were calculated from crystallographic structures that were deposited in the Protein Data Bank after the AF2 training set cut-off date of April 30, 2018. Only residues that unambiguously intersect between AF2 models deposited in EMBL [48] and crystal structures were considered, with a minimum per-chain length cutoff of 20 residues, resulting in 1810 structural models to be compared. Distributions for mean confidence levels (pLDDT₈₀) over the raw models at or above 90 (solid blue line) and below 88 (dotted orange line) are plotted. We also clustered the PDB using mmseqs [49] to select for sequences nearly identical to an existing structure (in clusters with 100% minimum sequence identity over 80% of the longest sequence and cluster mode 2, dot-dash green line) or decisively non-matching regions (out of 5% minimum sequence identity, dotted red line). The high-confidence (pLDDT₈₀>90) distribution on log(aRMSD) peaks at 1.7 Å aRMSD, with a long tail extending beyond 10 Å at the 10% level. The low-confidence distribution on log(aRMSD) has a broad flat shape suggesting peaks at 3 and 20 Å. The high-identity distribution looks similar to the high-confidence distribution, while the low-identity distribution has peaks near 2 and 20 Å, respectively. Plotted against all-atom LDDT, the high-confidence, high-identity, and low-identity distributions look similar to each other; only the low-confidence distribution is distinct, with a single peak at LDDT ~75. Notations on the x-axis indicate differences between structures of ligand-free vs. ligand bound myoglobin (MB, PDB entries 1A6N and 1A6G); R- vs. T-state hemoglobin (HB, PDB entries 6BWP and 6BWU); unphosphorylated vs. doubly-phosphorylated conformations of an extracellular signal-regulated kinase (ERK2, PDB entries 1ERK and 2ERK0); and calcium-free vs. calcium-bound calmodulin (CALM, PDB entries 1CLL and 1QX5). The cyan line shows the proposed LDDT cutoff for a structure that is likely to be useful for virtual screening.

The aRMSD metric is not well-suited for characterizing structural models because its non-local nature tends to exaggerate the effects in small number backbone angle changes [23]. The lack of a high-difference tail in the low-identity LDDT distribution, together with the suppression of the high-difference tail in the low-confidence distribution, suggest that most differences between model and structure are in a few local coordinates, rather than many. Using LDDT as the accuracy measure improved Spearman’s correlation on pLDDT₈₀ to 0.60 (Supplemental Figure S2).

Less than 1% of the high-confidence distribution appears in the range consistent with fold-level inaccuracies at LDDT<50. Distributions for the low-similarity, high-confidence, and high-identity subsets are approximately the same, with the caveat afforded by the paucity of low-similarity models. But the 4% of models in the low-confidence distribution are distinctly worse than the other subsets. These observations suggest that AF2 produces models that are correct at the fold level more than 95% of the time, better than the previous two-thirds estimate [11]. The high-confidence and high-identity subsets similarity suggest that sequence divergence with PDB entries is not the primary driver of AF2 model inaccuracy for this set of structures.

Discordance limits model accuracy

Another driver of model inaccuracy is how AF2 handles differences among structures in the PDB with similar or identical sequences, a phenomenon we call conformational discordance. AF2's training algorithm propagates conformational discordance through down-selecting among multiple PDB structures in a way that preserves maximum differences [11]. A key question is how well AF2 preserves correlations among conformational degrees of freedom–not just mean values and uncertainties–because those correlations are not needed to address the problem for which its algorithms were designed.

An illustrative case of conformational discordance is calmodulin (shown in Figure 3). Calmodulin is a kinase that, upon binding Ca⁺⁺ ions, changes from a globular to a dumb-bell shape primarily through differences in two adjacent hinge residues [24]. In the calmodulin AF2 model, the effects of both conformations present in the training set are reflected in low confidence scores at the two hinge residues. While different AF2 runs yield slightly different results, none of the resulting models that we have sampled accurately reflect either the ion-free or ion-bound structures, but rather seem to be variations around the average of the two states and correlated changes in the two hinge residues have been lost. This example suggests that conformational discordance in the PDB results in composite AF2 models that rarely sample underlying conformations in the PDB.

Figure 3. Comparison of discordant crystal structures of calmodulin with an AF model.

The calcium-bound crystal structure (PDB entry 1CLL, thin green cartoon with Ca⁺⁺ ions as spheres), with alignments against the first half of the calcium-free crystal structure (PDB entry 1QX5, thin black cartoon) and the AlphaFold2 model (P0DP23-F1-model_v1, thick yellow-red cartoon), aligned on their N-terminal halves. Yellow regions of the model represent very high confidence (pLDDT > 90) residues, while dark-red regions represent very low confidence (pLDDT<50) residues. The low confidence region at the center of the AF model corresponds to a hinge where the calcium-bound and calcium-free models diverge. When aligned in this manner, aRMSD values of 7.2 Å against the calcium-bound structure and 6.7 Å against the calcium-free structure were obtained. When aligned across all residues, the AF model yields aRMSDs of 10 Å against the calcium-bound structure and 17 Å against the calcium-free structure, respectively. Global LDDT scores for the experimental structures are 49% for all atoms and 56% for C_α only.

Druggability: Are AF2 models ready for virtual screening?

Important structural elements relevant for drug discovery, such as prosthetic groups, ion binding, and protons are not included in AF2 models. Known protein conformational changes (as shown in Figure 3) can help us assess the effects of model accuracy on AF2 model readiness for target-based virtual screening (TBVS). If the model needs to be as close to the crystal structure as deoxy-myoglobin is to carboxy-myoglobin [25], only a tiny fraction of the AF2 models would be suitable for TBVS. If the model needs to be as close as R-state is to T-state hemoglobin, AF2 models may be suitable for characterizing allosteric sites [26]. A more typical TBVS example, where accuracy needs to be similar in capturing conformational changes, is when ERK2 is doubly phosphorylated. Given this example, a practical lower bound of global pLDDT of 80 could serve as basis for a model to likely be TBVS-ready. A value of pLDDT of 80 indicates a 68% likelihood of sidechain rotamers falling into the correct hemisphere (Fig 2B in [11]). Surfaces formed by two adjacent residues with pLDDT ≥ 80 are very close to the 50% accuracy limit if rotamer errors are independent. Having previously introduced pLDDT₈₀, we set pLDDT₈₀ ≥ 91.2 as criterion for assessing AF2 model quality, combined with the fraction of protein length for which this holds true (pLDDT₈₀_frac) to evaluate TBVS potential; see Supplemental Information. These criteria allow us to calculate a confusion matrix (see Supplemental Figure S2) that gives the sensitivity (true positive rate of classification) of 90.1% and a precision (positive predictive value of classification) of 86.3%.

Given these criteria, we evaluated which AF2 models of the human understudied proteins, Tdark [7], which currently lack an experimental PDB structure might be TBVS-ready (Figure 4A). Of the set of 5592 “dark” proteins with AF2 models, 3051 (54.6%) meet our criteria for possibly being accurate enough for TBVS studies (Figure 4B). Taking into account the estimated false-positive rate (~6% of total), this implies that AF2 provides TBVS-ready models for about half of the understudied human proteins. Additionally, among the Tdark proteins associated with very high or confident AF2 models, 664 match druggable protein classes as follows: 235 enzymes; 23 GPCRs; 220 immune response proteins; 5 ion channels; 16 kinases; 45 receptors; 25 signaling proteins; 97 transporters. More strictly, by matching Tclin protein motif domains (according to PFAM, InterPro, and Prosite) with Tdark proteins, 32 of the above 664 may be more likely to have “druggable” binding domains[27]. In total, 50 Tclin associated motifs are present in “very high” or “confident” Tdark AF2 models; see Supplemental Information.

Figure 4. Fraction of the dark genome potentially illuminated by AF2 models.

A) Of the set of 5592 unique “dark” proteins with AF2 models, 3051 (54%) pass the proposed selection criteria of pLDDT80 greater to or equal to 91.2 while having at least 20 residues with pLDDT ≥ 80. B) Pie chart illustrating AF2 model quality according to pLDDT80-derived criteria (see Supplementary Information): 3051 (54%) proteins associated with “very high” or “confident” AF2 models are likely to be TBVS-ready, whereas 2541 proteins are not.

CONCLUSIONS

In our opinion, future work would do well to move away from the familiar aRMSD metric of overall model-structure agreement in favor of LDDT or other local measures. The aRMSD metric suggests that AF2 models are worse compared to pLDDT. Structural bioinformatics would also benefit from developing measures that disambiguate the effects of disorder, discordance, and divergence.

Proteins can take on different conformations, and which protein conformation is more druggable depends on the clinical need associated with a particular disease state. Screening the right target in the wrong conformation reduces the likelihood of finding valuable leads. A few proteins are represented in the PDB by structures determined in multiple conformations, while most have only one. Many of the biophysical drivers that determine protein conformation, such as the hydrophobic effect, are poorly understood at present. More work will be needed on the taxonomy of possible protein conformations before AI approaches can be expected to tackle the problem of conformation robustly. Until then, it might be helpful to eliminate discordant structures from the training set to predict a single conformation with higher accuracy than achieved with composite conformations at present.

AF2 forces us to reconsider the implications of disorder on druggability because it performs well at predicting IDRs [28]. Having trust that a region is disordered versus trusting the ordered region's accuracy leads to different conclusions. It is worth noting that the dataset used in this work primarily includes proteins enriched with relatively short IDRs, as 95% of proteins in the dataset are at most 29% unresolved/disordered. In a scenario where the dataset included proteins with significantly longer stretches of IDRs, the results from current work may not apply. Proteins containing IDRs play critical roles in many biological functions [29-36] and are associated with various diseases [37-40]. Thus, IDRs are potential targets in drug discovery [41-43]. “Disordered” does not mean “undruggable” because unique strategies for drug discovery in targets containing disordered regions are available [44]. Regions of pLDDT < 50 in an AF2 model indicate those strategies could be employed. Moreover, the existence of a structural model is neither necessary nor a sufficient condition for drug discovery. Even the use of high-quality experimental structures of the correct conformational state does not guarantee successful TBVS hits.

About 5% of the human "dark" proteome has structures in the PDB (Supplemental Information). Cost-to-benefit analyses of whether to deploy TBVS on AF2 models remain project-dependent. However, AF2 model quality may be "good enough" for rapid deployment for over 3000 understudied human proteins. AF2 models may help de-risk protein targets through protein expression and solubility and may provide protein engineering suggestions. By identifying likely boundaries of compact domains, disordered regions, or linkers, AF2 and other methods can enable synthesis-by-domain strategies that can break large proteins into more tractable modules to be expressed or synthesized then reconstituted in-vitro. Regardless of its impact on in silico technologies, AF2 does not preclude structural biology and structure-based drug design. However, AF2 is poised to become a powerful tool in the evolving drug discovery arsenal.

Computational models are very different from experimental structures in that they can be updated on-demand with the latest improvements. Public notebooks such as ColabFold [45] facilitate the removal of disordered termini, improving sequence alignment, adding a binding partner, and calculating new models within minutes. Although not designed with protein oligomers or assemblies in mind, multiple groups are working on use of AF2 to illuminate protein-protein interactions. In 2014, it was estimated that 40% of protein structures were experimentally determined [46]. With AF2 and future improvements, structural biology, and drug discovery are about to exponentially increase with new computational tools that combine sequence evolution, structures, and ligand binding knowledge [47].