How good are AlphaFold models for docking-based virtual screening?

Summary

A crucial component in structure-based drug discovery is the availability of high-quality three-dimensional structures of the protein target. Whenever experimental structures were not available, homology modeling has been, so far, the method of choice. Recently, AlphaFold (AF), an artificial-intelligence-based protein structure prediction method, has shown impressive results in terms of model accuracy. This outstanding success prompted us to evaluate how accurate AF models are from the perspective of docking-based drug discovery. We compared the high-throughput docking (HTD) performance of AF models with their corresponding experimental PDB structures using a benchmark set of 22 targets. The AF models showed consistently worse performance using four docking programs and two consensus techniques. Although AlphaFold shows a remarkable ability to predict protein architecture, this might not be enough to guarantee that AF models can be reliably used for HTD, and post-modeling refinement strategies might be key to increase the chances of success.

Graphical abstract

Highlights

• •Well-known AF models are evaluated for their HTD capability using 4 docking programs
• •The performance of as-is AF models is significantly lower compared with PDB structures
• •Even on very accurate models, small side-chain variations impact the performance
• •A refinement of AF models might be crucial to maximize the chances of success in HTD

Computational chemistry; Protein; Protein folding; Artificial intelligence.

Introduction

A crucial component in molecular docking is the availability of three-dimensional (3D) structures of the protein target. Although the number of deposited structures in the PDB¹ is continuously increasing (∼199,000 in November 2022), the gap between non-redundant protein sequences and experimental structures is steadily widening. For the last 20 years, the structural genomics consortia initiatives^2,3 have been accelerating the characterization of representative protein structures, mainly from families poorly represented in the PDB.

Whenever experimental structures were not available, or easily obtainable, in silico homology modeling has been widely used to obtain a reliable 3D representation of the target (or at least, of the binding site) for docking-based drug discovery endeavors.⁴ Homology modeling is a computational methodology to characterize an unknown protein structure (the target) using a related homologous protein whose experimental structure (the template) is known.⁵ This methodology is based on the underlying assumption that proteins with similar sequences should display similar structures.⁶ The use of homology models in docking projects is already consolidated with a performance comparable to experimental structures.^7,8,9,10

Although the quality of homology models depends on several aspects, such as target-template sequence similarity, accuracy of the alignment, and the choice and resolution of the template, it is acknowledged that the post-modeling refining process is critical to obtain a reliable 3D representation of the binding site (BS).^11,12,13,14 This can be understood in view of the dependence of the binding site structure on the bound ligand, what highlights the importance of accounting for protein flexibility, at least at a binding site level, in the homology modeling process.^15,16,17 Thus, it is natural to incorporate information about existing ligands in co-modeling the binding site, such as in the ligand-steered homology method,^16,18 in which the six rigid coordinates of the ligand, the conformational space of the ligand torsional angles, and the binding site sidechains are optimized through flexible-ligand—flexible-receptor Monte-Carlo-based docking.¹⁹ Similar approaches have been published, showing that refined models display an enhanced performance in high-throughput docking (HTD).^20,21,22,23

Recently, the implementation of DeepMind’s artificial intelligence model, AlphaFold (AF),²⁴ set a milestone within the field of protein structure prediction. The astonishing and outperforming results within the 14th Critical Assessment of protein Structure Prediction (CASP14)^25,26 set AlphaFold as the breakthrough of the year by Science (doi.org/10.1126/science.acx9810) and method of the year by Nature.²⁷ AlphaFold predictions have gained a notorious importance; not only the structure prediction of the entire human proteome has been already carried out²⁸ but a collaboration between DeepMind and the European Molecular Biology Laboratory’s European Bioinformatics Institute (EMBL-EBI) led to the creation of the AlphaFold Protein Structure Database,^29,30 which, at the time of writing (November 2022), contains over 200 million predicted structures. Evidently, the great excitement driven by AF is leading to a paradigm shift in the field of structural biology.³¹ Even the PDB, which contains experimentally determined structures, has incorporated AF predictions.³² Furthermore, not only different implementations of AF with specific refinements are being actively developed^33,34 but also developments implementing AF model predictions are emerging at a fast pace,^35,36 including coupling AlphaFold with cryogenic electron microscopy maps for structure determination,³⁷ molecular replacement,^38,39 NMR structural refinements,⁴⁰ prediction of protein-DNA binding sites,⁴¹ protein design,^42,43 and the prediction of protein-protein interactions,⁴⁴ among others.

Remarkably, “AlphaFold is trained to predict the structure of proteins as they might appear in the PDB” (https://alphafold.ebi.ac.uk/faq); moreover, “backbone and side chain coordinates are frequently consistent with the expected structure in the presence of ions (e.g., for zinc-binding sites) or co-factors (e.g., side chain geometry consistent with heme binding)” (https://alphafold.ebi.ac.uk/faq). These facts, and the public and impressive success of AF in terms of overall model accuracy, prompted us to evaluate how accurate and useful as-is AF models are in the context of docking-based drug discovery, as an alternative to using PDB structures. On 22 diverse proteins we compared the performance of AF models (extracted from the AlphaFold Protein Structure Database) versus PDB structures in HTD. We conclude that despite an overall very good accuracy in reproducing protein topology and the binding site, HTD on AF models exhibits a consistent worse performance compared with experimental structures, with zero enrichment factors in several proteins.

Results

We selected a benchmark set of 22 targets from diverse protein families used in an earlier work⁴⁵ (Table 1). Considering what has been said earlier of AF models in terms of their representativity of ligand-bound complexes, to evaluate the performance of as-is AF models in HTD we chose to compare with holo PDB structures. Because AlphaFold does not predict the positions of co-factors, metals, ligands, ions, or water molecules, to compare structures on an equal standing, we stripped PDB structures from water molecules, ions, co-factors, etc.; we also avoided any co-refinement of the PDB structure with the native or other ligands, what would also have enhanced the outcome. AF-modeled structures were obtained from the AlphaFold Protein Structure Database.³⁰ Four docking programs were used, AutoDock 4, ICM, rDock, and PLANTS, which have different search algorithms and scoring functions. We evaluated the HTD performance of AF models using two proven effective consensus techniques, ECR⁴⁶ and PRC.⁴⁵ Although the ECR is a ranking-based consensus method, PRC is a combination of both ranking- and docking-based consensus, which has shown a remarkable performance improvement over previous consensus methods and individual docking programs. In addition, we docked native ligands present in crystal structures to compare with their poses on AF models.

Table 1.

Target proteins used for HTD

Receptor	Receptor code	PDB	Resolution (Å)
β2 adrenergic receptor	ADRB2	4LDO	3.2
Androgen Receptor	ANDR	2AM9	1.6
Cyclin-dependent kinase 2	CDK2	1FVV	2.8
Cyclooxygenase-1	COX1	2OYU	2.7
Estrogen receptor α	ESR1	3ERD	2.0
Fatty-acid-binding protein 4	FABP4	2NNQ	1.8
Heat shock protein 90 α	HSP90a	1UYG	2.0
Insulin-like growth factor 1 receptor	IGF1R	2OJ9	2.0
Leukocyte-function associated antigen 1	LFA1	2ICA	1.6
Progesterone receptor	PRGR	3KBA	2.0
Protein kinase C β	KPCB	2I0E	2.6
Protein-tyrosine phosphatase 1B	PTN1	2AZR	2.0
Purine nucleoside phosphorylase	PNPH	3BGS	2.1
Renin	RENI	3G6Z	2.0
Tyrosine-protein kinase ABL	ABL1	2HZI	1.7
Urokinase-type plasminogen activator	UROK	1SQT	1.9
Dopamine D3 receptor	DRD3	3PBL	2.8
Thymidine kinase	KITH	2UZ3	2.5
Phosphodiesterase 5A	PDE5A	1UDT	2.3
Coagulation factor VII	FA7	1W7X	1.8
Hexokinase type IV	HXK4	3F9M	1.5
Dihydroorotate dehydrogenase	PYRD	1D3G	1.6

The topology of AF models is analyzed to assess whether they are suitable for HTD

The comparison of AF models to PDB structures is shown in Table 2. The pLDDT metric, as well as the RMSD values between backbones of the entire structure and within the binding site residues are displayed. Most AF models show very good overlap to their corresponding PDB structures measured using backbone RMSD for the complete protein and also for binding site residues (cf. columns 3–5 from Table 2). Some targets show subtle differences in certain secondary structure elements that interfere with the binding site, and a few of them show structural differences that directly impede carrying out docking within the binding site; for example, in RENI, where the pocket in the AF structure is blocked by the N-terminal loop, which adopts a completely disordered conformation compared with their corresponding residues in the crystal structure (see Figure 1).

Table 2.

Analysis of AF structural models and comparison to their corresponding experimental structures

Receptor	pLDDTa	Backboneb RMSD (Å)	Backbonec RMSD (Å)	Binding site backbone RMSD (Å)	General comments
ABL1	92 ± 5	1.43	0.47	0.79	The Gly-rich loop is pulled toward the binding pocket.
PNPH	95 ± 3	1.69	0.50	0.85	The N55:G66 loop is modeled toward the interior of the protein, near but not in contact with the ligands.
ADRB2	97 ± 2	2.53	2.06	0.81	PDB has missing residues K1232:S1262, which are included in the AF model.
IGF1R	82 ± 16	1.84	1.29	1.64	The Gly-rich loop is in a conserved position, whereas the DFG loop (D1123:E1132) is pulled toward the outside of the protein.
CDK2	92 ± 4	3.73	2.04	0.71	Large backbone differences in the activation loop and C-helix.
COX1	96 ± 1	0.59	0.49	0.61	PDB has D164G and S193G mutations, which have no effect on the binding site; the AF model and the PDB structure lack the heme group near the pocket, which does not affect docking.
PRGR	95 ± 1	0.61	0.52	0.47	—
ANDR	95 ± 1	0.61	0.44	0.16	—
LFA1	85 ± 12	0.73	0.68	1.52	Helix α7 (D297:I306) is pulled toward the inside of the protein, narrowing the binding cavity space.
PTN1	96 ± 6	0.34	0.27	0.22	—
UROK	72 ± 17	1.32	0.46	0.95	PDB has M36I mutation (far from pocket). PDB has crystal waters important for ligand binding.
FABP4	96 ± 3	0.46	0.39	0.47	PDB has crystal waters important for ligand binding.
KPCB	92 ± 5	2.71	2.50	1.4	Residues T500 and S660 are phosphorylated in the PDB but are far from binding site. There is a sequence difference within the C-terminal region (C622:H636), and the backbone is pulled toward the inside of the binding site.
HSP90	94 ± 5	9.23	4.91	4.56	High backbone RMSD of the whole protein. There is a large difference in the position of residues N106:G137, near binding site. PDB has crystal waters important for ligand binding.
ESR1	96 ± 2	1.36	0.38	0.29	The AF model is in the agonist-bound conformation.
RENI	84 ± 13	7.76	0.59	10.24	AF model shows a disordered N-terminal loop, which blocks the binding cavity and prevents using the AF structure for docking.
DRD3	93 ± 3	1.09	0.51	0.35	Big difference in the modeled structure between residues R219:G320, far from the binding site.
KITH	94 ± 6	0.75	0.63	0.69	—
PDE5A	95 ± 3	1.45	1.02	0.43	PDB has a gap between residues Y664:Y676. AF model shows a difference in the position of those two residues, which are pulled toward the outside of the protein expanding the binding site. PDB has crystal waters important for ligand binding.
FA7	73 ± 16	1.53	0.71	1.02	—
HXK4	90 ± 6	1.38	0.95	1.70	V62:G71 loop is pulled toward the inside of the binding site, narrowing the space available for ligand binding.
PYRD	98 ± 1	0.55	0.37	0.40	—

The pLDDT metric is reported for residues within the binding site as a measure of model confidence: pLDDT >90: highly confident prediction; 70 < pLDDT<90: confident prediction; 50 < pLDDT<70: low confident prediction; pLDDT<50: should not be interpreted. Reported values correspond to mean and SD. The RMSD values calculated at the backbone level are also displayed.

^aPer residue Local Distance Difference Test (pLDDT) for residues in the BS (see STAR Methods).

^bConsidering all protein amino acids.

^cConsidering only amino acids involved in secondary structure motifs.

Figure 1.

AF model of RENI receptor (cyan) showing an obstructed binding site

The N-terminal loop containing residue N80 is blocking the ligand-binding space (displayed in orange). The corresponding PDB structure 3G6Z is displayed in yellow for comparison.

Nuclear receptors ESR1, ANDR, and PRGR could be found in two structurally different biological conformations (agonist and antagonist-bound) in the PDB. In the case of ESR1, from visual inspection of the AF model, we found that helix 12 (H12) was pulled toward binding site, with a topology that corresponds best to an agonist-bound conformation. Thus, the agonist-bound PDB structure 3ERD had a more adequate backbone superposition than the corresponding antagonist-bound PDB (3ERT), as shown in Figure 2, and therefore it was chosen for comparison. AF models of ANDR and PRGR were also in the agonist-bound conformation.

Figure 2.

AF modeling of the estrogen receptor

ESR1 AF model (cyan) superimposed to the (A) antagonist-bound conformation (PDB 3ERT) and (B) agonist-bound conformation (PDB 3ERD). The ligand binding space is displayed with orange surfaces.

In the case of KPCB, where the AF model and the PDB structure had differences at the sequence level in the C-terminal section, we generated the modeled structure with the available AF Colab Notebook (https://github.com/deepmind/alphafold) using the PDB 2I0E sequence as input. However, almost no differences were observed between our generated model and the AF Protein Structure Database model. In both AF structures the C-terminal loop (C622:H636) is pulled toward the inside of protein, making near contact to the binding site and modifying its topology. In this case, however, because the binding pocket is not blocked, we still used the modeled AF structure for HTD to evaluate its performance.

Protein kinases CDK2, IGFR1 and ABL1 show, on average, very good RMSD compared with their PDB structures. The AF model of CDK2 has large differences within the activation loop (containing the DFG motif) and the C-helix (compared with PDB 1FVV). In the case of ABL1, the Gly-rich loop is modeled toward binding site (compared with PDB 2HZI). In KITH, two possible conformations of the flexible loop formed by K49:S68 can be found depending on the ligand bound, as stated by Kosinska and co-workers.⁴⁷ We found that although PDB 2B8T has a high backbone superposition to AF model of 4.11 Å in the binding site, PDB 2UZ3 has a better overlap showing an RMSD of 0.69 Å (cf. Table 2). Therefore, the latter PDB structure was used to compare AF model performance.

For the rest of the targets, very subtle differences were observed from the backbone superposition that are detailed in Table 2.

Small variations in the AF-modeled side chains could have a very large impact on the results obtained in molecular docking

Table 3 shows the results of HTD using AF structures. The EF at 1% (EF1) is displayed for ICM, which on average was the best performing program. Column 2 shows the results obtained with the ECR consensus method. Moreover, the EF and HR results of PRC consensus method as well as the RMSD values of native ligand docking are also displayed. It can be readily seen that the AF models had a very low performance. On average, EF1 values of 8.4 and 8.8 were obtained with ICM and ECR, respectively. The same trend is observed with the PRC, where an average EF of 8.9 was obtained, with a low average HR of 0.16. Many targets had EF results less than 3.0 and even 0.0 in some cases. It should be noted that the PRC method provided, on average, better EFs on AF models than single docking programs, and the consensus ECR, what constitutes a small-scale validation of the PRC on protein models.

Table 3.

Docking results using AF structural models

Receptor	ICM EF1	ECR EF1	PRC			Native ligand RMSD (Å)
			A/Sa	EF	HR
ABL1	24.8	16.0	21/65	19.5	0.32	0.66
PNPH	13.6	18.6	18/69	17.9	0.26	1.2
ADRB2	6.3	3.4	1/16	2.5	0.06	2.03
IGF1R	9.5	7.5	3/19	10.1	0.16	5.01
CDK2	8.1	10.2	3/10	10.9	0.30	8.3
COX1	1.9	1.3	4/74	2.5	0.05	>10
PRGR	15.7	12.6	36/107	18.3	0.34	0.93
ANDR	0.8	0.0	0/169	0.0	0.00	6.5
LFA1	1.5	2.9	0/14	0.0	0.00	7.7
PTN1	24.1	29.5	15/40	21.3	0.38	1.6
UROK	17.3	2.5	1/25	2.5	0.04	2.01
FABP4	0.0	0.0	0/11	0.0	0.00	5.2
KPCB	3.7	11.8	1/35	1.9	0.03	6.3
HSP90	4.6	0.0	0/32	0.0	0.00	4.5
ESR1	1.1	8.3	36/206	10.2	0.17	2.5
DRD3	0.6	10.4	7/33	8.5	0.21	7.2
KITH	18.7	22.1	13/32	20.7	0.41	1.0
PDE5A	3.5	10.3	29/141	14.4	0.21	9.32
FA7	9.6	13.1	5/12	23.2	0.42	2.33
HXK4	4.3	1.1	0/5	0	0	9.64
PYRD	7.2	3.6	3/53	3.3	0.06	8.8
Average	8.4	8.8	–	8.9	0.16	–

EF1 is shown for ICM and ECR. The PRC consensus method is evaluated by EF and HR. The corresponding equations can be found in STAR Methods. All these metrics are dimensionless.

^aActive/Selected.

Table 4 shows a comparison of the results obtained in AF models versus PDB structures using the two consensus methods. It can be seen that, in general, AF models greatly worsen the HTD performance compared with their corresponding crystal structures. The same is also true for the four docking programs individually as seen in Table S1. PRGR, PTN1, DRD3, and KITH were the cases that obtained similar results to the PDB structures. UROK, KPCB, ANDR, FABP4, ADRB2, and PYRD show the largest ECR EF1 decrease compared with docking on PDB structures, followed by PNPH and LFA1. Consistent with this, Table 5 shows that although most PDB structures achieved very low native ligand docking RMSD values, the opposite trend was found for AF models.

Table 4.

Comparison of VS results between AF models and PDB structures

Receptor	ECR EF1		PRC EF		Visual inspection comments on binding sites comparison to PDB structures.
	PDB	AF	PDB	AF
ABL1	25.3	16.0	26.4	19.5	D381 is pulled toward the inside of the binding site. Small difference in the position of the Gly-rich loop.
PNPH	37.1	18.6	34.9	17.9	S33 has a difference in the OH group, which is 2.66 Å pulled to the inside of the pocket.
ADRB2	24.5	3.4	23.4	2.5	Small variation in N1293 and S1203 side chains.
IGF1R	18.3	7.5	38.6	10.1	DFG loop is located toward the outside of the protein. G1125 is 4 Å away in the AF model.
CDK2	12.8	10.2	16.3	10.9	K89 and F80 side chains are slightly pulled inside the pocket, narrowing the binding site.
COX1	3.4	1.3	5.8	2.5	F518 side chain slightly pulled inside the binding site.
PRGR	9.2	12.6	17.3	18.3	W755 is inverted. Difference in Q725 side chain: OH is at a 2.45 Å distance.
ANDR	9.0	0.0	13.5	0.0	Differences in Q711 and T877 side chains (see Figure 3C).
LFA1	10.9	2.9	11.6	0.0	Helix α7 (D297:I306) is pulled inside protein, shrinking the binding site.
PTN1	29.5	29.5	23.9	21.3	D48 and D181 side chains are rotated toward the binding site.
UROK	25.9	2.5	47.0	2.5	N322, S323, and T324 are pulled toward binding site with an average backbone RMSD of 2.28 Å.
FABP4	22.1	0.0	26.4	0.0	F57 is pulled outward of the pocket with an RMSD of 1.6 Å.
KPCB	45.3	11.8	53.8	1.9	C-terminal residues C622:H636 are greatly pulled toward the binding site, modifying its topology. F353 is pulled to the out.
HSP90	0.0	0.0	0.0	0.0	Big difference in structure in N106:G137, near binding site. Important crystal waters missing, which might be critical for ligand binding.
ESR1	34.3	8.3	29.7	10.2	Small difference in M421 and H524 side chains, slightly pulled toward the binding site.
DRD3	3.2	10.4	5.0	8.5	S192 is slightly pulled out of the pocket. T369 is inverted.
KITH	22.1	22.1	20.0	20.7	Small differences in the side chains of residues R53 and R61.
PDE5A	17.0	10.3	23.2	14.4	Y664 is noticeably pulled to the outside of the protein, whereas in the PDB it interferes with the binding site. Q817 and M816 side chains are inverted.
FA7	47.1	13.1	48.0	23.2	Differences in the position of residue K189, slightly pulled out of the pocket.
HXK4	5.5	1.1	15.2	0	Residues S64:P66 are notably pulled into the binding cavity, narrowing the space available for ligand binding. Y214 side chain is also pulled slightly toward the cavity.
PYRD	27.7	3.6	25.5	3.34	Small differences in R136 and Y147 side-chain positions. L68 points into the binding site, whereas it points away in the PDB. H56 and T360 side chains are flipped.
Average	20.5	8.8	24.1	8.9	—

Results of the two consensus methods ECR and PRC are displayed. Comments at the side-chain level of the binding site residues are found in the last column. For single docking programs results see Table S1.

Table 5.

Native ligand RMSD comparison with PDB structures using ICM docking poses

Receptor	PDB (Å)	AF (Å)
ABL1	0.15	0.66
PNPH	0.59	1.2
ADRB2	0.35	2.0
IGF1R	1.06	5.0
CDK2	1.5	8.3
COX1	1.8	>10.0
PRGR	1.03	0.93
ANDR	0.17	6.5
LFA1	1.9	7.7
PTN1	0.53	1.6
UROK	0.24	2.0
FABP4	0.54	5.2
KPCB	1.2	6.3
HSP90	6.3	4.5
ESR1	0.2	2.5
DRD3	0.65	7.2
KITH	0.51	1.0
PDE5A	3.37	9.32
FA7	3.13	2.33
HXK4	0.92	9.64
PYRD	0.23	8.8

Although the AF models used to perform HTD exhibit, in general, an adequate backbone superposition in the binding site to their corresponding PDB structures (cf. RMSD values in Table 2), some striking variations at the side-chain level within the binding site can be observed (cf. Column 6 in Table 4).

In UROK, differences can be observed at the backbone level for ligand binding residues N143, S144, and T145, which are pulled further into the pocket in the AF model with a backbone RMSD value of 2.3 Å, thus shrinking the available space for ligand binding. Moreover, deviations are also observed in side chains of Q194 and S192, as shown in Figure 3A. Regarding KPCB, the binding site of the AF model is also modified at the backbone level, with residues from C-terminal region C622:H636 pulled inside the protein, interfering with the BS. As expected, this had a huge impact on HTD results. For ANDR, variations can be noticed in Q711 and T877 side chains, shown in Figure 3B. Although for Q711 it was shown by Pereira de Jesús et al.⁴⁸ that it can appear in both conformations, T877 is essential for ligand binding, making important interactions with the native ligand in the crystallized PDB structure. In HSP90, a very poor performance was obtained, using both the AF model and the PDB structure without crystallized waters. It should be noted that the PDB structure with waters had a PRC EF of 15.4 in a previous study,⁴⁵ which shows how critical it is to include them for HTD. In PYRD, L68 side chain points into the binding pocket, interfering in ligand binding, whereas it points away in the PDB structure. Small variations are also observed in the side chains of residues R136, Y147, H56, and T360.

Figure 3.

Comparison of binding sites for selected targets

AF models are displayed in cyan and PDB structures in yellow. Native ligands are displayed in stick representation and the binding sites represented with orange surfaces.

(A) UROK binding site: differences in backbone can be observed for N143:T145.

(B) ANDR binding site: small variation in T877 side chain can be observed, which makes important interactions for ligand-binding.

(C) PNPH binding site: the most notable difference can be seen in S33 side chain.

(D) LFA1 binding site: backbone differences in the helix containing K305, and small variations in the side chains of E284 and K287 are observed.

In the case of FABP4, although most of the side chains are correctly modeled, F57 is pulled further back, thus opening more space within the BS. This residue participates in important hydrophobic interactions with the native ligand in the PDB. For PNPH, almost only one significant difference is found in the OH group from S33, which is pulled 2.7 Å further into the pocket in the AF model, as shown in Figure 3C. This might be critical, as serine residues are often involved in important interactions for ligand binding. Figure 3D shows LFA1 binding site where a notable difference can be observed at the backbone level in helix α7 containing residues L302:I306. This helix is pulled inside the pocket in the AF model, thus modifying the space available for ligand binding. Small variations in the side chains of residues E284 and K287 are also observed.

It can be seen from this analysis that small changes at the side-chain level of essential ligand-binding residues could have a very large impact on the EFs obtained from HTD campaigns and on the docking of native ligand structures. However, this impact could not have been expected in advance by looking at the backbone RMSD nor at the pLDDT metric, because overall, those were acceptable. In four out of the five AF models that worsened the HTD performance the most, the pLDDT metric is equal to or greater than 70 for every residue in the binding site (cf. Column 1 in Table 2), indicating high confidence in these modeled structures.

Discussion

In a real-world structure-based drug discovery scenario, most of the researchers would directly use a structure from the PDB, and if not available, it is now possible to select an AlphaFold structure from the AlphaFold Protein Structure Database. The objective of this study is to judge how good are these as-is AlphaFold structures for docking-based virtual screening.

To assess the docking performance of these AlphaFold models, we chose to compare with the performance of HTD in holo PDB structures. As AlphaFold structures present no bound ligand, it could be tempting to judge this holo-PDB versus "apo-like" AF comparison as unfair, as it has been shown that holo structures are more suitable for HTD.^49,50 However, this is not the case, because AF was not designed to predict structures in the apo conformation: AF was trained both with apo and holo structures, and as stated in the Introduction, backbone and side chain coordinates are frequently consistent with the expected structure in the presence of non-protein components (https://alphafold.ebi.ac.uk/faq).

Moreover, given that the goal of this study is to assess how fit AF models are for HTD, it is evident that the comparison must be made between the best option from the AF database and the best option from the PDB database. Given a protein target, the AF database offers a single structure; in the case of the PDB, the reasonable option would be to select a holo structure. Then, the comparison made herein is the one that best serves the main goal of the study.

As it can be seen in Table 4, HTD on AF models shows consistently lower EF values assessed with two consensus methods (ECR and PRC) when compared with the HTD on the corresponding PDB structures, also complemented with poor native ligand RMSD values (cf. Table 5); in several cases, the EF on AF models is even zero. Results also deteriorated for each individual docking program. From Tables 2 and 4, it can be inferred that these poor EF values could be due to (i) large differences at the backbone level within the binding site (as in RENI, where no docking could be performed due to the distortion of the binding site) and (ii) small variations either at the backbone level (UROK, for example) or at the side-chain level (ANDR and PYRD, for example). In several cases, even very subtle differences within the binding site could have a huge impact on the EF, such as in ANDR and FABP4. In agreement with what has been shown by others,^24,25,51 the AF models exhibit low backbone RMSD values compared with PDB structures, thus demonstrating the remarkable ability of AlphaFold to predict protein architecture; moreover, from Table 2, it can be readily seen that our models also show low backbone RMSD and good pLDDT values within the binding site. Therefore, we must conclude that the accuracy of AlphaFold in reproducing protein topology and binding site anatomy with very good values of the pLDDT metric is not enough to guarantee that AF models can be reliably used for molecular docking purposes. Thus, crude AF models do not seem to be suitable for HTD without performing post-modeling refinement techniques.¹¹ On the one hand, these results are in agreement with two contemporary studies, namely, Zhang et al.,⁵² who evaluated AF models for 28 targets extracted from DUD-E with the Glide docking software,⁵³ and Díaz-Rovira et al.,⁵⁴ who evaluated AF models for 10 targets of the DUD-E. Although in the latter study the utilized docking software was also Glide, the assessment was carried out in a "real-world scenario" by developing a customized AF version that excludes all high-sequence identity templates from the training set.⁵⁵ In addition to assessing out-of-the-box AF structures, Zhang et al. have shown that refining AF structures using the IFD-MD induced-fit docking method⁵⁶ significantly improves enrichment factors. On the other hand, Wong et al.⁵⁷ developed a model to predict protein-ligand interactions based on AF structures and molecular docking and indicated, contrary to our results, that "molecular docking using AlphaFold2-predicted structures is similar to using experimentally determined ones." On top of mentioning that the comparison that yields this conclusion was only made with eight experimental structures, it is also worth considering that model performance was weak by using either experimental structures or AlphaFold structures: the mean area under the receiver operating characteristic curve (AUROC) was, approximately, 0.48, which is worse than random. A slight improvement was obtained when using machine learning scoring functions (mean AUROC of 0.63).

It should be also highlighted that the single structural model provided by AF from a given sequence cannot represent (i) different biological states of the proteins (such as agonist- and antagonist-bound conformations, as in the case of GPCRs and nuclear receptors, or open versus closed, as in channels); (ii) protein dynamics (such as different conformations of the Gly-rich, catalytic, and activation loops in protein kinases); (iii) structural conformational differences, especially within the binding site associated with ligand binding. In fact, it has been highlighted that modeling a receptor not in the desired biological state is one of the current main limitations of AF;⁵⁸ although it is probable that the AF model corresponds to the state that is most represented in the training set, an intermediate state conformation could also be observed.⁵⁸ It should be thus acknowledged that different structures of the same protein available in the PDB might indeed represent structural diversity to a certain degree, which right now is not available for AF models.

In this contribution, we compared the AF models with their best PDB match in terms of backbone RMSD. However, in a real-world prospective case, biological and biochemical knowledge should be taken into consideration at the modeling stage to ensure that the modeled structure is in the desired biological conformation. It should be noted that this issue is many times avoided by using homology modeling, where the structural template from the PDB is chosen taking into consideration the sought biological state of the target;⁶ for example, for modeling a given GPCR in the agonist bound conformation, the templates from the PDB are selected among those exhibiting an agonist-bound conformation.⁵⁹ It should also be noted that efforts extending the use of AlphaFold to predict both active and inactive states of a protein target have been recently reported.⁶⁰

Regarding AlphaFold limitations, which have been discussed elsewhere,^32,35,36,58 it is observed that, from a structure-based drug discovery perspective, AF also provides an incomplete structural model due to the lack of water molecules, metal ions, and co-factors. Just to further illustrate this issue, in HSP90 a very poor performance was obtained using both the AF model and the PDB structure omitting crystallized waters (cf. Table 4), whereas by including water molecules in docking a PRC EF of 15.4 is obtained⁴⁵ (the ligand RMSD values with and without water molecules (Table 5) were 0.8 Å and 6.3 Å, respectively), which highlights the importance of including water molecules for HTD in some targets. As routinely done with PDB structures, AF models should be also carefully checked for correct histidine tautomers, asparagine and glutamine flipping, protonation states (especially acidic residues, histidines, and cysteines eventually involved in metal binding), and polar hydrogens conformation.

From a practical point of view and provided the AF model is in the desired biological state, a co-refinement of the binding pocket together with known ligands (whenever available) in a ligand-steered fashion¹⁶ might be the best strategy to sample binding site conformational diversity and maximize the chances of success in a prospective HTD endeavor.

Although the analysis of this study has been focused on the regions of AlphaFold models that superimpose with the crystalized domains of their corresponding PDB structure, it is worth mentioning that, in some cases, the regions that were cut out from the AF models seem to exhibit, by simple visual inspection, a high degree of disorder. As expected, these a priori disordered regions present low values of pLDDT, but the notorious contrast of the perceived model quality in matching and non-matching regions results is striking. Even though low pLDDT regions (pLDDT<50) were suggested to have a high likelihood of being unstructured in isolation, or only structured as part of a complex,²⁸ this issue clearly deserves further analysis.

Our conclusions will help to understand the current limitations of AlphaFold models in HTD and from this knowledge to develop strategies to circumvent its drawbacks and thus enhance its further application in drug discovery.

Limitations of the study

The conclusions drawn from this study to assess the impact of AF models on HTD enrichments are based on a benchmark of 22 different proteins; although this benchmark could be extended, we expect the conclusions drawn in that case to be qualitative like the ones outlined earlier. This study utilizes AlphaFold structures reported in the AlphaFold Database (accessed November 2022). Although updates in the AlphaFold database or structures generated with the latest version of AlphaFold may lead to slightly different results, we do not expect significant modifications of the results obtained nor the conclusions drawn from them.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
PDB	(Berman et al.,1 2002)	https://www.rcsb.org
DUD-E	(Mysinger et al.,61 2012)	http://dude.docking.org
NRLiSt	(Lagarde et al.,62 2014)	http://nrlist.drugdesign.fr
GLL/GDD	(Gatica and Cavasotto,63 2012)	https://cavasotto-lab.net
Alpha-Fold Database	(Jumper et al.,24 2021; Varadi et al.,30 2022)	https://alphafold.ebi.ac.uk
Alpha-Fold (Colab version)	(Jumper et al.,24 2021)	https://github.com/deepmind/alphafold
ICM	(Abagyan et al.,64 1994)	https://www.molsoft.com
Auto Dock 4	(Morris et al.,65 2009)	https://autodock.scripps.edu
PLANTS	(Korb et al.66 2009)	www.tcd.uni-konstanz.de
rDock	(Ruiz-Carmona et al.,67 2014)	https://rdock.sourceforge.net

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Claudio Cavasotto (CCavasotto@austral.edu.ar; cnc@cavasotto-lab.net).

Materials availability

This study did not generate new unique reagents.

Method details

Target preparation

The 22 protein targets used in this study (Table 1) were downloaded from the PDB. Water molecules and co-factors were deleted in all of them. For each target, an AF model was retrieved from the Alpha-Fold Protein Structure Database³⁰ using the corresponding Uniprot identification. An additional Alpha-Fold structure was utilized for KPCB, which was generated using a slightly simplified version of AF which is publicly available (https://github.com/deepmind/alphafold). In every case, AF models were cut to match their corresponding crystalized domains present in the PDB.

Both PDB structures and AF models were prepared in the same way using the ICM program⁶⁴ (version 3.9-2e; MolSoft, San Diego, CA, May 2022), in a similar fashion as in earlier works.^45,68 Missing amino acids and hydrogen atoms were added to PDB structures; local energy minimization was performed both on PDB structures and AF models. Polar hydrogens within the binding site were optimized using a Monte Carlo sampling in the dihedral space. Glutamate and aspartate residues were assigned a −1 charge, and lysine and arginine were assigned a +1 charge. For PDB structures, asparagine and glutamine residues were inspected for flipping and corrected whenever, and His tautomers were assigned according to their hydrogen bonding network.

Protein metrics

For comparison with PDB structures, AF models were superimposed to them using backbone atoms (C, C_α, N) considering: i) the complete protein; ii) residues which participate in defined secondary structure elements (α-, π- or 3.10 helices, or β-sheets) (cf. Table 2). RMSD values between backbones were calculated for the whole structure and for the ligand-binding residues, which were determined according to their distance to the native ligand in the PDB structures: if a heavy atom is within 4.0 Å of any heavy atom in the ligand, that residue is considered a binding site residue. The predicted Local Distance Difference Test (pLDDT) is a per residue metric reported in the Alpha-Fold Protein Structure Database³⁰ as an estimate of model confidence on a scale from 0 to 100; the LDDT is a superposition-free score that evaluates local distance differences of all atoms in a model and includes validation of stereochemical plausibility.⁶⁹ Following this evaluation criterion, we looked at the pLDDT metric especially for binding site residues.

Docking libraries

For each target, the corresponding docking chemical libraries consist of a set of active molecules and their corresponding matching decoys according to similar physico-chemical properties and structural dissimilarity, which has been shown to ensure unbiased calculations in docking simulations.^63,70 For all molecules, chirality and protonation states were inherited from the corresponding original databases. Libraries were obtained from the DUD-E database,⁶¹ except for the ESR1 agonists library which was obtained from NRLiSt⁶² database, and the ADRB2 library which was taken from GLL/GDD.⁶³ The number of molecules present varies from ∼2,200 in CDK2 to ∼23,000 in ESR1.

Docking methods

Four docking programs were used in total: ICM,⁶⁴ Auto Dock 4,⁶⁵ rDock⁶⁷ and PLANTS.⁶⁶ These programs have different search algorithms and scoring functions as described in previous studies.^45,46 Auto Dock Tools utilities⁶⁵ were used to prepare the input files for Auto Dock 4. The Lamarckian genetic algorithm was used for a 20-run search for each compound using 1.75 million energy evaluation. For ICM, a thoroughness of 2 was used for the search algorithm. The ChemPLP scoring function was used in PLANTS and speed 1 was set as search speed. For rDock, a radius of 8.0 Å ± 2.0 Å from a reference ligand binding mode was used to represent the cavity. For Vina, an exhaustiveness value of 8 was set. All the other parameters for every software remained at their default values. This parameter setting is the same used in a previous study,⁴⁵ what allowed direct comparison of AF docking results with earlier calculations. Only when needed, docking boxes on AF models were slightly modified to be accommodated due to small differences in binding sites.

Consensus methods

Two consensus methods were used to combine the results of the docking programs. The Exponential Consensus Ranking (ECR)⁴⁶ combines the ranks of each molecule determined using different scoring functions with an exponential distribution, calculated as

$$ECR\left(i\right)=\frac{1}{\sigma }\sum _j\exp \left[-\frac{r_j\left(i\right)}{\sigma }\right]$$

where r_j(i) is the rank of molecule i idetermined using the scoring function of program j, and σ is the expected value of the exponential distribution and establishes the number of molecules for each scoring function that will be considered; the ECR was found to be quasi-independent on σ, and we used σ = 10% of the total number of molecules for each docking library.

The Pose/Ranking Consensus method (PRC)⁴⁵ consists of a hybrid consensus technique that combines ranks and docking poses obtained with different docking programs and selects the molecules that meet the following criteria: if a molecule has a maximum of two matching poses, the corresponding ranks should be within the top 5% of the corresponding docking programs; with a maximum of three matching poses, those corresponding three ranks should be within the top 10%, and with four matching poses, the four ranks ought to be in the top 20%. Finally, only the molecules that are also in the top 1.5% of ECR consensus method described above are selected. It was shown that this subset of molecules increases the chance of finding real hits, measured through the Enrichment Factor (EF) and the hit rate (HR).

The EF is defined as

$$EF\left(x\right)=\frac{Hit{s}_x}{N_x}/\frac{Hit{s}_{total}}{N_{total}}$$

where Hits_x represents the number of actives present in a subset x of the docked library, N_x the number of molecules in subset x, Hits_total is the total number of ligands within the entire chemical library, and N_total its total number of molecules. When subset x is a percentage of the total number of molecules, for example the top 1%, we call it the EF at 1% (EF1).

The hit rate (HR) is calculated as

$$HR\left(x\right)=\frac{Hit{s}_x}{N_x}$$

and is a measure between 0 and 1 which represents the probability of finding an actual ligand within the subset x.

Published: January 20, 2023

Data and code availability

• •This paper analyzes existing, publicly available data. Databases are listed in the key resources table.
• •This paper does not report original code.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.