Enhancing Virtual Screening Performance of Protein Kinases with Molecular Dynamics Simulations

Abstract

In silico virtual screening (VS) is a powerful hit identification technique used in drug discovery projects that aims to effectively distinguish true actives from inactive or decoy molecules. To better capture the dynamic behavior of protein drug targets, compound databases may be screened against an ensemble of protein conformations, which may be experimentally determined or generated computationally, i.e. via molecular dynamics (MD) simulations. Several studies have shown that conformations generated by MD are useful in identifying novel hit compounds, in part because structural rearrangements sampled during MD can provide novel targetable areas. However, it remains difficult to predict a priori when an MD conformation will outperform a VS against the crystal structure alone. Here, we assess whether MD conformations result in improved VS performance for six protein kinases. MD conformations are selected using three different methods, and their VS performances are compared to the corresponding crystal structures. Additionally, these conformations are used to train ensembles, and their VS performance is compared to the individual MD conformations and the corresponding crystal structures using receiver-operating characteristic curve (ROC) metrics. We show that performing MD results in at least one conformation that offers better VS performance than the crystal structure, and that, while it is possible to train ensembles to outperform the crystal structure alone, the extent of this enhancement is target dependent. Lastly, we show that the optimal structural selection method is also target dependent and recommend optimizing virtual screens on a kinase-by-kinase basis to improve the likelihood of success.

Graphical Abstract

INTRODUCTION

In drug discovery projects, high-throughput biochemical screens (HTS) are commonly used to identify pharmacologically active compounds. Despite extensive automation these screens still require expensive equipment and labor, contributing to the ~$1.8 billion cost to bring a drug to market.^{1, 2} Therefore, improving the efficacy of the hit discovery process has the potential to benefit multiple stakeholders, from patients to pharmaceutical companies. Structure-based virtual screening (SBVS) utilizes structural information from the drug target to predict ligand-protein interactions and can be more cost-effective than traditional HTS alone.³ During SBVS, ligand-protein interactions are used in a scoring function that predicts the binding affinities of a database of compounds against a drug target. These predicted affinities can then be used to prioritize a smaller subset of compounds for experimental testing.⁴ A good scoring function reliably distinguishes known active compounds from inactive compounds.

While it is common practice to use a receptor whose coordinates are determined by X-ray crystallography for VS, the approach has limitations. For example, a single crystal structure only captures one conformation and provides limited information about a protein’s dynamic behavior, which can be an important regulator of ligand binding, as explained in two contemporary models. In the late 1950’s, Koshland suggested that ligand binding induces a conformational change in its cognate target that enhances ligand-binding affinity.⁵ With the advent of energy landscape theory, this concept was extended to the conformational selection method, which states that ligand binding biases conformational populations toward a single state.^6–10 Consistently ignoring the importance of protein dynamics can have a detrimental impact on VS outcomes. This can occur when the crystallographic binding site conformation is not predictive, and large numbers of false positives and false negatives result. Therefore, it is important to consider the dynamic properties of proteins when predicting ligand-binding affinities. To address the importance of protein flexibility in SBVS, ensemble docking, which docks ligands into multiple target conformations, was developed.

There are several ways to generate protein conformations for ensemble docking. One can use experimentally determined protein structures solved using X-ray crystallography or NMR, in various ligand-bound and unbound states.^11–20 However, the amount of time and expertise required to perform these experiments limits their utility. Alternatively, molecular dynamics (MD) simulations can reveal novel protein conformations with practical value to ensemble docking virtual screens. A number of studies have successfully used MD-generated ensembles to identify active compounds.^21–27 Nevertheless, despite efforts to determine how the use of MD structures affects chemical database enrichment,^28–31 defining protocols for selecting MD structures across various protein targets for virtual screens remains difficult. Also, it is challenging to know a priori, which protein targets will benefit from incorporating multiple target conformations in virtual screening experiments.

Here, we determine if MD-generated ensembles can be trained to maximize VS performance for six protein kinases. By analyzing the trained ensembles on an independent test set, we determine whether the addition of MD snapshots boosts the VS performance compared to the crystal structure alone. Specifically, we explore the impact of protein dynamics and ensemble training on VS performance for a set of kinase targets.

Protein kinases mediate most of the signal transduction in eukaryotic cells via phosphorylation of substrates.³² A comprehensive genome-wide study found that there are ~500 protein kinases in humans, comprising ~1.7% of the human genome.³³ They are involved in many cellular processes including: metabolism, transcription, cell cycle progression, cytoskeletal rearrangement and cell movement, apoptosis, and differentiation.³⁴ Given the importance of phosphorylation, it is not surprising that abnormal phosphorylation can lead to cancers,^35–37 cardiovascular diseases,³⁸ neurodegenerative diseases,³⁹ inflammatory diseases,^{40, 41} and diabetes,⁴² thereby making protein kinases an important drug target. Consistently, an analysis of FDA-approved drugs since the 1980s indicated that kinases have surpassed GPCRs as the most sought-after targets for cancer treatments.⁴³ To date, the US Food and Drug Administration have approved 27 small molecule protein kinase inhibitors and 1 lipid kinase inhibitor.⁴⁴

Although drug discovery for protein kinases has achieved a great deal of success, several significant challenges remain in the development of future drugs. First, evolutionary pressure results in the accumulation of point mutations in the kinase domain, which compromises inhibitor potency and leads to long-term drug resistance.⁴⁵ Second, the conserved architecture of the kinase domain within a class of protein kinases (for example, JAK kinases), makes obtaining selectivity challenging.^{40, 46} This lack of specificity sometimes leads to adverse side effects. Third, the current kinase inhibitors on the market only cover a small subset of the human kinome, with 18 of the 27 approved covering only three out of more than 90 groups of tyrosine kinases, BCR-Abl, ErbBs, and VEGFRs. Given these shortcomings and the importance of the target, there is a need to improve kinase drug discovery - optimizing the enrichment of actives in virtual screening methods by using ensemble docking is one important avenue.

In the present study, we determine if MD structures result in enhanced virtual screening performance compared to the crystal structure alone. The effect of structural selection is examined by considering three methods: RMSD clustering, volume-based clustering and random selection. The impact of training heterogeneous ensembles that consist of both MD conformations and a crystal structure is also considered. Performance analysis is conducted using receiver-operating characteristic (ROC) curve metrics, and the analysis is conducted for six protein kinases that cover three different kinase classes.

MATERIALS AND METHODS

Protein Kinase Systems

Six protein kinases from the directory of useful decoys-enhanced (DUD-E)⁴⁷ were included in this study. These include: (1) MK2: inactive conformation of MAP-kinase-activated protein kinase 2 (PDB code: 3M2W), (2) CDK2: inactivation conformation of cyclin-dependent protein kinase 2 (PDB code: 4GCJ), (3) ROCK1: active conformation of rho-associated protein kinase 1 (PDB code: 2ETR), (4) AKT1: inactive conformation of serine/threonine-protein kinase AKT1 (PDB code 4GV1), (5) IGF1R: inactive conformation of insulin-like growth factor 1-receptor (PDB code: 2OJ9), and (6) ABL: inactive conformation of non-receptor tyrosine kinase ABL1 (PDB code 2HZI). The assembly of the hydrophobic spine was used to determine the conformational state of each protein kinase (Figure 1).⁴⁸ Each protein kinase was aligned to the sequence of the cyclic adenosine monophosphate-dependent protein kinase (PDB 2CPK) to determine the residues that make up the hydrophobic spine. Visual analysis of the hydrophobic spine assembly was conducted to determine the conformational state: an ordered hydrophobic spine indicates an active conformation, and a disordered spine reveals an inactive conformation.⁴⁸

Figure 1.

Protein Kinases involved in study. For each protein kinase the crystallographic inhibitor is shown as a purple stick. The orientation of the DFG (Asp-Phe-Gly) motif was used to determine the inhibition type and is depicted as licorice colored by atom type (C: cyan, O: red, N: blue). The assembly of the hydrophobic spine residues⁴⁸ (shown as green balls) was used to determine the conformational state of the protein kinase.

The human kinome organizes all human kinases into 7 major groups, and the 6 kinases shown in Figure 1 cover three of these classes.³³ The CMGC kinome class, including MK2 and CDK2, consists of a diverse group of kinases named after cyclin-dependent kinases, mitogen-activated protein kinases, glycogen synthases, and CDK-like kinases. The AGC kinome class, including ROCK1 and AKT1, contains serine/threonine kinases regulated by cyclic AMP or lipids. The TK class, including IGF1R and ABL1, contains both receptor and cytosolic tyrosine kinases.

For all protein kinases, the crystal structures were chosen such that all activation loop residues were resolved with a resolution better than 3.0Å (Table 1). For all protein kinases except CDK2 and AKT1,, the inhibitor-bound complex referenced in the DUD-E dataset was used for the study, whose kinase conformation state matched the actives in the dataset. In the case of CDK2 and AKT1, whose DUD-E crystal structure contained missing activation loop residues, alternative crystal structures that matched the conformational states of the DUD-E crystal structures were selected. Also, it should be noted that the ROCK1 kinase also included the N-terminal domain; for the remaining five kinases, the N-terminal domain was excluded, and only included the catalytic domain.

Table 1.

Protein Kinase Systems Setup for MD Simulations and VS Training

Protein Kinase	PDB Code	a Atoms simulated	Total simulation time	b Actives	b Decoys
MK2	3m2w	54161	100ns	50	3050
CDK2	4gcj	50644	100ns	236	13688
ROCK1	2etr	74700	100ns	49	3038
AKT1	4gv1	52854	100ns	146	8030
IGF1R	2oj9	54844	100ns	73	4526
ABL	2hzi	49818	100ns	90	5220

^aThe total number of atoms in simulation includes the protein, ligand, ions, and explicit water molecules.

^bThe number of actives and decoys used in the trained models are shown. These numbers are the same across the training and test sets for both clustering methods.

Preparation of Systems for MD Simulations

The six inhibitor-bound protein kinase crystal structures were obtained from the Protein Data Bank (PDB).⁴⁹ Using Schrödinger’s Protein Preparation wizard version 2014-4, all six protein kinases were prepared.⁵⁰ Both ABL and ROCK1 kinases were crystallized as dimers; chain B was deleted and chain A was retained for both crystal structures. The remaining protein kinases were crystallized as monomers.

During the import and process step, the following boxes were checked for all six crystal structures: assign bond order, add hydrogens, create zero-order bonds to metals, create disulfide bonds, convert selenomethionines to methionines, delete waters beyond 5 Å from heteroatom groups, fill in missing side chains using Prime, and fill in missing loops using Prime^51–53 MK2 contained two missing residues in the N-terminus region, and seven in a loop region. CDK2 only contained two missing N-terminus residues. For ROCK1, there were five N-terminus and ten C-terminus residues missing. For AKT1, there were two N-terminus residues, five residues in a loop region, and three residues in the C-terminus missing. For IGF1R, there were four missing residues within the N-terminus and eight missing residues in a loop region. For ABL kinase, there were seven missing residues at the N-terminus. The missing N-terminus residues were ignored since they were located at the beginning and were far from the active site. However, the missing residues within a loop region were built in using Prime.

During the second preparation step, crystallization molecules and ions were deleted (MK2: magnesium ion; CDK2: 4 1,2-ethanediol molecules; AKT1: 4 glycerol molecules; ROCK1, IGF1R, and ABL: no crystallization molecules or ions present). AKT1 contained a phosphorylated threonine residue (Thr308-phospo), which was mutated back to threonine.

During the final preparation step, water molecules with less than three hydrogen bonds to protein residues were deleted. The protonation states of residues were assigned at pH 7 using PROPKA3.^54–56 Hydrogen bonds were optimized, followed by an all-atom minimization with termination based on convergence or reaching a heavy atom RMSD of 0.30 Å using the OPLS_2005 force field.⁵⁷ The resulting inhibitor-protein complexes were built for MD simulations in Amber14 xLeap.^{58, 59} Antechamber was used to determine the atom types, bond orders, atomic partial charges, and assign force field parameters to the inhibitors using the gaff force field.^{60, 61} The kinase-inhibitor complex was neutralized using chloride or sodium ions as described by Joung and Cheatham.⁶² The TIP4PEWBOX water model⁶³ was used to solvate the inhibitor 10Å in the x-, y-, and z- direction.

MD Workflow

All-atom explicit-solvent MD simulations were performed for each inhibitor-bound protein kinase on GPUs using the CUDA version of pmemd in AMBER14.^{64, 65} The general MD workflow consisted of three stages: minimization, equilibration, and production. The prepared systems were minimized in four steps using the steepest descent minimization method as follows: (i) Minimization of the protons, while restraining the protein, ligand, and solvent; (ii) Minimization of the solvent, while restraining the protein and ligand; (iii) Minimization of the ligand and solvent, while restraining the protein; (iv) Minimization of the protein side chains and water, while restraining the protein backbone; (v) Minimization of all atoms in the system. Harmonic force constraint energy of 10 kcal/mol-Ų was used for the restrained minimizations. The systems were then equilibrated at 300K and 1 atm for 200ps with backbone restraints using the NPT ensemble. The backbone restraints were then removed, and the system was allowed to equilibrate for an additional 200ps. MD was run for 5ns in the NVT ensemble using the SHAKE algorithm,⁶⁶ and restart files were written every 1ns. These restart files were used to begin 5, 20ns NPT simulations at 300K and 1 atm with a 2 fs time step. A total simulation time of 100ns was generated for each protein kinase target.

Selection of Protein Conformations

After the MD simulations, the solvent and neutralizing ions were removed from each system. The 20 ns MD trajectories were loaded every 40ps, resulting in 2500 MD snapshots, or frames, for each protein target. Residues within 10Å of the inhibitor were selected and defined as the active site. MD trajectories were aligned on active site alpha carbon atoms using cpptraj.^{58, 59} The aligned trajectories were clustered using two different clustering methods: 1) Gromos RMSD-based method^{67, 68} and POVME 2.0.⁶⁹

Using the Gromos algorithm, pairwise root-mean-square deviations were calculated for all the active site heavy atoms for each frame. MD frames were clustered together if their RMSD value was below the specified cutoff. The cutoff value for each protein kinase system was selected using the following criteria: i) there were no more than 40 clusters, ii) 90% of the trajectory was within the first 10 clusters, and iii) there were no more than 5 clusters with one single frame. The cluster centroids, or the representative for each cluster, of the number of clusters that contained at least 80% of the MD trajectory were selected to make up the ensemble for virtual screening.

The active site was also clustered based on its volume and shape using POVME 2.0. POVME 2.0 flooded the active site with equidistant grid points, and removed points that clashed with the receptor. The resulting grid points were used to calculate the active-site volume. These grid points were clustered, generating the same number of clusters as in the RMSD-based method, using their Tanimoto similarity scores. The cluster centroids were extracted for virtual screening.

Frames were also randomly selected from the MD trajectory in which the number of random frames matched the same number of RMSD and POVME cluster centroids. For example, for systems with five cluster centroids, every 500^th frame was selected. It is important to note that frames are not extracted in a time-dependent manner since one long 100ns trajectory was not run; instead five 20ns trajectories were grouped together.

Principal Components Analysis

GROMACS⁷⁰ was used to determine the two most dominant modes of motion of the active site of the MD conformations using principal components analysis (PCA). The trajectories aligned on the active site heavy atoms were used to remove translation and rotation before generating PCA. The variances of the atomic coordinates were determined using the first MD frame as the reference; these variances were used to generate the covariance matrix A. Diagonalization of the covariance matrix was used to identify eigenvectors 1 and 2. In other words, these variances were projected onto 2D space corresponding to the first two principal components or the components with the greatest amount of variance. Calculation of the covariance matrix A was conducted as follows, yielding the eigenvalues, λ: Aμ = λμ. A plot of these eigenvalues along eigenvectors 1 and 2 was used to compare how the cluster centroids represented the structural changes throughout the MD trajectory.

Bio3D⁷¹ was used to determine the principal modes of motion for the MD conformations and crystal structures. A PDB search using the UniProt ID and a BLAST search in the Bio3D R package was used to find all available inhibitor-bound crystal structures for all six protein kinases. A consensus-binding site was found between all crystal structures and MD conformations, and the conserved binding site residues were used for protein alignment. The Cartesian coordinates of the aligned conserved binding site residues were the elements of the covariance matrix. Diagonalization of the covariance matrix was used to derive principal components 1 and 2, and calculation of the matrix resulted in the eigenvalues. A plot of these eigenvalues along eigenvectors 1 and 2 was used to compare the PC space of the MD sampled and available inhibitor-bound crystal structures.

Ensemble Docking

For all cluster centroids from the MD simulations, water molecules and ions were removed. Schrödinger’s Maestro protein preparation wizard was used to determine the correct atom types of the cluster centroids.⁵⁰ The protonation states used during MD simulations were retained for docking. The inhibitor was used to generate the receptor grid, using the default settings.

The DUD-E database of compounds for each protein kinase target was used for the virtual screen.⁴⁷ This database consists of experimentally determined actives and property-matched decoys. For every active compound, fifty decoys are selected to ensure physicochemical similarity and topologically dissimilarity to each active. The bond orders, stereochemistry, hydrogen atoms, and protonation states were generated for the actives and decoys using Schrodinger’s LigPrep OPLS_2005 force field.⁷²

The prepared active and decoy molecules were docked into their respective crystal structure and ensemble of cluster centroids using Schrödinger’s Glide single precision (SP) scoring function.^73–75 The crystallographic pose of the inhibitor in the inhibitor-kinase complex was reproduced to validate the use of the Glide scoring function (Figure S1).

Measuring VS Performance

To ensure adequate VS performance, the ability to distinguish active and inactive, or decoy, molecules should be determined before performing a prospective virtual screen. While there are various metrics available to determine how well a virtual screen performs, this study focuses on two metrics: 1) the area under the receiver-operating characteristic curve (AUC) and 2) ROC-enrichment factor (ROC-EF). Both the AUC and ROC-EF are determined from the ROC curve, which plots the true positive fraction (TPF) against the false positive fraction (FPF) at various threshold settings.^{76, 77} The global classification ability of a virtual screen is given by the AUC, which is identical to the probability that an active will be ranked ahead of a decoy. However, the ability of a VS to enrich actives ahead of decoys early in the ranked list is often of more interest in drug discovery applications and enrichment factors (EF) are used to measure this ability. “Traditional” EF measures the ratio of actives in an early portion of the ranked list to the ratio of total actives in the database screened.⁷⁸ While this commonly-used metric provides a useful indication of early enrichment, its maximum value depends on the ratio of decoy to active molecules in the database.⁷⁹ This can be problematic when making performance comparisons across targets where the decoy to active ratio varies, as is the case for each of the targets considered in this work. Therefore, the ROC-EF metric does not suffer from the same liability. It is determined by calculating the ratio of the TPF (at a given FPF) to the FPF, TPF(FPF)/FPF.⁷⁸ Random classification is indicated by a value of 1 and perfect separation of actives and decoys is indicated by a value of FPF⁻¹. We select a FPF of 0.001 in measuring the early chemical enrichment. However, it should be noted that an alternative FPF can be used, and to our knowledge there is no standard or generally accepted protocol for selecting a FPF.

Training Ensembles to Optimize VS Performance

A recently developed method, EnsembleBuilder, was used to train ensembles to maximize AUC or ROC-EF.⁸⁰ The docking results of each centroid and crystal structure were merged together and randomly split into a training and test set, maintaining the same active-to-decoy ratio (Table 1). Using the training set, all combinatorial possibilities at each ensemble size was constructed and either AUC or ROC-EF values were used to rank the performance of the resulting ensembles. For example, given two cluster centroids and the crystal structure, (labeled A, B, and xtal, respectively), there are seven possible ensembles: three of ensemble size one (A, B, or xtal), three of sizes two (AB, A and xtal, B and xtal), and one of ensemble size three (A, B, and xtal). For each ensemble size, the best docking score value across all ensemble members was used to rank each compound,¹⁵ and AUC and ROC-EF values were determined from the resulting ranked lists. Finally, the ensemble combination with the largest AUC or ROC-EF was identified and retained. These best-performing ensemble combinations were used to screen the test set, and the resulting AUC and ROC-EF values were used to gauge prospective VS performance.

Quantifying VS Performance Gain

To compare the VS performance between the cluster centroids (or trained ensembles) and the crystal structure, the gain in AUC or ROC-EF was calculated as follows:

$$\begin{array}{c}\mathit{AUC}\mathit{gain}(\%)=[{\mathit{AUC}}_{\mathit{centroid}/\mathit{ensemble}}-{\mathit{AUC}}_{\mathit{crystal}}]\ast 100\\ \mathit{ROCEF}\mathit{gain}(\%)=({\mathit{ROCEF}}_{\mathit{centroid}/\mathit{ensemble}}-{\mathit{ROCEF}}_{\mathit{crystal}})\ast 100/1000\end{array}$$

To insure the percent gain values are between 1 and 100, the ROC-EF is scaled by the FPF of interest.

RESULTS

We performed 100 ns of MD on each protein kinase and wrote out snapshots every 40 ps, which resulted in a total of 2500 frames. Docking into every MD frame is computationally expensive, and because of long conformational relaxation times, likely unnecessary. Therefore, various metrics are available to reduce the MD ensemble in a meaningful way without losing critical structural information about the dynamic behavior of the protein. In this study, we utilized three methods for reducing the MD trajectories: RMSD and POVME clustering in addition to random selection. The resulting cluster centroids and random frames were used for VS experiments. The VS performances of the individual cluster centroids were compared to the crystal structure and random frames performance. Lastly, the performance of ensembles trained to maximize either the AUC or ROC-EF was also considered.

Comparison Between Structural Selection Methods

Comparison between RMSD and POVME Clustering Methods

Both RMSD and POVME clustering adequately capture large-scale conformational changes of the binding pocket that occur during the MD simulations (Figure 2). Each method samples a collection of conformations that collectively represent the conformational space spanned by the first two principal components. However, the conformational representation used by each (binding pocket RMSD and shape) is different and results in a unique pool of cluster centroids for each method. To determine how these differences impact VS performance, we compared the VS performance of each cluster centroid to the corresponding crystal structure using the training set.

Figure 2.

Comparison between how RMSD and POVME clusters the MD trajectory. Principal components 1 and 2 of the binding site in the MD snapshots, cluster representatives (centroids), and the crystal structure projected onto 2D space for each protein kinase are shown.

When considering global classification ability, as measured by the AUC, POVME yields the highest individual performing conformation (Figure S2), but this result is not consistent across all targets, and both RMSD and POVME clustering return centroids with VS utility. For instance, RMSD and POVME both yield high performing MK2 centroids that perform identically (AUC = 0.97) and marginally outperform the crystal structure (AUC = 0.96), representing a 1% performance gain. This small gain is not surprising: the large crystal structure AUC value leaves little margin for improvement, and any performance gain must necessarily be small. Also while performance gains varied across targets, high crystal structure AUC values and small performance gains were the norm. For instance, while AKT1 and ABL realize the largest AUC gains using the highest performing conformer, these gains were only 5.53% and 5.51%, respectively. Similarly, the highest performing conformations of CDK2, ROCK1, and IGF1R, resulted in AUC gains of 3.62%, 3.67%, and 2.83%, respectively. While base line crystal structure AUC values were high across all targets, this was not the case for early enrichment, where crystal structure performance was less than perfect, a case we consider next.

Similar to the global classification results, when early enrichment is considered, as measured by ROC-EF values, the clustering method that identifies the highest performing conformation is target dependent, and both methods reveal centroids with VS utility (Figure 3). For instance RMSD produces the highest performing cluster centroids for MK2, ROCK1, and ABL, while POVME yields the best performers for CDK2 and IGF1R. For AKT1, both clustering methods yield a conformer with the same ROC-EF of 95.89. Additionally, for four out of six targets (MK2, CDK2, AKT1, and ABL), both methods were able to identify conformations that outperformed the crystal structure. At 10%, the largest gain in early enrichment ability was realized by an MK2 centroid, which was identified by RMSD clustering. For CDK2, AKT1, and ABL, gains using the highest performing conformer are 4.66%, 3.42%, and 5.49% respectively. In contrast to these performance gains, neither ROCK1 nor IGF1R yielded centroids that outperformed the crystal structure. However, the highest performing ROCK1 centroid performed identically to the crystal structure (Figure 3). For IGF1R, where all centroids perform worse than the crystal structure, we note that the PCA plots of the crystal relative to the MD frames indicates little overlap, which may explain this ROC-EF difference.

Figure 3.

The ROC-EF of the cluster centroids and crystal structures against the training set for each protein kinase. The ROC-EF of the optimal trained ensemble (the ensemble with the highest ROC-EF) using RMSD and POVME centroids reveal an increase in ROC-EF compared to the cluster centroids.

Comparison between Clustering Methods and Random Selection

The randomly selected frames sample conformations that collectively represent the conformational space spanned by the first two principal components for AKT1 (Fig S3). However, for the remaining five protein kinases, the random frames samples a subset of PC space. Similarly to the clustering methods, the conformational representations of the random frames are unique and differ from both RMSD and POVME cluster centroids. To determine if clustering the MD simulation in a meaningful way versus randomly selecting frames has an impact on the VS performance, the VS performance of the random frames were compared to the cluster centroids against the entire dataset.

When considering global VS performance, the AUC of the random frames and cluster centroids are generally comparable, as indicated by the overlapping confidence intervals (Fig S4). For two systems, ROCK1 and IGF1R, the randomly selected frames yield the single highest MD conformer. However, these AUC values are not statistically higher than the single highest yielding cluster centroids. Therefore, we conclude that the structural selection method does not impact global VS performance. Next, we considered the early enrichment metric in comparing the structural selection methods.

Similar to global VS performance, when early enrichment is considered, the randomly selected frames yield similar ROC-EF values to the cluster centroids for majority of the targets, with the exception of AKT1 and IGF1R (Figure S5). For AKT, there are four random frames that have a higher EF than all cluster centroids; however, this increase is not statistically significant. For IGF1R, the single best MD conformer comes from the random selection method, which is not statistically higher than the single highest cluster centroid. Similarly to the results seen with AUC, the structural selection method does not impact the early enrichment of VS.

Even though our results show that the use of clustering methods does not boost VS performance over random selection, the clustering methods does a better job in representing the MD conformations as shown with PCA. Therefore, we limit the use of randomly selected frames here, and only consider the cluster centroids for ensemble training. To determine whether synergism between cluster centroids and the crystal structure could result in performance that exceeded that of any individual conformation, we used these structures to train ensembles.

Comparing Trained Ensemble and Crystal Structure VS Performance

Training ensembles on AUC yielded at least one ensemble with a higher AUC than the crystal structure against the training set (Figure S6). The optimal trained ensembles contained no more than four members across all six targets (Table 2). For MK2 and CDK2, the members consist of RMSD centroids, and POVME centroids for the remaining targets. Interestingly, the optimal trained ensemble’s AUC value is statistically significantly higher than the crystal structure across all six targets (p<0.05). The largest AUC gain using the optimal ensemble over the crystal is seen with ABL with a gain of 8.62%, while the smallest AUC gain (1.71%) is seen with MK2. For CDK2, ROCK1, AKT1, and IGF1R, the AUC gains are 5.20%, 6.93%, 7.06%, and 3.09% respectively. Next, we determined if training ensembles on ROC-EF would result in increased early enrichment performance over that of the crystal structure.

Table 2.

The Global Performance of the Virtual Screen (AUC) of the Optimal Trained Ensemble against the Test Set

Kinase	Ensemble Size	Cluster Method	Ensemble Members	Higher than Xtal?	a Statistically significant
MK2	3	RMSD	Centroids 1, 3, & xtal	YES	YES
CDK2	3	RMSD	Centroids 2, 3, & 4	YES	YES
ROCK1	3	POVME	Centroids 2, 3, & xtal	YES	YES
AKT1	4	POVME	Centroids 4, 5, 6, & xtal	YES	YES
IGF1R	2	POVME	Centroids 2 & 3	YES	YES
ABL	3	POVME	Centroids 2, 3, & 4	YES	YES

^aStatistical significance is determined at the 95% confidence level; p<0.05.

The centroids highlighted in red contributed the most to the VS performance.

Similarly to training on AUC, ensembles trained on ROC-EF resulted in at least one ensemble that outperformed the crystal using the training set (Figure 4). All optimal trained ensembles contained no more than three members (Table 3). The members consist of RMSD centroids for MK2 and ABL, and POVME centroids for CDK2, ROCK1, and IGFIR.

Figure 4.

The trained ensemble sizes and crystal structures ROC-EF (fpf=0.001) values against the training and test set across all six protein kinases. The 95% confidence intervals overlap between the training and test set, validating the training method.

Table 3.

The Early Chemical Enrichment of Actives (ROC-EF_fpf=0.001) of the Optimal Trained Ensemble against the Test Set

Kinase	Ensemble Size	Cluster Method	Ensemble Members	Higher than Xtal?	a Statistically significant
MK2	1	RMSD	Centroid 3	YES	YES
CDK2	2	POVME	Centroids 1 & 4	YES	NO
ROCK1	2	POVME	Centroid 2 & xtal	NO	NO
b AKT1	2	POVME	Centroids 4 & 5	YES	NO
	3	RMSD	Centroids 5, 6, & 7	YES	NO
IGF1R	3	POVME	Centroids 2, 3, & xtal	YES	NO
ABL	2	RMSD	Centroids 2 & 4	YES	YES

^aStatistical significance is determined at the 95% confidence level; p<0.05.

^bTwo ensemble sizes are shown because both results in the same ROC-EF value which is the highest.

The centroids highlighted in red contributed the most to the VS performance.

For AKT1, there are two optimal trained ensembles (RMSD size 3 and POVME size 2) as they have the same ROC-EF value. The optimal trained ensembles’ ROC-EF is statistically significantly higher than the crystal for CDK2 and AKT1 (p<0.05), with a gain of 6.36% and 6.16% respectively. For MK2, ROCK1, IGF1R, and ABL, the ROC-EF gains are 10.00%, 6.00%, 4.05%, and 6.59%, although these gains are not statistically significant (p>0.05). Interestingly, results demonstrate that trained ensembles of cluster centroids and the corresponding crystal structure yielded VS performance that exceeds that of the crystal structure. To determine if the optimal trained ensembles performance was due to synergism between structures or due mostly in part to a single high performing conformer; we also compared the optimal trained ensemble’s VS performance to the single highest performing MD conformer.

VS Performance of Trained Ensembles vs Cluster Centroids

Ensembles trained on AUC resulted in at least one ensemble with a higher AUC than the cluster centroids against the training set (Figure S2). However, this increase in AUC using the optimal trained ensemble over the single highest performing conformer is not statistically significant for any targets (p>0.05), with gains of only 0.85%, 1.58%, 3.26%, 1.53%, 0.26%, and 3.11% for MK2, CDK2, ROCK1, AKT1, IGF1R, and ABL respectively. However, it is noteworthy that the highest individual performing conformer is a member of the optimal trained ensemble for all targets except CDK2 (Table 2). Recall for CDK2, POVME yields the single highest performing conformation; however, the optimal trained ensemble consists of RMSD cluster centroids. Therefore, if we look within the RMSD cluster centroids only, we see that the optimal ensemble contains the highest performing RMSD conformer, thereby supporting the results seen with the other five targets. In addition to being a member of the optimal ensemble, the single highest performing conformation contributes the most to the VS performance of the optimal trained ensemble for all targets except AKT1 (Table 2, centroids highlighted in red). This means that the single highest performing conformer was more successful at correctly ranking actives ahead of decoys than any other ensemble member. While results suggested that the global performance of the trained ensembles is due to the single highest performing centroid, next we investigated if the same was true for early enrichment.

Ensembles trained to maximize ROC-EF resulted in at least one ensemble with a higher ROC-EF than the cluster centroids against the training set for all targets except MK2 (Figure 3). This exception is due to the fact that the single highest performing conformation yields a higher ROC-EF than all trained ensembles. A statistically significantly (p>0.05) higher gain in ROC-EF using the optimal trained ensemble over the single highest performing conformation is seen for IGF1R only, with a gain of 10.81%. For the remaining four targets, CDK2, ROCK1, AKT1, and ABL, the gains in ROC-EF are 1.69%, 6.00%, 2.74%, and 1.10% respectively, although this enhancement is not statistically significant (p>0.05). Similarly to the members of the optimal ensemble trained on AUC, the members of the optimal trained ensembles on ROC-EF contain the single highest performing conformation across all six targets (Table 3). Also, the single highest performing conformation is the largest contributor to the optimal ensemble for all kinases except CDK2, suggesting the early enrichment performance of the optimal trained ensembles is due to the single highest performing conformer (Table 3, centroids highlighted in red). Next we measured the VS performance of the trained ensembles against a test set of compounds it has never seen before in order to predict how the trained ensembles would perform prospectively.

Performance of Trained Ensembles Against the Test Sets

The VS performance of the trained ensembles against the test set validates the ensemble training method (Figures 4 and S6). Since the AUC and ROC-EF values on the test set are within the confidence intervals of the values against the training set, this suggests that the trained models may perform similarly prospectively. Similar to the analysis against the training set, we compared the VS performance of the optimal trained ensembles’ and crystal structure against the test set.

The optimal trained ensembles resulted in slightly higher AUC values than the crystal structures against the test set across all six protein kinases (Table 2). This enhancement in AUC is statistically significant across all six targets (p<0.05), similar to the optimal ensemble’s performance against the training set. The AUC gains are 2.16%, 7.26%, 2.61, 5.73, 6.03%, and 3.38% for MK2, CDK2, ROCK1, AKT1, IGF1R, and ABL respectively; these gains are modest as expected due to the large crystal AUC values. Next, we compared the ROC-EF of the optimal trained ensembles’ and crystal against the test set.

The optimal trained ensemble resulted in a higher ROC-EF than the crystal structure against the test set for all targets except ROCK1 (Table 3). The enhancement in ROC-EF is statistically significant for MK2 and ABL only (p<0.05), differing from the results against the training set where the ROC-EF is statistically significantly higher for CDK2 and AKT1. Similar to the early enrichment performance against the training set, the largest ROC-EF gain is seen for MK2 with a gain of 14%. Differing from the performance against the training set, no ROC-EF gain is seen for ROCK1, and the optimal trained ensemble and crystal perform identically (Figure 4). The gains in ROC-EF for CDK2, AKT1, IGF1R, and ABL are 3.81%, 2.74%, 8.22%, and 13.33% respectively. We also compared the VS performance of the optimal trained ensembles to the single highest performing conformer against the test set. The AUC and ROC-EF of the optimal trained ensemble was either the same or higher than the single highest performer against the test set (Figures S7 and S8). However, this increase is not statistically significant (p>0.05) for any target, so we limit discussion of our results here.

DISCUSSION

All Structural Selection Methods Yield Conformers that Enhance VS Performance

The impact that RMSD and POVME clustering methods have on VS performance is comparable. Both clustering methods selected at least one conformer that performed as well or better than the crystal structure, as judged by both AUC and ROC-EF values. While POVME identified the single conformer with the highest AUC across all targets, the improvement was not statistically significantly higher than the AUC of the best performing RMSD conformer. Therefore, we cannot conclude that POVME identifies conformers that result in meaningful performance gains relative to RMSD clustering. Either clustering method can successfully cluster the MD trajectory and reveal conformers that will be successful in VS experiments.

While the impact that clustering and randomly selecting MD frames have on VS performance is comparable, there are differences with how the methods represent the MD conformational space. The use of clustering methods ensures that the large-scale conformational changes of the binding site are captured in the VS experiments, which held true for all six targets. However, randomly selecting frames may or may not capture these conformational changes as seen with majority of the protein kinases within this study. Since, there is no way of knowing a priori the correlation between VS performance and binding site conformation, it is important to represent all conformations sampled during MD in VS methods. Therefore, we still conclude that clustering on some physical property of the binding pocket is important in ensuring that all conformations sampled during MD are represented in VS experiments. In choosing which clustering method to utilize, factors other than VS performance comparison should be considered.

In order to reduce computational cost and time, it is optimal to choose one clustering method; this decision may be made based on the advantages and limitations with each method. RMSD-based clustering is commonly used, and has proven successful in revealing conformations that enriched novel active compounds.^{21, 27, 81} However, degeneracy may be a major limitation. Two conformers with the same RMSD would be grouped into the same cluster, even though visual inspection may reveal key structural differences. To overcome this limitation, clustering on the volume and shape of the active site may be an attractive alternative. Also, it may be easier to visualize differences in active site shapes and volumes as opposed to changes in side chain positions.

Training Ensembles Enhances VS Performance

Training ensembles improved the early enrichment of actives. Although we saw a statistically significant increase in AUC with the trained ensembles, it is important to highlight that the crystal structures yielded high AUC values (>0.9 for 1 kinase, >0.8 for 3 kinases). Therefore, there was little room for improvement in the global VS performance. Based on the ROC-EF values of the crystal structure, which were further from the maximum value, there was greater potential to realize improved early enrichment.

While it is important to optimize the overall performance of the virtual screen (AUC), very often in drug discovery projects the ability of the VS method to enrich actives early in the ranking lists is an important feature. This is especially critical in cases where thousands or even millions of compounds are screened in silico, and only a small subset of compounds can be experimentally tested (i.e., in academic labs or limited resource settings, or for targets that do not yet have their assays ported to high throughput frameworks). In providing a top fraction of compounds for experimental testing, it is crucial to reduce the number of recommended false positives and enhance the hit rate. A boost in this early enrichment of actives is seen when ensembles are trained on ROC-EF. However, in the examples presented here, this performance boost is only statistically significant for MK2 and ABL, suggesting that the ROC-EF values of the trained ensembles are comparable to the crystal structure for the remaining four protein kinases at a false positive fraction of 0.001. At this false positive fraction, the total numbers of compounds enriched in the early ranking lists were 18, 20, 8, 22, 19, and 23 for MK2, CDK2, ROCK2, AKT1, IGF1R, and ABL kinases. To the best of our knowledge, there does not appear to be a general protocol established for choosing false positive fractions with early enrichment. Therefore, it may make sense to choose a false positive fraction based on the available resources for experimental testing. For example, if in future prospective studies we are able to screen 80–100 compounds, we can select an optimal ensemble that gives a higher ROC-EF at a later false positive fraction, 0.01 for example. If we look at the ROC-EF for CDK2, ROCK1, IGF1R, and AKT at a later false positive fraction of 0.01, there are trained ensembles with a statistically significantly higher ROC-EF than the crystal structure (Figures S9 and S10).

MD Simulations Reveal at Least One Better Predictive Conformation than the Crystal Structure

One hypothesis for why trained ensembles enhance the virtual screening performance over the crystal structure is because the MD simulation finds at least one single conformation that is more predictive than the crystal structure. Consistently, the optimal trained ensembles contained the single highest performing centroid for majority of the targets. It is interesting that the conformer that contributed the most to the optimal trained ensemble’s performance is never the crystal structure and always an MD centroid (Tables 2 and 3). These results align with the studies of Barril et al, who observed that ensemble member’s that performed well alone also did well in ensemble docking.¹⁶ Our results suggest that performing MD simulations on protein kinases may be worthwhile in virtual screens as one single MD conformation may significantly enhance virtual screening performance. While identifying a single high-performing MD conformation may be promising, identifying it requires docking to every MD snapshot or cluster centroid, which is limiting. It would be more straightforward to select the highest performing centroid a priori; however, there is little consensus about the target descriptors that would allow such a conformation to be identified. In an effort to address this, we explored whether there was a correlation between VS performance and the binding site volume, and found no correlation. Ellingson et al. investigated whether there was a correlation between several physicochemical and thermodynamics properties and descriptors of MD snapshots and enrichment factors.⁸² Some of these properties included the number of MD snapshots that make up the cluster, the trajectory frame number of the largest neighbor conformation that is used as the representative conformation, largest pairwise RMSD within the cluster, and the largest RMSD within the cluster to the representative conformation. Additional properties included the number of contact atoms in the binding site, propensity for ligand binding, number of hydrophobic contact atoms in the binding site, number of side chain contact atoms in the binding site, the number of contact residues in the binding site, the van der Waals (vdW) surface area, hydrophilic surface area, hydrophobic surface area, and vdW volume. However, this attempt to identify a meaningful feature set that could reliably recognize high performing MD snapshots was unsuccessful. Future studies that elucidate MD snapshot selection for enhanced VS performance will be groundbreaking and greatly enable the use of MD-generated ensemble docking approaches.

While MD simulations within this study revealed a single high performing conformation, NMR and X-ray crystal structures may have also reveal a single high performing conformer. Although, we did not explore the use of NMR or crystallographic ensembles, we speculate that the virtual screening performance would be similar to the results seen using MD-generated ensembles for different reasons.

NMR experiments provide an ensemble of protein conformations, where each conformer could be extracted for VS experiments. However, we suggest that training an ensemble of conformations obtained from NMR would yield similar VS performance as the MD-generated ensembles since both methods explore protein dynamics. Damm and Carlson make a similar conclusion in a study where they found that both MD and NMR captured similar structural variations of HIV-1 protease as indicated in their similar pharmacophore models.⁸³

While multiple crystallographic structures are not always available for proteins, this is not the case for protein kinases, with over 300 crystal structures for Cdk2 for example (Supporting Table S1). Therefore, all the available crystallographic structures for each protein kinase could be used to generate an ensemble to train for VS performance. It is in fact possible that the VS performance of crystal structure ensembles may differ from the MD-generated ensembles. However, since PCA analysis reveals an overlap between the binding site of the crystal structures and the MD conformations (Figure S11), we conjecture that the ensemble of crystal structures would perform similarly to the MD ensemble and limit our studies to MD-generated ensembles.

Smaller Ensemble Sizes Result in Optimal VS Performance

When looking at the performance values of ensembles trained to maximize either the AUC or ROC-EF, it is interesting to note that the optimal ensemble size is fairly small. For some protein kinases, adding more conformations either left the performance unaltered or degraded it. For example, if we look at the results of training AKT1 ensembles to maximize ROC-EF values (see Figure 4), the RMSD optimal ensemble was of size 3. For ensemble sizes 4, 5, and 6, the ROC-EF is the same as ensemble size 3; adding three additional MD centroids did not alter the early enrichment of actives. Furthermore, for ensemble sizes 7 and 8, the ROC-EF value decreases. Previous studies have shown similar results - the addition of more conformers can degrade performance.^{12, 14–16, 84–86} While ensemble docking studies, ours included, consistently highlight the importance of incorporating multiple protein conformations in VS experiments, that performance does not necessarily scale with ensemble size. Clearly, even though protein kinases may adopt multiple conformations, it appears that only a small number of conformations (1–4) are important for ligand binding, at least in the context of ensemble docking. The key to fully leveraging the power of ensemble docking will be developing strategies and protocols to find these crucial conformations, which remains an outstanding challenge to the field.

Structural Comparison between MD Conformer and Crystal Structure for MK2

To gain insight into why a MD cluster centroid may result in greater enrichment than the crystal structure, we analyzed key structural differences in an exemplar case. We determined several key structural differences that exist between the third RMSD centroid and the crystal structure of MK2. First, when we compare the actives enriched in the ranking lists at a false positive fraction of 0.001, we see that RMSD centroid 3 enriches an additional eight classes of actives that are not enriched by the crystal structure. One of these actives, CHEMBL272309, is unable to fit in the active site of the crystal structure, indicated by the steric clashes made with the crystal structure (Figure 5). Interactions between the active compound and the hinge region (Leu) and a C-lobe beta strand (Lys) are seen when bound to both the MD conformation and the crystal structure. However, three hydrogen-bonding interactions are lost with residues Asp, Glu, and Asn when bound to the crystal structure, which compounds the steric clash and greatly lowers the predicted binding affinity. Although the docked pose is shown for one active, the steric clash and sub-optimal-hydrogen-bonding network observed with the crystal structure is consistently observed for all of the actives uniquely enriched by the third RMSD centroid. These side-chain movements help explain the difference in early enrichment between the MD conformer and the crystal structure, and the subtle rearrangement underscores the difficulty of identifying a general set of structural features that reliably predicts conformations with strong classification ability.

Figure 5.

The docked pose of active compound, CHEMBL272309, reveals favorable interactions with RMSD centroid 3 (purple) and steric clashes (circled in left figure) with the crystal structure (pink). The compound makes H-bonding interactions with active site residues in the RMSD centroid 3, but interactions with the Glu, Asp, and Asn residues are lost when bound to the crystal structure. CHEMBL272309 is shown as licorice and is colored by atom type (C: gray, N: blue, O: red, S: yellow, H: white).

CONCLUSIONS

In this study, we analyzed the VS performance of MD structures against six protein kinases in three different kinome classes. We compared two clustering methods to determine whether clustering the RMSD values of active site heavy atoms resulted in a significant advantage over clustering the active-site shape. We compared these clustering methods to random selection of MD conformations. We also determined if ensembles of MD structures and crystal structures could be trained to optimize AUC and ROC-EF, and we analyzed their performance on a test set of compounds.

Results from this study suggest that running MD is worthwhile in conducting virtual screens against protein kinases, because it may result in that at least one conformation is more predictive than the crystal structure. Further, ensembles trained to maximize the AUC or ROC-EF can result in better performance than using the crystal structure alone. However, the extent of this enhancement is system dependent. For the majority of the protein kinases in this study, it does not seem to matter whether MD structures are selected using RMSD or POVME clustering. We find that the virtual screening performance differs between targets. This held true for members of the same protein class and for members of the same kinome class. The performance variability across targets implies that optimizing virtual screening protocols on a target-by-target basis is a reliable way to improve the likelihood of a successful prospective virtual screen.

Although the results presented are encouraging, they are limited to the six protein kinases within this study. Exploring larger datasets will ultimately lead to a greater understanding of the fundamental promise and limitations of applying ensemble docking to kinase drug discovery. Parallel studies within our group are focusing similar analysis on nuclear hormone receptors. Consistent with our conclusions here, the VS performance within the same nuclear hormone receptor class differs between each target, which supports the case for target-specific optimization prior to applying a VS method prospectively.

ABBREVIATIONS

SBVS

structure-based virtual screening

VS

virtual screening

MD

molecular dynamics

ROC

receiver-operating characteristic

AUC

area under the ROC curve

ROC-EF

ROC enrichment factor

DUD-E

directory of useful decoys enhanced

MK2

MAP-kinase-activated protein kinase 2

CDK2

cyclin-dependent protein kinase 2

ROCK1

rho-associated protein kinase 1

AKT1

serine/threonine-protein kinase AKT1

IGF1R

insulin-like growth factor 1-receptor

ABL

non-receptor tyrosine protein kinase ABL1