Toward Fast and Accurate Binding Affinity Prediction with pmemdGTI: An Efficient Implementation of GPU-Accelerated Thermodynamic Integration

Abstract

We report the implementation of the thermodynamic integration method on the pmemd module of the AMBER 16 package on GPUs (pmemdGTI). The pmemdGTI code typically delivers over 2 orders of magnitude of speed-up relative to a single CPU core for the calculation of ligand–protein binding affinities with no statistically significant numerical differences and thus provides a powerful new tool for drug discovery applications.

Graphical abstract

Accurate prediction of the binding affinity between a drug candidate and the target protein is one of the biggest challenges for computer-aided drug design (CADD).^1-3 A broad range of methods exists for the prediction of protein–ligand binding affinities (henceforth referred to simply as ”binding affinities”) from molecular dynamics (MD) simulations, including fast empirical or semiempirical methods such as the linear interaction energy (LIE) model⁴ and its variants,⁵ the molecular mechanics (MM) combined with Poisson–Boltzmann or generalized Born solvation plus surface area correction (MM/PBSA and MM/GBSA),⁶ and more rigorous and time-consuming alchemical free energy methods.^7-10 Currently, a wide range of alchemical free energy methods provides the most robust and accurate estimates of binding affinities from molecular dynamics simulations.^3,7,11-15 These methods include Thermodynamic Integration (TI)^11,16-18 and Free Energy Perturbation (FEP),^11,19-22 as well as analysis through Bennett’s acceptance ratio and its variants (BAR/MBAR)^23-28 and are augmented with enhanced sampling such as replica exchange molecular dynamics,²⁹ metadynamics,³⁰ driven adiabatic free energy dynamics,³¹ orthogonal space random walk,³² adaptive integration,³³ and other methods.^34,35 Advanced alchemical free energy methods for binding affinity prediction^9,10,36 have evolved to the point that they approach quantitative predictive accuracy for ligand-binding affinities,^27,37,38 although much work remains, such as addressing sampling and convergence issues,^27,37 to improve precision so as to afford meaningful comparison with experimental uncertainties.³⁹

Nevertheless, the calculation of binding affinities through alchemical free energy simulations is computationally intensive, and often limitations in computing resources and/or required turn-around time for calculations render these methods impractical, particularly for industrial applications.⁴⁰ A promising solution is to develop free energy simulation software that can leverage affordable graphical processing units (GPUs)^10,38,41 to increase throughput, in some cases by more than 2 orders of magnitude.

GPU-accelerated molecular dynamics has been implemented in software packages such as NAMD,⁴² ACEMD,⁴³ AMBER,^44-46 OpenMM,⁴⁷ GROMACS,⁴⁸ and CHARMM⁴⁹ and has rapidly extended the time and spacial scales that MD simulation can reach.¹⁰ On the other hand, GPU-accelerated alchemical free energy methods have only very recently emerged in a few simulation codes. Recent implementation of the FEP method into the Desmond/GPU MD engine,⁵⁰ together with replica exchange with solute tempering,⁵¹ has demonstrated promising results for the accurate calculation of known binding affinities for a wide range of target classes and ligands.³⁸ The impact of these methods on making true predictions in blind tests remains to be fully determined.⁵²

We report a GPU-accelerated TI implementation of the AMBER 16 pmemd program^41,45,46 (henceforth referred to as pmemdGTI). Object-oriented programming concepts of encapsulation and inheritance were utilized to create a highly efficient, maintainable, and expandable code. We demonstrate that pmemdGTI can be used to explore simulation times scales up to 100 times longer, opening the door to more meaningful assessment of errors in free energy estimates, and more practical, reliable predictions.

Alchemical free energy methods such as TI and FEP/BAR have different advantages and disadvantages relating to the phase-space overlap of the quantities being averaged that may vary widely in different applications, and while these issues remain a topic of considerable discussion,^11,27,53-55 both classes of methods are widely used. In principle, the calculated quantities required by these methods appear similar, but they are not exactly the same: for each time step, TI only needs the derivative of the Hamiltonian with respect to the current λ_i; while FEP+BAR/MBAR needs the Hamiltonian values at the current λ_i and at neighboring λ_i+1 values as well as the exponential of the difference of the Hamiltonian values. As a result, their implementation on GPUs requires separate consideration in terms of storage and transfer of data, as well as the precision requirements in the averaging. For this reason, extension of the current implementation of the GPU-accelerated TI method to FEP/BAR is forthcoming.

1. IMPLEMENTATION

The software design and execution flow for the pmemdGTI code is illustrated in Figure 1, and addresses the following key implementation goals: 1) to meet the different precision requirements of the TI method relative to MD; 2) to maintain the same level of single-precision performance as in the current AMBER package; 3) to deliver the same level of accuracy for binding free energies as in the current AMBER CPU implementation of TI. To fulfill these goals, we utilized two architectural concepts of object-oriented programming: encapsulation and inheritance.⁵⁶

Figure 1.

Schematic view of the pmemdGTI software design and execution flow (GPU context is terminology referring to a GPU program unit representing the executing instance on the GPU.) Upper panel (software design): The original AMBER GPU context is packaged as a base class and contains all simulation data structures, including all coordinates, forces, energy terms, simulation parameters, and settings. The MD GPU simulation constant set is a collection of fast-access mathematical and physical constants and the pointers to the data structures of the GPU context and is packaged as a base class as well. Lower panel (execution flow): The CUDA multiple-stream capability is utilized, and the original AMBER MD code runs on a separate stream from the newly added TI code. Color codes: light green: the original AMBER modules; yellow: the add-on to encapsulate AMBER modules; light orange: the new TI add-on modules.

Encapsulation

Encapsulation is the concept to hide the internal functionality and data of a programming object from the outside. In our implementation, the original AMBER GPU data structures are encapsulated into base C++ classes. Here we use a GPU terminology, ”GPU context”, to refer to a GPU program unit representing the executing instance on the GPU. Shown in the upper panel of Figure 1, the original AMBER GPU context, which contains all simulation data structures, including all coordinates, forces, energy terms, all simulation parameters and settings, is now encapsulated as a base class. The MD GPU simulation constant set, a collection of fast-access mathematical and physical constants and the pointers to the data structures of the GPU context, is packaged as a base class as well.

Inheritance

Inheritance enables newly derived classes to receive, or inherit, the properties and methods of existing classes. In our GPU-TI implementation, new TI classes are derived from the base classes that contain the original GPU functionality and data structures for MD simulations. Since the TI classes inherit the base classes, they can be used just as the original base classes, and further, new functionality and data structures can be added without modifying the original GPU functionality and data structures.

Benefits

Through encapsulation and inheritance (upper panel of Figure 1), TI capability can be implemented so that 1) there is little or no need to modify the original MD GPU code, since all new add-ons can be implemented in the derived TI classes; 2) the new specific free energy algorithms and associated data structures are transparent to the base classes such that modifying or optimizing the base classes will have minimal effects on the derived classes; 3) derived TI classes can utilize, for example, different precision models and even different force fields if necessary.

Execution Flow

Shown in the lower panel of Figure 1, the execution of TI modules is implemented to execute on different CUDA streams (see the Supporting Information for CUDA multistream) and is independent from the execution of the original MD modules. The workflow of the particle mesh Ewald (PME⁵⁷) modules is shown in the Supporting Information as an example to demonstrate our implementation and to evaluate the contribution from two end point states in a TI calculation.

Precision Model

Two precision models were implemented in pmemdGTI, the SPFP (single precision evaluation/fixed precision accumulation) and the DPFP (double precision evaluation/fixed precision accumulation); both already utilized the pmemd.cuda code of AMBER 16.^45,46 Similar to the MD code, the performance of pmemdGTI with DPFP is significantly slower than SPFP (see the Results section). Since it has been shown that the GPU MD code with SPFP is sufficient for MD simulations to reproduce the results from the CPU MD code with the high precision DPDP (double precision evaluation/double precision accumulation) model,⁴⁶ there is no strong incentive to perform MD simulations with DPFP for production, although it is valuable for validation purposes in some cases. In this work, we report benchmark performance timings for pmemdGTI with both SPFP and DPFP, as well as the SPFP binding free energy results compared to the CPU results with the high precision DPDP model.

Development Platforms

The reported work was developed with Microsoft Visual C++ 2015, Intel Visual Fortran, Nvidia CUDA 7.5/8.0, and Nvidia’s Nsight on Windows 10 and with Eclipse IDE, GNU C++, GFortran, Nvidia CUDA 7.5/8.0, and Nvidia’s Nsight on Linux systems. The current version was tested on Windows 10 and various x86-based Linux systems, as well as on an IBM Minsky system with PowerPC8 CPUs and Tesla P100 GPUs. The current GPU implementation extends the GPU version of the pmemd MD code with TI calculations, hence CPU-only functionalities in pmemd are not supported with the current implementation. The current pmemdGTI code only runs on a single GPU.

Test Case Setup

In order to test our implementation, we selected an important well-studied benchmark protein-ligand system: the blood coagulation factor Xa protein with a series of ligands (Figure 2, right panel),⁵⁸ from which we recently reported an extensive study on the effect of ligand protonation and tautomerization state in the prediction of relative binding affinities.⁵⁹ Standard AMBER TI calculations were performed in the NVT ensemble, the same as in recently reported GPU benchmarks,^45,60 and a 1 fs integration time step at 300 K with PME (detailed parameters are described in the Supporting Information). Binding affinities were calculated using the TI method with 11 evenly spaced λ-windows ranging from 0.0 to 1.0. It has been suggested that a 2 fs time step can be used with covalent bonds to hydrogen atoms constrained, or else a 1 fs time step is recommended for MD simulations.^61,62 As there are no detailed studies of the impact of the bond constraint SHAKE algorithm on the soft-core implementation of alchemical free energy calculations, we disabled SHAKE for all TI mutations and consequently used 1 fs time steps for all reported simulations. For comparison, sets of softcore and nonsoftcore simulations with SHAKE enabled and 2 fs time steps were performed (see the Supporting Information).

Figure 2.

Binding affinities of various ligands with Factor Xa calculated from pmemdGTI (GPU) and pmemd (CPU). Ligands are from ref 59, and their structures are provided in the Supporting Information. Binding affinities were calculated with TI using 11 evenly spaced λ-windows ranging from 0.0 to 1.0 and integrated with Simpson’s rule. Listed are average ΔΔG’s values from 10 independent runs with 4 ns of data collection for each λ-window. The ΔΔG is the difference of ΔG_Complex and ΔG_Soln (see text). The softcore approach^63,64 was used for all TI calculations, except a → bt* where no softcore potential was used. All numbers are in kcal/mol. Left Panel: The plot of ΔΔG results from GPU (x-axis) and CPU (y-axis) with the associated standard errors as the error bars. The squared correlation coefficient (R²) between these two sets of data is 0.996. The dotted line is the ideal y = x regression line. Right Panel: The numerical values of ΔΔG and the associated standard errors (in parentheses), with the mutated chemical function groups shown with softcore atoms colored red.

Independent 5 ns TI simulations were performed for each λ-window, the last 4 ns of which were used for free energy analysis.

The chemical formulas of all compounds and mutations are listed in Figure 2. The compound ”bt” refers to the tautomer form of the compound b. The softcore approach^63,64 was utilized for both van der Waals and electrostatic interactions in all TI calculations, except a → bt* where no softcore potential was used (”*” denotes the simulation does not utilize softcore potential), which is to test the nonsoftcore implementation as this mutation is the only one that can be reasonably treated with nonsoftcore. The complex of Factor Xa and L51a ligand has total 41,563 atoms, including solvent molecules. The ligand L51a has 62 atoms, and other ligands have slightly different numbers of atoms as shown in their chemical formulas. In the TI simulations, the whole ligands were defined as the perturbed regions.

The free energy change of each mutation was calculated by numerical integration over all λ-windows using Simpson’s rule.^53,65 For all mutations, both the free energy changes of ligands in solution (ΔG_Soln) and ligands in complexes (ΔG_Complex) were calculated, and the relative binding affinities (ΔΔG) were obtained from the differences of free energy changes of ligands in complexes and in solution. Detailed simulation protocols can be found in the Supporting Information.

Simulations were carried out on both CPU and GPU platforms. While we previously reported CPU results derived from a single simulation run,⁵⁹ to obtain benchmark quality statistical sampling and error estimates (see the section below), in the present work each mutation was repeated 10 times with different initial velocities for both CPU and GPU versions. An extra set of 20 ns (for each λ-window) simulations of the a → bt* mutation was performed on the GPU. Various types of GPUs (K40, K80, M40, and P100) of the Nvidia’s testing cluster were utilized.

2. RESULTS AND DISCUSSION

2.1. Ensemble Average Results

There is an abundance of evidence that suggests an ”ensemble average approach”,^66-69 referring to performing an ensemble of independent simulations to calculate averaged free energies, is more effective than simply running a single long simulation for producing reliable, reproducible free energy results with meaningful error estimates.^70-72 Taking this approach, we have performed 10 independent TI simulation runs for each mutation (10 runs for each ligand mutation in solution, and 10 runs for each ligand mutation in the complex). The resulting error estimates and statistical analysis will allow assessment as to whether our TI implementation delivers the same results as the CPU version with proper statistical confidence.

The GPU Implementation Delivers Statistically Equivalent Results to That of the Current CPU Implementation

Figure 2 lists and plots 10-run averaged relative binding affinities for all mutations, along with the corresponding standard errors. We calculated the GPU results with the SPFP precision model and the CPU results with the default DPDP precision model. The 10-run average relative binding affinities from GPU and from CPU differ in the range between 0.01 and 0.2 kcal/mol. Based on the standard errors of these 10-run data (shown in the left panel of Figure 2), the unpaired t test p-values at the 95% confidence level (shown in the Supporting Information) of the null hypothesis ”GPU and CPU results are from the same distribution” are in the range of 0.11 to 0.92. Hence we conclude that there is no statistically significant difference between the GPU and CPU results.

The Ensemble Average Approach Is an Effective Way To Obtain Reproducible and Reliable Free Energies

Additional simulations were performed (not shown), and we found that the ensemble average relative binding affinities (ΔΔG) and their corresponding standard deviations are able to predict the results of new simulations, which confirmed the reproducibility of the calculated ΔΔG within the range of the estimated errors. We have performed 10-run 20 ns simulations of the a → bt mutation (detailed results in the Supporting Information). The standard deviation of binding affinity in the 10-run 20 ns simulations is similar to the 10-run 5 ns simulations, for both the ligand in solvent (0.010 vs 0.013 kcal/mol) and in the complex (0.162 vs 0.152 kcal/mol), consistent with recent TI studies.^39,72 This observation suggests that performing appropriate ensemble simulations may be more effective for convergence than a single long time TI simulation. In fact, ”converged” long time TI results from a single simulation may be misleading in some cases, as our results demonstrate that the 20 ns simulations manifest almost exactly the same ensemble deviation as seen in 5 ns simulations.

2.2. Timing Benchmarks

Timing benchmarks are shown in Table 1 on the simulation performance rates reported in terms of ns per day and the ratios between MD and TI simulation rates. The simulation system is for the “a → bt” mutation in Factor Xa. MD results are for L51a in Factor Xa (Figure 2, right panel), while TI results are for the whole ligand (62 atoms) defined as the TI region. Benchmarks were measured based on a 100 ps simulation period in the NVT ensemble using a 1 fs time step.

Table 1.

Timing Benchmarks (ns/day) for pmemdGTI Running on Various Nvidia GPUs^a

	MD	TI	TI/MD
GPU: pmemdGTI: SPFP
GTX 980 TI	53.31	38.20	0.72
GTX Titan X Pascal	66.10	43.28	0.65
Tesla K40	28.62	19.82	0.69
Tesla K80	31.22	22.74	0.73
Tesla M40	43.83	32.85	0.75
Tesla P40	58.99	43.04	0.73
Tesla P100	84.13	58.49	0.70
GPU: pmemdGTI: DPFP
GTX 980 TI	4.08	3.77	0.92
GTX Titan X Pascal	6.21	5.91	0.95
Tesla K40	9.74	7.91	0.81
Tesla K80	9.03	7.89	0.87
Tesla M40	3.42	3.19	0.93
Tesla P40	5.98	5.51	0.92
Tesla P100	37.61	30.73	0.82
CPU: pmemd.MPI
Xeon E5520 (8 cores)	2.07	1.50	0.72
Xeon E5-2620 (8 cores)	1.36	0.99	0.73
Xeon E5-2698 v3 (2 cores)	1.02	0.78	0.76
Xeon E5-2698 v3 (16 cores)	4.64	3.79	0.82
Xeon E5-2698 v3 (2 × 16 cores)	7.61	5.95	0.78

^aPerformance rates are shown in units of ns per day for standard MD simulation (MD) and TI simulation (TI), as well as the ratio (TI/MD). Benchmarks were measured based on 100 ps of simulation in the NVT ensemble using a 1 fs time step and PME electrostatics, as discussed in the text. SPFP (single precision/fixed precision) and DPFP (double precision/fixed precision) refer to the different precision models used in AMBER. The CPU version always runs with the default DPDP model (double precision/double precision). MD results are for L51a in Factor Xa; TI results (total 41563 atoms, including solvent) are for the L51a (62 atoms) to L51bt (62 atoms) mutation with the whole ligand defined as the TI region. See the Implementation section for further details. The $\frac{\mathrm{d}H}{\mathrm{d}\lambda }$ sampling frequency is set to 10 fs in order to ensure benchmark-level statistics. Preliminary tests indicate that the overhead associated with the $\frac{\mathrm{d}H}{\mathrm{d}\lambda }$ evaluation required for TI sampling, relative to the force-only requirement for each MD step, is about 30%, and increasing the stride for TI sampling to 1 ps only improves performance by around 3%.

The relative speeds of different GPUs and CPUs vary significantly (Table 1). In SPFP mode, the GPU performance ranges from 19.8 (K40) to 58.5 (P100) ns per day, whereas the fastest CPU (Xeon E5-2698) performance ranges from 0.78 (2-cores) to 5.95 (32-cores) ns per day. The speed-up of fastest single GPU (P100) to single CPU (Xeon E5-2698) core is 150X, whereas the speed-up for the slowest GPU (K40) to single CPU (Xeon E5-2620) core is 160X. Overall, pmemdGTI typically delivers over 2 orders of magnitude of speed-up relative to a single CPU core. Note that usually multiple CPU cores are utilized for simulations, thus the single CPU core timing estimates are merely used as a reference to establish normalized performance ratio comparisons.

For standard linear thermodynamic coupling, TI formally requires the evaluation of the Hamiltonians at both end points of the perturbed region. Avoiding explicit computation of the redundant energy and force terms, the CPU and GPU version (SPFP) both only need ~30% overhead for TI compared to plain MD. This 30% overhead derives mainly from the need to repeat the long-range PME calculations, even in the case that only a relatively small region is changing charges. Whereas the interpolation of the changed charges to the PME B-spline grid can be optimized, the cost of the fast Fourier transforms and interpolation of the potentials and forces back to all atoms results in the bulk of the computational overhead. The TI calculations of the nonelectrostatic nonbonded terms only account for less than 5% of time when SPFP is utilized. Hence a smaller TI region does not make a tremendous difference in reducing this overhead. When running with the DPFP precision model, where the GPU is much slower in calculating nonbonding terms and the time needed for repeating the long-range PME part becomes relatively smaller, the TI/MD ratio becomes higher as a result.

In short, our benchmark timing results demonstrate that currently available GPUs, when performing the reported TI free energy simulations, are able to deliver at least 2 orders of magnitude speed-up compared to a single CPU core for typical applications. The pmemdGTI code performs TI roughly at the speed of 70% of running an MD simulation with the fast SDFP precision mode, similar to the ratios seen in CPU versions.

It should be noted that the current implementation of pmemdGTI is not parallelized for multiple GPUs, whereas the CPU codes do run in parallel, and one can expect that significantly greater throughput could be obtained from use of a sufficiently high number of CPU cores. Nonetheless, typical free energy applications such as those presented here require, for each calculation, multiple windows (11 in the current work) to be examined and, for each window, multiple independent simulations (10 in the current work) to obtain converged averages and error estimates.^70-72 Further, for most pharmaceutical applications, predictions are typically required for an entire library of ligand compounds.⁵⁹ This leads to the need to perform many independent simulations (1,100 in the current work) which can be performed in parallel. Given that the parallel scaling performance of GPU MD codes is typically much worse than that for parallel CPU codes, it is usually overwhelmingly the case that the most practical and efficient strategy for high-throughput free energy predictions is to run multiple single-GPU TI simulations at the same time in parallel.

3. CONCLUSION

We report the implementation of the thermodynamic integration method based on the pmemd module of the AMBER 16 package on GPU platforms. The pmemdGTI code utilizes object-oriented programming principles so that the newly add-on codes are inherited from, not a direct extension of, the original pmemd GPU codes, resulting in a software implementation which is easy to maintain, test, and extend. Benchmarks with selected protein–ligand systems demonstrate that pmemdGTI is capable of delivering ligand–protein binding affinities with 2 orders of magnitude of speed-up relative to a single CPU core. The pmemdGTI code enables routine TI simulations to be performed efficiently on affordable GPU computing platforms and makes practical robust estimation of ligand binding affinities with reliable precision. This enabling technology advances the state of the art and provides a powerful tool for new and emerging drug discovery applications.

Software Availability

We plan to package pmemdGTI with the next official AMBER release (AMBER 18). The current version of pmemdGTI source code can be installed as a patch of AMBER16 for Unix platforms, as well as precompiled pmemdGTI binary for 64-bit Windows 7/8/10.