Boosting Free-Energy Perturbation Calculations with GPU-Accelerated NAMD

Abstract

Harnessing the power of graphics processing units (GPUs) to accelerate molecular dynamics (MD) simulations in the context of free-energy calculations has been a longstanding effort towards the development of versatile, high-performance MD engines. We report a new GPU-based implementation in NAMD of free-energy perturbation (FEP), one of the oldest, most popular importance-sampling approaches for the determination of free-energy differences that underlie alchemical transformations. Compared to the CPU implementation available since 2001 in NAMD, our benchmarks indicate that the new implementation of FEP in traditional GPU code is about four times faster, without any noticeable loss of accuracy, thereby paving the way towards more affordable free-energy calculations on large biological objects. Moreover, we have extended this new FEP implementation to a code path highly optimized for a single-GPU node, which proves to be up to nearly 30 times faster than the CPU implementation. Through optimized GPU performance, the present developments provide the community with a cost-effective solution for conducting FEP calculations. The new FEP-enabled code has been released with NAMD 3.0.

Graphical Abstract

Introduction

Free-energy calculations applied to molecular objects can be roughly categorized to (i) alchemical transformations, and (ii) geometrical transformations.^1–3 Used in conjunction with either molecular dynamics (MD) or Monte Carlo (MC) simulations, this approach has been broadly applied to study systems and processes of biological relevance, for instance to accelerate the process of drug discovery.^4–9 Free-energy perturbation (FEP)¹⁰ has been one of the earliest methodologies applied to transformations of an alchemical nature, whereby one chemical species is altered into another. The first concrete, chemically relevant application of FEP can be traced back to the mid-eighties and the determination of the relative hydration free energy of ethane and methanol.¹¹ Bolstered by early success stories,^12–16 FEP has subsequently been employed successfully to address a broad gamut of problems, including, albeit not limited to the prediction of protein-ligand binding affinities,^17–19 in-silico alanine scanning mutagenesis,^20,21 and the determination of solvation free energies.^4,22 FEP has been implemented in a variety of popular MD engines, e.g., GROMACS,²³ AMBER,²⁴ openMM²⁵ and NAMD.²⁶

In recent years, MD engines have leveraged the acceleration of graphics processing units (GPUs) and ported their core functionalities—in particular, the evaluation of bonded and nonbonded forces, from central processing units (CPUs) to GPUs, since the CPU-to-GPU transition results in a noteworthy gain in performance for a modest investment. To this date, however, GPU acceleration with spectacular speedups has primarily applied to equilibrium, unbiased simulations. Aside from the core functionalities of MD engines, a variety of importance-sampling schemes and free-energy methods, which include FEP, are still being adapted for GPU computation towards improved performance that would render these notoriously expensive calculations more affordable. Although FEP has been available in the GPU-accelerated version of AMBER,^27,28 openMM²⁹ and Desmond,³⁰ NAMD²⁶ still lacks the support of running FEP with GPU acceleration. In this contribution, the implementation of FEP in the GPU-accelerated NAMD is first outlined. The reliability and numerical accuracy of the implementation are then validated through four examples. We compare in performance benchmarks the FEP implementations in traditional GPU code, single-node GPU code, and the CPU code paths. Finally, we illustrate the usability of the GPU-accelerated FEP through a real-world biological process, namely an in-silico alanine-scanning mutagenesis of a peptide-protein complex.

Implementation details

Free-energy perturbation

Here, we first outline the guiding principles of the FEP theory, which can be traced back about hundred years ago.³¹ The free-energy difference, ΔA_0→1, that underlies the transformation between a reference state, “0”, and a target state, “1”, can be determined as,¹⁰

$$\Delta A_{0\to 1}=A{}_1-A_0=-\frac{1}{\beta }\text{In}{\langle \exp \left[-\beta \Delta U\left(\text{x}\right)\right]\rangle }_0.$$ (1)

Here, ΔU(x) = U₁(x) − U₀(x), where U₀(x) and U₁(x) are the potential energy functions of the reference and the target state of the transformation, respectively. The brackets,〈⋯〉 ₀, denote an ensemble average over configurations representative of the reference state, “0”. β = 1/k_BT, where k_B is the Boltzmann constant, and T is the temperature. Formally, the perturbation term, ΔU(x), in Eq. (1) ought to be replaced by the difference between the reference and the target Hamiltonians, but assuming that the mass is conserved during the transformation, the kinetic terms cancel out, and ΔA_0→1 corresponds to an excess Helmholtz free-energy difference.

In general, spontaneous transition between the reference state and the target state is rare and configurations of the latter are not sampled in finite-length simulation of the former, which is manifested in a slow convergence of the ensemble average. A simple strategy to circumvent the shortcomings associated to the direct application of Eq. (1) consists in introducing a series of nonphysical intermediate states to stratify the transition between the initial and the end physical states, and a general-extent, or coupling parameter, λ, connecting them. Assuming that the initial and the end states are connected by N−1 intermediate states and N windows or strata, the potential energy function of the i-th intermediate state can be written as,

$$U_{\lambda i}\left(\text{x}\right)=U\left(\text{x,}{\lambda }_i\right)={\lambda }_i{U}_1\left(\text{x}\right)+\left(1-{\lambda }_i\right)U_0\left(\text{x}\right)=U_0\left(\text{x}\right)+{\lambda }_i\Delta U\left(\text{x}\right)\forall 0\le {\lambda }_i\le 1.$$ (2)

Here, λ_i = i/N. Under these premises, the free-energy difference ΔA_0→1 is computed as a sum of free-energy differences over the N windows,

$$\Delta A_{0\to 1}=\sum _{i=1}^N\Delta A_{{\lambda }_{i-1}\to {\lambda }_i}=-\frac{1}{\beta }\sum _{i=1}^N\text{In}\langle \exp \left\{-\beta \left[U{\lambda }_i\left(\text{x}\right)-U{\lambda }_{i-1}\left(\text{x}\right)\right]\right\}\rangle {\lambda }_{i-1}.$$ (3)

In practice, a complete FEP workflow subsumes two stages, namely (I) evaluating the potential energy function at the different intermediate states, and (II) computing the ensemble average of Eq. (3), or the free-energy gradients required by the thermodynamic integration (TI) estimator.^32–34 To achieve stage (I), the MD engine must scale the potential energy function with respect to λ_i, following Eq. (2), which means that its different terms, i.e., the bonded contributions, the Lennard-Jones (LJ) potential, as well as the electrostatic potential become λ-dependent, and are scaled accordingly. Stage (II) requires bookkeeping of the exponential terms of the ensemble average in Eq. (3) to yield the intermediate free-energy differences, ΔA_λi−1→i, and possibly post-processing the data to obtain an alternate estimator, e.g., simple overlap sampling (SOS)³⁵ and Bennett acceptance ratio (BAR),³⁶ and its associated precision and accuracy—the latter step falling beyond the scope of the present article. Our novel implementation involves scaling of the potential energy function at stage (I), and output of the energy terms for computing either the FEP or TI ensemble average at stage (II). Although a TI estimator was also incorporated in our implementation, the discussion below focuses primarily on FEP, which is employed predominantly for computing the free-energy differences in alchemical transformations.

FEP implementation in GPU-enabled NAMD

The GPU-enabled version of NAMD organizes the computation of bonded, nonbonded LJ and electrostatic forces into compute objects according to an object-oriented design, and the data from compute objects pertaining to the same same C++ class are aggregated to schedule the workload to the CUDA kernels in a more effective fashion, which enables code reuse for the new GPU-resident FEP implementations. In NAMD, there are two code paths for executing the CUDA kernel functions,³⁷ namely (i) a traditional GPU code path, which utilizes the CHARM++³⁸ parallel programming framework and distributes the compute objects to the computational units, and (ii) the newly developed single-node optimized code path, which invokes the compute objects by direct function calls to bypass CHARM++, and incorporates the newly developed GPU-accelerated thermostats, barostats and propagation of motion to leverage the GPU in a more efficient way in equilibrium simulations.³⁷ In the discussion below, for the sake of clarity, code path (i) and (ii) will be referred to as “traditional GPU code” and “single-node code”, respectively. At this time, the single-node code is still under development, which is why we recommend the FEP-enabled traditional GPU code for production use, some of the popular features and modules of NAMD, like Colvars,³⁹ having not yet been ported to the single-node code. Furthermore, according to our benchmarks, the traditional GPU code path that leans on CHARM++ can be utilized for very large systems, consisting of several million atoms.³⁷

Our FEP implementation in CUDA kernels has been adapted to both code paths. The new implementation leans on the same dual-topology paradigm⁴⁰ employed by the CPU code,⁴¹ whereby the interaction of atoms of the reference state with those of the target state is ignored. All the potential energy function contributions, namely the bonded terms, as well as the nonbonded LJ and the long-range electrostatic interactions, are scaled according to the following rules:

Bonded interactions

Assuming that an alchemical transformation is performed at the i-th window, the bonded energies at the neighboring intermediate states, $U_{{\lambda }_i}^{\text{bonded}}\left(\text{x}\right)$ and $U_{{\lambda }_{i-1}}^{\text{bonded}}\left(\text{x}\right)$, which are obtained as a linear combination of the bonded energies of the reference and target states, are utilized to compute the energy difference required by Eq (3).

Nonbonded LJ potential

For each computed pair of atoms contained in the corresponding pair list, the kernel, i.e., the function executed on the GPU in NAMD, computes energy terms with a soft-core potential⁴² to avoid end-point catastrophes—i.e., singularities in the van der Waals potential due to particles appearing in a locus that is already occupied, and scales these terms accordingly with the corresponding value of the coupling parameter, λ, as,

$$\begin{array}{l}\Delta U_{{\lambda }_{i-1}\to {\lambda }_i}^{\text{LJ}}=U_{{\lambda }_i}^{\text{LJ}}\left(\text{x}\right)-U_{{\lambda }_{i-1}}^{\text{LJ}}\left(\text{x}\right)\\ U_{{\lambda }_k}^{\text{LJ}}\left(\text{x}\right)=\left(1-{\lambda }_k\right)\left[\frac{A_1}{{\left(r^2+\delta {\lambda }_k\right)}^6}-\frac{B_1}{{\left(r^2+\delta {\lambda }_k\right)}^3}\right]+{\lambda }_k\left[\frac{A_0}{{\left(r^2+\delta {\lambda }_k\right)}^6}-\frac{B_0}{{\left(r^2+\delta {\lambda }_k\right)}^3}\right]\\ \forall k=i,i-1,\end{array}$$ (4)

where r is the distance separating atomic pairs, A and B are the repulsive and attractive LJ parameters and δ is a shift parameter in the scale-shift soft-core potential function.⁴² The LJ parameters and the alchemical parameters, namely λ and δ, remain constant within a window, usually comprising thousands of MD steps. This behavior makes them good candidates for storage in the constant memory space instead of direct register storage, allowing for fast memory access through cached reads, and efficient broadcasting of the fetched values through individual threads in a warp. The kernel then calculates energies and applies the corresponding coupling parameter on-the-fly. The kernel is also capable of handling additional corner cases such as ignoring the scaling between interactions among atoms in the same state (decoupling), by adding only a slight additional computational cost during the energy evaluation.

Long-range electrostatic interactions

In the dual-topology paradigm,⁴⁰ the reference and target states coexist, albeit do not interact through either bonded or nonbonded forces, which implies that the long-range electrostatic interactions of appearing and vanishing atoms can be ignored. Consequently, in lieu of scaling the atomic charges and computing the long-range electrostatic forces on a single particle-mesh Ewald (PME)⁴³ grid, multiple grids are employed in our GPU-accelerated implementation of FEP. This strategy partitions the appearing, the vanishing and the unchanged atoms to different PME grids, and launches independent PME calculations over these grids. As shown in Fig. S1 in the Supporting Information, grid 0, referring to the reference state, contains the vanishing atoms and the unchanged ones, while grid 1, referring to the target state of the alchemical transformation, includes the appearing atoms and the unchanged atoms. The corresponding long-range electrostatic energies associated with the two grids, namely $U_0^{\text{elec}}\left(\text{x}\right)$ and $U_1^{\text{elec}}\left(\text{x}\right)$, are then computed using a PME solver. If the coupling parameter, λ, for electrostatic interactions is specified to change asynchronously with the one used for bonded interactions (NAMD option alchElecLambdaStart set to a value greater than 0), or if the intramolecular interactions of appearing or vanishing atoms are ignored (NAMD option alchDecouple is enabled), additional PME grids are required. A detailed description of the option alchDecouple can be found in the Supporting Information.

Simulation details

It is crucial to ensure that the computational accuracy of the new FEP-enabled NAMD GPU implementation is on par with that of the original CPU code. In order to validate the reliability and numerical accuracy of our FEP-enabled traditional GPU implementation, four prototypical applications, which can be found in the NAMD FEP tutorial,⁴⁴ have been simulated with both the new GPU and the original CPU implementations. These test cases include (i) the zero-sum ethane-to-ethane transformation in vacuum,^45,46 (ii) charging of an electrically neutral LJ particle into a sodium ion in water, (iii) alanine replacement of a tyrosine amino acid in the Ala-Tyr-Ala tripeptide in water,⁴⁷ and (iv) binding of a potassium ion to a crown ether.^48,49 In each application, we repeated both the forward (changing λ from 0 to 1) and backward (changing λ from 1 to 0) transformations five times, thereby providing a measure of the simulation accuracy. The reader is referred to the NAMD FEP tutorial⁴⁴ for all practical details of the simulations.

Increasing computational efficiency constitutes the main thrust of the present work. To evaluate the performance gain of our new implementation beyond paradigmatic computational assays, an ethane-to-ethane transformation was performed in explicit solvent (98,320 atoms), using the CPU code, the FEP-enabled traditional GPU code, and the single-node code, respectively. The benchmark was carried out on two different hardware environments: (i) a computer server with two 24-core Intel Xeon 8168 CPUs, 1.5 terabytes of memory and an NVIDIA Tesla V100 GPU, and (ii) a laptop with a quad-core Intel i7–7700HQ CPU, 16 gigabytes of memory, and an NVIDIA GTX1060 GPU. The side effect of increasing the number of PME grids imposed by the decoupling scheme that modifies λ associated with electrostatic interactions is also considered in the benchmark.

Aside from the aforementioned test cases, to illustrate the usefulness of the GPU-accelerated FEP implementations in real-world problems, an in-silico asparagine-to-alanine mutation was performed in the peptide-protein complex formed by the high-affinity peptide inhibitor (PMI) binding the ubiquitin ligase MDM2,⁵⁰ using the CPU code, the FEP-enabled traditional GPU code, and the single-node code on a workstation with an octa-core Intel i7–9800X CPU and an NVIDIA Quadro RTX 6000 GPU. The detail of the simulations can be found in Supporting Information.

Results and discussion

Reliability and numerical accuracy

The variation of the free energy as a function of λ, from 0 to 1, shown in Fig. 1 and Table S1–S8, was obtained by combining the different strata, using ParseFEP.⁵¹ The error bars in the figures represent the standard deviation associated with the free-energy difference in each stratified window. Comparing with the results of simulations carried out using the CPU code (Fig. 1A, C, E and G), the results of the FEP-enabled traditional GPU implementation (Fig. 1B, D, F, and H) do not reveal any conspicuous growth in error, and the free-energy differences obtained from λ = 0 to 1 with GPU acceleration closely match those of the standard CPU simulations. We therefore conclude that our FEP-enabled traditional GPU implementation possesses overall the same reliability and numerical accuracy as the CPU code in NAMD.

Figure 1:

Free-energy variation as a function of λ for the zero-sum ethane-to-ethane transformation in vacuum (A and B), the mutation of Ala-Tyr-Ala into Ala-Ala-Ala in water (C and D), charging of a naked LJ particle into a sodium ion in water (E and F), and binding of a potassium ion to a 18-crown-6 ether in water (G and H). In G and H, the curves marked as “complex” and “potassium“ denote the free-energy changes of the potassium ion in its bound state associated with the 18-crown-6 ether and its free state, respectively. The results obtained with the CPU code are shown in panels A, C, E and G, and those obtained with the FEP-enabled traditional GPU code are presented in panels B, D, F and H. The hysteresis between the forward and backward transformations and the associated errors shown in C and D are significant due to suboptimal simulation protocols.

Performance

The simulation times for ethane-to-ethane transformation in explicit solvent are shown in Fig. 2A and 2B. Our results indicate that the FEP-enabled traditional GPU implementation and the novel single-node code path can be as much as 4–12 times faster than the original CPU version of NAMD, even in the case where additional PME grids are required by a decoupled scaling scheme. Furthermore, the gain in performance can be observed on the different hardware environments, namely the laptop and the server. Moreover, considering that the retail price of the two CPUs (Intel Xeon Platinum 8168) and that of the GPU card (NVIDIA Tesla V100) used in our server are similar, our new implementation allows NAMD end-users to run much faster FEP calculations for the same cost.

Figure 2:

Comparison of code performance between the CPU implementation, the new FEP-enabled traditional GPU implementation, and the new, single-node implementation. Performance improvement of the GPU code paths is expressed as ratios to the simulation speeds with the CPU code path reported in parentheses. (A) Simulation speeds for the zero-sum, ethane-to-ethane transformation in explicit water run on a computer server. (B) Simulation speeds for the same transformation run on a laptop. In (A) and (B), the performances with a decoupled scaling scheme (additional PME grids) are shown with orange bars, while the ones with a coupled scaling scheme are shown with blue bars. (C) Simulation speeds for the asparagine-to-alanine mutation in both the bound state (PMI-MDM2 host-guest complex) and unbound state (PMI host only).

Asparagine-to-alanine mutation in PMI-MDM2

As shown in Fig. 2C and Table 1, offloading onto GPUs the computation hitherto done by CPUs results in a dramatic performance gain for this real-world application, without sacrificing numerical accuracy. It should be noted that the simulations described herein are provided as an illustration of the practical usability of our implementations, rather than benchmarks of the best protocol for optimal reproduction of experimental relative binding free energies,⁵⁰ which not only depends on sufficient sampling, but also, among other factors, on force-field accuracy. In the present example of an alanine mutation, the FEP-enabled traditional GPU code path results in an approximately eightfold speedup, compared with the CPU code, and the performance of the single-node version is increased by about 29 times with respect to the latter. This dramatic increase in the performance of the single-node version stems from the fact that the single-node code (i) bypasses a major part of CHARMM++ to avoid superfluous function calls on a single-GPU node, and (ii) offloads the integration of the equations of motion onto the GPU to reduce the operations of migrating data between the host memory and the GPU memory.

Table 1:

BAR free-energy estimators for the asparagine-to-alanine mutation in the bound and unbound states of the PMI-MDM2 complex and associated errors using three different code paths. The ΔΔG° value represents the relative binding free energy. The unit of ΔG^BAR and ΔΔG° is kcal/mol.

Code Path	$\Delta G_{\text{bound}}^{\text{BAR}}$	$\Delta G_{\text{unbound}}^{\text{BAR}}$	ΔΔG°
CPU	77.6 ± 0.1	78.0 ± 0.1	−0.4
GPU	77.8 ± 0.1	78.0 ± 0.1	−0.2
Single-node	77.6 ± 0.1	78.1 ± 0.1	−0.5
Experimenta			−1.1

^aThe experimental data is taken from reference 50.

Conclusion

In this article, we have reported new GPU-accelerated implementations of FEP, including the FEP-enabled traditional GPU and the single-node code paths, in the popular MD program NAMD,^26,37 and we have performed a series of numerical tests and benchmarks to probe the reliability and the performance of the code. Our benchmarks indicate that the present GPU implementations result in 4–30 fold acceleration compared to the original CPU version, without any loss of accuracy. Moreover, the resulting GPU acceleration can be achieved on both GPU-equipped laptops and dedicated GPU servers. Furthermore, the case of in-silico alanine replacement in the solvated PMI-MDM2 peptide-protein complex cogently illustrates the power and applicability of our new implementations in addressing biomolecular problems with utmost numerical efficiency. In summary, we have developed a robust, reliable and cost-effective workflow for performing FEP calculations in a time-bound fashion. Both the FEP-enabled traditional GPU and the single-node NAMD code paths have been released along with the 3.0 version of NAMD. Ongoing development will include porting additional FEP-specific features from the CPU code to GPU, e.g., the interleaved double-wide sampling⁵² and the Weeks-Chandler-Andersen decomposition,⁵³ and redesigning NAMD output of alchemical free-energy calculations to facilitate parallel running of independent windows.