Improving the Speed of Volumetric Density Map Generation via Cubic Spline Interpolation

Abstract

Visualizing data generated from molecular dynamics simulations can be difficult, particularly when there can be thousands to millions of trajectory frames. The creation of a 3D grid of atomic density (i.e. a volumetric map) is one way to easily view the long-time average behavior of a system. One way to generate volumetric maps is by approximating each atom with a Gaussian function centered on that atom and spread over neighboring grid cells. However the calculation of the Gaussian function requires evaluation of the exponential function, which is computationally costly. Here we report on speeding up the calculation of volumetric maps from molecular dynamics trajectory data by replacing the expensive exponential function evaluation with an approximation using interpolating cubic splines. We also discuss the errors involved in this approximation, and recommend settings for volumetric map creation based on this.

Graphical Abstract

Introduction

Over the past decade, the time scales that molecular dynamics (MD) simulations can reach has increased dramatically, largely thanks to the introduction of GPU-accelerated MD software.[1–5] This in turn has vastly increased the amount of data that needs to be analyzed.[6,7] Therefore, it is desirable to improve the speed and efficiency of the tools used in analyzing MD trajectory data. One of the many common ways to analyze MD trajectory data is to calculate a three dimensional histogram (i.e. a volumetric map) from the atomic positional data over the entire trajectory. The resulting “density map” provides a straightforward way to observe atomic positions over long periods of simulation time. For example, creating a volumetric map of ion density can help identify regions of solute where ions tend to associate.[8] A volumetric map of water density can help identify regions where water behavior is significantly different than bulk behavior, which in turn is important for various types of water analysis tools such as SPAM[9].

The most straightforward way to construct volumetric maps from atomic positions is to calculate which “bin” (i.e. voxel) an atomic position falls into and then increment that bin, repeating for all atoms and all frames. However, the results from this approach are heavily dependent on the choice of bin size, and because the binning is discrete and dependent solely on the atomic position things like atom size are not taken into account (so e.g. a hydrogen atom is treated the same as a chlorine atom, though they have different sizes). An alternative method for constructing volumetric maps is to use kernel density estimation (KDE) with a Gaussian kernel to smoothly apply the density of each atom to each voxel of the map. This typically results in smoother density maps than straightforward “binning” that are more independent of voxel size. However, it is computationally more demanding, largely due to the number of calls to the exponential function.

Various software packages currently support constructing volumetric maps of atomic density via Gaussian KDE. MDanalysis was able to create volumetric maps from PDB data with the now deprecated ‘density_from_PDB()’ function.[10] CPPTRAJ[11] and VMD[12] can construct volumetric maps of atomic density average over trajectory frames. VMD also has fast and efficient code to generate volumetric maps using GPUs.[13,14] Briones et al. have developed a tool called GROmaps[15] which can be used to create volumetric map data from GROMACS[16] trajectories. The GROmaps tool uses a factorization of the Gaussian function[17] that reduces the number of calls to the exponential function to only 2 per spreading direction.

In this manuscript we show how calculation of the volumetric map via Gaussian KDE can be sped up by using cubic splines to interpolate the exponential (exp()) function. This idea has already been employed in MD simulations for decades in order to speed up calculation of the complimentary error function in particle mesh Ewald calculations.[18] We quantify the deviations of the resulting volumetric maps from those generated using the regular exponential function; the deviations are shown to be a function of both the voxel size and the table spacing used for the interpolating function. The speedup achieved is also shown to be a function of interpolating table size, which we show is related to cache performance. Finally, we make recommendations for optimal settings for grid size and table spacing when using interpolating cubic splines in this manner. The code has been implemented in the ‘volmap’ command of the MD software analysis program CPPTRAJ[11,19], freely available both on GitHub (https://github.com/Amber-MD/cpptraj) and as part of the AmberTools[20] software package.

Methods

Calculation of the Volumetric Map

Volumetric maps of atomic positions are constructed using Gaussian kernel density estimation for each atom to each voxel of the map:

$${\rho }_j={\sum _{i=1}^N\frac{1}{\sqrt{8{\pi }^3}\left({\alpha }_i^3\right)}e}^{-\frac{{\left|r_j-r_i\right|}^2}{2{\alpha }_i^2}}$$ Eq. 1

where ρ_j is the map density of voxel j, i refers to a specific atom, N is the number of atoms, r_j is the position of the voxel, r_i is the position of the atom, and α_i, determines the width of the Gaussian kernel so it is set to the half radius of atom i. In CPPTRAJ the radius used is the van der Waals radius if Lennard-Jones 6-12 parameters are available, otherwise radii based on the atomic elements are used.

This is done for each input frame. After all input frames have been processed, each voxel is divided by the number of input frames.

The magnitude of the Gaussian kernel decays rapidly as the distance from the atom increases, so it is possible to omit the contribution of a given atom to the map density beyond a certain cutoff, which greatly speeds up the calculation. The number of voxels V in a given dimension k direction (plus or minus X, Y, or Z) to apply the density for a given atom i is chosen via the following equation:

$$V_{i,k}=\mathrm{ceil}\left(\frac{S\ast {\alpha }_i}{d_k}\right)$$ Eq. 2

where ceil() is a function that rounds floating-point numbers up to the next integer, S is a scaling factor (by default 4.1 to be consistent with the VMD volmap command), α_i, is half the radius of atom i, and d is the grid spacing in dimension k (X, Y, or Z).

Calculation of Cubic Splines

Cubic spline interpolation is a way to approximate an arbitrary function via multiple 3^rd order polynomials (called splines) connected at points (called knots) that are usually spaced at regular intervals of size dx; the second derivatives of each polynomial is set to zero at each endpoint to ensure a continuous function. Each spline has the form:

$$Y_i(t)=a_i+b_{i}t+c_i{t}^2+d_i{t}^3$$ Eq. 3

where t is a parameter that ranges from 0 to dx, and a_i, b_i, c_i, and d_i are the so-called “spline coefficients” of spline i. Given an input set of n Y values along with corresponding X values, the values for a_i can be solved as:

$$a_i=Y_i\left(0\right)$$ Eq. 4

The remaining values for b_i, c_i, and d_i can be solved after arranging the equations into a symmetric tridiagonal matrix. Once all parameters are determined, any arbitrary value of Y at X value B within the original range of given X values can be evaluated using Eq. 3. The parameter t is determined using the following:

$$t=B-X_i$$ Eq. 5

where X_i, is the original X value of the knot i preceding the spline within which the value of B falls; i can be determined via:

$$i=\mathrm{int}\left(\frac{B-X_{Min}}{dx}\right)$$ Eq. 6

where int() is a function that rounds down to the nearest integer, and X_Min is the lowest knot X value.

Implementation and Testing

The new code using cubic splines to interpolate the exponential function in calculation of volumetric maps with Gaussian KDE was implemented in the ‘volmap’ command of CPPTRAJ, and will be available in versions greater than 4.31. The current ‘volmap’ implementation in CPPTRAJ does not take into account periodic boundary conditions or support non-orthogonal grids; support for both features is planned for a future release. The spline parameter table was set up so that it could handle all possible values that the exponential function in Eq. 1 might see:

$$M=\frac{-{{\sum }_{k=0}^2\left(V_{i,k}\ast d_k\right)}^2}{2{\alpha }_i^2}$$ Eq. 7

where k refers to dimension (e.g. 0 = X, 1 = Y, 2 = Z), i is the index of the atom with the largest radius, V_i,k is determined from Eq. 2, d_k is the spacing in dimension k, and α_i is the half the radius of the atom with the largest radius. The table is then set up using X values ranging from M (which is negative) to 1.0 in steps of a certain size (referred to as the spline table spacing). The max value used is 1.0 and not 0.0 (as might be expected since you cannot have value of M > 0.0) to avoid deviations in spline interpolation from the target exponential function due to Runge’s phenomenon, a well-known problem of oscillation in polynomial interpolation near the edges of an interval.[21] This is illustrated in Figure 1.

Figure 1.

Illustration of Runge’s phenomenon via deviation of exponential function interpolation for two different spline tables with the same knot spacing and function interval but different ranges: one from −40 to 0, the other from −40 to 1. Dashed X axis marks are aligned with the spline table knots.

The code was tested on trajectories from previously run (and unpublished) simulations of chain A of the enoyl reductase (FabI) from F. Tularensis (PDB ID 2JJY). Missing residues were modelled in via homology modelling; a compatible sequence (PDB ID 2PD3) was identified using SWISSMODEL.[22] The protein was parameterized with ff99SB,[23] the NAD with parameters from Walker and Pavelites et al,[24,25] and the triclosan was parameterized using GAFF[26] and AM1-BCC charges[27] via the antechamber program from AmberTools. The system was solvated with 15022 TIP3P[28] solvent molecules using a truncated octahedron unit cell for a total system size of 49,209 atoms. The system was minimized using the protocol from Roe and Brooks[29] and simulated for ^~39 ns in the NPT ensemble using PMEMD from Amber 11.[20] Simulations used a time step of 2 fs; coordinate trajectory frames were written every 1 ps. Bonds to hydrogen were constrained with SHAKE.[30] Long range electrostatics were handled using particle mesh Ewald with a cut off of 8.0 Å. Temperature was regulated at 300 K using a Langevin scheme with a collision frequency of 5 ps⁻¹. Pressure was regulated at 1 bar using isotropic scaling and a weak-coupling barostat[31] with a relaxation time of 5 ps. All other parameters were Amber defaults. The initial portion of the simulation was discarded as equilibration; depending on the grid size either the first 20 or first 100 frames from the last 2 ns (i.e. the last 2000 frames) of simulation time were used for testing. Except where noted, benchmarks were run using CPPTRAJ compiled with GNU 5.4.0 using optimized compile flags; the specific flags are reported in Supporting Information.

Results

There are two main factors that affect how well the volumetric maps generated via the spline interpolation scheme match those generated with the regular exponential function. The first is the number of splines used in interpolation; fewer splines means a larger distance between knots, hence more interpolation of values. The second is the grid spacing of the map; a smaller grid spacing means more voxels to populate for each given data point (as according to Eq. 2) and hence more calls to the exponential function/spline interpolation evaluation.

We determined how well the maps generated with interpolation match the maps originally generated with exponential function via the root mean square error (RMSE) over all voxels. All data structures used were full double-precision in order to minimize differences due to precision round-off. We tested various combinations of spline table spacing values (0.0002, 0.0004, 0.0006, 0.0008, 0.0010, 0.0025, 0.0050, 0.0075, 0.0100, 0.0500, and 0.1000 Å) and grid spacing values (0.2, 0.4, 0.5, 0.8, and 1.0 Å).

Figure 2 shows the spline table spacing vs volumetric map RMSE for various grid spacing values. Two general trends are apparent. One is that smaller grid spacing values lead to larger deviations, although this effect tends to plateau below a spline table spacing of about 0.001 Å. The second is that larger spline table spacing values lead to larger deviations above a certain point. However, it should be noted that all of the RMSE values reported here are on the order of or better than single-precision floating-point machine epsilon (1.19209x10⁻⁰⁷ according to the C++98 standard).

Figure 2.

Spline table spacing vs RMSE for different grid spacing values (0.2, 0.4, 0.5, 0.8, and 1.0 Å, full double-precision). Both axes are logarithmic. Each calculation was run for the same 100 MD trajectory frames except for 0.2 Å grid spacing which was run on 20 MD frames due to speed. Benchmarks run on an Intel Xeon E5-2630, CentOS 7.

Figure 3 shows spline table spacing vs speedup of spline interpolation over the regular exp() function for various grid spacing values. The speedup is most pronounced for a grid spacing of 0.2 Å, but most speedups are on the order of 2x. There is a slight but noticeable increase in speedup as the spline table spacing increases (and hence number of splines decreases).

Figure 3.

Spline table spacing vs speedup of interpolation code over exp() function for different grid spacing values (0.2, 0.4, 0.5, 0.8, and 1.0 Å, full double-precision). The X axis is logarithmic. Each calculation was run for the same 100 MD trajectory frames except for 0.2 Å grid spacing which was run on 20 MD frames due to speed. Benchmarks run on an Intel Xeon E5-2630, CentOS 7.

It should be noted that the grid spacing has the largest impact on speed. This is because the grid spacing determines the number of voxels over which the Gaussian function will be spread (see Eq. 7). Table 1 shows the calculation speed in terms of frames per second for the original and new code compared with spline interpolation for the same spline table size (spacing of 0.1 Å). The speed when using a very small grid spacing of 0.2 Å is extremely slow, below 0.1 FPS, and so not recommended for general use. It is due to this slow speed that the data reported for 0.2 Å spacing in Figure 2 and Figure 3 is calculated for 20 MD trajectory frames instead of 100 frames like the other grid spacing values. Note that for each increase in grid spacing there is roughly an order of magnitude increase in the frames per second achieved. However, for each grid spacing, the speedup of the new code over the old code is consistent, around 2x. Based on these results, it is recommended that a grid spacing of no lower than 0.5 Å be used for volumetric mapping, at least as implemented in CPPTRAJ. This grid spacing should be suitable for most applications, and is e.g. the recommended grid spacing for grid inhomogeneous solvation theory (GIST) as it is found to provide good convergence of local water density.[32]

Table 1.

Frames per second (FPS) achieved calculating volumetric maps using either the exponential function (exp()) or spline interpolation for various grid spacing values. The spline table spacing in each case is 0.1 Å. Frames per second are those reported by CPPTRAJ for the trajectory processing phase. Benchmarks run on an Intel Xeon E5-2630, CentOS 7.

Grid Spacing (Å)	Exp() FPS	Spline FPS	Speedup
0.2	0.0293	0.0699	2.39
0.4	0.8628	1.9028	2.21
0.5	3.2487	7.2142	2.22
0.8	44.5597	97.1281	2.18
1.0	137.6398	287.1657	2.09

In order to test the effect that different compilers have on speed up (due to e.g. different implementations of the exp() function, etc), runs were repeated for various compilers using a grid spacing of 0.5 Å and a spline table spacing of 0.01 Å. Runs were also performed using the “fast” exponential as suggested by Schraudolph,[33] as well as a “fast” exponential with a polynomial correction of degree 4 as suggested by Malossi et al..[34] The code for these “fast” exponentials was written by Simon Pfreundschuh and is available from https://github.com/simonpf/fastexp. The results are shown in Table 2. The specific compiler flags used and a plot of spline table spacing vs speedup/RMSE for each compiler are given in Supporting Information.

Table 2.

Performance of various exp() implementations for calculating volumetric maps (full double-precision) over 100 trajectory frames using a grid spacing of 0.5 Å and a spline table spacing of 0.01 Å for various compilers. Performance is shown for the base implementations of exp(), the spline interpolated versions, and the “fast” versions of the exp() function (with “Poly.” indicating a version with a polynomial correction). Benchmarks run on an Intel I5-2310, Ubuntu 16.

New Implementation	Base Exp() FPS	New FPS	Speedup	RMSE
GNU 5.4.0 Spline	3.8530	11.0049	2.86	4.498x10−13
Intel 19.1.0 Spline	6.2889	10.7803	1.71	4.498x10−13
PGI 18.10 Spline	3.7897	8.3001	2.19	4.498x10−13
Clang 3.8.0 Spline	4.0903	11.0103	2.69	4.498x10−13
GNU Fast Exp()	3.8530	10.6377	2.76	5.553x10−05
GNU Fast Exp() Poly.	3.8530	9.2015	2.39	5.134x10−09
Intel Fast Exp()	6.2889	10.8630	1.73	5.553x10−05
Intel Fast Exp() Poly.	6.2889	8.2815	1.32	5.134x10−09

Of the four compilers tested, the Intel compilers have the best base exp() performance; over 1.5x better than the other compilers. As a result, the spline interpolation does not result in as much speedup compared to the other compilers, although it is still a respectable 1.71 times faster for the settings tested. PGI compilers have both the poorest base exp() and spline interpolated performance. All of the other compilers are able to achieve about 11 FPS using spline interpolation. The volumetric map RMSE for all 4 compilers is identical, indicating the base exp() for each compiler is returning the same value within numerical precision.

The use of “fast” exponential functions results in a speedup of 1.32x to 2.76x. However, this speed comes at the price of accuracy, with the RMSE increasing from 4.5x10⁻¹³ for the spline versions to 5.5x10⁻⁰⁹ for the “fast” exponential with polynomial correction and 5 x10⁻⁰⁵ without the correction. Because of this, the spline interpolation seems overall a better alternative to the “fast” exponential function when calculating volumetric maps. Unlike the regular exp() function, changing the compiler from GNU to Intel for the “fast” exp() functions has no major effect on the speed.

In order to determine how the size of the spline table may be affecting the speedup achieved, we examined the cache behavior of the spline interpolation code directly by writing a separate stripped-down program containing only evaluations using the spline interpolation table class from CPPTRAJ. The code was tested for various values of spline table, using 50,000,000 randomly selected X values ranging from −40.0 to 0.0. Each spline table spacing value was run for 10 times to obtain some statistics. Cache performance was monitored with the linux ‘perf’ command. The cache was monitored for 3 different CPUs with different cache sizes. The number of cache misses for each spline table spacing using the stripped-down program compared to the speedup achieved in volumetric mapping with CPPTRAJ are shown in Figure 4. The cache behavior and speedup achieved appear to depend on the CPU cache size. For a relatively small cache size like 6 MB (Intel i5-2310), the number of cache misses drops as the spline table spacing increases (and therefore the number of splines/spline table size decreases), and the overall speedup increases. This behavior points to the smaller number of splines being able to fit into the cache easier, and fewer cache misses means faster lookup and therefore better speedup. The number of cache hits and misses levels off around a spline spacing of 0.01 Å (where the spline table takes up roughly 122 kB). However, for the 16 MB (Intel Xeon E5-2630) cache the decrease in the number of cache misses as spline table spacing increases is not as pronounced, and for the 20 MB cache (Intel Xeon 5120) the decrease is barely noticeable. Similarly the speedup for the larger cache sizes is not as noticeable.

Figure 4.

Top) Average cache misses for various spline table spacing values for 3 different CPUs. Average for each point is over 10 runs; error bars represent 1 standard deviation. Bottom) Speedup of spline interpolation over regular exp() when calculating volumetric maps for various spline table spacing values for 3 different CPUs (Xeons on CentOS 7, I5 on Ubuntu 16). The X axis for both top and bottom is logarithmic.

Although all calculations to this point have been done with full double-precision, the default in CPPTRAJ is to store the grid in single-precision in order to reduce the overall size of the grid. Figure 5 shows the RMSE and speedup for various spline table spacing values when the grid is stored in single-precision for 3 different CPUs. The RMSE is identical for all three processors tested. As expected from the double-precision results shown in Figure 2, the RMSE for all spline table spacing values is zero until the spacing is greater than 0.01 Å. Also as expected from the cache test results shown in Figure 4, the CPU with the smallest cache benefits greatly from a larger spline table spacing (and hence a smaller spline table size), reaching a speedup of about 3x. In all cases, at a speedup of 2x or greater is achieved.

Figure 5.

Top) Spline table spacing values vs RMSE for single-precision grid with a spacing of 0.5 Å. Bottom) Spline table spacing vs speedup of interpolation code over exp() function for single-precision grid with a spacing of 0.5 Å. The X axis for both top and bottom is logarithmic. Xeons on CentOS 7, 15 on Ubuntu 16.

Conclusions

We have increased the speed of calculating volumetric maps using Gaussian KDE from 3D atomic positional coordinates by using cubic spline interpolation to approximate the computationally expensive exponential (exp()) function. We find that use of spline interpolation results in a speedup of at least 2x over the original code using the exp() function. We tested a variety of spline table spacing values from 0.0002 to 0.1 Å, and found that for all values tested the RMSE of generated volumetric maps was less than 1x10⁻⁷, approximately equal to the machine epsilon for single-precision. We also find that there is a slight but measurable effect of spline table size on speed that appears to be due to cache utilization, indicating that there is a slight benefit to choosing a spline table size that can comfortably fit in cache. Based on the results, it seems using a spline table spacing of 0.01 Å provides good speedup (2-3x) for various CPU types while still maintaining low error from the original volumetric maps. We expect that this relatively easy-to-implement speedup of volumetric map calculation can be combined with other strategies for speeding up this calculation (such as factorization of the Gaussian calculation or tree-based optimizations to increase locality) to achieve even more speedup.

Highlights for “Improving the Speed of Volumetric Density Map Generation via Cubic Spline Interpolation”.

• The use of cubic spline interpolation can speed up generation of volumetric maps using Gaussian Kernels by 2 to 3 times.

• The speedup provided by spline interpolation is affected by the number of splines and available CPU cache.

• A spline table spacing of 0.01 Å for interpolation provides good speedup while maintaining better than single-precision fidelity to the original maps.