Greedy replica exchange algorithm for heterogeneous computing grids

Abstract

Replica exchange molecular dynamics (REMD) has become a valuable tool in studying complex biomolecular systems. However, its application on distributed computing grids is limited by the heterogeneity of this environment. In this study we propose a REMD implementation referred to as greedy REMD (gREMD) suitable for computations on heterogeneous grids. To decentralize replica management gREMD utilizes a precomputed schedule of exchange attempts between temperatures. Our comparison of gREMD against standard REMD suggests four main conclusions. First, gREMD accelerates grid REMD simulations by as much as 40%. Second, gREMD increases CPU utilization rates in grid REMD by up to 60%. Third, we argue that gREMD is expected to maintain approximately constant CPU utilization rates and simulation wall-clock times with the increase in the number of replicas. Finally, we show that gREMD correctly implements REMD algorithm and reproduces the conformational ensemble of a short peptide sampled in our previous standard REMD simulations. We believe that gREMD can find its place in large scale REMD simulations on heterogeneous computing grids.

Graphical Abstract

Standard replica exchange molecular dynamics (REMD) requires all replicas to complete prior to initiation of the replica exchange protocol. Greedy REMD decentralizes this process by using precomputed schedule of exchange attempts. Therefore, it only requires a replica and its predetermined exchange partner to have finished simulations prior to initiating replica exchange. Because greedy REMD reduces the idle time associated with replica exchange tasks, it becomes well suited for performing REMD on heterogeneous distributed computing environments.

1. Introduction

All-atom molecular dynamics (MD) simulations are a valuable tool in probing microscopic mechanisms of various biological processes. However, a common issue, which hinders the applicability and prediction power of MD, is limited conformational sampling caused by trapping of biomolecules in local free energy minima. As a result probability distributions of quantities computed from MD data are often inconsistent with those expected from equilibrium statistical mechanics. One of the most successful approaches addressing the sampling problem in all-atom MD has been the replica exchange method (REM), which was first introduced by Swendsen and Wang [1] and then, in the context of biomolecular simulations, by Hansmann [2] and by Sugita and Okamoto [3]. The original version of REM was designed to reproduce canonical (or NVT) ensembles in biomolecular systems. In these REM simulations the system is replicated R times and each replica is simulated independently at different temperatures. Periodically all R simulations are stopped and swaps of replicas adjacent along the temperature scale are attempted. The probability of accepting the exchange is derived from the detailed balance condition, which preserves the probability distribution of states at each temperature. The power of this method lies in the significant acceleration of sampling of the rugged free energy landscapes compared to conventional constant temperature simulations. In recent years, other versions of REM have been developed, which produce canonical ensembles by exchanging rescaling factors applied to vander-Waals interactions (Hamiltonian REMD) [4] or by considering more than one exchange parameter [5]. Moreover, REM methodology has been extended to reproduce other ensembles, such as isobaric-isothermal (NPT) REM [6] or surface tension (NPγT) REM [7]. Although REM is computationally expensive, it offers a possibility of probing equilibrium properties of complex biomolecular systems. In principle, REM can be applied either to Monte Carlo or molecular dynamics simulations performed in individual replicas. Because the latter is far more common, this paper considers replica exchange molecular dynamics (REMD) as an example of REM.

Straightforward implementations of REMD are based on synchronous execution of simulations in all R replicas, collecting and analyzing the final R replica states, computing exchange probabilities, swapping the replicas (over, e.g., temperatures), and restarting new R replica simulations. The described replica management strategy which we refer to as standard requires a master process (server), which centrally performs all of these tasks. Accordingly, the approach has two serious limitations. First, because replica simulations are launched sequentially, each replica must wait for its own turn in a launching queue leading to considerable delays in initiating the simulations of the last few replicas. Second, the underlying assumption in the standard REMD is that all replica simulations are performed on similar computing resources with similar speeds. Although this assumption is usually satisfied on dedicated Linux clusters, it is not valid if REMD simulations are performed on heterogeneous distributed computing nodes, such as those typically found on computational grids. As a rule these nodes have wide-ranging levels of performance and reliability. Furthermore, the compute nodes on public grids frequently operate on a cycle-stealing basis, where grid computations are only permitted to run when the machine is otherwise idle. Computations running on these nodes are then subject to interruptions causing their delays or forcing their restart on other nodes. In these cases the performance of REMD is limited by the slowest computing node on the grid assigned to simulations. The consequences are (i) slowdown in REMD simulations and (ii) low utilization of each processor tasked with executing replica simulations. As a result performance of REMD on heterogeneous computational grids is severely degraded compared to execution on local dedicated clusters.

It is important to emphasize that both limitations outlined above are exclusively related to the management of replica simulations. For example, the theory of REMD does not require that exchange attempts are performed synchronously for all selected replica pairs. This opportunity has been used to develop alternative replica management strategies. For instance, REMD implementation in NAMD [8] uses local replica management, in which pairs of replicas are attempted to exchange once they become available. This local management strategy utilizes MPI that also makes it unsuitable for public distributed grids. Consequently, REMD distributed computations require a different approach. For example, asynchronous REMD decouples the exchanges between different replica pairs by local management of their exchange attempts [9]. An advantage of asynchronous REMD, which is particularly important for grid-based simulations, is that it does not rely on MPI. Other studies have addressed the REMD synchronization problem by replica multiplexing [10], which increases the pool of replicas available for exchange attempt. A radically different approach was implemented in serial REMD [11, 12], which allows one to completely decouple replica simulations while still preserving REMD-like random walk over temperatures.

In this paper, we propose another local replica management scheme suitable for heterogeneous grids termed greedy REMD or gREMD, which is commercially available as part of the Parabon Frontier Grid Platform. gREMD exploits the fact that a pair of replicas can attempt an exchange independent of the states of other R−2 replicas. To utilize this circumstance we precompute the schedule of replica (more precisely, temperature) exchange attempts before initiating REMD simulations. Then, when a pair of replicas set for exchange attempt according to the schedule becomes available, the exchange is attempted irrespective of current states in other replicas. Similar to asynchronous REMD, gREMD breaks the synchronicity in the management of replica simulations. To illustrate application of gREMD we used the massively parallel grid computing architecture, Parabon Frontier (www.parabon.com). We show that greedy management of replica simulations offers significant wall-clock acceleration of standard REMD simulations and increases CPU utilization. We apply gREMD to study the equilibrium conformational ensemble of the C-terminal fragment of Aβ peptide and show that it is consistent with the ensemble probed by us earlier using standard REMD [13]. We conclude our article with the discussion comparing gREMD with other approaches in managing replica simulations.

2. Methods

2.1. Replica exchange molecular dynamics

Replica exchange molecular dynamics (REMD) [3] was developed to accelerate sampling of rugged free energy landscapes in biomolecules. REMD has been widely applied to study protein thermodynamics [7, 14–18] and has been extensively used by us to explore peptide conformational ensembles (for recent studies see [13, 19–23]). The main steps in REMD implementation are as follows. A simulation system is replicated R times and each replica i(= 1, .., R) is described by the state $x_m^i$, where the index m corresponds to the temperature T_m. In all, there are R unique temperatures, each corresponding to its own replica. Potential energy of a replica is E_pot(q), where q are atomic coordinates. Short canonical (NVT) MD simulations are performed in all replicas. Then, Monte Carlo simulations in the replica space attempt to exchange the replicas i and j in the states $x_m^i$ and $x_n^j$ as $x_m^i\to x_m^j$, $x_n^j\to x_n^i$. The probability ω of accepting the exchange follows from the detailed balance condition

$$\omega =\{\begin{array}{ll}1,\hfill & \Delta \le 0\hfill \\ e^{-\Delta },\hfill & \Delta >0\hfill \end{array}$$ (1)

where Δ = (β_m−β_n)(E_pot(q^j)−E_pot(qⁱ)) (β = 1/(k_BT)). In order to preserve the Maxwell-Boltzmann distribution at the temperature T_n the velocities v_i must be rescaled by the factor ^{$\sqrt{T_n/T_m}$}. Similarly, upon an exchange rescaling factor $\sqrt{T_m/T_n}$ should be applied to the velocities v_j. Velocity rescaling allows one to exclude kinetic energies from exchange probability ω (Eq. (1)). An application of exchange attempt detailed in Eq. (1) to all replica pairs constitutes a single iteration of REMD algorithm. As a result REMD implements a random walk of replicas over temperatures, which prevents their trapping in local energy minima. Although REMD breaks continuity in replica simulations by transferring replicas to different temperatures, it rigorously generates canonical (NVT) ensembles at each temperature. Therefore, REMD can be used for probing equilibrium properties of biomolecules.

2.2. Greedy REMD management

As described in the Introduction standard REMD management scheme assumes synchronous execution of all replica simulations and exchanges. In other words, this scheme has a constraint that all replicas must complete before any replicas can be exchanged and the next REMD iteration started (Fig. 1a). Given that there is a heterogeneous computing environment this constraint results in subpar utilization of compute nodes and, more importantly, in that REMD performance is limited by the slowest computing node. To bypass these limitations we propose greedy management of REMD simulations or gREMD (Fig. 1a). Its key requirement is a precomputed schedule of exchange attempts between replicas (more precisely, temperatures) for the entire length of REMD simulations. For R replicas there are two sets of temperature pairs adjacent along the temperature scale, (2m − 1, 2m) or (2m, 2m + 1), where m = 1, …, R/2. Typically, these sets are either alternated from one REMD iteration to the next or selected at each iteration at random. In any case, the occurrence of exchange pairs does not depend on the states of replicas and can be predetermined before REMD simulations are initiated. Consequently, gREMD first computes the schedule of replica exchange attempts and then starts replica iterations. When replica at temperature T_m completes simulations, gREMD checks the schedule to determine if its “partner” replica at temperature T_n (n = m±1 depending on the schedule) has also finished. If so, replicas at temperatures T_m and T_n are attempted to exchange and then the simulations in both replicas are restarted. If the simulation at temperature T_n has not yet finished, replica at T_m will wait for its completion. This schedule alleviates both limitations identified above, because the exchange attempt and restart of REMD simulations no longer depend on the completion of all replica simulations, but are determined by the states of only two replicas.

Fig. 1.

(a) Diagram illustrating the management of replicas in standard and greedy REMD. Green and red arrows represent MD simulations running for the time τ_sim and wait (idle) times τ_w, respectively. Vertical black lines mark exchange attempts. In standard REMD exchange attempts are applied to all replica pairs simulteneously, whereas gREMD localizes exchange attempts to individual replica pairs. (b) Probability distributions P(τ_it) of iteration times τ_it for cluster-based REMD (black), Mason grid REMD (red), and CAAD grid REMD (blue). The data were collected from standard REMD simulations of Aβ10–40 peptides binding to DMPC bilayer. Log scale for τ_it underscores the narrow distribution produced by cluster-based REMD and broad distributions associated with grid REMD. For clusters with a faster switch the standard deviation can even be smaller.

The pseudo code implementing gREMD strategy is given below (variables are italicized):

completionQueue = EMPTY
foreach replica in replicas do
     run task simulateReplica(replica)
done
while (replicas are running or completionQueue not empty) do
    replica1 = get next completed replica from completionQueue
    iteration = get current iteration for replica1
    schedule = get schedule for iteration
    replica2 = get partner replica for replica1 in schedule
    if replica2 is in completionQueue and has completed iteration then
          omega = get exchange probability for replica1 and replica2
          if omega meets acceptance criteria then
               exchange configurations for replica1 and replica2
          endif
          if iteration is not the last iteration then
               run task simulateReplica(replica1)
               run task simulateReplica(replica2)
          endif
      endif
endwhile
task procedure simulateReplica(replica) do
     perform simulation of replica (one replica iteration)
     add replica to completionQueue
enddo

In this pseudo code completionQueue is used for communication between replica simulation tasks and the scheduler. When simulation task completes a replica iteration, it puts the replica on completionQueue to notify the scheduler that the replica iteration has completed. The scheduler then takes replicas off the queue for scheduling. The pseudo code checks the status of all replicas to identify those which have finished simulations (replica1). Using current iteration and corresponding schedule schedule a replica replica2 scheduled for exchange attempt with replica1 is determined. If its simulation is completed, an exchange probability omega for replica1 and replica2 is computed and based on its value replicas attempt an exchange. Unless the current iteration is the last one, simulations in replica1 and replica2 are then restarted.

2.3. Computational model and simulation details

To test gREMD we have selected the C-terminal fragment of Aβ peptide, Aβ29–40. This system has been investigated in our previous study [13] and below we provide only its brief summary. Molecular dynamics simulations of Aβ29–40 were performed using all-atom explicit solvent CHARMM22 force field with CMAP corrections [24]. The simulation system consisted of a single Aβ29–40 monomer solvated with 1600 TIP3 water molecules in the unit cell with the dimensions 36.9Å × 36.9Å × 36.9Å. In all, the system included 4969 atoms. Simulations utilized periodic boundary conditions. Electrostatic interactions were computed using Ewald summation. Van-der-Waals interactions were smoothly switched off in the interval from 8 to 12Å. All covalent bonds were constrained by the SHAKE algorithm. In addition, we considered larger simulation system consisting of two Aβ10–40 peptides binding to DMPC lipid bilayer composed of 98 lipids in explicit water. This system studied by us earlier included 25,586 atoms and was simulated using the NPT ensemble [22, 23].

gREMD utilized two heterogeneous networks of computing nodes controlled by the Parabon Frontier server. The first grid referred to as Mason grid consisted of 22 Windows workstations with i5 and i7 quad-core processors located at George Mason University. The second grid referred to as CAAD grid consisted of the distributed nodes donated by volunteer users within the scope of the Compute Against Alzheimer’s Disease campaign (CAAD, www.computeagainstalzheimers.org). This grid included laptops, desktops, and servers running Windows, Linux, and Mac OS X. The node capabilities ranged from four-core workstations to 16-core Intel Xeon Servers. For benchmark purposes we used both grids and our in-house cluster to perform short gREMD and standard REMD runs. The latter were treated as controls.

To evaluate the accuracy of mapping the conformations of Aβ29–40 peptide we performed gREMD on Mason grid and compared its sampling against our previous REMD simulations on a dedicated 32 core AMD Opteron Linux workstation [13]. gREMD conformational sampling of Aβ29–40 peptide was performed using NVT replica exchange molecular dynamics [3] coupled with the NAMD program [8]. We have used R = 16 replicas distributed exponentially in the temperature range from 320 to 400K. Canonical ensembles in the replicas were produced by underdamped Langevin simulations of “virtual” solvent with the damping coefficient γ = 5ps⁻¹ and the integration step of 1 fs. Replica exchanges were attempted every 2 ps between all neighboring replicas along the temperature axis with the average acceptance rate of 36%. Using gREMD we have produced two trajectories collecting in total 0.64 μs of simulation time (40 ns per replica). Initial portions of REMD simulations in each replica must be discarded as non-equilibrated. Thus, the cumulative equilibrium simulation time was reduced to τ_eq ≈ 0.59μs. Implementation of gREMD for the large system (Aβ10–40 peptides binding to DMPC bilayer) followed our earlier studies [22, 23], including a modified exchange criterion in Eq. (1) for NPT ensemble.

Tertiary interactions were probed by finding contacts between amino acid side chains. A contact occurs, if the distance between the centers of mass of two side chains is less than 6.5Å. Secondary structure was computed using the STRIDE program [25]. An amino acid is assumed to be in a helical state, if it samples α-, 3₁₀-, or π-helices. Thermodynamic averages of structural quantities (denoted as < ‥ >) were computed using the multiple histogram method [26].

3. Results

3.1. Greedy replica management improves REMD performance

We begin the analysis of gREMD by presenting the results of its computational performance. We denote as τ_it the wall-clock time of completing one REMD iteration, which is the sum of two contributions τ_w and τ_sim, i.e., τ_it = τ_w +τ_sim (Fig. 1a). The wait time τ_w accounts for launching a replica simulation and for an idling period during which replica expects an exchange attempt. The simulation time τ_sim measures actual duration of MD simulations between exchange attempts. Fig. 1b shows the probability distributions P(τ_it) of iteration times τ_it collected from standard REMD simulations of large system (Aβ10–40 peptides binding to DMPC bilayer). An extremely narrow distribution of τ_it is characteristic of standard REMD simulations performed on dedicated Linux cluster. As expected for this homogeneous environment the standard deviation in τ_it, σ, is very small (8 s). However, standard REMD simulations performed on heterogeneous Mason and CAAD grids result in broad P(τ_it) distributions characterized by σ equal to 1103 and 1880 s, which are two orders of magnitude larger than for the cluster-based REMD. Then, one might expect that heterogeneous grids severely degrade the performance of standard REMD.

To test this expectation and quantify the performance of standard vs greedy REMD we obtained several benchmarks using two (small and large) simulation systems described in Methodology. For each combination of system and REMD algorithm we have performed 50 REMD iterations, which were used to compute $\overline{{\tau }_{it}}$ and $\overline{{\tau }_{sim}}$ as well as utilization rates $\gamma =\overline{{\tau }_{sim}/{\tau }_{it}}$, where bars indicate the averages over iterations. The results for small and large simulation systems are presented in Tables 1 and 2, respectively. It follows from Table 1 that the application of gREMD to cluster-based simulations does not reduce the time required to complete one iteration τ_it or impact γ. However, compared to standard REMD gREMD does reduce τ_it on Mason and CAAD grids by a factor of 1.3 and 1.5, respectively. Also, gREMD marginally improves utilization rates γ.

Table 1.

Benchmark results for Aβ29–40 peptide in water (small system)

Simulation	$\overline{{\tau }_{it}},s$	$\overline{{\tau }_{sim}},s$	γ
cluster-based REMD	136	127	0.93
cluster-based gREMD	131	127	0.97
Mason grid REMD	565	57	0.12
Mason grid gREMD	435	56	0.19
CAAD grid REMD	614	71	0.14
CAAD grid gREMD	420	73	0.22

Table 2.

Benchmark results for Aβ10–40 peptides binding to the lipid bilayer (large system)

Simulation	$\overline{{\tau }_{it}},s$	$\overline{{\tau }_{sim}},s$	γ
cluster-based REMD	463	420	0.91
cluster-based gREMD	452	420	0.95
Mason grid REMD	1361	304	0.27
Mason grid gREMD	782	305	0.43
CAAD grid REMD	1713	355	0.26
CAAD grid gREMD	1026	362	0.42

Table 2 refers to the simulation system, which is more than five time larger than that considered in Table 1. Similar to Table 1 it shows that gREMD does not affect iteration times or utilization rates in cluster-based simulations. In contrast, when applied to Mason and CAAD grids gREMD accelerates simulations by a factor of 1.7. Simultaneously, gREMD increases the utilization rates from about 0.26 to 0.43. It is important to note that Tables 1 and 2 demonstrate that grid-based simulations are slower than cluster-based, even though the former execute MD simulation faster as measured by τ_sim. This observation is related to relatively low γ values in Tables 1 and 2 for grid-based REMD. In the Discussion we argue that the primary reason for low utilization rates is large replica management overheads associated with REMD simulations performed on distributed networks of computing nodes. There we also examine possible ways of reducing the overheads.

Therefore, taken together Tables 1 and 2 imply that for grid simulations gREMD accelerates run-time execution and increases utilization rate. Importantly, comparison of Tables 1 and 2 indicates that the benefits of gREMD become more apparent with the increase in system size.

3.2. Greedy management correctly implements REMD algorithm

It is important to check that greedy management correctly implements REMD algorithm. To this end, we compared gREMD simulations of Aβ29–40 fragment in water (small system, see Methodology) with our earlier simulations of the same system performed using standard REMD [13]. First, we tested the sampling of Aβ29–40 conformational states. To quantify this process we consider the number N_s of unique states (E_p, H) sampled at least once in the course of simulations. Each state (E_p, H) is defined by the potential energy E_p and the fraction of helical residues, H. Aβ29–40 fragment contains 12 residues and, therefore, H has 13 distinct values. To distinguish the states (E_p, H) collected from all replicas potential energy was discretized using the energy interval of 1 kcal/mol in the range from −15450 to −13682 kcal/mol. Then Aβ29–40 peptide can adopt up to 22,984 distinct states. However, the actual number of accessible unique (E_p, H) states is much lower, because many combinations of E_p and H are incompatible. In Fig. 2a we plot N_s as a function of the equilibrium simulation time τ_eq accumulated in gREMD. As gREMD simulations surpass τ_eq ≈ 0.3μs the growth in N_s dramatically slows down indicating the onset of exhaustion of new states. Importantly, Fig. 2a compares N_s computed using greedy and standard REMD [13]. It is clear that the collection of new conformational states (E_p, H) in both REMD implementations is very similar. (It should be noted that small differences in the N_s baselines reflect independent stochastic sampling performed by REMD and gREMD.) In our previous study dramatic slowdown in the growth of N_s was taken as an indicator of convergence of REMD simulations [13].

Fig. 2.

Testing the implementation of replica exchange algorithm by gREMD. (a) The number N_s of unique states (E_p, H) sampled at least once in the course of simulations as a function of equilibrium simulation time τ_eq for standard REMD (black) and greedy REMD (red). Leveling off of N_s growth over time indicates that simulation systems are converging. (b) Random walk of individual replicas over temperatures in gREMD. Colors in the right scale represent the instantaneous distribution of replicas over temperatures at the beginning of gREMD trajectory. (c) The replica mixing parameter m(T) computed as a function of REMD temperature T using standard REMD (black) and gREMD (red) trajectories. The dashed line marks optimum theoretical value of m(T) = 0.75. Nearly constant values of m(T) across temperatures for both standard and greedy REMD indicate efficient replica mixing. The data in both panels represent Aβ29–40 system.

In REMD simulations individual replicas are expected to execute a random walk over temperatures. Using gREMD probing Aβ29–40 (small) system Fig. 2b checks this expectation by presenting a heatmap illustrating the distributions of replicas over temperatures as a function of REMD iterations. As expected Fig. 2b demonstrates rapid random mixing of replicas over temperatures and shows no evidence of their trapping at any of REMD temperatures. A similar heatmap has been obtained for the second gREMD trajectory generated for Aβ29–40 system. To provide a quantitative measure of replica visits to individual temperatures we used the replica mixing parameter m(T) [27],

$$m(T)=1-\frac{\sqrt{{\sum }_{r=1}^R{t}_r^2}}{{\sum }_{r=1}^R{t}_r},$$ (2)

where t_r is the total number of REMD iterations spent by the replica r at the REMD temperature T. The quantity m(T) is designed to measure equal participation of all replicas at a given temperature T. For example, if all R=16 replicas are equally represented at the temperature T, then m(T) reaches the maximum theoretical value of $1-1/\sqrt{R}=0.75$. Approach of computed m(T) to the maximum theoretical value can be taken as an evidence of correct gREMD performance. The function m(T) computed from gREMD simulations is shown in Fig. 2c, where it is compared to m(T) obtained from standard REMD [13]. Two conclusions follow from this figure. First, m(T) computed for gREMD is constant and approximately equal to the maximum theoretical value of 0.75 at all REMD temperatures. Therefore, Fig. 2c indicates efficient mixing of replicas over temperatures in gREMD. Second, similar to Fig. 2a the performance of gREMD measured by m(T) is virtually identical to that of standard REMD. Taken together, comparisons presented in Fig. 2 suggest that gREMD correctly implements REMD algorithm and its conformational sampling is as exhaustive as in standard REMD.

3.3. gREMD reproduces conformational ensemble of Aβ29–40 peptide

In this section we test the ability of gREMD to reproduce equilibrium conformational ensembles. As a test system we again use Aβ29–40 peptide fragment solvated in water (see Methodology). We first evaluate Aβ29–40 secondary structure sampled by gREMD. Our previous study has shown that this fragment adopts mostly helical structure at 330K [13]. Consequently, Fig. 3a compares the fractions of helical structure < H(i) > formed by amino acids i computed using greedy and standard REMD. It follows from this plot that for all i the differences between the two < H(i) > are within or close to sampling errors. Indeed, normalized root mean square deviation (nRMSD) measuring the difference in helical propensities along the sequence between greedy and standard REMD is equal to 0.04. To put this result in perspective, we computed nRMSD between the helical propensities < H(i) > in Aβ29–40 and in the equivalent C-terminal region of Aβ10–40, which forms random coil [13]. We found this nRMSD to be equal to 0.49, which is more than 10 times larger than that comparing greedy and standard REMD. The average helical fraction < H > obtained from gREMD is 0.51 ± 0.02, which matches exactly < H >= 0.51 ± 0.02 computed from standard REMD [13]. Using gREMD we have also determined the fractions of turn structure < T >= 0.16 ± 0.02 and random coil < RC >= 0.32 ± 0.02. These values are in excellent agreement with those computed from standard REMD in our previous study (< T >= 0.17 ± 0.01 and < RC >= 0.33 ± 0.02). The nRMSD values for turn and coil structure distributions are 0.12 and 0.03, which are small with respect to the nRMSD values comparing Aβ29–40 and the C-terminal of Aβ10–40 (0.50 and 0.23). Therefore, consistent with previous simulations [13] gREMD predicts mostly helical conformational ensemble for Aβ29–40 peptide.

Fig. 3.

Comparing the conformational sampling generated by gREMD and standard REMD. (a) Fractions of helix structure < H(i) > for each amino acid i for standard REMD (black) and gREMD (red). Errors are presented as vertical bars. (b) The difference contact map < ΔC(i, j) >=< C(i, j) >_s − < C(i, j) >_g for amino acid pairs i and j, where < C(i, j) >_s and < C(i, j) >_g are contact maps computed using standard and greedy REMD, respectively. (c) Probability distributions P(r_1N) of end-to-end distance r_1N for standard REMD (black) and gREMD (red). The data in all panels represent Aβ29–40 system.

To examine tertiary structure in Aβ29–40 we consider the contact map < C(i, j) >, which reports the probabilities of forming side chain contacts between amino acids i and j. For comparison, we define the difference contact map < ΔC(i, j) >=< C(i, j) >_s − < C(i, j) >_g, where < C(i, j) >_s and < C(i, j) >_g are computed using standard and greedy REMD, respectively. Fig. 3b presents < ΔC(i, j) >, which reveals close similarity between the two simulations. The value of nRMSD measuring the differences between the contact maps is just 0.04. For comparison, the nRMSD computed between the contact maps of Aβ29–40 and the C-terminal region of Aβ10–40 is 0.17 [13]. For almost all contacts (i, j) the difference in contact probabilities | < ΔC(i, j) > | does not exceed 0.06. The exceptions are the five contacts, Leu34–Val39 (−0.12), Met35-Val40 (−0.10), Val36–Val39 (0.08), Leu34–Val40 (0.08), and Val36–Gly38 (0.07). All of these contacts occur near the C-terminal indicating minor difference in the tertiary structure in this region that we attribute to limited conformational sampling.

The third comparison of Aβ29–40 conformational ensembles produced by greedy and standard REMD utilizes the end-to-end distance r_1N. Fig. 3c displays two probability distributions P(r_1N) determined from greedy and standard REMD. The figure demonstrates a good agreement between them, which is reflected in the small value of respective nRMSD (0.03). For comparison, the nRMSD probing the differences in P(r_1N) obtained for Aβ29–40 and the C-terminal region of Aβ10–40 is three-fold larger (0.09) [13]. We have also found from Fig. 3c that the average values < r_1N > computed using greedy and standard REMD are equal within the sampling error (14.9 ± 0.2Å versus 15.0 ± 0.7Å, respectively). Thus, the analysis of three measures, which probe secondary and tertiary structures, suggests that Aβ29–40 conformational ensembles sampled by greedy and standard REMD are consistent.

4. Discussion

4.1. gREMD is a useful implementation of REMD algorithm for grid-based simulations

REMD simulations are an increasingly important tool in studying complex biomolecular systems. Nevertheless, they remain expensive in terms of required computational resources that limits REMD applicability. A relatively cheap alternative to REMD performed on dedicated compute clusters is grid-based REMD simulations utilizing networks of computing nodes, such as, Windows PCs in university classrooms or labs. However, due to intrinsic heterogeneity of such networks the times of completing one REMD iteration, τ_it, show very broad distributions P(τ_it) compared to those observed in cluster-based simulations. This circumstance is dramatically illustrated in Fig. 1b revealing that the standard deviations in P(τ_it) characterizing standard REMD on heterogeneous Mason and CAAD grids are about two orders of magnitude larger than that obtained for cluster-based REMD. Because standard REMD management requires synchronous execution and completion of all replicas before advancement to next iteration, standard REMD simulations are limited by the slowest node and may not be efficient on a heterogenous network. Indeed, Tables 1 and 2 demonstrate that standard REMD performs three to four times slower on a grid than on a dedicated cluster, even though individual grid nodes are typically faster than cluster ones.

Recognizing the necessity to improve REMD performance on computing grids, we propose a new scheme managing replicas in REMD simulations, which we termed greedy REMD or gREMD (Fig. 1a). From theoretical perspective REMD does not require synchronous replica execution and exchanges as the outcome of exchange in replica pair solely depends on their states. gREMD exploits this circumstance and precomputes the schedule of replica exchange attempts ahead of REMD simulations. Once a pair of replicas at the temperatures set a priori for exchange attempt by the schedule becomes available, gREMD attempts an exchange irrespective of states of other replicas. Consequently, gREMD breaks the synchronicity in the management of replica simulations.

Our analysis of Tables 1 and 2 revealed that depending on the system size gREMD accelerates grid-based REMD by factors of 1.3 to 1.7. Importantly, the largest speed-up factor corresponds to the large simulation system involving Aβ peptides interacting with the lipid bilayer. Furthermore, gREMD boosts utilization rates γ for the large simulation system. However, Tables 1 and 2 also revealed that γ for grid-based REMD are significantly lower than those observed for cluster-based simulations. One can argue that low γ values are due to inherent variability of iteration times τ_it produced by the heterogeneous grid environment and/or are the consequence of the overhead associated with a grid replica management. To evaluate the two possibilities and to gain better insight into gREMD performance, we have developed a simple program, which emulates execution of standard REMD and gREMD. This code adds individual wait and simulation times, τ_w and τ_sim, to the total time accumulated by each temperature applying the rules taken from standard REMD and gREMD. As a test using this emulator with individual τ_w and τ_sim taken from the actual distributions collected from Mason and CAAD grids we were able to reproduce the data in Tables 1 and 2.

According to our analysis the distribution of τ_sim obtained from grid-based simulations is relatively narrow with the standard deviation, which is about two orders of magnitude smaller than that characterizing the distribution of τ_w. Accordingly, we focus on the impact of τ_w variations on utilization rates γ. To this end, we used the emulation code and applied the following approach. The observed wait times τ_w were scaled according to ${\tau }_w(\Delta )=\overline{{\tau }_w}+\Delta \left({\tau }_w-\overline{{\tau }_w}\right)$, where bars denote an average over the observed distribution. By varying Δ we can control the contribution of dispersion in the distribution of τ_w. When Δ = 1 we recover the observed distribution of wait times, whereas Δ = 0 completely eliminates variations in τ_w across iterations. Fig. 4a shows the dependence of utilization rate on Δ, γ(Δ), for Mason grid gREMD and standard REMD. First, this figure demonstrates that for all Δ values gREMD provides better utilization rate compared to standard REMD. Second, γ reveals weak dependence on the heterogeneity of τ_w distribution as γ at Δ = 0 is just 10% lower than γ at Δ = 1. (A slight decrease in γ as Δ approaches 0 is due to a negative skew in τ_w distribution. For instance, the times τ_w for Mason grid gREMD are split almost 60% and 40% to the left and right of the mean $\overline{{\tau }_w}$. A similar trend is seen in τ_w obtained for Mason grid REMD.) Note that γ(0) for gREMD is still higher than for standard REMD, because in standard REMD τ_w at each iteration is R/2 times longer than τ_w in gREMD. Thus, complete elimination of τ_w variations across iterations does not improve utilization rates and we conclude that low values of γ in Tables 1 and 2 compared to cluster-based REMD are primarely determined by the grid replica management overheads expressed in large τ_w.

Fig. 4.

Probing theoretical gREMD performance for large simulation system of Aβ10–40 peptides binding to the lipid bilayer [22, 23]. (a) Dependence of utilization rate γ on the scaling factor Δ for standard REMD (black) and gREMD (red), where Δ scales the contribution of heterogeneity in τ_w to γ. Dashed lines mark the observed utilization rates in standard REMD or gREMD. (b) Dependence of utilization rate γ on the scaling factor Δ for standard REMD (black) and gREMD (red), where Δ scales the contribution of network communications to γ. Dashed lines mark the observed utilization rates in standard REMD or gREMD. (c) Dependence of utilization rate γ on the number of replicas R for standard REMD (black) and gREMD (red). Minor increase in γ at small R in gREMD is due to the contribution of first and last replicas, which participate less frequently in exchanges. Better utilization rates in standard REMD compared to gREMD at small R are related to the overhead produced by executing the code with greedy management. Panels (a-c) use actual τ_sim and τ_w times observed on Mason grid using either standard REMD or gREMD.

To a large extent, the management overhead is related to the transfer of data over networks and simulation scheduling. In principle, τ_w can be decomposed into the wait time associated with replica scheduling, τ_w,sch, and the wait time associated with data transfer between master and distributed nodes, τ_w,net. For the Mason grid executing standard REMD for large simulation system the average τ_w,sch and τ_w,net are 614 and 443 s, respectively. The same quantities for the Mason grid executing gREMD are 106 and 370 s. As expected τ_w,sch in gREMD is much smaller than in standard REMD. These data also indicate that τ_w,net is a significant contributor to the total τ_w for both the standard (42%) and greedy (78%) algorithms. Complete elimination of τ_w,net for Mason grid simulations would emulate an environment closely related to a cluster, where τ_w,net ≈ 0. To explore this further, we defined a scaled τ_w(Δ) = τ_w,sch + Δτ_w,net and plotted γ as a function of the scaling factor Δ in Fig. 4b. A value of Δ = 1 reproduces γ computed in Table 2. As Δ is decreased γ increases to 0.45 and 0.84 at Δ = 0 for the standard and greedy algorithms, respectively. These results demonstrate that grid gREMD performance may approach that of cluster simulations, if the data transfer over network is reduced to minimum. A possible solution to reduce τ_w,net would be to use local Frontier server installed on the same network as compute nodes that permits us to limit the data exchange between replicas to temperature values.

Using our emulator code it is also useful to examine the dependence of utilization rates in standard and greedy REMD on the number of replicas R. Fig. 4c presents γ(R), in which the distributions of τ_w and τ_sim were taken from Mason grid simulations with R₀ = 40. Note that in standard REMD simulations τ_w must be rescaled proportional to the number of replicas. It follows from Fig. 4c that utilization rate in standard REMD sharply declines with R. In contrast, gREMD maintains γ approximately constant as the number of replicas increases. Theoretically, this behavior can be represented as $\gamma =\overline{{\tau }_{sim}/\left({\tau }_{sim}+\left(R/R_0\right){\tau }_w\right)}$ for standard REMD and $\gamma =\overline{{\tau }_{sim}/\left({\tau }_{sim}+{\tau }_w\right)}$ for gREMD. Note that these equations consider only unique values of τ_w in the distributions, and the factor R/R₀ reflects the synchronization of all replicas upon exchange attempts in standard REMD. Simple theoretical expressions for γ allow us to predict the impact of the number of replicas on simulation time. Because the denominator in γ represents the time of one iteration, we expect that gREMD simulation time is not impacted by R, but it grows linearly with R in standard REMD. These theoretical considerations argue that gREMD is useful in overcoming the inefficencies of grid REMD simulations with large numbers of replicas.

We have examined the ability of gREMD to accurately implement REMD algorithm. Apart from minor stochastic variations we have detected no differences between gREMD and standard REMD in collecting unique conformational states (Fig. 2a). Furthermore, gREMD implements a random walk of replicas over temperatures as expected from REMD algorithm (Fig. 2b). Finally, gREMD efficiently mixes replicas over temperatures achieving almost uniform participation of individual replicas at each REMD temperature (Fig. 2c). Taken together, these three tests suggest that gREMD conforms to the protocol of REMD algorithm.

We have also checked the ability of gREMD to properly sample peptide conformations. Our results demonstrate that gREMD reproduces almost exactly the sampling of secondary structure of Aβ29–40 peptide probed using standard REMD (Fig. 3a). Two additional comparisons focusing on tertiary structure, which involved the computations of side chain contact maps and end-to-end distance distributions, also reveal excellent consistency between gREMD and standard REMD. Taken together, these results strongly argue that gREMD correctly samples bimolecular conformational ensemble.

In summary, this paper extends local replica management to gREMD simulations performed on heterogeneous computing grids. gREMD implements local replica management through precomputed exchange schedule. Comparative analysis of gREMD and standard REMD suggest four main conclusions: (i) gREMD accelerates grid REMD simulations by as much as 40%; (i) gREMD increases utilization rates in grid REMD by up to 60%; (iii) gREMD is predicted to maintain approximately constant utilization rates and simulation times with the increase in the number of replicas; (iv) gREMD correctly implements REMD algorithm and properly samples conformational space. It is important to emphasize that gREMD offers no benefits for cluster-based simulations, which are characterized by small and uniform wait times τ_w. In fact, cluster based REMD simulations perform equally well utilizing local or global management of replicas. It is true that grid REMD may be slower than REMD simulations on a dedicated compute cluster. However, gREMD, which offers sizable improvement in performance over standard REMD on computing grids, is often the only available cheap option for REMD simulations.

4.2. Comparing gREMD to other grid-based REMD

As pointed out above the theoretical foundations of REMD do not stipulate synchronous exchange attempts in all selected replica pairs. This circumstance has been exploited in several replica management strategies. The strategy closest to our gREMD has been proposed in asynchronous REMD, which decouples the exchanges between different replica pairs [9]. In asynchronous REMD each replica exists in two states, computing and posted. In the computing state replicas are not available for exchange, whereas in the posted state replica is looking for a neighboring replica, which is also in the posted state. Once such “partner” is identified, the exchange is attempted, and then simulations are restarted on both replicas, which now enter the computing state. Similar to gREMD asynchronous REMD decentralizes the management of replicas, according to which the slowest node can no longer dictate the overall REMD performance. However, asynchronous REMD differs from gREMD in two important aspects. First, in contrast to asynchronous REMD gREMD uses precomputed schedule of exchange attempts that provides an easy tool for decentralizing replica management. Second, in asynchronous REMD the amount of sampling per temperature is not constant, whereas gREMD following precomputed schedule samples each temperature equally.

Most current REMD implementations for cluster simulations use local replica management. For example, by utilizing MPI the NAMD replica exchange protocol implements a strategy similar to gREMD [8]. NAMD replica exchange is started with the number of MPI ranks proportional to the number of replicas. NAMD assigns the schedule of replica exchanges based on a simple scheme, where a given replica alternates between using neighboring replicas below or above itself on the temperature scale as exchange partners. Exchanges are attempted as soon as a replica and its partner have both finished simulations for a current iteration. This decentralized scheme relies on the synchonization provided by MPI. However, within public grids the nodes may not be directly accessible except through few standard protocols (e.g., HTTP). Consequently, this circumstance limits applicability of standard MPI in grid computing.

Another approach to synchronization problem in REMD is based on replica multiplexing [10], in which a replica at each temperature is further replicated M times. Because a replica ready for exchange attempt can now pair with any of M replicas at neighboring temperatures, the requirement for rigid synchronization is relaxed, albeit not entirely eliminated. Multiplexing boosts the overall REMD performance at a price of expanding the scope of simulations M times. In contrast, gREMD does not change the overall amount of simulations compared to standard REMD.

A completely different strategy is proposed in serial REMD [11, 12]. In this case, each replica performs a random walk over temperatures by attempting exchanges with “virtual” replicas created from precomputed energy distributions. This strategy completely eliminates inter-replica comminication making all R replica simulations fully indepedent. The main difficulty in performing serial REMD is the requirement to obtain unbiased energy probability distributions at each REMD temperature prior to production REMD simulations. However, this strategy may be advantageous, because it largely eliminates overhead associated with replica management by a central server for grid-based REMD.

5. Conclusions

In this study we have proposed a new REMD implementation, greedy REMD, which is particularly suitable for computations performed on heterogeneous grids. To decentralize replica management gREMD utilizes a precomputed schedule of exchange attempts between temperatures. Our comparison of gREMD against standard REMD suggests four main conclusions. First, gREMD accelerates grid REMD simulations by as much as 40% and the boost in performance increases with the system size. Second, gREMD increases CPU utilization rates in grid REMD by up to 60%. Third, we argued that gREMD is expected to maintain approximately constant CPU utilization rates and simulation times with the increase in the number of replicas. Fourth, we showed that gREMD correctly implements REMD algorithm and reproduces the conformational ensemble of short peptide sampled in our previous standard REMD simulations. Additionally, gREMD preserves equal amount of sampling per temperature and requires relatively minor changes in master programs managing REMD. We believe that gREMD can find its place in large scale REMD simulations on heterogeneous computing grids.