Density-dependent analysis of nonequilibrium paths improves free energy estimates

Abstract

When a system is driven out of equilibrium by a time-dependent protocol that modifies the Hamiltonian, it follows a nonequilibrium path. Samples of these paths can be used in nonequilibrium work theorems to estimate equilibrium quantities such as free energy differences. Here, we consider analyzing paths generated with one protocol using another one. It is posited that analysis protocols which minimize the lag, the difference between the nonequilibrium and the instantaneous equilibrium densities, will reduce the dissipation of reprocessed trajectories and lead to better free energy estimates. Indeed, when minimal lag analysis protocols based on exactly soluble propagators or relative entropies are applied to several test cases, substantial gains in the accuracy and precision of estimated free energy differences are observed.

INTRODUCTION

The accurate and efficient estimation of free energy differences is an important goal in chemical physics and remains an active area of research. One promising approach to free energy estimation entails measuring the work done on a system over repetitions of an irreversible process. According to the second law of thermodynamics, the mean work is greater than the free energy difference between the end states of the process F_Λ. Nonequilibrium work theorems^{1, 2, 3, 4, 5} supplement this upper bound by rigorously equating F_Λ with other averaged functions of the work. These theorems have been empirically validated in single-molecule pulling experiments^{6, 7} and computer simulations (e.g., Ref. 8).

Jarzynski’s equality,^{1, 2} a unidirectional nonequilibrium work theorem, relates the free energy difference to an exponential average of the work. Unfortunately, because it uses a nonlinear (specifically, a logarithmic) function of the average, the free energy estimator based on this equality suffers from a systematic finite-sampling bias.^{9, 10, 11} While accurate in the limit of infinite sampling, this estimator is usually dominated by rare events where the work is less than the free energy difference and thereby converges slowly.¹²

If the average amount of work dissipated as heat is reduced, these low-work events will be more frequent and accurate free energy estimation will usually require fewer work samples. The most straightforward way to reduce heat dissipation is to slow the rate of the process; in the limit of infinitely slow switching, the process is reversible and the work is equal to the free energy difference. Unfortunately, reducing the switching rate requires additional time and lowers the signal-to-noise ratio in single-molecule pulling experiments.¹³ Under the constraint of constant experiment length, it is possible to reduce heat dissipation by optimizing the switching protocol that controls how the thermodynamic state changes with time. Protocol variation predates Jarzynski’s equality, having been applied to tightening free energy bounds from the second law of thermodynamics.^{14, 15, 16, 17, 18} More recently, variational calculus has been applied to find optimal protocols that minimize the mean work.^{19, 20, 21}

While protocol variation is, in principle, feasible in laboratory experiments, many more approaches to improving nonequilibrium-based free energy estimation are possible in computer simulations. For example, Wu and Kofke²² were inspired by the Rosenbluth chain sampling scheme to develop methods for generating low-work nonequilibrium paths. Vaikuntanathan and Jarzynski²³ took another approach, altering the system dynamics, to reduce heat dissipation and improve free energy estimates. The approach most mathematically similar to this work, however, is importance sampling in nonequilibrium path space.^{24, 25, 26, 27, 28}

In importance sampling, samples from one distribution are used to estimate expectations in another. The technique is often applied to Markov chain Monte Carlo and molecular dynamics simulations (where it is usually called umbrella sampling): After applying a configurational bias to overcome energy barriers and promote ergodicity, expectations are calculated for the unbiased ensemble. Importance sampling has been extended to transition path sampling^{29, 30} with nonequilibrium trajectories. In this algorithm, a biasing function modifies the Monte Carlo acceptance criteria of proposed paths in a way that improves the convergence of free energy estimates.^{24, 25, 26, 27, 28}

Here, we apply the importance sampling formalism in a completely different way. Instead of sampling nonequilibrium trajectories in a biased manner, we focus on the analysis of previously generated paths. Instead of asking which path ensemble we would like to sample from, we ask which path-ensemble average we would like to evaluate. This is accomplished by processing paths generated using one protocol—the sampling protocol Λ_s—using another—the analysis protocol Λ.

While we have infinite freedom in selecting an analysis protocol, not all choices will improve the convergence of free energy estimates. One reasonable strategy for choosing Λ is to minimize the lag, the difference between the nonequilibrium and instantaneous equilibrium densities; Vaikuntanathan and Jarzynski²³ found that under certain dynamics, dissipation is eliminated if there is no lag, leading to a zero-variance estimator of F_Λ. To reduce the lag, they modified their equation of motion with an additional flow-field term that “escorts” the system along a near-equilibrium path. Essentially, this strategy modifies the nonequilibrium density. In this paper, we take the opposite approach: Using the analysis protocol to choose an instantaneous equilibrium density that closely matches the sampled nonequilibrium density.

As an illustrative case, consider a Brownian particle in a harmonic oscillator, or spring, which moves at a constant velocity (Fig. 1). If the system starts in thermal equilibrium, its density is Gaussian about the initial spring position. When the spring starts moving, the density remains Gaussian with the same variance, but its mean position x_T(t) lags behind the spring position.^{31, 32} For this particular system, an analysis protocol based on x_T(t) will have no lag. We shall further explore this system in Sec. 3.

Figure 1.

Lag in a moving harmonic oscillator: Potential energy (solid line) U(x,t) and density (dashed line) p(x,t) as a function of position at (a) t=0 and (b) t=0.1, where v=10. Sampling protocol (solid line) Λ_s and mean position x_T(t) as a function of time for (c) v=10 and (d) v=15.8019. For all parts of this figure, D=1 and k=25.

One complication with using an analysis protocol that minimizes the lag is that its end state is usually not the same as in the sampling protocol. Thus, the free energy difference being estimated differs. To estimate the same F_Λ with a minimal lag analysis protocol, it may be necessary to extend or otherwise modify the sampling. To distinguish the two situations, we shall refer to the former as protocol postprocessing and the latter as nonequilibrium density-dependent sampling (NEDDS). Both fall under the aegis of density-dependent analysis.

The structure of this paper is as follows. In Sec. 2, the importance sampling form of Jarzynski’s equality is detailed. In Sec. 3, density-dependent analysis is demonstrated on two cases in which the propagator is analytically known. In Sec. 4, a general method for finding minimal lag analysis protocols is described, applied to an adaptive algorithm for NEDDS, and tested on the model system. Lastly, implications of this method and possible future directions are discussed.

FREE ENERGY FORMALISM

Consider a system whose Hamiltonian H=H(x;λ) depends on x, its position in phase space (or configuration space), and a control parameter λ. Initially, the system is prepared in thermal equilibrium at λ(0). The parameter λ is perturbed according to a protocol Λ=λ(t) until it reaches a final state at λ(τ). Jarzynski’s equality^{1, 2}

$$e^{-F_{\Lambda }}=\frac{\int dX{e}^{-W\left[X\mid \Lambda \right]}{\rho }_{\Lambda }\left[X\right]}{\int dX{\rho }_{\Lambda }\left[X\right]}\equiv {⟨e^{-W\left[X\mid \Lambda \right]}⟩}_{\Lambda }$$ (1)

relates the free energy difference between the initial and final states of the protocol F_Λ to an average over all possible paths X=x(t), resulting from this nonequilibrium procedure. Specifically, this expectation (denoted by the angled brackets ⟨…⟩_Λ), is a path integral over infinitesimal elements dX with the protocol-dependent density ρ_Λ[X]. During each process, the work done on the system is $W\left[X\mid \Lambda \right]={\int }_0^{\tau }dt\dot{\lambda }\left(\partial H∕\partial \lambda \right)$. (In this paper, all energies will be expressed in units of k_BT.)

Suppose that instead of ρ_Λ[X], we consider an alternate density of paths ρ_s[X]. The free energy difference F_Λ can be calculated by applying a reweighed form of Jarzynski’s equality²⁶

$$e^{-F_{\Lambda }}=\frac{\int dX{e}^{-W\left[X\mid \Lambda \right]}\left(\frac{{\rho }_{\Lambda }\left[X\right]}{{\rho }_s\left[X\right]}\right){\rho }_s\left[X\right]}{\int dX\left(\frac{{\rho }_{\Lambda }\left[X\right]}{{\rho }_s\left[X\right]}\right){\rho }_s\left[X\right]}\equiv \frac{{⟨r{e}^{-W\left[X\mid \Lambda \right]}⟩}_s}{{⟨r⟩}_s},$$ (2)

where r=ρ_Λ[X]∕ρ_s[X] is the ratio of probabilities of observing the trajectory given the densities. To analyze a finite sample of paths drawn from ρ_s[X], we replace the expectations with sample mean estimators obtaining²⁶

$${\overline{F}}_{\Lambda }=-\text{ln}\left(\frac{{\sum }_{n=1}^{N_s}r{e}^{-W\left[X_n\mid \Lambda \right]}}{{\sum }_{n=1}^{N_s}r}\right),$$ (3)

where N_s is the sample size. In a standard Jarzynski estimate, r=1.

Previous workers have improved the convergence properties of Eq. 3 by choosing ρ_s[X] to be various work-weighted functionals of the original density ρ_Λ[X].^{26, 27, 28} When introducing the single-ensemble biased path sampling approach, Ytreberg and Zuckerman²⁶ picked ρ_s[X]=ρ_Λ[X]e^{−W[X∣Λ]∕2}, such that r=e^{W[X∣Λ]∕2}. In a paper comparing the method with conventional equilibrium procedures, Oberhofer et al.²⁷ considered ρ_s[X]=ρ_Λ[X]∕P(W[X∣Λ]). By variation in the asymptotic variance with respect to the sampling bias, Oberhofer and Dellago²⁸ found that optimal work-weighted sampling is given by ρ_s[X]=ρ_Λ[x]∣e^{−(W[X∣Λ]−F_Λ)}−1∣. Unfortunately, this optimal choice is impractical because it includes the sought quantity F_Λ.

In the present method, which applies Eq. 3 in a novel manner, ρ_s[X]=ρ_Λs[X] depends on the sampling protocol and r differs from unity when the analysis protocol Λ varies from Λ_s. Notably, the relevant work is W[X∣Λ], not W[X∣Λ_s], meaning that different choices of Λ will result in various work distributions and convergence properties. This new way of applying importance sampling leads to different, albeit analogous, asymptotic variance expressions.³³ The present approach is more general than previous applications of Eq. 3, which require transition path sampling, because it does not require biased sampling and paths can be generated by ordinary dynamical equations. Indeed, under certain assumptions, such as those suggested by Nummela and Andricioaei,³⁴ it may be possible to apply the present method to laboratory experiments.

We note, as a caveat, that the importance sampling form of Jarzynski’s equality will only be useful for stochastic dynamics where r can be computed. Under deterministic dynamics, r is a delta function and having different sampling and analysis protocols will not improve free energy estimates.

CASES WITH AN ANALYTICAL PROPAGATOR

As mentioned earlier, we would like to choose an analysis protocol that minimizes the lag, such that the instantaneous equilibrium density corresponds to the sampled nonequilibrium density. This is particularly tractable when the propagator is exactly known. Here, we demonstrate Eq. 3 on two such cases: A Brownian particle in a harmonic oscillator (i) moving at a constant velocity or (ii) with a time-dependent natural frequency. With both, the potential energy has the general form $U\left(x\right)=k{\left(x-\overline{x}\right)}^2∕2$ and the nonequilibrium density is

$$p_{\text{neq}}\left(x,t\right)=\sqrt{\frac{k_T\left(t\right)}{2\pi }}{e}^{-\frac{k_T\left(t\right)}{2}{\left(x-x_T\left(t\right)\right)}^2},$$ (4)

where x_T(t) and k_T(t) are the most typical paths and spring coefficients, respectively. As these propagators can be obtained by close analogy to the path integral derivation of work-weighted propagators,³² their derivations are not detailed here. In case (i), k is constant and λ moves the spring position according to $\overline{x}={\Lambda }_s=vt$, such that ΔF=0, k_T(t)=k, and

$$x_T\left(t\right)=vt-\frac{v}{Dk}\left(1-e^{-Dkt}\right).$$ (5)

In case (ii), $\overline{x}$ is zero and λ controls the spring coefficient k=Λ_s, such that $\Delta F=\frac{1}{2}\text{ln}\left[k\left(0\right)∕k\left(\tau \right)\right]$. In the corresponding nonequilibrium density, x_T(t)=0 and

$$k_T\left(t\right)=\frac{k\left(0\right)e^{2D{\int }_0^{t}dsk\left(s\right)}}{1+2Dk\left(0\right)\left[{\int }_0^{t}du{e}^{2D{\int }_0^{u}dsk\left(s\right)}\right]}.$$ (6)

Based on these expressions, it is evident that for case (i), the minimal lag analysis protocol is Λ_ml=x_T(t) from Eq. 5, and for case (ii), it is Λ_ml=k_T(t) from Eq. 6. In these special cases, the nonequilibrium density is exactly the equilibrium density corresponding to Λ_ml and there is no lag.

To test whether density-dependent analysis leads to improved free energy estimates, one-dimensional Brownian dynamics simulations were run with the equation of motion

$$x_{j+1}=x_j-D\Delta t{U}_j^{\prime }+\sqrt{2D\Delta t}{R}_j,$$ (7)

where x_j is the position at time jΔt, Δt is the time step, and R_j is a standard normal random variable. The primes denote spatial derivatives such that $U_j^{\prime }=\partial U\left(x_j;{\lambda }_j\right)∕\partial x_j$ and $U_j^{″}={\partial }^{2}U\left(x_j;{\lambda }_j\right)∕\partial x_j^2$. For a discrete trajectory X={x₀,x₁,…,x_J} sampled with the protocol Λ_s={λ₀,λ₁,…,λ_J}, where J is the total number of steps, the probability ratio is r=e^−ΔS, where ΔS=S[X∣Λ]−S[X∣Λ_s] and S[X∣Λ] is the stochastic action (discretized from Ref. 32),

$$S\left[X\mid \Lambda \right]=\frac{U\left(x_J;{\lambda }_J\right)+U\left(x_0;{\lambda }_0\right)}{2}+\frac{\Delta t}{4D}\sum _{j=0}^{J-1}\left[{\left(\frac{x_{j+1}-x_j}{\Delta t}\right)}^2+{\left(D{U}_j^{\prime }\right)}^2-2D^2{U}_j^{″}\right]-\frac{W\left[X\mid \Lambda \right]}{2}.$$ (8)

The work was evaluated with the discrete formula $W\left[X\mid \Lambda \right]={\sum }_{j=0}^{J-1}\left[U\left(x_{j+1};{\lambda }_{j+1}\right)-U\left(x_{j+1};{\lambda }_j\right)\right]$. This action is valid in the continuum limit as J→∞ and Δt→0. To approach this limit, we chose D=1 and a time step of Δt=0.001.

The simulations were performed over 10^m steps (truncated to be an integer), where m refers to seven evenly spaced numbers between 1.5 and 3. In case (i), k was set to 25 and Λ_s was chosen to start from λ₀=0 and linearly progress to the target state at λ_f=1. With case (ii), Λ_s is a linear interpolation between 1 and 100. Afterward, the trajectories were both analyzed with the standard Jarzynski estimate and subjected to protocol postprocessing with Λ_ml.

For comparison, NEDDS was implemented by switching λ at a faster rate such that the final state went beyond λ_f and the final nonequilibrium density, according to the propagators, corresponded to the target state. This is illustrated in Fig. 1d, where moving the harmonic oscillator at a faster rate than in Fig. 1c allows for the final density to correspond to the equilibrium state with λ=1. These trajectories, which took the same amount of simulation time for the same number of steps, were then reanalyzed with the appropriate Λ_ml.

In case (i), we find that protocol postprocessing with Λ_ml leads to a desirable result: Most work values are reduced such that a larger fraction of them are less than the free energy difference (Fig. 2). Of these negative dissipation trajectories, most have a probability ratio less than one. Conversely, several positive dissipation trajectories have a probability ratio greater than one. For this set of trajectories, the modified work distribution leads to a more accurate free energy estimate.

Figure 2.

Representative work-weight plot for a moving harmonic oscillator: W[X∣Λ] and r of 50 paths with v=10 analyzed with Λ=Λ_s (squares) or Λ=x_T(t) (circles). The free energy difference (shaded line) and ${\overline{F}}_{\Lambda }$ from Eq. 3 using Λ=Λ_s (solid line) and Λ=x_T(t) (dashed line) are denoted by horizontal lines.

Over a large number of repetitions and range of switching speeds, we find that free energy estimates based on Λ=Λ_ml are vastly improved over the standard procedure Λ=Λ_s having significantly less variance and systematic bias (Fig. 3). The standard estimator only approaches the accuracy and precision of protocol processing for slow switches. Clearly, these trajectories are much better at estimating the end state free energy differences for Λ_ml than for Λ_s. The estimates of F_Λ from NEDDS also require considerably less sampling than the standard procedure, although the effect is somewhat less dramatic.

Figure 3.

Comparison of free energy estimates for a moving harmonic oscillator: Mean and standard deviation of 10 000 ${\overline{F}}_{\Lambda }$ estimates using 50 trajectories each analyzed with Λ=Λ_s (squares), Λ=Λ_ml (circles), or by NEDDS (triangles). The latter two are slightly offset to prevent error bar overlap.

Similarly, in case (ii), density-dependent methods also show improvement over the standard Jarzynski estimate. For the time-dependent natural frequency, the systematic bias of the standard estimate is relatively small but nonetheless evident at all sampled switching rates (Fig. 4). Estimates from both density-dependent methods have reduced bias and variance and are found to be of similar quality to each other.

Figure 4.

Comparison of free energy estimates for a harmonic oscillator with a time-dependent natural frequency: Mean and standard deviation of 10 000 ${\overline{F}}_{\Lambda }-F_{\Lambda }$ estimates using 50 trajectories each analyzed with Λ=Λ_s (squares), Λ=k_T(t) (circles), or by NEDDS (triangles). The latter two are slightly offset to prevent error bar overlap.

GENERAL CASE

In most practical situations, unfortunately, the propagator is not known ahead of time. Thus, prior to simulations, it is unclear how long paths need to be generated before the nonequilibrium density matches a density characteristic of the target state. While paths are being generated, however, it is possible to estimate the difference between the sampled density and arbitrary equilibrium states. States which minimize this difference can be collected in an analysis protocol with minimal lag.

One measure of the distance between two probability distributions is the Kullback–Leibler divergence or the relative entropy. The relative entropy between the nonequilibrium density and an arbitrary equilibrium state T is

$$D_{\text{KL}}\left(p_{\text{neq}}\left(x,t\right)\parallel p_T\left(x\right)\right)\equiv \int dx{p}_{\text{neq}}\left(x,t\right)\text{ln}\frac{p_{\text{neq}}\left(x,t\right)}{p_T\left(x\right)}.$$ (9)

When the integral is separated into two at the logarithm, one part is a constant with respect to T. The divergence is minimized by finding a state where the other −∫dxp_neq(x,t)ln p_T(x) is the least. Using sampled discrete paths, this integral can be estimated by ${\sum }_{n=1}^{N_s}\text{ln\hspace{0.17em}}{p}_T\left(x_{jn}\right)$, where x_jn is the position at step j of path n. For a state T, the equilibrium density is p_T(x)=exp[−(H_T(x)−F_T(x))], where H_T(x) is the test state Hamiltonian and F_T is its free energy. Thus, the relative entropy is minimized by the smallest value of

$$D_T\left(x_{j1},x_{j2},\dots ,x_{j{N}_s}\right)=\frac{1}{N_s}\left[\sum _{n=1}^{N_s}{H}_T\left(x_{jn}\right)\right]-F_T$$ (10)

among different states T. Generally, the free energy F_T is unknown, but for states which occur along the switching protocol, F_T−F₀ (where F₀ is the free energy at λ₀) can be estimated using the standard form of Jarzynski’s equality. These states constitute our search space for minimizing the lag.

Suppose we are interested in the free energy difference between the states defined by λ₀ and λ_f. We can use D_T to estimate Λ_ml on the fly and determine when to stop sampling via the following adaptive algorithm:

• (1)Start with j=0 and the work W₀=0. For each of N_s paths, obtain x₀ by drawing samples from the equilibrium ensemble at λ₀.
• (2)Propagate each path calculating x_j+1 using a dynamical equation such as Eq. 7. To obtain W_j+1, calculate the work done on the system during the time step and add it to W_j. The next step in the sampling protocol λ_j+1 is found by adding a predetermined value μ to λ_j. The sign of μ must be the same as λ_f−λ₀. Increment j by one.
• (3)Using W_j values in the standard form of Jarzynski’s equality, estimate F_j−F₀, the free energy difference between the states with λ_j and λ₀.
• (4)For each state T corresponding to {λ₀,λ₁,…λ_j}, use H_T(x) and the free energy difference estimated in the previous algorithm step to calculate D_T−F₀. The λ which minimizes D_T−F₀ is λ_ml. Add λ_ml to the minimal lag protocol Λ_ml.
• (5)If λ_ml has not crossed λ_f, repeat from algorithm step (2). Otherwise, set the final value in Λ_ml to λ_f.
• (6)Estimate the free energy difference using Eq. 3 with Λ=Λ_ml.

This algorithm was tested on Sun’s system,²⁴ where the potential energy is U(x)=x⁴−16λx². Using Eq. 3, the free energy difference was estimated between the initial state with λ₀=0, where the potential is a single well, and the target state λ_f=1, where it is a double well such that ΔF=−62.9407.²⁷ Brownian dynamics simulations were performed with the same diffusion coefficient, time step, and equation of motion as in Sec. 3. The increment of λ at each time step was μ=vΔt, where v=10^m and m refers to nine evenly spaced values between 0 and 2. For comparison, the standard Jarzynski estimate was applied to simulations where λ is switched between 0 and 1 at a slower velocity, taking the same total time as in the corresponding NEDDS simulations.

In a representative set of simulations, the density most noticeably lags behind the sampling state at the beginning (Fig. 5). Around the state defined by λ=0.9, the lag quickly diminishes. However, the minima of D_T does not reach the target state until the sampling λ is beyond 1.

Figure 5.

Representative divergence landscape for Sun’s system: Contour plot of D_T as a function of sampling λ estimated using 50 paths with v=10. Λ_ml is shown with a dashed line. Note that only half of this information, where the sampling λ is less than test λ, is available on the fly.

Based on many repetitions of this procedure at different pulling speeds, we find that our NEDDS algorithm converges much more quickly than the standard Jarzynski estimate (Fig. 6). The systematic bias is largely eliminated with simulations that are switched nearly an order of magnitude faster. At the fastest switching rates, NEDDS remains biased but still outperforms the standard Jarzynski estimate. With these fast switchings, it is possible that the nonequilibrium density does not correspond well to any traversed equilibrium state.

Figure 6.

Comparison of free energy estimates for Sun’s system: Mean and standard deviation of 10 000 ${\overline{F}}_{\Lambda }$ estimates using 50 trajectories each analyzed with Λ=Λ_s (squares) or Λ=Λ_ml (circles slightly offset to prevent error bar overlap).

DISCUSSION AND CONCLUSION

With the goal of minimizing the lag via the choice of analysis protocol, we have developed density-dependent methods to analyze nonequilibrium paths, to estimate which states may constitute a protocol that minimizes the lag, and to adaptively sample paths until the desired density is achieved. Our promising results validate the strategy and provide further evidence for the link between lag and heat dissipation. They also hint that the accurate estimation of free energy differences may require adequate sampling in the important regions of both end states.³⁵

Analysis protocols provide another degree of freedom for lag reduction and can be used in conjunction with other methods such as sampling protocol optimization or biased path sampling. Furthermore, their use should extend beyond Jarzynski’s equality; they can potentially be applied in bidirectional nonequilibrium work expressions³³ or any relationship between a nonequilibrium process and a state function such as the expression of Hummer and Szabo³⁶ for the potential of mean force. Quite possibly, our results are just the tip of an iceberg and this paper will open up new research directions for sampling and analyzing nonequilibrium trajectories.