Quantifying Configuration-Sampling Error in Langevin Simulations of Complex Molecular Systems

Abstract

While Langevin integrators are popular in the study of equilibrium properties of complex systems, it is challenging to estimate the timestep-induced discretization error: the degree to which the sampled phase-space or configuration-space probability density departs from the desired target density due to the use of a finite integration timestep. Sivak et al., introduced a convenient approach to approximating a natural measure of error between the sampled density and the target equilibrium density, the Kullback-Leibler (KL) divergence, in phase space, but did not specifically address the issue of configuration-space properties, which are much more commonly of interest in molecular simulations. Here, we introduce a variant of this near-equilibrium estimator capable of measuring the error in the configuration-space marginal density, validating it against a complex but exact nested Monte Carlo estimator to show that it reproduces the KL divergence with high fidelity. To illustrate its utility, we employ this new near-equilibrium estimator to assess a claim that a recently proposed Langevin integrator introduces extremely small configuration-space density errors up to the stability limit at no extra computational expense. Finally, we show how this approach to quantifying sampling bias can be applied to a wide variety of stochastic integrators by following a straightforward procedure to compute the appropriate shadow work, and describe how it can be extended to quantify the error in arbitrary marginal or conditional distributions of interest.

1. Introduction

Langevin dynamics [1] is a system of stochastic differential equations which describes the behavior of condensed phase systems subject to random weak collisions with fictitious bath particles at thermal equilibrium. In this article, we are concerned with the efficient numerical simulation of the Langevin dynamics system. The equations governing the ith atom of an N-body Langevin system are

$$\begin{array}{}\mathrm{(1)}& \hfill {\dot{\mathbf{x}}}_i& =& {\mathbf{v}}_i,\hfill \mathrm{(2)}& \hfill {\dot{\mathbf{v}}}_i& =& -m_i^{-1}{\nabla }_{{\mathbf{x}}_i}U\left(\mathbf{x}\right)-\gamma {\mathbf{v}}_i+\sigma m_i^{-1/2}{\dot{\mathbf{W}}}_i.\hfill \end{array}$$

Here, ${\mathbf{x}}_i$ and ${\mathbf{v}}_i$ denote the position vector and velocity vector of the ith particle of the system (typically each is a vector in ${\Re }^3$), $m_i$ is the particle mass, and $U\left(\mathbf{x}\right)$ is the total potential energy, assumed to be a function of all the coordinates, $\mathbf{x}=({\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N)$. The constant ${\sigma }^2\equiv 2k_{\mathrm{B}}T\gamma$ quantifies the rate of heat exchange with the bath, where $k_{\mathrm{B}}T$ denotes the thermal energy per degree of freedom, $\gamma$ is the collision rate (with dimensions of inverse time), and ${\mathbf{W}}_i\left(t\right)$ is a standard 3-dimensional Wiener process [2,3]. It is easily shown that Langevin dynamics preserves the canonical distribution with stationary density

$$\begin{array}{c}\hfill \pi (\mathbf{x},\mathbf{v})\propto e^{-\beta E(\mathbf{x},\mathbf{v})}\propto e^{-\beta U\left(\mathbf{x}\right)}{e}^{-\beta K\left(\mathbf{v}\right)},\end{array}$$ (3)

where $\beta \equiv {\left(k_{\mathrm{B}}T\right)}^{-1}$ is the inverse temperature, $E(\mathbf{x},\mathbf{v})$ is a separable energy function $E(\mathbf{x},\mathbf{v})=U\left(\mathbf{x}\right)+K\left(\mathbf{v}\right)$, and $K\left(\mathbf{v}\right)={\mathbf{v}}^{T}M\mathbf{v}/2$ is the kinetic energy in which M is a diagonal matrix constructed from the masses. We assume that the system (1) and (2) is ergodic, meaning that the canonical density is the unique stationary distribution; almost all stochastic paths consistent with (1) and (2) will sample the canonical distribution with density (3).

On a computer, approximating the solution of Equations (1) and (2) requires discretizing these equations in time to produce a finite-timestep Langevin integrator which can be iterated to compute equilibrium or dynamical properties [4]. A wide variety of schemes have been proposed for this discretization [5,6,7,8,9,10,11,12,13,14]. For compact presentation of integration methods, we recast the Equations (1) and (2) in vectorial form:

$$\begin{array}{l}\dot{\mathbf{x}}=\mathbf{v},\\ \dot{\mathbf{v}}=M^{-1}\nabla U\left(\mathbf{x}\right)-\gamma \mathbf{v}+\sigma M^{-1/2}\dot{\mathbf{W}}.\end{array}$$ (4)

Popular numerical methods for integrating (4) can then be viewed as defining Markov chains on the phase space $(\mathbf{x},\mathbf{v})$, with $({\mathbf{x}}_{k+1},{\mathbf{v}}_{k+1})$ defined in relation to $({\mathbf{x}}_k,{\mathbf{v}}_k)$ where the subindex k from this point forward should be taken to be a timestep index. Assuming ergodicity, these schemes provide a practical means of sampling the equilibrium distribution.

Until now, assessing whether specific integrators sample the true equilibrium density with greater fidelity than others in specific settings has relied on computing low-dimensional marginal distributions of specific observables perceived to be sensitive to configuration-space sampling errors [15], such as radial distribution functions, marginal distributions of internal coordinates [16], or the configurational temperature [13,14,17]. While it is clear that some observables are more sensitive to errors in configuration-space density than others (Figure 1), and the error in the observables of interest is paramount for a particular application, this highlights the risk of using the error in a single physical property as a surrogate for judging integrator quality, as the error in other observables of interest may be large despite small error in the test observable.

Figure 1.

Different numerical integrators introduce different error structure in phase space, illustrated in a double-well system. Here, we illustrate the timestep-dependent discretization error introduced by four integrators on a 1D double-well potential [$U\left(\mathbf{x}\right)\equiv {\mathbf{x}}^6+2cos\left(5(\mathbf{x}+1)\right)$]. The top row of 2D contour plots illustrates the difference between the phase-space density $\rho (\mathbf{x},\mathbf{v})$ sampled at the maximum timestep considered ($\Delta t=0.7$, close to the stability limit) and the equilibrium density $\pi (\mathbf{x},\mathbf{v})$; solid lines indicate positive contours, while dashed lines indicate negative contours. The bottom row of 1D density plots shows timestep-dependent perturbation in the sampled marginal distribution in configuration space, ${\rho }_{\mathbf{x}}$, with the equilibrium distribution ${\pi }_{\mathbf{x}}$ depicted as a solid black line. The sampled marginal distributions ${\rho }_{\mathbf{x}}$ are shown for increasingly large timestep, denoted ${\rho }_{\Delta t}$, depicted by increasingly light dotted lines, for $\Delta t=0.3,0.5,0.7$ (arbitrary units). Inspecting the contour plots suggests that some integrator splittings (especially VRORV) induce error that fortuitously “cancels out” when the density is marginalized by integrating over $\mathbf{v}$, while the error in other integrator splittings (ORVRO, OVRVO) constructively sums to amplify the error in configuration space.

To evaluate numerical Langevin integrators, there would be great utility in a computable, universal measure of the bias they introduce in specific concrete settings, such that low error in this measure ensures low error in all observables of interest. There is currently no computable measure of the total configuration-sampling bias for concrete choices of integrator parameters and target system.

Controlling the magnitude of the integrator-induced sampling bias is crucial when computing quantitative predictions from simulation. Following [18], we assume that the numerical method samples an associated probability density $\rho (\mathbf{x},\mathbf{v})$ which differs from the exact canonical probability density $\pi (\mathbf{x},\mathbf{v})$ of (3). Because $\rho$ does not have a closed-form, easily computable expression, it is difficult to quantify the error introduced by a given choice of timestep or integrator. (The concept of a shadow Hamiltonian has been used to embed this density in a canonical density context, but the shadow Hamiltonian cannot be directly computed, though some approaches to approximate it via expansion (generally requiring higher-order derivatives than gradients) have been proposed—see [19], Section 3 of [3], and references therein.)

We will show how, for a particularly useful measure of error, we can circumvent this problem and develop a simple, effective approach to measuring error in complex molecular systems. Our focus in the sequel is on the error in configurational averages. We therefore introduce the notation ${\rho }_{\mathbf{x}}$ and ${\pi }_{\mathbf{x}}$ to indicate the position-dependent configurational marginal densities of $\rho$ and $\pi$.

KL Divergence as a Natural Measure of Sampling Bias

An ideal measure of the discrepancy between the sampled distribution ${\rho }_{\mathbf{x}}$ and the equilibrium distribution ${\pi }_{\mathbf{x}}$ should be “universal” in the sense that driving that measure to zero implies that error in any expectation also goes to zero. It should also be defined for all densities, and not rely on a system-specific choice of observables. One such measure is the Kullback–Leibler (KL) divergence,

$${\mathcal{D}}_{\mathrm{KL}}(p\parallel q)\equiv \int d\mathbf{x}p\left(\mathbf{x}\right)ln\left(\frac{p\left(\mathbf{x}\right)}{q\left(\mathbf{x}\right)}\right).$$ (5)

The KL divergence is defined and non-negative for any pair of distributions on the same support, and ${\mathcal{D}}_{\mathrm{KL}}(p\parallel q)=0$ if and only if $p=q$ almost everywhere.

In [20], Sivak and colleagues demonstrated how to approximate the KL divergence of the sampled distribution $\rho$ from the target distribution $\pi$ over the full phase-space distribution, ${\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$, in terms of a work-like quantity—the shadow work—that is readily computable for a large family of Langevin integrators (Figure 2a). This estimator depends only on the ability to draw samples from $\pi$ and to measure a suitable work-like quantity. This method was applied in [20] to measure the phase-space sampling bias introduced by a particular Langevin integrator (OVRVO) on periodic boxes of TIP3P water [21].

Figure 2.

A simple nonequilibrium protocol allows measurement of the KL divergence in phase and configuration space close to equilibrium. Simple nonequilibrium protocols can be used in complex molecular systems to rapidly estimate—utilizing the Crooks fluctuation theorem—the KL divergence of sampled Langevin densities from equilibrium. In both panels, the x-axis is the number of steps taken so far in the length-$2T$ protocol, and ${\langle w_{\mathrm{shad}}\rangle }_{\pi }$ indicates the average (reduced, unitless) shadow work accumulated over T steps of Langevin dynamics, initialized from equilibrium $(({\mathbf{x}}_0,{\mathbf{v}}_0)$~$\pi )$. (a) The original scheme described in Sivak et al. [20] to measure the KL divergence between the sampled phase-space density $\rho$ and the equilibrium phase-space density $\pi$. ${\langle w_{\mathrm{shad}}\rangle }_{\rho }$ is the average shadow work accumulated over T steps of Langevin dynamics, initialized from the integrator’s steady state $(({\mathbf{x}}_0,{\mathbf{v}}_0)$~$\rho )$. (b) The modified scheme introduced here to measure the KL divergence in the configuration-space marginal density between the marginal sampled configuration-space density ${\rho }_{\mathbf{x}}$ and marginal equilibrium density ${\pi }_{\mathbf{x}}$. ${\langle w_{\mathrm{shad}}\rangle }_{\omega }$ is the average shadow work accumulated over T steps of Langevin dynamics, where the initial configuration is drawn from the integrator’s steady state, and the initial velocities are drawn from equilibrium $({\mathbf{x}}_0$~${\rho }_{\mathbf{x}},{\mathbf{v}}_0$~$\pi \left(\mathbf{v}\right|{\mathbf{x}}_0\left)\right)$. We denote this distribution $\omega (\mathbf{x},\mathbf{v})\equiv {\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x})$. The top row schematically illustrates “distance from equilibrium”, with y-axis ticks for ${\mathcal{D}}_{\mathrm{KL}}(\pi \parallel \pi )=0$, ${\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )\le {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$. The bottom row illustrates the average work (here, just shadow work) accumulated throughout each protocol.

Since the velocity marginal is not generally of interest, and since some integrators are thought to preserve the configuration marginal of the target distribution with higher fidelity than the phase-space joint distribution, we sought to generalize the technique to estimate the KL divergence of the sampled configuration-space marginal, ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})$. Below, we show how a simple modification of the estimator described in [20] can achieve this goal, and illustrate how this provides a useful tool for measuring the integrator-induced error in configuration-space densities for real molecular systems.

2. Numerical Discretization Methods and Timestep-Dependent Bias

There are several possible ways to discretize Langevin dynamics. A flexible approach to this task is via operator splitting, where the Langevin system is split into components, for example,

$$\left[\begin{array}{c}\dot{\mathbf{x}}\\ \dot{\mathbf{v}}\end{array}\right]=\underset{\mathsf{R}}{\underbrace{\left[\begin{array}{c}\mathbf{v}\\ 0\end{array}\right]}}+\underset{\mathsf{V}}{\underbrace{\left[\begin{array}{c}0\\ -M^{-1}\nabla U\left(\mathbf{x}\right)\end{array}\right]}}+\underset{\mathsf{O}}{\underbrace{\left[\begin{array}{c}0\\ -\gamma \mathbf{v}+\sqrt{2\gamma }{\left(\beta M\right)}^{-1/2}\dot{\mathbf{W}}\end{array}\right]}}$$ (6)

where each component can be solved “exactly” (in the sense of distributions) for a small time increment. The label O indicates that the corresponding part of the splitting has the form of an Ornstein–Uhlenbeck process. The labels of the deterministic parts, R and V, have been chosen to reflect the deterministic updates of position and velocity, respectively; this notation has been used in some previous articles [20,22]. Note that in the articles of Leimkuhler and Matthews [3,13], the Langevin equations are cast in position-momentum form instead of position-velocity form and the labels A, B are then used to indicate the deterministic updates of positions and momenta, respectively. The choices of components of the splitting we use here are not the only options. For example, the Stochastic Position Verlet method [23] groups together both of the contributions V and O. Other splitting-based methods for Langevin dynamics are discussed for example in [24,25,26].

Once a splitting is defined, the propagator $e^{\mathcal{L}\Delta t}$ can be approximated as a Trotter factorization, i.e., a product of the propagators corresponding to the individual components, defining thus a family of numerical methods indexed by the string indicating the order of appearance of these individual terms. For example, we would use OVRVO to refer to the composition method,

$$\begin{array}{ccc}\hfill e^{\mathcal{L}\Delta t}\approx e^{{\mathcal{L}}_{\mathsf{OVRVO}}\Delta t}& =& e^{{\mathcal{L}}_{\mathsf{O}}\frac{\Delta t}{2}}{e}^{{\mathcal{L}}_{\mathsf{V}}\frac{\Delta t}{2}}{e}^{{\mathcal{L}}_{\mathsf{R}}\Delta t}{e}^{{\mathcal{L}}_{\mathsf{V}}\frac{\Delta t}{2}}{e}^{{\mathcal{L}}_{\mathsf{O}}\frac{\Delta t}{2}}.\hfill \end{array}$$ (7)

Due to the lack of commutativity of the operators, equality between the true propagator and the Strang splitting is only achieved in the limit $\Delta t\to 0$, i.e., for vanishing timestep. However, in the case of splitting methods as defined above, it is possible to analyze the error in the effective probability distribution sampled by the finite timestep method [27].

The advantage of the splitting approach is that each component propagator, $e^{{\mathcal{L}}_{\mathsf{R}}\tau }$, $e^{{\mathcal{L}}_{\mathsf{V}}\tau }$, and $e^{{\mathcal{L}}_{\mathsf{O}}\tau }$, has a straightforward interpretation in terms of arithmetic operations:

$$\begin{array}{}\mathrm{(8)}& \hfill \mathsf{R}:e^{{\mathcal{L}}_{\mathsf{R}}\tau }& :\left[\begin{array}{c}\Delta \mathbf{x}\\ \Delta \mathbf{v}\end{array}\right]& =\left[\begin{array}{c}\mathbf{v}\\ 0\end{array}\right]\tau \hfill \mathrm{(9)}& \hfill \mathsf{V}:e^{{\mathcal{L}}_{\mathsf{V}}\tau }& :\left[\begin{array}{c}\Delta \mathbf{x}\\ \Delta \mathbf{v}\end{array}\right]& =\left[\begin{array}{c}0\\ -M^{-1}\nabla U\left(\mathbf{x}\right)\end{array}\right]\tau \hfill \mathrm{(10)}& \hfill \mathsf{O}:e^{{\mathcal{L}}_{\mathsf{O}}\tau }& :\left[\begin{array}{c}\Delta \mathbf{x}\\ \Delta \mathbf{v}\end{array}\right]& =\left[\begin{array}{c}0\\ (a\left(\tau \right)-1)\mathbf{v}+\sqrt{1-a{\left(\tau \right)}^2}{\left(\beta M\right)}^{-1/2}\xi \end{array}\right]\hfill \end{array}$$

where $a\left(\tau \right)=e^{-\gamma \tau }$ and $\xi \sim \mathcal{N}{(0,1)}^{3N}$ is a vector of standard normal random variables drawn for each degree of freedom in each O step.

By chaining the operations in the order specified by the splitting string, we can unroll them into the sequence of mathematical updates needed to implement one cycle of the integrator for a total time $\Delta t$. For VRORV, for example, translating the splitting string into the appropriate sequence of update Equations in (8)–(10) produces the following equations for one complete integrator timestep:

$$\begin{array}{ccc}\hfill {\mathbf{v}}_{k+1/4}& =& {\mathbf{v}}_k-\frac{\Delta t}{2}{M}^{-1}\nabla U\left({\mathbf{x}}_k\right)\hfill \\ \hfill {\mathbf{x}}_{k+1/2}& =& {\mathbf{x}}_k+\frac{\Delta t}{2}{\mathbf{v}}_{k+1/4}\hfill \\ \hfill {\mathbf{v}}_{k+3/4}& =& a_1{\mathbf{v}}_{k+1/4}+\sqrt{1-a_1^2}{\left(\beta M\right)}^{-1/2}{\xi }_{k+1/2}\hfill \\ \hfill {\mathbf{x}}_{k+1}& =& {\mathbf{x}}_{k+1/2}+\frac{\Delta t}{2}{\mathbf{v}}_{k+3/4}\hfill \\ \hfill {\mathbf{v}}_{k+1}& =& {\mathbf{v}}_{k+3/4}-\frac{\Delta t}{2}{M}^{-1}\nabla U\left({\mathbf{x}}_{k+1}\right)\hfill \end{array}$$ (11)

Due to the different naming convention they adopt, this is referred to as the “BAOAB” method in the work of Leimkuhler and Matthews [3,13]. (Note that the subscripts 1/4, 1/2, etc., have no relation to intermediate physical times, and are used solely to denote the sequence of intermediate computational steps.)

As another example, the Langevin integrator of Bussi and Parrinello [11] corresponds to the splitting OVRVO; this splitting is also known as velocity Verlet with velocity randomization [22] due to its use of a velocity Verlet integrator core (substeps VRV) [28].

While both the VRORV and OVRVO discrete time integration schemes reduce to the same stochastic differential equations in the limit that $\Delta t\to 0$, they can behave quite differently for finite timesteps ($\Delta t>0$), especially for timesteps of practical interest for atomistic molecular simulation.

Langevin Integrators Introduce Sampling Bias That Grows with the Size of the Timestep

In many types of molecular simulations, only configuration-space properties are of interest. The configurational canonical density is defined by marginalization, viz,

$$\begin{array}{ccc}\hfill {\pi }_{\mathbf{x}}\left(\mathbf{x}\right)& =& \int d\mathbf{v}\pi (\mathbf{x},\mathbf{v})\propto e^{-U\left(\mathbf{x}\right)}.\hfill \end{array}$$ (12)

In practice, the velocities are simply discarded in computations while the positions are used to estimate configuration-dependent properties.

The continuous-time Langevin equations of motion (4) possess the target equilibrium density $\pi$ (3) as their unique stationary density, suggesting that, at least in principle, temporal averages along Langevin trajectories can be used to approximate averages with respect to the equilibrium distribution. However, numerical simulations with a finite timestep $\Delta t>0$ will generally sample a different distribution, which we will denote by $\rho (\mathbf{x},\mathbf{v})$, which implicitly depends on timestep $\Delta t$. The discrepancy between the distributions $\rho$ and $\pi$ will grow with the size of $\Delta t$ according to some power law (e.g., $\mathcal{O}(\Delta t^2)$ or $\mathcal{O}(\Delta t^4)$). For sufficiently large stepsize (typically inversely proportional to the fastest oscillatory mode present) the discretization will become unstable, but the value of the stepsize for which the bias is unacceptable may occur well below the molecular dynamics stability threshold [13].

Note that this phenomenon is completely separate from numerical issues introduced by finite-precision arithmetic on digital computers, which introduces roundoff error in mathematical operations; here, we presume that computations can be carried out to arbitrary precision, and analyze only the effects of time discretization. The timestep-dependent bias is also unrelated to the Monte Carlo or sampling error which is due to finite approximation of the long-term average of a given quantity.

In Figure 3, we illustrate a few key behaviors of this stepsize-dependent sampling bias in a simple quartic 1D system. Note that: (1) the numerically sampled distribution deviates from the target distribution; (2) this deviation increases with timestep $\Delta t$; and (3) the deviation in phase space ($\mathbf{x},\mathbf{v}$) may be different than the deviation in configuration space only ($\mathbf{x}$).

Figure 3.

Comparison of Langevin integrators in terms of phase-space and marginal distributions. For a simple 1D system with the quartic potential $U\left(\mathbf{x}\right)={\mathbf{x}}^4$, the error in sampled phase-space density $\rho$ and its marginal density ${\rho }_{\mathbf{x}}$ grows as a function of timestep $\Delta t$. However, different Langevin integrators (OVRVO and VRORV shown here) derived from symmetric Strang splittings can lead to drastically different error structures in phase space, which can induce fortuitous cancellation of error in the marginal distribution under certain circumstances (VRORV), see [18]. In the top row, we illustrate the definition of the 1D system (left: the potential energy function, $U\left(\mathbf{x}\right)={\mathbf{x}}^4$; middle: the equilibrium marginal density over configuration space, ${\pi }_{\mathbf{x}}\left(\mathbf{x}\right)\propto e^{-\beta U\left(\mathbf{x}\right)}$; right: the equilibrium joint distribution over phase space $\pi (\mathbf{x},\mathbf{v})$). In the middle row, we illustrate the increasing discrepancy between the sampled distribution ρ and the equilibrium distribution π, for both the full phase-space and the marginal configuration space, as a function of timestep $\Delta t$, for the given model problem and the particular choice of the Bussi–Parinello Langevin integrator OVRVO (7). Here the differences between exact and discrete configurational measures are plotted above the contours of the phase space density, for four values of the stepsize $\Delta t=$ [0.43, 0.66, 0.88, 1.1]. In the bottom row, we illustrate the timestep-dependent error in a similar way for another integrator VRORV (12).

It has been proposed that some integrators of Langevin dynamics (particularly the VRORV also known as “BAOAB” integrator of Leimkuhler and Matthews) preserve the configuration distribution with significantly greater fidelity than other equal-cost integration algorithms [3,13,18], a property that could have significant ramifications for the efficiency of sampling given fixed computational budgets. However, as the formal arguments for this “superconvergence” property rely on a high-friction limit, it is unclear how large the friction coefficient needs to be in practice for the argument to apply. Formal descriptions of the error are typically generic, in that they do not provide guidance on precisely which $\Delta t$ introduces a tolerable amount of bias for a particular system, and they do not provide a way to predict how other choices, such as mass matrix modifications (e.g., hydrogen mass repartitioning) [16,29,30,31], will affect the error for a system of interest.

3. Estimators for KL Divergence and the Configurational KL Divergence

3.1. Near-Equilibrium Estimators for KL Divergence

By using work averages, Sivak and Crooks [32] derived a near-equilibrium estimator for the KL divergence between an arbitrary distribution $\rho$ and the equilibrium distribution $\pi$:

$$\begin{array}{}\mathrm{(13)}& \hfill {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )& \approx \frac{1}{2}\left({\langle w\rangle }_{\pi ;\tilde{\mathsf{\Lambda }}}-{\langle w\rangle }_{\pi ,\mathsf{\Lambda };\tilde{\mathsf{\Lambda }}}\right)\mathrm{(14)}& & =\frac{1}{2}\left({\langle w\rangle }_{\pi ;\tilde{\mathsf{\Lambda }}}-{\langle w\rangle }_{\rho ;\tilde{\mathsf{\Lambda }}}\right).\end{array}$$

Here ${\langle \cdots \rangle }_{p;\tilde{\mathsf{\Lambda }}}$ indicates an average over the dynamical ensemble produced by initialization in microstates sampled from density p and subsequent driving by protocol $\tilde{\mathsf{\Lambda }}$ that is the time-reversal of protocol $\mathsf{\Lambda }$. The expectation ${\langle w\rangle }_{\pi ,\mathsf{\Lambda };\mathsf{\Lambda }}={\langle w\rangle }_{\rho ;\tilde{\mathsf{\Lambda }}}$ represents a procedure in which initial microstates are sampled from an initial density $\rho =(\pi ,\mathsf{\Lambda })$ prepared by sampling $\pi$ and applying protocol $\mathsf{\Lambda }$; the expectation is subsequently computed by averaging the work during the application of the time-reversed protocol $\tilde{\mathsf{\Lambda }}$ over many realizations of this sampling process. Note that this is distinct from ${\langle w\rangle }_{\pi ;\tilde{\mathsf{\Lambda }}}$, the expectation where the initial sample is selected from $\pi$ and the work is measured during the execution of time-reversed protocol $\tilde{\mathsf{\Lambda }}$.

Sivak, Chodera, and Crooks [20] demonstrated how to apply this estimator when $\rho$ is the biased (nonequilibrium) stationary distribution resulting from protocol $\mathsf{\Lambda }$, the repeated application of a particular numerical Langevin integrator for sufficiently long to reach steady state. In particular, because the numerical Langevin integrator is symmetric (so $\tilde{\mathsf{\Lambda }}=\mathsf{\Lambda }$) and because the time-independent Hamiltonian produces no explicit protocol work (so $w=w_{\mathrm{shad}}$), in this case the KL divergence is approximately

$$\begin{array}{}\mathrm{(15)}& \hfill {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )& \approx & \frac{1}{2}\left({\langle w\rangle }_{\pi ;\mathsf{\Lambda }}-{\langle w\rangle }_{\rho ;\mathsf{\Lambda }}\right)\hfill \mathrm{(16)}& & =& \frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi ;\mathsf{\Lambda }}-{\langle w_{\mathrm{shad}}\rangle }_{\rho ;\mathsf{\Lambda }}\right),\hfill \end{array}$$

the halved difference of two work averages: the work ${\langle w_{\mathrm{shad}}\rangle }_{\pi }$ required to drive from equilibrium $\pi$ into the steady state $\rho$, and the steady-state work ${\langle w_{\mathrm{shad}}\rangle }_{\rho }$ expended over the same length of time, but starting in $\rho$ (Figure 4a). (Note that in the limit that $\Delta t\to 0$, this relation is exact, since ${\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )\to 0$ and both ${\langle w_{\mathrm{shad}}\rangle }_{\pi ;\mathsf{\Lambda }}\to 0$ and ${\langle w_{\mathrm{shad}}\rangle }_{\rho ;\mathsf{\Lambda }}\to 0$.)

Figure 4.

Using the near-equilibrium approximation, some numerical methods introduce far less configuration-space bias in molecular mechanics models than others. The results here are reported for a small cluster of rigid TIP3P waters, described in more detail in the Detailed Methods section, and illustrated in the leftmost panel. On the x-axis is the timestep $\Delta t$, measured in femtoseconds (fs). On the y-axis is the estimated KL divergence ${\mathcal{D}}_{\mathrm{KL}}$. (a) The error over the joint distribution on ${\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$. (b) The error over the configuration-space marginal ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})$. Each colored curve corresponds to a numerical scheme for Langevin dynamics. The shaded region is the mean ± 95% confidence interval.

3.2. A Simple Modification to the Near-Equilibrium Estimator Can Compute KL Divergence in Configuration Space

In this study, we are especially interested in the configuration-space marginal distribution ${\rho }_{\mathbf{x}}\equiv \int d\mathbf{v}\rho (\mathbf{x},\mathbf{v})$. The KL divergence ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})$ between the respective configuration-space marginal distributions ${\rho }_{\mathbf{x}}$ and ${\pi }_{\mathbf{x}}\equiv \int d\mathbf{v}\pi (\mathbf{x},\mathbf{v})$ equals ${\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )$, the KL divergence between the full equilibrium distribution $\pi$ and the distribution

$$\omega (\mathbf{x},\mathbf{v})\equiv {\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x}),$$ (17)

that differs from $\pi$ only in its $\mathbf{x}$-marginal:

$$\begin{array}{}\mathrm{(18)}& \hfill {\mathcal{D}}_{\mathrm{KL}}\left(\omega \right|\pi )& =& \int d\mathbf{x}d\mathbf{v}\omega (\mathbf{x},\mathbf{v})ln\left[\frac{\omega (\mathbf{x},\mathbf{v})}{\pi (\mathbf{x},\mathbf{v})}\right]\hfill \mathrm{(19)}& & =& \int d\mathbf{x}d\mathbf{v}{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x})ln\left[\frac{{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x})}{{\pi }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x})}\right]\hfill \mathrm{(20)}& & =& \int d\mathbf{x}{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\left[\int d\mathbf{v}\pi \left(\mathbf{v}\right|\mathbf{x})\right]ln\left[\frac{{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)}{{\pi }_{\mathbf{x}}\left(\mathbf{x}\right)}\right]\hfill \mathrm{(21)}& & =& \int d\mathbf{x}{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)ln\left[\frac{{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)}{{\pi }_{\mathbf{x}}\left(\mathbf{x}\right)}\right]\hfill \mathrm{(22)}& & =& {\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})\hfill \end{array}$$

This distribution $\omega$ is reached via an augmented protocol ${\mathsf{\Lambda }}^{\prime }$ consisting of the original protocol $\mathsf{\Lambda }$ (repeated application of the numerical Langevin integrator) followed by final randomization of velocities according to the equilibrium conditional distribution $\pi \left(\mathbf{v}\right|\mathbf{x})$. Thus, we construct an analogous near-equilibrium estimator of ${\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )$ based on the average shadow work $w_{\mathrm{shad}}$ accumulated by trajectories initiated from the respective distributions:

$$\begin{array}{}\mathrm{(23)}& \hfill {\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})& ={\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )\hfill \mathrm{(24)}& & \approx \frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi ;{\tilde{\mathsf{\Lambda }}}^{\prime }}-{\langle w_{\mathrm{shad}}\rangle }_{\pi ,{\mathsf{\Lambda }}^{\prime };{\tilde{\mathsf{\Lambda }}}^{\prime }}\right)\hfill \mathrm{(25)}& & =\frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi ;{\tilde{\mathsf{\Lambda }}}^{\prime }}-{\langle w_{\mathrm{shad}}\rangle }_{\omega ;{\tilde{\mathsf{\Lambda }}}^{\prime }}\right)\hfill \mathrm{(26)}& & =\frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi ;\tilde{\mathsf{\Lambda }}}-{\langle w_{\mathrm{shad}}\rangle }_{\omega ;\tilde{\mathsf{\Lambda }}}\right)\hfill \mathrm{(27)}& & =\frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi ;\mathsf{\Lambda }}-{\langle w_{\mathrm{shad}}\rangle }_{\omega ;\mathsf{\Lambda }}\right)\hfill \mathrm{(28)}& & =\frac{1}{2}\left({\langle w_{\mathrm{shad}}\rangle }_{\pi }-{\langle w_{\mathrm{shad}}\rangle }_{\omega }\right).\hfill \end{array}$$

Equation (25) follows from (26) because applying the time-reversed protocol ${\tilde{\mathsf{\Lambda }}}^{\prime }$ (velocity randomization followed by repeated application of the numerical Langevin integrator) to $\pi$ or to $\omega$ is equivalent to applying $\tilde{\mathsf{\Lambda }}$: $\pi$ and $\omega$ already have velocity distributions randomized according to $\pi \left(\mathbf{v}\right|\mathbf{x})$, and the velocity randomization step does no work. In Equation (28), we suppress the explicit protocol dependence, since henceforth all work averages will be averaged over the same protocol $\mathsf{\Lambda }$.

Compared to the full-distribution KL divergence, the configuration-only KL divergence replaces the second work average ${\langle w_{\mathrm{shad}}\rangle }_{\rho }$ with ${\langle w_{\mathrm{shad}}\rangle }_{\omega }$, computing the expectation of the shadow work over a modified initial density constructed from the nonequilibrium steady-state configuration distribution but with Maxwell–Boltzmann velocity distribution $\pi \left(\mathbf{v}\right|\mathbf{x})$. Practically, this corresponds to drawing samples from the nonequilibrium steady-state $\rho$ and replacing the velocities $\mathbf{v}$ with an i.i.d. sample from $\pi \left(\mathbf{v}\right|\mathbf{x})$. The modified procedure is depicted schematically in Figure 2b.

3.3. Comparison of Phase-Space Error for Different Integrators

We first applied the original near-equilibrium method of Sivak et al. [20] to measure the timestep-dependent phase-space error introduced by four common Langevin integrators on a molecular mechanics model system (a cluster of 20 TIP3P waters [21] in a harmonic restraining potential), for a range of timesteps $\Delta t$ between 0.1 and 8.0 femtoseconds (0.1 fs, 0.5 fs, 1.0 fs, ..., 7.5 fs, 8.0 fs). As illustrated in Figure 4a, while this approach can resolve statistically significant differences among schemes, none of the four integrator splitting schemes offers a considerable reduction in phase space error. This may be unsurprising, as none of these integrator schemes, when Metropolized, are known to provide a significant reduction in acceptance rates, which depend in some manner on the induced phase-space sampling error (Figure 5a; see Section 4 for more details about the relationship between Metropolized acceptance rates and KL divergence). Consistent with the results of [20], the phase-space error appears to scale approximately as $\mathcal{O}(\Delta t^4)$.

Figure 5.

No consistent relationship between the GHMC acceptance rate and the steady-state bias is apparent for a 1D system. Since the GHMC rejection rate grows similarly with $\Delta t$ across all four schemes, but the configuration-space KL divergence does not, the GHMC rejection rate can be overly “conservative” for some splittings. Panel (a) shows the growth in the GHMC rejection rate as a function of timestep $\Delta t$, for the 1D double-well model considered in Figure 1 and Figure 6. On the x-axis is an evenly spaced grid of 50 timesteps between 0.1 and 0.7. On the y-axis is the estimated rejection rate, which is based on a sample average of the GHMC acceptance criterion. The shaded region is the mean ± 95% confidence interval. Panel (b) compares the GHMC rejection rate vs. the phase-space bias at steady state, over the range of timesteps plotted in panel (a). The y-axis is KL divergence between the phase-space histograms, plotted on a log-scale. Panel (c) compares the GHMC rejection rate vs. the configuration-space bias at steady state, over the range of timesteps plotted in panels (a,b). The y-axis is the KL divergence between the configuration-space histograms, plotted on a log-scale. Note that in panel (c), we have truncated the leftmost parts of the curves for RVOVR and VRORV rejection rates less than 0.05 and 0.1, respectively, due to noise in histogram estimates of very small ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})$.

3.4. Comparison of Configurational KL Divergence for Different Integrators

Figure 4b shows that the measured KL divergence between the configuration-space marginals ${\rho }_{\mathbf{x}}$ and ${\pi }_{\mathbf{x}}$ can be drastically different among the four integrator schemes, and in some cases grow much more slowly than the associated phase-space sampling error (Figure 4a). In particular, for VRORV, the error in the $\mathbf{x}$-marginal is very nearly zero for the entire range of feasible timesteps, and it can be run at $\Delta t\approx 6$ fs while introducing the same amount of configuration error as other methods at $\Delta t\approx 2$ fs. This is consistent with prior findings [3,13], which showed that the VRORV scheme introduces very little error in the average potential energy and multiple other system-specific observables.

We also note that for the OVRVO scheme, ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})\approx {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$ over the range of measured timesteps (Figure 4), consistent with prior findings that estimates of ${\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$ tracked well with several measures of configuration-sampling error [20].

3.5. Influence of the Collision Rate

The results reported above are for a single, relatively weak collision rate of 1 ps${}^{-1}$. This collision rate was selected, following prior work [13], because it is low enough that barrier-crossing and conformational exploration should be fast, but large enough that low errors are observed empirically for configuration-space averages. As the formal “superconvergence” properties of VRORV were derived in the high-friction limit [3,13,18], it is worth considering how robustly the splitting VRORV introduces low configuration-space error at various collision rates. Additionally, as the collision rate goes to zero, We would expect the differences between pairs of the schemes that are equivalent upon removal of the O step (such as OVRVO and VRORV) to become smaller as the collision-rate goes to zero.

In Figure 7, we report near-equilibrium estimates of ${\mathcal{D}}_{KL}$ (as in Figure 4) over a range of collision rates spanning $\gamma$ from 0.1–100 ps${}^{-1}$. Strikingly, the configuration-space error introduced by VRORV remains low over this entire range, while other integrators (such as OVRVO) display a significant sensitivity to collision rate. In general, increasing collision rate increases phase- and configuration-space error for all integrators for which differences can statistically be differentiated, though the effect is modest over several orders of magnitude relative to the sensitivity with respect to timestep. For each condition, 50,000 protocol samples were collected (one half the number of samples used in Figure 4), and the protocol length was 2000 steps (twice the protocol length used in Figure 4). Figure A7 demonstrates that this result is robust to protocol length for all collision rates considered.

Figure 7.

The choice of collision rate influences sampling bias. As we vary the collision rate $\gamma$ over a few orders of magnitude, the resulting measured KL divergence responds in different ways for the different schemes. The phase-space bias appears to increase with increasing collision rate for all schemes. The configuration-space bias for OVRVO and ORVRO appears to increase with increasing collision rate, but the configuration-space bias for RVOVR appears to decrease with increasing collision rate. The anomalous low configuration-space error for VRORV is observed across all collision rates tested. The non-monotonic curves in the $\gamma =100$ ps${}^{-1}$ condition are expected to be due to finite-sampling error, and are expected to be attenuated at a larger number of protocol samples. (Note that one condition is omitted from these plots for clarity: estimates of ${\mathcal{D}}_{\mathrm{KL}}$ for OVRVO at $\Delta t=8$ fs. At that timestep, the variance of the resulting ${\mathcal{D}}_{\mathrm{KL}}$ estimates for this scheme was much larger than for the other schemes.) See Figure A6 for a comparison grouped by collision rate, rather than by integrator.

3.6. Comparison with Reference Methods Validates the Near-Equilibrium Estimate

The accuracy of the near-equilibrium approximation introduced by Sivak et al., is largely unexplored. While the near-equilibrium approximation is computationally and statistically appealing, it is important to validate the accuracy of the approximation over the practical timestep $\Delta t$ range of relevance to molecular simulation. In particular, it is unknown whether the near-equilibrium approximation produces an over-estimate or under-estimate of the KL divergence, or how accurate the approximation is for high-dimensional systems. Further, it is unknown whether any bias introduced by the approximation is uniform across different numerical methods for Langevin dynamics.

How well does the near-equilibrium estimator approximate the true KL divergence of relevant timestep ranges? The task of validating the near-equilibrium approximation is numerically challenging, since we are unaware of exact estimators for ${\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )$ that remain tractable in high dimensions. Spatial discretization or density estimation are infeasible, due to the curse of dimensionality. There are direct estimators of the KL divergence based on Euclidean nearest-neighbor distances that perform well in some high-dimensional settings (e.g., [33]), but Euclidean distance is an unsuitable metric on molecular configurations. In the case of simple fluids, approximate methods are available that express the KL divergence in terms of a series of N-body correlations (as in [34]), typically truncating to two-body correlation functions (i.e., comparing the radial distribution functions). However, in general we do not know the effect of truncating the expansion, since the successive terms in the series do not necessarily have decreasing magnitude.

To validate the near-equilibrium estimate of the KL-divergence, we attempt to “sandwich” it between two reference estimates that are unlikely to substantially over- or under-estimate the KL-divergence, and verify that the proposed method is consistent. In Section 3.6.1, we derive an asymptotically exact nested Monte Carlo method, that is computationally inefficient and an under-estimate in practice. In Section 3.6.2, we note that the results from the nested Monte Carlo method can be reprocessed to yield an over-estimate. In Section 3.6.3, we compute both, and compare with the near-equilibrium estimator.

3.6.1. Practical Lower Bound from Nested Monte Carlo

First, we will derive an exact expression for the KL divergence between $\rho$ and $\pi$ in terms of quantities that we can measure, then discuss practical challenges that arise when using this expression, and under what conditions it becomes impractical. We start by writing the KL divergence as an expectation over $\rho$ (30), since we cannot evaluate $\rho (\mathbf{x},\mathbf{v})$ pointwise, but we can draw samples $(\mathbf{x},\mathbf{v})\sim \rho$.

$$\begin{array}{}\mathrm{(29)}& \hfill {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )& =& \int d\mathbf{x}d\mathbf{v}\rho (\mathbf{x},\mathbf{v})ln\left[\frac{\rho (\mathbf{x},\mathbf{v})}{\pi (\mathbf{x},\mathbf{v})}\right]\hfill \mathrm{(30)}& & =& {〈ln\left[\frac{\rho (\mathbf{x},\mathbf{v})}{\pi (\mathbf{x},\mathbf{v})}\right]〉}_{\rho }\hfill \mathrm{(31)}& & =& {〈ln\left[\frac{\pi (\mathbf{x},\mathbf{v}){\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\tilde{\mathsf{\Lambda }}}}{\pi (\mathbf{x},\mathbf{v})}\right]〉}_{\rho }\hfill \mathrm{(32)}& & =& {〈ln{\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}〉}_{\rho }\hfill \mathrm{(33)}& & \approx & \frac{1}{N}\sum _{i=1}^N\left(ln\sum _{j=1}^M\frac{1}{M}{e}^{-w_{ij}}\right)\hfill \end{array}$$

We note that the inner ratio of nonequilibrium steady-state to equilibrium densities, $\rho (\mathbf{x},\mathbf{v})/\pi (\mathbf{x},\mathbf{v})$, can be expressed in terms of ${\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\tilde{\mathsf{\Lambda }}}$—the average of exponentiated nonequilibrium work measured under the application of the time-reversed protocol $\tilde{\mathsf{\Lambda }}$ starting from $(\mathbf{x},\mathbf{v})$ (31). $\mathsf{\Lambda }$ denotes the protocol used to generate $\rho$ from a sample from $\pi$—in this case, T applications of the Langevin integrator step kernel; $\tilde{\mathsf{\Lambda }}$ denotes the time-reverse of this protocol. Since the protocol we apply to generate $\rho$ from $\pi$ is time-symmetric for integrators derived from symmetric Strang splittings, we can substitute $\mathsf{\Lambda }=\tilde{\mathsf{\Lambda }}$. In the final step, we substitute a simple Monte Carlo estimator of that average, in terms of work samples $w_{ij}$, where $w_{ij}$ is the jth reduced (unitless) work measurement collected from initial condition i. Here, N is the number of initial conditions sampled (i.e., the number of “outer-loop” samples), and M is the number of work samples (i.e., the number of “inner-loop” samples) collected at each initial condition $({\mathbf{x}}_i,{\mathbf{v}}_i)$~$\rho$. (Note that applying this methodology to non-symmetric integrators (where the sequence of operations for the integrator and its time-reverse are not identical) would require modifications to this scheme, as well as the manner in which shadow work is computed.)

Related nested estimators have been proposed in the literature. Notably, an estimator for the nonequilibrium entropy $S\left(\rho \right)\equiv -\int d\mathbf{x}\rho \left(\mathbf{x}\right)ln\rho \left(\mathbf{x}\right)$ in terms of work averages is given in equation 18 of [35], and this can in turn be used to estimate the desired KL divergence if we also have suitable estimates for the equilibrium entropy $S\left(\pi \right)$.

The required work values $w_{ij}$ can be easily computed from simulations. To sample an initial condition $({\mathbf{x}}_i,{\mathbf{v}}_i)$ from $\rho$, we simply run the Langevin integrator of interest for a sufficient number of steps to sample a new uncorrelated configuration from the nonequilibrium steady-state sampled by the integrator. To compute the work accumulated from a given starting condition, we use the notion of shadow work [20]. For numerical methods constructed from symmetric Strang splittings involving the R, V, and O operations described above, we simply need to compute the sum of the total energy changes during the deterministic substeps (i.e., the potential energy change during deterministic updates of the position variables, and the kinetic energy change during deterministic updates of the momentum variables). For convenience, we use reduced (unitless) energies and work values throughout, where factors of $k_{\mathrm{B}}T$ have been removed, without loss of generality. See Detailed Methods (Section 6.4) for a detailed description on how shadow work can be computed from this family of Langevin integrators in general.

Like the near-equilibrium scheme, this nested scheme can be modified analogously to measure the configuration-space error in isolation, by initializing instead from the distribution $\omega (\mathbf{x},\mathbf{v})\equiv {\rho }_{\mathbf{x}}\left(\mathbf{x}\right)\pi \left(\mathbf{v}\right|\mathbf{x})$, allowing us to compute ${\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )$, a quantity that is identical to ${\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})$ (see Section 3.2). Specifically, to measure the full KL divergence, we sample initial conditions from the Langevin integrator’s steady state: $({\mathbf{x}}_i,{\mathbf{v}}_i)$~$\rho$. To measure configuration-space-only KL divergence, we draw initial configuration from the integrator’s steady state, and velocities from equilibrium: ${\mathbf{x}}_i$~${\rho }_{\mathbf{x}}$, ${\mathbf{v}}_i$ ~$\pi \left({\mathbf{v}}_i\right|{\mathbf{x}}_i)$. Note that, for constrained systems, $\pi \left(\mathbf{v}\right|\mathbf{x})$ is not independent of $\mathbf{x}$, and care must be taken to eliminate velocity components along constrained degrees of freedom before measuring the contribution of the integrator substep to the shadow work (see Detailed Methods).

We note that the nested plug-in Monte Carlo estimator of the KL divergence is asymptotically exact only when both N (the number of “outer-loop” samples) and M (the number of “inner-loop” samples) go to infinity. For a finite number of samples, the nested estimator will produce an under-estimate of the KL divergence. To see this, note that the nested estimator is a simple average of many likely underestimates, so it should itself be an underestimate. More specifically, although the outer-loop expectation can be approximated without bias (since samples can be drawn from $\rho$), each of the inner-loop expectations is an exponential average ($log\langle exp(-w)\rangle$), which we will underestimate when we plug in a finite sample of w’s (for the same reasons that the EXP estimator for free energies is biased [36]—To leading order, that under-estimate is related to the variance of w [36], which here grows rapidly with $\Delta t$ as shown in Figure A2).

In practice, we use a simple adaptive scheme (described in detail in Section 6.6) that draws inner- and outer-loop samples until uncertainty thresholds are met, which should minimize the magnitude of this bias.

3.6.2. Practical Upper Bound from Jensen’s Inequality

A simple, but practically useful, upper bound for the KL divergence can be obtained from the application of Jensen’s inequality, $\langle lnx\rangle \le ln\langle x\rangle$, to (32):

$$\begin{array}{}\mathrm{(34)}& \hfill {\mathcal{D}}_{\mathrm{KL}}(\rho \parallel \pi )& =& {\langle ln{\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}\rangle }_{\rho }\hfill \mathrm{(35)}& & \le & ln{\langle {\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}\rangle }_{\rho }\hfill \mathrm{(36)}& & =& ln{\langle e^{-w}\rangle }_{\rho ;\mathsf{\Lambda }}.\hfill \end{array}$$

The analogous inequality for the configuration-space marginal is

$$\begin{array}{}\mathrm{(37)}& \hfill {\mathcal{D}}_{\mathrm{KL}}({\rho }_{\mathbf{x}}\parallel {\pi }_{\mathbf{x}})& =& {\mathcal{D}}_{\mathrm{KL}}(\omega \parallel \pi )\hfill \mathrm{(38)}& & =& {\langle ln{\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}\rangle }_{\omega }\hfill \mathrm{(39)}& & \le & ln{\langle {\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}\rangle }_{\omega }\hfill \mathrm{(40)}& & =& ln{\langle e^{-w}\rangle }_{\omega ;\mathsf{\Lambda }}.\hfill \end{array}$$

This provides a particularly convenient upper bound on the KL-divergence, since it can be computed by reprocessing work samples collected from the nested Monte Carlo scheme. Note that when we approximate this bound with a finite number of samples, we will underestimate it for the reasons mentioned in Section 3.6.1. However, since we are pooling all work samples, the magnitude of this underestimate should be much smaller than for the inner-loop underestimates in Section 3.6.1 above, and we expect the magnitude of this bias to be negligible compared with the effect of invoking Jensen’s inequality.

3.6.3. Sandwiching the KL Divergence to Validate the Near-Equilibrium Estimate

We compared these three estimates of the KL divergence on the molecular mechanics system introduced in Figure 4, and confirmed that the near equilibrium estimate falls between the likely over- and under-estimate for all four integrator schemes, over a range of feasible timesteps (Figure 8, and in log-scale in Figure A5). We conclude that the near-equilibrium approximation is empirically reliable for measuring integrator bias on molecular mechanics models for practical timesteps.

Figure 8.

The near-equilibrium estimator is consistent with reference estimators for a practical range of $\Delta t$. We compared the near-equilibrium estimates reported in Figure 4 for the water cluster against a likely under-estimate and a likely over-estimate of the ${\mathcal{D}}_{\mathrm{KL}}$. In the top row, we validate near-equilibrium estimates of the KL divergence on the full state space $(\mathbf{x},\mathbf{v})$. In the bottom row, we validate near-equilibrium estimates of the KL divergence on configuration space $\left(\mathbf{x}\right)$ alone. Each column corresponds to a numerical method for Langevin dynamics. The darker band in each plot corresponds to the near-equilibrium estimate ± 95% confidence intervals from asymptotic uncertainty estimate (details in Section 3.1). The lighter band with a solid line corresponds to the nested Monte Carlo estimate ± 95% confidence intervals from bootstrapping (details in Section 6.6). The lighter band with a dotted line corresponds to the exponential average estimate ± 95% confidence intervals from bootstrapping (details in Section 6.6). Log-scale versions of these plots are provided in the appendix also, Figure A5. In the lower two panels, we summarize these results by plotting all near-equilibrium estimates vs. all exponential-average estimates (left) and all near-equilibrium estimates vs. all nested Monte Carlo estimates (right). The colored dots and bars correspond to the means ± uncertainties used in the earlier panels. The dashed diagonal line shows parity. Grey error dots and error bars correspond to conditions where the nested Monte Carlo estimate reached the computational budget ($5\times {10}^4$ inner-loop samples) but failed to reach the inner-loop uncertainty threshold, and is thus more biased. See Section 6.6 for additional details.

4. Relation to GHMC Acceptance Rates

What is the relationship between the bias introduced by an integrator at steady state, and the acceptance rate of the corresponding Metropolized integrator? Specifically, why not construct a Metropolized version of VRORV to guarantee samples are drawn appropriately from the equilibrium target density $\pi \left(\mathbf{x}\right)$? Following [2,37], we can construct an exact MCMC method that uses one or more steps of Langevin dynamics as a proposal, by using

$$\begin{array}{ccc}\hfill \alpha & \equiv & min\{1,e^{-w_{\mathrm{shad}}}\}\hfill \end{array}$$ (41)

as the acceptance criterion. The resulting method is called generalized hybrid Monte Carlo (GHMC), and eliminates time-discretization error at the cost of increasing sample autocorrelation (see also the comments in Section 6.3). A natural question arises: if an uncorrected Langevin integrator introduces low configuration-space error, is the rejection rate of the corresponding Metropolis-corrected method also low?

To answer this question, we estimated the GHMC acceptance rate at all conditions for which we have estimated steady-state ${\mathcal{D}}_{\mathrm{KL}}$. Given a collection of equilibrium samples (described in Section 6.3), we can efficiently estimate the acceptance rate of an MCMC proposal by taking the sample average of the acceptance ratio $\alpha$ over proposals originating from equilibrium, $({\mathbf{x}}_0,{\mathbf{v}}_0)$~$\pi$.

We compared the GHMC acceptance rate to the histogram-based ${\mathcal{D}}_{\mathrm{KL}}$ estimates for the 1D double-well system in Figure 5. There does not appear to be a consistent relationship between ${\mathcal{D}}_{\mathrm{KL}}$ and acceptance rate across the four schemes. Notably, the GHMC rejection rate can be extremely “conservative” for splittings such as VRORV.

Next, we compared the GHMC rejection rate with the near-equilibrium ${\mathcal{D}}_{\mathrm{KL}}$ estimates for the water cluster considered in Figure 9. A similar pattern is recapitulated in this molecular mechanics model as in the 1D system—there is not a consistent relationship between configuration-space bias introduced by a Langevin integrator and the rejection rate of its corresponding GHMC method.

Figure 9.

Near-equilibrium measurements recapitulate the relationship between steady-state ${\mathcal{D}}_{\mathrm{KL}}$ and GHMC acceptance rate for the water cluster test system. Panel (a) shows the growth in the GHMC rejection rate (1 minus the acceptance rate) as a function of timestep $\Delta t$ (in femtoseconds), for the water cluster test system illustrated in Figure 4. On the x-axis are timesteps [0.1 fs, 0.5 fs, 1.0 fs, ...7.5 fs, 8 fs]. On the y-axis is the estimated rejection rate, which is based on a sample average of the GHMC acceptance criterion, over 10,000 proposals per condition. The shaded region is the mean ± 95% confidence interval. Panel (b) compares the GHMC rejection rate vs. the phase-space bias at steady state, over the range of timesteps plotted in panel (a). The y-axis is the KL divergence between the phase-space distributions as measured by the near-equilibrium estimate, plotted on a log-scale. Panel (c) compares the GHMC rejection rate vs. the configuration-space bias at steady state, over the range of timesteps plotted in panels (a,b). Note that in panels (b,c), we have truncated at ${\mathcal{D}}_{\mathrm{KL}}\le {10}^{-4}$, due to noise in near-equilibrium estimates of very small ${\mathcal{D}}_{\mathrm{KL}}$.

This complicates the decision of whether to Metropolize or not. As noted in Section 6.3, incurring even a small rejection rate in GHMC can have a large effect on statistical efficiency, due to the effect of momentum flipping. An open challenge is to construct Metropolis criteria for GHMC that might be less “wasteful” for Langevin splittings that introduce low configuration-space bias. One possibility is to use a multi-proposal Metropolization procedure to correct estimates of equilibrium expectations, as done in [38].

5. Discussion

We have introduced and validated a work-based estimator of the KL divergence over the configuration-space marginal sampled by Langevin dynamics. We demonstrated that we could use this estimator to measure differences between the timestep-dependent configuration-sampling error introduced by each member of a small class of Langevin integrators on molecular mechanics models. Work-based estimators are especially attractive for biomolecular systems, since expectations over work distributions are often tractable when other approaches are not.

Reliable estimates of KL divergence using the work-based estimator considered here require knowledge of the time to reach nonequilibrium steady state. This near-equilibrium approach requires that the user select a trajectory length T sufficiently large to reach the nonequilibrium steady state, or else the KL divergence estimate could be substantially biased. Opposing the choice of large T is the variance of the estimate, since the contribution of the steady-state work to the variance of the estimate grows as T. Taken together, this suggests the smallest time T that produces unbiased estimates is optimal. In our calculations, it was sufficient to use a protocol that was twice the average collision time, but in general this choice should be validated for the system under study. One way to do this, for example, is to perform the same computation for T and $2T$ and ensure estimates are concordant to within statistical error.

Generating equilibrium samples from $\pi (\mathbf{x},\mathbf{v})$ can be difficult for large, complex molecular systems. The near-equilibrium method requires access to a large number of independent samples from the equilibrium distribution of the system under study. In this work, we used extra-chance HMC [37,39] to construct a large cache of independent equilibrium samples, amortizing the cost of equilibrium sampling across the many integrator variants (described in Section 6.3). In cases where we would like to compare a large number of integrator variants on the same system, this can be an acceptable cost, but in other cases, it may be prohibitive. It is unknown whether we can relax this requirement in practice, and use only samples in “local equilibrium,” since integrator error may be dominated by local features of the energy landscape. An alternative would be that, if the primary contributions to the dissipation process that drive integration errors arise from high-frequency motions, adding a weak restraint to parts of the system under study (such as a biological macromolecule) may also allow rapid assessment of integrator schemes in a region of configuration space where estimates can be easily converged. We have not yet tested this hypothesis, but if it is possible to relax the requirement of i.i.d. equilibrium samples and retain accurate estimates, then the method will be much cheaper to apply in difficult settings.

Future Directions

The validation of the near-equilibrium estimate makes it possible to apply the technique to a systematic comparison of sampling bias in integrator variants and biomolecular systems. Although we considered only four Langevin integrators here, this approach can be applied to any stochastic integrator for which the relative path action can be computed (see [40] for examples of how to compute the relative action for stochastic integrators not necessarily derived from operator splitting).

Independently, the work-based estimate for $ln[{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)/\pi \left(\mathbf{x}\right)]$ we used in the expensive lower bound (Section 3.6) could be useful for other analyses. For example, an estimate of $ln[{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)/\pi \left(\mathbf{x}\right)]$ could be used to interpret what features of $\mathbf{x}$ are most distorted by integrator bias, e.g., by checking which features of $\mathbf{x}$ are most predictive of extreme values of $ln[{\rho }_{\mathbf{x}}\left(\mathbf{x}\right)/\pi \left(\mathbf{x}\right)]$.

We also note that nothing about the derivation is specific to the partition between configuration degrees of freedom and velocities. We could also use this method to measure the KL divergence over any subset S of the state variables $\mathbf{z}=(\mathbf{x},\mathbf{v})$, provided we can sample from the conditional distribution for the complementary subset $S^{\prime }$ of the state variables: $\pi \left({\mathbf{z}}_{S^{\prime }}\right|{\mathbf{z}}_S)$. To measure KL divergence over the configuration variables, we need only sample from the conditional distribution of velocities given positions, which is typically tractable. Provided that the required conditional distribution is tractable, this method could also prove useful in contexts other than measuring integrator error.

Finally, in this study, we have only considered sampling the canonical ensemble (NVT; constant temperature, particle number, and volume), but the isothermal-isobaric ensemble (NpT; constant temperature, particle number, and pressure) also has wide practical relevance. In OpenMM, the isothermal-isobaric ensemble is simulated by alternating between sampling the canonical ensemble (using thermostatted dynamics, such as Langevin dynamics), and periodically sampling the volume using a molecular-scaling Monte Carlo barostat. For sufficiently infrequent barostat proposals, the bias introduced by interaction between the finite-timestep Langevin integrator and the volume perturbations is expected to be minimal, so applying the proposed near-equilibrium method is expected to approximate well the overall sampling bias. (In [20], the error in phase space was measured for an ensemble of constant-volume Langevin trajectories with initial conditions drawn from the isothermal-isobaric ensemble.)

6. Detailed Methods

All code used in this paper, along with a manifest of all conda-installable prerequsites and version numbers needed to run the code, is available at https://github.com/choderalab/integrator-benchmark under the permissive Open Source Initiative approved MIT license.

A byproduct of this work is a flexible implementation of Langevin integrators derived from operator splitting for the GPU-accelerated OpenMM molecular simulation framework [41], also available under the MIT license in the openmmtools library: https://github.com/choderalab/openmmtools. This implementation allows the user to specify a Langevin integrator using a splitting string (like OVRVO) and can automatically compute shadow work for each splitting.

6.1. One-Dimensional Model System: Double Well

For illustration and to have a model system where the exact ${\mathcal{D}}_{\mathrm{KL}}$ was readily computable using histograms, we constructed and analyzed a double-well model in 1D. The potential energy function of this model is $U\left(x\right)\equiv x^6+2cos\left(5(x+1)\right)$, illustrated in Figure 6. We implemented the four Langevin schemes under study using Numba 0.35.0 [42] for use with 1D toy models. We used a temperature of $\beta =1$, a collision rate of $\gamma =10$, and a mass of $m=10$. For these conditions, we found a maximum stable timestep of approximately $\Delta t=0.7$. Histogram-based estimates of the configuration-space density and phase-space density used 100 bins per dimension, where the bin edges were set by bounding box of a trial run at the maximum $\Delta t$. The equilibrium density of each bin was computed using numerical quadrature (using the trapezoidal rule, numpy.trapz). The KL divergence between a given $\rho$ and $\pi$ was then computed using scipy.stats.entropy on the histogram representation.

Figure 6.

KL divergence is a natural measure of sampling error, although system-specific observables display different sensitivities to sampling error. Even for the simple double-well potential considered in Figure 1, configuration-space properties display different sensitivities to sampling error, motivating the use of a “universal” error measure, such as the KL divergence. The top left panel illustrates the double-well potential energy function from Figure 1, and the top center panel shows the resulting marginal equilibrium density, ${\pi }_{\mathbf{x}}$, at $\beta =1$. The bottom left panel shows, as a function of $\Delta t$, growth in the magnitude of the error in average potential energy, ${|\langle U\rangle }_{\rho }-{\langle U\rangle }_{\pi }|$, which has been used previously as a sensitive measure of sampling error [13]. The bottom center panel shows the error in the apparent free energy difference between the two wells as a function of $\Delta t$. Note that the timestep-dependent behavior of these two observables implies different rankings of integrator fidelity that may mislead one into believing error in all observables remains low with increasing timestep. However, as is clear here, just because an integrator introduces low timestep-dependent error in one observable does not mean that the method will introduce low error in another observable: for example, OVRVO preserves the well populations as accurately as VRORV, but introduces much larger errors in the average potential energy. The right column summarizes the growth in timestep-dependent error, as measured by the KL divergence. While all four integrators introduce comparable levels of $\Delta t$-dependent error in the phase-space distribution, they induce dramatically different magnitudes of error in the configuration-space marginal.

6.2. Model Molecular Mechanics System: A Harmonically Restrained Water Cluster

As noted, the exact Monte Carlo method involves exponential work averages, resulting in a statistical inefficiency that grows rapidly with both the size of the system and the distance from equilibrium. Since we are interested in identifying whether the near-equilibrium approximation breaks down over the timestep $\Delta t$ range of interest to molecular simulations, it is important to be able to compute a reliable estimate of ${\mathcal{D}}_{\mathrm{KL}}$ far from equilibrium. Thus, we aim to select the smallest system we think will be representative of the geometry of molecular mechanics models generally in order to allow the exact estimate to be computable with reasonable computing resources.

To compare the proposed method with a reference estimator, we needed to select a test system which met the following criteria:

1. The test system must have interactions typical of solvated molecular mechanics models, so that we would have some justification for generalizing from the results. This rules out 1D systems, for example, and prompted us to search for systems that were not alanine dipeptide in vacuum.

2. The test system must have sufficiently few degrees of freedom that the nested Monte Carlo estimator remains feasible. Because the nested estimator requires converging many exponential averages, the cost of achieving a fixed level of precision grows dramatically with the standard deviation of the steady-state shadow work distribution. The width of this distribution is extensive in system size. Empirically, this ruled out using the first water box we had tried (with approximately 500 rigid TIP3P waters [21], with 3000 degrees of freedom). Practically, there was also a limit to how small it is possible to make a water system with periodic boundary conditions in OpenMM (about 100 waters, or 600 degrees of freedom), which was also infeasible.

3. The test system must have enough disordered degrees of freedom that the behavior of work averages is typical of larger systems. This was motivated by our observation that it was paradoxically much easier to converge estimates for large disordered systems than it was to converge estimates for the 1D toy system.

To construct a test system that met all of those criteria, we used a WaterCluster test system, which comprises 20 rigid TIP3P waters weakly confined in a central harmonic restraining potential with force constant $K=1\mathrm{kJ}/\mathrm{mol}/{\mathrm{nm}}^2$ applied to all atoms. This test system is available in version 0.14.0 of the openmmtools package [43]. Simulations were performed in double-precision using the Reference platform in OpenMM 7.2 [41] to minimize the potential for introducing significant round-off error due to finite floating point precision.

6.3. Caching Equilibrium Samples

To enable this study, we attempted to amortize the cost of collecting i.i.d. samples from each test system’s equilibrium distribution $\pi$ and various integrator-and-$\Delta t$-specific distributions $\rho$. Since there are many different distributions $\rho$, and all are relatively small perturbations of $\pi$, we invest initial effort into sampling $\pi$ exhaustively, and then we draw samples from each integrator-specific $\rho$ by running the integrator of interest from initial conditions $({\mathbf{x}}_0,{\mathbf{v}}_0)$~$\pi$.

For each test system, we pre-computed a large collection of $K=1000$ equilibrium samples

$$\begin{array}{c}\hfill {\pi }_{\mathbf{x}}\left(\mathbf{x}\right)\approx {\pi }^{\mathrm{cache}}\left(\mathbf{x}\right)\equiv \frac{1}{K}\sum _{k=1}^K\delta (\mathbf{x}-{\mathbf{x}}^{\left(k\right)})\end{array}$$ (42)

using extra-chance Hamiltonian Monte Carlo (XC-HMC) [37,39], implemented as a CustomIntegrator in OpenMM [41]. In brief, XC-HMC is a strategy to reduce the adverse effects of momentum flipping on sampling autocorrelation times in GHMC. GHMC uses Langevin integration of Hamiltonian dynamics as a proposal mechanism, and accepts or rejects each proposal according to a Metropolis criterion. Whenever the Metropolis test fails, the proposal is rejected and the momentum must be reversed; this is necessary to maintain detailed balance [2,44], but can lead to extremely large autocorrelation times when the acceptance rate is not sufficiently close to 100% (see [39] for empirical examples and further discussion). In “extra-chance” HMC [37,39], rather than immediately flipping the momentum whenever the Metropolis criterion fails, the proposal trajectory is instead extended to generate a new proposal, and another (suitably modified) Metropolis criterion is checked. For suitable choices of parameters (length of trajectory proposal, timestep, number of extra chances, length of “extra-chance” trajectories), this strategy can virtually eliminate the effect of momentum flipping, at the cost of increasing the average length of proposal trajectories.

Our initial experiments (on larger systems than reported here) suggested that the cost of collecting uncorrelated samples using GHMC without “extra-chances” was prohibitive, since we needed to make the timestep extremely small (around 0.1–0.25 fs) to keep the acceptance rate sufficiently near 100% that the effect of momentum flipping was acceptable. Instead, we equilibrated for approximately 1 ns (${10}^5$ XC-HMC iterations, 10 steps per XC-HMC proposal trajectory, 15 extra-chance trajectories per iteration, 1 fs per timestep) from an energy-minimized starting structure. We then saved one sample ${\mathbf{x}}^{\left(i\right)}$ every ${10}^4$ XC-HMC iterations afterwards.

To draw an i.i.d. sample from $\pi (\mathbf{x},\mathbf{v})$, we draw a sample $\mathbf{x}$ uniformly from ${\pi }^{\mathrm{cache}}$, and then sample $\mathbf{v}$ from the equilibrium distribution of velocities conditioned on $\mathbf{x}$, $\mathbf{v}$~$\pi \left(\mathbf{x}\right|\mathbf{v})$. (In the presence of holonomic constraints, the velocity distribution is not independent of the configuration distribution. For example, if bond lengths involving hydrogen atoms are constrained, the velocity of a hydrogen minus the velocity of the bonded heavy atom cannot have any component parallel to the bond.)

To draw an i.i.d. sample from $\rho$, we draw an i.i.d. sample from $\pi$ and then simulate Langevin dynamics for a large number of steps. We tested using 1000 steps, $(1/\gamma )/\Delta t$ steps, and $(2/\gamma )/\Delta t$ steps.

6.4. Computing Shadow Work for Symmetric Strang Splittings

Here, we demonstrate how to compute the appropriate shadow work for a given discrete-timestep Langevin integration scheme. We note that the log-ratio of forward and reverse conditional path probabilities under specific time-discretizations of Langevin dynamics has been calculated in many prior works, such as [9,38,45]. While there has been a great deal of confusion in the literature about how nonequilibrium work should be computed [46]; fortunately, there is an unambiguous mechanical (if tedious) approach to the computation of the appropriate work-like quantity.

Assemble the sequence of all steps and substeps of an integrator cycle into a trajectory Z. For example, for OVRVO, we have

$$\begin{array}{ccc}\hfill Z& \equiv & \{z_0\stackrel{\mathsf{O}}{⟶}{z}_1\stackrel{\mathsf{V}}{⟶}{z}_2\stackrel{\mathsf{R}}{⟶}{z}_3\stackrel{\mathsf{V}}{⟶}{z}_4\stackrel{\mathsf{O}}{⟶}{z}_5\}\hfill \end{array}$$ (43)

where $z_n\equiv ({\mathbf{x}}_n,{\mathbf{v}}_n)$ are phase-space points. Let $\tilde{Z}$ denote the time-reversal of all substeps of Z with all velocities negated,

$$\begin{array}{ccc}\hfill \tilde{Z}& \equiv & \{{\tilde{z}}_5\stackrel{\mathsf{O}}{⟶}{\tilde{z}}_4\stackrel{\mathsf{V}}{⟶}{\tilde{z}}_3\stackrel{\mathsf{R}}{⟶}{\tilde{z}}_2\stackrel{\mathsf{V}}{⟶}{\tilde{z}}_1\stackrel{\mathsf{O}}{⟶}{\tilde{z}}_0\}\hfill \end{array}$$ (44)

where ${\tilde{z}}_n=({\mathbf{x}}_n,-{\mathbf{v}}_n)$ denotes negating the velocity of phase-space point $z_n$.

To compute the reduced, unitless shadow work $w\left[Z\right]$, we use the definition of work that satisfies the Crooks fluctuation theorem for starting with a sample from the target equilibrium density $\pi \left(z_0\right)$ and taking one integrator cycle step (eqn 4 of [20]):

$$\begin{array}{c}\hfill w\left[Z\right]=ln\frac{P[Z|\mathsf{\Lambda }]}{P[\tilde{Z}|\tilde{\mathsf{\Lambda }}]}+\Delta f_{\mathrm{eq}}\end{array}$$ (45)

where the target equilibrium density is $\pi \left(z\right)=e^{f-h\left(z\right)}$, f is a log normalizing constant (dimensionless free energy), $h\left(z\right)\equiv u\left(\mathbf{x}\right)+t\left(\mathbf{v}\right)$ is the reduced Hamiltonian, $u\left(\mathbf{x}\right)$ the reduced potential, and $t\left(\mathbf{v}\right)$ the reduced kinetic energy [47]. The quantity $P[\tilde{Z}|\tilde{\mathsf{\Lambda }}]$ denotes the probability of generating the time-reversed trajectory $\tilde{Z}$ by starting with ${\tilde{z}}_5$ drawn from the target equilibrium density $\pi \left({\tilde{z}}_5\right)$ (and not the nonequilibrium steady state) and applying the reverse sequence of integrator operations $\tilde{\mathsf{\Lambda }}$, which is identical to the forward sequence of integrator operations $\mathsf{\Lambda }$ because the integrators we consider here are symmetric. Since the Hamiltonian is time-independent, the free energy change $\Delta f_{\mathrm{eq}}=0$, and this simplifies to

$$\begin{array}{}\mathrm{(46)}& \hfill w\left[Z\right]& =& ln\frac{P\left[Z\right|\mathsf{\Lambda }]}{P\left[\tilde{Z}\right|\tilde{\mathsf{\Lambda }}]}=ln\frac{\pi \left(z_0\right)}{\pi \left({\tilde{z}}_5\right)}\frac{P\left[Z\right|z_0]}{P\left[\tilde{Z}\right|{\tilde{z}}_5]}\hfill \mathrm{(47)}& & =& ln\frac{e^{f-h\left(z_0\right)}}{e^{f-h\left({\tilde{z}}_5\right)}}+ln\frac{P\left[Z\right|z_0]}{P\left[\tilde{Z}\right|{\tilde{z}}_5]}\hfill \mathrm{(48)}& & =& \Delta h\left[Z\right]+ln\frac{P\left[Z\right|z_0]}{P\left[\tilde{Z}\right|{\tilde{z}}_5]}\hfill \end{array}$$

Computation of the shadow work then proceeds by simple mechanical algebra by computing the log ratio of conditional path probabilities in the last term.

For the family of integrators considered here (symmetric Strang splittings of the propagator, composed of R, V, and O steps), the shadow work has an especially simple form:

$$\begin{array}{ccc}\hfill w\left[Z\right]& =& \Delta h\left[Z\right]-\Delta q\left[Z\right]\hfill \end{array}$$ (49)

where $\Delta h$ is the total change in reduced Hamiltonian, and $\Delta q$ is the total change in reduced heat across each of the O substeps. We note that accumulation of the shadow work during integration requires no extra force evaluations, and simply requires knowledge of the potential energy at the beginning and end of the integrator cycle as well as the changes in kinetic energy for each O substep.

For OVRVO, this is

$$\begin{array}{ccc}\hfill \Delta h\left[Z\right]& \equiv & [u\left({\mathbf{x}}_5\right)+t\left({\mathbf{v}}_5\right)]-[u\left({\mathbf{x}}_0\right)+t\left({\mathbf{v}}_0\right)]\hfill \end{array}$$ (50)

$$\begin{array}{ccc}\hfill \Delta q\left[Z\right]& \equiv & [t\left({\mathbf{v}}_1\right)-t\left({\mathbf{v}}_0\right)]+[t\left({\mathbf{v}}_5\right)-t\left({\mathbf{v}}_4\right)]\hfill \end{array}$$ (51)

We illustrate how to arrive at this result in detail for OVRVO below.

6.5. Computation of Shadow Work for OVRVO

For OVRVO, we can represent the forward and reverse of a single integrator cycle diagrammatically as

$$\begin{array}{ccc}\hfill Z& \equiv & \{z_0\stackrel{\mathsf{O}}{⟶}{z}_1\stackrel{\mathsf{V}}{⟶}{z}_2\stackrel{\mathsf{R}}{⟶}{z}_3\stackrel{\mathsf{V}}{⟶}{z}_4\stackrel{\mathsf{O}}{⟶}{z}_5\}\hfill \end{array}$$ (52)

$$\begin{array}{ccc}\hfill \tilde{Z}& \equiv & \{{\tilde{z}}_5\stackrel{\mathsf{O}}{⟶}{\tilde{z}}_4\stackrel{\mathsf{V}}{⟶}{\tilde{z}}_3\stackrel{\mathsf{R}}{⟶}{\tilde{z}}_2\stackrel{\mathsf{V}}{⟶}{\tilde{z}}_1\stackrel{\mathsf{O}}{⟶}{\tilde{z}}_0\}\hfill \end{array}$$ (53)

where ${\tilde{z}}_n=({\mathbf{x}}_n,-{\mathbf{v}}_n)$ denotes negating the velocity of phase-space point $z_n$.

To compute the conditional path probability $P\left[Z\right|z_0]$, we write a transition probability density kernel for each substep:

$$\begin{array}{ccc}\hfill P\left[Z\right|z_0]& =& K_{\mathsf{O}}(z_0,z_1)K_{\mathsf{V}}(z_1,z_2)K_{\mathsf{R}}(z_2,z_3)K_{\mathsf{V}}(z_3,z_4)K_{\mathsf{O}}(z_4,z_5)\hfill \end{array}$$ (54)

We can write the log ratio of conditional path probabilities as

$$\begin{array}{ccc}\hfill ln\frac{P\left[Z\right|z_0]}{P\left[\tilde{Z}\right|{\tilde{z}}_5]}& =& ln\frac{K_{\mathsf{O}}(z_0,z_1)}{K_{\mathsf{O}}({\tilde{z}}_1,{\tilde{z}}_0)}\frac{K_{\mathsf{V}}(z_1,z_2)}{K_{\mathsf{V}}({\tilde{z}}_2,{\tilde{z}}_1)}\frac{K_{\mathsf{R}}(z_2,z_3)}{K_{\mathsf{R}}({\tilde{z}}_3,{\tilde{z}}_2)}\frac{K_{\mathsf{V}}(z_3,z_4)}{K_{\mathsf{V}}({\tilde{z}}_4,{\tilde{z}}_3)}\frac{K_{\mathsf{O}}(z_4,z_5)}{K_{\mathsf{O}}({\tilde{z}}_5,{\tilde{z}}_4)}\hfill \end{array}$$ (55)

The probability kernels $K_{\mathsf{V}}$ and $K_{\mathsf{R}}$ are both deterministic, so as long as we are considering a trajectory Z and its time-reverse $\tilde{Z}$ generated by a symmetric integrator splitting, the ratios involving these kernels are unity.

To compute the ratios involving $K_{\mathsf{O}}$ kernels, we note that $K_{\mathsf{O}}(z_0,z_1)$ perturbs the velocity according to the update equation

$$\begin{array}{c}\hfill {\mathbf{v}}_1=a_2{\mathbf{v}}_0+\sqrt{1-a_2^2}{\left(\beta M\right)}^{-1/2}\xi \end{array}$$ (56)

where $\xi$ is a random variate drawn from the unit normal density, which allows us to solve for the random variate required to propagate from $z_0$ to $z_1$,

$$\begin{array}{ccc}\hfill \xi & =& {(1-a_2^2)}^{-1/2}{\left(\beta M\right)}^{+1/2}({\mathbf{v}}_1-a_2{\mathbf{v}}_0)\hfill \end{array}$$ (57)

$$\begin{array}{ccc}\hfill {\xi }^{\prime }& =& {(1-a_2^2)}^{-1/2}{\left(\beta M\right)}^{+1/2}({\tilde{\mathbf{v}}}_0-a_2{\tilde{\mathbf{v}}}_1)\hfill \end{array}$$ (58)

where the probability density is given by

$$\begin{array}{ccc}\hfill p\left(\xi \right)& \propto & e^{-\frac{1}{2}{\left|\xi \right|}^2}.\hfill \end{array}$$ (59)

We can then rewrite the log ratio of $\mathsf{O}$ kernels as

$$\begin{array}{}\mathrm{(60)}& {\displaystyle ln\frac{K_{\mathsf{O}}(z_0,z_1)}{K_{\mathsf{O}}({\tilde{z}}_1,{\tilde{z}}_0)}=ln\frac{p\left(\xi \right)}{p\left({\xi }^{\prime }\right)}}\hfill \mathrm{(61)}& =-\frac{\beta M}{2{(1-a_2)}^2}\left[\left(|{\mathbf{v}}_1{|}^2-2a_2{\mathbf{v}}_1·{\mathbf{v}}_0+a_2^2{\left|{\mathbf{v}}_0\right|}^2\right)-\left(|{\mathbf{v}}_0{|}^2-2a_2{\mathbf{v}}_0·{\mathbf{v}}_1+a_2^2{\left|{\mathbf{v}}_1\right|}^2\right)\right]\hfill \mathrm{(62)}& =-\frac{\beta M}{2{(1-a_2)}^2}{(1-a_2)}^2\left(|{\mathbf{v}}_1{|}^2-{\left|{\mathbf{v}}_0\right|}^2\right)\hfill \mathrm{(63)}& =-[t\left({\mathbf{v}}_1\right)-t\left({\mathbf{v}}_0\right)]\hfill \end{array}$$

Combining this with (49), this provides the overall work as

$$\begin{array}{ccc}\hfill w\left[Z\right]& =& [u\left({\mathbf{x}}_5\right)-u\left({\mathbf{x}}_0\right)]+[t\left({\mathbf{v}}_5\right)-t\left({\mathbf{v}}_0\right)]-[t\left({\mathbf{v}}_1\right)-t\left({\mathbf{v}}_0\right)]-[t\left({\mathbf{v}}_5\right)-t\left({\mathbf{v}}_4\right)]\hfill \end{array}$$ (64)

6.6. Variance-Controlled Adaptive Estimator for KL Divergence

As noted in Section 3.6, the nested Monte Carlo estimator we use as an expensive, but in principle asymptotically exact, estimate of the KL divergence requires converging a separate exponential average $ln{\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}$ for every sample $(\mathbf{x},\mathbf{v})$~$\rho$ or $\omega$. It is obviously impossible to compute this exactly; any practical approach employing finite computational resources can only estimate this quantity to some finite statistical precision, and even then, the logarithm will induce some bias in the computed estimate for finite sample sizes. Here, we take the approach of setting a sensible target statistical error for this inner estimate, arbitrarily selecting 0.01, since we would like to resolve features in ${\mathcal{D}}_{\mathrm{KL}}$ larger than this magnitude. Notably, the difficulty in achieving this threshold increases exponentially as the width of the sampled distribution $p\left(w\right)$ increases with increasing timestep $\Delta t$.

To determine the number of inner-loop samples required to meet this statistical error threshold, we periodically compute an estimate of ${\sigma }_{\mathrm{inner}}(\mathbf{x},\mathbf{v})=\mathrm{stddev}(\widehat{ln{\langle e^{-w}\rangle }_{\mathbf{x},\mathbf{v};\mathsf{\Lambda }}})$ from the available samples using standard first-order Taylor series propagation of uncertainty. We compare the estimated standard deviation with the user-defined threshold, and continue to draw samples until we meet the threshold or exceed the sample budget of $5\times {10}^4$ samples. This limited the maximum CPU time spent collecting an “outer-loop” sample to approximately 2 h—conditions where this limit was met are colored grey in the lower panel of Figure 8. The scaling of computational effort with $\Delta t$ is shown in Figure A4.

Choosing the inner-loop threshold is subtle. If the inner-loop threshold is chosen too large, then the resulting estimate will be very biased (the whole procedure only becomes unbiased in the limit that $\sigma \mathrm{inner}\to 0$). If the threshold is chosen too small, then the computational effort becomes prohibitive. Controlling the precision of the inner-loop estimates should also be expected to control their bias, since the bias of the inner-loop estimates is approximately ${\sigma }_{\mathrm{inner}}^2/2M$ (see Section II.B, Equation (8) in [36]), in the direction of under-estimating the ${\mathcal{D}}_{\mathrm{KL}}$.

To compute and report uncertainty in Figure 8, we use bootstrap resampling, rather than Taylor propagation. The data for each condition is a jagged array of sampled work values, where each row represents an “outer-loop sample” (i.e., a different initial condition $(\mathbf{x},\mathbf{v})$ sampled from $\rho$ (or $\omega$)), and the length of each row is variable, reflecting the number of “inner-loop” samples required to estimate $ln\left[\rho (\mathbf{x},\mathbf{v})/\pi (\mathbf{x},\mathbf{v})\right]$ (or $ln\left[\omega (\mathbf{x},\mathbf{v})/\pi (\mathbf{x},\mathbf{v})\right]$) to the desired precision. To generate a single bootstrap sample, we resample first the rows uniformly with replacement, and then, within each row, resample the columns uniformly with replacement. The error bands for the “exact” estimator in Figure 8 are computed from 100 bootstrap samples per condition.

Appendix A. Statistics of Shadow Work Distributions

The exact expression for the KL divergence used for validation in Section 3.6 requires estimating the expectation of $e^{-w}$ averaged over $p\left(w\right)$. Work distributions for various integrators and timesteps are plotted in Figure A1, and appear to be approximately Gaussian, as can be seen by comparison with Gaussian fits (solid lines). As expected, the width of the work distribution increases with increasing timestep, which can be seen more clearly in Figure A2, which plots the standard deviation of the work distribution as a function of timestep.

Figure A1.

Shadow work distributions for the water cluster are approximately Gaussian for all integrators examined. In all panels, solid lines and shaded regions denote Gaussian fits, while dots denote histogram estimates. The top row depicts work distributions where initial conditions are sampled from the nonequilibrium steady-state induced by the corresponding integrator and timestep $\left[(\mathbf{x},\mathbf{v})\sim \rho \right]$; these shadow work values are used to measure phase-space error in the near-equilibrium estimates of ${\mathcal{D}}_{\mathrm{KL}}$. The bottom row depicts work distributions where initial conditions are sampled from the $\omega$ ensemble $\left[\mathbf{x}\sim {\rho }_{\mathbf{x}},\mathbf{v}\sim \pi \left(\mathbf{v}\right|\mathbf{x})\right]$; these work values are used to estimate configuration-space error in the near-equilibrium estimates of ${\mathcal{D}}_{\mathrm{KL}}$.

Figure A2.

Standard deviation of water cluster shadow work distribution grows with time step. The left panel summarizes the top row of Figure A1 (shadow work distributions for trajectories initialized in $\pi$), and the right panel summarizes the bottom row of Figure A1 (trajectories initialized in $\omega$).

While the near-equilibrium estimate will find the difficulty of reaching estimates of a given statistical precision grows linearly with the variance in $p\left(w\right)$, the exact estimator must converge expectations of the exponentiated shadow work, which becomes exponentially difficult with increasing variance. This effect is illustrated in Figure A3, where we plot $e^{-w}$ along with the Gaussian fits to these work distributions.

This is reflected in the computational effort required by the variance-controlled estimator, illustrated in Figure A4.

Figure A3.

Exponential averages with respect to the shadow work distribution become increasingly difficult with increasing timestep. It becomes increasingly difficult to estimate the expectation of $e^{-w}$ with respect to Gaussian fits to work distributions $p\left(w\right)$ for the water cluster, as the timestep $\Delta t$ increases. The four panels increase in $\Delta t$ from left to right: note the changing x-axis scales. The solid line is the shadow work distribution $p\left(w\right)$ measured at each timestep. The dashed line is $e^{-w}$. The dash-dotted line is $e^{-w}·p\left(w\right)$.

Figure A4.

The computational effort required to reach a fixed uncertainty threshold depends sharply on $\Delta t$. The total computational effort is determined by the total number of trajectories sampled in that condition (i.e.,${\sum }_{i=1}^N{M}_i$), multiplied by the length of each trajectory. Note that the curves are not monotonic, since the number of required trajectories increases superlinearly with $\Delta t$, but the number of timesteps in each trajectory decreases linearly with $\Delta t$.

Appendix B. Log-Scale Plots

To better visualize agreement between near-eq and nested estimates at small timesteps, we also show a version of Figure 8 with a log-scale y-axis in Figure A5.

Figure A5.

Agreement between near-equilibrium and reference methods may be better visualized at small timesteps on a log-scale. This is a repeat of Figure 8, but with a log-scale y-axis. Vertical artifacts are caused by the estimate becoming transiently negative, which cannot be represented on the logarithmic scale.

Appendix C. Further Comments on the Collision Rate

If we are interested only in configuration-sampling, the collision rate is a free parameter. However, the collision rate affects the sampled kinetics. Depending on the application, we may choose a value of the collision rate that reproduces kinetic observables such as diffusion constants, or we may choose a value of the collision rate that minimizes correlation times. In the over-damped limit (large $\gamma$) conformational search is significantly hampered, as the trajectories become diffusive. In the under-damped limit (small $\gamma$), trajectories are not random walks, but mixing between energy levels is slower. (At the extreme, $\gamma =0$, Langevin dynamics reduces to Hamiltonian dynamics, and explores a single constant-energy ensemble.)

We do not provide any guidance here on the choice of the collision rate, but we investigated how sensitive the measured ${\mathcal{D}}_{\mathrm{KL}}$ is to the choice of $\gamma$ for each integrator. To supplement the figure in the main text (Figure 7), we plot the same information again, but grouped by $\gamma$ rather than by scheme in Figure A6.

As noted above, the accuracy of the proposed estimator hinges on the assumption that the system has been driven into steady-state during the finite-length protocol. To validate that protocols of length 2000 steps were sufficient, we compare results for protocols that are 1000 steps long and 2000 steps long, and note that they produce very similar estimates for timestep-dependent error for the range of collision rates examined. The results reported in Figure 7 and Figure A6 were performed using a protocol of length $T=2000$ steps, two times longer than the protocols used in Figure 4. We also computed near-equilibrium estimates over the same range of collision rates using a shorter protocol of length of $T=1000$ steps Figure A7. Note that, for computational convenience, we have used substantially fewer protocol samples to compute the estimates in Figure A7 than were used to compute the results reported in the main text (10,000 samples, rather than 50,000 samples).

Figure A6.

Integrator accuracy can be compared at different collision rates. Here we have plotted the same results as in Figure 7 but grouped by the collision rate $\gamma$ rather than by scheme. Note that the ordering over schemes induced by configuration-space error can vary as a function of collision rate—i.e., at the lowest measured $\gamma$, the ordering from lowest to highest error is (1) VRORV, (2) OVRVO, (3) RVOVR, (4) ORVRO, but at the highest measured $\gamma$, the ordering is different.

Figure A7.

Similar results are obtained for 1000-step protocols as for 2000-step protocols. For the same wide array of conditions reported in Figure A6, near-equilibrium estimates generated using 5x fewer samples of 2x shorter protocols are broadly consistent with the results obtained by more exhaustive sampling.

Author Contributions

J.F. performed the numerical experiments and analyzed the data. G.E.C. and J.D.C. conceived the novel estimator and contributed to the study design. D.A.S. and B.L. contributed to the study design. K.A.B. contributed software implementation. J.F., D.A.S., B.L., and J.D.C. wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest.