Eigenvector method for umbrella sampling enables error analysis

Abstract

Umbrella sampling efficiently yields equilibrium averages that depend on exploring rare states of a model by biasing simulations to windows of coordinate values and then combining the resulting data with physical weighting. Here, we introduce a mathematical framework that casts the step of combining the data as an eigenproblem. The advantage to this approach is that it facilitates error analysis. We discuss how the error scales with the number of windows. Then, we derive a central limit theorem for averages that are obtained from umbrella sampling. The central limit theorem suggests an estimator of the error contributions from individual windows, and we develop a simple and computationally inexpensive procedure for implementing it. We demonstrate this estimator for simulations of the alanine dipeptide and show that it emphasizes low free energy pathways between stable states in comparison to existing approaches for assessing error contributions. Our work suggests the possibility of using the estimator and, more generally, the eigenvector method for umbrella sampling to guide adaptation of the simulation parameters to accelerate convergence.

I. INTRODUCTION

One of the main uses of molecular simulations is the calculation of equilibrium averages. For understanding reaction processes, the free energy projected onto selected coordinates (collective variables) is of special interest. It relates directly to the probabilities of the coordinates taking particular values, and it can provide valuable information about the stable states, the barriers between them, and the origin of their stabilization. Furthermore, it is the starting point for most rate theories. Although in principle the free energy can be estimated from a long unbiased simulation, in practice doing so is challenging because bottlenecks slow the exploration of the configuration space. In other words, transitions between regions of the space are very infrequent in comparison to local fluctuations.

Various methods have been introduced to overcome this problem. Here, we consider one of the oldest and still most widely used such methods, umbrella sampling (US).^1,2 In this approach, the collective-variable interval of interest is covered by a series of simulations, in each of which the system is biased such that sampling is restricted to a relatively narrow window of values of the collective variables. This can be accomplished by addition of a biasing potential that is small in the window and large outside it. The information from the different simulations must be combined, and the effect of the bias removed, to obtain the overall free energy profile. This requires consistently normalizing the probabilities in different windows, a task that is complicated by the fact that the simulations are run independently.

Considerable effort has been devoted to determining how best to combine the results from different simulations. Initially, researchers manually adjusted the zero of free energy in each window to make the full free energy profile continuous and, often, smooth; conflicting results arising from limited sampling at the window peripheries were removed. The desire to use all the simulation data motivated the introduction of estimators that allow for systematically combining the data from different simulations. By far, the most widely used of these in chemical physics applications is the weighted histogram analysis method (WHAM). The multistate Bennett acceptance ratio (MBAR) method, as it is referred to in the molecular-simulation literature and will be referred to here, is closely related but does not rely on binning the data.^3–6 Both WHAM and MBAR can be derived from maximum-likelihood or minimum asymptotic variance principles, assuming independent, identically distributed sampling in each window, and have corresponding statistical optimality properties under those conditions. Recent extensions seek to improve performance when the sampling is limited and to extend the algorithm to more general ensembles.^7,8

In the present paper, we introduce an alternative scheme for estimating the free energy from US simulation data. In this approach, the normalization constants needed to combine information from separate simulation windows are the components of the eigenvector of a stochastic matrix that can be constructed from running averages in the windows. We thus term our method Eigenvector Method for Umbrella Sampling (EMUS). The advantage of our method is that it lends itself to error analysis. Following previous work,^9–11 we measure error with the asymptotic variance.

Our paper is organized as follows. After giving some background on US in Section II, we formulate EMUS in Section III. In Sections IV and V, we show that EMUS performs comparably to WHAM and MBAR, and discuss its connection with the latter. In Section VI, we use scaling arguments with simplifying assumptions to show that accounting for the error associated with combining the data is important and limits the speedups that can be achieved by increasing the number of simulation windows. In Section VII, we provide the full numerical analysis, which applies generally, without simplifying assumptions. Specifically, we derive a central limit theorem for averages from EMUS and use it to develop a means for estimating the error contributions from individual windows. We demonstrate the method for the free energy projected onto the ϕ and ψ dihedral angles of the alanine dipeptide and compare the error contributions with those from an estimator introduced by Zhu and Hummer.¹² We conclude in Section VIII.

II. BACKGROUND ON UMBRELLA SAMPLING

Here, we review umbrella sampling and establish basic terms and notation. The goal is the calculation of an average of an observable g over a time-independent probability distribution π,

$${\displaystyle 〈g〉=\int g\left(x\right)\pi \left(x\right)dx.}$$ (1)

At thermal equilibrium, π is the Boltzmann distribution,

$${\displaystyle \pi \left(x\right)=\frac{exp\left(-H_0\left(x\right)/k_{B}T\right)}{\int exp\left(-H_0\left(x\right)/k_{B}T\right)dx},}$$ (2)

where H₀ is the system Hamiltonian, k_B is Boltzmann’s constant, and T is the temperature. In particular, we can express the free energy difference between two states S₁ and S₂ as

$${\displaystyle \mathrm{\Delta }G=-k_{B}Tln\left(\frac{〈{\mathbb{1}}_{S1}〉}{〈{\mathbb{1}}_{S2}〉}\right),}$$ (3)

where 1 is the indicator function

$${\displaystyle {\mathbb{1}}_S\left(x\right)=\{\begin{array}{ll}1\hfill & \text{if}x\in S,\text{and}\hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}}$$ (4)

Similarly, the reversible work to constrain a collective variable q(x) to a particular value q′, also known as the potential of mean force (PMF), may be written as

$${\displaystyle W\left(q^{\prime }\right)=-k_{B}Tln〈\delta \left(q-q^{\prime }\right)〉.}$$ (5)

For complex systems, averages of the form in (1) must be evaluated numerically. Typically, this is done by generating a chain of related configurations, X_t, using Metropolis Monte Carlo methods or molecular dynamics, and by assuming ergodicity. Namely, as the number of configurations N goes to infinity, $〈g〉$ is the limit of the sample mean,

$${\displaystyle \overline{g}=\frac{1}{N}\sum _{t=0}^{N-1}g\left(X_t\right).}$$ (6)

In all practical sampling methods, successive configurations are strongly correlated. While ergodicity guarantees that sample means converge to averages over π, convergence can be extremely slow if the correlation between subsequent points is strong. This is the case when sampling π relies on visiting low-probability states, such as transition states of chemical reactions.

US methods address this issue by enforcing sampling of different regions of configuration space (windows), introducing L nonnegative bias functions ψ_i and then using L independent simulations to sample from the biased probability distributions,

$${\displaystyle {\pi }_i\left(x\right)=\frac{{\psi }_i\left(x\right)\pi \left(x\right)}{\int {\psi }_i\left(x\right)\pi \left(x\right)dx}.}$$ (7)

The essential idea is that sampling each π_i is fast because ψ_i is chosen so that relatively likely states under ψ_i are not separated by relatively unlikely states. This is accomplished by restricting the set of states on which ψ_i is non-negligible so that π is closer to constant on that set. In Section VI C we make this point more carefully by examining a regime in which umbrella sampling can be shown to be exponentially more efficient than direct simulation. A popular choice is to use bias functions that take a Gaussian form,

$${\displaystyle {\psi }_i\left(q\right)=exp\left(-\frac{1}{2}{k}_i{\left(q-q_i^0\right)}^2/k_{B}T\right),}$$ (8)

such that

$${\displaystyle {\pi }_i\left(x\right)\propto exp\left[-\left(H_0\left(x\right)+\frac{1}{2}{k}_i{\left(q-q_i^0\right)}^2\right)/k_{B}T\right].}$$ (9)

This corresponds to adding a harmonic potential centered at $q_i^0$ with spring constant k_i to the system Hamiltonian. We call the relative normalization constant of the ith biased distribution z_i,

$${\displaystyle z_i=\frac{\int {\psi }_i\left(x\right)\pi \left(x\right)dx}{\sum _{k=1}^L\int {\psi }_k\left(x\right)\pi \left(x\right)dx}.}$$ (10)

We also define the free energy in window i as

$${\displaystyle G_i=-k_{B}Tln{z}_i.}$$ (11)

We denote averages over the biased distributions by

$${\displaystyle {〈g〉}_i=\int g\left(x\right){\pi }_i\left(x\right)\hspace{0.17em}dx.}$$ (12)

Overall averages of interest, 〈g〉, can be estimated as z_i-weighted sums of averages computed in each of the windows. We detail our prescription in Section III.

III. THE EIGENVECTOR METHOD FOR UMBRELLA SAMPLING

In this section, we present the Eigenvector Method for Umbrella Sampling (EMUS). We begin by defining

$${\displaystyle g^{*}\equiv \frac{g}{\sum _{k=1}^L{\psi }_k}.}$$ (13)

for any function g. Then, we observe that

$${\displaystyle 〈g〉=\int g\left(x\right)\pi \left(x\right)\hspace{0.17em}dx}$$

$${\displaystyle =\int g\left(x\right)\left\{\frac{\sum _{i=1}^L{\psi }_i\left(x\right)\left[\frac{\int {\psi }_i\left(x\right)\pi \left(x\right)dx}{\int {\psi }_i\left(x\right)\pi \left(x\right)dx}\right]}{\sum _{k=1}^L{\psi }_k\left(x\right)}\right\}\pi \left(x\right)\hspace{0.17em}dx}$$

$${\displaystyle =\sum _{i=1}^L\int {\psi }_i\left(x\right)\pi \left(x\right)dx\frac{\int g^{*}\left(x\right){\psi }_i\left(x\right)\pi \left(x\right)dx}{\int {\psi }_i\left(x\right)\pi \left(x\right)dx}}$$

$${\displaystyle =\sum _{i=1}^L{z}_i\left(\sum _{k=1}^L\int {\psi }_k\left(x\right)\pi \left(x\right)dx\right){〈g^{*}〉}_i.}$$ (14)

The factor in parentheses can be taken out of the sum over i. To express this factor in terms of computable averages, we repeat the same steps with g = 1,

$${\displaystyle \sum _{k=1}^L\int {\psi }_k\left(x\right)\pi \left(x\right)dx=\frac{1}{\sum _{i=1}^L{z}_i{〈1^{*}〉}_i}.}$$ (15)

Substituting (15) into (14),

$${\displaystyle 〈g〉=\frac{\sum _{i=1}^L{z}_i{〈g^{\ast }〉}_i}{\sum _{i=1}^L{z}_i{〈1^{\ast }〉}_i}.}$$ (16)

Consequently, if we can evaluate the z_i, the ${〈1^{\ast }〉}_i$, and the ${〈g^{\ast }〉}_i$, then we can assemble the original average $〈g〉$ of interest. The averages ${〈g^{\ast }〉}_i$ can be computed from sequences $X_t^i$ (typically independent for each i) that sample the π_i. Umbrella sampling methods differ primarily in how the z_i are computed.

To express the constants z_i in terms of averages over the biased distributions, we take g(x) = ψ_j(x) in (14). Then, z_i solves

$${\displaystyle z_j=\sum _{i=1}^L{z}_i{F}_{ij},}$$

where

$${\displaystyle F_{ij}={〈{\psi }_j^{*}〉}_i.}$$ (17)

That is, the vector of normalization constants z is a left eigenvector of the matrix F with eigenvalue one. Under conditions to be elaborated in Section III B, the solution to (17) is uniquely specified when we notice that

$${\displaystyle \sum _{i=1}^L{z}_i=1.}$$ (18)

A. Computational procedure

In the EMUS algorithm, we estimate the entries of F and the averages ${〈g^{*}〉}_i$ and ${〈1^{*}〉}_i$ by sample means, then assemble the estimate of $〈g〉$ using (16). To be precise, we denote the sample means by

$${\displaystyle {\overline{g}}_i^{\ast }=\frac{1}{N_i}\sum _{t=0}^{N_i-1}\frac{g\left(\underset{t}{\overset{i}{X}}\right)}{\sum _k{\psi }_k\left(X_t^i\right)},}$$ (19)

$${\displaystyle {\overline{1}}_i^{\ast }=\frac{1}{N_i}\sum _{t=0}^{N_i-1}\frac{1}{\sum _k{\psi }_k\left(X_t^i\right)},\text{and}}$$ (20)

$${\displaystyle {\overline{F}}_{ij}=\frac{1}{N_i}\sum _{t=0}^{N_i-1}\frac{{\psi }_j\left(X_t^i\right)}{\sum _k{\psi }_k\left(X_t^i\right)}.}$$ (21)

EMUS proceeds as follows:

• 1.Choose the biasing functions ψ_i.
• 2.Compute trajectories that sample states $X_t^i$ from the biased distributions π_i.
• 3.Calculate the matrix $\overline{F}$ and the averages ${\overline{g}}_i^{\ast }$ and ${\overline{1}}_i^{*}$.
• 4.
  Calculate the vector of estimated normalization constants z^EMUS as the solution to

  $${\displaystyle z_j^{\text{EMUS}}=\sum _{i=1}^L{z}_i^{\text{EMUS}}{\overline{F}}_{ij}\text{with}\sum _{i=1}^L{z}_i^{\text{EMUS}}=1.}$$ (22)

  We use QR factorization as in Golub and Meyer.¹³ See also Section VII B 1.
• 5.
  Compute the estimate of $〈g〉$,

  $${\displaystyle 〈g{〉}^{\text{EMUS}}=\frac{\sum _{i=1}^L{z}_i^{\text{EMUS}}\hspace{0.17em}{\overline{g}}_i^{*}}{\sum _{i=1}^L{z}_i^{\text{EMUS}}\hspace{0.17em}{\overline{1}}_i^{*}}}$$ (23)

  by substituting z^EMUS and the sample means in (16).

We have provided an implementation of this algorithm online, along with implementations of the iteration described in Section IV and the asymptotic error estimate found in Section VII B 1.¹⁴ We remark that when one wishes to compute a free energy difference or a ratio of two observables, as in Equation (3), it is not necessary to compute the $1_i^{*}$. Instead, one may use the formula

$${\displaystyle \frac{〈g_1〉}{〈g_2〉}=\frac{\sum _{i=1}^L{z}_i\hspace{0.17em}〈g_1{〉}_i}{\sum _{i=1}^L{z}_i\hspace{0.17em}〈g_2{〉}_i}}$$ (24)

where g₁ and g₂ are arbitrary functions.

B. The eigenvector problem

In this section, we give conditions under which the eigenvector problem has a unique solution. First, we show that F is a stochastic matrix; that is, each element F_ij is nonnegative and every row of F sums to one:

$${\displaystyle \sum _{j=1}^L{F}_{ij}=\sum _{j=1}^L{〈\frac{{\psi }_j}{\sum _{k=1}^L{\psi }_k}〉}_i={〈\frac{\sum _{j=1}^L{\psi }_j}{\sum _{k=1}^L{\psi }_k}〉}_i=1.}$$ (25)

The entries of F are nonnegative since we require that the bias functions be nonnegative. One can show that the matrix $\overline{F}$ is also stochastic by similar arguments.

A stochastic matrix J has a unique eigenvector with eigenvalue one if it is irreducible: for every possible grouping of the indices into two distinct sets, A and B, J_ij ≠ 0 for some i ∈ A and j ∈ B.¹⁵ In fact, this statement remains true when J is nonnegative with largest eigenvalue equal to one. For any such matrix we let z(J) denote the continuous function returning the unique left eigenvector of J corresponding to eigenvalue one.

In the case of the particular stochastic matrix F defined in (17) these statements imply that if, for any division of the indices into sets A and B, there is a sufficient overlap between the sets ∪_i∈A{x :  ψ_i(x) > 0} and ∪_j∈B{x :  ψ_j(x) > 0} then there will be a unique solution z(F) to (17) which necessarily equals the relative normalization constants z defined in (10). Because z(J) is a continuous function of its arguments, $z^{\text{EMUS}}=z\left(\overline{F}\right)$ converges to z as $\overline{F}$ converges to F. Consequently, EMUS produces a consistent estimator in the sense that if the sample averages used to estimate the entries F_ij and ${〈g^{\ast }〉}_i$ converge (in probability or with probability one) to the true values, then the estimate of $〈g〉$ also converges (in the same sense).

IV. THE CONNECTION BETWEEN EMUS AND MBAR

Building upon earlier work in the statistics literature,^3,4,16 Shirts and Chodera⁶ suggested a class of algorithms for estimating free energy differences between states, which they termed MBAR. This method is similar to WHAM but does not require binning the simulation data to form histograms (see Tan et al.⁹). In this section, we explain the relation between EMUS and MBAR.⁶ We also derive a new iterative method for solving the MBAR equations, and we show that our iteration leads naturally to a new family of related consistent estimators.

The starting point of Shirts and Chodera⁶ is the identity (see their (5))

$${\displaystyle z_j\sum _{i=1}^L{〈{\alpha }_{ij}\left(x\right){\psi }_i\left(x\right)\pi \left(x\right)〉}_j=\sum _{i=1}^L{z}_i{〈{\alpha }_{ij}\left(x\right){\psi }_j\left(x\right)\pi \left(x\right)〉}_i,}$$ (26)

where α_ij(x) is an arbitrary function. They proposed the choice

$${\displaystyle {\alpha }_{ij}^{\text{MBAR}}\left(x\right)=\frac{n_i/z_i}{\sum _k{\psi }_k\left(x\right)\pi \left(x\right)n_k/z_k},}$$ (27)

where n_i is the number of uncorrelated samples in window i. Substituting (27) into (26) gives

$${\displaystyle z_j=\sum _{i=1}^L{z}_i{〈\frac{{\psi }_j\left(x\right)n_i/z_i}{\sum _k{\psi }_k\left(x\right)n_k/z_k}〉}_i.}$$ (28)

We can cast (28) in a form reminiscent of EMUS by writing

$${\displaystyle z_j=\sum _{i=1}^L{z}_i{F}_{ij}\left(z\right),}$$ (29)

where

$${\displaystyle F_{ij}\left(w\right)={〈\frac{{\psi }_j\left(x\right)n_i/w_i}{\sum _k{\psi }_k\left(x\right)n_k/w_k}〉}_i}$$ (30)

for any vector w with positive entries. EMUS corresponds to setting w = n so that

$${\displaystyle {\alpha }_{ij}^{\text{EMUS}}\left(x\right)=\frac{1}{\sum _k\pi \left(x\right){\psi }_k\left(x\right)},}$$ (31)

and (26) reduces to the eigenproblem (17).

In practice, one must replace the matrix F_ij(w) in (30) by the sample mean approximation

$${\displaystyle {\overline{F}}_{ij}\left(w\right)=\frac{1}{N_i}\sum _{t=0}^{N_i-1}\left[\frac{{\psi }_j\left(X_t^i\right)n_i/w_i}{\sum _k{\psi }_k\left(X_t^i\right)n_k/w_k}\right].}$$ (32)

Substituting ${\overline{F}}_{ij}\left(z\right)$ for F_ij(z) in (29) yields the equation

$${\displaystyle z_j^{\text{MBAR}}=\sum _{i=1}^L{z}_i^{\text{MBAR}}{\overline{F}}_{ij}\left(z^{\text{MBAR}}\right)}$$ (33)

for z^MBAR, which we refer to here as the MBAR estimator. If the samples $X_t^i$ are independent, MBAR is the nonparametric maximum-likelihood estimator of z.³

In practice, the samples $X_t^i$ are not independent for a given i, and the n_i must be estimated from data. Several algorithms for estimating the n_i have been proposed.^17–19 Shirts and Chodera⁶ base their estimates on the integrated autocorrelation times of physically motivated coordinates, and we follow this common practice here. In fact, once the n_i have been estimated, Shirts and Chodera⁶ suggest replacing sample averages over all N_i points by sample averages over the n_i points obtained by including only every N_i/n_i-th sample along the trajectory. We note that both the subsampling approach and the one in (32) correspond to approximations of expression (26) with (27), and we regard both as variations on the MBAR estimator. When the samples are independent, the two approaches are the same. In tests of the iterative EMUS algorithm introduced below, we find estimates to be insensitive to the choice of n_i and they can be set equal to 1, though in that case the estimator no longer corresponds directly to MBAR.

As written above, the MBAR estimator (33) resembles an eigenvector problem. However, the dependence of $\overline{F}\left(z\right)$ on z implies that the solution must be obtained self-consistently. The approach advocated by Shirts and Chodera for computing the MBAR estimator corresponds in the framework described here to solving (33) by a Newton-type iteration. However, the eigenvector form of (33) suggests an alternative approach. Rather than Newton’s method, we employ the following algorithm:

• 1.As an initial guess for z^MBAR, choose a vector z⁰ with positive entries. Estimate the n_i. Set m = 0.
• 2.
  • (a)
    Calculate ${\overline{F}}_{ij}\left(z^m\right)$ according to (32).
  • (b)
    Calculate a new estimate z^m+1 of z^MBAR by solving the eigenproblem

    $${\displaystyle z_j^{m+1}=\sum _{i=1}^L{z}_i^{m+1}{\overline{F}}_{ij}\left(z^m\right).}$$ (34)
• 3.
  If $\underset{i}{max}|z_i^{m+1}-z_i^m|/z_i^m>\mathrm{Tolerance}$,

  • (a)
    Increment m;
  • (b)
    Go to Step 2.

A similar algorithm was proposed by Meng and Wong.²⁰

To show that this iteration makes sense, we must prove that the eigenproblem (34) always has a unique solution and that z^m converges to z^MBAR as m goes to infinity. To see that the eigenproblem has a solution, first observe that if ${\overline{F}}_{ij}\left(w\right)$ is irreducible for one vector with positive entries, w, then it is irreducible for all vectors with positive entries. When applying the EMUS method, we thus assume that $\overline{F}\left(w\right)$ is irreducible. Moreover, observe that for any positive vector w, the vector with entries n_i/w_i is a right eigenvector of $\overline{F}\left(w\right)$ with eigenvalue one and positive entries. It follows from the Perron-Frobenius theorem that the matrix $\overline{F}\left(w\right)$ has a unique left eigenvector $z\left(\overline{F}\left(w\right)\right)$ with eigenvalue one and that $z\left(\overline{F}\left(w\right)\right)$ has positive entries. Thus, the eigenproblem always has a unique solution. We do not have a proof that the iterates converge. However, since z^MBAR is a fixed point of the iteration, if the iterates do converge, their limit must be z^MBAR. In practice, we find that the iteration converges quickly, usually to a relative error of 10⁻⁶ within 10 iterates.

In addition to its apparently rapid convergence, another argument in favor of the algorithm that we introduce above for solving (33) is that each iteration of the scheme results in a new consistent estimator. We will use the term iterative EMUS to refer to this family of estimators. With the initial guess z⁰ = n, the result, z¹, of the first iteration is the EMUS estimator defined in Section III. In the Appendix, we show that for any fixed finite number of iterations m, z^m is also a consistent estimator of the vector z of normalization constants. By contrast, other schemes, such as Newton’s method, for solving (33) may require that the number of iterations goes to infinity to obtain a consistent estimate. We also remark that the consistency result in the Appendix holds as long as the n_i converge to non-random, positive values with increasing numbers of samples N_i. They can be chosen as described above, or simply set to a fixed value.

Differences between the iterative EMUS scheme above and the application of Newton’s method proposed by Shirts and Chodera⁶ are mostly matters of implementation. As we will see in Section V, the results are not very sensitive to these computational details; most of the accuracy in the iterative EMUS approach is achieved in the first step. In any case, we remind the reader that the primary goal of this paper is to characterize those properties of the broader umbrella sampling approach that are essential to its success, not to analyze details of implementation.

While we focus here on potentials of mean force, the MBAR estimator has been applied to a broader category of free energy problems, including the analysis of single-molecule pulling experiments and alchemical free energy calculations.^6,21,22 The close relation between EMUS and MBAR indicates that error analysis of EMUS may provide insight into the sources of error in MBAR for these problems, but we do not pursue this idea further in the present work.

V. NUMERICAL COMPARISON

To test the algorithm numerically, we performed 100 independent umbrella sampling calculations for the PMF of the ϕ coordinate of the alanine dipeptide (i.e., N-acetyl-alanyl-N′-methylamide) in vacuum. Simulations were run using GROMACS version 5.1.1 with harmonic bias potentials applied using the PLUMED 2.2.0 software package.^23,24 The molecule was represented by the CHARMM 27 force field without CMAP corrections,²⁵ with covalent bonds to hydrogen atoms constrained by the LINCS algorithm.²⁶ Twenty windows were evenly spaced along the ϕ dihedral angle. The force constant k_i = 0.007 605 35 × 10⁻² kcal mol⁻¹ deg⁻² such that the standard deviation of the Gaussian bias functions was 9^∘. In each window, we integrated the equations of motion with the GROMACS leap-frog Langevin integrator with a 1 fs time step and a time constant of 0.1 ps. The temperature was 300 K. The system was equilibrated for 40 ps and then sampled for 100 ps, saving structures every 10 fs.

The data were then analyzed with EMUS, Grossfield’s implementation of WHAM,²⁷ and the algorithm proposed by Zhu and Hummer (ZH) (see Equation (A1) and the discussion following it in Appendix A of that paper¹²). The data were also analyzed with pyMBAR;⁶ as pyMBAR gave results virtually identical to WHAM, the results are not shown. In Figure 1, we show the resulting average potentials of mean force, as well as the standard deviation of the estimates over the 100 runs. WHAM and EMUS converge to the same result. This is to be expected, as both algorithms are consistent (i.e., they converge to the exact result as the amount of samples in each window tends to infinity; see Section VII), although WHAM exhibits a small bias from the binning of data for the histograms.⁶ The standard deviations of the free energies are slightly higher for EMUS than for the other two algorithms, but these differences are negligible compared with those expected for physically weighted simulations. Moreover, the relative performances are likely to be problem dependent. We note that ZH is based on thermodynamic integration, and the finite number of integration points causes quadrature error,²⁸ and, in turn, a systematic error in the barrier height.

FIG. 1.

Comparison of umbrella sampling methods applied to simulation data for the alanine dipeptide. (a) Average window free energies, G_i, for the indicated methods. Error bars are estimated standard deviations of the means. (b) Standard deviation of each method relative to that of the WHAM algorithm. Colors are the same as in (a). (c) EMUS as the first step in a self-consistent iteration to solve the MBAR equations (see text). The number of uncorrelated samples in each window (n_i) was estimated by calculating the integrated autocorrelation of the ϕ dihedral angle from each trajectory. Results shown are for identical molecular dynamics data (see text for simulation details); the methods differ only with respect to combination of the data to estimate the free energies.

In Figure 1(c), we apply the self-consistent iteration described in Section IV. For this calculation, we estimate the number of independent samples in each window (n_i) from the integrated autocorrelation time of the ϕ dihedral angle time series. We plot the standard deviation of the values of z calculated after the first iteration (EMUS), the second iteration, and after convergence to a relative residual smaller than 10⁻⁶. In general, convergence is achieved after an average of 9 iterations; none of the 100 data sets required more than 15 iterations. However, we note that after two iterations, the estimates of z already have a standard deviation equivalent to that of the WHAM algorithm. In this article we focus on the scheme corresponding to the first iteration only and do not attempt to analyze the improvement due to multiple iterations. In our tests the performance gain from multiple iterations is negligible compared to the improvement over direct approximation of free energy differences using long unbiased trajectories.

VI. JUSTIFICATION FOR UMBRELLA SAMPLING BY SCALING ARGUMENTS

The quality of a statistical estimate from umbrella sampling depends strongly on the choices made for the simulation windows. In this section, we discuss how the error scales as properties of the simulation change. We begin in Section VI A with a description of a prevalent justification for the use of US. We show in Section VI B that this argument is incomplete and, in turn, misleading. In Section VI C, we provide an alternative justification; namely, we show that in the low temperature limit, the cost to achieve a fixed accuracy by US grows slowly compared to direct simulation. In this section we make several simplifying assumptions that allow us to draw precise conclusions about the scaling properties of EMUS. In Section VII we provide error bounds for EMUS under much more general assumptions.

A. Scaling in the limit of many windows

To justify umbrella sampling, it is often suggested that the total computational time required to accurately sample statistics is inversely proportional to the number of windows, L.^17,29–32 The argument for this scaling proceeds as follows.

• •Divide a one-dimensional collective variable space into L windows of equal length, inversely proportional to L (i.e., L⁻¹).
• •Assume the windows are small enough that no free energy barriers exist in each window. The time to explore a window should be diffusion limited and proportional to the length of the window squared. Therefore, the simulation time required to accurately sample statistics in one window is also proportional to L⁻².
• •Because there are L windows, the total simulation time required to compute averages to fixed accuracy should scale as L × L⁻² = L⁻¹.

While this argument is now standard,^17,29–32 Virnau and Müller³³ observed that the error for computing the free energy difference between phases of Lennard-Jones particles with an approximately fixed amount of sampling was insensitive to the number of windows in practice, and they noted that the argument above neglects the error associated with combining the data from different simulation windows. Nguyen and Minh recently made a similar suggestion for a related class of methods.³⁴ This intuition is supported by our analysis in Section VI B, which shows that the total computational cost to achieve a fixed accuracy should be at best insensitive to the choice of L, so long as it is sufficiently large.

B. A simple model problem

To perform a more precise analysis, we make a number of simplifying assumptions. We emphasize that these assumptions are in force only for the purposes of the scaling arguments in this section. We provide more general error bounds for EMUS in Section VII.

Assumption 6.1.

The total computation time, N, is divided equally among the windows such that N_i = N/L.

Assumption 6.2.

The ψ_i are functions on the one-dimensional interval [0, 1], and the set of points where ψ_i is non-zero, {q :  ψ_i > 0}, is an interval of length |{q :  ψ_i > 0}| ≤ γ/L. We also assume that ψ_iψ_j = 0 unless |j − i| ≤ 1. Consequently, both the exact matrix F and the sample mean $\overline{F}$ are tri-diagonal. This assumption clearly does not hold when the bias functions ψ_i are Gaussian. Nonetheless, the rapid decay of Gaussian bias functions away from their peaks guarantees that entries of F and $\overline{F}$ far from the diagonal are very small, such that we expect our conclusions to still hold (though their justification would be more complicated).

Assumption 6.3.

The overlap of ψ_i and ψ_i±1 (i.e., the integral of their product) is large enough that

$${\displaystyle min\{F_{i,i+1},F_{i,i-1}\}>\delta >0}$$ (35)

for all L and for all i ≤ L. If our last assumption holds, but this one does not, then we can find more than one vector z satisfying Equations (17) and (18). This assumption is a slightly stronger version of the notion of irreducibility that we defined earlier (see Section III B). Note that we require the irreducibility to hold uniformly in the large L limit, and we thus introduce the δ, which is independent of L.

Assumption 6.4.

Sample averages computed in different windows are independent, i.e., ${\overline{F}}_{i,i\pm 1}$ and ${\overline{F}}_{j,j\pm 1}$ for j ≠ i are independent. We do not assume (here or anywhere else in this paper) that samples generated within a single window are independent. Indeed, even if the samples from π_i are independent, ${\overline{F}}_{i,i+1}$ and ${\overline{F}}_{i,i-1}$ are dependent random variables.

As an example average, let us consider the error in the free energy difference between the first and last windows,

$${\displaystyle \mathrm{\Delta }{\overline{G}}_{L,1}=-k_{B}Tln\left(\frac{z_L}{z_1}\right).}$$ (36)

Assumption 6.2 is sufficient for $z\left(\overline{F}\right)$ to be in detailed balance with $\overline{F}$ (Kelly Ref. 35, Lemma 1.5 and Section 1.3),

$${\displaystyle z_{i+1}\left(\overline{F}\right)=z_i\left(\overline{F}\right)\frac{{\overline{F}}_{i,i+1}}{{\overline{F}}_{i+1,i}}.}$$ (37)

Using (37) recursively,

$${\displaystyle \mathrm{\Delta }{\overline{G}}_{L,1}=-k_{B}Tln\left(\frac{\prod _{i=1}^{L-1}{\overline{F}}_{i,i+1}}{\prod _{i=1}^{L-1}{\overline{F}}_{i+1,i}}\right)=-k_{B}Tln{\overline{F}}_{1,2}+k_{B}Tln{\overline{F}}_{L,L-1}+k_{B}T\sum _{i=2}^{L-1}ln\left(\frac{{\overline{F}}_{i,i+1}}{{\overline{F}}_{i,i-1}}\right).}$$ (38)

To understand the error (variance) of the terms in (38), we must further specify F_i,i+1 and F_i,i−1.

Assumption 6.5.

For N_min and L_min sufficiently large, when N_i ≥ N_min and L ≥ L_min,

$${\displaystyle \frac{K_{min}}{N_i{L}^2}\le \mathrm{var}\left(ln\left(\frac{{\overline{F}}_{i,i+1}}{{\overline{F}}_{i,i-1}}\right)\right)\le \frac{K_{max}}{N_i{L}^2}}$$ (39)

for i = 2, 3, …, L − 1, and the same upper and lower bounds hold for $\mathrm{var}\left(ln{F}_{1,2}\right)$ and $\mathrm{var}\left(ln{F}_{L,L-1}\right)$. This is just a precise interpretation of the diffusion limited sampling assumption made in the standard justification of US reproduced in Section VI A. Under such an assumption we expect both ${\overline{F}}_{i,i+1}$ and ${\overline{F}}_{i,i-1}$ to have variance on the order of 1/(N_iL²) and, in light of (35), the function ln(x/y) is smooth near (x, y) = (F_i,i+1, F_i,i−1). These considerations are closely related to Lemma 7.2 in Section VII A.

With all the assumptions in hand, we now complete the argument by taking the variance of both sides of (38). Since samples from different windows are independent, the variance of ${\overline{G}}_{L,1}$ is a sum of contributions from each window,

$${\displaystyle \mathrm{var}\left(\mathrm{\Delta }{\overline{G}}_{L,1}\right)=\mathrm{var}\left(k_{B}Tln{\overline{F}}_{1,2}\right)+\mathrm{var}\left(k_{B}Tln{\overline{F}}_{L,L-1}\right)+\sum _{i=2}^{L-1}\mathrm{var}\left(k_{B}Tln\left(\frac{{\overline{F}}_{i,i+1}}{{\overline{F}}_{i,i-1}}\right)\right).}$$ (40)

Using (39) and substituting N_i = N/L, we find that, as long as N/L ≥ N_min and L ≥ L_min,

$${\displaystyle {\left(k_{B}T\right)}^2\frac{K_{min}}{N}\le \mathrm{var}\left(\mathrm{\Delta }{\overline{G}}_{L,1}\right)\le {\left(k_{B}T\right)}^2\frac{K_{max}}{N}.}$$ (41)

To verify that this conclusion carries over to harmonic bias potentials, we performed multiple umbrella sampling calculations for a Brownian particle on a flat potential on the interval $\left[0,1\right]$ with a stepsize of 1.0 × 10⁻⁶ and k_BT = 1.0 using Gaussian bias functions with a standard deviation of 1/L. The number of windows was varied from L = 10 to 46 in steps of 2. For each value, a total of 10⁷ steps were distributed equally in the windows, and the US calculation was repeated 480 times. We then calculated the mean square error of the free energy difference between the first and last window over the 480 replicates and determined how the mean square error scaled with L. Rather than the mean square error varying inversely with the number of windows, the data plotted in Figure 2 support a scaling of L⁰, consistent with (41).

FIG. 2.

The scaling of umbrella sampling error with number of windows on a flat potential. A Brownian particle on a flat, one-dimensional potential was simulated for 480 identical runs, and the free energy difference between the first and last windows was calculated, as described in the text. Here, the mean square error from the exact result is plotted against the number of windows. The lines show the scaling in error predicted by the L⁻¹ and L⁰ scalings. Fitting the data on a log-log scale give a scaling exponent of −0.026 ± 0.028.

It is worth noting that the inverse scaling with total cost N in (41) is exactly the scaling one would expect for the variance of an estimate of the free energy difference $\mathrm{\Delta }{\overline{G}}_{L,1}$ constructed from a molecular dynamics trajectory of length N. Because US and direct simulations of comparable total numbers of steps require comparable computational effort (ignoring the overhead associated with combining the simulation data, which is typically small in comparison with the computational cost of the sampling), the benefits of US must be encoded in the constants K_min and K_max. A dramatic demonstration of this observation is the purpose of Section VI C.

C. The low temperature limit

To understand the benefits of umbrella sampling, we must study its performance in the presence of free energy barriers. In particular, we compare the performance of umbrella sampling to physically weighted sampling as the temperature goes to zero. In this limit, the cost of direct sampling increases exponentially with 1/T, while, as we show, the cost of umbrella sampling increases only algebraically. A formal discussion is given in a separate publication;³⁶ here, we present a simple plausibility argument.

Owing to the free energy barriers, the assumption of diffusive dynamics in each window no longer holds. Instead, we expect a form typical of reaction rate theories in each window. We define ΔW_i as the maximum difference in the PMF in window i,

$${\displaystyle \mathrm{\Delta }{W}_i=\underset{\{q:\hspace{0.17em}\underset{i}{\psi }\left(q\right)>0\}}{max}\left\{W\left(q\right)\right\}-\underset{\{q:\hspace{0.17em}\underset{i}{\psi }\left(q\right)>0\}}{min}\left\{W\left(q\right)\right\}.}$$ (42)

Assumption 6.5 ’:

We now replace the upper and lower bounds in (39) by the upper bound

$${\displaystyle \mathrm{var}\left(ln\left(\frac{{\overline{F}}_{i,i+1}}{{\overline{F}}_{i,i-1}}\right)\right)\le \frac{K}{k_{B}T{N}_i{L}^2}exp\left(\frac{\mathrm{\Delta }{W}_i}{k_{B}T}\right)}$$ (43)

for i = 2, 3, …, L − 1 with analogous replacements for i = 1 and i = L, as long as N_i ≥ N_min and L ≥ L_min. The constant K here is assumed to be independent of temperature. This bound captures the diffusion limited sampling assumption when L is very large, but is more detailed than (39) in that it captures (crudely) the increasing difficulty of the sampling problem as the temperature decreases with all other parameters held fixed. Under reasonable additional assumptions on the underlying potential, the bias functions ψ_i, and the sampling scheme, one can rigorously establish an asymptotic (large N_i) bound of the form in (43).³⁶

Substituting this new bound into (40), we find that, if L ≥ L_min and N/L ≥ N_min, then

$${\displaystyle \mathrm{var}\left(\mathrm{\Delta }{\overline{G}}_{L,1}\right)\le \frac{K{k}_{B}T}{NL}\sum _{i=1}^{L}exp\left(\frac{\mathrm{\Delta }{W}_i}{k_{B}T}\right).}$$ (44)

As the temperature decreases, we choose to increase L such that ΔW_i/k_BT is bounded above. This can be achieved by scaling L linearly with 1/T: if the derivative of the PMF is bounded (in absolute value) by $W_{max}^{\prime }$, choosing L so that

$${\displaystyle L\ge \frac{W_{max}^{\prime }}{\mathrm{\Omega }{k}_{B}T}}$$ (45)

ensures that ΔW_i/k_BT is bounded by Ω (since we have assumed that the argument of W is in [0, 1]). On the other hand, our assumption that the length of {q :  ψ_i > 0} (Assumption 6.2) does not exceed γ/L implies that

$${\displaystyle \frac{\mathrm{\Delta }{W}_i}{k_{B}T}\le \frac{W_{max}^{\prime }\gamma }{k_{B}TL}.}$$ (46)

Consequently, as long as (45) holds,

$${\displaystyle \frac{\mathrm{\Delta }{W}_i}{k_{B}T}\le \mathrm{\Omega }\gamma .}$$ (47)

Finally, substituting this result into (44) we find that if L ≥ L_min and N/L ≥ N_min, then

$${\displaystyle \mathrm{var}\left(\mathrm{\Delta }{\overline{G}}_{L,1}\right)\le \frac{K{k}_{B}Texp\left(\mathrm{\Omega }\gamma \right)}{N}\le \frac{KL{k}_{B}Texp\left(\mathrm{\Omega }\gamma \right)}{N_{min}}.}$$ (48)

With the best possible (smallest) choice of L allowed by (45), this bound becomes

$${\displaystyle \mathrm{var}\left(\mathrm{\Delta }{\overline{G}}_{L,1}\right)\le \frac{K{W}_{max}^{\prime }exp\left(\mathrm{\Omega }\gamma \right)}{\mathrm{\Omega }{N}_{min}}.}$$ (49)

The remarkable feature of the bound in (49) is that it is independent of T. This does not mean that the cost to achieve a fixed accuracy is independent of T. However, it does imply that as the temperature is decreased, we do not have to increase N_min to maintain a fixed accuracy. Expression (45) and the fact that N_i ≥ N_min together imply that, under the assumptions of this section, the computational cost of obtaining an accurate estimate of $\mathrm{\Delta }{\overline{G}}_{L,1}$ by US increases algebraically with (k_BT)⁻¹. That scaling is to be compared to exponential in (k_BT)⁻¹ to achieve the same accuracy by direct simulation.

Finally, we remark that while our analysis provides a convincing explanation for the performance benefits offered by umbrella sampling, it neglects a number of practical realities. First, we have ignored the cost of equilibrating the simulation in each window, which can be challenging. Second, we have ignored the practical difficulties that arise when the number of windows grows large. The bias functions (restraints) can introduce additional free energy barriers that slow mixing in the degrees of freedom orthogonal to the collective variable, and, if sufficiently restrictive, could in principle necessitate modifying the elementary simulation step sizes. Our analysis reveals, however, that these difficulties are not responsible for the slow (L⁰) error scaling observed for large L.

VII. ANALYSIS OF THE ERROR OF EMUS

In this section, we study the error of EMUS in full generality, without imposing the simplifying assumptions of Section VI. Our main results are a central limit theorem for EMUS (Theorem 7.4) and an easily computed, practical error estimator which reveals the contributions of the different windows to the total error. These results may be used to compare the efficiency of EMUS and other methods and to study how the efficiency of EMUS depends on parameters such as the number of samples allocated to each window.

A. A central limit theorem for EMUS

Before developing the error analysis, we define a single notation for EMUS which incorporates both the case of a free-energy difference and the case of an ensemble average. In either case, one must compute $\overline{F}$ and also ${\overline{g}}_{1,i}^{*}$ and ${\overline{g}}_{2,i}^{*}$ for two real valued functions g₁ and g₂. To compute a free energy difference, we choose based on (3),

$${\displaystyle g_1={\mathbb{1}}_{S1}\text{and}{g}_2={\mathbb{1}}_{S2}.}$$ (50)

To compute an ensemble average 〈g〉, we choose based on (16),

$${\displaystyle g_1=g\text{and}{g}_2=1.}$$ (51)

We furthermore define the function

$${\displaystyle v_i\left(x\right)=\left({\psi }_1^{*}\left(x\right),\dots ,{\psi }_L^{*}\left(x\right),g_{1,i}^{*}\left(x\right),g_{2,i}^{*}\left(x\right)\right)}$$ (52)

so that

$${\displaystyle {\overline{v}}_i=\frac{1}{N_i}\sum _{t=0}^{N_i-1}{v}_i\left(X_t^i\right)=\left({\overline{F}}_{i1},\dots ,{\overline{F}}_{iL},{\overline{g}}_{1,i}^{*},{\overline{g}}_{2,i}^{*}\right),}$$ (53)

where we remind the reader that, for each i, the process $X_t^i$ samples the biased distribution π_i. Define

$${\displaystyle \overline{v}=({\overline{v}}_1,\dots ,{\overline{v}}_L),}$$ (54)

and let

$${\displaystyle 〈v〉=\left({〈v_1〉}_1,\dots ,{〈v_L〉}_L\right)}$$ (55)

denote the corresponding vector of exact averages. Using the notation defined in Section III B, the EMUS estimator takes the form $B\left(\overline{v}\right)$, where for a free-energy difference,

$${\displaystyle B\left(\overline{v}\right)=-k_{B}Tln\left(\frac{\sum _{i=1}^L{z}_i\left(\overline{F}\right){\overline{g}}_{1,i}^{*}}{\sum _{i=1}^L{z}_i\left(\overline{F}\right){\overline{g}}_{2,i}^{*}}\right),}$$ (56)

and for an ensemble average,

$${\displaystyle B\left(\overline{v}\right)=\frac{\sum _{i=1}^L{z}_i\left(\overline{F}\right){\overline{g}}_{1,i}^{*}}{\sum _{i=1}^L{z}_i\left(\overline{F}\right){\overline{g}}_{2,i}^{*}}.}$$ (57)

We now proceed with the error analysis. First, we characterize the error of the sample means over the biased distributions. As discussed by Frenkel and Smit (Ref. 17, Appendix D), the variance of a sample mean may be expanded in terms of the integrated autocovariance of the process. We define the autocovariance function of $v_i\left(X_t^i\right)$ to be

$${\displaystyle C_i\left(t\right)={〈(v_i\left(X_0^i\right)-{〈v_i〉}_i){(v_i\left(X_t^i\right)-{〈v_i〉}_i)}^{\mathrm{T}}〉}_i,}$$ (58)

where T denotes a vector transpose, and here the outer 〈…〉_i denotes the exact average not only over $X_0^i$ sampled from π_i but also subsequent points of the sequence $X_t^i$. Note that C_i(t) is a (L + 2) × (L + 2) matrix. We define the integrated autocovariance to be

$${\displaystyle {\mathrm{\Sigma }}_i=\sum _{t=-\infty }^{\infty }{C}_i\left(t\right).}$$ (59)

The integrated autocovariance is the leading order coefficient in an expansion of the covariance ${\overline{v}}_i$ (see Ref. 17, D.1.3),

$${\displaystyle \mathrm{cov}\left({\overline{v}}_i\right)=\frac{{\mathrm{\Sigma }}_i}{N_i}+o\left(\frac{1}{N_i}\right),}$$ (60)

where o(1/N_i) denotes terms that go to zero faster than 1/N_i (i.e., N_io(1/N_i) → 0).

Under certain conditions on the process $X_t^i$, one can strengthen the expansion of the covariance (60) to a central limit theorem (CLT) for ${\overline{v}}_i$. We expect such a CLT to hold for most problems and most sampling methods in computational statistical physics. However, to avoid a lengthy and technical digression, we simply take the CLT as an assumption; we justify this assumption in more detail in another work,³⁶ and we refer to the work of Lelièvre et al. (Ref. 37, Section 2.3.1.2) for a general discussion of the CLT in the context of computational statistical physics.

Assumption 7.1 Central limit theorem for ${\overline{v}}_i$ —

We assume that

$${\displaystyle \sqrt{N_i}\left({\overline{v}}_i-〈v_i〉\right)\stackrel{\mathrm{d}}{\to }\mathbf{N}\left(0,{\mathrm{\Sigma }}_i\right),}$$ (61)

where Σ_i ∈ ℝ^(L+2)×(L+2) is the integrated autocovariance matrix defined in (59). The symbol $\stackrel{\mathrm{d}}{\to }$ denotes convergence in distribution as N_i → ∞. Notice that when the elements of the sequence $X_t^i$ are independent and drawn from π_i then ${\mathrm{\Sigma }}_i={〈(v_i-{〈v_i〉}_i){(v_i-{〈v_i〉}_i)}^{\mathrm{T}}〉}_i/N_i$. More generally, samples are correlated, so Σ_i includes a factor that accounts for the time to decorrelate.

Having characterized the errors in the sample means, we now study how these errors propagate through the EMUS algorithm. Our goal is to prove a CLT for EMUS. We accomplish this using the delta method.

Lemma 7.2 The delta method; Proposition 6.2 of Bilodeau and Brenner³⁸ —

Let θ_N be a sequence of random variables taking values in ℝ^d. Assume that a central limit theorem holds for θ_N with mean μ ∈ ℝ^d and asymptotic covariance matrix Σ ∈ ℝ^d×d; that is, assume

$${\displaystyle \sqrt{N}({\theta }_N-\mu )\stackrel{\mathrm{d}}{\to }\mathbf{N}(0,\mathrm{\Sigma }).}$$ (62)

Let Φ : ℝ^d → ℝ be a function differentiable at μ. Then we have the central limit theorem

$${\displaystyle \sqrt{N}\left(\mathrm{\Phi }\left({\theta }_N\right)-\mathrm{\Phi }\left(\mu \right)\right)\stackrel{\mathrm{d}}{\to }\mathbf{N}\left(0,\nabla {\mathrm{\Phi }}^{\mathrm{T}}\left(\mu \right)\mathrm{\Sigma }\nabla \mathrm{\Phi }\left(\mu \right)\right)}$$ (63)

for the sequence of random variables Φ(θ_N).

To motivate the delta method, we observe that if X has distribution N(μ, Σ), then ∇Φ(μ)^TX has distribution $\mathbf{N}\left(\mathrm{\Phi }\left(\mu \right),\nabla {\mathrm{\Phi }}^{\mathrm{T}}\left(\mu \right)\mathrm{\Sigma }\nabla \mathrm{\Phi }\left(\mu \right)\right)$. That is, according to the delta method, the asymptotic distribution of Φ(X) is the linearization of Φ at μ applied to the asymptotic distribution of X. Thus, one may regard the delta method as a rigorous version of the standard error propagation formula based on linearization.

We prove the CLT for EMUS by applying the delta method with $\overline{v}$ taking the place of θ_N and with the function B taking the place of Φ. We require the following assumptions in addition to Assumption 7.1.

Assumption 7.3.

We assume the following:

• 1.
  The proportion of the total number of samples drawn from each window is constant in the limit as N → ∞; that is,

  $${\displaystyle \underset{N\to \infty }{lim}{N}_i/N={\kappa }_i.}$$ (64)
• 2.Sampling in different windows is independent; that is, ${\overline{v}}_i$ is independent of ${\overline{v}}_j$ when j ≠ i.
• 3.The biasing functions ψ_i are chosen so that F is irreducible; see Section III B.

We now give the CLT for EMUS.

Theorem 7.4 Central Limit Theorem for EMUS —

Let Assumptions 7.1 and 7.3 hold. Let

$${\displaystyle \frac{\partial B}{\partial {\overline{v}}_i}=\left(\frac{\partial B}{\partial {\overline{F}}_{i1}},\dots ,\frac{\partial B}{\partial {\overline{F}}_{iL}},\frac{\partial B}{\partial {\overline{g}}_{1,i}^{*}},\frac{\partial B}{\partial {\overline{g}}_{2,i}^{*}}\right)\in {\mathbb{R}}^{L+2}}$$ (65)

denote the partial derivative of B with respect to ${\overline{v}}_i$. Under the assumptions stated above,

$${\displaystyle \sqrt{N}\left(B\left(\overline{v}\right)-B\left(〈v〉\right)\right)\stackrel{\mathrm{d}}{\to }\mathbf{N}(0,{\sigma }^2),}$$ (66)

where

$${\displaystyle {\sigma }^2=\sum _{i=1}^L\frac{1}{{\kappa }_i}\left({\frac{\partial B}{\partial {\overline{v}}_i}}^{\mathrm{T}}{\mathrm{\Sigma }}_i\frac{\partial B}{\partial {\overline{v}}_i}\right).}$$ (67)

We refer to σ² as the asymptotic variance of EMUS.

Proof.

First, we write down a central limit theorem for $({\overline{v}}_1,\dots ,{\overline{v}}_L)$. We have that

$${\displaystyle \sqrt{N}({\overline{v}}_i-{〈v_i〉}_i)\stackrel{\mathrm{d}}{\to }\mathbf{N}(0,{\kappa }_i^{-1}{\mathrm{\Sigma }}_i)}$$ (68)

by Assumption 7.1 and (64). Since the sampling in different windows is assumed to be independent, (68) implies

$${\displaystyle \sqrt{N}(\overline{v}-〈v〉)\stackrel{\mathrm{d}}{\to }\mathbf{N}(0,\mathrm{\Sigma }),}$$ (69)

where Σ ∈ ℝ^{L(L+2)×L(L+2)} is the block diagonal matrix

$${\displaystyle \mathrm{\Sigma }=\left[\begin{array}{cccc}\hfill {\mathrm{\Sigma }}_1/{\kappa }_1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \dots \hfill \\ \hfill 0\hfill & \hfill {\mathrm{\Sigma }}_2/{\kappa }_2\hfill & \hfill 0\hfill & \hfill \dots \hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill {\mathrm{\Sigma }}_3/{\kappa }_3\hfill & \hfill \dots \hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill \end{array}\right].}$$ (70)

Second, we verify that B is differentiable at $\overline{v}$. Since F is assumed to be an irreducible stochastic matrix, $z\left(\overline{F}\right)$ is differentiable at $\overline{F}$. We refer to Thiede et al.³⁹ Lemma 3.1 for a complete explanation. It follows from the chain rule that B is differentiable at $\overline{v}$.

Finally, applying Lemma 7.2 with B playing the role of Φ and $\overline{v}$ the role of θ_N concludes the proof.□

The asymptotic variance σ² appearing in Theorem 7.4 measures the rate at which the error of EMUS decreases with the number of samples. To make this precise, we observe that Theorem 7.4 is equivalent to the following asymptotic result concerning confidence intervals. For every α > 0,

$${\displaystyle \underset{N\to \infty }{lim}\mathbf{P}\left[\left|B\left(\overline{v}\right)-B\left(〈v〉\right)\right|\le \frac{\alpha \sigma }{\sqrt{N}}\right]=\mathrm{erf}\left(\frac{\alpha }{\sqrt{2}}\right),}$$ (71)

where P denotes a probability and erf denotes the error function.

The asymptotic variance is commonly used to measure the efficiency of an estimator. We refer to the work of van der Vaart⁴⁰ for an explanation and for a discussion of other possibilities. In Section VII B, we explain how the proportion κ_i of samples allocated to each window may be adjusted to minimize the asymptotic variance of EMUS, thereby maximizing efficiency.

We note that a central limit theorem similar to Theorem 7.4 has been proved for the MBAR estimator by Gill et al. (Ref. 4, Proposition 2.2). However, the authors of this work do not study the dependence of the asymptotic variance on the parameters, as we do. In fact, the MBAR estimator is significantly more complicated than EMUS, and its dependence on the number of windows and the allocation of samples is harder to understand.

We³⁶ use a result similar to Theorem 7.4 to generalize the conclusions of Section VI to periodic and multi-dimensional reaction coordinates and to a wider class of observables than free energy differences. We show both that the asymptotic variance is constant in the limit of large L and that the work required to compute an average to fixed precision increases only algebraically in the low temperature limit. In addition, we use recently developed perturbation estimates for Markov chains³⁹ to quantify the dependence of the asymptotic variance of EMUS on the degree to which the bias functions overlap.

B. Estimating the asymptotic variance of EMUS

Our goal in this section is to derive a computable estimate ${\overline{\sigma }}^2$ of the asymptotic variance σ², which can be decomposed to assess the contributions from individual windows to errors in averages. We recall that formula (67) for σ² involves partial derivatives of B. Our estimate ${\overline{\sigma }}^2$ of σ² requires explicit formulas for these partial derivatives. We provide the appropriate expressions, both for ensemble averages and for free-energy differences, in Lemma 7.5. Following the partial derivatives, we present an algorithm for evaluating ${\overline{\sigma }}^2$ and demonstrate it for the alanine dipeptide. Finally, we compare with the output of a procedure from Zhu and Hummer (ZH)¹² in Section VII B 3.

Lemma 7.5.

We have the following formulas for $\partial B/\partial {\overline{v}}_i$:

• 1.
  When EMUS is used to compute an ensemble average 〈g〉, B is defined by (57), and we have

  $${\displaystyle \frac{\partial B}{\partial {\overline{F}}_{ij}}\left(\overline{v}\right)=\frac{\sum _k{z}_i\left(\overline{F}\right){(I-\overline{F})}_{jk}^{\#}({\overline{g}}_k^{*}-B\left(\overline{v}\right){\overline{1}}_k^{*})}{\sum _k{z}_k\left(\overline{F}\right){\overline{1}}_k^{*}},}$$ (72)

  $${\displaystyle \frac{\partial B}{\partial {\overline{g}}_{1,i}^{*}}\left(\overline{v}\right)=\frac{z_i\left(\overline{F}\right)}{\sum _k{z}_k\left(\overline{F}\right){\overline{1}}_k^{*}},\mathit{\text{and}}}$$ (73)

  $${\displaystyle \frac{\partial B}{\partial {\overline{g}}_{2,i}^{*}}\left(\overline{v}\right)=-\frac{B\left(\overline{v}\right)z_i\left(\overline{F}\right)}{\sum _k{z}_k\left(\overline{F}\right){\overline{1}}_k^{*}},}$$ (74)

  where # denotes the group inverse (see Ref. 13 and discussion below).
• 2.
  When EMUS is used to compute a free-energy difference, B is defined by (56), and we have

  $${\displaystyle \frac{\partial B}{\partial {\overline{F}}_{ij}}\left(\overline{v}\right)=k_{B}T{z}_i\left(\overline{F}\right)\left(\frac{\sum _k{(I-F)}_{jk}^{\#}{\overline{\mathbb{1}}}_{S2,k}^{*}}{\sum _k{z}_k\left(\overline{F}\right){\overline{\mathbb{1}}}_{S2,k}^{*}}\right.\left.-\frac{\sum _k{(I-F)}_{jk}^{\#}{\overline{\mathbb{1}}}_{S1,k}^{*}}{\sum _k{z}_k\left(\overline{F}\right){\overline{\mathbb{1}}}_{S1,k}^{*}}\right),}$$ (75)

  $${\displaystyle \frac{\partial B}{\partial {\overline{g}}_{1,i}^{*}}\left(\overline{v}\right)=k_{B}T\frac{z_i\left(\overline{F}\right)}{\sum _k{z}_k\left(\overline{F}\right){\overline{\mathbb{1}}}_{S1,k}^{*}},\mathit{\text{and}}}$$ (76)

  $${\displaystyle \frac{\partial B}{\partial {\overline{g}}_{2,i}^{*}}\left(\overline{v}\right)=-k_{B}T\frac{z_i\left(\overline{F}\right)}{\sum _k{z}_k\left(\overline{F}\right){\overline{\mathbb{1}}}_{S2,k}^{*}}.}$$ (77)
• 3.
  When EMUS is used to compute the free energy of the k th window, B is defined by (11), and we have

  $${\displaystyle -k_{B}T\frac{\partial B}{\partial {\overline{F}}_{ij}}\left(\overline{v}\right)=\frac{z_i\left(\overline{F}\right)}{z_k\left(\overline{F}\right)}{(I-\overline{F})}_{jk}^{\#}\mathit{\text{and}}}$$ (78)

  $${\displaystyle \frac{\partial B}{\partial {\overline{g}}_{i,1}^{*}}\left(\overline{v}\right)=\frac{\partial B}{\partial {\overline{g}}_{i,2}^{*}}\left(\overline{v}\right)=0.}$$ (79)

  Note that for a free energy difference between windows, we can simply subtract derivatives for the corresponding windows.

Proof.

We begin by reminding the reader that the output of EMUS is the vector of window normalization constants, z, which depends on the sample mean $\overline{F}$. Because all other averages and, in turn, their derivatives rely on z, we need to determine the sensitivity of each element of z to each element of $\overline{F}$ (i.e., $\partial z_k/\partial {\overline{F}}_{ij}$). Since $\overline{F}$ is a stochastic matrix, some care must be taken in defining this derivative. We resolve the technical difficulties in detail elsewhere; see Ref. 36 and Ref. 39 (Lemma 3.1). Here, to obtain the derivative $\partial z_k/\partial {\overline{F}}_{ij}$, evaluated at $\overline{F}$, we perturb around $\overline{F}$,

$${\displaystyle {\left.\frac{d}{d\epsilon }\right|}_{\epsilon =0}{z}_k(\overline{F}+\epsilon E)=\sum _{i,j=1}^L\frac{\partial z_k}{\partial {\overline{F}}_{ij}}\left(\overline{F}\right)E_{ij},}$$ (80)

where E is an arbitrary matrix, ε is a scalar, and we assume that the sum $\overline{F}+\epsilon E$ is also a stochastic matrix. The right hand side follows from the chain rule, effectively treating each element of the matrix as a separate argument to each element z. Then, we employ a relation from Golub and Meyer (Ref. 13, Theorem 3.1),

$${\displaystyle {\left.\frac{d}{d\epsilon }\right|}_{\epsilon =0}{z}_k(\overline{F}+\epsilon E)=z{\left(\overline{F}\right)}^{\mathrm{T}}E{(I-\overline{F})}^{\#}{e}_k,}$$ (81)

where # denotes the group inverse, a generalized matrix inverse similar to the Moore-Penrose inverse. It is defined as satisfying AA^#A = A, A^#AA^# = A^#, AA^# = A^#A. We refer to Golub and Meyer¹³ for further discussion of the group inverse and an algorithm for computing it. Finally, we equate (80) and (81) and solve for the derivative of interest,

$${\displaystyle \frac{\partial z_k}{\partial {\overline{F}}_{ij}}\left(\overline{F}\right)=z_i\left(\overline{F}\right){(I-\overline{F})}_{jk}^{\#}.}$$ (82)

Thus the sensitivity of each element of z to each element of $\overline{F}$ can be computed from linear algebra operations.

With (82), we can now compute the derivatives of B. We derive the formulas for the free-energy difference explicitly; the other cases are similar. In this case,

$${\displaystyle B\left(\overline{v}\right)=k_{B}Tln\left(\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{2,k}^{*}\right)-k_{B}Tln\left(\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{1,k}^{*}\right).}$$ (83)

By the chain rule,

$${\displaystyle \frac{\partial }{\partial {\overline{g}}_{1,i}^{*}}ln\left(\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{1,k}^{*}\right)=\frac{z_i\left(\overline{F}\right)}{\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{1,k}^{*}}}$$ (84)

and

$${\displaystyle \frac{\partial }{\partial {\overline{F}}_{ij}}ln\left(\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{1,k}^{*}\right)=\frac{\sum _{k=1}^L\frac{\partial z_k}{\partial {\overline{F}}_{ij}}\left(\overline{F}\right)\underset{1,k}{\overset{*}{\overline{g}}}}{\sum _{k=1}^L{z}_k\left(\overline{F}\right){\overline{g}}_{1,k}^{*}}.}$$ (85)

The stated result follows by substituting g₁ = 1_S1, g₂ = 1_S2, and the expression in (82) for $\partial z_k/\partial {\overline{F}}_{ij}$.□

1. Computational procedure

We now provide a practical procedure that uses the derivatives above to estimate σ² from trajectories that sample the distributions π_i. For clarity, we assume that the system is equilibrated (i.e., $X_0^i$ has distribution π_i, so that the process $X_t^i$ is stationary) throughout this section.

We begin by rewriting (67) as

$${\displaystyle {\sigma }^2=\sum _{i=1}^L\frac{{\chi }_i^2}{{\kappa }_i},}$$ (86)

where

$${\displaystyle {\chi }_i^2={\frac{\partial B}{\partial {\overline{v}}_i}}^{\mathrm{T}}{\mathrm{\Sigma }}_i\frac{\partial B}{\partial {\overline{v}}_i}={\frac{\partial B}{\partial {\overline{v}}_i}}^{\mathrm{T}}\left(\sum _{t=-\infty }^{\infty }{C}_i\left(t\right)\right)\frac{\partial B}{\partial {\overline{v}}_i}.}$$ (87)

Defining the sequence

$${\displaystyle {\zeta }_t^i={\left.\frac{\partial B}{\partial {\overline{v}}_i}\right|}_{〈v〉}\cdot \left(v_i\left(X_t^i\right)-{〈v_i〉}_i\right)}$$ (88)

we find that

$${\displaystyle {\chi }_i^2=\sum _{t=-\infty }^{\infty }〈{\zeta }_t^i{\zeta }_0^i〉}$$ (89)

which is the integrated autocovariance of ${\zeta }_t^i$.

We thus propose the following algorithm, given simulation data:

• 1.Compute $\overline{v}$.
• 2.Compute $z\left(\overline{F}\right)$ and ${(I-\overline{F})}^{\#}$ using the algorithm of Golub and Meyer.¹³
• 3.Evaluate $\partial B/\partial {\overline{v}}_i$ at $\overline{v}$ using the formulas in Lemma 7.5.
• 4.
  Compute

  $${\displaystyle {\overline{\zeta }}_t^i={\left.\frac{\partial B}{\partial {\overline{v}}_i}\right|}_{{\overline{v}}_i}\cdot (v_i\left(X_t^i\right)-{\overline{v}}_i).}$$ (90)
• 5.Compute an estimate ${\overline{\chi }}_i^2$ of the integrated autocovariance of ${\overline{\zeta }}_t^i$ using an algorithm such as ACOR.⁴¹
• 6.
  Compute the estimate of σ²,

  $${\displaystyle {\overline{\sigma }}^2=\sum _{i=1}^L\frac{{\overline{\chi }}_i^2}{{\kappa }_i}.}$$ (91)

Since $\overline{F}$, $\overline{v}$, and z are all computed in the process of obtaining the EMUS averages, estimating ${\overline{\sigma }}^2$ only requires one additional pass over the simulation data. This additional cost is insignificant compared with that of computing the trajectories.

Both (67) and its approximation (91) decompose the asymptotic variance of EMUS into a sum of contributions from each window. By comparing the sizes of terms in the sum, we can determine the degrees to which different windows contribute to the error. In principle, this information can be used to guide modification of the parameters of the simulation to improve efficiency. For instance, one might adjust the proportion of samples allocated to each window, κ_i, to minimize the asymptotic variance. From (86), the asymptotic variance σ² is minimized when κ_i ∝ χ_i (see Equation (42) of Ref. 12). Consequently, we can define the relative importance of window i as

$${\displaystyle {\mu }_i=L\frac{{\chi }_i}{\sum _{k=1}^L{\chi }_i},}$$ (92)

where the normalization is chosen so that μ_i = 1, regardless of L, if all windows have the same importance. The relative importance represents how many samples would be allocated to a window to optimally estimate a specific observable, compared to a uniform distribution over all windows.

2. Numerical results

To study the behavior of these estimates, we performed a two-dimensional umbrella sampling calculation with restraints on the ϕ and ψ dihedral angles of the alanine dipeptide. Parameters were the same as in the one-dimensional calculation above, with the addition of 20 bias functions in the ψ dihedral with the same force constant, creating a grid of 400 windows. Each window was equilibrated for 40 ps and sampled for a further 150 ps, with the collective variable values output every 10 fs.

In Figures 3 and 4(a), we plot the two-dimensional PMF from EMUS and the importances for the free energy difference between two windows located at the C₇ equatorial and C₇ axial configurations. Comparison shows that the importances are high for windows on low free energy pathways between the two windows of interest. Two such pathways exist. In the representation in Figure 3, one proceeds up and to the left of the C₇ equatorial basin and then (via the periodic boundaries) enters the C₇ axial basin through transition state 1 (TS1 in Figure 3). The other pathway proceeds down then right through transition state 2 (TS2 in Figure 3). Of these two pathways, the first has a lower free energy barrier. We observe that the EMUS importances are larger for windows located on this pathway. In contrast, windows off these pathways in regions with high free energies have very low importances presumably because, though sampling error may be large in those regions, they do not contribute significantly to the desired averages. We refer the reader to Thiede et al.³⁹ for a mathematical discussion of the sensitivities of the averages.

FIG. 3.

Potential of mean force obtained from US with biases on the ϕ and ψ dihedral angles. Major basins and barriers on pathways connecting them are indicated. Angles are measured in degrees. The scale bar indicates PMF values in kcal/mol, and the contour spacing is 2 k_BT. The surface is constructed from simulation data accumulated in histograms with 100 bins in each collective variable. See text for simulation details.

FIG. 4.

EMUS relative importances. Angles are measured in degrees. (a) Relative importances for the free energy difference between windows in the C₇ axial and C₇ equatorial basins. The window in the C₇ equatorial basin is centered at (–81°,81°), and the window in the C₇ axial basin at (63°,–63°). (b) Window importances for the free energy difference between windows in the C₇ axial basin and at TS1. Windows are centered at (63°,–63°) and (135°,−117°), respectively. (c) Importances for the free energy of the window at TS1. (d) Importances for the window in the C₇ axial basin.

We expect the importances to depend on the computed average. To illustrate that this is the case numerically, we show the log importances for the free energy difference between a window in the C₇ axial basin and one located on TS1 in Figure 4(b). Compared to Figure 4(a), the importances are higher in the C₇ axial basin and lower in the C₇ equatorial basin, which highlights that the importances depend on the average computed and do not simply mirror the free energy. In Figures 4(c) and 4(d), we plot the importances for estimating the window free energy (not the free energy difference) of the window on TS1 and the window in the C₇ axial basin, respectively. We note that the importances in the C₇ equatorial basin are higher in Figures 4(c) and 4(d) than in 4(b). This suggests that when the free energy difference between the two windows is considered, there is some cancellation of the errors arising in the C₇ equatorial basin.

3. Comparison with other algorithms for determining error contributions

Zhu and Hummer¹² proposed an algorithm for determining window free energies by calculating the mean restraining forces for each window and using thermodynamic integration to estimate free energy differences between adjacent windows. These are combined using least squares to calculate window free energies. Like EMUS, this algorithm allows one to construct error estimates that can be decomposed into contributions from individual windows. The authors give an expression for the error in the free energy of one window. This expression can be easily extended to the free energy difference between two windows, giving

$${\displaystyle \mathrm{var}\left(\mathrm{\Delta }{G}_{ji}\right)=\sum _k^L\mathrm{var}\left(\sum _{\alpha }^D\left(c_{jk\alpha }-c_{ik\alpha }\right){\overline{f}}_{k\alpha }\right),}$$ (93)

where ${\overline{f}}_{k\alpha }$ is the average force exerted by the bias function for window k in the αth dimension. The constants c_ikα and c_jkα are defined in Appendix A of Zhu and Hummer.¹² The authors propose that these error estimates are applicable to WHAM and other umbrella sampling algorithms.

Both (67) and (93) are sums of contributions from individual windows. Using the formalism introduced in Section VII B 1, we define the process

$${\displaystyle {\zeta }_t^{k,ZH}=\sum _{\alpha }^D\left(c_{jk\alpha }-c_{ik\alpha }\right)\left(f_{k\alpha }\left(X_t^k\right)-〈f_{k\alpha }〉\right),}$$ (94)

and ${\left({\chi }_i^{ZH}\right)}^2$ as the integrated autocovariance of ${\zeta }_t^{k,ZH}$. This allows us to define importance for the Zhu and Hummer algorithm analogously to those for EMUS (see (92)).

We applied the ZH error analysis to the two-dimensional umbrella sampling data used in Section VII B 2 and calculated the importances for the same free energy difference as in Figure 4(a) (Figure 5(a)). Rather than falling along the low free energy pathways, as for EMUS, the ZH importances mirror the autocorrelation times (Figure 5(b)). This indicates that windows have large ZH importances if they have large fluctuations in free energy. We thus see that different algorithms emphasize different windows in US. We can understand the behaviors of these two algorithms by considering (88) and (94). The factor $\partial B/\partial {\overline{v}}_i$ in (88) depends explicitly on the normalization constant for each window (see Lemma 7.5). By contrast, the factor $\left(c_{jk\alpha }-c_{ik\alpha }\right)$ in (94) depends only on the relative positions of the windows and not on their free energies.

FIG. 5.

Comparison to the work of Zhu and Hummer. (a) ZH estimates for the relative importances for the free energy difference between windows in the C₇ axial and C₇ equatorial basins. Compare with Figure 4(a). (b) Autocorrelation times of the trajectory ${\zeta }_t^{k,ZH}$ in each window. The largest value observed is 3 ps, but the scale is limited to 1 ps for visual clarity.

VIII. CONCLUSIONS

The success of an umbrella sampling simulation depends on the choice of windows (i.e., how the system is biased) and the estimator used to determine the normalization constants of the windows from trajectory data. Here, we show that the normalization constants can be obtained from an eigenvector of a stochastic matrix. This eigenvector method for umbrella sampling (EMUS) can be viewed as the first step in an implementation of the MBAR estimator. In our experience, this first step is nearly converged, and machine precision is reached in only a few iterations. Moreover, each iteration yields a consistent estimate. Most importantly, error analysis is considerably easier for EMUS than MBAR because the elements of the stochastic matrix do not depend on the normalization constants.

Within this framework, we revisited a common scaling argument for justifying umbrella sampling and showed that once the number of windows becomes sufficiently large, the scheme does not benefit from the addition of more windows (i.e., the variance is not further reduced for a fixed computational effort). We show that an alternative scaling regime in which temperature decreases (or, equivalently, free-energy barrier heights increase) as the number of windows increases best demonstrates the potential benefits of the umbrella sampling strategy; in that regime the efficiency improvement over direct simulation is exponential in the (inverse) temperature.

Our main theoretical result is a central limit theorem for the statistical averages obtained from EMUS. This result relies on the delta method, which we use to characterize the propagation of the asymptotic error through the solution of a stochastic matrix eigenproblem. The central limit theorem provides an expression for the asymptotic variance of the averages of interest. It is a sum of contributions from individual windows, and we use it to develop a prescription for estimating the relative importances of windows for averages from the trajectory data. For free energy differences of states of the alanine dipeptide, we find numerically that the importances are largest for low-free energy pathways that connect the specific states of interest. These results suggest that the importances could serve as the basis for adaptive schemes that focus computational effort on the windows of most importance. Even more interesting would be to adjust the bias functions as the simulation progresses. How best to do this remains an open area of investigation.

APPENDIX: CONSISTENCY OF ITERATIVE EMUS

Here, we prove that for fixed finite m, z^m is a consistent estimator of the vector of normalization constants z. With the initial guess z⁰ = n, the result, z¹, of the first iteration is the EMUS estimator. We now show that z² is also consistent in the sense that if the trajectory averages defining $\overline{F}\left(w\right)$ converge then z² converges to z. Because the various sequences in question are sequences of random variables, one must specify what is meant by convergence. The argument below applies when convergence refers either to convergence in probability or convergence with probability one (almost sure convergence) as long as the notion of convergence is consistent throughout. Consistency of z^m follows by induction on m using a similar argument.

For any positive vector w, we define

$${\displaystyle u_k=\prod _{j\ne k}\frac{w_j}{n_j},}$$ (A1)

and we write

$${\displaystyle {\overline{F}}_{ij}\left(w\right)=\frac{1}{N_i}\sum _{t=0}^{N_i-1}{h}_{ij}(u,x),}$$ (A2)

where

$${\displaystyle h_{ij}(u,x)=\frac{u_i{\psi }_j\left(x\right)}{\sum _k{u}_k{\psi }_k\left(x\right)}.}$$ (A3)

We then observe that

$${\displaystyle {\partial }_{u_k}{h}_{ij}(u,x)=\{\begin{array}{ll}\frac{1}{u_i}{h}_{ij}(u,x)[1-h_{ii}(u,x)],\hfill & \text{if}k=i\hfill \\ -\frac{1}{u_i}{h}_{ij}(u,x)h_{ik}(u,x),\hfill & \text{if}k\ne i\hfill \end{array}.}$$ (A4)

Because h_ij(u, x) ≤ u_i/u_j,

$${\displaystyle \left|{\partial }_{u_k}{h}_{ij}(u,x)\right|\le \frac{1}{u_j}max\left\{1,\frac{u_i}{u_k}\right\}.}$$ (A5)

Therefore,

$${\displaystyle |h_{ij}(\tilde{u},x)-h_{ij}(u,x)|\le \gamma (u,\tilde{u}),}$$ (A6)

for a continuous function γ defined for positive vectors u and $\tilde{u}$ and such that γ(u, u) = 0 for any u. The function $\gamma (u,\tilde{u})$ must explode when the entries of u or $\tilde{u}$ approach 0. Now define

$${\displaystyle u_k=\prod _{j\ne k}\frac{z_j^1}{n_j}\text{and}{\tilde{u}}_k=\prod _{j\ne k}\frac{z_j}{n_j},}$$ (A7)

where z is the exact vector of normalization constants. By (A6), we have

$${\displaystyle |{\overline{F}}_{ij}\left(z^1\right)-{\overline{F}}_{ij}\left(z\right)|\le \gamma (u,\tilde{u}).}$$ (A8)

As the number of samples N increases, $\overline{F}\left(z\right)$ converges to F(z). Moreover, since z¹ is the EMUS estimate of z, z¹ converges to z. Therefore, u converges to $\tilde{u}$, and (A8) implies that ${\overline{F}}_{ij}\left(z^1\right)$ converges to F_ij(z). Finally, since the function mapping an irreducible, stochastic matrix to its invariant vector is continuous, it follows that z² converges to the invariant vector of F(z), which is z. This verifies the consistency of z².