Derivatives of the Stochastic Growth Rate

Abstract

We consider stochastic matrix models for population driven by random environments which form a Markov chain. The top Lyapunov exponent a, which describes the long-term growth rate, depends smoothly on the demographic parameters (represented as matrix entries) and on the parameters that define the stochastic matrix of the driving Markov chain. The derivatives of a — the “stochastic elasticities” — with respect to changes in the demographic parameters were derived by Tuljapurkar (1990). These results are here extended to a formula for the derivatives with respect to changes in the Markov chain driving the environments. We supplement these formulas with rigorous bounds on computational estimation errors, and with rigorous derivations of both the new and the old formulas.

1 Introduction

Stochastic matrix models for structured populations are widely used in evolutionary biology, demographic forecasting, ecology, and population viability analysis (e.g., Tuljapurkar (1990); Lee and Tuljapurkar (1994); Morris and Doak (2002); Caswell (2001); Lande et al. (2003)). In these models, a discrete-time stochastic process drives changes in environmental conditions that determine the population's stage-transition rates (survival, fertility, growth, regression and so on). Population dynamics are described by a product of randomly chosen population projection matrices. In most biological situations the population's stage structure converges to a time-varying but stable structure Cohen (1977), and in the long run the population grows at a stochastic growth rate a that is not random and is the leading Lyapunov exponent of the random product of population projection matrices Furstenberg and Kesten (1960a); Cohen (1977); Lange (1979); Lange and Holmes (1981); Tuljapurkar and Orzack (1980). This growth rate a is of considerable biological interest, as a fitness measure for a stage-structured phenotype Tuljapurkar (1982), as a determinant of population viability and persistence Tuljapurkar and Orzack (1980); Morris and Doak (2002); Lande et al. (2003), and in a variety of invasion problems in evolution and epidemiology Metz et al. (1992).

The map between environments and projection matrices describes how pheno-type change depends on the environment — the phenotypic norm of response — and we are often interested in how populations respond to changes in, say, the mean or variance of the projection matrix elements. Such questions are answered by computing the derivatives of a with respect to changes in the projection matrices, using a formula derived by Tuljapurkar (1990). Tuljapurkar et al. (2003) called these derivatives stochastic elasticities, to contrast with the elasticity of the dominant eigenvalue of a fixed projection matrix to the elements of that matrix (Caswell, 2001). Stochastic elasticity has been used to examine evolutionary questions (Haridas and Tuljapurkar, 2005) and the effects of climate change (Morris et al., 2008). At the same time, a is also a function of the stochastic process that drives environments. Many processes, such as climate change (Boyce et al., 2006), will result in changes in the frequencies of, or the probabilities of transition between, environmental states. How is a affected by a change in the pattern and distribution of environments, rather than by a change in the population projection matrices? To answer this question, we consider a model in which the environment makes transitions among several discrete states, according to a Markov chain. Then what we want is the derivative of a with respect to changes in the transition probabilities of this Markov chain. This derivative exists (at least away from the boundaries of the space of stochastic matrices), and in fact we know from Peres (1992) that a is an analytic function of the parameters of both the projection matrices and the parameters defining the stochastic matrix, in an open neighborhood of the set of stochastic matrices. In deterministic models (Caswell, 2001), the growth rate is represented as λ = e^r; then sensitivities are derivatives of the form (∂λ/∂x) with respect to a parameter x whereas elasticities are proportional derivatives of the form (∂r/∂ log x). In stochastic models we compute derivatives of a, and these can be used to compute elasticities (as in Tuljapurkar et al. (2003)) or sensitivities.

Our first contribution here is a new formula for computing the derivative of a with respect to changes in the transition probabilities of the environmental Markov chain, given in abstract form as equation (40). To obtain this result we show how an initial environmental state affects future growth, using coupling and importance sampling; this analysis may be of independent interest. Even with a formula in hand we must compute derivatives of a by numerical simulation which is subject to both sampling (Monte Carlo) error and bias. Our second contribution here is to show how one can bound these estimation errors. Our third contribution is a rigorous proof of the heuristically derived formula given by Tuljapurkar (1990) for the derivatives of a to the elements of the population projection matrices.

In Section 2 of this paper we set out the model and assumptions, the approach to finding derivatives, along with necessary facts about the convergence of population structures and distributions. In Section 3 we discuss systematic and sampling errors and show how we can bound them. We illustrate this approach in Section 4 by presenting bounds (in Theorem 1) for simulation estimates of the stochastic growth rate a and (in Theorem 2) for the derivatives of a with respect to projection matrix elements. In Section 5 we define a measure of the effect of an initial environmental state on subsequent population growth and show how to estimate this measure using coupling arguments. Section 6 presents (in Theorem 4) the formula, algorithm, and error bounds for the derivative of a with respect to the elements of the Markov chain that drives environments. We end by discussing how these theorems can be applied and some related issues concerning parameter estimation in such models. Proofs are in the Appendix.

2 The Model

We consider a population whose individuals exist in K different stages (these may be, for example, ages, developmental stages or size classes). Newborns are in stage i = 1. The progression between stages occurs at discrete time intervals at rates that depend on the environment in each time interval. The environment e_t in period t is in one of M possible states; we denote the set of possible environments by $\mathcal{M}$ = {1, …, M}. Individuals in stage i at time t move to stage j at a rate X_et+1(j, i). These rates are elements of a nonnegative population projection matrix, and at time t when the environment is e_t this matrix is denoted by X_et; there are M such matrices, one for each environmental state. We assume that allocation of individuals to classes and the identification of environment states are certain. We also assume that the total number of individuals in the population is large enough that we can ignore sampling variation. Successive environments are chosen according to a Markov process with transition matrix P whose elements are $P(e,\breve{e})$ and whose stationary distribution is ν = {ν(e)}. We follow the standard convention for Markov chains, that $P(e,\breve{e})$ represents the probability of a transition from state e to state $\breve{e}$; note that this is the reverse of the convention used in matrix population models.

To guarantee demographic weak ergodicity (Cohen 1977) we assume that

• (i)There exists some R > 0 such that any product X_e1 … X_eR has all entries positive. This implies, in particular, that each population projection matrix is row-allowable and column-allowable. That is, every row and every column has at least one positive entry.
• (ii)The chain is ergodic (so transitive and aperiodic), and environments are in the stationary distribution of P, which will also be denoted by ν(e).

The population in year t is represented by a vector $N_t\in {\mathbb{R}}_{+}^K≔\{{(n_t\left(1\right),\dots ,n_t\left(K\right))}^T:n_t\left(i\right)>0\}$. The superscript T will always mean transpose; here it indicates that population vectors are column vectors. (There may be population classes early on that have 0 members; condition (ii) above forces all classes eventually to have positive membership, and so we assume without loss of generality that we start with all population classes occupied.) The population structure changes according to N_t+1 = X_et+1N_t, and

$$N_t=X_{e_t}{X}_{e_{t-1}}\cdots X_{e_1}{N}_0.$$ (1)

The normalized population structure N_t/(Σ_i N_t(i)) does not converge to a fixed limit (as it would if the environment were constant) but it does converge in distribution.

At each stage i, we define

$$a≔\underset{t\to \infty }{\text{lim}}{t}^{-1}\text{log}{N}_t\left(i\right)\underset{t\to \infty }{\text{lim}}{t}^{-1}\text{log}\frac{{\Sigma }_i{N}_t\left(i\right)}{{\Sigma }_i{N}_0\left(i\right)}$$ (2)

A fundamental result in this area of research is the Furstenberg-Kesten Theorem Furstenberg and Kesten (1960b), which tells us that the long-run growth rate exists and is not random. This a is called the “stochastic growth rate”. The proof of this fact is fairly straightforward, within the framework of modern Markov chain theory. We present a version in section 3.1 and a proof in the appendix, both for the reader's convenience and because it illustrates some of the important basic ideas that we draw on throughout this work.

The main result of this paper is formula (40), which represents the derivative of a with respect to an incremental change in the probability of transitioning from e to $\tilde{e}$. If we denote this derivative by $A_{e,\tilde{e}}$, then the change corresponding to shifting P in the direction of a matrix W is a linear functional

$$\frac{\mathit{da}(P+∊W)}{d∊}=\sum _{e,\tilde{e}\in \mathcal{M}}{A}_{e,\tilde{e}}{W}_{e,\tilde{e}}.$$

We note that, while the existence of such a derivative follows from the general theory of Peres (1992), the computable formula is new.

Counting the K² parameters in each matrix, there are at most (MK² + M²) parameters. While all parameters must be nonnegative, and all elements of the population projection matrices but the birth rates must be ≤ 1, the only universal constraint is ${\Sigma }_{e\in \mathcal{M}}P(e,\breve{e})=1$. (There may, however, be further constraints imposed, as some transitions may be impossible. If we are considering age-structured populations, the matrices are Leslie matrices, each with only 2K − 1 potentially nonzero parameters.) The sensitivities we examine are derivatives of a with respect to these parameters. We will define a one-dimensional family of population matrices or environment transition matrices, and we refer to change “in the direction of” the derivative of this family. This will be made precise in the analyses that follow.

3 Technical background

3.1 Contraction

We denote the K-dimensional column vector with 1's in all places by 1. By default we use the L¹ norm ∥x∥ := Σ |x_i| when x is a vector in ${\mathbb{R}}^K$, and write $\parallel X\parallel ≔{\sum }_{i,j=1}^K\mid X_{i,j}\mid$ when X is a K × K matrix. Note that ∥X∥ = ∥X1∥ when X is a nonnegative matrix. Our assumptions imply that there are positive constants $\widehat{k}$ and $\widehat{r}$, such that for any environments e₁, …, e_m,

$$\mid \text{log}\parallel X_{e_m}\cdots X_{e_1}\parallel \mid \le \widehat{k}+m\widehat{r}$$ (3)

We use the Hilbert projective metric ρ, described in Golubitsky et al. (1975) and in section 3.1 of Seneta (1981), defined by

$$\rho (x,y)≔\text{log}\underset{1\le i\le K}{\text{max}}x\left(i\right)∕y\left(i\right)+\text{log}\underset{1\le i\le K}{\text{max}}y\left(i\right)∕x\left(i\right).$$ (4)

This is a pseudometric on ${\mathbb{R}}_{+}^K$ that is a metric on $\mathcal{S}≔\{{(x\left(1\right),\dots ,x\left(K\right))}^T:x\left(i\right)>0\}$ and Σx(i) = 1}. The distance between two vectors is defined by the ray from the origin; that is, ρ(x, y) = ρ(x/∥x∥, y/∥y∥). It is straightforward to show — cf. Bushell (1973) — that

$$\frac{1}{2}\left[\text{min}\left\{x\left(i\right)\right\}+\text{min}\left\{y\left(i\right)\right\}\right]e^{-\rho (x,y)}\rho (x,y)\le \parallel x-y\parallel \le e^{\rho (x,y)}-1$$

for any x, y ∈ $\mathcal{S}$. (The upper bound is shown by Bushell with respect to the Euclidean norm, but the same argument holds for any L^p norm.) Thus, convergence in the projective metric implies that the projections onto $\mathcal{S}$ converge in the standard norms. It is also straightforward to show that

$$\underset{1\le i\le K}{\text{max}}\mid \text{log}\frac{x\left(i\right)}{y\left(i\right)}\mid \le \rho (x,y)+\mid \text{log}\frac{\parallel x\parallel }{\parallel y\parallel }\mid .$$ (5)

The Birkhoff contraction coefficient (cf. Seneta (1981)) is defined for nonnegative matrices X as

$${\tau }_B\left(X\right)≔\underset{u\ne v\in \mathcal{S}}{\text{sup}}\frac{\rho (\mathit{Xu},\mathit{Xv})}{\rho (x,v)}.$$ (6)

It is clear that τ_B is sub-multiplicative (that is, τ_B(XY) ≤ τ_B(X)τ_B(Y)). We also have the formula (Theorem 3.12 of Seneta (1981))

$${\tau }_B\left(X\right)=\frac{1-\varphi {\left(X\right)}^{1∕2}}{1+\varphi {\left(X\right)}^{1∕2}},\text{where}\varphi \left(X\right)=\{\begin{array}{cc}{\text{min}}_{i,j,k,\ell }\frac{X_{i,k}{X}_{j\ell }}{X_{\mathit{jk}}{X}_{i\ell }}\hfill & \hfill \text{if}X>0,\hfill \\ 0\hfill & \hfill \text{ohterwise}.\hfill \end{array}\phantom{\}}$$ (7)

(This depends on the assumption that X is row-allowable. We will be applying this result only to cases in which X is positive, so our bound on τ_B will be strictly less than 1.)

An immediate consequence is Lemma 3.1.

Lemma 3.1. For any fixed R, set $r≔{\text{max}}_{e_1,\dots ,e_R\in \mathcal{M}}{\tau }_B{(X_{e_R}\cdots X_{e_1})}^{1∕R}$. Then for any $e_1,\dots ,e_n\in \mathcal{M}$,

$${\tau }_B(X_{e_n}\cdots X_{e_1})\le k_{1^{r^n}},$$ (8)

where k₁ := r^1/R−1. In particular, if R is chosen so that every product X_eR … X_e1 is positive, then r < 1.

We also know that for any u, $\breve{u}\in \mathcal{S}$, and $v^T$ any positive row vector,

$$\text{log min}\left\{\frac{u\left(i\right)}{\breve{u}\left(i\right)}\right\}\le \text{log}\frac{v^{T}u}{v^T\breve{u}}\le \text{log max}\left\{\frac{u\left(i\right)}{\breve{u}\left(i\right)}\right\}.$$

Since the left-hand side is ≤ 0 and the right-hand side ≥ 0, so

$$\mid \text{log}{v}^{T}u-\text{log}{v}^T\breve{u}\mid \le \rho (u,\breve{u}).$$ (9)

Following Lemma 1 of Lange (1979), we define a compact subset $\mathcal{U}\subset \mathcal{S}$ which is stable under the transformations u → X_eu/∥X_eu∥ for any $e\in \mathcal{M}$ and includes the vector 1/K, as well as all vectors of the form X_eR … X_e1 y/∥X_eR … X_e1 y∥ where $y\in \mathcal{S}$; and a compact subset $\mathcal{V}\subset {\mathcal{S}}^T$ which is stable under the transformations v^T → v^T X_{e/∥v^T Xe}∥, and includes the vector 1^T/K as well as all vectors of the form y^T X_eR … X_e1/∥y^T X_eR … X_e1∥ where $y\in \mathcal{S}$. Note that we could take $\mathcal{U}$ to be any ${\mathcal{U}}_n$ defined to be the union of all products X_e1 … X_en $\mathcal{S}$. These are a decreasing sequence of compact sets, with $X_e{\mathcal{U}}_n\subset {\mathcal{U}}_{n+1}$. whose limit ${\mathcal{U}}_{\infty }$ is in general a fractal set, which is the support of the stationary distribution π. For practicality we are likely to work with a $\mathcal{U}$ which is a finite-stage approximation to ${\mathcal{U}}_{\infty }$.

Since $\mathcal{U}$ is a compact subset of $\mathcal{S}$, its diameter $\Delta ≔{\text{sup}}_{u,\breve{u}\in \mathcal{U}}\rho (u,\breve{u})$ is finite. Combining this with Lemma 3.1, there is a constant k₂ = k₁Δ such that

$$\rho (X_{e_m}\cdots X_{e_1}u,X_{e_m}\cdots X_{e_1}\breve{u}){\le }_{1^{r^m}}\rho (u,\breve{u})\le k_{2^{r^m}}.$$ (10)

(Cf. Theorem 2 of Lange and Holmes (1981).) Of course, the constants may be chosen so that the same relation holds for the transposed matrices, with $u^T,{\breve{u}}^T\in \mathcal{V}$. We use then the definition

$$\Delta ≔\text{max}\left\{\underset{u,\breve{u}\in \mathcal{U}}{\text{sup}}\rho (u,\breve{u}),\underset{v,\breve{v}\in \mathcal{U}}{\text{sup}}\rho (v,\breve{v})\right\}.$$ (11)

We also define

$${\Delta }_{\mathcal{U}}\left(u\right)≔\underset{\breve{u}\in \mathcal{U}}{\text{sup}}(u,\breve{u});{\Delta }_{\mathcal{V}}\left(u\right)≔\underset{\breve{v}\in \mathcal{V}}{\text{sup}}(v,\breve{v}).$$ (12)

Since for any positive vectors u, u′, u″ we have ρ(u+u′, u″) ≤ max {ρ(u, u″), ρ(u′, u″)}, the maximum of ρ(u, u′) among vectors in a cone is taken between extreme points of the cone, we can compute Δ as the maximum of all distances of the form $\rho (X_{e_R}\cdots X_{e_1}1,X_{e_R^{\prime }}\cdots X_{e_1^{\prime }}1)$; and distances of the form $\rho (1X_{e_R}\cdots X_{e_1},1X_{e_R^{\prime }}\cdots X_{e_1^{\prime }})$.

It follows (as in Lemma 2 of Lange and Holmes (1981)) that for any u, $\breve{u}\in \mathcal{U}$ and environments e, e₁, …, e_m,

$$\begin{array}{cc}\hfill \mid \text{log}\frac{\parallel X_e{X}_{e_m}\cdots X_{e_1}u\parallel }{\parallel X_{e_m}\cdots X_{e_1}u\parallel }-\text{log}\frac{\parallel X_e{X}_{e_m}\cdots X_{e_1}\breve{u}\parallel }{\parallel X_{e_m}\cdots X_{e_1}\breve{u}\parallel }\mid \hfill & \le k_{1^{r^m}}\rho (u,\breve{u})\hfill \\ \hfill \hfill & \le k_{2^{r^m}}.\hfill \end{array}$$ (13)

The same relation holds when the matrices X are replaced by their transposes, with u^T, ${\breve{u}}^T\in \mathcal{V}$.

Since ∥X∥ = ∥X1∥, and 1/K is in $\mathcal{U}$, it immediately follows that if ${\breve{e}}_1,\dots ,{\breve{e}}_i$ are any other environments,

$$\mid \text{log}\frac{\parallel X_e{X}_{e_m}\cdots X_{e_1}\parallel }{\parallel X_{e_m}\cdots X_{e_1}\parallel }-\text{log}\frac{\parallel X_e{X}_{e_m}\cdots X_{e_{i+1}}{X}_{{\breve{e}}_i}\cdots X_{{\breve{e}}_1}\parallel }{\parallel X_{e_m}\cdots X_{e_{i+1}}{X}_{{\breve{e}}_i}\cdots X_{{\breve{e}}_1}\parallel }\mid \le k_{2^{r^{m-i}}}.$$ (14)

We note that the results in Lange and Holmes (1981) depend only on the set of matrices X being compact, not on it being finite. In section 5 and beyond we will be letting the matrices X_e and/or the transition matrix P depend smoothly on a parameter x, which will take values either in [−x₀, x₀] or [0, x₀]. We may then choose the sets $\mathcal{U}$ and $\mathcal{V}$ and constants k₁, k₂, r such that the properties above — in particular, the stability of $\mathcal{U}$ and $\mathcal{V}$ and the bounds (10) and (13) — hold simultaneously for all values of the parameter.

One slightly more complicated bound is proved in the appendix.

Lemma 3.2 Given v, $\breve{v}$, u, $\breve{u}\in {\mathbb{R}}_{+}^R$, and constants k_u, $k_v\le \frac{1}{4}$ with $\text{max}\mid \text{log}{v}_i∕{\breve{v}}_i\mid \le k_v$ and $\text{max}\mid \text{log}{u}_j∕{\breve{u}}_j\mid \le k_u$ we have

$$\mid \text{log}\frac{v^{T}u}{{\breve{v}}^{T}u}-\text{log}\frac{v^{T}u}{{\breve{v}}^T\breve{u}}\mid \le 8k_u{k}_v.$$ (15)

This condition is satisfied if

$$k_v≔\rho (v,\breve{v})+\mid \text{log}\parallel v\parallel ∕\parallel \breve{v}\parallel \mid \le \frac{1}{4}$$

and

$$k_u≔\rho (u,\breve{u})+\mid \text{log}\parallel v\parallel ∕\parallel \breve{v}\parallel \mid \le \frac{1}{4}.$$

3.2 Time reversal and the stationary distribution

The transition matrix for the time-reversal of P will be denoted $\tilde{P}$, and is given by

$$\tilde{P}(e,\breve{e})≔\frac{\nu \left(\breve{e}\right)}{\nu \breve{e}}P(\breve{e},e).$$

A standard result (for example, see Theorem 6.5.1 of Grimmett and Stirzaker (2001)) tells us that if e₁, e₂, …, e_m form a stationary Markov chain with transition matrix P, for a fixed m, the reversed sequence e_m, e_m−1, …, e₁ is a Markov chain with transition matrix $\tilde{P}$. (We use the boldface e to represent a random environment — a random variable with values in $\mathcal{M}$ — and e to represent a particular environment or a realisation of a random variable e.)

As described in Lange and Holmes (1981), if e₁, e₂, … forms a stationary Markov chain with transition probabilities P, there is a unique distribution π on $\mathcal{U}\times \mathcal{M}$ which is stable under the transformation (u, e_t) ↦ (X_etu/∥X_etu∥, e_t+1). That is, if the normalized population structure N_t/∥N_t∥ paired with e_t+1 is chosen from the distribution π, then the pair (N_t+1/∥N_t+1∥, e_t+2) will also be in the distribution π.

This Markov chain is naturally represented as an iterated function system; that is, a step in the chain is made by choosing randomly from a collection of possible functions, and applying it to the current position of the chain. In this case, the function is u ↦ X_eu/∥X_eu∥, where e is the next random environment. Time reversal is a convenient approach for studying the asymptotic behavior of such systems, and more general random dynamical systems. For applications, see Barnsley and Elton (1988); Elton (1990); Steinsaltz (1999).

To understand how this works, suppose we extend our Markov chain of environments forward and backward in time to infinity: …, e₋₂, e₋₁, e₀, e₁, e₂, …. Starting at any s, positive or negative, moving forward we have a Markov chain with transition matrix P: e_s, e_s+1, e_s+2, …. Moving backward, e_s, e_s−1, e_s−2, … is a Markov chain with transition matrix $\tilde{P}$. Choose any population vector $u_0\in \mathcal{U}$, and define for s > t,

$$U_{s,t}≔\frac{X_{{\mathbf{e}}_s}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_t}{u}_0}{\parallel X_{{\mathbf{e}}_s}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_t}{u}_0\parallel }.$$ (16)

From (10), we know that for any t₁, t₂ ≥ m, we have the bound ρ(U_s,−t1, U_s−t2) ≤ k₂r^m+s, which means that ${\left(U_{s,-t}\right)}_{t=0}^{\infty }$ is a Cauchy sequence. We may denote its limit by U_s,−∞.

We note four important features of this backward iterated sequence:

• (i)If the environments all shift one to the left, so that we start at e_s+1 instead of e_s, the distribution remains exactly the same. Thus, the distribution of U_s,−∞ is the same for any s.
• (ii)
  If we start our population Markov chain in the state (U_0,−∞, e₁), then the next state will be

  $$\left(\frac{X_{{\mathbf{e}}_1}{U}_{0,-\infty }}{\parallel X_{{\mathbf{e}}_1}{U}_{0,-\infty }\parallel },{\mathbf{e}}_2\right)=(U_{1,-\infty },{\mathbf{e}}_2).$$

  Thus, the distribution of (U_0,−∞, e₁) is a stationary distribution π for the chain.
• (iii)
  For each fixed t, U_0,−t has the same distribution as

  $$U_{t,0}=\frac{U_{{\mathbf{e}}_t}{X}_{{\mathbf{e}}_{t-1}}\cdots X_{{\mathbf{e}}_0}{u}_0}{\parallel X_{{\mathbf{e}}_t}{X}_{{\mathbf{e}}_{t-1}}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel },$$ (17)

  which is a realization of the normalized population vector N_t/∥N_t∥, when it is started at the vector u₀. Thus, the population-environment Markov chain (N_t/∥N_t∥) converges in distribution to the same stationary distribution π, no matter what the initial condition.
• (iv)
  We will use frequently the fact that (U_0,−t, e₁) may be considered an approximate sample from the distribution π, with an error that is bounded by

  $$\begin{array}{cc}\hfill \rho (U_{0,-t},U_{0,-\infty })& =\rho (X_{{\mathbf{e}}_0}\cdots X_{{\mathbf{e}}_{-t}}{u}_0,X_{{\mathbf{e}}_0}\cdots X_{{\mathbf{e}}_{-t}}\cdot U_{-t-1,-\infty })\hfill \\ \hfill & \le {\Delta }_{\mathcal{U}}\left(u_0\right){\tau }_B(X_{{\mathbf{e}}_0}\cdots X_{{\mathbf{e}}_{-t}}),\hfill \end{array}$$

  which is straightforward to bound (cf. section 3.1), independent of any information about U_−t,−∞.

The same holds true, of course, if we reverse the matrix multiplication: Starting from any nonnegative K-dimensional row vector $v_0^T$, we define the sequence of row vectors

$$V_{0,t}^T≔\frac{v_0^T{X}_{{\mathbf{e}}_t}\cdot X_{{\mathbf{e}}_0}}{\parallel v_0^T{X}_{{\mathbf{e}}_t}\cdot X_{{\mathbf{e}}_0}\parallel }\in \mathcal{V}.$$

Then V_0,t converges pointwise to a vector V_0,∞ := lim_t→∞ V_0,t. We denote the distribution of (V_0,∞, e₋₁) by $\tilde{\pi }$. As before, $\tilde{\pi }$ is the stationary distribution for the Markov chain on $\mathcal{V}\times \mathcal{M}$, defined by taking (v^T, e_−t) at time t to (v^T X_e−t/∥v^T X_e−t∥, e_−(t+1)) at time t + 1.

We also define the regular conditional distributions π_e on $\mathcal{U}$ as follows: pick (U, e) from the distribution π, conditioned on e being e, and take π_e to be the distribution of U. We define ${\tilde{\pi }}_e$ similarly. In the case of i.i.d. environments, of course, π and $\tilde{\pi }$ would be simply products of an independent population vector and the stationary environment with distribution ν.

4 Errors and How to Bound Them

A standard approach to estimating a — and a similar approach to the derivatives that we describe later on — is to choose a fixed starting vector $u_0\in \mathcal{U}$, simulate sequences e₀(i), e₁(i), …, emi (i independently from the stationary Markov chain with transition probabilities P (i = 1, …, J), and then compute

$$\begin{array}{cc}\hfill a_m≔& \mathbb{E}\left[\text{log}\frac{X_{{\mathbf{e}}_{m_i}\left(i\right)}{X}_{{\mathbf{e}}_{m_i-1}\left(i\right)}\cdots X_{{\mathbf{e}}_0\left(i\right)}{u}_0}{\parallel X_{{\mathbf{e}}_{m_i-1}\left(i\right)}\cdots X_{{\mathbf{e}}_0\left(i\right)}{u}_0\parallel }\right]\hfill \\ \hfill & \approx \frac{1}{J}\sum _{i=1}^J\left(\text{log}\parallel X_{e_{m_i}\left(i\right)}{X}_{e_{m-1}\left(i\right)}\cdots X_{e_0\left(i\right)}{u}_0\parallel -\text{log}\parallel X_{e_{m_i-1}\left(i\right)}\cdots X_{e_0\left(i\right)}{u}_0\parallel \right).\hfill \end{array}$$ (18)

It is important not only to know what would be an appropriate approximation to a or its derivatives in the sense of being asymptotically correct, but also to have rigorous bounds for the error arising from any finite simulation procedure, such as (18). There are two sources of error: systematic error, arising from the fact that (X_e0, u₀) is not exactly a sample from the distribution π, and sampling error, arising from the fact that we have estimated the expectation by averaging over a random sample.

For obtaining an unbiased estimate there is no reason the sample Markov chain realizations should need to be independent. For instance, a standard approach would be to take a single realization of the chain e₀; e₁; e₂,…, and take e_t(i) = e_t, with m_i = B + i for some “burn-in time” B. The problem with this approach is that it becomes more difficult to bound the errors; in principle, the convergence could be extremely slow for these kinds of sums. We leave the problem of bounding errors for these cumulative simulations to a future work.

4.1 Systematic error

By “systematic error” we mean the error in our estimate of a arising from the difference between the distribution we are aiming for and the distribution we are actually sampling from. The quantity we are trying to estimate may be represented as a = π[F], the expectation of a certain function F with respect to the distribution π. If we can simulate Z₁,…, Z_J from π, then ${\widehat{a}}_j≔{\sum }_{j=1}^{J}F\left(Z_j\right)$ is an unbiased estimator of π[F], and will be consistent under modest Passumptions on F and the independence of the samples. Suppose, though that what we have are not samples from π, but samples $Z_j^{\ast }$ from a “similar” distribution π*. Then we can bound the error by

$$\mid a-\widehat{a}\mid \le \mid J^{-1}\sum _{j=1}^{J}F\left(Z_j^{\ast }\right)-\pi \ast \left[F\right]\mid +\mid \pi \left[F\right]-\pi \ast \left[F\right]\mid .$$ (19)

Here the first term on the right-hand side is the sampling error, and the second term is the bias, the expected value of systematic error. The problem is that the bounds we can obtain for the bias are likely to be crude, absent good computational tools for the distribution π. (And if we could compute analytically from π, we wouldn't need to be simulating).

An alternative is to couple the samples $Z_j^{\ast }$ from the approximate distribution π′ to exact samples Z_j from the distribution π, we can break up the error in a slightly di erent way:

$$\begin{array}{cc}\hfill \mid a-\widehat{a}\mid & \le \mid J^{-1}\sum _{j=1}^{J}F\left(Z_j\right)-\pi \left[F\right]\mid +\mid J^{-1}\sum _{j=1}^{J}F\left(Z_j\right)-J^{-1}\sum _{j=1}^{J}F\left[Z_j^{\ast }\right]\mid \hfill \\ \hfill & \le \mid J^{-1}\sum _{j=1}^{J}F\left(Z_j\right)-\pi \left[F\right]\mid +J^{-1}\sum _{j=1}^J\mid F\left(Z_j\right)-F\left(Z_J^{\ast }\right)\mid \hfill \end{array}$$ (20)

Bounds for the sampling error in (19) will generally also be bounds for the first term in (20). The second term in (20), on the other hand, which takes the place of the bias, is a random variable, computed from the samples $Z_j^{\ast }$. Its expectation is still a bound on the bias. The crucial fact is that the last line may be computable without knowing in detail what the “true” sample Z_j is.

A small disadvantage of this approach is that the systematic error varies with the sample. To achieve a particular fixed error bound we need an adaptive approach, whereby we successively extend our sequence of matrices until the error crosses the desired threshold. We note here that this approach to estimating the systematic error in simulations is essentially just a version of the Propp-Wilson algorithm (cf. Chapter 10 of Häggström (2002)). Unlike standard applications, though, the space is continuous, so the systematic error never reaches 0.

4.2 Sampling error

The sampling error is difficult to control with current techniques, because the distribution of the samples is so poorly understood — the very reason why we resort to the Monte Carlo approximation in the rst place. The best we can do for a rigorous bound is to use Hoeffiding's inequality (see Hoeffding (1963)), relying on crude bounds on the terms in the expectation. Hoeffding's inequality tells us that if X₁,…,X_J are i.i.d. random variables such that α ≤ X_i β almost surely, then for any z > 0,

$$\mathbb{P}\left\{\mid \frac{1}{J}\sum X_i-\mathbb{E}\left[X\right]\mid >z\right\}\le 2e^{-2J{z}^2∕{(\beta -\alpha )}^2}.$$ (21)

This is essentially the same bound that we would estimate from the normal approximation if the standard deviation of X were (β – α)/2. Generally we will want to fix p₀, the con dence level, and compute the corresponding z, which will be

$$z_0=(\beta -\alpha )\sqrt{-\frac{1}{2J}\text{log}(p_0∕2)}.$$ (22)

Of course, the standard deviation will be smaller than this, but we do not know how much smaller. An alternative approach then would be to use the bound $2\tau (z\sqrt{J}∕\widehat{\sigma })$, where $\widehat{\sigma }$ is the standard deviation of the simulated samples, and τ is the cumulative distribution function of the Student t distribution with J – 1 degrees of freedom. This will be a smaller bound, in that sense “better”, but not precisely true for finite samples, to the extent that the sample distribution is not normal. The corresponding bound on the error, at probability level p₀, is

$$z_0=\frac{\widehat{\sigma }}{\sqrt{J}}{t}_{1-p_0∕2}(J-1),$$ (23)

where t_p(J – 1) is the p quantile of the Student T distribution with J – 1 degrees of freedom; that is, if T has this distribution then P{T > t_p(J – 1) = p.

These asymptotic bounds are perfectly conventional in Markov chain Monte Carlo analysis (see, for example, Asmussen and Glynn (2007)), and there is no reason particularly to eschew them in this context. They are likely to be quite accurate, and superior to the rigorous bounds, and might also be applied to the setting where the expectations are estimated not from independent samples, but from a single run of the environment chain. We have nonetheless emphasised the Hoeffding-based rigorous bounds in the statements of our theorems for three reasons: First, they are likely to be less familiar, and the reader may require more guidance in applying them to the individual cases; second, because some residue of skepticism must remain for the asymptotic bounds, while these results may be applied to calculations that are otherwise analytically precise; and third, as a spur to further thought on the best ways to bound errors rigorously in these sorts of problems.

5 Growth Rate and Sensitivity to Projection Matrices

We present here extensions of two known results. In these cases (and in later results) we start by defining an estimator that converges to the the quantity we desire, and follow that by bounds on the systematic and sampling errors, as well as an error bound for estimates from a simulation estimator. We state our results on error bounds in the form “The quantity Q may be approximated by the expectation of A, with systematic error bounded by B and sampling error bounded by C(J, p).” This means that if A₁,…, A_J are independent realizations of A, then the probability that the true value of Q is not in the interval J⁻¹ ΣA_i ± [B + C(J, p)] is no bigger than p. When describing an adaptive bound on the systematic error, B will depend upon the particular simulation result. Again, the sampling error may be bounded either by a universally valid Hoeffding bound, based on known upper bounds on the samples, or by the Student t distribution, using the standard deviation estimated from the sample, which provides a generally much superior bound, but which can only be treated as an approximation.

5.1 Computing a

The stochastic growth rate a is commonly estimated by numerical simulation but, as discussed with examples by Caswell (2001), there is no general way to bound the errors in the estimated values. The following result provides suitable bounds.

Theorem 5.1 Let u₀ be any fixed element of $\mathcal{U}$, and Y_m := X_em X_em–1 …X_e1, where e₀; e₁, … form a Markov chain with transition rates P. The stochastic growth rate may be approximated by the simulated expectation of

$$\text{log}\frac{\parallel X_{{\mathbf{e}}_{m+1}}{Y}_m{u}_0\parallel }{\parallel Y_m{u}_0\parallel },$$ (24)

with systematic error bounded by $k_{2^{r^m}}$ and sampling error at level p on J samples bounded by

$$\left(\text{log}\frac{{\text{sup}}_{u\in \mathcal{U}}{\text{max}}_{e\in \mathcal{M}}\parallel X_{e}u\parallel }{{\text{inf}}_{u\in \mathcal{U}}{\text{min}}_{e\in \mathcal{M}}\parallel X_{e}u\parallel }\right){\left(\frac{-\text{log}p}{2J}\right)}^{1∕2}.$$ (25)

When the simulated expectation is

$$\frac{1}{J}\sum _{i=1}^J\text{log}\frac{\parallel X_{e_{m+1}}\left(i\right)Y_m\left(i\right)u_0\parallel }{\parallel Y_m\left(i\right)u_0\parallel }$$

we may also bound the systematic error by

$$\frac{1}{J}\sum _{i=1}^J\underset{u,\breve{u}\in \mathcal{U}}{\text{sup}}\rho (Y_m\left(i\right)u,Y_m\left(i\right)\breve{u})\le \frac{{\Delta }_{\mathcal{U}}\left(u_0\right)}{J}\sum _{i=1}^J{\tau }_B\left(Y_m\left(i\right)\right),$$ (26)

where ${\Delta }_{\mathcal{U}}$(u₀) is de ned as in (12) and τ_B is defined as in (6).

5.2 Derivatives with respect to Projection Matrices

We need care in defining derivatives of a with respect to elements of the population projection matrices. As discussed in Tuljapurkar and Horvitz (2003) we must define how the matrix entries change, e.g., do we change fertility rates in a particular environment, or in all possible environments? Although the main formula here is known, Tuljapurkar's (1990) derivation did not justify a crucial exchange of limits (between taking the perturbation to zero and time to infinity). We provide a rigorous proof (see Appendix) and of course the error bounds here are new.

We will suppose that the matrices X_e depend smoothly on a parameter ε, so that we may define X′_e := ∂X_e/∂ε, and we define the base matrices to be at ε = 0. In some cases, the parametrization will be defined only for ε ≥ 0, and in those cases we will understand the partial derivatives to be one-sided derivatives, and the limits lim_ε→0 will be the one-sided likits lim_ε↓0.

Theorem 5.2 Let U_e and V_e be independent random variables with distributions π_e and ${\tilde{\pi }}_e$ (the conditional stationary distributions defined in section 3.2). Then

$$a\prime \left(0\right)=\sum _{e\in \mathcal{M}}{\nu }_e\mathbb{E}\left[\frac{V_e^T{X}_e^{\prime }{U}_e}{V_e^T{X}_e{U}_e}\right]$$ (27)

Each term may be approximated by averaging samples of the form

$$\sum _{e\in \mathcal{M}}{\nu }_e\frac{V^{\left(m\right)T}{X}_e^{\prime }{U}^{\left(m\right)}}{V^{\left(m\right)T}{X}_e{U}^{\left(m\right)}},$$ (28)

where u₀; $v_0^T$ are any fixed elements of $\mathcal{U}$, $\mathcal{V}$ respectively, $U^{\left(m\right)}=X_{{\tilde{\mathbf{e}}}_1}\cdots X_{{\tilde{\mathbf{e}}}_m}{u}_0$ and $V^{\left(m\right)T}=v_0^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}$, e = ẽ₀; ẽ₁,…,ẽ_m form a sample from the Markov chain P̃, and e = e₀, e₁,…, e_m form a sample from the Markov chain P. The systematic error may be bounded uniformly by

$$2(\text{exp}\left(4k_{2^{r^m}}\right)-1)\mid a\prime \left(0\right)\mid ,$$ (29)

while the sampling error at level p on J samples is bounded by

$${\left(\frac{-\text{log}(p∕2)}{2J}\right)}^{1∕2}\underset{\underset{\breve{u}\in \mathcal{U},{\breve{v}}^T\in \mathcal{V}}{u\in \mathcal{U},v^T\in \mathcal{V}}}{\text{sup}}\sum _{e\in \mathcal{M}}{\nu }_e\left(\frac{v^T{X}_e^{\prime }u}{v^T{X}_{e}u}-\frac{\breve{v}{X}_e^{\prime }\breve{u}}{{\breve{v}}^T{X}_e\breve{u}}\right).$$ (30)

Suppose the simulated expectation is

$$\frac{1}{J}\sum _{j=1}^J\sum _{e\in \mathcal{M}}{\nu }_e\frac{{\left(V^{\left(m\right)}\left(j\right)\right)}^T{X}_e^{\prime }{U}^{\left(m\right)}\left(j\right)}{{\left(V^{\left(m\right)}\left(j\right)\right)}^T{X}_e{U}^{\left(m\right)}\left(j\right)},$$

where

$$U^{\left(m\right)}\left(j\right)=X_{{\tilde{\mathbf{e}}}_1\left(j\right)}\cdots X_{{\tilde{\mathbf{e}}}_m\left(j\right)}{u}_0≔{\tilde{Y}}_m\left(j\right)u_0$$

and

$${\left(V^{\left(m\right)}\left(j\right)\right)}^T=v_0^T{X}_{{\mathbf{e}}_m\left(j\right)}\cdots X_{{\mathbf{e}}_1\left(j\right)}≕v_0^T{Y}_m\left(j\right).$$

Let

$$\mathcal{U}\left(j\right)≔{\tilde{Y}}_m\left(j\right)\mathcal{U}=\{{\tilde{Y}}_m\left(j\right)u:u\in \mathcal{U}\},$$

$$\mathcal{V}\left(j\right)≔\mathcal{V}{\tilde{Y}}_m\left(j\right)=\{v^T{Y}_m\left(j\right):v^T\in \mathcal{V}\}.$$

Then we may also bound the systematic error by

$$\begin{array}{cc}\hfill & \sum _{e\in \mathcal{M}}\frac{{\nu }_e}{J}\sum _{j=1}^J\underset{\underset{v^T\in \mathcal{V}\left(j\right)}{u\in \mathcal{U}\left(j\right)}}{\text{sup}}\mid \frac{v^T{X}_e^{\prime }u}{v^T{X}_{e}u}-\frac{{\left(V^{\left(m\right)}\left(j\right)\right)}^T{X}_e^{\prime }{U}^{\left(m\right)}\left(j\right)}{{\left(V^{\left(m\right)}\left(j\right)\right)}^T{X}_e{U}^{\left(m\right)}\left(j\right)}\mid \hfill \\ \hfill & \le \sum _{e\in \mathcal{M}}\frac{{\nu }_e}{J}\sum _{j=1}^J\left(\text{exp}\left\{2\underset{u\in \mathcal{U}\left(j\right)}{\text{sup}}\rho (u,U^{\left(m\right)}\left(j\right))+2\underset{v^T\in \mathcal{V}\left(j\right)}{\text{sup}}\rho (u,V^{\left(m\right)}\left(j\right))\right\}-1\right)\times \underset{\underset{v^T\mathcal{V}\left(j\right)}{u\in \mathcal{U}\left(j\right)}}{\text{sup}}\frac{v^T\mid X_e^{\prime }\mid u}{v^T{X}_{e}u}\hfill \\ \hfill & \le \sum _{e\in \mathcal{M}}\frac{{\nu }_e}{J}\sum _{j=1}^J\underset{\underset{v^T\in \mathcal{V}\left(j\right)}{u\in \mathcal{U}\left(j\right)}}{\text{su[}}\frac{v^T\mid X_e^{\prime }u\mid }{v^T{X}_{e}u}\left(\text{exp}\left\{2\Delta \left({\tau }_B\left({\tilde{Y}}_m\left(j\right)\right)+{\tau }_B{\left(Y_m\left(j\right)\right)}^T\right)\right\}-1\right).\hfill \end{array}$$ (31)

Note that the bound (29) is given as a proportion of the unknown a…(0). It can be turned into an explicit bound by using an upper bound on a…(0). We have the trivial bound

$$\mid a\prime \left(0\right)\mid \le \sum _{e\in \mathcal{M}}{\nu }_e\underset{\underset{v^T\in \mathcal{V}}{u\in \mathcal{U}}}{\text{sup}}\frac{\mid v^T{X}_e^{\prime }u\mid }{v^T{X}_{e}u}.$$ (32)

We recall that Δ and τ_B may be bounded according to the formulas given in section 3.1.

It may seem surprising that the vectors U_e and V_e are independent. If we think of the environments as a doubly infinite sequence with e₀ = e, then the vector U_e in equation (27) depends only on the past (that is, e_i with i < 0) and V_e depends only on the future (e_i with i > 0). By the Markov property, these two are independent, when conditioned on e₀ = e.

6 Environments and Coupling

Suppose we change the transition matrix P to a slightly different matrix P^(ε), and want to compare population growth along environmental sequences generated by the original and the perturbed matrix. We expect that the perturbed environmental sequences will only occasionally deviate from the environment that we “would have had” in the original distribution of environments. Computing the derivative of a is then a matter of measuring the cumulative deviations due to these changes. These may be split into two parts: First, the process moves under P^(ε) to a state $\breve{e}$ , different from the state e that it would have moved to under P. Then there is a sequence of environments following on from $\breve{e}$ that is different from the sequence that would have followed from e, until the Markov chain gradually “forgets” its starting point. The change to a new sequence of matrices induces two separate changes on the growing population: The new sequence accumulates a difference in magnitude on its way to stationarity; and it produces a different stationary distribution of unit vectors depending on the starting environment $\breve{e}$ rather than e.

In this section we examine the first effect. We fix the transition matrix P and compare the growth of total population size when starting from environment e to the growth starting from the stationary distribution ν. A standard method for doing this is coupling. For an outline of coupling techniques in MCMC, see Kendall (2005) and Roberts and Rosenthal (2004). We use coupling in two ways, corresponding to the two components of the Markov chain: the environment and the population vector.

Fix environments e and $\breve{e}$ (possibly the same). We define sequences e₀, e₁, …; ${\breve{\mathbf{e}}}_0$, ${\breve{\mathbf{e}}}_1$, …; and ${\stackrel{‒}{\mathbf{e}}}_0$, ${\stackrel{‒}{\mathbf{e}}}_1$, …: all three are Markov chains with transition probabilities P, but with e₀ = e, ${\breve{\mathbf{e}}}_0=\breve{e}$ and ${\stackrel{‒}{\mathbf{e}}}_0$ having distribution ν (so that (${\stackrel{‒}{\mathbf{e}}}_i$) is stationary). We define the total population effect of starting in state e rather than $\breve{e}$ as

$$\begin{array}{cc}\hfill {}_{\breve{e}}\zeta _e& ≔\underset{t\to \infty }{\text{lim}}\left(\mathbb{E}\left[\text{log}\parallel X_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_0}\parallel \right]-\mathbb{E}\left[\text{log}\parallel X_{{\breve{\mathbf{e}}}_t}\cdots X_{{\breve{\mathbf{e}}}_0}\parallel \right]\right)\hfill \\ \hfill {\zeta }_e& ≔\underset{t\to \infty }{\text{lim}}\left(\mathbb{E}\left[\text{log}\parallel X_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_0}\parallel \right]-\mathbb{E}\left[\text{log}\parallel X_{{\stackrel{‒}{\mathbf{e}}}_t}\cdots X_{{\stackrel{‒}{\mathbf{e}}}_0}\parallel \right]\right)\hfill \\ \hfill & =\sum _{\breve{e}\in \mathcal{M}}{\nu }_{\breve{e}}\cdot {}_{\breve{e}}\zeta _e.\hfill \end{array}$$ (33)

This is one of the terms that will come into formula (40), the derivative of a with respect to shifting transition probabilities from e to $\breve{e}$.

Note that when the environments are i.i.d. — so $P(e,\breve{e})={\nu }_{\breve{e}}$ — we have

$${}_{\breve{e}}\zeta _e=\mathbb{E}\left[\text{log}\frac{\parallel V^T{X}_e\parallel }{\parallel V^T{X}_{\breve{e}}\parallel }\right],$$

where V^T has the distribution π.

Computing ζ_e depends on coupling the version of the Markov chain starting at e, to another version starting in the distribution ν. We define the coupling time τ to be the first time such that ${\mathbf{e}}_{\tau }={\stackrel{‒}{\mathbf{e}}}_{\tau }$; after this time the chains follow identical trajectories. If we know the distribution of τ and of the sequences followed by the two chains from time 0 to τ, we can average the diferences in (33) to nd ζ. The advantage of coupling is, first, that it reduces the variability of the estimates, and second, that we know from the simulation when the coupling time has been achieved, which gives bounds on the error. A suitable choice is Griffeath's maximal coupling (Griffeath, 1975) which we will apply in Pitman's (Pitman, 1976) path-decomposition representation. (The coupling is “maximal” in the sense of making the coupling time, and hence the variance of the estimate, as small as possible.) However we must be careful about sampling values of τ, because they may be large if the Markov chain mixes slowly. To deal with this we overweight large coupling times that generate a large contribution to ζ.

Beginning with a fixed environment e, the procedure is as follows:

• (i)Define the sequence of vectors α_t := P^t(e, ·) − ν(·). We also define ${\alpha }_t^{+}$ and ${\alpha }_t^{-}$ to be the vectors of pointwise positive and negative parts respectively. Let C(t) be a bound on $\mid \text{log}\parallel X_{e_1}\cdots X_{e_t}\parallel -\text{log}\parallel X_{{\breve{e}}_1}\cdots X_{{\breve{e}}_t}\parallel \mid$, where the e_i and ${\breve{e}}_i$ are any environments. From (3) we know that $2\widehat{k}+2t\widehat{r}$ is a possible choice for C(t).
• (ii)
  For pairs (t, e), where t is a positive integer and $e\ast \in \mathcal{M}$, define a probability distribution

  $$q(t,\breve{e})≔\{\begin{array}{cc}{\nu }_e\hfill & \text{if}e=\breve{e},t=0;\hfill \\ 0\hfill & \text{if}e\ne \breve{e},t=0;\hfill \\ \left[{\alpha }_{t-1}^{+}P\right]\left(\breve{e}\right)-{\alpha }_t^{+}\left(\breve{e}\right)\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$

  This is the distribution of the pair (τ, e_τ) for the maximally coupled chain. Define

  $$A≔\sum _{t=1}^{\infty }\sum _{\breve{e}\in \mathcal{M}}q(t,\breve{e})C\left(t\right),$$

  and a probability distribution on $\mathbb{N}\times \mathcal{M}$

  $$\mathring{q}(t,\breve{e})≔\frac{q(t,\breve{e})C\left(t\right)}{A}.$$
• (iii)
  Average J independent realizations of the following random variable: Let (τ, e₀) be chosen from the distribution $\mathring{q}$ on $\mathbb{N}\times \mathcal{M}$, and ${\breve{\mathbf{e}}}_0$ independently from the distribution ν. From these starting states e₀ and ${\breve{\mathbf{e}}}_0$, let (e₀, …, e_τ, …, e_m) and (${\breve{\mathbf{e}}}_0$, …, ${\breve{\mathbf{e}}}_{\tau }$, …, ${\breve{\mathbf{e}}}_m$) be realizations of the coupled pair of Markov chains with transition probabilities P, conditioned on the coupling time being τ and ${\mathbf{e}}_{\tau }={\breve{\mathbf{e}}}_{\tau }=\breve{\mathbf{e}}$. These realizations are generated from independent inhomogeneous Markov chains running backward, with transition probabilities

  $$\mathbb{P}\{{\mathbf{e}}_{i-1}=x\mid {\mathbf{e}}_i=y\}=\frac{{\alpha }_{i-1}^{+}\left(x\right)P(x,y)}{{\Sigma }_{z\in \mathcal{M}}{\alpha }_{i-1}^{+}\left(z\right)P(z,y)},$$

  $$\mathbb{P}\{{\tilde{e}}_{i-1}=x\mid {\breve{\mathbf{e}}}_i=y\}=\frac{{\alpha }_{i-1}^{-}\left(x\right)P(x,y)}{{\Sigma }_{z\in \mathcal{M}}{\alpha }_{i-1}^{-}\left(z\right)P(z,y)}.$$

  We extend the chain past τ, requiring ${\mathbf{e}}_t={\breve{\mathbf{e}}}_t$ for t > τ, as a realization of the Markov transition probabilities P, to obtain a total sample of predetermined length m. The random variable is then

  $$Z≔\frac{A}{C\left(\tau \right)}\text{log}\frac{\parallel X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_{\tau }}\cdots X_{{\mathbf{e}}_0}\parallel }{\parallel X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_{\tau +1}}{X}_{{\breve{\mathbf{e}}}_{\tau }}\cdots X_{{\breve{\mathbf{e}}}_0}\parallel }.$$

  (Note that the realizations corresponding to τ = 0 are identically 0. The possibility of τ = 0 has been included only to simplify the notation. In practice, we are free to condition on τ > 0.)

The change from q to $\mathring{q}$ is an example of importance sampling (cf. Chapter V.1 in Asmussen and Glynn (2007)). We oversample the values of the random variable with high τ to reduce the variability of the estimate. The importance sampling makes Z(j) a bounded random variable, with bound A. Imagine that we had a source of perfect samples V^T (j) from the distribution ${\tilde{\pi }}_{{\mathbf{e}}_m}$, and define

$$\tilde{Z}\left(j\right)≔\frac{A}{C\left(\tau \left(j\right)\right)}\text{log}\frac{\parallel V^T\left(j\right)X_{{\mathbf{e}}_m\left(j\right)}\cdots X_{{\mathbf{e}}_0\left(j\right)}\parallel }{\parallel V^T\left(j\right)X_{{\breve{\mathbf{e}}}_m\left(j\right)}\cdots X_{{\breve{\mathbf{e}}}_0\left(j\right)}\parallel }.$$

Let Y(j) := X_em(j) ⋯ X_e0(j) and $\breve{Y}\left(j\right)≔X_{{\breve{\mathbf{e}}}_m\left(j\right)}\cdots X_{{\breve{\mathbf{e}}}_0\left(j\right)}$. Then

$$\mid \tilde{Z}\left(j\right)-Z\left(j\right)\mid \le \frac{A{\Delta }_{\nu }\left(1\right)}{C\left(\tau \left(j\right)\right)}\left({\tau }_B\left(Y\left(j\right)\right)+{\tau }_B\left(\breve{Y}\left(j\right)\right)\right).$$ (34)

Since $\mathbb{E}\left[\tilde{Z}\right]={\zeta }_e$, we may use (22) to compute the bound

$$\mathbb{P}\left\{\mid {\zeta }_e-n^{-1}\sum _{j=1}^n\tilde{Z}\left(j\right)\mid >2A\sqrt{-\frac{1}{2J}\text{log}(p_0∕2)}\right\}\le p_0.$$ (35)

Lemma 6.1 The limits defining the coefficients ${}_{\breve{e}}\zeta _e$ and ζ_e exist and are finite. We may approximate ζ_e by

$$\frac{1}{J}\sum _{j=1}^J\frac{A}{C\left(\tau \left(j\right)\right)}\text{log}\frac{\parallel X_{e_{\tau }\left(j\right)}\cdots X_{e_0\left(j\right)}\parallel }{\parallel X_{{\tilde{e}}_{\tau }\left(j\right)}\cdots X_{{\tilde{e}}_0\left(j\right)}\parallel }.$$ (36)

If 0 < p₀ ≤ 1, the probability is no more than p₀ that the error in this estimation is larger than

$$\frac{1}{J}\sum _{j=1}^J\frac{A{\Delta }_{\nu }\left(1\right)}{C\left(\tau \left(j\right)\right)}\left({\tau }_B\left(Y\left(j\right)\right)+{\tau }_B\left(\breve{Y}\left(j\right)\right)\right)+2A\sqrt{-\frac{1}{2J}\text{log}(p_0∕2)},$$ (37)

It remains to bound A. From standard Markov chain theory, for any vector $v\in \mathcal{S}$,

$$\parallel v^T{P}^t-v^t\parallel \le D{\xi }^t.$$ (38)

Setting setting Q := P − 1ν^T, ${\alpha }_t^T$ is the e-th row of Q_t, which may also be written as

$${\alpha }_t=1_e^T{\left(Q^t\right)}^T,$$

where 1_e is the column vector with 1 in place e and 0 elsewhere. Thus

$$\parallel {\alpha }_t\parallel \le D{\xi }^t.$$

If we use the bound $C\left(t\right)=\widehat{k}+t\widehat{r}$, then

$$\begin{array}{cc}\hfill A& =\sum _{t=1}^{\infty }(\widehat{k}+t\widehat{r})(\parallel {\alpha }_{t-1}\parallel -\parallel {\alpha }_t\parallel )\hfill \\ \hfill & =\widehat{k}\parallel {\alpha }_1\parallel +\widehat{r}\sum _{t=1}^{\infty }\parallel {\alpha }_t\parallel \hfill \\ \hfill & \le D\xi \left(\widehat{k}+\frac{D\widehat{r}}{1-\xi }\right)\hfill \end{array}$$ (39)

7 Derivatives with respect to Environmental Transitions

We are now ready to compute derivatives of a with respect to changes in the distribution of environments, as determined by P. Complicating the notation slightly is the constraint {$P:{\sum }_{\breve{e}}P(e,\breve{e})=1$ for each e}; thus, there can be no sense in speaking of the derivative with respect to changes in $P(e,\breve{e})$ for some particular e, $\breve{e}$. Instead, we must compute directional derivatives along the direction of some matrix W, in the plane ${\sum }_{\breve{e}}{W}_{e,\breve{e}}=0$.

For the purposes of this result we write a(ε) = a(P^(ε)), where P^(ε) is a differentiable curve of M × M matrices, where the parameter ε takes values either in a two-sided interval [−ε₀, ε₀], or a one-sided interval [0, ε₀]. Let W := ∂P^(ε)/∂ε, an M × M matrix whose rows all sum to 0. The perturbations are such that P^(ε) retains the ergodicity and irreducibility of P. (The result should be the same whether ε is positive or negative. If P is on the boundary of the set of possible values, one or the other sign may be impossible. Some choices of W may be impossible in both directions.) In the special case in which $W_{e,\breve{e}}=1$ and $W_{e,\breve{e}}=-1$, with all other entries 0, we are computing the derivative corresponding to a small increase in the rate of transitioning from environment e to $\breve{e}$, and a decrease in the rate of transitioning to $\breve{e}$.

In this section the matrices X₁, …, X_M are assumed fixed.

Theorem 7.1 The derivative of the stochastic growth rate is

$$a\prime \left(0\right)=\sum _{e,\breve{e}\in \mathcal{M}}{\nu }_{\breve{e}}{W}_{\breve{e},e}\left({\zeta }_e+\mathbb{E}\left[\text{log}\frac{V_e^T{X}_e{X}_{\breve{e}}{U}_{\breve{e}}}{\parallel V_e^T{X}_e\parallel }\right]\right),$$ (40)

where $U_{\breve{e}}$, $V_e^T$ are independent random variables with distributions ${\pi }_{\breve{e}}$ and ${\tilde{\pi }}_e$ respectively.

The quantities ζ_e may be approximated, with error bounds, according to the algorithm described in section 6.

The other part of the expression may be approximated by averaging samples of the form

$$\sum _{e,\tilde{e}\in \mathcal{M}}{\nu }_{\tilde{e}}{W}_{\tilde{e},e}\text{log}\frac{V_e^{\left(m\right)}{X}_e{X}_{\tilde{e}}{U}_{\tilde{e}}^{\left(m\right)}}{\parallel V_e^{\left(m\right)T}{X}_e\parallel },$$ (41)

where

$$U_{\tilde{e}}^{\left(m\right)}=\frac{X_{{\tilde{\mathbf{e}}}_1}\cdots X_{{\tilde{\mathbf{e}}}_m}{u}_0}{\parallel X_{{\tilde{\mathbf{e}}}_1}\cdots X_{{\tilde{\mathbf{e}}}_m}{u}_0\parallel }$$

and

$$V_e^{\left(m\right)T}=\frac{v_0^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}}{\parallel v_0^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}\parallel },$$

and e = e₀, e₁, …, e_m is a Markov chain with transition matrix P, and $\tilde{e}={\tilde{\mathbf{e}}}_0$, ${\tilde{\mathbf{e}}}_1$, …, ${\tilde{\mathbf{e}}}_m$ is an independent Markov chain with transition probabilities $\tilde{P}$, and $u_0\in \mathcal{U}$ and $v_0^T\in \mathcal{V}$.

The systematic error may be bounded uniformly by 2k₂r^m∥(ν^T|W|)∥, while the sampling error at level p on J samples bounded by

$$2\parallel {\nu }^T\mid W\mid \parallel \underset{\underset{v^T\in \mathcal{V}}{u\in \mathcal{M}}}{\text{sup}}\underset{e\in \mathcal{M}}{\text{max}}\mid \text{log}{v}^T{X}_{e}u\mid {\left(\frac{-\text{log}p}{2J}\right)}^{1∕2.}.$$ (42)

Suppose the simulated expectation is

$$\frac{1}{J}\sum _{j=1}^J\sum _{\tilde{e},e\in \mathcal{M}}{\nu }_{\tilde{e}}{W}_{e,\tilde{e}}\text{log}\frac{V^{(m,\tilde{e})T}\left(j\right)X_{\tilde{e}}{X}_e{U}^{(m,e)}\left(j\right)}{\parallel V^{(m,\tilde{e})T}\left(j\right)X_{\tilde{e}}\parallel },$$

where

$$U_{\tilde{e}}^{\left(m\right)}\left(j\right)≔\frac{X_{{\tilde{\mathbf{e}}}_1\left(j\right)}\cdots X_{{\tilde{\mathbf{e}}}_m\left(j\right)}{u}_0}{\parallel X_{{\tilde{\mathbf{e}}}_1\left(j\right)}\cdots X_{{\tilde{\mathbf{e}}}_m\left(j\right)}{u}_0\parallel }≕\frac{{\tilde{Y}}_{\tilde{e}}^{\left(m\right)}\left(j\right)u_0}{\parallel {\tilde{Y}}_{\tilde{e}}^{\left(m\right)}\left(j\right)u_0\parallel }$$

and

$$V_e^{\left(m\right)T}\left(j\right)=\frac{v_0^T{X}_{{\mathbf{e}}_m\left(j\right)}\cdots X_{{\mathbf{e}}_1\left(j\right)}}{\parallel v_0^T{X}_{{\mathbf{e}}_m\left(j\right)}\cdots X_{{\mathbf{e}}_1\left(j\right)}\parallel }≕\frac{v_0^T{Y}_e^{\left(m\right)}\left(j\right)}{\parallel v_0^T{Y}_e^{\left(m\right)}\left(j\right)\parallel }.$$

We may also bound the systematic error by

$$\begin{array}{cc}\hfill \frac{1}{J}\sum _{j=1}^J& \sum _{\tilde{e},e\in \mathcal{M}}{\nu }_{\tilde{e}}\mid W_{\tilde{e},e}\mid \left(\underset{u,\breve{u}\in \mathcal{U}}{\text{sup}}\rho ({\tilde{Y}}_{\tilde{e}}^{\left(m\right)}\left(j\right)u,{\tilde{Y}}_{\tilde{e}}^{\left(m\right)}\left(j\right)\breve{u})+\underset{v^T,{\breve{v}}^T\in \mathcal{V}}{\text{sup}}\rho (v^T{Y}_e^{\left(m\right)}\left(j\right)X_e,{\breve{v}}^T{Y}_e^{\left(m\right)}\left(j\right)X_e)\right)\hfill \\ \hfill & \le \frac{\Delta }{J}\sum _{j=1}^J\sum _{\tilde{e},e\in \mathcal{M}}{\nu }_{\tilde{e}}\mid W_{e,\tilde{e}}\mid \left({\tau }_B\left({\tilde{Y}}_{\tilde{e}}^{\left(m\right)}\left(j\right)\right)+{\tau }_B\left(X_e^T{Y}_e^{\left(m\right)}{\left(j\right)}^T\right)\right).\hfill \end{array}$$ (43)

We note that expressions like (40) are examples of what Brémaud (1992) calls “ersatz derivatives”. In a rather different class of applications Brémaud suggests applying maximal coupling.

8 Discussion

Our results provide analytical formulas and simulation estimators for the derivatives of stochastic growth rate with respect to the transition probability matrix or the population projection matrices. We have concentrated here on the theoretical results; although this may not be obvious, we have made considerable effort at brevity. Partly for this reason, we will present elsewhere numerical applications of these results. We expect that our results should carry over to integral population models (IPMs), given the strong parallels between the stochastic ergodicity properties of IPMs and matrix models (Ellner and Rees, 2007).

Our results apply not only to stochastic structured populations but to any stochastic system in which a Lyapunov exponent of a product of random matrices determines stability or other dynamic properties. Examples include the net reproductive rate in epidemic models and some models of network dynamics. An obvious application of our results is to the analysis of optimal life histories, i.e., environment-to-projection matrix maps that maximize the stochastic growth rate. As discussed by McNamara (1997), this optimization problem translates into what is called an average reward problem in stochastic control theory, and so our results may be more generally useful in such control problems.

Highlights

• >We analyse the growth rate a in matrix population models driven by Markov environments.
• >We derive formulas for derivatives of a with respect to changes in the Markov parameters.
• >We prove known formulas for the derivatives with respect to demographic parameters.

Appendix

Proofs of the theorems.

A.1 Proof of Lemma 3.2

We have

$$\begin{array}{cc}\hfill \mid \log \frac{v^{T}u}{{\breve{v}}^{T}u}-\log \frac{v^T\breve{u}}{{\breve{v}}^T\breve{u}}\mid & =\mid \log \frac{\sum _{i,j}{v}_i{u}_i{\breve{v}}_j{\breve{u}}_j}{\sum _{i,j}{v}_i{\breve{u}}_i{\breve{v}}_j{u}_j}\mid \hfill \\ \hfill & =\mid \log \frac{\sum _i{u}_i{v}_i{\breve{u}}_i{\breve{u}}_i+\sum _{i<j}(v_i{u}_i{\breve{v}}_j{\breve{u}}_j+v_j{u}_j{\breve{v}}_i{\breve{u}}_i)}{\sum _i{u}_i{v}_i{\breve{u}}_i{\breve{u}}_i+\sum _{i<j}(v_i{\breve{u}}_i{\breve{v}}_j{u}_j+v_j{\breve{u}}_j{\breve{v}}_i{u}_i)}\mid \hfill \\ \hfill & \le \underset{i,j}{\max }\mid \log \frac{v_i{u}_i{\breve{v}}_j{\breve{u}}_j+v_j{u}_j{\breve{v}}_i{\breve{u}}_i}{v_i{\breve{u}}_i{\breve{v}}_j{u}_j+v_j{\breve{u}}_j{\breve{v}}_i{u}_i}\mid \hfill \end{array}$$

For some choice of i, j, let

$$\alpha ≔\log \frac{u_i}{{\breve{u}}_i},\beta ≔\log \frac{v_i}{{\breve{v}}_i},\gamma ≔\log \frac{u_j}{{\breve{u}}_j},\delta ≔\log \frac{v_j}{{\breve{v}}_j}.$$

Note that by (5),

$$\begin{array}{c}\hfill \mid \alpha \mid \le \rho (u,\breve{u})+\mid \log \parallel u\parallel ∕\parallel \breve{u}\parallel \mid \le \frac{1}{4},\hfill \\ \hfill \mid \beta \mid \le \rho (v,\breve{v})+\mid \log \parallel v\parallel ∕\parallel \breve{v}\parallel \mid \le \frac{1}{4},\hfill \\ \hfill \mid \gamma \mid \le \rho (u,\breve{u})+\mid \log \parallel u\parallel ∕\parallel \breve{u}\parallel \mid \le \frac{1}{4},\hfill \\ \hfill \mid \delta \mid \le \rho (v,\breve{v})+\mid \log \parallel u\parallel ∕\parallel \breve{u}\parallel \mid \le \frac{1}{4}.\hfill \end{array}$$ (44)

Then we need to bound

$$\begin{array}{cc}\hfill \mid \log \frac{e^{\alpha +\beta }+e^{\gamma +\delta }}{e^{\beta +\delta }+e^{\alpha +\gamma }}\mid & \le \frac{\mid e^{\alpha +\beta }+e^{\gamma +\delta }-e^{\beta +\delta }-e^{\alpha +\gamma }\mid }{\min \{e^{\alpha +\beta }+e^{\gamma +\delta },e^{\beta +\delta }+e^{\alpha +\delta }\}}\hfill \\ \hfill & \le \mid e^{\alpha +\beta }+e^{\gamma +\delta }-e^{\beta +\delta }-e^{\alpha +\gamma }\mid ,\hfill \end{array}$$ (45)

where we have used the relation |log x| ≤ max{x, 1/x} − 1.

We now use the Taylor series expansion

$$e^x=\sum _{k=0}^n\frac{x^k}{k!}+\frac{C{x}^{k+1}}{(k+1)!},$$

where |C| ≤ max{e^x, 1}. This turns the right-hand side of (45) into

$$2\sum _{k=2}^{\infty }\frac{1}{k!}\mid {(\alpha +\beta )}^k+{(\gamma +\delta )}^k-{(\beta +\delta )}^k-{(\alpha +\gamma )}^k\mid =2\sum _{k=2}^{\infty }\mid \sum _{\ell =1}^{k-1}\frac{1}{\ell !(k-\ell )!}({\alpha }^{\ell }{\beta }^{k-\ell }+{\gamma }^{\ell }{\delta }^{k-\ell }-{\beta }^{\ell }{\delta }^{k-\ell }-{\alpha }^{\ell }{\gamma }^{k-\ell })\mid \le 2\sum _{k=0}^{\infty }\frac{1}{(k-2)!}\left(\alpha \beta \sum _{\ell =0}^{k-2}\left(\underset{\ell }{\overset{k-2}{}}\right){\alpha }^{\ell }{\beta }^{k-2-\ell }+\mid \gamma \delta \mid \sum _{\ell =0}^{k-2}\left(\underset{\ell }{\overset{k-2}{}}\right){\gamma }^{\ell }{\delta }^{k-2-\ell }+\mid \beta \delta \mid \sum _{\ell =0}^{k-2}\left(\underset{\ell }{\overset{k-2}{}}\right){\beta }^{\ell }{\delta }^{k-2-\ell }+\mid \alpha \gamma \mid \sum _{\ell =0}^{k-2}\left(\underset{\ell }{\overset{k-2}{}}\right){\alpha }^{\ell }{\gamma }^{k-2-\ell }\right).$$

Applying the bounds (44) yields finally the upper bound in (15).

A.2 Estimating the stochastic growth rate

We prove here Theorem 5.1. The quantity we are trying to compute is

$$a=\mathbb{E}\left[\log \parallel X_{\mathbf{e}}U\parallel \right],\text{where}(U,\mathbf{e})\text{is selected from the distribution}\pi .$$ (46)

Let e₀, e₁, e₂, … be a realization of the stationary Markov chain with transition matrix P. Let Y_m := X_emX_em−1 ⋯ X_e1. Choose $u_0\in \mathcal{U}$, and let U be a random variable with distribution π_e0. Then $a=\mathbb{E}[\log \Vert X_{{\mathbf{e}}_{m+1}}{Y}_{m}U\Vert ∕\Vert Y_{m}U\Vert ]$, which may be approximated by $\mathbb{E}[\log \Vert X_{{\mathbf{e}}_{m+1}}{Y}_m{u}_0\Vert ∕\Vert Y_m{u}_0\Vert ]$.

If we identify systematic error with bias, this is

$${\text{Error}}_{\text{sys}}=\mid \mathbb{E}\left[\log \frac{\parallel X_{{\mathbf{e}}_{m+1}}{Y}_m{u}_0\parallel }{\parallel Y_m{u}_0\parallel }\right]-\mathbb{E}\left[\log \frac{\parallel X_{{\mathbf{e}}_{m+1}}{Y}_{m}U\parallel }{\parallel Y_{m}U\parallel }\right]\mid ,$$

since (Y_mU/∥Y_mU∥, e_m) also has the distribution π (if Y_m and U are taken to be independent, conditioned on e₀). Thus

$$\begin{array}{cc}\hfill {\text{Error}}_{\text{sys}}& \le \mathbb{E}\left[\mid \log \frac{\parallel X_{e_{m+1}}{Y}_m{u}_0\parallel }{\parallel Y_m{u}_0\parallel }-\log \frac{\parallel X_{e_{m+1}}{Y}_{m}U\parallel }{\parallel Y_{m}U\parallel }\mid \right]\hfill \\ \hfill & \le \mathbb{E}\left[\underset{u,\breve{u}\in \mathcal{U}}{\sup }\mid \log \frac{\parallel X_{e_{m+1}}{Y}_{m}u\parallel }{\parallel Y_{m}u\parallel }-\log \frac{\parallel X_{e_{m+1}}{Y}_m\breve{u}\parallel }{\parallel Y_m\breve{u}\parallel }\mid \right]\hfill \\ \hfill & \le k_2{r}^m,\hfill \end{array}$$

by (13). The corresponding bound on the sampling error may be computed from (22).

For a particular choice of of e₁, …, e_m+1 and U we can also represent the random systematic error as

$$\mid \log \frac{\parallel X_{{\mathbf{e}}_{m+1}}{Y}_m{u}_0\parallel }{\parallel Y_m{u}_0\parallel }-\log \frac{\parallel X_{{\mathbf{e}}_{m+1}}{Y}_{m}U\parallel }{\parallel Y_{m}U\parallel }\mid ,$$

which may be bounded by the summand in (26).

A.3 Estimating sensitivities: Matrix entries

We prove here Theorem 5.2. As discussed at the end of section 3.1, we may assume that the compact sets $\mathcal{U}$ and $\mathcal{V}$ are stable and satisfy the bounds of section 3.1 simultaneously for all $X_e^{\left(∊\right)}$. The stationary distributions corresponding to products of the perturbed matrices are denoted π^(ε) and ${\tilde{\pi }}_e^{\left(∊\right)}$, and the corresponding regular conditional distributions are π_e^(ε) and ${\tilde{\pi }}_e^{\left(∊\right)}$

The derivative a′(0) may be written as

$$\begin{array}{cc}\hfill \underset{∊\to 0}{\lim }{∊}^{-1}\left(\underset{m\to \infty }{\lim }\mathbb{E}\left[\log \frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_0}^{\left(∊\right)}{u}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_0}^{\left(∊\right)}{u}_0\parallel }\right]-\underset{m\to \infty }{\lim }\mathbb{E}\left[\log \frac{\parallel X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }\right]\right)& =\left(\underset{m\to \infty }{\lim }\mathbb{E}\left[\log \frac{\parallel X_{{\tilde{e}}_0}^{\left(∊\right)}{X}_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_m}^{\left(∊\right)}{u}_0\parallel }{\parallel X_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_m}^{\left(∊\right)}{u}_0\parallel }\right]-\underset{m\to \infty }{\lim }\mathbb{E}\left[\log \frac{\parallel X_{{\tilde{e}}_0}{X}_{{\tilde{e}}_1}\cdots X_{{\mathbf{e}}_m}{u}_0\parallel }{\parallel X_{{\tilde{e}}_1}\cdots X_{{\mathbf{e}}_m}{u}_0\parallel }\right]\right)\hfill \\ \hfill & =\underset{∊\to 0}{\lim }\underset{m\to \infty }{\lim }\sum _{s=0}^m{a}_{s,m}\left(∊\right)=:\underset{∊\to 0}{\lim }\underset{m\to \infty }{\lim }A(m,∊),\hfill \end{array}$$

where

$$a_{s,m}\left(∊\right):=\mathbb{E}\left[{∊}^{-1}\left(\log \frac{\parallel X_{{\tilde{e}}_0}^{\left(∊\right)}{X}_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_s}^{\left(∊\right)}{X}_{{\tilde{e}}_{s+1}}\cdots X_{{\tilde{e}}_m}{u}_0\parallel }{\parallel X_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_s}^{\left(∊\right)}{X}_{{\tilde{e}}_{s+1}}\cdots X_{{\tilde{e}}_m}{u}_0\parallel }-\log \frac{\parallel X_{{\tilde{e}}_0}^{\left(∊\right)}{X}_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_{s-1}}^{\left(∊\right)}{X}_{{\tilde{e}}_s}\cdots X_{{\tilde{e}}_m}{u}_0\parallel }{\parallel X_{{\tilde{e}}_1}^{\left(∊\right)}\cdots X_{{\tilde{e}}_{s-1}}^{\left(∊\right)}{X}_{{\tilde{e}}_s}\cdots X_{{\tilde{e}}_m}{u}_0\parallel }\right)\right].$$

Here e₀, e₁, … is the stationary Markov chain with transition probabilities P, and ${\tilde{\mathbf{e}}}_0$, ${\tilde{\mathbf{e}}}_1$, … is the reverse stationary Markov chain, with transition probabilities $\tilde{P}$.

By (13), for ∊ > 0 sufficiently small

$$\mid a_{s,m}\left(∊\right)\mid \le {∊}^{-1}{k}_2{r}^{s-1}\underset{u\in \mathcal{U}}{\sup }\mathbb{E}\left[\rho (X_{{\tilde{e}}_s}^{\left(∊\right)}u,X_{{\tilde{e}}_s}u)\right]\le 2k_{2}C{r}^s,$$

where

$$C:=r^{-1}\underset{e\in \mathcal{M}}{\max }\underset{1\le \ell \le K}{\max }\left\{\underset{u\in \mathcal{U}}{\max }\frac{{\mid X_e^{\prime }u\mid }_{\ell }}{{\left(X_{e}u\right)}_{\ell }},\underset{v\in \mathcal{V}}{\max }\frac{{\mid v^T{X}_e^{\prime }\mid }_{\ell }}{{\left(v^T{X}_e\right)}_{\ell }}\right\}.$$

If we define

$$V_{i,j}:=\frac{1^T{X}_{{\tilde{e}}_i}^{\left(∊\right)}{X}_{{\tilde{e}}_{i+1}}^{\left(∊\right)}\cdots X_{{\tilde{e}}_j}^{\left(∊\right)}}{\parallel 1^T{X}_{{\tilde{e}}_i}^{\left(∊\right)}{X}_{{\tilde{e}}_{i+1}}^{\left(∊\right)}\cdots X_{{\tilde{e}}_j}^{\left(∊\right)}\parallel }$$

$$U_{i,j}:=\frac{X_{{\tilde{e}}_i}{X}_{{\tilde{e}}_{i+1}}\cdots X_{{\tilde{e}}_j}{u}_0}{\parallel X_{{\tilde{e}}_i}{X}_{{\tilde{e}}_{i+1}}\cdots X_{{\tilde{e}}_j}{u}_0\parallel },$$

then for m ≤ n,

$$\mid a_{s,m}\left(∊\right)-a_{s,n}\left(∊\right)\mid \le {∊}^{-1}\mathbb{E}\left[\mid \log \frac{V{(0,s-1)}^T{X}_{{\tilde{e}}_s}^{\left(∊\right)}U(s+1,m)}{V{(0,s-1)}^T{X}_{{\tilde{e}}_s}U(s+1,m)}-\log \frac{V{(0,s-1)}^T{X}_{{\tilde{e}}_s}^{\left(∊\right)}U(s+1,n)}{V{(0,s-1)}^T{X}_{{\tilde{e}}_s}U(s+1,n)}\mid +\mid \log \frac{V_{1,s-1}^T{X}_{{\tilde{e}}_s}^{\left(∊\right)}{U}_{s+1,m}}{V_{1,s-1}^T{X}_{{\tilde{e}}_s}{U}_{s+1,m}}-\log \frac{V_{1,s-1}^T{X}_{{\tilde{e}}_s}^{\left(∊\right)}{U}_{s+1,n}}{V_{1,s-1}^T{X}_{{\tilde{e}}_s}{U}_{s+1,n}}\mid \right]$$

We note by (10) that ρ(U_s+1,m, U_s+1,n) ≤ k₂r^m−s; and for ∊ > 0 sufficiently small and any $v\in \mathcal{V}$ we have

$$\underset{\underset{e\in \mathcal{M}}{1\le j\le K,}}{\text{max}},\mid \text{log}\frac{{\left(v^T{X}_e^{\left(∊\right)}\right)}_j}{{\left(v^T{X}_e\right)}_j}\mid \le 2C∊.$$

It follows by Lemma 3.2 that

$$\mid a_{s,m}\left(∊\right)-a_{s,n}\left(∊\right)\mid \le \text{min}\{16k_2{\mathit{Cr}}^{m-s},4k_2{\mathit{Cr}}^s\}.$$ (47)

Putting these together, we get, for all n ≥ m,

$$\begin{array}{cc}\hfill \mid A(m,∊)-A(n,∊)\mid & \le \sum _{s=m+1}^n\mid a_{s,n}\left(∊\right)\mid +\sum _{s=0}^m\mid a_{s,m}\left(∊\right)-a_{s,n}\left(∊\right)\mid \hfill \\ \hfill & \le \sum _{s=m+1}^{n}2k_2{\mathit{Cr}}^s+\sum _{1\le s\le m∕2}16k_2{\mathit{Cr}}^{m-s}+\sum _{m∕2<\le m}4k_2{\mathit{Cr}}^s\hfill \\ \hfill & \le \frac{20k_{2}C}{1-r}{r}^{m∕2}.\hfill \end{array}$$

By Lemma A.1 we may exchange the order of the limits, to see that

$$a\prime =\underset{m\to \infty }{\text{lim}}\underset{∊\to 0}{\text{lim}}{∊}^{-1}\sum _{s=1}^m\left(\mathbb{E}\left[\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }\right]-\mathbb{E}\left[\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}\cdots X_{{\mathbf{e}}_0}{u}_0\parallel }\right]\right).$$ (48)

This limit is the same for any choice of u₀, hence would also be the same if we replaced u₀ by a random U, with any distribution on $\mathcal{U}$. We choose U to have the distribution π_e0, independent of the rest of the Markov chain. By the invariance property of the distributions π,

$$\begin{array}{cc}\hfill a\prime & =\underset{m\to \infty }{\text{lim}}\underset{∊\to 0}{\text{lim}}{∊}^{-1}\left(\sum _{s=1}^m\mathbb{E}\left[\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_0}{U}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{X}_{{\mathbf{e}}_{s-1}}\cdots X_{{\mathbf{e}}_0}{U}_0\parallel }\right]-\mathbb{E}\left[\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}\cdots X_{{\mathbf{e}}_0}{U}_0\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}\cdots X_{{\mathbf{e}}_0}{U}_0\parallel }\right]\right)\hfill \\ \hfill & =\underset{m\to \infty }{\text{lim}}\sum _{s=0}^m\underset{∊\to 0}{\text{lim}}{∊}^{-1}\mathbb{E}\left[\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}\parallel }-\text{log}\frac{\parallel X_{{\mathbf{e}}_m}^{\left(∊\right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}{U}_{s-1}\parallel }{\parallel X_{{\mathbf{e}}_{m-1}}^{\left(∊\right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(∊\right)}{X}_{{\mathbf{e}}_s}{U}_{s-1}\parallel }\right],\hfill \end{array}$$ (49)

where (U_s−1, e_s) has distribution π.

For m ≥ s ≥ 1 define functions

$$f_s(∊,\delta )≔\frac{1^T{X}_{{\mathbf{e}}_m}^{\left(\delta \right)}{X}_{{\mathbf{e}}_{m-1}}^{\left(\delta \right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(\delta \right)}{X}_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}}{1^T{X}_{{\mathbf{e}}_{m-1}}^{\left(\delta \right)}\cdots X_{{\mathbf{e}}_{s+1}}^{\left(\delta \right)}{X}_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}},$$

where the denominator is understood to be 1 for s = m. The summand on the right of (49) may be written as

$$\underset{∊\to 0}{\text{lim}}{∊}^{-1}\mathbb{E}[\text{log}{f}_s(∊,∊)-\text{log},f_s(0,∊)].$$ (50)

By (13),

$${∊}^{-1}\mathbb{E}\mid \text{log}{f}_s(∊,\delta )-\text{log},f_s(0,\delta )]\le {∊}^{-1}{k}_{2^{r^s}}\rho (X_{e_s}^{\left(∊\right)}U,X_{e_s}U)<2{\mathit{Ck}}_{2^{r^s}}$$

for ∊ in a neighborhood of 0, so the Bounded Convergence Theorem turns (50) into

$$\mathbb{E}\left[\underset{∊\to 0}{\text{lim}}{∊}^{-1}(\text{log}{f}_s(∊,∊)-\text{log},f_s(0,∊))\right]=\mathbb{E}\left[\frac{\partial \text{log}{f}_s}{\partial ∊}(0,0)\right].$$ (51)

We have, by linearity of the matrix product and ∥ · ∥,

$$\begin{array}{cc}\hfill \frac{\partial \text{log}{f}_s}{\partial ∊}(0,0)& =\frac{1^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_{s+1}}\frac{\partial }{\partial ∊}{X}_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}}{\parallel X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_s}{U}_{s-1}\parallel }-\frac{1^T{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_{s+1}}\frac{\partial }{\partial ∊}{X}_{{\mathbf{e}}_s}^{\left(∊\right)}{U}_{s-1}}{\parallel X_{e_{m-1}}\cdots X_{{\mathbf{e}}_s}{U}_{s-1}\parallel }\hfill \\ \hfill & =\{\begin{array}{cc}\left(\frac{1^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_{s+1}}{X}_{{\mathbf{e}}_s}^{\prime }{U}_{s-1}}{\parallel X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_s}{U}_{s-1}\parallel }-\frac{1^T{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_{s+1}}{X}_{{\mathbf{e}}_s}^{\prime }{U}_{s-1}}{\parallel X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_s}{U}_{s-1}\parallel }\right)\hfill & \text{for}1\le s\le m-1,\hfill \\ \frac{1X_{{\mathbf{e}}_s}^{\prime }{U}_{m-1}}{\parallel X_{{\mathbf{e}}_m}{U}_{m-1}\parallel }\text{for}s=m\hfill & \hfill \end{array}\phantom{\}}\hfill \end{array}$$

Combining this with (49) yields the telescoping sum

$$\begin{array}{cc}\hfill a\prime \left(0\right)=& \underset{m\to \infty }{\text{lim}}\left(\sum _{t=1}^m\mathbb{E}\left[\frac{1^T{X}_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}^{\prime }U}{1^T{X}_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}U}\right]\right)-\sum _{t=1}^{m-1}\mathbb{E}\left[\frac{1^T{X}_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}^{\prime }U}{1^T{X}_{{\mathbf{e}}_t}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}U}\right]\hfill \\ \hfill & =\underset{m\to \infty }{\text{lim}}\mathbb{E}\left[\frac{1^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}^{\prime }U}{1^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}{X}_{{\mathbf{e}}_0}U}\right],\hfill \end{array}$$

where in the last line (U, e₀) has the distribution π. De ne $V_m^T:=1^T{X}_{e_m}\cdots X_{e_1}∕\parallel 1^T{X}_{e_m}{\cdots }_{e_1}\parallel$. Then $V_m^T$ converges in distribution to V, with distribution ${\tilde{\pi }}_{e_0}$ (conditioned on e₀), and so

$$a\prime \left(0\right)=\underset{m\to \infty }{\text{lim}}\mathbb{E}\frac{V_m^T{X}_{{\mathbf{e}}_0}^{\prime }U}{V_m^T{X}_{{\mathbf{e}}_0}U}=\sum _{e\in \mathcal{M}}{\nu }_e\mathbb{E}\frac{V_e^T{X}_e^{\prime }{U}_e}{V_e^T{X}_e{U}_e},$$

which is identical to (27).

Now we estimate the error. We use the representation

$$U≔\frac{X_{{\tilde{e}}_1}\cdots X_{{\tilde{e}}_m}{U}_0}{\parallel X_{{\tilde{e}}_1}\cdots X_{{\tilde{e}}_m}{U}_0\parallel }$$

$$V^T≔\frac{V_0^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}}{\parallel V_0^T{X}_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}\parallel },$$

where U₀ and $V_0^T$ are assumed to have distributions ${\pi }_{{\tilde{e}}_m}$ and ${\tilde{\pi }}_{e_m}$ respectively. Then

$$\begin{array}{cc}\hfill \mid \frac{V^{\left(m\right)T}{X}_e^{\prime }{U}^{\left(m\right)}}{V^{\left(m\right)T}{X}_e{U}^{\left(m\right)}}-\frac{V^T{X}_e^{\prime }}{V^T{X}_{e}U}\mid & \le 2\left(e^{2\rho (V,V^{\left(m\right)T})+2\rho (U,U^{\left(m\right)})}-1\right)\frac{\mid V{X}_e^{\prime }U\mid }{\mid V{X}_{e}U\mid }\hfill \\ \hfill & \le 2\left(e^{4k_{2^{r^m}}}-1\right)\frac{\mid V{X}_e^{\prime }U\mid }{V{X}_{e}U},\hfill \end{array}$$ (52)

by (13). This implies the uniform bound on systematic error, and the bound on sampling error (30) follows from applying (22) to a trivial bound on the terms in the average. The simulated bound (31) also follows directly from (52).

Lemma A.1 Let A(m, ∊) be a two-dimensional array of real numbers, indexed by m ∈ $\mathbb{N}$ and ∊ > 0, with A(m, 0) := lim_∊↓0 A(m, ∊) existing for each m, and A(∞, ∊) := lim_m→∞ A(m, ∊) existing for all ∊ sufficiently small (independent of m). Suppose A satisfies

$$\underset{M\to \infty }{\text{lim}}\underset{∊\downarrow 0}{\text{lim sup}}\underset{m,n>M}{\text{sup}}\mid A(m,∊)-A(n,∊)\mid =0.$$ (53)

Then the two limits

$$\underset{m\to \infty }{\text{lim}}\underset{∊\downarrow 0}{\text{lim}}A(m,∊)$$

and

$$\underset{∊\downarrow 0}{\text{lim}}\underset{m\to \infty }{\text{lim}}A(m,∊)$$

are equal; in particular, if one exists the other exists as well.

Proof. Suppose A_* := lim_∊↓0 A(∞, ∊) exists. Then we need to show that A_* = lim_m→∞ A(m, 0).

Choose any δ > 0, and choose M such that

$$\underset{∊\downarrow 0}{\text{lim sup}}\underset{m,n>M}{\text{sup}}\mid A(m,∊)-A(n,∊)\mid <\delta .$$

This means that we may find ∊₀ > 0 such that |A(m, ∊) − A(n, ∊)| < 2δ when 0 < ∊ < ∊₀ and m, n < M, and such that also |A(∞, ∊) − A_*| < 2δ for all 0 < ∊ < ∊₀. It follows, in particular, that |A(m, ∊) − A_*| < 4δ when m > M and 0 < ∊ < ∊₀. Thus |A(m, 0) − A_*| < 4δ for m > M, and consequently |lim sup_m→∞ A(m, 0) − A_*| < 4δ. Since δ is arbitrary, it follows that lim_m→∞ A(m, 0) = A_*.

The converse result (starting from the assumption that lim_m→∞ A(m, 0) exists) follows identically.

A.4 Estimating sensitivities: Markov environments

We derive here the formula (40) by a combination of the coupling method and importance sampling. We use importance sampling for the actual computation, but coupling provides a more direct path to validating the crucial exchange of limits. As in the proof of Theorem 5.2, the error bounds are an obvious consequence of the formula (40) and the general formulas for errors described in section 4.

Given two distributions q and q* on {1, …,M}, we define a standard coupling between q and q*. Suppose we are given a uniform random variable ω on [0, 1]. Let ${\mathcal{M}}_{-}≔\{e:q_e<q_e^{\ast }\}$ and ${\mathcal{M}}_{+}≔\{e:q_e^{\ast }<q_e\}$. Let $\delta ≔{\Sigma }_{e\in {\mathcal{M}}_{-}}(q_e^{\ast }-q_e)={\Sigma }_{e\in {\mathcal{M}}_{+}}(q_e-q_e^{\ast })$. We define three random variables $\breve{e}$ on $\mathcal{M}$, ${\breve{e}}_{+}$ on ${\mathcal{M}}_{+}$, and ${\breve{e}}_{-}$ on ${\mathcal{M}}_{-}$, according to the following distributions:

$$\mathbb{P}\{\breve{e}=e\}=\text{min}\{q_e,q_e^{\ast }\}∕(1-\delta ),$$

$$\mathbb{P}\{{\breve{e}}_{+}=e\}={\{q_e-q_e^{\ast }\}}_{+}∕\delta ,$$

$$\mathbb{P}\{{\breve{e}}_{-}=e\}={\{q_e^{\ast }-q_e\}}_{+}∕\delta ,$$

The joint distribution is irrelevant, but for definiteness we let them be independent. Then we define the coupled pair (e, e*) to have the values

$$\begin{array}{cc}\hfill (\breve{e},\breve{e})\text{if}& \omega >\delta .\hfill \\ \hfill ({\breve{e}}_{+},{\breve{e}}_{-})\text{if}& \omega \le \delta .\hfill \end{array}$$ (54)

Then e has distribution q, e* has distribution q*, and e = e* with probability 1 − δ. This δ is called the total-variation distance between q and q*.

We write ${\mathbb{E}}_P$ for the expectation with respect to the distribution that makes e₀, …, e_m a stationary Markov chain with transition matrix P. Define ν^(∊) to be the stationary distribution corresponding to P^(∊), and define ${\tilde{P}}^{\left(∊\right)}$ to be the time-reversed chain of P^(∊). We define

$$g(m;∊;u)≔{\mathbb{E}}_{P\left(∊\right)}\left[\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]-{\mathbb{E}}_P\left[\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right].$$

By the time-reversal property,

$$g(m;∊;u)≔{\mathbb{E}}_{\tilde{P}\left(∊\right)}\left[\log \frac{\Vert X_{{\mathbf{e}}_0}{X}_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }{\Vert X_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }\right]-{\mathbb{E}}_{\tilde{P}}\left[\log \frac{\Vert X_{{\mathbf{e}}_0}{X}_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }{\Vert X_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }\right].$$

For ∊ > 0 we couple a sequence e₀, …, e_m selected from the distribution $\tilde{P}$ to a sequence ${\mathbf{e}}_0^{\left(∊\right)},\dots ,{\mathbf{e}}_m^{\left(∊\right)}$ selected from the distribution ${\tilde{P}}^{\left(∊\right)}$ as follows: We start by choosing $({\mathbf{e}}_0,{\mathbf{e}}_0^{\left(∊\right)})$ according to the standard coupling of (ν, ν^(∊). Assume now that we have produced sequences of length i, ending in e_i−1 and ${\mathbf{e}}_{i-1}^{\left(∊\right)}$. We then produce $({\mathbf{e}}_i,{\mathbf{e}}_i^{\left(∊\right)})$ according to the standard coupling of row e_i−1 of $\tilde{P}$ to row $e_{i-1}^{\left(∊\right)}$ of ${\tilde{P}}^{\left(∊\right)}$. (To simplify the typography in some places we use e(i) and e^(∊)(i) interchangeably with e_i and $e_i^{\left(∊\right)}$.)

Let δ = δ(∊) be the maximum of the total variation distance between ν and ν^(∊), and all of the pairs of rows. It is easy to see that there is a constant c such that δ ≤ c∊ for ∊ sufficiently small. Define ω₁, ω₂, … to be an i.i.d. sequence of uniform random variables on [0; 1], and two sequences of random times as follows: T₀ := S₀ := −1, and

$$T_{i+1}=\min \{t>S_i:{\omega }_t\le \delta \},$$

$$S_{i+1}=\min \{t>T_{i+1}:{\mathbf{e}}_t^{\left(∊\right)}={\mathbf{e}}_t\}.$$

Thus, $e_t^{\left(∊\right)}=e_t$ for all S_i ≤ t < T_i+1. Define for any $u_0\in \mathcal{U}$ the random vector

$$U_t≔\underset{m\to \infty }{\lim }\frac{X_{\mathbf{e}\left(t\right)}\cdots X_{\mathbf{e}(t+m)}{u}_0}{\Vert X_{\mathbf{e}\left(t\right)}\cdots X_{\mathbf{e}(t+m)}{u}_0\Vert },$$

and define a version of g conditioned on T₁ and T₂

$$g(m;∊;u;T_1,T_2)≔{\mathbb{E}}_{\tilde{P}\left(∊\right)}\left[\log \frac{\Vert X_{{\mathbf{e}}_0}{X}_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }{\Vert X_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }\mid T_1,T_2\right]-{\mathbb{E}}_{\tilde{P}}\left[\log \frac{\Vert X_{{\mathbf{e}}_0}{X}_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }{\Vert X_{{\mathbf{e}}_1}\cdots X_{{\mathbf{e}}_m}u\Vert }\mid T_1,T_2\right].$$

Then for any $u\in \mathcal{U}$,

$${\nabla }_{W}a\left(P\right)=a\prime \left(0\right)=\underset{∊\downarrow 0}{\lim }\underset{m\to \infty }{\lim }{∊}^{-1}\mathbb{E}\left[g(m;∊;u;T_1,T_2)\right].$$ (55)

We also define

$$\gamma (∊;T_1,T_2)≔\mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}^{\left(∊\right)}\left(0\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}(S_1-1)}{U}_{S_1}\Vert }{\Vert X_{{\mathbf{e}}^{\left(∊\right)}\left(1\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}(S_1-1)}{U}_{S_1}\Vert }-\log \frac{\Vert X_{\mathbf{e}\left(0\right)}{X}_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}(S_1-1)}{U}_{S_1}\Vert }{\Vert X_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}(S_1-1)}{U}_{S_1}\Vert }\mid T_1,T_2\right]$$

We break up these expectations into their portion overlapping three different events:

• (i){T₁ > m};
• (ii){T₂ > m ≥ T₁};
• (iii){m ≥ T₂}.

On the event {T₁ > m} we have g(m, ∊ u; T₁, T₂) = 0, and T₁ − m is geometrically distributed with parameter δ. By (13), γ is bounded by k₂r^T₁−1.

On the event {T₂ > m ≥ T₁}: We have e^(∊)(i) = e(i) for i < T₁ and for S₁ ≤ i ≤ m. If S₁ ≤ m,

$$\begin{array}{cc}\hfill U_{S_1}& =X_{\mathbf{e}\left(S_1\right)}\cdots X_{\mathbf{e}\left(m\right)}{U}_{m+1}∕\Vert X_{\mathbf{e}\left(S_1\right)}\cdots X_{\mathbf{e}\left(m\right)}{U}_{m+1}\Vert \hfill \\ \hfill & =X_{{\mathbf{e}}_{\left(S_1\right)}^{\left(∊\right)}}\cdots X_{{\mathbf{e}}_{\left(m\right)}^{\left(∊\right)}}{U}_{m+1}∕\Vert X_{{\mathbf{e}}_{\left(S_1\right)}^{\left(∊\right)}}\cdots X_{{\mathbf{e}}_{\left(m\right)}^{\left(∊\right)}}{U}_{m+1}\Vert \hfill \end{array}$$

Thus we may write where

$$\begin{array}{cc}\hfill \mid \gamma (∊;T_1,T_2)-g(m,∊;u;T_1,T_2)\mid & \le \mid \mathbb{E}\left[\log \frac{\Vert X_{\mathbf{e}\left(0\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{{\mathbf{e}}^{\left(∊\right)}\left(T_1\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}\left(m\right)}{\breve{U}}^{\left(∊\right)}\Vert }{\Vert X_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{{\mathbf{e}}^{\left(∊\right)}\left(T_1\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}\left(m\right)}{U}^{\left(∊\right)}\Vert }\mid T_1,T_2\right]-\mathbb{E}\left[\log \frac{\Vert X_{\mathbf{e}\left(0\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{{\mathbf{e}}^{\left(∊\right)}\left(T_1\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}\left(m\right)}u\Vert }{\Vert X_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{{\mathbf{e}}^{\left(∊\right)}\left(T_1\right)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}\left(m\right)}u\Vert }\mid T_1,T_2\right]\mid +\mid \mathbb{E}\left[\log \frac{\Vert X_{\mathbf{e}\left(0\right)}{X}_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}\left(m\right)}\breve{U}\Vert }{\Vert X_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}\left(m\right)}\breve{U}\Vert }-\log \frac{\Vert X_{\mathbf{e}\left(0\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{\mathbf{e}\left(T_1\right)}\cdots X_{\mathbf{e}\left(m\right)}u\Vert }{\Vert X_{\mathbf{e}\left(1\right)}\cdots X_{\mathbf{e}(T_1-1)}{X}_{\mathbf{e}\left(T_1\right)}\cdots X_{\mathbf{e}\left(m\right)}u\Vert }\mid T_1,T_2\right]\mid \hfill \\ \hfill & \le 2k_2{r}^m,\hfill \end{array}$$

where ${\breve{U}}^{\left(∊\right)}=\breve{U}=U_{m+1}$ if S₁ ≤ m; otherwise

$${\breve{U}}^{\left(∊\right)}=\frac{X_{{\mathbf{e}}^{\left(∊\right)}(m+1)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}(S_1-1)}{U}_{S_1}}{\Vert X_{{\mathbf{e}}^{\left(∊\right)}(m+1)}\cdots X_{{\mathbf{e}}^{\left(∊\right)}(S_1-1)}{U}_{S_1}\Vert };$$

$$\breve{U}=\frac{X_{(m+1)}\cdots X_{(S_1-1)}{U}_{S_1}}{\Vert X_{(m+1)}\cdots X_{(S_1-1)}{U}_{S_1}\Vert }.$$

On the event {T₂ ≤ m}: The above approach shows that

$$\mid \gamma (∊;T_1,T_2)-g(m,∊;u;T_1,T_2)\mid \le 2k_2{r}^{T_2-1}.$$ (57)

Combining these bounds, we obtain

$$\mid \gamma (∊;T_1,T_2)-g(m,∊;u;T_1,T_2)\mid \le k_2{r}^{T_1-1}{1}_{\{T_1>m\}}+2k_2{r}^m{1}_{\{T_1\le m\}}+2k_2{r}^{T_2-1}.$$ (58)

Taking the expectation with respect to the distribution of T₁ and T₂, using the fact that T₁ and T₂ − S₁ are independent with distribution geometric with parameter δ, we obtain

$$\mathbb{E}\left[\mid \gamma (∊;T_1,T_2)-g(m,∊;u;T_1,T_2)\mid \right]\le \frac{k_2}{1-r}{r}^{m-1}\delta +2k_2\delta m{r}^m+\frac{2k_2}{r^2{(1-r)}^2}{\delta }^2.$$ (59)

Since δ is bounded by a constant times |∊|, we may find a constant C such that (by the triangle inequality) for all ∊, positive integers m, and $u\in \mathcal{U}$,

$$\mid \mathbb{E}\left[\gamma (∊;T_1,T_2)\right]-\mathbb{E}\left[g(m,∊;u;T_1,T_2)\right]\mid \le \mathbb{E}\left[\mid \gamma (∊;T_1,T_2)-g(m,∊;u;T_1,T_2)\mid \right]\le C(m{r}^m\mid ∊\mid +{∊}^2).$$ (60)

This bound allows us to exchange the limits in (55):

$$\begin{array}{cc}\hfill a\prime \left(0\right)& =\underset{∊\to 0}{\lim }\underset{m\to \infty }{\lim }{∊}^{-1}\mathbb{E}\left[g(m,∊;u;T_1,T_2)\right]\hfill \\ \hfill & =\underset{∊\to 0}{\lim }{∊}^{-1}\mathbb{E}\left[\gamma (∊;T_1,T_2)\right]\hfill \\ \hfill & =\underset{m\to \infty }{\lim }\underset{∊\to 0}{\lim }{∊}^{-1}\mathbb{E}\left[g(m,∊;u;T_1,T_2)\right]\hfill \\ \hfill & =\underset{m\to \infty }{\lim }\frac{d}{d∊}{\mid }_{∊=0}{\mathbb{E}}_{P\left(∊\right)}\left[\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]\hfill \end{array}$$ (61)

Now we apply the method of importance sampling. We may assume without loss of generality that W (e, e′) = 0 whenever P (e, e′) = 0 (using the analyticity of a, and the fact that the formula (40) is nonsingular on the nonnegative orthant). For any function $Z:{\mathcal{M}}^{m+1}\to \mathbb{R}$,

$${\mathbb{E}}_{P\left(∊\right)}\left[Z({\mathbf{e}}_0,\dots ,{\mathbf{e}}_m)\right]={\mathbb{E}}_P\left[Z({\mathbf{e}}_0,\dots ,{\mathbf{e}}_m)F(∊;{\mathbf{e}}_0,\dots ,{\mathbf{e}}_m)\right],$$

where F is the Radon-Nikodym derivative

$$\begin{array}{cc}\hfill F(∊;{\mathbf{e}}_0,\dots ,{\mathbf{e}}_m)& =\frac{d{P}^{\left(∊\right)}}{dP}({\mathbf{e}}_0,\dots ,{\mathbf{e}}_m)\hfill \\ \hfill & =\frac{{\nu }_{{\mathbf{e}}_0}^{\left(∊\right)}}{{\nu }_{{\mathbf{e}}_0}}\prod _{i=0}^{m-1}\frac{P^{\left(∊\right)}({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}{P({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}.\hfill \end{array}$$

This allows us to rewrite

$$a\prime \left(0\right)=\underset{m\to \infty }{\lim }\frac{d}{d∊}\mid {}_{∊=0}\phantom{\mid }{\mathbb{E}}_P\left[\frac{{\nu }_{{\mathbf{e}}_0}^{\left(∊\right)}}{{\nu }_{e_0}}\prod _{i=0}^{m-1}\frac{P^{\left(∊\right)}({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}{P({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]$$ (62)

For any fixed m, there is an upper bound on ∊⁻¹(F(∊ e₀, …, e_m) −1), so we may move the differentiation inside the expectation, to obtain

$$\begin{array}{cc}\hfill a\prime \left(0\right)& =\underset{m\to \infty }{\lim }{\mathbb{E}}_P\left[\frac{d}{d∊}{\mid }_{∊=0}\frac{{\nu }_{{\mathbf{e}}_0}^{\left(∊\right)}}{{\nu }_{\mathbf{e}0}}\prod _{i=0}^{m-1}\frac{P^{\left(∊\right)}({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}{P({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]\hfill \\ \hfill & =\underset{m\to \infty }{\lim }{\mathbb{E}}_P\left[{\left({\nu }_{{\mathbf{e}}_0}\right)}^{-1}\frac{d{\nu }_{{\mathbf{e}}_0}^{\left(∊\right)}}{d∊}{\mid }_{∊=0}\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]+\underset{m\to \infty }{\lim }\sum _{i=0}^{m-1}{\mathbb{E}}_P\left[\frac{W({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}{P({\mathbf{e}}_i,{\mathbf{e}}_{i+1})}\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]\hfill \end{array}$$ (63)

The first limit is 0. To see this, rewrite it as a sum over possible values of e₀:

$$\underset{m\to \infty }{\lim }\sum _{e\in \mathcal{M}}{\nu }_e{\mathbb{E}}_{P_e}\left[{\left({\nu }_e\right)}^{-1}\frac{d{\nu }_e^{\left(∊\right)}}{d∊}{\mid }_{∊=0}\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{e}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{e}u\Vert }\right]=\underset{m\to \infty }{\lim }\sum _{e\in \mathcal{M}}\frac{d{\nu }_e^{\left(∊\right)}}{d∊}{\mathbb{E}}_{P_e}\left[\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_1}{X}_{e}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_1}{X}_{e}u\Vert }\right]$$

Since ν^(∊) is a probability distribution, it must be that ${\sum }_{\tilde{e}=1}^{\mathcal{M}}\frac{d{\nu }_{\tilde{e}}^{\left(∊\right)}}{d∊}=0$. Thus, the expression in the limit becomes 0 if we replace the expectation by a constant, independent of e. By Lemma A.2 it follows that the limit is 0.

To compute the other limit, we sum over all possible pairs (e_i, e_i+1) = $(\tilde{e},e)$. The summand becomes

$$\sum _{\tilde{e},e\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e)\mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}_m}{X}_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_{m-1}}\cdots X_{{\mathbf{e}}_0}u\Vert }\mid {\mathbf{e}}_i=\tilde{e},{\mathbf{e}}_{i+1}=e\right]$$ (64)

In order to analyze this, we need to consider the distribution of e₀, …, e_m, conditioned on ${\mathbf{e}}_i=\tilde{e}$ and e_i+1 = e. By the Markov property, this splits into two independent Markov chains: e = e_i+1, …, e_m is a Markov chain of length m − i, with transition probabilities P and starting point e, while $\tilde{e}={\mathbf{e}}_i$, e_i−1, …, e₀ is a Markov chain of length i + 1 with transition probabilities $\tilde{P}$ and starting point $\tilde{e}$. Define two independent infinite sequences ${\tilde{\mathbf{e}}}_0$, ${\tilde{\mathbf{e}}}_1$, … and e₀e₁, …, which are Markov chains with transitions $\tilde{P}$ and P respectively, beginning in ${\tilde{\mathbf{e}}}_0=\tilde{e}$ and e_i+1 = e. Define for $i\ge 1,{\check{U}}_i\left(\tilde{e}\right)≔X_{\tilde{e}}{X}_{{\tilde{\mathbf{e}}}_1}\cdots X_{{\tilde{\mathbf{e}}}_i}u$ with ${\check{U}}_0\left(\tilde{e}\right)≔1$, and ${\check{V}}_i^T\left(e\right)≔1^T{X}_{{\mathbf{e}}_{i-1}}\cdots X_{{\mathbf{e}}_1}{X}_e$ with ${\check{V}}_0^T\left(e\right)≔1^T$. Also define

$$U_i\left(e\right)≔\frac{{\check{U}}_i\left(e\right)}{\Vert {\check{U}}_i\left(e\right)\Vert },V_i^T\left(e\right)≔\frac{{\check{V}}_i^T\left(e\right)}{\Vert {\check{V}}_i^T\left(e\right)\Vert }.$$

Since ∥u∥ = 1^T u for any nonnegative column vector u, the expression (63) becomes

$$\begin{array}{cc}\hfill a\prime \left(0\right)& =\sum _{\tilde{e},e\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e)\underset{m\to \infty }{\lim }\left\{\sum _{i=0}^{m-1}\left(\mathbb{E}\left[\log {\check{V}}_{i+1}^T\left(e\right){\check{U}}_{m-1}\left(\tilde{e}\right)\right]-\mathbb{E}\left[\log {\check{V}}_i^T\left(e\right){\check{U}}_{m-1}\left(\tilde{e}\right)\right]\right)\right\}\hfill \\ \hfill & =\sum _{\tilde{e},e\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e)\underset{m\to \infty }{\lim }\left\{\sum _{i=0}^{m-1}\left(\mathbb{E}\left[\log \Vert {\check{V}}_{i+1}^T\left(e\right)\Vert \right]-\mathbb{E}\left[\log \Vert {\check{V}}_i^T\left(e\right)\Vert \right]+\mathbb{E}\left[\log V_{i+1}^T\left(e\right)U_i\left(\tilde{e}\right)\right]-\mathbb{E}\left[\log V_i^T\left(e\right)U_i\left(\tilde{e}\right)\right]\right)\right\}\hfill \\ \hfill & =\sum _{\tilde{e}\in \mathcal{M}}\nu \left(\tilde{e}\right)\underset{m\to \infty }{\lim }\sum _{e\in \mathcal{M}}W(\tilde{e},e)\mathbb{E}\left[\log \Vert {\check{V}}_m^T\left(e\right)\Vert \right]+\sum _{e,\tilde{e}\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e)\underset{m\to \infty }{\lim }\sum _{i=0}^{m-1}\mathbb{E}\left[\log \frac{V_{i+1}^T\left(e\right)U_{m-1}\left(\tilde{e}\right)}{V_i^T\left(e\right)U_{m-1}\left(\tilde{e}\right)}\right]\hfill \end{array}$$ (65)

In the last line we have used the fact that ${\Sigma }_{e\in \mathcal{M}}W(\tilde{e},e)=0$, which means that ${\Sigma }_{e\in \mathcal{M}}W(\tilde{e},e)\mathbb{E}\left[\text{log}\parallel {\check{V}}_0^T\left(e\right)\parallel \right]=0$ as well, since ${\check{V}}_0^T\left(e\right)=1^T$ is independent of e. The same reasoning implies that if we define ${\check{V}}_i^T\left(\nu \right)$ to be the version of ${\check{V}}_i^T$ started in the stationary distribution — for instance, starting from realizations of ${\check{V}}_i^T\left(e\right)$, define ${\check{V}}_i^T\left(\nu \right)$ to be equal to ${\check{V}}_i^T\left(e\right)$ with probability ν_e — then ${\Sigma }_{e\in \mathcal{M}}W(\tilde{e},e)\mathbb{E}\left[\text{log}\parallel {\check{V}}_m^T\left(\nu \right)\parallel \right]=0$. The first term on the right-hand side of (65) may then be written as

$$\sum _{\tilde{e},e\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e)\underset{m\to \infty }{\lim }\mathbb{E}\left[\log \frac{\Vert {\check{V}}_m^T\left(e\right)\Vert }{\Vert {\check{V}}_m^T\left(\nu \right)\Vert }\right]=\sum _{\tilde{e},e\in \mathcal{M}}\nu \left(\tilde{e}\right)W(\tilde{e},e){\zeta }_e.$$ (66)

To compute the second term, we note that $U_{\infty }\left(\tilde{e}\right)≔{\text{lim}}_{i\to \infty }{U}_i\left(\tilde{e}\right)$ exists, with distribution ${\pi }_{\tilde{e}}$, and $\rho (U_i\left(\tilde{e}\right),U_{\infty }\left(\tilde{e}\right))\le k_2{r}^i$; similarly, $V_{\infty }^T\left(e\right)≔{\text{lim}}_{i\to \infty }{V}_i^T\left(e\right)$ exists, with distribution ${\tilde{\pi }}_e$, and $\rho (V_i\left(e\right),V_{i+1}\left(e\right))\le k_2{r}^i$. Thus

$$\mid \log V_{i+1}^T\left(e\right)U_{m-1}\left(\tilde{e}\right)-\log V_i^T\left(e\right)U_{m-1}\left(\tilde{e}\right)\mid \le \rho (V_i\left(e\right),V_{i+1}\left(e\right))\le k_2{r}^i,$$

We break up the sum on the right-hand side of (65) into three pieces:

$$\sum _{0\le i\le m-1}\mathbb{E}[\log V_{i+1}^T\left(e\right)U_{m-1}\left(\tilde{e}\right)-\log V_i^T\left(e\right)U_{m-1}\left(\tilde{e}\right)]=\sum _{0\le i\le m∕2}\mathbb{E}[\log V_{i+1}^T\left(e\right)U_{\infty }\left(\tilde{e}\right)-\log V_i^T\left(e\right)U_{\infty }\left(\tilde{e}\right)]+\sum _{0\le i\le m∕2}\mathbb{E}\left[\log \frac{V_{i+1}^T\left(e\right)U_{m-1}\left(\tilde{e}\right)}{V_{i+1}^T\left(e\right)U_{\infty }\left(\tilde{e}\right)}-\log \frac{V_i^T\left(e\right)U_{m-1}\left(\tilde{e}\right)}{V_i^T\left(e\right)U_{\infty }\left(\tilde{e}\right)}\right]+\sum _{m∕2\le i\le m-1}\mathbb{E}[\log V_{i+1}^T\left(e\right)U_{m-i}\left(\tilde{e}\right)-\log V_i^T\left(e\right)U_{m-i}\left(\tilde{e}\right)].$$

The first sum telescopes to

$$\mathbb{E}[\log V_{1+m∕2}^T\left(e\right)U\left(\tilde{e}\right)-\log V_0^T\left(e\right)U\left(\tilde{e}\right)]=\mathbb{E}\left[\log V_{1+m∕2}^T\left(e\right)U\left(\tilde{e}\right)\right]-\log K,$$

applying the fact that $V_0^T=1^T∕K$, so that $V_0^T\left(e\right)U\left(\tilde{e}\right)=\parallel U\left(\tilde{e}\right)\parallel ∕K-1∕K$. Applying (9), the second and third sums are bounded by

$$\sum _{0\le i\le m∕2}2k_2{r}^{m-1}+\sum _{m∕2<i\le m-1}{k}_2{r}^i\le \frac{3k_2{r}^{m∕2}}{1-r}.$$

Thus

$$\underset{m\to \infty }{\lim }\sum _{i=0}^{m-1}\mathbb{E}\left[\log \frac{V_{i+1}^T\left(e\right)U_{m-1}\left(\tilde{e}\right)}{V_i^T\left(e\right)U_{m-i}\left(\tilde{e}\right)}\right]=\mathbb{E}\left[V^T\left(e\right)U\left(\tilde{e}\right)\right],$$ (67)

completing the proof of Theorem 7.1.

Lemma A.2. For any u, $\breve{u}\in \mathcal{U}$ and e, $e^{\ast }\in \mathcal{M}$, if we let e₀, e₁, … and ${\mathbf{e}}_0^{\ast },{\mathbf{e}}_1^{\ast }$, … be realisations of the Markov chain P starting at e₀ = e and ${\mathbf{e}}_0^{\ast }=e^{\ast }$ respectively. Then

$$\mid \mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}_{m+1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]-\mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}_{m+1}^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }{\Vert X_{{\mathbf{e}}_m^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }\right]\mid \le \frac{k_{2}D}{1-r}(m+1){(\xi \vee r)}^m,$$ (68)

where ξ and D are the constants that satisfy (38).

Proof. Using the maximal coupling, we create coupled versions of (e_i, ${\mathbf{e}}_i^{\ast }$), such that the coupling time τ satisfies

$$\mathbb{P}\{\tau \ge t\}\le \Vert \nu -P^t(e,\cdot )\Vert +\Vert \nu -P^t(\mathbf{e}\ast ,\cdot )\Vert \le 2D{\xi }^t.$$

Define

$$u_{\tau }≔\frac{X_{{\mathbf{e}}_{\tau -1}}\cdots X_{{\mathbf{e}}_0}u}{\Vert X_{{\mathbf{e}}_{\tau -1}}\cdots X_{{\mathbf{e}}_0}u\Vert },{\breve{u}}_{\tau }≔\frac{X_{{\mathbf{e}}_{\tau -1}^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}}{\Vert X_{{\mathbf{e}}_{\tau -1}^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }.$$

Then by the bound (13),

$$\begin{array}{cc}\hfill \mid \mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}_{m+1}}\cdots X_{{\mathbf{e}}_0}u\Vert }{\Vert X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_0}u\Vert }\right]-\mathbb{E}\left[\log \frac{\Vert X_{{\mathbf{e}}_{m+1}^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }{\Vert X_{{\mathbf{e}}_m^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }\right]\mid & \le \mathbb{E}\left[\mid \log \frac{\Vert X_{{\mathbf{e}}_{m+1}}\cdots X_{{\mathbf{e}}_0}u}{X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_0}u\Vert }-\log \frac{\Vert X_{{\mathbf{e}}_{m+1}^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }{\Vert X_{{\mathbf{e}}_m^{\ast }}\cdots X_{{\mathbf{e}}_0^{\ast }}\breve{u}\Vert }\mid \right]\hfill \\ \hfill & =\mathbb{E}\left[\mid \log \frac{\Vert X_{{\mathbf{e}}_{m+1}}\cdots X_{{\mathbf{e}}_{\tau }}{u}_{\tau }\Vert }{\Vert X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_0}u\Vert }-\log \frac{\Vert X_{{\mathbf{e}}_{m+1}}\cdots X_{{\mathbf{e}}_{\tau }}\breve{u}\Vert }{\Vert X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_{\tau }}{\breve{u}}_{\tau }\Vert }\mid \right]\hfill \\ \hfill & \le \mathbb{E}\left[k_2{r}^{m-\tau }\right]\hfill \\ \hfill & \le \frac{k_{2}D}{1-r}\sum _{t=0}^m{r}^{m-t}{\xi }^t\hfill \\ \hfill & \le \frac{k_{2}D}{1-r}(m+1){(\xi \vee r)}^m.\hfill \end{array}$$

A.5 A version of the Furstenberg-Kesten Theorem

Theorem A.3 (Furstenberg-Kesten) The limit defined in (2) exists almost surely, and is deterministic, given by (46).

Proof. Let e₁, e₂,… be a stationary Markov chain with transition matrix P. If we choose $N_0\in \mathcal{U}$, this induces a Markov chain Y_t := (N_t–1/∥N_t–1∥, e_t) with state space $\mathcal{U}\times \mathcal{M}$. We extend the Hilbert metric to $\mathcal{U}\times \mathcal{M}$ by ρ((u, e), (u_*, e_*)) = ρ(u, u_*) + 1_{e≠e*}.

If we define the function F : $\mathcal{U}\times \mathcal{M}\to \mathbb{R}$ by

$$F(u,m)≔\log \Vert X_{m}u\Vert ,$$

then

$$\begin{array}{cc}\hfill \log \Vert N_t\Vert & =\sum _{i=1}^t\log \frac{\Vert N_i\Vert }{\Vert N_{i-1}\Vert }\hfill \\ \hfill & =\sum _{i=1}^t\log \Vert X_{{\mathbf{e}}_i}\frac{N_{i-1}}{\Vert N_{i-1}\Vert }\Vert \hfill \\ \hfill & =\sum _{i=1}^{t}F\left(Y_i\right).\hfill \end{array}$$

If (Y_i) were a chain on a finite space, we would invoke the law of large numbers for Markov chains, also known as the pathwise Individual Ergodic Theorem (cf. Theorem 1.10.2 of Norris (1998)) to see that

$$\underset{t\to \infty }{\lim }{t}^{-1}\log \Vert N_t\Vert =\underset{t\to \infty }{\lim }{t}^{-1}\sum _{i=1}^{t}F\left(Y_i\right)\mathrm{exists}\mathrm{and}\mathrm{equals}\pi \left[F\right]≔\sum {\pi }_{y}F\left(y\right),$$

where π is the unique stationary distribution. The same would be true for a positive recurrent Markov chain on a countable state space.

On a more general state space, the pathwise ergodic theorem still holds as long as the Markov chain is uniquely ergodic; that is, it has a unique stationary distribution π (see Theorem 6.1 of Hernández-Lerma and Lasserre (1998), or Chapter 6 of Walters (1982)), and π is ergodic.

We first show the existence of π. Choose $u_0\in \mathcal{U}$, and let …, e₋₁, e₀, e₁, e₂, … be a stationary Markov chain on $\mathcal{M}$ with transitions P, infinite in both directions (as in section 3.2). Then

$$\rho (U_{-1,-n},U_{-1,-n-1})=\rho (X_{{\mathbf{e}}_{-1}}\cdots X_{{\mathbf{e}}_{-n}}{u}_0,X_{{\mathbf{e}}_{-1}}\cdots X_{{\mathbf{e}}_{-n}}{X}_{{\mathbf{e}}_{-n-1}}{u}_0)\le k_2{r}^n,$$

implying that ${\left(U_{-1,-n}\right)}_{n=1}^{\infty }$ is always a Cauchy sequence, hence converges to a random variable $U_{-1,-\infty }\in \mathcal{U}$. Define π to be the distribution of the pair (U_−1,−,∞, e₀).

Now, conditioned on starting at Y₀ = (U_−1,−∞, e₀), the next step Y₁ is (X_e0U_−1,−∞/∥X_e0U_−1,−∞∥ e₁). Notice that

$$\frac{X_{e_0}{U}_{-1,-\infty }}{\Vert X_{e_0}{U}_{-1,-\infty }\Vert }=U_{0,-\infty }$$

has the same distribution as U_−1,−∞, so that π is a stationary distribution for (Y_t).

It remains to show that π is uniquely ergodic. Suppose that we start the chain in an alternative distribution μ. Define Y₀ = (U, e₁) to have distribution π and $Y_0^{\ast }=(U^{\ast },{\mathbf{e}}_1^{\ast })$ to have distribution μ. Let e₁, e₂, … and ${\mathbf{e}}_1^{\ast },{\mathbf{e}}_2^{\ast }$ … be coupled versions of the environment chain, started in e₁ and ${\mathbf{e}}_1^{\ast }$ respectively. For definiteness, we say that they are independent up to the first time τ when ${\mathbf{e}}_{\tau }={\mathbf{e}}_{\tau }^{\ast }$ — and then ${\mathbf{e}}_{\tau +i}={\mathbf{e}}_{\tau +i}^{\ast }$ for i ≥ 0. Then

$$Y_m≔\left(\frac{X_{e_m}\cdots X_{{\mathbf{e}}_1}U}{\Vert X_{{\mathbf{e}}_m}\cdots X_{{\mathbf{e}}_1}U\Vert },{\mathbf{e}}_{m+1}\right)$$

$$Y_m^{\ast }≔\left(\frac{X_{e_m^{\ast }}\cdots X_{{\mathbf{e}}_1^{\ast }}U}{\Vert X_{{\mathbf{e}}_m^{\ast }}\cdots X_{{\mathbf{e}}_1^{\ast }}U\ast \Vert },{\mathbf{e}}_{m+1}^{\ast }\right)$$

are both realisations of the Markov chain Y, with initial distribution of π and μ respectively. We write μ_m for the distribution of $Y_m^{\ast }$. We see that

$$\rho (Y_m,Y_m^{\ast })\le k_2{r}^{m-\tau }{1}_{r>m}{\to }_{P}0\mathrm{as}m\to \infty .$$

Thus, μ_m → π weakly. By Theorem 6.12 of Walters (1982), it follows that π is ergodic (in fact, strong mixing). Furthermore, if μ were an invariant distribution then μ_m = μ, so the convergence implies μ = π. Therefore, π is uniquely ergodic.