SHARP ENTRYWISE PERTURBATION BOUNDS FOR MARKOV CHAINS

Abstract

For many Markov chains of practical interest, the invariant distribution is extremely sensitive to perturbations of some entries of the transition matrix, but insensitive to others; we give an example of such a chain, motivated by a problem in computational statistical physics. We have derived perturbation bounds on the relative error of the invariant distribution that reveal these variations in sensitivity.

Our bounds are sharp, we do not impose any structural assumptions on the transition matrix or on the perturbation, and computing the bounds has the same complexity as computing the invariant distribution or computing other bounds in the literature. Moreover, our bounds have a simple interpretation in terms of hitting times, which can be used to draw intuitive but rigorous conclusions about the sensitivity of a chain to various types of perturbations.

1. Introduction

The invariant distribution of a Markov chain is often extremely sensitive to perturbations of some entries of the transition matrix, but insensitive to others. However, most perturbation estimates bound the error in the invariant distribution by a single condition number times a matrix norm of the perturbation. That is, perturbation estimates usually take the form

$$\mid \pi \left(\tilde{F}\right)-\pi \left(F\right)\mid \le \kappa \left(F\right)\Vert \tilde{F}-F\Vert ,$$ (1)

where F and F̃ are the exact and perturbed transition matrices, π(F) and π(F̃) are the invariant distributions of these matrices, ∥·∥ is a matrix norm, |·| is either a vector norm or some measure of the relative error between the two distributions, and κ(F) is a condition number depending on F. For example, see [2,5–7,9,13,16,19–21] and the survey given in [3]. No bound of this form can capture wide variations in the sensitivities of different entries of the transition matrix.

Alternatively, one might approximate the difference between π and $\tilde{\pi }$ using the linearization of π at F. (The derivative of π can be computed efficiently using the techniques of [6].) The linearization will reveal variations in sensitivities, but only yields an approximation of the form

$$\pi \left(\tilde{F}\right)-\pi \left(F\right)={\pi }^{\prime }\left(F\right)\left[\tilde{F}-F\right]+o\left(\Vert \tilde{F}-F\Vert \right),$$

not an upper bound on the error. That is, unless global bounds on π′(F) can be derived, linearization provides only a local, not a global estimate.

In this article, we give upper bounds that yield detailed information about the sensitivity of π(F) to perturbations of individual entries of F. Given an irreducible substochastic matrix S, we show that for all stochastic matrices F, F̃ satisfying the entry-wise bound F, F̃ ≥ S,

$$\underset{i}{\max }\mid \log {\pi }_i\left(\tilde{F}\right)-\log {\pi }_i\left(F\right)\mid \le \sum _{i\ne j}\mid \log \left({\tilde{F}}_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij}\right)-\log (F_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij})\mid ,$$ (2)

where Q_ij(S) is defined in Section 4. As a corollary, we also have

$$\underset{i}{\max }\mid \log {\pi }_i\left(\tilde{F}\right)-\log {\pi }_i\left(F\right)\mid \le \sum _{i\ne j}{\mathbf{Q}}_{ij}{\left(S\right)}^{-1}\mid {\tilde{F}}_{ij}-F_{ij}\mid$$ (3)

for all stochastic F, F̃ ≥ S.

The difference in logarithms on the left hand sides of (2) and (3) measures relative error. Usually, when x̃ ∈ (0, ∞) is computed as an approximation to x ∈ (0, ∞), the error of x̃ relative to x is defined to be either

$$\mid \frac{\tilde{x}-x}{x}\mid \text{or}\max \left\{\frac{\tilde{x}}{x},\frac{x}{\tilde{x}}\right\}.$$ (4)

Instead, we define the relative error to be |log x̃ – log x|. Our definition is closely related to the other two: it is the logarithm of the second definition in (4); and by Taylor expansion,

$$\mid \log \tilde{x}-\log x\mid =\mid \frac{\tilde{x}-x}{x}\mid +O\left({\mid \frac{\tilde{x}-x}{x}\mid }^2\right),$$

so it is equivalent with the first in the limit of small error. We chose our definition because it allows for simple arguments based on logarithmic derivatives of π(F).

We call the coefficient Q_ij(S)^–1 in (3) the sensitivity of the ij^th entry. Q_ij(S) has a simple probabilistic interpretation in terms of hitting times, which can sometimes be used to draw intuitive conclusions about the sensitivities. In Theorem 4, we show that our coefficients Q_ij(S)^–1 are within a factor of two of the smallest possible so that a bound of form (3) holds. Thus, our bound is sharp. (We note that our definition of sharp differs slightly from other standard definitions; see Remark 5.) In Theorem 5, we give an algorithm by which the sensitivities may be computed in O(L³) time for L the number of states in the chain. Therefore, computing the error bound has the same order of complexity as computing the invariant measure or computing most other perturbation bounds in the literature; see Remark 8.

Since our result takes an unusual form, we now give three examples to illustrate its use. We discuss the examples only briefly here; details follow in Sections 4 and 6. First, suppose that F̃ has been computed as an approximation to an unknown stochastic matrix F and that we have a bound on the error between F̃ and F, for example |F̃_ij – F_ij| ≤ α_ij. In this case, we define S_ij := max{F̃_ij – α_ij, 0}, and we have estimate

$$\underset{i}{\max }\mid \log {\pi }_i\left(\tilde{F}\right)-\log {\pi }_i\left(F\right)\mid \le \sum _{i\ne j}{\mathbf{Q}}_{ij}{\left(S\right)}^{-1}\mid {\tilde{F}}_{ij}-F_{ij}\mid$$

for all F so that |F̃_ij – F_ij| ≤ α_ij. See Remark 7 for a more detailed explanation.

Now suppose instead that F ≥ αP where 0 < α < 1 and P is the transition matrix of an especially simple Markov chain, for example a symmetric random walk. Then we choose S := αP, and we compute or approximate Q_ij(αP) by easy to understand probabilistic arguments. This method can be used to draw intuitive but rigorous conclusions about the sensitivity of a chain to various types of perturbations. See Section 6.5 for details.

Finally, suppose that the transition matrix F has a large number of very small positive entries and that we desire a sparse approximation F̃ to F with approximately the same invariant distribution. In this case, we take S to be F with all its small positive entries set to zero. If the sensitivity Q_ij(S)^–1 is very large, it is likely that the the value of F_ij is important and cannot be set to zero. If Q_ij(S)^–1 is small, then setting F̃_ij = 0 and F̃_ii = F_ii + F_ij will not have much effect on the invariant distribution.

We are aware of two other bounds on relative error in the literature. By [9, Theorem 4.1],

$$\mid \frac{{\pi }_i\left(\tilde{F}\right)-{\pi }_i\left(F\right)}{{\pi }_i\left(F\right)}\mid \le {\kappa }_i\left(F\right){\Vert \left(\tilde{F}-F\right)\left(I-e_i{e}_i^{\mathrm{t}}\right)\Vert }_{\infty },$$ (5)

where ${\kappa }_i\left(F\right)={\Vert {(I-F_i)}^{-1}\Vert }_{\infty }$ for F_i the j^th principal submatrix of F. This bound fails to identify the sensitive and insensitive entries of the transition matrix since the error in the i^th component of the invariant distribution is again controlled only by the single condition number κ_i(F). Moreover, computing κ_i(F) for all i is of the same complexity as computing all of our sensitivities Q_ij(S)^–1; see Remark 8. Therefore, in many respects, our result provides more detailed information at the same cost as [9, Theorem 4.1]. On the other hand, we observe that [9, Theorem 4.1] holds for all perturbations: one does not have to restrict the admissible perturbations by requiring F̃ ≥ S as we do for our result. However, this is not always an advantage, since we anticipate that in many applications bounds on the error in F are available and, as we will see, the benefit from using this information is significant.

In [18, Theorem 1], another bound on the relative error is given. Here, the relative error in the invariant distribution is bounded by the relative error in the transition matrix. Precisely, if F, $\tilde{F}\in {\mathbb{R}}^{L\times L}$ are irreducible stochastic matrices with F_ij = 0 if and only if F̃_ij = 0, then

$$\underset{m}{\max }\max \left\{\frac{{\tilde{\pi }}_m}{{\pi }_m},\frac{{\pi }_m}{{\tilde{\pi }}_m}\right\}\le {\left(\underset{i\ne j}{\max }\max \left\{\frac{{\tilde{F}}_{ij}}{F_{ij}},\frac{F_{ij}}{{\tilde{F}}_{ij}}\right\}\right)}^L$$ (6)

This surprising result requires no condition number depending on F, but it does require that F and F̃ have the same sparsity pattern, which greatly restricts the admissible perturbations. Our result may be understood as a generalization of (6) which allows perturbations changing the sparsity pattern.

Our result also bears some similarities with the analysis in [2, Section 4], which is based on the results of [9]. In [2], a state m of a Markov chain is said to be centrally located if E_i[τ_m] is small for all states i. (Here, τ_m is the first passage time to state m; see Section 2.) It is shown that if |E_i[τ_m] – E_j[τ_m]| is small, then π_m(F) is insensitive to F_ij in relative error. Therefore, if m is centrally located, π_m(F) is not sensitive to any entry of the transition matrix. Our ${\mathbf{Q}}_{ij}^{-1}$ can also be expressed in terms of first passage times, and they provide a better measure of the sensitivity of π_m(F) to F_ij than |E_i[τ_m] – E_j[τ_m]|; see Section 6.3.

Our bounds on derivatives of π(F) in Theorem 2 and our estimates (2) and (3) share some features with structured condition numbers [12, Section 5.3]. The structured condition number of an irreducible, stochastic matrix F is defined to be

$${\mathcal{C}}^{(p,q)}\left(F\right)≔\underset{\epsilon \to 0}{\lim }\sup \left\{{\epsilon }^{-1}{\Vert \pi (F+E)-\pi \left(F\right)\Vert }_p:F+E\text{stochastic and}{\Vert E\Vert }_q<\epsilon \right\}.$$

Structured condition numbers yield approximate bounds valid for small perturbations. These bounds are useful, since for small perturbations, estimates of type (1) are often far too pessimistic. We remark that our results (2) and (3) give the user control over the size of the perturbation through the choice of S. (If S is nearly stochastic, then only small perturbations are allowed.) Therefore, like structured condition numbers, our results are good for small perturbations. In addition, our results are true upper bounds, so they are more robust than approximations derived from structured condition numbers.

Our interest in perturbation bounds for Markov chains arose from a problem in computational statistical physics; we present a drastically simplified version below in Section 6. For this problem, the invariant distribution is extremely sensitive to some entries of the transition matrix, but insensitive to others. We use the problem to illustrate the differences between our result, [18, Theorem 1], [9, Theorem 4.1], and the eight bounds on absolute error surveyed in [3]. Each of the eight bounds has form (1), and we demonstrate that the condition number κ(F) in each bound blows up exponentially with the inverse temperature parameter in our problem. By contrast, many of the sensitivities ${\mathbf{Q}}_{ij}^{-1}$ from our result are bounded as the inverse temperature increases. Thus, our result gives a great deal more information about which perturbations can lead to large changes in the invariant distribution.

2. Notation

We fix $L\in \mathbb{N}$, and we let X be a discrete time Markov chain with state space Ω = {1, 2, . . . , L} and irreducible, row-stochastic transition matrix $F\in {\mathbb{R}}^{L\times L}$. Since F is irreducible, X has a unique invariant distribution $\pi \in {\mathbb{R}}^L$ satisfying

$${\pi }^{\mathrm{t}}F={\pi }^{\mathrm{t}},\sum _{i=1}^L{\pi }_i=1,\text{and}{\pi }_i>0\text{for all}i=1,\dots ,L.$$

We let e_i denote the i^th standard basis vector in ${\mathbb{R}}^L$, e denote the vector of ones, and I denote the identity matrix. We treat all vectors, including π, as column vectors (that is, as L × 1 matrices). For $S\in {\mathbb{R}}^{L\times L}$, we let $S_j:e_j^{\perp }\to e_j^{\perp }$ be the operator defined by

$$S_{j}x≔(I-e_j{e}_j^{\mathrm{t}})Sx.$$

Instead of defining S_j as above, we could define S_j to be S with the j^th row and column set to zero. We could also define S_j to be the j^th principal submatrix. We chose our definition to emphasize that we treat S_j as an operator on $e_j^{\perp }$. If S and T are matrices of the same dimensions, we say S ≥ T if and only if S_ij ≥ T_ij for all indices i, j. For any $v\in {\mathbb{R}}^L$ with v > 0, we define $\log \left(v\right)\in {\mathbb{R}}^L$ by (log v)_i = log(v_i).

For k ∈ {1, 2, . . . , L}, we define 1_k to be the indicator function of the set {k}, and

$${\tau }_k≔\min \{s>0:X_s=k\}$$

to be the first return time to state k. We also define

$${\mathbf{P}}_k\left[A\right]≔\mathbf{P}[A\mid X_0=k]\text{and}{\mathbf{E}}_k\left[Y\right]≔\mathbf{E}[Y\mid X_0=k]$$

to be the probability of the event A conditioned on X₀ = k and the expectation of the random variable Y conditioned on X₀ = k, respectively. Finally, for Y a random variable and B an event, we let

$$\mathbf{E}[Y,B]≔\mathbf{E}\left[Y{\chi }_B\right]=\mathbf{E}[Y\mid B]\mathbf{P}\left[B\right],$$

where χ_B is the indicator function of the event B.

3. Partial derivatives of the invariant distribution

Given an irreducible, stochastic matrix $F\in {\mathbb{R}}^{L\times L}$, let $\pi \left(F\right)\in {\mathbb{R}}^L$ be the invariant distribution of F; that is, let π(F) be the unique solution of

$$\pi {\left(F\right)}^{\mathrm{t}}F=\pi {\left(F\right)}^{\mathrm{t}}\text{and}\pi {\left(F\right)}^{\mathrm{t}}e=1.$$

We regard π as a function defined on the set of irreducible stochastic matrices, and in Lemma 1, we show that π is differentiable in a certain sense. We give a proof of the lemma in Appendix A.

Lemma 1

The function π admits a continuously differentiable extension $\stackrel{‒}{\pi }$ to an open neighborhood $\mathcal{V}$ of the set of irreducible stochastic matrices in ${\mathbb{R}}^{L\times L}$. The extension may be chosen so that $\stackrel{‒}{\pi }\left(G\right)>0$ for all $G\in \mathcal{V}$ and so that if Ge = e, then

$$\stackrel{‒}{\pi }{\left(G\right)}^{\mathrm{t}}G=\stackrel{‒}{\pi }{\left(G\right)}^{\mathrm{t}}and\stackrel{‒}{\pi }{\left(G\right)}^{\mathrm{t}}e=1.$$

Remark 1

The set of stochastic matrices is not a vector space; it is a compact, convex polytope lying in the affine space $\{G\in {\mathbb{R}}^{L\times L}:Ge=e\}\subset {\mathbb{R}}^{L\times L}$. As a consequence, we need the extension guaranteed by Lemma 1 to define the derivative of π on the boundary of the polytope, which is the set of all stochastic matrices with at least one zero entry. We introduce $\stackrel{‒}{\pi }$ only to resolve this unpleasant technicality, not to define π(F) for matrices which are not stochastic. In fact, all our results are independent of the particular choice of extension, as long as it meets the conditions in the second sentence of the lemma.

Our perturbation bounds are based on partial derivatives of π with respect to entries of F. As usual, the partial derivatives are defined in terms of a coordinate system, and we choose the off-diagonal entries of F as coordinates: Any stochastic F is determined by its off-diagonal entries through the formula

$$F=I+\sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}{F}_{ij}\left(e_i{e}_j^{\mathrm{t}}-e_i{e}_i^{\mathrm{t}}\right).$$

Accordingly, for i, j ∈ Ω with i ≠ j, we define

$$\begin{array}{cc}\hfill \frac{\partial {\pi }_m}{\partial F_{ij}}\left(F\right)& =\frac{\partial }{\partial F_{ij}}{\stackrel{‒}{\pi }}_m\left(I+\sum _{k\ne l}{F}_{kl}\left(e_k{e}_l^{\mathrm{t}}-e_k{e}_k^{\mathrm{t}}\right)\right)\hfill \\ \hfill & ={\phantom{\mid }\frac{d}{d\epsilon }\mid }_{\epsilon =0}{\stackrel{‒}{\pi }}_m\left(F+\epsilon \left(e_i{e}_j^{\mathrm{t}}-e_i{e}_i^{\mathrm{t}}\right)\right).\hfill \end{array}$$ (7)

These partial derivatives must be understood as derivatives of the extension $\stackrel{‒}{\pi }$ guaranteed by Lemma 1. Otherwise, if F_ij or F_ii were zero, the right hand side of (7) would be undefined.

Remark 2

We chose to define partial derivatives by (7), since that definition leads to the global bounds on the invariant distribution presented in Section 4. Other choices are reasonable: for example, one might consider derivatives of the form

$${\phantom{\mid }\frac{d}{d\epsilon }\mid }_{\epsilon =0}\pi \left(F+\epsilon \left(e_i{e}_j^{\mathrm{t}}-e_i{e}_k^{\mathrm{t}}\right)\right),$$

where k ≠ i and j ≠ i. However, to the best of our knowledge, only definition (7) leads easily to global bounds.

In Theorem 1, we derive a convenient formula for $\frac{\partial {\pi }_m}{\partial F_{ij}}$. Comparable results relating derivatives and perturbations of π to the matrix of mean first passage times were given in [2,8]. A formula for the derivative of the invariant distribution in terms of the group inverse of I – F was given in [15]; a general formula for the derivative of the Perron vector of a nonnegative matrix was given in [4].

Theorem 1

Let F be an irreducible stochastic matrix, and let X be a Markov chain with transition matrix F. Define π(F) and $\frac{\partial {\pi }_m}{\partial F_{ij}}$ as above. We have

$$\frac{\partial {\pi }_m}{\partial F_{ij}}\left(F\right)={\pi }_i\left({\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\pi }_m{\mathbf{E}}_j\left[{\tau }_i\right]\right)$$

for all i, j ∈ Ω with i ≠ j.

Proof

Define $\stackrel{‒}{\pi }$ and $\mathcal{V}$ as in Lemma 1, let $v\left(G\right)≔\stackrel{‒}{\pi }\left(G\right)∕{\stackrel{‒}{\pi }}_i\left(G\right)$ for all $G\in \mathcal{V}$, and let $G^{\epsilon }≔F+\epsilon (e_i{e}_j^{\mathrm{t}}-e_i{e}_i^{\mathrm{t}})$. Define

$$\frac{\partial v}{\partial F_{ij}}\left(F\right)≔\left(\frac{\partial v_i}{\partial F_{ij}}\left(F\right),\dots ,\frac{\partial v_L}{\partial F_{ij}}\left(F\right)\right)={\phantom{\mid }\frac{d}{d\epsilon }v\left(G^{\epsilon }\right)\mid }_{\epsilon =0}.$$

Since G^εe = e, Lemma 1 implies

$$v{\left(G^{\epsilon }\right)}^{\mathrm{t}}(I-G^{\epsilon })=0\text{and}{v}_i\left(G^{\epsilon }\right)=1$$ (8)

for all ε sufficiently close to zero.

We derive an equation for $\frac{\partial v}{\partial F_{ij}}\left(F\right)$ from (8); differentiating (8) with respect to ε gives

$$\frac{\partial v}{\partial F_{ij}}{\left(F\right)}^{\mathrm{t}}(I-F)=e_j^{\mathrm{t}}-e_i^{\mathrm{t}}\text{and}\frac{\partial v_i}{\partial F_{ij}}\left(F\right)=0.$$ (9)

Recalling the definition of F_i from Section 2, (9) implies

$$\frac{\partial v}{\partial F_{ij}}{\left(F\right)}^{\mathrm{t}}(I-F_i)=e_j^{\mathrm{t}}.$$

Moreover, by [1, Chapter 6, Theorem 4.16],

$$F\text{irreducible and stochastic or substochastic implies}{(I-F_i)}^{-1}=\sum _{s=0}^{\infty }{F}_i^s.$$ (10)

Therefore, we have

$$\frac{\partial v}{\partial F_{ij}}=e_j^{\mathrm{t}}{(I-F_i)}^{-1}=e_j^{\mathrm{t}}\sum _{k=0}^{\infty }{F}_i^k.$$ (11)

We now interpret (11) in terms of the Markov chain X_t with transition matrix F. We observe that for any m ∈ Ω \ {i},

$$e_j^{\mathrm{t}}{F}_i^k{e}_m={\mathbf{P}}_j[X_k=m,k<{\tau }_i],$$

where τ_i := min{t > 0 : X_t = i} is the first passage time to state i. Therefore, for m, j ∈ Ω \ {i}, (11) yields

$$\frac{\partial v_m}{\partial F_{ij}}=\sum _{k=0}^{\infty }{\mathbf{P}}_j[X_k=m,k<{\tau }_i]={\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right].$$ (12)

In fact, this formula also holds for m = i, since we have

$$\frac{\partial v_i}{\partial F_{ij}}={\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_i\left(X_s\right)\right]=0$$

for all j ∈ Ω \ {i}.

Finally, we convert our formula for $\frac{\partial v_m}{\partial F_{ij}}$ to a formula for $\frac{\partial {\pi }_m}{\partial F_{ij}}$. We have

$${\pi }_m=\frac{v_m}{{\sum }_{k=1}^L{v}_k},$$

and so by (12),

$$\begin{array}{cc}\hfill \frac{\partial {\pi }_m}{\partial F_{ij}}& =\frac{\frac{\partial v_m}{\partial F_{ij}}\left({\sum }_{k=1}^L{v}_k\right)-v_m{\sum }_{k=1}^L\frac{\partial v_k}{\partial F_{ij}}}{{\left({\sum }_{k=1}^L{v}_k\right)}^2}\hfill \\ \hfill & =\frac{{\mathbf{E}}_j\left[{\sum }_{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\pi }_m{\mathbf{E}}_j\left[{\tau }_i\right]}{{\sum }_k{v}_k}.\hfill \end{array}$$ (13)

Now by [17, Theorem 1.7.5], we have

$$v_k\left(F\right)=\frac{{\pi }_k\left(F\right)}{{\pi }_i\left(F\right)}={\mathbf{E}}_i\left[\sum _{s=0}^{{\tau }_i-1}{1}_k\left(X_s\right)\right]\text{and}\frac{1}{{\pi }_i\left(F\right)}={\mathbf{E}}_i\left[{\tau }_i\right].$$ (14)

Therefore, (13) implies

$$\frac{\partial {\pi }_m}{\partial F_{ij}}={\pi }_i\left({\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\pi }_m{\mathbf{E}}_j\left[{\tau }_i\right]\right).$$

Our goal is to bound the relative errors of the entries of the invariant measure π(F), where for x̃, x ∈ (0, ∞), we define the relative error between x̃ and x to be

$$\mid \log \tilde{x}-\log x\mid .$$ (15)

Our definition of relative error is unusual, but it is closely related to the common definitions, as we explain in the introduction. In Theorem 2, we derive sharp bounds on the logarithmic partial derivatives of the invariant distribution. We want bounds on logarithmic derivatives, since we will ultimately prove bounds on the relative error in π, with relative error defined by (15).

The following lemma will be used in the proof of Theorem 2.

Lemma 2

We have

$${\mathbf{P}}_i[{\tau }_j<{\tau }_i]{\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]={\mathbf{E}}_i\left[\sum _{s={\tau }_j}^{{\tau }_i-1}{1}_m\left(X_s\right)\right],$$

and

$${\mathbf{P}}_i[{\tau }_j<{\tau }_i]{\mathbf{E}}_j\left[{\tau }_i\right]={\mathbf{E}}_i[{\tau }_i-{\tau }_j,{\tau }_j<{\tau }_i].$$

Proof

For n ≥ 0, define Y_n := X_τj+n. For i ∈ Ω, let ${\tau }_i^X$ and ${\tau }_i^Y$ denote the first return times to i for X and Y, respectively. By the strong Markov property, (a) Y is a Markov process with the same transition matrix as X, (b) the distribution of Y₀ is $e_j^{\mathrm{t}}$, and (c) conditional on Y₀, Y_n is independent of X₀, X₁, . . . , X_τj. Therefore,

$$\begin{array}{cc}\hfill \mathbf{E}\left[\phantom{\mid }\sum _{s={\tau }_j^X}^{{\tau }_i^X-1}{1}_m\left(X_s\right)\mid X_0=i\right]& =\mathbf{E}\left[\phantom{\mid }\sum _{s={\tau }_j^X}^{{\tau }_i^X-1}{1}_m\left(X_s\right),{\tau }_j^X<{\tau }_i^X\mid X_0=i\right]\hfill \\ \hfill & =\mathbf{E}\left[\phantom{\mid }\sum _{s=0}^{{\tau }_i^Y-1}{1}_m\left(Y_s\right),{\tau }_j^X<{\tau }_i^X\mid X_0=i\right]\hfill \\ \hfill & =\mathbf{E}\left[\phantom{\mid }\sum _{s=0}^{{\tau }_i^Y-1}{1}_m\left(Y_s\right)\mid Y_0=j\right]{\mathbf{P}}_i\left[{\tau }_j^X<{\tau }_i^X\right].\hfill \end{array}$$

(The second equality above follows from the definition of Y; the third equality follows from the strong Markov property, since the event ${\tau }_j^X<{\tau }_i^X$ is determined by X₀, X₁, . . . , X_τj.) This proves the first formula in the statement of the lemma; the second formula follows on summing the first over all m.

Using Lemma 2, we now prove our bounds on the logarithmic derivatives.

Theorem 2

We have

$$\frac{1}{2}\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\le \underset{m}{\max }\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\mid \le \frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]},$$

and

$$\underset{m}{\max }\frac{\partial \log {\pi }_m}{\partial F_{ij}}-\underset{m}{\min }\frac{\partial \log {\pi }_m}{\partial F_{ij}}=\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}.$$

Proof

Using Theorem 1, Lemma 2, and (14), we have

$$\frac{\partial \log {\pi }_m}{\partial F_{ij}}=\frac{{\pi }_i}{{\pi }_m}{\mathbf{E}}_j\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right]$$ (16)

$$\begin{array}{cc}\hfill & =\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\frac{{\pi }_i}{{\pi }_m}{\mathbf{E}}_i\left[\sum _{s={\tau }_j}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right]\hfill \\ \hfill & =\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\frac{{\pi }_i}{{\pi }_m}\left({\mathbf{E}}_i\left[\sum _{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]-{\mathbf{E}}_i\left[\sum _{s=0}^{{\tau }_{j-1}}{1}_m\left(X_s\right),{\tau }_j<{\tau }_i\right]\right)-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right]\hfill \\ \hfill & =\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\left(1-\frac{{\pi }_i}{{\pi }_m}{\mathbf{E}}_i\left[\sum _{s=0}^{{\tau }_j-1}{1}_m\left(X_s\right),{\tau }_j<{\tau }_i\right]\right)-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right].\hfill \end{array}$$ (17)

We observe that the term $\frac{{\pi }_i}{{\pi }_m}{\mathbf{E}}_j\left[{\sum }_{s=0}^{{\tau }_i-1}{1}_m\left(X_s\right)\right]$ in formula (16) is nonnegative, hence that term attains a minimum value of zero when m = i. Therefore, we have

$$\underset{m}{\min }\frac{\partial \log {\pi }_m}{\partial F_{ij}}=-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right],$$ (18)

with the minimum attained when m = i, since the other term in formula (16) does not depend on m. By a similar argument using (17),

$$\underset{m}{\max }\frac{\partial \log {\pi }_m}{\partial F_{ij}}=\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}-{\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right],$$ (19)

and the maximum is attained when m = j.

Subtracting (18) from (19) gives

$$\underset{m}{\max }\frac{\partial \log {\pi }_m}{\partial F_{ij}}-\underset{m}{\min }\frac{\partial \log {\pi }_m}{\partial F_{ij}}=\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]},$$

hence

$$\frac{1}{2}\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\le \underset{m}{\max }\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\mid .$$

Finally, by Lemma 2 and (14),

$$0\le {\pi }_i{\mathbf{E}}_j\left[{\tau }_i\right]=\frac{{\pi }_i{\mathbf{E}}_i[{\tau }_i-{\tau }_j,{\tau }_j<{\tau }_i]}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\le \frac{{\pi }_i{\mathbf{E}}_i\left[{\tau }_i\right]}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}=\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]},$$

so

$$\underset{m}{\max }\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\mid \le \frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}.$$

Corollary 1 gives a simplified version of the bound in Theorem 2. This estimate can be used to derive [18, Theorem 1], which we have stated in equation (6) above. We omit the proof.

Corollary 1

Whenever F_ij ≠ 0,

$$\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\mid \le \frac{1}{F_{ij}}.$$

Proof

We have

$${\mathbf{P}}_i[{\tau }_j<{\tau }_i]\ge {\mathbf{P}}_i[X_1=j]=F_{ij},$$

and so the result follows by Theorem 2.

4. Global perturbation bounds

In this section, we use our bounds on the derivatives of the invariant distribution to prove global perturbation estimates. Our estimates assume that both the exact transition matrix F and the perturbed matrix F̃ are bounded below by some irreducible substochastic matrix S. As a consequence, coefficients Q_ij(S) depending on S arise.

We define Q_ij(S) in terms of a Markov chain with transition matrix depending on S, and our perturbation results are based on comparisons between this chain and other chains with transition matrices G ≥ S. Therefore, to avoid confusion, we let

$${\mathbf{P}}_i\left[A\right]\left(G\right)$$

denote the probability that X^G ∈ A conditioned on $X_0^G=i$ for X^G a chain with transition matrix G. To give a specific example, we intend P_i[τ_i < τ_j](G) to mean the probability that X^G hits j before returning to i, conditional on $X_0^G=i$.

We now define Q_ij(S).

Definition 1

For S an irreducible and substochastic or stochastic matrix, let X^ω be the Markov chain with state space $\Omega \cup \left\{\omega \right\}$ and transition matrix

$$S^{\omega }≔\begin{array}{c}\hfill \Omega \hfill \\ \hfill \omega \hfill \end{array}\left(\begin{array}{cc}\hfill \Omega \hfill & \hfill \omega \hfill \\ \hfill S\hfill & \hfill e-S_e\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right).$$

We think of X^ω as a chain with transition probability S, but augmented by an absorbing state ω to adjust for the fact that S is substochastic. For all i, j ∈ Ω with i ≠ j, we define

$${\mathbf{Q}}_{ij}\left(S\right)≔{\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right).$$

Remark 3

We observe that for F stochastic,

$${\mathbf{Q}}_{ij}\left(F\right)={\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(F^{\omega }\right)={\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(F\right),$$

since the absorbing state ω does not communicate with the other states Ω when F is stochastic.

We now show that Q_ij(S) is monotone as a function of S. This is the crucial step in deriving global perturbation bounds from the bounds on derivatives in Theorem 2.

Lemma 3

Let S be an irreducible substochastic matrix. If F is a stochastic or substochastic matrix with F ≥ S, then

$${\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(F^{\omega }\right)={\mathbf{Q}}_{ij}\left(F\right)\ge {\mathbf{Q}}_{ij}\left(S\right)>0.$$

In addition, for any substochastic or stochastic matrix S,

$${\mathbf{Q}}_{ij}\left(S\right)\ge S_{ij}.$$

Proof

Let ${\mathcal{P}}_{ij}^M$ be the set of all walks of length M in Ω which start at i, end at j, and visit i and j only at the endpoints. To be more precise, define

$${\mathcal{P}}_{ij}^M≔\{\gamma :\mathbb{Z}\cap [0,M]\to \Omega :\gamma \left(0\right)=i,\gamma \left(M\right)=j,\gamma \left(k\right)\notin \{i,j\}\forall 0<k<M\}.$$

We observe that since F ≥ S,

$$\begin{array}{cc}\hfill {\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(F^{\omega }\right)& =\sum _{M=1}^{\infty }\sum _{\gamma \in {\mathcal{P}}_{ij}^M}{\mathbf{P}}_i[X_k=\gamma \left(k\right)\text{for}k=1,\dots ,M]\left(F^{\omega }\right)\hfill \\ \hfill & \ge \sum _{M=1}^{\infty }\sum _{\gamma \in {\mathcal{P}}_{ij}^M}{\mathbf{P}}_i[X_k=\gamma \left(k\right)\text{for}k=1,\dots ,M]\left(S^{\omega }\right)\hfill \\ \hfill & ={\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)\hfill \\ \hfill & ={\mathbf{Q}}_{ij}\left(S\right).\hfill \end{array}$$

Since S is irreducible, we also have that

$${\mathbf{Q}}_{ij}\left(S\right)={\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)>0.$$

Finally,

$${\mathbf{Q}}_{ij}\left(S\right)={\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)\ge {\mathbf{P}}_i[X_1=j]\left(S^{\omega }\right)=S_{ij},$$

which concludes the proof.

Combining Theorem 2 with Lemma 3 yields our global perturbation estimate.

Theorem 3

Let F, F̃ be stochastic matrices, let S be substochastic and irreducible, and assume that F, F̃ ≥ S. We have

$$\mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\mid \log \left({\tilde{F}}_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right)\right)-\log (F_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right))\mid .$$

Remark 4

We allow S to be stochastic in Definition 1 and also in the hypotheses of Theorem 3. However, we observe that if S is stochastic, then the conclusion of the theorem is trivial since S is the unique stochastic F with F ≥ S.

Proof

Let G ∈ {tF̃+(1–t)F : t ∈ [0, 1]}. Since Q_ij(S) = P_i[τ_j < min{τ_i, τ_ω}](S^ω), we have

$$\begin{array}{cc}\hfill {\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(G\right)& =G_{ij}+{\mathbf{P}}_i[1<{\tau }_j<{\tau }_i]\left(G\right)\hfill \\ \hfill & \ge G_{ij}+{\mathbf{P}}_i[1<{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)\hfill \\ \hfill & =G_{ij}+{\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)-S_{ij}\hfill \\ \hfill & =G_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij}.\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill \mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid & =\mid \sum _{i\ne j}{\int }_0^I\frac{\partial \log {\pi }_m}{\partial F_{ij}}\left(t\tilde{F}+(1-t)F\right)\left({\tilde{F}}_{ij}-F_{ij}\right)dt\mid \hfill \\ \hfill & \le \sum _{i\ne j}{\int }_0^1\frac{\mid {\tilde{F}}_{ij}-F_{ij}\mid }{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(t\tilde{F}+(1-t)F\right)}dt\hfill \\ \hfill & \le \sum _{i\ne j}{\int }_0^1\frac{\mid {\tilde{F}}_{ij}-F_{ij}\mid }{t{\tilde{F}}_{ij}+(1-t)F_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij}}dt\hfill \\ \hfill & =\sum _{i\ne j}\mid \log \left({\tilde{F}}_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right)\right)-\log (F_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right))\mid .\hfill \end{array}$$

The first equality holds since by Lemma 1, the invariant distribution π is Fréchet differentiable on an open neighborhood of the set of irreducible, stochastic matrices. Directional derivatives can then be computed using partial derivatives as above. The denominator in the third line is positive since F, F̃ ≥ S implies

$$t{\tilde{F}}_{ij}+(1-t)F_{ij}-S_{ij}\ge 0,$$

and by Lemma 3, Q_ij(S) > 0.

Theorem 3 takes a somewhat complicated form, so in Corollary 2, we present a simplified version. The proof of Corollary 2 shows that the bound in Theorem 3 is always smaller than the bound in Corollary 2.

Corollary 2

Let F, F̃ be stochastic matrices, let S be substochastic and irreducible, and assume that F, F̃ ≥ S. We have

$$\mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\frac{\mid {\tilde{F}}_{ij}-F_{ij}\mid }{{\mathbf{Q}}_{ij}\left(S\right)}.$$

Proof

We have

$$\mid \log \left({\tilde{F}}_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij}\right)-\log (F_{ij}+{\mathbf{Q}}_{ij}\left(S\right)-S_{ij})\mid \le \underset{x\in [{\mathbf{Q}}_{ij}\left(S\right),\infty ]}{\max }\frac{d\log }{dx}\left(x\right)\mid ({\tilde{F}}_{ij}-S_{ij})-(F_{ij}-S_{ij})\mid =\frac{\mid {\tilde{F}}_{ij}-F_{ij}\mid }{{\mathbf{Q}}_{ij}\left(S\right)},$$

and so the result follows directly from Theorem 3.

We show in Theorem 4 that both Theorem 3 and Corollary 2 are sharp. That is, we show that ρ_ij(S) = Q_ij(S) is within a factor of two of the largest value of ρ_ij(S) so that a bound of the form

$$\mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\mid \log ({\tilde{F}}_{ij}-S_{ij}+{\rho }_{ij}\left(S\right))-\log (F_{ij}-S_{ij}+{\rho }_{ij}\left(S\right))\mid ,$$

and we show that η_ij(S) = Q_ij(S)^–1 is within a factor of two of the smallest value of η_ij(S) such that a bound of the form

$$\mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}{\eta }_{ij}\left(S\right)\mid {\tilde{F}}_{ij}-F_{ij}\mid$$

holds.

Remark 5

We note that some authors call a bound sharp if it is possible for equality to hold. For example, a bound of form (1) may be called sharp if for every stochastic F, there exists a stochastic F̃ so that equality holds. We prefer to call a bound sharp if it is the best bound of a given form, possibly up to a small constant factor. Thus, for bounds of the type which we consider, we require that for every S and every i ≠ j, Q_ij(S)^–1 is as small as possible.

Theorem 4

Let S be an irreducible substochastic matrix, and let i, j ∈ Ω with i ≠ j. For every ε > 0, there exist stochastic matrices F̃, F with F̃, F ≥ S so that

$$\underset{m\in \Omega }{\max }\mid \log {\pi }_m\left(\tilde{F}\right)-\log {\pi }_m\left(F\right)\mid \ge \frac{1}{2}{({\mathbf{Q}}_{ij}\left(S\right)+\epsilon )}^{-1}\mid {\tilde{F}}_{ij}-F_{ij}\mid \ge \mid \log ({\tilde{F}}_{ij}-S_{ij}+2({\mathbf{Q}}_{ij}\left(S\right)+\epsilon ))-\log (F_{ij}-S_{ij}+2({\mathbf{Q}}_{ij}\left(S\right)+\epsilon ))\mid$$

and $\mid {\tilde{F}}_{kl}-F_{kl}\mid =0$ for all (k, l) except (i, j) and (i, i).

Proof

Define

$$F_{kl}≔\{\begin{array}{cc}{S}_{kl}\hfill & \text{if}l\ne i,\text{and}\hfill \\ 1-{\sum }_{m\ne i}{S}_{km}\hfill & \text{if}l=i.\hfill \end{array}\phantom{\}}$$

F is stochastic, F ≥ S, $F_j-F_j{e}_i{e}_i^{\mathrm{t}}=S_j-S_j{e}_i{e}_i^{\mathrm{t}}$ and F_j^⊥,j = S_j^⊥,j. Therefore,

$${\mathbf{Q}}_{ij}\left(S\right)={\mathbf{Q}}_{ij}\left(F\right)={\mathbf{P}}_i[{\tau }_j<{\tau }_i]\left(F\right).$$

We now distinguish two cases: ${\sum }_{m\in \Omega }{S}_{im}=1$ and ${\sum }_{m\in \Omega }{S}_{im}<1$. In the first case, if F ≥ S and F is stochastic, then F_im = S_im for all m ∈ Ω. Therefore, $\mid {\tilde{F}}_{ij}-F_{ij}\mid =0$ for all stochastic F, F̃ ≥ S, and so the conclusion of the theorem holds. In the second case, we observe that ${\sum }_{m\in \Omega }{S}_{im}<1$ implies F_ii > S_ii ≥ 0. Therefore, for any sufficiently small η > 0,

$$F^{\eta }≔F+\eta (e_i{e}_j^{\mathrm{t}}-e_i{e}_i^{\mathrm{t}})$$

is stochastic, and F^η ≥ S. By Theorem 2, we have

$$\underset{m\in \Omega }{\max }\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\left(F\right)\mid \ge \frac{1}{2}\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]}=\frac{1}{2}\frac{1}{{\mathbf{Q}}_{ij}\left(S\right)}.$$

It follows that for every ε > 0 there exists an η > 0 with

$$\underset{m\in \Omega }{\max }\mid \log {\pi }_m\left(F^{\eta }\right)-\log {\pi }_m\left(F\right)\mid \ge \frac{1}{2}{({\mathbf{Q}}_{ij}\left(S\right)+\epsilon )}^{-1}\mid F_{ij}^{\eta }-F_{ij}\mid \ge \mid \log (F_{ij}^{\eta }-S_{ij}+2({\mathbf{Q}}_{ij}\left(S\right)+\epsilon ))-\log (F_{ij}-S_{ij}+2({\mathbf{Q}}_{ij}\left(S\right)+\epsilon ))\mid .$$

(The second inequality follows by an argument similar to the proof of Corollary 2.)

Theorem 3 and Corollary 2 take unusual forms, and at first glance the condition F, F̃ ≥ S may seem inconvenient. In Remarks 6 and 7, we explain how to use these estimates in the common case when only one matrix and a bound on the error are known.

Remark 6

For a given application, the best upper bounds are obtained by choosing the largest possible S. This is a consequence of Lemma 3. We apply this principle in Remark 7.

Remark 7

Suppose that F̃ has been computed as an approximation to an unknown stochastic matrix F and that we have some bound on the error between F̃ and F. For example, suppose that for some matrix α ≥ 0,

$$\mid {\tilde{F}}_{ij}-F_{ij}\mid \le {\alpha }_{ij}\text{for all}i,j\in \Omega .$$

In this case, we define

$$S_{ij}≔\max \{{\tilde{F}}_{ij}-{\alpha }_{ij},0\}\text{for all}i,j\in \Omega .$$

We observe that this choice of S is the largest possible so that F ≥ S for all F so that |F_ij – F̃_ij| ≤ α_ij. Therefore, by Lemma 3, the coefficients Q_ij(S) are as large as possible, giving the best possible upper bounds.

If S is irreducible, we have

$$\underset{k\in \Omega }{\max }\mid \log {\pi }_k\left(\tilde{F}\right)-\log {\pi }_k\left(F\right)\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\mid \log \left({\tilde{F}}_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right)\right)-\log (F_{ij}-S_{ij}+{\mathbf{Q}}_{ij}\left(S\right))\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\frac{\mid {\tilde{F}}_{ij}-F_{ij}\mid }{{\mathbf{Q}}_{ij}\left(S\right)}.$$

In general, if S is reducible, then no statement can be made about the error of the invariant distribution. In fact, if S is reducible, then there is a reducible, stochastic F with $\mid {\tilde{F}}_{ij}-F_{ij}\mid \le {\alpha }_{ij}$, and the invariant distribution of F is not even unique.

5. An efficient algorithm for computing sensitivities

In Theorem 5 below, we show that the coefficients Q_ij(S) can be computed by inverting an L × L matrix and performing additional operations of cost O(L²). Therefore, the cost of estimating the error in π using either Theorem 3 or Corollary 2 is comparable to the cost of computing π. Moreover, the cost of computing our bounds is the same as the cost of computing most other bounds in the literature, for example, those based on the group invserse; see Remark 8.

Our first step is to characterize Q_ij(S) as the solution of a linear equation. We advise the reader that we will make extensive use of the notation introduced in Section 2. In addition, we define

$$S_{j^{\perp },j}≔(I-e_j{e}_j^{\mathrm{t}})S{e}_j\text{and}{S}_{j,j^{\perp }}≔e_j^{\mathrm{t}}S(I-e_j{e}_j^{\mathrm{t}})$$ (20)

to be the j^th column and row of S with the j^th entry set to zero, respectively.

Lemma 4

Let S be irreducible and substochastic or stochastic. For i, j ∈ Ω with i ≠ j, let $q^{ij}\left(S\right)\in e_j^{\perp }$ be the vector defined by

$$q_k^{ij}\left(S\right)≔{\mathbf{P}}_k[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(S^{\omega }\right)$$

for all k ∈ Ω \ {j}. Define S_j as in Section 2, and S_j^⊥,j by (20). The operator $I-S_j+S_j{e}_i{e}_i^{\mathrm{t}}$ is invertible on $e_j^{\perp }$, and q^ij(S) is the unique solution of the equation

$$(I-S_j+S_j{e}_i{e}_i^{\mathrm{t}})q^{ij}\left(S\right)=S_{j^{\perp },j}.$$ (21)

Proof

Let i, j ∈ Ω with i ≠ j. We have

$${\mathbf{P}}_k[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]=\sum _{l\in \Omega \cup \left\{\omega \right\}}{\mathbf{P}}_k[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\},X_1=l],$$

and for k ≠ j,

$${\mathbf{P}}_k[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\},X_1=l]=\{\begin{array}{cc}0\hfill & \text{if}l\in \{i,\omega \},\hfill \\ S_{kj}\hfill & \text{if}l=j,\text{and}\hfill \\ {\mathbf{P}}_l[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]S_{kl}\hfill & \text{if}l\notin \{i,j,\omega \}.\hfill \end{array}\phantom{\}}$$

Therefore,

$${\mathbf{P}}_k[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]=\sum _{l\notin \{i,j,\omega \}}{S}_{kl}{\mathbf{P}}_l[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]+S_{kj}\text{for}k\ne j.$$ (22)

We observe that equation (22) above can be expressed as

$$q^{ij}=(S_j-S_j{e}_i{e}_i^{\mathrm{t}})q^{ij}+S_{j^{\perp },j}.$$ (23)

We now claim that if S is irreducible, then $I-S_j+S_j{e}_i{e}_i^{\mathrm{t}}$ is invertible, which shows that q^ij is the unique solution of (23). By (10), ${(I-S_j)}^{-1}={\sum }_{m=0}^{\infty }{S}_j^m$. We now observe that

$$0\le S_j-S_j{e}_i{e}_i^{\mathrm{t}}\le S_j,$$

so ${\sum }_{m=0}^{\infty }{(S_j-S_j{e}_i{e}_i^{\mathrm{t}})}^m$ converges, hence $I-S_j+S_j{e}_i{e}_i^{\mathrm{t}}$ is invertible.

The proof of Theorem 5 uses the following lemma.

Lemma 5

For S substochastic and irreducible,

$${\mathbf{Q}}_{ij}\left(S\right)=\frac{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{S}_{j^{\perp },j}}{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{e}_i}.$$

Proof

We recall that

$$(I-S_j+S_j{e}_i{e}_i^{\mathrm{t}})q^{ij}\left(S\right)=S_{j^{\perp },j}.$$

Multiplying both sides by (I – S_j)^–1 then yields

$$(I+{(I-S_j)}^{-1}{S}_j{e}_i{e}_i^{\mathrm{t}})q^{ij}\left(S\right)={(I-S_j)}^{-1}{S}_{j^{\perp },j}.$$

Now S_j ≥ 0 is a substochastic matrix, so by (10), ${(I-S_j)}^{-1}={\sum }_{m=0}^{\infty }{S}_j^m\ge 0$. Therefore, $1+e_i^{\mathrm{t}}{(I-S_j)}^{-1}{S}_j{e}_i\ge 1$, and by the Sherman-Morrison formula,

$$\begin{array}{cc}\hfill {(I+{(I-S_j)}^{-1}{S}_j{e}_i{e}_i^{\mathrm{t}})}^{-1}& =I-\frac{{(I-S_j)}^{-1}{S}_j{e}_i{e}_i^{\mathrm{t}}}{1+e_i^{\mathrm{t}}{(I-S_j)}^{-1}{S}_j{e}_i}\hfill \\ \hfill & =I-\frac{{(I-S_j)}^{-1}{S}_j{e}_i{e}_i^{\mathrm{t}}}{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{e}_i}.\hfill \end{array}$$

Thus,

$$\begin{array}{cc}\hfill {\mathbf{Q}}_{ij}\left(S\right)=q_i^{ij}\left(S\right)& =e_i^{\mathrm{t}}{(I+{(I-S_j)}^{-1}{S}_j{e}_i{e}_i^{\mathrm{t}})}^{-1}{(I-S_j)}^{-1}{S}_{j^{\perp },j}\hfill \\ \hfill & =\frac{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{S}_{j^{\perp },j}}{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{e}_i}.\hfill \end{array}$$ (24)

Theorem 5

Let $S\in {\mathbb{R}}^{L\times L}$ be irreducible and substochastic. The set of all coefficients Q_ij(S) can be computed by inverting an L × L matrix and then performing additional operations of cost O(L²).

Proof

We begin with some definitions and notation. Define

$$A\left(j\right)≔I-S+e_j{e}_j^{\mathrm{t}}S=\begin{array}{c}\hfill e_j^{\perp }\hfill \\ \hfill e_j\hfill \end{array}\left(\begin{array}{cc}\hfill e_j^{\perp }\hfill & \hfill e_j\hfill \\ \hfill I-S_j\hfill & \hfill -S_{j^{\perp },j}\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right).$$

(The right hand side above denotes the block decomposition of A with respect to the decomposition ${\mathbb{R}}^L\simeq e_j^{\perp }\oplus \mathbb{R}{e}_j$. Thus, for example, I – S_j is to be interpreted as an operator on $e_j^{\perp }$; cf. the definition of S_j in Section 2.) We observe that A(j) is invertible, since by (10), I – S_j is invertible, so

$$A{\left(j\right)}^{-1}=\begin{array}{c}\hfill e_j^{\perp }\hfill \\ \hfill e_j\hfill \end{array}\left(\begin{array}{cc}\hfill e_j^{\perp }\hfill & \hfill e_j\hfill \\ \hfill {(I-S_j)}^{-1}\hfill & \hfill {(I-S_j)}^{-1}{S}_{j^{\perp },j}\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right).$$ (25)

In the first step of the algorithm, we compute A(1)^–1, which costs O(L³) operations. Second, we compute SA(1)^–1. Given A(1)^–1, this can be done in O(L²) operations using the formula

$$SA{\left(1\right)}^{-1}=\begin{array}{c}\hfill e_1^{\perp }\hfill \\ \hfill e_1\hfill \end{array}\left(\begin{array}{cc}\hfill e_1^{\perp }\hfill & \hfill e_1\hfill \\ \hfill A{\left(1\right)}_1^{-1}-I\hfill & \hfill A{\left(1\right)}_{1^{\perp },1}^{-1}\hfill \\ \hfill S_{1,1^{\perp }}A{\left(1\right)}_1^{-1}\hfill & \hfill S_{11}+S_{1,1^{\perp }}A{\left(1\right)}_{1^{\perp },1}^{-1}\hfill \end{array}\right).$$

(The formula is easily proved by direct calculation using (25).) Third, we compute Q_i1(S) for all i ≠ 1. By Lemma 5 and 25, we have

$${\mathbf{Q}}_{i1}\left(S\right)=\frac{e_i^{\mathrm{t}}{(I-S_1)}^{-1}{S}_{1^{\perp },1}}{e_i^{\mathrm{t}}{(I-S_1)}^{-1}{e}_i}=\frac{A{\left(1\right)}_{i1}^{-1}}{A{\left(1\right)}_{ii}^{-1}}.$$ (26)

Therefore, once A(1)^–1 has been computed, it costs O(L) operations to compute Q_i1(S) for all i ≠ 1.

We compute the remaining sensitivities Q_ij(S) for j ≠ 1 by a formula analogous to (28), but with A(j)^–1 in place of A(1)^–1. To do so efficiently, we use the Sherman-Morrison-Woodbury identity to derive a formula expressing A(j)^–1 in terms of A(1)^–1:

$$\begin{array}{cc}\hfill A{\left(j\right)}^{-1}& ={(I-S+e_1{e}_1^{\mathrm{t}}S+(e_j{e}_j^{\mathrm{t}}S-e_1{e}_1^{\mathrm{t}}S))}^{-1}\hfill \\ \hfill & =A{\left(1\right)}^{-1}-A{\left(1\right)}^{-1}(e_j-e_1)C{\left(j\right)}^{-1}\left(\begin{array}{c}\hfill e_j^{\mathrm{t}}\hfill \\ \hfill e_1^{\mathrm{t}}\hfill \end{array}\right)SA{\left(1\right)}^{-1},\hfill \end{array}$$ (27)

where

$$C\left(j\right)≔I+\left(\begin{array}{c}\hfill e_j^{\mathrm{t}}\hfill \\ \hfill e_1^{\mathrm{t}}\hfill \end{array}\right)SA{\left(1\right)}^{-1}(e_j-e_1).$$

In the fourth step of the algorithm, we loop over all j ≠ 1. For each j, we first compute C(j)^–1, which requires a total of O(L) operations. We then compute A(j)^–1e_j at a cost of O(L) operations using a formula derived from (27):

$$\begin{array}{cc}\hfill A{\left(j\right)}^{-1}{e}_j& =A{\left(1\right)}^{-1}{e}_j=A{\left(1\right)}^{-1}(e_j-e_1)C{\left(j\right)}^{-1}\left(\begin{array}{c}\hfill e_j^{\mathrm{t}}\hfill \\ \hfill e_1^{\mathrm{t}}\hfill \end{array}\right)SA{\left(1\right)}^{-1}{e}_j\hfill \\ \hfill & =A{\left(1\right)}^{-1}{e}_j-(A{\left(1\right)}^{-1}{e}_j-A{\left(1\right)}^{-1}{e}_1)C{\left(j\right)}^{-1}\left(\begin{array}{c}\hfill SA{\left(1\right)}_{jj}^{-1}\hfill \\ \hfill SA{\left(1\right)}_{1j}^{-1}\hfill \end{array}\right).\hfill \end{array}$$

Next, we must compute $A{\left(j\right)}_{ii}^{-1}$ for all i ≠ j. By (27),

$$A{\left(j\right)}_{ii}^{-1}=A{\left(1\right)}_{ii}^{-1}-(A{\left(1\right)}_{ij}^{-1}-A{\left(1\right)}_{i1}^{-1})C{\left(j\right)}^{-1}\left(\begin{array}{c}\hfill SA{\left(1\right)}_{ji}^{-1}\hfill \\ \hfill SA{\left(1\right)}_{1i}^{-1}\hfill \end{array}\right),$$

so the cost of computing $A{\left(j\right)}_{ii}^{-1}$ for each i ≠ j is O(L). Finally, we compute Q_ij(S) for all i ≠ j. By Lemma 5 and 25, we have

$${\mathbf{Q}}_{ij}\left(S\right)=\frac{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{S}_{j^{\perp },j}}{e_i^{\mathrm{t}}{(I-S_j)}^{-1}{e}_i}=\frac{A{\left(1\right)}_{ij}^{-1}}{A{\left(1\right)}_{ii}^{-1}},$$ (28)

so this last step costs O(L).

The total cost of the algorithm described above is a single L × L matrix inversion plus O(L²).

Remark 8

Most perturbation bounds in the literature have the same computational complexity as our bound. For example, some bounds are based on the group inverse of I – F [13,16]. The cost of computing the group inverse is O(L³) [6], so our bound has the same complexity as [13, 16]. Computing the bound on relative error in [9] requires finding ${\Vert {(I-F_j)}^{-1}\Vert }_{\infty }$ for all j; see (5). This could be done in O(L³) operations by methods similar to Theorem 5, so we conjecture that our bound and the bound of [9] have the same complexity. On the other hand, the bound on relative error in [18] (see (6)) requires almost no calculation at all.

Remark 9

We give the algorithm above to show that the cost of computing our bounds is comparable to the cost of other bounds, in principle. We do not claim that the algorithm is always reliable, since we have not performed a complete stability analysis. Nonetheless, in many cases, the computation of Q_ij(S) is stable even when the computation of π(F) is unstable. For example, suppose that S = αF for F a stochastic matrix and α ∈ (0, 1). (This would be a good choice of S if all entries of F were known with relative error α^–1; cf. Remark 7.) Let ∥M∥_∞ denote the operator norm of the matrix $M\in {\mathbb{R}}^{L\times L}$ with respect to the ${\ell }^{\infty }$-norm

$${\Vert v\Vert }_{\infty }≔\underset{i=1,\dots ,L}{\max }\mid v_i\mid \text{for}v\in {\mathbb{R}}^L.$$

It is a standard result that

$${\Vert M\Vert }_{\infty }=\underset{i=1,\dots ,L}{\max }\sum _{j=1}^L\mid M_{ij}\mid .$$

Therefore, since F is stochastic and 0 ≤ S_j ≤ S = αF, ∥S_j∥_∞ ≤ α, and we have

$${\Vert {(I-S_j)}^{-1}\Vert }_{\infty }={\Vert \sum _{n=0}^{\infty }{S}_j^n\Vert }_{\infty }\le \sum _{n=0}^{\infty }{\Vert S_j^n\Vert }_{\infty }\le \sum _{n=0}^{\infty }{\alpha }^n=\frac{1}{1-\alpha }.$$

Moreover,

$${\Vert I-S_j\Vert }_{\infty }\le 2,$$

and so the condition number for the inversion of I – S_j satisfies

$${\kappa }_{\infty }(I-S_j)≔{\Vert I-S_j\Vert }_{\infty }{\Vert {(I-S_j)}^{-1}\Vert }_{\infty }\le \frac{2}{1-\alpha }.$$

We conclude that if α is not too close to one, then the algorithm is stable for any F. For example, if the entries of F are known with 2% error, then we choose S = 0.98F, and we have κ_∞(F) ≤ 100.

6. The hilly landscape example

In this section, we discuss an example in which the invariant distribution is very sensitive to some entries of the transition matrix, but insensitive to others. The example arose from a problem in computational statistical physics. We will use the example to compare our results with previous work, especially [2,9,18] and the bounds on absolute error summarized in [3].

6.1. Transition matrix and physical interpretation

Our hilly landscape example is a simple analogue of the dynamics of a single particle in contact with a heat bath. Define $V:\mathbb{R}\to \mathbb{R}$ by

$$V\left(x\right)=\frac{1}{4\pi }\cos \left(4\pi x\right)$$

Take $L\in \mathbb{N}$, and let Ω := {1, 2, . . . , L} with periodic boundary conditions; that is, take $\Omega ≔\mathbb{Z}∕L\mathbb{Z}$. Given V, we define a probability distribution on Ω by

$$\pi \left(i\right)≔\frac{\exp (-LV(i∕L))}{{\sum }_{k=1}^L\exp (-LV(k∕L))}.$$

The measure π is in detailed balance with the Markov chain X having transition matrix $F\in {\mathbb{R}}^{\Omega \times \Omega }$ defined by

$$\begin{array}{cc}\hfill F_{ii}≔\frac{1}{2}\left(\frac{\pi \left(i\right)}{\pi (i-1)+\pi \left(i\right)}+\frac{\pi \left(i\right)}{\pi (i+1)+\pi \left(i\right)}\right)\hfill & \text{for all}i\in \Omega ,\hfill \\ \hfill F_{i,i+1}≔\frac{1}{2}\frac{\pi (i+1)}{\pi (i+1)+\pi \left(i\right)}\hfill & \text{for all}i\in \Omega ,\hfill \\ \hfill F_{i,i-1}≔\frac{1}{2}\frac{\pi (i-1)}{\pi (i-1)+\pi \left(i\right)}\hfill & \text{for all}i\in \Omega ,\text{and}\hfill \\ \hfill F_{ij}≔0\hfill & \text{otherwise}.\hfill \end{array}$$

(In the definition of F, F_L,L+1 means F_L,1 and F_1,0 means F_1,L, since we take Ω with periodic boundary conditions.)

We interpret X as the position of a particle which moves through the interval (0, 1] with periodic boundary conditions. If V(i) > V(j), we say that j is downhill from i. When the inequality is reversed we say that j is uphill from i. Under the dynamics prescribed by F, the particle is more likely to move downhill than uphill. In fact, as L tends to infinity, π becomes more and more concentrated near the minima of V. For large L, the particle spends most of the time near minima of V, and transitions of the particle between minima occur rarely.

6.2. Sensitivities for the hilly landscape transition matrix

In Figure 1, we plot –logQ_ij(αF) versus i and j for F the hilly landscape transition matrix with L = 40 and α ∈ {0.7, 0.8, 0.9, 0.95, 0.98, 1}. The purpose of this section is to give an intuitive explanation of the main features observed in the figure. Recall that the potential V is shaped roughly like a “W” with peaks at 0, ½, and 1 and valleys at ¼ and ¾. When L = 40, the peaks correspond to the indices 0, 20, and 40 in Ω, and the valleys correspond to 10 and 30. (To be precise, 0 and 40 are identical since we take periodic boundary conditions.)

Figure 1.

Sensitivities for the hilly landscape transition matrix.

Now consider the case α = 1. We observe that –logQ_20,j(F) is small for all j, so Q_20,j(F)^–1 is small, and π(F) is insensitive to perturbations which change the transition probabilities from the peak to other points. This is as expected, since the probability P₂₀[τ_j < τ₂₀](F) = Q_20,j(F) of hitting a point j in the valley before returning to the peak should be fairly large. On the other hand, –logQ_30,10(F) is enormous, so π(F) is sensitive to the transition probability from the valley to the peak. This is also as expected, since the probability P₃₀[τ₁₀ < τ₃₀](F) of climbing from the valley to the peak without falling back into the valley should be small. To explain the small values of –logQ_ij(F) observed near the diagonal, we observe that for all i ∈ Ω and all L,

$$F_{i,i+1}=\frac{1}{2}\frac{1}{1+\exp \left[L(V\left(\frac{i+1}{L}\right)-V\left(\frac{i}{L}\right))\right]}\ge \frac{1}{2}\frac{1}{1+\exp \left(\mathrm{Lip}\left(V\right)\right)}=\frac{1}{2}\frac{1}{1+\exp \left(1\right)}.$$

where Lip(V) = 1 is the Lipschitz constant of the potential V. Therefore, by Corollary 1,

$${\mathbf{Q}}_{i,i+1}{\left(F\right)}^{-1}\le \frac{1}{F_{i,i+1}}\le 2(1+\exp \left(1\right)).$$

The same estimate holds for Q_i,i–1(F).

The coefficients Q_ij(αF) for α < 1 share many features with Q_ij(F). These coefficients would be relevant if F were known with relative error 1 – α; see Remark 7. The main difference between Q_ij(αF) and Q_ij(F) is that Q_ij(αF) is small whenever the minimum number of time steps required to transition between i and j is large. The reader is directed to Section 6.5 for discussion of a related phenomenon. Observe that this effect grows more dominant as α decreases. We also note that Q_ij(αF)^–1 is again small near the diagonal. In fact, by Lemma 3, we have

$${\mathbf{Q}}_{i,i+1}{\left(\alpha F\right)}^{-1}\le \frac{1}{\alpha F_{i,i+1}}\le \frac{2(1+\exp \left(1\right))}{\alpha }.$$

The same estimate holds for Q_i,i–1(αF).

6.3. Mean first passage times and related relative error bounds

Section 4 of [2] also suggests bounds on relative error in terms of certain first passage times. A comparison is therefore in order. We record a simplified version of these results below.

Theorem 6

[2, Corollaries 4.1,4.2] Let F and F̃ be irreducible stochastic matrices. We have

$$\frac{{\pi }_m\left(\tilde{F}\right)-{\pi }_m\left(F\right)}{{\pi }_m\left(F\right)}=\sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}{\pi }_i\left(\tilde{F}\right)((1-{\delta }_{jm}){\mathbf{E}}_j\left[{\tau }_m\right]-(1-{\delta }_{im}){\mathbf{E}}_i\left[{\tau }_m\right])\left({\tilde{F}}_{ij}-F_{ij}\right),$$

where the expectations are taken for the chain with transition matrix F and δ is the Kronecker delta function. Therefore,

$$\mid \frac{{\pi }_m\left(\tilde{F}\right)-{\pi }_m\left(F\right)}{{\pi }_m\left(F\right)}\mid \le \sum _{\begin{array}{c}\hfill i,j\in \Omega \hfill \\ \hfill i\ne j\hfill \end{array}}\mid (1-{\delta }_{jm}){\mathbf{E}}_j\left[{\tau }_m\right]-(1-{\delta }_{im}){\mathbf{E}}_i\left[{\tau }_m\right]\mid \mid {\tilde{F}}_{ij}-F_{ij}\mid .$$

Taking the maximum over all m in the second sentence of Theorem 6 yields a pertubation result having a form similar to Theorem 3, but with

$${\beta }_{ij}\left(F\right)≔\underset{m\in \Omega }{\max }\mid (1-{\delta }_{jm}){\mathbf{E}}_j\left[{\tau }_m\right]-(1-{\delta }_{im}){\mathbf{E}}_i\left[{\tau }_m\right]\mid$$

in place of Q_ij(S)^–1. In the next paragraph, we show for the hilly landscape transition matrix that for some values of i and j, β_ij grows exponentially with L while Q_ij(S)^–1 remains bounded. Thus, the results of [2] dramatically overestimate the error due to some perturbations.

To derive the estimate in the second sentence of Theorem 6 from the exact formula in the first sentence, one discards a factor of π_i(F̃). Therefore, roughly speaking, the estimate is poor when π_i(F̃) is small. To give a specific example, let F be the hilly landscape transition matrix, assume that L is even, and let i = L/2 and j = L/2 + 1. Observe that we have chosen i at a peak of the potential V, so

$${\pi }_i\left(F\right)\gtrsim \exp (-L)$$

(Here, we use the symbols ≲ and ≳ to denote bounds up to multiplicative constants, so in the last line above, we mean that there is some C > 0 so that the left hand side is bounded above by C exp(–L) for all L.) Let $F^{\epsilon }=F+\epsilon (e_i{e}_j^{\mathrm{t}}-e_i{e}_i^{\mathrm{t}})$. We have

$${\phantom{\mid }\frac{\partial \log {\pi }_m}{\partial F_{ij}}\left(F\right)=\frac{d}{d\epsilon }\mid }_{\epsilon =0}{\pi }_m\left(F^{\epsilon }\right)={\pi }_i\left(F\right)((1-{\delta }_{jm}){\mathbf{E}}_j\left[{\tau }_m\right]-(1-{\delta }_{im}){\mathbf{E}}_i\left[{\tau }_m\right])$$

by Theorem 6. Therefore, by Theorem 2,

$$\underset{m\in \Omega }{\max }\mid \frac{\partial \log {\pi }_m}{\partial F_{ij}}\left(F\right)\mid ={\pi }_i\left(F\right)\underset{m\in \Omega }{\max }\mid (1-{\delta }_{jm}){\mathbf{E}}_j\left[{\tau }_m\right]-(1-{\delta }_{im}){\mathbf{E}}_i\left[{\tau }_m\right]\mid \ge \frac{1}{2}\frac{1}{{\mathbf{P}}_i[{\tau }_j<{\tau }_i]},$$

and so

$${\beta }_{ij}\left(F\right)\ge \frac{1}{2}\frac{1}{{\pi }_i\left(F\right){\mathbf{P}}_i[{\tau }_j<{\tau }_i]}\gtrsim \exp \left(L\right).$$

Now suppose that the substochastic matrix S appearing in our bound is chosen for each L so that S_ij for i = L/2 and j = L/2 + 1 is bounded above zero uniformly as L → ∞. For example, one might choose S to be a multiple of F as in Section 6.2 or a multiple of a simple random walk transition matrix as in Section 6.5. Then by Lemma 3, we have

$${\mathbf{Q}}_{ij}{\left(S\right)}^{-1}\le \frac{1}{S_{ij}},$$

so Q_ij(S)^–1 is bounded. Thus, β_ij(F) is a poor estimate of the sensitivity of the ij^th entry for this problem.

6.4. The spectral gap and related absolute error bounds

The survey article [3] lists eight condition numbers κ_i(F) for i = 1, 2, . . . , 8 for which bounds of the form

$${\Vert \pi \left(F\right)-\pi \left(\tilde{F}\right)\Vert }_p\le {\kappa }_i\left(F\right){\Vert F-\tilde{F}\Vert }_{p^{\prime }}$$

hold. (The Hölder exponents p and p′ vary with the choice of condition number.) Some of these condition numbers are based on ergodicity coefficients [20,21], some on mean first passage times [2], and some on generalized inverses of the characteristic matrix I – F [5,7,9,16,19]. We prove for the hilly landscape transition matrix F that κ_i(F) increases exponentially with L for all i. By contrast, we have already seen that many of the coefficients Q_ij(αF)^–1 are bounded as L tends to infinity.

Our proof that the condition numbers increase exponentially is based on an analysis of the spectral gap of F. Let σ(F) denote the spectrum of F. The spectral gap γ is defined to be

$$\gamma ≔1-\max \{\mid \lambda \mid :\lambda \in \sigma \left(F\right)\backslash \left\{1\right\}\}.$$ (29)

We use the bottleneck inequality [14, Theorem 7.3] to show that the spectral gap of the hilly landscape transition matrix decreases exponentially with L. For convenience, assume that L is even, and let

$$E≔\left\{1,\dots ,\frac{L}{2}\right\}.$$

The bottleneck ratio [14, Section 7.2] for the partition {E, E^c} is

$$\begin{array}{cc}\hfill \Phi (E,E^{\mathrm{c}})& =\frac{\pi \left(\frac{L}{2}\right)F_{\frac{L}{2},\frac{L}{2}+1}+\pi \left(1\right)F_{1,0}}{\frac{1}{2}}\hfill \\ \hfill & \lesssim \frac{\exp (-L)}{1+\exp \left[L(V\left(\frac{1}{L}\right)-V\left(0\right))\right]}\hfill \\ \hfill & \lesssim \exp (-L).\hfill \end{array}$$

(As in the last section, we use the symbols ≲ and ≳ to denote bounds up to multiplicative constants.) Therefore, by the bottleneck inequality, the mixing time t_mix [14, Section 4.5] satisfies

$$t_{\mathrm{mix}}\ge \frac{1}{4\Phi (E,E^{\mathrm{c}})}\gtrsim \exp \left(L\right).$$

By [14, Theorem 12.3],

$$\frac{1}{\gamma }\log \left(\frac{4}{\min \{\pi \left(i\right):i\in \Omega \}}\right)=\frac{\log \left(4\right)+L}{\gamma }\ge t_{\mathrm{mix}}.$$

Therefore,

$$\frac{1}{\gamma }\gtrsim \frac{\exp \left(L\right)}{L},$$ (30)

and we see that the spectral gap decreases exponentially in L.

We now relate the condition numbers to the spectral gap. Using Equation (3.3) and the table at the bottom of page 147 in [3], and also [11, Corollary 2.6], we have

$${\kappa }_i\gtrsim \frac{1}{L\min \{\mid 1-\lambda \mid :\lambda \in \sigma \left(F\right)\backslash \left\{1\right\}\}}$$ (31)

for all i = 1, . . . , 8. Now we claim that for sufficiently large L,

$$\gamma =1-\max \{\mid \lambda \mid :\lambda \in \sigma \left(F\right)\backslash \left\{1\right\}\}=\min \{\mid 1-\lambda \mid :\lambda \in \sigma \left(F\right)\backslash \left\{1\right\}\},$$ (32)

in which case (30) and (31) imply that all condition numbers grow exponentially with L. To see this, we first observe that since F is reversible, its spectrum is real [14, Lemma 12.2]. Moreover,

$$F_{ii}\ge \frac{1}{1+\exp \left(\mathrm{Lip}\left(V\right)\right)}=\frac{1}{1+\exp \left(1\right)}>0\text{for all}i\in \Omega ,$$

where Lip(V) = 1 is the Lipschitz constant of V. Therefore, using the Gershgorin circle theorem we have

$$\lambda \ge \frac{2}{1+\exp \left(1\right)}-1$$ (33)

for all λ ∈ σ(F). Inequality (33) shows that σ(F) is bounded above –1 uniformly in L, and by (30),

$$\underset{L\to \infty }{\lim }\gamma =\underset{L\to \infty }{\lim }1-\max \{\mid \lambda \mid :\lambda \in \sigma \left(F\right)\backslash \left\{1\right\}\}=0.$$

It follows that for sufficiently large L, max{|λ| : λ ∈ σ(F) \ {1}} is attained for λ > 0. Thus, equation (32) holds, and we conclude using (30) and (31) that

$${\kappa }_i\gtrsim \frac{\exp \left(L\right)}{L^2}$$

for all i = 1, . . . , 8.

6.5. Bounds below by a random walk

Let Y be the random walk on Ω with transition matrix

$$\begin{array}{cc}\hfill P_{ii}=P_{i,i+1}=P_{i,i-1}& ≔\frac{1}{3}\text{for all}i\in \Omega ,\text{and}\hfill \\ \hfill P_{ij}& ≔0\text{otherwise}.\hfill \end{array}$$

(As above, since Ω has periodic boundaries, P_L,L+1 means P_L,1, etc.) In this section, we use Theorem 3 to relate Q_ij(P) with Q_ij(F) for F the hilly landscape transition matrix. First, using the lower bounds on entries of F derived in Section 6.2, we have

$$F\ge \frac{3}{2}\frac{1}{1+\exp \left(1\right)}P.$$

Therefore, by (35) and Lemma 3,

$${\mathbf{Q}}_{ij}\left(\alpha F\right)\ge {\mathbf{Q}}_{ij}\left(\frac{3\alpha }{2(1+\exp \left(1\right))}P\right).$$ (34)

Now for any β ∈ (0, 1),

$$\begin{array}{cc}\hfill {\mathbf{Q}}_{ij}\left(\beta P\right)& ={\mathbf{P}}_i[{\tau }_j<\min \{{\tau }_i,{\tau }_{\omega }\}]\left(\beta P^{\omega }\right)\hfill \\ \hfill & =\sum _{M=1}^{\infty }{\mathbf{P}}_i[{\tau }_j=M,\min \{{\tau }_i,{\tau }_{\omega }\}>M]\left(\beta P^{\omega }\right).\hfill \end{array}$$

Let |i – j| denote the minimum number of time steps required for the chain to reach state j from state i. Adopting the notation used in the proof of Lemma 3, there is some path $\gamma \in {\mathcal{P}}_{ij}^{\mid i-j\mid }$ of length |i – j| for which

$$\begin{array}{cc}\hfill {\mathbf{Q}}_{ij}\left(\beta P\right)& \ge {\mathbf{P}}_i[{\tau }_j=\mid i-j\mid ,\min \{{\tau }_i,{\tau }_{\omega }\}>\mid i-j\mid ]\left(\beta P^{\omega }\right)\hfill \\ \hfill & \ge {\mathbf{P}}_i[X_k=\gamma \left(k\right)\text{for}k=1,\dots ,\mid i-j\mid ]\left(\beta P^{\omega }\right)\hfill \\ \hfill & ={\left(\frac{\beta }{3}\right)}^{\mid i-j\mid }.\hfill \end{array}$$ (35)

Combining (34) and (35) then yields

$${\mathbf{Q}}_{ij}\left(\alpha F\right)\ge {\mathbf{Q}}_{ij}\left(\frac{3\alpha }{2(1+\exp \left(1\right))}P\right)\ge {\left(\frac{3\alpha }{2(1+\exp \left(1\right))}\right)}^{\mid i-j\mid }.$$

Appendix A. Proof of Lemma 1

Proof

Let F be irreducible and stochastic. By [6, Equation (3.1)], det(I – F_i) > 0 for all i ∈ Ω, and

$$\pi {\left(F\right)}^{\mathrm{t}}=\frac{1}{{\sum }_{i=1}^L\det (I-F_i)}(\det (I-F_1),\det (I-F_2),\dots ,\det (I-F_L)).$$ (36)

The right hand side of (36) yields the desired extension. To show this, we first observe that there exists an open neighborhood ${\mathcal{V}}_F\subset {\mathbb{R}}^{L\times L}$ of F and a disc ${\mathcal{D}}_F\subset \mathbb{C}$ with $1\in {\mathcal{D}}_F$ such that $G\in {\mathcal{V}}_F$ implies (a) ${\sum }_i\det (I-G_i)>0$ and (b) G has exactly one eigenvalue in ${\mathcal{D}}_F$ and that eigenvalue is simple. There exists a neighborhood with property (a), since ${\sum }_i\det (I-G_i)$ is continuous in G and ${\sum }_i\det (I-F_i)>0$. The existence of a neighborhood ${\mathcal{V}}_F$ and disc ${\mathcal{D}}_F$ with property (b) follows from standard results in perturbation theory, since F irreducible and stochastic implies that 1 is a simple eigenvalue of F; see [10, Ch. II, Theorem 5.14]. We now let $\mathcal{U}$ be the union of the sets ${\mathcal{V}}_F$ over all irreducible, stochastic F, and we define $\stackrel{‒}{\pi }:\mathcal{U}\to {\mathbb{R}}^L$ by extending the formula on the right hand side of (36). By (a), $\stackrel{‒}{\pi }$ is continuously differentiable. By property (b), we know that if $G\in \mathcal{U}$ with Ge = e then e is a simple eigenvalue of G, and so $\ker (I-G)=\mathbb{R}e$. Following the proof of [6, Theorem 3.1], one may then use the identity

$$\left(\mathrm{adj}(I-G)\right)(I-G)=(I-G)\left(\mathrm{adj}(I-G)\right)=0,$$

where adj(·) denotes the adjugate matrix, to show that $\stackrel{‒}{\pi }{\left(G\right)}^{\mathrm{t}}G=\stackrel{‒}{\pi }{\left(G\right)}^{\mathrm{t}}$.