Chained Kullback-Leibler Divergences

Abstract

We define and characterize the “chained” Kullback-Leibler divergence min_w D(p‖w) + D(w‖q) minimized over all intermediate distributions w and the analogous k-fold chained K-L divergence min D(p‖w_k−1) + … + D(w₂‖w₁) + D(w₁‖q) minimized over the entire path (w₁,…,w_k−1). This quantity arises in a large deviations analysis of a Markov chain on the set of types – the Wright-Fisher model of neutral genetic drift: a population with allele distribution q produces offspring with allele distribution w, which then produce offspring with allele distribution p, and so on.

The chained divergences enjoy some of the same properties as the K-L divergence (like joint convexity in the arguments) and appear in k-step versions of some of the same settings as the K-L divergence (like information projections and a conditional limit theorem). We further characterize the optimal k-step “path” of distributions appearing in the definition and apply our findings in a large deviations analysis of the Wright-Fisher process. We make a connection to information geometry via the previously studied continuum limit, where the number of steps tends to infinity, and the limiting path is a geodesic in the Fisher information metric.

Finally, we offer a thermodynamic interpretation of the chained divergence (as the rate of operation of an appropriately defined Maxwell’s demon) and we state some natural extensions and applications (a k-step mutual information and k-step maximum likelihood inference). We release code for computing the objects we study.

I. Introduction

We investigate the properties and applications of the k-fold chained Kullback-Leibler divergence between distributions p and q:

$$D^{\left(k\right)}\left(p\Vert q\right)\Vert \underset{w_1,\dots ,w_{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D\left(w_i\Vert w_{i-1}\right)}$$ (1)

where w₀ ≜ q, w_k ≜ p, D(·‖·) denotes the Kullback-Leibler divergence

$$D^{\left(k\right)}\left(p\Vert q\right)={\displaystyle \sum _{x\in \mathcal{X}}p\left[x\right]\log \frac{p\left[x\right]}{q\left[x\right]}}$$ (2)

and the minimum, if it exists, is taken over a “path” of distributions w_i in some family of distributions. k counts the number of hops and D⁽¹⁾(p‖q) ≜ D(p‖q).

Just as the K-L divergence appears in statistical applications of information theory and in the method of types [1], so the chained divergence arises in the analysis of stochastic processes with an iterative resampling flavor: in each round one samples from a distribution w_i determined by the outcome of the previous round of sampling from w_i−1. A motivating example that we consider in detail is the Wright-Fisher model of neutral genetic drift [2], [3], wherein each generation is obtained by sampling with replacement from the previous generation.

This work is devoted to characterizing the chained divergences in the finite alphabet setting and the optimal k-step path appearing in their definition, and to pointing out their applications. Section II defines the chained divergence in tandem with a discussion of the Wright-Fisher process, in which it naturally appears. Section III derives some properties (like joint convexity) of the chained divergence, including several “operationally” meaningful properties characterizing the rate of large deviations, information projections, and a conditional limit theorem; these are very similar to results on the K-L divergence [4], [5], [6]. Section IV-A characterizes the optimal path of intermediate distributions (w₁, …, w_k−1) for all values of k and states a method for computing this path. Section IV-B considers the continuum limit of k → ∞ (letting the number of distributions in the optimal path tend to infinity) and recapitulates results previously obtained by [7] in this setting: the limiting path is a geodesic in the Fisher information metric; equivalently, part of a great circle connecting $\sqrt{p}$ and $\sqrt{q}$¹. Section V considers extensions and other applications of the chained divergences – in particular a k-step version of the mutual information related to the likelihood that two independent genetic loci become dependent through neutral genetic drift in k generations. Finally, we offer a thermodynamic interpretation of the chained divergence as the optimal rate of operation of an appropriately defined Maxwell’s demon.

We release code for computing the objects we study at². An Appendix contains the details of some of the proofs omitted here due to space constraints and is available at [8].

II. The Wright-Fisher Markov chain

We define the Wright-Fisher (“iterative resampling”) process and review briefly some results about the K-L divergence that we extend to the chained divergences (1).

Denote by $\mathrm{\Delta }\Vert \left\{p\in {ℝ}^{\mathcal{X}}:{\displaystyle {\sum }_{x\in \mathcal{X}}p\left[x\right]}=1,p⩾0\right\}$ the set of distributions on finite alphabet $\mathcal{X}$. Denote by Δ_n the set of distributions with integer denominator n for each component:

$${\mathrm{\Delta }}_n\Vert \mathrm{\Delta }\cap \left\{p:np\in {\mathbb{N}}^{\mathcal{X}}\right\}$$ (3)

Δ_n is the set of types [1], equivalently the possible empirical distributions of n $\mathcal{X}$-valued samples.

Consider the Markov chain whose state space is the set of types Δ_n (3) defined by the following rule: Let the current state of the chain be q ∈ Δ_n. Draw n samples $x_1^n$ i.i.d. from q and let $p\left[x\right]=\frac{1}{n}|\left\{i:x_i=x\right\}|$ be the empirical distribution. p is the next state of the chain. This is the Wright-Fisher model [2], [3] of neutral genetic drift among $|\mathcal{X}|$ alleles in a population of size n: Each of n individuals in a generation is sampled by cloning a uniformly chosen (hence “neutral”) individual from the previous generation. The process is illustrated in Figure 1. As time increases, alleles “die out” (once q[x] = 0, no samples of x can be observed again) until only one allele remains, or “fixes” (that is, q[x] = 1 for some $x\in \mathcal{X}$), so the vertices of the simplex Δ are absorbing states. We will study fluctuations of this chain before fixation.

Fig. 1.

(left) The Wright-Fisher process, a Markov chain on the set of types. (right) A trajectory of the Wright-Fisher process for $|\mathcal{X}|=3$ alleles plotted on the simplex. Eventually only one allele remains (“fixes”).

Denote by P_n(p|q) the transition matrix for this Markov chain:

$$P_n\left(p|q\right)=\mathrm{MultinominalPMF}\left(np;q\right)$$ (4)

$$=\left(\begin{array}{c}n\\ {\left(np\left[x\right]\right)}_{x\in \mathcal{X}}\end{array}\right){\displaystyle \prod _{x\in \mathcal{X}}q{\left[x\right]}^{np\left[x\right]}}$$ (5)

$${\doteq }_n{e}^{-nD\left(p\Vert q\right)}$$ (6)

where the prefactor in (5) is a multinomial coefficient, ≐_n denotes equality to leading exponential order in n³, and D(·‖·) denotes the Kullback-Leibler divergence (2). We can evaluate P_n(p|q) for any p, q ∈ Δ and n, but since the state space for the Wright-Fisher process is the set of types Δ_n (3), n ∈ ℕ, we can think of P_n(p|q) as a |Δ_n| × |Δ_n| stochastic matrix.

Given a set of distributions E ⊂ Δ, we define $P_n\left(E|q\right)\Vert {\displaystyle {\sum }_{p\in E\cap {\mathrm{\Delta }}_n}{P}_n\left(p|q\right)}$ as the probability that the chain hops to somewhere in E starting from q. If E ∩ Δ_n = ∅, then P_n(E|q) ≜ 0. If E is the closure of its interior (so E ∩ Δ_n is nonempty for all large enough n) then Sanov’s theorem tells us [4], [5]

$$P_n\left(E|q\right)\Vert {\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{P}_n\left(p|q\right)}\doteq e^{-nD\left(E\Vert q\right)}$$ (7)

where

$$D\left(E\Vert q\right)\Vert \underset{p\in E}{\inf }D\left(p\Vert q\right)=D\left(p^{\ast }\Vert q\right)$$ (8)

where p* ≜ arg min_p∈E D(p‖q) is the I-projection [6], where the minimum exists and is unique since E is closed (thus compact since E ⊂ Δ) by assumption and D(p‖q) is strictly convex in p. Sanov’s theorem and the conditional limit theorem tell us [4], [5], [1] that conditioned on drawing empirical distribution p ∈ E, distribution p is close⁴ to p* as n → ∞. If E is not closed (so min_p∈E D(p‖q)might not exist) but is convex, then there exists a unique distribution p* – the generalized I-projection [5] – such that D(p‖q) ⩾ D(p‖p*) + D(E‖q) for all p ∈ E.

What happens when we iterate the resampling chain? Denote by $P_n^{\left(k\right)}\left(p|q\right)$ the probability to draw distribution p starting from q in k steps, corresponding to the transition matrix $P_n^{\left(k\right)}=P_n^k$. We recursively express $P_n^{\left(k\right)}\left(\cdot |\cdot \right)$:

$$P_n^{\left(k\right)}\left(p|q\right)\Vert {\displaystyle \sum _{w\in {\mathrm{\Delta }}_n}{P}_n\left(p|w\right)P_n^{\left(k-1\right)}\left(w|q\right)}$$ (9)

$$P_n^{\left(k\right)}\left(E|q\right)\Vert {\displaystyle \sum _{w\in {\mathrm{\Delta }}_n}{P}_n\left(E|w\right)P_n^{\left(k-1\right)}\left(w|q\right)}$$ (10)

with $P_n^{\left(1\right)}\left(\cdot |q\right)\Vert P_n\left(\cdot |q\right)$ as in (4) and (7).

Theorem 1 establishes that the k-fold chained K-L divergence (1) plays the same role for the k-step resampling chain as the K-L divergence plays for one step of the chain, as described above. It is convenient to define recursively the k-step chained divergence D^(k) (p‖q) for p, q ∈ Δ:

$$D^{\left(k\right)}\left(p\Vert q\right)\Vert \underset{w\in \mathrm{\Delta }}{min}\left(D\left(p\Vert w\right)+D^{\left(k-1\right)}\left(w\Vert q\right)\right)$$ (11)

with D⁽¹⁾ (p‖q) ≜ D(p‖q), the K-L divergence (2). Note that we optimize over the simplex Δ (rather than the types Δ_n). Theorem 1 establishes the existence and uniqueness of the minimum and the equivalence of the recursive definition (11) with our earlier definition for the chained divergence (1).

We further define D^(k) (E‖q) for a closed, convex set E ⊂ Δ by analogy with (8):

$$D^{\left(k\right)}\left(E\Vert q\right)\Vert \underset{p\in E}{\inf }{D}^{\left(k\right)}\left(p\Vert q\right)=D^{\left(k\right)}\left(p^{\left(k\right)\ast }\Vert q\right)$$ (12)

and p^(k)* ≜ arg min_p∈E D^(k) (p‖q) is the I-projection of $w_{k-1}^{\left(k\right)\ast }$ on E, where $w_{k-1}^{\left(k\right)\ast }$ is the next-to-last point in the optimal path (15). If E is convex but min_p∈E D^(k) (p‖q) does not exist, then there exists a unique distribution p^(k)* that is the generalized I-projection of $w_{k-1}^{\left(k\right)\ast }$ on E.

III. Characterizing the k-fold chained K-L divergence

Theorem 1

Let k be a positive integer.

1. The k-fold chained divergence (11) satisfies for all p, q ∈ Δ such that D(p‖q) < ∞:

   $$D^{\left(k\right)}\left(p\Vert q\right)\Vert \underset{w\in \mathrm{\Delta }}{min}\left(D\left(p\Vert w\right)+D^{\left(k-1\right)}\left(w\Vert q\right)\right)$$ (13)

   $$={\displaystyle \sum _{i=1}^{k}D\left(w_i^{\left(k\right)\ast }\Vert w_{i-1}^{\left(k\right)\ast }\right)}$$ (14)

   with boundary conditions $w_k^{\left(k\right)\ast }\Vert p$, $w_0^{\left(k\right)\ast }\Vert q$, and

   $$\left(w_1^{\left(k\right)\ast },\dots ,w_{k-1}^{\left(k\right)\ast }\right)\Vert \arg \underset{{\mathrm{\Delta }}^{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D\left(w_i\Vert w_{i-1}\right)}$$ (15)

   where the minimum is over the (k − 1)-fold product set Δ^k−1 = Δ × ⋯ × Δ. The minimum exists and is unique. For closed, convex E ⊂ Δ, (12) defines D^(k)(E‖q). If D(p‖q) = ∞, then D^(k)(E‖q) ≜ ∞ and $w_i^{\left(k\right)\ast }$ is not defined.
2. Convexity: D^(k)(E‖q) is jointly strictly convex in p and q:

   $$D^{\left(k\right)}\left(\alpha p_1+\left(1-\alpha \right)p_2\Vert \alpha q_1+\left(1-\alpha \right)q_2\right)<\alpha D^{\left(k\right)}\left(p_1\Vert q_1\right)+\left(1-\alpha \right)D^{\left(k\right)}\left(p_2\Vert q_2\right)$$ (16)

   for all distributions (p₁, q₁) ≠ (p₂, q₂), α ∈ (0,1).
3. Scaling in k: Suppose 0 < D(p‖q) < ∞. Then D^(k)(E‖q) is strictly decreasing in k and $D^{\left(k\right)}\left(p\Vert q\right)=O\left(\frac{1}{k}\right)$. Theorem 3 of Section IV-B (on the continuum limit k → ∞) gives the more precise scaling:

   $$\underset{k\to \infty }{\lim }k{D}^{\left(k\right)}\left(p\Vert q\right)=2{\theta }_{\sqrt{p},\sqrt{q}}^2$$ (17)

   where ${\theta }_{\sqrt{p},\sqrt{q}}=\mathrm{across}\left(\sqrt{p}\cdot \sqrt{q}\right)$ is the angle between vectors $\sqrt{p}$ and $\sqrt{q}$ (see Section IV-B for details).
4. Markov chain transition matrix $P_n^{\left(k\right)}\left(p|q\right)$ (9), upper bound for p ∈ Δ:

   $$P_n^{\left(k\right)}\left(p|q\right)\le {\left(n+1\right)}^{\left(k-1\right)|\mathcal{X}|}{e}^{-n{D}^{\left(k\right)}\left(p\Vert q\right)}$$ (18)

   Moreover, for p ∈ Δ and sequence (p_n)_n∈ℕ, p_n ∈ Δ_n such that lim_n→∞ D(p_n‖p) = 0

   $$\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{\left(k\right)}\left(p|q\right)=\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{\left(k\right)}\left(p_n|q\right)=-D^{\left(k\right)}\left(p\Vert q\right)$$ (19)

   Equivalently, $P_n^{\left(k\right)}\left(p|q\right)\doteq e^{-n{D}^{\left(k\right)}\left(p\Vert q\right)}$.
5. Sanov’s theorem: Let E ⊂ Δ. The probability $P_n^{\left(k\right)}\left(E|q\right)$ (10) to draw a type in E from q ∈ Δ in k steps is upper bounded:

   $$P_n^{\left(k\right)}\left(E|q\right)\le {\left(n+1\right)}^{k|\mathcal{X}|}{e}^{-n{D}^{\left(k\right)}\left(E\Vert q\right)}$$ (20)

   where D^(k)(E‖q) is defined in (12). Moreover, if E is closed, convex, and E ∩ Δ_n ≠ ∅ for sufficiently large n then

   $$\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{\left(k\right)}\left(E|q\right)=-D^{\left(k\right)}\left(p^{\left(k\right)\ast }\Vert q\right)$$ (21)

   where p^(k)* the k-step I-projection of q on E (12); equivalently, $P_n^{\left(k\right)}\left(E|q\right)\doteq e^{-n{D}^{\left(k\right)}\left(E\Vert q\right)}$.
6. Conditional limit theorem: Let E ⊂ Δ be closed, convex, and E ⊂ Δ_n ≠ ∅ for sufficiently large n, q ∈ Δ − E, D(E‖q) < ∞. Let ${\left(X_t^{\left(i\right)}\right)}_{t=1}^n$ be the i-th sample of the Wright-Fisher process (Figure 1): that is, $X_t^{\left(i\right)}$ drawn i.i.d. $\sim {\widehat{w}}_{i-1}$ with ${\widehat{w}}_0\Vert q$ and ${\widehat{w}}_i$ the empirical distribution of $X_t^{\left(i\right)}$. Then for all ε > 0, i ∈ {1, …, k}, $x\in \mathcal{X}$

   $$\underset{n\to \infty }{\lim }\mathbb{P}\left(|{\widehat{w}}_i\left[x\right]-w_i^{\ast }\left[x\right]|⩾\epsilon |{\widehat{w}}_k\in E\right)=0$$ (22)

   with ${\left(w_i^{\left(k\right)\ast }\right)}_{i=1}^{k-1}$ as in (15) and $w_k^{\ast }=p^{\left(k\right)\ast }=\arg {min}_{p\in E}{D}^{\left(k\right)}\left(p\Vert q\right)$ (the k-step I-projection of q on E (12)).

Joint convexity 2) follows because minimization of jointly convex functions with respect to some of the arguments over a convex set preserves convexity [9]. Existence and uniqueness of the optimal path 1) follow from joint strict convexity and the compactness of Δ ∩ Support(q). Theorem 3 contains 3). The proofs of 4), 5), 6) all follow closely the proofs of analogous results for the K-L divergence in [1], Chapter 11. See Appendix VI-A in [8] for details of the proofs.

IV. Characterizing the k-fold path

What can we say about the optimal k-fold “path” ${\left(w_i^{\left(k\right)\ast }\right)}_{i=1}^{k-1}={min}_{{\mathrm{\Delta }}^{k-1}}{\displaystyle {\sum }_{i=1}^{k}D\left(w_i\Vert w_{i-1}\right)}$ (15)? In the event of a large deviation, we observe a path close to this one with high probability as n → ∞ (Theorem 1.6)).

A. Finite number of steps k

Let’s start with the case k = 2 and find w*(p, q) = arg min_w∈Δ D(p‖w) + D(w‖q). For k > 2, we will then use this local characterization of the path of distributions to compute $w_i^{\ast }=w^{\ast }\left(w_{i+1}^{\ast },w_{i-1}^{\ast }\right)$.

We set up an optimization problem with Lagrangian $\mathrm{\Lambda }=D\left(p\Vert w\right)+D\left(w\Vert q\right)+\left(\mathrm{\lambda }-1\right)\left({\displaystyle {\sum }_{x\in \mathcal{X}}w\left[x\right]-1}\right)$ where λ − 1 is a Lagrange multiplier⁵ enforcing normalization of w. Solving the $|\mathcal{X}|$ equations $\nabla \mathrm{\Lambda }{|}_{w^{\ast }}=0$ for w* we find:

$$w^{\ast }\left[x\right]=w^{\ast }\left(p,q\right)\left[x\right]\Vert \{\begin{array}{cc}\frac{p\left[x\right]}{W\left(e^{\mathrm{\lambda }\ast }\frac{p\left[x\right]}{q\left[x\right]}\right)}& :p\left[x\right]>0\\ e^{-\mathrm{\lambda }\ast }q\left[x\right]& :p\left[x\right]=0\end{array}$$ (23)

where W(·) is the principal branch of the Lambert W function⁶ and λ* is chosen so that w* ∈ Δ is normalized (in fact λ* ∈ [0, 1] for all p, q). The case of p[x] = 0 is obtained by analytic continuation of the solution of the case p[x] > 0. if D(p‖q) = ∞ then w*(p, q) is not defined. Note that the Lagrange multiplier λ* lives inside the Lambert W function (rather than outside like a partition function prefactor), and so can not generally be expressed analytically in terms of p and q; one could try to find λ* numerically⁷. Theorem 1.1) tells us that w* is a global minimum.

Theorem 2

(local characterization of the optimal k-step path) Let k ⩾ 2 and D(p‖q) < ∞ for p, q ∈ Δ. The optimal path ${\left(w_i^{\left(k\right)\ast }\right)}_{i=1}^{k-1}$ (15) used in defining the chained K-L divergence D^(k)(p‖q) satisfies

$$w_i^{\left(k\right)\ast }\left[x\right]=w^{\ast }\left(w_{i+1}^{\left(k\right)\ast },w_{i-1}^{\left(k\right)\ast }\right)\left[x\right]$$ (24)

$$⩾p{\left[x\right]}^{i/k}q{\left[x\right]}^{\left(k-i\right)/k}$$ (25)

for all $x\in \mathcal{X}$, 1 ≤ i ≤ k − 1, where w*(·,·) is defined in (23) and with boundary conditions $w_k^{\left(k\right)\ast }=p$, $w_0^{\left(k\right)\ast }=q$. As a corollary of (24), for i ∈ {1,…, k − 1}

$$\mathrm{Support}\left(w_i^{\left(k\right)\ast }\right)=\mathrm{Support}\left(q\right)$$ (26)

Proof

Suppose (24) does not hold for some p, q, $w_i^{\left(k\right)\ast }$. Then ${\displaystyle {\sum }_{i=1}^{k}D\left(w_i^{\left(k\right)\ast }\Vert w_{i-1}^{\left(k\right)\ast }\right)}$ (14) can be strictly decreased by replacing $w_i^{\left(k\right)\ast }$ with $w^{\ast }\left(w_{i+1}^{\left(k\right)\ast },w_{i-1}^{\left(k\right)\ast }\right)$. Thus (24) holds. ■

To derive the geometric lower bound (25) we massage the function w*(p, q) (23) into an f-divergence⁸, then use the non-negativity of f-divergences to bound λ* and derive $w^{\ast }\left(p,q\right)\left[x\right]⩾\sqrt{p\left[x\right]q\left[x\right]}$, and then derive (25). See Appendix VI-B [8] for details.

Remarks

Figure 2 plots the optimal k-step path for k ∈ {2, 6} and in the k → ∞ limit (see Section IV-B). One way to compute the optimum path is to start with some guess (a good initial point is given in Theorem 3) and repeatedly apply (24) until numerical convergence; this is the method used in making Figure 2 and implemented in the code we release⁹.

Fig. 2.

(left) k-step path (15) from q to p (corresponding to D^(k)(p‖q)) computed by successifly applying (24) for $w_1^{\left(k=2\right)\ast }$ and ${\left(w_i^{\left(k=6\right)}\ast \right)}_{1\le i\le 5}$. The green line shows the limiting k → ∞ path (31). The dark grey lines show the arithmetic and (normalized) geometric mean of p and q for α ∈ [0, 1]. (right) The limiting path as k → ∞, part of a great circle joining $\sqrt{p}$ and $\sqrt{q}$.

The optimal k-step path from q to p is not the same as the optimal path from p to q − an asymmetry inherited from the asymmetry of D^(k)(p‖q); but in the limit k → ∞ these two paths converge to the same limiting path (see Section IV-B). From (23) we see that q[x] > 0 ⇒ w* (p, q)[x] > 0, so $\mathrm{Support}\left(w_i^{\left(k\right)\ast }\right)=\mathrm{Support}\left(q\right)$ for i ∈ {1, …, k − 1}.

B. The limit k → ∞

What happens in the continuum limit, as the number of steps k → ∞? This setting was investigated by [7], who states the limiting path and large deviations rate function. We include this Section to make the story more complete, and to check consistency with and provide a finite-k intuition for the results [7] obtained with variational calculus.

A useful perspective in the following is to map the simplex Δ to the part of the unit sphere in ${ℝ}^{|\mathcal{X}|}$ in the non-negative orthant, $\mathrm{\Psi }\Vert \left\{\mathrm{\psi }\in {ℝ}^{|\mathcal{X}|}:{\Vert \mathrm{\psi }\Vert }_2=1,\mathrm{\psi }⩾0\right\}$ via the bijection $p\to \sqrt{p}\Vert {\left(\sqrt{p\left[x\right]}\right)}_{x\in \mathcal{X}}$. This square root reparametrization appears in [10], [11], [12], [13]. Consider the K-L divergence between nearby points on the orthant Ψ. That is, if p = q + ε ∈ Δ, $\epsilon \in {ℝ}^{|\mathcal{X}|}$, then $\exists {\delta }_{\epsilon }\in {ℝ}^{|\mathcal{X}|}$ such that $\sqrt{p}=\sqrt{q}+{\delta }_{\epsilon }\in \mathrm{\Psi }$ where ${\delta }_{\epsilon }\left[x\right]=\frac{1}{2}\epsilon \left[x\right]p{\left[x\right]}^{-1/2}+O\left(\epsilon {\left[x\right]}^{2}p{\left[x\right]}^{-3/2}\right)$. Then

$$D\left(p\Vert q\right)=\stackrel{2\Vert \sqrt{p}-\sqrt{p}\Vert \begin{array}{c}2\\ 2\end{array}}{\overbrace{4\left(1-\underset{\cos \theta }{\underbrace{\sqrt{p}\cdot \sqrt{p}}}\right)}}+{\displaystyle \sum _{x\in \mathcal{X}}O\left(\frac{{\delta }_{\epsilon }{\left[x\right]}^3}{\sqrt{q\left[x\right]}}\right)}$$ (27)

$$=2{\theta }^2+O\left({\theta }^4\right)+{\displaystyle \sum _{x\in \mathcal{X}}O\left(\frac{{\theta }^3}{\sqrt{q\left[x\right]}}\right)}$$ (28)

where in (27) we rewrote the higher order terms in terms of ε, expanded the cos about θ = 0, and where

$$\theta ={\theta }_{\sqrt{p},\sqrt{q}}\Vert \mathrm{across}\left(\sqrt{p}\cdot \sqrt{q}\right)$$ (29)

is the angle between vectors $\sqrt{p}$ and $\sqrt{q}$ and $\sqrt{p}\cdot \sqrt{q}=B\left(p,q\right)$ is the Bhattacharyya coefficient between p and q.

The above recapitulates the familiar fact that the K-L divergence is close to the squared Euclidean distance between nearby distributions [13] (here written in square root space); the leading order term is symmetric in p and q and depends on p and q only through the angle ${\theta }_{\sqrt{p},\sqrt{q}}$. This enables us to guess that the limiting optimal path is a geodesic in the Euclidean metric restricted to the surface of the unit sphere and that the angle between adjacent points ${\theta }_{\sqrt{w_i^{\left(k\right)\ast }},\sqrt{w_{i-1}^{\left(k\right)\ast }}}\to \frac{1}{k}{\theta }_{\sqrt{p},\sqrt{q}}$. Theorem 3 confirms this intuition. See Appendix VI-C [8] for proof details.

Theorem 3

Let D(p‖q), D^(k)(p‖q) be as in (2), (14), respectively. Let p, q ∈ Δ and D(p‖q) < ∞ and D(q‖p) < ∞.

1. Scaling of D^(k)(p‖q) in $k:D\left(p\Vert q\right)=O\left(\frac{1}{k}\right)$ and

   $$\underset{k\to \infty }{\lim }k{D}^{\left(k\right)}\left(p\Vert q\right)=\underset{k\to \infty }{\lim }k{D}^{\left(k\right)}\left(q\Vert p\right)=2{\theta }_{\sqrt{p},\sqrt{q}}^2$$ (30)

   where (as in (29)) $\cos {\theta }_{\sqrt{p},\sqrt{q}}=\sqrt{p}\cdot \sqrt{q}=B\left(p,q\right)$.
2. The limiting path $\left(w_i^{\left(k\right)\ast }\right)$ (15) as k → ∞ is a geodesic in the Euclidean metric restricted to the unit sphere (equivalently, a geodesic in the Fisher information metric on the simplex): part of a great circle connecting $\sqrt{p}$ and $\sqrt{q}$ with constant angular speed. Let t ∈ [0,1], then

   $$w_{\lfloor tk\rfloor }^{\left(k\right)\ast }\to \frac{1}{Z_{{\tau }_t}}{\left({\tau }_t\sqrt{p}+\left(1-{\tau }_t\right)\sqrt{q}\right)}^2$$ (31)

   in L² norm as k → ∞, where ⎿x⏌ denotes the floor function, $Z_{{\tau }_t}$ is a constant¹⁰, and τ_t : [0,1] → [0,1] is a reparametrization of “time” t that ensures constant angular speed¹¹ on the unit sphere.

Remarks

The condition D(p‖q) < ∞ and D(q‖p) < ∞ ensures that p[x] > 0 ⇔ q[x] > 0, so the series expansion (27) does not diverge and the limits in (30) match. Figure 2 depicts the limiting optimal path for a particular choice of p and q. For a finite number of steps k, the limiting value of the quantity $w_{{\tau }_{i/k}}^{\left(k\right)\ast }$ (31) provides a decent initial guess for the iterative computation of $w_i^{\left(k\right)\ast }$ described in the remarks following the statement of Theorem 2.

V. Applications and further directions

We conclude by offering an application of the chained K-L divergences in maximum likelihood inference and an interpretation in terms of a thought experiment in thermodynamics.

A. ML inference and mutual information

The k-step K-L divergence D^(k)(p‖E) = inf_q∈E D^(k)(p‖q) “from” a set of distributions E has a maximum likelihood interpretation: q* = min_q∈E D^(k)(p‖q) maximizes the likelihood to draw empirical distribution p ∈ Δ_n in k steps of the Wright-Fisher process with initial distribution q ∈ E. A special case of this is a k-step generalization of the mutual information I(X; Y) between random variables X and Y with finite alphabets $\mathcal{X}$, $\mathcal{Y}$, respectively, jointly distributed as $p_{XY}\in {\mathrm{\Delta }}^{\mathcal{X}\times \mathcal{Y}}$:

$$I^{\left(k\right)}\left(X;Y\right)\Vert \underset{q_X,q_Y}{min}{D}^{\left(k\right)}\left(p_{XY}║q_X{q}_Y\right)$$ (32)

with the minimum attained by the k-step “marginals” $\left(q_X^{\left(k\right)\ast },q_Y^{\left(k\right)\ast }\right)\Vert \arg {min}_{q_X,q_Y}{D}^{\left(k\right)}\left(p_{XY}║q_X{q}_Y\right)$. We can check that the minimum exists and is unique¹². For k = 1, we have I⁽¹⁾ (X; Y) = I(X; Y) and $\left(q_X^{\left(1\right)\ast },q_Y^{\left(1\right)\ast }\right)=\left(p_X,p_Y\right)$ (the marginals of p_XY), but for k > 1 this is not the case. I^(k) is monotonically decreasing in k.

Written in the minimization form (32), computing the mutual information I(X; Y) corresponds to finding the maximum likelihood distribution under which X and Y are independent given data with empirical distribution p_XY. In the context of the Wright-Fisher process, suppose we assume that two genetic loci X and Y are independently distributed k generations ago, but are no longer independent due to the intervening neutral genetic drift; then the maximum likelihood ancenstral distribution is $q_X^{\left(k\right)\ast }{q}_Y^{\left(k\right)\ast }$ given the current distribution p_XY. After enough time, all but one pair (x, y) of alleles fixes and the two loci become indepedent again, but neutral drift induces dependence before fixation occurs.

A related object we can construct with the chained K-L divergence is D^(k)(p_XY‖p_Xp_Y); this, like (32), matches the mutual information I(X; Y) for k = 1. The authors do not yet give an operational meaning for this quantity.

B. Maxwell’s demon

Maxwell’s demon is a thought experiment in thermodynamics, here envisioned as a model of a desalination plant: let distribution q correspond to the ambient relative concentrations of some chemicals in the “ocean” (so $\mathcal{X}=\left\{\text{sodium}\text{ion},\text{water},\text{potassium}\text{ion},\dots \right\}$). The demon’s goal is to achieve a desired concentration of chemicals in the water supply; let E denote the set of concentrations the demon considers acceptable. The demon operates by admitting n molecules drawn from the ambient distribution q into a vesicle (temporary storage); if it so happens that p − the empirical distribution of the n molecules − satisfies p ∈ E, then the demon releases the contents of the vesicle into the water supply, and otherwise the demon dumps the contents back in the ocean. See Figure 3.

Fig. 3.

(left) Maxwell’s demon stands ready to release particles into the water supply below if the distribution of a random sample of n happens to be in E. (right) Two demons use an intermediate stage E′ to desalinate faster.

How quickly does the demon desalinate? From the preceding discussion we know that the demon releases the n molecules into the water supply with probability about e^{−nD (E‖q)}. Suppose we now break up the desalination process into two demons, so there is an intermediate stage between the ocean and the water supply. If either of the two demons rejects, then both reject¹³. Then the optimal 2-demon desalination rate is $e^{-n{D}^{\left(2\right)}}\left(E║q\right)$¹⁴, which is about twice as large as the 1-demon rate. Adding more stages increases the rate further; in the k → ∞ limit, we have a concentration gradient across the k stages with no large “jumps.” This is not how actual desalinators work, but the design principle of breaking up a large concentration difference into many small stages is used in multi-stage flash desalination and in the loop of Henle in the kidney.

VI. Appendix

A. Proof of Theorem 1

1) (Equivalence of recursive and non-recursive definitions of the chained divergence)

We proceed by induction on k. For the base case k = 2 the two definitions (13) and (14) are the same. Now suppose the definitions are equivalent up to k − 1 (abusing notation). Then we expand out the min:

$$D^{(k)}(p\Vert q)\Vert \underset{w_{k-1}}{min}\left(D(p\Vert w_{k-1})+D^{(k-1)}(w\Vert q)\right)$$ (33)

$$=\underset{w_{k-1}}{min}\left(D(p\Vert w_{k-1})+\underset{w_1,\dots ,w_{k-2}}{min}{\displaystyle \sum _{i=1}^{k-1}D(w_i\Vert w_{i-1})}\right)$$ (34)

$$\underset{w_1,\dots ,w_{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}$$ (35)

where the second line follows from the inductive hypothesis. □

Existence and uniquness of the path follows from the joint strict convexity of D^(k)(p‖q) (Theorem 1.2)).

2) (Joint convexity of D^(k)(·‖·))

We use the following result [9] (Section 3.2.5): Suppose function f (x, y, z) is jointly convex in x, y, z ∈ C, where C is a nonempty convex subset of ℝ^d. Then

$$g(x,y)\Vert \underset{z\in C}{min}f(x,y,z)$$ (36)

is jointly convex in x and y.

We proceed by induction on k. For the base case k = 1, D⁽¹⁾(p‖q) = D(p‖q) is jointly strictly convex in p, q ∈ Δ. Now assume the statement holds up to k − 1. Then D(p‖w) + D^{(k − 1)}(w‖q) is jointly strictly convex in p, q,w ∈ Δ and

$$D^{(k)}(p\Vert q)\Vert \underset{w\in \mathrm{\Delta }}{min}\left(D(p\Vert w)+D^{(k-1)}(w\Vert q)\right)$$ (37)

is jointly strictly convex in p, q by the above lemma (36) with g(p, q) = D^(k)(p‖q) and f(p, q, w) = D(p‖w) + D^{(k − 1)}(w‖q).

Since D^(k)(p‖q) is the minimum of a strictly convex function over a closed convex set Δ, there is a unique global minimum $(w_i^{(k)\ast })$ for 1 ≤ i ≤ k − 1. □

3) (Monotonicity of D^(k)(p‖q) in k)

Consider the optimal k-step path ( $q=w_0^{(k)\ast }$, $w_1^{(k)\ast },\dots ,w_{k-1}^{(k)\ast }$, $p=w_k^{(k)\ast }$) (15) and modify it by replacing $w_{k-1}^{(k)\ast }$ with p:

$$D^{(k)}(p\Vert q)={\displaystyle \sum _{i=1}^{k}D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })}$$ (38)

$$\stackrel{(\mathrm{a})}{<}D(p\Vert p)+{\displaystyle \sum _{i=1}^{k-1}D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })}$$ (39)

$$\stackrel{(\mathrm{b})}{\le }{D}^{(k-1)}(p\Vert q)$$ (40)

(a) follows because we know from Theorem 2 that $w_{k-1}^{(k)\ast }\ne p$ and since we showed above that the optimal path $(w_i^{(k)\ast })$ is unique. (b) follows since in general $w_i^{(k-1)\ast }\ne w_i^{(k)\ast }$. □

4) (Bounding $P_n^{(k)}(p|q)$)

First the upper bound (18). We proceed by induction on k. For k = 1, the statement holds (see [1] Theorem 11.1.4). Now assume the statement holds up to k − 1. Now using (9) we write

$$P_n^{(k)}(p|q)={\displaystyle \sum _{w\in {\mathrm{\Delta }}_n}{P}_n(p|w)P_n^{(k-1)}(w|q)}$$ (41)

$$\stackrel{(\mathrm{a})}{\le }{(n+1)}^{(k-2)|\mathcal{X}|}{\displaystyle \sum _{w\in {\mathrm{\Delta }}_n}{e}^{-n\left(D(p\Vert w)+D^{(k-1)}(w\Vert q)\right)}}$$ (42)

$$\le {(n+1)}^{(k-2)|\mathcal{X}|}|{\mathrm{\Delta }}_n|\underset{w\in {\mathrm{\Delta }}_n}{max}{e}^{-n\left(D(p\Vert w)+D^{(k-1)}(w\Vert q)\right)}$$ (43)

$$\stackrel{(\mathrm{b})}{\le }{(n+1)}^{(k-1)|\mathcal{X}|}\underset{w\in \mathrm{\Delta }}{max}{e}^{-n\left(D(p\Vert w)+D^{(k-1)}(w\Vert q)\right)}$$ (44)

$$={(n+1)}^{(k-1)|\mathcal{X}|}{e}^{-n{min}_{w\in \mathrm{\Delta }}\left(D(p\Vert w)+D^{(k-1)}(w\Vert q)\right)}$$ (45)

$$\stackrel{(\mathrm{c})}{=}{(n+1)}^{(k-1)|\mathcal{X}|}{e}^{-n{D}^{(k)}(p\Vert q)}$$ (46)

where (a) follows from the inductive hypothesis, (b) follows from $|{\mathrm{\Delta }}_n|\le {(n+1)}^{|\mathcal{X}|}$ (see [1] Theorem 11.1.1) and from optimizing over Δ rather than Δ_n ⊂ Δ, and (c) follows from the recursive definition of the chained K-L divergence (11).

Next we find an asymptotic (in n ⟶ ∞) lower bound. Let (p_n)_n∈ℕ, p_n ∈ Δ_n, lim_{n ⟶ ∞} D(p_n‖p) = 0. Choose an n-indexed sequence ${\left(v_{n,i}\right)}_{i=1}^{k-1}$ such that v_n,i ∈ Δ_n, $D(v_{n,i}\Vert w_i^{\ast })\to 0$ as n ⟶ ∞. Such a sequence exists because ∪_nΔ_n is dense in Δ. Denote v_n,k ≜ p_n and $w_k^{\ast }\Vert p$. Denote

$${\epsilon }_{n,i}\Vert v_{n,i}-w_i^{\ast }$$ (47)

By Pinsker’s inequality

$$\underset{n\to \infty }{\lim }|{\epsilon }_{n,i}[x]|=0$$ (48)

for all i ∈ {1,…, k}, $x\in \mathcal{X}$. Moreover, since ${\lim }_{n\to \infty }D(v_{n,i}\Vert w_i^{\ast })=0$ by assumption, then for large enough n we have $\text{Support}(v_{n,i})=\text{Support}(w_i^{\ast })$, so for large enough n

$$\text{Support}({\epsilon }_{n,i})\subset \text{Support}(w_i^{\ast })$$ (49)

Therefore for i ∈ {1,…, k}

$$\underset{n\to \infty }{\lim }D(v_{n,i}\Vert v_{n,i-1})=\underset{n\to \infty }{\lim }{\displaystyle \sum _{x\in \mathcal{X}}(w_i^{\ast }[x]+{\epsilon }_{n,i}[x])\log \left(\frac{w_i^{\ast }[x]+{\epsilon }_{n,i}[x]}{w_{i-1}^{\ast }[x]+{\epsilon }_{n,i-1}[x]}\right)}$$ (50)

$$={\displaystyle \sum _{x\in \mathcal{X}}{w}_i^{\ast }[x]\log \left(\frac{w_i^{\ast }[x]}{w_{i-1}^{\ast }[x]}\right)}$$ (51)

$$=D(w_i^{\ast }\Vert w_{i-1}^{\ast })$$ (52)

where the second equality follows from (48) and (49) and ε_n, 0[x] ≜ 0.

Using (9) we write

$$P_n^{(k)}(p_n|q)={\displaystyle \sum _{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{\displaystyle \prod _{i=1}^{k}P(w_i|w_{i-1})}}$$ (53)

$$\stackrel{(\mathrm{a})}{⩾}\frac{1}{{(n+1)}^{k|\mathcal{X}|}}{\displaystyle \sum _{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{e}^{-n{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}}$$ (54)

$$⩾\frac{1}{{(n+1)}^{k|\mathcal{X}|}}{e}^{-n{min}_{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}$$ (55)

with w₀ = q, w_k = p_n, (w_i) = (w₁,…,w_k−1), where (a) follows from the lower bound $P_n(p|q)⩾e^{-nD(p\Vert q)}/{(n+1)}^{|\mathcal{X}|}$ (see [1], Theorem 11.1.4). Therefore

$$\underset{n\to \infty }{\lim \inf }\frac{1}{n}\log P_n^{(k)}(p_n|q)⩾\underset{n\to \infty }{\lim \inf }\left(-\frac{k|\mathcal{X}|\log (n+1)}{n}-\underset{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}\right)$$ (56)

$$=\underset{n\to \infty }{\lim \inf }\left(-\underset{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}\right)$$ (57)

$$⩾\underset{n\to \infty }{\lim \inf }\left(-{\displaystyle \sum _{i=1}^{k}D(v_{n,i}\Vert v_{n,i-1})}\right)$$ (58)

$$\stackrel{(\mathrm{a})}{=}-{\displaystyle \sum _{i=1}^{k}D(w_i^{\ast }\Vert w_{i-1}^{\ast })}$$ (59)

$$=-D^{(k)}(p\Vert q)$$ (60)

where (a) follows by (52). Recalling the upper bound (18) we conclude

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{(k)}(p_n|q)=-D^{(k)}(p\Vert q)$$ (61)

The first equality in (19) follows by letting v_n,k = p_n= p, ε_n,k[x] = 0 for all n, $x\in \mathcal{X}$, so ${\lim }_{n\to \infty }D(p\Vert v_{n,k-1})=D(p\Vert w_{k-1}^{\ast })$, so (59) holds, and

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{(k)}(p|q)=\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{(k)}(p_n|q)=-D^{(k)}(p\Vert q)$$ (62)

Equivalently, $P_n^{(k)}(p|q)\doteq e^{-n{D}^{(k)}(p\Vert q)}$. □

5) (Sanov’s theorem)

This proof is very similar to the previous proof for the case E = {p} and to the proof of Sanov’s theorem in [1].

First the upper bound (20). Using (7) and (10) we write

$$P_n^{(k)}(E|q)={\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{\displaystyle \sum _{w\in {\mathrm{\Delta }}_n}{P}_n(p|w)P_n^{(k-1)}}(w|q)}$$ (63)

$$\stackrel{(\mathrm{a})}{\le }{(n+1)}^{(k-1)|\mathcal{X}|}{\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{e}^{-n{D}^{(k)}(p\Vert q)}}$$ (64)

$$\le {(n+1)}^{(k-1)|\mathcal{X}|}{\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{e}^{-n{\inf }_{p\in E}{D}^{(k)}(p\Vert q)}}$$ (65)

$$={(n+1)}^{(k-1)|\mathcal{X}|}{\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{e}^{-n{D}^{(k)}(E\Vert q)}}$$ (66)

$$\stackrel{(\mathrm{c})}{\le }{(n+1)}^{k|\mathcal{X}|}{e}^{-n{D}^{(k)}(E\Vert q)}$$ (67)

where (a) follows from the upper bound (18) and (b) follows from $|E\cap {\mathrm{\Delta }}_n|\le |{\mathrm{\Delta }}_n|\le {(n+1)}^{|\mathcal{X}|}$.

By assumption E is closed and convex, so

$$p^{(k)\ast }=\arg \underset{p\in E}{min}{D}^{(k)}(p\Vert q)$$ (68)

is the k-step I-projection of q on E (12). We find an asymptotic (in n ⟶ ∞) lower bound. Using (7) and (10) we write

$$P_n^{(k)}(E|q)={\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{\displaystyle \sum _{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{\displaystyle \prod _{i=1}^{k}P(w_i|w_{i-1})}}}$$ (69)

$$\stackrel{(\mathrm{a})}{⩾}\frac{1}{{(n+1)}^{k|\mathcal{X}|}}{\displaystyle \sum _{p\in E\cap {\mathrm{\Delta }}_n}{\displaystyle \sum _{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{e}^{-n{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}}}$$ (70)

$$⩾\frac{1}{{(n+1)}^{k|\mathcal{X}|}}{e}^{-n{min}_{p\in E\cap {\mathrm{\Delta }}_n,(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}$$ (71)

with w₀ = q, w_k = p, (w_i) = (w₁,…,w_k−1), where (a) follows from the lower bound $P_n(p|q)⩾e^{-nD(p\Vert q)}/{(n+1)}^{|\mathcal{X}|}$ (see [1], Theorem 11.1.4).

By assumption E ∩ Δ_n is non-empty for sufficiently large n, so the lower bound (71) is not vacuous. We can then find a sequence (p_n)_n, p_n ∈ E ∩ Δ_n and lim_n⟶∞ D(p_n‖p^(k)*) = 0 and an n-indexed sequence ${(v_{n,i})}_{i=1}^{k-1}$ such that v_n,i ∈ Δ_n, $D(v_{n,i}\Vert w_i^{\ast })\to 0$ as n ⟶ ∞. Such a sequence exists because ∪_nΔ_n is dense in Δ. Recycling the argument in (52) we conclude

$$\underset{n\to \infty }{\lim }D(v_{n,i}\Vert v_{n,i-1})=D(w_i^{\ast }\Vert w_{i-1}^{\ast })$$ (72)

where $w_k^{\ast }\Vert p^{(k)\ast }$. Then

$$\underset{n\to \infty }{\lim \inf }\frac{1}{n}\log P_n^{(k)}(E|q)⩾\underset{n\to \infty }{\lim \inf }\left(-\frac{k|\mathcal{X}|\log (n+1)}{n}-\underset{p\in E\cap {\mathrm{\Delta }}_n}{min}\underset{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}\right)$$ (73)

$$=\underset{n\to \infty }{\lim \inf }\left(-\underset{p\in E\cap {\mathrm{\Delta }}_n}{min}\underset{(w_i)\in {\mathrm{\Delta }}_n^{k-1}}{min}{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}\right)$$ (74)

$$⩾\underset{n\to \infty }{\lim \inf }\left(-D(p_n||v_{n,k-1})-{\displaystyle \sum _{i=1}^{k-1}D(v_{n,i}\Vert v_{n,i-1})}\right)$$ (75)

$$\stackrel{(\mathrm{a})}{=}-{\displaystyle \sum _{i=1}^{k}D(w_i^{\ast }\Vert w_{i-1}^{\ast })}$$ (76)

$$=-D^{(k)}(E\Vert q)$$ (77)

where (a) follows by (72).

Recalling the upper bound (20) we conclude

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log P_n^{(k)}(E|q)=D^{(k)}(p^{(k)\ast }\Vert q)$$ (78)

Equivalently, $P_n^{(k)}(E|q)\doteq e^{-n{D}^{(k)}(E\Vert q)}$. □

6) (Conditional limit theorem)

We follow closely the proof for the conditional limit theorem in [1]. Let $(w_1^{\ast },\dots ,w_{k-1}^{\ast })=\arg {min}_{(w_i)\in {\mathrm{\Delta }}^{k-1}}D(p^{(k)\ast }\Vert q)$ be the optimal path (15) and p^(k)* = argmin_p∈E D^(k)(p‖q) (12) be the k-step I-projection of q on E (p^(k)* exists and is unique since E is closed and convex by assumption).

Next define the set

$$S_t\Vert \{(w_1,\dots ,w_{k-1},p)\in {\mathrm{\Delta }}^k:{\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}\le t\}$$ (79)

Let

$$D\ast ={\displaystyle \sum _{i=1}^k{D}_i^{\ast }}=D^{(k)}(E\Vert q)$$ (80)

Now define the set

$$A\Vert S_{D\ast +2\delta }\cap ({\mathrm{\Delta }}^{k-1}\times E)$$ (81)

where the last term ensures that ${(w_i)}_{i=1}^k\in A\Rightarrow w_k\in E$. Define the set

$$B\Vert ({\mathrm{\Delta }}^{k-1}\times E)-A$$ (82)

Thus A ∪ B = Δ^k−1 × E. In the summations below, $v=(w_1,\dots ,w_{k-1},w_k)\in {\mathrm{\Delta }}_n^{k-1}\times (E\cap {\mathrm{\Delta }}_n)$. Then

$$P_n^{(k)}(B|q)={\displaystyle \sum _{v:{\displaystyle {\sum }_{i=1}^{k}D(w_i||w_{i-1})>D\ast +2\delta }}{\displaystyle \prod _{i=1}^{k}P(w_i|w_{i-1})}}$$ (83)

$$\stackrel{(\mathrm{a})}{\le }{\displaystyle \sum _{v:{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})>D\ast +2\delta }}{e}^{-n{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}}$$ (84)

$$\le {\displaystyle \sum _{v:{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})>D\ast +2\delta }}{e}^{-n(D\ast +2\delta )}}$$ (85)

$$\stackrel{(\mathrm{b})}{\le }{(n+1)}^{k|\mathcal{X}|}{e}^{-n(D\ast +2\delta )}$$ (86)

where (a) follows from the upper bound (18) in the case k = 1 and (b) follows since $|{\mathrm{\Delta }}_n^{k-1}\times (E_i\cap {\mathrm{\Delta }}_n)|\le |{\mathrm{\Delta }}_n{|}^k\le {(n+1)}^{k|\mathcal{X}|}$.

In the summations below, $v=(w_1,\dots ,w_{k-1},w_k)\in {\mathrm{\Delta }}_n^{k-1}\times (E\cap {\mathrm{\Delta }}_n)$. We write

$$P_n^{(k)}(A|q)\stackrel{(\mathrm{a})}{⩾}{P}_n^{(k)}(A\cap S_{D\ast +\delta }|q)$$ (87)

$$={\displaystyle \sum _{v:{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})\le D\ast +\delta }}{\displaystyle \prod _{i=1}^{k}P(w_i|w_{i-1})}}$$ (88)

$$\stackrel{(\mathrm{b})}{⩾}{\displaystyle \sum _{v:{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})\le D\ast +\delta }}\frac{e^{-n{\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}}}{{(n+1)}^{k|\mathcal{X}|}}}$$ (89)

$$\stackrel{(\mathrm{c})}{⩾}\frac{e^{-n(D\ast +\delta )}}{{(n+1)}^{k|\mathcal{X}|}}\text{for sufficiently large }n$$ (90)

where in (a) we used $S_{D\ast +\delta }\subset S_{D\ast +2\delta }$, in (b) we used the lower bound $P_n(p|q)⩾e^{-nD(p||q)}/{(n+1)}^{|\mathcal{X}|}$ (see [1], Theorem 11.1.4), and (c) follows since $S_{D\ast +\delta }\cap ({\mathrm{\Delta }}^{k-1}\times E)\cap {\mathrm{\Delta }}_n^k\ne \varnothing$ for n sufficiently large (since E ∩ Δ_n ≠ Ø for n sufficiently large by assumption).

Now recall the random variables ${(X_t^{(i)})}_{t\in \{1,\dots ,n\}}$ defined in the statement of Theorem 1.6) and define the empirical distributions ŵ_i ∈ Δ_n

$${\widehat{w}}_i[x]\Vert \frac{1}{n}|\{t:x_t^{(i)}=x\}|$$ (91)

for all $x\in \mathcal{X}$, i ∈ {1,…, k}. Then for n sufficiently large

$$\mathbb{P}(({\widehat{w}}_1,\dots ,{\widehat{w}}_k)\in B|{\widehat{w}}_k\in E)=\frac{P_n^{(k)}(B\cap ({\mathrm{\Delta }}^{k-1}\times E))}{P_n^{(k)}({\mathrm{\Delta }}^{k-1}\times E)}$$ (92)

$$\le \frac{P_n^{(k)}(B)}{P_n^{(k)}(A)}$$ (93)

$$\stackrel{(\mathrm{a})}{\le }\frac{{(n+1)}^{k|\mathcal{X}|}{e}^{-n(D\ast +2\delta )}}{\frac{e^{-n(D\ast +\delta )}}{{(n+1)}^{k|\mathcal{X}|}}}$$ (94)

$$={(n+1)}^{2k|\mathcal{X}|}{e}^{-n\delta }$$ (95)

where (a) follows from (86) and (90). The above quantity goes to 0 as n ⟶ ∞, so $\mathbb{P}(({\widehat{w}}_1,\dots ,{\widehat{w}}_k)\in A|{\widehat{w}}_k\in E)\to 1$ as n ⟶ ∞.

Next let’s show that elements of A are close to $(w_1^{\ast },\dots ,w_k^{\ast })$ in L¹ norm. This follows from the strict joint convexity of the function

$$\mathrm{\Lambda }({(w_i)}_{i=1}^k)\Vert {\displaystyle \sum _{i=1}^{k}D(w_i\Vert w_{i-1})}$$ (96)

on the compact, convex set Δ^k−1 × E. We state a lemma:

Lemma 4

Let f(x) be jointly strictly convex in x ∈ C, where C is a compact, convex subset of ℝ^d, let x* ≜ argmin_x∈C f(x). For δ > 0 denote the sublevel set

$$T_{\delta }\Vert \{x:f(x)-f(x\ast )\le \delta \}$$ (97)

Then

$$\underset{\delta \to 0}{\lim }\underset{x\in T_{\delta }}{max}{\Vert x-x\ast \Vert }_1=0$$ (98)

Proof

Suppose (98) does not hold. Then there exists ε > 0 such that for all δ > 0

$$\underset{x\in T_{\delta }}{max}{\Vert x-x\ast \Vert }_1⩾\epsilon$$ (99)

Let T′ ≜ {x : ║x − x*║₁ ⩾ ε} and ${\delta }_{\epsilon }\Vert {min}_{x\in T^{\prime }}f(x)>0$ where the min exists by compactness of T′ and the inequality follows since x ∈ T′ ⇒ x ≠ x* and f(x) is jointly strictly convex. Then

$$x\in T_{{\delta }_{\epsilon }/2}\Rightarrow f(x)-f(x\ast )\le {\delta }_{\epsilon }/2<{\delta }_{\epsilon }$$ (100)

$$\Rightarrow x\cancel{\in }{T}^{\prime }$$ (101)

$$\Rightarrow {\Vert x-x\ast \Vert }_1<\epsilon$$ (102)

This contradicts (99), so (98) holds. ■

The function $\mathrm{\Lambda }({(w_i)}_{i=1}^k)$ is jointly strictly convex in ${(w_i)}_{i=1}^k$ with compact, convex support Δ^k−1 × E and attains its minimum at $\mathrm{\Lambda }({(w_i^{\ast })}_{i=1}^k)=D^{(k)}(E\Vert q)$, so by Lemma 4

$$\underset{\delta \to 0}{\lim }\underset{{(w_i^{\ast })}_{i=1}^k\in A}{max}{\Vert {(w_i)}_{i=1}^{\ast }-{(w_i^{\ast })}_{i=1}^k\Vert }_1=0$$ (103)

Now recalling our earlier conclusion that for all δ > 0 $\mathbb{P}(({\widehat{w}}_1,\dots ,{\widehat{w}}_k)\in A|{\widehat{w}}_k\in E)\to 1$ as n ⟶ ∞ and using (103), we conclude that for all ε > 0, all i ∈ {1,…, k}, and all $x\in \mathcal{X}$

$$\underset{n\to \infty }{\lim }\mathbb{P}(|{\widehat{w}}_i[x]-w_i^{\ast }[x]|⩾\epsilon |{\widehat{w}}_k\in E)=0$$ (104)

□

B. Proof of Theorem 2

Proof

The proof of the local optimality condition (24) is given after the statement of this Theorem.

Now let’s prove the geometric lower bound (25). We first state a lemma.

Lemma 5

Let p, q ∈ Δ, D(p‖q) < ∞, w*(p, q) be as in (23), and λ^⋆ the value of the Lagrange multiplier so that w*(p, q) is normalized. Then

$${\mathrm{\lambda }}^{\star }\le 1$$ (105)

Proof

First define the function

$$Z_{\mathrm{\lambda }}(p,q)\Vert {\displaystyle \sum _{x\in \mathcal{X}}q[x]f_{\mathrm{\lambda }}}\left(\frac{p[x]}{q[x]}\right)$$ (106)

where

$$f_{\mathrm{\lambda }}(t)\Vert \{\begin{array}{cc}\frac{-t}{W(t{e}^{\mathrm{\lambda }})}+\frac{t}{W(e^{\mathrm{\lambda }})}& :t>0\\ -e^{-\mathrm{\lambda }}& :t=0\end{array}$$ (107)

We can check that f_λ(·) is convex and f_λ(1) = 0, so Z_λ defines an f-divergence. Now letting λ^⋆ so that w*(p, q) (23) is normalized, we write

$$Z_{{\mathrm{\lambda }}^{\star }}(p,q)={\displaystyle \sum _{x\in \mathcal{X}}q[x]}{f}_{{\mathrm{\lambda }}^{\star }}\left(\frac{p[x]}{q[x]}\right)$$ (108)

$$\stackrel{(\mathrm{a})}{=}{\displaystyle \sum _{x\in \mathcal{X}}\left(-w\ast (p,q)[x]+\frac{p[x]}{W(e^{{\mathrm{\lambda }}^{\star }})}\right)}$$ (109)

$$=-1+\frac{1}{W(e^{{\mathrm{\lambda }}^{\star }})}$$ (110)

$$\begin{array}{l}\stackrel{(\mathrm{b})}{⩾}0\\ \Downarrow \end{array}$$ (111)

$${\mathrm{\lambda }}^{\star }\le 1$$ (112)

where in (a) we substituted w*(p, q) (23) and in (b) we used the non-negativity of f-divergences [ref?]. ■

Next let’s state another lemma.

Lemma 6

Let p, q ∈ Δ, D(p‖q) < ∞ and w*(p, q) be as in (23). Then

$$w\ast (p,q)[x]⩾\sqrt{p[x]q[x]}$$ (113)

for all $x\in \mathcal{X}$.

Proof

Suppose there exists some $x\in \mathcal{X}$ such that (113) does not hold, so $w\ast (p,q)[x]<\sqrt{p[x]q[x]}$. This implies p[x], q[x] > 0, so

$$\begin{array}{c}w\ast (p,q)[x]=\frac{p[x]}{W\left(e^{{\mathrm{\lambda }}^{\star }}\frac{p[x]}{q[x]}\right)}<\sqrt{p[x]q[x]}\\ \Downarrow \end{array}$$ (114)

$${\mathrm{\lambda }}^{\star }>\sqrt{\frac{p[x]}{q[x]}}-\log \left(\frac{p[x]}{q[x]}\right)⩾1$$ (115)

This contradicts Lemma 5, so the statement holds. ■

Lemma 6 implies

$$\log (w_i^{(k)\ast }[x])=\log (w\ast (w_{i+1}^{(k)\ast },w_{i-1}^{(k)\ast })[x])$$ (116)

$$⩾\frac{1}{2}\log (w_{i+1}^{(k)\ast }[x])+\frac{1}{2}\log (w_{i-1}^{(k)\ast }[x])$$ (117)

Thus $\log w_i^{(k)\ast }[x]$ is concave in i, so

$$\log (w_i^{(k)\ast }[x])⩾\frac{i}{k}\log (w_k^{(k)\ast }[x])+\frac{k-i}{k}\log (w_0^{(k)\ast }[x])$$ (118)

$$=\frac{i}{k}\log (p[x])+\frac{k-i}{k}\log (q[x])$$ (119)

so (25) follows. ■

C. Proof of Theorem 3

Proof

Let v_t denote the continuous path (31)

$$v_t\Vert \frac{1}{Z_{{\tau }_t}}{({\tau }_t\sqrt{p}+(1-{\tau }_t)\sqrt{q})}^2$$ (120)

where $Z_{{\tau }_t}$ and ${\tau }_t$ are as in the statement of Theorem 3. Let $v_i^{(k)}$ denote the continuous path (31) sampled at k uniformly spaced points in time:

$${(v_i^{(k)})}_{i=0}^k\Vert {\left(\frac{1}{Z_{{\tau }_t}}{({\tau }_t\sqrt{p}+(1-{\tau }_t)\sqrt{q})}^2\right)}_{t\in \{0,\frac{1}{k},\frac{2}{k},\dots ,1\}}$$ (121)

Let ${(w_i^{(k)\ast })}_{i=0}^k$ denote the optimal k-step path (15). Then $v_0^{(k)}=w_0^{(k)\ast }=q$ and $v_k^{(k)}=w_k^{(k)\ast }=p$. For 1 ≤ i ≤ k denote by ${\theta }_i^{(k)}$ the angle between (the square roots of) adjacent distributions on the path ${(v_i^{(k)})}_{i=0}^k$

$${\varphi }_i^{(k)}\Vert \arccos \left(\sqrt{v_i^{(k)}·\sqrt{v_{i-1}^{(k)}}}\right)=\frac{1}{k}{\theta }_{\sqrt{p},\sqrt{q}}$$ (122)

$${\theta }_i^{(k)}\Vert \arccos \left(\sqrt{w_i^{(k)\ast }·\sqrt{w_{i-1}^{(k)\ast }}}\right)$$ (123)

Next let

$$\mu \Vert \underset{x\in \text{Support}(q)}{min}min(p[x],q[x])>0$$ (124)

We have μ > 0 since D(p‖q) < ∞ and D(q‖p) < ∞, so Support(p) = Support(q) by assumption.

Now apply the series expansion for the K-L divergence (27) to ${(v_i^{(k)})}_{i=0}^k$ and ${(w_i^{(k)\ast })}_{i=0}^k$ and sum over i ∈ {1,…, k}:

$${\displaystyle \sum _{i=1}^{k}D(v_i^{(k)}\Vert v_{i-1}^{(k)})}={\displaystyle \sum _{i=1}^{k}2\left({\varphi }_i^{(k)2}+O({\varphi }_i^{(k)4})+{\displaystyle \sum _{x\in \mathcal{X}}O\left(\frac{{\varphi }_i^{(k)3}}{\sqrt{v_i^{(k)}}}\right)}\right)}$$ (125)

$$={\displaystyle \sum _{i=1}^k\left(2{\varphi }_i^{(k)2}+O({\varphi }_i^{(k)3})\right)}$$ (126)

where the second equality comes from the i- and k- independent lower bound

$$\underset{x\in \mathcal{X}}{min}\underset{t\in [0,1]}{min}{v}_t[x]⩾\mu \Vert \underset{x\in \mathcal{X}}{min}min(p[x],q[x])>0$$ (127)

Since D(p‖q) < ∞ and D(q‖p) < ∞ by assumption, we may assume p[x], q[x] > 0 for all $x\in \mathcal{X}$ (since otherwise we could restrict attention to $\text{Support}(p)=\text{Support}(q)\subset \mathcal{X}$). Thus the last inequality in (127) follows. Similarly we obtain

$${\displaystyle \sum _{i=1}^{k}D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })}={\displaystyle \sum _{i=1}^{k}2\left({\theta }_i^{(k)2}+O({\theta }_i^{(k)4})+{\displaystyle \sum _{x\in \mathcal{X}}O}\left(\frac{{\theta }_i^{(k)3}}{\sqrt{w_i^{(k)\ast }}}\right)\right)}$$ (128)

$$={\displaystyle \sum _{i=1}^k\left(2{\theta }_i^{(k)2}+O({\theta }_i^{(k)3})\right)}$$ (129)

where the last inequality follows from the i- and k-independent lower bound

$$\underset{x\in \mathcal{X}}{min}\underset{i\in \{0,\dots ,k\}}{min}{w}_i^{(k)\ast }⩾\underset{x\in \mathcal{X}}{min}p{[x]}^{i/k}q{[x]}^{(k-i)/k}⩾\mu >0$$ (130)

where the first inequality follows from Theorem 2, (25).

Now we write

$$\underset{k\to \infty }{\lim }k{\displaystyle \sum _{i=1}^k\left(D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })-D(v_i^{(k)}\Vert v_{i-1}^{(k)})\right)}=\underset{k\to \infty }{\lim }2k{\displaystyle \sum _{i=1}^k\left({\theta }_i^{(k)2}-{\varphi }_i^{(k)2}\right)}$$ (131)

$$=\underset{k\to \infty }{\lim }2k{\displaystyle \sum _{i=1}^k\left({\theta }_i^{(k)2}-\frac{{\theta }_{\sqrt{p},\sqrt{q}}^2}{k^2}\right)}$$ (132)

$$⩾0$$ (133)

where the inequality follows since the path v_t is part of a great circle joining $\sqrt{p}$ and $\sqrt{q}$ at constant angular speed (uniformly spacing the angles ${\varphi }_i^{(k)}=\frac{1}{k}{\theta }_{\sqrt{p},\sqrt{q}}$ minimizes the sum of their squares subject to their sum ${\varphi }_i^{(k)}={\theta }_{\sqrt{p},\sqrt{q}}$). On the other hand by the optimality of ${(w_i^{\ast })}_{i=1}^k$

$$\underset{k\to \infty }{\lim }k{\displaystyle \sum _{i=1}^k\left(D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })-D(v_i^{(k)}\Vert v_{i-1}^{(k)})\right)}\le 0$$ (134)

Thus, combining the above inequalities,

$$\underset{k\to \infty }{\lim }k{\displaystyle \sum _{i=1}^k\left(D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })-D(v_i^{(k)}\Vert v_{i-1}^{(k)})\right)}=0$$ (135)

We then have

$${\displaystyle \sum _{i=1}^k\left(D(w_i^{(k)\ast }\Vert w_{i-1}^{(k)\ast })-D(v_i^{(k)}\Vert v_{i-1}^{(k)})\right)}={\displaystyle \sum _{i=1}^k(O({\theta }_i^{(k)3})+O({\varphi }_i^3))}$$ (136)

$$=kO(1/k^3)=O(1/k^2)$$ (137)

[why θ^(k) = O(1/k)?]

[Now use joint strict convexity of ${\displaystyle {\sum }_{i=1}^{k}D(w_i\Vert w_{i-1})}$ on the compact, convex set Δ^k−1 × E to complete proof?] ■