The Generalized Asymptotic Equipartition Property: Necessary and Sufficient Conditions

Abstract

Suppose a string $X_1^n=\left(X_1,X_2,\dots ,X_n\right)$ generated by a memoryless source (X_n)_n≥1 with distribution P is to be compressed with distortion no greater than D ≥ 0, using a memoryless random codebook with distribution Q. The compression performance is determined by the “generalized asymptotic equipartition property” (AEP), which states that the probability of finding a D-close match between $X_1^n$ and any given codeword $Y_1^n$, is approximately 2^{−nR(P, Q, D)}, where the rate function R(P, Q, D) can be expressed as an infimum of relative entropies. The main purpose here is to remove various restrictive assumptions on the validity of this result that have appeared in the recent literature. Necessary and sufficient conditions for the generalized AEP are provided in the general setting of abstract alphabets and unbounded distortion measures. All possible distortion levels D ≥ 0 are considered; the source (X_n)_n≥1 can be stationary and ergodic; and the codebook distribution can have memory. Moreover, the behavior of the matching probability is precisely characterized, even when the generalized AEP is not valid. Natural characterizations of the rate function R(P, Q, D) are established under equally general conditions.

I. Introduction

Suppose a random string $X_1^n=\left(X_1,X_2,\dots ,X_n\right)$ produced by a memoryless source (X_n : n ≥ 1) with distribution P on a source alphabet S, is to be compressed with distortion no more than some D ≥ 0 with respect to a single-letter distortion measure ρ(x, y).¹ The basic information-theoretic model for understanding the best performance that can be achieved is the study of random codebooks. If we generate memoryless random strings $Y_1^n=\left(Y_1,Y_2,\dots ,Y_n\right)$ according to some distribution Q on the reproduction alphabet T, we would like to know how many such strings are needed so that, with high probability, we will be able to find at least one codeword $Y_1^n$ that matches the source string $X_1^n$ with distortion D or less. The crucial mathematical problem in answering this question is the evaluation of the probability that a given, typical $X_1^n$, will be D-close to a random $Y_1^n$. This probability can be expressed as

$$\mathrm{Prob}\left\{Y_1^n\in B_n\left(X_1^n,D\right)|X_1^n\right\}=Q^n\left(B_n\left(X_1^n,D\right)\right)$$ (1)

where $B_n\left(X_1^n,D\right)$ denotes the “distortion ball” consisting of all reproduction strings that are within distortion D (or less) from $X_1^n$; note that the matching probability in (1) is itself a random quantity, as it depends on the source string $X_1^n$.

The importance of evaluating (1) was already identified by Shannon in his classic study of rate-distortion theory [4], where he showed that, for the best codebook distribution Q = Q_*, we have

$$Q_{\ast }^n\left(B_n\left(X_1^n,D\right)\right)\approx 2^{-nR\left(P,D\right)}$$ (2)

where R(P, D) is the rate-distortion function of the source.

The more general question of evaluating the matching probability (1) for distributions Q perhaps different from the optimal reproduction distribution Q_*, arises naturally in a variety of contexts, including problems in pattern-matching, mismatched codebooks, Lempel-Ziv compression, combinatorial optimization on random strings, and others; see, e.g., [5]-[13], and the review and references in [14]. In this case, Shannon’s estimate (2) is replaced by the so-called “generalized asymptotic equipartition property” (or generalized AEP), which states that

$$-n^{-1}\log Q^n\left(B_n\left(X_1^n,D\right)\right)\to R\left(P,Q,D\right)\mathrm{a.s.}$$ (3)

where “a.s.” stands for “almost surely” and refers to the random string $X_1^n$. The rate function R(P, Q, D) is defined in a way that closely resembles the rate-distortion function definition,

$$R\left(P,Q,D\right):=\underset{W}{\inf }H\left(W\parallel P\times Q\right)$$

where H(·∥·) denotes the relative entropy, and the infimum is over all (bivariate) probability distributions of random variables (U, V) with values on S and T, respectively, such that U has distribution P and the expected distortion E[ρ(U, V)] ≤ D. (For a broad introduction to the generalized AEP, its applications and refinements, see [14] and the references therein.)

Although much is known about the generalized AEP and about R(P, Q, D) [14], all known results are established under certain restrictive conditions. In particular, it is always assumed that D_ave(P, Q) < ∞ and that D ≠ D_min(P, Q), where

$$\begin{array}{l}{D}_{\mathrm{ave}}\left(P,Q\right):=E_{\left(X,Y\right)\sim P\times Q}\rho \left(X,Y\right)\\ D_{\min }\left(P,Q\right):=\inf \left\{D:R\left(P,Q,D\right)<\infty \right\}.\end{array}$$

Often, the codebook distribution is required to be memoryless, and when it is not, it is further assumed that the distortion measure is bounded.

The main point of this paper is to remove these constraints, and to analyze which (if any) are essential for the validity of the generalized AEP. Our motivation is twofold. On one hand, unnecessarily stringent conditions make the theoretical picture incomplete. On the other, there are applications which naturally require more general statements, such as universal lossy compression, where the source distribution is not known a priori [3].

Thus motivated, we give necessary and sufficient conditions for the generalized AEP in (3), and we precisely characterize the behavior of the matching probability in the pathological situations when the generalized AEP fails. Our results hold for all values of D, and they cover arbitrary abstract alphabets and distortion measures. We also allow the source to be stationary and ergodic, and the codebook distribution to have a certain amount of memory. We similarly extend the characterization of the rate function R(P, Q, D) to the same level of generality. We show that it can always be written as a convex dual, and that a minimizer W in the definition of R(P, Q, D) always exists (unless, of course, the infimum is taken over the empty set).

II. Notation and Assumptions

The source and reproduction alphabets S and T, along with their associated σ-algebras are assumed to be Borel spaces² and the distortion measure ρ : S × T ↦ [0, ∞) is assumed to be product measurable. P and Q denote generic probability distributions on S and T, respectively, and let X ~ P and Y ~ Q be independent random variables (r.v.). Define³

$${\rho }^Q\left(x\right):=\mathrm{ess\; inf}\rho \left(x,Y\right).$$

If W is a probability distribution on S × T, then we use W^S to denote the marginal distribution of W on S, and similarly for W^T. Define

$$R\left(P,Q,D\right):=\underset{W\in W\left(P,D\right)}{\inf }H\left(W\parallel P\times Q\right)$$

where the infimum is over the subset of probability distributions on S × T defined by

$$W\left(P,D\right):=\left\{W:W^S=P,E_{\left(U,V\right)\sim W}\rho \left(U,V\right)\le D\right\}$$

and where H(μ∥ν) denotes the relative entropy (in nats), namely,

$$H\left(\mu \parallel \nu \right):=\{\begin{array}{ll}{E}_{\mu }\log \frac{d\mu }{d\nu },& \mathrm{if}\mu \ll \nu \\ \infty ,& \mathrm{otherwise}.\end{array}$$

We use the convention that 0̸ = +∞ and logarithms are base e. For independent r.v.s X ~ P and Y ~ Q, define

$$\begin{array}{c}\Lambda \left(P,Q,\lambda \right):=E\left[\log E[e^{\lambda \rho \left(X,Y\right)}|X]\right]\\ {\Lambda }^{\ast }\left(P,Q,D\right):=\underset{\lambda \le 0}{\sup }\left[\lambda D-\Lambda \left(P,Q,\lambda \right)\right].\end{array}$$

Consider now the product spaces Sⁿ and Tⁿ with the usual σ-algebras and with generic probability distributions P_n and Q_n, respectively. Let $X_1^n\sim P_n$ and $Y_1^n\sim Q_n$ be independent and let W_n be a probability distribution on Sⁿ × Tⁿ. We use Qⁿ to denote the product distribution of Q on Tⁿ. Define the sequence of single-letter distortion measures (ρ_n : n ≥ 1) by

$${\rho }_n\left(x_1^n,y_1^n\right):=n^{-1}\sum _{k=1}^n\rho \left(x_k,y_k\right).$$

Since we are working with abstract alphabets, it makes sense to formally replace S, T, ρ, P, Q, W, X and Y with Sⁿ, Tⁿ, ρ_n, P_n, Q_n, W_n, $X_1^n$ and $Y_1^n$, respectively, in the above definitions of ρ^Q, W(P, D), R(P, Q, D), Λ(P, Q, D) and Λ*(P, Q, D) in order to define ${\rho }_n^{Q_n}$, W_n(P_n, D), R_n(P_n, Q_n, D), Λ_n(P_n, Q_n, D) and ${\Lambda }_n^{\ast }\left(P_n,Q_n,D\right)$.

For each $x_1^n\in S^n$, let $P_{x_1^n}$ denote the empirical probability distribution of $x_1^n$ on S and let ${\delta }_{x_1^n}$ denote the probability distribution on Sⁿ that assigns probability 1 to the sequence $x_1^n$. Define

$$L_n\left(x_1^n,Q_n,D\right):=-n^{-1}\log Q_n\left(B_n\left(x_1^n,D\right)\right)$$

where

$$B_n\left(x_1^n,D\right):=\left\{y_1^n\in T^n:{\rho }_n\left(x_1^n,y_1^n\right)\le D\right\}$$

denotes the distortion ball of radius D at the point $x_1^n$.

Finally, consider the one-sided infinite sequence spaces S^∞ and T^∞ with the usual σ-algebras and with generic probability distributions ℙ and ℚ, respectively. Let (X_n : n ≥ 1) and (Y_n : n ≥ 1) be independent random sequences with distributions ℙ and ℚ, respectively. Let P_n and Q_n denote the marginal distributions of ℙ and ℚ on Sⁿ and Tⁿ, respectively, so that $X_1^n\sim P_n$ and $Y_1^n\sim Q_n$ as before. Define

$$\begin{array}{c}{R}_{\infty }\left(\mathbb{P},\mathbb{Q},D\right):=\underset{n\to \infty }{\mathrm{lim\; sup}}{n}^{-1}{R}_n\left(P_n,Q_n,D\right)\\ {\Lambda }_{\infty }\left(\mathbb{P},\mathbb{Q},\lambda \right):=\underset{n\to \infty }{\mathrm{lim\; sup}}{n}^{-1}{\Lambda }_n\left(P_n,Q_n,n\lambda \right)\\ {\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right):=\underset{\lambda \le 0}{\sup }\left[\lambda D-{\Lambda }_{\infty }\left(\mathbb{P},\mathbb{Q},\lambda \right)\right].\end{array}$$

It is straightforward to verify that when ℙ is stationary with P₁ = P and ℚ is memoryless with Q_n = Qⁿ, then Λ_∞ (ℙ, ℚ, λ) = Λ(P, Q, λ) and hence ${\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right)={\Lambda }^{\ast }\left(P,Q,D\right)$.

III. Memoryless Codebook Distributions

Our goal is provide necessary and sufficient conditions for the generalized AEP in (3) and for the characterization of the limit as a convex dual. In this section we restrict attention to the case where the code-book distribution is memoryless and where the source distribution is stationary and ergodic.

Under appropriate regularity conditions, it is well known in the literature that R(P, Q, D) can be expressed as a convex dual [9]. (See [14] for a review and further references). Our first result is that the regularity conditions are not necessary. (Proofs of all results are collected in Sections V and VI.)

Proposition 1

R(P, Q, D) = Λ* (P, Q, D) for all D. If W(P, D) is not empty, then this set contains a W such that R(P, Q, D) = H(W∥P × Q).

We separate our results about the generalized AEP into two theorems. The first theorem asserts that the generalized AEP always holds along some subsequence (of n’s). Consequently, the generalized AEP can only fail if the limit does not exist.⁴ The second theorem gives necessary and sufficient conditions for the existence of the limit and also characterizes the pathology. Taken together, they provide necessary and sufficient conditions for the generalized AEP.

Theorem 2

Suppose ℙ is stationary and ergodic with P₁ = P. Then

$$\underset{n\to \infty }{\mathrm{lim\; inf}}{L}_n\left(X_1^n,Q^n,D\right)=R\left(P,Q,D\right)\mathrm{a.s.}$$ (4)

for all D. The result also holds with $L_n\left(X_1^n,Q^n,D\right)$ replaced by $R(P_{X_1^n},Q,D)$.

Theorem 3

Suppose ℙ is stationary and ergodic with P₁ = P. Then $\mathrm{Prob}\left\{{\lim }_n{L}_n\left(X_1^n,Q^n,D\right)\mathrm{exists}\right\}<1$ if and only if 0 < D = D_min (P, Q) < ∞ and R(P, Q, D) < ∞ and ρ^Q (X₁) is not a.s. constant. Furthermore, in this pathological situation

$$\mathrm{Prob}\left\{L_n\left(X_1^n,Q^n,D\right)=\infty \mathrm{infinitely\; often}\right\}>0$$ (5a)

$$\mathrm{Prob}\left\{L_n\left(X_1^n,Q^n,D\right)<\infty \mathrm{infinitely\; often}\right\}=1$$ (5b)

$$\underset{m\to \infty }{\lim }{L}_{N_m}(X_1^{N_m},Q^{N_m},D)=R\left(P,Q,D\right)\mathrm{a.s.}$$ (5c)

where (N_m : m ≥ 1) is the (a.s.) infinite random subsequence of (n ≥ 1) for which $L_n\left(X_1^n,Q^n,D\right)$ is finite. All of the above also hold with $L_n\left(X_1^n,Q^n,D\right)$ replaced by $R(P_{X_1^n},Q,D)$.

The proofs below show that (N_m : m ≥ 1) can also be (a.s.) characterized as the random subsequence for which

$$n^{-1}\sum _{k=1}^n{\rho }^Q\left(X_k\right)\le D.$$ (6)

Note that D_min(P, Q) = Eρ^Q(X), whenever the former is finite.

A simple example that illustrates the pathology is the following: Let (X_n : n ≥ 1) be the sequence 1, 0, 1, 0, … with probability 1/2 and the sequence 0, 1, 0, 1, … with probability 1/2, namely, the binary, stationary, periodic Markov chain (which is ergodic). Let Q be the point mass at 0, let ρ(x, y) := |x − y| and let D = 1/2. Note that ρ^Q(X₁) = X₁ is not constant, that D = D_min (P, Q) = 1/2 and that R(P, Q, D) = 0 is finite. In the case when X₁ = 0, $L_n\left(X_1^n,Q^n,D\right)=0$ for all n. In the case when X₁ = 1, however, $L_{2n}\left(X_1^{2n},Q^{2n},D\right)=0$ and $L_{2n-1}\left(X_1^{2n-1},Q^{2n-1},D\right)=\infty$ for all n.

IV. Extensions to the Case With Memory

Although the source (X_n : n ≥ 1) can have memory, the generalized AEP stated thus far is restricted to the case where the reproduction distribution is memoryless, that is, L_n is evaluated with a product measure Qⁿ. We relax this assumption here and replace it with a strong mixing condition, namely

$$C^{-1}\mathbb{Q}\left(A\right)\mathbb{Q}\left(B\right)\le \mathbb{Q}\left(A\cap B\right)\le C\mathbb{Q}\left(A\right)\mathbb{Q}\left(B\right)$$ (7)

for some fixed 1 ≤ C < ∞ and all $A\in \sigma \left(Y_1^n\right)$ and $B\in \sigma (Y_{n+1}^{\infty })$ and all n. Examples include the cases where ℚ is memoryless (C = 1) and where ℚ is a hidden Markov model (HMM) whose underlying Markov chain has a finite state space with all (strictly) positive transition probabilities.

Proposition 4

If ℙ and ℚ are stationary and ℚ satisfies (7), then

$$R_{\infty }\left(\mathbb{P},\mathbb{Q},D\right)=\underset{n\to \infty }{\lim }{n}^{-1}{R}_n\left(P_n,Q_n,D\right)={\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right)$$

for all D.

The proof shows that limit supremum in the definition of Λ_∞ is actually a limit and Proposition 1 implies that $R_n\left(P_n,Q_n,D\right)={\Lambda }_n^{\ast }\left(P_n,Q_n,D\right)$. For the special case when S and T are finite and ℚ is a finite state Markov chain, a formula for R_∞(ℙ, ℚ, D) not involving limits was identified in [7].

Theorem 5

If ℚ is stationary and satisfies (7), then Theorems 2 and 3 remain valid when Qⁿ, $Q^{N_m}$, R(P, Q, D) and $R(P_{X_1^n},Q,D)$ are replaced by Q_n, Q_Nm, R_∞(ℙ, ℚ, D) and $n^{-1}{R}_n({\delta }_{X_1^n},Q_n,D)$, respectively.

The mixing conditions here are strong enough to ensure that

$$D_{\min }\left(P,Q\right)=D_{\min }\left(\mathbb{P},\mathbb{Q}\right):=\inf \left\{D:R_{\infty }\left(\mathbb{P},\mathbb{Q},D\right)<\infty \right\}$$ (8)

and that

$${\rho }_n^{Q_n}\left(x_1^n\right)=n^{-1}\sum _{k=1}^n{\rho }^Q\left(x_k\right)$$ (9)

which is why the results for memory can still be in terms of D_min(P, Q) and ρ^Q. Extending Theorem 3 to situations where these do not hold seems difficult. The generalized AEP for ℚ with memory can also be found in [7], [12], [14], [16], in some cases under more general mixing conditions, but always for bounded distortion measure ρ and for D ≠ D_min (ℙ, ℚ).

V. Proofs: Standard Cases

Owing to space constraints, the proofs presented here focus primarily on the case D = D_min (P, Q), which has received little or no attention in the literature. We also provide some justification for removing the standard assumption that D_ave (P, Q) < ∞. This is a moment condition on ρ and it is assumed in all previous treatments of the subject. More detailed proofs, including measurability issues, can be found in a longer version of this paper [2] and the technical report that preceded it [1].

When ℚ is stationary and satisfies (7), then

$$\left|\log Q_n\left(A_n\right)+\log Q_m\left(A_m\right)-\log Q_{n+m}\left(A_n\times A_m\right)\right|\le \log C$$ (10)

and

$$\left|{\Lambda }_n({\delta }_{x_1^n},Q_n,n\lambda )+{\Lambda }_m({\delta }_{x_{n+1}^{n+m}},Q_m,m\lambda )-{\Lambda }_{n+m}({\delta }_{x_1^{n+m}},Q_{n+m},\left(n+m\right)\lambda )\right|\le \log C.$$ (11)

These two bounds make it relatively straightforward to extend the memoryless case to the case with memory. Replacing x with X in (11), taking expectations and using well-known facts about subadditive sequences (cf., [15, Lemma 10.21]) gives Proposition 4. Equation (10) combined with a blocking argument can be used to extend the memoryless generalized AEP to the case with memory. The details are tedious and analogous to existing arguments in the literature. We refer the interested reader to [2] and [1]. Henceforth, except for the special case D = D_min (P, Q) and our remarks about the lower bound below, we consider only the memoryless setting.

It is convenient in the proofs to temporarily redefine

$$D_{\min }\left(P,Q\right):=\inf \left\{D:{\Lambda }^{\ast }\left(P,Q,D\right)<\infty \right\}.$$

Proposition 1, once established, shows that D_min agrees with the definition in the text.

A. The Lower Bound and Replacing L_n With R_n

For each λ ≤ 0, the exponential Chebyshev’s inequality gives

$$\begin{array}{lll}n{L}_n\left(X_1^n,Q_n,D\right)& \ge & {\Lambda }_n^{\ast }({\delta }_{X_1^n},Q_n,D)\\ & \ge & n\lambda D-{\Lambda }_n({\delta }_{X_1^n},Q_n,n\lambda ).\end{array}$$

Taking limits, using (11) to justify the sub-additive ergodic theorem [15, Th. 10.22], and then optimizing over λ ≤ 0 gives

$$\underset{n\to \infty }{\mathrm{lim\; inf}}{L}_n\left(X_1^n,Q_n,D\right)\ge {\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right)\mathrm{a.s.}$$ (12)

This holds for all D. Combined with the previous bound this also shows that whenever $L_n\left(X_1^n,Q_n,D\right)$ converges to ${\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right),n^{-1}{\Lambda }_n^{\ast }({\delta }_{X_1^n},Q_n,D)$ converges to the same limit. Furthermore, using Lemma 7 below it is straightforward to show that ${\Lambda }_n^{\ast }$ must be infinite when L_n is infinite. Using Propositions 1 and 4 to equate R_n and ${\Lambda }_n^{\ast }$, these remarks justify our claims that L_n can be replaced by n⁻¹ R_n in each of the theorems.

There is also a lower bound for Proposition 1.

$$H\left(W\parallel P\times Q\right)\ge {\Lambda }^{\ast }\left(P,Q,D\right)\mathrm{for\; all}W\in W\left(P,D\right).$$ (13)

This only makes sense if W (P, D) ≠ 0̸. A simple proof can be found in [14, Th. 2]. Note that the proof there does not make use of any regularity assumptions, other than the existence of a regular conditional distribution W(y∣x).

B. The Upper Bound: D < D_min(P, Q) or D > D_ave(P, Q)

The upper bound is

$$\underset{n\to \infty }{\mathrm{lim\; sup}}{L}_n\left(X_1^n,Q_n,D\right)\le {\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right)\mathrm{a.s.}$$ (14)

If D < D_min(P, Q) then the upper bound is trivial since the right side is +∞. If D > D_ave(P, Q) (the latter of which must then be finite), then the left side converges to 0 (Chebyshev’s inequality and the ergodic theorem) and again the upper bound must hold.

For Proposition 1, the upper bound is

$$H\left(W\parallel P\times Q\right)\le {\Lambda }^{\ast }\left(P,Q,D\right)\mathrm{for\; some}W\in W\left(P,D\right).$$ (15)

Again, this only makes sense if W(P, D) ≠ 0̸. If D < D_min(P, Q), then any W ∈ W(P, D) trivially satisfies (15). If D ≥ D_ave(P, Q), then W = P × Q ∈ W(P, D) trivially satisfies (15).

C. The Upper Bound: D_min(P, Q) < D ≤ D_ave(P, Q)

This is the case considered in the literature, at least under the assumption that D_ave(P, Q) < ∞. Under appropriate regularity conditions, the upper bound (14) is an immediate consequence of the one-sided large deviations theorem in [17] applied to the random variables $W_n:=-n{\rho }_n\left(x_1^n,Y_1^n\right)$ and the constants a_n := −D for a typical, fixed realization (x_n : n ≥ 1) of the source. See [14, Th. 1] for an overview of this technique using the Gärtner-Ellis theorem [18, Th. V.6], which is a generalization of [17]. The required regularity conditions are placed on Λ_∞(ℙ, ℚ, λ).

For the memoryless codebook case, Λ_∞(ℙ, ℚ, λ) = Λ(P, Q, λ) and the regularity conditions needed in [17] are given by the following Lemma.

Lemma 6

If D_min(P, Q) < ∞, then Λ(λ) := Λ(P, Q, λ) is convex and continuously differentiable on (−∞, 0) with lim_λ↓−∞ Λ′(λ) = E(ρ^Q(X)) = D_min(P, Q) and lim_λ↑0 Λ′(λ) = D_ave(P, Q) (which may be infinite). If D_min(P, Q) < D_ave(P, Q), then Λ(λ) is strictly convex on (−∞, 0).

Remarks

The regularity assumption D_ave(P, Q) < ∞ is used in the literature to establish these properties of Λ. It can be replaced by the much weaker assumption that D_min(P, Q) < ∞. If Lemma 6 were known to be true for Λ_∞, then the more general case with memory would also follow immediately from [17]. Establishing the strict convexity of Λ_∞ seems challenging, however, and we resorted to a blocking argument as mentioned above to derive the general case from the memoryless case.

Proof

Let Z be a real-valued, nonnegative random variable. Define Γ(λ) := log Ee^λZ. It is well-known [19, Lem. 2.2.5, Ex. 2.2.24], [9], [14] that Γ is nondecreasing, convex and C^∞ on (−∞, 0) with Γ′(λ) ↓ ess inf Z as λ ↓ ∞ and Γ′(λ) ↑ EZ (which may be infinite) as λ ↑ 0. Furthermore, if ess inf Z < EZ then Γ is strictly convex on (−∞, 0).

Now define Γ(x, λ) := log Ee^{λρ(x, Y)}. Using Z := ρ(x, Y), we see that Γ(x, λ) has all the above properties of Γ(λ) for each fixed x. Since Λ(λ) = EΓ(X, λ), which must be finite on (−∞, 0] if D_min < ∞, we need only show that these properties are preserved by expectation. Convexity and strict convexity are immediately preserved. Preserving properties of the derivative requires some justification. Moving the expectation inside the derivative is justified by convexity and the monotone convergence theorem for the left hand derivative (including λ ↑ 0) and then finiteness and the dominated convergence theorem for the right hand derivative (including λ ↓ −∞). The fact that D_min = lim_λ↓−∞ Λ′(λ) is a well-known property of Fenchel–Legendre transforms.

Turning to Proposition 1, if D_min(P, Q) < D < D_ave(P, Q), a proof of the upper bound (15) can be found in [14, Th. 2]. The achieving W ∈ W(P, D) is identified as

$$\frac{\mathit{dW}}{d\left(P\times Q\right)}\left(x,y\right):=\frac{e^{{\lambda }_D\rho \left(x,y\right)}}{E{e}^{{\lambda }_D\rho \left(x,Y\right)}}$$

where λ_D < 0 is uniquely chosen so that Λ*(P, Q, D) = λ_DD − Λ(P, Q, λ_D). Note that the properties of Λ given in Lemma 6 are sufficient regularity conditions for ensuring that λ_D exists and has the desired properties. See [19, Lemma 2.2.5] and [20, Th. 23.5, Corollary 23.5.1, Th. 25.1].

VI. Proofs: Nonstandard Cases

Here we consider the special case when D = D_min := D_min(P, Q) < ∞. Lemma 6 is applicable, so

$$D_{\min }=E\left({\rho }^Q\left(X\right)\right).$$

Unlike most of the previous section, we explicitly treat the case of code-book distributions with memory. In particular, we assume that ℚ is stationary and satisfies (7). The bounds in (10) and (11) easily give (9) and

$$D_{\min }=\inf \left\{D:{\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D\right)<\infty \right\}$$

which means that Λ*(D_min) := Λ*(P, Q, D_min) is finite (infinite) exactly when ${\Lambda }_{\infty }^{\ast }\left(D_{\min }\right):={\Lambda }_{\infty }^{\ast }\left(\mathbb{P},\mathbb{Q},D_{\min }\right)$ is finite (infinite). The lower bounds in (12) and (13) are also valid. These lower bounds address the upper bounds in (14) and (15) in the case Λ*(D_min) = ∞. So in this section we will restrict attention to Λ*(D_min) < ∞. The proofs make use of the sets

$$A_n\left(x_1^n\right):=\left\{y_1^n\in T^n:{\rho }_n\left(x_1^n,y_1^n\right)={\rho }_n^{Q_n}\left(x_1^n\right)\right\}.$$

We use A(x) := A₁(x) = {y : ρ(x, y) = ρ^Q(x)} and we use the notation 1{B} to denote the indicator function of the event B.

Lemma 7

If Λ*(D_min) < ∞, then

$${\Lambda }^{\ast }\left(D_{\min }\right)=-E\left[\log Q\left(A\left(X\right)\right)\right]=H\left(W\parallel P\times Q\right)$$

for W ∈ W(P, D_min) defined by

$$\frac{\mathit{dW}}{d\left(P\times Q\right)}\left(x,y\right):=\frac{1\left\{y\in A\left(x\right)\right\}}{Q\left(A\left(x\right)\right)}.$$

Proof

Define

$$\stackrel{\sim }{\rho }\left(x,y\right):=\max \left\{\rho \left(x,y\right)-{\rho }^Q\left(x\right),0\right\}$$

so that ρ̃ is a valid distortion measure and so that

$$Q\left(A\left(x\right)\right)=Q\left(\left\{y:\stackrel{\sim }{\rho }\left(x,y\right)=0\right\}\right).$$

Let Λ̃ be defined analogously to Λ, except with ρ̃ instead of ρ. We have Λ(λ) = Λ̃(λ) + λD_min so that

$$\begin{array}{l}{\Lambda }^{\ast }\left(D_{\min }\right)\\ =\underset{\lambda \le 0}{\sup }\left[\lambda D_{\min }-\stackrel{\sim }{\Lambda }\left(\lambda \right)-\lambda D_{\min }\right]\\ =\underset{\lambda \downarrow -\infty }{\lim }-\stackrel{\sim }{\Lambda }\left(\lambda \right)=-\underset{\lambda \downarrow -\infty }{\lim }E\left[\log E\left[e^{\lambda \stackrel{\sim }{\rho }\left(X,Y\right)}|X\right]\right]\\ =-E\left[\log E\left[\underset{\lambda \downarrow -\infty }{\lim }{e}^{\lambda \stackrel{\sim }{\rho }\left(X,Y\right)}|X\right]\right]\\ =-E\left[\log E\left[1\left\{\stackrel{\sim }{\rho }\left(X,Y\right)=0\right\}|X\right]\right]\\ =-E\left[\log Q\left(\left\{y:\stackrel{\sim }{\rho }\left(X,y\right)=0\right\}\right)\right]=-E\left[\log Q\left(A\left(X\right)\right)\right].\end{array}$$

We moved the limit inside the expectations using first the monotone convergence theorem and then the dominated convergence theorem. This also shows that the denominator in the definition of W is strictly positive P-a.s., so W is well-defined.

It is easy to see that $W\in W\left(P,D_{\min }\right)$ and that

$$\begin{array}{l}H\left(W\parallel P\times Q\right)\\ =E\left[\frac{dW}{d\left(P\times Q\right)}\left(X,Y\right)\log \frac{dW}{d\left(P\times Q\right)}\left(X,Y\right)\right]\\ =E\left[\frac{1\left\{Y\in A\left(X\right)\right\}}{Q\left(A\left(X\right)\right)}\log 1\left\{Y\in A\left(X\right)\right\}\right]-E\left[\frac{1\left\{Y\in A\left(X\right)\right\}}{Q\left(A\left(X\right)\right)}\log Q\left(A\left(X\right)\right)\right]\\ =0-E\left[\log Q\left(A\left(X\right)\right)\right]={\Lambda }^{\ast }\left(D_{\min }\right)\end{array}$$

which completes the proof.

Lemma 7 gives the upper bound (15) and completes the proof of Proposition 1 for all possible values of D. We now address the upper bound (14) for the generalized AEP. Because of (9)

$$Q_{n+m}(A_{n+m}(x_1^{n+m}))=Q_{n+m}(A_n(x_1^n)\times A_m(x_{n+1}^{n+m}))$$

and we can use (10) in conjunction with the subadditive ergodic theorem [15, Th. 10.22] to conclude that

$$\begin{array}{l}\underset{n\to \infty }{\lim }-\frac{1}{n}\log Q_n(A_n(X_1^n))\\ \stackrel{\mathrm{a.s.}}{=}\underset{n\to \infty }{\lim }-\frac{1}{n}E[-\log Q_n(A_n(X_1^n))]\\ =\underset{n\to \infty }{\lim }\frac{1}{n}{\Lambda }_n^{\ast }(P_n,Q_n,D_{\min })={\Lambda }_{\infty }^{\ast }(\mathbb{P},\mathbb{Q},D_{\min }).\end{array}$$ (16)

The limit in the subadditive ergodic theorem is deterministic because it is shift invariant and the source is ergodic. The second equality comes from Lemma 7 and the third from Propositions 1 and 4. Note that if ρ^Q(X) is a.s. constant, then $Q_n(A_n(X_1^n))\stackrel{\mathrm{a.s.}}{=}{Q}_n(B_n(X_1^n,D_{\min }))$ and (16) gives the upper bound (14).

Now suppose ρ^Q(X₁) is not a.s. constant (and D = D_min and Λ*(D_min) < ∞). This is the only pathological situation where the upper bound does not hold. Our analysis makes use of recurrence properties for random walks with stationary and ergodic increments. What we need is summarized in the following lemma. The notation “i.o.” means “infinitely often.”

Lemma 8

Let (U_n : n ≥ 1) be a real-valued stationary and ergodic process and define $W_n:={\sum }_{k=1}^n{U}_k$ for n ≥ 1. If EU₁ = 0 and Prob{U₁ ≠ 0} > 0, then Prob{W_n > 0 i.o.} > 0 and Prob{W_n ≥ 0 i.o.} = 1.

Proof

Define W₀ := 0 so that (W_n : n ≥ 0) is a random walk with stationary and ergodic increments [21]. [22] shows that {lim inf_n⁻¹W_n > 0} and {W_n → ∞} differ by a null set. The ergodic theorem gives Prob{n⁻¹W_n → 0} = 1, so Prob{W_n → ∞} = 0. Similarly, by considering the process −W_n, we see that Prob{W_n → −∞} = 0.

Now {|W_n| → ∞} is invariant and must have probability 0 or 1. If it has probability 1, then since we cannot have W_n → ∞ or W_n → −∞ we must have W_n oscillating between increasingly larger positive and negative values, which means Prob{W_n > 0i.o.} = 1 and completes the proof.

Suppose Prob{|W_n| → ∞} = 0. Define

$$N\left(A\right):=\sum _{n\ge 0}1\left\{W_n\in A\right\},A\subset ℝ$$

to be the number of times the random walk visits the set A. [21, Corollary 2.3.4] shows that either N(J) < ∞ a.s. for all bounded intervals J or {N(J)=0}∪{N(J)=∞} has probability 1 for all intervals J (open or closed, bounded or unbounded, but not a single point). By assumption |W_n|↛∞, so we can rule out the first possibility. Since Prob{W₀ = 0} = 1, we see that for any interval J containing {0} we must have Prob{N(J) = ∞} = 1. In particular, taking J := [0, ∞) shows that Prob{W_n ≥ 0 i.o.} = 1. Similarly, taking J := (0, ∞) shows that

$$\begin{array}{l}\mathrm{Prob}\left\{W_n>0\mathrm{i.o.}\right\}\\ =\mathrm{Prob}\left\{N\left(J\right)=\infty \right\}\\ =\mathrm{Prob}\left\{N\left(J\right)>0\right\}\ge \mathrm{Prob}\left\{U_1>0\right\}>0.\end{array}$$

Returning to the main argument

$$\begin{array}{l}{L}_n\left(X_1^n,Q_n,D_{\min }\right)\\ \ge -n^{-1}\log Q_n\left\{y_1^n:n^{-1}\sum _{k=1}^n{\rho }^Q\left(X_k\right)\le D_{\min }\right\}\\ =\{\begin{array}{ll}0,& \mathrm{if}{\sum }_{k=1}^n{\rho }^Q\left(X_k\right)\le n{D}_{\min }\\ \infty ,& \mathrm{if}{\sum }_{k=1}^n{\rho }^Q\left(X_k\right)>n{D}_{\min }\end{array}\\ =\{\begin{array}{ll}0,& \mathrm{if}{W}_n\le 0\\ \infty ,& \mathrm{if}{W}_n>0\end{array}\end{array}$$ (17)

where $W_n:={\sum }_{k=1}^n({\rho }^Q(X_k)-D_{\min })$. Lemma 8 shows that Prob{W_n > 0 i.o.} > 0. This and (17) prove (5a).

Lemma 8 also shows that Prob{W_n ≤ 0 i.o.} = 1. Let (N_m : m ≥ 1) be the (a.s.) infinite, random subsequence of (n ≥ 1) such that W_n ≤ 0. Note that

$$N_m^{-1}\sum _{k=1}^{N_m}{\rho }^Q\left(X_k\right)\le D_{\min }$$

so

$$\begin{array}{l}{L}_{N_m}(X_1^{N_m},Q_{N_m},D_{\min })\\ \le L_{N_m}(X_1^{N_m},Q_{N_m},N_m^{-1}\sum _{k=1}^{N_m}{\rho }^Q(X_k))\\ =-N_m^{-1}\log Q_{N_m}(A_{N_m}(X_1^{N_m})).\end{array}$$ (18)

Now, the final expression in (18) is a.s. finite because $E[-\log Q_n(A_n(X_1^n))]={\Lambda }_n^{\ast }(P_n,Q_n,D_{\min })<\infty$. This proves (5b) and shows that (N_m : m ≥ 1) satisfies the claims of the theorem, including (6). Letting m → ∞ in (18) and using (16) gives (5c), the upper bound along the sequence (N_m : m ≥ 1). Note that it also shows that the lim inf_n L_n is a.s. ${\Lambda }_{\infty }^{\ast }\left(D_{\min }\right)$ even in this pathological case, which proves Theorem 2 and its generalization in Theorem 5.