Sharp Second-Order Pointwise Asymptotics for Lossless Compression with Side Information

Abstract

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs, the conditional information density is shown to provide a sharp asymptotic lower bound for the description lengths achieved by an arbitrary sequence of compressors. This implies that for ergodic source-side information pairs, the conditional entropy rate is the best achievable asymptotic lower bound to the rate, not just in expectation but with probability one. Under appropriate mixing conditions, a central limit theorem and a law of the iterated logarithm are proved, describing the inevitable fluctuations of the second-order asymptotically best possible rate. An idealised version of Lempel-Ziv coding with side information is shown to be universally first- and second-order asymptotically optimal, under the same conditions. These results are in part based on a new almost-sure invariance principle for the conditional information density, which may be of independent interest.

1. Introduction

It is well-known that the presence of correlated side information can potentially offer dramatic benefits for data compression [1,2]. Important applications where such side information is naturally present include the compression of genomic data [3,4], file and software management [5,6], and image and video compression [7,8].

In practice, the most common approach to the design of effective compression methods with side information is based on generalisations of the Lempel-Ziv family of algorithms [9,10,11,12,13]. A different approach based on grammar-based codes was developed in [14], turbo codes were applied in [15], and a generalised version of context-tree weighting was used in [16].

In this work, we examine the theoretical fundamental limits of the best possible performance that can be achieved in such problems. Let $(\mathit{X},\mathit{Y})=\{(X_n,Y_n);n\ge 1\}$ be a source-side information pair; X is the source to be compressed, and Y is the associated side information process which is assumed to be available both to the encoder and the decoder. Under appropriate conditions, the best average rate that can be achieved asymptotically [2], is the conditional entropy rate,

$$H(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{lim}\frac{1}{n}H(X_1^n|Y_1^n),\mathrm{bits}/\mathrm{symbol},$$

where $X_1^n=(X_1,X_2,\dots ,X_n)$, $Y_1^n=(Y_1,Y_2,\dots ,Y_n)$, and $H(X_1^n|Y_1^n)$ denotes the conditional entropy of $X_1^n$ given $Y_1^n$; precise definitions will be given in Section 2.

Our main goal is to derive sharp asymptotic expressions for the optimum compression rate (with side information available to both the encoder and decoder), not only in expectation but with probability 1. In addition to the best first-order performance, we also determine the best rate at which this performance can be achieved, as a function of the length of the data being compressed. Furthermore, we consider an idealised version of a Lempel-Ziv compression algorithm, and we show that it can achieve asymptotically optimal first- and second-order performance, universally over a broad class of stationary and ergodic source-side information pairs $(\mathit{X},\mathit{Y})$.

Specifically, we establish the following. In Section 2.1 we describe the theoretically optimal one-to-one compressor $f_n^{\ast }(X_1^n|Y_1^n)$, for arbitrary source-side information pairs $(\mathit{X},\mathit{Y})$. In Section 2.2 we prove our first result, stating that the description lengths $\ell (f_n^{\ast }(X_1^n|Y_1^n))$ can be well-approximated, with probability one, by the conditional information density, $-logP(X_1^n|Y_1^n)$. Theorem 2 states that for any jointly stationary and ergodic source-side information pair $(\mathit{X},\mathit{Y})$, the best asymptotically achievable compression rate is $H(\mathit{X}|\mathit{Y})$ bits/symbol, with probability 1. This generalises Kieffer’s corresponding result [17] to the case of compression with side information.

Furthermore, in Section 2.4 we show that there is a sequence of random variables $\{Z_n\}$ such that the description lengths $\ell (f_n(X_1^n|Y_1^n))$ of any sequence of compressors $\{f_n\}$ satisfy a “one-sided” central limit theorem (CLT): Eventually, with probability 1,

$$\begin{array}{c}\hfill \ell (f_n(X_1^n|Y_1^n))\ge nH(\mathit{X}|\mathit{Y})+\sqrt{n}{Z}_n+o(\sqrt{n}),\mathrm{bits},\end{array}$$ (1)

where the $Z_n$ converge to a $N(0,{\sigma }^2(\mathit{X}|\mathit{Y}))$ distribution, and the term $o(\sqrt{n})$ is negligible compared to $\sqrt{n}$. The lower bound (1) is established in Theorem 3 where it is also shown that it is asymptotically achievable. This means that the rate obtained by any sequence of compressors has inevitable $O(\sqrt{n})$ fluctuations around the conditional entropy rate, and that the size of these fluctuations is quantified by the conditional varentropy rate,

$${\sigma }^2(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{lim}\frac{1}{n}\mathrm{Var}\left(-logP(X_1^n|Y_1^n)\right).$$

This generalises the minimal coding variance of [18]. The bound (1) holds for a broad class of source-side information pairs, including all Markov chains with positive transition probabilities. Under the same conditions, a corresponding “one-sided” law of the iterated logarithm (LIL) is established in Theorem 4, which gives a precise description of the inevitable almost-sure fluctuations above $H(\mathit{X}|\mathit{Y})$, for any sequence of compressors.

The proofs of all the results in Section 2.3 and Section 2.4 are based, in part, on analogous asymptotics for the conditional information density, $-logP(X_1^n|Y_1^n)$. These are established in Section 2.5, where we state and prove a corresponding CLT and an LIL for $-logP(X_1^n|Y_1^n)$. These results, in turn, follow from the almost sure invariance principle for $-logP(X_1^n|Y_1^n)$, proved in Appendix A. Theorem A1, which is of independent interest, generalises the invariance principle established for the (unconditional) information density $-logP(X_1^n)$ by Philipp and Stout [19]. In fact, Theorem A1 along with the identification of the conditions under which it holds (Assumption 1) in Section 2.4), are the more novel contributions of this work.

In a different direction, Nomura and Han [20] establish finer coding theorems for the Slepian-Wolf problem, when the side information is only available to the decoder. There, they obtain general second-order asymptotics for the best achievable rate region, under an excess-rate probability constraint.

Section 3 is devoted to universal compression. We consider a simple, idealised version of Lempel-Ziv coding with side information. As in the case of Lempel-Ziv compression without side information [21,22], the performance of this scheme is determined by the asymptotics of a family of conditional recurrence times, ${\mathcal{R}}_n={\mathcal{R}}_n(\mathit{X}|\mathit{Y})$. Under appropriate, general conditions on the source-side information pair $(\mathit{X},\mathit{Y})$, in Theorem 8 we show that the ideal description lengths, $log{\mathcal{R}}_n$, can be well-approximated by the conditional information density $-logP(X_1^n|Y_1^n)$. Combining this with our earlier results on the conditional information density, in Corollary 1 and Theorem 9 we show that the compression rate of this scheme converges to $H(\mathit{X}|\mathit{Y})$, with probability 1, and that it is universally second-order optimal. The results of this section generalise the corresponding asymptotics without side information established in [23,24].

The proofs of the more technical results needed in Section 2 and Section 3 are given in the appendix.

2. Pointwise Asymptotics

In this section, we derive general, fine asymptotic bounds for the description lengths of arbitrary compressors with side information, as well as corresponding achievability results.

2.1. Preliminaries

Let $\mathit{X}=\{X_n;n\ge 1\}$ be an arbitrary source to be compressed, and $\mathit{Y}=\{Y_n;n\ge 1\}$ be an associated side information process. We let $\mathcal{X},\mathcal{Y}$, denote their finite alphabets, respectively, and we refer to the joint process $(\mathit{X},\mathit{Y})=\{(X_n,Y_n);n\ge 1\}$ as a source-side information pair.

Let $x_1^n=(x_1,x_2,\dots ,x_n)$ be a source string, and let $y_1^n=(y_1,y_2,\dots ,y_n)$ an associated side information string which is available to both the encoder and decoder. A fixed-to-variable one-to-one compressor with side information, of blocklength n, is a collection of functions $f_n$, where each $f_n(x_1^n|y_1^n)$ takes a value in the set of all finite-length binary strings,

$${\{0,1\}}^{\ast }=\underset{k=0}{\overset{\infty }{\mathrm{U}}}{\{0,1\}}^k=\{Ø,0,1,00,01,000,\dots \},$$

with the convention that ${\{0,1\}}^0=\{Ø\}$ consists of just the empty string Ø of length zero. For each $y_1^n\in {\mathcal{Y}}^n$, we assume that $f_n(·|y_1^n)$ is a one-to-one function from ${\mathcal{X}}^n$ to ${\{0,1\}}^{\ast }$, so that the compressed binary string $f_n(x_1^n|y_1^n)$ is always correctly decodable.

The main figure of merit in lossless compression is of course the description length,

$$\ell (f_n(x_1^n|y_1^n))=\mathrm{length} \mathrm{of}{f}_n(x_1^n|y_1^n),\mathrm{bits},$$

where throughout, $\ell (s)$ denotes the length, in bits, of a binary string s. It is easy to see that under quite general criteria, the optimal compressor $f_n^{\ast }$ is easy to describe; see [25] for an extensive discussion. For $1\le i\le j\le \infty$, we use the shorthand notation $z_i^j$ for the string $(z_i,z_{i+1},\dots ,z_j)$, and similarly $Z_i^j$ for the corresponding collection of random variables $Z_i^j=(Z_i,Z_{i+1},\dots ,Z_j)$.

Definition 1

(The optimal compressor ${\mathit{f}}_{\mathit{n}}^{\ast }$). For each side information string $y_1^n$, $f_n^{\ast }(·|y_1^n)$ is the optimal compressor for the distribution $\mathbb{P}(X_1^n=·|Y_1^n=y_1^n)$, namely the compressor that orders the strings $x_1^n$ in order of decreasing probability $\mathbb{P}(X_1^n=x_1^n|Y_1^n=y_1^n)$, and assigns them codewords from ${\{0,1\}}^{\ast }$ in lexicographic order.

2.2. The Conditional Information Density

Definition 2

(Conditional information density). For an arbitrary source-side information pair $(\mathit{X},\mathit{Y})$, the conditional information density of blocklength n is the random variable: $-logP(X_1^n|Y_1^n)=-log{P}_{X_1^n|Y_1^n}(X_1^n|Y_1^n)$.

[Throughout the paper, ‘log’ denotes ‘${log}_2$’, the logarithm taken to base 2, and all familiar information theoretic quantities are expressed in bits.]

The starting point is the following almost sure (a.s.) approximation result between the description lengths $\ell (f_n(X_1^n|Y_1^n))$ of an arbitrary sequence of compressors and the conditional information density $-logP(X_1^n|Y_1^n))$ of an arbitrary source-side information pair $(\mathit{X},\mathit{Y})$. When it causes no confusion, we drop the subscripts for PMFs and conditional PMFs, e.g., simply writing $P(x_1^n|y_1^n)$ for $P_{X_1^n|Y_1^n}(x_1^n|y_1^n)$ as in the definition above. Recall the definition of the optimal compressors $\{f_n^{\ast }\}$ from Section 2.1.

Theorem 1.

For any source-side information pair $(\mathit{X},\mathit{Y})$, and any sequence $\{B_n\}$ that grows faster than logarithmically, i.e., such that $B_n/logn\to \infty$ as $n\to \infty$, we have:

• (a) 
  For any sequence of compressors with side information $\{f_n\}$:

  $$\begin{array}{c}\hfill \underset{n\to \infty }{\lim \inf }\frac{\ell (f_n(X_1^n|Y_1^n))-[-logP(X_1^n|Y_1^n)]}{B_n}\ge 0,a.s.\end{array}$$
• (b)  The optimal compressors $\{f_n^{\ast }\}$ achieve the above bound with equality.

Proof. 

Fix $ϵ>0$ arbitrary and let $\tau ={\tau }_n=ϵB_n$. Applying the general converse in ([25], Theorem 3.3) with $X_1^n,Y_1^n$ in place of $\mathit{X},\mathit{Y}$ and ${\mathcal{X}}^n,{\mathcal{Y}}^n$ in place of $\mathcal{X},\mathcal{Y}$, gives,

$$\mathbb{P}\left[\ell (f(X_1^n|Y_1^n))\le -logP(X_1^n|Y_1^n)-ϵB_n\right]\le 2^{logn-ϵB_n}(\lfloor log|\mathcal{X}|\rfloor +1),$$

which is summable in n. Therefore, by the Borel-Cantelli lemma we have that eventually, a.s.,

$$\ell (f(X_1^n|Y_1^n))+logP(X_1^n|Y_1^n)>-ϵB_n,$$

Since $ϵ>0$ was arbitrary, this implies $(a)$. Part $(b)$ follows from $(a)$ together with the fact that $\ell (f_n^{\ast }(X_1^n|Y_1^n))+logP(X_1^n|Y_1^n)\le 0$, a.s., by the general achievability result in ([25], Theorem 3.1). □

2.3. First-Order Asymptotics

For any source-side information pair $(\mathit{X},\mathit{Y})$, the conditional entropy rate $H(\mathit{X}|\mathit{Y})$ is defined as:

$$H(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{\lim \sup }\frac{1}{n}H(X_1^n|Y_1^n).$$

Throughout $H(Z)$ and $H(Z|W)$ denote the discrete entropy of Z and the conditional entropy of Z given W, in bits. If $(\mathit{X},\mathit{Y})$ are jointly stationary, then the above $\lim \sup$ is in fact a limit, and it is equal to $H(\mathit{X},\mathit{Y})-H(\mathit{Y})$, where $H(\mathit{X},\mathit{Y})$ and $H(\mathit{Y})$ are the entropy rates of $(\mathit{X},\mathit{Y})$ and of Y, respectively [2]. Moreover, if $(\mathit{X},\mathit{Y})$ are also jointly ergodic, then by applying the Shannon-McMillan-Breiman theorem [2] to Y and to the pair $(\mathit{X},\mathit{Y})$, we obtain its conditional version:

$$-\frac{1}{n}logP(X_1^n|Y_1^n)\to H(\mathit{X}|\mathit{Y}),\mathrm{a}.\mathrm{s}.$$ (2)

The next result states that the conditional entropy rate is the best asymptotically achievable compression rate, not only in expectation but also with probability 1. It is a consequence of Theorem 1 with $B_n=n$, combined with (2).

Theorem 2.

Suppose $(\mathit{X},\mathit{Y})$ is a jointly stationary and ergodic source-side information pair with conditional entropy rate $H(\mathit{X}|\mathit{Y})$.

• (a) 
  For any sequence of compressors with side information $\{f_n\}$:

  $$\begin{array}{c}\hfill \underset{n\to \infty }{\lim \inf }\frac{\ell (f_n(X_1^n|Y_1^n))}{n}\ge H(\mathit{X}|\mathit{Y}),a.s.\end{array}$$
• (b)  The optimal compressors $\{f_n^{\ast }\}$ achieve the above bound with equality.

2.4. Finer Asymptotics

The refinements of Theorem 2 presented in this section will be derived as consequences of the general approximation results in Theorem 1, combined with corresponding refined asymptotics for the conditional information density $-logP(X_1^n|Y_1^n)$. For clarity of exposition these are stated separately, in Section 2.5 below.

The results of this section will be established for a class of jointly stationary and ergodic source-side information pairs $(\mathit{X},\mathit{Y})$, that includes all Markov chains with positive transition probabilities. The relevant conditions, in their most general form, will be given in terms of the following mixing coefficients.

Definition 3.

Suppose $\mathit{Z}=\{Z_n;n\in \mathbb{Z}\}$ is a stationary process on a finite alphabet $\mathcal{Z}$. For any pair of indices $-\infty \le i\le j\le \infty$, let ${\mathcal{F}}_i^j$ denote the σ-algebra generated by $Z_i^j$. For $d\ge 1$, define:

$$\begin{array}{cc}\hfill {\alpha }^{(\mathit{Z})}(d)& =sup\left\{|\mathbb{P}(A\cap B)-\mathbb{P}(A)\mathbb{P}(B)|;A\in {\mathcal{F}}_{-\infty }^0,B\in {\mathcal{F}}_d^{\infty }\right\},\hfill \\ \hfill {\gamma }^{(\mathit{Z})}(d)& =\underset{z\in \mathcal{Z}}{max}\mathbb{E}\left(|log\mathbb{P}(Z_0=z|Z_{-\infty }^{-1})-log\mathbb{P}(Z_0=z|Z_{-d}^{-1})|\right).\hfill \end{array}$$

Note that if Z is an ergodic Markov chain of order k, then ${\alpha }^{(\mathit{Z})}(d)$ decays exponentially fast [26], and ${\gamma }^{(\mathit{Z})}(d)=0$ for all $d\ge k$. Moreover, if $(\mathit{X},\mathit{Y})$ is a Markov chain with all positive transition probabilities, then ${\gamma }^{(\mathit{Y})}(d)$ also decays exponentially fast; cf. ([27], Lemma 2.1).

Throughout this section we will assume that the following conditions hold:

Assumption 1.

The source-side information pair $(\mathit{X},\mathit{Y})$ is stationary and satisfies one of the following three conditions:

• $(a)$ $(\mathit{X},\mathit{Y})$ is a Markov chain with all positive transition probabilities; or
• $(b)$ $(\mathit{X},\mathit{Y})$ as well as Y are kth order, irreducible and aperiodic Markov chains; or
• $(c)$
  $(\mathit{X},\mathit{Y})$ is jointly ergodic and satisfies the following mixing conditions: [Our source-side information pairs $(\mathit{X},\mathit{Y})$ are only defined for $(X_n,Y_n)$ with $n\ge 1$, whereas the coefficients ${\alpha }^{(\mathit{Z})}(d)$ and ${\gamma }^{(\mathit{Z})}(d)$ are defined for two-sided sequences $\{Z_n;n\in \mathbb{Z}\}$. However, this does not impose an additional restriction, since any one-sided stationary process can be extended to a two-sided one by the Kolmogorov extension theorem [28].]

  $$\begin{array}{c}\hfill {\alpha }^{(\mathit{X},\mathit{Y})}(d)=O(d^{-336}),{\gamma }^{(\mathit{X},\mathit{Y})}(d)=O(d^{-48}),and{\gamma }^{(\mathit{Y})}(d)=O(d^{-48}).\end{array}$$ (3)

In view of the discussion following Definition 3, $(a)\Rightarrow (c)$ and $(b)\Rightarrow (c)$. Therefore, all results stated under Assumption 1 will be proved under the weakest set of conditions, namely that (3) hold.

Definition 4.

For a source-side information pair $(\mathit{X},\mathit{Y})$, the conditional varentropy rate is:

$${\sigma }^2(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{\lim \sup }\frac{1}{n}\mathrm{Var}\left(-logP(X_1^n|Y_1^n)\right).$$ (4)

Under the above assumptions, the $\lim \sup$ in (4) is in fact a limit. Lemma 1 is proved in the Appendix A.

Lemma 1.

Under Assumption 1, the conditional varentropy rate ${\sigma }^2(\mathit{X}|\mathit{Y})$ is:

$${\sigma }^2(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{lim}\frac{1}{n}\mathrm{Var}\left(-logP(X_1^n|Y_1^n)\right)=\underset{n\to \infty }{lim}\frac{1}{n}\mathrm{Var}\left(-log\left(\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right)\right).$$

Our first main result in this section is a “one-sided” central limit theorem (CLT), which states that the description lengths $\ell (f_n(X_1^n|Y_1^n))$ of an arbitrary sequence of compressors with side information, $\{f_n\}$, are asymptotically at best Gaussian, with variance ${\sigma }^2(\mathit{X}|\mathit{Y})$. Recall the optimal compressors $\{f_n^{\ast }\}$ described in Section 2.1

Theorem 3

(CLT for codelengths). Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate (4). Then there exists a sequence of random variables $\{Z_n;n\ge 1\}$ such that:

• (a) 
  For any sequence of compressors with side information, $\{f_n\}$, we have,

  $$\underset{n\to \infty }{\lim \inf }\left[\frac{\ell (f_n(X_1^n|Y_1^n))-H(X_1^n|Y_1^n)}{\sqrt{n}}-Z_n\right]\ge 0,a.s.,$$ (5)

  where $Z_n\to N(0,{\sigma }^2),$ in distribution, as $n\to \infty .$

• (b) The optimal compressors $\{f_n^{\ast }\}$ achieve the lower bound in (5) with equality.

Proof. 

Letting $Z_n=[-logP(X_1^n|Y_1^n)]/\sqrt{n}$, $n\ge 1$, and taking $B_n=\sqrt{n}$, both results follow by combining the approximation results of Theorem 1 with the corresponding CLT for the conditional information density in Theorem 5. □

Our next result is in the form of a “one-sided” law of the iterated logarithm (LIL) which states that with probability 1, the description lengths of any compressor with side information will have inevitable fluctuations of order $\sqrt{2{\sigma }^{2}n{log}_e{log}_{2}n}$ bits around the conditional entropy rate $H(\mathit{X}|\mathit{Y})$; throughout, ${log}_e$ denotes the natural logarithm to base e.

Theorem 4

(LIL for codelengths). Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate (4). Then:

• (a) 
  For any sequence of compressors with side information, $\{f_n\}$, we have:

  $$\begin{array}{cc}\hfill & \underset{n\to \infty }{\lim \sup }\frac{\ell (f_n(X_1^n|Y_1^n))-H(X_1^n|Y_1^n)}{\sqrt{2n{log}_e{log}_{e}n}}\ge \sigma ,a.s.,\hfill \end{array}$$ (6)

  $$\begin{array}{cc}\hfill and& \underset{n\to \infty }{\lim \inf }\frac{\ell (f_n(X_1^n|Y_1^n))-H(X_1^n|Y_1^n)}{\sqrt{2n{log}_e{log}_{e}n}}\ge -\sigma ,a.s.\hfill \end{array}$$ (7)
• (b) The optimal compressors $\{f_n^{\ast }\}$ achieve the lower bounds in (6) and (7) with equality.

Proof. 

Taking $B_n=\sqrt{2n{log}_2{log}_{e}n}$, the results of the theorem again follow by combining the approximation results of Theorem 1 with the corresponding LIL for the conditional information density in Theorem 6. □

Remark 1.

1. Although the results in Theorems 3 and 4 are stated for one-to-one compressors $\{f_n\}$, they remain valid for the class of prefix-free compressors. Since prefix-free codes are certainly one-to-one, the converse bounds in Theorem 3 $(a)$ and 4 $(a)$ are valid as stated, while for the achievability results it suffices to consider compressors $f_n^p$ with description lengths $\ell (f_n^p(x_1^n|y_1^n)))=\lceil -logP(x_1^n|y_1^n)\rceil$, and then apply Theorem 5.

2. Theorem 3 says that the compression rate of any sequence of compressors $\{f_n\}$ will have at best Gaussian fluctuations around $H(\mathit{X}|\mathit{Y})$,

   $$\frac{1}{n}\ell (f_n^{\ast }(X_1^n|Y_1^n))\approx N\left(H(\mathit{X}|\mathit{Y}),\frac{{\sigma }^2(\mathit{X}|\mathit{Y})}{n}\right),bits/symbol,$$

   and similarly Theorem 4 says that with probability 1, the description lengths will have inevitable fluctuations of approximately $\pm \sqrt{2n{\sigma }^2{log}_e{log}_{e}n}$ bits around $nH(\mathit{X}|\mathit{Y})$.

   As both of these vanish when ${\sigma }^2(\mathit{X}|\mathit{Y})$ is zero, we note that if the source-side information pair $(\mathit{X},\mathit{Y})$ is memoryless, so that $\{(X_n,Y_n)\}$ are independent and identically distributed, then the conditional varentropy rate reduces to,

   $${\sigma }^2(\mathit{X}|\mathit{Y})=\mathrm{Var}(-logP(X_1|Y_1)),$$

   which is equal to zero if and only if, for each $y\in \mathcal{Y}$, the conditional distribution of $X_1$ given $Y_1=y$ is uniform on a subset ${\mathcal{X}}_y\subset \mathcal{X}$, where all the ${\mathcal{X}}_y$ have the same cardinality.

   In the more general case when both the pair process $(\mathit{X},\mathit{Y})$ and the side information Y are Markov chains, necessary and sufficient conditions for ${\sigma }^2(\mathit{X}|\mathit{Y})$ to be zero were recently established in [25].

3. In analogy with the source dispersion for the problem of lossless compression without side information [29,30], for an arbitrary source-side information pair $(\mathit{X},\mathit{Y})$ the conditional dispersion $D(\mathit{X}|\mathit{Y})$ was recently defined [25] as,

   $$D(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{\lim \sup }\frac{1}{n}\mathrm{Var}[\ell (f_n^{\ast }(X_1^n|Y_1^n))].$$
   There, it was shown that when both the pair $(\mathit{X},\mathit{Y})$ and Y itself are irreducible and aperiodic Markov chains, the conditional dispersion coincides with the conditional varentropy rate:

   $$D(\mathit{X}|\mathit{Y})=\underset{n\to \infty }{lim}\frac{1}{n}\mathrm{Var}[\ell (f_n^{\ast }(X_1^n|Y_1^n))]={\sigma }^2(\mathit{X}|\mathit{Y})<\infty .$$

2.5. Asymptotics of the Conditional Information Density

Here we show that the conditional information density itself, $-logP(X_1^n|Y_1^n)$, satisfies a CLT and a LIL. The next two theorems are consequences of the almost sure invariance principle established in Theorem A1, in the Appendix A.

Theorem 5

(CLT for the conditional information density). Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate (4). Then, as $n\to \infty$:

$$\frac{-logP(X_1^n|Y_1^n)-H(X_1^n|Y_1^n)}{\sqrt{n}}\to N(0,{\sigma }^2),in distribution.$$ (8)

Proof. 

The conditions (3), imply that as $n\to \infty$, $[nH(\mathit{X},\mathit{Y})-H(X_1^n,Y_1^n)]/\sqrt{n}\to 0$, and $[nH(\mathit{Y})-H(Y_1^n)]/\sqrt{n}\to 0$, cf. [19], therefore also, $[nH(\mathit{X}|\mathit{Y})-H(X_1^n|Y_1^n)]/\sqrt{n}\to 0$, so it suffices to show that as $n\to \infty$,

$$\frac{-logP(X_1^n|Y_1^n)-nH(\mathit{X}|\mathit{Y})}{\sqrt{n}}\to N(0,{\sigma }^2).\mathrm{in} \mathrm{distribution}.$$ (9)

Let $D=D([0,1],\mathbb{R})$ denote the space of cadlag (right-continuous with left-hand limits) functions from $[0,1]$ to $\mathbb{R}$, and define, for each $t\ge 0$, $S(t)=logP(X_1^{\lfloor t\rfloor }|Y_1^{\lfloor t\rfloor })+tH(\mathit{X}|\mathit{Y})$, as in Theorem A1 in the Appendix A. For all $n\ge 1,t\in [0,1]$, define $S_n(t)=S(nt)$. Then Theorem A1 implies that as $n\to \infty$,

$$\left\{\frac{1}{\sigma \sqrt{n}}{S}_n(t);t\in [0,1]\right\}\to \{B(t);t\in [0,1]\},\mathrm{weakly}\mathrm{in}D,$$

where $\{B(t)\}$ is a standard Brownian motion; see, e.g., ([19], Theorem E, p. 4). In particular, this implies that

$$\frac{1}{\sigma \sqrt{n}}{S}_n(1)\to B(1)\sim N(0,1),\mathrm{in} \mathrm{distribution},$$

which is exactly (9). □

Theorem 6

(LIL for the conditional information density). Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate (4). Then:

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{\lim \sup }\frac{-logP(X_1^n|Y_1^n)-H(X_1^n|Y_1^n)}{\sqrt{2n{log}_e{log}_{e}n}}=\sigma ,a.s.,\hfill \end{array}$$ (10)

$$\begin{array}{cc}\hfill and& \underset{n\to \infty }{\lim \inf }\frac{-logP(X_1^n|Y_1^n)-H(X_1^n|Y_1^n)}{\sqrt{2n{log}_e{log}_{e}n}}=-\sigma ,a.s.\hfill \end{array}$$ (11)

Proof. 

As in the proof of (8), it suffices to prove (10) with $nH(\mathit{X}|\mathit{Y})$ in place of $H(X_1^n|Y_1^n)$. However, this is immediate from Theorem A1, since, for a standard Brownian motion $\{B(t)\}$,

$$\underset{t\to \infty }{\lim \sup }\frac{B(t)}{\sqrt{2t{log}_e{log}_{e}t}}=1,\mathrm{a}.\mathrm{s}.,$$

see, e.g., ([31], Theorem 11.18). In addition, similarly for (11). □

3. Idealised LZ Compression with Side Information

Consider the following idealised version of Lempel-Ziv-like compression with side information. For a given source-side information pair $(\mathit{X},\mathit{Y})=\{(X_n,Y_n);n\in \mathbb{Z}\}$, the encoder and decoder both have access to the infinite past $(X_{-\infty }^0,Y_{-\infty }^0)$ and to the current side information $Y_1^n$. The encoder describes $X_1^n$ to the decoder as follows. First she searches for the first appearance of $(X_1^n,Y_1^n)$ in the past $(X_{-\infty }^0,Y_{-\infty }^0)$, that is, for the first $r\ge 1$ such that $(X_{-r+1}^{-r+n},Y_{-r+1}^{-r+n})=(X_1^n,Y_1^n)$. Then she counts how many times $Y_1^n$ appears in $Y_{-\infty }^0$ between locations $-r+1$ and 0, namely how many indices $1\le j<r$ there are, such that $Y_{-j+1}^{-j+n}=Y_1^n$. Say there are $({\mathcal{R}}_n-1)$ such js. She describes $X_1^n$ to the decoder by telling him to look at the ${\mathcal{R}}_n$th position where $Y_1^n$ appears in the past $Y_{-\infty }^0$, and read off the corresponding X string.

This description takes $\approx log{\mathcal{R}}_n$ bits, and, as it turns out, the resulting compression rate is asymptotically optimal: As $n\to \infty$, with probability 1,

$$\begin{array}{c}\hfill \frac{1}{n}log{\mathcal{R}}_n\to H(\mathit{X}|\mathit{Y}),\mathrm{bits}/\mathrm{symbol}.\end{array}$$ (12)

Moreover, it is second-order optimal, in that it achieves equality in the CLT and LIL bounds given in Theorems 3 and 4 of Section 2.

Our purpose in this section is to make these statements precise. We will prove (12) as well as its CLT and LIL refinements, generalising the corresponding results for recurrence times without side information in [24].

The use of recurrence times in understanding the Lempel-Ziv (LZ) family of algorithms was introduced by Willems [21] and Wyner and Ziv [22,32]. In terms of practical methods for compression with side information, Subrahmanya and Berger [9] proposed a side information analog of the sliding window LZ algorithm [33], and Uyematsu and Kuzuoka [10] proposed a side information version of the incremental parsing LZ algorithm [34]. The Subrahmanya-Berger algorithm was shown to be asymptotically optimal in [12,13]. Different types of LZ-like algorithms for compression with side information were also considered in [11].

Throughout this section, we assume $(\mathit{X},\mathit{Y})$ is a jointly stationary and ergodic source-side information pair, with values in the finite alphabets $\mathcal{X},\mathcal{Y}$, respectively. We use bold lower-case letters $\mathit{x},\mathit{y}$ without subscripts to denote infinite realizations $x_{-\infty }^{\infty },y_{-\infty }^{\infty }$ of $\mathit{X},\mathit{Y}$, and the corresponding bold capital letters $\mathit{X},\mathit{Y}$ without subscripts to denote the entire process, $\mathit{X}=X_{-\infty }^{\infty },\mathit{Y}=Y_{-\infty }^{\infty }$.

The main quantities of interest are the recurrence times defined next.

Definition 5

(Recurrence times). For a realization x of the process X, and $n\ge 1$, define the repeated recurrence times ${\mathcal{R}}_n^{(j)}(\mathit{x})$ of $x_1^n$, recursively, as:

$$\begin{array}{cc}\hfill {\mathcal{R}}_n^{(1)}(x)& =inf\{i\ge 1:x_{-i+1}^{-i+n}=x_1^n\},\hfill \\ \hfill {\mathcal{R}}_n^{(j)}(x)& =inf\{i>{\mathcal{R}}_n^{(j-1)}(x):x_{-i+1}^{-i+n}=x_1^n\},j>1.\hfill \end{array}$$

For a realization $(\mathit{x},\mathit{y})$ of the pair $(\mathit{X},\mathit{Y})$ and $n\ge 1$, the joint recurrence time ${\mathcal{R}}_n(\mathit{x},\mathit{y})$ of $(x_1^n,y_1^n)$ is defined as,

$${\mathcal{R}}_n(\mathit{x},\mathit{y})=inf\{i\ge 1:{(x,y)}_{-i+1}^{-i+n}={(x,y)}_1^n\},$$

and the conditional recurrence time ${\mathcal{R}}_n(\mathit{x}|\mathit{y})$ of $x_1^n$ among the appearances $y_1^n$ is:

$${\mathcal{R}}_n(\mathit{x}|\mathit{y})=inf\left\{i\ge 1:x_{-{\mathcal{R}}_n^{(i)}(y)+1}^{-{\mathcal{R}}_n^{(i)}(y)+n}=x_1^n\right\}.$$

An important tool in the asymptotic analysis of recurrence times is Kac’s Theorem [35]. Its conditional version in Theorem 7 was first established in [12] using Kakutani’s induced transformation [36,37].

Theorem 7

(Conditional Kac’s theorem). [12] Suppose $(\mathit{X},\mathit{Y})$ is a jointly stationary and ergodic source-side information pair. For any pair of strings $x_1^n\in {\mathcal{X}}^n$, $y_1^n\in {\mathcal{Y}}^n$:

$$\mathbb{E}[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})|X_1^n=x_1^n,Y_1^n=y_1^n]=\frac{1}{P(x_1^n|y_1^n)}.$$

The following result states that we can asymptotically approximate $log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})$ by the conditional information density not just in expectation as in Kac’s theorem, but also with probability 1. Its proof is in Appendix B.

Theorem 8.

Suppose $(\mathit{X},\mathit{Y})$ is a jointly stationary and ergodic source-side information pair. For any sequence $\{c_n\}$ of non-negative real numbers such that ${\sum }_{n}n{2}^{-c_n}<\infty$, we have:

$$\begin{array}{cc}\hfill (i)& log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})-log\left(\frac{1}{P(X_1^n|Y_1^n)}\right)\le c_n,\mathit{eventually}a.s.\hfill \\ \hfill (ii)& log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})-log\left(\frac{1}{P(X_1^n|Y_1^n,Y_{-\infty }^0,X_{-\infty }^0)}\right)\ge -c_n,\mathit{eventually}a.s.\hfill \\ \hfill (iii)& log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})-log\left(\frac{P(Y_1^n|Y_{-\infty }^0)}{P(X_1^n,Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}\right)\ge -2c_n,\mathit{eventually}a.s.\hfill \end{array}$$

Next we state the main consequences of Theorem 8 that we will need. Recall the definition of the coefficients ${\gamma }^{(\mathit{Z})}(d)$ from Section 2.4. Corollary 1 is proved in Appendix B.

Corollary 1.

Suppose $(\mathit{X},\mathit{Y})$ are jointly stationary and ergodic.

• (a) 
  If, in addition, ${\sum }_d{\gamma }^{(X,Y)}(d)<\infty$ and ${\sum }_d{\gamma }^{(Y)}(d)<\infty$, then for any $\beta >0$:

  $$log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n)]=o(n^{\beta }),a.s.$$
• (b) 
  In the general jointly ergodic case, we have:

  $$log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n)]=o(n),a.s.$$

From part $(b)$ combined with the Shannon-McMillan-Breiman theorem as in (2), we obtain the result (12) promised in the beginning of this section:

$$\underset{n\to \infty }{lim}\frac{1}{n}log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})\to H(\mathit{X}|\mathit{Y}),\mathrm{a}.\mathrm{s}.$$

This was first established in [12]. However, at this point we have already done the work required to obtain much finer asymptotic results for the conditional recurrence time.

For any pair of infinite realizations $(\mathit{x},\mathit{y})$ of $(\mathit{X},\mathit{Y})$, let $\{{\mathcal{R}}^{(\mathit{x}|\mathit{y})}(t);t\ge 0\}$ be the continuous-time path, defined as:

$$\begin{array}{ccc}\hfill {\mathcal{R}}^{(\mathit{x}|\mathit{y})}(t)& =& 0,\mathrm{for}t<1,\hfill \\ \hfill {\mathcal{R}}^{(\mathit{x}|\mathit{y})}(t)& =& log{\mathcal{R}}_{\lfloor t\rfloor }(\mathit{x}|\mathit{y})-\lfloor t\rfloor H(\mathit{X}|\mathit{Y}),\mathrm{for}t\ge 1.\hfill \end{array}$$

The following theorem is a direct consequence of Corollary 1 $(a)$ combined with Theorem A1 in the Appendix A. Recall Assumption 1 from Section 2.4.

Theorem 9.

Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate. Then $\{{\mathcal{R}}^{(\mathit{X}|\mathit{Y})}(t)\}$ can be redefined on a richer probability space that contains a standard Brownian motion $\{B(t);t\ge 0\}$ such that for any $\lambda <1/294$:

$${\mathcal{R}}^{(\mathit{X}|\mathit{Y})}(t)-\sigma B(t)=O(t^{1/2-\lambda }),a.s.$$

Two immediate consequences of Theorem 9 are the following:

Theorem 10

(CLT and LIL for the conditional recurrence times). Suppose $(\mathit{X},\mathit{Y})$ satisfy Assumption 1, and let ${\sigma }^2={\sigma }^2(\mathit{X}|\mathit{Y})>0$ denote the conditional varentropy rate. Then:

$$\begin{array}{cc}\hfill (a)& \frac{log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})-H(X_1^n|Y_1^n)}{\sqrt{n}}\to N(0,{\sigma }^2),\mathit{in} \mathit{distribution},asn\to \infty .\hfill \\ \hfill (b)& \underset{n\to \infty }{\lim \sup }\frac{log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})-H(X_1^n|Y_1^n)}{\sqrt{2n{log}_e{log}_{e}n}}=\sigma ,a.s.\hfill \end{array}$$

Appendix A. Invariance Principle for the Conditional Information Density

This Appendix is devoted to the proof of Theorem A1, which generalises the corresponding almost sure invariance principle of Philipp and Stout ([19], Theorem 9.1) for the (unconditional) information density $-logP(X_1^n)$.

Theorem A1.

Suppose $(\mathit{X},\mathit{Y})$ is a jointly stationary and ergodic process, satisfying the mixing conditions (3). For $t\ge 0$, let,

$$S(t)=logP(X_1^{\lfloor t\rfloor }|Y_1^{\lfloor t\rfloor })+tH(\mathit{X}|\mathit{Y}).$$ (A1)

Then the following series converges:

$$\begin{array}{cc}\hfill {\sigma }^2& =\mathbb{E}{\left[logP(X_0,Y_0|X_{-\infty }^{-1},Y_{-\infty }^{-1})+H(\mathit{X}|\mathit{Y})\right]}^2\hfill \\ \hfill & +2\sum _{k=1}^{\infty }\mathbb{E}\{\left[logP(X_0,Y_0|X_{-\infty }^{-1},Y_{-\infty }^{-1})+H(\mathit{X}|\mathit{Y})\right]\left[logP(X_k,Y_k|X_{-\infty }^{k-1},Y_{-\infty }^{k-1})+H(\mathit{X}|\mathit{Y})\right]\}.\hfill \end{array}$$

If ${\sigma }^2>0$, then, without changing its distribution, we can redefine the process $\{S(t);t\ge 0\}$ on a richer probability space that contains a standard Brownian motion $\{B(t);t\ge 0\}$, such that

$$S(t)-\sigma B(t)=O(t^{\frac{1}{2}-\lambda }),a.s.,$$ (A2)

as $t\to \infty$, for each $\lambda <1/294$.

To simplify the notation, we write $h=H(\mathit{X}|\mathit{Y})$ and define,

$$f_j=log\left(\frac{P(X_j,Y_j|X_{-\infty }^{j-1},Y_{-\infty }^{j-1})}{P(Y_j|Y_{-\infty }^{j-1})}\right),j\ge 0,$$ (A3)

so that for example, the variance ${\sigma }^2$ in the theorem becomes,

$$\begin{array}{c}\hfill {\sigma }^2=\mathbb{E}[{(f_0+h)}^2]+2\sum _{k=1}^{\infty }\mathbb{E}[(f_0+h)(f_k+h)].\end{array}$$ (A4)

Lemma A1.

If ${\sum }_d{\gamma }^{(\mathit{X},\mathit{Y})}(d)<\infty$ and ${\sum }_d{\gamma }^{(\mathit{Y})}(d)<\infty$ then, as $n\to \infty$:

$$\sum _{k=1}^n{f}_k-logP(X_1^n|Y_1^n)=O(1),a.s.$$

Proof. 

Let,

$$g_j=log\left(\frac{P(X_j,Y_j|X_1^{j-1},Y_1^{j-1})}{P(Y_j|Y_1^{j-1})}\right),j\ge 2,$$

and,

$$g_1=log\left(\frac{P(X_1,Y_1)}{P(Y_1)}\right)=logP(X_1|Y_1).$$

We have, for $k\ge 2$,

$$\begin{array}{cc}\hfill \mathbb{E}|f_k-g_k|\le & \mathbb{E}|logP(X_k,Y_k|X_{-\infty }^{k-1},Y_{-\infty }^{k-1})-logP(X_k,Y_k|X_1^{k-1},Y_1^{k-1})|\hfill \\ \hfill & +\mathbb{E}|logP(Y_k|Y_{-\infty }^{k-1})-logP(Y_k|Y_1^{k-1})|\hfill \\ \hfill \le & \sum _{x,y}\mathbb{E}|logP(X_k=x,Y_k=y|X_{-\infty }^{k-1},Y_{-\infty }^{k-1})-logP(X_k=x,Y_k=y|X_1^{k-1},Y_1^{k-1})|\hfill \\ \hfill & +\sum _y\mathbb{E}|logP(Y_k=y|Y_{-\infty }^{k-1})-logP(Y_k=y|Y_1^{k-1})|\hfill \\ \hfill \le & |\mathcal{X}||\mathcal{Y}|{\gamma }^{(X,Y)}(k-1)+|\mathcal{Y}|{\gamma }^{(Y)}(k-1).\hfill \end{array}$$

Therefore, ${\sum }_{k=1}^{\infty }\mathbb{E}|f_k-g_k|<\infty$, and by the monotone convergence theorem we have,

$$\sum _{k=1}^{\infty }|f_k-g_k|<\infty ,\mathrm{a}.\mathrm{s}.$$

Hence, as $n\to \infty$,

$$\left|\sum _{k=1}^n{f}_k-logP(X_1^n|Y_1^n)\right|\le \sum _{k=1}^n|f_k-g_k|=O(1),\mathrm{a}.\mathrm{s}.,$$

as claimed. □

The following bounds are established in the proof of ([19], Theorem 9.1):

Lemma A2.

Suppose $\mathit{Z}=\{Z_n;n\in \mathbb{Z}\}$ is a stationary and ergodic process on a finite alphabet, with entropy rate $H(\mathit{Z})$, and such that ${\alpha }^{(\mathit{Z})}(d)=O(d^{-336})$ and ${\gamma }^{(\mathit{Z})}(d)=O(d^{-48}),$ as $d\to \infty$.

Let $f_k^{(\mathit{Z})}=logP(Z_k|Z_{-\infty }^{k-1})$, $k\ge 0$, and put ${\eta }_n^{(\mathit{Z})}=f_n^{(\mathit{Z})}+H(\mathit{Z})$, $n\ge 0$. Then:

• 1.  For each $r>0$, $\mathbb{E}\left[|f_0^{(\mathit{Z})}{|}^r\right]<\infty .$
• 2. 
  For each $r\ge 2$ and $ϵ>0$,

  $$\mathbb{E}\left[|f_0^{(\mathit{Z})}-logP(Z_0|Z_{-k}^{-1}){|}^r\right]\le C(r,ϵ){({\gamma }^{(\mathit{Z})}(k))}^{\frac{1}{2}-ϵ},$$

  where $C(r,ϵ)$ is a constant depending only on r and ϵ.
• 3.  For a constant $C>0$ independent of n, $\parallel {\eta }_n^{(\mathit{Z})}{\parallel }_4\le C.$
• 4. 
  Let ${\eta }_{n\ell }^{(\mathit{Z})}=\mathbb{E}[{\eta }_n^{(\mathit{Z})}|{\mathcal{F}}_{n-\ell }^n]$. Then, as $\ell \to \infty$:

  $$\parallel {\eta }_n^{(\mathit{Z})}-{\eta }_{n\ell }^{(\mathit{Z})}{\parallel }_4=O({\ell }^{-11/2}).$$

Please note that under the assumptions of Theorem A1, the conclusions of Lemma A2 apply to Y as well as to the pair process $(\mathit{X},\mathit{Y})$.

Lemma A3.

For each $r>0$, we have, $\mathbb{E}[|f_0{|}^r]<\infty .$

Proof. 

Simple algebra shows that

$$f_0=f_0^{(\mathit{X},\mathit{Y})}-f_0^{(Y)}.$$

Therefore, by two applications of Lemma A2, part 1,

$$\parallel f_0{\parallel }_r\le \parallel f_0^{(\mathit{X},\mathit{Y})}{\parallel }_r+{\parallel f_0^{(\mathit{Y})}\parallel }_r<\infty .$$

□

The next bound follows from Lemma A2, part 2, upon applying the Minkowski inequality.

Lemma A4.

For each $r\ge 2$ and each $ϵ>0$,

$${∥f_0-log\left(\frac{P(X_0,Y_0|X_{-k}^{-1},Y_{-k}^{-1})}{P(Y_0|Y_{-k}^{-1})}\right)∥}_r\le C_1(r,ϵ){[{\gamma }^{(\mathit{X},\mathit{Y})}(k)]}^{\frac{1-2ϵ}{2r}}+C_2(r,ϵ){[{\gamma }^{(\mathit{Y})}(k)]}^{\frac{1-2ϵ}{2r}}.$$

Lemma A5.

As $N\to \infty$:

$$\mathbb{E}\left\{{\left[\sum _{k\le N}(f_k+h)\right]}^2\right\}={\sigma }^{2}N+O(1).$$

Proof. 

First we examine the definition of the variance ${\sigma }^2$. The first term in (A4),

$$\parallel f_0{+h\parallel }_2^2\le (\parallel f_0{{\parallel }_2+h)}^2<\infty ,$$

is finite by Lemma A3. For the series in (A4), let, for $k\ge 0$,

$${\varphi }_k=log\left(\frac{P(X_k,Y_k|X_{\lfloor k/2\rfloor }^{k-1},Y_{\lfloor k/2\rfloor }^{k-1})}{P(Y_k|Y_{\lfloor k/2\rfloor }^{k-1})}\right),$$

and write,

$$\begin{array}{c}\hfill \mathbb{E}(f_0+h)(f_k+h)=\mathbb{E}(f_0+h)(f_k-{\varphi }_k)+\mathbb{E}(f_0+h)({\varphi }_k+h).\end{array}$$ (A5)

For the first term in the right-hand side above, we can bound, for any $ϵ>0$,

$$\begin{array}{cc}\hfill |\mathbb{E}(f_0+h)(f_k-{\varphi }_k)|& \stackrel{(a)}{\le }\parallel f_0{+h\parallel }_2{\parallel f_k-{\varphi }_k\parallel }_2\hfill \\ \hfill & \le [\parallel f_0{\parallel }_2+h]\parallel f_k-{\varphi }_k{\parallel }_2\hfill \\ \hfill & \stackrel{(b)}{\le }A{C}_1(2,ϵ){\left[{\gamma }^{(X,Y)}(\lfloor k/2\rfloor )\right]}^{\frac{1}{4}-\frac{1}{2}ϵ}+A{C}_2(2,ϵ){\left[{\gamma }^{(Y)}(\lfloor k/2\rfloor )\right]}^{\frac{1}{4}-\frac{1}{2}ϵ},\hfill \end{array}$$

where $(a)$ follows by the Cauchy-Schwarz inequality, and $(b)$ follows by Lemmas A3 and A4, with $A=\parallel f_0{\parallel }_2+h<\infty$. Therefore, taking $ϵ>0$ small enough and using the assumptions of Theorem A1,

$$\begin{array}{c}\hfill |\mathbb{E}(f_0+h)(f_k-{\varphi }_k)|=O(k^{-12+24ϵ})=O(k^{-3}),\mathrm{as}k\to \infty .\end{array}$$ (A6)

For the second term in (A5), we have that for any $r>0$, $\parallel {\varphi }_k{\parallel }_r<\infty$, uniformly over $k\ge 1$ by stationarity. Also, since $f_0,{\varphi }_k$ are measurable with respect to the $\sigma$-algebras generated by $(X_{-\infty }^0,Y_{-\infty }^0)$ and $(X_{\lfloor k/2\rfloor }^{\infty },Y_{\lfloor k/2\rfloor }^{\infty })$, respectively, we can apply ([19], Lemma 7.2.1) with $p=r=s=3$, to obtain that

$$|\mathbb{E}(f_0+h)({\varphi }_k+h)|\le 10\parallel f_0{+h\parallel }_3{\parallel {\varphi }_k+h\parallel }_3\alpha {(\lfloor k/2\rfloor )}^{1/3},$$

where $\alpha (k)={\alpha }^{(\mathit{X},\mathit{Y})}(k)=O(k^{-48})$, as $k\to \infty$, by assumption. Therefore, a fortiori,

$$\mathbb{E}(f_0+h)(f_k+h)=O(k^{-3}),$$

and combining this with (A6) and substituting in (A5), implies that ${\sigma }^2$ in (A4) is well defined and finite.

Finally, we have that as $N\to \infty$,

$$\begin{array}{cc}\hfill \mathbb{E}\left\{{\left[\sum _{k\le N}(f_k+h)\right]}^2\right\}& =N\mathbb{E}{(f_0+h)}^2+2\sum _{k=0}^{N-1}(N-k)\mathbb{E}(f_0+h)(f_k+h)\hfill \\ \hfill & =N{\sigma }^2-2\sum _{k=1}^{N-1}k\mathbb{E}(f_0+h)(f_k+h)-2N\sum _{k=N}^{\infty }\mathbb{E}(f_0+h)(f_k+h)\hfill \\ \hfill & ={\sigma }^{2}N+O(1),\hfill \end{array}$$

as required. □

Proof of Lemma 1. 

Lemma A5 states that the limit,

$$\underset{n\to \infty }{lim}\frac{1}{n}\mathrm{Var}\left(-log\left(\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right)\right).$$ (A7)

exists and is finite. Moreover, by Lemma A4, after an application of the Cauchy-Schwarz inequality, we have that as $n\to \infty$,

$$\mathbb{E}\left\{{\left[\sum _{k\le n}\left|log\left(\frac{P(X_k,Y_k|X_1^{k-1},Y_1^{k-1})}{P(Y_k|Y_1^{k-1})}\right)-log\left(\frac{P(X_k,Y_k|X_{-\infty }^{k-1},Y_{-\infty }^{k-1})}{P(Y_k|Y_{-\infty }^{k-1})}\right)\right|\right]}^2\right\}=O(1),$$

therefore,

$$\frac{1}{n}\left\{\mathrm{Var}\left(-logP(X_1^n|Y_1^n)\right)-\mathrm{Var}\left(-log\left(\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right)\right)\right\}=o(1).$$

Combining this with (A7) and the definition of ${\sigma }^2$, completes the proof. □

Proof of Theorem A1. 

Note that we have already established the fact that the expression for the variance converges to some ${\sigma }^2<\infty$. Also, in view of Lemma A1, it is sufficient to prove the theorem for $\{\tilde{S}(t)\}$ instead of $\{S(t)\}$, where:

$$\tilde{S}(t)=\sum _{k\le t}(f_k+h),t\ge 0.$$

This will be established by an application of ([19], Theorem 7.1), once we verify that conditions (7.1.4), (7.1.5), (7.1.6), (7.1.7) and (7.1.9) there are all satisfied.

For each $n\ge 0$, let ${\eta }_n=f_n+h$, where $f_n$ is defined in (A3) and h is the conditional entropy rate. First we observe that by stationarity,

$$\begin{array}{cc}\hfill \mathbb{E}[{\eta }_n]& =\mathbb{E}\left[log\left(\frac{P(X_n,Y_n|X_{-\infty }^{n-1},Y_{-\infty }^{n-1})}{P(Y_n|Y_{-\infty }^{n-1})}\right)\right]+H(\mathit{X}|\mathit{Y})\hfill \\ \hfill & =\mathbb{E}\left[logP(X_0,Y_0|X_{-\infty }^{-1},Y_{-\infty }^{-1})\right]+H(\mathit{X},\mathit{Y})-\mathbb{E}\left[logP(Y_0|Y_{-\infty }^{-1})\right]-H(\mathit{Y})\hfill \\ \hfill & =0,\hfill \end{array}$$ (A8)

where $H(\mathit{X},\mathit{Y})$ and $H(Y)$ denote the entropy rates of $(\mathit{X},\mathit{Y})$ and Y, respectively [2]. Observe that in the notation of Lemma A2, ${\eta }_n={\eta }_n^{(\mathit{X},\mathit{Y})}-{\eta }_n^{(\mathit{Y})}$, and ${\eta }_{n\ell }={\eta }_{n\ell }^{(\mathit{X},\mathit{Y})}-{\eta }_{n\ell }^{(\mathit{Y})}.$ By Lemma A2, parts 3 and 4, there exist a constant C, independent of n such that

$$\parallel {\eta }_n{\parallel }_4\le C<\infty ,$$ (A9)

and,

$$\parallel {\eta }_n-{\eta }_{n\ell }{\parallel }_4=O({\ell }^{-11/2}).$$ (A10)

In addition, from Lemma A5 we have,

$$\mathbb{E}\left\{{\left(\sum _{n\le N}\frac{1}{\sigma }{\eta }_n\right)}^2\right\}=N+O(1).$$ (A11)

From (A8)–(A11) and the assumption that ${\alpha }^{(\mathit{X},\mathit{Y})}(d)=O(d^{-336})$, we have that all of the conditions (7.1.4), (7.1.5), (7.1.6), (7.1.7) and (7.1.9) of ([19], Theorem 7.1) are satisfied for the random variables $\{{\eta }_n/\sigma \}$, with $\delta =2$. Therefore, $\{\tilde{S}(t);t\ge 0\}$ can be redefined on a possibly richer probability space, where there exists a standard Brownian motion $\{B(t);t\ge 0\}$, such that as $t\to \infty$:

$$\frac{1}{\sigma }\tilde{S}(t)-B(t)=O(t^{1/2-\lambda }),\mathrm{a}.\mathrm{s}.$$

By Lemma A1, this completes the proof. □

Appendix B. Recurrence Times Proofs

In this appendix, we provide the proofs of some of the more technical results in Section 3. First we establish the following generalisation of ([2], Lemma 16.8.3).

Lemma A6.

Suppose $(\mathit{X},\mathit{Y})$ is an arbitrary source-side information pair. Then, for any sequence $\{t_n\}$ of non-negative real numbers such that ${\sum }_n{2}^{-t_n}<\infty$, we have:

$$log\frac{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\ge -t_n,\mathit{eventually}a.s.$$

Proof. 

Let $B(X_{-\infty }^0,Y_{-\infty }^0)\subset {\mathcal{Y}}^n$ denote the support of $P(·|X_{-\infty }^0,Y_{-\infty }^0)$. We can compute,

$$\begin{array}{cc}\hfill \mathbb{E}\left(\frac{P(Y_1^n|Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}\right)& =\mathbb{E}\left(\mathbb{E}\left(\frac{P(Y_1^n|Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}|Y_{-\infty }^0,X_{-\infty }^0\right)\right)\hfill \\ \hfill & =\mathbb{E}\left(\sum _{y_1^n\in B(X_{-\infty }^0,Y_{-\infty }^0)}\frac{P(y_1^n|Y_{-\infty }^0)}{P(y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}P(y_1^n|Y_{-\infty }^0,X_{-\infty }^0)\right)\hfill \\ \hfill & \le 1.\hfill \end{array}$$

By Markov’s inequality,

$$\mathbb{P}\left[log\left(\frac{P(Y_1^n|Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}\right)>t_n\right]=\mathbb{P}\left[\frac{P(Y_1^n|Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}>2^{t_n}\right]\le 2^{-t_n},$$

and so, by the Borel-Cantelli lemma,

$$log\frac{P(Y_1^n|Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}\le t_n,\mathrm{eventually}\mathrm{a}.\mathrm{s}.,$$

as claimed. □

Proof of Theorem 8. 

Let $K>0$ arbitrary. By Markov’s inequality and Kac’s theorem,

$$\begin{array}{cc}\hfill \mathbb{P}({\mathcal{R}}_n(\mathit{X}|\mathit{Y})>K|X_1^n=x_1^n,Y_1^n=y_1^n)& \le \frac{\mathbb{E}\left({\mathcal{R}}_n(\mathit{X}|\mathit{Y})|X_1^n=x_1^n,Y_1^n=y_1^n\right)}{K}\hfill \\ \hfill & =\frac{1}{KP(x_1^n|y_1^n)}.\hfill \end{array}$$

Taking $K=2^{c_n}/P(X_1^n|Y_1^n)$, we obtain,

$$\begin{array}{cc}\hfill & \mathbb{P}\left(log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n)]>c_n|X_1^n=x_1^n,Y_1^n=y_1^n\right)\hfill \\ \hfill & =\mathbb{P}\left({\mathcal{R}}_n(\mathit{X}|\mathit{Y})>\frac{2^{c_n}}{P(X_1^n|Y_1^n)}|X_1^n=x_1^n,Y_1^n=y_1^n\right)\hfill \\ \hfill & \le 2^{-c_n}.\hfill \end{array}$$

Averaging over all $x_1^n\in {\mathcal{X}}^n,y_1^n\in {\mathcal{Y}}^n$,

$$\mathbb{P}(log{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n)>c_n)\le 2^{-c_n},$$

and the Borel-Cantelli lemma gives $(i)$.

For $(ii)$ we first note that the probability,

$$\mathbb{P}\left(log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n,X_{-\infty }^0,Y_{-\infty }^0)]<-c_n|Y_1^n=y_1^n,X_{-\infty }^0=x_{-\infty }^0,Y_{-\infty }^0=y_{-\infty }^0\right)$$ (A12)

is the probability, under $P(X_1^n=·|Y_1^n=y_1^n,X_{-\infty }^0=x_{-\infty }^0,Y_{-\infty }^0=y_{-\infty }^0)$, of those $z_1^n$ such that

$$\begin{array}{c}\hfill P(X_1^n=z_1^n|X_{-\infty }^0,Y_{-\infty }^n)<\frac{2^{-c_n}}{{\mathcal{R}}_n(x_{-\infty }^0\ast z_1^n|y_{-\infty }^n)},\end{array}$$

where ‘*’ denotes the concatenation of strings. Let $G_n=G_n(x_{-\infty }^0,y_{-\infty }^n)\subset {\mathcal{X}}^n$ denote the set of all such $z_1^n$. Then the probability in (A12) is,

$$\sum _{z_n\in G_n}P(z_1^n|x_{-\infty }^0,y_{-\infty }^n)\le \sum _{z_n\in G_n}\frac{2^{-c_n}}{{\mathcal{R}}_n(x_{-\infty }^0\ast z_1^n|y_{-\infty }^n)}\le 2^{-c_n}\sum _{z_n\in {\mathcal{X}}^n}\frac{1}{{\mathcal{R}}_n(x_{-\infty }^0\ast z_1^n|y_{-\infty }^n)}.$$

Since both $x_{-\infty }^0$ and $y_{-\infty }^n$ are fixed, for each $j\ge 1$, there is exactly one $z_1^n\in {\mathcal{X}}^n$, such that ${\mathcal{R}}_n(x_{-\infty }^0\ast z_1^n|y_{-\infty }^n)=j$. Thus, the last sum is bound above by,

$$\sum _{j=1}^{{|\mathcal{X}|}^n}\frac{1}{j}\le Dn,$$

for some positive constant D. Therefore, the probability in (A12) is bounded above by $Dn{2}^{-c_n}$, which is independent of $x_{-\infty }^0,y_{-\infty }^n$ and, by assumption, summable over n. Hence, after averaging over all infinite sequences $x_{-\infty }^0,y_{-\infty }^n$, the Borel-Cantelli lemma gives $(ii)$.

For part $(iii)$ we have, eventually, almost surely,

$$\begin{array}{cc}\hfill log& \left[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})\frac{P(X_1^n,Y_1^n|Y_{-\infty }^0,X_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right]\hfill \\ \hfill & =log\left[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})\frac{P(X_1^n|Y_1^n,X_{-\infty }^0,Y_{-\infty }^0)P(Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right]\hfill \\ \hfill & =log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n,X_{-\infty }^0,Y_{-\infty }^0)]+log\left[\frac{P(Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right]\ge -2c_n,\hfill \end{array}$$

where the last inequality follows from $(ii)$ and Lemma A6, and we have shown $(iii)$. □

Proof of Corollary 1. 

If we take $c_n=ϵn^{\beta }$ in theorem 8, with $ϵ>0$ arbitrary, we get from $(i)$ and $(iii)$,

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{\lim \sup }\frac{1}{n^{\beta }}log[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})P(X_1^n|Y_1^n)]\le 0,\mathrm{a}.\mathrm{s}.\hfill \end{array}$$ (A13)

$$\begin{array}{cc}\hfill and& \underset{n\to \infty }{\lim \inf }\frac{1}{n^{\beta }}log\left[{\mathcal{R}}_n(\mathit{X}|\mathit{Y})\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right]\ge 0,\mathrm{a}.\mathrm{s}.\hfill \end{array}$$ (A14)

Hence, to prove $(a)$ it is sufficient to show that as $n\to \infty$,

$$logP(X_1^n|Y_1^n)-log\left[\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right]=O(1),\mathrm{a}.\mathrm{s}.,$$

which is exactly Lemma A1 in Appendix A.

To prove $(b)$, taking $\beta =1$ in (A13) and (A14), it suffices to show that

$$\underset{n\to \infty }{lim}\left\{\frac{1}{n}logP(X_1^n|Y_1^n)-\frac{1}{n}log\left(\frac{P(X_1^n,Y_1^n|X_{-\infty }^0,Y_{-\infty }^0)}{P(Y_1^n|Y_{-\infty }^0)}\right)\right\}=0,\mathrm{a}.\mathrm{s}.$$

However, the first term converges almost surely to $-H(\mathit{X}|\mathit{Y})$ by the Shannon-McMillan-Breiman theorem, as in (2), and the second term is,

$$-\frac{1}{n}\sum _{i=1}^{n}logP(X_i,Y_i|X_{-\infty }^{i-1},Y_{-\infty }^{i-1})+\frac{1}{n}\sum _{i=1}^{n}logP(Y_i|Y_{-\infty }^{i-1}),$$

which, by the ergodic theorem, converges almost surely to,

$$-\mathbb{E}[logP(X_0,Y_0|X_{-\infty }^0,Y_{-\infty }^0)]+\mathbb{E}[logP(Y_0|Y_{-\infty }^0)]=H(\mathit{X},\mathit{Y})-H(\mathit{Y})=H(\mathit{X}|\mathit{Y}).$$

This completes the proof. □

Author Contributions

Conceptualization, L.G. and I.K.; methodology, L.G. and I.K.; formal analysis, L.G. and I.K.; investigation, L.G. and I.K.; writing—original draft preparation, L.G. and I.K.; writing—review and editing, L.G. and I.K.; funding acquisition, I.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded in part by the Hellenic Foundation for Research and Innovation (H.F.R.I.) under the “First Call for H.F.R.I. Research Projects to support Faculty members and Researchers and the procurement of high-cost research equipment grant,” project number 1034, and also in part by EPSRC grant number RG94782.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.