Exponential Strong Converse for Successive Refinement with Causal Decoder Side Information

Abstract

We consider the k-user successive refinement problem with causal decoder side information and derive an exponential strong converse theorem. The rate-distortion region for the problem can be derived as a straightforward extension of the two-user case by Maor and Merhav (2008). We show that for any rate-distortion tuple outside the rate-distortion region of the k-user successive refinement problem with causal decoder side information, the joint excess-distortion probability approaches one exponentially fast. Our proof follows by judiciously adapting the recently proposed strong converse technique by Oohama using the information spectrum method, the variational form of the rate-distortion region and Hölder’s inequality. The lossy source coding problem with causal decoder side information considered by El Gamal and Weissman is a special case ($k=1$) of the current problem. Therefore, the exponential strong converse theorem for the El Gamal and Weissman problem follows as a corollary of our result.

1. Introduction

We consider the k-user successive refinement problem with causal decoder side information shown in Figure 1, which we refer to as the k-user causal successive refinement problem. The decoders aim to recover the source sequence based on the encoded symbols and causally available private side information sequences. Specifically, given the source sequence $X^n$, each encoder $f_j$ where $j\in \{1,\dots ,k\}$ compresses $X^n$ into a codeword $S_j$. At time $i\in \{1,\dots ,n\}$, for each $j\in \{1,\dots ,k\}$, the j-th user aims to recover the i-th source symbol using the codewords from encoders $(f_1,\dots ,f_j)$, the side information up to time i and a decoding function ${\varphi }_{j,i}$, i.e., ${\widehat{X}}_{j,i}={\varphi }_{j,i}(S_1,\dots ,S_j,Y_{j,1},\dots ,Y_{j,i})$. Finally, at time n, for all $j\in \{1,\dots ,k\}$, the j-th user outputs the source estimate ${\widehat{X}}_j^n$ which, under a distortion measure $d_j$, is required to be less than or equal to a specified distortion level $D_j$.

Figure 1.

Encoder-decoder system model for the k-user successive refinement problem with causal decoder side information at time $i\in [n]$. Each encoder $f_j$ where $j\in [k]$ compresses the source information into codewords $S_j$. Given accumulated side information $(Y_{j,1},\dots ,Y_{j,i})$ and the codewords $(S_1,\dots ,S_j)$, decoder ${\varphi }_{j,i}$ reproduces the i-th source symbol as ${\widehat{X}}_{j,i}$. At time n, for $j\in [k]$, the estimate ${\widehat{X}}_j^n$ for user j is required to satisfy distortion constraint $D_j$ under a distortion measure $d_j$.

The causal successive refinement problem was first considered by Maor and Merhav in [1] who fully characterized the rate-distortion region for the two-user version. Maor and Merhav showed that, unlike the case with non-causal side information [2,3], no special structure e.g., degradedness, is required between the side information $Y_1^n$ and $Y_2^n$. Furthermore, Maor and Merhav discussed the performance loss due to causal decoder side information compared with non-causal side information [2,3]. In general, for the k-user successive refinement problem, the loss of performance due to causal decoder side information can be derived using Theorem 1 of the present paper and the results in [2,3] for the k-user case, under certain conditions on the degradedness of the side information in [2,3].

However, Maor and Merhav only presented a weak converse in [1]. In this paper, we strengthen the result in [1] by providing an exponential strong converse theorem for the full k-user causal successive refinement problem, which states that the joint excess-distortion probability approaches one exponentially fast if the rate-distortion tuple falls outside the rate-distortion region.

1.1. Related Works

We first briefly summarize existing works on the successive refinement problem. The successive refinement problem was first considered by Equitz and Cover [4] and by Koshelev [5] who considered necessary and sufficient conditions for a source-distortion triple to be successively refinable. Rimoldi [6] fully characterized the rate-distortion region of the successive refinement problem under the joint excess-distortion probability criterion while Kanlis and Narayan [7] derived the excess-distortion exponent in the same setting. The second-order asymptotic analysis of No and Weissman [8], which provides approximations to finite blocklength performance and implies strong converse theorems, was derived under the marginal excess-distortion probabilities criteria. This analysis was extended to the joint excess-distortion probability criterion by Zhou, Tan and Motani [9]. Other frameworks for successive refinement decoding include [10,11,12,13].

The study of source coding with causal decoder side information was initiated by Weissman and El Gamal in [14] where they derived the rate-distortion function for the lossy source coding problem with causal side information at the decoders (i.e., $k=1$, see also [15], Chapter 11.2). Subsequently, Timo and Vellambi [16] characterized the rate-distortion regions of the Gu-Effros two-hop network [17] and the Gray-Wyner problem [18] with causal decoder side information; Maor and Merhav [19] derived the rate-distortion region for the successive refinement of the Heegard-Berger problem [20] with causal side information available at the decoders; Chia and Weissman [21] considered the cascade and triangular source coding problem with causal decoder side information. In all the aforementioned works, the authors used Fano’s inquality to prove a weak converse. The weak converse implies that as the blocklength tends to infinity, if the rate-distortion tuple falls outside the rate-distortion region, then the joint excess-distortion probability is bounded away from zero. However, in this paper, we prove an exponential strong converse theorem for the k-user causal successive refinement problem, which significantly strengthens the weak converse as it implies that the joint excess-distortion probability tends to one exponentially fast with respect to the blocklength if the rate-distortion tuple falls outside the rate-distortion region (cf. Theorem 3). As a corollary of our result, for any $\epsilon \in [0,1)$, the $\epsilon$-rate-distortion region (cf. Definition 2) remains the same as the rate-distortion region (cf. Equation (27)). Please note that with weak converse, one can only assert that the $\epsilon$-rate-distortion region equals the rate-distortion region when $\epsilon =0$. See [22] for yet another justification for the utility of a strong converse compared to a weak converse theorem.

As the information spectrum method will be used in this paper to derive an exponential strong converse theorem for the causal successive refinement problem, we briefly summarize the previous applications of this method to network information theory problems. In [23,24,25], Oohama used this method to derive exponential strong converses for the lossless source coding problem with one-helper [26,27] (i.e., the Wyner-Ahlswede-Körner (WAK) problem), the asymmetric broadcast channel problem [28], and the Wyner-Ziv problem [29] respectively. Furthermore, Oohama’s information spectrum method was also used to derive exponential strong converse theorems for content identification with lossy recovery [30] by Zhou, Tan, Yu and Motani [31] and for Wyner’s common information problem under the total variation distance measure [32] by Yu and Tan [33].

1.2. Main Contribution and Challenges

We consider the k-user causal successive refinement problem and present an exponential strong converse theorem. For given rates and blocklength, define the joint excess-distortion probability as the probability that any decoder incurs a distortion level greater than the specified distortion level (see (3)) and define the probability of correct decoding as the probability that all decoders satisfy the specified distortion levels (see (24)). Our proof proceeds as follows. First, we derive a non-asymptotic converse (finite blocklength upper) bound on the probability of correct decoding of any code for the k-user causal successive refinement problem using the information spectrum method. Subsequently, by using Cramér’s inequality and the variational formulation of the rate-distortion region, we show that the probability of correct decoding decays exponentially fast to zero as the blocklength tends to infinity if the rate-distortion tuple falls outside the rate-distortion region of the causal successive refinement problem.

As far as we are aware, this paper is the first to establish a strong converse theorem for any lossy source coding problem with causal decoder side information. Furthermore, our methods can be used to derive exponential strong converse theorems for other lossy source coding problems with causal decoder side information discussed in Section 1.1. In particular, since the lossy source coding problems with causal decoder side information in [1,14] are special cases of the k-user causal successive refinement problem, the exponential strong converse theorems for the problems in [1,14] follow as a corollary of our result.

To establish the strong converse in this paper, we must overcome several major technical challenges. The main difficulty lies in the fact that for the causal successive refinement problem, the side information is available to the decoder causally instead of non-causally. This causal nature of the side information makes the design of the decoder much more complicated and involved, which complicates the analysis of the joint excess-distortion probability. We find that classical strong converse techniques like the image size characterization [34] and the perturbation approach [35] cannot lead to a strong converse theorem due to the above-mentioned difficulty. However, it is possible that other approaches different from ours can be used to obtain a strong converse theorem for the current problem. For example, it is interesting to explore whether two recently proposed strong converse techniques in [36,37] can be used for this purpose considering the fact that the methods in [36,37] have been successfully applied to problems including the Wyner-Ziv problem [29], the Wyner-Ahlswede-Körner (WAK) problem [26,27] and hypothesis testing problems with communication constraints [38,39,40].

2. Problem Formulation and Existing Results

2.1. Notation

Random variables and their realizations are in upper (e.g., X) and lower case (e.g., x) respectively. Sets are denoted in calligraphic font (e.g., $\mathcal{X}$). We use ${\mathcal{X}}^{\mathrm{c}}$ to denote the complement of $\mathcal{X}$ and use $X^n:=(X_1,\dots ,X_n)$ to denote a random vector of length n. Furthermore, given any $j\in [n]$, we use $X^{n\backslash j}$ to denote $(X_1,\dots ,X_{j-1},X_{j+1},\dots ,X^n)$. We use ${\mathbb{R}}_{+}$ and $\mathbb{N}$ to denote the set of positive real numbers and integers respectively. Given two integers a and b, we use $[a:b]$ to denote the set of all integers between a and b and use $[a]$ to denote $[1:a]$. The set of all probability distributions on $\mathcal{X}$ is denoted as $\mathcal{P}(\mathcal{X})$ and the set of all conditional probability distributions from $\mathcal{X}$ to $\mathcal{Y}$ is denoted as $\mathcal{P}(\mathcal{Y}|\mathcal{X})$. For information-theoretic quantities such as entropy and mutual information, we follow the notation in [34]. In particular, when the joint distribution of $(X,Y)$ is $P_{XY}\in \mathcal{P}(\mathcal{X}\times \mathcal{Y})$, we use $I(P_X,P_{Y|X})$ and $I(X;Y)$ interchangeably.

2.2. Problem Formulation

Let $k\in \mathbb{N}$ be a fixed finite integer and let $P_{X{Y}^k}$ be a joint probability mass function (pmf) on the finite alphabet $\mathcal{X}\times ({\prod }_{j\in [k]}{\mathcal{Y}}_j)$ with its marginals denoted in the customary way, e.g., $P_X$, $P_{X{Y}_1}$. Throughout the paper, we consider memoryless sources $(X^n,Y_1^n,\dots ,Y_k^n)$, which are generated i.i.d. according to $P_{X{Y}^k}$. Let a finite alphabet ${\widehat{\mathcal{X}}}_j$ be the alphabet of the reproduced source symbol for user $j\in [k]$. Recall the encoder-decoder system model for the k-user causal successive refinement problem in Figure 1.

A formal definition of a code for the causal successive refinement problem is as follows.

Definition 1.

An $(n,M_1,\dots ,M_k)$-code for the causal successive refinement problem consists of

• k encoding functions

  $$\begin{array}{c}\hfill f_j:{\mathcal{X}}^n\to {\mathcal{M}}_j:=\{1,\dots ,M_j\},j\in [k],\end{array}$$ (1)
• and $kn$ decoding functions: for each $i\in [n]$

  $$\begin{array}{cc}\hfill {\varphi }_{j,i}:& (\prod _{l\in [j]}{\mathcal{M}}_l)\times {({\mathcal{Y}}_j)}^i\to {\widehat{\mathcal{X}}}_j,j\in [k].\hfill \end{array}$$ (2)

For $j\in [k]$, let $d_j:\mathcal{X}\times {\widehat{\mathcal{X}}}_j\to [0,\infty )$ be a distortion measure. Given the source sequence $x^n$ and a reproduced version ${\widehat{x}}_j^n$, we measure the distortion between them using the additive distortion measure $d_j(x^n,{\widehat{x}}_j^n):=\frac{1}{n}{\sum }_{i\in [n]}{d}_j(x_i,{\widehat{x}}_{j,i})$. To evaluate the performance of an $(n,M_1,\dots ,M_k)$-code for the causal successive refinement problem, given distortion specified levels $(D_1,\dots ,D_k)$, we consider the following joint excess-distortion probability

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{e}}^{(n)}(D_1,\dots ,D_k):=Pr\left\{\exists j\in [k]\mathrm{s}.\mathrm{t}.d_j(X^n,{\widehat{X}}_j^n)>D_j\right\}.\end{array}$$ (3)

For ease of notation, throughout the paper, we use $D^k$ to denote $(D_1,\dots ,D_k)$, $M^k$ to denote $(M_1,\dots ,M_k)$ and $R^k$ to denote $(R_1,\dots ,R_k)$.

Given $\epsilon \in (0,1)$, the $\epsilon$-rate-distortion region for the k-user causal successive refinement problem is defined as follows.

Definition 2.

Given any $\epsilon \in (0,1)$, a rate-distortion tuple $(R^k,D^k)$ is said to be ε-achievable if there exists a sequence of $(n,M^k)$-codes such that

$$\begin{array}{cc}\hfill \underset{n\to \infty }{limsup}\frac{1}{n}log{M}_1& \le R_1,\hfill \end{array}$$ (4)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{limsup}\frac{1}{n}log{M}_j& \le R_j-\sum _{l\in [j-1]}{R}_l,\forall j\in [2:k],\hfill \end{array}$$ (5)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{limsup}{\mathrm{P}}_{\mathrm{e}}^{(n)}(D^k)& \le \epsilon .\hfill \end{array}$$ (6)

The closure of the set of all ε-achievable rate-distortion tuples is called the ε-rate-distortion region and is denoted as $\mathcal{R}(\epsilon )$.

Please note that in Definition 2, $R_j$ is the sum rate of the first j decoders. Using Definition 2, the rate-distortion region for the problem is defined as

$$\begin{array}{cc}\hfill \mathcal{R}& :=\bigcap _{\epsilon \in (0,1)}\mathcal{R}(\epsilon ).\hfill \end{array}$$ (7)

2.3. Existing Results

For the two-user causal successive refinement problem, the rate-distortion region was fully characterized by Maor and Merhav (Theorem 1 in [1]). With slight generalization, the result can be extended to the k-user case.

For $j\in [k]$, let $W_j$ be a random variable taking values in a finite alphabet ${\mathcal{W}}_j$. For simplicity, throughout the paper, we let

$$\begin{array}{cc}\hfill T& :=(X,Y^k,W^k,{\widehat{X}}^k),\hfill \end{array}$$ (8)

and let $(t,\mathcal{T})$ be a particular realization of T and its alphabet set, respectively.

Define the following set of joint distributions:

$$\begin{array}{cc}\hfill {\mathcal{P}}^{\ast }& :=\{Q_T\in \mathcal{P}(\mathcal{T}):Q_{X{Y}^k}=P_{X{Y}^k},W^k-X-Y^k,|{\mathcal{W}}_1|\le |\mathcal{X}|+3,\mathrm{and}\forall j\in [k]:\hfill \\ & |{\mathcal{W}}_j|\le |\mathcal{X}|\left(\prod _{l\in [j-1]}|{\mathcal{W}}_l|\right)+1,{\widehat{X}}_j={\varphi }_j(W^j,Y_j)\mathrm{for}\mathrm{some}{\varphi }_j:\left(\prod _{l\in [j]}{\mathcal{W}}_l\right)\times {\mathcal{Y}}_j\to {\widehat{\mathcal{X}}}_j\}.\hfill \end{array}$$ (9)

Given any joint distribution $Q_T\in \mathcal{P}(\mathcal{T})$, define the following set of rate-distortion tuples

$$\begin{array}{cc}\hfill \mathcal{R}(Q_T):=\{(R^k,D^k):R_1& \ge I(Q_X,Q_{W_1|X}),D_1\ge \mathbb{E}[d_1(X,{\varphi }_1(W_1,Y_1))],\mathrm{and}\forall j\in [2:k]:\hfill \\ \hfill R_j-\sum _{l\in [j-1]}{R}_l& \ge I(Q_{X|W^{j-1}},Q_{W_j|X{W}^{j-1}}|Q_{W_{j-1}}),D_j\ge \mathbb{E}[d_j(X,{\varphi }_j(W^j,Y_j))]\}.\hfill \end{array}$$ (10)

For $k=2$, Maor and Merhav [1] defined the following information theoretical sets of rate-distortion tuples

$$\begin{array}{c}\hfill {\mathcal{R}}^{\ast }:=\bigcup _{Q_T\in {\mathcal{P}}^{\ast }}\mathcal{R}(Q_T).\end{array}$$ (11)

Theorem 1.

The rate-distortion region for the causal successive refinement problem satisfies

$$\begin{array}{c}\hfill \mathcal{R}={\mathcal{R}}^{\ast }.\end{array}$$ (12)

We remark that in [1], Maor and Merhav considered the average distortion criterion for $k=2$, i.e.,

$$\begin{array}{c}\hfill \underset{n\to \infty }{limsup}\mathbb{E}[d_j(X^n,{\widehat{X}}_j^n)]\le D_k,\forall j\in [k],\end{array}$$ (13)

instead of the vanishing joint excess-distortion probability criterion (see (6)) in Definition 2. However, with slight modification to the proof of [1], it can be verified (see Appendix A) that the rate-distortion region $\mathcal{R}$ under the vanishing joint excess-distortion probability criterion, is identical to the rate-distortion region ${\mathcal{R}}^{\ast }$ derived by Maor and Merhav under the average distortion criterion.

Theorem 1 implies that if a rate-distortion tuple falls outside the rate-distortion region, i.e., $(R^k,D^k)\notin \mathcal{R}$, then the joint excess-distortion probability ${\mathrm{P}}_{\mathrm{e}}^{(n)}(D^k)$ is bounded away from zero. We strengthen the converse proof of Theorem 1 by showing that if $(R^k,D^k)\notin \mathcal{R}$, then the joint excess-distortion probability ${\mathrm{P}}_{\mathrm{e}}^{(n)}(D^k)$ approaches one exponentially fast as the blocklength n tends to infinity.

3. Main Results

3.1. Preliminaries

In this subsection, we present necessary definitions and a key lemma before stating our main result.

Define the following set of distributions

$$\begin{array}{cc}\hfill \mathcal{Q}& :=\left\{Q_T\in \mathcal{P}(\mathcal{T}):|{\mathcal{W}}_j|\le {\left(|\mathcal{X}||\mathcal{Y}||\mathcal{Z}||{\widehat{\mathcal{X}}}_1||\widehat{{\mathcal{X}}_2}|\right)}^j,\forall j\in [k]\right\}.\hfill \end{array}$$ (14)

Throughout the paper, we use ${\alpha }^k$ to denote $({\alpha }_1,\dots ,{\alpha }_k)$ and use ${\beta }^k$ similarly. Given any $(\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}$ such that

$$\begin{array}{c}\hfill \sum _{i\in [k]}({\alpha }_i+{\beta }_i)=1,\end{array}$$ (15)

for any $Q_T\in \mathcal{Q}$, define the following linear combination of log likelihoods

$$\begin{array}{cc}\hfill {\omega }_{Q_T}^{(\mu ,{\alpha }^k,{\beta }^k)}(t)& :=log\frac{Q_X(x)}{P_X(x)}+log\frac{Q_{Y^k|X{W}^k}(y^k|x,w^k)}{P_{Y^k|X}(y^k|x)}+log\frac{Q_{X{Y}^{k\backslash 1}{W}^{k\backslash 1}|Y_1{W}_1{\widehat{X}}_1}(x,y^{k\backslash 1},w^{k\backslash 1}|y_1,w_1,{\widehat{x}}_1)}{Q_{X{Y}^{k\backslash 1}{W}^{k\backslash 1}|Y_1{W}_1}(x,y^{k\backslash 1},w^{k\backslash 1}|y_1,w_1)}\hfill \\ & +\sum _{j\in [2:k]}log\frac{Q_{{\widehat{X}}_j|X{Y}^k{W}^k{\widehat{X}}^{j-1}}({\widehat{x}}_j|x,y^k,w^k,{\widehat{x}}^{j-1})}{Q_{{\widehat{X}}_j|Y_j{W}^j}({\widehat{x}}_j|y_j,w^j)}+\mu {\alpha }_{1}log\frac{Q_{X|W_1}(x|w_1)}{P_X(x)}\hfill \\ & +\sum _{j\in [2:k]}\mu {\alpha }_{j}log\frac{Q_{X|W^j}(x|w^j)}{Q_{X|W^{j-1}}(x|w^{j-1})}+\sum _{j\in [k]}\mu {\beta }_j{d}_j(x,{\widehat{x}}_j).\hfill \end{array}$$ (16)

Given any $\theta \in {\mathbb{R}}_{+}$ and any $Q_T\in \mathcal{Q}$, define the negative cumulant generating function of ${\omega }_{Q_T}^{(\mu ,{\alpha }^k,{\beta }^k)}(·)$ as

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)& :=-log{\mathbb{E}}_{Q_T}\left[exp\left(-\theta {\omega }_{Q_T}^{(\mu ,{\alpha }^k,{\beta }^k)}(T)\right)\right].\hfill \end{array}$$ (17)

Furthermore, define the minimal negative cumulant generating function over distributions in $\mathcal{Q}$ as

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}& :=\underset{Q_T\in \mathcal{Q}}{min}{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T).\hfill \end{array}$$ (18)

Finally, given any rate-distortion tuple $(R^k,D^k)$, define

$$\begin{array}{cc}\hfill {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)& :={\alpha }_1{R}_1+{\beta }_1{D}_1+\sum _{j\in [2:k]}({\alpha }_j(R_j-\sum _{l\in [j-1]}{R}_l)+{\beta }_j{D}_j),\hfill \end{array}$$ (19)

$$\begin{array}{cc}\hfill F^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k)& :=\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}-\theta \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j},\hfill \end{array}$$ (20)

$$\begin{array}{cc}\hfill F(R^k,D^k)& :=\underset{(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}:{\sum }_{i\in [k]}({\alpha }_i+{\beta }_i)=1}{sup}{F}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k).\hfill \end{array}$$ (21)

With the above definitions, we have the following lemma establishing the properties of the exponent function $F(R^k,D^k)$.

Lemma 1.

The following holds.

• (i) 
  For any rate-distortion tuple outside the rate-distortion region, i.e., $(R^k,D^k)\notin \mathcal{R}$, we have

  $$\begin{array}{c}\hfill F(R^k,D^k)>0,\end{array}$$ (22)
• (ii) 
  For any rate-distortion tuple inside the rate-distortion region, i.e., $(R^k,D^k)\in \mathcal{R}$, we have

  $$\begin{array}{c}\hfill F(R^k,D^k)=0.\end{array}$$ (23)

The proof of Lemma 1 is inspired by Property 4 in [25], Lemma 2 in [31] and is given in Section 5. As will be shown in Theorem 2, the exponent function $F(R^k,D^k)$ is a lower bound on the exponent of the probability of correct decoding for the k-user causal successive refinement problem. Thus, Claim (i) in Lemma 1 is crucial to establish the exponential strong converse theorem which states that the joint excess-distortion probability (see (3)) approaches one exponentially fast with respect to the blocklength of the source sequences.

3.2. Main Result

Define the probability of correct decoding as

$$\begin{array}{cc}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)& :=1-{\mathrm{P}}_{\mathrm{e}}^{(n)}(D^k)=Pr\left\{\forall j\in [k],d_j(X^n,{\widehat{X}}_j^n)\le D_j\right\}.\hfill \end{array}$$ (24)

Theorem 2.

Given any $(n,M^k)$-code for the k-user causal successive refinement problem such that

$$\begin{array}{c}\hfill log{M}_1\le n{R}_1,\mathrm{and}\forall j\in [2:k],log{M}_j\le n\left(R_j-\sum _{l\in [j-1]}{R}_l\right),\end{array}$$ (25)

we have the following non-asymptotic upper bound on the probability of correct decoding

$$\begin{array}{cc}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)& \le (2k+3)exp(-nF(R^k,D^k)).\hfill \end{array}$$ (26)

The proof of Theorem 2 is given in Section 4. Several remarks are in order.

First, our result is non-asymptotic, i.e., the bound in (26) holds for any $n\in \mathbb{N}$. To prove Theorem 2, we adapt the recently proposed strong converse technique by Oohama [25] to analyze the probability of correct decoding. We first obtain a non-asymptotic upper bound using the information spectrum of log-likelihoods involved in the definition of ${\omega }_{Q_T}^{(\mu ,{\alpha }^k,{\beta }^k)}$ (see (16)) and then apply Cramér’s bound on large deviations (see e.g., Lemma 13 in [31]) to obtain an exponential type non-asymptotic upper bound. Subsequently, we apply the recursive method [25] and proceed similarly as in [31] to obtain the desired result. Our method can also be used to establish similar results for other source coding problems with causal decoder side information [16,19,21].

Second, we do not believe that classical strong converse techniques including the image size characterization [34] and the perturbation approach [35] can be used to obtain a strong converse theorem for the causal successive refinement problem (e.g., Theorem 3). The main obstacle is that the side information is available causally and thus complicates the decoding analysis significantly.

Invoking Lemma 1 and Theorem 2, we conclude that the exponent on the right hand side of (26) is positive if and only if the rate-distortion tuple is outside the rate-distortion region, which implies the following exponential strong converse theorem.

Theorem 3.

For any sequence of $(n,M^k)$-codes satisfying the rate constraints in (25), given any distortion levels $D^k$, we have that if $(R^k,D^k)\notin \mathcal{R}$, then the probability of correct decoding ${\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)$ decays exponentially fast to zero as the blocklength of the source sequences tends to infinity.

As a result of Theorem 3, we conclude that for every $\epsilon \in (0,1)$, the $\epsilon$-rate distortion region (see Definition 2) satisfies that

$$\begin{array}{c}\hfill \mathcal{R}(\epsilon )=\mathcal{R},\end{array}$$ (27)

i.e., a strong converse holds for the k-user causal successive refinement problem. Using the strong converse theorem and Marton’s change-of-measure technique [41], similarly to Theorem 5 in [31], we can also derive an upper bound on the exponent of the joint excess-distortion probability. Furthermore, applying the one-shot techniques in [42], we can also establish a non-asymptotic achievability bound. Applying the Berry-Esseen theorem to the achievability bound and analyzing the non-asymptotic converse bound in Theorem 2, similarly to [25], we conclude that the backoff from the rate-distortion region at finite blocklength scales on the order of $\mathsf{\Theta }(\frac{1}{\sqrt{n}})$. However, nailing down the exact second-order asymptotics [43,44] is challenging and is left for future work.

Our main results in Lemma 1, Theorems 2 and 3 can be specialized to the settings in [1,14] with $k=1$ and $k=2$ decoders (users) respectively.

4. Proof of the Non-Asymptotic Converse Bound (Theorem 2)

4.1. Preliminaries

Given any $(n,M^k)$-code with encoding functions $(f_1,\dots ,f_k)$ and and decoding functions ${\left\{({\varphi }_{1,i},\dots ,{\varphi }_{k,i})\right\}}_{i\in [n]}$, we define the following induced conditional distributions on the encoders and decoders: for each $j\in [k]$,

$$\begin{array}{cc}\hfill P_{S_j|X^n}(s_j|x^n)& :=1\{s_j=f_j(x^n)\},\hfill \end{array}$$ (28)

$$\begin{array}{cc}\hfill P_{{\widehat{X}}_j^n|S^j{Y}_j^n}({\widehat{x}}_j^n|s^j,y_j^n)& :=\prod _{i\in [n]}1\{{\widehat{x}}_{j,i}={\varphi }_{j,i}(s^j,y_{j,1},\dots ,y_{j,i})\}.\hfill \end{array}$$ (29)

For simplicity, in the following, we define

$$\begin{array}{cc}\hfill G& :=(X^n,Y_1^n,\dots ,Y_k^n,S^k,{\widehat{X}}_1^n,\dots ,{\widehat{X}}_k^n),\hfill \end{array}$$ (30)

and let $(g,\mathcal{G})$ be a particular realization and the alphabet of G respectively. With above definitions, we have that the distribution $P_G$ satisfies that for any $g\in \mathcal{G}$,

$$\begin{array}{ccc}\hfill P_G(g)& :=P_{X{Y}^k}^n(x^n,y_1^n,\dots ,y_k^n)\hfill & \hfill \left(\prod _{j\in [k]}{P}_{S_j|X^n}(s_j|x^n)\right)\left(\prod _{j\in [k]}{P}_{{\widehat{X}}_j^n|S^j{Y}_j^n}({\widehat{x}}_j^n|s^j,y_j^n)\right).\end{array}$$ (31)

In the remaining part of this section, all distributions denoted by P are induced by the joint distribution $P_G$.

To simplify the notation, given any $(i,j)\in [n]\times [k]$, we use $Y_{j,1}^{j,i}$ to denote $(Y_{j,1},\dots ,Y_{j,i})$ and we use $Y_{1,i}^{k,i}$ to denote $(Y_{1,i},\dots ,Y_{k,i})$. Similarly, we use $W_{1,i}^{k,i}$ and ${\widehat{X}}_{1,i}^{k,i}$. For each $i\in [n]$, let auxiliary random variables be $W_{1,i}:=(X^{i-1},Y_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1},S_1)$ and $W_{j,i}=S_j$ for all $j\in [2:k]$. Please note that as a function of $i\in [n]$, the Markov chain $(W_{1,i}^{k,i})\leftrightarrow X_i\leftrightarrow (Y_i,Z_i)$ holds under $P_G$. Throughout the paper, for each $i\in [n]$, we let

$$\begin{array}{cc}\hfill T_i& :=(X_i,Y_{1,i}^{k,i},W_{1,i}^{k,i},{\widehat{X}}_{1,i}^{k,i}),\hfill \end{array}$$ (32)

and let $(t_i,{\mathcal{T}}_i)$ be a particular realization of $T_i$ and the alphabet of $T_i$, respectively.

For each $i\in [n]$, let $Q_{C_i|D_i}$ be arbitrary distributions where $C_i\in {\mathcal{T}}_i$ and $D_i\in {\mathcal{T}}_i$. Given any positive real number $\eta$ and rate-distortion tuple $(R^k,D^k)$, define the following subsets of $\mathcal{G}$:

$$\begin{array}{cc}\hfill {\mathcal{B}}_1& :=\left\{g:0\ge \frac{1}{n}\sum _{i\in [n]}log\frac{Q_{X_i}(x_i)}{P_X(x_i)}-\eta \right\},\hfill \end{array}$$ (33)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_2& :=\hfill & \hfill \left\{g:0\ge \frac{1}{n}\sum _{i\in [n]}log\frac{Q_{Y_{1,i}^{k,i}|X_i{W}_{1,i}^{k,i}}(y_{1,i}^{k,i}|x_i,w_{1,i}^{k,i})}{P_{Y^k|X}(y_{1,i}^{k,i}|x_i)}-\eta \right\},\end{array}$$ (34)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_3& :=\hfill & \hfill \{g:0\ge \frac{1}{n}\sum _{i\in [n]}log\frac{Q_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}{\widehat{X}}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i},{\widehat{x}}_{1,i})}{P_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i})}-\eta \},\end{array}$$ (35)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_4& :=\hfill & \hfill \{g:0\ge \frac{1}{n}\sum _{i\in [n]}log\frac{Q_{{\widehat{X}}_{j,i}|X_i{Y}_{1,i}^{k,i}{W}_{1,i}^{k,i}{\widehat{X}}_{1,i}^{j-1,i}}({\widehat{x}}_{j,i}|x_i,y_{1,i}^{k,i},w_{1,i}^{k,i},{\widehat{x}}_{1,i}^{j-1,i})}{P_{{\widehat{X}}_{j,i}|Y_{j,i}{W}_{1,i}^{j,i}}({\widehat{x}}_{j,i}|y_{j,i},w_{1,i}^{j,i})}-\eta ,\forall j\in [2:k]\},\end{array}$$ (36)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_5& :=\hfill & \hfill \{g:R_1\ge \frac{1}{n}\sum _{i\in [n]}log\frac{P_{X_i|W_{1,i}}(x_i|w_{1,i})}{P_X(x_i)}-\eta \},\end{array}$$ (37)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_6& :=\hfill & \hfill \{g:R_j-\sum _{l\in [j-1]}{R}_l\ge \frac{1}{n}\sum _{i\in [n]}log\frac{P_{X_i|W_{1,i}^{j,i}}(x_i|w_{1,i}^{j,i})}{P_{X_i|W_{1,i}^{j-1,i}}(x_i|w_{1,i}^{j-1,i})}-\eta ,\forall j\in [2:k]\}\end{array}$$ (38)

$$\begin{array}{ccc}\hfill {\mathcal{B}}_7& :=\hfill & \hfill \{g:D_j\ge \frac{1}{n}\sum _{i\in [n]}logexp(d_j(x_i,{\widehat{x}}_{j,i})),\forall j\in [k]\}.\end{array}$$ (39)

4.2. Proof Steps of Theorem 2

We first present the following non-asymptotic upper bound on the probability of correct decoding using the information spectrum method.

Lemma 2.

For any $(n,M^k)$-code satisfying (25), given any distortion levels $D^k$, we have

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)\le Pr\left\{\bigcap _{i\in [7]}{\mathcal{B}}_i\right\}+(2k+2)exp(-n\eta ).\end{array}$$ (40)

The proof of Lemma 2 is given in Appendix B and is divided into two steps. First, we derive a n-letter non-asymptotic upper bound which holds for certain arbitrary n-letter auxiliary distributions. Subsequently, we single-letterize the derived bound by proper choice of auxiliary distributions and careful decomposition of induced distributions of $P_G$.

Subsequently, we will apply Cramér’s bound on Lemma 2 to obtain an exponential type non-asymptotic upper bound on the probability of correct decoding. For simplicity, we will use $P_i$ to denote $P_{T_i}$ and use $Q_i$ to denote $Q_{T_i}$. To present our next result, we need the following definitions. Given any $\mu \in {\mathbb{R}}_{+}$ and any $({\alpha }^k,{\beta }^k)\in {[0,1]}^{2k}$ satisfying (15), let $f_{Q_i,P_i}^{({\alpha }^k,{\beta }^k)}(t_i)$ be the weighted sum of log likelihood terms in the summands to the right of the inequalities in ${\left\{{\mathcal{B}}_i\right\}}_{i\in [7]}$, i.e.,

$$\begin{array}{cc}\hfill f_{Q_i,P_i}^{({\alpha }^k,{\beta }^k)}(t_i)& :=log\frac{Q_{X_i}(x_i)}{P_X(x_i)}+log\frac{Q_{Y_{1,i}^{k,i}|X_i{W}_{1,i}^{k,i}}(y_{1,i}^{k,i}|x_i,w_{1,i}^{k,i})}{P_{Y^k|X}(y_{1,i}^{k,i}|x_i)}+log\frac{Q_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}{\widehat{X}}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i},{\widehat{x}}_{1,i})}{P_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i})}\hfill \\ & +\sum _{j\in [2:k]}log\frac{Q_{{\widehat{X}}_{j,i}|X_i{Y}_{1,i}^{k,i}{W}_{1,i}^{k,i}{\widehat{X}}_i^{j-1}}({\widehat{x}}_{j,i}|x_i,y_{1,i}^{k,i},w_{1,i}^{k,i},{\widehat{x}}_{1,i}^{j-1,i})}{P_{{\widehat{X}}_{j,i}|Y_{j,i}{W}_{1,i}^{j,i}}({\widehat{x}}_{j,i}|y_{j,i},w_{1,i}^{j,i})}+\mu {\alpha }_{1}log\frac{P_{X_i|W_{1,i}}(x_i|w_{1,i})}{P_X(x_i)}\hfill \\ & +\sum _{j\in [2:k]}\mu {\alpha }_{j}log\frac{P_{X_i|W_{1,i}^{j,i}}(x_i|w_{1,i}^{j,i})}{P_{X_i|W_{1,i}^{j-1,i}}(x_i|w_{1,i}^{j-1,i})}+\sum _{j\in [k]}\mu {\beta }_j{d}_j(x_i,{\widehat{x}}_{j,i}).\hfill \end{array}$$ (41)

Furthermore, given any non-negative real number $\lambda \in {\mathbb{R}}_{+}$, define the following negative cumulant generating function

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})& :=-log\mathbb{E}\left[exp\left(-\lambda \sum _{i\in [n]}{f}_{Q_i,P_i}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_i)\right)\right].\hfill \end{array}$$ (42)

Recall the definition of ${\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)$ in (19). Please note that ${\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)$ is a linear combination of the rate-distortion tuple. Using Lemma 2 and Cramér’s bound (Lemma 13 in [31]), we obtain the following non-asymptotic exponential type upper bound on the probability of correct decoding, whose proof is given in in Appendix D.

Lemma 3.

For any $(n,M^k)$-code satisfying the conditions in Lemma 2, given any distortion levels $D^k$, we have

$$\begin{array}{cc}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)& \le (2k+3)exp\left(-n\frac{\frac{1}{n}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})-\lambda \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+\lambda (k+2+{\sum }_{j\in [k]}\mu {\alpha }_j)}\right).\hfill \end{array}$$ (43)

For subsequent analyses, let ${\underset{\_}{\mathsf{\Omega }}}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\left\{P_i\right\}}_{i\in [n]})$ be the lower bound on the Q-maximal negative cumulant generating function ${\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})$ obtained by optimizing over the choice of auxiliary distributions ${\left\{Q_i\right\}}_{i\in [n]}$, i.e.,

$$\begin{array}{cc}\hfill {\underset{\_}{\mathsf{\Omega }}}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\left\{P_i\right\}}_{i\in [n]})& :=\underset{n\in \mathbb{N}}{inf}\underset{{\left\{Q_i\right\}}_{i\in [n]}}{sup}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]}).\hfill \end{array}$$ (44)

Here the supremum over ${\left\{Q_i\right\}}_{i\in [n]}$ is taken since we want the bound to hold for favorable auxiliary distributions and the infimum over $n\in \mathbb{N}$ is taken to yield a non-asymptotic bound.

In the following, we derive a relationship between ${\underset{\_}{\mathsf{\Omega }}}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\left\{P_i\right\}}_{i\in [n]})$ and ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}$ (cf. (18)), which, as we shall see later, is a crucial step in proving Theorem. For this purpose, given any $(\lambda ,\mu ,{\alpha }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^k$ such that

$$\begin{array}{c}\hfill \lambda (k+\sum _{j\in [k]}\mu {\alpha }_j)\le 1,\end{array}$$ (45)

let

$$\begin{array}{c}\hfill \theta :=\frac{\lambda }{1-k\lambda -{\sum }_{j\in [k]}\lambda \mu {\alpha }_j}.\end{array}$$ (46)

Then we have the following lemma which shows that ${\underset{\_}{\mathsf{\Omega }}}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\left\{P_i\right\}}_{i\in [n]})$ in Equation (44) can be lower bounded by a scaled version of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}$ in Equation (18).

Lemma 4.

Given any $(\lambda ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^3$ satisfying (15) and (45), for θ defined in (46), we have:

$$\begin{array}{c}\hfill {\underset{\_}{\mathsf{\Omega }}}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\left\{P_i\right\}}_{i\in [n]})\ge \frac{n{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}}{1+k\theta +{\sum }_{j\in [k]}\theta \mu {\alpha }_j}.\end{array}$$ (47)

The proof of Lemma 4 uses Hölder’s inequality and the recursive method in [25] and is given in Appendix E.

Combining Lemmas 3 and 4, we conclude that for any $(n,M^k)$-code satisfying the conditions in Lemma 2 and for any $(\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^3$, given any $\lambda \in {\mathbb{R}}_{+}$ satisfying (45), we have

$$\begin{array}{cc}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)& \le (2k+3)exp\left(-n\frac{\frac{1}{n}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})-\lambda \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+\lambda (k+2+{\sum }_{j\in [k]}\mu {\alpha }_j)}\right)\hfill \end{array}$$ (48)

$$\begin{array}{cc}\hfill & \le (2k+3)exp\left(-n\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}-\theta \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j}\right)\hfill \end{array}$$ (49)

$$\begin{array}{cc}\hfill & \le (2k+3)exp\left(-n{F}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k)\right),\hfill \end{array}$$ (50)

where (49) follows from the definitions of ${\kappa }^{({\alpha }^k,{\beta }^k)}(·)$ in (19) and $\theta$ in (46), and (50) is simply due to the definition of $F^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(·)$ in (20).

5. Proof of Properties of Strong Converse Exponent: Proof of Lemma 1

5.1. Alternative Expressions for the Rate-Distortion Region

In this section, we present preliminaries for the proof of Lemma 1, including several definitions and two alternative characterizations of the rate-distortion region $\mathcal{R}$ (cf. (7)).

Recall that we use $Y^{k\backslash j}$ to denote $(Y_1,\dots ,Y_{j-1},Y_{j+1},\dots ,Y_k)$. First, paralleling (9), we define the following set of joint distributions

$$\begin{array}{cc}\hfill \mathcal{P}& :=\{Q_T\in \mathcal{P}(\mathcal{T}):Q_{X{Y}^k}=P_{X{Y}^k},W^k\leftrightarrow X\leftrightarrow Y^k,\mathrm{and}\forall j\in [k]:\hfill \\ & |{\mathcal{W}}_j{|\le (|\mathcal{X}|+1)}^j,{\widehat{X}}_j\leftrightarrow (W^j,Y_j)\leftrightarrow (X,Y^{k\backslash j},W_{j+1}^k,{\widehat{X}}^{j-1})\}.\hfill \end{array}$$ (51)

Please note that compared with (9), the deterministic decoding functions ${\varphi }_j$ are now replaced by stochastic functions, which are characterized by transition matrices and induce Markov chains, and the cardinality bounds on auxiliary random variables are changed accordingly. Using the definitions of $\mathcal{P}$ and $\mathcal{R}(Q_T)$ (cf. (10)), we can define the following rate-distortion region denoted by ${\mathcal{R}}_{\mathrm{ran}}$ where the subscript “ran” refers to the randomness of the stochastic functions in the definition of $\mathcal{P}$:

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{ran}}& :=\bigcup _{Q_T\in \mathcal{P}}\mathcal{R}(Q_T).\hfill \end{array}$$ (52)

As we shall see later, ${\mathcal{R}}_{\mathrm{ran}}={\mathcal{R}}^{\ast }$.

To present the alternative characterization of the rate-distortion region using supporting hyperplanes, we need the following definitions. First, we let ${\mathcal{P}}_{\mathrm{sh}}$ be the following set of joint distributions

$$\begin{array}{cc}\hfill {\mathcal{P}}_{\mathrm{sh}}& :=\{Q_T\in \mathcal{P}(\mathcal{T}):Q_{X{Y}^k}=P_{X{Y}^k},W^k-X-Y^k,\mathrm{and}\forall j\in [k],\hfill \\ & |{\mathcal{W}}_j{|\le (|\mathcal{X}|)}^j,{\widehat{X}}_j-(W^j,Y_j)-(X,Y^{k\backslash j},W_{j+1}^k,{\widehat{X}}^{j-1})\}.\hfill \end{array}$$ (53)

Please note that ${\mathcal{P}}_{\mathrm{sh}}$ are the same as $\mathcal{P}$ (cf. (51)) except that the cardinality bounds are reduced. Given any $({\alpha }^k,{\beta }^k)\in {[0,1]}^{2k}$ satisfying (15), define the following linear combination of achievable rate-distortion tuples

$$\begin{array}{cc}\hfill {\mathrm{R}}^{({\alpha }^k,{\beta }^k)}& :=\underset{Q_T\in {\mathcal{P}}_{\mathrm{sh}}}{min}\left\{{\alpha }_{1}I(Q_X,Q_{W_1|X})+\sum _{j\in [2:k]}{\alpha }_{j}I(Q_{X|W^{j-1}},Q_{W_j|X{W}^{j-1}}|Q_{W^{j-1}})+\sum _{j\in [k]}{\beta }_j\mathbb{E}[d_j(X,{\widehat{X}}_j)]\right\}.\hfill \end{array}$$ (54)

Recall the definition of linear combination of rate-distortion tuples ${\kappa }^{·}(·)$ in (19) and let ${\mathcal{R}}_{\mathrm{sh}}$ be the following collection of rate-distortion tuples defined using supporting hyperplane ${\mathrm{R}}^{({\alpha }^k,{\beta }^k)}$:

$$\begin{array}{ccc}\hfill {\mathcal{R}}_{\mathrm{sh}}& :=\bigcap _{({\alpha }^k,{\beta }^k)\in {[0,1]}^{2k}:{\sum }_{i\in [k]}({\alpha }_i+{\beta }_i)=1}\{(R^k,D^k):\hfill & \hfill {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)\ge {\mathrm{R}}^{({\alpha }^k,{\beta }^k)}\}.\end{array}$$ (55)

Finally, recall the definitions of the rate-distortion region $\mathcal{R}$ in (7) and the characterization ${\mathcal{R}}^{\ast }$ in (11). Similarly to Properties 2 and 3 in [25], one can establish the following lemma, which states that: (i) the rate-distortion region $\mathcal{R}$ for the k-user causal successive refinement problem remains unchanged even if one uses stochastic decoding functions; and (ii) the rate-distortion region $\mathcal{R}$ has alternative characterization ${\mathcal{R}}_{\mathrm{sh}}$ in terms of supporting hyperplanes in (54).

Lemma 5.

The rate-distortion region for the causal successive refinement problem satisfies

$$\begin{array}{c}\hfill \mathcal{R}={\mathcal{R}}^{\ast }={\mathcal{R}}_{\mathrm{ran}}={\mathcal{R}}_{\mathrm{sh}}.\end{array}$$ (56)

5.2. Proof of Claim (i)

Recall that we use T (cf. (8)) to denote the collection of random variables $(X,Y^k,S^k,{\widehat{X}}^k)$ and use $t,\mathcal{T}$ similarly to denote a realization of T and its alphabet, respectively. For any $P_T\in {\mathcal{P}}_{\mathrm{sh}}$ (recall (53)), any $({\alpha }^k,{\beta }^k)\in {[0,1]}^{2k}$ satisfying (15) and any $\lambda \in {\mathbb{R}}_{+}$, for any $t\in \mathcal{T}$, paralleling (16) and (17), define the following linear combination of log likelihoods and its negative cumulative generating function:

$$\begin{array}{cc}\hfill {\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(t)& :={\alpha }_{1}log\frac{P_{X|W_1}(x|w_1)}{P_X(x)}+\sum _{j\in [2:k]}{\alpha }_{j}log\frac{P_{X|W^j}(x|w^j)}{P_{X|W_{j-1}}(x|w_{j-1})}+\sum _{j\in [k]}{\beta }_j{d}_j(x,{\widehat{x}}_j),\hfill \end{array}$$ (57)

$$\begin{array}{cc}\hfill {\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)& :=-log{\mathbb{E}}_{P_T}\left[exp(-\lambda {\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T))\right].\hfill \end{array}$$ (58)

For simplicity, we let

$$\begin{array}{c}\hfill {\alpha }^{+}:=\underset{j\in [k]}{max}{\alpha }_j.\end{array}$$ (59)

Furthermore, paralleling the steps used to go from (18) to (21) and recalling the definition of ${\kappa }^{({\alpha }^k,{\beta }^k)}(·)$ in (19), let

$$\begin{array}{cc}\hfill {\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}& :=\underset{P_T\in {\mathcal{P}}_{\mathrm{sh}}}{min}{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T),\hfill \end{array}$$ (60)

$$\begin{array}{cc}\hfill {\tilde{F}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(R^k,D^k)& :=\frac{{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}-\lambda {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{2k+3+\lambda {\alpha }^{+}+{\sum }_{j\in [2:k]}\lambda (2k+3){\alpha }_j+{\sum }_{l\in [k]}2\lambda {\alpha }_l},\hfill \end{array}$$ (61)

$$\begin{array}{cc}\hfill \tilde{F}(R^k,D^k)& :=\underset{(\lambda ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:{\sum }_{i\in [k]}({\alpha }_i+{\beta }_i)=1}{sup}{\tilde{F}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(R^k,D^k).\hfill \end{array}$$ (62)

To prove Claim (i), we will need the following two definitions of the tilted distribution and the dispersion function:

$$\begin{array}{cc}\hfill P_T^{(\lambda ,{\alpha }^k,{\beta }^k)}(t)& :=\frac{P_T(t)exp(-\lambda {\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(t))}{{\mathbb{E}}_{P_T}\left[exp(-\lambda {\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T))\right]},\hfill \end{array}$$ (63)

$$\begin{array}{cc}\hfill \rho & :=\underset{P_T\in {\mathcal{P}}_{\mathrm{sh}}}{sup}\underset{(\lambda ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:{\sum }_{i\in [k]}({\alpha }_i+{\beta }_i)=1}{sup}{\mathrm{Var}}_{P_T^{(\lambda ,{\alpha }^k,{\beta }^k)}}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right].\hfill \end{array}$$ (64)

Please note that $\rho$ is positive and finite.

The proof of Claim (i) in Lemma 1 is completed by the following lemma which relates $F(R^k,D^k)$ in Equation (21) to $\tilde{F}(R^k,D^k)$ in Equation (62).

Lemma 6.

The following holds.

• (i) 
  For any rate-distortion tuple $(R^k,D^k)$,

  $$\begin{array}{c}\hfill F(R^k,D^k)\ge \tilde{F}(R^k,D^k).\end{array}$$ (65)
• (ii) 
  For any rate-distortion tuple $(R^k,D^k)$ outside the rate-distortion region, i.e., $(R^k,D^k)\notin \mathcal{R}$, there exists $\delta \in (0,\rho ]$ such that:

  $$\begin{array}{c}\hfill \tilde{F}(R^k,D^k)\ge \frac{{\delta }^2}{2(2k+9)\rho }>0.\end{array}$$ (66)

The proof of Lemma 6 is inspired by [25,31] and given in Appendix F. To prove Lemma 6, we use the alternative characterizations of the rate-distortion region $\mathcal{R}$ in Lemma 5 and analyze the connections between the two exponent functions $F(R^k,D^k)$ and $\tilde{F}(R^k,D^k)$.

5.3. Proof of Claim (ii)

Recall the definition of the linear combination of rate-distortion tuple ${\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)$ in Equation (19). If a rate-distortion tuple falls inside the rate-distortion region, i.e., $(R^k,D^k)\in \mathcal{R}$, then there exists a distribution $Q_T^{\ast }\in {\mathcal{P}}_{\mathrm{sh}}$ (see (53)) such that for any $({\alpha }^k,{\beta }^k)\in {[0,1]}^{2k}$ satisfying (15), we have the following lower bound on ${\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)$:

$$\begin{array}{cc}\hfill {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)& \ge {\alpha }_{1}I(Q_{X_1}^{\ast },Q_{W_1|X_1}^{\ast })+{\beta }_1^{\ast }\mathbb{E}[d_1(X,{\widehat{X}}_1)]\hfill \\ & +\sum _{j\in [2:k]}({\alpha }_j^{\ast }I(Q_{X_1|W^{j-1}}^{\ast },Q_{W_j|X{W}^{j-1}}^{\ast }|Q_{W^{j-1}}^{\ast })+{\beta }_j^{\ast }\mathbb{E}[d_j(X,{\widehat{X}}_j)]).\hfill \end{array}$$ (67)

Recall the definition of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)$ in (17). Simple calculation establishes

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(0,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)& =0,\hfill \end{array}$$ (68)

$$\begin{array}{cc}\hfill \frac{\partial {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)}{\partial \theta }{|}_{\theta =0}& ={\mathbb{E}}_{Q_T}\left[{\omega }_{Q_T}^{\mu ,{\alpha }^k,{\beta }^k}(T)\right].\hfill \end{array}$$ (69)

Combining (68) and (69), by concavity of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)$ in $\theta$, it follows that for any $(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}$,

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)& \le \theta {\mathbb{E}}_{Q_T}\left[{\omega }_{Q_T}^{\mu ,{\alpha }^k,{\beta }^k}(T)\right].\hfill \end{array}$$ (70)

Using the definition of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}$ in (18), it follows that

$$\begin{array}{cc}\hfill {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}& \le \underset{Q_T\in {\mathcal{P}}_{\mathrm{sh}}}{min}{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)\hfill \end{array}$$ (71)

$$\begin{array}{cc}\hfill & \le \underset{Q_T\in {\mathcal{P}}_{\mathrm{sh}}}{min}\theta {\mathbb{E}}_{Q_T}\left[{\omega }_{Q_T}^{\mu ,{\alpha }^k,{\beta }^k}(T)\right]\hfill \\ \hfill & \le {\alpha }_{1}I(Q_{X_1}^{\ast },Q_{W_1|X_1}^{\ast })+{\beta }_1^{\ast }\mathbb{E}[d_1(X,{\widehat{X}}_1)]\hfill \end{array}$$ (72)

$$\begin{array}{cc}\hfill & +\sum _{j\in [2:k]}({\alpha }_j^{\ast }I(Q_{X_1|W^{j-1}}^{\ast },Q_{W_j|X{W}^{j-1}}^{\ast }|Q_{W^{j-1}}^{\ast })+{\beta }_j^{\ast }\mathbb{E}[d_j(X,{\widehat{X}}_j)])\hfill \end{array}$$ (73)

$$\begin{array}{cc}\hfill & \le \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k),\hfill \end{array}$$ (74)

where (71) follows from ${\mathcal{P}}_{\mathrm{sh}}\subseteq \mathcal{Q}$ (recall (14)), (72) follows from the result in (70), (73) follows from the definitions of ${\omega }_{Q_T}^{\mu ,{\alpha }^k,{\beta }^k}(t)$ in (17) and ${\mathcal{P}}_{\mathrm{sh}}$ in (53), and (74) follows from the result in (67).

Using the definition of $F^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k)$ in (21) and the result in (74), we conclude that for any $(R^k,D^k)\in \mathcal{R}$,

$$\begin{array}{c}\hfill F^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k)\le 0.\end{array}$$ (75)

The proof of Claim (ii) is completed by noting that

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}{F}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(R^k,D^k)=0.\end{array}$$ (76)

6. Conclusions

We considered the k-user causal successive refinement problem [1] and established an exponential strong converse theorem using the strong converse techniques proposed by Oohama [25]. Our work appears to be the first to derive a strong converse theorem for any source coding problem with causal decoder side information. The methods we adopted can also be used to obtain exponential strong converse theorems for other source coding problems with causal decoder side information. This paper further illustrates the usefulness and generality of Oohama’s information spectrum method in deriving exponential strong converse theorems. The discovered duality in [45] between source coding with decoder side information [46] and channel coding with encoder state information [47] suggests that Oohama’s techniques [25] can also be used to establish the strong converse theorem for channel coding with causal encoder state information, e.g., [48,49,50].

There are several natural future research directions. In Theorem 2, we presented only a lower bound on the strong converse exponent. It would be worthwhile to obtain an exact expression for the strong converse exponent and thus characterize the speed at which the probability of correct decoding decays exponentially fast with respect to the blocklength of source sequences when the rate-distortion tuple falls outside the rate-distortion region. Furthermore, one can explore whether the methods in this paper can be used to establish strong converse theorems for causal successive refinement under the logarithmic loss [51,52], which corresponds to soft decoding of each source symbol. Finally, one can also explore extensions to continuous alphabet by considering Gaussian memoryless sources under bounded distortion measures and derive second-order asymptotics [44,53,54,55,56] for the causal successive refinement problem.

Appendix A. Proof of Theorem 1

Replacing (6) with Definition 2, we can define the $\epsilon$-rate-distortion region ${\mathcal{R}}_{\mathrm{ad}}(\epsilon )$ under the average distortion criterion. Furthermore, let

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{ad}}& :=\bigcap _{\epsilon \in [0,1)}{\mathcal{R}}_{\mathrm{ad}}(\epsilon ).\hfill \end{array}$$ (A1)

Maor and Merhav [1] showed that for $k=2$,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{ad}}={\mathcal{R}}^{\ast }.\end{array}$$ (A2)

Actually, in Section 7 of [1], in order to prove that ${\mathcal{R}}^{\ast }\subseteq {\mathcal{R}}_{\mathrm{ad}}$, it was already shown that ${\mathcal{R}}^{\ast }\subseteq \mathcal{R}$. Furthermore, it is straightforward to show that the above results hold for any finite $k\in \mathbb{N}$. Thus, to prove Theorem 1, it suffices to show

$$\begin{array}{c}\hfill \mathcal{R}\subseteq {\mathcal{R}}^{\ast }={\mathcal{R}}_{\mathrm{ad}}.\end{array}$$ (A3)

For this purpose, given any $j\in [k]$, let

$$\begin{array}{cc}\hfill {\overline{d}}_j& :=\underset{(x,{\widehat{x}}_j)\in \mathcal{X}\times {\widehat{\mathcal{X}}}_j}{max}{d}_j(x,{\widehat{x}}_j).\hfill \end{array}$$ (A4)

From the problem formulation, we know that ${\overline{d}}_j<\infty$ for all $j\in [k]$. Now consider any rate-distortion tuple $(R^k,D^k)\in \mathcal{R}$, then we have (4) to (6). Therefore, for any $j\in [k]$,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{limsup}\mathbb{E}[d_j(X^n,{\widehat{X}}_j^n)]& \le \underset{n\to \infty }{limsup}\left(\mathbb{E}[d_j(X^n,{\widehat{X}}_j^n)1\{d_j(X^n,{\widehat{X}}_j^n)\le D_j\}]+{\overline{d}}_{j}Pr\{d_j(X^n,{\widehat{X}}_j^n)>D_j\}\right)\hfill \end{array}$$ (A5)

$$\begin{array}{c}\le D_j.\hfill \end{array}$$ (A6)

As a result, we have $(R^k,D^k)\in {\mathcal{R}}_{\mathrm{ad}}$. Thus establishes that $\mathcal{R}\subseteq {\mathcal{R}}_{\mathrm{ad}}={\mathcal{R}}^{\ast }$.

Appendix B. Proof of Lemma 2

Recall the definition of G and $\mathcal{G}$ in (30). Given any $C\in \mathcal{G}$ and $D\in \mathcal{G}$, let $Q_{C|D}$ be arbitrary distributions. For simplicity, given each $j\in [k]$, we use ${\mathbf{Y}}^j$ to denote $(Y_1^n,\dots ,Y_j^n)$ and use ${\mathbf{Y}}^{j\backslash l}$ to denote $(Y_1^n,\dots ,Y_{l-1}^n,Y_{l+1}^n,\dots ,Y_l^n)$ where $l\in [j]$. Similarly we use ${\widehat{\mathbf{X}}}^j$ and ${\widehat{\mathbf{X}}}^{j\backslash l}$.

Given any positive real number $\eta$, define the following sets:

$$\begin{array}{cc}\hfill {\mathcal{A}}_1& :=\left\{g:\frac{1}{n}log\frac{P_X^n(x^n)}{Q_{X^n}(x^n)}\ge -\eta \right\},\hfill \end{array}$$ (A7)

$$\begin{array}{ccc}\hfill {\mathcal{A}}_2& :=\hfill & \hfill \left\{g:\frac{1}{n}log\frac{P_{Y^k|X}^n({\mathbf{y}}^k|x^n)}{Q_{{\mathbf{Y}}^k|X^n{S}^k}({\mathbf{y}}^k|x^n,s^k)}\ge -\eta \right\},\end{array}$$ (A8)

$$\begin{array}{cc}\hfill {\mathcal{A}}_3& :=\left\{g:\frac{1}{n}log\frac{P_{X^n{\mathbf{Y}}^{k\backslash 1}{S}^{k\backslash 1}|Y_1^n{S}_1}(x^n,{\mathbf{y}}^{k\backslash 1},s^{k\backslash 1}|y_1^n,s_1)}{Q_{X^n{\mathbf{Y}}^{k\backslash 1}{S}^{k\backslash 1}|Y_1^n{S}_1{\widehat{X}}_1^n}(x^n,{\mathbf{y}}^{k\backslash 1},s^{k\backslash 1}|y_1^n,s_1,{\widehat{x}}_1^n)}\ge -\eta \right\},\hfill \end{array}$$ (A9)

$$\begin{array}{ccc}\hfill {\mathcal{A}}_4& :=\hfill & \hfill \left\{g:\frac{1}{n}log\frac{P_{{\widehat{X}}_j^n|Y_j^n{S}^j}({\widehat{x}}_j^n|y_j^n,s^j)}{Q_{{\widehat{X}}_j^n|X^n{\mathbf{Y}}^k{S}^k{\widehat{\mathbf{X}}}^{j-1}}({\widehat{x}}_j^n|x^n,{\mathbf{y}}^k,s^k,{\widehat{\mathbf{x}}}^{j-1})}\ge -\eta ,\forall j\in [2:k]\right\},\end{array}$$ (A10)

$$\begin{array}{cc}\hfill {\mathcal{A}}_5& :=\left\{g:R_1\ge \frac{1}{n}log\frac{P_{X^n|S_1}(x^n|s_1)}{P_X^n(x^n)}-\eta \right\},\hfill \end{array}$$ (A11)

$$\begin{array}{cc}\hfill {\mathcal{A}}_6& :=\left\{g:R_j-\sum _{l\in [j-1]}{R}_l\ge \frac{1}{n}log\frac{P_{X^n|S^j}(x^n|s^j)}{P_{X^n|S^{j-1}}(x^n|s^{j-1})}-\eta ,\forall j\in [2:k]\right\},\hfill \end{array}$$ (A12)

$$\begin{array}{ccc}\hfill {\mathcal{A}}_7& :=\left\{g:D_j\ge d_j(x^n,{\widehat{x}}_j^n)\forall j\in [k]\right\}\hfill & \hfill =\left\{g:D_j\ge \frac{1}{n}\sum _{i\in [n]}{d}_j(x_i,{\widehat{x}}_{j,i})\forall j\in [k]\right\}.\end{array}$$ (A13)

Then we have the following non-asymptotic upper bound on the probability of correct decoding.

Lemma A1.

Given any $(n,M^k)$-code satisfying (25) and any distortion levels $D^k$, we have

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)\le Pr\left\{\bigcap _{i\in [7]}{\mathcal{A}}_i\right\}+(2k+2)exp(-n\eta ).\end{array}$$ (A14)

The proof of Lemma A1 is given in Appendix C.

In the remainder of this subsection, we single-letterize the bound in Lemma A1. Recall that given any $(i,j)\in [n]\times [k]$, we use $Y_{j,1}^{j,i}$ to denote $(Y_{j,1},\dots ,Y_{j,i})$. Recalling that the distributions starting with P are all induced by the joint distribution $P_G$ in (31) and using the choice of auxiliary random variables $(W_{1,i},\dots ,W_{k,i},V_i)$, we have

$$\begin{array}{c}{P}_{X^n{\mathbf{Y}}^{k\backslash 1}{S}^{k\backslash 1}|Y_1^n{S}_1}(x^n,{\mathbf{y}}^{k\backslash 1},s^{k\backslash 1}|y_1^n,s_1)\hfill \end{array}$$

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{X_i{Y}_{2,i}^{k,i}{S}^{k\backslash 1}|X^{i-1},Y_{2,1}^{2,i-1},\dots ,Y_{k,1}^{k,i-1},Y_1^n,S_1}(x_i,y_{2,i}^{k,i},s^{k\backslash 1}|x^{i-1},y_{2,1}^{2,i-1},\dots ,y_{k,1}^{k,i-1},y_1^n,s_1)\hfill \end{array}$$ (A15)

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{X_i{Y}_{2,i}^{k,i}{S}^{k\backslash 1}|X^{i-1},Y_{1,1}^{1,i-1},\dots ,Y_{k,1}^{i-1},Y_{1,i},S_1}(x_i,y_{2,i}^{k,i},s^{k\backslash 1}|x^{i-1},y_{1,1}^{1,i-1},\dots ,y_{k,1}^{k,i-1},y_{1,i},s_1)\hfill \end{array}$$ (A16)

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i})\hfill \end{array}$$ (A17)

$$\begin{array}{cc}\hfill P_{{\widehat{X}}_j^n|Y_j^n{S}^j}({\widehat{x}}_j^n|y_j^n,s_j)& =\prod _{i\in [n]}{P}_{{\widehat{X}}_{j,i}|Y_{j,1}^{j,i}{S}^j}({\widehat{x}}_{j,i}|y_{j,1}^{j,i},s^j)\hfill \end{array}$$ (A18)

$$\begin{array}{cc}\hfill & =\prod _{i\in [n]}{P}_{{\widehat{X}}_{j,i}|X^{i-1},Y_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1},Y_{j,i},S^j}({\widehat{x}}_{j,i}|x^{i-1},y_{1,1}^{1,i-1},\dots ,y_{k,1}^{k,i-1},y_{j,i},s^j)\hfill \end{array}$$ (A19)

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{{\widehat{X}}_{j,i}|Y_{j,i}{W}_{1,i}^{k,i}}({\widehat{x}}_{j,i}|y_{j,i},w_{1,i}^{k,i}),\hfill \end{array}$$ (A20)

$$\begin{array}{cc}\hfill P_{X^n|S_1}(x^n|s_1)& =\prod _{i\in [n]}{P}_{X_i|X^{i-1}{S}_1}(x_i|x^{i-1},S_1)\hfill \end{array}$$ (A21)

$$\begin{array}{cc}\hfill & =\prod _{i\in [n]}{P}_{X_i|X^{i-1}{Y}_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1}{S}_1}(x_i|x^{i-1}{y}_{1,1}^{1,i-1},\dots ,y_{k,1}^{k,i-1},S_1)\hfill \end{array}$$ (A22)

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{X_i|W_{1,i}}(x_i|w_{1,i})\hfill \end{array}$$ (A23)

$$\begin{array}{cc}\hfill P_{X^n|S^{j-1}}(x^n|s^{j-1})& =\prod _{i\in [n]}{P}_{X_i|X^{i-1}{S}^{j-1}}(x_i|x^{i-1},s^{j-1})\hfill \end{array}$$ (A24)

$$\begin{array}{cc}\hfill & =\prod _{i\in [n]}{P}_{X_i|X^{i-1}{Y}_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1},S^{j-1}}(x_i|x^{i-1},y_{1,1}^{1,i-1},\dots ,y_{k,1}^{k,i-1},s^{j-1})\hfill \end{array}$$ (A25)

$$\begin{array}{c}=\prod _{i\in [n]}{P}_{X_i|W_{1,i}^{j-1,i}}(x_i,w_{1,i}^{j-1,i}),\hfill \end{array}$$ (A26)

$$\begin{array}{cc}\hfill P_{X^n|S^j}(x^n|s^j)& =\prod _{i\in [n]}{P}_{X_i|W_{1,i}^{j,i}}(x_i,w_{1,i}^{j,i}),\hfill \end{array}$$ (A27)

where (A16) follows from the Markov chain $(X_i,Y_{2,i}^{k,i},S_2^k)-(X^{i-1},Y_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1},Y_{1,i},S_1)-Y_{1,i+1}^{1,n}$, (A19) follows from the Markov chain ${\widehat{X}}_{j,i}-(Y_{j,1}^{j,i},S^j)-(X^{i-1},Y_{1,1}^{1,i-1},\dots ,Y_{j-1,1}^{j-1,i-1},Y_{j+1,1}^{j+1,i-1},\dots ,Y_{k,1}^{k,i-1})$, (A22) follows from the Markov chain $X_i-(X^{i-1},S_1)-(Y_1^{i-1},\dots ,Y_k^{i-1})$, and (A25) follows from the Markov chain $X_i-(X^{i-1},S^{j-1})-(Y_{1,1}^{1,i-1},\dots ,Y_{k,1}^{k,i-1})$.

Furthermore, recall that for $i\in [n]$, $Q_{C_i|D_i}$ are arbitrary distributions where $C_i\in {\mathcal{T}}_i$ and $D_i\in {\mathcal{T}}_i$. Please note that Lemma A1 holds for arbitrary choices of distributions $Q_{C|D}$ where $C\in \mathcal{G}$ and $D\in \mathcal{G}$. The proof of Lemma 2 is completed by using Lemma A1 with the following choices of auxiliary distributions and noting that ${\mathcal{B}}_7={\mathcal{A}}_7$:

$$\begin{array}{cc}\hfill Q_{X^n}(x^n)& :=\prod _{i\in [n]}{Q}_{X_i}(x_i),\hfill \end{array}$$ (A28)

$$\begin{array}{cc}\hfill Q_{{\mathbf{Y}}^k|X^n{S}^k}({\mathbf{y}}^k|x^n,s^k)& :=\prod _{i\in [n]}{Q}_{Y_{1,i}^{k,i}|X_i,W_{1,i}^{k,i}}(y_{1,i}^{k,i}|x_i,w_{1,i}^{k,i}),\hfill \end{array}$$ (A29)

$$\begin{array}{cc}\hfill Q_{X^n{\mathbf{Y}}^{k\backslash 1}{S}_2^k|Y_1^n{S}_1{\widehat{X}}_1^n}(x^n,{\mathbf{y}}^{k\backslash 1},s_2^k|y_1^n,s_1,{\widehat{x}}_1^n)& :=\prod _{i\in [n]}{Q}_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}{\widehat{X}}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i},{\widehat{x}}_{1,i})\hfill \end{array}$$ (A30)

$$\begin{array}{cc}\hfill Q_{{\widehat{X}}_j^n|X^n{\mathbf{Y}}^k{S}^k{\widehat{\mathbf{X}}}^{j-1}}({\widehat{x}}_j^n|x^n,{\mathbf{y}}^k,s^k,{\widehat{\mathbf{x}}}^{k\backslash j})& :=\prod _{i\in [n]}{Q}_{{\widehat{X}}_{j,i}|X_i,Y_{1,i}^{k,i},W_{1,i}^{k,i},{\widehat{X}}_{1,i}^{j-1,i}}({\widehat{x}}_{j,i}|x_i,y_{1,i}^{k,i},w_{1,i}^{k,i},{\widehat{x}}_{1,i}^{j-1,i}).\hfill \end{array}$$ (A31)

Appendix C. Proof of Lemma A1

Recall the definition of the probability of correct decoding ${\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)$ in (24) and the definitions of sets ${\left\{{\mathcal{A}}_j\right\}}_{j\in [7]}$ in (A7) to (A13). For any $(n,M^k)$-code, we have that

$$\begin{array}{cc}\hfill {\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)& =Pr\left\{{\mathcal{A}}_7\right\}\hfill \end{array}$$ (A32)

$$\begin{array}{c}=Pr\left\{{\mathcal{A}}_7\bigcap (\bigcap _{j\in [6]}{\mathcal{A}}_j)\right\}+Pr\left\{{\mathcal{A}}_7\bigcap (\bigcup _{j\in [6]}{\mathcal{A}}_j^{\mathrm{c}})\right\}\hfill \end{array}$$ (A33)

$$\begin{array}{cc}\hfill & \le Pr\left\{\bigcap _{j\in [7]}{\mathcal{A}}_j\right\}+\sum _{j\in [6]}Pr\left\{{\mathcal{A}}_j^{\mathrm{c}}\right\},\hfill \end{array}$$ (A34)

where (A34) follows from the union bound and the fact that $Pr\{\mathcal{A}\cap \mathcal{B}\}\le Pr\left\{\mathcal{B}\right\}$ for any two sets $\mathcal{A}$ and $\mathcal{B}$. The proof of Lemma A1 is completed by showing that

$$\begin{array}{c}\hfill \sum _{j\in [6]}Pr\left\{{\mathcal{A}}_j^{\mathrm{c}}\right\}\le (2k+2)exp(-n\eta ).\end{array}$$ (A35)

In the remainder of this subsection, we show that (A35) holds. Recall the joint distribution of G in (31). In the following, when we use a (conditional) distribution starting with P, we mean that the (conditional) distribution is induced by the joint distribution $P_G$ in (31).

Using the definition of ${\mathcal{A}}_1$ in (A7),

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_1^{\mathrm{c}}\right\}& =\sum _{x^n\in {\mathcal{X}}^n}{P}_X^n(x^n)1\{P_X^n(x^n)\le exp(-n\eta )Q_{X^n}(x^n)\}\hfill \end{array}$$ (A36)

$$\begin{array}{cc}\hfill & \le exp(-n\eta ).\hfill \end{array}$$ (A37)

Similarly to (A37), it follows that

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_2^{\mathrm{c}}\right\}& =\sum _{g\in {\mathcal{A}}_2^{\mathrm{c}}}{P}_G(g)\hfill \end{array}$$ (A38)

$$\begin{array}{c}=\sum _{x^n,s^k,{\mathbf{y}}^k}{P}_{X{Y}^k}(x^n,{\mathbf{y}}^k)\left(\prod _{j\in [k]}{P}_{S_j|X^n}(s_j|x^n)\right)1\{P_{Y^k|X}^n({\mathbf{y}}^k|x^n)\le exp(-n\eta )Q_{{\mathbf{Y}}^k|X^n{S}^k}({\mathbf{y}}^k|x^n,s^k)\}\hfill \end{array}$$ (A39)

$$\begin{array}{c}\le exp(-n\eta )\sum _{x^n,s^k,{\mathbf{y}}^k}{P}_X^n(x^n)Q_{{\mathbf{Y}}^k|X^n{S}^k}({\mathbf{y}}^k|x^n,s^k)\left(\prod _{j\in [k]}{P}_{S_j|X^n}(s_j|x^n)\right)\hfill \end{array}$$ (A40)

$$\begin{array}{c}\le exp(-n\eta ),\hfill \end{array}$$ (A41)

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_3^{\mathrm{c}}\right\}& =\sum _{g\in {\mathcal{A}}_3^{\mathrm{c}}}{P}_G(g)\hfill \end{array}$$ (A42)

$$\begin{array}{c}\le exp(-n\eta )\sum _{x^n,{\mathbf{y}}^k,s^k,{\widehat{x}}_1^n}{P}_{Y_1^n{S}_1}(y_1^n,s_1)P_{{\widehat{X}}_1^n|Y_1^n{S}_1}({\widehat{x}}_1^n|y_1^n,s_1)Q_{X^n{\mathbf{Y}}^{k\backslash 1}{S}_2^k|Y_1^n{S}_1{\widehat{X}}_1^n}(x^n,{\mathbf{y}}^{k\backslash 1},s_2^k|y_1^n,s_1,{\widehat{x}}_1^n)\hfill \end{array}$$ (A43)

$$\begin{array}{c}\le exp(-n\eta ),\hfill \end{array}$$ (A44)

Furthermore, using the definition of ${\mathcal{A}}_4$ in (A10) and the union bound,

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_4^{\mathrm{c}}\right\}& \le \sum _{j\in [2:k]}exp(-n\eta )\sum _{x^n,{\mathbf{y}}^k,s^k,{\widehat{\mathbf{x}}}^j}{P}_{X{Y}^k}^n(x^n,{\mathbf{y}}^k)\left(\prod _{l\in [k]}{P}_{S_l|X^n}(s_l|x^n)\right)\left(\prod _{l\in [j-1]}{P}_{{\widehat{X}}_l^n|Y_l^n{S}^l}({\widehat{x}}_l^n|y_l^n,s_l)\right)\hfill \end{array}$$ (A45)

$$\begin{array}{c}\times Q_{{\widehat{X}}_j^n|X^n{\mathbf{Y}}^k{S}^k{\widehat{\mathbf{X}}}^{j-1}}({\widehat{x}}_j^n|x^n,{\mathbf{y}}^k,s^k,{\widehat{\mathbf{x}}}^{j-1})\hfill \end{array}$$ (A46)

$$\begin{array}{cc}\hfill & \le (k-1)exp(-n\eta ).\hfill \end{array}$$ (A47)

Furthermore, using the definition of ${\mathcal{A}}_5$ in (A11),

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_5^{\mathrm{c}}\right\}& \le \sum _{x^n,s_1}{P}_{S_1|X^n}(s_1|x^n)exp(-n(R_1+\eta ))P_{X^n|S_1}(x^n|s_1)\hfill \end{array}$$ (A48)

$$\begin{array}{cc}\hfill & \le \sum _{x^n,s_1}exp(-n(R_1+\eta ))P_{X^n|S_1}(x^n|s_1)\hfill \end{array}$$ (A49)

$$\begin{array}{c}=\sum _{s_1}exp(-n(\eta +R_1))\hfill \end{array}$$ (A50)

$$\begin{array}{cc}\hfill & \le exp(-n\eta ),\hfill \end{array}$$ (A51)

where (A49) follows since $P_{S_1|X^n}(s_1|x^n)\le 1$ for all $(x^n,s_1)$, and (A51) follows since ${\sum }_{s_1}=|{\mathcal{W}}_1|=M_1\le exp(n{R}_1)$.

Using the definition of ${\mathcal{A}}_6$ in (A12) and the union bound similarly to (A47), it follows that

$$\begin{array}{cc}\hfill Pr\left\{{\mathcal{A}}_6^{\mathrm{c}}\right\}& \le \sum _{j\in [2:k]}\sum _{x^n,s^j}{P}_{S^{j-1}}(s^{j-1})exp(-n\eta )P_{X^n|S^j}(x^n|s^j)exp(-n(R_j-\sum _{l\in [j-1]}{R}_l))P_{S_j|X^n}(s_j|x^n)\hfill \end{array}$$ (A52)

$$\begin{array}{cc}\hfill & \le \sum _{j\in [2:k]}exp(-n\eta )\sum _{x^n,s^j}{P}_{S^{j-1}}(s^{j-1})P_{X^n|S^j}(x^n|s^j)exp(-n(R_j-\sum _{l\in [j-1]}{R}_l))\hfill \end{array}$$ (A53)

$$\begin{array}{c}\le \sum _{j\in [2:k]}exp(-n\eta )\sum _{s_j}exp(-n(R_j-\sum _{l\in [j-1]}{R}_l))\hfill \end{array}$$ (A54)

$$\begin{array}{cc}\hfill & \le (k-1)exp(-n\eta ),\hfill \end{array}$$ (A55)

where (A53) follows since $P_{S_j|X^n}(s_j|x^n)\le 1$ for all $(x^n,s_j)$ and (A55) follows since ${\sum }_{s_j}=|{\mathcal{M}}_j|=M_j\le exp(n(R_j-{\sum }_{l\in [j-1]}{R}_l))$.

Appendix D. Proof of Lemma 3

For any $(\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}$ satisfying (15), for $i\in [4]$, define ${\mathcal{F}}_i={\mathcal{B}}_i$ (cf. (33) to (36)) and for $i\in [5:7]$, define

$$\begin{array}{cc}\hfill {\mathcal{F}}_5& :=\left\{g:\mu {\alpha }_1{R}_1\ge \frac{\mu {\alpha }_1}{n}\sum _{i\in [n]}log\frac{Q_{X_i|W_{1,i}}(x_i|w_{1,i})}{P_X(x_i)}-\mu {\alpha }_1\eta \right\},\hfill \end{array}$$ (A56)

$$\begin{array}{ccc}\hfill {\mathcal{F}}_6& :=\hfill & \hfill \{g:\mu {\alpha }_j(R_j-\sum _{l\in [j-1]}{R}_l)\ge \sum _{i\in [n]}\frac{\mu {\alpha }_j}{n}log\frac{Q_{X_i|W_{1,i}{W}_{2,i}}(x_i|w_{1,i},w_{2,i})}{P_{X_i|W_{1,i}}(x_i|w_{1,i})}-\mu {\alpha }_j\eta ,\forall j\in [2:k]\},\end{array}$$ (A57)

$$\begin{array}{ccc}\hfill {\mathcal{F}}_7& :=\hfill & \hfill \{g:\mu {\beta }_j{D}_j\ge \frac{\mu {\beta }_j}{n}\sum _{i\in [n]}logexp(d_1(x_i,{\widehat{x}}_{1,i})),\forall j\in [k]\}.\end{array}$$ (A58)

Furthermore, let

$$\begin{array}{cc}\hfill c(\mu ,{\alpha }^k)& :=k+2+\sum _{j\in [k]}\mu {\alpha }_j.\hfill \end{array}$$ (A59)

Using Lemma 2 and definitions in (A56) to (A59), we obtain that

$$\begin{array}{c}{\mathrm{P}}_{\mathrm{c}}^{(n)}(D^k)-(2k+2)exp(-n\eta )\hfill \end{array}$$

$$\begin{array}{c}\le Pr\left\{\bigcap _{i\in [7]}{\mathcal{F}}_i\right\}\hfill \end{array}$$ (A60)

$$\begin{array}{c}\le Pr\left\{n(\mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)+c(\mu ,{\alpha }^k)\eta )\ge \sum _{i\in [n]}{f}_{Q_i,P_i}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_i)\right\}\hfill \end{array}$$ (A61)

$$\begin{array}{cc}\hfill & \le exp\left\{n\lambda (\mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)+c(\mu ,{\alpha }^k)\eta )+log\mathbb{E}\left[exp\left(-\lambda \sum _{i\in [n]}{f}_{Q_i,P_i}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_i)\right)\right]\right\}\hfill \end{array}$$ (A62)

$$\begin{array}{cc}\hfill & =exp\left\{n\left(\lambda \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)+\lambda c(\mu ,{\alpha }^k)\eta -\frac{1}{n}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})\right)\right\},\hfill \end{array}$$ (A63)

where (A62) follows from Cramér’s bound in Lemma 13 of [31] and (A63) follows from the definition of ${\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})$ in (42).

Choose $\eta$ such that

$$\begin{array}{c}\hfill -\eta =\lambda \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)+\lambda c(\mu ,{\alpha }^k)\eta -\frac{1}{n}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]}),\end{array}$$ (A64)

i.e.,

$$\begin{array}{c}\hfill \eta =\frac{\frac{1}{n}{\mathsf{\Omega }}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{P_i,Q_i\}}_{i\in [n]})-\lambda \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+\lambda c(\mu ,{\alpha }^k)}.\end{array}$$ (A65)

The proof of Lemma 3 is completed by combining (A63) and (A65).

Appendix E. Proof of Lemma 4

Recall that for each $i\in [n]$, we use $t_i$ to denote $(x_i,y_{1,i}^{k,i},w_{1,i}^{k,i},{\widehat{x}}_{1,i}^{k,i})$ and use $T_i$ similarly. Recall that the auxiliary random variables are chosen as $w_{1,i}=(x^{i-1},y_1^{i-1},\dots ,y_k^{i-1},s_1)$ and $w_{j,i}=s_j$ for all $j\in [2:k]$. Using the definition of $f_{Q_i,P_i}^{(\mu ,{\alpha }^k,{\beta }^k)}$ in (41), define

$$\begin{array}{cc}\hfill h_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i)& :=exp\left(-\lambda f_{Q_i,P_i}^{(\mu ,{\alpha }^k,{\beta }^k)}(t_i)\right).\hfill \end{array}$$ (A66)

Recall the joint distribution of G in (31). For each $j\in [n]$, define

$$\begin{array}{cc}\hfill {\tilde{C}}_j& :=\sum _g{P}_G(g)\prod _{i\in [j]}{h}_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i),\hfill \end{array}$$ (A67)

$$\begin{array}{cc}\hfill P_G^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)|j}(g)& :=\frac{P_G(g){\prod }_{i\in [j]}{h}_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i)}{{\tilde{C}}_i},\hfill \end{array}$$ (A68)

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_j^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{Q_i,P_i\}}_{i\in [n]})& :=\frac{{\tilde{C}}_j}{{\tilde{C}}_{j-1}}.\hfill \end{array}$$ (A69)

Combining (42) and (A69),

$$\begin{array}{cc}\hfill exp\left(-{\mathsf{\Omega }}_{({\{P_i,Q_i\}}_{i\in [n]})}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}\right)& =\mathbb{E}\left[\prod _{i\in [n]}{h}_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(T_i)\right]\hfill \end{array}$$ (A70)

$$\begin{array}{c}=\sum _{g\in \mathcal{G}}{P}_G(g)\prod _{i\in [n]}{h}_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i)\hfill \end{array}$$ (A71)

$$\begin{array}{cc}\hfill & =\prod _{i\in [n]}{\mathsf{\Lambda }}_i^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(\{Q_i,P_i\}).\hfill \end{array}$$ (A72)

Furthermore, similar to Lemma 5 of [25], we obtain the following lemma, which is critical in the proof of Lemma 4.

Lemma A2.

For each $j\in [n]$,

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_j^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{Q_i,P_i\}}_{i\in [n]})& =\sum _{g\in \mathcal{G}}{P}_G^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)|j-1}(g)h_{Q_j,P_j}^{(\mu ,{\alpha }^k,{\beta }^k)}(t_i).\hfill \end{array}$$ (A73)

Furthermore, for each $j\in [n]$, define

$$\begin{array}{c}\hfill P^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_j):=\sum _{\begin{array}{c}{x}_{j+1}^n,y_{1,j+1}^n,\dots ,y_{k,j+1}^n,\\ {\widehat{x}}_1^{j-1},\dots ,{\widehat{x}}_k^{j-1},{\widehat{x}}_{1,j+1}^n,\cdots ,{\widehat{x}}_{k,j+1}^n\end{array}}{P}_G^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)|j-1}(g).\end{array}$$ (A74)

Using Lemma A2 and (A74), it follows that for each $j\in [n]$,

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_j^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{Q_i,P_i\}}_{i\in [n]})& =\sum _{t_j}{P}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_j)h_{Q_j,P_j}^{(\mu ,{\alpha }^k,{\beta }^k)}(t_j).\hfill \end{array}$$ (A75)

Recall that the auxiliary distributions ${\left\{Q_i\right\}}_{i\in [n]}$ can be arbitrary distributions. Following the recursive method in [25], for each $i\in [n]$, we choose $Q_i$ such that

$$\begin{array}{c}\hfill Q_i(t_i)=P^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i).\end{array}$$ (A76)

Let $Q_{C_i|D_i}$, where $C_i\in {\mathcal{T}}_i$ and $D_i\in {\mathcal{T}}_i$, be induced by $Q_i$. Using the definition of $h_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i)$ in (A66), we define

$$\begin{array}{cc}\hfill {\xi }_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i)& :=h_{Q_i,P_i}^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}(t_i){\left(\frac{P_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i})}{Q_{X_i{Y}_{2,i}^{k,i}{W}_{2,i}^{k,i}|Y_{1,i}{W}_{1,i}}(x_i,y_{2,i}^{k,i},w_{2,i}^{k,i}|y_{1,i},w_{1,i})}\right)}^{-\lambda }\hfill \\ & \times {\left(\prod _{j\in [2:k]}\frac{P_{{\widehat{X}}_{j,i}|Y_{j,i}{W}_{1,i}^{j,i}}({\widehat{x}}_{j,i}|y_{j,i},w_{1,i}^{j,i})}{Q_{{\widehat{X}}_{j,i}|Y_{j,i}{W}_{1,i}^{j,i}}({\widehat{x}}_{j,i}|y_{j,i},w_{1,i}^{j,i})}\right)}^{-\lambda }{\left(\frac{P_{X_i|W_{1,i}}(x_i|w_{1,i})}{Q_{X_i|W_{1,i}}(x_i|w_{1,i})}\right)}^{-\lambda \mu {\alpha }_1}\hfill \\ & \times \prod _{j\in [2:k]}{\left(\frac{P_{X_i|W_{1,i}^{j-1,i}}(x_i,w_{1,i}^{j-1,i})}{Q_{X_i|W_{1,i}^{j-1,i}}(x_i,w_{1,i}^{j-1,i})}\right)}^{-\lambda \mu {\alpha }_j}.\hfill \end{array}$$ (A77)

In the following, for simplicity, we let $\mathsf{\Psi }:=1-k\lambda -{\sum }_{j\in [k]}\lambda \mu {\alpha }_j$. Combining (A74) and (A75), we obtain that for each $l\in [n]$,

$$\begin{array}{c}{\mathsf{\Lambda }}_l^{(\lambda ,\mu ,{\alpha }^k,{\beta }^k)}({\{Q_i,P_i\}}_{i\in [n]})\hfill \end{array}$$

$$\begin{array}{c}={\mathbb{E}}_{Q_l}\left[h_{Q_l,P_l}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_l)\right]\hfill \end{array}$$ (A78)

$$\begin{array}{c}={\mathbb{E}}_{Q_l}[{\xi }_{Q_l,P_l}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_l){\left(\frac{P_{X_l{Y}_{2,l}^{k,l}{W}_{2,l}^{k,l}|Y_{1,l}{W}_{1,l}}(x_l,y_{2,l}^{k,l},w_{2,l}^{k,l}|y_{1,l},w_{1,l})}{Q_{X_l{Y}_{2,l}^{k,l}{W}_{2,l}^{k,l}|Y_{1,l}{W}_{1,l}}(x_l,y_{2,l}^{k,l},w_{2,l}^{k,l}|y_{1,l},w_{1,l})}\right)}^{\lambda }{\left(\prod _{j\in [2:k]}\frac{P_{{\widehat{X}}_{j,l}|Y_{j,l}{W}_{1,l}^{j,l}}({\widehat{x}}_{j,l}|y_{j,l},w_{1,l}^{j,l})}{Q_{{\widehat{X}}_{j,l}|Y_{j,l}{W}_{1,l}^{j,l}}({\widehat{x}}_{j,l}|y_{j,l},w_{1,l}^{j,l})}\right)}^{\lambda }\hfill \end{array}$$

$$\begin{array}{c}\times {\left(\frac{P_{X_l|W_{1,l}}(x_l|w_{1,l})}{Q_{X_l|W_{1,l}}(x_l|w_{1,l})}\right)}^{\lambda \mu {\alpha }_1}\prod _{j\in [2:k]}{\left(\frac{P_{X_l|W_{1,l}^{j-1,l}}(x_l,w_{1,l}^{j-1,l})}{Q_{X_l|W_{1,l}^{j-1,l}}(x_l,w_{1,l}^{j-1,l})}\right)}^{\lambda \mu {\alpha }_j}]\hfill \end{array}$$ (A79)

$$\begin{array}{c}\le {\left({\mathbb{E}}_{Q_l}{\left[\left({\xi }_{Q_l,P_l}^{(\mu ,{\alpha }^k,{\beta }^k)}(T_l)\right]\right)}^{\frac{1}{\mathsf{\Psi }}}\right)}^{\mathsf{\Psi }}{\left(\mathbb{E}\left[\frac{P_{X_l{Y}_{2,l}^{k,l}{W}_{2,l}^{k,l}}|Y_{1,l}{W}_{1,l}(x_l,y_{2,l}^{k,l},w_{2,l}^{k,l}|y_{1,l},w_{1,l})}{Q_{X_l{Y}_{2,l}^{k,l}{W}_{2,l}^{k,l}}|Y_{1,l}{W}_{1,l}(x_l,y_{2,l}^{k,l},w_{2,l}^{k,l}|y_{1,l},w_{1,l})}\right]\right)}^{\lambda }\hfill \\ \times \prod _{j\in [2:k]}{\left(\mathbb{E}\left[\frac{P_{{\widehat{X}}_{j,l}|Y_{j,l}{W}_{1,l}^{j,l}}({\widehat{x}}_{j,l}|y_{j,l},w_{1,l}^{j,l})}{Q_{{\widehat{X}}_{j,l}|Y_{j,l}{W}_{1,l}^{j,l}}({\widehat{x}}_{j,l}|y_{j,l},w_{1,l}^{j,l})}\right]\right)}^{\lambda }{\left(\mathbb{E}\left[\frac{P_{X_l|W_{1,l}}(x_l|w_{1,l})}{Q_{X_l|W_{1,l}}(x_l|w_{1,l})}\right]\right)}^{\lambda \mu {\alpha }_1}\hfill \end{array}$$

$$\begin{array}{c}\times \prod _{j\in [2:k]}{\left(\mathbb{E}\left[\frac{P_{X_l|W_{1,l}^{j-1,l}}(x_l,w_{1,l}^{j-1,l})}{Q_{X_l|W_{1,l}^{j-1,l}}(x_l,w_{1,l}^{j-1,l})}\right]\right)}^{\lambda \mu {\alpha }_j}\hfill \end{array}$$ (A80)

$$\begin{array}{c}\le exp\left(-\mathsf{\Psi }{\mathsf{\Omega }}^{(\frac{\lambda }{\mathsf{\Psi }},\mu ,{\alpha }^k,{\beta }^k)}(Q_j)\right)\hfill \end{array}$$ (A81)

$$\begin{array}{c}=exp\left(-\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_j)}{1+k\theta +{\sum }_{j\in [k]}\theta \mu {\alpha }_j}\right)\hfill \end{array}$$ (A82)

$$\begin{array}{c}\le exp\left(-\underset{Q_j\in \mathcal{P}({\mathcal{T}}_j)}{min}\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_j)}{1+k\theta +{\sum }_{j\in [k]}\theta \mu {\alpha }_j}\right)\hfill \end{array}$$ (A83)

$$\begin{array}{c}=exp\left(-\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}}{1+k\theta +{\sum }_{j\in [k]}\theta \mu {\alpha }_j}\right)\hfill \end{array}$$ (A84)

where (A80) results from Hölder’s inequality, (A81) follows from the definitions of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(·)$ in (17) and ${\xi }_{Q_j,P_j}^{(\mu ,{\alpha }^k,{\beta }^k)}(·)$ in (A77), (A82) follows from the result in (46), and (A84) follows from the definition of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}$ in (18) and the fact it is sufficient to consider distributions $Q_j$ with cardinality bounds $W_{1,j}\le |\mathcal{X}|$ and $W_{2,j}\le {|\mathcal{X}|}^2$ for the optimization problem in (A83) (the proof of this fact is similar to Property 4(a) in [25] and thus omitted).

The proof of Lemma 4 is completed by combining (A72) and (A84).

Appendix F. Proof of Lemma 6

Appendix F.1. Proof of Claim (i)

For any $Q_T\in \mathcal{Q}$ (see (14)), let $P_T\in {\mathcal{P}}_{\mathrm{sh}}$ (see (53)) be chosen such that $P_{W^k|X}=Q_{W^k|X}$ and $P_{{\widehat{X}}_j|Y_j{W}^j}=Q_{{\widehat{X}}_j|Y_j{W}^j}$ for all $j\in [k]$.

In the following, we drop the subscript of distributions when there is no confusion. For any $(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}$ satisfying (15) and

$$\begin{array}{c}\hfill \sum _{j\in [2:k]}\mu {\alpha }_j\le 1\mathrm{and}\forall l\in [k],\theta \le \frac{1}{1+\mu {\alpha }_l},\end{array}$$ (A85)

using the definition of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)$ in (17), we obtain

$$\begin{array}{c}exp\left(-{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}(Q_T)\right)\hfill \\ ={\mathbb{E}}_{Q_T}[{\left(\frac{P(X,Y^k)Q(X,Y^{k\backslash 1},W^{k\backslash 1}|Y_1,W_1)\left({\prod }_{j\in [2:k]}Q({\widehat{X}}_j|Y,W^j)\right)}{Q(X)Q(Y^k|X,W^k)Q(X,Y^{k\backslash 1},W^{k\backslash 1}|Y_1,W_1,{\widehat{X}}_1)\left({\prod }_{j\in [2:k]}Q({\widehat{X}}_j|X,Y^k,W^k,{\widehat{X}}^{j-1})\right)}\right)}^{\theta }\hfill \end{array}$$

$$\begin{array}{c}\times {\left(\frac{P(X)}{Q(X|W_1)}\right)}^{\theta \mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{Q(X|W^j)}\right)}^{\theta \mu {\alpha }_j}\right)exp\left(-\theta \mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)]\hfill \end{array}$$ (A86)

$$\begin{array}{c}={\mathbb{E}}_{Q_T}\left[{\left(\frac{P(T)}{Q(T)}\right)}^{\theta }{\left(\frac{P(X)}{Q(X|W_1)}\right)}^{\theta \mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{Q(X|W^j)}\right)}^{\theta \mu {\alpha }_j}\right)exp\left(-\theta \mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)\right]\hfill \end{array}$$ (A87)

$$\begin{array}{c}={\mathbb{E}}_{Q_T}\left[{\left(\frac{P(T)}{Q(T)}\right)}^{\theta }{\left(\frac{P(X)}{P(X|W_1)}\right)}^{\theta \mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{P(X|W^j)}\right)}^{\theta \mu {\alpha }_j}\right)exp\left(-\theta \mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)\right]\hfill \end{array}$$

$$\begin{array}{c}\times \left(\prod _{j\in [k]}{\left(\frac{P(X|W^j)}{Q(X|W^j)}\right)}^{\theta \mu {\alpha }_j}\right)\hfill \end{array}$$ (A88)

$$\begin{array}{c}\le {\left({\mathbb{E}}_{Q_T}\left[\left(\frac{P(T)}{Q(T)}\right){\left(\frac{P(X)}{P(X|W_1)}\right)}^{\mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{P(X|W^j)}\right)}^{\mu {\alpha }_j}\right)exp\left(-\mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)\right]\right)}^{\theta }\hfill \end{array}$$

$$\begin{array}{c}\times \prod _{j\in [k]}{\left({\mathbb{E}}_{Q_T}\left[{\left(\frac{P(X|W^j)}{Q(X|W^j)}\right)}^{\frac{\theta \mu {\alpha }_j}{1-\theta }}\right]\right)}^{1-\theta }\hfill \end{array}$$ (A89)

$$\begin{array}{cc}\le ({\mathbb{E}}_{P_T}[{\left(\frac{P(X)}{P(X|W_1)}\right)}^{\mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{P(X|W^j)}\right)}^{\mu {\alpha }_j}\right)\hfill & \hfill exp\left(-\mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)]{)}^{\theta }\end{array}$$ (A90)

$$\begin{array}{c}=({\mathbb{E}}_{P_T}[{\left(\frac{P(X)}{P(X|W_1)}\right)}^{\mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{P(X|W^{j-1})}{P(X|W^j)}\right)}^{\mu {\alpha }_j}\right)exp\left(-\mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)\hfill \end{array}$$

$$\begin{array}{c}\times \prod _{j\in [2:k]}{\left(\frac{Q(X|W^{j-1})}{P(X|W^{j-1})}\right)}^{\mu {\alpha }_j}]{)}^{\theta }\hfill \end{array}$$ (A91)

$$\begin{array}{c}={\left({\mathbb{E}}_{P_T}\left[{\left({\left(\frac{P(X)}{P(X|W_1)}\right)}^{\mu {\alpha }_1}\left(\prod _{j\in [2:k]}{\left(\frac{P(X|W^{j-1})}{P(X|W^j)}\right)}^{\mu {\alpha }_j}\right)exp\left(-\mu \left(\sum _{j\in [k]}{\beta }_j{d}_j(X,{\widehat{X}}_j)\right)\right)\right)}^{\frac{1}{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j}}\right]\right)}^{\theta (1-{\sum }_{j\in [2:k]}\mu {\alpha }_j)}\hfill \end{array}$$

$$\begin{array}{c}\times \prod _{j\in [2:k]}{\left({\mathbb{E}}_{P_T}\left[\left(\frac{Q(X|W^{j-1})}{P(X|W^{j-1})}\right)\right]\right)}^{\theta \mu {\alpha }_j}\hfill \end{array}$$ (A92)

$$\begin{array}{c}=exp\left(-\theta (1-\sum _{j\in [2:k]}\mu {\alpha }_j)\tilde{\mathsf{\Omega }}\left(\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j},{\alpha }^k,{\beta }^k\right)\right),\hfill \end{array}$$ (A93)

where (A87) follows since (i) with our choice of $P_T\in {\mathcal{P}}_{\mathrm{sh}}$, we have

$$\begin{array}{c}\hfill P(T)=P(X,Y^k)P(W^k|X)\left(\prod _{j\in [k]}P({\widehat{X}}_j|Y_j,W^j)\right)\end{array}$$ (A94)

and (ii) the following equality holds

$$\begin{array}{cc}\hfill \frac{Q(X,Y^{k\backslash 1},W^{k\backslash 1}|Y_1,W_1)}{Q(X,Y^{k\backslash 1},W^{k\backslash 1}|Y_1,W_1,{\widehat{X}}_1)}& =\frac{Q({\widehat{X}}_1|Y_1,W_1)}{Q({\widehat{X}}_1|X,Y^k,W^k)},\hfill \end{array}$$ (A95)

Equation (A89) follows from Hölder’s inequality, (A90) follows from the concavity of $X^a$ for $a\in [0,1]$ and the choice of $\theta$ which ensures $\frac{\theta \mu {\alpha }_j}{1-\theta }\le 1$ for all $j\in [k]$, (A92) follows by applying Hölder’s inequality and recalling that ${\sum }_{j\in [2:k]}\mu {\alpha }_j\le 1$, and (A93) follows from the definition of ${\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)$ in (58).

Therefore, for any $(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}$ satisfying (15) and (A85), using the definition of ${\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}$ in (18) and the result in (A93), we have that

$$\begin{array}{c}\hfill {\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}\ge \theta (1-\sum _{j\in [2:k]}\mu {\alpha }_j)\tilde{\mathsf{\Omega }}\left(\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j},{\alpha }^k,{\beta }^k\right).\end{array}$$ (A96)

Recalling the definition of $F(R^k,D^k)$ in (21) and using the result in (A96), we have

$$\begin{array}{c}F(R^k,D^k)\hfill \end{array}$$

$$\begin{array}{c}=\underset{(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}:{\sum }_{i\in [k]}({\alpha }_i+{\beta }_i)=1}{sup}\frac{{\mathsf{\Omega }}^{(\theta ,\mu ,{\alpha }^k,{\beta }^k)}-\theta \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j}\hfill \end{array}$$ (A97)

$$\begin{array}{c}\ge \underset{\begin{array}{c}(\theta ,\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}^2\times {[0,1]}^{2k}:\\ (15)\mathrm{and}(\mathrm{A}85)\end{array}}{sup}\frac{\theta (1-{\sum }_{j\in [2:k]}\mu {\alpha }_j)\tilde{\mathsf{\Omega }}\left(\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j},{\alpha }^k,{\beta }^k\right)-\theta \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j}\hfill \end{array}$$ (A98)

$$\begin{array}{c}=\underset{\begin{array}{c}(\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:\\ (15)\mathrm{and}\mu \le \frac{1}{{\sum }_{j\in [2:k]}{\alpha }_j}\end{array}}{sup}\underset{\theta \in {\mathbb{R}}_{+}:{max}_{j\in [k]}\theta (1+\mu {\alpha }_j)\le 1}{sup}\frac{\theta (1-{\sum }_{j\in [2:k]}\mu {\alpha }_j)\tilde{\mathsf{\Omega }}\left(\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j},{\alpha }^k,{\beta }^k\right)-\theta \mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j}\hfill \end{array}$$ (A99)

$$\begin{array}{c}=\underset{\begin{array}{c}(\mu ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:\\ (15)\mathrm{and}\mu \le \frac{1}{{\sum }_{j\in [2:k]}{\alpha }_j}\end{array}}{sup}\frac{(1-{\sum }_{j\in [2:k]}\mu {\alpha }_j)\tilde{\mathsf{\Omega }}\left(\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j},{\alpha }^k,{\beta }^k\right)-\mu {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{2k+3+\mu {\alpha }^{+}+{\sum }_{l\in [k]}2\mu {\alpha }_l}\hfill \end{array}$$ (A100)

$$\begin{array}{cc}\hfill & =\underset{\begin{array}{c}(\lambda ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:\\ (15)\end{array}}{sup}\frac{{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}-\lambda {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{2k+3+\lambda {\alpha }^{+}+{\sum }_{j\in [2:k]}\lambda (2k+3){\alpha }_j+{\sum }_{l\in [k]}2\lambda {\alpha }_l}\hfill \end{array}$$ (A101)

$$\begin{array}{c}=\tilde{F}(R^k,D^k),\hfill \end{array}$$ (A102)

where (A100) follows since

$$\begin{array}{cc}\hfill \underset{\theta \in {\mathbb{R}}_{+}:{max}_{j\in [k]}\theta (1+\mu {\alpha }_j)\le 1}{sup}\frac{\theta }{1+(2k+2)\theta +{\sum }_{j\in [k]}2\theta \mu {\alpha }_j}& =\underset{j\in [k]}{min}\frac{1}{2k+3+\mu {\alpha }_j+{\sum }_{l\in [k]}2\mu {\alpha }_l}\hfill \end{array}$$ (A103)

$$\begin{array}{c}=\frac{1}{2k+3+\mu {\alpha }^{+}+{\sum }_{l\in [k]}2\mu {\alpha }_l},\hfill \end{array}$$ (A104)

and (A101) follows by choosing $\lambda =\frac{\mu }{1-{\sum }_{j\in [2:k]}\mu {\alpha }_j}$ and (A102) follows from the definition of $\tilde{F}$ in (62).

Appendix F.2. Proof of Claim (ii)

Recall the definitions of ${\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)$ in (57) and $P_T^{(\lambda ,{\alpha }^k,{\beta }^k)}$ in (63). By simple calculation, one can verify that

$$\begin{array}{cc}\hfill \frac{\partial {\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)}{\partial \lambda }& ={\mathbb{E}}_{P_T^{(\lambda ,{\alpha }^k,{\beta }^k)}}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right],\hfill \end{array}$$ (A105)

$$\begin{array}{cc}\hfill \frac{{\partial }^2{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)}{\partial {\lambda }^2}& =-{\mathrm{Var}}_{P_T^{(\lambda ,{\alpha }^k,{\beta }^k)}}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right].\hfill \end{array}$$ (A106)

Applying Taylor expansion to ${\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)$ at around $\lambda =0$ and combining (A105), (A106), we have that for any $P_T\in {\mathcal{P}}_{\mathrm{sh}}$ and any $\lambda \in [1,\frac{1}{{\sum }_{j\in [k]}{\alpha }_j}]$, there exists $\tau \in [0,\lambda ]$ such that

$$\begin{array}{cc}\hfill {\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)& ={\tilde{\mathsf{\Omega }}}^{(0,{\alpha }^k,{\beta }^k)}(P_T)+\lambda {\mathbb{E}}_{P_T^{(0,{\alpha }^k,{\beta }^k)}}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right]-\frac{{\lambda }^2}{2}{\mathrm{Var}}_{P_T^{(\tau ,{\alpha }^k,{\beta }^k)}}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right]\hfill \end{array}$$ (A107)

$$\begin{array}{c}\ge \lambda {\mathbb{E}}_{P_T}\left[{\tilde{\omega }}_{P_T}^{({\alpha }^k,{\beta }^k)}(T)\right]-\frac{{\lambda }^2\rho }{2},\hfill \end{array}$$ (A108)

where (A108) follows from the definitions in (57), (63) and (64).

Using the definitions in (54), (57) and (60) and the result in (A108), we have that for any $\lambda \in [0,\frac{1}{{\sum }_{j\in [k]}{\alpha }_j}]$,

$$\begin{array}{cc}\hfill {\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}& =\underset{P_T\in {\mathcal{P}}_{\mathrm{sh}}}{min}{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}(P_T)\hfill \end{array}$$ (A109)

$$\begin{array}{cc}\hfill & \ge \lambda {\mathrm{R}}^{({\alpha }^k,{\beta }^k)}-\frac{{\lambda }^2\rho }{2}.\hfill \end{array}$$ (A110)

For any rate-distortion tuple outside the rate-distortion region, i.e., $(R^k,D^k)\notin \mathcal{R}$, from Lemma 5, we conclude that there exists $({\alpha }^{k,\ast },{\beta }^{k,\ast })\in {[0,1]}^{2k}$ satisfying (15) such that for some positive $\delta \in [0,\rho ]$

$$\begin{array}{cc}\hfill {\kappa }^{({\alpha }^{k{,}^{\ast }},{\beta }^{k,\ast })}(R^k,D^k)& \le R^{({\alpha }^{k{,}^{\ast }},{\beta }^{k,\ast })}-\delta .\hfill \end{array}$$ (A111)

Using the definition of $\tilde{F}(R^k,D^k)$ in (62), we have

$$\begin{array}{cc}\hfill \tilde{F}(R^k,D^k)& =\underset{\begin{array}{c}(\lambda ,{\alpha }^k,{\beta }^k)\in {\mathbb{R}}_{+}\times {[0,1]}^{2k}:(15)\end{array}}{sup}\frac{{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^k,{\beta }^k)}-\lambda {\kappa }^{({\alpha }^k,{\beta }^k)}(R^k,D^k)}{2k+3+\lambda {\alpha }^{+}+{\sum }_{j\in [2:k]}\lambda (2k+3){\alpha }_j+{\sum }_{l\in [k]}2\lambda {\alpha }_l}\hfill \end{array}$$ (A112)

$$\begin{array}{c}\ge \underset{\lambda \in [0,1]}{sup}\frac{{\tilde{\mathsf{\Omega }}}^{(\lambda ,{\alpha }^{k,\ast },{\beta }^{k,\ast })}-\lambda {\kappa }^{({\alpha }^{k,\ast },{\beta }^{k,\ast })}(R^k,D^k)}{2k+3+\lambda {max}_{j\in [k]}{\alpha }_j^{\ast }+{\sum }_{j\in [2:k]}\lambda (2k+3){\alpha }_j^{\ast }+{\sum }_{l\in [k]}2\lambda {\alpha }_l^{\ast }}\hfill \end{array}$$ (A113)

$$\begin{array}{cc}\hfill & \ge \underset{\lambda \in [0,1]}{sup}\frac{\lambda \delta -\frac{{\lambda }^2\rho }{2}}{2k+9}\hfill \end{array}$$ (A114)

$$\begin{array}{c}=\frac{{\delta }^2}{2(2k+9)\rho },\hfill \end{array}$$ (A115)

where (A114) follows from the results in (A110), (A111) and the inequality

$$\begin{array}{cc}\hfill 2k+3+\lambda \underset{j\in [k]}{max}{\alpha }_j^{\ast }+\sum _{j\in [2:k]}\lambda (2k+3){\alpha }_j^{\ast }+\sum _{l\in [k]}2\lambda {\alpha }_l^{\ast }& \le 2k+9,\hfill \end{array}$$ (A116)

resulting from the constraints that $({\alpha }^{k,\ast },{\beta }^{k{,}^{\ast }})\in {[0,2]}^{2k}$ satisfying (15) and $\lambda \in [0,1]$.

Author Contributions

Formal analysis, L.Z.; Funding acquisition, A.H.; Supervision, A.H.; Writing—original draft, L.Z.; Writing—review & editing, A.H.

Funding

This work was partially supported by ARO grant W911NF-15-1-0479.

Conflicts of Interest

The authors declare no conflict of interest.