The Cauchy Distribution in Information Theory

Abstract

The Gaussian law reigns supreme in the information theory of analog random variables. This paper showcases a number of information theoretic results which find elegant counterparts for Cauchy distributions. New concepts such as that of equivalent pairs of probability measures and the strength of real-valued random variables are introduced here and shown to be of particular relevance to Cauchy distributions.

1. Introduction

Since the inception of information theory [1], the Gaussian distribution has emerged as the paramount example of a continuous random variable leading to closed-form expressions for information measures and extremality properties possessing great pedagogical value. In addition, the role of the Gaussian distribution as a ubiquitous model for analog information sources and for additive thermal noise has elevated the corresponding formulas for rate–distortion functions and capacity–cost functions to iconic status in information theory. Beyond discrete random variables, by and large, information theory textbooks confine their coverage and examples to Gaussian random variables.

The exponential distribution has also been shown [2] to lead to closed-form formulas for various information measures such as differential entropy, mutual information and relative entropy, as well as rate–distortion functions for Markov processes and the capacity of continuous-time timing channels with memory such as the exponential-server queue [3].

Despite its lack of moments, the Cauchy distribution also leads to pedagogically attractive closed-form expressions for various information measures. In addition to showcasing those, we introduce an attribute, which we refer to as the strength of a real-valued random variable, under which the Cauchy distribution is shown to possess optimality properties. Along with the stability of the Cauchy law, those properties result in various counterparts to the celebrated fundamental limits for memoryless Gaussian sources and channels.

To enhance readability and ease of reference, the rest of this work is organized in 120 items grouped into 17 sections, plus an appendix.

Section 2 presents the family of Cauchy random variables and their basic properties as well as multivariate generalizations, and the Rider univariate density which includes the Cauchy density as a special case and finds various information theoretic applications.

Section 3 gives closed-form expressions for the differential entropies of the univariate and multivariate densities covered in Section 2.

Introduced previously for unrelated purposes, the Shannon and $\eta$-transforms reviewed in Section 4 prove useful to derive several information theoretic results for Cauchy and related laws.

Applicable to any real-valued random variable and inspired by information theory, the central notion of strength is introduced in Section 5 along with its major properties. In particular, it is shown that convergence in strength is an intermediate criterion between convergence in probability and convergence in $L_q$, $q>0$, and that differential entropy is continuous with respect to the addition of independent vanishing strength noise.

Section 6 shows that, for any $\rho >0$ the maximal differential entropy density satisfying $\mathbb{E}\left[log\left(1+{\left|Z\right|}^{\rho }\right)\right]\le \theta$ can be obtained in closed form, but its shape (not just its scale) depends on the value of $\theta$. In particular, the Cauchy density is the solution only if $\rho =2$, and $\theta =log4$. In contrast, we show that, among all the random variables with a given strength, the centered Cauchy density has maximal differential entropy, regardless of the value of the constraint. This result suggests the definition of entropy strength of Z, as the strength of a Cauchy random variable whose differential entropy is the same as that of Z. Modulo a factor, entropy power is the square of entropy strength. Section 6 also gives a maximal differential entropy characterization of the standard spherical Cauchy multivariate density.

Information theoretic terminology for the logarithm of the Radon–Nikodym derivative, as well as its distribution, the relative information spectrum is given in Section 7. The relative information spectrum for Cauchy distributions is found and shown to depend on their location and scale through a single scalar. This is a rare property, not satisfied by most common families such as Gaussian, exponential, Laplace, etc. Section 8 introduces the notion of equivalent pairs of probability measures, which plays an important role not only in information theory but in statistical inference. Distinguishing $P_1$ from $Q_1$ has the same fundamental limits as distinguishing $P_2$ from $Q_2$ if $(P_1,Q_1)$ and $(P_2,Q_2)$ are equivalent pairs. Section 9 studies the interplay between f-divergences and equivalent pairs. A simple formula for the f-divergence between Cauchy distributions results from the explicit expression for the relative information spectrum found in Section 7. These results are then used to easily derive a host of explicit expressions for ${\chi }^2$-divergence, relative entropy, total variation distance, Hellinger divergence and Rényi divergence in Section 10, Section 11, Section 12, Section 13 and Section 14, respectively.

In addition to the Fisher information matrix of the Cauchy family, Section 15 finds a counterpart of de Bruijn’s identity [4] for convolutions with scaled Cauchy random variables, instead of convolutions with scaled Gaussian random variables as in the conventional setting.

Section 16 is devoted to mutual information. The mutual information between a Cauchy random variable and its noisy version contaminated by additive independent Cauchy noise exhibits a pleasing counterpart (modulo a factor of two) with the Gaussian case, in which the signal-to-noise ratio is now given by the ratio of strengths rather than variances. With Cauchy noise, Cauchy inputs maximize mutual information under an output strength constraint. The elementary fact that an output variance constraint translates directly into an input variance constraint does not carry over to input and output strengths, and indeed we identify non-Cauchy inputs that may achieve higher mutual information than a Cauchy input with the same strength. Section 16 also considers the dual setting in which the input is Cauchy, but the additive noise need not be. Lower bounds on the mutual information, attained by Cauchy noise, are offered. However, as the bounds do not depend exclusively on the noise strength, they do not rule out the possibility that a non-Cauchy noise with identical strength may be least favorable. If distortion is measured by strength, the rate–distortion function of a Cauchy memoryless source is shown to admit (modulo a factor of two) the same rate–distortion function as the memoryless Gaussian source with mean–square distortion, replacing the source variance by its strength. Theorem 17 gives a very general continuity result for mutual information that encompasses previous such results. While convergence in probability to zero of the input to an additive-noise transformation does not imply vanishing input-output mutual information, convergence in strength does under very general conditions on the noise distribution.

Some concluding observations about generalizations and open problems are collected in Section 17, including a generalization of the notion of strength.

Those definite integrals used in the main body are collected and justified in the Appendix A.

2. The Cauchy Distribution and Generalizations

In probability theory, the Cauchy (also known as Lorentz and as Breit–Wigner) distribution is the prime example of a real-valued random variable none of whose moments of order one or higher exists, and as such it is not encompassed by either the law of large numbers or the central limit theorem.

1. A real-valued random variable V is said to be standard Cauchy if its probability density function is

   $$\begin{array}{c}\hfill f_V\left(x\right)=\frac{1}{\pi }\frac{1}{x^2+1},x\in \mathbb{R}.\end{array}$$ (1)
   Furthermore, X is said to be Cauchy if there exist $\lambda \ne 0$ and $\mu \in \mathbb{R}$ such that $X=\lambda V+\mu$, in which case

   $$\begin{array}{c}\hfill f_X\left(x\right)=\frac{\left|\lambda \right|}{\pi }\frac{1}{{(x-\mu )}^2+{\lambda }^2},x\in \mathbb{R},\end{array}$$ (2)

   where $\mu$ and $\left|\lambda \right|$ are referred to as the location (or median) and scale, respectively, of the Cauchy distribution. If $\mu =0$, (2) is said to be centered Cauchy.
2. Since $\mathbb{E}[max\{0,V\left\}\right]=\mathbb{E}[max\{0,-V\left\}\right]=\infty$, the mean of a Cauchy random variable does not exist. Furthermore, $\mathbb{E}\left[\right|V{|}^q]=\infty$ for $q\ge 1$, and the moment generating function of V does not exist (except, trivially, at 0). The characteristic function of the standard Cauchy random variable is

   $$\begin{array}{c}\hfill \mathbb{E}\left[{\mathrm{e}}^{i\omega V}\right]={\mathrm{e}}^{-\left|\omega \right|},\omega \in \mathbb{R}.\end{array}$$ (3)

3. Using (3), we can verify that a Cauchy random variable has the curious property that adding an independent copy to it has the same effect, statistically speaking, as adding an identical copy. In addition to the Gaussian and Lévy distributions, the Cauchy distribution is stable: a linear combination of independent copies remains in the family, and is infinitely divisible: it can be expressed as an n-fold convolution for any n. It follows from (3) that if $\{V_1,V_2,\dots \}$ are independent, standard Cauchy, and $\mathbf{a}$ is a deterministic sequence with finite ${\ell }_1$-norm ${\parallel \mathbf{a}\parallel }_1$, then ${\sum }_{i=1}^{\infty }{a}_i{V}_i$ has the same distribution as ${\parallel \mathbf{a}\parallel }_{1}V$. In particular, the time average of independent identically distributed Cauchy random variables has the same distribution as any of the random variables. The families $\{\lambda V,\lambda \in \mathcal{I}\}$ and $\{V+\mu ,\mu \in \mathcal{I}\}$, with $\mathcal{I}$ any interval of the real line, are some of the simplest parametrized random variables that are not an exponential family.

4. If $\mathrm{\Theta }$ is uniformly distributed on $[-\frac{\pi }{2},\frac{\pi }{2}]$, then $tan\mathrm{\Theta }$ is standard Cauchy. This follows since, in view of (1) and (A1), the standard Cauchy cumulative distribution function is

   $$\begin{array}{c}\hfill F_V\left(x\right)=\frac{1}{2}+\frac{1}{\pi }arctan\left(x\right),x\in \mathbb{R}.\end{array}$$ (4)
   Therefore, V has unit semi-interquartile length. The functional inverse of (4) is the standard Cauchy quantile function given by

   $$\begin{array}{c}\hfill {\mathbb{Q}}_V\left(t\right)=tan\left(\pi \left(t-\frac{1}{2}\right)\right),t\in (0,1).\end{array}$$ (5)

5. If $X_1$ and $X_2$ are standard Gaussian with correlation coefficient $\rho \in (-1,1)$, then $X_1/X_2$ is Cauchy with scale $\sqrt{1-{\rho }^2}$ and location $\rho$. This implies that the reciprocal of a standard Cauchy random variable is also standard Cauchy.

6. Taking the cue from the Gaussian case, we say that a random vector is multivariate Cauchy if any linear combination of its components has a Cauchy distribution. Necessary and sufficient conditions for a characteristic function to be that of a multivariate Cauchy were shown by Ferguson [5]. Unfortunately, no general expression is known for the corresponding probability density function. This accounts for the fact that one aspect, in which the Cauchy distribution does not quite reach the wealth of information theoretic results attainable with the Gaussian distribution, is in the study of multivariate models of dependent random variables. Nevertheless, special cases of multivariate Cauchy distribution do admit some interesting information theoretic results as we will see below. The standard spherical multivariate Cauchy probability density function on ${\mathbb{R}}^n$ is (e.g., [6])

   $$\begin{array}{c}\hfill f_{V^n}\left(\mathbf{x}\right)=\frac{\mathrm{\Gamma }\left(\frac{n+1}{2}\right)}{{\pi }^{\frac{n+1}{2}}}{\left(1+{\parallel \mathbf{x}\parallel }^2\right)}^{-\frac{n+1}{2}},\end{array}$$ (6)

   where $\mathrm{\Gamma }(·)$ is the Gamma function. Therefore, $V^n=(V_1,\dots ,V_n)$ are exchangeable random variables. If $X_0,X_1,\dots ,X_n$ are independent standard normal, then the vector $X_0^{-1}{X}^n$ has the density in (6). With the aid of (A10), we can verify that any subset of $k\in \{1,\dots ,n-1\}$ components of $V^n$ is distributed according to $V^k$. In particular, the marginals of (6) are given by (1). Generalizing (3), the characteristic function of (6) is

   $$\begin{array}{c}\hfill \mathbb{E}\left[{\mathrm{e}}^{i{\mathbf{t}}^{\top }{V}^n}\right]={\mathrm{e}}^{-\parallel \mathbf{t}\parallel },\mathbf{t}\in {\mathbb{R}}^n.\end{array}$$ (7)
7. In parallel to Item 1, we may generalize (6) by dropping the restriction that it be centered at the origin and allowing ellipsoidal deformation, i.e., letting $Z^n={\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+\mathbf{\mu }$ with $\mathbf{\mu }\in {\mathbb{R}}^n$ and a positive definite $n\times n$ matrix $\mathrm{\Lambda }$. Therefore,

   $$\begin{array}{c}\hfill f_{Z^n}\left(\mathbf{x}\right)=\frac{\mathrm{\Gamma }\left(\frac{n+1}{2}\right)}{{\pi }^{\frac{n+1}{2}}{det}^{\frac{1}{2}}\left(\mathrm{\Lambda }\right)}{\left(1+{(\mathbf{x}-\mathbf{\mu })}^{\top }{\mathrm{\Lambda }}^{-1}(\mathbf{x}-\mathbf{\mu })\right)}^{-\frac{n+1}{2}}.\end{array}$$ (8)

   While ${\mathbf{\rho }}^{\top }{Z}^n$ is a Cauchy random variable for $\mathbf{\rho }\in {\mathbb{R}}^n-\left\{0\right\}$, (8) fails to encompass every multivariate Cauchy distribution—in particular, the important case of independent Cauchy random variables. Another reason the usefulness of the model in (8) is limited is that it is not closed under independent additions: if $V^n$ and ${\overline{V}}^n$ are independent, each distributed according to (6); then, ${\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\overline{\mathrm{\Lambda }}}^{\frac{1}{2}}{\overline{V}}^n$, while multivariate Cauchy, does not have a density of the type in (8) unless $\mathrm{\Lambda }=\alpha \overline{\mathrm{\Lambda }}$ for some $\alpha >0$.

8. Another generalization of the (univariate) Cauchy distribution, which comes into play in our analysis, was introduced by Rider in 1958 [7]. With $\rho >0$ and $\beta \rho >1$,

   $$\begin{array}{cc}\hfill f_{V_{\beta ,\rho }}\left(x\right)& =\frac{{\kappa }_{\beta ,\rho }}{{(1+|x|}^{\rho }{)}^{\beta }},x\in \mathbb{R},\hfill \end{array}$$ (9)

   $$\begin{array}{cc}\hfill {\kappa }_{\beta ,\rho }& =\frac{\rho \mathrm{\Gamma }\left(\beta \right)}{2\mathrm{\Gamma }\left(\frac{1}{\rho }\right)\mathrm{\Gamma }\left(\beta -\frac{1}{\rho }\right)}.\hfill \end{array}$$ (10)

   In addition to the $(\beta ,\rho )$ parametrization in (9), we may introduce scale and location parameters by means of $\lambda V_{\beta ,\rho }+\mu$, just as we did in the Cauchy case $(\beta ,\rho )=(1,2)$. Another notable special case is $\sqrt{\nu }{V}_{\frac{\nu +1}{2},2}$, which is the centered Student-t random variable, itself equivalent to a Pearson type VII distribution.

3. Differential Entropy

• 9.
  The differential entropy of a Cauchy random variable is

  $$\begin{array}{cc}\hfill h(\lambda V+\mu )& =log\left|\lambda \right|+h\left(V\right),\hfill \end{array}$$ (11)

  $$\begin{array}{cc}\hfill h\left(V\right)& =-{\int }_{-\infty }^{\infty }{f}_V\left(t\right)log{f}_V\left(t\right)\mathrm{d}t=log\left(4\pi \right),\hfill \end{array}$$ (12)

  using (A3). Throughout this paper, unless the logarithm base is explicitly shown, it can be chosen by the reader as long as it is the same on both sides of the equation. For natural logarithms, the information measure unit is the nat.
• 10.
  An alternative, sometimes advantageous, expression for the differential entropy of a real-valued random variable is feasible if its cumulative distribution function $F_X$ is continuous and strictly monotonic. Then, the quantile function is its functional inverse, i.e., $F_X\left({\mathbb{Q}}_X\left(t\right)\right)=t$ for all $t\in (0,1)$, which implies that ${\dot{\mathbb{Q}}}_X\left(t\right)f_X\left({\mathbb{Q}}_X\left(t\right)\right)=1$ for all $t\in (0,1)$. Moreover, since X and ${\mathbb{Q}}_X\left(U\right)$ with U uniformly distributed on $[0,1]$ have identical distributions, we obtain

  $$\begin{array}{c}\hfill h\left(X\right)=\mathbb{E}[-log{f}_X\left(X\right)]=\mathbb{E}[-log{f}_X\left({\mathbb{Q}}_X\left(U\right)\right)]={\int }_0^{1}log{\dot{\mathbb{Q}}}_X\left(t\right)\mathrm{d}t.\end{array}$$ (13)

  Since (4) is indeed continuous and strictly monotonic, we can verify that we recover (12) by means of (5), (13) and (A2).

• 11.
  Despite not having finite moments, an independent identically distributed sequence of Cauchy random variables $\left\{Z_i\right\}$ is information stable in the sense that

  $$\begin{array}{c}\hfill \frac{1}{n}\sum _{i=1}^{n}log{f}_Z\left(Z_i\right)\to -h\left(Z\right),a.s.\end{array}$$ (14)

  because of the strong law of large numbers.
• 12.
  With $V^n$ distributed according to the standard spherical multivariate Cauchy density in (6), it is shown in [8] that

  $$\begin{array}{c}\hfill \mathbb{E}\left[{log}_{\mathrm{e}}\left(1+\parallel V^n{\parallel }^2\right)\right]=\psi \left(\frac{n+1}{2}\right)+{log}_{\mathrm{e}}4+\gamma ,\end{array}$$ (15)

  where $\gamma$ is the Euler–Mascheroni constant and $\psi (·)$ is the digamma function. Therefore, the differential entropy of (6) is, in nats, (see also [9])

  $$\begin{array}{cc}\hfill h\left(V^n\right)& =\frac{n+1}{2}\mathbb{E}\left[{log}_{\mathrm{e}}\left(1+\parallel V^n{\parallel }^2\right)\right]+\frac{n+1}{2}{log}_{\mathrm{e}}\pi -{log}_{\mathrm{e}}\mathrm{\Gamma }\left(\frac{n+1}{2}\right)\hfill \end{array}$$ (16)

  $$\begin{array}{cc}\hfill & =\frac{n+1}{2}\left({log}_{\mathrm{e}}\left(4\pi \right)+\gamma +\psi \left(\frac{n+1}{2}\right)\right)-{log}_{\mathrm{e}}\mathrm{\Gamma }\left(\frac{n+1}{2}\right),\hfill \end{array}$$ (17)

  whose growth is essentially linear with n: the conditional differential entropy
  $h\left(V_{n+1}\right|V^n)=h\left(V^{n+1}\right)-h\left(V^n\right)$ is monotonically decreasing with

  $$\begin{array}{cc}\hfill h\left(V_2\right|V_1)& =\frac{3}{2}(\gamma +\psi \left(\frac{3}{2}\right))+{log}_{\mathrm{e}}4=2.306...\hfill \end{array}$$ (18)

  $$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}h\left(V_{n+1}\right|V^n)& =\frac{1}{2}(1+\gamma +{log}_{\mathrm{e}}\left(4\pi \right))=2.054...\hfill \end{array}$$ (19)
• 13.
  By the scaling law of differential entropy and its invariance to location, we obtain

  $$\begin{array}{c}\hfill h\left({\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+\mathbf{\mu }\right)=h\left(V^n\right)+\frac{1}{2}log|det\left(\mathrm{\Lambda }\right)|.\end{array}$$ (20)
• 14.
  Invoking (A6), we obtain a closed-form formula for the differential entropy, in nats, of the generalized Cauchy distribution (9) as

  $$\begin{array}{cc}\hfill h\left(V_{\beta ,\rho }\right)& =-{log}_{\mathrm{e}}{\kappa }_{\beta ,\rho }+\beta \mathbb{E}\left[{log}_{\mathrm{e}}\left(1+|V_{\beta ,\rho }{|}^{\rho }\right)\right]\hfill \end{array}$$ (21)

  $$\begin{array}{cc}\hfill & =-{log}_{\mathrm{e}}{\kappa }_{\beta ,\rho }+\beta \psi \left(\beta \right)-\beta \psi \left(\beta -\frac{1}{\rho }\right),\hfill \end{array}$$ (22)

  with ${\kappa }_{\beta ,\rho }$ defined in (10).
• 15.
  The Rényi differential entropy of order $\alpha \in (0,1)\cup (1,\infty )$ of an absolutely continuous random variable with probability density function $f_X$ is

  $$\begin{array}{c}\hfill h_{\alpha }\left(X\right)=\frac{1}{1-\alpha }log\int f_X^{\alpha }\left(t\right)\mathrm{d}t.\end{array}$$ (23)
  For Cauchy random variables, we obtain, with the aid of (A12),

  $$\begin{array}{cc}\hfill h_{\alpha }(\lambda V+\mu )& =log\left|\lambda \right|+h_{\alpha }\left(V\right),\hfill \end{array}$$ (24)

  $$\begin{array}{cc}\hfill h_{\alpha }\left(V\right)& =\frac{\frac{1}{2}-\alpha }{1-\alpha }log\pi +\frac{1}{1-\alpha }log\frac{\mathrm{\Gamma }(\alpha -\frac{1}{2})}{\mathrm{\Gamma }\left(\alpha \right)},\alpha >\frac{1}{2},\hfill \end{array}$$ (25)

  which is infinite for $\alpha \in (0,\frac{1}{2}]$, converges to $log\left(4\pi \right)$ (cf. (12)) as $\alpha \to 1$, and to $log\pi$, the reciprocal of the mode height, as $\alpha \to \infty$.
• 16.
  Invoking (A13), the Rényi differential entropy of order $\alpha \in \left(\frac{1}{\beta \rho },1\right)\cup (1,\infty )$ of the generalized Cauchy distribution (9) is

  $$\begin{array}{c}\hfill h_{\alpha }\left(V_{\beta ,\rho }\right)=\frac{\alpha }{1-\alpha }log{\kappa }_{\beta ,\rho }+\frac{1}{1-\alpha }log\frac{2\mathrm{\Gamma }\left(\beta \alpha -\frac{1}{\rho }\right)\mathrm{\Gamma }\left(\frac{1}{\rho }\right)}{\rho \mathrm{\Gamma }\left(\beta \alpha \right)}.\end{array}$$ (26)

4. The Shannon- and $\mathbf{\eta }$-Transforms

In this section, we recall the definitions of two notions introduced in [10] for the unrelated purpose of expressing the asymptotic singular value distribution of large random matrices.

• 17.
  The Shannon transform of a nonnegative random variable X is the function ${\mathcal{V}}_X:[0,\infty )\to [0,\infty )$, defined by

  $$\begin{array}{cc}\hfill {\mathcal{V}}_X\left(\theta \right)& =\mathbb{E}\left[{log}_{\mathrm{e}}\left(1+\theta X\right)\right].\hfill \end{array}$$ (27)

  Unless ${\mathcal{V}}_X\left(\theta \right)=\infty$ for all $\theta >0$ (e.g., if X has the log-Cauchy density $\frac{1}{\pi x}\frac{1}{1+{log}^{2}x},x>0$), or ${\mathcal{V}}_X\left(\theta \right)=0$, $\theta \ge 0$, (which occurs if $X=0$ a.s.), the Shannon transform is a strictly concave continuous function from ${\mathcal{V}}_X\left(0\right)=0$, which grows without bound as $\theta \to \infty$.

• 18.
  If V is standard Cauchy, then (A4) results in

  $$\begin{array}{cc}\hfill {\mathcal{V}}_{V^2}\left({\theta }^2\right)& =2{log}_{\mathrm{e}}\left(1+\left|\theta \right|\right),\hfill \end{array}$$ (28)

  and the handy relationship

  $$\begin{array}{c}\hfill \mathbb{E}\left[log\left({\beta }^2+{\lambda }^2{V}^2\right)\right]=2log\left(\left|\beta \right|+\left|\lambda \right|\right).\end{array}$$ (29)
• 19.
  For the distribution in (9) with $(\beta ,\rho )=(2,2)$, (A7) results in

  $$\begin{array}{cc}\hfill {\mathcal{V}}_{V_{2,2}^2}\left({\theta }^2\right)& =2{log}_{\mathrm{e}}\left(1+\left|\theta \right|\right)-\frac{2\left|\theta \right|}{1+\left|\theta \right|}.\hfill \end{array}$$ (30)
• 20.
  The $\eta$-transform ${\eta }_X:[0,\infty )\to (0,1]$ of a non-negative random variable is defined as the function

  $$\begin{array}{c}\hfill {\eta }_X\left(\theta \right)=\mathbb{E}\left[\frac{1}{1+\theta X}\right]=1-\theta {\dot{\mathcal{V}}}_X\left(\theta \right),\end{array}$$ (31)

  which is intimately related to the Cauchy–Stieltjes transform [11]. For example,

  $$\begin{array}{cc}\hfill {\eta }_{V^2}\left({\theta }^2\right)& =\frac{1}{1+\left|\theta \right|},\hfill \end{array}$$ (32)

  $$\begin{array}{cc}\hfill {\eta }_{V_{2,2}^2}\left({\theta }^2\right)& =\frac{1+2\left|\theta \right|}{{(1+|\theta \left|\right)}^2}.\hfill \end{array}$$ (33)

5. Strength

The purpose of this section is to introduce an attribute which is particularly useful to compare random variables that do not have finite moments.

• 21.
  The strength $\varsigma \left(Z\right)\in [0,+\infty ]$ of a real-valued random variable Z is defined as

  $$\begin{array}{c}\hfill \varsigma \left(Z\right)=inf\left\{\varsigma >0:\mathbb{E}\left[log\left(1+\frac{Z^2}{{\varsigma }^2}\right)\right]\le log4\right\}.\end{array}$$ (34)
  It follows that the only random variable with zero strength is $Z=0$, almost surely. If the inequality in (34) is not satisfied for any $\varsigma >0$, then $\varsigma \left(Z\right)=\infty$. Otherwise, $\varsigma \left(Z\right)$ is the unique positive solution $\varsigma >0$ to

  $$\begin{array}{c}\hfill \mathbb{E}\left[log\left(1+\frac{Z^2}{{\varsigma }^2}\right)\right]=log4.\end{array}$$ (35)

  If $\varsigma \left(Z\right)\le \varsigma$, then (35) holds with ≤.

• 22.
  The set of probability measures whose strength is upper bounded by a given finite nonnegative constant,

  $$\begin{array}{c}\hfill {\mathcal{A}}_{\varsigma }=\left\{P_Z:\varsigma \left(Z\right)\le \varsigma \right\},\end{array}$$ (36)

  is convex: The set ${\mathcal{A}}_0$ is a singleton as seen in Item 21, while, for $0<\varsigma <\infty$, we can express (36) as

  $$\begin{array}{c}\hfill {\mathcal{A}}_{\varsigma }=\left\{P_Z:\mathbb{E}\left[log\left(1+\frac{Z^2}{{\varsigma }^2}\right)\right]\le log4\right\}.\end{array}$$ (37)

  Therefore, if $P_{Z_0}\in {\mathcal{A}}_{\varsigma }$ and $P_{Z_1}\in {\mathcal{A}}_{\varsigma }$, we must have $\alpha P_{Z_1}+(1-\alpha )P_{Z_0}\in {\mathcal{A}}_{\varsigma }$.

• 23.
  The peculiar constant in the definition of strength is chosen so that if V is standard Cauchy, then its strength is $\varsigma \left(V\right)=1$ because, in view of (29),

  $$\begin{array}{c}\hfill \mathbb{E}\left[log\left(1+V^2\right)\right]=log4.\end{array}$$ (38)
• 24.
  If $Z=k\in \mathbb{R}$, a.s., then its strength is

  $$\begin{array}{c}\hfill \varsigma \left(Z\right)=\frac{\left|k\right|}{\sqrt{3}}.\end{array}$$ (39)
• 25.
  The left side of (35) is the Shannon transform of $Z^2$ evaluated at ${\varsigma }^{-2}$, which is continuous in ${\varsigma }^2$. If $\varsigma \left(Z\right)\in (0,\infty )$ then, (35) can be written as

  $$\begin{array}{c}\hfill {\varsigma }^2\left(Z\right)=\frac{1}{{\mathcal{V}}_{Z^2}^{-1}\left({log}_{\mathrm{e}}4\right)},\end{array}$$ (40)

  where, on the right side, we have denoted the functional inverse of the Shannon transform. Clearly, the square root of the right side of (40) cannot be expressed as the expectation with respect to Z of any $\mathsf{b}:\mathbb{R}\to \mathbb{R}$ that does not depend on $P_Z$. Nevertheless, thanks to (37), (36) can be expressed as

  $$\begin{array}{c}\hfill {\mathcal{A}}_{\varsigma }=\left\{P_Z:\mathbb{E}\left[{\mathsf{b}}_{{\varsigma }^2}\left(Z\right)\right]\le 1\right\},with{\mathsf{b}}_{{\varsigma }^2}\left(x\right)={log}_4\left(1+\frac{x^2}{{\varsigma }^2}\right).\end{array}$$ (41)
• 26.

  Theorem 1.

  The strength of a real-valued random variable satisfies the following properties:

  • (a) 
    $$\begin{array}{c}\hfill \\ \hfill \varsigma \left(\lambda Z\right)=\left|\lambda \right|\varsigma \left(Z\right).\end{array}$$ (42)
  • (b) 
    $$\begin{array}{c}\hfill {\varsigma }^2\left(Z\right)\le \frac{1}{3}\mathbb{E}\left[Z^2\right],\end{array}$$ (43)
    with equality if and only if $\left|Z\right|$ is deterministic.
  • (c) 
    If $0<q<2$, and ${\parallel Z\parallel }_q={\mathbb{E}}^{\frac{1}{q}}{\left[\right|Z|}^q]<\infty$, then

    $$\begin{array}{c}\hfill \varsigma \left(Z\right)\le {\kappa }_q^{\frac{1}{q}}{\parallel Z\parallel }_q,with{\kappa }_q=\underset{x>0}{max}\frac{{log}_4(1+x^2)}{x^q}.\end{array}$$ (44)
  • (d) 
    If V is standard Cauchy, independent of X, then $\varsigma (X+V)$ is the solution to

    $$\begin{array}{c}\hfill {\mathcal{V}}_{X^2}\left({(\varsigma +1)}^{-2}\right)=2log\frac{2}{1+{\varsigma }^{-1}},\end{array}$$ (45)

    if it exists, otherwise, $\varsigma (X+V)=\infty$. Moreover, ≤ holds in (45) if $\varsigma (X+V)\le \varsigma$.
  • (e) 
    $$\begin{array}{c}\hfill 2log\left(2min\{1,\varsigma (Z\left)\right\}\right)\le \mathbb{E}\left[log\left(1+Z^2\right)\right]\le 2log\left(2max\{1,\varsigma (Z\left)\right\}\right).\end{array}$$ (46)
  • (f) 
    If $0<\varsigma \left(Z\right)<\infty$, then

    $$\begin{array}{c}\hfill h\left(Z\right)=log\left(4\pi \varsigma \right(Z\left)\right)-D(Z\parallel \varsigma (Z\left)V\right),\end{array}$$ (47)

    where V is standard Cauchy, and $D(X\parallel Y)$ stands for the relative entropy with reference probability measure $P_Y$ and dominated measure $P_X$.
  • (g) 
    $$\begin{array}{c}\hfill h\left(Z\right)<\infty ⟸\varsigma \left(Z\right)<\infty ⟺\mathbb{E}\left[log\left(1+Z^2\right)\right]<\infty .\end{array}$$ (48)
  • (h) 
    If V is standard Cauchy, then

    $$\begin{array}{c}\hfill \varsigma \left(Z\right)<\infty andh\left(Z\right)\in \mathbb{R}⟺D(Z\parallel \lambda V)<\infty ,forall\lambda >0.\end{array}$$ (49)
  • (i) 
    The finiteness of strength is sufficient for the finiteness of the entropy of the integer part of the random variable, i.e.,

    $$H(\lfloor Z\rfloor )=\infty ⟹\varsigma \left(Z\right)=\infty .$$
  • (j) 
    If $Z_n\to Z$ in $L_q$ for any $q\in (0,1]$, then $\varsigma \left(Z_n\right)\to \varsigma \left(Z\right)$.
  • (k) 
    $$\begin{array}{c}\hfill Z_n\to 0i.p.⟸\mathbb{E}\left[log\left(1+Z_n^2\right)\right]\to 0⟺\varsigma \left(Z_n\right)\to 0.\end{array}$$ (50)
  • (l) 
    If $\varsigma \left(X_n\right)\to 0$, then $\varsigma (Z+X_n)\to \varsigma \left(Z\right)$.
  • (m) 
    If $\varsigma \left(X_n\right)\to 0$, $\varsigma \left(Z\right)<\infty$ and Z is independent of $X_n$, then $h(Z+X_n)\to h\left(Z\right)$.

  Proof. 

  For the first three properties, it is clear that they are satisfied if $\varsigma \left(Z\right)=0$, i.e., $Z=0$ almost surely.

  • (a)
    If ${\varsigma }^2\in (0,\infty )$ is the solution to (35), then ${\lambda }^2{\varsigma }^2$ is a solution to (35) with $\lambda Z$ taking the role of Z. If (35) has no solution, neither does its version in which $\lambda Z$ takes the role of Z.
  • (b)
    Jensen’s inequality applied to the left side of (35) results in $3{\varsigma }^2\le \mathbb{E}\left[Z^2\right]$. The strict concavity of $log(1+t)$ implies that equality holds if and only if $Z^2$ is deterministic. If (35) has no solution, the same reasoning implies that $\mathbb{E}\left[Z^2\right]=\infty$.
  • (c)
    First, it is easy to check that, for $q\in (0,2)$, the function $f_q:(0,\infty )\to (0,\infty )$ given by $f_q\left(t\right)=t^{-q}{log}_4(1+t^2)$ attains its maximum ${\kappa }_q$ at a unique point. Assume $\varsigma \left(Z\right)\in (0,\infty )$. Since ${\kappa }_q{t}^q\ge {log}_4(1+t^2)$ for all $t>0$, letting $t=\left|Z\right|/\varsigma \left(Z\right)$ and taking expectations, (35) (choosing 4 as the logarithm base) results in

    $$\begin{array}{c}\hfill \frac{{\kappa }_q}{{\varsigma }^q\left(Z\right)}\mathbb{E}\left[{\left|Z\right|}^q\right]\ge 1,\end{array}$$ (51)

    which is the same as (44). If $\varsigma \left(Z\right)=\infty$, then $\infty =\mathbb{E}[log(1+Z^2)]\le {\kappa }_q{\mathbb{E}\left[\right|Z|}^q]$.
  • (d)
    Invoking (A4) with ${\alpha }^2={\varsigma }^2+x^2$ and $|sin\beta |=\frac{\varsigma }{\sqrt{x^2+{\varsigma }^2}}$, we obtain

    $$\begin{array}{cc}\hfill \mathbb{E}\left[log\left(1+\frac{{(x+V)}^2}{{\varsigma }^2}\right)\right]& =log\left(\frac{{(1+\varsigma )}^2+x^2}{{\varsigma }^2}\right)\hfill \end{array}$$ (52)

    $$\begin{array}{cc}\hfill & =log\left(1+\frac{x^2}{{(1+\varsigma )}^2}\right)-2log\frac{\varsigma }{\varsigma +1}.\hfill \end{array}$$ (53)
    Substituting x by X and averaging over X, the result follows from the definition of strength.
  • (e)
    The result holds trivially if either $\varsigma \left(Z\right)=0$ or $\varsigma \left(Z\right)=\infty$. Otherwise, we simply rewrite (35) as

    $$\begin{array}{c}\hfill 2log\left(2\varsigma \left(Z\right)\right)=\mathbb{E}\left[log\left({\varsigma }^2\left(Z\right)+Z^2\right)\right],\end{array}$$ (54)

    and upper/lower bound the right side by $\mathbb{E}\left[log\left(1+Z^2\right)\right]$.
  • (f)
    $$\begin{array}{cc}\hfill D(Z\parallel \varsigma (Z\left)V\right)& =-h\left(Z\right)+log\left(\varsigma \left(Z\right)\pi \right)+\mathbb{E}\left[log\left(1+\frac{Z^2}{{\varsigma }^2\left(Z\right)}\right)\right]\hfill \end{array}$$ (55)

    $$\begin{array}{cc}\hfill & =log\left(4\pi \varsigma \right(Z\left)\right)-h\left(Z\right),\hfill \end{array}$$ (56)

    where (55) and (56) follow from (2) and (35), respectively.
  • (g)
    • If $\varsigma \left(Z\right)<\infty$, then $\mathbb{E}\left[log\left(1+Z^2\right)\right]<\infty$ and $h\left(Z\right)<\infty$ follow from (46) and (47), respectively.
    • If $\mathbb{E}\left[log(1+Z^2)\right]<\infty$, the dominated convergence theorem implies

      $$\begin{array}{c}\hfill \underset{\varsigma \to \infty }{lim}\mathbb{E}\left[log\left(1+{\left(\frac{Z}{\varsigma }\right)}^2\right)\right]=0.\end{array}$$ (57)

      Excluding the case $Z=0$ a.s. for which both $\mathbb{E}\left[log(1+Z^2)\right]$ and $\varsigma \left(Z\right)$ are zero, we have

      $$\begin{array}{c}\hfill \underset{\varsigma \downarrow 0}{lim}\mathbb{E}\left[log\left(1+{\left(\frac{Z}{\varsigma }\right)}^2\right)\right]=\underset{\varsigma \downarrow 0}{lim}{\mathcal{V}}_{Z^2}\left(\frac{1}{{\varsigma }^2}\right)=\infty .\end{array}$$ (58)

      Since (35) is continuous in $\varsigma$, it must have a finite solution in view of (57) and (58).
  • (h)
    It is sufficient to assume $\lambda =1$ for the condition on the right of (49) because the condition on the left holds if and only if it holds for $\alpha Z$, for any $\alpha >0$ and $D(\alpha Z\parallel \alpha V)=D(Z\parallel V)$. If $h\left(Z\right)<\infty$, then

    $$\begin{array}{c}\hfill D(Z\parallel V)=-h\left(Z\right)+log\pi +\mathbb{E}\left[log\left(1+Z^2\right)\right],\end{array}$$ (59)

    which is finite unless either $h\left(Z\right)=-\infty$ or $\mathbb{E}[log(1+Z^2)]=\infty$. This establishes ⟹ in view of (48). To establish ⟸, it is enough to show that

    $$\begin{array}{c}\hfill D(Z\parallel V)<\infty ⟹\mathbb{E}\left[log\left(1+Z^2\right)\right]<\infty ,\end{array}$$ (60)

    in view of (48) and the fact that, according to (59), $h\left(Z\right)>-\infty$ if both $D(Z\parallel V)$ and $\mathbb{E}\left[log\left(1+Z^2\right)\right]$ are finite. To show (60), we invoke the following variational representation of relative entropy (first noted by Kullback [12] for absolutely continuous random variables): If $P_Z\ll P_V$, then

    $$\begin{array}{c}\hfill D(Z\parallel V)=\underset{Q:Q\ll P_V}{max}\mathbb{E}\left[log\frac{\mathrm{d}Q}{\mathrm{d}{P}_V}\left(Z\right)\right],\end{array}$$ (61)

    attained only at $Q=P_Z$. Let Q be the absolutely continuous random variable with probability density function

    $$\begin{array}{c}\hfill q\left(x\right)=\frac{{log}_{\mathrm{e}}2}{4\left|x\right|{log}_{\mathrm{e}}^2\left|x\right|}1\left\{\right|x|\ge 2\}+\frac{1}{8}1\left\{\right|x|<2\}.\end{array}$$ (62)
    Then,

    $$\begin{array}{cc}\hfill \infty & >D(Z\parallel V)>\mathbb{E}\left[log\frac{q\left(Z\right)}{f_V\left(Z\right)}\right]\hfill \\ \hfill & =\mathbb{E}\left[1\left\{\right|Z|\ge 2\}\left(log\frac{\pi {log}_{\mathrm{e}}2}{4}+log\left(1+Z^2\right)-log\left(\left|Z\right|{log}_{\mathrm{e}}^2\left|Z\right|\right)\right)\right]\hfill \end{array}$$ (63)

    $$\begin{array}{cc}\hfill & +\mathbb{E}\left[1\left\{\right|Z|<2\}\left(log\frac{\pi }{8}+log\left(1+Z^2\right)\right)\right]\hfill \end{array}$$ (64)

    $$\begin{array}{cc}\hfill & >\frac{1}{5}\mathbb{E}\left[1\{Z\ge 2\}log\left(1+Z^2\right)\right]+log\frac{5\pi }{8}\hfill \end{array}$$ (65)

    $$\begin{array}{cc}\hfill & \ge \frac{1}{5}\mathbb{E}\left[log\left(1+Z^2\right)\right]+\frac{4}{5}log5-log\frac{8}{\pi },\hfill \end{array}$$ (66)

    where (65) holds since

    $$\begin{array}{c}\hfill \frac{4}{5}log(1+x^2)\ge -log\left(\pi {log}_{\mathrm{e}}2\right)+2log{log}_{\mathrm{e}}\left|x\right|+log\left|x\right|,\left|x\right|>2.\end{array}$$ (67)
  • (i)
    $$\begin{array}{cc}\hfill \varsigma \left(Z\right)<\infty & ⟹\mathbb{E}\left[log(1+Z^2)\right]<\infty \hfill \end{array}$$ (68)

    $$\begin{array}{cc}\hfill & ⟹\mathbb{E}\left[log(1+|Z\left|\right)\right]<\infty \hfill \end{array}$$ (69)

    $$\begin{array}{cc}\hfill & ⟹H(\lfloor Z\rfloor )<\infty ,\hfill \end{array}$$ (70)

    where (68)–(70) follow from (48), $log(1+x^2)\le 2log(1+|x\left|\right)$, and p. 3743 in [13], respectively.
  • (j)
    If $\varsigma \left(Z\right)=0$, then $Z=0$ a.e., and the result follows from (44). For all $(x,z)\in {\mathbb{R}}^2$,

    $$\begin{array}{cc}\hfill \left|{log}_{\mathrm{e}}\left(\frac{1+{(x+z)}^2}{1+z^2}\right)\right|& \le {log}_{\mathrm{e}}\left(1+\frac{1}{2}(x^2+\left|x\right|\sqrt{4+x^2})\right)\hfill \end{array}$$ (71)

    $$\begin{array}{cc}\hfill & \le \frac{2}{q}{\left|x\right|}^q,\hfill \end{array}$$ (72)

    where (71) follows by maximizing the left side over $z\in \mathbb{R}$. Denote the difference between the right side and the left side of (72) by $f_q\left(x\right)$, an even function which satisfies $f_q\left(0\right)=0$, and

    $$\begin{array}{c}\hfill {\dot{f}}_q\left(x\right)=2x^{q-1}-\frac{2}{\sqrt{4+x^2}}>0,x>0,0<q\le 1.\end{array}$$ (73)
    Therefore, (72) follows. Assuming $0<\varsigma \left(Z\right)<\infty$, we have

    $$\begin{array}{cc}\hfill \left|\mathbb{E}\left[log(1+Z_n^2)\right]-\mathbb{E}\left[log(1+Z^2)\right]\right|& \le \mathbb{E}\left[\left|log(1+Z_n^2)-log(1+Z^2)\right|\right]\hfill \end{array}$$ (74)

    $$\begin{array}{cc}\hfill & \le \frac{2}{q}\mathbb{E}\left[|Z_n{-Z|}^q\right]log\mathrm{e}.\hfill \end{array}$$ (75)
    Now, because of the scaling property in (42), we may assume without loss of generality that $\varsigma \left(Z\right)=1$. Thus, (74) and (75) result in

    $$\begin{array}{c}\hfill \left|\mathbb{E}\left[log(1+Z_n^2)\right]-log4\right|\le \frac{2}{q}\mathbb{E}\left[|Z_n{-Z|}^q\right]log\mathrm{e},\end{array}$$ (76)

    which requires that $\varsigma \left(Z_n\right)\to 1$, since, by assumption, the right side vanishes. Assume now that $\varsigma \left(Z\right)=\infty$, and therefore, $\mathbb{E}\left[log(1+Z^2)\right]=\infty$. Inequality (75) remains valid in this case, implying that, as soon as the right side is finite (which it must be for all sufficiently large n), $\mathbb{E}\left[log(1+Z_n^2)\right]=\infty$, and therefore, $\varsigma \left(Z_n\right)=\infty$ in view of (48).
  • (k)
    • 1st ⟸
           For any $ϵ>0$, Markov’s inequality results in

      $$\begin{array}{c}\hfill \mathbb{P}\left[\right|Z_n|>ϵ]=\mathbb{P}\left[log\left(1+Z_n^2\right)>log\left(1+{ϵ}^2\right)\right]\le \frac{\mathbb{E}\left[log\left(1+Z_n^2\right)\right]}{log\left(1+{ϵ}^2\right)}.\end{array}$$ (77)
    • ⟹
      First, we show that, for any $\alpha >0$, we have

      $$\begin{array}{c}\hfill \mathbb{E}\left[log\left(1+Z_n^2\right)\right]\to 0⟹\mathbb{E}\left[log\left(1+\alpha Z_n^2\right)\right]\to 0.\end{array}$$ (78)

      The case $0<\alpha <1$ is trivial. The case $\alpha >1$ follows because $\mathbb{E}\left[log\left(1+Z_n^2\right)\right]$$\to 0$ implies

      $$\begin{array}{c}\hfill \mathbb{E}\left[log\left(1+\alpha Z_n^2\right)\right]=\mathbb{E}\left[log\left(1+(\alpha -1)Z_n^2\right)\right],\end{array}$$ (79)

      where ≥ is obvious, and ≤ holds because

      $$\begin{array}{cc}\hfill log\left(1+\alpha t^2\right)& =log\left(1+t^2\right)+log\left(1+(\alpha -1)\frac{t^2}{1+t^2}\right)\hfill \end{array}$$ (80)

      $$\begin{array}{cc}\hfill & \le log\left(1+t^2\right)+log\left(1+(\alpha -1)t^2\right).\hfill \end{array}$$ (81)

      If $\varsigma \left(Z_n\right)=\infty$ infinitely often, so is $\mathbb{E}\left[log\left(1+Z_n^2\right)\right]$ in view of (48). Assume that $lim\; sup\varsigma \left(Z_n\right)=\varsigma \in (0,\infty ]$, and $\varsigma \left(Z_n\right)$ is finite for all sufficiently large. Then, there is a subsequence such that $\varsigma \left(Z_{n_i}\right)\nrightarrow$, and

      $$\begin{array}{c}\hfill log4=\mathbb{E}\left[log\left(1+{\left(\frac{Z_{n_i}}{\varsigma \left(Z_{n_i}\right)}\right)}^2\right)\right]\le \mathbb{E}\left[log\left(1+{\left(\frac{Z_{n_i}}{\lambda }\right)}^2\right)\right],\end{array}$$ (82)

      for all sufficiently large i and $\lambda <\varsigma$. Consequently, (78) implies that $\mathbb{E}\left[log\left(1+Z_n^2\right)\right]\neg \to 0$.
    • 2nd ⟸
         Suppose that $\mathbb{E}\left[log\left(1+Z_n^2\right)\right]\nrightarrow 0$. Therefore, there is a subsequence along which $\mathbb{E}\left[log\left(1+Z_{n_i}^2\right)\right]>\eta >0$. If $\eta \ge log4$, then $\varsigma \left(Z_{n_i}\right)>1$ along the subsequence. Because of the continuity of the Shannon transform and the fact that it grows without bound as its argument goes to infinity (Item 25), if $0<\eta <log4$, we can find $1<\alpha <\infty$ such that $\mathbb{E}\left[log\left(1+\alpha Z_{n_i}^2\right)\right]>log4$, which implies $\varsigma \left(Z_{n_i}\right)>{\alpha }^{-1/2}$. Therefore, $\varsigma \left(Z_n\right)\nrightarrow 0$ as we wanted to show.
  • (l)
    We start by showing that

    $$\begin{array}{c}\hfill \mathbb{E}\left[log\left(1+X_n^2\right)\right]\to 0⟺\mathbb{E}\left[f\left(X_n\right)\right]\to 0,\end{array}$$ (83)

    where we have denoted the right side of (71) with arbitrary logarithm base by $f\left(x\right)$. Since $\dot{f}\left(x\right)=\frac{2log\mathrm{e}}{\sqrt{4+x^2}}$, it is easy to verify that

    $$\begin{array}{c}\hfill 0\le f\left(x\right)-log(1+x^2)\le log\frac{4}{3},x\in \mathbb{R},\end{array}$$ (84)

    where the lower and upper bounds are attained uniquely at $x=0$ and $\left|x\right|=\frac{1}{\sqrt{2}}$, respectively. The lower bound results in ⟸ in (83). To show ⟹, decompose, for arbitrary $ϵ>0$,

    $$\begin{array}{cc}\hfill \mathbb{E}\left[f\left(X_n\right)\right]& =\mathbb{E}\left[f\left(X_n\right)1\left\{\right|X_n|<ϵ\}\right]+\mathbb{E}\left[f\left(X_n\right)1\left\{\right|X_n|\ge ϵ\}\right]\hfill \end{array}$$ (85)

    $$\begin{array}{cc}\hfill & \le f\left(ϵ\right)+\mathbb{E}\left[f\left(X_n\right)1\left\{\right|X_n|\ge ϵ\}\right]\hfill \end{array}$$ (86)

    $$\begin{array}{cc}\hfill & \le f\left(ϵ\right)+A_{ϵ}\mathbb{E}\left[log\left(1+X_n^2\right)1\left\{\right|X_n|\ge ϵ\}\right]\hfill \end{array}$$ (87)

    $$\begin{array}{cc}\hfill & \le f\left(ϵ\right)+A_{ϵ}{ϵ}^3,\hfill \end{array}$$ (88)

    where

    $$\begin{array}{cc}\hfill A_{ϵ}& =1+\frac{log\frac{4}{3}}{log(1+{ϵ}^2)},\hfill \end{array}$$ (89)

    (87) holds from the upper bound in (84), and the fact that (89) is decreasing in $ϵ$, and (88) holds for all sufficiently large n if $\mathbb{E}\left[log\left(1+X_n^2\right)\right]\to 0$. Since the right side of (88) goes to 0 as $ϵ\to 0$, (83) is established. Assume $0<\varsigma \left(Z\right)<\infty$. From the linearity property (42), we have $\varsigma (Z+X_n)=\varsigma \left(Z\right)·\varsigma (\overline{Z}+{\overline{X}}_n)$ with $\overline{Z}={\varsigma }^{-1}\left(Z\right)Z$ and ${\overline{X}}_n={\varsigma }^{-1}\left(Z\right)X_n$ which satisfies $\varsigma \left({\overline{X}}_n\right)\to 0$. Therefore, we may restrict attention to $\varsigma \left(Z\right)=1$ without loss of generality. Following (71) and (74), and abbreviating $Z_n=Z+X_n$, we obtain

    $$\begin{array}{cc}\hfill \left|\mathbb{E}\left[log(1+Z_n^2)\right]-log4\right|& \le \mathbb{E}\left[\left|log(1+Z_n^2)-log(1+Z^2)\right|\right]\hfill \end{array}$$ (90)

    $$\begin{array}{cc}\hfill & \le \mathbb{E}\left[f\left(X_n\right)\right].\hfill \end{array}$$ (91)
    Thus, the desired result follows in view of (50) and (83). To handle the case $\varsigma \left(Z\right)=\infty$, we use the same reasoning as in the proof of (i) since (83) remains valid in that case.
  • (m)
    If $\varsigma \left(Z\right)=0$, then $Z=0$ a.s., $h\left(Z\right)=-\infty$ and $h\left(X_n\right)\to -\infty$ in view of Part (f). Assume henceforth that $\varsigma \left(Z\right)>0$. Since $h(Z+X_n)\ge h\left(Z\right)$, it suffices to show

    $$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}h(X_n+Z)\le h\left(Z\right).\end{array}$$ (92)

    Under the assumptions, Part (l) guarantees that

    $$\begin{array}{c}\hfill \varsigma (X_n+Z)\to \varsigma \left(Z\right).\end{array}$$ (93)

    If V is a standard Cauchy random variable, then $\varsigma (Z+X_n)V\to \varsigma \left(Z\right)V$ in distribution as the characteristic function converges: ${\mathrm{e}}^{-\varsigma (Z+X_n)\left|t\right|}\to {\mathrm{e}}^{-\varsigma \left(Z\right)\left|t\right|}$ for all t. Analogously, according to Part (k), $Z+X_n\stackrel{D}{⟶}Z$ since $X_n\to 0$ in probability. Since the strength of $X_n+Z$ is finite for all sufficiently large n, we may invoke (47) to express, for those n,

    $$\begin{array}{cc}\hfill h(X_n+Z)-h\left(Z\right)& =log\frac{\varsigma (Z+X_n)}{\varsigma \left(Z\right)}\hfill \\ \hfill & -D(Z+X_n\parallel \varsigma (Z+X_n)V)+D(Z\parallel \varsigma \left(Z\right)V).\hfill \end{array}$$ (94)

    The lower semicontinuity of relative entropy under weak convergence (which, in turn, is a corollary to the Donsker–Varadhan [14,15] variational representation of relative entropy) results in

    $$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}D(Z+X_n\parallel \varsigma (Z+X_n)V)\ge D(Z\parallel \varsigma \left(Z\right)V),\end{array}$$ (95)

    because $Z+X_n\stackrel{D}{⟶}Z$ and $\varsigma (Z+X_n)V\stackrel{D}{⟶}\varsigma \left(Z\right)V$. Therefore, (92) follows from (94) and (95).
       □

• 27.
  In view of (42) and Item 23, $\varsigma \left(\lambda V\right)=\left|\lambda \right|$ if V is standard Cauchy. Furthermore, if $X_1$ and $X_2$ are centered independent Cauchy random variables, then their sum is centered Cauchy with

  $$\begin{array}{c}\hfill \varsigma (X_1+X_2)=\varsigma \left(X_1\right)+\varsigma \left(X_2\right).\end{array}$$ (96)

  More generally, it follows from Theorem 1-(d) that, if $X_1$ is centered Cauchy, and (96) holds for $X_2=\alpha X$ and all $\alpha \in \mathbb{R}$, then X must be centered Cauchy. Invoking (52), we obtain

  $$\begin{array}{c}\hfill \varsigma (\lambda V+\mu )=\frac{\left|\lambda \right|}{3}+\frac{1}{3}\sqrt{4{\lambda }^2+3{\mu }^2},\end{array}$$ (97)

  which is also valid for $\lambda =0$ as we saw in Item 24.
• 28.
  If X is standard Gaussian, then ${\varsigma }^2\left(X\right)=0.171085\cdots$, and ${\varsigma }^2\left(\sigma X\right)={\sigma }^2{\varsigma }^2\left(X\right)$. Therefore, if $X_1$ and $X_2$ are zero-mean independent Gaussian random variables, then

  $$\begin{array}{c}\hfill {\varsigma }^2(X_1+X_2)={\varsigma }^2\left(X_1\right)+{\varsigma }^2\left(X_2\right).\end{array}$$ (98)

  Thus, in this case, $\varsigma (X_1+X_2)<\varsigma \left(X_1\right)+\varsigma \left(X_2\right)$.
• 29.
  It follows from Theorem 1-(d) that, with X independent of standard Cauchy V, we obtain $\varsigma (X+V)>\varsigma \left(X\right)+\varsigma \left(V\right)$ whenever X is such that

  $$\begin{array}{c}\hfill {\mathcal{V}}_{X^2}\left({(2+\varsigma \left(X\right))}^{-2}\right)>2{log}_{\mathrm{e}}\left(1+\frac{\varsigma \left(X\right)}{2+\varsigma \left(X\right)}\right).\end{array}$$ (99)

  An example is the heavy-tailed probability density function

  $$\begin{array}{c}\hfill f_X\left(x\right)=\frac{1}{\pi }\frac{{log}_4(1+x^2)}{1+x^2},\end{array}$$ (100)

  for which $7.0158\dots =\varsigma (X+V)>\varsigma \left(X\right)+\varsigma \left(V\right)=6.8457\dots$.
• 30.
  Using (A8), we can verify that, if X is zero-mean uniform with variance ${\sigma }^2$, then

  $$\begin{array}{c}\hfill {\varsigma }^2\left(X\right)=\frac{3}{c^2}{\sigma }^2=0.221618\dots {\sigma }^2,\end{array}$$ (101)

  where c is the solution to ${log}_{\mathrm{e}}(1+c^2)+\frac{2}{c}arctan\left(c\right)=2+{log}_{\mathrm{e}}4$.
• 31.
  We say that $Z_n\to 0$ in strength if $\varsigma \left(Z_n\right)\to 0$. Parts (j) and (k) of Theorem 1 show that this convergence criterion is intermediate between the traditional in probability and $L_q$ criteria. It is not equivalent to either one: If

  $$\begin{array}{c}\hfill Z_n=\left\{\begin{array}{cc}0,\hfill & \mathrm{with} \mathrm{probability}1-\frac{1}{n};\hfill \\ 2^n,\hfill & \mathrm{with} \mathrm{probability}\frac{1}{n},\hfill \end{array}\right.\end{array}$$ (102)

  then $\varsigma \left(Z_n\right)\to 1$, while $Z_n\to 0$ in probability. If, instead, $Z_n={\frac{3}{2}}^n$, with probability $\frac{1}{n}$, then $Z_n\to 0$ in strength, but not in $L_q$ for any $0<q$.
• 32.
  The assumption in Theorem 1-(m) that $X_n\to 0$ in strength cannot be weakened to convergence in probability. Suppose that $X_n$ is absolutely continuous with probability density function

  $$\begin{array}{c}\hfill f_{X_n}\left(t\right)=\left\{\begin{array}{cc}n-1,\hfill & t\in \left[0,\frac{1}{n}\right];\hfill \\ 0,\hfill & t\in (-\infty ,0)\cup \left(\frac{1}{n},2\right);\hfill \\ \frac{1}{n}\frac{{log}_{\mathrm{e}}2}{t{log}_{\mathrm{e}}^{2}t},\hfill & t\in [2,\infty ).\hfill \end{array}\right.\end{array}$$ (103)
  We have $X_n\to 0$ in probability since, regardless of how small $ϵ>0$, $\mathbb{P}[X_n>ϵ]=\frac{1}{n}$ for all $n\ge \frac{1}{ϵ}$. Furthermore,

  $$\begin{array}{c}\hfill h(X_n+Z)\ge h\left(X_n\right)=\infty ,\end{array}$$ (104)

  because (103) is the mixture of a uniform and an infinite differential entropy probability density function, and differential entropy is concave. We conclude that $h(X_n+Z)\nrightarrow h\left(Z\right)$, since $h\left(Z\right)<\infty$.
• 33.
  The following result on the continuity of differential entropy is shown in [16]: if X and Z are independent, $\mathbb{E}\left[\right|Z\left|\right]<\infty$ and $\mathbb{E}\left[\right|X\left|\right]<\infty$, then

  $$\begin{array}{c}\hfill \underset{ϵ\downarrow 0}{lim}h(ϵX+Z)=h\left(Z\right).\end{array}$$ (105)

  This result is weaker than Theorem 1-(m) because finite first absolute moment implies finite strength as we saw in (44), and $ϵX\to 0$ in $L_1$ if $ϵ\to 0$, and therefore, it vanishes in strength too.

• 34.
  If Z and V are centered and standard Cauchy, respectively, then ${min}_{\lambda }D(Z\parallel \lambda V)$ is achieved by $\lambda =\varsigma \left(Z\right)$. Otherwise, in general, this does not hold. Since $D(Z\parallel \lambda V)={\mathcal{V}}_{Z^2}\left({\lambda }^{-2}\right)-h\left(Z\right)+{log}_{\mathrm{e}}\left(\pi \lambda \right)$, the minimum is attained at the solution to

  $$\begin{array}{c}\hfill {\eta }_{Z^2}\left(\frac{1}{{\lambda }_{\ast }^2}\right)=\frac{1}{2},\end{array}$$ (106)

  where we have used the $\eta$-transform in (31). If $Z=V_{2,2}$, recalling (32), (106), results in ${\lambda }_{\ast }=\sqrt{2}-1$, while $\varsigma \left(V_{2,2}\right)=0.302\dots$.
• 35.
  Using (28) and the concavity of $log(1+x)$, we can verify that

  $$\begin{array}{c}\hfill \varsigma \left(X_{\alpha }\right)\le \alpha \varsigma \left(X_1\right)+(1-\alpha )\varsigma \left(X_0\right),X_{\alpha }\sim \alpha P_{X_1}+(1-\alpha )P_{X_0},\end{array}$$ (107)

  if $X_0$ and $X_1$ are centered Cauchy, or, more generally, if $X_0={\lambda }_{0}X$, $X_1={\lambda }_{1}X$ and ${\mathcal{V}}_{X^2}\left({\theta }^2\right)$ is concave on $\theta$. Not only is this property not satisfied if $X=1$ but (107) need not hold in that case, as we can verify numerically for $\alpha =0.1$, ${\lambda }_1=1$ and ${\lambda }_0>20$.

6. Maximization of Differential Entropy

• 36.
  Among random variables with a given second moment (resp. first absolute moment), differential entropy is maximized by the zero-mean Gaussian (resp. Laplace) distribution. More generally, among random variables with a given p-absolute moment $\mu$, differential entropy is maximized by the parameter-p Subbotin (or generalized normal) distribution with p-absolute moment $\mu$ [17]

  $$\begin{array}{c}\hfill f_X\left(x\right)=\frac{p^{1-\frac{1}{p}}}{2\mathrm{\Gamma }\left(\frac{1}{p}\right){\mu }^{\frac{1}{p}}}{\mathrm{e}}^{-\frac{{\left|x\right|}^p}{p\mu }},x\in \mathbb{R}.\end{array}$$ (108)

  Among nonnegative random variables with a given mean, differential entropy is maximized by the exponential distribution. In those well-known solutions, the cost function is an affine function of the negative logarithm of the maximal differential entropy probability density function. Is there a cost function such that, among all random variables with a given expected cost, the Cauchy distribution is the maximal differential entropy solution? To answer this question, we adopt a more general viewpoint. Consider the following result, whose special case $\rho =2$ was solved in [18] using convex optimization:

  Theorem 2.

  Fix $\rho >0$ and $\theta >0$.

  $$\begin{array}{c}\hfill \underset{Z:\mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|Z\right|}^{\rho }\right)\right]\le \theta }{max}h\left(Z\right)=h\left(V_{\beta ,\rho }\right),\end{array}$$ (109)

  where $V_{\beta ,\rho }$ is defined in (9), the right side of (109) is given in (22), and $\beta >{\rho }^{-1}$ is the solution to

  $$\begin{array}{c}\hfill \theta =\psi \left(\beta \right)-\psi \left(\beta -\frac{1}{\rho }\right).\end{array}$$ (110)

  Therefore, the standard Cauchy distribution is the maximal differential entropy distribution provided that $\rho =2$ and $\theta ={log}_{\mathrm{e}}4.$

  Proof. 

  • (a)
    For every $\rho >0$ and $\theta >0$, there is a unique $\beta >{\rho }^{-1}$ that satisfies (110) because the function of $\beta$ on the right side is strictly monotonically decreasing, grows without bound as $\beta \downarrow \frac{1}{\rho }$, and goes to zero as $\beta \to \infty$.
  • (b)
    For any Z which satisfies $\mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|Z\right|}^{\rho }\right)\right]\le \theta$, its relative entropy, in nats, with respect to $V_{\beta ,\rho }$ is

    $$\begin{array}{cc}\hfill D(Z\parallel V_{\beta ,\rho })& =-h\left(Z\right)-{log}_{\mathrm{e}}{\kappa }_{\beta ,\rho }+\beta \mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|Z\right|}^{\rho }\right)\right]\hfill \end{array}$$ (111)

    $$\begin{array}{cc}\hfill & \le -h\left(Z\right)-{log}_{\mathrm{e}}{\kappa }_{\beta ,\rho }+\beta \theta \hfill \end{array}$$ (112)

    $$\begin{array}{cc}\hfill & =-h\left(Z\right)-{log}_{\mathrm{e}}{\kappa }_{\beta ,\rho }+\beta \psi \left(\beta \right)-\beta \psi \left(\beta -\frac{1}{\rho }\right)\hfill \end{array}$$ (113)

    $$\begin{array}{cc}\hfill & =h\left(V_{\beta ,\rho }\right)-h\left(Z\right),\hfill \end{array}$$ (114)

    where (113) and (114) follow from (110) and (22), respectively. Since relative entropy is nonnegative, and zero only if both measures are identical, not only does (2) hold but any random variable other than $V_{\beta ,\rho }$ achieves strictly lower differential entropy.
       □

• 37.An unfortunate consequence stemming from Theorem 2 is that, while we were able to find out a cost function such that the Cauchy distribution is the maximal differential entropy distribution under an average cost constraint, this holds only for a specific value of the constraint. This behavior is quite different from the classical cases discussed in Item 36 for which the solution is, modulo scale, the same regardless of the value of the cost constraint. As we see next, this deficiency is overcome by the notion of strength introduced in Section 5.
• 38.

  Theorem 3.

  Strength constraint. The differential entropy of a real-valued random variable with strength $\varsigma \left(Z\right)$ is upper bounded by

  $$\begin{array}{c}\hfill h\left(Z\right)\le log\left(4\pi \varsigma \left(Z\right)\right).\end{array}$$ (115)

  If $0<\varsigma \left(Z\right)<\infty$, equality holds if and only if Z has a centered Cauchy density, i.e., $Z=\lambda V$ for some $\lambda >0$.

  Proof. 

  • (a)
    If Z is not an absolutely continuous random variable, or more generally, $h\left(Z\right)=-\infty$ such as in the case $\varsigma \left(Z\right)=0$ in which $Z=0$ with probability one, then (115) is trivially satisfied.
  • (b)
    If $0<\varsigma \left(Z\right)<\infty$ and $h\left(Z\right)>-\infty$, then we invoke (47) to conclude that not only does (115) hold, but it is satisfied with equality if and only if $Z=\varsigma \left(Z\right)V$.

     □

• 39.
  The entropy power of a random variable Z is the variance of a Gaussian random variable whose differential entropy is $h\left(Z\right)$, i.e.,

  $$\begin{array}{c}\hfill N\left(Z\right)=\frac{1}{2\pi \mathrm{e}}exp\left(2h\left(Z\right)\right).\end{array}$$ (116)
  While the power of a Cauchy random variable is infinite, its entropy power is given by

  $$\begin{array}{c}\hfill N(\lambda V+\mu )=\frac{1}{2\pi \mathrm{e}}exp\left(2h(\lambda V+\mu )\right)=\frac{8\pi {\lambda }^2}{\mathrm{e}}.\end{array}$$ (117)
  In the same spirit as the definition of entropy power, Theorem 3 suggests the definition of $N_{\mathtt{C}}\left(Z\right)$, the entropy strength of Z, as the strength of a centered Cauchy random variable whose differential entropy is $h\left(Z\right)$, i.e., $h\left(Z\right)=h\left(N_{\mathtt{C}}\left(Z\right)V\right)$. Therefore,

  $$\begin{array}{cc}\hfill N_{\mathtt{C}}\left(Z\right)& =\frac{1}{4\pi }exp\left(h\left(Z\right)\right)\hfill \end{array}$$ (118)

  $$\begin{array}{cc}\hfill & =\varsigma \left(Z\right)exp\left(-D\left(Z\parallel \varsigma \left(Z\right)V\right)\right)\hfill \end{array}$$ (119)

  $$\begin{array}{cc}\hfill & \le \varsigma \left(Z\right),\hfill \end{array}$$ (120)

  where (119) follows from (56), and (120) holds with equality if and only if Z is centered Cauchy. Note that, for all $(\alpha ,\mu )\in {\mathbb{R}}^2$,

  $$\begin{array}{c}\hfill N_{\mathtt{C}}(\alpha Z+\mu )=\left|\alpha \right|N_{\mathtt{C}}\left(Z\right).\end{array}$$ (121)
  Comparing (116) and (118), we see that entropy power is simply a scaled version of the entropy strength squared,

  $$\begin{array}{c}\hfill N\left(Z\right)=\frac{8\pi }{\mathrm{e}}{N}_{\mathtt{C}}^2\left(Z\right).\end{array}$$ (122)
  The entropy power inequality (e.g., [19,20]) states that, if $X_1$ and $X_2$ are independent real-valued random variables, then

  $$\begin{array}{c}\hfill N(X_1+X_2)\ge N\left(X_1\right)+N\left(X_2\right),\end{array}$$ (123)

  regardless of whether they have moments. According to (122), we may rewrite the entropy power inequality (123) replacing each entropy power by the corresponding squared entropy strength. Therefore, the squared entropy strength of the sum of independent random variables satisfies

  $$\begin{array}{c}\hfill N_{\mathtt{C}}^2(X_1+X_2)\ge N_{\mathtt{C}}^2\left(X_1\right)+N_{\mathtt{C}}^2\left(X_2\right).\end{array}$$ (124)

  It is well-known that equality holds in (123), and hence (124), if and only if both random variables are Gaussian. Indeed, if $X_1$ and $X_2$ are centered Cauchy with respective strengths ${\varsigma }_1>0$ and ${\varsigma }_2>0$, then (124) becomes ${\left({\varsigma }_1+{\varsigma }_2\right)}^2>{\varsigma }_1^2+{\varsigma }_2^2$.

• 40.
  Theorem 3 implies that any random variable with infinite differential entropy has infinite strength. There are indeed random variables with finite differential entropy and infinite strength. For example, let $Z\in [2,\infty )$ be an absolutely continuous random variable with probability density function

  $$\begin{array}{c}\hfill f_Z\left(t\right)=\left\{\begin{array}{cc}0.473991...{log}_{\mathrm{e}}^{-2}n,\hfill & t\in \left[n,n+\frac{1}{n}\right],n\in \{2,3,\dots \};\hfill \\ 0,\hfill & \mathrm{elsewhere}.\hfill \end{array}\right.\end{array}$$ (125)

  Then, $h\left(Z\right)=1.99258...$ nats, while the entropy of the quantized version as well as the strength satisfy $H(\lfloor Z\rfloor )=\infty =\varsigma \left(Z\right)$.

• 41.
  With the same approach, we may generalize Theorem 3 to encompass the full slew of the generalized Cauchy distributions in (9). To that end, fix $\rho >0$ and define the $(\rho ,\theta )$-strength of a random variable as

  $$\begin{array}{c}\hfill {\varsigma }_{\rho ,\theta }\left(Z\right)=inf\left\{\varsigma >0:\mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|\frac{Z}{\varsigma }\right|}^{\rho }\right)\right]\le \theta \right\}.\end{array}$$ (126)
  Therefore, ${\varsigma }_{\rho ,\theta }\left(Z\right)=\varsigma \left(Z\right)$ for $(\rho ,\theta )=(2,{log}_{\mathrm{e}}4)$, and if $(\beta ,\rho ,\theta )$ satisfy (110), then ${\varsigma }_{\rho ,\theta }\left(V_{\beta ,\rho }\right)=1$. As in Item 25, if ${\varsigma }_{\rho ,\theta }\left(Z\right)\in (0,\infty )$, we have

  $$\begin{array}{c}\hfill {\varsigma }_{\rho ,\theta }^{\rho }\left(Z\right)=\frac{1}{{\mathcal{V}}_{{\left|Z\right|}^{\rho }}^{-1}\left(\theta \right)}.\end{array}$$ (127)
• 42.

  Theorem 4.

  Generalized strength constraint. Fix $\rho >0$ and $\theta >0$. The differential entropy of a real-valued random variable with $(\rho ,\theta )$-strength ${\varsigma }_{\rho ,\theta }\left(Z\right)$ is upper bounded by

  $$\begin{array}{c}\hfill h\left(Z\right)\le log\left({\varsigma }_{\rho ,\theta }\left(Z\right)\right)+h\left(V_{\beta ,\rho }\right),\end{array}$$ (128)

  where β is given by the solution to (110), $V_{\beta ,\rho }$ has the generalized Cauchy density (9), and $h\left(V_{\beta ,\rho }\right)$ is given in (21). If ${\varsigma }_{\rho ,\theta }\left(Z\right)<\infty$, equality holds if and only if Z is a constant times $V_{\beta ,\rho }$.

  Proof. 

  As with Theorem 3, in the proof, we may assume $0<{\varsigma }_{\rho ,\theta }\left(Z\right)<\infty$ to avoid trivialities. Then,

  $$\begin{array}{c}\hfill \mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|\frac{Z}{{\varsigma }_{\rho ,\theta }\left(Z\right)}\right|}^{\rho }\right)\right]=\theta ,\end{array}$$ (129)

  and, in nats,

  $$\begin{array}{cc}\hfill D\left(\frac{Z}{{\varsigma }_{\rho ,\theta }\left(Z\right)}\parallel V_{\beta ,\rho }\right)& =-h\left(Z\right)-{log}_{\mathrm{e}}\frac{{\kappa }_{\beta ,\rho }}{{\varsigma }_{\rho ,\theta }\left(Z\right)}+\beta \mathbb{E}\left[{log}_{\mathrm{e}}\left(1+{\left|\frac{Z}{{\varsigma }_{\rho ,\theta }\left(Z\right)}\right|}^{\rho }\right)\right]\hfill \end{array}$$ (130)

  $$\begin{array}{cc}\hfill & =-h\left(Z\right)-{log}_{\mathrm{e}}\frac{{\kappa }_{\beta ,\rho }}{{\varsigma }_{\rho ,\theta }\left(Z\right)}+\beta \theta \hfill \end{array}$$ (131)

  $$\begin{array}{cc}\hfill & =-h\left(Z\right)-{log}_{\mathrm{e}}\frac{{\kappa }_{\beta ,\rho }}{{\varsigma }_{\rho ,\theta }\left(Z\right)}+\beta \psi \left(\beta \right)-\beta \psi \left(\beta -\frac{1}{\rho }\right)\hfill \end{array}$$ (132)

  $$\begin{array}{cc}\hfill & =-h\left(Z\right)+{log}_{\mathrm{e}}\left({\varsigma }_{\rho ,\theta }\left(Z\right)\right)+h\left(V_{\beta ,\rho }\right),\hfill \end{array}$$ (133)

  where (130), (131), (132), and (133) follow from (9), (129), (110), and (22), respectively.    □

• 43.In the multivariate case, we may find a simple upper bound on differential entropy based on the strength of the norm of the random vector.

  Theorem 5.

  The differential entropy of a random vector $Z^n$ is upper bounded by

  $$\begin{array}{c}\hfill h\left(Z^n\right)\le nlog\left(\varsigma (\parallel Z^n\parallel )\right)+\frac{n+1}{2}log\left(4\pi \right)-log\mathrm{\Gamma }\left(\frac{n+1}{2}\right).\end{array}$$ (134)

  Proof. 

  As in the proof of Theorem 3, we may assume that $0<\varsigma (\parallel Z^n\parallel )<\infty$. As usual, $V^n$ denotes the standard spherical multivariate Cauchy density in (6). Since for $\alpha \ne 0$, $f_{\alpha V^n}\left(x^n\right)={\left|\alpha \right|}^{-n}{f}_{V^n}\left({\alpha }^{-1}{x}^n\right)$, we have

  $$\begin{array}{cc}\hfill & D(Z^n\parallel \varsigma (\parallel Z^n\parallel )V^n)=-h\left(Z^n\right)-\mathbb{E}\left[log{f}_{\varsigma (\parallel Z^n\parallel )V^n}\left(Z^n\right)\right]\hfill \end{array}$$ (135)

  $$\begin{array}{cc}\hfill & =-h\left(Z^n\right)+nlog\left(\varsigma (\parallel Z^n\parallel )\right)-log\frac{\mathrm{\Gamma }\left(\frac{n+1}{2}\right)}{{\pi }^{\frac{n+1}{2}}}+\frac{n+1}{2}\mathbb{E}\left[log\left(1+\frac{\parallel Z^n{\parallel }^2}{{\varsigma }^2(\parallel Z^n\parallel )}\right)\right]\hfill \end{array}$$ (136)

  $$\begin{array}{cc}\hfill & =-h\left(Z^n\right)+nlog\left(\varsigma (\parallel Z^n\parallel )\right)-log\mathrm{\Gamma }\left(\frac{n+1}{2}\right)+\frac{n+1}{2}log\left(4\pi \right),\hfill \end{array}$$ (137)

  where (136) and (137) follow from (6) and the definition of strength, respectively.    □

  For $n=1$, Theorem 5 becomes the bound in (115). For $n=2,3,\dots$, the right side of (15) is greater than ${log}_{\mathrm{e}}4$, and, therefore, $\varsigma (\parallel Z^n\parallel )>1$. Consequently, in the multivariate case, there is no $Z^n$ such that (134) is tight.

• 44.
  To obtain a full generalization of Theorem 3 in the multivariate case, it is advisable to define the strength of a random n-vector as

  $$\begin{array}{cc}\hfill \varsigma \left(Z^n\right)& =inf\left\{\varsigma >0:-\mathbb{E}\left[log{f}_{V^n}\left({\varsigma }^{-1}{Z}^n\right)\right]\le h\left(V^n\right)\right\}\hfill \end{array}$$ (138)

  $$\begin{array}{cc}\hfill & ={\varsigma }_{2,{\theta }_n}(\parallel Z^n\parallel )\hfill \end{array}$$ (139)

  for ${\theta }_n=\psi \left(\frac{n+1}{2}\right)+\gamma +{log}_{\mathrm{e}}4$. To verify (139), note (15)–(17). Notice that $\varsigma \left(\lambda V^n\right)=\left|\lambda \right|$ and for $n=1$, (138) is equal to (34). The following result provides a maximal differential entropy characterization of the standard spherical multivariate Cauchy density.

  Theorem 6.

  Let $V^n$ have the standard multivariate Cauchy density (6), Then,

  $$\begin{array}{c}\hfill h\left(Z^n\right)\le nlog\varsigma \left(Z^n\right)+h\left(V^n\right),\end{array}$$ (140)

  where $h\left(V^n\right)$ is given in (17). If $0<\varsigma \left(Z^n\right)<\infty$, equality holds in (140) if and only if $Z^n=\lambda V^n$ for some $\lambda \ne 0$.

  Proof. 

  Assume $0<\varsigma \left(Z^n\right)<\infty$. Then,

  $$\begin{array}{cc}\hfill D\left(\frac{Z^n}{\varsigma \left(Z^n\right)}\parallel V^n\right)& =-h\left(Z^n\right)+nlog\varsigma \left(Z^n\right)-\mathbb{E}\left[log{f}_{V^n}\left({\varsigma }^{-1}\left(Z^n\right)Z^n\right)\right]\hfill \end{array}$$ (141)

  $$\begin{array}{cc}\hfill & =-h\left(Z^n\right)+nlog\varsigma \left(Z^n\right)+h\left(V^n\right)\hfill \end{array}$$ (142)

  in view of (138). Hence, the difference between right and left sides of (140) is equal to zero if and only if $Z^n=\lambda V^n$ for some $\lambda \ne 0$; otherwise, it is positive.    □

7. Relative Information

• 45.
  For probability measures P and Q on the same measurable space $(\mathcal{A},\mathcal{F})$, such that $P\ll Q$, the logarithm of their Radon–Nikodym derivative is the relative information denoted by

  $$\begin{array}{c}\hfill {ı}_{P\parallel Q}\left(x\right)=log\frac{\mathrm{d}P}{\mathrm{d}Q}\left(x\right).\end{array}$$ (143)
• 46.
  As usual, we may employ the notation ${ı}_{X\parallel Y}\left(x\right)$ to denote ${ı}_{P_X\parallel P_Y}\left(x\right)$. The distributions of the random variables ${ı}_{X\parallel Y}\left(X\right)$ and ${ı}_{X\parallel Y}\left(Y\right)$ are referred to as relative information spectra (e.g., [21]). It can be shown that there is a one-to-one correspondence between the cumulative distributions of ${ı}_{X\parallel Y}\left(X\right)$ and ${ı}_{X\parallel Y}\left(Y\right)$. For example, if they are absolutely continuous random variables with respective probability density functions $f_{X\parallel Y}$ and ${\overline{f}}_{X\parallel Y}$, then

  $$\begin{array}{c}\hfill f_{X\parallel Y}\left(\alpha \right)=exp\left(\alpha \right){\overline{f}}_{X\parallel Y}\left(\alpha \right),\alpha \in \mathbb{R}.\end{array}$$ (144)
  Obviously, the distributions of ${ı}_{X\parallel Y}\left(X\right)$ and $\frac{\mathrm{d}{P}_X}{\mathrm{d}{P}_Y}\left(X\right)$ determine each other. One caveat is that relative information may take the value $-\infty$. It can be shown that

  $$\begin{array}{cc}\hfill \mathbb{P}[{ı}_{X\parallel Y}\left(X\right)=-\infty ]& =0,\hfill \end{array}$$ (145)

  $$\begin{array}{cc}\hfill \mathbb{P}[{ı}_{X\parallel Y}\left(Y\right)=-\infty ]& =1-\mathbb{E}\left[exp(-{ı}_{X\parallel Y}\left(X\right))\right].\hfill \end{array}$$ (146)
• 47.The information spectra determine all measures of the distance between the respective probability measures of interest (e.g., [22,23]), including f-divergences and Rényi divergences. For example, the relative entropy (or Kullback–Leibler divergence) of the dominated measure P with respect to the reference measure Q is the average of the relative information when the argument is distributed according to P, i.e., $D(X\parallel Y)=\mathbb{E}\left[{ı}_{X\parallel Y}\left(X\right)\right]$. If $P\neg \ll Q$, then $D(P\parallel Q)=\infty$.
• 48.
  The information spectra also determine the fundamental trade-off in hypothesis testing. Let ${\alpha }_{\nu }(P_{\mathtt{1}},P_{\mathtt{0}})$ denote the minimal probability of deciding $P_{\mathtt{0}}$ when $P_{\mathtt{1}}$ is true subject to the constraint that the probability of deciding $P_{\mathtt{1}}$ when $P_{\mathtt{0}}$ is true is no larger than $\nu$. A consequence of the Neyman–Pearson lemma is

  $$\begin{array}{c}\hfill {\alpha }_{\nu }(P_{\mathtt{1}},P_{\mathtt{0}})=\underset{\gamma \in \mathbb{R}}{min}\left\{\mathbb{P}\left[{ı}_{P_{\mathtt{1}}\parallel P_{\mathtt{0}}}\left(Y_{\mathtt{1}}\right)\le \gamma \right]-exp\left(\gamma \right)\left(\nu -\mathbb{P}\left[{ı}_{P_{\mathtt{1}}\parallel P_{\mathtt{0}}}\left(Y_{\mathtt{0}}\right)>\gamma \right]\right)\right\},\end{array}$$ (147)

  where $Y_{\mathtt{0}}\sim P_{\mathtt{0}}$ and $Y_{\mathtt{1}}\sim P_{\mathtt{1}}$.
• 49.
  Cauchy distributions are absolutely continuous with respect to each other and, in view of (2),

  $$\begin{array}{c}\hfill {ı}_{{\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0}\left(x\right)=log\frac{|{\lambda }_1|}{|{\lambda }_0|}+log\frac{{(x-{\mu }_0)}^2+{\lambda }_0^2}{{(x-{\mu }_1)}^2+{\lambda }_1^2}.\end{array}$$ (148)
• 50.
  The following result, proved in Item 58, shows that the relative information spectrum corresponding to Cauchy distributions with respective scale/locations $({\lambda }_1,{\mu }_1)$ and $({\lambda }_0,{\mu }_0)$ depends on the four parameters through the single scalar

  $$\begin{array}{c}\hfill \zeta ({\lambda }_1,{\mu }_1,{\lambda }_0,{\mu }_0)=\frac{{\lambda }_1^2+{\lambda }_0^2+{({\mu }_1-{\mu }_0)}^2}{2|{\lambda }_0{\lambda }_1|}\ge 1,\end{array}$$ (149)

  where equality holds if and only if $({\lambda }_1,{\mu }_1)=({\lambda }_0,{\mu }_0)$.

  Theorem 7.

  Suppose that ${\lambda }_1{\lambda }_0\ne 0$, and V is standard Cauchy. Denote

  $$\begin{array}{c}\hfill Z=\frac{\mathrm{d}{P}_{{\lambda }_{1}V+{\mu }_1}}{\mathrm{d}{P}_{{\lambda }_{0}V+{\mu }_0}}({\lambda }_{1}V+{\mu }_1).\end{array}$$ (150)

  Then,

  • (a) 
    $$\begin{array}{c}\hfill \mathbb{E}\left[Z\right]=\zeta ({\lambda }_1,{\mu }_1,{\lambda }_0,{\mu }_0),\end{array}$$ (151)
  • (b) 
    Z has the same distribution as the random variable

    $$\begin{array}{c}\hfill \zeta +\sqrt{{\zeta }^2-1}cos\mathrm{\Theta },\end{array}$$ (152)
    where Θ is uniformly distributed on $[-\pi ,\pi ]$ and $\zeta =\zeta ({\lambda }_1,{\mu }_1,{\lambda }_0,{\mu }_0)$. Therefore, the probability density function of Z is

    $$\begin{array}{c}\hfill f_Z\left(z\right)=\frac{1}{\pi }\frac{1}{\sqrt{{\zeta }^2-1-{(z-\zeta )}^2}},\end{array}$$ (153)
    on the interval $0<\zeta -\sqrt{{\zeta }^2-1}<z<\zeta +\sqrt{{\zeta }^2-1}$.

• 51.
  The indefinite integral (e.g., see 2.261 in [24])

  $$\begin{array}{c}\hfill \int \frac{\mathrm{d}x}{\sqrt{2\zeta x-x^2-1}}=arcsin\frac{x-\zeta }{\sqrt{{\zeta }^2-1}}\end{array}$$ (154)

  results, with $X_i={\lambda }_{i}V+{\mu }_i$, $i=0,1$, in

  $$\begin{array}{c}\hfill \mathbb{P}[{ı}_{X_1\parallel X_0}\left(X_1\right)\le logt]=\left\{\begin{array}{cc}1,\hfill & \zeta +\sqrt{{\zeta }^2-1}\le t;\hfill \\ \frac{1}{2}+\frac{1}{\pi }arcsin\frac{t-\zeta }{\sqrt{{\zeta }^2-1}},\hfill & \zeta -\sqrt{{\zeta }^2-1}<t<\zeta +\sqrt{{\zeta }^2-1};\hfill \\ 0,\hfill & 0<t\le \zeta -\sqrt{{\zeta }^2-1}.\hfill \end{array}\right.\end{array}$$ (155)
• 52.
  For future use, note that the endpoints of the support of (153) are their respective reciprocals. Furthermore,

  $$\begin{array}{c}\hfill f_Z\left(\frac{1}{z}\right)=z{f}_Z\left(z\right),\end{array}$$ (156)

  which implies

  $$\begin{array}{c}\hfill f_{\frac{1}{Z}}\left(z\right)=\frac{1}{z}{f}_Z\left(z\right).\end{array}$$ (157)

8. Equivalent Pairs of Probability Measures

• 53.Suppose that $P_1$ and $Q_1$ are probability measures on $({\mathcal{A}}_1,{\mathcal{F}}_1)$ such that $P_1\ll Q_1$ and $P_2$ and $Q_2$ are probability measures on $({\mathcal{A}}_2,{\mathcal{F}}_2)$ such that $P_2\ll Q_2$. We say that $(P_1,Q_1)$ and $(P_2,Q_2)$ are equivalent pairs, and write $(P_1,Q_1)\equiv (P_2,Q_2)$, if the cumulative distribution functions of ${ı}_{P_1\parallel Q_1}\left(X_1\right)$ and ${ı}_{P_2\parallel Q_2}\left(X_2\right)$ are identical with $X_1\sim P_1$ and $X_2\sim P_2$. Naturally, ≡ is an equivalence relationship. Because of the one-to-one correspondence indicated in Item 46, the definition of equivalent pairs does not change if we require equality of the information spectra under the dominated measure, i.e., that ${ı}_{P_1\parallel Q_1}\left(Y_1\right)$ and ${ı}_{P_2\parallel Q_2}\left(Y_2\right)$ be equally distributed $Y_1\sim Q_1$ and $Y_2\sim Q_2$. Obviously, the requirement that the information spectra coincide is the same as requiring that the distributions of $\frac{\mathrm{d}{P}_1}{\mathrm{d}{Q}_1}\left(Y_1\right)$ and $\frac{\mathrm{d}{P}_2}{\mathrm{d}{Q}_2}\left(Y_2\right)$ are equal. As in Item 46, we also employ the notation $(X_1,Y_1)\equiv (X_2,Y_2)$ to indicate $(P_1,Q_1)\equiv (P_2,Q_2)$ if $X_1\sim P_1$, $X_2\sim P_2$, $Y_1\sim Q_1$, and $Y_2\sim Q_2$.
• 54.Suppose that the output probability measures of a certain (random or deterministic) transformation are $Q_{\mathtt{0}}$ and $Q_{\mathtt{1}}$ when the input is distributed according to $P_{\mathtt{0}}$ and $P_{\mathtt{1}}$, respectively. If $(P_{\mathtt{0}},P_{\mathtt{1}})\equiv (Q_{\mathtt{0}},Q_{\mathtt{1}})$, then the transformation is a sufficient statistic for deciding between $P_{\mathtt{0}}$ and $P_{\mathtt{1}}$ (i.e., the case of a binary parameter).
• 55.
  If $(\mathcal{A},\mathcal{F})$ is a measurable space on which the probability measures $P_{X_1}\ll P_{X_2}$ are defined, and $\varphi :\mathcal{A}\to \mathcal{B}$ is a $(\mathcal{F},\mathcal{G})$-measurable injective function, then $P_{\varphi \left(X_1\right)}\ll P_{\varphi \left(X_2\right)}$ are probability measures on $(\mathcal{B},\mathcal{G})$ and

  $$\begin{array}{c}\hfill {ı}_{X_1\parallel X_2}\left(x\right)={ı}_{\varphi \left(X_1\right)\parallel \varphi \left(X_2\right)}\left(\varphi \left(x\right)\right).\end{array}$$ (158)

  Consequently, $(X_1,X_2)\equiv (\varphi \left(X_1\right),\varphi \left(X_2\right))$.

• 56.
  The most important special case of Item 55 is an affine transformation of an arbitrary real-valued random variable X, which enables the reduction of four-parameter problems into two-parameter problems: for all $({\lambda }_2,{\mu }_1,{\mu }_2)\in {\mathbb{R}}^3$ and ${\lambda }_1\ne 0$,

  $$\begin{array}{c}\hfill ({\lambda }_{1}X+{\mu }_1,{\lambda }_{2}X+{\mu }_2)\equiv (X,\lambda X+\mu ),\end{array}$$ (159)

  with

  $$\begin{array}{c}\hfill \lambda =\frac{{\lambda }_2}{{\lambda }_1}and\mu =\frac{{\mu }_2-{\mu }_1}{{\lambda }_1},\end{array}$$ (160)

  by choosing the affine function $\varphi \left(x\right)=\frac{x-{\mu }_1}{{\lambda }_1}$.
• 57.

  Theorem 8.

  If $X^n\in {\mathbb{R}}^n$ is an even random vector, i.e., $P_{X^n}=P_{-X^n}$, then

  $$\begin{array}{c}\hfill (X^n+{\mathbf{\mu }}_1,X^n+{\mathbf{\mu }}_2)\equiv (X^n+{\mathbf{\mu }}_3,X^n+{\mathbf{\mu }}_4),\end{array}$$ (161)

  whenever $|{\mathbf{\mu }}_1-{\mathbf{\mu }}_2|=|{\mathbf{\mu }}_3-{\mathbf{\mu }}_4|$.

  Proof. 

  • (a)
    If ${\mathbf{\mu }}_1-{\mathbf{\mu }}_2={\mathbf{\mu }}_3-{\mathbf{\mu }}_4$, then (161) holds even if $X^n$ is not even because the function $\mathbf{x}-\mathbf{\mu }$ is injective, in particular, with $\mathbf{\mu }={\mathbf{\mu }}_3-{\mathbf{\mu }}_1={\mathbf{\mu }}_4-{\mathbf{\mu }}_2$.
  • (b)
    If ${\mathbf{\mu }}_1-{\mathbf{\mu }}_2={\mathbf{\mu }}_4-{\mathbf{\mu }}_3$, then

    $$\begin{array}{cc}\hfill (X^n+{\mathbf{\mu }}_1,X^n+{\mathbf{\mu }}_2)& \equiv (X^n,X^n+{\mathbf{\mu }}_2-{\mathbf{\mu }}_1)\hfill \end{array}$$ (162)

    $$\begin{array}{cc}\hfill & \equiv (X^n,X^n+{\mathbf{\mu }}_3-{\mathbf{\mu }}_4)\hfill \end{array}$$ (163)

    $$\begin{array}{cc}\hfill & \equiv (-X^n+{\mathbf{\mu }}_3-{\mathbf{\mu }}_4,-X^n)\hfill \end{array}$$ (164)

    $$\begin{array}{cc}\hfill & \equiv (X^n+{\mathbf{\mu }}_3-{\mathbf{\mu }}_4,X^n)\hfill \end{array}$$ (165)

    $$\begin{array}{cc}\hfill & \equiv (X^n+{\mathbf{\mu }}_3,X^n+{\mathbf{\mu }}_4),\hfill \end{array}$$ (166)

    where (162) and (166) follow from Part (a), (164) follows because $-\mathbf{x}+{\mathbf{\mu }}_3-{\mathbf{\mu }}_4$ is injective, and (165) holds because $X^n$ is even.
       □

• 58.We now proceed to prove Theorem 7.

  Proof. 

  Since $\lambda V$ and $-\lambda V$ have identical distributions, we may assume for convenience that ${\lambda }_1>0$ and ${\lambda }_0>0$. Furthermore, capitalizing on Item 56, we may assume ${\lambda }_1=1$, ${\mu }_1=0$, ${\lambda }_0=\lambda$, and ${\mu }_0=\mu$, and then recover the general result letting $\lambda =\frac{{\lambda }_0}{{\lambda }_1}$ and $\mu =\frac{{\mu }_0-{\mu }_1}{{\lambda }_1}$. Invoking (A9) and (A10), we have

  $$\begin{array}{cc}\hfill \mathbb{E}\left[\frac{\mathrm{d}{P}_V}{\mathrm{d}{P}_{\lambda V+\mu }}\left(V\right)\right]& =\frac{1}{\lambda }\mathbb{E}\left[\frac{{(V-\mu )}^2+{\lambda }^2}{V^2+1}\right]\hfill \end{array}$$ (167)

  $$\begin{array}{cc}\hfill & =\frac{1}{\pi \lambda }{\int }_{-\infty }^{\infty }\frac{{(t-\mu )}^2+{\lambda }^2}{{(t^2+1)}^2}\mathrm{d}t\hfill \end{array}$$ (168)

  $$\begin{array}{cc}\hfill & =\frac{{\lambda }^2+{\mu }^2+1}{2\lambda },\hfill \end{array}$$ (169)

  and we can verify that we recover (151) through the aforementioned substitution. Once we have obtained the expectation of $Z=\frac{\mathrm{d}{P}_V}{\mathrm{d}{P}_{\lambda V+\mu }}\left(V\right)$, we proceed to determine its distribution. Denoting the right side of (169) by $\zeta$, we have

  $$\begin{array}{cc}\hfill Z-\mathbb{E}\left[Z\right]& =\frac{1}{\lambda }\frac{{\lambda }^2+{(V-\mu )}^2}{1+V^2}-\zeta \hfill \end{array}$$ (170)

  $$\begin{array}{cc}\hfill & =\frac{1}{2\lambda }\frac{(1-{\lambda }^2-{\mu }^2)(V^2-1)-4\mu V}{1+V^2}\hfill \end{array}$$ (171)

  $$\begin{array}{cc}\hfill & =\frac{1}{2\lambda }(1-{\lambda }^2-{\mu }^2)({sin}^2\mathrm{\Theta }-{cos}^2\mathrm{\Theta })-4\mu sin\mathrm{\Theta }cos\mathrm{\Theta }\hfill \end{array}$$ (172)

  $$\begin{array}{cc}\hfill & =\frac{1}{2\lambda }\left(({\lambda }^2+{\mu }^2-1)cos2\mathrm{\Theta }-2\mu sin2\mathrm{\Theta }\right)\hfill \end{array}$$ (173)

  $$\begin{array}{cc}\hfill & =\frac{1}{2\lambda }\sqrt{{({\lambda }^2+{\mu }^2-1)}^2+4{\mu }^2}cos\left(2\mathrm{\Theta }+{\varphi }_{\lambda ,\mu }\right)\hfill \end{array}$$ (174)

  $$\begin{array}{cc}\hfill & =\sqrt{{\zeta }^2-1}cos\left(2\mathrm{\Theta }+{\varphi }_{\lambda ,\mu }\right),\hfill \end{array}$$ (175)

  where $\mathrm{\Theta }$ is uniformly distributed on $[-\pi ,\pi ]$. We have substituted $V=tan\mathrm{\Theta }$ (see Item 4) in (172), and invoked elementary trigonometric identities in (173) and (174). Since the phase in (175) does not affect it, the distribution of Z is indeed as claimed in (152), and (153) follows because the probability density function of $cos\mathrm{\Theta }$ is

  $$\begin{array}{c}\hfill f_{cos\mathrm{\Theta }}\left(t\right)=\frac{1}{\pi }\frac{1}{\sqrt{1-t^2}},\left|t\right|<1.\end{array}$$ (176)

     □

• 59.
  In general, it need not hold that $(X,Y)\equiv (Y,X)$—for example, if X and Y are zero-mean Gaussian with different variances. However, the class of scalar Cauchy distributions does satisfy this property since the result of Theorem 7 is invariant to swapping ${\lambda }_1\leftrightarrow {\lambda }_0$ and ${\mu }_1\leftrightarrow {\mu }_0$. More generally, Theorem 7 implies that, if ${\lambda }_1{\lambda }_0{\gamma }_1{\gamma }_0\ne 0$, then

  $$\begin{array}{cc}\hfill ({\lambda }_{1}V+{\mu }_1,{\lambda }_{0}V+{\mu }_0)& \equiv ({\gamma }_{1}V+{\nu }_1,{\gamma }_{0}V+{\nu }_0)\hfill \\ \hfill & \Updownarrow \hfill \\ \hfill \frac{{\lambda }_1^2+{\lambda }_0^2+{({\mu }_1-{\mu }_0)}^2}{|{\lambda }_0{\lambda }_1|}& =\frac{{\gamma }_1^2+{\gamma }_0^2+{({\nu }_1-{\nu }_0)}^2}{|{\gamma }_0{\gamma }_1|}.\hfill \end{array}$$ (177)

  Curiously, (177) implies that $(V,V+1)\equiv (V,2V+1)$.

• 60.For location–dilation families of random variables, we saw in Item 56 how to reduce a four-parameter problem into a two-parameter problem since $({\lambda }_{1}V+{\mu }_1,{\lambda }_{0}V+{\mu }_0)\equiv (V,\lambda V+\mu )$ with the appropriate substitution. In the Cauchy case, Theorem 7 reveals that, in fact, we can go one step further and turn it into a one-parameter problem. We have two basic ways of doing this:
  • (a)$({\lambda }_{1}V+{\mu }_1,{\lambda }_{0}V+{\mu }_0)\equiv (V,V+\mu )$ with ${\mu }^2=2\zeta -2$.
  • (b)
    $({\lambda }_{1}V+{\mu }_1,{\lambda }_{0}V+{\mu }_0)\equiv (V,\lambda V)$ with either

    $$\begin{array}{c}\hfill \lambda =\zeta -\sqrt{{\zeta }^2-1}<1,or\lambda =\zeta +\sqrt{{\zeta }^2-1}>1,\end{array}$$ (178)

    which are the solutions to $\zeta =\frac{{\lambda }^2+1}{2\lambda }$.

9. $\mathit{f}$-Divergences

This section studies the interplay of f-divergences and equivalent pairs of measures.

• 61.
  If $P\ll Q$ and $f:[0,\infty )\to \mathbb{R}$ is convex and right-continuous at 0, f-divergence is defined as

  $$\begin{array}{c}\hfill D_f(P\parallel Q)=\mathbb{E}\left[f\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\left(Y\right)\right)\right],Y\sim Q.\end{array}$$ (179)
• 62.
  The most important property of f-divergence is the data processing inequality

  $$\begin{array}{c}\hfill D_f(P_X\parallel Q_X)\ge D_f(P_Y\parallel Q_Y),\end{array}$$ (180)

  where $P_Y$ and $Q_Y$ are the responses of a (random or deterministic) transformation to $P_X$ and $Q_X$, respectively. If f is strictly convex at 1 and $D_f(P_X\parallel Q_X)<\infty$, then $(P_X,Q_X)\equiv (P_Y,Q_Y)$ is necessary and sufficient for equality in (180).
• 63.If $(P,Q)\equiv (Q,P)$, then $D_f(P\parallel Q)=D_{f^{☆}}(P\parallel Q)$ with the transform $f^{☆}\left(t\right)=tf\left(\frac{1}{t}\right)$, which satisfies $f^{☆☆}=f$.
• 64.

  Theorem 9.

  If $P_1\ll Q_1$ and $P_2\ll Q_2$, then

  $$\begin{array}{c}\hfill (P_1,Q_1)\equiv (P_2,Q_2)⟺D_f(P_1\parallel Q_1)=D_f(P_2\parallel Q_2),\forall f,\end{array}$$ (181)

  where $\forall f$ stands for all convex right-continuous $f:[0,\infty )\to \mathbb{R}$.

  Proof. 

  As mentioned in Item 53, $(P_1,Q_1)\equiv (P_2,Q_2)$ is equivalent to $\frac{\mathrm{d}{P}_1}{\mathrm{d}{Q}_1}\left(Y_1\right)$ and $\frac{\mathrm{d}{P}_2}{\mathrm{d}{Q}_2}\left(Y_2\right)$ having identical distributions with $Y_1\sim Q_1$ and $Y_2\sim Q_2$.

  • ⟹
    According to (179), $D_f(P\parallel Q)$ is determined by the distribution of the random variable $\frac{\mathrm{d}P}{\mathrm{d}Q}\left(Y\right)$, $Y\sim Q$.
  • ⟸
    For $t\in \mathbb{R}$, the function $f_t\left(x\right)={\mathrm{e}}^{tx}$, $x\ge 0$, is convex and right-continuous at 0, and $D_{f_t}(P\parallel Q)$ is the moment generating function, evaluated at t, of the random variable $\frac{\mathrm{d}P}{\mathrm{d}Q}\left(Y\right)$, $Y\sim Q$. Therefore, $D_{f_t}(P_1\parallel Q_1)=D_{f_t}(P_2\parallel Q_2)$ for all t implies that $(P_1,Q_1)\equiv (P_2,Q_2)$.
       □

• 65.Since $P\ll Q$ is not necessary in order to define (finite) $D_f(P\parallel Q)$, it is possible to enlarge the scope of Theorem 9 by defining $(P_1,Q_1)\equiv (P_2,Q_2)$ dropping the restriction that $P_1\ll Q_1$ and $P_2\ll Q_2$. For that purpose, let ${\mu }_1$ and ${\mu }_2$ be $\sigma$-finite measures on $({\mathcal{A}}_1,{\mathcal{F}}_1)$ and $({\mathcal{A}}_2,{\mathcal{F}}_2)$, respectively, and denote $p_i=\frac{\mathrm{d}{P}_i}{\mathrm{d}{\mu }_i}$, $q_i=\frac{\mathrm{d}{Q}_i}{\mathrm{d}{\mu }_i}$, $i=1,2$. Then, we say $(P_1,Q_1)\equiv (P_2,Q_2)$ if
  • (a)when restricted to $[0,1]$, the random variables $\frac{p_1\left(Y_1\right)}{q_1\left(Y_1\right)}$ and $\frac{p_2\left(Y_2\right)}{q_2\left(Y_2\right)}$ have identical distributions with $Y_1\sim Q_1$ and $Y_2\sim Q_2$;
  • (b)when restricted to $[0,1]$, the random variables $\frac{q_1\left(X_1\right)}{p_1\left(X_1\right)}$ and $\frac{q_2\left(X_2\right)}{p_2\left(X_2\right)}$ have identical distributions with $X_1\sim P_1$ and $X_2\sim P_2$.

  Note that those conditions imply that

  • (c)$Q_1\left(\{\omega \in {\mathcal{A}}_1:p_1\left(\omega \right)=q_1\left(\omega \right)\}\right)=Q_2\left(\{\omega \in {\mathcal{A}}_2:p_2\left(\omega \right)=q_2\left(\omega \right)\}\right)$;
  • (d)$Q_1\left(\{\omega \in {\mathcal{A}}_1:p_1\left(\omega \right)=0\}\right)=Q_2\left(\{\omega \in {\mathcal{A}}_2:p_2\left(\omega \right)=0\}\right)$;
  • (e)$P_1\left(\{\omega \in {\mathcal{A}}_1:q_1\left(\omega \right)=0\}\right)=P_2\left(\{\omega \in {\mathcal{A}}_2:q_2\left(\omega \right)=0\}\right)$.
  For example, if $P_1\perp Q_1$ and $P_2\perp Q_2$, then $(P_1,Q_1)\equiv (P_2,Q_2)$. To show the generalized version of Theorem 9, it is convenient to use the symmetrized form

  $$\begin{array}{c}\hfill D_f(P\parallel Q)={\int }_{0\le p<q}qf\left(\frac{p}{q}\right)\mathrm{d}\mu +{\int }_{0\le q<p}p{f}^{☆}\left(\frac{q}{p}\right)\mathrm{d}\mu +f\left(1\right)Q[p=q].\end{array}$$ (182)
• 66.
  Suppose that there is a class $\mathcal{C}$ of probability measures on a given measurable space with the property that there exists a convex function $g:(0,\infty )\to \mathbb{R}$ (right-continuous at 0) such that, if $(P_1,Q_1)\in {\mathcal{C}}^2$ and $(P_2,Q_2)\in {\mathcal{C}}^2$, then

  $$\begin{array}{c}\hfill D_g(P_1\parallel Q_1)=D_g(P_2\parallel Q_2)⟺(P_1,Q_1)\equiv (P_2,Q_2).\end{array}$$ (183)

  In such case, Theorem 9 indicates that ${\mathcal{C}}^2$ can be partitioned into equivalence classes such that, within every equivalence class, the value of $D_f(P\parallel Q)$ is constant, though naturally dependent on f. Throughout ${\mathcal{C}}^2$, the value of $D_g(P\parallel Q)$ determines the value of $D_f(P\parallel Q)$, i.e., we can express $D_f(P\parallel Q)={\vartheta }_{f,g}\left(D_g(P\parallel Q)\right)$, where ${\vartheta }_{f,g}$ is a non-decreasing function. Consider the following examples:

  • (a)
    Let $\mathcal{C}$ be the class of real-valued Gaussian probability measures with given variance ${\sigma }^2>0$. Then,

    $$\begin{array}{c}\hfill D\left(\mathcal{N}\left({\mu }_1,{\sigma }^2\right)\parallel \mathcal{N}\left({\mu }_2,{\sigma }^2\right)\right)=\frac{{({\mu }_1-{\mu }_2)}^2}{{\sigma }^2}log\mathrm{e}.\end{array}$$ (184)

    Since Theorem 8 implies that $(\mathcal{N}\left({\mu }_1,{\sigma }^2\right),\mathcal{N}\left({\mu }_2,{\sigma }^2\right))\equiv (\mathcal{N}\left({\mu }_3,{\sigma }^2\right),\mathcal{N}\left({\mu }_4,{\sigma }^2\right))$ as long as ${({\mu }_1-{\mu }_2)}^2={({\mu }_3-{\mu }_4)}^2$, (184) indicates that (183) is satisfied with $g\left(t\right)$ given by the right-continuous extension of $tlogt$. Therefore, we can conclude that, regardless of f, $D_f\left(\mathcal{N}\left({\mu }_1,{\sigma }^2\right)\parallel \mathcal{N}\left({\mu }_2,{\sigma }^2\right)\right)$ depends on $({\mu }_1,{\mu }_2,{\sigma }^2)$ only through ${({\mu }_1-{\mu }_2)}^2/{\sigma }^2$.

  • (b)
    Let $\mathcal{C}$ be the collection of all Cauchy random variables. Theorem 7 reveals that (183) is also satisfied if $g\left(x\right)=x^2$ because, if $X\sim P$ and $Y\sim Q$, then

    $$\begin{array}{c}\hfill \mathbb{E}\left[\frac{\mathrm{d}P}{\mathrm{d}Q}\left(X\right)\right]=\mathbb{E}\left[{\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\left(Y\right)\right)}^2\right].\end{array}$$ (185)
• 67.
  An immediate consequence of Theorems 7 and 9 is that, for any valid f, the f-divergence between Cauchy densities is symmetric,

  $$\begin{array}{c}\hfill D_f({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)=D_f({\lambda }_{0}V+{\mu }_0\parallel {\lambda }_{1}V+{\mu }_1).\end{array}$$ (186)
  This property does not generalize to the multivariate case. While, in view of Theorem 8,

  $$\begin{array}{c}\hfill ({\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\mathbf{\mu }}_1,{\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\mathbf{\mu }}_2)\equiv ({\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\mathbf{\mu }}_2,{\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\mathbf{\mu }}_1),\end{array}$$ (187)

  in general, $({\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n,V^n)\not\equiv (V^n,{\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n)$ since the corresponding relative entropies do not coincide as shown in [8].
• 68.It follows from Item 66 and Theorem 7 that any f-divergence between Cauchy probability measures $D_f({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)$ is a monotonically increasing function of $\zeta ({\lambda }_1,{\mu }_1,{\lambda }_0,{\mu }_0)$ given by (149). The following result shows how to obtain that function from f.

  Theorem 10.

  With $f_Z$ given in (153),

  $$\begin{array}{cc}\hfill D_f({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)& ={\int }_{\zeta -\sqrt{{\zeta }^2-1}}^{\zeta +\sqrt{{\zeta }^2-1}}f\left(\frac{1}{z}\right)f_Z\left(z\right)\mathrm{d}z\hfill \end{array}$$ (188)

  $$\begin{array}{cc}\hfill & =\mathbb{E}\left[f\left({\left(\zeta +\sqrt{{\zeta }^2-1}cos\mathrm{\Theta }\right)}^{-1}\right)\right]\hfill \end{array}$$ (189)

  $$\begin{array}{cc}\hfill & ={\int }_{\zeta -\sqrt{{\zeta }^2-1}}^{\zeta +\sqrt{{\zeta }^2-1}}\frac{1}{z}f\left(z\right)f_Z\left(z\right)\mathrm{d}z.\hfill \end{array}$$ (190)

  where Θ is uniformly distributed on $[0,\pi ]$ in (189).

  Proof. 

  In view of (179) and the definition of Z in Theorem 7,

  $$\begin{array}{c}\hfill D_f({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)=\mathbb{E}\left[f\left(\frac{1}{Z}\right)\right],\end{array}$$ (191)

  thereby justifying (188) and (189) since we saw in Theorem 7 that Z has the distribution of $\zeta +\sqrt{{\zeta }^2-1}cos\mathrm{\Theta }$ with $\mathrm{\Theta }$ uniformly distributed on $[0,\pi ]$. Item 52 results in (190). Alternatively, we can rely on Item 63 and substitute f by $f^{☆}$ on the right side of (188).    □

• 69.
  Suppose now that we have two sequences of Cauchy measures with respective parameters $({\lambda }_1^{\left(n\right)},{\mu }_1^{\left(n\right)})$ and $({\lambda }_0^{\left(n\right)},{\mu }_0^{\left(n\right)})$ such that $\zeta ({\lambda }_1^{\left(n\right)},{\mu }_1^{\left(n\right)},{\lambda }_0^{\left(n\right)},{\mu }_0^{\left(n\right)})\to 1$. Then, Theorem 10 indicates that

  $$\begin{array}{c}\hfill D_f\left({\lambda }_1^{\left(n\right)}V+{\mu }_1^{\left(n\right)}\parallel {\lambda }_0^{\left(n\right)}V+{\mu }_0^{\left(n\right)}\right)\to f\left(1\right).\end{array}$$ (192)
  The most common f-divergences are such that $f\left(1\right)=0$ since in that case $D_f(P\parallel Q)\ge 0$. In addition, adding the function $\alpha t-\alpha$ to $f\left(t\right)$ does not change the value of $D_f(P\parallel Q)$ and with appropriately chosen $\alpha$, we can turn $f\left(t\right)$ into canonical form in which not only $f\left(1\right)=0$ but $f\left(t\right)\ge 0$. In the special case in which the second measure is fixed, Theorem 9 in [25] shows that, if $\mathrm{ess}sup\frac{\mathrm{d}{P}_n}{\mathrm{d}Q}\left(Y\right)\to 1$ with $Y\sim Q$, then

  $$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_f(P_n\parallel Q)}{D_g(P_n\parallel Q)}=\underset{t\to 1}{lim}\frac{f\left(t\right)}{g\left(t\right)},\end{array}$$ (193)

  provided the limit on the right side exists; otherwise, the left side lies between the left and right limits at 1. In the Cauchy case, we can allow the second probability to depend on n and sharpen that result by means of Theorem 10. In particular, it can be shown that

  $$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_f\left({\lambda }_1^{\left(n\right)}V+{\mu }_1^{\left(n\right)}\parallel {\lambda }_0^{\left(n\right)}V+{\mu }_0^{\left(n\right)}\right)}{D_g\left({\lambda }_1^{\left(n\right)}V+{\mu }_1^{\left(n\right)}\parallel {\lambda }_0^{\left(n\right)}V+{\mu }_0^{\left(n\right)}\right)}=\frac{\dot{f}\left(0^{-}\right)+\dot{f}\left(0^{+}\right)}{\dot{g}\left(0^{-}\right)+\dot{g}\left(0^{+}\right)}\end{array}$$ (194)

  provided the right side is not $\frac{0}{0}$.

10. ${\mathbf{\chi }}^{\mathbf{2}}$-Divergence

• 70.
  With either $f\left(x\right)={(x-1)}^2$ or $f\left(x\right)=x^2-1$, f-divergence is the ${\chi }^2$-divergence,

  $$\begin{array}{cc}\hfill {\chi }^2(P\parallel Q)& =\mathbb{E}\left[\frac{\mathrm{d}P}{\mathrm{d}Q}\left(X\right)\right]-1,X\sim P.\hfill \end{array}$$ (195)
• 71.
  If P and Q are Cauchy distributions, then (149), (151) and (195) result in

  $$\begin{array}{cc}\hfill {\chi }^2({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)& =\zeta ({\lambda }_1,{\mu }_1,{\lambda }_0,{\mu }_0)-1\hfill \end{array}$$ (196)

  $$\begin{array}{cc}\hfill & =\frac{\left(\right|{\lambda }_0|-|{\lambda }_1{\left|\right)}^2+{({\mu }_1-{\mu }_0)}^2}{2|{\lambda }_0{\lambda }_1|},\hfill \end{array}$$ (197)

  a formula obtained in Appendix D of [26] using complex analysis and the Cauchy integral formula. In addition, invoking complex analysis and the maximal group invariant results in [27,28], ref. [26] shows that any f-divergence between Cauchy distributions can be expressed as a function of their ${\chi }^2$ divergence, although [26] left open how to obtain that function, which is given by Theorem 10 substituting $\zeta =1+{\chi }^2$.

11. Relative Entropy

• 72.
  The relative entropy between Cauchy distributions is given by

  $$\begin{array}{c}\hfill D({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)=log\left(\frac{\left(\right|{\lambda }_0|+|{\lambda }_1{\left|\right)}^2+{({\mu }_1-{\mu }_0)}^2}{4|{\lambda }_0{\lambda }_1|}\right),\end{array}$$ (198)

  where ${\lambda }_1{\lambda }_0\ne 0$. The special case ${\lambda }_1={\lambda }_0$ of (198) was found in Example 4 of [29]. The next four items give different simple justifications for (198). An alternative proof was recently given in Appendix C of [26] using complex analysis holomorphisms and the Cauchy integral formula. Yet another, much more involved, proof is reported in [30]. See also Remark 19 in [26] for another route invoking the Lévy–Khintchine formula and the Frullani integral.
• 73.
  Since for absolutely continuous random variables $D(X\parallel Y)=-h\left(X\right)-\mathbb{E}[log{f}_Y\left(X\right)]$,

  $$\begin{array}{cc}\hfill D\left(V\parallel \lambda V+\mu \right)& =-h\left(V\right)+log\frac{\pi }{\left|\lambda \right|}+\mathbb{E}\left[log\left({\lambda }^2+{(V-\mu )}^2\right)\right]\hfill \end{array}$$ (199)

  $$\begin{array}{cc}\hfill & =-log\left(4\right|\lambda \left|\right)+log\left({(1+|\lambda \left|\right)}^2+{\mu }^2\right),\hfill \end{array}$$ (200)

  where (200) follows from (12) and (A4) with ${\alpha }^2={\lambda }^2+{\mu }^2$ and $cos\beta =\frac{\mu }{\left|\alpha \right|}.$

  Now, substituting $\lambda =\frac{{\lambda }_0}{{\lambda }_1}$ and $\mu =\frac{{\mu }_0-{\mu }_1}{{\lambda }_1}$, we obtain (198) since, according to Item 56, $(V,\lambda V+\mu )\equiv ({\lambda }_{1}V+{\mu }_1,{\lambda }_{0}V+{\mu }_0)$.

• 74.
  From the formula found in Example 4 of [29] and the fact that, according to (197), ${\chi }^2=\frac{{\mu }^2}{2{\lambda }^2}$ when ${\lambda }_1={\lambda }_0=\lambda$, we obtain

  $$\begin{array}{c}\hfill D(\lambda V+\mu \parallel \lambda V)=log\left(1+\frac{{\mu }^2}{4{\lambda }^2}\right)=log\left(1+\frac{1}{2}{\chi }^2\right).\end{array}$$ (201)

  Moreover, as argued in Item 60, (201) is also valid for the relative entropy between Cauchy distributions with ${\lambda }_1\ne {\lambda }_0$ as long as ${\chi }^2$ is given in (197). Indeed, we can verify that the right side of (201) becomes (198) with said substitution.

• 75.
  By the definition of relative entropy, and Theorem 7,

  $$\begin{array}{cc}\hfill D({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)& =\mathbb{E}\left[logZ\right]\hfill \end{array}$$ (202)

  $$\begin{array}{cc}\hfill & =\frac{1}{2\pi }{\int }_0^{2\pi }log\left(\zeta +\sqrt{{\zeta }^2-1}cos\theta \right)\mathrm{d}\theta \hfill \end{array}$$ (203)

  $$\begin{array}{cc}\hfill & =log\left(\frac{1+\zeta }{2}\right),\hfill \end{array}$$ (204)

  where (204) follows from (A14). Then, (198) results by plugging into (204) the value of $\zeta$ in (149).
• 76.Evaluating (190) with $f\left(t\right)=tlogt$ results in (202).
• 77.
  If V is standard Cauchy, independent of Cauchy $V_1$ and $V_0$, then (198) results in

  $$\begin{array}{c}\hfill D(\lambda V+ϵV_1\parallel \lambda V+ϵV_0)=\frac{{ϵ}^2}{4{\lambda }^2}\left({({\lambda }_1-{\lambda }_0)}^2+{({\mu }_1-{\mu }_0)}^2\right)log\mathrm{e}+o\left({ϵ}^2\right),\end{array}$$ (205)

  where $V_1={\lambda }_1{V}^{\prime }+{\lambda }_1$ and $V_0={\lambda }_1{V}^{\prime }+{\lambda }_1$, and $V^{\prime }$ is an independent (or exact) copy of V. In contrast, the corresponding result in the Gaussian case in which X, $X_1$, $X_0$ are independent Gaussian with means $\mu ,{\mu }_1,{\mu }_0$ and variances ${\sigma }^2,{\sigma }_1^2,{\sigma }_0^2$, respectively, is

  $$\begin{array}{c}\hfill D(X+ϵX_1\parallel X+ϵX_0)=\frac{{ϵ}^2}{2{\sigma }^2}{({\mu }_1-{\mu }_0)}^{2}log\mathrm{e}+o\left({ϵ}^2\right).\end{array}$$ (206)

  In fact, it is shown in Lemma 1 of [31] that (206) holds even if $X_1$ and $X_0$ are not Gaussian but have finite variances. It is likely that (205) holds even if $V_1$ and $V_0$ are not Cauchy, but have finite strengths.

• 78.
  An important information theoretic result due to Csiszár [32] is that if $Q_1\ll Q_2$ and P is such that

  $$\begin{array}{cc}\hfill \mathbb{E}\left[{ı}_{Q_1\parallel Q_2}\left(X\right)\right]& =D(Q_1\parallel Q_2),X\sim P,\hfill \end{array}$$ (207)

  then the following Pythagorean identity holds

  $$\begin{array}{cc}\hfill D(P\parallel Q_2)& =D(P\parallel Q_1)+D(Q_1\parallel Q_2).\hfill \end{array}$$ (208)
  Among other applications, this result leads to elegant proofs of minimum relative entropy results. For example, the closest Gaussian to a given P with a finite second moment has the same first and second moments as P. If we let $Q_1$ and $Q_2$ be centered Cauchy with strengths ${\lambda }_1$ and ${\lambda }_2$, respectively, then the orthogonality condition (207) becomes, with the aid of (148) and (198),

  $$\begin{array}{c}\hfill {\mathcal{V}}_{X^2}\left({\lambda }_2^{-1}\right)-{\mathcal{V}}_{X^2}\left({\lambda }_1^{-1}\right)=2{log}_{\mathrm{e}}\left(1+\frac{{\lambda }_1}{{\lambda }_2}\right)-2{log}_{\mathrm{e}}2.\end{array}$$ (209)

  If, in addition, P is centered Cauchy, we can use (28) to verify that (209) holds only in the trivial cases in which either ${\lambda }_1={\lambda }_2$ or $P=Q_1$. For non-Cauchy P, (208) may indeed be satisfied with ${\lambda }_1\ne {\lambda }_2$. For example, using (30), if $X=V_{2,2}$, then (209), and therefore (208), holds with $({\lambda }_1,{\lambda }_2)=(2,0.35459\dots )$.

• 79.
  Mutually absolutely continuous random variables may be such that

  $$\begin{array}{c}\hfill D(X\parallel Z)<\infty =D(Z\parallel X).\end{array}$$ (210)

  An easy example is that of Gaussian X and Cauchy Z, or, if we let X be Cauchy, (210) holds with Z having the very heavy-tailed density function in (62).

• 80.
  While relative entropy is lower semi-continuous, it is not continuous. For example, using the Cauchy distribution, we can show that relative entropy is not stable against small contamination of a Gaussian random variable: if X is Gaussian independent of V, then no matter how small $\lambda \ne 0$,

  $$\begin{array}{c}\hfill D\left(\lambda \right|V|+X\parallel -\lambda |V|+X)=\infty .\end{array}$$ (211)

12. Total Variation Distance

• 81.
  With $f\left(x\right)=|x-1|$, f-divergence becomes the total variation distance (with range [0,2]). Moreover, we have the following representation:

  Theorem 11.

  If $P\ll \gg Q$ and $(P,Q)\equiv (Q,P)$, then

  $$\begin{array}{c}\hfill \frac{1}{2}|P-Q|=2\mathbb{P}[Z>1]-\mathbb{P}[Z\ne 1],\end{array}$$ (212)

  with $Z=\frac{\mathrm{d}P}{\mathrm{d}Q}\left(X\right)$, $X\sim P$.

  Proof. 

  $$\begin{array}{cc}\hfill \frac{1}{2}|P-Q|& =\underset{A\in \mathcal{F}}{max}\left\{P\left(A\right)-Q\left(A\right)\right\}\hfill \end{array}$$ (213)

  $$\begin{array}{cc}\hfill & =P\left(\omega :\frac{\mathrm{d}P}{\mathrm{d}Q}\left(\omega \right)>1\right)-Q\left(\omega :\frac{\mathrm{d}P}{\mathrm{d}Q}\left(\omega \right)>1\right)\hfill \end{array}$$ (214)

  $$\begin{array}{cc}\hfill & =P\left(\omega :\frac{\mathrm{d}P}{\mathrm{d}Q}\left(\omega \right)>1\right)-P\left(\omega :\frac{\mathrm{d}Q}{\mathrm{d}P}\left(\omega \right)>1\right)\hfill \end{array}$$ (215)

  $$\begin{array}{cc}\hfill & =\mathbb{P}[Z>1]-\mathbb{P}[Z<1]\hfill \end{array}$$ (216)

  where (215) and (216) follow from $(P,Q)\equiv (Q,P)$ and $P\ll \gg Q$, respectively.    □

• 82.
  Example 15 of [33] shows that the total variation distance between centered Cauchy distributions is

  $$\begin{array}{cc}\hfill \left|P_{{\lambda }_{1}V}-P_{{\lambda }_{0}V}\right|& =\frac{4}{\pi }arctan\left(\frac{\left|\right|{\lambda }_1|-|{\lambda }_0\left|\right|}{2\sqrt{|{\lambda }_0{\lambda }_1|}}\right)\hfill \end{array}$$ (217)

  $$\begin{array}{cc}\hfill & =\frac{4}{\pi }arctan\left(\sqrt{\frac{1}{2}{\chi }^2(P_{{\lambda }_{1}V}\parallel P_{{\lambda }_{0}V})}\right)\hfill \end{array}$$ (218)

  in view of (197). Since any f-divergence between Cauchy distributions depends on the parameters only through the corresponding ${\chi }^2$-divergence, (217)–(218) imply the general formula

  $$\begin{array}{c}\hfill \left|P_{{\lambda }_{1}V+{\mu }_1}-P_{{\lambda }_{0}V+{\mu }_0}\right|=\frac{4}{\pi }arctan\left(\sqrt{\frac{1}{2}{\chi }^2(P_{{\lambda }_{1}V+{\mu }_1}\parallel P_{{\lambda }_{0}V+{\mu }_0})}\right).\end{array}$$ (219)
  Alternatively, applying Theorem 11 to the case of Cauchy random variables, note that, in this case, Z is an absolutely continuous random variable with density function (153). Therefore, $\mathbb{P}[Z\ne 1]$ = 1, and

  $$\begin{array}{cc}\hfill \mathbb{P}[Z>1]& =\frac{1}{\pi }{\int }_1^{\zeta +\sqrt{{\zeta }^2-1}}\frac{1}{\sqrt{2z\zeta -z^2-1}}\mathrm{d}z\hfill \end{array}$$ (220)

  $$\begin{array}{cc}\hfill & =\frac{1}{2}+\frac{1}{\pi }arctan\sqrt{\frac{1}{2}{\chi }^2},\hfill \end{array}$$ (221)

  where (221) follows from (154) and the identity $arcsin\frac{\sqrt{\delta }}{\sqrt{1+\delta }}=arctan\sqrt{\delta }$ specialized to $\delta =\frac{1}{2}{\chi }^2=\frac{1}{2}(\zeta -1)$. Though more laborious (see [26]), (219) can also be verified by direct integration.

13. Hellinger Divergence

• 83.
  The Hellinger divergence, ${\mathcal{H}}_{\alpha }(P\parallel Q)$ of order $\alpha \in (0,1)\cup (1,\infty )$, is the $f_{\alpha }$-divergence with

  $$\begin{array}{c}\hfill f_{\alpha }\left(t\right)=\frac{t^{\alpha }-1}{\alpha -1}.\end{array}$$ (222)
  Notable special cases are

  $$\begin{array}{cc}\hfill {\mathcal{H}}_2(P\parallel Q)& ={\chi }^2(P\parallel Q),\hfill \end{array}$$ (223)

  $$\begin{array}{cc}\hfill \underset{\alpha \downarrow 1}{lim}{\mathcal{H}}_{\alpha }(P\parallel Q)& =D(P\parallel Q),\hfill \end{array}$$ (224)

  $$\begin{array}{cc}\hfill {\mathcal{H}}_{\frac{1}{2}}(P\parallel Q)& =2{\mathcal{H}}^2(P\parallel Q),\hfill \end{array}$$ (225)

  where ${\mathcal{H}}^2(P\parallel Q)$ is known as the squared Hellinger distance.
• 84.
  For Cauchy random variables, Theorem 10 yields

  $$\begin{array}{cc}\hfill {\mathcal{H}}_{\alpha }({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)& =\frac{1}{\alpha -1}\left(\mathbb{E}\left[Z^{-\alpha }\right]-1\right)\hfill \end{array}$$ (226)

  $$\begin{array}{cc}\hfill & =\frac{{\mathrm{P}}_{\alpha -1}\left(\zeta \right)-1}{\alpha -1},\hfill \end{array}$$ (227)

  where $\zeta$ is as given in (149), and we have used (A15) and ${\mathrm{P}}_{\alpha }(·)$ denotes the Legendre function of the first kind, which satisfies ${\mathrm{P}}_{-\alpha }={\mathrm{P}}_{\alpha -1}$ (see 8.2.1. in [34]).

14. Rényi Divergence

• 85.
  For absolutely continuous probability measures P and Q, with corresponding probability density functions p and q, the Rényi divergence of order$\alpha \in [0,1)\cup (1,\infty )$ is [35]

  $$\begin{array}{cc}\hfill D_{\alpha }(P\parallel Q)& =\frac{1}{\alpha -1}log\left({\int }_{-\infty }^{\infty }{p}^{\alpha }\left(t\right)q^{1-\alpha }\left(t\right)\mathrm{d}t\right).\hfill \end{array}$$ (228)
  Note that, if $(P_1,Q_1)\equiv (P_2,Q_2)$, then $D_{\alpha }(P_1\parallel Q_1)=D_{\alpha }(P_2\parallel Q_2)$. Moreover, although Rényi divergence of order $\alpha$ is not an f-divergence, it is in one-to-one correspondence with the Hellinger divergence of order $\alpha$:

  $$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)=\frac{1}{\alpha -1}log\left(1+(\alpha -1){\mathcal{H}}_{\alpha }(P\parallel Q)\right).\end{array}$$ (229)
• 86.
  An extensive table of order-$\alpha$ Rényi divergences for various continuous random variables can be found in [36]. An addition to that list for Cauchy random variables can be obtained plugging (227) into (229):

  $$\begin{array}{cc}\hfill D_{\alpha }({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)& =\frac{log{\mathrm{P}}_{\alpha -1}\left(\zeta \right)}{\alpha -1}\hfill \end{array}$$ (230)

  $$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log{\mathrm{P}}_{\alpha -1}\left(\frac{{\lambda }_1^2+{\lambda }_0^2+{({\mu }_1-{\mu }_0)}^2}{2|{\lambda }_0{\lambda }_1|}\right),\hfill \end{array}$$ (231)

  for $\alpha \in (0,1)\cup (1,\infty )$.
• 87.
  Suppose that $\lambda \in (0,1)$. Then, (A16) yields

  $$\begin{array}{c}\hfill D_{\frac{1}{2}}(V\parallel \lambda V)=-2log\left(\frac{2\sqrt{\lambda }}{\pi }\mathbf{K}\left(\sqrt{1-{\lambda }^2}\right)\right),\end{array}$$ (232)

  where $\mathbf{K}(·)$ stands for the complete elliptical integral of the first kind in (A18). As indicated in Item 60, to obtain $D_{\frac{1}{2}}({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)$, we just need to substitute $\lambda$ by $\zeta -\sqrt{{\zeta }^2-1}$ in (232), with $\zeta$ given by (149).
• 88.
  Notice that, specializing (86) to $(\alpha ,{\mu }_0,{\mu }_1,{\lambda }_0,{\lambda }_1)=(\frac{1}{2},0,0,\lambda ,1)$, (232) results in the identity

  $$\begin{array}{c}\hfill {\mathrm{P}}_{-\frac{1}{2}}\left(\frac{1}{2\lambda }+\frac{\lambda }{2}\right)=\frac{2\sqrt{\lambda }}{\pi }\mathbf{K}\left(\sqrt{1-{\lambda }^2}\right),\lambda \in (0,1).\end{array}$$ (233)
  Writing the complete elliptical integral of the first kind and the Legendre function of the first kind as special cases of the Gauss hypergeometric function, González [37] noticed the simpler identity (see also 8.13.8 in [34])

  $$\begin{array}{c}\hfill {\mathrm{P}}_{-\frac{1}{2}}\left(\lambda \right)=\frac{2}{\pi }\mathbf{K}\left(\sqrt{\frac{1-\lambda }{2}}\right),\lambda \in (0,1).\end{array}$$ (234)

  We can view (233) and (234) as complementary of each other since they constrain the argument of the Legendre function to belong to $(1,\infty )$ and $(0,1)$, respectively.

• 89.
  Since ${\mathrm{P}}_1\left(z\right)=z$, particularizing (230), we obtain

  $$\begin{array}{cc}\hfill D_2({\lambda }_{1}V+{\mu }_1\parallel {\lambda }_{0}V+{\mu }_0)=log\zeta & =log\left(\frac{{\lambda }_1^2+{\lambda }_0^2+{({\mu }_1-{\mu }_0)}^2}{2|{\lambda }_0{\lambda }_1|}\right).\hfill \end{array}$$ (235)
• 90.
  Since ${\mathrm{P}}_2\left(z\right)=\frac{1}{2}(3z^2-1)$, for Cauchy random variables, we obtain

  $$\begin{array}{c}\hfill D_3(P\parallel Q)=\frac{1}{2}log\left(1+3{\chi }^2(P\parallel Q)+\frac{3}{2}{\chi }^4(P\parallel Q)\right).\end{array}$$ (236)
• 91.
  For Cauchy random variables, the Rényi divergence for integer order 4 or higher can be obtained through (235), (236) and the recursion (dropping $(P\parallel Q)$ for typographical convenience)

  $$\begin{array}{c}\hfill (n+1)exp\left((n+1)D_{n+2}\right)=(2n+1)\zeta exp\left(n{D}_{n+1}\right)-nexp\left((n-1)D_n\right),\end{array}$$ (237)

  which follows from (230) and the recursion of the Legendre polynomials

  $$\begin{array}{c}\hfill (n+1){\mathrm{P}}_{n+1}\left(z\right)=(2n+1)z{\mathrm{P}}_n\left(z\right)-n{\mathrm{P}}_{n-1}\left(z\right),\end{array}$$ (238)

  which, in fact, also holds for non-integer n (see 8.5.3 in [34]).
• 92.
  The Chernoff information

  $$\begin{array}{c}\hfill C(P\parallel Q)=\underset{\lambda \in (0,1)}{sup}(1-\lambda )D_{\lambda }(P\parallel Q)\end{array}$$ (239)

  satisfies $C(P\parallel Q)=C(Q\parallel P)$ regardless of $(P,Q)$. If, as in the case of Cauchy measures, $(P,Q)\equiv (Q,P)$, then Chernoff information is equal to the Bhattacharyya distance:

  $$\begin{array}{c}\hfill C(P\parallel Q)=\frac{1}{2}{D}_{\frac{1}{2}}(P\parallel Q)=log\frac{1}{{\int }_{-\infty }^{\infty }\sqrt{p\left(t\right)q\left(t\right)}\mathrm{d}t}=-log\left(1-{\mathcal{H}}^2(P\parallel Q)\right),\end{array}$$ (240)

  where ${\mathcal{H}}^2(P\parallel Q)$ is the squared Hellinger distance, which is the f-divergence with $f\left(t\right)=\frac{1}{2}{(1-\sqrt{t})}^2$. Together with Item 87, (240) gives the Chernoff information for Cauchy distributions. While it involves the complete elliptical integral function, its simplicity should be contrasted with the formidable expression for Gaussian distributions, recently derived in [38]. The reason (240) holds is that the supremum in (239) is achieved at ${\lambda }^{\ast }=\frac{1}{2}$. To see this, note that

  $$\begin{array}{cc}\hfill f\left(\lambda \right)=(1-\lambda )D_{\lambda }(P\parallel Q)& =\lambda D_{1-\lambda }(Q\parallel P)\hfill \end{array}$$ (241)

  $$\begin{array}{cc}\hfill & =\lambda D_{1-\lambda }(P\parallel Q)\hfill \end{array}$$ (242)

  $$\begin{array}{cc}\hfill & =f(1-\lambda ),\hfill \end{array}$$ (243)

  where (241) reflects the skew-symmetry of Rényi divergence, and (242) holds because $(P,Q)\equiv (Q,P)$. Since $f\left(\lambda \right):\lambda \in [0,1]$ is concave and its own mirror image, it is maximized at ${\lambda }^{\ast }=\frac{1}{2}$.

15. Fisher’s Information

• 93.
  The score function of the standard Cauchy density (1) is

  $$\begin{array}{c}\hfill {\rho }_V\left(x\right)=\nabla {log}_{\mathrm{e}}{f}_V\left(x\right)=-\nabla {log}_{\mathrm{e}}(1+x^2)=-\frac{2x}{1+x^2}.\end{array}$$ (244)
  Then, ${\rho }_V\left(V\right)$ is a zero-mean random variable with second moment equal to Fisher’s information

  $$\begin{array}{c}\hfill J\left(V\right)=\mathbb{E}\left[{\rho }_V^2\left(V\right)\right]=\frac{1}{\pi }{\int }_{-\infty }^{\infty }\frac{4t^2}{{(1+t^2)}^3}\mathrm{d}t=\frac{1}{2},\end{array}$$ (245)

  where we have used (A11). Since Fisher’s information is invariant to location and scales as $J\left(X\right)={\alpha }^{2}J\left(\alpha X\right)$, we obtain

  $$\begin{array}{c}\hfill J(\lambda V+\mu )=\frac{1}{2{\lambda }^2}.\end{array}$$ (246)

  Together with (117), the product of entropy power and Fisher information is $\frac{4\pi }{\mathrm{e}}$, thereby abiding by Stam’s inequality [4], $1\le N\left(X\right)J\left(X\right)$.

• 94.
  Introduced in [39], Fisher’s information of a density function (245) quantifies its similarity with a slightly shifted version of itself. A more general notion is the Fisher information matrix of a random transformation $P_{Y|X}:{\mathbb{R}}^k\to \mathcal{Y}$ satisfying the regularity condition

  $$\begin{array}{c}\hfill D(P_{Y|X=\mathbf{\alpha }}\parallel P_{Y|X=\mathbf{\theta }})=o(\parallel \mathbf{\alpha }-\mathbf{\theta }\parallel ).\end{array}$$ (247)
  Then, the Fisher information matrix of $P_{Y|X}$ at $\mathbf{\theta }$ has coefficients

  $$\begin{array}{cc}\hfill J_{ij}(\mathbf{\theta },P_{Y|X})& =\mathbb{E}\left[\frac{\partial }{\partial {\alpha }_i}{ı}_{P_{Y|X=\mathbf{\alpha }}\parallel P_{Y|X=\mathbf{\theta }}}\left(Y_{\mathbf{\theta }}\right)\frac{\partial }{\partial {\alpha }_j}{ı}_{P_{Y|X=\mathbf{\alpha }}\parallel P_{Y|X=\mathbf{\theta }}}\left(Y_{\mathbf{\theta }}\right)\right]{|}_{\mathbf{\alpha }\leftarrow \mathbf{\theta }},\hfill \end{array}$$ (248)

  and satisfies (with relative entropy in nats)

  $$\begin{array}{c}\hfill D(P_{Y|X=\mathbf{\alpha }}\parallel P_{Y|X=\mathbf{\theta }})=\frac{1}{2}{(\mathbf{\alpha }-\mathbf{\theta })}^{\top }\mathbf{J}(\mathbf{\theta },P_{Y|X})(\mathbf{\alpha }-\mathbf{\theta }){+o(\parallel \mathbf{\alpha }-\mathbf{\theta }\parallel }^2).\end{array}$$ (249)
  For the Cauchy family, the parametrization vector has two components, location and strength, namely, ${\mathbf{\theta }}^{\top }=(\mu ,\lambda )$. The regularity condition (247) is satisfied in view of (205), and we can use the closed-form expression in (205) to obtain

  $$\begin{array}{cc}\hfill J_{11}(\mathbf{\theta },P_{Y|X})=J_{22}(\mathbf{\theta },P_{Y|X})& =\frac{1}{2{\lambda }^2},\hfill \end{array}$$ (250)

  $$\begin{array}{cc}\hfill J_{12}(\mathbf{\theta },P_{Y|X})=J_{21}(\mathbf{\theta },P_{Y|X})& =0.\hfill \end{array}$$ (251)
• 95.
  The relative Fisher information is defined as

  $$\begin{array}{c}\hfill J(P\parallel Q)=\mathbb{E}\left[{\left(\nabla {ı}_{P\parallel Q}\left(X\right)\right)}^2\right],X\sim P.\end{array}$$ (252)
  Although the purpose of this definition is to avoid some of the pitfalls of the classical definition of Fisher’s information, not only do equivalent pairs fail to have the same relative Fisher information but, unlike relative entropy or f-divergence, relative Fisher information is not transparent to injective transformations. For example, $J(X\parallel Y)={\lambda }^{2}J(\lambda X\parallel \lambda Y)$. Centered Cauchy random variables illustrate this fact since

  $$\begin{array}{c}\hfill J(V\parallel \lambda V)=\frac{(4+\lambda ){(\lambda -1)}^2}{2\lambda {(1+\lambda )}^2}\mathrm{and}J(\lambda V\parallel V)=\frac{(4\lambda +1){(\lambda -1)}^2}{2{\lambda }^2{(1+\lambda )}^2}.\end{array}$$ (253)
• 96.
  de Bruijn’s identity [4] states that, if $N\sim \mathcal{N}\left(0,1\right)$ is independent of X, then, in nats,

  $$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}t}h\left(X+\sqrt{t}N\right)=\frac{1}{2}J\left(X+\sqrt{t}N\right),t>0.\end{array}$$ (254)

  As well as serving as the key component in the original proofs of the entropy power inequality, the differential equation in (254) provides a concrete link between Shannon theory and its prehistory. As we show in Theorem 12, it turns out that there is a Cauchy counterpart of de Bruijn’s identity (254). Before stating the result, we introduce the following notation for a parametrized random variable $Y_t$ (to be specified later):

  $$\begin{array}{cc}\hfill \nabla {log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)& =\frac{\partial }{\partial y}{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)=f_{Y_t}^{-1}\left(y\right)\frac{\partial }{\partial y}{f}_{Y_t}\left(y\right),\hfill \end{array}$$ (255)

  $$\begin{array}{cc}\hfill {\nabla }_2{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)& =\frac{\partial }{\partial t}{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)=f_{Y_t}^{-1}\left(y\right)\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right),\hfill \end{array}$$ (256)

  $$\begin{array}{cc}\hfill J\left(Y_t\right)& =\mathbb{E}\left[{\left(\nabla {log}_{\mathrm{e}}{f}_{Y_t}\left(Y_t\right)\right)}^2\right],\hfill \end{array}$$ (257)

  $$\begin{array}{cc}\hfill K\left(Y_t\right)& =\mathbb{E}\left[{\left({\nabla }_2{log}_{\mathrm{e}}{f}_{Y_t}\left(Y_t\right)\right)}^2\right],\hfill \end{array}$$ (258)

  i.e., $J\left(Y_t\right)$ and $K\left(Y_t\right)$ are the Fisher information with respect to location and with respect to dilation, respectively (corresponding to the coefficients $J_{11}$ and $J_{22}$ of the Fisher information matrix when ${\mathbf{\theta }}^{\top }=(\mu ,\lambda )$ as in Item 94. The key to (254) is that $Y_t=X+\sqrt{t}N$, $N\sim \mathcal{N}\left(0,1\right)$ satisfies the partial differential equation

  $$\begin{array}{c}\hfill \frac{{\partial }^2}{\partial y^2}{f}_{Y_t}\left(y\right)=\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right).\end{array}$$ (259)

  Theorem 12.

  Suppose that X is independent of standard Cauchy V. Then, in nats,

  $$\begin{array}{c}\hfill \frac{{\mathrm{d}}^2}{\mathrm{d}{t}^2}h\left(X+tV\right)=-J\left(X+tV\right)-K\left(X+tV\right),t>0.\end{array}$$ (260)

  Proof. 

  Equation (259) does not hold in the current case in which $Y_t=X+tV$, and

  $$\begin{array}{c}\hfill f_{Y_t}\left(y\right)=\frac{t}{\pi }\mathbb{E}\left[\frac{1}{t^2+{(X-y)}^2}\right].\end{array}$$ (261)

  However, some algebra (the differentiation/integration swaps can be justified invoking the bounded convergence theorem) indicates that the convolution with the Cauchy density satisfies the Laplace partial differential equation

  $$\begin{array}{c}\hfill \frac{{\partial }^2}{\partial y^2}{f}_{Y_t}\left(y\right)=-\frac{{\partial }^2}{\partial t^2}{f}_{Y_t}\left(y\right)=\frac{2t}{\pi }\mathbb{E}\left[\frac{3{(X-y)}^2-t^2}{{(t^2+{(X-y)}^2)}^3}\right].\end{array}$$ (262)

  The derivative of the differential entropy of $Y_t$ is, in nats,

  $$\begin{array}{cc}\hfill \frac{\mathrm{d}}{\mathrm{d}t}h\left(Y_t\right)& =-{\int }_{-\infty }^{\infty }\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right)\mathrm{d}y-{\int }_{-\infty }^{\infty }{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right)\mathrm{d}y\hfill \end{array}$$ (263)

  $$\begin{array}{cc}\hfill & =-\frac{\partial }{\partial t}{\int }_{-\infty }^{\infty }{f}_{Y_t}\left(y\right)\mathrm{d}y-{\int }_{-\infty }^{\infty }{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right)\mathrm{d}y.\hfill \end{array}$$ (264)

  Taking another derivative, the left side of (260) becomes

  $$\begin{array}{cc}\hfill \frac{{\mathrm{d}}^2}{\mathrm{d}{t}^2}h\left(Y_t\right)& =-{\int }_{-\infty }^{\infty }\frac{{\partial }^2}{\partial t^2}{f}_{Y_t}\left(y\right){log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\mathrm{d}y-{\int }_{-\infty }^{\infty }\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right)\frac{\partial }{\partial t}{log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\mathrm{d}y\hfill \end{array}$$ (265)

  $$\begin{array}{cc}\hfill & ={\int }_{-\infty }^{\infty }\frac{{\partial }^2}{\partial y^2}{f}_{Y_t}\left(y\right){log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\mathrm{d}y-{\int }_{-\infty }^{\infty }{f}_{Y_t}^{-1}\left(y\right){\left(\frac{\partial }{\partial t}{f}_{Y_t}\left(y\right)\right)}^2\mathrm{d}y\hfill \end{array}$$ (266)

  $$\begin{array}{cc}\hfill & ={\int }_{-\infty }^{\infty }\frac{{\partial }^2}{\partial y^2}{f}_{Y_t}\left(y\right){log}_{\mathrm{e}}{f}_{Y_t}\left(y\right)\mathrm{d}y-K\left(Y_t\right)\hfill \end{array}$$ (267)

  $$\begin{array}{cc}\hfill & =-J\left(Y_t\right)-K\left(Y_t\right),\hfill \end{array}$$ (268)

  where

  • (265) ⟸ the first term on the right side of (264) is zero;
  • (266) ⟸ (262);
  • (267) ⟸ (258);
  • (268) ⟸ integration by parts, exactly as in [4] (or p. 673 of [19]).
       □

• 97.Theorem 12 reveals that the increasing function $f_X\left(t\right)=h\left(X+tV\right)$ is concave (which does not follow from the concavity of differential entropy functional of the density). In contrast, it was shown by Costa [40] that the entropy power $N\left(X+\sqrt{t}N\right)$, with $N\sim \mathcal{N}\left(0,1\right)$ is concave in t.

16. Mutual Information

• 98.
  Most of this section is devoted to an additive noise model. We begin with the simplest case in which $X_{\mathtt{C}}$ is centered Cauchy independent of $W_{\mathtt{C}}$, also centered Cauchy with $\varsigma \left(W_{\mathtt{C}}\right)>0$. Then, (11) yields

  $$\begin{array}{cc}\hfill I(X_{\mathtt{C}};X_{\mathtt{C}}+W_{\mathtt{C}})& =h(X_{\mathtt{C}}+W_{\mathtt{C}})-h\left(W_{\mathtt{C}}\right)\hfill \end{array}$$ (269)

  $$\begin{array}{cc}\hfill & =log\left(4\pi (\varsigma \left(X_{\mathtt{C}}\right)+\varsigma \left(W_{\mathtt{C}}\right))\right)-log\left(4\pi \varsigma \left(W_{\mathtt{C}}\right)\right)\hfill \end{array}$$ (270)

  $$\begin{array}{cc}\hfill & =log\left(1+\frac{\varsigma \left(X_{\mathtt{C}}\right)}{\varsigma \left(W_{\mathtt{C}}\right)}\right),\hfill \end{array}$$ (271)

  thereby establishing a pleasing parallelism with Shannon’s formula [1] for the mutual information between a Gaussian random variable and its sum with an independent Gaussian random variable. Aside from a factor of $\frac{1}{2}$, in the Cauchy case, the role of the variance is taken by the strength. Incidentally, as shown in [2], if N is standard exponential on $(0,\infty )$, an independent X on $[0,\infty )$ can be found so that $X+N$ is exponential, in which case the formula (271) also applies because the ratio of strengths of exponentials is equal to the ratio of their means. More generally, if input and noise are independent non-centered Cauchy, their locations do not affect the mutual information, but they do affect their strengths, so, in that case, (271) holds provided that the strengths are evaluated for the centered versions of the Cauchy random variables.
• 99.
  It is instructive, as well as useful in the sequel, to obtain (271) through a more circuitous route. Since $Y_{\mathtt{C}}=X_{\mathtt{C}}+W_{\mathtt{C}}$ is centered Cauchy with strength $\varsigma \left(Y_{\mathtt{C}}\right)=\varsigma \left(X_{\mathtt{C}}\right)+\varsigma \left(W_{\mathtt{C}}\right)$, the information density (e.g., [41]) is defined as

  $$\begin{array}{cc}\hfill {ı}_{X_{\mathtt{C}};Y_{\mathtt{C}}}(x;y)& =log\frac{\mathrm{d}{P}_{X_{\mathtt{C}}{Y}_{\mathtt{C}}}}{\mathrm{d}(P_{X_{\mathtt{C}}}\times P_{Y_{\mathtt{C}}})}(x,y)\hfill \end{array}$$ (272)

  $$\begin{array}{cc}\hfill & =log\frac{f_{Y_{\mathtt{C}}|X_{\mathtt{C}}}\left(y\right|x)}{f_{Y_{\mathtt{C}}}\left(y\right)}\hfill \end{array}$$ (273)

  $$\begin{array}{cc}\hfill & =log\frac{\varsigma \left(Y_{\mathtt{C}}\right)}{\varsigma \left(W_{\mathtt{C}}\right)}+log\left(1+\frac{y^2}{{\varsigma }^2\left(Y_{\mathtt{C}}\right)}\right)-log\left(1+\frac{{(y-x)}^2}{{\varsigma }^2\left(W_{\mathtt{C}}\right)}\right).\hfill \end{array}$$ (274)
  Averaging with respect to $(X_{\mathtt{C}},Y_{\mathtt{C}})=(X_{\mathtt{C}},X_{\mathtt{C}}+W_{\mathtt{C}})$, we obtain

  $$\begin{array}{cc}\hfill I(X_{\mathtt{C}};Y_{\mathtt{C}})& =\mathbb{E}\left[{ı}_{X_{\mathtt{C}};Y_{\mathtt{C}}}(X_{\mathtt{C}};Y_{\mathtt{C}})\right]\hfill \end{array}$$ (275)

  $$\begin{array}{cc}\hfill & =log\frac{\varsigma \left(Y_{\mathtt{C}}\right)}{\varsigma \left(W_{\mathtt{C}}\right)}+log4-log4=log\left(1+\frac{\varsigma \left(X_{\mathtt{C}}\right)}{\varsigma \left(W_{\mathtt{C}}\right)}\right).\hfill \end{array}$$ (276)
• 100.
  If the strengths of output $Y=X+N$ and independent noise N are finite and their differential entropies are not $-\infty$, we can obtain a general representation of the mutual information without requiring that either input or noise be Cauchy. Invoking (56) and $I(X;X+N)=h(X+N)-h\left(N\right)$, we have

  $$\begin{array}{cc}\hfill I(X;Y)& =log\frac{N_{\mathtt{C}}\left(Y\right)}{N_{\mathtt{C}}\left(N\right)}\hfill \end{array}$$ (277)

  $$\begin{array}{cc}\hfill & =log\frac{\varsigma \left(Y\right)}{\varsigma \left(N\right)}+D(N\parallel \varsigma \left(N\right)V)-D(Y\parallel \varsigma \left(Y\right)V),\hfill \end{array}$$ (278)

  since, as we saw in (49), the finiteness of the strengths guarantees the finiteness of the relative entropies in (278). We can readily verify the alternative representation in which strength is replaced by standard deviation, and the standard Cauchy V is replaced by standard normal W:

  $$\begin{array}{cc}\hfill I(X;Y)& =\frac{1}{2}log\frac{N\left(Y\right)}{N\left(N\right)}\hfill \end{array}$$ (279)

  $$\begin{array}{cc}\hfill & =log\frac{\sigma \left(Y\right)}{\sigma \left(N\right)}+D(N\parallel \sigma \left(N\right)W)-D(Y\parallel \sigma \left(Y\right)W).\hfill \end{array}$$ (280)
  A byproduct of (278) is the upper bound

  $$\begin{array}{cc}\hfill I(X;Y)& \le log\frac{\varsigma \left(Y\right)}{N_{\mathtt{C}}\left(N\right)}\hfill \end{array}$$ (281)

  $$\begin{array}{cc}\hfill & =log\frac{\varsigma \left(Y\right)}{\varsigma \left(N\right)}+D(N\parallel \varsigma \left(N\right)V),\hfill \end{array}$$ (282)

  where (281) follows from $N_{\mathtt{C}}\left(Y\right)\le \varsigma \left(Y\right)$, and (282) follows by dropping the last term on the right side of (278). Note that (281) is the counterpart of the upper bound given by Shannon [1] in which the standard deviation of Y takes the place of the strength in the numerator, and the square root of the noise entropy power takes the place of the entropy strength in the denominator. Shannon gave his bound three years before Kullback and Leibler introduced relative entropy in [42]. The counterpart of (282) with analogous substitutions of strengths by standard deviations was given by Pinsker [43], and by Ihara [44] for continuous-time processes.
• 101.We proceed to investigate the maximal mutual information between the (possibly non-Cauchy) input and its additive Cauchy-noise contaminated version.

  Theorem 13.

  Maximal mutual information: output strength constraint. For any $\eta \ge \varsigma \left(W_{\mathtt{C}}\right)>0$,

  $$\begin{array}{c}\hfill \underset{X:\varsigma (X+W_{\mathtt{C}})\le \eta }{max}I(X;X+W_{\mathtt{C}})=log\frac{\eta }{\varsigma \left(W_{\mathtt{C}}\right)},\end{array}$$ (283)

  where $W_{\mathtt{C}}$ is centered Cauchy independent of X. The maximum in (283) is attained uniquely by the centered Cauchy distribution with strength $\eta -\varsigma \left(W_{\mathtt{C}}\right)$.

  Proof. 

  For centered Cauchy noise, the upper bound in (282) simplifies to

  $$\begin{array}{cc}\hfill I(X;X+W_{\mathtt{C}})& \le log\frac{\varsigma (X+W_{\mathtt{C}})}{\varsigma \left(W_{\mathtt{C}}\right)},\hfill \end{array}$$ (284)

  which shows ≤ in (283). If the input is centered Cauchy $X_{\mathtt{C}}$ with strength $\eta -\varsigma \left(W_{\mathtt{C}}\right)$, then $\varsigma (X_{\mathtt{C}}+W_{\mathtt{C}})=\eta$, and $I(X_{\mathtt{C}};X_{\mathtt{C}}+W_{\mathtt{C}})$ is equal to the right side in view of (271).

     □

• 102.
  In the information theory literature, the maximization of mutual information over the input distribution is usually carried out under a constraint on the average cost $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]$ for some real-valued function $\mathsf{b}$. Before we investigate whether the optimization in (283) can be cast into that conventional paradigm, it is instructive to realize that the maximization of mutual information in the case of input-independent additive Gaussian noise can be viewed as one in which we allow any input such that the output variance is constrained, and because the output variance is the sum of input and noise variances that the familiar optimization over variance constrained inputs obtains. Likewise, in the case of additive exponential noise and random variables taking nonnegative values, if we constrain the output mean, automatically we are constraining the input mean. In contrast, the output strength is not equal to the sum of Cauchy noise strength and the input strength, unless the input is Cauchy. Indeed, as we saw in Theorem 1-(d), the output strength depends not only on the input strength but on the shape of its probability density function. Since the noise is Cauchy, (45) yields

  $$\begin{array}{cc}\hfill \varsigma (X+W_{\mathtt{C}})\le \eta & ⟺{\varsigma }_{2,\theta }\left(X\right)\le \varsigma \left(W_{\mathtt{C}}\right)+\eta ,with\theta =2log\frac{2\eta }{\eta +\varsigma \left(W_{\mathtt{C}}\right)}\hfill \end{array}$$ (285)

  $$\begin{array}{cc}\hfill & ⟺\mathbb{E}\left[log\left(\varsigma \left(W_{\mathtt{C}}\right)+\eta +X^2\right)\right]\le 2log\left(2\eta \right),\hfill \end{array}$$ (286)

  which is the same input constraint found in [45] (see also Lemma 6 in [46] and Section V in [47]) in which $\eta$ affects not only the allowed expected cost but the definition of the cost function itself. If X is centered Cauchy with strength $\eta -\varsigma \left(W_{\mathtt{C}}\right)$, then (286) is satisfied with equality, in keeping with the fact that that input achieves the maximum in (283). Any alternative input with the same strength that produces output strength lower than or equal to $\eta$ can only result in lower mutual information. However, as we saw in Item 29, we can indeed find input distributions with strength $\eta -\varsigma \left(W_{\mathtt{C}}\right)$ that can produce output strength higher than $\eta$. Can any of those input distributions achieve $I(X;Y)>log\frac{\eta }{\varsigma \left(W_{\mathtt{C}}\right)}$? The answer is affirmative. If we let $X=V_{\beta ,2}$, defined in (9), we can verify numerically that, for $\beta \in [0.8,1)$,

  $$\begin{array}{c}\hfill I(X;X+V)>log\left(\varsigma \left(X\right)+1\right).\end{array}$$ (287)
  We conclude that, at least for $\frac{\theta }{\varsigma \left(W_{\mathtt{C}}\right)}\in \left(1,\varsigma \left(V_{0.8,2}\right)\right)=(1,3.126\dots )$, the capacity–input–strength function satisfies

  $$\begin{array}{c}\hfill C\left(\theta \right)=\underset{X:\varsigma \left(X\right)\le \theta }{max}I(X;X+W_{\mathtt{C}})>log\left(1+\frac{\theta }{\varsigma \left(W_{\mathtt{C}}\right)}\right).\end{array}$$ (288)
• 103.Although not always acknowledged, the key step in the maximization of mutual information over the input distribution for a given random transformation is to identify the optimal output distribution. The results in Items 101 and 102 point out that it is mathematically more natural to impose constraints on the attributes of the observed noisy signal than on the transmitted noiseless signal. In the usual framework of power constraints, both formulations are equivalent as an increase in the gain of the receiver antenna (or a decrease in the front-end amplifier thermal noise) of $\kappa$ dB has the same effect as an increase of $\kappa$ dB in the gain of the transmitter antenna (or increase in the output power of the transmitted amplifier). When, as in the case of strength, both formulations lead to different solutions, it is worthwhile to recognize that what we usually view as transmitter/encoder constraints also involve receiver features.
• 104.
  Consider a multiaccess channel $Y_i=X_{1i}+X_{2i}+W_i$, where $W_i$ is a sequence of strength $\varsigma \left(W\right)$ independent centered Cauchy random variables. While the capacity region is unknown if we place individual cost or strength constraints on the transmitters, it is easily solvable if we impose an output strength constraint. In that case, the capacity region is the triangle

  $$\begin{array}{c}\hfill C_{\eta }=\left\{(R_1,R_2)\in {[0,\infty )}^2:R_1+R_2\le log\frac{\eta }{\varsigma \left(W\right)}\right\},\end{array}$$ (289)

  where $\eta >\varsigma \left(W\right)$ is the output strength constraint. To see this, note (a) the corner points are achievable thanks to Theorem 13; (b) if the transmitters are synchronous, a time-sharing strategy with Cauchy distributed inputs satisfies the output strength constraint in view of (107); (c) replacing the independent encoders by a single encoder which encodes both messages would not be able to achieve higher rate sum. It is also possible to achieve (289) using the successive decoding strategy invented by Cover [48] and Wyner [49] for the Gaussian multiple-access channel: fix $\alpha \in (0,1)$; to achieve $R_1=\alpha log\frac{\eta }{\varsigma \left(W\right)}$ and $R_2=(1-\alpha )log\frac{\eta }{\varsigma \left(W\right)}$, we let the transmitters use random coding with sequences of independent Cauchy random variables with respective strengths

  $$\begin{array}{cc}\hfill {\varsigma }_1& =\eta -{\varsigma }^{\alpha }\left(W\right){\eta }^{1-\alpha }>0,\hfill \end{array}$$ (290)

  $$\begin{array}{cc}\hfill {\varsigma }_2& ={\varsigma }^{\alpha }\left(W\right){\eta }^{1-\alpha }-\varsigma \left(W\right)>0,\hfill \end{array}$$ (291)

  which abide by the output strength constraint since ${\varsigma }_1+{\varsigma }_2+\varsigma \left(W\right)=\eta$, and

  $$\begin{array}{cc}\hfill R_1& =log\left(1+\frac{{\varsigma }_1}{{\varsigma }_2+\varsigma \left(W\right)}\right),\hfill \end{array}$$ (292)

  $$\begin{array}{cc}\hfill R_2& =log\left(1+\frac{{\varsigma }_2}{\varsigma \left(W\right)}\right),\hfill \end{array}$$ (293)

  a rate-pair which is achievable by successive decoding by using a single-user decoder for user 1, which treats the codeword transmitted by user 2 as noise; upon decoding the message of user 1, it is re-encoded and subtracted from the received signal, thereby presenting a single-user decoder for user 2 with a signal devoid of any trace of user 1 (with high probability).
• 105.
  The capacity per unit energy of the additive Cauchy-noise channel $Y_i=X_i+\lambda V_i$, where $\left\{V_i\right\}$ is an independent sequence of standard Cauchy random variables, was shown in [29] to be equal to ${\left(4{\lambda }^2\right)}^{-1}log\mathrm{e}$, even though the capacity-cost function of such a channel is unknown. A corollary to Theorem 13 is that the capacity per unit output strength of the same channel is

  $$\begin{array}{c}\hfill {\mathbf{C}}_O=\frac{1}{\lambda }\underset{\eta \ge \lambda }{max}\frac{\lambda }{\eta }log\frac{\eta }{\lambda }=\frac{log\mathrm{e}}{\lambda \mathrm{e}}.\end{array}$$ (294)
  By only considering Cauchy distributed inputs, the capacity per unit input strength is lower bounded by

  $$\begin{array}{c}\hfill {\mathbf{C}}_I\ge \underset{\gamma >0}{max}\frac{1}{\gamma }log\left(1+\frac{\gamma }{\lambda }\right)=\frac{log\mathrm{e}}{\lambda }\end{array}$$ (295)

  but is otherwise unknown as it is not encompassed by the formula in [29].
• 106.
  We turn to the scenario, dual to that in Theorem 13, in which the input is Cauchy but the noise need not be. As Shannon showed in [1], if the input is Gaussian, among all noise distributions with given second moment, independent Gaussian noise is the least favorable. Shannon showed that fact applying the entropy power inequality to the numerator on the right side of (279), and then further weakened the resulting lower bound by replacing the noise entropy power in the denominator by its variance. Taking a cue from this simple approach, we apply the entropy strength inequality (124) to (277) to obtain

  $$\begin{array}{cc}\hfill I(X_{\mathtt{C}};X_{\mathtt{C}}+W)& =\frac{1}{2}log\frac{N_{\mathtt{C}}^2\left(Y\right)}{N_{\mathtt{C}}^2\left(W\right)}\hfill \end{array}$$ (296)

  $$\begin{array}{cc}\hfill & \ge \frac{1}{2}log\frac{N_{\mathtt{C}}^2\left(X_{\mathtt{C}}\right)+N_{\mathtt{C}}^2\left(W\right)}{N_{\mathtt{C}}^2\left(W\right)}\hfill \end{array}$$ (297)

  $$\begin{array}{cc}\hfill & =\frac{1}{2}log\left(1+\frac{{\varsigma }^2\left(X_{\mathtt{C}}\right)}{N_{\mathtt{C}}^2\left(W\right)}\right)\hfill \end{array}$$ (298)

  $$\begin{array}{cc}\hfill & \ge \frac{1}{2}log\left(1+\frac{{\varsigma }^2\left(X_{\mathtt{C}}\right)}{{\varsigma }_{\mathtt{C}}^2\left(W\right)}\right),\hfill \end{array}$$ (299)

  where (299) follows from $N_{\mathtt{C}}^2\left(W\right)\le {\varsigma }_{\mathtt{C}}^2\left(W\right)$. Unfortunately, unlike the case of Gaussian input, this route falls short of showing that Cauchy noise of a given strength is least favorable because the right side of (299) is strictly smaller than the Cauchy-input Cauchy-noise mutual information in (271). Evidently, while the entropy power inequality is tight for Gaussian random variables, it is not for Cauchy random variables as we observed in Item 39. For this approach to succeed showing that, under a strength constraint, the least favorable noise is centered Cauchy we would need that, if W is independent of standard Cauchy V, then $N_{\mathtt{C}}(V+W)-N_{\mathtt{C}}\left(W\right)\ge 1$. (See Item 119c-(a).)
• 107.As in Item 102, the counterpart in the Cauchy-input case is more challenging due to the fact that, unlike variance, the output strength need not be equal to the sum of input and noise strength. The next two results give lower bounds which, although achieved by Cauchy noise, do not just depend on the noise distribution through its strength.

  Theorem 14.

  If $X_{\mathtt{C}}$ is centered Cauchy, independent of W with $0<\varsigma \left(W\right)<\infty$, denote $Y=X_{\mathtt{C}}+W$. Then,

  $$\begin{array}{c}\hfill I(X_{\mathtt{C}};X_{\mathtt{C}}+W)\ge log\frac{\varsigma \left(Y\right)}{\varsigma \left(W\right)}-\left|log\frac{\varsigma \left(W\right)}{\varsigma \left(Y\right)-\varsigma \left(X_{\mathtt{C}}\right)}\right|,\end{array}$$ (300)

  with equality if W is centered Cauchy.

  Proof. 

  Let us abbreviate $\varsigma =\varsigma \left(Y\right)-\varsigma \left(X_{\mathtt{C}}\right)$. Consider the following chain:

  $$\begin{array}{cc}\hfill D(Y\parallel \varsigma (Y\left)V\right)-D(W\parallel \varsigma (W\left)V\right)& =D\left(X_{\mathtt{C}}+W\parallel X_{\mathtt{C}}+\varsigma V\right)-D(W\parallel \varsigma \left(W\right)V)\hfill \end{array}$$ (301)

  $$\begin{array}{cc}\hfill & \le D\left(W\parallel \varsigma V\right)-D(W\parallel \varsigma \left(W\right)V)\hfill \end{array}$$ (302)

  $$\begin{array}{cc}\hfill & =log\frac{\varsigma \left(W\right)}{\varsigma }+\mathbb{E}\left[log\frac{{\varsigma }^2+W^2}{{\varsigma }^2\left(W\right)+W^2}\right]\hfill \end{array}$$ (303)

  $$\begin{array}{cc}\hfill & \le \left|log\frac{\varsigma \left(W\right)}{\varsigma }\right|,\hfill \end{array}$$ (304)

  where

  • (301) ⟸$X_{\mathtt{C}}$ is centered Cauchy;
  • (302) ⟸ relative entropy data processing theorem applied to a random transformation that consists of the addition of independent “noise” $X_{\mathtt{C}}$;
  • (303) ⟸ both relative entropies are finite since $\varsigma \left(W\right)<\infty$;
  • (304) ⟸ the elementary observation

    $$\begin{array}{c}\hfill log\frac{{\varsigma }^2+t^2}{{\varsigma }^2\left(W\right)+t^2}\le \left\{\begin{array}{cc}0,\hfill & \varsigma <\varsigma \left(W\right);\hfill \\ 2log\frac{\varsigma }{\varsigma \left(W\right)},\hfill & \varsigma \ge \varsigma \left(W\right).\hfill \end{array}\right.\end{array}$$ (305)

  The desired bound (300) now follows in view of (278). It holds with equality in W being centered Cauchy as, in that case, $\varsigma \left(Y\right)=\varsigma \left(X_{\mathtt{C}}\right)+\varsigma \left(W_{\mathtt{C}}\right)$.    □

  Although the lower bound in Theorem 14 is achieved by a centered Cauchy, it does not rule out the existence of W such that $\varsigma \left(W\right)=\varsigma \left(W_{\mathtt{C}}\right)$ and $I(X_{\mathtt{C}};X_{\mathtt{C}}+W)<I(X_{\mathtt{C}};X_{\mathtt{C}}+W_{\mathtt{C}})$.

• 108.For the following lower bound, it is advisable to assume for notational simplicity and without loss of generality that $\varsigma \left(X_{\mathtt{C}}\right)=1$. To remove that restriction, we may simply replace W by $\varsigma \left(X_{\mathtt{C}}\right)W$.

  Theorem 15.

  Let V be standard Cauchy independent of W. Then,

  $$\begin{array}{c}\hfill I(V;V+W)\ge log\left(1+\frac{1}{\lambda \left(W\right)}\right),\end{array}$$ (306)

  where $\lambda \left(W\right)$ is the solution to

  $$\begin{array}{c}\hfill \mathbb{E}\left[log\frac{{(2+\lambda )}^2+W^2}{{\lambda }^2+W^2}\right]=2log\left(1+\frac{1}{\lambda }\right).\end{array}$$ (307)

  Equality holds in (306) if W is a centered Cauchy random variable, in which case, $\lambda \left(W\right)=\varsigma \left(W\right)$.

  Proof. 

  It can be shown that, if $P_{XY}=P_X{P}_{Y|X}=P_Y{P}_{X|Y}$ and $Q_{Y|X}$ is an auxiliary random transformation such that $P_X{Q}_{Y|X}=Q_Y{Q}_{X|Y}$ where $Q_Y$ is the response of $Q_{Y|X}$ to $P_X$, then

  $$\begin{array}{cc}\hfill I(X;Y)& =D(P_{X|Y}\parallel Q_{X|Y}\left|P_Y\right)+\mathbb{E}\left[{ı}_{X;\overline{Y}}\right(X;Y\left)\right],\hfill \end{array}$$ (308)

  where $(X,Y)\sim P_X{P}_{Y|X}$ and the information density ${ı}_{X;\overline{Y}}$ corresponds to the joint probability measure $P_X{Q}_{Y|X}$. We can participate decomposition of mutual information to the case where $P_X=P_V,P_{Y|X=ₓ}=P_{W+ₓ},Q_{Y|X=ₓ}=P_{W_{c_{+x}}}$ where W_c is centered Cauchy with strenght λ > 0. Then, $P_X{Q}_{Y|X}$ is the joint distribution of V and V + W_C, and

  $${ı}_{X;\overline{Y}}(x;y=\log \frac{\lambda }{1+\lambda }-\log ({\lambda }^2+{(y-x)}^2)+\log ({(1+\lambda )}^2+y^2).$$ (309)

  Taking expectation with respect to $(x,y)=(V,V+t)$, and invoking (52), we obtain

  $$\mathbb{E}\left[{ı}_{X;\overline{Y}}\right(V;V+t\left)\right]=\log \frac{\lambda }{1+\lambda }+\mathbb{E}\left[\log \frac{{(1+\lambda )}^2+{(\mathrm{V}+\mathrm{t})}^2}{{\lambda }^2+{\mathrm{t}}^2}\right]$$ (310)

  $$=\log \frac{\lambda }{1+\lambda }+\log \frac{{(2+\lambda )}^2+{\mathrm{t}}^2}{{\lambda }^2+{\mathrm{t}}^2}.$$ (311)

  Finally, taking expectation with respect to $t=W$, we obtain

  $$\mathbb{E}\left[{ı}_{X;\overline{Y}}\right(V;V+W\left)\right]=\mathbb{E}\left[\log \frac{{(2+\lambda )}^2+{\mathrm{t}}^2}{{\lambda }^2+{\mathrm{W}}^2}\right]-\log (1+\frac{1}{\lambda }).$$ (312)

  If $\lambda =\lambda \left(W\right)$, namely, the solution to (307), then (306) follows as a result of (108). If $W=\varsigma \left(W\right)V$, then the solution to (307) is $\lambda \left(W\right)=\varsigma \left(W\right)$, and the equality in (306) can be seen by specializing (271) to $(\varsigma \left(X_{\mathtt{C}}\right),\varsigma \left(W_{\mathtt{C}}\right))=(1,\varsigma \left(W\right))$.    □

• 109.
  As we just saw, if W is centered Cauchy, then the solution to (307) satisfies $\lambda \left(W\right)=\varsigma \left(W\right)$. On the other hand, we have

  $$\begin{array}{cc}\hfill 0.302..=\varsigma \left(V_{2,2}\right)& <\lambda \left(V_{2,2}\right)=0.349\dots \hfill \end{array}$$ (313)

  $$\begin{array}{cc}\hfill 4.961...=\lambda \left(W\right)& <\varsigma \left(W\right)=5.845\dots \hfill \end{array}$$ (314)

  if W has the probability density function in (100).
• 110.
  As the proof indicates, at the expense of additional computation, we may sharpen the lower bound in Theorem 15 to show

  $$\begin{array}{c}\hfill I(V;V+W)\ge \underset{\lambda >0}{max}\left\{\mathbb{E}\left[log\frac{{(2+\lambda )}^2+W^2}{{\lambda }^2+W^2}\right]-log\left(1+\frac{1}{\lambda }\right)\right\},\end{array}$$ (315)

  which is attained at the solution to

  $$\begin{array}{c}\hfill \frac{\lambda }{2+\lambda }{\eta }_{W^2}\left(\frac{1}{{(2+\lambda )}^2}\right)-{\eta }_{W^2}\left(\frac{1}{{\lambda }^2}\right)+\frac{1}{2\lambda +2}=0.\end{array}$$ (316)
• 111.

  Theorem 16.

  The rate–distortion function of a memoryless source whose distribution is centered Cauchy with strength $\varsigma \left(X\right)$ such that the time-average of the distortion strength is upper bounded by D is given by

  $$\begin{array}{c}\hfill R\left(D\right)=\left\{\begin{array}{cc}log\frac{\varsigma \left(X\right)}{D},\hfill & 0<D<\varsigma \left(X\right);\hfill \\ 0,\hfill & \phantom{0<}D\ge \varsigma \left(X\right).\hfill \end{array}\right.\end{array}$$ (317)

  Proof. 

  If $D\ge \varsigma \left(X\right)$, reproducing the source by $(0,\dots ,0)$ results in time-average of the distortion strength equal to $\frac{1}{n}{\sum }_{i=1}^n\varsigma \left(X_i\right)=\varsigma \left(X\right)$. Therefore, $R\left(D\right)=0$. If $0<D<\varsigma \left(X\right)$, we proceed to determine the minimal $I(X;\widehat{X})$ among all $P_{\widehat{X}|X}$ such that $\varsigma (X-\widehat{X})\le D$. For any such random transformation,

  $$\begin{array}{cc}\hfill I(X;\widehat{X})& =h\left(X\right)-h\left(X\right|\widehat{X})\hfill \end{array}$$ (318)

  $$\begin{array}{cc}\hfill & =h\left(X\right)-h(X-\widehat{X}|\widehat{X})\hfill \end{array}$$ (319)

  $$\begin{array}{cc}\hfill & \ge h\left(X\right)-h(X-\widehat{X})\hfill \end{array}$$ (320)

  $$\begin{array}{cc}\hfill & =log\left(4\pi \varsigma \left(X\right)\right)-h(X-\widehat{X})\hfill \end{array}$$ (321)

  $$\begin{array}{cc}\hfill & \ge log\left(4\pi \varsigma \left(X\right)\right)-log\left(4\pi \varsigma (X-\widehat{X})\right)\hfill \end{array}$$ (322)

  $$\begin{array}{cc}\hfill & \ge log\frac{\varsigma \left(X\right)}{D},\hfill \end{array}$$ (323)

  where (320) holds because conditioning cannot increase differential entropy, and (322) follows from Theorem 3 applied to $Z=X-\widehat{X}$. The fact that there is an allowable $P_{\widehat{X}|X}$ that achieves the lower bound with equality is best seen by letting $X=\widehat{X}+Z$, where Z and $\widehat{X}$ are independent centered Cauchy random variables with $\varsigma \left(Z\right)=D$ and $\varsigma \left(\widehat{X}\right)=\varsigma \left(X\right)-D$. Then, $P_{\widehat{X}|X}{P}_X=P_{X|\widehat{X}}{P}_{\widehat{X}}$ is such that the X marginal is indeed centered Cauchy with strength $\varsigma \left(X\right)$, and $\varsigma (X-\widehat{X})=D$. Recalling (271),

  $$\begin{array}{c}\hfill I(\widehat{X};X)=log\left(1+\frac{\varsigma \left(X\right)-D}{\varsigma \left(Z\right)}\right)=log\frac{\varsigma \left(X\right)}{D},\end{array}$$ (324)

  and the lower bound in (323) can indeed be satisfied with equality. We are not finished yet since we need to justify that the rate–distortion function is indeed

  $$\begin{array}{c}\hfill R\left(D\right)=\underset{P_{\widehat{X}|X}:\varsigma (X-\widehat{X})\le D}{min}I(X;\widehat{X}),\end{array}$$ (325)

  which does not follow from the conventional memoryless lossy compression theorem with average distortion because, although the distortion measure is separable, it is not the average of a function with respect to the joint probability measure $P_{X\widehat{X}}$. This departure from the conventional setting does not impact the direct part of the theorem (i.e., ≤ in (325)), but it does affect the converse and in particular the proof of the fact that the n-version of the right side of (325) single-letterizes. To that end, it is sufficient to show that the function of D on the right side of (325) is convex (e.g., see pp. 316–317 in [19]). In the conventional setting, this follows from the convexity of the mutual information in the random transformation since, with a distortion function $d(·,·)$, we have

  $$\begin{array}{c}\hfill \mathbb{E}\left[d(X,{\widehat{X}}_{\alpha })\right]=\alpha \mathbb{E}\left[d(X,{\widehat{X}}_1)\right]+(1-\alpha )\mathbb{E}\left[d(X,{\widehat{X}}_0)\right],\end{array}$$ (326)

  where $(X,{\widehat{X}}_1)\sim P_X{P}_{\widehat{X}|X}^1$, $(X,{\widehat{X}}_0)\sim P_X{P}_{\widehat{X}|X}^0$, and $(X,{\widehat{X}}_{\alpha })\sim \alpha P_X{P}_{\widehat{X}|X}^1+(1-\alpha )P_X{P}_{\widehat{X}|X}^0$. Unfortunately, as we saw in Item 35, strength is not convex on the probability measure so, in general, we cannot claim that

  $$\begin{array}{c}\hfill \varsigma (X-{\widehat{X}}_{\alpha })\le \alpha \varsigma (X-{\widehat{X}}_1)+(1-\alpha )\varsigma (X-{\widehat{X}}_0).\end{array}$$ (327)

  The way out of this quandary is to realize that (327) is only needed for those $P_{\widehat{X}|X}^0$ and $P_{\widehat{X}|X}^1$ that attain the minimum on the right side of (325) for different distortion bounds $D_0$ and $D_1$. As we saw earlier in this proof, those optimal random transformations are such that $X-{\widehat{X}}_0$ and $X-{\widehat{X}}_1$ are centered Cauchy. Fortuitously, as we noted in (107), (327) does indeed hold when we restrict attention to mixtures of centered Cauchy distributions.    □

  Theorem 16 gives another example in which the Shannon lower bound to the rate–distortion function is tight. In addition to Gaussian sources with mean–square distortion, other examples can be found in [50]. Another interesting aspect of the lossy compression of memoryless Cauchy sources under strength distortion measure is that it is optimally successively refinable in the sense of [51,52]. As in the Gaussian case, this is a simple consequence of the stability of the Cauchy distribution and the fact that the strength of the sum of independent Cauchy random variables is equal to the sum of their respective strengths (Item 27).

• 112.The continuity of mutual information can be shown under the following sufficient conditions

  Theorem 17.

  Suppose that $X_n$ is a sequence of real-valued random variables that vanishes in strength, Z is independent of $X_n$, $h\left(Z\right)>-\infty$ and $0<\varsigma \left(Z\right)<\infty$. Then,

  $$\begin{array}{c}\hfill \underset{n\to \infty }{lim}I(X_n;X_n+Z)=0.\end{array}$$ (328)

  Proof. 

  Under the assumptions, $h\left(Z\right)\in \mathbb{R}$. Therefore, $I(X_n;X_n+Z)=h(X_n+Z)-h\left(Z\right)$, and (328) follows from Theorem 1-(m).    □

• 113.
  The assumption $h\left(Z\right)>-\infty$ is not superfluous for the validity of Theorem 17 even though it was not needed in Theorem 1-(m). Suppose that Z is integer valued, and $X_n={\left(nL\right)}^{-1}\in (0,\frac{1}{2})$ where $L\in \{2,3,\dots \}$ has probability mass function

  $$\begin{array}{c}\hfill P_L\left(k\right)=\frac{0.986551...}{k{log}_2^{2}k},k=2,3,\dots \end{array}$$ (329)

  Then, $I(X_n;X_n+Z)=H\left(X_n\right)=H\left(L\right)=\infty$, while $\mathbb{E}\left[\right|X_n\left|\right]=\frac{0.328289...}{n}$, and therefore, $\varsigma \left(X_n\right)\to 0$.

• 114.
  In the case in which $V^n$ and $W^n$ are standard spherical multivariate Cauchy random variables with densities in (6), it follows from (7) that ${\lambda }_X{V}^n+{\lambda }_W{W}^n$ has the same distribution as $\left(\right|{\lambda }_X|+|{\lambda }_W\left|\right)V^n$. Therefore,

  $$\begin{array}{cc}\hfill I\left(V^n;{\lambda }_X{V}^n+{\lambda }_W{W}^n\right)& =h\left({\lambda }_X{V}^n+{\lambda }_W{W}^n\right)-h\left({\lambda }_W{W}^n\right)\hfill \end{array}$$ (330)

  $$\begin{array}{cc}\hfill & =nlog\left(1+\frac{|{\lambda }_X|}{|{\lambda }_W|}\right),\hfill \end{array}$$ (331)

  where we have used the scaling law $h\left(\alpha X^n\right)=nlog\left|\alpha \right|+h\left(X^n\right)$. There is no possibility of a Cauchy-counterpart of the celebrated log-determinant formula for additive Gaussian vectors (e.g., Theorem 9.2.1 in [41]) because, as pointed out in Item 7, ${\mathrm{\Lambda }}^{\frac{1}{2}}{V}^n+{\overline{\mathrm{\Lambda }}}^{\frac{1}{2}}{W}^n$ is not distributed according to the ellipsoidal density in (8) unless $\mathrm{\Lambda }$ and $\overline{\mathrm{\Lambda }}$ are proportional, in which case the setup reverts to that in (330).
• 115.
  To conclude this section, we leave aside additive noise models and consider the mutual information between a partition of the components of the standard spherical multivariate Cauchy density (6). If $\mathcal{I}\cap \mathcal{J}=⌀$, then (17) yields

  $$\begin{array}{c}\hfill I\left(\{V_i,i\in \mathcal{I}\};\{V_j,j\in \mathcal{J}\}\right)=h_{\left|\mathcal{I}\right|}+h_{\left|\mathcal{J}\right|}-h_{\left|\mathcal{I}\right|+\left|\mathcal{J}\right|},\end{array}$$ (332)

  where $h_n$ stands for the right side of (17). For example, if $i\ne j$, then, in nats,

  $$\begin{array}{cc}\hfill I(V_i;V_j)& =2h\left(V_1\right)-h(V_1,V_2)\hfill \end{array}$$ (333)

  $$\begin{array}{cc}\hfill & =2{log}_{\mathrm{e}}\left(4\pi \right)-\frac{3}{2}\left({log}_{\mathrm{e}}\left(4\pi \right)+\gamma +\psi \left(\frac{3}{2}\right)\right)-{log}_{\mathrm{e}}\mathrm{\Gamma }\left(\frac{3}{2}\right)\hfill \end{array}$$ (334)

  $$\begin{array}{cc}\hfill & ={log}_{\mathrm{e}}\left(8\pi \right)-3=0.22417\dots \hfill \end{array}$$ (335)
  More generally, the dependence index among the n random variables in the standard spherical multivariate Cauchy density is (see also [9,53]), in nats,

  $$\begin{array}{cc}\hfill & D(P_{V^n}\parallel P_{V_1}\times \cdots \times P_{V_n})=nh\left(V_1\right)-h\left(V^n\right)\hfill \end{array}$$ (336)

  $$\begin{array}{cc}\hfill & =\frac{n-1}{2}{log}_{\mathrm{e}}\left(4\pi \right)+{log}_{\mathrm{e}}\mathrm{\Gamma }\left(\frac{n+1}{2}\right)-\frac{n+1}{2}\left(\gamma +\psi \left(\frac{n+1}{2}\right)\right)\hfill \end{array}$$ (337)

  $$\begin{array}{cc}\hfill & =\left\{\begin{array}{cc}\frac{n}{2}{log}_{\mathrm{e}}\left(8\pi \right)+\sum _{k=1}^{\frac{n}{2}}\left({log}_{\mathrm{e}}(2k-1)-\frac{n+1}{2k-1}\right),\hfill & neven;\hfill \\ \frac{n-1}{2}{log}_{\mathrm{e}}\left(4\pi \right)+\sum _{k=1}^{\frac{n-1}{2}}\left({log}_{\mathrm{e}}k-\frac{n+1}{2k}\right),\hfill & nodd.\hfill \end{array}\right.\hfill \end{array}$$ (338)
• 116.
  The shared information of n random variables is a generalization of mutual information introduced in [54] for deriving the fundamental limit of interactive data exchange among agents who have access to the individual components and establish a dialog to ensure that all of them find out the value of the random vector. The shared information of $X^n$ is defined as

  $$\begin{array}{cc}\hfill S\left(X^n\right)& =\underset{\mathsf{\Pi }}{min}\frac{1}{\left|\mathsf{\Pi }\right|-1}D\left(P_{X^n}\parallel \prod _{\ell =1}^{\left|\mathsf{\Pi }\right|}{P}_{X\left({\mathcal{I}}_{\ell }\right)}\right),\hfill \end{array}$$ (339)

  where $X\left(\mathcal{J}\right)=\{X_i,i\in \mathcal{J}\}$, with $\mathcal{J}\subset \mathcal{I}=\{1,\dots ,n\}$, and the minimum is over all partitions of $\mathcal{I}$:

  $$\mathsf{\Pi }=\{{\mathcal{I}}_{\ell }\ne ⌀,\ell =1,\dots ,\left|\mathsf{\Pi }\right|\},with{\cup }_{\ell =1}^{\left|\mathsf{\Pi }\right|}{\mathcal{I}}_{\ell }=\mathcal{I},{\mathcal{I}}_{\ell }\cap {\mathcal{I}}_j=⌀,\ell \ne j,$$

  such that $\left|\mathsf{\Pi }\right|>1$. If we divide (338) by $n-1$, we obtain the shared information of n random variables distributed according to the standard spherical multivariate Cauchy model. This is a consequence of the following result, which is of independent interest.

  Theorem 18.

  If $X^n$ are exchangeable random variables, any subset of which have finite differential entropy, then for any partition Π of $\{1,\dots ,n\}$,

  $$\begin{array}{c}\hfill \frac{1}{\left|\mathsf{\Pi }\right|-1}D\left(P_{X^n}\parallel \prod _{\ell =1}^{\left|\mathsf{\Pi }\right|}{P}_{X\left({\mathcal{I}}_{\ell }\right)}\right)\ge \frac{1}{n-1}D(P_{X^n}\parallel P_{X_1}\times \cdots \times P_{X_n}).\end{array}$$ (340)

  Proof. 

  Fix any partition $\mathsf{\Pi }$ with $\left|\mathsf{\Pi }\right|=L\in \{2,\dots ,n-1\}$ chunks. Denote by $n_{\ell }$ the number of chunks in $\mathsf{\Pi }$ with cardinality $\ell \in \{1,\dots ,n-1\}$. Therefore,

  $$\begin{array}{c}\hfill \sum _{\ell =1}^{n-1}{n}_{\ell }=L,and\sum _{\ell =1}^{n-1}\ell n_{\ell }=n.\end{array}$$ (341)

  By exchangeability, any chunk of cardinality k has the same differential entropy, which we denote by $h_k$. Then,

  $$\begin{array}{c}\hfill D\left(P_{X^n}\parallel \prod _{\ell =1}^{\left|\mathsf{\Pi }\right|}{P}_{X\left({\mathcal{I}}_{\ell }\right)}\right)=-h_n+\sum _{\ell =1}^{n-1}{n}_{\ell }{h}_{\ell },\end{array}$$ (342)

  and the difference of the left minus the right sides of (340) multiplied by $(n-1)(L-1)$ is readily seen to equal

  $$\begin{array}{cc}\hfill & -(n-1)h_n+(n-1)\sum _{\ell =1}^{n-1}{n}_{\ell }{h}_{\ell }+(L-1)h_n-(L-1)n{h}_1\hfill \end{array}$$

  $$\begin{array}{cc}\hfill & =\left((n-1)n_1-n(L-1)\right)h_1+(L-n)h_n+(n-1)\sum _{\ell =2}^{n-1}{n}_{\ell }{h}_{\ell }\hfill \end{array}$$ (343)

  $$\begin{array}{cc}\hfill & \ge \left((n-1)n_1-n(L-1)+\sum _{\ell =2}^{n-1}(n-\ell )n_{\ell }\right)h_1+\left(L-n+\sum _{\ell =2}^{n-1}(\ell -1)n_{\ell }\right)h_n\hfill \end{array}$$ (344)

  $$\begin{array}{cc}\hfill & =0\hfill \end{array}$$ (345)

  where

  • (344) ⟸ for all $\ell \in \{2,\dots ,n-1\}$,

    $$\begin{array}{c}\hfill h_{\ell }\ge \frac{\ell -1}{n-1}{h}_n+\frac{n-\ell }{n-1}{h}_1,\end{array}$$ (346)

    since $h_1,\dots ,h_n$ is a concave sequence, i.e., $2h_k\ge h_{k-1}+h_{k+1}$ as a result of the sub-modularity of differential entropy.
  • (345) ⟸ (341).
       □

  Naturally, the same proof applies to n discrete exchangeable random variables with finite joint entropy.

17. Outlook

• 117.
  We have seen that a number of key information theoretic properties pertaining to the Gaussian law are also satisfied in the Cauchy case. Conceptually, those extensions shed light on the underlying reason the conventional Gaussian results hold. Naturally, we would like to explore how far beyond the Cauchy law those results can be expanded. As far as the maximization of differential entropy is concerned, the essential step is to redefine strength tailoring it to the desired law: Fix a reference random variable W with probability density function $f_W$ and finite differential entropy $h\left(W\right)\in \mathbb{R}$, and define the W-strength of a real valued random variable Z as

  $$\begin{array}{cc}\hfill {\varsigma }_W\left(Z\right)& =inf\left\{\varsigma >0:-\mathbb{E}\left[log{f}_W\left(\frac{Z}{\varsigma }\right)\right]\le h\left(W\right)\right\}.\hfill \end{array}$$ (347)

  For example,

  • (a)
    For $\alpha >0$, ${\varsigma }_W\left(\alpha W\right)=\alpha$;
  • (b)
    if W is standard normal, then ${\varsigma }_W^2\left(Z\right)=\mathbb{E}\left[Z^2\right]$;
  • (c)
    if V is standard Cauchy, then ${\varsigma }_V\left(Z\right)=\varsigma \left(Z\right)$;
  • (d)
    if W is standard exponential, then ${\varsigma }_W\left(Z\right)=\mathbb{E}\left[Z\right]$ if $Z\ge 0$ a.s., otherwise, ${\varsigma }_W\left(Z\right)=\infty$;
  • (e)
    if W is standard $(\mu =1)$ Subbotin (108) with $p>0$, then, ${\varsigma }_W^p\left(Z\right)={\mathbb{E}\left[\right|Z|}^p]$;
  • (f)
    if W has the Rider distribution in (9), then ${\varsigma }_W\left(Z\right)={\varsigma }_{\rho ,\theta }\left(Z\right)$ defined in (126) for $\theta$ chosen as in (110);
  • (g)
    if W is uniformly distributed on $[-1,1]$, ${\varsigma }_W\left(Z\right)=esssup\left|Z\right|$;
  • (h)
    if W is standard Rayleigh, then ${\varsigma }_W\left(Z\right)=inf\left\{\varsigma >0:\mathbb{E}\left[\frac{Z^2}{{\varsigma }^2}-{log}_{\mathrm{e}}\frac{Z^2}{2{\varsigma }^2}\right]\le 2+\gamma \right\}$ if $Z\ge 0$ a.s., otherwise, ${\varsigma }_W\left(Z\right)=\infty$.

  The pivotal Theorems 3 and 4 admit the following generalization.

  Theorem 19.

  Suppose $h\left(W\right)\in \mathbb{R}$ and $\varsigma >0$. Then,

  $$\begin{array}{c}\hfill \underset{Z:{\varsigma }_W\left(Z\right)\le \varsigma }{max}h\left(Z\right)=h\left(W\right)+log\varsigma .\end{array}$$ (348)

  Proof. 

  Fix any Z in the feasible set. For any $\sigma \ge {\varsigma }_W\left(Z\right)$ such that $-\mathbb{E}\left[log{f}_W\left(\frac{Z}{\sigma }\right)\right]\le h\left(W\right)$, we have

  $$\begin{array}{cc}\hfill 0\le D({\sigma }^{-1}Z\parallel W)& =-h\left(Z\right)+log\sigma -\mathbb{E}\left[log{f}_W\left(\frac{Z}{\sigma }\right)\right]\hfill \end{array}$$ (349)

  $$\begin{array}{cc}\hfill & \le -h\left(Z\right)+log\sigma +h\left(W\right).\hfill \end{array}$$ (350)

  Therefore, $h\left(Z\right)\le h\left(W\right)+log{\varsigma }_W\left(Z\right)$, by definition of ${\varsigma }_W\left(Z\right)$, thereby establishing ≤ in (348). Equality holds since ${\varsigma }_W\left(\varsigma W\right)=\varsigma$.    □

  A corollary to Theorem 19 is a very general form of the Shannon lower bound for the rate–distortion function of a memoryless source Z such that the distortion is constrained to have W-strength not higher than D, namely,

  $$\begin{array}{c}\hfill R\left(D\right)\ge h\left(Z\right)-h\left(W\right)-logD.\end{array}$$ (351)
  Theorem 19 finds an immediate extension to the multivariate case

  $$\begin{array}{c}\hfill \underset{Z^n:{\varsigma }_{W^n}\left(Z^n\right)\le \varsigma }{max}h\left(Z^n\right)=h\left(W^n\right)+nlog\varsigma ,\end{array}$$ (352)

  where, for $W^n$ with $h\left(W^n\right)\in \mathbb{R}$, we have defined

  $$\begin{array}{cc}\hfill {\varsigma }_{W^n}\left(Z^n\right)& =inf\left\{\varsigma >0:-\mathbb{E}\left[log{f}_{W^n}\left({\varsigma }^{-1}{Z}^n\right)\right]\le h\left(W^n\right)\right\}.\hfill \end{array}$$ (353)

  For example, if $W^n$ is zero-mean multivariate Gaussian with positive definite covariance $\Sigma$, then ${\varsigma }_{W^n}^2\left(Z^n\right)=\frac{1}{n}\mathbb{E}\left[Z^{n\top }{\Sigma }^{-1}{Z}^n\right]$.

• 118.One aspect in which we have shown that Cauchy distributions lend themselves to simplification unavailable in the Gaussian case is the single-parametrization of their likelihood ratio, which paves the way for a slew of closed-form expressions for f-divergences and Rényi divergences. It would be interesting to identify other multiparameter (even just scale/location) families of distributions that enjoy the same property. To that end, it is natural, though by no means hopeful, to study various generalizations of the Cauchy distribution such as the Student-t random variable, or more generally, the Rider distribution in (9). The information theoretic study of general stable distributions is hampered by the fact that they are characterized by their characteristic functions (e.g., p. 164 in [55]), which so far, have not lent themselves to the determination of relative entropy or even differential entropy.
• 119.Although we cannot expect that the cornucopia of information theoretic results in the Gaussian case can be extended to other domains, we have been able to show that a number of those results do find counterparts in the Cauchy case. Nevertheless, much remains to be explored. To name a few,
  • (a)
    The concavity of the entropy-strength $N_{\mathtt{C}}(X+tV)$—a counterpart of Costa’s entropy power inequality [40] would guarantee the least favorability of Cauchy noise among all strength-constrained noises as well as the entropy strength inequality

    $$\begin{array}{c}\hfill N_{\mathtt{C}}(X+tV)\ge N_{\mathtt{C}}\left(tV\right)+N_{\mathtt{C}}\left(X\right).\end{array}$$ (354)
  • (b)Information theoretic analyses quantifying the approach to normality in the central limit theorem are well-known (e.g., [56,57,58]). It would be interesting to explore the decrease in the relative entropy (relative to the Cauchy law) of independent sums distributed according to a law in the domain of attraction of the Cauchy distribution [55].
  • (c)Since de Bruijn’s identity is one of the ancestors of the i-mmse formula of [59], and we now have a counterpart of de Bruijn’s identity for convolutions with scaled Cauchy, it is natural to wonder if there may be some sort of integral representation of the mutual information between a random variable and its noisy version contaminated by additive Cauchy noise. In this respect, note that counterparts for the i-mmse formula for models other than additive Gaussian noise have been found in [60,61,62].
  • (d)Mutual information is robust against the addition of small non-Gaussian contamination in the sense that its effects are the same as if it were Gaussian [63]. The proof methods rely on Taylor series expansions that require the existence of moments. Any Cauchy counterparts (recall Item 77) would require substantially different methods.
  • (e)Pinsker [41] showed that Gaussian processes are information stable imposing only very mild assumptions. The key is that, modulo a factor, the variance of the information density is upper bounded by its mean, the mutual information. Does the spherical multivariate Cauchy distribution enjoy similar properties?
• 112.Although not surveyed here, there are indeed a number of results in the engineering literature advocating Cauchy models in certain heavy-tailed infinite-variance scenarios (see, e.g., [45] and the references therein.) At the end, either we abide by the information theoretic maxim that “there is nothing more practical than a beautiful formula”, or we pay heed to Poisson, who after pointing out in [64] that Laplace’s proof of the central limit theorem broke down for what we now refer to as the Cauchy law, remarked that “Mais nous ne tiendrons pas compte de ce cas particulier, quil nous suffira d’avoir remarqué à cause de sa singularité, et qui ne se recontre sans doute pas dans la pratique”.

Appendix A. Definite Integrals

$$\begin{array}{cc}\hfill {\int }_0^x\frac{1}{1+t^2}\mathrm{d}t& =arctan\left(x\right),\hfill \end{array}$$ (A1)

$$\begin{array}{cc}\hfill {\int }_{-\frac{1}{2}}^{\frac{1}{2}}logcos\left(\pi t\right)\mathrm{d}t& =log\frac{1}{2},\hfill \end{array}$$ (A2)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{log(1+t^2)}{1+t^2}\mathrm{d}t& =\pi log4,\hfill \end{array}$$ (A3)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{log({\alpha }^2-2\alpha tcos\beta +t^2)}{1+t^2}\mathrm{d}t& =\pi log(1+{\alpha }^2+2\alpha |sin\beta \left|\right),\hfill \end{array}$$ (A4)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{log(1+t^2)}{1+{(\xi t-\kappa )}^2}\mathrm{d}t& =\frac{\pi }{\xi }\left(log\left({\kappa }^2+{\left(\xi +1\right)}^2\right)-2log\xi \right),\xi >0,\hfill \end{array}$$ (A5)

$$\begin{array}{cc}\hfill {\kappa }_{\beta ,\rho }{\int }_{-\infty }^{\infty }\frac{{log}_{\mathrm{e}}{(1+|t|}^{\rho })}{{(1+|t|}^{\rho }{)}^{\beta }}\mathrm{d}t& =\psi \left(\beta \right)-\psi \left(\beta -\frac{1}{\rho }\right),\beta \rho >1,\hfill \end{array}$$ (A6)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{{log}_{\mathrm{e}}\left(1+{\theta }^2{t}^2\right)}{{\left(1+t^2\right)}^2}& =\pi \left({log}_{\mathrm{e}}(1+|\theta \left|\right)-\frac{\left|\theta \right|}{1+\left|\theta \right|}\right),\hfill \end{array}$$ (A7)

$$\begin{array}{cc}\hfill {\int }_{-\alpha }^{\alpha }{log}_{\mathrm{e}}(t^2+{\varsigma }^2)\mathrm{d}t& =4\varsigma arctan\left(\frac{\alpha }{\varsigma }\right)-4\alpha +2\alpha {log}_{\mathrm{e}}\left({\alpha }^2+{\varsigma }^2\right),\hfill \end{array}$$ (A8)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{t^2}{{(1+t^2)}^2}\mathrm{d}t& =\frac{\pi }{2},\hfill \end{array}$$ (A9)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{1}{{(1+t^2)}^2}\mathrm{d}t& =\frac{\pi }{2},\hfill \end{array}$$ (A10)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{t^2}{{(1+t^2)}^3}\mathrm{d}t& =\frac{\pi }{8},\hfill \end{array}$$ (A11)

$$\begin{array}{cc}\hfill {\int }_{-\infty }^{\infty }\frac{1}{{({\beta }^2+t^2)}^{\nu }}\mathrm{d}t& =\sqrt{\pi }{\beta }^{1-2\nu }\frac{\mathrm{\Gamma }\left(\nu -\frac{1}{2}\right)}{\mathrm{\Gamma }\left(\nu \right)},\nu >\frac{1}{2},\hfill \end{array}$$ (A12)

$$\begin{array}{cc}\hfill {\int }_0^{\infty }\frac{1}{{(1+t^{\rho })}^{\nu }}\mathrm{d}t& =\frac{\mathrm{\Gamma }\left(\nu -\frac{1}{\rho }\right)\mathrm{\Gamma }\left(1+\frac{1}{\rho }\right)}{\mathrm{\Gamma }\left(\nu \right)},\nu >\frac{1}{\rho }>0,\hfill \end{array}$$ (A13)

$$\begin{array}{cc}\hfill {\int }_0^{\pi }log\left(\alpha +\beta cos\theta \right)\mathrm{d}\theta & =\pi log\left(\frac{\alpha }{2}+\frac{1}{2}\sqrt{{\alpha }^2-{\beta }^2}\right),\alpha \ge \left|\beta \right|>0,\hfill \end{array}$$ (A14)

$$\begin{array}{cc}\hfill {\int }_0^{\pi }log{\left(\beta +\sqrt{{\beta }^2-1}cos\theta \right)}^{\alpha }\mathrm{d}\theta & =\pi {\mathrm{P}}_{\alpha }\left(\beta \right),\beta >0,\hfill \end{array}$$ (A15)

$$\begin{array}{cc}\hfill {\int }_0^{\infty }\frac{\mathrm{d}t}{\sqrt{1+t^2}\sqrt{{\beta }^2+t^2}}& =\mathbf{K}\left(\sqrt{1-{\beta }^2}\right),\beta \in (0,1),\hfill \end{array}$$ (A16)

where

• (A2) is a special case of 4.384.21 in [24];

• (A3) is a special case of (A4);

• (A4) is 4.296.2 in [24];

• (A5) follows from (A4) by change of variable;

• (A6), with ${\kappa }_{\beta ,\rho }$ defined in (10) and $\psi (·)$ denoting the digamma function, follows from 4.256 in [24] by change of variable $x={(1+t^p)}^{-\frac{1}{2n}}$ and $n=mp$;

• (A7) is a special case of 4.295.25 in [24];

• (A8) follows from 2.733.1 in [24];

• (A9)–(A10) follow from 3.252.6 in [24];

• (A11) can be obtained by integration by parts and (A10);

• (A12), with $\mathrm{\Gamma }(·)$ denoting the gamma function, is a special case of 3.251.11 in [24];

• (A13) can be obtained from 3.251.11 in [24] by change of variable;

• (A14) is 4.224.9 in [24];

• (A15) is 8.822.1 in [24] with ${\mathrm{P}}_{\alpha }\left(x\right)$ the Legendre function of the first kind, which is a solution to

  $$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}x}\left(\left(1-x^2\right)\frac{\mathrm{d}u\left(x\right)}{\mathrm{d}x}\right)+\alpha (\alpha +1)u\left(x\right)=0;\end{array}$$ (A17)
• (A16) is a special case of 3.152.1 in [24] with the complete elliptic integral of the first kind defined as 8.112.1 in [24], namely,

  $$\begin{array}{c}\hfill \mathbf{K}\left(k\right)={\int }_0^{\frac{\pi }{2}}\frac{\mathrm{d}\alpha }{\sqrt{1-k^2{sin}^2\alpha }},\left|k\right|<1.\end{array}$$ (A18)
  Note that mathematica defines the complete elliptic integral function EllipticK such that

  $$\begin{array}{c}\hfill \mathbf{K}\left(k\right)=\frac{\mathtt{EllipticK}\left(\frac{-k^2}{1-k^2}\right)}{\sqrt{1-k^2}},\left|k\right|<1.\end{array}$$ (A19)

Institutional Review Board Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The author declares no conflict of interest.

Funding Statement

This research received no external funding.