On f-Divergences: Integral Representations, Local Behavior, and Inequalities

Abstract

This paper is focused on f-divergences, consisting of three main contributions. The first one introduces integral representations of a general f-divergence by means of the relative information spectrum. The second part provides a new approach for the derivation of f-divergence inequalities, and it exemplifies their utility in the setup of Bayesian binary hypothesis testing. The last part of this paper further studies the local behavior of f-divergences.

1. Introduction

Probability theory, information theory, learning theory, statistical signal processing and other related disciplines, greatly benefit from non-negative measures of dissimilarity (a.k.a. divergence measures) between pairs of probability measures defined on the same measurable space (see, e.g., [1,2,3,4,5,6,7]). An axiomatic characterization of information measures, including divergence measures, was provided by Csiszár [8]. Many useful divergence measures belong to the set of f-divergences, independently introduced by Ali and Silvey [9], Csiszár [10,11,12,13], and Morimoto [14] in the early sixties. The family of f-divergences generalizes the relative entropy (a.k.a. the Kullback- Leibler divergence) while also satisfying the data processing inequality among other pleasing properties (see, e.g., [3] and references therein).

Integral representations of f-divergences serve to study properties of these information measures, and they are also used to establish relations among these divergences. An integral representation of f-divergences, expressed by means of the DeGroot statistical information, was provided in [3] with a simplified proof in [15]. The importance of this integral representation stems from the operational meaning of the DeGroot statistical information [16], which is strongly linked to Bayesian binary hypothesis testing. Some earlier specialized versions of this integral representation were introduced in [17,18,19,20,21], and a variation of it also appears in [22] Section 5.B. Implications of the integral representation of f-divergences, by means of the DeGroot statistical information, include an alternative proof of the data processing inequality, and a study of conditions for the sufficiency or $\epsilon$-deficiency of observation channels [3,15].

Since many distance measures of interest fall under the paradigm of an f-divergence [23], bounds among f-divergences are very useful in many instances such as the analysis of rates of convergence and concentration of measure bounds, hypothesis testing, testing goodness of fit, minimax risk in estimation and modeling, strong data processing inequalities and contraction coefficients, etc. Earlier studies developed systematic approaches to obtain f-divergence inequalities while dealing with pairs of probability measures defined on arbitrary alphabets. A list of some notable existing f-divergence inequalities is provided, e.g., in [22] Section 1 and [23] Section 3. State-of-the-art techniques which serve to derive bounds among f-divergences include:

• (1)Moment inequalities which rely on log-convexity arguments ([22] Section 5.D, [24,25,26,27,28]);
• (2)Inequalities which rely on a characterization of the exact locus of the joint range of f-divergences [29];
• (3)f-divergence inequalities via functional domination ([22] Section 3, [30,31,32]);
• (4)Sharp f-divergence inequalities by using numerical tools for maximizing or minimizing an f-divergence subject to a finite number of constraints on other f-divergences [33];
• (5)Inequalities which rely on powers of f-divergences defining a distance [34,35,36,37];
• (6)Vajda and Pinsker-type inequalities for f-divergences ([4,10,13,22] Sections 6–7, [38,39]);
• (7)Bounds among f-divergences when the relative information is bounded ([22] Sections 4–5, [40,41,42,43,44,45,46,47]), and reverse Pinsker inequalities ([22] Section 6, [40,48]);
• (8)Inequalities which rely on the minimum of an f-divergence for a given total variation distance and related bounds [4,33,37,38,49,50,51,52,53];
• (9)Bounds among f-divergences (or functions of f-divergences such as the Rényi divergence) via integral representations of these divergence measures [22] Section 8;
• (10)Inequalities which rely on variational representations of f-divergences (e.g., [54] Section 2).

Following earlier studies of the local behavior of f-divergences and their asymptotic properties (see related results by Csiszár and Shields [55] Theorem 4.1, Pardo and Vajda [56] Section 3, and Sason and Vérdu [22] Section 3.F), it is known that the local behavior of f-divergences scales, such as the chi-square divergence (up to a scaling factor which depends on f) provided that the first distribution approaches the reference measure in a certain strong sense. The study of the local behavior of f-divergences is an important aspect of their properties, and we further study it in this work.

This paper considers properties of f-divergences, while first introducing in Section 2 the basic definitions and notation needed, and in particular the various measures of dissimilarity between probability measures used throughout this paper. The presentation of our new results is then structured as follows:

Section 3 is focused on the derivation of new integral representations of f-divergences, expressed as a function of the relative information spectrum of the pair of probability measures, and the convex function f. The novelty of Section 3 is in the unified approach which leads to integral representations of f-divergences by means of the relative information spectrum, where the latter cumulative distribution function plays an important role in information theory and statistical decision theory (see, e.g., [7,54]). Particular integral representations of the type of results introduced in Section 3 have been recently derived by Sason and Verdú in a case-by-case basis for some f-divergences (see [22] Theorems 13 and 32), while lacking the approach which is developed in Section 3 for general f-divergences. In essence, an f-divergence $D_f(P\parallel Q)$ is expressed in Section 3 as an inner product of a simple function of the relative information spectrum (depending only on the probability measures P and Q), and a non-negative weight function ${\omega }_f:(0,\infty )\mapsto [0,\infty )$ which only depends on f. This kind of representation, followed by a generalized result, serves to provide new integral representations of various useful f-divergences. This also enables in Section 3 to characterize the interplay between the DeGroot statistical information (or between another useful family of f-divergence, named the $E_{\gamma }$ divergence with $\gamma \ge 1$) and the relative information spectrum.

Section 4 provides a new approach for the derivation of f-divergence inequalities, where an arbitrary f-divergence is lower bounded by means of the $E_{\gamma }$ divergence [57] or the DeGroot statistical information [16]. The approach used in Section 4 yields several generalizations of the Bretagnole-Huber inequality [58], which provides a closed-form and simple upper bound on the total variation distance as a function of the relative entropy; the Bretagnole-Huber inequality has been proved to be useful, e.g., in the context of lower bounding the minimax risk in non-parametric estimation (see, e.g., [5] pp. 89–90, 94), and in the problem of density estimation (see, e.g., [6] Section 1.6). Although Vajda’s tight lower bound in [59] is slightly tighter everywhere than the Bretagnole-Huber inequality, our motivation for the generalization of the latter bound is justified later in this paper. The utility of the new inequalities is exemplified in the setup of Bayesian binary hypothesis testing.

Section 5 finally derives new results on the local behavior of f-divergences, i.e., the characterization of their scaling when the pair of probability measures are sufficiently close to each other. The starting point of our analysis in Section 5 relies on the analysis in [56] Section 3, regarding the asymptotic properties of f-divergences.

The reading of Section 3, Section 4 and Section 5 can be done in any order since the analysis in these sections is independent.

2. Preliminaries and Notation

We assume throughout that the probability measures P and Q are defined on a common measurable space $(\mathcal{A},\mathcal{F})$, and $P\ll Q$ denotes that P is absolutely continuous with respect to Q, namely there is no event $\mathcal{F}\in \mathcal{F}$ such that $P(\mathcal{F})>0=Q(\mathcal{F})$.

Definition 1.

The relative information provided by$a\in \mathcal{A}$according to$(P,Q)$, where$P\ll Q$, is given by

$$\begin{array}{c}\hfill {ı}_{P\parallel Q}(a):=log\frac{\mathrm{d}P}{\mathrm{d}Q}(a).\end{array}$$ (1)

More generally, even if$P\neg \ll Q$, let R be an arbitrary dominating probability measure such that$P,Q\ll R$(e.g.,$R=\frac{1}{2}(P+Q)$); irrespectively of the choice of R, the relative information is defined to be

$$\begin{array}{c}\hfill {ı}_{P\parallel Q}(a):={ı}_{P\parallel R}(a)-{ı}_{Q\parallel R}(a),a\in \mathcal{A}.\end{array}$$ (2)

The following asymmetry property follows from (2):

$$\begin{array}{c}\hfill {ı}_{P\parallel Q}=-{ı}_{Q\parallel P}.\end{array}$$ (3)

Definition 2.

The relative information spectrum is the cumulative distribution function

$$\begin{array}{c}\hfill {\mathbb{F}}_{P\parallel Q}(x)=\mathbb{P}\left[{ı}_{P\parallel Q}(X)\le x\right],x\in \mathbb{R},X\sim P.\end{array}$$ (4)

The relative entropy is the expected valued of the relative information when it is distributed according to P:

$$\begin{array}{c}\hfill D(P\parallel Q):=\mathbb{E}\left[{ı}_{P\parallel Q}(X)\right],X\sim P.\end{array}$$ (5)

Throughout this paper, $\mathcal{C}$ denotes the set of convex functions $f:(0,\infty )\mapsto \mathbb{R}$ with $f(1)=0$. Hence, the function $f\equiv 0$ is in $\mathcal{C}$; if $f\in \mathcal{C}$, then $af\in \mathcal{C}$ for all $a>0$; and if $f,g\in \mathcal{C}$, then $f+g\in \mathcal{C}$. We next provide a general definition for the family of f-divergences (see [3] p. 4398).

Definition 3

(f-divergence [9,10,12]). Let P and Q be probability measures, let μ be a dominating measure of P and Q (i.e., $P,Q\ll \mu$; e.g., $\mu =P+Q$), and let $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q:=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$. The f-divergence from P to Q is given, independently of μ, by

$$\begin{array}{c}\hfill D_f(P\parallel Q):=\int qf\left(\frac{p}{q}\right)\mathrm{d}\mu ,\end{array}$$ (6)

where

$$\begin{array}{c}f(0):=\underset{t\downarrow 0}{lim}f(t),\hfill \end{array}$$ (7)

$$\begin{array}{c}0f\left(\frac{0}{0}\right):=0,\hfill \end{array}$$ (8)

$$\begin{array}{c}0f\left(\frac{a}{0}\right):=\underset{t\downarrow 0}{lim}tf\left(\frac{a}{t}\right)=a\underset{u\to \infty }{lim}\frac{f(u)}{u},a>0.\hfill \end{array}$$ (9)

We rely in this paper on the following properties of f-divergences:

Proposition 1.

Let$f,g\in \mathcal{C}$. The following conditions are equivalent:

• (1)
  $$\begin{array}{c}\hfill D_f(P\parallel Q)=D_g(P\parallel Q),\forall P,Q;\end{array}$$ (10)
• (2)
  there exists a constant$c\in \mathbb{R}$such that

  $$\begin{array}{c}\hfill f(t)-g(t)=c(t-1),\forall t\in (0,\infty ).\end{array}$$ (11)

Proposition 2.

Let$f\in \mathcal{C}$, and let$f^{\ast }:(0,\infty )\mapsto \mathbb{R}$be the conjugate function, given by

$$\begin{array}{c}\hfill f^{\ast }(t)=tf\left(\frac{1}{t}\right)\end{array}$$ (12)

for$t>0$. Then,$f^{\ast }\in \mathcal{C}$; $f^{\ast \ast }=f$, and for every pair of probability measures$(P,Q)$,

$$\begin{array}{c}\hfill D_f(P\parallel Q)=D_{f^{\ast }}(Q\parallel P).\end{array}$$ (13)

By an analytic extension of $f^{\ast }$ in (12) at $t=0$, let

$$\begin{array}{c}\hfill f^{\ast }(0):=\underset{t\downarrow 0}{lim}{f}^{\ast }(t)=\underset{u\to \infty }{lim}\frac{f(u)}{u}.\end{array}$$ (14)

Note that the convexity of $f^{\ast }$ implies that $f^{\ast }(0)\in (-\infty ,\infty ]$. In continuation to Definition 3, we get

$$\begin{array}{}\mathrm{(15)}& \hfill D_f(P\parallel Q)& =\int qf\left(\frac{p}{q}\right)\mathrm{d}\mu \hfill \mathrm{(16)}& & ={\int }_{\{pq>0\}}qf\left(\frac{p}{q}\right)\mathrm{d}\mu +Q(p=0)f(0)+P(q=0)f^{\ast }(0)\hfill \end{array}$$

with the convention in (16) that $0·\infty =0$, We refer in this paper to the following f-divergences:

• (1)
  Relative entropy:

  $$\begin{array}{cc}\hfill D(P\parallel Q)& =D_f(P\parallel Q).\hfill \end{array}$$ (17)

  with

  $$\begin{array}{c}\hfill f(t)=tlogt,t>0.\end{array}$$ (18)
• (2)
  Jeffrey’s divergence [60]:

  $$\begin{array}{cc}\hfill J(P\parallel Q)& :=D(P\parallel Q)+D(Q\parallel P)\hfill \end{array}$$ (19)

  $$\begin{array}{cc}\hfill & =D_f(P\parallel Q)\hfill \end{array}$$ (20)

  with

  $$\begin{array}{c}\hfill f(t)=(t-1)logt,t>0.\end{array}$$ (21)
• (3)
  Hellinger divergence of order$\alpha \in (0,1)\cup (1,\infty )$ [2] Definition 2.10:

  $$\begin{array}{c}\hfill {\mathcal{H}}_{\alpha }(P\parallel Q)=D_{f_{\alpha }}(P\parallel Q)\end{array}$$ (22)

  with

  $$\begin{array}{c}\hfill f_{\alpha }(t)=\frac{t^{\alpha }-1}{\alpha -1},t>0.\end{array}$$ (23)
  Some of the significance of the Hellinger divergence stems from the following facts:

  • -
    The analytic extension of ${\mathcal{H}}_{\alpha }(P\parallel Q)$ at $\alpha =1$ yields

    $$\begin{array}{c}\hfill D(P\parallel Q)=H_1(P\parallel Q)loge.\end{array}$$ (24)
  • -
    The chi-squared divergence [61] is the second order Hellinger divergence (see, e.g., [62] p. 48), i.e.,

    $$\begin{array}{c}\hfill {\chi }^2(P\parallel Q)={\mathcal{H}}_2(P\parallel Q).\end{array}$$ (25)
    Note that, due to Proposition 1,

    $$\begin{array}{c}\hfill {\chi }^2(P\parallel Q)=D_f(P\parallel Q),\end{array}$$ (26)

    where $f:(0,\infty )\mapsto \mathbb{R}$ can be defined as

    $$\begin{array}{c}\hfill f(t)={(t-1)}^2,t>0.\end{array}$$ (27)
  • -
    The squared Hellinger distance (see, e.g., [62] p. 47), denoted by ${\mathcal{H}}^2(P\parallel Q)$, satisfies the identity

    $$\begin{array}{c}\hfill {\mathcal{H}}^2(P\parallel Q)=\frac{1}{2}{\mathcal{H}}_{\frac{1}{2}}(P\parallel Q).\end{array}$$ (28)
  • -
    The Bhattacharyya distance [63], denoted by $B(P\parallel Q)$, satisfies

    $$\begin{array}{c}\hfill B(P\parallel Q)=log\frac{1}{1-{\mathcal{H}}^2(P\parallel Q)}.\end{array}$$ (29)
  • -
    The Rényi divergence of order $\alpha \in (0,1)\cup (1,\infty )$ is a one-to-one transformation of the Hellinger divergence of the same order [11] (14):

    $$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)=\frac{1}{\alpha -1}log\left(1+(\alpha -1){\mathcal{H}}_{\alpha }(P\parallel Q)\right).\end{array}$$ (30)
  • -
    The Alpha-divergence of order $\alpha$, as it is defined in [64] and ([65] (4)), is a generalized relative entropy which (up to a scaling factor) is equal to the Hellinger divergence of the same order $\alpha$. More explicitly,

    $$\begin{array}{c}\hfill D_{\mathrm{A}}^{(\alpha )}(P\parallel Q)=\frac{1}{\alpha }{\mathcal{H}}_{\alpha }(P\parallel Q),\end{array}$$ (31)

    where $D_{\mathrm{A}}^{(\alpha )}(·\parallel ·)$ denotes the Alpha-divergence of order $\alpha$. Note, however, that the Beta and Gamma-divergences in [65], as well as the generalized divergences in [66,67], are not f-divergences in general.
• (4)
  ${\chi }^s$divergence for $s\ge 1$ [2] (2.31), and the total variation distance: The function

  $$\begin{array}{c}\hfill f_s(t)={|t-1|}^s,t>0\end{array}$$ (32)

  results in

  $$\begin{array}{cc}\hfill {\chi }^s(P\parallel Q)& =D_{f_s}(P\parallel Q).\hfill \end{array}$$ (33)
  Specifically, for $s=1$, let

  $$\begin{array}{c}\hfill f(t):=f_1(t)=|t-1|,t>0,\end{array}$$ (34)

  and the total variation distance is expressed as an f-divergence:

  $$\begin{array}{cc}\hfill |P-Q|& =D_f(P\parallel Q).\hfill \end{array}$$ (35)
• (5)
  Triangular Discrimination [39] (a.k.a. Vincze-Le Cam distance):

  $$\begin{array}{c}\hfill \Delta (P\parallel Q)=D_f(P\parallel Q)\end{array}$$ (36)

  with

  $$\begin{array}{c}\hfill f(t)=\frac{{(t-1)}^2}{t+1},t>0.\end{array}$$ (37)
  Note that

  $$\begin{array}{cc}\hfill \frac{1}{2}\Delta (P\parallel Q)& ={\chi }^2(P\parallel \frac{1}{2}P+\frac{1}{2}Q)={\chi }^2(Q\parallel \frac{1}{2}P+\frac{1}{2}Q).\hfill \end{array}$$ (38)
• (6)
  Lin’s measure [68] (4.1):

  $$\begin{array}{}\mathrm{(39)}& \hfill L_{\theta }(P\parallel Q)& :=H\left(\theta P+(1-\theta )Q\right)-\theta H(P)-(1-\theta )H(Q)\hfill (40)& & =\theta D\left(P\parallel \theta P+(1-\theta )Q\right)+(1-\theta )D\left(Q\parallel \theta P+(1-\theta )Q\right),\hfill \end{array}$$

  for $\theta \in [0,1]$. This measure can be expressed by the following f-divergence:

  $$\begin{array}{c}\hfill L_{\theta }(P\parallel Q)=D_{f_{\theta }}(P\parallel Q),\end{array}$$ (41)

  with

  $$\begin{array}{c}\hfill f_{\theta }(t):=\theta tlogt-\left(\theta t+1-\theta \right)log\left(\theta t+1-\theta \right),t>0.\end{array}$$ (42)
  The special case of (41) with $\theta =\frac{1}{2}$ gives the Jensen-Shannon divergence (a.k.a. capacitory discrimination):

  $$\begin{array}{}\mathrm{(43)}& \hfill \mathrm{JS}(P\parallel Q)& :=L_{\frac{1}{2}}(P\parallel Q)\hfill \mathrm{(44)}& & =\frac{1}{2}D\left(P\parallel \frac{1}{2}P+\frac{1}{2}Q\right)+\frac{1}{2}D\left(Q\parallel \frac{1}{2}P+\frac{1}{2}Q\right).\hfill \end{array}$$
• (7)
  $E_{\gamma }$divergence [57] p. 2314: For $\gamma \ge 1$,

  $$\begin{array}{}\mathrm{(45)}& \hfill E_{\gamma }(P\parallel Q)& :=\underset{\mathcal{U}\in \mathcal{F}}{max}\left(P(\mathcal{U})-\gamma Q(\mathcal{U})\right)\hfill \mathrm{(46)}& & =\mathbb{P}[{ı}_{P\parallel Q}(X)>log\gamma ]-\gamma \mathbb{P}[{ı}_{P\parallel Q}(Y)>log\gamma ]\hfill \end{array}$$

  with $X\sim P$ and $Y\sim Q$, and where (46) follows from the Neyman-Pearson lemma. The $E_{\gamma }$ divergence can be identified as an f-divergence:

  $$\begin{array}{c}\hfill E_{\gamma }(P\parallel Q)=D_{f_{\gamma }}(P\parallel Q)\end{array}$$ (47)

  with

  $$\begin{array}{c}\hfill f_{\gamma }(t):={(t-\gamma )}^{+},t>0\end{array}$$ (48)

  where ${(x)}^{+}:=max\{x,0\}$. The following relation to the total variation distance holds:

  $$\begin{array}{c}\hfill E_1(P\parallel Q)=\frac{1}{2}|P-Q|.\end{array}$$ (49)
• (8)
  DeGroot statistical information [3,16]: For $\omega \in (0,1)$,

  $$\begin{array}{c}\hfill {\mathcal{I}}_{\omega }(P\parallel Q)=D_{{\varphi }_{\omega }}(P\parallel Q)\end{array}$$ (50)

  with

  $$\begin{array}{c}\hfill {\varphi }_{\omega }(t)=min\{\omega ,1-\omega \}-min\{\omega t,1-\omega \},t>0.\end{array}$$ (51)
  The following relation to the total variation distance holds:

  $$\begin{array}{c}\hfill {\mathcal{I}}_{\frac{1}{2}}(P\parallel Q)=\frac{1}{4}|P-Q|,\end{array}$$ (52)

  and the DeGroot statistical information and the $E_{\gamma }$ divergence are related as follows [22] (384):

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{I}}_{\omega }(P\parallel Q)=\{\begin{array}{cc}\omega E_{\frac{1-\omega }{\omega }}(P\parallel Q),\hfill & \omega \in \left(0,\frac{1}{2}\right],\\ (1-\omega )E_{\frac{\omega }{1-\omega }}(Q\parallel P),\hfill & \omega \in (\frac{1}{2},1).\end{array}\end{array}\end{array}$$ (53)

3. New Integral Representations of $\mathit{f}$-Divergences

The main result in this section provides new integral representations of f-divergences as a function of the relative information spectrum (see Definition 2). The reader is referred to other integral representations (see [15] Section 2, [4] Section 5, [22] Section 5.B, and references therein), expressing a general f-divergence by means of the DeGroot statistical information or the $E_{\gamma }$ divergence.

Lemma 1.

Let$f\in \mathcal{C}$be a strictly convex function at 1. Let$g:\mathbb{R}\mapsto \mathbb{R}$be defined as

$$\begin{array}{c}\hfill g(x):=exp(-x)f\left(exp(x)\right)-f_{+}^{\prime }(1)\left(1-exp(-x)\right),x\in \mathbb{R}\end{array}$$ (54)

where$f_{+}^{\prime }(1)$denotes the right-hand derivative of f at 1 (due to the convexity of f on$(0,\infty )$, it exists and it is finite). Then, the function g is non-negative, it is strictly monotonically decreasing on$(-\infty ,0]$, and it is strictly monotonically increasing on$[0,\infty )$with$g(0)=0$.

Proof. 

For any function $u\in \mathcal{C}$, let $\tilde{u}\in \mathcal{C}$ be given by

$$\begin{array}{c}\hfill \tilde{u}(t)=u(t)-u_{+}^{\prime }(1)(t-1),t\in (0,\infty ),\end{array}$$ (55)

and let $u^{\ast }\in \mathcal{C}$ be the conjugate function, as given in (12). The function g in (54) can be expressed in the form

$$\begin{array}{c}\hfill g(x)={(\tilde{f})}^{\ast }\left(exp(-x)\right),x\in \mathbb{R},\end{array}$$ (56)

as it is next verified. For $t>0$, we get from (12) and (55),

$$\begin{array}{c}\hfill {(\tilde{f})}^{\ast }(t)=t\tilde{f}\left(\frac{1}{t}\right)=tf\left(\frac{1}{t}\right)+f_{+}^{\prime }(1)(t-1),\end{array}$$ (57)

and the substitution $t:=exp(-x)$ for $x\in \mathbb{R}$ yields (56) in view of (54).

By assumption, $f\in \mathcal{C}$ is strictly convex at 1, and therefore these properties are inherited to $\tilde{f}$. Since also $\tilde{f}(1)={\tilde{f}}^{\prime }(1)=0$, it follows from [3] Theorem 3 that both $\tilde{f}$ and ${\tilde{f}}^{\ast }$ are non-negative on $(0,\infty )$, and they are also strictly monotonically decreasing on $(0,1]$. Hence, from (12), it follows that the function ${(\tilde{f})}^{\ast }$ is strictly monotonically increasing on $[1,\infty )$. Finally, the claimed properties of the function g follow from (56), and in view of the fact that the function ${(\tilde{f})}^{\ast }$ is non-negative with ${(\tilde{f})}^{\ast }(1)=0$, strictly monotonically decreasing on $(0,1]$ and strictly monotonically increasing on $[1,\infty )$. ☐

Lemma 2.

Let$f\in \mathcal{C}$be a strictly convex function at 1, and let$g:\mathbb{R}\mapsto \mathbb{R}$be as in (54). Let

$$\begin{array}{cc}\hfill a& :=\underset{x\to \infty }{lim}g(x)\in (0,\infty ],\hfill \end{array}$$ (58)

$$\begin{array}{cc}\hfill b& :=\underset{x\to -\infty }{lim}g(x)\in (0,\infty ],\hfill \end{array}$$ (59)

and let ${\ell }_1:[0,a)\mapsto [0,\infty )$ and ${\ell }_2:[0,b)\mapsto (-\infty ,0]$ be the two inverse functions of g. Then,

$$\begin{array}{c}\hfill D_f(P\parallel Q)={\int }_0^a\left[1-{\mathbb{F}}_{P\parallel Q}\left({\ell }_1(t)\right)\right]\mathrm{d}t+{\int }_0^b{\mathbb{F}}_{P\parallel Q}\left({\ell }_2(t)\right)\mathrm{d}t.\end{array}$$ (60)

Proof. 

In view of Lemma 1, it follows that ${\ell }_1:[0,a)\mapsto [0,\infty )$ is strictly monotonically increasing and ${\ell }_2:[0,b)\mapsto (-\infty ,0]$ is strictly monotonically decreasing with ${\ell }_1(0)={\ell }_2(0)=0$.

Let $X\sim P$, and let $V:=exp\left({ı}_{P\parallel Q}(X)\right)$. Then, we have

$$\begin{array}{}\mathrm{(61)}& \hfill D_f(P\parallel Q)& =D_{\tilde{f}}(P\parallel Q)\hfill \mathrm{(62)}& & =D_{{(\tilde{f})}^{\ast }}(Q\parallel P)\hfill \mathrm{(63)}& & =\int {(\tilde{f})}^{\ast }\left(exp\left({ı}_{Q\parallel P}(x)\right)\right)\mathrm{d}P(x)\hfill \mathrm{(64)}& & =\int {(\tilde{f})}^{\ast }\left(exp\left(-{ı}_{P\parallel Q}(x)\right)\right)\mathrm{d}P(x)\hfill \mathrm{(65)}& & =\int g\left({ı}_{P\parallel Q}(x)\right)\mathrm{d}P(x)\hfill \mathrm{(66)}& & =\mathbb{E}\left[g(V)\right]\hfill \mathrm{(67)}& & ={\int }_0^{\infty }\mathbb{P}\left[g(V)>t\right]\mathrm{d}t\hfill \mathrm{(68)}& & ={\int }_0^a\mathbb{P}\left[V\ge 0,g(V)>t\right]\mathrm{d}t+{\int }_0^b\mathbb{P}\left[V<0,g(V)>t\right]\mathrm{d}t\hfill \mathrm{(69)}& & ={\int }_0^a\mathbb{P}\left[V>{\ell }_1(t)\right]\mathrm{d}t+{\int }_0^b\mathbb{P}\left[V\le {\ell }_2(t)\right]\mathrm{d}t\hfill \mathrm{(70)}& & ={\int }_0^a\left[1-{\mathbb{F}}_{P\parallel Q}\left({\ell }_1(t)\right)\right]\mathrm{d}t+{\int }_0^b{\mathbb{F}}_{P\parallel Q}\left({\ell }_2(t)\right)\mathrm{d}t\hfill \end{array}$$

where (61) relies on Proposition 1; (62) relies on Proposition 2; (64) follows from (3); (65) follows from (56); (66) holds by the definition of the random variable V; (67) holds since, in view of Lemma 1, $Z:=g(V)\ge 0$, and $\mathbb{E}\left[Z\right]={\int }_0^{\infty }\mathbb{P}[Z>t]\mathrm{d}t$ for any non-negative random variable Z; (68) holds in view of the monotonicity properties of g in Lemma 1, the definition of a and b in (58) and (59), and by expressing the event $\{g(V)>t\}$ as a union of two disjoint events; (69) holds again by the monotonicity properties of g in Lemma 1, and by the definition of its two inverse functions ${\ell }_1$ and ${\ell }_2$ as above; in (67)–(69) we are free to substitute > by ≥, and < by ≤; finally, (70) holds by the definition of the relative information spectrum in (4). ☐

Remark 1.

The function$g:\mathbb{R}\mapsto \mathbb{R}$in (54) is invariant to the mapping $f(t)\mapsto f(t)+c(t-1)$, for $t>0$, with an arbitrary $c\in \mathbb{R}$. This invariance of g (and, hence, also the invariance of its inverse functions ${\ell }_1$ and ${\ell }_2$) is well expected in view of Proposition 1 and Lemma 2.

Example 1.

For the chi-squared divergence in (26), letting f be as in (27), it follows from (54) that

$$\begin{array}{c}\hfill g(x)=4{sinh}^2\left(\frac{1}{2loge}x\right),x\in \mathbb{R},\end{array}$$ (71)

which yields, from (58) and (59), $a=b=\infty$. Calculation of the two inverse functions of g, as defined in Lemma 2, yields the following closed-form expression:

$$\begin{array}{cc}\hfill {\ell }_{1,2}(u)& =\pm 2log\left(\frac{u+\sqrt{u+4}}{2}\right),u\ge 0.\hfill \end{array}$$ (72)

Substituting (72) into (60) provides an integral representation of ${\chi }^2(P\parallel Q)$.

Lemma 3.

$$\begin{array}{c}\hfill {\int }_0^{\infty }\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta =1.\end{array}$$ (73)

Proof. 

Let $X\sim P$. Then, we have

$$\begin{array}{}\mathrm{(74)}& \hfill {\int }_0^{\infty }\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta & ={\int }_0^{\infty }\frac{1}{{\beta }^2}\mathbb{P}[{ı}_{P\parallel Q}(X)\le log\beta ]\mathrm{d}\beta \hfill \mathrm{(75)}& & ={\int }_0^{\infty }\frac{1}{{\beta }^2}\mathbb{P}\left[exp\left({ı}_{Q\parallel P}(X)\right)\ge \frac{1}{\beta }\right]\mathrm{d}\beta \hfill \mathrm{(76)}& & ={\int }_0^{\infty }\mathbb{P}\left[exp\left({ı}_{Q\parallel P}(X)\right)\ge u\right]\mathrm{d}u\hfill \mathrm{(77)}& & =\mathbb{E}\left[exp\left({ı}_{Q\parallel P}(X)\right)\right]\hfill \mathrm{(78)}& & =1,\hfill \end{array}$$

where (74) holds by (4); (75) follows from (3); (76) holds by the substitution $u:=\frac{1}{\beta }$; (77) holds since $exp\left({ı}_{Q\parallel P}(X)\right)\ge 0$, and finally (78) holds since $X\sim P$. ☐

Remark 2.

Unlike Example 1, in general, the inverse functions${\ell }_1$and${\ell }_2$in Lemma 2 are not expressible in closed form, motivating our next integral representation in Theorem 1.

The following theorem provides our main result in this section.

Theorem 1.

The following integral representations of an f-divergence, by means of the relative information spectrum, hold:

• (1) 
  Let

  • -
    $f\in \mathcal{C}$be differentiable on$(0,\infty )$;
  • -
    $w_f:(0,\infty )\mapsto [0,\infty )$be the non-negative weight function given, for$\beta >0$, by

    $$\begin{array}{cc}\hfill w_f(\beta )& :=\frac{1}{\beta }\left|f^{\prime }(\beta )-\frac{f(\beta )+f^{\prime }(1)}{\beta }\right|;\hfill \end{array}$$ (79)
  • -
    the function $G_{P\parallel Q}:(0,\infty )\mapsto [0,1]$ be given by

    $$G_{P\parallel Q}(\beta ):=\{\begin{array}{cc}1-{\mathbb{F}}_{P\parallel Q}(log\beta ),\hfill & \beta \in [1,\infty ),\hfill \\ {\mathbb{F}}_{P\parallel Q}(log\beta ),\hfill & \beta \in (0,1).\hfill \end{array}$$ (80)
  Then,

  $$\begin{array}{c}\hfill D_f(P\parallel Q)=\langle w_f,G_{P\parallel Q}\rangle ={\int }_0^{\infty }{w}_f(\beta )G_{P\parallel Q}(\beta )\mathrm{d}\beta .\end{array}$$ (81)
• (2) 
  More generally, for an arbitrary$c\in \mathbb{R}$, let${\tilde{w}}_{f,c}:(0,\infty )\mapsto \mathbb{R}$be a modified real-valued function defined as

  $$\begin{array}{c}\hfill {\tilde{w}}_{f,c}(\beta ):=w_f(\beta )+\frac{c}{{\beta }^2}\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right).\end{array}$$ (82)
  Then,

  $$\begin{array}{c}\hfill D_f(P\parallel Q)=\langle {\tilde{w}}_{f,c},G_{P\parallel Q}\rangle .\end{array}$$ (83)

Proof. 

We start by proving the special integral representation in (81), and then extend our proof to the general representation in (83).

• (1)
  We first assume an additional requirement that f is strictly convex at 1. In view of Lemma 2,

  $$\begin{array}{cc}\hfill & {\ell }_1\left(g(u)\right)=u,u\in [0,\infty ),\hfill \end{array}$$ (84)

  $$\begin{array}{cc}\hfill & {\ell }_2\left(g(u)\right)=u,u\in (-\infty ,0].\hfill \end{array}$$ (85)
  Since by assumption $f\in \mathcal{C}$ is differentiable on $(0,\infty )$ and strictly convex at 1, the function g in (54) is differentiable on $\mathbb{R}$. In view of (84) and (85), substituting $t:=g\left(log\beta \right)$ in (60) for $\beta >0$ implies that

  $$\begin{array}{c}\hfill D_f(P\parallel Q)={\int }_1^{\infty }\left[1-{\mathbb{F}}_{P\parallel Q}\left(log\beta \right)\right]{\overline{w}}_f(\beta )\mathrm{d}\beta -{\int }_0^1{\mathbb{F}}_{P\parallel Q}\left(log\beta \right){\overline{w}}_f(\beta )\mathrm{d}\beta ,\end{array}$$ (86)

  where ${\overline{w}}_f:(0,\infty )\mapsto \mathbb{R}$ is given by

  $$\begin{array}{}\mathrm{(87)}& \hfill {\overline{w}}_f(\beta )& :=\frac{g^{\prime }\left(log\beta \right)}{\beta }log\mathrm{e}\hfill \mathrm{(88)}& & =\frac{1}{\beta }\left[f^{\prime }(\beta )-\frac{f(\beta )+f^{\prime }(1)}{\beta }\right]\hfill \end{array}$$

  for $\beta >0$, where (88) follows from (54). Due to the monotonicity properties of g in Lemma 1, (87) implies that ${\overline{w}}_f(\beta )\ge 0$ for $\beta \ge 1$, and ${\overline{w}}_f(\beta )<0$ for $\beta \in (0,1)$. Hence, the weight function $w_f$ in (79) satisfies

  $$\begin{array}{c}\hfill w_f(\beta )=\left|{\overline{w}}_f(\beta )\right|={\overline{w}}_f(\beta )\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right),\beta >0.\end{array}$$ (89)

  The combination of (80), (86) and (89) gives the required result in (81).

  We now extend the result in (81) when $f\in \mathcal{C}$ is differentiable on $(0,\infty )$, but not necessarily strictly convex at 1. To that end, let $s:(0,\infty )\mapsto \mathbb{R}$ be defined as

  $$\begin{array}{c}\hfill s(t):=f(t)+(t^2-1),t>0.\end{array}$$ (90)
  This implies that $s\in \mathcal{C}$ is differentiable on $(0,\infty )$, and it is also strictly convex at 1. In view of the proof of (81) when f is strict convexity of f at 1, the application of this result to the function s in (90) yields

  $$\begin{array}{c}\hfill D_s(P\parallel Q)=\langle w_s,G_{P\parallel Q}\rangle .\end{array}$$ (91)
  In view of (6), (22), (23), (25) and (90),

  $$\begin{array}{c}\hfill D_s(P\parallel Q)=D_f(P\parallel Q)+{\chi }^2(P\parallel Q);\end{array}$$ (92)

  from (79), (89), (90) and the convexity and differentiability of $f\in \mathcal{C}$, it follows that the weight function $w_s\in (0,\infty )\mapsto [0,\infty )$ satisfies

  $$\begin{array}{c}\hfill w_s(\beta )=w_f(\beta )+\left(1-\frac{1}{{\beta }^2}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)\end{array}$$ (93)

  for $\beta >0$. Furthermore, by applying the result in (81) to the chi-squared divergence ${\chi }^2(P\parallel Q)$ in (25) whose corresponding function $f_2(t):=t^2-1$ for $t>0$ is strictly convex at 1, we obtain

  $$\begin{array}{c}\hfill {\chi }^2(P\parallel Q)={\int }_0^{\infty }\left(1-\frac{1}{{\beta }^2}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)G_{P\parallel Q}(\beta )\mathrm{d}\beta .\end{array}$$ (94)

  Finally, the combination of (91)–(94), yields $D_f(P\parallel Q)=\langle w_f,G_{P\parallel Q}\rangle$; this asserts that (81) also holds by relaxing the condition that f is strictly convex at 1.

• (2)
  In view of (80)–(82), in order to prove (83) for an arbitrary $c\in \mathbb{R}$, it is required to prove the identity

  $$\begin{array}{c}\hfill {\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}\left(log\beta \right)}{{\beta }^2}\mathrm{d}\beta ={\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}\left(log\beta \right)}{{\beta }^2}\mathrm{d}\beta .\end{array}$$ (95)

  Equality (95) can be verified by Lemma 3: by rearranging terms in (95), we get the identity in (73) (since ${\int }_1^{\infty }\frac{\mathrm{d}\beta }{{\beta }^2}=1$). ☐

Remark 3.

Due to the convexity of f, the absolute value in the right side of (79) is only needed for $\beta \in (0,1)$ (see (88) and (89)). Also, $w_f(1)=0$ since $f(1)=0$.

Remark 4.

The weight function$w_f$only depends on f, and the function$G_{P\parallel Q}$only depends on the pair of probability measures P and Q. In view of Proposition 1, it follows that, for$f,g\in \mathcal{C}$, the equality$w_f=w_g$holds on$(0,\infty )$if and only if (11) is satisfied with an arbitrary constant $c\in \mathbb{R}$. It is indeed easy to verify that (11) yields $w_f=w_g$ on $(0,\infty )$.

Remark 5.

An equivalent way to write$G_{P\parallel Q}$in (80) is

$$G_{P\parallel Q}(\beta )=\{\begin{array}{cc}\hfill \mathbb{P}\left[\frac{\mathrm{d}P}{\mathrm{d}Q}(X)>\beta \right],& \beta \in [1,\infty )\hfill \\ \hfill \mathbb{P}\left[\frac{\mathrm{d}P}{\mathrm{d}Q}(X)\le \beta \right],& \beta \in (0,1)\hfill \end{array}$$ (96)

where $X\sim P$. Hence, the function $G_{P\parallel Q}:(0,\infty )\mapsto [0,1]$ is monotonically increasing in $(0,1)$, and it is monotonically decreasing in $[1,\infty )$; note that this function is in general discontinuous at 1 unless ${\mathbb{F}}_{P\parallel Q}(0)=\frac{1}{2}$. If $P\ll \gg Q$, then

$$\begin{array}{c}\hfill \underset{\beta \downarrow 0}{lim}{G}_{P\parallel Q}(\beta )=\underset{\beta \to \infty }{lim}{G}_{P\parallel Q}(\beta )=0.\end{array}$$ (97)

Note that if$P=Q$, then$G_{P\parallel Q}$is zero everywhere, which is consistent with the fact that$D_f(P\parallel Q)=0$.

Remark 6.

In the proof of Theorem 1-(1), the relaxation of the condition of strict convexity at 1 for a differentiable function $f\in \mathcal{C}$ is crucial, e.g., for the ${\chi }^s$ divergence with $s>2$. To clarify this claim, note that in view of (32), the function $f_s:(0,\infty )\mapsto \mathbb{R}$ is differentiable if $s>1$, and $f_s\in \mathcal{C}$ with $f_s^{\prime }(1)=0$; however, $f_s^{″}(1)=0$ if $s>2$, so $f_s$ in not strictly convex at 1 unless $s\in [1,2]$.

Remark 7.

Theorem 1-(1) with $c\ne 0$ enables, in some cases, to simplify integral representations of f-divergences. This is next exemplified in the proof of Theorem 2.

Theorem 1 yields integral representations for various f-divergences and related measures; some of these representations were previously derived by Sason and Verdú in [22] in a case by case basis, without the unified approach of Theorem 1. We next provide such integral representations. Note that, for some f-divergences, the function $f\in \mathcal{C}$ is not differentiable on $(0,\infty )$; hence, Theorem 1 is not necessarily directly applicable.

Theorem 2.

The following integral representations hold as a function of the relative information spectrum:

• (1) 
  Relative entropy [22] (219):

  $$\begin{array}{cc}\hfill \frac{1}{loge}D(P\parallel Q)& ={\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{\beta }\mathrm{d}\beta -{\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{\beta }\mathrm{d}\beta .\hfill \end{array}$$ (98)
• (2) 
  Hellinger divergence of order $\alpha \in (0,1)\cup (1,\infty )$ [22] (434) and (437):

  $${\mathcal{H}}_{\alpha }(P\parallel Q)=\{\begin{array}{cc}\frac{1}{1-\alpha }-{\int }_0^{\infty }{\beta }^{\alpha -2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta ,\hfill & \alpha \in (0,1)\hfill \\ {\int }_0^{\infty }{\beta }^{\alpha -2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -\frac{1}{\alpha -1},\hfill & \alpha \in (1,\infty ).\hfill \end{array}$$ (99)
  In particular, the chi-squared divergence, squared Hellinger distance and Bhattacharyya distance satisfy

  $$\begin{array}{cc}\hfill {\chi }^2(P\parallel Q)& ={\int }_0^{\infty }\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -1;\hfill \end{array}$$ (100)

  $$\begin{array}{cc}\hfill {\mathcal{H}}^2(P\parallel Q)& =1-\frac{1}{2}{\int }_0^{\infty }{\beta }^{-\frac{3}{2}}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta ;\hfill \end{array}$$ (101)

  $$\begin{array}{cc}\hfill B(P\parallel Q)& =log2-log\left({\int }_0^{\infty }{\beta }^{-\frac{3}{2}}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \right),\hfill \end{array}$$ (102)

  where (100) appears in [22] (439).
• (3) 
  Rényi divergence [22] (426) and (427): For $\alpha \in (0,1)\cup (1,\infty )$,

  $$D_{\alpha }(P\parallel Q)=\{\begin{array}{cc}\frac{1}{\alpha -1}log\left((1-\alpha ){\int }_0^{\infty }{\beta }^{\alpha -2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \right),\hfill & \alpha \in (0,1)\hfill \\ \frac{1}{\alpha -1}log\left((\alpha -1){\int }_0^{\infty }{\beta }^{\alpha -2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta \right),\hfill & \alpha \in (1,\infty ).\hfill \end{array}$$ (103)
• (4) 
  ${\chi }^s$ divergence: For $s\ge 1$

  $$\begin{array}{cc}{\chi }^s(P\parallel Q)\hfill & ={\int }_1^{\infty }\frac{1}{\beta }\left(s-1+\frac{1}{\beta }\right){(\beta -1)}^{s-1}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta \hfill \\ & +{\int }_0^1\frac{1}{\beta }\left(s-1+\frac{1}{\beta }\right){(1-\beta )}^{s-1}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta .\hfill \end{array}$$ (104)
  In particular, the following identities hold for the total variation distance:

  $$\begin{array}{}\mathrm{(105)}& \hfill |P-Q|& =2{\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta \hfill \mathrm{(106)}& & =2{\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ,\hfill \end{array}$$

  where (105) appears in [22] (214).
• (5) 
  DeGroot statistical information:

  $${\mathcal{I}}_w(P\parallel Q)=\{\begin{array}{cc}(1-w){\int }_0^{\frac{1-w}{w}}\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ,\hfill & w\in (\frac{1}{2},1)\hfill \\ (1-w){\int }_{\frac{1-w}{w}}^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ,\hfill & w\in \left(0,\frac{1}{2}\right].\hfill \end{array}$$ (107)
• (6) 
  Triangular discrimination:

  $$\begin{array}{cc}\hfill \Delta (P\parallel Q)& =4{\int }_0^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{(\beta +1)}^2}\mathrm{d}\beta -2.\hfill \end{array}$$ (108)
• (7) 
  Lin’s measure: For $\theta \in [0,1]$,

  $$\begin{array}{cc}\hfill L_{\theta }(P\parallel Q)& =h(\theta )-(1-\theta ){\int }_0^{\infty }\frac{log\left(1+\frac{\theta \beta }{1-\theta }\right)}{{\beta }^2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta ,\hfill \end{array}$$ (109)

  where $h:[0,1]\mapsto [0,log2]$ denotes the binary entropy function. Specifically, the Jensen-Shannon divergence admits the integral representation:

  $$\begin{array}{cc}\hfill \mathrm{JS}(P\parallel Q)& =log2-{\int }_0^{\infty }\frac{log(\beta +1)}{2{\beta }^2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta .\hfill \end{array}$$ (110)
• (8) 
  Jeffrey’s divergence:

  $$\begin{array}{cc}\hfill J(P\parallel Q)& ={\int }_1^{\infty }\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\left(\frac{loge}{\beta }+\frac{log\beta }{{\beta }^2}\right)\mathrm{d}\beta \hfill \\ \hfill & -{\int }_0^1{\mathbb{F}}_{P\parallel Q}(log\beta )\left(\frac{loge}{\beta }+\frac{log\beta }{{\beta }^2}\right)\mathrm{d}\beta .\hfill \end{array}$$ (111)
• (9) 
  $E_{\gamma }$ divergence: For $\gamma \ge 1$,

  $$\begin{array}{c}\hfill E_{\gamma }(P\parallel Q)=\gamma {\int }_{\gamma }^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta .\end{array}$$ (112)

Proof. 

See Appendix A. ☐

An application of (112) yields the following interplay between the $E_{\gamma }$ divergence and the relative information spectrum.

Theorem 3.

Let $X\sim P$, and let the random variable ${ı}_{P\parallel Q}(X)$ have no probability masses. Denote

$$\begin{array}{cc}\hfill & {\mathcal{A}}_1:=\left\{E_{\gamma }(P\parallel Q):\gamma \ge 1\right\},\hfill \end{array}$$ (113)

$$\begin{array}{cc}\hfill & {\mathcal{A}}_2:=\left\{E_{\gamma }(Q\parallel P):\gamma >1\right\}.\hfill \end{array}$$ (114)

Then,

• $E_{\gamma }(P\parallel Q)$ is a continuously differentiable function of γ on $(1,\infty )$, and $E_{\gamma }^{\prime }(P\parallel Q)\le 0$;

• the sets ${\mathcal{A}}_1$ and ${\mathcal{A}}_2$ determine, respectively, the relative information spectrum ${\mathbb{F}}_{P\parallel Q}(·)$ on $[0,\infty )$ and $(-\infty ,0)$;

• for $\gamma >1$,

  $$\begin{array}{cc}\hfill & {\mathbb{F}}_{P\parallel Q}(+log\gamma )=1-E_{\gamma }(P\parallel Q)+\gamma E_{\gamma }^{\prime }(P\parallel Q),\hfill \end{array}$$ (115)

  $$\begin{array}{cc}\hfill & {\mathbb{F}}_{P\parallel Q}(-log\gamma )=-E_{\gamma }^{\prime }(Q\parallel P),\hfill \end{array}$$ (116)

  $$\begin{array}{}\mathrm{(117)}& {\mathbb{F}}_{P\parallel Q}(0)\hfill & =1-E_1(P\parallel Q)+\underset{\gamma \downarrow 1}{lim}{E}_{\gamma }^{\prime }(P\parallel Q)\hfill \mathrm{(118)}& & =-\underset{\gamma \downarrow 1}{lim}{E}_{\gamma }^{\prime }(Q\parallel P).\hfill \end{array}$$

Proof. 

We start by proving the first item. By our assumption, ${\mathbb{F}}_{P\parallel Q}(·)$ is continuous on $\mathbb{R}$. Hence, it follows from (112) that $E_{\gamma }(P\parallel Q)$ is continuously differentiable in $\gamma \in (1,\infty )$; furthermore, (45) implies that $E_{\gamma }(P\parallel Q)$ is monotonically decreasing in $\gamma$, which yields $E_{\gamma }^{\prime }(P\parallel Q)\le 0$.

We next prove the second and third items together. Let $X\sim P$ and $Y\sim Q$. From (112), for $\gamma >1$,

$$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}\gamma }\left(\frac{E_{\gamma }(P\parallel Q)}{\gamma }\right)=-\frac{1-{\mathbb{F}}_{P\parallel Q}(log\gamma )}{{\gamma }^2},\end{array}$$ (119)

which yields (115). Due to the continuity of ${\mathbb{F}}_{P\parallel Q}(·)$, it follows that the set ${\mathcal{A}}_1$ determines the relative information spectrum on $[0,\infty )$.

To prove (116), we have

$$\begin{array}{}\mathrm{(120)}& \hfill E_{\gamma }(Q\parallel P)& =\mathbb{P}[{ı}_{Q\parallel P}(Y)>log\gamma ]-\gamma \mathbb{P}[{ı}_{Q\parallel P}(X)>log\gamma ]\hfill \mathrm{(121)}& & =1-{\mathbb{F}}_{Q\parallel P}(log\gamma )-\gamma \mathbb{P}[{ı}_{Q\parallel P}(X)>log\gamma ]\hfill \mathrm{(122)}& & =E_{\gamma }(Q\parallel P)-\gamma E_{\gamma }^{\prime }(Q\parallel P)-\gamma \mathbb{P}[{ı}_{Q\parallel P}(X)>log\gamma ]\hfill \mathrm{(123)}& & =E_{\gamma }(Q\parallel P)-\gamma E_{\gamma }^{\prime }(Q\parallel P)-\gamma \mathbb{P}[{ı}_{P\parallel Q}(X)<-log\gamma ]\hfill \mathrm{(124)}& & =E_{\gamma }(Q\parallel P)-\gamma E_{\gamma }^{\prime }(Q\parallel P)-\gamma {\mathbb{F}}_{P\parallel Q}(-log\gamma )\hfill \end{array}$$

where (120) holds by switching P and Q in (46); (121) holds since $Y\sim Q$; (122) holds by switching P and Q in (115) (correspondingly, also $X\sim P$ and $Y\sim Q$ are switched); (123) holds since ${ı}_{Q\parallel P}=-{ı}_{P\parallel Q}$; (124) holds by the assumption that $\frac{\mathrm{d}P}{\mathrm{d}Q}(X)$ has no probability masses, which implies that the sign < can be replaced with ≤ at the term $\mathbb{P}[{ı}_{P\parallel Q}(X)<-log\gamma ]$ in the right side of (123). Finally, (116) readily follows from (120)–(124), which implies that the set ${\mathcal{A}}_2$ determines ${\mathbb{F}}_{P\parallel Q}(·)$ on $(-\infty ,0)$.

Equalities (117) and (117) finally follows by letting $\gamma \downarrow 1$, respectively, on both sides of (115) and (116). ☐

A similar application of (107) yields an interplay between DeGroot statistical information and the relative information spectrum.

Theorem 4.

Let $X\sim P$, and let the random variable ${ı}_{P\parallel Q}(X)$ have no probability masses. Denote

$$\begin{array}{cc}\hfill & {\mathcal{B}}_1:=\left\{{\mathcal{I}}_{\omega }(P\parallel Q):\omega \in \left(0,\frac{1}{2}\right]\right\},\hfill \end{array}$$ (125)

$$\begin{array}{cc}\hfill & {\mathcal{B}}_2:=\left\{{\mathcal{I}}_{\omega }(P\parallel Q):\omega \in \left(\frac{1}{2},1\right)\right\}.\hfill \end{array}$$ (126)

Then,

• ${\mathcal{I}}_{\omega }(P\parallel Q)$ is a continuously differentiable function of ω on $(0,\frac{1}{2})\cup (\frac{1}{2},1)$,

  $$\begin{array}{c}\hfill \underset{\omega \uparrow \frac{1}{2}}{lim}{\mathcal{I}}_{\omega }^{\prime }(P\parallel Q)-\underset{\omega \downarrow \frac{1}{2}}{lim}{\mathcal{I}}_{\omega }^{\prime }(P\parallel Q)=2,\end{array}$$ (127)

  and ${\mathcal{I}}_{\omega }^{\prime }(P\parallel Q)$ is, respectively, non-negative or non-positive on $\left(0,\frac{1}{2}\right)$ and $\left(\frac{1}{2},1\right)$;

• the sets ${\mathcal{B}}_1$ and ${\mathcal{B}}_2$ determine, respectively, the relative information spectrum ${\mathbb{F}}_{P\parallel Q}(·)$ on $[0,\infty )$ and $(-\infty ,0)$;

• for $\omega \in \left(0,\frac{1}{2}\right)$

  $$\begin{array}{c}\hfill {\mathbb{F}}_{P\parallel Q}\left(log\frac{1-\omega }{\omega }\right)=1-{\mathcal{I}}_{\omega }(P\parallel Q)-(1-\omega ){\mathcal{I}}_{\omega }^{\prime }(P\parallel Q),\end{array}$$ (128)

  for $\omega \in \left(\frac{1}{2},1\right)$

  $$\begin{array}{c}\hfill {\mathbb{F}}_{P\parallel Q}\left(log\frac{1-\omega }{\omega }\right)=-{\mathcal{I}}_{\omega }(P\parallel Q)-(1-\omega ){\mathcal{I}}_{\omega }^{\prime }(P\parallel Q),\end{array}$$ (129)

  and

  $$\begin{array}{cc}\hfill {\mathbb{F}}_{P\parallel Q}(0)& =-{\mathcal{I}}_{\frac{1}{2}}(P\parallel Q)-\frac{1}{2}\underset{\omega \downarrow \frac{1}{2}}{lim}{\mathcal{I}}_{\omega }^{\prime }(P\parallel Q).\hfill \end{array}$$ (130)

Remark 8.

By relaxing the condition in Theorems 3 and 4 where $\frac{\mathrm{d}P}{\mathrm{d}Q}(X)$ has no probability masses with $X\sim P$, it follows from the proof of Theorem 3 that each one of the sets

$$\begin{array}{cc}\hfill & \mathcal{A}:={\mathcal{A}}_1\cup {\mathcal{A}}_2=\left\{\left(E_{\gamma }(P\parallel Q),E_{\gamma }(Q\parallel P)\right):\gamma \ge 1\right\},\hfill \end{array}$$ (131)

$$\begin{array}{cc}\hfill & \mathcal{B}:={\mathcal{B}}_1\cup {\mathcal{B}}_2=\left\{{\mathcal{I}}_{\omega }(P\parallel Q):\omega \in (0,1)\right\}\hfill \end{array}$$ (132)

determines ${\mathbb{F}}_{P\parallel Q}(·)$ at every point on $\mathbb{R}$ where this relative information spectrum is continuous. Note that, as a cumulative distribution function, ${\mathbb{F}}_{P\parallel Q}(·)$ is discontinuous at a countable number of points. Consequently, under the condition that $f\in \mathcal{C}$ is differentiable on $(0,\infty )$, the integral representations of $D_f(P\parallel Q)$ in Theorem 1 are not affected by the countable number of discontinuities for ${\mathbb{F}}_{P\parallel Q}(·)$.

In view of Theorems 1, 3 and 4 and Remark 8, we get the following result.

Corollary 1.

Let $f\in \mathcal{C}$ be a differentiable function on $(0,\infty )$, and let $P\ll \gg Q$ be probability measures. Then, each one of the sets $\mathcal{A}$ and $\mathcal{B}$ in (131) and (132), respectively, determines $D_f(P\parallel Q)$.

Remark 9.

Corollary 1 is supported by the integral representation of $D_f(P\parallel Q)$ in [3] Theorem 11, expressed as a function of the set of values in $\mathcal{B}$, and its analogous representation in [22] Proposition 3 as a function of the set of values in $\mathcal{A}$. More explicitly, [3] Theorem 11 states that if $f\in \mathcal{C}$, then

$$\begin{array}{c}\hfill D_f(P\parallel Q)={\int }_0^1{\mathcal{I}}_{\omega }(P\parallel Q)\mathrm{d}{\Gamma }_f(\omega )\end{array}$$ (133)

where ${\Gamma }_f$ is a certain σ-finite measure defined on the Borel subsets of $(0,1)$; it is also shown in [3] (80) that if $f\in \mathcal{C}$ is twice differentiable on $(0,\infty )$, then

$$\begin{array}{cc}\hfill D_f(P\parallel Q)& ={\int }_0^1{\mathcal{I}}_{\omega }(P\parallel Q)\frac{1}{{\omega }^3}{f}^{″}\left(\frac{\omega }{1-\omega }\right)\mathrm{d}\omega .\hfill \end{array}$$ (134)

4. New $\mathbf{f}$-Divergence Inequalities

Various approaches for the derivation of f-divergence inequalities were studied in the literature (see Section 1 for references). This section suggests a new approach, leading to a lower bound on an arbitrary f-divergence by means of the $E_{\gamma }$ divergence of an arbitrary order $\gamma \ge 1$ (see (45)) or the DeGroot statistical information (see (50)). This approach leads to generalizations of the Bretagnole-Huber inequality [58], whose generalizations are later motivated in this section. The utility of the f-divergence inequalities in this section is exemplified in the setup of Bayesian binary hypothesis testing.

In the following, we provide the first main result in this section for the derivation of new f-divergence inequalities by means of the $E_{\gamma }$ divergence. Generalizing the total variation distance, the $E_{\gamma }$ divergence in (45)–(47) is an f-divergence whose utility in information theory has been exemplified in [17] Chapter 3, [54],[57] p. 2314 and [69]; the properties of this measure were studied in [22] Section 7 and [54] Section 2.B.

Theorem 5.

Let $f\in \mathcal{C}$, and let $f^{\ast }\in \mathcal{C}$ be the conjugate convex function as defined in (12). Let P and Q be probability measures. Then, for all $\gamma \in [1,\infty )$,

$$\begin{array}{c}\hfill D_f(P\parallel Q)\ge f^{\ast }\left(1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q)\right)+f^{\ast }\left(\frac{1}{\gamma }\left(1-E_{\gamma }(P\parallel Q)\right)\right)-f^{\ast }\left(\frac{1}{\gamma }\right).\end{array}$$ (135)

Proof. 

Let $p=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$ be the densities of P and Q with respect to a dominating measure $\mu$$(P,Q\ll \mu )$. Then, for an arbitrary $a\in \mathbb{R}$,

$$\begin{array}{}\mathrm{(136)}& \hfill D_f(P\parallel Q)& =D_{f^{\ast }}(Q\parallel P)\hfill \mathrm{(137)}& & =\int p{f}^{\ast }\left(\frac{q}{p}\right)\mathrm{d}\mu \hfill \mathrm{(138)}& & =\int p\left[f^{\ast }\left(max\left\{a,\frac{q}{p}\right\}\right)+f^{\ast }\left(min\left\{a,\frac{q}{p}\right\}\right)-f^{\ast }(a)\right]\mathrm{d}\mu \hfill \mathrm{(137)}& & \ge f^{\ast }\left(\int pmax\left\{a,\frac{q}{p}\right\}\mathrm{d}\mu \right)+f^{\ast }\left(\int pmin\left\{a,\frac{q}{p}\right\}\mathrm{d}\mu \right)-f^{\ast }(a)\hfill \end{array}$$

where (139) follows from the convexity of $f^{\ast }$ and by invoking Jensen’s inequality.

Setting $a:=\frac{1}{\gamma }$ with $\gamma \in [1,\infty )$ gives

$$\begin{array}{}\mathrm{(140)}& \hfill \int pmax\left\{a,\frac{q}{p}\right\}\mathrm{d}\mu & =\int max\left\{\frac{p}{\gamma },q\right\}\mathrm{d}\mu \hfill \mathrm{(141)}& & =\int q\mathrm{d}\mu +\int max\left\{\frac{p}{\gamma }-q,0\right\}\mathrm{d}\mu \hfill \mathrm{(142)}& & =1+\frac{1}{\gamma }\int qmax\left\{\frac{p}{q}-\gamma ,0\right\}\mathrm{d}\mu \hfill \mathrm{(143)}& & =1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q),\hfill \end{array}$$

and

$$\begin{array}{}\mathrm{(143)}& \hfill \int pmin\left\{a,\frac{q}{p}\right\}\mathrm{d}\mu & =\int p\left(a+\frac{q}{p}-max\left\{a,\frac{q}{p}\right\}\right)\mathrm{d}\mu \hfill \mathrm{(145)}& & =a+1-\int pmax\left\{a,\frac{q}{p}\right\}\mathrm{d}\mu \hfill \mathrm{(146)}& & =\frac{1}{\gamma }\left(1-E_{\gamma }(P\parallel Q)\right)\hfill \end{array}$$

where (146) follows from (143) by setting $a:=\frac{1}{\gamma }$. Substituting (143) and (146) into the right side of (139) gives (135). ☐

An application of Theorem 5 gives the following lower bounds on the Hellinger and Rényi divergences with arbitrary positive orders, expressed as a function of the $E_{\gamma }$ divergence with an arbitrary order $\gamma \ge 1$.

Corollary 2.

For all $\alpha >0$ and $\gamma \ge 1$,

$${\mathcal{H}}_{\alpha }(P\parallel Q)\ge \{\begin{array}{cc}\frac{1}{\alpha -1}\left[{\left(1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q)\right)}^{1-\alpha }+{\left(\frac{1-E_{\gamma }(P\parallel Q)}{\gamma }\right)}^{1-\alpha }-1-{\gamma }^{\alpha -1}\right],\hfill & \alpha \ne 1\hfill \\ -{log}_{\mathrm{e}}\left(\left(1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q)\right)\left(1-E_{\gamma }(P\parallel Q)\right)\right),\hfill & \alpha =1,\hfill \end{array}$$ (147)

and

$$D_{\alpha }(P\parallel Q)\ge \{\begin{array}{cc}\frac{1}{\alpha -1}log\left({\left(1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q)\right)}^{1-\alpha }+{\gamma }^{\alpha -1}\left[{\left(1-E_{\gamma }(P\parallel Q)\right)}^{1-\alpha }-1\right]\right),\hfill & \alpha \ne 1\\ -log\left(\left(1+\frac{1}{\gamma }{E}_{\gamma }(P\parallel Q)\right)\left(1-E_{\gamma }(P\parallel Q)\right)\right),\hfill & \alpha =1.\end{array}$$ (148)

Proof. 

Inequality (147), for $\alpha \in (0,1)\cup (1,\infty )$, follows from Theorem 5 and (22); for $\alpha =1$, it holds in view of Theorem 5, and equalities (17) and (24). Inequality (148), for $\alpha \in (0,1)\cup (1,\infty )$, follows from (30) and (147); for $\alpha =1$, it holds in view of (24), (147) and since $D_1(P\parallel Q)=D(P\parallel Q)$. ☐

Specialization of Corollary 2 for $\alpha =2$ in (147) and $\alpha =1$ in (148) gives the following result.

Corollary 3.

For $\gamma \in [1,\infty )$, the following upper bounds on $E_{\gamma }$ divergence hold as a function of the relative entropy and ${\chi }^2$ divergence:

$$\begin{array}{cc}\hfill & E_{\gamma }(P\parallel Q)\le \frac{1}{2}\left[1-\gamma +\sqrt{{(\gamma -1)}^2+\frac{4\gamma {\chi }^2(P\parallel Q)}{1+\gamma +{\chi }^2(P\parallel Q)}}\right],\hfill \end{array}$$ (149)

$$\begin{array}{cc}\hfill & E_{\gamma }(P\parallel Q)\le \frac{1}{2}\left[1-\gamma +\sqrt{{(\gamma -1)}^2+4\gamma \left(1-exp(-D(P\parallel Q))\right)}\right].\hfill \end{array}$$ (150)

Remark 10.

From [4] (58),

$${\chi }^2(P\parallel Q)\ge \{\begin{array}{cc}{|P-Q|}^2,\hfill & |P-Q|\in [0,1)\hfill \\ \hfill \frac{|P-Q|}{2-|P-Q|},& |P-Q|\in [1,2)\hfill \end{array}$$ (151)

is a tight lower bound on the chi-squared divergence as a function of the total variation distance. In view of (49), we compare (151) with the specialized version of (149) when $\gamma =1$. The latter bound is expected to be looser than the tight bound in (151), as a result of the use of Jensen’s inequality in the proof of Theorem 5; however, it is interesting to examine how much we loose in the tightness of this specialized bound with $\gamma =1$. From (49), the substitution of $\gamma =1$ in (149) gives

$$\begin{array}{cc}\hfill {\chi }^2(P\parallel Q)\ge \frac{{2|P-Q|}^2}{{4-|P-Q|}^2},& |P-Q|\in [0,2),\hfill \end{array}$$ (152)

and, it can be easily verified that

• if $|P-Q|\in [0,1)$, then the lower bound in the right side of (152) is at most twice smaller than the tight lower bound in the right side of (151);

• if $|P-Q|\in [1,2)$, then the lower bound in the right side of (152) is at most $\frac{3}{2}$ times smaller than the tight lower bound in the right side of (151).

Remark 11.

Setting $\gamma =1$ in (150), and using (49), specializes to the Bretagnole-Huber inequality [58]:

$$\begin{array}{c}\hfill |P-Q|\le 2\sqrt{1-exp\left(-D(P\parallel Q)\right)}.\end{array}$$ (153)

Inequality (153) forms a counterpart to Pinsker’s inequality:

$$\begin{array}{c}\hfill \frac{1}{2}{|P-Q|}^{2}loge\le D(P\parallel Q),\end{array}$$ (154)

proved by Csiszár [12] and Kullback [70], with Kemperman [71] independently a bit later. As upper bounds on the total variation distance, (154) outperforms (153) if $D(P\parallel Q)\le 1.594$ nats, and (153) outperforms (154) for larger values of $D(P\parallel Q)$.

Remark 12.

In [59] (8), Vajda introduced a lower bound on the relative entropy as a function of the total variation distance:

$$\begin{array}{c}\hfill D(P\parallel Q)\ge log\left(\frac{2+|P-Q|}{2-|P-Q|}\right)-\frac{2|P-Q|loge}{2+|P-Q|},|P-Q|\in [0,2).\end{array}$$ (155)

The lower bound in the right side of (155) is asymptotically tight in the sense that it tends to ∞ if $|P-Q|\uparrow 2$, and the difference between $D(P\parallel Q)$ and this lower bound is everywhere upper bounded by $\frac{{2|P-Q|}^3}{{(2+|P-Q|)}^2}\le 4$ (see [59] (9)). The Bretagnole-Huber inequality in (153), on the other hand, is equivalent to

$$\begin{array}{c}\hfill D(P\parallel Q)\ge -log\left(1-\frac{1}{4}{|P-Q|}^2\right),|P-Q|\in [0,2).\end{array}$$ (156)

Although it can be verified numerically that the lower bound on the relative entropy in (155) is everywhere slightly tighter than the lower bound in (156) (for $|P-Q|\in [0,2)$), both lower bounds on $D(P\parallel Q)$ are of the same asymptotic tightness in a sense that they both tend to ∞ as $|P-Q|\uparrow 2$ and their ratio tends to 1. Apart of their asymptotic tightness, the Bretagnole-Huber inequality in (156) is appealing since it provides a closed-form simple upper bound on $|P-Q|$ as a function of $D(P\parallel Q)$ (see (153)), whereas such a closed-form simple upper bound cannot be obtained from (155). In fact, by the substitution $v:=-\frac{2-|P-Q|}{2+|P-Q|}$ and the exponentiation of both sides of (155), we get the inequality $v{e}^v\ge -\frac{1}{e}exp\left(-D(P\parallel Q)\right)$ whose solution is expressed by the Lambert W function [72]; it can be verified that (155) is equivalent to the following upper bound on the total variation distance as a function of the relative entropy:

$$\begin{array}{cc}\hfill & |P-Q|\le \frac{2\left(1+W(z)\right)}{1-W(z)},\hfill \end{array}$$ (157)

$$\begin{array}{cc}\hfill & z:=-\frac{1}{e}exp\left(-D(P\parallel Q)\right),\hfill \end{array}$$ (158)

where W in the right side of (157) denotes the principal real branch of the Lambert W function. The difference between the upper bounds in (153) and (157) can be verified to be marginal if $D(P\parallel Q)$ is large (e.g., if $D(P\parallel Q)=4$ nats, then the upper bounds on $|P-Q|$ are respectively equal to 1.982 and 1.973), though the former upper bound in (153) is clearly more simple and amenable to analysis.

The Bretagnole-Huber inequality in (153) is proved to be useful in the context of lower bounding the minimax risk (see, e.g., [5] pp. 89–90, 94), and the problem of density estimation (see, e.g., [6] Section 1.6). The utility of this inequality motivates its generalization in this section (see Corollaries 2 and 3, and also see later Theorem 7 followed by Example 2).

In [22] Section 7.C, Sason and Verdú generalized Pinsker’s inequality by providing an upper bound on the $E_{\gamma }$ divergence, for $\gamma >1$, as a function of the relative entropy. In view of (49) and the optimality of the constant in Pinsker’s inequality (154), it follows that the minimum achievable $D(P\parallel Q)$ is quadratic in $E_1(P\parallel Q)$ for small values of $E_1(P\parallel Q)$. It has been proved in [22] Section 7.C that this situation ceases to be the case for $\gamma >1$, in which case it is possible to upper bound $E_{\gamma }(P\parallel Q)$ as a constant times $D(P\parallel Q)$ where this constant tends to infinity as we let $\gamma \downarrow 1$. We next cite the result in [22] Theorem 30, extending (154) by means of the $E_{\gamma }$ divergence for $\gamma >1$, and compare it numerically to the bound in (150).

Theorem 6.

([22] Theorem 30) For every $\gamma >1$,

$$\begin{array}{c}\hfill sup\frac{E_{\gamma }(P\parallel Q)}{D(P\parallel Q)}=c_{\gamma }\end{array}$$ (159)

where the supremum is over $P\ll Q,P\ne Q$, and $c_{\gamma }$ is a universal function (independent of $(P,Q)$), given by

$$\begin{array}{cc}\hfill & c_{\gamma }=\frac{t_{\gamma }-\gamma }{t_{\gamma }log{t}_{\gamma }+(1-t_{\gamma })loge},\hfill \end{array}$$ (160)

$$\begin{array}{cc}\hfill & t_{\gamma }=-\gamma W_{-1}\left(-\frac{1}{\gamma }{e}^{-\frac{1}{\gamma }}\right)\hfill \end{array}$$ (161)

where $W_{-1}$ in (161) denotes the secondary real branch of the Lambert W function [72].

As an immediate consequence of (159), it follows that

$$\begin{array}{c}\hfill E_{\gamma }(P\parallel Q)\le c_{\gamma }D(P\parallel Q),\end{array}$$ (162)

which forms a straight-line bound on the $E_{\gamma }$ divergence as a function of the relative entropy for $\gamma >1$. Similarly to the comparison of the Bretagnole-Huber inequality (153) and Pinsker’s inequality (154), we exemplify numerically that the extension of Pinsker’s inequality to the $E_{\gamma }$ divergence in (162) forms a counterpart to the generalized version of the Bretagnole-Huber inequality in (150).

Figure 1 plots an upper bound on the $E_{\gamma }$ divergence, for $\gamma \in \{1.1,2.0,3.0,4.0\}$, as a function of the relative entropy (or, alternatively, a lower bound on the relative entropy as a function of the $E_{\gamma }$ divergence). The upper bound on $E_{\gamma }(P\parallel Q)$ for $\gamma >1$, as a function of $D(P\parallel Q)$, is composed of the following two components:

• the straight-line bound, which refers to the right side of (162), is tighter than the bound in the right side of (150) if the relative entropy is below a certain value that is denoted by $d(\gamma )$ in nats (it depends on $\gamma$);

• the curvy line, which refers to the bound in the right side of (150), is tighter than the straight-line bound in the right side of (162) for larger values of the relative entropy.

Figure 1.

Upper bounds on the $E_{\gamma }$ divergence, for $\gamma >1$, as a function of the relative entropy (the curvy and straight lines follow from (150) and (162), respectively).

It is supported by Figure 1 that $d:(1,\infty )\mapsto (0,\infty )$ is positive and monotonically increasing, and $\underset{\gamma \downarrow 1}{lim}d(\gamma )=0$; e.g., it can be verified that $d(1.1)\approx 0.02$, $d(2)\approx 0.86$, $d(3)\approx 1.61$, and $d(4)\approx 2.10$ (see Figure 1).

Bayesian Binary Hypothesis Testing

The DeGroot statistical information [16] has the following meaning: consider two hypotheses $H_{\mathtt{0}}$ and $H_{\mathtt{1}}$, and let $\mathbb{P}\left[H_{\mathtt{0}}\right]=\omega$ and $\mathbb{P}\left[H_{\mathtt{1}}\right]=1-\omega$ with $\omega \in (0,1)$. Let P and Q be probability measures, and consider an observation Y where $Y|H_{\mathtt{0}}\sim P$, and $Y|H_{\mathtt{1}}\sim Q$. Suppose that one wishes to decide which hypothesis is more likely given the observation Y. The operational meaning of the DeGroot statistical information, denoted by ${\mathcal{I}}_{\omega }(P\parallel Q)$, is that this measure is equal to the minimal difference between the a-priori error probability (without side information) and a posteriori error probability (given the observation Y). This measure was later identified as an f-divergence by Liese and Vajda [3] (see (50) here).

Theorem 7.

The DeGroot statistical information satisfies the following upper bound as a function of the chi-squared divergence:

$${\mathcal{I}}_{\omega }(P\parallel Q)\le \{\begin{array}{cc}\omega -\frac{1}{2}+\sqrt{\frac{1}{4}-\frac{\omega (1-\omega )}{1+\omega {\chi }^2(P\parallel Q)}},\hfill & \omega \in \left(0,\frac{1}{2}\right],\hfill \\ \frac{1}{2}-\omega +\sqrt{\frac{1}{4}-\frac{\omega (1-\omega )}{1+\omega {\chi }^2(Q\parallel P)}},\hfill & \omega \in \left(\frac{1}{2},1\right),\hfill \end{array}$$ (163)

and the following bounds as a function of the relative entropy:

• (1) 
  $${\mathcal{I}}_{\omega }(P\parallel Q)\le \{\begin{array}{cc}\omega c_{\frac{1-\omega }{\omega }}D(P\parallel Q),\hfill & \omega \in \left(0,\frac{1}{2}\right),\hfill \\ \sqrt{\frac{1}{8loge}min\left\{D(P\parallel Q),D(Q\parallel P)\right\}},\hfill & \omega =\frac{1}{2},\hfill \\ (1-\omega )c_{\frac{\omega }{1-\omega }}D(Q\parallel P),\hfill & \omega \in \left(\frac{1}{2},1\right),\hfill \end{array}$$ (164)

  where $c_{\gamma }$ for $\gamma >1$ is introduced in (160);
• (2) 
  $${\mathcal{I}}_{\omega }(P\parallel Q)\le \{\begin{array}{cc}\hfill \omega -\frac{1}{2}+\sqrt{\frac{1}{4}-\omega (1-\omega )exp\left(-D(P\parallel Q)\right)},& \omega \in \left(0,\frac{1}{2}\right],\hfill \\ \hfill \frac{1}{2}-\omega +\sqrt{\frac{1}{4}-\omega (1-\omega )exp\left(-D(Q\parallel P)\right)},& \omega \in \left(\frac{1}{2},1\right).\hfill \end{array}$$ (165)

Proof. 

The first bound in (163) holds by combining (53) and (149); the second bound in (164) follows from (162) and (53) for $\omega \in \left(0,\frac{1}{2}\right)\cup \left(\frac{1}{2},1\right)$, and it follows from (52) and (154) when $\omega =\frac{1}{2}$; finally, the third bound in (165) follows from (150) and (53). ☐

Remark 13.

The bound in (164) forms an extension of Pinsker’s inequality (154) when $\omega \ne \frac{1}{2}$ (i.e., in the asymmetric case where the hypotheses $H_{\mathtt{0}}$ and $H_{\mathtt{1}}$ are not equally probable). Furthermore, in view of (52), the bound in (165) is specialized to the Bretagnole-Huber inequality in (153) by letting $\omega =\frac{1}{2}$.

Remark 14.

Numerical evidence shows that none of the bounds in (163)–(165) supersedes the others.

Remark 15.

The upper bounds on ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ in (163) and (165) are asymptotically tight when we let $D(P\parallel Q)$ and $D(Q\parallel P)$ tend to infinity. To verify this, first note that (see [23] Theorem 5)

$$\begin{array}{cc}\hfill D(P\parallel Q)& \le log\left(1+{\chi }^2(P\parallel Q)\right),\hfill \end{array}$$ (166)

which implies that also ${\chi }^2(P\parallel Q)$ and ${\chi }^2(Q\parallel P)$ tend to infinity. In this case, it can be readily verified that the bounds in (163) and (165) are specialized to ${\mathcal{I}}_{\omega }(P\parallel Q)\le min\{\omega ,1-\omega \}$; this upper bound, which is equal to the a-priori error probability, is also equal to the DeGroot statistical information since the a-posterior error probability tends to zero in the considered extreme case where P and Q are sufficiently far from each other, so that $H_{\mathtt{0}}$ and $H_{\mathtt{1}}$ are easily distinguishable in high probability when the observation Y is available.

Remark 16.

Due to the one-to-one correspondence between the $E_{\gamma }$ divergence and DeGroot statistical information in (53), which shows that the two measures are related by a multiplicative scaling factor, the numerical results shown in Figure 1 also apply to the bounds in (164) and (165); i.e., for $\omega \ne \frac{1}{2}$, the first bound in (164) is tighter than the second bound in (165) for small values of the relative entropy, whereas (165) becomes tighter than (164) for larger values of the relative entropy.

Corollary 4.

Let $f\in \mathcal{C}$, and let $f^{\ast }\in \mathcal{C}$ be as defined in (12). Then,

• (1) 
  for $w\in (0,\frac{1}{2}]$,

  $$\begin{array}{c}\hfill D_f(P\parallel Q)\ge f^{\ast }\left(1+\frac{{\mathcal{I}}_w(P\parallel Q)}{1-w}\right)+f^{\ast }\left(\frac{w-{\mathcal{I}}_w(P\parallel Q)}{1-w}\right)-f^{\ast }\left(\frac{w}{1-w}\right);\end{array}$$ (167)
• (2) 
  for $w\in \left(\frac{1}{2},1\right)$,

  $$\begin{array}{c}\hfill D_f(P\parallel Q)\ge f^{\ast }\left(1+\frac{{\mathcal{I}}_w(Q\parallel P)}{w}\right)+f^{\ast }\left(\frac{1-w-{\mathcal{I}}_w(Q\parallel P)}{w}\right)-f^{\ast }\left(\frac{1-w}{w}\right).\end{array}$$ (168)

Proof. 

Inequalities (167) and (168) follow by combining (135) and (53). ☐

We end this section by exemplifying the utility of the bounds in Theorem 7.

Example 2.

Let $\mathbb{P}\left[H_{\mathtt{0}}\right]=\omega$ and $\mathbb{P}\left[H_{\mathtt{1}}\right]=1-\omega$ with $\omega \in (0,1)$, and assume that the observation Y given that the hypothesis is $H_{\mathtt{0}}$ or $H_{\mathtt{1}}$ is Poisson distributed with the positive parameter μ or λ, respectively:

$$\begin{array}{c}Y|H_{\mathtt{0}}\sim P_{\mu },\hfill \end{array}$$ (169)

$$\begin{array}{c}Y|H_{\mathtt{1}}\sim P_{\lambda }\hfill \end{array}$$ (170)

where

$$\begin{array}{c}\hfill P_{\lambda }\left[k\right]=\frac{e^{-\lambda }{\lambda }^k}{k!},k\in \{0,1,\dots \}.\end{array}$$ (171)

Without any loss of generality, let $\omega \in \left(0,\frac{1}{2}\right]$. The bounds on the DeGroot statistical information ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ in Theorem 7 can be expressed in a closed form by relying on the following identities:

$$\begin{array}{c}D(P_{\mu }\parallel P_{\lambda })=\mu log\left(\frac{\mu }{\lambda }\right)+(\lambda -\mu )loge,\hfill \end{array}$$ (172)

$$\begin{array}{c}{\chi }^2(P_{\mu }\parallel P_{\lambda })=e^{\frac{{(\mu -\lambda )}^2}{\lambda }}-1.\hfill \end{array}$$ (173)

In this example, we compare the simple closed-form bounds on ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ in (163)–(165) with its exact value

$$\begin{array}{cc}\hfill {\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })& =min\{\omega ,1-\omega \}-\sum _{k=0}^{\infty }min\left\{\omega P_{\mu }\left[k\right],(1-\omega )P_{\lambda }\left[k\right]\right\}.\hfill \end{array}$$ (174)

To simplify the right side of (174), let $\mu >\lambda$, and define

$$\begin{array}{c}\hfill k_0=k_0(\lambda ,\mu ,\omega ):=⌊\frac{\mu -\lambda +ln\frac{1-\omega }{\omega }}{ln\frac{\mu }{\lambda }}⌋,\end{array}$$ (175)

where for $x\in \mathbb{R}$, $\lfloor x\rfloor$ denotes the largest integer that is smaller than or equal to x. It can be verified that

$$\{\begin{array}{cc}\hfill \omega P_{\mu }\left[k\right]\le (1-\omega )P_{\lambda }\left[k\right],& fork\le k_0\hfill \\ \hfill \omega P_{\mu }\left[k\right]>(1-\omega )P_{\lambda }\left[k\right],& fork>k_0.\hfill \end{array}$$ (176)

Hence, from (174)–(176),

$$\begin{array}{}\mathrm{(177)}& \hfill {\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })& =min\{\omega ,1-\omega \}-\omega \sum _{k=0}^{k_0}{P}_{\mu }\left[k\right]-(1-\omega )\sum _{k=k_0+1}^{\infty }{P}_{\lambda }\left[k\right]\hfill \mathrm{(178)}& & =min\{\omega ,1-\omega \}-\omega \sum _{k=0}^{k_0}{P}_{\mu }\left[k\right]-(1-\omega )\left(1-\sum _{k=0}^{k_0}{P}_{\lambda }\left[k\right]\right).\hfill \end{array}$$

To exemplify the utility of the bounds in Theorem 7, suppose that μ and λ are close, and we wish to obtain a guarantee on how small ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ is. For example, let $\lambda =99$, $\mu =101$, and $\omega =\frac{1}{10}$. The upper bounds on ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ in (163)–(165) are, respectively, equal to $4.6·{10}^{-4}$, $5.8·{10}^{-4}$ and $2.2·{10}^{-3}$; we therefore get an informative guarantee by easily calculable bounds. The exact value of ${\mathcal{I}}_{\omega }(P_{\mu }\parallel P_{\lambda })$ is, on the other hand, hard to compute since $k_0=209$ (see (175)), and the calculation of the right side of (178) appears to be sensitive to the selected parameters in this setting.

5. Local Behavior of f-Divergences

This section studies the local behavior of f-divergences; the starting point relies on [56] Section 3 which studies the asymptotic properties of f-divergences. The reader is also referred to a related study in [22] Section 4.F.

Lemma 4.

Let

• $\left\{P_n\right\}$ be a sequence of probability measures on a measurable space $(\mathcal{A},\mathcal{F})$;

• the sequence $\left\{P_n\right\}$ converge to a probability measure Q in the sense that

  $$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathrm{ess}\sup \frac{\mathrm{d}{P}_n}{\mathrm{d}Q}(Y)=1,Y\sim Q\end{array}$$ (179)

  where $P_n\ll Q$ for all sufficiently large n;

• $f,g\in \mathcal{C}$ have continuous second derivatives at 1 and $g^{″}(1)>0$.

Then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_f(P_n\parallel Q)}{D_g(P_n\parallel Q)}=\frac{f^{\prime \prime }(1)}{g^{\prime \prime }(1)}.\end{array}$$ (180)

Proof. 

The result in (180) follows from [56] Theorem 3, even without the additional restriction in [56] Section 3 which would require that the second derivatives of f and g are locally Lipschitz at a neighborhood of 1. More explicitly, in view of the analysis in [56] p. 1863, we get by relaxing the latter restriction that (cf. [56] (31))

$$\begin{array}{c}\hfill \left|D_f(P_n\parallel Q)-\frac{1}{2}{f}^{″}(1){\chi }^2(P_n\parallel Q)\right|\le \frac{1}{2}\underset{y\in [1-{\epsilon }_n,1+{\epsilon }_n]}{sup}\left|f^{″}(y)-f^{″}(1)\right|{\chi }^2(P_n\parallel Q),\end{array}$$ (181)

where ${\epsilon }_n\downarrow 0$ as we let $n\to \infty$, and also

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\chi }^2(P_n\parallel Q)=0.\end{array}$$ (182)

By our assumption, due to the continuity of $f^{″}$ and $g^{″}$ at 1, it follows from (181) and (182) that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_f(P_n\parallel Q)}{{\chi }^2(P_n\parallel Q)}=\frac{1}{2}{f}^{″}(1),\end{array}$$ (183)

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_g(P_n\parallel Q)}{{\chi }^2(P_n\parallel Q)}=\frac{1}{2}{g}^{″}(1),\end{array}$$ (184)

which yields (180) (recall that, by assumption, $g^{\prime \prime }(1)>0$). ☐

Remark 17.

Since f and g in Lemma 4 are assumed to have continuous second derivatives at 1, the left and right derivatives of the weight function $w_f$ in (79) at 1 satisfy, in view of Remark 3,

$$\begin{array}{c}\hfill w_f^{\prime }(1^{+})=-w_f^{\prime }(1^{-})=f^{″}(1).\end{array}$$ (185)

Hence, the limit in the right side of (180) is equal to $\frac{w_f^{\prime }(1^{+})}{w_g^{\prime }(1^{+})}$ or also to $\frac{w_f^{\prime }(1^{-})}{w_g^{\prime }(1^{-})}$.

Lemma 5.

$$\begin{array}{c}\hfill {\chi }^2(\lambda P+(1-\lambda )Q\parallel Q)={\lambda }^2{\chi }^2(P\parallel Q),\forall \lambda \in [0,1].\end{array}$$ (186)

Proof. 

Let $p=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$ be the densities of P and Q with respect to an arbitrary probability measure $\mu$ such that $P,Q\ll \mu$. Then,

$$\begin{array}{}\mathrm{(187)}& \hfill {\chi }^2(\lambda P+(1-\lambda )Q\parallel Q)& =\int \frac{{\left((\lambda p+(1-\lambda )q)-q\right)}^2}{q}\mathrm{d}\mu \hfill \mathrm{(188)}& & ={\lambda }^2\int \frac{{(p-q)}^2}{q}\mathrm{d}\mu \hfill \mathrm{(189)}& & ={\lambda }^2{\chi }^2(P\parallel Q).\hfill \end{array}$$

☐

Remark 18.

The result in Lemma 5, for the chi-squared divergence, is generalized to the identity

$$\begin{array}{c}\hfill {\chi }^s(\lambda P+(1-\lambda )Q\parallel Q)={\lambda }^s{\chi }^s(P\parallel Q),\forall \lambda \in [0,1],\end{array}$$ (190)

for all $s\ge 1$ (see (33)). The special case of $s=2$ is required in the continuation of this section.

Remark 19.

The result in Lemma 5 can be generalized as follows: let $P,Q,R$ be probability measures, and $\lambda \in [0,1]$. Let $P,Q,R\ll \mu$ for an arbitrary probability measure μ, and $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$, $q:=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$, and $r:=\frac{\mathrm{d}R}{\mathrm{d}\mu }$ be the corresponding densities with respect to μ. Calculation shows that

$$\begin{array}{cc}\hfill {\chi }^2(\lambda P+(1-\lambda )Q\parallel R)-{\chi }^2(Q\parallel R)& =c\lambda +\left[{\chi }^2(P\parallel R)-{\chi }^2(Q\parallel R)-c\right]{\lambda }^2\hfill \end{array}$$ (191)

with

$$\begin{array}{cc}\hfill c& :=\int \frac{(p-q)q}{r}\mathrm{d}\mu .\hfill \end{array}$$ (192)

If $Q=R$, then $c=0$ in (192), and (191) is specialized to (186). However, if $Q\ne R$, then c may be non-zero. This shows that, for small $\lambda \in [0,1]$, the left side of (191) scales linearly in λ if $c\ne 0$, and it has a quadratic scaling in λ if $c=0$ and ${\chi }^2(P\parallel R)\ne {\chi }^2(Q\parallel R)$ (e.g., if $Q=R$, as in Lemma 5). The identity in (191) yields

$$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}\lambda }{\chi }^2(\lambda P+(1-\lambda )Q\parallel R){|}_{\lambda =0}=\underset{\lambda \downarrow 0}{lim}\frac{{\chi }^2(\lambda P+(1-\lambda )Q\parallel R)-{\chi }^2(Q\parallel R)}{\lambda }=c.\end{array}$$ (193)

We next state the main result in this section.

Theorem 8.

Let

• P and Q be probability measures defined on a measurable space $(\mathcal{A},\mathcal{F})$, $Y\sim Q$, and suppose that

  $$\begin{array}{c}\hfill \mathrm{ess}\sup \frac{\mathrm{d}P}{\mathrm{d}Q}(Y)<\infty ;\end{array}$$ (194)

• $f\in \mathcal{C}$, and $f^{″}$ be continuous at 1.

Then,

$$\begin{array}{}\mathrm{(195)}& \hfill \underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_f(\lambda P+(1-\lambda )Q\parallel Q)& =\underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_f(Q\parallel \lambda P+(1-\lambda )Q)\hfill \mathrm{(196)}& & =\frac{1}{2}{f}^{″}(1){\chi }^2(P\parallel Q).\hfill \end{array}$$

Proof. 

Let ${\left\{{\lambda }_n\right\}}_{n\in \mathbb{N}}$ be a sequence in $[0,1]$, which tends to zero. Define the sequence of probability measures

$$\begin{array}{c}\hfill R_n:={\lambda }_{n}P+(1-{\lambda }_n)Q,n\in \mathbb{N}.\end{array}$$ (197)

Note that $P\ll Q$ implies that $R_n\ll Q$ for all $n\in \mathbb{N}$. Since

$$\begin{array}{c}\hfill \frac{\mathrm{d}{R}_n}{\mathrm{d}Q}={\lambda }_n\frac{\mathrm{d}P}{\mathrm{d}Q}+(1-{\lambda }_n),\end{array}$$ (198)

it follows from (194) that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathrm{ess}\sup \frac{\mathrm{d}{R}_n}{\mathrm{d}Q}(Y)=1.\end{array}$$ (199)

Consequently, (183) implies that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D_f(R_n\parallel Q)}{{\chi }^2(R_n\parallel Q)}=\frac{1}{2}{f}^{″}(1)\end{array}$$ (200)

where $\left\{{\lambda }_n\right\}$ in (197) is an arbitrary sequence which tends to zero. Hence, it follows from (197) and (200) that

$$\begin{array}{c}\hfill \underset{\lambda \downarrow 0}{lim}\frac{D_f(\lambda P+(1-\lambda )Q\parallel Q)}{{\chi }^2(\lambda P+(1-\lambda )Q\parallel Q)}=\frac{1}{2}{f}^{″}(1),\end{array}$$ (201)

and, by combining (186) and (201), we get

$$\begin{array}{c}\hfill \underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_f(\lambda P+(1-\lambda )Q\parallel Q)=\frac{1}{2}{f}^{″}(1){\chi }^2(P\parallel Q).\end{array}$$ (202)

We next prove the result for the limit in the right side of (195). Let $f^{\ast }:(0,\infty )\mapsto \mathbb{R}$ be the conjugate function of f, which is given in (12). By the assumption that f has a second continuous derivative, so is $f^{\ast }$ and it is easy to verify that the second derivatives of f and $f^{\ast }$ coincide at 1. Hence, from (13) and (202),

$$\begin{array}{}\mathrm{(203)}& \hfill \underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_f(Q\parallel \lambda P+(1-\lambda )Q)& =\underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_{f^{\ast }}(\lambda P+(1-\lambda )Q\parallel Q)\hfill \mathrm{(204)}& & =\frac{1}{2}{f}^{″}(1){\chi }^2(P\parallel Q).\hfill \end{array}$$

☐

Remark 20.

Although an f-divergence is in general not symmetric, in the sense that the equality $D_f(P\parallel Q)=D_f(Q\parallel P)$ does not necessarily hold for all pairs of probability measures $(P,Q)$, the reason for the equality in (195) stems from the fact that the second derivatives of f and $f^{\ast }$ coincide at 1 when f is twice differentiable.

Remark 21.

Under the conditions in Theorem 8, it follows from (196) that

$$\begin{array}{cc}\hfill & \frac{\mathrm{d}}{\mathrm{d}\lambda }{D}_f(\lambda P+(1-\lambda )Q\parallel Q){|}_{\lambda =0}=\underset{\lambda \downarrow 0}{lim}\frac{1}{\lambda }{D}_f(\lambda P+(1-\lambda )Q\parallel Q)=0,\hfill \end{array}$$ (205)

$$\begin{array}{cc}\hfill & \underset{\lambda \downarrow 0}{lim}\frac{{\mathrm{d}}^2}{\mathrm{d}{\lambda }^2}{D}_f(\lambda P+(1-\lambda )Q\parallel Q)=2\underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_f(\lambda P+(1-\lambda )Q\parallel Q)=f^{″}(1){\chi }^2(P\parallel Q)\hfill \end{array}$$ (206)

where (206) relies on L’Hôpital’s rule. The convexity of $D_f(P\parallel Q)$ in $(P,Q)$ also implies that, for all $\lambda \in [0,1]$,

$$\begin{array}{c}\hfill D_f(\lambda P+(1-\lambda )Q\parallel Q)\le \lambda D_f(P\parallel Q).\end{array}$$ (207)

The following result refers to the local behavior of Rényi divergences of an arbitrary non-negative order.

Corollary 5.

Under the condition in (194), for every $\alpha \in [0,\infty ]$,

$$\begin{array}{}\mathrm{(208)}& \hfill \underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_{\alpha }(\lambda P+(1-\lambda )Q\parallel Q)& =\underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{D}_{\alpha }(Q\parallel \lambda P+(1-\lambda )Q)\hfill \mathrm{(209)}& & =\frac{1}{2}\alpha {\chi }^2(P\parallel Q)loge.\hfill \end{array}$$

Proof. 

Let $\alpha \in (0,1)\cup (1,\infty )$. In view of (23) and Theorem 8, it follows that the local behavior of the Hellinger divergence of order $\alpha$ satisfies

$$\begin{array}{}\mathrm{(210)}& \hfill \underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{\mathcal{H}}_{\alpha }(\lambda P+(1-\lambda )Q\parallel Q)& =\underset{\lambda \downarrow 0}{lim}\frac{1}{{\lambda }^2}{\mathcal{H}}_{\alpha }(Q\parallel \lambda P+(1-\lambda )Q)\hfill \mathrm{(211)}& & =\frac{1}{2}\alpha {\chi }^2(P\parallel Q).\hfill \end{array}$$

The result now follows from (30), which implies that

$$\begin{array}{}\mathrm{(212)}& \hfill \underset{\lambda \downarrow 0}{lim}\frac{D_{\alpha }(\lambda P+(1-\lambda )Q\parallel Q)}{{\mathcal{H}}_{\alpha }(\lambda P+(1-\lambda )Q\parallel Q)}& =\underset{\lambda \downarrow 0}{lim}\frac{D_{\alpha }(Q\parallel \lambda P+(1-\lambda )Q)}{{\mathcal{H}}_{\alpha }(Q\parallel \lambda P+(1-\lambda )Q)}\hfill \mathrm{(213)}& & =\frac{1}{\alpha -1}\underset{u\to 0}{lim}\frac{log\left(1+(\alpha -1)u\right)}{u}\hfill \mathrm{(214)}& & =loge.\hfill \end{array}$$

The result in (208) and (209), for $\alpha \in (0,1)\cup (1,\infty )$, follows by combining the equalities in (210)–(214).

Finally, the result in (208) and (209) for $\alpha \in \{0,1,\infty \}$ follows from its validity for all $\alpha \in (0,1)\cup (1,\infty )$, and also due to the property where $D_{\alpha }(·\parallel ·)$ is monotonically increasing in $\alpha$ (see [73] Theorem 3). ☐

Appendix A. Proof of Theorem 2

We prove in the following the integral representations of f-divergences and related measures in Theorem 2.

• (1)
  Relative entropy: The function $f\in \mathcal{C}$ in (18) yields the following weight function in (79):

  $$\begin{array}{c}\hfill w_f(\beta )=\left(\frac{1}{\beta }-\frac{1}{{\beta }^2}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)loge,\beta >0.\end{array}$$ (A1)
  Consequently, setting $c:=loge$ in (82) yields

  $$\begin{array}{c}\hfill {\tilde{w}}_{f,c}(\beta )=\frac{1}{\beta }\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)loge,\end{array}$$ (A2)

  for $\beta >0$. Equality (98) follows from the substitution of (A2) into the right side of (83).
• (2)
  Hellinger divergence: In view of (22), for $\alpha \in (0,1)\cup (1,\infty )$, the weight function $w_{f_{\alpha }}:(0,\infty )\mapsto [0,\infty )$ in (79) which corresponds to $f_{\alpha }:(0,\infty )\mapsto \mathbb{R}$ in (23) can be verified to be equal to

  $$\begin{array}{c}\hfill w_{f_{\alpha }}(\beta )=\left({\beta }^{\alpha -2}-\frac{1}{{\beta }^2}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)\end{array}$$ (A3)

  for $\beta >0$. In order to simplify the integral representation of the Hellinger divergence ${\mathcal{H}}_{\alpha }(P\parallel Q)$, we apply Theorem 1-(1). From (A3), setting $c:=1$ in (82) implies that ${\tilde{w}}_{f_{\alpha },1}:(0,\infty )\to \mathbb{R}$ is given by

  $$\begin{array}{c}\hfill {\tilde{w}}_{f_{\alpha },1}(\beta )={\beta }^{\alpha -2}\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)\end{array}$$ (A4)

  for $\beta >0$. Hence, substituting (80) and (A4) into (83) yields

  $$\begin{array}{c}\hfill {\mathcal{H}}_{\alpha }(P\parallel Q)={\int }_1^{\infty }{\beta }^{\alpha -2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -{\int }_0^1{\beta }^{\alpha -2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta .\end{array}$$ (A5)
  For $\alpha >1$, (A5) yields

  $$\begin{array}{}\mathrm{(A6)}& \hfill {\mathcal{H}}_{\alpha }(P\parallel Q)& ={\int }_0^{\infty }{\beta }^{\alpha -2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -{\int }_0^1{\beta }^{\alpha -2}\mathrm{d}\beta \hfill \mathrm{(A7)}& & ={\int }_0^{\infty }{\beta }^{\alpha -2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -\frac{1}{\alpha -1},\hfill \end{array}$$

  and, for $\alpha \in (0,1)$, (A5) yields

  $$\begin{array}{}\mathrm{(A8)}& \hfill {\mathcal{H}}_{\alpha }(P\parallel Q)& ={\int }_1^{\infty }{\beta }^{\alpha -2}\mathrm{d}\beta -{\int }_0^{\infty }{\beta }^{\alpha -2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \hfill \mathrm{(A9)}& & =\frac{1}{1-\alpha }-{\int }_0^{\infty }{\beta }^{\alpha -2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta .\hfill \end{array}$$
  This proves (99). We next consider the following special cases:

  • -
    In view of (25), equality (100) readily follows from (99) with $\alpha =2$.
  • -
    In view of (28), equality (101) readily follows from (99) with $\alpha =\frac{1}{2}$.
  • -
    In view of (29), equality (102) readily follows from (101).
• (3)Rényi divergence: In view of the one-to-one correspondence in (30) between the Rényi divergence and the Hellinger divergence of the same order, (103) readily follows from (99).
• (4)
  ${\chi }^s$ divergence with $s\ge 1$: We first consider the case where $s>1$. From (33), the function $f_s:(0,\infty )\mapsto \mathbb{R}$ in (32) is differentiable and $f_s^{\prime }(1)=0$. Hence, the respective weight function $w_{f_s}:(0,\infty )\mapsto (0,\infty )$ can be verified from (79) to be given by

  $$\begin{array}{c}\hfill w_{f_s}(\beta )=\frac{1}{\beta }\left(s-1+\frac{1}{\beta }\right){|\beta -1|}^{s-1},\beta >0.\end{array}$$ (A10)

  The result in (104), for $s>1$, follows readily from (33), (80), (81) and (A10).

  We next prove (104) with $s=1$. In view of (32), (34), (35) and the dominated convergence theorem,

  $$\begin{array}{}\mathrm{(A11)}& \hfill |P-Q|& =\underset{s\downarrow 1}{lim}{\chi }^s(P\parallel Q)\hfill \mathrm{(A12)}& & ={\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta +{\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta .\hfill \end{array}$$

  This extends (104) for all $s\ge 1$, although $f_1(t)=|t-1|$ for $t>0$ is not differentiable at 1. For $s=1$, in view of (95), the integral representation in the right side of (A12) can be simplified to (105) and (106).

• (5)
  DeGroot statistical information: In view of (50)–(51), since the function ${\varphi }_w:(0,\infty )\mapsto \mathbb{R}$ is not differentiable at the point $\frac{1-\omega }{\omega }\in (0,\infty )$ for $\omega \in (0,1)$, Theorem 1 cannot be applied directly to get an integral representation of the DeGroot statistical information. To that end, for $(\omega ,\alpha )\in {(0,1)}^2$, consider the family of convex functions $f_{\omega ,\alpha }:(0,\infty )\mapsto \mathbb{R}$ given by (see [3] (55))

  $$\begin{array}{c}\hfill f_{\omega ,\alpha }(t)=\frac{1}{1-\alpha }\left({\left[{(\omega t)}^{\frac{1}{\alpha }}+{(1-\omega )}^{\frac{1}{\alpha }}\right]}^{\alpha }-{\left[{\omega }^{\frac{1}{\alpha }}+{(1-\omega )}^{\frac{1}{\alpha }}\right]}^{\alpha }\right),\end{array}$$ (A13)

  for $t>0$. These differentiable functions also satisfy

  $$\begin{array}{c}\hfill \underset{\alpha \downarrow 0}{lim}{f}_{\omega ,\alpha }(t)={\varphi }_w(t),\end{array}$$ (A14)

  which holds due to the identities

  $$\begin{array}{cc}\hfill & \underset{\alpha \downarrow 0}{lim}{\left(a^{\frac{1}{\alpha }}+b^{\frac{1}{\alpha }}\right)}^{\alpha }=max\{a,b\},a,b\ge 0;\hfill \end{array}$$ (A15)

  $$\begin{array}{cc}\hfill & min\{a,b\}=a+b-max\{a,b\},a,b\in \mathbb{R}.\hfill \end{array}$$ (A16)
  The application of Theorem 1-(1) to the set of functions $f_{\omega ,\alpha }\in \mathcal{C}$ with

  $$\begin{array}{c}\hfill c:=\frac{{(1-\omega )}^{\frac{1}{\alpha }}}{\alpha -1}{\left[{\omega }^{\frac{1}{\alpha }}+{(1-\omega )}^{\frac{1}{\alpha }}\right]}^{\alpha -1}\end{array}$$ (A17)

  yields

  $$\begin{array}{c}\hfill {\tilde{w}}_{f_{\omega ,\alpha },c}(\beta )=\frac{1-\omega }{1-\alpha }\frac{1}{{\beta }^2}{\left[1+{\left(\frac{\omega \beta }{1-\omega }\right)}^{\frac{1}{\alpha }}\right]}^{\alpha -1}\left[1\{0<\beta <1\}-1\{\beta \ge 1\}\right],\end{array}$$ (A18)

  for $\beta >0$, and

  $$\begin{array}{c}\hfill D_{f_{\omega ,\alpha }}(P\parallel Q)={\int }_0^{\infty }{\tilde{w}}_{f_{\omega ,\alpha },c}(\beta )G_{P\parallel Q}(\beta )\mathrm{d}\beta \end{array}$$ (A19)

  with $G_{P\parallel Q}(·)$ as defined in (80), and $(\omega ,\alpha )\in {(0,1)}^2$. From (A15) and (A18), it follows that

  $$\begin{array}{c}\hfill \underset{\alpha \downarrow 0}{lim}{\tilde{w}}_{f_{\omega ,\alpha },c}(\beta )=\frac{1-\omega }{{\beta }^2}\left[1\{0<\beta <1\}-1\{\beta \ge 1\}\right]\left[\frac{1}{2}1\left\{\beta =\frac{1-\omega }{\omega }\right\}+1\left\{0<\beta <\frac{1-\omega }{\omega }\right\}\right],\end{array}$$ (A20)

  for $\beta >0$. In view of (50), (51), (80), (A14), (A19) and (A20), and the monotone convergence theorem,

  $$\begin{array}{}& {\mathcal{I}}_{\omega }(P\parallel Q)\hfill \mathrm{(A21)}& =D_{{\varphi }_{\omega }}(P\parallel Q)\hfill \mathrm{(A22)}& =\underset{\alpha \downarrow 0}{lim}{D}_{f_{\omega ,\alpha }}(P\parallel Q)\hfill \mathrm{(A23)}& =(1-\omega ){\int }_0^{min\{1,\frac{1-\omega }{\omega }\}}\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta -(1-\omega ){\int }_1^{max\{1,\frac{1-\omega }{\omega }\}}\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ,\hfill \end{array}$$

  for $\omega \in (0,1)$. We next simplify (A23) as follows:

  • -
    if $\omega \in \left(1,\frac{1-\omega }{\omega }\right)$, then $\frac{1-\omega }{\omega }<1$ and (A23) yields

    $$\begin{array}{cc}\hfill {\mathcal{I}}_{\omega }(P\parallel Q)& =(1-\omega ){\int }_0^{\frac{1-\omega }{\omega }}\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ;\hfill \end{array}$$ (A24)
  • -
    if $\omega \in \left(0,\frac{1}{2}\right]$, then $\frac{1-\omega }{\omega }\ge 1$ and (A23) yields

    $$\begin{array}{}\mathrm{(A25)}& \hfill {\mathcal{I}}_{\omega }(P\parallel Q)& =(1-\omega ){\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta -(1-\omega ){\int }_1^{\frac{1-\omega }{\omega }}\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta \hfill \mathrm{(A26)}& & =(1-\omega ){\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta -(1-\omega ){\int }_1^{\frac{1-\omega }{\omega }}\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta \hfill \mathrm{(A27)}& & =(1-\omega ){\int }_{\frac{1-\omega }{\omega }}^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta ,\hfill \end{array}$$

    where (A26) follows from (95) (or its equivalent from in (73)).

  This completes the proof of (107). Note that, due to (95), the integral representation of ${\mathcal{I}}_{\omega }(P\parallel Q)$ in (107) is indeed continuous at $\omega =\frac{1}{2}$.

• (6)
  Triangular discrimination: In view of (36)–(37), the corresponding function ${\tilde{w}}_{f,1}:(0,\infty )\mapsto \mathbb{R}$ in (82) (i.e., with $c:=1$) can be verified to be given by

  $$\begin{array}{c}\hfill {\tilde{w}}_{f,1}(\beta )=\frac{4}{{(\beta +1)}^2}\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)\end{array}$$ (A28)

  for $\beta >0$. Substituting (80) and (A28) into (83) proves (108) as follows:

  $$\begin{array}{}\mathrm{(A29)}& \hfill \Delta (P\parallel Q)& =4\left({\int }_1^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{(\beta +1)}^2}\mathrm{d}\beta -{\int }_0^1\frac{{\mathbb{F}}_{P\parallel Q}(log\beta )}{{(\beta +1)}^2}\mathrm{d}\beta \right)\hfill \mathrm{(A30)}& & =4\left({\int }_0^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{(\beta +1)}^2}\mathrm{d}\beta -{\int }_0^1\frac{1}{{(\beta +1)}^2}\mathrm{d}\beta \right)\hfill \mathrm{(A31)}& & =4{\int }_0^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{(\beta +1)}^2}\mathrm{d}\beta -2.\hfill \end{array}$$
• (7)
  Lin’s measure and the Jensen-Shannon divergence: Let $\theta \in (0,1)$ (if $\theta \in \{0,1\}$, then (39) and (40) imply that $L_{\theta }(P\parallel Q)=0$). In view of (41), the application of Theorem 1-(1) with the function $f_{\theta }:(0,\infty )\mapsto \mathbb{R}$ in (42) yields the weight function $w_{f_{\theta }}:(0,\infty )\mapsto [0,\infty )$ defined as

  $$\begin{array}{cc}\hfill w_{f_{\theta }}(\beta )& =\frac{(1-\theta )log(\theta \beta +1-\theta )}{{\beta }^2}\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right).\hfill \end{array}$$ (A32)
  Consequently, we get

  $$\begin{array}{}\mathrm{(A33)}& \hfill L_{\theta }(P\parallel Q)& =(1-\theta )\left({\int }_1^{\infty }\frac{log(\theta \beta +1-\theta )}{{\beta }^2}\left(1-{\mathbb{F}}_{P\parallel Q}(log\beta )\right)\mathrm{d}\beta -{\int }_0^1\frac{log(\theta \beta +1-\theta )}{{\beta }^2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \right)\hfill \mathrm{(A34)}& & =(1-\theta )\left({\int }_1^{\infty }\frac{log(\theta \beta +1-\theta )}{{\beta }^2}\mathrm{d}\beta -{\int }_0^{\infty }\frac{log(\theta \beta +1-\theta )}{{\beta }^2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \right)\hfill \mathrm{(A35)}& & =\theta log\frac{1}{\theta }-(1-\theta ){\int }_0^{\infty }\frac{log(\theta \beta +1-\theta )}{{\beta }^2}{\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \hfill \mathrm{(A36)}& & =h(\theta )-(1-\theta ){\int }_0^{\infty }\frac{1}{{\beta }^2}log\left(\frac{\theta \beta }{1-\theta }+1\right){\mathbb{F}}_{P\parallel Q}(log\beta )\mathrm{d}\beta \hfill \end{array}$$

  where (A33) follows from (80), (81) and (A32); for $\theta \in (0,1)$, equality (A35) holds since

  $$\begin{array}{c}\hfill {\int }_1^{\infty }\frac{log(\theta \beta +1-\theta )}{{\beta }^2}\mathrm{d}\beta =\frac{\theta }{1-\theta }log\frac{1}{\theta };\end{array}$$ (A37)

  finally, (A36) follows from (73) where $h:[0,1]\mapsto [0,log2]$ denotes the binary entropy function. This proves (109). In view of (43), the identity in (110) for the Jensen-Shannon divergence follows from (109) with $\theta =\frac{1}{2}$.
• (8)
  Jeffrey’s divergence: In view of (21)–(20), the corresponding weight function $w_f:(0,\infty )\mapsto [0,\infty )$ in (79) can be verified to be given by

  $$\begin{array}{c}\hfill w_f(\beta )=\left(\frac{loge}{\beta }+\frac{1}{{\beta }^2}log\frac{\beta }{e}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right).\end{array}$$ (A38)
  Hence, setting $c:=loge$ in (82) implies that

  $$\begin{array}{c}\hfill {\tilde{w}}_{f,c}(\beta )=\left(\frac{loge}{\beta }+\frac{log\beta }{{\beta }^2}\right)\left(1\{\beta \ge 1\}-1\{0<\beta <1\}\right)\end{array}$$ (A39)

  for $\beta >0$. Substituting (80) and (A39) into (83) yields (111).
• (9)
  $E_{\gamma }$ divergence: Let $\gamma \ge 1$, and let $\omega \in \left(0,\frac{1}{2}\right]$ satisfy $\frac{1-\omega }{\omega }=\gamma$; hence, $\omega =\frac{1}{1+\gamma }$. From (53), we get

  $$\begin{array}{c}\hfill E_{\gamma }(P\parallel Q)=(1+\gamma ){\mathcal{I}}_{\frac{1}{1+\gamma }}(P\parallel Q).\end{array}$$ (A40)
  The second line in the right side of (107) yields

  $$\begin{array}{c}\hfill {\mathcal{I}}_{\frac{1}{1+\gamma }}(P\parallel Q)=\frac{\gamma }{1+\gamma }{\int }_{\gamma }^{\infty }\frac{1-{\mathbb{F}}_{P\parallel Q}(log\beta )}{{\beta }^2}\mathrm{d}\beta .\end{array}$$ (A41)

  Finally, substituting (A41) into the right side of (A40) yields (112).

Remark A1.

In view of (95), the integral representation for the ${\chi }^s$ divergence in (104) specializes to (100), (105) and (106) by letting $s=2$ and $s=1$, respectively.

Remark A2.

In view of (49), the first identity for the total variation distance in (105) follows readily from (112) with $\gamma =1$. The second identity in (106) follows from (73) and (105), and since ${\int }_1^{\infty }\frac{\mathrm{d}\beta }{{\beta }^2}=1$.

Conflicts of Interest

The author declares no conflict of interest.