Alternative Entropy Measures and Generalized Khinchin–Shannon Inequalities

Abstract

The Khinchin–Shannon generalized inequalities for entropy measures in Information Theory, are a paradigm which can be used to test the Synergy of the distributions of probabilities of occurrence in physical systems. The rich algebraic structure associated with the introduction of escort probabilities seems to be essential for deriving these inequalities for the two-parameter Sharma–Mittal set of entropy measures. We also emphasize the derivation of these inequalities for the special cases of one-parameter Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropy measures.

1. Introduction

In the present contribution we derive the Generalized Khinchin–Shannon inequalities (GKS) [1,2] associated to entropy measures of the Sharma–Mittal (SM) set [3]. We stress that the derivations to be presented here are a tentative way of implementing the ideas of the literature on interdisciplinary topics of Statistical Mechanics and Theory of Information [4,5,6]. The algebraic structure of the escort probability distributions on these derivations seems to be essential, contrary to the intuitive derivation of the usual Khinchin–Shannon inequalities for the Gibbs–Shannon entropy measures. We start on Section 2 with the construction of a generic probabilistic space with their elements—the probabilities of occurrence—arranged on blocks of m rows and n columns. It then follows the introduction of the definitions of simple, joint, conditional and marginal probabilities through the use of the Bayes’ law. In Section 3, we make use of the assumption of concavity in order to unveil the Synergy of the distribution of values of Gibbs–Shannon entropy measures [2]. In Section 4, we present the same development but for the SM set of entropy measures, after the introduction of the concept of escort probabilities. We then specialize the derivations to Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropies [7,8,9]. A detailed study is then undertaken in this section to treat the eventual ordering between the probabilities of occurrence and their associated escort probabilities. This is then enough for deriving the GKS inequalities for the SM entropy measures. In Section 5, we present a proposal for Information measure associated to SM entropies and we derive its related inequalities [10]. At this point we stress once more the upsurge of the synergy effect on the comparison of the information obtained from the entropy calculated with joint probabilities of occurrence and the entropies corresponding to simple probabilities. In Section 6, we present an alternative derivation of the GKS inequalities based on Hölder inequalities [11]. These can provide, in association with Bayes’ law, the same assumptions of concavity which have been used in Section 3 and Section 4 and a consequent identical derivation of the GKS inequalities given in Section 4.

2. The Probability Space. Probabilities of Occurrence

We consider that the data could be represented on two-dimensional arrays of m rows and n columns. We then have $m\times n$ blocks of data to undertake the statistical analysis. The joint probabilities of occurrence of a set of t variables $a_1,\dots ,a_t$ in columns $j_1,\dots ,j_t$, respectively, are given by

$$p_{j_1\dots j_t}(a_1,\dots ,a_t)=\frac{n_{j_1\dots j_t}(a_1,\dots ,a_t)}{m},$$ (1)

where m is the number of rows in the subarray $m\times t$ of the array $m\times n$, and $n_{j_1\dots j_t}(a_1,\dots ,a_t)$ is the number of occurrences of the set $a_1,\dots ,a_t$. The values assumed by the variables $j_1,\dots ,j_t$, $j_1<j_2<\dots <j_t$, are respectively given by:

$$\begin{array}{cc}\hfill j_1& =1,2,\dots ,n-t+1\hfill \\ \hfill j_2& =j_1+1,j_1+2,\dots ,n-t+2\hfill \\ \hfill ⋮& \hfill \\ \hfill j_{t-1}& =j_{t-2}+1,j_{t-2}+2,\dots ,n-1\hfill \\ \hfill j_t& =j_{t-1}+1,j_{t-1}+2,\dots ,n,\hfill \end{array}$$ (2)

or,

$$\begin{array}{cc}\hfill j_1& =1,2,\dots ,n-t+1\hfill \\ \hfill j_2& =j_1+1,j_1+2,\dots ,n-t+2\hfill \\ \hfill ⋮& \hfill \\ \hfill j_{t-1}& =j_1+t-2,j_1+t-1,\dots ,n-1\hfill \\ \hfill j_t& =j_1+t-1,j_1+t,\dots ,n.\hfill \end{array}$$ (3)

There are then $\left(\genfrac{}{}{0pt}{}{n}{t}\right)=\frac{n!}{t!(n-t)!}$ objects of t columns each, $1\le t\le n$, and if the variables $a_1,\dots ,a_t$ take on values $1,\dots ,W$, then we will have ${\left(W\right)}^t$ components for each of these objects.

Since:

$$\sum _{a_1,\dots ,a_t=1}^W{n}_{j_1\dots j_t}(a_1,\dots ,a_t)=m,$$ (4)

we can write:

$$\sum _{a_1,\dots ,a_t=1}^W{p}_{j_1\dots j_t}(a_1,\dots ,a_t)=1.$$ (5)

On the study of distributions of bases of nucleotides or distributions of amino acids in proteins, the related values of W are $W=4$ and $W=20$, respectively.

The Bayes’ law for the probabilities of occurrence of Equation (1) is written as:

$$\begin{array}{cc}\hfill p_{j_1\dots j_t}(a_1,\dots ,a_t)& =p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)·p_{j_t}\left(a_t\right)\hfill \\ \hfill & =p_{j_t{j}_1\dots j_{t-1}}\left(a_t\right|a_1,\dots ,a_{t-1})·p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\hfill \\ \hfill & =p_{j_t{j}_1\dots j_{t-1}}(a_t,a_1,\dots ,a_{t-1}),\hfill \end{array}$$ (6)

where $p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)$ stands for the conditional probability of occurrence of the values associated to the variables $a_1,\dots ,a_{t-1}$ in the columns $j_1,\dots ,j_{t-1}$, respectively, if the values associated to $a_t$ in the jth column are given a priori. This also means that:

$$\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)=1.$$ (7)

The marginal probabilities related to $p_{j_1\dots j_t}(a_1,\dots ,a_t)$ are then given by

$$\begin{array}{cc}\hfill p_{j_t}\left(a_t\right)& =\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_t}(a_1,\dots ,a_t)\hfill \\ \hfill & =\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)p_{j_t}\left(a_t\right).\hfill \end{array}$$ (8)

We then have from Equations (6) and (8):

$$1=\sum _{a_t}{p}_{j_t}\left(a_t\right)=\sum _{a_1,\dots ,a_t}{p}_{j_1\dots j_t}(a_1,\dots ,a_t),$$ (9)

which is the same result of Equation (5).

3. The Assumption of Concavity and the Synergy of Gibbs–Shannon Entropy Measures

A concave function of several variables should satisfy the following inequality:

$$\sum _l{\lambda }_{l}f\left(x_l\right)\le f\left(\sum _l{\lambda }_l{x}_l\right);{\lambda }_l\ge 0;\sum _l{\lambda }_l=1.$$ (10)

We shall apply Equation (10) to the Gibbs–Shannon entropies:

$$S_{j_1\dots j_t}=-\sum _{a_1,\dots ,a_t}{p}_{j_1\dots j_t}(a_1,\dots ,a_t)log{p}_{j_1\dots j_t}(a_1,\dots ,a_t),$$ (11)

$$S_{j_1\dots j_{t-1}|j_t}=-\sum _{a_1,\dots ,a_t}{p}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)log{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t).$$ (12)

where Equation (12) stands for the definition of Gibbs–Shannon entropy which is related to the conditional probabilities $p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)$. It is a measure of the uncertainty [2] on the distribution of probabilities of the columns $j_1,\dots ,j_{t-1}$, when we have previous information on the distribution of the column $j_t$.

From Bayes’ law, Equation (6) and from Equations (8), (11) and (12), we get:

$$S_{j_1\dots j_t}=S_{j_1\dots j_{t-1}|j_t}+S_{j_t}.$$ (13)

We now use the correspondences:

$$\begin{array}{c}{\lambda }_l\leftrightarrow p_{j_t}\left(a_t\right);x_l\leftrightarrow p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t),\\ f\left(x_l\right)\leftrightarrow -\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)log{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t),\end{array}$$ (14)

and we then have:

$$\sum _l{\lambda }_l{x}_l\leftrightarrow \sum _{a_t}{p}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)=\sum _{a_t}{p}_{j_1\dots j_t}(a_1,\dots ,a_t)=p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1}),$$ (15)

$$\sum _l{\lambda }_{l}f\left(x_l\right)\leftrightarrow -\sum _{a_t,a_1,\dots ,a_{t-1}}{p}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)log{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t),$$ (16)

and

$$f\left(\sum _l{\lambda }_l{x}_l\right)\leftrightarrow -\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})log{p}_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1}).$$ (17)

After substituting Equations (12), (16) and (17) into Equation (10), we get:

$$\begin{array}{cc}\hfill S_{j_1\dots j_{t-1}|j_t}& =-\sum _{a_1,\dots ,a_t}{p}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)log{p}_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\hfill \\ \hfill & \le -\sum _{a_1,\dots ,a_{t-1}}{p}_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})log{p}_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1}),\hfill \end{array}$$ (18)

or,

$$S_{j_1\dots j_{t-1}|j_t}\le S_{j_1\dots j_{t-1}}.$$ (19)

This means that the uncertainty of the distribution on the columns $j_1,\dots ,j_{t-1}$ cannot be increased when we have previous information on the distribution of column $j_t$.

From Equations (13) and (19), we then write:

$$S_{j_1\dots j_t}\le S_{j_1\dots j_{t-1}}+S_{j_t},$$ (20)

and by iteration we get the Khinchin–Shannon inequality for the Gibbs–Shannon entropy measure:

$$S_{j_1\dots j_t}\le \sum _{l=1}^t{S}_{j_l}.$$ (21)

The usual meaning given to Equation (21) is that the minimum of the information to be obtained from the analysis of the joint probabilities of a set of t columns is given by the sum of the informations associated with the t columns if considered as independent [1,2,10]. This is also seen as an aspect of Synergy [12,13] of the distribution of probabilities of occurrence.

4. The Assumption of Concavity and the Synergy of Sharma–Mittal (SM) Entropy Measures. The GKS Inequalities

We shall now use the assumption of concavity given by Equation (10) on Sharma–Mittal (SM) entropy measures:

$${\left(SM\right)}_{j_1\dots j_t}=\frac{{\left({\alpha }_{j_1\dots j_t}\right)}^{\frac{1-r}{1-s}}-1}{1-r},$$ (22)

where,

$${\alpha }_{j_1\dots j_t}=\sum _{a_1,\dots ,a_t}{\left[p_{j_1\dots j_t}(a_1,\dots ,a_t)\right]}^s,$$ (23)

and r, s are non-dimensional parameters.

Analogously to Equation (12), we also introduce the “conditional entropy measure”

$${\left(SM\right)}_{j_1\dots j_{t-1}|j_t}=\frac{{\left({\beta }_{j_1\dots j_{t-1}|j_t}\right)}^{\frac{1-r}{1-s}}-1}{1-r},$$ (24)

where

$${\beta }_{j_1\dots j_{t-1}|j_t}=\sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s,$$ (25)

and ${\widehat{p}}_{j_t}\left(a_t\right)$ stands for the escort probability

$${\widehat{p}}_{j_t}\left(a_t\right)=\frac{{\left(p_{j_t}\left(a_t\right)\right)}^s}{{\alpha }_{j_t}}=\frac{{\left(p_{j_t}\left(a_t\right)\right)}^s}{{\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s}.$$ (26)

We have in general:

$${\widehat{p}}_{j_1\dots j_t}(a_1,\dots ,a_t)=\frac{{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s}{{\alpha }_{j_1\dots j_t}}=\frac{{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s}{{\displaystyle \sum _{a_1,\dots ,a_t}}{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s}.$$ (27)

The inverse transformations are given by:

$$p_{j_t}\left(a_t\right)=\frac{{\left({\widehat{p}}_{j_t}\left(a_t\right)\right)}^{1/s}}{{\displaystyle \sum _{a_t}}{\left({\widehat{p}}_{j_t}\left(a_t\right)\right)}^{1/s}},$$ (28)

$$p_{j_1\dots j_t}(a_1,\dots ,a_t)=\frac{{\left({\widehat{p}}_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^{1/s}}{{\displaystyle \sum _{a_1,\dots ,a_t}}{\left({\widehat{p}}_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^{1/s}},$$ (29)

with

$$\sum _{a_t}{p}_{j_t}\left(a_t\right)=1=\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right),$$ (30)

$$\sum _{a_1,\dots ,a_t}{p}_{j_1\dots j_t}(a_1,\dots ,a_t)=1=\sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_1\dots j_t}(a_1,\dots ,a_t).$$ (31)

A range of variation for the parameters r, s of Sharma–Mittal entropies, Equation (22), should be derived from a requirement for strict concavity. In order to do so, let us remember that for each set of t columns ($m\times t$ subarray) there are m rows of t values each (t-sequences). We now denote these t-sequences by:

$$(a_1^{q_{\mu }},\dots ,a_t^{q_{\mu }}),\mu =1,\dots ,m$$ (32)

A sufficient requirement for strict concavity is the negative definiteness of the quadratic form associated to the Hessian matrix [14], whose elements are given by:

$$H_{q_{\mu }{q}_{\nu }}=\frac{{\partial }^2{\left(SM\right)}_{j_1\dots j_t}}{\partial p_{j_1\dots j_t}(a_1^{q_{\mu }},\dots ,a_t^{q_{\mu }})\partial p_{j_1\dots j_t}(a_1^{q_{\nu }},\dots ,a_t^{q_{\nu }})},\mu ,\nu =1,\dots ,m.$$ (33)

We then consider the m submatrices along the diagonal of the Hessian matrix. Their determinants should be alternately negative or positive according to whether their order is odd or even [15], respectively:

$$det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1)=s{\left({\alpha }_{j_1\dots j_t}\right)}^{\frac{s-r}{1-s}}{\left[p_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})\right]}^{s-2}\left[\frac{s(s-r)}{{(1-s)}^2}·{\widehat{p}}_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})-1\right],$$ (34)

$$\begin{array}{cc}\hfill det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1,2)=-s^2{\left({\alpha }_{j_1\dots j_t}\right)}^{2\left(\frac{s-r}{1-s}\right)}{\left[p_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})·p_{j_1\dots j_t}(a_1^{q_2},\dots ,a_t^{q_2})\right]}^{s-2}& \hfill \\ \hfill \left\{\frac{s(s-r)}{{(1-s)}^2}·\left[{\widehat{p}}_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})+{\widehat{p}}_{j_1\dots j_t}(a_1^{q_2},\dots ,a_t^{q_2})\right]-1\right\},& \hfill \end{array}$$ (35)

$$\begin{array}{cc}\hfill det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1,2,3)=s^3{\left({\alpha }_{j_1\dots j_t}\right)}^{3\left(\frac{s-r}{1-s}\right)}{\left[p_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})·p_{j_1\dots j_t}(a_1^{q_2},\dots ,a_t^{q_2})·p_{j_1\dots j_t}(a_1^{q_3},\dots ,a_t^{q_3})\right]}^{s-2}& \hfill \\ \hfill ·\left\{\frac{s(s-r)}{{(1-s)}^2}·\left[{\widehat{p}}_{j_1\dots j_t}(a_1^{q_1},\dots ,a_t^{q_1})+{\widehat{p}}_{j_1\dots j_t}(a_1^{q_2},\dots ,a_t^{q_2})+{\widehat{p}}_{j_1\dots j_t}(a_1^{q_3},\dots ,a_t^{q_3})\right]-1\right\}.& \hfill \end{array}$$ (36)

We then choose:

$$1>r\ge s>0,$$ (37)

and we have from Equations (34)–(36):

$$det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1)<0,$$ (38)

$$det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1,2)>0,$$ (39)

$$det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1,2,3)<0.$$ (40)

We then have generally:

$$\begin{array}{cc}\hfill det{H}_{q_{\mu }{q}_{\nu }}(\mu ,\nu =1,\dots ,m)={(-1)}^{m-1}·s^m{\left({\alpha }_{j_1\dots j_t}\right)}^{m\left(\frac{s-r}{1-s}\right)}{\left[\prod _{\mu =1}^m{p}_{j_1\dots j_t}(a_1^{q_{\mu }},\dots ,a_t^{q_{\mu }})\right]}^{s-2}& \hfill \\ \hfill \left\{\frac{s(s-r)}{{(1-s)}^2}\sum _{\mu =1}^m{\widehat{p}}_{j_1\dots j_t}(a_1^{q_{\mu }},\dots ,a_t^{q_{\mu }})-1\right\}.\end{array}$$ (41)

This completes the proof.

From Bayes’ law, Equation (6) and from Equations (8), (22)–(25), we can write:

$${\left(SM\right)}_{j_1\dots j_t}={\left(SM\right)}_{j_1\dots j_{t-1}|j_t}+{\left(SM\right)}_{j_t}+(1-r){\left(SM\right)}_{j_1\dots j_{t-1}|j_t}·{\left(SM\right)}_{j_t}.$$ (42)

We are now ready to use the concavity assumption, Equation (10) for deriving the GKS inequalities. In order to do so, we make the correspondences:

$$\begin{array}{c}{\lambda }_l\leftrightarrow {\widehat{p}}_{j_t}\left(a_t\right);x_l\leftrightarrow p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t),\\ f\left(x_l\right)\leftrightarrow {\displaystyle \sum _{a_1,\dots ,a_{t-1}}}{\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s.\end{array}$$ (43)

We can then write:

$$\sum _l{\lambda }_l{x}_l\leftrightarrow \sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t),$$ (44)

$$\sum _l{\lambda }_{l}f\left(x_l\right)\leftrightarrow \sum _{a_t,a_1,\dots ,a_{t-1}}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s,$$ (45)

and

$$f\left(\sum _l{\lambda }_l{x}_l\right)\leftrightarrow \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s.$$ (46)

With the correspondences above, Equation (10) will turn into:

$$\sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s\ge \sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s.$$ (47)

An additional information should be taken into consideration before we derive the GKS inequalities:

On each $j_t$ column, $t=1,\dots ,n$ of a $m\times n$ block, there will be values $a_t^{\prime }$, $a_t^{″}$ of $a_t$ such that

$$p_{j_t}\left(a_t^{\prime }\right)\ge {\widehat{p}}_{j_t}\left(a_t^{\prime }\right),$$ (48)

and

$$p_{j_t}\left(a_t^{″}\right)\le {\widehat{p}}_{j_t}\left(a_t^{″}\right).$$ (49)

After multiplying inequalities (48) and (49) by $p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime })$ and $p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″})$, respectively, and summing up in $a_t^{\prime }$ and $a_t^{″}$, respectively, we get:

$$\sum _{a_t^{\prime }}{p}_{j_t}\left(a_t^{\prime }\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime })\ge \sum _{a_t^{\prime }}{\widehat{p}}_{j_t}\left(a_t^{\prime }\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime }),$$ (50)

and

$$\sum _{a_t^{″}}{p}_{j_t}\left(a_t^{″}\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″})\le \sum _{a_t^{″}}{\widehat{p}}_{j_t}\left(a_t^{″}\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″}).$$ (51)

From Equations (48) and (49), any sum over the $a_t$ values can be partitioned as sums over the sets of values $a_t^{\prime }$ and $a_t^{″}$:

$$\sum _{a_t}{p}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)=\sum _{a_t^{\prime }}{p}_{j_t}\left(a_t^{\prime }\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime })+\sum _{a_t^{″}}{p}_{j_t}\left(a_t^{″}\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″}).$$ (52)

Substituting Equation (52) into Equations (50) and (51), we have:

$$\begin{array}{cc}\hfill \sum _{a_t}& p_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)-\sum _{a_t^{″}}{p}_{j_t}\left(a_t^{″}\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″})\hfill \\ \hfill & \ge \sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)-\sum _{a_t^{″}}{\widehat{p}}_{j_t}\left(a_t^{″}\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″}),\hfill \end{array}$$ (53)

and

$$\begin{array}{cc}\hfill \sum _{a_t}& p_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)-\sum _{a_t^{\prime }}{p}_{j_t}\left(a_t^{\prime }\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime })\hfill \\ \hfill & \le \sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)-\sum _{a_t^{\prime }}{\widehat{p}}_{j_t}\left(a_t^{\prime }\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime }),\hfill \end{array}$$ (54)

respectively.

After applying the Bayes’ law, Equation (6), to the first term on the left hand side of Equations (53) and (54), we get:

$$p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\ge \sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{″},$$ (55)

$$p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\le \sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{\prime },$$ (56)

where

$$B^{″}=\sum _{a_t^{″}}\left(p_{j_t}\left(a_t^{″}\right)-{\widehat{p}}_{j_t}\left(a_t^{″}\right)\right)·p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{″}),$$ (57)

$$B^{\prime }=\sum _{a_t^{\prime }}\left(p_{j_t}\left(a_t^{\prime }\right)-{\widehat{p}}_{j_t}\left(a_t^{\prime }\right)\right)·p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t^{\prime }),$$ (58)

and we have, $B^{″}\le 0$, $B^{\prime }\ge 0$, according to Equations (48) and (49), respectively.

After taking the s-power in Equations (55) and (56) and summing up in $a_1,\dots ,a_{t-1}$, we have:

$$\sum _{a_1,\dots ,a_{t-1}}{\left[p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\right]}^s\ge \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{″}\right]}^s,$$ (59)

$$\sum _{a_1,\dots ,a_{t-1}}{\left[p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\right]}^s\le \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{\prime }\right]}^s.$$ (60)

We now write, the concavity assumption, Equation (10), as:

$$C\ge D,$$ (61)

where

$$C\equiv \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s,$$ (62)

and

$$D\equiv \sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s.$$ (63)

Equations (59) and (60) are now written as:

$$A\ge C^{″},$$ (64)

$$A\le C^{\prime },$$ (65)

where

$$A\equiv \sum _{a_1,\dots ,a_{t-1}}{\left[p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\right]}^s,$$ (66)

and

$$C^{″}\equiv \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{″}\right]}^s,$$ (67)

$$C^{\prime }\equiv \sum _{a_1,\dots ,a_{t-1}}{\left[\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)+B^{\prime }\right]}^s.$$ (68)

Since $B^{″}\le 0$ and $B^{\prime }\ge 0$, we have trivially that:

$$C^{″}\le C,$$ (69)

and

$$C^{\prime }\ge C.$$ (70)

The set of inequalities, Equations (61), (64) and (69), or

$$C\ge D;A\ge C^{″};C^{″}\le C,$$ (71)

and the set of inequalities, Equations (61), (65) and (70), or

$$C\ge D;A\le C^{\prime };C^{\prime }\ge C,$$ (72)

can be arranged as the chains of inequalities

$$C\ge A\ge D\ge C^{″},$$ (73)

and

$$C^{\prime }\ge A\ge C\ge D,$$ (74)

respectively.

The inequality $A\ge D$ is common to the two chains above and it can be written as:

$$\sum _{a_1,\dots ,a_{t-1}}{\left[p_{j_1\dots j_{t-1}}(a_1,\dots ,a_{t-1})\right]}^s\ge \sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s.$$ (75)

From the definition of the escort probabilities, Equations (26) and (27), we can write the right-hand side of Equation (75), as:

$$\sum _{a_1,\dots ,a_t}{\widehat{p}}_{j_t}\left(a_t\right){\left[p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right]}^s=\frac{{\displaystyle \sum _{a_1,\dots ,a_t}}{\left[p_{j_1\dots j_t}(a_1,\dots ,a_t)\right]}^s}{\sum _{a_t}{\left[p_{j_t}\left(a_t\right)\right]}^s}.$$ (76)

From Equations (75) and (76) and the definition of the $\alpha$-symbols, Equation (23), we have:

$${\alpha }_{j_1\dots j_t}\le {\alpha }_{j_1\dots j_{t-1}}·{\alpha }_{j_t}.$$ (77)

We then get by iteration,

$${\alpha }_{j_1\dots j_t}\le \prod _{l=1}^t{\alpha }_{j_l}.$$ (78)

Equation (78) do correspond to the Generalized Khinchin–Shannon Inequalities (GKS) here derived for Sharma–Mittal entropies.

From Equations (22) and (37) we can also write for the GKS inequalities:

$${\left(SM\right)}_{j_1\dots j_t}\le \frac{{\displaystyle \prod _{l=1}^t}\left[1+(1-r){\left(SM\right)}_{j_l}\right]-1}{1-r}.$$ (79)

The same words which have been written after Equation (21), could be written also here for the Sharma–Mittal entropy measures as the aspect of Synergy is concerned. We will introduce a proposal for information measure to stress this aspect on the next section.

For $t=2,3$, we can write from Equation (79):

$${\left(SM\right)}_{j_1{j}_2}\le {\left(SM\right)}_{j_1}+{\left(SM\right)}_{j_2}+(1-r){\left(SM\right)}_{j_1}{\left(SM\right)}_{j_2},$$ (80)

$$\begin{array}{cc}\hfill {\left(SM\right)}_{j_1{j}_2{j}_3}\le {\left(SM\right)}_{j_1}& +{\left(SM\right)}_{j_2}+{\left(SM\right)}_{j_3}+(1-r)[{\left(SM\right)}_{j_1}{\left(SM\right)}_{j_2}+{\left(SM\right)}_{j_2}{\left(SM\right)}_{j_3}\hfill \\ \hfill & +{\left(SM\right)}_{j_1}{\left(SM\right)}_{j_3}]+{(1-r)}^2{\left(SM\right)}_{j_1}{\left(SM\right)}_{j_2}{\left(SM\right)}_{j_3}.\hfill \end{array}$$ (81)

The Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropies are easily obtained by taking the convenient limits in Equation (24):

$$\underset{r\to s}{lim}{\left(SM\right)}_{j_1\dots j_t}={\left(HC\right)}_{j_1\dots j_t}=\frac{{\alpha }_{j_1\dots j_t}-1}{1-s},$$ (82)

$$\underset{r\to 1}{lim}{\left(SM\right)}_{j_1\dots j_t}=R_{j_1\dots j_t}=\frac{log\left({\alpha }_{j_1\dots j_t}\right)}{1-s},$$ (83)

$$\underset{r\to 2-s}{lim}{\left(SM\right)}_{j_1\dots j_t}={\left(LV\right)}_{j_1\dots j_t}=\frac{{\alpha }_{j_1\dots j_t}-1}{(1-s){\alpha }_{j_1\dots j_t}}.$$ (84)

The Gibbs–Shannon entropy measure $S_{j_1\dots j_t}$, Equation (11), is included in all these entropies through:

$$\underset{s\to 1}{lim}{\left(HC\right)}_{j_1\dots j_t}=\underset{s\to 1}{lim}{R}_{j_1\dots j_t}=\underset{s\to 1}{lim}{\left(LV\right)}_{j_1\dots j_t}=S_{j_1\dots j_t}.$$ (85)

Equations (83) and (85) have been derived via the l’Hôpital theorem.

For $t=2,3$, we write from Equations (82)–(84):

$${\left(HC\right)}_{j_1{j}_2}\le {\left(HC\right)}_{j_1}+{\left(HC\right)}_{j_2}+(1-s){\left(HC\right)}_{j_1}{\left(HC\right)}_{j_2},$$ (86)

$$\begin{array}{cc}\hfill {\left(HC\right)}_{j_1{j}_2{j}_3}\le {\left(HC\right)}_{j_1}& +{\left(HC\right)}_{j_2}+{\left(HC\right)}_{j_3}+(1-s)[{\left(HC\right)}_{j_1}{\left(HC\right)}_{j_2}+{\left(HC\right)}_{j_2}{\left(HC\right)}_{j_3}\hfill \\ \hfill & +{\left(HC\right)}_{j_1}{\left(HC\right)}_{j_3}]+{(1-s)}^2{\left(HC\right)}_{j_1}{\left(HC\right)}_{j_2}{\left(HC\right)}_{j_3},\hfill \end{array}$$ (87)

$$R_{j_1{j}_2}\le R_{j_1}+R_{j_2},$$ (88)

$$R_{j_1{j}_2{j}_3}\le R_{j_1}+R_{j_2}+R_{j_3},$$ (89)

$${\left(LV\right)}_{j_1{j}_2}\le {\left(LV\right)}_{j_1}+{\left(LV\right)}_{j_2}-(1-s){\left(LV\right)}_{j_1}{\left(LV\right)}_{j_2},$$ (90)

$$\begin{array}{cc}\hfill {\left(LV\right)}_{j_1{j}_2{j}_3}\le {\left(LV\right)}_{j_1}& +{\left(LV\right)}_{j_2}+{\left(LV\right)}_{j_3}-(1-s)[{\left(LV\right)}_{j_1}{\left(LV\right)}_{j_2}+{\left(LV\right)}_{j_2}{\left(LV\right)}_{j_3}\hfill \\ \hfill & +{\left(LV\right)}_{j_1}{\left(LV\right)}_{j_3}]+{(1-s)}^2{\left(LV\right)}_{j_1}{\left(LV\right)}_{j_2}{\left(LV\right)}_{j_3}.\hfill \end{array}$$ (91)

As a last result of this section, we note that Equation (79) could be also derived from Equation (75), since this equation could be also written as:

$${\left(SM\right)}_{j_1\dots j_{t-1}|j_t}\le {\left(SM\right)}_{j_1\dots j_{t-1}},$$ (92)

where we have used Equations (22)–(25).

After the comparison of Equation (42) and (92), we get the result of Equation (79) again.

5. An Information Measure Proposal Associated to Sharma–Mittal Entropy Measures

We are looking for a proposal of information measure which can fulfill a requirement of clear interpretation of the upsurge of Synergy in a probabilistic distribution and is supported by the usual idea of entropy as a measure of uncertainty.

For the Sharma–Mittal set of entropy measures the proposal for the associated information measure would be:

$$I_{j_1\dots j_t}=-\frac{{\left(SM\right)}_{j_1\dots j_t}}{{\left({\alpha }_{j_1\dots j_t}\right)}^{\frac{1-r}{1-s}}},1>r\ge s>0,$$ (93)

where ${\left(SM\right)}_{j_1\dots j_t}$ and ${\alpha }_{j_1\dots j_t}$ are given by Equations (22) and (23). We then have from Equation (93)

$${\left(SM\right)}_{j_1\dots j_t}=-\frac{I_{j_1\dots j_t}}{1+(1-r)I_{j_1\dots j_t}}.$$ (94)

From the GKS inequalities, Equation (79), and from Equation (94), we get:

$$I_{j_1\dots j_t}\ge \frac{{\displaystyle \prod _{l=1}^t}\left[1+(1-r)I_{j_l}\right]-1}{1-r}\ge {\displaystyle \sum _{l=1}^t}{I}_{j_l}.$$ (95)

The meaning of Equation (95) is that the minimum of information associated with t columns of probabilities of occurrence is given by the sum of informations associated to each column. This corresponds to the expression of Synergy of the distribution of probabilities of occurrences which we have derived on the previous section.

The inequalities (95), for $t=2,3$ are written as:

$$I_{j_1{j}_2}\ge I_{j_1}+I_{j_2}+(1-r)I_{j_1}{I}_{j_2}\ge I_{j_1}+I_{j_2},$$ (96)

$$\begin{array}{cc}\hfill I_{j_1{j}_2{j}_3}& \ge I_{j_1}+I_{j_2}+I_{j_3}+(1-r)\left[I_{j_1}{I}_{j_2}+I_{j_2}{I}_{j_3}+I_{j_1}{I}_{j_3}\right]+{(1-r)}^2{I}_{j_1}{I}_{j_2}{I}_{j_3}\hfill \\ \hfill & \ge I_{j_1}+I_{j_2}+I_{j_3}.\hfill \end{array}$$ (97)

It seems worthwhile to derive yet another result which unveils once more the fundamental aspect of synergy of the distribution of probabilities of occurrence. From Equation (93), we have:

$$\mathrm{d}{I}_{j_1\dots j_t}=-\frac{\mathrm{d}{\left(SM\right)}_{j_1\dots j_t}}{{\left({\alpha }_{j_1\dots j_t}\right)}^{2\left(\frac{1-r}{1-s}\right)}},$$ (98)

and we then write from the GKS inequalities, Equation (78):

$$\frac{1}{{\left({\alpha }_{j_1\dots j_t}\right)}^{2\left(\frac{1-r}{1-s}\right)}}\ge \frac{1}{{\displaystyle \prod _{l=1}^t}{\left({\alpha }_{j_l}\right)}^{2\left(\frac{1-r}{1-s}\right)}}={\displaystyle \prod _{l=1}^t}\frac{1}{{\left({\alpha }_{j_l}\right)}^{2\left(\frac{1-r}{1-s}\right)}}.$$ (99)

We then get:

$$-\frac{\mathrm{d}{I}_{j_1\dots j_t}}{\mathrm{d}{\left(SM\right)}_{j_1\dots j_t}}\ge {\displaystyle \prod _{l=1}^t}\left(-\frac{\mathrm{d}{I}_{j_l}}{\mathrm{d}{\left(SM\right)}_{j_l}}\right).$$ (100)

Equation (100) do correspond to another result which originates from the Synergy of the distribution of probabilities of occurrence. It can be written as: The minimum of the rate of information increase with decreasing entropy in probability distribution for sets of t columns, is given by the product of the rates of information increase pertaining to each of the t columns.

6. The Use of Hölder’s Inequality for an Alternative Derivation of the GKS Inequalities

We firstly note that:

$$\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)=\frac{{\displaystyle \sum _{a_t}}{p}_{j_1\dots j_t}(a_1,\dots ,a_t){\left(p_{j_t}\left(a_t\right)\right)}^{s-1}}{{\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s},$$ (101)

and we now introduce the Hölder’s inequality: Ref. [11]

$${\left(\sum _l{x}_l{y}_l\right)}^{ab}\le {\left(\sum _l{\left(x_l\right)}^a\right)}^b·{\left(\sum _l{\left(y_l\right)}^b\right)}^a,ab=a+b,a\ne 1.$$ (102)

We can also write:

$$\sum _l{x}_l{y}_l\ge {\left(\sum _l{\left(x_l\right)}^a\right)}^{\frac{1}{a}}·{\left(\sum _l{\left(y_l\right)}^b\right)}^{\frac{1}{b}},0\le a<1,ab\le 0,$$ (103)

or

$${\left(\sum _l{x}_l{y}_l\right)}^a\ge \sum _l{\left(x_l\right)}^a·{\left(\sum _l{\left(y_l\right)}^b\right)}^{\frac{a}{b}},0\le a<1,b\le 0.$$ (104)

We now make the correspondences:

$$x_l\leftrightarrow p_{j_1\dots j_t}(a_1,\dots ,a_t);y_l\leftrightarrow {\left(p_{j_t}\left(a_t\right)\right)}^{s-1}.$$ (105)

We take the s-power of the sides of Equation (101) and after using Equations (104) and (105) with $a=s$ and $b=\frac{s}{s-1}$, we get:

$$\begin{array}{cc}\hfill {\left(\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_{t-1}|a_t)\right)}^s& ={\left(\frac{{\displaystyle \sum _{a_t}}{p}_{j_1\dots j_t}(a_1,\dots ,a_t){\left(p_{j_t}\left(a_t\right)\right)}^{s-1}}{{\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s}\right)}^s\hfill \\ \hfill & \ge \frac{\left({\displaystyle \sum _{a_t}}{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s\right){\left({\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s\right)}^{s-1}}{{\left({\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s\right)}^s}\hfill \\ \hfill & =\frac{{\displaystyle \sum _{a_t}}{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s}{{\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s}.\hfill \end{array}$$ (106)

After summing up in $a_1,\dots ,a_{t-1}$, we then have:

$$\sum _{a_1,\dots ,a_{t-1}}{\left(\sum _{a_t}{\widehat{p}}_{j_t}\left(a_t\right)p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s\ge \frac{{\displaystyle \sum _{a_1,\dots ,a_t}}{\left(p_{j_1\dots j_t}(a_1,\dots ,a_t)\right)}^s}{{\displaystyle \sum _{a_t}}{\left(p_{j_t}\left(a_t\right)\right)}^s}.$$ (107)

From the application of Bayes’ law to the right-hand side of Equation (107), we get Equation (47) again, i.e., the correspondence of the assumption of concavity has been derived once more. All the subsequent development of Section 4 will then follows accordingly.

7. Concluding Remarks

It should be stressed that the introduction of escort probabilities has been efficient on the construction of generalized entropy measures. These can be used for the classification of databases in terms of their clustering as driven by their intrinsic synergy and the resulting formation of more complex structures like families and clans.

A fundamental aim would be the derivation of a dynamical theory which would be able to describe the process of formation of these structures. A theory based on the evolution of the entropy values on databases which we hope that could be realized by methods introduced by the exhaustive study of Fokker–Planck equations.

Some introductory results on this promising line of research have been already published [16] and a forthcoming publication of a comprehensive review will summarize all of them.

Abbreviations

The following abbreviations are used in this manuscript:

GKS	Generalized Khinchin–Shannon
SM	Sharma–Mittal

Author Contributions

Conceptualization, R.P.M. and S.C.d.A.N.; methodology, R.P.M. and S.C.d.A.N.; formal analysis, R.P.M. and S.C.d.A.N.; writing—original draft preparation, R.P.M.; writing—review and editing, R.P.M. and S.C.d.A.N.; visualization, R.P.M. and S.C.d.A.N.; supervision, R.P.M.; project administration, R.P.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.