Table 1.

Concepts of entropy and MI defined by Shannon's theory of information.

Concepts of Shannon's theory of information	Descriptions
$H(X)=-{\displaystyle \sum }_x^{}p(x){\log \hspace{0.17em}}_2\hspace{0.17em}p(x)$	The uncertainty of a random variable X is measured by its entropy H(X); p(x) is the probability density of X
$H(X\mid Y)=-{\displaystyle \sum }_x^{}p(x\mid y){\log \hspace{0.17em}}_2\hspace{0.17em}p(x\mid y)$	The uncertainty of a random variably X given knowledge of another random variable Y is measured by the conditional entropy H(X ∣ Y)
$H(X,Y)=-{\displaystyle \sum }_x^{}{\displaystyle \sum }_y^{}p(x,y)\text{lo}{\text{g}}_{2}p(x,y)$	The uncertainty of a pair of random variables X, Y is measured by the entropy H(X ∣ Y)
H(X, Y) = H(X) + H(Y ∣ X) = H(Y) + H(X ∣ Y)
$\text{MI}(X;Y)={\displaystyle \sum }_x^{}{\displaystyle \sum }_y^{}p(x,y)\text{lo}{\text{g}}_2\frac{p(x,y)}{p(x)p(y)}$	Given two random variables X and Y, the amount of information that each one of them provides about the other is the mutual information MI(X; Y)
MI(X; Y) = H(X) + H(Y) − H(X, Y)