Shannon's monotonicity problem for free and classical entropy

Abstract

We give a short unified proof of the following theorem, valid in the context of both classical probability theory and Voiculescu's free probability theory: let (X_j⁽¹⁾, …, X_j⁽ⁿ⁾) be independent (resp., freely independent) n-tuples of random variables. Let Z_N^(p) = N^−1/2(X₁^(p) + … + X_N^(p)) be their central limit sums. Then the entropy (resp., free entropy) of the n-tuple (Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾) is a monotone function of N. The classical case (for n = 1) is a celebrated result of Artstein, Ball, Barthe, and Naor, and our proof is an adaptation and simplification of their argument.

Voiculescu's free probability theory (1) is an amazing non-commutative parallel to classical probability theory. Classical probability theory deals with random variables X₁, X₂, … which are commutative variables: X_iX_j = X_jX_i. Associated to any polynomial P in the variables X₁, X₂, … one has its expected value E(P). A typical model for random variables are functions X_j ∈ L^∞ (𝔛, μ) on a measure space 𝔛 endowed with a positive measure μ satisfying μ(𝔛) = 1. Then E(P(X₁,X₂, …) = ∫ P(X₁(σ), X₂(σ), …)dμ(σ). Two groups of variables X₁,X₂, … and Y₁,Y₂ … are independent if E(PQ) = E(P)E(Q) whenever P is a polynomial depending only on X₁,X₂, … and Q is a polynomial depending only on Y₁,Y₂….

In the noncommutative setting, one again has random variables, only now they are not assumed to commute; yet one still can associate an expected value τ(P) to any (noncommutative) polynomial P in X₁,X₂…. A prototypical example is that of bounded operators X_j on a Hilbert space, with expectation functional given by τ(P) = 〈P(X₁,X₂, …)ξ,ξ〉 for some fixed ξ ∈ H of unit norm. An amazing discovery made by Voiculescu (2) is that, in addition to a straightforward extension of the notion of independence, the noncommutative setting allows for a new form of independence, called free independence (see ref. 1 for an introduction). The notion of free independence and free probability theory plays an important role in the description of large N asymptotics of random multimatrix models.

Free probability thus provides a noncommutative parallel to classical probability theory; strangely, many statements and notions from classical probability have their free analogs (though the correspondence is far from being straightforward: Some classical statements fail to have free analogs, while others have stronger free analogs than one would expect). An example of this correspondence is the theory of free entropy and free information (3), which in many respects parallels the classical theory pioneered by Shannon.

This article is devoted to giving a proof of monotonicity of free and classical entropy computed on central limit sums of n-tuples of random variables. The classical statement was conjectured by Shannon in the 50s and proved (at least in the n = 1 case) by ref. 4. In trying to adapt their proof to the free context, we discovered a simplification which at the same time allowed us to give a unified proof for the free and classical statement (5). In the present paper, we extend our proof to the case of n variables. Appendix 1, by Hanne Schultz, characterizes strict monotonicity.

Background

The Classical Case.

Let X₁, …, X_n, … be independent identically distributed (iid) random variables, and assume that each X_j is centered [i.e., its expectation E(X_j) is zero] and that the variance E(X_j²) = σ² is fixed. The classical central limit theorem then states that the central limit sums

coverge in law to a Gaussian variable G with variance σ², $d{\mu }_G=(1/\sqrt{2\pi \sigma })\exp (-x^2/{\sigma }^2)dx$. Recall that if the law of a random variable X has density dμ_X(x) = p(x)dx then the entropy of X is defined to be

If the law of X is not Lebesgue absolutely continuous, the entropy is defined to be +∞. The entropy H(X) is a measure of how far the law of X is from that of a Gaussian variable. In fact, if E(X) = 0 and E(X²) = σ², then H(X) ≥ H(G), with equality iff μ_X = μ_G. One can thus expect that H(Z_N) approaches H(G) as N → ∞ (6). This is in general false (e.g., the measure associated to Z_N could be atomic for all N). The following conjecture, going back to Shannon (see refs. 7–9), was proved by Artstein, Ball, Barthe, and Naor (4):

Theorem.

(See ref. 4.) The function N → H(Z_N) is monotone nonincreasing in N.

In the case of several variables X₁, …, X_n, their joint entropy can be expressed in terms of the joint law p(x₁, …, x_n)dx₁…dx_n by the formula

We prove the following generalization of the main result of ref. 4 for n-tuples:

Theorem 1.

Let (X_j⁽¹⁾, …, X_j^(p)) be a sequence of p-tuples of random variables, so that {(X_j⁽¹⁾, …, X_j^(p)) : j = 1,2, …} are independent and identically distributed and have finite second moments. Let Z_N^(k) = (X₁^(k)+ … +X_N^(k))/$\sqrt{N}$. Then the function N → H(Z_N⁽¹⁾, …, Z_N^(p)) is monotone nonincreasing.

Note that (Z_N⁽¹⁾, …, Z_N^(p)) → (G⁽¹⁾, …, G^(p)) in law, where G^(j) are correlated Gaussians determined by E(G⁽ⁱ⁾G^(j)) = E(X₁⁽ⁱ⁾X₁^(j)).

A version of this theorem also holds for conditionally independent n-tuples (see Relative Entropy before Appendix 1).

The Free Case.

We now describe the case of freely independent random variables (see ref. 1 for basic definitions).

Let X₁, …, X_n, … ∈ (M, τ) be freely independent identically distributed random variables, so that each X_j is centered and τ(X_j²) = σ². Consider their central limit sums $Z_N=(X_1+\dots +X_N)/\sqrt{N}$. According to Voiculescu's free central limit theorem (1, 2), Z_N → σS in law, where S is a random variable with the semicircular law $d\mu s(x)=(2/\pi )\sqrt{2-t^{2}dx}$.

For a single random variable X with law μ_X its free entropy was defined by Voiculescu (3, 10) to be

This quantity plays the role of the classical entropy H in free probability theory. An important difference, though, is its sign: χ(X) ∈ [−∞,∞).

For any random variable X with τ(X) = 0 and having variance τ(X²) = σ², τ(X) = 0, the free entropy of X satisfies χ(X) ≤ χ(S) with equality if and only if X has the same law as S.

Theorem.

(See ref. 5.) Let X_j be freely independent, identically distributed random variables, and let $Z_N=(X_1+\dots +X_N)/\sqrt{N}$. Then the function N → χ(Z_N) is monotone nondecreasing.

In the case of several variables X₁, …, X_n, Voiculescu introduced two definitions of free entropy, denoted χ(X₁, …, X_n) and χ*(X₁, …, X_n). It is not known whether these quantities are ever different; they are the same when n = 1. For n > 1, our proof works only for the free entropy χ* (see below for its definition).

Theorem 2.

Let (X_j⁽¹⁾, …, X_j^(p)) be a sequence of p-tuples of random variables, so that {(X_j⁽¹⁾, …, X_j^(p)): j = 1,2, …} are freely independent and identically distributed and have finite second moments. Let Z_N^(k) = (X₁^(k)+ … +X_N^(k))/$\sqrt{N}$. Then the function N → χ*(Z_N⁽¹⁾, …, Z_N^(p)) is monotone nondecreasing.

Preliminaries on Entropy

Let as before X₁, …, X_n be some (perhaps noncommutative) random variables in an operator algebra A equipped with a positive tracial linear functional τ : A → ℂ [thus we write τ(X) for the expected value E(X) if X is a classical random variable]. We recall the definitions of the score function, Fisher information and entropy in the classical case (see, e.g., ref. 11 and references therein) and the free case (see refs. 3 and 12).

The Classical Case.

Consider the derivations d_j : A → L²(A, τ) given by d_jX_k = 0 for j ≠ k and d_jX_j = 1. If 1 is in the domain of d_j*, one defines the score function by

In other words, for any function g ∈ L2(A, τ), one has 〈gj, f〉L2(A,τ) = 〈djf,1〉L2(A,τ). That is, ∫ fjgdμ(x1, …, xn) = ∫ ∂g/∂xjdμ(x1, …, xn), where μ is the joint law of X1, …, Xn. It is not hard to see that this exists iff μ is Lebesgue absolutely-continuous; and if p(x1, …, xn) is the density of μ, then 1 is in the domain of dj iff the following expression for fj is in L2(μ):

The Fisher information F(X₁, …, X_n) is then defined by the equation F(X₁, …, X_n) = Σ_j ‖f_j‖_L²(A,τ)². It turns out that the entropy of X₁, …, X_n is up to a universal constant the expression

where X_j^t = X_j + $\sqrt{t}$G_j and G₁, …, G_n are independent iid centered Gaussian random variables of variance 1, independent from X₁, …, X_n.

The Free Case.

Consider the derivations ∂_j: A → L²(A, τ)⊗̄L²(A, τ) given by ∂_jX_k = 0 for j ≠ k and ∂_jX_j = 1 ⊗ 1. If 1 ⊗ 1 is in the domain of ∂_j*, one defines the free score function (also called conjugate variable) by

In other words, for any g ∈ L²(A, τ), one has 〈 g, ξ_j〉 = 〈∂_j f,1 ⊗ 1〉_{L²(A,τ);⊗̄L²(A,τ)}. In the case that n = 1 one can check that 1 ⊗ 1 is in the domain of ∂ iff the law of X = X₁ is Lebesgue absolutely continuous with density p, for which the Hilbert transform H p lies in L²(μ). There is no explicit description of ξ_j in the case that n > 1, since the joint law of several noncommutative random variables is no longer encoded by a measure.

One defines the free Fisher information by Φ* (X₁, …, X_n) = Σ_j‖ξ_j‖_L²(A,τ)². The free entropy χ* of X₁, …, X_n is up to a universal constant the expression

where X_j^t = X_j + $\sqrt{t}$S_j and S₁, …, S_n are freely independent iid centered semicircular random variables of variance 1, freely independent from X₁, …, X_n.

In the case that n = 1, this definition of free entropy χ*(X) coincides with the free entropy χ(X) defined in Eq. 1.

Properties of Free and Classical Fisher Information

Lemma 3.

Assume that Z is freely independent (resp., classically independent) from X,Y₁, …, Y_m. Then one has the equality J(X : Y₁, …, Y_m,Z) = J(X : Y₁, …, Y_m) [resp., f(X : Y₁, …, Y_m,Z) = f(X : Y₁, …, Y_m)].

We refer the reader to e.g., ref. 12, for the proof in the free case (the proof in the classical case is immediate from the explicit formula for the score function).

Lemma 4.

Assume that {X_j^(k)}, k = 1, …, n, j = 1,2, … are (noncommutative) random variables. Then for each j = 1,2, …, N + 1 and each k = 1,2, …, n one has:

assuming that the score function appearing on the right-hand side of the respective equation exists.

Proof: Let Y_k = Σ_i=1^N+1 X_i^(k), Y′_k = Σ_i≠j X_i^(k). Thus Y_k = Y′_k + X_j^(k). Let P be a polynomial in Y₁, …, Y_n, viewed also as a polynomial in Y′₁, …, Y′_n,X₁^(k), …, X_j^(k). Then

Indeed, the values of a derivation on an arbitrary polynomial P are determined by the Leibnitz rule and the values of the derivation on the generators. However, one has

and similarly for d. It follows that for any such P,

which, in view of the fact that P ∈ W* (Σ_i=1^N+1 X_i^(r) : r = 1, 2, …, n), proves the lemma.

Monotonicity for Fisher Information

Theorem 5.

Let (X_j⁽¹⁾, …, X_j^(p)) be a sequence of p-tuples of random variables, so that {(X_j⁽¹⁾, …, X_j^(p)) : j = 1,2, …} are classically (resp., freely) independent and identically distributed and have finite second moments. Let Z_N^(k) = X₁^(k)+ … +X_N^(k)/$\sqrt{N}$. Then the function N → F(Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾) (resp., N → Φ* (Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾)) is monotonenon-increasing.

Proof: We give the details in the free case. The argument in the classical case is the same, if one replaces everywhere J by f and Φ* by F and free independence by classical independence.

Let M = W*(Z_N+1⁽¹⁾, …, Z_N+1⁽ⁿ⁾). Then using Lemma 3, we have that for all k,

where in the last line we used Lemma 3 and free independence of X_j^(k) and {X_i^(k)x}_i≠j. Thus,

Since E_M is a contraction on L² we obtain that

where

Now let M_j = W* (X_j⁽¹⁾, …, X_j⁽ⁿ⁾), M = W* ({M_j}_j=1^N+1) and E_j : M → Q_j = *_i≠jM_i be the conditional expectation. Then E_j : L²(M) → L²(M) are projections and moreover {E_j : j = 1, …, N + 1} form a commuting family. Indeed, because of the freeness assumptions, we may write

Hence if i < j,

In particular, note that E₁ ∘ … ∘ E_n = τ. Since ζ_j ∈ Q_j, ζ_j = E_jζ_j and τ(ζ_j) = 〈1, ζ_j〉 = 〈∂_{Σj≠iXi^(k):{ZN+1^(r):r≠k}} 1,1 ⊗ 1〉 = 0, we may now apply lemma 5 in ref. 4 to conclude that

On the other hand, because the joint distribution of (X_i⁽¹⁾, …, X_i⁽ⁿ⁾) does not depend on i, we find that the L² norm of ζ_j is the same for all j; hence

Combining this with the previous estimates (Eqs. 2 and 3) we obtain

By summing over k we obtain the inequality Φ*(Z_N+1⁽¹⁾, …, Z_N+1⁽ⁿ⁾) ≤ Φ* (Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾).

Proof of Theorems 1 and 2.

We now show that Theorem 5 implies Theorem 1 and Theorem 2. Indeed, let X_j^(k,t) = X_j^(k) + $\sqrt{t}$Y_j^(k), where {Y_j^(k)}_j,k are independent, iid centered Gaussian random variables (resp., freely iid centered semicircular variables) of variance 1, which are independent (resp., freely independent) from {X_j^(k)}_j,k. Let Z_N^(k,t) = N^−1/2(X₁^(k,t) + … +X_N^(k,t)). Applying Theorem 5 for fixed t gives that

(and similarly for Φ* in the free case). But Z_N^(k,t) = Z_N^(k) + $\sqrt{t}$Y^(N,k), where for each fixed N, Y^(N,k) = N^−1/2(Y₁^(k) + … + Y_N^(k)), k = 1, …, n, is a family of centered iid Gaussian random variables (resp., centered freely iid semicircular variables), independent (resp., freely independent) from {Z_N^(k)}_k and having variance 1. Hence

implying Theorem 1. The argument is the same in the free case, except for a change of sign in the definition of entropy.

Relative Entropy.

The same argument implies the following result: for an arbitrary von Neumann algebra B, keeping the notation of Theorem 1 (resp., Theorem 2), and assuming that the n-tuples {(X_j⁽¹⁾, …, X_j⁽ⁿ⁾): j = 1,2, …} are conditionally independent over B (resp., free with amalgamation over B), the relative entropy H(Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾ : B) (resp., the relative free entropy, χ*(Z_N⁽¹⁾, …, Z_N⁽ⁿ⁾ : B)) is monotone nonincreasing (resp., monotone nondecreasing).

Abbreviation

iid

independent identically distributed.

Appendix 1

In the case of single random variables, we characterize, in the case n = 1, when equality holds in Theorem 1 or Theorem 2.

Theorem 6.

Let X₁, …, X_N+1 be freely iid bounded with χ(X₁) > −∞. If

then X₁ is semicircular.

Theorem 7.

Let X₁, …, X_N+1 be iid square integrable random variables with H(X₁) < + ∞. If

then X₁ is Gaussian.

Proof in the Free Case.

We first note that if we assume that if we assume that Eq. 4 holds, then by writing entropy as an integral of Fisher information, we find that it is sufficient to prove the theorem under the assumption that Φ*(N^−1/2Σ_i=1^N X_i) = Φ* ((N + 1)^−1/2Σ_i=1^N+1X_i).

Let ζ_j = J(Σ_i≠j X_i). Then equality in Eq. 4 entails

Let E be the orthogonal projection onto W* (Σ_i=1^N+1 X_i). Then E(ζ_j) = J(Σ_i=1^N+1X_i) and as in the proof of Theorem 2 we have the inequalities

So equality in Eq. 4 forces E(Σζ_j) = Σζ_j and ‖Σζ_j‖₂² = N Σ‖ζ_j‖₂². We now use the following lemma (13):

Lemma 8.

Let P₁, …, P_m be commuting projections on a Hilbert space ℋ. If ξ₁, …, ξ_m ∈ ℋ satisfy that for all 1 ≤ i ≤ m, P₁P₂ … P_mξ_i = 0, and the equality

holds, then for each j, ξ_j ∈ ⊕_i≠j ℋ_i, where we have set ℋ_j = (∩_k≠jP_k(ℋ))∩P_j^⊥(ℋ).

Applying Lemma 8 with m = N + 1, ξ_j = ζ_j, P₁ = E₁, …, P_N+1 = E_N+1, where E_j is the projection onto L²(W*(X₁, …, X̂_j, …, X_N+1)), and noticing that ℋ_j = L²(W*(X_j)), we obtain that

Combining this with the equality (N + 1) J (Σ_j=1^N+1 X_j) = Σ_j=1^N+1 ζ_j, we conclude from Eq. 7 that J(Σ_j=1^N+1 a_jX_j) ∈ ⊕_j=1^N+1 (L²(W* (X_i)) ⊝ ℂ1). Now choose η _j ∈ L²(W*(X_i)) ⊝ ℂ1 so that

Then

A standard application of freeness shows that for (i, j) ≠ (k, l), the terms X_iη _j − η_i X_j and X_kη_l − η_kX_l are perpendicular elements of L²(M) [when τ(X_j) = 0]. Thus, the above identity implies that for all i ≠ j,

It follows from the unique decomposition within the free product that there is only one way that Eq. 9 can be fulfilled: There exist c₁, …, c_n+1 ∈ ℝ such that η _j = c_j X_j. Eq. 9 shows that all c_j must be the same, so that for some C,

This implies that the variable Σ_j=1^N+1 X_j is semicircular (12). Since X_i are freely iid, one can conclude by additivity of R-transform that this can only happen if X₁ is semicircular.

Modifications of the Proof in the Classical Case.

The proof in the classical case proceeds in the same way until we arrive at Eq. 8. Since we are in a commutative setting, the commutator trick which we applied in the free case does not work here. We use the following lemma (see ref. 13 for a proof) instead.

Lemma 9.

Let N ∈ ℕ. Then for every m ∈ ℕ, the mth Hermite polynomial, H_m, satisfies

Because of finiteness of Fisher information, we may assume (see lemma 13.3 in ref. 13 for details) that the variables X_j are functions of iid Gaussian variables. Thus we can assume that the Hermite polynomials form a basis for the L²-spaces with which we are working. That is, there exist scalars (α_m)_m=1^∞ and (β_m)_m=1^∞ such that the score function f for X₁ + … + X_N+1 is given by

and that the vectors η_j in Eq. 8 are equal to

By Lemma 9, this implies that

The functions (H_k1 (x₁)H_k2 (x₂) … H_kN+1 (x_N+1))_{k1,…,kN+1≥0} are mutually perpendicular in L²(ℝ^N+1, ⊗_j=1^N+1σ₁), where σ₁ denotes the standard Gaussian law. Fix m ≥ 2, and take k₁, …, k_N+1 with Σ_jk_j = m and k_j ≥ 1 for at least two j. Then take inner product with H_k1 (x₁)H_k2 (x₂) … H_kN+1 (x_N+1) on both sides of Eq. 12 to see that α_m must be zero. Thus the score function to X₁ + … + X_N+1 must be proportional to X₁ + … + X_N+1. This shows that X₁ + … + X_N+1 is Gaussian. As in the free case, using additivity of the logarithm of the Fourier transform, this can only happen if X₁ is Gaussian.