Partitioned and Hadamard Product Matrix Inequalities

Abstract

This note is partly expositor). Inequalities relating inversion with, respectively, extraction of principal submatriees and the Hadamard product in the two possible orders are developed in a simple and unified way for positive definite matrices. These inequalities are known, hut we also characterize the cases of equality and strict inequality. A by-product is, for example, a pleasant proof of an inequality due to Fiedler. In addition, it is shown that the Hadamard product preserves inequalities in a generalization of Schur’s observation. In the process, many tools for dealing with the positive semi-definite partial ordering are exhibited.

The inequalities we discuss are with respect to the positive semi-definite partial ordering on hermitian n-by-n matrices. By

$$P\ge Q$$

we mean that P − Q is positive semi-definite, and by

$$\text{P}>\text{Q}$$

we mean that P − Q is positive definite. In particular, P ≥ 0 and P > 0 mean, respectively, that P is positive semi-definite and P is positive definite. See [4]¹ for an exposition of standard facts about positive definite matrices used herein. We first note that congruence preserves this ordering.

(1) Observation: If P and Q are n-by-n hermitian matrices, then

$$P\ge Q\text{ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}implies \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{T}^{*}PT\ge T^{*}QT$$

for all n-by-m complex matrices T, and

$$P>Q\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} implies\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} }{T}^{*}PT>T^{*}QT$$

for all n-by-m complex matrices T of rank m.

Proof: If P − Q is positive semi-definite then y*(P − Q)y ≥ 0 for all y ∈Cⁿ. Thus, x*(T*PT − T*QT)x = (Tx)*(P − Q)Tx ≥ 0 for all x ∈ C^m which, in turn, means that T*PT −T*QT is positive semi-definite and therefore, that T*PT ≥ T*QT. The second statement is verified analogously.

(2) Observation: For n-by-n positive definite hermitian matrices P and Q,

$$P\ge Q\text{\hspace{1em} if and only if \hspace{1em}}{Q}^{-1}\ge P^{-1}$$

and

$$P>Q\text{\hspace{1em} \hspace{0.17em}if and only if \hspace{1em}}{Q}^{-1}>P^{-1}.$$

Proof: Since P is positive definite, it may be written P = T*T where T is n-by-n and nonsingular. From observation 1 and P ≥ Q it follows that I ≥ T⁻¹*QT⁻¹. Since this means that I − T⁻¹*QT⁻¹ is positive semi-definite, it follows that all eigenvalues of T⁻¹*QT⁻¹ are less than or equal to 1 and, therefore, that all those of TQ⁻¹T* are greater than or equal to 1. This is to say that TQ⁻¹T* − I is positive semi-definite, or that TQ⁻¹T* ≥ I. Again using observation 1, this translates into Q⁻¹ ≥ T⁻¹T⁻¹* = P⁻¹ This proves the first statement because the two implications are equivalent. An analogous argument verifies the second statement.

A key observation is the partitioned form for the inverse of a general nonsingular matrix.

(3) Observation: Suppose that A is an n-by-n nonsingular matrix partitioned as

$$A=\left(\begin{array}{cc}{A}_1& A_2\\ A_3& A_4\end{array}\right)$$

where A₁ is n₁-by-n₁ and A₄ is n₂-by-n₂, n = n₁ + n₂. Then

$$A^{-1}=\left(\begin{array}{cc}{\left(A_1-A_2{A}_4^{-1}{A}_3\right)}^{-1}& A_1^{-1}{A}_2{\left(A_3{A}_1^{-1}{A}_2-A_4\right)}^{-1}\\ {\left(A_3{A}_1^{-1}{A}_2-A_4\right)}^{-1}{A}_3{A}_4^{-1}& {\left(A_4-A_3{A}_1^{-1}{A}_2\right)}^{-1}\end{array}\right)$$

partitioned conformally. This assumes all the relevant inverses exist.

Proof: This may be verified by direct matrix multiplication or by solving for the blocks of A⁻¹.

In case the matrix is positive definite hermitian, the relevant inverses do exist and the form of observation 3 translates into

(4) Observation: For a partitioned n-by-n hermitian positive definite matrix,

$${\left(\begin{array}{cc}A& B\\ B^{*}& C\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\left(A-B{C}^{-1}{B}^{*}\right)}^{-1}& A^{-1}B{\left(B^{*}{A}^{-1}B-C\right)}^{-1}\\ {\left(B^{*}{A}^{-1}B-C\right)}^{-1}{B}^{*}{A}^{-1}& {\left(C-B^{*}{A}^{-1}B\right)}^{-1}\end{array}\right)$$

where the partitioning is conformal.

(5) Corollary: For an n-by-n by hermitian positive definite matrix partitioned as in observation 4, the following inequalities hold

$$A>B{C}^{-1}{B}^{*}\text{\hspace{1em} and \hspace{1em}}C>B^{*}{A}^{-1}B.$$

Proof: This follows from observations 2 and 4 and the facts that the inverse of and any principal submatrix of a positive definite matrix must be positive definite.

Observation (3) has been made by many authors in this and several equivalent forms. See, for example, [1] where many related facts are also developed. Corollary (5) has been noted also in [3] by different means and probably has a more extensive history. We next depart from known observations and note a fact which allows us to link several types of inequalities and explore more deeply and simply some observations made in [9].

The main result is a comparison between the inverse of a principal submatrix of a positive definite matrix P and the corresponding principal submatrix of P⁻¹ (both of which are necessarily positive definite). We denote the set {1, 2, ⋯, n} by N and then I, J ⊆ N are index sets. For an arbitrary n-by-n matrix A, we denote the submatrix of A built from the rows indicated by I and the columns indicated by J by

$$A\text{(}I,J),$$

and we abbreviate A(I, I) to A(I). The complement of I (with respect to N) is written I′.

(6) Theorem: For an n-by-n positive definite matrix P we have

$$P^{-1}(I)\ge P{(I)}^{-1}.$$

Furthermore, rank (P⁻¹(I) − P(I)⁻¹) = rank P(I, I′), so that the above inequality is strict if and only if P(I, I′) has full row rank and equality holds precisely when P (I, I′) =0.

Remark: Statement (6) may he paraphrased, “The inverse of a principal suhmatrix is less than or equal to the corresponding principal suhmatrix of the inverse.”

Proof of (6): Because the statement in question is invariant under permutation similarity, it follows from observation (4) that

$$P^{-1}(I)={\left(P(I)-P\left(I,I^{\prime }\right)P{\left(I^{\prime }\right)}^{-1}P{\left(I,I^{\prime }\right)}^{*}\right)}^{-1}.$$

Therefore,

$${\left(P^{-1}(I)\right)}^{-1}=P(I)-P\left(I,I^{\prime }\right)P{\left(I^{\prime }\right)}^{-1}P{\left(I,I^{\prime }\right)}^{*},$$

and

$$P(I)-{\left(P^{-1}(I)\right)}^{-1}=P\left(I,I^{\prime }\right)P{\left(I^{\prime }\right)}^{-1}P{\left(I,I^{\prime }\right)}^{*}.$$

Since P(I′) is positive definite by virtue of being a principal suhmatrix of a positive definite matrix, the right-hand side of the equality just above is positive semi-definite (and, in fact, has rank equal to that of P(I, I′)). We then have that P(I) ≥ (P⁻¹(I))⁻¹ and, employing observation 2, P⁻¹(I) ≥ P(I)⁻¹. The cases of equality and strict inequality are, clearly, determined by P(I, I′) as asserted, and an analogous argument passing from P⁻¹ to P shows that rank (P⁻¹(I) − P(I)⁻¹) = rank P(I, I′).

Remark: We note that the inequality of theorem 6 cannot be strict whenever the cardinality of I is greater than $\frac{1}{2}n$.

The Hadamard product of two m-by-n matrices A = (a_ij) and B = (b_ij) is the m-by-n matrix

$$A\circ B=C=\left(c_{ij}\right)$$

where c_ij = a_ij b_ij for i = 1, ⋯, m; j = 1, ⋯, n. Further, the Kronecker product of an m_l-by-n_l matrix A = (a_ij) and an m₂-by-n₂ matrix B is the m₁m₂-by-n₁n₂ matrix

$$A\otimes B\equiv \left(\begin{array}{ccc}{a}_{11}B& \cdots & a_{1n_1}B\\ .& & .\\ .& & .\\ .& & .\\ a_{m_{1}1}B& \cdots & a_{m_1{n}_1}B\end{array}\right).$$

(7) Observation: The Hadamard product of two matrices (for which it is defined) is a submatrix of the Kronecker product of those two matrices, and if the two matrices are n-by-n, the submatrix is principal.

Proof: This is immediate, and in the n-by-n case we actually have

$$A\circ B=(A\otimes B)(I)$$

where I = {1, n + 2, 2n + 3, ⋯, n²}.

(8) Observation: If H ≥ (>)K are n-by-n hermitian matrices, then

$$H(I)\ge (>)K(I)$$

for any index set I ⊂ N.

Proof: This simply follows from the fact that H − K positive semi-definite (positive definite) implies (H − K)(I) = H(I) − K(I) is also [4]. Observations 7 and 8 are well known and provide a useful device for deducing inequalities for Hadamard products from those for Kronecker products (which, although stronger, are often easier to see) since H ⊗ K is positive definite when H and K are [8]. We note also the well-known result attributed to Schur that the Hadamard product of positive definite, matrices is positive definite [6] (which is a classic illustration of the comment just made). See also [8] for manipulative facts relating to these products.

The next main result deals with Hadamard products and inverses of positive definite matrices.

(9) Theorem. For n-by-n positive definite hermitian matrices H and K:

$$H^{-1}\circ K^{-1}\ge {(H\circ K)}^{-1}.$$

Moreover, rank (H⁻¹ ◦ K⁻¹ − (H ◦ K)⁻¹) = rank L where

$$L=[H,H,\cdots ,H]\circ \left(K\left[P,P^2,\cdots ,P^{n-1}\right]\right)$$

and

$$P=\left(\begin{array}{ccccccc}0& 0& .& .& .& 0& 1\\ 1& 0& .& .& .& & 0\\ .& 1& & & & & .\\ .& & & .& & & .\\ 0& & .& & .& 1& 0\end{array}\right).$$

Thus, the inequality is strict if and only if rank L = n, and equality holds if and only if H ◦ KP^t = 0, t = 1, ⋯, n − 1.

Proof: Note that (H ⊗ K)⁻¹ = H⁻¹ ⊗ K⁻¹ and the proof is a direct application of theorem 6 to H ⊗ K using observations 7 and 8 and the fact that (H ⊗ K)(I, I′) = L where I is as in the proof of observation 7.

(10) Corollary. Equality holds in the inequality of Theorem 9 if and only if H and K are both diagonal.

Proof: If both are diagonal, equality is clear, and, on the other hand, H ◦ KP^t = 0 for t = 1, ⋯, n − 1

only when H and K are diagonal (for positive definite matrices), so that the converse follows from the condition for equality in theorem 9.

The inequality of theorem 9 is known (see, for example [9] for variations); but the cases of equality and strict inequality seem not to have been treated, and the deduction from theorem 6 is especially simple. This, of course, shows the link between the Hadamard type inequality (9) and the partitioned inequality (6) and in some sense it explains why the otherwise rather mystifying inequality (9) is true. We note that the cases of equality and strict inequality have nothing to do with unitary invariants but are rather more combinatorial.

For a matrix A, denote the p-th Hadamard “power”

$$\underset{p\text{-times }}{\underbrace{A\circ A\circ \cdots \cdot \circ A}}$$

by A^(p). Several special cases of theorem (9) are worth noting.

(11) Corollary: For a positive definite n-by-n hermitian matrix H,

$$H^{-1}\circ H^{-1}\ge {(H\circ H)}^{-1}.$$

Inductively, (11) may be generalized using (9) and using the fact that A ≥ B ≥ 0 and C ≥ D ≥ 0 hermitian imply that A ◦ C ≥ B ◦ D. (The latter is a special case of a statement to be proved later.)

(12) Corollary: For a positive definite n-by-n hermitian matrix H,

$${\left(H^{-1}\right)}^{(p)}\ge {\left(H^{(p)}\right)}^{-1}$$

for all p = 1, 2, 3, ⋯.

Taking K = H⁻¹, and noting the commutativity of the Hadamard product, produces another interesting special case of (9).

(13) Corollary: For a positive definite n-by-n hermitian matrix H,

$$H\circ H^{-1}\ge {\left(H\circ H^{-1}\right)}^{-1}.$$

Statement (13) is of particular interest in that it yields an immediate proof of an inequality due to Fiedler [2]. Note that (13) states that a certain matrix is greater than or equal to its inverse; therefore, that matrix must be greater than or equal to the identify.

(14) Corollary: For a positive definite n-by-n hermitian matrix H,

$$\left(H\circ H^{-1}\right)\ge I\ge {\left(H\circ H^{-1}\right)}^{-1}.$$

(We note that this means that all eigenvalues of H ◦ H⁻¹ are at least one.) Actually, either half of inequality (14) is equivalent to the other and, in turn, equivalent to (13). Fiedler’s proof of H ◦ H⁻¹ ≥ I is quite computational, so that the approach herein provides an attractive alternative. This inequality may he interpreted as saying that the Hadamard product (of H and H⁻¹) dominates the usual product. It would he interesting to know if there are other instances of such a phenomenon. For another view of this inequality see [7].

The cases of equality in (11)–(14) are, of course, entirely covered by (10). Strict inequality can never attain in (13) or (14) because the row sums of H ◦ H⁻¹ are all equal to one, and thus

$$e^T\left(H\circ H^{-1}\right)e=e^{T}Ie=e^T{\left(H\circ H^{-1}\right)}^{-1}e=n.$$

Strict inequality can occur in (11) and (12) and, in fact, is likely. (We conjecture that strict inequality holds in (11) and (12) unless H has a row with (n − 1) entries equal to zero.)

Inequality (11) is of an intriguing type: two matrix operations (in this case inversion and Hadamard squaring) commute except for an inequality. There are other examples of this phenomenon (e.g., see [5]), and a more general understanding of it would be worthwhile. Inequality (6) is, of course, also of this type.

We turn finally to a different sort of Hadamard product inequality which generalizes Schur’s theorem (the Hadamard product of two positive definite matrices is positive definite) and points out that matrices under Hadamard multiplication are quite analogous to complex numbers in rectangular coordinates as far as simple multiplicative inequalities. For an arbitrary n-by-n complex matrix define

$$H(A)\equiv \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\left(A+A^{*}\right)$$

the “hermitian part” (or real part) of A. A natural extension to all matrices of the partial ordering for hermitian matrices discussed so far is the following. For n-by-n matrices A, B we write

$$A\ge B$$

if and only if H(A) ≥ H(B) in the positive semi-definite partial ordering on hermitian matrices. (Note that A ≥ B and B ≥ A do not imply A = B, only that H(A) = H(B).) We make the analogous extension for strict inequality. A straightforward calculation yields

(15) Observation: For two n-by-n complex matrices A and B, A ≥ B if and only if Re(x*Ax) ≥ Re(x*Bx) for all x ∈ Cⁿ.

Of the observations made earlier for the positive semi-definite partial ordering on hermitian matrices, those which carry over to the extended partial ordering (under obvious interpretation) are (1), (2), and (5). It is obvious that A ≥ B and C ≥ D imply A + C ≥ B + D.

(16) Example: That theorem 6 does not in general extend to all matrices satisfying H(A) > 0 is shown by

$$A=\left(\begin{array}{rr}\hfill 1& \hfill 1\\ \hfill -1& \hfill 1\end{array}\right).$$

Then,

$$A^{-1}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\left(\begin{array}{cc}1& -1\\ 1& 1\end{array}\right)$$

and 1/2 ≱ 1.

Our principal observation is:

(17) Theorem: Let P ≥ Q be n-by-n positive semi-definite hermitian matrices, and suppose that A ≥ B ≥ 0 are n-by-n complex matrices. Then

$$P\circ A\ge Q\circ B.$$

Proof: We assume, without loss of generality, that P is positive definite because then the positive semi-definite case follows by a continuity argument. Because of observations 7 and 8, the desired inequality holds if

$$P\otimes A\ge Q\otimes B$$

and, because of the extension of observation 1, this inequality holds if there is a nonsingular n²-by-n² matrix R such that

$$R*\left(P\otimes A\right)R\ge R*(Q\otimes B)R.$$

By virtue of the assumption that P is positive definite, there is a non-singular n-by-n matrix T such that

$$T*PT=D\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}T*QT=E$$

are simultaneously diagonal, and, again because of observation 1, D ≥ E ≥ 0 (i.e., d_ii ≥ e_ii ≥ 0 for all i = 1, ⋯, n). We now take R = T ⊗ I, and, then

$$R*(P\otimes A)R=\left(T*\otimes I\right)(P\otimes A)(T\otimes I)=T*PT\otimes A=D\otimes A$$

and, similarly

$$R*(Q\otimes B)R=E\otimes B.$$

Thus it suffices to show that

$$D\otimes A\ge E\otimes B.$$

However, if we partition an arbitrary $x\in C^{n^2}$ into $x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right)$ where each x₁ ∈ Cⁿ, i = 1, ⋯, n, then

$$x^{*}(D\otimes A)x=\sum _{i=1}^n{d}_{ii}{x}_i^{*}A{x}_i\text{\hspace{1em} and \hspace{1em}}{x}^{*}(E\otimes B)x=\sum _{i=1}^n{e}_{ii}{x}_i^{*}B{x}_i.$$

Since $Re\left(x_i^{*}A{x}_i\right)\ge Re\left(x_i^{*}B{x}_i\right)\ge 0$ and d_ii ≥ e_ii ≥ 0, i = 1, ⋯, n, we have

$$Re\left(x^{*}(D\otimes A)x\right)\ge Re\left(x^{*}(E\otimes B)x\right)$$

and D ⊗ A ≥ E ⊗ B by observation 15. Thus

$$P\circ A\ge Q\circ B,$$

and the proof is complete.

It is clear that theorem 17 may be extended to the case of strict inequality which we mention without proof.

(18) Observation: For n-by-n hermitian matrices P > Q ≥ 0 and n-by-n complex matrices A > B ≥ 0, P ◦ A > Q ◦ B.

Finally, (17) may be extended to the case in which none of the factors is hermitian under proper assumptions. This completes the analogy to multiplication of complex numbers. Define

$$S\left(A\right)\equiv \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\left(A-A*\right)$$

the “skew-hermitian part” of an n-by-n complex matrix A. Then A = H (A) + S(A) and iS(A) is hermitian. Now for four matrices, A, B, C, D a calculation yields that

$$H\left(A\circ C\right)=H\left(A\right)\circ H\left(C\right)+S(A)\circ S(C)$$

and

$$H\left(B\circ D\right)=H\left(B\right)\circ H\left(D\right)+S(B)\circ S(D).$$

Therefore, A ◦ C ≥ B ◦ D if and only if

$$H(A)\circ H\left(C\right)=S\left(A\right)\circ S\left(C\right)\ge H(B)\circ H(D)+S(B)\circ S(D).$$

Theorem 17 treats the case in which S(A) = S(B) = 0 and may be used to exhibit several more circumstances in which A ◦ C ≥ B ◦ D. We list some of these as examples and each may be proved quite formally with the aid of (17).

(19) Observation: For n-by-n complex matrices A, B, C, and D, each of the following sets of conditions is sufficient for

$$A\circ C\ge B\circ D:$$

(a) H(A) ≥ H(B) ≥ 0; H(C) ≥ H(D) ≥ 0; iS(B) ≥ iS(A) ≥ 0; and iS(D) ≥ iS(C) ≥ 0.

(b) H(A) ≥ H(B) ≥ 0; H(C) ≥ H(D) ≥ 0; 0 ≥ iS(B) ≥ iS(A); and iS(C) ≥ iS(D) ≥ 0. and

(c) H(A) ≥ −H(B) ≥ 0; and H(C) ≥ −H(D) ≥ 0; −iS(A) ≥ iS(B) ≥ 0; and iS(C) ≥ −iS(D) ≥ 0.