Roy’s largest root under rank-one perturbations: the complex valued case and applications

Abstract

The largest eigenvalue of a single or a double Wishart matrix, both known as Roy’s largest root, plays an important role in a variety of applications. Recently, via a small noise perturbation approach with fixed dimension and degrees of freedom, Johnstone and Nadler derived simple yet accurate approximations to its distribution in the real valued case, under a rank-one alternative. In this paper, we extend their results to the complex valued case for five common single matrix and double matrix settings. In addition, we study the finite sample distribution of the leading eigenvector. We present the utility of our results in several signal detection and communication applications, and illustrate their accuracy via simulations.

1. Introduction

Wishart matrices, both real and complex valued, play a central role in statistics, with numerous engineering applications, specifically signal processing and communications. Of particular interest are the roots of a single Wishart matrix H, and of a double Wishart matrix E⁻¹ H, with H and E independent [1]. The latter can be viewed as the multivariate analogue of the univariate F distribution and is also closely related to the multivariate beta distribution [32, Section 3.3]. Here we consider the largest eigenvalue ℓ₁ of either the matrix H or the matrix E⁻¹ H, a test statistic proposed by Roy [38, 39], known as Roy’s largest root [32, Section 10.6]. Specifically, we focus on the complex-valued case where H, E are independent complex-valued Wishart matrices. Throughout this paper, we consider m × m matrices, where E follows a complex valued central Wishart distribution with n_E degrees of freedom and identity covariance matrix ∑_E = I, denoted $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$. The distribution of the matrix H will either be central $H~\mathcal{C}{\mathcal{W}}_m(n_H,{\Sigma }_H)$, or non-central $H~\mathcal{C}{\mathcal{W}}_m(n_H,{\Sigma }_H,\Omega )$. For the definition of central and non-central complex valued Wishart matrices, see for example [15] and [19, Section 8].

Obtaining simple expressions, exact or approximate, for the distribution of this top eigenvalue, denoted by ℓ₁, in the single or double matrix case has been a subject of intense research for more than 50 years. Khatri [27] derived an exact expression for the distribution of ℓ₁ in the single central matrix case with an identity covariance matrix (∑_H = I). His result was generalized to several other settings, such as an arbitrary covariance matrix or a non-centrality matrix [24, 28, 36, 37, 41]. The resulting expressions are, in general, challenging to evaluate numerically. More recently, Zanella et al. [44] derived simpler exact, yet recursive expressions, both for the central case with arbitrary ∑_H and for the non-central case but with ∑_H = I. Alternative recursive formulas in the real-valued case and in the complex-valued case were derived by Chiani [4–6].

A different approach to derive approximate distributions for the largest eigenvalue when ∑_E = ∑_H = I, is based on random matrix theory. Considering the limit as n_H and m (and in the double matrix case also n_E) tend to infinity, with their ratios converging to constants, ℓ₁ in the single matrix case and ln(ℓ₁) in the double matrix case, asymptotically follow a Tracy-Widom distribution [20–22]. Furthermore, with suitable centering and scaling, the convergence to these limiting distributions is quite fast [10, 31].

In this paper, motivated by statistical signal detection and communication applications, we consider complex valued Wishart matrices H whose population covariance is a rank-one perturbation of a base covariance matrix. Specifically, in the central case we assume ∑_H = I + λvv^†, where λ is a measure of signal strength, the unit norm vector $\mathbf{v}\in {\mathbb{C}}^m$ is its direction, and v^† denotes the conjugate transpose of v. Similarly, in the non-central case, we assume that $H~\mathcal{C}{\mathcal{W}}_m(n_H,I,\Omega )$, with a rank-one non-centrality matrix Ω = λvv^†. Our goal is to study the distribution of ℓ₁ and its dependence on λ, which as discussed below is a central quantity of interest in various applications. A classical result in the single-matrix case is that with dimension m fixed, as n_H → ∞, the largest eigenvalue of H converges to a Gaussian distribution [1]. In the random matrix setting, as both n_H and m tend to infinity with their ratio tending to a constant, Baik et al. [3] and Paul [34] proved that if $\lambda >\sqrt{m∕n_H}$ then ℓ₁ still converges to a Gaussian distribution, but with a different variance. In the two-matrix case, the location of the phase transition and the limiting value of the largest eigenvalue of E⁻¹ H were recently studied by Nadakuditi and Silverstein [33]. Dharmawansa et al. [8] proved that above the phase transition, ℓ₁ converges to a Gaussian distribution and provided an explicit expression for its asymptotic variance.

Whereas the above results assume that dimension and degrees of freedom tend to infinity, in various common applications these quantities are relatively small. In such settings, the above mentioned asymptotic results may provide a poor approximation to the distribution of the largest eigenvalue ℓ₁, which can be quite far from Gaussian, see Fig. 1 (left) for an illustrative example. Accurate expressions for the distribution of ℓ₁, for small dimension and degrees of freedom, were recently derived for single and double real-valued Wishart matrices by Johnstone and Nadler [23], via a small noise perturbation approach. In this paper, we build upon their work and extend their results to the complex valued case and to the study of the distribution of the leading sample eigenvector, not considered in their work. As discussed below, both are important quantities in various applications.

Fig. 1.

Density of the largest eigenvalue in Case 1 (left) and Case 2 (right). The parameters for Case 1 are n_H = m = 5, λ = 5 and σ² = 0.01. For Case 2 they are n_H = m = 5, ω = 1 and σ² = 0.01. The red solid line corresponds to Propositions 1 and 2.

Propositions 1-5 in Section 2 provide approximate expressions for the distribution of ℓ₁ under the five single-matrix and double-matrix cases outlined in Table 1. In Section 3 we study the finite sample fluctuations of the leading eigenvector and its overlap with the population eigenvector. Next, in Section 4 we illustrate the utility of these approximations in signal detection and communication applications. Specifically, Section 4.1 considers the power of Roy’s largest root test under two common signal models, whereas Section 4.2 considers the outage probability in a specific multiple-input and multiple-output (MIMO) communication system [24]. For a rank-one Rician fading channel, we show analytically that to minimize the outage probability it is preferable to have an equal number of transmitting and receiving antennas. This important design property was previously observed via simulations [24].

Table 1.

Five common single-matrix and double-matrix cases. The middle column describes the distribution of the covariance matrices of the observed data. In the first two cases only one sample covariance matrix is computed. The right column describes several relevant applications.

Case	General Form of Distribution	Application
1	$H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma +\lambda {\mathbf{vv}}^{†})$ ∑ is known	Signal detection in noise, known noise covariance matrix.
2	$H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma ,\omega {\mathbf{vv}}^{†})$ Σ is known	Constant modulus signal detection in noise, known noise covariance matrix.
3	$H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma +\lambda {\mathbf{vv}}^{†})$ $E~\mathcal{C}{\mathcal{W}}_m(n_E,\Sigma )$	Signal detection in noise, estimated noise covariance matrix.
4	$H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma ,\omega {\mathbf{vv}}^{†})$ $E~\mathcal{C}{\mathcal{W}}_m(n_E,\Sigma )$	Constant modulus signal detection in noise, estimated noise covariance matrix.
5	$H~\mathcal{C}{\mathcal{W}}_p(q,\Phi ,\Omega )$ $E~\mathcal{C}{\mathcal{W}}_p(n-q,\Phi )$ Ω is a rank-one matrix	canonical correlation analysis between two groups of sizes p ≤ q

2. On the Distribution of Roy’s Largest Root

Table 1 outlines five common single matrix and double matrix complex Wishart cases, along with some representative applications. Propositions 1-5 below, are the complex analogues of those in [23], and provide simple approximations to the distribution of Roy’s largest root in these cases. As outlined in the appendix, their proof follows those of [23], with some notable differences. In particular, we present complex valued analogues of some well known results for real valued Wishart matrices. In what follows we denote by $\mathbb{E}$ the expectation operator. We also denote by ${\chi }_k^2$ the chi-squared distribution with k degrees of freedom and by ${\chi }_k^2(\eta )$ the non-central chi-squared distribution with non-centrality parameter η. Throughout the manuscript we follow the standard definition of complex valued multivariate Gaussian random variables, see [15]. Specifically, if $X~\mathcal{C}\mathcal{N}(0,{\sigma }^2)$ then it can be written as $(A+\iota B)∕\sqrt{2}$ where A, B ∈ ℝ are independent $\mathcal{N}(0,{\sigma }^2)$ random variables and $\iota =\sqrt{-1}$.

We start with the simplest Case 1 in Table 1, involving a single central Wishart matrix, $H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma +\lambda {\mathbf{vv}}^{†})$. In various engineering applications the matrix ∑ denotes the covariance of the noise measured at m sensors and is often assumed to be known, whereas λ is a measure of the signal strength and the unit norm vector v denotes its direction. Without loss of generality, we thus assume ∑ = σ²I, where σ² then denotes the noise variance. In contrast to previous asymptotic approaches, whereby the number of samples n_H → ∞ and possibly also the dimension m → ∞, in the following we keep n_H and m fixed, and study the distribution of the largest eigenvalue in the limit of small noise, namely as σ → 0. To emphasize that we study the dependence of the largest eigenvalue of H on the parameter σ, we shall denote it by ℓ₁ (σ).

Proposition 1. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,\lambda {\mathbf{vv}}^{†}+{\sigma }^{2}I)$, with ||v|| = 1, λ > 0 and let ℓ₁(σ) be its largest eigenvalue. Then, with (m, n_H, λ) fixed, as σ → 0

$${\ell }_1(\sigma )=\frac{\lambda +{\sigma }^2}{2}A+\frac{{\sigma }^2}{2}B+\frac{{\sigma }^4}{2(\lambda +{\sigma }^2)}\frac{BC}{A}+O_P({\sigma }^4)$$ (1)

where A, B, C are independent random variables, distributed as $A~{\chi }_{2n_H}^2$, $B~{\chi }_{2m-2}^2$, and $C~{\chi }_{2n_H-2}^2$.

Remark 1. Given that ℓ₁ is the largest eigenvalue of a Wishart matrix, it has finite mean and variance. Approximate formulas for these quantities follow directly from (1). Since $\mathbb{E}\{{\chi }_k^2\}=k\mathrm{Var}\{{\chi }_k^2\}=2k$, and $\mathbb{E}\{1∕{\chi }_k^2\}=1∕(k-2)$ for k > 2 then for n_H > 1

$$\mathbb{E}\{{\ell }_1(\sigma )\}=\lambda n_H+(n_H+m-1){\sigma }^2+\frac{{\sigma }^4}{\lambda +{\sigma }^2}(m-1)+o({\sigma }^4),$$

and similarly,

$$\mathrm{Var}\{{\ell }_1(\sigma )\}={\lambda }^2{n}_H+2\lambda n_H{\sigma }^2+(n_H+m-1){\sigma }^4+o({\sigma }^4).$$

Remark 2. The exact distribution of the largest eigenvalue ℓ₁ in the setting of Proposition 1, with number of samples larger than the dimension, has been recently derived by Chiani [6, Theorem 4, part 3]. The result is given in terms of the determinant of an m × m matrix whose entries depend on the generalized incomplete gamma function, with parameters that depend on λ and on σ. In contrast, while (1) is approximate, the dependence on the values of λ and σ is more explicit.

The next proposition considers a non-central single Wishart, Case 2 in Table 1.

Proposition 2. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,{\sigma }^{2}I,(\omega ∕{\sigma }^2){\mathbf{vv}}^{†})$ with ||v|| = 1, ω > 0 and let ℓ₁(σ) be its largest eigenvalue. Then, with (m, n_H, ω) fixed, as σ → 0

$${\ell }_1(\sigma )=\frac{{\sigma }^2}{2}\left(A+B+\frac{BC}{A}\right)+O_P({\sigma }^4)$$ (2)

where A, B, C are all independent and distributed as $A~{\chi }_{2n_H}^2(2\omega ∕{\sigma }^2)$, $B~{\chi }_{2m-2}^2$ and $C~{\chi }_{2n_H-2}^2$.

Remark 3. By definition, $\mathbb{E}\{{\chi }_k^2(\eta )\}=k+\eta$. Furthermore, it is easy to show that as η → ∞, $\mathbb{E}\{{({\chi }_k^2(\eta ))}^{-1}\}={(k-2+\eta )}^{-1}\{1+O({\eta }^{-1})\}$ and $\mathrm{Var}\{{({\chi }_k^2(\eta ))}^{-1}\}=2∕\{{(k+\eta -2)}^2(k+\eta -4)\}\cdot \{1+O({\eta }^{-1})\}$. Note that as σ → 0 the non-centrality parameter 2ω/σ² which appears in the random variable A in (2) tends to infinity. Hence, for small σ, we can approximate the mean and variance of ℓ₁(σ) in (2) by

$$\mathbb{E}\{{\ell }_1(\sigma )\}\approx \omega +(n_H+m-1){\sigma }^2+\frac{(n_H-1)(m-1)}{{\sigma }^2(n_H-1)+\omega }{\sigma }^4$$

and

$$\mathrm{Var}\{{\ell }_1(\sigma )\}\approx 8\omega +4{\sigma }^2\left\{n_H+m-1+\frac{(n_H-1)(m-1)}{2{(n_H+\frac{{\sigma }^2}{\omega }-1)}^2(n_H+\frac{{\sigma }^2}{\omega }-2)}\right\}.$$

The next two propositions provide approximations to the distribution of Roy’s largest root in the central and non-central double matrix settings, which correspond to Cases 3 and 4 in Table 1. For Case 3, for example, in principle we need to study ℓ₁(E⁻¹ H) where $E~\mathcal{C}{\mathcal{W}}_m(n_E,\Sigma )$ and $H~\mathcal{C}{\mathcal{W}}_m(n_H,\Sigma +\tilde{\lambda }{\mathbf{ww}}^{†})$. However, a simplification can be made based on the following observations: (i) The matrix E⁻¹ H has the same eigenvalues as ∑^1/2E⁻¹ H∑^−1/2, which is equal to (Σ^−1/2EΣ^−1/2)⁻¹(Σ^−1/2HΣ^−1/2), (ii) the matrix ${\Sigma }^{-1∕2}E{\Sigma }^{-1∕2}~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ and (iii) the matrix ${\Sigma }^{-1∕2}H{\Sigma }^{-1∕2}~\mathcal{C}{\mathcal{W}}_m(n_H,I+\lambda {\mathbf{vv}}^{†})$, where v = Σ^−1/2w/||Σ^−1/2w|| has unit norm, and $\lambda ={\Vert {\Sigma }^{-1∕2}\mathbf{w}\Vert }^2\tilde{\lambda }$. Hence, in the following propositions we assume without loss of generality that the covariance matrix of E is Σ = I.

Proposition 3. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,I+\lambda {\mathbf{vv}}^{†})$ and $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ be independent, with n_E > m + 1 and ||v|| = 1. Let ℓ₁ be the largest eigenvalue of E⁻¹H. Then, with (m, n_H, n_E) fixed, as λ becomes large

$${\ell }_1(\lambda )\approx (1+\lambda )a_1{F}_{b_1,c_1}+a_2{F}_{b_2,c_2}+a_3$$ (3)

where the two F distributed random variates are independent and

$$\begin{gathered} a_1=\frac{n_H}{n_E-m+1},a_2=\frac{m-1}{n_E-m+2},a_3=\frac{m-1}{(n_E-m)(n_E-m-1)}, \\ {b}_1=2n_H,b_2=2m-2,c_1=2n_E-2m+2,c_2=2n_E-2m+4. \end{gathered}$$ (4)

Proposition 4. Suppose that $H~\mathcal{C}{\mathcal{W}}_m(n_H,I,\omega {\mathbf{vv}}^{†})$ and $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ are independent, with n_E > m + 1, ω > 0, and ||v|| = 1. Let ℓ₁ be the largest eigenvalue of E⁻¹H. Then, with (m, n_H, n_E) fixed, as ω becomes large

$${\ell }_1(\omega )\approx a_1{F}_{b_1,c_1}(2\omega )+a_2{F}_{b_2,c_2}+a_3$$ (5)

where the two F distributed random variates are independent and the parameters a_i, b_i, c_i are given in (4).

Remark 4. In the limit as n_E → ∞, the two F-distributed random variables in (3) and (5) converge to χ² distributed random variables, thus recovering the leading order terms in (1) and (2), respectively.

Let us illustrate the accuracy of our approximations via several simulations. Fig. 1 compares the empirical density of the largest eigenvalue, computed from 10⁵ independent Monte Carlo realizations, in Cases 1 and 2 defined in Table 1, to the two corresponding propositions. For reference, we also plot the standard Gaussian density. The accuracy of our proposition for computing tail probabilities of the form Pr(ℓ₁ > t) is illustrated in Fig. 2 for Case 1. Similar results (not shown) hold for other cases. Results for Cases 3 and 4 of Table 1 are shown in Fig. 3. As can be seen, in all cases, due to the small sample size and dimension, the distribution of the largest root deviates significantly from the asymptotic Gaussian one, with our propositions being significantly more accurate.

Fig. 2.

Tail probabilities for largest eigenvalue in Case 1, same parameters as in Fig. 1.

Fig. 3.

Density of the largest eigenvalue in Case 3 (left) and Case 4 (right). In both plots n_E = n_H = 10, m = 5. In Case 3, λ = 50 and in Case 4 ω = 150. The blue solid line corresponds to Propositions 3 and 4.

2.1. On the leading canonical correlation coefficient

We now consider the fifth Case of Table 1 and study the largest sample canonical correlation coefficient between a first group of p variables and a second group of q variables, in the presence of a single large canonical correlation coefficient in the population. Canonical correlation analysis is widely used in a variety of applications, for example in medical image processing [7, 26, 30], signal processing [2, 35, 40], and array processing [11].

Since the canonical correlation is invariant under unitary transformations within each of the two groups of variables, in the presence of a single large correlation coefficient, without loss of generality we can choose the following form for the matrix Σ,

$$\Sigma =\left(\begin{array}{cc}{I}_p& \tilde{P}\\ {\tilde{P}}^{⊺}& I_q\end{array}\right).$$

Here $\tilde{P}=\left(\begin{array}{cc}\hfill P\hfill & \hfill 0_{p\times (q-p)}\hfill \end{array}\right)$ with P = diag (ρ, 0,…, 0) ∈ ℝ^p×p and ρ is the value of the correlation coefficient.

To study the sample canonical correlation, consider n + 1 complex-valued m-dimensional multivariate Gaussian observations ${\mathbf{x}}_i~\mathcal{C}\mathcal{N}(0,\Sigma )$, i = {1, …, n + 1} on m = p + q variables, where without loss of generality p ≤ q. The corresponding sample covariance matrix S decomposes as

$$nS=\left(\begin{array}{cc}{Y}^{†}Y& Y^{†}X\\ X^{†}Y& X^{†}X\end{array}\right),$$

where $Y\in {\mathbb{C}}^{n\times p}$ and $X\in {\mathbb{C}}^{n\times q}$ represent the first p variables and the remaining q variables, respectively.

Our interest is in the largest sample canonical correlation coefficient, denoted by r₁. Similar to the real valued case [32, Chapter 10], its square $r_1^2$ is the largest root of the following characteristic equation

$$\det \left(r^2{Y}^{†}Y-Y^{†}QY\right)=0,$$ (6)

where $Q=X{\left(X^{†}X\right)}^{-1}{X}^{†}$. Introducing the notation H = Y^†QY and E = Y^†(I_p − Q)Y, (6) can be rewritten as

$$\det (r^2(H+E)-H)=0.$$

Hence, we may equivalently study the largest root of E⁻¹H, since it is related to $r_1^2$ by ${\ell }_1=r_1^2∕(1-r_1^2)$.

Similar to [23], it can be shown that with Φ = I_p − P², conditional on X, the two matrices H and E are independent and distributed as

$$H|X\sim \mathcal{C}{\mathcal{W}}_p(q,\Phi ,\Omega )\text{and}E|X\sim \mathcal{C}{\mathcal{W}}_p(n-q,\Phi )$$ (7)

with the non-centrality matrix given by

$$\Omega ={\Phi }^{-1}\tilde{P}{X}^{†}X{\tilde{P}}^{⊺}=\frac{{\rho }^2}{1-{\rho }^2}{\left(X^{†}X\right)}_{11}{\mathbf{e}}_1{\mathbf{e}}_1^{†}=\omega {\mathbf{e}}_1{\mathbf{e}}_1^{†}\text{where}\omega =\frac{{\rho }^2}{1-{\rho }^2}{\left(X^{†}X\right)}_{11}.$$ (8)

Since $X^{†}X~\mathcal{C}{\mathcal{W}}_q\left(n,I_q\right)$, then all diagonal entries of X^†X follow a chi-square distribution. In particular, ${\left(X^{†}X\right)}_{11}~{\chi }_{2n}^2∕2$. The next proposition provides an approximation to the distribution of the largest sample canonical correlation in the presence of a single population canonical correlation. To this end, we introduce the following notation. We denote by $F_{a,b}^{\chi }(c,n)$ a random variable, which is defined as a function of three other random variables as follows: First, generate a random variable $Z~c{\chi }_n^2$. Next, generate two independent random variables, one distributed as ${\chi }_a^2(Z)$ and the other as ${\chi }_b^2$. Finally, compute their ratio

$$\frac{{\chi }_a^2(Z)/a}{{\chi }_b^2/b}\sim F_{a,b}^{\chi }(c,n).$$ (9)

Proposition 5. Let ${\ell }_1=r_1^2∕(1-r_1^2)$, where r₁ is the largest sample canonical correlation between two groups of size p ≤ q computed from n + 1 i.i.d. observations with v = n − p − q > 1. Then in the presence of a single large population correlation coefficient ρ between the two groups, asymptotically as ρ → 1,

$${\ell }_1\approx a_1{F}_{b_1,c_1}^{\chi }\left(\frac{{\rho }^2}{1-{\rho }^2},2n\right)+a_2{F}_{b_2,c_2}+a_3$$

where

$$\begin{gathered} a_1=q/(v+1),a_2=(p-1)/(v+2),a_3=(p-1)/v(v-1), \\ {b}_1=2q,b_2=2p-2,c_1=2(v+1),c_2=2(v+2). \end{gathered}$$

Remark 5. It can be shown that the probability density of $F_{a,b}^{\chi }(c,n)$ is

$$f_X(x)=\frac{1}{\mathcal{B}\left(\frac{c_1}{2},\frac{b_1}{2}\right){(1+c)}^{n/2}}{\left(\frac{b}{a}\right)}^{\frac{b}{2}}\frac{x^{\frac{a}{2}-1}}{{\left(x+\frac{b}{a}\right)}^{\frac{1}{2}(a+b)}}·{}_{2}F_1\left(\frac{n}{2},\frac{1}{2}(a+b);\frac{a}{2};\frac{xc}{(c+1)\left(x+\frac{b}{a}\right)}\right)$$

where ${}_{2}F_1(a,b;c;z)$ is the Gauss hypergeometric function and $\mathcal{B}(p,q)$ is the beta function. This formula is useful for numerical evaluation for small parameter values.

Fig. 4 illustrates the accuracy of Proposition 5. A good match between the theoretical approximation formula and simulation results is clearly visible, particularly at the right tail of the distribution.

Fig. 4.

Density function of ℓ₁(E⁻¹H) in canonical correlation analysis.

3. Distribution of the Leading Sample Eigenvector

Another key quantity of both theoretical and practical importance is the squared dot product between the leading sample eigenvector, denoted $\widehat{\mathbf{v}}$, and its corresponding population eigenvector v. Assuming $\Vert \mathbf{v}\Vert =\Vert \widehat{\mathbf{v}}\Vert =1$,

$$R={|{\widehat{\mathbf{v}}}^{†}\mathbf{v}|}^2.$$ (10)

A practical application where it is important to understand the behavior of R under a rank one spike, involves the design of dominant mode rejection (DMR) adaptive beamformers in array processing [42]. The main purpose of this beamformer is to eliminate interferences from undesired directions other than the steering direction. As shown in [43], an important parameter which determines the performance of the DMR scheme is the correlation between the random sample eigenvectors and the unknown population eigenvectors. Specifically, in the presence of a single dominant interferer, the population covariance matrix takes the form of a rank one spiked model [43, Eq. 17], and the effectiveness of the DMR depends on the quantity R. Another application where the quantity R plays a key role is passive radar detection with digital illuminators having several periodic identical pulses [14]. In a sequence of papers [12–14], the authors developed a new framework for passive radar detection based on the leading eigenvector of the sample covariance matrix. This detection scheme outperforms traditional detectors [14]. Motivated by these and other applications, we now develop stochastic approximations to R. For Case 1 of Table 1, we have:

Proposition 6. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,\lambda {\mathbf{vv}}^{†}+{\sigma }^{2}I)$, with ||v|| = 1 and λ > 0. Let $\widehat{\mathbf{v}}$ be the eigenvector corresponding to the largest eigenvalue of H. Then, with (m, n_H, λ) fixed, for small σ

$$R\approx \frac{1}{1+\frac{{\sigma }^2}{\lambda +{\sigma }^2}\frac{B}{A}+\frac{2{\sigma }^4}{{(\lambda +{\sigma }^2)}^2}\frac{BC}{A^2}},$$

where $A~{\chi }_{2n_H}^2$, $B~{\chi }_{2m-2}^2$ and $C~{\chi }_{2n_H-2}^2$ are all independent.

The distribution of R in Case 2 of Table 1 is given by the following proposition.

Proposition 7. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,{\sigma }^{2}I,(\omega ∕{\sigma }^2){\mathbf{vv}}^{†})$, with ||v|| = 1 and ω > 0. Let $\widehat{\mathbf{v}}$ be the eigenvector corresponding to the largest eigenvalue of H. Then, with (m, n_H, ω) fixed, for small σ

$$R\approx \frac{1}{1+\frac{B}{A_{\sigma }}+2\frac{BC}{A_{\sigma }^2}},$$

where $A_{\sigma }~{\chi }_{2n_H}^2(2\omega ∕{\sigma }^2)$, $B~{\chi }_{2m-2}^2$ and $C~{\chi }_{2n_H-2}^2$ are all independent.

Propositions 6 and 7 can be useful to analyze theoretically various DMR and radar detections schemes, and shed light on their dependence on the relevant system parameters.

For the double-matrix Case 3 in Table 1, we have

Proposition 8. Let $H~\mathcal{C}{\mathcal{W}}_m(n_H,\lambda {\mathbf{vv}}^{†}+I)$ and $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ be independent, with n_E > m + 1 and ||v|| = 1. Let $\widehat{\mathbf{v}}$ be the eigenvector corresponding to the largest eigenvalue of E⁻¹H. Then, with (m, n_H, n_E) fixed, for large λ

$$R\approx \frac{1}{1+\frac{B}{D}},$$

where $B~{\chi }_{2m-2}^2$ and $D~{\chi }_{2n_E+4-2m}^2$ are independent.

In the context of array processing, the double matrix Case 3 of Table 1 corresponds to a setting where the noise characteristics of the m sensors are not perfectly known, but rather their covariance matrix is estimated from n_E samples that do not contain any signal. Comparing Proposition 8 with Proposition 6 sheds light on the effect of estimating the covariance matrix of the noise. Whereas in Case 1, as signal strength λ → ∞ the quantity R converges to one, in Case 3, the random variable R does not converge to one, but rather to a Beta distribution.

Figs. 5 and 6 illustrate the accuracy of our approximate distributions of the squared inner product between the leading sample and population eigenvectors.

Fig. 5.

Empirical versus theoretical density of R in Case 1 (left) and Case 2 (right).

Fig. 6.

Comparison of empirical density of R in Case 3 of Table 1 with Proposition 8, for n_H = 10, n_E = 16, m = 5 and λ = 100.

4. Applications

We now demonstrate the utility of our approximations to Roy’s largest root distribution under a rank-one perturbation in three different engineering applications. The first two are concerned with common problems in signal detection, whereas the third with the outage probability of a rank-one Rician fading MIMO channel.

4.1. Signal Detection in Noise

Detecting the presence of a signal in a noisy environment is a fundamental problem in detection theory. Specific examples include spectrum sensing in cognitive radio [17] and target detection in sonar and radar [42]. Assuming additive Gaussian noise, the observed vector $\mathbf{y}(t)\in {\mathbb{C}}^m$ at time t is of the form

$$\mathbf{y}(t)=\sqrt{\lambda }s(t)\mathbf{u}+\mathbf{n}(t),$$ (11)

where $s(t)\in \mathbb{C}$ is the time dependent signal, $\mathbf{u}\in {\mathbb{C}}^m$ is normalized such that ||u|| = 1 is its direction, λ ≥ 0 is a measure of the signal strength and the vector $\mathbf{n}\in {\mathbb{C}}^m$ is a zero mean complex valued random noise, assumed to be independent of the signal and distributed as $\mathbf{n}~\mathcal{C}\mathcal{N}(0,\Sigma )$. The positive definite Hermitian matrix Σ is thus the population covariance of the additive random noise. In some cases it is assumed to be explicitly known, whereas in others it needs to be estimated. The signal s(t) is often modeled as a random quantity with $\mathbb{E}\{{\mid s(t)\mid }^2\}=1$. For example, in multiple antenna spectrum sensing for cognitive radio a common model is that $s(t)~\mathcal{C}\mathcal{N}(0,1)$, namely s(t) = s₁(t) + ιs₂(t) where s₁(t) and s₂(t) are real valued and independent random variables distributed $\mathcal{N}(0,1)$ [45,46]. Similarly, in detection of constant modulus signals (e.g., FM signals [18]), s(t) = exp(ιϕ(t)), where ϕ(t) is random.

When the covariance matrix Σ of the noise vector n is assumed known, the observed data used to detect if a signal is present are often n_H i.i.d. observations y₁, …, y_nH, from (11). A popular approach is to compute the sample covariance matrix $H={\Sigma }_{j=1}^{n_H}{\mathbf{y}}_j{\mathbf{y}}_j^{†}$, and declare that a signal is present if some function of its eigenvalues is larger than a suitable threshold. Several such detection tests have been proposed [18, 45, 46], including Roy’s largest root [29]. As discussed below, depending on the model of the signal, this leads precisely to Cases 1 and 2 in Table 1.

In other situations, Σ is unknown, but it is possible to observe both the n_H samples y_i of (11) as well as an additional set of n_E independent realizations n₁, …, n_nE of the noise vector n. The latter are measured, for example, in time slots at which it is a-priori known that no signals are emitted. Here, a typical approach is to form both the matrix H as above and the matrix $E={\Sigma }_{j=1}^{n_E}{\mathbf{n}}_j{\mathbf{n}}_j^{†}$ and detect the presence of a signal via some function of the eigenvalues of E⁻¹H. Signal detection based on the largest eigenvalue of E⁻¹H leads to Cases 3 and 4 in Table 1.

As discussed in Section 2, one may assume without loss of generality that Σ = σ²I. Thus, when $s~\mathcal{C}\mathcal{N}(0,1)$,

$$H\sim \mathcal{C}{\mathcal{W}}_m(n_H,\lambda \mathbf{u}{\mathbf{u}}^{†}+{\sigma }^{2}I).$$

In contrast, if s = exp(ιϕ), conditional on ϕ₁,…, ϕ_nH,

$$H\sim \mathcal{C}{\mathcal{W}}_m\left(n_H,{\sigma }^{2}I,\frac{\lambda n_H}{{\sigma }^2}\mathbf{u}{\mathbf{u}}^{†}\right).$$

Propositions 1-4 can thus be used to approximate the detection power of Roy’s largest root test as a function of signal strength λ in both the single matrix cases and the double matrix cases,

$$P_D=\mathrm{Pr}\{{\ell }_1>\mu |\text{signal present with strength}\lambda \},$$ (12)

where μ is a given threshold parameter. The accuracy of (12) is illustrated in Fig. 7.

Fig. 7.

Detection power profile for several signal to noise ratios as a function of threshold μ for a known covariance matrix or an unknown covariable. In both cases λ = 1, n_H = 5, m = 5. In the right panel n_E = 10. From top to bottom, σ = 1/10,2/10 and 3/10.

4.2. Rank-One Rician-Fading MIMO Channel

As a last application, consider the outage probability of a MIMO communication channel with n_T transmitters and n_R receivers. Here, the transmitted signals $\mathbf{x}\in {\mathbb{C}}^{n_T}$ and received signals $\mathbf{y}\in {\mathbb{C}}^{n_R}$ are related as

$$\mathbf{y}=H\mathbf{x}+\mathbf{n}$$

where H is the n_R × n_T channel matrix and n is additive random complex valued noise, assumed to be distributed as $\mathbf{n}~\mathcal{C}\mathcal{N}\left(n,{\sigma }_n^{2}I\right)$, where ${\sigma }_n^2$ is its (real-valued) variance. Due to fluctuations in the environment, the channel matrix H is modeled as a random quantity. In particular, under a common Rician fading model [16], H has the form

$$H=\sqrt{\frac{K}{K+1}}{H}_1+\sqrt{\frac{1}{K+1}}{H}_2$$ (13)

where H₁ represents the specular (Rician) component from a direct line-of-sight between transmitter and receiver antennas and H₂ represents the scattered Rayleigh-fading component. With fixed sender and receiver locations, the matrix H₁ is constant whereas H₂ is random with entries modeled as i.i.d. complex Gaussians, $\mathcal{C}\mathcal{N}(0,{\sigma }_H^2)$. Under the normalization $\mathrm{tr}(H_1{H}_1^{†})=n_R{n}_T$, the factor K represents the ratio of deterministic-to-scattered power of the environment.

Under the maximal ratio transmission strategy, where the transmitter sends information along the leading eigenvector of HH^†, the channel signal to noise ratio is given by

$$\mu =\frac{{\Omega }_D}{{\sigma }_n^2}{\ell }_1(H{H}^{†})$$ (14)

where ${\Omega }_D=\mathbb{E}[{\Vert \mathbf{x}\Vert }^2]$ is the power of the transmitted signal vectors [24]. An important quantity is the channel’s outage probability, defined as the probability of failing to achieve a specified minimal SNR μ_min required for satisfactory reception. Based on (14), the outage probability P_out can be written as

$$P_{\mathrm{out}}=\mathrm{Pr}\left(\frac{{\Omega }_D}{{\sigma }_n^2}{\ell }_1\le {\mu }_{\min }\right).$$ (15)

One particularly interesting case is when the Rician component H₁ is assumed to be of rank one, H₁ = uv^†, where $\mathbf{u}\in {\mathbb{C}}^{n_R}$, $\mathbf{v}\in {\mathbb{C}}^{n_T}$. An important design question is which configuration of antennas minimizes (15), under the constraint that the total number of transmitting and receiving antennas is fixed. Via simulations, [24] showed it is best to have an equal number of transmitting and receiving antennas. Here we analytically prove this result asymptotically in the limit of small scattering variance (i.e., σ_H « 1).

Proposition 9. Consider a rank-one Rician fading channel with a fixed number of antennas, n_T + n_R = N. Then, for σ_H « 1, the outage probability is minimized at n_T = n_R = N/2 for N even (or say n_T = ⌊N/2⌋, n_R = ⌈N/2⌉| for N odd).

Proof. Under the model in (13) and the assumption that H₁ = uv^† is rank one, the j-th column of H, of dimension n_R, is distributed as $\mathcal{C}\mathcal{N}\left(\sqrt{K∕(K+1)}\mathbf{u}{v}_j,{\sigma }_H^2∕(K+1)I_{n_R}\right)$. Therefore,

$$H{H}^{†}\sim \mathcal{C}{\mathcal{W}}_{n_R}(n_T,{\alpha }^2{I}_{n_R},{\beta }^2/{\alpha }^2\mathbf{w}{\mathbf{w}}^{†})$$

is non-central Wishart, with

$$\mathbf{w}=\mathbf{v}/\Vert \mathbf{v}\Vert ,{\alpha }^2=\frac{1}{K+1}{\sigma }_H^2\text{and}{\beta }^2=\frac{K}{K+1}{\Vert \mathbf{u}\Vert }^2{\Vert \mathbf{v}\Vert }^2=\frac{K}{K+1}{n}_R{n}_T.$$

Thus, Proposition 2 implies that for fixed (n_T, n_R, K),

$$\mu =\frac{{\Omega }_D}{{\sigma }_n^2}{\ell }_1=c_1\left(A+B+\frac{BC}{A}\right)+O_P({\sigma }_H^4)$$ (16)

where A, B, C are independent random variables distributed as

$$A\sim {\chi }_{2n_T}^2(c_2),B\sim {\chi }_{2n_R-2}^2,C\sim {\chi }_{2n_T-2}^2$$

and

$$c_1=\frac{{\Omega }_D{\sigma }_H^2}{2(K+1){\sigma }_n^2},c_2=\frac{2{\beta }^2}{{\alpha }^2}=\frac{2K}{{\sigma }_H^2}{n}_R{n}_T.$$ (17)

Since $\mathbb{E}(A)=2n_T+c_2⪢1$, and c₂ → as σ_H → 0, we may neglect the third term in (16). Furthermore, since A and B are independent,

$$\mu \approx c_1(A+B)=c_1({\chi }_{2n_T}^2(c_2)+{\chi }_{2n_R-2}^2)=c_1{\chi }_{2N-2}^2(c_2).$$

Clearly P_out of (15) is minimal when the largest eigenvalue ℓ₁ is stochastically as large as possible, or in turn, when its non-centrality parameter c₂ is maximal. Since by (17), c₂ ∝ n_Tn_R, the proposition follows. □

Fig. 8.

Outage probability as a function of n_T, with n_T + n_R fixed. Circles represent a Monte-Carlo simulation whereas the solid line is our approximation (which can be computed for any non-integer n_T ∈ ℝ⁺). These graphs support Proposition 9 and demonstrate the accuracy of our approximations. In both graphs, K = 2, σ_H = 0.3, σ_n = 1 and Ω_D = 5.

Appendix A. Proofs of main propositions

We prove our main results using the analytical framework developed in [23]. For a complex-valued number $z\in \mathbb{C}$, its real and imaginary parts are denoted $\mathfrak{R}(z)$ and $\mathfrak{J}(z)$, respectively, whereas $\stackrel{‒}{z}$ is its complex conjugate. We begin with the following auxiliary lemma, which describes the analytic structure of the leading eigenvalue and eigenvector of a covariance matrix constructed from vectors all in the same direction, which without loss of generality we choose as the standard vector e₁ = (1, 0, …, 0)^⊺, but corrupted by small perturbations. Its proof is in Appendix B.

Lemma 1. Let ${\{{\mathbf{x}}_j\}}_{j=1}^n$ be n vectors in ${\mathbb{C}}^m$ of the form

$${\mathbf{X}}_j=u_j{\mathbf{e}}_1+ϵ{\xi }_j^{\perp }$$ (A.1)

where u_j are complex valued scalars, ${\xi }_j^{\perp }=\left(\begin{array}{c}\hfill 0\hfill \\ \hfill {\xi }_j\hfill \end{array}\right)$ with ${\xi }_j\in {\mathbb{C}}^{m-1}$ are the perturbations in orthogonal directions to e₁ and ϵ ∈ ℝ is a small parameter. Define z ∈ ℝ, $b\in {\mathbb{C}}^{m-1}$ and Z ∈ C^{(m−1)×(m−1)}

$$z=\underset{j=1}{\overset{n}{\Sigma }}{u}_j{\overline{u}}_j,b=z^{-\frac{1}{2}}\underset{j=1}{\overset{n}{\Sigma }}{\overline{u}}_j{\xi }_j,Z=\underset{j=1}{\overset{n}{\Sigma }}{\xi }_j{\xi }_j^{†}.$$ (A.2)

Let ℓ₁(ϵ) be the largest eigenvalue of $H(ϵ)=\underset{j=1}{\overset{n}{\Sigma }}{\mathbf{x}}_j{\mathbf{x}}_j^{†}$ with corresponding leading eigenvector v₁(ϵ) normalized such that ${\mathbf{e}}_1^{†}{v}_1(ϵ)=1$. Then ℓ₁(ϵ) is an even analytic function of ϵ, whereas v₁(ϵ)−e₁ is an odd function of ϵ. In particular, the Taylor expansions of ℓ₁(ϵ) and ϵ = 0 are given by

$${\ell }_1(ϵ)=z+{\parallel b\parallel }^2{ϵ}^2+z^{-1}{b}^{†}(Z-{\mathit{bb}}^{†})b{ϵ}^4+\dots$$ (A.3)

$$v_1(ϵ)={\mathbf{e}}_1+z^{-1/2}\left(\begin{array}{c}0\\ b\end{array}\right)ϵ+z^{-3/2}\left(\begin{array}{c}0\\ Zb-\Vert b{\Vert }^{2}b\end{array}\right){ϵ}^3+\dots$$

Proof of Propositions 1 and 2. Since the eigenvalues of H do not depend on the direction of the vector v, without loss of generality we thus assume that v = e₁. Then, H may be realized from n_H i.i.d. observations of the form (A.1) with ϵ replaced by σ,

$${\xi }_j\sim \mathcal{C}\mathcal{N}(0,I_{m-1}),u_j\sim \{\begin{array}{c}\mathcal{C}\mathcal{N}(0,{\sigma }^2+\lambda ),\text{Proposition}1,\\ \mathcal{C}\mathcal{N}({\mu }_j,{\sigma }^2),\text{Proposition}2,\end{array}$$ (A.4)

and μ_j are arbitrary complex numbers satisfying Σ_j|μ_j|² = ω.

For each realization of u = (u_k) and $\Xi =[{\xi }_1,\dots ,{\xi }_{n_H}]\in {\mathbb{C}}^{(m-1)\times n_H}$, Lemma 1 yields the approximation (A.3) for ℓ₁(σ). To derive the distributions of the various terms in (A.3) we proceed as follows. Define $o_1=\stackrel{‒}{u}∕\Vert u\Vert \in {\mathbb{C}}^{n_H}$, choose columns o₂, …, o_n so that O = [o₁, …, o_nH] is an n_H×n_H unitary matrix, and consider the following (m−1)×n_H matrix V = ΞO. Its first column is $v_1=\Xi \stackrel{‒}{u}∕\Vert u\Vert =b$, and thus the O(ϵ²) term in (A.3) is b^†b = ||v₁||². For the forth order term, observe that Z = ΞΞ^† = VV^† and so the quantity D = b^†(Z − bb^†)b may be written as

$$D=v_1^{†}(V{V}^{†}-v_1{v}_1^{†})v_1=(v_1^{†}V){(v_1^{†}V)}^{†}-(v_1^{†}v){(v_1^{†}v)}^{†}=\underset{j=2}{\overset{n_H}{\Sigma }}|v_1^{†}{v}_j{|}^2.$$

Hence, (A.3) becomes

$${\ell }_1(ϵ)=V_0+V_2{ϵ}^2+V_4{ϵ}^4+\dots$$

where V₀ = ||u||², V₂ = ||v₁||² and $V_4=V_0^{-1}D$. To study the distributions of V₀, V₂, V₄, note that by assumption in (A.4), $u_j=\left(a_j+\iota b_j\right)∕\sqrt{2}$ with

$$\begin{array}{cc}{a}_j\sim \{\begin{array}{c}\mathcal{N}(0,\lambda +{\sigma }^2)\text{Proposition}1\hfill \\ \mathcal{N}(\sqrt{2}\mathfrak{R}({\mu }_j),{\sigma }^2)\text{Proposition}2\hfill \end{array}\hfill & b_j\sim \{\begin{array}{c}\mathcal{N}(0,\lambda +{\sigma }^2)\text{Proposition}1\hfill \\ \mathcal{N}(\sqrt{2}\Im ({\mu }_j),{\sigma }^2)\text{Proposition}2.\end{array}\hfill \end{array}$$

Therefore, ${\Vert u\Vert }^2=\frac{1}{2}\underset{j=1}{\overset{n_H}{\Sigma }}(a_j^2+b_j^2)$ is a sum of 2n_H independent squares of either mean centered or non-centered Gaussian random variables. This in turn gives

$$V_0={\Vert u\Vert }^2\sim \{\begin{array}{c}\frac{{\sigma }^2+\lambda }{2}{\chi }_{2n_H}^2,\text{Proposition}1,\\ \frac{{\sigma }^2}{2}{\chi }_{2n_H}^2(\frac{2\omega }{{\sigma }^2}),\text{Proposition}2.\end{array}$$

Since given u, O is unitary and fixed, then $v_j\mid u~\mathcal{C}\mathcal{N}(0,I_{m-1})$. Since this distribution is independent of u, $v_j~\mathcal{C}\mathcal{N}(0,I_{m-1})$. By similar arguments

$$V_2={\Vert v_1\Vert }^2\sim \frac{1}{2}{\chi }_{2m-2}^2$$

which is independent of ||u||². Finally, conditioned on(u, v₁), we have $v_1^{†}{v}_j~\mathcal{C}\mathcal{N}(0,{\Vert v_1\Vert }^2)$ and ${\mid v_1^{†}{v}_j\mid }^2~{\Vert v_1\Vert }^2{\chi }_2^2∕2$. Thus,

$$D|(u,v_1)=\underset{j=2}{\overset{n_H}{\Sigma }}{|v_1^{†}{v}_j|}^2|(u,v_1)\sim \frac{{\Vert v_1\Vert }^2}{2}{\chi }_{2n_H-2}^2,$$

where the ${\chi }_{2n_H-2}^2$ variate is independent of (u, v₁). We conclude that

$$V_4\sim \{\begin{array}{cc}\frac{1}{2{\sigma }^2+2\lambda }{({\chi }_{2n_H}^2)}^{-1}{\chi }_{2m-2}^2{\chi }_{2n_H-2}^2,& proposition1,\\ \frac{1}{2{\sigma }^2}{({\chi }_{2n_H}^2(\frac{2\omega }{{\sigma }^2}))}^{-1}{\chi }_{2m-2}^2{\chi }_{2n_H-2}^2,& proposition2.\end{array}$$

Since the random variables V₀, V₂, V₄ are independent, then so are A, B, C in either (1) or (2). This completes the proof of Propositions 1 and 2. □

To prove Propositions 3 and 4, we first introduce some additional notation and two auxiliary lemmas, whose proofs are deferred to Appendix B. For a matrix S, denote by S _jk and S ^jk the (j, k)-th entries of S and S⁻¹, respectively.

Lemma 2. Let $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ and $M=[{\mathbf{e}}_1,b]\in {\mathbb{C}}^{m\times 2}$, with the vector b fixed and orthogonal to e₁. Define a 2 × 2 diagonal matrix $D=\mathit{diag}\left(1,1∕{\Vert b\Vert }^2\right)$. Then

$$S={(M^{†}{E}^{-1}M)}^{-1}\sim C{W}_2(n_E-m+2,D),$$

and the two random variables S¹¹ and S ₂₂ are independent with

$$\begin{array}{cc}{S}^{11}\sim \frac{2}{{\chi }_{2n_E-2m+2}^2},& S_{22}\end{array}\sim \frac{{\chi }_{2n_E-2m+4}^2}{2{\Vert b\Vert }^2}.$$

Lemma 3. Let $E~\mathcal{C}{\mathcal{W}}_m(n_E,I)$ and let $A_2=\left(\begin{array}{cc}\hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill Z\hfill \end{array}\right)$, where Z is an (m − 1) × (m − 1) random matrix independent of E, with $\mathbb{E}(Z)=I_{m-1}$. Then

$$\mathbb{E}\left(\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}\right)=\frac{m-1}{(n_E-m)(n_E-m+1)}.$$

Proof of Propositions 3 and 4. Without loss of generality we may assume that the signal direction is v = e₁. Hence

$$H\sim \{\begin{array}{cc}\mathcal{C}{\mathcal{W}}_m(n_H,I+\lambda {\mathbf{e}}_1{\mathbf{e}}_1^{†}),\hfill & \text{Proposition}3,\hfill \\ \mathcal{C}{\mathcal{W}}_m(n_H,I,\omega {\mathbf{e}}_1{\mathbf{e}}_1^{†}),\hfill & \text{Proposition}4.\hfill \end{array}$$

Next, we apply a perturbation approach similar to the one used in the previous proof. To introduce a small parameter, set

$${ϵ}^2=\{\begin{array}{cc}1/(1+\lambda ),\hfill & \text{Proposition}3,\hfill \\ 1/\omega ,\hfill & \text{Proposition}4.\end{array}$$

The matrix H_ϵ = ϵ²H has a representation of the form X^†X with X [x₁, …, x_nH] where each x_j follows (A.1) but now with

$${\xi }_j~\mathcal{C}\mathcal{N}(0,I_{m-1}),u_j~\{\begin{array}{cc}\mathcal{C}\mathcal{N}(0,1),\hfill & \text{Proposition 3,}\\ \mathcal{C}\mathcal{N}({\mu }_j/\sqrt{\omega },1/\omega )& \text{Proposition 4,}\end{array}$$

where Σ |μ_j|² = ω. In particular,

$$z=\underset{j=1}{\overset{n_H}{\Sigma }}{|u_j|}^2~\{\begin{array}{cc}\frac{1}{2}{\chi }_{2n_H}^2,\hfill & \text{Proposition 3,}\\ \frac{1}{2\omega }{\chi }_{2n_H}^2(2\omega ),& \text{Proposition 4}.\end{array}$$

With b as in (A.2), using the same arguments as in the previous proof, we have that $b~\mathcal{C}\mathcal{N}(0,I_{m-1})$, independently of u.

The matrix H_ϵ may be written as H_ϵ = A₀ + ϵA₁ + ϵ²A₂, where

$$\begin{array}{ccc}{A}_0=\left(\begin{array}{cc}z& 0\\ 0& 0_{m-1}\end{array}\right),& A_1=\sqrt{z}\left(\begin{array}{cc}0& b^{†}\\ b& 0_{m-1}\end{array}\right),& A_2=\left(\begin{array}{cc}0& 0\\ 0& z\end{array}\right)\end{array}$$ (A.5)

with Z as in (A.2). For future use we define the following quantities

$$\begin{array}{cccc}{E}^{11}={\mathbf{e}}_1^{†}{E}^{-1}{\mathbf{e}}_1,& \widehat{b}=\left(\begin{array}{c}0\\ b\end{array}\right),& E^{b1}={\widehat{b}}^{†}{E}^{-1}{\mathbf{e}}_1,& E^{bb}={\widehat{b}}^{†}{E}^{-1}\widehat{b}.\end{array}$$

Note that the condition n_E ≥ m ensures that E is invertible with probability 1. This follows for example from Theorem 3.2 in [9].

The matrix E⁻¹H_ϵ is similar to the Hermitian matrix E^−1/2H_ϵE^−1/2. Therefore, all its eigenvalues are real-valued for any value of ϵ. Furthermore, since E^−1/2 H_ϵE^−1/2 is a holomorphic symmetric function of ϵ, it follows from Kato ([25], Theorem 6.1 page 120) that the largest eigenvalue ℓ₁ and its eigenprojection $\tilde{P}(ϵ)$ are analytic functions of ϵ in some neighborhood of zero, where the largest eigenvalue has multiplicity one. The projection to the corresponding eigenspace of E⁻¹H_ϵ is $P(ϵ)=E^{-1∕2}\tilde{P}(ϵ)$. As the matrix E does not depend on ϵ, this projection is also an analytic function in some neighborhood of ϵ = 0.

At ϵ = 0, E⁻¹e₁ is an eigenvector with eigenvalue E¹¹z, that is,

$$E^{-1}{H}_0{E}^{-1}{\mathbf{e}}_1=z{E}^{-1}{\mathbf{e}}_1{\mathbf{e}}_1^{†}{E}^{-1}{\mathbf{e}}_1=z{E}^{11}{E}^{-1}{\mathbf{e}}_1,$$

from which we obtain

$${\mathbf{e}}_1^{†}\tilde{P}(0)E^{-1}{\mathbf{e}}_1={\mathbf{e}}_1^{†}{E}^{-1}{\mathbf{e}}_1=E^{11}.$$ (A.6)

Since $\tilde{P}(ϵ)$ is an analytic function of ϵ and the inner product is a smooth function, then there exists a neighborhood of ϵ = 0 where ${\mathbf{e}}_1^{†}\tilde{P}(ϵ)E^{-1}{\mathbf{e}}_1$, e₁ is both analytic in ϵ and strictly positive. In this neighborhood, we may define

$$v_1(ϵ)=\frac{E^{11}}{{\mathbf{e}}_1^{†}\tilde{P}(ϵ)E^{-1}{\mathbf{e}}_1}\tilde{P}(ϵ)E^{-1}{\mathbf{e}}_1.$$ (A.7)

Clearly v₁(ϵ) is the eigenvector corresponding to the eigenvalue ℓ₁(ϵ) and it is also analytic. We thus expand

$${\ell }_1(ϵ)=\underset{j=0}{\overset{\infty }{\Sigma }}{\lambda }_j{ϵ}^j,v_1(ϵ)=\underset{j=0}{\overset{\infty }{\Sigma }}{w}_j{ϵ}^j.$$ (A.8)

Inserting these expansions into the eigenvalue-eigenvector equations E⁻¹H_ϵv₁ = ℓ₁v₁ gives the following equations: at the O(1) level,

$$E^{-1}{A}_0{w}_0={\lambda }_0{w}_0$$

whose solution is

$${\lambda }_0=z{E}^{11},w_0=\text{const}\cdot E^{-1}{\mathbf{e}}_1.$$ (A.9)

By (A.6)-(A.7), w₀ = v₁(0) = E⁻¹e₁, so the above constant is one.

By (A.7), ${\mathbf{e}}_1^{†}{v}_1(ϵ)=E^{11}={\mathbf{e}}_1^{†}{w}_0$. Hence ${\mathbf{e}}_1^{†}{w}_j=0$ for all j ≥ 1. Furthermore, since $A_0=z{\mathbf{e}}_1{\mathbf{e}}_1^{†}$, then A₀w_j = 0 for all j ≥ 1. The O(ϵ) equation is thus

$$E^{-1}{A}_1{w}_0+E^{-1}{A}_0{w}_1={\lambda }_1{w}_0+{\lambda }_0{w}_1.$$ (A.10)

However, A₀w₁ = 0. Multiplying this equation by ${\mathbf{e}}_1^{†}$ gives that

$$\begin{gathered} {\lambda }_1=\frac{{\mathbf{e}}_1^{†}{E}^{-1}{w}_0}{E^{11}}=\frac{\sqrt{z}}{E^{11}}\left\{{\mathbf{e}}_1^{†}{E}^{-1}\left(\begin{array}{cc}0& b^{†}\\ b& 0\end{array}\right)E^{-1}{\mathbf{e}}_1\right\} \\ =\frac{\sqrt{z}}{E^{11}}\left\{{\mathbf{e}}_1^{†}{E}^{-1}\left(\begin{array}{cc}0& b^{†}\\ 0& 0\end{array}\right)E^{-1}{\mathbf{e}}_1+{\mathbf{e}}_1^{†}{E}^{-1}\left(\begin{array}{cc}0& 0\\ b& 0\end{array}\right)E^{-1}{\mathbf{e}}_1\right\}=2\sqrt{z}\mathfrak{R}(E^{b1}). \end{gathered}$$ (A.11)

Inserting the expression for λ₁ into (A.10) gives that

$$\begin{gathered} w_1=\frac{1}{\sqrt{z}{E}^{11}}\left\{E^{-1}\left(\begin{array}{cc}0\hfill & b^{†}\hfill \\ b\hfill & 0\hfill \end{array}\right)E^{-1}{\mathbf{e}}_1-2\mathfrak{R}(E^{b1})E^{-1}{\mathbf{e}}_1\right\} \\ =\frac{1}{\sqrt{z}{E}^{11}}\left(E^{b1}{E}^{-1}{\mathbf{e}}_1+E^{11}{E}^{-1}\widehat{b}-2\mathfrak{R}(E^{b1})E^{-1}{\mathbf{e}}_1\right)=\frac{1}{\sqrt{z}}\left(E^{-1}\widehat{b}-\frac{\overline{E^{b1}}}{E^{11}}{E}^{-1}{\mathbf{e}}_1\right). \end{gathered}$$

The next O(ϵ²) equation is

$$E^{-1}{A}_2{w}_0+E^{-1}{A}_1{w}_1+E^{-1}{A}_0{w}_2={\lambda }_2{w}_0+{\lambda }_1{w}_1+{\lambda }_0{w}_2.$$

Multiply this equation by ${\mathbf{e}}_1^{†}$ and recall that A₀w₂ = 0 and ${\mathbf{e}}_1^{†}{w}_0=E^{11}$ gives

$$\begin{gathered} {\lambda }_2=\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}+\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_1\frac{1}{\sqrt{z}}(E^{-1}\widehat{b}-\frac{\overline{E^{b1}}}{E^{11}}{E}^{-1}{\mathbf{e}}_1)}{E^{11}} \\ =\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}+\frac{E^{11}{E}^{bb}+{(\overline{E^{b1}})}^2-2\overline{E^{b1}}\mathfrak{R}(E^{b1})}{E^{11}}=\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}+\frac{E^{11}{E}^{bb}-E^{b1}\overline{E^{b1}}}{E^{11}}. \end{gathered}$$ (A.12)

Combining (A.9)-(A.12), we obtain the following approximate stochastic representation for the largest eigenvalue ℓ₁ of E⁻¹H_ϵ

$${\ell }_1(ϵ)=z{E}^{11}+2ϵ\sqrt{z}\mathfrak{R}(E^{b1})+{ϵ}^2\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}+{ϵ}^2\frac{E^{11}{E}^{bb}-E^{b1}\overline{E^{b1}}}{E^{11}}+O_P({ϵ}^3).$$ (A.13)

Next, to derive the approximate distribution of ℓ₁ corresponding to the above equation, we study a 2 × 2 Hermitian matrix S, whose inverse is defined by

$$S^{-1}=\left(\begin{array}{cc}{E}^{11}& \overline{E^{b1}}\\ E^{b1}& E^{bb}\end{array}\right)=M^{†}{E}^{-1}M,$$

where $M=[{\mathbf{e}}_1,\widehat{b}]$ is a 2 × 2 matrix. Inverting this matrix gives

$$S=\frac{1}{E^{11}{E}^{bb}-E^{b1}\overline{E^{b1}}}\left(\begin{array}{cc}{E}^{bb}& -E^{b1}\\ -\overline{E^{b1}}& E^{11}\end{array}\right).$$

Hence in terms of the matrices S and S⁻¹, (A.13) can be written as

$${\ell }_1(ϵ)=z{S}^{11}+2ϵ\sqrt{z}\mathfrak{R}(E^{b1})+\frac{{ϵ}^2}{S_{22}}+{ϵ}^2\frac{{\mathbf{e}}_1^{†}{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}+O_P({ϵ}^3).$$ (A.14)

To establish Propositions 3 and 4, we start from (A.14). We neglect the second term $T_1=2ϵ\sqrt{z}\mathfrak{R}(E^{b1})$ which is symmetric with mean zero, and whose variance is much smaller than that of the first term. We also approximate the last term, denoted by T₂, by its mean value, using Lemma 3. We now have

$${\ell }_1(ϵ)\approx z{S}^{11}+{ϵ}^2\left\{\frac{1}{S_{22}}+c(m,n_E)\right\},$$

where c(m, n) is the expectation from Lemma 3. Since ℓ(ϵ) is the largest eigenvalue of E⁻¹H_ϵ = ϵ²E⁻¹H, (A.14) should be divided by ϵ² to obtain the largest eigenvalue of E⁻¹H. By doing so, and inserting the distributions of S¹¹ and combining this with the S₂₂ from Lemma 2 gives

$${\ell }_1\approx \frac{2z}{{ϵ}^2{\chi }_{2n_E-2m+2}^2}+\frac{2\Vert {b\Vert }^2}{{\chi }_{2n_E-2m+4}^2}+\frac{m-1}{(n_E-m)(n_E-m-1)}.$$

Next, by inserting the distributions of ||b||², z and the relevant value of ϵ, we get that for Proposition 3

$${\ell }_1(\lambda )\approx (1+\lambda )\frac{{\chi }_{2n_H}^2}{{\chi }_{2n_E-2m+2}^2}+\frac{{\chi }_{2m-2}^2}{{\chi }_{2n_E-2m+4}^2}+\frac{m-1}{(n_E-m)(n_E-m-1)}$$

and for Proposition 4

$${\ell }_1(\omega )\approx \frac{{\chi }_{2n_H}^2(2\omega )}{{\chi }_{2n_E-2m+2}^2}+\frac{{\chi }_{2m-2}^2}{{\chi }_{2n_E-2m+4}^2}+\frac{m-1}{(n_E-m)(n_E-m-1)}.$$

From Lemma 2 and the independency of u and z, all of the above χ² random variables are independent. Finally, since ratios of independent χ² random variables follow an F distribution, the two propositions follow. □

Proof of Proposition 5. By (8), the non-centrality parameter ω depends on the data only through X^†X. Conditioning on X^†X, following (7), we invoke Proposition 4 with the parameters m = p, n_H = q, and n_E = n − q to obtain

$${\ell }_1(E^{-1}H)|X\approx a_1{F}_{b_1,c_1}\left(\frac{2{\rho }^2}{1-{\rho }^2}{(X^{†}X)}_{11}\right)+a_2{F}_{b_2,c_2}+a_3.$$

Now the final result follows by integrating over the distribution of ${\left(X^{†}X\right)}_{11}~\frac{1}{2}{\chi }_{2n}^2$, and using the definition of $F_{a,b}^{\chi }(c,n)$ given in (9). □

Proof of Proposition 6 and 7. Let us assume without loss of generality that v = e₁. If $\widehat{\mathbf{v}}$ is not normalized, then we can write (10) as $R={\mid {\widehat{\mathbf{v}}}^{†}{\mathbf{e}}_1\mid }^2∕{\Vert \widehat{\mathbf{v}}\Vert }^2$. From Lemma 1, we have

$$\widehat{\mathbf{v}}=w_0+\sigma w_1+{\sigma }^3{w}_3+\dots$$

where

$$w_0={\mathbf{e}}_1,w_1=\frac{1}{\Vert u\Vert }\left(\begin{array}{c}0\\ v_1\end{array}\right),w_3=\frac{1}{{\Vert u\Vert }^3}\left(\begin{array}{c}0\\ {\Sigma }_{j=1}^{n_H}{v}_j{v}_j^{†}{v}_1\end{array}\right),$$

with i.i.d. variables $v_j~\mathcal{C}\mathcal{N}(0,I_{m-1})$ all independent of $uϵ{\mathbb{C}}^n$,

$$u\sim \{\begin{array}{c}\mathcal{C}\mathcal{N}(0,(\lambda +{\sigma }^2)I_n)\text{for}H\sim \mathcal{C}{\mathcal{W}}_m(n_H,\lambda {\mathbf{e}}_1{\mathbf{e}}_1^{†}+{\sigma }^2{I}_m)\\ \mathcal{C}\mathcal{N}(\mu ,{\sigma }^2{I}_n)\text{for}H\sim \mathcal{C}{\mathcal{W}}_m(n_H,{\sigma }^2{I}_m,(\omega /{\sigma }^2){\mathbf{e}}_1{\mathbf{e}}_1^{†})\hfill \end{array}$$

and ||μ||² = ω. Therefore,

$$R=\frac{1}{1+{\sigma }^2\frac{\Vert v_1|{|}^2}{\Vert u_1|{|}^2}+2{\sigma }^4\frac{{\Sigma }_{j=1}^{n_H}|v_1^{†}{v}_j{|}^2}{\Vert u_1|{|}^4}+O_P({\sigma }^6)}.$$

The result follows from the distribution of these quantities. □

Proof of Proposition 8. Let us rewrite (A.8) as follows

$$\widehat{v}(ϵ)=w_0+w_1ϵ+O_P({ϵ}^2),$$

where w₀ = E⁻¹e₁, $w_1=\frac{1}{\sqrt{z}}\left(E^{-1}\widehat{b}-\frac{\overline{E^{b1}}}{E^{11}}{E}^{-1}{\mathbf{e}}_1\right)$. For convenience, decompose the matrices E and E⁻¹ as

$$E=\left(\begin{array}{cc}{E}_{11}& E_{12}^{†}\\ E_{12}& E_{22}\end{array}\right),E^{-1}=\left(\begin{array}{cc}{E}^{11}& E^{{12}^{†}}\\ E^{12}& E^{22}\end{array}\right)$$ (A.15)

where E₁₁ ∈ ℝ, $E_{12}ϵ{\mathbb{C}}^{(m-1)\times 1}$, and $E_{22}ϵ{\mathbb{C}}^{(m-1)\times (m-1)}$. Consequently, E¹¹ = 1/(E₁₁ − E₁₂^†E₂₂⁻¹E₁₂) ∈ ℝ and $E^{12}=-E^{11}{E_{22}}^{-1}{E}_{12}ϵ{\mathbb{C}}^{(m-1)\times 1}$. The exact form of E²² is unimportant as it does not affect our calculations.

Let us now focus on the numerator of R. Since e^†w₁ = 0, we have

$${\widehat{v}}^{†}{e}_1=E^{11}+O_P({ϵ}^2),$$

from which we obtain

$$|{\widehat{v}}^{†}{\mathbf{e}}_1{|}^2=(E^{11})+O_P({ϵ}^2).$$

The denominator of R can be written as

$$\Vert \widehat{v}|{|}^2={\mathbf{e}}_1^{†}(E^{-1}){\mathbf{e}}_1+\frac{2}{\sqrt{z}}\mathfrak{R}\left\{{\mathbf{e}}_1^{†}{(E^{-1})}^2\widehat{b}\right\}ϵ-\frac{2}{\sqrt{z}{E}^{11}}\mathfrak{R}\left\{E^{b1}\right\}{\mathbf{e}}_1^{†}{(E^{-1})}^2{\mathbf{e}}_1ϵ+O_P({ϵ}^2)$$

Using the decomposition of E⁻¹ given in (A.15), we get

$$\Vert \widehat{v}|{|}^2={\left(E^{11}\right)}^2+\Vert E^{12}|{|}^2+\frac{2}{\sqrt{z}}\mathfrak{R}\left\{E^{11}{E}^{{12}^{†}}b+E^{{21}^{†}}{E}^{22}b\right\}ϵ-\frac{2}{\sqrt{z}{E}^{11}}\mathfrak{R}\left\{E^{b1}\right\}\left\{{\left(E^{11}\right)}^2+\Vert E^{12}|{|}^2\right\}ϵ+O_P({ϵ}^2).$$

Now we can conveniently express R as

$$R=\frac{{\left(E^{11}\right)}^2}{{\left(E^{11}\right)}^2+\left|\right|E^{12}|{|}^2}+\frac{{\left(E^{11}\right)}^2}{{\left\{{\left(E^{11}\right)}^2+\left|\right|E^{12}|{|}^2\right\}}^2}(P_E-Q_E)ϵ+O_P({ϵ}^2),$$

where

$$P_E=\frac{2}{\sqrt{z}{E}^{11}}\left\{{\left(E^{11}\right)}^2+{\Vert E^{12}\Vert }^2\right\}\mathfrak{R}\left\{E^{b1}\right\},Q_E=\frac{2}{\sqrt{z}}\mathfrak{R}\left\{E^{12†}\left(E^{11}{I}_{m-1}+E^{22}\right)b\right\}.$$

Since P_E and Q_E are zero mean random variables, we neglect them to obtain

$$R\approx \frac{1}{1+\Vert E_{22}^{-1}{E}_{22}|{|}^2},$$

where we have used the relation $E^{12}=-E^{11}{E}_{22}^{-1}{E}_{12}ϵ{\mathbb{C}}^{(m-1)\times 1}$. Noting that $E_{22}^{-1}{E}_{12}\mid E_{22}~\mathcal{C}\mathcal{N}(0,E_{22}^{-1})$ with $E_{22}~\mathcal{C}{\mathcal{W}}_{m-1}(n_E,I_{m-1})$, we can show that $1∕\left(1+{\Vert E_{22}^{-1}{E}_{12}\Vert }^2\right)$ is beta distributed with parameters n_E − m + 2 and m − 1. Now the final result follows from the observation that, for $X~{\chi }_p^2$ and $Y~{\chi }_q^2$, X/(X + Y) is beta distributed with parameters p/2 and q/2. □

Appendix B. Proof of Auxiliary Lemmas

Proof of Lemma 1. Write the m × n matrix X(ϵ) = [x₁, …, x_n] and observe that X(−ϵ) = UX(ϵ), where U = diag(1, −1, …, −1), is an orthogonal matrix. Thus, H(−ϵ) = U^TH(ϵ)U has the same eigenvalues as H(ϵ). In particular, the largest eigenvalue ℓ₁ and its corresponding eigenvector v₁ satisfy

$${\ell }_1(-ϵ)={\ell }_1(ϵ),v_1(-ϵ)=U{v}_1(ϵ).$$ (B.1)

Hence ℓ₁ and the first component of v₁ are even functions of ϵ whereas the remaining components of v₁ are odd.

We decompose the matrix $H(ϵ)={\Sigma }_{j=1}^n{\mathbf{x}}_j{\mathbf{x}}_j^{†}$ as

$$\begin{gathered} H(ϵ)=\sum _{j=1}^n(u_j{\mathbf{e}}_1+ϵ{\xi }_j^{\perp }){(u_j{\mathbf{e}}_1+ϵ{\xi }_j^{\perp })}^{†}=\sum _{j=1}^n{\mid u_j\mid }^2{\mathbf{e}}_1{\mathbf{e}}_1^{†}+ϵ\sum _{j=1}^n[{\xi }_j^{\perp }\cdot \overline{u_j}{\mathbf{e}}_1^{†}\cdot {\xi }_j^{{\perp }^{†}}]+{ϵ}^2\sum _{j=1}^n{\xi }_j^{\perp }\cdot {\xi }_j^{{\perp }^{†}} \\ =\left(\begin{array}{cc}\hfill z\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0_{m-1}\hfill \end{array}\right)+ϵ\sqrt{z}\left(\begin{array}{cc}\hfill 0\hfill & \hfill b^{†}\hfill \\ \hfill b\hfill & \hfill 0_{m-1}\hfill \end{array}\right)+{ϵ}^2\left(\begin{array}{cc}\hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill Z\hfill \end{array}\right)=A_0+ϵA_1+{ϵ}^2{A}_2, \end{gathered}$$

with the matrices A₀, A₁ and A₂ are given in (A.5). Following similar arguments which lead to (A.7) and (A.8) with E = I, we can establish that ℓ₁(ϵ) and v₁(ϵ) are analytic in some neighborhood of zero. Therefore, we have the following Taylor series expansions:

$${\ell }_1(ϵ)={\lambda }_0+{ϵ}^2{\lambda }_2+{ϵ}^4{\lambda }_4+\dots \text{and}{v}_1(ϵ)=w_0+ϵw_1+{ϵ}^2{w}_2+{ϵ}^3{w}_3+{ϵ}^4{w}_4+\dots$$ (B.2)

Also, the eigenprojection P(ϵ) of ℓ₁ satisfies

$$v_1(ϵ)=\frac{1}{{\mathbf{e}}_1^{†}P(ϵ){\mathbf{e}}_1}P(ϵ){\mathbf{e}}_1.$$ (B.3)

Inserting the expansions (B.2) into the eigenvalue equation Hv₁ = ℓ₁v₁ gives the following set of equations for r ≥ 0

$$A_0{w}_r+A_1{w}_{r-1}+A_2{w}_{r-2}={\lambda }_0{w}_r+{\lambda }_2{w}_{r-2}+{\lambda }_4{w}_{r-4}+\dots$$ (B.4)

with the convention that vectors with negative subscripts are zero. From the r = 0 equation, A₀w₀ = λ₀w₀, we readily find that

$${\lambda }_0=z,w_0=const·{\mathbf{e}}_1.$$

Eq. (B.3) implies that ${\mathbf{e}}_1^{†}{v}_1=1$ and w₀ = v₁(0) = e₁. This implies that w_j, for j ≥ 1, is orthogonal to e₁, that is orthogonal to w₀.

From the eigenvector remarks following (B.1) it follows that w_2j = 0 for j ≥ 1. These remarks allow considerable simplification of (B.4); we use those for r = 1 and r = 3

$$A_1{w}_0={\lambda }_0{w}_1,A_2{w}_1={\lambda }_0{w}_3+{\lambda }_2{w}_1$$ (B.5)

from which we obtain

$$w_1=z^{-1/2}\widehat{b},w_3={\lambda }_0^{-1}(A_2-{\lambda }_{2}I)w_1.$$ (B.6)

Multiply (B.4) on the left by $w_0^{†}$ and use the first equation of (B.5) to obtain, for r even,

$${\lambda }_r={(A_1{w}_0)}^{†}{w}_{r-1}={\lambda }_0{w}_1^{†}{w}_{r-1}$$

and hence

$${\lambda }_2={\lambda }_0{w}_1^{†}{w}_1=b^{†}b\text{and}{\lambda }_4=w_1^{†}(A_2-{\lambda }_{2}I)w_1=z^{-1}{b}^{†}(z-b{b}^{†})b.$$

Therefore, we can further simplify (B.6) to yield

$$w_1=z^{-1/2}\widehat{b},w_2=z^{-3/2}(A_2-{\Vert b\Vert }^2{I}_{m-1})\widehat{b}=z^{-3/2}\left(\begin{array}{c}0\\ Zb-{\Vert b\Vert }^{2}b\end{array}\right).\square$$

To prove Lemmas 2 and 3, we shall use the following two claims, which are the complex analogues of Theorems 3.2.10 and 3.2.11 in Muirhead [32]. While their proofs are similar to those in the real valued case, for completeness we present them below.

Claim 1. Suppose $A~\mathcal{C}{\mathcal{W}}_m(n,\Sigma )$ with n > m − 1 where A and Σ are partitioned as follows

$$A=\left(\begin{array}{cc}{A}_{11}& A_{12}\\ A_{21}& A_{22}\end{array}\right)\Sigma =\left(\begin{array}{cc}{\Sigma }_{11}& {\Sigma }_{12}\\ {\Sigma }_{21}& {\Sigma }_{22}\end{array}\right)$$

and let $A_{11\cdot 2}=A_{11}-A_{12}{A}_{22}^{-1}{A}_{21}$, and ${\Sigma }_{11\cdot 2}={\Sigma }_{11}-{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21}$. Then, A_11.2 is distributed as $\mathcal{C}{\mathcal{W}}_k(n-m+k,{\Sigma }_{11\cdot 2})$ and is independent of A₁₂, A₂₁ and A₂₂.

Claim 2. Let $A~\mathcal{C}{\mathcal{W}}_m(n,\Sigma )$ and let M be a k×m matrix of rank k, where M is independent of A. Then ${({\mathit{MA}}^{-1}{M}^{†})}^{-1}~\mathcal{C}{\mathcal{W}}_k(n-m+k,{(M{\Sigma }^{-1}{M}^{†})}^{-1})$.

Proof of Claim 1. Let C = Σ⁻. We partition it as follows,

$$C=\left(\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& C_{22}\end{array}\right),$$ (B.7)

where $C_{11}ϵ{\mathbb{C}}^{k\times k}$, $C_{22}ϵ{\mathbb{C}}^{(m-k)\times (m-k)}$, and $C_{12}ϵ{\mathbb{C}}^{k\times (m-k)}$ with $C_{12}^{†}=C_{21}$. Consequently, ${\Sigma }_{11\cdot 2}^{-1}=C_{11}$.

Following [15, 19], the density of A is given by

$$f(A)=\frac{{\det }^{n-m}(A)}{{\Gamma }_m(n){\det }^n(\Sigma )}{e}^{-\mathrm{tr}({\Sigma }^{-1}A)}$$ (B.8)

where tr(·) denotes the trace operator and

$${\Gamma }_m(n)={\pi }^{\frac{m}{2}(m-1)}\prod _{j=1}^m\Gamma (n-j+1)$$

with Γ(·) denoting the classical gamma function.

To prove the claim we shall study the form of det(A) and of tr(Σ⁻¹A). First of all, we have that

$$\det (A)=\det (A_{22})\det (A_{11·2}).$$

Next, we introduce a change of variables from the entries of the matrix A, to $A_{11\cdot 2}=A_{11}-A_{12}{A}_{22}^{-1}{A}_{21}$, B₁₂ = A₁₂, B₂₂ = A₂₂. The Jacobian of this transformation is an upper triangular matrix, with all diagonal entries equal to one. Hence, the volume element in (B.8) is dA = dA₁₁dA₁₂dA₂₂ = dA_11.2dB₁₂dB₂₂. Furthermore, using the expansion

$$\begin{gathered} \text{tr}\left({\Sigma }^{-1}A\right)=\text{tr}\left(\left(\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& C_{22}\end{array}\right)\left(\begin{array}{cc}{A}_{11·2}+B_{12}{B}_{22}^{-1}{B}_{21}& B_{12}\\ B_{21}& B_{22}\end{array}\right)\right) \\ =\text{tr}(C_{11}{A}_{11·2})+\text{tr}\left(C_{11}{B}_{12}{B}_{22}^{-1}{B}_{21}\right)+\text{tr}(C_{12}{B}_{21})+\text{tr}(C_{21}{B}_{12})+\text{tr}(C_{22}{B}_{22}) \end{gathered}$$

along with the fact that $B_{21}=B_{12}^{†}$ yields that

$$f(A_{11·2},B_{12},B_{22})=\frac{{\det }^{n-m}(B_{22}){\det }^{n-m}(A_{11·2})}{{\det }^n({\Sigma }_{22}){\det }^n({\Sigma }_{11·2}){\Gamma }_m(n)}\times e^{-\text{tr}({\Sigma }_{11·2}^{-1}{A}_{11·2})-\text{tr}\left({\Sigma }_{11.2}^{-1}{B}_{12}{B}_{22}^{-1}{B}_{12}^{†}\right)}\times e^{-\text{tr(}{C}_{21}{B}_{12})-\text{tr}(C_{21}{B}_{12}{)}^{†}-\text{tr}(C_{22}{B}_{22})}.$$ (B.9)

Now we may use the decomposition

$$\begin{gathered} {\Gamma }_m(n)={\pi }^{\frac{k}{2}(k-1)}\prod _{j=1}^k\Gamma (n-m+k-j+1)\times {\pi }^{\frac{(m-k)}{2}(m+k-1)}\prod _{j=1}^{m-k}\Gamma (n-j+1) \\ ={\Gamma }_k(n-m+k)\times {\pi }^{\frac{(m-k)}{2}(m+k-1)}\prod _{j=1}^{m-k}\Gamma (n-j+1) \end{gathered}$$

to rewrite (B.9) as

$$f(A_{11·2},B_{12},B_{22})=f_1(A_{11·2})\times f_2(B_{12},B_{22}),$$ (B.10)

where

$$f_1(A_{11·2})=\frac{{\det }^{n-m+k-k}(A_{11·2})}{{\det }^{n-m+k}({\Sigma }_{11·2}){\Gamma }_k(n-m+k)}{e}^{-\text{tr(}{\Sigma }_{11·2}^{-1}{A}_{11·2})},$$ (B.11)

and

$$f_2(B_{12},B_{22})=\frac{{\det }^{n-m}(B_{22})}{{\pi }^{\frac{(m-k)}{2}(m+k-1)}{\prod }_{j=1}^{m-k}\Gamma (n-j+1){\det }^{m-k}({\Sigma }_{11.2}){\det }^n({\Sigma }_{22})}\times e^{-\text{tr}\left({\Sigma }_{11·2}^{-1}{B}_{12}{B}_{22}^{-1}{B}_{12}^{†}\right)-\text{tr}(C_{\text{21}}{B}_{\text{12}})-\text{tr}(C_{\text{21}}{B}_{12}{)}^{†}-\text{tr}(C_{22}{B}_{22})}.$$

The factorization in (B.10) establishes that A_11.2 is independent of A₁₂ and A₂₂. Finally, (B.11) implies that $A_{11\cdot 2}~\mathcal{C}{\mathcal{W}}_k(n-m+k,{\Sigma }_{11\cdot 2})$ which concludes the proof. □

Proof of Claim 2. Set B = Σ^−1/2AΣ^−1/2. Now $B~\mathcal{C}{\mathcal{W}}_m(n,I)$. For R = MΣ^−1/2, (MA⁻¹M^†)⁻¹ = (RB⁻¹R^†)⁻¹ and (MΣ⁻¹M^†)⁻¹ = (RR^†)⁻¹. Thus, it is sufficient to prove that ${({\mathit{RB}}^{-1}{R}^{†})}^{-1}~\mathcal{C}{\mathcal{W}}_k(n-m+k,{({\mathit{RR}}^{†})}^{-1})$. Let R = L[I_k : 0]H be the SVD decomposition of R, where L is k × k and nonsingular and H is m × m unitary. Now,

$$\begin{gathered} {({\mathit{RB}}^{-1}{R}^{†})}^{-1}={\left(L[I_k:0]{\mathit{HB}}^{-1}{H}^{†}{[I_k:0]}^{\prime }{L}^{†}\right)}^{-1} \\ ={(L^{-1})}^{†}{\left([I_k:0]{({\mathit{HBH}}^{†})}^{-1}{[I_k:0]}^{\prime }\right)}^{-1}{L}^{-1}={(L^{-1})}^{†}{\left([I_k:0]C^{-1}{[I_k:0]}^{\prime }\right)}^{-1}{L}^{-1} \end{gathered}$$

where $C={\mathit{HBH}}^{†}~\mathcal{C}{\mathcal{W}}_m(n,I)$. Let

$$F=C^{-1}=\left(\begin{array}{cc}{F}_{11}& F_{12}\\ F_{21}& F_{22}\end{array}\right),C=\left(\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& C_{22}\end{array}\right)$$

where F₁₁ and C₁₁ are k × k. Then ${({\mathit{RB}}^{-1}{R}^{†})}^{-1}=(L^{-1}{)}^{†}{F}_{11}^{-1}{L}^{-1}$, and since $F_{11}^{-1}=C_{11}-C_{12}{C}_{22}^{-1}{C}_{21}$, it follows from Claim 1 that $F_{11}^{-1}~\mathcal{C}{\mathcal{W}}_k(n-m+k,I_k)$. Hence ${(L^{-1})}^{†}{F}_{11}^{-1}{L}^{-1}~\mathcal{C}{\mathcal{W}}_k(n-m+k,{({\mathit{LL}}^{†})}^{-1})$, and since (LL^†)⁻¹ = (RR^†)⁻¹, the proof is complete. □

Proof of Lemma 2. Note that $S^{11}=E^{11}={\mathbf{e}}_1^T{E}^{-1}{\mathbf{e}}_1$. Then, by Claim 2, ${(S^{11})}^{-1}~\mathcal{C}{\mathcal{W}}_1(n_E-m+1,I_1)={\chi }_{2n_E-2m+2}^2∕2$, meaning $S^{11}~2∕{\chi }_{2n_E-2m+2}^2$. Next, by definition S = (M^†E⁻¹M)⁻¹, with fixed M. Thus, by the same claim, $S~\mathcal{C}{\mathcal{W}}_2(n_E-m+2,D)$ from which we obtain $S_{22}~{\chi }_{2n_E-2m+4}^2∕(2{\Vert b\Vert }^2)$. Finally, since ${(S^{11})}^{-1}=S_{11}-S_{12}{S}_{22}^{-1}{S}_{21}$, by Claim 1, (S¹¹)⁻¹ is independent of S₂₂. □

Proof of Lemma 3. First we decompose the expectation as follows:

$$\mathbb{E}\left(\frac{{\mathbf{e}}_1^T{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}\right)={\mathbb{E}}_E\left\{{\mathbb{E}}_{A\mid E}\left(\frac{{\mathbf{e}}_1^T{E}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}\right)\right\}.$$

Next, since A₂ is independent of E,

$$\mathbb{E}(A_2\mid E)=\mathbb{E}(A_2)=\left(\begin{array}{cc}\hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill I_{m-1}\hfill \end{array}\right).$$

Combining the above two equations gives that

$$\mathbb{E}\left(\frac{{\mathbf{e}}_1^t{\mathit{opE}}^{-1}{A}_2{E}^{-1}{\mathbf{e}}_1}{E^{11}}\right)=\mathbb{E}\left(\sum _{j=2}^m\frac{{\Vert E^{1_j}\Vert )}^2}{E^{11}}\right)=(m-1)\mathbb{E}\left(\frac{{\Vert E^{12}\Vert }^2}{E^{11}}\right).$$

To compute this expectation, consider the matrix $S^{-1}={[\begin{array}{cc}\hfill {\mathbf{e}}_1\hfill & \hfill {\mathbf{e}}_2\hfill \end{array}]}^{⊺}{E}^{-1}[\begin{array}{cc}\hfill {\mathbf{e}}_1\hfill & \hfill {\mathbf{e}}_2\hfill \end{array}]=\left(\begin{array}{cc}\hfill E^{11}\hfill & \hfill E^{21}\hfill \\ \hfill {\overline{E}}^{21}\hfill & \hfill E^{22}\hfill \end{array}\right)$. Since S²² = E²² and S₂₂ = E¹¹/(E¹¹E²² − ‖E¹²‖²), we have

$$\frac{1}{S_{22}}=E^{22}-\frac{{\Vert E^{12}\Vert }^2}{E^{11}}.$$ (B.12)

Noting that $S_{22}~\frac{1}{2}{\chi }_{2n_E-2m+4}^2$ and $E^{22}~2∕{\chi }_{2n_E-2m+2}^2$, we take the expectation of both sides of (B.12) to obtain

$$\mathbb{E}\left\{\frac{{(E^{12})}^2}{E^{11}}\right\}=\mathbb{E}\left(E^{22}\right)-\mathbb{E}\left(\frac{2}{{\chi }_{2n_E-2m+4}^2}\right)=\frac{1}{(n_E-m)(n_E-m+1)}$$

which completes the proof. □