Asymptotics of eigenstructure of sample correlation matrices for high-dimensional spiked models

Abstract

Sample correlation matrices are widely used, but for high-dimensional data little is known about their spectral properties beyond “null models”, which assume the data have independent coordinates. In the class of spiked models, we apply random matrix theory to derive asymptotic first-order and distributional results for both leading eigenvalues and eigenvectors of sample correlation matrices, assuming a high-dimensional regime in which the ratio p/n, of number of variables p to sample size n, converges to a positive constant. While the first-order spectral properties of sample correlation matrices match those of sample covariance matrices, their asymptotic distributions can differ significantly. Indeed, the correlation-based fluctuations of both sample eigenvalues and eigenvectors are often remarkably smaller than those of their sample covariance counterparts.

1. Introduction

Estimating a correlation matrix is a fundamental statistical task. It is widely applied in areas such as viral sequence analysis and vaccine design in biology (Dahirel et al., 2011, Quadeer et al., 2014, 2018), large portfolio design in finance (Plerou et al., 2002), signal detection in radio astronomy (Leshem and van der Veen, 2001), and collaborative filtering (Liu et al., 2014, Ruan et al., 2016), among many others. In classical statistical settings, with a limited number of variables p and a large sample size n, the sample correlation matrix performs well and its statistical properties are well understood; see, for example, Girshick (1939), Konishi (1979), Fang and Krishnaiah (1982), Schott (1991), Kollo and Neudecker (1993), and Boik (2003). Modern applications, however, often exhibit high dimensionality, with large p and, in many cases, limited n. In such cases, sample correlation matrices become inaccurate owing to an aggregation of statistical noise across the matrix coordinates that is visible in the eigen-spectrum (El Karoui, 2009). This is particularly important in principal component analysis (PCA), which often involves projecting data onto the leading eigenvectors of the sample correlation matrix or, equivalently, onto those of the sample covariance matrix after standardizing the data.

Despite the extensive use of sample correlation matrices, relatively little is known about theoretical properties of their eigen-spectra in high dimensions. In contrast, sample covariance matrices have been studied extensively, and a rich body of literature now exists (e.g., Yao et al. (2015)). Their asymptotic properties have typically been described in high-dimensional settings in which the number of samples and variables both grow large, often though not always at the same rate, based on the theory of random matrices. Specific first- and second-order results for the eigenvalues and eigenvectors of sample covariance matrices are reviewed in Bai and Silverstein (2009), Couillet and Debbah (2011), and Yao et al. (2015).

For the spectra of high-dimensional sample correlation matrices, current theoretical results focus on the simplest “null model” scenario, in which the data are assumed to be independent. In this null model, correlation matrices share many of the same asymptotic properties as covariance matrices from independent and identically distributed (i.i.d.) data, with zero mean and unit variance. Thus, the empirical eigenvalue distribution converges to the Marchenko–Pastur distribution, almost surely (Jiang, 2004b), and the largest and smallest eigenvalues converge to the edges of this distribution (Jiang, 2004b, Xiao and Zhou, 2010). Moreover, the rescaled largest and smallest eigenvalues asymptotically follow the Tracy–Widom law (Bao et al., 2012, Pillai and Yin, 2012). Central limit theorems (CLTs) for linear spectral statistics have also been derived (Gao et al., 2017). A separate line of work studies the maximum absolute off-diagonal entry of sample correlation matrices, referred to as “coherence” (Jiang, 2004a, Cai and Jiang, 2011, 2012), which has been proposed as a statistic for conducting independence tests; see also Cochran et al. (1995), Mestre and Vallet (2017), and the references therein. Hero and Rajaratnam (2011, 2012) use a related statistic to identify variables exhibiting strong correlations, an approach referred to as “correlation screening.”

For non-trivial correlation models, however, asymptotic results for the spectra of sample correlation matrices are quite scarce. Notably, El Karoui (2009) shows that, for a fairly general class of covariance models with bounded spectral norm, to first order, the eigenvalues of sample correlation matrices asymptotically coincide with those of sample covariance matrices with unit-variance data, generalizing earlier results of Jiang (2004b) and Xiao and Zhou (2010). Under similar covariance assumptions, recent work also presents CLTs for linear spectral statistics of sample correlation matrices (Mestre and Vallet, 2017), extending the work of Gao et al. (2017). First order behavior again coincides with that of sample covariances. However, the asymptotic fluctuations are quite different for sample correlation matrices.

This study considers a particular class of correlation matrix models, the so-called “spiked models,” in which a few large or small eigenvalues of the population covariance (or correlation) matrix are assumed to be well separated from the rest (Johnstone, 2001). Spiked covariance models are relevant in applications in which the primary covariance information lies in a relatively small number of eigenmodes. Such applications include collaborative signal detection in cognitive radio systems (Bianchi et al., 2009), fault detection in sensor networks (Couillet and Hachem, 2013), adaptive beamforming in array processing (Hachem et al., 2013, Vallet et al., 2015, Yang et al., 2018), and protein contact prediction in biology (Cocco et al., 2011, 2013). The spectral properties of spiked covariance models have been well studied, with precise analytical results established for the asymptotic first-order and distributional properties of both eigenvalues and eigenvectors; see, for example, Baik et al. (2005), Baik and Silverstein (2006), Paul (2007), Bai and Yao (2008), Benaych-Georges and Nadakuditi (2011), Couillet and Hachem (2013), Bloemendal et al. (2016). For reviews, see also Couillet and Debbah (2011, Chapter 9) and Yao et al. (2015, Chapter 11).

Less is known about the spectrum of sample correlation matrices under spiked models. Although the asymptotic first-order behavior is expected to coincide with that of the sample covariance, as a consequence of El Karoui (2009), a simple simulation reveals striking differences in the fluctuations of both sample eigenvalues and eigenvectors; see Figure 1.

Figure 1:

A simple simulation shows remarkable distributional differences between sample covariance and sample correlation. From n = 200 i.i.d. Gaussian samples, $x_i\in {ℝ}^{100}$, with covariance Σ = blkdiag(Σ_s, I₉₀), where ${\left({\mathrm{\Sigma }}_s\right)}_{i,j=1}^{10}={\left(r^{|i-j|}\right)}_{i,j=1}^{10}$, for r = 0.95, we compute the sample covariance and sample correlation, and show: (a) the empirical density (normalized histogram) of the largest sample eigenvalue, along with a Gaussian distribution with its estimated mean and standard deviation (solid line), and (b) a scatter plot of the leading sample eigenvector, projected onto the second (x-axis) and fourth (y-axis) population eigenvectors. A striking variance reduction is observed in the sample correlation for both (a) and (b). A similar variance reduction is observed for different choices of population eigenvectors in (b); the selected choice (being the second and fourth eigenvectors) facilitates the illustration of an additional correlation effect in the sample-to-population eigenvector projections.

Here, we present theoretical results to describe these observed phenomena. We obtain asymptotic first-order and distribution results for the eigenvalues and eigenvectors of sample correlation matrices under a spiked model. Paul (2007) proved theorems for sample covariance matrices in the special case of Gaussian data. In essence, we present analogs of these theorems for sample correlation matrices, and extend them to non-Gaussian data. To first order, the eigenvalues and eigenvectors coincide asymptotically with those of sample covariance matrices; however, their fluctuations can be very different. Indeed, for both the largest sample correlation eigenvalues (Theorem 1) and the projections of the corresponding eigenvectors (Theorem 2), the asymptotic variances admit a decomposition into three terms. The first term is just the asymptotic variance for sample covariance matrices generated from Gaussian data; the second adds corrections due to non-Gaussianity, and the third captures further corrections due to data normalization imposed by the sample correlation matrix. (This last amounts to normalizing the entries of the sample covariance matrix using the sample variances). Consistent with the example shown in Figure 1(a), in the CLT for the leading sample eigenvalues, the sample correlation eigenvalues often show lower fluctuations—despite the variance normalization—than those of the sample covariance eigenvalues. As seen in Figure 1(b), the (normalized) eigenvector projections are typically asymptotically correlated, even for Gaussian data, unlike the sample covariance setting of Paul (2007, Theorem 5).

Technical contributions

We build on and extend a set of random matrix tools for studying spiked covariance models. The companion manuscript (Johnstone and Yang, 2018) [JY], gives an exposition and parallel treatment for sample covariance matrices. Important adaptations are needed here to account for the data normalization imposed by sample correlation matrices. Among key technical contributions of our work, basic to our main theorems, are asymptotic first-order and distributional properties for bilinear forms and matrix quadratic forms with normalized entries, Section 4. A novel regularization-based proof strategy is used to establish the inconsistency of eigenvector projections in the case of “subcritical” spiked eigenvalues, Theorem 3.

Model M

Let $x\in {ℝ}^{m+p}$ be a random vector with finite (4+δ)th moment for some δ > 0. Consider the partition

$$x=\left[\begin{array}{l}\xi \\ \eta \end{array}\right].$$

Assume that $\xi \in {ℝ}^m$ has mean zero and covariance Σ, and is independent of $\eta \in {ℝ}^p$, which has i.i.d components η_i with mean zero and unit variance. Let ${\mathrm{\Sigma }}_D=\mathrm{diag}\left({\sigma }_1^2,\dots ,{\sigma }_m^2\right)$ be the diagonal matrix containing the variances of ξ_i, and let $\mathrm{\Gamma }={\mathrm{\Sigma }}_D^{-1/2}\mathrm{\Sigma }{\mathrm{\Sigma }}_D^{-1/2}$ be the correlation matrix of ξ with eigen-decomposition Γ = PLP^T, where P = [p₁, …, p_m] is the eigenvector matrix, and L = diag(ℓ₁, …, ℓ_m) contains the spike correlation eigenvalues ℓ₁ ≥ … ≥ ℓ_m > 0.

The correlation matrix of x is therefore Γ_x = blkdiag(Γ, I), with eigenvalues ℓ₁, …, ℓ_m,1, …, 1, and corresponding eigenvectors ${\mathfrak{p}}_1,\dots ,{\mathfrak{p}}_m$, e_m+1, …, e_m+p, where ${\mathfrak{p}}_i={\left[p_i^T{0}_p^T\right]}^T$ and e_j is the jth canonical vector (i.e., a vector of all zeros, except for a one in the jth coordinate).

Consider a sequence of i.i.d. copies of x, the first n of which fill the columns of the (m + p) × n data matrix X = (x_ij). We assume m is fixed, whereas p and n increase with

$${\gamma }_n=p/n\to \gamma >0\text{as}p,n\to \infty .$$

Notation

Let S = n⁻¹XX^T be the sample covariance matrix, and $S_D=\mathrm{diag}\left({\widehat{\sigma }}_1^2,\dots ,{\widehat{\sigma }}_{m+p}^2\right)$ be the diagonal matrix containing the sample variances. Let $R=S_D^{-1/2}S{S}_D^{-1/2}$ be the sample correlation matrix, with corresponding νth sample eigenvalue and eigenvector satisfying

$$R{\widehat{\mathfrak{p}}}_{\nu }={\widehat{\ell }}_{\nu }{\widehat{\mathfrak{p}}}_{\nu },$$

where, for later use, we partition ${\widehat{\mathfrak{p}}}_{\nu }={\left[{\widehat{p}}_{\nu }^T,{\widehat{v}}_{\nu }^T\right]}^T$. Here ${\widehat{p}}_{\nu }$ is the subvector of ${\widehat{\mathfrak{p}}}_{\nu }$ restricted to the first m coordinates.

For $\ell >1+\sqrt{\gamma }$, define

$$\rho (\ell ,\gamma )=\ell +\gamma \frac{\ell }{\ell -1},\dot{\rho }(\ell ,\gamma )=\frac{\partial \rho (\ell ,\gamma )}{\partial \ell }=1-\frac{\gamma }{{(\ell -1)}^2}.$$

For an index ν, for which ${\ell }_{\nu }>1+\sqrt{\gamma }$ is a simple eigenvalue, set

$${\rho }_{\nu }=\rho \left({\ell }_{\nu },\gamma \right),{\rho }_{\nu n}=\rho \left({\ell }_{\nu },{\gamma }_n\right),{\dot{\rho }}_{\nu }=\dot{\rho }\left({\ell }_{\nu },\gamma \right),{\dot{\rho }}_{vn}=\dot{\rho }\left({\ell }_{\nu },{\gamma }_n\right).$$ (1.1)

We refer to eigenvalues satisfying ${\ell }_{\nu }>1+\sqrt{\gamma }$ as “supercritical,” and those satisfying ${\ell }_{\nu }\le 1+\sqrt{\gamma }$ as “subcritical,” with the quantity $1+\sqrt{\gamma }$ referred to as the “phase transition.”

To describe and interpret the variance terms in the limiting distributions to follow, we need some definitions. Let ${\overline{\xi }}_i={\xi }_i/{\sigma }_i$ and ${\kappa }_{ij}=\mathbb{E}{\overline{\xi }}_i{\overline{\xi }}_j$ denote the scaled components of ξ and their covariances; of course κ_ii = 1. The corresponding scaled fourth-order cumulants are

$${\kappa }_{ij{i}^{\prime }{j}^{\prime }}=\mathbb{E}\left[{\overline{\xi }}_i{\overline{\xi }}_j{\overline{\xi }}_{i^{\prime }}{\overline{\xi }}_{j^{\prime }}\right]-{\kappa }_{ij}{\kappa }_{i^{\prime }{j}^{\prime }}-{\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}-{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}.$$ (1.2)

When ξ is Gaussian,${\kappa }_{ij{i}^{\prime }{j}^{\prime }}\equiv 0$.

The effect of variance scaling in the correlation matrix is described using additional quadratic functions of $\left({\overline{\xi }}_i\right)$, defined by

$${\chi }_{ij}={\overline{\xi }}_i{\overline{\xi }}_j,{\psi }_{ij}={\kappa }_{ij}\left({\overline{\xi }}_i^2+{\overline{\xi }}_j^2\right)/2$$ (1.3)

$${\check{\kappa }}_{ij{i}^{\prime }{j}^{\prime }}=\mathrm{Cov}\left({\psi }_{ij},{\psi }_{i^{\prime }{j}^{\prime }}\right)-\mathrm{Cov}\left({\psi }_{ij},{\chi }_{i^{\prime }{j}^{\prime }}\right)-\mathrm{Cov}\left({\chi }_{ij},{\psi }_{i^{\prime }{j}^{\prime }}\right).$$ (1.4)

Tensor notation

For convenience, it is useful to consider ${\kappa }_{ij{i}^{\prime }{j}^{\prime }}$ and ${\check{\kappa }}_{ij{i}^{\prime }{j}^{\prime }}$ as entries of four-dimensional tensor arrays κ and $\check{\kappa }$, respectively, and to define an additional array ${\mathcal{P}}^{\mu {\mu }^{\prime }\nu {\nu }^{\prime }}$ with entries $p_{\mu ,i}{p}_{{\mu }^{\prime },j}{p}_{\nu ,i^{\prime }}{p}_{{\nu }^{\prime },j^{\prime }}$. In addition, define ${\mathcal{P}}^{\nu }$ as ${\mathcal{P}}^{\nu \nu \nu \nu }$. Finally, for a second array A of the same dimensions,

$$\left[{\mathcal{P}}^{\nu },A\right]=\sum _{i,j,i^{\prime },j^{\prime }}{P}_{ij{i}^{\prime }{j}^{\prime }}^{\nu }{A}_{ij{i}^{\prime }{j}^{\prime }}.$$

2. Main results

Our first main result, proved in Section 5, gives the asymptotic properties of the largest (spike) eigenvalues of the sample correlation matrix:

Theorem 1

Assume Model M, and that ${\ell }_{\nu }>1+\sqrt{\gamma }$ is a simple eigenvalue. As p/n → γ > 0,

$$\begin{gathered} (i){\widehat{\ell }}_{\nu }\stackrel{\text{a.s.}}{\to }{\rho }_{\nu }, \\ (ii)\sqrt{n}({\widehat{\ell }}_{\nu }-{\rho }_{\nu n})\stackrel{\mathcal{D}}{\to }N\left(0,{\tilde{\sigma }}_{\nu }^2\right), \end{gathered}$$ (2.5)

where

$${\tilde{\sigma }}_{\nu }^2=2{\dot{\rho }}_{\nu }{\ell }_{\nu }^2+{\dot{\rho }}_{\nu }^2\left[{\mathcal{P}}^{\nu },\kappa \right]+{\dot{\rho }}_{\nu }^2\left[{\mathcal{P}}^{\nu },\check{\kappa }\right].$$ (2.6)

Centering at ρ_νn rather than at ρ_ν is important. If, for example, γ_n = γ + an^−1/2, then

$$\sqrt{n}({\widehat{\ell }}_{\nu }-{\rho }_{\nu })\stackrel{\mathcal{D}}{\to }N(a{\ell }_{\nu }{\left({\ell }_{\nu }-1\right)}^{-1},{\tilde{\sigma }}_{\nu }^2),$$

and we see a limiting shift. Furthermore, it may also be beneficial to consider ${\tilde{\sigma }}_{\nu n}^2$ instead of ${\tilde{\sigma }}_{\nu }^2$, obtained by replacing ${\dot{\rho }}_{\nu }$ with ${\dot{\rho }}_{\nu n}$ in (2.6), such that

$$\sqrt{n}({\widehat{\ell }}_{\nu }-{\rho }_{\nu n})/{\tilde{\sigma }}_{\nu n}\stackrel{\mathcal{D}}{\to }N(0,1).$$

The asymptotic first-order limit in (i), which follows as an easy consequence of El Karoui (2009), coincides with that of the νth largest eigenvalue of a sample covariance matrix computed from data with population covariance Γ (Paul, 2007). This implies that, when constructing R, normalizing by the sample variances has no effect on the leading eigenvalues, at least to first order.

However, key differences are seen when looking at the asymptotic distribution, given in (ii), and in the variance formula (2.6) in particular. This can be readily interpreted. The first term corresponds to the variance in the Gaussian-covariance case of Paul (2007), again for samples with covariance Γ. The second provides a correction of that result for non-Gaussian data, see the companion article [JY]. The third term describes the contribution specific to sample correlation matrices, representing the effect of normalizing the data by the sample variances. This term is often negative, and is evaluated explicitly for Gaussian data in Corollary 1 below, proved in the Supplementary Material, S1.1.

Corollary 1

For ξ Gaussian, the asymptotic variance in Theorem 1 simplifies to

$${\tilde{\sigma }}_{\nu }^2=2{\ell }_{\nu }^2{\dot{\rho }}_{\nu }\left[1-{\dot{\rho }}_{\nu }\left(2{\ell }_{\nu }\mathrm{tr}{P}_{D,\nu }^4-\mathrm{tr}{\left(P_{D,\nu }\mathrm{\Gamma }{P}_{D,\nu }\right)}^2\right)\right],$$

where P_D,ν = diag(p_ν,1, …, p_ν,m).

Thus, computing the sample correlation results in the asymptotic variance being scaled by $1-{\dot{\rho }}_{\nu }{\mathrm{\Delta }}_{\nu }$, relative to the sample covariance, where

$${\mathrm{\Delta }}_{\nu }=2{\ell }_{\nu }\mathrm{tr}{P}_{D,\nu }^4-\mathrm{tr}{\left(P_{D,\nu }\mathrm{\Gamma }{P}_{D,\nu }\right)}^2=2{\ell }_{\nu }\sum _i{p}_{\nu ,i}^4-\sum _{i,j}{\left(p_{\nu ,i}{\kappa }_{ij}{p}_{\nu ,j}\right)}^2$$

is often positive, implying that spiked eigenvalues of the sample correlation often exhibit a smaller variance than those of the sample covariance. Indeed, such variance reduction occurs iff

$$\sum _{i,j}{\left(p_{\nu ,i}{\kappa }_{ij}{p}_{\nu ,j}\right)}^2<2{\ell }_{\nu }\sum _i{p}_{\nu ,i}^4=\sum _{i,j}{p}_{\nu ,i}{\kappa }_{ij}{p}_{\nu ,j}(p_{\nu ,i}^2+p_{\nu ,j}^2),$$ (2.7)

with the last identity following from the fact that ${\ell }_{\nu }{p}_{\nu ,i}={\sum }_j{\kappa }_{ij}{p}_{\nu ,j}$. Condition (2.7), and variance reduction, holds in the following cases:

1. both Γ and p_ν have nonnegative entries, or

2. $2{\ell }_{\nu }{\sum }_i{p}_{\nu ,i}^4>1$, or

3. $2{\ell }_{\nu }>{\ell }_1^2.$

In case (i), the inequalities $0\le p_{\nu ,i}{\kappa }_{ij}{p}_{\nu ,j}\le 2p_{\nu ,i}{p}_{\nu ,j}\le p_{\nu ,i}^2+p_{\nu ,j}^2$ yield (2.7). Note that if Γ has nonnegative entries, then the Perron–Frobenius theorem establishes the existence of an eigenvector with nonnegative components for ℓ₁; furthermore, if Γ has positive entries, by the same theorem, ℓ₁ is simple and associated with an eigenvector with positive components. Case (ii) follows from ${\sum }_{i,j}{\left(p_{\nu ,i}{\kappa }_{ij}{p}_{\nu ,j}\right)}^2\le {\sum }_{i,j}{\left(p_{\nu ,i}{p}_{\nu ,j}\right)}^2=1$, and holds if ℓ_ν > m/2, because ${\sum }_i{p}_{\nu ,i}^4\ge 1/m$. Case (iii) follows from the inequalities $2p_{\nu ,i}^2{p}_{\nu ,j}^2\le p_{\nu ,i}^4+p_{\nu ,j}^4$ and ${\sum }_j{\kappa }_{ij}^2={\left({\mathrm{\Gamma }}^2\right)}_{ii}\le \Vert {\mathrm{\Gamma }}^2\Vert ={\ell }_1^2$. Note that this is rather special, in that it has nothing to do with eigenvectors, and a necessary condition for it to hold is ℓ₁ ≤ 2.

Condition (2.7) can fail, however. For example, for even m and r ∈ (0, 1), consider

$$\mathrm{\Gamma }=\left(\begin{array}{cc}1& -r\\ -r& 1\end{array}\right)\otimes 1_{m/2}{1}_{m/2}^T,$$

where 1_m/2 is the (m/2)-dimensional vector of all ones, which corresponds to two negatively correlated groups of identical random vectors. This has simple supercritical eigenvalues ℓ₁ = (1 + r)m/2 and ℓ₂ = (1 − r)m/2 when $m>2(1+\sqrt{\gamma })/(1-r)$, with $p_{\nu ,i}^2=m^{-1}$ for ν = 1, 2. One finds that Δ₂ = (1 − 2r − r²)/2 < 0 for $r>\sqrt{2}-1$, although Δ₁ > 0 because ℓ₁ > m/2, which implies case (ii).

We turn now to the eigenvectors. Again, fix an index ν for which ${\ell }_{\nu }>1+\sqrt{\gamma }$ is a simple eigenvalue of Γ, with corresponding eigenvector ${\mathfrak{p}}_{\nu }={\left[p_{\nu }^T{0}_p^T\right]}^T$. Recall that ${\widehat{\mathfrak{p}}}_{\nu }={\left[{\widehat{p}}_{\nu }^T{\widehat{v}}_{\nu }^T\right]}^T$ is the νth sample eigenvector of R, and let $a_{\nu }={\widehat{p}}_{\nu }/\Vert {\widehat{p}}_{\nu }\Vert$ be the corresponding normalized subvector of ${\widehat{\mathfrak{p}}}_{\nu }$, restricted to the first m coordinates. The next result establishes a limit for the eigenvector projection $\langle {\widehat{\mathfrak{p}}}_{\nu },{\mathfrak{p}}_{\nu }\rangle$, and a CLT for the normalized cross-projections $P^T{a}_{\nu }={\left[p_1^T{a}_{\nu },\dots ,p_m^T{a}_{\nu }\right]}^T$; see Sections 6.1 and 6.2.

Theorem 2

Assume Model M, and that ${\ell }_{\nu }>1+\sqrt{\gamma }$ is a simple eigenvalue. Then, as p/n → γ > 0,

$$\begin{gathered} (i){\langle {\widehat{\mathfrak{p}}}_{\nu },{\mathfrak{p}}_{\nu }\rangle }^2\stackrel{\text{a.s.}}{\to }{\dot{\rho }}_{\nu }{\ell }_{\nu }/{\rho }_{\nu }, \\ (ii)\sqrt{n}\left(P^T{a}_{\nu }-e_{\nu }\right)\stackrel{\mathcal{D}}{\to }N\left(0,{\mathrm{\Sigma }}_{\nu }\right), \end{gathered}$$

where ${\mathrm{\Sigma }}_{\nu }={\mathcal{D}}_{\nu }{\tilde{\mathrm{\Sigma }}}_{\nu }{\mathcal{D}}_{\nu }$ with

$${\mathcal{D}}_{\nu }=\sum _{k\ne \nu }^m{\left({\ell }_{\nu }-{\ell }_k\right)}^{-1}{e}_k{e}_k^T$$ (2.8)

$${\tilde{\mathrm{\Sigma }}}_{\nu ,kl}={\dot{\rho }}_{\nu }^{-1}{\ell }_k{\ell }_{\nu }{\delta }_{k,l}+\left[{\mathcal{P}}^{k\nu l\nu },\kappa \right]+\left[{\mathcal{P}}^{k\nu l\nu },\check{\kappa }\right],$$ (2.9)

where δ_k,l = 1 if k = l, and zero otherwise.

The CLT result in (ii) can be rephrased in terms of the entries of a_ν, for which we readily obtain $\sqrt{n}\left(a_{\nu }-p_{\nu }\right)\stackrel{\mathcal{D}}{\to }N\left(0,P{\mathrm{\Sigma }}_{\nu }{P}^T\right)$; note that Σ_ν has zeros in the νth row and the νth column.

As for the eigenvalues, Theorem 2 shows that the spiked eigenvectors of sample correlation matrices exhibit the same first-order behavior as those of the sample covariance (Paul, 2007). The difference again lies in the asymptotic fluctuations, captured by the covariance matrix Σ_ν. Note that this is decomposed as a product of ${\mathcal{D}}_{\nu }-\text{a}$ diagonal matrix—and the matrix ${\tilde{\mathrm{\Sigma }}}_{\nu }$, which involves the three terms in (2.9). These terms have similar interpretations as those discussed previously in (2.6). That is, the first term captures the asymptotic fluctuations for a Gaussian-covariance model (Paul, 2007), the second term captures the effect of non-Gaussianity in the covariance case [JY], and the third term captures information specific to the correlation case, representing fluctuations due to sample variance normalization. Note that only the first term is diagonal in general, suggesting that the eigenvector projections may be asymptotically correlated, as seen earlier in Figure 1(b), right panel. This holds also for Gaussian data, evaluated explicitly in Corollary 2 below; see Supplementary Material, S1.2, for the proof. We note an interesting contrast with the eigenvector projections for covariance matrices (Paul, 2007), described only by the leading term in (2.9).

Corollary 2

For ξ Gaussian, the asymptotic covariance in Theorem 2 reduces to ${\mathrm{\Sigma }}_{\nu }={\mathcal{D}}_{\nu }{\tilde{\mathrm{\Sigma }}}_{\nu }{\mathcal{D}}_{\nu }$,

$${\tilde{\mathrm{\Sigma }}}_{\nu }=\frac{{\ell }_{\nu }}{{\dot{\rho }}_{\nu }}L+\left({\ell }_{\nu }I+L\right)\left(\frac{1}{2}\mathcal{Z}-{\ell }_{\nu }\mathcal{Y}\right)\left({\ell }_{\nu }I+L\right)+{\ell }_{\nu }\left({\ell }_{\nu }^2\mathcal{Y}-L\mathcal{Y}L\right),$$

where $\mathcal{Z}=P^T{P}_{D,\nu }(\mathrm{\Gamma }\circ \mathrm{\Gamma })P_{D,\nu }P$, $\mathcal{Y}=P^T{P}_{D,\nu }^{2}P$, and ∘ denotes the Hadamard product.

Thus, for Gaussian data, the entries of the asymptotic covariance matrix are given by (for k, l ≠ ν)

$${\mathrm{\Sigma }}_{\nu ,kl}={\left({\ell }_{\nu }-{\ell }_k\right)}^{-1}{\left({\ell }_{\nu }-{\ell }_l\right)}^{-1}\left[\frac{{\ell }_{\nu }}{{\dot{\rho }}_{\nu }}{\ell }_k{\delta }_{k,l}+\left({\ell }_{\nu }+{\ell }_k\right)\left({\ell }_{\nu }+{\ell }_l\right)\frac{{\mathcal{Z}}_{kl}}{2}-{\ell }_{\nu }\left({\ell }_{\nu }\left({\ell }_k+{\ell }_l\right)+2{\ell }_k{\ell }_l\right){\mathcal{Y}}_{kl}\right].$$

Consider now the subcritical case in which ν is such that $1<{\ell }_{\nu }\le 1+\sqrt{\gamma }$. Let ${\mathfrak{p}}_{\nu }$ denote the corresponding population eigenvector, and let ${\widehat{\ell }}_{\nu }$ and ${\widehat{\mathfrak{p}}}_{\nu }$ denote the corresponding sample eigenvalue and eigenvector, respectively. With proofs deferred to Sections 5.1 and 6.3, we have the following result:

Theorem 3

Assume Model M, and that $1<{\ell }_{\nu }\le 1+\sqrt{\gamma }$ is a simple eigenvalue. Then, as p/n → γ > 0,

$$\begin{gathered} (i){\widehat{\ell }}_{\nu }\stackrel{\text{a.s.}}{\to }{(1+\sqrt{\gamma })}^2, \\ (ii){\langle {\widehat{\mathfrak{p}}}_{\nu },{\mathfrak{p}}_{\nu }\rangle }^2\stackrel{\text{a.s.}}{\to }0. \end{gathered}$$

Once again, the asymptotic first-order limits of the sample eigenvalue and its associated eigenvector are the same as those obtained for the sample covariance (Paul, 2007).

Recall that our high-dimensional results assume an asymptotic regime where p/n → γ > 0, as opposed to the classical regime where p is fixed and n → ∞. The case of fixed p corresponds to γ = 0 and the spectral properties of the sample correlation matrix are well understood; see, for example, Girshick (1939), Konishi (1979), Fang and Krishnaiah (1982), Schott (1991), Kollo and Neudecker (1993), and Boik (2003). When γ = 0, the function ρ(ℓ) reduces to the identity. Indeed, for fixed p, there is no high-dimensional component η in Model M, and hence no biasing effect on ρ(ℓ, γ) that occurs when γ > 0. In particular, for fixed p there is no counterpart to our Theorem 3.

To summarize, in comparison to the high-dimensional (p/n → γ > 0) sample covariance setting, our results for the spiked eigenvalues and eigenvectors of sample correlation matrices confirm that the first-order asymptotic behavior is indeed equivalent to that of sample covariance matrices, in agreement with previous results and observations (El Karoui, 2009, Mestre and Vallet, 2017). While the eigenvalue limits in Theorem 1 and Theorem 3 follow as a straightforward consequence of El Karoui (2009), the eigenvector results of Theorem 2-(i) and Theorem 3-(ii) do not. In contrast to the first-order equivalences, important differences arise in the fluctuations of both the eigenvalues and eigenvectors, as shown by the asymptotic distributions of Theorem 1-(ii) and Theorem 2-(ii).

We illustrate these differences with a simple example having covariance $\mathrm{\Gamma }=(1-r)I_m+r{1}_m{1}_m^T$, where r ∈ [0, 1]; that is, a model with unit variances and constant correlation r across all components. Moreover, ξ is assumed to be Gaussian for simplicity. In this setting, L = diag(ℓ₁, 1 − r, …, 1 − r), where ℓ₁ = 1 + r(m − 1) is supercritical iff $r>\sqrt{\gamma }/(m-1)$. Consider the largest sample eigenvalue ${\widehat{\ell }}_1$ in such a supercritical case. From Corollary 1, the asymptotic variances for the sample covariance and the sample correlation can be computed, yielding

$${\sigma }_1^2=2{\ell }_1^2{\dot{\rho }}_1,{\tilde{\sigma }}_1^2={\sigma }_1^2\left(1-{\dot{\rho }}_1\mathrm{\Delta }\right),$$

respectively, with $\mathrm{\Delta }=2{\ell }_1\mathrm{tr}{P}_D^4-\mathrm{tr}{\left(P_D\mathrm{\Gamma }{P}_D\right)}^2$, and where

$$P_D\triangleq P_{D,1}=m^{-1/2}{I}_m,{\dot{\rho }}_1=1-\frac{\gamma }{r^2{(m-1)}^2}.$$

Figure 2(a) plots these asymptotic variances versus r for various (γ,m). Indeed, the variance (fluctuation) for the sample correlation is consistently smaller than for the sample covariance. The difference is striking, becoming extremely large as r ↗ 1. Similar trends are observed for various choices of m and γ, being more pronounced for higher m, while not much affected by varying γ. This may be understood from the fact that, after writing Δ = r(2 − r) + (1 − r)²m⁻¹ = 1 − (1 − r)²(1 − m⁻¹),

$$\frac{{\tilde{\sigma }}_1^2}{{\sigma }_1^2}=1-{\dot{\rho }}_1\mathrm{\Delta }\to \{\begin{array}{lll}\frac{\gamma }{{(m-1)}^2}& \text{as}& r\to 1,m\text{fixed}\\ {(1-r)}^2& \text{as}& m\to \infty ,r\text{fixed.}\end{array}$$

Turn now to the fluctuations of the leading sample eigenvector, in the same setting as above. Note that, in Corollary 2, for this particular case, one can deduce from P^TΓP = L that

$$\mathcal{Z}=m^{-1}\left(1-r^2\right)I_m+r^2{e}_1{e}_1^T,\mathcal{Y}=m^{-1}{I}_m.$$

Also from Corollary 2, the asymptotic variances for the normalized sample-to-population eigenvector projection $p_2^T{a}_1$, in the sample covariance and sample correlation cases, are computed as

$${\mathrm{\Sigma }}_{1,22}^{\text{cov}}=\frac{{\ell }_1{\ell }_2}{{(rm)}^2{\dot{\rho }}_1},{\mathrm{\Sigma }}_{1,22}={\mathrm{\Sigma }}_{1,22}^{\text{cov}}-\frac{\zeta }{{(\text{rm})}^2}\frac{{\ell }_1{\ell }_2\left({\ell }_1+{\ell }_2\right)}{m},$$

respectively, where $\zeta =1-r+\frac{1}{2}(1+r){\left(1+\frac{1-r}{rm}\right)}^{-1}$, and we recall that ℓ₁ = 1 − r + rm and ℓ₂ = 1 − r. These variances are numerically evaluated in Figure 2(b) for the same parameter choices as before and, again, as functions of r. Note, however, that for better visual appreciation, the range of r has been restricted to supercritical values sufficiently above the critical point $\sqrt{\gamma }/(m-1)$, because the variance explodes at that point. The comparative evaluation again shows smaller variances for the sample correlation. The variance reduction here is less visible in the graphs, because both Σ_1,22 and ${\mathrm{\Sigma }}_{1,22}^{\text{cov}}$ vanish as r → 1. The ratio, however, behaves quite similarly to the variance ratio ${\tilde{\sigma }}_1^2/{\sigma }_1^2$:

$$\frac{{\mathrm{\Sigma }}_{1,22}}{{\mathrm{\Sigma }}_{1,22}^{\text{cov}}}=1-{\zeta }_{{\dot{\rho }}_1}\frac{\left({\ell }_1+{\ell }_2\right)}{m}\to \{\begin{array}{ll}\frac{\gamma }{{(m-1)}^2}& \text{as}r\to 1,m\text{fixed}\\ (1-r)(1-r/2)& \text{as}m\to \infty ,r\text{fixed.}\end{array}$$

Figure 2:

Differences in the fluctuations of sample eigenvalues and eigenvectors for an example Gaussian model with $\mathrm{\Gamma }=(1-r)I_m+r{1}_m{1}_m^T$. Asympotic variances are shown for (a) the largest sample eigenvalue ${\widehat{\ell }}_1$, and (b) the normalized sample-to-population eigenvector projection $p_2^T{a}_1$.

We end the discussion of our main results with a few remarks about possible extensions. Our results assume that ℓ_ν > 1 is a simple eigenvalue, but extensions for small spikes with ℓ_ν < 1 and for spikes with multiplicities should be possible. Analogous results for eigenvalues have been obtained for sample covariance matrices for ℓ_ν < 1, including multiplicities greater than one (e.g., see Bai and Yao (2008)), giving reason to expect corresponding results for correlation matrices. Extensions of our results for eigenvalues and eigenvectors of sample correlation matrices for simple ℓ_ν < 1 should be fairly straightforward, though the cases γ < 1, γ = 1, and γ > 1 would need separate treatment. Extensions for spikes with multiplicities are also possible, but in this case the eigenvectors are not well defined and one would need to consider subspace projections, requiring non-trivial modifications of our technical arguments.

The remainder of the paper proceeds as follows. First, in Section 3, we introduce key quantities and identities used in the derivations. Section 4 presents necessary asymptotic properties for bilinear forms and matrix quadratic forms with normalized entries, with the corresponding proofs relegated to the Supplementary Material, Section S3. These properties provide a foundation for describing the asymptotic convergence and distribution of eigenvalues and eigenvectors of sample correlation matrices, derived in Sections 5 and 6 respectively.

As already noted, a parallel treatment for the simpler case of covariance matrices is given in a supplementary manuscript [JY]. This aims at a unified exposition of known spectral properties of spiked covariance matrices as a benchmark for the current work, along with additional citations to the literature.

3. Preliminaries

We begin with a block representation and some associated reductions for the sample correlation matrix R. These are well known in the covariance matrix setting. As with the partition of x in Model M, consider

$$X=\left[\begin{array}{l}{X}_1\\ X_2\end{array}\right],X_1\in {ℝ}^{m\times n},X_2\in {ℝ}^{p\times n}.$$

Write S_D = blkdiag(S_D1, S_D2), with S_D1 containing the sample variances corresponding to ξ, and S_D2 containing those corresponding to η. Define the “normalized” data matrices ${\overline{X}}_1=S_{D1}^{-1/2}{X}_1$ and ${\overline{X}}_2=S_{D2}^{-1/2}{X}_2$, such that

$$R=n^{-1}\left[\begin{array}{cc}{\overline{X}}_1{\overline{X}}_1^T& {\overline{X}}_1{\overline{X}}_2^T\\ {\overline{X}}_2{\overline{X}}_1^T& {\overline{X}}_2{\overline{X}}_2^T\end{array}\right]=\left[\begin{array}{ll}{R}_{11}& R_{12}\\ R_{21}& R_{22}\end{array}\right];{\widehat{\mathfrak{p}}}_{\nu }=\left[\begin{array}{l}{\widehat{p}}_{\nu }\\ {\widehat{v}}_{\nu }\end{array}\right].$$

This partitioning of the eigenvector equation $R{\widehat{\mathfrak{p}}}_{\nu }={\widehat{\ell }}_{\nu }{\widehat{\mathfrak{p}}}_{\nu }$, along with ${\widehat{\mathfrak{p}}}_{\nu }={\left[{\widehat{p}}_{\nu }^T,{\widehat{v}}_{\nu }^T\right]}^T$, yields

$$R_{11}{\widehat{p}}_{\nu }+R_{12}{\widehat{v}}_{\nu }={\widehat{\ell }}_{\nu }{\widehat{p}}_v$$

$$R_{21}{\widehat{p}}_{\nu }+R_{22}{\widehat{v}}_{\nu }={\widehat{\ell }}_{\nu }{\widehat{v}}_{\nu }.$$

From the second equation, ${\widehat{v}}_{\nu }={\left({\widehat{\ell }}_{\nu }{I}_p-R_{22}\right)}^{-1}{R}_{21}{\widehat{p}}_{\nu }$. Substituting this into the first equation yields

$$K\left({\widehat{\ell }}_{\nu }\right){\widehat{p}}_{\nu }={\widehat{\ell }}_{\nu }{\widehat{p}}_{\nu },\text{with}K(t)=R_{11}+R_{12}{\left(t{I}_p-R_{22}\right)}^{-1}{R}_{21}.$$

Thus, ${\widehat{\ell }}_v$ is an eigenvalue of $K\left({\widehat{\ell }}_{\nu }\right)$, with associated eigenvector ${\widehat{p}}_{\nu }$; this is central to our derivations. Note that $K\left({\widehat{\ell }}_{\nu }\right)$ is well defined if ${\widehat{\ell }}_v$ is well separated from the eigenvalues of R₂₂; Section 5.1 shows that this occurs with probability one for all large n when ℓ_ν is supercritical. Furthermore, the normalization condition, ${\widehat{p}}_{\nu }^T{\widehat{p}}_{\nu }+{\widehat{v}}_{\nu }^T{\widehat{v}}_{\nu }=1$ yields

$${\widehat{p}}_{\nu }^T\left(I_m+Q_{\nu }\right){\widehat{p}}_{\nu }=1,Q_{\nu }=R_{12}{\left({\widehat{\ell }}_{\nu }{I}_p-R_{22}\right)}^{-2}{R}_{21}.$$

Phrased in terms of the signal-space normalized eigenvector $a_{\nu }={\widehat{p}}_{\nu }/\Vert {\widehat{p}}_{\nu }\Vert$, we have

$$K\left({\widehat{\ell }}_{\nu }\right)a_{\nu }={\widehat{\ell }}_{\nu }{a}_{\nu },a_{\nu }^T\left(I_m+Q_{\nu }\right)a_{\nu }={\Vert {\widehat{p}}_{\nu }\Vert }^{-2}.$$ (3.10)

Note also that the sample-to-population inner product can be rewritten as

$$\langle {\widehat{\mathfrak{p}}}_{\nu },{\mathfrak{p}}_{\nu }\rangle =\langle {\widehat{p}}_{\nu },p_{\nu }\rangle =\Vert {\widehat{p}}_{\nu }\Vert \langle a_{\nu },p_{\nu }\rangle .$$ (3.11)

In the derivation of our CLT results, we use an eigenvector perturbation formula with quadratic error bound given in [JY, Lemma 13], itself a modification of the arguments in Paul (2007). This yields the key expansion

$$a_{\nu }-p_{\nu }=-{\mathcal{R}}_{\nu n}{D}_{\nu }{p}_{\nu }+r_{\nu },$$ (3.12)

where

$${\mathcal{R}}_{\nu n}=\frac{{\ell }_{\nu }}{{\rho }_{\nu n}}\sum _{k\ne \nu }^m{\left({\ell }_k-{\ell }_{\nu }\right)}^{-1}{p}_k{p}_k^T,D_{\nu }=K\left({\widehat{\ell }}_{\nu }\right)-\left({\rho }_{\nu n}/{\ell }_{\nu }\right)\mathrm{\Gamma },\Vert r_{\nu }\Vert =O\left({\Vert D_{\nu }\Vert }^2\right).$$

The derivations of our eigenvalue and eigenvector results, presented in Sections 5 and 6 respectively, take (3.10), (3.11) and (3.12) as points of departure, and rely on asymptotic properties of the key objects $K\left({\widehat{\ell }}_{\nu }\right)$ and Q_ν. In particular, K(t) can be expressed as the random matrix quadratic form

$$K(t)=n^{-1}{\overline{X}}_1{B}_n(t){\overline{X}}_1^T,$$ (3.13)

where, using the Woodbury identity,

$$\begin{gathered} B_n(t)=I_n+n^{-1}{\overline{X}}_2^T{\left(t{I}_p-R_{22}\right)}^{-1}{\overline{X}}_2 \\ =t{\left(t{I}_n-n^{-1}{\overline{X}}_2^T{\overline{X}}_2\right)}^{-1}. \end{gathered}$$

Thus, our key objects are random quadratic forms involving the normalized data matrices ${\overline{X}}_1$ and ${\overline{X}}_2$. The asymptotic properties of these forms are foundational to our results, and are presented next.

4. Quadratic forms with normalized entries

In this section, we establish the first-order (deterministic) convergence and a CLT for matrix quadratic forms of the type $n^{-1}{\overline{X}}_1{B}_n{\overline{X}}_1^T$, where B_n is a matrix with bounded spectral norm. While being essential to our purposes, some of the technical results may be of independent interest; thus, we first present the general results, and then apply these in the context of Model M.

4.1. First-order convergence

To establish the first-order convergence, we first require some results on bilinear forms involving correlated random vectors of unit length. A main technical result (see Supplementary Material, S3.1) is the following:

Lemma 1

Let B be an n × n nonrandom symmetric matrix, and let $x,y\in {ℝ}^n$ be random vectors of i.i.d. entries with mean zero, variance one, $\mathbb{E}{\left|x_i\right|}^l$, $\mathbb{E}{\left|y_i\right|}^l\le {\nu }_l$, and $\mathbb{E}\left[x_i{y}_i\right]=\rho$. Let $\overline{x}=\sqrt{n}x/\Vert x\Vert$ and $\overline{y}=\sqrt{n}y/\Vert y\Vert$. Then, for any s ≥ 1,

$$\mathbb{E}{\left|n^{-1}{\overline{x}}^{T}B\overline{y}-\rho n^{-1}\mathrm{tr}B\right|}^s\le {\mathcal{C}}_s\left[n^{-s}\left({\nu }_{2s}\mathrm{tr}{B}^s+{\left({\nu }_4\mathrm{tr}{B}^2\right)}^{s/2}\right)+\Vert B{\Vert }^s\left(n^{-s/2}{\nu }_4^{s/2}+n^{-s+1}{\nu }_{2s}\right)\right],$$

where ${\mathcal{C}}_s$ is a constant depending only on s.

This is a generalization of Gao et al. (2017, Lemma 5), which established a corresponding bound for normalized quadratic forms. Lemma 1 leads to the following first-order convergence result:

Corollary 3

Let x,y be random vectors of i.i.d. entries with mean zero, variance one, $\mathbb{E}{\left|x_i\right|}^{4+\delta }$, $\mathbb{E}{\left|y_i\right|}^{4+\delta }<\infty$ for some δ > 0, and $\mathbb{E}\left[x_i{y}_i\right]=\rho$. Define $\overline{x}=\sqrt{n}x/\Vert x\Vert$ and $\overline{y}=\sqrt{n}y/\Vert y\Vert$, and let B_n be a sequence of n × n symmetric matrices, with ‖B_n‖ bounded. Then,

$$n^{-1}{\overline{x}}^T{B}_n\overline{y}-n^{-1}\rho \mathrm{tr}{B}_n\stackrel{\text{a.s.}}{\to }0.$$

Proof. Because the (4 + δ)th moment and ‖B_n‖ are bounded, from Lemma 1,

$$\mathbb{E}{\left|n^{-1}{\overline{x}}^T{B}_n\overline{y}-n^{-1}\rho \mathrm{tr}{B}_n\right|}^{2+\delta /2}\le O\left(n^{-(1+\delta /4)}\right).$$

The convergence then follows from Markov’s inequality and the Borel–Cantelli lemma. ☐

We now apply this to our Model M with random matrices $B_n\left({\overline{X}}_2\right)$, independent of ${\overline{X}}_1$:

Lemma 2

Assume Model M, and suppose that $B_n=B_n\left({\overline{X}}_2\right)$ is a sequence of random symmetric matrices, for which ‖B_n‖ is O_a.s.(1). Then,

$$n^{-1}{\overline{X}}_1{B}_n\left({\overline{X}}_2\right){\overline{X}}_1^T-n^{-1}\mathrm{tr}{B}_n\left({\overline{X}}_2\right)\mathrm{\Gamma }\stackrel{\text{a}.\text{s}.}{\to }0.$$

Proof. This follows from Fubini’s theorem. Specifically, one may use the arguments in the proof of [JY, Lemma 5], applying Corollary 3, and noting that ${\overline{X}}_1$ is independent of $B_n\left({\overline{X}}_2\right)$. ☐

4.2. Central Limit Theorem

To establish our main matrix quadratic-form CLT result, we first derive a CLT for scalar bilinear forms involving normalized random vectors. To this end, we must introduce some further notation. Consider zero-mean random vectors $(x,y)\in {ℝ}^M\times {ℝ}^M$, with

$$\mathrm{Cov}\left(\begin{array}{l}x\\ y\end{array}\right)=C=\left(\begin{array}{ll}{C}^{xx}& C^{xy}\\ C^{yx}& C^{yy}\end{array}\right),$$

where $C_{l{l}^{\prime }}^{xy}=\mathbb{E}\left[x_l{y}_{l^{\prime }}\right]$. Assume $C_{ll}^{xx}=C_{ll}^{yy}=1$; that is, all components of the x and y vectors have unit variance and ${\rho }_l=C_{ll}^{xy}=\mathbb{E}\left[x_l{y}_l\right]$. We first introduce notation for some quadratic functions of x_l, y_l. Let $z,w\in {ℝ}^M$, with

$$z_l=x_l{y}_l,w_l={\rho }_l\left(x_l^2+y_l^2\right)/2,C^{zz}=\mathrm{Cov}(z),C^{wz}=\mathrm{Cov}(z,w),etc.$$

Let X = (x_li)_M×n and Y = (y_li)_M×n be data matrices based on n i.i.d. observations of (x, y), and define the “normalized” data matrices $\overline{X}={\widehat{\mathrm{\Sigma }}}_x^{-1/2}X$ and $\overline{Y}={\widehat{\mathrm{\Sigma }}}_y^{-1/2}Y$, where ${\widehat{\mathrm{\Sigma }}}_x=\mathrm{diag}\left({\widehat{\sigma }}_{x_1}^2,\dots ,{\widehat{\sigma }}_{x_M}^2\right)$, ${\widehat{\mathrm{\Sigma }}}_y=\mathrm{diag}\left({\widehat{\sigma }}_{y_1}^2,\dots ,{\widehat{\sigma }}_{y_M}^2\right)$, and ${\widehat{\sigma }}_{x_l}^2=n^{-1}{\sum }_{i=1}^n{x}_{li}^2$,${\widehat{\sigma }}_{y_l}^2=n^{-1}{\sum }_{i=1}^n{y}_{li}^2$. Then, we use the following notation for the rows ${\overline{x}}_l^T$ and ${\overline{y}}_l^T$ of the normalized data matrices

$$\overline{X}={\left({\overline{x}}_{li}\right)}_{M\times n}=\left[\begin{array}{c}{\overline{x}}_{1.}^T\\ ⋮\\ {\overline{x}}_{M\cdot }^T\end{array}\right],\overline{Y}={\left({\overline{y}}_{li}\right)}_{M\times n}=\left[\begin{array}{c}{\overline{y}}_1^T\\ ⋮\\ {\overline{y}}_M^T\end{array}\right].$$

With this setup, we have the following result, proved in the Supplementary Material, S3.2:

Proposition 1

Let B_n = (b_n,ij) be random symmetric n × n matrices, independent of X, Y, such that for some finite β, ‖B_n‖ ≤ β for all n, and

$$n^{-1}\sum _{i=1}^n{b}_{n,ii}^2\stackrel{p}{\to }\omega ,n^{-1}\mathrm{tr}{B}_n^2\stackrel{p}{\to }\theta ,{\left(n^{-1}\mathrm{tr}{B}_n\right)}^2\stackrel{p}{\to }\varphi ,$$

all finite. In addition, define $Z_n\in {ℝ}^M$, with components

$$Z_{n,l}=n^{-1/2}\left[{\overline{x}}_{l.}^T{B}_n{\overline{y}}_{l.}-{\rho }_l\mathrm{tr}{B}_n\right].$$

Then, $Z_n\stackrel{\mathcal{D}}{\to }{N}_M(0,D)$, with

$$D=(\theta -\omega )J+\omega K_1+\varphi K_2=\theta J+\omega K+\varphi K_2,$$ (4.14)

where K = K₁ − J and J,K₁,K₂ are matrices defined by

$$\begin{gathered} J=C^{xy}\circ C^{yx}+C^{xx}\circ C^{yy} \\ {K}_1=C^{zz} \\ {K}_2=C^{ww}-C^{wz}-C^{zw}. \end{gathered}$$ (4.15)

The entries of K are fourth-order cumulants of x and y:

$$K_{l{l}^{\prime }}=\mathbb{E}\left(x_l{y}_l{x}_{l^{\prime }}{y}_{l^{\prime }}\right)-\mathbb{E}\left(x_l{y}_l\right)\mathbb{E}\left(x_{l^{\prime }}{y}_{l^{\prime }}\right)-\mathbb{E}\left(x_l{y}_{l^{\prime }}\right)\mathbb{E}\left(y_l{x}_{l^{\prime }}\right)-\mathbb{E}\left(x_l{x}_{l^{\prime }}\right)\mathbb{E}\left(y_l{y}_{l^{\prime }}\right).$$ (4.16)

Hence, K vanishes if x, y are Gaussian.

The corresponding result with unnormalized vectors is established in [JY Theorem 10]. The terms θJ + ωK appear in that case, and the additional term ϕK₂ reflects the normalization in ${\overline{x}}_{l.}$ and ${\overline{y}}_{l.}$. As in [JY], the proof is based on the martingale CLT, rather than the moment method used in Bai and Yao (2008), which stated a similar result for quadratic forms involving unnormalized random vectors.

While potentially of independent interest, Proposition 1 is important for our purposes through its application to Model M.

Proposition 2

Assume Model M, and consider B_n as in Proposition 1. Then,

$$W_n=n^{-1/2}\left[{\overline{X}}_1{B}_n{\overline{X}}_1^T-\left(\mathrm{tr}{B}_n\right)\mathrm{\Gamma }\right]\stackrel{\mathcal{D}}{\to }W,$$

where W is a symmetric m × m Gaussian matrix with entries W_ij, mean zero, and covariances given by

$$\mathrm{Cov}\left[W_{ij},W_{i^{\prime }{j}^{\prime }}\right]=\theta \left({\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}+{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}\right)+\omega {\kappa }_{ij{i}^{\prime }{j}^{\prime }}+\varphi {\check{\kappa }}_{ij{i}^{\prime }{j}^{\prime }},$$ (4.17)

for i ≤ j and i′ ≤ j′.

Proof. The result follows from Proposition 1 by turning the matrix quadratic form ${\overline{X}}_1{B}_n{\overline{X}}_1^T$ into a vector of bilinear forms; see, for example, [JY, Proposition 6] and Bai and Yao (2008, Proposition 3.1). Specifically, use an index l for the M = m(m + 1)/2 pairs (i, j), with 1 ≤ i ≤ j ≤ m. Build the random vectors (x, y) for Proposition 1 as follows: if l = (i, j), then set x_l = ξ_i/σ_i and y_l = ξ_j/σ_j. In the resulting covariance matrix C for (x, y), if also l′ = (i′, j′),

$$C_{l{l}^{\prime }}^{xy}=\mathbb{E}\left[{\xi }_i{\xi }_{j^{\prime }}\right]/\left({\sigma }_i{\sigma }_{j^{\prime }}\right)={\kappa }_{i{j}^{\prime }},C_{l{l}^{\prime }}^{yx}={\kappa }_{j{i}^{\prime }},C_{l{l}^{\prime }}^{xx}={\kappa }_{i{i}^{\prime }},C_{l{l}^{\prime }}^{yy}={\kappa }_{j{j}^{\prime }}$$

and, in particular, ${\rho }_l=C_{ll}^{xy}={\kappa }_{ij}$ and ${\rho }_{l^{\prime }}={\kappa }_{i^{\prime }{j}^{\prime }}$, whereas $C_{ll}^{xx}=C_{ll}^{yy}=1$. Component W_n,ij corresponds to component Z_l in Proposition 1. Thus, we conclude that $W_n\stackrel{\mathcal{D}}{\to }W$, where W is a Gaussian matrix with zero mean and $\mathrm{Cov}\left(W_{ij},W_{i^{\prime },j^{\prime }}\right)=D_{l{l}^{\prime }}$, given by Proposition 1. It remains to interpret the quantities in (4.14) in terms of Model M. Substituting $x_l={\overline{\xi }}_i$ and $y_l={\overline{\xi }}_j$ into (4.16) and chasing definitions, we obtain $J_{l{l}^{\prime }}={\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}+{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}$ and $K_{l{l}^{\prime }}={\kappa }_{ij{i}^{\prime }{j}^{\prime }}$. Observing that z_l = x_ly_l = χ_ij and $w_l={\rho }_l\left(x_l^2+y_l^2\right)/2={\psi }_{ij}$, we similarly find that $K_{2,l{l}^{\prime }}={\check{\kappa }}_{ij{i}^{\prime }{j}^{\prime }}$. ☐

5. Proofs of the eigenvalue results

In this section, we derive the main eigenvalue results, presented in Theorem 1 and Theorem 3-(i).

5.1. Preliminaries

Convergence properties of the eigenvalues of R₂₂

It is well known that the empirical spectral density (ESD) of S₂₂ converges weakly a.s. to the Marchenko–Pastur (MP) law F_γ, and that the extreme non-trivial eigenvalues converge to the edges of the support of F_γ. For the sample correlation case, Jiang (2004b) shows that the same is true for R₂₂. That is, the empirical distribution of the eigenvalues μ₁ ≥ … ≥ μ_p of the “noise” correlation matrix $R_{22}=n^{-1}{\overline{X}}_2{\overline{X}}_2^T$ converges weakly a.s. to the MP law F_γ, supported on $\left[a_{\gamma },b_{\gamma }\right]=[{(1-\sqrt{\gamma })}^2,{(1+\sqrt{\gamma })}^2]$, if γ ≤ 1, and on {0} ∪ [a_γ, b_γ] otherwise. Furthermore, the ESD of the n × n companion matrix $C_n=n^{-1}{\overline{X}}_2^T{\overline{X}}_2$, denoted by F_n, converges weakly a.s. to the “companion MP law” F_γ = (1 − γ)1_[0,∞) + γF_γ, where 1_A denotes the indicator function on set A.

In addition, Jiang (2004b) shows that

$${\mu }_1\stackrel{\text{a}.\text{s}.}{\to }{b}_{\gamma }\text{and}{\mu }_{p\wedge n}\stackrel{\text{a}.\text{s}.}{\to }{a}_{\gamma }.$$ (5.18)

Based on these results, if f_n → f uniformly as continuous functions on the closure $\mathcal{I}$ of a bounded neighborhood of the support of F_γ, then:

$$\int f_n(x){\text{F}}_n(dx)\stackrel{\text{a}.\text{s}.}{\to }\int f(x){\text{F}}_{\gamma }(dx).$$ (5.19)

If supp(F_n) is not contained in $\mathcal{I}$, then the left side integral may not be defined. However, such an event occurs for at most finitely many n with probability one.

Almost sure limit of ${\widehat{\ell }}_{\nu }$

The statements in Theorem 1-(i) and Theorem 3-(i) follow easily from known results. Specifically, denote the νth eigenvalue of the sample covariance S by ${\widehat{\lambda }}_{\nu }$. The almost sure limits

$${\widehat{\lambda }}_{\nu }\stackrel{\text{a}.\text{s}.}{\to }\{\begin{array}{cc}{\rho }_{\nu },& {\ell }_{\nu }>1+\sqrt{\gamma }\\ {(1+\sqrt{\gamma })}^2,& 1<{\ell }_{\nu }\le 1+\sqrt{\gamma }\end{array}$$ (5.20)

were established in Baik and Silverstein (2006). From the proof of El Karoui (2009, Lemma 1),

$$\underset{i=1,\mathrm{...},m}{\max }\left|{\widehat{\lambda }}_i-{\widehat{\ell }}_i\right|\stackrel{\text{a}.\text{s}.}{\to }0.$$

Therefore, the same almost sure limits as (5.20) hold for ${\widehat{\ell }}_{\nu }$.

High-probability events J_nϵ, J_nϵ1

When necessary, we may confine attention to the event $J_{nϵ}=\left\{{\widehat{\ell }}_{\nu }>\min \left({\rho }_{\nu },{\rho }_{\nu n}\right)-ϵ,{\mu }_1\le b_{\gamma }+ϵ\right\}$ or $J_{nϵ1}=\left\{{\mu }_1\le b_{\gamma }+ϵ\right\}$, with ϵ > 0 chosen such that ρ_ν – b_γ ≥ 3ϵ, because from (2.5) (proven above) and (5.18), these events occur with probability one for all large n.

Asymptotic expansion of $K\left({\widehat{\ell }}_{\nu }\right)$

We establish an asymptotic stochastic expansion for the quadratic form $K\left({\widehat{\ell }}_{\nu }\right)$. Specifically, using the decomposition

$$K\left({\widehat{\ell }}_{\nu }\right)=K\left({\rho }_{\nu n}\right)+\left[K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)\right],$$ (5.21)

we show that

$$K\left({\rho }_{\nu n}\right)\stackrel{\text{a}.\text{s}.}{\to }-{\rho }_{\nu }\text{m}\left({\rho }_{\nu };\gamma \right)\mathrm{\Gamma }=\left({\rho }_{\nu }/{\ell }_{\nu }\right)\mathrm{\Gamma }$$ (5.22)

and

$$K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)=-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)\left[c\left({\rho }_{\nu }\right)\mathrm{\Gamma }+o_{\text{a}.\text{s}.}(1)\right],$$ (5.23)

where, for $t\notin \mathrm{supp}\left({\text{F}}_{\gamma }\right)$,

$$\text{m}(t;\gamma )=\int {(x-t)}^{-1}{\text{F}}_{\gamma }(dx),c(t)=\int x{(t-x)}^{-2}{\text{F}}_{\gamma }(dx).$$

Here, m is the Stieltjes transform of the companion distribution F_γ.

In establishing (5.22), start by taking sufficiently large n such that |ρ_νn – ρ_ν| ≤ ϵ, with ϵ defined as above. For such n, on J_nϵ1, we have

$$\Vert B_n\left({\rho }_{\nu n}\right)\Vert \le \frac{{\rho }_{\nu }+ϵ}{ϵ}.$$

Because J_nϵ1 holds with probability one for all large n, ‖B_n(ρ_νn)‖ = O_a.s.(1) and, therefore, it follows from Lemma 2 that

$$K\left({\rho }_{\nu n}\right)-n^{-1}\mathrm{tr}{B}_n\left({\rho }_{\nu n}\right)\mathrm{\Gamma }\stackrel{\text{a}.\text{s}.}{\to }0.$$

In addition, (5.19) yields

$$n^{-1}\mathrm{tr}{B}_n\left({\rho }_{\nu n}\right)=\int {\rho }_{\nu n}{\left({\rho }_{\nu n}-x\right)}^{-1}{F}_n(dx)\stackrel{\text{a.s.}}{\to }\int {\rho }_{\nu }{\left({\rho }_{\nu }-x\right)}^{-1}{F}_{\gamma }(dx)=-{\rho }_{\nu }m\left({\rho }_{\nu };\gamma \right).$$

Explicit evaluation gives m(ρ_ν; γ) = −1/ℓ_ν, [JY, Appendix A], and (5.22) follows.

To establish (5.23), we start by recalling that $C_n=n^{-1}{\overline{X}}_2^T{\overline{X}}_2$, and introduce the resolvent notation Z(t) = (tI_n − C_n)⁻¹, such that B_n(t) = tZ(t) and $K\text{(}t)=n^{-1}{\overline{X}}_{1}tZ(t){\overline{X}}_1^T$. From the resolvent identity, that is, A⁻¹ − B⁻¹ = A⁻¹(B − A)B⁻¹ for square invertible A and B, and noting that tZ(t) = C_nZ(t) + I from the Woodbury identity, we have, for t₁, t₂ > b_γ,

$$t_{1}Z\left(t_1\right)-t_{2}Z\left(t_2\right)=-\left(t_1-t_2\right)C_{n}Z\left(t_1\right)Z\left(t_2\right)$$

and, therefore,

$$K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)=-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)n^{-1}{\overline{X}}_1{C}_{n}Z\left({\widehat{\ell }}_{\nu }\right)Z\left({\rho }_{\nu n}\right){\overline{X}}_1^T.$$

Moreover, again by the resolvent identity, $Z\left({\widehat{\ell }}_{\nu }\right)=Z\left({\rho }_{\nu n}\right)-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)Z\left({\widehat{\ell }}_{\nu }\right)Z\left({\rho }_{\nu n}\right)$, which yields

$$K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)=-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)n^{-1}{\overline{X}}_1{B}_{n1}\left({\rho }_{\nu n},{\rho }_{\nu n}\right){\overline{X}}_1^T+{\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)}^2{n}^{-1}{\overline{X}}_1{B}_{n2}\left({\widehat{\ell }}_{\nu },{\rho }_{\nu n}\right){\overline{X}}_1^T,$$ (5.24)

with B_nr(t₁, t₂) defined as

$$B_{nr}\left(t_1,t_2\right)=C_{n}Z\left(t_1\right)Z^r\left(t_2\right).$$ (5.25)

We now characterize the first-order behavior of the two matrix quadratic forms in (5.24). For the first, we simply mirror the arguments of the proof of (5.22) to obtain

$$n^{-1}{\overline{X}}_1{B}_{n1}\left({\rho }_{\nu n},{\rho }_{\nu n}\right){\overline{X}}_1^T\stackrel{\text{a.s.}}{\to }c\left({\rho }_{\nu }\right)\mathrm{\Gamma }.$$

For the second, we again apply similar reasoning, operating on the event J_nϵ. Specifically, it is easy to establish that on J_nϵ, and for n sufficiently large that |ρ_νn – ρ_ν| ≤ ϵ, $\Vert B_{n2}\left({\widehat{\ell }}_{\nu },{\rho }_{\nu n}\right)\Vert$ is bounded. Hence, $\Vert B_{n2}\left({\widehat{\ell }}_{\nu },{\rho }_{\nu n}\right)\Vert =O_{\text{a}.\text{s}.}(1)$, and it follows from Lemma 2 and (5.19) that

$$n^{-1}{\overline{X}}_1{B}_{n2}\left({\widehat{\ell }}_{\nu },{\rho }_{\nu n}\right){\overline{X}}_1^T=O_{\text{a}.\text{s}.}(1).$$

The expansion in (5.23) is obtained by combining the latter two equations with (5.24).

CLT of K(ρ_νn)

We now specialize Proposition 2 for the matrix quadratic form K(ρ_νn).

Proposition 3

Assume Model M, and define ρ_νn by (1.1) and K(ρ_νn) by (3.13). Then,

$$W_n\left({\rho }_{\nu n}\right)=\sqrt{n}\left[K\left({\rho }_{\nu n}\right)-n^{-1}\mathrm{tr}{B}_n\left({\rho }_{\nu n}\right)\mathrm{\Gamma }\right]\stackrel{\mathcal{D}}{\to }{W}^{\nu },$$

which is a symmetric Gaussian random matrix with entries $W_{ij}^{\nu }$, mean zero, and covariances given by

$$\mathrm{Cov}\left[W_{ij}^{\nu },W_{i^{\prime }{j}^{\prime }}^{\nu }\right]=\frac{{\rho }_{\nu }^2}{{\ell }_{\nu }^2{\dot{\rho }}_{\nu }}\left({\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}+{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}\right)+\frac{{\rho }_{\nu }^2}{{\ell }_{\nu }^2}\left({\kappa }_{ij{i}^{\prime }{j}^{\prime }}+{\check{\kappa }}_{ij{i}^{\prime }{j}^{\prime }}\right),$$ (5.26)

where ρ_ν and ${\dot{\rho }}_{\nu }$ are defined in (1.1), and the terms in parentheses are defined in (1.2) and (1.4).

Proof. Recall that J_nϵ1 = {μ₁ ≤ b_γ + ϵ}, and consider sufficiently large n such that ρ_νn > ρ_ν – ϵ. Then, we may apply Proposition 2 with $B_n=B_n\left({\rho }_{\nu n}\right)1_{J_{n\in 1}}$, which is independent of ${\overline{X}}_1$, and for which ‖B_n‖ is bounded. Specifically, the result follows by applying Proposition 2 to $W_n\left({\rho }_{\nu n}\right)1_{J_{n\in 1}}$, along with the fact that $1_{J_{n\in 1}}\stackrel{\text{a.s.}}{\to }1$, and particularizing ω, θ, and ϕ in (4.17). These quantities, denoted respectively by ω_ν, θ_ν, and ϕ_ν, can be computed as in [JY, Appendix A], yielding

$${\omega }_{\nu }={\varphi }_{\nu }=\frac{{\left({\ell }_{\nu }-1+\gamma \right)}^2}{{\left({\ell }_{\nu }-1\right)}^2}=\frac{{\rho }_{\nu }^2}{{\ell }_{\nu }^2},{\theta }_{\nu }=\frac{{\left({\ell }_{\nu }-1+\gamma \right)}^2}{{\left({\ell }_{\nu }-1\right)}^2-\gamma }=\frac{{\omega }_{\nu }}{{\dot{\rho }}_{\nu }}.$$

Tightness properties

Lastly, we establish some tightness properties essential to the derivation of our second-order results.

We first establish a refinement of (5.22). Define K₀(ρ; γ) := −ρm(ρ; γ)Γ, such that (5.22) is rewritten as $K\left({\rho }_{\nu n}\right)\stackrel{\text{a.s.}}{\to }{K}_0\left({\rho }_{\nu };\gamma \right)$. Set g_ρ(x) = ρ(ρ − x)⁻¹, and write

$$\mathrm{tr}{B}_n(\rho )=\sum _{i=1}^n\rho {\left(\rho -{\mu }_i\right)}^{-1}=\sum _{i=1}^n{g}_{\rho }\left({\mu }_i\right).$$

In addition, introducing

$$G_n(g):=\sum _{i=1}^{n}g\left({\mu }_i\right)-n\int g(x)F_{{\gamma }_n}(dx),$$

we have

$$\begin{gathered} K(\rho )-K_0\left(\rho ;{\gamma }_n\right)=K(\rho )-n^{-1}\mathrm{tr}{B}_n(\rho )\mathrm{\Gamma }+\rho n^{-1}\left[\sum _{i=1}^n{\left(\rho -{\mu }_i\right)}^{-1}-n\int {(\rho -x)}^{-1}{F}_{{\gamma }_n}(dx)\right]\mathrm{\Gamma } \\ =n^{-1/2}{W}_n(\rho )+n^{-1}{G}_n\left(g_{\rho }\right)\mathrm{\Gamma }. \end{gathered}$$ (5.27)

Lemma 3

Assume that Model M holds, and that ${\ell }_{\nu }>1+\sqrt{\gamma }$ is simple. For some b > ρ₁, let I denote the interval [b_γ + 3ϵ, b]. Then,

$$\left\{G_n\left(g_{\rho }\right),\rho \in I\right\}isuniformlytight,$$ (5.28)

$$\left\{n^{1/2}\left[K(\rho )-K_0\left(\rho ;{\gamma }_n\right)\right],\rho \in I\right\}isuniformlytight,$$ (5.29)

$${\widehat{\ell }}_{\nu }-{\rho }_{\nu n}=O_p\left(n^{-1/2}\right),$$ (5.30)

$$a_{\nu }-p_{\nu }=O_p\left(n^{-1/2}\right).$$ (5.31)

Proof. The proofs of (5.28)–(5.30) appear in the Supplementary Material, S2. We show (5.31) using the expansion $a_{\nu }-p_{\nu }=-{\mathcal{R}}_{\nu n}{D}_{\nu }{p}_{\nu }+r_{\nu }$, given in (3.12), from which we recall ‖r_ν‖ = O(‖D_ν‖²) and note that $\Vert {\mathcal{R}}_{\nu n}\Vert \le C$ and $D_{\nu }=K\left({\widehat{\ell }}_{\nu }\right)-K_0\left({\rho }_{\nu n};{\gamma }_n\right)$. We then have a_ν − p_ν = O_p(‖D_ν‖ + ‖D_ν‖²). Furthermore, from

$$\Vert D_{\nu }\Vert \le \Vert K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)\Vert +\Vert K\left({\rho }_{\nu n}\right)-K_0\left({\rho }_{\nu n};{\gamma }_n\right)\Vert ,$$

the first term is O_p(n^−1/2) by (5.23) and (5.30), as is the second term by (5.29). Hence,

$$\Vert D_{\nu }\Vert =O_p\left(n^{-1/2}\right),$$ (5.32)

and the proof is completed. ☐

5.2. Eigenvalue uctuations (Theorem 1-(ii))

The proof of Theorem 1-(ii) relies on the key expansion

$$\sqrt{n}\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)\left[1+c\left({\rho }_{\nu }\right){\ell }_{\nu }+o_p(1)\right]=p_{\nu }^T{W}_n\left({\rho }_{\nu n}\right)p_{\nu }+o_p(1),$$ (5.33)

which is obtained by combining the vector equations $K\left({\widehat{\ell }}_{\nu }\right)a_{\nu }={\widehat{\ell }}_{\nu }{a}_{\nu }$ and K₀(ρ_νn;γ_n)p_ν = ρ_νnp_ν with expansions (5.24) for $K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)$ and (5.27) for K(ρ_νn) − K₀(ρ_νn;γ_n). Specifically, we first use $\left[K\left({\widehat{\ell }}_{\nu }\right)-{\widehat{\ell }}_{\nu }{I}_m\right]a_{\nu }=0$ to obtain

$$p_{\nu }^T\left[K\left({\widehat{\ell }}_{\nu }\right)-{\widehat{\ell }}_{\nu }{I}_m\right]p_{\nu }={\left(a_{\nu }-p_{\nu }\right)}^T\left[K\left({\widehat{\ell }}_{\nu }\right)-{\widehat{\ell }}_{\nu }{I}_m\right]\left(a_{\nu }-p_{\nu }\right)=O_p\left(n^{-1}\right),$$ (5.34)

because $\Vert K\left({\widehat{\ell }}_{\nu }\right)-{\widehat{\ell }}_{\nu }{I}_m\Vert =O_p(1)$ from (5.21)–(5.23) and (2.5), and a_ν − p_ν = O_p(n^−1/2) from Lemma 3. In addition, because [K₀(ρ_νn;γ_n) − ρ_νnI_m]p_ν = 0, it follows that

$$\begin{gathered} p_{\nu }^T\left[K\left({\widehat{\ell }}_{\nu }\right)-{\widehat{\ell }}_{\nu }{I}_m\right]p_{\nu }=p_{\nu }^T\left[K\left({\widehat{\ell }}_{\nu }\right)-K_0\left({\rho }_{\nu n};{\gamma }_n\right)-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)I_m\right]p_{\nu } \\ =p_{\nu }^T\left[K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)I_m\right]p_{\nu }+p_{\nu }^T\left[K\left({\rho }_{\nu n}\right)-K_0\left({\rho }_{\nu n};{\gamma }_n\right)\right]p_{\nu } \\ =-\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)\left[1+c\left({\rho }_{\nu }\right){\ell }_{\nu }+o_p(1)\right]+n^{-1/2}{p}_{\nu }^T{W}_n\left({\rho }_{\nu n}\right)p_{\nu }+o_p\left(n^{-1/2}\right), \end{gathered}$$ (5.35)

where the last equality follows from (5.23), (5.27), and (5.28). Combining (5.34) and (5.35) yields (5.33).

The asymptotic normality of $\sqrt{n}\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)$ now follows from Proposition 3, with asymptotic variance

$${\tilde{\sigma }}_{\nu }^2={\left[1+c\left({\rho }_{\nu }\right){\ell }_{\nu }\right]}^{-2}\mathrm{Var}\left[p_{\nu }^T{W}^{\nu }{p}_{\nu }\right]={\left({\dot{\rho }}_{\nu }{\ell }_{\nu }/{\rho }_{\nu }\right)}^2\sum _{i,j,i^{\prime },j^{\prime }}{\mathcal{P}}_{ij{i}^{\prime }{j}^{\prime }}^{\nu }\mathrm{Cov}\left[W_{ij}^{\nu },W_{i^{\prime }{j}^{\prime }}^{\nu }\right],$$

where W^ν is the m × m symmetric Gaussian random matrix defined in Proposition 3, with covariance $\mathrm{Cov}\left[W_{ij}^{\nu },W_{i^{\prime }{j}^{\prime }}^{\nu }\right]$ given by (5.26). Using this in the developed expression for the variance above leads to

$${\tilde{\sigma }}_{\nu }^2={\dot{\rho }}_{\nu }\sum _{i,j,i^{\prime },j^{\prime }}{\mathcal{P}}_{ij{i}^{\prime }{j}^{\prime }}^{\nu }\left({\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}+{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}\right)+{\dot{\rho }}_{\nu }^2\left[{\mathcal{P}}^{\nu },\kappa +\check{\kappa }\right].$$ (5.36)

By symmetry and the eigen equation ${\left(\mathrm{\Gamma }{p}_{\nu }\right)}_i=\sum _j{\kappa }_{ij}{p}_{\nu ,j}={\ell }_{\nu }{p}_{\nu ,i}$, we have

$$\sum _{i,j,i^{\prime },j^{\prime }}{\mathcal{P}}_{ij{i}^{\prime }{j}^{\prime }}^{\nu }{\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}=\sum _{i,j,i^{\prime },j^{\prime }}{\mathcal{P}}_{ij{i}^{\prime }{j}^{\prime }}^{\nu }{\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}=\sum _{i,j}{p}_{\nu ,i}{p}_{\nu ,j}{\left(\mathrm{\Gamma }{p}_{\nu }\right)}_i{\left(\mathrm{\Gamma }{p}_{\nu }\right)}_j={\ell }_{\nu }^2\sum _{i,j}{\left(p_{\nu ,i}{p}_{\nu ,j}\right)}^2={\ell }_{\nu }^2.$$

Therefore, the first sum in (5.36) reduces to $2{\dot{\rho }}_{\nu }{\ell }_{\nu }^2$, yielding formula (2.6) of Theorem 1.

6. Proofs of the eigenvector results

We now derive the main eigenvector results, presented in Theorem 2 and Theorem 3-(ii).

6.1. Eigenvector inconsistency (Theorem 2-(i))

The convergence result of Theorem 2-(i) follows from two facts: $a_{\nu }\stackrel{\text{a}.\text{s}.}{\to }{p}_{\nu }$ and $Q_{\nu }\stackrel{\text{a}.\text{s}.}{\to }c\left({\rho }_{\nu }\right)\mathrm{\Gamma }$, which are shown below. Once these facts are established, from (3.10),

$${\Vert {\widehat{p}}_{\nu }\Vert }^{-2}\stackrel{\text{a.s.}}{\to }{p}_{\nu }^T\left(I_m+c\left({\rho }_{\nu }\right)\mathrm{\Gamma }\right)p_{\nu }=1+c\left({\rho }_{\nu }\right){\ell }_{\nu }=\frac{{\rho }_{\nu }}{{\ell }_{\nu }{\dot{\rho }}_{\nu }},$$

which leads to

$$\text{a.s.}\lim {\langle {\widehat{\mathfrak{p}}}_{\nu },{\mathfrak{p}}_{\nu }\rangle }^2=\text{a.s.}\lim {\langle {\widehat{p}}_{\nu },p_{\nu }\rangle }^2=\text{a.s.}\lim {\Vert {\widehat{p}}_{\nu }\Vert }^2=\frac{{\ell }_{\nu }{\dot{\rho }}_{\nu }}{{\rho }_{\nu }}.$$

Proof of $a_{\nu }\stackrel{\text{a.s.}}{\to }{p}_{\nu }$

This is a direct consequence of (3.12) and

$$D_{\nu }=K\left({\rho }_{\nu n}\right)-\left({\rho }_{\nu n}/{\ell }_{\nu }\right)\mathrm{\Gamma }+K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)\stackrel{\text{a}.\text{s}.}{\to }0,$$

which follows from (5.22), (5.23), and the fact that ${\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\stackrel{\text{a.s.}}{\to }0$, given in (2.5).

Proof of $Q_{\nu }\stackrel{\text{a}.\text{s}.}{\to }c\left({\rho }_{\nu }\right)\mathrm{\Gamma }$

With $\check{Z}(t)={\left(t{I}_p-R_{22}\right)}^{-1}$, we have

$$Q_{\nu }=R_{12}{\check{Z}}^2\left({\rho }_{\nu }\right)R_{21}+R_{12}\left[{\check{Z}}^2\left({\widehat{\ell }}_{\nu }\right)-{\check{Z}}^2\left({\rho }_{\nu }\right)\right]R_{21}\triangleq Q_{\nu 1}+Q_{\nu 2}.$$

Rewrite $Q_{\nu 1}=n^{-1}{\overline{X}}_1{\check{B}}_{n1}{\overline{X}}_1^T$, with ${\check{B}}_{n1}=n^{-1}{\overline{X}}_2^T{\check{Z}}^2\left({\rho }_{\nu }\right){\overline{X}}_2$. On the high-probability event J_nϵ1 = {μ₁ ≤ b_γ + ϵ}, with ϵ > 0 such that ρ_ν – b_γ ≥ 2ϵ, it is easily established that $\Vert {\check{B}}_{n1}\Vert$ is bounded and, consequently, that $\Vert {\check{B}}_{n1}\Vert =O_{\text{a}.\text{s}.}(1)$. Hence, Lemma 2 can be applied to Q_ν1. Moreover, from (5.19) and noting that

$$n^{-1}\mathrm{tr}{\check{B}}_{n1}=n^{-1}\mathrm{tr}{B}_{n1}\left({\rho }_{\nu },{\rho }_{\nu }\right),$$

with B_n1 defined in (5.25), we have

$$n^{-1}\mathrm{tr}{\check{B}}_{n1}\stackrel{\text{a}.\text{s}.}{\to }\int x{\left({\rho }_{\nu }-x\right)}^{-2}{\text{F}}_{\gamma }(dx)=c\left({\rho }_{\nu }\right).$$

This and Lemma 2 imply that $Q_{\nu 1}\stackrel{\text{a.s.}}{\to }c\left({\rho }_{\nu }\right)\mathrm{\Gamma }$.

It remains to show $Q_{\nu 2}\stackrel{\text{a}.\text{s}.}{\to }0$. Using a variant of the resolvent identity, that is, A⁻² − B⁻² = −A⁻²(A² − B²)B⁻² for square invertible A and B, we rewrite

$$Q_{\nu 2}=-2\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu }\right)n^{-1}{\overline{X}}_1{\check{B}}_{n2}{\overline{X}}_1^T,$$

with ${\check{B}}_{n2}=n^{-1}{\overline{X}}_2^T{\check{Z}}^2\left({\widehat{\ell }}_{\nu }\right)\left[\frac{1}{2}\left({\widehat{\ell }}_{\nu }+{\rho }_{\nu }\right)I-R_{22}\right]{\check{Z}}^2\left({\rho }_{\nu }\right){\overline{X}}_2$. Working on the high-probability event J_nϵ, it can be verified that $\Vert {\check{B}}_{n2}\Vert =O_{\text{a}.\text{s}.}(1)$. Thus, Lemma 2 together with (5.19) imply that $n^{-1}{\overline{X}}_1{\check{B}}_{n2}{\overline{X}}_1^T=O_{\text{a}.\text{s}.}(1)$. Because ${\widehat{\ell }}_{\nu }\stackrel{\text{a}.\text{s}.}{\to }{\rho }_{\nu }$, we conclude that $Q_{\nu 2}\stackrel{\text{a.s.}}{\to }0$.

6.2. Eigenvector fluctuations (Theorem 2-(ii))

Again, we use the key expansion (3.12). Because ‖r_ν‖ = O(‖D_ν‖²) = O_p(n⁻¹) from (5.32), we have

$$\sqrt{n}\left(a_{\nu }-p_{\nu }\right)=-{\mathcal{R}}_{\nu n}\sqrt{n}{D}_{\nu }{p}_{\nu }+o_p(1).$$

Furthermore, using a similar decomposition to the derivation of (5.35),

$$\begin{gathered} \sqrt{n}{D}_{\nu }=\sqrt{n}\left[K\left({\widehat{\ell }}_{\nu }\right)-K\left({\rho }_{\nu n}\right)\right]+\sqrt{n}\left[K\left({\rho }_{\nu n}\right)-K_0\left({\rho }_{\nu n},{\gamma }_n\right)\right] \\ =W_n\left({\rho }_{\nu n}\right)-\sqrt{n}\left({\widehat{\ell }}_{\nu }-{\rho }_{\nu n}\right)c\left({\rho }_{\nu }\right)\mathrm{\Gamma }+o_p(1), \end{gathered}$$

where we use (5.23) and (5.27), along with (5.28) and (5.30) of Lemma 3. Hence, noting that ${\mathcal{R}}_{\nu n}\mathrm{\Gamma }{p}_{\nu }={\ell }_{\nu }{\mathcal{R}}_{\nu n}{p}_{\nu }=0$ from the definition of ${\mathcal{R}}_{\nu n}$ in (3.12), we have

$$\sqrt{n}\left(a_{\nu }-p_{\nu }\right)=-{\mathcal{R}}_{\nu n}{W}_n\left({\rho }_{\nu n}\right)p_{\nu }+o_p(1),$$

or equivalently,

$$\sqrt{n}\left(P^T{a}_{\nu }-e_{\nu }\right)=-{\tilde{\mathcal{R}}}_{\nu n}{\tilde{W}}_n\left({\rho }_{\nu n}\right)e_{\nu }+o_p(1),$$

where

$${\tilde{\mathcal{R}}}_{\nu n}=\frac{{\ell }_{\nu }}{{\rho }_{\nu n}}\sum _{k\ne \nu }^m{\left({\ell }_k-{\ell }_{\nu }\right)}^{-1}{e}_k{e}_k^T,{\tilde{W}}_n\left({\rho }_{\nu n}\right)=P^T{W}_n\left({\rho }_{\nu n}\right)P.$$

The CLT for P^T a_ν now follows from Proposition 3. In particular,

$$\sqrt{n}\left(P^T{a}_{\nu }-e_{\nu }\right)\stackrel{\mathcal{D}}{\to }{\tilde{\mathcal{R}}}_{\nu }{w}_{\nu }~N\left(0,{\mathrm{\Sigma }}_{\nu }\right),$$

where ${\tilde{\mathcal{R}}}_{\nu }=\left({\ell }_{\nu }/{\rho }_{\nu }\right){\mathcal{D}}_{\nu }$, recall (2.8), and w_ν = P^T W^νp_ν, with W^ν defined in Proposition 3. The covariance matrix ${\mathrm{\Sigma }}_{\nu }={\tilde{\mathcal{R}}}_{\nu }\mathbb{E}\left[w_{\nu }{w}_{\nu }^T\right]{\tilde{\mathcal{R}}}_{\nu }={\mathcal{D}}_{\nu }{\tilde{\mathrm{\Sigma }}}_{\nu }{\mathcal{D}}_{\nu }$, with ${\tilde{\mathrm{\Sigma }}}_{\nu }={\left({\ell }_{\nu }/{\rho }_{\nu }\right)}^2\mathbb{E}\left[w_{\nu }{w}_{\nu }^T\right]$. The kth component of w_ν is given by $w_{\nu }(k)=p_k^T{W}^{\nu }{p}_{\nu }={\sum }_{i,j}{p}_{k,i}{W}_{ij}^{\nu }{p}_{\nu ,j}$ and, therefore,

$${\tilde{\mathrm{\Sigma }}}_{\nu ,kl}=\sum _{i,j,i^{\prime },j^{\prime }}{p}_{k,i}{p}_{\nu ,j}{p}_{l,i^{\prime }}{p}_{\nu ,j^{\prime }}{\left({\ell }_{\nu }/{\rho }_{\nu }\right)}^2\mathrm{Cov}\left[W_{ij}^{\nu },W_{i^{\prime }{j}^{\prime }}^{\nu }\right].$$ (6.37)

Theorem 2-(ii) follows after substituting (5.26) for $\mathrm{Cov}\left[W_{ij}^{\nu },W_{i^{\prime }{j}^{\prime }}^{\nu }\right]$ and noting that, when k, l ≠ ν,

$$\sum _{i,j,i^{\prime },j^{\prime }}{p}_{k,i}{p}_{\nu ,j}{p}_{l,i^{\prime }}{p}_{\nu ,j^{\prime }}\left({\kappa }_{i{i}^{\prime }}{\kappa }_{j{j}^{\prime }}+{\kappa }_{i{j}^{\prime }}{\kappa }_{j{i}^{\prime }}\right)=p_k^T\mathrm{\Gamma }{p}_l\cdot p_{\nu }^T\mathrm{\Gamma }{p}_{\nu }+p_k^T\mathrm{\Gamma }{p}_{\nu }\cdot p_{\nu }^T\mathrm{\Gamma }{p}_l={\delta }_{kl}{\ell }_k{\ell }_{\nu }.$$

6.3. Eigenvector inconsistency in the subcritical case (Theorem 3-(ii))

From (3.10) and (3.11), it suffices to show that $a_{\nu }^T{Q}_{\nu }{a}_{\nu }\stackrel{\text{a}.\text{s}.}{\to }\infty$ in order for Theorem 3-(ii) to hold. We establish this by showing that ${\lambda }_{\min }\left(Q_{\nu }\right)\stackrel{\text{a.s.}}{\to }\infty$. The approach uses a regularized version of Q_ν,

$$Q_{\nu ϵ}(t)=R_{12}{\left[{\left(t{I}_p-R_{22}\right)}^2+{ϵ}^2{I}_p\right]}^{-1}{R}_{21},$$

for ϵ > 0. Observe that $Q_{\nu }\succ Q_{\nu ϵ}\left({\widehat{\ell }}_{\nu }\right)$, such that

$$\lim \inf {\lambda }_{\min }\left(Q_{\nu }\right)\ge \lim \inf {\lambda }_{\min }\left(Q_{\nu ϵ}\left({\widehat{\ell }}_{\nu }\right)\right)=\lim \inf {\lambda }_{\min }\left(Q_{\nu ϵ}\left(b_{\gamma }\right)+{\mathrm{\Delta }}_{\nu ϵ}\right),$$

where ${\mathrm{\Delta }}_{\nu ϵ}:=Q_{\nu ϵ}\left({\widehat{\ell }}_{\nu }\right)-Q_{\nu ϵ}\left(b_{\gamma }\right)$ (Recall that ${\widehat{\ell }}_{\nu }\stackrel{\text{a.s.}}{\to }{b}_{\gamma })$. We show that ${\mathrm{\Delta }}_{\nu ϵ}\stackrel{\text{a}.\text{s}.}{\to }0$, and

$$Q_{\nu ϵ}\left(b_{\gamma }\right)\stackrel{\text{a}.\text{s}.}{\to }\int x{\left[{\left(b_{\gamma }-x\right)}^2+{ϵ}^2\right]}^{-1}{\text{F}}_{\gamma }(dx)\cdot \mathrm{\Gamma }=c_{\gamma }(ϵ)\mathrm{\Gamma },$$ (6.38)

say. Because λ_min(·) is a continuous function on m × m matrices, we conclude that

$$\lim \inf {\lambda }_{\min }\left(Q_{\nu }\right)\ge c_{\gamma }(ϵ){\lambda }_{\min }(\mathrm{\Gamma }),$$ (6.39)

and because c_γ(ϵ) ≥ c(b_γ + ϵ) and c(b_γ + ϵ) ↗ ∞ as ϵ ↘ 0, by [JY, Appendix A], we obtain ${\lambda }_{\min }\left(Q_{\nu }\right)\stackrel{\text{a.s.}}{\to }\infty$. We write $Q_{\nu ϵ}(t)=n^{-1}{\overline{X}}_1{\check{B}}_{nϵ}(t){\overline{X}}_1$, with

$$\begin{gathered} {\check{B}}_{nϵ}(t)=n^{-1}{\overline{X}}_2^T{\left[{\left(t{I}_p-n^{-1}{\overline{X}}_2{\overline{X}}_2^T\right)}^2+{ϵ}^2{I}_p\right]}^{-1}{\overline{X}}_2 \\ =H\mathrm{diag}\left\{f_{ϵ}\left({\mu }_i,t\right)\right\}{H}^T, \end{gathered}$$

if we write the singular-value decomposition of $n^{-1/2}{\overline{X}}_2=V{\mathcal{M}}^{1/2}{H}^T$, with $\mathcal{M}=\mathrm{diag}{\left({\mu }_i\right)}_{i=1}^p$ and define $f_{ϵ}(\mu ,t)=\mu {\left[{(t-\mu )}^2+{ϵ}^2\right]}^{-1}$. Evidently, $\Vert {\check{B}}_{nϵ}(t)\Vert \le {ϵ}^{-2}{\mu }_1$ is bounded almost surely. Thus, Lemma 2 may be applied to Q_νϵ(b_γ), and because

$$n^{-1}\mathrm{tr}{\check{B}}_{nϵ}\left(b_{\gamma }\right)\stackrel{\text{a}.\text{s}.}{\to }\int f_{ϵ}\left(x,b_{\gamma }\right){\text{F}}_{\gamma }(dx)=c_{\gamma }(ϵ)$$

from (5.19), our claim (6.38) follows.

Now consider Δ_νϵ. Fix $a\in {ℝ}^m$ such that ‖a‖₂ = 1, and set $b=n^{-1/2}{H}^T{\overline{X}}_1^{T}a$. We have

$$a^T{\mathrm{\Delta }}_{\nu ϵ}a=\sum _{i=1}^p{b}_i^2\left[f_{ϵ}\left({\mu }_i,{\widehat{\ell }}_{\nu }\right)-f_{ϵ}\left({\mu }_i,b_{\gamma }\right)\right].$$

Because $\left|\partial f_{ϵ}(\mu ,t)/\partial t\right|=|2\mu (t-\mu )|/{\left[{(t-\mu )}^2+{ϵ}^2\right]}^2\le \mu /{ϵ}^3$, for μ, ϵ > 0, by the arithmeticmean–geometric-mean inequality, we have

$$\left|a^T{\mathrm{\Delta }}_{\nu ϵ}a\right|\le {\mu }_1{ϵ}^{-3}\left|{\widehat{\ell }}_{\nu }-b_{\gamma }\right|\cdot \Vert b{\Vert }_2^2={\mu }_1{ϵ}^{-3}\left|{\widehat{\ell }}_{\nu }-b_{\gamma }\right|a^T{R}_{11}a\le {\mu }_1{ϵ}^{-3}\left|{\widehat{\ell }}_{\nu }-b_{\gamma }\right|{\widehat{\ell }}_1\stackrel{\text{a.s.}}{\to }0,$$

from Cauchy’s interlacing inequality for eigenvalues of symmetric matrices, Theorem 1-(i) and Theorem 3-(i). Therefore, ${\mathrm{\Delta }}_{\nu ϵ}\stackrel{\text{a}.\text{s}.}{\to }0$, and the proof of (6.39) and, hence, of Theorem 3-(ii) is complete.