Robust high dimensional factor models with applications to statistical machine learning

Abstract

Factor models are a class of powerful statistical models that have been widely used to deal with dependent measurements that arise frequently from various applications from genomics and neuroscience to economics and finance. As data are collected at an ever-growing scale, statistical machine learning faces some new challenges: high dimensionality, strong dependence among observed variables, heavy-tailed variables and heterogeneity. High-dimensional robust factor analysis serves as a powerful toolkit to conquer these challenges.

This paper gives a selective overview on recent advance on high-dimensional factor models and their applications to statistics including Factor-Adjusted Robust Model selection (FarmSelect) and Factor-Adjusted Robust Multiple testing (FarmTest). We show that classical methods, especially principal component analysis (PCA), can be tailored to many new problems and provide powerful tools for statistical estimation and inference. We highlight PCA and its connections to matrix perturbation theory, robust statistics, random projection, false discovery rate, etc., and illustrate through several applications how insights from these fields yield solutions to modern challenges. We also present far-reaching connections between factor models and popular statistical learning problems, including network analysis and low-rank matrix recovery.

1. INTRODUCTION

In modern data analytics, dependence across high-dimensional outcomes or measurements is ubiquitous. For example, stocks within the same industry exhibit significantly correlated returns, housing prices of a country depend on various economic factors, gene expressions can be stimulated by cytokines. Ignoring such dependence structure can produce significant systematic bias and yields inefficient statistical results and misleading insights. The problems are more severe for high-dimensional big data, where dependence, non-Gaussianity and heterogeneity of measurements are common.

Factor models aim to capture such dependence by assuming several variates or “factors”, usually much fewer than the outcomes, that drive the dependence of the entire outcomes (Lawley and Maxwell, 1962; Stock and Watson, 2002). Stemming from the early works on measuring human abilities (Spearman, 1927), factor models have become one of the most popular and powerful tools in multivariate analysis and have made profound impact in the past century on psychology (Bartlett, 1938; McCrae and John, 1992), economics and finance (Chamberlain and Rothschild, 1982; Fama and French, 1993; Stock and Watson, 2002; Bai and Ng, 2002), biology (Hirzel et al., 2002; Hochreiter et al., 2006; Leek and Storey, 2008), etc. Suppose x₁, …, x_n are n i.i.d. p-dimensional random vectors, which may represent financial returns, housing prices, gene expressions, etc. The generic factor model assumes that

$${\mathbf{\text{x}}}_i=\mathit{\mu }+\mathbf{\text{B}}{\mathbf{\text{f}}}_i+{\mathbf{\text{u}}}_i,\hspace{1em}\text{or in matrix form},\hspace{1em}\mathbf{\text{X}}=\mathit{\mu }{\mathbf{\text{1}}}_n^{\top }+\mathbf{\text{B}}{\mathbf{\text{F}}}^{\top }+\mathbf{\text{U}},$$ (1.1)

where $\mathbf{\text{X}}=\left({\mathbf{\text{x}}}_1,\dots ,{\mathbf{\text{x}}}_n\right)\in {ℝ}^{p\times n}$, $\mathit{\mu }={\left({\mu }_1,\dots ,{\mu }_p\right)}^{\top }$ is the mean vector, $\mathbf{\text{B}}={\left({\mathbf{\text{b}}}_1,\dots ,{\mathbf{\text{b}}}_p\right)}^{\top }\in {ℝ}^{p\times K}$ is the matrix of factor loadings, $\mathbf{\text{F}}={\left({\mathbf{\text{f}}}_1,\dots ,{\mathbf{\text{f}}}_n\right)}^{\top }\in {ℝ}^{n\times K}$ stores K-dimensional vectors of common factors with $\mathbb{E}{\mathbf{\text{f}}}_i=0$, and $\mathbf{\text{U}}=\left({\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_n\right)\in {ℝ}^{p\times n}$ represents the error terms (a.k.a. idiosyncratic components), which has mean zero and is uncorrelated with or independent of F. We emphasize that, for most of our discussions in the paper (except Section 3.1), only ${\left\{{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$ are observable, and the goal is to infer B and ${\left\{{\mathbf{\text{f}}}_i\right\}}_{i=1}^n$ through ${\left\{{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$. Here we use the name “factor model” to refer to a general concept where the idiosyncratic components u_i are allowed to be weakly correlated. This is also known as the “approximate factor model” in the literature, in contrast to the “strict factor model” where the idiosyncratic components are assumed to be uncorrelated.

Note that the model (1.1) has identifiability issues: given any invertible matrix $\mathbf{\text{R}}\in {ℝ}^{K\times K}$, simultaneously replacing B with BR and f_i with R⁻¹f_i does not change the observation x_i. To resolve this ambiguity issue, the following identifiability assumption is usually imposed:

Assumption 1.1 (Identifiability). B^⊤B is diagonal and cov(f_i) = I_p.

Other identifiability assumptions as well as detailed discussions can be found in Bai and Li (2012) and Fan et al. (2013).

Factor analysis is closely related to principal component analysis (PCA), which breaks down the covariance matrix into a set of orthogonal components and identifies the subspace that explains the most variation of the data (Pearson, 1901; Hotelling, 1933). In this selective review, we will mainly leverage PCA, or more generally, spectral methods, to estimate the factors ${\left\{{\mathbf{\text{f}}}_i\right\}}_{i=1}^n$ and the loading matrix B in (1.1). Other popular estimators, mostly based on the maximum likelihood principle, can be found in Lawley and Maxwell (1962); Anderson and Amemiya (1988); Bai and Li (2012), etc. The covariance matrix of x_i consists of two components: cov(Bf_i) and cov(u_i). Intuitively, when the contribution of the covariance from the error term u_i is negligible compared with those from the factor term Bf_i, the top-K eigenspace (namely, the space spanned by top K eigenvectors) of the sample covariance of ${\left\{{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$ should be well aligned with the column space of B. This can be seen from the assumption that $\text{cov}\left({\mathbf{\text{x}}}_i\right)=\mathbf{\text{B}}{\mathbf{\text{B}}}^{\top }+\text{cov}\left({\mathbf{\text{u}}}_i\right)\approx \mathbf{\text{B}}{\mathbf{\text{B}}}^{\top }$, which occurs frequently in high-dimensional statistics (Fan et al., 2013).

Here is our main message: applying PCA to well-crafted covariance matrices (including vanilla sample covariance matrices and their robust version) consistently estimates the factors and loadings, as long as the signal-to-noise ratio is large enough. The core theoretical challenge is to characterize how idiosyncratic covariance cov(u_i) perturb the eigenstructure of the factor covariance BB^⊤. In addition, the situation is more complicated with the presence of heavy-tailed data, missing data, computational constraints, heterogeneity, etc.

The rest of the paper is devoted to solutions to these challenges and a wide range of applications to statistical machine learning problems. In Section 2, we will elucidate the relationship between factor models and PCA and present several useful deterministic perturbation bounds for eigenspaces. We will also discuss robust covariance inputs for the PCA procedure to guard against corruption from heavy-tailed data. Exploiting the factor structure of the data helps solve many statistical and machine learning problems. In Section 3, we will see how the factor models and PCA can be applied to high-dimensional covariance estimation, regression, multiple testing and model selection. In Section 4, we demonstrate the connection between PCA and a wide range of machine learning problems including Gaussian mixture models, community detection, matrix completion, etc. We will develop useful tools and establish strong theoretical guarantees for our proposed methods.

Here we collect all the notations for future convenience. We use [m] to refer to {1, 2, …, m}. We adopt the convention of using regular letters for scalars and using bold-face letters for vectors or matrices. For $\mathbf{\text{x}}={\left(x_1,\dots ,x_p\right)}^{\top }\in {ℝ}^p$, and 1 ≤ q < ∞, we define ${\Vert \mathbf{\text{x}}\Vert }_q={({\sum }_{j=1}^p{\left|x_j\right|}^q)}^{1/q}$, ${\Vert \mathbf{\text{x}}\Vert }_0=\left|\text{supp}\left(\mathbf{\text{x}}\right)\right|$, where $\text{supp}\left(\mathbf{\text{x}}\right)=\left\{j:x_j\ne 0\right\}$, and ${\Vert \mathbf{\text{x}}\Vert }_{\infty }={\text{max}}_{1\le j\le p}\left|x_j\right|$. For a matrix M, we use ${\Vert \mathbf{\text{M}}\Vert }_2$, ${\Vert \mathbf{\text{M}}\Vert }_F$, ${\Vert \mathbf{\text{M}}\Vert }_{\text{max}}$ and ${\Vert \mathbf{\text{M}}\Vert }_1$ to denote its operator norm (spectral norm), Frobenius norm, entry-wise (element-wise) max-norm, and vector ℓ₁ norm, respectively. To be more specific, the last two norms are defined by ${\Vert \mathbf{\text{M}}\Vert }_{\text{max}}={\text{max}}_{j,k}\left|M_{jk}\right|$ and ${\Vert \mathbf{\text{M}}\Vert }_1={\sum }_{j,k}\left|M_{jk}\right|$. Let I_p denote the p × p identity matrix, 1_p denote the p-dimensional all-one vector, and $1_A$ denote the indicator of event A, i.e., $1_A=1$ if A happens, and 0 otherwise. We use $\mathcal{N}\left(\mathit{\mu },\mathbf{\text{Σ}}\right)$ to refer to the normal distribution with mean vector μ and covariance matrix Σ. For two nonnegative numbers a and b that possibly depend on n and p, we use the notation a = O(b) and $a\lesssim b$ to mean a ≤ C₁b for some constant C₁ > 0, and the notation a = Ω(b) and $a\gtrsim b$ to mean a ≥ C₂b for some constant C₂ > 0. We write $a\asymp b$ if both a = O(b) and a = Ω(b) hold. For a sequence of random variables ${\left\{X_n\right\}}_{n=1}^{\infty }$ and a sequence of nonnegative deterministic numbers ${\left\{a_n\right\}}_{n=1}^{\infty }$, we write $X_n=O_{\mathbb{P}}\left(a_n\right)$ if for any ε > 0, there exists C > 0 and N > 0 such that $\mathbb{P}\left(\left|X_n\right|\ge C{a}_n\right)\le \epsilon$ holds for all n > N; and we write $X_n=o_{\mathbb{P}}\left(a_n\right)$ if for any ε > 0 and C > 0, there exists N > 0 such that $\mathbb{P}\left(\left|X_n\right|\ge C{a}_n\right)\le \epsilon$ holds for all n > N. We omit the subscripts when it does not cause confusion.

2. FACTOR MODELS AND PCA

2.1. Relationship between PCA and factor models in high dimensions

Under model (1.1) with the identifiability condition, Σ = cov(x_i) is given by

$$\mathbf{\text{Σ}}=\mathbf{\text{B}}{\mathbf{\text{B}}}^{\top }+{\mathbf{\text{Σ}}}_u,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathbf{\text{Σ}}}_u={\left({\sigma }_{u,jk}\right)}_{1\le j,k\le p}=\text{cov}\left({\mathbf{\text{u}}}_i\right).$$ (2.1)

Intuitively, if the magnitude of BB^⊤ dominates Σ_u, the top-K eigenspace of Σ should be approximately aligned with the column space of B. Naturally we expect a large gap between the eigenvalues of BB^⊤ and Σ_u to be important for estimating the column space of B through PCA (see Figure 1). On the other hand, if this gap is small compared with the eigenvalues of Σ_u, it is known that PCA leads to inconsistent estimation (Johnstone and Lu, 2009). The above discussion motivates a simple vanilla PCA-based method for estimating B and F as follows (assuming the Identifiability Assumption).

Fig 1.

The left panel is the histogram of the eigenvalue distribution from a synthetic dataset. Fix n = 1000, p = 400 and K = 2 and let all the entries of B be i.i.d. Gaussian $\mathcal{N}\left(0,1/4\right)$. Each entry of F and U is generated from i.i.d. $\mathcal{N}\left(0,1\right)$ and i.i.d. $\mathcal{N}\left(0,5^2\right)$ respectively. The data matrix X is formed according to the factor model (1.1). The right diagram illustrates the Pervasiveness Assumption.

Step 1. Obtain an estimator $\widehat{\mathit{\mu }}$ and $\widehat{\mathbf{\text{Σ}}}$ of μ and Σ, e.g., the sample mean and covariance matrix or their robust versions.

Step 2. Compute the eigen-decomposition of $\widehat{\mathbf{\text{Σ}}}={\sum }_{j=1}^p{\widehat{\lambda }}_j{\widehat{\mathbf{\text{v}}}}_j{\widehat{\mathbf{\text{v}}}}_j^{\top }$. Let ${\left\{{\widehat{\lambda }}_k\right\}}_{k=1}^K$ be the top K eigenvalues and ${\left\{{\widehat{\mathbf{\text{v}}}}_k\right\}}_{k=1}^K$ be their corresponding eigenvectors. Set $\widehat{\mathbf{\text{V}}}=\left({\widehat{\mathbf{\text{v}}}}_1,\dots ,{\widehat{\mathbf{\text{v}}}}_K\right)\in {ℝ}^{p\times K}$ and $\widehat{\mathbf{\text{Λ}}}=\text{diag}\left({\widehat{\lambda }}_1,\dots ,{\widehat{\lambda }}_K\right)\in {ℝ}^{K\times K}$.

Step 3. Obtain PCA estimators $\widehat{\mathbf{\text{B}}}=\widehat{\mathbf{\text{V}}}{\widehat{\mathbf{\text{Λ}}}}^{1/2}$ and $\widehat{\mathbf{\text{F}}}={\left(\mathbf{\text{X}}-\widehat{\mathit{\mu }}{\mathbf{\text{1}}}^{\top }\right)}^{\top }\widehat{\mathbf{\text{V}}}{\widehat{\mathbf{\text{Λ}}}}^{-1/2}$, namely, $\widehat{\mathbf{\text{B}}}$ consists of the top-K rescaled eigenvectors of $\widehat{\mathbf{\text{Σ}}}$ an ${\widehat{\mathbf{\text{f}}}}_i$ is just the rescaled projection of ${\mathbf{\text{x}}}_i-\widehat{\mathit{\mu }}$ onto the space spanned by the eigen-space:${\widehat{\mathbf{\text{f}}}}_i={\widehat{\mathbf{\text{Λ}}}}^{-1/2}{\widehat{\mathbf{\text{V}}}}^T\left({\mathbf{\text{x}}}_i-\widehat{\mathit{\mu }}\right)$.

Let us provide some intuitions for the estimators in Step 3. Recall that b_j is the jth column of B. Then, by model (1.1),${\mathbf{\text{B}}}^{\top }\left({\mathbf{\text{x}}}_i-\mathit{\mu }\right)={\mathbf{\text{B}}}^{\top }\mathbf{\text{B}}{\mathbf{\text{f}}}_i+{\mathbf{\text{B}}}^{\top }{\mathbf{\text{u}}}_i$. In the high-dimensional setting, the second term is averaged out when u_i is weakly dependent across its component. This along with the identifiability condition delivers that

$${\mathbf{\text{f}}}_i\approx \text{diag}{\left({\mathbf{\text{B}}}^{\top }\mathbf{\text{B}}\right)}^{-1}{\mathbf{\text{B}}}^{\top }\left({\mathbf{\text{x}}}_i-\mathit{\mu }\right)=\text{diag}{\left({\Vert {\mathbf{\text{b}}}_1\Vert }^2,\cdots ,{\Vert {\mathbf{\text{b}}}_K\Vert }^2\right)}^{-1}{\mathbf{\text{B}}}^{\top }\left({\mathbf{\text{x}}}_i-\mathit{\mu }\right).$$ (2.2)

Now, we estimate ${\sum }_{j=1}^K{\widehat{\lambda }}_j{\widehat{\mathbf{\text{v}}}}_j{\widehat{\mathbf{\text{v}}}}_j^{\top }$ and hence b_j by ${\widehat{\lambda }}_j^{1/2}{\widehat{\mathbf{\text{v}}}}_j$ and ${\Vert {\mathbf{\text{b}}}_j\Vert }^2$ by ${\widehat{\lambda }}_j$. Using the substitution method, we obtain the estimators in Step 3.

The above heuristic also reveals that the PCA-based methods work well if the effect of the factors outweighs the noise. To quantify this, we introduce a form of Pervasiveness Assumption from the factor model literature. While this assumption is strong¹, it simplifies our discussion and captures the above intuition well: it holds when the factor loadings ${\left\{{\mathbf{\text{b}}}_j\right\}}_{j=1}^p$ are random samples from a nondegenerate population (Fan et al., 2013).

Assumption 2.1 (Pervasiveness). The first K eigenvalues of BB^⊤ have order Ω(p), whereas ${\Vert {\mathbf{\text{Σ}}}_u\Vert }_2=O\left(1\right)$.

Note that cov(f_i) = I_K under the Identifiability Assumption 1.1. The first part of this assumption holds when each factor influences a non-vanishing proportion of outcomes. Mathematically speaking, it means that for any k ∈ [K] := {1, 2, …, K}, the average of squared loadings of the kth factor satisfies $p^{-1}{\sum }_{j=1}^p{B}_{jk}^2=\Omega \left(1\right)$ (right panel of Figure 1). This holds with high probability if, for example, ${\left\{B_{jk}\right\}}_{j=1}^p$ are i.i.d. realizations from a non-degenerate distribution, but we will not make such assumption in this paper. The second part of the assumption is reasonable, as cross-sectional correlation becomes weak after we take out the common factors. Typically, if Σ_u is a sparse matrix, the norm bound ${\Vert {\mathbf{\text{Σ}}}_u\Vert }_2=O\left(1\right)$ holds; see Section 3.1 for details. Under this Pervasiveness Assumption, the first K eigenvalues of Σ will be well separated with the rest of eigenvalues. By the Davis-Kahan theorem (Davis and Kahan, 1970), which we present as Theorem 2.1, we can consistently estimate the column space of B through the top-K eigenspace of Σ. This explains why we can apply PCA to factor model analysis (Fan et al., 2013).

Though factor models and PCA are not identical (see Jolliffe, 1986), they are approximately the same for high-dimensional problems with the pervasiveness assumption(Fan et al., 2013). Thus, PCA-based ideas are important components of estimation and inference for factor models. In later sections (especially Section 4), we discuss statistical and machine learning problems with factor-model-type structures. There PCA is able to achieve consistent estimation even when the Pervasiveness Assumption is weakened—and somewhat surprisingly—PCA can work well down to the information limit. For perspectives from random matrix theory, see Baik et al. (2005); Paul (2007); Johnstone and Lu (2009); Benaych-Georges and Nadakuditi (2011); O’Rourke et al. (2016); Wang and Fan (2017), among others.

2.2. Estimating the number of factors

In high-dimensional factor models, if the factors are unobserved, we need to choose the number of factors K before estimating the loading matrix, factors, etc. The number K can be usually estimated from the eigenvalues of the the sample covariance matrix or its robust version. With certain conditions such as separation of the top K eigenvalues from the others, the estimation is consistent. Classical methods include likelihood ratio tests (Bartlett, 1950), the scree plot (Cattell, 1966), parallel analysis (Horn, 1965), etc. Here, we introduce a few recent methods: the first one is based on the eigenvalue ratio, the second on eigenvalue differences, and the third on the eigenvalue magnitude.

For simplicity, let us use the sample covariance and arrange its eigenvalues in descending order: λ₁ ≥ λ₂ ≥ ⋯ ≥ λ_n∧p, where n ∧ p = min{n, p} (the remaining eigenvalues, if any, are zero). Lam and Yao (2012) and Ahn and Horenstein (2013) proposed an estimator ${\widehat{K}}_1$ based on ratios of consecutive eigenvalues. For a pre-determined k_max, the eigenvalue ratio estimator is

$${\widehat{K}}_1=\underset{i\le k_{\text{max}}}{\text{argmax}}\frac{{\lambda }_i}{{\lambda }_{i+1}}.$$

Intuitively, when the signal eigenvalues are well separated from the other eigenvalues, the ratio at k = K should be large. Under some conditions, the consistency of this estimator, which does not involve complicated tuning parameters, is established.

In an earlier work, Onatski (2010) proposed to use the differences of consecutive eigenvalues. For a given δ > 0 and pre-determined integer k_max, define

$${\widehat{K}}_2\left(\delta \right)=\text{max}\left\{i\le k_{\text{max}}:{\lambda }_i-{\lambda }_{i+1}\ge \delta \right\}.$$

Using a result on eigenvalue empirical distribution from random matrix theory, Onatski (2010) proved consistency of ${\widehat{K}}_2\left(\delta \right)$ under the Pervasiveness Assumption. The intuition is that, the Pervasiveness Assumption implies that λ_K − λ_K+1 tends on ∞ in probability as n → ∞; whereas λ_i − λ_i+1 → 0 almost surely for K < i < k_max because these λ_i-s converge to the same limit, which can be determined using random matrix theory. Onatski (2010) also proposed a data-driven way to determine δ from the empirical eigenvalue distribution of the sample covariance matrix.

A third possibility is to use an information criterion. Define

$$V\left(k\right)=\frac{1}{np}\underset{\widehat{\mathbf{\text{B}}}\in {ℝ}^{p\times k},\widehat{\mathbf{\text{F}}}\in {ℝ}^{n\times k}}{\text{min}}{\Vert \mathbf{\text{X}}-\widehat{\mathit{\mu }}{\mathbf{\text{1}}}_n^{\top }-\widehat{\mathbf{\text{B}}}{\widehat{\mathbf{\text{F}}}}^{\top }\Vert }_F^2=p^{-1}\sum _{j>k}{\lambda }_j,$$

where $\widehat{\mathit{\mu }}$ is the sample mean, and the equivalence (second equality) is well known. For a given k, V (k) is interpreted as the scaled sum of squared residuals, which measures how well k factors fit the data. A very natural estimator ${\widehat{K}}_3$ is to find the best k ≤ k_max such that the following penalized version of V (k) is minimized (Bai and Ng, 2002):

$$PC\left(k\right)=V\left(k\right)+k{\widehat{\sigma }}^{2}g\left(n,p\right),\hspace{0.17em}\hspace{1em}\text{where}\hspace{1em}g\left(n,p\right):=\frac{n+p}{np}\text{log}\left(\frac{np}{n+p}\right),$$

and ${\widehat{\sigma }}^2$ is any consistent estimate ${\left(np\right)}^{-1}{\sum }_{i=1}^n{\sum }_{j=1}^d\mathbb{E}{u}_{ji}^2$. The upper limit k_max is assumed to be no smaller than K, and is typically chosen as 8 or 15 in empirical studies in Bai and Ng (2002). Consistency results are established under more general choices of g(n, p).

We conclude this section by remarking that in general, it is impossible to consistently estimate K if the smallest nonzero eigenvalue B^⊤B is much smaller than ${\Vert {\mathbf{\text{Σ}}}_u\Vert }_2$, because the ‘signals’ (eigenvalues of B^⊤B) would not be distinguishable from the the noise (eigenvalues of UU^⊤). As mentioned before, consistency of PCA is well studied in the random matrix theory literature. See Dobriban (2017) for a recent work that justifies parallel analysis using random matrix theory.

2.3. Robust covariance inputs

To extract latent factors and their factor loadings, we need an initial covariance estimator. Given independent observations x₁, …, x_n with mean zero, the sample covariance matrix, namely ${\widehat{\mathbf{\text{Σ}}}}_{\text{sam}}:=n^{-1}{\sum }_{i=1}^n{\mathbf{\text{x}}}_i{\mathbf{\text{x}}}_i^{\top }$, is a natural choice to estimate $\mathbf{\text{Σ}}\in {ℝ}^{p\times p}$. The finite sample bound on ${\Vert {\widehat{\mathbf{\text{Σ}}}}_{\text{sam}}-\mathbf{\text{Σ}}\Vert }_2$ has been well studied in the literature (Vershynin, 2010; Tropp, 2012; Koltchinskii and Lounici, 2017). Before presenting the result from Vershynin (2010), let us review the definition of sub-Gaussian variables.

A random variable ξ is called sub-Gaussian if ${\Vert \xi \Vert }_{{\psi }_2}\equiv {\text{sup}}_{q\ge 1}{q}^{-1/2}{\left(\mathbb{E}{\left|\xi \right|}^q\right)}^{1/q}$ is finite, in which case this quantity defines a norm ${\Vert \cdot \Vert }_{{\psi }_2}$ called the sub-Gaussian norm. Sub-Gaussian variables include as special cases Gaussian variables, bounded variables, and other variables with tails similar to or lighter than Gaussian tails. For a random vector ξ, we define ${\Vert \mathbf{\xi }\Vert }_{{\psi }_2}:={\text{sup}}_{{\Vert \mathbf{\text{v}}\Vert }_2=1}{\Vert {\mathbf{\xi }}^{\top }\mathbf{\text{v}}\Vert }_{{\psi }_2}$; we call ξ sub-Gaussian if ${\Vert \mathbf{\xi }\Vert }_{{\psi }_2}$ is finite.

Theorem 2.1. Let Σ be the covariance matrix of x_i. Assume that ${\left\{{\mathbf{\text{Σ}}}^{-\frac{1}{2}}{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$ are i.i.d. sub-Gaussian random vectors, and denote $\kappa ={\text{sup}}_{{\Vert \mathbf{\text{v}}\Vert }_2=1}{\Vert {\mathbf{\text{x}}}_i^{\top }\mathbf{\text{v}}\Vert }_{{\psi }_2}$. Then for any t ≥ 0, there exist constants C and c only depending on κ such that

$$\mathbb{P}\left({\Vert {\widehat{\mathbf{\text{Σ}}}}_{\text{sam}}-\mathbf{\text{Σ}}\Vert }_2\ge \text{max}\left(\delta ,{\delta }^2\right){\Vert \mathbf{\text{Σ}}\Vert }_2\right)\le 2\text{exp}\left(-c{t}^2\right),$$ (2.3)

where $\delta =C\sqrt{p/n}+t/\sqrt{n}$.

Remark 2.1. The spectral-norm bound above depends on the ambient dimension p, which can be large in high-dimensional scenarios. Interested readers can refer to Koltchinskii and Lounici (2017) for a refined result that only depends on the intrinsic dimension (or effective rank) of Σ.

An important asepect of the above result is the sub-Gaussian concentration in (2.3), but this depends heavily on the sub-Gaussian or sub-exponential behaviors of observed random vectors. This condition can not be validated in high dimensions when tens of thousands of variables are collected. See Fan et al. (2016b). When the distribution is heavy-tailed², one cannot expect sub-Gaussian or sub-exponential behaviors of the sample covariance in the spectral norm (Catoni, 2012). See also Vershynin (2012) and Srivastava and Vershynin (2013). Therefore, to perform PCA for heavy-tailed data, the sample covariance is not a good choice to begin with. Alternative robust estimators have been constructed to achieve better finite sample performance.

Catoni (2012), Fan et al. (2017b) and Fan et al. (2016b) approached the problem by first considering estimation of a univariate mean μ from a sequence of i.i.d random variables X₁, ⋯, X_n with variance σ². In this case, the sample mean $\overline{X}$ provides an estimator but without exponential concentration. Indeed, by Markov inequality, we have $\mathbb{P}\left(\left|\overline{X}-\mu \right|\ge t\sigma /\sqrt{n}\right)\le t^{-2}$, which is tight in general and has a Cauchy tail (in terms of t). On the other hand, if we truncate the data ${\tilde{X}}_i=\text{sign}\left(X_i\right)\text{min}\left(\left|X_i\right|,\tau \right)$ with $\tau \asymp \sigma \sqrt{n}$ and compute the mean of the truncated data, then we have (Fan et al., 2016b)

$$\mathbb{P}\left(\left|\frac{1}{n}\sum _{i=1}^n{\tilde{X}}_i-\mu \right|\ge t\frac{\sigma }{\sqrt{n}}\right)\le 2\text{exp}\left(-c{t}^2\right),$$

for a universal constant c > 0. In other words, the mean of truncated data with only a finite second moment behaves very much the same as the sample mean from the normal data: both estimators have Gaussian tails (in terms of t). This sub-Gaussian concentration is fundamental in high-dimensional statistics as the sample mean is computed tens of thousands or even millions of times.

As an example, estimating the high-dimensional covariance matrix Σ = (σ_ij) involves O(p²) univariate mean estimation, since the covariance can be expressed as an expectation: as ${\sigma }_{ij}=\mathbb{E}\left(X_i{X}_j\right)-\mathbb{E}\left(X_i\right)\mathbb{E}\left(X_j\right)$. Estimating each component by the truncated mean yields a covariance matrix $\widetilde{\mathbf{\text{Σ}}}$. Assuming the fourth moment is bounded (as the covariance itself are second moments), by using the union bound and the above concentration inequality, we can easily obtain

$$\mathbb{P}\left({\Vert \widetilde{\mathbf{\text{Σ}}}-\mathbf{\text{Σ}}\Vert }_{\text{max}}\ge \sqrt{\frac{a\text{log}p}{c^{\prime }n}}\right)\lesssim p^{2-a}$$

for any a > 0 and a constant c′ > 0. In other words, with truncation, when the data have merely bounded fourth moments, we can achieve the same estimation rate as the sample covariance matrix under the Gaussian data.

Fan et al. (2016b) and Minsker (2016) independently proposed shrinkage variants of the sample covariance with sub-Gaussian behavior under the spectral norm, as long as the fourth moments of X are finite. For any $\tau \in {ℝ}^{+}$, Fan et al. (2016b) proposed the following shrinkage sample covariance matrix

$${\widehat{\mathbf{\text{Σ}}}}_s\left(\tau \right)=\frac{1}{n}\sum _{i=1}^n{\widetilde{\mathbf{\text{x}}}}_i{\widetilde{\mathbf{\text{x}}}}_i^{\top },\hspace{1em}\hspace{0.17em}\hspace{0.17em}{\widetilde{\mathbf{\text{x}}}}_i:=\left({\Vert {\mathbf{\text{x}}}_i\Vert }_4\wedge \tau \right){\mathbf{\text{x}}}_i/{\Vert {\mathbf{\text{x}}}_i\Vert }_4,$$ (2.4)

to estimate Σ, where ${\Vert \cdot \Vert }_4$ is the ℓ₄-norm. The following theorem establishes the statistical error rate of ${\widetilde{\mathbf{\text{Σ}}}}_s\left(\tau \right)$ in terms of the spectral norm.

Theorem 2.2. Suppose $\mathbb{E}{\left({\mathbf{\text{v}}}^{\top }{\mathbf{\text{x}}}_i\right)}^4\le R$ for any unit vector $\mathbf{\text{v}}\in {\mathcal{S}}^{p-1}$. Then it holds that for any δ > 0,

$$\mathbb{P}({\Vert {\widehat{\mathbf{\text{Σ}}}}_s\left(\tau \right)-\mathbf{\text{Σ}}\Vert }_2\ge \sqrt{\frac{\delta Rp\text{log}p}{n}})\le p^{1-C\delta },$$ (2.5)

where $\tau \asymp {\left(nR/\left(\delta \text{log}p\right)\right)}^{1/4}$ and C is a universal constant.

Applying PCA to the robust covariance estimators as described above leads to more reliable estimation of principal eigenspaces in the presence of heavy-tailed data.

In Theorem 2.2, we assume that the mean of x_i is zero. When this does not hold, a natural estimator of $\mathbf{\text{Σ}}=\frac{1}{2}\mathbb{E}\left({\mathbf{\text{x}}}_1-{\mathbf{\text{x}}}_2\right){\left({\mathbf{\text{x}}}_1-{\mathbf{\text{x}}}_2\right)}^{\top }$ is to use the shrunk U-statistic (Fan et al., 2017a):

$$\begin{gathered} {\widehat{\mathbf{\text{Σ}}}}_U\left(\tau \right)=\frac{1}{2\left(\begin{array}{l}n\\ 2\end{array}\right)}\sum _{j\ne k}\frac{{\psi }_{\tau }\left({\Vert {\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\Vert }_2^2\right)}{{\Vert {\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\Vert }_2^2}\left({\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\right){\left({\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\right)}^{\top } \\ =\frac{1}{2\left(\begin{array}{l}n\\ 2\end{array}\right)}\sum _{j\ne k}\text{min}\left(1,\tau /{\Vert {\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\Vert }_2^2\right)\left({\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\right){\left({\mathbf{\text{x}}}_j-{\mathbf{\text{x}}}_k\right)}^{\top }, \end{gathered}$$

where ${\psi }_{\tau }\left(u\right)=\left(\left|u\right|\wedge \tau \right)\text{sign}\left(u\right)$. When τ = ∞, it reduces to the usual U-statistics. It possesses a similar concentration property to that in Theorem 2.2 with a proper choice of τ.

2.4. Perturbation bounds

In this section, we introduce several perturbation results on eigenspaces, which serve as fundamental technical tools in factor models and related learning problems. For example, in relating the factor loading matrix B to the principal components of covariance matrix Σ in (2.1), one can regard Σ as a perturbation of BB^⊤ by an amount of Σ_u and take A = BB^⊤ and $\widetilde{\mathbf{\text{A}}}=\mathbf{\text{Σ}}$ in Theorem 2.3 below. Similarly, we can also regard a covariance matrix estimator $\widehat{\mathbf{\text{Σ}}}$ as a perturbation of Σ by an amount of $\widehat{\mathbf{\text{Σ}}}-\mathbf{\text{Σ}}$.

We will begin with a review of the Davis-Kahan theorem (Davis and Kahan, 1970), which is usually useful for deriving ℓ₂-type bounds (which includes spectral norm bounds) for symmetric matrices. Then, based on this classical result, we introduce entry-wise (ℓ_∞) bounds, which typically give refined results under structural assumptions. We also derive bounds for rectangular matrices that are similar to Wedin’s theorem (Wedin, 1972). Several recent works on this topic can be found in Yu et al. (2014); Fan et al. (2018b); Koltchinskii and Xia (2016); Abbe et al. (2017); Zhong (2017); Cape et al. (2017); Eldridge et al. (2017).

First, for any two subspaces $\mathcal{S}$ and $\tilde{\mathcal{S}}$ of the same dimension K in ${ℝ}^p$, we choose any V, $\widetilde{\mathbf{\text{V}}}\in {ℝ}^{p\times K}$ with orthonormal columns that span $\mathcal{S}$ and $\tilde{\mathcal{S}}$, respectively. We can measure the closeness between two subspaces though the difference between their projectors:

$$d_2\left(\mathcal{S},\tilde{\mathcal{S}}\right)={\Vert \widetilde{\mathbf{\text{V}}}{\widetilde{\mathbf{\text{V}}}}^{\top }-\mathbf{\text{V}}{\mathbf{\text{V}}}^{\top }\Vert }_2\hspace{1em}\text{or}\hspace{1em}{\mathbf{\text{d}}}_F\left(\mathcal{S},\tilde{\mathcal{S}}\right)={\Vert \widetilde{\mathbf{\text{V}}}{\widetilde{\mathbf{\text{V}}}}^{\top }-\mathbf{\text{V}}{\mathbf{\text{V}}}^{\top }\Vert }_F.$$

The above definitions are both proper metrics (or distances) for subspaces S and $\widetilde{\mathbf{\text{S}}}$ and do not depend on the specific choice of V and $\widetilde{\mathbf{\text{V}}}$, since $\widetilde{\mathbf{\text{V}}}{\widetilde{\mathbf{\text{V}}}}^{\top }$ and VV^⊤ are projection operators. Importantly, these two metrics are connected to the well-studied notion of canonical angles (or principal angles). Formally, let the singular values of ${\widetilde{\mathbf{\text{V}}}}^{\top }\mathbf{\text{V}}$ be ${\left\{{\sigma }_k\right\}}_{k=1}^K$, and define the canonical angles θ_k = cos⁻¹ σ_k for k = 1, …, K. It is often useful to denote the sine of the canonical (principal) angles by $\text{sin}\Theta \left(\widehat{\mathbf{\text{V}}},\mathbf{\text{V}}\right):=\text{diag}\left(\text{sin}{\theta }_1,\dots ,\text{sin}{\theta }_K\right)\in {ℝ}^{K\times K}$, which can be interpreted as a generalization of sine of angles between two vectors. The following identities are well known (Stewart and Sun, 1990).

$${\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert }_2=d_2\left(\mathcal{S},\tilde{\mathcal{S}}\right),\hspace{1em}\sqrt{2}{\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert }_F=d_F\left(\mathcal{S},\tilde{\mathcal{S}}\right).$$

In some cases, it is convenient to fix a specific choice of $\widetilde{\mathbf{\text{V}}}$ and V. It is known that for both Frobenius norm and spectral norm,

$$\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert \le \underset{\mathbf{\text{R}}\in \mathcal{O}\left(K\right)}{\text{min}}\Vert \widetilde{\mathbf{\text{V}}}\mathbf{\text{R}}-\mathbf{\text{V}}\Vert \le \sqrt{2}\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert ,$$

where $\mathcal{O}\left(K\right)$ is the space of orthogonal matrices of size K × K. The minimizer (best rotation of basis) can be given by the singular value decomposition (SVD) of ${\widetilde{\mathbf{\text{V}}}}^{\top }\mathbf{\text{V}}$. For details, see Cape et al. (2017) for example.

Now, we present the Davis-Kahan sin θ theorem (Davis and Kahan, 1970).

Theorem 2.3. Suppose A, $\widetilde{\mathbf{\text{A}}}\in {ℝ}^{n\times n}$ are symmetric, and that V, $\widetilde{\mathbf{\text{V}}}\in {ℝ}^{n\times K}$ have orthonormal column vectors which are eigenvectors of A and $\widetilde{\mathbf{\text{A}}}$ respectively. Let $\mathcal{L}\left(\mathbf{\text{V}}\right)$ be the set of eigenvalues corresponding to the eigenvectors given in V, and let $\mathcal{L}\left({\mathbf{\text{V}}}^{\perp }\right)$ (respectively $\mathcal{L}\left({\widetilde{\mathbf{\text{V}}}}^{\perp }\right)$) be the set of eigenvalues corresponding to the eigenvectors not given in V (respectively $\widetilde{\mathbf{\text{V}}}$). If there exists an interval [α, β] and δ > 0 such that $\mathcal{L}\left(\mathbf{\text{V}}\right)\subset \left[\alpha ,\beta \right]$ and $\mathcal{L}\left({\widetilde{\mathbf{\text{V}}}}^{\perp }\right)\subset \left(-\infty ,\alpha -\delta \right]\cup \left[\beta +\delta ,+\infty \right)$, then for any orthogonal-invariant norm³

$$\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert \le {\delta }^{-1}\Vert \left(\widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\right)\mathbf{\text{V}}\Vert .$$

This theorem can be generalized to singular vector perturbation for rectangular matrices; see Wedin (1972). A slightly unpleasant feature of this theorem is that δ depends on the eigenvalues of both A and $\widetilde{\mathbf{\text{A}}}$. However, with the help of Weyl’s inequality, we can immediately obtain a corollary that does not involve the eigenvalues of $\widetilde{\mathbf{\text{A}}}$. Let λ_j(·) denote the jth largest eigenvalue of a real symmetric matrix. Recall that Weyl’s inequality bounds the differences between the eigenvalues of A and $\widetilde{\mathbf{\text{A}}}$:

$$\underset{1\le j\le n}{\text{max}}\left|{\lambda }_j\left(\widetilde{\mathbf{\text{A}}}\right)-{\lambda }_j\left(\mathbf{\text{A}}\right)\right|\le {\Vert \widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\Vert }_2.$$ (2.6)

This inequality suggests that, if the eigenvalues in $\mathcal{L}\left({\widetilde{\mathbf{\text{V}}}}^{\perp }\right)$ have the same ranks (in descending order) as those in $\mathcal{L}\left({\mathbf{\text{V}}}^{\perp }\right)$, then $\mathcal{L}\left({\widetilde{\mathbf{\text{V}}}}^{\perp }\right)$ and $\mathcal{L}\left({\mathbf{\text{V}}}^{\perp }\right)$ are similar. Below we state our corollary, whose proof is in the appendix.

Corollary 2.1. Assume the setup of the above theorem, and suppose the eigenvalues in $\mathcal{L}\left(\widetilde{\mathbf{\text{V}}}\right)$ have the same ranks as those in $\mathcal{L}\left(\mathbf{\text{V}}\right)$. If $\mathcal{L}\left(\mathbf{\text{V}}\right)\subset \left[\alpha ,\beta \right]$ and $\mathcal{L}\left({\mathbf{\text{V}}}^{\perp }\right)\subset \left(-\infty ,\alpha -{\delta }_0\right]\cup \left[\beta +{\delta }_0,+\infty \right)$ for some δ₀ > 0, then

$${\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert }_2\le 2{\delta }_0^{-1}{\Vert \widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\Vert }_2.$$

We can then use ${\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert }_F\le \sqrt{K}{\Vert \text{sin}\mathbf{\text{Θ}}\left(\widetilde{\mathbf{\text{V}}},\mathbf{\text{V}}\right)\Vert }_2$ to obtain a bound under the Frobenius norm. In the special case where $\mathcal{L}\left(\mathbf{\text{V}}\right)=\left\{\lambda \right\}$ and V = v, $\widetilde{\mathbf{\text{V}}}=\widetilde{\mathbf{\text{v}}}$ reduce to vectors, we can choose α = β = λ, and the above corollary translates into

$$\underset{s\in \left\{\pm 1\right\}}{\text{min}}{\Vert \widehat{\mathbf{\text{v}}}-s\mathbf{\text{v}}\Vert }_2\le \sqrt{2}\text{sin}\theta \left(\widehat{\mathbf{\text{v}}},\mathbf{\text{v}}\right)\le 2\sqrt{2}{\delta }_0^{-1}{\Vert \widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\Vert }_2.$$ (2.7)

We can now see that the factor model and PCA are approximately the same with sufficiently large eigen-gap. Indeed, under Identifiability Assumption 1.1, we have $\mathbf{\text{Σ}}=\mathbf{\text{B}}{\mathbf{\text{B}}}^{\top }+{\mathbf{\text{Σ}}}_u$. Applying Weyl’s inequality and Corollary 2.1 to BB^⊤ (as A) and Σ (as $\widetilde{\mathbf{\text{A}}}$), we can easily control the eigenvalue/eigenvector differences by ${\Vert {\mathbf{\text{Σ}}}_u\Vert }_2$ and the eigengap, which is comparably small under Pervasiveness Assumption 2.1. This difference can be interpreted as the bias incurred by PCA on approximating factor models.

Furthermore, given any covariance estimator $\widehat{\mathbf{\text{Σ}}}$, we can similarly apply the above results by setting A = Σ and $\widetilde{\mathbf{\text{A}}}=\widehat{\mathbf{\text{Σ}}}$ to bound the difference between the estimated eigenvalues/eigenvectors and the population counterparts. Note that the above corollary gives us an upper bound on the subspace estimation error in terms of the ratio ${\Vert \widehat{\mathbf{\text{Σ}}}-\mathbf{\text{Σ}}\Vert }_2/{\delta }_0$.

Next, we consider entry-wise bounds on the eigenvectors. For simplicity, here we only consider eigenvectors corresponding to unique eigenvalues rather than the general eigenspace. Often, we want to have a bound on each entry of the eigenvector difference $\widetilde{\mathbf{\text{v}}}-\mathbf{\text{v}}$, instead of an ℓ₂ norm bound, which is an average-type result. In many cases, none of these entries has dominant perturbation, but the Davis-Kahan’s theorem falls short of providing a reasonable bound (the naïve bound ${\Vert \cdot \Vert }_{\infty }\le {\Vert \cdot \Vert }_2$ gives a suboptimal result).

Some recent papers (Abbe et al., 2017) have addressed this problem, and in particular, entry-wise bounds of the following form are established.

$$\left|{\left[\widetilde{\mathbf{\text{v}}}-\mathbf{\text{v}}\right]}_m\right|\lesssim \mu \frac{{\Vert \widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\Vert }_2}{{\delta }_0}+\text{small term},\hspace{1em}m\hspace{0.17em}\in \left[n\right],$$

where μ ∈ [0, 1] is related to the structure of the statistical problem and typically can be as small as $O\left(1/\sqrt{n}\right)$, which is very desirable in high-dimensional setting. The small term is often related to independence pattern of the data, which is typically small under mild independence conditions.

We illustrate this idea in Figure 2 through a simulated data example (left) and a real data example (right), both of which have factor-type structure. For the left plot, we generated a network data according to the stochastic block model with K = 2 blocks (communities), each having nodes n/2 = 2500: the adjacency matrix that represents the links between nodes is a symmetric matrix, with upper triangular elements generated independently from Bernoulli trials (diagonal elements are taken as 0), with the edge probability 5 log n/n for two nodes within blocks and log n/(4n) otherwise. Our task is to classify (cluster) these two communities based on the adjacency matrix. We used the second eigenvector ${\mathbf{\text{v}}}_2\in {ℝ}^{5000}$ (that is, corresponding to the second largest eigenvalue) of the adjacency matrix as a classifier. The left panel of Figure 2 represents the values of the 5000 coordinates (or entries) [v₂]_i in the y-axis against the indices i = 1, …, 5000 in the x-axis. For comparison, the second eigenvector ${\mathbf{\text{v}}}_2^{*}\in {ℝ}^{5000}$ of the expectation of the adjacency matrix—which is of interest but unknown—have entries taking values only in $\left\{\pm 1/\sqrt{5000}\right\}$, depending on the unknown nature of which block a vertex belongs to (this statement is not hard to verify). We used the horizontal line to represent these ideal values: they indicate exactly the membership of each vertex. Clearly, the magnitude of entry-wise perturbation is $O\left(1/\sqrt{n}\right)$. Therefore, we can use $\text{sign}\left({\mathbf{\text{v}}}_2\right)/\sqrt{5000}$ as an estimate of v* and classify all nodes with the same sign as the same community. See Section 4.2 for more details.

Fig 2.

The left plot shows the entries (coordinates) of the second eigenvectors v₂ computed from the adjacency matrix from the SBM with two equal-sized blocks (n = 5000, K = 2). The plot also shows the expectation counterpart ${\mathbf{v}}_2^{\ast }$, whose entries have the same magnitude $O\left(1/\sqrt{n}\right)$. The deviation of v₂ from ${\mathbf{v}}_2^{\ast }$ is quite uniform, which is a phenomenon not captured by the Davis-Kahan’s theorem. The right plot shows the coordinates of two leading eigenvectors of the sample covariance matrix calculated from 2012–2017 daily return data of 484 stocks (tiny black dots). We also highlight six stocks during three time windows (2012–2015, 2013–2016, 2014–2017) with big markers, so that the fluctuation/perturbation is shown. The magnitude of these coordinates is typically small, and the fluctuation is also small.

For the right plot, we used daily return data of stocks that are constituents of S&P 500 index from 2012.1.1–2017.12.31. We considered stocks with exactly n = 1509 records and excluded stocks with incomplete/missing values, which resulted in p = 484 stocks. Then, we calculated the sample covariance matrix ${\widehat{\mathbf{\text{Σ}}}}_{\text{sam}}\in {ℝ}^{p\times p}$ using the data in the entire period, and computed two leading eigenvectors (note that they span the column space of B) and plotted the coordinates (entries) using small dots. Stocks with an coordinate smaller than 5% quantile or larger than 95% quantile are potentially outlying values and are not shown in the plot. In addition, we also highlighted the fluctuation of six stocks during three time windows: 2012.1–2015.12, 2013.1–2016.12 and 2014.1–2017.12, with different big markers. That is, for each of the three time windows, we re-computed the covariance matrices and the two leading eigenvectors, and then highlighted coordinates that correspond to the six major stocks. Clearly, the magnitude for these stocks is small, which is roughly $O\left(1/\sqrt{p}\right)$, and the fluctuation of coordinates is also very small. Both plots suggest an interesting phenomenon of eigenvectors in high dimensions: entry-wise behavior of eigenvectors can be benign under factor model structure.

To state our results rigorously, let us suppose that A, $\widetilde{\mathbf{\text{A}}}$, $\mathbf{\text{W}}\in {ℝ}^{n\times n}$ are symmetric matrices, with $\widetilde{\mathbf{\text{A}}}=\mathbf{\text{A}}+\mathbf{\text{W}}$ and rank(A) = K < n. Let the eigen-decomposition of A and $\widetilde{\mathbf{\text{A}}}$ be

$$\mathbf{\text{A}}=\sum _{k=1}^K{\lambda }_k{\mathbf{\text{v}}}_k{\mathbf{\text{v}}}_k^{\top },\hspace{1em}\text{and}\hspace{1em}\widetilde{\mathbf{\text{A}}}=\sum _{k=1}^K{\tilde{\lambda }}_k{\widetilde{\mathbf{\text{v}}}}_k{\widetilde{\mathbf{\text{v}}}}_k^{\top }+\sum _{k=K+1}^n{\tilde{\lambda }}_k{\widetilde{\mathbf{\text{v}}}}_k{\widetilde{\mathbf{\text{v}}}}_k^{\top }.$$ (2.8)

Here the eigenvalues ${\left\{{\lambda }_k\right\}}_{k=1}^K$ and ${\left\{{\tilde{\lambda }}_k\right\}}_{k=1}^K$ are the K largest ones of A and $\widetilde{\mathbf{\text{A}}}$, respectively, in terms of absolute values. Both sequences are sorted in descending order. ${\left\{{\tilde{\lambda }}_k\right\}}_{k=K+1}^K$ are eigenvalues of $\widetilde{\mathbf{\text{A}}}$ whose absolute values are smaller than $\left|{\tilde{\lambda }}_K\right|$. The eigenvectors ${\left\{{\mathbf{\text{v}}}_k\right\}}_{k=1}^K$ and ${\left\{{\widetilde{\mathbf{\text{v}}}}_k\right\}}_{k=1}^n$ are normalized to have unit norms.

Here ${\left\{{\lambda }_k\right\}}_{k=1}^K$ are allowed to take negative values. Thanks to Weyl’s inequality, ${\left\{{\tilde{\lambda }}_k\right\}}_{k=1}^K$ and ${\left\{{\tilde{\lambda }}_k\right\}}_{k=K+1}^n$ are well-separated when the size of perturbation W is not too large. In addition, we have the freedom to choose signs for eigenvectors, since they are not uniquely defined. Later, we will use ‘up to sign’ to signify that our statement is true for at least one choice of sign. With the conventions λ₀ = +∞ and λ_K+1 = −∞, we define the eigen-gap as

$${\delta }_k=\text{min}\left\{{\lambda }_{k-1}-{\lambda }_k,{\lambda }_k-{\lambda }_{k+1},\left|{\lambda }_k\right|\right\},\hspace{1em}\forall k\in \left[K\right],$$ (2.9)

which is the smallest distance between λ_k and other eigenvalues (including 0). This definition coincides with the (usual) eigen-gap in Corollary 2.1 in the special case $\mathcal{L}\left({\mathbf{\text{v}}}_k\right)=\left\{{\lambda }_k\right\}$ where we are interested in a single eigenvalue and its associated eigenvector.

We now present an entry-wise perturbation result. Let us first look at only one eigenvector. In this case, when $\Vert \widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\Vert$ is small, heuristically,

$${\widetilde{\mathbf{\text{v}}}}_k=\frac{\widetilde{\mathbf{\text{A}}}{\widetilde{\mathbf{\text{v}}}}_k}{{\tilde{\lambda }}_k}\approx \frac{\widetilde{\mathbf{\text{A}}}{\mathbf{\text{v}}}_k}{{\lambda }_k}={\mathbf{\text{v}}}_k+\frac{\left(\widetilde{\mathbf{\text{A}}}-\mathbf{\text{A}}\right){\mathbf{\text{v}}}_k}{{\lambda }_k}$$

holds uniformly for each entry. When $\mathbf{\text{A}}=\mathbb{E}\widetilde{\mathbf{\text{A}}}$, that is, $\widetilde{\mathbf{\text{A}}}$ is unbiased, this gives the first-order approximation (rather than bounds on the difference ${\widetilde{\mathbf{\text{v}}}}_k-{\mathbf{\text{v}}}_k$) of the random vector ${\widetilde{\mathbf{\text{v}}}}_k$. Abbe et al. (2017) proves rigorously this result and generalizes to eigenspaces. The key technique for the proof is similar to Theorem 2.4 below, which simplifies the one in Abbe et al. (2017) in various ways but holds under more general conditions. It is stated in a deterministic way, and can be powerful if there is certain structural independence in the perturbation matrix W. A self-contained proof can be found in the appendix.

For each m ∈ [n], let ${\mathbf{\text{W}}}^{\left(m\right)}\in {ℝ}^{n\times n}$ be a modification of W with the mth row and mth column zeroed out, i.e.,

$$W_{ij}^{\left(m\right)}=W_{ij}{1}_{\left\{i\ne m\right\}}{1}_{\left\{j\ne m\right\}},\hspace{1em}\forall i,j\in \left[n\right].$$

We also define ${\widetilde{\mathbf{\text{A}}}}^{\left(m\right)}=\mathbf{\text{A}}+{\mathbf{\text{W}}}^{\left(m\right)}$, and denote its eigenvalues and eigenvectors by ${\left\{{\tilde{\lambda }}_k^{\left(m\right)}\right\}}_{k=1}^n$ and ${\left\{{\widetilde{\mathbf{\text{v}}}}_k^{\left(m\right)}\right\}}_{k=1}^n$, respectively. This construction is related to the leave-one-out technique in probability and statistics. For recent papers using this technique, see Bean et al. (2013); Zhong and Boumal (2018); Abbe et al. (2017) for example.

Theorem 2.4. Fix any ℓ ∈ [K]. Suppose that $\left|{\lambda }_{\ell }\right|\asymp {\text{max}}_{k\in \left[K\right]}\left|{\lambda }_k\right|$, and that the eigen-gap δ_ℓ as defined in (2.9) satisfies ${\delta }_{\ell }\ge 5{\Vert \mathbf{\text{W}}\Vert }_2$. Then, up to sign,

$$\left|{\left[{\widetilde{\mathbf{\text{v}}}}_{\ell }-{\mathbf{\text{v}}}_{\ell }\right]}_m\right|\lesssim \frac{{\Vert \mathbf{\text{W}}\Vert }_2}{{\delta }_{\ell }}{\left(\sum _{k=1}^K{\left[{\mathbf{\text{v}}}_k\right]}_m^2\right)}^{1/2}+\frac{\left|\langle {\mathbf{\text{w}}}_m,{\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(m\right)}\rangle \right|}{{\delta }_{\ell }},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall m\in \left[n\right],$$ (2.10)

where w_m is the mth column of W.

To understand this theorem, let us compare it with the standard ℓ₂ bound (Theorem 2.3), which implies ${\Vert {\widetilde{\mathbf{\text{v}}}}_{\ell }-{\mathbf{\text{v}}}_{\ell }\Vert }_2\le {\Vert \mathbf{\text{W}}\Vert }_2/{\delta }_{\ell }$. The first term of the upper bound in (2.10) says the perturbation on the mth entry can be much smaller, because the factor ${({\sum }_{k=1}^K{\left[{\mathbf{\text{v}}}_k\right]}_m^2)}^{1/2}$, always bounded by 1, can be usually much smaller. For example, if v_k’s are uniformly distributed on the unit sphere, then this factor is typically of order $O(\sqrt{K\text{log}n/n})$. This factor is related to the notion of incoherence in Candès and Recht (2009); Candès et al. (2011), etc.

The second term of the upper bound in (2.10) is typically much smaller than ${\Vert \mathbf{\text{W}}\Vert }_2/{\delta }_{\ell }$, especially under certain independence assumption. For example, if w_m is independent of other entries, then, by construction, ${\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(m\right)}$ and w_m are independent. If, moreover, entries of w_m are i.i.d. standard Gaussian, $|\langle {\mathbf{\text{w}}}_m,{\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(m\right)}\rangle |$ is of order $O_{\mathbb{P}}\left(1\right)$, whereas ${\Vert \mathbf{\text{W}}\Vert }_2$ typically scales with $\sqrt{n}$. This gives a bound for the mth entry, and can be extended to an ℓ_∞ bound if we are willing to make independence assumption for all m ∈ [n] (which is typical for random graphs for example).

We remark that this result can be generalized to perturbation bounds for eigenspaces (Abbe et al., 2017), and the conditions on eigenvalues can be relaxed using certain random matrix assumptions (Koltchinskii and Xia, 2016; O’Rourke et al., 2017; Zhong, 2017).

Now, we extend this perturbation result to singular vectors of rectangular matrices. Suppose L, $\widetilde{\mathbf{\text{L}}}$, $\mathbf{\text{E}}\in {ℝ}^{n\times p}$ satisfy $\widetilde{\mathbf{\text{L}}}=\mathbf{\text{L}}+\mathbf{\text{E}}$ and rank(L) = K < min{n, p}. Let the SVD of L and $\widetilde{\mathbf{\text{L}}}$ be⁴

$$\mathbf{\text{L}}=\sum _{k=1}^K{\sigma }_k{\mathbf{\text{u}}}_k{\mathbf{\text{v}}}_k^{\top }\hspace{1em}\text{and}\hspace{1em}\widetilde{\mathbf{\text{L}}}=\sum _{k=1}^K{\tilde{\sigma }}_k{\widetilde{\mathbf{\text{u}}}}_k{\widetilde{\mathbf{\text{v}}}}_k^{\top }+\sum _{k=K+1}^{\text{min}\left\{n,p\right\}}{\tilde{\sigma }}_k{\widetilde{\mathbf{\text{u}}}}_k{\widetilde{\mathbf{\text{v}}}}_k^{\top },$$

where σ_k and ${\tilde{\sigma }}_k$ are respectively non-increasing in k, and u_k and v_k are all normalized to have unit ℓ₂ norm. As before, let ${\left\{{\tilde{\sigma }}_k\right\}}_{k=1}^K$ have K largest absolute values. Similar to (2.9), we adopt the conventions σ₀ = +∞, σ_K+1 = 0 and define the eigen-gap as

$${\gamma }_k=\text{min}\left\{{\sigma }_{k-1}-{\sigma }_k,{\sigma }_k-{\sigma }_{k+1}\right\},\hspace{1em}\forall k\in \left[K\right].$$ (2.11)

For j ∈ [p] and i ∈ [n], we define unit vectors ${\{{\widetilde{\mathbf{\text{u}}}}_k^{\left(j\right)}\}}_{k=1}^{\text{min}\left\{n,p\right\}}\subseteq {ℝ}^n$ and ${\{{\widetilde{\mathbf{\text{v}}}}_k^{\left(i\right)}\}}_{k=1}^{\text{min}\left\{n,p\right\}}\subseteq {ℝ}^p$ by replacing certain row or column of E with zeros. To be specific, in our expression $\widetilde{\mathbf{\text{L}}}=\mathbf{\text{L}}+\mathbf{\text{E}}$, if we replace the ith row of E by zeros, then the normalized right singular vectors of the resulting perturbed matrix are denoted by ${\{{\widetilde{\mathbf{\text{v}}}}_k^{\left(i\right)}\}}_{k=1}^{\text{min}\left\{n,p\right\}}$; and if we replace the jth column of E by zeros, then the normalized left singular vectors of the resulting perturbed matrix are denoted by ${\left\{{\widetilde{\mathbf{\text{u}}}}_k^{\left(j\right)}\right\}}_{k=1}^{\text{min}\left\{n,p\right\}}$.

Corollary 2.2. Fix any ℓ ∈ [K]. Suppose that ${\sigma }_{\ell }\asymp {\text{max}}_{k\in \left[K\right]}{\sigma }_k$, and that ${\gamma }_{\ell }\ge 5{\Vert \mathbf{\text{E}}\Vert }_2$. Then, up to sign,

$$\left|{\left[{\widetilde{\mathbf{\text{u}}}}_{\ell }-{\mathbf{\text{u}}}_{\ell }\right]}_i\right|\lesssim \frac{{\Vert \mathbf{\text{E}}\Vert }_2}{\gamma \ell }{\left(\sum _{k=1}^K{\left[{\mathbf{\text{u}}}_k\right]}_i^2\right)}^{1/2}+\frac{\left|\langle {\left({\mathbf{\text{e}}}_i^{\text{row}}\right)}^{\top },{\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(i\right)}\rangle \right|}{\gamma \ell },\hspace{1em}\forall i\in \left[n\right],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and$$

$$\left|{\left[{\widetilde{\mathbf{\text{v}}}}_{\ell }-{\mathbf{\text{v}}}_{\ell }\right]}_j\right|\lesssim \frac{{\Vert \mathbf{\text{E}}\Vert }_2}{\gamma \ell }{\left(\sum _{k=1}^K{\left[{\mathbf{\text{v}}}_k\right]}_j^2\right)}^{1/2}+\frac{\left|\langle {\mathbf{\text{e}}}_j^{\text{col}},{\widetilde{\mathbf{\text{u}}}}_{\ell }^{\left(j\right)}\rangle \right|}{\gamma \ell },\hspace{1em}\forall j\in \left[p\right],$$

where ${\mathbf{\text{e}}}_i^{\text{row}}\in {ℝ}^p$ is the ith row vector of E, and ${\mathbf{\text{e}}}_j^{\text{col}}\in {ℝ}^n$ is the jth column vector of E.

If we view $\widetilde{\mathbf{\text{L}}}$ as the data matrix (or observation) X, then, the low rank matrix L can be interpreted as BF^⊤. The above result provides a tool of studying estimation errors of the singular subspace of this low rank matrix. Note that ${\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(i\right)}$ can be interpreted as the result of removing the idiosyncratic error of the ith observation, and ${\widetilde{\mathbf{\text{u}}}}_{\ell }^{\left(j\right)}$ as the result of removing the jth covariate of the idiosyncratic error.

To better understand this result, let us consider a very simple case: K = 1 and each row of E is i.i.d.$\mathcal{N}\left(0,{\mathbf{\text{I}}}_p\right)$. We are interested in bounding the singular vector difference between the rank-1 matrix L = σ₁uv^⊤ and its noisy observation $\widetilde{\mathbf{\text{L}}}=\mathbf{\text{L}}+\mathbf{\text{E}}$. This is a spiked matrix model with a single spike. By independence between ${\mathbf{\text{e}}}_i^{\text{row}}$ and ${\widetilde{\mathbf{\text{v}}}}_{\ell }^{\left(i\right)}$ as well as elementary properties of Gaussian variables, Corollary 2.2 implies that with probability 1 – o(1), up to sign,

$${\Vert {\widetilde{\mathbf{\text{u}}}}_1-{\mathbf{\text{u}}}_1\Vert }_{\infty }\le \frac{{\Vert \mathbf{\text{E}}\Vert }_2}{{\sigma }_1}{\Vert {\mathbf{\text{u}}}_1\Vert }_{\infty }+\frac{O\left(\sqrt{\text{log}n}\right)}{{\sigma }_1}.$$ (2.12)

Random matrix theory gives ${\Vert \mathbf{\text{E}}\Vert }_2\asymp \sqrt{n}+\sqrt{p}$ with high probability. Our ℓ₂ perturbation inequality (Corollary 2.1) implies that ${\Vert {\widetilde{\mathbf{\text{u}}}}_1-{\mathbf{\text{u}}}_1\Vert }_2\le {\Vert \mathbf{\text{E}}\Vert }_2/{\sigma }_1$. This upper bound is much larger than the two terms in (2.12), as ${\Vert {\mathbf{\text{u}}}_1\Vert }_{\infty }$ is typically much smaller than 1 in high dimensions. Thus, (2.12) gives a better entry-wise control over the ℓ₂ counterpart.

Beyond this simple case, there are many desirable features of Corollary 2.2. First of all, we allow K to be moderately large, in which case, as mentioned before, the factor ${({\sum }_{k=1}^K{\left[{\mathbf{\text{u}}}_k\right]}_i^2)}^{1/2}$ is related to the incoherence structure in the matrix completion and robust PCA literature. Secondly, the result holds deterministically, so random matrices are also applicable. Finally, the result holds for each i ∈ [n] and j ∈ [p], and thus it is useful even if the entries of E are not independent, e.g. when a subset of covariates are dependent.

To sum up, our results Theorem 2.4 and Corollary 2.2 provide flexible tools of studying entry-wise perturbation of eigenvectors and singular vectors. It is also easy to adapt to other problems since their proofs are not complicated (see the appendix).

3. APPLICATIONS TO HIGH-DIMENSIONAL STATISTICS

3.1. Covariance estimation

Estimation of high-dimensional covariance matrices has wide applications in modern data analysis. When the dimensionality p exceeds the sample size n, the sample covariance matrix becomes singular. Structural assumptions are necessary in order to obtain a consistent estimator in this challenging scenario. One typical assumption in the literature is that the population covariance matrix is sparse, with a large fraction of entries being (close to) zero, see Bickel and Levina (2008) and Cai and Liu (2011). In this setting, most variables are nearly uncorrelated. In financial and genetic data, however, the presence of common factors leads to strong dependencies among variables (Fan et al., 2008). The approximate factor model (1.1) better characterizes this structure and helps construct valid estimates. Under this model, the covariance matrix Σ has decomposition (2.1), where ${\mathbf{\text{Σ}}}_u=\text{cov}\left({\mathbf{\text{u}}}_i\right)={\left({\sigma }_{u,jk}\right)}_{1\le j,k\le p}$ is assumed to be sparse (Fan et al., 2013). Intuitively, we may assume that Σ_u only has a small number of nonzero entries. Formally, we require the sparsity parameter

$$m_0:=\underset{j\in \left[p\right]}{\text{max}}\sum _{k=1}^{p}1\left\{{\sigma }_{u,jk}\ne 0\right\}$$

to be small. This definition can be generalized to a weaker sense of sparsity, which is characterized by $m_q={\text{max}}_{j\in \left[p\right]}{\sum }_{k=1}^p{\left|{\sigma }_{u,jk}\right|}^q$, where q ∈ (0g, 1) is a parameter. Note that small m_q forces Σ_u to have few large entries. However, for simplicity, we choose not to use this more general definition when presenting theoretical results below.

The approximate factor model has the following two important special cases, under which the parameter estimation has been well studied.

• The sparse covariance model is (2.1) without factor structure, i.e. Σ = Σ_u; typically, entry-wise thresholding is employed for estimation.

• The strict factor model corresponds to (2.1) with Σ_u being diagonal; usually, PCA-based methods are used.

The approximate factor model is a combination of the above two models, as it comprises both a low-rank component and a sparse component. A natural idea is to fuse methodologies for the two models into one, by estimating the two components using their corresponding methods. This motivated our high-level idea for estimation under the approximate factor model: (1) estimating the low-rank component (factors and loadings) using regression (when factors are observable) or PCA (when factors are latent); (2) after eliminating it from Σ, employing standard techniques such as thresholding in the sparse covariance matrix literature to estimate Σ_u; (3) adding the two estimated components together.

First, let us consider the scenario where the factors ${\left\{{\mathbf{\text{f}}}_i\right\}}_{i=1}^n$ are observable. In this setting, we do not need the Identifiability Assumption 1.1. Fan et al. (2008) focused on the strict factor model where the Σ_u in (2.1) is diagonal. It is then extended to the approximate factor model (1.1) by Fan et al. (2011). Later, Fan et al. (2018b) relaxed the sub-Gaussian assumption on the data to moment condition, and proposed a robust estimator. We are going to present the main idea of these methods using the one in Fan et al. (2011).

Step 1. Estimate B using the ordinary least-squares: $\widehat{\mathbf{\text{B}}}={\left({\widehat{\mathbf{\text{b}}}}_1,\dots ,{\widehat{\mathbf{\text{b}}}}_p\right)}^{\top }$ where

$$({\widehat{a}}_j,{\widehat{\mathbf{\text{b}}}}_j)={\text{argmin}}_{a,\widehat{\mathbf{\text{b}}}}\frac{1}{n}\sum _{i=1}^n{\left(x_{ij}-a-{\mathbf{\text{b}}}^{\top }{\mathbf{\text{f}}}_i\right)}^2.$$

Step 2. Let $\widehat{\mathbf{\text{a}}}={\left({\widehat{a}}_1,\cdots ,{\widehat{a}}_p\right)}^{\top }$ be the vector of intercepts, ${\widehat{\mathbf{\text{u}}}}_i={\mathbf{\text{x}}}_i-\widehat{\mathbf{\text{a}}}-\widehat{\mathbf{\text{B}}}{\mathbf{\text{f}}}_i$ be the vector of residual for i ∈ [n], and ${\mathbf{\text{S}}}_u=\frac{1}{n}{\sum }_{i=1}^n{\widehat{\mathbf{\text{u}}}}_i{\widehat{\mathbf{\text{u}}}}^{\top }$ be the sample covariance. Apply thresholding to S_u and obtain a regularized estimator ${\widehat{\mathbf{\text{Σ}}}}_u$.

Step 3. Estimate cov(f_i) by $\widehat{\text{cov}}\left({\mathbf{\text{f}}}_i\right)=\frac{1}{n}{\sum }_{i=1}^n\left({\mathbf{\text{f}}}_i-\overline{\mathbf{\text{f}}}\right){\left({\mathbf{\text{f}}}_i-\overline{\mathbf{\text{f}}}\right)}^{\top }$.

Step 4. The final estimator is $\widehat{\mathbf{\text{Σ}}}=\widehat{\mathbf{\text{B}}}\widehat{\text{cov}}\left({\mathbf{\text{f}}}_i\right){\widehat{\mathbf{\text{B}}}}^{\top }+{\widehat{\mathbf{\text{Σ}}}}_u$.

We remark that in Step 2, there are many thresholding rules for estimating sparse covariance matrices. Two popular choices are the t-statistic-based adaptive thresholding (Cai and Liu, 2011) and correlation-based adaptive thresholding (Fan et al., 2013), with the entry-wise thresholding level chosen to be $\omega \asymp K\sqrt{\frac{\text{log}p}{n}}$. As the sparsity pattern of correlation and covariance are the same and the correlation matrix is scale-invariant, one typically applies the thresholding on the correlation and then scales it back to the covariance. Except for the number of factors K, this coincides with the commonly-used threshold for estimating sparse covariance matrices.

While it is not possible to achieve better convergence of Σ in terms of the operator norm or the Frobenius norm, Fan et al. (2011) considered two other important norms. Under regularity conditions, it is shown that

$$\begin{gathered} {\Vert \widehat{\mathbf{\text{Σ}}}-\mathbf{\text{Σ}}\Vert }_{\mathbf{\text{Σ}}}=O_{\mathbb{P}}\left(m_{0}K\sqrt{\frac{\text{log}p}{n}}+\frac{K\sqrt{p}\text{log}p}{n}\right), \\ {\Vert \widehat{\mathbf{\text{Σ}}}-\mathbf{\text{Σ}}\Vert }_{\text{max}}=O_{\mathbb{P}}\left(K\sqrt{\frac{\text{log}p}{n}}+K^2\sqrt{\frac{\text{log}n}{n}}\right). \end{gathered}$$ (3.1)

Here for $\mathbf{\text{A}}\in {ℝ}^{p\times p}$, ${\Vert \mathbf{\text{A}}\Vert }_{\mathbf{\text{Σ}}}$ and ${\Vert \mathbf{\text{A}}\Vert }_{\text{max}}$ refer to its entropy-loss norm $p^{-1/2}{\Vert {\mathbf{\text{Σ}}}^{-1/2}\mathbf{\text{A}}{\mathbf{\text{Σ}}}^{-1/2}\Vert }_F$ and entry-wise max-norm max_i,j |A_ij|. As is pointed out by Fan et al. (2011) and Wang and Fan (2017), they are relevant to portfolio selection and risk management. In addition, convergence rates for ${\Vert {\widehat{\mathbf{\text{Σ}}}}^{-1}-{\mathbf{\text{Σ}}}^{-1}\Vert }_2$, ${\Vert {\widehat{\mathbf{\text{Σ}}}}_u-{\mathbf{\text{Σ}}}_u\Vert }_2$ and ${\Vert {\widehat{\text{Σ}}}_u^{-1}-{\text{Σ}}_u^{-1}\Vert }_2$ are also established.

Now we come to covariance estimation with latent factors. As is mentioned in Section 2.1, the Pervasiveness Assumption 2.1 helps separate the low-rank part BB^⊤ from the sparse part Σ_u in (2.1). Fan et al. (2013) proposed a Principal Orthogonal complEment Thresholding (POET) estimator, motivated by the relationship between PCA and factor model, and the estimation of sparse covariance matrix Σ_u in Fan et al. (2011). The procedure is described as follows.

Step 1. Let $\mathbf{\text{S}}=\frac{1}{n}{\sum }_{i=1}^n{\mathbf{\text{x}}}_i{\mathbf{\text{x}}}_i^{\top }$ be the sample covariance matrix, ${\{{\widehat{\lambda }}_j\}}_{j=1}^p$ be the eigenvalues of S in non-ascending order, ${\{{\widehat{\mathbf{\xi }}}_j\}}_{j=1}^p$ be their corresponding eigenvectors.

Step 2. Apply thresholding to ${\mathbf{\text{S}}}_u=\mathbf{\text{S}}-{\sum }_{j=1}^K{\widehat{\lambda }}_j{\widehat{\mathbf{\xi }}}_j{\widehat{\mathbf{\xi }}}_j^{\top }$ and obtain a regularized estimator ${\widehat{\mathbf{\text{Σ}}}}_u$.

Step 3. The final estimator is $\widehat{\mathbf{\text{Σ}}}={\sum }_{j=1}^K{\widehat{\lambda }}_j{\widehat{\mathbf{\xi }}}_j{\widehat{\mathbf{\xi }}}_j^{\top }+{\widehat{\mathbf{\text{Σ}}}}_u$.

Here K is assumed to be known and bounded to simplify presentation and emphasize the main ideas. The methodology and theory in Fan et al. (2013) also allow using a data-driven estimate $\widehat{K}$ of K. In Step 2 above we can choose from a large class of thresholding rules, and it is recommended to use the correlation-based adaptive thresholding. However, the thresholding level should be set to $\tilde{\omega }\asymp \sqrt{\frac{\text{log}p}{n}}+\frac{1}{\sqrt{p}}$. Compared to the level $\sqrt{\frac{\text{log}p}{n}}$ we use in covariance estimation with observed factors, the extra term $1/\sqrt{p}$ here is the price we pay for not knowing the latent factors. It can be negligible when p grows much faster than n. Intuitively, thanks to the Pervasiveness Assumption, the latent factors can be estimated accurately in high dimensions. Fan et al. (2013) obtained theoretical guarantees for the POET that are similar to (3.1). The analysis allows for general sparsity patterns of Σ_u by considering m_q as the measure of sparsity for q ∈ [0, 1).

Robust procedures handling heavy-tailed data are proposed and analyzed by Fan et al. (2018a,b). In another line of research, Li et al. (2017) considered estimation of the covariance matrix of a set of targeted variables, when additional data beyond the variables of interest are available. By assuming a factor model structure, they constructed an estimator taking advantage of all the data and justified the information gain theoretically.

The Pervasiveness Assumption rules out the case where factors are weak and the leading eigenvalues of Σ are not as large as O(p). Shrinkage of eigenvalues is a powerful technique in this scenario. Donoho et al. (2013) systematically studied the optimal shrinkage in spiked covariance model where all the eigenvalues except several largest ones are assumed to be the same. Wang and Fan (2017) considered the approximate factor model, which is more general, and proposed a new version of POET with shrinkage for covariance estimation.

3.2. Principal component regression with random sketch

Principal component regression (PCR), first proposed by Hotelling (1933) and Kendall (1965), is one of the most popular methods of dimension reduction in linear regression. It employs the principal components of the predictors x_i to explain or predict the response y_i. Why do principal components, not other components, have more prediction power? Here we offer an insight from the perspective of high-dimensional factor models.

The basic assumption is that the unobserved latent factors ${\mathbf{\text{f}}}_i\in {ℝ}^K$ drive simultaneously the covariates via (1.1) and responses, as shown in Figure 3. As a specific example, we assume

$$y_i={\mathbf{\theta }}^{\ast \top }{\mathbf{\text{f}}}_i+{\epsilon }_i,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,n,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{or in matrix form},\hspace{1em}\mathbf{\text{y}}=\mathbf{\text{F}}{\mathbf{\theta }}^{\ast }+\epsilon ,$$

where y = (y₁, …, y_n)^⊤ and the noise ε = (ε₁, …, ε_n)^⊤ has zero means. Since f_i is latent and the covariate vector is high dimensional, we naturally infer the latent factors from the observed covariates via PCA. This yields the PCR.

Fig 3.

Illustration of the data generation mechanism in PCR. Both predictors x_i and responses y_i are driven by the latent factors f_i. PCR extracts latent factors via the principal components from X, and uses the resulting estimate $\widehat{\mathbf{F}}$ as the new predictor. Regressing y against $\widehat{\mathbf{F}}$ leads to the PCR estimator $\widehat{\theta }\in {ℝ}^K$, which typically enjoys a smaller variance due to its reduced dimension, though it introduces bias.

By (2.2) (assume μ = 0 for simplicity), $y_i\approx {\left({\mathit{\beta }}^{†}\right)}^{\top }{\mathbf{\text{x}}}_i+{\epsilon }_i$, where ${\mathit{\beta }}^{†}:=\mathbf{\text{B}}{\left({\mathbf{\text{B}}}^{\top }\mathbf{\text{B}}\right)}^{-1}{\mathbf{\theta }}^{\ast }\in {ℝ}^p$. This suggests that if we directly regress y_i over x_i, then the regression coefficient β^† should lie in the column space spanned by B. This inspires the core idea of PCR, i.e., instead of seeking the least square estimator in the entire ${ℝ}^p$ space, we restrict our search scope to be the left leading singular space of X, which is approximately the column space of B under the Pervasiveness Assumption.

Let us discuss PCR more rigorously. To be consistent with the rest of this paper, we let $\mathbf{\text{X}}\in {ℝ}^{p\times n}$, which is different from conventions, and

$$y_i={\mathbf{\text{x}}}_i^{\top }{\mathit{\beta }}^{\ast }+{\epsilon }_i,\hspace{1em}i=1,\dots ,n,\hspace{1em}\text{or in matrix form},\hspace{1em}\mathbf{\text{y}}={\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }+\mathbf{\epsilon }.$$ (3.2)

Let $\mathbf{\text{X}}=\left({\mathbf{\text{x}}}_1,\dots ,{\mathbf{\text{x}}}_n\right)=\mathbf{\text{PΣ}}{\mathbf{\text{Q}}}^{\top }$ be the SVD of X, where $\mathbf{\text{Σ}}=\text{diag}\left({\sigma }_1,\dots ,{\sigma }_{\text{min}\left(n,p\right)}\right)$ with non-increasing singular values. For some integer K satisfying 1 ≤ K ≤ min(n, p), write P = (P_K, P_K+) and Q = (Q_K, Q_K+). The PCR estimator ${\widehat{\mathit{\beta }}}_K$ solves the following optimization problem:

$${\widehat{\mathit{\beta }}}_K:={\text{argmin}}_{{\mathbf{\text{P}}}_{K+}^{\top }\mathit{\beta }=0}{\Vert \mathbf{\text{y}}-{\mathbf{\text{X}}}^{\top }\mathit{\beta }\Vert }_2.$$ (3.3)

It is easy to verify that

$${\widehat{\mathit{\beta }}}_K={\mathbf{\text{P}}}_K{\mathbf{\text{Σ}}}_K^{-1}{\mathbf{\text{Q}}}_K^{\top }\mathbf{\text{y}}={\mathbf{\text{P}}}_K{\mathbf{\text{P}}}_K^{\top }{\mathit{\beta }}^{\ast }+{\mathbf{\text{P}}}_K{\mathbf{\text{Σ}}}_K^{-1}{\mathbf{\text{Q}}}_K^{\top }\mathbf{\epsilon },$$ (3.4)

where ${\mathbf{\text{Σ}}}_K\in {ℝ}^{K\times K}$ is the top left submatrix of Σ. The following lemma calculates the excess risk of ${\widehat{\mathit{\beta }}}_K$, i.e., $\mathcal{E}\left({\widehat{\mathit{\beta }}}_K\right):={\mathbb{E}}_{\mathbf{\epsilon }}\left[{\Vert {\mathbf{\text{X}}}^{\top }{\widehat{\mathit{\beta }}}_K-{\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2/n\right]$, treating X as fixed. The proof is relegated to the appendix.

Lemma 3.1. Let ${\mathbf{\text{p}}}_1,\dots ,{\mathbf{\text{p}}}_{\text{min}\left\{n,p\right\}}\in {ℝ}^p$ be the column vectors of P. For j = 1, …, p, denote ${\alpha }_j={\left({\mathit{\beta }}^{\ast }\right)}^{\top }{\mathbf{\text{p}}}_j$. We have

$$\mathcal{E}({\widehat{\mathit{\beta }}}_K)=\frac{K{\sigma }^2}{n}+\sum _{j=K+1}^p{\lambda }_j^2{\alpha }_j^2.$$

Define the ordinary least squares (OLS) estimator $\widehat{\mathit{\beta }}:={\left(\mathbf{\text{X}}{\mathbf{\text{X}}}^{\top }\right)}^{-1}\mathbf{\text{Xy}}$. Note that $\mathcal{E}(\widehat{\mathit{\beta }})={\mathbb{E}}_{\epsilon }\left[{\Vert {\mathbf{\text{X}}}^{\top }\widehat{\mathit{\beta }}-{\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2/n\right]=\text{min}\left(n,p\right){\sigma }^2/n$. Comparing $\mathcal{E}({\widehat{\mathit{\beta }}}_K)$ and $\mathcal{E}(\widehat{\mathit{\beta }})$, one can clearly see a variance-bias tradeoff: PCR reduces the variance by introducing a bias term ${\sum }_j{\lambda }_j^2{\alpha }_j^2$, which is typically small and vanishes in the ideal case ${\mathbf{\text{P}}}_{K+}^{\top }{\mathit{\beta }}^{\ast }=0$—this is the bias incurred by imposing the constraint in (3.3).

In the high-dimensional setting where p is large, calculating P_K using SVD is computationally expensive. Recently, sketching has gained growing attention in statistics community and is used for downscaling and accelerating inference tasks with massive data. See recent surveys by Woodruff (2014) and Yang et al. (2016). The essential idea is to multiply the data matrix by a sketch matrix to reduce its dimension while still preserving the statistical performance of the procedure, since random projection reduces the strength of the idiosyncratic noise. To apply sketching to PCR, we first multiply the design matrix X by an appropriately chosen matrix $\mathbf{\text{R}}\in {ℝ}^{p\times m}$ with K ≤ m < p:

$$\widetilde{\mathbf{\text{X}}}:={\mathbf{\text{R}}}^{\top }\mathbf{\text{X}},$$ (3.5)

where R is called the “sketching matrix”. This creates m indices based on X. From the factor model perspective (assuming μ = 0), with a proper choice of R, we have $\widetilde{\mathbf{\text{X}}}\approx {\mathbf{\text{R}}}^{\top }\mathbf{\text{B}}{\mathbf{\text{F}}}^{\top }$, since the idiosyncratic components in (1.1) is averaged out due to weak dependence of u. Hence, the indices in $\widetilde{\mathbf{\text{X}}}$ are approximately linear combinations of the factors ${\left\{{\mathbf{\text{f}}}_i\right\}}_{i=1}^n$. At the same time, since m ≥ K and R is nondegenerate, the row space of $\widetilde{\mathbf{\text{X}}}$ is approximately the same as that spanned by F^⊤. This shows running linear regression on $\widetilde{\mathbf{\text{X}}}$ is approximately the same as running it on F^⊤, without using the computationally expensive PCA.

We now examine the property of sketching approach beyond the factor models. Let $\widetilde{\mathbf{\text{X}}}=\widetilde{\mathbf{\text{P}}}\widetilde{\mathbf{\text{Σ}}}{\widetilde{\mathbf{\text{Q}}}}^{\top }$ be the SVD of $\widetilde{\mathbf{\text{X}}}$, and write $\widetilde{\mathbf{\text{P}}}=\left({\widetilde{\mathbf{\text{P}}}}_K,{\widetilde{\mathbf{\text{P}}}}_{K+}\right)$ and $\widetilde{\mathbf{\text{Q}}}=({\widetilde{\mathbf{\text{Q}}}}_K,{\widetilde{\mathbf{\text{Q}}}}_{K+})$. Imitating the form of (3.4), we consider the following sketched PCR estimator:

$${\tilde{\mathit{\beta }}}_K:=\mathbf{\text{R}}{\widetilde{\mathbf{\text{P}}}}_K{\widetilde{\mathbf{\text{Σ}}}}_K^{-1}{\widetilde{\mathbf{\text{Q}}}}_K^{\top }\mathbf{\text{y}},$$ (3.6)

where ${\widetilde{\mathbf{\text{Σ}}}}_K\in {ℝ}^{K\times K}$ is the top left submatrix of $\widetilde{\mathbf{\text{Σ}}}$.

We now explain the above construction for ${\tilde{\mathit{\beta }}}_K$. It is easy to derive from (3.4) that given R^⊤X and y as the design matrix and response vector, the PCR estimator should be ${\tilde{\mathit{\beta }}}_K^0:={\widetilde{\mathbf{\text{P}}}}_K{\widetilde{\mathbf{\text{Σ}}}}_K^{-1}{\widetilde{\mathbf{\text{Q}}}}_K^{\top }\mathbf{\text{y}}$. Then the corresponding PCR projection of y onto R^⊤X should be ${\mathbf{\text{X}}}^{\top }\mathbf{\text{R}}{\tilde{\mathit{\beta }}}_K^0={\mathbf{\text{X}}}^{\top }\mathbf{\text{R}}{\widetilde{\mathbf{\text{P}}}}_K{\widetilde{\mathbf{\text{Σ}}}}_K^{-1}{\widetilde{\mathbf{\text{Q}}}}_K^{\top }\mathbf{\text{y}}={\mathbf{\text{X}}}^{\top }{\tilde{\mathit{\beta }}}_K$. This leads to the construction of ${\tilde{\mathit{\beta }}}_K$ in (3.6). Theorem 4 in Mor-Yosef and Avron (2018) gives the excess risk of ${\tilde{\mathit{\beta }}}_K$, which holds for any R satisfying the conditions of the theorem.

Theorem 3.1. Assume m ≥ K and rank(R^⊤X) ≥ K. If ${\Vert \text{sin}\Theta \left({\widetilde{\mathbf{\text{P}}}}_K,{\mathbf{\text{P}}}_K\right)\Vert }_2\le \nu <1$, then

$$\mathcal{E}({\tilde{\mathit{\beta }}}_K)\le \mathcal{E}({\widehat{\mathit{\beta }}}_K)+\frac{\left(2\nu +{\nu }^2\right){\Vert {\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2}{n},$$ (3.7)

This theorem shows that the extra bias induced by sketching is $(2\nu +{\nu }^2){\Vert {\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2/n$. Given the bound of $\mathcal{E}({\widehat{\mathit{\beta }}}_K)$ in Lemma 3.1, we can deduce that

$$\mathcal{E}({\tilde{\mathit{\beta }}}_K)\le \frac{K{\sigma }^2}{n}+\sum _{j=K+1}^p{\alpha }_j^2{\sigma }_j^2+\frac{\left(2\nu +{\nu }^2\right){\Vert {\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2}{n}.$$

As we will see below, a smaller ν requires a larger m, and thus more computation. Therefore, we observe a tradeoff between statistical accuracy and computational resources: if we have more computational resources, we can allow a large dimension of sketched matrix $\widetilde{\mathbf{\text{X}}}$, and the sketched PCR is more accurate, and vice versa.

One natural question thus arises: which R should we choose to guarantee a small ν to retain the statistical rate of ${\widehat{\mathit{\beta }}}_K$? Recent results (Cohen et al., 2015) on approximate matrix multiplication (AMM) suggest several candidate sketching matrices for R. Define the stable rank $\text{sr}\left(\mathbf{\text{X}}\right):={\Vert \mathbf{\text{X}}\Vert }_F^2/{\Vert \mathbf{\text{X}}\Vert }_2^2$, which can be interpreted as a soft version of the usual rank—indeed, sr(X) ≤ rank(X) always holds, and sr(X) can be small if X is approximately low-rank. An example of candidate sketching matrices for R is a random matrix with independent and suitably scaled sub-Gaussian entries. As long as the sketch size m = Ω(sr(X) + log(1/δ)/ε²), it will hold for any ε, δ ∈ (0, 1/2) that

$$\mathbb{P}({\Vert {\mathbf{\text{X}}}^{\top }\mathbf{\text{R}}{\mathbf{\text{R}}}^{\top }\mathbf{\text{X}}-{\mathbf{\text{X}}}^{\top }\mathbf{\text{X}}\Vert }_2^2\ge \epsilon {\Vert \mathbf{\text{X}}\Vert }_2^2)\le \delta .$$ (3.8)

Combining this with the Davis-Kahan Theorem (Corollary 2.1), we can deduce that ${\Vert \text{sin}\Theta \left({\widetilde{\mathbf{\text{P}}}}_K,{\mathbf{\text{P}}}_K\right)\Vert }_2$ is small with certain eigen-gap condition. We summarize our argument by presenting a corollary of Theorem 9 in Mor-Yosef and Avron (2018) below. Readers can find more candidate sketching matrices in the examples after Theorem 1 in Cohen et al. (2015).

CorollarY 3.1. For any ν, δ ∈ (0, 1/2), let

$$\epsilon =\nu {\left(1+\nu \right)}^{-1}\left({\sigma }_K^2-{\sigma }_{K+1}^2\right)/{\sigma }_1^2.$$

Let $\mathbf{\text{R}}\in {ℝ}^{p\times m}$ a random matrix with i.i.d. $\mathcal{N}\left(0,1/m\right)$ entries. Then there exists a universal constant C > 0 such that for any δ > 0, if m ≥ C(sr(X)+ log(1/δ)/ε²), it holds with probability at least 1 − δ that

$$\mathcal{E}({\tilde{\mathit{\beta }}}_K)\le \mathcal{E}({\widehat{\mathit{\beta }}}_K)+\frac{\left(2\nu +{\nu }^2\right){\Vert {\mathbf{\text{X}}}^{\top }{\mathit{\beta }}^{\ast }\Vert }_2^2}{n}.$$ (3.9)

Remark 3.1. Note that $\epsilon \le \nu \left({\sigma }_K^2-{\sigma }_{K+1}^2\right)/{\sigma }_1^2$, and this bound is tight with a small ν. Some algebra yields that (3.9) holds when

$$m=\Omega (\text{sr}\left(\mathbf{\text{X}}\right)+\frac{{\sigma }_1^2\text{log}\left(1/\delta \right)}{{\nu }^2{\left({\sigma }_K^2-{\sigma }_{K+1}^2\right)}^2}).$$

One can see that reducing ν requires a larger sketch size m. Besides, a large eigengap of the design matrix X helps reduce the required sketch size.

3.3. Factor-Adjust Robust Multiple (FARM) tests

Large-scale multiple testing is a fundamental problem in high-dimensional inference. In genome-wide association studies and many other applications, tens of thousands of hypotheses are tested simultaneously. Standard approaches such as Benjamini and Hochberg (1995) and Storey (2002) can not control well both false and missed discovery rates in the presence of strong correlations among test statistics. Important efforts on dependence adjustment include Efron (2007), Friguet et al. (2009), Efron (2010), and Desai and Storey (2012). Fan et al. (2012) and Fan and Han (2017) considered FDP estimation under the approximate factor model. Wang et al. (2017) studied a more complicated model with both observed variables and latent factors. All these existing papers heavily rely on the joint normality assumption of the data, which is easily violated in real applications. A recent paper (Fan et al., 2017a) developed a factor-adjusted robust procedure that can handle heavy-tailed data while controlling FDP. We are going to introduce this method in this subsection.

Suppose our i.i.d. observations ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$ satisfy the approximate factor model (1.1) where $\mathit{\mu }\in {ℝ}^p$ is an unknown mean vector. To make the model identifiable, we use the Identifiability Assumption 1.1. We are interested in simultaneously testing

$$H_{0j}:{\mu }_j=0\hspace{1em}\text{versus}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\text{H}}_{1j}:{\mu }_j\ne 0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}j\in \left[p\right].$$

Let T_j be a generic test statistic for H_0j. For a pre-specified level z > 0, we reject H_0j whenever |T_j| ≥ z. The numbers of total discoveries R(z) and false discoveries V (z) are defined as

$$R\left(z\right)=\#\left\{j:\left|T_j\right|\ge z\right\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}V\left(z\right)=\#\left\{j:\left|T_j\right|\ge z,\hspace{0.17em}\hspace{0.17em}{\mu }_j=0\right\}.$$

Note that R(z) is observable while V (z) needs to be estimated. Our goal is to control the false discovery proportion FDP(z) = V (z)/R(z) with the convention 0/0 = 0.

Naïve tests based on sample averages $\frac{1}{n}{\sum }_{i=1}^n{\mathbf{x}}_i$ suffer from size distortion of FDP control due to dependence of common factors in (1.1). On the other hand, the factor-adjusted test based on the sample averages of x_i − Bf_i (B and f_i need to be estimated) has two advantages: the noise u_i is now weakly dependent so that FDP can be controlled with high accuracy, and the variance of u_i is smaller than that of Bf_i + u_i in model (1.1), so that it is more powerful. This will be convincingly demonstrated in Figure 5 below. The factor-adjusted robust multiple test (FarmTest) is a robust implementation of the above idea (Fan et al., 2017a), which replaces the sample mean by its adaptive Huber estimation and extracts latent factors from a robust covariance input.

Fig 5.

Histograms of four different mean estimators for simultaneous inference. Fix n = 100, p = 500 and K = 3, and data are generated i.i.d. from t₃, which is heavy-tailed. Dashes lines correspond to μ_j = 0 and μ_j = 0.6, which is unknown. Robustification and factor adjustment help distinguish nulls and alternatives.

To begin with, we consider the Huber loss (Huber, 1964) with the robustification parameter τ ≥ 0:

$${\ell }_{\tau }\left(u\right)=\{\begin{array}{ll}{u}^2/2,\hfill & \text{if}\hspace{0.17em}\left|u\right|\le \tau \hfill \\ \tau \left|u\right|-{\tau }^2/2,\hfill & \text{if}\hspace{0.17em}\left|u\right|>\tau \hfill \end{array},$$

and use ${\widehat{\mu }}_j={\text{argmax}}_{\theta \in ℝ}{\sum }_{i=1}^n{\ell }_{\tau }\left(x_{ij}-\theta \right)$ as a robust M -estimator of μ_j. Fan et al. (2017a) suggested choosing $\tau \asymp \sqrt{n/\log \left(np\right)}$ to deal with possible asymmetric distribution and called it adaptive Huber estimator. They showed, assuming bounded fourth moments only, that

$$\sqrt{n}\left({\widehat{\mu }}_j-{\mu }_j-{\text{b}}_j^{\top }\overline{\mathbf{\text{f}}}\right)=\mathcal{N}\left(0,{\sigma }_{u,jj}\right)+o_{\mathbb{P}}\left(1\right)\hspace{0.17em}\text{uniformly over}\hspace{0.17em}j\in \left[p\right];$$ (3.10)

where $\overline{\mathbf{\text{f}}}=\frac{1}{n}{\sum }_{i=1}^n{\mathbf{\text{f}}}_i$, and σ_u,jj is the (j, j)th entry of Σ_u as is defined in (2.1). Assuming for now that ${\left\{{\text{b}}_j\right\}}_{j=1}^p$, $\overline{\mathbf{\text{f}}}$ and ${\left\{{\sigma }_{u,jj}\right\}}_{j=1}^p$ are all observable, then the factor-adjusted test statistic $T_j=\sqrt{n/{\sigma }_{u,jj}}\left({\widehat{\mu }}_j-{\text{b}}_j^{\top }\overline{\mathbf{\text{f}}}\right)$ is asymptotically $\mathcal{N}\left(0,1\right)$. The law of large numbers implies that V (z) should be close to 2p₀Φ(−z) for z ≥ 0, where Φ(·) is the cumulative distribution function of $\mathcal{N}\left(0,1\right)$, and p₀ = #{j : μ_j = 0} is the number of true nulls. Hence

$$\text{FDP}\left(z\right)=\frac{V\left(z\right)}{R\left(z\right)}\approx \frac{2p_0\Phi \left(-z\right)}{R\left(z\right)}\le \frac{2p\Phi \left(-z\right)}{R\left(z\right)}=:{\text{FDP}}^{\text{A}}\left(z\right).$$

Note that in the high-dimensional and sparse regime, we have p₀ = p − o(p) and thus FDP^A(z) is only a slightly conservative surrogate. However, we can also estimate the proportion π₀ = p₀/p and use less conservative estimate ${\text{FDP}}^{\text{A}}\left(z\right)=2p{\widehat{\pi }}_0\Phi \left(-z\right)/R\left(z\right)$ instead, where ${\widehat{\pi }}_0$ is an estimate of π₀ whose idea is depicted in Figure 4; see Storey (2002). Finally, we define the critical value z_α = inf{z ≥ 0 : FDP^A(z) ≤ α} and reject H_0j whenever |T_j| ≥ z_α.

Fig 4.

Estimation of proportion of true nulls. The observed P-values (right panel) consist of those from significant variables (genes), which are usually small, and those from insignificant variables, which are uniformly distributed. Assuming the P-values for significant variables are mostly less than λ (taken to be 0.5 in this illustration, left panel), the contributions of observed P-values > λ are mostly from true nulls and this yields a natural estimator ${\widehat{\pi }}_0\left(\lambda \right)=\frac{1}{(1-\lambda )}{\sum }_{j=1}^{p}1\left({\widehat{P}}_j>\lambda \right)$, which is the average height of the histogram with P-values > λ (red line). Note that the histograms above the red line estimates the distributions of P-values from the significant variavles (genes) in the left panel.

In practice, we have no access to ${\left\{{\mathbf{b}}_j\right\}}_{j=1}^p$, $\overline{\mathbf{\text{f}}}$ or ${\left\{{\sigma }_{u,jj}\right\}}_{j=1}^p$ in (3.10) and need to use their estimates. This results in the Factor-Adjusted Robust Multiple test (FarmTest) in Fan et al. (2017a). The inputs include ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$, a generic robust covariance matrix estimator $\widehat{\mathbf{\Sigma }}\in {ℝ}^{p\times p}$ from the data, a pre-specified level α ∈ (0, 1) for FDP control , the number of factors K, and the robustification parameters γ and ${\left\{{\tau }_j\right\}}_{j=1}^p$. Note that K can be estimated by the methods in Section 2.2, and overestimating K has little impact on final outputs.

Step 1. Denote by $\widehat{\mathbf{\Sigma }}\in {ℝ}^{p\times p}$ a generic robust covariance matrix estimator. Compute the eigen-decomposition of $\widehat{\mathbf{\Sigma }}$, set ${\left\{{\widehat{\lambda }}_j\right\}}_{j=1}^K$ to be its top K eigenvalues in descending order, and ${\left\{{\widehat{\text{v}}}_j\right\}}_{j=1}^K$ to be their corresponding eigenvectors. Let $\widehat{\text{B}}=\left({\tilde{\lambda }}_1^{1/2}{\widehat{\text{v}}}_1,\dots ,{\tilde{\lambda }}_1^{1/2}{\widehat{\text{v}}}_K\right)\in {ℝ}^{p\times K}$ where ${\tilde{\lambda }}_j=\max \left\{{\widehat{\lambda }}_j,0\right\}$, and denote its rows by ${\left\{{\widehat{\text{b}}}_j\right\}}_{j=1}^p$.

Step 2. Let ${\overline{x}}_j=\frac{1}{n}{\sum }_{i=1}^n{x}_{ij}$ for j ∈ [p] and $\widehat{\text{f}}={\text{argmax}}_{\text{f}\in {ℝ}^K}{\sum }_{j=1}^p{\ell }_{\gamma }\left({\overline{x}}_j-{\widehat{\text{b}}}_j^{\top }\text{f}\right)$. Construct factor-adjusted test statistics

$$T_j=\sqrt{n/{\widehat{\sigma }}_{u,jj}}({\widehat{\mu }}_j-{\widehat{\text{b}}}_j^{\top }\widehat{\text{f}})\hspace{1em}\text{for}\hspace{0.17em}\hspace{0.17em}\text{j}\in \left[p\right],$$ (3.11)

where ${\widehat{\sigma }}_{u,jj}={\widehat{\theta }}_j-{\widehat{\mu }}_j^2-{\Vert {\widehat{\text{b}}}_j\Vert }_2^2$, ${\widehat{\theta }}_j={\text{argmin}}_{\theta \ge {\widehat{\mu }}_j^2+{\Vert {\widehat{\text{b}}}_j\Vert }_2^2}{\ell }_{{\tau }_j}\left(x_{ij}^2-\theta \right)$.

Step 3. Calculate the critical value z_α = inf{z ≥ 0 : FDP^A(z) ≤ α}, where ${\text{FDP}}^{\text{A}}\left(z\right)=2{\widehat{\pi }}_{0}p\Phi \left(-z\right)/R\left(z\right)$, and reject H_0j whenever |T_j| ≥ z_α.

In Step 2, we estimate $\overline{\mathbf{\text{f}}}$ based on ${\overline{x}}_j={\mu }_j+{\widehat{\text{b}}}_j^{\top }\overline{\mathbf{\text{f}}}+{\overline{u}}_j$, which is implied by the factor model (1.1), and regard non-vanishing μ_j as an outlier. In the estimation of σ_u,jj, we used the identity ${\theta }_j:=\mathbb{E}{x}_{ij}^2={\mu }_j^2+{\Vert {\text{b}}_j\Vert }^2+{\sigma }_{u,jj}$ and robustly estimated the second moment θ_j.

Figure 5 is borrowed from Figure 1 in Fan et al. (2017a) that illustrates the effectiveness of this procedure. Here n = 100, p = 500, K = 3, ${\text{f}}_i~\mathcal{N}\left(\text{0},{\text{I}}_3\right)$ and the entries of u_i are generated independently from the t-distribution with 3 degrees of freedom. It is known that t—distributions are not sub-Guassian variables and are often used to model heavy-tailed data. The unknown means $\mu \in {ℝ}^p$ are fixed as μ_j = 0.6 for j ≤ 125 and μ_j = 0 otherwise. We plot the histograms of sample means, robust mean estimators, and their counterparts with factor-adjustment. The latent factors and heavy-tailed errors make it difficult to distinguish μ_j = 0.6 from μ_j = 0, and that explains why the sample means behave poorly. As is shown in Figure 5, better separation can be obtained by factor adjustment and robustification.

While existing literature usually imposes the joint normal assumption on ${\left\{{\text{f}}_i,{\text{u}}_i\right\}}_{i=1}^n$, the FarmTest only requires the coordinates of ${\left\{{\text{u}}_i\right\}}_{i=1}^n$ to have bounded fourth-order moments, and ${\left\{{\text{f}}_i\right\}}_{i=1}^n$ to be sub-Gaussian. Under standard regularity conditions for the approximate factor model, it is proved by Fan et al. (2017a) that

$${\text{FDA}}^{\text{A}}\left(z\right)-\text{FDP}\left(z\right)=o_{\mathbb{P}}\left(1\right).$$

We see that FDP^A is a valid approximation of FDP, which is therefore faithfully controlled by the FARM-Test.

3.4. Factor-Adjusted Robust Model (FARM) selection

Model selection is one of the central tasks in high dimensional data analysis. Parsimonious models enjoy interpretability, stability and oftentimes, better prediction accuracy. Numerous methods for model selection have been proposed in the past two decades, including, Lasso (Tibshirani, 1996), SCAD (Fan and Li, 2001), the elastic net Zou and Hastie (2005), the Dantzig selector (Candes and Tao, 2007), among others. However, these methods work only when the covariates are weakly dependent or statisfy certain regularity conditions (Zhao and Yu, 2006; Bickel et al., 2009). When covariates are strongly correlated, Paul et al. (2008); Kneip and Sarda (2011); Wang (2012); Fan et al. (2016a) used factor model to eliminate the dependencies caused by pervasive factors, and to conduct model selection using the resulting weakly correlated variables.

Assume that ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$ follow the approximate factor model (1.1). As a standard assumption, the coordinates of ${\mathbf{w}}_i={\left({\mathbf{f}}_i^{\top },{\mathbf{u}}_i^{\top }\right)}^{\top }\in {ℝ}^{K+p}$ are weakly dependent. Thanks to this condition and the decomposition

$${\mathbf{x}}_i^{\top }\mathbf{\beta }={\left(\mathit{\mu }+\mathbf{B}{\mathbf{f}}_i+{\mathbf{u}}_i\right)}^{\top }\mathbf{\beta }=\alpha +{\mathbf{u}}_i^{\top }\mathbf{\beta }+{\mathbf{f}}_i^{\top }\gamma$$ (3.12)

where α = μ^T β and γ = B^⊤β. we may treat w_i as the new predictors. In other words, by lifting the number of variables from p to p + K, the covariates of w_i are now weakly dependent. The usual regularized estimation can now be applied to this new set of variables. Note that we regard the coefficients B^⊤β as free parameters to facilitate the implementation (ignoring the relation γ = B^⊤β) and this requires an additional assumption to make this valid (Fan et al., 2016a).

Suppose we wish to fit a model $y_i=g\left({\mathbf{x}}_i^{\top }\mathbf{\beta },{\epsilon }_i\right)$ via a loss function $L\left(y_i,{\mathbf{x}}_i^{\top }\mathbf{\beta }\right)$. The above idea suggests the following two-step approach, which is called Factor-Adjusted Regularized (or Robust when so implemented) Model selection (FarmSelect) (Fan et al., 2016a).

Step 1: Factor estimation. Fit the approximate factor model (1.1) to get $\widehat{\mathbf{B}}$, ${\widehat{\mathbf{f}}}_i$ and ${\widehat{\mathbf{u}}}_i={\mathbf{x}}_i-\widehat{\mathbf{B}}{\widehat{\mathbf{f}}}_i$.

Step 2: Augmented regularization. Find α, β and γ to minimize

$$\sum _{i=1}^{n}L\left(y_i,\alpha +{\widehat{\mathbf{u}}}_i^{\top }\mathbf{\beta }+{\widehat{\mathbf{f}}}_i^{\top }\gamma \right)+\sum _{j=1}^p{p}_{\lambda }\left(\left|{\beta }_j\right|\right),$$

where p_λ(·) is a folded concave penalty (Fan and Li, 2001) with parameter λ.

In Step 1, standard estimation procedures such as POET (Fan et al., 2013) and S-POET (Wang and Fan, 2017) can be applied, as long as they produce consistent estimators of B, ${\left\{{\mathbf{f}}_i\right\}}_{i=1}^n$ and ${\left\{{\mathbf{u}}_i\right\}}_{i=1}^n$. Step 2 is carried out using usual regularization methods with new covariates.

Figure 6, borrowed from Figure 3 (a) in Fan et al. (2016a), shows that the proposed method outperforms other popular ones for model selection including Lasso (Tibshirani, 1996), SCAD (Fan and Li, 2001) and elastic net (Zou and Hastie, 2005), in the presence of correlated covariates. The basic setting is sparse linear regression y = x^⊤β* + ε with p = 500 and n growing from 50 to 160. The true coefficients are β* = (β₁, ··· , β₁₀, 0_p−10)^⊤, where ${\left\{{\mathbf{\beta }}_j\right\}}_{j=1}^{10}$ are drawn uniformly at random from [2, 5], and $\epsilon \sim \mathcal{N}\left(0,0.3\right)$. The correlation structure of covariates x is calibrated from S&P 500 monthly excess returns between 1980 and 2012.

Fig 6.

Model selection consistency rate, i.e., the proportion of simulations that the selected model is identical to the true one, with p = 500 and n varying from 50 to 160. With moderate sample size, the proposed method faithfully identifies the correct model while other methods cannot.

Under the generalized linear model, L(y, z) = −yz + b(z) and b(·) is a convex function. Fan et al. (2016a) analyzed theoretical properties of the above procedure. As long as the coordinates of w_i (rather than x_i) are not too strongly dependent and the factor model is estimated to enough precision, $\widehat{\mathbf{\beta }}$ enjoys optimal rates of convergence ${\Vert \widehat{\mathbf{\beta }}-{\mathbf{\beta }}^{\ast }\Vert }_q=O_{\mathbb{P}}\left({\left|S\right|}^{1/q}\sqrt{\log p/n}\right)$, where q = 1, 2 or ∞. When the minimum entry of |β*| is at least $\Omega \left(\sqrt{\log p/n}\right)$, the model selection consistency is achieved.

When we use the square loss, this method reduces to the one in Kneip and Sarda (2011). By using the square loss and replacing the penalized multiple regression in Step 2 with marginal regression, we recover the factor-profiled variable screening method in Wang (2012). While these papers aim at modeling and then eliminating the dependencies in x_i via (1.1), Paul et al. (2008) used a factor model to characterize the joint distribution of ${\left(y_i,{\mathbf{x}}_i^{\top }\right)}^{\top }$ and develops a related but different approach.

4. RELATED LEARNING PROBLEMS

4.1. Gaussian mixture model

PCA, or more generally, spectral decomposition, can be also applied to learn mixture models for heterogeneous data. A thread of recent papers (Hsu and Kakade, 2013; Anandkumar et al., 2014; Yi et al., 2016; Sedghi et al., 2016) apply spectral decomposition to lower-order moments of the data to recover the parameters of interest in a wide class of latent variable models. Here we use the Gaussian mixture model to illustrate their idea. Consider a mixture of K Gaussian distributions with spherical covariances. Let w_k ∈ (0, 1) be the probability of choosing component k ∈ {1, … , K}, and $\left\{{\mathit{\mu }}_1,\dots ,{\mathit{\mu }}_k\right\}\subseteq {ℝ}^p$ be the component mean vectors, and ${\left\{{\sigma }_k^2{\text{I}}_p\right\}}_{k=1}^K$ be the component covariance matrices, which is required by Hsu and Kakade (2013) and Anandkumar et al. (2014). Each data vector $\mathbf{x}~w_1\mathcal{N}\left({\mathit{\mu }}_1,{\sigma }_1^2{\text{I}}_p\right)+\cdots +w_K\mathcal{N}\left({\mathit{\mu }}_K,{\sigma }_K^2{\text{I}}_p\right)$ follows the mixture of the Gaussian distribution. The parameters of interest are ${\left\{w_k,{\mathit{\mu }}_k,{\sigma }_k^2\right\}}_{k=1}^K$.

Hsu and Kakade (2013) and Anandkumar et al. (2014) shed lights on the close connection between the lower-order moments of the data and the parameters of interest, which motivates the use of Method of Moments (MoM). Denote the population covariance $\mathbb{E}\left[\left(\mathbf{x}-\mathbb{E}\mathbf{x}\right){\left(\mathbf{x}-\mathbb{E}\mathbf{x}\right)}^{\top }\right]$ by Σ. Below we present Theorem 1 in Hsu and Kakade (2013) to elucidate the moment structure of the problem.

Theorem 4.1. Suppose that ${\left\{{\mathit{\mu }}_k\right\}}_{k=1}^K$ are linearly independent. Then the average variance ${\sigma }_{\text{ave}}^2:=K^{-1}\sum _{k=1}^K{\sigma }_k^2$ is the smallest eigenvalue of Σ. Let v be any eigenvector of Σ that is associated with the eigenvalue ${\sigma }_{\text{ave}}^2$. Define the following quantities:

$${\mathbf{M}}_1=\mathbb{E}\left[{\left({\mathbf{v}}^{\top }\left(\mathbf{x}-\mathbb{E}\mathbf{x}\right)\right)}^2\mathbf{x}\right]\in {ℝ}^p,$$

$${\mathbf{M}}_2=\mathbb{E}\left[\mathbf{x}\otimes \mathbf{x}\right]-{\sigma }_{\text{ave}}^2\cdot {\mathbf{I}}_p\in {ℝ}^{p\times p},$$

$${\mathbf{M}}_3=\mathbb{E}\left[\mathbf{x}\otimes \mathbf{x}\otimes \mathbf{x}\right]-\sum _{j=1}^p\left({\mathbf{M}}_1\otimes {\mathbf{e}}_j\otimes {\mathbf{e}}_j+{\mathbf{e}}_j\otimes {\mathbf{M}}_1\otimes {\mathbf{e}}_j+{\mathbf{e}}_j\otimes {\mathbf{e}}_j\otimes {\mathbf{M}}_1\right)\in {ℝ}^{p\times p\times p}.$$

Then we have

$${\mathbf{M}}_1=\sum _{k=1}^K{w}_k{\sigma }_k^2{\mathit{\mu }}_k,\hspace{1em}{\mathbf{M}}_2=\sum _{k=1}^K{w}_k{\mathit{\mu }}_k\otimes {\mathit{\mu }}_k,\hspace{1em}{\mathbf{M}}_3=\sum _{k=1}^K{w}_k{\mathit{\mu }}_k\otimes {\mathit{\mu }}_k\otimes {\mathit{\mu }}_k,$$ (4.1)

where the notation ⊗ represents the tensor product.

Theorem 4.1 gives the relationship between the moments of the first three orders of x and the parameters of interest. With ${\left\{{\mathbf{M}}_i\right\}}_{i=1}^3$ replaced by their empirical versions, the remaining task is to solve for all the parameters of interest via (4.1). Hsu and Kakade (2013) and Anandkumar et al. (2014) proposed a fast method called robust tensor power method to compute the estimators. The crux therein is to construct an estimable third-order tensor ${\tilde{\mathbf{M}}}_3$ that can be decomposed as the sum of orthogonal tensors based on μ_k. This orthogonal tensor decomposition can be regarded as an extension of spectral decomposition to third-order tensors (simply speaking, three-dimensional arrays). Then the power iteration method is applied to the estimate of ${\tilde{\mathbf{M}}}_3$ to recover each μ_k, as well as other parameters.

Specifically, consider first the following linear transformation of μ_k:

$${\tilde{\mathit{\mu }}}_k:=\sqrt{{\omega }_k}{\text{W}}^{\top }{\mathit{\mu }}_k$$ (4.2)

for k ∈ [K], where $\text{W}\in {ℝ}^{p\times K}$. The key is to use the whitening transformation by setting W to be a square root of M₂. This ensures that ${\left\{{\tilde{\mathit{\mu }}}_k\right\}}_{k=1}^K$ are orthogonal to each other. Denoting a^⊗3 = a ⊗ a ⊗ a,

$${\tilde{\mathbf{M}}}_3:=\sum _{k=1}^K{\omega }_k{\left({\text{W}}^{\top }{\mathit{\mu }}_k\right)}^{\otimes 3}=\sum _{k=1}^K\frac{1}{\sqrt{{\omega }_k}}{\tilde{\mathit{\mu }}}_k^{\otimes 3}\in {ℝ}^{K\times K\times K}$$ (4.3)

is an orthogonal tensor decomposition; that is, it satisfies orthogonality of ${\left\{{\tilde{\mathit{\mu }}}_k\right\}}_{k=1}^K$. The following theorem from Anandkumar et al. (2014) summarizes the above argument, and more importantly, it shows how to obtain μ_k back from ${\tilde{\mathit{\mu }}}_k$.

Theorem 4.2. Suppose the vectors ${\left\{{\tilde{\mathit{\mu }}}_k\right\}}_{k=1}^K$ are linearly independent, and the scalars ${\left\{{\omega }_k\right\}}_{k=1}^K$ are strictly positive. Let M₂ = UDU^⊤ be the spectral decomposition of M₂ and let W = UD^−1/2. Then ${\left\{{\tilde{\mathit{\mu }}}_k\right\}}_{k=1}^K$ in (4.2) are orthogonal to each other. Furthermore, the Moore-Penrose pseudo-inverse of W is ${\mathbf{W}}^{†}:={\text{D}}^{1/2}{\mathbf{u}}^{\top }\in {ℝ}^{K\times p}$, and we have ${\mathit{\mu }}_k={\left({\mathbf{W}}^{†}\right)}^{\top }{\tilde{\mathit{\mu }}}_k/\sqrt{{\omega }_k}$ for k ∈ [K].

As promised, the orthogonal tensor ${\tilde{\mathbf{M}}}_3$ can be estimated from empirical moments. We will make use of the following identity, which is similar to Theorem 4.1.

$${\tilde{\mathbf{M}}}_3=\mathbb{E}\left[{\left({\mathbf{W}}^{\top }\mathbf{x}\right)}^{\otimes 3}\right]-\sum _{j=1}^p\sum _{\text{cyc}}\left({\mathbf{W}}^{\top }{\mathbf{M}}_1\right)\otimes \left({\mathbf{W}}^{\top }{\mathbf{e}}_j\right)\otimes \left({\mathbf{W}}^{\top }{\mathbf{e}}_j\right),$$ (4.4)

where we used the cyclic sum notation

$$\sum _{\text{cyc}}\mathbf{a}\otimes \mathbf{b}\otimes \mathbf{c}:=\mathbf{a}\otimes \mathbf{b}\otimes \mathbf{c}+\mathbf{b}\otimes \mathbf{c}\otimes \mathbf{a}+\mathbf{c}\otimes \mathbf{a}\otimes \mathbf{b}.$$

Note that ${\mathbf{W}}^{\top }{\mathbf{e}}_j\in {ℝ}^K$ is simply the jth row of W. To obtain an estimate of ${\tilde{\mathbf{M}}}_3$, we replace the expectation $\mathbb{E}$ by the empirical average, and substitute W and M₁ by their plug-in estimates. It is worth mentioning that, because ${\tilde{\mathbf{M}}}_3$ has a smaller size than M₃, computations involving ${\tilde{\mathbf{M}}}_3$ can be implemented more efficiently.

Once we obtain an estimate of ${\tilde{\mathbf{M}}}_3$, which we denote by ${\overline{\mathbf{M}}}_3$, to recover ${\left\{{\mathit{\mu }}_k\right\}}_{k=1}^K$, ${\left\{{\omega }_k\right\}}_{k=1}^K$ and ${\left\{{\sigma }_k^2\right\}}_{k=1}^K$, the only task left is computing the orthogonal tensor decomposition (4.3) for ${\overline{\mathbf{M}}}_3$. The tensor power method in Anandkumar et al. (2014) is shown to solve this problem with provable computational guarantees. We omit the details of the algorithm here. Interested readers are referred to Section 5 of Anandkumar et al. (2014) for the introduction and analysis of this algorithm.

To conclude this subsection, we summarize the entire procedure of estimating ${\left\{{\mathit{\mu }}_k,{\sigma }_k,{\omega }_k\right\}}_{k=1}^K$ as below.

Step 1. Calculate the sample covariance matrix $\widehat{\mathbf{\Sigma }}:=n^{-1}\sum _{i=1}^n\left({\mathbf{x}}_i-\overline{\mathbf{x}}\right){\left({\mathbf{x}}_i-\overline{\mathbf{x}}\right)}^{\top }$, its minimum eigenvalue ${\widehat{\sigma }}_{\text{ave}}^2$ and its associated eigenvector $\widehat{\mathbf{v}}$.

Step 2. Derive the estimators ${\widehat{\mathbf{M}}}_1$, ${\widehat{\mathbf{M}}}_2$, ${\widehat{\mathbf{M}}}_3$ based on Theorem 4.1 by plug-in of empirical moments of x, $\widehat{\mathbf{v}}$ and ${\widehat{\sigma }}_{\text{ave}}^2$.

Step 3. Calculate the spectral decomposition ${\widehat{\mathbf{M}}}_2=\widehat{\mathbf{U}}\widehat{\text{D}}{\widehat{\mathbf{U}}}^{\top }$. Let $\widehat{\mathbf{W}}=\widehat{\mathbf{U}}{\widehat{\text{D}}}^{-1/2}$. Construct an estimator of ${\tilde{\mathbf{M}}}_3$, denoted by ${\overline{\mathbf{M}}}_3$, based on (4.4) by plug-in of empirical moments of ${\widehat{\mathbf{W}}}^{\top }\mathbf{x}$, $\widehat{\mathbf{W}}$ and ${\widehat{\mathbf{M}}}_1$. Apply the robust tensor power method in Anandkumar et al. (2014) to ${\overline{\mathbf{M}}}_3$ and obtain ${\left\{{\overline{\mathit{\mu }}}_k\right\}}_{k=1}^K$ and ${\left\{{\widehat{\omega }}_k\right\}}_{k=1}^K$.

Step 4. Set ${\widehat{\mathbf{W}}}^{†}={\widehat{\text{D}}}^{1/2}{\widehat{\mathbf{U}}}^{\top }$ and ${\widehat{\mathit{\mu }}}_k={\left({\widehat{\mathbf{W}}}^{†}\right)}^{\top }{\overline{\mathbf{\mu }}}_k/\sqrt{{\widehat{\omega }}_k}$. Solve the linear equation ${\widehat{\mathbf{M}}}_1=\sum _{k=1}^K{\widehat{\omega }}_k{\widehat{\sigma }}_k^2{\widehat{\mathit{\mu }}}_k$ for ${\left\{{\widehat{\sigma }}_k^2\right\}}_{k=1}^K$.

4.2. Community detection

In statistical modeling of networks, the stochastic block model (SBM), first proposed by Holland et al. (1983), has gained much attention in recent years (see Abbe, 2017 for a recent survey). Suppose our observation is a graph of n vertices, each of which belongs to one of K communities (or blocks). Let the vertices be indexed by [n], and the community that vertex i belongs to is indicated by an unknown θ_i ∈ [K]. In SBM, the probability of an edge between two vertices depends entirely on the membership of the communities. To be specific, let $\mathbf{W}\in {ℝ}^{K\times K}$ be a symmetric matrix where each entry takes value in [0, 1], and let $\mathbf{A}\in {ℝ}^{n\times n}$ be the adjacency matrix, i.e., A_ij = 1 if there is an edge between vertex i and j, and A_ij = 0 otherwise. Then, the SBM assumes

$$\mathbb{P}\left(A_{ij}=1\right)=W_{k\ell }\hspace{1em}\text{with}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\theta }_i=k,{\theta }_j=\ell$$

and {A_ij}_i>j are independent. Here, for ease of presentation, we allow self-connecting edges. Figure 7 gives one realization of the network with two communities.

Fig 7.

In both heatmaps, a dark pixel represents an entry with value 1 in a matrix, and a white pixel represents an entry with value 0. The left heatmap shows the (observed) adjacency matrix A of size n = 40 generated from the SBM with two equal-sized blocks (K = 2), with edge probabilities 5 log n/n (within blocks) and log n/(4n) (between blocks). The right heatmap shows the same matrix with its row indices and column indices suitably permutated based on unobserved z_i. Clearly, we observe an approximate rank-2 structure in the right heatmap. This motivates estimating z_i via the second eigenvector.

Though seemingly different, this problem shares a close connection with PCA and spectral methods. Let z_i = e_k (namely, the kth canonical basis in ${ℝ}^K$) if θ_i = k, indicating the membership of ith node, and define $\mathbf{Z}={\left[{\mathbf{z}}_1,\dots ,{\mathbf{z}}_n\right]}^{\top }\in {ℝ}^{n\times K}$. The expectation of A has a low-rank decomposition $\mathbb{E}\mathbf{A}=\text{ZW}{\text{Z}}^{\top }$ and

$$\mathbf{A}=\text{ZW}{\text{Z}}^{\top }+\left(\mathbf{A}\hspace{0.17em}–\hspace{0.17em}\mathbb{E}\mathbf{A}\right).$$ (4.5)

Loosely speaking, the matrix Z plays a similar role as factors or loading matrices (unnormalized), and $\mathbf{A}\hspace{0.17em}–\hspace{0.17em}\mathbb{E}\mathbf{A}$ is similar to the noise (idiosyncratic component). In the ideal situation, the adjacency matrix A and its expectation are close, and naturally we expect the eigenvectors of A to be useful for estimating θ_i. Indeed, this observation is the underpinning of many methods (Rohe et al., 2011; Gao et al., 2015; Abbe and Sandon, 2015). The vanilla spectral method for network/graph data is as follows:

Step 1. Construct the adjacency matrix A or other similarity-based matrices;

Step 2. Compute eigenvectors v₁, … , v_L corresponding to the largest eigenvalues, and form a matrix $\mathbf{V}=\left[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_{\ell }\right]\in {ℝ}^{n\times L}$;

Step 3. Run a clustering algorithm on the row vectors of V.

There are many variants and improvements of this vanilla spectral method. For example, in Step 1, very often the graph Laplacian D − A or normalized Laplacian D^−1/2(D − A)D^−1/2 is used in place of the adjacency matrix, where D = diag(d₁, … , d_n), and $d_i={\sum }_j{A}_{ij}$ is the degree of vertex i. If real-valued similarities or distances between vertices are available, weighted graphs are usually constructed. In Step 2, there are many other refinements over raw eigenvectors in the construction of V, for example, projecting row vectors of V onto the unit sphere (Ng et al., 2002), and calculating scores based on eigenvector ratios (Jin, 2015), etc. In Step 3, a very popular algorithm for clustering is the K-means algorithm.

We will look at the vanilla spectral algorithm in its simplest form. Our goal is exact recovery, which means finding an estimator $\widehat{\theta }$ of $\mathbf{\theta }={\left({\theta }_1,\dots ,{\theta }_n\right)}^{\top }$ such that as n → ∞,

$$\mathbb{P}\left(\text{there}\hspace{0.17em}\text{exists}\hspace{0.17em}\text{a}\hspace{0.17em}\text{permutation}\hspace{0.17em}\pi \hspace{0.17em}\text{of}\hspace{0.17em}\left[K\right]\hspace{0.17em}\text{s}.\text{t}.\hspace{0.17em}{\widehat{\theta }}_i=\pi \left({\theta }_i\right),\hspace{0.17em}\forall i\in \left[n\right]\right)=1-o\left(1\right).$$

Note that we can only determine θ up to a permutation since the distribution of our observation is invariant to permutations of [K]. There are nice theoretical results, including information limits for exact recovery in Abbe et al. (2016).

Despite its simplicity, spectral methods can be quite sharp for exact recovery in SBM, which succeed in a regime that matches the information limit. The next theorem from Abbe et al. (2017) will make this point clear. Consider the SBM with two balanced blocks, i.e., K = 2 and {i : θ_i = 1} = {i : θ_i = 2} = n/2, and suppose W₁₁ = W₂₂ = a log n/n, W₁₂ = b log n/n where a > b > 0. In this case, one can easily see that the second eigenvector of $\mathbb{E}\mathbf{A}$ is given by ${\mathbf{v}}_2^{\ast }$ whose ith entry is given by $1/\sqrt{n}$ if θ_i = 1 and −1 otherwise. In other words, $\mathrm{sgn}\left({\mathbf{v}}_2^{\ast }\right)$ classifies the two communities, where sgn(·) is the sign function applied to each entry of a vector. This is shown in Figure 2 for the case that #{i : θ_i = 1} = 2500 (red curve, left panel), where the second eigenvector v₂ of A is also depicted (blue curve). The entrywise closeness between these two quantities is guaranteed by the perturbation theory under ℓ_∞-norm (Abbe et al., 2017).

Theorem 4.3. Let v₂ be the normalized second eigenvector of A. If $\sqrt{a}-\sqrt{b}<\sqrt{2}$, then no estimator achieves exact recovery; if $\sqrt{a}-\sqrt{b}>\sqrt{2}$, then both the maximum likelihood estimator and the eigenvector estimator sgn(v₂) achieves exact recovery.

The proof of this result is based on entry-wise analysis of eigenvectors in a spirit similar to Theorem 2.4, together with a probability tail bound for differences of binomial variables.

4.3. Matrix completion

In recommendation systems, an important problem is to estimate users’ preferences based on history data. Usually, the available data per user is very small compared with the total number of items (each user sees only a small number of movies and buys only a small fraction of books, comparing to the total). Matrix completion is one formulation of such problem.

The goal of (noisy) matrix completion is to estimate a low-rank matrix ${\mathbf{M}}^{\ast }\in {ℝ}^{n_1\times n_2}$ from noisy observations of some entries (n₁ users and n₂ items). Suppose we know rank(M*) = K. For each i ∈ [n₁] and j ∈ [n₂], let I_ij be i.i.d. Bernoulli variable with $\mathbb{P}\left(I_{ij}=1\right)=p$ that indicates if we have observed information about the entry $M_{ij}^{\ast }$, i.e., I_ij = 1 if and only if it is observed. Also suppose that our observation is $M_{ij}=M_{ij}^{\ast }+{\epsilon }_{ij}$ if I_ij = 1, where ε_ij is i.i.d. $\mathcal{N}\left(0,{\sigma }^2\right)$ jointly independent of I_ij.

One natural way to estimate M* is to solve

$$\underset{\mathbf{X}\in {ℝ}^{n_1\times n_2}}{\min }\frac{1}{2}{\Vert {\mathcal{P}}_{\Omega }\left(\mathbf{M}\right){\mathcal{P}}_{\Omega }\left(\mathbf{X}\right)\Vert }^2\hspace{1em}\text{subject}\hspace{0.17em}\text{to}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{rank}\left(\mathbf{X}\right)=K,$$

where ${\mathcal{P}}_{\Omega }:{ℝ}^{n_1\times n_2}\to {ℝ}^{n_1\times n_2}$ is the sampling operator defined by ${\left[{\mathcal{P}}_{\Omega }\left(\mathbf{x}\right)\right]}_{ij}=I_{ij}{X}_{ij},\hspace{0.17em}\forall i,j$. The minimizer of this problem is essentially the MLE for M *. Due to the nonconvex constraint rank(X) = K, it is desirable to relax this optimization into a convex program. A popular way to achieve that is to transform the rank constraint into a penalty term λǁXǁ_* that is added to the quadratic objective function, where λ is a tuning parameter and ${\Vert \cdot \Vert }_{\ast }$ is the nuclear norm (that is, the ℓ₁ norm of the vector of all its singular values), which encourages a solution with low rank (number of nonzero components in that vector). A rather surprising conclusion from Candès and Recht (2009) is that in the noiseless setting, solving the relaxed problem yields the same solution as the nonconvex problem with high probability.

We can view this problem from the perspective of factor models. The assumption that M* has low rank can be justified by interpreting each M_ij as the linear combination of a few latent factors. Indeed, if M_ij is the preference score of user i for item j, then it is reasonable to posit $M_{ij}={\mathbf{b}}_i^{\top }{\mathbf{f}}_j$, where ${\mathbf{f}}_j\in {ℝ}^K$ is the features item j possesses and ${\mathbf{b}}_i\in {ℝ}^K$ is the tendency of user i towards the features. In this regard, M* = BF^⊤ can be viewed as the part explained by the factors in the factor models.

This discussion motivates us to write our observation as

$${\mathcal{P}}_{\Omega }\left(\mathbf{M}\right)=p{\mathbf{M}}^{\ast }+\left({\mathcal{P}}_{\Omega }\left({\mathbf{M}}^{\ast }\right)-\mathbb{E}{\mathcal{P}}_{\Omega }\left({\mathbf{M}}^{\ast }\right)+{\mathcal{P}}_{\Omega }\left(\mathbf{E}\right)\right),\hspace{1em}\text{where}\hspace{0.17em}\mathbf{E}:={\left({\epsilon }_{ij}\right)}_{i,j}\in {ℝ}^{n_1\times n_2},$$

since $\mathbb{E}{\mathcal{P}}_{\Omega }\left({\mathbf{M}}^{\ast }\right)=p{\mathbf{M}}^{\ast }$. This decomposition gives the familiar “low-rank plus noise” structure. It is natural to conduct PCA on ${\mathcal{P}}_{\Omega }\left(\mathbf{M}\right)$ to extract the low-rank part.

Let the best rank-K approximation of ${\mathcal{P}}_{\Omega }\left(\mathbf{M}\right)$ be given by $\mathbf{U}\text{diag}\left({\sigma }_1,\dots ,{\sigma }_K\right){\mathbf{V}}^{\top }$, where ${\left\{{\sigma }_k\right\}}_{k=1}^K$ are the largest K singular values in descending order, and columns of $\mathbf{U}\in {ℝ}^{n_1\times K}$, $\mathbf{V}\in {ℝ}^{n_2\times K}$ correspond to their normalized left and right singular vectors, respectively. Similarly, we have singular value decomposition ${\mathbf{M}}^{\ast }={\mathbf{U}}^{\ast }\text{diag}\left({\sigma }_1^{\ast },\dots ,{\sigma }_K^{\ast }\right){\left({\mathbf{V}}^{\ast }\right)}^{\top }$. The following result from Abbe et al. (2017) provides entry-wise bounds for our estimates. For a matrix, denote by ${\Vert \cdot \Vert }_{\max }$ the largest absolute value of all entries, and ${\Vert \cdot \Vert }_{2\to \infty }$ the largest ℓ₂ norm of all row vectors.

Theorem 4.4. Let n = n₁ + n₂, $\eta =\max \left\{{\Vert \mathbf{U}\Vert }_{2\to \infty },{\Vert \mathbf{V}\Vert }_{2\to \infty }\right\}$ and $\kappa ={\sigma }_1^{\ast }/{\sigma }_K^{\ast }$. There exist constants C, C′ > 0 and an orthogonal matrix $\text{R}\in {ℝ}^{K\times K}$ such that the following holds. If p ≥ 6 log n/n and . There exist constants C, C′ > 0 and an orthogonal matrix $\kappa \frac{n\left({\Vert {\mathbf{M}}^{\ast }\Vert }_{\max }+\sigma \right)}{{\sigma }_r^{\ast }}\sqrt{\frac{\log n}{np}}\le 1/C$, then with at least probability 1 − C/n,

$$\text{max}\left\{{\Vert \mathbf{U}\mathbf{R}-{\mathbf{U}}^{\ast }\Vert }_{\max },{\Vert \mathbf{V}\mathbf{R}-{\mathbf{V}}^{\ast }\Vert }_{\max }\right\}\le C^{\prime }\eta \kappa \frac{n\left({\Vert {\mathbf{M}}^{\ast }\Vert }_{\max }+\sigma \right)}{{\sigma }_r^{\ast }}\sqrt{\frac{\log n}{np}},$$

$${\Vert \mathbf{U}\text{diag}\left\{{\sigma }_1,\dots ,{\sigma }_K\right\}{\mathbf{V}}^{\top }-{\mathbf{M}}^{\ast }\Vert }_{\max }\le C^{\prime }{\eta }^2{\kappa }^4\left({\Vert {\mathbf{M}}^{\ast }\Vert }_{\max }+\sigma \right)\sqrt{\frac{n\log n}{p}}.$$

We can simplify the bounds with a few additional assumptions. If $n_1\asymp n_2$, then η is of order $O\left(\sqrt{K/n}\right)$ assuming a bounded coherence number. In addition, if κ is also bounded, then

$${\Vert \mathbf{U}\text{diag}\left\{{\sigma }_1,\dots ,{\sigma }_K\right\}{\mathbf{V}}^{\top }-{\mathbf{M}}^{\ast }\Vert }_{\max }\lesssim \left({\Vert {\mathbf{M}}^{\ast }\Vert }_{\max }+\sigma \right)\sqrt{\frac{\log n}{np}}.$$

We remark that the requirement on the sample ratio $p\gtrsim \log n/n$ is the weakest condition necessary for matrix completion, which ensures each row and column and sampled with high probability. Also, the entry-wise bound above can recover the Frobenius bound (Keshavan et al., 2010) up to a log factor. It is more precise than the Frobenius bound, because the latter only provides control on average error.

4.4. Synchronization problems

Synchronization problems are a class of problems in which one estimates signals from their pairwise comparisons. Consider the phase synchronization problem as an example, that is, estimating n angles θ₁, … , θ_n from noisy measurements of their differences. We can express an angle θ_ℓ in the equivalent form of a unit-modulus complex number z_ℓ = exp(iθ_ℓ), and thus, the task is to estimate a complex vector $\text{z}={\left(z_1,\dots ,z_n\right)}^{\top }\in {ℂ}^n$. Suppose our measurements have the form $C_{\ell k}={\overline{z}}_{\ell }{z}_k+\sigma w_{\ell k}$, where ${\overline{z}}_{\ell }$ denotes the conjugate of z_ℓ, and for all ℓ > k, $w_{\ell k}\in ℂ$ is i.i.d. complex Gaussian variable (namely, the real part and imaginary part of w_ℓk are $\mathcal{N}\left(0,1/2\right)$ and independent). Then, the phase of C_ℓk (namely arg(C_ℓk)) encodes the noisy difference θ_k − θ_A.

More generally, the goal of a synchronization problem is to estimate n signals from their pairwise measurements, where each signal is an element from a group, e.g., the group of rotations in three dimensions. Synchronization problems are motivated from imaging problems such as cryo-EM (Shkolnisky and Singer, 2012), camera calibration (Tron and Vidal, 2009), etc.

Synchronization problems also admit the “low-rank plus noise” structure. Consider our phase synchronization problem again. If we let w_kℓ = w_ℓk (ℓ > k) and w_ℓℓ = 0, and write $\mathbf{W}={\left(w_{\ell k}\right)}_{\ell ,k=1}^n$, then our measurement matrix $\mathbf{C}={\left(C_{\ell k}\right)}_{\ell ,k=1}^n$ has the structure

$$\mathbf{C}=\mathbf{z}{\mathbf{z}}^{\ast }+\sigma \mathbf{W},$$

where * denotes the conjugate transpose. This decomposition has a similar form to (4.5) in community detection. Note that zz* is a complex matrix with a single nonzero eigenvalue n, and ${\Vert \sigma \mathbf{W}\Vert }_2$ is of order $\sigma \sqrt{n}$ with high probability (which is a basic result in random matrix theory). Therefore, we expect that no estimators can do well if $\sigma \gtrsim \sqrt{n}$. Indeed, the information-theoretic limit is established in Lelarge and Miolane (2016). Our next result from Zhong and Boumal (2018) gives estimation guarantees if the reverse inequality is true (up to a log factor).

Theorem 4.5. Let $\mathbf{v}\in {ℂ}^n$ be the leading eigenvector of C such that ${\Vert \mathbf{v}\Vert }_2=\sqrt{n}$ and ${\mathbf{v}}^{\ast }\text{z}=\left|{\mathbf{v}}^{\ast }\text{z}\right|$. Then, if $\sigma \lesssim \sqrt{n/\log n}$, then with probability 1 − O(n⁻²), the relative errors satisfy

$$n^{-1/2}{\Vert \mathbf{v}-\text{z}\Vert }_2\lesssim \sigma /\sqrt{n},\hspace{1em}and\hspace{1em}{\Vert \mathbf{v}-\text{z}\Vert }_{\infty }\lesssim \sigma \sqrt{\log n/n}$$

Moreover, the above two inequalities also hold for the maximum likelihood estimator.

Note that the eigenvector of a complex matrix is not unique: for any $\alpha \in ℝ$, the vector e^iαv is also an eigenvector, so we fix the global phase e^iα by restricting ${\mathbf{v}}^{\ast }\text{z}=\left|{\mathbf{v}}^{\ast }\text{z}\right|$. Note also that the maximum likelihood estimator is different from v, because the MLE must satisfy the entry-wise constraint |z_ℓ| = 1 for any ℓ ∈ [n]. This result implies consistency of v in terms of both the ℓ₂ norm and the ℓ_∞ norm if $\sigma \ll \sqrt{n/\log n}$, and thus, provides good evidence that spectral methods (or PCA) are simple, generic, yet powerful.

APPENDIX A: PROOFS

Proof of Corollary 2.1. Notice that the result is trivial if ${\delta }_0\le 2{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2$, since ${\Vert \left(\tilde{\mathbf{A}}-\mathbf{A}\right)\mathbf{V}\Vert }_2\le {\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2$ and ${\Vert \sin \Theta \left(\tilde{\mathbf{A}},\mathbf{A}\right)\Vert }_2\le 1$ always hold. If ${\delta }_0>2{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2$, then by Weyl’s inequality,

$$\mathcal{L}\left({\tilde{\mathbf{V}}}^{\perp }\right)\subset \left(-\infty ,\alpha -{\delta }_0+{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2\right]\cup \left[\beta +{\delta }_0-{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2,+\infty \right).$$

Thus, we can set $\delta ={\delta }_0-{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2$ in Theorem 2.3 and derive

$${\Vert \sin \Theta \left(\tilde{\mathbf{V}},\mathbf{V}\right)\Vert }_2\le \frac{{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2}{{\delta }_0-{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2}\le \frac{{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2}{{\delta }_0-{\delta }_0/2}=2{\delta }_0^{-1}{\Vert \tilde{\mathbf{A}}-\mathbf{A}\Vert }_2.$$

This proves the spectral norm case.

Proof of Theorem 2.4. Step 1: First, we derive a few elementary inequalities: for any m ∈ [n],

$${\Vert {\mathbf{W}}^{\left(m\right)}\Vert }_2\le {\Vert \mathbf{W}\Vert }_2,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\Vert {\mathbf{W}}_m\Vert }_2\le {\Vert \mathbf{W}\Vert }_2,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\Vert \mathbf{W}-{\mathbf{W}}^{\left(m\right)}\Vert }_2\le 2{\Vert \mathbf{W}\Vert }_2.$$ (A.1)

To prove these inequalities, recall the (equivalent) definition of spectral norm for symmetric matrices:

$${\Vert \mathbf{W}\Vert }_2=\underset{\mathit{x},\mathit{y}\in S^{n-1}}{\max }{\mathit{x}}^{\top }\mathbf{W}\mathit{y}=\underset{\mathit{x}\in S^{n-1}}{\max }{\Vert \mathbf{W}\mathit{x}\Vert }_2,$$

where Sⁿ⁻¹ is the unit sphere in ${ℝ}^n$, and x = (x₁, … , x_n)^⊤, y = (y₁, … , y_n)^⊤. The first and second inequalities follow from

$${\Vert \mathbf{W}\Vert }_2\ge \max \left\{{\mathit{x}}^{\top }\mathbf{W}\mathit{y};\mathit{x},\mathit{y}\in S^{n-1},x_m=y_m=0\right\}={\Vert {\mathbf{W}}^{\left(m\right)}\Vert }_2,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}$$

$${\Vert \mathbf{W}\Vert }_2\ge \underset{\mathit{x}\in S^{n-1}}{\max }\left|\langle {\mathbf{W}}_m,\mathit{x}\rangle \right|={\Vert {\mathbf{W}}_m\Vert }_2.$$

The third inequality follows from the first one and the triangle inequality.

Step 2: Next, by the definition of eigenvectors,

$${\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }=\frac{\tilde{\mathbf{A}}{\tilde{\mathbf{v}}}_{\ell }}{{\tilde{\lambda }}_{\ell }}-{\mathbf{v}}_{\ell }=\left(\frac{\mathbf{A}{\tilde{\mathbf{v}}}_{\ell }}{{\tilde{\lambda }}_{\ell }}-{\mathbf{v}}_{\ell }\right)+\frac{\mathbf{W}{\tilde{\mathbf{v}}}_{\ell }}{{\tilde{\lambda }}_{\ell }}.$$ (A.2)

We first control the entries of the first term on the right-hand side. Using the decomposition (2.8), we have

$$\left[\frac{\mathbf{A}{\tilde{\mathbf{v}}}_{\ell }}{{\tilde{\lambda }}_{\ell }}-{\mathbf{v}}_{\ell }\right]=\left(\frac{{\lambda }_{\ell }}{{\tilde{\lambda }}_{\ell }}\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle -1\right){\left[{\mathbf{v}}_{\ell }\right]}_m+\sum _{k\ne \ell }\frac{{\lambda }_{\ell }}{{\tilde{\lambda }}_{\ell }}\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle {\left[{\mathbf{v}}_k\right]}_m,\hspace{1em}\forall m\in \left[n\right].$$ (A.3)

Using the triangle inequality, we have

$$\left|\frac{{\lambda }_{\ell }}{{\tilde{\lambda }}_{\ell }}\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle -1\right|\le \left|\frac{{\lambda }_{\ell }}{{\tilde{\lambda }}_{\ell }}\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle -\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle \right|+\left|\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle -1\right|\le \frac{\left|{\lambda }_{\ell }-{\tilde{\lambda }}_{\ell }\right|}{\left|{\tilde{\lambda }}_{\ell }\right|}+\frac{1}{2}{\Vert {\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }\Vert }^2.$$

By Weyl’s inequality, $\left|{\tilde{\lambda }}_{\ell }-{\lambda }_{\ell }\right|\le {\Vert \mathbf{W}\Vert }_2$, and thus $\left|{\tilde{\lambda }}_{\ell }\right|\ge \left|{\lambda }_{\ell }\right|-{\Vert \mathbf{W}\Vert }_2\ge {\delta }_{\ell }-{\Vert \mathbf{W}\Vert }_2$. Also, by Corollary 2.1 (simplified Davis-Kahan’s theorem) and its following remark, ${\Vert {\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }\Vert }_2\le 2\sqrt{2}{\Vert \mathbf{W}\Vert }_2/{\delta }_{\ell }$. Therefore, under the condition δ_ℓ ≥ 2ǁWǁ₂,

$$\left|\frac{{\lambda }_{\ell }}{{\tilde{\lambda }}_{\ell }}\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle -1\right|\le \frac{{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }-{\Vert \mathbf{W}\Vert }_2}+\frac{4{\Vert \mathbf{W}\Vert }_2^2}{{\delta }_{\ell }^2}\le \frac{2{\Vert \mathbf{W}\Vert }_2^2}{{\delta }_{\ell }}+\frac{2{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }}=\frac{4{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }}.$$

Using Corollary 2.1 again, we obtain

$$\sum _{k\ne \ell ,k\le K}\frac{{\lambda }_k^2}{{\tilde{\lambda }}_{\ell }^2}{\langle {\mathbf{v}}_k,{\tilde{\mathbf{v}}}_{\ell }\rangle }^2\lesssim \sum _{k\ne \ell ,k\le K}{\langle {\mathbf{v}}_k,{\tilde{\mathbf{v}}}_{\ell }\rangle }^2\le 1-{\langle {\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\rangle }^2={\sin }^2\theta \left({\mathbf{v}}_{\ell },{\tilde{\mathbf{v}}}_{\ell }\right)\le \frac{4{\Vert \mathbf{W}\Vert }_2^2}{{\delta }_{\ell }^2},$$

where the first inequality is due to $\left|{\tilde{\lambda }}_{\ell }\right|\ge \left|{\lambda }_{\ell }\right|-{\Vert \mathbf{W}\Vert }_2\ge 4\left|{\lambda }_{\ell }\right|/5$ and the condition $\left|{\lambda }_{\ell }\right|\asymp {\max }_{k\in \left[K\right]}\left|{\lambda }_k\right|$, and the second inequality is due to the fact that ${\left\{{\mathbf{v}}_k\right\}}_{k=1}^K$ is a subset of orthonormal basis. Now we use the Cauchy-Schwarz inequality to bound the second term on the right-hand side of (A.3) and get

$$\left|{\left[\frac{\mathbf{A}{\tilde{\mathbf{v}}}_{\ell }}{{\tilde{\lambda }}_{\ell }}-{\mathbf{v}}_{\ell }\right]}_m\right|\lesssim \frac{{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }}{\left(\sum _{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}.$$ (A.4)

Step 3: To bound the entries of the second term in (A.2), we use the leave-one-out idea as follows.

$${\left[\mathbf{W}{\tilde{\mathbf{v}}}_{\ell }\right]}_m={\left[\mathbf{W}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m+{\left[\mathbf{W}\left({\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right)\right]}_m=\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle +\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle ,\hspace{1em}\forall m\in \left[n\right].$$ (A.5)

We can bound the second term using the Cauchy-Schwarz inequality: $\left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|\le {\Vert {\mathbf{w}}_m\Vert }_2{\Vert {\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2$. The crucial observation is that, if we view ${\tilde{\mathbf{v}}}_{\ell }$ as the perturbed version of ${\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}$, then by Theorem 2.3 (Davis-Kahan’s theorem) and Weyl’s inequality, for any ℓ ∈ [K],

$${\Vert {\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2\le \frac{\sqrt{2}{\Vert {\mathbf{\Delta }}^{\left(m\right)}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2}{{\tilde{\delta }}_{\ell }^{\left(m\right)}-{\Vert {\mathbf{\Delta }}^{\left(m\right)}\Vert }_2},\hspace{1em}\text{where}\hspace{0.17em}{\mathbf{\Delta }}^{\left(m\right)}:=\mathbf{W}-{\mathbf{W}}^{\left(m\right)}.$$

Here, ${\tilde{\delta }}_{\ell }^{\left(m\right)}$ is the eigen-gap of A + W^(m), and it satisfies ${\tilde{\delta }}_{\ell }^{\left(m\right)}\ge {\delta }_{\ell }-2{\Vert {\mathbf{W}}^{\left(m\right)}\Vert }_2$ since $\left|{\tilde{\lambda }}_i^{\left(m\right)}-{\lambda }_i\right|\le {\Vert {\mathbf{W}}^{\left(m\right)}\Vert }_2$ for all i ∈ [n], by Weyl’s inequality. By (A.1), we have ${\tilde{\delta }}_{\ell }^{\left(m\right)}-{\Vert {\mathbf{\Delta }}^{\left(m\right)}\Vert }_2\ge {\delta }_{\ell }-4{\Vert \mathbf{W}\Vert }_2$. Thus, under the condition δ_ℓ ≥ 5ǁWǁ₂, we have

$${\Vert {\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2\lesssim \frac{{\Vert {\mathbf{\Delta }}^{\left(m\right)}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2}{{\delta }_{\ell }}.$$

Note that the mth entry of the vector ${\mathbf{\Delta }}^{\left(m\right)}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}$ is exactly $\langle {\mathbf{W}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle$, and other entries are $W_{im}{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m$ where $i\ne m$. Thus,

$${\Vert {\tilde{\mathbf{v}}}_{\ell }-{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\Vert }_2\lesssim \frac{1}{{\delta }_{\ell }}{\left({\langle {\mathbf{W}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle }^2+\sum _{i\ne m}{W}^2{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m^2\right)}^{1/2}\le \frac{1}{{\delta }_{\ell }}\left(\left|\langle {\mathbf{W}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|+{\Vert {\mathbf{W}}_m\Vert }_2\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right|\right),$$

where we used $\sqrt{a+b}\le \sqrt{a}+\sqrt{b}\left(a,b\ge 0\right)$. The above inequality, together with $\left|\langle {\text{w}}_m,{\tilde{\text{v}}}_{\ell }-{\tilde{\text{v}}}_{\ell }^{\left(m\right)}\rangle \right|\le {\Vert {\text{w}}_m\Vert }_2{\Vert {\tilde{\text{v}}}_{\ell }-{\tilde{\text{v}}}_{\ell }^{\left(m\right)}\Vert }_2$, leads to a bound on ${\left[\mathbf{W}{\tilde{\mathbf{v}}}_{\ell }\right]}_m$ in (A.5).

$$\begin{gathered} \left|{\left[\mathbf{W}{\tilde{\mathbf{v}}}_{\ell }\right]}_m\right|\lesssim \left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|+\frac{{\Vert {\mathbf{w}}_m\Vert }_2}{{\delta }_{\ell }}\left(\left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|+{\Vert {\mathbf{w}}_m\Vert }_2\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right|\right) \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\lesssim \left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|+{\Vert {\mathbf{w}}_m\Vert }_2\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right| \end{gathered}$$ (A.6)

where we used ${\delta }_{\ell }^{-1}{\Vert {\mathbf{w}}_m\Vert }_2\le {\delta }_{\ell }^{-1}{\Vert \mathbf{W}\Vert }_2<1$. We claim that $\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right|\lesssim {\left({\sum }_{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}$. Once this is proved, combining it with (A.4) and (A.6) yields the desired bound on the entries of ${\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }$ in (A.2):

$$\begin{gathered} \left|{\left[{\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }\right]}_m\right|\lesssim \frac{{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }}{\left(\sum _{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}+\frac{1}{{\delta }_{\ell }}\left(\left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|+{\Vert {\mathbf{w}}_m\Vert }_2\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right|\right) \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}\lesssim \frac{{\Vert \mathbf{W}\Vert }_2}{{\delta }_{\ell }}{\left(\sum _{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}+\frac{\left|\langle {\mathbf{w}}_m,{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\rangle \right|}{{\delta }_{\ell }}, \end{gathered}$$

where, in the first inequality, we used $\left|{\tilde{\lambda }}_{\ell }\right|\ge \left|{\lambda }_{\ell }\right|-{\Vert \mathbf{W}\Vert }_2\ge {\delta }_{\ell }-{\delta }_{\ell }/5=4{\delta }_{\ell }/5$, and in the second inequality, we used ${\Vert {\mathbf{W}}_m\Vert }_2\le {\Vert \mathbf{W}\Vert }_2$ and the claim.

Step 4: Finally, we prove our claim that $\left|{\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m\right|\lesssim {\left({\sum }_{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}$. By definition, ${\tilde{\lambda }}_{\ell }^{\left(m\right)}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}=\left(\text{A+}{\mathbf{W}}^{\left(m\right)}\right){\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}$. Note that the mth row of ${\mathbf{W}}^{\left(m\right)}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}$ is 0, since W^(m) has only zeros in its mth row. Thus,

$${\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m=\left({\left[{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}\right]}_m-{\left[{\mathbf{v}}_{\ell }\right]}_m\right)+{\left[{\mathbf{v}}_{\ell }\right]}_m={\left[\frac{\mathbf{A}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}}{{\tilde{\lambda }}_{\ell }^{\left(m\right)}}-{\mathbf{v}}_{\ell }\right]}_m+{\left[{\mathbf{v}}_{\ell }\right]}_m.$$

With an argument similar to the one that leads to (A.4), we can bound the first term on the right-hand side.

$$\left|{\left[\frac{\mathbf{A}{\tilde{\mathbf{v}}}_{\ell }^{\left(m\right)}}{{\tilde{\lambda }}_{\ell }^{\left(m\right)}}-{\mathbf{v}}_{\ell }\right]}_m\right|\lesssim \frac{{\Vert {\mathbf{W}}^{\left(m\right)}\Vert }_2}{{\delta }_{\ell }}{\left(\sum _{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}\le {\left(\sum _{k=1}^K{\left[{\mathbf{v}}_k\right]}_m^2\right)}^{1/2}.$$

Clearly, $\left|{\left[{\mathbf{v}}_{\ell }\right]}_m\right|$ is also upper bounded by the right-hand side above. This proves our claim and concludes the proof.

Proof of Corollary 2.2. Let us construct symmetric matrices A, W, $\tilde{\mathbf{W}}$ of size n + p via a standard dilation technique (Paulsen, 2002). Define

$$\mathbf{A}=\left(\begin{array}{cc}\mathbf{0}& \mathbf{L}\\ {\mathbf{L}}^{\top }& \mathbf{0}\end{array}\right),\hspace{1em}\mathbf{W}=\left(\begin{array}{cc}\mathbf{0}& \mathbf{E}\\ {\mathbf{E}}^{\top }& \mathbf{0}\end{array}\right),\hspace{1em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{W}.$$

It can be checked that rank(A) = 2K, and importantly,

$$\mathbf{A}=\frac{1}{2}\sum _{k=1}^K{\sigma }_k\left(\begin{array}{l}{\mathbf{u}}_k\\ {\mathbf{v}}_k\end{array}\right)\left(\begin{array}{cc}{\mathbf{u}}_k^{\top }& {\mathbf{v}}_k^{\top }\end{array}\right)-\frac{1}{2}\sum _{k=1}^K{\sigma }_k\left(\begin{array}{c}{\mathbf{u}}_k\\ -{\mathbf{v}}_k\end{array}\right)\left(\begin{array}{cc}{\mathbf{u}}_k^{\top }& -{\mathbf{v}}_k^{\top }\end{array}\right).$$ (A.7)

Step 1: Check the conditions of Theorem 2.4. The nonzero eigenvalues of A are ±σ_k, (k ∈ [K]), and the corresponding eigenvectors are ${\left({\mathbf{u}}_k^{\top },\pm {\mathbf{v}}_k^{\top }\right)}^{\top }/\sqrt{2}\in {ℝ}^{n+p}$. It is clear that the eigenvalue condition $\left|{\lambda }_{\ell }\right|\asymp {\max }_{k\in \left[K\right]}\left|{\lambda }_k\right|$ in Theorem 2.4 is satisfied, and the eigen-gap δ_ℓ of A is exactly γ_ℓ. Since the identity (A.7) holds for any matrix constructed from dilation, by applying it to W we get ${\Vert \mathbf{W}\Vert }_2={\Vert \mathbf{E}\Vert }_2$.

Step 2: Apply the conclusion of Theorem 2.4. Similarly as before, we write W^(m) as the matrix obtained by setting mth row and mth column of W to zero, where m ∈ [n + p]. We also denote ${\tilde{\mathbf{A}}}^{\left(m\right)}=\mathbf{A}+{\mathbf{W}}^{\left(m\right)}$. Using a similar argument as Step 1, we find

1. the eigenvectors of $\tilde{\mathbf{A}}$ are $\left(\begin{array}{c}{\tilde{\mathbf{u}}}_k\\ \pm {\tilde{\mathbf{v}}}_k\end{array}\right)/\sqrt{2}$,

2. the eigenvectors of ${\tilde{\mathbf{A}}}^{\left(i\right)}$ are $\left(\begin{array}{c}\ast \\ \pm {\tilde{\mathbf{v}}}_k^{\left(i\right)}\end{array}\right)/\sqrt{2},\hspace{0.17em}\forall i\in \left[n\right]$, and

3. the eigenvectors of ${\tilde{\mathbf{A}}}^{\left(n+j\right)}$ are $\left(\begin{array}{c}{\tilde{\mathbf{u}}}_k^{\left(j\right)}\\ \ast \end{array}\right)/\sqrt{2},\hspace{0.17em}\forall j\in \left[p\right]$,

where * means some appropriate vectors we do not need in the proof (we do not bother introducing notations for them). We also observe that

$${\mathbf{W}}_m=\{\begin{array}{ll}{\left(\begin{array}{cc}\text{0}& {\mathbf{e}}_i^{\text{row}}\end{array}\right)}^{\top },\hfill & m=i\in \left[n\right]\hfill \\ {\left(\begin{array}{cc}{\left({\mathbf{e}}_j^{\text{col}}\right)}^{\top }& \text{0}\end{array}\right)}^{\top },\hfill & m=n+j,\hspace{0.17em}j\in \left[p\right]\hfill \end{array}$$

Note that the inner product between w_m and the eigenvector of ${\tilde{\mathbf{A}}}^{\left(m\right)}$ is $\langle {\left({\mathbf{e}}_i^{\text{row}}\right)}^{\top },\pm {\tilde{\mathbf{v}}}_k^{\left(i\right)}\rangle$ if m = i ∈ [n], or $\langle {\mathbf{e}}_j^{\text{col}},{\tilde{\mathbf{u}}}_k^{\left(j\right)}\rangle$ if m = n + j, j ∈ [p]. Therefore, applying Theorem 2.4 to the first n entries of

$$\frac{1}{\sqrt{2}}\left(\begin{array}{c}{\tilde{\mathbf{u}}}_{\ell }-{\mathbf{u}}_{\ell }\\ {\tilde{\mathbf{v}}}_{\ell }-{\mathbf{v}}_{\ell }\end{array}\right),$$

we obtain the first inequality of Corollary 2.2, and applying Theorem 2.4 to the last p entries leads to the second inequality.

Proof of Lemma 3.1.

$$\begin{gathered} {\mathbb{E}}_{\mathbf{\epsilon }}\left[{\Vert {\mathbf{X}}^{\top }{\widehat{\mathbf{\beta }}}_K-{\mathbf{X}}^{\top }{\mathbf{\beta }}^{\ast }\Vert }_2^2/n\right]={\mathbb{E}}_{\mathbf{\epsilon }}\left[{\Vert {\mathbf{Q}}_K{\mathbf{\Sigma }}_K{\mathbf{P}}_K^{\top }{\mathbf{\beta }}^{\ast }+{\mathbf{Q}}_K{\mathbf{Q}}_K^{\top }\mathbf{\epsilon }-{\mathbf{X}}^{\top }{\mathbf{\beta }}^{\ast }\Vert }_2^2/n\right] \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} ={\mathbb{E}}_{\mathbf{\epsilon }}\left[{\Vert {\mathbf{Q}}_K{\mathbf{Q}}_K^{\top }\mathbf{\epsilon }-{\mathbf{Q}}_{K+}{\mathbf{\Sigma }}_{K+}{\mathbf{P}}_{K+}^{\top }{\mathbf{\beta }}^{\ast }\Vert }_2^2/n\right] \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} =\frac{K{\sigma }^2}{n}+\underset{{\mathbf{\alpha }}^{\top }}{\underbrace{{\mathbf{\beta }}^{\ast \top }{\mathbf{P}}_{K+}}}{\mathbf{\Sigma }}_{K+}^2\underset{\mathbf{\alpha }}{\underbrace{{\mathbf{P}}_{K+}^{\top }{\mathbf{\beta }}^{\ast }}}. \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} =\frac{K{\sigma }^2}{n}+\sum _{j=K+1}^d{\lambda }_j^2{\alpha }_j^2. \end{gathered}$$