ASYMMETRY HELPS: EIGENVALUE AND EIGENVECTOR ANALYSES OF ASYMMETRICALLY PERTURBED LOW-RANK MATRICES

Abstract

This paper is concerned with the interplay between statistical asymmetry and spectral methods. Suppose we are interested in estimating a rank-1 and symmetric matrix $M^{\star }\in {ℝ}^{n\times n}$, yet only a randomly perturbed version M is observed. The noise matrix M − M^⋆ is composed of independent (but not necessarily homoscedastic) entries and is, therefore, not symmetric in general. This might arise if, for example, we have two independent samples for each entry of M^⋆ and arrange them in an asymmetric fashion. The aim is to estimate the leading eigenvalue and the leading eigenvector of M^⋆.

We demonstrate that the leading eigenvalue of the data matrix M can be $O(\sqrt{n})$ times more accurate (up to some log factor) than its (unadjusted) leading singular value of M in eigenvalue estimation. Moreover, the eigen-decomposition approach is fully adaptive to heteroscedasticity of noise, without the need of any prior knowledge about the noise distributions. In a nutshell, this curious phenomenon arises since the statistical asymmetry automatically mitigates the bias of the eigenvalue approach, thus eliminating the need of careful bias correction. Additionally, we develop appealing non-asymptotic eigenvector perturbation bounds; in particular, we are able to bound the perterbation of any linear function of the leading eigenvector of M (e.g. entrywise eigenvector perturbation). We also provide partial theory for the more general rank-r case. The takeaway message is this: arranging the data samples in an asymmetric manner and performing eigen-decomposition could sometimes be quite beneficial.

1. Introduction.

Consider an unknown symmetric and low-rank matrix $M^{\star }\in {ℝ}^{n\times n}$. What we have observed is a corrupted version

$$M=M^{\star }+H,$$ (1)

with H denoting a noise matrix. A classical problem is concerned with estimating the leading eigenvalues and eigenspace of M^⋆ given observation M.

The current paper concentrates on a scenario where the noise matrix H (and hence M) consists of independently generated random entries and is hence asymmetric in general. This might arise, for example, when we have available multiple (e.g. two) samples for each entry of M^⋆ and arrange the samples in an asymmetric fashion. A natural approach that immediately comes to mind is based on singular value decomposition (SVD), which employs the leading singular values (resp. subspace) of M to approximately estimate the eigenvalues (resp. eigenspace) of M^⋆. By contrast, a much less popular alternative is based on eigen-decomposition of the asymmetric data matrix M, which attempts approximation using the leading eigenvalues and eigenspace of M. Given that eigen-decomposition of an asymmetric matrix is in general not as numerically stable as SVD, conventional wisdom often favors the SVD-based approach, unless certain symmetrization step is implemented prior to eigen-decomposition.

When comparing these two approaches numerically, however, a curious phenomenon arises, which largely motivates the research in this paper. Let us generate M^⋆ as a random rank-1 matrix with leading eigenvalue λ^⋆ = 1, and let H be a Gaussian random matrix whose entries are i.i.d. $\mathcal{N}\left(0,{\sigma }^2\right)$ with $\sigma =1/\sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}$. Fig. 1(a) compares the empirical accuracy of estimating the 1st eigenvalue of M^⋆ via the leading eigenvalue (the blue line) and via the leading singular value of M (the red line). As it turns out, eigen-decomposition significantly outperforms vanilla SVD in estimating λ^⋆, and the advantage seems increasingly more remarkable as the dimensionality n grows. To facilitate comparison, we include an additional green line in Fig. 1(a), obtained by rescaling the red line by $2.5/\sqrt{n}$. Interestingly, this green line coincides almost perfectly with the blue line, thus suggesting orderwise gain of eigen-decomposition compared to SVD. What is more, this phenomenon does not merely happen under i.i.d. noise. Similar numerical behaviors are observed in the problem of matrix completion — as displayed in Fig. 1(b) — even though the components of the equivalent perturbation matrix are apparently far from identically distributed or homoscedastic.

Fig 1:

Numerical error |λ − λ^⋆| vs. the matrix dimension n, where λ is either the leading eigenvalue (the blue line) or the leading singular value (the red line) of M. Here, (a) is the case when {H_ij} are i.i.d. $\mathcal{N}\left(0,{\sigma }^2\right)$ with $\sigma =1/\sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}$, and (b) is the matrix completion case with sampling rate p = 3 log n/n, where $M_{i,j}=\frac{1}{p}{M}_{i,j}^{\star }$ independently with probability p and 0 otherwise. The results are averaged over 100 independent trials. The green lines are obtained by rescaling the corresponding red lines by $2.5/\sqrt{n}$.

The goal of the current paper is thus to develop a systematic understanding of this phenomenon, that is, why statistical asymmetry empowers eigen-decomposition and how to exploit this feature in statistical estimation. Informally, our findings suggest that: when M^⋆ is rank-1 and H is composed of zero-mean and independent (but not necessarily identically distributed or homoscedastic) entries,

1. the leading eigenvalue of M could be $O(\sqrt{n})$ times (up to some logarithmic factor) more accurate than the (unadjusted) leading singular value of M when estimating the 1st eigenvalue of M^⋆;¹

2. the perturbation of the leading eigenvector is well-controlled along an arbitrary deterministic direction; for example, the eigenvector perturbation is well-controlled in any coordinate, indicating that the eigenvector estimation error is spread out across all coordinates.

We will further provide partial theory to accommodate the rank-r case. As an important application, such a theory allows us to estimate the leading singular value and singular vectors of an asymmetric rank-1 matrix via eigen-decomposition of a certain dilation matrix, which also outperforms the vanilla SVD approach.

We would like to immediately remark that: for some scenarios (e.g. the case with i.i.d. Gaussian noise), it is possible to adjust the leading singular value of M to obtain the same accuracy as the leading eigenvalue of M. As it turns out, the advantages of the eigen-decomposition approach may become more evident in the presence of heteroscedasticity — the case where the noise has location-varying and unknown variance. We shall elaborate on this point in Section 4.1.2.

All in all, when it comes to low-rank matrix estimation, arranging the observed matrix samples in an asymmetric manner and invoking eigen-decomposition properly could sometimes be statistically beneficial.

2. Problem formulation.

2.1. Models and assumptions.

In this section, we formally introduce our models and assumptions. Consider a symmetric and low-rank matrix $M^{\star }={\left[M_{ij}^{\star }\right]}_{1\le i,j\le n}\in {ℝ}^{n\times n}$. Suppose we are given a random copy of M^⋆ as follows

$$M=M^{\star }+H,$$ (2)

where H = [H_ij]_1≤i,j≤n is a random noise matrix.

The current paper concentrates on independent — but not necessarily identically distributed or homoscedastic — noise. Specifically, we impose the following assumptions on H throughout this paper.

Assumption 1.

1. (Independent entries) The entries {H_ij}1_≤i,j≤n are independently generated;

2. (Zero mean) $\mathbb{E}\left[H_{ij}\right]=0$ for all 1 ≤ i, j ≤ n;

3. (Variance) $\text{Var}\left(H_{ij}\right)=\mathbb{E}\left[H_{ij}^2\right]\le {\sigma }_n^2$ for all 1 ≤ i, j ≤ n;

4. (Magnitude) Each H_ij (1 ≤ i, j ≤ n) satisfies either of the following conditions:

   1. |H_ij| ≤ B_n;
   2. H_ij has a symmetric distribution obeying $\mathbb{P}\left\{|H_{ij}|>B_n\right\}\le c_{\text{b}}{n}^{-12}$ for some universal constant c_b > 0.

Remark 1 (Notational convention).

In what follows, the dependency of σ_n and B_n on n shall often be suppressed whenever it is clear from the context, so as to simplify notation.

Note that we do not enforce the constraint H_ij = H_ji, and hence H and M are in general asymmetric matrices. Also, Condition 3 does not require the H_ij’s to have equal variance across different locations; in fact, they can be heteroscedastic. In addition, while Condition 4(a) covers the class of bounded random variables, Condition 4(b) allows us to accommodate a large family of heavy-tailed distributions (e.g. sub-exponential distributions). An immediate consequence of Assumption 1 is the following bound on the spectral norm ∥H∥ of H.

Lemma 1.

Under Assumption 1, there exist some universal constants c₀, C₀ > 0 such that with probability exceeding 1 − C₀n⁻¹⁰,

$$\Vert H\Vert \le c_0\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}+c_{0}B\text{\hspace{0.17em}log\hspace{0.17em}}n.$$ (3)

Proof. This is a standard non-asymptotic result that follows immediately from the matrix Bernstein inequality Tropp (2015) and the union bound (for Assumption 4(b)). We omit the details for conciseness. □

2.2. Our goal.

The aim is to develop non-asymptotic eigenvalue and eigenvector perturbation bounds under this family of random and asymmetric noise matrices. Our theoretical development is divided into two parts. Below we introduce our goal as well as some notation used throughout.

Rank-1 symmetric case.. For the rank-1 case, we assume the eigen-decomposition of M^⋆ to be

$$M^{\star }={\lambda }^{\star }{u}^{\star }{u}^{\star \top }$$ (4)

with λ^⋆ and u^⋆ being its leading eigenvalue and eigenvector, respectively. We also denote by λ and u the leading eigenvalue and eigenvector of M, respectively. The following quantities are the focal points of this paper (see Section 4):

1. Eigenvalue perturbation: |λ − λ^⋆|;

2. Perturbation of linear forms of eigenvectors: min{|a^⊤(u − u^⋆)|, |a^⊤(u + u^⋆)|} for any fixed unit vector $\text{min}\left\{|a^{\top }\left(u-u^{\star }\right)|,|{\mathit{a}}^{\top }\left(u+u^{\star }\right)|\right\}$;

3. Entrywise eigenvector perturbation: min{∥u − u^⋆∥_∞, ∥u + u^⋆∥_∞}.

Rank-r symmetric case.. For the general rank-r case, we let the eigen-decomposition of M^⋆ be

$$M^{\star }=U^{\star }{\Sigma }^{\star }{U}^{\star \top },$$ (5)

where the columns of $U^{\star }=\left[u_1^{\star },\cdots ,u_r^{\star }\right]\in {ℝ}^{n\times r}$ are the eigenvectors, and ${\mathbf{\Sigma }}^{\star }=\text{diag}\left({\lambda }_1^{\star },\cdots ,{\lambda }_r^{\star }\right)\in {ℝ}^{r\times r}$ is a diagonal matrix with the eigenvalues arranged in descending order by their magnitude, i.e. $\left|{\lambda }_1^{\star }\right|\ge \cdots \ge \left|{\lambda }_r^{\star }\right|$. We let ${\lambda }_{\text{max}}^{*}=\left|{\lambda }_1^{\star }\right|$ and ${\lambda }_{\text{min}}^{*}=\left|{\lambda }_r^{\star }\right|$. In addition, we let the top-r eigenvalues (in magnitude) of M be λ₁, ⋯, λ_r (obeying |λ₁| ≥ ⋯ ≥ |λ_r|) and their corresponding normalized eigenvectors be u₁, ⋯, u_r. We will present partial eigenvalue perturbation results for this more general case, as detailed in Section 5.

As is well-known, eigen-decomposition can be applied to estimate the singular values and singular vectors of an asymmetric matrix M^⋆ via the standard dilation trick Tropp (2015). As a consequence, our results are also applicable for singular value and singular vector estimation. See Section 5.2 for details.

2.3. Incoherence conditions.

Finally, we single out an incoherence parameter that plays an important role in our theory, which captures how well the energy of the eigenvectors is spread out across all entries.

Definition 1 (Incoherence parameter).

The incoherence parameter of a rank-r symmetric matrix M^⋆ with eigen-decomposition M^⋆ = U^⋆Σ^⋆U^⋆⊤ is defined to be the smallest quantity μ obeying

$${\Vert U^{\star }\Vert }_{\infty }\le \sqrt{\frac{\mu }{n}},$$ (6)

where ∥·∥_∞ denotes the entrywise ℓ_∞ norm.

Remark 2.

An alternative definition of the incoherence parameter (Candès and Recht, 2009; Keshavan, Montanari and Oh, 2010; Chi, Lu and Chen, 2019; Chen et al., 2019a) is the smallest quantity μ₀ satisfying $\sqrt{{\mu }_{0}r/n}$. This is a weaker assumption than Definition 1, as it only requires the energy of U^⋆ to be spread out across all of its rows rather than all of its entries. Note that these two incoherent parameters are consistent in the rank-1 case; in the rank-r case one has μ₀ ≤ μ ≤ μ₀r.

2.4. Notation.

The standard basis vectors in ${ℝ}^n$ are denoted by e₁, ⋯, e_n. For any vector z, we let ∥z∥₂ and ∥z∥_∞ denote the ℓ₂ norm and the ℓ_∞ norm of z, respectively. For any matrix M, denote by ∥M∥, ∥M∥_F and ∥M∥_∞ the spectral norm, the Frobenius norm and the entrywise ℓ_∞ norm (the largest magnitude of all entries) of M, respectively. Let [n] := {1, ⋯, n}. In addition, the notation f(n) = O (g(n)) or f(n) ≲ g(n) means that there is a constant c > 0 such that |f(n)| ≤ c|g(n)|, f(n) ≳ g(n) means that there is a constant c > 0 such that |f(n)| ≥ c|g(n)|, and f(n) ≍ g(n) means that there exist constants c₁, c₂ > 0 such that c₁|g(n)| ≤ |f(n)| ≤ c₂|g(n)|.

3. Preliminaries.

Before continuing, we gather several preliminary facts that will be useful throughout. The readers familiar with matrix perturbation theory may proceed directly to the main theoretical development in Section 4.

3.1. Perturbation of eigenvalues of asymmetric matrices.

We begin with a standard result concerning eigenvalue perturbation of a diagonalizable matrix Bauer and Fike (1960). Note that the matrices under study might be asymmetric.

Theorem 1 (Bauer-Fike Theorem).

Consider a diagonalizable matrix $A\in {ℝ}^{n\times n}$ with eigen-decomposition A = VΛV⁻¹, where $V\in {ℂ}^{n\times n}$ is a non-singular eigenvector matrix and Λ is diagonal. Let $\tilde{\lambda }$ be an eigenvalue of A + H. Then there exists an eigenvalue λ of A such that

$$\left|\lambda -\tilde{\lambda }\right|\le \Vert V\Vert ‖V^{-1}‖\Vert H\Vert .$$ (7)

In addition, if A is symmetric, then there exists an eigenvalue λ of A such that

$$|\lambda -\tilde{\lambda }|\le \Vert H\Vert .$$ (8)

However, caution needs to be exercised as the Bauer-Fike Theorem does not specify which eigenvalue of A is close to an eigenvalue of A + H. Encouragingly, in the low-rank case of interest, the Bauer-Fike Theorem together with certain continuity of the spectrum allows one to localize the leading eigenvalues of the perturbed matrix.

Lemma 2.

Suppose M^⋆ is a rank-r symmetric matrix whose top-r eigenvalues obey $|{\lambda }_1^{\star }|\ge \cdots \ge |{\lambda }_r^{\star }|>0$. If $\Vert H\Vert <|{\lambda }_r^{\star }|/2$, then the top-r eigenvalues λ₁, ⋯, λ_r of M = M^⋆ + H, sorted by modulus, obey that: for any 1 ≤ l ≤ r,

$$|{\lambda }_l-{\lambda }_j^{\star }|\le \Vert H\Vert \mathit{\text{ for some }}1\le j\le r.$$ (9)

In addition, if r = 1, then both the leading eigenvalue and the leading eigenvector of M are real-valued.

This result, which we establish in Appendix 1.1, parallels Weyl’s inequality for symmetric matrices. Note, however, that the above bound (9) might be quite loose for specific settings. We will establish much sharper perturbation bounds when H contains independent random entries (see, e.g. Corollary 1).

3.2. The Neumann trick and eigenvector perturbation.

Next, we introduce a classical result dubbed as the “Neumann trick” Eldridge, Belkin and Wang (2018). This theorem, which is derived based on the Neumann series for a matrix inverse, has been applied to analyze eigenvectors in various settings Erdős et al. (2013); Jain and Netrapalli (2015); Eldridge, Belkin and Wang (2018).

Theorem 2 (Neumann trick).

Consider the matrices M^⋆ and M (see (5) and (2)). Suppose ∥H∥ < |λ_l| for some 1 ≤ l ≤ n. Then

$$u_l=\sum _{j=1}^r\frac{{\lambda }_j^{\star }}{{\lambda }_l}\left(u_j^{\star \top }{u}_l\right)\left\{\sum _{s=0}^{\infty }\frac{1}{{\lambda }_l^s}{H}^s{u}_j^{\star }\right\}.$$ (10)

Proof. We supply the proof in Appendix 1.2 for self-containedness. □

Remark 3.

In particular, if M^⋆ is a rank-1 matrix and ∥H∥ < |λ₁|, then

$$u_1=\frac{{\lambda }_1^{\star }}{{\lambda }_1}\left(u_1^{\star \top }{u}_1\right)\left\{\sum _{s=0}^{\infty }\frac{1}{{\lambda }_1^s}{H}^s{u}_1^{\star }\right\}.$$ (11)

An immediate consequence of the Neumann trick is the following lemma, which asserts that each of the top-r eigenvectors of M resides almost within the top-r eigen-subspace of M^⋆, provided that ∥H∥ is sufficiently small. The proof is deferred to Appendix 1.3.

Lemma 3.

Suppose M^⋆ is a rank-r symmetric matrix with r non-zero eigenvalues obeying $1={\lambda }_{\text{max}}^{*}=\left|{\lambda }_1^{\star }\right|\ge \cdots \ge \left|{\lambda }_r^{\star }\right|={\lambda }_{\text{min}}^{*}>0$ and associated eigenvectors $\kappa \triangleq {\lambda }_{\text{max}}^{*}/{\lambda }_{\text{min}}^{*}$. If ∥H∥ ≤ 1/(4κ), then the top-r eigenvectors u₁, ⋯, u_r of M = M^⋆ + H obey

$$\sum _{j=1}^r{|u_j^{\star \top }{u}_l|}^2\ge 1-\frac{64{\kappa }^4}{9}\Vert H{\Vert }^2,1\le l\le r.$$ (12)

In addition, if r = 1, then one further has

$$\text{min}\left\{{\Vert u_1-u_1^{\star }\Vert }_2,{\Vert u_1+u_1^{\star }\Vert }_2\right\}\le \frac{8\sqrt{2}}{3}\Vert H\Vert .$$ (13)

4. Perturbation analysis for the rank-1 case.

4.1. Main results: the rank-1 case.

This section presents perturbation analysis results when the truth M^⋆ is a symmetric rank-1 matrix. We shall start by presenting a master bound which, as we will see, immediately leads to our main findings.

4.1.1. A master bound.

Our master bound is concerned with the perturbation of linear forms of eigenvectors, as stated below.

Theorem 3 (Perturbation of linear forms of eigenvectors (rank-1)).

Consider a rank-1 symmetric matrix $M^{\star }={\lambda }^{\star }{u}^{\star }{u}^{\star \top }\in {ℝ}^{n\times n}$ with incoherence parameter μ (cf. Definition 1). Suppose the noise matrix H obeys Assumption 1, and assume the existence of some sufficiently small constant c₁ > 0 such that

$$\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\le c_1|{\lambda }^{\star }|.$$ (14)

Then for any fixed vector $a\in {ℝ}^n$ with ∥a∥₂ = 1, probability at least 1 − O(n⁻¹⁰) one has

$$|a^{\top }\left(u-\frac{u^{\star \top }u}{\lambda /{\lambda }^{\star }}{u}^{\star }\right)|\lesssim \frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{|{\lambda }^{\star }|}\sqrt{\frac{\mu }{n}}.$$ (15)

Remark 4 (The noise size).

We would like to remark on the range of the noise size covered by our theory. If the incoherence parameter of the truth M^⋆ obeys μ ≍ 1, then even the magnitude of the largest entry of M^⋆ cannot exceed the order of |λ^⋆|/n. One can thus interpret the condition (14) in this case as

$$\sigma \lesssim \sqrt{\frac{n}{\text{log\hspace{0.17em}}n}}{\Vert {\mathit{M}}^{\star }\Vert }_{\infty }\text{\hspace{1em} and \hspace{1em}}B\lesssim \frac{n}{\text{log\hspace{0.17em}}n}{\Vert M^{\star }\Vert }_{\infty }.$$

In other words, the standard deviation σ of each noise component is allowed to be substantially larger (i.e. $\sqrt{n/\text{log\hspace{0.17em}}n}$ times larger) than the magnitude of any of the true entries. In fact, this condition (14) matches, up to some log factor, the one required for spectral methods to perform noticeably better than random guessing.

In words, Theorem 3 tells us that: the quantity $\frac{u^{\star \top }u}{\lambda /{\lambda }^{\star }}{a}^{\top }{u}^{\star }$ serves as a remarkably accurate approximation of the linear form a^⊤u. In particular, the approximation error is at most $O(1/\sqrt{n})$ under the condition (14) for incoherent matrices. Encouragingly, this approximation accuracy holds true for an arbitrary deterministic direction (reflected by a). As a consequence, one can roughly interpret Theorem 3 as

$$u\approx \frac{{\lambda }^{\star }}{\lambda }{u}^{\star }{u}^{\star \top }u=\frac{1}{\lambda }{M}^{\star }u,$$ (16)

where such an approximation is fairly accurate along any fixed direction. Compared with the identity $u=\frac{1}{\lambda }Mu=\frac{1}{\lambda }\left(M^{\star }+H\right)u$, our results imply that Hu is exceedingly small along any fixed direction, even though H and u are highly dependent. As we shall explain in Section 4.3, this observation usually cannot happen when H is a symmetric random matrix or when one uses the leading singular vector instead, due to the significant bias resulting from symmetry.

This master theorem has several interesting implications, as we shall elucidate momentarily.

4.1.2. Eigenvalue perturbation.

To begin with, Theorem 3 immediately yields a much sharper non-asymptotic perturbation bound regarding the leading eigenvalue λ of M.

Corollary 1.

Under the assumptions of Theorem 3, with probability at least 1 − O(n⁻¹⁰) we have

$$|\lambda -{\lambda }^{\star }|\lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\sqrt{\frac{\mu }{n}}.$$ (17)

Proof. Without loss of generality, assume that λ^⋆ = 1. Taking a = u^⋆ in Theorem 3, we get

$$|u^{\star \top }u|\frac{|\lambda -1|}{\lambda }=|u^{\star \top }u-u^{\star \top }{u}^{\star }\frac{u^{\star \top }u}{\lambda }|\lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\sqrt{\frac{\mu }{n}}.$$ (18)

From Lemma 1 and the condition (14), we know ∥H∥ < 1/4, which combines with Lemma 2 and Lemma 3 yields λ ≍ |u^⋆⊤u| ≍ 1. Substitution into (18) yields

$$\begin{array}{ll}\left|\lambda -1\right|\hfill & \lesssim \left|\frac{\lambda }{u^{\star \top }u}\right|\text{max}\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\}\sqrt{\frac{\mu }{n}}\hfill \\ \hfill & \lesssim \text{max}\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\}\sqrt{\frac{\mu }{n}}.\hfill \end{array}$$

For the vast majority of applications we encounter, the maximum possible noise magnitude B (cf. Assumption 1) obeys $B\lesssim \sigma \sqrt{n/\text{log\hspace{0.17em}}n}$, in which case the bound in Corollary 1 simplifies to

$$|\lambda -{\lambda }^{\star }|\lesssim \sigma \sqrt{\mu \text{\hspace{0.17em}log\hspace{0.17em}}n}.$$ (19)

This means that the eigenvalue estimation error is not much larger than the variability of each noise component. In addition, we remind the reader that for a fairly broad class of noise (see Remark 4), the leading eigenvalue λ of M is guaranteed to be real-valued, an observation that has been made in Lemma 2. In practice, however, one might still encounter some scenarios where λ is complex-valued. As a result, we recommend the practitioner to use the real part of λ as the eigenvalue estimate, which clearly enjoys the same statistical guarantee as in Corollary 1.

Comparison to the vanilla SVD-based approach.. In order to facilitate comparison, we denote by λ_svd the largest singular value of M, and look at |λ_svd − λ^⋆|. Combining Weyl’s inequality, Lemma 1 and the condition (14), we arrive at

$$|{\lambda }_{\text{svd }}-{\lambda }^{\star }|\le \Vert H\Vert \lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}.$$ (20)

When μ ≍ 1, this error bound w.r.t. this (unadjusted) singular value could be $\sqrt{n}$ times larger than the perturbation bound (17) derived for the leading eigenvalue. This corroborates our motivating experiments in Fig. 1.

Comparison to vanilla eigen-decomposition after symmetrization.. The reader might naturally wonder what would happen if we symmetrize the data matrix before performing eigen-decomposition. Consider, for example, the i.i.d. Gaussian noise case where $H_{ij}\stackrel{\text{ i.i}.\text{d.}}{~}\mathcal{N}\left(0,{\sigma }^2\right)$, and assume λ^⋆ > 0 for simplicity. The leading eigenvalue λ_sym of the symmetrized matrix (M + M^⊤)/2 has been extensively studied in the literature Füredi and Komlós (1981); Yin, Bai and Krishnaiah (1988); Péché (2006); Féral and Péché (2007); Benaych-Georges and Nadakuditi (2011); Renfrew and Soshnikov (2013); Knowles and Yin (2013). In particular, it has been shown (e.g. Capitaine, Donati-Martin and Féral (2009)) that, with probability approaching one,

$${\lambda }_{\text{sym}}={\lambda }^{\star }+\frac{n{\sigma }^2}{2{\lambda }^{\star }}+O(\sigma \sqrt{\text{log\hspace{0.17em}}n}).$$ (21)

If $\sigma =|{\lambda }^{\star }|\sqrt{1/(n\text{\hspace{0.17em}log\hspace{0.17em}}n)}$ (which is the setting in our numerical experiment), then this can be translated into

$$\frac{{\lambda }_{\text{sym }}-{\lambda }^{\star }}{{\lambda }^{\star }}=\frac{1}{2\text{\hspace{0.17em}log\hspace{0.17em}}n}+O\left(\frac{1}{\sqrt{n}}\right).$$

This implies that λ_sym suffers from a substantially larger bias than the leading eigenvalue λ obtained without symmetrization, since in this case we have (cf. Corollary 1)

$$|\frac{\lambda -{\lambda }^{\star }}{{\lambda }^{\star }}|\lesssim \frac{1}{\sqrt{n}}.$$ (22)

Comparison to properly adjusted eigen-decomposition and SVD-based methods.. Armed with the approximation (21) in the i.i.d. Gaussian noise case, the careful reader might naturally suggest a properly corrected estimate λ_sym,c as follows (again assuming λ > 0)

$${\lambda }_{\text{sym},\text{c}}=\frac{1}{2}\left({\lambda }_{\text{sym}}+\sqrt{{\lambda }_{\text{sym}}^2-2n{\sigma }^2}\right),$$ (23)

which is a shrinkage-type estimate chosen to satisfy ${\lambda }_{\text{sym}}={\lambda }_{\text{sym},\text{c}}+\frac{n{\sigma }^2}{2{\lambda }_{\text{sym},\text{c}}}$. A little algebra reveals that: if $\sigma =1/\sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}$, then

$$|\frac{{\lambda }_{\text{sym},\text{c}}-{\lambda }^{\star }}{{\lambda }^{\star }}|\lesssim \frac{1}{\sqrt{n}},$$

thus matching the estimation accuracy of λ (cf. (22)). In addition, some sort of universality results has been established as well in the literature (Capitaine, Donati-Martin and Féral, 2009), implying that the same approximation and correction are applicable to a broad family of zero-mean noise with identical variance. As we shall illustrate numerically in Section 4.2, this approach (i.e. λ_sym,c) performs almost identically to the one using vanilla eigen-decomposition without symmetrization. In addition, very similar observations have been made for the SVD-based approach (Silverstein, 1994; Yin, Bai and Krishnaiah, 1988; Péché, 2006; Féral and Péché, 2007; Benaych-Georges and Nadakuditi, 2012; Bryc and Silverstein, 2018); for the sake of brevity, we do not repeat the arguments here.

We would nevertheless like to single out a few statistical advantages of the eigen-decomposition approach without symmetrization. To begin with, λ is obtained via vanilla eigen-decomposition, and computing it does not rely on any kind of noise statistics. This is in stark contrast to the bias correction (23) in the presence of symmetric data, which requires prior knowledge about (or a very precise estimate of) the noise variance σ². Leaving out this prior knowledge matter, a more important issue is that the approximation formula (21) assumes identical variance of noise components across all entries (i.e. homoscedasticity). While an approximation of this kind has been found for more general cases beyond homoscedastic noise (e.g. Bryc and Silverstein (2018)), the approximation formula (e.g. (Bryc and Silverstein, 2018, Theorem 1.1)) becomes fairly complicated, requires prior knowledge about all variance parameters, and is thus difficult to implement in practice. In comparison, the vanilla eigen-decomposition approach analyzed in Corollary 1 imposes no restriction on the noise statistics and is fully adaptive to heteroscedastic noise.

Lower bounds.. To complete the picture, we provide a simple information-theoretic lower bound for the i.i.d. Gaussian noise case, which will be established in Appendix 2.

Lemma 4.

Fix any small constant ε > 0. Suppose that $H_{ij}\stackrel{i.i.d.}{~}\mathcal{N}\left(0,{\sigma }^2\right)$. Consider three matrices

$$M={\lambda }^{\star }{u}^{\star }{u}^{\star \top }+H,\tilde{M}=\left({\lambda }^{\star }+\Delta \right)u^{\star }{u}^{\star \top }+H,\widehat{M}=\left({\lambda }^{\star }-\Delta \right)u^{\star }{u}^{\star \top }+H$$

with ${\Vert u^{\star }\Vert }_2=1.$ if $\Delta \le \sigma \sqrt{\left({\text{log}}_{2}1.5-\epsilon \right)\text{\hspace{0.17em}log\hspace{0.17em}}2}$, then no algorithm can distinguish M, $\tilde{M}$ and $\hat{M}$ with p_e ≤ ε, where p_e is the minimax probability of error for testing three hypotheses (namely, the ones claiming that the true eigenvalues are λ^⋆, λ^⋆ + Δ, and λ^⋆ − Δ, respectively).

In short, Lemma 4 asserts that one cannot possibly locate an eigenvalue to within a precision of Δ much better than σ, which reveals a fundamental limit that cannot be broken by any algorithm. In comparison, the vanilla eigen-decomposition method based on asymmetric data achieves an accuracy of $|\lambda -{\lambda }^{\star }|\lesssim \sigma \sqrt{\text{log }n}$ (cf. Corollary 1 and (19)) for the incoherent case, thus matching the information-theoretic lower bound up to some log factor. In fact, the extra $\sqrt{\text{log }n}$ factor arises simply because we are aiming for a high-probability guarantee.

4.1.3. Perturbation of linear forms of eigenvectors.

The master bound in Theorem 3 admits a more convenient form when controlling linear functions of the eigenvectors. The result is this:

Corollary 2.

Under the same setting of Theorem 3, with probability at least 1 − O(n⁻¹⁰) we have

$$\text{min}\left\{|a^{\top }\left(u-u^{\star }\right)|,|a^{\top }\left(u+u^{\star }\right)|\right\}\lesssim \left(|a^{\top }{u}^{\star }|+\sqrt{\frac{\mu }{n}}\right)\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{|{\lambda }^{\star }|}.$$ (24)

Proof. Without loss of generality, assume that u^⋆⊤u ≥ 0 and that λ^⋆ = 1. Then one has

$$\begin{array}{ll}\left|a^{\top }\left(u-u^{\star }\right)\right|\hfill & \le \left|a^{\top }u-a^{\top }{u}^{\star }\frac{u^{\star \top }u}{\lambda }\right|+\left|a^{\top }{u}^{\star }\right|\left|\frac{u^{\star \top }u}{\lambda }-1\right|\hfill \\ \hfill & \le \text{max}\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\}\sqrt{\frac{\mu }{n}}+\left|a^{\top }{u}^{\star }\right|\left|\frac{u^{\star \top }u}{\lambda }-1\right|,\hfill \end{array}$$

where the last inequality arises from Theorem 3 as well as the definition of μ. In addition, apply Lemma 2 and Lemma 3 to obtain

$$|\frac{u^{\star \top }u}{\lambda }-1|\le \frac{u^{\star \top }u}{\lambda }|1-\lambda |+|u^{\star \top }u-1|\lesssim \Vert H\Vert \lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}.$$

Putting the above bounds together concludes the proof. □

The perturbation of linear forms of eigenvectors (or singular vectors) has not yet been well explored even for the symmetric case. One scenario that has been studied is linear forms of singular vectors under i.i.d. Gaussian noise (Koltchinskii and Xia, 2016; Xia, 2016). Our analysis — which is certainly different from Koltchinskii and Xia (2016) as our emphasis is eigen-decomposition — does not rely on the Gaussianality assumption, and accommodates a much broader class of random noise. Another work that has looked at linear forms of the leading singular vector is Ma et al. (2019) for phase retrieval and blind deconvolution, although the vector a therein is specific to the problems (i.e. the design vectors) and cannot be made general.

Remark 5.

The perturbation theory for linear forms of eigenvectors has been substantially extended in our follow-up work; the interested reader is referred to Cheng, Wei and Chen (2020) for details.

4.1.4. Entrywise eigenvector perturbation.

A straightforward consequence of Corollary 2 that is worth emphasizing is sharp entrywise control of the leading eigenvector as follows.

Corollary 3.

Under the same setting of Theorem 3, with probability at least 1 − O(n⁻⁹) we have

$$\text{min}\left\{{\Vert u-u^{\star }\Vert }_{\infty },{\Vert u+u^{\star }\Vert }_{\infty }\right\}\lesssim \frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{|{\lambda }^{\star }|}\sqrt{\frac{\mu }{n}}.$$ (25)

Proof. Recognizing that ${\Vert u-u^{\star }\Vert }_{\infty }={\text{max}}_i|e_i^{\top }u-e_i^{\top }{u}^{\star }|$ and recalling our assumption $|e_i^{\top }u|\le \sqrt{\mu /n}$, we can invoke Corollary 2 and the union bound to establish this entrywise bound. □

We note that: while the ℓ₂ perturbation (or sin Θ distance) of eigenvectors or singular vectors has been extensively studied Davis and Kahan (1970); Wedin (1972); Vu (2011); Wang (2015); O’Rourke, Vu and Wang (2013); Cai and Zhang (2018), the entrywise eigenvector behavior was much less explored. The prior literature contains only a few entrywise eigenvector perturbation analysis results for settings very different from ours, e.g. the i.i.d. random matrix case Vu and Wang (2015); O’Rourke, Vu and Wang (2016), the symmetric low-rank case Fan, Wang and Zhong (2018); Abbe et al. (2017); Eldridge, Belkin and Wang (2018), and the case with transition matrices for reversible Markov chains (Chen et al., 2019b). Our results add another instance to this body of works in providing entrywise eigenvector perburation bounds.

4.2. Applications.

We apply our main results to two concrete matrix estimation problems and examine the effectiveness of these bounds. As before, M^⋆ is a rank-1 matrix with incoherence parameter μ and leading eigenvalue λ^⋆.

Low-rank matrix estimation from Gaussian noise.

Suppose that H is composed of i.i.d. Gaussian random variables $\mathcal{N}\left(0,{\sigma }^2\right)$.² If $\sigma \lesssim \frac{1}{\sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}}$, applying Corollaries 1–3 reveals that with high probability,

$$|\lambda -{\lambda }^{\star }|\lesssim \sigma \sqrt{\mu \text{\hspace{0.17em}log\hspace{0.17em}}n}$$ (26a)

$$\text{min}\left\{{\Vert u-u^{\star }\Vert }_{\infty },{\Vert u+u^{\star }\Vert }_{\infty }\right\}\lesssim \frac{\sigma \sqrt{\mu \text{\hspace{0.17em}log\hspace{0.17em}}n}}{|{\lambda }^{\star }|}$$ (26b)

$$\text{min}\left\{|a^{\top }\left(u-u^{\star }\right)|,|a^{\top }\left(u+u^{\star }\right)|\right\}\lesssim \left(|a^{\top }{u}^{\star }|+\sqrt{\frac{\mu }{n}}\right)\frac{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}}{|{\lambda }^{\star }|}$$ (26c)

for any fixed unit vector $a\in {ℝ}^n$. We have conducted additional numerical experiments in Fig. 2, which confirm our findings. It is also worth noting that empirically, eigen-decomposition and SVD applied to M achieve nearly identical ℓ₂ and ℓ_∞ errors when estimating the leading eigenvector of M^⋆. In addition, we also include the numerical estimation error of the corrected eigenvalue λ_sym,c (cf. (23)) of the symmetrized matrix (M + M^⊤)/2. As can be seen from Fig. 2, vanilla eigen-decomposition without symmetrization performs nearly identically to the one with symmetrization and proper correction.

Fig 2:

Numerical simulation for rank-1 matrix estimation under i.i.d. Gaussian noise $\mathcal{N}\left(0,{\sigma }^2\right)$, where the rank-1 truth M^⋆ is generated randomly with leading eigenvalue 1. (a): |λ − λ^⋆| vs. σ when n = 1000; (b) and (c): ℓ_∞ and ℓ₂ eigenvector estimation errors vs. n with $\sigma =1/\sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n}$, respectively. The blue (resp. red) lines represent the average errors over 100 independent trials using the vanilla eigen-decomposition (resp. SVD) approach applied to M. The orange line in (a) represents the average errors over 100 independent trials using the corrected leading eigenvalue λ_sym,c of the symmetrized matrix (M + M^⊤)/2 (cf. (23)).

Low-rank matrix completion.

Suppose that M is generated using random partial entries of M^⋆ as follows

$$M_{ij}=\{\begin{array}{ll}\frac{1}{p}{M}_{ij}^{\star },\hfill & \text{ with probability }p,\hfill \\ 0,\hfill & \text{ else, }\hfill \end{array}$$ (27)

where p denotes the fraction of the entries of M^⋆ being revealed. It is straightforward to verify that H = M − M^⋆ is zero-mean and obeys $|H_{ij}|\le \frac{\mu }{np}≔B$ and $\text{Var}\left(H_{ij}\right)\le \frac{{\mu }^2}{p{n}^2}$. Consequently, if $p\gtrsim \frac{{\mu }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}{n}$, then invoking Corollaries 1–3 yields

$$\frac{|\lambda -{\lambda }^{\star }|}{|{\lambda }^{\star }|}\lesssim \frac{1}{\sqrt{n}}\sqrt{\frac{{\mu }^3\text{\hspace{0.17em}log\hspace{0.17em}}n}{pn}}$$ (28a)

$$\text{min}\left\{{\Vert u-u^{\star }\Vert }_{\infty },{\Vert u+u^{\star }\Vert }_{\infty }\right\}\lesssim \frac{1}{\sqrt{n}}\sqrt{\frac{{\mu }^3\text{\hspace{0.17em}log\hspace{0.17em}}n}{pn}}$$ (28b)

$$\text{min}\left\{|a^{\top }\left(u-u^{\star }\right)|,|a^{\top }\left(u+u^{\star }\right)|\right\}\lesssim \left(|a^{\top }{u}^{\star }|+\sqrt{\frac{\mu }{n}}\right)\sqrt{\frac{{\mu }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}{pn}}$$ (28c)

with high probability, where $a\in {ℝ}^n$ is any fixed unit vector. Additional numerical simulations have been carried out in Fig. 3 to verify these findings. Empirically, eigen-decomposition outperforms SVD in estimating both the leading eigenvalue and eigenvector of M^⋆.

Fig 3:

Numerical simulation for rank-1 matrix completion, where the rank-1 truth M^⋆ is randomly generated with leading eigenvalue 1 and sampling rate is p = 3 log n/n. (a) |λ − λ^⋆| vs. p when n = 1000; (b) and (c): ℓ_∞ and ℓ₂ eigenvector estimation errors vs. n, respectively. The blue (resp. red) lines represent the average errors over 100 independent trials using the eigen-decomposition (resp. SVD) approach.

Finally, we remark that all the above applications assume the availability of an asymmetric data matrix M. One might naturally wonder whether there is anything useful we can say if only a symmetric matrix M is available. While this is in general difficult, our theory does have direct implications for both matrix completion and the case with i.i.d. Gaussian noise in the presence of symmetric data matrices; that is, it is possible to first asymmetrize the data matrix followed by eigen-decomposition. The interested reader is referred to Appendix 10 for details.

4.3. Why asymmetry helps?.

We take a moment to develop some intuition underlying Theorem 3, focusing on the case with λ^⋆ = 1 for simplicity. The key ingredient is the Neumann trick stated in Theorem 2. Specifically, in the rank-1 case we can expand

$$u=\frac{1}{\lambda }\left(u^{\star \top }u\right){\sum }_{s=0}^{\infty }\frac{1}{{\lambda }^s}{H}^s{u}^{\star }.$$

A little algebra yields

$$|a^{\top }\left(u-\frac{u^{\star \top }u}{\lambda }{u}^{\star }\right)|=|\frac{u^{\star \top }u}{\lambda }\sum _{s=1}^{\infty }\frac{a^{\top }{H}^s{u}^{\star }}{{\lambda }^s}|\lesssim \sum _{s=1}^{\infty }|\frac{a^{\top }{H}^s{u}^{\star }}{{\lambda }^s}|,$$ (29)

where the last inequality holds since (i) |u^⋆⊤u| ≤ 1, and (ii) λ is real-valued and obeys λ ≈ 1 if ∥H∥ ≪ 1 (in view of Lemma 2). As a result, the perturbation can be well-controlled as long as |a^⊤H^su^⋆| is small for every s ≥ 1.

As it turns out, a^⊤H^su^⋆ might be much better controlled when H is random and asymmetric, in comparison to the case where H is random and symmetric. To illustrate this point, it is perhaps the easiest to inspect the second-order term.

• Asymmetric case: when H is composed of independent zero-mean entries each with variance ${\sigma }_n^2$, one has

  $$\mathbb{E}\left[a^{\top }{H}^2{u}^{\star }\right]=a^{\top }\mathbb{E}\left[H^2\right]u^{\star }=a^{\top }\left({\sigma }^{2}I\right)u^{\star }={\sigma }^2{a}^{\top }{u}^{\star }.$$
• Symmetric case: when H is symmetric and its upper trangular part consists of independent zero-mean entries with variance ${\sigma }_n^2$, it holds that

  $$\mathbb{E}\left[a^{\top }{H}^2{u}^{\star }\right]=a^{\top }\mathbb{E}\left[H^2\right]u^{\star }=a^{\top }\left(n{\sigma }^{2}I\right)u^{\star }=n{\sigma }^2{a}^{\top }{u}^{\star }.$$

In words, the term a^⊤H²u^⋆ in the symmetric case might have a significantly larger bias compared to the asymmetric case. This bias effect is substantial when a^⊤u^⋆ is large (e.g. when a = u^⋆), which plays a crucial role in determining the size of eigenvalue perturbation.

The vanilla SVD-based approach can be interpreted in a similar manner. Specifically, we recognize that the leading singular value (resp. left singular vector) can be computed via the leading eigenvalue (resp. eigenvector) of the symmetric matrix MM^⊤. Given that MM^⊤ − M^⋆M^⋆⊤ is also symmetric, the aforementioned bias issue arises as well. This explains why vanilla eigen-decomposition might have an advantage over vanilla SVD when dealing with asymmetric matrices.

Finally, we remark that the aforementioned bias issue becomes less severe as ∥H∥ decreases. For example, when ∥H∥ is exceedingly small, the only dominant term on the right-hand side of (29) is a^⊤Hu^⋆, with all higher-order terms being vanishingly small. In this case, $\mathbb{E}\left[a^{\top }H{u}^{\star }\right]=0$ for both symmetric and asymmetric zero-mean noise matrices. As a consequence, the advantage of eigen-decomposition becomes negligible when dealing with nearly-zero noise. This observation is also confirmed in the numerical experiments reported in Fig. 2(a) and Fig. 3(a), where the two approaches achieve similar eigenvalue estimation accuracy when σ → 0 (resp. p → 1) in matrix estimation under Gaussian noise (resp. matrix completion). In fact, the case with very small ∥H∥ has been studied in the literature O’Rourke, Vu and Wang (2013); Vu (2011); Eldridge, Belkin and Wang (2018). For example, it was shown in O’Rourke, Vu and Wang (2013) that when $\Vert H\Vert \lesssim \frac{1}{\sqrt{n}}{M}^{\star }$, the singular value perturbation is also $\sqrt{n}$ times smaller than the bound predicted by Weyl’s theorem; similar improvement can be observed w.r.t. eigenvalue perturbation when H is symmetric (cf. (Eldridge, Belkin and Wang, 2018, Theorem 6)). By contrast, our eigenvalue perturbation results achieve this gain even when ∥H∥ is nearly as large as ∥M^⋆∥ (up to some logarithmic factor).

4.4. Proof outline of Theorem 3.

This subsection outlines the main steps for establishing Theorem 3. To simplify presentation, we shall assume without loss of generality that

$${\lambda }^{\star }=1.$$ (30)

Throughout this paper, all the proofs are provided for the case when Conditions 1–3, 4(a) in Assumption 1 are valid. Otherwise, if Condition 4(b) is valid, then we can invoke the union bound to show that

$$M=M^{\star }+\tilde{H}$$ (31)

with probability exceeding 1 − O(n⁻¹⁰), where ${\tilde{H}}_{ij}\triangleq H_{ij}{\mathbb{1}}_{\left\{|H_{ij}|\le B\right\}}$ is the truncated noise and has magnitude bounded by B. Since H_ij has symmetric distribution, it is seen that $\mathbb{E}\left[{\tilde{H}}_{ij}\right]=0$ and $\text{Var}\left({\tilde{H}}_{ij}\right)\le {\sigma }^2$, which coincides with the case obeying Conditions 1–3, 4(a) in Assumption 1.

As already mentioned in Section 4.3, everything boils down to controlling |a^⊤H^su^⋆| for s ≥ 1. This is accomplished via the following lemma.

Lemma 5 (Bounding higher-order terms).

Consider any fixed unit vector $a\in {ℝ}^n$ and any positive integers s, k satisfying Bsk ≤ 2 and nσ²sk ≤ 2. Under the assumptions of Theorem 3,

$$|\mathbb{E}\left[{\left(a^{\top }{H}^s{u}^{\star }\right)}^k\right]|\le \frac{sk}{2}\text{max}\left\{{(Bsk)}^{sk},{\left(2n{\sigma }^{2}sk\right)}^{sk/2}\right\}{\left(\sqrt{\frac{\mu }{n}}\right)}^k.$$ (32)

Proof. The proof of Lemma 5 is combinatorial in nature, which we defer to Appendix 3. □

Remark 6.

A similar result in (Tao, 2013, Lemma 2.3) has studied the bilinear forms of the high order terms of an i.i.d. random matrix, with a few distinctions. First of all, Tao (2013) assumes that each entry of the noise matrix is i.i.d. and has finite fourth moment (if the noise variance is rescaled to be 1); these assumptions break in examples like matrix completion. Moreover, Tao (2013) focuses on the case with k = 2, and does not lead to high-probability bounds (which are crucial for, e.g. entrywise error control).

Using Markov’s inequality and the union bound, we can translate Lemma 5 into a high probability bound as follows.

Corollary 4.

Under the assumptions of Lemma 5, there exists some universal constant c₂ > 0 such that

$$|a^{\top }{H}^s{u}^{\star }|\le {\left(c_2\text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\right)}^s\sqrt{\frac{\mu }{n}},\forall s\le 20\text{\hspace{0.17em}log\hspace{0.17em}}n$$

with probability 1 − O(n⁻¹⁰).

Proof. See Appendix 6. □

In addition, in view of Lemma 1 and the condition (14), one has

$$\Vert H\Vert \lesssim \text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}<1/10$$ (33)

with probability 1 − O(n⁻¹⁰), which together with Lemma 2 implies λ ≥ 3∥H∥. This further leads to

$$\begin{array}{ll}\sum _{s:s\ge 20\text{\hspace{0.17em}log\hspace{0.17em}}n}{\left(\frac{\Vert H\Vert }{\lambda }\right)}^s\hfill & \le \frac{\Vert H\Vert }{\lambda }\sum _{s:s\ge 20\text{\hspace{0.17em}log\hspace{0.17em}}n-1}{\left(\frac{\Vert H\Vert }{\lambda }\right)}^s\le \frac{\Vert H\Vert }{\lambda }\sum _{s:s\ge 20\text{\hspace{0.17em}log\hspace{0.17em}}n-1}\frac{1}{3^s}\hfill \\ \hfill & \lesssim \text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\cdot n^{-10}.\hfill \end{array}$$

Putting the above bounds together and using the fact that λ is real-valued and λ ≥ 1/2 (cf. Lemma 2), we have

$$\begin{array}{l}|a^{\top }\left(u-\frac{u^{\star \top }u}{\lambda }{u}^{\star }\right)|\hfill \\ =|\frac{u^{\star \top }u}{\lambda }\sum _{s=1}^{+\infty }\frac{a^{\top }{H}^s{u}^{\star }}{{\lambda }^s}|\hfill \\ \lesssim \sum _{s=1}^{20\text{\hspace{0.17em}log\hspace{0.17em}}n}\frac{1}{{\lambda }^s}|a^{\top }{H}^s{u}^{\star }|+\sum _{s=20\text{\hspace{0.17em}log\hspace{0.17em}}n}^{+\infty }{\left(\frac{\Vert H\Vert }{\lambda }\right)}^s\hfill \\ \le \sqrt{\frac{\mu }{n}}\sum _{s=1}^{20\text{\hspace{0.17em}log\hspace{0.17em}}n}{\left(2c_2\text{\hspace{0.17em}max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\right)}^s+\frac{\text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}}{n^{10}}\hfill \\ \lesssim \text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\sqrt{\frac{\mu }{n}},\hfill \end{array}$$

as long as max $\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}$ is sufficiently small. Here, the last line also uses the fact that μ ≥ 1 (and hence $\sqrt{\mu /n}\gg n^{-10}$). This concludes the proof.

5. Extension: perturbation analysis for the rank-r case.

5.1. Eigenvalue perturbation for the rank-r case.

The eigenvalue perturbation analysis in Section 4 can be extended to accommodate the case where M^⋆ is symmetric and rank-r, as detailed in this section. As before, assume that the r non-zero eigenvalues of M^⋆ obey ${\lambda }_{{\text{max}}^{\star }}=|{\lambda }_1^{\star }|\ge \cdots \ge |{\lambda }_r^{\star }|={\lambda }_{{\text{min}}^{\star }}$. Once again, we start with a master bound.

Theorem 4 (Perturbation of linear forms of eigenvectors (rank-r)).

Consider a rank-r symmetric matrix $M^{\star }\in {ℝ}^{n\times n}$ with incoherence parameter μ. Define $\kappa \triangleq {\lambda }_{\text{max}}^{*}/{\lambda }_{\text{min}}^{*}$. Suppose that

$$\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }_{\text{max}}^{*}}\le \frac{c_1}{\kappa }$$ (34)

for some sufficiently small constants c₁ > 0. Then for any fixed unit vector $a\in {ℝ}^n$ and any 1 ≤ l ≤ r, with probability at least 1 − O(n⁻¹⁰) one has

$$\begin{array}{lll}(35)\hfill & a^{\top }\left(u_l-\sum _{j=1}^r\frac{{\lambda }_j^{\star }{u}_j^{\star \top }{u}_l}{{\lambda }_l}{u}_j^{\star }\right)\mid \hfill & \lesssim \text{max}\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\}\frac{\kappa }{\left|{\lambda }_l\right|}\sqrt{\frac{\mu r}{n}}\hfill \\ (36)\hfill & \hfill & \lesssim \frac{\text{max}\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\}}{{\lambda }_{\text{max}}^{*}}{\kappa }^2\sqrt{\frac{\mu r}{n}}.\hfill \end{array}$$

This result allows us to control the perturbation of the linear form of eigenvectors. The perturbation upper bound grows as either the rank r or the condition number κ increases.

One of the most important consequences of Theorem 4 is a refinement of the Bauer-Fike theorem concerning eigenvalue perturbations as follows.

Corollary 5.

Consider the lth (1 ≤ l ≤ r) eigenvalue λ_l of M. Under the assumptions of Theorem 4, with probability at least 1 − O(n⁻¹⁰), there exists 1 ≤ j ≤ r such that

$$|{\lambda }_l-{\lambda }_j^{\star }|\lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\kappa r\sqrt{\frac{\mu }{n}},$$ (37)

provided that

$$\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }_{\text{max}}^{*}}\le c_1/{\kappa }^2$$ (38)

for some sufficiently small constant c₁ > 0.

Proof. See Appendix 7. □

In comparison, the Bauer-Fike theorem (Lemma 2) together with Lemma 1 gives a perturbation bound

$$|{\lambda }_l-{\lambda }_j^{\star }|\le \Vert H\Vert \lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\text{\hspace{1em} for some }1\le j\le r.$$ (39)

For the low-rank case where $r\ll \sqrt{n}$, the eigenvalue perturbation bound derived in Corollary 5 can be much sharper than the Bauer-Fike theorem.

Another result that comes from Theorem 4 is the following bound that concerns linear forms of the eigen-subspace.

Corollary 6.

Under the same setting of Theorem 4, with probability 1 − O(n⁻⁹) we have

$${‖a^{\top }U‖}_2\lesssim \kappa \sqrt{r}{‖a^{\top }{U}^{\star }‖}_2+\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }_{\text{max}}^{*}}{\kappa }^{2}r\sqrt{\frac{\mu }{n}}.$$ (40)

Proof. See Appendix 8. □

Consequently, by taking a = e_i (1 ≤ i ≤ n) in Corollary 6, we arrive at the following statement regarding the alternative definition of the incoherence of the eigenvector matrix U (see Remark 2).

Corollary 7.

Under the same setting of Theorem 4, with probability 1 − O(n⁻⁸) we have

$$\Vert U{\Vert }_{2,\infty }\lesssim \kappa r\sqrt{\frac{\mu }{n}}.$$ (41)

Proof. Given that $\Vert U{\Vert }_{2,\infty }={\text{max}}_{1\le i\le n}{\Vert e_i^{\top }U\Vert }_2$ and recalling our assumption implies ${\Vert U^{\star }\Vert }_{2,\infty }\le \sqrt{\mu r/n}$, we can invoke Corollary 6 and the union bound to derive the advertised entrywise bounds. □

Remark 7.

The eigenvector matrix is often employed to form a reasonably good initial guess for several nonconvex statistical estimation problems Keshavan, Montanari and Oh (2010), and the above kind of incoherence property is crucial in guaranteeing fast convergence of the subsequent nonconvex iterative refinement procedures Ma et al. (2019).

Unfortunately, these results fall short of providing simple perturbation bounds for the eigenvectors; in other words, the above-mentioned bounds do not imply the size of the difference between U and U^⋆. The challenge arises in part due to the lack of orthonormality of the eigenvectors when dealing with asymmetric matrices. Analyzing the eigenspace perturbation for the general rank-r case will likely require new analysis techniques, which we leave for future work. There is, however, some special case in which we can develop eigenvector perturbation theory, as detailed in the next subsection.

Remark 8.

The theory for the rank-r case has recently been significantly improved; see our follow-up work Cheng, Wei and Chen (2020) for details.

5.2. Application: spectral estimation when M^⋆ is asymmetric and rank-1.

In some scenarios, the above general rank results allow us to improve spectral estimation when M^⋆ is asymmetric. Consider the case where $M^{\star }={\lambda }^{\star }{u}^{\star }{v}^{\star \top }\in {ℝ}^{n_1\times n_2}$ is an asymmetric rank-1 matrix with leading singular value λ^⋆. Suppose that we observe two independent noisy copies of M^⋆, namely,

$$M_1=M^{\star }+H_1,M_2=M^{\star }+H_2,$$ (42)

where H₁ and H₂ are independent noise matrices. The goal is to estimate the singular value and singular vectors of M^⋆ from M₁ and M₂.

We attempt estimation via the standard dilation trick (e.g. Tao (2012)). This consists of embedding the matrices of interest within a larger block matrix

$$M_{\text{d}}^{\star }\triangleq \left[\begin{array}{cc}0& M^{\star }\\ M^{\star \top }& 0\end{array}\right],M_{\text{d}}\triangleq \left[\begin{array}{cc}0& M_1\\ M_2^{\top }& 0\end{array}\right].$$ (43)

Here, we place M₁ and M₂ in two different subblocks, in order to “asymmetrize” the dilation matrix. The rationale is that $M_{\text{d}}^{\star }$ is a rank-2 symmetric matrix with exactly two nonzero eigenvalues

$${\lambda }_1\left(M_{\text{d}}^{\star }\right)={\lambda }^{\star }\text{\hspace{1em} and \hspace{1em}}{\lambda }_2\left(M_{\text{d}}^{\star }\right)=-{\lambda }^{\star },$$

whose corresponding eigenvectors are given by

$$\frac{1}{\sqrt{2}}\left(\begin{array}{l}{u}^{\star }\hfill \\ v^{\star }\hfill \end{array}\right)\text{\hspace{1em} and \hspace{1em}}\frac{1}{\sqrt{2}}\left(\begin{array}{l}{u}^{\star }\hfill \\ -v^{\star }\hfill \end{array}\right),$$

respectively. This motivates us to perform eigen-decomposition of M_d, and use the top-2 eigenvalues and eigenvectors to estimate λ^⋆, u^⋆ and v^⋆, respectively.

Eigenvalue perturbation analysis.

As an immediate consequence of Corollary 5, the two leading eigenvalues of M_d provide fairly accurate estimates of the leading singular value λ^⋆ of M^⋆, as stated below.

Corollary 8.

Assume $M^{\star }\in {ℝ}^{n_1\times n_2}$ is a rank-1 matrix with leading singular value λ^⋆ and incoherence parameter μ. Define $n\triangleq n_1+n_2$. Suppose that ${\lambda }_1^{\text{d}}\ge {\lambda }_2^{\text{d}}$ are the two leading eigenvalues of M_d (cf. (43)), and that H₁ and H₂ are independent and satisfy Assumption 1. Then with probability at least 1 − O(n⁻¹⁰),

$$\text{max}\left\{|{\lambda }_1^{\text{d}}-{\lambda }^{\star }|,|{\lambda }_2^{\text{d}}+{\lambda }^{\star }|\right\}\lesssim \text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}\sqrt{\frac{\mu }{n}},$$ (44)

provided that

$$\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }}\le c_1$$ (45)

for some sufficiently small constant c₁ > 0.

Proof. To begin with, it follows from Corollary 5 that both ${\lambda }_1^{\text{d}}$ and ${\lambda }_2^{\text{d}}$ are close to either λ^⋆ or −λ^⋆. Repeating similar arguments as in the proof of Lemma 2 (which we omit here), we can immediately show the separation between these two eigenvalues, namely, ${\lambda }_1^{\text{d}}$ (resp. ${\lambda }_2^{\text{d}}$) is close to λ^⋆ (resp. −λ^⋆). □

Eigenvector perturbation analysis.

We then move on to studying the eigenvector perturbation bounds. Specifically, denote by $u_1^{\text{d}}$ and $u_2^{\text{d}}$ the eigenvectors of M_d associated with its two leading eigenvalues ${\lambda }_1^{\text{d}}$ and ${\lambda }_2^{\text{d}}$, respectively. Without loss of generality, we assume that ${\lambda }_1^{\text{d}}\ge {\lambda }_2^{\text{d}}$. If we write

$$u_1^{\text{dilation }}=\left({}_{u_{1,2}^{\text{d}}}^{u_{1,1}^{\text{d}}}\right)\text{\hspace{1em} with }{u}_{1,1}^{\text{d}}\in {ℝ}^{n_1},u_{1,2}^{\text{d}}\in {ℝ}^{n_2},$$

then we can employ $u_{1,1}^{\text{d}}$ and $u_{1,2}^{\text{d}}$ to estimate u^⋆ and v^⋆ after proper normalization, namely,

$$u\triangleq \frac{u_{1,1}^{\text{d}}}{{‖u_{1,1}^{\text{d}}‖}_2},v\triangleq \frac{u_{1,2}^{\text{d}}}{{‖u_{1,2}^{\text{d}}‖}_2}.$$ (46)

The following theorem develops error bounds for both u and v, which we establish in Appendix 9. Here, we denote min∥x ± y∥₂ = min{∥x − y∥₂, ∥x + y∥₂}, and min∥x ± y∥_∞ = min{∥x − y∥_∞, ∥x + y∥_∞}.

Theorem 5.

Suppose $M^{\star }={\lambda }^{\star }{u}^{\star }{v}^{\star \top }\in {ℝ}^{n_1\times n_2}$ is a rank-1 matrix with leading singular value λ^⋆ and incoherence parameter μ, where ∥u^⋆∥₂ = ∥v^⋆∥₂ = 1. Define $n\triangleq n_1+n_2$, and fix any unit vectors $a\in {ℝ}^{n_1}$ and $b\in {ℝ}^{n_2}$. Then with probability at least 1 − O(n⁻¹⁰), the estimates u and v (cf. (46)) obey

$$\text{max}\left\{\text{min}{\Vert u\pm u^{\star }\Vert }_2,\text{min}{\Vert v\pm v^{\star }\Vert }_2\right\}\lesssim \frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }},$$ (47a)

$$\text{max}\left\{\text{min}{\Vert u\pm u^{\star }\Vert }_{\infty },\text{min}{\Vert v\pm v^{\star }\Vert }_{\infty }\right\}\lesssim \frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }}\sqrt{\frac{\mu }{n}},$$ (47b)

$$\text{min}|a^{\top }\left(u\pm u^{\star }\right)|\lesssim \left(|a^{\top }{u}^{\star }|+\sqrt{\frac{\mu }{n}}\right)\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }},$$ (47c)

$$\text{min}|b^{\top }\left(v\pm v^{\star }\right)|\lesssim \left(|b^{\top }{v}^{\star }|+\sqrt{\frac{\mu }{n}}\right)\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }},$$ (47d)

provided that there exists some some sufficiently small constant c₁ > 0 such that

$$\frac{\text{max}\left\{\sigma \sqrt{n\text{\hspace{0.17em}log\hspace{0.17em}}n},B\text{\hspace{0.17em}log\hspace{0.17em}}n\right\}}{{\lambda }^{\star }}\le c_1.$$ (48)

Similar to the symmetric rank-1 case, the estimation errors of the estimates u and v are well-controlled in any deterministic direction (e.g. the entrywise errors are well-controlled). This allows us to complete the theory for the case when M^⋆ is a real-valued and rank-1 matrix.

Further, we conduct numerical experiments for matrix completion when M^⋆ is a rank-1 and asymmetric matrix in Fig. 4. Here, we suppose that at most 1 sample is observed for each entry, and we estimate the singular value and singular vectors of M^⋆ via the above-mentioned dilation trick, coupled with the asymmetrization procedure discussed in Section 10. The numerical performance confirms that the proposed technique outperforms vanilla SVD in spectral estimation.

Fig 4:

Numerical experiments for rank-1 matrix completion, where $M^{\star }=u^{\star }{v}^{\star \top }\in {ℝ}^{n_1\times n_2}$ is randomly generated with leading singular value λ^⋆ = 1. Let n = n₁ = 2n₂. Each entry is observed independently with probability p. (a) |λ − λ^⋆| vs. n with p = 3 log n/n; (b) |λ − λ^⋆| vs. p with n = 1000; (c) ℓ_∞ eigenvector estimation error vs. n with p = 3 log n/n. The blue (resp. red) lines represent the average errors over 100 independent trials using using the eigen-decomposition (resp. SVD) approach applied to M_dilation (resp. M).

Finally, we remark that the asymptotic behavior of the eigenvalues of asymmetric random matrices has been extensively explored in the physics literature (e.g. (Sommers et al., 1988; Khoruzhenko, 1996; Brezin and Zee, 1998; Chalker and Mehlig, 1998; Feinberg and Zee, 1997; Lytova and Tikhomirov, 2018)). Their focus, however, has largely been to pin down the asymptotic density of the eigenvalues, similar to the semi-circle law in the symmetric case. Nevertheless, a sharp perturbation bound for the leading eigenvalue — particularly for the low-rank case — is beyond their reach. A few recent papers began to explore the locations of eigenvalue outliers that fall outside the bulk predicted by the circular law Tao (2013); Rajagopalan (2015); Benaych-Georges and Rochet (2016); Bordenave and Capitaine (2016). The results reported therein either do not focus on obtaining the right convergence rate (e.g. providing only a bound like |λ − λ^⋆| = o(|λ^⋆|)) or are restricted to a special family of ground truth (e.g. the one with a diagonal block equal to identity) or i.i.d. noise. As a result, these prior results are insufficient to demonstrate the power and benefits of the eigen-decomposition method in the presence of data asymmetry.

5.3. Proof of Theorem 4.

Without loss of generality, we shall assume ${\lambda }_{\text{max}}^{*}={\lambda }_1^{\star }=1$ throughout the proof. To begin with, Lemma 2 implies that for all 1 ≤ l ≤ r,

$$\left|{\lambda }_l\right|\ge \left|{\lambda }_{\text{max}}^{*}\right|-\Vert H\Vert >1/(2\kappa )>\Vert H\Vert$$ (49)

as long as ∥H∥ < 1/(2κ). In view of the Neumann trick (Theorem 2), we can derive

$$\begin{array}{ll}(50)\hfill & \left|a^{\top }{u}_l-\sum _{j=1}^r\frac{{\lambda }_j^{\star }{u}_j^{\star \top }{u}_l}{{\lambda }_l}{a}^{\top }{u}_j^{\star }\right|\hfill \\ \hfill & =\left|\sum _{j=1}^r\frac{{\lambda }_j^{\star }}{{\lambda }_l}\left(u_j^{\star \top }{u}_l\right)\left\{\sum _{s=1}^{\infty }\frac{1}{{\lambda }_l^s}{a}^{\top }{H}^s{u}_j^{\star }\right\}\right|\hfill \\ \hfill & \le \left(\sum _{j=1}^r\frac{\left|{\lambda }_j^{\star }\right|}{\left|{\lambda }_l\right|}\left|u_j^{\star \top }{u}_l\right|\right)\left\{\underset{1\le j\le r}{\text{max}}\sum _{s=1}^{\infty }\frac{1}{{\left|{\lambda }_l\right|}^s}\left|a^{\top }{H}^s{u}_j^{\star }\right|\right\}\hfill \\ \hfill & \le \sqrt{r\sum _{j=1}^r{\left|u_j^{\star \top }{u}_l\right|}^2}\left\{\underset{1\le j\le r}{\text{max}}\frac{\left|{\lambda }_j^{\star }\right|}{\left|{\lambda }_l\right|}\right\}\left\{\underset{1\le j\le r}{\text{max}}\sum _{s=1}^{\infty }\frac{1}{{\left|{\lambda }_l\right|}^s}\left|a^{\top }{H}^s{u}_j^{\star }\right|\right\}\hfill \\ (51)\hfill & \le \sqrt{r}\cdot \frac{1}{\left|{\lambda }_l\right|}\cdot \left\{\underset{1\le j\le r}{\text{max}}\sum _{s=1}^{\infty }\frac{1}{{\left|{\lambda }_l\right|}^s}\left|a^{\top }{H}^s{u}_j^{\star }\right|\right\},\hfill \end{array}$$

where the third line follows since ${\sum }_{J=1}^R{|u_j^{\star \top }{u}_l|}^2\le {\Vert u_l\Vert }_2^2=1$, and the last inequality makes use of (49). Apply Corollary 4 to reach

$$\begin{array}{ll}(51)\hfill & \le \frac{\sqrt{r}}{\left|{\lambda }_l\right|}\sum _{s=1}^{\infty }{\left(2c_2\kappa \text{\hspace{0.17em}max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\right)}^s\sqrt{\frac{\mu }{n}}\hfill \\ \hfill & \lesssim \frac{\kappa }{\left|{\lambda }_l\right|}\text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\sqrt{\frac{\mu r}{n}}\hfill \\ \hfill & \lesssim {\kappa }^2\text{\hspace{0.17em}max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\sqrt{\frac{\mu r}{n},}\hfill \end{array}$$

with the proviso that |λ_l| > 1/(2κ) and $\text{max}\left\{B\text{\hspace{0.17em}log\hspace{0.17em}}n,\sqrt{n{\sigma }^2\text{\hspace{0.17em}log\hspace{0.17em}}n}\right\}\le c_1/\kappa$ for some sufficiently small constant c₁ > 0. The condition |λ_l| > 1/(2κ) follows immediately by combining Lemma 2, Lemma 1 and the condition (34).

6. Discussions.

In this paper, we demonstrate the remarkable advantage of eigen-decomposition over SVD in the presence of asymmetric noise matrices. This is in stark contrast to conventional wisdom, which is generally not in favor of eigen-decomposition for asymmetric matrices. Our results only reflect the tip of an iceberg, and there are many outstanding issues left answered. We conclude the paper with a few future directions.

Sharper eigenvalue perturbation bounds for the rank-r case.

Our current results in Section 5 provide an eigenvalue perturbation bound on the order of $r/\sqrt{n}$, assuming the truth is rank-r. However, numerical experiments suggest that the dependency on r might be improvable. It would be interesting to see whether further theoretical refinement is possible, e.g. whether it is possible to improve it to $O(\sqrt{r/n})$.

Eigenvector perturbation bounds for the rank-r case.

As mentioned before, the current theory falls short of providing eigenvector perturbation bounds for the general rank-r case. The main difficulty lies in the lack of orthogonality of the eigenvectors of the observed matrix M. Nevertheless, when the size of the noise is not too large, it is possible to establish certain near-orthogonality of the eigenvectors, which might in turn lead to sharp control of eigenvector perturbation.

A challenging signal-to-noise ratio regime.

Take the rank-1 case for example: the present work focuses on the regime where $\Vert H\Vert \lesssim ‖M^{\star }‖/\sqrt{\text{log\hspace{0.17em}}n}$, and it is known that spectral methods fail to yield reliable estimation if ∥H∥ ≫ ∥M^⋆∥. There is, however, a “gray” region (which includes, for example, the case with ∥H∥ ≈ ∥M^⋆∥) that has not been addressed. Developing non-asymptotic yet informative perturbation bounds for this regime is likely very challenging and requires new analysis techniques, which we leave for future investigation.

Correlated noise.

The current theoretical development relies heavily on the assumption that the noise matrix H contains independent random entries. There is no shortage of examples where the noise matrix is asymmetric but is not composed of independent entries. For instance, in blind deconvolution Li et al. (2018), the noise matrix is a sum of independent asymmetric matrices. Can we develop eigenvalue perturbation theory for this class of noise?

Statistical inference of eigenvalues and eigenvectors.

In various applications like network analysis and inference, one might be interested in determining the (asymptotic) eigenvalue and eigenvector distributions of a random data matrix, in order to produce valid confidence intervals Johnstone (2001); Bai and Yao (2008); Cai, Han and Pan (2017); Cape, Tang and Priebe (2018); Xia (2018); Bao, Ding and Wang (2018); Chen et al. (2019c). Can we use the current framework to characterize the distributions of the leading eigenvalues as well as certain linear forms of the eigenvectors of M when the noise matrix is non-symmetric?

Asymmetrization for other applications.

Given the abundant applications of spectral estimation, our findings are likely to be useful for other matrix eigenvalue problems and might extend to the tensor case Zhang and Xia (2018); Cai et al. (2019). Here, we conclude the paper with an example in covariance estimation (Baik and Silverstein, 2006; Fan, Wang and Zhong, 2018). Imagine that we observe a collection of n independent Gaussian vectors $X_1,\cdots ,X_n\in {ℝ}^d$, which have mean zero and covariance matrix

$${\mathbf{\Sigma }}^{\star }=v{v}^{\top }+I_d$$ (52)

with v being a unit vector. This falls under the category of the spiked covariance model (Johnstone and Lu, 2009). One strategy to estimate the spectral norm λ^⋆ = 2 of Σ^⋆ is to look at the spectrum of the sample covariance matrix $\widehat{\mathbf{\Sigma }}=\frac{1}{n}{\sum }_{i=1}^n{\mathit{X}}_i{\mathit{X}}_i^{\top }$. Motivated by the results of this paper, we propose an alternative strategy by looking at the following asymmetrized sample covariance matrix

$${\widehat{\mathbf{\Sigma }}}_{\text{asym }}=\frac{2}{n}\left({{\sum }_{i=1}^{n/2}\text{Upper}}_{}\left(X_i{X}_i^{\top }\right)+{\sum }_{i=n/2+1}^n\text{Lower }\left(X_i{X}_i^{\top }\right)\right),$$ (53)

where Upper(·) (resp. Lower(·)) extracts out the upper (resp. lower) triangular part of the matrix, including (resp. excluding) the diagonal entries. As can be seen from Fig. 5, the largest eigenvalue of the asymmetrized ${\widehat{\mathbf{\Sigma }}}_{\text{asym }}$ is much closer to the true spectral norm of Σ^⋆, compared to the largest singular value of the sample covariance matrix $\widehat{\mathbf{\Sigma }}$. We leave the theoretical understanding of such findings to future investigation.

Fig 5:

Numerical experiments for the spiked covariance model, where the sample vectors are zero-mean Gaussian vectors with covariance matrix Σ^⋆ (cf. (52)). We plot |λ − λ^⋆| vs. n with d = n/10. The blue (resp. red) lines represent the average errors over 100 independent trials when λ is the leading eigenvalue of ${\widehat{\mathbf{\Sigma }}}_{\text{asym }}$ (resp. $\widehat{\mathbf{\Sigma }}$).