ENTRYWISE EIGENVECTOR ANALYSIS OF RANDOM MATRICES WITH LOW EXPECTED RANK

Abstract

Recovering low-rank structures via eigenvector perturbation analysis is a common problem in statistical machine learning, such as in factor analysis, community detection, ranking, matrix completion, among others. While a large variety of bounds are available for average errors between empirical and population statistics of eigenvectors, few results are tight for entrywise analyses, which are critical for a number of problems such as community detection.

This paper investigates entrywise behaviors of eigenvectors for a large class of random matrices whose expectations are low-rank, which helps settle the conjecture in Abbe et al. (2014b) that the spectral algorithm achieves exact recovery in the stochastic block model without any trimming or cleaning steps. The key is a first-order approximation of eigenvectors under the ℓ_∞ norm:

$$u_k\approx \frac{A{u}_k^{*}}{{\lambda }_k^{*}},$$

where {u_k} and $\left\{u_k^{*}\right\}$ are eigenvectors of a random matrix A and its expectation $\mathbb{E}A$, respectively. The fact that the approximation is both tight and linear in A facilitates sharp comparisons between u_k and $u_k^{*}$. In particular, it allows for comparing the signs of u_k and $u_k^{*}$ even if ${\Vert u_k-u_k^{*}\Vert }_{\infty }$ is large. The results are further extended to perturbations of eigenspaces, yielding new ℓ_∞-type bounds for synchronization (${\mathbb{Z}}_2$-spiked Wigner model) and noisy matrix completion.

1. Introduction

Many estimation problems in statistics involve low-rank matrix estimators that are NP-hard to compute, and many of these estimators are solutions to nonconvex programs. This is partly because of the widespread use of maximum likelihood estimation (MLE) which, while enjoying good statistical properties, often poses computational challenges due to nonconvex or discrete constraints inherent in the problems.

Fortunately, computationally efficient algorithms using eigenvectors often afford good performance. The eigenvectors either directly lead to final estimates (Shi and Malik, 2000; Ng et al., 2002), or serve as warm starts followed by further refinements (Keshavan et al., 2010a; Jain et al., 2013; Candès et al., 2015). Such algorithms mostly rely on computation of leading eigenvectors and matrix-vector multiplications, which are easily implemented.

While various heuristics abound, theoretical understanding remains scarce on the entrywise analysis, and on when refinements are needed or can be avoided. In particular, it remains open in various cases to determine whether a vanilla eigenvector-based method without preprocessing steps (e.g., trimming of outliers) or without refinement steps (e.g., cleaning with local improvements) enjoys the same optimality results as the MLE (or SDP) does. A crucial missing step is a sharp entrywise perturbation analysis of eigenvectors. This is party because the ℓ_∞ distance between the eigenvectors of a random matrix and their expected counterparts may not be the correct quantity to look at; errors per entry can be asymmetrically distributed, as we shall see in this paper.

This paper investigates entrywise behaviors of eigenvectors and more generally, eigenspaces, for random matrices with low expected rank using the following approach. Let A be a random matrix, $A^{*}=\mathbb{E}A$, and E = A − A* be the ‘error’ of A. In many cases, A* is a symmetric matrix with low rank determined by the structure of a statistical problem, such as low-rank with blocks in community detection.

Consider for now the case of symmetric A, and let u_k, resp. $u_k^{*}$, be the eigenvector corresponding to the k-th largest eigenvalue of A, resp. A*. Roughly speaking, if E is moderate, our first-order approximation reads

$$u_k=\frac{A{u}_k}{{\lambda }_k}\approx \frac{A{u}_k^{*}}{{\lambda }_k^{*}}=u_k^{*}+\frac{E{u}_k^{*}}{{\lambda }_k^{*}}.$$

While u_k is a nonlinear function of A (or equivalently E), the approximation is linear in A, which greatly facilitates the analysis. Under certain conditions, the maximum entrywise approximation error ${\Vert u_k-A{u}_k^{*}/{\lambda }_k^{*}\Vert }_{\infty }$ can be much smaller than ${\Vert u_k^{*}\Vert }_{\infty }$, allowing us to study u_k through $A{u}_k^{*}/{\lambda }_k^{*}$. To obtain such results, a key part in our theory is to characterize concentration properties of A and structural assumptions on its expectation A*.

This perturbation analysis leads to new and sharp theoretical guarantees. In particular, we find that for the exact recovery problem in stochastic block model, the vanilla spectral algorithm (without trimming or cleaning) achieves the information-theoretic limit, and it coincides with the MLE estimator whenever the latter succeeds. This settles in particular a conjecture left open in Abbe et al. (2014b, 2016). Therefore, MLE and SDP do not have advantage over the spectral method in terms of exact recovery, if the model is correct. SDP may be preferred in some applications for its robustness and optimality certificates, but that is beyond the scope of this paper.

1.1. A sample problem

Let us consider a network model that has received widespread interest in recent years: the stochastic block model (SBM). Suppose that we have a graph with vertex set {1, 2, ⋯, n}, and assume for simplicity that n is even. There is an unknown index set J G {1, 2, ⋯, n} with |J| = n/2 such that the vertex set is partitioned into two groups J and J^c. Within groups, there is an edge between each pair of vertices with probability p, and between groups, there is an edge with probability q. Let $x\in {ℝ}^n$ be the group membership vector with x_i = 1 if i ∈ J and x_i = −1 otherwise. The goal is to recover x from the observed edges of the graph.

This random-graph-based model was first proposed for social relationship networks (Holland et al., 1983), and many more realistic models have been developed based on the SBM since then. Given its fundamental importance, there are a plurality of papers addressing statistical properties and algorithmic efficiencies; see Abbe (2017) for a survey.

Under the regime $p=\frac{a\log n}{n}$, $q=\frac{b\log n}{n}$ where a > b > 0 are constants, Abbe et al. (2016) and Mossel et al. (2014) proved that exact recovery is possible if and only if $\sqrt{a}-\sqrt{b}>\sqrt{2}$, and that the limit can be achieved by efficient algorithms. They used two-round procedures (with a clean-up phase) to achieve the threshold. Semidefinite relaxations are also known to achieve the threshold (Abbe et al., 2016; Hajek et al., 2016; Agarwal et al., 2015; Bandeira, 2015), as well as spectral methods with local refinements (Abbe and Sandon, 2015; Yun and Proutiere, 2016; Gao et al., 2015). We will discuss more in Sections 1.5 and 3.2.

While existing works tackle exact recovery rather successfully, some fundamental questions remain unsolved: how do the simple statistics—top eigenvectors of the adjacency matrix—behave? Are they informative enough to reveal the group structure under very challenging regimes?

To study these questions, we start with the eigenvectors of $A^{*}=\mathbb{E}A$. By definition, A_ij is a Bernoulli random variable, and $\mathbb{P}\left(A_{ij}=1\right)$ depends on whenever i and j are from the same groups. The expectation $\mathbb{E}A$ must be a block matrix of the following form:

$$\mathbb{E}A=\frac{\log n}{n}\left(\begin{array}{cc}a{\mathbf{1}}_{\frac{n}{2}\times \frac{n}{2}}& b{\mathbf{1}}_{\frac{n}{2}\times \frac{n}{2}}\\ b{\mathbf{1}}_{\frac{n}{2}\times \frac{n}{2}}& a{\mathbf{1}}_{\frac{n}{2}\times \frac{n}{2}}\end{array}\right),$$

where 1_m×m is the m × m all-one matrix. Here, for convenience, we represent $\mathbb{E}A$ as if J = {1, 2, ⋯, n/2}. But in general J is unknown, and there is a permutation of indices {1, ⋯, n} in the matrix representation.

From the matrix representation it is clear that $\mathbb{E}A$ has rank 2, with two nonzero eigenvalues ${\lambda }_1^{*}=\frac{a+b}{2}\log n$ and ${\lambda }_2^{*}=\frac{a-b}{2}\log n$. Simple calculations give the corresponding (normalized) eigenvectors: $u_1^{*}=\frac{1}{\sqrt{n}}{\mathbf{1}}_n$, and ${\left(u_2^{*}\right)}_i=1/\sqrt{n}$ if i ∈ J and ${\left(u_2^{*}\right)}_i=-1/\sqrt{n}$ if i ∈ J^c. Since $u_2^{*}$ perfectly aligns with the group assignment vector x, we hope to show its counterpart u₂, i.e., the second eigenvector of A, also has desirable properties.

The first reassuring fact is that, the top eigenvalues preserve proper ordering: by Weyl’s inequality, the deviation of any eigenvalue λ_i (i ∈ [n]) from ${\lambda }_i^{*}$ is bounded by ||A−A*||₂, which is $O(\sqrt{\log n})$ with high probability; see supplementary materials (Abbe et al., 2018). The Davis-Kahan sin Θ theorem asserts that u₁ and u₂ are weakly consistent estimators for $u_1^{*}$ and $u_2^{*}$ respectively, in the sense that $\left|\langle u_k,u_k^{*}\rangle \right|\stackrel{\mathbb{P}}{\to }1$ for k = 1, 2. However, this is not helpful for understanding their entrywise behaviors in the uniform sense, which is crucial for exact recovery. Nor can it explain the sharp phase transition phenomenon. This makes entrywise analysis both interesting and challenging.

This problem motivates some simulations about the coordinates of top eigenvectors of A. In Figure 1, we calculate the rescaled second eigenvector $\sqrt{n}{u}_2$ of one typical realization A, and make a histogram plot of its coordinates. (Note the first eigenvector is aligned with the all-one vector 1_n, which is uninformative.) The parameters we choose are n = 5000, a = 4.5 and b = 0.25, for which exact recovery is possible with high probability. Visibly, the coordinates of $\sqrt{n}{u}_2$ form two clusters around ±1 which, marked by red dashed lines, are coordinates of $\sqrt{n}{u}_2^{*}$. Intuitively, the signs of the former should suffice to reveal the group structure.

Fig 1:

The second eigenvector and its first-order approximation in SBM. Left: The histogram of coordinates of $\sqrt{n}{u}_2$ computed from a single realization of adjacency matrix A, where n is 5000, a is 4.5 and b is 0.25. Exact recovery is expected as coordinates form two well-separated clusters. Right: boxplots showing three different distance/errors (up to sign) over 100 realizations: (i) $\sqrt{n}{\Vert u_2-u_2^{*}\Vert }_{\infty }$, (ii) $\sqrt{n}{\Vert A{u}_2^{*}/{\lambda }_2^{*}-u_2^{*}\Vert }_{\infty }$, (iii) $\sqrt{n}{\Vert u_2-A{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty \cdot }A{u}_2^{*}/{\lambda }_2^{*}$ is a good approximation of u₂ under ℓ_∞ norm even though ${\Vert u_2-u_2^{*}\Vert }_{\infty }$ may be large.

To probe into the second eigenvector u₂, we expand the perturbation $u_2-u_2^{*}$ as follows:

$$u_2-u_2^{*}=\left(\frac{A{u}_2^{*}}{{\lambda }_2^{*}}-u_2^{*}\right)+\left(u_2-\frac{A{u}_2^{*}}{{\lambda }_2^{*}}\right).$$ (1.1)

The first term is exactly $E{u}_2^{*}/{\lambda }_2^{*}$, which is linear in E and can be viewed as the first-order perturbation. The second term is nonlinear in general, representing the error of higher order. Figure 1 shows boxplots of the infinity norm of rescaled perturbation errors over 100 realizations (see (i)-(iii)), which illustrates that ${\Vert u_2-A{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty }$ is much smaller than ${\Vert u_2-u_2^{*}\Vert }_{\infty }$ and ${\Vert A{u}_2^{*}/{\lambda }_2^{*}-u_2^{*}\Vert }_{\infty }$. Indeed, we will see in Theorem 1.1 that

$${\Vert u_2-A{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty }=o_{\mathbb{P}}\left(\underset{i}{\min }\left|{\left(u_2^{*}\right)}_i\right|\right)=o_{\mathbb{P}}(1/\sqrt{n}).$$ (1.2)

The result holds ‘up to sign’, i.e. can choose an appropriate sign for the eigenvector u₂ as it is not uniquely defined; see Theorem 1.1 for its precise meaning. Therefore, the entrywise behavior of $u_2-u_2^{*}$ is captured by its first-order term, which is much more amenable to analysis. This observation will finally lead to sharp eigenvector results in Section 3.2.

We remark that it is also possible to study the top eigenvector (denoted as $\overline{u}$) of the centered adjacency matrix $\overline{A}=A-\frac{\widehat{d}}{n}{\mathbf{1}}_n{\mathbf{1}}_n^T$, where $\widehat{d}={\sum }_{i,j}{A}_{ij}/n$ is the average degree of all vertices. The top eigenvector of $\mathbb{E}\overline{A}$ is exactly $u_2^{*}$, and its empirical counterpart $\overline{u}$ is very similar to u₂. In fact, the same reasoning and analysis applies to $\overline{u}$, and one obtains similar plots as Figure 1 (omitted here).

1.2. First-order approximation of eigenvectors

Now we present a simpler version of our result that justifies the intuitions above. Consider a general symmetric random matrix (more precisely, this should be a sequence of random matrices with growing dimensions) $A\in {ℝ}^{n\times n}$ with independent entries on and above its diagonal. Suppose its expectation $A^{*}=\mathbb{E}A\in {ℝ}^{n\times n}$ is low-rank and has r nonzero eigenvalues. Let us assume that

• (a)r = O(1), these r eigenvalues are positive and in descending order $\left({\lambda }_1^{*}\ge {\lambda }_2^{*}\ge \cdots \ge {\lambda }_r^{*}>0\right)$, and ${\lambda }_1^{*}\asymp {\lambda }_r^{*}$.

Their corresponding eigenvectors are denoted by $u_1^{*},\cdots ,u_r^{*}\in {ℝ}^n$. In other words, we have spectral decomposition $A^{*}={\sum }_{j=1}^r{\lambda }_j^{*}{u}_j^{*}{\left(u_j^{*}\right)}^T$.

We fix k ∈ [r] and study the k-th eigenvector u_k. Define the eigen-gap (or spectral gap) as ${\mathrm{\Delta }}^{*}=\min \left\{{\lambda }_{k-1}^{*}-{\lambda }_k^{*},{\lambda }_k^{*}-{\lambda }_{k+1}^{*}\right\}$, where we adopt the convention ${\lambda }_0^{*}=+\infty$ and ${\lambda }_{n+1}^{*}=-\infty$. Assume that

• (b)A concentrates under the spectral norm, i.e., there is a suitable γ = γ_n = o(1) such that ${\Vert A-A^{*}\Vert }_2\le \gamma {\mathrm{\Delta }}^{*}$ holds with probability 1 − o(1).

A direct yet important implication is that, the fluctuation of λ_k is much smaller than the gap Δ*, since Weyl’s inequality forces $\left|{\lambda }_k-{\lambda }_k^{*}\right|\le {\Vert A-A^{*}\Vert }_2$. Thus, λ_k is well separated from other eigenvalues, including the ‘bulk’ n−r eigenvalues whose magnitudes are at most ‖E‖₂.

In addition, we assume that A concentrates in a row-wise sense:

• (c)there exists a continuous non-decreasing function $\phi :{ℝ}_{+}\to {ℝ}_{+}$ that possibly depends on n, such that φ(0) = 0, φ(x)/x is non-increasing, and that for any m ∈ [n], $w\in {ℝ}^n$, with probability 1 − o(n⁻¹),

$$\left|{\left(A-A^{*}\right)}_{m\cdot }w\right|\le {\mathrm{\Delta }}^{*}\Vert w{\Vert }_{\infty }\phi \left(\frac{\Vert w{\Vert }_2}{\sqrt{n}\Vert w{\Vert }_{\infty }}\right).$$

Here, the notation (A − A*)_m. means the m-th row vector of A − A*.

For the Gaussian case where $A_{ij}\sim N\left(A_{ij}^{*},{\sigma }^2\right)$, we can simply choose a linear function $\phi (x)=c{\left({\mathrm{\Delta }}^{*}\right)}^{-1}\sigma \sqrt{n\log n}x$ where c > 0 is some proper constant. The condition then reads

$$\mathbb{P}\left(\left|{\left(A-A^{*}\right)}_{m\cdot }w\right|\le c\sigma \sqrt{\log n}\Vert w{\Vert }_2\right)=1-o\left(n^{-1}\right),$$

which directly follows from Gaussian tail bound since ${\left(A-A^{*}\right)}_{m\cdot }w\sim N\left(0,{\sigma }^2\Vert w{\Vert }_2^2\right)$. The tail of (A − A*)_m.ω is completely determined by ||w||₂. For Bernoulli variables, we will use Bernstein-type inequalities to study (A − A*)_m.ω, which will inevitably involve both ||w||₂ and ||w||_∞. Hence the function φ(x) can no longer be linear. It turns out that φ(x) ∝ (1∨log(1/x))⁻¹, shown in Figure 2, is a suitable choice. More details can be found in Section 2.1 and the supplementary material Abbe et al. (2018). In both cases we have φ(1) = O(1) under suitable signal-to-noise conditions.

Fig 2:

Typical choices of φ for Gaussian noise and Bernoulli noise.

Theorem 1.1

(Simpler form of Theorem 2.1). Let k ∈ [r] = {1, 2, ⋯, r} be fixed. Suppose that Assumptions (a), (b) and (c) hold, and ${\Vert u_k^{*}\Vert }_{\infty }\le \gamma$. Then, with probability 1 − o(1),

$$\underset{s\in \{\pm 1\}}{\min }{\Vert u_k-sA{u}_k^{*}/{\lambda }_k^{*}\Vert }_{\infty }=O\left((\gamma +\phi (\gamma )){\Vert u_k^{*}\Vert }_{\infty }\right)=o\left({\Vert u_k^{*}\Vert }_{\infty }\right),$$ (1.3)

where the notations O(·) and o(·) hide dependencies on φ(1).

On the left-hand side, we are allowed to choose a suitable sign s as eigenvectors are not uniquely defined. The second bound is a consequence of the first one, since γ = o(1) and lim_γ→0 φ(γ) = 0 by continuity. We hide dependency on φ(1) in the above bound, since φ(1) is bounded by a constant under suitable signal-to-noise ratio. More details can be found in Theorem 2.1. Therefore, the approximation error ${\Vert u_k-A{u}_k^{*}/{\lambda }_k^{*}\Vert }_{\infty }$ is much smaller than ${\Vert u_k^{*}\Vert }_{\infty }$. This rigorously confirms the intuitions in Section 1.1.

Here are some remarks. (1) This theorem enables us to study u_k via its linearization $A{u}_k^{*}/{\lambda }_k^{*}$, since the approximation error is usually small order-wise. (2) The conditions of the theorem are fairly mild. For SBM, the theorem is applicable as long as we are in the $\frac{\log n}{n}$ regime ($p=a\frac{\log n}{n}$ and $q=b\frac{\log n}{n}$), regardless of the relative sizes of a and b.

1.3. MLE, spectral algorithm, and strong consistency

Once we obtain the approximation result (1.3), the analysis of entrywise behavior of eigenvectors boils down to that of $A{u}_k^{*}/{\lambda }_k^{*}$. In the SBM example, suppose we have (1.2) and with probability 1−o(1), $\mathrm{sgn}\left(A{u}_2^{*}/{\lambda }_2^{*}\right)=\mathrm{sgn}\left(u_2^{*}\right)$ and all the entries of $A{u}_2^{*}/{\lambda }_2^{*}$ are bounded away from zero by an order of $1/\sqrt{n}$. Then $\mathrm{sgn}\left(u_2\right)=\mathrm{sgn}\left(A{u}_2^{*}/{\lambda }_2^{*}\right)$ holds with probability 1−o(1). Here sgn(·) denotes the entrywise sign function. The eigenvector-based estimator sgn(u₂) for block membership can be conveniently analyzed through $A{u}_2^{*}/{\lambda }_2^{*}$, whose entries are just linear combinations of Bernoulli variables.

We remark on a subtlety of our result: our central analysis is a good control of ${\Vert u_k-A{u}_k^{*}/{\lambda }_k^{*}\Vert }_{\infty }$, not necessarily of ${\Vert u_k-u_k^{*}\Vert }_{\infty }$. For example, in SBM, an inequality such as ${\Vert u_2-u_2^{*}\Vert }_{\infty }<{\Vert u_2^{*}\Vert }_{\infty }$ is not true in general. In Figure 1, the second boxplot shows that $\sqrt{n}{\Vert u_2-u_2^{*}\Vert }_{\infty }$ may well exceed 1 even if $\mathrm{sgn}\left(u_2\right)=\mathrm{sgn}\left(u_2^{*}\right)$. This suggests that the distributions of the coordinates of the two clusters, though well separated, have asymmetric tails. Our Theorem 3.3 asserts that it is in vain to seek a good bound for ${\Vert u_2-u_2^{*}\Vert }_{\infty }$. Instead, one should resort to the central quantity $A{u}_2^{*}/{\lambda }_2^{*}$. This may partly explain why the conjecture has remained open for long.

The vector $A{u}_k^{*}/{\lambda }_k^{*}$ also plays a pivotal role in the information-theoretic lower bound for exact recovery in SBM, established in Abbe et al. (2016). It is necessary to ensure ${\left(A{u}_2^{*}/{\lambda }_2^{*}\right)}_i>0,\forall i\in J$ to hold with probability at least 1/3. Otherwise, by symmetry and the union bound, with probability at least 1/3 we can find some i ∈ J and i’ ∈ J^c with ${\left(A{u}_2^{*}/{\lambda }_2^{*}\right)}_i<0$ and ${\left(A{u}_2^{*}/{\lambda }_2^{*}\right)}_{i^{\prime }}>0$. Elementary calculation shows that in that case, a swap of group assignments of i and i’ increases the likelihood. Thus the MLE ${\widehat{x}}_{\text{MLE}}$ fails to exactly recover J. With a uniform prior on group assignments, the MLE is equivalent to the maximum a posteriori estimator, which is optimal for exact recovery. Therefore, we must eliminate such local refinements to make exact recovery possible. This forms the core argument in Abbe et al. (2016). The analysis above suggests an interesting property about the eigenvector-based estimator ${\widehat{x}}_{eig}(A):=\mathrm{sgn}\left(u_2\right)$:

Corollary 1.1

Suppose we are given a > b > 0 such that $\sqrt{a}\ne \sqrt{b}+\sqrt{2}$, i.e., we exclude the regime where (a, b) is at the boundary of the phase transition. Then, whenever the MLE is successful, in the sense that ${\widehat{x}}_{MLE}=x$ (up to sign) with probability 1 − o(1), we have

$${\widehat{x}}_{eig}(A)={\widehat{x}}_{MLE}(A)=x$$

with probability 1 − o(1). Here x is the signed indicator of true communities.

This is because the success of ${\widehat{x}}_{MLE}$ hinges on $\mathrm{sgn}\left(A{u}_2^{*}/{\lambda }_2^{*}\right)=\mathrm{sgn}\left(u_2^{*}\right)$, which also guarantees ${\widehat{x}}_{eig}$ to work. See Section 3.2 for details. Such phenomenon appears in two applications considered in this paper.

1.4. An iterative perspective: power iterations

In the SBM, a key observation is that ${\Vert u_2-A{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty }$ is small. Here we give some intuitions from an iterative (or algorithmic) perspective. For simplicity, we will focus on the top eigenvector $\overline{u}$ of the centered adjacency matrix $\overline{A}=A-\frac{\widehat{d}}{n}{\mathbf{1}}_n{\mathbf{1}}_n^T$.

It is well known that the top eigenvector of a symmetric matrix can be computed via the power method. For almost any possible initialization u⁰, the iterations $u^{t+1}=\overline{A}{u}^t/{\Vert \overline{A}{u}^t\Vert }_2$ converge to $\overline{u}$. Suppose we set $u^0=u_2^{*}$, the top eigenvector of $\mathbb{E}\overline{A}$. Although this is not a real algorithm due to the initialization, it helps us gain theoretical insights.

The first, iterate after initialization is $u^1=\overline{A}{u}_2^{*}/{\Vert \overline{A}{u}_2^{*}\Vert }_2$. Standard concentration inequalities show that ${\Vert \overline{A}{u}_2^{*}\Vert }_2\approx {\overline{\lambda }}^{*}$, the top eigenvalue of $\mathbb{E}\overline{A}$ Therefore, u¹ is approximately $\overline{A}{u}_2^{*}/{\overline{\lambda }}^{*}$, which coincides with our first-order approximation. If u^t converges to $\overline{u}$ sufficiently fast, u¹ can already be good enough. This is similar to the rationale of one-step estimator (Bickel, 1975): a single, carefully designed iterate may improve the precision of a good initialization to the desired level. Figure 3 helps illustrate this idea.

Fig 3:

Error decay in power iterations. The larger and smaller squares represent ℓ_∞ balls centered at $\overline{u}$, with radii ${\Vert u^0-\overline{u}\Vert }_{\infty }$ and ${\Vert u^1-\overline{u}\Vert }_{\infty }$, respectively.

The iterative perspective has been explored in recent works (Zhong, 2017; Zhong and Boumal, 2018), where the latter studied both the eigenvector estimator and the MLE of a nonconvex problem. We are not going to show any proof with iterations or induction. Instead, we resort to the Davis-Kahan sin Θ theorem, combined with a “leave-one-out” technique. Nevertheless, we believe the iterative perspective is helpful to many other (nonconvex) problems where a counterpart of Davis-Kahan theorem is absent.

1.5. Related works

The study of eigenvector perturbation dates back to Rayleigh (Rayleigh, 1896) and Schrödinger (Schrödinger, 1926), in which asymptotic expansions were obtained. Later, Davis and Kahan (1970) developed elegant nonasymptotic perturbation bounds for eigenspaces gauged by unitary-invariant norms. These were extended to general rectangular matrices in Wedin (1972). See Stewart and Sun (1990) for a comprehensive investigation. Recently, O’Rourke et al. (2018) showed significant improvements of classical, deterministic bounds when the perturbation is random. Norms that depend on the choice of basis, such as the ℓ_∞ norm, are not addressed in these works but are of great interest in statistics.

There are several recent papers related to the study of entrywise perturbation. Fan et al. (2016) obtained ℓ_∞ eigenvector perturbation bounds. Their results were improved by Cape et al. (2017), in which the authors focused on 2 → ∞ norm bounds for eigenspaces. Eldridge et al. (2017) developed an ℓ_∞ perturbation bound by expanding the eigenvector perturbation into infinite series. These results are deterministic by nature, and thus yield suboptimal bounds under challenging stochastic regimes with small signal-to-noise ratio. By taking advantage of randomness, Koltchinskii et al. (2016) and Koltchinskii and Xia (2016) studied bilinear forms of singular vectors, leading to a sharp bound on ℓ_∞ error that was later extended to tensors (Xia and Zhou, 2017). Zhong (2017) characterized entrywise behaviors of eigenvectors and explored their connections with Rayleigh-Schrödinger perturbation theory. Zhong and Boumal (2018) worked on a related but slighted more complicated problem named “phase synchronization”, and analyzed entrywise behaviors of both the spectral estimator and MLE under a near-optimal regime. Chen et al. (2017) used similar ideas to derive the optimality of both the spectral estimator and MLE in top-K ranking problem.

There is a rich literature on the three applications in this paper. The synchronization problems (Singer, 2011; Cucuringu et al., 2012) aim at estimating unknown signals (usually group elements) from their noisy pairwise measurements, and have attracted much attention in optimization and statistics community recently (Bandeira et al., 2016; Javanmard et al., 2016). They are very relevant models for cryo-EM, robotics (Singer, 2011; Rosen et al., 2016) and more.

The stochastic block model has been studied extensively in the past decades, with renewed activity in the recent years (Coja-Oghlan, 2006; Decelle et al., 2011; Massoulié, 2014; Mossel et al., 2013; Krzakala et al., 2013; Abbe et al., 2016; Guédon and Vershynin, 2016; Amini and Levina, 2014; Abbe and Sandon, 2015; Montanari and Sen, 2016; Bordenave et al., 2015; Abbe and Sandon, 2017; Banks et al., 2016), see Abbe (2017) for further references, and in particular McSherry (2001), Vu (2014), Yun and Proutiere (2014), Lelarge et al. (2015), Chin et al. (2015) and Yun and Proutiere (2016), which are closest to this paper in terms of regimes and algorithms. The matrix completion problems (Candès and Recht, 2009; Candès and Plan, 2010; Keshavan et al., 2010b) have seen great impacts in many areas, and new insights and ideas keep flourishing in recent works (Ge et al., 2016; Sun and Luo, 2016). These lists are only a small fraction of the literature and are far from complete.

We organize our paper as follows: we present our main theorems of eigenvector and eigenspace perturbation in Section 2, which are rigorous statements of the intuitions introduced in Section 1. In Section 3, we apply the theorems to three problems: ${\mathbb{Z}}_2$-synchronization, SBM, and matrix completion from noisy entries. In Section 4, we present simulation results to verify our theories. Finally, we conclude and discuss future works in Section 5.

1.6. Notations

We use the notation [n] to refer to {1, 2, ⋯, n} for $n\in {\mathbb{Z}}_{+}$, and let ${ℝ}_{+}=[0,+\infty )$. For any real numbers $a,b\in ℝ$, we denote a ∨ b = max{a, b} and a ∧ b = min{a, b}. For nonnegative a_n and b_n that depend on n (e.g., problem size), we write a_n ≲ b_n to mean a_n ≤ Cb_n for some constant C > 0. The notation ≍ is similar, hiding two constants in upper and lower bounds. For any vector $x\in {ℝ}^n$, we define $\Vert x{\Vert }_2=\sqrt{{\sum }_{i=1}^n{x}_i^2}$ and ||x||_∞ = max_i |x_i|. For any matrix $M\in {ℝ}^{n\times d}$, M_i· refers to its i-th row, which is a row vector, and M_·i refers to its i-th column, which is a column vector. The matrix spectral norm is $\Vert M{\Vert }_2={\max }_{\Vert x{\Vert }_2=1}\Vert Mx{\Vert }_2$, the matrix max-norm is $\Vert M{\Vert }_{\max }={\max }_{i,j}\left|M_{ij}\right|$, and the matrix 2 → ∞ norm is $\Vert M{\Vert }_{2\to \infty }={\max }_{\Vert x{\Vert }_2=1}\Vert Mx{\Vert }_{\infty }={\max }_i{\Vert M_{i\cdot }\Vert }_2$. The set of n × r matrices with orthonormal columns is denoted by ${\mathcal{O}}_{n\times r}$.

2. Main results

2.1. Random matrix ensembles

Suppose $A\in {ℝ}^{n\times n}$ is a symmetric random matrix and $A^{*}=\mathbb{E}A$. Denote the eigenvalues of A by λ₁ ≥ ⋯ ≥ λ_n, and their associated eigenvectors by ${\left\{u_j\right\}}_{j=1}^n$. Analogously for A*, the eigenvalues and eigenvectors are ${\lambda }_1^{*}\ge \cdots \ge {\lambda }_n^{*}$ and ${\left\{u_j^{*}\right\}}_{j=1}^n$, respectively. We also adopt the convention ${\lambda }_0={\lambda }_0^{*}=+\infty$ and ${\lambda }_{n+1}={\lambda }_{n+1}^{*}=-\infty$. We allow some eigenvalues to be identical. Thus, some eigenvectors may be defined up to rotations.

Suppose r and s are two integers satisfying 1 ≤ r ≤ n and 0 ≤ s ≤ n − r. Let $U=\left(u_{s+1},\cdots ,u_{s+r}\right)\in {ℝ}^{n\times r}$, $U^{*}=\left(u_{s+1}^{*},\cdots ,u_{s+r}^{*}\right)\in {ℝ}^{n\times r}$ and ${\mathrm{\Lambda }}^{*}=\text{diag}\left({\lambda }_{s+1}^{*},\cdots ,{\lambda }_{s+r}^{*}\right)\in {ℝ}^{r\times r}$. We are interested in the eigenspace span(U). To this end, we assume there is an eigen-gap Δ* seperating ${\left\{{\lambda }_{s+j}^{*}\right\}}_{j=1}^r$ from 0 and other eigenvalues (see Figure 4), i.e.,

$${\mathrm{\Delta }}^{*}=\left({\lambda }_s^{*}-{\lambda }_{s+1}^{*}\right)\wedge \left({\lambda }_{s+r}^{*}-{\lambda }_{s+r+1}^{*}\right)\wedge \underset{i\in [r]}{\min }\left|{\lambda }_{s+i}^{*}\right|.$$ (2.1)

Compared with the usual eigen-gap (Davis and Kahan, 1970), our definition also takes the distances between eigenvalues and 0 into consideration. When A* is rank-deficient, 0 is itself an eigenvalue.

Fig 4:

Eigen-gap Δ*

We define $\kappa :={\max }_{i\in [r]}\left|{\lambda }_{s+i}^{*}\right|/{\mathrm{\Delta }}^{*}$, which is always bounded from below by 1. In our applications, κ is usually bounded from above by a constant, i.e., Δ* is comparable to ${\left\{{\lambda }_{s+j}^{*}\right\}}_{j=1}^r$ in terms of magnitude.

The concentration property is characterized by a parameter γ ≥ 0, and a function $\phi (x):{ℝ}_{+}\to {ℝ}_{+}$. Roughly speaking, γ⁻¹ resembles the signal-to-noise ratio, and γ typically vanishes as n tends to infinity. φ(x) is chosen according to the distribution of A, and is typically bounded by a constant for x ∈ [0,1]. In particular, we take φ(x) ∝ x for Gaussian matrices and φ(x) ∝ (1 ∨ log(1/x))⁻¹ for Bernoulli matrices —see Figure 2. In addition, we will also make a mild structural assumption: ||A*||_2⟶∞ ≤ γΔ*. In many applications involving low-rank structure, the eigenvalues of interest (and thus Δ*) typically scale with n, whereas ||A*||_2⟶∞ scales with $\sqrt{n}$.

Based on the quantities above, we make the following assumptions.

• A1 (Incoherence) ||A*||_2⟶∞ ≤ γΔ*.

• A2 (Row- and column-wise independence) For any m ∈ [n], the entries in the mth row and column of A are independent with others, i.e. {A_ij : i = m or j = m} are independent of {A_ij : i ≠ m, j ≠ m}.

• A3 (Spectral norm concentration) 32κ max{γ, φ(γ)} ≤ 1 and for some δ₀ ∈ (0, 1),

  $$\mathbb{P}\left({\Vert A-A^{*}\Vert }_2\le \gamma {\mathrm{\Delta }}^{*}\right)\ge 1-{\delta }_0.$$ (2.2)
• A4 (Row concentration) Suppose φ(x) is continuous and non-decreasing in ${ℝ}_{+}$ with φ(0) = 0, φ(x)/x is non-increasing in ${ℝ}_{+}$, and δ₁ ∈ (0, 1). For any m ∈ [n] and $W\in {ℝ}^{n\times r}$,

  $$\mathbb{P}\left({\Vert {\left(A-A^{*}\right)}_{m\cdot }W\Vert }_2\le {\mathrm{\Delta }}^{*}\Vert W{\Vert }_{2\to \infty }\phi \left(\frac{\Vert W{\Vert }_F}{\sqrt{n}\Vert W{\Vert }_{2\to \infty }}\right)\right)\ge 1-\frac{{\delta }_1}{n}.$$ (2.3)

Here are some remarks and intuitions. Assumption 1 requires that no row of A* is dominant. To relate it to the usual concept of incoherence (Candès and Recht, 2009; Candès et al., 2011), we consider the case A* = U*∧*(U*)^T and let $\mu \left(U^{*}\right)=\frac{n}{r}{\max }_{i\in [n]}{\sum }_k{\left(U_{ik}^{*}\right)}^2=\frac{n}{r}{\Vert U^{*}\Vert }_{2\to \infty }^2$. Note that

$${\Vert U^{*}{\mathrm{\Lambda }}^{*}{\left(U^{*}\right)}^T\Vert }_{2\to \infty }\le {\Vert U^{*}\Vert }_{2\to \infty }{\Vert {\mathrm{\Lambda }}^{*}{\left(U^{*}\right)}^T\Vert }_2={\Vert U^{*}\Vert }_{2\to \infty }{\Vert {\mathrm{\Lambda }}^{*}\Vert }_2$$ (2.4)

and $\kappa ={\Vert {\mathrm{\Lambda }}^{*}\Vert }_2/{\mathrm{\Delta }}^{*}$. Then Assumption 1 is satisfied as long as $\mu \left(U^{*}\right)\le \frac{n{\gamma }^2}{r{\kappa }^2}$, which is very mild.

Assumption 2 is a mild independence assumption, and it encompasses common i.i.d. noise assumptions.

Assumption 3 requires the spectral norm of the noise matrix A − A* to be dominated by Δ*, which can be interpreted as signal strength. In our example of ${\mathbb{Z}}_2$ synchronization (see Section 3.1), we have Δ* = n, and A − A* have i.i.d. N(0, σ²) entries above the diagonal. Since ${\Vert A-A^{*}\Vert }_2\lesssim \sigma \sqrt{n}$ by standard concentration results, we need to require $\sigma =O(\gamma \sqrt{n})$.

Assumption 4 is a generalization of the row concentration assumption in Section 1.2, and the function φ is problem-dependent. Here we explain the role of φ using a special case where r = 1 and A ∈ {0, 1}^n×n has i.i.d. Bernoulli entries with parameter p = p_n on and above its diagonal. Then Δ* = np, ${\sum }_{i=1}^n{A}_{mi}^{*}=np$. If p is not too small, with high probability we have ${\sum }_{i=1}^n{A}_{mi}\lesssim np$ and thus

$$\left|{\left(A-A^{*}\right)}_{m\cdot }W\right|\le \Vert W{\Vert }_{\infty }\sum _{i=1}^n\left|{\left(A-A^{*}\right)}_{mi}\right|\lesssim \Vert W{\Vert }_{\infty }np={\mathrm{\Delta }}^{*}\Vert W{\Vert }_{\infty }.$$

If many entries in W have magnitudes much less than ||W||_∞, there should be less fluctuation and better concentration. Indeed, Assumption 4 stipulates a tighter bound by a factor of $\phi \left(\frac{\Vert W{\Vert }_2}{\sqrt{n}\Vert W{\Vert }_{\infty }}\right)$, where $\frac{\Vert W{\Vert }_2}{\sqrt{n}\Vert W{\Vert }_{\infty }}$ is typically much smaller than 1 in this case. This delicate concentration bound turns out to be crucial in the analysis of SBM, where A is a sparse binary matrix.

2.2. Entrywise perturbation of general eigenspaces

In this section, we generalize Theorem 1.1 from individual eigenvectors to eigenspaces under milder conditions that are characterized by additional parameters. Note that neither U nor U* is uniquely defined, and they can only be determined up to a rotation if the eigenvalues are identical. For this reason, our result has to involve an r × r orthogonal matrix. Beyond asserting our result holds up to a suitable rotation, we give an explicit form of such orthogonal matrix.

Let $H=U^T{U}^{*}\in {ℝ}^{r\times r}$, and its singular value decomposition be $\text{H}=\overline{U}\overline{\mathrm{\Sigma }}{\overline{V}}^T$, where $\overline{U},\overline{V}\in {ℝ}^{r\times r}$ are orthonormal matrices, and $\overline{\mathrm{\Sigma }}\in {ℝ}^{r\times r}$ is a diagonal matrix. Define an orthonormal matrix $\mathrm{sgn}(H)\in {ℝ}^{r\times r}$ as

$$\mathrm{sgn}(H):=\overline{U}{\overline{V}}^T.$$ (2.5)

This orthogonal matrix is called the matrix sign function (Gross, 2011). Now we are able to extend the results in Section 1.2 to general eigenspaces.

Theorem 2.1

Under Assumptions A1–A4, with probability at least 1 – δ₀ – 2δ₁ we have

$$\begin{gathered} \Vert U{\Vert }_{2\to \infty }\lesssim (\kappa +\phi (1)){\Vert U^{*}\Vert }_{2\to \infty }+\gamma {\Vert A^{*}\Vert }_{2\to \infty }/{\mathrm{\Delta }}^{*}, \\ {\Vert U\mathrm{sgn}(H)-A{U}^{*}{\left({\mathrm{\Lambda }}^{*}\right)}^{-1}\Vert }_{2\to \infty }\lesssim \kappa (\kappa +\phi (1))(\gamma +\phi (\gamma )){\Vert U^{*}\Vert }_{2\to \infty }+\gamma {\Vert A^{*}\Vert }_{2\to \infty }/{\mathrm{\Delta }}^{*}, \\ {\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_{2\to \infty }\le {\Vert U\mathrm{sgn}(H)-A{U}^{*}{\left({\mathrm{\Lambda }}^{*}\right)}^{-1}\Vert }_{2\to \infty }+\phi (1){\Vert U^{*}\Vert }_{2\to \infty }. \end{gathered}$$

Here the notation ≲ only hides absolute constants.

The third inequality is derived by simply writing Usgn(H) − U* as a sum of the first-order error EU*(∧*)⁻¹ and higher-order error Usgn(H) − AU*(∧*)⁻¹, and bounding EU*(∧*)⁻¹ by the row concentration Assumption A4. It will be useful for the noisy matrix completion problem. It is worth pointing out that Theorem 2.1 is applicable to any eigenvector of A that is not necessarily the leading one. This is particularly powerful in SBM (Section 3.2) where we need to analyze the second eigenvector. In addition, we do not need A* to have low rank, although the examples to be presented have such structure. For low-rank A*, estimation errors of all the eigenvectors can be well controlled by the following corollary of Theorem 2.1.

Corollary 2.1

Let Assumptions A1–A4 hold, and suppose that A* = U*∧*(U*)^T. With probability at least 1 – δ₀ – 2δ₁, we have

$$\begin{gathered} \Vert U{\Vert }_{2\to \infty }\lesssim (\kappa +\phi (1)){\Vert U^{*}\Vert }_{2\to \infty }, \\ {\Vert U\mathrm{sgn}(H)-A{U}^{*}{\left({\mathrm{\Lambda }}^{*}\right)}^{-1}\Vert }_{2\to \infty }\lesssim \kappa (\kappa +\phi (1))(\gamma +\phi (\gamma )){\Vert U^{*}\Vert }_{2\to \infty }, \\ {\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_{2\to \infty }\le {\Vert U\mathrm{sgn}(H)-A{U}^{*}{\left({\mathrm{\Lambda }}^{*}\right)}^{-1}\Vert }_{2\to \infty }+\phi (1){\Vert U^{*}\Vert }_{2\to \infty }. \end{gathered}$$

Here the notation ≲ only hides absolute constants.

Corollary 2.1 directly follows from Theorem 2.1, inequality (2.4) and the fact that κ ≥ 1. Below we use a simple example to illustrate the results above. Let A* = λ*u*(u*)^T be a rank-one matrix with λ* > 0 and ||u*||₂ =1. Set r = 1 and s = 0. This structure implies Δ* = λ* and κ = 1. Suppose A has independent entries on and above the diagonal. Such A is usually called a spiked Wigner matrix in statistics and random matrix theory.

Let Assumptions A1-A4 hold. The first two inequalities in Corollary 2.1 are simplified as

$$\Vert u{\Vert }_{\infty }\lesssim (1+\phi (1)){\Vert u^{*}\Vert }_{\infty },$$ (2.6)

$${\Vert u-A{u}^{*}/{\lambda }^{*}\Vert }_{\infty }\lesssim (\gamma +\phi (\gamma ))(1+\phi (1)){\Vert u^{*}\Vert }_{\infty }.$$ (2.7)

In many applications, φ(1) ≲ 1 and γ = o(1) as n goes to infinity. Then (2.6) controls the magnitude of the empirical eigenvector u by that of the true eigenvector u* in the ℓ^∞ sense. Furthermore, (2.7) has the same form as the main result in Theorem 1.1, stating that Au*/λ* is an ℓ_∞ approximation of u with error much smaller than ||u*||_∞. Therefore, it is possible to study u via its linearization Au*/λ*, which usually makes analysis much easier.

The regularity conditions in Theorem 1.1 imply our Assumptions A1-A4. In particular, the condition ||u*||_∞ ≤ γ there is equivalent to Assumption A1. As a result, Theorem 1.1 with r = 1 is a special case of Corollary 2.1 and hence of Theorem 2.1. It is not hard to generalize to r = O(1).

3. Applications

3.1. ${\mathbb{Z}}_2$-synchronization and spiked Wigner model

The problem of ${\mathbb{Z}}_2$-synchronization is to recover n unknown labels ±1 from noisy pairwise measurements. This is a prototype of more general SO(d)-synchronization problems including phase synchronization and SO(3)-synchronization, in which one wishes to estimate the phases of signals or rotations of cam- eras/molecules, etc. Such problems arise in time synchronization of distributed networks (Giridhar and Kumar, 2006), calibration of cameras (Tron and Vidal, 2009), and cryo-EM (Shkolnisky and Singer, 2012).

Consider an unknown signal x ϵ {±1}ⁿ. Suppose we have independent measurements of the form Y_ij = x_ix_j + σW_ij, where i < j, W_ij ~ N (0, 1) and σ > 0. We can define W_ii = 0 and W_ij = W_ji for simplicity, and write our model into a matrix form as follows:

$$Y=x{x}^T+\sigma W,x\in {\{\pm 1\}}^n.$$ (3.1)

This is sometimes called the Gaussian ${\mathbb{Z}}_2$-synchronization problem, in contrast to the one with ${\mathbb{Z}}_2$-noise, also known as the censored block model (Abbe et al., 2014a). This problem can be further generalized: each entry x_j is a unit-modulus complex number $e^{i{\theta }_j}$, if the goal is to estimate unknown angles from pairwise measurements; or, each entry x_j is an orthogonal matrix from SO(3), if the goal is to estimate unknown orientations of molecules, cameras, etc. Here we focus on the simplest case x_j ∈ {±1}.

Note that in (3.1), both Y and W are symmetric matrices in ${ℝ}^{n\times n}$, and the data matrix Y has a noisy rank-one decomposition. This falls into the spiked Wigner model. The quality of an estimator $\widehat{x}$ is usually gauged either by its correlation with x, or by the proportion of labels x_i it correctly recovers. It has been shown that the information-theoretic threshold for a nontrivial correlation is $\sigma =\sqrt{n}$ (Javanmard et al., 2016; Deshpande et al., 2015; Lelarge and Miolane, 2016; Perry et al., 2016), and the threshold for exact recovery (i.e., $\widehat{x}=\pm x$ with probability tending to 1) is $\sigma =\sqrt{\frac{n}{2\log n}}$ (Bandeira et al., 2016).

When $\sigma \le \sqrt{\frac{n}{(2+\epsilon )\log n}}$ (ε > 0 is any constant), it was proved by Bandeira et al. (2016) that semidefinite programming (SDP) finds the maximum likelihood estimator and achieves exact recovery. We are going to show that a very simple method, both conceptually and computationally, also achieves exact recovery. This method is outlined as follows:

1. Compute the leading eigenvector of Y, denoted by u;

2. Take the estimate $\widehat{x}=\mathrm{sgn}(u)$.

Our next theorem asserts that the eigenvector-based method above succeeds in finding x consistently under $\sigma \le \sqrt{\frac{n}{(2+\epsilon )\log n}}$. Thus, under any regime where the MLE achieves exact recovery, our eigenvector estimator $\widehat{x}$ equals the MLE with high probability. This phenomenon also holds for the stochastic block model.

Theorem 3.1

Suppose $\sigma \le \sqrt{\frac{n}{(2+\epsilon )\log n}}$ for some ε > 0. With probability 1 − o(1), the leading eigenvector of Y with unit ℓ₂ norm satisfies

$$\sqrt{n}\underset{i\in [n]}{\min }\left\{s{x}_i{u}_i\right\}\ge 1-\sqrt{\frac{2}{2+\epsilon }}+\frac{C}{\sqrt{\log n}},$$

for a suitable s ϵ {±1}, where C > 0 is an absolute constant. As a consequence, our eigenvector-based method achieves exact recovery.

Note that our approach does not utilize the structural constraints |x_i| = 1, ∀ i ∈ [n]; whereas such constraints appear in the SDP formulation (Bandeira et al., 2016). A natural question is an analysis of both methods with an increased noise level σ. A seminal work by Javanmard et al. (2016) complements our story: the authors showed via non-rigorous statistical mechanics arguments that when σ is on the order of $\sqrt{n}$, the SDP-based approach outperforms the eigenvector approach. Nevertheless, with a slightly larger signal strength, there is no such advantage of the SDP approach.

When $\sigma \asymp \sqrt{n}$, general results for spiked Wigner models (Baik et al., 2005; Féral and Péché, 2007; Benaych-Georges and Nadakuditi, 2011) imply that $\frac{1}{n}{\left|u^{T}x\right|}^2\to 1-{\sigma }^2/n$ for $\sigma /\sqrt{n}<1-\epsilon$ with any small constant ε> 0. Deshpande et al. (2015) proved that non-trivial correlation with x cannot be obtained by any estimator if $\sigma /\sqrt{n}>1+\epsilon$.

3.2. Stochastic Block Model

As is briefly discussed in Section 1, we focus on the symmetric SBM with two equally-sized groups. (Though the second eigenvector of A* depends on relative sizes of the groups, our analysis only requires slight modification if groups have different sizes.) For simplicity, we allow for self-loops (i.e. edges from vertices to themselves) in the random graph, and it makes no much difference if they are excluded. In that case, the expectation of the adjacency matrix changes by a negligible quantity O(log n/n) under the spectral norm and moreover, Assumptions A1–A4 still hold with the same parameters.

Definition 3.1

Let n be even, 0 ≤ q ≤ p ≤ 1, and J ⊆ [n] with |J| = n/2. SBM(n, p, q, J) is the ensemble of n × n symmetric random matrices A = (A_ij)_i,j∈[n] where {A_ij}_{1≤i≤j≤n} are independent Bernoulli random variables, and

$$\mathbb{P}\left(A_{ij}=1\right)=\{\begin{array}{l}p,\text{if}i\in J,j\in J\mathit{\text{or}}i\in J^c,j\in J^c\hfill \\ q,otherwise\hfill \end{array}.$$ (3.2)

The community detection problem aims at finding the bi-partition (J, J^c) given only one realization of A. Let z_i = 1 if i ∈ J and z_i = −1 otherwise. We want to find an estimator $\widehat{z}$ for the unknown labels z ∈ {±1}ⁿ. Intuitively, the task is more difficult when p is close to q, and when the magnitudes of p, q are small. It is impossible, for instance, to produce any meaningful estimator when p = q. The task is also impossible when p and q are as small as o(n⁻²), since A is a zero matrix with high probability.

As is already discussed in Section 1, under the regime $p=\frac{a\log n}{n}$, $q=\frac{b\log n}{n}$ where a and b are constants independent of n, it is information theoretically impossible to achieve exact recovery (the estimate $\widehat{z}$ equals z or −z with probability tending to 1) when $\sqrt{a}-\sqrt{b}<\sqrt{2}$. In contrast, when $\sqrt{a}-\sqrt{b}>\sqrt{2}$, the goal is efficiently achievable. Further, it is known that SDP succeeds down to the threshold. Under the regime $p=\frac{a}{n}$, $q=\frac{b}{n}$, it is impossible to obtain nontrivial correlation (i.e. the correlation between $\widehat{z}$ and z is at least some positive constant ε, as a random guess gets roughly half the signs correct and almost zero correlation with z) between any estimator $\widehat{z}$ and z if (a − b)² < 2(a + b), and when (a − b)² > 2(a + b), nontrivial correlation can efficiently be obtained (Massoulié, 2014; Mossel et al., 2013).

Here we focus on the regime where $p=a\frac{\log n}{n}$, $q=b\frac{\log n}{n}$ and a > b > 0 are constants. Note that $\mathbb{E}A$, or equivalently A*, is a rank-2 matrix. Its nonzero eigenvalues are ${\lambda }_1^{*}=(p+q)n/2$ and ${\lambda }_2^{*}=(p-q)n/2$, whose associated eigenvectors are $u_1^{*}=\frac{1}{\sqrt{n}}{\mathbf{1}}_n$ and $u_2^{*}=\frac{1}{\sqrt{n}}{1}_J-\frac{1}{\sqrt{n}}{1}_{J^c}$. As $u_2^{*}$ is aligned with z and perfectly reveals the desired partition, the following vanilla spectral method is a natural candidate:

1. Compute u₂, the eigenvector of A corresponding to its second largest eigenvalue λ₂;

2. Set $\widehat{z}=\mathrm{sgn}\left(u_2\right)$.

It has been empirically observed and conjectured that as soon as the signal strength $\sqrt{a}-\sqrt{b}$ exceed the information threshold $\sqrt{2}$, the vanilla spectral method achieves exact recovery (Abbe et al., 2014b). Moreover, in regimes where exact recovery is impossible, Zhang and Zhou (2016) established the following minimax result. It has not been clear whether the vanilla spectral method achieves the minimax misclassification rate.

If we define the misclassification rate as

$$r(\widehat{z},z)=\underset{s\in \{\pm 1\}}{\min }{n}^{-1}\sum _{i=1}^n{\mathbf{1}}_{\left\{{\widehat{z}}_i\ne s{z}_i\right\}},$$ (3.3)

then the results of Zhang and Zhou (2016) imply that

$$\underset{\widehat{z}}{\inf }\sup \mathbb{E}r(\widehat{z},z)=\exp \left(-(1+o(1))\cdot {(\sqrt{a}-\sqrt{b})}^2\frac{\log n}{2}\right),$$ (3.4)

where the supremum is taken over approximately equal-sized SBM with 2-blocks. Note that this parameter space is slightly different from our Definition 3.1, but as explained before, we can modify our proofs accordingly such that the same conclusions still hold. See the supplementary materials (Abbe et al., 2018) for further explanation of (3.4).

Here we prove that the vanilla spectral method indeed succeeds in exact recovery whenever it is information-theoretic possible, which resolves the conjecture of (Abbe et al., 2014b); and if it is not, vanilla spectral method achieves the optimal misclassification rate.

Theorem 3.2

(i) If $\sqrt{a}-\sqrt{b}>\sqrt{2}$ then there exists η = η(a, b) > 0 and s ∈ {±1} such that with probability 1 − o(1),

$$\sqrt{n}\underset{i\in [n]}{\min }s{z}_i{\left(u_2\right)}_i\ge \eta .$$

As a consequence, our spectral method achieves exact recovery.

(ii) Let the misclassification rate $r(\widehat{z},z)$ be defined in (3.3). If $\sqrt{a}-\sqrt{b}\in (0,\sqrt{2}]$, then

$$\mathbb{E}r(\widehat{z},z)\le n^{-(1+o(1)){(\sqrt{a}-\sqrt{b})}^2/2}.$$

This upper bound matches the minimax lower bound.

The first part implies that, under the regime where the MLE achieves exact recovery, our eigenvector estimator is exactly the MLE with high probability. This proves Corollary 1.1 in the introduction. Moreover, the second part asserts that for more challenging regime where exact recovery is impossible, the eigenvector estimator has the optimal misclassification rate.

Before further explaining our results, we give a brief review of previous endeavors and an analysis of difficulties. Various papers have investigated this algorithm and its variants such as McSherry (2001), Coja-Oghlan (2006), Rohe et al. (2011), Sussman et al. (2012), Vu (2014), Lelarge et al. (2015), Yun and Proutiere (2014), Yun and Proutiere (2016), Lei and Rinaldo (2015), Gao et al. (2015), among others. However, it is not known if the simple algorithm above achieves exact recovery down to the information-theoretic threshold, nor the optimal misclassification rate studied in Zhang and Zhou (2016) while below the threshold. An important reason for the unsettlement of this question is that the entrywise behavior of u₂ is not fully understood. In particular, people have been focusing on the ℓ_∞ error ${\Vert u_2-u_2^{*}\Vert }_{\infty }$, which may well exceed ${\Vert u_2^{*}\Vert }_{\infty }$ (see Theorem 3.3), suggesting that the algorithm may potentially fail by rounding on the incorrect sign. This is not necessarily the case—as errors could have larger magnitudes on the ‘good side’ of the signal range—but ${\Vert u_2-u_2^{*}\Vert }_{\infty }$ cannot capture this. To avoid suboptimal theoretical results, multi-round algorithms are popular choices in the literature (Coja-Oghlan, 2006; Vu, 2014), which typically have a preprocessing step of trimming and/or a postprocessing step refining the initial solution. Yun and Proutiere (2014) and Yun and Proutiere (2016) showed that such variants can achieve the exact recovery threshold. We are going to prove that the vanilla spectral algorithm alone achieves the threshold and the minimax lower-bound in one shot.

The key to proving Theorem 3.2 is the following first-order approximation result for u₂ under the ℓ_∞ norm, which is a consequence of Theorem 2.1.

Corollary 3.1

If $A\sim \mathrm{SBM}\left(n,a\frac{\log n}{n},b\frac{\log n}{n},J\right)$, then with probability 1 − O(n⁻³) we have

$$\underset{s\in \{\pm 1\}}{\min }{\Vert u_2-sA{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty }\le \frac{C}{\sqrt{n}\log \log n}.$$ (3.5)

where C = C (a, b) is some constant depending only on a and b.

The above result holds for any constants a and b, and does not depend on the gap $\sqrt{a}-\sqrt{b}$. This fact will be useful for analyzing the misclassification rate. By Corollary 3.1, the ℓ_∞ approximation error is negligible, and thus the analysis of vanilla spectral algorithm boils down to analyzing the entries in $A{u}_2^{*}/{\lambda }_2^{*}$, which are just weighted sums of Bernoulli random variables.

As a by-product, we can show that entrywise analysis through ${\Vert u_2-u_2^{*}\Vert }_{\infty }$ is not a good strategy. As is mentioned earlier, our sharp result for the eigenvector estimator stems from careful analysis of the linearized version $A{u}_2^{*}/{\lambda }_2^{*}$ of u₂, and the approximation error ${\Vert u_2-A{u}_2^{*}/{\lambda }_2^{*}\Vert }_{\infty }$. This is superior to direct analysis of the ℓ_∞ perturbation ${\Vert u_2-u_2^{*}\Vert }_{\infty }$, as the next theorem implies that ${\Vert u_2-u_2^{*}\Vert }_{\infty }>{\Vert u_2^{*}\Vert }_{\infty }$ is possible even if $\mathrm{sgn}\left(u_2\right)=\mathrm{sgn}\left(u_2^{*}\right)$.

Theorem 3.3

(Asymptotic lower bound for eigenvector perturbation). Let J = [n/2] and $A\sim \mathrm{SBM}\left(n,a\frac{\log n}{n},b\frac{\log n}{n},J\right)$, where a > b > 0 are constants and n → ∞. For any fixed η > 1 with η log η − η + 1 < 2/a, with probability 1 − o(1) we have

$$\sqrt{n}{\Vert u_2-u_2^{*}\Vert }_{\infty }\ge \frac{a(\eta -1)}{a-b}.$$

Now let us consider the case in Figure 1, where a = 4.5 and b = 0.25. On the one hand, exact recovery is achievable since $\sqrt{a}-\sqrt{b}>1.62>\sqrt{2}$. On the other hand, by taking η = 2 we get h(η) = 2log2 − 2 + 1 < 4/9 = 2/a and $\frac{a(\eta -1)}{a-b}>1.05$. Theorem 3.3 implies

$$\underset{n\to \infty }{\lim }\mathbb{P}\left({\Vert u_2-u_2^{*}\Vert }_{\infty }>1.05/\sqrt{n}\right)=1.$$

In words, the size of fluctuation is consistently larger than the signal strength. As a result, by merely looking at ${\Vert u_2-u_2^{*}\Vert }_{\infty }$ we cannot expect sharp analysis of the spectral method in exact recovery.

Finally, we point out that it is not straightforward to develop a simple spectral method to achieve the information threshold for exact recovery in SBM with K > 2 blocks. Spectral methods in this scenario (Rohe et al., 2011; Lei and Rinaldo, 2015) typically start with r > 1 eigenvectors ${\left\{v_j\right\}}_{j=1}^r\subseteq {ℝ}^n$ of some data matrix (e.g. the adjacency matrix or Laplacian matrix). Then, the n rows of $V=\left(v_1,\cdots ,v_r\right)\in {ℝ}^{n\times r}$ are treated as embeddings of the n nodes into ${ℝ}^r$, from which one infers block memberships using clustering techniques. In our vanilla spectral method for 2 blocks, we only look at a single eigenvector and return the blocks based on signs of coordinates. This method always returns the same memberships (up to a global swap), even though the eigenvector is identifiable only up to a sign. When K > 2 and r > 1, due to possible multiplicity of eigenvalues, the embeddings of n nodes may be identifiable only up to an orthonormal transform in ${ℝ}^r$. Such ambiguity causes trouble for effective clustering, although we can still study the embedding using Theorem 2.1. Due to space constraints, we put a brief discussion in the supplementary material Abbe et al. (2018).

3.3. Matrix completion from noisy entries

Matrix completion based on partial observations has wide applications including collaborative filtering, system identification, global positioning, remote sensing, etc., see Candes and Plan (2010). A popular version is the “Netflix problem”, where one is given a incomplete table of customer ratings and wants to predict the missing entries. This could be useful for targeted recommendation in the future. Since it has been intensively studied in the past decade, our brief review below is by no means exhaustive. Candès and Recht (2009), Candès and Tao (2010), and Gross (2011) focused on exact recovery of low-rank matrices based on noiseless observations. More realistic models with noisy observations were studied in Candès and Plan (2010), Keshavan et al. (2010b), Koltchinskii et al. (2011), Jain et al. (2013) and Chatterjee (2015).

As an application of Theorem 2.1, we are going to study a model similar to the one in Chatterjee (2015) where both sampling scheme and noise are random. It can be viewed as a statistical problem with missing values. Suppose we have an unknown signal matrix $M^{*}\in {ℝ}^{n_1\times n_2}$. For each entry of M*, we have a noisy observation $M_{ij}^{*}+{\epsilon }_{ij}$ with probability p, and have no observation otherwise. Let $M^{obs}\in {ℝ}^{n_1\times n_2}$ record our observations, with missing entries treated as zeros. We consider the rescaled partial observation matrix M = M^obs/p for simplicity. It is easy to see that M is an unbiased estimator for M*, and hence a popular starting point for further analysis. The definition of our model is formalized below.

Definition 3.2

Let $M^{*}\in {ℝ}^{n_1\times n_2}$, p ∈ (0, 1) and σ ≥ 0. We define NMC(M*,p, σ) to be the ensemble of n₁ × n₂ random matrices $M={\left(M_{ij}\right)}_{i\in \left[n_1\right],j\in \left[n_2\right]}$ with $M_{ij}=\left(M_{ij}^{*}+{\epsilon }_{ij}\right)I_{ij}/p$, where ${\left\{I_{ij},{\epsilon }_{ij}\right\}}_{i\in \left[n_1\right],j\in \left[n_2\right]}$ are jointly independent, $\mathbb{P}\left(I_{ij}=1\right)=p=1-\mathbb{P}\left(I_{ij}=0\right)$and ${\epsilon }_{ij}\sim N\left(0,{\sigma }^2\right)$.

Let r = rank(M*) and M* = U*Σ*V* be its singular value decomposition (SVD), where $U^{*}\in {\mathcal{O}}_{n_1\times r}$, $V^{*}\in {\mathcal{O}}_{n_2\times r}$, ${\mathrm{\Sigma }}^{*}=\mathrm{diag}\left({\sigma }_1^{*},\cdots ,{\sigma }_r^{*}\right)$ is diagonal, and ${\sigma }_1^{*}\ge \cdots \ge {\sigma }_r^{*}$. We are interested in estimating U*, V*, and M*. The rank r is assumed to be known, which is usually easily estimated otherwise, see Keshavan and Oh (2009) for example. We work on a very simple spectral algorithm that often serves as an initial estimate of M* in iterative methods.

1. Compute the r largest singular values σ₁ ≥ ⋯ ≥ σ_r of M, and their associated left and right singular vectors ${\left\{u_j\right\}}_{j=1}^r$ and ${\left\{v_j\right\}}_{j=1}^r$. Define Σ = diag(σ₁, ⋯, σ_r), $U=\left(u_1,\cdots ,u_r\right)\in {\mathcal{O}}_{n_1\times r}$ and $V=\left(v_1,\cdots ,v_r\right)\in {\mathcal{O}}_{n_2\times r}$.

2. Return U, V and UΣV^T as estimators for U*, V*, and M*, respectively.

Note that the matrices in Definition 3.2 are asymmetric in general, due to the rectangular shape and independent sampling. Hence, Theorem 2.1 is not directly applicable. Nevertheless, it could be tailored to fit into our framework by a “symmetric dilation” trick. See the supplementary materials (Abbe et al., 2018) for details. Below we present our results.

Theorem 3.4

Let M ~ NMS(M*, p, σ), n = n₁ + n₂, $\kappa ={\sigma }_1^{*}/{\sigma }_r^{*}$, $H=\frac{1}{2}\left(U^T{U}^{*}+V^T{V}^{*}\right)$, and $\eta =\left({\Vert U^{*}\Vert }_{2\to \infty }\vee {\Vert V^{*}\Vert }_{2\to \infty }\right)$. There exist constants C and C’ such that the followings hold. Suppose $p\ge 6\frac{\log n}{n}$ and $\kappa \frac{n\left({\Vert M^{*}\Vert }_{\max }+\sigma \right)}{{\sigma }_r^{*}}\sqrt{\frac{\log n}{np}}\le 1/C$. With probability at least 1 − C/n, we have

$$\begin{gathered} \left(\Vert U{\Vert }_{2\to \infty }\vee \Vert V{\Vert }_{2\to \infty }\right)\le C^{\prime }\kappa \eta , \\ \left({\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_{2\to \infty }\vee {\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_{2\to \infty }\right)\le C^{\prime }\eta {\kappa }^2\frac{n\left({\Vert M^{*}\Vert }_{\max }+\sigma \right)}{{\sigma }_r^{*}}\sqrt{\frac{\log n}{np}}, \\ {\Vert U\mathrm{\Sigma }{V}^T-M^{*}\Vert }_{\max }\le C^{\prime }{\eta }^2{\kappa }^4\left({\Vert M^{*}\Vert }_{\max }+\sigma \right)\sqrt{\frac{n\log n}{p}}. \end{gathered}$$

To our best knowledge, the results for singular vectors are the first of this type for the spectral algorithm. Our bound on ||UΣV^T – M*||_max is a by-product of that, and a similar result was derived by Jain and Netrapalli (2015) using a different approach.

There are two reasons why entrywise type bounds are important. First, in applications such as recommender systems, it is often desirable to have uniform guarantees for all individuals. If we directly use existing ℓ₂-type inequalities to control entrywise errors, the resulting bounds can be highly sub-optimal in high dimensions. Thus new results are needed. Second, in algorithms based on non-convex optimization (Keshavan et al., 2010b; Sun and Luo, 2016; Jain and Netrapalli, 2015), entrywise bounds are critical for the analysis of initializations and iterations. After the first draft of this paper came out, the entrywise bounds on singular subspaces were applied by Ma et al. (2017) as a guarantee for spectral intialization. The relevance of entrywise bounds goes well beyond matrix completion; see Section 1.5.

For the rest of this subsection, we will illustrate the results in Theorem 3.4 by comparing them with existing ones based on Frobenius norm.

Suppose $p>c\frac{\log n}{n}$ for some large constant c > 0. By Theorems 1.1 and 1.3 of Keshavan et al. (2010b), an upper bound for the root-mean squared error (RMSE) gives:

$$\frac{1}{n}{\Vert U\mathrm{\Sigma }{V}^T-M^{*}\Vert }_F\lesssim \left({\Vert M^{*}\Vert }_{\max }+\sigma \right)\sqrt{\frac{r}{np}}.$$ (3.6)

This implies that the spectral algorithm is rate-optimal when σ ≳ ||M*||_max as Candes and Plan (2010) established a lower bound $\frac{1}{n}{\Vert \widehat{M}-M^{*}\Vert }_F\gtrsim \sigma \sqrt{\frac{r}{np}}$ for any estimator $\widehat{M}$. On the other hand, our Theorem 3.4 asserts that

$${\Vert U\mathrm{\Sigma }{V}^T-M^{*}\Vert }_{\max }{\lesssim }_{\kappa ,r,\eta }\left({\Vert M^{*}\Vert }_{\max }+\sigma \right)\sqrt{\frac{\log n}{np}}.$$

where ≲_κ,r,η hides a factor $O(\kappa ,r,\eta \sqrt{n/r})$ that is not large if certain matrix incoherence structure is assumed; see Candès and Recht (2009) for example. Note that our result recovers (3.6) up to a factor of $\sqrt{\log n}$, since $\Vert X{\Vert }_F\le \sqrt{n_1{n}_2}\Vert X{\Vert }_{\max }$ always holds for any X of size n₁ × n₂.

We also compare the estimation errors of singular vectors under the Frobenius norm and the max-norm. On the one hand, the perturbation inequality in Wedin (1972) and spectral norm concentration yield the following.

$$\begin{gathered} \max \left\{{\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_F,{\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_F\right\}\lesssim \sqrt{r}{\Vert M-M^{*}\Vert }_2/{\sigma }_r^{*} \\ \lesssim \frac{\sqrt{rn/p}\left({\Vert M^{*}\Vert }_{\max }+\sigma \right)}{{\sigma }_r^{*}}\lesssim \frac{n\Vert M^{*}\Vert \max }{{\sigma }_r^{*}}\left(1+\frac{\sigma }{\Vert M^{*}\Vert \max }\right)\sqrt{\frac{r}{np}}. \end{gathered}$$ (3.7)

On the other hand, by our entry-wise bound in Theorem 3.4 we have

$$\begin{gathered} \sqrt{n}\max \left\{{\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_{2\to \infty },{\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_{2\to \infty }\right\} \\ \lesssim \kappa ,r,\eta \frac{n{\Vert M^{*}\Vert }_{\max }}{{\sigma }_r^{*}}\left(1+\frac{\sigma }{\Vert M^{*}\Vert \max }\right)\sqrt{\frac{r\log n}{np}}. \end{gathered}$$ (3.8)

where, as before, ≲_κ,r,η hides a factor $O(\kappa ,r,\eta \sqrt{n/r})$ that is usually not large. Therefore, we also recover (3.7) up to a factor of $\sqrt{\log n}$, since $\Vert X{\Vert }_F\le \sqrt{nr}\Vert X{\Vert }_{\max }$ holds for any X of size n × r. Note that our goal is to derive good max-norm bounds rather than improving Frobenius-norm bounds. The comparisons above demonstrate that our bounds have the ‘correct’ order. To a certain extent, our results better portrait the behavior of spectral algorithm and provide more information than their Frobenius counterparts.

4. Numerical experiments

4.1. ${\mathbb{Z}}_2$-synchronization

We present our numerical results for the phase transition phenomenon of ${\mathbb{Z}}_2$-synchronization—see Figure 5. Fix q₁ = 500^1/50 and q₂ = 2^1/10. For each n in the geometric sequence $\left\{2,2q_1,2q_1^2,\cdots ,2q_1^{50}\right\}$ (rounded to the nearest integers), and each σ in the geometric sequence $\left\{q_2^{-32},q_2^{-31},\cdots ,q_2^{50}\right\}$, we compare our eigenvector-based estimator $\widehat{x}$ with the unknown signal x, and report the proportion of success (namely $\widehat{x}=\pm x$) out of 100 independent runs in the heat map.

Fig 5:

Phase transition of ${\mathbb{Z}}_2$-synchronization: the x-axis is the dimension n, and the y-axis is σ. Lighter pixels refer to higher proportions of runs that $\widehat{x}$ recovers x. The red curve shows the theoretical boundary $\sigma =\sqrt{\frac{n}{2\log n}}$.

A theoretical curve $\sigma =\sqrt{\frac{n}{2\log n}}$ is added onto the heat map. It is clear that below the curve, the eigenvector approach almost always recovers the signal perfectly; and above the curve, it fails to recover the signal.

4.2. Stochastic Block Model

Now we present our simulation results for exact recovery and misclassification rates of SBM. The phase transition phenomenon of SBM is exhibited on the left of Figure 6. In this simulation, n is fixed as 300, and parameters a (y-axis) and b (x-axis) vary from 0 to 30 and 0 to 10, with increments 0.3 and 0.1 respectively. We compare the labels returned by our eigenvector-based method with the true cluster labels, and report the the proportion of success (namely $\widehat{z}=\pm z$) out of 100 independent runs. As before, lighter pixels represent higher chances of success. Two theoretical curves $\sqrt{a}-\sqrt{b}=\pm \sqrt{2}$ are also added onto the heat map. Clearly, theoretical predictions match numerical results.

Fig 6:

Vanilla spectral method for SBM. Left: phase transition of exact recovery. The x-axis is b, the y-axis is a, and lighter pixels represent higher chances of success. Two red curves $\sqrt{a}-\sqrt{b}=\pm \sqrt{2}$ represent theoretical boundaries for phase transtion, matched by numerical results. Right: mean misclassification rates on the logarithmic scale with b = 2. The x-axis is a, varying from 2 to 8, and the y-axis is $\log \mathbb{E}r(\widehat{z},z)/\log n$. No marker: theoretical curve; circles: n = 5000; crosses: n = 500; squares: n = 100.

The right plot of Figure 6 shows misclassification rates of our eigenvector approach with a fixed parameter b and a varying parameter a, where a is not large enough to reach the exact recovery threshold. We fix b = 2, and increase a from 2 to 8 by 0.2 for three different choices of n from {100, 500, 5000}. Then we calculate the mean misclassification rates $\mathbb{E}r(\widehat{z},z)$ averaged over 100 independent runs, and plot $\log \mathbb{E}r(\widehat{z},z)/\log n$ against varying b. We also add a theoretical curve (with no markers), whose y-coordinates are $-{(\sqrt{a}-\sqrt{b})}^2/2$; see Theorem 3.2 (ii). It is clear that with n tending to infinity, the curves of mean misclassification rates move closer to the theoretical one.

4.3. Matrix completion from noisy entries

Finally we come to experiments of matrix completion from noisy entries. The performance of the spectral algorithm in terms of root-mean squared error (RMSE) has already been demonstrated in Keshavan et al. (2010b), among others. In this part, we focus on the comparison between the maximum entrywise errors and RMSEs, for both the singular vectors and the matrix itself. The settings are mainly adopted from Candès and Plan (2010) and Keshavan et al. (2010b). Each time we first create a rank-r matrix $M^{*}\in {ℝ}^{n\times n}$ using the product $M_L{M}_R^T$, where $M_L,M_R\in {ℝ}^{n\times r}$ have i.i.d. $N(0,20/\sqrt{n})$ entries. Then, each entry of M* is picked with probability p and contaminated by random noise drawn from N(0, σ²), independently of others. While increasing n from 500 to 5000 by 500, we choose $p=\frac{10\log n}{n}$, fix r = 5 and σ = 1. All the data presented in the plot are averaged over 100 independent experiments.

In support of our discussions in Section 3.3, Figure 7 shows that the following two ratios

$$\begin{gathered} R_{\text{mat}}=\frac{{\Vert U\mathrm{\Sigma }{V}^T-M^{*}\Vert }_{\max }}{{\eta }^2\sqrt{\log n}\cdot {\Vert U\mathrm{\Sigma }{V}^T-M^{*}\Vert }_F}, \\ {R}_{\text{vec}}=\frac{\max \left\{{\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_{2\to \infty },{\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_{2\to \infty }\right\}}{\eta \sqrt{\log n}\cdot \max \left\{{\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_F,{\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_F\right\}}, \end{gathered}$$

approximately remain constant as n grows. Here the RMSEs n⁻¹||UΣV^T – M*||_F and $n^{-1/2}\max \left\{{\Vert U\mathrm{sgn}(H)-U^{*}\Vert }_F,{\Vert V\mathrm{sgn}(H)-V^{*}\Vert }_F\right\}$ are scaled by ${(\sqrt{n}\eta )}^2\sqrt{\log n}$ and $(\sqrt{n}\eta )\sqrt{\log n}$, respectively. Hence our analysis is sharp, and the perturbations are obviously delocalized among the entries.

Fig 7:

R_mat and R_vec in matrix completion from noisy entries. The x-axis is n, varying from 500 to 5000 by 500, and the y-axis is the ratio. Crosses and circles stand for R_mat and R_vec, respectively.

5. Discussions

We have developed first-order approximations for eigenvectors and eigenspaces with small ℓ_∞ errors under random perturbations. These results lead to sharp guarantees for three statistical problems.

Several future directions deserve exploration. First, the main perturbation theorems are currently stated only for symmetric matrices. We think it may be possible to extend the current analysis to SVD of general rectangular matrices, which has broader applications such as principal component analysis. Second, there are many other graph-related matrices beyond adjacency matrices, including graph Laplacians and non-backtracking matrices, which are important both in theory and in practice. Third, we believe our assumption of row- and column-wise independence can be relaxed to block-wise independence, which is relevant to cryo-EM and other problems.

Finally, in our examples, the spectral algorithm is strongly consistent if and only if the MLE is, though the latter can be NP-hard to compute in general. It would be interesting to see how general this phenomenon is, in view of better understanding the statistical and computational tradeoffs.