NOISY MATRIX COMPLETION: UNDERSTANDING STATISTICAL GUARANTEES FOR CONVEX RELAXATION VIA NONCONVEX OPTIMIZATION

Abstract

This paper studies noisy low-rank matrix completion: given partial and noisy entries of a large low-rank matrix, the goal is to estimate the underlying matrix faithfully and efficiently. Arguably one of the most popular paradigms to tackle this problem is convex relaxation, which achieves remarkable efficacy in practice. However, the theoretical support of this approach is still far from optimal in the noisy setting, falling short of explaining its empirical success.

We make progress towards demystifying the practical efficacy of convex relaxation vis-à-vis random noise. When the rank and the condition number of the unknown matrix are bounded by a constant, we demonstrate that the convex programming approach achieves near-optimal estimation errors — in terms of the Euclidean loss, the entrywise loss, and the spectral norm loss — for a wide range of noise levels. All of this is enabled by bridging convex relaxation with the nonconvex Burer–Monteiro approach, a seemingly distinct algorithmic paradigm that is provably robust against noise. More specifically, we show that an approximate critical point of the nonconvex formulation serves as an extremely tight approximation of the convex solution, thus allowing us to transfer the desired statistical guarantees of the nonconvex approach to its convex counterpart.

1. Introduction.

Suppose we are interested in a large low-rank data matrix, but only get to observe a highly incomplete subset of its entries. Can we hope to estimate the underlying data matrix in a reliable manner? This problem, often dubbed as low-rank matrix completion, spans a diverse array of science and engineering applications (e.g. collaborative filtering [82], localization [86], system identification [70], magnetic resonance parameter mapping [99], joint alignment [21]), and has inspired a flurry of research activities in the past decade. In the statistics literature, matrix completion also falls under the category of factor models with a large amount of missing data, which finds numerous statistical applications such as controlling false discovery rates for dependence data [36,37,39,40], factor-adjusted variable selection [41,63], principal component regression [3, 44, 58, 79], and large covariance matrix estimation [42, 43]. Recent years have witnessed the development of many tractable algorithms that come with statistical guarantees, with convex relaxation being one of the most popular paradigms [14, 15, 46]. See [25, 34] for an overview of this topic.

This paper focuses on noisy low-rank matrix completion, assuming that the revealed entries are corrupted by a certain amount of noise. Setting the stage, consider the task of estimating a rank-r data matrix $M^{\star }={\left[M_{ij}^{\star }\right]}_{1\le i,j\le n}\in {ℝ}^{n\times n}$,¹ and suppose that this needs to be performed on the basis of a subset of noisy entries

$$M_{ij}=M_{ij}^{\star }+E_{ij},(i,j)\in \Omega ,$$ (1.1)

where Ω ⊆ {1, ⋯ , n} × {1, ⋯ , n} denotes a set of indices, and E_ij stands for the additive noise at the location (i, j). As we shall elaborate shortly, solving noisy matrix completion via convex relaxation, while practically exhibiting excellent stability (in terms of the estimation errors against noise), is far less understood theoretically compared to the noiseless setting.

1.1. Convex relaxation: limitations of prior results.

Naturally, one would search for a low-rank solution that best fits the observed entries. One choice is the regularized least-squares formulation given by

$$\underset{Z\in {ℝ}^n{}^{\times n}}{\text{minimize}}\frac{1}{2}\sum _{(i,j)\in \Omega }{\left(Z_{ij}-M_{ij}\right)}^2+\lambda \text{rank}(Z),$$ (1.2)

where λ > 0 is some regularization parameter. In words, this approach optimizes certain trade-off between the goodness of fit (through the squared loss expressed in the first term of (1.2)) and the low-rank structure (through the rank function in the second term of (1.2)). Due to computational intractability of rank minimization, we often resort to convex relaxation in order to obtain computationally feasible solutions. One notable example is the following convex program:

$$\underset{Z\in {ℝ}^n{}^{\times n}}{\text{minimize}}g(Z)\triangleq \frac{1}{2}\sum _{(i,j)\in \Omega }{\left(Z_{ij}-M_{ij}\right)}^2+\lambda \Vert Z{\Vert }_{*},$$ (1.3)

where ‖Z‖_* denotes the nuclear norm (i.e. the sum of singular values) of Z — a convex surrogate for the rank function. A significant portion of existing theory supports the use of this paradigm in the noiseless setting: when E_ij vanishes for all (i, j) ∈ Ω, the solution to (1.3) is known to be faithful (i.e. the estimation error becomes zero) even under near-minimal sample complexity [13–15,20,51,80].

By contrast, the performance of convex relaxation remains largely unclear when it comes to noisy settings (which are often more practically relevant). Candès and Plan [13] first studied the stability of an equivalent variant² of (1.3) against noise. The estimation error ‖Z_cvx – M^⋆‖_F derived therein, of the solution Z_cvx to (1.3), is significantly larger than the oracle lower bound. This does not explain well the effectiveness of (1.3) in practice. In fact, the numerical experiments reported in [13] already indicated that the performance of convex relaxation is far better than their theoretical bounds. This discrepancy between numerical performance and existing theoretical bounds gives rise to the following natural yet challenging questions: Where does the convex program (1.3) stand in terms of its stability vis-à-vis additive noise? Can we establish statistical performance guarantees that match its practical effectiveness?

We note in passing that several other convex relaxation formulations have been thoroughly analyzed for noisy matrix completion, most notably by Negahban and Wainwright [74] and by Koltchinskii et al. [64]. These works have significantly advanced our understanding of the power of convex relaxation. However, the estimators studied therein, particularly the one in [64], are quite different from the one (1.3) considered here; as a consequence, the analysis therein does not lead to improved statistical guarantees of (1.3). Moreover, the performance guarantees provided for these variants are also suboptimal when restricted to the class of “incoherent” or “de-localized” matrices, unless the magnitudes of the noise are fairly large. See Section 1.4 for more detailed discussions as well as numerical comparisons of these algorithms.

1.2. A detour: nonconvex optimization.

While the focus of the current paper is convex relaxation, we take a moment to discuss a seemingly distinct algorithmic paradigm: nonconvex optimization, which turns out to be remarkably helpful in understanding convex relaxation. Inspired by the Burer–Monteiro approach [7], the nonconvex scheme starts by representing the rank-r decision matrix (or parameters) Z as Z = XY^⊤ via low-rank factors $X,Y\in {ℝ}^{n\times r}$, and proceeds by solving the following nonconvex (regularized) least-squares problem [60]

$$\underset{X,Y\in {ℝ}^{n\times r}}{\text{minimize}}\frac{1}{2}\sum _{(i,j)\in \Omega }{\left[{\left(X{Y}^{\top }\right)}_{ij}-M_{ij}\right]}^2+\text{reg}(X,Y).$$ (1.4)

Here, reg(·, ·) denotes a certain regularization term that promotes additional structural properties.

To see its intimate connection with the convex program (1.3), we make the following observation: if the solution to (1.3) has rank r, then it must coincide with the solution to

$$\underset{X,Y\in {ℝ}^{n\times r}}{\text{minimize}}\frac{1}{2}\sum _{(i,j)\in \Omega }{\left[{\left(X{Y}^{\top }\right)}_{ij}-M_{ij}\right]}^2+\underset{\text{reg}(X,Y)}{\underbrace{\frac{\lambda }{2}\Vert X{\Vert }_{\text{F}}^2+\frac{\lambda }{2}\Vert Y{\Vert }_{\text{F}}^2}}.$$ (1.5)

This can be easily verified by recognizing the elementary fact that

$$\Vert Z{\Vert }_{*}=\underset{X,Y\in {ℝ}^n{}^{\times r}:X{Y}^{\top }=Z}{\text{inf}}\left\{\frac{1}{2}\Vert X{\Vert }_{\text{F}}^2+\frac{1}{2}\Vert Y{\Vert }_{\text{F}}^2\right\}$$ (1.6)

for any rank-r matrix Z [73, 87]. Note, however, that it is very challenging to predict when the key assumption in establishing this connection — namely, the rank-r assumption of the solution to the convex program (1.3) — can possibly hold (and in particular, whether it can hold under minimal sample complexity requirement).

Despite the nonconvexity of (1.4), simple first-order optimization methods, in conjunction with proper initialization, are often effective in solving (1.4). Partial examples include gradient descent on manifold [60, 61, 96], gradient descent [71, 88], and projected gradient descent [31, 101]. Apart from their practical efficiency, the nonconvex optimization approach is also appealing in theory. To begin with, algorithms tailored to (1.4) often enable exact recovery in the noiseless setting. Perhaps more importantly, for a wide range of noise settings, the nonconvex approach achieves appealing estimation accuracy [31, 71], which could be significantly better than those bounds derived for convex relaxation discussed earlier. See [25, 33] for a summary of recent results. The appealing estimation accuracy together with the lower computational cost makes nonconvex approaches suitable for large-scale problems. Having said that, the convex approach is also widely used in practice. The convex programming is often observed to enjoy better stability against model mismatch (e.g. non-uniform sampling of the matrix entries [32]), which makes it appealing for moderate-size problems. In contrast, the theoretical understanding of the effectiveness of the convex method does not explain well its empirical performance; see Section 1.1. This motivates us to take a closer inspection of the underlying connection between the two contrasting algorithmic frameworks in the hope that one can utilize the existing theory for the nonconvex approach to improve the stability analysis of the convex relaxation approach.

1.3. Empirical evidence: convex and nonconvex solutions are often close.

In order to obtain a better sense of the relationships between convex and nonconvex approaches, we begin by comparing the estimates returned by the two approaches via numerical experiments. Fix n = 1000 and r = 5. We generate M^⋆ = X^⋆Y^⋆⊤, where $X^{\star },Y^{\star }\in {ℝ}^{n\times r}$ are random orthonormal matrices. Each entry $M_{ij}^{\star }$ of M^⋆ is observed with probability p = 0.2 independently, and then corrupted by an independent Gaussian noise $E_{ij}~\mathcal{N}\left(0,{\sigma }^2\right)$. Throughout the experiments, we set $\lambda =5\sigma \sqrt{np}$. The convex program (1.3) is solved by the proximal gradient method [77], whereas we attempt solving the nonconvex formulation (1.5) by gradient descent with spectral initialization (see [33] for details). Let Z_cvx (resp. $Z_{\text{ncvx}}=X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }$) be the solution returned by the convex program (1.3) (resp. the nonconvex program (1.5)). Figure 1 displays the relative estimation errors of both methods (‖Z_cvx – M^⋆‖_F/‖M^⋆‖_F and (‖Z_ncvx – M^⋆‖_F/‖M^⋆‖_F as well as the relative distance ‖Z_cvx − Z_ncvx‖_F/‖M^⋆‖_F between the two estimates. The results are averaged over 20 independent trials.

Fig. 1.

The relative estimation errors of both Z_cvx (the estimate of the convex program (1.3)) and Z_ncvx (the estimate returned by the nonconvex approach tailored to (1.5)) and the relative distance between them vs. the standard deviation σ of the noise. The results are reported for n = 1000, r = 5, p = 0.2, $\lambda =5\sigma \sqrt{np}$ and are averaged over 20 independent trials.

Interestingly, the distance between the convex and the nonconvex solutions seems extremely small (e.g. ‖Z_cvx −Z_ncvx‖_F/‖M^⋆‖_F is typically below 10⁻⁷); in comparison, the relative estimation errors of both Z_cvx and Z_ncvx are substantially larger. In other words, the estimate returned by the nonconvex approach serves as a remarkably accurate approximation of the convex solution. Given that the nonconvex approach is often guaranteed to achieve intriguing statistical guarantees vis-à-vis random noise [71], this suggests that the convex program is equally stable — a phenomenon that was not captured by prior theory [13]. Can we leverage existing theory for the nonconvex scheme to improve the statistical analysis of the convex relaxation approach?

Before continuing, we remark that the above numerical connection between convex relaxation (1.3) and nonconvex optimization (1.5) has already been observed multiple times in prior literature [45, 61, 73, 81, 87]. Nevertheless, all prior observations on this connection were either completely empirical, or provided in a way that does not lead to improved statistical error bounds of the convex paradigm (1.3). In fact, the difficulty in rigorously justifying the above numerical observations has been noted in the literature; see e.g. [61].³

1.4. Models and main results.

The numerical experiments reported in Section 1.3 suggest an alternative route for analyzing convex relaxation for noisy matrix completion. If one can formally justify the proximity between the convex and the nonconvex solutions, then it is possible to propagate the appealing stability guarantees from the nonconvex scheme to the convex approach. As it turns out, this simple idea leads to significantly enhanced statistical guarantees for the convex program (1.3), which we formally present in this subsection.

1.4.1. Models and assumptions.

Before proceeding, we introduce a few model assumptions that play a crucial role in our theory.

Assumption 1.

1. (Random sampling) Each index (i, j) belongs to the index set Ω independently with probability p.

2. (Random noise) The noise matrix E = [E_ij]_1≤i,j≤n is composed of i.i.d. zero-mean sub-Gaussian random variables with sub-Gaussian norm at most σ > 0, i.e. ${\Vert E_{ij}\Vert }_{{\psi }_2}\le \sigma$ (see [94, Definition 5.7]).

In addition, let M^⋆ = U^⋆Σ^⋆V^⋆⊤ be the singular value decomposition (SVD) of M^⋆, where U^⋆, $V^{\star }\in {ℝ}^{n\times r}$ consist of orthonormal columns and ${\Sigma }^{\star }=\text{diag}\left({\sigma }_1^{\star },{\sigma }_2^{\star },\cdots ,{\sigma }_r^{\star }\right)\in {ℝ}^{r\times r}$ is a diagonal matrix obeying ${\sigma }_{\text{max}}\triangleq {\sigma }_1^{\star }\ge {\sigma }_2^{\star }\ge \cdots \ge {\sigma }_r^{\star }\triangleq {\sigma }_{\text{min}}$. Denote by $\kappa \triangleq {\sigma }_{\text{max}}/{\sigma }_{\text{min}}$ the condition number of M^⋆. We impose the following incoherence condition on M^⋆, which is known to be crucial for reliable recovery of M^⋆ [14, 20].

Definition 1.1.

A rank-r matrix $M^{\star }\in {ℝ}^{n\times n}$ with SVD M^⋆ = U^⋆Σ^⋆V^⋆⊤ is said to be μ-incoherent if

$${\Vert U^{\star }\Vert }_{2,\infty }\le \sqrt{\frac{\mu }{n}}{\Vert U^{\star }\Vert }_{\text{F}}=\sqrt{\frac{\mu r}{n}}\mathit{\text{and}}{\Vert V^{\star }\Vert }_{2,\infty }\le \sqrt{\frac{\mu }{n}}{\Vert V^{\star }\Vert }_{\text{F}}=\sqrt{\frac{\mu r}{n}}.$$

Here, ‖U‖_2,∞ denotes the largest ${\mathcal{l}}_2$ norm of all rows of a matrix U.

Remark 1.2.

It is worth noting that several other conditions on the low-rank matrix have been proposed in the noisy setting. Examples include the spikiness condition [74] and the bounded ${\mathcal{l}}_{\infty }$ norm condition [64]. However, these conditions alone are often unable to ensure identifiability of the true matrix even in the absence of noise.

1.4.2. Theoretical guarantees: when both the rank and the condition number are constants.

With these in place, we are positioned to present our improved statistical guarantees for convex relaxation. For convenience of presentation, we shall begin with a simple yet fundamentally important class of settings when the rank r and the condition number κ are both fixed constants. As it turns out, this class of problems arises in a variety of engineering applications. For example, in a fundamental problem in cryo-EM called angular synchronization [85], one needs to deal with rank-2 or rank-3 matrices with κ = 1; in a joint shape mapping problem that arises in computer graphics [29, 54], the matrix under consideration has low rank and a condition number equal to 1; and in structure from motion in computer vision [91], one often seeks to estimate a matrix with r ≤ 3 and a small condition number. Encouragingly, our theory delivers near-optimal statistical guarantees for such practically important scenarios.

Theorem 1.3.

Let M^⋆ be rank-r and μ-incoherent with a condition number κ, where the rank and the condition number satisfy r, κ = O(1).⁴ Suppose that Assumption 1 holds and take $\lambda =C_{\lambda }\sigma \sqrt{np}$ in (1.3) for some large enough constant C_λ > 0. Assume the sample size obeys n²p ≥ Cμ²n log³ n for some sufficiently large constant C > 0, and the noise satisfies $\sigma \lesssim \sqrt{\frac{np}{{\mu }^3\text{log}n}}{\Vert M^{\star }\Vert }_{\infty }$ for some sufficiently small constant c > 0. Then with probability exceeding 1 − O(n⁻³):

1. Any minimizer Z_cvx of (1.3) obeys

   $${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\text{F}}\lesssim \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}{\Vert M^{\star }\Vert }_{\text{F}};\Vert Z_{\text{cvx}}-M^{\star }\Vert \lesssim \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}\Vert M^{\star }\Vert ;$$ (1.7a)

   $${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\infty }\lesssim \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{\mu n\text{log}n}{p}}{\Vert M^{\star }\Vert }_{\infty }.$$ (1.7b)
2. Letting $Z_{\text{cvx},r}\triangleq \text{arg}{\text{min}}_{Z:\text{rank}(Z)\le r}{\Vert Z-Z_{\text{cvx}}\Vert }_{\text{F}}$ be the best rank-r approximation of Z_cvx, we have⁵

   $${\Vert Z_{\text{cvx},r}-Z_{\text{cvx}}\Vert }_{\text{F}}\le \frac{1}{n^3}\cdot \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}\Vert M^{\star }\Vert ,$$ (1.8)

   and the error bounds in (1.7) continue to hold if Z_cvx is replaced by Z_cvx,r.

To explain the applicability of the above theorem, we first remark on the conditions required for this theorem to hold; for simplicity, we assume that μ = O(1).

• Sample complexity. To begin with, the sample size needs to exceed the order of npoly log n, which is information-theoretically optimal up to some logarithmic term [15].

• Noise size. We then turn attention to the noise requirement, i.e. $\sigma \lesssim \sqrt{np/(\text{log}n)}{\Vert M^{\star }\Vert }_{\infty }$. Note that under the sample size condition $n^{2}p\ge Cn{\text{log}}^{3}n$, the size of the noise in each entry is allowed to be substantially larger than the maximum entry in the matrix. In other words, the signal-to-noise ratio w.r.t. each observed entry could be very small. According to prior literature (e.g. [61, Theorem 1.1] and [71, Theorem 2]), such noise conditions are typically required for spectral methods to perform noticeably better than random guessing.

• Regularization parameter. In the end, we remark on the choice of the regularization parameter λ. As in most regularized estimators, we need to pick the regularization parameter λ large enough so as to suppress the noise E (which controls the variance), and small enough so as not to shrink the signal M^⋆ too much (which controls the estimation bias). It turns out that setting $\lambda \asymp \sigma \sqrt{np}$ achieves the desired bias-variance trade-off; see Lemma 3.

  Further, Theorem 1.3 has several important implications about the power of convex relaxation. The discussions below again concentrate on the case where μ = O(1).

• Near-optimal stability guarantees. Our results reveal that the Euclidean error of any convex optimizer Z_cvx of (1.3) obeys

  $${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\text{F}}\lesssim \sigma \sqrt{n/p},$$ (1.9)

  implying that the performance of convex relaxation degrades gracefully as the signal-to-noise ratio decreases. This result matches the oracle lower bound derived in [13, Eq. (III.13)], which also improves upon their statistical guarantee. Specifically, Candès and Plan [13] provided a stability guarantee in the presence of arbitrary bounded noise. When applied to the random noise model assumed here, their results yield ‖Z_cvx – M^⋆‖_F ≲ σn^3/2, which could be $O\left(\sqrt{n^{2}p}\right)$ times more conservative than our bound (1.9).

• Nearly low-rank structure of the convex solution. In light of (1.8), the optimizer of the convex program (1.3) is almost, if not exactly, rank-r. When the true rank r is known a priori, it is not uncommon for practitioners to return the rank-r approximation of Z_cvx. Our theorem formally justifies that there is no loss of statistical accuracy — measured in terms of either ‖·‖_F or ‖·‖_∞ — when performing the rank-r projection operation.

• Entrywise and spectral norm error control. Moving beyond the Euclidean loss, our theory uncovers that the estimation errors of the convex optimizer are fairly spread out across all entries, thus implying near-optimal entrywise error control. This is a stronger form of error bounds, as an optimal Euclidean estimation accuracy alone does not preclude the possibility of the estimation errors being spiky and localized. Furthermore, the spectral norm error of the convex optimizer is also well-controlled. See Figure 2 for the numerical support.

• Implicit regularization. As a byproduct of the entrywise error control, this result indicates that the additional constraint ‖Z‖_∞ ≤ α suggested by [74] is automatically satisfied and is hence unnecessary. In other words, the convex approach implicitly controls the spikiness of its entries, without resorting to explicit regularization. This is also confirmed by the numerical experiments reported in Figure 3, where we see that the estimation error of (1.3) and that of the constrained version considered in [74] are nearly identical.

• Statistical guarantees for fast iterative optimization methods. Various iterative algorithms have been developed to solve the nuclear norm regularized least-squares problem (1.3) up to an arbitrarily prescribed accuracy, examples including SVT (or proximal gradient methods) [8], FPC [72], SOFT–IMPUTE [73], FISTA [5, 90], to name just a few. Our theory immediately provides statistical guarantees for these algorithms. As we shall make precise in Section 2, any point Z with g(Z) ≤ g(Z_cvx) + ε (where g(·) is defined in (1.3)) enjoys the same error bounds as in (1.7) (with Z_cvx replaced by Z in (1.7)), provided that ε > 0 is sufficiently small. In other words, when these convex optimization algorithms converge w.r.t. the objective value, they are guaranteed to return a statistically reliable estimate.

Fig. 2.

The relative estimation error of Z_cvx measured by both ‖ · ‖_∞ (i.e. ‖Z_cvx – M^⋆‖_∞/‖M^⋆‖_∞ and ‖ · ‖ (i.e. ‖Z_cvx – M^⋆‖/‖M^⋆‖) vs. the standard deviation σ of the noise. The results are reported for n = 1000, r = 5, p = 0.2, $\lambda =5\sigma \sqrt{np}$ and are averaged over 20 independent trials.

Fig. 3.

The relative estimation errors of $\widehat{Z}$, measured in terms of ${\mathcal{l}}_{\text{F}}$ and ${\mathcal{l}}_{\infty }$, vs. the standard deviation σ of the noise. Here $\widehat{Z}$ can be either the modified convex estimator in [64], the constrained convex estimator in [74] or the vanilla convex estimator (1.3). The results are reported for n = 1000, r = 5, p = 0.2, and are averaged over 20 Monte-Carlo trials. For the modified convex estimator in [64], we choose the regularization parameter λ therein to be 1.5 $\text{max}\left\{\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right\}\sqrt{1/\left(n^{3}p\right)}$, as suggested by their theory. For the constrained one in [74], the regularization parameter λ is set to be $5\sigma \sqrt{np}$ and the constraint α is set to be ‖M^⋆‖_∞. Both choices are recommended by [74]. As for (1.3), we set $\lambda =5\sigma \sqrt{np}$.

To better understand our contributions, we take a moment to discuss two important but different convex programs studied in [74] and [64]. To begin with, under a spikiness assumption on the low-rank matrix, Negahban and Wainwright [74] proposed to enforce an extra entrywise constraint ‖Z‖_∞ ≤ α when solving (1.3), in order to explicitly control the spikiness of the estimate. When applied to our model with r, κ, μ ≍ 1, their results read (up to some logarithmic factor)

$${\Vert \widehat{Z}-M^{\star }\Vert }_{\text{F}}\lesssim \text{max}\left\{\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right\}\sqrt{n/p},$$ (1.10)

where $\widehat{Z}$ is the estimate returned by their modified convex algorithm. While this matches the optimal bound when σ ≳ ‖M^⋆‖_∞, it becomes suboptimal when σ ≪ ‖M^⋆‖_∞ (under our models). Moreover, as we have already discussed, the extra spikiness constraint becomes unnecessary in the regime considered herein. This also means that our result complements existing theory about the convex program in [74] by demonstrating its minimaxity for an additional range of noise. Another work by Koltchinskii et al. [64] investigated a completely different convex algorithm, which is effiectively a spectral method (namely, one round of soft singular value thresholding on a rescaled zero-padded data matrix). The algorithm is shown to be minimax optimal over the class of low-rank matrices with bounded ${\mathcal{l}}_{\infty }$ norm (note that this is very different from the set of incoherent matrices studied here). When specialized to our model, their error bound is the same as (1.10) (modulo some log factor), which also becomes suboptimal as ˙ decreases. The advantage of the convex program (1.3) is shown in Figure 3.

1.4.3. Theoretical guarantees: extensions to more general settings.

So far we have presented results when the true matrix has bounded rank and condition number, i.e. r, κ = O(1). Our theory actually accommodates a significantly broader range of scenarios, where the rank and the condition number are both allowed to grow with the dimension n.

Theorem 1.4.

Let M^⋆ be rank-r and μ-incoherent with a condition number κ. Suppose Assumption 1 holds and take $\lambda =C_{\lambda }\sigma \sqrt{np}$ in (1.3) for some large enough constant C_λ > 0. Assume the sample size obeys n²p ≥ C κ⁴μ²r²n log³ n for some sufficiently large constant C > 0, and the noise satisfies $\sigma \sqrt{\frac{n}{p}}\le c\frac{{\sigma }_{\text{min}}}{\sqrt{{\kappa }^4\mu r\text{log}n}}$ for some sufficiently small constant c > 0. Then with probability exceeding 1 − O(n⁻³),

1. Any minimizer Z_cvx of (1.3) obeys

   $${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\text{F}}\lesssim \kappa \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}{\Vert M^{\star }\Vert }_{\text{F}},$$ (1.11a)

   $${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\infty }\lesssim \sqrt{{\kappa }^3\mu r}\cdot \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n\text{log}n}{p}}{\Vert M^{\star }\Vert }_{\infty },$$ (1.11b)

   $$\Vert Z_{\text{cvx}}-M^{\star }\Vert \lesssim \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}\Vert M^{\star }\Vert ;$$ (1.11c)
2. Letting $Z_{\text{cvx},r}\triangleq \text{arg}{\text{min}}_{Z:\text{rank}(Z)\le r}{\Vert Z-Z_{\text{cvx}}\Vert }_{\text{F}}$ be the best rank-r approximation of Z_cvx, we have

   $${\Vert Z_{\text{cvx},r}-Z_{\text{cvx}}\Vert }_{\text{F}}\le \frac{1}{n^3}\cdot \frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}\Vert M^{\star }\Vert ,$$ (1.12)

   and the error bounds in (1.11) continue to hold if Z_cvx is replaced by Z_cvx,r.

Remark 1.5 (The noise condition).

The incoherence condition (cf. Definition 1.1) guarantees that the largest entry ‖M^⋆‖_∞ of the matrix M^⋆ is no larger than κμrσ_min/n. As a result, the noise condition stated in Theorem 1.4 covers all scenarios obeying

$$\sigma \lesssim \sqrt{\frac{np}{{\kappa }^6{\mu }^3{r}^3\text{log}n}}{\Vert M^{\star }\Vert }_{\infty }.$$

Therefore, the typical size of the noise is allowed to be much larger than the size of the largest entry of M^⋆, provided that $p\gg \frac{{\kappa }^6{\mu }^3{r}^3\text{log}n}{n}$. In particular, when r, κ = O(1), this recovers the noise condition in Theorem 1.3.

Notably, the sample size condition for noisy matrix completion (i.e. n²p ≥ Cκ⁴μ²r²n log³ n) is more stringent than that in the noiseless setting (i.e. n²p ≍ nr log² n), and our statistical guarantees are likely suboptimal with respect to the dependency on r and κ. It turns out that both convex and nonconvex methods work well numerically even when the number of samples n²p is on the order of nr log n, which is much smaller than the required sample complexity κ⁴r²n log³ n in Theorem 1.4; see Figure 4 for an illustration. From a technical point of view, this sub-optimality is mainly due to the analysis of nonconvex optimization, a key ingredient of our analysis of convex relaxation. In fact, the state-of-the-art nonconvex analysis [31, 61, 71] requires the sample size to be much larger than the optimal one (e.g. n²p ≫ npoly(r)poly(κ)) even in the noiseless setting. It would certainly be interesting, and in fact important, to see whether it is possible to develop a theory with optimal dependency on r and κ. We leave this for future investigation.

Fig. 4.

The estimation errors of Z_cvx and Z_ncvx, measured in terms of ${\mathcal{l}}_{\text{F}}$, vs. the rescaled sample complexity n²p/(2nr log n). The results are reported for n = 1000, p = 0.2, σ = 10⁻⁴, and are averaged over 20 Monte-Carlo trials. The rank r is varied from 1 to 25 and the condition number κ is chosen from {1, 10, 100}.

It is also instrumental to compare our sample complexity and error bounds with those in the prior literature [10, 62, 64, 74]. See Table 1 for a comparison when the incoherence parameter μ is O(1). Here

$${\alpha }^{\star }\triangleq n\frac{{\Vert M^{\star }\Vert }_{\infty }}{{\Vert M^{\star }\Vert }_{\text{F}}},\text{\hspace{1em} and \hspace{1em}}{\Vert M^{\star }\Vert }_{\text{max}}\triangleq \underset{M^{\star }=U{V}^{\top }}{\text{min}}\Vert U{\Vert }_{2,\infty }\Vert V{\Vert }_{2,\infty }.$$

Indeed, all the papers [10, 62, 64, 74] studied convex relaxation approaches for noisy matrix completion. However, there are two main differences from our analysis here. The first is regarding the convex method itself. All four papers [10, 62, 64,74] require extra knowledge about the underlying low-rank matrix for the convex approach, e.g. the ${\mathcal{l}}_{\infty }$ norm of the low-rank matrix in [10, 62, 74] and the sharpness constant in [74]. This results in different convex formulations from what we analyze here. The second major difference lies in the performance guarantees. In both the well-conditioned and the highly ill-conditioned regimes, the estimation error provided in [10, 62, 64, 74] does not vanish as the size of the noise decreases to zero. This is in stark contrast to our theory that guarantees the stability of convex approach for noisy matrix completion.

Table 1.

Comparisons of sample complexities and Euclidean estimation errors when μ = O(1).

	Sample complexity	Euclidean estimation error
[64]	n log2 n	$\text{max}\left(\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right)\sqrt{(nr\text{log}n)/p}$
[75]	n log n	$\text{max}(\sigma ,1/n){\alpha }^{\star }\sqrt{(nr\text{log}n)/p}$
[62]	n log3 n	$\text{max}\left(\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right)\sqrt{(nr\text{log}n)/p}$
[10]	n	$\sqrt{\text{max}\left(\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right)}{\Vert M^{\star }\Vert }_{\text{max}}{n}^{3/4}/p^{1/4}$
Ours	κ4r2n log3 n	${\kappa }^2\sigma \sqrt{nr/p}$

Despite the above sub-optimality issue, implications similar to those of Theorem 1.3 hold for this general setting. To begin with, the nearly low-rank structure of the convex solution is preserved (cf. (1.12)). In addition, the estimation error of the convex estimate is spread out across entries (cf. (1.11b)), thus uncovering an implicit regularization phenomenon underlying convex relaxation (which implicitly regularizes the spikiness constraint on the solution). Last but not least, the upper bounds (1.11) and (1.12) continue to hold for approximate minimizers of the convex program (1.3), thus yielding statistical guarantees for numerous iterative algorithms aimed at minimizing (1.3).

1.5. Numerical experiments.

This subsection collects numerical supports for our theoretical findings in Section 1.4.

Entrywise and spectral norm error of the convex approach..

The experimental setting is similar to that in producing Figure 1. For completeness, we repeat here. Fix n = 1000 and r = 5. We generate M^⋆ = X^⋆Y^⋆⊤, where $X^{\star },Y^{\star }\in {ℝ}^{n\times r}$ are random orthonormal matrices. Each entry $M_{ij}^{\star }$ of M^⋆ is observed with probability p = 0.2 independently, and then corrupted by an independent Gaussian noise $E_{ij}~\mathcal{N}\left(0,{\sigma }^2\right)$. Throughout the experiments, we set $\lambda =5\sigma \sqrt{np}$. The convex program (1.3) is solved by the proximal gradient method [77]. Figure 2 displays the relative estimation errors of the convex approach (1.3) in both the ${\mathcal{l}}_{\infty }$ norm and the spectral norm. As can be seen, both forms of estimation errors scale linearly with the noise level, corroborating our theory.

Comparisons with other convex approaches..

Utilizing the same experimental setting as before, we compare the numerical performance of (1.3) with two important but different convex programs studied in [74] and [64]. For the modified convex estimator in [64], we choose the regularization parameter λ therein to be 1.5 $\text{max}\left\{\sigma ,{\Vert M^{\star }\Vert }_{\infty }\right\}\sqrt{1/\left(n^{3}p\right)}$, as suggested by their theory. For the constrained one in [74], the regularization parameter λ is set to be $5\sigma \sqrt{np}$ and the constraint α is set to be ‖M^⋆‖_∞. Both choices are recommended by [74]. Figure 3 displays the relative Euclidean and entrywise estimation error of the three convex programs. As can be seen, the estimation error of this thresholding-based spectral algorithm [64] does not decrease as the noise shrinks, and its performance seems uniformly outperformed by that of convex relaxation (1.3) and the constrained estimator in [74]. In fact, this is part of our motivation to pursue an improved theoretical understanding of the formulation (1.3).

Numerical sample complexity of both convex and nonconvex methods..

Theorem 1.4 requires the sample complexity n²p to exceed κ⁴r²n log³ n, which we believe could be improved. To justify this, under a similar setting to that of Figure 1, we plot the estimation errors of Z_cvx and Z_ncvx vs. the rescaled sample complexity n²p/(2nr log n) in Figure 4. It can be seen that the estimation errors degrade gracefully as the rescaled sample size gets smaller and the performance does not change w.r.t. the condition number κ.

2. Strategy and novelty.

In this section, we introduce the strategy for proving our main theorem, i.e. Theorem 1.4. Theorem 1.3 follows immediately. Informally, the main technical difficulty stems from the lack of closed-form expressions for the primal solution to (1.3), which in turn makes it difficult to construct a dual certificate. This is in stark contrast to the noiseless setting, where one clearly anticipates the ground truth M^⋆ to be the primal solution; in fact, this is precisely why the analysis for the noisy case is significantly more challenging. Our strategy, as we shall detail below, mainly entails invoking an iterative nonconvex algorithm to “approximate” such a primal solution.

Before continuing, we introduce a few more notations. Let ${\mathcal{P}}_{\Omega }(\cdot ):{ℝ}^{n\times n}\mapsto {ℝ}^{n\times n}$ represent the projection onto the subspace of matrices supported on Ω, namely,

$${\left[{\mathcal{P}}_{\Omega }(Z)\right]}_{ij}=\{\begin{array}{ll}{Z}_{ij},\hfill & \text{ for }(i,j)\in \Omega \hfill \\ 0,\hfill & \text{ otherwise }\hfill \end{array}$$ (2.1)

for any matrix $Z\in {ℝ}^{n\times n}$. For a rank-r matrix M with singular value decomposition UΣV^⊤, denote by T the tangent space of the rank-r manifold at M, i.e.

$$T=\left\{U{A}^{\top }+B{V}^{\top }\mid A,B\in {ℝ}^{n\times r}\right\}.$$ (2.2)

Correspondingly, let ${\mathcal{P}}_T(\cdot )$ be the orthogonal projection onto the subspace T, that is,

$${\mathcal{P}}_T(Z)=U{U}^{\top }Z+ZV{V}^{\top }-U{U}^{\top }ZV{V}^{\top }$$ (2.3)

for any matrix $Z\in {ℝ}^{n\times n}$. In addition, let T^⊥ and ${\mathcal{P}}_{T^{\perp }}(\cdot )$ denote the orthogonal complement of T and the projection onto T^⊥, respectively. Regarding the ground truth, we denote

$$X^{\star }=U^{\star }{\left({\Sigma }^{\star }\right)}^{1/2}\text{\hspace{1em} and \hspace{1em}}{Y}^{\star }=V^{\star }{\left({\Sigma }^{\star }\right)}^{1/2}.$$ (2.4)

The nonconvex problem (1.5) is equivalent to

$$\underset{X,Y\in {ℝ}^{n\times r}}{\text{minimize}}f(X,Y)\triangleq \frac{1}{2p}{\Vert {\mathcal{P}}_{\Omega }\left(X{Y}^{\top }-M\right)\Vert }_{\text{F}}^2+\frac{\lambda }{2p}\Vert X{\Vert }_{\text{F}}^2+\frac{\lambda }{2p}\Vert Y{\Vert }_{\text{F}}^2,$$ (2.5)

where we have inserted an extra factor 1/p (compared to (1.5)) to simplify the presentation of the analysis later on.

2.1. Exact duality.

In order to analyze the convex program (1.3), it is natural to start with the first-order optimality condition. Specifically, suppose that $Z\in {ℝ}^{n\times n}$ is a (primal) solution to (1.3) with SVD Z = UΣV^⊤.⁶ As before, let T be the tangent space of Z, and let T^⊥ be the orthogonal complement of T. Then the first-order optimality condition for (1.3) reads: there exists a matrix W ∈ T^⊥ (called a dual certificate) such that

$$\frac{1}{\lambda }{\mathcal{P}}_{\Omega }(M-Z)=U{V}^{\top }+W;$$ (2.6a)

$$\Vert W\Vert \le 1.$$ (2.6b)

This condition is not only necessary to certify the optimality of Z, but also “almost sufficient” in guaranteeing the uniqueness of the solution Z; see Appendix SM2 for in-depth discussions.

The challenge then boils down to identifying such a primal-dual pair (Z, W) satisfying the optimality condition (2.6). For the noise-free case, the primal solution is clearly Z = M^⋆ if exact recovery is to be expected; the dual certificate can then be either constructed exactly by the least-squares solution to a certain underdetermined linear system [14, 15], or produced approximately via a clever golfing scheme pioneered by Gross [51]. For the noisy case, however, it is often difficult to hypothesize on the primal solution Z, as it depends on the random noise in a complicated way. In fact, the lack of a suitable guess of Z (and hence W) was the major hurdle that prior works faced when carrying out the duality analysis.

2.2. A candidate primal solution via nonconvex optimization.

Motivated by the numerical experiment in Section 1.3, we propose to examine whether the optimizer of the nonconvex problem (1.5) stays close to the solution to the convex program (1.3). Towards this, suppose that $X,Y\in {ℝ}^{n\times r}$ form a critical point of (1.5) with rank(X) = rank(Y) = r.⁷ Then the first-order condition reads

$$\frac{1}{\lambda }{\mathcal{P}}_{\Omega }\left(M-X{Y}^{\top }\right)Y=X;$$ (2.7a)

$$\frac{1}{\lambda }{\left[{\mathcal{P}}_{\Omega }\left(M-X{Y}^{\top }\right)\right]}^{\top }X=Y.$$ (2.7b)

To develop some intuition about the connection between (2.6) and (2.7), let us take a look at the case with r = 1. Denote X = x and Y = y and assume that the two rank-1 factors are “balanced”, namely, ‖x‖₂ = ‖y‖₂ ≠ 0. It then follows from 2.7) that ${\lambda }^{-1}{\mathcal{P}}_{\Omega }\left(M-x{y}^{\top }\right)$ has a singular value 1, whose corresponding left and right singular vectors are x/‖x‖₂ and y/‖y‖₂, respectively. In other words, one can express

$$\frac{1}{\lambda }{\mathcal{P}}_{\Omega }\left(M-x{y}^{\top }\right)=\frac{1}{\Vert x{\Vert }_2\Vert y{\Vert }_2}x{y}^{\top }+W,$$ (2.8)

where W is orthogonal to the tangent space of xy^⊤; this is precisely the condition (2.6a). It remains to argue that (2.6b) is valid as well. Towards this end, the first-order condition (2.7) alone is insufficient, as there might be non-global critical points (e.g. saddle points) that are unable to approximate the convex solution well. Fortunately, as long as the candidate xy^⊤ is not far away from the ground truth M^⋆, one can guarantee ‖W‖ < 1 as required in (2.6b).

The above informal argument about the link between the convex and the nonconvex problems can be formalized. To begin with, we introduce the following conditions on the regularization parameter λ.

Condition 1 (Regularization parameter).

The regularization parameter λ satisfies

1. (Relative to noise) $\Vert {\mathcal{P}}_{\Omega }(E)\Vert <\lambda /8$.

2. (Relative to nonconvex solution) $\Vert {\mathcal{P}}_{\Omega }\left(X{Y}^{\top }-M^{\star }\right)-p\left(X{Y}^{\top }-M^{\star }\right)\Vert <\lambda /8$.

Remark 2.1.

Condition 1 requires that the regularization parameter λ should dominate a certain norm of the noise, as well as of the deviation of XY^⊤ − M^⋆ from its mean p(XY^⊤ − M^⋆); as will be seen shortly, the latter condition can be met when (X, Y) is sufficiently close to (X^⋆, Y^⋆).

With the above condition in place, the following result demonstrates that a critical point (X, Y) of the nonconvex problem (1.5) readily translates to the unique minimizer of the convex program (1.3). This lemma is established in Appendix SM3.1.

Lemma 2.2 (Exact nonconvex vs. convex optimizers).

Suppose that (X, Y) is a critical point of (1.5) satisfying rank(X) = rank(Y) = r, and the sampling operator ${\mathcal{P}}_{\Omega }$ is injective when restricted to the elements of the tangent space T of XY^⊤, namely,

$${\mathcal{P}}_{\Omega }(H)=0\iff H=0,forallH\in T.$$ (2.9)

Under Condition 1, the point $Z\triangleq X{Y}^{\top }$ is the unique minimizer of (1.3).

In order to apply Lemma 2.2, one needs to locate a critical point of (1.5) that is sufficiently close to the truth, for which one natural candidate is the global optimizer of (1.5). The caveat, however, is the lack of theory characterizing directly the properties of the optimizer of (1.5). Instead, what is available in prior theory is the characterization of some iterative sequence (e.g. gradient descent iterates) aimed at solving (1.5). It is unclear from prior theory whether the iterative algorithm under study (e.g. gradient descent) converges to the global optimizer in the presence of noise. This leads to technical difficulty in justifying the proximity between the nonconvex optimizer and the convex solution via Lemma 2.2.

2.3. Approximate nonconvex optimizers.

Fortunately, perfect knowledge of the nonconvex optimizer is not pivotal. Instead, an approximate solution to the nonconvex problem (1.5) (or equivalently (2.5)) suffices to serve as a reasonably tight approximation of the convex solution. More precisely, we desire two factors (X, Y) that result in nearly zero (rather than exactly zero) gradients:

$${\nabla }_{X}f(X,Y)\approx \mathbf{0}\text{\hspace{1em} and \hspace{1em}}{\nabla }_{Y}f(X,Y)\approx \mathbf{0},$$

where f(·, ·) is the nonconvex objective function as defined in (2.5). This relaxes the condition discussed in Lemma 2.2 (which only applies to critical points of (1.5) as opposed to approximate critical points). As it turns out, such points can be found via gradient descent tailored to (1.5). The sufficiency of the near-zero gradient condition is made possible by slightly strengthening the injectivity assumption (2.9), which is stated below.

Condition 2 (Injectivity).

Let T be the tangent space of XY^⊤. There is a quantity c_inj > 0 such that

$$p^{-1}{\Vert {\mathcal{P}}_{\Omega }(H)\Vert }_{\text{F}}^2\ge c_{\text{inj}}\Vert H{\Vert }_{\text{F}}^2,forallH\in T.$$ (2.10)

The following lemma states quantitatively how an approximate nonconvex optimizer serves as an excellent proxy of the convex solution, which we establish in Appendix SM3.2.

Lemma 2.3 (Approximate nonconvex vs. convex optimizers).

Suppose that (X, Y) obeys

$$\Vert \nabla f(X,Y){\Vert }_{\text{F}}\le c\frac{\sqrt{c_{\text{inj}}p}}{\kappa }\cdot \frac{\lambda }{p}\sqrt{{\sigma }_{\text{min}}}$$ (2.11)

for some sufficiently small constant c > 0. Further assume that any singular value of X and Y lies in $\left[\sqrt{{\sigma }_{\text{min}}/2},\sqrt{2{\sigma }_{\text{max}}}\right]$. Then under Conditions 1 and 2, any minimizer Z_cvx of (1.3) satisfies

$${\Vert X{Y}^{\top }-Z_{\text{cvx}}\Vert }_{\text{F}}\lesssim \frac{\kappa }{c_{\text{inj}}}\frac{1}{\sqrt{{\sigma }_{\text{min}}}}\Vert \nabla f(X,Y){\Vert }_{\text{F}}.$$ (2.12)

Remark 2.4.

In fact, this lemma continues to hold if Z_cvx is replaced by any Z obeying g(Z) ≤ g(XY^⊤), where g(·) is the objective function defined in (1.3) and X and Y are low-rank factors obeying conditions of Lemma 2.3. This is important in providing statistical guarantees for iterative methods like SVT [8], FPC [72], SOFT– IMPUTE [73], FISTA [5], etc. To be more specific, suppose that (X, Y) results in an approximate optimizer of (1.3), namely, g(XY^⊤) = g(Z_cvx) + ε for some sufficiently small ε > 0. Then for any Z obeying g(Z) ≤ g(XY^⊤) = g(Z_cvx) + ε, one has

$${\Vert X{Y}^{\top }-Z\Vert }_{\text{F}}\lesssim \frac{\kappa }{c_{\text{inj}}}\frac{1}{\sqrt{{\sigma }_{\text{min}}}}\Vert \nabla f(X,Y){\Vert }_{\text{F}}.$$ (2.13)

As a result, as long as the above-mentioned algorithms converge in terms of the objective value, they must return a solution obeying (2.13), which is exceedingly close to XY^⊤ if ‖∇f(X, Y)‖_F is small.

It is clear from Lemma 2.3 that, as the size of the gradient ∇f(X, Y) gets smaller, the nonconvex estimate XY^⊤ becomes an increasingly tighter approximation of any convex optimizer of (1.3), which is consistent with Lemma 2.2. In contrast to Lemma 2.2, due to the lack of strong convexity, a nonconvex estimate with a near-zero gradient does not imply the uniqueness of the optimizer of the convex program (1.3); rather, it indicates that any minimizer of (1.3) lies within a sufficiently small neighborhood surrounding XY^⊤ (cf. (2.12)).

2.4. Construction of an approximate nonconvex optimizer.

So far, Lemmas 2.2–2.3 are both deterministic results based on Condition 1. As we will soon see, under Assumption 1, we can derive simpler conditions that — with high probability — guarantee Condition 1. We start with Condition 1(a).

Lemma 2.5.

Suppose n²p ≥ Cn log² n for some sufficiently large constant C > 0. Then with probability at least 1 − O(n⁻¹⁰), one has $\Vert {\mathcal{P}}_{\Omega }(E)\Vert \lesssim \sigma \sqrt{np}$. As a result, Condition 1 holds (i.e. $\Vert {\mathcal{P}}_{\Omega }(E)\Vert <\lambda /8$) as long as $\lambda =C_{\lambda }\sigma \sqrt{np}$ for some sufficiently large constant C_λ > 0.

Proof.

This follows from [31, Lemma 11] with a slight and straightforward modification to accommodate the asymmetric noise here. For brevity, we omit the proof. □

Turning attention to Condition 1(b) and Condition 2, we have the following lemma, the proof of which is deferred to Appendix SM3.3.

Lemma 2.6.

Under the assumptions of Theorem 1.4, with probability exceeding 1 – O(n⁻¹⁰) we have $\Vert {\mathcal{P}}_{\Omega }\left(X{Y}^{\top }-M^{\star }\right)-p\left(X{Y}^{\top }-M^{\star }\right)\Vert <\lambda /8$ (Condition 1 (b)) $\frac{1}{p}{\Vert {\mathcal{P}}_{\Omega }(H)\Vert }_{\text{F}}^2\ge \frac{1}{32\kappa }\Vert H{\Vert }_{\text{F}}^2$, for all H ∈ T (Condition 2 with c_inj = (32κ)⁻¹) hold simultaneously for all (X, Y) obeying

$$\text{max}\left\{{\Vert X-X^{\star }\Vert }_{2,\infty },{\Vert Y-Y^{\star }\Vert }_{2,\infty }\right\}\le C_{\infty }\kappa \left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n\text{log}n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right)\text{max}\left\{{\Vert X^{\star }\Vert }_{2,\infty },{\Vert Y^{\star }\Vert }_{2,\infty }\right\}.$$ (2.14)

Here, T denotes the tangent space of XY^⊤, and C_∞ > 0 is some absolute constant.

This lemma is a uniform result, namely, the bounds hold irrespective of the statistical dependency between (X, Y) and Ω. As a consequence, to demonstrate the proximity between the convex and the nonconvex solutions (cf. (2.12)), it remains to identify a point (X, Y) with vanishingly small gradient (cf. (2.11)) that is sufficiently close to the truth (cf. (2.14)).

As we already alluded to previously, a simple gradient descent algorithm aimed at solving the nonconvex problem (1.5) might help us produce an approximate nonconvex optimizer. This procedure is summarized in Algorithm 2.1. Our hope is this: when initialized at the ground truth and run for sufficiently many iterations, the GD trajectory produced by Algorithm 2.1 will contain at least one approximate stationary point of (1.5) with the desired properties (2.11) and (2.14). We shall note that Algorithm 2.1 is not practical since it starts from the ground truth (X^⋆, Y^⋆); this is an auxiliary step mainly to simplify the theoretical analysis. While we can certainly make it practical by adopting spectral initialization as in [19, 71], it requires more lengthy proofs without further improving our statistical guarantees.

2.5. Properties of the nonconvex iterates.

In this subsection, we will build upon the literature on nonconvex low-rank matrix completion to justify that the estimates returned by Algorithm 2.1 satisfy the requirement stated in (2.14). Our theory will be largely established upon the leave-one-out strategy introduced by Ma et al. [71], which is an effective analysis technique to control the ${\mathcal{l}}_{2,\infty }$ error of the estimates. This strategy has recently been extended by Chen et al. [19] to the more general rectangular case with an improved sample complexity bound.

Before continuing, we introduce several useful notations. Notice that the matrix product of X^⋆ and Y^⋆⊤ is invariant under global orthonormal transformation, namely, for any orthonormal matrix $R\in {ℝ}^{r\times r}$ one has X^⋆R(Y^⋆R)^⊤ = X^⋆Y^⋆⊤. Viewed in this light, we shall consider distance metrics modulo global rotation. In particular, the theory relies heavily on a specific global rotation matrix defined as follows

$$H^t\triangleq \text{arg}\underset{R\in {\mathcal{O}}^{r\times r}}{\text{min}}{\left({\Vert X^{t}R-X^{\star }\Vert }_{\text{F}}^2+{\Vert Y^{t}R-Y^{\star }\Vert }_{\text{F}}^2\right)}^{1/2},$$ (2.16)

where ${\mathcal{O}}^{r\times r}$ is the set of r × r orthonormal matrices.

We are now ready to present the performance guarantees for Algorithm 2.1.

Lemma 2.7 (Quality of the nonconvex estimates).

Instate the notation and hypotheses of Theorem 1.4. With probability at least 1 – O (n⁻³), the iterates ${\left\{\left(X^t,Y^t\right)\right\}}_{0\le t\le t_0}$ of Algorithm 2.1 satisfy

$$\text{max}\left\{{\Vert X^t{H}^t-X^{\star }\Vert }_{\text{F}},{\Vert Y^t{H}^t-Y^{\star }\Vert }_{\text{F}}\right\}\le C_{\text{F}}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert X^{\star }\Vert }_{\text{F}},$$ (2.17a)

$$\text{max}\left\{\Vert X^t{H}^t-X^{\star }\Vert ,\Vert Y^t{H}^t-Y^{\star }\Vert \right\}\le C_{\text{op}}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right)\Vert X^{\star }\Vert ,$$ (2.17b)

$$\text{max}\left\{{\Vert X^t{H}^t-X^{\star }\Vert }_{2,\infty },{\Vert Y^t{H}^t-Y^{\star }\Vert }_{2,\infty }\right\}\le C_{\infty }\kappa \left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n\text{log}n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right)\text{max}\left\{{\Vert X^{\star }\Vert }_{2,\infty },{\Vert Y^{\star }\Vert }_{2,\infty }\right\},$$ (2.17c)

$$\underset{0\le t<t_0}{\text{min}}{\Vert \nabla f\left(X^t,Y^t\right)\Vert }_{\text{F}}\le \frac{1}{n^5}\frac{\lambda }{p}\sqrt{{\sigma }_{\text{min}}},$$ (2.18)

where C_F, C_op, C_∞ > 0 are some absolute constants, provided that $\eta \asymp 1/\left(n{\kappa }^3{\sigma }_{\text{max}}\right)$ and that t₀ = n¹⁸.

This lemma, which we establish in Appendix SM4, reveals that for a polynomially large number of iterations, all iterates of the gradient descent sequence — when initialized at the ground truth — remain fairly close to the true low-rank factors. This holds in terms of the estimation errors measured by the Frobenius norm, the spectral norm, and the ${\mathcal{l}}_{2,\infty }$ norm. In particular, the proximity in terms of the ${\mathcal{l}}_{2,\infty }$ norm error plays a pivotal role in implementing our analysis strategy (particularly Lemmas 2.3–2.6) described previously. In addition, this lemma (cf. (2.18)) guarantees the existence of a small-gradient point within this sequence ${\left\{\left(X^t,Y^t\right)\right\}}_{0\le t\le t_0}$, a somewhat straightforward property of GD tailored to smooth problems [76]. This in turn enables us to invoke Lemma 2.3.

As immediate consequences of Lemma 2.7, with high probability we have

$${\Vert X^t{Y}^{t\top }-M^{\star }\Vert }_{\text{F}}\le 3\kappa C_{\text{F}}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert M^{\star }\Vert }_{\text{F}}$$ (2.19a)

$${\Vert X^t{Y}^{t\top }-M^{\star }\Vert }_{\infty }\le 3C_{\infty }\sqrt{{\kappa }^3\mu r}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n\text{log}n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert M^{\star }\Vert }_{\infty }$$ (2.19b)

$$\Vert X^t{Y}^{t\top }-M^{\star }\Vert \le 3C_{\text{op}}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right)\Vert M^{\star }\Vert$$ (2.19c)

for all 0 ≤ t ≤ t₀. The proof is deferred to Appendix SM4.12.

2.6. Proof of Theorem 1.4.

Let $t_{*}\triangleq \text{arg}{\text{min}}_{0\le t<t_0}{\Vert \nabla f\left(X^t,Y^t\right)\Vert }_{\text{F}}$, and take (X_ncvx, Y_ncvx) = (X^t* H^t*, Y^t* H^t* ) (cf. (2.16)). It is straightforward to verify that (X_ncvx, Y_ncvx) obeys (i) the small-gradient condition (2.11), and (ii) the proximity condition (2.14). We are now positioned to invoke Lemma 2.3: for any optimizer Z_cvx of (1.3), one has

$$\begin{gathered} {\Vert Z_{\text{cvx}}-X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }\Vert }_{\text{F}}\lesssim \frac{\kappa }{c_{\text{inj}}}\frac{1}{\sqrt{{\sigma }_{\text{min}}}}{\Vert \nabla f\left(X_{\text{ncvx}},Y_{\text{ncvx}}\right)\Vert }_{\text{F}}\lesssim \frac{{\kappa }^2}{n^5}\frac{\lambda }{p} \\ =\frac{\kappa }{n^5}\frac{\lambda }{p{\sigma }_{\text{min}}}\left(\kappa {\sigma }_{\text{min}}\right)=\frac{\kappa }{n^5}\frac{\lambda }{p{\sigma }_{\text{min}}}\Vert M^{\star }\Vert \\ \lesssim \frac{1}{n^4}\frac{\lambda }{p{\sigma }_{\text{min}}}\Vert M^{\star }\Vert . \end{gathered}$$ (2.20)

The last line arises since n ≫ κ — a consequence of the sample complexity condition np ≳ κ⁴μ²r² log³ n (and hence n ≥ np ≳ κ⁴μ²r² log³ n ≫ κ⁴). This taken collectively with the property (2.19) implies that

$$\begin{gathered} {\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\text{F}}\le {\Vert Z_{\text{cvx}}-X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }\Vert }_{\text{F}}+{\Vert X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }-M^{\star }\Vert }_{\text{F}} \\ \lesssim \frac{1}{n^4}\frac{\lambda }{p{\sigma }_{\text{min}}}\Vert M^{\star }\Vert +\kappa \left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert M^{\star }\Vert }_{\text{F}} \\ \asymp \kappa \left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert M^{\star }\Vert }_{\text{F}}. \end{gathered}$$

In other words, since X_ncvxY_ncvx^⊤ and Z_ncvx are exceedingly close, the error Z_cvx – M^⋆ is mainly accredited to $X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }-M^{\star }$. Similar arguments lead to

$$\Vert Z_{\text{cvx}}-M^{\star }\Vert \lesssim \left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right)\Vert M^{\star }\Vert ,$$

$${\Vert Z_{\text{cvx}}-M^{\star }\Vert }_{\infty }\lesssim \sqrt{{\kappa }^3\mu r}\left(\frac{\sigma }{{\sigma }_{\text{min}}}\sqrt{\frac{n\text{log}n}{p}}+\frac{\lambda }{p{\sigma }_{\text{min}}}\right){\Vert M^{\star }\Vert }_{\infty }.$$

We are left with proving the properties of Z_cvx,r. Since Z_cvx,r is defined to be the best rank-r approximation of Z_cvx, one can invoke (2.20) to derive

$${\Vert Z_{\text{cvx}}-Z_{\text{cvx},r}\Vert }_{\text{F}}\le {\Vert Z_{\text{cvx}}-X_{\text{ncvx}}{Y}_{\text{ncvx}}^{\top }\Vert }_{\text{F}}\lesssim \frac{1}{n^4}\frac{\lambda }{p{\sigma }_{\text{min}}}\Vert M^{\star }\Vert ,$$

from which (1.12) follows. Repeating the above calculations implies that (1.11) holds if Z_cvx is replaced by Z_cvx,r, thus concluding the proof.

3. Prior art.

Nuclear norm minimization, pioneered by the seminal works [14, 15, 45, 81], has been a popular and principled approach to low-rank matrix recovery. In the noiseless setting, i.e. E = 0, it amounts to solving the following constrained convex program

$${\text{minimize}}_{Z\in {ℝ}^n{}^{\times n}}\Vert Z{\Vert }_{*}\text{\hspace{1em} subject to \hspace{1em}}{\mathcal{P}}_{\Omega }(Z)={\mathcal{P}}_{\Omega }\left(M^{\star }\right),$$ (3.1)

which enjoys great theoretical success. Informally, this approach enables exact recovery of a rank-r matrix $M^{\star }\in {ℝ}^{n\times n}$ as soon as the sample size is about the order of nr — the intrinsic degrees of freedom of a rank-r matrix [20,51,80]. In particular, Gross [51] blazed a trail by developing an ingenious golfing scheme for dual construction — an analysis technique that has found applications far beyond matrix completion. When it comes to the noisy case, CandÚs and Plan [13] first studied the stability of convex programming when the noise is bounded and possibly adversarial, followed by [74] and [64] using two modified convex programs. As we have already discussed, none of these papers provide optimal statistical guarantees under our model when r = O(1). Other related papers such as [10,62] include similar estimation error bounds and suffer from similar sub-optimality issues.

Turning to nonconvex optimization, we note that this approach has recently received much attention for various low-rank matrix factorization problems, owing to its superior computational advantage compared to convex programming (e.g. [12, 22, 56, 60, 92, 98]). The convergence guarantees for matrix completion have been established for various algorithms such as gradient descent on manifold [60, 61], alternating minimization [53, 56], gradient descent [19, 71, 88, 95], and projected gradient descent [31], provided that a suitable initialization (like spectral initialization) is available [23, 56, 60, 71, 88]. Our work is mostly related to [19, 71], which studied (vanilla) gradient descent for nonconvex matrix completion. This algorithm was first analyzed by [71] via a leave-one-out argument — a technique that proves useful in analyzing various statistical algorithms [1, 26, 27, 35, 38, 67, 89, 102]. In the absence of noise and omitting logarithmic factors, [71] showed that O(nr³) samples are sufficient for vanilla GD to yield ε accuracy in $O\left(\text{log}\frac{1}{\epsilon }\right)$ iterations (without the need of extra regularization procedures); the sample complexity was further improved to O(nr²) by [19]. Apart from gradient descent, other nonconvex methods (e.g. [16, 35, 48, 52, 53, 55–57, 65, 82, 83, 93, 96, 97, 100]) and landscape / geometry properties have been investigated [18, 49, 50, 78, 84]; these are, however, beyond the scope of the current paper.

Another line of works asserted that a large family of SDPs admits low-rank solutions [4], which in turn motivates the Burer-Monteiro approach [6, 7]. When applied to matrix completion, however, the generic theoretical guarantees therein lead to conservative results. Take the noiseless case (3.1) for instance: these results revealed the existence of a solution of rank at most $O\left(\sqrt{n^{2}p}\right)$, which however is often much larger the true rank (e.g. when r ≍ 1 and p ≍ poly log(n)/n, one has $\sqrt{n^{2}p}\gg \sqrt{n}\gg r$). Moreover, this line of works does not imply that all solutions to the SDP of interest are (approximately) low-rank.

Finally, the connection between convex and nonconvex optimization has also been explored in line spectral estimation [66], although the context therein is drastically different from ours.

4. Discussion.

This paper provides an improved statistical analysis for the natural convex program (1.3), without the need of enforcing additional spikiness constraint. Our theoretical analysis uncovers an intriguing connection between convex relaxation and nonconvex optimization, which we believe is applicable to many other problems beyond matrix completion. Having said that, our current theory leaves open a variety of important directions for future exploration. Here we sample a few interesting ones.

• Improving dependency on r and κ. While our theory is optimal when r and κ are both constants, it becomes increasingly looser as either r or κ grows. For instance, in the noiseless setting, it has been shown that the sample complexity for convex relaxation scales as O(nr) — linear in r and independent of κ — which is better than the current results. It is worth noting that existing theory for nonconvex matrix factorization typically falls short of providing optimal scaling in r and κ [19, 31, 60, 71, 88]. Thus, tightening the dependency of sample complexity on r and κ might call for new analysis tools.

• Approximate low-rank structure. So far our theory is built upon the assumption that the ground-truth matrix M^⋆ is exactly low-rank, which falls short of accommodating the more realistic scenario where M^⋆ is only approximately low-rank. For the approximate low-rank case, it is not yet clear whether the nonconvex factorization approach can still serve as a tight proxy. In addition, the landscape of nonconvex optimization for the approximately low-rank case [18] might shed light on how to handle this case.

• Extension to deterministic noise. Our current theory — in particular, the leave-one-out analysis for the nonconvex approach — relies heavily on the randomness assumption (i.e. i.i.d. sub-Gaussian) of the noise. In order to justify the broad applicability of convex relaxation, it would be interesting to see whether one can generalize the theory to cover deterministic noise with bounded magnitudes.

• Extension to structured matrix completion. Many applications involve low-rank matrices that exhibit additional structures, enabling a further reduction of the sample complexity [9, 24, 47]. For instance, if a matrix is Hankel and low-rank, then the sample complexity can be O(n) times smaller than the generic low-rank case. The existing stability guarantee of Hankel matrix completion, however, is overly pessimistic compared to practical performance [24]. The analysis framework herein might be amenable to the study of Hankel matrix completion and help close the theory-practice gap.

• Extension to robust PCA and blind deconvolution. Moving beyond matrix completion, there are other problems that are concerned with recovering low-rank matrices. Notable examples include robust principal component analysis [11, 17, 30], blind deconvolution [2, 68] and blind demixing [59, 69]. The stability analyses of the convex relaxation approaches for these problems [2, 69, 103] often adopt a similar approach as [13], and consequently are sub-optimal. The insights from the present paper might promise tighter statistical guarantees for such problems.

Finally, we remark that the intimate link between convex and nonconvex optimization enables statistically optimal inference and uncertainty quantification for noisy matrix completion (e.g. construction of optimal confidence intervals for each missing entry). The interested readers are referred to our companion paper [28] for in-depth discussions.