Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrieval

Abstract

This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest ${\mathit{x}}^{♮}\in {ℝ}^n$ from m quadratic equations/samples $y_i={\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2,1\le i\le m$. This problem, also dubbed as phase retrieval, spans multiple domains including physical sciences and machine learning.

We investigate the efficacy of gradient descent (or Wirtinger flow) designed for the nonconvex least squares problem. We prove that under Gaussian designs, gradient descent — when randomly initialized — yields an ϵ-accurate solution in O(log n + log(1/ϵ)) iterations given nearly minimal samples, thus achieving near-optimal computational and sample complexities at once. This provides the first global convergence guarantee concerning vanilla gradient descent for phase retrieval, without the need of (i) carefully-designed initialization, (ii) sample splitting, or (iii) sophisticated saddle-point escaping schemes. All of these are achieved by exploiting the statistical models in analyzing optimization algorithms, via a leave-one-out approach that enables the decoupling of certain statistical dependency between the gradient descent iterates and the data.

1. Introduction

Suppose we are interested in learning an unknown object ${\mathit{x}}^{♮}\in {ℝ}^n$, but only have access to a few quadratic equations of the form

$$y_i={\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2,1\le i\le m,$$ (1)

where y_i is the sample we collect and a_i is the design vector known a priori. Is it feasible to reconstruct x^♮ in an accurate and efficient manner?

The problem of solving systems of quadratic equations (1) is of fundamental importance and finds applications in numerous contexts. Perhaps one of the best-known applications is the so-called phase retrieval problem arising in physical sciences [CESV13,SEC⁺15]. In X-ray crystallography, due to the ultra-high frequency of the X-rays, the optical sensors and detectors are incapable of recording the phases of the diffractive waves; rather, only intensity measurements are collected. The phase retrieval problem comes down to reconstructing the specimen of interest given intensity-only measurements. If one thinks of x^♮ as the specimen under study and uses {y_i} to represent the intensity measurements, then phase retrieval is precisely about inverting the quadratic system (1).

Moving beyond physical sciences, the above problem also spans various machine learning applications. One example is mixed linear regression, where one wishes to estimate two unknown vectors β₁ and β₂ from unlabeled linear measurements [CYC14]. The acquired data {a_i, b_i}_1≤i≤m take the form of either $b_i\approx a_i^{\top }{\beta }_1$ or $b_i\approx a_i^{\top }{\beta }_2$, without knowing which of the two vectors generates the data. In a simple symmetric case with β₁ = β₂ = x^♮ (so that $b_i\approx \pm a_i^{\top }{\mathit{x}}^{♮}$), the squared measurements $y_i=b_i^2\approx {\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2$ become the sufficient statistics, and hence mixed linear regression can be converted to learning x^♮ from {a_i, y_i}. Furthermore, the quadratic measurement model in (1) allows to represent a single neuron associated with a quadratic activation function, where {a_i, y_i} are the data and x^♮ encodes the parameters to be learned. As described in [SJL17, LMZ17], _{learning neural nets with quadratic activations} involves solving systems of quadratic equations.

1.1. Nonconvex optimization via gradient descent

A natural strategy for inverting the system of quadratic equations (1) is to solve the following nonconvex least squares estimation problem

$${\mathrm{minimize}}_{\mathit{x}\in {ℝ}^n}f(\mathit{x}):=\frac{1}{4m}\sum _{i=1}^m{\left[{\left(a_i^{\top }\mathit{x}\right)}^2-y_i\right]}^2.$$ (2)

Under Gaussian designs where $a_i\stackrel{\text{i.i.d}}{\sim }\mathcal{N}\left(\mathbf{0},{\mathit{I}}_n\right)$, the solution to (2) is known to be exact — up to some global sign — with high probability, as soon as the number m of equations (samples) exceeds the order of the number n of unknowns [BCMN14]. However, the loss function in (2) is highly nonconvex, thus resulting in severe computational challenges. With this issue in mind, can we still hope to find the global minimizer of (2) via low-complexity algorithms which, ideally, run in time proportional to that taken to read the data?

Fortunately, in spite of nonconvexity, a variety of optimization-based methods are shown to be effective in the presence of proper statistical models. Arguably, one of the simplest algorithms for solving (2) is vanilla gradient descent (GD), which attempts recovery via the update rule

$${\mathit{x}}^{t+1}={\mathit{x}}^t-{\eta }_t\nabla f\left({\mathit{x}}^t\right),t=0,1,\cdots$$ (3)

with η_t being the stepsize/learning rate. The above iterative procedure is also dubbed Wirtinger flow for phase retrieval, which can accommodate the complex-valued case as well [CLS15]. This simple algorithm is remarkably efficient under Gaussian designs: in conjunction with carefully-designed initialization and stepsize rules, GD provably converges to the truth x^♮ at a linear rate¹, provided that the ratio m/n of the number of equations to the number of unknowns exceeds some logarithmic factor [CLS15,Sol14,MWCC17].

One crucial element in prior convergence analysis is initialization. In order to guarantee linear convergence, prior works typically recommend spectral initialization or its variants [CLS15,CC17,WGE17,ZZLC17,MWCC17,LL17,MM17]. Specifically, the spectral method forms an initial estimate x⁰ using the (properly scaled) leading eigenvector of a certain data matrix. Two important features are worth emphasizing:

• x⁰ convexity; falls within a local ℓ₂-ball surrounding x^♮ with a reasonably small radius, where f(·) enjoys strong convexity;

• x⁰ is incoherent with all the design vectors {a_i} — in the sense that $\left|a_i^{\top }{\mathit{x}}^0\right|$ is reasonably small for all 1 ≤ i ≤ m — and hence x⁰ falls within a region where f(·) enjoys desired smoothness conditions.

These two properties taken collectively allow gradient descent to converge rapidly from the very beginning.

1.2. Random initialization?

The enormous success of spectral initialization gives rise to a curious question: is carefully-designed initialization necessary for achieving fast convergence? Obviously, vanilla GD cannot start from arbitrary points, since it may get trapped in undesirable stationary points (e.g. saddle points). However, is there any simpler initialization approach that avoids such stationary points and works equally well as spectral initialization?

A strategy that practitioners often like to employ is to initialize GD randomly. The advantage is clear: compared with spectral methods, random initialization is model-agnostic and is usually more robust visa-vis model mismatch. Despite its wide use in practice, however, GD with random initialization is poorly understood in theory. One way to study this method is through a geometric lens [SQW16]: under Gaussian designs, the loss function f(·) (cf. (2)) does not have any spurious local minima as long as the sample size m is on the order of n log³ n. Moreover, all saddle points are strict [GHJY15], meaning that the associated Hessian matrices have at least one negative eigenvalue if they are not local minima. Armed with these two conditions, the theory of Lee et al. [LSJR16] implies that vanilla GD converges almost surely to the truth. However, the convergence rate remains unsettled. In fact, we are not aware of any theory that guarantees polynomial-time convergence of vanilla GD for phase retrieval in the absence of carefully-designed initialization.

Motivated by this, we aim to pursue a formal understanding about the convergence properties of GD with random initialization. Before embarking on theoretical analyses, we first assess its practical efficiency through numerical experiments. Generate the true object x^♮ and the initial guess x⁰ randomly as

$${\mathit{x}}^{♮}\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right)\text{and}{\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right).$$

We vary the number n of unknowns (i.e. n = 100, 200, 500, 800, 1000), set m = 10n, and take a constant stepsize η_t ≡ 0.1. Here the measurement vectors are generated from Gaussian distributions, i.e. $a_i\stackrel{\text{i.i.d.}}{\sim }\mathcal{N}\left(\mathbf{0},{\mathit{I}}_n\right)$ for 1 ≤ i ≤ m. The relative ℓ₂ errors dist(x^t, x^♮)/∥x^♮∥₂ of the GD iterates in a random trial are plotted in Figure 1, where

$$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right):=\min \left\{{\Vert {\mathit{x}}^t-{\mathit{x}}^{♮}\Vert }_2,{\Vert {\mathit{x}}^t+{\mathit{x}}^{♮}\Vert }_2\right\}$$ (4)

represents the ℓ₂ distance between x^t and x^♮ modulo the unrecoverable global sign.

Figure 1:

The relative ℓ₂ error vs. iteration count for GD with random initialization, plotted semilogarithmically. The results are shown for n = 100, 200, 500, 800, 1000 with m = 10n and η_t ≡ 0.1.

In all experiments carried out in Figure 1, we observe two stages for GD: (1) Stage 1: the relative error of x^t stays nearly flat; (2) Stage 2: the relative error of x^t experiences geometric decay. Interestingly, Stage 1 lasts only for a few tens of iterations. These numerical findings taken together reveal appealing computational efficiency of GD in the presence of random initialization — it attains 5-digit accuracy within about 200 iterations!

To further illustrate this point, we take a closer inspection of the signal component ⟨x^t, x^♮⟩x^♮ and the orthogonal component x^t − ⟨x^t, x^♮⟩x^♮, where we normalize ${\Vert {\mathit{x}}^{♮}\Vert }_2=1$ for simplicity. Denote by ${\Vert {\mathit{x}}_{\perp }^t\Vert }_2$ the ℓ₂ norm of the orthogonal component. We highlight two important and somewhat surprising observations that allude to why random initialization works.

• The strength ratio of the signal to the orthogonal components grows exponentially. The ratio, $\left|\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle \right|/{\Vert {\mathit{x}}_{\perp }^t\Vert }_2$, grows exponentially fast throughout the execution of the algorithm, as demonstrated in Figure 2(a). This metric $\left|\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle \right|/{\Vert {\mathit{x}}_{\perp }^t\Vert }_2$ in some sense captures the signal-to-noise ratio of the running iterates.

• Exponential growth of the signal strength in Stage 1. While the ℓ₂ estimation error of x^t may not drop significantly during Stage 1, the size |⟨x^t, x^♮⟩| of the signal component increases exponentially fast and becomes the dominant component within several tens of iterations, as demonstrated in Figure 2(b). This helps explain why Stage 1 lasts only for a short duration.

Figure 2:

(a) The ratio $\left|\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle \right|/{\Vert {\mathit{x}}_{\perp }^t\Vert }_2$, and (b) the size $\left|\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle \right|$ of the signal component and the ℓ₂ error vs. iteration count, both plotted on semilogarithmic scales. The results are shown for n = 100, 200, 500, 800, 1000 with m = 10n, η_t ≡ 0.1, and ${\Vert {\mathit{x}}^{♮}\Vert }_2=1$.

The central question then amounts to whether one can develop a mathematical theory to interpret such intriguing numerical performance. In particular, how many iterations does Stage 1 encompass, and how fast can the algorithm converge in Stage 2?

1.3. Main findings

The objective of the current paper is to demystify the computational efficiency of GD with random initialization, thus bridging the gap between theory and practice. Assuming a tractable random design model in which a_i’s follow Gaussian distributions, our main findings are summarized in the following theorem. Here and throughout, the notation f(n) ≲ g(n) or f(n) = O(g(n)) (resp. f(n) ≳ g(n), f(n) ≍ g(n)) means that there exist constants c₁, c₂ > 0 such that f(n) ≤ c₁g(n) (resp. f(n) ≥ c₂g(n), c₁g(n) ≤ f(n) ≤ c₂g(n)).

Theorem 1

Fix ${\mathit{x}}^{♮}\in {ℝ}^n$ with ∥x^♮∥₂ = 1. Suppose that $a_i\stackrel{\text{i.i.d}.}{\sim }\mathcal{N}\left(\mathbf{0},{\mathit{I}}_n\right)$ for 1 ≤ i ≤ m, ${\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right)$, and ${\eta }_t\equiv \eta =c/{\Vert {\mathit{x}}^{♮}\Vert }_2^2$ for some sufficiently small constant c > 0. Then with probability approaching one, there exist some sufficiently small constant 0 < γ < 1 and T_γ ≲ log n such that the GD iterates (3) obey

$$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le \gamma {(1-\rho )}^{t-T_{\gamma }},\forall t\ge T_{\gamma }$$

for some absolute constant 0 < ρ < 1, provided that the sample size m ≳ n poly log(m).

Remark 1

The readers are referred to Theorem 2 for a more general statement.

Here, the stepsize is taken to be a fixed constant throughout all iterations, and we reuse the same data across all iterations (i.e. no sample splitting is needed to establish this theorem). The GD trajectory is divided into 2 stages: (1) Stage 1 consists of the first T_γ iterations, corresponding to the first tens of iterations discussed in Section 1.2; (2) Stage 2 consists of all remaining iterations, where the estimation error contracts linearly. Several important implications/remarks follow immediately.

• Stage 1 takes O(log n) iterations. When seeded with a random initial guess, GD is capable of entering a local region surrounding x^♮ within T_γ ≲ log n iterations, namely,

  $$\mathrm{dist}\left({\mathit{x}}^T{}^{{}_{\gamma }},{\mathit{x}}^{♮}\right)\le \gamma$$

  for some sufficiently small constant γ > 0. Even though Stage 1 may not enjoy linear convergence in terms of the estimation error, it is of fairly short duration.

• Stage 2 takes O(log (1/ϵ)) iterations. After entering the local region, GD converges linearly to the ground truth x^♮ with a contraction rate 1−ρ. This tells us that GD reaches ϵ-accuracy (in a relative sense) within O(log (1/ϵ)) iterations.

• Near linear-time computational complexity. Taken collectively, these imply that the iteration complexity of GD with random initialization is

  $$O\left(\log n+\log \frac{1}{ϵ}\right).$$

  Given that the cost of each iteration mainly lies in calculating the gradient ∇f(x^t), the whole algorithm takes nearly linear time, namely, it enjoys a computational complexity proportional to the time taken to read the data (modulo some logarithmic factor).

• Near-minimal sample complexity. The preceding computational guarantees occur as soon as the sample size exceeds m ≳ n poly log(m). Given that one needs at least n samples to recover n unknowns, the sample complexity of randomly initialized GD is optimal up to some logarithmic factor.

• Saddle points? The GD iterates never hit the saddle points (see Figure 3 for an illustration). In fact, after a constant number of iterations at the very beginning, GD will follow a path that increasingly distances itself from the set of saddle points as the algorithm progresses. There is no need to adopt sophisticated saddle-point escaping schemes developed in generic optimization theory (e.g. cubic regularization [NP06], perturbed GD [JGN⁺17]).

• Weak dependency w.r.t. the design vectors. As we will elaborate in Section 4, the statistical dependency between the GD iterates {x^t} and certain components of the design vectors {a_i} stays at an exceedingly weak level. Consequently, the GD iterates {x^t} proceed as if fresh samples were employed in each iteration. This statistical observation plays a crucial role in characterizing the dynamics of the algorithm without the need of sample splitting.

Figure 3:

The trajectory of (α_t, β_t), where α_t = |⟨x^t, x^♮⟩| and β_t = ∥x^t − ⟨x^t, x^♮⟩x^♮∥₂ represent respectively the size of the signal component and that of the orthogonal component of the GD iterates (assume ∥x^♮∥₂ = 1). (a) The results are shown for n = 1000 with m = 10n, and η_t = 0.01, 0.05, 0.1. (b) The results are shown for n = 1000 with m approaching infinity, and η_t = 0.01, 0.05, 0.1. The blue filled circles represent the population-level saddle points, and the orange arrows indicate the directions of increasing t.

It is worth emphasizing that the entire trajectory of GD is automatically confined within a certain region enjoying favorable geometry. For example, the GD iterates are always incoherent with the design vectors, stay sufficiently away from any saddle point, and exhibit desired smoothness conditions, which we will formalize in Section 4. Such delicate geometric properties underlying the GD trajectory are not explained by prior papers. In light of this, convergence analysis based on global geometry [SQW16] — which provides valuable insights into algorithm designs with arbitrary initialization — results in suboptimal (or even pessimistic) computational guarantees when analyzing a specific algorithm like GD. In contrast, the current paper establishes near-optimal performance guarantees by paying particular attention to finer dynamics of the algorithm. As will be seen later, this is accomplished by heavily exploiting the statistical properties in each iterative update.

2. Why random initialization works?

Before diving into the proof of the main theorem, we pause to develop intuitions regarding why gradient descent with random initialization is expected to work. We will build our understanding step by step: (i) we first investigate the dynamics of the population gradient sequence (the case where we have infinite samples); (ii) we then turn to the finite-sample case and present a heuristic argument assuming independence between the iterates and the design vectors; (iii) finally, we argue that the true trajectory is remarkably close to the one heuristically analyzed in the previous step, which arises from a key property concerning the “near-independence” between {x^t} and the design vectors {a_i}.

Without loss of generality, we assume x^♮ = e₁ throughout this section, where e₁ denotes the first standard basis vector. For notational simplicity, we denote by

$${\mathit{x}}_{\Vert }^t:={\mathit{x}}_1^t\text{and}{\mathit{x}}_{\perp }^t:={\left[{\mathit{x}}_i^t\right]}_{2\le i\le n}$$ (5)

the first entry and the 2nd through the nth entries of x^t, respectively. Since x^♮ = e₁, it is easily seen that

$$\underset{\text{signal}\text{component}}{\underbrace{x_{\Vert }^t{e}_1=\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle {\mathit{x}}^{♮}}}\text{and}\underset{\text{orthogonal}\text{component}}{\underbrace{\left[\begin{array}{c}0\\ {\mathit{x}}_{\perp }^t\end{array}\right]={\mathit{x}}^t-\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle {\mathit{x}}^{♮}}}$$ (6)

represent respectively the components of x^t along and orthogonal to the signal direction. In what follows, we focus our attention on the following two quantities that reflect the sizes of the preceding two components²

$${\alpha }_t:={\mathit{x}}_{\Vert }^t\text{and}{\beta }_t:={\Vert {\mathit{x}}_{\perp }^t\Vert }_2.$$ (7)

Without loss of generality, assume that α₀ > 0.

2.1. Population dynamics

To start with, we consider the unrealistic case where the iterates {x^t} are constructed using the population gradient (or equivalently, the gradient when the sample size m approaches infinity), i.e.

$${\mathit{x}}^{t+1}={\mathit{x}}^t-\eta \nabla F\left({\mathit{x}}^t\right).$$

Here, ∇F(x) represents the population gradient given by

$$\nabla F(\mathit{x}):=\left(3\Vert \mathit{x}{\Vert }_2^2-1\right)\mathit{x}-2\left({\mathit{x}}^{♮\top }\mathit{x}\right){\mathit{x}}^{♮},$$

which can be computed by $\nabla F(\mathit{x})=\mathbb{E}[\nabla f(\mathit{x})]=\mathbb{E}\left[\left\{\left(a_i^{\top }\mathit{x}\right){\mathit{x}}^2-{\left({\mathbf{a}}_i^{\top }{\mathit{x}}^{♮}\right)}^2\right\}{a}_i{a}_i^{\top }\mathit{x}\right]$ assuming that x and the a_i’s are independent. Simple algebraic manipulation reveals the dynamics for both the signal and the orthogonal components:

$${\mathit{x}}_{\Vert }^{t+1}=\left\{1+3\eta \left(1-{\Vert {\mathit{x}}^t\Vert }_2^2\right)\right\}{x}_{\Vert }^t;$$ (8a)

$${\mathit{x}}_{\perp }^{t+1}=\left\{1+\eta \left(1-3{\Vert {\mathit{x}}^t\Vert }_2^2\right)\right\}{\mathit{x}}_{\perp }^t.$$ (8b)

Assuming that η is sufficiently small and recognizing that ${\Vert {\mathit{x}}^t\Vert }_2^2={\alpha }_t^2+{\beta }_t^2$, we arrive at the following population-level state evolution for both α_t and β_t (cf. (7)):

$${\alpha }_{t+1}=\left\{1+3\eta \left[1-\left({\alpha }_t^2+{\beta }_t^2\right)\right]\right\}{\alpha }_t;$$ (9a)

$${\beta }_{t+1}=\left\{1+\eta \left[1-3\left({\alpha }_t^2+{\beta }_t^2\right)\right]\right\}{\beta }_t.$$ (9b)

This recursive system has three fixed points:

$$(\alpha ,\beta )=(1,0),(\alpha ,\beta )=(0,0),\text{and}(\alpha ,\beta )=(0,1/\sqrt{3}),$$

which correspond to the global minimizer, the local maximizer, and the saddle points, respectively, of the population objective function.

We make note of the following key observations in the presence of a randomly initialized x⁰, which will be formalized later in Lemma 1:

• the ratio α_t/β_t of the size of the signal component to that of the orthogonal component increases exponentially fast;

• the size α_t of the signal component keeps growing until it plateaus around 1;

• the size β_t of the orthogonal component eventually drops towards zero.

In other words, when randomly initialized, (α^t, β^t) converges to (1, 0) rapidly, thus indicating rapid convergence of x^t to the truth x^♮, without getting stuck at any undesirable saddle points. We also illustrate these phenomena numerically. Set n = 1000, η_t ≡ 0.1 and ${\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right)$. Figure 4 displays the dynamics of α_t/β_t, α_t, and β_t, which are precisely as discussed above.

Figure 4:

Population-level state evolution, plotted semilogarithmically: (a) the ratio α_t/β_t vs. iteration count, and (b) α_t and β_t vs. iteration count. The results are shown for n = 1000, η_t ≡ 0.1, and ${\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right)$ (assuming α₀ > 0 though).

2.2. Finite-sample analysis: a heuristic treatment

We now move on to the finite-sample regime, and examine how many samples are needed in order for the population dynamics to be reasonably accurate. Notably, the arguments in this subsection are heuristic in nature, but they are useful in developing insights into the true dynamics of the GD iterates.

Rewrite the gradient update rule (3) as

$${\mathit{x}}^{t+1}={\mathit{x}}^t-\eta \nabla f\left({\mathit{x}}^t\right)={\mathit{x}}^t-\eta \nabla F\left({\mathit{x}}^t\right)-\eta \underset{:=r\left({\mathit{x}}^t\right)}{\underbrace{\left(\nabla f\left({\mathit{x}}^t\right)-\nabla F\left({\mathit{x}}^t\right)\right)}},$$ (10)

where $\nabla f(\mathit{x})=m^{-1}{\sum }_{i=1}^m\left[{\left(a_i^{\top }\mathit{x}\right)}^2-{\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2\right]a_i{a}_i^{\top }\mathit{x}$. Assuming (unreasonably) that the iterate x^t is independent of {a_i}, the central limit theorem (CLT) allows us to control the size of the fluctuation term r(x^t). Take the signal component as an example: simple calculations give

$$x_{\Vert }^{t+1}=x_{\Vert }^t-\eta {\left(\nabla F\left({\mathit{x}}^t\right)\right)}_1-\eta r_1\left({\mathit{x}}^t\right),$$

where

$$r_1(\mathit{x}):=\frac{1}{m}\sum _{i=1}^m\left[{\left(a_i^{\top }\mathit{x}\right)}^3-a_{i,1}^2\left(a_i^{\top }\mathit{x}\right)\right]a_{i,1}-\mathbb{E}\left[\left\{{\left(a_i^{\top }\mathit{x}\right)}^3-a_{i,1}^2\left(a_i^{\top }\mathit{x}\right)\right\}{a}_{i,1}\right]$$ (11)

with a_i,1 the first entry of a_i. Owing to the preceding independence assumption, r₁ is the sum of m i.i.d. zero-mean random variables. Assuming that x^t never blows up so that ∥x^t∥₂ = O(1), one can apply the CLT to demonstrate that

$$\left|r_1\left({\mathit{x}}^t\right)\right|\lesssim \sqrt{\mathrm{Var}\left(r_1\left({\mathit{x}}^t\right)\right)\mathrm{poly}\log (m)}\lesssim \sqrt{\frac{\mathrm{poly}\log (m)}{m}}$$ (12)

with high probability, which is often negligible compared to the other terms. For instance, for the random initial guess ${\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\mathit{I}}_n\right)$ one has $\left|{\mathit{x}}_{\Vert }^0\right|\gtrsim 1/\sqrt{n\log n}$ with probability approaching one, telling us that

$$\left|r_1\left({\mathit{x}}^0\right)\right|\lesssim \sqrt{\frac{\mathrm{poly}\log (m)}{m}}\ll \left|{\mathit{x}}_{\Vert }^0\right|$$

as long as m ≳ n poly log(m). This combined with the fact that $\left|{\mathit{x}}_{\Vert }^0-\eta {\left(\nabla F\left({\mathit{x}}^0\right)\right)}_1\right|\asymp \left|{\mathit{x}}_{\Vert }^0\right|$ reveals $\left|r_1\left({\mathit{x}}^0\right)\right|\lesssim \left|{\mathit{x}}_{\Vert }^0-\eta {\left(\nabla F\left({\mathit{x}}^0\right)\right)}_1\right|$. Similar observations hold true for the orthogonal component ${\mathit{x}}_{\perp }^t$.

In summary, by assuming independence between x^t and {a_i}, we arrive at an approximate state evolution for the finite-sample regime:

$${\alpha }_{t+1}\approx \left\{1+3\eta \left[1-\left({\alpha }_t^2+{\beta }_t^2\right)\right]\right\}{\alpha }_t;$$ (13a)

$${\beta }_{t+1}\approx \left\{1+\eta \left[1-3\left({\alpha }_t^2+{\beta }_t^2\right)\right]\right\}{\beta }_t,$$ (13b)

with the proviso that m ≳ n poly log(m).

2.3. Key analysis ingredients: near-independence and leave-one-out tricks

The preceding heuristic argument justifies the approximate validity of the population dynamics, under an independence assumption that never holds unless we use fresh samples in each iteration. On closer inspection, what we essentially need is the fluctuation term r(x^t) (cf. (10)) being well-controlled. For instance, when focusing on the signal component, one need $\left|r_1\left({\mathit{x}}^t\right)\right|\ll \left|{\mathit{x}}_{\Vert }^t\right|$ for all t ≥ 0. In particular, in the beginning iterations, $\left|{\mathit{x}}_{\Vert }^t\right|$ is as small as $O(1/\sqrt{n})$. Without the independence assumption, the CLT types of results fail to hold due to the complicated dependency between x^t and {a_i}. In fact, one can easily find many points that result in much larger remainder terms (as large as O(1)) and that violate the approximate state evolution (13). See Figure 5 for a caricature of the region where the fluctuation term r(x^t) is well-controlled. As can be seen, it only occupies a tiny fraction of the neighborhood of x^♮.

Figure 5:

Illustration of the region satisfying the “near-independence” property. Here, the green arrows represent the directions of {a_i}_1≤i≤20, and the blue region consists of all points such that the first entry r₁ (x) of the fluctuation r(x) = ∇f(x) − ∇F(x) is bounded above in magnitude by $\left|{\mathit{x}}_{\Vert }^t\right|/5\left(\text{or}\left|\langle \mathit{x},{\mathit{x}}^{♮}\rangle \right|/5\right)$.

Fortunately, despite the complicated dependency across iterations, one can provably guarantee that x^t always stays within the preceding desirable region in which r(x^t) is well-controlled. The key idea is to exploit a certain “near-independence” property between {x^t} and {a_i}. Towards this, we make use of a leave-one-out trick proposed in [MWCC17] for analyzing nonconvex iterative methods. In particular, we construct auxiliary sequences that are

1. independent of certain components of the design vectors {a_i}; and

2. extremely close to the original gradient sequence {x^t}_t≥0.

As it turns out, we need to construct several auxiliary sequences {x^t,(l)}_t≥0, {x^t,sgn}_t≥0 and {x^t,sgn,(l)}_t≥0, where {x^t,(l)}_t≥0 is independent of the lth sampling vector a_l, {x^t,sgn}_t≥0 is independent of the sign information of the first entries of all a_i’s, and {x^t;sgn;(l)} is independent of both. In addition, these auxiliary sequences are constructed by slightly perturbing the original data (see Figure 6 for an illustration), and hence one can expect all of them to stay close to the original sequence throughout the execution of the algorithm. Taking these two properties together, one can propagate the above statistical independence underlying each auxiliary sequence to the true iterates {x^t}, which in turn allows us to obtain near-optimal control of the fluctuation term r(x^t). The details are postponed to Section 4.

Figure 6:

Illustration of the leave-one-out and random-sign sequences. (a) {x^t} is constructed using all data {a_i, y_i}; (b) {x^t,(l)} is constructed by discarding the lth sample {a_l, y_l}; (c) {x^t,sgn} is constructed by using auxiliary design vectors $\left\{a_i^{\text{sgn}}\right\}$, where $a_i^{\text{sgn}}$ is obtained by randomly flipping the sign of the first entry of a_i; (d) {x^t,sgn,(l)} is constructed by discarding the lth sample $\left\{a_l^{\text{sgn}},y_l\right\}$.

3. Related work

Solving systems of quadratic equations, or phase retrieval, has been studied extensively in the recent literature; see [SEC⁺15] for an overview. One popular method is convex relaxation (e.g. PhaseLift [CSV13]), which is guaranteed to work as long as m/n exceeds some large enough constant [CL14,DH14,CCG15,CZ15,KRT17]. However, the resulting semidefinite program is computationally prohibitive for solving large-scale problems. To address this issue, [CLS15] proposed the Wirtinger flow algorithm with spectral initialization, which provides the first convergence guarantee for nonconvex methods without sample splitting. Both the sample and computation complexities were further improved by [CC17] with an adaptive truncation strategy. Other nonconvex phase retrieval methods include [NJS13,CLM16,Sol17,WGE17,ZZLC17,WGSC17,CL16,DR17,GX16,CFL15,Wei15,BEB17,TV17,CLW17,ZWGC17,QZEW17,ZCL16,YYF⁺17,CWZG17,Zha17,MXM18,CLC18]. Almost all of these nonconvex methods require carefully-designed initialization to guarantee a sufficiently accurate initial point. One exception is the approximate message passing algorithm proposed in [MXM18], which works as long as the correlation between the truth and the initial signal is bounded away from zero. This, however, does not accommodate the case when the initial signal strength is vanishingly small (like random initialization). Other works [Zha17,LGL15] explored the global convergence of alternating minimization/projection with random initialization which, however, require fresh samples at least in each of the first O(log n) iterations in order to enter the local basin. In addition, [LMZ17] explored low-rank recovery from quadratic measurements with near-zero initialization. Using a truncated least-squares objective, [LMZ17] established approximate (but non-exact) recovery of over-parametrized GD. Notably, if we do not over-parametrize the phase retrieval problem, then GD with near-zero initialization is (nearly) equivalent to running the power method for spectral initialization³, which can be understood using prior theory.

Another related line of research is the design of generic saddle-point escaping algorithms, where the goal is to locate a second-order stationary point (i.e. the point with a vanishing gradient and a positive-semidefinite Hessian). As mentioned earlier, it has been shown by [SQW16] that as soon as m ≫ log³ n, all local minima are global and all the saddle points are strict. With these two geometric properties in mind, saddle-point escaping algorithms are guaranteed to converge globally for phase retrieval. Existing saddle-point escaping algorithms include but are not limited to Hessian-based methods [NP06,SQW16] (see also [AAZB⁺16,AZ17,JGN⁺17] for some reviews), noisy stochastic gradient descent [GHJY15], perturbed gradient descent [JGN⁺17], and normalized gradient descent [MSK17]. On the one hand, the results developed in these works are fairly general: they establish polynomial-time convergence guarantees under a few generic geometric conditions. On the other hand, the iteration complexity derived therein may be pessimistic when specialized to a particular problem.

Take phase retrieval and the perturbed gradient descent algorithm [JGN⁺17] as an example. It has been shown in [JGN⁺17, Theorem 5] that for an objective function that is L-gradient Lipschitz, ρ-Hessian Lipschitz, (θ, γ, ζ)-strict saddle, and also locally α-strongly convex and β-smooth (see definitions in [JGN⁺17]), it takes⁴

$$O\left(\frac{L}{{\left[\min \left(\theta ,{\gamma }^2/\rho \right)\right]}^2}+\frac{\beta }{\alpha }\log \frac{1}{ϵ}\right)=O\left(n^3+n\log \frac{1}{ϵ}\right)$$

iterations (ignoring logarithmic factors) for perturbed gradient descent to converge to ϵ-accuracy. In fact, even with Nesterov’s accelerated scheme [JNJ17], the iteration complexity for entering the local region is at least

$$O\left(\frac{L^{1/2}{\rho }^{1/4}}{{\left[\min \left(\theta ,{\gamma }^2/\rho \right)\right]}^{7/4}}\right)=O\left(n^{2.5}\right).$$

Both of them are much larger than the O(log n + log(1/ϵ)) complexity established herein. This is primarily due to the following facts: (i) the Lipschitz constants of both the gradients and the Hessians are quite large, i.e. L ≍ n and ρ ≍ n (ignoring log factors), which are, however, treated as dimension-independent constants in the aforementioned papers; (ii) the local condition number is also large, i.e. β/α ≍ n. In comparison, as suggested by our theory, the GD iterates with random initialization are always confined within a restricted region enjoying much more benign geometry than the worst-case / global characterization.

Furthermore, the above saddle-escaping first-order methods are often more complicated than vanilla GD. Despite its algorithmic simplicity and wide use in practice, the convergence rate of GD with random initialization remains largely unknown. In fact, Du et al. [DJL⁺17] demonstrated that there exist non-pathological functions such that GD can take exponential time to escape the saddle points when initialized randomly. In contrast, as we have demonstrated, saddle points are not an issue for phase retrieval; the GD iterates with random initialization never get trapped in the saddle points.

Finally, the leave-one-out arguments have been invoked to analyze other high-dimensional statistical inference problems including robust M-estimators [EKBB⁺13,EK15], and maximum likelihood theory for logistic regression [SCC18], etc. In addition, [ZB17,CFMW17,AFWZ17] made use of the leave-one-out trick to derive entrywise perturbation bounds for eigenvectors resulting from certain spectral methods. The techniques have also been applied by [MWCC17,LMCC18] to establish local linear convergence of vanilla GD for nonconvex statistical estimation problems in the presence of proper spectral initialization.

4. Analysis

In this section, we first provide a more general version of Theorem 1 as follows. It spells out exactly the conditions on x⁰ in order for vanilla GD with random initialization to succeed.

Theorem 2

Fix ${\mathit{x}}^{♮}\in {ℝ}^n$. Suppose $a_i\stackrel{\text{i.i.d}.}{\sim }\mathcal{N}\left(\mathbf{0},{\mathit{I}}_n\right)(1\le i\le m)$ and m ≥ Cn log¹³m for some sufficiently large constant C > 0. Assume that the initialization x⁰ is independent of {a_i} and obeys

$$\frac{\left|\langle {\mathit{x}}^0,{\mathit{x}}^{♮}\rangle \right|}{{\Vert {\mathit{x}}^{♮}\Vert }_2^2}\ge \frac{1}{\sqrt{n\log n}}and\left(1-\frac{1}{\log n}\right){\Vert {\mathit{x}}^{♮}\Vert }_2\le {\Vert {\mathit{x}}^0\Vert }_2\le \left(1+\frac{1}{\log n}\right){\Vert {\mathit{x}}^{♮}\Vert }_2,$$ (14)

and that the stepsize satisfies ${\eta }_t\equiv \eta =c/{\Vert {\mathit{x}}^{♮}\Vert }_2^2$ for some sufficiently small constant c > 0. Then there exist a sufficiently small absolute constant 0 < γ < 1 and T_γ ≲ log n _{such that with probability at least} 1 − O(m²e^−1.5n) − O(m⁻⁹),

1. the GD iterates (3) converge linearly to x^♮ after t ≥ T_γ, namely,

   $$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le {\left(1-\frac{\eta }{2}{\Vert {\mathit{x}}^{♮}\Vert }_2^2\right)}^{t-T_{\gamma }}\cdot \gamma {\Vert {\mathit{x}}^{♮}\Vert }_2,\forall t\ge T_{\gamma };$$
2. the strength ratio of the signal component $\frac{\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle }{{\Vert {\mathit{x}}^{♮}\Vert }_2^2}{\mathit{x}}^{♮}$ to the orthogonal component ${\mathit{x}}^t-\frac{\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle }{{\Vert {\mathit{x}}^{♮}\Vert }_2^2}{\mathit{x}}^{♮}$ obeys

   $$\frac{{\Vert \frac{\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle }{{\Vert {\mathit{x}}^{♮}\Vert }_2^2}{\mathit{x}}^{♮}\Vert }_2}{{\Vert {\mathit{x}}^t-\frac{\langle {\mathit{x}}^t,{\mathit{x}}^{♮}\rangle }{{\Vert {\mathit{x}}^{♮}\Vert }_2^2}{\mathit{x}}^{♮}\Vert }_2}\gtrsim \frac{1}{\sqrt{n\log n}}{\left(1+c_1{\eta }^2\right)}^t,t=0,1,\cdots$$ (15)

   for some constant c₁ > 0.

Several remarks regarding Theorem 2 are in order.

• Our current sample complexity reads m ≳ n log¹³ m, which is optimal up to logarithmic factors. It is possible to further reduce the logarithmic factors using more refined probabilistic tools, which we leave for future work.

• We can also prove similar performance guarantees for noisy phase retrieval. For brevity, we do not provide the exact theorem and the detailed proofs. The readers will find them in the last author’s Ph.D. thesis.

• The random initialization ${\mathit{x}}^0\sim \mathcal{N}\left(\mathbf{0},n^{-1}{\Vert {\mathit{x}}^{♮}\Vert }_2^2{\mathit{I}}_n\right)$ obeys the condition (14) with probability exceeding $1-O(1/\sqrt{\log n})$, which in turn establishes Theorem 1.

• Theorem 2 requires an initialization x⁰ which is independent of the data and the knowledge of ∥x^♮∥, which is not practical. One possible method is to estimate it from the data, which results in an initial value that depends on the data. The following theorem demonstrate both independent initial value and known ∥x^♮∥ are not necessary, resulting a practical algorithm.

Theorem 3

Let

$${\mathit{x}}^0=\sqrt{\frac{1}{m}\sum _{i=1}^m{y}_i\cdot u},$$

where u is uniformly distributed over the unit sphere. With probability at least $1-O(1/\sqrt{\log n})$ all the claims in Theorem 2 continue to hold.

Proof. The proof is very similar to that of Theorem 2, with only a few changes. See Appendix N for detailed explanations. □

The remainder of this section is then devoted to proving Theorem 2. Without loss of generality⁵, we will assume throughout that

$${\mathit{x}}^{♮}=e_1\text{and}{\mathit{x}}_1^0>0.$$ (16)

Given this, one can decompose

$${\mathit{x}}^t={\mathit{x}}_{\Vert }^t{e}_1+\left[\begin{array}{c}0\\ {\mathit{x}}_{\perp }^t\end{array}\right]$$ (17)

where ${\mathit{x}}_{\Vert }^t={\mathit{x}}_1^t$ and ${\mathit{x}}_{\perp }^t={\left[{\mathit{x}}_i^t\right]}_2{}_{\le i\le n}$ as introduced in Section 2. For notational simplicity, we define

$${\alpha }_t:={\mathit{x}}_{\Vert }^t\text{and}{\beta }_t:={\Vert {\mathit{x}}_{\perp }^t\Vert }_2.$$ (18)

Intuitively, α_t represents the size of the signal component, whereas β_t measures the size of the component orthogonal to the signal direction. In view of (16), we have α₀ > 0.

4.1. Outline of the proof

To begin with, it is easily seen that if α_t and β_t (cf. (18)) obey |α_t − 1| ≤ γ/2 and β_t ≤ γ/2, then

$$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le {\Vert {\mathit{x}}^t-{\mathit{x}}^{♮}\Vert }_2\le \left|{\alpha }_t-1\right|+\left|{\beta }_t\right|\le \gamma .$$

Therefore, our first step — which is concerned with proving $\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le \gamma$ — comes down to the following two steps.

1. Show that if α_t and β_t satisfy the approximate state evolution (see (13)), then there exists some T_γ = O (log n) such that

   $$\left|{\alpha }_{T_{\gamma }}-1\right|\le \gamma /2\text{and}{\beta }_{T_{\gamma }}\le \gamma /2,$$ (19)

   which would immediately imply that

   $$\mathrm{dist}\left({\mathit{x}}^{T_{\gamma }},{\mathit{x}}^{♮}\right)\le \gamma .$$

   Along the way, we will also show that the ratio α_t/β_t grows exponentially fast.

2. Justify that α_t and β_t satisfy the approximate state evolution with high probability, using (some variants of) leave-one-out arguments.

After t ≥ T_γ, we can invoke prior theory [MWCC17] concerning local convergence to show that with high probability,

$$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le {(1-\rho )}^{t-T_{\gamma }}{\Vert {\mathit{x}}^{T_{\gamma }}-{\mathit{x}}^{♮}\Vert }_2,\forall t>T_{\gamma }$$

for some constant 0 < ρ < 1 independent of n and m.

4.2. Dynamics of approximate state evolution

This subsection formalizes our intuition in Section 2: as long as the approximate state evolution holds, then one can find T_γ ≲ log n obeying condition (19). In particular, the approximate state evolution is given by

$${\alpha }_{t+1}=\left\{1+3\eta \left[1-\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\zeta }_t\right\}{\alpha }_t$$ (20a)

$${\beta }_{t+1}=\left\{1+\eta \left[1-3\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\rho }_t\right\}{\beta }_t,$$ (20b)

where {ζ_t} and {ρ_t} represent the perturbation terms. Our result is this:

Lemma 1

Let γ > 0 be some sufficiently small constant, and consider the approximate state evolution (20). Suppose the initial point obeys

$${\alpha }_0\ge \frac{1}{\sqrt{n\log n}}\mathit{\text{and}}1-\frac{1}{\log n}\le \sqrt{{\alpha }_0^2+{\beta }_0^2}\le 1+\frac{1}{\log n}.$$ (21)

and the perturbation terms satisfy

$$\max \left\{\left|{\zeta }_t\right|,\left|{\rho }_t\right|\right\}\le \frac{c_3}{\log n},t=0,1,\cdots$$

for some sufficiently small constant c₃ > 0.

(a) Let

$$T_{\gamma }:=\min \left\{t:\left|{\alpha }_t-1\right|\le \gamma /2\text{and}{\beta }_t\le \gamma /2\right\}.$$ (22)

Then for any sufficiently large n and m and any sufficiently small constant η > 0, one has

$$T_{\gamma }\lesssim \log n,$$ (23)

and there exist some constants c₅, c₁₀ > 0 independent of n and m such that

$$\frac{1}{2\sqrt{n\log n}}\le {\alpha }_t\le 2,c_5\le {\beta }_t\le 1.5\text{and}\frac{{\alpha }_{t+1}/{\alpha }_t}{{\beta }_{t+1}/{\beta }_t}\ge 1+c_{10}{\eta }^2,0\le t\le T_{\gamma }.$$ (24)

(b) If we define

$$T_0:=\min \left\{t:{\alpha }_{t+1}\ge c_6/{\log }^{5}m\right\},$$ (25)

$$T_1:=\min \left\{t:{\alpha }_{t+1}>c_4\right\},$$ (26)

for some arbitrarily small constants c₄, c₆ > 0, then

1. T₀ ≤ T₁ ≤ T_γ ≲ log n; T₁ − T₀ ≲ log log m; T_γ − T₁ ≲ 1;

2. For T₀ < t ≤ T_γ, one has α_t ≥ c₆/log⁵ m.

Proof. See Appendix B. □

Remark 2

Recall that γ is sufficiently small and (α, β) = (1, 0) represents the global minimizer. Since |α₀−1| ≈ 1, one has T_γ > 0, which denotes the first time when the iterates enter the local region surrounding the global minimizer. In addition, the fact that ${\alpha }_0\lesssim 1/\sqrt{n}$ gives T₀ > 0 and T₁ > 0, both of which indicate the first time when the signal strength is sufficiently large.

Lemma 1 makes precise that under the approximate state evolution, the first stage enjoys a fairly short duration T_γ ≲ log n. Moreover, the size of the signal component grows faster than that of the orthogonal component for any iteration t < T_γ, thus confirming the exponential growth of α_t/β_t.

In addition, Lemma 1 identifies two midpoints T₀ and T₁ when the sizes of the signal component α_t become sufficiently large. These are helpful in our subsequent analysis. In what follows, we will divide Stage 1 (which consists of all iterations up to T_γ) into two phases:

• Phase I: consider the duration 0 ≤ t ≤ T₀;

• Phase II: consider all iterations with T₀ < t ≤ T_γ.

We will justify the approximate state evolution (20) for these two phases separately.

4.3. Motivation of the leave-one-out approach

As we have alluded in Section 2.3, the main difficulty in establishing the approximate state evolution (20) lies in controlling the perturbation terms to the desired orders (i.e. |ζ_t|, |ρ_t| ≪ 1/log n in Lemma 1). To achieve this, we advocate the use of (some variants of) leave-one-out sequences to help establish certain “near-independence” between x^t and certain components of {a_i}.

We begin by taking a closer look at the perturbation terms. Regarding the signal component, it is easily seen from (11) that

$${\mathit{x}}_{\Vert }^{t+1}=\left\{1+3\eta \left(1-{\Vert {\mathit{x}}^t\Vert }_2^2\right)\right\}{\mathit{x}}_{\Vert }^t-\eta r_1\left({\mathit{x}}^t\right),$$

where the perturbation term r₁(x^t) obeys

$$\begin{gathered} r_1\left({\mathit{x}}^t\right)=\underset{:={\mathit{I}}_1}{\underbrace{\left[1-{\left({\mathit{x}}_{\Vert }^t\right)}^2\right]{\mathit{x}}_{\Vert }^t\left(\frac{1}{m}\sum _{i=1}^m{a}_{i,1}^4-3\right)}}+\underset{:={\mathit{I}}_2}{\underbrace{\left[1-3{\left({\mathit{x}}_{\Vert }^t\right)}^2\right]\frac{1}{m}\sum _{i=1}^m{a}_{i,1}^3{a}_{i,\perp }^{\top }{\mathit{x}}_{\perp }^t}} \\ -\underset{:={\mathit{I}}_3}{\underbrace{3{\mathit{x}}_{\Vert }^t\left(\frac{1}{m}\sum _{i=1}^m{\left(a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^t\right)}^2{a}_{i,1}^2-{\Vert {\mathit{x}}_{\perp }^t\Vert }_2^2\right)}}-\underset{:={\mathit{I}}_4}{\underbrace{\frac{1}{m}\sum _{i=1}^m{\left(a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^t\right)}^3{a}_{i,1}}}. \end{gathered}$$ (27)

Here and throughout the paper, for any vector $v\in {ℝ}^n$, $v_{\perp }\in {ℝ}^{n-1}$ denotes the 2nd through the nth entries of v. Due to the dependency between x^t and {a_i}, it is challenging to obtain sharp control of some of these terms.

In what follows, we use the term I₄ to explain and motivate our leave-one-out approach. As discussed in Section 2.3, I₄ needs to be controlled to the level $O(1/(\sqrt{n}\mathrm{poly}\log (n)))$. This precludes us from seeking a uniform bound on the function $h(\mathit{x}):=m^{-1}{\sum }_{i=1}^m{\left(a_{i,\perp }^{\top }{\mathit{x}}_{\perp }\right)}^3{a}_{i,1}$ over all x (or even all x within the set $\mathcal{C}$ incoherent with {a_i}), since the uniform bound ${\sup }_{\mathit{x}\in \mathcal{C}}|h(\mathit{x})|$ can be $O(\sqrt{n}/\mathrm{poly}\log (n))$ times larger than the desired order.

Algorithm 1.

The lth leave-one-out sequence

Input: {ai}1≤i≤m,i≠l, {yi}1≤i≤m,i≠l, and x0.

Gradient updates: for t = 0, 1, 2, …, T − 1 do $${\mathit{x}}^{t+1,(l)}={\mathit{x}}^{t,(l)}-{\eta }_t\nabla f^{(l)}\left({\mathit{x}}^{t,(l)}\right),$$ (29) where x0,(l) = x0 and $f^{(l)}(\mathit{x})=(1/4m)\cdot {\sum }_{i:i\ne l}{\left[{\left(a_i^{\top }\mathit{x}\right)}^2-{\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2\right]}^2$.

In order to control I₄ to the desirable order, one strategy is to approximate it by a sum of independent variables and then invoke the CLT. Specifically, we first rewrite I₄ as

$${\mathit{I}}_4=\frac{1}{m}\sum _{i=1}^m{\left(a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^t\right)}^3\left|a_{i,1}\right|{\xi }_i$$

with ξ_i := sgn(a_i,1). Here sgn(·) denotes the usual sign function. To exploit the statistical independence between ξ_i and {|a_i,1|,a_i,⊥}, we would like to identify some vector independent of ξ_i that well approximates x^t. If this can be done, then one may treat I₄ as a weighted independent sum of {ξ_i}. Viewed in this light, our plan is the following:

1. Construct a sequence {x^t,sgn} independent of {ξ_i} obeying x^t,sgn ≈ x^t, so that

   $${\mathit{I}}_4\approx \frac{1}{m}\sum _{i=1}^m\underset{:=w_i}{\underbrace{{\left(a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^{t,\text{sgn}}\right)}^3\left|a_{i,1}\right|}}{\xi }_i.$$

   One can then apply standard concentration results (e.g. the Bernstein inequality) to control I₄, as long as none of the weight w_i is exceedingly large.
2. Demonstrate that the weight w_i is well-controlled, or equivalently, $|a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^{t,\text{sgn}}|(1\le i\le m)$ is not much larger than its typical size. This can be accomplished by identifying another sequence {x^t,(i)} independent of a_i such that x^t,(i) ≈ x^t ≈ x^t,sgn, followed by the argument:

   $$\left|a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^{t,\text{sgn}}\right|\approx \left|a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^t\right|\approx \left|a_{i,\perp }^{\top }{\mathit{x}}_{\perp }^{t,(i)}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^{t,(i)}\Vert }_2\approx \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^t\Vert }_2.$$ (28)

   Here, the inequality follows from standard Gaussian tail bounds and the independence between a_i and x^t,(i). This explains why we would like to construct {x^t,(i)} for each 1 ≤ i ≤ m.

As we will detail in the next subsection, such auxiliary sequences are constructed by leaving out a small amount of relevant information from the collected data before running the GD algorithm, which is a variant of the “leave-one-out” approach rooted in probability theory and random matrix theory.

4.4. Leave-one-out and random-sign sequences

We now describe how to design auxiliary sequences to help establish certain independence properties between the gradient iterates {x^t} and the design vectors {a_i}. In the sequel, we formally define the three sets of auxiliary sequences {x^t,(l)},{x^t,sgn},{x^t,sgn,(l)} as introduced in Section 2.3 and Section 4.3.

• Leave-one-out sequences {x^t,(l)}_t≥0. For each 1 ≤ l ≤ m, we introduce a sequence {x^t,(l)}, which drops the lth sample and runs GD w.r.t. the auxiliary objective function

  $$f^{(l)}(\mathit{x})=\frac{1}{4m}\sum _{i:i\ne l}{\left[{\left(a_i^{\top }\mathit{x}\right)}^2-{\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2\right]}^2.$$ (32)

  See Algorithm 1 for details and also Figure 6(a) for an illustration. One of the most important features of {x^t,(l)} is that all of its iterates are statistically independent of (a_l, y_l), and hence are incoherent with a_l with high probability, in the sense that $\left|a_l^{\top }{\mathit{x}}^{t,(l)}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}^{t,(l)}\Vert }_2$. Such incoherence properties further allow us to control both $\left|a_l^{\top }{\mathit{x}}^t\right|$ and $\left|a_l^{\top }{\mathit{x}}^{t,\text{sgn}}\right|$ (see (28)), which is crucial for controlling the size of the residual terms (e.g. r₁(x^t) as defined in (11)). Notably, the sequence {x^t,(l)} has also been applied by [MWCC17] to justify the success of GD with spectral initialization for several nonconvex statistical estimation problems.

Algorithm 2.

The random-sign sequence

Input: {|ai,1|}1≤i≤m, {ai,⊥}1≤i≤m, ${\left\{{\xi }_i^{\text{sgn}}\right\}}_{1\le i\le m}$, {yi}1≤i≤m, x0.

Gradient updates: for t = 0, 1, 2, …, T − 1 do $${\mathit{x}}^{t+1,\text{sgn}}={\mathit{x}}^{t,\text{sgn}}-{\eta }_t\nabla f^{\text{sgn}}\left({\mathit{x}}^{t,\text{sgn}}\right),$$ (30) where ${\mathit{x}}^{0,\text{sgn}}={\mathit{x}}^0,f^{\text{sgn}}(\mathit{x})=\frac{1}{4m}{\sum }_{i=1}^m{\left[{\left(a_i^{\text{sgn}\top }\mathit{x}\right)}^2-{\left(a_i^{\text{sgn}\top }{\mathit{x}}^{♮}\right)}^2\right]}^2$ with $a_i^{\text{sgn}}:=\left[\begin{array}{c}{\xi }_i^{\text{sgn}}\left|a_{i,1}\right|\\ a_{i,\perp }\end{array}\right]$.

Algorithm 3.

The lth leave-one-out and random-sign sequence

Input: {|ai,1|}1≤i≤m,i≠l, {ai,⊥}1≤i≤m,i≠l, ${\left\{{\xi }_i^{\mathrm{sgn}}\right\}}_{1\le i\le m,,i\ne l}$, {yi}1≤i≤m,i≠l, x0.

Gradient updates: for t = 0, 1, 2, …, T − 1 do $${\mathit{x}}^{t+1,\mathrm{sgn},(l)}={\mathit{x}}^{t,\mathrm{sgn},(l)}-{\eta }_t\nabla f^{\text{sgn},(l)}\left({\mathit{x}}^{t,\mathrm{sgn},(l)}\right),$$ (31) where ${\mathit{x}}^{0,\mathrm{sgn},(l)}={\mathit{x}}^0,f^{\mathrm{sgn},(l)}(\mathit{x})=\frac{1}{4m}{\sum }_{i:i\ne l}{\left[{\left(a_i^{\mathrm{sgn}\top }\mathit{x}\right)}^2-{\left(a_i^{\mathrm{sgn}\top }{\mathit{x}}^{♮}\right)}^2\right]}^2$ with $a_i^{\text{sgn}}:=\left[\begin{array}{c}{\xi }_i^{\text{sgn}}\left|a_{i,1}\right|\\ a_{i,\perp }\end{array}\right]$.

• Random-sign sequence {x^t,sgn}_t≥0. Introduce a collection of auxiliary design vectors ${\left\{a_i^{\text{sgn}}\right\}}_{1\le i\le m}$ defined as

  $$a_i^{\text{sgn}}:=\left[\begin{array}{c}{\xi }_i^{\text{sgn}}\left|a_{i,1}\right|\\ a_{i,\perp }\end{array}\right],$$ (33)

  where ${\left\{{\xi }_i^{\text{sgn}}\right\}}_{1\le i\le m}$ is a set of Rademacher random variables independent of {a_i}, i.e.

  $${\xi }_i^{\text{sgn}}\stackrel{\text{i.i.d.}}{=}\{\begin{array}{ll}1,\hfill & \text{with}\text{probability}1/2,\hfill \\ -1,\hfill & \text{else}\hfill \end{array}1\le i\le m.$$ (34)

  In words, $a_i^{\text{sgn}}$ is generated by randomly flipping the sign of the first entry of a_i. To simplify the notations hereafter, we also denote

  $${\xi }_i=\mathrm{sgn}\left(a_{i,1}\right).$$ (35)

  As a result, a_i and $a_i^{\text{sgn}}$ differ only by a single bit of information. With these auxiliary design vectors in place, we generate a sequence {x^t,sgn} by running GD w.r.t. the auxiliary loss function

  $$f^{\text{sgn}}(\mathit{x})=\frac{1}{4m}\sum _{i=1}^m{\left[{\left(a_i^{\text{sgn}\top }\mathit{x}\right)}^2-{\left(a_i^{\text{sgn}\top }{\mathit{x}}^{♮}\right)}^2\right]}^2.$$ (36)

  One simple yet important feature associated with these new design vectors is that it produces the same measurements as {a_i}:

  $${\left(a_i^{\top }{\mathit{x}}^{♮}\right)}^2={\left(a_i^{\text{sgn}\top }{\mathit{x}}^{♮}\right)}^2={\left|a_{i,1}\right|}^2,1\le i\le m.$$ (37)

  See Figure 6(b) for an illustration and Algorithm 2 for the detailed procedure. This sequence is introduced in order to “randomize” certain Gaussian polynomials (e.g. I₄ in (27)), which in turn enables optimal control of these quantities. This is particularly crucial at the initial stage of the algorithm.

• Leave-one-out and random-sign sequences {x^t,sgn,(l)}_t≥0. Furthermore, we also need to introduce another collection of sequences {x^t,sgn,(l)} by simultaneously employing the new design vectors $\left\{a_i^{\text{sgn}}\right\}$ and discarding a single sample $\left(a_l^{\text{sgn}},y_l^{\text{sgn}}\right).$. This enables us to propagate the kinds of independence properties across the above two sets of sequences, which is useful in demonstrating that x^t is jointly “nearly-independent” of both a_l and {sgn(a_i,1)}. See Algorithm 3 and Figure 6(c).

As a remark, all of these leave-one-out and random-sign procedures are assumed to start from the same initial point as the original sequence, namely,

$${\mathit{x}}^0={\mathit{x}}^{0,(l)}={\mathit{x}}^{0,\text{sgn}}={\mathit{x}}^{0,\mathrm{sgn},(l)},1\le l\le m.$$ (38)

4.5. Justification of approximate state evolution for Phase I of Stage 1

Recall that Phase I consists of the iterations 0 ≤ t ≤ T₀, where

$$T_0=\min \left\{t:{\alpha }_{t+1}\ge \frac{c_6}{{\log }^{5}m}\right\}.$$ (39)

Our goal here is to show that the approximate state evolution (20) for both the size α_t of the signal component and the size β_t of the orthogonal component holds true throughout Phase I. Our proof will be inductive in nature. Specifically, we will first identify a set of induction hypotheses that are helpful in proving the validity of the approximate state evolution (20), and then proceed by establishing these hypotheses via induction.

4.5.1. Induction hypotheses

For the sake of clarity, we first list all the induction hypotheses.

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,(l)}\Vert }_2\le {\beta }_t{\left(1+\frac{1}{\log m}\right)}^t{C}_1\frac{\sqrt{n{\log }^{5}m}}{m},$$ (40a)

$$\underset{1\le l\le m}{\max }\left|{\mathit{x}}_{\Vert }^t-{\mathit{x}}_{\Vert }^{t,(l)}\right|\le {\alpha }_t{\left(1+\frac{1}{\log m}\right)}^t{C}_2\frac{\sqrt{n{\log }^{12}m}}{m},$$ (40b)

$${\Vert {\mathit{x}}^t-{\mathit{x}}^{t,\mathrm{sgn}}\Vert }_2\le {\alpha }_t{\left(1+\frac{1}{\log m}\right)}^t{C}_3\sqrt{\frac{n{\log }^{5}m}{m}},$$ (40c)

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,\text{sgn}}-{\mathit{x}}^{t,(l)}+{\mathit{x}}^{t,\text{sgn},(l)}\Vert }_2\le {\alpha }_t{\left(1+\frac{1}{\log m}\right)}^t{C}_4\frac{\sqrt{n{\log }^{9}m}}{m},$$ (40d)

$$c_5\le {\Vert {\mathit{x}}_{\perp }^t\Vert }_2\le {\Vert {\mathit{x}}^t\Vert }_2\le C_5,$$ (40e)

$${\Vert {\mathit{x}}^t\Vert }_2\le 4{\alpha }_t\sqrt{n\log m},$$ (40f)

where C₁,· · ·, C₅ and c₅ are some absolute positive constants.

Now we are ready to prove an immediate consequence of the induction hypotheses (40): if (40) hold for the t^th iteration, then α_t+1 and β_t+1 follow the approximate state evolution (see (20)). This is justified in the following lemma.

Lemma 2

Suppose m ≥ Cn log¹¹ m for some sufficiently large constant C > 0. For any 0 ≤ t ≤ T₀ (cf. (39)), if the t^th iterates satisfy the induction hypotheses (40), then with probability at least 1 − O(me^−1.5n) − O(m⁻¹⁰),

$${\alpha }_{t+1}=\left\{1+3\eta \left[1-\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\zeta }_t\right\}{\alpha }_t;$$ (41a)

$${\beta }_{t+1}=\left\{1+\eta \left[1-3\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\rho }_t\right\}{\beta }_t$$ (41b)

hold for some |ζ_t| ≪ 1/log m and |ρ_t| ≪ 1/log m.

Proof. See Appendix C. □

It remains to inductively show that the hypotheses hold for all 0 ≤ t ≤ T₀. Before proceeding to this induction step, it is helpful to first develop more understanding about the preceding hypotheses.

1. In words, (40a), (40b), (40c) specify that the leave-one-out sequences {x^t,(l)} and {x^t,sgn} are exceedingly close to the original sequence {x^t}. Similarly, the difference between x^t − x^t,sgn and x^t,(l) − x^t,sgn,(l) is extremely small, as asserted in (40d). The hypothesis (40e) says that the norm of the iterates {x^t} is always bounded from above and from below in Phase I. The last one (40f) indicates that the size α_t of the signal component is never too small compared with ∥x^t∥₂.

2. Another property that is worth mentioning is the growth rate (with respect to t) of the quantities appeared in the induction hypotheses (40). For instance, $\left|{\mathit{x}}_{\Vert }^t-{\mathit{x}}_{\Vert }^{t,(l)}\right|$, ∥x^t−x^t,sgn∥₂ and ∥x^t−x^t,sgn−x^t,(l)+x^t,sgn,(l)∥₂ grow more or less at the same rate as α_t (modulo some ${(1+1/\log m)}^{T_0}$ factor). In contrast, ∥x^t −x^t,(l)∥2 shares the same growth rate with β_t (modulo the ${(1+1/\log m)}^{T_0}$ factor). See Figure 7 for an illustration. The difference in the growth rates turns out to be crucial in establishing the advertised result.

3. Last but not least, we emphasize the sizes of the quantities of interest in (40) for t = 1 under the Gaussian initialization. Ignoring all of the log m terms and recognizing that ${\alpha }_1\asymp 1/\sqrt{n}$ and β₁ ≍ 1, one sees that ${\Vert {\mathit{x}}^1-{\mathit{x}}^{1,(l)}\Vert }_2\lesssim 1/\sqrt{m}$, $\left|{\mathit{x}}_{\Vert }^1-{\mathit{x}}_{\Vert }^{1,(l)}\right|\lesssim 1/m$, ${\Vert {\mathit{x}}^1-{\mathit{x}}^{1,\text{sgn}}\Vert }_2\lesssim 1/\sqrt{m}$ and ∥x¹−x^1,sgn−x^1,(l)+x^1,sgn,(l)∥₂ ≲ 1/m. See Figure 7 for an illustration of the trends of the above four quantities.

Figure 7:

Illustration of the differences among leave-one-out and original sequences vs. iteration count, plotted semilogarithmically. The results are shown for n = 1000 with m = 10n, η_t ≡ 0.1, and ${\Vert {\mathit{x}}^{♮}\Vert }_2=1$. (a) The four differences increases in Stage 1. From the induction hypotheses (40), our upper bounds on $\left|{\mathit{x}}_{\Vert }^t-{\mathit{x}}_{\Vert }^{t,(l)}\right|$, ∥x^t − x^t,sgn∥₂ and ∥x^t − x^t,sgn − x^t,(l) + x^t,sgn,(l)∥₂ scale linearly with α_t, whereas the upper bound on ∥x^t − x^t,(l)∥₂ is proportional to β_t. In addition, ${\Vert {\mathit{x}}^1-{\mathit{x}}^{1,(l)}\Vert }_2\lesssim 1/\sqrt{m}$, $\left|{\mathit{x}}_{\Vert }^1-{\mathit{x}}_{\Vert }^{1,(l)}\right|\lesssim 1/m$, ${\Vert {\mathit{x}}^1-{\mathit{x}}^{1,\text{sgn}}\Vert }_2\lesssim 1/\sqrt{m}$ and ∥x¹ − x^1,sgn − x^1,(l) + x^1,sgn,(l)∥₂ ≲ 1/m. (b) The four differences converge to zero geometrically fast in Stage 2, as all the (variants of) leave-one-out sequences and the original sequence converge to the truth x^♮.

Several consequences of (40) regarding the incoherence between {x^t}, {x^t,sgn} and {a_i}, $\left\{a_i^{\text{sgn}}\right\}$ are immediate, as summarized in the following lemma.

Lemma 3

Suppose that m ≥ Cn log⁶m for some sufficiently large constant C > 0 and the t^th iterates satisfy the induction hypotheses (40) for t ≤ T₀, then with probability at least 1 − O(me^−1.5n) − O(m⁻¹⁰),

$$\begin{gathered} \underset{1\le l\le m}{\max }\left|a_l^{\top }{\mathit{x}}^t\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}^t\Vert }_2; \\ \underset{1\le l\le m}{\max }\left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^t\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^t\Vert }_2; \\ \underset{1\le l\le m}{\max }\left|a_l^{\top }{\mathit{x}}^{t,\text{sgn}}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}^{t,\text{sgn}}\Vert }_2; \\ \underset{1\le l\le m}{\max }\left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^{t,\text{sgn}}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^{t,\text{sgn}}\Vert }_2; \\ \underset{1\le l\le m}{\max }\left|a_l^{\mathrm{sgn}\top }{\mathit{x}}^{t,\mathrm{sgn}}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}^{t,\mathrm{sgn}}\Vert }_2. \end{gathered}$$

Proof. These incoherence conditions typically arise from the independence between {x^t,(l)} and a_l. For instance, the first line follows since

$$\left|a_l^{\top }{\mathit{x}}^t\right|\approx \left|a_l^{\top }{\mathit{x}}^{t,(l)}\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}^{t,(l)}\Vert }_2\asymp \sqrt{\log m}{\Vert {\mathit{x}}^t\Vert }_2.$$

See Appendix M for detailed proofs. □

4.5.2. Induction step

We then turn to showing that the induction hypotheses (40) hold throughout Phase I, i.e. for 0 ≤ t ≤ T₀. The base case can be easily verified because of the identical initial points (38). Now we move on to the inductive step, i.e. we aim to show that if the hypotheses (40) are valid up to the t^th iteration for some t ≤ T₀, then they continue to hold for the (t + 1)^th iteration.

The first lemma concerns the difference between the leave-one-out sequence x^t+1,(l) and the true sequence x^t+1 (see (40a)).

Lemma 4

Suppose m ≥ Cnlog⁵ m for some sufficiently large constant C > 0. If the induction hypotheses (40) hold true up to the t^th iteration for some t ≤ T₀, then with probability at least 1−O(me^−1.5n)−O(m⁻¹⁰),

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^{t+1}-{\mathit{x}}^{t+1,(l)}\Vert }_2\le {\beta }_{t+1}{\left(1+\frac{1}{\log m}\right)}^{t+1}{C}_1\frac{\sqrt{n{\log }^{5}m}}{m}$$ (43)

holds as long as η > 0 is a sufficiently small constant and C₁ > 0 is sufficiently large.

Proof. See Appendix D. □

The next lemma characterizes a finer relation between x^t+1 and x^t+1,(l) when projected onto the signal direction (cf. (40b)).

Lemma 5

Suppose m ≥ Cnlog⁶ m for some sufficiently large constant C > 0. If the induction hypotheses (40) hold true up to the t^th iteration for some t ≤ T₀, then with probability at least 1−O(me^−1.5n)−O(m⁻¹⁰),

$$\underset{1\le l\le m}{\max }\left|{\mathit{x}}_{\Vert }^{t+1}-{\mathit{x}}_{\Vert }^{t+1,(l)}\right|\le {\alpha }_{t+1}{\left(1+\frac{1}{\log m}\right)}^{t+1}{C}_2\frac{\sqrt{n{\log }^{12}m}}{m}$$ (44)

holds as long as η > 0 is a sufficiently small constant and C₂ ≫ C₄.

Proof. See Appendix E. □

Regarding the difference between x^t and x^t,sgn (see (40c)), we have the following result.

Lemma 6

Suppose m ≥ Cnlog⁵ m for some sufficiently large constant C > 0. If the induction hypotheses (40) hold true up to the t^th iteration for some t ≤ T₀, then with probability at least 1−O(me^−1.5n)−O(m⁻¹⁰),

$${\Vert {\mathit{x}}^{t+1}-{\mathit{x}}^{t+1,\mathrm{sgn}}\Vert }_2\le {\alpha }_{t+1}{\left(1+\frac{1}{\log m}\right)}^{t+1}{C}_3\sqrt{\frac{n{\log }^{5}m}{m}}$$ (45)

holds as long as η > 0 is a sufficiently small constant and C₃ is a sufficiently large positive constant.

Proof. See Appendix F. □

We are left with the double difference x^t+1 − x^t+1,sgn − x^t+1,(l) + x^t+1,sgn,(l) (cf. (40d)), for which one has the following lemma.

Lemma 7

Suppose m ≥ Cnlog⁸ m for some sufficiently large constant C > 0. If the induction hypotheses (40) hold true up to the t^th iteration for some t ≤ T₀, then with probability at least 1−O(me^−1.5n)−O(m⁻¹⁰),

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^{t+1}-{\mathit{x}}^{t+1,\mathrm{sgn}}-{\mathit{x}}^{t+1,(l)}+{\mathit{x}}^{t+1,\mathrm{sgn},(l)}\Vert }_2\le {\alpha }_{t+1}{\left(1+\frac{1}{\log m}\right)}^{t+1}{C}_4\frac{\sqrt{n{\log }^{9}m}}{m}$$ (46)

holds as long as η > 0 is a sufficiently small constant and C₄ > 0 is sufficiently large.

Proof. See Appendix G. □

Assuming the induction hypotheses (40) hold up to the t^th iteration for some t ≤ T₀, we know from Lemma 2 that the approximate state evolution for both α_t and β_t (see (20)) holds up to t + 1. As a result, the last two hypotheses (40e) and (40f) for the (t + 1)^th iteration can be easily verified.

4.6. Justification of approximate state evolution for Phase II of Stage 1

Recall from Lemma 1 that Phase II refers to the iterations T₀ < t ≤ T_γ (see the definition of T₀ in Lemma 1), for which one has

$${\alpha }_t\ge \frac{c_6}{{\log }^{5}m}$$ (47)

as long as the approximate state evolution (20) holds. Here c₆ > 0 is the same constant as in Lemma 1. Similar to Phase I, we invoke an inductive argument to prove that the approximate state evolution (20) continues to hold for T₀ < t ≤ T_γ.

4.6.1. Induction hypotheses

In Phase I, we rely on the leave-one-out sequences and the random-sign sequences {x^t,(l)}, {x^t,sgn} and {x^t,sgn,(l)} to establish certain “near-independence” between {x^t} and {a_l}, which in turn allows us to obtain sharp control of the residual terms r (x^t) (cf. (10)) and r₁ (x^t) (cf. (11)). As it turns out, once the size α_t of the signal component obeys α_t ≳ 1/poly log(m), then {x^t,(l)} alone is sufficient for our purpose to establish the “near-independence” property. More precisely, in Phase II we only need to impose the following induction hypotheses.

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,(l)}\Vert }_2\le {\alpha }_t{\left(1+\frac{1}{\log m}\right)}^t{C}_6\frac{\sqrt{n{\log }^{15}m}}{m};$$ (48a)

$$c_5\le {\Vert {\mathit{x}}_{\perp }^t\Vert }_2\le {\Vert {\mathit{x}}^t\Vert }_2\le C_5.$$ (48b)

A direct consequence of (48) is the incoherence between x^t and {a_l}, namely,

$$\underset{1\le l\le m}{\max }\left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^t\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^t\Vert }_2;$$ (49a)

$$\underset{1\le l\le m}{\max }\left|a_l^{\top }{\mathit{x}}^t\right|\lesssim \sqrt{\log m}\Vert {\mathit{x}}^t{\Vert }_2.$$ (49b)

To see this, one can use the triangle inequality to show that

$$\begin{gathered} \left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^t\right|\le \left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^{t,(l)}\right|+\left|a_{l,\perp }^{\top }\left({\mathit{x}}_{\perp }^t-{\mathit{x}}_{\perp }^{t,(l)}\right)\right| \\ \stackrel{(\text{i})}{\lesssim }\sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^{t,(l)}\Vert }_2+\sqrt{n}{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,(l)}\Vert }_2 \\ \lesssim \sqrt{\log m}\left({\Vert {\mathit{x}}_{\perp }^t\Vert }_2+{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,(l)}\Vert }_2\right)+\sqrt{n}{\Vert {\mathit{x}}^t-{\mathit{x}}^{t,(l)}\Vert }_2 \\ \stackrel{(\text{ii})}{\lesssim }\sqrt{\log m}+\frac{\sqrt{n{\log }^{15}m}}{m}\sqrt{n}\lesssim \sqrt{\log m}, \end{gathered}$$

where (i) follows from the independence between a_l and x^t,(l) and the Cauchy-Schwarz inequality, and the last line (ii) arises from (1 + 1/log m)^t ≲ 1 for t ≤ T_γ ≲ log n and m ≫ n log^15/2m. This combined with the fact that ${\Vert {\mathit{x}}_{\perp }^t\Vert }_2\ge c_5/2$ results in

$$\underset{1\le l\le m}{\max }\left|a_{l,\perp }^{\top }{\mathit{x}}_{\perp }^t\right|\lesssim \sqrt{\log m}{\Vert {\mathit{x}}_{\perp }^t\Vert }_2.$$ (50)

The condition (49b) follows using nearly identical arguments, which are omitted here.

As in Phase I, we need to justify the approximate state evolution (20) for both α_t and β_t, given that the t^th iterates satisfy the induction hypotheses (48). This is stated in the following lemma.

Lemma 8

Suppose m ≥ Cnlog¹³ m for some sufficiently large constant C > 0. If the t^th iterates satisfy the induction hypotheses (48) for T₀ < t < T_γ, then with probability at least 1 − O(me^−1.5n) − O(m⁻¹⁰),

$${\alpha }_{t+1}=\left\{1+3\eta \left[1-\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\zeta }_t\right\}{\alpha }_t;$$ (51a)

$${\beta }_{t+1}=\left\{1+\eta \left[1-3\left({\alpha }_t^2+{\beta }_t^2\right)\right]+\eta {\rho }_t\right\}{\beta }_t,$$ (51b)

for some |ζ_t| ≪ 1/log m and |ρ_t| ≪ 1/log m

Proof. See Appendix H for the proof of (51a). The proof of (51b) follows exactly the same argument as in proving (41b), and is hence omitted. □

4.6.2. Induction step

We proceed to complete the induction argument. Towards this end, one has the following lemma in regard to the induction on max_1≤l≤m ∥x^t+1 − x^t+1,(l)∥₂ (see (48a)).

Lemma 9

Suppose m ≥ Cnlog⁵ m for some sufficiently large constant C > 0, and consider any T₀ < t < T_γ. If the induction hypotheses (40) are valid throughout Phase I and (48) are valid from the T₀th to the t^th iterations, then with probability at least 1 − O(me^−1.5n) − O(m⁻¹⁰),

$$\underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^{t+1}-{\mathit{x}}^{t+1,(l)}\Vert }_2\le {\alpha }_{t+1}{\left(1+\frac{1}{\log m}\right)}^{t+1}{C}_6\frac{\sqrt{n{\log }^{13}m}}{m}$$

holds as long as η > 0 is sufficiently small and C₆ > 0 is sufficiently large.

Proof. See Appendix I. □

As in Phase I, since we assume the induction hypotheses (40) (resp. (48)) hold for all iterations up to the T₀th iteration (resp. between the T₀th and the t^th iteration), we know from Lemma 8 that the approximate state evolution for both α_t and β_t (see (20)) holds up to t + 1. The last induction hypothesis (48b) for the (t + 1)^th iteration can be easily verified from Lemma 1.

It remains to check the case when t = T₀ + 1. It can be seen from the analysis in Phase I that

$$\begin{gathered} \underset{1\le l\le m}{\max }{\Vert {\mathit{x}}^{T_0+1}-{\mathit{x}}^{T_0+1,(l)}\Vert }_2\le {\beta }_{T_0+1}{\left(1+\frac{1}{\log m}\right)}^{T_0+1}{C}_1\frac{\sqrt{n{\log }^{5}m}}{m} \\ \le {\alpha }_{T_0+1}{\left(1+\frac{1}{\log m}\right)}^{T_0+1}{C}_6\frac{\sqrt{n{\log }^{15}m}}{m}, \end{gathered}$$

for some constant condition C₆ ≫ 1, where the second line holds since ${\beta }_{T_0+1}\le C_5$, ${\alpha }_{T_0+1}\ge c_6/{\log }^{5}m$.

4.7. Analysis for Stage 2

Combining the analyses in Phase I and Phase II, we finish the proof of Theorem 2 for Stage 1, i.e. t ≤ T_γ. In addition to $\left({\mathit{x}}^{T_{\gamma }},{\mathit{x}}^{♮}\right)\le \gamma$, we can also see from (49b) that

$$\underset{1\le i\le m}{\max }\left|a_i^{\top }{\mathit{x}}^{T_{\gamma }}\right|\lesssim \sqrt{\log m},$$

which in turn implies that

$$\underset{1\le i\le m}{\max }\left|a_i^{\top }\left({\mathit{x}}^{T_{\gamma }}-{\mathit{x}}^{♮}\right)\right|\lesssim \sqrt{\log m}.$$

Armed with these properties, one can apply the arguments in [MWCC17, Section 6] to prove that for t ≥ T_γ + 1,

$$\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)\le {\left(1-\frac{\eta }{2}\right)}^{t-T_{\gamma }}\mathrm{dist}\left({\mathit{x}}^{T_{\gamma }},{\mathit{x}}^{♮}\right)\le {\left(1-\frac{\eta }{2}\right)}^{t-T_{\gamma }}\cdot \gamma .$$ (52)

Notably, the theorem therein [MWCC17, Theorem 1] works under the stepsize η_t ≡ η ≍ c/log n when m ≫ n log n. Nevertheless, as remarked by the authors, when the sample complexity exceeds m ≫ n log³m, a constant stepsize is allowed.

We are left with proving (15) for Stage 2. Note that we have already shown that the ratio α_t/β_t increases exponentially fast in Stage 1. Therefore,

$$\frac{{\alpha }_{T_1}}{{\beta }_{T_1}}\ge \frac{1}{\sqrt{2n\log n}}{\left(1+c_{10}{\eta }^2\right)}^{T_1}$$

and, by the definition of T₁ (see (26)) and Lemma 1, one has ${\alpha }_{T_1}\asymp {\beta }_{T_1}\asymp 1$ and hence

$$\frac{{\alpha }_{T_1}}{{\beta }_{T_1}}\asymp 1.$$ (53)

When it comes to t > T_γ, in view of (52), one has

$$\begin{gathered} \frac{{\alpha }_t}{{\beta }_t}\ge \frac{1-\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)}{\mathrm{dist}\left({\mathit{x}}^t,{\mathit{x}}^{♮}\right)}\ge \frac{1-\gamma }{{\left(1-\frac{\eta }{2}\right)}^{t-T_{\gamma }}\cdot \gamma } \\ \ge \frac{1-\gamma }{\gamma }{\left(1+\frac{\eta }{2}\right)}^{t-T_{\gamma }}\stackrel{(\text{i})}{\asymp }\frac{{\alpha }_{T_1}}{{\beta }_{T_1}}{\left(1+\frac{\eta }{2}\right)}^{t-T_{\gamma }} \\ \gtrsim \frac{1}{\sqrt{n\log n}}{\left(1+c_{10}{\eta }^2\right)}^{T_1}{\left(1+\frac{\eta }{2}\right)}^{t-T_{\gamma }} \\ \stackrel{(\text{ii})}{\asymp }\frac{1}{\sqrt{n\log n}}{\left(1+c_{10}{\eta }^2\right)}^{T_{\gamma }}{\left(1+\frac{\eta }{2}\right)}^{t-T_{\gamma }} \\ \gtrsim \frac{1}{\sqrt{n\log n}}{\left(1+c_{10}{\eta }^2\right)}^t, \end{gathered}$$

where (i) arises from (53) and the fact that γ is a constant, (ii) follows since T_γ – T₁ ≍ 1 according to Lemma 1, and the last line holds as long as c₁₀ > 0 and η are sufficiently small. This concludes the proof regarding the lower bound on α_t/β_t.

5. Discussions

The current paper justifies the fast global convergence of gradient descent with random initialization for phase retrieval. Specifically, we demonstrate that GD with random initialization takes only O(log n + log(1/ϵ)) iterations to achieve a relative ϵ-accuracy in terms of the estimation error. It is likely that such fast global convergence properties also arise in other nonconvex statistical estimation problems. The technical tools developed herein may also prove useful for other settings. We conclude our paper with a few directions worthy of future investigation.

• Sample complexity and phase transition. We have proved in Theorem 2 that GD with random initialization enjoys fast convergence, with the proviso that m ≫ n log¹³m. It is possible to improve the sample complexity via more sophisticated arguments. In addition, it would be interesting to examine the phase transition phenomenon of GD with random initialization.

• Other nonconvex statistical estimation problems. We use the phase retrieval problem to showcase the efficiency of GD with random initialization. It is certainly interesting to investigate whether this fast global convergence carries over to other nonconvex statistical estimation problems including low-rank matrix and tensor recovery [KMO10,SL16,CW15,TBS⁺16,ZL16,ZWL15,MWCC17,CL17,CC18,CCF18,HZC18], blind deconvolution [LLSW18,MWCC17,HH17] and neural networks [SJL17,LMZ17,FCL18]. The leave-one-out sequences and the “near-independence” property introduced/identified in this paper might be useful in proving efficiency of randomly initialized GD for the aforementioned problems.

• Noisy setting and other activation functions. Throughout this paper, our focus is on inverting noiseless quadratic systems. Extensions to the noisy case is definitely worth investigating. Moving beyond quadratic samples, one may also study other activation functions, including but not limited to Rectified Linear Units (ReLU), polynomial functions and sigmoid functions. Such investigations might shed light on the effectiveness of GD with random initialization for training neural networks.

• Other iterative optimization methods. Apart from gradient descent, other iterative procedures have been applied to solve the phase retrieval problem. Partial examples include alternating minimization, Kaczmarz algorithm, and truncated gradient descent (Truncated Wirtinger flow). In conjunction with random initialization, whether the iterative algorithms mentioned above enjoy fast global convergence is an interesting open problem. For example, it has been shown that truncated WF together with truncated spectral initialization achieves optimal sample complexity (i.e. m ≍ n) and computational complexity simultaneously [CC17]. Does truncated Wirtinger flow still enjoy optimal sample complexity when initialized randomly?

• Beyond Gaussian sampling vectors. In this work, we consider the Gaussian phase retrieval problem where the sampling vectors are i.i.d. Gaussian vectors. We expect our results to generalize to other sampling vectors. Experimentally, we can verify that random initialization also converges fast under a Rademacher sampling model; see Figure 8.

• Applications of leave-one-out tricks. In this paper, we heavily deploy the leave-one-out trick to demonstrate “near-independence” between the iterates x^t and the sampling vectors {a_i}. The basic idea is to construct an auxiliary sequence that is (i) independent w.r.t. certain components of the design vectors, and (ii) extremely close to the original sequence. These two properties allow us to propagate the desired independence properties to x^t. As mentioned in Section 3, the leave-one-out trick has served as a very powerful hammer for decoupling the dependency between random vectors in several high-dimensional estimation problems. We expect this powerful trick to be useful in broader settings.

Figure 8:

The relative ℓ₂ error vs. iteration count for GD with random initialization, plotted semilogarithvectorsmically. The results are shown for n = 1000 with m = 10n and ηt ≡ 0.1. Here the entries of the sampling a_i are drawn _i.i.d. from a Rademacher distribution.