Fig. 4.

Separating two isotropic Gaussians, with a nonmonotone activation function (see Predicting Failure for details). Here $N=800$, $d=320$, and $\mathrm{\Delta }=0.5$. The main frame presents the evolution of the population risk along the SGD trajectory, starting from two different initializations of ${\left({\mathbf{w}}_i^0\right)}_{i\le N}{\sim }_{iid}\mathsf{N}\left(\mathbf{0},{\kappa }^2/d\cdot {\mathbf{I}}_d\right)$ for either $\kappa =0.1$ or $\kappa =0.4$. In Inset, we plot the evolution of the average of $\Vert \mathbf{w}{\Vert }_2$ for the same conditions. Symbols are empirical results. Continuous lines are predictions obtained with the reduced PDE (Eq. 13).