Understanding Implicit Regularization in Over-Parameterized Single Index Model

Abstract

In this paper, we leverage over-parameterization to design regularization-free algorithms for the high-dimensional single index model and provide theoretical guarantees for the induced implicit regularization phenomenon. Specifically, we study both vector and matrix single index models where the link function is nonlinear and unknown, the signal parameter is either a sparse vector or a low-rank symmetric matrix, and the response variable can be heavy-tailed. To gain a better understanding of the role played by implicit regularization without excess technicality, we assume that the distribution of the covariates is known a priori. For both the vector and matrix settings, we construct an over-parameterized least-squares loss function by employing the score function transform and a robust truncation step designed specifically for heavy-tailed data. We propose to estimate the true parameter by applying regularization-free gradient descent to the loss function. When the initialization is close to the origin and the stepsize is sufficiently small, we prove that the obtained solution achieves minimax optimal statistical rates of convergence in both the vector and matrix cases. In addition, our experimental results support our theoretical findings and also demonstrate that our methods empirically outperform classical methods with explicit regularization in terms of both ℓ₂-statistical rate and variable selection consistency.

1. Introduction

With the astonishing empirical success in various application domains such as computer vision [Voulodimos et al., 2018], natural language processing [Otter et al., 2020, Torfi et al., 2020], and reinforcement learning [Arulkumaran et al., 2017, Li, 2017], deep learning [LeCun et al., 2015, Goodfellow et al., 2016, Fan et al., 2021a] has become one of the most prevalent classes of machine learning methods. When applying deep learning to supervised learning tasks such as regression and classification, the regression function or classifier is represented by a deep neural network, which is learned by minimizing a loss function of the network weights. Here the loss function is defined as the empirical risk function computed based on the training data and the optimization problem is usually solved by gradient-based optimization methods. Due to the nonlinearity of the activation function and the multi-layer functional composition, the landscape of the loss function is highly nonconvex, with many saddle points and local minima [Dauphin et al., 2014, Swirszcz et al., 2016, Yun et al., 2019]. Moreover, oftentimes the neural network is over-parameterized in the sense that the total number of network weights exceeds the number of training data, making the regression or classification problem ill-posed from a statistical perspective. Surprisingly, however, it is often observed empirically that simple algorithms such as (stochastic) gradient descent tend to find the global minimum of the loss function despite non-convexity. Moreover, the obtained solution also generalizes well to unseen data with small test error [Neyshabur et al., 2015, Zhang et al., 2017]. These mysterious observations cannot be fully explained by the classical theory of nonconvex optimization and generalization bounds via uniform convergence.

To understand such an intriguing phenomenon, Neyshabur et al. [2015], Zhang et al. [2017] show empirically that the generalization stems from an “implicit regularization” of the optimization algorithm. Specifically, they observe that, in over-parametrized statistical models, although the optimization problems consist of bad local minima with large generalization error, the choice of optimization algorithm, usually a variant of gradient descent algorithm, usually guard the iterates from bad local minima and prefers the solution that generalizes well. Thus, without adding any regularization term in the optimization objective, the implicit preference of the optimization algorithm itself plays the role of regularization. Implicit regularization has been shown indispensable in training deep learning models [Neyshabur et al., 2015, 2017, Zhang et al., 2017, Keskar et al., 2017, Poggio et al., 2017, Wilson et al., 2017].

With properly designed algorithm, Gunasekar et al. [2017] and Li et al. [2018] provide empirical evidence and theoretical guarantees for the implicit regularization of gradient descent for least-squares regression with a two-layer linear neural network, i.e., low-rank matrix sensing. They show that gradient descent biases towards the minimum nuclear norm solution when the initialization is close to the origin, stepsizes are sufficiently small, and no explicit regularization is imposed. More specifically, when the true parameter is a rank r positive-semidefinite matrix in ${ℝ}^{d\times d}$, they rewrite the parameter as UU^⊤, where $\text{U}\in {ℝ}^{d\times d}$, and propose to estimate the true parameter by updating U via gradient descent. Li et al. [2018] proves that, with $\tilde{\mathcal{O}}\left(r^{2}d\right)$ i.i.d. observations of the model, gradient descent provably recovers the true parameter with accuracy, where $\tilde{\mathcal{O}}(\cdot )$ hides absolute constants and polylogarithmic terms. Thus, in over-parametrized matrix sensing problems, the implicit regularization of gradient descent can be viewed as equivalent to adding a nuclear norm penalty explicitly. See also Arora et al. [2019] for a related topic on deep linear network.

Moreover, Zhao et al. [2019], Vaškevičius et al. [2019] recently design a noval regularization-free algorithm and study the implicit regularization of gradient descent for high-dimensional linear regression with a sparse signal parameter, which is a vector in ${ℝ}^p$ with s nonzero entries. They propose to re-parametrize the parameter using two vectors in ${ℝ}^p$ via the Hadamard product and estimate the true parameter via un-regularized gradient descent with proper initialization, stepsizes, and the number of iterations. They prove independently that, with $n=\mathcal{O}\left(s^2\text{ log }p\right)$ i.i.d. observations, gradient descent yields an estimator of the true parameter with the optimal statistical accuracy. More interestingly, when the nonzero entries of the true parameter all have sufficiently large magnitude, the proposed estimator attains the oracle $\mathcal{O}(\sqrt{s\text{ log }s/n})$ rate that is independent of the ambient dimension p. Hence, for sparse linear regression, the implicit regularization of gradient descent has the same effect as the folded concave penalties [Fan et al., 2014] such as smoothly clipped absolute deviation (SCAD) [Fan and Li, 2001] and minimax concave penalty (MCP) [Zhang et al., 2010].

The aforementioned works all design algorithms and establish theoretical results for linear statistical models with light-tailed noise, which is slightly restricted since linear models with sub-Gaussian noise only comprise a small proportion of the models of interest in statistics. For example, in the field of finance, linear models only bring limited contributions and the datasets are always corrupted by heavy-tailed noise. Thus, one questions is left open:

Can we leverage over-parameterization and implicit regularization to establish statistically accurate estimation procedures for a more general class of high-dimensional statistical models with possibly heavy-tailed data?

In this work, we focus on the single index model, where the response variable Y and the covariate X satisfy Y = f(〈X, β*〉) + ϵ, with β* being the true parameter, ϵ being the random noise, and $f:ℝ\to ℝ$ being an unknown (nonlinear) link function. Here β* is either a s-sparse vector in ${ℝ}^p$ or a rank r matrix in ${ℝ}^{d\times d}$. Since f is unknown, the norm of β is not identifiable. Thus, for the vector and matrix cases respectively, we further assume that the ℓ₂- or Frobenius norms of β* are equal to one. Our goal is to recover the true parameter β* given n i.i.d. observations of the model. Such a model can be viewed as the misspecified version of the compressed sensing [Donoho, 2006, Candés, 2008] and phase retrieval [Shechtman et al., 2015, Candés et al., 2015] models, which corresponds to the identical and quadratic link functions respectively.

In a single index model, due to the unknown link function, it is infeasible to directly estimate β* via nonlinear least-squares. Moreover, jointly minimizing the least-squares loss function with respect to β* and f is computationally intractable. To overcome these challenges, a recent line of research proposes to estimate β* by the method of moments when the distribution of X is known. This helps us provide a deep understanding on the implicit regularization induced by over-parameterization in the nonlinear models without excessive technicality and eliminate other complicated factors that convolve insights. Specifically, when X is a standard Gaussian random variable, Stein’s identity [Stein et al., 1972] implies that the expectation of Y · X is proportional to β*. Thus, despite the nonlinear link function, β* can be accurately estimated by neglecting f and fitting a regularized least-squares regression. In particular, when β* is a sparse vector, Plan and Vershynin [2016], Plan et al. [2017] prove that the Lasso estimator achieves the optimal statistical rate of convergence. Subsequently, such an approach has been extended to the cases beyond Gaussian covariates. In particular, Goldstein et al. [2018], Wei [2018], Goldstein and Wei [2019] allow the covariates to follow an elliptically symmetric distribution that can be heavy-tailed. In addition, utilizing a generalized version of Stein’s identity [Stein et al., 2004], Yang et al. [2017] extends the Lasso approach to the setting where the covariate X has a known density p₀. Specifically, when p₀ is known, we can define the score function $S_{p_0}(\cdot )$ as $S_{p_0}(\cdot )=-\nabla \text{ log }{p}_0(\cdot )$, which enjoys the property that $\mathbb{E}\left[Y\cdot S_{p_0}(X)\right]$ identifies the direction of β*. Thus, the true parameter can be estimated via M-estimation with $S_{p_0}(X)$ serving as the covariate.

To answer the question given above, in this work, we leverage over-parameterization to design regularization-free algorithms for single index model and provide theoretical guarantees for the induced implicit regularization phenomenon. To be more specific, we first adopt the quadratic loss function in Yang et al. [2017] and rewrite the parameter of interest by over-parameterization. When β* is a sparse vector in ${ℝ}^p$, we adopt a Hadamard product parameterization [Hoff, 2017, Zhao et al., 2019, Vaškevičius et al., 2019] and write β* as w ⊙ w − v ⊙ v, where both w and v are vectors in ${ℝ}^p$. We propose to minimize the loss function as a function of the new parameters via gradient descent, where both w and v are initialized near an all-zero vector and the stepsizes are fixed to be a sufficiently small constant η > 0. Furthermore, when β* is a low-rank matrix, we similarly represent β* as WW^⊤ − VV^⊤ and propose to recover β* by applying the gradient descent algorithm to the quadratic loss function under the new parameterization.

Furthermore, the analysis of our algorithm faces the following two challenges. First, due to over-parameterization, there exist exponentially many stationary points of the population loss function that are far from the true parameter. Thus, it seems that the gradient descent algorithm would be likely to return a stationary point that incurs a large error. Second, both the response Y and the score $S_{p_0}(X)$ can be heavy-tailed random variables. Thus, the gradient of the empirical loss function can deviate significantly from its expectation, which poses an additional challenge to establishing the statistical error of the proposed estimator.

To overcome these difficulties, in our algorithm, instead of estimating $\mathbb{E}\left[Y\cdot S_{p_0}(X)\right]$ by its empirical counterpart, we construct robust estimators via proper truncation techniques, which have been widely applied in high-dimensional M-estimation problems with heavy-tailed data [Fan et al., 2021c, Zhu, 2017, Wei and Minsker, 2017, Minsker, 2018, Fan et al., 2021b, Ke et al., 2019, Minsker and Wei, 2020]. These robust estimators are then employed to compute the update directions of the gradient descent algorithm. Moreover, despite the seemingly perilous loss surface, we prove that, when initialized near the origin and sufficiently small stepsizes, the gradient descent algorithm guard the iterates from bad stationary points. More importantly, when the number of iterations is properly chosen, the obtained estimator provably enjoys (near-)optimal $\mathcal{O}(\sqrt{s\text{ log }p/n})$ and $\mathcal{O}(\sqrt{rd\text{ log }d/n})$ ℓ₂-statistical rates under the sparse and low-rank settings, respectively. Moreover, for sparse β*, when the magnitude of the nonzero entries is sufficiently large, we prove that our estimator enjoys an oracle $\mathcal{O}(\sqrt{s\text{ log }n/n})$ ℓ₂-statistical rate, which is independent of the dimensionality p. Our proof is based on a jointly statistical and computational analysis of the gradient descent dynamics. Specifically, we decompose the iterates into a signal part and a noise part, where the signal part share the same sparse or low-rank structures as the true signal and the noise part are orthogonal to the true signal. We prove that the signal part converges to the true parameter efficiently whereas the noise part accumulates at a rather slow rate and thus remains small for a sufficiently large number of iterations. Such a dichotomy between the signal and noise parts characterizes the implicit regularization of the gradient descent algorithm and enables us to establish the statistical error of the final estimator.

Furthermore, our method has several merits compared with classical regularized methods. From the theoretical perspective, our strengths are two-fold. First, as we mentioned in the last paragraph, under mild conditions, our estimator enjoys oracle statistical rate whereas the most commonly used ℓ₁-regularized method always results in large bias. In this case, our method is equivalent with adding folded-concave regularizers (e.g. SCAD, MCP) to the loss function. Second, for all estimators inside the wide optimal time interval, our range of choosing the truncating parameter to achieve variable selection consistency (rank consistency) is much wider than classical regularized methods. Thus, our method is more robust than all regularized methods in terms of selecting the truncating parameter. Meanwhile, from the aspect of applications, our strengths are three-fold. First, in terms of ℓ₂-statistical rate, numerical studies show that our method generalizes even better than adding folded-concave penalties. Second, from the aspect of variable selection, experimental results also show that the robustness of our method helps reduce false positive rates greatly. Last but not least, as we only need to run gradient descent and the gradient information is able to be efficiently transferred among different machines, our method is easier to be paralleled and generalized to large-scale problems. Thus, our method can be applied to modern machine learning applications such as federated learning.

To summarize, our contribution is several-fold. First, for sparse and low-rank single index models where the random noise is possible heavy-tailed, we employ a quadratic loss function based on a robust estimator of $\mathbb{E}\left[Y\cdot S_{p_0}(X)\right]$ and propose to estimate β* by combining over-parameterization and regularization-free gradient descent. Second, we prove that, when the initialization, stepsizes, and stopping time of the gradient descent algorithm are properly chosen, the proposed estimator achieves optimal statistical rates of convergence up to logarithm terms under both the sparse and low-rank settings. This captures the implicit regularization phenomenon induced by our algorithm. Third, in order to corroborate our theories, we did extensive numerical studies. The experimental results support our theoretical findings and also show that our method outperforms classical regularized methods in terms of both ℓ₂-statistical rates and variable selection consistency.

1.1. Related Works

Our work belongs to the recent line of research on understanding the implicit regularization of gradient-based optimization methods in various statistical models. In addition, our work is also closely related to the large body of literature on single index models. Due to the space limit, we defer the discussions on related works to Appendix A in the supplement.

1.2. Notation

In this subsection, we give an introduction to our notations. Throughout this work, we use [n] to denote the set {1, 2, …, n}. For a subset S in [n] and a vector u, we use u_S to denote the vector whose i-th entry is u_i if i ∈ S and 0 otherwise. For any vector u and q ⩾ 0, we use $\Vert \text{u}{\Vert }_{{\ell }_q}$ to represent the vector ℓ_q norm. In addition, the inner product 〈u, v〉 between any pair of vectors u, v is defined as the Euclidean inner product u^⊤v. Moreover, we define u⊙v as the Hardmard product of vectors u, v. For any given matrix $\text{X}\in {ℝ}^{d_1\times d_2}$, we use ∥X∥_op, ∥X∥_F and ∥X∥_* to represent the operator norm, Frobenius norm and nuclear norm of matrix X respectively. In addition, for any two matrices X, $\text{Y}\in {ℝ}^{d_1\times d_2}$, we define their inner product 〈X, Y〉 as 〈X, Y〉 = tr(X^⊤Y). Moreover, if we write X ≽ 0 or X ≼ 0, then the matrix X is meant to be positive semidefinite or negative semidefinite. We let {a_n, b_n}_n⩾1 be any two positive series. We write a_n ≲ b_n if there exists a universal constant C such that a_n ⩽ C · b_n and we write a_n ≪ b_n if a_n/b_n → 0. In addition, we write a_n ≍ b_n, if we have a_n ≲ b_n and b_n ≲ a_n and the notations of $a_n=\mathcal{O}\left(b_n\right)$ and a_n = o(b_n) share the same meaning with a_n ≲ b_n and a_n ≪ b_n. Moreover, $a_n=\tilde{\mathcal{O}}\left(b_n\right)$ means a_n ⩽ Cb_n up to some logarithm terms. Finally, we use a_n = Ω(b_n) if there exists a universal constant c > 0 such that a_n/b_n ⩾ c and we use a_n = Θ(b_n) if c ⩽ a_n/b_n ⩽ C where c, C > 0 are universal constants.

1.3. Roadmap

The organization of our paper is as follows. We introduce the background knowledge in §2. In §3 and §4 we investigate the implicit regularization effect of gradient descent in over-parameterized SIM under the vector and matrix settings, respectively. Extensive simulation studies are presented in §B to corroborate our theory.

2. Preliminaries

In this section, we introduce the phenomenon of implicit regularization via over-parameterization, high dimensional single index model, and generalized Stein’s identity [Stein et al., 2004].

2.1. Related Works on Implicit Regularization

Both Gunasekar et al. [2017] and Li et al. [2018] have studied least squares objectives over positive semidefinite matrices $\beta \in {ℝ}^{d\times d}$ of the following form

$$\underset{\beta ⩾0}{\text{min }}F(\beta )=\frac{1}{n}\sum _{i=1}^n{\left(y_i-\langle {\text{X}}_i,\beta \rangle \right)}^2,$$ (1)

where the labels ${\left\{y_i\right\}}_{i=1}^n$ are generated from linear measurements y_i = 〈X_i, β*〉, i ∈ [n], with ${\beta }^{*}\in {ℝ}^{d\times d}$ being positive semidefinite and low rank. Here β* is of rank r where r is much smaller than d.

Instead of working on parameter β directly, they write β as β = UU^⊤ where $\text{U}\in {ℝ}^{d\times d}$, and study the optimization problem related to U,

$$\underset{\mathbf{\text{U}}\in {ℝ}^{d\times d}}{\text{min }}f(\mathbf{\text{U}})=\frac{1}{2n}\sum _{i=1}^n{\left(y_i-\langle {\mathbf{\text{X}}}_i,\mathbf{\text{U}}{\mathbf{\text{U}}}^{\top }\rangle \right)}^2.$$ (2)

The least-squares problem in (2) is over-parameterized because here β is parameterized by U, which has d² degrees of freedom, whereas β*, being a rank-r matrix, has $\mathcal{O}(rd)$ degrees of freedom. Gunasekar et al. [2017] proves that when ${\left\{{\text{X}}_i\right\}}_{i=1}^m$ are commutative and U is properly initialized, if the gradient flow of (2) converges to a solution $\widehat{\text{U}}$ such that $\widehat{\beta }=\widehat{\text{U}}{\widehat{\text{U}}}^{\top }$ is a globally optimal solution of (1), then $\widehat{\text{U}}$ has the minimum nuclear norm over all global optima. Namely,

$$\begin{gathered} \widehat{\beta }\in \underset{\beta \succcurlyeq 0}{\text{argmin}}\Vert \beta {\Vert }_{*}, \\ \text{subject to }\langle {\mathbf{\text{X}}}_i,\widehat{\beta }\rangle =y_i,\forall i\in [n]. \end{gathered}$$

Subsequently, Li et al. [2018] assumes ${\left\{{\text{X}}_i\right\}}_{i=1}^n$ satisfy the restricted isometry property (RIP) condition [Candés, 2008] and proves that by applying gradient descent to (2) with the initialization close to zero and sufficiently small fixed stepsizes, the near exact recovery of β* is achieved.

Recently, Li et al. [2021] proves that the algorithm of gradient flow with infinitesimal initialization on the general covariate of (2) tends to be equivalent to the Greedy Low-Rank Learning (GLRL) algorithm, which is a greedy rank minimization algorithm. Results in Gunasekar et al. [2017] with commutable ${\left\{{\text{X}}_i\right\}}_{i=1}^m$ serves as a special case to Li et al. [2021].

As for noisy statistical model, both Zhao et al. [2019] and Vaškevičius et al. [2019] study over-parameterized high dimensional noisy linear regression problem independently. Specifically, here the response variables ${\left\{y_i\right\}}_{i=1}^n$ are generated from a linear model

$$y_i={\mathbf{\text{x}}}_i^{\top }{\beta }^{*}+{ϵ}_i,i\in [n],$$ (3)

where ${\beta }^{*}\in {ℝ}^p$ and ${\left\{{ϵ}_i\right\}}_{i=1}^n$ are i.i.d. sub-Gaussian random variables that are independent with the covariates ${\left\{{\text{x}}_i\right\}}_{i=1}^n$. Moreover, here β* has only s nonzero entries where s ≪ p. Instead of adding sparsity-enforcing penalties, they propose to estimate β* via gradient descent with respect to w, v on a loss function L,

$$\underset{\mathbf{\text{w}}\in {ℝ}^p,\mathbf{\text{v}}\in {ℝ}^p}{\text{min}}L(\mathbf{\text{w}},\mathbf{\text{v}})=\frac{1}{2n}\sum _{i=1}^n{\left[{\mathbf{\text{x}}}_i^{\top }(\mathbf{\text{w}}\odot \mathbf{\text{w}}-\mathbf{\text{v}}\odot \mathbf{\text{v}})-y_i\right]}^2,$$ (4)

where the parameter β is over-parameterized as β = w ⊙ w − v ⊙ v. Under the restricted isometry property (RIP) condition on the covariates, these works prove that, when the hyperparameters is proper selected, gradient descent on (4) finds an estimator of β* with optimal statistical rate of convergence.

2.2. High Dimensional Single Index Model

In this subsection, we first introduce the score functions associated with random vectors and matrices, which are utilized in our algorithms. Then we formally define the high dimensional single index model (SIM) in both the vector and matrix settings.

Definition 1. Let $\mathbf{\text{x}}\in {\mathbb{R}}^p$ be a random vector with density function $p_0(\text{x}):{ℝ}^p\to ℝ$. The score function $S_{p_0}(\text{x}):{ℝ}^p\to {ℝ}^p$ associated with x is defined as

$$S_{p_0}(\mathbf{\text{x}})≔-{\nabla }_{\text{x}}\text{log }{p}_0(\text{x})=-{\nabla }_{\text{x}}{p}_0(\text{x})/p_0(\text{x}).$$

Here the score function $S_{p_0}(\text{x})$ relies on the density function p₀(x) of the covariate x. In order to simplify the notations, in the rest of the paper, we omit the subscript p₀ from $S_{p_0}$ when the underlying distribution of x is clear to us.

Remark: If the covariate $\text{X}\in {ℝ}^{d\times d}$ is a random matrix whose entries are i.i.d. with a univariate density $p_0(x):ℝ\to ℝ$, we then define the score function $S(\text{X})\in {ℝ}^{d\times d}$ entrywisely. In other words, for any {i, j} ∈ [d] × [d], we obtain

$$S{(\mathbf{\text{X}})}_{i,j}≔-p_0^{\prime }\left({\mathbf{\text{X}}}_{i,j}\right)/p_0\left({\mathbf{\text{X}}}_{i,j}\right).$$ (5)

Next, we introduce the first-order general Stein’s identity.

Lemma 1. (First-Order General Stein’s Identity, [Stein et al., 2004]) We assume that the covariate $\text{x}\in {ℝ}^p$ follows a distribution with density function $p_0(\text{x}):{ℝ}^p\to ℝ$ which is differentiable and satisfies the condition that |p₀(x)| converges to zero as ∥x∥₂ goes to infinity. Then for any differentiable function f(x) with $\mathbb{E}[|f(\text{x})S(\text{x})|]\vee \mathbb{E}[\Vert {\nabla }_{\mathbf{\text{x}}}f{(\text{x})\Vert }_2]<\infty$, it holds that,

$$\mathbb{E}[f(\mathbf{\text{x}})S(\mathbf{\text{x}})]=\mathbb{E}\left[{\nabla }_{\mathbf{\text{x}}}f(\mathbf{\text{x}})\right],$$

where S(x) = −∇_xp₀(x)/p₀(x) is the score function with respect to x defined in Definition 1.

Remark: In the case of having matrix covariate, we are able to achieve the same conclusion by simply replacing $\text{x}\in {ℝ}^p$ by $\text{X}\in {ℝ}^{d\times d}$ in Lemma 1 with the definition of matrix score function in (5).

In the sequel, we introduce the single index models considered in this work. We first define sparse vector single index models as follows.

Definition 2. (Sparse Vector SIM) We assume the response $Y\in ℝ$ is generated from model

$$Y=f\left(\langle \mathbf{\text{x}},{\beta }^{*}\rangle \right)+ϵ,$$ (6)

with unknown link $f:ℝ\to ℝ$, p-dimensional covariate x as well as signal β* which is the parameter of interest. Here, we let $ϵ\in ℝ$ be an exogenous random noise with mean zero. In addition, if not particularly indicated, we assume the entries of x are i.i.d. random variables with a known univariate density p₀(x). As for the underlying true signal β*, it is assumed to be s-sparse with s ≪ p. Note that the length of β* can be absorbed by the unknown link f, we then let ∥β*∥₂ = 1 for model identifiability.

By the definition of sparse vector SIM, we notice that many well-known models are included in this category, such as linear regression $y_i={\text{x}}_i^{\top }{\beta }^{*}+ϵ$, phase retrieval $y_i={\left({\text{x}}_i^{\top }{\beta }^{*}\right)}^2+ϵ$, as well as one-bit compressed sensing $y=\text{sign}\left({\text{x}}_i^{\top }{\beta }^{*}\right)+ϵ$.

Finally, we define the low rank matrix SIM as follows.

Definition 3. (Symmetric Low Rank Matrix SIM) For the low rank matrix SIM, we assume the response $Y\in ℝ$ is generated from

$$Y=f\left(\langle \text{X},{\beta }^{*}\rangle \right)+ϵ,$$ (7)

in which ${\beta }^{*}\in {ℝ}^{d\times d}$ is a low rank symmetric matrix with rank r ≪ d and the link function f is unknown. For the covariate $\text{X}\in {ℝ}^{d\times d}$, we assume the entries of X are i.i.d. with a known density p₀(x). Besides, since ∥β*∥_F can be absorbed in the unknown link function f, we further assume ∥β*∥_F = 1 for model identifiability. In addition, the noise term ϵ is also assumed additive and mean zero.

As we have discussed in the introduction, almost all existing literature designs algorithms and studies the corresponding implicit regularization phenomenon in linear models with sub-Gaussian data. The scope of this work is to leverage over-parameterization to design regularization-free algorithms and delineate the induced implicit regularization phenomenon for a more general class of statistics models with possibly heavy-tailed data. Specifically, in §3 and §4, we design algorithms and capture the implicit regularization induced by the gradient descent algorithm for over-parameterized vector and matrix SIMs, respectively.

3. Main Results for Over-Parameterized Vector SIM

Leveraging our conclusion from Lemma 1 as well as our definition of sparse vector SIM in Definitions 2, we have

$$\mathbb{E}[Y\cdot S(\mathbf{\text{x}})]=\mathbb{E}\left[f\left(\langle \text{x},{\beta }^{*}\rangle \right)\cdot S(\text{x})\right]=\mathbb{E}\left[f^{\prime }\left(\langle \text{x},{\beta }^{*}\rangle \right)\right]\cdot {\beta }^{*}≔{\mu }^{*}{\beta }^{*},$$

which recovers our true signal β* up to scaling. Here we define ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{x},{\beta }^{*}\rangle \right)\right]$, which is assumed nonzero throughout this work. Hence, Y · S(x) serves as an unbiased estimator of μ*β*, and we can correctly identify the direction of β* by solving a population level optimization problem:

$$\underset{\beta }{\text{min }}L(\beta )≔\langle \beta ,\beta \rangle -2\langle \beta ,\mathbb{E}[Y\cdot S(\text{x})]\rangle .$$

Since we only have access to finite data, we replace $\mathbb{E}[Y\cdot S(\text{x})]$ by its sample version estimator $\frac{1}{n}{\sum }_{i=1}^n{y}_{i}S\left({\text{x}}_i\right)$, and plug the sample-based estimator into the loss function. In a high dimensional SIM given in Definition 2, where the true signal β* is assumed to be sparse, various works [Plan and Vershynin, 2016, Plan et al., 2017, Yang et al., 2017] have shown that the ℓ₁-regularized estimator $\widehat{\beta }$ given by

$$\widehat{\beta }\in \underset{\beta }{\text{argmin }}L(\beta )≔\langle \beta ,\beta \rangle -2\langle \beta ,\frac{1}{n}\sum _{i=1}^n{y}_{i}S\left({\text{x}}_i\right)\rangle +\lambda \Vert \beta {\Vert }_1$$ (8)

attains the optimal statistical rate of convergence rate to μ*β*.

In contrast, instead of imposing an ℓ₁-norm regularization term, we propose to obtain an estimator by minimizing the loss function L directly, with β re-parameterized using two vectors w and v in ${ℝ}^p$. Specifically, we write β as β = w ⊙ w − v ⊙ v and thus equivalently write the loss function L(β) as L(w, v), which is given by

$$L(\text{w},\text{v})=\langle \text{w}\odot \text{w}-\text{v}\odot \text{v},\text{w}\odot \text{w}-\text{v}\odot \text{v}\rangle -2\langle \text{w}\odot \text{w}-\text{v}\odot \text{v},\frac{1}{n}\sum _{i=1}^n{y}_{i}S\left({\text{x}}_i\right)\rangle .$$ (9)

Note that the way of writing β in terms of w and v is not unique. In particular, β has p degrees of freedom but we use 2p parameters to represent β. Thus, by using w and v instead of β, we employ over-parameterization in (9).

We briefly describe our motivation on over-parameterizing β by w ⊙ w – v ⊙ v. Suppose that β is sparse, an explicit regularization is to use ℓ₁-penalty. Note that ∥β∥₁ = min_γ⊙δ=β{∥γ∥² + ∥δ∥²}/2, where ⊙ denotes the Hadamard (componentwise) product. Thus, an explicit regularization is to ${\text{min}}_{\gamma ,\delta }{\sum }_{i=1}^n{\left\{Y_i-f\left({\text{x}}_i^T\gamma \odot \delta \right)\right\}}^2+\lambda \left\{\Vert \gamma {\Vert }^2+\Vert \delta {\Vert }^2\right\}$ for a penalty parameter λ, following the method in Hoff [2017]. To gain understanding on implicit regularization by over parametrization, we let w = (γ + δ)/2 and v = (γ − δ)/2. Then β = γ⊙δ = w⊙w − v⊙v with 2p new parameters w and v that over parameterize the problem. This leads to the empirical loss $L(\text{w},\mathbf{\text{v}})={\sum }_{i=1}^n{\left\{Y_i-f\left({\text{x}}_i^T(\text{w}\odot \text{w}-\text{v}\odot \text{v})\right)\right\}}^2$. Following the neural network training, we drop the explicit penalty and run the gradient decent to minimize L(w, v).

To be more specific, for the sparse SIM, we propose to construct an estimator of β* by applying gradient descent to L in (9) with respect to w and v, without any explicit regularization. Such an estimator, if achieves desired statistical accuracy, demonstrates the efficacy of implicit regularization of gradient descent in over-parameterized sparse SIM. Specifically, the gradient updates for the vector (w^⊤, v^⊤)^⊤ for solving (9) are given by

$${\text{w}}_{t+1}={\text{w}}_t-\eta {\nabla }_{w}L\left({\text{w}}_t,{\text{v}}_t\right)={\text{w}}_t-\eta \left({\text{w}}_t\odot {\text{w}}_t-{\text{v}}_t\odot {\text{v}}_t-\frac{1}{n}\sum _{i=1}^{n}S\left({\text{x}}_i\right)y_i\right)\odot {\text{w}}_t,$$ (10)

$${\text{v}}_{t+1}={\text{v}}_t+\eta {\nabla }_{\text{v}}L\left({\text{w}}_t,{\text{v}}_t\right)={\text{v}}_t+\eta \left({\text{w}}_t\odot {\text{w}}_t-{\text{v}}_t\odot {\text{v}}_t-\frac{1}{n}\sum _{i=1}^{n}S\left({\text{x}}_i\right)y_i\right)\odot {\text{v}}_t.$$ (11)

Here η > 0 is the stepsize. By the parameterization of β, {w_t, v_t}_t⩾0 leads to a sequence of estimators {β_t}_t⩾0 given by

$${\beta }_{t+1}={\text{w}}_{t+1}\odot {\text{w}}_{t+1}-{\text{v}}_{t+1}\odot {\text{v}}_{t+1}.$$ (12)

Meanwhile, in terms of chooisng initial values, since the zero vector is a stationary point of the algorithm, we cannot set the initial values of w and v to the zero vector. To utilize the structure of β*, ideally we would like to initialize w and v such that they share the same sparsity pattern as β*. That is, we would like to set the entries in the support of β* to nonzero values, and set those outside of the support to zero. However, such an initialization scheme is infeasible since the support of β* is unknown. Instead, we initialize w₀ and v₀ as w₀ = v₀ = α · 1_p×1, where α > 0 is a small constant and 1_p×1 is an all-one vector in ${ℝ}^p$. By setting w₀ = v₀, we equivalently set β₀ to the zero vector. And more importantly, such a construction provides a good compromise: zero components get nearly zero initializations, which are the majority under the sparsity assumption, and nonzero components get nonzero initializations. Even though we initialize every component at the same value, the nonzero components move quickly to their stationary component, while zero components remain small. This is how over-parameterization differentiate active components from inactive components. We illustrate this by a simulation experiment.

A simulation study.

In this simulation, we fix sample size n = 1000, dimension p = 2000, number of non-zero entries s = 5. Let $S≔\left\{i:\left|{\beta }_i^{*}\right|>0\right\}$. The responses ${\left\{y_i\right\}}_{i=1}^n$ are generated from y_i = f(〈x, β*〉) + ϵ_i, i ∈ [n] with link functions f₁(x) = x (linear regression) and f₂(x) = sin(x). Here we assume β* is s-sparse with ${\beta }_i=1/\sqrt{s}$, i ∈ S, and ${\left\{{\text{x}}_i\right\}}_{i=1}^n$ are standard Gaussian random vectors. We over-parameterize β as w⊙w − v⊙v and set w₀ = v₀ = 10⁻⁵ · 1_p×1. Then we update w, v and β regarding equations (10), (11), and (12) with stepsize η = 0.01. The evolution of the distance between our unnormalized iterates β_t and μ*β*, trajectories of β_j,t for j ∈ S and ${\text{max}}_{j\in S^c}\left|{\beta }_{j,t}\right|$ are depicted in Figures 1 and 2.

Figure 1:

With link function f(x) = x, (a) characterizes the evolution of distance ${\Vert {\beta }_t-{\mu }^{*}{\beta }^{*}\Vert }_2^2$ against iteration number t; (b) depicts the trajectories β_j,t (j ∈ S) for five nonzero components, and (c) presents the trajectory ${\text{max}}_{j\in S^c}\left|{\beta }_{j,t}\right|$.

Figure 2:

With link function f(x) = sin(x), similar to Figure 1, here (a) characterizes the evolution of distance ${\Vert {\beta }_t-{\mu }^{*}{\beta }^{*}\Vert }_2^2$ against iteration number t; (b) depicts the trajectories β_j,t (j ∈ S) for five nonzero components, and (c) presents the trajectory ${\text{max}}_{j\in S^c}\left|{\beta }_{j,t}\right|$.

From the simulation results given in Figure 1-(a) and Figure 2-(a), we notice that there exists a time interval, where we can nearly recover μ*β*. From plots (b) in Figures 1 and 2, we can see with over-parameterization, five nonzero components all increase rapidly and converge quickly to their stationary points. Meanwhile, the maximum estimation error for inactive component, represented by ${\Vert {\beta }_{S^c,t}\Vert }_{\infty }$, still remains small, as shown in Figure 1-(c) and Figure 2-(c). In other words, running gradient descent with respect to over-parameterized parameters helps us distinguish non-zero components from zero components, while applying gradient descent to the ordinary loss can not.

It is worth noting that, with over-parameterization, there are Ω(2^p) stationary points of L satisfying ∇_wL(w, v) = ∇_vL(w, v) = 0_p×1, where 0_p×1 is the zero vector. To see this, for any subset I ⊆ [p], we define vectors $\overline{\mathbf{\text{w}}}$ and $\overline{\mathbf{\text{v}}}$ as follows. For any j ∉ I, we set the j-th entries of $\overline{\mathbf{\text{w}}}$ and $\overline{\mathbf{\text{v}}}$ to zero. Meanwhile, for any j ∈ I, we choose ${\overline{\mathbf{\text{w}}}}_j$ and ${\overline{\mathbf{\text{v}}}}_j$ such that ${\overline{\mathbf{\text{w}}}}_j^2-{\overline{\mathbf{\text{v}}}}_j^2=n^{-1}{{\sum }_{i=1}^{n}S\left({\text{x}}_i\right)}_j{y}_i$, where ${\overline{\mathbf{\text{w}}}}_j$, ${\overline{\mathbf{\text{v}}}}_j$, and S(x_i)_j are the j-th entries of $\overline{\mathbf{\text{w}}}$, $\overline{\mathbf{\text{v}}}$, and S(x_i), respectively. By direct computation, it can be shown that $(\overline{\mathbf{\text{w}}},\overline{\mathbf{\text{v}}})$ is a stationary point of L, and thus there are at least 2^p stationary points. However, our numerical results demonstrate that not all of these stationary points are likely to be found by the gradient descent algorithm — gradient descent favors the stationary points that correctly recover μ*β*. Such an intriguing observation captures the implicit regularization induced by the optimization algorithm and over-parameterization.

3.1. Gaussian Design

In this subsection, we discuss over-parameterized SIM with Gaussian covariates. In this subsection, we assume the distribution of x in (6) is N(μ, Σ), where both μ and Σ are assumed known. Moreover, only in this subsection, we slightly modify the identifiability condition in Definition 2 from assuming ∥β*∥₂ = 1 to ∥Σ^1/2β*∥₂ = 1.

3.1.1. Theoretical Results for Gaussian Covariates

We first introduce an structural assumption on the SIM.

Assumption 1. Assume that ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{x},{\beta }^{*}\rangle \right)\right]\ne 0$ is a constant and the following two conditions hold.

1. Covariance matrix Σ is positive-definite and has bounded spectral norm. To be more specific, there exist constants C_min and C_max such that $C_{\text{min}}{\mathbb{I}}_{p\times p}\preccurlyeq \sum ⩽C_{\text{max}}{\mathbb{I}}_{p\times p}$ holds, where ${\mathbb{I}}_{p\times p}$ is the identity matrix.

2. Both ${\left\{f\left(\langle {\text{x}}_i,{\beta }^{*}\rangle \right)\right\}}_{i=1}^n$ and ${\left\{{ϵ}_i\right\}}_{i=1}^n$ are i.i.d. sub-Gaussian random variables, with the sub-Gaussian norms denoted by $\Vert f{\Vert }_{{\psi }_2}=\mathcal{O}(1)$ and $\sigma =\mathcal{O}(1)$ respectively. Here we let $\Vert f{\Vert }_{{\psi }_2}$ denote the sub-Gaussian norm of f(〈x_i, β*〉). In addition, we further assume that $\left|{\mu }^{*}\right|/\Vert f{\Vert }_{{\psi }_2}=\Theta (1)$, |μ*|/σ = Ω(1).

The score function for the Gaussian distribution N(μ, Σ) is S(x) = Σ⁻¹(x – μ) and Assumption 1-(a) makes the Gaussian distributed covariates non-degenerate. Assumption 1-(b) enables the the empirical estimator $n^{-1}{\sum }_{i=1}^n{y}_{i}S\left({\text{x}}_i\right)$ to concentrate to its expectation μ*β*, and also sets a lower bound to the signal noise ratio |μ*|/σ. Note that this assumption is quite standard and easy to be satisfied by a broad class of models as long as there exists a lower bound on the signal noise ratio, which include models with link functions f(x) = x, sin x, tanh(x), and etc. In addition, in §3.2, the assumption that both f(〈x, β*〉) and the noise ϵ are sub-Gaussian random variables will be further relaxed to simply assuming they have bounded finite moments with perhaps heavy-tailed distributions.

We present the details of the proposed method for the Gaussian case in Algorithm 1. In the following, we present the statistical rates of convergence for the estimator constructed by Algorithm 1. Let us divide the support seta $S=\left\{i:\left|{\beta }_i^{*}\right|>0\right\}$ into $S_0=\left\{i:\left|{\beta }_i\right|⩾C_s\sqrt{\text{log }p/n}\right\}$ and $S_1=\left\{i:0<\left|{\beta }^{*}\right|<C_s\sqrt{\text{log }p/n}\right\}$, which correspond to the sets of strong and weak signals, respectively. Here C_s is an absolute constant. We let s₀ and s₁ be the cardinalities of S₀ and S₁, respectively. In addition, we let $s_m={\text{min}}_{i\in S_0}\left|{\beta }_i^{*}\right|$ be the smallest value of strong signals.

Theorem 1. Apart from Assumption 1, if we further let our initial value α satisfy $0<\alpha ⩽M_0^2/p$ and set stepsize η as 0 < η ⩽ 1/(12(|μ*| + M₀)) in Algorithm 1 with M₀ being a constant proportional to $\text{max}\left\{\Vert f{\Vert }_{{\psi }_2},\sigma \right\}$, there exist absolute constants a₁, a₂ > 0 such that, with probability at least 1 − 2p⁻¹ − 2n⁻², we have

$${\Vert {\beta }_{T_1}-{\mu }^{*}{\beta }^{*}\Vert }_2^2\lesssim \frac{s_0\text{log }n}{n}+\frac{s_1\text{log }p}{n},$$

for all $T_1\in \left[a_1\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|s_m-M_0\sqrt{\text{log }p/n}\right)\right),a_2\text{ log}(1/\alpha )\sqrt{n/\text{log }p}/\left(\eta M_0\right)\right]$. Meanwhile, the statistical rate of convergence for the normalized iterates are given by

$${\Vert \frac{{\beta }_{T_1}}{{\Vert {\Sigma }^{1/2}{\beta }_{T_1}\Vert }_2}-\frac{{\mu }^{*}{\beta }^{*}}{\left|{\mu }^{*}\right|}\Vert }_2^2\lesssim \frac{s_0\text{ log }n}{n}+\frac{s_1\text{ log }p}{n}.$$

Theorem 2. (Variable Selection Consistency) Under the setting of Theorem 1, for all

$$T_1\in \left[a_1\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|s_m-M_0\sqrt{\text{log }p/n}\right)\right),a_2\text{ log}(1/\alpha )\sqrt{n/\text{log }p}/\left(\eta M_0\right)\right],$$

we let ${\left[{\tilde{\beta }}_{T_1}\right]}_i={\left[{\beta }_{T_1}\right]}_i\cdot {\mathbb{I}}_{\left|{\left[{\beta }_{T_1}\right]}_i\right|⩾\lambda }$, for all i ∈ [p]. Then, with probability at least 1 – 2p⁻¹ – 2n⁻², for all $\lambda \in \left[\alpha ,\left(C_s\left|{\mu }^{*}\right|-2M_0\right)\sqrt{\text{log }p/n}\right]$, we have $\mathit{\text{supp}}\left({\tilde{\beta }}_{T_1}\right)\subset \mathit{\text{supp}}\left({\beta }^{*}\right)$. Moreover, when there only exists strong signals in S₀. We further have $\mathit{\text{supp}}\left({\tilde{\beta }}_{T_1}\right)=\mathit{\text{supp}}\left({\beta }^{*}\right)$ and $\mathit{\text{sign}}\left({\tilde{\beta }}_{T_1}\right)=\mathit{\text{sign}}\left({\beta }^{*}\right)$.

Theorem 1 shows that if we just have strong signals, then with high probability, for any $T_1\in \left[a_1\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|s_m-M_0\sqrt{\text{log }p/n}\right)\right),a_2\text{ log}(1/\alpha )\sqrt{n/\text{log }p}/\left(\eta M_0\right)\right]$, we get the oracle statistical rate $\mathcal{O}(\sqrt{s\text{ log }n/n})$ in terms of the ℓ₂-norm, which is independent of the ambient dimension p. Besides, when β* also consists of weak signals, we achieve $\mathcal{O}(\sqrt{s\text{ log }p/n})$ statistical rate in terms of the ℓ₂-norm, where s is the sparsity of β*. Such a statistical rate matches the minimax rate of sparse linear regression [Raskutti et al., 2011] and is thus minimax optimal. Notice that the oracle rate is achievable via explicit regularization using folded concave penalties [Fan et al., 2014] such as SCAD [Fan and Li, 2001] and MCP [Zhang et al., 2010]. Thus, Theorem 1 shows that, with over-parameterization, the implicit regularization of gradient descent has the same effect as adding a folded concave penalty function to the loss function in (9) explicitly.

Furthermore, comparing our work to Plan and Vershynin [2016], Plan et al. [2017], which study high dimensional SIM with ℓ₁-regularization, thanks to the implicit regularization phenomenon, we avoid bias brought by the ℓ₁-penalty and attain the oracle statistical rate. Moreover, our another advantage over regularized methods is shown in Theorem 2. It shows that by properly truncating ${\beta }_{T_1}$ when T₁ falls in the optimal time interval, we are able to recover the support of β* with high probability. Comparing to existing literatures on support recovery via using explicit regularization on single index model [Neykov et al., 2016], our method offers a wider range for choosing tuning parameter λ with a known left boundary α, instead of only using $\lambda =\Theta (\sqrt{\text{log }p/n})$. This efficiently reduces false discovery rate, see §D.1 for more details. Last but not least, as we only need to run gradient descent, comparing to regularized methods, it is easier to parallel our algorithm since the gradient information is able to be efficiently transferred among different machines. The use of implicit regularization allows our methodology to be generalized to large-scale problems easily [McMahan et al., 2017, Richards and Rebeschini, 2020, Richards et al., 2020]. The detailed discussions are given in §C.5.

Theorem 1 and Theorem 2 generalizes the results in Zhao et al. [2019] and Vaškevičius et al. [2019] for the linear model to high-dimensional SIMs. In addition, to satisfy the RIP condition, their sample complexity is at least $\mathcal{O}\left(s^2\text{ log }p\right)$ if their covariate x follows the Gaussian distribution. Whereas, by using the loss function in (9) motivated by the Stein’s identity [Stein et al., 1972, 2004], the RIP condition is unnecessary in our analysis. Instead, our theory only requires that $n^{-1}{\sum }_{i=1}^{n}S\left({\text{x}}_i\right)\cdot y_i$ concentrates at a fast rate. As a result, our sample complexity is $\mathcal{O}(s\text{ log }p)$ for ℓ₂-norm consistency, which is better than $\mathcal{O}\left(s^2\text{ log }p\right)$.

The ideas of proof behind Theorem 1 and Theorem 2 are as follows. First, we are able to control the strengths of error component, denoted by ${\Vert {\beta }_t\odot 1_{S^c}\Vert }_{\infty }$, at the same order with the square root of their initial values until $\mathcal{O}\left(\text{log}(1/\alpha )\cdot \sqrt{n/\text{log }p}/\left(\eta M_0\right)\right)$ steps. This gives us the right boundary of the stopping time T₁. Meanwhile, every entry of strong signal part ${\beta }_t\odot 1_{S_0}$ grows at exponential rates to $ϵ=\mathcal{O}(\sqrt{\text{log }n/n})$ accuracy around ${\mu }^{*}{\beta }^{*}\odot 1_{S_0}$ within $\mathcal{O}\left(\text{log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|s_m-M_0\sqrt{\text{log }p/n}\right)\right)\right)$ steps, which offers us the left boundary of the stopping time T₁. Finally, we prove for weak signals, their strengths will not exceed $\mathcal{O}(\sqrt{\text{log }p/n})$ for all steps as long as we properly choose the stepsize. Thus, by letting the stopping time T₁ be in the interval given in Theorem 1, we obtain converged signal component and well controlled error component. The final statistical rates are obtained by combining the results on the active and inactive components together. Moreover, the conclusion of Theorem 2 holds by truncating the β_t properly, since we are able to control the error component of β_t uniformly as mentioned above. See Appendix §E.1 for the detail. As shown in the proof, we observe that with small initialization and over-parameterized loss function, the signal component converges rapidly to the true signal, while the the error component grows in a relatively slow pace. Thus, gradient descent rapidly isolates the signal components from the noise, and with a proper stopping time, finds a near-sparse solution with high statistical accuracy. Thus, with proper initialization, over-parameterization plays the role of an implicit regularization by favoring approximately sparse saddle points of the loss function in (9).

Finally, we remark that Theorem 1 establishes optimal statistical rates for the estimator ${\beta }_{T_1}$, where T₁ is any stopping time that belongs to the interval given in Theorem 1. However, in practice, such an interval is infeasible to compute as it depends on unknown constants. To make the proposed method practical, in the following, we introduce a method for selecting a proper stopping time T₁.

3.1.2. Choosing the Stopping Time T₁

We split the dataset into training data and testing data. We utilize the training data to implement Algorithm 1 and get the estimator β_t as well as the value of the training loss (9) at step t. We notice β_t varies slowly inside the optimal time interval specified in Theorem 1, so that the fluctuation of the training loss (9) can be smaller than a threshold. Based on that, we choose m testing points on the flatted curve of the training loss (9) and denote their corresponding number of iterations as {t_j}, j ∈ [m]. For each j ∈ [m], we then reuse the training data and normalized estimator ${\beta }_{t_j}/{\Vert {\sum }^{1/2}{\beta }_{t_j}\Vert }_2$, j ∈ [m] to fit the link function f. Let the obtained estimator be ${\widehat{f}}_j$. For the testing dataset, we perform out-of-sample prediction and get m prediction losses:

$$l_j=\frac{1}{n_{\text{test}}}\sum _{i=1}^{n_{\text{test}}}{\left[Y_i-{\widehat{f}}_j\left(\langle {\text{x}}_i,{\beta }_{t_j}/{\Vert {\Sigma }^{1/2}{\beta }_{t_j}\Vert }_2\rangle \right)\right]}^2,\forall j\in [m].$$

Next, we choose T₁ as $t_{j^{*}}$ where we define j* = argmin_j∈[m] l_j.

We remark that each ${\widehat{f}}_j$ can be obtained by any nonparametric regression methods. To show case our method, in the following, we apply univariate kernel regression to obtain each ${\widehat{f}}_j$ and establish its theoretical guarantee.

3.1.3. Prediction Risk

We now consider estimating the nonparametric component and the prediction risk. Suppose we are given an estimator $\widehat{\beta }$ of β and n i.i.d. observations ${\left\{y_i,{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$ of the model. For simplicity of the technical analysis, we assume that $\widehat{\beta }$ is independent of ${\left\{y_i,{\mathbf{\text{x}}}_i\right\}}_{i=1}^n$, which can be achieved by data-splitting. Moreover, we assume that $\widehat{\beta }$ is an estimator of β* such that

$${\Vert \widehat{\beta }-{\beta }^{*}\Vert }_2=o\left(n^{-1/3}\right),{\Vert {\Sigma }^{1/2}\widehat{\beta }\Vert }_2=1,\text{ and }{\Vert {\Sigma }^{1/2}{\beta }^{*}\Vert }_2=1.$$ (13)

Our goal is to construct an estimate the regression function f(〈·, β*〉) based on $\widehat{\beta }$ and ${\left\{y_i,{\text{x}}_i\right\}}_{i=1}^n$.

Note that, when β* is known, we can directly estimate f based on y_i and $Z_i^{*}≔{\text{x}}_i^{\top }{\beta }^{*}$, i ∈ [n] via standard non-parametric regression. When $\widehat{\beta }$ is accurate, a direct idea is to replace $Z_i^{*}$ by $Z_i≔{\text{x}}_i^{\top }\widehat{\beta }$ and follow the similar route. For a new observation x, we define Z as $Z≔{\mathbf{\text{x}}}^{\top }\widehat{\beta }$ and Z* as Z* ≔ x^⊤ β* respectively.

To predict Y, we estimate function g(z) using kernel regression with data ${\left\{\left(y_i,{\text{x}}_i^{\top }\widehat{\beta }\right)\right\}}_{i=1}^n$. Specifically, we let the function K_h(u) be K_h(u) ≔ 1/h · K(u/h), in which $K:ℝ\to ℝ$ is a kernel function with $K(u)={\mathbb{I}}_{\left\{\left|u\right|⩽1\right\}}$ and h is a bandwidth. By the definitions of Z*, Z, and Z_i, i ∈ [n] given above, the prediction function $\widehat{g}(Z)$ is defined as

$$\widehat{g}(Z)=\{\begin{array}{cc}\frac{{\sum }_{i=1}^n{y}_i{K}_h\left(Z-Z_i\right)}{{\sum }_{i=1}^n{K}_h\left(Z-Z_i\right)},& \left|Z-{\mu }^{\top }\widehat{\beta }\right|⩽R,\\ 0,& \text{otherwise},\end{array}$$ (14)

where we follow the convention that 0/0 = 0. In what follows, we consider the ℓ₂-prediction risk of $\widehat{g}$, which is given by

$$\mathbb{E}\left[{\left\{\widehat{g}\left(\langle \text{x},\widehat{\beta }\rangle \right)-f\left(\langle \text{x},{\beta }^{*}\rangle \right)\right\}}^2\right],$$

where the expectation is taken with respect to x and ${\left\{{\text{x}}_i,y_i\right\}}_{i=1}^n$. Before proceeding to the theoretical guarantees, we make the following assumption on the regularity of f.

Assumption 2. There exists an α₁ > 0 and a constant C > 0 such that |f(x)|, $\left|f^{\prime }(x)\right|⩽C+|x{|}^{{\alpha }_1}$.

For the rationality of the Assumption 2, we note that the constraint on f′(x) and f(x) given above is weaker than assuming f′(x) and f(x) are bounded functions directly. Next, we present Theorem 3 which characterizes the convergence rate of mean integrated error of our prediction function $\widehat{g}(Z)$.

Theorem 3. If we set $R=2\sqrt{\text{log}(n)}$ and h ≍ n^−1/3 in (14), under Assumption 2, the ℓ₂-prediction risk of $\widehat{g}$ defined in (14) is given by

$$\mathbb{E}\left[{\left\{\widehat{g}\left(\langle \text{x},\widehat{\beta }\rangle \right)-f\left(\langle \text{x},{\beta }^{*}\rangle \right)\right\}}^2\right]\lesssim \frac{\mathit{\text{polylog}}(n)}{n^{2/3}},$$

where $\widehat{\beta }$ is any vector that satisfies (13) and polylog(n) contains terms that are polynomials of log n.

It is worth noting that the estimator $\widehat{\beta }={\beta }_{T_1}/{\Vert {\Sigma }^{1/2}{\beta }_{T_1}\Vert }_2$ constructed in Theorem 1 with any T₁ belongs to the optimal time interval given in Theorem 1 satisfy (13). Thus, under such regimes, Theorem 3 also holds. The proof of Theorem 3 is given in §E.3. Note that it is possible to refine the analysis on the prediction risk for f with higher order derivatives by utilizing higher order kernels (see Tsybakov [2008] therein) this is not the key message of our paper.

3.2. General Design

In this subsection, we extend our methodology to the setting with covariates generated from a general distribution. Following our discussions at the beginning of §3, ideally we aim at solving the loss function with over-parameterized variable given in (9). However, when the distribution of x has density p₀, the score S(x) can be heavy-tailed such that $\mathbb{E}[Y\cdot S(\text{x})]$ and its empirical counterpart may not be sufficiently close.

To remedy this issue, we modify the loss function in (9) by replacing y_i and S(x_i) by their truncated (Winsorized) version $\check{y_i}$ and $\check{S}$, respectively. Specifically, we propose to apply gradient descent to the following modified loss function with respect to u and v:

$$\underset{\text{w},\text{v}}{\text{min }}L(\text{w},\text{v})≔\langle \text{w}\odot \text{w}-\text{v}\odot \text{v},\text{w}\odot \text{w}-\text{v}\odot \text{v}\rangle -\frac{2}{n}\sum _{i=1}^n\check{y_i}\langle \text{w}\odot \text{w}-\text{v}\odot \text{v},\check{S}\left({\text{x}}_i\right)\rangle .$$ (15)

Let $\check{\text{a}}\in {ℝ}^d$ denote the truncated version of vector $\text{a}\in {ℝ}^d$ based on a parameter τ [Fan et al., 2021b]. That is, its entries are given by ${[\check{\text{a}}]}_j={[\text{a}]}_j$ if |a_i| ⩽ τ and τ otherwise. Applying elementwise truncation to ${\left\{y_i\right\}}_{i=1}^n$ and ${\left\{S\left({\text{x}}_i\right)\right\}}_{i=1}^n$ in (15), we allow the score S(x) and the response Y to both have heavy-tailed distributions. By choosing a proper threshold τ, such a truncation step ensures $n^{-1}{\sum }_{i=1}^n{\check{y}}_i\check{S}\left({\text{x}}_i\right)$ converge to $\mathbb{E}[Y\cdot S(\text{x})]$ with a desired rate in ℓ_∞-norm. Compared with Algorithm 1, here we only modify the definition of the loss function. Thus, we defer the details of the proposed algorithm for this setting to Algorithm 3 in §E.5.

Before stating our main theorem, we first present an assumption on the distributions of the covariate and the response variables.

Assumption 3. Assume there exists a constant M such that

$$\mathbb{E}\left[Y^4\right]⩽M,\mathbb{E}\left[S{(\text{x})}_j^4\right]⩽M,\forall j\in [p].$$

Here S(x)_j is the j-th entry of S(x). Moreover, recall that we denote ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{x},{\beta }^{*}\rangle \right)\right]$. We assume that μ* is a nonzero constant such that M/|μ*| = Θ(1).

Assuming the fourth moments exist and are bounded is significant weaker than the sub-Gaussian assumption. Moreover, such an assumption is prevalent in robust statistics literature [Fan et al., 2021c, 2018, 2019]. Now we are ready to introduce the theoretical results for the setting with general design.

Theorem 4. Under Assumption 3, we set the thresholding parameter τ = (M · n/log p)^1/4/2, let the initialization parameter α satisfy $0<\alpha ⩽M_g^2/p$, and set the stepsize η such that 0 < η ⩽ 1/(12(|μ*| + M_g)) in Algorithm 3 given in §E.5 where M_g is a constant proportional to M. There exist absolute constants a₃, a₄, such that, with probability at least 1 − 2p⁻²,

$${\Vert {\beta }_{T_1}-{\mu }^{*}{\beta }^{*}\Vert }_2^2\lesssim \frac{s\text{ log }p}{n}$$

holds for all $T_1\in \left[a_3\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|s_m-M_g\sqrt{\text{log }p/n}\right)\right),a_4\text{ log}(1/\alpha )\sqrt{n/\text{log }p}/\left(\eta M_g\right)\right]$. Here s is the cardinality of the support set S and $s_m={\text{min}}_{i\in S_0}\left|{\beta }_j^{*}\right|$, where $S_0=\left\{j\in i:\left|{\beta }_i\right|⩾C_s\sqrt{\text{log }p/n}\right\}$ is the set of strong signals. In addition, for the normalized iterates, we further have

$${\Vert \frac{{\beta }_{T_1}}{{\Vert {\beta }_{T_1}\Vert }_2}-\frac{{\mu }^{*}{\beta }^{*}}{\left|{\mu }^{*}\right|}\Vert }_2^2\lesssim \frac{s\text{ log }p}{n},$$

with probability at least 1 − 2p⁻².

Compared with Theorem 1 for the Gaussian design, here we achieve the $\mathcal{O}(\sqrt{s\text{ log }p/n})$ statistical rate of convergence in terms of the ℓ₂-norm. These rates are the same of those achieved by adding an ℓ₁-norm regularization explicitly [Plan and Vershynin, 2016, Plan et al., 2017, Yang et al., 2017] and are minimax optimal [Raskutti et al., 2011]. Moreover, we note that here S(x) and Y can be both heavy-tailed and our truncation procedure successfully tackles such a challenge without sacrificing the statistical rates. Moreover, similar to the Gaussian case, here C_s can be set as a sufficiently large absolute constant, and the statistical rates established in Theorem 4 holds for all choices of C_s. In addition, for heavy-tailed case, we also let ${\left[{\tilde{\beta }}_{T_1}\right]}_i={\left[{\beta }_{T_1}\right]}_i\cdot {\mathbb{I}}_{\left|{\left[{\beta }_{T_1}\right]}_i\right|⩾\lambda }$, for all i ∈ [p]. Then for all $\lambda \in \left[\alpha ,\left(C_s\left|{\mu }^{*}\right|-2M_g\right)\sqrt{\text{log }p/n}\right]$, we obtain similar theoretical guarantees as in Theorem 2.

4. Main Results for Over-Parametrized Low Rank SIM

In this section, we present the results for over-parameterized low rank matrix SIM introduced in Definition 3 with both standard Gaussian and generally distributed covariates. Similar to the results in §3, here we also focus on matrix SIM with first-order links, i.e., we assume that ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{X},{\beta }^{*}\rangle \right)\right]\ne 0$, where β* is a low rank matrix with rank r. Note that we assume that the entries of covariate $\text{X}\in {ℝ}^{d\times d}$ are i.i.d. with a univariate density p₀. Also recall that we define the score function $S(\text{X})\in {ℝ}^{d\times d}$ in (5). Then, similar to the loss function in (9), we consider the loss function

$$L(\beta )≔\langle \beta ,\beta \rangle -2\langle \beta ,\frac{1}{n}\sum _{i=1}^n{y}_{i}S\left({\text{X}}_i\right)\rangle ,$$

where $\beta \in {ℝ}^{d\times d}$ is a symmetric matrix. Hereafter, we rewrite β as WW^⊤ − VV^⊤, where both W and V are matrices in ${ℝ}^{d\times d}$. The intuitions of re-parameterizing β = WW^⊤ − VV^⊤ are as follows. Any (low rank) symmetric matrix is able to be written as the difference of two positive semidefinite matrices, namely WW^⊤ − VV^⊤ with W, $\text{V}\in {ℝ}^{d\times d}$. Re-parameterizing the symmetric matrix this way is a generalization of re-parameterizing its eigenvalues by the Hadamard products. Thus this can be regarded as an extension of the re-parameterization mechanism from the vector case to the spectral domain. With such an over-parameterization, we propose to estimate β* by applying gradient descent to the loss function

$$L(\text{W},\text{V})≔\langle \text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top },\text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top }\rangle -2\langle \text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top },\frac{1}{n}\sum _{i=1}^n{y}_{i}S\left({\text{X}}_i\right)\rangle .$$ (16)

Since the rank of β* is unknown, we initialization W₀ and V₀ as ${\text{W}}_0={\text{V}}_0=\alpha \cdot {\mathbb{I}}_{d\times d}$ for a small α > 0 and construct a sequence of iterates {W_t, V_t, β_t}_t⩾0 via the gradient decent method as follows:

$${\text{W}}_{t+1}={\text{W}}_t-\eta \left({\text{W}}_t{\text{W}}_t^{\top }-{\text{V}}_t{\text{V}}_t^{\top }-\frac{1}{2n}\sum _{i=1}^{n}S\left({\text{X}}_i\right)y_i-\frac{1}{2n}\sum _{i=1}^{n}S{\left({\text{X}}_i\right)}^{\top }{y}_i\right){\text{W}}_t,$$ (17)

$$\begin{gathered} {\text{V}}_{t+1}={\text{V}}_t+\eta \left({\text{W}}_t{\text{W}}_t^{\top }-{\text{V}}_t{\text{V}}_t^{\top }-\frac{1}{2n}\sum _{i=1}^{n}S\left({\text{X}}_i\right)y_i-\frac{1}{2n}\sum _{i=1}^{n}S{\left({\text{X}}_i\right)}^{\top }{y}_i\right){\text{V}}_t, \\ {\beta }_{t+1}={\text{W}}_t{\text{W}}_t^{\top }-{\text{V}}_t{\text{V}}_t^{\top }, \end{gathered}$$ (18)

where η in (17) and (18) is the stepsize. Note that here the algorithm does not impose any explicit regularization. In the rest of this section, we show that such a procedure yields an estimator of the true parameter β* with near-optimal statistical rates of convergence.

Similar to the vector case, for theoretical analysis, here we also divide eigenvalues of β* into different groups by their strengths. We let $r_i^{*}$, i ∈ [d] be the i-th eigenvalue of β*. The support set R of the eigenvalues is defined as $R≔\left\{i:\left|r_i^{*}\right|>0\right\}$, whose cardinality is r. We then divide the support set R into $R_0≔\left\{i:\left|r_i^{*}\right|⩾C_{\mathit{\text{ms}}}\sqrt{d\text{ log }d/n}\right\}$ and $R_1≔\left\{i:0<\left|r_i^{*}\right|<C_{\mathit{\text{ms}}}\sqrt{d\text{ log }d/n}\right\}$, which correspond to collections of strong and weak signals with cardinality denoting by r₀ and r₁, respectively. Here C_ms > 0 is an absolute constant and we have R = R₀ ⋃ R₁. Moreover, we use r_m to denote the minimum strong eigenvalue in magnitude, i.e. $r_m={\text{min}}_{i\in R_0}\left|r_i^{*}\right|$.

4.1. Gaussian Design

In this subsection, we focus on the model in (7) with the entries of covariate X being i.i.d. N(0, 1) random variables. In this case, S(X_i) = X_i. This leads to Algorithm 4 given in §F.1, where we place S(X_i) by X_i in (16)–(18).

Similar to the case in §3.1, here we also impose the following assumption for the function class of the low rank SIM.

Assumption 4. We assume that ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{X},{\beta }^{*}\rangle \right)\right]$ is a nonzero constant. Moreover, we assume that both ${\left\{f\left(\langle {\text{X}}_i,{\beta }^{*}\rangle \right)\right\}}_{i=1}^n$ and ${\left\{{ϵ}_i\right\}}_{i=1}^n$ are i.i.d. sub-Gaussian random variables, with sub-Gaussian norm denoted by $\Vert f{\Vert }_{{\psi }_2}=\mathcal{O}(1)$ and $\sigma =\mathcal{O}(1)$ respectively. Here we let $\Vert f{\Vert }_{{\psi }_2}$ denote the sub-Gaussian norm of f(〈X, β*〉). In addition, we further assume $\left|{\mu }^{*}\right|/\Vert f{\Vert }_{{\psi }_2}=\Theta (1)$, |μ*|/σ = Ω(1).

The following theorem establishes the statistical rates of convergence for the estimator constructed by Algorithm 4.

Theorem 5. We set and stepsize $0<\alpha ⩽M_m^2/d$ and stepsize $0<\eta ⩽1/\left[12\left(\left|{\mu }^{*}\right|+M_m\right)\right]$ in Algorithm 4, where M_m is a constant proportional to $\text{max}\left\{\Vert f{\Vert }_{{\psi }_2},\sigma \right\}$. Under Assumption 4, there exist constants a₅, a₆ such that, with probability at least 1 – 1/(2d) – 3/n², we have

$${\Vert {\beta }_{T_1}-{\mu }^{*}{\beta }^{*}\Vert }_F^2\lesssim \frac{rd\text{ log }d}{n}$$

for all $T_1\in \left[a_5\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|r_m-M_m\sqrt{d\text{ log }d/n}\right)\right),a_6\text{ log}(1/\alpha )\sqrt{n/(d\text{ log }d)}/\left(\eta M_m\right)\right]$. Moreover, for the normalized iterates β_t/∥β_t∥_F, we have

$${\Vert \frac{{\beta }_{T_1}}{{\Vert {\beta }_{T_1}\Vert }_F}-\frac{{\mu }^{*}{\beta }^{*}}{\left|{\mu }^{*}\right|}\Vert }_F^2\lesssim \frac{rd\text{ log }d}{n}.$$

Similar to the vector case given in §3.1, as shown in the proof in Appendix §F, here we require C_ms to satisfy C_ms ⩾ max{(a₅/a₆ + 1)M_m{|μ*|, 2M_m/|μ*|} in order to let the strong signals in R₀ dominate the noise and let the interval for T₁ to exist. The statistical rates hold for all such a C_ms. As shown in Theorem 5, with the proper choices of initialization parameter α, stepsize η, and the stopping time T₁, Algorithm 4 constructs an estimator that achieves near-optimal statistical rates of convergence (up to logarithmic factors compared to minimax lower bound [Rohde and Tsybakov, 2011]). Notice that the statistical rates established in Theorem 5 are also enjoyed by the M-estimator based on the least-squares loss function with nuclear norm penalty [Plan and Vershynin, 2016, Plan et al., 2017]. Thus, in terms of statistical estimation, applying gradient descent to the over-parameterized loss function in (16) is equivalent to adding a nuclear norm penalty explicitly, hence demonstrating the implicit regularization effect. Except for obtaining the optimal ℓ₂-statistical rate, we are able to recover the true rank with high-probability by properly truncating the eigenvalues of ${\beta }_{T_1}$ for all $T_1\in \left[a_5\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|r_m-M_m\sqrt{d\text{ log }d/n}\right)\right),a_6\text{ log}(1/\alpha )\sqrt{n/(d\text{ log }d)}/\left(\eta M_m\right)\right]$. Comparing with the literature Lee et al. [2015] which studies the rank consistency via ℓ₁-regularization, we offer a wider range for choosing the tuning parameter with known left boundary α, instead of only setting the nuclear tuning parameter $\lambda =\tilde{\Theta }(\sqrt{rd/n})$.

Theorem 6. (Rank Consistency) Under the setting of Theorem 5, for all

$$T_1\in \left[a_5\text{log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|r_m-M_m\sqrt{d\text{ log }d/n}\right)\right),a_6\text{ log}(1/\alpha )\sqrt{n/(d\text{ log }d)}/\left(\eta M_m\right)\right],$$

we let ${\tilde{\beta }}_{T_1}={\sum }_{i=1}^d{\text{u}}_i{\text{u}}_i^{\top }{\lambda }_i\left({\beta }_{T_1}\right)\cdot {\mathbb{I}}_{\left\{\left|{\lambda }_i\left({\beta }_{T_1}\right)\right|⩾\lambda \right\}}$, for all i ∈ [d]. Here u_k, k ∈ [d] are eigenvectors of ${\beta }_{T_1}$. Then, with probability at least 1 − 2d⁻¹ − 3n⁻², for all $\lambda \in \left[\alpha ,\left(C_{\mathit{\text{ms}}}\left|{\mu }^{*}\right|-2M_m\right)\sqrt{d\text{ log }d/n}\right]$, we have ${\tilde{\beta }}_{T_1}$ enjoys the conclusion of Theorem 5, and $\mathit{\text{rank}}\left({\tilde{\beta }}_{T_1}\right)⩽\mathit{\text{rank}}\left({\beta }^{*}\right)$. Moreover, when there only exists strong signals in R₀, we further have $\mathit{\text{rank}}\left({\tilde{\beta }}_{T_1}\right)=\mathit{\text{rank}}\left({\beta }^{*}\right)$.

Furthermore, our method extends the existing works that focus on designing algorithms and studying implicit regularization phenomenon in noiseless linear matrix sensing models with positive semidefinite signal matrices [Gunasekar et al., 2017, Li et al., 2018, Arora et al., 2019, Gidel et al., 2019]. Specifically, we allow a more general class of (noisy) models and symmetric signal matrices. Compared with Li et al. [2018], our methodology possesses several strengths, which include achieving low sample complexity ($\tilde{\mathcal{O}}(rd)$ insted of $\tilde{\mathcal{O}}(r^{2}d)$., allowing weak signals (${\text{min}}_{i\in R}\left|r_i^{*}\right|z\mathcal{O}\left({(1/n)}^{1/2}\right)$ instead of ${\text{min}}_{i\in R}\left|r_i^{*}\right|\gtrsim \mathcal{O}\left({(1/n)}^{1/6}\right)$), getting tighter statistical rate under noisy models ($\tilde{\mathcal{O}}(dr/n)$ instead of $\tilde{\mathcal{O}}(\kappa rd/n)$), and applying to a more general class of noisy statistical models. These strengths are achieved by the use of score transformation together with a refined trajectory analysis, which involves studying the dynamics of eigenvalues inside the strong signal set elementwisely with multiple stages instead of only studying the dynamics of the minimum eigenvalue with two stages.

The way of choosing stopping time T₁ in the case of matrix SIM is almost the same with our method in §3.1.2. The only difference between them is that here we replace x^⊤β* by tr(X^⊤β*) Indeed, as we assume ∥Σ^1/2β*∥₂ = 1 in vector SIM and ∥β*∥_F = 1 in matrix version for model identifiability, both x^⊤β_t and tr(X^⊤β_t) follow the standard normal distribution. Thus, our results on the prediction risk in §3.1.3 can be applied here directly.

4.2. General Design

In the rest of this section, we focus on the low rank matrix SIM beyond Gaussian covariates. Hereafter, we assume the entries of X are i.i.d. random variables with a known density function $p_0:ℝ\to ℝ$. Recall that, according to the remarks following Definition 1, the score function $S(\text{X})\in {ℝ}^{d\times d}$ is defined as

$$S{(\text{X})}_{j,k}≔S\left({\text{X}}_{j,k}\right)=-{p^{\prime }}_0\left({\text{X}}_{j,k}\right)/p_0\left({\text{X}}_{j,k}\right),$$

where S(X)_j,k and X_j,k are the (j, k)-th entries of S(X) and X for all j, k ∈ [d]. However, similar to the results in §3.2, the entries of S(X) can have heavy-tailed distributions and thus $n^{-1}{\sum }_{i=1}^n{y}_i\cdot S\left({\text{X}}_i\right)$ may not converge its expectation $\mathbb{E}[Y\cdot S(\text{X})]$ efficiently in terms of spectral norm. Here X_i is the i-th observation of the covariate X. To tackle such a challenge, we employ a shrinkage approach [Catoni et al., 2012, Fan et al., 2021c, Minsker, 2018] to construct a robust estimator of $\mathbb{E}[Y\cdot S(\text{X})]$. Specifically, we let

$$\varphi (x)=\{\begin{array}{cc}\text{log}\left(1-x+x^2/2\right),& x⩽0,\\ \text{log}\left(1+x+x^2/2\right),& x>0\end{array},$$

which is approximately x when x is small and grows at logarithmic rate for large x. The rescaled version λ⁻¹ϕ(λx) for λ → 0 behaves like a soft-winsorizing function, which has been widely used in statistical mean estimation with finite bounded moments [Catoni et al., 2012, Brownlees et al., 2015]. For any matrix $\text{X}\in {ℝ}^{d\times d}$, we apply spectral decomposition to its Hermitian dilation and obtain

$${\mathbf{\text{X}}}^{*}≔\left[\begin{array}{cc}0& \mathbf{\text{X}}\\ {\mathbf{\text{X}}}^{\top }& 0\end{array}\right]=\mathbf{\text{Q}}{\mathbf{\Sigma }}^{*}{\mathbf{\text{Q}}}^{\top },$$

where ${\Sigma }^{*}\in {ℝ}^{2d\times 2d}$ is a diagonal matrix. Based on such a decomposition, we define $\tilde{X}=\text{Q}\varphi \left({\Sigma }^{*}\right){\text{Q}}^{\top }$, where ϕ applies elementwisely to Σ*. Then we write $\tilde{X}$ as a block matrix as

$$\tilde{X}≔\left[\begin{array}{cc}{\tilde{X}}_{11}& {\tilde{X}}_{12}\\ {\tilde{X}}_{21}& {\tilde{X}}_{22}\end{array}\right],$$

where each block of $\tilde{X}$ is in ${ℝ}^{d\times d}$. We further define a mapping ${\varphi }_1:{ℝ}^{d\times d}\to {ℝ}^{d\times d}$ by letting ${\varphi }_1(\text{X})≔{\tilde{X}}_{12}$, which is a regularized version of X. Given data y₁, X₁, we finally define $\mathcal{H}(\cdot )$ as

$$\mathcal{H}\left(y_{1}S\left({\text{X}}_1\right),\kappa \right)≔1/\kappa \cdot {\varphi }_1\left(\kappa y_1\cdot S\left({\text{X}}_1\right)\right),\forall \kappa >0,$$ (19)

where κ is a thresholding parameter, converging to zero. This method is in a similar spirit of robustifying the singular value of X. Based on the operator $\mathcal{H}$ defined in (19), we define a loss function L(W, V) as

$$L(\text{W},\text{V})≔\langle \text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top },\text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top }\rangle -\frac{2}{n}\sum _{i=1}^n\langle \text{W}{\text{W}}^{\top }-\text{V}{\text{V}}^{\top },\mathcal{H}\left(y_{i}S\left({\text{X}}_i\right),\kappa \right))\rangle .$$ (20)

After over-parameterizing β as WW^⊤ − VV^⊤, we propose to construct an estimator of β* by applying gradient descent on the following loss function in (20) with respect to W, V. See Algorithm 5 in §F.5 for the details of the algorithm.

In the following, we present the statistical rates of convergence for the obtained estimator. We first introduce the assumption on Y and p₀.

Assumption 5. We assume that both the response variable Y and entries of S(X) have bounded fourth moments. Specifically, there exists an absolute constant M such that

$$\mathbb{E}\left[Y^4\right]⩽M,\mathbb{E}\left[S{(\text{X})}_{i,j}^4\right]⩽M,\forall (i,j)\in [d]\times [d].$$

Moreover, we assume that ${\mu }^{*}=\mathbb{E}\left[f^{\prime }\left(\langle \text{X},{\beta }^{*}\rangle \right)\right]$ is a nonzero constant such that |μ*|/M = Θ(1).

Next, we present the main theorem for low rank matrix SIM.

Theorem 7. In Algorithm 5, we set parameter κ in (19) as $\kappa =\sqrt{\text{log}(4d)/(nd\cdot M)}$ and let the initialization parameter α and the stepsize η satisfy $0<\alpha ⩽M_{mg}^2/d$ and $0<\eta ⩽1/\left[12\left(\left|{\mu }^{*}\right|+M_{mg}\right)\right]$, where M_mg is a constant proportional to M. Then, under Assumption 5, there exist absolute constants a₇, a₈ such that, with probability at least 1 – (4d)⁻², we have

$${\Vert {\beta }_{T_1}-{\mu }^{*}{\beta }^{*}\Vert }_F^2\lesssim \frac{rd\text{ log }d}{n},$$

for all $T_1\in \left[a_7\text{ log}(1/\alpha )/\left(\eta \left(\left|{\mu }^{*}\right|r_m-M_{mg}\sqrt{d\text{ log }d/n}\right)\right),a_8\text{ log}(1/\alpha )\sqrt{n/(d\text{ log }d)}/\left(\eta M_{mg}\right)\right]$, Moreover, for the normalized iterate β_t/∥β_t∥_F, we have

$${\Vert \frac{{\beta }_{T_1}}{{\Vert {\beta }_{T_1}\Vert }_F}-\frac{{\mu }^{*}{\beta }^{*}}{\left|{\mu }^{*}\right|}\Vert }_F^2\lesssim \frac{rd\text{ log }d}{n}.$$

For low rank matrix SIM, when the hyperparameters of the gradient descent algorithm are properly chosen, we also capture the implicit regularization phenomenon by applying a simple optimization procedure to over-parameterized loss function with heavy-tailed measurements. Here, applying the thresholding operator $\mathcal{H}$ in (19) can also be viewed as a data pre-processing step, which arises due to handling heavy-tailed observations. Note that the way of choosing C_ms here is similar with the way in Theorem 5, in order to ensure the convergence rate and existence of a time interval, so we omit the details. Note that the ℓ₂-statistical rate given in Theorem 7 are minimax optimal up to a logarithmic term [Rohde and Tsybakov, 2011]. Similar results were also obtained by Plan and Vershynin [2016], Yang et al. [2017], Goldstein et al. [2018], Na et al. [2019] via adding explicit nuclear norm regularization. Thus, in terms of statistical recovery, when employing the thresholding in (19) and over-parameterization, gradient descent enforces implicit regularization that has the same effect as the nuclear norm penalty. In addition, in terms of the rank consistency result for the heavy-tailed case, if we also let ${\tilde{\beta }}_{T_1}={\sum }_{i=1}^d{\text{u}}_i{\text{u}}_i^{\top }{\lambda }_i\left({\beta }_{T_1}\right)\cdot {\mathbb{I}}_{\left\{\left|{\lambda }_i\left({\beta }_{T_1}\right)\right|⩾\lambda \right\}}$, then for all $\lambda \in \left[\alpha ,\left(C_s\left|{\mu }^{*}\right|-2M_{\mathit{\text{mg}}}\right)\sqrt{d\text{ log }d/n}\right]$, we achieve the same results with Theorem 6.

5. Conclusion

In this paper, we leverage over-parameterization to design regularization-free algorithms for single index model and provide theoretical guarantees for the induced implicit regularization phenomenon. We consider the case where the link function is unknown, the distribution of the covariates is known as a prior, and the signal parameter is either a s-sparse vector in ${ℝ}^p$ or a rank-r matrix in ${ℝ}^{d\times d}$. Using the score function and the Stein’s identity, we propose an over-parameterized nonlinear least-squares loss function. To handle the possibly heavy-tailed distributions of the score functions and the response variables, we adopt additional truncation techniques that robustify the loss function. For both the vector and matrix SIMs, we construct an estimator of the signal parameter by applying gradient descent to the proposed loss function, without any explicit regularization. We prove that, when initialized near the origin, gradient descent with a small stepsize finds an estimator that enjoys minimax-optimal statistical rates of convergence. Moreover, for vector SIM with Gaussian design, we further obtain the oracle statistical rates that are independent of the ambient dimension. Furthermore, our experimental results support our theoretical findings and also demonstrate that our methods empirically outperform classical methods with explicit regularization in terms of both ℓ₂-statistical rate and variable selection consistency.