Benign overfitting in linear regression

Abstract

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.

Deep learning methodology has revealed a surprising statistical phenomenon: overfitting can perform well. The classical perspective in statistical learning theory is that there should be a tradeoff between the fit to the training data and the complexity of the prediction rule. Whether complexity is measured in terms of the number of parameters, the number of nonzero parameters in a high-dimensional setting, the number of neighbors averaged in a nearest neighbor estimator, the scale of an estimate in a reproducing kernel Hilbert space, or the bandwidth of a kernel smoother, this tradeoff has been ubiquitous in statistical learning theory. Deep learning seems to operate outside the regime where results of this kind are informative since deep neural networks can perform well even with a perfect fit to the training data.

As one example of this phenomenon, consider the experiment illustrated in figure 1C in ref. 1: standard deep network architectures and stochastic gradient algorithms, run until they perfectly fit a standard image classification training set, give respectable prediction performance, even when significant levels of label noise are introduced. The deep networks in the experiments reported in ref. 1 achieved essentially zero cross-entropy loss on the training data. In statistics and machine learning textbooks, an estimate that fits every training example perfectly is often presented as an illustration of overfitting [“…interpolating fits…[are] unlikely to predict future data well at all” (ref. 2, p. 37)]. Thus, to arrive at a scientific understanding of the success of deep learning methods, it is a central challenge to understand the performance of prediction rules that fit the training data perfectly.

In this paper, we consider perhaps the simplest setting where we might hope to witness this phenomenon: linear regression. That is, we consider quadratic loss and linear prediction rules, and we assume that the dimension of the parameter space is large enough that a perfect fit is guaranteed. We consider data in an infinite-dimensional space (a separable Hilbert space), but our results apply to a finite-dimensional subspace as a special case. There is an ideal value of the parameters, ${\theta }^{*}$, corresponding to the linear prediction rule that minimizes the expected quadratic loss. We ask when it is possible to fit the data exactly and still compete with the prediction accuracy of ${\theta }^{*}$. Since we require more parameters than the sample size in order to fit exactly, the solution might be underdetermined, and therefore, there might be many interpolating solutions. We consider the most natural: choose the parameter vector $\widehat{\theta }$ with the smallest norm among all vectors that gives perfect predictions on the training sample. (This corresponds to using the pseudoinverse to solve the normal equations; see below.) We ask when it is possible to overfit in this way—and embed all of the noise of the labels into the parameter estimate $\widehat{\theta }$—without harming prediction accuracy.

Our main result is a finite sample characterization of when overfitting is benign in this setting. The linear regression problem depends on the optimal parameters ${\theta }^{*}$ and the covariance $\mathrm{\Sigma }$ of the covariates $x$. The properties of $\mathrm{\Sigma }$ turn out to be crucial since the magnitude of the variance in different directions determines both how the label noise gets distributed across the parameter space and how errors in parameter estimation in different directions in parameter space affect prediction accuracy. There is a classical decomposition of the excess prediction error into two terms. The first is rather standard: provided that the scale of the problem (that is, the sum of the eigenvalues of $\mathrm{\Sigma }$) is small compared with the sample size $n$, the contribution to $\widehat{\theta }$ that we can view as coming from ${\theta }^{*}$ is not too distorted. The second term is more interesting since it reflects the impact of the noise in the labels on prediction accuracy. We show that this part is small if and only if the effective rank of $\mathrm{\Sigma }$ in the subspace corresponding to low-variance directions is large compared with $n$. This necessary and sufficient condition of a large effective rank can be viewed as a property of significant overparameterization: fitting the training data exactly but with near-optimal prediction accuracy occurs if and only if there are many low-variance (and hence, unimportant) directions in parameter space where the label noise can be hidden.

The details are more complicated. The characterization depends in a specific way on two notions of effective rank, $r$ and $R$; the smaller one, $r$, determines a split of $\mathrm{\Sigma }$ into large and small eigenvalues, and the excess prediction error depends on the effective rank as measured by the larger notion $R$ of the subspace corresponding to the smallest eigenvalues. For the excess prediction error to be small, the smallest eigenvalues of $\mathrm{\Sigma }$ must decay slowly.

Studying the patterns of eigenvalues that allow benign overfitting reveals an interesting role for large but finite dimensions: in an infinite-dimensional setting, benign overfitting occurs only for a narrow range of decay rates of the eigenvalues. On the other hand, it occurs with any suitably slowly decaying eigenvalue sequence in a finite-dimensional space with dimension that grows faster than the sample size. Thus, for linear regression, data that lie in a large but finite-dimensional space exhibit the benign overfitting phenomenon with a much wider range of covariance properties than data that lie in an infinite-dimensional space.

The phenomenon of interpolating prediction rules has been an object of study by several authors over the last two years since it emerged as an intriguing mystery at the Simons Institute program on Foundations of Machine Learning in the spring of 2017. Belkin et al. (3) described an experimental study demonstrating that this phenomenon of accurate prediction for functions that interpolate noisy data also occurs for prediction rules chosen from reproducing kernel Hilbert spaces and explained the mismatch between this phenomenon and classical generalization bounds. Belkin et al. (4) gave an example of an interpolating decision rule—simplicial interpolation—with an asymptotic consistency property as the input dimension gets large. That work and subsequent work of Belkin et al. (5) studied kernel smoothing methods based on singular kernels that both interpolate and, with suitable bandwidth choice, give optimal rates for nonparametric estimation [building on earlier consistency results (6) for these unusual kernels]. Liang and Rakhlin (7) considered minimum norm interpolating kernel regression with kernels defined as nonlinear functions of the Euclidean inner product and showed that, with certain properties of the training sample (expressed in terms of the empirical kernel matrix), these methods can have good prediction accuracy. Belkin et al. (8) studied experimentally the excess risk as a function of the dimension of a sequence of parameter spaces for linear and nonlinear classes.

Subsequent to our work, ref. 9 considered the properties of the interpolating linear prediction rule with minimal expected squared error. After this work was presented at the NAS Colloquium on the Science of Deep Learning (10), we became aware of the concurrent work of Belkin et al. (11) and of Hastie et al. (12). Belkin et al. (11) calculated the excess risk for certain linear models (a regression problem with identity covariance and sparse optimal parameters, both with and without noise, and a problem with random Fourier features with no noise), and Hastie et al. (12) considered linear regression in an asymptotic regime, where sample size $n$ and input dimension $p$ go to infinity together with asymptotic ratio $p/n\to \gamma$. They assumed that, as $p$ gets large, the empirical spectral distribution of $\mathrm{\Sigma }$ (the discrete measure on its set of eigenvalues) converges to a fixed measure, and they applied random matrix theory to explore the range of behaviors of the asymptotics of the excess prediction error as $\gamma$, the noise variance, and the eigenvalue distribution vary. They also studied the asymptotics of a model involving random nonlinear features. In contrast, we give upper and lower bounds on the excess prediction error for arbitrary finite sample size, for arbitrary covariance matrices, and for data of arbitrary dimension.

The next section introduces notation and definitions used throughout the paper, including definitions of the problem of linear regression and of various notions of effective rank of the covariance operator. The following section gives the characterization of benign overfitting, illustrates why the effective rank condition corresponds to significant overparameterization, and presents several examples of patterns of eigenvalues that allow benign overfitting, suggesting that slowly decaying covariance eigenvalues in input spaces of growing but finite dimension are the generic example of benign overfitting. Then we discuss the connections between these results and the benign overfitting phenomenon in deep neural networks and outline the proofs of the results.

Definitions and Notation

We consider linear regression problems, where a linear function of covariates $x$ from a (potentially infinite-dimensional) Hilbert space $\mathbb{H}$ is used to predict a real-valued response variable $y$. We use vector notation, so that $x^{\top }\theta$ denotes the inner product between $x$ and $\theta$ and $x{z}^{\top }$ denotes the tensor product of $x,z\in \mathbb{H}$.

Definition 1 (Linear Regression): A linear regression problem in a separable Hilbert space $\mathbb{H}$ is defined by a random covariate vector $x\in \mathbb{H}$ and outcome $y\in \mathbb{R}$. We define

• 1)the covariance operator $\mathrm{\Sigma }=\mathbb{E}\left[x{x}^{\top }\right]$ and
• 2)the optimal parameter vector ${\theta }^{*}\in \mathbb{H}$, satisfying $\mathbb{E}{\left(y-x^{\top }{\theta }^{*}\right)}^2=\underset{\theta }{min}\mathbb{E}{\left(y-x^{\top }\theta \right)}^2$.

We assume that

• 1)$x$ and $y$ are mean zero;
• 2)$x=V{\mathrm{\Lambda }}^{1/2}z$, where $\mathrm{\Sigma }=V\mathrm{\Lambda }{V}^{\top }$ is the spectral decomposition of $\mathrm{\Sigma }$ and $z$ has components that are independent ${\sigma }_x^2$ sub-Gaussian with ${\sigma }_x$ a positive constant: that is, for all $\lambda \in \mathbb{H}$,

$$\mathbb{E}\left[\exp \left({\lambda }^{\top }z\right)\right]\le \exp \left({\sigma }_x^2{\Vert \lambda \Vert }^2/2\right),$$

• where $\Vert \cdot \Vert$ is the norm in the Hilbert space $\mathbb{H}$;

• 3)the conditional noise variance is bounded below by some constant ${\sigma }^2$,

$$\mathbb{E}\left[{\left(y-x^{\top }{\theta }^{*}\right)}^2|x\right]\ge {\sigma }^2;$$

• 4)$y-x^{\top }{\theta }^{*}$ is ${\sigma }_y^2$ sub-Gaussian conditionally on $x$: that is, for all $\lambda \in \mathbb{R}$,

$$\mathbb{E}\left[\exp \left(\lambda \left(y-x^{\top }{\theta }^{*}\right)\right)|x\right]\le \exp \left({\sigma }_y^2{\lambda }^2/2\right)$$

• (note that this implies $\mathbb{E}\left[y|x\right]=x^{\top }{\theta }^{*}$); and

• 5)almost surely, the projection of the data $X$ on the space orthogonal to any eigenvector of $\mathrm{\Sigma }$ spans a space of dimension $n$.

Given a training sample $\left(x_1,y_1\right),\dots ,\left(x_n,y_n\right)$ of $n$ independent pairs with the same distribution as $\left(x,y\right)$, an estimator returns a parameter estimate $\theta \in \mathbb{H}$. The excess risk of the estimator is defined as

$$R\left(\theta \right)≔\hspace{0.17em}{\mathbb{E}}_{x,y}\left[{\left(y-x^{\top }\theta \right)}^2-{\left(y-x^{\top }{\theta }^{*}\right)}^2\right],$$

where ${\mathbb{E}}_{x,y}$ denotes the conditional expectation given all random quantities other than $x,y$ (in this case, given the estimate $\theta$). Define the vectors $\mathit{y}\in {\mathbb{R}}^n$ with entries $y_i$ and $\mathit{\epsilon }\in {\mathbb{R}}^n$ with entries ${\epsilon }_i=y_i-x_i^{\top }{\theta }^{*}$. We use infinite matrix notation: $X$ denotes the linear map from $\mathbb{H}$ to ${\mathbb{R}}^n$ corresponding to $x_1,\dots ,x_n\in \mathbb{H}$ so that $X\theta \in {\mathbb{R}}^n$ has $i$th component $x_i^{\top }\theta$. We use similar notation for the linear map $X^{\top }$ from ${\mathbb{R}}^n$ to $\mathbb{H}$.

Notice that Assumptions 1 to 5 are satisfied when $x$ and $y$ are jointly Gaussian with zero mean and $rank\left(\mathrm{\Sigma }\right)>n$.

We shall be concerned with situations where an estimator $\theta$ can fit the data perfectly: that is, $X\theta =\mathit{y}$. Typically, this implies that there are many such vectors. We consider the interpolating estimator with minimal norm in $\mathbb{H}$. We use $\Vert \cdot \Vert$ to denote both the Euclidean norm of a vector in ${\mathbb{R}}^n$ and the Hilbert space norm.

Definition 2 (Minimum Norm Estimator): Given data $X\in {\mathbb{H}}^n$ and $\mathit{y}\in {\mathbb{R}}^n$, the minimum norm estimator $\widehat{\theta }$ solves the optimization problem

$$\begin{array}{ccc}\hfill \underset{\theta \in \mathbb{H}}{min}\hspace{1em}\hspace{1em}& \hfill & {∥\theta ∥}^2\hfill \\ \hfill \text{such that}\hspace{1em}\hspace{1em}& \hfill & {∥X\theta -\mathit{y}∥}^2=\underset{\beta }{min}{∥X\beta -\mathit{y}∥}^2.\hfill \end{array}$$

By the projection theorem, parameter vectors that solve the least squares problem $\underset{\beta }{min}{∥X\beta -\mathit{y}∥}^2$ solve the normal equations, and therefore, we can equivalently write $\widehat{\theta }$ as the minimum norm solution to the normal equations

$$\begin{array}{ccc}\hfill \widehat{\theta }& =\hfill & \arg \hspace{0.17em}\underset{\theta }{min}\left\{{∥\theta ∥}^2:X^{\top }X\theta =X^{\top }\mathit{y}\right\}\hfill \\ \hfill & =\hfill & {\left(X^{\top }X\right)}^{†}{X}^{\top }\mathit{y}\hfill \\ \hfill & =X^{\top }{\left(X{X}^{\top }\right)}^{†}\mathit{y},\hfill & \hfill \end{array}$$

where ${\left(X^{\top }X\right)}^{†}$ denotes the pseudoinverse of the bounded linear operator $X^{\top }X$ (for infinite-dimensional $\mathbb{H}$, the existence of the pseudoinverse is guaranteed because $X^{\top }X$ is bounded and has a closed range) (13). When $\mathbb{H}$ has dimension $p$ with $p<n$ and $X$ has rank $p$, there is a unique solution to the normal equations. On the contrary, Assumption 5 in Definition 1 implies that we can find many solutions $\theta \in \mathbb{H}$ to the normal equations that achieve $X\theta =y$. The minimum norm solution is given by

$$\widehat{\theta }=X^{\top }{\left(X{X}^{\top }\right)}^{-1}\mathit{y}.$$ [1]

Our main result gives tight bounds on the excess risk of this minimum norm estimator in terms of certain notions of effective rank of the covariance that are defined in terms of its eigenvalues.

We use ${\mu }_1\left(\mathrm{\Sigma }\right)\ge {\mu }_2\left(\mathrm{\Sigma }\right)\ge \cdots \hspace{0.17em}$ to denote the eigenvalues of $\mathrm{\Sigma }$ in descending order, and we denote the operator norm of $\mathrm{\Sigma }$ by $\Vert \mathrm{\Sigma }\Vert$. We use $I$ to denote the identity operator on $\mathbb{H}$ and $I_n$ to denote the $n\times n$ identity matrix.

Definition 3 (Effective Ranks): For the covariance operator $\mathrm{\Sigma }$, define ${\lambda }_i={\mu }_i\left(\mathrm{\Sigma }\right)$ for $i=\mathrm{1,2},\dots$. If ${\sum }_{i=1}^{\infty }{\lambda }_i<\infty$ and ${\lambda }_{k+1}>0$ for $k\ge 0$, define

$$r_k\left(\mathrm{\Sigma }\right)=\frac{{\sum }_{i>k}{\lambda }_i}{{\lambda }_{k+1}},\hspace{1em}\hspace{1em}{R}_k\left(\mathrm{\Sigma }\right)=\frac{{\left(\sum _{i>k}{\lambda }_i\right)}^2}{\sum _{i>k}{\lambda }_i^2}.$$

Main Results

The following theorem establishes nearly matching upper and lower bounds for the risk of the minimum norm interpolating estimator.

Theorem 1. For any ${\sigma }_x$, there are $b,c,c_1>1$ for which the following holds. Consider a linear regression problem from Definition 1. Define

$$k^{*}=min\left\{k\ge 0:r_k\left(\mathrm{\Sigma }\right)\ge bn\right\},$$

where the minimum of the empty set is defined as $\infty$. Suppose that $\delta <1$ with $\log \left(1/\delta \right)<n/c$. If $k^{*}\ge n/c_1$, then $\mathbb{E}R\left(\widehat{\theta }\right)\ge {\sigma }^2/c$. Otherwise,

$$\begin{array}{ccc}\hfill R\left(\widehat{\theta }\right)& \le \hfill & c\left({\Vert {\theta }^{*}\Vert }^2∥\mathrm{\Sigma }∥max\left\{\sqrt{\frac{r_0\left(\mathrm{\Sigma }\right)}{n}},\frac{r_0\left(\mathrm{\Sigma }\right)}{n},\sqrt{\frac{\log \left(1/\delta \right)}{n}}\right\}\right)\hfill \\ \hfill & \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+c\log \left(1/\delta \right){\sigma }_y^2\left(\frac{k^{*}}{n}+\frac{n}{R_{k^{*}}\left(\mathrm{\Sigma }\right)}\right)\hfill \end{array}$$

with probability at least $1-\delta$, and

$$\mathbb{E}R\left(\widehat{\theta }\right)\ge \frac{{\sigma }^2}{c}\left(\frac{k^{*}}{n}+\frac{n}{R_{k^{*}}\left(\mathrm{\Sigma }\right)}\right).$$

Moreover, there are universal constants $a_1,a_2,n_0$ such that, for all $n\ge n_0$, for all $\mathrm{\Sigma }$, and for all $t\ge 0$, there is a ${\theta }^{*}$ with $\Vert {\theta }^{*}\Vert =t$ such that, for $x\sim \mathcal{N}\left(0,\mathrm{\Sigma }\right)$ and $y|x\sim \mathcal{N}\left(x^{\top }{\theta }^{*},{\Vert {\theta }^{*}\Vert }^2\Vert \mathrm{\Sigma }\Vert \right)$ with probability at least $1/4$,

$$R\left(\widehat{\theta }\right)\ge \frac{1}{a_1}{\Vert {\theta }^{*}\Vert }^2∥\mathrm{\Sigma }∥\mathbb{1}\left[\frac{r_0\left(\mathrm{\Sigma }\right)}{n\log \left(1+r_0\left(\mathrm{\Sigma }\right)\right)}\ge a_2\right].$$

Effective Ranks and Overparameterization.

In order to understand the implications of Theorem 1, we now study relationships between the two notions of effective rank, $r_k$ and $R_k$, and establish sufficient and necessary conditions for the sequence $\left\{{\lambda }_i\right\}$ of eigenvalues to lead to small excess risk.

The following lemma shows that the two notions of effective rank are closely related. SI Appendix, section H has its proof and other properties of $r_k$ and $R_k$.

Lemma 1. $r_k\left(\mathrm{\Sigma }\right)\ge 1$, $r_k^2\left(\mathrm{\Sigma }\right)=r_k\left({\mathrm{\Sigma }}^2\right)R_k\left(\mathrm{\Sigma }\right)$, and

$$r_k\left({\mathrm{\Sigma }}^2\right)\le r_k\left(\mathrm{\Sigma }\right)\le R_k\left(\mathrm{\Sigma }\right)\le r_k^2\left(\mathrm{\Sigma }\right).$$

Notice that $r_0\left(I_p\right)=R_0\left(I_p\right)=p$. More generally, if all of the nonzero eigenvalues of $\mathrm{\Sigma }$ are identical, then $r_0\left(\mathrm{\Sigma }\right)=R_0\left(\mathrm{\Sigma }\right)=rank\left(\mathrm{\Sigma }\right)$. For $\mathrm{\Sigma }$ with finite rank, we can express both $r_0\left(\mathrm{\Sigma }\right)$ and $R_0\left(\mathrm{\Sigma }\right)$ as a product of the rank and a notion of symmetry. In particular, for $rank\left(\mathrm{\Sigma }\right)=p$, we can write

$$\begin{array}{ccc}\hfill r_0\left(\mathrm{\Sigma }\right)& =\hfill & rank\left(\mathrm{\Sigma }\right)s\left(\mathrm{\Sigma }\right),\hspace{1em}\hspace{1em}{R}_0\left(\mathrm{\Sigma }\right)=rank\left(\mathrm{\Sigma }\right)S\left(\mathrm{\Sigma }\right),\hfill \\ \hfill \text{with}\hspace{0.17em}\hspace{0.17em}s\left(\mathrm{\Sigma }\right)& =\hfill & \frac{\frac{1}{p}{\sum }_{i=1}^p{\lambda }_i}{{\lambda }_1},\hspace{1em}\hspace{1em}\hspace{1em}S\left(\mathrm{\Sigma }\right)=\frac{{\left(\frac{1}{p}\sum _{i=1}^p{\lambda }_i\right)}^2}{\frac{1}{p}\sum _{i=1}^p{\lambda }_i^2}.\hfill \end{array}$$

Both notions of symmetry $s$ and $S$ lie between $1/p$ (when ${\lambda }_2\to 0$) and 1 (when the ${\lambda }_i$ are all equal).

Theorem 1 shows that, for the minimum norm estimator to have near-optimal prediction accuracy, $r_0\left(\mathrm{\Sigma }\right)$ should be small compared with the sample size $n$ (from the first term) and $r_{k^{*}}\left(\mathrm{\Sigma }\right)$ and $R_{k^{*}}\left(\mathrm{\Sigma }\right)$ should be large compared with $n$. Together, these conditions imply that overparameterization is essential for benign overfitting in this setting: the number of nonzero eigenvalues should be large compared with $n$, they should have a small sum compared with $n$, and there should be many eigenvalues no larger than ${\lambda }_{k^{*}}$. If the number of these small eigenvalues is not much larger than $n$, then they should be roughly equal, but they can be more asymmetric if there are many more of them.

The following theorem shows that the kind of overparameterization that is essential for benign overfitting requires $\mathrm{\Sigma }$ to have a heavy tail. (The proof—and some other examples illustrating the boundary of benign overfitting—are in SI Appendix, section I.) In particular, if we fix $\mathrm{\Sigma }$ in an infinite-dimensional Hilbert space and ask when the excess risk of the minimum norm estimator approaches zero as $n\to \infty$, it imposes tight restrictions on the eigenvalues of $\mathrm{\Sigma }$. However, there are many other possibilities for these asymptotics if $\mathrm{\Sigma }$ can change with $n$. Since rescaling $X$ affects the accuracy of the least norm interpolant in an obvious way, we may assume without loss of generality that $\Vert \mathrm{\Sigma }\Vert =1$. If we restrict our attention to this case, then informally, Theorem 1 implies that, when the covariance operator for data with $n$ examples is ${\mathrm{\Sigma }}_n$, the least norm interpolant converges if $\frac{r_0\left({\mathrm{\Sigma }}_n\right)}{n}\to 0$, $\frac{k_n^{*}}{n}\to 0$, and $\frac{n}{R_{k_n^{*}}\left({\mathrm{\Sigma }}_n\right)}\to 0$ and only if $\frac{r_0\left({\mathrm{\Sigma }}_n\right)}{n\log \left(1+r_0\left({\mathrm{\Sigma }}_n\right)\right)}\to 0$, $\frac{k_n^{*}}{n}\to 0$, and $\frac{n}{R_{k_n^{*}}\left({\mathrm{\Sigma }}_n\right)}\to 0$, where $k_n^{*}=min\left\{k\ge 0:r_k\left({\mathrm{\Sigma }}_n\right)\ge bn\right\}$ for the universal constant $b$ in Theorem 1. This motivates the following definition.

Definition 4: A sequence of covariance operators ${\mathrm{\Sigma }}_n$ with $\Vert {\mathrm{\Sigma }}_n\Vert =1$ is benign if

$$\underset{n\to \infty }{lim}\frac{r_0\left({\mathrm{\Sigma }}_n\right)}{n}=\underset{n\to \infty }{lim}\frac{k_n^{*}}{n}=\underset{n\to \infty }{lim}\frac{n}{R_{k_n^{*}}\left({\mathrm{\Sigma }}_n\right)}=0.$$

We give some examples of benign and nonbenign settings.

Theorem 2.

• 1)If ${\mu }_k\left(\mathrm{\Sigma }\right)=k^{-\alpha }{\ln }^{-\beta }\left(\left(k+1\right)e/2\right)$, then $\mathrm{\Sigma }$ is benign if and only if $\alpha =1$ and $\beta >1$.
• 2)If

$${\mu }_k\left({\mathrm{\Sigma }}_n\right)=\left\{\begin{array}{cc}{\gamma }_k+{ϵ}_n\hspace{1em}\hfill & \mathit{\text{if}}k\le p_n,\hfill \\ 0\hspace{1em}\hfill & \mathit{\text{otherwise}}\hfill \end{array}\right.$$

and ${\gamma }_k=\mathrm{\Theta }\left(\exp \left(-k/\tau \right)\right)$, then ${\mathrm{\Sigma }}_n$ with $\Vert {\mathrm{\Sigma }}_n\Vert =1$ is benign if and only if $p_n=\omega \left(n\right)$ and $n{e}^{-o\left(n\right)}={ϵ}_n{p}_n=o\left(n\right)$. Furthermore, for $p_n=\mathrm{\Omega }\left(n\right)$ and ${ϵ}_n{p}_n=n{e}^{-o\left(n\right)}$,

$$R\left(\widehat{\theta }\right)\hspace{0.17em}=\hspace{0.17em}O\left(\frac{{ϵ}_n{p}_n+1}{n}+\frac{\ln \left(n/\left({ϵ}_n{p}_n\right)\right)}{n}+max\left\{\frac{1}{n},\frac{n}{p_n}\right\}\right).$$

Compare the situations described by Theorem 2.1 and 2.2. Theorem 2.1 shows that, for infinite-dimensional data with a fixed covariance, benign overfitting occurs if and only if the eigenvalues of the covariance operator decay just slowly enough for their sum to remain finite. Theorem 2.2 shows that the situation is very different if the data have finite dimension and a small amount of isotropic noise is added to the covariates. In that case, even if the eigenvalues of the original covariance operator (before the addition of isotropic noise) decay very rapidly, benign overfitting occurs if and only if both the dimension is large compared with the sample size and the isotropic component of the covariance is sufficiently small—but not exponentially small—compared with the sample size.

These examples illustrate the tension between the slow decay of eigenvalues that is needed for $k/n+n/R_k$ to be small and the summability of eigenvalues that is needed for $r_0\left(\mathrm{\Sigma }\right)/n$ to be small. There are two ways to resolve this tension. First, in the infinite-dimensional setting, slow decay of the eigenvalues suffices—decay just fast enough to ensure summability—as shown by Theorem 2.1. (SI Appendix, section I gives another example—Theorem S14.2—where the eigenvalue decay is allowed to vary with $n$; in that case, ${\mathrm{\Sigma }}_n$ is benign iff the decay rate gets close—but not too close—to $1/k$ as $n$ increases.) Second, the other way to resolve the tension is to consider a finite-dimensional setting (which ensures that the eigenvalues are summable), and in this case, arbitrarily slow decay is possible. Theorem 2.2 gives an example of this: eigenvalues that are all at least as large as a small constant. SI Appendix, section I gives other examples with a similar flavor, including a truncated infinite series that decays sufficiently slowly that the sum does not converge (SI Appendix, section I, Theorem S14.3). Theorem 2.1 shows that a very specific decay rate is required in infinite dimensions, which suggests that this is an unusual phenomenon in that case. The more generic scenario where benign overfitting will occur is demonstrated by Theorem 2.2, with eigenvalues that are either at least a constant or slowly decaying in a very high—but finite-dimensional—space.

Proof

Throughout the proofs, we treat ${\sigma }_x$ (the sub-Gaussian norm of the covariates) as a constant. Therefore, we use the symbols $b,c,c_1,c_2,\dots$ to refer to constants that only depend on ${\sigma }_x$. Their values are suitably large (and always at least one) but do not depend on any parameters of the problems that we consider other than ${\sigma }_x$. For universal constants that do not depend on any parameters of the problem at all, we use the symbol $a$. Also, whenever we sum over eigenvectors of $\mathrm{\Sigma }$, the sum is restricted to eigenvectors with nonzero eigenvalues.

Outline.

The first step is a standard decomposition of the excess risk into two pieces, a term that corresponds to the distortion that is introduced by viewing ${\theta }^{*}$ through the lens of the finite sample and a term that corresponds to the distortion introduced by the noise $\mathit{\epsilon }=\mathit{y}-X\theta$. The impact of both sources of error in $\widehat{\theta }$ on the excess risk is modulated by the covariance $\mathrm{\Sigma }$, which gives different weight to different directions in parameter space.

Lemma 2. The excess risk of the minimum norm estimator satisfies $R\left(\widehat{\theta }\right)\le 2{{\theta }^{*}}^{\top }B{\theta }^{*}+c{\sigma }^2\log \left(1/\delta \right)tr\left(C\right)$ with probability at least $1-\delta$ over $ϵ$, and ${\mathbb{E}}_{\mathit{\epsilon }}R\left(\widehat{\theta }\right)\ge {{\theta }^{*}}^{\top }B{\theta }^{*}+{\sigma }^{2}tr\left(C\right)$, where

$$\begin{array}{ccc}\hfill B& =\hfill & \left(I-X^{\top }{\left(X{X}^{\top }\right)}^{-1}X\right)\mathrm{\Sigma }\left(I-X^{\top }{\left(X{X}^{\top }\right)}^{-1}X\right),\hfill \\ \hfill C& =\hfill & {\left(X{X}^{\top }\right)}^{-1}X\mathrm{\Sigma }{X}^{\top }{\left(X{X}^{\top }\right)}^{-1}.\hfill \end{array}$$

The proof of this lemma is in SI Appendix, section A. SI Appendix, sections J and K give bounds on the term ${{\theta }^{*}}^{\top }B{\theta }^{*}$. The heart of the proof is controlling $tr\left(C\right)$.

Before continuing with the proof, let us make a quick digression to note that Lemma 2 already begins to give an idea that many low-variance directions are necessary for the least norm interpolator to succeed. Let us consider the extreme case that $p=n$ and $\mathrm{\Sigma }=I$. In this case, $C={\left(X{X}^{\top }\right)}^{-1}$. For Gaussian data, for instance, standard random matrix theory implies that, with high probability, the eigenvalues of $X{X}^{\top }$ will all be within a constant factor of $n$, which implies that $tr\left(C\right)$ is bounded below by a constant, and then, Lemma 2 implies that the least norm interpolant’s excess risk is at least a constant.

To prove that $tr\left(C\right)$ can be controlled for suitable $\mathrm{\Sigma }$, the first step is to express it in terms of sums of outer products of unit-covariance, independent, sub-Gaussian random vectors. We show that, when there is a $k^{*}$ with $k^{*}/n$ small and $r_{k^{*}}\left(\mathrm{\Sigma }\right)/n$ large, all of the smallest eigenvalues of these matrices are suitably concentrated, and this implies that $tr\left(C\right)$ is bounded above by

$$\underset{l\le k^{*}}{min}\left(\frac{l}{n}+n\frac{\sum _{i>l}{\lambda }_i^2}{{\left({\lambda }_{k^{*}+1}{r}_{k^{*}}\left(\mathrm{\Sigma }\right)\right)}^2}\right).$$

(Later, we show that the minimizer is $l=k^{*}$.) Next, we show that this expression is also a lower bound on $tr\left(C\right)$ provided that there is such a $k^{*}$. Conversely, we show that, for any $k$ for which $r_k\left(\mathrm{\Sigma }\right)$ is not large compared with $n$, $tr\left(C\right)$ is at least as big as a constant times $min\left(1,k/n\right)$. Combining shows that, when $k^{*}/n$ is small, $tr\left(C\right)$ is upper and lower bounded by constant factors times

$$\frac{k^{*}}{n}+\frac{n}{R_{k^{*}}\left(\mathrm{\Sigma }\right)}.$$

Unit Variance Sub-Gaussians.

Our assumptions allow the trace of $C$ to be expressed as a function of many independent sub-Gaussian vectors.

Lemma 3. Consider a covariance operator $\mathrm{\Sigma }$ with ${\lambda }_i={\mu }_i\left(\mathrm{\Sigma }\right)$ and ${\lambda }_n>0$. Write its spectral decomposition $\mathrm{\Sigma }={\sum }_j{\lambda }_j{v}_j{v}_j^{\top }$, where the orthonormal $v_j\in \mathbb{H}$ are the eigenvectors corresponding to the ${\lambda }_j$. For $i$ with ${\lambda }_i>0$, define $z_i=X{v}_i/\sqrt{{\lambda }_i}$. Then,

$$tr\left(C\right)=\sum _i\left[{\lambda }_i^2{z}_i^{\top }{\left(\sum _j{\lambda }_j{z}_j{z}_j^{\top }\right)}^{-2}{z}_i\right],$$

and these $z_i\in {\mathbb{R}}^n$ are independent ${\sigma }_x^2$ sub-Gaussian. Furthermore, for any $i$ with ${\lambda }_i>0$, we have

$${\lambda }_i^2{z}_i^{\top }{\left(\sum _j{\lambda }_j{z}_j{z}_j^{\top }\right)}^{-2}{z}_i=\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2},$$

where $A_{-i}={\sum }_{j\ne i}{\lambda }_j{z}_j{z}_j^{\top }$.

Proof: By Assumption 2 in Definition 1, the random variables $x^{\top }{v}_i/\sqrt{{\lambda }_i}$ are independent ${\sigma }_x^2$ sub-Gaussian. We consider $X$ in the basis of eigenvectors of $\mathrm{\Sigma }$, $X{v}_i=\sqrt{{\lambda }_i}{z}_i$, to see that

$$X{X}^{\top }=\sum _i{\lambda }_i{z}_i{z}_i^{\top },\hspace{1em}\hspace{1em}X\mathrm{\Sigma }{X}^{\top }=\sum _i{\lambda }_i^2{z}_i{z}_i^{\top },$$

and therefore, we can write

$$\begin{array}{ll}\hfill tr\left(C\right)& \hspace{0.17em}=tr\left({\left(X{X}^{\top }\right)}^{-1}X\mathrm{\Sigma }{X}^{\top }{\left(X{X}^{\top }\right)}^{-1}\right)\hfill \\ \hfill & =\sum _i\left[{\lambda }_i^2{z}_i^{\top }{\left(\sum _j{\lambda }_j{z}_j{z}_j^{\top }\right)}^{-2}{z}_i\right].\hfill \end{array}$$

For the second part, we use SI Appendix, section B, Lemma S3, which is a consequence of the Sherman–Woodbury–Morrison formula

$$\begin{array}{ll}\hfill {\lambda }_i^2{z}_i^{\top }{\left(\sum _j{\lambda }_j{z}_j{z}_j^{\top }\right)}^{-2}{z}_i& ={\lambda }_i^2{z}_i^{\top }{\left({\lambda }_i{z}_i{z}_i^{\top }+A_{-i}\right)}^{-2}{z}_i\hfill \\ \hfill & =\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2},\hfill \end{array}$$

by SI Appendix, section B, Lemma S3 for the case $k=1$ and $Z=\sqrt{{\lambda }_i}{z}_i$. Note that $A_{-i}$ is invertible by Assumption 5 in Definition 1. $\square$

The weighted sum of outer products of these sub-Gaussian vectors plays a central role in the rest of the proof. Define

$$A=\sum _i{\lambda }_i{z}_i{z}_i^{\top },\hspace{1em}{A}_{-i}=\sum _{j\ne i}{\lambda }_j{z}_j{z}_j^{\top },\hspace{1em}{A}_k=\sum _{i>k}{\lambda }_i{z}_i{z}_i^{\top },$$

where the $z_i\in {\mathbb{R}}^n$ are independent vectors with independent ${\sigma }_x^2$ sub-Gaussian coordinates with unit variance defined in Lemma 3. Note that the vector $z_i$ is independent of the matrix $A_{-i}$, and therefore, in the last part of Lemma 3, all of the random quadratic forms are independent of the points where those forms are evaluated.

Concentration of $A$.

The next step is to show that eigenvalues of $A$, $A_{-i}$, and $A_k$ are concentrated. The proof of the following inequality is in SI Appendix, section C. Recall that ${\mu }_1\left(A\right)$ and ${\mu }_n\left(A\right)$ denote the largest and the smallest eigenvalues of the $n\times n$ matrix $A$.

Lemma 4. There is a constant $c$ such that, for any $k\ge 0$ with probability at least $1-2e^{-n/c}$,

$$\frac{1}{c}\sum _{i>k}{\lambda }_i-c{\lambda }_{k+1}n\le {\mu }_n\left(A_k\right)\le {\mu }_1\left(A_k\right)\le c\left(\sum _{i>k}{\lambda }_i+{\lambda }_{k+1}n\right).$$

The following lemma uses this result to give bounds on the eigenvalues of $A_k$, which in turn, give bounds on some eigenvalues of $A_{-i}$ and $A$. For these upper and lower bounds to match up to a constant factor, the sum of the eigenvalues of $A_k$ should dominate the term involving its leading eigenvalue, which is a condition on the effective rank $r_k\left(\mathrm{\Sigma }\right)$. The lemma shows that, after $r_k\left(\mathrm{\Sigma }\right)$ is sufficiently large, all of the eigenvalues of $A_k$ are identical up to a constant factor.

Lemma 5. There are constants $b,c\ge 1$ such that, for any $k\ge 0$, with probability at least $1-2e^{-n/c}$,

• 1)for all $i\ge 1$,

$${\mu }_{k+1}\left(A_{-i}\right)\le {\mu }_{k+1}\left(A\right)\le {\mu }_1\left(A_k\right)\le c\left(\sum _{j>k}{\lambda }_j+{\lambda }_{k+1}n\right);$$

• 2)for all $1\le i\le k$,

$${\mu }_n\left(A\right)\ge {\mu }_n\left(A_{-i}\right)\ge {\mu }_n\left(A_k\right)\ge \frac{1}{c}\sum _{j>k}{\lambda }_j-c{\lambda }_{k+1}n;$$

• and

• 3)if $r_k\left(\mathrm{\Sigma }\right)\ge bn$, then

$$\frac{1}{c}{\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\le {\mu }_n\left(A_k\right)\le {\mu }_1\left(A_k\right)\le c{\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right).$$

Proof: By Lemma 4, we know that, with probability at least $1-2e^{-n/c_1}$,

$$\begin{array}{ll}\hfill \frac{1}{c_1}\sum _{j>k}{\lambda }_j-c_1{\lambda }_{k+1}n& \le {\mu }_n\left(A_k\right)\hfill \\ \hfill & \le {\mu }_1\left(A_k\right)\le c_1\left(\sum _{j>k}{\lambda }_j+{\lambda }_{k+1}n\right).\hfill \end{array}$$

First, the matrix $A-A_k$ has rank at most $k$ (as a sum of $k$ matrices of rank 1). Thus, there is a linear space $\mathcal{L}$ of dimension $n-k$ such that, for all $v\in \mathcal{L}$, $v^{\top }Av=v^{\top }{A}_{k}v\le {\mu }_1\left(A_k\right){\Vert v\Vert }^2$ and therefore, ${\mu }_{k+1}\left(A\right)\le {\mu }_1\left(A_k\right)$.

Second, by the Courant–Fischer–Weyl Theorem, for all $i$ and $j$, ${\mu }_j\left(A_{-i}\right)\le {\mu }_j\left(A\right)$ (SI Appendix, section G, Lemma S11). On the other hand, for $i\le k$, $A_k⪯A_{-i}$, and therefore, all of the eigenvalues of $A_{-i}$ are lower bounded by ${\mu }_n\left(A_k\right)$.

Finally, if $r_k\left(\mathrm{\Sigma }\right)\ge bn$,

$$\begin{array}{ll}\hfill \sum _{j>k}{\lambda }_j+{\lambda }_{k+1}n& ={\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)+{\lambda }_{k+1}n\hfill \\ \hfill & \le \left(1+\frac{1}{b}\right){\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right),\hfill \\ \hfill \frac{1}{c_1}\sum _{j>k}{\lambda }_j-c_1{\lambda }_{k+1}n& =\frac{1}{c_1}{\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)-c_1{\lambda }_{k+1}n\hfill \\ \hfill & \ge \left(\frac{1}{c_1}-\frac{c_1}{b}\right){\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right).\hfill \end{array}$$

Choosing $b>c_1^2$ and $c>max\left\{c_1+1/c_1,{\left(1/c_1-c_1/b\right)}^{-1}\right\}$ gives the third claim of the lemma. $\square$

Upper Bound on the Trace Term.

Lemma 6. There are constants $b,c\ge 1$ such that, if $0\le k\le n/c$, $r_k\left(\mathrm{\Sigma }\right)\ge bn$, and $l\le k$, then with probability at least $1-7e^{-n/c}$,

$$tr\left(C\right)\le c\left(\frac{l}{n}+n\frac{\sum _{i>l}{\lambda }_i^2}{{\left(\sum _{i>k}{\lambda }_i\right)}^2}\right).$$

The proof uses the following lemma and its corollary. Their proofs are in SI Appendix, section C.

Lemma 7. Suppose that ${\left\{{\lambda }_i\right\}}_i^{\infty }$ is a nonincreasing sequence of nonnegative numbers such that ${\sum }_{i=1}^{\infty }{\lambda }_i<\infty$ and that ${\left\{{\xi }_i\right\}}_{i=1}^{\infty }$ are independent centered $\sigma$-subexponential random variables. Then, for some universal constant $a$ for any $t>0$, with probability at least $1-2e^{-t}$,

$$\left|\sum _i{\lambda }_i{\xi }_i\right|\le a\sigma max\left(t{\lambda }_1,\sqrt{t\sum _i{\lambda }_i^2}\right).$$

Corollary 1. Suppose that $z\in {\mathbb{R}}^n$ is a centered random vector with independent ${\sigma }^2$ sub-Gaussian coordinates with unit variances, $\mathcal{L}$ is a random subspace of ${\mathbb{R}}^n$ of codimension $k$, and $\mathcal{L}$ is independent of $z$. Then, for some universal constant $a$ and any $t>0$, with probability at least $1-3e^{-t}$,

$$\begin{array}{ll}\hfill \Vert {z\Vert }^2& \le n+a{\sigma }^2\left(t+\sqrt{nt}\right),\hfill \\ \hfill \Vert {\mathrm{\Pi }}_{\mathcal{L}}{z\Vert }^2& \ge n-a{\sigma }^2\left(k+t+\sqrt{nt}\right),\hfill \end{array}$$

where ${\mathrm{\Pi }}_{\mathcal{L}}$ is the orthogonal projection on $\mathcal{L}$.

Proof of Lemma 6: Fix $b$ to its value in Lemma 5. By Lemma 3,

$$\begin{array}{ll}\hfill tr\left(C\right)& =\sum _i{\lambda }_i^2{z}_i^{\top }{A}^{-2}{z}_i\hfill \\ \hfill & =\sum _{i=1}^l\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2}+\sum _{i>l}{\lambda }_i^2{z}_i^{\top }{A}^{-2}{z}_i.\hfill \end{array}$$ [2]

First, consider the sum up to $l$. If $r_k\left(\mathrm{\Sigma }\right)\ge bn$, Lemma 5 shows that, with probability at least $1-2e^{-n/c_1}$ for all $i\le k$, ${\mu }_n\left(A_{-i}\right)\ge {\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)/c_1$ and for all $i$, ${\mu }_{k+1}\left(A_{-i}\right)\le c_1{\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)$. The lower bounds on the ${\mu }_n\left(A_{-i}\right)$ imply that, for all $z\in {\mathbb{R}}^n$ and $1\le i\le l$,

$$z^{\top }{A}_{-i}^{-2}z\le \frac{c_1^2\Vert {z\Vert }^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2},$$

and the upper bounds on the ${\mu }_{k+1}\left(A_{-i}\right)$ give

$$z^{\top }{A}_{-i}^{-1}z\ge {\left({\mathrm{\Pi }}_{{\mathcal{L}}_i}z\right)}^{\top }{A}_{-i}^{-1}{\mathrm{\Pi }}_{{\mathcal{L}}_i}z\ge \frac{\Vert {\mathrm{\Pi }}_{{\mathcal{L}}_i}{z\Vert }^2}{c_1{\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)},$$

where ${\mathcal{L}}_i$ is the span of the $n-k$ eigenvectors of $A_{-i}$ corresponding to its smallest $n-k$ eigenvalues. Therefore, for $i\hspace{0.17em}\le \hspace{0.17em}l$,

$$\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2}\le \frac{z_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(z_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2}\le c_1^4\frac{{\Vert z_i\Vert }^2}{{\Vert {\mathrm{\Pi }}_{{\mathcal{L}}_i}{z}_i\Vert }^4}.$$ [3]

Next, we apply Corollary 1 $l$ times together with a union bound to show that, with probability at least $1-3e^{-t}$ for all $1\le i\le l$,

$${\Vert z_i\Vert }^2\le n+a{\sigma }_x^2\left(t+\ln k+\sqrt{n\left(t+\ln k\right)}\right)\le c_{2}n,$$ [4]

$${\Vert {\mathrm{\Pi }}_{{\mathcal{L}}_i}{z}_i\Vert }^2\ge n-a{\sigma }_x^2\left(k+t+\ln k+\sqrt{n\left(t+\ln k\right)}\right)\ge n/c_3,$$ [5]

provided that $t<n/c_0$ and $c>c_0$ for some sufficiently large $c_0$ (note that $c_2$ and $c_3$ only depend on $c_0$, $a$, and ${\sigma }_x$, and we can still take $c$ large enough in the end without changing $c_2$ and $c_3$). Combining Eqs. 3–5, with probability at least $1-5e^{-n/c_0}$,

$$\sum _{i=1}^l\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2}\le c_4\frac{l}{n}.$$

Second, consider the second sum in Eq. 2. Lemma 5 shows that, on the same high-probability event that we considered in bounding the first half of the sum, ${\mu }_n\left(A\right)\ge {\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)/c_1$. Hence,

$$\sum _{i>l}{\lambda }_i^2{z}_i^{\top }{A}^{-2}{z}_i\le \frac{c_1^2\sum _{i>l}{\lambda }_i^2{\Vert z_i\Vert }^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}.$$

Notice that ${\sum }_{i>l}{\lambda }_i^2{\Vert z_i\Vert }^2$ is a weighted sum of ${\sigma }_x^2$-subexponential random variables, with the weights given by the ${\lambda }_i^2$ in blocks of size $n$. Lemma 7 implies that, with probability at least $1\hspace{0.17em}-\hspace{0.17em}2e^{-t}$,

$$\begin{array}{ll}\hfill \sum _{i>l}{\lambda }_i^2{\Vert z_i\Vert }^2& \le n\sum _{i>l}{\lambda }_i^2+a{\sigma }_x^{2}max\left(\underset{l+1}{\overset{2}{\lambda }}t,\sqrt{tn\sum _{i>l}{\lambda }_i^4}\right)\hfill \\ \hfill & \le n\sum _{i>l}{\lambda }_i^2+a{\sigma }_x^{2}max\left(t\sum _{i>l}\underset{i}{\overset{2}{\lambda }},\sqrt{tn}\sum _{i>l}\underset{i}{\overset{2}{\lambda }}\right)\hfill \\ \hfill & \le c_{5}n\sum _{i>l}{\lambda }_i^2\hfill \end{array}$$

because $t<n/c_0$. Combining the above gives

$$\sum _{i>l}{\lambda }_i^2{z}_i^{\top }{A}^{-2}{z}_i\le c_{6}n\frac{{\sum }_{i>l}{\lambda }_i^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}.$$

Finally, putting both parts together and taking $c>max\left\{c_0,c_4,c_6\right\}$ gives the lemma. $\square$

Lower Bound on the Trace Term.

We first give a bound on a single term in the expression for $tr\left(C\right)$ in Lemma 3 that holds regardless of $r_k\left(\mathrm{\Sigma }\right)$. The proof is in SI Appendix, section D.

Lemma 8. There is a constant $c$ such that, for any $i\ge 1$ with ${\lambda }_i>0$ and any $0\le k\le n/c$, with probability at least $1-5e^{-n/c}$,

$$\frac{{\lambda }_i^2{z}_i^{\top }{A}_{-i}^{-2}{z}_i}{{\left(1+{\lambda }_i{z}_i^{\top }{A}_{-i}^{-1}{z}_i\right)}^2}\ge \frac{1}{cn}{\left(1+\frac{{\sum }_{j>k}{\lambda }_j+n{\lambda }_{k+1}}{n{\lambda }_i}\right)}^{-2}.$$

We can extend these bounds to a lower bound on $tr\left(C\right)$ using the following lemma. The proof is in SI Appendix, section E.

Lemma 9. Suppose that $n\le \infty$, ${\left\{{\eta }_i\right\}}_{i=1}^n$ is a sequence of nonnegative random variables, and that ${\left\{t_i\right\}}_{i=1}^n$ is a sequence of nonnegative real numbers (at least one of which is strictly positive) such that, for some $\delta \in \left(\mathrm{0,1}\right)$ and any $i\le n$, $Pr\left({\eta }_i>t_i\right)\ge 1-\delta$. Then,

$$Pr\left(\sum _{i=1}^n{\eta }_i\ge \frac{1}{2}\sum _{i=1}^n{t}_i\right)\ge 1-2\delta .$$

These two lemmas imply the following lower bound.

Lemma 10. There are constants $c$ such that, for any $0\le k\le n/c$ and any $b>1$ with probability at least $1-10e^{-n/c}$,

• 1)if $r_k\left(\mathrm{\Sigma }\right)<bn$, then $tr\left(C\right)\ge \frac{k+1}{c{b}^{2}n}$; and
• 2)if $r_k\left(\mathrm{\Sigma }\right)\ge bn$, then

$$tr\left(C\right)\ge \frac{1}{c{b}^2}\underset{l\le k}{min}\left(\frac{l}{n}+\frac{b^{2}n\sum _{i>l}{\lambda }_i^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}\right).$$

In particular, if all choices of $k\le n/c$ give $r_k\left(\mathrm{\Sigma }\right)<bn$, then $r_{n/c}\left(\mathrm{\Sigma }\right)<bn$ implies that, with probability at least $1-10e^{-n/c}$, $tr\left(C\right)={\mathrm{\Omega }}_{{\sigma }_x}\left(1\right)$.

Proof: From Lemmas 3, 8, and 9, with probability at least $1-10e^{-n/c_1}$,

$$\begin{array}{ll}\hfill tr\left(C\right)& \ge \frac{1}{c_{1}n}\sum _i{\left(1+\frac{{\sum }_{j>k}{\lambda }_j+n{\lambda }_{k+1}}{n{\lambda }_i}\right)}^{-2}\hfill \\ \hfill & \ge \frac{1}{c_{2}n}\sum _{i}min\left\{1,\frac{n^2{\lambda }_i^2}{{\left({\sum }_{j>k}{\lambda }_j\right)}^2},\frac{\underset{i}{\overset{2}{\lambda }}}{\underset{k+1}{\overset{2}{\lambda }}}\right\}\hfill \\ \hfill & \ge \frac{1}{c_2{b}^{2}n}\sum _{i}min\left\{1,{\left(\frac{bn}{r_k\left(\mathrm{\Sigma }\right)}\right)}^2\frac{{\lambda }_i^2}{{\lambda }_{k+1}^2},\frac{{\lambda }_i^2}{{\lambda }_{k+1}^2}\right\}.\hfill \end{array}$$

Now, if $r_k\left(\mathrm{\Sigma }\right)<bn$, then the second term in the minimum is always bigger than the third term, and in that case,

$$tr\left(C\right)\ge \frac{1}{c_2{b}^{2}n}\sum _{i}min\left\{1,\frac{\underset{i}{\overset{2}{\lambda }}}{\underset{k+1}{\overset{2}{\lambda }}}\right\}\ge \frac{k+1}{c_2{b}^{2}n}.$$

On the other hand, if $r_k\left(\lambda \right)\ge bn$,

$$\begin{array}{ll}\hfill tr\left(C\right)& \ge \frac{1}{c_2{b}^2}\sum _{i}min\left\{\frac{1}{n},\frac{b^{2}n{\lambda }_i^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}\right\}\hfill \\ \hfill & =\frac{1}{c_2{b}^2}\underset{l\le k}{min}\left(\frac{l}{n}+\frac{b^{2}n\sum _{i>l}{\lambda }_i^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}\right),\hfill \end{array}$$

where the equality follows from the fact that the ${\lambda }_i$ are nonincreasing. $\square$

A Simple Choice of $l$.

Recall that ${\sigma }_x$ is a constant. If no $k\le n/c$ has $r_k\left(\mathrm{\Sigma }\right)\ge bn$, then Lemmas 2 and 10 imply that the expected excess risk is $\mathrm{\Omega }\left({\sigma }^2\right)$, which proves the first paragraph of Theorem 1 for large $k^{*}$. If some $k\le n/c$ does have $r_k\left(\mathrm{\Sigma }\right)\ge bn$, then the upper and lower bounds of Lemmas 6 and 10 are constant multiples of

$$\underset{l\le k}{min}\left(\frac{l}{n}+n\frac{\sum _{i>l}{\lambda }_i^2}{{\left({\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)\right)}^2}\right).$$

It might seem surprising that any suitable choice of $k$ suffices to give upper and lower bounds: what prevents one choice of $k$ from giving an upper bound that falls below the lower bound that arises from another choice of $k$? However, the freedom to choose $k$ is somewhat illusory: Lemma 5 shows that, for any qualifying value of $k$, the smallest eigenvalue of $A$ is within a constant factor of ${\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)$. Thus, any two choices of $k$ satisfying $k\le n/c$ and $r_k\left(\mathrm{\Sigma }\right)\ge bn$ must have values of ${\lambda }_{k+1}{r}_k\left(\mathrm{\Sigma }\right)$ within constant factors. The smallest such $k$ simplifies the bound on $tr\left(C\right)$ as the following lemma shows. The proof is in SI Appendix, section F.

Lemma 11. For any $b\ge 1$ and $k^{*}≔min\left\{k:r_k\left(\mathrm{\Sigma }\right)\ge bn\right\}$, if $k^{*}<\infty$, we have

$$\begin{array}{l}\hfill \underset{l\le k^{*}}{min}\left(\frac{l}{bn}+\frac{bn\sum _{i>l}{\lambda }_i^2}{{\left({\lambda }_{k^{*}+1}{r}_{k^{*}}\left(\mathrm{\Sigma }\right)\right)}^2}\right)\\ \hfill & =\frac{k^{*}}{bn}+\frac{bn\sum _{i>k^{*}}{\lambda }_i^2}{{\left({\lambda }_{k^{*}+1}{r}_{k^{*}}\left(\mathrm{\Sigma }\right)\right)}^2}=\frac{k^{*}}{bn}+\frac{bn}{R_{k^{*}}\left(\mathrm{\Sigma }\right)}.\hfill \end{array}$$

Finally, we can finish the proof of Theorem 1. Set $b$ in Lemma 10 and Theorem 1 to the constant $b$ from Lemma 6. Take $c_1$ to be the maximum of the constants $c$ from Lemmas 6 and 10.

By Lemma 10, if $k^{*}\ge n/c_1$, then with high probability $tr\left(C\right)\ge 1/c_2$. However, by Lemma 10.2 and by Lemma 6, if $k^{*}<n/c_1$, then with high probability $tr\left(C\right)$ is within a constant factor of $\underset{l\le k^{*}}{min}\left(\frac{l}{n}+n\frac{{\sum }_{i>l}{\lambda }_i^2}{{\left({\lambda }_{k^{*}+1}{r}_{k^{*}}\left(\mathrm{\Sigma }\right)\right)}^2}\right)$, which by Lemma 11, is within a constant factor of $\frac{k^{*}}{n}+\frac{n}{R_{k^{*}}\left(\mathrm{\Sigma }\right)}$. Taking $c$ sufficiently large and combining these results with Lemma 2 and with the upper bound on the term ${{\theta }^{*}}^{\top }B{\theta }^{*}$ in SI Appendix, section J completes the proof of the first paragraph of Theorem 1.

The proof of the second paragraph is in SI Appendix, section K.

Deep Neural Networks

How relevant are Theorems 1 and 2 to the phenomenon of benign overfitting in deep neural networks? One connection appears by considering regimes where deep neural networks are well approximated by linear functions of their parameters. This so-called neural tangent kernel (NTK) viewpoint has been vigorously pursued recently in an attempt to understand the optimization properties of deep learning methods. Very wide neural networks, trained with gradient descent from a suitable random initialization, can be accurately approximated by linear functions in an appropriate Hilbert space, and in this case, gradient descent finds an interpolating solution quickly (14–19). (Note that these papers do not consider prediction accuracy, except when there is no noise; for example, ref. 14, Assumption A1 implies that the network can compute a suitable real-valued response exactly, and the data-dependent bound of ref. 19, Theorem 5.1 becomes vacuous when independent noise is added to the $y_i$.) The eigenvalues of the covariance operator in this case can have a heavy tail under reasonable assumptions on the data distribution (20, 21), and the dimension is very large but finite as required for benign overfitting. However, the assumptions of Theorem 1 do not apply in this case. In particular, the assumption that the random elements of the Hilbert space are a linearly transformed vector with independent components is not satisfied. Thus, our results are not directly applicable in this—somewhat unrealistic—setting. Note that the slow decay of the eigenvalues of the NTK is in contrast to the case of the Gaussian and other smooth kernels, where the eigenvalues decay nearly exponentially quickly (22).

The phenomenon of benign overfitting was first observed in deep neural networks. Theorems 1 and 2 are steps toward understanding this phenomenon by characterizing when it occurs in the simple setting of linear regression. Those results suggest that covariance eigenvalues that are constant or slowly decaying in a high (but finite)-dimensional space might be important in the deep network setting also. Some authors have suggested viewing neural networks as finite-dimensional approximations to infinite-dimensional objects (23–25), and there are generalization bounds—although not for the overfitting regime—that are applicable to infinite-width deep networks with parameter norm constraints (26–30). However, the intuition from the linear setting suggests that truncating to a finite-dimensional space might be important for good statistical performance in the overfitting regime. Confirming this conjecture by extending our results to the setting of prediction in deep neural networks is an important open problem.

Conclusions and Further Work

Our results characterize when the phenomenon of benign overfitting occurs in high-dimensional linear regression with Gaussian data and more generally. We give finite sample excess risk bounds that reveal the covariance structure that ensures that the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization depends on two notions of the effective rank of the data covariance operator. It shows that overparameterization (that is, the existence of many low-variance and hence, unimportant directions in parameter space) is essential for benign overfitting and that data that lie in a large but finite-dimensional space exhibit the benign overfitting phenomenon with a much wider range of covariance properties than data that lie in an infinite-dimensional space.

There are several natural future directions. Our main theorem requires the conditional expectation $\mathbb{E}\left[y|x\right]$ to be a linear function of $x$, and it is important to understand whether the results are also true in the misspecified setting, where this assumption is not true. Our main result also assumes that the covariates are distributed as a linear function of a vector of independent random variables. We would like to understand the extent to which this assumption can be relaxed since it rules out some important examples, such as infinite-dimensional reproducing kernel Hilbert spaces with continuous kernels defined on finite-dimensional spaces. We would also like to understand how our results extend to other loss functions other than squared error and what we can say about overfitting estimators beyond the minimum norm interpolating estimator. The most interesting future direction is understanding how these ideas could apply to nonlinearly parameterized function classes, such as neural networks, the methodology that uncovered the phenomenon of benign overfitting.

Data Availability.

There are no data associated with this manuscript.