A brief prehistory of double descent

In their thought-provoking paper, Belkin et al. (1) illustrate and discuss the shape of risk curves in the context of modern high-complexity learners. Given a fixed training sample size $n$, such curves show the risk of a learner as a function of some (approximate) measure of its complexity $N$. With $N$ the number of features, these curves are also referred to as feature curves. A salient observation in ref. 1 is that these curves can display what they call double descent: With increasing $N$, the risk initially decreases, attains a minimum, and then increases until $N$ equals $n$, where the training data are fitted perfectly. Increasing $N$ even further, the risk decreases a second and final time, creating a peak at $N=n$. This twofold descent may come as a surprise, but as opposed to what ref. 1 reports, it has not been overlooked historically. Our letter draws attention to some original earlier findings of interest to contemporary machine learning.

Already in 1989, using artificial data, Vallet et al. (2) experimentally demonstrated double descent for learning curves of classifiers trained through minimum norm linear regression (MNLR; see ref. 3)—termed the pseudo-inverse solution in ref. 2. In learning curves the risk is displayed as a function of $n$, as opposed to $N$ for risk curves. What intuitively matters in learning behavior, however, is the sample size relative to the measure of complexity. This idea is made explicit in various physics papers on learning (e.g., refs. 2, 4, and 5), where the risk is often plotted against $\alpha =\frac{n}{N}$. A first theoretical result on double descent, indeed using such $\alpha$, is given by Opper et al. (4). They prove that in particular settings, for $N$ going to infinity, the pseudo-inverse solution improves as soon as one moves away from the peak at $\alpha =1$.

Employing a so-called pseudo-Fisher linear discriminant (PFLD, equivalent to MNLR), Duin (6) was the first to show feature curves on real-world data quite similar to the double-descent curves in ref. 1. Compare, for instance, figure 2 in ref. 1 with figures 6 and 7 in ref. 6. Skurichina and Duin (7) demonstrate experimentally that increasing PFLD’s complexity simply by adding random features can improve performance when $N=n$ (i.e., $\alpha =1$). The benefit of some form of regularization has been shown already in ref. 2. For semisupervised PFLD, Krijthe and Loog (8) demonstrate that unlabeled data can regularize but also worsen the peak in the curve. Their work builds on the original analysis of double descent for the supervised PFLD by Raudys and Duin (9).

Interestingly, results from refs. 4–7 suggest that some losses may not exhibit double descent in the first place. In refs. 6 and 7, the linear support vector machine (SVM) shows regular monotonic behavior. Analytic results from refs. 4 and 5 show the same for the so-called perceptron of optimal (or maximal) stability, which is closely related to the SVM (5).

The findings in ref. 1 go, significantly, beyond those for the MNLR. Also shown, for instance, is double descent for two-layer neural networks and random forests. Combining this with observations such as those from Loog et al. (10), which show striking multiple-descent learning curves (even in the underparameterized regime), the need to further our understanding of such rudimentary learning behavior is evident.