Cross-validation: what does it estimate and how well does it do it?

Abstract

Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow’s ${\text{C}}_{\text{p}}$. Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not re-fit the model on the combined data, since this invalidates the confidence intervals.

1. Introduction

When deploying a predictive model, it is important to understand its prediction accuracy on future test points, so both good point estimates and accurate confidence intervals for prediction error are essential. Cross-validation (CV) is a widely-used approach for these two tasks, but in spite of its seeming simplicity, its operating properties remain opaque. Considering first estimation, it is challenging to precisely state the estimand corresponding to the cross-validation point estimate. In this work, we show that the the estimand of CV is not the accuracy of the model fit on the data at hand, but is instead the average accuracy over many hypothetical data sets. Specifically, we show that the CV estimate of error has larger mean squared error (MSE) when estimating the prediction error of the final model than when estimating the average prediction error of models across many unseen data sets for the special case of linear regression. Turning to confidence intervals for prediction error, we show that naïve intervals based on CV can fail badly, giving coverage far below the nominal level; we provide a simple example soon in Section 1.1. The source of this behavior is the estimation of the variance used to compute the width of the interval: it does not account for the correlation between the error estimates in different folds, which arises because each data point is used for both training and testing. As a result, the estimate of variance is too small and the intervals are too narrow. To address this issue, we develop nested cross-validation (NCV) that achieves coverage near the nominal level.

1.1. A simple illustration

As a motivating example where naïve cross-validation confidence intervals fail, we consider a sparse logistic regression model

$$P\left(Y_i=1|X_i=x_i\right)=\frac{1}{1+\exp \left\{-x_i^{\top }\theta \right\}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,n,$$ (1)

with $n=90$ observations of $p=1000$ features, and a coefficient vector $\theta =c\cdot {\left(1,1,1,1,0,0,\dots \right)}^{\top }\in {ℝ}^p$ with four nonzero entries of equal strength. The feature matrix $X\in {ℝ}^{n\times p}$ is comprised of independent and identically distributed (i.i.d.) standard normal variables, and we chose the signal strength $c$ so that the Bayes misclassification rate is 20%. We estimate the parameters using ${\ell }_1$-penalized logistic regression with a fixed penalty level. In this case, naïve confidence intervals for prediction error are far too small: intervals with desired miscoverage of 10% give 31% miscoverage in our simulation. We visualize this in Figure 1. The intervals need to be made larger by a factor of about 1.6 to obtain coverage at the desired level in this case.

Fig. 1.

A plot of the true error of a model versus the CV estimates for 1000 replicates of the model from Section 1.1. The blue curve shows the average midpoint of the naïve CV confidence intervals. The green bands show the average 90% confidence interval for prediction error given by naïve CV. The red curves show the 5% and 95% quantiles from a quantile regression fit. To achieve nominal coverage, the green curves should approximate the red curves, but they are too narrow in this case.

1.2. Related work

Cross-validation is used ubiquitously to estimate the prediction error of a model (Allen, 1974; Geisser, 1975; Stone, 1977). The enduring popularity of CV is due to the fact that it is a conceptually simple improvement over a one-time train-test split (Blum et al., 1999). CV is part of a broader landscape of resampling techniques to estimate prediction error, with bootstrap-based techniques as the most common alternative (Efron, 1983, 1986; Efron and Tibshirani, 1997, 1993). The other main category of prediction error estimates are based on analytic adjustments such as Mallow’s $C_p$ (Mallows, 1973), AIC (Akaike, 1974), BIC (Schwarz, 1978), and general covariance penalties (Stein, 1981; Efron, 2004). The present work is primarily concerned with CV, but also addresses the properties of bootstrap, data splitting and covariance penalty methods.

In spite of CV’s apparent simplicity, the formal properties of this procedure are subtle; the seemingly basic question “what is cross-validation estimating?” has engendered considerable debate. Although the predictive accuracy of the model fit on the observed training data may seem like a natural estimand, it has been observed that the CV estimator tracks this quantity only weakly, suggesting that CV should instead be treated as an estimator of the average prediction error across training sets (Zhang, 1995; Hastie et al., 2009; Yousef, 2020). See also Rosset and Tibshirani (2020) and Wager (2020) for a discussion about different potential estimands. In this work, we discuss this phenomenon in detail for the case of the linear model. Our main result uses a conditional independence argument to explain the aforementioned weak relationship between CV and the instance-specific error.

Turning to the question of inference, one important use of CV is to deliver confidence intervals for the prediction error (or, similarly, an estimate of the standard error) to accompany a point estimate. The second primary goal in this work is to provide such confidence intervals, which cannot be reliably created with naïve methods, as shown in our example in Figure 1. A fundamental prior result shows that there is no unbiased estimator of the standard error of the CV point estimate based on one instance of CV (Bengio and Grandvalet, 2004). As a result, to obtain standard error estimates, one would either need to modify the CV procedure or make additional assumptions. Pursuing the former, Dietterich (1998) and Nadeau and Bengio (2003) proposes sampling schemes where the data is split in half, and CV is carried out within each half separately. This yields an estimate of standard error, but it will typically be much too conservative since the internal CV model fits each use a samples size that is less than half of the full sample. A related proposal due to Austern and Zhou (2020) involves repeatedly performing leave-one-out CV with data sets of half of the original size, but this proposed estimator is not computationally feasible for most learning algorithms.

In a different direction, Nadeau and Bengio (2003) and Markatou et al. (2005) propose alternative estimates of standard error, but these are based only on the sample size and higher moments of the errors and so do not address the source of the problem: a covariance term that we describe in Section 4.1. For bootstrap estimators, there are proposals to estimate the standard error of the (bootstrap) point estimates of prediction error with methods based on influence functions (Efron, 1983; Efron and Tibshirani, 1997). The CV proposal of Austern and Zhou (2020) similarly involves leave-one-out resampling, which can be interpreted as an empirical estimate of the influence functions.

Accompanying these algorithmic proposals, there is some theoretical understanding of the asymptotic behavior of CV. Dudoit and van der Laan (2005) proves a central limit theorem (CLT) for a cross-validation estimator, showing asymptotic coverage with a non-CV plug-in estimator for standard error. LeDell et al. (2015) provides a consistent estimator for the standard error in the special case of estimating the AUC, and Benkeser et al. (2020) conducts a higher-order asymptotic analyses for AUC and other metrics, yielding a more efficient estimator for accuracy with a consistent standard error estimate. Further theoretical results establish the asymptotic normality of the CV estimate in more general cases (Austern and Zhou, 2020; Bayle et al., 2020). The former considers the average prediction error across training sets (similar to our goal herein), and introduces an asymptotically valid estimate of the standard error; see Supplementary Appendix F.9 for an experiment with this estimator. The latter estimates a different estimand: the average prediction error of the models fit on the subsamples, and introduces a valid estimate of standard error for this quantity. We explain this estimand and the proposed standard error estimator in more detail in Supplementary Appendix I. Both use arguments relying on notions of algorithmic stability (Kumar et al., 2013; Kale et al., 2011; Celisse and Guedj, 2016). At present, it is not clear how the large-sample regime considered in these works relates to the behavior we see in small samples such as in the experiment in Section 1.1.In particular, algorithmic stability may not be satisfied in high-dimensional settings or with small sample sizes; see Supplementary Appendix I and Bayle et al. (2020) for more discussion.

Lastly, we note that CV is often used to compare predictive models, such as when selecting a model or a good value of a learning algorithm’s hyperparameters (e.g., Stoica et al., 1986; Shao, 1993; Zhang, 1993; Dietterich, 1998; Xu and Liang, 2001). To this end, Yang (2007); Wager (2020) show that for CV, comparing two models is a statistically easier task than estimating the prediction error, in some sense. While we expect that our proposed estimator would be of use for hyperparameter selection because it yields more accurate confidence intervals for prediction error, we do not pursue this problem further in the present work.

1.3. Our contribution

This work has two main thrusts. First, we study the choice of estimand for CV, giving results for the special case of the linear model. We prove a finite-sample conditional independence result (Theorem 1) with a supporting asymptotic result (Theorem 2) that together show that CV does not estimate the error of the specific model fit on the observed training set, but is instead estimating the average error over many training sets (Corollary 2 and Corollary 3). We also show that this holds for the other common estimates of prediction error: data splitting (Section 3.4), Mallow’s $C_p$ (Section 3.5), and bootstrap (Supplementary Appendix A). Second, we introduce a modified cross-validation scheme to give accurate confidence intervals for prediction error. We prove that our estimate for the MSE of the CV point estimate is unbiased (Theorem 3). Moreover, we validate our method with extensive numerical experiments, confirming that the coverage is consistently better than that of standard cross-validation (Section 5).

2. Setting and notation

We consider the supervised learning setting where we have features $X=\left(X_1,\dots ,X_n\right)\in {\mathcal{X}}^n$ and response $Y=\left(Y_1,\dots ,Y_n\right)\in {\mathcal{Y}}^n$, and we assume that the data points $\left(X_i,Y_i\right)$ for $i=1,\dots ,n$ are i.i.d. from some distribution $P$. We wish to understand how well fitted models generalize to unseen data points, which we formalize with a loss function $\ell \left(\widehat{y},y\right):\mathcal{Y}\times \mathcal{Y}\to {ℝ}_{\ge 0}$ such that $\ell \left(y,y\right)=0$ for all $y$. For example, $\ell$ could be squared error loss, misclassification error, or deviance (cross-entropy). Now consider a class of models parameterized by $\theta$. Let $\widehat{f}\left(x,\theta \right)$ be the function that predicts $y$ from $X\in {ℝ}^p$ using the model with parameters $\theta$, which takes values in some space Θ. Let $\mathcal{A}$ be a model-fitting algorithm that takes any number of data points and returns a parameter vector $\widehat{\theta }\in \text{Θ}$. Let $\widehat{\theta }=\mathcal{A}\left(X,Y\right)$ be the fitted value of the parameter based on the observed data $X$ and $Y$. We are interested in the out-of-sample error with this choice of parameters:

$${\text{Err}}_{XY}:=\mathbb{E}\left[\ell \left(\widehat{f}\left(X_{n+1},\widehat{\theta }\right),Y_{n+1}\right)|\left(X,Y\right)\right],$$

where $\left(X_{n+1},Y_{n+1}\right)\sim P$ is an independent test point from the same distribution. Notice ${\text{Err}}_{XY}$ is a random quantity, depending on the training data. We denote the expectation of this quantity across possible training sets as

$$\text{Err}:=\mathbb{E}\left[{\text{Err}}_{XY}\right]$$

We will discuss the relationship between these two quantities further in Section 3. We note that out-of-sample error is materially different from in-sample-error which is the focus of methods like the $C_p$ and AIC statistics, and covariance penalties. These are discussed in Section 3.5.

In cross-validation, we partition the observations $\mathcal{I}=\left\{1,\dots ,n\right\}$ into $K$ disjoint subsets (folds) ${\mathcal{I}}_1,\dots ,{\mathcal{I}}_K$ of size $m=n/K$ at random. Throughout this work, we will assume $K$ divides $n$ for convenience, and we will choose $K=10$ in all of our numerical results. Consider the first fold, and let ${\widehat{\theta }}^{\left(-1\right)}=\mathcal{A}\left({\left(X_j,Y_j\right)}_{j\in \mathcal{I}\backslash {\mathcal{I}}_1}\right.$ be the model fit to only those points that are not in fold one. Then, let $e_i=\ell \left(\widehat{f}\left(x_i,{\widehat{\theta }}^{\left(-1\right)}\right),y_i\right)$ for each $i\in {\mathcal{I}}_1$. The errors $e_i$ for points in other folds are defined analogously. We let

$${\text{Err}}^{\left(\text{CV}\right)}:=\overline{e}=\frac{1}{n}\sum _{i=1}^n{e}_i$$ (2)

be the average error, which is the usual CV estimate of prediction error. If one desires a confidence interval for the prediction error, a straightforward approach is to compute the empirical standard deviation of the $e_i$ divided by $\sqrt{n}$ to get an estimate of the standard error:

$$\text{se}:=\frac{1}{\sqrt{n}}\sqrt{\frac{1}{\left(n-1\right)}\sum _{i=1}^n{\left(e_i-\overline{e}\right)}^2}.$$

From here, we can create a confidence interval as

$$\left(\overline{e}-z_{1-\alpha /2}\cdot \text{se},\overline{e}+z_{1-\alpha /2}\cdot \text{se}\right),$$

where $z_{1-\alpha /2}$ is the $1-\alpha /2$ quantile of the standard normal distribution. We call these the naïve cross-validation intervals and they serve as our baseline approach. Importantly, we find that these naïve CV intervals are on average too small because the true standard deviation of $\overline{e}$ is larger than the naïve estimate $\text{se}$ would suggest, so a better estimate of the standard error is needed.

3. What prediction error are we estimating?

We next discuss targets of inference when assessing prediction accuracy. We discuss both $\text{Err}$ and ${\text{Err}}_{XY}$, and also introduce an intermediate quantity ${\text{Err}}_X$ that explains the connection between these two. While cross-validation is our focus, our results hold identically for other estimates of prediction error: covariance penalties (Section 3.5), data splitting (Section 3.4), and bootstrap (Supplementary Appendix A).

3.1. ${\text{Err}}_X$: a different target of inference

The two most natural estimands of interest to the analyst are ${\text{Err}}_{XY}$, the error of the model that was fit on our actual training set, and $\text{Err}$, the average error of the fitting algorithm run on same-sized datasets drawn from the underlying distribution $P$. The former quantity is of the most interest to a practitioner deploying a specific model, whereas the latter may be of interest to a researcher comparing different fitting algorithms. While it may initially appear that the quantity ${\text{Err}}_{XY}$ is easier to estimate—since it concerns the model at hand—it has been observed that the cross-validation estimate provides little information about ${\text{Err}}_{XY}$ (Zhang, 1995; Hastie et al., 2009; Yousef, 2020), a phenomenon sometimes called the weak correlation issue.

We now prove that CV has lower MSE for estimating $\text{Err}$ than it does for ${\text{Err}}_{XY}$, for the special case of the linear model. In this sense, CV should be viewed as an estimate of $\text{Err}$ rather than of ${\text{Err}}_{XY}$. In order to state this formally, for this section only, assume the homoskedastic linear model holds:

$$y_i=x_i^{\top }\theta +{ϵ}_i\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{where}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{ϵ}_i\stackrel{i.i.d}{\sim }\mathcal{N}\left(0,{\sigma }^2\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,n,$$ (3)

with $ϵ=\left({ϵ}_1,\dots ,{ϵ}_n\right)$ independent of $X$. In this setting, a key quantity in our analysis is ${\text{Err}}_X:=\mathbb{E}\left[{\text{Err}}_{XY}|X\right],$.

which falls between $\text{Err}$ and ${\text{Err}}_{XY}$; see Figure 2 for a visualization. This quantity is also considered by Hastie et al. (2019) in a high-dimensional regression setting, but to the best of our knowledge has not been considered in the literature on estimation of prediction error.

Fig. 2.

Possible targets of inference for cross-validation. Here, $\left(X,Y\right)$ is the training data and ${\text{Err}}_{XY}$ is the average error of the model fit on $\left(X,Y\right)$ on a test data set of infinite size. From left to right, the random variables above are a constant, a function of X only, and a function of $\left(X,Y\right)$.

While our current focus is on cross-validation, the conclusions hold for a broad class of estimates of prediction error. In particular, we consider estimators of prediction error that satisfy the following property:

Definition 1 (Linearly invariant estimator). We say that an estimator of prediction error $\text{Err}\left(\left(X_1,Y_1\right),\dots ,\left(X_n,Y_n\right),U\right)$ is linearly invariant if for all $x_i,y_i,u$ we have

$$\text{Err}\left(\left(x_1,y_1\right),\dots ,\left(x_n,y_n\right),u\right)=\text{Err}\left(\left(x_1,y_1+x_1^{\top }\kappa \right),\dots ,\left(x_n,y_n+x_n^{\top }\kappa \right),u\right).$$ (4)

Here $\kappa$ is any p-vector and the random variable U is included to allow for randomized procedures like cross-validation, and without loss of generality it is taken to be $\text{unif}\left[0,1\right]$ and independent of $\left(X,Y\right)$.

With ordinary least square (OLS) fitting, cross-validation satisfies this property:

Lemma 1. When using OLS as the fitting algorithm and squared-error loss, the cross-validation estimate of prediction error, ${\text{Err}}^{\left(\text{CV}\right)}$, is linearly invariant.

Note that linear invariance is a deterministic property of an estimator and does not rely on any distributional assumptions.

Recall from classical linear regression theory that when using ordinary least squares (OLS), the estimated coefficient vector is independent of the residual sum of squares. This implies that the sum of squared residuals is independent of the true predictive error. It turns out that even further, the CV estimate of error (and all linearly invariant estimates of error) is independent of the true error, conditional on the feature matrix $X$.

Theorem 1. Assume the homoskedastic Gaussian linear model (3) holds and that we use squared-error loss. Let $\text{Err}$ be a linearly invariant estimate of prediction error (such as ${\text{Err}}^{\left(\text{CV}\right)}$ using OLS as the fitting algorithm). Then,

$$\text{Err}\perp \perp {\text{Err}}_{XY}|X.$$ (5)

The proof of this theorem rests primarily on the fact that the OLS residuals are independent of the fitted coefficient vector in the linear model, together with the observation that linearly invariant estimators are a function only of the residuals of an OLS model fit. Due to its simplicity, we give the proof explicitly here; all other proofs are given in Supplementary Appendix D.

Proof of Theorem 1. The true predictive error $\left({\text{Err}}_{XY}\right)$ is a function only of $\widehat{\theta }$, the OLS estimate of $\theta$ based on the full sample $\left(x_1,y_1\right),\dots ,\left(x_n,y_n\right)$. On the other hand, any linearly invariant $\text{Err}$ is a function only of the residuals $Y-X\widehat{\theta }=\left(I-X{\left(X^{\top }X\right)}^{-1}{X}^{\top }\right)Y$ by the invariance property (see Supplementary Appendix Lemma 6). Since $\widehat{\theta }\perp \perp \left(Y-X\widehat{\theta }\right)|X$, from classical linear model results, the proof is complete. □

As a result, any linearly invariant estimator (such as cross-validation) has lower MSE as an estimate of ${\text{Err}}_X$ than as an estimate of ${\text{Err}}_{XY}$:

Corollary 1. Under the conditions of Theorem 1,

$$\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_{XY}\right)}^2\right]=\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_X\right)}^2\right]+\underset{\ge 0}{\underbrace{\mathbb{E}\left[\mathrm{var}\left({\text{Err}}_{XY}|X\right)\right]}}.$$

We demonstrate this in an experiment in a simple linear model with $n=100$ observations and $p=20$ features, where the features are i.i.d. standard normal variables; see Figure 3. As predicted by Corollary 1, we see that the CV point estimate has lower MSE for ${\text{Err}}_X$ than for ${\text{Err}}_{XY}$. Similarly, the naïve CV intervals cover ${\text{Err}}_X$ more often than they cover ${\text{Err}}_{XY}$.

Fig. 3.

Left: mean squared error of the CV point estimate of prediction error relative to three different estimands: $\text{Err}$, ${\text{Err}}_X$, and ${\text{Err}}_{XY}$. Center: coverage of $\text{Err}$, ${\text{Err}}_X$, and ${\text{Err}}_{XY}$ by the naïve cross-validation intervals in a homoskedastic Gaussian linear model. The nominal miscoverage rate is 10%. Each pair of points connected by a line represents 2000 replicates with the same feature matrix $X$. Right: 2000 replicates with the same feature matrix and the line of best fit (blue).

3.2. Relationship with average error

The results of the previous section suggest that ${\text{Err}}_X$ is a more natural target of inference than ${\text{Err}}_{XY}$ . Next, we examine the relationship between $\text{Err}$ and ${\text{Err}}_X$, showing that ${\text{Err}}_X$ is close to $\text{Err}$, in that the variance of ${\text{Err}}_X$ (which has mean $\text{Err}$) is small compared with the variance of ${\text{Err}}_{XY}$ (which also has mean $\text{Err}$). Combined with the results of the previous section, this gives a formal statement that cross-validation is a better estimator for $\text{Err}$ than for ${\text{Err}}_{XY}$.

To make this precise, consider the conditional variance decomposition of the variance of ${\text{Err}}_{XY}$,

$$\mathrm{var}\left({\text{Err}}_{XY}\right)=\underset{\mathrm{var}\hspace{0.17em}\text{due}\hspace{0.17em}\hspace{0.17em}\text{to}\hspace{0.17em}\hspace{0.17em}Y|X\hspace{0.17em}\hspace{0.17em}}{\underbrace{{\mathbb{E}}_X\left[\mathrm{var}\left({\text{Err}}_{XY}|X\right)\right]}}+\underset{\text{vardueto}X\hspace{0.17em}\hspace{0.17em}}{\underbrace{\mathrm{var}\left({\text{Err}}_X\right)}}.$$ (6)

To quantify the relative contribution of the two terms in the right-hand side of (6), we will use a proportional asymptotic limit, where

$$n>p,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}n,p\to \infty ,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}n/p\to \lambda >1.$$ (7)

We use the proportional asymptotic limit rather than traditional $p$ fixed, $n\to \infty$ asymptotics, because in the latter asymptotic regime, the difference between $\text{Err}$, ${\text{Err}}_X$, and ${\text{Err}}_{XY}$ is asymptotic order lower than $1/\sqrt{n}$, so one always estimates these three targets with equal precision, and the analysis is less informative. See Supplementary Appendix H for a complementary analysis in the traditional $p$ fixed, $n\to \infty$ asymptotic regime and Yang (2007); Wager (2020) for a related discussion. By contrast, in the proportional asymptotic limit we will see that ${\text{Err}}^{\left(\text{CV}\right)}$ is closer to $\text{Err}$ and ${\text{Err}}_X$ than to ${\text{Err}}_{XY}$.

Theorem 2. Suppose the homoskedastic Gaussian linear model in (3) holds and that we use squared-error loss. In addition, assume that feature vectors $X_i\sim \mathcal{N}\left(0,{\Sigma }_p\right)$ for any full-rank ${\Sigma }_p$. Then, in the proportional asymptotic limit in (7), we have ${\mathbb{E}}_X\left[\mathrm{var}\left({\text{Err}}_{XY}|X\right)\right]=\text{Θ}\left(1/n\right)$ and $\mathrm{var}\left({\text{Err}}_X\right)=\mathbb{E}{\left({\text{Err}}_X-\text{Err}\right)}^2=\text{Θ}\left(1/n^2\right)$, as $n,p\to \infty$.

We summarize the asymptotic relationship among the various estimands in Figure 4. We see that the randomness caused by $Y$ given $X$ is of a larger order than that due to the randomness in $X$. This explains why in Figure 3, the coverage and MSE of cross-validation is similar when estimating either $\text{Err}$ or ${\text{Err}}_X$, but is significantly different when estimating ${\text{Err}}_{XY}$. As a result, ${\text{Err}}_X$ and ${\text{Err}}_{XY}$ are asymptotically uncorrelated, and moreover, combining this with Theorem 1 shows that ${\text{Err}}^{\left(\text{CV}\right)}$ is asymptotically uncorrelated with ${\text{Err}}_{XY}$, as stated next.

Fig. 4.

The relationship among various notions of prediction error in the proportional asymptotic limit (7). Recall that ${\sigma }^2$ is the Bayes error: the error rate of the best possible model. See Figure 5 for a simulation experiment demonstrating these rates. * The variance of $\text{Err}$ scales as $1/\sqrt{n}$; see Section 3.3 for details about the bias.

Corollary 2. In the setting of Theorem 2, $\text{cor}\left({\text{Err}}_{XY},{\text{Err}}_X\right)\to 0$ as $n,p\to \infty$. Moreover, for any linearly invariant estimator $\text{Err}$ (such as ${\text{Err}}^{\left(\text{CV}\right)}$ using OLS as the fitting algorithm),

$\text{cor}\left({\text{Err}}_{XY},{\text{Err}}_X\right)\to 0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{as}\hspace{0.17em}\hspace{0.17em}n,p\to \infty$

Notice that this is a marginal result, whereas the similar Theorem 1 is conditional on $X$. With respect to Figure 4, this result means that the fluctuations of ${\text{Err}}_{XY}$ around $\text{Err}$ are asymptotically uncorrelated with the fluctuations of of $\text{Err}$ around $\text{Err}$. Combining Theorem 2 with Theorem 1, we conclude that CV has larger error for estimating ${\text{Err}}_{XY}$ than for $\text{Err}$ or ${\text{Err}}_X$:

Corollary 3. In the setting of Theorem 2, let $\text{Err}$ be any linearly invariant estimator (such as ${\text{Err}}^{\left(\text{CV}\right)}$ using OLS as the fitting algorithm). Suppose in addition that $\mathrm{var}\left(\text{Err}\right)\to 0$ (an extremely weak condition satisfied by any reasonable estimator). Then,

$$\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_{XY}\right)}^2\right]-\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_X\right)}^2\right]=\Omega \left(1/n\right),$$

$$\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_{XY}\right)}^2\right]-\mathbb{E}\left[{\left(\text{Err}-\text{Err}\right)}^2\right]=\Omega \left(1/n\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}$$

$$\left|\mathbb{E}\left[{\left(\text{Err}-\text{Err}\right)}^2\right]-\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_X\right)}^2\right]\right|=o\left(1/n\right).$$

The asymptotic theory perfectly predicts the experimental results presented in Figure 5; we see that even for moderate sample size, the scalings is exactly as anticipated. The main conclusion is that for a linearly invariant estimate of prediction error that has precision $1/\sqrt{n}$, our results show that asymptotically one has lower estimation error when estimating $\text{Err}$ compared to ${\text{Err}}_{XY}$. Similarly, the correlation between a linearly invariant estimate and ${\text{Err}}_{XY}$ goes to zero. These theoretical predictions are also corroborated by the experimental results presented in Supplementary Appendix Figure 6. Thus, cross-validation is estimating the average error $\text{Err}$ more so than the specific error ${\text{Err}}_{XY}$.

Fig. 5.

Simulation results demonstrating the asympotic scaling presented in Figure 4. The fitted slopes of the lines (after log-transforming both axes) are 0.00, −0.46, −0.50, −1.01, from top to bottom. See Section 3.3 for details about the rate of $\text{Err}$

Remark 1. Note that the results in this section apply both to K-fold cross-validation with fixed K, and leave-one-out cross validation where K = n. Formally, the results require only that one is using some sequence of linearly invariant estimators.

3.3. The bias of cross-validation

Up until this point, we have not explicitly mentioned the bias in the CV point estimate $\text{Err}$ that comes from the difference in sample size. That is, $\text{Err}$ uses models of size $n\left(K-1\right)/K$, whereas $\text{Err}$ and ${\text{Err}}_{XY}$ are defined for models fit on data of size $n$, so $\mathbb{E}\left[\text{Err}\right]$ is typically smaller than $\text{Err}=\mathbb{E}\left[{\text{Err}}_{XY}\right]$. We now pause for a few remarks about this bias. First, notice that Corollary 3 sidesteps the bias issue by considering differences between two mean squared error quantities. The bias is important, however, if we wish to understand absolute quantities such as $\mathbb{E}\left[{\left(\text{Err}-\text{Err}\right)}^2\right]$. To this end, the bias exhibits different behavior in different regimes¹:

• The parametric regime. Suppose $p$ is fixed, $n\to \infty$, and the model class has fixed dimension. Here, the bias will typically be of order $1/n$, which means that it is negligible compared to the variance. (In fact, the dimension of the model class can grow, provided the rate is slow enough; see Wager (2020) for discussion.)

• The proportional, dense regime. Consider the setting above in (7), fitting a dense model. If the number of folds is fixed, the bias of Err will converge to a nonzero constant as $n$ and $p$ grow (e.g., Liu and Dobriban, 2020). What this means is that in Figure 5, the $\left|\text{Err}-\text{Err}\right|$ curve will eventually cease to decay at a $1/\sqrt{n}$ rate, bottoming out due to the constant bias. We do not see this behavior in the plot, because the bias is still much smaller than the variance at the sample sizes we consider; Supplementary Appendix Figure F.1 reports the bias and variance in this setting.

• The proportional, sparse regime. The setting is the most delicate. Here, the behavior of sparse regression algorithms may have very different behavior on samples of size $n\left(K-1\right)/K$ versus samples of size $n$ (e.g., Reeves et al., 2019). Thus, the bias here may be appreciable.

In all cases, the bias can be mitigated by taking a larger number of folds as $n$ grows.

Incorporating bias alongside our results from Section 3.2 leads to an interesting bias-variance-variance decomposition of $\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_{XY}\right)}^2\right]$. Since both $\text{Err}$ and ${\text{Err}}_{XY}$ are random quantities, we cannot use the usual bias-variance decomposition. However, by Corollary 2, these two quantities are asymptotically uncorrelated, yielding the following:

$$\mathbb{E}\left[{\left(\text{Err}-{\text{Err}}_{XY}\right)}^2\right]\approx \underset{\text{bias}}{\underbrace{{\left(\mathbb{E}\left[\text{Err}\right]-\text{Err}\right)}^2}}+\underset{\text{variance}\hspace{0.17em}\hspace{0.17em}\text{of}\hspace{0.17em}\hspace{0.17em}\text{Err}}{\underbrace{\mathbb{E}\left[{\left(\text{Err}-\mathbb{E}\left[\text{Err}\right]\right)}^2\right]}}+\underset{\text{variance}\hspace{0.17em}\hspace{0.17em}\text{of}\hspace{0.17em}\hspace{0.17em}{\text{Err}}_{XY}}{\underbrace{\mathbb{E}\left[{\left({\text{Err}}_{XY}-\text{Err}\right)}^2\right]}}.$$

The first two terms on the right hand side are the bias-variance decomposition for $\text{Err}$ as an estimate of $\text{Err}$. Thus, because the additional third term is positive, we again see that Err is a more precise estimate of Err than of ${\text{Err}}_{XY}$.

3.4. Data splitting

Perhaps the simplest way to estimate prediction error is to split the data into two disjoint sets, one for training and one for estimating the prediction accuracy. The previous results also shed light on the properties of data splitting. In particular, we will show that when estimating prediction error with data splitting, refitting the model on the full data incurs additional variance that can make the confidence intervals slightly too small, even asymptotically. This is not a cause for practical concern, but it is another manifestation of the fact that linearly invariant estimators are estimating average prediction error, and ${\text{Err}}_{XY}$ contains additional, independent variation. We report on the details in Supplementary Appendix B.

3.5. Connection with covariance penalties

For parametric models, there is an alternative theory of the estimation of prediction accuracy based on covariance penalties; see Stein (1981); Efron (2004); Rosset and Tibshirani (2020) for overviews of this approach. For the linear model with OLS and squared error loss, this approach specializes to the well-known Mallows $C_p$ (Mallows, 1973; Akaike, 1974) estimate of prediction error:

$${\text{Err}}^{\left(C_p\right)}:=\frac{1}{n}\sum _{i=1}^n{\left(y_i-\widehat{f}\left(x_i,\widehat{\theta }\right)\right)}^2+\frac{2p{\widehat{\sigma }}^2}{n}.$$

We first consider the estimation of prediction error with Mallows $C_p$, showing that (like cross-validation) it is a worse estimator for ${\text{Err}}_{XY}$ than for $\text{Err}$. The results from Section 3.1 and Section 3.2 continue to hold for ${\text{Err}}^{\left(C_p\right)}$ and ${\text{Err}}^{\left(R{C}_p\right)}$, since they are linearly invariant:

Lemma 2. The estimators ${\text{Err}}^{\left(C_p\right)}$ and ${\text{Err}}^{\left(R{C}_p\right)}$ are linearly invariant.

This result is immediate from the fact the ${\text{Err}}^{\left(C_p\right)}$ and ${\text{Err}}^{\left(R{C}_p\right)}$ are functions only of the residuals of the OLS fit. Thus, the conclusions of Theorem 1, Corollary 1, Corollary 2, and Corollary 3 hold for ${\text{Err}}^{\left(C_p\right)}$. In particular, ${\text{Err}}^{\left(C_p\right)}$ has lower error for estimating $\text{Err}$ and ${\text{Err}}_X$ than for estimating ${\text{Err}}_{XY}$, and ${\text{Err}}^{\left(C_p\right)}$ is asymptotically uncorrelated with ${\text{Err}}_{XY}$. In summary, as before with cross-validation, Mallow’s $C_p$ is not able to estimate ${\text{Err}}_{XY}$, but is rather an estimate of ${\text{Err}}_{\left(\text{in}\right)}$, $\text{Err}$ or ${\text{Err}}_X$ (the latter two are close for large samples).

3.6. Bootstrap estimates of prediction error

Bootstrap estimates of prediction error are also linearly invariant, and so they are also estimates of the average prediction error. For brevity, we present these results in Supplementary Appendix A.

4. Confidence intervals with nested cross-validation

In this section, we develop an estimator for the MSE of the cross-validation point estimate. Our ultimate goal is then to use the estimated MSE to give confidence intervals for prediction error with approximately valid coverage.

4.1. Dependence structure of CV errors

Before developing our estimator for the cross-validation MSE, we pause here to build up intuition for why the naïve CV confidence intervals for prediction error can fail, as seen previously in our example in Section 1.1. The naïve CV intervals are too small, on average, because the true sampling variance of ${\text{Err}}^{\left(\text{CV}\right)}$ is larger than the naïve estimate $\text{se}$ would suggest. In particular, this estimate of the variance of the CV point estimate assumes that the observed errors $e_1,\dots ,e_n$ are independent. This is not true—the observed errors have less information than an independent sample since each point is used for both training and testing, which induces dependence among these terms. In particular, the covariance matrix of the errors $e_1,\dots ,e_n$ has the block structure shown in Figure 6; see (Bengio and Grandvalet, 2004). Thus, the usual estimate of the variance of ${\text{Err}}^{\left(\text{CV}\right)}$ is too small, resulting in poor coverage. To remedy this issue, we now develop an estimator that empirically estimates the variance of ${\text{Err}}^{\left(\text{CV}\right)}$ across many subsamples. Avoiding the faulty independence approximation leads to intervals with superior coverage.

Fig. 6.

Covariance structure of the CV errors. Red corresponds to the covariance between points in the same fold, and blue corresponds to the covariance between points in different folds.

4.2. Our target of inference

In this section, our primary goal will be to give confidence intervals for test accuracy by estimating the mean-squared error (MSE) of cross-validation:

Definition 2. For a sample of size n split into K folds, the cross-validation MSE is

$${\text{MSE}}_{K,n}:=\mathbb{E}\left[{\left({\text{Err}}^{\left(\text{CV}\right)}-{\text{Err}}_{XY}\right)}^2\right].$$ (8)

In particular, we define MSE with respect to ${\text{Err}}_{XY}$ and thus will calibrate our test intervals to cover the quantity ${\text{Err}}_{XY}$. At this point, the reader should wonder why we define the MSE with respect to ${\text{Err}}_{XY}$ in view of the results from Section 3 that show that ${\text{Err}}^{\left(\text{CV}\right)}$ is a more precise estimate of $\text{Err}$ than of ${\text{Err}}_{XY}$, and we will next discuss this issue carefully.

To be clear, we choose to pursue confidence intervals for ${\text{Err}}_{XY}$ because we are able to do so; the MSE quantity above can be estimated in a convenient way due to an upcoming decomposition (Lemma 3). At present, we do not know how to obtain a similar MSE estimate with respect to $\text{Err}$. Second, we emphasize that our results from Section 3 do not mean that giving confidence intervals for ${\text{Err}}_{XY}$ is impossible. Rather, our results say that in the linear model, confidence intervals for ${\text{Err}}_{XY}$ will be larger than confidence intervals for $\text{Err}$. Still, confidence intervals for either Err or ${\text{Err}}_{XY}$ would be of interest to the analyst. We are able to derive an estimator for the MSE with respect to ${\text{Err}}_{XY}$, and we will turn to the details next.

The MSE in (8) contains both a bias term (due to the reduced sample size used by ${\text{Err}}^{\left(\text{CV}\right)}$) and variance term. See Section 3.3 for a discussion of the bias. Thus, we can view the MSE as a slightly conservative version of the variance of the cross-validation estimator. In any case, the MSE is the relevant quantity for creating confidence intervals around a possibly biased point estimate, since it accounts for both bias and variance. With this in mind, we will use an estimate of the MSE to construct confidence intervals for ${\text{Err}}_{XY}$. Previewing the remainder of this section, we will produce confidence intervals for ${\text{Err}}_{XY}$ as follows:

$$\left({\text{Err}}^{\left(\text{NCV}\right)}-\text{bias}-z_{1-\alpha /2}\cdot \sqrt{\text{MSE}},\hspace{0.17em}\hspace{0.17em}{\text{Err}}^{\left(\text{NCV}\right)}-\text{bias}+z_{1-\alpha /2}\cdot \sqrt{\text{MSE}}\right).$$ (9)

Above, ${\text{Err}}^{\left(\text{NCV}\right)}$ is similar to the CV estimate of error except across many random splits, and $\text{MSE}$ is our estimator for MSE—the heart of this section. In addition, we allow for the possibility of correcting for the sample size bias with an estimator $\text{bias}$; we use one that arises naturally from the computations already carried out to estimate the MSE (Section 4.3.3).

4.3. A nested CV estimate of MSE

4.3.1. A holdout MSE identity

We now give a generic decomposition of the mean-squared error of an estimate of prediction error, which will enable use to estimate ${\text{MSE}}_{K,n}$. Consider a single split of the data into a training set and holdout set, i.e., we partition $\mathcal{I}=\left\{1,\dots ,n\right\}$ into ${\mathcal{I}}_{\left(\text{train}\right)}$ and ${\mathcal{I}}_{\left(\text{out}\right)}$ calling the training set $\left(\tilde{X},\tilde{Y}\right)$. Using only $\left(\tilde{X},\tilde{Y}\right)$, we use our fitting procedure to obtain estimated parameters ${\widehat{\theta }}^{\left(\text{train}\right)}=\mathcal{A}\left(\tilde{X},\tilde{Y}\right)$, and further assume we have some estimate of ${\text{Err}}_{\tilde{X}\tilde{Y}}$ of the prediction error ${\text{Err}}_{\tilde{X}\tilde{Y}}$ defined in Supplementary Appendix (12). Here, ${\text{Err}}_{\tilde{X}\tilde{Y}}$ is any estimator of ${\text{Err}}_{\tilde{X}\tilde{Y}}$ based only on $\left(\tilde{X},\tilde{Y}\right)$, such as

cross-validation using only $\left(\tilde{X},\tilde{Y}\right)$. Let ${\left\{e_i^{\left(\text{out}\right)}\right\}}_{i\in {\mathcal{I}}_{\left(\text{out}\right)}}$ be the losses of the fitted model $\widehat{f}\left(\cdot ,{\widehat{\theta }}^{\left(\text{train}\right)}\right)$ on the holdout set, and let ${\overline{e}}^{\left(\text{out}\right)}$ be their average. The MSE of ${\text{Err}}_{\tilde{X}\tilde{Y}}$ can be written as follows:

Lemma 3 (Holdout MSE identity). In the setting above

$$\underset{\text{MSE}}{\underbrace{\mathbb{E}\left[{\left({\text{Err}}_{\tilde{X}\tilde{Y}}-{\text{Err}}_{\tilde{X}\tilde{Y}}\right)}^2\right]}}=\underset{\left(\text{a}\right)}{\underbrace{\mathbb{E}\left[{\left({\text{Err}}_{\tilde{X}\tilde{Y}}-{\overline{e}}^{\left(\text{out}\right)}\right)}^2\right]}}-\underset{\left(\text{b}\right)}{\underbrace{\mathbb{E}\left[{\left({\overline{e}}^{\left(\text{out}\right)}-{\text{Err}}_{\tilde{X}\tilde{Y}}\right)}^2\right]}}.$$ (10)

The expectations above are over the complete data $\left(X,Y\right)$. The lemma follows from adding and subtracting ${\text{Err}}_{\tilde{X}\tilde{Y}}$ within term (a) then showing the cross-term is zero with a nested conditional expectation argument.

This identity is of interest, since both (a) and (b) can be estimated from the data, which leads to an estimate of the MSE term. Specifically, we propose the following estimation strategy:

1. Repeatedly split the data into ${\mathcal{I}}_{\left(\text{train}\right)}$ and ${\mathcal{I}}_{\left(\text{out}\right)}$, and for each split, do the following:

   1. Apply cross-validation to ${\mathcal{I}}_{\left(\text{train}\right)}$ to obtain ${\text{Err}}_{\tilde{X}\tilde{Y}}$ and use ${\mathcal{I}}_{\left(\text{out}\right)}$ to obtain ${\overline{e}}^{\left(\text{out}\right)}$, and then estimate (a) with ${\left({\text{Err}}_{\tilde{X}\tilde{Y}}-{\overline{e}}^{\left(\text{out}\right)}\right)}^2$.
   2. Estimate (b) with empirical variance of ${\left\{e_i\right\}}_{i\in {\mathcal{I}}_{\left(\text{out}\right)}}$ divided by the size of ${\mathcal{I}}_{\left(\text{out}\right)}$.

2. Average together estimates of (a) and (b) across all random splits and take their difference as in (10) to obtain an estimate of MSE.

Note that the estimates for both (a) and (b) are unbiased, so the resulting MSE estimate is unbiased for the MSE term in (10). In the next section, we will pursue this strategy for the particular case where ${\text{Err}}_{\tilde{X}\tilde{Y}}$ is itself a cross-validation estimate based only on $\left(\tilde{X},\tilde{Y}\right)$.

4.3.2. The MSE estimator

Building from Lemma 3, we now turn to our proposed estimate of MSE, the heart of this section. We follow the estimation strategy described above, using $\left(K-1\right)$-fold CV as the estimator ${\text{Err}}_{\tilde{X}\tilde{Y}}$. This gives an estimate of (a) and (b), and hence an estimate for the MSE, as described above. We also get a point estimate of error by taking the empirical mean of ${\text{Err}}_{\tilde{X}\tilde{Y}}$ across the many splits. See Figure 7 for a visualization of the nested CV sample splitting, and see Algorithm ?? in the Supplementary Appendix for a detailed description. We denote the resulting estimate of mean squared error by ${\text{MSE}}^{\left(\text{NCV}\right)}$ and the point estimate for prediction error ${\text{Err}}^{\left(\text{NCV}\right)}$ In view of Lemme 3, we see that the estimator ${\text{MSE}}^{\left(\text{NCV}\right)}$ is targeting the MSE of ${\text{Err}}^{\left(\text{CV}\right)}$ as an estimate of ${\text{Err}}_{XY}$, as we record formally next.

Fig. 7.

Visualization of nested CV. Using only the folds on left of the vertical line, we perform the usual cross-validation by holding out one fold at a time (the dark grey fold) and then fitting on the remaining folds (the light grey folds). The fresh holdout points (the blue fold) are never used in the inner CV step.

Theorem 3 (Estimand of nested CV). For a nested CV with a sample of size $n$, $\mathbb{E}\left[{\text{MSE}}^{\left(\text{NCV}\right)}\right]\approx {\text{MSE}}_{K-1,n\text{'}}$, where $n\text{'}=n\left(K-1\right)/K$.

This result shows that ${\text{MSE}}^{\left(\text{NCV}\right)}$ obtained by nested CV is estimating the MSE of $\left(K-1\right)$-fold cross-validation on a sample of size $n\left(K-1\right)/K$. Since nested CV uses an inner loop with samples of size $n\left(K-1\right)/K$, we recommend re-scaling to obtain an estimate for a sample of size n by instead taking $\text{MSE}=\left(K-1\right)/K\cdot {\text{MSE}}^{\left(\text{NCV}\right)}$ (although this re-scaled version is not guaranteed to be exactly unbiased for ${\text{MSE}}_{K,n}$). As a minor detail, in practice we also restrict $\sqrt{\text{MSE}}$ to fall between $\text{se}$ (the estimated standard error if one had n independent points) and $\sqrt{K}\cdot \text{se}$ (the estimated standard error if one had only $n/K$ independent points). This is a minor implementation detail prevents implausible values of $\text{MSE}$ from arising. After adjusting the point estimate ${\text{Err}}^{\left(\text{NCV}\right)}$ with a bias correction discussed next, we form our final confidence intervals as in (9).

Remark 2 (The sample size difference and the target of inference.). Note that the estimator ${\text{Err}}^{\left(\text{NCV}\right)}$ uses models fit on with $n\left(K-2\right)/K$ data points, whereas the target of inference in Theorem 3, ${\text{MSE}}_{K-1,n\text{'}}$, is defined with respect to the prediction accuracy of a model fit with $n\left(K-2\right)/K$ points. How can the former be used to estimate the latter? The answer is that the nested CV procedure relies also on fits of size $n\left(K-2\right)/K$, see the definition of $\widehat{\theta }$ in the “nested_crossval” subroutine of Algorithm ??. Nested CV compares the predicted accuracy on the models from $n\left(K-2\right)/K$ data points to the estimated accuracy of the models with $n\left(K-2\right)/K$ data points, using the extra holdout data to asses this accuracy.

4.3.3. Estimation of bias

The nested CV computations also yield a convenient estimate of the bias of the NCV point estimate of error, ${\text{Err}}^{\left(\text{NCV}\right)}$. They key idea is that nested CV considers both models fit with $n\left(K-2\right)/K$ data points and with $n\left(K-2\right)/K$ data points, and comparing their these models gives an estimate of bias; see Supplementary Appendix C. This aspect of nested CV is not critical—the MSE estimation above is the core of our proposal.

5. Simulation experiments

We now explore the coverage of nested CV in a variety of settings. In each case, we will report the coverage of naïve CV (CV), nested CV (NCV), and data splitting with refitting (DS), where the nominal miscoverage rate is 10% (5% miscoverage in each tail). We also report on the width of the intervals, expressed relative to the width of the standard CV intervals. (We wish to produce intervals that are as narrow as possible while maintaining correct coverage.) We use 10-fold CV (the number of folds has little impact; see Supplementary Appendix F.4) and NCV, with 200 random splits for the latter; see Supplementary Appendix F.2 for the runtime of each experiment. For classification examples we use binary loss, and form confidence intervals for CV, NCV, and data splitting after taking the binomial variance-stabilizing transformation, described in detail in Supplementary Appendix G. For regression examples, we use squared error loss.

For data splitting, we use 80% of the samples for training and 20% for estimating prediction error. Note that the data splitting without refitting intervals are the same as the data splitting with refitting intervals; the difference is that they are intended to cover different quantities. To make this comparable to CV and nested CV, we report on the coverage of $\text{Err}$ and ${\text{Err}}_{XY}$ here, which corresponds to data splitting with refitting. Data splitting without refitting (which seeks to cover the quantity in Supplementary Appendix (12)) will typically have better coverage; we observed relatively accurate coverage in the classification examples and worse coverage in the regression examples, but do not explicitly report these results herein.

Scripts reproducing these experiments are available at https://github.com/stephenbates19/nestedcv_experiments.

5.1. Classification

5.1.1. Low-dimensional logistic regression

We consider the logistic regression data generating model 1 with $n=100$ observations and $p=20$ features, sampled as i.i.d. standard Gaussian variables. Due to the rotational symmetry of the features, the only parameter that affects behavior is the signal strength, and we explore models with Bayes error of either 33% or 23%. Here, we use (un-regularized) logistic regression as our fitting algorithm. We report the results in Table 1, finding that nested CV gives coverage much closer to the nominal level. Moreover, the point estimates have slightly less bias. We report the size of the NCV intervals relative to their CV counterparts per instance in Supplementary Appendix Figure F.2.

Table 1.

Performance of cross-validation (CV), nested cross-validation (NCV), and data splitting with refitting (DS) in the low-dimensional logistic regression model from Section 5.1.1. Each row is a setting with a different signal strength, indexed by the Bayes error: the error of the true model. The nominal total error rate is 10%, i.e., 5% above and below. A “Hi” miscoverage is one where the confidence interval is too large and the point estimate falls below the interval; conversely for a “Lo” miscoverage. The standard error in each coverage estimate reported is about 0.5%. The “Target” column indicates the target of coverage—the intervals are always generated identically, but we report the coverage of both $\text{Err}$ and ${\text{Err}}_{XY}$.

Setting		Width		Point estimates				Miscoverage
								CV		NCV		DS
Bayes Error	Target	NCV	DS	Err	CV	NCV	DS	Hi	Lo	Hi	Lo	Hi	Lo
33.2%	${\text{Err}}_{XY}$	1.23	2.23	39.1%	39.6%	39.0%	40.1%	10%	8%	3%	5%	7%	6%
”	$\text{Err}$	”	”	”	”	”	”	9%	8%	3%	4%	6%	5%
22.5%	${\text{Err}}_{XY}$	1.47	2.25	28.7%	30.4%	28.1%	33.3%	11%	3%	4%	1%	16%	4%
”	$\text{Err}$	”	”	”	”	”	”	10%	2%	5%	0%	15%	3%

Next, we return to the question of estimands, as in Section 3. Since we do not have analytical results for the logistic regression model, we explore this in simulation. Here, we consider problems where the Bayes error rate is 22.5%, and vary $n$ and $p$. We investigate two quantities. First, we investigate the correlation of ${\text{Err}}^{\left(\text{CV}\right)}$ and ${\text{Err}}_{XY}$, and we find that it is small but larger than in the OLS case. See Figure 8. Next, we check whether Err has higher precision for $\text{Err}$ than for ${\text{Err}}_{XY}$. To this end, we compute the expected value of $\left|{\text{Err}}^{\left(\text{CV}\right)}-\text{Err}\right|$ and of $\left|{\text{Err}}^{\left(\text{CV}\right)}-{\text{Err}}_{XY}\right|$, and plot their relative difference in the right panel of Figure 8. We find that the CV point estimate is again slightly more precise as an estimate of $\text{Err}$ than of ${\text{Err}}_{XY}$ in this setting.

Fig. 8.

Behavior of cross-validation with a logistic regression model. Left: the correlation between the point estimate and instance-specific error. Right: fraction change in mean absolute deviation of the point estimate with respect to ${\text{Err}}_{XY}$ versus $\text{Err}$.

5.1.2. High-dimensional sparse logistic regression

We return to the high-dimensional logistic regression model introduced in Section 1.1, generalizing slightly. We consider $n\in \left\{90,200\right\}$ with $p=1000$ features. The feature matrix has standard normal entries with an autoregressive covariance pattern such that adjacent columns have covariance $\rho$. In each case, we take $k=4$ nonzero entries of the covariance matrix and use sparse logistic regression. We report on the results in Table 2 and give the width² in Figure 9. Again, NCV gives intervals with coverage much closer to the nominal level.

Table 2.

Performance of cross-validation (CV), nested cross-validation (NCV), and data splitting (DS) in the high-d logistic regression model from Section 5.1.2. The nominal (target) error rate is 10%, i.e., 5% above and below. Other details as in Table 1.

Setting				Width		Point estimates				Miscoverage
										CV		NCV		DS
n		ρ	Target	NCV	DS	Bayes error	Err	CV	NCV	Hi	Lo	Hi	Lo	Hi	Lo
90	0		${\text{Err}}_{XY}$	1.53	2.24	22%	41.3%	41.8%	41.1%	16%	12%	6%	7%	9%	7%
”		”	$\text{Err}$	”	”	”	”	”	”	17%	13%	6%	8%	11%	9%
200	0		${\text{Err}}_{XY}$	1.66	2.26	22%	25.6%	26.7%	25.6%	14%	7%	3%	5%	9%	4%
”		”	$\text{Err}$	”	”	”	”	”	”	15%	7%	4%	6%	8%	5%
90		0.5	${\text{Err}}_{XY}$	1.80	2.25	13%	25.6%	27.5%	28.6%	20%	10%	5%	8%	15%	4%
”	”		$\text{Err}$	”	”	”	”	”	”	20%	11%	7%	9%	14%	3%

Fig. 9.

Size of the nested CV intervals relative to the size of the naïve CV intervals in the high-dimensional sparse logistic regression experiment from Section 5.1.2.

5.2. Regression

We next consider an OLS example. We take $X\in {ℝ}^{n\times p}$ with $p=20$ comprised of i.i.d. $\mathcal{N}\left(0,1\right)$. Further, we generate a response from the standard linear model: $Y=X\theta +ϵ$, where $ϵ$ is likewise i.i.d. $\mathcal{N}\left(0,1\right)$. We use OLS to estimate $\theta$. Note that by Lemma 1, the choice of $\theta$ does not affect the coverage rate of CV. The same argument shows that the choice of $\theta$ will not affect the coverage rate of nested CV, so we can take $\theta$ to be 0 without loss of generality. Similarly, both CV and NCV are unchanged when $X$ is transformed by an full-rank linear operator, so the results in this section would remain unchanged for Gaussian features with any full-rank correlation structure. We report the coverage of nested CV in Figure 10. We find that this scheme works well and has good coverage for any n, overcovering somewhat for very small $n$. By contrast, naïve CV has poor coverage until n is 400. In Supplementary Appendix Figure F.3 we report on the width of the NCV intervals relative to their CV counterparts—the usual ratio is not that large for samples sizes of $n=100$ or greater. See Supplementary Appendix Section F.11 for an experiment in a high-dimensional regression setting.

Fig. 10.

Coverage of CV, data splitting, and nested CV in the OLS case.

6. Real data examples

Lastly, we evaluate the nested CV procedure on real data sets from the UCI repository (Dua and Graff, 2017). In each case, we repeatedly subsample a small number of observations, perform nested CV on the subsample, and then use the many remaining observations to determine the accuracy of the fitted model. We consider the following data sets:

Communities and crimes (CC).

This data set is comprised of measurements of 1994 communities in the US. We predict the crime rate of each community, a real number normalized to be between 0 and 1, based on 99 demographic features of the community.

Crop mapping (crp).

This data set is comprised of optical radar measurements of cropland in Manitoba in 2012. We filter the data set to contain two classes, corn and oats, and then do binary classification based on 174 features. Here, we add a small amount of label noise so that the best possible classifier has a misclassification rate of about 5%.

We again use sparse linear or logistic regression as our fitting algorithm. The results are reported in Table 3. We find that nested CV generally has coverage that is much closer to the nominal rate than naïve CV. Data splitting has poor coverage in this case due to the small sample size, but is significantly better with =100 samples than with $n=50$ samples.

Table 3.

Performance of cross-validation (CV),nested cross-validation (NCV), and data splitting (DS) with the real data sets. The nominal (target) error rate is 10%, i.e., 5% above and below. Other details as in Table 1.

Setting			Width		Point estimates				Miscoverage
									CV		NCV		DS
data	n	Target	NCV	DS	Err	CV	NCV	DS	Hi	Lo	Hi	Lo	Hi	Lo
CC	50	${\text{Err}}_{XY}$	2.82	1.77	0.029	0.031	0.029	.034	4%	20%	1%	13%	1%	33%
”	”	$\text{Err}$	”	”	”	”	”	”	2%	22%	0%	12%	1%	37%
CC	100	${\text{Err}}_{XY}$	1.46	2.05	0.023	0.024	0.023	.025	4%	13%	2%	7%	1%	24%
”	”	$\text{Err}$	”	”	”	”	”	”	2%	12%	1%	4%	1%	24%
crp	50	${\text{Err}}_{XY}$	1.21	1.89	10.6%	10.7%	10.6%	10.8%	6%	8%	2%	6%	4%	31%
”	”	$\text{Err}$	”	”	”	”	”	”	7%	12%	3%	8%	2%	31%
crp	100	${\text{Err}}_{XY}$	1.52	2.00	9.5%	9.7%	9.5%	9.4%	6%	6%	4%	5%	4%	15%
”	”	$\text{Err}$	”	”	”	”	”	”	8%	9%	5%	7%	4%	15%

7. Discussion

Our investigation had two main main components. First, we discussed point estimates of prediction error via subsampling techniques. Our primary result is that common estimates of prediction error—cross-validation, bootstrap, data splitting, and covariance penalties—should be viewed as estimates of the average prediction error, averaged across other hypothetical data sets from the same distribution. The formal results here were all for the special case of the linear model using unregularized OLS for model-fitting, although we also saw similar behavior in simulation for logistic regression; see Figure 8. A further important question is how regularization affects this behavior. In an additional experiment, we find that $\text{Err}$ does track ${\text{Err}}_{XY}$, albeit weakly, when there is regularization; see Supplementary Appendix F.10. We look forward to future work explaining the behavior of cross-validation and other estimates of prediction error in these settings.

Second, we discussed inference for cross-validation, deriving an estimator for the MSE of the CV point estimate, nested CV. The nested CV scheme has consistently superior coverage compared to naïve cross-validation confidence intervals, which makes it an appealing choice for providing confidence intervals for prediction error. Nonetheless, we wish to be clear that nested CV is more computationally intensive than standard CV—we use about 1000 times more model fits per example because of the repeated splitting. For example, in the logistic regression example from Section 1.1, nested CV takes about 10 seconds on a personal computer.

A fundamental open question is to understand under what conditions the standard CV intervals will be badly behaved, making the nested CV computations necessary. Roughly speaking, we expect the standard CV intervals to perform better when $n/p$ is larger and when more regularization is used. In our experiments, we saw that even in the mundane linear model with $n/p=10$, the miscoverage rate of standard CV was about 50% larger than the nominal rate. As n increases, however, the violation decreases. Moreover, the asymptotic results in Austern and Zhou (2020) and Bayle et al. (2020) show that the coverage is correct in the $p$ fixed, $n\to \infty$ limiting regime. The stability conditions therein may also be able to shed light on settings with small samples or high-dimensions. We look forward to future work in this direction.