On the bias in the AUC variance estimate

Abstract

The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a standard metric for quantifying and comparing binary classifiers. A popular approach to estimating the AUCs and the associated variabilities – the variance of the AUC or the full covariance matrix of multiple correlated AUCs – is the one proposed by DeLong et al [1], which is based on the Mann Whitney two-sample U-statistics. The bias of a variance estimator is an important factor in applications such as hypothesis testing and construction of confidence intervals – a negatively biased variance estimator may lead to incorrect conclusions, and a positive bias is conservative hence preferable. In this work, we show that the (co-)variance estimate in DeLong’s approach is always positively biased. More specifically, the difference matrix between the expectation of the estimated covariance and the true covariance is a positive semi-definite matrix. This bias is non-negligible when the sample size is small, and quickly diminishes as the sample size increases. Our method relies on constructing, from the AUC kernel, a random variable whose (co-)variance matrix coincides with the bias, thereby establishing the claim. We also discuss alternative approaches to AUC variance estimation that may potentially reduce the bias.

1. Introduction

The area under the receiver operating characteristic (ROC) curve (AUC) is widely used to quantify binary classification performance [2, 3, 4]. Unlike other performance measures, such as accuracy or precision, the AUC is independent of the decision threshold and the class prevalence. These properties make the AUC a more meaningful metric in cost sensitive situations or with imbalanced data, which are common in machine learning and medicine [5, 6]. In addition to being a performance metric, the AUC can also be used as a training objective for learning to improve classification performance [7].

Given a fixed (trained) classifier, a set of classifier scores and the corresponding class labels (Fig. 1), a popular way to estimate the AUC and the associated statistical variability is to apply DeLong’s nonparametric approach [1], which is equivalent to the Mann Whitney (MW) U-statistics [8]. When comparing the performance of multiple classifiers using the same test samples, DeLong’s approach can estimate the covariance matrix of the multiple AUCs that properly takes into account the correlated nature of the classifier scores.

Fig. 1.

(a) A binary classifier $ℱ$ takes input class-1 features $U_i\in R^d$ and generates rating scores $X_i\in R,i=1,\dots ,n_1$; it then takes input class-2 features $V_j\in R^d$ and generates class-2 scores $Y_j\in R,j=1,\cdots ,n_2$. (b) Comparing multiple classifiers ${ℱ}^{\left(q\right)},q=1,\cdots ,Q$, using the same $U_i,V_j$ leads to correlated scores $X_i^{\left(q\right)}$ for class-1 and $Y_j^{\left(q\right)}$ for class-2.

A main application of a variance estimate is to construct confidence intervals (CIs) of a statistic of interest. For such tasks, a less biased estimator can have more favorable properties [9, 10]. Furthermore, a negatively biased variance estimator leads to CIs with low coverage probabilities which is undesirable; a positively biased variance estimator is conservative and more preferable [9].

In this work, we show that the covariance estimate in DeLong’s approach is always positively biased. That is, the difference matrix between the expectation of DeLong’s covariance estimate and the true covariance is a positive semi-definite matrix. For the special case of a single AUC variance estimate, the positive bias in DeLong’s approach has been empirically observed [11]. The conservative (larger) standard error estimation also leads to conservative confidence intervals (regions) and sample size estimation, which is more preferable than the contrary for practical applications.

The rest of the paper is organized as follows. We discuss our problem setup and the related work in Sec 2. In Sec 3 we first derive the bias of DeLong’s variance estimate of a single AUC; this bias is then shown to be non-negative by constructing a random variable, from the AUC kernel, whose variance coincides with the bias. This construction is then extended to the covariance estimate of multiple correlated AUCs. We conclude the paper in Sec 5. Some calculation details are deferred to the appendix.

2. Related work

2.1. Binary AUC and its estimation

As shown in Fig. 1, we denote by $X,Y\in R$ the classifier scores for class 1 and class 2 features $U,V$, respectively. Without loss of generality, we assume $X~F_1$, and $Y~F_2$, i.e., $X\left(Y\right)$ follows the cumulative distribution function $F_1\left(F_2\right)$, and that $X$ and $Y$ are independent. The area under the empirical ROC curve (AUC) can be characterized as [12, 13]

$$\theta =\text{Pr}\left(X>Y\right)+\frac{1}{2}\text{Pr}\left(X=Y\right).$$ (1)

Given two sets of i.i.d. samples $\left\{X_i\right\},\left\{Y_j\right\},i=1,\cdots ,n_1,j=1,\cdots ,n_2$, a popular nonparametric AUC estimate is the Mann-Whitney two-sample U-statistic:

$$\hat{\theta }=\frac{1}{n_1{n}_2}\sum _{i=1}^{n_1}\sum _{j=1}^{n_2}\varphi \left(X_i,Y_j\right),$$ (2)

where the kernel $\varphi (a,b)$ is given by

$$\varphi \left(a,b\right)=\{\begin{array}{cc}1& a>b\\ 1/2& a=b\\ 0& a<b\end{array}$$ (3)

It is clear that $\text{E}\left[\hat{\theta }\right]=\text{Pr}\{X>Y\}+1/2\text{Pr}\{X=Y\}=\theta ;\hat{\theta }\left(2\right)$ is an unbiased estimate of the AUC (1).

The variance of the AUC estimate (2) can be expressed using product moments of the kernels $\varphi (\cdot ,\cdot )$:

$$\text{Var}\left[\hat{\theta }\right]=\frac{\left(n_2-1\right){\xi }_{10}+\left(n_1-1\right){\xi }_{01}+{\xi }_{11}}{n_1{n}_2}$$ (4)

where ${\xi }_{10},{\xi }_{01}$, and ${\xi }_{11}$ are second central moments of the kernels $\varphi (\cdot ,\cdot )$:

$${\xi }_{10}=\text{E}\left[\varphi \left(X_i,Y_j\right)\varphi \left(X_i,Y_k\right)\right]-{\theta }^2,j\ne k$$ (5a)

$${\xi }_{01}=\text{E}\left[\varphi \left(X_j,Y_i\right)\varphi \left(X_k,Y_i\right)\right]-{\theta }^2,j\ne k$$ (5b)

$${\xi }_{11}=\text{E}\left[\varphi \left(X_i,Y_j\right)\varphi \left(X_i,Y_j\right)\right]-{\theta }^2$$ (5c)

The derivation of (4) can be found in textbooks, see e.g., [14, chapter 6.3] and [15, chapter 3.3].

2.2. The variance estimate of a single AUC

The variance expression $\text{Var}\left[\hat{\theta }\right]$ (4) is given in terms of the population quantities ${\xi }_{10},{\xi }_{01},{\xi }_{11}$. In practice, these population quantities need to be estimated from sample observations. DeLong’s approach [1] employs the so-called structural components [16], given by

$${\varphi }_{10,i\cdot }\triangleq \frac{1}{n_2}\sum _j^{n_2}\varphi \left(X_i,Y_j\right),i=1,\cdots ,n_1$$ (6a)

$${\varphi }_{01,\cdot j}\triangleq \frac{1}{n_1}\sum _i^{n_1}\varphi \left(X_i,Y_j\right),j=1,\cdots ,n_2$$ (6b)

Obviously, $\text{E}\left[{\varphi }_{10,i·}\right]=\text{E}\left[{\varphi }_{01,·j}\right]=\theta ,\forall i,j$. Let ${\hat{s}}_{10}$ and ${\hat{s}}_{01}$ be the sample variances of the structural components (6):

$${\widehat{s}}_{10}=\frac{1}{n_1-1}\sum _i^{n_1}{\left({\varphi }_{10,i\cdot }-\widehat{\theta }\right)}^2$$ (7a)

$${\widehat{s}}_{01}=\frac{1}{n_2-1}\sum _j^{n_2}{\left({\varphi }_{01,\cdot j}-\widehat{\theta }\right)}^2,$$ (7b)

then a consistent estimate of the AUC variance (4) is [1]

$$\widehat{\text{Var}}\left[\widehat{\theta }\right]=\frac{{\hat{s}}_{10}}{n_1}+\frac{{\hat{s}}_{01}}{n_2}$$ (8)

2.3. Multiple correlated AUCs

The previous discussion on a single AUC can be modified to accommodate multiple correlated AUCs. A relevant scenario (Fig. 1(b)) is when we compare multiple classifiers using the same set of test data. In this case, the classifier scores are correlated; and this correlation must be properly accounted for in order to make correct inference about the observed difference in the classifier performance [17, 18].

Assume there are $Q$ competing classifiers ${ℱ}^{\left(q\right)},q=1,\cdots ,Q$, and $X_i^{\left(q\right)}{Y}_j^{\left(q\right)},i=1,\cdots ,n_1,j=1,\cdots ,n_2$ be the $q$-th classifier scores for class 1 and class 2 features, respectively. The classifiers’ individual AUCs (${\theta }^{\left(q\right)},{\hat{\theta }}^{\left(q\right)}$) and the AUC variance $\left(\text{Var}\left[{\hat{\theta }}^{\left(q\right)}\right],\widehat{\text{Var}}\left[{\hat{\theta }}^{\left(q\right)}\right]\right)$ can be obtained from (1), (2), (4), and (12) by inserting the proper classifier index to the classifier scores. Furthermore, the correlation between the AUC estimates for the $p$-th and $q$-th classifier can be shown to be [1]

$$\text{Cov}\left[{\hat{\theta }}^{\left(p\right)},{\hat{\theta }}^{\left(q\right)}\right]=\frac{\left(n_2-1\right){\xi }_{10}^{p,q}+\left(n_1-1\right){\xi }_{01}^{p,q}+{\xi }_{11}^{p,q}}{n_1{n}_2}$$ (9)

where ${\xi }_{10}^{p,q},{\xi }_{10}^{p,q}{\xi }_{11}^{p,q}$, parallel to (5), are the kernel’s cross (central) moments:

$${\xi }_{10}^{p,q}=\text{E}\left[\varphi \left(X_i^{\left(p\right)},Y_j^{\left(p\right)}\right)\varphi \left(X_i^{\left(q\right)},Y_k^{\left(q\right)}\right)\right]-{\theta }^{\left(p\right)}{\theta }^{\left(q\right)},j\ne k$$ (10a)

$${\xi }_{01}^{p,q}=\text{E}\left[\varphi \left(X_j^{\left(p\right)},Y_i^{\left(p\right)}\right)\varphi \left(X_k^{\left(q\right)},Y_i^{\left(q\right)}\right)\right]-{\theta }^{\left(p\right)}{\theta }^{\left(q\right)},j\ne k$$ (10b)

$${\xi }_{11}^{p,q}=\text{E}\left[\varphi \left(X_i^{\left(p\right)},Y_j^{\left(p\right)}\right)\varphi \left(X_i^{\left(q\right)},Y_j^{\left(q\right)}\right)\right]-{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}$$ (10c)

Similar to the single AUC case, the covariance expression (9) involves population quantities that need to be estimated from sample observations. From the theory of U-statistics, a consistent covariance estimate is

$$\widehat{\text{Cov}}\left[{\hat{\theta }}^{\left(p\right)},{\hat{\theta }}^{\left(q\right)}\right]=\frac{{\hat{s}}_{10}^{p,q}}{n_1}+\frac{{\hat{s}}_{01}^{p,q}}{n_2}$$ (11)

where, as in (7), ${\hat{s}}_{10}^{p,q}$ and ${\hat{s}}_{01}^{p,q}$ are the sample covariance of the structural components ${\varphi }_{10,i\cdot }^{\left(q\right)},{\varphi }_{01,·j}^{\left(q\right)}$:

$${\hat{s}}_{10}^{p,q}=\frac{1}{n_1-1}\sum _i^{n_1} \left({\varphi }_{10,i\cdot }^{\left(p\right)}-{\hat{\theta }}^{\left(p\right)}\right)\left({\varphi }_{10,i\cdot }^{\left(q\right)}-{\hat{\theta }}^{\left(q\right)}\right)$$ (12a)

$${\hat{s}}_{01}^{p,q}=\frac{1}{n_2-1}\sum _j^{n_2} \left({\varphi }_{01,\cdot j}^{\left(p\right)}-{\hat{\theta }}^{\left(p\right)}\right)\left({\varphi }_{01,\cdot j}^{\left(q\right)}-{\hat{\theta }}^{\left(q\right)}\right)$$ (12b)

and ${\varphi }_{10,i·}^{\left(p\right)},{\varphi }_{01,\cdot j}^{\left(p\right)}{\varphi }_{10,i\cdot }^{\left(q\right)},{\varphi }_{01,\cdot j}^{\left(q\right)}$ are defined similar to (6) by inserting proper classifier indices.

3. Method

We first calculate the bias in the AUC variance estimates (8), (11). Then we consider the case of a single AUC, and show that the bias is always non-negative. We further extend our analysis to multiple correlated AUCs.

3.1. The bias in (8) and (11)

We consider the two terms in (8) separately, and define correspondingly the following short-hand notation:

$$A\triangleq \frac{\sum _{i=1}^{n_1} {\left({\varphi }_{10,i\cdot }-\hat{\theta }\right)}^2}{n_1\left(n_1-1\right)},B\triangleq \frac{\sum _{j=1}^{n_2} {\left({\varphi }_{01,\cdot j}-\hat{\theta }\right)}^2}{n_2\left(n_2-1\right)}$$

then (8) is equivalent to $\widehat{\text{Var}}\left[\hat{\theta }\right]=A+B$. We show in the appendix that

$$\text{E}\left[A\right]=\frac{\frac{\left(n_2-1\right)\left(n_1-1\right)}{n_1}{\xi }_{10}-\frac{n_1-1}{n_1}{\xi }_{01}+\frac{n_1-1}{n_1}{\xi }_{11}}{\left(n_1-1\right)n_2}$$ (13a)

$$\text{E}\left[B\right]=\frac{\frac{\left(n_1-1\right)\left(n_2-1\right)}{n_2}{\xi }_{01}-\frac{n_2-1}{n_2}{\xi }_{10}+\frac{n_2-1}{n_2}{\xi }_{11}}{n_1\left(n_2-1\right)}$$ (13b)

Then combining (13a) and (13b),

$$E\left[\widehat{\text{Var}}\left(\hat{\theta }\right)\right]=E\left[A\right]+E\left[B\right]\stackrel{\left(13\right)}{=}\frac{\frac{\left(n_2-1\right)}{n_1}{\xi }_{10}-\frac{1}{n_1}{\xi }_{01}+\frac{1}{n_1}{\xi }_{11}}{n_2}+\frac{\frac{\left(n_1-1\right)}{n_2}{\xi }_{01}-\frac{1}{n_2}{\xi }_{10}+\frac{1}{n_2}{\xi }_{11}}{n_1}=\frac{\left(n_2-1\right){\xi }_{10}-{\xi }_{10}+\left(n_1-1\right){\xi }_{10}-{\xi }_{10}+2{\xi }_{11}}{n_1{n}_2} \stackrel{\left(4\right)}{=}\text{Var}\left[\hat{\theta }\right]+\underset{\_}{\frac{{\xi }_{11}-{\xi }_{01}-{\xi }_{10}}{n_1{n}_2}}$$ (14)

where the underlined term is the bias. The numerator of the bias, ${\xi }_{11}-{\xi }_{01}-{\xi }_{10}$, is a property of the population distribution; the bias in (14) is of the order $𝒪\left(1/\left(n_1{n}_2\right)\right)$, which may be non-negligible for small sample sizes but diminishes quickly as the sample size increases.

By inserting proper classifier indices, a similar derivation as above leads to the following bias expression for the covariance

$$\text{E}\left\{\widehat{\text{Cov}}\left[{\hat{\theta }}^{\left(p\right)},{\hat{\theta }}^{\left(q\right)}\right]\right\}-\text{Cov}\left[{\hat{\theta }}^{\left(p\right)},{\hat{\theta }}^{\left(q\right)}\right]=\frac{{\xi }_{11}^{p,q}-{\xi }_{10}^{p,q}-{\xi }_{01}^{p,q}}{n_1{n}_2}$$ (15)

3.2. The bias in (8) is non-negative

We define the following kernel conditional expectations that will be used shortly.

$${\zeta }_{10}\left(X\right)=\text{E}\left[\varphi \left(X,Y\right)∣X\right],$$ (16a)

$${\zeta }_{01}\left(Y\right) =\text{E}\left[\varphi \right(X,Y)\mid Y]$$ (16b)

$${\zeta }_{11}\left(X,Y\right)=\text{E}\left[\varphi \left(X,Y\right)∣X,Y\right],$$ (16c)

Apparently ${\zeta }_{11}$ is just $\varphi (X,Y)$; this definition is simply for notational consistency. Note that all three random variables in (16) have mean $\theta :\text{E}\left[{\zeta }_{10}\left(X\right)\right]=\text{E}\left[{\zeta }_{01}\left(Y\right)\right]=\text{E}\left[{\zeta }_{11}(X,Y)\right]=\theta$. Further, it can be verified that the (co-)variance of these kernel conditional expectations can be related to the kernel moments (5), as shown in Table 1.

Table 1.

(Co-)variance of the kernel conditional expectations, where ${\xi }_{10},{\xi }_{01},{\xi }_{11}$ are the kernel (central) moments in (5).

	${\zeta }_{10}\left(X\right)$	${\zeta }_{01}\left(Y\right)$	${\zeta }_{11}\left(X,Y\right)$
${\zeta }_{10}\left(X\right)$	${\xi }_{10}$	0	${\xi }_{10}$
${\zeta }_{01}\left(Y\right)$	0	${\xi }_{01}$	${\xi }_{01}$
${\zeta }_{11}\left(X,Y\right)$	${\xi }_{10}$	${\xi }_{01}$	${\xi }_{11}$

As an example, we derive the first entry in Table 1. Starting from the definition of ${\xi }_{10}$ in (5a),

$${\xi }_{10}+{\theta }^2=\text{E}\left[\varphi \left(X,Y_1\right)\varphi \left(X,Y_2\right)\right] =\underset{y_1}{\int }\underset{y_2}{\int }\underset{x}{\int }\varphi \left(x,y_1\right)\varphi \left(x,y_2\right)d{F}_2\left(y_1\right)d{F}_2\left(y_2\right)d{F}_1\left(x\right) =\underset{x}{\int }d{F}_1\left(x\right)\underset{y_1}{\int }\varphi \left(x,y_1\right)d{F}_2\left(y_1\right)\underset{y_2}{\int }\varphi \left(x,y_2\right)d{F}_2\left(y_2\right) \stackrel{\left(16\text{a}\right)}{=}\underset{x}{\int }d{F}_1\left(x\right){\zeta }_{10}\left(x\right){\zeta }_{10}\left(x\right)=\text{E}\left[{\zeta }_{10}\left(X\right){\zeta }_{10}\left(X\right)\right]$$ (17)

As the mean of ${\zeta }_{10}\left(X\right)$ is $\theta$, (17) says that the central kernel moment ${\xi }_{10}$ is simply the variance of the conditional expectation ${\zeta }_{10}(\cdot )$. Similar derivations can verify the other entries in Table 1.

With Table 1 set up, we are ready to construct the random variable whose variance coincides with the bias in $\widehat{\text{Var}}\left[\hat{\theta }\right]$. Denote by $V_{11}$ the following quantity:

$$V_{11}\left(X,Y\right)=\left({\zeta }_{11}\left(X,Y\right)-\theta \right)-\left({\zeta }_{10}\left(X\right)-\theta \right)-\left({\zeta }_{01}\left(Y\right)-\theta \right)$$ (18)

Obviously, $\text{E}\left[V_{11}(X,Y)\right]=0$. Aided by Table 1, the variance of $V_{11}$ is calculated as

$$\text{E}\left[V_{11}^2\left(X,Y\right)\right]={\xi }_{11}+{\xi }_{10}+{\xi }_{01}-2{\xi }_{10}-2{\xi }_{01}={\xi }_{11}-{\xi }_{10}-{\xi }_{01}$$ (19)

Note that the right hand side of (19) coincides, up to a constant factor, with the bias in the variance estimate (14), establishing our claim that the bias is always non-negative.

3.3. Extension to covariance estimate of multiple AUCs

The previous result on a single AUC can be extended in a straightforward manner to multiple correlated AUCs. For this, we define $V_{11}$ as in (18) for each classifier $q$:

$$V_{11}\left(X^{\left(q\right)},Y^{\left(q\right)}\right)=\left({\zeta }_{11}\left(X^{\left(q\right)},Y^{\left(q\right)}\right)-{\theta }^{\left(q\right)}\right)-\left({\zeta }_{10}\left(X^{\left(q\right)}\right)-{\theta }^{\left(q\right)}\right)-\left({\zeta }_{01}\left(Y^{\left(q\right)}\right)-{\theta }^{\left(q\right)}\right)$$ (20)

where ${\zeta }_{10},{\zeta }_{01},{\zeta }_{11}$ are modified from (16) by inserting the proper classifier index. Similar to (17), we can relate these kernel conditional expectations to the kernel cross moments (10). The results are tabulated in Table 2.

Table 2.

Cross covariance of the kernel conditional expectations, where ${\xi }_{10},{\xi }_{01},{\xi }_{11}$ are the kernel (central) moments in (10). The only difference with Table 1 is the insertion of classifier indices $p,q$.

	${\zeta }_{10}\left(X^{\left(q\right)}\right)$	${\zeta }_{01}\left(Y^{\left(q\right)}\right)$	${\zeta }_{11}\left(X^{\left(q\right)},Y^{\left(q\right)}\right)$
${\zeta }_{10}\left(X^{\left(q\right)}\right)$	${\xi }_{10}^{p,q}$	0	${\xi }_{10}^{p,q}$
${\zeta }_{01}\left(Y^{\left(q\right)}\right)$	0	${\xi }_{01}^{p,q}$	${\xi }_{01}^{p,q}$
${\zeta }_{11}\left(X^{\left(q\right)},Y^{\left(q\right)}\right)$	${\xi }_{10}^{p,q}$	${\xi }_{01}^{p,q}$	${\xi }_{11}^{p,q}$

Using $V_{11}^{\left(q\right)}$ as a shorthand notation for $V_{11}\left(X^{\left(q\right)},Y^{\left(q\right)}\right)$ (20), we assemble all of them from the $Q$ classifiers to form $\Phi \in R^{Q\times 1}:$

$${\Phi }^t=\left[V_{11}^{\left(1\right)},\cdots ,V_{11}^{\left(Q\right)}\right]\in R^{1\times Q},$$

The covariance matrix of $\Phi$ is of the form $\text{E}\left[\Phi {\Phi }^t\right]=\left[\text{Cov}\left(V_{11}^{\left(p\right)},V_{11}^{\left(q\right)}\right)\right]\in R^{Q\times Q}$. Using Table 2, the entries of the covariance matrix are

$$\text{Cov}\left(V_{11}^{\left(p\right)},V_{11}^{\left(q\right)}\right)=\text{E}\left(V_{11}^{\left(p\right)}{V}_{11}^{\left(q\right)}\right) ={\xi }_{11}^{p,q}-{\xi }_{10}^{p,q}-{\xi }_{01}^{p,q}-{\xi }_{10}^{p,q}+{\xi }_{10}^{p,q}+0-{\xi }_{01}^{p,q}+0+{\xi }_{01}^{p,q} ={\xi }_{11}^{p,q}-{\xi }_{10}^{p,q}-{\xi }_{01}^{p,q}$$ (21)

Again, up to a constant, (21) coincides with the bias of the covariance estimate (15). As the covariance matrix (21) is always positive semi-definite, so is the bias (15).

3.4. Comments on the construction of V₁₁

The construction of $V_{11}$ (18) can be motivated by the H-decomposition originally developed by Hoeffding [19] for analyzing U-statistics. Analogous forms of the decomposition can be found in [20, 21], and also [22, chapter 2] discusses the H-decomposition for generalized (multi-sample) U-statistics. It turns out that, for the two-sample, degree-1 U-statistic such as the AUC $\hat{\theta }\left(2\right)$, this decomposition has a simple form.

Consider the following definition:

$$u_{10}\left(X_i\right) \triangleq \text{E}\left[\hat{\theta }\mid X_i\right]-\theta ,i=1,\cdots ,n_1$$ (22a)

$$u_{01}\left(Y_j\right) \triangleq \text{E}\left[\hat{\theta }\mid Y_j\right]-\theta ,j=1,\cdots ,n_2$$ (22b)

$$u_{11}\left(X_i,Y_j\right) \triangleq \text{E}\left[\hat{\theta }\mid X_i,Y_j\right]-u_{10}\left(X_i\right)-u_{01}\left(Y_j\right)-\theta$$ (22c)

From the expression $\hat{\theta }\left(2\right)$, we can further deduce that

$$u_{10}\left(X_i\right) =\frac{1}{n_1}\left[{\zeta }_{10}\left(X_i\right)-\theta \right],$$ (23a)

$$u_{01}\left(Y_j\right) =\frac{1}{n_2}\left[{\zeta }_{01}\left(Y_j\right)-\theta \right],$$ (23b)

$$u_{11}\left(X_i,Y_j\right) =\frac{1}{n_1{n}_2}\left[\left({\zeta }_{11}\left(X_i,Y_j\right)-\theta \right)\right.\left.-\left({\zeta }_{10}\left(X_i\right)-\theta \right)-\left({\zeta }_{01}\left(Y_j\right)-\theta \right)\right]$$ (23c)

where (23c) involves the expression $V_{11}$. Summing all equations in (23), and recall ${\zeta }_{11}\left(X_i,Y_j\right)\equiv \varphi \left(X_i,Y_j\right)$, we arrive at the following decomposition of $\hat{\theta }$ into the constituents (22):

$$\hat{\theta }=\theta +\sum _{i=1}^{n_1} u_{10}\left(X_i\right)+\sum _{j=1}^{n_2} u_{01}\left(Y_j\right)+\sum _{i,j} u_{11}\left(X_i,Y_j\right)$$ (24)

Unlike the original expression of $\hat{\theta }\left(2\right)$ where all terms in the summation are correlated, it can be shown that in (24) all random variables are zero-mean and uncorrelated due to their construction (22). The decomposition form (24) often provides more insights into the analysis of variance (ANOVA) of $\hat{\theta }$.

Our discussion here is intentionally brief as for our purpose the decomposition form (24) is not needed per se but provides the background for the origin of $V_{11}$. Interested readers should consult the publications we cited.

4. Numerical examples

We performed numerical studies to empirically verify the bias in DeLong’s (co-)variance estimate (8), (11). To obtain the ground truth (4), (9) for bias calculation, we assumed the classifier ratings $X^{\left(p\right)},Y^{\left(p\right)},p=\mathrm{1,2}$, were random variables (r.v.s) following either (a) uni-variate normal $𝒩\left(\mu ,{\sigma }^2\right)$ for a single classifier or (b) bivariate normal, $𝒩\left({\mu }_1,{\mu }_2;{\sigma }_1^2,{\sigma }_2^2,\rho \right),\rho$ the correlation coefficient, for a pair of classifiers. Under these assumptions, the analytical results related to the ground truth calculation are provided in Appendix B.

Our numerical parameter setting was based on the application of DeLong’s approach to lesion detection using medical images. For such studies, it is often the case that the rating scores share a common (co-)variance. More specifically, we set the variance ${\sigma }^2=1$ for all classifier scores (both class-1 and class-2), and varied the classifier correlation coefficient $\rho$ and the class mean score differences ${\mu }_x-{\mu }_y$ to achieve certain target AUC values. We considered two settings: (i) a single classifier, and (ii) a pair of classifiers. To estimate the empirical mean, we ran $M=100,000$ Monte Carlo (MC) trials of DeLong’s approach. These MC trials were repeated for different sample sizes that are relevant in medical imaging.

4.1. A single classifier

The classifier ratings were generated as independent uni-variate r.v.s, $X~𝒩\left({\mu }_x,1.0\right)$ for class-1 and $Y~𝒩\left({\mu }_y,1.0\right)$ for class-2. We fixed ${\mu }_y=0$, and adjusted ${\mu }_x$ to {0.358, 0.954, 1.812} to achieve AUCs of $\theta =\{0.6,0.75,0.9\}$. The percent bias of AUC variance was defined as

$$\%\text{bias}=\frac{\frac{{\sum }_m{\widehat{Var}}_m\left[\widehat{\theta }\right]}{M}-\text{Var}\left[\widehat{\theta }\right]}{\text{Var}\left[\widehat{\theta }\right]}\times 100$$ (25)

where $\text{Var}\left[\hat{\theta }\right]$ is the ground truth AUC variance (4), ${\widehat{\text{Var}}}_m\left[\theta \right]$, for each MC trial $m$, was calculated as in (12).

In Fig. 2, we show the ground truth AUC variance ($\text{Var}\left[\hat{\theta }\right]$, the denominator of (25)) on the left for reference, and show on the right the bias estimated from MC trials and the analytically calculated bias using (14). As the AUC increases, the ground truth AUC variance $\text{Var}\left[\hat{\theta }\right]$ decreases while the percent bias increases. At each fixed AUC value, both the AUC variance itself and the bias (25) decrease as a function of the sample size. For the high AUC $\theta =0.9$, the bias decreases from ~ 12% for the small sample size to under 1% as the sample size increases.

Fig. 2.

(Left) AUC ground truth variance $\text{Var}\left[\hat{\theta }\right]$ (4) and (right) percent bias in AUC variance (25) using DeLong’s method for different sample sizes. The ratio of class-1 and class-2 samples was fixed at $n_1:n_2=1:2$ for all cases.

4.2. A pair of classifiers

The rating values of two correlated classifiers were simulated as bivariate normal r.v.s. $\left(X^{\left(1\right)},X^{\left(2\right)}\right)~𝒩\left({\mu }_x^{\left(1\right)},{\mu }_x^{\left(2\right)},1.0,1.0,\rho \right)$ for class-1 scores, and $\left(Y^{\left(1\right)},Y^{\left(2\right)}\right)~𝒩\left({\mu }_y,{\mu }_y;1.0,1.0,\rho \right)$ for class-2 scores. We fixed ${\mu }_y=0$ and ${\mu }_x^{\left(1\right)}$, and adjusted ${\mu }_x^{\left(2\right)}$ to achieve a fixed ${\theta }^{\left(1\right)}=0.75$ and varying ${\theta }^{\left(2\right)}=\{0.6,0.75,0.9\}$. Also, we varied the correlation coefficient $\rho =\{0.3,0.8\}$ to emulate both low and high correlation scenarios. A scalar measure of the covariance bias was defined as

$$B=\frac{\text{Det}\left(\frac{\sum _m {\widehat{\text{Cov}}}_m\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]}{M}-\text{Cov}\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]\right)}{\text{Det}\left(\text{Cov}\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]\right)}$$ (26a)

$$\text{\%bias}=\text{sign}\left(B\right)\sqrt{\left|B\right|}\times 100$$ (26b)

where Det in (26a) is the matrix determinant, $\text{Cov}\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]$ is the ground truth AUC covariance (9), and ${\widehat{\text{Cov}}}_m\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]$, for each $m$, was calculated using DeLong’s approach (11). The square root in (26b) was to place the bias for the bi-variate case on an equal footing as the uni-variate case (25). The definition (26) ensures that the sign of the bias reflects the definiteness of the bias in the covariance matrix estimate.

In Fig. 3, we show on the left the baseline covariance $\left(\sqrt{\text{Det}\left(\text{Cov}\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]\right)}\right)$ for reference, and show on the right the bias estimated from MC trials and from the analytic result (15). For each sample size, a higher AUC $\left({\theta }^{\left(2\right)}\right)$ or a higher correlation $\rho$ reduces the baseline covariance, while at the same time leads to a higher bias. As the sample size increases, both the covariance itself and the bias in the covariance decrease, ranging from ~ 10% for the small sample size to under 1% for the larger sample size.

Fig. 3.

(Left) Baseline covariance $\left(\sqrt{\text{Det}\left(\text{Cov}\left[{\hat{\theta }}^{\left(1\right)},{\hat{\theta }}^{\left(2\right)}\right]\right)}\right)$ and the percent bias of the AUC covariance (26) for different sample sizes. The ratio of class-1 and class-2 samples was fixed at $n_1:n_2=1:2$ for all cases.

5. Summary and discussions

In this work, we showed that DeLong’s nonparametric approach always produces positively biased variance (for a single AUC) and covariance (for multiple correlated AUCs) estimates of the AUC statistics. Our approach is based on constructing a random variable whose (co-)variance coincides with the bias. The bias is of the order $𝒪\left(1/\left(n_1{n}_2\right)\right)$, where $n_1,n_2$ are the number of class-1 and class-2 observations. For moderate to large sample sizes, the bias would be negligible.

DeLong’s approach is quite popular in biomedicine [18, 23, 24] and has been applied to, in addition to AUC, many AUC variants such as the partial AUC [25], the clustered AUC [26], and the multi-class AUC [27]. The positive bias is expected to hold in these applications as well.

It is known that DeLong’s approach is equivalent to the jackknife variance estimation applied to the two sample U-statistics [28]. Other resampling methods, including the Monte Carlo bootstrap [29] and the ideal bootstrap [30], have also been used to assess the statistical variability of the AUC. Another interesting approach is the one-shot method [31, 32] that produces an unbiased variance estimate of the AUC. These alternative methods may potentially reduce or eliminate the bias in the AUC variance estimate. As the main utility of a variance estimator is for hypotheses testing and CI construction, it is important to consider both the bias and the variance [9]. For the one-sample U-statistics, these properties have been investigated both analytically and numerically for different variance estimation methods [33]. It is a natural next step to perform a similar analysis for the two-sample U-statistics such as the AUC.

Highlights.

• We claim that the bias in DeLong’s AUC variance estimation is always non-negative.

• We claim that the bias matrix in DeLong’s covariance estimation of multiple correlated AUCs is semi-positive definite.

• We construct a random variable whose (co-)variance coincides with the bias, thereby establishing the claims.

Appendix A. Calculation of the bias in (8)

For convenience, we copy the following definitions and notations from the main text.

$$\hat{\theta } =\frac{1}{n_1{n}_2}\sum _{i=1}^{n_1} \sum _{j=1}^{n_2} \varphi \left(X_i,Y_j\right)$$ (A.1a)

$$A =\frac{1}{n_1\left(n_1-1\right)}\sum _{i=1}^{n_1} {\left({\varphi }_{10,i\cdot }-\hat{\theta }\right)}^2$$ (A.1b)

$$B =\frac{1}{n_2\left(n_2-1\right)}\sum _{j=1}^{n_2} {\left({\varphi }_{01,\cdot j}-\hat{\theta }\right)}^2$$ (A.1c)

where $\hat{\theta }$ is the AUC estimate, ${\varphi }_{10,i}$. and ${\varphi }_{01,\cdot j}$ are the structural components (same as (6)):

$${\varphi }_{10,i\cdot }=\frac{1}{n_2}\sum _{j=1}^{n_2} \varphi \left(X_i,Y_j\right)$$ (A.2a)

$${\varphi }_{01,\cdot j}=\frac{1}{n_1}\sum _{i=1}^{n_1} \varphi \left(X_i,Y_j\right)$$ (A.2b)

Note that from (A.1a) and (A.2), we also have

$$\hat{\theta }=\frac{1}{n_1}\sum _{i=1}^{n_1} {\varphi }_{10,i\cdot }=\frac{1}{n_2}\sum _{j=1}^{n_2} {\varphi }_{01,\cdot j}$$ (A.3)

Consider the following which differs only slightly from the summand in (A.1b) (instead of $\hat{\theta }$ we use $\theta$):

$${\left[{\varphi }_{10,i\cdot }-\theta \right]}^2 ={\left[\frac{1}{n_2}\sum _j {\varphi }_{ij}-\theta \right]}^2={\left[\frac{1}{n_2}\sum _j \left({\varphi }_{ij}-\theta \right)\right]}^2 =\frac{1}{n_2^2}\sum _{jk} \left({\varphi }_{ij}-\theta \right)\left({\varphi }_{ik}-\theta \right)$$ (A.4)

where we use the shorthand notation ${\varphi }_{ij}\triangleq \varphi \left(X_i,Y_j\right)$. With simple algebraic manipulations, it is easy to check that

$$\text{E}{\left[{\varphi }_{10,i\cdot }-\theta \right]}^2=\frac{n_2\left(n_2-1\right){\xi }_{10}+n_2{\xi }_{11}}{n_2^2}$$ (A.5)

where ${\xi }_{10}$ and ${\xi }_{11}$ are defined in (5). Now rewrite $A$ to incorporate (A.4):

$$A =\frac{\sum _i {\left[\left({\varphi }_{10,i\cdot }-\theta \right)-(\hat{\theta }-\theta )\right]}^2}{n_1\left(n_1-1\right)} \stackrel{\left(\text{ A. }3\right)}{=}\frac{\sum _i {\left({\varphi }_{10,i\cdot }-\theta \right)}^2-n_1[\hat{\theta }-\theta {]}^2}{n_1\left(n_1-1\right)}$$ (A.6)

Taking expectations on both sides of (A.6), we obtain

$$\text{E}\left[A\right] =\frac{\sum _i \text{E}\left[{\left({\varphi }_{10,i}-\theta \right)}^2\right]}{n_1\left(n_1-1\right)}-\frac{n_1\text{E}[\hat{\theta }-\theta {]}^2}{n_1\left(n_1-1\right)} \stackrel{(\text{A}.5)}{=}\frac{\left(n_2-1\right){\xi }_{10}+{\xi }_{11}}{\left(n_1-1\right)n_2}-\frac{E[\hat{\theta }-\theta {]}^2}{n_1-1} \stackrel{\left(4\right)}{=}\frac{\left(n_2-1\right){\xi }_{10}+{\xi }_{11}-\left(\frac{\left(n_2-1\right){\xi }_{10}+\left(n_1-1\right){\xi }_{01}+{\xi }_{11}}{n_1}\right)}{\left(n_1-1\right)n_2} =\frac{\frac{\left(n_2-1\right)\left(n_1-1\right)}{n_1}{\xi }_{10}-\frac{n_1-1}{n_1}{\xi }_{01}+\frac{n_1-1}{n_1}{\xi }_{11}}{\left(n_1-1\right)n_2}$$ (A.7)

which is (13a). A similar derivation can be carried out for $B$ (A.1c) to obtain $\text{E}\left[B\right]$ (13b). The details are omitted.

Appendix B. A bivariate normal example

Let two classifiers ${ℱ}^{\left(p\right)},{ℱ}^{\left(q\right)}$ generate correlated class-1 $\left(X^{\left(p\right)},X^{\left(q\right)}\right)$ and class-2 $\left(Y^{\left(p\right)},Y^{\left(q\right)}\right)$ ratings. We make the standard assumption that class-1 and class-2 ratings are independent; moreover, we assume

• $\left(X^{\left(p\right)},X^{\left(q\right)}\right)$ follow a bivariate normal distribution with mean $\left({\mu }_x^{\left(p\right)},{\mu }_x^{\left(q\right)}\right)$ and covariance

  $${\Sigma }_{x,\left(pq\right)}=\left[\begin{array}{cc}{\sigma }_{x,\left(p\right)}^2& {\sigma }_{x,\left(pq\right)}\\ {\sigma }_{x,\left(pq\right)}& {\sigma }_{x,\left(q\right)}^2\end{array}\right]$$
• $\left(Y^{\left(p\right)},Y^{\left(q\right)}\right)$ follow a bivariate normal distribution with mean $\left({\mu }_y^{\left(p\right)},{\mu }_y^{\left(q\right)}\right)$ and covariance

  $${\Sigma }_{y,\left(pq\right)}=\left[\begin{array}{cc}{\sigma }_{y,\left(p\right)}^2& {\sigma }_{y,\left(pq\right)}\\ {\sigma }_{y,\left(pq\right)}& {\sigma }_{y,\left(q\right)}^2\end{array}\right]$$

Denote by $G\left(x\right)$ the cumulative distribution function (CDF) of the standard normal distribution, and BvN $(h,k;\rho )$ the CDF of a zero-mean, unit-variance, bivariate normal with the correlation coefficient $\rho$:

$$\text{BvN}(h,k;\rho )=\frac{1}{2\pi \sqrt{1-{\rho }^2}}{\int }_{-\infty }^h \text{ d}s{\int }_{-\infty }^k \text{ d}t{e}^{-\frac{s^2-2\rho st+t^2}{2\left(1-{\rho }^2\right)}}$$ (B.1)

In particular, $G\left(x\right)=\text{BvN}(x,\infty ;0)$.

For continuous random variables (r.v.s), the definition of AUC (1) simplifies.

$${\theta }^{\left(p\right)}=\text{Pr}\left(X^{\left(p\right)}>Y^{\left(p\right)}\right),{\theta }^{\left(q\right)}=\text{Pr}\left(X^{\left(q\right)}>Y^{\left(q\right)}\right)$$ (B.2)

Given our assumptions, $X^{(\cdot )}-Y^{(\cdot )}$ is normally distributed with mean ${\mu }_x^{(\cdot )}-{\mu }_y^{(\cdot )}$ and variance ${\sigma }_{x,(\cdot )}^2+{\sigma }_{y,(\cdot )}^2$. Then (B.2) can be evaluated using the standard normal CDF $G(\cdot )$ as

$${\theta }^{(\cdot )}=G\left(\frac{{\mu }_x^{(\cdot )}-{\mu }_y^{(\cdot )}}{\sqrt{{\sigma }_{x,(\cdot )}^2+{\sigma }_{y,(\cdot )}^2}}\right),(\cdot )=\left\{\right(p),(q\left)\right\}$$ (B.3)

The continuous r.v. assumption also simplifies the kernel $\varphi$ definition (3), which becomes $\varphi (X,Y)=1$ if $X>Y$, and 0 otherwise. Then the kernel central moment (10) can be written as

$${\xi }_{10}^{p,q}+{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{Pr}\left(X^{\left(p\right)}>Y_1^{\left(p\right)},X^{\left(q\right)}>Y_2^{\left(q\right)}\right)$$ (B.4a)

$${\xi }_{01}^{p,q}+{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{Pr}\left(X_1^{\left(p\right)}>Y^{\left(p\right)},X_2^{\left(q\right)}>Y^{\left(q\right)}\right)$$ (B.4b)

$${\xi }_{11}^{p,q}+{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{Pr}\left(X^{\left(p\right)}>Y^{\left(p\right)},X^{\left(q\right)}>Y^{\left(q\right)}\right)$$ (B.4c)

Similar to (B.2), all quantities in (B.4) can be expressed using BvN $(h,k;\rho )$ (B.1). These kernel moments can be plugged into (4), (9), (14), and (15) to calculate analytically the (co-)variance of the AUC(s) and the bias in the (co-)variance estimate of DeLong’s approach.

Appendix B.1. A pair of classifiers

We outline the steps using (B.4a) as an example. Define

$$D_{10}^{\left(p\right)}=X^{\left(p\right)}-Y_1^{\left(p\right)},D_{10}^{\left(q\right)}=X^{\left(q\right)}-Y_2^{\left(q\right)}$$

then $\left(D_{10}^{\left(p\right)},D_{10}^{\left(q\right)}\right)$ is a bivariate normally distributed r.v. with mean $\left({\mu }_{x,\left(p\right)}-{\mu }_{y,\left(p\right)},{\mu }_{x,\left(q\right)}-{\mu }_{y,\left(q\right)}\right)$ and covariance

$${\Sigma }_{10,\left(pq\right)}=\left[\begin{array}{cc}{\sigma }_{x,\left(p\right)}^2+{\sigma }_{y,\left(p\right)}^2& {\sigma }_{x,\left(pq\right)}\\ {\sigma }_{x,\left(pq\right)}& {\sigma }_{x,\left(q\right)}^2+{\sigma }_{y,\left(q\right)}^2\end{array}\right]$$ (B.5)

The probability in (B.4a) can be evaluated as

$${\xi }_{10}^{p,q} +{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{BvN}\left(h^{\left(p\right)},h^{\left(q\right)};{\rho }_{10}^{\left(pq\right)}\right)$$ (B.6a)

$$h^{(\cdot )} =\frac{\left({\mu }_x^{(\cdot )}-{\mu }_y^{(\cdot )}\right)}{\sqrt{{\sigma }_{x,(\cdot )}^2+{\sigma }_{y,(\cdot )}^2},(\cdot )=\left\{\right(p),(q\left)\right\}}$$ (B.6b)

$${\rho }_{10}^{\left(pq\right)} =\frac{{\sigma }_{x,pq}}{\sqrt{\left({\sigma }_{x,\left(p\right)}^2+{\sigma }_{y,\left(p\right)}^2\right)\left({\sigma }_{x,\left(q\right)}^2+{\sigma }_{y,\left(q\right)}^2\right)}}$$ (B.6c)

A similar derivation can be applied to other equations in (B.4) to obtain

$${\xi }_{01}^{p,q}+{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{BvN}\left(h^{\left(p\right)},h^{\left(q\right)},{\rho }_{01}^{\left(pq\right)}\right)$$ (B.7a)

$${\rho }_{01}^{\left(pq\right)}=\frac{{\sigma }_{y,\left(pq\right)}}{\sqrt{\left({\sigma }_{x,\left(p\right)}^2+{\sigma }_{y,\left(p\right)}^2\right)\left({\sigma }_{x,\left(q\right)}^2+{\sigma }_{y,\left(q\right)}^2\right)}}$$ (B.7c)

$${\xi }_{11}^{p,q}+{\theta }^{\left(p\right)}{\theta }^{\left(q\right)}=\text{BvN}\left(h^{\left(p\right)},h^{\left(q\right)},{\rho }_{11}^{\left(pq\right)}\right)$$ (B.7b)

$${\rho }_{11}^{\left(pq\right)}={\rho }_{10}^{\left(pq\right)}+{\rho }_{01}^{\left(pq\right)}$$ (B.7d)

In this work, we calculate BvN $(h,k;\rho )$ in (B.6),(B.7) using Owen’s T function^{1 $T(h,a)$} [34] which is readily available in standard libraries such as Scipy.

Appendix B.2. A single classifier

Let $p=q$ in the previous section, the covariance between two classifiers $\left(p\right)$ and $\left(q\right)$ in e.g., (B.5), coincides with the variance. In other words, applying the substitution ${\sigma }_{x,\left(pq\right)}\leftrightarrow {\sigma }_{x,\left(p\right)}^2$ and ${\sigma }_{y,\left(pq\right)}\leftrightarrow {\sigma }_{y,\left(p\right)}^2$ in (B.6c), (B.7c), (B.7d), and omitting the classifier index, we have

$${\rho }_{10}=\frac{{\sigma }_x^2}{{\sigma }_x^2+{\sigma }_y^2},{\rho }_{01}=\frac{{\sigma }_y^2}{{\sigma }_x^2+{\sigma }_y^2},{\rho }_{11}={\rho }_{10}+{\rho }_{01}=1$$

Plugging them into (B.6a), (B.7a), (B.7b),

$${\xi }_{10}+{\theta }^2=\text{BvN}\left(h,h;{\rho }_{10}\right)\stackrel{\left(a\right)}{=}\theta -2T\left(h,\sqrt{\frac{1-{\rho }_{10}}{1+{\rho }_{10}}}\right)$$

$${\xi }_{01}+{\theta }^2=\text{BvN}\left(h,h;{\rho }_{01}\right)\stackrel{\left(a\right)}{=}\theta -2T\left(h,\sqrt{\frac{1-{\rho }_{01}}{1+{\rho }_{10}}}\right)$$

$${\xi }_{11}+{\theta }^2=\text{BvN}\left(h,h,1\right)\stackrel{\left(a\right)}{=}\theta -2T\left(h,0\right)\stackrel{\left(b\right)}{=}\theta$$

where (a) is due to a relationship between BvN and Owen’s T function [34], (b) is from the property that $T\left(h,0\right)=0,h=\left({\mu }_x-{\mu }_y\right)/\sqrt{{\sigma }_x^2+{\sigma }_y^2}$ from (B.6b), and $\theta =G\left(h\right)$ is the AUC (B.3).

Data availability

Data will be made available upon request.