Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curve

Abstract

Recent scientific and technological innovations have produced an abundance of potential markers that are being investigated for their use in disease screening and diagnosis. In evaluating these markers, it is often necessary to account for covariates associated with the marker of interest. Covariates may include subject characteristics, expertise of the test operator, test procedures or aspects of specimen handling. In this paper, we propose the covariate-adjusted receiver operating characteristic curve, a measure of covariate-adjusted classification accuracy. Nonparametric and semiparametric estimators are proposed, asymptotic distribution theory is provided and finite sample performance is investigated. For illustration we characterize the age-adjusted discriminatory accuracy of prostate-specific antigen as a biomarker for prostate cancer.

1. Introduction

Research into new markers for disease diagnosis, screening and prognosis has exploded in recent years. The primary question in each setting is of classification accuracy: how well does the marker distinguish between the two groups of individuals, the ‘cases’ and the ‘controls’?

The receiver operating characteristic or roc curve plays a central role in evaluating classification accuracy (Baker, 2003; Pepe et al., 2001). Let D denote the binary group variable such as disease status, and let Y_D and Y_D̄ denote case and control marker observations with survivor functions S_D(y) = pr(Y_D > y) and S_D̄ = pr(Y_D̄ > y). The roc curve is a plot of the true-positive fraction or sensitivity versus the false-positive fraction or 1-specificity for rules that classify an individual as ‘test-positive’ if Y > c, where the threshold c varies over all possible values. Equivalently, if the false-positive fraction is t, then the receiver operating characteristic curve is $\text{ROC}(t)=\text{pr}\{Y_D>S_{\overline{D}}^{-1}(t)\}=S_D\{S_{\overline{D}}^{-1}(t)\}$ (Pepe, 2003, p. 69).

There are often factors, such as patient or disease characteristics or features of the specimen handling or test procedure, which affect the marker and its associated classification accuracy. These effects may mean that the definition of testing positive on the basis of the marker should depend on covariates, or that the accuracy of the test is less than optimal in certain settings (Pepe, 2003, pp. 48–9). This paper proposes a covariate-adjusted summary of classification accuracy, the covariate-adjusted roc curve.

2. The covariate-adjusted roc curve

Let Y be a continuous marker and let Z be a covariate. The methods generalize naturally to multiple covariates. Let Z_D and Z_D̄ denote case and control covariate observations with cumulative distribution functions P_D and P_D̄. Denote by S_D(y | z) = pr(Y_D > y | Z_D = z) and S_D̄ (y | z) = pr(Y_D̄ > y | Z_D̄ = z) the continuous survivor functions for Y conditional on Z = z, where $S_D^{-1}(t|z)$ and $S_{\overline{D}}^{-1}(t|z)$ are the inverse conditional survivor functions, and let f_D(y | z) and f_D̄ (y | z) be the conditional densities.

The covariate-adjusted roc curve at a false-positive fraction of t is the overall true-positive fraction when thresholds for defining ‘test-positive’ are covariate-specific, chosen to ensure that the value of the false-positive fraction is t in each covariate-specific population. Mathematically, we write $\text{AROC}(t)=\text{pr}\{Y_D>S_{\overline{D}}^{-1}(t|Z_D)\}$, where we note that the threshold equal to the quantile $S_{\overline{D}}^{-1}(t|Z_D)$ yields a false-positive fraction of t in the covariate-specific population. Naturally, the marginal false-positive fraction is also equal to t.

The covariate-specific receiver operating characteristic curve, $\text{ROC}(t|z)=S_D\{S_{\overline{D}}^{-1}(t|z)|z\}$ is the roc curve for the marker in the population with covariate level Z = z. Interestingly, the adjusted roc can be written as a weighted average of covariate-specific curves,

$$\begin{array}{lll}\text{AROC}(t)\hfill & =\hfill & \int \text{pr}\{Y_D>S_{\overline{D}}^{-1}(t|Z_D)|Z_D=z\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}d{P}_D(z)\hfill \\ \hfill & =\hfill & \int \text{ROC}(t|z)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}d{P}_D(z).\hfill \end{array}$$ (1)

Equivalently, aroc(t) = E{roc(t | Z_D)}, where the expectation is taken with respect to Z_D. Equation (1) shows that when Z affects marker observations but not discriminatory accuracy as defined by the roc curve, the adjusted roc is the common covariate-specific roc curve; it is perhaps best motivated in this setting. On the other hand, when Z affects discrimination, the adjusted roc reports a weighted average of covariate-specific true-positive fractions, holding the covariate-specific false-positive fractions constant. While estimation of covariate-specific roc curves is typically of primary interest in this context, the adjusted roc curve is a summary of covariate-adjusted accuracy that is useful for comparing markers and in small studies where covariate-specific roc curves cannot be estimated with precision.

Another interpretation follows from noting that aroc(t) = pr{S_D̄ (Y_D | Z_D) ⩽ t}, the cumulative distribution function of S_D̄ (Y_D | Z_D), where S_D̄ (Y_D | Z_D) is the placement of a case observation relative to a reference distribution for controls with the same covariate value. Contrast this with the unadjusted or pooled roc curve which has been shown to be equal to the cumulative distribution function of a case observation standardized relative to the general control distribution (Pepe & Cai, 2004), roc(t) = pr{S_D̄ (Y_D) ⩽ t}.

Expression (1) has some attractive mathematical properties, including invariance with respect to monotone increasing transformations of Y and/or Z. It is also unaffected by control covariate-dependent sampling. This follows because such designs sample controls randomly conditional on Z, and cases randomly marginally. Janes & Pepe (2008) argue for its use in studies that employ frequency matching, where controls are sampled to have the same distribution of Z as cases. This is one type of covariate-dependent sampling.

3. Estimation

3.1. Estimators

We propose two estimators for $\text{AROC}(t)=\text{pr}\{Y_D>S_{\overline{D}}^{-1}(t|Z_D)\}$ using n_D and n_D̄ case and control observations. In both instances, the outside probability is estimated empirically. With the nonparametric estimator, valid for a discrete covariate Z ∈ {1, . . . , K}, we estimate the quantiles $S_{\overline{D}}^{-1}(t|z)$ empirically in each covariate stratum. With the semiparametric estimator, the quantiles are estimated based on a model for Y_D̄ as a function of Z_D̄. We lay out the general framework for the adjusted roc estimator, of which the nonparametric estimator is a special case.

We assume the quantile model Y_D̄ = f (Z_D̄, ∊; θ), where ∊ is random error and θ are parameters. With the semiparametric adjusted roc estimator, this model may be parametric, such as a normal linear model, or semiparametric (Heagerty & Pepe, 1999). Let $q(t;z,\theta )=S_{\overline{D}}^{-1}(t|z;\theta )$ be the function which extracts the 1 − t quantile from the set of control quantiles with covariate value Z = z. We write

$${\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)=n_D^{-1}\sum _{i=1}^{n_D}I\{Y_{Di}>q(t;Z_{D_i},\widehat{\theta })\}.$$

With the nonparametric estimator, $\theta ={\{S_{\overline{D}}^{-1}\left(t|z=1\right),\dots ,S_{\overline{D}}^{-1}(t|z=K)\}}^{\text{T}}$ are the quantiles themselves, ${\widehat{S}}_{\overline{D}}(y|z)=n_{\overline{D}}{\left(z\right)}^{-1}{\sum }_{i=1}^{n_{\overline{D}}(z)}I\{Y_{{\overline{D}}_i}>y,Z_{{\overline{D}}_i}=z\}$, and $q\left(t;z,\widehat{\theta }\right)={\widehat{S}}_{\overline{D}}^{-1}(t|z)={\text{inf}}_{s\in [0,1]}\{{\widehat{S}}_{\overline{D}}(s|z)⩾t\}$, all estimated using n_D̄(z) observations in each covariate stratum. This estimator depends only on the ranks of the data, and thus is invariant with respect to monotone transformations.

3.2. Asymptotic distribution theory

We make the following assumptions in establishing asymptotic distribution theory. Recall that the distribution of Y_D is not a function of θ.

Assumption 1. We assume that observations are randomly sampled conditional on D, that n_D + n_D̄ → ∞ and that n_D/n_D̄ → λ ∈ (0, 1).

Assumption 2. We assume that $n_{\overline{D}}^{1/2}\left(\widehat{\theta }-\theta \right)$ converges in distribution to a normal zero-mean random variable with covariance matrix Σ_θ as n_D̄ → ∞.

Assumption 3. We assume that aroc_θ(t) is differentiable, and hence continuous, in θ.

Assumption 4. We assume that lim_{nD̄ → ∞}pr{aroc_θ̂(t) ∉ {0, 1}} = 1, where aroc_θ̂(t)= pr{Y_D > q(t; Z_D, θ̂) | θ̂} is the adjusted roc based on estimated quantiles.

Assumption 5. We assume that t ∉ {0, 1}.

In relation to Assumption 1, covariate-dependent sampling can also be accommodated; see § 3.4. A wide variety of quantile models satisfy Assumption 2, including parametric models (Cole, 1990; Cole & Green, 1992; Pepe, 2003, p. 140), semiparametric models (Heagerty & Pepe, 1999; Zheng & Heagerty, 2004), empirical methods, as proven in the Appendix, and any θ̂ based on unbiased estimating equations satisfying standard regularity conditions. Assumption 3 is also valid for a diversity of quantile and roc models, such as the location-scale quantile model (Heagerty & Pepe, 1999) with bounded (∂/∂t) roc(t | z) and E(Z_D) < ∞; see an unpublished University of Washington technical report by H. Janes and M. S. Pepe, available at http://www.bepress.com/uwbiostat/paper283. Assumption 4 is violated if the support for Y_D is entirely above or below the estimated quantile of interest. This will not occur as long as the support for Y_D includes the support for Y_D̄ or if the support for Y_D is unbounded, as under the normal distribution. We also require that t ∉ {0, 1}, but by definition aroc(0) = 0 and aroc(1) = 1. Finally, imposing continuity of S_D(y | z) and S_D̄ (y | z) implies that roc(t | z) and aroc(t) are continuous in t. The proof of the following theorem is given in the Appendix.

Theorem 1. Under Assumptions 1–5, $n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)-{\text{AROC}}_{\theta }(t)\}$ converges in distribution to a normal zero-mean random variable with variance V(t) as n_D, n_D̄ → ∞, where

$$V(t)={\text{AROC}}_{\theta }(t)\{1-{\text{AROC}}_{\theta }(t)\}+\lambda \frac{\partial }{\partial \theta }{\text{AROC}}_{\theta }(t)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathrm{\Sigma }}_{\theta }\frac{\partial }{\partial {\theta }^{\text{T}}}{\text{AROC}}_{\theta }(t).$$ (2)

The form of V(t) is intuitively plausible. The second component comes from estimating the Z-specific quantiles, while the first is a binomial variance associated with estimating the true-positive fraction, given the quantiles.

For the nonparametric estimator, Assumptions 2 and 3 are satisfied when the following assumption holds.

Assumption 6. We assume that f_D̄(y | z) is continuous and positive in a neighbourhood of $S_{\overline{D}}^{-1}(t|z)$ for all z.

Under Assumptions 1–6, V(t) reduces to

$$V(t)={\text{AROC}}_{\theta }(t)\{1-{\text{AROC}}_{\theta }(t)\}+\lambda \left[\sum _{z=1}^K\frac{p_D^2(z)}{p_{\overline{D}}(z)}\frac{f_D{\{S_{\overline{D}}^{-1}(t|z)|z\}}^2}{f_{\overline{D}}{\{S_{\overline{D}}^{-1}(t|z)|z\}}^2}\right]t(1-t),$$ (3)

where p_D and p_D̄ are the probability mass functions for Z_D and Z_D̄; see the Appendix for a proof.

3.3. Consistent variance estimation

Our first variance estimator can be used to estimate the variance of the semiparametric adjusted roc estimator, namely expression (2) multiplied by $n_D^{-1}$. The semiparametric estimator is consistent by Theorem 1. We assume that a consistent estimator of Σ_θ exists; for example, if θ̂ is based on a set of unbiased estimating equations, a sandwich-type variance estimator can be used. The jth component of (∂/∂θ) aroc_θ(t) is estimated by (arôc_θ̂+hj(n) − arôc_θ̂−hj(n))/{2h(n)}, where h(n) is $o\left(n_D^{-1/3}\right)$, and θ̂ + h_j(n), respectively θ̂ − h_j (n), denotes the vector θ̂ with h(n) added to, respectively subtracted from, the jth component only. In the Appendix, the composite variance estimator is shown to be consistent under Assumptions 1–5 and the following Assumption 7.

Assumption 7. We assume that

$$\underset{n_{\overline{D}}\to \infty }{\text{lim}}\text{pr}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }+h_j(n)}\notin \{0,1\}\}=\underset{n_{\overline{D}}\to \infty }{\text{lim}}\text{pr}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }-h_j(n)}\notin \{0,1\}\}=1,$$

for all j.

With small sample sizes, the estimate of (∂/∂θ_j) aroc_θ(t) may be sensitive to the choice of bandwidth, h(n). We have used h(n) = 0.04 in applications and simulations; this value ensured that (aroc_θ+hj(n) − aroc_θ−hj(n))/{2h(n)} ≃ (∂/∂θ_j) aroc_θ(t) in one example and has worked well in practice. We leave exploration of the optimal choice of h(n) for future research.

Our second variance estimator can be used to estimate the variance of the nonparametric estimator, namely expression (3) multiplied by $n_D^{-1}$. Here, aroc_θ(t) is estimated using the non-parametric estimator, p_D and p_D̄ by binomial proportions, $S_{\overline{D}}^{-1}(t|z)$ empirically, and f_D(y | z) and f_D̄ (y | z) with uniformly consistent kernel density estimators (Silverman, 1986, § 3.7). In the Appendix we prove that the composite function is consistent under Assumptions 1–6 and the following Assumption 8.

Assumption 8. We assume that f_D(y | z) and f_D̄ (y | z) are continuous density functions for all z.

Bootstrap variance estimation is a simple alternative which accommodates clustered sampling as well as performing well in practice and in simulations; see § 4.

3.4. Sampling based on covariates

In many situations, sampling may depend on both D and Z. Two examples of this are frequency matching and sampling subjects in a specified range of Z. Our asymptotic results hold under such designs, but all population distributions should be replaced with sampling distributions.

3.5. Estimation using roc regression

The aroc can also be estimated using the roc regression method of Pepe (2000) and Alonzo & Pepe (2002), which requires estimation of covariate-specific control quantiles and specifying and fitting a model for the roc curve, typically as a function of covariates. A model for the adjusted roc curve is obtained by including Z in the quantile calculations, while omitting Z from the roc model. A classic example is the binormal model, aroc(t) = Φ{α + βΦ⁻¹(t)}, where Φ is the standard normal cumulative distribution function. This approach results in a smooth parametric estimate of the adjusted roc, but the marker distributions remain unspecified. The semiparametric roc regression estimator of Cai & Pepe (2002) reduces to our semiparametric estimator.

4. Small sample performance

We evaluate the finite sample properties of our estimators using simulations, starting with the nonparametric estimator and its variance which can be used for discrete Z. We assume that there exists a monotone increasing transformation under which Y is normally distributed conditional on a binary covariate, Z, in both cases and controls: Y_D̄ ∼ N (0, 1) and Y_D ∼ N (0.9, 1) conditional on Z = 0; Y_D̄ ∼ N (0.2, 1) and Y_D ∼ N (0.9, 1) conditional on Z = 1; pr(Z_D̄ = 1) = 0.7 and pr(Z_D = 1) = 0.3. All of the assumptions laid out in § 3 are satisfied under this model. The values of aroc(t) at t = 0.05, 0.10, 0.20 and 0.50 are 0.21, 0.33, 0.50 and 0.80 and the two components of asymptotic variance are $n_D^{-1}$ (0.33, 0.44, 0.50, 0.32) and $n_{\overline{D}}^{-1}$(1.41, 1.40, 1.14, 0.40).

We simulated 5000 datasets, where n_D = n_D̄ varies between 100 and 1000; see Table 1. In terms of percentage bias, defined as 100[avg{arôc_θ̂(t)} − aroc_θ(t)]/aroc_θ(t), where avg{arôc_θ̂(t)} is the average estimate, the estimator performs very well, except for some modest bias when both t and n_D = n_D̄ are small. The percentage bias in the nonparametric variance estimator, based on rectangular kernel density estimates, is $100\left[n_D^{-1}\text{med}\{\widehat{V}(t)\}-\text{v}\widehat{\text{a}}\text{r}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)\}\right]/\text{v}\widehat{\text{a}}\text{r}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)\}$, where the median is calculated because of the skewed distribution of the variance estimates, and vâr{arôc_θ̂(t)} is the sample variance of the estimates. The variance estimator tends to underestimate the true variance. There is substantial bias when t is small, mostly due to estimation of the second component of variance, but this disappears for larger t. The percentage difference between the asymptotic and sample variances, $100\left[n_D^{-1}V(t)-\text{v}\widehat{\text{a}}\text{r}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)\}\right]/\text{v}\widehat{\text{a}}\text{r}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)\}$, tends to be small, with differences only when both t and n_D = n_D̄ are small. Coverage probabilities using nonparametric variance estimators are provided. Coverage based on logit transformations, which have been shown to improve coverage for the pooled roc when t is close to 0 or 1 (Pepe, 2003, p. 102), are also shown. Only logit-based coverage is shown when both t and n_D = n_D̄ are small, as adjusted roc estimates are close to zero. We find that coverage can be low for small t but is very good for moderate t.

Table 1.

Small sample performance of the nonparametric estimator of the adjusted roc based on 5000 simulations. The sample size, n_D = n_D̄, varies between 100 and 1000. The nonparametric variance estimator uses rectangular kernel density estimates. Results are shown at four false-positive fractions of interest. Only coverage based on logit-transformations is shown for small t and n_D = n_D̄, since arôc_θ̂(t) is frequently close to zero. Bootstrap variance estimates based on 100 bootstrap samples and associated coverage are also shown

nD	% Bias	% Diff	% Bias var	Cov	Logit cov	% Bias Boot var	Cov	Logit cov
t = 0.05
100	6.15	21.34	−30.65	–	89.36	−25.25	–	87.32
200	10.05	7.83	−29.67	–	88.14	−6.30	–	91.68
500	2.40	6.88	−20.25	–	91.58	4.36	–	94.86
1000	1.71	2.28	−16.24	91.90	92.14	2.92	94.40	94.18
t = 0.10
100	7.83	15.48	−19.75	–	91.22	−7.32	–	93.38
200	4.37	7.06	−17.53	–	91.88	0.79	–	94.82
500	0.75	3.88	−12.94	–	93.22	2.27	–	94.60
1000	0.94	1.39	−10.53	92.86	92.18	1.68	94.10	94.40
t = 0.20
100	2.21	12.70	−4.42	91.92	95.36	1.04	93.88	97.08
200	1.45	6.64	−6.00	92.76	94.38	3.08	94.26	95.68
500	0.24	3.51	−3.96	93.74	94.42	2.96	94.44	95.08
1000	0.33	4.26	−1.12	94.14	94.42	5.04	94.16	94.56
t = 0.50
100	0.02	2.05	−3.67	92.16	96.98	8.41	95.00	97.26
200	−0.11	−0.10	−0.47	93.42	95.86	4.38	94.18	95.98
500	−0.05	−1.23	1.78	94.78	95.98	0.71	94.64	95.42
1000	−0.01	2.48	6.37	95.36	95.88	3.76	94.78	95.20

% Bias, percentage bias in aroc_θ(t); % Diff, percentage difference between asymptotic and sample variances; % Bias var, percentage bias in the nonparametric variance estimator; Cov, coverage of 95% confidence intervals; Logit cov, coverage based on logit-transformations; Boot var, bootstrap variance estimates.

We also evaluate the performance of bootstrap variance estimates. Data are resampled 100 times conditional on D, and the sample variance of the adjusted roc estimates is calculated. The bootstrap variance estimator exhibits substantially less bias than the nonparametric estimator. Bootstrap coverage also tends to be better; coverage is good except when both t and n_D = n_D̄ are small.

Table 2 compares the nonparametric estimator of the adjusted roc with the semiparametric estimator based on a normal linear quantile model. The percentage difference in the estimates, defined as avg{arôc_θ̂;semi(t) − arôc_θ̂(t)}/aroc_θ(t), where arôc_θ̂(t) is the nonparametric estimator and arôc_θ̂;semi(t) is the semiparametric estimator, tends to be small. The estimated relative efficiency of the two estimators, vâr{arôc_θ̂;semi(t)}/vâr{arôc_θ̂(t)}, where the variances are estimated from the 5000 simulations, shows that the semiparametric estimator yields substantial gains in efficiency, with larger gains for smaller t and larger n_D = n_D̄.

Table 2.

Comparison of the nonparametric and semiparametric estimates of the adjusted roc based on 5000 simulations. The sample size, n_D = n_D̄, varies between 100 and 1000. The semiparametric estimator uses a normal linear quantile model. The percentage difference in the adjusted roc estimates and relative efficiency are shown at four false-positive fractions of interest

	t = 0.05		t = 0.10		t = 0.20		t = 0.50
nD	% Diff	re	% Diff	re	% Diff	re	% Diff	re
100	−0.20	0.61	−5.33	0.68	−2.24	0.76	−0.75	0.82
200	−6.87	0.54	−2.98	0.64	−1.25	0.72	−0.33	0.81
500	−1.90	0.51	−0.88	0.61	−0.47	0.71	−0.17	0.79
1000	−0.92	0.50	−0.52	0.61	−0.12	0.71	−0.04	0.82

% Diff, percentage difference in the adjusted roc estimates; re, relative efficiency.

The performances of the semiparametric estimator and its variance are explored under the double binormal model; see Lin & Jeon (2003) and the authors’ technical report. Under this model, there is a monotone increasing transformation under which (Y, Z) is bivariate normal in both cases and controls. We set the parameter values to E(Y_D̄) = E(Z_D̄) = 0, E(Y_D) = 0.7, E(Z_D) = 0.5, var(Y_D̄) = var(Y_D) = 1, var(Z_D̄) = var(Z_D) = 1.5², corr(Y_D, Z_D) = 0.6 and corr(Y_D̄, Z_D̄) = 0.2. This is an extension of the classic binormal model for the pooled roc curve (Swets, 1986; Hanley, 1988, 1996). The induced adjusted roc is a binormal roc curve, as shown in the authors’ technical report. All of the assumptions laid out in § 3 are satisfied under this model. We apply the semiparametric estimator using a normal linear quantile model; this is the true model for Y_D̄ given Z_D̄. The values of aroc(t) at t = 0.05, 0.10, 0.20 and 0.50 are 0.16, 0.25, 0.39 and 0.67. The two components of asymptotic variance are $n_D^{-1}$ (0.24, 0.37, 0.49, 0.36) and $n_{\overline{D}}^{-1}$(0.35, 0.53, 0.56, 0.23).

We simulated 5000 datasets, in which n_D = n_D̄ varies between 100 and 1000. The estimator performs very well, except for some modest bias for very small n_D = n_D̄ and t ; see Table 3. The semiparametric variance estimator exhibits moderate small sample bias for the smallest sample sizes; the variance is consistently overestimated. This is primarily due to bias in the second component of variance, which involves (∂/∂θ) aroc_θ(t). However, coverage is reasonable. The asymptotic and sample variances agree quite well, except for some minor differences with the smallest sample sizes. Bootstrap variance estimates are good alternatives: they tend to exhibit less bias and have excellent coverage.

Table 3.

Small sample performance of the semiparametric estimator of the adjusted roc based on 5000 simulations. The sample size, n_D = n_D̄, varies between 100 and 1000. The quantiles are estimated using a normal linear model, and the semiparametric variance estimator uses a bandwidth of h = 0.04. Results are shown at four false-positive fractions of interest. Only coverage based on logit transformations is shown for small t and n_D = n_D̄, since arôc_θ̂(t) is frequently close to zero. Bootstrap variance estimates based on 100 bootstrap samples and associated coverage are also shown

nD	% Bias	% Diff	% Bias var	Cov	Logit cov	% Bias Boot var	Cov	Logit cov
t = 0.05
100	14.35	−7.64	1.50	–	91.52	6.25	–	93.20
200	6.47	0.28	5.17	–	93.88	6.67	–	94.64
500	2.97	0.21	3.70	–	94.60	1.97	–	95.40
1000	0.85	1.64	1.95	94.32	95.08	2.20	94.42	95.30
t = 0.10
100	7.08	2.27	8.25	–	93.30	2.47	–	94.96
200	3.23	3.26	4.87	–	94.28	3.79	–	95.48
500	1.34	−2.22	−2.72	–	94.22	−3.09	–	94.82
1000	0.34	2.47	1.48	94.66	95.10	2.48	95.00	95.44
t = 0.20
100	2.46	5.14	9.02	92.54	94.02	−0.12	93.50	95.86
200	1.26	2.31	4.54	93.80	94.70	0.25	94.38	95.54
500	0.37	−0.55	−0.26	94.10	95.42	−1.71	94.30	94.72
1000	0.03	−0.33	0.79	94.48	94.72	−0.17	94.42	94.66
t = 0.50
100	−0.62	3.66	13.89	93.52	95.98	9.06	94.70	96.76
200	−0.44	1.08	9.75	94.58	95.44	4.63	94.92	95.68
500	−0.21	1.38	5.50	95.02	95.26	4.35	94.82	95.18
1000	−0.14	0.45	3.13	95.00	95.24	3.24	94.68	94.88

% Bias, percentage bias in aroc_θ(t); % Diff, percentage difference between asymptotic and sample variances; % Bias var, percentage bias in the nonparametric variance estimator; Cov, coverage of 95% confidence intervals; Logit cov, coverage based on logit-transformations; Boot var, bootstrap variance estimates.

5. Illustration

Data from the Physicians’ Health Study (Gann et al., 1995) are used for illustration. This was a randomized controlled study of aspirin and β-carotene among 22 071 U.S. male physicians of ages 40 to 84 years in 1982. A blood sample taken at enrolment was stored. For 429 men diagnosed with prostate cancer up to 12 years after enrolment, most before PSA, Prostate-Specific Antigen, was widely used for screening, and for 1287 controls not diagnosed with prostate cancer during 12 years of follow-up, the serum was assayed for PSA. Cases and controls were matched on age; for each case, three controls within one year of age were selected (Gann et al., 1995; Etzioni et al., 2004).

The goal of this sub-study is to assess the ability of PSA to discriminate between men who did and did not develop prostate cancer. The pooled roc curve in the matched data is not of practical interest (Janes & Pepe, 2008). It describes the ability of PSA to distinguish between cases and age-matched controls, an artificial control group. More importantly, this curve is attenuated by the matching on age. We use the adjusted roc to quantify the age-adjusted classification accuracy of PSA.

Age-specific roc curves for PSA, estimated using a binormal roc regression model (Alonzo & Pepe, 2002) with quantiles from a linear location-scale model (Heagerty & Pepe, 1999), reveal very little variation in discrimination due to age; see Fig. 1(a). Hence, the adjusted roc represents the common age-specific roc curve for PSA.

Fig. 1.

roc curves for PSA in the Physicians’ Health Study data. (a) Age-specific roc curves, 50 (dotted), 60 (dash-dotted), 70 (dashed), estimated using the roc regression model. (b) The age-adjusted roc curve, estimated using the semiparametric estimator (the dotted line) and the roc regression model (the solid line). The 95% confidence intervals, based on bootstrapped variance estimates, are overlaid at t = 0.025 and t = 0.05.

The adjusted roc for PSA is shown in Fig. 1(b), estimated using both the semiparametric estimator and a binormal roc regression model, where the control quantiles are estimated using a linear location-scale model (Heagerty & Pepe, 1999) for both methods. Bootstrapping is used for inference, and logit-based confidence intervals for the semiparametric estimator are overlaid at t = 0.025 and t = 0.05. The curve describes the ability of PSA to discriminate between cases and controls of the same age. We estimate that when the age-specific false-positive fraction is held at 0.025, 17% of cases can be detected, with a 95% confidence interval of 13% to 21%. When the common false-positive fraction is increased to 0.05, 27% of cases can be detected, with a 95% confidence interval of 21% to 33%.

6. Discussion

When covariates affect discrimination, there are various ways of combining covariate-specific roc curves. The adjusted roc curve is a simple vertical average. A horizontal average may be more appropriate in certain settings, and is a simple extension of our methods.

The area under the adjusted roc, equivalent to pr(Y_D > Y_D̄ | Z_D = Z_D̄), can be interpreted as the probability of correctly ordering a randomly chosen case and a control observation with the same covariate value. This statistical summary deserves further development and may be used to compare covariate-adjusted roc curves for different markers.

Appendix

Proof of Theorem 1. We write

$$\begin{array}{lll}{n}_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)-{\text{AROC}}_{\theta }(t)\}\hfill & =\hfill & n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)-{\text{AROC}}_{\widehat{\theta }}(t)\}\hfill \\ \hfill & \hfill & +n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)-{\text{AROC}}_{\theta }(t)\}\hfill \\ \hfill & =\hfill & n_D^{-1/2}\sum _{i=1}^{n_D}I\{Y_{D_i}>q\left(t;Z_{D_i},\widehat{\theta }\right)\}-\text{pr}\{Y_D>q\left(t;Z_D,\widehat{\theta }\right)|\widehat{\theta }\}\hfill \\ \hfill & \hfill & +n_D^{1/2}\left[\text{pr}\{Y_D>q\left(t;Z_D,\widehat{\theta }\right)|\widehat{\theta }\}-\text{pr}\{Y_D>q\left(t;Z_D,\theta \right)\}\right]\hfill \\ \hfill & \equiv \hfill & A_n+B_n.\hfill \end{array}$$

Note that $B_n=n_D^{1/2}\{g(\widehat{\theta })-g(\theta )\}$, and by Assumptions 1–3, the delta method (Ferguson, 1996, p. 45) and Slutsky’s theorem, B_n converges in distribution to a normal zero-mean random variable with variance matrix

$${\sigma }_b^2=\lambda \frac{\partial }{\partial \theta }{\text{AROC}}_{\theta }(t)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathrm{\Sigma }}_{\theta }\frac{\partial }{\partial {\theta }^{\text{T}}}{\text{AROC}}_{\theta }(t).$$

Now, we write $A_n=n_D^{-1/2}{\sum }_{i=1}^{n_D}{A}_{n_i}$, and find its asymptotic distribution conditional on θ̂, using the Lindeberg–Feller central limit theorem. First, note that E(A_ni | θ̂) = 0 and var(A_ni | θ̂) = aroc_θ̂(t){1 − aroc_θ̂(t)}. Convergence under the Lindeberg–Feller central limit theorem requires that

$$\frac{1}{n_D{\text{AROC}}_{\widehat{\theta }}(t)\{1-{\text{AROC}}_{\widehat{\theta }}(t)\}}\sum _{i=1}^{n_D}E\left(A_{n_i}^{2}I\left[\left|A_{n_i}\right|⩾∊n_D\hspace{0.17em}\hspace{0.17em}{\text{AROC}}_{\widehat{\theta }}(t)\{1-{\text{AROC}}_{\widehat{\theta }}(t)\}\right]\right)$$ (A1)

converges to zero as n_D → ∞ for all ∊ > 0. However, $A_{n_i}^2\text{I}\left[\left|A_{n_i}\right|⩾∊n_D{\text{AROC}}_{\widehat{\theta }}(t)\{1-{\text{AROC}}_{\widehat{\theta }}(t)\}\right]$ takes the value {1 − aroc_θ̂(t)}²I {aroc_θ̂(t)⁻¹ ⩽ ∊n_D} with probability aroc_θ̂(t), and aroc_θ̂(t)²I [{1 − aroc_θ̂(t)}⁻¹ ⩽ ∊n_D] with probability 1 − aroc_θ̂(t). Hence, (A1) becomes

$$\{1-{\text{AROC}}_{\widehat{\theta }}(t)\}I\{{\text{AROC}}_{\widehat{\theta }}{\left(t\right)}^{-1}⩾∊n_D\}+{\text{AROC}}_{\widehat{\theta }}(t)I\left[{\{1-{\text{AROC}}_{\widehat{\theta }}(t)\}}^{-1}⩾∊n_D\right].$$

Assumptions 4 and 5 ensure that this converges to zero. Thus, conditional on θ̂, A_n [aroc_θ̂(t){1 − aroc_θ̂(t)}]^−1/2 converges in distribution to a standard normal random variable as n_D → ∞. Finally, the asymptotic distribution of A_n [aroc_θ̂(t){1 − aroc_θ̂(t)}]^−1/2 conditional on θ̂ is the same as that of A_n [aroc_θ̂(t){1 − aroc_θ̂(t)}]^−1/2 conditional on B_n, since it is functionally independent of θ̂ and $B_n=n_D^{1/2}\{g(\widehat{\theta })-g(\theta )\}$. By Assumptions 2 and 3, aroc_θ̂(t){1 − aroc_θ̂(t)} converges in probability to aroc_θ(t){1 − aroc_θ(t)} as n_D̄ → ∞. By Slutsky’s theorem,

$$\left(\begin{array}{c}{A}_n{\left[{\text{AROC}}_{\theta }(t)\{1-{\text{AROC}}_{\theta }(t)\}\right]}^{-1/2}\\ B_n\end{array}\right),$$

converges in distribution to a zero-mean bivariate normal random variable with covariance matrix

$$\left(\begin{array}{cc}1& 0\\ 0& {\sigma }_b^2\end{array}\right),$$

as n_D, n_D̄ → ∞. The continuous mapping theorem then yields the desired result.

Nonparametric estimation of aroc(t)

We prove that Assumptions 2 and 3 are satisfied with empirical quantile estimators. Under Assumption 6, by standard empirical process theory (Ferguson, 1996, p. 91), for a fixed stratum Z = z and conditional on n_D̄(z), $n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}$ converges in distribution to a normal zero-mean random variable with variance ${\sigma }_z^2=t\left(1-t\right)f_{\overline{D}}^{-2}\{S_{\overline{D}}^{-1}(t|z)|z\}$. By Assumption 1, $n_{\overline{D}}(z)n_{\overline{D}}^{-1}$ converges in probability to p_D̄(z) as n_D̄ → ∞. Hence, for all ∊, there exists N such that

$$\begin{array}{l}\text{pr}\left[n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}⩽y\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=E\left(\text{pr}\left[n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}⩽y|n_{\overline{D}}(z)\right]\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=E\left(\text{pr}\left[n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}⩽y|n_{\overline{D}}(z)\right]I\{n_{\overline{D}}(z)>N\}\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+E\left(\text{pr}\left[n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}⩽y|n_{\overline{D}}(z)\right]I\{n_{\overline{D}}(z)⩽N\}\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}<\{\mathrm{\Phi }(y/{\sigma }_z)+∊\}\text{pr}\{n_{\overline{D}}(z)>N\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+E\left(\text{pr}\left[n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}⩽y|n_{\overline{D}}(z)\right]I\{n_{\overline{D}}(z)⩽N\}\right).\end{array}$$

Since n_D̄(z) → ∞, the second term can be made arbitrarily small, and pr{n_D̄(z) > N} arbitrarily close to 1, by choosing N large enough. Thus, $n_{\overline{D}}{\left(z\right)}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}$ converges in distribution to a normal zero-mean random variable with variance ${\sigma }_z^2$, and by Slutsky’s theorem, $n_{\overline{D}}^{1/2}\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)-S_{\overline{D}}^{-1}(t|z)\}$ converges in distribution to a normal zero-mean random variable with variance ${\sigma }_z^2/p_{\overline{D}}(z)$ as n_D̄ → ∞. Since observations in different strata are independent, marginal convergence implies joint asymptotic normality, with the variance-covariance matrix ${\mathrm{\Sigma }}_{\theta }=\text{diag}\{{\sigma }_z^2/p_{\overline{D}}(z)\}$. We also calculate the form of(∂/∂θ)aroc_θ(t). We have

$$\begin{array}{lll}\frac{\partial }{\partial {\theta }_z}{\text{AROC}}_{\theta }(t)\hfill & =\hfill & \frac{\partial }{\partial S_{\overline{D}}^{-1}(t|z)}E\left[\text{pr}\{Y_D>S_{\overline{D}}^{-1}(t|Z_D)|Z_D\}\right]\hfill \\ \hfill & =\hfill & -E\left[f_D\{S_{\overline{D}}^{-1}(t|Z_D)|Z_D\}\frac{\partial }{\partial S_{\overline{D}}^{-1}(t|z)}{S}_D^{-1}(t|Z_D)\right]\hfill \\ \hfill & =\hfill & -f_D\{S_D^{-1}(t|z)|z\}{p}_D(z),\hfill \end{array}$$

and V(t) reduces to (3).

Consistency of the semiparametric variance estimator

We write the estimator of (∂/∂θ_j) aroc_θ(t) as

$$\frac{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }+h_j(n)}(t)-{\text{AROC}}_{\theta +h_j(n)}(t)}{2h(n)}+\frac{{\text{AROC}}_{\theta +h_j(n)}(t)-{\text{AROC}}_{\theta -h_j(n)}(t)}{2h(n)}-\frac{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }-h_j(n)}(t)-{\text{AROC}}_{\theta -h_j(n)}(t)}{2h(n)}.$$ (A2)

Consider the first component. We claim that $n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }+h_j(n)}(t)-{\text{AROC}}_{\theta +h_j(n)}(t)\}$ converges in distribution to a normal zero-mean random variable with variance V(t). The proof of this fact is similar to the proof that $n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }}(t)-{\text{AROC}}_{\theta }(t)\}$ is asymptotically normal, and hence O_p(1), as proven earlier. Hence, $n_D^{1/2}\{{\text{AR}\widehat{\text{O}}\text{C}}_{\widehat{\theta }+h_j(n)}(t)-{\text{AROC}}_{\theta +h_j(n)}(t)\}/\{2n_D^{1/2}h(n)\}\to 0$ in probability since the denominator converges to ∞. A similar argument can be used to prove that the third term in (A2) converges to 0. Finally, {aroc_θ+hj(n)(t) − aroc_θ−hj(n)(t)}/{2h(n)} converges in probability to (∂/∂θ_j)aroc_θ(t) by continuity of aroc_θ(t) in θ; see Assumption 3. Hence, our estimator of (∂/∂θ_j) aroc_θ(t) is consistent. Now, with Σ^_θ converging in probability to Σ_θ, arôc_θ̂(t) converging in probability to aroc_θ(t) and consistency of the derivative estimator, we have consistency of the composite variance estimator.

Consistency of the nonparametric variance estimator

We have p̂_D(z) converging in probability to p_D(z) and p̂_D̄(z) converging in probability to p_D̄(z) for z = 1, . . . , K, and, by standard empirical process theory (Ferguson, 1996, p. 91) and Assumption 6, ${\widehat{S}}_{\overline{D}}^{-1}(t|z)$ converges in probability to $S_D^{-1}(t|z)$ as n_D̄ → ∞. We write

$$\begin{array}{lll}\left|{\widehat{f}}_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}-f_D\{S_{\overline{D}}^{-1}(t|z)|z\}\right|\hfill & =\hfill & |{\widehat{f}}_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}-f_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}\hfill \\ \hfill & \hfill & +f_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}-f_D\{S_{\overline{D}}^{-1}(t|z)|z\}|\hfill \\ \hfill & ⩽\hfill & \left|{\widehat{f}}_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}-f_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}\right|\hfill \\ \hfill & \hfill & +\left|f_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}-f_D\{S_{\overline{D}}^{-1}(t|z)|z\}\right|.\hfill \end{array}$$

The first term converges in probability to zero by the uniform consistency of f̂_D(y | z), while the second term converges in probability to zero by the consistency of ${\widehat{S}}_{\overline{D}}^{-1}(t|z)$, Assumption 8 and the continuous mapping theorem. Hence, ${\widehat{f}}_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}$ converges in probability to $f_D\{S_{\overline{D}}^{-1}(t|z)|z\}$ as n_D, n_D̄ → ∞. A similar argument shows that ${\widehat{f}}_D\{{\widehat{S}}_{\overline{D}}^{-1}(t|z)|z\}$ is also consistent. The variance estimator is a continuous function of these components, and under Assumption 6 it is consistent.