Asymptotic Properties of the Sequential Empirical ROC, PPV and NPV Curves Under Case-Control Sampling

Abstract

The receiver operating characteristic (ROC) curve, the positive predictive value (PPV) curve and the negative predictive value (NPV) curve are three measures of performance for a continuous diagnostic biomarker. The ROC, PPV and NPV curves are often estimated empirically to avoid assumptions about the distributional form of the biomarkers. Recently, there has been a push to incorporate group sequential methods into the design of diagnostic biomarker studies. A thorough understanding of the asymptotic properties of the sequential empirical ROC, PPV and NPV curves will provide more flexibility when designing group sequential diagnostic biomarker studies. In this paper we derive asymptotic theory for the sequential empirical ROC, PPV and NPV curves under case-control sampling using sequential empirical process theory. We show that the sequential empirical ROC, PPV and NPV curves converge to the sum of independent Kiefer processes and show how these results can be used to derive asymptotic results for summaries of the sequential empirical ROC, PPV and NPV curves.

1. Introduction

Several recent papers have discussed the application of group sequential methodology to diagnostic biomarker studies (Tang, Emerson and Zhou, 2008; Tang and Liu, 2010; Pepe et al., 2009). Group sequential study designs (i.e. study designs with multiple interim analyses) provide an opportunity to improve the efficiency of diagnostic biomarker studies by allowing studies to terminate early when the candidate marker is clearly superior or inferior to established markers or historical levels of marker performance. Many group sequential methods assume the existence of a test statistic with an independent increments covariance structure (Jennison and Turnbull, 2000). A thorough understanding of the asymptotic properties of the sequential empirical ROC, PPV and NPV curves and, specifically, verifying that their summary measures have an independent increments covariance structure, would provide great flexibility when designing group sequential diagnostic biomarker studies.

Diagnostic biomarkers are used to classify a patient as a case or a control. A dichotomous biomarker results in either a positive test, indicating that the subject should be classified as a case, or a negative test, indicating that the subject should be classified as a control. Many biomarkers are measured on a continuous scale and a threshold must be defined in order to translate a continuous biomarker into a positive or negative test result. Let D be a Bernoulli random variable indicating disease status with prevalence ρ and let X be a biomarker value with conditional distribution F(x|D = 1) ≡ F_D(x) and F(x|D = 0) ≡ F_D¯(x), where F_D(x) is the distribution function for the cases and F_D¯(x) is the distribution function for the controls. Furthermore, we define F(x) ≡ F_D(x) + (1 − ρ) F_D¯(x) to be the biomarker distribution function for the entire population. Without loss of generality, assume that larger biomarker values are more indicative of disease. For a threshold c, a biomarker value X is translated into a positive test result if it is greater than c and a negative test result if it is less than or equal to c.

The receiver operating characteristic (ROC) curve summarizes the classification accuracy of a continuous diagnostic biomarker (Pepe, 2003) by reporting the true positive fraction (TPF) and the false positive fraction (FPF) for all possible cut-offs of the marker. For a threshold c, TPF(c) = P [X > c|D = 1] and FPF(c) = P [X > c|D = 0]. The ROC curve is defined as,

$$\mathit{\text{ROC}}\left(c\right)=\left\{\right(\mathit{\text{TPE}}\left(c\right),\mathit{\text{FPE}}\left(c\right)),c\in (-\infty ,\infty \left)\right\},$$

and can alternately be expressed as,

$$\mathit{\text{ROC}}(t)=S_D\left(S_{\overline{D}}^{-1}(t)\right),t\in (0,1),$$ (1.1)

where S_D(x) = 1−F_D(x) and S_D¯(x) = 1−F_D¯(x). ROC(t) can be interpreted as the TPF corresponding to a FPF of t. Alternately, one might be interested in the inverse of the ROC curve,

$${\mathit{\text{ROC}}}^{-1}(\upsilon )=S_{\overline{D}}\left(S_D^{-1}(\upsilon )\right),\upsilon \in (0,1).$$ (1.2)

ROC⁻¹ (υ) is indexed by the TPF and can be interpreted as the FPF corresponding to a TPF of υ.

The predictive accuracy of a dichotomous biomarker can be summarized by the positive predictive value (PPV) and negative predictive value (NPV).

The PPV and NPV curves were proposed as an extension of PPV and NPV to continuous markers (Moskowitz and Pepe, 2004; Zheng et al., 2008). For a threshold c, PPV (c) = P[D = 1|X > c] and NPV (c) = P[D = 0|X ≤ c]. The PPV and NPV curves are defined as PPV (c) and NPV (c) for all c ∊ (−∞, ∞). In practice, PPV and NPV curves are indexed by a summary of the marker distribution rather than a generic threshold (Moskowitz and Pepe, 2004; Zheng et al., 2008). In this paper, we consider the PPV and NPV curves indexed by the FPF and the percentile value in the entire population.

The ROC, PPV and NPV curves are commonly estimated nonparametrically to avoid making assumptions about the form of F_D(x) and F_D̄(x). This is particularly important in the case of the ROC, PPV and NPV curves because we are often interested in regions of the curve that correspond to the tails of these distributions. For example, a biomarker must possess a high specificity in order to be clinically useful in a low disease risk population screening setting, which corresponds to the upper tail of the biomarker distribution among controls.

Our understanding of the empirical ROC curve is enhanced by knowledge of its asymptotic properties. Hsieh and Turnbull (1996) showed that the empirical ROC curve converges to the sum of two independent Brownian bridges. The asymptotic normality of summary measures of the empirical ROC curve, such as the area under the ROC curve or a point on the ROC curve, can be derived from their work. To our knowledge, no asymptotic theory is available for the empirical PPV and NPV curves.

Tang, Emerson and Zhou (2008) showed that a family of weighted area under the ROC curve (wAUC) statistics has an independent increments covariance structure. It would be beneficial to show that this assumption holds for a larger class of summaries of the ROC curve. In this paper, we develop asymptotic theory for the sequential empirical ROC, PPV and NPV curves. Our results allow us to develop distribution theory for other summaries of the ROC curve and to develop distribution theory for summaries of the PPV and NPV curves.

2. Notation and Definitions

Before beginning our discussion of the sequential empirical ROC, PPV and NPV curves, we provide definitions of the sequential empirical estimates for the underlying distribution and quantile functions. Let X_D,1, X_D,2,…, X_D,nD be i.i.d. marker values for the cases with distribution function, F_D(x), and X_D̄,1, X_D̄,2,…, X_D̄,nD̄ be i.i.d. marker values for the controls with distribution function, F_D̄(x). Furthermore, let r_D and r_D̄ refer to the proportion of case and controls, respectively, that are observed at a given time point. The sequential empirical estimate of F_D(x) is defined as

$${\widehat{F}}_{D,r_D}(x)=\{\begin{array}{cc}0,& 0\le r_D<\frac{1}{n_D},\\ \frac{1}{\left[r_D{n}_D\right]}{\sum }_{i=1}^{\left[r_D{n}_D\right]}1\left\{X_{D,i}\le x\right\},& -\infty <x<\infty ,\frac{1}{n_D}\le r_D\le 1,\end{array}$$

and the sequential empirical estimate of $F_D^{-1}(t)$ is defined as

$${\widehat{F}}_{D,r_D}^{-1}(t)=\{\begin{array}{cc}{X}_{D,1,\left[r_D{n}_D\right]}& \mathit{\text{if}}t=0,0\le r_D\le 1,\\ X_{D,k,\left[r_D{n}_D\right]}& \mathit{\text{if}}\frac{k-1}{\left[r_D{n}_D\right]}<t\le \frac{k}{\left[r_D{n}_D\right]},1\le k\le \left[r_D{n}_D\right],0\le t\le 1\end{array}$$

where X_D,1,[rDnD], X_D,2,[rDnD], …, X_{D,[rDnD],[rDnD]} are the sequential order statistics of the biomarker values for the cases. The sequential empirical estimates of S_D(x) and $S_D^{-1}(t)$ are defined as Ŝ_D,rD(x) = 1 − F̂_D,rD(x) and ${\widehat{S}}_{D,r_D}^{-1}(t)={\widehat{F}}_{D,r_D}^{-1}(1-t)$. The sequential empirical estimates for the control population are defined in an analogous fashion. The sequential empirical estimates of F_D(x) and F_D̄(x) lead to a natural definition of the sequential empirical estimates of F(x) and F⁻¹(t),

$${\widehat{F}}_{r_D,r_{\overline{D}}}(x)=\rho {\widehat{F}}_{D,r_D}(x)+\left(1-\rho \right){\widehat{F}}_{\overline{D},r_{\overline{D}}}(x)$$

and

$${\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(t)=inf\left\{x:{\widehat{F}}_{r_D,r_{\overline{D}}}(x)\ge t\right\},$$

where ρ is assumed to be known. F̂_rD,rD̄(x) is a linear combination of F̂_D,rD(x) and F̂_D̄,rD̄(x) and is therefore indexed by both r_D, the proportion of cases observed at a given time point, and r_D̄, the proportion of controls observed at a given time point.

Throughout this paper We let 0 < a < b < 1, 0 < c < 1, 0 < d < 1 and make the following assumptions:

• A1 F_D(x) and F_D̄(x) are continuous distribution functions with continuous densities f_D(x) and f_D̄(x), respectively,

• A2 f_D(x) > 0 for x ∊ (sup{x: F_D(x) = 0}, inf{x: F_D(x) = 1}),

• A3 f_D̄(x) > 0 for x ∊ (sup{x: F_D̄(x) = 0}, inf{x: F_D̄(x) = 1}),

• A4 $\frac{n_D}{n_D}\to \lambda >0$ as n_D → ∞ and n_D̄ → ∞, i.e. the ratio of cases to controls converges to a constant that is greater than 0.

The asymptotic results in Section 3 make use of the Kiefer process. The Kiefer process, K(t,r), is a two-dimensional, mean-zero Gaussian process with covariance:

$$\mathit{\text{Co}}\upsilon \left(K\right(t_1,r_1),K(t_2,r_2\left)\right)=(t_1\wedge t_2-t_1{t}_2)(r_1\wedge r_2)$$

where ∧ represents the minimum. The Kiefer process behaves like a Brownian Bridge in t and Brownian Motion in r.

The remainder of this paper proceeds as follows. In Section 3, we develop asymptotic theory for the sequential empirical ROC, PPV and NPV curves. First, we generalize the work of Hsieh and Turnbull (1996) to the sequential empirical ROC curve by showing that the sequential empirical ROC curve converges to the sum of independent Kiefer processes. Next, we develop asymptotic theory for the sequential empirical PPV and NPV curves indexed by the FPF by writing them as functions of the sequential empirical ROC curve. Finally, we follow the approach of Pyke and Shorack (1968) to develop asymptotic theory for the PPV and NPV curves indexed by the percentile value of the marker distribution. We validate our asymptotic results by simulation in Section 4 and illustrate how they can be used to design group sequential diagnostic biomarker studies in Section 5. We conclude with a discussion in Section 6.

3. Asymptotic Results

3.1. The Sequential Empirical ROC Curve

In this section we provide asymptotic results for the sequential empirical ROC curve. Results for the inverse of the sequential empirical ROC curve are nearly identical; we direct the reader to an associated technical report for details (Koopmeiners and Feng, 2010). The sequential empirical ROC curve, ${\widehat{\mathit{\text{ROC}}}}_{r_D,r_D}(t)$ is defined by substituting the sequential empirical estimates of S_D(x) and S_D¯ (x) into (1.1), yielding,

$${\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)={\widehat{S}}_{D,r_D}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t)\right),$$

and for ease of notation, we define,

$$R_{r_D,r_{\overline{D}}}(t)\equiv n_D^{-1/2}\left[n_D{r}_D\right]({\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)-\mathit{\text{ROC}}(t)).$$

The primary result in this section provides asymptotic theory for R_rD,_rD̄(t). By developing asymptotic theory for R_rD,_rD̄(t), we are also able to develop asymptotic theory for functionals of R_rD,_rD̄(t) as a special case. Theorem 3.1 establishes the convergence of R_rD,_rD̄(t) to the sum of independent Kiefer processes.

Theorem 3.1

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. As n_D → ∞ and n_D̄ → ∞

$$R_{r_D,r_{\overline{D}}}(t)\to {}_d{K}_1(\mathit{\text{ROC}}(t),r_D)+{\lambda }^{1/2}\frac{r_D}{r_{\overline{D}}}\left(\frac{f_D(S_{\overline{D}}^{-1}\left(t\right))}{f_{\overline{D}}(S_{\overline{D}}^{-1}\left(t\right))}\right)K_2\left(t,r_{\overline{D}}\right)$$

uniformly for t ∊ [a, b], r_D ∊ [c, 1] and r_D̄ ∊ [d, 1] where K₁ and K₂ are independent Kiefer Processes.

A proof of Theorem 3.1 can be found in the Appendix. Theorem 3.1 generalizes the results of Hsieh and Turnbull (1996) to the sequential empirical ROC curve. The proof of Theorem 3.1 is similar to the proof found in Hsieh and Turnbull (1996) but our proof relies on the more powerful sequential empirical process theory. Sequential empirical process theory generalizes asymptotic theory for the standard empirical process by introducing a parameter for time. In doing so, asymptotic results for the sequential empirical process involve the Kiefer Process. Using properties of the Kiefer Process, we are able to easily derive asymptotic results for summaries of the sequential empirical ROC curve and verify that the independent increments assumption holds in many cases. Furthermore, we can recover Hsieh and Turnbull's result as a special case of Theorem 3.1 by letting r_D and r_D̄ both equal 1.

Corrollary 3.2

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. As n_D→ ∞ and n_D̄ → ∞

$$R_{1,1}(t)\to {}_d{B}_1(\mathit{\text{ROC}}(t))+{\lambda }^{1/2}\left(\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}\right)B_2(t)$$

uniformly for t ∊ [a, b] where B₁ and B₂ are independent Brownian Bridges.

Proof. Immediate from Theorem 3.1 and by noting that K(t, 1) =_dB(t).

An advantage to studying the asymptotic behavior of the sequential empirical ROC curve at the process level, rather than a single point on the sequential empirical ROC curve, is that we are able to study the joint behavior of multiple points on the ROC curve. Corollary 3.3 provides a normal approximation for a vector of points on the sequential empirical ROC curve.

Corollary 3.3

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. For t₁,t₂, …,t_J ∊ (0,1), r_D,1, r_D,2, …, r_D,J ∊ (0,1] and r_D̄,₁, r_D̄,2, …, r_D̄,J ∊ (0,1], a vector of arbitrary points on the sequential empirical ROC curve, $\left({\widehat{\mathit{\text{ROC}}}}_{r_{D,1},r_{\overline{D},1}}(t_1),{\widehat{\mathit{\text{ROC}}}}_{r_{D,2},r_{\overline{D},2}}(t_2),\dots ,{\widehat{\mathit{\text{ROC}}}}_{r_{D,J},r_{\overline{D},J}}(t_J)\right),$ is approximately multivariate normal with,

$${\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)\sim N(\mathit{\text{ROC}}(t_j),{\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)}^2)j=1,2,\dots ,J$$

where

$${\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}(t_j)}}^2=\frac{\mathit{\text{ROC}}(t_j)(1-\mathit{\text{ROC}}(t_j))}{n_D{r}_{D,j}}+{\left(\frac{f_D(S_{\overline{D}}^{-1}(t_j))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t_j))}\right)}^2\frac{t_j(1-t_j)}{n_{\overline{D}}{r}_{\overline{D},j}},$$

and

$$\mathit{\text{Co}}\upsilon [{\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}\left(t_i\right)},{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)]=\frac{(r_{D,i}\wedge r_{D,j})\left(\mathit{\text{ROC}}\right(t_i)\wedge \mathit{\text{ROC}}(t_j)-\mathit{\text{ROC}}(t_i\left)\mathit{\text{ROC}}\right(t_j\left)\right)}{n_D{r}_{D,i}{r}_{D,j}}+\left(\frac{f_D\left(S_{\overline{D}}^{-1}\right(t_i\left)\right)}{f_{\overline{D}}\left(S_{\overline{D}}^{-1}\right(t_i\left)\right)}\right)\left(\frac{f_D\left(S_{\overline{D}}^{-1}\right(t_j\left)\right)}{f_{\overline{D}}\left(S_{\overline{D}}^{-1}\right(t_j\left)\right)}\right)\frac{(r_{\overline{D},i}\wedge r_{\overline{D},j})(t_i\wedge t_j-t_i{t}_j)}{n_{\overline{D}}{r}_{\overline{D},i}{r}_{\overline{D},j}}.$$

Proof. Immediate from Theorem 3.1.

Corollary 3.3 provides the asymptotic covariance for two points at different locations and different times on the sequential empirical ROC curve. This allows us to fully specificy the joint sequential distribution of multiple points on the ROC curve, which allows us to design group sequential diagnostic biomarker studies where multiple points on the ROC curve are treated as multiple endpoints of a group sequential study. For example, we might be interested in ROC(t₁) and ROC(t₂), where t₁ is chosen for high specificity to rule patients in for work up and t₂ is chosen for high sensitivity to rule out patients for invasive work.

Our interest in the sequential empirical ROC curve is motivated by the need to design group sequential diagnostic biomarker studies. Our ability to design group sequential diagnostic biomarker studies would be enhanced by showing that summaries of the sequential empirical ROC curve have an independent increments covariance structure. The simplest summary of the ROC curve is a point on the ROC curve, ROC(t). ROC(t) can be interpreted as the sensitivity at a specificity of 1 − t. Corollary 3.4 shows that the sequential empirical estimator of ROC(t) is asymptotically normal and has independent increments when divided its variance.

Corollary 3.4

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. For t ∊ (0,1) and J stopping times, $({\widehat{\mathit{\text{ROC}}}}_{r_{D,1},r_{\overline{D},1}}(t),{\widehat{\mathit{\text{ROC}}}}_{r_{D,2},r_{\overline{D},2}}(t),\dots ,{\widehat{\mathit{\text{ROC}}}}_{r_{D,J},r_{\overline{D},J}}(t)),$ is approximately multivariate normal with,

$${\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}}(t)\sim N(\mathit{\text{ROC}}(t),{\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}(t)}}^2)i=1,2,\dots ,J$$

and

$$\mathit{\text{Co}}\upsilon [{\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}}(t),{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t)]=\mathit{\text{Var}}[{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t)]={\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t)}^2,r_i\le r_j$$

where ${\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t)}^2$ is defined as in Corollary 3.3.

Proof. Immediate from Corollary 3.3.

Asymptotic theory for other summary measures of the ROC curve, such as the area under the curve or the partial area under the curve, can also be derived from Theorem 3.1. This illustrates the flexibility of Theorem 3.1. By developing distribution theory for the sequential empirical ROC curve, we are able to derive distribution theory for summaries of the ROC curve as a special case.

3.2. The sequential empirical PPV and NPV curves indexed by the False Positive Fraction

In this section, we consider the sequential empirical PPV and NPV curves indexed by the false positive fraction, t. The PPV and NPV curve indexed by the false positive fraction can be written as a function of the ROC curve and their asymptotic properties can be derived using the results from Section 3.1. Asymptotic results for the PPV and NPV curve indexed by the true positive fraction, υ, can similarly be derived by writing the PPV and NPV curve as a function of the inverse of the ROC curve but are not presented in this paper. The interested reader is directed to Koopmeiners and Feng (2010) for details.

The PPV and NPV curves indexed by the false positive fraction are defined as $\mathit{\text{PPV}}(t)=P[D=1|X>S_{\overline{D}}^{-1}(t)]$ and $\mathit{\text{NPV}}(t)=P[D=0|X\le S_{\overline{D}}^{-1}(t)]$ for all t ∊ (0,1) and can be written as functions of the ROC curve as follows:

$$\mathit{\text{PPV}}(t)=\frac{\mathit{\text{ROC}}(t)\rho }{\mathit{\text{ROC}}(t)\rho +t(1-\rho )},$$ (3.1)

and

$$\mathit{\text{NPV}}(t)=\frac{(1-t)(1-\rho )}{(1-\mathit{\text{ROC}}(t))\rho +(1-t)(1-\rho )}.$$ (3.2)

The sequential empirical estimators of PPV(t) and NPV(t) are defined be plugging the sequential empirical estimator of ROC(t) into (3.1) and (3.2), yielding,

$${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(t)=\frac{{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)\rho }{{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)\rho +t\left(1-\rho \right)},$$

and

$${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(t)=\frac{\left(1-t\right)\left(1-\rho \right)}{\left(1-{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)\right)\rho +\left(1-t\right)\left(1-\rho \right)}.$$

From this point forward we only consider ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(t)$ and note that results for ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(t)$ are nearly identical. Again, for ease of notation, we define,

$$P_{r_D,r_{\overline{D}}}(t)\equiv n_D^{-1/2}\left[n_D{r}_D\right]({\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(t)-\mathit{\text{PPV}}(t)).$$

We begin by using the results of Section 3.1 to derive asymptotic theory for P_rD,rD¯(t). Theorem 3.5 establishes the convergence of P_rD,rD¯(t) to the sum of two independent Kiefer Processes.

Theorem 3.5

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. As n_D→ ∞ and n_D̄ → ∞

$$P_{r_D,r_{\overline{D}}}(t)\to d\left(\frac{t(1-\rho )\rho }{{(\mathit{\text{ROC}}(t)\rho +t(1-\rho ))}^2}\right)(K_1(\mathit{\text{ROC}}(t),r_D)+{\lambda }^{1/2}\frac{r_D}{r_{\overline{D}}}(\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))})K_2(t,r_{\overline{D}}))$$

uniformly for t ∊ [a, b], r_D ∊ [c, 1] and r_D̄ ∊ [d, 1] where K₁ and K₂ are independent Kiefer Processes.

The proof of Theorem 3.5 relies on writing P_rD,rD̄(t) as a function of R_rD,rD̄(t)

$$P_{r_D,r_{\overline{D}}}(t)=\frac{(\frac{{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)\rho }{{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)\rho +t(1-\rho )}-\frac{\mathit{\text{ROC}}(t)\rho }{\mathit{\text{ROC}}(t)\rho +t(1-\rho )})}{{\widehat{\mathit{\text{ROC}}}}_{r_D,r_{\overline{D}}}(t)-\mathit{\text{ROC}}(t)}{R}_{r_D,r_{\overline{D}}}(t),$$

and applying the results of Theorem 3.1. The first term converges to $\left(\frac{t(1-\rho )\rho }{{(\mathit{\text{ROC}}(t)\rho +t(1-\rho ))}^2}\right)$ and R_rD,rD̄(t) converges to the sum of two independent Keifer process by Theorem 3.1. A formal proof of Theorem 3.5 can be found in Koopmeiners and Feng (2010).

From Theorem 3.5, we can prove analogous results to Corollaries 3.3 and 3.4 for the sequential empirical PPV curve indexed by the FPF. Namely, that an arbitrary vector of points on the sequential empirical PPV curve follows a multivariate normal distribution and the sequential empirical estimate of a point on the PPV curve is approximately normally distributed with an independent increments covariance structure. We leave the formal statement of these corollaries for the Appendix but present the form of the covariance between two arbitrary points on the sequential empirical PPV curve:

$$\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{PPV}}}}_{r_{D¯i}¯r_{\overline{D}¯i}}(t_i)¯{\widehat{\mathit{\text{PPV}}}}_{r_{D¯j}¯r_{\overline{D}¯j}}(t_j)]=\left(\frac{t_i(1-\rho )\rho }{{(\mathit{\text{ROC}}(t_i)\rho +t_i(1-\rho ))}^2}\right)\left(\frac{t_i(1-\rho )\rho }{{(\mathit{\text{ROC}}(t_i)\rho +t_i(1-\rho ))}^2}\right)\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{ROC}}}}_{r_{D¯i}¯r_{\overline{D}¯i}}(t_i),{\widehat{\mathit{\text{ROC}}}}_{r_{D¯j}¯r_{\overline{D}¯j}}(t_j)].$$

PPV(t) is a function of ROC(t) and, therefore, distribution theory for a vector of points on the PPV curve can also be derived using the delta method and Corollary 3.3.

Asymptotic theory for the fixed-sample empirical PPV curve indexed by the FPF, which was previously unavailable, can be derived as a special case of Theorem 3.5 by letting r_D and r_D̂ equal 1. The fixed-sample empirical PPV curve converges to the sum of independent Brownian bridges,

$$P_{\mathrm{1,1}}(t){\to }_d\left(\frac{t(1-\rho )\rho }{{(\mathit{\text{ROC}}(t)\rho +t-\rho ))}^2}\right)(B_1(\mathit{\text{ROC}}(t))+{\lambda }^{1/2}(\frac{f_D\left(S_{\overline{D}}^{-1}\right(t\left)\right)}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}\left)B_2\right(t\left)\right),$$

which allows us to derive a normal approximation for the empirical estimate of a point on the PPV curve,

$${\widehat{\mathit{\text{PPV}}}}_{\mathrm{1,1}}(t)\sim N(\mathit{\text{PPV}}(t),{\left(\frac{t(1-\rho )\rho }{{\left(\mathit{\text{ROC}}\right(t)\rho +t-\rho ))}^2}\right)}^2{\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{\mathrm{1,1}}\left(t\right)}^2)$$

where ${\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{\mathrm{1,1}}(t)}^2$ is defined as in Corollary 3.3.

3.3. The sequential empirical PPV and NPV curves indexed by the Percentile Value

Finally, we consider the PPV and NPV curves indexed by the proportion of the population that are classified as negative, u, and positive, 1 – u. In this case, the PPV and NPV curves are defined as PPV(u) = P[D = 1|X > F⁻¹(u)] and NPV(u) = P[D = 0|X ≤ F⁻¹(u)] for all u ∈ (0,1). Under this indexing, the PPV curve can be written as

$$\mathit{\text{PPV}}(u)=\frac{S_D(F^{-1}(u))\rho }{1-u},$$ (3.3)

and since the NPV curve can be written as,

$$\mathit{\text{NPV}}(u)=\frac{u-\rho }{u}+\frac{1-u}{u}\mathit{\text{PPV}}(u),$$ (3.4)

it suffices to study the PPV curve when considering estimation of the NPV curve.

The sequential empirical estimator of PPV(u) is found by substituting the sequential empirical estimators of S_D(x) and F(x), along with the known value of ρ, into (3.3),

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,}{r}_{\overline{D}}}(u)=\frac{{\widehat{S}}_{D,r_D}({\widehat{F}}_{r_{D,}{r}_{\overline{D}}}^{-1}(u))\rho }{1-u},$$ (3.5)

and the sequential empirical estimator of NPV(u) is found by substituting the sequential empirical estimator of PPV(u) into (3.4),

$${\widehat{\mathit{\text{NPV}}}}_{r_{D,}{r}_{\overline{D}}}(u)=\frac{u-\rho }{u}+\frac{1-u}{u}{\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(u).$$ (3.6)

Finally, we define,

$$P_{r_D,r_{\overline{D}}}\left(u\right)=n_D^{-1/2}[n_D{r}_D]\left({\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}\right(u)-\mathit{\text{PPV}}(u\left)\right)$$

and

$$N_{r_D,r_{\overline{D}}}\left(u\right)=n_D^{-1/2}\left[n_D{r}_D\right]\left({\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}\right(u)-\mathit{\text{NPV}}(u\left)\right)$$

for mathematical convenience. We begin by developing distribution theory for Pr_D,r_D̂(u). Theorem 3.6 establishes the convergence of the sequential empirical PPV curve to the sum of two independent Kiefer Processes.

Theorem 3.6

Assume A1 - A4 hold and let $f_D(F^{-1}(u))f(F^{-1}(u))$ be bounded on [a, b]. As n_D → ∞ and n_D̂ → ∞

$$P_{r_D,r_{\overline{D}}}(u){\to }_d-\frac{\rho (1-\rho )}{1-u}\frac{f_{\overline{D}}(F^{-1}(u))}{f(F^{-1}(u))}{K}_1(F_D(F^{-1}(u)),r_D)+\frac{\rho (1-\rho )}{1-u}\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}\sqrt{\lambda }\frac{r_D}{r_{\overline{D}}}{K}_2(F_{\overline{D}}(F^{-1}(u)),r_{\overline{D}})$$

uniformly for u ∈ [a, b], r_D ∈ [c, 1] and r_D̂ ∈ [d,1] where K1 and K2 are independent Kiefer Processes.

The proof of Theorem 3.6 is complicated by the fact that S̄_D,r_D(x) and ${\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(t)$ are correlated because F̄r_D,r_D̂(x) is a linear combination of F̄_D,r_D(x) and F̄_D̂,r_D̂.(x) In contrast, the sequential empirical ROC curve and the sequential empirical PPV curve indexed by the FPF are functionals of two independent sequential empirical estimators, S̄_D,r_D(x) and ${\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t),$ which makes it easier to show that Rr_D,r_D̂(t) and Pr_D,r_D̂(t) converge to the sum of independent Kiefer processes. To account for the correlation between S̄_D,r_D(x) and ${\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(t)$ we follow the approach of Pyke and Shorack (1968), who prove a similar result for two correlated, fixed-sample empirical processes. The proof of Theorem 3.6 can be found in the Appendix.

Theorem 3.6 also establishes asymptotic theory for the sequential empirical NPV curve because ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(t)$ is a function of ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(t).$ Corollary 3.7 establishes the convergence of Nr_D,r_D̂ (t) to the sum of two independent Kiefer processes.

Corollary 3.7

Assume A1 - A4 hold and let $\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}$ be bounded on [a,b]. As n_D → ∞ and n_D̂ → ∞

$$N_{r_D,r_{\overline{D}}}(u){\to }_d-\frac{\rho (1-\rho )}{1-u}\frac{f_{\overline{D}}(F^{-1}(u))}{f(F^{-1}(u))}{K}_1(F_D(F^{-1}(u)),r_D)+\frac{\rho (1-\rho )}{1-u}\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}\sqrt{\lambda }\frac{r_D}{r_{\overline{D}}}{K}_2(F_{\overline{D}}(F^{-1}(u)),r_{\overline{D}})$$

uniformly for u ∈ [a, b], r_D ∈ [c, 1] and r_D̂ ∈ [D, 1] where K1 and K2 are independent Kiefer Processes.

Corollary 3.7 is immediate from Theorem 3.6 by noting that,

$$N_{r_D,r_{\overline{D}}}(t)=\frac{1-u}{u}{P}_{r_D,r_{\overline{D}}}(t).$$

As with the ROC curve and the PPV curve indexed by the FPF, Theorem 3.6 and Corollary 3.7 allow us to develop distribution theory for summaries of the PPV and NPV curve indexed by u. Distribution theory for a vector of points on the PPV or NPV curve is left for the Appendix but we choose to highlight the joint distribution of the sequential empirical estimate of a single point on the PPV or NPV curve. Corollary 3.8 establishes that the sequential empirical estimate of a point on the PPV or NPV curve is asymptotically normal and has independent increments when divided by its variance.

Corollary 3.8

Assume A1 - A4 hold and let $\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}$ be bounded on [a, b]. For u ∈ (0,1) and J stopping times,

A. $({\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}(u),\dots ,{\widehat{\mathit{\text{PPV}}}}_{r_{D,J},r_{\overline{D},J}}(u))$, is approximately multivariate normal with,

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(u)~N(\mathit{\text{PPV}}(u),{\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(u)}^2)i=1,2,\dots ,J$$

and

$$\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(u),{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)]=\mathit{\text{V ar}}[{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)]={\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)}^2,r_i\le r_j$$

where

$${\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)}^2=\frac{{(\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}(1-\rho ))}^2\mathit{\text{PPV}}(u)(\frac{\rho }{1-u}-\mathit{\text{PPV}}(u))}{n_D{r}_{D,j}}+\frac{{\left(\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}\rho \right)}^2(1-\mathit{\text{PPV}}(u))(\frac{u-\rho }{1-u}+\mathit{\text{PPV}}(u))}{n_{\overline{D}}{r}_{\overline{D},j}}.$$

B. $({\widehat{\mathit{\text{NPV}}}}_{r_{D,1},r_{\overline{D},1}}(u),{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}(u),\dots ,{\widehat{\mathit{\text{NPV}}}}_{r_{D,J},r_{\overline{D},J}}(u)),$ is approximately multivariate normal with,

$${\widehat{\mathit{\text{NPV}}}}_{r_{D,i},r_{\overline{D},i}}(u)~N(\mathit{\text{NPV}}(u),{\sigma }_{{\widehat{\mathit{\text{NPV}}}}_{r_{D,i},r_{\overline{D},i}}(u)}^2)i=1,2,\dots ,J$$

and

$$\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{NPV}}}}_{r_{D,i},r_{\overline{D},i}}(u),{\widehat{\mathit{\text{NPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)]=\mathit{\text{V ar}}[{\widehat{\mathit{\text{NPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)]={\sigma }_{{\widehat{\mathit{\text{NPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)}^2,r_i\le r_j$$

where

$${\sigma }_{{\widehat{\mathit{\text{NPV}}}}_{r_{D,j},r_{\overline{D},j}}(u)}^2=\frac{{(\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}(1-\rho ))}^2(\mathit{\text{NPV}}(u)+\frac{\rho -u}{u})(1-\mathit{\text{NPV}}(u))}{n_D{r}_{D,j}}+\frac{{\left(\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}\rho \right)}^2(1-\mathit{\text{NPV}}(u))(\frac{1-\rho }{u}+\mathit{\text{NPV}}(u))}{n_{\overline{D}}{r}_{\overline{D},j}}.$$

It is immediate from Theorem 3.6 and Corollary 3.7 that ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(u)$ and ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(u)$ are asymptotically normal with an independent increments covariance structure. By noting that,

$$F_D(F^{-1}(u))=1-\frac{1-u}{\rho }\mathit{\text{PPV}}(u)=\frac{u}{\rho }(1-\mathit{\text{NPV}}(u)),$$

and

$$F_{\overline{D}}(F^{-1}(u))=1-\frac{1-u}{1-\rho }(1-\mathit{\text{PPV}}(u))=\frac{u}{1-\rho }\mathit{\text{NPV}}(u),$$

we can write the asymptotic variances of ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(u)$ and ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(u)$ as functions of PPV (u) and NPV (u), respectively. This provides a better understanding of the mean-variance relationship for the asymptotic distributions of ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(u)$ and ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(u)$ and, perhaps, provides a form of the variance that is easier to work with in practical situations (i.e. study design, estimating the standard error, etc.).

An important component of Theorem 3.6 and Corollary 3.7 is that not only do Pr_D,r_D̂(u) and Nr_D,r_D̂(u) converge to the sum of independent Kiefer processes, but they both converge to the same two Kiefer processes. As a result, we are able to derive the correlation between a point on the PPV curve and a point on the NPV curve. Corollary 3.9 provides a bivariate normal approximation for a point on the PPV and a point on the NPV curve.

Corollary 3.9

Assume A1 - A4 hold and let $\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}$ be bounded on [a,b]. For u1,u2 ∈ (0,1), $({\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1),{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}(u_2)),$, is approximately bivariate normally distributed with,

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1)~N(\mathit{\text{PPV}}(u),{\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1)}^2),$$

and

$${\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}\left(u_2\right)~N\left(\mathit{\text{NPV}}(u),{\sigma }_{{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}\left(u_2\right)}^2\right),$$

with,

$$\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1),{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}(u_2)]=\frac{{(1-\rho )}^2{u}_1(1-u_2)}{(1-u_1)u_2}\frac{f_{\overline{D}}(F^{-1}(u_1))}{f(F^{-1}(u_1))}\frac{f_{\overline{D}}(F^{-1}(u_2))}{f(F^{-1}(u_2))}\frac{(r_{D,1}\wedge r_{D,2})(1-\mathit{\text{NPV}}(u_1))\mathit{\text{PPV}}(u_2)}{n_D{r}_{D,1}{r}_{D,2}}+\frac{{\rho }^2{u}_1(1-u_2)}{(1-u_1)u_2}\frac{f_D(F^{-1}(u_1))}{f(F^{-1}(u_1))}\frac{f_D(F^{-1}(u_2))}{f(F^{-1}(u_2))}\frac{(r_{\overline{D},2}\wedge r_{\overline{D},2})\mathit{\text{NPV}}(u_1)(1-\mathit{\text{PPV}}(u_2))}{n_D{r}_{D,1}{r}_{D,2}},$$

when u₁ ≤ u₂ and,

$$\mathit{\text{CO}}\upsilon [{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1),{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}}(u_2)]={(1-\rho )}^2\frac{f_{\overline{D}}(F^{-1}(u_1))}{f(F^{-1}(u_1))}\frac{f_{\overline{D}}(F^{-1}(u_2))}{f(F^{-1}(u_2))}\frac{(r_{D,1}\wedge r_{D,2})(1-\mathit{\text{NPV}}(u_2))\mathit{\text{PPV}}(u_1)}{n_D{r}_{D,1}{r}_{D,2}}+{\rho }^2\frac{f_D(F^{-1}(u_1))}{f(F^{-1}(u_1))}\frac{f_D(F^{-1}(u_2))}{f(F^{-1}(u_2))}\frac{(r_{\overline{D},2}\wedge r_{\overline{D},2})\mathit{\text{NPV}}(u_2)(1-\mathit{\text{PPV}}(u_1))}{n_D{r}_{D,1}{r}_{D,2}},$$

when u₂ ≤ u₁, where ${\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1)}^2$ and ${\sigma }_{{\widehat{\mathit{\text{NPV}}}}_{r_{D,2},r_{\overline{D},2}(u_2)}}^2$ are defined as in Corollary 3.8.

The case of a point on the PPV curve and a point on the NPV curve is presented for simplicity but Corollary 3.9 can be extended to an arbitrary vector of points on the PPV and NPV curves. Corollary 3.9 has obvious practical implications. It is not uncommon to classify the bottom u₁ × 100% of the population as “low-risk”, the top (1 —u₂) × 100% of the population as “high-risk” and the remainder of the population as “moderate-risk”. In this case, one would be interested in the NPV of the low-risk group and the PPV of the high-risk group. Corollary 3.9 provides the joint convergence of these two estimates.

Finally, we note that asymptotic results for the fixed-sample empirical PPV and NPV curves indexed by the percentile value of the marker distribution can be derived as a special case of the results in this section. It is immediate from Theorem 3.6 and Corollary 3.7 that the fixed-sample empirical PPV and NPV curves converge to the sum of independent Brownian Bridges by letting r_D and r_D̂ both equal 1. Furthermore, Corollary 3.8 provides a normal approximation for the fixed-sample empirical estimate of a point on the PPV or NPV curve for the special case when J = 1.

4. Finite Sample Properties

A simulation study was completed to assess the finite sample properties of the results in Theorems 3.1, 3.5 and 3.6. We simulated 10,000 studies with n_D̂ controls and n_D cases. Biomarker values for the controls were drawn from a standard normal distribution and biomarker values for the cases were drawn from a normal distribution with mean and standard deviation equal to 1. A prevalence of .2 was used for estimation of the PPV curve. Figure 1 presents the true ROC and PPV curves for this scenario. For each realization, we calculated Rr_D,r_D̂(t), Pr_D,r_D̂(t) and Pr_D,r_D̂(u) and evaluated the expected value, normality and covariance for various combinations of r_D, r_D̂ and t or u. Normality was evaluated by providing a summary of information found in a normal q-q plot. Instead of providing the entire plot, we provide the (simulated) probability of being less than the 5th, 25th, 50th, 75th and 95th percentile of a normal distribution with variance derived using the results in Theorems 3.1, 3.5 and 3.6. Similarly, the simulated covariance matrices were compared to the theoretical covariance matrices derived using the results in Theorems 3.1, 3.5 and 3.6.

Fig 1.

True ROC and PPV curves for the scenario considered in Section 4.

Table 1 presents simulation results for Rr_D,r_D̂(t). The expected value was close to 0 in all cases with only a small amount of bias observed when t = .2. The probability of being less than the theoretical 5th and 95th percentile was close to the nominal value for all sample sizes, while the probability of being less than the 25th, 50th and 75th percentile was less than the nominal value with 50 cases and 50 controls but approached the correct values as sample size increases. The observed variance and covariance were less than expected with 50 cases and 50 controls but the observed covariance matrix approached the theoretical covariance matrix in larger sample sizes. This phenomenon is likely due to the sample space for ROC(t) being restricted to the unit interval. $\widehat{\mathit{\text{ROC}}}(t)$ is less likely to equal 0 or 1 as sample size increases and the normal approximation will be more accurate. Similar results were observed for Pr_D,r_D̂(t) and Pr_D,r_D̂(u) but were omitted for brevity.

Table 1.

Simulation results to evaluate the finite sample properties of Theorem 3.1. Presented are the expected value, simulated probability of being less than 5th, 25th, 50th, 75th and 95th percentile of the normal distribution, the simulated covariance matrix and the theoretical covariance matrix for Rr_D,r_D̂ (t). 10000 simulations were performed for each scenario.

	Mean	5th %tile	25th %tile	50th %tile	75th %tile	95th %tile	Observed				Theoretical
							Covariance Matrix				Covariance Matrix
					nD = 50,		nD̂ = 50
R0.4,0.7(0.4)	0.01	0.05	0.17	0.46	0.63	0.98	0.1	0.117	0.079	0.103	0.104	0.129	0.081	0.104
R1,1(0.4)	0.02	0.07	0.2	0.44	0.74	0.97		0.318	0.104	0.262		0.322	0.104	0.26
R0.4,0.7(0.2)	0.03	0.04	0.22	0.47	0.73	0.96			0.161	0.201			0.171	0.225
R1,1(0.2)	0.05	0.04	0.2	0.47	0.68	0.93				0.544				0.563
					nD = 100,		nD̂ = 100
R0.4,0.7(0.4)	0.01	0.05	0.21	0.41	0.78	0.97	0.101	0.12	0.08	0.102	0.104	0.129	0.081	0.104
R1,1(0.4)	0.02	0.05	0.24	0.48	0.76	0.96		0.318	0.104	0.26		0.322	0.104	0.26
R0.4,0.7(0.2)	0.03	0.04	0.2	0.45	0.73	0.95			0.164	0.205			0.171	0.225
R1,1(0.2)	0.05	0.04	0.23	0.47	0.73	0.95				0.55				0.563
					nD = 200,		nD̂ = 200
R0.4,0.7(0.4)	0.01	0.06	0.22	0.44	0.7	0.96	0.104	0.121	0.081	0.102	0.104	0.129	0.081	0.104
R1,1(0.4)	0.02	0.05	0.25	0.48	0.72	0.95		0.317	0.104	0.259		0.322	0.104	0.26
R0.4,0.7(0.2)	0.03	0.04	0.25	0.5	0.7	0.94			0.168	0.212			0.171	0.225
R1,1(0.2)	0.05	0.05	0.23	0.46	0.72	0.95				0.555				0.563

5. Application

The results of Section 3 provide fundamental theory that allows existing group sequential methodology to be applied to summaries of the ROC, PPV and NPV curves. In this section, we present an example of how these results can be used to design group sequential diagnostic biomarker studies. Our application is presented in the context of a study to evaluate the diagnostic accuracy of des-gamma carboxyprothrombin (DCP), a novel biomarker for the early detection of hepatocellular carcinoma (HCC). A multi-center study was completed to compare the diagnostic accuracy of DCP to that of alpha-fetoprotein (AFP), the most widely used biomarker for the detection of HCC (Marrero et al., 2009) but in our application we will only consider the design of a study to compare DCP to historical levels of diagnostic accuracy for AFP.

We consider a study to evaluate the predictive accuracy of DCP using the following novel design that makes use of the joint asymptotic theory for the PPV and NPV curve derived in Section 3.8. Assume that the prevalence of HCC in the population of interest is 0.2. In this case, one might call the bottom 60% percent of biomarker values “negative”, the top 10% of the biomarker values “positive” and refer the remaining subjects for further evaluation. Under this scenario, we would desire a high NPV for negative test results, NPV(0.6), and a high PPV for positive test results, PPV(0.9). The NPV(0.6) for AFP is 0.92 and the PPV(0.9) is 0.82. To determine if DCP improves on the predictive accuracy of AFP, we would test the hypothesis,

$$H_0:\mathit{\text{N PV}}\left(0.6\right)\le 0.9\mathit{\text{or PPV}}\left(0.9\right)\le 0.8,$$

versus,

$$H_a:\mathit{\text{NPN}}\left(0.6\right)>0.9\mathit{\text{and PPV}}\left(0.9\right)>0.8,$$

using the test statistics, Z_NPV(u1) and Z_PPV(u2), where Z_NPV(u1) is defined as

$$Z_{\mathit{\text{NPV}}(u_1)}=\frac{N\widehat{P}V(0.6)-\mathit{\text{NPV}}{(0.6)}_0}{{\sigma }_{\mathit{\text{NPV}}{\left(0.6\right)}_0}},$$

and Z_PPV(u2) is defined in an analogous fashion.

We consider a group sequential design using the error spending approach proposed by Hwang, Shih and De Cani (1990). The overall null hypothesis will only be rejected if the null hypotheses for both NPV (0.6) and PPV (0.9) are rejected. In the context of a group sequential study, this means that the study will stop early to reject the null hypothesis if Z_{NPV (u1)} and Z_{PPV (u2)} both cross the boundary for rejecting the null hypothesis but the study will stop early for futility if either Z_NPV(u1) or Z_PPV(u2) cross the futility boundary. This implies that we do not need to adjust the type-I error rate to account for multiple endpoints but we do need to consider the joint probability of rejecting the null hypothesis when determining the power.

The sample size for our study is chosen to achieve 90% power under the alternative hypothesis NPV(0.6) = 0.95 and PPV(0.9) = 0.90. A closed-form formula for determining the required sample size is not available. Instead, the sample size for a fixed sample design is derived by numerically solving,

$$P(Z_{\mathit{\text{NPV}}(u_1)}>Z_{1-\alpha /2},\mathit{\text{PPV}}(u_2)>Z_{1-\alpha /2}|\mathit{\text{NPV}}(u_1)=0.95,\mathrm{PPV}(u_2)=0.90),$$

for n_D, where the joint distribution of Z_NPV(u1) and Z_PPV(u2) is derived by applying the delta method to the joint asymptotic normal distribution of ${\widehat{\mathit{\text{NPV}}}}_{r_D,r_{\overline{D}}}(u_1)$ and ${\widehat{\mathit{\text{PPV}}}}_{r_D,r_{\overline{D}}}(u_2)$ found in Corollary 3.9. Assuming a one-to-one ratio of cases to controls, 702 cases are required to achieve 90% power under the alternative hypothesis. This sample size must be multiplied by an inflation factor to determine the maximum sample size for a group sequential design (i.e. the sample size if the study does not stop at the interim analyses) in order for the group sequential design to maintain the same type-I error rate and power as the fixed-sample design (Jennison and Turnbull, 2000). Using the gsDesign package in R, we find that the maximum sample size for group sequential studies with two, three and four stopping times are 724, 737 and 745 cases, respectively. However, as illustrated in the simulation which follows, the actual sample sizes required in group sequential studies are generally smaller than these maximum values.

Table 2 presents simulation results using a fixed-sample design and group sequential designs with two, three and four stopping times. Biomarker values for the controls were simulated from a standard normal distribution and biomarker values for the cases were simulated from a normal distribution with mean and variance chosen to achieve the desired value of NPV(0.6) and PPV(0.9). The advantages of group sequential designs are clear. The group sequential designs have similar type-I error rate and power to the fixed-sample design but with substantially smaller expected sample sizes in all scenarios.

Table 2.

Simulation results to evaluate the operating characteristics of a study to evaluate the predictive accuracy of DCP using a fixed-sample design and group sequential designs with two, three or four stopping times. Presented are the probability of rejecting the null hypothesis and expected sample size under the null and alternative hypotheses. 10000 simulations were performed for each scenario.

Stopping Times	NPV (0.6) = 0.90		NPV (0.6) = 0.95		NPV (0.6) = 0.90		NPV (0.6) = 0.95
	PPV (0.9) = 0.80		PPV (0.9) = 0.80		PPV (0.9) = 0.90		PPV (0.9) = 0.90
	P(reject)	E(nD)	P(reject)	E (nD)	P(reject) 0.026	E(nD)	P(reject)	E(nD)
J = 1	0.003	702	0.03	702	0.026	702	0.917	702
J = 2	0.004	432	0.026	492.4	0.024	489.5	0.924	624.5
J = 3	0.004	367.4	0.022	431.3	0.023	433	0.917	580.1
J = 4	0.002	340	0.023	410.7	0.024	417.2	0.911	571.1

6. Discussion

In this paper, we derived asymptotic properties of the sequential empirical ROC, PPV and NPV curves. We first extended the work of Hsieh and Turnbull (1996) to the sequential empirical ROC curve and used these results to develop distribution theory for summaries of the sequential empirical ROC curve. Next, we considered asymptotic theory for the sequential empirical PPV curve indexed by the FPF and percentile value in the entire population. These results were used to develop distribution theory for summaries of the sequential empirical PPV curve. Asymptotic theory for the fixed-sample PPV curve, which was previously unavailable, was developed as a special case.

This work was motivated by the desire to design group sequential diagnostic biomarker studies. In Section 5, we illustrated how our results can be used to design group sequential diagnostic biomarker studies. Our simulation results clearly illustrate the advantages of group sequential designs. In both cases, the group sequential designs have similar type-I error rate and power than the fixed-sample designs but with substantially smaller expected sample size.

An advantage to our approach is that we are able to investigate the joint behavior of multiple points on the ROC and PPV curve. The primary end-point of a diagnostic biomarker study may be a single point on the ROC or PPV curve but other points on the ROC or PPV curve may also be of interest. The results of Theorems 3.1, 3.5 and 3.6 allow us to apply existing group sequential methodology for analyzing multiple endpoints to scenarios where multiple points on the ROC or PPV curve are of interest in a group sequential diagnostic biomarker study (Liu and Hall, 2001).

We considered estimation of the sequential empirical ROC and PPV curve under case-control sampling. The asymptotic properties of the sequential empirical ROC and PPV curve under other sampling schemes are also of interest. We are currently working on extending the results of this paper to estimation of the sequential empirical ROC and PPV curve under cohort and nested case-control sampling.

The theory developed in this paper applies to sequential testing of the diagnostic accuracy of a continuous test. In many cases, diagnostic tests take the form of multi-level ordinal data (cancer staging, for example). Methods exist extending the ROC curve to ordinal data (Dorfman and Alf, 1960) but further work is needed to verify that group sequential methods can be applied in these settings.

Response adaptive clinical trials have been proposed as a means to provide greater flexibility when designing therapeutic clinical trials. Response adaptive clinical trials adjust the design characteristics of the study (sample size, percent randomized to each group, etc.) in response to outcomes for subjects enrolled earlier in the study. Recently, Zhu and Hu (2010) showed that a class of test statistics from a response adaptive clinical trial converges to Brownian Motion when considered sequentially (similar to what we have shown for the emprical ROC, PPV and NPV curves), which allows existing group sequential methodology to be applied to response adaptive clinical trials. Future work will be needed to consider how response adaptive designs can be applied in the setting of group sequential diagnostic biomarker studies.

Appendix A.

Supplementary Results for Section 3

A.1. Supplementary Results for Section 3.1

Proof of Theorem 3.1

First, note that

$$\begin{array}{l}{n}_D^{-1/2}\left[n_D{r}_D\right](R\widehat{O}{C}_{r_D,r_{\overline{D}}}(t)-\mathit{\text{ROC}}(t))=n_D^{-1/2}\left[n_D{r}_D\right]\left({\widehat{S}}_{r_D,r_D}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-S_D({\widehat{S}}_{\overline{D}}^{-1}\left(t\right))\right)\\ =n_D^{-1/2}\left[n_D{r}_D\right]\left({\widehat{S}}_{r_D,r_D}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right))-S_D({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right))\right)\\ +n_D^{-1/2}\left[n_D{r}_D\right]\left(S_D({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right))-S_D({\widehat{S}}_{\overline{D}}^{-1}\left(t\right))\right).\end{array}$$

The first term converges to a Kiefer process. We note that

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}|F_{\overline{D}}({\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|=\frac{n_{\overline{D}}}{\left[n_{\overline{D}}d\right]}\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}\frac{\left[n_{\overline{D}}d\right]}{n_{\overline{D}}}|F_{\overline{D}}({\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|\le \frac{n_{\overline{D}}}{\left[n_{\overline{D}}d\right]}\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}\frac{\left[n_{\overline{D}}{r}_{\overline{D}}\right]}{n_{\overline{D}}}|F_{\overline{D}}({\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|.$$

Therefore

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}\left|F_{\overline{D}}\right({\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|{\to }_{a.s.}0$$ (A.1)

by the Glivenko-Cantelli Theorems (1.51 and 1.52 in Csörgõ and Szyszkowicz (1998)) and because $\frac{n_{\overline{D}}}{\left[n_{\overline{D}}d\right]}\to \frac{1}{d}$. Furthermore, $F_{\overline{D}}^{-1}(t)$ will be continuous by A1-A3 and will be uniformly continuous on [a, b]. Therefore,

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}|{\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}(t)-F_{\overline{D}}^{-1}(t)|{\to }_{a.s.}0.$$ (A.2)

We note that due to the continuity of F_D̄(x), $S_{\overline{D}}^{-1}(t)=F_{\overline{D}}^{-1}(1-t)$ and therefore (A.2) also applies to $S_{\overline{D}}^{-1}(t)$. From corollary 1.A in Csörgõ and Szyszkowicz (1998), (A.2) and the uniform continuity of the Kiefer process, we have

$$n_D^{-1/2}\left[n_D{r}_D\right]\left({\widehat{S}}_{D,r_D}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right))-S_D({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right))\right)\to {}_d{K}_1\left(\mathit{\text{ROC}}\left(t\right),r_D\right).$$ (A.3)

The second term can be re-written as

$$\begin{array}{l}{n}_D^{-1/2}\left[n_D{r}_D\right]\left(S_D\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-S_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)\right)\\ =\frac{n_D^{-1/2}\left[n_D{r}_D\right]}{n_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]}\frac{\left(S_D\left(S_{\overline{D}}^{-1}\left(S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)\right)\right)-S_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)\right)}{S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-t}{n}_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]\left(S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-{\widehat{S}}_{\overline{D},r_{\overline{D}}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)\right)\\ +\frac{n_D^{-1/2}\left[n_D{r}_D\right]}{n_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]}\frac{\left(S_D\left(S_{\overline{D}}^{-1}\left(S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)\right)\right)-S_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)\right)}{S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-t}{n}_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-t\right).\end{array}$$

By the mean value theorem, there exists a $S_{\overline{D}}({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))$ between $S_{\overline{D}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))$ and t such that

$$\frac{S_D\left(S_{\overline{D}}^{-1}\left(S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)\right)\right)-S_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)}{S_{\overline{D}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-t}=\frac{f_D\left(S_{\overline{D}}^{-1}\left(S_{\overline{D}}\left({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)\right)\right)}{f_{\overline{D}}\left(S_{\overline{D}}^{-1}\left(S_{\overline{D}}({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))\right)\right)}.$$

From (A.1), we know that $S_{\overline{D}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t)){\to }_{a.s.}t$, uniformly for t ∈ [a,b], r_D ∈ [c, 1] and r_D̄ ∈ [d, 1], and, therefore, $S_{\overline{D}}({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t)){\to }_{a.s.}t$, uniformly for t ∈ [a,b], r_D ∈ [c, 1] and r_D̄ ∈ [d, 1]. This, along with the uniform continuity of $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$, allows us to conclude that

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}|\frac{f_D\left(S_{\overline{D}}^{-1}\right(S_{\overline{D}}({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))\left)\right)}{f_{\overline{D}}\left(S_{\overline{D}}^{-1}\right(S_{\overline{D}}({\stackrel{\sim }{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))\left)\right)}-\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}|{\to }_{a.s.}0,$$

which implies,

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}|\frac{S_D\left(S_{\overline{D}}^{-1}\right(S_{\overline{D}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))\left)\right)-S_D(S_{\overline{D}}^{-1}(t))}{S_{\overline{D}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t}-\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}|{\to }_{a.s.}0.$$ (A.4)

For all r_D̄ ∈ [d, 1],

$$\underset{a\le t\le b}{sup}\left|{\widehat{S}}_{\overline{D},r_{\overline{D}}}\right({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|{\le }_{a.s.}\frac{1}{\left[n_{\overline{D}}{r}_{\overline{D}}\right]}.$$

Therefore,

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}{n}_{\overline{D}}^{-1/2}[n_{\overline{D}}{r}_{\overline{D}}]|{\widehat{S}}_{\overline{D},r_{\overline{D}}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-t|{\le }_{a.s.}\frac{1}{n_{\overline{D}}^{1/2}},$$

and

$$\underset{c\le r_D\le 1}{sup}\underset{d\le r_{\overline{D}}\le 1}{sup}\underset{a\le t\le b}{sup}{n}_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]|{\widehat{S}}_{\overline{D},r_{\overline{D}}}\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\right(t\left)\right)-t|{\le }_{a.s.}0.$$ (A.5)

From corollary 1.A in Csörgõ and Szyszkowicz (1998), (A.2) and the uniform continuity of the Kiefer process, we have

$$n_{\overline{D}}^{-1/2}[n_{\overline{D}}{r}_{\overline{D}}](S_{\overline{D}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))-{\widehat{S}}_{\overline{D},r_{\overline{D}}}({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}(t))\left){\to }_d{K}_2\right(t,r_{\overline{D}}).$$ (A.6)

By (A.4), (A.5), (A.6) and noting that $\frac{n_D^{-1/2}[n_D{r}_D]}{n_{\overline{D}}^{-1/2}[n_{\overline{D}}{r}_{\overline{D}}]}\to {\lambda }^{1/2}\frac{r_D}{r_{\overline{D}}}$, we conclude that

$$n_D^{-1/2}\left[n_D{r}_D\right]\left(S_D\left({\widehat{S}}_{\overline{D},r_{\overline{D}}}^{-1}\left(t\right)\right)-S_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)\right)\to {}_d{\lambda }^{1/2}\frac{r_D}{r_{\overline{D}}}\left(\frac{f_D\left(S_{\overline{D}}^{-1}\left(t\right)\right)}{f_{\overline{D}}\left(S_{\overline{D}}^{-1}\left(t\right)\right)}\right)K_2\left(t,r_{\overline{D}}\right).$$ (A.7)

Summing (A.3) and (A.7) gives the desired result.

A.2. Supplementary Results for Section 3.2

Corollary A.1

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. For t₁, t₂, …, t_J ∈ (0,1), r_D,1, r_D,2, …, r_D,J ∈ (0,1] and r_D̄,1, r_D̄,2, …, r _D̄,J ∈ (0,1], a vector of arbitrary points on the sequential empirical PPV curve, $({\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(t_1),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}(t_2),\dots ,{\widehat{\mathit{\text{PPV}}}}_{r_{D,J},r_{\overline{D},J}}(t_J))$, is approximately multivariate normal with,

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)\sim N(\mathit{\text{PPV}}(t_j),{\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)}^2)j=1,2,\dots ,J,$$

$${\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(t_j\right)}^2={\left(\frac{t\left(1-\rho \right)\rho }{{\left(\mathit{\text{ROC}}\left(t\right)\rho +t\left(1-\rho \right)\right)}^2}\right)}^2{\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}\left(t_j\right)}^2,$$

and

$$\mathit{\text{Cov}}[{\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(t_i),{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)]=\left(\frac{t_i(1-\rho )\rho }{(\mathit{\text{ROC}}(t_i)\rho +t_i{(1-\rho ))}^2}\right)\left(\frac{t_j(1-\rho )\rho }{(\mathit{\text{ROC}}(t_j)\rho +t_j{(1-\rho ))}^2}\right)\mathit{\text{Cov}}[{\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}}(t_i),{\widehat{\mathit{\text{ROC}}}}_{r_{D,J},r_{\overline{D},J}}(t_j)],$$

where ${\sigma }_{{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)}^2$ and $\mathit{\text{Cov}}[{\widehat{\mathit{\text{ROC}}}}_{r_{D,i},r_{\overline{D},i}}(t_i),{\widehat{\mathit{\text{ROC}}}}_{r_{D,j},r_{\overline{D},j}}(t_j)]$ are as defined in Corollary 3.3.

Proof. Immediate from Theorem 3.5.

Corollary A.2

Assume A1-A4 hold and let $\frac{f_D(S_{\overline{D}}^{-1}(t))}{f_{\overline{D}}(S_{\overline{D}}^{-1}(t))}$ be bounded on [a, b]. For t ∈ (0,1) and J stopping times $({\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(t),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}(t),\dots ,{\widehat{\mathit{\text{PPV}}}}_{r_{D,J},r_{\overline{D},J}}(t))$, is approximately multivariate normal with,

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(t)\sim N(\mathit{\text{PPV}}(t),{\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}(t)}^2)i=1,2,\dots ,J$$

and

$$\mathit{\text{cov}}\left[{\widehat{\mathit{\text{PPV}}}}_{r_{D,i},r_{\overline{D},i}}\left(t\right),{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(t\right)\right]=\mathit{\text{var}}\left[{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(t\right)\right]={\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(t\right)}^2,r_i\le r_j$$

where ${\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}(t)}^2$ is defined as in Corollary A.1.

Proof. Immediate from Corollary A.1.

A.3. Supplementary Results for Section 3.3

Proof of Theorem 3.6

The proof of Theorem 3.6 follows the proofs found in Pyke and Shorack (1968). First, note that,

$$n_D^{-1/2}[n_D{r}_D]({\widehat{S}}_{D,r_D}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-S_D(F^{-1}(u)))=n_D^{-1/2}[n_D{r}_D](F_D(F^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)))+n_D^{-1/2}[n_D{r}_D](F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-{\widehat{F}}_{D,r_D}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))).$$

The first term can be rewritten as,

$$n_D^{-1/2}[n_D{r}_D](F_D(F^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)))>\frac{F_D(F^{-1}\left(F\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right))-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}=n_D^{-1/2}[n_D{r}_D](F_D(F^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)))+\frac{F_D(F^{-1}(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))))-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}\rho n_D^{-1/2}[n_D{r}_D]({\widehat{F}}_{D,r_D}^{-1}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)))+\frac{n_D^{-1/2}[n_D{r}_D]}{n_D^{-1/2}[n_D{r}_D]}\frac{F_D(F^{-1}(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))))-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}(1-\rho )n_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]({\widehat{F}}_{\overline{D},r_{\overline{D}}}^{-1}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_{\overline{D}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)))$$

We begin by showing that $F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ converges to u uniformly,

$$\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\left|F\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u|\le \underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\left|F\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|+\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\left|F\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u|.$$

We note that,

$$\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\left|F\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|\le \frac{n_D}{\left[n_{D}c\right]}\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\frac{[n_D{r}_D]}{n_D}\left|F_D\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|+\frac{n_{\overline{D}}}{\left[n_{\overline{D}}d\right]}\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}\frac{\left[n_{\overline{D}}{r}_{\overline{D}}\right]}{n_{\overline{D}}}\left|F_{\overline{D}}\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_{\overline{D},r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|{\to }_{a.s.}0,$$

by the Glivenko-Cantelli Theorems (1.51 and 1.52 in Csörgõ and Szyszkowicz (1998)), along with the fact that $\frac{n_D}{\left[n_{D}c\right]}\to \frac{1}{c}$ and $\frac{n_{\overline{D}}}{\left[n_{\overline{D}}d\right]}\to \frac{1}{d}$. For all r_D, r_D̄ ∈ (0,1] × (0,1],

$$\underset{a\le u\le b}{\text{sup}}|u-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))|{\le }_{a.s.}(\frac{\rho }{\left[r_D{n}_D\right]}\vee \frac{1-\rho }{\left[n_{\overline{D}}{r}_{\overline{D}}\right]}).$$

Therefore,

$$\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}|u-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|{\le }_{a.s.}(\frac{\rho }{\left[n_{D}c\right]}\vee \frac{1-\rho }{\left[n_{\overline{D}}d\right]})\to 0,$$

which implies that,

$$\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}|F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u|{\to }_{a.s.}0.$$ (A.8)

We note that (A.8) also implies that $F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ and $F_{\overline{D}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ converge uniformly to F_D (F⁻¹ (u)) and F_D̄(F⁻¹ (u)), respectively, which can be seen by noting that the difference between $F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ and F_D(F⁻¹ (u)) will always have the same sign as the difference between $F_{\overline{D}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ and F_D̄(F⁻¹ (u)).

By the mean value theorem, there exists $F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ between u and $F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))$ such that,

$$\frac{F_D\left(F^{-1}\right(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}=\frac{f_D\left(F^{-1}\right(F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)}{f\left(F^{-1}\right(F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)}.$$

The uniform continuity of $\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}$ combined with the fact that $F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u)){\to }_{a.s.}$ u uniformly, allows us to conclude,

$$\underset{c\le r_D\le 1}{\text{sup}}\underset{d\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}|\frac{f_D\left(F^{-1}\right(F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)}{f\left(F^{-1}\right(F({\stackrel{\sim }{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)}-\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}|{\to }_{a.s.}0.$$ (A.9)

For all r_D, r_D̄ ∈ (0, 1] × (0,1],

$$\underset{a\le u\le b}{\text{sup}}{n}_D^{-1/2}[n_D{r}_D]|u-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|{\le }_{a.s.}(\frac{\rho }{n_D^{-1/2}}\vee \frac{[n_D{r}_D]}{\left[n_{\overline{D}}{r}_{\overline{D}}\right]}\frac{1-\rho }{n_D^{-1/2}}).$$

Therefore, as n_D → ∞ and n_D̄ → ∞,

$$\underset{0\le r_D\le 1}{\text{sup}}\underset{0\le r_{\overline{D}}\le 1}{\text{sup}}\underset{a\le u\le b}{\text{sup}}{n}_D^{-1/2}[n_D{r}_D]|u-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right|{\to }_{a.s.}0.$$

Combining this result with (A.9) allows us to conclude that,

$$\frac{F_D\left(F^{-1}\right(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}{n}_D^{-1/2}[n_D{r}_D](u-{\widehat{F}}_{r_D,r_{\overline{D}}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))){\to }_{a.s.}0.$$

Corollary 1.A in Csörgõ and Szyszkowicz (1998), (A.9) and the uniform continuity of the Kiefer process allow us to conclude,

$$\frac{F_D\left(F^{-1}\right(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}\rho n_D^{-1/2}[n_D{r}_D]\left({\widehat{F}}_{D,r_D}\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right){\to }_d\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}\rho K_1\left(F_D\right(F^{-1}(u)),r_D),$$ (A.10)

and,

$$\frac{n_D^{-1/2}[n_D{r}_D]}{n_D^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]}\frac{F_D\left(F^{-1}\right(F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))\left)\right)-F_D(F^{-1}(u))}{F({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-u}(1-\rho )n_{\overline{D}}^{-1/2}\left[n_{\overline{D}}{r}_{\overline{D}}\right]\left({\widehat{F}}_{\overline{D},r_{\overline{D}}}\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_{\overline{D}}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right){\to }_d\sqrt{\lambda }\frac{r_D}{r_{\overline{D}}}\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}(1-\rho )K_2\left(F_{\overline{D}}\right(F^{-1}(u)),r_D).$$ (A.11)

The second term converges in distribution to a Kiefer process,

$$n_D^{-1/2}[n_D{r}_D]\left(F_D\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-{\widehat{F}}_{D,r_D}({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right)=-n_D^{-1/2}[n_D{r}_D]\left({\widehat{F}}_{D,r_D}\right({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u))-F_D({\widehat{F}}_{r_D,r_{\overline{D}}}^{-1}(u)\left)\right){\to }_d-K_1\left(F_D\right(F^{-1}(u)),r_D),$$ (A.12)

by Corollary 1.A in Csörgõ and Szyszkowicz (1998). Summing (A.10), (A.11)and (A.12) gives the desired result.

Corollary A.3

Assume A1 - A4 hold and let $\frac{f_D(F^{-1}(u))}{f(F^{-1}(u))}$ be bounded on [a, b]. For u₁, u₂, …, u_J ∈ (0,1), r_D,1, r_D,2, …, r_D,J ∈ (0,1] and r_D̄,1, r_{D̄, 2}, …, r _{D̄, J} ∈ (0,1], a vector of arbitrary points on the sequential empirical PPV curve, $({\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}(u_2),\dots ,{\widehat{\mathit{\text{PPV}}}}_{r_{D,J},r_{\overline{D},J}}(u_J))$, is approximately multivariate normal with,

$${\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(u_j\right)\sim N\left(\mathit{\text{PPV}}\left(u_j\right),{\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,j},r_{\overline{D},j}}\left(u_j\right)}^2\right)j=1,2,\dots ,J$$

with,

$$\begin{array}{l}\mathit{\text{Co}}\upsilon \left[{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}\left(u_1\right),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}\left(u_2\right)\right]\\ =\frac{{\left(1-\rho \right)}^2{u}_1}{\left(1-u_1\right)}\frac{f_{\overline{D}}\left(F^{-1}\left(u_1\right)\right)}{f\left(F^{-1}\left(u_1\right)\right)}\frac{f_{\overline{D}}\left(F^{-1}\left(u_2\right)\right)}{f\left(F^{-1}\left(u_2\right)\right)}\frac{\left(r_{D,1}\wedge r_{D,2}\right)\left(1-\mathit{\text{NPV}}\left(u_1\right)\right)\mathit{\text{PPV}}\left(u_2\right)}{n_D{r}_{D,1}{r}_{D,2}}\\ +\frac{{\rho }^2{u}_1}{\left(1-u_1\right)}\frac{f_D\left(F^{-1}\left(u_1\right)\right)}{f\left(F^{-1}\left(u_1\right)\right)}\frac{f_D\left(F^{-1}\left(u_2\right)\right)}{f\left(F^{-1}\left(u_2\right)\right)}\frac{\left(r_{\overline{D},2}\wedge r_{\overline{D},2}\right)\mathit{\text{NPV}}\left(u_1\right)\left(1-\mathit{\text{PPV}}\left(u_2\right)\right)}{n_D{r}_{D,1}{r}_{D,2}},\end{array}$$

when u₁ ≤ u₂ and,

$$\begin{array}{l}\mathit{\text{Co}}\upsilon \left[{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}\left(u_1\right),{\widehat{\mathit{\text{PPV}}}}_{r_{D,2},r_{\overline{D},2}}\left(u_2\right)\right]\\ =\frac{{\left(1-\rho \right)}^2{u}_2}{\left(1-u_2\right)}\frac{f_{\overline{D}}\left(F^{-1}\left(u_1\right)\right)}{f\left(F^{-1}\left(u_1\right)\right)}\frac{f_{\overline{D}}\left(F^{-1}\left(u_2\right)\right)}{f\left(F^{-1}\left(u_2\right)\right)}\frac{\left(r_{D,1}\wedge r_{D,2}\right)\left(1-\mathit{\text{NPV}}\left(u_2\right)\right)\mathit{\text{PPV}}\left(u_1\right)}{n_D{r}_{D,1}{r}_{D,2}}\\ +\frac{{\rho }^2{u}_2}{\left(1-u_2\right)}\frac{f_D\left(F^{-1}\left(u_1\right)\right)}{f\left(F^{-1}\left(u_1\right)\right)}\frac{f_D\left(F^{-1}\left(u_2\right)\right)}{f\left(F^{-1}\left(u_2\right)\right)}\frac{\left(r_{\overline{D},2}\wedge r_{\overline{D},2}\right)\mathit{\text{NPV}}\left(u_2\right)\left(1-\mathit{\text{PPV}}\left(u_1\right)\right)}{n_D{r}_{D,1}{r}_{D,2}},\end{array}$$

when u₂ ≤ u₁, where ${\sigma }_{{\widehat{\mathit{\text{PPV}}}}_{r_{D,1},r_{\overline{D},1}}(u_1)}^2$ is defined as in Corollary 3.8.

Proof. Immediate from Theorem 3.6.