Estimation of AUC or Partial AUC under Test-Result-Dependent Sampling

Abstract

The area under the ROC curve (AUC) and partial area under the ROC curve (pAUC) are summary measures used to assess the accuracy of a biomarker in discriminating true disease status. The standard sampling approach used in biomarker validation studies is often inefficient and costly, especially when ascertaining the true disease status is costly and invasive. To improve efficiency and reduce the cost of biomarker validation studies, we consider a test-result-dependent sampling (TDS) scheme, in which subject selection for determining the disease state is dependent on the result of a biomarker assay. We first estimate the test-result distribution using data arising from the TDS design. With the estimated empirical test-result distribution, we propose consistent nonparametric estimators for AUC and pAUC and establish the asymptotic properties of the proposed estimators. Simulation studies show that the proposed estimators have good finite sample properties and that the TDS design yields more efficient AUC and pAUC estimates than a simple random sampling (SRS) design. A data example based on an ongoing cancer clinical trial is provided to illustrate the TDS design and the proposed estimators. This work can find broad applications in design and analysis of biomarker validation studies.

1 Introduction

The area under the ROC curve (AUC) and partial area under the ROC curve (pAUC) are two summary measures of accuracy of a diagnostic test or a biomarker [1, 2]. We use the terms “biomarker” and “diagnostic test” in a generic sense to mean any measure that could signal the onset of a disease or a disease condition. Let D be the true disease status with D = 1 for presence of disease and D = 0 for absence of disease. Let Y be a continuous measure for a biomarker, where a higher value is indicative of increased severity of the disease. The true and false positive rates at the threshold c are TPR(c) = Pr(Y ≥ c|D = 1) = Pr(Y_D ≥ c) and FPR(c) = Pr(Y ≥ c|D = 0) = Pr(Y_D̄ ≥ c) respectively, where D and D̄ in indices denote the sets {D = 1} and {D = 0}. An ROC curve is a plot of the entire set of possible true and false positive rates ROC(c) = {(FPR(c), TPR(c)), c ∈ (−∞, ∞)}. The AUC is defined over the entire ROC curve as $\text{AUC}={\int }_0^1\text{TPR}({\text{FPR}}^{-1}(t))\text{d}t$. The larger the AUC value, the more accurate the diagnostic test is. However, in medical research, we often know a priori that a particular test would be useful only if its false-positive rate is in a restricted range of rates. Partial AUC (pAUC) is defined over the restricted range of false positive rates (t₀, t₁) with $\text{pAUC}={\int }_{t_0}^{t_1}\text{TPR}({\text{FPR}}^{-1}(t))\text{d}t$, where q₀ = FPR⁻¹(t₁) and q₁ = FPR⁻¹ (t₀).

Validation of the predictive value of a biomarker or a diagnostic test usually involves large prospective studies, in which subjects are sampled randomly from a target population. Observing the true disease status can be costly, invasive, or time consuming and the positivity rates for many markers can be low in the target population. Moreover, medical investigators are often interested in the performance of a diagnostic test in a narrow range of test results that corresponds to low false positive rates. For example, in cancer screening, where false positivity is associated with greater risk than benefit from aggressive intervention, partial AUC over the restricted false positive rates is of more interest [3, 4]. All of these factors make the conventional design, in which the true disease status is ascertained for a large number of randomly sampled subjects regardless of their test results, is inefficient and costly. In this article, we consider test-result-dependent sampling (TDS) as an alternative to simple random sampling (SRS) for biomarker validation. Rather than sampling all subjects randomly from the target population, TDS allows a subset of subjects to be sampled using a test-result-dependent sampling scheme. Depending on the specific questions, the investigators can select more subjects from those in the range of test results that contain the most information for statistical estimation at the design phase of the study. By oversampling subjects with the chosen ranges of test results and undersampling subjects outside of these ranges, the TDS design has potential to considerably improve efficiency of estimating AUC (or pAUC) for a given number of subjects that require their true disease status determined.

In this article, we consider test-result-dependent sampling (TDS) that consists of two sampling components, a simple random component of subjects from the targeted study population and a test-result-dependent component from the same population. As in the case of verification bias in diagnostic medicine [1, 2], the TDS design will introduce bias when standard methods are used to estimate AUC and pAUC based on TDS data. However, the existing methods of AUC and pAUC for verification bias are not applicable to the TDS data. Verification bias occurs primarily in population-based studies, in which a diagnostic test is applied to all N subjects in the cohort, yielding a test result Y for all study units. As only a subset of patients is selected to observe (verify) true disease status D, unselected patients in the cohort are missing D. When D is missing at random (MAR) [5], the selection probability of observing D is directly estimable. The bias in estimating AUC can be corrected by the inverse probability weighting method for binary, ordinal or continuous diagnostic tests (e.g. [6, 7, 8, 9, 10, 11]). In the TDS design, patients are selected to observe D depending on their ranges of test result Y, but the size of the parent cohort from which the patients are accrued and the test results of unscreened patients are unknown to the investigators. Indeed, such situations occur in most cancer biomarker validation studies, in which subjects come from hospital-based settings and randomized clinical trials and population-based study designs are not feasible. In the TDS design, for all the subjects in the selected sample, the data available for analysis are observed, both Y and D. Since there are no missing variables, the TDS design is different from the verification bias problem and the inverse probability weighting (IPW) approach for missing data is not directly applicable.

For data with a TDS design structure, standard methods of estimating AUC and pAUC yield biased results. In this article, an empirical likelihood method is used to estimate the distribution of the test result Y conditional on disease status D. Unbiased estimators for AUC and pAUC are then obtained by reweighing the standard estimators with the estimated empirical distribution. The estimators are nonparametric in that no parametric model is assumed for the relationship between the test result and the true disease status. Our method is related to the general approach of outcome-dependent sampling. Weaver and Zhou [12] and Zhou et al. [13] discussed the utility of outcome-dependent sampling schemes in estimating the coefficients of regression models for epidemiologic studies. Zhou et al. [14] and Wang and Zhou [15] studied empirical likelihood-based estimators for regression models where the sample depends on the outcome or an auxiliary variable. Outcome-dependent sampling is also used in econometrics. Cosslett [16, 17] proposed a constrained likelihood estimator for regression coefficients of a categorical regression model. Morgenthaler and Vardi [18] studied a weighted likelihood method for data arising from length-bias sampling.

For AUC and pAUC estimation, nonparametric methods are generally preferable to their parametric counterparts since they require fewer assumptions. Bamber [19] showed that AUC is the probability that a randomly observed test result in diseased subjects is higher than a randomly observed test result in non-diseased subjects, i.e. Pr(Y_D > Y_D̄). Hanley and McNeil [20] showed that the AUC estimator based on the trapezoidal rule is equivalent to the Mann-Whitney U statistic. DeLong et al. [21] and Wieand et al. [22] studied nonparametric estimation of AUC and testing of AUC differences for paired biomarkers. Partial AUC (pAUC) can be considered an average of TPR(c) across all FPR(c) where FPR(c) ∈ (t₀, t₁). Thompson and Zucchini [23], McClish [24] and Jiang et al. [25] studied parametric pAUC estimation assuming bivariate normal models. Zhang et al. [26], Dodd and Pepe [4], and He and Escobar [27] studied nonparametric pAUC estimation by extending the approach of DeLong et al. [21]. In all these papers, the data are assumed to be a simple random sample of the targeted patient population.

The rest of the paper is organized as follows. In Section 2, we specify the likelihood for data arising from a TDS design and propose a nonparametric empirical likelihood method to estimate the distribution of the test result conditional on the disease status. Section 3 presents the nonparametric AUC estimator under the TDS design and establishes its asymptotic properties. Section 4 studies the nonparametric pAUC estimator under the TDS design. Section 5 describes the finite sample properties of the proposed estimators under the TDS design. The efficiency gain of the proposed estimators under the TDS design relative to the standard estimators under the SRS design is also investigated. We illustrate in Section 6 the proposed method with an example. The article concludes with some remarks in Section 7. Proofs of the asymptotic properties for the proposed estimators are sketched in the Appendix.

2 Estimation of Test Result Distribution

We consider test-result-dependent sampling (TDS) which consists of a simple random component and a test-result-dependent component. Assume that the test result Y falls into one of K mutually exclusive intervals C_k = (a_k−1, a_k), k = 1, …, K where a_k−1 < a_k and a₀ = −∞, a_K = ∞. One observes a test-result-dependent component {Y_ki, D_ki}, i = 1, …, n_k conditional on whether Y belongs to stratum C_k as well as a simple random component {Y_0i, D_0i}, where i = 1, …, n₀. The combined sample size of the two components is $n={\sum }_{s=0}^K{n}_s$.

Denote by f(Y, D) the joint density function of Y and D. Let p_si = f (Y_si, D_si), s = 0, …, K, i = 1, …, n_s, and θ_k = Pr(Y ∈ C_k), k = 1, …, (K − 1), θ = (θ₁, …, θ_K−1)^T and ${\theta }_K=1-{\sum }_{k=1}^{K-1}{\theta }_s$. The empirical likelihood of {Y_si, D_si} under the TDS design is then

$$L(\theta ,p_{si})=\left[\prod _{i=1}^{n_0}f(Y_{0i},D_{0i})\right]\left[\prod _{k=1}^K\prod _{i=1}^{n_k}f(Y_{ki},D_{ki}\mid Y_{ki}\in C_k)\right]=\left[\prod _{s=0}^K\prod _{i=1}^{n_s}{p}_{si}\right]\left[\prod _{k=1}^{K-1}{\theta }_k^{-n_k}\right]{\theta }_K^{-n_K}.$$

Based on empirical likelihood theory [28, 29, 30], one can maximize the log(L(θ, p_si)) under the following constraints below to find estimates of θ and p_si:

$$\left\{p_{si}\ge 0,\sum _{s=0}^K\sum _{i=1}^{n_s}{p}_{si}=1,\sum _{s=0}^K\sum _{i=1}^{n_s}{p}_{si}(I(y_{si}\in C_k)-{\theta }_k)=0,k=1,\dots ,K-1\right\},$$

where I(·) is the indicator function. Using Lagrange multipliers, one can show that the maximum of log(L(θ, p_si)) is attained at

$${\widehat{p}}_{si}=\frac{1}{n_0+{\sum }_{k=1}^K{n}_k/{\widehat{\theta }}_{k}I(y_{si}\in C_k)},$$ (1)

where ${\widehat{\theta }}_K=1-{\sum }_{k=1}^{K-1}{\widehat{\theta }}_k$ and θ̂_k can be obtained by solving ∂l_p(θ)/∂θ = 0, where

$$l_p(\theta )=-\sum _{s=0}^K\sum _{i=1}^{n_s}log\left(n_0+\sum _{k=1}^K{n}_k/{\theta }_{k}I(y_{si}\in C_k)\right)-\sum _{k=1}^K(n_{k}log{\theta }_k).$$ (2)

Following Qin and Lawless [30], we can show θ̂ asymptotically follows a normal distribution. That is, under general regularity conditions, if n_s/n → ρ_s as n → ∞ where 0 < ρ_s < 1, s = 0, …, K − 1, we have $n^{1/2}{\mathrm{\Psi }}_n^{-1/2}(\widehat{\theta }-\theta )\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1)$, where ${\mathrm{\Psi }}_n={\sum }_{s=0}^K{\rho }_{s}var({\xi }_{si})$ with ξ_si is given in the Appendix.

3 Nonparametric Estimation of AUC

The AUC is defined as $A={\int }_0^1\text{TPR}({\text{FPR}}^{-1}(t))\text{d}t$. Under simple random sampling, the nonparametric AUC estimator [19, 20] is

$${\widehat{A}}^{\mathit{SRS}}=\left[\sum _{j=1}^n\sum _{i=1}^n{D}_i(1-D_j)I_{ij}\right]/\left[\sum _{j=1}^n\sum _{i=1}^n{D}_i(1-D_j)\right],$$ (3)

where I_ij = I(Y_i > Y_j), i, j = 1, …, n, and Â^SRS is known to have good properties under simple random sampling. However, Â^SRS yields biased estimates under the TDS design. To obtain unbiased AUC estimates for the TDS design, we reweight the standard AUC estimator with the estimates p̂_si in (1) and propose the following nonparametric estimator under the TDS design:

$${\widehat{A}}^{\mathit{TDS}}=\left[\sum _{j=1}^n\sum _{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)I_{ij}\right]/\left[\sum _{j=1}^n\sum _{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)\right].$$ (4)

For notational simplicity, we have replaced the double subscripts si in p̂_si with a single subscript i with i = 1, · · ·, n in p̂_i. Let $R_n(A,\theta )={\sum }_{i=1}^n{\sum }_{j=1}^n{\stackrel{\sim }{p}}_i{\stackrel{\sim }{p}}_j{D}_i(1-D_j)(I_{ij}-A)$ and

$${\stackrel{\sim }{p}}_i={[n_0+\sum _{k=1}^K{n}_k/{\theta }_{k}I(y_i\in C_k)]}^{-1}.$$ (5)

Assuming the U-process U_n(A, θ) = n^1/2 {R_n(A, θ) − E[R_n(A, θ)]} is equicontinuous, we are able to show that the proposed estimator Â^TDS for AUC converges asymptotically to a normal distribution, that is,

$$n^{1/2}{\mathrm{\Lambda }}_n^{-1/2}({\widehat{A}}^{\mathit{TDS}}-A)\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1),$$ (6)

where Λ_n is given in the Appendix.

4 Nonparametric Estimation of pAUC

Denote the partial AUC restricted in (t₀, t₁) by A_t. Let q₀ = FPR⁻¹(t₁) and q₁ = FPR⁻¹(t₀). By the definition of partial AUC, we have

$$A_t={\int }_{t_0}^{t_1}\text{TPR}({\text{FPR}}^{-1}(t))\text{d}t=Pr(Y_D>Y_{\overline{D}},Y_{\overline{D}}\in (q_0,q_1)).$$

Under simple random sampling, Dodd and Pepe (2003) proposed a nonparametric estimator for partial AUC: ${\widehat{A}}_t^{\mathit{SRS}}={(n_D{n}_{\overline{D}})}^{-1}{\sum }_i{\sum }_{j}I(Y_{Di}>Y_{\overline{D}j},Y_{\overline{D}j}\in (q_0,q_1))$, or equivalently,

$${\widehat{A}}_t^{\mathit{SRS}}=\left[\sum _{j=1}^n\sum _{i=1}^n{D}_i(1-D_j)I(Y_i>Y_j,Y_j\in (q_0,q_1))\right]/\left[\sum _{j=1}^n\sum _{i=1}^n{D}_i(1-D_j)\right],$$ (7)

and they illustrated that ${\widehat{A}}_t^{\mathit{SRS}}$ is consistent and more robust than other estimators that make parametric assumptions. We propose the following nonparametric pAUC estimator for the TDS design:

$${\widehat{A}}_t^{\mathit{TDS}}=\left[\sum _{j=1}^n\sum _{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)I_{ij}^{\prime }\right]/\left[\sum _{j=1}^n\sum _{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)\right],$$ (8)

where $I_{ij}^{\prime }=I\left(Y_i>Y_j,{\widehat{\text{FPR}}}_j\in (t_0,t_1)\right)$ and ${\widehat{\text{FPR}}}_j={\sum }_i{\widehat{p}}_i(1-D_i)I(Y_i>Y_j)/{\sum }_i{\widehat{p}}_i(1-D_i)$. To estimate ${\widehat{A}}_t^{\mathit{TDS}}$ in formula (8), we first estimate $\widehat{\text{FPR}}$, then limit $\widehat{\text{FPR}}$ to the range (t₀, t₁).

Besides ${\widehat{A}}_t^{\mathit{TDS}}$, we may also consider a standardization recommended by McClish [24] and Jiang et al. [25]. Let τ_t = A_t/(t₁ − t₀) = Pr(Y_D > Y_D̄| FPR ∈ (t₀, t₁)). The τ_t may be interpreted as the average true positive rate over the range of false positive rates of concern. The estimator τ̂_t is given by

$${\widehat{\tau }}_t^{\mathit{TDS}}={\sum }_{j=1}^n{\sum }_{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)I_{ij}^{\prime }/\left[{\sum }_{j=1}^n{\sum }_{i=1}^n{\widehat{p}}_i{\widehat{p}}_j{D}_i(1-D_j)I({\widehat{\text{FPR}}}_j\in (t_0,t_1))\right].$$

We establish the asymptotic properties of ${\widehat{\tau }}_t^{\mathit{TDS}}$ by letting $R_n^{\prime }({\tau }_t,\theta )={\sum }_{i=1}^n{\sum }_{j=1}^n{\stackrel{\sim }{p}}_i{\stackrel{\sim }{p}}_j{D}_i(1-D_j)\left(I_{ij}^{\prime }-{\tau }_{t}I({\text{F}\stackrel{\sim }{\text{P}}\text{R}}_j\in (t_0,t_1))\right)$ and FP̃R_j = Σ_ip̃_i(1 − D_i)I(Y_i > Y_j)/Σ_ip̃_i(1 − D_i) where p̃_i is given in (5). Let $\mathrm{\Delta }=\{j\mid \widehat{{\text{FPR}}_j}\in (t_0,t_1),D_j=0,j=1,\cdots ,n\}$, and $m_{\mathrm{\Delta }}^s$ be the number of elements of Δ falling into the sth stratum of the test result Y. Assuming the U-process $U_n^{\prime }({\tau }_t,\theta )=n^{1/2}[R_n^{\prime }({\tau }_t,\theta )-\text{E}(R_n^{\prime }({\tau }_t,\theta ))]$ is equicontinuous and $m_{\mathrm{\Delta }}^s/n\to {\rho }_{\mathrm{\Delta }}^s>0$ as n → ∞, we can show that the proposed estimator ${\widehat{\tau }}_t^{\mathit{TDS}}$ for τ_t follows asymptotically a normal distribution, that is, $n^{1/2}{\mathrm{\Omega }}_n^{-1/2}({\widehat{\tau }}_t^{\mathit{TDS}}-{\tau }_t)\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1)$, where Ω_n is given in the Appendix.

The asymptotic properties of ${\widehat{A}}_t^{\mathit{TDS}}$ can be easily obtained from ${\widehat{\tau }}_t^{\mathit{TDS}}$, that is,

$$n^{1/2}{\mathrm{\Gamma }}_n^{-1/2}({\widehat{A}}_t^{\mathit{TDS}}-A_t)\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1),$$ (9)

where ${\mathrm{\Gamma }}_n={\mathrm{\Omega }}_n{\left(\frac{{\sum }_j{p}_j(1-D_j)I({\text{FPR}}_j\in (t_0,t_1))}{{\sum }_j{p}_j(1-D_j)}\right)}^2$.

5 Simulation Study

Simulation studies were conducted to evaluate the performance of the AUC estimator Â^TDS and the pAUC estimator ${\widehat{A}}_t^{\mathit{TDS}}$ under the TDS design. First, we illustrate the unbiasedness of the estimators under the TDS design. Second, we assess the performance of the proposed variance estimators under different specifications of subject allocations and cutoff points for stratification. The structure of the TDS design is determined by several factors, including number of strata K, choice of cutoff points (a₁, a₂, · · ·, a_K−1) and subject allocations among strata n_s = (n₀, n₁, · · ·, n_K). Last, we examine the efficiency gain of the use of the TDS design and the proposed estimators over the SRS design and the standard estimators.

In all simulations, we assume D ~ Bernoulli(0.3), $Y_D~\mathcal{N}({\mu }_1,{\sigma }_1^2)$ and $Y_{\overline{D}}~\mathcal{N}({\mu }_0,{\sigma }_0^2)$. To generate the TDS data, we first generate a random sample of size n₀, and then generate the test-result-dependent (TDS) component. The TDS component consists of samples of sizes n₁, n₂, n₃ from the lower tail, the middle region and the upper tail of Y, respectively. These regions are defined by the cutoff points (a₁, a₂). For the purpose of illustration, we use K = 3 and (a₁, a₂) = (μ_Y − α × σ_Y, μ_Y +α × σ_Y ) for AUC, and (a₁, a₂) = (q₀, q₁) for partial AUC, where (q₀, q₁) are the quantiles of Y_D̄ at the pre-specified (t₀, t₁). All simulation studies are based on 5, 000 independent runs.

5.1 Unbiasedness of Proposed Estimators Under TDS

To illustrate the unbiasedness of the proposed estimator Â^TDS under the TDS design, we generated TDS data with K = 3, (a₁, a₂) = (μ_Y − ασ_Y, μ_Y + ασ_Y ) and n_s = (n₀, n₁, n₂, n₃) = (180, 60, 60, 60), where α = 1 or 1.5, μ_Y and σ_Y are mean and standard deviation of Y. Table 1 lists the mean (Estimate) and the relative bias (Bias%) of Â^SRS (3) and Â^TDS (4) under the TDS design. The standard AUC estimator Â^SRS yields significant bias under the TDS design, and the bias becomes larger as the cutoff points move away from the center of the test-result distribution. In a contrast, for the same TDS data, the proposed estimator Â^TDS yields unbiased estimates. In the illustration of the unbiasedness of the proposed pAUC estimator ${\widehat{A}}_t^{\mathit{TDS}}$, we chose the combinations: (t₀, t₁) = (0, 0.1), n_s = (180, 90, 90), and (t₀, t₁) = (0.1, 0.2), n_s = (180, 60, 60, 60). Notice that two methods were used to determine the cutoff points for stratification: either the true quantiles (q₀, q₁) of Y_D̄ or the quantiles (q̂₀, q̂₁) of the empirical distribution Ŷ_D̄ at (t₀, t₁). The standard estimator $A_t^{\mathit{SRS}}$ (7) and the proposed estimator ${\widehat{A}}_t^{\mathit{TDS}}$ (8) were applied to the same TDS data. As seen in Table 2, ${\widehat{A}}_t^{\mathit{SRS}}$ yields extreme bias under the TDS design, while the proposed estimator for the TDS design ${\widehat{A}}_t^{\mathit{TDS}}$ performs well with essentially no bias.

Table 1.

Unbiasedness of Â^TDS under TDS

True AUC	Method	Estimate	Bias %	Estimate	Bias %
		α = 1.0		α = 1.5
YD ~ (0.5, 1.5), YD̄ ~ (0, 1)
0.6092	ÂSRS	0.6298	3.38	0.6383	4.78
	ÂTDS	0.6080	−0.20	0.6079	−0.21
YD ~ (1, 1), YD̄ ~ (0, 1)
0.7602	ÂSRS	0.7969	4.83	0.8282	8.95
	ÂTDS	0.7594	−0.11	0.7587	−0.20

Note: TDS data were generated with n_s = (180, 60, 60, 60) and cutoff points (a₁, a₂) = (μ_Y − ασ_Y, μ_Y + ασ_Y). The Â^TDS and Â^SRS were applied to the same copies of the TDS data.

Table 2.

Unbiasedness of ${\widehat{A}}_t^{\mathit{TDS}}$ under TDS

(t0, t1)	True pAUC	Method	Estimate	Bias %	Estimate	Bias %
			(q0, q1)		(q̂0, q̂1)
YD ~ (0.5, 1.5), YD̄ ~ (0, 1)
(0, 0.1)	0.0209	${\widehat{A}}_t^{\mathit{SRS}}$	0.0817	290.91	0.0759	263.16
		${\widehat{A}}_t^{\mathit{TDS}}$	0.0206	−1.44	0.0208	−0.48
(0.1, 0.2)	0.0358	${\widehat{A}}_t^{\mathit{SRS}}$	0.1167	225.98	0.0985	175.14
		${\widehat{A}}_t^{\mathit{TDS}}$	0.0354	−1.12	0.0358	0
YD ~ (1, 1), YD̄ ~ (0, 1)
(0, 0.1)	0.0243	${\widehat{A}}_t^{\mathit{SRS}}$	0.0747	207.41	0.0701	188.48
		${\widehat{A}}_t^{\mathit{TDS}}$	0.0239	−1.65	0.0241	−0.82
(0.1, 0.2)	0.0481	${\widehat{A}}_t^{\mathit{SRS}}$	0.1220	153.64	0.1043	116.84
		${\widehat{A}}_t^{\mathit{TDS}}$	0.0474	−1.46	0.0481	0

Note: The TDS data were generated with n_s = (180, 90, 90) for (t₀, t₁) = (0, 0.1) and n_s = (180, 60, 60, 60) for (t₀, t₁) = (0.1, 0.2). The cutoff points for stratification are either the quantiles (q₀, q₁) of Y_D̄ or the quantiles (q̂₀, q̂₁) of Ŷ_D̄ at (t₀, t₁).

5.2 Performance of Variance Estimators under TDS

We examine the finite sample performance of the proposed variance estimators for Â^TDS and ${\widehat{A}}_t^{\mathit{TDS}}$ by simulating the TDS data with different structures. Table 3 and Table 4 lists the mean of estimates (Estimate), the relative bias (Bias%), the simulated standard error (SE), the mean of the estimated standard errors ( $\widehat{\text{SE}}$) and the coverage percentage of the 95% confidence interval (CP). Estimated standard errors were obtained by the proposed variance estimators, in which large sample quantities were replaced with finite sample quantities. As seen in Table 3, the proposed estimator Â^TDS has good finite sample property under TDS data with several specifications of subject allocation and cutoff points. The estimated standard errors in all cases are very close to their true values and the empirical 95% coverage percentage are close to the nominal value. Different structures of the TDS design result in slightly different true standard errors, but the proposed estimators and their variance estimators perform fairly well in all these cases. Similar results are observed in Table 4 for the proposed estimator ${\widehat{A}}_t^{\mathit{TDS}}$ for partial AUC under various TDS designs. Simulation studies were conducted under many combinations on subject allocation and cutoff points, but we have not seen a systematic pattern that a specific combination consistently yields most efficiency gain.

Table 3.

Performance of Â^TDS and Its Variance Estimator under TDS

True AUC	TDS Designs	Estimate	Bias%	SE	$\widehat{\text{SE}}$	95% CP
	ns		(α0, α1) = (−1.0, 1.0)
0.7602	(180, 60, 60, 60)	0.7594	−0.11	0.0274	0.0274	0.949
	(180, 30, 60, 90)	0.7596	−0.08	0.0277	0.0277	0.951
	(270, 30, 30, 30)	0.7591	−0.14	0.0270	0.0267	0.943
	(α0, α1)		ns = (180, 60, 60, 60)
0.7602	(−1.5, 1.5)	0.7587	−0.20	0.0304	0.0302	0.945
	(−1.0, 0.5)	0.7593	−0.12	0.0277	0.0274	0.943

Note: The TDS data were drawn assuming Y_D ~ (1, 1), Y_D̄ ~ (0, 1) with (a₁, a₂) = (μ_Y + α₀σ_Y, μ_Y + α₁σ_Y ) and n_s = (n₀, n₁, n₂, n₃).

Table 4.

Performance of ${\widehat{A}}_t^{\mathit{TDS}}$ and Its Variance Estimator under TDS

(t0, t1)	True pAUC	ns	Estimate	Bias%	SE	$\widehat{\text{SE}}$	95% CP
(q0, q1)
(0, 0.1)	0.0243	(180, 90, 90)	0.0239	−1.65	0.0039	0.0039	0.955
(0, 0.1)	0.0243	(180, 30, 150)	0.0238	−2.06	0.0036	0.0039	0.961
(q̂0, q̂1)
(0, 0.1)	0.0243	(180, 90, 90)	0.0241	−0.82	0.0040	0.0039	0.950
(0, 0.1)	0.0243	(180, 30, 150)	0.0240	−1.23	0.0038	0.0039	0.956
(0, 0.05)	0.0079	(180, 90, 90)	0.0078	−1.27	0.0016	0.0017	0.961
(0, 0.2)	0.0724	(180, 90, 90)	0.0721	−0.41	0.0093	0.0087	0.938
(0.1, 0.2)	0.0481	(180, 60, 60, 60)	0.0481	0	0.0057	0.0054	0.933

Note: The TDS data were drawn assuming Y_D ~ (1, 1), Y_D̄ ~ (0, 1).

5.3 Efficiency Comparison of TDS Design over SRS Design

One of the main reasons to use the TDS design over the standard SRS design in biomarker validation is that the TDS design improves efficiency in assessing the performance of a diagnostic test or biomarker with the same number of subjects. The strategy could be particularly useful for the estimation of the partial area under ROC curve (pAUC). In this case, oversampling can be done by concentrating those subjects with test results largely corresponding to the range of false-positive rates of interest. To illustrate this, we generated separate data for the SRS design and the TDS design of the same sample size n = 360. Efficiency comparison is made by applying ${\widehat{A}}_t^{\mathit{TDS}}$ to the TDS data while applying ${\widehat{A}}_t^{\mathit{SRS}}$ to the SRS data. If simulation shows that the TDS design yields better efficiency over the SRS design in estimating pAUC, it would support the utility of the TDS design over the SRS design in biomarker validation. Table 5 lists Estimate, Bias% and simulated standard error (SE), ratio of mean square error (RMSE). subject proportion (n_ratio%) for the TDS data and the SRS data, where n_s = (180, 90, 90) is taken for (t₀, t₁) = (0, 0.1). Table 5 shows considerable efficiency gain using the TDS design to estimate partial AUC under various scenarios, regardless of whether the true quantile (q₀, q₁) or (q̂₀, q̂₁) was used as cutoff points for stratification. The results indicate that a combined use of the TDS design and the proposed pAUC estimator can lead to more than 50% efficiency gain over that from the SRS design and the standard pAUC estimator. Based on similar simulations, the efficiency gain for estimating AUC under the TDS design is relatively small. This is understandable since the TDS design oversamples patients corresponding to the range of interest. Thus, the efficiency gain of the TDS design is not as high in estimating AUC, since it is an average across the full range of test results, thereby counteracting the effect of the TDS design by oversampling the two tails and undersampling in between.

Table 5.

Efficiency Comparison of TDS Design and SRS Design for Estimating pAUC

(t0, t1)	Design	nratio%	Method	True	Estimate	Bias %	SE	RMSE
YD ~ (0.1, 1), YD̄ ~ (0, 1)
					(q0, q1)
(0, 0.1)	SRS	(89, 11)	${\widehat{A}}_t^{\mathit{SRS}}$	0.0061	0.0060	−1.64	0.0023	1
	TDS	(70, 30)	${\widehat{A}}_t^{\mathit{TDS}}$	0.0061	0.0059	−3.28	0.0014	0.40
					(q̂0, q̂1)
(0, 0.1)	SRS	(91, 9)	${\widehat{A}}_t^{\mathit{SRS}}$	0.0061	0.0064	4.92	0.0024	1
	TDS	(70, 30)	${\widehat{A}}_t^{\mathit{TDS}}$	0.0061	0.0060	−1.64	0.0015	0.33
YD ~ (1, 1), YD̄ ~ (0, 1)
					(q0, q1)
(0, 0.1)	SRS	(84, 16)	${\widehat{A}}_t^{\mathit{SRS}}$	0.0243	0.0243	0	0.0062	1
	TDS	(67, 33)	${\widehat{A}}_t^{\mathit{TDS}}$	0.0243	0.0238	−2.06	0.0038	0.38
					(q̂0, q̂1)
(0, 0.1)	SRS	(77, 23)	${\widehat{A}}_t^{\mathit{SRS}}$	0.0243	0.0252	3.70	0.005	1
	TDS	(63, 37)	${\widehat{A}}_t^{\mathit{TDS}}$	0.0243	0.0241	−0.82	0.004	0.62

Note: RMSE represents ratio of mean square error relative to the SRS design. ${\widehat{A}}_t^{\mathit{SRS}}$ was applied to the SRS data; ${\widehat{A}}_t^{\mathit{TDS}}$ was applied to TDS data. The SRS and TDS designs have the same size n = 360. For TDS data n_ratio = (n₀₁ + n₁, n₀₂ + n₂, n₀₃ + n₃)/n, where n_0k is the number of patients in each stratum with the cutoff points (q₀, q₁) or (q̂₀, q̂₁) based on the SRS component of the TDS data. For the SRS data n_ratio% is the percentage of patients falling in each stratum defined by the cutoff points for stratification.

6 Data Example

We have focused on the TDS design and emphasized its potential to improve efficiency over the SRS design in AUC and partial AUC estimation. In practice, biomarker data with the TDS structure may also come from existing studies, in which subjects are not a simple random sample from the target population and analyzing the available data ignoring this fact will lead to biased results. In this section, we use a randomized clinical trial to illustrate this idea and show how the proposed TDS methods can be applied. CALGB 30801 is an ongoing phase III lung cancer trial conducted at the Cancer and Leukemia Group B (CALGB) [31]. A total of 216 patients with positive COX-2 expression are to be randomized with equal allocation to a standard chemotherapy (arm A) or celecoxib combined with standard chemotherapy (arm B). The primary objective of the trial is to evaluate the survival benefit of celecoxib (a COX-2 inhibitor) among patients with positive COX-2 expression. COX-2 is a protein over-expressed in lung cancers and its intensity is measured on a continuous scale ranging from 0 to 10. Based on preliminary data [32], the proportions of patients with negative (COX-2 < 2), moderate (2 ≤ COX-2 < 4) and positive (COX-2 ≥ 4) expression are about 60%, 13% and 27% respectively in the target population. Therefore, approximately 800 patients are to be screened to accrue the 216 positive patients. As a secondary objective, the investigators are interested in validating the prognostic value of COX-2 for survival in those patients who receive the standard chemotherapy. To answer this question, data from patients who have a full range of COX-2 scores (negative, moderate, positive) and are treated by standard chemotherapy are needed. To avoid the costly option of treating and following the large number of COX-2 negative patients, the investigators decided to select about 1/4 of all negatives to treat and follow for long term survival. At the end of the trial, the data available for assessing the prognostic value of COX-2 has a data structure of the TDS design. The simple random component (SRC) include all negatives when 1/4 of the positives are enrolled, all moderates when 1/4 of the positives are enrolled, and the first 54 positives on arm A. The expected sizes of the three SRC strata are n_0,1 = 120, n_0,2 = 26, n_0,3 = 54, respectively. The rest of patients receiving standard chemotherapy, including n₁ = 0 negatives, n₂ = 78 moderates and n₃ = 54 positives patients, makes up of the test-result-dependent component (TDC) of the TDS design. For this purpose of illustration, we define D = 1 for patients surviving less than 6 years and D = 0 otherwise. Since very few patients are expected to drop out before 6 years of follow-up in this trial, we ignore the issue of censoring for those patients who are followed less than 6 years. However, when censoring is non-ignorable and the predictive accuracy of the biomarker over time is of interest, an estimator that takes into account both time-to-event endpoint and censoring is warranted. This is a research topic for the future.

The CALGB trial is still open for patient accrual and the final data are not yet available for analysis. To illustrate the proposed AUC and pAUC estimators, we resampled a cohort of 800 patients with full range of COX-2 measures using the data from a preliminary COX-2 study [32], and used the cohort to create a dataset with the same data structure of the TDS design as CALGB 30801. The specific goal is to evaluate how well COX-2 expression levels predict death within 6 years) among the patients who receive only standard chemotherapy. Figure 1 shows the estimated ROC curves by the standard method ignoring the TDS data structure (Left) and the proposed method (Right), and the shaded areas indicate the estimated pAUCs. For AUC, the vector of cutoff points is set to (−∞, 3, 4, ∞). For pAUC, the range of t is set to (0, 0.1), which approximately corresponds to cutoff points (−∞, 4, ∞), such that n₁ = 0 and n₂ = 54 in the notation for pAUC in Section 5. The estimated AUC using Â^TDS is 0.8052 with a 95% confidence interval [0.7445, 0.8544]. The biased estimator Â^SRS yields an AUC estimate of 0.7749 with a 95% confidence interval [0.7192, 0.8223]. When Â^SRS is applied only to the SRS component of the dataset, the estimated AUC is 0.8125 with a 95% confidence interval [0.7400, 0.8685]. Assuming that the investigators are interested in the performance of the biomarker within a low FPR range t ∈ (0, 0.1), the estimated pAUC using ${\widehat{A}}_t^{\mathit{TDS}}$ is 0.0216 with a 95% confidence interval [0.0159, 0.0291]. The biased estimator ${\widehat{A}}_t^{\mathit{SRS}}$ yields a pAUC estimate of 0.0151. The estimated pAUC is 0.0115 when ${\widehat{A}}_t^{\mathit{SRS}}$ is applied only to the SRS component of the dataset.

Figure 1. COX-2 ROC Curve Estimated by Standard Method and Proposed Method.

Note: The left ROC curve is estimated by the standard method for a SRS design; the right ROC curve is estimated by the proposed method for a TDS design. The shaded areas denote the partial area under curve (pAUC) with t ∈ (0, 0.1)

7 Discussion

We propose nonparametric estimators for the area under the ROC curve (AUC) and the partial area under the ROC curve (pAUC) for data arising from a test-result-dependent sampling scheme (TDS). We demonstrate that the estimated empirical distribution of test results can be used to reweight the standard estimators of AUC and pAUC to provide valid inference for data arising from the TDS design. We establish the asymptotic properties of the proposed estimators under general regularity conditions. We also demonstrate good finite sample properties for the proposed estimators and their variance estimators under various specifications of the extent of separation of diseased vs. non-diseased subjects, cutoff points for stratification and subject allocation among strata. The proposed estimators are nonparametric in the same sense as the nonparametric AUC and pAUC estimators under the SRS design [4, 20]; and these estimators make no parametric assumptions about the relationship between test results and true disease status.

The test-result-dependent (TDS) design is motivated by the need for better efficiency for biomarker validation studies with limited resources. It is particularly useful when determining the true disease status is expensive, invasive or time consuming. In this article, the data arising from the TDS design consists of two components: a simple random component and a test-result-dependent component. The determination of the true disease status of a subject in the TDS component is dependent on the test results. More specifically, the selection of the TDS component is dependent on the intervals defined by pre-chosen cutoff points for stratification on the test result.

Simulations confirm that the proposed estimators for AUC and partial AUC effectively correct the bias introduced by the TDS design. Simulations also show that efficiency gains can be achieved by adopting a TDS design over a SRS design, particularly when the estimation of partial AUC is of interest. It is not surprising to see that much better precision is achieved by the TDS design for partial AUC estimation since more subjects fall into the false-positive range of interest as compared to the SRS design. In addition, greater efficiency gain occurs for pAUC estimation when (q₀, q₁) is unknown and has to be estimated than when (q₀, q₁) is known. The efficiency gain in estimating AUC under a TDS design is also understandable since a SRS design leads to an uneven distribution of subjects in different segments of the test results and hence leads to lower precision in AUC estimation.

Appendix

Let $B_{si}({\theta }_k)={\stackrel{\sim }{p}}_{si}\left[\frac{n_k}{{\theta }_k^2}I(y_{si}\in C_k)-\frac{n_K}{{\theta }_K^2}I(y_{si}\in C_K)\right]-\frac{n_k}{{\theta }_k}+\frac{n_K}{{\theta }_K}$, s = 0, · · ·, K, i = 1, · · ·, n_s, k = 1, · · ·, K − 1, where ${\stackrel{\sim }{p}}_{si}={\left[n_0+{\sum }_{k=1}^K{n}_k/{\theta }_{k}I(y_{si}\in C_k)\right]}^{-1}$.

Let

$$B_{si}(\theta )={(B_{si}({\theta }_1),\cdots ,B_{si}({\theta }_{K-1}))}^T$$

and

$$H_{si}(\theta )=\frac{1}{{\stackrel{\sim }{p}}_{si}}\frac{{\partial }^2{\stackrel{\sim }{p}}_{si}}{\partial \theta {\theta }^T}-\frac{1}{{\stackrel{\sim }{p}}_{si}^2}\left[\frac{\partial {\stackrel{\sim }{p}}_{si}}{\partial \theta }\frac{\partial {\stackrel{\sim }{p}}_{si}}{\partial {\theta }^T}+\frac{n_K}{{\theta }_K^2}{P}_{K-1}+\text{diag}\left(\frac{n_1}{{\theta }_1^2},\cdots ,\frac{n_{K-1}}{{\theta }_{K-1}^2}\right)\right],$$

where P_K−1 is the (K − 1)th-order matrix with all elements 1.

Proof for Asymptotic Distribution of θ̂

The proof follows Owen [28, 29] and Qin and Lawless [30]. Since n_s/n, s = 0, · · ·, K − 1 converge to ρ_s as n → ∞, n_K/n converges to ${\rho }_K={\sum }_{s=0}^{K-1}{\rho }_s$. Using the Taylor expansion of l_p(θ̂) (see (2)) at the true value θ and noticing $\frac{\partial l_p(\theta )}{\partial \theta }{\mid }_{\theta =\widehat{\theta }}=0$, we have

$$\begin{array}{l}{n}^{1/2}(\widehat{\theta }-\theta )=-{\left[\frac{1}{n}\frac{{\partial }^2{l}_p(\theta )}{\partial \theta \partial {\theta }^T}\right]}^{-1}{n}^{-1/2}\frac{\partial l_p(\theta )}{\partial \theta }+o_p(n^{-1/2})\\ ={\sum }_{s=0}^K{\rho }_s^{1/2}{n}_s^{-1/2}{\sum }_{i=1}^{n_s}{\xi }_{si}+o_p(1),\end{array}$$

where ${\xi }_{si}=-{\left[n^{-1}{\sum }_{s=0}^K{\sum }_{i=1}^{n_s}{H}_{si}(\theta )\right]}^{-1}{B}_{si}(\theta )$. Rewriting the sum $n_s^{-1/2}{\sum }_{i=1}^{n_s}{\xi }_{si}$ as $1/n_0{\sum }_{i=1}^{n_0}+1/n_1{\sum }_{i=1}^{n_1}+\cdots +1/n_K{\sum }_{i=1}^{n_K}$, we can apply the central limit theorem to each term, and it is easy to show $1/n{\sum }_{s=0}^K{\sum }_{i=1}^{n_s}{H}_{si}(\theta )$ converges as n tends to infinity. Thus we have n^1/2(θ̂− θ) converges to normal distribution as n tends to infinity. Let ${\mathrm{\Psi }}_n={\sum }_{s=0}^K{\rho }_{s}var({\xi }_{si})$, we have

$$n^{1/2}{\mathrm{\Psi }}_n^{-1/2}(\widehat{\theta }-\theta )\to \mathcal{N}(0,1).$$

Proof for Asymptotic Distribution of Â^TDS

It is obvious that R_n(Â^TDS, θ̂) = 0. Denote e(A, θ) = E(R_n(A, θ)). Expanding R_n(Â^TDS, θ̂) at (A, θ), it follows from the condition that U_n(A, θ) is equicontinuous that

$$\begin{array}{l}{n}^{1/2}{R}_n({\widehat{A}}^{\mathit{TDS}},\widehat{\theta })=n^{1/2}\left[R_n({\widehat{A}}^{\mathit{TDS}},\widehat{\theta })-e({\widehat{A}}^{\mathit{TDS}},\widehat{\theta })\right]-n^{1/2}[R_n(A,\theta )-e(A,\theta )]+n^{1/2}\left[e({\widehat{A}}^{\mathit{TDS}},\widehat{\theta })-e(A,\theta )\right]+n^{1/2}{R}_n(A,\theta )+o_p(1)\\ =o_p(1)+n^{1/2}\left[\frac{\partial e(A,\theta )}{\partial A}({\widehat{A}}^{\mathit{TDS}}-A)+\frac{\partial e(A,\theta )}{\partial \theta }(\widehat{\theta }-\theta )\right]+n^{1/2}{R}_n(A,\theta ),\end{array}$$

where $\frac{\partial e(A,\theta )}{\partial A}=Pr(D=1)Pr(D=0)$ and

$$\frac{\partial e(A,\theta )}{\partial \theta }={\sum }_{i=1}^n{\sum }_{j=1}^n\text{E}\left\{\left(\frac{\partial {\stackrel{\sim }{p}}_i}{\partial \theta }{\stackrel{\sim }{p}}_j+\frac{\partial {\stackrel{\sim }{p}}_j}{\partial \theta }{\stackrel{\sim }{p}}_i\right)[D_i(1-D_j)(I_{ij}-A)+D_j(1-D_i)(I_{ji}-A)]\right\}/2.$$

Rewrite ξ_si in the proof of asymptotic distribution for θ̂ using single subscript as ξ_i. We have

$$\begin{array}{l}{n}^{1/2}\left({\widehat{A}}^{\mathit{TDS}}-A\right)={\left(\frac{\partial e(A,\theta )}{\partial A}\right)}^{-1}{n}^{1/2}\left[-R_n(A,\theta )-\frac{\partial e(A,\theta )}{\partial \theta }(\widehat{\theta }-\theta )\right]\\ ={[Pr(D=0)Pr(D=1)]}^{-1}{n}^{-1/2}{\sum }_{i=1}^n{\eta }_i+o_p(1),\end{array}$$

where ${\eta }_i=-E(R_{ij}+R_{ji}\mid Y_i,D_i)-\frac{\partial e(A,\theta )}{\partial \theta }{\xi }_i$ where R_ij = p̃_ip̃_jD_i(1 − D_j)(I_ij − A). The last identity follows by the asymptotic property of θ̂ and by the limit distribution theorem of U-Statistic [33].

$$\begin{array}{l}\sqrt{n}{R}_n(A,\theta )=n^{-1/2}\left[{\sum }_{i=1}^{n}E(R_{ij}\mid Y_i,D_i)+{\sum }_{j=1}^{n}E(R_{ij}\mid Y_j,D_j)\right]\\ =n^{-1/2}{\sum }_{i=1}^{n}E(R_{ij}+R_{ji}\mid Y_i,D_i).\end{array}$$

Rewrite $n^{-1/2}{\sum }_{i=1}^n{\eta }_i$ using double subscripts as $n^{-1/2}{\sum }_{s=0}^K{\sum }_{i=1}^{n_s}{\eta }_{si}={\sum }_{s=0}^K{\rho }_s^{1/2}{n}_s^{-1/2}{\sum }_{i=1}^{n_s}{\eta }_i$, then we can apply the central limit theorem to each term ${\rho }_s^{1/2}{\eta }_s^{-1/2}{\sum }_{i=1}^{n_s}{\eta }_i$, s = 0, · · ·, K. Let ${\mathrm{\Lambda }}_n={[Pr(D=0)Pr(D=1)]}^2{\sum }_{s=0}^K{\rho }_{s}var({\eta }_{si})$, it follows that $n^{1/2}{\mathrm{\Lambda }}_n^{-1/2}\left({\widehat{A}}^{\mathit{TDS}}-A\right)\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1)$ as n → ∞.

Proof of Asymptotic Distribution for τ̂_t

Denote $e^{\prime }({\tau }_t,\theta )=\text{E}(R_n^{\prime }({\tau }_t,\theta ))$. Expanding $R_n^{\prime }({\widehat{\tau }}_t^{\mathit{TDS}},\widehat{\theta })$ at the true value (τ_t, θ), it follows from the condition that $U_n^{\prime }({\tau }_t,\theta )$ is equicontinuous that

$$n^{1/2}{R}_n^{\prime }({\widehat{\tau }}_t^{\mathit{TDS}},\widehat{\theta })=o_p(1)+n^{1/2}\left[\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial {\tau }_t}({\widehat{\tau }}_t^{\mathit{TDS}}-{\tau }_t)+\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial \theta }(\widehat{\theta }-\theta )\right]+n^{1/2}{R}_n^{\prime }({\tau }_t,\theta ),$$

where $\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial {\tau }_t}=Pr(D=1)Pr(\text{FPR}\in (t_0,t_1)\mid D=0)$ and

$$\begin{array}{l}\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial \theta }={\sum }_{i=1}^n{\sum }_{j\in \mathrm{\Delta }}\text{E}\left\{\left(\frac{\partial {\stackrel{\sim }{p}}_i}{\partial \theta }{\stackrel{\sim }{p}}_j+\frac{\partial {\stackrel{\sim }{p}}_j}{\partial \theta }{\stackrel{\sim }{p}}_i\right)[D_i(1-D_j)(I_{ij}^{\prime }-{\tau }_{t}I({\text{FPR}}_j\in (t_0,t_1)))]\right\}+\\ {\sum }_{i=1}^n{\sum }_{j\in \mathrm{\Delta }}\text{E}\{D_j(1-D_i)\left[I_{ji}^{\prime }-{\tau }_{t}I({\text{FPR}}_i\in (t_0,t_1))\right]/2\}.\end{array}$$

By the proof of asymptotic distribution for t̂heta, we have

$$\begin{array}{l}{n}^{1/2}({\widehat{\tau }}_t^{\mathit{TDS}}-{\tau }_t)={\left(\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial {\tau }_t}\right)}^{-1}{n}^{1/2}\left[-R_n^{\prime }({\tau }_t,\theta )-\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial \theta }(\widehat{\theta }-\theta )\right]\\ ={\left(\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial {\tau }_t}\right)}^{-1}{n}^{-1/2}\left\{{\sum }_{i=1}^n\left[-E(R_{ij}^{\prime }\mid Y_i,D_i)-\frac{\partial e^{\prime }({\tau }_t,\theta )}{\partial \theta }{\xi }_i\right]-{\sum }_{j=1}^{n}E(R_{ji}^{\prime }\mid Y_j,D_j)\right\}\\ ={[Pr(\text{FPR}\in (t_0,t_1),D=0)Pr(D=1)]}^{-1}\left(n^{-1/2}{\sum }_{i=1}^n{\zeta }_i+n^{-1/2}{\sum }_{j=1}^n{\zeta }_j^{\prime }\right)+o_p(1),\end{array}$$

where $R_{ij}^{\prime }={\stackrel{\sim }{p}}_i{\stackrel{\sim }{p}}_j{D}_i(1-D_j)\left(I_{ij}^{\prime }-{\tau }_{t}I(F\stackrel{\sim }{P}{R}_j\in (t_0,t_1))\right)$, and the second identity follows by the limit distribution theorem of U-statistic, similarly to the proof of Theorem 1. Rewrite $n^{-1/2}{\sum }_{i=1}^n{\zeta }_i$ using double subscripts as ${\sum }_{s=0}^K{\eta }_s^{-1/2}{\sum }_{i=1}^{n_s}{\rho }_s^{1/2}{\zeta }_{si}$. Similarly, rewrite $n^{-1/2}{\sum }_{j=1}^n{\zeta }_j^{\prime }$ as ${\sum }_{s=0}^K{(n_{\mathrm{\Delta }}^s)}^{-1/2}{\sum }_{j=1,j\in \mathrm{\Delta }}^{n_{\mathrm{\Delta }}^s}{({\rho }_{\mathrm{\Delta }}^s)}^{1/2}{\zeta }_{sj}^{\prime }$. Applying the central limit theorem to ${\sum }_{i=1}^{n_s}{\rho }_s^{1/2}{\zeta }_{si}$ and ${\sum }_{j=1,j\in \mathrm{\Delta }}^{n_{\mathrm{\Delta }}^s}{({\rho }_{\mathrm{\Delta }}^s)}^{1/2}{\zeta }_{sj}^{\prime }$ respectively. Let ${\mathrm{\Omega }}_n={\rho }_{s}var({\zeta }_{si})+{\rho }_{\mathrm{\Delta }}^{s}var({\zeta }_{sj}^{\prime })$, we have $n^{1/2}{\mathrm{\Omega }}_n^{-1/2}({\widehat{\tau }}_t-{\tau }_t)\stackrel{\mathcal{L}}{\to }\mathcal{N}(0,1)$.