Biomarker Evaluation Under Imperfect Nested Case–control Design

Summary

The nested case–control (NCC) design has been widely adopted as a cost-effective sampling design for biomarker research. Under the NCC design, markers are only measured for the NCC subcohort consisting of all cases and a fraction of the controls selected randomly from the matched risk sets of the cases. Robust methods for evaluating prediction performance of risk models have been derived under the inverse probability weighting (IPW) framework. The probabilities of samples being included in the NCC cohort can be calculated based on the study design¹ or estimated non-parametrically². Neither strategy works well due to model mis-specification and the curse of dimensionality in practical settings where the sampling does not entirely follow the study design or depends on many factors. In this paper, we propose an alternative strategy to estimate the sampling probabilities based on a varying coefficient model, which attains a balance between robustness and the curse of dimensionality. The complex correlation structure induced by repeated finite risk set sampling makes the standard resampling procedure for variance estimation fail. We propose a perturbation resampling procedure that provides valid interval estimation for the proposed estimators. Simulation studies show that the proposed method performs well in finite samples. We apply the proposed method to the Nurses’ Health Study II to develop and evaluate prediction models using clinical biomarkers for cardiovascular risk.

1 ∣. INTRODUCTION

Conducting rigorous biomarker validation studies is an important step in the translation of novel biomarkers into routine clinical practice for medical decision making. Such studies should follow design principles in sample selection to avoid bias³. Large prospective studies, such as the Women’s Health Initiative Study and the Nurses’ Health Study, with exposures captured and biologic samples collected and stored prior to disease onset, can serve as a platform for biomarker research^4,5. However, measuring biomarkers for large prospective cohorts is highly resource consuming. To make efficient use of stored samples from a cohort, two-phase sampling designs, including nested case–control (NCC) and case–cohort (CCH) studies, are often adopted as resource-efficient sampling strategies, especially when the outcome of interest is rare^6,7,8.

Under the NCC design, new markers are measured for all cases and a subset of controls randomly sampled without replacement from the risk sets of the cases. Sometimes controls are also matched to cases on variables such as gender and age. Many well-known biomarker studies nested in large cohorts have employed the NCC design^{9,10,11, e.g.}. For example, in the Nurses’ Health Study II (NHS_II), novel biomarkers, apoA1 concentration in whole plasma (WPA1) and concentration of apoE in whole plasma (apoE), were investigated for predicting the risk of Myocardial Infraction (MI)¹¹. Due to limited resources and low incidence of MI, the biomarkers were measured on a nested case–control set, which included all cases and controls sampled from the 1:1 matched risk set of the cases with matching variables including smoking status, fasting status, age, and timing of blood collection.

To analyze NCC data, conditional logistic regression (CLR) model has traditionally been used when the focus is on the estimation of hazard ratio (HR) parameters, and sometimes the estimation of absolute risk parameters^12,13. The CLR provides HR estimators under the Cox model from the full cohort, however cannot be extended to other models. Nor can the methods be used for estimating other parameters, such as the prediction accuracy parameters, which involve the distribution of the markers in the full cohort. For model parameter estimation, fully efficient maximum likelihood estimators (MLE) have also been proposed^14,15. The MLE relies on the correct specification of the failure time model and requires that censoring is independent of the novel markers as well as additional modeling assumptions when there are multiple novel biomarkers and routine clinical variables measured on the full cohort.

As a flexible alternative, the inverse probability weighting (IPW) approach has been developed¹⁶. Recently, IPW estimators have also been developed for fitting models beyond the Cox model as well as for prediction performance measures including the receiver operating characteristic (ROC) curve, positive predictive value (PPV) and negative predictive value (NPV)^17,18,19. Most existing IPW estimators for NCC studies calculates the true IPW (TIPW) sampling weights according to the study design and are consistent provided that the sampling weights are correctly obtained. However, the TIPW estimators may be invalid if the sampling is not implemented exactly according to the design. Such a scenario arises, for example, when the matching criteria are implemented only coarsely during the implementation of the sampling scheme due to practical concerns. Such an imperfect NCC design poses additional analytical challenges for estimating and evaluating risk prediction models. To overcome the bias, one may estimate the sampling weights non-parametrically via kernel smoothing as in Zheng et al.². Obtaining such a non-parametric augmented IPW (NP) estimator, however, is not feasible when the number of matching variables is not very small due to the curse of dimensionality.

In this paper, we propose a new semi-non-parametric AIPW estimator, where the selection probability is estimated based on a flexible varying coefficient model. The AIPW estimator can incorporate a larger number of matching variables while remaining robust to the deviation from the intended sampling scheme. We derive the the asymptotic properties for the proposed estimators and come up with a resampling method to assess the variability of our proposed AIPW estimators.

The remainder of the paper is organized as follows. In Section 2, we provide model specification and describe the proposed point estimation procedures. Our proposed interval estimation procedure is given in Section 3. In Section 4, we report results of simulation studies to assess the finite sample performance of the proposed method. In Section 5, the data from NHS II is analyzed as illustration. Concluding remarks are given in Section 6. Theoretical studies of the proposed estimators are provided in the Appendix.

2 ∣. ESTIMATING SAMPLING WEIGHTS VIA AIPW

Let T denote the survival outcome of interest and $\mathbf{Y}=({\mathbf{Y}}_{\text{old}}^{\mathsf{T}},{\mathbf{Y}}_{\text{new}}^{\mathsf{T}}{)}^{\mathsf{T}}$ denote the vector of predictors for T, where Y_old denotes the vector of routine markers and Y_new denotes the vector of novel biomarkers. Due to censoring, T is only observed up to a bivariate vector X = T Λ C and δ = I(T ≤ C), where C is the censoring time. Under the NCC design, Y_new is only measured if V = 1, where V = δ + (1 − δ)V₀ and V₀ is a binary indicator for whether a subject is sampled into the NCC subcohort as a control. We assume that the sampling of the controls is performed by matching to the cases according to a vector of matching variables M. Suppose that the underling data for the full cohort consists of N independent and identically distributed random vectors, $\mathcal{D}=\{{\mathbf{D}}_i=(X_i,{\delta }_i,{\mathbf{Y}}_i^{\mathsf{T}},{\mathbf{M}}_i^{\mathsf{T}}{)}^{\mathsf{T}},i=1,\dots ,N\}$, while the observed data consist of $\mathcal{O}=\{{\mathbf{O}}_i=(X_i,{\delta }_i,{\mathbf{Y}}_{\text{old},i}^{\mathsf{T}},V_i{\mathbf{Y}}_{\text{new},i}^{\mathsf{T}},{\mathbf{M}}_i^{\mathsf{T}}{)}^{\mathsf{T}},i=1,\dots ,N\}$. Let Ω = {i : 1 ≤ i ≤ N} and Ω_ncc = {i : 1 ≤ i ≤ N, V_i = 1} respectively denote the index sets for the full cohort and NCC subcohort.

Under the matched NCC design, for a case with event time X_i and matching variables M_i, m controls are sampled from the matched risk set

$${\mathcal{R}}_{{\mathbf{W}}_i}=\{k:X_k\ge X_i,\mid {\mathbf{M}}_k-{\mathbf{M}}_i\mid \le {\mathbf{a}}_0\},$$

where a₀ is a predetermined range vector and ${\mathbf{W}}_i=({\delta }_i,X_i,{\mathbf{M}}_i^{\mathsf{T}},{\mathbf{Y}}_{\text{old},i}{)}^{\mathsf{T}}$. Let ${\overline{\pi }}_i=P(V_i=1\mid \mathcal{O})$ and ${\overline{\pi }}_{0i}=P(V_i=1\mid \mathcal{O},{\delta }_i=0)$ denote the true sampling probabilities under possibly imperfect NCC sampling. If the NCC sampling were implemented exactly accordingly to design, then ${\overline{\pi }}_i={\delta }_i+(1-{\delta }_i){\overline{\pi }}_{0i}$ can be calculated as ${\tilde{\pi }}_i={\delta }_i+(1-{\delta }_i){\tilde{\pi }}_{0i}$, where ${\tilde{\pi }}_{0i}={\tilde{\pi }}_0({\mathbf{W}}_i)$,

$${\tilde{\pi }}_0({\mathbf{W}}_i)=1-\prod _{j:j\in {\mathcal{R}}_{{\mathbf{W}}_i}}\left\{1-\frac{m{\delta }_j}{\mid {\mathcal{R}}_{{\mathbf{W}}_j}\mid -1}\right\}$$

and $\mid {\mathcal{R}}_{{\mathbf{W}}_i}\mid$ is the size of ${\mathcal{R}}_{{\mathbf{W}}_i}$¹⁶. Under the perfect NCC design, TIPW estimators can be constructed by weighting observations with the true weights ${\tilde{\omega }}_i=V_i∕{\tilde{\pi }}_i$. To improve efficiency and robustness over the TIPW estimators, Zheng et al.² proposed NP estimators using non-parametrically estimated weights ${\widehat{\omega }}_i^{\text{NP}}=V_i∕{\widehat{\pi }}^{\text{NP}}({\mathbf{W}}_i)$, where ${\widehat{\pi }}^{\text{NP}}({\mathbf{W}}_i)={\delta }_i+(1-{\delta }_i){\widehat{\pi }}_0^{\text{NP}}({\mathbf{W}}_i)$,

$${\widehat{\pi }}_0^{\text{NP}}(\mathbf{w})=\frac{{\sum }_{i=1}^N(1-{\delta }_i)V_i{K}_b({\mathbf{W}}_i-\mathbf{w})}{{\sum }_{i=1}^N(1-{\delta }_i)K_b({\mathbf{W}}_i-\mathbf{w})}$$

is a non-parametric estimate of π₀(w) = P(V_i = 1 ∣ W_i = w, δ_i = 0), $K_b(\mathbf{w})=b^{-q}{\prod }_{j=1}^{q}K(w_j∕b)$, K(·) is a symmetric density function, and b > 0 denotes the bandwidth.

While the NP method can be used to incorporate imperfect NCC designs, it is infeasible when the dimension of W is not small. To overcome the limitations of TIPW and NP methods, we propose a semi-non-parametric AIPW method by approximating ${\overline{\pi }}_{0i}$ via a flexible varying coefficient model

$${\pi }_{0i}=g\{\beta ({\tilde{\pi }}_{0i},X_i{)}^{\mathsf{T}}{\mathbf{Z}}_i\}\text{with}g(x)=\frac{e^x}{1+e^x}$$ (2.1)

where Z_i = (1, Φ₁(Y_{old, i})^⊤, Φ₂(M_i)^⊤)^⊤, Φ₁(·) and Φ₂(·) are basis functions that allow potential non-linear effects, and β(π, x) is the unknown coefficient function. In practice, we find that the commonly used b-spline or natural splines basis with degree of freedom 3 works well. Equally spaced knots that covers most of the domain of the data are also desirable. We find that our results are not overly sensitive to the choice of the basis functions provided that they are reasonably flexible to capture non-linear effects. Under perfect NCC sampling, $\beta ({\tilde{\pi }}_{0i},X_i)=(g^{-1}({\tilde{\pi }}_{0i}),0^{\mathsf{T}}{)}^{\mathsf{T}}$. On the other hand, when the sampling is imperfect, the flexible model provides accurate approximation to the true sampling probabilities while overcoming the curse of dimensionality associated with NP procedures.

To estimate β(π, x), we maximize a local logistic log-likelihood using observed data on $\{(V_i,{\mathbf{Z}}_i,X_i,{\tilde{\pi }}_{0i}):{\delta }_i=0\}$. Specifically, for any given (π, x), we estimate β(π, x) as $\widehat{\beta }(\pi ,x)$, the solution to the estimating equation

$${\widehat{\mathbf{U}}}_{\pi ,x}(\beta )=N^{-1}\sum _{i=1}^n{K}_{\mathbf{b}}({\tilde{\pi }}_{0i}-\pi ,X_i-x)(1-{\delta }_i){\mathbf{Z}}_i\left\{V_i-g({\beta }^{\mathsf{T}}{\mathbf{Z}}_i)\right\}$$

where K_b(·) = (b₁b₂)⁻¹K(π/b₁)K(x/b₂), K(·) is a symmetric density function function, b = (b₁, b₂)^⊤ is the bandwidth parameters vector which tend to 0 as N → ∞. With $\widehat{\beta }(\pi ,x)$, we estimate the sampling probability for the ith subject as

$${\widehat{\pi }}_i={\delta }_i+(1-{\delta }_i){\widehat{\pi }}_{0i},\text{where}{\widehat{\pi }}_{0i}=g\{\widehat{\beta }({\tilde{\pi }}_{0i},X_i{)}^{\mathsf{T}}{\mathbf{Z}}_i\}.$$ (2.2)

Then we construct our AIPW estimator using the augmented weights ${\widehat{\omega }}_i=V_i∕{\widehat{\pi }}_i$. Under the correct specification of (2.1), we expect that ${\max }_{1\le i\le N}\mid {\widehat{\pi }}_i-{\overline{\pi }}_i\mid \to 0$ as N → ∞.

3 ∣. APPLICATION OF AIPW TO ROBUST RISK PREDICTION

In this section, we illustrate the application of the AIPW approach to developing and evaluating risk prediction models. Since one of the major goals of biomarker studies is to evaluate the predictive capacity of novel biomarkers, we consider quantifying the incremental value of Y_new in predicting T above and beyond routine markers Y_old.

3.1 ∣. Calibrated Risk Estimate

To predict risks based on $\mathbf{Y}=({\mathbf{Y}}_{\text{old}}^{\mathsf{T}},{\mathbf{Y}}_{\text{new}}^{\mathsf{T}}{)}^{\mathsf{T}}$ and Y_old, we fit two proportional hazards (PH) models,

$$\mathit{P}(\mathit{T}\ge t\mid \mathbf{Y})={\mathit{S}}_{{}_{\text{all}}}(t{)}^{\text{exp}({\gamma }_{\text{all}}^{\mathsf{T}}\mathbf{Y})},$$ (2.3)

$$\mathit{P}(\mathit{T}\ge t\mid {\mathbf{Y}}_{{}_{\text{old}}})={\mathit{S}}_{{}_{\text{old}}}(t{)}^{\text{exp}({\gamma }_{\text{old}}^{\mathsf{T}}{\mathbf{Y}}_{{}_{\text{old}}})},$$ (2.4)

where S_all(·) and S_old(·) are unknown baseline survival functions and γ_all and γ_old are the corresponding log hazard ratio parameters. To estimate γ_all and γ_old, we note that Y_new is only available for those in the NCC subcohort while Y_old is observed for all subjects. Thus, we propose to estimate γ_all by maximizing weighted log partial likelihood with AIPW weights ${\widehat{\omega }}_i$:

$${\widehat{\gamma }}_{{}_{\text{all}}}={\text{argmax}}_{\gamma }\sum _{i=1}^N{\widehat{\omega }}_i{\delta }_i\left[{\gamma }^{\mathsf{T}}{\mathbf{Y}}_i-\log \left\{\sum _{j=1}^N{\widehat{\omega }}_{j}I(X_j\ge X_i)\text{exp}({\gamma }^{\mathsf{T}}{\mathbf{Y}}_j)\right\}\right].$$

On the other hand, γ_old can be estimated as the standard maximum partial likelihood estimator, denoted by ${\widehat{\gamma }}_{{}_{\text{old}}}$. It follows from Lin and Wei²⁰ and the consistency of the sampling probabilities that ${\widehat{\gamma }}_{{}_{\text{all}}}$ and ${\widehat{\gamma }}_{{}_{\text{old}}}$ respectively converge to deterministic vectors ${\overline{\gamma }}_{{}_{\text{all}}}$ and ${\overline{\gamma }}_{{}_{\text{old}}}$ as N → ∞, regardless of the adequacy of the survival models (2.3) and (2.4). When models (2.3) and (2.4) hold, then ${\overline{\gamma }}_{{}_{\text{all}}}={\gamma }_{{}_{\text{all}}}$ and ${\overline{\gamma }}_{{}_{\text{old}}}={\gamma }_{{}_{\text{old}}}$.

To make a prediction for t-year survival, one may obtain a model-based estimate for P(T ≤ t ∣ Y) and P(T ≤ t ∣ Y_old) under (2.3) and (2.4). However, such a risk estimate may not be accurate under model mis-specifications. Following the calibrated risk prediction strategies proposed in Cai et al.²¹, we predict t-year survival risk given Y and Y_old based on

$${\mathcal{S}}_{{}_{\text{all}}}(t\mid {\mathit{R}}_{{}_{\text{all}}})\equiv \mathit{P}(\mathit{T}>t\mid {\mathit{R}}_{{}_{\text{all}}})\text{and}{\mathcal{S}}_{{}_{\text{old}}}(t\mid {\mathit{R}}_{{}_{\text{old}}})\equiv \mathit{P}(\mathit{T}>t\mid {\mathit{R}}_{{}_{\text{old}}}),$$

respectively, where ${\mathit{R}}_{{}_{\text{all}}}={\mathbf{Y}}^{\mathsf{T}}{\overline{\gamma }}_{{}_{\text{all}}}$ and ${\mathit{R}}_{{}_{\text{old}}}={\mathbf{Y}}_{{}_{\text{old}}}^{\mathsf{T}}{\overline{\gamma }}_{{}_{\text{old}}}$ are the limiting risk scores. The calibrated survival risk functions ${\mathcal{S}}_{{}_{\text{all}}}(t\mid r)$ and ${\mathcal{S}}_{{}_{\text{old}}}(t\mid r)$ can be non-parametrically estimated as ${\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid r)=\text{exp}\{-{\widehat{\Lambda }}_{{}_{\text{all}}}(t\mid r)\}$ and ${\widehat{\mathcal{S}}}_{{}_{\text{old}}}(t\mid r)=\text{exp}\{-{\widehat{\Lambda }}_{{}_{\text{old}}}(t\mid r)\}$, where

$${\widehat{\Lambda }}_{{}_{\text{all}}}(t\mid r)=\underset{0}{\overset{t}{\int }}\frac{{\sum }_i{\widehat{\omega }}_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{all},i}}-r)\mathrm{d}{N}_i(u)}{{\sum }_i{\widehat{\omega }}_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{all},i}}-r)I(X_i\ge u)},{\widehat{\Lambda }}_{{}_{\text{old}}}(t\mid r)=\underset{0}{\overset{t}{\int }}\frac{{\sum }_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{old},i}}-r)\mathrm{d}{N}_i(u)}{{\sum }_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{old},i}}-r)I(X_i\ge u)},$$

${\widehat{\mathit{R}}}_{{}_{\text{all},i}}={\widehat{\gamma }}_{{}_{\text{all}}}^{\mathsf{T}}{\mathbf{Y}}_i$, ${\widehat{\mathit{R}}}_{{}_{\text{old},i}}={\widehat{\gamma }}_{{}_{\text{old}}}^{\mathsf{T}}{\mathbf{Y}}_{{}_{\text{old},i}}$ and N_i(t) = I(X_i ≤ t)δ_i. The above calibrated risk prediction procedure essentially fits risk models (2.3) and (2.4) to summarize multi-variate risk markers into univariate risk scores, R_all and R_old, and then non-parametrically estimates the t-year risk given the risk score.

3.2 ∣. Evaluating Prediction Performance

The accuracy of the risk prediction based on a given risk score R can be summarized by commonly used time dependent accuracy measures including the true positive rate (TPR), false positive rate (FPR), the receiver operating characteristic (ROC) curve, the positive predictive value (PPV), and the negative predictive value (NPV). These prediction performance measures typically consider the accuracy of a binary classification rule R ≥ r in predicting the t-year survival status D_t = I(T ≥ t). Specifically, the TPR and FPR of R ≥ r in prediction D_t are respectively defined as

$$\text{TPR}(r\mid t)=\mathit{P}(\mathit{R}\ge r\mid \mathit{T}<t),\text{and}\text{FPR}(r\mid t)=\mathit{P}(\mathit{R}\ge r\mid \mathit{T}\ge t).$$

The ROC curve, ROC(u∣t) = TPR{FPR⁻¹(u∣t)∣t}, summarizes the trade-off between the FPR and TPR as the cut-off value varies. The PPV and NPV of R ≥ r are defined as

$$\text{PPV}(t\mid r)=\mathit{P}(\mathit{T}<t\mid \mathit{R}\ge r),\text{and}\text{NPV}(t\mid r)=\mathit{P}(\mathit{T}\ge t\mid \mathit{R}<r).$$

To estimate the prediction accuracy for R_all and R_old, we note that all the aforementioned parameters are functionals of ${\mathcal{S}}_{{}_{\text{all}}}(t\mid r)$, ${\mathcal{S}}_{{}_{\text{old}}}(t\mid r)$, ${\mathcal{F}}_{{}_{\text{all}}}(r)=P({\mathit{R}}_{{}_{\text{all}}}\le r)$ and ${\mathcal{F}}_{{}_{\text{old}}}(r)=P({\mathit{R}}_{{}_{\text{old}}}\le r)$. For example, the TPR and FPR of R_all ≥ r can be respectively written as

$${\text{TPR}}_{{}_{\text{all}}}(r\mid t)=\frac{1-{\mathcal{F}}_{{}_{\text{all}}}(r)-{\int }_r^{\infty }{\mathcal{S}}_{{}_{\text{all}}}(t\mid u)d{\mathcal{F}}_{{}_{\text{all}}}(u)}{1-\int {\mathcal{S}}_{{}_{\text{all}}}(t\mid u)d{\mathcal{F}}_{{}_{\text{all}}}(u)},\text{and}{\text{FPR}}_{{}_{\text{all}}}(r\mid t)=\frac{{\int }_r^{\infty }{\mathcal{S}}_{{}_{\text{all}}}(t\mid u)d{\mathcal{F}}_{{}_{\text{all}}}(u)}{\int {\mathcal{S}}_{{}_{\text{all}}}(t\mid u)d{\mathcal{F}}_{{}_{\text{all}}}(u)}.$$

The trade-off between TPR_all(r∣t) and FPR_all(r∣t) can be summarized based on the receiver operating characteristic (ROC) curve ${\text{ROC}}_{{}_{\text{all}}}(u\mid t)={\text{TPR}}_{{}_{\text{all}}}\{{\text{FPR}}_{{}_{\text{all}}}^{-1}(u\mid t)\mid t)\}$, where u is any specified FPR level of interest. The marginal distribution functions ${\mathcal{F}}_{{}_{\text{all}}}(r)$ and ${\mathcal{F}}_{{}_{\text{old}}}(r)$ can be respectively estimated as

$${\widehat{\mathcal{F}}}_{{}_{\text{all}}}(r)=\frac{{\sum }_{i=1}^N{\widehat{\omega }}_{i}I({\widehat{\mathit{R}}}_{{}_{\text{all},i}}\ge r)}{{\sum }_{i=1}^N{\widehat{\omega }}_i},\text{and}{\widehat{\mathcal{F}}}_{{}_{\text{old}}}(r)=N^{-1}\sum _{i=1}^N\mathit{I}({\widehat{\mathit{R}}}_{{}_{\text{old},i}}\ge r).$$

Subsequently, we may construct plug-in estimators for TPR_all(r∣t) and FPR_all(r∣t) as

$${\widehat{\text{TPR}}}_{{}_{\text{all}}}(r\mid t)=\frac{1-{\widehat{\mathcal{F}}}_{{}_{\text{all}}}(r)-{\int }_r^{\infty }{\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}(u)}{1-\int {\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}(u)}\text{and}{\widehat{\text{FPR}}}_{{}_{\text{all}}}(r\mid t)=\frac{{\int }_r^{\infty }{\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}(u)}{\int {\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}(u)},$$

respectively. Similar plug-in estimators can be constructed for other accuracy parameters. We may quantify the incremental value (IncV) of Y_new based on the difference between the accuracy of R_all and R_old. For example, the IncV of Y_new with respect to the ROC curve at FPR level of u₀ can be estimated as ${\widehat{\text{ROC}}}_{{}_{\text{all}}}(u_0\mid t)-{\widehat{\text{ROC}}}_{{}_{\text{old}}}(u_0\mid t)$, where ${\widehat{\text{ROC}}}_{{}_{\text{all}}}(u_0\mid t)={\widehat{\text{TPR}}}_{{}_{\text{all}}}\{{\widehat{\text{FPR}}}_{{}_{\text{all}}}^{-1}(u_0\mid t)\mid t\}$ and ${\widehat{\text{ROC}}}_{{}_{\text{old}}}$ is the estimated ROC curve for R_old.

3.3 ∣. Resampling Based Interval Estimation

To estimate the asymptotic variance of the proposed AIPW estimators, we propose a perturbation resampling procedure. Specifically, let I = (I₁, …, I_N)^⊤ be a vector of independent and identically distributed non-negative random variables with mean 1 and variance 1. We first obtain perturbed counterpart of $\widehat{\beta }(\pi ,x)$ as ${\widehat{\beta }}^{\ast }(\pi ,x)$, the solution to the estimating equation

$${\widehat{\mathbf{U}}}_{\pi ,x}^{\ast }(\beta )=N^{-1}\sum _{i=1}^n{\mathit{K}}_{\mathbf{b}}({\tilde{\pi }}_{0i}-\pi ,X_i-x)(1-{\delta }_i){\mathbf{Z}}_i\{V_i-g({\beta }^{\mathsf{T}}{\mathbf{Z}}_i)\}{\mathit{I}}_i.$$

Then we perturb the AIPW weights as

$${\widehat{\omega }}_i^{\ast }=\left\{{\delta }_i+(1-{\delta }_i)\frac{V_{0i}}{{\widehat{\pi }}_{0i}^{\ast }}\right\}{I}_i\text{with}{\widehat{\pi }}_{0i}^{\ast }=g\{({\widehat{\beta }}^{\ast }({\tilde{\pi }}_{0i},X_i{)}^{\mathsf{T}}{\mathbf{Z}}_i\}.$$

Subsequently, we perturb all AIPW estimators by replacing ${\widehat{\omega }}_i$ with ${\widehat{\omega }}_i^{\ast }$. Specifically, we perturb ${\widehat{\gamma }}_{{}_{\text{all}}}$ as

$${\widehat{\gamma }}_{\text{all}}^{\ast }={\text{argmax}}_{\gamma }\sum _{i=1}^N{\widehat{\omega }}_i^{\ast }{\delta }_i\left[{\gamma }^{\mathsf{T}}{\mathbf{Y}}_i-\log \left\{\sum _{j=1}^N{\widehat{\omega }}_j^{\ast }I(X_j\ge X_i)\text{exp}({\gamma }^{\mathsf{T}}{\mathbf{Y}}_j)\right\}\right],$$

and perturb ${\widehat{\mathcal{S}}}_{{}_{\text{all}}}(t\mid r)$ as ${\widehat{\mathcal{S}}}_{{}_{\text{all}}}^{\ast }(t\mid r)=\text{exp}\{-{\widehat{\Lambda }}_{{}_{\text{all}}}^{\ast }(t\mid r)\}$, where

$${\widehat{\Lambda }}_{\text{all}}^{\ast }(t\mid r)=\underset{0}{\overset{t}{\int }}\frac{{\sum }_i{\widehat{\omega }}_i^{\ast }{K}_h({\widehat{\mathit{R}}}_{{}_{\text{all},i}}^{{}^{\ast }}-r)\mathrm{d}{N}_i(u)}{{\sum }_i{\widehat{\omega }}_i^{\ast }{K}_h({\widehat{\mathit{R}}}_{{}_{\text{all},i}}^{{}^{\ast }}-r)I(X_i\ge u)},\text{and}{\widehat{\mathit{R}}}_{{}_{\text{all},i}}^{{}^{\ast }}={\mathbf{Y}}_{{}_{\text{all},i}}^{{}^{\mathsf{T}}}{\widehat{\gamma }}_{{}_{\text{all}}}^{{}^{\ast }}.$$

The accuracy parameters can be perturbed similarly. For example, we may obtain

$${\widehat{\text{TPR}}}_{{}_{\text{all}}}^{{}^{\ast }}(r\mid t)=\frac{1-{\widehat{\mathcal{F}}}_{{}_{\text{all}}}^{{}^{\ast }}(r)-{\int }_r^{\infty }{\widehat{\mathcal{S}}}_{{}_{\text{all}}}^{{}^{\ast }}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}^{{}^{\ast }}(u)}{1-\int {\widehat{\mathcal{S}}}_{{}_{\text{all}}}^{{}^{\ast }}(t\mid u)d{\widehat{\mathcal{F}}}_{{}_{\text{all}}}^{{}^{\ast }}(u)},$$

where ${\widehat{\mathcal{F}}}_{{}_{\text{all}}}^{\ast }(r)={\sum }_{i=1}^N{\widehat{\omega }}_i^{\ast }I({\widehat{\mathit{R}}}_{{}_{\text{all},i}}^{\ast }\le r)∕({\sum }_{i=1}^N{\widehat{\omega }}_i^{\ast })$.

For IncV parameters, the estimation of model parameters related to the reduced model only involve full cohort data and thus the perturbation will only involve weighting observations by {I_i}. Specifically, ${\widehat{\gamma }}_{{}_{\text{old}}}$ is perturbed as

$${\widehat{\gamma }}_{{}_{\text{old}}}^{{}^{\ast }}={\text{argmax}}_{\gamma }\sum _{i=1}^N{I}_i{\delta }_i\left[{\gamma }^{\mathsf{T}}{\mathbf{Y}}_{{}_{\text{old},i}}-\log \left\{\sum _{j=1}^N{I}_{j}I(X_j\ge X_i)\text{exp}({\gamma }^{\mathsf{T}}{\mathbf{Y}}_{{}_{\text{old},j}})\right\}\right],$$

and ${\widehat{\mathcal{S}}}_{{}_{\text{old}}}^{\ast }(t\mid r)=\text{exp}\{-{\widehat{\Lambda }}_{{}_{\text{old}}}^{\ast }(t\mid r)\}$, where

$${\widehat{\Lambda }}_{{}_{\text{old}}}^{{}^{\ast }}(t\mid r)=\underset{0}{\overset{t}{\int }}\frac{{\sum }_i{I}_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{old},i}}^{{}^{\ast }}-r)\mathrm{d}{N}_i(u)}{{\sum }_i{I}_i{K}_h({\widehat{\mathit{R}}}_{{}_{\text{old},i}}^{{}^{\ast }}-r)I(X_i\ge u)},\text{and}{\widehat{\mathit{R}}}_{{}_{\text{old},i}}^{{}^{\ast }}={\mathbf{Y}}_{{}_{\text{old},i}}^{{}^{\mathsf{T}}}{\widehat{\gamma }}_{{}_{\text{old}}}^{{}^{\ast }}.$$

Similar strategies can be used for accuracy parameters such as ${\widehat{\text{TPR}}}_{{}_{\text{old}}}^{\ast }(c\mid t)$ and ${\widehat{\text{FPR}}}_{{}_{\text{old}}}^{\ast }(c\mid t)$.

To obtain variance estimators and construct confidence intervals, we may obtain a large number, say B, of realizations of I. For each realization of I, we obtain the above perturbed estimates. The empirical distribution of the B sets of perturbed estimates can be used for inference. For example, the empirical variance of ${\widehat{\text{ROC}}}_{{}_{\text{all}}}^{\ast }(u_0\mid t)-{\widehat{\text{ROC}}}_{{}_{\text{old}}}^{\ast }(u_0\mid t)$ can be used to approximate the variance of ${\widehat{\text{ROC}}}_{{}_{\text{all}}}(u_0\mid t)-{\widehat{\text{ROC}}}_{{}_{\text{old}}}(u_0\mid t)$.

4 ∣. NUMERICAL STUDIES

We performed extensive simulations to evaluate the finite sample performance of the proposed estimators and to compare with other estimators under NCC design when the design is carried out perfectly or imperfectly. We generate Y = (Y_old, Y_new)^⊤ from a bivariate normal distribution with zero mean, unit variance and correlation 0.5. Given Y, we generate T from a PH model

$$\mathit{P}(\mathit{T}\ge t\mid \mathbf{Y})=\text{exp}\left[-\text{exp}\left\{\log (0.01t)+\log (2){\mathit{Y}}_{{}_{\text{new}}}+\log (3){\mathit{Y}}_{{}_{\text{old}}}\right\}\right].$$

The censoring time was generated from two settings: (I) C ~ C_IND = min(C_a, C_b), where C_a ~ Uniform(0.5, 2) and C_b ~ Gamma(shape = 2, rate = 2); (II) $\mathit{C}\sim {\mathit{C}}_{{}_{\text{DEP}}}=\text{min}\{{\mathit{C}}_a,{\mathit{C}}_b^{\prime }(\mathbf{Y})\}$, where ${\mathit{C}}_b^{\prime }(\mathbf{Y})=\text{exp}\{-({\mathit{Y}}_{{}_{\text{new}}}+{\mathit{Y}}_{{}_{\text{old}}})∕5\}+0.5$. This leads to covariate independent censoring in (I) and covariate dependent censoring in (II). The censoring rate and event rate (proportion of cases) are around 15% and 5%, respectively. We let N = 2000, and selected the NCC cohort by including all the cases and m = 1 control per case. Under each configuration, results were summarized based on 500 simulated datasets. We obtain estimators for ${\overline{\gamma }}_{{}_{\text{all}}}=({\overline{\gamma }}_1,{\overline{\gamma }}_2)$ in model (2.3) and TPR_u0, PPV_u0, NPV_u0 at FPR= u₀, with u₀ taken to be 0.05, 0.1, 0.2. We also compared the proposed approach with existing methods including the TIPW estimator of Cai and Zheng¹⁹, NP estimators of Zheng et al.² and conditional logistic regression method based estimator, denoted as CLR.

We considered three settings. In the first setting, setting (1), the matching covariate vector M = (M₁, M₂) with matching window a₀ = (0, 0), where $M_1={\sum }_{l=1}^{2}I({\mathit{Y}}_{{}_{\text{old}}}\le y_{q_l})$, y_q was the 100qth percentiles of Y_old and q₁ = 0.33, q₂ = 0.66, M₂ ~ Bernouli(0.5); In setting (2), matching variable M = (M₁, …, M₅)^⊤ with matching window a₀ = (0, 2, 2, 5, 0), where M₁ is the same as in setting (1), $M_2\sim \lfloor 0.3{\mathit{e}}^{\mathcal{N}}\rceil$, $M_3\sim \lfloor 5\varphi ({\mathit{Y}}_{{}_{\text{old}}}+\mathcal{N})\rceil$, M₄ ~ ⌊Uniform(0, 10)⌉, and M₅ ~ Bernouli(0.5), ϕ is a normal density function, and $\mathcal{N}\sim N(0,1)$; In setting (3), matching variable M = (M₁, M₂, M₃, M₄)^⊤ with a varying window in that we intend to match with window a₀ = (0, 0, 0, 0) but when the number of subjects is not sufficient in the risk set for some cases, we relax the criterion to matching window a = (0, 0, 2, 2) to select controls in the new risk set. Here M₁ is the same as in setting (1), $M_2=I({\mathit{Y}}_{{}_{\text{old}}}+\mathcal{N}>0)$ with $\mathcal{N}\sim N(0,1)$, $M_3\sim \lfloor 5\varphi ({\mathit{Y}}_{{}_{\text{old}}}+\mathcal{N})\rceil$ and $M_4\sim \lfloor 0.2{\mathit{e}}^{\mathcal{N}}\rceil$.

Results summarizing the performance of the proposed point and interval estimators across settings (1) - (3) are presented in Table 1-3. The point estimators have negligible biases. The average of the standard errors (ASEs) are close to the corresponding empirical standard errors (SEs), and the empirical coverage probabilities (CP) of the 95% confidence intervals are close to the nominal level. These results confirm the validity of the proposed estimation procedures in finite sample.

TABLE 1.

The Bias, empirical standard error (SE) and relative efficiency (RE) of the TIPW estimator, the proposed AIPW estimator, the nonparametric method based estimator (NP) and the CLR based estimator (CLR). For the proposed AIPW estimator, we also calculated the average of the estimated standard error (ASE), empirical coverage probabilities (CP) of the 95% CIs (×100) for settings (1).

Independent censoring (I)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.010	0.022	0.005	0.146	0.096	0.096	0.096	0.166	1.000	0.955	1.020	0.191	0.090	93.0
${\overline{\gamma }}_2$	1.099	−0.005	0.014	−0.022	0.344	0.100	0.095	0.092	0.340	1.000	1.086	1.116	0.043	0.090	93.6
TPR	0.460	−0.011	−0.010	−0.025		0.050	0.044	0.046		1.000	1.296	0.942		0.044	94.2
PPV	0.543	−0.012	−0.005	0.002		0.035	0.032	0.034		1.000	1.285	1.132		0.032	94.4
NPV	0.932	−0.001	−0.002	−0.008		0.011	0.007	0.009		1.000	2.220	0.863		0.008	96.8
TPR	0.596	−0.010	−0.009	−0.027		0.051	0.041	0.044		1.000	1.492	1.010		0.042	94.8
PPV	0.435	−0.009	−0.003	0.005		0.030	0.027	0.030		1.000	1.325	1.056		0.026	94.0
NPV	0.945	−0.001	−0.001	−0.008		0.011	0.007	0.008		1.000	2.586	1.007		0.007	97.0
TPR	0.748	−0.006	−0.005	−0.022		0.046	0.035	0.038		1.000	1.747	1.163		0.036	96.2
PPV	0.326	−0.005	−0.000	0.009		0.024	0.021	0.023		1.000	1.429	0.972		0.021	95.2
NPV	0.961	−0.001	−0.001	−0.006		0.011	0.006	0.007		1.000	2.986	1.229		0.007	97.2
Dependent censoring (II)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.018	0.044	0.007	0.086	0.116	0.111	0.115	0.169	1.000	0.952	1.027	0.382	0.106	91.9
${\overline{\gamma }}_2$	1.099	−0.004	−0.014	−0.019	0.210	0.122	0.105	0.101	0.315	1.000	1.319	1.398	0.104	0.101	93.3
TPR	0.460	−0.010	−0.005	−0.025		0.052	0.046	0.050		1.000	1.303	0.891		0.047	96.0
PPV	0.543	−0.010	−0.002	0.005		0.039	0.033	0.037		1.000	1.483	1.150		0.033	94.8
NPV	0.932	−0.001	−0.002	−0.009		0.010	0.008	0.009		1.000	1.623	0.575		0.008	96.8
TPR	0.596	−0.008	−0.004	−0.026		0.048	0.042	0.044		1.000	1.308	0.886		0.044	94.6
PPV	0.435	−0.006	−0.000	0.008		0.032	0.026	0.029		1.000	1.533	1.137		0.027	95.0
NPV	0.945	−0.001	−0.001	−0.008		0.009	0.007	0.009		1.000	1.578	0.606		0.008	96.0
TPR	0.748	−0.008	−0.004	−0.025		0.042	0.037	0.040		1.000	1.310	0.833		0.038	95.0
PPV	0.326	−0.005	0.000	0.010		0.025	0.021	0.024		1.000	1.560	0.987		0.021	95.6
NPV	0.961	−0.001	−0.001	−0.007		0.008	0.007	0.008		1.000	1.580	0.602		0.007	96.6

TABLE 3.

The Bias, empirical standard error (SE) and relative efficiency (RE) of the TIPW estimator, the proposed AIPW estimator, the nonparametric method based estimator (NP) and the CLR based estimator (CLR). For the proposed AIPW estimator, we also calculated the average of the estimated standard error (ASE), empirical coverage probabilities (CP) of the 95% CIs (×100) for settings (3).

Independent censoring (I)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.028	0.024	0.007	0.152	0.129	0.101	0.099	0.169	1.000	1.614	1.758	0.335	0.105	94.6
${\overline{\gamma }}_2$	1.099	−0.086	−0.002	−0.063	0.335	0.128	0.094	0.092	0.345	1.000	2.716	1.908	0.103	0.103	95.6
TPR	0.460	−0.039	−0.009	−0.036		0.053	0.048	0.050		1.000	1.777	1.130		0.050	94.0
PPV	0.543	0.015	−0.002	0.020		0.040	0.033	0.036		1.000	1.671	1.056		0.037	96.6
NPV	0.932	−0.017	−0.003	−0.017		0.014	0.008	0.011		1.000	6.487	1.131		0.010	98.0
TPR	0.596	−0.043	−0.006	−0.038		0.058	0.048	0.051		1.000	2.266	1.293		0.048	92.8
PPV	0.435	0.020	0.001	0.025		0.036	0.027	0.032		1.000	2.251	1.040		0.031	96.6
NPV	0.945	−0.015	−0.001	−0.015		0.014	0.008	0.011		1.000	6.411	1.182		0.009	96.2
TPR	0.748	−0.045	−0.001	−0.035		0.057	0.040	0.045		1.000	3.244	1.621		0.042	94.6
PPV	0.326	0.022	0.003	0.027		0.033	0.021	0.025		1.000	3.568	1.099		0.025	96.8
NPV	0.961	−0.014	−0.001	−0.013		0.013	0.007	0.010		1.000	6.815	1.307		0.008	96.2
Dependent censoring (II)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.019	0.032	0.003	0.074	0.138	0.115	0.116	0.159	1.000	1.372	1.443	0.629	0.109	92.1
${\overline{\gamma }}_2$	1.099	−0.082	−0.016	−0.061	0.213	0.129	0.106	0.104	0.354	1.000	2.032	1.603	0.136	0.102	93.9
TPR	0.460	−0.032	−0.001	−0.034		0.063	0.052	0.054		1.000	1.816	1.206		0.050	91.9
PPV	0.543	0.022	0.006	0.027		0.045	0.034	0.040		1.000	2.154	1.116		0.036	94.1
NPV	0.932	−0.017	−0.003	−0.019		0.014	0.009	0.012		1.000	5.339	1.012		0.010	97.0
TPR	0.596	−0.036	0.001	−0.035		0.061	0.049	0.049		1.000	2.134	1.427		0.047	93.1
PPV	0.435	0.026	0.008	0.032		0.039	0.028	0.033		1.000	2.692	1.042		0.030	94.5
NPV	0.945	−0.015	−0.002	−0.016		0.013	0.009	0.011		1.000	5.235	1.097		0.009	96.3
TPR	0.748	−0.038	0.003	−0.035		0.053	0.042	0.043		1.000	2.340	1.379		0.040	94.3
PPV	0.326	0.026	0.008	0.033		0.034	0.021	0.027		1.000	3.593	1.009		0.024	96.1
NPV	0.961	−0.014	−0.001	−0.014		0.012	0.008	0.010		1.000	4.634	1.091		0.008	97.6

In setting (1), sampling is correctly carried out and M is low dimensional, and hence all three methods (TIPW, AIPW, NP) are valid. As shown in Table 1, all three estimators have negligible biases, TIPW and NP have comparable efficiency with respect to mean squared error (MSE), and AIPW is a little more efficient than the TIPW and NP estimators. In setting (2), the sampling is carried out correctly but the matching variable is of a higher dimension, which leads to curse of dimensionality for the NP method while the TIPW remains valid. As shown in Table 2, the TIPW and AIPW both have negligible biases, while the NP exhibits higher biases. Setting (3) is a commonly encountered imperfect NCC sampling setting that is similar to the motivating example of the NHS II study. In this case, the TIPW estimator is biased as expected. There is also bias observed for the NP estimators due to the curse of dimensionality, whereas the AIPW estimator still maintains negligible bias. In addition, the AIPW estimator is substantially more efficient than both the TIPW and NP estimators with respect to MSE, with relative efficiency as high as 6 compared to the TIPW estimator and 5 compared to the NP estimator. In all the settings considered, the CLR estimator is either more biased or less efficient compared to other estimators.

TABLE 2.

The Bias, empirical standard error (SE) and relative efficiency (RE) of the TIPW estimator, the proposed AIPW estimator, the nonparametric method based estimator (NP) and the CLR based estimator (CLR). For the proposed AIPW estimator, we also calculated the average of the estimated standard error (ASE), empirical coverage probabilities (CP) of the 95% CIs (×100) for settings (2).

Independent censoring (I)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.014	0.020	−0.009	0.139	0.110	0.105	0.118	0.171	1.000	1.078	0.880	0.254	0.096	91.9
${\overline{\gamma }}_2$	1.099	−0.018	0.013	−0.097	0.343	0.109	0.097	0.145	0.339	1.000	1.263	0.399	0.052	0.096	94.0
TPR	0.460	−0.015	−0.006	−0.052		0.049	0.043	0.057		1.000	1.383	0.446		0.046	95.8
PPV	0.543	−0.003	−0.001	0.054		0.034	0.030	0.041		1.000	1.309	0.261		0.033	96.8
NPV	0.932	−0.004	−0.002	−0.036		0.011	0.008	0.017		1.000	2.124	0.085		0.009	97.2
TPR	0.596	−0.016	−0.006	−0.064		0.048	0.043	0.055		1.000	1.391	0.360		0.043	95.6
PPV	0.435	−0.001	0.000	0.057		0.029	0.025	0.040		1.000	1.302	0.176		0.027	96.6
NPV	0.945	−0.004	−0.002	−0.033		0.010	0.007	0.016		1.000	2.101	0.086		0.008	97.4
TPR	0.748	−0.013	−0.004	−0.072		0.045	0.039	0.051		1.000	1.480	0.287		0.039	94.6
PPV	0.326	0.002	0.002	0.056		0.025	0.021	0.035		1.000	1.444	0.142		0.022	95.6
NPV	0.961	−0.003	−0.001	−0.031		0.009	0.007	0.015		1.000	2.091	0.085		0.008	97.8
Dependent censoring (II)
		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.024	0.044	−0.001	0.088	0.121	0.115	0.131	0.172	1.000	1.008	0.892	0.408	0.106	91.7
${\overline{\gamma }}_2$	1.099	−0.009	−0.008	−0.098	0.230	0.130	0.110	0.133	0.331	1.000	1.389	0.622	0.104	0.102	93.6
TPR	0.460	−0.010	−0.002	−0.049		0.056	0.050	0.058		1.000	1.302	0.565		0.048	92.8
PPV	0.543	0.001	0.004	0.064		0.040	0.035	0.046		1.000	1.311	0.263		0.034	94.1
NPV	0.932	−0.004	−0.003	−0.038		0.011	0.008	0.017		1.000	2.181	0.085		0.009	95.6
TPR	0.596	−0.012	−0.003	−0.067		0.053	0.045	0.056		1.000	1.426	0.385		0.044	94.9
PPV	0.435	0.002	0.005	0.066		0.034	0.029	0.040		1.000	1.335	0.190		0.028	92.6
NPV	0.945	−0.004	−0.002	−0.036		0.011	0.007	0.016		1.000	2.360	0.080		0.008	97.9
TPR	0.748	−0.011	−0.000	−0.072		0.045	0.036	0.047		1.000	1.631	0.289		0.038	95.3
PPV	0.326	0.004	0.006	0.065		0.027	0.022	0.036		1.000	1.506	0.137		0.022	95.1
NPV	0.961	−0.003	−0.001	−0.033		0.010	0.006	0.014		1.000	2.594	0.079		0.007	97.7

To examine whether our proposed method performs well under settings with a very low event rate, we also generated data under a slight variation of the above PH model with a substantially lower baseline hazard leading to about 0.5% of event rate and independent censoring. We sampled the NCC cohort under setting (3) and obtained estimates as above. As shown in Table 4, the proposed AIPW estimates have small biases and high relative efficiencies.

TABLE 4.

The Bias, empirical standard error (SE) and relative efficiency (RE) of the TIPW estimator, the proposed AIPW estimator, the nonparametric method based estimator (NP) and the CLR based estimator (CLR). For the proposed AIPW estimator, we also calculated the average of the estimated standard error (ASE), empirical coverage probabilities (CP) of the 95% CIs (×100).

		Bias				SE				RE				AIPW
	true	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	TIPW	AIPW	NP	CLR	ASE	CP
${\overline{\gamma }}_1$	0.693	0.022	0.053	0.006	0.022	0.122	0.103	0.106	0.108	1.000	1.152	1.372	1.275	0.105	93.5
${\overline{\gamma }}_2$	1.099	−0.062	−0.008	−0.027	0.043	0.116	0.087	0.085	0.193	1.000	2.245	2.146	0.440	0.093	94.9
TPR	0.457	−0.044	−0.009	−0.036		0.047	0.038	0.043		1.000	2.623	1.305		0.038	91.5
PPV	0.163	0.004	−0.002	0.002		0.022	0.016	0.020		1.000	1.866	1.291		0.016	94.9
NPV	0.988	−0.003	−0.000	−0.002		0.002	0.001	0.002		1.000	8.090	1.539		0.001	95.9
TPR	0.601	−0.049	−0.008	−0.039		0.045	0.034	0.038		1.000	3.471	1.465		0.036	93.3
PPV	0.113	0.005	−0.000	0.004		0.014	0.010	0.012		1.000	2.334	1.446		0.010	95.9
NPV	0.991	−0.003	−0.001	−0.002		0.002	0.001	0.001		1.000	8.275	1.503		0.001	95.1
TPR	0.757	−0.047	−0.005	−0.035		0.037	0.028	0.031		1.000	4.325	1.631		0.031	96.1
PPV	0.075	0.005	−0.001	0.003		0.008	0.006	0.007		1.000	2.804	1.482		0.006	96.7
NPV	0.994	−0.003	−0.001	−0.002		0.001	0.001	0.001		1.000	8.069	1.548		0.001	93.3

5 ∣. REAL DATA ANALYSIS

High-density lipoprotein (HDL) is a protein-lipid complex which carries a range of proteins. These proteins differ in size and structure, which determines the functional properties and metabolism of HDL²². The plasma total apoA-1 concentration (WPA1) is well known to be strongly and consistently predictive of cardiovascular risk²³. In addition to apoA-1, HDL also contains other proteins including apoA2, apoC3 and apoE. ApoC3, present on 8-15% of HDL particles, has been shown to be associated with the risk of obesity and diabetes^24,25. To assess the predictiveness of these lipoprotein markers for the risk of developing myocardial infarction (MI), an NCC biomarker study was performed within the NHS II blood cohort consisting of 29,240 registered nurses enrolled around 1989²⁶. Among participants who were free of diagnosed cardiovascular disease or cancer at blood draw, 144 women were identified in the cohort with incident MI between blood draw and January 2016. Using a risk-set sampling, 144 controls were to be selected randomly and 1:1 matched on age, fasting (yes, no), smoking (never, past, current <15 cigarettes/day, current > 15 cigarettes/day, resulting in three dummy variables), and month of blood drawn. However, due to the lack of samples satisfying the matching criteria and having sufficient stored plasma for biomarker quantification, NCC design was not followed exactly during the control sampling process, yielding an imperfect NCC design. If the matching criteria is followed, the matching window should be a₀ = (0, 0, 0, 0, 2, 2). But if for some case, there is no control in its risk set, the matching criteria is relaxed but not known. For example, the age difference maybe relaxed to 5 years so that there are controls to select from for this case.

The outcome of interest is the time from blood drawn to diagnosis of MI. For an individual without an event, failure time was censored at the earlier date between the last contact date and January 2016. Routine risk factors included smoking, age, diabetes, high cholesterol, and medication for HBP. These factors are available from the full cohort. Measures of the new biomarkers, WPA1 and apoE, are only available for the NCC subcohort. To account for the subcohort sample, we fitted a weighted Cox PH model including WPA1 and apoE and other baseline clinical variables as covariates using the data from the NCC subset. Since the sampling depends on many levels of covariates, it was difficult to estimate the weights using the NP approach (Exiting packages for nonparametric estimation of the selection probability all failed). Due to the additional adjustment in matching criteria, the ‘true’ weights were not retrievable. Therefore, we calculated the weights using the proposed AIPW techniques. As presented in Table 5, more frequent smoking (>15 cig/d), having diabetes or high cholesterol, or medication for high blood pressure, and high values of apoE are significantly associated with high risk of MI. In particular apoE predicts the time to MI beyond clinical factors, with an HR of 1.427 (95% CI: 1.140, 1.786). We also considered fitting the Cox model using the weights calculated strictly from the original protocol, the IPW method. For the variable more frequent smoking (>15 cig/d) versus never smoking, the estimated HR by the AIPW method is significantly above 1 while not significantly above 1 by the IPW method, which does not reflect the findings based on the existing literature. This is as expected, as the weights in this situation do not accurately account for the sampling procedures actually implemented, and this might potentially lead to biased estimates in the main regression model. The results highlight the importance of robust procedures in the calculation of the sampling weights, though the difference between the IPW and AIPW estimators is less pronounced for new markers. The estimated effects of CLR are a little different from the IPW and AIPW estimators.

TABLE 5.

Hazard ratio (HR) estimates for MI risk using sampling weights based on the original protocol (IPW), the proposed AIPW method and CLR method.

covariate	IPW (95% CI)	AIPW (95% CI)	CLR (95% CI)
smoking (past)	0.804 (0.151, 1.457)	1.054 (0.815, 1.363)	0.723 (0.431, 1.214)
smoking <15 cig/d	0.890 (0.240, 1.541)	1.222 (0.854, 1.748)	1.067 (0.838, 1.359)
smoking >15 cig/d	1.160 (0.847, 1.473)	1.253 (1.083, 1.449)	NA
age	1.014 (0.502, 1.526)	1.156 (0.852, 1.569)	0.379 (0.077, 1.863)
diabetes	1.573 (1.346, 1.800)	1.359 (1.121, 1.649)	1.335 (1.015, 1.757)
high cholesterol	1.380 (1.073, 1.686)	1.369 (1.073, 1.747)	1.349 (1.049, 1.736)
medication for HBP	1.307 (1.063, 1.550)	1.430 (1.193, 1.714)	1.301 (1.059, 1.599)
WPA1	0.600 (0.101, 1.098)	0.730 (0.500, 1.066)	0.688 (0.496, 0.953)
apoE	1.420 (1.107, 1.733)	1.427 (1.140, 1.786)	1.214 (0.950, 1.550)

We then calculated the in-sample accuracy measures of the model scores for predicting risk of MI by 158 months (t = 158) using the proposed method. The estimates of TPR, PPV, NPV at FPR=0.05, 0.1, 0.2 and AUC for the Cox model with baseline covariates as well as WPA1 and apoE are listed in Table 6 along with the IncV of the corresponding accuracy measures compared to the performance of a Cox model without the biomarkers. Results show that adding WPA1 and apoE to the Cox model with baseline covariates leads to no significant improvement in the accuracy measures, though apoE has a significant association with time to MI.

TABLE 6.

Estimated accuracy measures for a MI risk model with clinical predictors and biomarkers WPA1 and apoE and the incremental values (incV) of WPA1 and apoE over a model with only clinical predictors.

measure	FPR	est (95% CI)	incv (95% CI)
TPR	0.050	0.265 (0.146, 0.385)	−0.022 (−0.106, 0.061)
PPV	0.050	0.021 (0.009, 0.033)	−0.001 (−0.009, 0.007)
NPV	0.050	0.997 (0.996, 0.998)	0.000 (−0.000, 0.000)
TPR	0.100	0.360 (0.234, 0.487)	0.001 (−0.081, 0.083)
PPV	0.100	0.014 (0.008, 0.021)	−0.001 (−0.005, 0.004)
NPV	0.100	0.997 (0.996, 0.998)	0.000 (−0.000, 0.000)
TPR	0.200	0.503 (0.378, 0.627)	0.025 (−0.065, 0.115)
PPV	0.200	0.010 (0.007, 0.014)	0.000 (−0.002, 0.003)
NPV	0.200	0.998 (0.997, 0.998)	0.000 (−0.000, 0.001)
AUC		0.688 (0.610, 0.765)	−0.005 (−0.057, 0.047)

6 ∣. DISCUSSION

Cost-effective two-phase sampling designs have been widely adopted in biomarker research in recent years. The nonrandom sampling of the NCC designs introduces complex data structures, which should be dealt with carefully to avoid bias. One well-recognized barrier in the analysis of two-phase designs is that the control selection procedures are often complicated in practical implementation: many matching factors are considered, and the window of selection for each variable might be adjusted in an ad-hoc fashion over the course of study, making it infeasible to retrieve the ‘true’ sampling weights. Robust nonparametric procedures for estimating the weights can consistently recover the weights according to the actual sampling, however they are limited in handling more than a few matching factors. In the case that the number of matching variables and routine markers Y_old exceed 5, the NP method of Zheng et al.² often becomes infeasible both theoretically and practically. On the other hand, our proposed AIPW method leverages the true sampling weights as a reasonable starting point and uses a sufficiently flexible model to estimate the effect on sampling of both variables involved in control selection and other correlated variables. Compared to the NP approach, the proposed AIPW procedure is able to incorporate a larger number of variables to augment the weights, while maintaining reasonable robustness and efficiency. It is important to note that matching on a large number of variables is generally not desirable since it inherently increases the chance of the matched risk sets being empty. We therefore do not recommend that in practice.

There are a couple of future directions/limitations in this line of research. The approach we proposed can easily be extended to other types of two-phase sampling such as a covariate-stratified case–cohort studies. Flexible methods are also needed to account for other practical complications in two-phase sampling. Our methods here assume a NCC study where all cases will be selected due to a low incidence rate. However in practice, due to cases and sample availability, not all cases can be sampled²⁷. This may complicate the inference procedure and warrants future research.

The R code for carrying out the proposed AIPW procedure is available upon request.

APPENDIX

APPENDIX A.

Note that in the appendixes, the derivations are with respect to the whole data and the proposed AIPW estimator, so we omit the subscript ‘all’ for notation convenience.

In this section, we show the asymptotic normality of the proposed AIPW estimator.

Assume C has a finite support [0, τ], P(T > τ) > 0 and the markers Y are continuous and bounded. The limit of $\widehat{\gamma }$, which is $\overline{\gamma }$, is in the interior of a compact parameter space Ω_γ. Suppose the regularity conditions in Andersen and Gill²⁸ hold. Similarly to Du and Akritas²⁹, we assume the kernel function K is a symmetric probability density function with finite support and bounded second derivative. In addition, we assume the joint density of $\mathit{R}={\mathbf{Y}}^{\mathsf{T}}\overline{\gamma }$, T, and C has continuous derivatives.

Denote ${\beta }_i=\beta ({\tilde{\pi }}_{0i},X_i)$, we first get the asymptotic expression of $N^{1∕2}({\widehat{\beta }}_i-{\beta }_i)$, which will be used in later derivations. Recalling that

$${\widehat{\mathbf{U}}}_{{\tilde{\pi }}_{0i},X_i}({\beta }_i)=\frac{1}{N}\sum _{j=1}^N(1-{\delta }_j)[V_j-\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)∕\{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)\}]{\mathbf{Z}}_j{K}_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j)).$$

The derivative of ${\widehat{\mathbf{U}}}_{{\tilde{\pi }}_{0i},X_i}({\beta }_i)$ with respect to β_i is

$$\frac{\partial {\widehat{\mathbf{U}}}_{{\tilde{\pi }}_{0i},X_i}({\beta }_i)}{\partial {\beta }_i}=-\frac{1}{N}\sum _{j=1}^N\frac{\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}{\{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j){\}}^2}{\mathbf{Z}}_j{\mathbf{Z}}_j^{\mathsf{T}}(1-{\delta }_j)K_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j)),$$

which converges to $-{\Sigma }_i≔-{\tilde{\pi }}_{0i}(1-{\tilde{\pi }}_{0i})E[{\mathbf{Z}}_j{\mathbf{Z}}_j^{\mathsf{T}}\mid {\tilde{\pi }}_{0j}={\tilde{\pi }}_{0i}$, $X_j=X_i]f({\tilde{\pi }}_{0i},X_i)$, where f (·, ·) is the density function of $({\tilde{\pi }}_{0i},X_i)$. It follows that

$$N^{1∕2}({\widehat{\beta }}_i-{\beta }_i)={\Sigma }_i^{-1}{N}^{-1∕2}\sum _{j=1}^N[V_j-\frac{\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}]{\mathbf{Z}}_j(1-{\delta }_j)K_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j))+o_p(1).$$ (A.1)

For the proposed AIPW estimators with a general form

$$\widehat{U}=N^{-1∕2}\sum _{i=1}^N{\widehat{\omega }}_i{U}_i,$$ (A.2)

where $E(U_i)=0$, ${\widehat{\omega }}_i=V_i∕{\widehat{\pi }}_i$ and ${\widehat{\pi }}_i={\delta }_i+(1-{\delta }_i){\widehat{\pi }}_{0i}$, we have

$$\begin{gathered} \widehat{U}=N^{-1∕2}\sum _{i=1}^N{\widehat{\omega }}_i{U}_i=N^{-1∕2}\sum _{i=1}^N{U}_i+N^{-1∕2}\sum _{i=1}^N({\tilde{\omega }}_i-1)U_i+N^{-1∕2}\sum _{i=1}^N({\widehat{\omega }}_i-{\tilde{\omega }}_i)U_i \\ \equiv I_1+I_2+I_3, \end{gathered}$$

$$\begin{gathered} \text{where}{I}_3=N^{-1∕2}\sum _{i=1}^N{V}_i(\frac{1}{{\widehat{\pi }}_{0i}}-\frac{1}{{\tilde{\pi }}_{0i}})U_i=-N^{-1∕2}\sum _{i=1}^N{V}_i\frac{{\widehat{\pi }}_{0i}-{\tilde{\pi }}_{0i}}{{\widehat{\pi }}_{0i}{\tilde{\pi }}_{0i}}{U}_i \\ =-N^{-1∕2}\sum _{i=1}^N{\tilde{\omega }}_i\frac{{\widehat{\pi }}_{0i}-{\tilde{\pi }}_{0i}}{{\widehat{\pi }}_{0i}}{U}_i=-N^{-1∕2}\sum _{i=1}^N{\tilde{\omega }}_i{U}_i\frac{{\widehat{\pi }}_{0i}-{\tilde{\pi }}_{0i}}{{\tilde{\pi }}_{0i}}+o_p(1) \\ =-N^{-1∕2}\sum _{i=1}^N{\tilde{\omega }}_i{U}_i\frac{\stackrel{.}{g}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_i)}{g({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_i)}{\mathbf{Z}}_i^{\mathsf{T}}({\widehat{\beta }}_i-{\beta }_i)+o_p(1) \\ =-N^{-1}\sum _{i=1}^N{\tilde{\omega }}_i{U}_i(1-{\tilde{\pi }}_{0i}){\mathbf{Z}}_i^{\mathsf{T}}{\Sigma }_i^{-1}{N}^{-1∕2}\sum _{j=1}^N(V_j-\frac{\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}){\mathbf{Z}}_j(1-{\delta }_j)K_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j))+o_p(1) \\ =-N^{-1∕2}\sum _{j=1}^{N}E[U_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j},X_i=X_j]E[{\mathbf{Z}}_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j},X_i=X_j{]}^{-1}\times ({\tilde{\omega }}_j-1){\mathbf{Z}}_j(1-{\delta }_j)+o_p(1) \\ =-N^{-1∕2}\sum _{j=1}^N({\tilde{\omega }}_j-1)(1-{\delta }_j){\Pi }_j+o_p(1), \end{gathered}$$

where ${\Pi }_j=E[U_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j}$, $X_i=X_j]E[{\mathbf{Z}}_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j}$, X_i = X_j]⁻¹Z_j, which can be regarded as a linear (conditional) projection of U_j onto the space of Z_j under the inner product ⟨X_i, Y_i⟩ = E(X_iY_i). Also note that $E[U_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j},X_i=X_j]E[{\mathbf{Z}}_i{\mathbf{Z}}_i^{\mathsf{T}}\mid {\tilde{\pi }}_{0i}={\tilde{\pi }}_{0j},X_i=X_j{]}^{-1}$ is the minimizer of

$$\frac{1}{N}\sum _{i=1}^N(U_i-\theta {\mathbf{Z}}_i{)}^2{K}_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j))$$

with respect to θ. So $E[{\mathbf{Z}}_i(U_i-{\Pi }_i)\mid {\tilde{\pi }}_{0i},X_i]=0$. Since the first component of Z_j is one, we have that $E[(U_i-{\Pi }_i)\mid {\tilde{\pi }}_{0i},X_i]=0$. So $\widehat{\mathit{U}}$ can be rewritten as

$$\widehat{U}=N^{-1∕2}\sum _{i=1}^N{U}_i+N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)({\tilde{\omega }}_i-1)(U_i-{\Pi }_i)+o_p(1).$$ (A.3)

It follows from Cai and Zheng¹ that $\widehat{\mathit{U}}$ is asymptotically normal, with asymptotic variance

$$\begin{gathered} {\Sigma }_U=E(U_i^2)+E{N}^{-1}\sum _{i=1}^N(1-{\delta }_i)({\tilde{\omega }}_i-1{)}^2(U_i-{\Pi }_i{)}^2+o_p(1) \\ =E(U_i^2)+E[(\frac{1-{\tilde{\pi }}_{0i}}{{\tilde{\pi }}_{0i}})(1-{\delta }_i)(U_i-{\Pi }_i{)}^2]+o_p(1). \end{gathered}$$

Because the interaction term is

$$\begin{gathered} E[N^{-1}\sum _{i\ne j}^N({\tilde{\omega }}_i-1)({\tilde{\omega }}_j-1)(U_i-{\Pi }_i)(U_j-{\Pi }_j)] \\ =(N-1)E\text{Cov}\left({\tilde{\omega }}_i\{U_i-{\Pi }_i\},{\tilde{\omega }}_j\{U_j-{\Pi }_j\}\mid \mathcal{D}\right) \\ =-m(N-1)∕N\int \eta (t,X_i,{\delta }_i)\eta (t,X_j,{\delta }_j)\frac{\mathrm{d}{\Lambda }_{NCC}(t)}{P(X\ge t)}=0, \end{gathered}$$

where ${\Lambda }_{NCC}(t)={\int }_0^t\mathrm{d}{\mathit{A}}_{NCC}(u)∕\mathit{P}(X\ge u),{\mathit{A}}_{NCC}(t)=E\{N_i(t)\}$ and

$$\eta (t,X_i,{\delta }_i)E[\{U_i-{\Pi }_i\}I(X_i\ge t)(1-{\tilde{\pi }}_{0i})∕{\tilde{\pi }}_{0i}]=E\left(E[\{U_i-{\Pi }_i\}I(X_i\ge t)(1-{\tilde{\pi }}_{0i})∕{\tilde{\pi }}_{0i}\mid {\tilde{\pi }}_{0i},X_i]\right)=0$$ (A.4)

by the arguments before (A.3) and similar arguments to those of Samuelsen¹⁶.

From Cai and Zheng¹, we know that the asymptotic variance of the TIPW estimator $\widehat{\mathit{U}}=N^{-1∕2}{\sum }_{i=1}^N{\tilde{\omega }}_i{\mathit{U}}_i$ is

$$\begin{gathered} {\Sigma }^{TIPW}=E(U_i^2)+E(U_i^2\frac{1-{\tilde{\pi }}_{0i}}{{\tilde{\pi }}_{0i}})-m\int {\eta }_u(t,X_i,{\delta }_i{)}^2\frac{\mathrm{d}{\Lambda }_{NCC}(t)}{P(X\ge t)}+o_p(1) \\ =E(U_i^2∕{\tilde{\pi }}_{0i})-m\int {\eta }_u(t,X_i,{\delta }_i{)}^2\frac{\mathrm{d}{\Lambda }_{NCC}(t)}{P(X\ge t)}+o_p(1), \end{gathered}$$

where ${\eta }_u(t,X_i,{\delta }_i)=E[U_{i}I(X_i\ge t)(1-{\tilde{\pi }}_{0i})∕{\tilde{\pi }}_{0i}]$.

Comparing these two asymptotic variances, we have

$$\begin{gathered} {\Sigma }^{TIPW}-{\Sigma }_U=E[(1-{\delta }_i)(\frac{1-{\tilde{\pi }}_{0i}}{{\tilde{\pi }}_{0i}})\{U_i^2-(U_i-{\Pi }_i{)}^2\}]-m\int {\eta }_u(t,X_i,{\delta }_i{)}^2\frac{\mathrm{d}{\Lambda }_{NCC}(t)}{P(X\ge t)} \\ =E\{(1-{\delta }_i)(\frac{1-{\tilde{\pi }}_{0i}}{{\tilde{\pi }}_{0i}}){\Pi }_i^2\}-m\int {\eta }_u(t,X_i,{\delta }_i{)}^2\frac{\mathrm{d}{\Lambda }_{NCC}(t)}{P(X\ge t)} \\ =var\{N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)({\tilde{\omega }}_i-1){\Pi }_i\}\ge 0, \end{gathered}$$

where the last equality holds similarly to (A.4). That is, $E[U_{i}I(X_i\ge t)(1-{\tilde{\pi }}_{0i})∕{\tilde{\pi }}_{0i}]=E[{\Pi }_{i}I(X_i\ge t)(1-{\tilde{\pi }}_{0i})∕{\tilde{\pi }}_{0i}]$. Therefore, the proposed AIPW estimators are more efficient than the true weight based TIPW estimators.

APPENDIX B.

Now we derive the specific forms of U_i in the general form (A.2) for all the related estimators of interest. Then the asymptotic variances of these estimators can be obtained using the results in Appendix A.

For $\widehat{\gamma }$, similarly to Cai and Zheng¹, we have that

$$\begin{gathered} N^{1∕2}(\widehat{\gamma }-\overline{\gamma })=N^{-1∕2}\sum _{i=1}^N{\widehat{\omega }}_i{U}_{\overline{\gamma }i}+o_p(1), \\ \text{where}{U}_{\overline{\gamma }i}=D(\overline{\gamma }{)}^{-1}\int \{{\mathbf{Y}}_i-\frac{I^{(1)}(t)}{I^{(0)}(t)}\}\mathrm{d}{M}_i(t), \\ D(\overline{\gamma })=N^{-1}\sum _{i=1}^N{\delta }_i\left\{\frac{I^{(2)}(X_i)I^{(0)}(X_i)-I^{(1)}(X_i{)}^{\otimes 2}}{I^{(0)}(X_i{)}^{\otimes 2}}\right\}, \\ {I}^{(k)}(t,\gamma )=N^{-1}\sum _{i=1}^N{\widehat{\omega }}_{i}I(X_i\ge t)\text{exp}({\mathit{Y}}_i^{\mathsf{T}}\gamma ){\mathbf{Y}}_i^k,k=0,1,2, \\ {I}^{(k)}(t)=N^{-1}\sum _{i=1}^N{\widehat{\omega }}_{i}I(X_i\ge t)\text{exp}({\mathit{Y}}_i^{\mathsf{T}}\overline{\gamma }){\mathbf{Y}}_i^k,k=0,1,2, \\ {A}_i(t)=\underset{0}{\overset{t}{\int }}I(X_i\ge u)\text{exp}({\mathit{Y}}_i^{\mathsf{T}}\overline{\gamma })\mathrm{d}{\Lambda }_0(u), \\ \text{and}{M}_i(t)=N_i(t)-A_i(t). \end{gathered}$$

For $\widehat{\Lambda }(t\mid r)$, we have

$$\begin{gathered} N^{1∕2}\{\widehat{\Lambda }(t\mid r)-\Lambda (t\mid r)\} \\ =N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)\mathrm{d}{N}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)I(X_i\ge u)}-N^{1∕2}\Lambda (t\mid r) \\ =N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)I(X_i\ge u)} \\ =N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\widehat{\gamma }}^{\mathsf{T}}{\mathbf{Y}}_i-r)I(X_i\ge u)}-N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)I(X_i\ge u)}+N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)I(X_i\ge u)} \\ =\left[\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{\stackrel{.}{K}}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)∕h{\mathbf{Y}}_i\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)I(X_i\ge u)}-\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)\mathrm{d}{M}_i(u)\{{\sum }_{i=1}^N{\widehat{\omega }}_i{\stackrel{.}{K}}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)∕h{\mathbf{Y}}_{i}I(X_i\ge u)\}}{\{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)I(X_i\ge u){\}}^2}\right]N^{1∕2}(\widehat{\gamma }-\overline{\gamma }) \\ +N^{1∕2}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)\mathrm{d}{M}_i(u)}{{\sum }_{i=1}^N{\widehat{\omega }}_i{K}_h({\mathbf{Y}}_i^{\mathsf{T}}\overline{\gamma }-r)I(X_i\ge u)}. \end{gathered}$$

So the U_i form in (A.2) for $\widehat{\Lambda }(t\mid r)$ is

$$\begin{gathered} U_{\Lambda i}(t\mid r)=U_{\overline{\gamma }i}^{\mathsf{T}}[\underset{0}{\overset{t}{\int }}\frac{N^{-1}{\sum }_{j=1}^N{\widehat{\omega }}_j{\stackrel{.}{K}}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})∕h{\mathbf{Y}}_j\mathrm{d}{M}_j(u)}{N^{-1}{\sum }_{j=1}^N{\widehat{\omega }}_j{K}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})I(X_j\ge u)}\phantom{]} \\ -\phantom{[}\underset{0}{\overset{t}{\int }}\frac{{\sum }_{j=1}^N{\widehat{\omega }}_j{K}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})\mathrm{d}{M}_j(u)\{{\sum }_{l=1}^N{\widehat{\omega }}_l{\stackrel{.}{K}}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})∕h{\mathbf{Y}}_{l}I(X_l\ge u)\}}{\{{\sum }_{j=1}^N{\widehat{\omega }}_j{K}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})I(X_j\ge u){\}}^2}]+\underset{0}{\overset{t}{\int }}\frac{K_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})\mathrm{d}{M}_j(u)}{N^{-1}{\sum }_{j=1}^N{\widehat{\omega }}_j{K}_h({\mathbf{Y}}_j^{\mathsf{T}}\overline{\gamma }-\mathit{r})I(X_j\ge u)}. \end{gathered}$$

Recalling that $\widehat{S}(t\mid r)=\text{exp}\{-\widehat{\Lambda }(t\mid r)\}$, we have that the U_i form in (A.2) for $\widehat{S}(t\mid r)$ is

$$U_{Si}(t\mid r)=-S(t\mid r)U_{\Lambda i}(t\mid r).$$

Recalling that $\widehat{F}(r)=\frac{\sum _{i=1}^N{\widehat{\omega }}_{i}I({\widehat{\mathit{R}}}_{{}_{\text{all},i}}\le r)}{\sum _{i=1}^N{\widehat{\omega }}_i}$, we get that the U_i form in (A.2) for $\widehat{F}(r)$ is

$$U_{Fi}(r)=I({\mathit{R}}_i\le r)-F(c)+D_{\overline{\gamma }}(r)U_{\overline{\gamma }i},\text{where}{D}_{\overline{\gamma }}(r)=\partial E[I({\mathit{R}}_i\le r)]∕\partial \gamma {\mid }_{\gamma =\overline{\gamma }}.$$

Recalling $\widehat{S}(r,t)={\int }_r^{\infty }\widehat{S}(t\mid u)\mathrm{d}\widehat{F}(u)$, we have that the U_i form in (A.2) for $\widehat{S}(r,t)$ is

$$U_{Si}(t,r)=\underset{r}{\overset{\infty }{\int }}{U}_{Si}(t\mid u)\mathrm{d}F(u)+\underset{r}{\overset{\infty }{\int }}S(t\mid u)\mathrm{d}{U}_{F_i}(u).$$

It follows that of U_i forms for the accuracy parameter estimators are

$$\begin{gathered} U_{TP{R}_{t}i}(r)=\frac{TP{R}_t(r)U_{Si}(t,r_l)-U_{Fi}(r)-U_{Si}(t,r)}{1-S(t)}, \\ {U}_{FP{R}_{t}i}(r)=\frac{U_{Si}(t,r)-FP{R}_t(r)U_{Si}(t,r_l)}{S(t)}, \\ {U}_{PP{V}_{t}i}(r)=\frac{\{PP{V}_t(r)-1\}{U}_{Fi}(r)-U_{Si}(t,r)}{1-F(r)}, \\ {U}_{NP{V}_{t}i}(r)=\frac{U_{Si}(t)-U_{Si}(t,r)-NP{V}_t(r)U_{Fi}(r)}{F(r)}. \end{gathered}$$

Thus, we get the forms of U_i in (A.2) for the regression parameter estimator $\widehat{\gamma }$ and the accuracy parameter estimators $\widehat{\text{TPR}}(c\mid t)$, $\widehat{\text{FPR}}(c\mid t)$, $\widehat{\text{PPV}}(c\mid t)$, $\widehat{\text{NPV}}(c\mid t)$.

APPENDIX C.

In this section, we show the validity of the proposed resampling technique.

The derivative of ${\widehat{\mathbf{U}}}_{{\tilde{\pi }}_{0i},X_i}^{\ast }({\beta }_i)$ with respect to ${\beta }_i^{\ast }$ is

$$\frac{\partial {\widehat{\mathbf{U}}}_{{\tilde{\pi }}_{0i},X_i}^{\ast }({\beta }_i^{\ast })}{\partial {\beta }_i^{\ast }}=-\frac{1}{N}\sum _{j=1}^N{I}_j\frac{\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}{\{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j){\}}^2}{\mathbf{Z}}_j{\mathbf{Z}}_j^{\mathsf{T}}(1-{\delta }_j)K_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j))+o_p(1),$$

which also converges to −Σ_i. It follows that

$$N^{1∕2}({\widehat{\beta }}_i^{\ast }-{\beta }_i)={\Sigma }_i^{-1}{N}^{-1∕2}\sum _{j=1}^N{I}_j(V_j-\frac{\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}{1+\text{exp}({\beta }_i^{\mathsf{T}}{\mathbf{Z}}_j)}){\mathbf{Z}}_j(1-{\delta }_j)K_b(({\tilde{\pi }}_{0i},X_i)-({\tilde{\pi }}_{0j},X_j))+o_p(1).$$

The perturbed form of (A.2) is

$$\begin{gathered} {\widehat{U}}^{\ast }=N^{-1∕2}\sum _{i=1}^N[{\delta }_i{I}_i+(1-{\delta }_i)V_{0i}{I}_i∕{\widehat{\pi }}_{0i}^{\ast }]U_i \\ =N^{-1∕2}\sum _{i=1}^N\{I_i{\delta }_i+(1-{\delta }_i)I_i+(1-{\delta }_i)(\frac{V_{0i}}{{\tilde{\pi }}_{0i}}-1)I_i+(1-{\delta }_i)[\frac{V_{0i}{I}_i}{{\widehat{\pi }}_{0i}^{\ast }}-\frac{V_{0i}{I}_i}{{\tilde{\pi }}_{0i}}]\}{U}_i \\ =N^{-1∕2}\sum _{i=1}^N{I}_i{U}_i+N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)I_i(\frac{V_{0i}}{{\tilde{\pi }}_{0i}}-1)U_i-N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)\frac{V_{0i}{I}_i}{{\tilde{\pi }}_{0i}}\frac{{\widehat{\pi }}_{0i}^{\ast }-{\tilde{\pi }}_{0i}}{{\widehat{\pi }}_{0i}^{\ast }}{U}_i \\ =N^{-1∕2}\sum _{i=1}^N{I}_i{U}_i+N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)I_i({\tilde{\omega }}_i-1)(U_i-{\Pi }_i)+o_p(1), \end{gathered}$$

where the last equation follows similarly to the derivation of I₃ in Appendix A.

From (A.3), we know

$$\widehat{U}=N^{-1∕2}\sum _{i=1}^N{U}_i+N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)({\tilde{\omega }}_i-1)(U_i-{\Pi }_i)+o_p(1).$$

It follows that

$${\widehat{U}}^{\ast }-\widehat{U}=N^{-1∕2}\sum _{i=1}^N(I_i-1)U_i+N^{-1∕2}\sum _{i=1}^N(1-{\delta }_i)(I_i-1)({\tilde{\omega }}_i-1)(U_i-{\Pi }_i)+o_p(1).$$

Therefore,

$$\text{Var}({\widehat{U}}^{\ast }-\widehat{U}\mid \mathcal{D})=\text{Var}(\widehat{U}).$$