New weighting methods when cases are only a subset of events in a nested case-control study

Abstract

Nested case-control (NCC) is a sampling method widely used for developing and evaluating risk models with expensive biomarkers on large prospective cohort studies. In a typical NCC design, biomarker values are obtained on a sub-cohort, where cases consist of all the events (subjects who experience the event during the follow-up). However, when the number of events is not small, due to the cost and limited availability of bio-specimen, one may select only a subset of events as cases. We refer to such a variation as the untypical NCC. Unfortunately, existing inverse probability weighted (IPW) estimators for the untypical NCC are biased, and they only focus on relative risk parameters under the proportional hazards (PH) model. In this manuscript, we propose new weighting methods that produce consistent IPW estimators for not only relative risk parameters but also several metrics that evaluate a risk model’s predictive performance. We also provide the inference procedure via perturbation resampling, which captures all the variance and between-subject covariance induced by the sampling processes for both case and control selections. In addition, our methods are not limited to the PH model, and they can be applied to the time-specific generalized linear model. Under the typical NCC design, our new weights are equivalent to the weight proposed in Samuelsen; under the untypical NCC, the IPW estimators using our weights have smaller bias and variance than the existing methods. We will demonstrate this improved performance via both analytical and numerical investigations.

1. Introduction

Risk prediction using novel biomarkers plays a vital role in disease prevention and management. The development and evaluation of risk models require rich information from large-scale cohort studies where participants are followed prospectively to the clinical outcome of interest, and their clinical information is collected at baseline. Many cohorts also obtain biological specimens that are used later for investigating new biomarkers to improve the predictive capacity of the risk model. Often, the cost and effort are expensive for ascertaining biomarkers on a large population, and there is a need to preserve precious biological samples. Thus, sub-cohort sampling such as nested case-control (NCC) is often employed, where new biomarkers are measured on the sub-cohort instead of the entire cohort.

An NCC sub-cohort is constructed in two steps: first, cases are selected, and second, for each case, a number of controls are selected from a group of subjects who are event-free at the failure time of the case; this group is referred to as the risk set of the case. For a typical NCC study, all the events (subjects who encounter the event during the follow-up) are included as cases at the first step. However, when the number of events is not small, the above-described practical constraints prevent using all the events to save cost or samples.

To address this challenge, some studies modified the typical NCC design by selecting only a subset of events as cases. We refer to this variation as the untypical NCC. For example, Jakszyn et al. (2006) investigated the association of Helicobacter pylori infection and vitamin C levels with the risk of gastric cancer. They included 229 out of 314 gastric cancer patients as cases, and for each case, two to four controls were selected; the same design was used in Jakszyn et al. (2012). Lü et al. (2018) conducted a multi-center study on fatality, hospitalization costs, and length of stay due to healthcare-associated infection (HAI). They selected 10 HAI events as cases from each of the 51 hospitals, and for each case, one control is selected and matched on several criteria. Besides these applications, Edelmann et al. (2020) and Graziano et al. (2021) compared the untypical NCC with other sub-cohort sampling methods, such as case-cohort, via simulation studies. Unlike these two papers, our manuscript focuses on just NCC, particularly the untypical variation.

For analyzing typical NCC data, conditional logistic regression has been widely used to estimate hazard ratios under the proportional hazards (PH) model (Goldstein and Langholz 1992). Another popular approach is the inverse probability weighted (IPW) estimation (Samuelsen 1997. Cai and Zheng 2012: 2011, Zhou et al. 2015). Compared to conditional logistic regression, the IPW can be applied to a variety of models, including the PH model and time-specific generalized linear model (GLM) (Uno et al. 2007). Samuelsen (1997) provided the weight formula: the inverse of the selection probability. For the typical NCC, since all the events are selected as cases at the first step, their selection probability is 1, and thus, their weights are 1.

What about the untypical NCC? Conditional logistic regression would still be valid, but some studies misused it. For example, Jakszyn et al. (2006) fit an unconditional logistic regression but included the matching variables as the covariates in the regression. Edelmann et al. (2020) and Graziano et al. (2021) employed the IPW method. However, their weights for the events in the sub-cohort are different: Edelmann et al. (2020) still used 1, but Graziano et al. (2021) used $1/{\pi }_1$, where ${\pi }_1$ is the proportion of the events that are selected as cases at the first step.

Unfortunately, our investigations show that these two existing IPW estimators are both biased. It is because they failed to recognize the fact that for the untypical NCC, some events would be selected as cases at the first step, and others would be selected only at the second step through the control selection. To differentiate these two scenarios, we refer to the former as the event case and the latter as the event control. Because of this difference, we need to carefully design the weight for avoiding biased estimation and improving estimation efficiency.

In this manuscript, we propose three weighting methods that all result in consistent IPW estimators. Our first two methods weight event cases and event controls differently based on how they are selected. In contrast, our third weight has the same formula for these two groups, which is the inverse of the overall selection probability: the probability of being ever selected to the sub-cohort as a case or a control. For this weighting method, we formulate the NCC design as a two-stage stratified sampling framework described in Breslow et al. (2009). Under this framework, our third weight follows the method of Horvitz and Thompson (1952). We will prove that the expectation of each weight given the data is either exactly or asymptotically 1, which is the condition of the consistency for the IPW estimator.

Statistical inference is another challenge for the IPW estimation under the untypical NCC. It is difficult to obtain an analytical variance estimate for the IPW estimator because the variance expression is complicated. Standard resampling procedures such as bootstrap fail to capture the correlation induced by the finite-population sampling (Gray 2009). Cai and Zheng (2013) proposed a perturbation resampling method for the typical NCC, and their approach accounts for only between-control correlations. However, for the untypical NCC, sampling processes are involved for both case and control selections. Thus, we propose a new perturbation procedure that captures both between-case and between-control correlations.

In this manuscript, we provide IPW estimators under both the PH model and time-specific GLM. The model parameters characterize relative risks, such as log hazard ratios for the PH model, which are the focus of all the existing works on the untypical NCC. Besides these parameters, we are also interested in several time-dependent accuracy summaries for evaluating the risk model’s predictive performance, such as the true positive rate (TPR), false positive rate (FPR), positive predictive value (PPV), negative predictive value (NPV), and area under a receiver operating characteristic (ROC) curve (AUC) (Heagerty and Zheng 2005, Cai and Zheng 2012; 2011, Zhou et al. 2015). For allowing the public to implement our methods, we create an R package NCCIPW, available at https://github.com/michellezhou2009/NCCIPW.git.

The remaining manuscript is organized as follows. In Section 2, we introduce our three new weights and present the IPW estimators for model and accuracy parameters. In Section 3, we derive and compare the asymptotic properties of the IPW estimators using our weights. We also describe our perturbation resampling procedure for drawing inferences on the parameters. We compare our methods with the existing approaches via simulation studies in Section 4 and a data example in Section 5. Concluding remarks are given in Section 6.

2. IPW Estimation with Three New Weights

In this section, we introduce our three new weights and their resulting IPW estimators for model and accuracy parameters. Under the typical NCC, the three weights are equivalent to the weight of Samuelsen (1997). Under the untypical NCC, their differences are displayed in Table 1, which lists the expressions of each weight for three groups of subjects in the sub-cohort: event cases, event controls, and non-event controls. We have defined event cases and event controls earlier: they have the same disease status but enter the sub-cohort in different ways. Non-event controls are subjects who are event-free during the follow-up and selected as controls. Before these descriptions, we first introduce notation that we will use throughout the manuscript.

Table 1:

Expressions of our three weights: ${\tilde{w}}_j,{\widehat{w}}_j,w_{\text{HT},j}$ for three groups of subjects in the subcohort under the untypical NCC, i.e., ${\pi }_1<1$ : (i) events cases, events that are selected as cases, (ii) events controls, events that are selected only through the control selection, (iii) non-events controls, non-events that are selected as controls. Note that some event cases might be selected as other cases’ control, and thus, their $V_{0j}$ could be 1 or 0.

	$({\delta }_j,V_{1j},V_{0j})$	${\tilde{w}}_j$	${\tilde{w}}_j$	$w_{\text{HT},j}$
event cases	(1, 1, $V_{0j}$)	1	$1/{\pi }_1$	$1/[{\pi }_1+(1-{\pi }_1){\widehat{p}}_{0j}]$
event controls	(1, 0, 1)	$1/{\widehat{p}}_{0j}$	0	$1/[{\pi }_1+(1-{\pi }_1){\widehat{p}}_{0j}]$
non-event controls	(0, 0, 1)	$1/{\widehat{p}}_{0j}$	$1/{\widehat{p}}_{0j}$	$1/{\widehat{p}}_{0j}$

2.1. Notation

An NCC data set includes variables on survival outcome, markers, and sub-cohort sampling indicators. For survival outcome, let $T_j^{†}$ denote the time to the event of interest. For some subjects, $T_j^{†}$ might not be observed due to censoring; for instance, the subject is lost to follow-up, or the follow-up ends. Let ${\tilde{C}}_j$ denote a random censoring time, and let $\tau$ denote the follow-up duration. For each subject, we only observe $T_j=\text{min}\left(T_j^{†},C_j\right)$ and ${\delta }_j=I\left(T_j^{†}\le C_j\right)$, where $C_j=\text{min}\left\{{\tilde{C}}_j,\tau \right\}$, and $I(\cdot )$ is an indicator function. Subjects who experience the event during the follow-up, i.e., with ${\delta }_j=1$ are referred to as events, and those with ${\delta }_j=0$ are referred to as non-events. Let ${\mathbf{Z}}_j$ denote a $p$-dimensional vector of clinical markers and biomarkers.

We define two sub-cohort sampling indicator variables: $V_{1j}=1$ if subject $j$ is selected as a case, and $V_{0j}=1$ if subject $j$ is selected as a control. Let $V_j={\delta }_j\left[V_{1j}+\left(1-V_{i1}\right)V_{0j}\right]+\left(1-{\delta }_j\right)V_{0j}$ indicate whether subject $j$ is ever selected to the sub-cohort. In an NCC data set, $\left(T_j,{\delta }_j,V_{1j},V_{0j}\right)$ and clinical markers are observed on the full cohort, but biomarkers are usually available for only subjects with $V_j=1$.

Furthermore, we express the control indicator variable as

$$V_{0j}=1-\prod _{i:j\in {ℛ}_i}\left(1-V_{1i}{V}_{0j}^i\right),$$ (1)

where $V_{0j}^i$ is an indictor variable: $V_{0j}^i=1$ if subject $j$ is a control of case subject $i$, and ${ℛ}_i$ denotes the risk set for subject $i$. Specifically, ${ℛ}_i=\left\{k:1\le k\le n,T_k\ge T_i\right\}$ consists of all the subjects who have not experienced the event before subject $i\text{'}$s event time. For some studies, the controls are also matched on some variables. For these situations, the risk set is expressed as ${ℛ}_i=\{k:1\le \left.k\le n,T_k\ge T_i,\left|{\mathbf{M}}_k-{\mathbf{M}}_i\right|\le {\mathbf{a}}_0\right\}$, where ${\mathbf{M}}_i$ is a vector of matching variables, and $\left|\mathbf{a}\right|\le {\mathbf{a}}_0$ denotes $\left|\mathbf{a}\right|$ being less than or equal to ${\mathbf{a}}_0$ component-wise.

In addition, as described earlier, ${\pi }_1\in \left(\mathrm{0,1}\right]$ is the proportion of events that are selected as cases. When ${\pi }_1=1$, all the events are cases, and this study is a typical NCC; when ${\pi }_1<1$, it is an untypical variation.

2.2. Three New Weights

Our first two methods assign different weights to event cases and event controls. Like Edelmann et al. (2020) and Graziano et al. (2021), we consider two different values: 1 or $1/{\pi }_1$ for event cases. To make the expectation of each weight to be 1, we need to weight event controls correspondingly. Specifically, our first weight is defined as

$${\tilde{w}}_j={\delta }_j{V}_{1j}+\left(1-{\delta }_j{V}_{1j}\right)V_{0j}/{\widehat{p}}_{0j},$$ (2)

and the second weight is

$${\widehat{w}}_j={\delta }_j{V}_{1j}/{\pi }_1+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j},$$ (3)

where

$${\widehat{p}}_{0j}=1-\prod _{i:j\in {ℛ}_i}\left\{1-\frac{m}{{\mathfrak{n}}_i-1}{\delta }_i{V}_{1i}\right\}$$ (4)

is the probability of subject $j$ being selected as a control, $m$ is the number of controls for each case, and ${\mathfrak{n}}_i$ is the size of the risk set ${ℛ}_i$ for subject $i$. By Equations (2) and (3), we have: for non-event controls, ${\tilde{w}}_j={\widehat{w}}_j=1/{\widehat{p}}_{0j};$ for event cases, ${\tilde{w}}_j=1$ but ${\widehat{w}}_j=1/{\pi }_1;$ for event controls, ${\tilde{w}}_j=1/{\widehat{p}}_{0j}$ but ${\widehat{w}}_j=0$.

We want to point out that our formula of ${\hat{p}}_{0j}$ is different from Equation (2.6) in Samuelsen (1997), given as ${\widehat{p}}_{0j}=1-{\prod }_{i:j\in {ℛ}_i} \left\{1-\frac{m}{{\mathfrak{n}}_i-1}{\delta }_i\right\}$, which does not contain $V_{1i}$. We modify this formula because, under the untypical NCC, controls are selected only for selected cases, and not all the events are selected as cases. Thus, $V_{i1}$ is needed. When ${\pi }_1=1,V_{1i}=1$ whenever ${\delta }_i=1$; our formula (4) is equivalent to Equation (2.6) in Samuelsen (1997) under the typical NCC.

Remark 1 The probability ${\widehat{p}}_{0j}$ is usually small for large cohort studies because $m$ is often very small relative to the size of the risk set. In addition, if an event, say subject $j$, has a short event time, its probability ${\widehat{p}}_{0j}$ can be close to zero since there are very few cases that subject $j$ is eligible to be included in their risk sets.

Remark 2 Using our notations, the weight in Edelmann et al. (2020) is expressed as

$${\kappa }_{1,j}={\delta }_j\left[V_{1j}+\left(1-V_{1j}\right)V_{0j}\right]+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j},$$ (5)

and the weight in Graziano et al. (2021) is expressed as

$${\kappa }_{2,j}={\delta }_j\left[V_{1j}+\left(1-V_{1j}\right)V_{0j}\right]/{\pi }_1+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j}.$$ (6)

For non-event controls, ${\kappa }_{1,j}={\kappa }_{2,j}=1/{\widehat{p}}_{0j}$; for both events cases and event controls, ${\kappa }_{1,j}=1$ but ${\kappa }_{2,j}=1/{\pi }_1$. In Appendix A.1 of Supplementary Material, we show that neither weight has an expectation of 1 when ${\pi }_1<1$, and consequently, the resulting IPW estimators are biased for the untypical NCC.

Unlike $\tilde{w}$ and $\widehat{w}$, our third weight has the same formula for event cases and event controls. This method is inspired by Breslow et al. (2009), which formulated the case-cohort sampling as a two-stage stratified sampling. Following this idea, we render the NCC design as such a framework and, based on it, propose a Horvitz-Thompson’s type of weight.

NCC as a two-stage stratified sampling.

In the first stage, the subjects in the full cohort are classified into $N_1$ strata, where $N_1$ is the number of events, i.e., $N_1={\sum }_{j=1}^N {\delta }_j$ with $N$ being the size of the full cohort. Each stratum includes one event and his/her risk set. At the second stage, a two-step sampling is performed: first, select $n_1\le N_1$ strata with ${\pi }_1=n_1/N_1$, and second, within each selected stratum, select the case and $m$ subjects from the risk set of this case.

This framework differs from the one described in Breslow et al. (2009) in two aspects. First, the strata in Breslow et al. (2009) are disjoint, but the strata in NCC usually overlap because a subject can be eligible for the risk set of more than one case. Second, the sampling in the untypical NCC is not independent. The strata selection is a finite-population sampling; if one stratum is selected, the others have a lower chance of being selected.

Horvitz-Thompson’s weight.

Our third weight is defined as the inverse probability of being ever selected into the sub-cohort, given as

$$w_{\text{HT},j}={\delta }_j\left[V_{1j}+\left(1-V_{1j}\right)V_{0j}\right]/\left[{\pi }_1+\left(1-{\pi }_1\right){\widehat{p}}_{0j}\right]+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j}.$$ (7)

This expression indicates that for both event cases and event controls, $w_{\text{HT},j}=1/\left[{\pi }_1+\left(1-{\pi }_1\right){\widehat{p}}_{0j}\right]$; for non-event controls, $w_{\text{HT},j}=1/{\widehat{p}}_{0j}$.

2.3. Comparison of Three Weights

When ${\pi }_1=1$, all three weights have the same expression and are equivalent to the weight proposed in Samuelsen (1997): ${\tilde{w}}_j={\widehat{w}}_j=w_{\text{HT},j}={\delta }_j+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j}$.

When ${\pi }_1<1$, the three weights have the same expression for non-events: ${\tilde{w}}_j={\widehat{w}}_j=w_{\text{HT},j}=V_{0j}/{\widehat{p}}_{0j}$, but for events, they are completely different. Specifically,

• for event cases, ${\tilde{w}}_j=1,{\widehat{w}}_j=1/{\pi }_1$, and $w_{\text{HT},j}=1/\left[{\pi }_1+\left(1-{\pi }_1\right){\widehat{p}}_{0j}\right]$; among them, ${\widehat{w}}_j>w_{\text{HT},j}>{\tilde{w}}_j$;

• for event controls, ${\tilde{w}}_j=1/{\widehat{p}}_{0j},{\widehat{w}}_j=0$, and $w_{\text{HT},j}=1/\left[{\pi }_1+\left(1-{\pi }_1\right){\widehat{p}}_{0j}\right]$; among them, ${\widehat{w}}_j<w_{\text{HT},j}<{\tilde{w}}_j$. We also want to point out that the weight ${\widehat{w}}_j$ complete discards this group of subjects for estimation.

In Appendix A, we derive each weight’s expectation, variance, and covariance. These properties determine the consistency and asymptotic variance of its resulting IPW estimator, which will be delineated in Section 3.

2.4. IPW estimation

We consider two models for characterizing the relationship between the event time $T^{†}$ and markers Z: (i) the PH model and (ii) the time-specific GLM. Under each model, we present the IPW estimators of the model parameters and accuracy parameters. We want to point out that these parameters are associated with the probability model in Equation (8) below, which describes a larger population where the full cohort is from (Breslow et al. 2009).

Since the IPW estimators using the three weights have a similar expression except for the weight, we will present the estimators using $w_j$, representing one of the three weights.

2.4.1. Model Parameters Estimation

Both the PH model and time-specific GLM can be expressed in the following form:

$$P\left(T_j^{†}\le t_0\mid Z_j\right)=g\left({\alpha }_{t_0}+{\mathit{\beta }}_{t_0}^{\top }{Z}_j\right)\triangleq {\mathbb{P}}_{t_{0,j}},$$ (8)

where $g(\cdot )$ is a link function, and ${\mathit{\beta }}_{t_0}$, a $p$-dimensional vector, consists of the relative risk parameters that characterize the effects of the markers ${\mathbf{Z}}_j$ on the risk ${\mathbb{P}}_{t_0,j}$.

PH model.

This model can be expressed as ${\mathbb{P}}_{t_0,j}=1-\text{exp} \left[-\text{exp} \left\{\text{log} {\Lambda }_0\left(t_0\right)+{\mathit{\beta }}^{\mathrm{\top }}{\mathbf{Z}}_j\right\}\right]$, where ${\Lambda }_0\left(t_0\right)$ is the baseline cumulative hazard function. Based on Equation (8), ${\alpha }_{t_0}=\text{log} {\Lambda }_0\left(t_0\right)$, which can be estimated by the IPW Breslow’s estimator with the weight $w_j$ (Cai and Zheng 2012). The relative risk parameters $\mathit{\beta }$ are estimated via maximizing the IPW log partial likelihood function with the weight $w_j$ (Samuelsen 1997).

The PH model assumes the relative risks $\mathit{\beta }$ to be constant over time. However, in practice, the biomarkers may have strong effects on the short-term risk but weak for the long-term risk, or vice versa (Zhou et al. 2015). In these situations, the time-specific GLM allows the marker effects to vary over time $t_0$.

Time-specific GLM.

This model can be expressed as ${\mathbb{P}}_{t_0,j}=g\left(\alpha \left(t_0\right)+\mathit{\beta }{\left(t_0\right)}^{\mathrm{\top }}{\mathbf{Z}}_j\right)$, where both $\alpha \left(t_0\right)$ and $\mathit{\beta }\left(t_0\right)$ are functions of $t_0$. Given a $t_0$, these parameters can be estimated via double IPW. Each observation is weighted by $w_j*{\widehat{\omega }}_{t_0,j}$, where $w_j$ is the sampling weight accounting for the missing values of ${\mathbf{Z}}_j$ due to the sub-cohort sampling, and ${\widehat{\omega }}_{t_0,j}$ is the censoring weight accounting for the missing disease status $I\left(T_j^{†}\le t_0\right)$ due to censoring. The censoring weight ${\widehat{\omega }}_{t_{0,j}}$ is given as ${\widehat{\omega }}_{t_0,j}={\delta }_{j}I\left(T_j\le t_0\right)/\widehat{𝒢}\left(T_j\right)+I\left(T_j>t_0\right)/\widehat{𝒢}\left(t_0\right)$, where $\widehat{𝒢}\left(t\right)$ is a consistent estimator of $𝒢\left(t\right)=P\left(C_j\ge t\right)$, the survival function of the censoring time. If the censoring time is independent of both the event time and markers, $\widehat{𝒢}\left(t\right)$ can be the Kaplan-Meier estimator (Kaplan and Meier 1958). If the censoring time depends on the markers ${\mathbf{Z}}_j$, a PH model can be fit to estimate $P\left(C_j\ge t\mid {\mathbf{Z}}_j\right)$.

Let ${\widehat{\alpha }}_{t_0}$ and ${\widehat{\mathit{\beta }}}_{t_0}$ denote the IPW estimates of the model parameters under either the PH model or time-specific GLM. Let $\left({\alpha }_{t_0}^{*},{\mathit{\beta }}_{t_0}^{*}\right)$ be the limiting values of $\left({\widehat{\alpha }}_{t_0},{\widehat{\mathit{\beta }}}_{t_0}\right)$ as $N\to \mathrm{\infty }$, and they can be regarded as the true values of the model parameters.

2.4.2. Accuracy Parameters Estimation

The probability ${\mathbb{P}}_{t_0,j}$ in Equation (8) can be used as a risk score that classifies subjects into different risk categories. Given a cut-off value $c$, subjects with ${\mathbb{P}}_{t_0,j}>c$ are classified as the highrisk group, and the low-risk group consists of subjects with ${\mathbb{P}}_{t_0,j}\le c$. We evaluate the above models’ predictive accuracy with the following time-dependent TPR, FPR, PPV, and NPV.

These accuracy parameters are based on the population risk score ${\mathbb{P}}_{t_0,j}^{*}=g\left({\alpha }_{t_0}^{*}+{\mathit{\beta }}_{t_0}^{*\mathrm{\top }}{\mathbf{Z}}_j\right)$. They are defined as ${\text{TPR}}_{t_0} \left(c\right)=\mathit{\text{Pr}} \left({\mathbb{P}}_{t_0,j}^{*}>c\mid T_j^{†}\le t_0\right),{\text{FPR}}_{t_0} \left(c\right)=\mathit{\text{Pr}} \left({\mathbb{P}}_{t_0,j}^{*}>c\mid T_j^{†}>t_0\right)$, ${\text{PPV}}_{t_0} \left(c\right)=\mathit{\text{Pr}} \left(T_j^{†}\le t_0\mid {\mathbb{P}}_{t_0,j}^{*}>c\right)$, and ${\text{NPV}}_{t_0}\left(c\right)=\mathit{\text{Pr}} \left(T_j^{†}>t_0\mid {\mathbb{P}}_{t_0,j}^{*}\le c\right)$. In addition, the time-dependent AUC is the area under the time-dependent ROC curve, which is a curve of ${\text{TPR}}_{t_0}\left(c\right)$ versus ${\text{FPR}}_{t_0}\left(c\right)$ over all possible values of $c$. The time-dependent AUC can be expressed as ${\text{AUC}}_{t_0}\left(c\right)=\mathit{\text{Pr}} \left({\mathbb{P}}_{t_0,i}^{*}>{\mathbb{P}}_{t_0,j}^{*}\mid T_i^{†}\le t_0,T_j^{†}>t_0\right)$, a conditional probability that, given a pair of an event and a non-event by time $t_0$, the event has a higher risk score.

With the estimates $\left({\widehat{\alpha }}_{t_0},{\widehat{\mathit{\beta }}}_{t_0}\right)$, we can calculate the estimated risk ${\widehat{\mathbb{P}}}_{t_0,j}=g\left({\widehat{\alpha }}_{t_0}+{\widehat{\mathit{\beta }}}_{t_0}^{\mathrm{\top }}{\mathbf{Z}}_j\right)$. The time-dependent accuracy measures described above can be estimated by the double IPW estimators:

$$\begin{array}{rr}& {\widehat{\text{TPR}}}_{t_0}\left(c\right)=\frac{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,j}>c\right)I\left(T_j\le t_0\right)}{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left(T_j\le t_0\right)},{\widehat{\text{FPR}}}_{t_0}\left(c\right)=\frac{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,j}>c\right)I\left(T_j>t_0\right)}{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left(T_j>t_0\right)},\\ & {\widehat{\text{PPV}}}_{t_0}\left(c\right)=\frac{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,j}>c\right)I\left(T_j\le t_0\right)}{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,j}>c\right)},{\widehat{\text{NPV}}}_{t_0}\left(c\right)=\frac{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_{0,j},j}I\left({\widehat{\mathbb{P}}}_{t_{0,j},j}\le c\right)I\left(T_j>t_0\right)}{\sum _{j=1}^N w_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,j}\le c\right)},\end{array}$$

and

$${\widehat{\text{AUC}}}_{t_0}=\frac{\sum _{i=1}^N \sum _{j=1}^N w_i{\widehat{\omega }}_{t_0,i}{w}_j{\widehat{\omega }}_{t_0,j}I\left({\widehat{\mathbb{P}}}_{t_0,i}>{\widehat{\mathbb{P}}}_{t_0,j}\right)I\left(T_i\le t_0\right)I\left(T_j>t_0\right)}{\sum _{i=1}^N \sum _{j=1}^N w_i{\widehat{\omega }}_{t_0,i}{w}_j{\widehat{\omega }}_{t_0,j}I\left(T_i\le t_0\right)I\left(T_j>t_0\right)}.$$ (9)

3. Asymptotic Properties of IPW estimators

In this section, we will present the consistency and asymptotic distribution of the IPW estimators using our three weights: ${\tilde{w}}_j,{\widehat{w}}_j$, and $w_{\text{HT},j}$.

3.1. Consistency of IPW estimators

In Appendix A.1, we prove that given the data $\mathfrak{D}=\left\{\left(T_i,{\delta }_i,{\mathbf{M}}_i\right),i=1,\cdots ,N\right\},\text{E}\left({\tilde{w}}_j\mid \mathfrak{D}\right)=1$, $\text{E}\left({\widehat{w}}_j\mid \mathfrak{D}\right)=1$, and $\text{E}\left(w_{\text{HT},j}\mid \mathfrak{D}\right)=1+O_p\left(N^{-1}\right)$ for any $0<{\pi }_1\le 1$. Thus, the IPW estimator using each weight is consistent for both the typical and untypical NCC.

3.2. Asymptotic Distribution of IPW Estimators

In Appendix B, we show that the IPW estimator with each weight can be expressed as a weighted sum of independent zero-mean random variables that are functions of data $\left(T_j,{\delta }_j,{\mathbf{Z}}_j\right)$, shown in Equation (B.1). Thus, they are asymptotically normally distributed, and the asymptotic variance is the sum of two components. The first component is the model-based variance of the estimator as if the full cohort data $\left\{\left(T_j,{\delta }_j,{\mathbf{Z}}_j\right),j=1,\cdots ,N\right\}$ are available (Equation (B.3)). The second component is the sampling variance, i.e., variability from the sub-cohort sampling, including the variance and covariance of $V_{1j}$ and $V_{0j}$ given the data (Equations (B.4) and (B.5)). Breslow et al. (2009) and Edelmann et al. (2020) expressed their IPW estimator variance by a similar decomposition.

3.3. Comparison of IPW Estimators

Since the three IPW estimators are all consistent, we compare their efficiency. As pointed out in Section 2.3, when ${\pi }_1=1$, the three weights have the same expression, and thus, their IPW estimators have the same asymptotic variance for the typical NCC.

Our comparison focuses on ${\pi }_1<1$, i.e., the untypical NCC. The three IPW estimators have the same model-based variance since this part does not involve the sampling weight. Their sampling variance for non-events is also the same since the three weights have the same expression for this group. Therefore, the difference of the total variance is on the sampling variance for the events, mainly controlled by the variance of the sampling weight.

Equations (A.7) - (A.9) in Appendix A.2 express each weight’s variance for events: $\text{Var} \left[{\tilde{w}}_j\mid \mathfrak{D}\right]=\left(1-{\pi }_1\right){\text{E}}_{{\mathbf{V}}_1}\left[\frac{1-{\widehat{p}}_{0j}}{{\widehat{p}}_{0j}}\mid \mathfrak{D}\right]+O_p\left(N^{-1}\right),\text{Var} \left[{\widehat{w}}_j\mid \mathfrak{D}\right]=\frac{1-{\pi }_1}{{\pi }_1}$, and $\text{Var} \left[w_{\text{HT},j}\mid \mathfrak{D}\right]={\text{E}}_{{\mathbf{V}}_1}\left[\frac{1-{\widehat{p}}_j}{{\widehat{p}}_j}\mid \mathfrak{D}\right]+O_p\left(N^{-1}\right)$ with ${\hat{p}}_j={\pi }_1+\left(1-{\pi }_1\right){\hat{p}}_{0j}$, where ${\mathbf{V}}_1=\left\{V_{1j},j=1,\cdots ,N\right\}$. When ${\pi }_1<1$, we have:

$$\text{Var}\left[{\tilde{w}}_j\mid \mathfrak{D}\right]>\text{Var}\left[{\widehat{w}}_j\mid \mathfrak{D}\right]\gtrsim \text{Var}\left[w_{\text{HT},j}\mid \mathfrak{D}\right].$$ (10)

The first inequality is because the probability ${\widehat{p}}_{0j}$ is usually very small (Remark 1), leading to ${\widehat{p}}_{0j}<{\pi }_1$, and thus, $\text{Var} \left[{\tilde{w}}_j\mid \mathfrak{D}\right]>\text{Var} \left[{\widehat{w}}_j\mid \mathfrak{D}\right]$. The second inequality is because ${\pi }_1<{\pi }_1+\left(1-{\pi }_1\right){\widehat{p}}_{0j}$, and thus, $\text{Var} \left[{\widehat{w}}_j\mid \mathfrak{D}\right]>\text{Var} \left[w_{\text{HT},j}\mid \mathfrak{D}\right]$. Again, because ${\widehat{p}}_{0j}$ is small, these two variances are very close. The order in Equation (10) determines that the IPW estimator with $\tilde{w}$ is less efficient than the other two, which have similar variances. These analytical conclusions agree with the numerical results of the simulation study in Section 4.2 and the data example in Section 5.

3.4. Inference via Perturbation

To draw inferences on the parameters of interest, we need a variance estimate for the IPW estimator. Samuelsen (1997) provided an analytical variance estimate for the PH model parameters under the typical NCC. However, for the untypical NCC, the IPW estimator’s variance expression is more complicated, as shown above, and therefore, obtaining an analytical estimate is challenging. Resampling procedures, such as bootstrap, fail to estimate the variance accurately because they cannot emulate the between-subject correlations (Gray 2009. Cai and Zheng 2013).

Cai and Zheng (2013) proposed a perturbation resampling procedure for variance estimation under the typical NCC. Their method mimics the variance and covariance of $V_{0j}$ via repeatedly perturbing these indicator variables. However, under the untypical NCC, there is also a sampling process for the case selection, so the perturbation of $V_{0j}$ alone is not sufficient. Thus, we extend this procedure by perturbing both $V_{1j}{}^{}$’s and $V_{0j}{}^{}$’s to recover all the variances and covariances.

Like bootstrap, the perturbation method creates many perturbed counterparts for the estimator. We can calculate their empirical variance, which approximates the finite-sample variance of the estimator.

Perturbed counterpart of IPW estimator.

Here, we use $w_j$ to represent one of our weights. For the model parameters ${\alpha }_{t_0}$ and ${\mathit{\beta }}_{t_0}$, the perturbed counterparts of their IPW estimators, denoted by ${\widehat{\alpha }}_{t_0}^{\left(b\right)}$ and ${\widehat{\mathit{\beta }}}_{t_0}^{\left(b\right)}$, are obtained by replacing the sampling weight $w_j$ and censoring weight ${\widehat{\omega }}_{t_0,j}$ with their respective perturbed counterparts, $w_j^{\left(b\right)}$ and ${\widehat{\omega }}_{t_0,j}^{\left(b\right)}$. We will explain how to perturb these two weights later.

For the double IPW estimators of accuracy parameters, their counterparts are obtained using the perturbed weight $w_j^{\left(b\right)}*{\widehat{\omega }}_{t_{0,j}}^{\left(b\right)}$ and the perturbed risk score ${\widehat{\mathbb{P}}}_{t_{0,j}}^{\left(b\right)}=g\left({\widehat{\alpha }}_{t_0}^{\left(b\right)}+{\mathbf{Z}}_j^{\mathrm{\top }}{\widehat{\mathit{\beta }}}_{t_0}^{\left(b\right)}\right)$. For example, the perturbed counterpart of ${\widehat{\text{AUC}}}_{t_0}$ in Equation (9) is given as

$${\widehat{\text{AUC}}}_{t_0}^{\left(b\right)}=\frac{\sum _{i=1}^N \sum _{j=1}^N w_i^{\left(b\right)}{\widehat{\omega }}_{t_{0,i}}^{\left(b\right)}{w}_j^{\left(b\right)}{\widehat{\omega }}_{t_{0,j}}^{\left(b\right)}I\left\{{\widehat{\mathbb{P}}}_{t_0,i}^{\left(b\right)}>{\widehat{\mathbb{P}}}_{t_{0,j}}^{\left(b\right)},T_i\le t_0,T_j>t_0\right)}{\sum _{i=1}^N \sum _{j=1}^N w_i^{\left(b\right)}{\widehat{\omega }}_{t_{0,i}}^{\left(b\right)}{w}_j^{\left(b\right)}{\widehat{\omega }}_{t_{0,j}}^{\left(b\right)}I\left(T_i\le t_0,T_j>t_0\right)}.$$

Perturbed censoring weight ${\widehat{\mathit{\omega }}}_{\mathit{j}}^{\left(\mathit{b}\right)}$.

Let $\left\{I_{ij},i=1,\cdots ,N,j=1,\cdots ,N\right\}$ be independent and identically distributed random variables with mean 1 and variance 1. The perturbed censoring weight ${\widehat{\omega }}_{t_0,j}^{\left(b\right)}$ is given as ${\widehat{\omega }}_{t_0,j}^{\left(b\right)}={\delta }_{j}I\left(T_j\le t_0\right)/{\widehat{𝒢}}^{\left(b\right)}\left(T_j\right)+I\left(T_j>t_0\right)/{\widehat{𝒢}}^{\left(b\right)}\left(t_0\right)$, where ${\widehat{𝒢}}^{\left(b\right)}\left(t\right)$ is the estimate of $𝒢\left(t\right)$ with each subject weighted by $I_{jj},j=1,\cdots ,N$.

Previously, we concluded that the two IPW estimators using ${\widehat{w}}_j$ and $w_{\text{HT},j}$ are more efficient than ${\tilde{w}}_j$. Thus, we describe how to perturb these two weights.

Perturbed sampling weight ${\widehat{\mathit{w}}}_{\mathit{j}}^{\left(\mathit{b}\right)}$.

This perturbed sampling weight is ${\widehat{w}}_j^{\left(b\right)}={\delta }_j\frac{V_{1j}^{\left(b\right)}}{{\pi }_1^{\left(b\right)}}+\left(1-{\delta }_j\right)\frac{V_{0j}^{\left(b\right)}}{{\widehat{p}}_{0j}^{\left(b\right)}}$, obtained by replacing the indicator variables $V_{1j}$ and $V_{0j}$ as well as their probabilities ${\pi }_1$ and ${\hat{p}}_{0j}$ with their perturbed counterparts. Specifically, the perturbed counterpart of the case indicator is $V_{1j}^{\left(b\right)}=V_{1j}{I}_{jj}$. The probability ${\pi }_1$ can be written as ${\pi }_1={\sum }_{i=1}^N {\delta }_i{V}_{1i}/{\sum }_{i=1}^N {\delta }_i$, and its perturbed counterpart is given as ${\pi }_1^{\left(b\right)}={\sum }_{i=1}^N I_{ii}{\delta }_i{V}_{1i}/{\sum }_{i=1}^N I_{ii}{\delta }_i$. By Equation (1), the perturbed counterpart of the control indicator variable is given as $V_{0j}^{\left(b\right)}=1-{\prod }_{i:j\in {ℛ}_i} \left(1-V_{1i}{V}_{0j}^i{I}_{ij}\right)$. The probability ${\widehat{p}}_{0j}$ in Equation (4) can be written as ${\widehat{p}}_{0j}=1-{\prod }_{i:j\in {ℛ}_i} \left\{1-\frac{{\sum }_{l\in {ℛ}_i} V_{0l}^i}{{\mathfrak{n}}_i-1}{\delta }_i{V}_{1i}\right\}$ where $m={\sum }_{l\in {ℛ}_i} V_{0l}^i$, and its perturbed counterpart is ${\widehat{p}}_{0j}^{(b)}=1-{\prod }_{i:j\in {ℛ}_i}\left\{1-\frac{{\sum }_{l\in {ℛ}_i}{V}_{0k}^i{I}_{il}}{{\mathfrak{n}}_i-1}{\delta }_i{V}_{1i}\right\}$.

Perturbed sampling weight ${\mathit{w}}_{\mathbf{H}\mathbf{T},\mathit{j}}^{\left(\mathit{b}\right)}$.

With the notations above, this perturbed weight is $w_{\text{HT},j}^{\left(b\right)}={\delta }_j\frac{{\left[V_{1j}+\left(1-V_{1j}\right)V_{0j}\right]}_{jj}}{{\pi }_1^{\left(b\right)}+\left(1-{\pi }_1^{\left(b\right)}\right){\hat{p}}_{0j}^{\left(b\right)}}+\left(1-{\delta }_j\right)\frac{V_{0j}^{\left(b\right)}}{{\widehat{p}}_{0j}^{\left(b\right)}}$.

It is worth noting that our perturbation procedure is valid for both the typical and untypical NCC. When ${\pi }_1=1,V_{1j}={\delta }_j$, and ${\pi }_1^{\left(b\right)}={\pi }_1=1$. The perturbed weights ${\widehat{w}}_j^{\left(b\right)}$ and $w_{\text{HT},j}^{\left(b\right)}$ are the same as those proposed in Cai and Zheng (2013) for the typical NCC studies.

4. Simulation Studies

We investigate the proposed methods via the following two simulation studies. The first one focuses on comparing the IPW estimators using our three new weights: ${\tilde{w}}_j$, ${\widehat{w}}_j$, and $w_{\text{HT},j}$ and two weights of Edelmann et al. (2020) and Graziano et al. (2021): ${\kappa }_{1,j}$ and ${\kappa }_{2,j}$ in terms of their bias and variability. For the ease of presentation, we use ${\widehat{\theta }}_{\tilde{w}},{\widehat{\theta }}_{\hat{w},},{\widehat{\theta }}_{w_{\text{HT}}},{\widehat{\theta }}_{{\kappa }_1}$, and ${\widehat{\theta }}_{{\kappa }_2}$ to denote the five IPW estimators using the five weights. The second study investigates the validity of the proposed perturbation resampling procedure for estimating the variance of the IPW estimator.

As defined earlier, the true values of the model and accuracy parameters are governed by the underlying data generating mechanism. Appendix C of Supplementary Material explains how to calculate the true parameter values based on the simulation scheme below. In both studies, we use the true parameter values to evaluate our proposed estimation and inference procedures as well as the existing estimators.

4.1. Simulation setting

For both studies, we use the same simulation setting. We consider one clinical marker $Z_j$ that is measured on the full cohort and one biomarker $B_j$ that is measured only on the NCC sub-cohort. The marker $Z_j$ is first generated from the standard normal distribution $N\left(\mathrm{0,1}\right)$. In many situations, biomarkers are correlated with clinical markers, and thus, we simulate $B_j$ depending on $Z_j:B_j=Z_j+e_{B,j}$, where $e_{B,j}~N\left(\mathrm{0,1}\right)$. Given these two markers, the event time $T_j^{†}$ is obtained from

$$\text{log}\left(T_j^{†}\right)=1.5-0.25Z_j-0.25B_j+0.5{ϵ}_{T,j},$$ (11)

where ${ϵ}_{T,j}$ is generated from an extreme value distribution with the cumulative distribution function $F\left(x\right)=1-\mathit{\text{exp}} \left(-e^x\right)$. As described in Section 2.1, the event time might be censored by a random censoring time variable ${\tilde{C}}_j$ and a follow-up duration of $\tau =2$ years, where ${\tilde{C}}_j$ is generated from ${\tilde{C}}_j=0.1+C_{j1}$ with $C_{j1}~\text{Gamma} \left(\mathrm{2,2}\right)$. Thus, about 90% of the event times are censored by $C_j=\text{min}\left\{{\tilde{C}}_j,\tau \right\}$.

We consider two full cohort sizes: $N=\mathrm{5,000}$ and 10,000. To construct the NCC sub-cohort, we set ${\pi }_1$ (the percentage of events that are selected as cases) to be 20%, 50%, and 80%. For each selected case, $m=1$ or 3 controls are selected from either (i) the case’s risk set without matching, or (ii) the case’s risk set with exact matching on a variable $M_1$ and matching up to \pm 1 on another variable $M_2$. The two matching variables $M_1$ and $M_2$ are generated as follows. Let $M_1=I\left({\tilde{M}}_1>0.5\right)$ where ${\tilde{M}}_1=\Phi \left(Z+e_1\right)$ with $e_1~N\left(\mathrm{0,1}\right)$, and let $M_2$ be the closest integer to ${\tilde{M}}_2=5\Phi \left(B+e_2\right)$ where $e_2~N\left(\mathrm{0,1}\right)$.

We set the prediction time $t_0=1$ with the event rate $\mathit{\text{Pr}} \left(T^{†}<1\right)\approx 8\%$. To estimate the absolute risk $P\left(T_j^{†}<1\mid Z_j,B_j\right)$, we fit the PH model and the time-specific GLM with a logit link. As explained in Section 2.4.1, both models can be expressed as the general form in Equation (8): $P\left(T_j^{†}\le t_0\mid Z_j,B_j\right)=g\left({\alpha }_{t_0}+{\beta }_{Z,t_0}{Z}_j+{\beta }_{B,t_0}{B}_j\right)$. Since we consider only one $t_0$ value, we suppress $t_0$ from the parameters. We also want to point out that the true data generating mechanism in Equation (11) follows the PH assumption, and thus the GLM is a misspecified model.

Under each model, we obtain the IPW estimates of the relative risk parameters $\left({\beta }_Z,{\beta }_B\right)$. For the PH model, as mentioned earlier, conditional logistic regression (clogit) is an alternative method to obtain the estimates of $\left({\beta }_Z,{\beta }_B\right)$. Thus, we compare the five IPW estimators with the clogit estimator. In addition, each model is evaluated on the following time-dependent accuracy measures: TPR, PPV, and NPV at a cutoff value making FPR = 0.05, and AUC.

As described in Section 3.2, the total variance of the IPW estimator includes the model-based variance as if the full cohort data are available. To capture this part, we generate 1000 replicates of the full cohort data, and for each replication, an NCC sub-cohort is constructed following one of the sampling schemes described above.

4.2. Study I: Comparison of IPW Estimators using Five Weights

In the first study, we compare the empirical bias and standard deviation (SD) of the five IPW estimators. Tables 2 and 3 report these statistics for the relative risk parameters ${\beta }_Z$ and ${\beta }_B$ under the PH model and time-specific GLM respectively. The summary statistics for each model’s accuracy parameters are shown in Figures 1 and 2. All these results are for the settings where the controls are selected with matching, and those for the without-matching settings are included in Appendix D of Supplementary Material. We also report the square root of mean square error (RMSE) in Tables 4 – 7 of Appendix D.

Table 2:

Simulation Results: Estimation of ${\beta }_Z$ and ${\beta }_B$ under the PH model. For each parameter, the results include the bias and empirical SD (in the parentheses) relative to the true parameter value in 100%. The NCC sub-cohort is constructed with matching.

$N$	${\pi }_1$	$1:m$	$\tilde{w}$	$\tilde{w}$	$w_{\text{HT}}$	$k_1$	$k_2$	clogit
			${\beta }_Z=0.5$
5000	0.2	1:1	9.8 (99.4)	0.5 (41.4)	0.5 (41.4)	19.4 (53.0)	21.7 (55.9)	1.0 (56.6)
5000	0.2	1:3	6.8 (66.9)	0.6 (32.4)	1.4 (31.6)	20.2 (38.1)	21.2 (40.0)	1.2 (39.5)
5000	0.5	1:1	3.6 (53.7)	2.0 (27.4)	2.1 (27.2)	11.2 (30.7)	11.4 (30.4)	1.6 (36.2)
5000	0.5	1:3	2.4 (38.9)	1.1 (19.8)	1.3 (19.4)	10.1 (21.0)	10.1 (21.1)	0.1 (25.5)
5000	0.8	1:1	2.5 (33.1)	2.1 (21.1)	2.1 (21.0)	5.2 (21.9)	5.2 (21.8)	2.2 (31.0)
5000	0.8	1:3	1.3 (22.7)	1.2 (16.5)	1.2 (16.3)	4.1 (16.7)	4.1 (16.6)	−1.2 (22.7)
10000	0.2	1:1	8.2 (80.3)	0.0 (29.1)	0.6 (29.0)	18.7 (37.2)	22.6 (39.4)	0.3 (38.8)
10000	0.2	1:3	2.5 (54.0)	−0.5 (21.8)	0.1 (20.7)	18.6 (24.9)	19.7 (26.8)	−0.9 (27.4)
10000	0.5	1:1	−0.9 (47.1)	−0.6 (18.5)	−0.6 (18.3)	8.2 (20.5)	8.4 (20.4)	0.2 (26.3)
10000	0.5	1:3	2.4 (31.4)	0.0 (14.7)	0.2 (14.3)	9.0 (15.3)	9.1 (15.3)	−0.9 (18.6)
10000	0.8	1:1	1.5 (24.1)	0.4 (14.7)	0.5 (14.6)	3.5 (15.2)	3.5 (15.1)	0.1 (20.1)
10000	0.8	1:3	−0.7 (18.8)	0.2 (11.8)	0.2 (11.6)	3.1 (11.9)	3.1 (11.9)	−0.8 (15.9)
			${\beta }_B=0.5$
5000	0.2	1:1	11.8 (76.3)	2.7 (29.6)	2.7 (29.5)	23.8 (39.7)	27.3 (41.8)	12.1 (46.0)
5000	0.2	1:3	6.4 (49.9)	1.4 (22.3)	1.4 (21.8)	22.1 (26.2)	24.0 (28.0)	5.3 (30.8)
5000	0.5	1:1	6.3 (39.6)	1.2 (18.7)	1.3 (18.7)	10.9 (21.1)	11.5 (21.0)	6.4 (28.8)
5000	0.5	1:3	1.5 (29.2)	0.5 (14.6)	0.6 (14.4)	10.4 (15.7)	11.0 (15.7)	1.8 (21.5)
5000	0.8	1:1	1.0 (24.7)	0.0 (14.9)	0.0 (14.9)	3.2 (15.6)	3.2 (15.5)	4.3 (23.4)
5000	0.8	1:3	1.2 (16.7)	0.1 (11.9)	0.2 (11.8)	3.4 (12.1)	3.5 (12.1)	0.5 (17.7)
10000	0.2	1:1	6.1 (58.7)	0.7 (20.8)	0.6 (20.6)	19.3 (26.8)	23.3 (28.1)	7.3 (31.6)
10000	0.2	1:3	4.9 (38.2)	0.1 (15.4)	0.5 (14.9)	20.4 (17.7)	23.8 (19.0)	3.1 (21.6)
10000	0.5	1:1	5.1 (34.2)	1.6 (12.9)	1.6 (12.8)	10.9 (14.5)	11.5 (14.4)	5.9 (21.1)
10000	0.5	1:3	0.3 (24.6)	0.4 (10.8)	0.5 (10.5)	10.2 (11.4)	10.9 (11.2)	0.9 (15.4)
10000	0.8	1:1	2.0 (17.8)	0.8 (10.5)	0.8 (10.5)	4.0 (10.9)	4.0 (10.9)	4.3 (16.3)
10000	0.8	1:3	1.0 (13.3)	0.5 (8.3)	0.6 (8.1)	3.8 (8.3)	3.9 (8.3)	−0.3 (12.4)

Table 3:

Simulation Results: Estimation of ${\beta }_Z$ and ${\beta }_B$ under the time-specific GLM. For each parameter, the results include the bias and empirical SD (in the parentheses) relative to the true parameter value in 100%. The NCC sub-cohort is constructed with matching. In addition, for the IPW estimator using $\tilde{w}$, its summary statistics are calculated using only the replications for which the GLM converged. The number of replications (out of the total 1000 replications) for which the GLM did not converge is reported as numbers labelled with *.

$N$	${\pi }_1$	$1:m$	$\tilde{w}$	$\tilde{w}$	$w_{HT}$	$k_1$	$k_2$
			${\beta }_Z=0.54$
5000	0.2	1:1	1.8e+14 (9.5e+15); 34*	2.2 (54.8)	2.1 (54.2)	19.5 (64.8)	18.6 (69.1)
5000	0.2	1:3	1.3e+13 (8.7e+15); 8*	3.2 (45.7)	3.8 (43.0)	19.7 (47.7)	19.2 (56.7)
5000	0.5	1:1	1.8e+14 (4.6e+15); 2*	3.5 (35.5)	3.6 (35.2)	11.7 (38.1)	11.3 (38.0)
5000	0.5	1:3	5.8 (57.3); 0*	2.4 (28.1)	2.3 (27.1)	9.3 (28.0)	8.1 (28.6)
5000	0.8	1:1	5.2e+13 (1.6e+15); 2*	4.0 (27.3)	4.0 (27.3)	6.7 (27.9)	6.6 (27.9)
5000	0.8	1:3	3.6 (33.7); 0*	2.8 (23.0)	2.8 (22.8)	5.0 (23.0)	4.8 (23.0)
10000	0.2	1:1	−8.3e+14 (1.4e+16); 30*	1.4 (37.1)	1.8 (36.3)	19.0 (44.3)	19.9 (46.8)
10000	0.2	1:3	−3.2e+14 (7.6e+15); 7*	0.7 (30.2)	0.7 (28.6)	16.6 (31.6)	14.2 (38.8)
10000	0.5	1:1	3.6e+13 (1.1e+15); 5*	−0.6 (24.3)	−0.6 (24.1)	7.2 (25.8)	6.7 (25.9)
10000	0.5	1:3	5.5 (43.3); 1*	0.5 (19.9)	0.7 (19.1)	7.8 (19.7)	6.8 (20.2)
10000	0.8	1:1	3.1 (34.8); 0*	1.1 (19.2)	1.1 (19.1)	3.7 (19.5)	3.6 (19.5)
10000	0.8	1:3	0.0 (25.8); 0*	0.8 (15.9)	0.8 (15.7)	3.1 (15.8)	2.9 (15.8)
			${\beta }_B=0.54$
5000	0.2	1:1	5.3e+13 (6.7e+15); 34*	4.0 (38.9)	4.2 (38.3)	23.4 (47.2)	23.6 (50.2)
5000	0.2	1:3	2.5e+14 (6.4e+15); 8*	3.4 (32.8)	2.9 (30.6)	20.3 (34.0)	18.3 (40.8)
5000	0.5	1:1	−7e+13 (2.4e+15); 2*	1.8 (25.0)	2.0 (24.9)	10.3 (26.8)	10.1 (26.9)
5000	0.5	1:3	6.3 (43.5); 0*	1.9 (21.2)	2.0 (20.4)	9.7 (21.2)	9.1 (21.4)
5000	0.8	1:1	−2.7e+13 (8.5e+14); 2*	0.7 (19.6)	0.6 (19.5)	3.3 (20.0)	3.3 (20.0)
5000	0.8	1:3	3.3 (24.5); 0*	1.1 (17.0)	1.2 (16.9)	3.7 (17.1)	3.6 (17.1)
10000	0.2	1:1	5.3e+14 (9e+15); 30*	2.1 (26.4)	2.1 (25.8)	19.9 (31.5)	20.0 (33.7)
10000	0.2	1:3	4.3e+13 (3.2e+15); 7*	1.8 (22.1)	2.2 (21.3)	19.2 (23.4)	19.5 (28.7)
10000	0.5	1:1	−2.4e+13 (7.6e+14); 5*	3.2 (17.0)	3.2 (17.0)	11.3 (18.2)	11.0 (18.3)
10000	0.5	1:3	3.4 (34.4); 1*	2.2 (14.8)	2.2 (14.0)	9.8 (14.6)	9.1 (14.7)
10000	0.8	1:1	4.5 (25.6); 0*	2.0 (13.3)	2.0 (13.3)	4.7 (13.6)	4.6 (13.6)
10000	0.8	1:3	2.9 (18.9); 0*	1.9 (11.1)	1.9 (10.9)	4.4 (11.0)	4.3 (11.0)

Figure 1:

Simulation Results: Estimation of AUC and TPR, PPV, and NPV at the cut-off value which corresponds to FPR=0.05 under the PH model. For each parameter, the results include the absolute bias (the marker) and empirical SD (half of the bar length) relative to the true parameter value in 100%. The NCC sub-cohort is constructed with matching.

Figure 2:

Simulation Results: Estimation of AUC and TPR, PPV, and NPV at the cut-off value which corresponds to FPR=0.05 under the time-specific GLM. For each parameter, the results include the absolute bias (the marker) and empirical SD (half of the bar length) relative to the true parameter value in 100%. The NCC sub-cohort is constructed with matching. In addition, for the IPW estimator using $\tilde{w}$, its summary statistics are calculated using only the replications for which the GLM converged.

When calculating these summary statistics, we take into account the magnitude of the parameter values: we compute the bias, SD, and RMSE relative to the true parameter value. Specifically, let ${\widehat{\theta }}^{\left[r\right]}$ denote the IPW estimate of a parameter $\theta$ based on the data generated in the $r$-th replication, $r=1,\cdots ,1000$, and let ${\theta }^{*}$ denote the true value of $\theta$. The above-mentioned tables and figures present the relative bias, calculated as rBias $=\left(\stackrel{-}{\widehat{\theta }}-{\theta }^{*}\right)/{\theta }^{*}$ with $\stackrel{-}{\widehat{\theta }}=\frac{1}{1000}{\sum }_{r=1}^{1000} \widehat{\theta }\left[r\right]$, and the relative SD as $\text{rSD}=\sqrt{\frac{1}{999}{\sum }_{r=1}^{1000} \left(\widehat{\theta }\right[r]-\bar{\widehat{\theta }}{)}^2}/{\theta }^{*}$. The RMSE is an overall metric accounting for both bias and variability. The relative RMSE is calculated as rRMSE $=\sqrt{{\text{rBias}}^2+{\text{rSD}}^2}$. In addition, Figures 1 and 2 plot the absolute value of the relative bias to better compare its magnitude.

Results for PH model.

For estimating the relative risk parameters $\left({\beta }_Z,{\beta }_B\right)$, we find that a larger full cohort size $N$, a larger ${\pi }_1$, or a larger number $m$ of controls per case tend to produce smaller bias and smaller variance. Among the five IPW estimators and the clogit estimator, ${\widehat{\theta }}_{\hat{w}}$ and ${\widehat{\theta }}_{w_{\text{HT}}}$ perform the best with the least bias and highest efficiency, and therefore, their RMSEs are the smallest. Between them, the variance of ${\widehat{\theta }}_{w_{\text{HT}}}$ is slightly smaller, but their difference is minimal, especially when ${\pi }_1$ is close to 1. This observation agrees with our analytical conclusions in Section 3.3.

The IPW estimator ${\widehat{\theta }}_{\tilde{w}}$ has a smaller bias than ${\widehat{\theta }}_{{\kappa }_1}$ and ${\widehat{\theta }}_{{\kappa }_2}$. However, its disadvantage is variability, which is the largest among all the estimators, and because of this, the RMSE of ${\widehat{\theta }}_{\tilde{w}}$ is also the largest. As explained in Section 3.3, its large variance is due to small values of ${\hat{p}}_{0j}$. In addition, when ${\pi }_1$ decreases, we find that ${\hat{\theta }}_{\tilde{w}}$ gets less efficient because there are fewer cases that a control can be selected for, which means ${\hat{p}}_{0j}$ drops even more.

The results clearly show that ${\widehat{\theta }}_{{\kappa }_1}$ and ${\widehat{\theta }}_{{\kappa }_2}$ are biased. As explained in Remark 2, the bias results from the expectations of their weights not equal to 1. In addition, as derived in Appendix A.1, when ${\pi }_1$ approaches 1, these expectations get closer to 1. Accordingly, we observe that their biases decrease when ${\pi }_1$ increases. When ${\pi }_1$ is small, for instance, ${\pi }_1=0.2$, doubling the full cohort size $N$ did not significantly reduce the bias. In terms of variability, their variances are slightly smaller than the clogit estimator. Overall, they have a similar RMSE as the clogit estimator.

For the accuracy parameters, we observe similar results: ${\widehat{\theta }}_{\hat{w}}$ and ${\widehat{\theta }}_{w_{\text{HT}}}$ perform the best and similarly, ${\widehat{\theta }}_{\tilde{w}}$ has the largest variance, and ${\widehat{\theta }}_{{\kappa }_1}$ and ${\widehat{\theta }}_{{\kappa }_2}$ have the largest bias. When ${\pi }_1$ is closer to 1, these five IPW estimators are more alike because their weights are more similar to each other.

Results for time-specific GLM.

Most of the results are similar to those under the PH model. The main difference is that, besides being inefficient, ${\widehat{\theta }}_{\tilde{w}}$ did not converge in some replications. Table 3 lists the number of the “nonconverged” replications for each scenario. The summary statistics for all the parameters are calculated using only the estimates from the converged replications. However, some of them still yield “abnormal” estimates, which makes the bias and SD go through the roof.

We believe that this issue is caused by event controls that experienced the event early. As explained in Remark 1, for such subjects, their ${\widehat{p}}_{0j}$ values are close to zero, leading to a huge weight ${\tilde{w}}_j$, but their binary outcomes $I\left(T_j\le t_0\right)$ are 1. In addition, when ${\pi }_1$ decreases, as explained earlier, ${\hat{p}}_{0j}$ would be even smaller, and consequently, this issue gets worse. In comparison, other IPW estimators do not have this problem because their weights are smaller for these subjects.

4.3. Study II: Perturbation Resampling for Inference

The second study examines the validity of the perturbation resampling procedure for two IPW estimators ${\widehat{\theta }}_{\widehat{w}}$ and ${\widehat{\theta }}_{w_{\text{HT}}}$ since they are the best among all the estimators. In the following, we use $\widehat{\theta }$ to denote one of these two estimators. In each replication, besides the estimate ${\widehat{\theta }}^{\left[r\right]}$, we obtain 500 perturbed counterparts with $I_{ij}$ generated from an exponential distribution with rate 1. The perturbed counterpart is denoted as ${\widehat{\theta }}^{[r,b]}$ where $b$ indexes the perturbation, and $r$ still indexes the replication. We calculate the SD of $\left\{{\widehat{\theta }}^{[r,b]},b=1,\cdots ,500\right\}$, which is the standard error (SE) associated with the estimate ${\widehat{\theta }}^{\left[r\right]}$. This SE is denoted as $S{E}^{\left[r\right]}$ and used to construct a level $1-\alpha$ confidence interval (CI) for the parameter: ${\widehat{\theta }}^{\left[r\right]}\pm z_{1-\alpha /2}*S{E}^{\left[r\right]}$, where $z_{1-\alpha /2}$ is the $100*(1-\alpha /2)\%$ quantile of a standard normal distribution.

We evaluate this procedure with two metrics, reported in Tables 4 and 5. The first metric compares the average SE, i.e., $\frac{1}{1000}{\sum }_{r=1}^{1000} S{E}^{\left[r\right]}$ with the empirical SD of the estimates $\left\{{\widehat{\theta }}^{\left[r\right]},r=\right.1,\cdots ,1000\}$. Specifically, this metric is the ratio of the average SE over the empirical SD. The second metric is the empirical coverage probability of the 95% CI constructed with the perturbation-based SE. We observe that for both estimators, the average SE is close to the empirical SD, and the empirical coverage probability is close to the nominal level 95% for most of the scenarios. These results indicate that the proposed perturbation resampling can accurately capture the variability of the IPW estimator, and thus, the inference procedure is valid.

Table 4:

Simulation Results: the ratios of the average SE to empirical SD and empirical coverage probabilities (in the parentheses) of 95% CIs for relative risk parameters. Both the SEs and 95% CIs are obtained via perturbation resampling. The NCC sub-cohort is constructed with matching.

			$B_Z$		${\beta }_B$
$N$	${\pi }_1$	$1:m$	$\tilde{w}$	$w_{\text{HT}}$	$\tilde{w}$	$w_{\text{HT}}$
			PH Model
5000	0.2	1:1	0.927 (92.8%)	0.914 (92.5%)	0.926 (92.5%)	0.915 (92.5%)
5000	0.2	1:3	0.963 (94.2%)	0.934 (93.7%)	1.005 (95.2%)	0.976 (94.9%)
5000	0.5	1:1	0.912 (92.0%)	0.910 (92.4%)	0.952 (94.0%)	0.948 (93.7%)
5000	0.5	1:3	1.025 (94.5%)	1.006 (94.8%)	0.997 (94.5%)	0.977 (94.1%)
5000	0.8	1:1	0.959 (94.2%)	0.959 (94.7%)	0.967 (94.3%)	0.966 (94.4%)
5000	0.8	1:3	1.001 (95.2%)	1.001 (94.8%)	0.995 (94.0%)	0.990 (94.4%)
10000	0.2	1:1	0.944 (93.6%)	0.930 (93.6%)	0.938 (94.1%)	0.935 (94.1%)
10000	0.2	1:3	1.012 (94.9%)	1.006 (95.4%)	1.021 (94.7%)	1.000 (94.6%)
10000	0.5	1:1	0.965 (94.3%)	0.965 (94.3%)	0.987 (94.8%)	0.983 (94.2%)
10000	0.5	1:3	0.977 (94.4%)	0.971 (94.1%)	0.949 (94.2%)	0.944 (94.1%)
10000	0.8	1:1	0.981 (94.5%)	0.986 (94.0%)	0.978 (94.3%)	0.978 (94.5%)
10000	0.8	1:3	0.988 (94.3%)	0.989 (94.4%)	1.015 (94.4%)	1.018 (94.7%)
			Time-specific GLM
5000	0.2	1:1	0.963 (94.8%)	0.955 (94.6%)	0.976 (95.2%)	0.972 (94.7%)
5000	0.2	1:3	0.981 (94.8%)	0.982 (94.8%)	0.990 (95.2%)	1.000 (94.6%)
5000	0.5	1:1	0.936 (93.5%)	0.936 (93.0%)	0.956 (94.5%)	0.951 (94.0%)
5000	0.5	1:3	1.012 (94.7%)	1.012 (95.3%)	0.969 (94.3%)	0.967 (94.5%)
5000	0.8	1:1	0.973 (93.5%)	0.969 (93.8%)	0.972 (94.4%)	0.973 (94.1%)
5000	0.8	1:3	0.983 (94.6%)	0.981 (94.9%)	0.964 (94.3%)	0.957 (94.3%)
10000	0.2	1:1	0.993 (94.4%)	0.996 (94.4%)	1.003 (95.7%)	1.008 (95.5%)
10000	0.2	1:3	1.037 (95.2%)	1.033 (95.2%)	1.019 (95.5%)	1.002 (95.6%)
10000	0.5	1:1	0.963 (93.9%)	0.962 (93.4%)	0.994 (94.6%)	0.986 (94.0%)
10000	0.5	1:3	1.000 (94.8%)	1.010 (95.8%)	0.975 (94.7%)	0.992 (95.0%)
10000	0.8	1:1	0.976 (93.6%)	0.978 (94.0%)	1.012 (94.8%)	1.011 (95.1%)
10000	0.8	1:3	1.002 (94.7%)	1.004 (94.8%)	1.037 (96.2%)	1.046 (96.1%)

Table 5:

Simulation Results: the ratios of the average SE to empirical SD and empirical coverage probabilities (in the parentheses) of 95% CIs for accuracy parameters. Both the SEs and 95% CIs are obtained via perturbation resampling. The NCC sub-cohort is constructed with matching.

			AUC		TPR		PPV		NPV
			$\tilde{w}$	$w_{\text{HT}}$	$\tilde{w}$	$w_{\text{HT}}$	$\tilde{w}$	$w_{\text{HT}}$	$\tilde{w}$	$w_{\text{HT}}$
$N$	${\pi }_1$	$1:m$				PH Model
5000	0.2	1:1	0.922 (90.9%)	0.914 (91.1%)	1.006 (94.0%)	1.008 (94.8%)	1.107 (96.1%)	1.093 (96.0%)	0.928 (92.2%)	0.925 (93.1%)
5000	0.2	1:3	0.973 (92.8%)	0.967 (93.9%)	1.027 (94.4%)	1.027 (94.7%)	1.036 (94.9%)	1.031 (95.2%)	0.913 (92.6%)	0.929 (92.5%)
5000	0.5	1:1	0.949 (93.7%)	0.949 (94.2%)	1.011 (94.9%)	1.007 (93.6%)	1.027 (94.4%)	1.036 (95.6%)	0.911 (91.9%)	0.908 (92.4%)
5000	0.5	1:3	0.958 (93.7%)	0.954 (93.5%)	1.015 (94.4%)	1.002 (94.2%)	0.979 (93.3%)	0.973 (93.9%)	0.896 (91.6%)	0.891 (91.3%)
5000	0.8	1:1	0.987 (93.6%)	0.981 (93.3%)	1.012 (95.3%)	1.013 (95.1%)	1.005 (94.9%)	1.007 (94.8%)	0.941 (92.9%)	0.943 (93.3%)
5000	0.8	1:3	1.024 (95.2%)	1.019 (95.3%)	1.011 (95.2%)	1.014 (94.8%)	0.973 (93.6%)	0.986 (94.1%)	0.860 (90.0%)	0.857 (90.3%)
10000	0.2	1:1	0.967 (94.2%)	0.961 (94.1%)	1.022 (94.5%)	1.018 (94.8%)	1.063 (96.3%)	1.060 (95.2%)	0.953 (93.7%)	0.948 (93.6%)
10000	0.2	1:3	0.968 (94.5%)	0.949 (93.5%)	0.991 (93.5%)	0.988 (93.3%)	0.967 (94.3%)	0.972 (94.7%)	0.915 (92.3%)	0.919 (92.8%)
10000	0.5	1:1	0.957 (93.1%)	0.955 (93.4%)	1.016 (95.1%)	1.021 (95.4%)	1.034 (95.1%)	1.046 (95.1%)	0.912 (91.7%)	0.911 (91.9%)
10000	0.5	1:3	0.970 (93.4%)	0.974 (94.1%)	0.997 (94.9%)	1.002 (94.6%)	0.976 (94.7%)	0.994 (94.8%)	0.876 (90.6%)	0.883 (90.7%)
10000	0.8	1:1	0.964 (93.8%)	0.967 (93.7%)	1.006 (94.8%)	1.007 (94.7%)	1.026 (94.8%)	1.029 (95.3%)	0.897 (90.7%)	0.898 (90.5%)
10000	0.8	1:3	0.982 (95.1%)	0.985 (94.6%)	1.031 (95.8%)	1.050 (96.3%)	1.008 (95.5%)	1.024 (95.4%)	0.833 (89.3%)	0.838 (89.0%)
			Time-specific GLM
5000	0.2	1:1	0.916 (91.1%)	0.908 (90.8%)	1.029 (95.1%)	1.023 (95.2%)	1.107 (95.9%)	1.109 (96.5%)	0.938 (93.3%)	0.925 (93.4%)
5000	0.2	1:3	0.958 (92.3%)	0.954 (92.7%)	1.050 (94.9%)	1.059 (95.5%)	1.039 (95.2%)	1.055 (95.3%)	0.917 (91.9%)	0.937 (92.7%)
5000	0.5	1:1	0.947 (93.5%)	0.946 (93.9%)	1.022 (94.5%)	1.021 (94.6%)	1.038 (94.4%)	1.046 (95.0%)	0.910 (92.0%)	0.906 (92.2%)
5000	0.5	1:3	0.956 (93.6%)	0.952 (93.6%)	1.014 (94.3%)	1.003 (95.0%)	0.974 (93.7%)	0.972 (94.1%)	0.894 (92.0%)	0.891 (91.7%)
5000	0.8	1:1	0.987 (93.2%)	0.982 (93.3%)	1.027 (95.4%)	1.017 (95.2%)	1.012 (94.6%)	1.010 (94.7%)	0.943 (93.3%)	0.941 (92.8%)
5000	0.8	1:3	1.020 (95.2%)	1.017 (95.2%)	1.012 (95.5%)	1.009 (94.6%)	0.976 (93.6%)	0.981 (93.4%)	0.859 (90.2%)	0.858 (90.1%)
10000	0.2	1:1	0.962 (94.2%)	0.956 (93.8%)	1.027 (94.7%)	1.036 (95.2%)	1.058 (95.4%)	1.072 (95.6%)	0.954 (93.8%)	0.955 (94.2%)
10000	0.2	1:3	0.966 (94.2%)	0.945 (93.0%)	1.000 (94.3%)	1.016 (94.9%)	0.969 (94.3%)	0.989 (95.1%)	0.922 (92.4%)	0.931 (92.5%)
10000	0.5	1:1	0.956 (92.9%)	0.955 (92.9%)	1.029 (94.9%)	1.030 (95.5%)	1.055 (95.7%)	1.054 (95.7%)	0.912 (91.8%)	0.912 (91.8%)
10000	0.5	1:3	0.968 (93.4%)	0.973 (94.2%)	1.004 (94.5%)	1.011 (95.3%)	0.981 (95.3%)	1.005 (95.4%)	0.875 (91.3%)	0.880 (91.0%)
10000	0.8	1:1	0.959 (93.5%)	0.963 (93.7%)	1.014 (94.9%)	1.019 (94.6%)	1.036 (95.2%)	1.044 (95.7%)	0.900 (91.5%)	0.901 (91.1%)
10000	0.8	1:3	0.979 (95.0%)	0.982 (94.4%)	1.036 (95.8%)	1.059 (96.6%)	1.010 (95.9%)	1.022 (95.7%)	0.835 (89.7%)	0.847 (89.9%)

5. Data Example: the Framingham Offspring Study

To further compare our three new weights with existing methods, we use the Offspring Cohort of Framingham Heart Study (Wawrzyniak 2013) as an example, in which the outcome of interest is a cardiovascular disease (CVD) event. This cohort includes 1501 males and 1644 females; among them, 989 subjects have encountered a CVD event during the follow-up of about 35 years.

To build a risk model for CVD, we consider two markers: the Framingham risk score (FRS), and a biomarker, C-reactive protein (CRP). The FRS was developed by Wilson et al. (1998) to estimate the 10-year CVD risk. This score is gender-specific and based on several risk factors including age, systolic blood pressure, diastolic blood pressure, total cholesterol, high-density lipoprotein cholesterol, current smoking status, and diabetes status. The CRP is an inflammation biomarker and shown to improve the prediction on top of the risk variables from the FRS (Ridker 2003, Cook et al. 2006).

The Offspring Cohort is the full cohort, and we construct NCC sub-cohorts from it and obtain the IPW estimates using our three weights: $\tilde{w},\widehat{w}$, and $w_{\text{HT}}$, and two existing weights ${\kappa }_1$ and ${\kappa }_2$. To evaluate the accuracy of these IPW estimators, we use the estimates obtained from the full cohort as the reference values since both FRS and CRP are available for the full cohort. We want to point out that unlike the simulation studies, we cannot compute the true parameter values for this example because the underlying data generating mechanism is unknown. Thus, the full-cohort estimates are the best estimates we could have in this situation.

Specifically, we draw 100 sub-cohorts following each of the two sampling schemes: (i) selecting ${\pi }_1=50\%$ of the 989 CVD events as cases and 1 control for each case, and (ii) selecting ${\pi }_1=26\%$ of the 989 CVD events as cases and 3 controls for each case. These two schemes produce a similar sub-cohort size despite different ${\pi }_1$ : out of 100 samples, on average, about 904 subjects are selected based on Scheme (i) and 896 based on Scheme (ii).

We consider two prediction times $t_0=15$ and 30 years. Within 15-year follow-up, 213 subjects experienced a CVD event and 6 subjects were lost to follow-up; the 15-year event risk is estimated to be about 7%; within 30-year follow-up, 805 subjects experienced a CVD event, and 169 subjects were lost-to-follow up; the 30-year event risk is estimated to be about 34%. We fit both the PH model and time-specific GLM with the FRS and log transformed CRP. Figures 3, and 5 include the boxplots of the five IPW estimates for the relative risk and accuracy parameters. In these figures, the horizontal lines represent the full-cohort estimates. In addition, for the relative risk parameters of the PH model, we compare the five IPW estimators with the clogit estimator in Figure 3(a).

Figure 3:

Data Example: Boxplots of the IPW estimates using five weights: $\tilde{w},\widehat{w},w_{\text{HT}},{\kappa }_1$, and ${\kappa }_2$ for the marker effects under the PH model and time-specific GLM based on 100 NCC sub-cohorts. For the PH model, the IPW estimates are also compared with the clogit estimates. The triangle inside each box represents the mean of the estimates; the dashed horizontal lines represent the full-cohort estimates.

Figure 5:

Data Example: Boxplots of the IPW estimates using five weights: $\tilde{w},\widehat{w},w_{\text{HT}},{\kappa }_1$, and ${\kappa }_2$ for the accuracy parameters under the time-specific GLM based on 100 NCC sub-cohorts. The triangle inside each box represents the mean of the estimates; the dashed horizontal lines represent the full-cohort estimates.

We observe similar results with those of the simulation study. The two IPW estimators ${\widehat{\theta }}_{\hat{w}}$ and ${\widehat{\theta }}_{w_{\text{HT}}}$ perform the best: they are closest to the full-cohort estimates on average and have the smallest variability. In addition, these two estimators perform similarly. By contrast, ${\widehat{\theta }}_{{\kappa }_1}$ and ${\widehat{\theta }}_{{\kappa }_2}$ are farthest from the full-cohort estimates on average, especially for the accuracy parameters. For ${\widehat{\theta }}_{\tilde{w}}$, although the GLM estimation converged for all the samples, it still has the largest variance.

From this data example, we also observe that the advantage of ${\hat{\theta }}_{\hat{w}}$ and ${\widehat{\theta }}_{w_{\text{HT}}}$ over all the other methods is more obvious for predicting the 15-year risk than the 30-year risk. In addition, each estimator tend to have a larger variance for ${\pi }_1=26\%$, compared to ${\pi }_1=50\%$, although these sampling schemes lead to a similar sub-cohort size. This phenomenon is more apparent for the 15-year risk prediction. This indicates that if the sub-cohort size is fixed given a budget, selecting more cases, i.e., a larger ${\pi }_1$, and a lower control-to-case ratio would lead to more efficient estimates.

6. Concluding Remarks

In this paper, we are interested in the untypical NCC design where a subset, not all, of the events are selected as cases. Such a design, although not generally considered in the statistical literature, is useful in practice for various reasons. In particular, for biomarker studies, samples from events may be more easily depleted and require careful preservation. To analyze the untypical NCC data with the IPW approach, we need to address two challenges. First, event cases and event controls are selected to the sub-cohort in different ways. Failing to account for this difference would lead to biased estimation, like Edelmann et al. (2020) and Graziano et al. (2021). In contrasts, our two weights $\tilde{w}$ and $\widehat{w}$ weight these two groups differently based on how an event enters the sub-cohort. Although our third weight $w_{\text{HT}}$ has the same formula for both groups, its selection probability for events counts both case and control selections. In addition, when all the events are selected as cases, all these three weights are equivalent to the Samuelsen’s weight.

The second challenge is statistical inference since the IPW estimator for the untypical NCC has a complicated variance structure, including both between-case and between-control correlations induced by the finite-population sampling. We provided a perturbation resampling procedure for drawing inferences on both model and accuracy parameters. Our simulation study has demonstrated that this procedure can well approximate the empirical variance of the IPW estimator, and consequently, the coverage probability of the perturbation-based CI is close to the nominal level.

Among our three IPW estimators, ${\widehat{\theta }}_{w_{\text{HT}}}$ and ${\widehat{\theta }}_{\hat{w}}$ have a similar performance, and they are more efficient than ${\widehat{\theta }}_{\tilde{w}}$. We have provided the analytical derivation and numerical evidence. In addition, ${\widehat{\theta }}_{w_{\text{HT}}}$ and ${\widehat{\theta }}_{\hat{w}}$ perform better than all the existing IPW and clogit estimators. Thus, we recommend these two weighting methods for the IPW estimation under the untypical NCC.

Our proposed framework can be further extended to NCC studies where the cases are sampled via a more complex design, such as stratified sampling (Lü et al. 2018). In such a situation, we can design the following two weights: ${\widehat{w}}_j={\delta }_j{V}_{1j}/{\widehat{p}}_{1j}+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_{0j}$, where ${\widehat{p}}_{1j}$ is the probability that subject $j$ is selected as a case, and $w_{\text{HT},j}={\delta }_j\left[V_{1j}+\left(1-V_{1j}\right)V_{0j}\right]/\left[{\widehat{p}}_{1j}+\left(1-{\widehat{p}}_{1j}\right){\widehat{p}}_{0j}\right]+\left(1-{\delta }_j\right)V_{0j}/{\widehat{p}}_j$. Together, our proposed work could open the door for more efficient and practical biomarker studies.

In this manuscript, all the weights are design-based because the selection probability is calculated based on the sampling scheme. An alternative approach is to augment the weight by either calibrating or estimating the selection probability from a model with auxiliary variables (Breslow et al. 2009). The model-based weighting method is not the focus of this manuscript, but it is worth future exploration for NCC designs since it has a potential to improve the estimation efficiency.

Figure 4:

Data Example: Boxplots of the IPW estimates using five weights: $\tilde{w},\widehat{w},w_{\text{HT}},{\kappa }_1$, and ${\kappa }_2$ for the accuracy parameters under the PH model based on 100 NCC sub-cohorts. The triangle inside each box represents the mean of the estimates; the dashed horizontal lines represent the full-cohort estimates.