Evaluating incremental values from new predictors with net reclassification improvement in survival analysis

Abstract

Developing individualized prediction rules for disease risk and prognosis has played a key role in modern medicine. When new genomic or biological markers become available to assist in risk prediction, it is essential to assess the improvement in clinical usefulness of the new markers over existing routine variables. Net reclassification improvement (NRI) has been proposed to assess improvement in risk reclassification in the context of comparing two risk models and the concept has been quickly adopted in medical journals. We propose both nonparametric and semiparametric procedures for calculating NRI as a function of a future prediction time t with a censored failure time outcome. The proposed methods accommodate covariate-dependent censoring, therefore providing more robust and sometimes more efficient procedures compared with the existing nonparametric-based estimators. Simulation results indicate that the proposed procedures perform well in finite samples. We illustrate these procedures by evaluating a new risk model for predicting the onset of cardiovascular disease.

1 Introduction

Developing individualized prediction rules for disease risk and prognosis is fundamental for successful disease prevention and treatment selection. For many diseases, risk prediction models have been developed and incorporated into clinical practice guidelines. For example, the Gail model was developed for predicting individual breast cancer risk (Gail et al. 1989) and a risk calculator based on that model can be used to assist physicians making screening recommendations. For cardiovascular disease (CVD), prediction models such as the Framingham Risk Score (FRS) are used for stratifying patients into different levels of risks. However, much refinement is needed even for the best of these models because of their limited discriminatory accuracy. For example, the Framingham model, largely based on traditional clinical risk factors, has recognized limitations in its clinical utility (Hemann et al. 2007). A considerable fraction of patients who experienced CVD events had none of the identified risk factors, indicating a need to explore avenues beyond routine clinical measures for more accurate prediction (Khot et al. 2003). This fuels much of the current search for novel biologic markers and genetic factors that, when combined with routine clinical risk factors, may provide accurate prediction at the individual level.

When new genomic or biological markers become available to assist in risk prediction, it is essential to assess the clinical usefulness of these new markers compared to existing routine markers. Careful evaluation of the incremental value is particularly crucial when markers are either expensive or invasive to measure. To quantify the added clinical value of new markers over a conventional risk scoring system for predicting disease risk, one may calculate the difference in the prediction measures for the existing conventional model and the new model, which includes information from the new markers. For example the difference in the areas under the receiver operating characteristic curves (AUC of ROC) are often used to quantify the improvement in discrimination attributable to added markers. Since a risk model is often used to stratify patients into proper risk categories, statistical summaries that depend on clinically meaningful risk thresholds may be more relevant (Cook 2007; Cui 2009; Lloyd-Jones 2010). As an alternative to measuring the difference between AUCs, net reclassification improvement (NRI) has also been proposed to assess improvement in risk reclassification in the context of comparing two risk models constructed with and without novel markers (Pencina et al. 2008). Using “up” and “down” to denote changes in one or more risk categories in the upward and downward directions, respectively, for a subject between their baseline and augmented risk values, the NRI is defined as

$$\mathrm{NRI}=[\mathrm{Pr}(\text{up}\mid \text{Diseased})+\mathrm{Pr}(\text{down}\mid \text{Healthy})]-[\mathrm{Pr}(\text{down}\mid \text{Diseased})+\mathrm{Pr}(\text{up}\mid \text{Healthy})].$$

Such a measure is appealing because it acknowledges both desirable risk reclassifications (up for diseased and down for healthy subjects) and undesirable risk reclassifications (down for diseased and up for healthy subjects). Due to its simplicity, NRI has been quickly adopted in medical journals. However, compared with many other measures for incremental values, the concept has not received much attention in the statistical literature.

Since a risk model is often used for predicting an individual's future outcome, it is essential to incorporate the additional dimension of time when assessing the performance of a risk model in a cohort study. For both deriving and evaluating risk models, prospective cohort data is often used. In this setting a subject's health status at a future time t is sometimes unknown due to loss of follow-up, termination of a study or the occurrence of a competing risk event. Such censoring poses additional challenges compared with settings previously examined in the literature which focus on incremental value calculation with a dichotomous outcome. Currently there is limited development in methods to estimate the incremental value of novel markers with censored failure time outcomes. Recently Pencina and D'Agostino (2011) proposed a method for calculating time-dependent NRI, based on nonparametric Kaplan–Meier (KM) estimators in order to account for censoring in cohort data. The asymptotic properties of a similar estimator is studied in detail in Uno et al. (2009). However, the validity of these estimators relies critically on the assumption that censoring is independent of predictors used in the risk models. Furthermore, the nonparametric procedure considered in these estimators may potentially lead to efficiency loss. A more flexible and more efficient estimating procedure is needed in practice.

In this manuscript, we propose quantitative procedures for calculating NRI as a function of a future prediction time t with a censored failure time outcome. Compared with existing nonparametric estimators, our procedures do not require the assumption that censoring is independent of predictors, therefore the methods would be widely applicable to many practical situations. We also consider procedures that aim to improve efficiency while maintaining robustness. This manuscript is organized as follows. In Sect. 2, we specify models and define NRI suitable for event time outcomes. In Sect. 3, we describe procedures for estimating time-dependent NRI using data obtained from a prospective cohort study with a failure time outcome. We comment on the theoretical properties of our proposed estimators in Sect. 4. We then describe simulation studies to evaluate the performance of the proposed estimators. The results are reported in Sect. 5. An application of our procedures for comparing two CVD risk models is presented in Sect. 6. Concluding remarks are in Sect. 7.

2 Measures of risk stratification and reclassification

Consider the situation that a vector of predictor Y measured at baseline is used for predicting the time to event outcome T. Risk models can be built using sub-vectors of Y. Let Y₍₁₎, a function of Y, denote a vector of conventional predictor values in the existing model. Let Y₍₂₎, also a function of Y, denote a vector of predictors used in the new model that contains Y₍₁₎, but also new predictor values. Individual-level risk at a future time t can be derived as $P=\mathrm{Pr}(T\le t\mid {\mathbf{Y}}_{\left(1\right)})$, based on the conventional model, and $Q=\mathrm{Pr}(T\le t\mid {\mathbf{Y}}_{\left(2\right)})$, the corresponding risk based on the new model, respectively. Since, in practice, risk categories are often uncertain for many diseases, a more objective and flexible measure of improvement in risk prediction would be based on P or Q in their original continuous scales. Therefore, following the definition of Pencina and D'Agostino (2011), in this manuscript we focus on the time-dependent continuous NRI, which is a more general definition that does not rely on the existence of risk categories. In the time-dependent setting, we further denote an ‘event’ person at time t as those with $T\le t$, and a ‘nonevent’ person as T > t. Here, NRI(t) is equal to the sum of ‘event NRI’ and ‘nonevent NRI’, which are defined as:

$$\begin{array}{cc}\hfill \text{event}{\mathrm{NRI}}_u\left(t\right)& =\mathrm{Pr}(Q-P>u\mid T\le t)-\mathrm{Pr}(Q-P\le u\mid T\le t)\hfill \\ \hfill & \equiv 2\mathrm{Pr}(Q-P>u\mid T\le t)-1,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill \text{nonevent}{\mathrm{NRI}}_v\left(t\right)& =\mathrm{Pr}(Q-P\le v\mid T>t)-\mathrm{Pr}(Q-P>v\mid T>t)\hfill \\ \hfill & \equiv 1-2\mathrm{Pr}(Q-P>v\mid T>t).\hfill \end{array}$$

Since, NRI_u,v(t) = event NRI_u(t) + nonevent NRI_v(t), it follows that ${\mathrm{NRI}}_{u,v}\left(t\right)=2\{\mathrm{Pr}(Q-P>u\mid T\le t)-\mathrm{Pr}(Q-P>v\mid T>t)\}$. In practice we may chose u and v such that improvement in risk estimates is meaningful (Uno et al. 2009). Setting u = v = 0 gives the ‘continuous NRI’ considered in Pencina and D'Agostino (2011). For the ease of presentation, in the sequel, we'll omit the subscript u and v from our notations and assume u = v = 0, but note that our estimators can be constructed for any arbitrary u and v. In the next section, we show how each component of NRI(t) can be estimated.

3 Estimation

Suppose we have a cohort of N individuals from the targeted population followed prospectively. Due to censoring, the observed data consist of N i.i.d copies of vector, $\mathcal{D}=\{{\mathbf{D}}_i={(X_i,{\delta }_i,{\mathbf{Y}}_i)}^{\mathsf{T}},i=1,\dots ,N\}$, where $X_i=\text{min}(T_i,C_i),{\delta }_i=I(T_i\le C_i)$ for T_i and C_i denote failure time and censoring time respectively. Y_i are predictors from individual i measured at time 0, including subset Y_i(1) used in the existing model (model 1) and Y_i(2) in the new model (model 2) such that Y_i(1) ∈ Y_i(2). For illustration, we first assume that P and Q both follow the conventional Cox regression models. Specifically, at time t, we assume $P\left({\theta }_1\right)=1-\text{exp}\left[{\Lambda }_{01}\left(t\right)\text{exp}\left\{{\beta }_1^{\mathsf{T}}{\mathbf{Y}}_{\left(1\right)}\right\}\right]$ and $Q\left({\theta }_2\right)=1-\text{exp}\left[{\Lambda }_{02}\left(t\right)\text{exp}\left\{{\beta }_2^{\mathsf{T}}{\mathbf{Y}}_{\left(2\right)}\right\}\right]$, where Λ_0k is the baseline cumulative hazard function, β_k are unknown vector of parameters, for model k = 1, 2, and ${\theta }_1={({\Lambda }_{01}\left(t\right),{\beta }_1^{\mathsf{T}})}^{\mathsf{T}}$, ${\theta }_2={({\Lambda }_{02}\left(t\right),{\beta }_2^{\mathsf{T}})}^{\mathsf{T}}$. It is important to note that these models are most likely not correctly specified. Nevertheless under a mild regularity condition, the standard maximum partial likelihood estimator ${\widehat{\beta }}_k$ for β_k converges to a constant vector, as n → ∞ (Hjort 1992). This provides theoretical ground for our asymptotic studies.

To estimate NRI(t), Pencina and D'Agostino (2011) first expressed the two key components as

$$\mathrm{Pr}\{B\left(\theta \right)>0\mid T\le t\}=\frac{\mathrm{Pr}\{T\le t\mid B\left(\theta \right)>0\}\mathrm{Pr}\{B\left(\theta \right)>0\}}{\mathrm{Pr}(T\le t)}$$

and

$$\mathrm{Pr}\{B\left(\theta \right)>0\mid T>t\}=\frac{\mathrm{Pr}\{T>t\mid B\left(\theta \right)>0\}\mathrm{Pr}\{B\left(\theta \right)>0\}}{\mathrm{Pr}(T>t)},$$

where B(θ) = Q(θ₂) – P(θ₁) and $\theta ={({\theta }_1,{\theta }_2)}^{\mathsf{T}}$. To account for censoring, Pencina and D'Agostino (2011) proposed to use the KM estimator to estimate the survival function using data from all subjects for $\mathrm{Pr}(T\le t)$ and using subjects with B(θ) > 0 for estimation of $\mathrm{Pr}[T\le t\mid \{B\left(\theta \right)>0\}]$. We refer to the resulting estimator as the ‘KM estimator’ hereafter.

Uno et al. (2009) considered estimating NRI(t) based on an inverse-probability-of-censoring weighted (IPW) estimator (hereafter referred to as the ‘IPW estimator’), with its key components estimated as

$${\widehat{\mathrm{Pr}}}^{\mathrm{IPW}}\{B\left(\theta \right)>0\mid T\le t\}=\frac{{\sum }_{i}I\{B_i\left(\widehat{\theta }\right)>0,X_i\le t\}{\widehat{W}}_i\left(t\right)}{{\sum }_{i}I(X_i\le t){\widehat{W}}_i\left(t\right)}$$ (3.1)

$${\widehat{\mathrm{Pr}}}^{\mathrm{IPW}}\{B\left(\theta \right)>0\mid T>t\}=\frac{{\sum }_{i}I\{B_i\left(\widehat{\theta }\right)>0,X_i>t\}{\widehat{W}}_i\left(t\right)}{{\sum }_{i}I(X_i>t){\widehat{W}}_i\left(t\right)}$$ (3.2)

where $\widehat{\theta }={({\widehat{\theta }}_1,{\widehat{\theta }}_2)}^{\mathsf{T}}$, ${\widehat{\theta }}_1={({\widehat{\Lambda }}_{01}\left(t\right),{\widehat{\beta }}_1^{\mathsf{T}})}^{\mathsf{T}}$, ${\widehat{\theta }}_2={({\widehat{\Lambda }}_{02}\left(t\right),{\widehat{\beta }}_2^{\mathsf{T}})}^{\mathsf{T}}$, ${\widehat{W}}_i\left(t\right)=I(X_i\le t){\delta }_i∕\widehat{H}\left(X_i\right)+I(X_i>t)∕\widehat{H}\left(t\right)$ and $\widehat{H}(\cdot )$ is the KM estimator of H(·) = P(C > ·). Due to the equivalence between the KM estimator and the IPW estimator for marginal survival functions under independent censoring (Satten and Datta 2001), the two estimators are likely to have very similar robustness and efficiency. Both estimators are consistent under an independent censoring assumption regardless of the adequacy of the two fitted models, P(θ₁) and Q(θ₂). This is particularly appealing for model comparisons.

One potential weakness of both estimators is that they can be biased if censoring is dependent on a subset of Y₍₂₎. On the other hand, when model 2 is correctly specified, such covariate-dependent censoring can be incorporated based on the model since C ⊥ T given ${\beta }_2^{\mathsf{T}}{\mathbf{Y}}_{\left(2\right)}$ or Q(θ₂). This motivates us to propose a more robust alternative to the Uno et al. (2009) estimator by estimating censoring probabilities given Y₍₂₎ via kernel smoothing over Q(θ₂). Let $H_q^1\left(t\right)=P(C>t\mid Q\left({\theta }_2\right)=q,{\Delta }_i\left(\theta \right)=1)$ and $H_q^{•}\left(t\right)=P(C>t\mid Q\left({\theta }_2\right)=q)$ where Δ_i(θ) = I{B_i(θ) > 0}. To estimate NRI(t), we propose to modify equations (3.1) and (3.2) by considering the following more robust IPW censoring weights

$${\tilde{W}}_i^{\left(\iota \right)}\left(t\right)=\frac{I(X_i\le t){\delta }_i}{{\tilde{H}}_{Q_i\left({\widehat{\theta }}_2\right)}^{\left(\iota \right)}\left(X_i\right)}+\frac{I(X_i>t)}{{\widehat{H}}_{Q_i\left({\widehat{\theta }}_2\right)}^{\left(\iota \right)}\left(t\right)}\text{for}\iota =1\text{and}•,$$

where ${\tilde{H}}_q^{\left(\iota \right)}\left(t\right)=\text{exp}\{-{\widehat{\Lambda }}_q^{\left(\iota \right)}\left(t\right)\}=\text{exp}\{-{\int }_0^t{\widehat{\pi }}_q^{\left(\iota \right)}{\left(s\right)}^{-1}d{\widehat{N}}_{Cq}^{\left(\iota \right)}\left(s\right)\}$,

$$\begin{array}{cc}\hfill {\widehat{N}}_{Cq}^{\left(\iota \right)}\left(s\right)& =n^{-1}\sum _{i:{\Delta }_i\left(\widehat{\theta }\right)\in {\mho }_{\iota }}{K}_h\{Q_i\left({\widehat{\theta }}_2\right)-q\}{N}_{Ci}\left(s\right),\hfill \\ \hfill {\widehat{\pi }}_q^{\left(\iota \right)}\left(s\right)& =n^{-1}\sum _{i:{\Delta }_i\left(\widehat{\theta }\right)\in {\mho }_{\iota }}{K}_h\{Q_i\left({\widehat{\theta }}_2\right)-q\}I(X_i\ge s),\hfill \end{array}$$

$N_{Ci}\left(s\right)=I(X_i\le s)(1-{\delta }_i),{\mho }_1=1$ and ${\mho }_{•}=\{0,1\},K_h(\cdot )=\frac{1}{h}K\left\{\frac{q-Q_i\left({\theta }_2\right)}{h}\right\}$, K is a symmetric kernel density function, with h = h(n) → 0 as the bandwidth. Note that ${\Delta }_i\left(\widehat{\theta }\right)\in {\mho }_1$ is simply the subset of individuals with $B_i\left(\widehat{\theta }\right)>0$ and ${\Delta }_i\left(\widehat{\theta }\right)\in {\mho }_{•}$ is the set of all individuals. Consequently we can then use these more robust kernel smoothing weights in the IPW estimator, to obtain the ‘Smooth-IPW (S-IPW) estimators’,

$${\widehat{\mathrm{Pr}}}^{\mathrm{S}\text{-}\mathrm{IPW}}\{B\left(\theta \right)>0\mid T\le t\}=\frac{{\sum }_i{\Delta }_i\left(\widehat{\theta }\right){\tilde{W}}_i^{\left(1\right)}\left(t\right)I(X_i\le t)}{{\sum }_i{\tilde{W}}_i^{\left(•\right)}\left(t\right)I(X_i\le t)}\text{and}$$ (3.3)

$${\widehat{\mathrm{Pr}}}^{\mathrm{S}\text{-}\mathrm{IPW}}\{B\left(\theta \right)>0\mid T>t\}=\frac{{\sum }_i{\Delta }_i\left(\widehat{\theta }\right){\tilde{W}}_i^{\left(1\right)}\left(t\right)I(X_i>t)}{{\sum }_i{\tilde{W}}_i^{\left(•\right)}\left(t\right)I(X_i>t)}.$$ (3.4)

This resulting estimator for NRI(t) is

$$\widetilde{\mathrm{NRI}}(\widehat{\theta },t)=2\times \left[{\widehat{\mathrm{Pr}}}^{\mathrm{S}\text{-}\mathrm{IPW}}\{B\left(\widehat{\theta }\right)>0\mid T\le t\}-{\widehat{\mathrm{Pr}}}^{\mathrm{S}\text{-}\mathrm{IPW}}\{B\left(\widehat{\theta }\right)>0\mid T>t\}\right].$$

The estimator can be shown to have the property of ‘double robustness’, i.e., it only requires that the risk model Q is correctly specified or that the independent censoring assumption holds.

Additionally, to improve upon the efficiency of the class of nonparametric estimators, we propose considering a semiparametric estimator. Note that

$$\begin{array}{cc}\hfill \mathrm{Pr}\{B\left(\theta \right)>0\mid T>t\}& =\frac{E\left[E\{I(B\left(\theta \right)>0,T>t)\mid {\mathbf{Y}}_{\left(2\right)}\}\right]}{E\left[E\{I(T>t)\mid {\mathbf{Y}}_{\left(2\right)}\}\right]}\hfill \\ \hfill & =\frac{E\left\{I(B\left(\theta \right)>0)P(T>t\mid {\mathbf{Y}}_{\left(2\right)})\right\}}{E\left\{P(T>t\mid {\mathbf{Y}}_{\left(2\right)})\right\}}.\hfill \end{array}$$

Therefore NRI(t) can be estimated semiparametrically as

$$\widehat{\mathrm{NRI}}(\widehat{\theta },t)=2\times \left\{{\widehat{\mathrm{Pr}}}^{\mathrm{SEM}}(B\left(\theta \right)>0\mid T\le t)-{\widehat{\mathrm{Pr}}}^{\mathrm{SEM}}(B\left(\theta \right)\le 0\mid T>t)\right\},$$

with the ‘SEM’ estimators,

$${\widehat{\mathrm{Pr}}}^{\mathrm{SEM}}(B\left(\theta \right)>0\mid T\le t)=\frac{{\sum }_{i=1}^n{\Delta }_i\left(\widehat{\theta }\right)Q_i\left({\widehat{\theta }}_2\right)}{{\sum }_{i=1}^n{Q}_i\left({\widehat{\theta }}_2\right)},$$ (3.5)

$${\widehat{\mathrm{Pr}}}^{\mathrm{SEM}}(B\left(\theta \right)>0\mid T>t)=\frac{{\sum }_{i=1}^n{\Delta }_i\left(\widehat{\theta }\right)\{1-Q_i\left({\widehat{\theta }}_2\right)\}}{{\sum }_{i=1}^n\{1-Q_i\left({\widehat{\theta }}_2\right)\}}.$$ (3.6)

Under the correctly specified model Q(θ₂), the semiparametric estimator accommodates a covariate-dependent censoring situation and would be more efficient compared to the Smooth-IPW estimator. In practice, to estimate NRI(t), if estimates from such a semiparametric method agree well with those of the nonparametric methods, one may choose to report results based on the semiparametric method for additional gain in efficiency. To automatize the procedure, we suggest considering a combined estimator (hereafter referred as the ‘combined estimator’), which takes the form

$$\widehat{p}\times \widehat{\mathrm{NRI}}(\widehat{\theta },t)+(1-\widehat{p})\times \widetilde{\mathrm{NRI}}(\widehat{\theta },t),$$

with $\widehat{p}$ being a weight that is dependent on the aptness of the semiparametric model. For example, $\widehat{p}$ can be taken to be the p-value from a consistent test of the proportional hazards assumption for a Cox regression model fit. Such an estimator provides a simple procedure which is robust over a wide variety of situations. In numerical studies, we show that such a combined estimator can be more efficient compared with the nonparametric estimators, while maintaining the double robustness property.

We note that the proposed estimators can be easily generalized to NRI based on risk categories. Consider a situation where individuals are classified as low, intermediate or high risk: low risk if their risks are below r₁, and high risk if their risks are above r₂. The reclassification accuracy of risk models in such a setting can be quantified with a 3-category NRI of the form ${\mathrm{NRI}}^{category}(\widehat{\theta },t)=P(up\mid T\le t)-P(down\mid T\le t)+P(down\mid T>t)-P(up\mid T>t)$. To estimate $P(up\mid T\le t)$ and P(up|T > t), we may simply replace ${\Delta }_i\left(\widehat{\theta }\right)$ with ${\Omega }_i^{up}\left(\widehat{\theta }\right)=I(P_i\left({\theta }_1\right)\le r_1{Q}_i\left({\theta }_2\right)>r_1)+I(r_1<P_i\left({\theta }_1\right)\le r_2,Q_i\left({\theta }_2\right)>r_2)$ in Eqs. 3.3 and 3.4, respectively. Similarly, to estimate $P(down\mid T\le t)$ and P(down|T > t), one may replace ${\Delta }_i\left(\widehat{\theta }\right)$ with ${\Omega }_i^{down}\left(\widehat{\theta }\right)=I(Q_i\left({\theta }_1\right)\le r_1,P_i\left({\theta }_2\right)>r_1)+I(r_1<Q_i\left({\theta }_1\right)\le r_2,P_i\left({\theta }_2\right)>r_2)$ in Eqs. 3.3 and 3.4. Similarly, one may obtain a semiparametric estimator of ${\mathrm{NRI}}^{category}(\widehat{\theta },t)$ by replacing ${\Delta }_i\left(\widehat{\theta }\right)$ with ${\Omega }_i^{up}\left(\widehat{\theta }\right)$, or ${\Omega }_i^{down}\left(\widehat{\theta }\right)$ in Eqs. 3.5 and 3.6.

4 Inference

To make inference about $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$, we study the asymptotic properties of proposed estimators. In the Appendix, we show that $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$ is uniformly consistent for NRI(θ₀, t), where ${\theta }_0={({\Lambda }_{k0}(\cdot ),{\beta }_{k0}^{\mathsf{T}})}^{\mathsf{T}}$ with β_k0 being the unique maximizer of the expected value of the corresponding partial likelihood. Furthermore, we show that the process $\tilde{\mathcal{W}}\left(t\right)=\sqrt{n}\{\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)\}$ is asymptotically equivalent to a sum of i.i.d terms, $n^{-1∕2}{\sum }_{i=1}^n{∊}_i\left(t\right)$ where ε_i(t) is defined in the the Appendix. By a functional central limit theorem of Pollard (1990), the process $\tilde{\mathcal{W}}\left(t\right)$ converges weakly to a mean zero Gaussian process in t. We also show that $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$ is uniformly consistent for NRI(θ₀, t), and that the process $\tilde{\mathcal{N}}\left(t\right)=\sqrt{n}[\widehat{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)]$ is asymptotically equivalent to a sum of i.i.d terms $n^{-1∕2}{\sum }_{i=1}^n{\zeta }_i\left(t\right)$ where ζ_i(t) is defined in the Appendix. Again, by a functional central limit theorem, the process $\tilde{\mathcal{N}}\left(t\right)$ converges weakly to a mean zero Gaussian process in t. With weak convergence of both $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$ and $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$, it follows that the combined estimator converges to a zero-mean process. Due to the variation in $\widehat{p}$, the combined estimators may not be a Gaussian process. We show in our simulation that to make inference, resampling procedures such as a bootstrap method can provide a valid approximation of the limit distribution. Specifically, at each of the bth bootstrap iterations, with b = 1, . . . , B, we conduct a random sampling with replacement of the original dataset, and fit our new and old risk models based on the sampled dataset, denoted as $P^b\left(\widehat{\theta }\right)$ and $Q^b\left(\widehat{\theta }\right)$. These estimates from the fitted models are then used to calculate ${\widehat{NRI}}^b(\widehat{\theta },t)$ and $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$ based on the bootstrapped samples. This procedure will be repeated B times, and confidence intervals can be constructed either based on the percentile method, or a normal approximation where the standard error is calculated based on the empirical standard errors of $\{{\widehat{NRI}}^b(\widehat{\theta },t),b=1,\dots ,B\}$ and $\{{\widetilde{NRI}}^b(\widehat{\theta },t),b=1,\dots ,B\}$. The combined estimator can be inferred similarly by repeatedly calculate the weights based on each bootstrap sample in addition to ${\widehat{NRI}}^b(\widehat{\theta },t)$ and $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$.

In the absence of an independent validating set, often in practice the same dataset is used for both fitting the model with several predictors and calculating a measure such as NRI(t). Such an ‘apparent’ summary may potentially lead to the so-called ‘overfitting’ phenomenon, i.e. estimates of model performance will tend to be more optimistic compared with the corresponding estimates if the model were to applied to a new dataset. Several methods for correcting the bias from apparent estimates can be considered. The 0.632 Bootstrap method (Efron and Tibshirani 1997) has been shown to have better performance compared with a simple cross-validated approach. The estimator was derived in our simulation as follows: we first obtained a bootstrapped estimate ${\widehat{\mathrm{NRI}}}^{bt}\left(t\right)$ by sampling the data with replacement to obtain the training set. The training set is used to estimate the model parameters $\{{\widehat{\theta }}_k^{\left(\text{train}\right)},k=1,2\}$. The remaining subjects make up the validation set, and are used to calculate the various estimates of NRI using parameter values $\{{\widehat{\theta }}_k^{\left(\text{train}\right)},k=1,2\}$. This is repeated B times and ${\widehat{\mathrm{NRI}}}^{bt}\left(t\right)$ is the mean across the repetitions. The 0.632 bootstrap estimate is,

$${\widehat{\mathrm{NRI}}}^{0.632bt}\left(t\right)=0.632{\widehat{\mathrm{NRI}}}^{bt}\left(t\right)+(1-0.632){\widehat{\mathrm{NRI}}}^{apparent}\left(t\right),$$

where ${\widehat{NRI}}^{apparent}\left(t\right)$ is the estimate without using cross-validation. To construct a confidence interval based on ${\widehat{\mathrm{NRI}}}^{0.632bt}\left(t\right)$, we follow the suggestions given in Tian et al. (2007) and Uno et al. (2007) by shifting the apparent error based confidence interval in the amount of bias estimated as $\widehat{\text{bias}}={\widehat{\mathrm{NRI}}}^{apparent}\left(t\right)-{\widehat{\mathrm{NRI}}}^{0.632bt}\left(t\right)$. Specifically, if [L, R] is the confidence interval calculated based on the procedure described above, then the bias corrected confidence interval is $[L-\widehat{\text{bias}},R-\widehat{\text{bias}}]$.

5 Simulation studies

To examine the performance of various NRI(t) estimators, we conducted simulation studies under several different scenarios. Throughout we chose n = 500 and used 200 bootstrap samples to calculate standard errors. Results for each setting were produced from 1,000 simulations. We calculated NRI(t), for t = 3 for comparing two risk models using the KM, IPW, Smooth-IPW, SEM and the combined estimators described in Sect. 3. We fitted Cox regression models to calculate risks for both the new and existing models using corresponding predictors.

For the first setting presented in Table 1, two predictors Y₁ and Y₂ were simulated from a multivariate normal distribution with mean (0, 0.5), σ_y1 = σ_y2 = 1 and a correlation ρ of 0.25. The relationship between survival time T and Y followed a proportional hazards model with parameters β₁ = log(3) and β₂ equal to log(1.5). Censoring time was generated from a U(0, a) distribution where a was chosen to produce approximately 40% censoring. Note that in this setting, model Q is correctly specified and the independent censoring assumption is correct. We took the baseline model to consist of Y₁ and the new model to include both predictors. As expected, all estimators shown in Table 1 provide unbiased estimates. The bootstrap-based variance estimators perform well with coverage percentage close to the 95% nominal level. Since the risk based on the new model is correctly specified, the semiparametric method is the most efficient. Improvement in efficiency over the nonparametric procedures is observed with our combined estimators.

Table 1.

Simulation results under noninformative censoring and correctly specified new risk model with mean (mean of bias (mean(bias)) and standard deviation (Std. Dev.) of the estimated parameters across simulations, the mean of the standard error estimates calculated for each simulation using bootstrapping (mean(std error)), and coverage of the 95 % bootstrap confidence interval based on the normal approximation

Method		$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	NRI (t)
	True values	0.592	0.358	0.468
KM
	Mean(Bias)	0.003	0.001	0.002
	Std. Dev.	0.034	0.030	0.104
	Mean(std error)	0.034	0.030	0.103
	95 % bootstrap CI cov.	0.946	0.946	0.946
IPW
	Mean(Bias)	0.002	0.002	–0.001
	Std. Dev.	0.034	0.030	0.105
	Mean(std error)	0.034	0.031	0.104
	95 % bootstrap CI cov.	0.943	0.95	0.951
Smooth IPW
	Mean(Bias)	0.001	0.003	–0.003
	Std. Dev.	0.034	0.030	0.104
	Mean(std error)	0.034	0.030	0.103
	95 % bootstrap CI cov.	0.946	0.942	0.949
SEM
	Mean(Bias)	0.001	0.003	–0.003
	Std. Dev.	0.024	0.029	0.082
	Mean(std error)	0.025	0.028	0.080
	95 % bootstrap CI cov.	0.952	0.942	0.937
Combined
	Mean(Bias)	0.002	0.003	–0.002
	Std. Dev.	0.029	0.028	0.089
	Mean(std error)	0.031	0.029	0.095
	95 % bootstrap CI cov.	0.968	0.949	0.969

KM Kaplan–Meier estimator, IPW inverse probability weighted estimator, Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

Under this setting we also considered a null model where β₂ = 0 i.e. there is no incremental value of the new marker and NRI(t) = 0. We found that in this situation all estimators tend to slightly over estimate NRI(t), and variance estimators based on the bootstrap estimators tend to be conservative (see Table 2). We do not recommend calculating NRI(t) in the case when the new marker does not independently predict outcome in a model with conventional predictors. Note that all theoretical results in the Appendix are derived under the assumption that β₂ ≠ 0 and thus our proposed procedures are only valid under this assumption. In practice, if the null setting is a likely possibility, estimation should be treated with care.

Table 2.

Simulation results under noninformative censoring and correctly specified new risk model with mean of bias (mean(Bias)) and standard deviation (Std. Dev.) of the estimated parameters across simulations, the mean of the standard error estimates calculated for each simulation using bootstrapping (mean(std error)), and coverage of the 95 % bootstrap confidence interval based on the normal approximation. Data is generated under the null model that β₂ = 0

Method		$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	NRI(t)
Null model: β2 = 0
	True values	0.5	0.5	0
KM
	Mean(Bias)	0.01	–0.02	0.061
	Std. Dev.	0.034	0.026	0.091
	Mean(std error)	0.043	0.033	0.118
	95 % bootstrap CI cov.	0.996	0.971	0.98
IPW
	Mean(Bias)	0.01	–0.019	0.058
	Std. Dev.	0.034	0.026	0.092
	Mean(std error)	0.044	0.033	0.119
	95 % bootstrap CI cov.	0.996	0.972	0.981
Smooth IPW
	Mean(Bias)	0.009	–0.019	0.055
	Std. Dev.	0.034	0.026	0.092
	Mean(std error)	0.044	0.033	0.118
	95 % bootstrap CI cov.	0.996	0.972	0.981
SEM
	Mean(Bias)	0.009	–0.019	0.057
	Std. Dev.	0.023	0.025	0.067
	Mean(std error)	0.029	0.031	0.081
	95 % bootstrap CI cov.	0.99	0.967	0.957
Combined
	Mean(Bias)	0.008	–0.019	0.055
	Std. Dev.	0.029	0.025	0.077
	Mean(std error)	0.039	0.032	0.104
	95 % bootstrap CI cov.	0.997	0.971	0.977

KM Kaplan–Meier estimator, IPW inverse probability weighted estimator, Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

The second setting we considered was identical to the first setting, except that censoring time was dependent on marker values. Here, censoring time,

$$C=U\cdot B+\text{exp}(X-3Y_2)\cdot (1-B),$$

where U was generated from a Uniform(0, a) distribution where with a chosen to yield about 40% censoring, X was generated from a N(0, 1) distribution and B was generated from a N(2·Y₁, 1) distribution. Note that in this setting, model Q is correctly specified but the independent censoring assumption is not correct. As seen in the results presented in Table 3, the KM estimator yields biased estimators for both NRI(t) and its two key components. The IPW estimator is biased for both $\mathrm{Pr}(P>Q\mid T\le t)$ and NRI(t), whereas the smooth-IPW estimator substantially alleviates such biases. However, we observed large variation in nonparamatric estimators of NRI(t) as compared with the semiparametric and combined estimators (Table 3).

Table 3.

Simulation results under covariate-dependent censoring and correctly specified new risk model with mean of bias (mean(Bias)) and standard deviation (Std. Dev.) of the estimated parameters across simulations, the mean of the standard error estimates calculated for each simulation using bootstrapping (mean(std error)), and coverage of the 95 % bootstrap confidence interval based on the normal approximation

Method		$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	NRI(t)
	True values	0.611	0.45	0.322
KM
	Mean(Bias)	0.067	–0.062	0.259
	Std. Dev.	0.040	0.040	0.126
	Mean(std error)	0.041	0.040	0.129
	95 % bootstrap CI cov.	0.615	0.659	0.483
IPW
	Mean(Bias)	–0.024	0.005	–0.057
	Std. Dev.	0.034	0.045	0.131
	Mean(std error)	0.035	0.044	0.130
	95 % bootstrap CI cov.	0.897	0.944	0.918
Smooth IPW
	Mean(Bias)	–0.013	0.007	–0.038
	Std. Dev.	0.041	0.041	0.133
	Mean(std error)	0.040	0.040	0.132
	95 % bootstrap CI cov.	0.937	0.939	0.941
SEM
	Mean(Bias)	0	–0.001	0.002
	Std. Dev.	0.025	0.039	0.098
	Mean(std error)	0.026	0.037	0.095
	95 % bootstrap CI cov.	0.951	0.932	0.938
Combined
	Mean(Bias)	–0.006	0.002	–0.016
	Std. Dev.	0.031	0.039	0.109
	Mean(std error)	0.035	0.039	0.117
	95 % bootstrap CI cov.	0.975	0.951	0.971

KM Kaplan–Meier estimator, IPW inverse probability weighted estimator, Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

The third setting we investigated considers the case where survival time depends on four markers Y_i, for i = 1, . . . , 4, but we only have access to the first two. In particular, Y comes from a multivariate normal distribution with mean 0, and σ_ij = 1 for i = j and 0.25 otherwise. Survival time relates to the marker data through a model where the hazard function is specified as λ(t|Y) = 0.1*{3Y₁ + 1.5Y₂ + 2Y₃ + 2.5Y₄ + exp(3Y₁)}. Note that in this setting, model Q is misspecified as depending only on Y₁ and Y₂. Censoring time in this setting is generated the same as in the first setting, which does not dependent on T or Y. Since the SEM estimator misspecified the relationship between T and Y as λ(t|Y) = λ₀ exp(β₁Y₁ + β₂Y₂), it yields biased results. All other estimators are unbiased (Table 4). Throughout the three settings we considered, the combined estimator remained unbiased and more efficient than other nonparametric estimators.

Table 4.

Simulation results under noninformative censoring and misspecified new risk model with mean of bias (mean(Bias)) and standard deviation (Std. Dev.) of the estimated parameters across simulations, the mean of the standard error estimates calculated for each simulation using bootstrapping (mean(std error)),and coverage of the 95 % bootstrap confidence interval based on the normal approximation

Method		$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	NRI (t)
	True values	0.646	0.395	0.504
KM
	Mean(Bias)	0.007	–0.002	0.016
	Std. Dev.	0.072	0.023	0.160
	Mean(std error)	0.074	0.024	0.164
	95 % bootstrap CI cov.	0.94	0.945	0.947
IPW
	Mean(Bias)	0.004	–0.001	0.008
	Std. Dev.	0.072	0.023	0.160
	Mean(std error)	0.074	0.024	0.165
	95 % bootstrap CI cov.	0.945	0.942	0.95
Smooth IPW
	Mean(Bias)	0.003	–0.001	0.007
	Std. Dev.	0.072	0.023	0.160
	Mean(std error)	0.074	0.024	0.164
	95 % bootstrap CI cov.	0.943	0.946	0.95
SEM
	Mean(Bias)	–0.046	0.003	–0.099
	Std. Dev.	0.022	0.022	0.068
	Mean(std error)	0.022	0.023	0.068
	95 % bootstrap CI cov.	0.448	0.943	0.682
Combined
	Mean(Bias)	–0.009	0.000	–0.020
	Std. Dev.	0.057	0.022	0.128
	Mean(std error)	0.062	0.023	0.139
	95 % bootstrap CI cov.	0.970	0.947	0.976

KM Kaplan–Meier estimator, IPW inverse probability weighted estimator, Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

To evaluate the procedures described above, we simulated 10 markers from a multivariate normal distribution with mean 0, σ_Yi = 1 and pairwise correlations equal to 0.25. The number of parameters and sample size were chosen to mimic the setting of our data example described in Sect. 6. We consider a Cox model for failure time, with hazard ratio parameters for 10 markers specified as β = (log(2), log(.77), 0, log(1.81), 0, 0, 0, log(0.5), 0, log(1.2)). The baseline model consists only of the first marker. To derive a new model based on the information on all 10 markers, for each simulation, we first fit a model with all ten markers. The expanded model consists of all markers that have non-zero β at an α = 0.05 level. We found that in the case of estimating NRI, under our simulated scenario, the apparent summaries are quite close to the true values in many cases. Since the bias is at the rate of g/N, where g is the number of predictors under consideration for risk model building, overfitting may be of more concern when large numbers of genetic markers are involved with a relatively small sample size. In the situation there is a slight indication of overfitting, the 0.632 bootstrap procedure appears to be adequate in correcting the bias (see Table 5).

Table 5.

Simulation results comparing apparent estimates and the 0.632 bootstrap for correcting overfitting

Estimator		$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	NRI(t)
	True values	0.684	0.275	0.817
Smooth IPW	Apparent
	Mean(Bias)	0.000	0.004	–0.007
	Std. Dev.	0.036	0.028	0.108
	CI coverage	0.962	0.963	0.964
	0.632 Bootstrap
	Mean(Bias)	–0.008	0.008	–0.032
	Std. Dev.	0.034	0.027	0.102
	CI coverage	0.971	0.969	0.968
	Bootstrapped SE
	Mean(std error)	0.039	0.030	0.114
SEM	Apparent
	Mean(Bias)	0.003	–0.001	0.009
	Std. Dev.	0.023	0.025	0.072
	CI coverage	0.955	0.954	0.945
	0.632 Bootstrap
	Mean(Bias)	0.005	–0.003	0.015
	Std. Dev.	0.022	0.024	0.072
	CI coverage	0.953	0.962	0.937
	Bootstrapped SE
	Mean(std error)	0.024	0.025	0.071
Combined	Apparent
	Mean(Bias)	0.001	0.001	0.001
	Std. Dev.	0.028	0.026	0.087
	CI coverage	0.982	0.969	0.975
	0.632 Bootstrap
	Mean(Bias)	–0.002	0.003	–0.008
	Std. Dev.	0.027	0.025	0.085
	CI coverage	0.989	0.975	0.983
	Bootstrapped SE
	Mean(std error)	0.035	0.028	0.102

Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

6 Example

The Framingham risk model (FRM) has been used for population-wide CVD risk assessment. The model was developed based on several common clinical risk factors, including age, gender, total cholesterol level, high-density lipoprotein (HDL) cholesterol level, smoking, systolic blood pressure and high blood pressure treatment (Wilson et al. 1998). To improve the predictive capacity of the FRM, a new risk model has been developed recently using data from the Women's Health Study (Cook et al. 2006), based on variables in the Framingham risk model and an inflammation marker, C-reactive protein (CRP). Prior to adapting the new model in routine practice, it is important to quantify its prediction performance, especially in comparison to that of FRM. We illustrate here how our proposed procedures can be used to evaluate and compare the clinical utility of the two risk models using an independent dataset from the Framingham Offspring Study (Kannel et al. 1979).

The Framingham Offspring Study was established in 1971 with 5,124 participants who were monitored prospectively for epidemiological and genetic risk factors for CVD. We consider here 1,728 female participants who have CRP measurement and other clinical information at the second exam and are free of CVD at the time of examination. The average age of this subset was about 44 years (standard deviation = 10). The outcome we consider is the time from exam date to first major CVD event including CVD-related death. During the followup period 269 participants were observed to encounter at least one CVD event and the 10-year event rate was about 4%. For illustration we chose t = 10 years as in Wilson et al. (1998). For each individual, two risk scores were calculated: one based on the FRM (Model 1), combining information on age, systolic blood pressure, smoking status, high-density lipoprotein (HDL), total cholesterol, medication for hypertension; the other based on an algorithm developed in Cook et al. (2006) (Model 2), with the addition of CRP concentration. We use Cox models to specify the relation between the time-to-CVD events and model scores (linear predictors from the models).

Both models are well calibrated based on calibration plots (not shown). For comparison, we first give AUC results and use the bootstrap to obtain confidence intervals. The AUC for an ROC curve at 10-years is 0.752 (95% CI: 0.721,0.783) for Model 1 and 0.758 (95% CI: 0.729,0.787) for Model 2. The difference between the two AUCs is not statistically significant: 0.006 (95% CI: –0.033, 0.046). We now investigate whether the new models reclassify patients in terms of their risks and CVD outcome at 10 years. We consider NRI (10-years) for such an evaluation using the methods described in Sect. 3. Table 6 shows that estimates from the three nonparametric models are quite consistent, all indicating that the new model does not add significant improvement gauged by NRI. The semiparametric model, however, does indicate a significant incremental value with NRI = 0.167 (SE = 0.067), and the combined estimator indicates a similar magnitude of improvement, though not significant (NRI = 0.132, SE = 0.137). Note that since we considered a continuous NRI with u = v = 0, the observed improvement at this magnitude may not be interpreted as clinically substantial. Since different conclusions could be reached depending on which estimation method is chosen, this analysis highlights the need to consider multiple robust approaches for calculating NRI.

Table 6.

NRI estimates for two risk models for predicting 10-year CVD risk among women in the Framingham offspring cohort

Method	$\mathrm{Pr}(P_i-Q_i>0\mid T_i\le t)$	Pr(Pi – Qi > 0|Ti > t)	$NRI(t)
KM
Est	0.483	0.508	–0.049
SE	0.069	0.028	0.176
IPW
Est	0.478	0.508	–0.059
SE	0.070	0.028	0.178
Smooth IPW
Est	0.480	0.508	–0.057
SE	0.070	0.028	0.178
SEM
Est	0.587	0.503	0.167
SE	0.015	0.026	0.067
Combined
Est	0.570	0.504	0.132
SE	0.054	0.027	0.137

KM Kaplan–Meier estimator, IPW inverse probability weighted estimator, Smooth IPW smooth inverse probability weighted estimator, SEM semiparametric estimator, Combined combined estimator, as defined in the text

7 Discussion

NRI provides an alternative tool for evaluating risk prediction models (Pencina et al. 2008) beyond the traditional ROC curve framework. The concept has continued to gain popularity in the medical literature, yet its statistical properties have not been well studied to date in the statistical literature, and existing methods for calculating NRI under the failure time outcome setting are limited. In this manuscript, we provide a more thorough investigation of a variety of estimation procedures. Our proposed nonparametric and semiparametric estimators improve upon existing methods both in terms of robustness and efficiency under a variety of practical situations. Such improvement is quite important, since we observe that compared with other measures such as AUC, NRI estimates, in general, are not very stable with substantial variations in the estimators we have considered. The proposed procedures can be used for estimating both continuous NRI and NRI with pre-specified fixed categories. As illustrated in the example, the choice of estimation method can lead to different conclusions. In practice, the method chosen should depend on a number of important considerations including the likelihood that the model has been correctly specified and that the assumptions concerning censoring are correct. In addition, in situations where the new marker may be expensive or difficult to ascertain, an approach which considers both the risks and benefits of obtaining the marker should be considered in a decision-making process. We recommend such measures to be used in practice with caution. A thorough evaluation of a risk model should consider a wide spectrum of measures for assessing discrimination and calibration, and NRI may be better served as one of the summary measures to complement graphical displays of risk distributions(Gu and Pepe 2009). All analyses were performed in R. Code for implementing the proposed procedures is available upon request.

Appendix

Throughout, we assume that the joint density of (T, C, Y) is twice continuously differentiable, Y are bounded, and 1 > P(T > t) > 0, 1 > P(C > t) > 0. The kernel function K is a symmetric probability density function with compact support and bounded second derivative. The bandwidth h → 0 such that nh⁴ → 0. In addition, the estimator ${\widehat{\theta }}_k$ converges to θ_0k for k = 1, 2 as n → ∞ (Hjort 1992), where β_k0 is the unique maximizer of the expected value of the corresponding partial likelihood and Λ_k0 is the baseline cumulative hazard for k = 1, 2. We denote the parameter space for θ_k by Ω_k and assume that Ω_k is a compact set containing θ_0k. Furthermore, we assume that β₂ ≠ 0 and note that $Q\left({\theta }_2\right)=1-\text{exp}\left\{{\Lambda }_{02}\left(t\right)e^{{\beta }_2^{\mathsf{T}}{\mathbf{Y}}_{\left(2\right)}}\right\}$ and $P\left({\theta }_1\right)=1-\text{exp}\left\{{\Lambda }_{01}\left(t\right)e^{{\beta }_1^{\mathsf{T}}{\mathbf{Y}}_{\left(1\right)}}\right\}$ are the respective limits of $Q\left({\widehat{\theta }}_2\right)$ and $P\left({\widehat{\theta }}_1\right)$, for any given Y₍₂₎ and Y₍₁₎. The in-probability convergence of $Q\left({\widehat{\theta }}_2\right)\to Q\left({\theta }_{02}\right)$ and $P\left({\widehat{\theta }}_1\right)$ and P(θ₀₁) are uniform in Y₍₂₎ and Y₍₁₎ due to the convergence of $\widehat{\theta }\to {\theta }_0={({\theta }_{01}^{\mathsf{T}},{\theta }_{02}^{\mathsf{T}})}^{\mathsf{T}}$.

Asymptotic Properties of $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$

From the same arguments as given in Cai et al. (2010) and Dabrowska (1997), it follows that we have the uniform consistency of ${\tilde{H}}_q^{\left(\iota \right)}\left(t\right)$ to ${\tilde{H}}_q^{\left(\iota \right)}\left(t\right)=P(C\ge t\mid Q\left({\theta }_2\right)=q,\Delta \left(\theta \right)\in {\mho }_{\iota }$, where ${\mho }_1=1$ and ${\mho }_{•}=\{0,1\}$, for ι = 1 and •. It follows, using the law of numbers (Pollard 1990), that

$$\underset{\theta }{\text{sup}}\mid \widetilde{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)\mid \to 0.$$

This along with the convergence of $\widehat{\theta }$ to θ₀ implies that $\widetilde{\mathrm{NRI}}(\widehat{\theta },t)$ is uniformly consistent for NRI(θ₀, t).

Throughout, we will use the fact that $E\{{\Delta }_i\left(\theta \right)I(X_i\le t){\delta }_i{H}_{Q_i\left({\theta }_2\right)}^{\left(1\right)}{\left(X_i\right)}^{-1}\mid Q_i\left({\theta }_2\right)=q\}=P({\Delta }_i\left(\theta \right)=1,T_i\le t\mid Q_i\left({\theta }_2\right)=q)$ if either C ⊥ T, Y₍₂₎ (model may be misspecified) or $Q\left({\theta }_2\right)=\mathrm{Pr}(T\le t\mid Y_{\left(2\right)})$ i.e. the Cox model is correctly specified though censoring may be such that C ⊥ T | Y₍₂₎ (double robustness). We first write the i.i.d representation of $\sqrt{n}[\widetilde{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)]$ for any θ. Note that $\sqrt{n}\{\widetilde{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)\}=2\sqrt{n}\{\widetilde{\mathrm{Pr}}(\Delta \left(\theta \right)-1\mid T\le t)-\mathrm{Pr}\left(\Delta \left(\theta \right)=1\mid T\le t)\right\}-2\sqrt{n}\{\widetilde{\mathrm{Pr}}(\Delta \left(\theta \right)=1\mid T>t)-\mathrm{Pr}(\Delta \left(\theta \right)=1\mid T>t)\}$. We first examine the initial component,

$$\widetilde{\mathrm{Pr}}(\Delta \left(\widehat{\theta }\right)=1\mid T\le t)=\frac{{\sum }_i\Delta \left(\widehat{\theta }\right)I(X_i\le t){\delta }_i∕{\tilde{H}}_{Q_i\left({\widehat{\theta }}_2\right)}^{\left(1\right)}\left(X_i\right)}{{\sum }_{i}I(X_i\le t){\delta }_i∕{\tilde{H}}_{Q_i\left({\widehat{\theta }}_2\right)}^{\left(•\right)}\left(X_i\right)}\equiv \frac{\widehat{N}(t,\widehat{\theta },\tilde{H})}{\widehat{D}(t,\widehat{\theta },\tilde{H})}$$

where $\widehat{N}(t,\theta ,H)=n^{-1}{\sum }_i{\Delta }_i\left(\theta \right)I(X_i\le t){\delta }_i∕H_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)$ and $\widehat{D}(t,\theta ,H)=n^{-1}{\sum }_{i}I(X_i\le t){\delta }_i∕H_{Q_i\left({\theta }_2\right)}^{\left(•\right)}\left(X_i\right)$. Let $N(t,\theta )=\mathrm{Pr}(\Delta \left(\theta \right)=1,T\le t)$ and $D\left(t\right)=\mathrm{Pr}(T\le t)$. Then by the uniform consistency of the IPW weights, we have

$$\begin{array}{cc}\hfill & \sqrt{n}\{\widetilde{\mathrm{Pr}}(\Delta \left(\theta \right)=1\mid T\le t)-\mathrm{Pr}(\Delta \left(\theta \right)=1\mid T\le t)\}\hfill \\ \hfill & \approx \sqrt{n}\{\widehat{N}(t,\theta ,\tilde{H})D\left(t\right)-N(t,\theta )\widehat{D}(t,\theta ,\tilde{H})\}∕D{\left(t\right)}^2.\hfill \end{array}$$

Examining the numerator, $\sqrt{n}\{\widehat{N}(t,\theta ,\tilde{H})D\left(t\right)-N(t,\theta )\widehat{D}(t,\theta ,\tilde{H})\}=\sqrt{n}\{\left(1\right)+\left(2\right)-\left(3\right)\}$ where $\left(1\right)=\widehat{N}(t,\theta ,H)D\left(t\right)-\widehat{D}(t,\theta ,H)N(t,\theta ),\left(2\right)=\widehat{N}(t,\theta ,\tilde{H})D\left(t\right)-\widehat{N}(t,\theta ,H)D\left(t\right)$, and $\left(3\right)=[N(t,\theta )\widehat{D}(t,\theta ,\tilde{H})-\widehat{D}(t,\theta ,H)N(t,\theta )]$. Note that

$$\begin{array}{cc}\hfill \left(1\right)& =\sqrt{n}(\widehat{N}(t,\theta ,H)D\left(t\right)-\widehat{D}(t,\theta ,H)N(t,\theta ))=n^{-\frac{1}{2}}\sum U_{1i}\left(t\right),\text{where}\hfill \\ \hfill U_{1i}\left(t\right)& =\frac{I(X_i\le t){\delta }_i}{H_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)}{\Delta }_i\left(\theta \right)D\left(t\right)-\frac{I(X_i\le t){\delta }_i}{H_{Q_i\left({\theta }_2\right)}^{\left(•\right)}\left(X_i\right)}N(t,\theta )\hfill \end{array}$$

Using a Taylor series expansion, Lemma A.3 of Bilias et al. (1997) and the asymptotic expansion for ${\widehat{\Lambda }}_q\left(t\right)$ given in Du and Akritas (2002),

$$\begin{array}{cc}\hfill \left(2\right)& =D\left(t\right)\sqrt{n}\{\widehat{N}(t,\theta ,\tilde{H})-\widehat{N}(t,\theta ,H)\}\hfill \\ \hfill & =D\left(t\right)n^{-1∕2}\sum _i\frac{{\Delta }_i\left(\theta \right)I(X_i\le t){\delta }_i}{H_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)}\left[\frac{H_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)}{{\tilde{H}}_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)}-1\right]\hfill \\ \hfill & =D\left(t\right)n^{-1∕2}\int {\int }_0^t\left[\frac{H_q^{\left(1\right)}\left(s\right)}{{\tilde{H}}_q^{\left(1\right)}\left(s\right)}-1\right]d\sum _i\frac{{\Delta }_i\left(\theta \right){\delta }_{i}I(X_i\le s,Q_i\left({\theta }_2\right)\le q)}{H_{Q_i\left({\theta }_2\right)}^{\left(1\right)}\left(X_i\right)}\hfill \\ \hfill & \approx D\left(t\right)\int {\int }_0^t\sqrt{n}\left[{\widehat{\Lambda }}_q^{\left(1\right)}\left(s\right)-{\Lambda }_q^{\left(1\right)}\left(s\right)\right]d\left\{\frac{1}{n}\sum _i\frac{{\Delta }_i\left(\theta \right){\delta }_{i}I(X_i\le s,Q_i\left({\theta }_2\right)\le q)}{H_{Q_i\left({\theta }_2\right)}^{\left(1\right)},\left(X_i\right)}\right\}\hfill \\ \hfill & \approx D\left(t\right)\int {\int }_0^t\left[n^{-\frac{1}{2}}\sum K_h\{q-Q_i\left({\theta }_2\right)\}{M}_{Cq}^{\left(1\right)}(s,X_i,{\delta }_i)\right]dP(\Delta \left(\theta \right)=1,T\le t,Q\left({\theta }_2\right)\le q)\hfill \end{array}$$

where

$$M_{Cq}^{\left(1\right)}(t,X_i,{\delta }_i)={\int }_0^t\frac{d{N}_{Ci}\left(s\right)-I(X_i\ge s)d{\Lambda }_q^{\left(1\right)}\left(s\right)}{{\pi }_s^{\left(1\right)}\left(q\right)}.$$

Now by a change of variable, $\psi =\frac{q-Q_i\left({\theta }_2\right)}{h}$ and $f(t,q)\equiv {\partial }^{2}P(\Delta \left(\theta \right)=1,T\le t,Q\left({\theta }_2\right)\le q)∕\partial t\partial q$,

$$\begin{array}{cc}\hfill \left(2\right)& \approx D\left(t\right)\int {\int }_0^t\sqrt{n}\left[\frac{1}{n}\sum K\left(\psi \right)M_{C(\psi h+Q_i\left({\theta }_2\right))}(s,X_i,{\delta }_i)\right]f(t,\psi h+Q_i)dsd\psi \hfill \\ \hfill & =D\left(t\right)n^{-1∕2}\sum \int {\int }_0^{t}K\left(\psi \right)a\{s,h\psi +Q_i\left({\theta }_2\right),X_i\}dsd\psi =n^{-\frac{1}{2}}\sum U_{2i}\left(t\right),\hfill \end{array}$$

where $U_{2i}\left(t\right)=D\left(t\right){\int }_0^{t}a(s,q^{\ast },X_i)ds\text{and}a(t,q,X_i)=M_{C{q}^{\ast }}(t,X_i,{\delta }_i)f(t,q^{\ast })$. Similar arguments can be used to obtain an asymptotic expansion for (3) as $\left(3\right)\approx n^{-\frac{1}{2}}\sum U_{3i}\left(t\right)$ and therefore, the numerator, $\sqrt{n}\left[\widehat{N}(t,\theta ,\tilde{H})D\left(t\right)-N(t,\theta )\widehat{D}(t,\theta ,\tilde{H})\right]\approx n^{-\frac{1}{2}}\sum \{U_{1i}\left(t\right)+U_{2i}\left(t\right)+U_{3i}\left(t\right)\}$. The same arguments as given above can be used to obtain an asymptotic expansion for $\sqrt{n}\{\widetilde{\mathrm{Pr}}(\Delta \left(\theta \right)=1\mid T>t)-\mathrm{Pr}(\Delta \left(\theta \right)=1\mid T>t)\}$ as $n^{-\frac{1}{2}}{\sum }_{i=1}^{n}D{\left(t\right)}_{-}^{-2}\{U_{-1i}\left(t\right)+U_{-2i}\left(t\right)+U_{-3i}\left(t\right)\}$ where D(t)_–, U_–1i(t), U_–2i(t), and U_–3i(t) are defined similarly to D(t), U_1i(t), U_2i(t), and U_3i(t) with $T\le t$ replaced with T > t. Therefore, $\sqrt{n}\{\widetilde{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)\}\approx n^{-\frac{1}{2}}{\sum }_{i=1}^{n}2[D{\left(t\right)}^{-2}\{U_{1i}\left(t\right)+U_{2i}\left(t\right)+U_{3i}\left(t\right)\}-D{\left(t\right)}_{-}^{-2}\{U_{-1i}\left(t\right)+U_{-2i}\left(t\right)+U_{-3i}\left(t\right)\}]=n^{-\frac{1}{2}}{\sum }_{i=1}^n{\eta }_i\left(t\right)$.

Note that regardless of correct model specification, $\sqrt{n}(\widehat{\theta }-{\theta }_0)=n^{-1∕2}\sum {\psi }_i+o_p\left(1\right)$ where ψ_i are i.i.d mean zero random variables by Lin and Wei (1989) and Uno et al. (2009). Using a Taylor series approximation and the i.i.d representation of $\sqrt{n}[\widetilde{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)]$ for any θ, we can write $\tilde{\mathcal{W}}\left(t\right)=\sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)]$ as a sum of i.i.d terms, $n^{-1∕2}{\sum }_{i=1}^n{∊}_i\left(t\right)$ defined below.

$$\begin{array}{cc}\hfill \sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)]& =\sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}(\widehat{\theta },t)+\mathrm{NRI}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)]\hfill \\ \hfill & \approx \sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}(\widehat{\theta },t)+\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}(\widehat{\theta }-{\theta }_0)\hfill \\ \hfill & =\sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}(\widehat{\theta },t)]+\sqrt{n}(\widehat{\theta }-{\theta }_0)\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}\hfill \\ \hfill & \approx \sqrt{n}[\widetilde{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}(\widehat{\theta },t)]+n^{-1∕2}\sum {\psi }_i\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}\hfill \\ \hfill & \approx n^{-1∕2}\sum _{i=1}^n{\eta }_i\left(t\right)+n^{-1∕2}\sum {\psi }_i\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}\hfill \\ \hfill & =n^{-1∕2}\sum _{i=1}^n{∊}_i\left(t\right)\hfill \end{array}$$

where ${∊}_i(u,v,t)={\eta }_i(u,v,t)+{\psi }_i\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}$. By a functional central limit theorem of Pollard (1990), the process $\tilde{\mathcal{W}}\left(t\right)$ converges weakly to a mean zero Gaussian process in t.

Asymptotic Properties of $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$

Recall that we assume the Cox model is correctly specified and thus, $Q\left({\theta }_2\right)=Q({\theta }_2,t,{\mathbf{Y}}_{\left(2\right)})=\mathrm{Pr}(T\le t\mid Y_{\left(2\right)})=1-\text{exp}\left\{{\Lambda }_{02}\left(t\right)e^{{\beta }_2^{\mathsf{T}}{Y}_{\left(2\right)}}\right\}$ and $S_{Q_i\left({\theta }_2\right)}\left(t\right)=\mathrm{Pr}(T>t\mid Y_{\left(2\right)})=\text{exp}\left\{{\Lambda }_{02}\left(t\right)e^{{\beta }_2{Y}_{\left(2\right)}}\right\}$. To derive asymptotic properties of $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$ we assume the same regularity conditions as in Andersen and Gill (1982). The uniform consistency of $Q({\widehat{\theta }}_2,t,{\mathbf{Y}}_{\left(2\right)})$ for Q(θ₂, t, Y₍₂₎) in t and Y₍₂₎ follows directly from the uniform consistency of ${\widehat{\Lambda }}_{02}\left(t\right)$ and ${\widehat{\beta }}_2$. It follows from the uniform law of large numbers (Pollard 1990) that $\widehat{\mathrm{NRI}}(\widehat{\theta },t)$ is uniformly consistent for NRI(θ₀, t). Andersen and Gill (1982) show that $\sqrt{n}({\widehat{\beta }}_2-{\beta }_{02})$ is a normal random variable and $\sqrt{n}({\widehat{\Lambda }}_{02}\left(t\right)-{\Lambda }_{02}\left(t\right))$ converges to a Gaussian process. By the functional delta method it can be shown that $\sqrt{n}\{Q({\widehat{\theta }}_2,t,{\mathbf{Y}}_{\left(2\right)})-Q({\theta }_2,t,{\mathbf{Y}}_{\left(2\right)})\}$ converges to a zero mean Gaussian process in t and Y₍₂₎ (Zheng et al. 2008). Similar to the derivation for $\widetilde{\mathrm{NRI}}(\tilde{\theta },t)$, it can be shown that the process $\tilde{\mathcal{N}}\left(t\right)=\sqrt{n}[\widehat{\mathrm{NRI}}(\widehat{\theta },t)-\mathrm{NRI}({\theta }_0,t)]$ is asymptotically equivalent to $n^{-1∕2}{\sum }_{i=1}^n{\zeta }_i(u,v,t)$. In particular, for a fixed θ, $\sqrt{n}\{\widehat{\mathrm{NRI}}(\theta ,t)-\mathrm{NRI}(\theta ,t)\}\approx n^{-1∕2}{\sum }_{i=1}^n{\eta }_i^{\ast }\left(t\right)$ where ${\eta }_i^{\ast }\left(t\right)=2[D{\left(t\right)}^{-2}\{{\Delta }_i\left(\theta \right)Q_i\left({\theta }_2\right)-\mathrm{Pr}({\Delta }_i\left(\theta \right)=1\mid T_i\le t)Q_i\left({\theta }_2\right)\}-D{\left(t\right)}_{-}^{-2}\{{\Delta }_i\left(\theta \right)[1-Q_i\left({\theta }_2\right)]-\mathrm{Pr}({\Delta }_i\left(\theta \right)=1\mid T_i>t)[1-Q_i\left({\theta }_2\right)]\}]$. Thus, $\tilde{\mathcal{N}}\left(t\right)\approx n^{-1∕2}{\sum }_{i=1}^n{\zeta }_i\left(t\right)$ where ${\zeta }_i(u,v,t)={\eta }_i^{\ast }\left(t\right)+{\psi }_i\frac{\partial \mathrm{NRI}\left(t\right)}{\partial \theta }{\mid }_{{\theta }_0}$. Once again, using a functional central limit theorem, this implies that $\tilde{\mathcal{N}}\left(t\right)$ converges to a Gaussian process with mean zero.