Nonparametric estimation of the cumulative incidence function under outcome misclassification using external validation data

Abstract

Misclassification of outcomes or event types is common in health sciences research and can lead to serious bias when estimating the cumulative incidence functions in settings with competing risks. Recent work has shown how to estimate nonparametric cumulative incidence functions in the presence of nondifferential outcome misclassification when the misclassification probabilities are known. Here, we extend this approach to account for misclassification that is differential with respect to important predictors of the outcome using misclassification probabilities estimated from external validation data. Moreover, we propose a bootstrap approach in which the observations from both the main study data and the external validation study are resampled to allow the uncertainty in the misclassification probabilities to propagate through the analysis into the final confidence intervals, ensuring appropriate confidence interval coverage probabilities. The proposed estimator is shown to be uniformly consistent and simulation studies indicate that both the estimator and the standard error estimation approach perform well in finite samples. The methodology is applied to estimate the cumulative incidence of death and disengagement from HIV care in a large cohort of HIV infected individuals in sub-Saharan Africa, where a significant death underreporting issue leads to outcome misclassification. This analysis uses external validation data from a separate study conducted in the same country.

1 |. INTRODUCTION

One goal of health sciences research is to describe the natural history of a specific health outcome in a given target population. A useful parameter for describing this natural history is the risk^1,2 or the set of cumulative incidence functions for the outcomes of interest. We can consistently estimate the cumulative incidence functions under a set of assumptions, including (1) that the time to the event is measured without error; (2) that, in settings in which more than one event type is possible, such as when competing events may occur, the event type is observed without error; and (3) that participants in the sample who are under observation at time t are exchangeable with participants in the target population not under observation at time t.

In some settings, the event times are thought to be known, but the event type is subject to misclassification. When an internal validation sample is available, the problem reduces to the problem of missing cause of failure in the competing risks model and the analysis can be performed using methods for missing data, such as multiple imputation.^3–5 More recent proposals for the analysis of misclassified competing risks data and more general Markov processes with interval validation samples are based on maximum nonparametric pseudolikelihood estimation.⁶ In settings without internal validation data, Bakoyannis and Yiannoutsos⁷ explored the extent of bias in cumulative incidence functions due to nondifferential misclassification of event type and proposed a simple uniformly consistent estimator to correct for such missclassification when the misclassification probabilities are known.

However, the probability that event type is misclassified is usually not known with certainty. Rather, the probability of misclassification is often estimated from internal or external validation data. Ignoring the uncertainty of the estimated misclassification probabilities may lead to estimates that are overly precise, resulting in confidence interval (CI) coverage that is too low.⁸ Moreover, if the misclassification probabilities depend to one or more variables (ie, there is differential misclassification), applying average misclassification probabilities may produce biased estimates of the cumulative incidence of the event of interest. Here, we present an approach to account for both differential and nondifferential misclassification of event type when estimating nonparametric cumulative incidence functions in settings with external validation data. To account for the uncertainty in the misclassification probabilities that arises when estimating these probabilities from a finite external validation data set, we follow the works of Greenland⁸ and Chatterjee and Chatterjee⁹ and employ a bootstrap approach in which both the main study and the external validation data are resampled to compute the standard error of the risk estimates.

We illustrate the proposed approach to estimate the cumulative incidence of mortality in the Family AIDS Care and Education Services (FACES) cohort. In FACES, mortality is underascertained such that people who die are sometimes incorrectly classified as lost to care, leading to bias in estimates of the cumulative incidence of mortality and the cumulative incidence of loss to care. In this setting, an external validation data set, in which a study team traced a sample of patients lost to care in a nearby cohort to determine vital status, was available.

The outline of the remainder of this paper is as follows. Notation, parameters of interest, and the proposed estimator are defined in Section 2. In Section 3, we explore the finite-sample properties of the proposed estimator using a simulation experiment. Section 4 applies the proposed estimator to analysis of data from the motivating study outlined above. A discussion is provided in Section 5. The Appendix presents a proof of uniform consistency of the proposed estimators.

2 |. DATA AND METHOD

For simplicity, we consider two competing events throughout this paper. However, it is straightforward to generalize the methods presented here to the case of more than two competing outcomes. Let T_i and C_i denote the failure time and the true failure cause, respectively, for individual i, with C_i ∈ {1, 2}. Without loss of generality, assume that the cause of interest is C_i = 1. Let U_i denote the right censoring time. X_i represents the observed failure time such that X_i = min(T_i, U_i). Among censored individuals (eg, those for whom X_i = U_i), C_i = 0. The study duration interval is denoted by [0, τ], where τ is finite. The basic identifiable quantities from competing risks data are the cause-specific hazards and the cumulative incidence functions.^10–12 The cause specific hazard function for event type j is defined as

$${\lambda }_j(t)=\underset{\mathrm{\Delta }t\to 0}{\lim }\frac{1}{\mathbf{\Delta }t}P(t\le T<t+\mathbf{\Delta }t,C=j|T\ge t),j=1,2.$$

The cumulative incidence function is defined as

$$F_j(t)=P(T\le t,C=j)=\underset{0}{\overset{t}{\int }}S(v-)d{\mathrm{\Lambda }}_j(v),j=1,2,$$

where Λ_j(t) is the cumulative cause-specific hazard, ${\mathrm{\Lambda }}_j(t)={\int }_0^t{\lambda }_j(v)dv$. The cumulative incidence depends on the cause-specific hazards for both causes of failure through the overall survival function S(·). The cumulative incidence function can be estimated nonparametrically using the Aalen-Johansen estimator¹³

$${\widehat{F}}_j(t)=\underset{0}{\overset{t}{\int }}\widehat{S}(v-)d{\widehat{\mathrm{\Lambda }}}_j(v),j=1,2,$$

where $\widehat{S}$ and ${\widehat{\mathrm{\Lambda }}}_j$ are the Kaplan-Meier and Nelson-Aalen estimators for the overall survival function and the cumulative cause-specific hazard function, respectively.

In settings where the outcome ascertainment procedure is imperfect, we observe $C_i^{\star }$ in place of C_i, where $C_i^{\star }$ is the measured, and potentially misclassified cause of failure. Misclassification occurs when $C_i^{\star }\ne C_i$ for at least some study participants. We define misclassification probabilities a and b as

$$a=P\left(C_i^{\star }=2|C_i=1\right)$$

and

$$b=P\left(C_i^{\star }=1|C_i=2\right).$$

Here, as in our motivating example, we assume that right censoring is always correctly ascertained, such that $C_i^{\star }=0$ if and only if C_i = 0. Misclassification can be addressed by (a) measuring the outcome type using a gold standard diagnostic procedure in a small random subsample of the participants to retrieve the true C_i (as in an internal validation sample), (b) using estimates of the misclassification probabilities based on an external validation study, or (c) drawing on prior knowledge of the misclassification probabilities. In studies where there are no internal validation data, but the misclassification probabilities are a priori known, the estimator by Bakoyannis and Yiannoutsos⁷ can be used to account for misclassification

$${\widehat{F}}_1^{(1)}(t;a,b)=\sup \left\{{\widehat{F}}_1(v;a,b):0\le v\le t\right\},$$ (1)

and

$${\widehat{F}}_1(v;a,b)=\frac{1-b}{1-a-b}{\widehat{F}}_1^{\star }(v)-\frac{b}{1-a-b}{\widehat{F}}_2^{\star }(v),$$

where ${\widehat{F}}_j^{\star }(\cdot )$, j = 1, 2, is the Aalen-Johansen estimator based on the misclassified cause of failure C^⋆. Note that the supremum in (1) ensures that ${\widehat{F}}_1^{(1)}(t;a,b)$ will be monotonic; ${\widehat{F}}_1(t;a,b)$ may not be monotonically increasing.

If a and b are known, then ${\widehat{F}}_1^{(1)}(t;a,b)$ is uniformly consistent for F₁(t).⁷ If $\widehat{a}$ and $\widehat{b}$ are estimated from an external validation sample, ${\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})$ provides a reasonable estimator for F₁(t), provided that $\widehat{a}$ and $\widehat{b}$ are consistent for a and b in the main study (or transportable (see the work of Ha and Tsodikov^14p10) to the main study). More precisely, it can be shown that

$$\underset{t\in [0,\tau ]}{\sup }\left|{\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-F_1(t)\right|\stackrel{p}{\to }0,$$

as n → ∞ and n₀ → ∞ (where n is the sample size in the main study and n₀ is the number of events included in the external validation sample), which means that ${\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})$ is uniformly consistent for F₁(t).

2.1 |. Accounting for differential misclassification of event type

When misclassification depends on time T or participant characteristics Z (eg, gender), we can relax the misclassification probabilities a and b to be dependent on these factors such that $a(t,z)=P\left(C_i^{\star }=2|C_i=1,T_i=t,Z_i=z\right)$ and $b(t,z)=P\left(C_i^{\star }=1|C_i=2,T_i=t,Z_i=z\right)$. In this setting, we generalize the nonparametric estimator of the cumulative cause-specific hazard by Ha and Tsodikov¹⁴ to write

$${\widehat{\mathrm{\Lambda }}}_1^S(t;a,b)=\sup \left\{{\widehat{\mathrm{\Lambda }}}_1(v;a,b):0\le v\le t\right\}$$ (2)

and

$${\widehat{\mathrm{\Lambda }}}_1(v;a,b)=\underset{0}{\overset{v}{\int }}\frac{\sum _{i=1}^n{r}_1(s,z)d{N}_{i1}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)}-\underset{0}{\overset{v}{\int }}\frac{{\sum }_{i=1}^n{r}_2(s,z)d{N}_{i2}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)},$$

where r₁(t, z) = {1 − b(t, z_i)}∕{1 − a(t, z_i) − b(t, z_i)}, r₂(t, z) = b(t, z_i)∕{1 − a(t, z_i) − b(t, z_i)}, $N_{ij}^{\star }(t)=I\left(T_i\le t,C_i^{\star }=j\right)$, j = 1, 2, and Y_i(t) = I(T_i ≥ t).

Estimates of the misclassification probabilities $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ may be obtained using maximum likelihood. If the corresponding parametric models are correctly specified and $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ are consistent estimators of a(t, z) and b(t, z) in the main study, a reasonable estimator for F₁(t) is the plug-in estimator

$${\widehat{F}}_1^{(2)}(t;\widehat{a},\widehat{b})=\underset{0}{\overset{t}{\int }}\widehat{S}(v-)d{\widehat{\mathrm{\Lambda }}}_1^S(v;\widehat{a},\widehat{b}),t\in [0,\tau ].$$ (3)

More importantly,

$$\underset{t\in [0,\tau ]}{\sup }|{\widehat{F}}_1^{(2)}(t;\widehat{a},\widehat{b})-F_1(t)|\stackrel{p}{\to }0,$$

as n → ∞ and n₀ → ∞, which means that ${\widehat{F}}_1^{(2)}(t;\widehat{a},\widehat{b})$ is uniformly consistent for F₁(t) under differential (or nondifferential) misclassification. The proof of uniform consistency of this estimator is outlined in the Appendix.

Often, the choice between applying the estimator that allows for differential misclassification ${\widehat{F}}_1^{(2)}(t;\widehat{a},\widehat{b})$ and the estimator that does not ${\widehat{F}}_1^{(1)}$ relies on investigator knowledge of the misclassification process. If uncertain about whether a and b differ by T or Z, one may perform a likelihood ratio test comparing parametric models for $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ with and without terms for T and Z, provided that validation data containing information on T and Z are available.

2.2 |. Accounting for uncertainty $\widehat{\mathit{a}}$ and $\widehat{\mathit{b}}$ estimated from external validation data

Suppose an external validation data set is available in which both the gold standard outcome measure C_i and the possibly misclassified measure of the outcome type $C_i^{\star }$ are available for all participants. Information on the events occurring among those in the validation sample can be used to estimate $\widehat{a}$ and $\widehat{b}$ (if misclassification is nondifferential) or $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ if misclassification is differential. Note that, because we focus on accounting for misclassification of event type (ie, between C_i = 1 and C_i = 2), only events in the validation sample contribute to estimation of $\widehat{a}$ and $\widehat{b}$. We will let n₀ represent the number of events in the validation sample.

Assuming that misclassification is nondifferential with respect to predictors of the outcome, the misclassification probabilities a and b may be estimated in the external validation data as

$$\widehat{a}=\frac{{\sum }_{i=1}^{n_0}1\left(C_i^{\star }=2,C_i=1\right)}{{\sum }_{i=1}^{n_0}1\left(C_i=1\right)},\widehat{b}=\frac{{\sum }_{i=1}^{n_0}1\left(C_i^{\star }=1,C_i=2\right)}{{\sum }_{i=1}^{n_0}1\left(C_i=2\right)}$$

and one may consistently estimate F₁(t) by plugging $\widehat{a}$ and $\widehat{b}$ into (1).

As described above, we may relax the assumption that misclassification is nondifferential by allowing the misclassification probabilities to vary over time and strata of covariates z, provided that covariates z are available in both the main study data and the external validation data. In such cases, $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ may be estimated in the validation data using logistic regression such that

$$\widehat{a}(t,z)=\frac{\exp \left({\widehat{\beta }}_a^{T}w\right)}{1+\exp \left({\widehat{\beta }}_a^{T}w\right)}$$

and

$$\widehat{b}(t,z)=\frac{\exp \left({\widehat{\beta }}_b^{T}w\right)}{1+\exp \left({\widehat{\beta }}_b^{T}w\right)},$$

where w = (1, T_i, Z_i)^T. If the parametric models used to estimate $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ are correctly specified, one may consistently estimate F₁(t) by plugging $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ into (2).

However, simply plugging values of $\widehat{a}$ and $\widehat{b}$ or $\widehat{a}(t,z)$ and $\widehat{b}(t,z)$ estimated from the external validation study into (1) or (2) as in the work of Bakoyannis and Yiannoutsos⁷ ignores the uncertainty in the estimated misclassification probabilities, which will lead to CIs with a low coverage probability. Because the misclassification probabilities are estimated using an external and independent sample, we can define the following nonparametric bootstrap approach for 1−α CI calculation, when the external study data set is available.

For k = 1 to M,

1. Draw bootstrap sample k by sampling n participants with replacement from the main study data;

2. Draw bootstrap sample k by sampling n₀ events with replacement from the external validation data to estimate ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ or ${\widehat{a}}^{(k)}(t,z)$ and ${\widehat{b}}^{(k)}(t,z)$;

3. Apply ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ (or ${\widehat{a}}^{(k)}(t,z)$ and ${\widehat{b}}^{(k)}(t,z)$) to bootstrap sample k from the main study data to estimate the cumulative incidence using ${\widehat{F}}_1^{(1)(k)}(t;{\widehat{a}}^{(k)},{\widehat{b}}^{(k)})$ under an assumption of nondifferential misclassification (or ${\widehat{F}}_1^{(2)(k)}\{t;{\widehat{a}}^{(k)}(t,z),{\widehat{b}}^{(k)}(t,z)\}$ under an assumption of differential misclassification).

The α/2 and 1 –α/2 percentiles of the distribution of the estimates over the M bootstrap samples at time t are the limits of the 1 − α CI for the estimated cumulative incidence at t for t ∈[0, τ]. This approach for CI calculation will have the correct nominal coverage as it incorporates the uncertainty in the misclassification probabilities.

However, in some settings, resampling from the external validation data may not yield ideal results. For example, in small validation samples, individual bootstrap samples from the validation sample may contain zero events of type C = 1 or C = 2, resulting in errors in estimating a or b, respectively. Moreover, in some situations, published summary measures $\widehat{a}$ and $\widehat{b}$ are available for the external validation data, but the validation data themselves are not available. In these settings, one may replace the nonparametric bootstrap procedure of step 2 with a parametric approach by drawing ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ for each bootstrap from the asymptotic distribution of $\widehat{a}$ and $\widehat{b}$. For example, allowing for differential misclassification where w = (t, z)^T, the simulated ${\widehat{a}}^{(k)}(t,z)$ and ${\widehat{b}}^{(k)}(t,z)$ for each individual in bootstrap sample k are

$${\widehat{a}}^{(k)}(t,z)=\frac{\exp \left({\tilde{\beta }}_a^{T}w\right)}{1+\exp \left({\tilde{\beta }}_a^{T}w\right)}$$

and

$${\widehat{b}}^{(k)}(t,z)=\frac{\exp \left({\tilde{\beta }}_b^{T}w\right)}{1+\exp \left({\tilde{\beta }}_b^{T}w\right)},$$

where ${\tilde{\beta }}_a$ and ${\tilde{\beta }}_b$ are random draws from $N({\widehat{\beta }}_a,\widehat{V}({\widehat{\beta }}_a))$ and $N({\widehat{\beta }}_b,\widehat{V}({\widehat{\beta }}_b))$, respectively.

2.3 |. Transportability of the misclassification probabilities

The validity of the proposed approach rests of the assumption that a and b or a(t, z) and b(t, z) are the same in the study population as in the external validation study population. However, unlike other approaches to account for outcome misclassification,^15–18 the proposed approach does not require equivalence of π₁₂(t, z) = P(C = 1 C^⋆ = 2, T = t, Z = z) and π₂₁(t, z) = P(C = 2|C^⋆ = 1, T = t, Z = z) between the study sample and validation data. Note that these probabilities are the positive and negative predictive values if the outcome is binary and occurs at a single time point. Like the predictive values, which depend on both the misclassification probabilities a and b and the prevalence of disease in the study sample, π₁₂(t, z) and π₂₁(t, z) depend on the cause-specific hazards of both event types in the population in addition to the misclassification probabilities. Specifically,

$${\pi }_{12}(t,z)=\frac{{\lambda }_1(t;z)a(t,z)}{{\lambda }_1(t;z)a(t,z)+{\lambda }_2(t;z)[1-b(t,z)]},$$

and

$${\pi }_{21}(t,z)=\frac{{\lambda }_2(t;z)b(t,z)}{{\lambda }_1(t;z)[1-a(t,z)]+{\lambda }_2(t;z)b(t,z)},$$

where λ₁(t; z) and λ₂(t; z) are the cause specific hazard functions for outcomes 1 and 2, respectively.

Consequently, using π₁₂(t, z) and π₂₁(t, z) from an external validation study would require the assumption that both the event frequency for both event types and the misclassification probabilities are equal between the main and the external validation study. This assumption is much stronger than our assumption of transportability of only the misclassification probabilities between the two populations. Like sensitivity and specificity in the diagnostic testing context, the classification probabilities a(t, z) and b(t, z) are intrinsic properties of the event classification method, and therefore more likely to be transportable than the predictive values π₁₂(t, z) and π₂₁(t, z).

3 |. SIMULATION STUDIES

3.1 |. Simulation set up

We assess the finite sample performance of the proposed approach to account for outcome misclassification in cumulative incidence functions using a Monte Carlo simulation experiment. In each of 2000 simulated data sets, we let i = 1, …, 2000 index simulated individuals. We examined the performance of the proposed approach under scenarios in which the amount of misclassification, the size of the external validation set, and the proportion experiencing the competing event varied. We compared the proposed estimators under settings with no misclassification, nondifferential misclassification, and misclassification that was differential with respect to a predictor of the outcome.

In each simulated data set, half of simulated participants were randomly assigned covariate Z_i = 1 and half were assigned to Z_i = 0. Among simulated participants with Z_i = 1, 30% of the simulated subjects were randomly assigned to cause C_i = 1 and 70% were assigned to cause C_i = 2, while among those with Z_i = 0, 60% were assigned to cause C_i = 1 and 40% were assigned to cause C_i = 2, so that the overall P(C_i = 1) = 45%. Among those assigned to cause C_i = 1, the time to failure T_i followed a Weibull distribution with the shape parameter α set to 1.25 and the scale parameter λ₁ determined by Z_i such that

$$f_1(t|z)={\alpha }^{-1}{\lambda }_{1z}^{-1}{t}^{\alpha -1}{e}^{-{\left(t/{\lambda }_{1z}\right)}^{\alpha }},$$

where λ_1z = 2+log(10)−log(10)z. The time to failure for those assigned to causeC_i = 2 followed an exponential distribution with λ₂ determined by Z_i such that

$$f_2(t|z)={\lambda }_{2z}^{-1}{e}^{-\left(t/{\lambda }_{2z}\right)},$$

where λ_2z = 4 + log(5) − log(5)z. The time to censoring U_i was a uniform random number between 0 and 5. The observed failure time was the minimum of the time to failure from either cause and the time to censoring, X_i = min(T_i, U_i). If X_i = U_i, then C_i = 0. The observed cause of failure, $C_i^{\star }$, was determined by values a(t, z) and b(t, z). Note that, for simplicity, we allowed the misclassification of cause C_i to vary as a function of z, but not of t, such that a(t, z) = a(z) and b(t, z) = b(z). For simulated subjects with C_i = 1, $C_i^{\star }$ was drawn from a Bernoulli distribution with probability 1(z = 1){1 − a(1)} + 1(z = 0){1 − a(0)}. For subjects with C_i = 2, $P\left(C_i^{\star }=1\right)=1(z=1)b(1)+1(z=0)b(0)$; otherwise, $C_i^{\star }=2$. If C_i = 0, then $C_i^{\star }=0$.

3.1.1 |. Simulations under nondifferential misclassification

First, we compared the performance of the proposed estimator ${\widehat{F}}_1^{(1)}(t)$ under a range of settings with nondifferential misclassification. In these scenarios, we set a(1) = a(0) = a and b(1) = b(0) = b. We compared estimators under four sets of values for a and b, ie, (0, 0), (0.1, 0.1), (0.3, 0.1), and (0.1, 0.3).

For each simulated data set, we simulated a corresponding external validation data set with the number of events set to n₀, where n₀ took on a value between 25 and 1000 for the various simulated scenarios. In the first set of simulations, the external validation data set had the same data generating mechanism as the main data set. Specifically, in the first set of simulations, the proportion assigned to C_i = 1 was 0.45 in both the main study data and the validation data, or p_main = p_val = 0.45. To explore whether the proposed approach was sensitive to differences in the incidence of the competing event between the main study and the validation data, the second set of simulations assigned p_val = 0.75 while maintaining p_main = 0.45. In all simulation experiments, values of a and b were the same in the main study and in the validation data. In the simulated external validation data, we assumed that both C_i and $C_i^{\star }$ were observed.

The true cumulative incidence at time t of event type 1 was F₁(t) = ∑_zP(Z = z) × p_main,z × (1 − exp{(−t∕λ_1z)^1.25}) in all scenarios. For each of the simulated scenarios, we compared the bias, standard deviation of the bias, mean squared error, and 95% CI coverage between the following two estimators:

1. The standard Aalen-Johansen estimator ${\widehat{F}}_1^{\star }(t)$ based on the misclassified cause of failure $C_i^{*}$;

2. The proposed estimator ${\widehat{F}}_1^{(1)}(t)$, which accounted for the misclassification in the cumulative incidence function using Equation (1), with information on a and b from the external validation data.

We report bias, empirical standard error, average estimated standard error, root mean squared error, and 95% CI coverage probability for each approach under each scenario. For ease of interpretation, we have rescaled all values to refer to 100 times the cumulative incidence, which can be thought of as the percentage of study participants experiencing a particular outcome type. Bias was defined as 100 times the average of the difference between the estimated cumulative incidence and the true value, the empirical standard error was the standard deviation of the estimated cumulative incidence (multiplied by 100) over all simulated data sets, and the average estimated standard error was the average standard error of the cumulative incidence (multiplied by 100) estimated using the bootstrap approaches across all simulated data sets. Root mean squared error was the square root of the sum of the bias squared and the empirical standard error squared, and the 95% CI coverage was the proportion of 95% CIs constructed in the simulated data sets that contained the true value.

Confidence intervals for ${\widehat{F}}_1^{\star }(t)$ were the α∕2 and 1 − α∕2 percentiles of ${\widehat{F}}_1^{\star }(t)$ across 1000 bootstrap samples of the main study data. We compared coverage under two strategies, ie, the first approach constructed 95% CIs as the 2.5th and 97.5th percentiles of ${\widehat{F}}_1^{(1)}(t)$ across 1000 bootstrap samples in which the main study data only was resampled (the single bootstrap approach). In the second approach, 95% CIs around ${\widehat{F}}_1^{(1)}(t)$ were constructed as the 2.5th and 97.5th of ${\widehat{F}}_1^{(1)}(t)$ in 1000 bootstrap samples in which both the main study data were resampled and ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ were redrawn from their asymptotic distributions in each bootstrap. Results were evaluated at times t = (1, 2, 3).

3.1.2 |. Simulations under differential misclassification

We next compared the performance of the standard estimator ${\widehat{F}}_1^{\star }(t)$, the proposed estimator ${\widehat{F}}_1^{(1)}(t)$, which assumed nondifferential misclassification, and the second proposed estimator ${\widehat{F}}_1^{(2)}(t)$, which allowed for differential misclassification, under settings with no misclassification, nondifferential misclassification, and misclassification that was differential with respect to Z.

The second set of simulations used the same data generating mechanism as the simulations in Section 3.1.1 above, with the exception of how $C_i^{\star }$ was generated. In this set of simulations, we examined scenarios with three types of misclassification.

1. No misclassification: a(1) = a(0) = b(1) = b(0) = 0;

2. Nondifferential misclassification: a(1) = a(0) = 0.1; b(1) = b(0) = 0.3;

3. Differential misclassification: a(1) = 0; a(0) = 0.3; b(1) = 0; b(0) = 0.2.

In each scenario, the main study data was composed of 2000 individuals and the validation data included 300 simulated events. Results were evaluated at times t = 1.5 (true F₁(1.5) = 15%) and t = 3 (true F₁(3) = 26%). In all scenarios in this set of simulations, ${\widehat{a}}^k(t,z)$ and ${\widehat{b}}^k(t,z)$ were resampled from their asymptotic distribution in each bootstrap sample.

3.1.3 |. Simulations with error in $\widehat{\mathit{a}}$ and $\widehat{\mathit{b}}$

In some settings, outcomes measured in external validation data may not share the misclassification properties of outcomes in the main study data. In a final set of simulation experiments, we explored the setting with nondifferential misclassification in which the true value of a was 0.1 and the true value of b was 0.3. As in the aforementioned scenarios, the true prevalence of outcome type C = 1 in the main study data was 0.45. In each of 2000 simulated worlds, we simulated a series of 25 external validation data sets of size n₀ = 500 in which $\widehat{a}$ and $\widehat{b}$ took on values corresponding to a±0.1 and b±0.1, respectively. In each scenario, we estimated the cumulative incidence of event type C = 1 using the standard approach (ie, ${\widehat{F}}_1(t)$) and using the proposed estimator ${\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})$ in conjunction with $\widehat{a}$ and $\widehat{b}$ estimated from each external validation data set. We defined bias as 100 times the average difference between the true cumulative incidence at t = 3 and the estimated cumulative incidence over the 2000 iterations of the experiment.

3.2 |. Simulation results

Table 1 compares the performance of the standard estimator ${\widehat{F}}_1^{\star }(t)$ and the proposed ${\widehat{F}}_1^{(1)}(t)$ in settings with nondifferential misclassification and a validation data set composed of n₀ = 300 events. In scenarios with no misclassification (a = b = 0), both estimators had little bias and appropriate 95% CI coverage. Bias in the standard estimator ${\widehat{F}}_1^{\star }(t)$ increased as the amount of misclassification in C^⋆ increased. Bias was smaller for ${\widehat{F}}_1^{(1)}(t)$ than for ${\widehat{F}}_1^{\star }(t)$ in all scenarios examined where a ≠ 0 or b ≠ 0. The reduction in bias was slightly offset by an increase in the standard deviation of the bias for ${\widehat{F}}_1^{(1)}(t)$, but root mean squared error was smaller for ${\widehat{F}}_1^{(1)}(t)$ than for ${\widehat{F}}_1^{\star }(t)$ for many scenarios examined. Notably, ${\widehat{F}}_1^{(1)}(t)$ had appropriate CI coverage in all scenarios when standard errors were estimated by bootstrapping both the main study data and the validation data. In contrast, 95% CI coverage was too low for ${\widehat{F}}_1^{\star }(t)$ (likely due to bias in the point estimates), and for ${\widehat{F}}_1^{(1)}(t)$ when only the main study was resampled as part of the bootstrap (due to the failure to account for uncertainty in $\widehat{a}$ and $\widehat{b}$).

TABLE 1.

Bias^a, empirical standard error (ESE)^b, average estimated standard error (ASE)^c, root mean squared error (RMSE)^d, and 95% CI coverage of cumulative incidence estimates in a standard analysis ${\widehat{F}}_1^{\star }(t)$ and an analysis accounting for misclassification ${\widehat{F}}_1^{(1)}(t)$ from 2000 simulated cohorts of 2000 individuals evaluated at three time points t = (1, 2, 3)

			${\widehat{\mathit{F}}}_{\mathbf{1}}^{\star }(\mathit{t})$					${\widehat{\mathit{F}}}_{\mathbf{1}}^{(\mathbf{1})}(\mathit{t})$
pvale	a	b	Bias	ESE	ASE	RMSE	Coverage	Bias	ESE	ASE	RMSE	Coveragef	Coverageg
t = 1
0.45	0	0	−0.02	0.76	0.75	0.76	0.94	−0.02	0.76	0.75	0.76	0.94	0.94
0.45	0.1	0.1	0.08	0.76	0.75	0.77	0.94	−0.04	1.01	1.02	1.01	0.90	0.95
0.45	0.1	0.3	2.29	0.84	0.82	2.45	0.20	−0.06	1.36	1.37	1.36	0.88	0.94
0.45	0.3	0.1	−1.86	0.66	0.68	1.98	0.22	−0.08	1.34	1.41	1.34	0.89	0.95
0.75	0	0	−0.02	0.76	0.75	0.76	0.94	−0.02	0.76	0.75	0.76	0.94	0.94
0.75	0.1	0.1	0.08	0.76	0.75	0.77	0.94	−0.08	0.99	1.01	0.99	0.92	0.94
0.75	0.1	0.3	2.29	0.84	0.82	2.45	0.20	−0.08	1.40	1.40	1.40	0.86	0.95
0.75	0.3	0.1	−1.86	0.66	0.68	1.98	0.22	−0.09	1.24	1.33	1.24	0.92	0.96
t = 2
0.45	0	0	−0.01	0.98	1.00	0.98	0.94	−0.01	0.98	1.00	0.98	0.94	0.94
0.45	0.1	0.1	0.06	0.95	1.00	0.95	0.96	0.06	1.51	1.54	1.51	0.86	0.95
0.45	0.1	0.3	3.99	1.02	1.07	4.12	0.02	−0.03	2.16	2.15	2.16	0.83	0.94
0.45	0.3	0.1	−3.75	0.83	0.91	3.84	0.02	0.15	2.14	2.29	2.15	0.82	0.95
0.75	0	0	−0.01	0.98	1.00	0.98	0.94	−0.01	0.98	1.00	0.98	0.94	0.94
0.75	0.1	0.1	0.06	0.95	1.00	0.95	0.96	0.00	1.43	1.51	1.43	0.89	0.94
0.75	0.1	0.3	3.99	1.02	1.07	4.12	0.02	−0.05	2.16	2.23	2.16	0.79	0.94
0.75	0.3	0.1	−3.75	0.83	0.91	3.84	0.02	0.12	1.92	2.12	1.92	0.86	0.96
t = 3
0.45	0	0	0.02	1.15	1.11	1.15	0.93	0.02	1.15	1.11	1.15	0.93	0.93
0.45	0.1	0.1	0.04	1.12	1.11	1.12	0.95	0.02	1.81	1.87	1.81	0.83	0.95
0.45	0.1	0.3	5.36	1.16	1.18	5.48	0.00	−0.10	2.59	2.69	2.59	0.79	0.95
0.45	0.3	0.1	−5.25	0.98	1.03	5.34	0.00	0.17	2.71	2.92	2.71	0.75	0.97
0.75	0	0	0.02	1.15	1.11	1.15	0.93	0.02	1.15	1.11	1.15	0.93	0.93
0.75	0.1	0.1	0.04	1.12	1.11	1.12	0.95	0.02	1.79	1.86	1.79	0.85	0.96
0.75	0.1	0.3	5.36	1.16	1.18	5.48	0.00	−0.13	2.75	2.87	2.75	0.76	0.95
0.75	0.3	0.1	−5.25	0.98	1.03	5.34	0.00	0.12	2.49	2.68	2.49	0.80	0.95

Abbreviations: ASE, average estimated standard error; ESE, empirical standard error; RMSE: root mean squared error.

^aBias was 100 times the average difference between the true and estimated values of the cumulative incidence.

^bESE was the standard deviation of the estimated cumulative incidence across all simulations.

^cASE was the standard error estimated using the proposed bootstrap approach averaged across all simulated data sets.

^dRMSE was the sum of the bias squared and the ESE squared.

^eP(C = 1|C > 0) in the validation data. In the main study data, P(C = 1|C > 0) = 0.45.

^fCoverage based on 1000 bootstrap resamples of main study data only (treating $\widehat{a}$ and $\widehat{b}$ as fixed).

^gCoverage based on 1000 bootstrap samples in which both the main study data were resampled and ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ were redrawn from their asymptotic distributions.

Figure 1 illustrates the relationship between the size of the external validation data set n₀ and estimated standard error and 95% CI coverage under each approach. Using the standard estimator ${\widehat{F}}_1^{\star }(t)$, the standard error was small and independent of the size of the external validation data, as expected. In addition, due to the bias in the estimated cumulative incidence, 95% CI coverage was low for all scenarios examined. The standard error was also independent of the size of the validation sample when we applied ${\widehat{F}}_1^{(1)}(t)$, but ignored the uncertainty in estimation of a and b by bootstrapping only the main study data to obtain standard errors. When this estimator was applied, 95% CI coverage was much too low at low validation data set sizes, and rose as the size of the validation data grew. Finally, when we applied ${\widehat{F}}_1^{(1)}(t)$ accounting for the uncertainty in a and b using the approach that both bootstrapped the main study data and resampled a and b for each bootstrap sample, standard error was dependent on the size of the external validation data, and 95% coverage remained near its nominal value for all sizes of the validation data examined. Specifically, for external validation data sets with n₀ > 100, bias was low (approximately 0.05%) and 95% CI coverage was near 95%. At smaller validation data set sizes (eg, n₀ = 25), bias was slightly elevated (approximately 0.50%), though remained smaller than bias under the standard approach (3.30%). Moreover, at smaller validation study sizes, 95% CI coverage was somewhat elevated (approximately 98%).

FIGURE 1.

(A) Standard error and (B) 95% confidence interval coverage in simulations with nondifferential misclassification in which a = 0.1 and b = 0.3 under various sizes of the external validation data set. Solid black curves represent values using the standard approach that ignores misclassification, dotted curves represent values obtained by plugging in estimated values of $\widehat{a}$ and $\widehat{b}$ into Equation (1) and ignoring uncertainty in the estimation of $\widehat{a}$ and $\widehat{b}$, and the dashed curves represent values obtained using the proposed approach, including bootstrapping of the validation and main study data. The plot of (B) 95% confidence interval coverage also contains a grey reference line at 95%

Table 2 compares the performance of ${\widehat{F}}_1^{\star }(t)$, ${\widehat{F}}_1^{(1)}(t)$, and ${\widehat{F}}_1^{(2)}(t)$ in scenarios with no misclassification, nondifferential misclassification, and differential misclassification. When no misclassification was present, the three methods yielded nearly identical results with little bias and appropriate 95% CI coverage. When misclassification was nondifferential with respect to Z, the standard estimator ${\widehat{F}}_1^{\star }(t)$ was biased and had poor coverage, while ${\widehat{F}}_1^{(1)}(t)$ and ${\widehat{F}}_1^{(2)}(t)$ produced results will little bias and appropriate coverage. When misclassification was differential with respect to Z, ${\widehat{F}}_1^{(2)}(t)$ produced results with little bias and appropriate coverage, while ${\widehat{F}}_1^{\star }(t)$ produced biased results with very poor coverage. Notably, using the estimator that assumed nondifferential misclassification ${\widehat{F}}_1^{(1)}(t)$ in the setting where differential misclassification was present produced results with some bias and coverage probability that was too low. However, the reduction in bias seen when implementing ${\widehat{F}}_1^{(2)}(t)$ in settings with differential misclassification came at the cost of decreased precision.

TABLE 2.

Performance of estimators of the cumulative incidence at t = 1.5 and t = 3 years in 2000 simulated cohorts of 2000 individuals with external validation data of size n₀ = 300 under scenarios with no misclassification, nondifferential misclassification, and differential misclassification

	t=1.5					t=3
Approach	Bias	ESE	ASE	RMSE	Coverage	Bias	ESE	ASE	RMSE	Coverage
a1 = a2 = b1 = b2 = 0
${\widehat{F}}_1^{\star }(t)$	−0.01	0.92	0.90	0.90	0.94	−0.03	1.16	1.10	1.10	0.94
${\widehat{F}}_1^{(1)}(t)$	−0.01	0.92	0.90	0.90	0.94	−0.03	1.16	1.10	1.10	0.94
${\widehat{F}}_1^{(2)}(t)$	0.00	0.32	0.90	0.90	0.94	−0.01	1.16	1.10	1.10	0.94
a1 = a1 = 0.1; b1 = b2 = 0.3
${\widehat{F}}_1^{\star }(t)$	3.22	0.99	0.97	3.37	0.08	5.35	1.20	1.18	5.50	0.00
${\widehat{F}}_1^{(1)}(t)$	−0.03	1.69	1.82	1.69	0.97	−0.08	2.23	2.28	2.23	0.96
${\widehat{F}}_1^{(2)}(t)$	−0.02	1.71	1.86	1.71	0.97	−0.05	2.55	2.88	2.55	0.97
a1 = 0; a2 = 0.3; b1 = 0; b2 = 0.2
${\widehat{F}}_1^{\star }(t)$	−1.66	0.87	0.85	1.87	0.50	−3.45	1.10	1.05	3.62	0.11
${\widehat{F}}_1^{(1)}(t)$	−0.12	1.21	1.23	1.21	0.95	−0.71	1.69	1.76	1.84	0.92
${\widehat{F}}_1^{(2)}(t)$	0.10	1.47	1.55	1.47	0.96	0.19	2.32	2.52	2.33	0.97

Abbreviations: ASE, average estimated standard error; ESE, empirical standard error; RMSE, root mean squared error.

^aBias was 100 times the average difference between the true and estimated values of the cumulative incidence.

^bESE was the standard deviation of the estimated cumulative incidence across all simulations.

^cASE was the standard error estimated using the proposed bootstrap approach averaged across all simulated data sets.

^dRMSE was the sum of the bias squared and the ESE squared.

Table 3 illustrates the sensitivity of the proposed approach to violations in the assumption of transportability of a and b between the main study and the validation data. Each row displays the bias, estimated standard error, average standard error, root mean squared error, and 95% CI coverage of the standard approach (employing ${\widehat{F}}_1^{\star }(t)$) and the proposed approach (employing ${\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})$) applied in tandem with external validation data sets in which a₀ ranges from 0 to 0.2 and b₀ ranges from 0.2 to 0.4. Under most scenarios in which the misclassification probabilities in the external data, a₀ and b₀, were somewhat close to the true value of a and b in the main study data, bias and root mean squared error are reduced using the proposed approach. Moreover, 95% CI coverage is closer to the nominal value under the proposed approach (compared to the standard approach) under all scenarios examined.

TABLE 3.

Performance of estimators of the cumulative incidence at t = 3 years in 2000 simulated cohorts of 2000 individuals with external validation data of size n₀ = 500 when a = 0.1 and b = 0.3 in the main study under scenarios where the misclassification probabilities in the validation sample, a₀ and b₀, differ from a and b in the main study

Estimates		${\widehat{\mathit{F}}}_{\mathbf{1}}^{\star }(\mathit{t})$					$\widehat{\mathit{F}}{(\mathbf{1})}_{\mathbf{1}}(\mathit{t})$
a0	b0	Bias	ESE	ASE	RMSE	Coverage	Bias	ESE	ASE	RMSE	Coverage
0.00	0.20	5.35	1.20	1.18	5.48	0.00	−0.01	1.49	1.49	1.49	0.96
0.00	0.25	5.35	1.20	1.18	5.48	0.00	−1.82	1.62	1.62	2.44	0.80
0.00	0.30	5.35	1.20	1.18	5.48	0.00	−3.87	1.78	1.78	4.26	0.40
0.00	0.35	5.35	1.20	1.18	5.48	0.00	−6.23	1.96	1.98	6.53	0.10
0.00	0.40	5.35	1.20	1.18	5.48	0.00	−8.95	2.22	2.22	9.22	0.01
0.05	0.20	5.35	1.20	1.18	5.48	0.00	1.74	1.67	1.72	2.41	0.84
0.05	0.25	5.35	1.20	1.18	5.48	0.00	−0.06	1.81	1.85	1.81	0.95
0.05	0.30	5.35	1.20	1.18	5.48	0.00	−2.14	1.96	2.01	2.91	0.83
0.05	0.35	5.35	1.20	1.18	5.48	0.00	−4.55	2.16	2.21	5.03	0.45
0.05	0.40	5.35	1.20	1.18	5.48	0.00	−7.37	2.43	2.47	7.76	0.13
0.10	0.20	5.35	1.20	1.18	5.48	0.00	3.79	1.92	1.97	4.25	0.51
0.10	0.25	5.35	1.20	1.18	5.48	0.00	2.00	2.07	2.10	2.88	0.85
0.10	0.30	5.35	1.20	1.18	5.48	0.00	−0.08	2.23	2.28	2.23	0.96
0.10	0.35	5.35	1.20	1.18	5.48	0.00	−2.52	2.44	2.49	3.50	0.83
0.10	0.40	5.35	1.20	1.18	5.48	0.00	−5.43	2.73	2.77	6.08	0.49
0.15	0.20	5.35	1.20	1.18	5.48	0.00	6.16	2.23	2.28	6.55	0.20
0.15	0.25	5.35	1.20	1.18	5.48	0.00	4.42	2.39	2.43	5.03	0.56
0.15	0.30	5.35	1.20	1.18	5.48	0.00	2.37	2.56	2.63	3.49	0.87
0.15	0.35	5.35	1.20	1.18	5.48	0.00	−0.07	2.78	2.87	2.78	0.96
0.15	0.40	5.35	1.20	1.18	5.48	0.00	−3.04	3.09	3.18	4.34	0.85
0.20	0.20	5.35	1.20	1.18	5.48	0.00	8.88	2.60	2.67	9.25	0.05
0.20	0.25	5.35	1.20	1.18	5.48	0.00	7.24	2.78	2.86	7.76	0.23
0.20	0.30	5.35	1.20	1.18	5.48	0.00	5.28	2.98	3.09	6.06	0.62
0.20	0.35	5.35	1.20	1.18	5.48	0.00	2.89	3.24	3.38	4.34	0.90
0.20	0.40	5.35	1.20	1.18	5.48	0.00	−0.08	3.60	3.75	3.60	0.97

Abbreviations: ASE, average estimated standard error; ESE, empirical standard error; RMSE, root mean squared error.

^aBias was 100 times the average difference between the true and estimated values of the cumulative incidence.

^bESE was the standard deviation of the estimated cumulative incidence across all simulations.

^cASE was the standard error estimated using the proposed bootstrap approach averaged across all simulated data sets.

^dRMSE was the sum of the bias squared and the ESE squared.

4 |. DATA ANALYSIS

We applied the proposed approach to estimate the 1-year cumulative incidence of death while in care at a clinic supported by the FACES program in western Kenya. Dates of death were collected among people in care in FACES by clinic staff. Patients who did not return for follow-up visits and were not known to have died were classified as lost to clinical care. However, vital status among these patients who did not return for scheduled follow-up visits was uncertain; some of these patients were truly lost to clinical care while others likely died. Here, we apply the proposed approach to estimate the cumulative incidence of mortality in this cohort accounting for misclassification of some deaths as loss to care.

Let the 3886 patients enrolled in FACES between November 2001 and July 2011 be indexed from i = 1 to 3886. A patient was considered to be “lost to care” after 90 days without a clinic visit. The time from enrollment to the first of death or loss to care will be represented by T_i and the time from enrollment to censoring at 1 year or database close on July 27, 2011 will be represented by U_i. X_i = min(T_i, U_i) is the observed time each participant spends in the cohort. $C_i^{\star }$ is the observed event type. $C_i^{\star }=1$ if the participant has an observed date of death prior to loss to clinic or censoring, $C_i^{\star }=2$ if the participant is lost to clinic prior to censoring, and $C_i^{\star }=0$ if the participant is censored. C_i, the true event type at time X_i, was not observed in this cohort.

However, an Academic Model for the Prevention and Treatment of HIV/AIDS (AMPATH) program in the same geographic area conducted a tracing study during the same time period in which participants who did not appear at the clinic for a scheduled follow-up visit were sampled for an outreach program to determine vital status. While it is conceivable that some patients included in the FACES data set also sought care within the AMPATH program, the distance between clinics served by each program makes such overlap unlikely. The AMPATH conducted outreach visits among 4238 patients who had not returned to the clinic within 2 months of a scheduled visit, and were therefore classified as lost to care. Of these patients originally classified as lost to care, 1143 had died within 2 months of their missed visit. Data from this tracing study of 4238 patients were then used to impute the outcome status for the other 23 861 patients who were lost to care but were not traced conditional on age, sex, and time of loss to care, under the assumption that patients successfully traced were a stratified random subset of all patients lost. Note that all observed deaths were assumed to be true deaths. This implies that b = 0 by assumption in this example. After applying the approach describe above, we estimated that 77% of deaths were classified as loss to follow-up, implying that $\widehat{a}=0.77$. Data from the AMPATH program were used as validation data to account for misclassification of loss to care in FACES.

We compare the 1-year cumulative incidence of mortality estimated using the standard estimator ${\widehat{F}}_1^{\star }(t)$ and the proposed estimator for nondifferential misclassification ${\widehat{F}}_1^{(1)}(t)$. Specifically, the steps to estimate ${\widehat{F}}_1^{(1)}(t)$ were as follows.

1. Estimate $\widehat{a}$ in the validation data. As mentioned above, estimating $\widehat{a}$ first required imputing outcomes for patients lost to care and not traced, and then calculating $\widehat{a}$ in the imputed data set. To ease computational burden in this example, we used parametric fractional imputation,¹⁹ rather than multiple imputation, to impute outcomes for those not traced in the validation sample. $\widehat{b}$ was set to 0 by assumption in this example, and $\widehat{a}$ was estimated to be 0.77.

2. Plug $\widehat{a}$ and $\widehat{b}$ into Equation (1) to obtain point estimate ${\widehat{F}}_1^{(1)}(t)$.

3. Draw 500 bootstrap samples from the validation data and the main study data, respectively.

4. Estimate ${\widehat{a}}^{(k)}$ in each bootstrap sample from the validation data. Due to the nature of the validation data, estimating ${\widehat{a}}^{(k)}$ required performing the imputation of outcomes for patients lost but not traced in each bootstrap sample of the validation data to allow uncertainty in the imputation process to propagate through the analysis to the final CI. By assumption, $b^{(k)}$ was set to 0 for all k.

5. Plug ${\widehat{a}}^{(k)}$ and ${\widehat{b}}^{(k)}$ into Equation (1) to estimate ${\widehat{F}}_1^{(1)(k)}(t)$.

6. The α∕2 and 1 − α∕2 percentiles of the distribution of the estimates over the M bootstrap samples at time t year are the limits of the 1 − α CI for the estimated cumulative incidence of mortality.

Of the 3886 patients who enrolled in the FACES program during the study time period, 34% were male and the mean age at enrollment was 34 years. Of these patients, 67 died in the year after enrollment and 874 appeared to become lost to care. Using the standard approach, the estimated 1-year cumulative incidence of mortality was 1.76% (95% CI: 1.33, 2.19) and the estimated 1 year cumulative incidence of loss to care was 23.54% (95% CI: 22.17, 24.91). After applying the proposed approach to account for misclassification of some deaths as loss to care, the estimated 1-year cumulative incidence of mortality was 6.59% (95% CI: 4.92, 8.27) and the estimated 1-year cumulative incidence of loss to care was 18.71% (95% CI: 16.77, 20.65). As expected, standard errors estimated using the proposed approach (including bootstrapping the main and validation data) for cumulative incidence of mortality (standard error $\left({\widehat{\sigma }}_1\right)=0.85$) and loss to care $\left({\widehat{\sigma }}_2=0.99\right)$ were larger than standard errors resulting from an approach that accounted for the outcome misclassification without rebootstrapping the validation data (${\widehat{\sigma }}_{naive}=0.77$ and 0.89, respectively).

Figure 2 displays the estimated cumulative incidence of mortality and loss to care over time using the standard approach ${\widehat{F}}_1^{\star }(t)$ and the proposed approach ${\widehat{F}}_1^{(1)}(t)$ and pointwise 95% confidence limits calculated at each event time. The proposed approach yielded estimates of mortality that were higher than the standard approach at all time points under study and of loss to care that were lower than the standard approach at all time points.

FIGURE 2.

Estimated cumulative incidence of mortality and loss to care and pointwise 95% confidence intervals over 1 year after enrollment in the FACES program in western Kenya between 2001 and 2011 under the standard approach to estimate cumulative incidence ${\widehat{F}}_1^{\star }(t)$ (dashed lines) and the proposed approach ${\widehat{F}}_1^{(1)}(t)$ (solid lines)

5 |. DISCUSSION

Misclassification of event type is common in a variety of settings and can lead to biased estimates and invalid conclusions. Here, we have described a nonparametric approach to account for this misclassification when estimating cumulative incidence functions in the presence of competing events. This work extends previously proposed approaches⁷ in two ways. First, the approach proposed here allows consistent estimation of cumulative incidence functions in the presence of misclassification that is differential with respect to time and/or a set of covariates; second, we propose bootstrapping both the validation data and the main study data to incorporate uncertainty in the misclassification probabilities into the final 95% CI around the estimated cumulative incidence. The proposed estimators where shown to be uniformly consistent. Finally, in the example presented here, applying the proposed approach to estimate mortality in an HIV treatment program resulted in mortality estimates that were nearly four times the mortality that would be estimated assuming perfect outcome classification.

In simulations, we illustrated that the proposed approach works well in settings with both false negatives (a > 0) and false positives (b > 0). However, in the example, misclassification was unidirectional (ie, no patients lost to follow-up were erroneously classified as dead), meaning that b was set to 0. Other applications of this approach could leverage its ability to account for bidirectional misclassification. For example, this approach would be valuable to estimate the cumulative incidence of cause-specific mortality in the presence of misclassification of cause of death.

We presented two options for allowing uncertainty in the estimation of the misclassification probability estimates into the final CIs. In the first, the investigator would resample from both the main study data and the validation data in each bootstrap iteration. However, when the validation data set is small, this approach could fail due to small or zero cell sizes in a given sample of the validation data. To mitigate this challenge, we proposed a second option that involved resampling the misclassification probabilities from their asymptotic distributions in each bootstrap sample. This approach is appealing because it allows inference in a wide variety of settings (including settings where the validation data themselves are unavailable), but relies on correct specification of a model for the misclassification probabilities.

A key assumption of the proposed methodology is that the misclassification probabilities are the same between the main study and the external validation study populations. This assumption is reasonable in many settings as these probabilities are properties of the event ascertainment methods and typically do not depend on the underlying event frequency of the events for binary outcomes. Therefore, the assumptions required to implement the proposed approach are weaker than the assumptions required for methods that rely on transportability of the predictive values π₁₂(t, z) and π₂₁(t, z), which depend on the event frequency of both the event types, from external studies to account for misclassification.²⁰ However, results may be sensitive to differences in the misclassification probabilities a and b between the validation study and the main study data, as we illustrate in Table 3. In settings where the transportability of a and b is uncertain, sensitivity analysis in which the investigator varies these probabilities around the values estimated in the validation study (as in the work of Fox et al²¹) will provide insight into the possible bias due to the violation of this assumption.

To allow for differential misclassification, we also assumed correct specification of the logistic regression models for the misclassification probabilities. As an alternative, considering nonparametric regression functions of the covariates under study in the framework of generalized additive models or machine learning approaches²² could relax these assumptions. When the size of the validation data set is small, parametric models may be preferable to more flexible approaches. Small validation sample sizes may limit the number of factors by which misclassification probabilities may be allowed to vary. If logistic regression is used to estimate the misclassification probabilities in the validation data, one could use the rule of thumb that from five to ten events are needed per predictor included in the model.^23,24

In settings where the size of the validation data set is very small, estimation of a and b may be difficult, and, as a result, the asymptotic distribution of $\widehat{a}$ and $\widehat{b}$ may be unreliable or unavailable. In these settings, one may stabilize inference by applying a penalty to each of the parameters in the models used to estimate for $\widehat{a}$ and $\widehat{b}$ (eg, as described in the works of Cole et al²⁵ and Greenland and Mansournia²⁶) or pursue Bayesian approaches to stabilization.²⁷

Finally, we did not consider the potential for mismeasurement of the event time in addition to, or instead of, misclassification of event type. Such mismeasurement could result in censoring participants who in fact experienced one of the events of interest during the study. This is an important topic that has been addressed in a limited number of settings (eg, see the works of Meier et al²⁸ and Korn et al²⁹) and is an interesting area of future research.

APPENDIX

PROOF OF CONSISTENCY

In this appendix, we provide the proofs of uniform consistency for both estimators. Define ‖f(t)‖_∞ = sup_t∈[0,τ]|f(t)|. We first consider the case of nondifferential misclassification, ie, estimator ${\widehat{F}}_1^{(1)}(\cdot )$. It follows that

$${\Vert {\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-F_1(t)\Vert }_{\infty }\le {\Vert {\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-{\widehat{F}}_1^{(1)}(t;a,b)\Vert }_{\infty }+{\Vert {\widehat{F}}_1^{(1)}(t;a,b)-F_1(t)\Vert }_{\infty },$$

where the second term in the right side of the aforementioned inequality converges to 0 in probability.⁷ Denote

$$w_1=\frac{1-b}{1-a-b}$$

and

$$w_2=\frac{b}{1-a-b}.$$

Then, the first term becomes

$$\begin{gathered} {\Vert {\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-{\widehat{F}}_1^{(1)}(t;a,b)\Vert }_{\infty }\le {\Vert \underset{x\le t}{\sup }\left[\left({\widehat{w}}_1-w_1\right){\widehat{F}}_1^{\star }(x)-\left({\widehat{w}}_2-w_2\right){\widehat{F}}_2^{\star }(x)\right]\Vert }_{\infty } \\ \le {\Vert \left({\widehat{w}}_1-w_1\right){\widehat{F}}_1^{\star }(x)-\left({\widehat{w}}_2-w_2\right){\widehat{F}}_2^{\star }(x)\Vert }_{\infty } \\ \le |{\widehat{w}}_1-w_1|+|{\widehat{w}}_2-w_2|, \end{gathered}$$

where ${\widehat{w}}_j$ is equal to w_j, j = 1, 2, with a and b replaced by their consistent estimates $\widehat{a}$ and $\widehat{b}$. By the consistency of $\widehat{a}$ and $\widehat{b}$ and Slutsky’s theorem, it follows that ${\sum }_{j=1}^2|{\widehat{w}}_j-w_j|\stackrel{p}{\to }0$, and thus, ${\Vert {\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-{\widehat{F}}^{(1)}(t;a,b)\Vert }_{\infty }\stackrel{p}{\to }0$. Consequently,

$${\Vert {\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})-F_1(t)\Vert }_{\infty }\stackrel{p}{\to }0,$$

and thus, the proof of uniform consistency of ${\widehat{F}}_1^{(1)}(t;\widehat{a},\widehat{b})$ over [0, τ] is complete.

For the case of differential misclassification, we first need to prove the consistency of ${\widehat{\mathrm{\Lambda }}}_1^S(\cdot ;\widehat{a},\widehat{b})$. As before

$${\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})-{\mathrm{\Lambda }}_1(t)\Vert }_{\infty }\le {\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})-{\widehat{\mathrm{\Lambda }}}^S(t;a,b)\Vert }_{\infty }+{\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;a,b)-{\mathrm{\Lambda }}_1(t)\Vert }_{\infty },$$ (A1)

where the second term in the right side of the aforementioned inequality converges to 0 in probability.¹⁴ Next, it can be easily shown that

$$\left|r_j({\widehat{a}}_i(t),{\widehat{b}}_i(t))-r_j\left(a_i(t),b_i(t)\right)\right|\le \left(|{\widehat{\beta }}_a-{\beta }_a|+|{\widehat{\beta }}_b-{\beta }_b|\right)K_j,j=1,2,$$

due to the Lipschitz continuity of the exponential function. Note that K_j < ∞, j = 1, 2, by the usual regularity condition of boundedness of both the logistic regression parameters and the covariate vector W. Therefore, the first term in the right side of inequality (A1) is

$$\begin{gathered} {\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})-{\widehat{\mathrm{\Lambda }}}_1^S(t;a,b)\Vert }_{\infty }\le \Vert \underset{x\le t}{\sup }\{\underset{0}{\overset{x}{\int }}\frac{{\sum }_{i=1}^n\left[r_1\left({\widehat{a}}_i(s),{\widehat{b}}_i(s)\right)-r_1\left(a_i(s),b_i(s)\right)\right]d{N}_{i1}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)} \\ {-\underset{0}{\overset{x}{\int }}\frac{{\sum }_{i=1}^n\left[r_2\left({\widehat{a}}_i(s),{\widehat{b}}_i(s)\right)-r_2\left(a_i(s),b_i(s)\right)\right]d{N}_{i2}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)}\}\Vert }_{\infty } \\ \le \sum _{j=1}^2{\Vert \underset{0}{\overset{t}{\int }}\frac{\sum _{i=1}^n\left[r_j\left({\widehat{a}}_i(s),{\widehat{b}}_i(s)\right)-r_j\left(a_i(s),b_i(s)\right)\right]d{N}_{ij}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)}\Vert }_{\infty } \\ \le \left(|{\widehat{\beta }}_a-{\beta }_a|+|{\widehat{\beta }}_b-{\beta }_b|\right)\sum _{j=1}^2{K}_j{\Vert \underset{0}{\overset{t}{\int }}\frac{{\sum }_{i=1}^{n}d{N}_{ij}^{\star }(s)}{{\sum }_{i=1}^n{Y}_i(s)}\Vert }_{\infty } \\ \le \left(|{\widehat{\beta }}_a-{\beta }_a|+|{\widehat{\beta }}_b-{\beta }_b|\right)\sum _{j=1}^2{K}_j{\widehat{\mathrm{\Lambda }}}_j^{\star }(\tau ), \end{gathered}$$

where ${\widehat{\mathrm{\Lambda }}}_j^{\star }(\tau )$, j = 1, 2, is the nonparametric Nelson-Aalen estimator of the cumulative cause-specific hazard for the jth cause of failure at the maximum follow-up time τ, based on the misclassified counting processes $N_{ij}^{\star }(t)$, i = 1, …, n. Now, the usual regularity conditions that $E\left[{\sum }_{i=1}^n{Y}_i(\tau )\right]>0$, which means that the expected number of subjects at risk at the maximum follow-up time is positive and the boundedness of the cumulative cause-specific hazard at the maximum follow-up time, along with the consistency of ${\widehat{\beta }}_a$ and ${\widehat{\beta }}_b$ in probability lead to ${\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})-{\widehat{\mathrm{\Lambda }}}_1^S(t;a,b)\Vert }_{\infty }\stackrel{p}{\to }0$, and thus,

$${\Vert {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})-{\mathrm{\Lambda }}_1(t)\Vert }_{\infty }\stackrel{p}{\to }0.$$ (A2)

The same arguments can also be used to show that ${\Vert {\widehat{\mathrm{\Lambda }}}_2^S(t;\widehat{a},\widehat{b})-{\mathrm{\Lambda }}_2(t)\Vert }_{\infty }\stackrel{p}{\to }0$. Now, consider the competing risks model studied in this manuscript as the special case of a nonhomogeneous continuous time Markov process with one initial state and two absorbing (ie, terminal) states 1 and 2. The transition probability matrix of this process is

$$\mathbf{P}(0,t)=\left(\begin{array}{ccc}S(t)& F_1(t)& F_2(t)\\ 0& 1& 0\\ 0& 0& 1\end{array}\right),t\in [0,\tau ].$$

The Aalen-Johansen estimator of this matrix can be expressed as a product integral of the form³⁰

$$\widehat{\mathbf{P}}(0,t)=\prod _{{}_{(0,t]}}[\mathbf{I}+d\widehat{\mathbf{\Lambda }}(v)],$$

where

$$\widehat{\mathbf{\Lambda }}(t)=\left(\begin{array}{ccc}-{\sum }_{j=1}^2{\widehat{\mathrm{\Lambda }}}_j(t)& {\widehat{\mathrm{\Lambda }}}_1(t)& {\widehat{\mathrm{\Lambda }}}_2(t)\\ 0& 0& 0\\ 0& 0& 0\end{array}\right),t\in [0,\tau ],$$

with ${\widehat{\mathrm{\Lambda }}}_j(t)$, j = 1, 2, being the Nelson-Aalen estimator of the cumulative cause-specific hazard when there is no misclassification in the cause of failure. Moreover, it can be shown that the product-integral estimator is equivalent to³⁰

$${\widehat{F}}_j(t)=\underset{0}{\overset{t}{\int }}\widehat{S}(v-)d{\widehat{\mathrm{\Lambda }}}_j(v),t\in [0,\tau ],j=1,2.$$

This means that our estimator under differential misclassification can be expressed as

$$\widehat{\mathbf{P}}(0,t;\widehat{a},\widehat{b})=\prod _{(0,t]}\left[\mathbf{I}+d{\widehat{\mathbf{\Lambda }}}^S(v;\widehat{a},\widehat{b})\right],$$

where

$${\widehat{\mathrm{\Lambda }}}^S(t;\widehat{a},\widehat{b})=\left(\begin{array}{ccc}-{\sum }_{j=1}^2{\widehat{\mathrm{\Lambda }}}_j^S(t;\widehat{a},\widehat{b})& {\widehat{\mathrm{\Lambda }}}_1^S(t;\widehat{a},\widehat{b})& {\widehat{\mathrm{\Lambda }}}_2^S(t;\widehat{a},\widehat{b})\\ 0& 0& 0\\ 0& 0& 0\end{array}\right),t\in [0,\tau ].$$

By the uniform consistency (A2) of the components of ${\widehat{\mathbf{\Lambda }}}^S(t;\widehat{a},\widehat{b})$ and a continuity result from the Duhamel equation,³⁰ it follows that

$$\prod _{(0,t]}\left[\mathbf{I}+d{\widehat{\mathbf{\Lambda }}}^S(v;\widehat{a},\widehat{b})\right]\stackrel{p}{\to }\prod _{(0,t]}[\mathbf{I}+d\mathbf{\Lambda }(v)]=\mathbf{P}(0,t),$$

uniformly over [0, τ], and therefore,

$${\Vert {\widehat{F}}_1^{(2)}(t;\widehat{a},\widehat{b})-F_1(t)\Vert }_{\infty }\stackrel{p}{\to }0.$$