Validation of a Risk‐Prediction Model in the Presence of Outcome Misclassification

ABSTRACT

Electronic health records (EHRs) provide a rich data source for building prediction models to improve the quality of care. However, EHR data are prone to measurement error, and outcomes used to evaluate prediction model performance may be misclassified. Comparing risk predictions to misclassified outcomes will result in unreliable estimates of a prediction model's performance. We propose a method that leverages a smaller chart review sample with gold‐standard outcome measurements to adjust validation for outcome misclassification and provide more accurate prediction model evaluation. We derive formulae to estimate true positive rate (TPR), false positive rate (FPR), positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUC) in the presence of outcome misclassification. Different scenarios of misclassification are explored, including when the misclassification is independent or dependent on features, and when misclassification is unidirectional (e.g., only missed diagnoses) or bidirectional. In simulation studies, we compare the bias and 95% confidence interval coverage of performance estimates obtained using our proposed method to those estimated with misclassified outcomes (without accounting for misclassification) or in the smaller chart review sample. Simulation results indicate that, across all misclassification scenarios examined, our proposed estimates have good accuracy and improved precision. Outcome misclassification should be considered when evaluating a prediction model's performance in order to accurately inform decision‐making about whether and how to use a clinical prediction model.

Abbreviations

AUC

area under the curve

EHR

electronic health record

FPR

false positive rate

NPV

negative predictive value

PPV

positive predictive value

ROC

receiver operating characteristic

TPR

true positive rate

1. Introduction

Electronic health record (EHR) data provide an information‐rich resource for developing, validating, and implementing clinical prediction models [1, 2]. These prediction models can inform clinical decision‐making in a myriad of ways, including delivering personalized patient care [3, 4], detecting disease conditions early [5, 6, 7], and stratifying patients for care pathways based on their predicted risk [8, 9], which can potentially lead to better health outcomes [4, 10] and more efficient health care resource allocation [11]. Our work is motivated by our collaboration in estimating suicide risk‐prediction models used to guide suicide prevention care. Consider a prediction model for the risk of self‐harm or suicide attempt following an outpatient mental health visit. These prediction models can be used to inform providers that a patient is at higher risk for self‐harm. In response, providers then may conduct additional risk assessment or initiate safety planning or other self‐harm risk mitigating strategies [12].

Evaluation of a clinical prediction model's expected performance in the population of interest, known as validation, is needed to inform decision‐making about whether and how to use the model in clinical care. The area under the receiver operating characteristic (ROC) curve (AUC) is a commonly used metric of discrimination that measures how well a prediction model orders observations by risk of the event of interest. In the context of predicting risk of self‐harm, a poor AUC would indicate the prediction model is not effectively distinguishing between visits that are and are not followed by self‐harm events; such a model would not be recommended for clinical use.

In addition to AUC, model validation frequently also examines the accuracy of using specific risk thresholds to classify those at higher risk of the event of interest. Classification accuracy measures such as true positive rate (TPR), false positive rate (FPR), positive predictive value (PPV), and negative predictive value (NPV) may better guide prediction model implementation, as they reflect a model's impact when a threshold of the risk score is used to distinguish between patients at higher and lower risk of, for example, self‐harm. In our motivating example, TPR quantifies the potential maximum impact implementing a prediction model may have on reducing self‐harm events. Only those patients who are flagged as high‐risk and receive the intervention will have their self‐harm risk potentially reduced by the model implementation. FPR quantifies how frequently a visit would unnecessarily be recommended for the risk‐reducing intervention. FPR must be considered in the context of the potential harms of an intervention. While a brief risk assessment during a visit is relatively inexpensive—unnecessarily conducting this intervention takes away from limited visit time—could damage the therapeutic relationship between patient and provider, and could escalate to more intrusive interventions like involuntary hospitalization. PPV, the probability of an event among those visits classified as high‐risk, also informs the selection of an appropriate risk‐reducing intervention. Existing self‐harm prediction models have a PPV of 5% or less [13]. Given that more than 95% of the visits flagged as high‐risk will not be followed by self‐harm, accompanying interventions must be inexpensive, safe, and minimally disruptive [14]. NPV corresponds to the probability of no event among visits classified as low‐risk. Achieving high NPV is crucial in settings such as self‐harm risk prediction, in which the consequences of a false negative classification are significant; high NPV facilitates more confident management of lower risk patients. Overall, selection of an appropriate risk threshold must balance the benefits and harms of subsequent interventions.

While EHR data provide an excellent resource for rich data on a well‐defined population of individuals that can be leveraged for prediction model development and validation, EHR data are prone to measurement error [15, 16, 17], including misclassification of binary outcomes, like self‐harm [18, 19]. Patients may not seek care following self‐harm or may be hesitant to report injuries or poisonings as self‐inflicted, leading to missed diagnoses in our data [20]. It is also possible for accidental injuries or poisonings to be incorrectly documented as intentional self‐harm [21, 22]. Additionally, in many situations, outcome misclassification can be associated with patient characteristics [23, 24, 25]. For example, a key predictor of future self‐harm and suicide is a prior incident of self‐harm [26, 27]. Past self‐harm could also be associated with the risk of outcome misclassification, as self‐harm may be more likely to be recognized and diagnosed in those with a documented history of self‐harm.

Outcome misclassification in EHR data can lead to incorrectly estimating relationships between predictors and outcomes and inaccurate model validation [28, 29]. In the presence of binary outcome misclassification, naive model evaluation with the observed, potentially misclassified, outcomes will not reflect model performance for true (not misclassified) outcomes. Consider, for example, AUC: a prediction model's ability to discriminate between misclassified outcomes is typically not of scientific or clinical interest, and a prediction model that performs well in distinguishing misclassified outcomes may have limited discriminatory power for true outcomes. The impact of outcome misclassification is particularly concerning as accurate validation is necessary to correctly inform decisions about whether and how to use a prediction model. Consider PPV of a self‐harm prediction model: providers would take very different action for patients classified as high‐risk if the estimated PPV, or probability of a subsequent event, in that high‐risk stratum was 20% compared to 2%. If misclassification leads to biased estimates of PPV, clinical leaders may choose to pair the model with an intervention that is inappropriate for the true risk posed to patients classified as high‐risk.

While there are well‐established methods to adjust for outcome misclassification in an inferential setting [30, 31, 32, 33], few studies focus on the correct validation of prediction models in the presence of outcome misclassification. Wang et al. [28] derived formulas to estimate corrected ROC curves in the situation where misclassification is not related to risk factors or predictions, that is, misclassification is not risk‐differential. They demonstrated their approach in a setting where some events are not captured, but did not apply their method to the setting where events may be incorrectly present in the EHR. Zawistowski et al. [29] proposed a misclassification‐adjusted ROC procedure to account for misclassification in observed outcomes in both prediction model estimation and validation. This method relies on a likelihood‐based model for correcting logistic regression parameter estimates and, as a result, it is not applicable when a likelihood cannot be written explicitly, as is the case with common machine learning approaches, such as random forest. Demonstration of the proposed method of Zawistowski et al. [29] was also limited to a setting with only uncaptured events (underdiagnosis) with misclassification rates independent of the features. Thus, there is a need for validation approaches that account for outcome misclassification in complex misclassification scenarios and when flexible prediction modeling approaches are used.

Another limitation of existing literature is that validation methods have been demonstrated assuming misclassification rates are known [28, 29]. In reality, it is unlikely that researchers conducting model validation have prior knowledge of the true, underlying misclassification rates. And, naively assuming (estimated) misclassification rates elicited from earlier studies are precisely known will underestimate the variance of performance measures.

Manual review of clinical records is a common approach to quantifying misclassification in EHR data. In these chart review studies, trained annotators carefully examine patient records, including free text notes and other indicators in the data, to determine “gold standard” outcome measures. For example, a prior chart review study of EHR‐documented injuries or poisonings among patients reporting suicidal ideation found that nearly 90% of events with self‐harm diagnoses included documentation of self‐harm intent [19]. Some injuries or poisonings without a diagnosis of self‐harm had accompanying information in free text or other EHR documentation indicating self‐harm intent, including 8% of events coded as accidental, 28% coded as undetermined, and 11% without a coding of intent [19]. While a prediction model can be accurately validated within a chart review sample with gold‐standard outcomes, these samples are frequently quite small, as individual patient record review is expensive and time‐consuming to conduct. Validation of clinical prediction models in small chart review samples will yield estimates with high variability that, as a result, will be insufficient to inform decision‐making about whether and how to implement a prediction model. Ideally, information on model performance from much larger (unannotated) EHR datasets could also be leveraged to accurately evaluate model performance.

In this paper, we present a validation method that accommodates misclassification of binary outcomes for the AUC and classification measures (including TPR, FPR, PPV, and NPV) for true outcomes. Our proposed approach addresses the gaps in existing work as it is applicable to any prediction modeling approach and to complex misclassification scenarios, reflective of those encountered in real‐world EHR data. Instead of assuming misclassification rates are known, we illustrate how a model of the relationship between inaccessible true outcomes, observed misclassified outcomes, and risk predictors can be elicited using an external chart review data set. We further demonstrate how uncertainty in this estimation can be reflected in our proposed method to accurately quantify variability in performance measures. When available, EHR samples (with misclassified outcomes) are much larger than chart review samples (as is typically the case), and our proposed estimators are more efficient than those obtained by limiting validation to observations with gold‐standard outcomes.

We present results evaluating the proposed method in situations in which only events are misclassified (unidirectional misclassification) and for those in which both events and nonevents may be misclassified (bidirectional misclassification). We also consider scenarios when misclassification is independent of risk factors (non‐risk differential) as well as those when misclassification is associated with risk factors (risk differential) [34]. The method is evaluated through plasmode simulations leveraging data from a suicide and self‐harm prediction study [26, 35].

In Section 2, we present notation, define terms, and illustrate the impact of outcome misclassification on prediction model validation. In Section 3, we describe our proposed method to account for outcome misclassification. Section 4 contains details of our simulations, and Section 5 shows simulation results. We discuss how our methods could be useful in real‐world problems and further research directions in Section 6.

2. Prediction Model Validation with True and Misclassified Outcomes

2.1. Notation and Definitions

Consider a prediction model $f(·)$. We would like to evaluate the true performance of $f(·)$ in a validation sample, which may have misclassified outcomes. The methods presented in this paper are agnostic to the modeling strategy of $f(·)$. Further, we make no assumptions about the presence or absence of outcome misclassification in the training data used to estimate $f(·)$ and focus solely on estimating the accurate performance of the already estimated prediction model.

Let ${({\mathbf{\text{X}}}_i,Y_i)}_{i=1}^N$ represent the features and the observed potentially misclassified binary outcome, respectively, for each observation $i$ in the validation sample, $i=1,\dots ,N$. We calculate $f({\mathbf{X}}_i)$ for all observations to obtain predicted risk scores. We use $\alpha$ to denote some risk score threshold such that observations with $f({\mathbf{\text{X}}}_i)>\alpha$ would be labeled as positive predicted outcomes (that is, high‐risk for self‐harm in our motivating example) while observations with $f({\mathbf{\text{X}}}_i)\le \alpha$ would be considered as negative predicted outcomes (not classified as high‐risk). We would like to evaluate the performance of the risk model $f(·)$ in this validation set, but cannot easily do so because the true, correctly classified outcomes, denoted $T_i$, are not observed. In our motivating example, ${\mathbf{X}}_i$ represents risk factors for self‐harm, $T_i$ is a binary indicator of whether a self‐harm event occurred, and $Y_i$ is a binary indicator of whether a self‐harm event was documented in the EHR.

Suppose we also have a chart review sample containing ${({\mathbf{\text{X}}}_j,Y_j,T_j)}_{j=1}^n$, where ${\mathbf{\text{X}}}_j$ denotes the features, $Y_j$ denotes the observed binary outcome, and $T_j$ denotes the gold‐standard determination (referred throughout this paper as the “true” outcome) for each observation $j$ in our $n$‐size chart review sample, where $n<<N$. We assume observations in our validation and chart review sample are independently and identically distributed. (We consider the special case in which chart reviewed observations are a subset of the validation sample in Section 3.2).

We next define several misclassification scenarios. Misclassification may be unidirectional where only true events or only true nonevents are possibly mislabeled, but not both. The scenario in which some self‐harm events that truly occurred are not documented in the EHR, while no nonevents are erroneously recorded, can be expressed as $P(Y=1|T=1)\le 1$ and $P(Y=0|T=0)=1.$ The alternate scenario in which some nonevents are incorrectly diagnosed, while all true events are correctly labeled and some nonevents are incorrectly documented as events can be expressed as $P(Y=1|T=1)=1$ and $P(Y=0|T=0)\le 1$. We also consider bidirectional misclassification, in which both events and nonevents may potentially be mislabeled. Bidirectional misclassification can be written as $P(Y=1|T=1)\le 1$ and $P(Y=0|T=0)\le 1.$

We use nondifferential outcome misclassification to describe a scenario in which the likelihood of misclassification is independent of $\mathbf{\text{X}}$ such that $P(Y=1|T,\mathbf{\text{X}})=P(Y=1|T)$, while differential outcome misclassification indicates that misclassification rates are associated with at least some elements of $\mathbf{\text{X}}$, that is, $P(Y=1|T,\mathbf{\text{X}})\ne P(Y=1|T)$. In our motivating example, the correct recording of self‐harm in the EHR may be more likely for patients with other risk factors present, such as prior self‐harm or suicidal ideation. We note that both nondifferential and differential misclassification scenarios can be uni‐ or bidirectional.

2.2. ROC and Classification Accuracy Analysis With Gold‐Standard Outcomes

The ROC curve for a prediction model $f(·)$ is defined by the TPR and FPR of $f(·)$ across all possible risk score thresholds $\alpha$. The targets of interest are

$$\begin{array}{ll}\hfill \text{TPR}(f,\alpha )& =P(f(\mathbf{\text{X}})>\alpha |T=1)\text{and}\end{array}$$ (1)

$$\begin{array}{ll}\hfill \text{FPR}(f,\alpha )& =P(f(\mathbf{\text{X}})>\alpha |T=0).\end{array}$$ (2)

We will use the shorthand $\text{TPR}(\alpha )$ and $\text{FPR}(\alpha )$ below to refer to these quantities, since in all cases the prediction performance metric depends on the fixed $f(·)$.

In a chart review sample, we have access to true gold‐standard outcomes for observations, enabling us to obtain the following estimators for TPR and FPR at each threshold $\alpha$:

$${\widehat{\text{TPR}}}_{\mathrm{n}}(\alpha )=\frac{{\sum }_{j=1}^{n}I(T_j=1)I(f({\mathbf{\text{X}}}_j)>\alpha )}{{\sum }_{j=1}^{n}I(T_j=1)},$$

$${\widehat{\text{FPR}}}_{\mathrm{n}}(\alpha )=\frac{{\sum }_{j=1}^{n}I(T_j=0)I(f({\mathbf{\text{X}}}_j)>\alpha )}{{\sum }_{j=1}^{n}I(T_j=0)},$$

where $I(·)$ is a binary indicator function. The above estimators evaluated in the chart review sample are consistent for our target parameter of interest, the true population TPR and FPR of $f(·)$ at threshold $\alpha$.

The ROC curve is defined as the plot of TPR and FPR over all thresholds. The AUC is defined as the area under the ROC curve: $\text{AUC}={\int }_{\alpha }\text{TPR}(\alpha )d\text{FPR}(\alpha )$. A plug‐in estimator ${\widehat{\text{AUC}}}_n$ of $\text{AUC}$ can be obtained using ${\widehat{\text{TPR}}}_{\mathrm{n}}(\alpha )$ and ${\widehat{\text{FPR}}}_{\mathrm{n}}(\alpha )$.

We also consider two related measures of classification accuracy, PPV, the probability of an event among observations with a risk score above a threshold $\alpha$, and NPV, the probability of a nonevent among observations with a risk score no greater than a threshold $\alpha$. These targets can be written as:

$$\text{PPV}(\alpha )=P(T=1|f(\mathbf{\text{X}})>\alpha ),$$ (3)

$$\text{NPV}(\alpha )=P(T=0|f(\mathbf{\text{X}})\le \alpha ).$$ (4)

Similar to the estimator of TPR and FPR, gold‐standard outcomes can be used to obtain estimates of PPV($\alpha$) and NPV($\alpha$) consistent for the target:

$$\begin{array}{cc}\hfill & {\widehat{\text{PPV}}}_{\mathrm{n}}(\alpha )=\frac{{\sum }_{j=1}^{n}I(T_j=1)I(f({\mathbf{\text{X}}}_j)>\alpha )}{{\sum }_{j=1}^{n}I(f({\mathbf{\text{X}}}_j)>\alpha )},\hfill \\ \hfill & {\widehat{\text{NPV}}}_{\mathrm{n}}(\alpha )=\frac{{\sum }_{j=1}^{n}I(T_j=0)I(f({\mathbf{\text{X}}}_j)\le \alpha )}{{\sum }_{j=1}^{n}I(f({\mathbf{\text{X}}}_j)\le \alpha )}.\hfill \end{array}$$

2.3. ROC and Classification Accuracy Analysis With Observed, Potentially Misclassified Outcomes in a Validation Sample

In this section, we review the impact of unidirectional, nondifferential binary outcome misclassification on TPR, FPR, and AUC estimates, as provided in Wang et al. [28], and extend this presentation to consider the unidirectional differential scenario as well as bias in PPV and NPV estimates.

Outcome misclassification interferes with accurate ROC analysis. When outcome misclassification is present, true outcomes $T$ are not observed; only potentially incorrect outcome observations $Y$ are available. Having only access to $Y$, we could estimate TPR and FPR using

$${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )=\frac{{\sum }_{i=1}^{N}I(Y_i=1)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^{N}I(Y_i=1)}\text{and}$$ (5)

$${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )=\frac{{\sum }_{i=1}^{N}I(Y_i=0)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^{N}I(Y_i=0)},$$ (6)

where we use the subscript “mis” to highlight that the possibly‐misclassified outcomes $Y$ are used. ${\widehat{\text{AUC}}}_{\text{mis}}$ can be obtained by plugging in ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ and ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ into the formula for AUC: ${\widehat{\text{AUC}}}_{\text{mis}}={\int }_{\alpha }{\widehat{\text{TPR}}}_{\text{mis}}(\alpha )d{\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$. Additionally, the estimators for PPV and NPV when only $Y$ are observed is

$$\begin{array}{cc}\hfill & {\widehat{\text{PPV}}}_{\text{mis}}(\alpha )=\frac{{\sum }_{i=1}^{N}I(Y_i=1)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^{N}I(f({\mathbf{\text{X}}}_i)>\alpha )},\hfill \\ \hfill & {\widehat{\text{NPV}}}_{\text{mis}}(\alpha )=\frac{{\sum }_{i=1}^{N}I(Y_i=0)I(f({\mathbf{\text{X}}}_i)\le \alpha )}{{\sum }_{i=1}^{N}I(f({\mathbf{\text{X}}}_i)\le \alpha )}.\hfill \end{array}$$

Naively using the observed, potentially misclassified outcomes available in a validation sample may result in biased performance estimates. Below, we describe how misclassification of binary outcomes can introduce bias in the ROC analysis by first describing bias in each of its components. We focus our initial presentation on the setting of unidirectional misclassification in which only true events are missed before considering unidirectional misclassification in which only nonevents are incorrectly diagnosed.

The direction of bias for bidirectional misclassification cannot be as clearly outlined across all settings since total bias in performance metrics reflects bias from the rates of both false positives (nonevents incorrectly classified as events) and false negatives (events incorrectly classified as nonevents). However, in most cases, we expect the bias to be nonzero.

2.3.1. Estimation of the True Positive Rate

We start by considering the estimation of TPR. The target of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$, the estimator for TPR at threshold $\alpha$ evaluated with potentially misclassified $Y$, is $P(f(\mathbf{\text{X}})>\alpha |Y=1)$. We can express this target in terms of $Y$ and $T$ in order to compare the target of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ to (1):

$$\begin{array}{ll}P(f(\mathbf{\text{X}})>\alpha |Y=1)& =P(f(\mathbf{\text{X}})>\alpha |Y=1,T=1)\\ \hspace{0.17em}& =P(f(\mathbf{\text{X}})>\alpha |T=1)\frac{P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)}{P(Y=1|T=1)}.\end{array}$$ (7)

The first equation in (7) follows from our misclassification scenario; here, we are considering misclassification that is unidirectional such that only some true events are incorrectly labeled, such that, for those with $Y=1$, they must also have $T=1$. The second equation in (7) follows by using the conditional law of probability. We can now examine the amount of bias that will occur by using ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ to estimate $\widehat{\text{TPR}}(\alpha )$.

When misclassification is nondifferential, $P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)=P(Y=1|T=1)$, so equation (7) simplifies to $P(f(\mathbf{\text{X}})>\alpha |T=1)$, the same as (1). Thus, in this scenario, the target of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ is the parameter of interest $\text{TPR}(\alpha )$; estimates of the TPR using $Y$ will be consistent for the true TPR when unidirectional misclassification is nondifferential. We can also follow this reasoning intuitively in the context of our motivating example: when only some self‐harm events are undiagnosed and the probability of missed diagnosis is not associated with any risk factors, the distribution of risk factors (and the risk score $f(\mathbf{X})$) is the same among all the patients with a true self‐harm event ($T=1$) and the subset with self‐harm diagnosed ($Y=1$).

However, when outcome misclassification is differential, the target of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ does not equal $\text{TPR}(\alpha )$ but instead can be written as below, following from equation (7) above:

$$\begin{array}{cc}\hfill P(f(\mathbf{\text{X}})>\alpha |Y=1)& =\text{TPR}(\alpha )\frac{P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)}{P(Y=1|T=1)}:=\text{TPR}(\alpha )b(\alpha )\hfill \end{array}$$

Here, we can see that comparing $b(\alpha )$ to 1 indicates whether ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ is asymptotically biased upwards or downwards. When patients with higher risk scores are more likely to have true events correctly classified, $P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)\ge P(Y=1|T=1)$. As a result, the asymptotic bias of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will be nonnegative, and the estimated sensitivity of the model will be optimistic. We may expect this misclassification scenario under the assumption that patients with self‐harm risk factors like prior self‐harm or suicidal ideation are more likely to have future self‐harm events recognized and diagnosed in the EHR.

Alternatively, one could also hypothesize a scenario in which patients with lower predicted risk of self‐harm may be more likely to have self‐harm events correctly diagnosed, perhaps if those with lower predicted risk had better access to mental health care and were more likely to receive care following a self‐harm event. In this situation, $P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)\le P(Y=1|T=1)$ and ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will underestimate the $\text{TPR}(\alpha )$ for most $\alpha$, asymptotically.

Considering these two basic scenarios, we can conclude that using misclassified outcomes $Y$ to naively estimate the TPR can have positive or negative bias. One could also imagine more complex risk‐differential scenarios in which the probability of correctly diagnosing a true event is not monotone in $f(\mathbf{X})$ such that TPR is not uniformly over‐ or underestimated across thresholds $\alpha$.

We next consider the alternate unidirectional misclassification setting in which outcome misclassification is restricted to observations without an event, that is, observations with $T_i=0$ may be misdiagnosed or misrecorded as $Y_i=1$, and quantify the bias in TPR estimation for both nondifferential and differential scenarios. We start by writing $P(f(\mathbf{\text{X}})>\alpha |Y=1)$ as a weighted average of $P(f(\mathbf{\text{X}})>\alpha |Y=1,T=t)$ with weights equal to $P(T=t|Y=1)$ and $t\in \{0,1\}$:

$$\begin{array}{ll}\hspace{0.17em}& P(f(\mathbf{\text{X}})>\alpha |Y=1)\\ \hspace{0.17em}& =P(f(\mathbf{\text{X}})>\alpha |T=1,Y=1)P(T=1|Y=1)\\ \hspace{0.17em}& +P(f(\mathbf{\text{X}})>\alpha |T=0,Y=1)P(T=0|Y=1)\\ \hspace{0.17em}& =P(f(\mathbf{\text{X}})>\alpha |T=1)(1-P(T=0|Y=1))\\ \hspace{0.17em}& +P(f(\mathbf{\text{X}})>\alpha |T=0)\frac{P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=0)P(T=0|Y=1)}{P(Y=1|T=0)}\\ \hspace{0.17em}& =\text{TPR}(\alpha )-P(T=0|Y=1)(\text{TPR}(\alpha )-b^{\prime }(\alpha )\text{FPR}(\alpha )),\end{array}$$

where $b^{\prime }(\alpha ):=\frac{P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=0)}{P(Y=1|T=0)}$. When the misclassification in nondifferential, $b^{\prime }(\alpha )=1$ and TPR($\alpha$) is underestimated, as shown below:

$$\begin{array}{ll}\hfill & P(f(\mathbf{\text{X}})>\alpha |Y=1)=\text{TPR}(\alpha )-P(T=0|Y=1)(\text{TPR}(\alpha )-\text{FPR}(\alpha )).\end{array}$$

In the nondifferential setting, the asymptotic bias direction of ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ depends on how well the prediction model $f$ distinguishes those with $T=1$ and $T=0$ at the specific $\alpha$, that is, the sign of $\text{TPR}(\alpha )-\text{FPR}(\alpha )$. At the threshold values ($\alpha$) that $\text{TPR}(\alpha )<\text{FPR}(\alpha )$, ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will overestimate TPR($\alpha$); at the threshold values ($\alpha$) that $\text{TPR}(\alpha )>\text{FPR}(\alpha )$, ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will underestimate TPR($\alpha$); for $\alpha$ satisfying $\text{TPR}(\alpha )=\text{FPR}(\alpha )$, ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will correctly recover TPR($\alpha$). For the remainder of this manuscript, for simplicity, we assume the ROC curve is never below the diagonal line, that is, $\text{TPR}(\alpha )\ge \text{FPR}(\alpha )$ for all $\alpha$'s, which implies it performs no worse than random chance at all possible $\alpha$'s in distinguishing between those with and without true events.

For the differential case, the direction of bias depends on the difference between $\text{TPR}(\alpha )$ and $b^{\prime }(\alpha )\text{FPR}(\alpha )$. For example, when patients with higher predicted risk scores are less likely to have their true event status $T=0$ misclassified as $Y=1$, $b^{\prime }(\alpha )<1$ and, following our assumption that the ROC curve of $f$ is never below the diagonal line, $b^{\prime }(\alpha )\text{FPR}(\alpha )$<$\text{TPR}(\alpha )$ such that ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will underestimate TPR($\alpha$).

2.3.2. Estimation of the False Positive Rate

We next consider estimation of the FPR in the presence of unidirectional binary outcome misclassification. Analogous to the presentation above, we compare the target for ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ to $\text{FPR}(\alpha )$:

$$\begin{array}{cl}& P(f(\mathbf{\text{X}})>\alpha |Y=0)\\ & =P(f(\mathbf{\text{X}})>\alpha |Y=0,T=0)P(T=0|Y=0)\\ & +P(f(\mathbf{\text{X}})>\alpha |Y=0,T=1)P(T=1|Y=0)\\ & =P(f(\mathbf{\text{X}})>\alpha |T=0)\{1-P(T=1|Y=0)\}\\ & +P(T=1|Y=0)\frac{P(Y=0|f(\mathbf{\text{X}})>\alpha ,T=1)P(f(\mathbf{\text{X}})>\alpha )|T=1)}{P(Y=0|T=1)}\\ & =P(f(\mathbf{\text{X}})>\alpha |T=0)+P(T=1|Y=0)\\ & (P(f(\mathbf{\text{X}})>\alpha |T=1)\frac{P(Y=0|f(\mathbf{\text{X}})>\alpha ,T=1)}{P(Y=0|T=1)}\\ & -P(f(\mathbf{\text{X}})>\alpha |T=0))\\ & =\text{FPR}(\alpha )+P(T=1|Y=0)(\text{TPR}(\alpha )c(\alpha )-\text{FPR}(\alpha )),\end{array}$$ (8)

where $c(\alpha ):=\frac{P(Y=0|f(\mathbf{\text{X}})>\alpha ,T=1)}{P(Y=0|T=1)}$. The last line of (8) quantifies the deviation of the ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ from the target, $\text{FPR}(\alpha )$. In the nondifferential case, $c(\alpha )=1$, and provided that our ROC curve of $f$ is never below the diagonal line across all possible thresholds, the asymptotic bias of ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ is $P(T=1|Y=0)(\text{TPR}(\alpha )-\text{FPR}(\alpha ))\ge 0$. Given that a higher FPR indicates poor model performance—prediction models with lower FPR at a given $\alpha$ are preferred—over‐estimating the FPR is equivalent to underestimating model performance.

When unidirectional misclassification is risk‐differential, the direction of bias of ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ depends on the relationship between $\text{TPR}(\alpha )c(\alpha )$ and $\text{FPR}(\alpha )$. For example, if those with higher predicted risk scores are less likely to have true events recorded in the EHR, perhaps due to more barriers to accessing care, $c(\alpha )>1$ and ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ will overestimate FPR($\alpha$).

Under the alternate unidirectional misclassification scenario in which all events are correctly labeled but nonevents may be mislabeled, ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ will recover the desired estimand, $\text{FPR}(\alpha )$ in (2), if misclassification is nondifferential. To see this, we expand $P(f(\mathbf{\text{X}})>\alpha |Y=0)$ as follows:

$$\begin{array}{cc}\hfill & P(f(\mathbf{\text{X}})>\alpha |Y=0)\hfill \\ \hfill & =P(f(\mathbf{\text{X}})>\alpha |Y=0,T=0)\hfill \\ \hfill & =P(f(\mathbf{\text{X}})>\alpha |T=0)\frac{P(Y=0|f(\mathbf{\text{X}})>\alpha ,T=0)}{P(Y=0|T=0)}\hfill \\ \hfill & =\text{FPR}(\alpha )c^{\prime }(\alpha ),\hfill \end{array}$$

where $c^{\prime }(\alpha ):=\frac{P(Y=0|f(\mathbf{\text{X}})>\alpha ,T=0)}{P(Y=0|T=0)}$. When misclassification is nondifferential, $c^{\prime }(\alpha )=1$, so the target of ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ is FPR($\alpha$) in (2). When the misclassification is risk‐differential, the direction of bias depends on the comparison between $c^{\prime }(\alpha )$ and 1. For example, if patients with higher predicted risk scores are more likely to have their nonevents correctly classified, then $c^{\prime }(\alpha )>1$ and ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ will overestimate FPR($\alpha$). If, instead, patients with higher predicted scores are more likely to have their true nonevent status misclassified as $Y=1$, then we have $c^{\prime }(\alpha )<1$ leading to underestimation of FPR.

2.3.3. Estimation of the AUC

We next consider estimation of the AUC. When unidirectional (only true events possibly missed) binary outcome misclassification is nondifferential, we have shown that, for all possible $\alpha$, ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ are unbiased estimates of $\text{TPR}(\alpha )$, but ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ can overestimate the target asymptotically. Given that the ROC curve plots TPR on the y‐axis against FPR on the x‐axis for all possible thresholds of $\alpha$, overestimation of $\text{FPR}(\alpha )$ will shift the curve to the right as illustrated in Figure 1. Thus, the ${\widehat{\text{AUC}}}_{\text{mis}}$ for the ROC curve obtained by ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ and ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ will underestimate the target AUC, underestimating the discriminative ability of the prediction model.

FIGURE 1.

Effect of unidirectional, nondifferential outcome misclassification on ROC plot and the estimated AUC when (a) only true events are incorrectly labeled as nonevents and (b) only nonevents are incorrectly labeled as events.

As shown in Figure 1, on average, if the FPR is always overestimated while the TPR remains unbiased, one can imagine the ROC curve shifts to the right (toward higher FPR values) for a given TPR. This shift moves the curve away from the ideal top‐left corner, which corresponds to better performance. This magnitude of bias will increase as the misclassification gets worse because the proportion of $T=1$ in $Y=0$ will be greater, and thus the positive deviation of $P(f(\mathbf{\text{X}})>\alpha |Y=0)$ from $P(f(\mathbf{\text{X}})>\alpha |T=0)$ will get larger.

When the direction of unidirectional‐nondifferential misclassification is flipped, that is, the binary outcome misclassification only happens within patients with true nonevents, and provided that the ROC curve of $f$ is never below the diagonal line, ${\widehat{\text{AUC}}}_{\text{mis}}$ will also underestimate the target AUC, thereby underestimating the discriminative ability of the model, since ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ is asymptotically unbiased, while ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ will underestimate TPR($\alpha$).

We cannot make a general conclusion about the direction of bias for AUC estimates when misclassification is differential. As presented above, the direction of bias for ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$ and ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$ depends on the association between the risk score and misclassification risk for true events and nonevents, respectively. The asymptotic bias direction for ${\widehat{\text{AUC}}}_{\text{mis}}$ is then a balance between the bias from the estimates of each, and the resulting AUC estimate may over‐ or underestimate the model's discriminatory ability for true outcomes.

2.3.4. Estimation of the Positive and Negative Predictive Value

A similar approach can be taken to evaluate the potential impact of unidirectional binary outcome misclassification for other classification accuracy measures. We first compare the target of the estimate of ${\widehat{\text{PPV}}}_{\text{mis}}$ to the target measure evaluated in true outcomes, $\text{PPV}(\alpha )$ in the setting where only some true events may be mislabeled and incorrectly classified as $Y=0$ in the EHR:

$$\begin{array}{ll}P(Y=1|f(\mathbf{\text{X}})>\alpha )& =P(Y=1,T=1|f(\mathbf{\text{X}})>\alpha )\\ \hspace{0.17em}& =P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)P(T=1|f(\mathbf{\text{X}})>\alpha )\\ \hspace{0.17em}& =P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)\text{PPV}(\alpha ).\end{array}$$

Since $P(Y=1|f(\mathbf{\text{X}})>\alpha ,T=1)\le 1$, ${\widehat{\text{PPV}}}_{\text{mis}}(\alpha )$ tends to underestimate $\text{PPV}(\alpha )$; this holds when misclassification is nondifferential or differential. In our motivating example of self‐harm risk prediction, PPV obtained using misclassified outcomes would underestimate the rate of self‐harm at a threshold selected for clinical intervention.

When only nonevents are incorrectly classified as having the event, ${\widehat{\text{PPV}}}_{\text{mis}}(\alpha )$ tends to overestimate the desired estimand, PPV($\alpha$). To see this, we expand $P(Y=1|f(\mathbf{\text{X}})>\alpha )$ as follows:

$$\begin{array}{ll}P(Y=1|f(\mathbf{\text{X}})>\alpha )& =P(Y=1,T=1|f(\mathbf{\text{X}})>\alpha )+P(Y=1,T=0|f(\mathbf{\text{X}})>\alpha )\\ \hspace{0.17em}& =P(T=1|f(\mathbf{\text{X}})>\alpha )+P(Y=1,T=0|f(\mathbf{\text{X}})>\alpha )\\ \hspace{0.17em}& =\text{PPV}(\alpha )+P(Y=1,T=0|f(\mathbf{\text{X}})>\alpha ),\end{array}$$

since $P(Y=1,T=0|f(\mathbf{\text{X}})>\alpha )\ge 0$, in this situation, ${\widehat{\text{PPV}}}_{\text{mis}}(\alpha )$ will have nonnegative bias in the case of both nondifferential and risk‐differential misclassification.

Similarly, ${\widehat{\text{NPV}}}_{\text{mis}}$ will generally overestimate NPV($\alpha$) in the unidirectional misclassification setting in which only true events may be misclassified as $Y=0$, regardless of whether the misclassification is nondifferential or risk‐differential:

$$\begin{array}{ll}P(Y=0|f(\mathit{X})\le \alpha )& =P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=1)P(T=1|f(\mathbf{\text{X}})\le \alpha )\\ \hspace{0.17em}& +P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=0)P(T=0|f(\mathbf{\text{X}})\le \alpha )\\ \hspace{0.17em}& =P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=1)P(T=1|f(\mathbf{\text{X}})\le \alpha )\\ \hspace{0.17em}& +\text{NPV}(\alpha ).\end{array}$$

Under the alternate unidirectional binary outcome misclassification in which only nonevents are incorrectly labeled as $Y=1$,

$$\begin{array}{ll}\hfill & P(Y=0|f(\mathbf{\text{X}})\le \alpha )\\ \hfill & =P(Y=0,T=0|f(\mathbf{\text{X}})\le \alpha )\\ \hfill & =P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=0)P(T=0|f(\mathbf{\text{X}})\le \alpha )\\ \hfill & =P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=0)\text{NPV}(\alpha ).\end{array}$$

Given $P(Y=0|f(\mathbf{\text{X}})\le \alpha ,T=0)\le 1$, in this case, ${\widehat{\text{NPV}}}_{\text{mis}}$ tends to underestimate NPV($\alpha$), in both nondifferential or differential misclassification settings.

3. Leveraging Chart Review to Improve Model Validation When There is Outcome Misclassification

We now consider how gold‐standard outcomes obtained through chart review can be used in prediction model validation to produce better performance estimates under outcome misclassification. As a reminder, we are interested in validating a prediction model $f(\mathbf{X})$ in a validation set with misclassified binary outcomes $Y$; the true outcomes $T$ are not observed on the full sample.

We do, however, have a smaller chart review sample that is selected from the same population as the validation set and contains gold‐standard true outcome determinations $T$, in addition to risk predictions $f(\mathbf{X})$ and observed outcomes $Y$. While the prediction model $f(\mathbf{X})$ could be accurately validated in this chart review sample—as it is drawn from the same population as the validation set, estimates would be unbiased—the sample size is small, thus uncertainty around the performance estimates in the chart review sample is likely to be large. Ideally, one would like to take advantage of the considerably larger sample size in the validation set to more precisely estimate performance.

Toward that goal, with access to the true outcomes in the chart review set, we can estimate a model that relates true outcomes $T$ to the misclassified observed outcomes $Y$ and risk factors $\mathbf{X}$. Specifically, we can estimate $P(T=1|\mathbf{\text{X}}=\mathbf{\text{x}},Y=y)$ within the independent chart review set by ${ĝ}_n(\mathbf{\text{x}},y)$, which can be any function mapping that constrains fitted values to the interval [0,1], including a logistic regression model. This function ${ĝ}_n(·)$ can then be used to obtain fitted values ${ĝ}_n({\mathbf{\text{X}}}_i,Y_i)$ for $i=1,\dots ,N$ in the validation sample to approximate the unobserved true outcomes.

Our proposed method is then to substitute the unobserved true outcomes ${(T_i)}_{i=1}^N$ with ${ĝ}_n{({\mathbf{\text{X}}}_i,Y_i)}_{i=1}^N$. Proposed estimators for $\text{TPR}(\alpha )$ and $\text{FPR}(\alpha )$ are then defined as follows:

$${\widehat{\text{TPR}}}_{\text{proposed}}(\alpha )=\frac{{\sum }_{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)},$$ (9)

$${\widehat{\text{FPR}}}_{\text{proposed}}(\alpha )=\frac{{\sum }_{i=1}^N(1-{ĝ}_n({\mathbf{\text{X}}}_i,Y_i))I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^N(1-{ĝ}_n({\mathbf{\text{X}}}_i,Y_i))}.$$ (10)

${\widehat{\text{AUC}}}_{\text{proposed}}$ can be obtained as

$${\widehat{\text{AUC}}}_{\text{proposed}}={\int }_{\alpha }{\widehat{\text{TPR}}}_{\text{proposed}}(\alpha )d{\widehat{\text{FPR}}}_{\text{proposed}}(\alpha ).$$

This approach can also be used for other measures of classification accuracy, such as PPV and NPV:

$${\widehat{\text{PPV}}}_{\text{proposed}}(\alpha )=\frac{{\sum }_{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^{N}I(f({\mathbf{\text{X}}}_i)>\alpha )}.$$ (11)

$${\widehat{\text{NPV}}}_{\text{proposed}}(\alpha )=\frac{{\sum }_{i=1}^N(1-{ĝ}_n({\mathbf{\text{X}}}_i,Y_i))I(f({\mathbf{\text{X}}}_i)\le \alpha )}{{\sum }_{i=1}^{N}I(f({\mathbf{\text{X}}}_i)\le \alpha )}.$$ (12)

If $\Vert {ĝ}_n(\mathbf{\text{x}},y)-P(T=1|\mathbf{\text{X}}=\mathbf{\text{x}},Y=y){\Vert }_{\infty }\stackrel{p}{\to }0$ as $n\to \infty$, then, the proposed estimators in Equations ((9), (10), [Link], (11), (12)) are consistent to the targets of interest in Equations ((1), (2), [Link], [Link], (3), (4)). We note that this is a sufficient condition, but there may be weaker conditions under which consistency holds. For example, for $\text{TPR}(\alpha )$, both the numerator and denominator converge in probability to their targets, as shown below by applying the weak law of large numbers,

$$\begin{array}{ll}\hfill & |\frac{1}{N}\sum _{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)-𝔼[T]|\\ \hfill & \le |\frac{1}{N}\sum _{i=1}^{N}P(T_i=1|{\mathbf{\text{X}}}_i,Y_i)-𝔼[T]|\\ \hfill & +|\frac{1}{N}\sum _{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)-\frac{1}{N}\sum _{i=1}^{N}P(T_i=1|{\mathbf{\text{X}}}_i,Y_i)|\\ \hfill & \le o_p(1),\\ \hfill & |\frac{1}{N}\sum _{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )-𝔼[TI(f(\mathbf{\text{X}})>\alpha )]|\\ \hfill & \le |\frac{1}{N}\sum _{i=1}^{N}P(T_i=1|{\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )-𝔼[TI(f(\mathbf{\text{X}})>\alpha )]|\\ \hfill & +|\frac{1}{N}\sum _{i=1}^N\left[{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)-P(T_i=1|{\mathbf{\text{X}}}_i,Y_i)\right]I(f({\mathbf{\text{X}}}_i)>\alpha )|\\ \hfill & \le o_p(1)+||{ĝ}_n(\mathbf{\text{x}},y)-P(T=1|\mathbf{\text{X}}=\mathbf{\text{x}},Y=y)|{|}_{\infty }=o_p(1),\end{array}$$

and, thus,

$$\begin{array}{ll}\hfill & \frac{1}{N}\sum _{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)\stackrel{p}{\to }𝔼[T]\\ \hfill & \frac{1}{N}\sum _{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )\stackrel{p}{\to }𝔼[TI(f(\mathbf{\text{X}})>\alpha )].\end{array}$$

Provided that $𝔼[T]\ne 0$, by Slutsky's theorem,

$$\begin{array}{ll}\hfill {\widehat{\text{TPR}}}_{\text{proposed}}(\alpha )& =\frac{\frac{1}{N}{\sum }_{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )}{\frac{1}{N}{\sum }_{i=1}^N{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)}\\ \hfill & \stackrel{p}{\to }\frac{𝔼\left[T_{i}I(f({\mathbf{\text{X}}}_i)>\alpha )\right]}{𝔼\left[T_i\right]}=\text{TPR}(\alpha ).\end{array}$$

Similar calculations can be used for the other estimated classification measures, including ${\widehat{\text{FPR}}}_{\text{proposed}}\stackrel{p}{\to }\text{FPR}(\alpha )$, and thus ${\text{AUC}}_{\text{proposed}}\stackrel{p}{\to }\text{AUC}$.

3.1. Estimating Uncertainty in Performance Measures

We next propose a method for accurately quantifying variability in the proposed performance measures by accounting for both sampling variability in the validation set and variability in the model ${ĝ}_n(\mathbf{\text{X}},Y)$ approximating $T$ that is derived from chart review data. To capture sampling variability in the validation sample, one can repeatedly sample with replacement from the validation set (i.e., nonparametric bootstrap) to generate multiple bootstrap samples ${({\mathbf{\text{X}}}_i^b,Y_i^b)}_{i=1}^N$ for bootstrap replications $b=1,\dots ,B$ [36].

Since the method we proposed involves estimating $P(T=1|\mathbf{\text{X}},Y)$, we also need to account for the uncertainty in this estimation. Otherwise, we might be overly optimistic about the efficiency of the performance estimates. If we have access to individual observations in the chart review sample, we can also use a nonparametric bootstrap to obtain a sample ${({\mathbf{\text{X}}}_i^b,Y_i^b,T_i^b)}_{i=1}^n$ in which to estimate ${ĝ}_n^b$. Alternatively, the parametric bootstrap (i.e., resampling from some assumed distribution) can be used, depending on the form of ${ĝ}_n$ (e.g., fitted with a parametric model or a nonparametric model), to obtain ${ĝ}_n^b$ in both settings where we have access to the chart review data or we only have ${ĝ}_n$.

Bootstrap procedures for the validation sample and ${ĝ}_n$ must be integrated to obtain bootstrapped estimates of the performance metrics. For example, for TPR estimation, for each bootstrap $b$, we obtain an estimate

$${\text{TPR}}_{\text{proposed}}^b(\alpha )=\frac{\frac{1}{N}{\sum }_{i=1}^N{ĝ}_n^b({\mathbf{\text{X}}}_i^b,Y_i^b)I(f({\mathbf{\text{X}}}_i^b)>\alpha )}{\frac{1}{N}{\sum }_{i=1}^N{ĝ}_n^b({\mathbf{\text{X}}}_i^b,Y_i^b)}$$

from the bootstrapped validation sample ${({\mathbf{\text{X}}}_i^b,Y_i^b)}_{i=1}^N$and the bootstrapped ${ĝ}_n^b$. Then, one can use standard approaches to aggregate estimates across bootstrap samples to derive quantile‐ or Wald‐based 95% confidence intervals for each measure. We demonstrate one possible approach to uncertainty quantification when $P(T=1|\mathbf{\text{X}},Y)$ is estimated with a logistic regression in our simulation study, see below for details.

3.2. Pooling Chart Review and Validation Sample Data

The estimators proposed above in 3 can be used when the investigators only have access to ${ĝ}_n$ and do not require access to $n$ individual observations in the chart review sample. If individual‐level chart review data are accessible, then we can extend the proposed estimator to pool the chart review and validation samples when estimating performance metrics, for example, TPR:

$$\begin{array}{ll}\hfill & {\widehat{\text{TPR}}}_{\text{pooled}}:=\frac{{\sum }_{i=1}^n{T}_{i}I(f({\mathbf{\text{X}}}_i)>\alpha )+{\sum }_{i=n+1}^{n+N}{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)I(f({\mathbf{\text{X}}}_i)>\alpha )}{{\sum }_{i=1}^n{T}_i+{\sum }_{i=n+1}^{n+N}{ĝ}_n({\mathbf{\text{X}}}_i,Y_i)}.\end{array}$$

Under the additional assumption $\frac{n}{n+N}\to \lambda >0$, this pooled estimator is also consistent for the target. Details can be found in the Supporting Information. Variance estimates for the pooled estimators can be assessed via a nonparametric bootstrapping approach in which performance estimates are obtained in a series of bootstrap samples of size $N$ from the validation set (excluding any observations in the chart review set) and of size $n$ from the chart review set. This approach will account for variability in the (pooled) validation and chart review sets used for estimating prediction model performance and variability in the chart review set used to obtain ${ĝ}_n$. In the remainder of the manuscript, we mimic the more general setting in which we only have access to the fitted model ${ĝ}_n$ and the chart review and validation samples are not pooled.

4. Simulation

4.1. Data Generation

We evaluated the proposed method using a plasmode simulation study with real‐world observations from clinical records data [37]. Our plasmode simulation follows the motivating example of risk prediction for self‐harm. The simulation dataset was obtained from the population of all outpatient mental health visits for patients 11 years old and older at 7 health systems (HealthPartners, Henry Ford Health, and the Colorado, Hawaii, Northwest, Southern California, and Washington regions of Kaiser Permanente) in the United States during 2009–2017 [26, 35]. Predictor data associated with each visit includes more than 100 baseline risk factors (such as prior diagnoses, medications, utilization, and self‐harm) and a binary indicator of whether the person had a nonfatal self‐harm diagnosis or suicide death in the 90 days following the visit. These event indicators were considered to be the “true” outcome status for the purposes of this simulation study. In the original dataset, complete outcome observation was not available for people with non‐suicide death or disenrollment from the participating health systems during the 90‐day follow‐up period. The sample used for plasmode simulations was limited to visits with complete 90‐day outcome observation. Misclassified outcomes were simulated under different settings, following the scenarios described in Table 1. From the existing dataset with repeated visits per person, we created a dataset to use for simulation with $\sim$3 million independent observations by randomly sampling one visit per person from the entire population. For people with a self‐harm event following any visit, a visit with an event was sampled.

TABLE 1.

Misclassification scenarios explored in plasmode simulation studies. Data were generated with an event rate of 0.1 for all scenarios. Event rates of 0.05 and 0.2 were also explored for scenarios marked with a .

		Misclassification rates
Misclassification type	Description	$P(Y=1|T=1,X=x)$	$P(Y=1|T=0,X=x)$	Scenario label
Unidirectional Nondifferential	$P(Y=1|T=1,X=x)=P(Y=1|T=1)$	0.9 0.7 0.5	0 0 0	(a) (b)* (c)
Unidirectional Differential	$P(Y=1|T=1,X=x)$ ranges from $s_1$ for those with lowest “true” risk scores to $t_1$ for those with highest “true” risk scores	$s_1=0.7$, $t_1=0.95$ $s_1=0.25$, $t_1=0.75$ $s_1=0.95$, $t_1=0.7$	0 0 0	(d)* (e) (f)
Bidirectional Nondifferential	$P(Y=1|T=1,X=x)=P(Y=1|T=1)$, $P(Y=1|T=0,X=x)=P(Y=1|T=0)$	0.9 0.7 0.5	0.01 0.05 0.075	(g) (h)* (i)
Bidirectional differential	$P(Y=1|T=1,X=x)$ ranges from $s_1$ for those with lowest “true” risk scores to $t_1$ for those with highest “true” risk scores $P(Y=1|T=0,X=x)$ ranges from $s_0$ for those with lowest “true” risk scores to $t_0$ for those with highest “true” risk scores	$s_1=0.7$, $t_1=0.95$ $s_1=0.25$, $t_1=0.75$ $s_1=0.95$, $t_1=0.7$	$s_0=0.005$, $t_0=0.1$ $s_0=0.01$, $t_0=0.2$ $s_0=0.05$, $t_0=0.005$	(j)* (k) (l)

All misclassification scenarios (presented in Table 1 and described below) were explored with 1000 simulation iterations. For each iteration, we randomly sampled (without replacement) 50 000 observations for training and validation sets and 1000 observations for the chart review set (independent of the validation set), that is, $n=1000$ and $N=50000$, with stratified sampling to obtain a 10% event rate in training, validation, and chart review sets. Additional simulations considered smaller training and validation sets ($N=25000$) and chart review sets $(n=500)$ as well as event rates of 5% and 20%.

We generated misclassified outcome data under various scenarios in which we manipulated the misclassification types (uni‐ and bidirectional, differential and nondifferential), misclassification rates, and event rates. We explored a variety of potential misclassification rates, ranging from minimal to more substantive misclassification (e.g., the probability that a true event is correctly observed ranges from 50% to 90%). We generated misclassified outcomes ${(Y_i)}_{i=1}^{n+N}$ in the chart review set and validation set using Bernoulli distributions with specified misclassification rates $P(Y=1|T=0)$ and $P(Y=1|T=1)$ listed in Table 1. Misclassification rates were selected to explore a range of plausible clinical scenarios. For nondifferential misclassification, the probability of misclassification depended on the true event status, $T$, only. For differential misclassification, the probability of misclassification depended on the “true” self‐harm risk, that is, risk scores that were estimated using a random forest prediction model trained with true (not misclassified) outcomes. For example, in the first unidirectional risk‐differential scenario listed in Table 1, we defined parameters for the probability of correctly observing $Y=1$ for observations with events $(T=1)$ such that the probability of correctly diagnosing the event ranged from a minimum of 0.7 for observations with the lowest true risk (denoted $s_1$ in Table 1) of an event and a maximum of 0.95 for those at the highest risk (denoted $t_1$). More details on the data‐generating mechanism are provided in the Supporting Information.

While our proposed approach can be used to validate any existing prediction model (regardless of whether the model was trained with correct or error‐prone data), in this simulation, we estimated prediction models with misclassified outcomes following the data‐generating specification of each misclassification scenario. In each simulation iteration, we estimated a random forest prediction model $f(·)$ in the training set with misclassified outcomes. We then obtained fitted values, $f({\mathbf{\text{X}}}_i)$, for that prediction model for all observations in the validation and chart review sets. More details on how the random forest prediction models were estimated (e.g., tuning parameter value selection) in each simulation iteration are provided in the Supporting Information.

4.2. Estimation of Performance Measures

In each simulation iteration, we calculated 3 estimators for the target performance measures, $\text{TPR}(\alpha )$, $\text{FPR}(\alpha )$, $\text{PPV}(\alpha )$, $\text{NPV}(\alpha )$, and $\text{AUC}$:

1. those estimated using chart review data alone, ${\widehat{\text{TPR}}}_{\mathrm{n}}(\alpha )$, ${\widehat{\text{FPR}}}_{\mathrm{n}}(\alpha )$, ${\widehat{\text{PPV}}}_{\mathrm{n}}(\alpha )$, ${\widehat{\text{NPV}}}_{\mathrm{n}}(\alpha )$, and ${\widehat{\text{AUC}}}_{\mathrm{n}}$;

2. those estimated by naively using potentially misclassified outcomes, ${\widehat{\text{TPR}}}_{\text{mis}}(\alpha )$, ${\widehat{\text{FPR}}}_{\text{mis}}(\alpha )$, ${\widehat{\text{PPV}}}_{\text{mis}}(\alpha )$, ${\widehat{\text{NPV}}}_{\text{mis}}(\alpha )$, and ${\widehat{\text{AUC}}}_{\text{mis}}$; and

3. the proposed estimators presented above, ${\widehat{\text{TPR}}}_{\text{proposed}}(\alpha )$, ${\widehat{\text{FPR}}}_{\text{proposed}}(\alpha )$, ${\widehat{\text{PPV}}}_{\text{proposed}}(\alpha )$, ${\widehat{\text{NPV}}}_{\text{proposed}}(\alpha )$, and ${\widehat{\text{AUC}}}_{\text{proposed}}$.

The proposed estimators were obtained through 3 steps:

1. Estimate a logistic regression model for true outcomes, $g(·)$ from the chart review sample. Within the chart review sample, we estimate a regression model for the true outcomes $T$ given misclassified outcomes $Y$ and predicted risk $f(\mathbf{X})$. We note that the “true” $g(X,Y)$ was not specified directly for these plasmode simulations, as we instead specified $P(Y=1|T,f(\mathbf{X}))$ to generate misclassified outcomes $Y$ based on true outcomes $T$ sampled from our real‐world dataset. For simplicity, a logistic regression model was selected:

   $$\begin{array}{ll}\hfill \text{logit}({ĝ}_n({\mathbf{\text{X}}}_j,Y_j))& =\text{logit}(\widehat{P}(T_j=1|Y_j,{\mathbf{\text{X}}}_j))\\ \hfill & ={\widehat{\gamma }}_0+{\widehat{\gamma }}_1{Y}_j+{\widehat{\gamma }}_2\text{logit}(f({\mathbf{\text{X}}}_j))\\ \hfill & +{\widehat{\gamma }}_3\text{logit}(f({\mathbf{\text{X}}}_j))I(Y_j=1)\end{array}$$

   We save the coefficient estimates $\widehat{\gamma }$ and the corresponding model coefficient covariance matrices, ${\widehat{\sum }}_{\widehat{\gamma }}$. While we used a relatively simple model in our experiment, in practice and sample size allowing, one can use more flexible models, including splines, as the relationship would be unknown and could be complex.

2. Plug fitted values $\widehat{P}(T_i=1|Y_i,{\mathbf{\text{X}}}_i)={ĝ}_n(Y_i,{\mathbf{\text{X}}}_i)$ into proposed formulae. We calculated the proposed estimates for all performance measures, including TPR($\alpha$), FPR($\alpha$), PPV($\alpha$), and NPV($\alpha$) at all possible thresholds, that is, $\alpha =f{({\mathbf{\text{X}}}_i)}_{i=1}^N$, in the validation set by plugging in ${ĝ}_n(Y_i,{\mathbf{\text{X}}}_i)$obtained in step 1 into ((9), (10), [Link], (11), (12)), respectively. We also obtained proposed estimates for AUC.

3. Estimate uncertainty using bootstrap sampling. We used parametric and nonparametric bootstrapping to obtain interval estimates for all performance measure estimates. Within each simulation iteration, we obtained 2000 bootstrap samples of size 50 000 with replacement from the validation set. We indexed each bootstrapped sample by $b$, for $b=1,\dots ,2000$, and the corresponding misclassified outcomes and features for each bootstrap sample are denoted as $Y_i^b$ and ${\mathbf{\text{X}}}_i^b$, for $i=1,\dots ,n$.

We also captured uncertainty associated with estimating $\widehat{\gamma }$ via parametric bootstrap sampling. For each bootstrap sample $b$, we randomly drew coefficients ${\widehat{\gamma }}^b$ from the multivariate normal distribution $N(\widehat{\gamma },{\widehat{\sum }}_{\widehat{\gamma }})$. We obtained fitted values under the models

$$\begin{array}{ll}\hfill \text{logit}({ĝ}_n^b({\mathbf{\text{X}}}_j^b,Y_j^b))& ={\widehat{\gamma }}_0^b+{\widehat{\gamma }}_1^b{Y}_j^b+{\widehat{\gamma }}_2^b\text{logit}(f({\mathbf{\text{X}}}_j^b))\\ \hfill & +{\widehat{\gamma }}_3^b\text{logit}(f({\mathbf{\text{X}}}_j^b))I(Y_j^b=1)\end{array}$$

These predictions were then used to obtain proposed performance measures for each bootstrap sample. From the completion of 2000 bootstrap replications, we evaluated the empirical distribution of our estimated performance measures to derive 95% quantile‐based CIs.

We also estimated performance measures in the validation set using misclassified outcomes as described in Section 2.3 and in the chart review sample using gold‐standard outcomes. For each, we accounted for the sampling uncertainty via the 2000 bootstrap resamples from the validation set (for estimates using misclassified outcomes) and from the chart review sample (for gold‐standard estimates) and derived the estimates naively using the potentially misclassified outcomes and the estimates using the chart review sample alone in each bootstrapped sample. The 95% CIs were obtained for each estimate by using the 2.5% and 97.5% percentiles of these 2000 estimates.

4.3. Evaluation of Proposed Method

In each iteration, we calculated the empirical “true” performance measures for each metric in the validation set using true outcomes $T_i$ known to the investigators in this plasmode simulation scenario. Bias, mean squared errors (MSE), and 95% CI coverage with respect to the empirical targets (estimated with true outcomes) and average CI width were calculated for estimates naively using the potentially misclassified outcomes in the validation set, using the chart review sample alone, and our proposed estimates over 1000 simulation iterations. We computed the Monte Carlo standard errors (MCSE) for all bias, MSE, and 95% CI coverage and average CI width estimates [38].

5. Results

Random forest prediction models trained with misclassified outcomes had similar or slightly lower performance than prediction models trained with correct outcomes. The AUC of prediction models trained with correct outcomes (averaged over all simulation replications) at an event rate of 0.10 and training set size of $N=50000$ was 0.871. The AUC of prediction models trained with misclassified outcomes at that same event rate ranged from 0.853‐0.871 across the misclassification scenarios examined, with lower AUC in settings with larger misclassification rates.

We present the results for our evaluation of estimators for AUC under different misclassification scenarios and an event rate of 0.1 in Figures 2, 3, 4 and Table S1 in the Supporting Information. Figure 2 shows the average bias across simulation replications for ${\widehat{\text{AUC}}}_n$, ${\widehat{\text{AUC}}}_{\text{mis}}$, and ${\widehat{\text{AUC}}}_{\text{proposed}}$, where bias is defined as the difference between each estimate of AUC and the AUC as evaluated in the validation set with true outcomes observed. Across the misclassification scenarios examined, the proposed estimator showed negligible bias that was smaller in magnitude than the AUC estimate calculated with misclassified outcomes. For example, in the nondifferential unidirectional misclassifications scenario with $P(Y=1|T=1)=0.9$ (scenario [a]), the proposed method had an average bias(MCSE) of $0.00096(0.0002)$, while estimating AUC with misclassified outcomes showed an average bias of $0.0041(0.00003)$. For all scenarios, MCSEs were sufficiently small in magnitude to support making conclusions about comparative properties of the estimation methods. MCSEs for all evaluations are presented in the Supporting Information. The magnitude and direction of bias for ${\widehat{\text{AUC}}}_{\text{mis}}$ follows the behavior outlined in Section 2.3 for the unidirectional misclassification scenarios (scenarios a–f). The directions of bias for ${\widehat{\text{TPR}}}_{\text{mis}}$ and ${\widehat{\text{FPR}}}_{\text{mis}}$ are also as expected; tables detailing TPR and FPR estimators separately are in Supporting Information: Table S2‐S5. In the bidirectional misclassification scenarios examined (scenarios g–l), ${\widehat{\text{AUC}}}_{\text{mis}}$ underestimated the true AUC in the validation set, and the magnitude of the bias was larger than in the unidirectional examples. Recall, the data generation process for bidirectional misclassification used the same misclassification rates as the unidirectional scenarios for misclassifying events and added misclassification of nonevents as well. As expected, the estimate of AUC calculated with true outcomes identified in the gold‐standard set was unbiased in all scenarios.

FIGURE 2.

Absolute bias of AUC estimates under different misclassification scenarios, event $\text{rate}=0.1$, $N=50000$, and $n=1000$. Scenario labels and descriptions are provided in Table 1. For nondifferential cases ([a–c] and [g–i]), the x‐axis corresponds to worsening misclassification. For differential cases ([d–f] and [j–l]), the direction of association between risk and misclassification rates switches for the farthest right scenario. Shapes and colors denote the estimator using potentially misclassified outcomes (${\widehat{\text{AUC}}}_{\text{mis}}$, circles), the estimator using only chart reviewed data (${\widehat{\text{AUC}}}_n$, triangles) and our proposed estimator accounting for misclassification (${\widehat{\text{AUC}}}_{\text{proposed}}$, squares).

FIGURE 3.

The coverage of 95% CIs for AUC under different misclassification scenarios, event rate = 0.1, $N=50000$, and $n=1000$. Scenario labels and descriptions are provided in Table 1. For nondifferential cases ([a–c] and [g–i]), the x‐axis corresponds to worsening misclassification. For differential cases ([d–f] and [j–l]), the direction of association between risk and misclassification rates switches for the farthest right scenario. Shapes and colors denote the estimator using potentially misclassified outcomes (${\widehat{\text{AUC}}}_{\text{mis}}$, circles), the estimator using only chart reviewed data (${\widehat{\text{AUC}}}_n$, triangles) and our proposed estimator accounting for misclassification (${\widehat{\text{AUC}}}_{\text{proposed}}$, squares).

FIGURE 4.

Average 95% CI width under different misclassification scenarios, event rate=0.1, $N=50000$, and $n=1000$. For nondifferential cases ([a–c] and [g–i]), the x‐axis corresponding to worsening misclassification. Scenario labels and descriptions are provided in Table 1. For differential cases([d–f] and [j–l]), the direction of association between risk and misclassification rates switches for the farthest right scenario. Shapes and colors denote the estimator using potentially misclassified outcomes (${\widehat{\text{AUC}}}_{\text{mis}}$, circles), the estimator using only chart reviewed data (${\widehat{\text{AUC}}}_n$, triangles) and our proposed estimator accounting for misclassification (${\widehat{\text{AUC}}}_{\text{proposed}}$, squares).

Figure 3 displays the coverage, defined here as the percent of simulations for which each estimator's 95% CI contained the AUC of the prediction model as evaluated in the validation set using true observations. Coverage of 95% CIs for the proposed and gold‐standard estimators was close to 95% while coverage for the estimators calculated with misclassified outcomes was nearly 0 for all misclassification settings except for (a).

Figure 4 presents the average width of 95% CIs for each AUC estimator across the 1000 simulations. In all scenarios, the interval estimates of ${\widehat{\text{AUC}}}_{\text{proposed}}$ are wider than ${\widehat{\text{AUC}}}_{\text{mis}}$; this is expected, as the proposed method accounts for the additional uncertainty introduced by ${ĝ}_n$, the model approximating T, in the chart review set when constructing intervals. Despite this additional variability, we see that 95% CIs for the proposed estimator are narrower than that of the AUC estimate obtained in the smaller chart review set, indicating that evaluating performance in the larger validation set did improve precision under these simulation settings. As seen in Figure 3, the coverage of 95% CIs for ${\widehat{\text{AUC}}}_{\text{mis}}$ is poor; this is due to the considerable bias in estimates as well as the narrower CI width. Our ${\widehat{\text{AUC}}}_{\text{proposed}}$ has appropriate coverage due to minimal bias and an appropriate 95% CI width that reflects uncertainty.

We present evaluation metrics for PPV and NPV, including assessed bias, MSE, coverage, and 95% CI width, in Tables S6–S9. Comparisons between estimation methods are consistent to those described above for AUC. Bias in PPV and NPV estimates obtained using misclassified outcomes follow the behavior described in 2.3.

We present additional results in the Supporting Information, including bias, MSE, coverage, and 95% CI width for AUC and other performance measures in settings with 5% and 20% event rates (Table S10–S18), chart review set size of $n=500$ (Tables S19–S27), and training and validation set sizes of $N=25000$ (Tables S28–S36). Conclusions about the behavior of each estimator at these different event rates and sample sizes are consistent with those presented and discussed above. Notably, as the event rate increases, the absolute bias of PPV and NPV estimators calculated with misclassified outcomes is amplified due to the influence event rate has on PPV (independent of the discriminatory ability of the model). As expected, variability of all performance measure estimates was greater for all methods at smaller values of $N$ and $n$. All estimated MSEs and constructed 95% CIs accurately reflected this increased sampling variability. At the smallest chart review size examined ($n=500$), 95% CI width was still narrower when estimating performance with the proposed method than when using only the chart review set with gold‐standard outcome measures alone.

6. Discussion

Accurately estimating a prediction model's performance is important as it informs decision‐making about whether and how to use the model in clinical care [39, 40]. In the presence of outcome misclassification, validation estimates obtained by using the observed outcomes quantify how well a model predicts those misclassified outcomes, which, as we show, may not reflect predictive validity for true outcomes and may mislead decision‐makers. Our proposed method, which uses a small chart reviewed sample to train a model predicting the true outcome given the observed outcome and risk factors, improved the evaluation of predictive performance across various outcome misclassification scenarios examined. Estimates based on this approach showed small bias and confidence intervals with close to nominal coverage (86%–97% for AUC estimates).

The method we proposed is applicable across any type of prediction modeling strategy and computationally efficient to implement. The key idea is leveraging a chart review set to develop a model $g$ for the relationship between the true outcome status, which we do not have access to in our validation set, and the potentially misclassified outcomes and the features of our observations, which we do have full access to in our validation set. With this relationship model, one can obtain ${ĝ}_n{({\mathbf{\text{X}}}_i,Y_i)}_{i=1}^N$ for all the validation observations and use this approximation as a substitute for the true outcome status instead of naively relying on the misclassified outcomes to calculate performance estimates. This approach elicits information on the misclassification rate from a representative chart review sample rather than requiring previous knowledge of those rates. One can fit any appropriate model to the chart review set for deriving the proxy estimates. Uncertainty in estimating $g$ can be accounted for using standard bootstrapping methods; the resulting confidence intervals around proposed estimates provide better coverage than those based on the naive estimates. Rather than using a bootstrap, one could instead use imputation to account for uncertainty; our procedure can be viewed as a single imputation using $g$. Multiple imputation could proceed as follows: for each $m$ in $1,\dots ,M$ imputation rounds, impute ${\tilde{T}}_{m,i}=\text{Bern}({ĝ}_n({\mathbf{\text{X}}}_i,Y_i))$, then estimate TPR, FPR, or AUC using ${\tilde{T}}_{m,i}$ in place of ${ĝ}_n({\mathbf{\text{X}}}_i,Y_i)$ in our proposed estimators. Then Rubin's rules [41] can be used for variance estimation: the within‐imputation variance can be estimated using the asymptotic variance of the corresponding estimator [42, 43, 44, 45], while the across‐imputation variance can be estimated using the standard estimator. The adjustment method we proposed can be applied to any prediction model, regardless of whether it was developed with or without misclassified outcomes, since our objective is accurately validating a given prediction model in a validation set of interest. The proposed method can also be applied across all prediction modeling approaches, including parametric and nonparametric models, as the approach does not require any knowledge of or adjustment to how the prediction model was developed.

As shown in Section 3, the targets of our proposed estimates are not always guaranteed to be the targets of interest (i.e., the true TPR, FPR, PPV, or AUC). We provided a sufficient condition under which our proposed estimates will be asymptotically unbiased: $||{ĝ}_n(\mathbf{\text{X}},Y)-𝔼[T|X,Y]|{|}_{\infty }\stackrel{p}{\to }0$ as $n\to \infty$. However, this condition may not typically be met and cannot be verified in the available data. When this assumption does not hold, the method we proposed may be asymptotically biased because the proposed estimator is not consistent for the target. However, nonconsistency does not mean non‐improvement. This improvement is demonstrated in our simulations. The data‐generating approach used in our plasmode simulations did not include specifying a model $g(X,Y)=P(T=1|X,Y)$ to generate $T$; instead, we sampled true outcomes $T$ from existing data and specified a misclassification probability. As such, the ${ĝ}_n(\mathbf{\text{X}},Y)$ fitted from the chart review set in our simulations is likely misspecified. We found that the proposed estimates have less bias and better coverage than estimates of performance calculated with misclassified outcomes, and this improvement holds across various misclassification scenarios. And, though our proposed estimator is not guaranteed to be asymptotically unbiased for the target, the confidence intervals for the proposed estimators were more efficient than estimates obtained using the chart review set alone while maintaining expected coverage. Even when inconsistent, ${ĝ}_n(\mathbf{X},Y)$ may approximate $E(T|\mathbf{X},Y)$ sufficiently to better accommodate binary outcome misclassification. The next steps of this research include theoretical development to more clearly delineate the scenarios in which the proposed method reduces bias and designing approaches for examining the method's assumptions in a dataset.

The proposed method was demonstrated with a relatively large chart review sample randomly drawn from the same population as the validation set. Our simulation study used a chart review sample of 1000 observations, which is not typically practical due to the cost and time required for expert annotation. Decreasing the size of the chart review sample will increase the variability in modeling $𝔼[T|\mathbf{\text{X}},Y]$ and, in turn, the MSE of the proposed performance estimates. In practice, chart review samples are relatively small. Chart review samples may also not be representative of the target population, as researchers typically attempt to optimize chart review resources by oversampling observations with higher risk scores or with certain features predictive of the outcome or misclassification risk [19, 46]. Also, when the outcome of interest is a rare event, a small random sample may not contain enough events to be informative. The ideal method would enable accurate estimation of performance measures in the larger population of interest, even with more efficient stratified sampling of a smaller chart review set. To address this limitation, further research is needed to extend the proposed method so that it is robust under distribution shift, that is, when the distribution of $X$ and $Y$ varies between the chart review and validation sets. This is especially common in the context of data fusion, where, for example, chart review data and validation data may be collected from multiple sources. Exploration of sampling strategies, including two‐phase designs, may uncover more effective approaches for selecting chart review samples [47, 48].

This paper focused on improving estimates of prediction model discrimination and classification accuracy and did not consider measures of calibration. Chart review samples with gold‐standard outcome determinations could also be leveraged to improve prediction calibration in the presence of outcome misclassification. As with the performance measures explored here, outcome misclassification will bias model calibration when observed (misclassified) event rates are not similar to true event rates. The methods presented here could potentially be extended to accommodate binary outcome misclassification for calibration measures like Brier score, calibration‐in‐the‐large, and regression‐based intercept and slope calibration estimates.

The practical impact of outcome classification on prediction model validation was not severe in all scenarios examined. For example, in the unidirectional‐nondifferential setting where 10% of true events were not captured, the average bias in AUC was less than 0.01. Such minor biases in performance measures may not be clinically relevant, and in these cases, the additional time and resources required to collect chart review data and obtain adjusted estimates could be avoided. Health providers can leverage their expertise on the likely degree and magnitude of misclassification to determine whether the additional time and resources required to collect chart review data and obtain proposed estimates are necessary. The simulation framework we presented here could be used to explore how robust performance validation with potentially misclassified outcomes may be across reasonable misclassification scenarios.

Our method focuses on evaluating the performance of any given prediction model in a validation set. There is also a need for methods to improve prediction model estimation when outcomes are misclassified. Zawistowski et al. [29] proposed a method that simultaneously adjusts for outcome misclassification in model estimation and validation. However, their method relies on a likelihood‐based prediction model, which is not applicable to machine learning prediction models. Further research in this area would improve the performance of clinical prediction models estimated with real‐world data prone to outcome misclassification.

7. Conclusion

Our proposed method utilizes a chart review sample with gold‐standard outcomes to accommodate binary outcome misclassification when evaluating a prediction model's performance. Under various misclassification scenarios examined, empirically, our proposed estimators reduced bias compared to naively evaluating the prediction model with misclassified outcomes and provided reliable uncertainty quantification with satisfying efficiency. The resulting point and interval estimates for a prediction model's performance provide information about the targets of interest—how well a model predicts true (if unobserved) outcomes—and can guide decision‐making about whether and how to use the prediction model.

Funding

This work was supported by the National Institute of Mental Health (Grant Nos. R01‐MH125821, U19‐MH099201, and U19‐MH121738).

Conflicts of Interest

Dr. Shortreed has worked on grants awarded to Kaiser Permanente Washington Health Research Institute (KPWHRI) by Bristol Meyers Squibb and by Pfizer. She was also a co‐investigator on grants awarded to KPWHRI from Syneos Health, who represented a consortium of pharmaceutical companies carrying out FDA‐mandated studies regarding the safety of extended‐release opioids.

Supporting information

Data S1. Supporting Information.

Data Availability Statement

The dataset used in this study is not publicly available, as it contains detailed information from patient clinical records data.