Use of the Area Under the Precision-Recall Curve to Evaluate Prediction Models of Rare Critical Illness Events

In the critical care setting, where missed diagnoses and false alarms can have significant consequences, evaluating the performance of a classification (or binary prediction) model must be nuanced, critical, and thorough. Traditionally, the area under the receiver operating characteristic (AUROC) curve has been used as the primary evaluation metric for most models. The receiver operating characteristic (ROC) curve is a plot of a model’s sensitivity (also known as recall or true positive rate) and specificity (true negative rate) at each possible model output threshold (typically a score or probability). The AUROC, thus, provides a global evaluation of a model’s ability to discriminate between the two outcome classes by summarizing the sensitivity–specificity tradeoff over all model output thresholds.

However, AUROC may be misinterpreted clinically when the outcome of interest is rare. Such scenarios are referred to as “imbalanced” because of the large ratio of negative to positive (or vice versa) cases. In these situations, a model may have a high AUROC as a result of the robust true negative rate (specificity) (1, 2) yet be unable to reliably identify positive cases (Fig. 1). Accordingly, AUROC may not give clinicians the insights they seek when deciding whether a classification model will be clinically useful (3).

Figure 1.

Example impact of outcome class imbalance on model predictions. This figure shows how application of model predictions to balanced (A) and imbalanced (B) outcome sets can affect the distribution of true positive (TP) and false positive (FP) and false negative (FN) predictions. While the true negative (TN) rate (specificity) remains high in the imbalanced set, the ratio of FP predictions to TP predictions has increased leading to a low positive predictive value and a potentially less useful model.

Many events of interest in the critical care setting, including mortality, clinical deterioration, and acute kidney injury, are imbalanced in our datasets because only a minority of our patients suffer these events (< 10–20%). In such situations, the precision-recall (PR) curve and area under the PR curve (AUPRC) offer more clinically relevant and operationally useful measures of performance. The sensitivity (recall) of a model indicates the proportion of all positive cases the model can identify. The positive predictive value (PPV or precision) is the proportion of positive predictions that are correct. Both are intuitive concepts for clinicians. By measuring the tradeoff between sensitivity and PPV, AUPRC aligns better with what we care about in imbalanced problems: reliable identification of a rare event. Further, PR curves can illustrate the operational implications of model deployment in critical care settings. Here, we demonstrate the utility of AUPRC and PR curves when predicting rare critical illness events using simulated pediatric data.

A PREDICTION MODEL EXAMPLE USING SIMULATED CRITICAL CARE DATA

We synthesized clinical data for 200,000 virtual pediatric patients presenting with diabetic ketoacidosis and built prediction models for the outcome of cerebral edema (CE). Input variables, distributions, and CE prevalence were chosen to reflect observational studies (4–6). Virtual patients were randomly divided into training (80%) and test (20%) sets, and logistic regression (LR), random forest (RF), and extreme gradient boosting (XGBoost) models were trained to predict CE. We calculated AUROC and AUPRC and plotted ROC and PR curves using the pROC and PRROC packages (which use piecewise trapezoidal integration to determine area under the curve). AUROC and AUPRC comparisons and 95% CIs were computed using bootstrapping methods. The R software (Version 4.4.1; Vienna, Austria) we used is described in the Supplement (http://links.lww.com/PCC/C622).

The AUROCs for the LR, RF, and XGBoost models were 0.953 (95% CI, 0.939–0.964), 0.874 (0.851–0.897), and 0.947 (0.939–0.964), respectively, while AUPRCs were 0.116 (0.095–0.142), 0.083 (0.068–0.102), and 0.096 (0.082–0.112), respectively (Fig. 2). Importantly, interpretation of AUPRC and each model’s value is best done in relation to the frequency of the event of interest (specifically, by dividing AUPRC by the frequency of that event).

Figure 2.

Receiver operator characteristic (ROC) and precision-recall (PR) curves for simulation prediction models. This figure shows ROC (A) and PR (B) curves and associated area under the ROC curve (AUROC) and area under the PR curve (AUPRC) values (with 95% CIs) for three models (logistic regression, random forest, and extreme gradient boosting [XGBoost]) predicting cerebral edema in the simulated population test set. We have reversed the X-axis in A, which is more commonly shown as 1–specificity. PPV = positive predictive value.

Dividing the LR model AUPRC (0.116) by the CE outcome frequency (0.007), we find the LR model is 16.6-times more useful than a random model. Additionally, as observed in the PR curve in Figure 2B, a model threshold chosen to achieve a sensitivity of 0.85–0.90 would potentially result in a PPV (precision) 5–10% higher for the LR and XGBoost models than the RF model. None of this is readily apparent in the ROC curve (Fig. 2A). Finally, although the LR and XGBoost models had similar AUROC, the LR model AUPRC was statistically significantly higher, primarily as a result of its improved PPV at lower sensitivities. In such a scenario, sole use of AUROC and ROC curves for model selection could lead to selection of a model with a PPV-sensitivity tradeoff that does not provide the clinical utility required by the clinical team.

CAPTURING WHAT MATTERS: SPECIFICITY VS. POSITIVE PREDICTIVE VALUE

High sensitivity is important for prediction of critical illness events, including CE, because failing to identify events could have severe consequences for patients. As both AUROC and AUPRC use sensitivity in their measures, the main difference between AUROC and AUPRC is use of specificity vs. PPV, respectively. In imbalanced datasets, where negative cases are abundant, this difference becomes consequential (2).

When the number of true negatives vastly exceeds true positives, a model can have a high specificity (and AUROC) by labeling all cases as negative, but this would have no clinical value. A model that labels most cases as negative may have limited ability to detect the rarer positive cases. In our simulation, all three models have excellent AUROC over 0.85. However, in an application where a sensitivity of 50% is deemed necessary, the resulting PPV will be less than 0.2, indicating less than one in five positive predictions will be correct. While this tradeoff may still be acceptable to the clinician, it is a much more sober assessment of clinical utility than AUROC suggests.

Furthermore, a small change in model performance that converts a true positive into a false negative has only a small effect on specificity in imbalanced problems due to the overwhelming number of true negatives that partially mask this drop in performance. PPV will change more markedly in this situation, offering a more transparent measure of performance when some data points have high leverage. As illustrated in Figure 1, a “miss” by the model (a change from a true positive to a false negative) has an immediate, noticeable impact on PPV, without masking from true negatives.

OPERATIONAL RELEVANCE OF AUPRC IN CRITICAL CARE SETTINGS

When deploying a model to detect or predict critical illness events, the primary goals are often two-fold: minimizing the number of missed positive cases (high sensitivity) and ensuring clinicians are not overwhelmed by false positive alerts. A high false positive rate diminishes clinicians’ trust in an alert and increases the time to respond (7).

The PR curve illustrates these priorities effectively allowing one to gauge, at different sensitivities, how well the model maximizes PPV and minimizes the “number needed to alert” (NNA), defined as 1/PPV (8). The lower the PPV, the higher the number of false positive predictions and larger the NNA. Both PPV and NNA are intuitive operational metrics for clinicians. NNA reflects the burden of false positives expected on the clinical teams responding to the alerts for each correct prediction, which is crucial in balancing the cost of alert fatigue with the benefit of accurate predictions.

By illustrating what PPV is attainable at different sensitivities, clinicians can judge whether a model’s performance aligns with an acceptable threshold for operational use. Returning to our simulation, LR model PPV is greater than 0.25 only at low (< 0.05) sensitivities. Conversely, at sensitivities near 0.90, PPV ranges from 0.15 to 0.2. By examining this curve, the clinician can determine the impact of prioritizing PPV vs. sensitivity and consider the operational implications through discussion of the NNA. A clinician may decide, for example, that a threshold that provides a sensitivity of 0.90 and NNA of 6–7 is or is not acceptable depending on the actions involved (e.g., high-risk therapy vs. more intense monitoring).

In summary, inspection of PR curves allows the clinician to evaluate if a prediction model is appropriate for clinical use and enables selection of a probability threshold suitable for the prediction task and clinical environment. Although inspection of the ROC curve remains a key component of predictive model evaluation, it does not provide the same operational insights as the PR curve because the tradeoff between sensitivity and 1-specificity can be more challenging to translate into real-world clinical scenarios.

While AUPRC is advantageous for evaluating models on imbalanced datasets, it is important to recognize the potential challenges. AUPRC can be sensitive to small changes in precision and recall, especially when positive cases are extremely rare. Therefore, it is important to complement AUPRC with visual inspection of PR curves to ensure model and threshold selection match the priorities of the prediction task. Further, it is important to note that a model that consistently outperforms others across all thresholds in ROC space will also outperform these models in PR space.

CONCLUSIONS

AUPRC and the PR curve are informative, practical metrics that should be considered when assessing prediction models for rare clinical events. By prioritizing PPV over specificity, AUPRC aligns with the operational priorities of critical care clinicians, offering a clearer picture of model performance and insight on interpreting positive predictions. While AUROC is an important metric and should be used as one of several measures of discrimination, we recommend investigators working with imbalanced prediction problems in critical care use AUPRC and the PR curve to determine how a model will perform when applied clinically at various thresholds, considering the tradeoff between sensitivity and PPV.