Continuous Evaluation Frameworks for Retrospective Evaluation of Clinical Machine Learning Models

Abstract

Abstract

The performance of clinical machine learning (ML) models that continuously predict the risk of a future event is typically evaluated using metrics at a single time point. While these metrics are widely accepted to evaluate the performance of ML models, they are not comprehensive enough to capture the performance and behavior of the ML models when executing continuously in real-world settings. Additionally, these single time point metrics assess model discriminability, which is different from clinical utility. In this paper, we propose three novel frameworks to continuously evaluate clinical ML models on retrospective datasets: (1) Zone-based (2) Predictive Activity Monitoring Characteristic curve-based and (3) Notification-based. We demonstrate the value of these three continuous evaluation frameworks using predictions from sepsis models evaluated on two publicly available datasets. The three proposed frameworks provide a more comprehensive evaluation and comparison of different sepsis prediction models and reveal additional aspects of model performance beyond standard metrics computed at a single time point. These frameworks provide users of clinical ML models a more realistic understanding of the model’s utility.

Background

Clinical machine learning (ML) models are increasingly used to predict patient risk continuously for future adverse events, such as sepsis. Traditionally, these models are evaluated using performance metrics computed at a single timepoint. While these metrics—such as Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision Recall curve ( AUPRC)—are widely accepted, they fail to capture how a model behaves when operating continuously in real-world clinical environments. Moreover, single time point evaluations primarily assess discriminability, which may not directly reflect a model’s clinical utility.

Methods

We propose and implement three novel retrospective evaluation frameworks to continuously assess clinical ML models: zone-based, predictive activity monitoring characteristic curve (AMOC)-based and notification-based. These frameworks were applied to sepsis prediction models using two publicly available clinical datasets. Model outputs were continuously monitored, and results were compared against traditional single time point metrics to assess differences in performance.

Results

The three continuous evaluation frameworks reveal multiple dimensions of model performance not captured by conventional metrics. Across both datasets, models with high AUROC demonstrate markedly different performance when continuously evaluated.

Conclusions

The proposed zone-based, predictive AMOC-based, and notification-based frameworks offer a more comprehensive and realistic assessment of continuously operating clinical ML models. By uncovering temporal and behavioral aspects of model performance, these methods enable more informed model selection, deployment, and monitoring in healthcare settings. Implementing continuous evaluation can enhance trust, transparency, and clinical applicability of ML systems beyond what single time point metrics provide.

Trial Registration

Not Applicable

Introduction

The use of machine learning (ML) in healthcare is rapidly expanding. ML models are used in various branches of medicine such as radiology, cardiology, intensive care. In this paper, we focus on clinical ML models which are developed to continuously provide a predicted risk of future events or conditions at the bedside. Examples include early warning scores [1], patient deterioration index [2], risk of hypotension [3], and sepsis prediction [4].

Models intended to continuously provide risk-score predictions at the bedside are typically evaluated on retrospective data available at single time point, using metrics such as area under receiver operating curve (AUROC), sensitivity, specificity, precision, area under precision recall curve (AUPRC). These metrics are easy to implement and interpret and can be used to compare the performance of multiple models and their generalizability over different datasets. These metrics are popular and are widely reported in literature [3, 5, 6]. However, at time of deployment, these ML models continuously produce a score associated with probability of a future event, and classical metrics (e.g., AUROC) calculated at a single point ahead of events fail to adequately summarize the performance of these continuous predictions over time. In a recent example, the Epic sepsis model (ESM), a continuous sepsis prediction algorithm needed to be withdrawn and redeveloped at least partially due to low recall and high false alarm burden produced by the model despite having an acceptable AUROC [7, 8].

Continuous performance evaluation frameworks for clinical ML models are essential for assessing model efficacy and utility in dynamic medical environments. Continuous evaluation metrics for ML models are critical to establish anticipated model behavior and clinical utility, including lead time provided by the ML model, alarming rates (both true alarms and false alarms over time), and prediction patterns over time (e.g., risk score evolution and model performance within X hours vs. Y hours before event/condition). Continuous evaluation metrics could provide grounds to support that the model will provide actionable alarms or reveal model insufficiencies prior to embedding it in clinical workflows. Evaluation of continuous prediction models must be rigorous to ensure that the tool aids decision making, rather than creating additional burden for the clinician at the cost of decreased patient safety risk. These evaluation frameworks can be used to evaluate model performance post-deployment using prospectively collected data.

A key challenge in evaluating ML models continuously is the occurrence of event of interest. In a continuous setting, the ML model produces multiple predictions over a time period which may or may not be followed by an event of interest. The one-to-one mapping of prediction to event of interest (which are used to produce metrics at a single time point evaluation) are unavailable. Different strategies can be used to solve this problem. In the PhysioNet/Computing in Cardiology Challenge 2019 [4], the organizers proposed a utility score-based evaluation metric to assess continuous sepsis prediction. The utility score rewarded correct early predictions while penalizing missed predictions and false alarms. Another approach was proposed by authors performing an external validation of the ESM. In addition to the standard metrics, they calculated number of patients needed to make a correct positive prediction and time-horizon based PPV which evaluated every threshold crossing. In [9], the authors proposed lead time analysis for continuous evaluation of ML models. Lead time was defined as the time between the earliest positive prediction and the event such that > 80% of the predictions are positive. None of these approaches evaluated the controls and therefore the ability of the model to correctly identify a non-event was not evaluated. Lastly, regulatory agencies are increasingly identifying the need for continuous evaluation frameworks and metrics for comprehensive model evaluation. Hence this work aims to fill this gap by providing clinical model developers tools to evaluate models continuously on retrospective datasets.

In this paper, we propose three novel continuous evaluation frameworks (CEF) designed specifically to evaluate ML models continuously in retrospective datasets. We describe three approaches to aggregate multiple predictions for the same target event of interest. Our contributions can be summarized as follows:

1. Three continuous evaluation frameworks and metrics that can be applied to both case and control populations.

2. Parameterized continuous evaluation frameworks that can implement prediction window for actionable alarms for a multitude of clinical problems.

3. Relevant metrics to assess clinical utility and impact on clinical workflow.

We demonstrate the use of these frameworks in the domain of sepsis prediction. We use two publicly available datasets. The first is the Emory dataset from the 2019 Physionet challenge with a ∼5.7% prevalence of sepsis [4]. The second is the Medical Information Mart for Intensive Care IV (MIMIC-IV) v3.1, a large deidentified dataset of patients admitted to the intensive care unit at the BIDMC with a ∼9.4% prevalence of sepsis [10].

We utilize two sepsis prediction models: LGBM and SIRS. The LGBM is a light gradient boosted tree regression model which was the winner of the 2019 Physionet sepsis prediction challenge [11]. Additionally, we trained a gradient boosted tree (XGBoost) model to mimic the systemic inflammatory response syndrome (SIRS) score. We evaluate the continuous performance of the two sepsis models using our proposed CEFs on the two retrospective datasets. Through these evaluations, we aim to establish the utility of our proposed frameworks.

Methods

In this section, we propose three CEFs to evaluate ML models on retrospective datasets. We define patient unit stays with event of interest (e.g., sepsis) as cases and patient unit stays without event of interest as controls. Our train-test setup is as follows: we train the sepsis prediction models on the train partition of the datasets. The trained models are used to produce event risk scores, periodically (e.g., every hour), for the patient stays in the test partition. We convert these trajectories of event risk scores into trajectories of positive/negative predictions. A prediction is positive when the event risk score exceeds a chosen risk threshold, otherwise it is a negative prediction.

Zone-based CEF

In the zone-based CEF, we partition a patient unit stay in the hospital into time zones, compute scores per time zone and aggregate scores to get a performance metric. There is a total of five steps in the zone-based CEF, which we describe below,

I. Partition patient’s unit stay: We first partition a patient’s unit stay into time zones. The rationale for partitioning a stay into time zones is two-fold: one, zoning makes it possible to capture clinical relevance and domain knowledge e.g., time zones before onset of sepsis are more clinically relevant; two, by defining time zones, we control the contribution of individual scores (described next) from these time zones to the final score. We define time zones relative to the event onset for cases (e.g., sepsis onset). We define time zone A for A hours as the time period immediately preceding the event of interest since it is critical and clinically relevant. We expect the ML model to make many positive predictions in this zone. We define time zone B for B hours as the time period immediately preceding time zone A since we expect the ML model to make at least a few positive predictions in this zone. Lastly, we define time zone C as all other time periods in cases. For controls the entire patient unit stay is labeled as time zone C. In Fig. 1, we show the time zones for a case and control.

Fig. 1.

Partitioning patient unit stay into time zones. The case patient stay has event (e.g., sepsis) between time t_onset and unit discharge. We show the corresponding 2 × 2 confusion matrix for each zone. For control patient stay, the complete unit stay is labeled as time zone C

II. Assign scores for predictions: We assign scores for predictions in each time zone. This framework supports multiple periodic and aperiodic predictions per time zone e.g., predictions are made in 15-minute intervals. Like shown in the confusion matrices in Fig. 1, in time zones A and B, a positive prediction will result in a true positive (TP), and a negative prediction will result in a false negative (FN). In time zone C, a positive prediction will result in a false positive (FP), and a negative prediction will result in a true negative (TN). We assign scores to each prediction, in each time zone, respectively (e.g., a TP in zone A gets a score of ‘+1’). The scores are defined to reward TP and TN while penalizing FP and FN. The assignment of scores in each time zone is problem-specific and can be configured.

III. Compute cumulative score per patient unit stay: We compute a cumulative score for each patient unit stay. For each prediction in each time zone, we get the score assigned in step II. We sum the individual scores over the entire patient unit stay to get a cumulative score.

IV. Compute best possible score per patient unit stay: We compute the best possible score for each patient unit stay. The best possible score in each time zone is the score corresponding to the best possible outcome in each time zone, per prediction i.e., TP in time zones A/B, TN in time zone C. Hence, for a patient unit stay with multiple predictions in zones A, B and C, the best possible score would be the sum of scores corresponding to TPs in zone A, TPs in zone B and TNs in zone C. We add the scores over the entire patient unit stay to get the best possible score.

V. Compute mean continuous score over all patient unit stays: Patients with long unit stays result in larger cumulative scores. To offset this effect, we normalize the computed cumulative score by dividing it by the best possible score. We compute the continuous score per patient unit stay as the ratio of the cumulative score (step III) divided by the best possible score (step IV). The continuous score can take any value between the two extremes of ‘−1’ to ‘+1’. A patient receives a score of ‘+1’ if the cumulative score for a patient unit stay matches the best possible score. A patient receives a score of ‘−1’ if the individual scores for a patient unit stay are all ‘−1’ e.g., all FP for a control patient. Any other combination of scores will result in a continuous score between these two extremes. Lastly, we compute the mean over all patient unit stay continuous scores to get a mean continuous score per dataset.

Zone-based CEF Settings for the problem of Sepsis Prediction

For the problem of sepsis prediction, we set the duration of time zones A, B to be A = 12, B = 12 h, respectively. Multiple clinical ML models developed to predict sepsis has shown the utility of predicting sepsis 12 to 48 h prior to onset of sepsis [4, 12–15]. Hence setting A and B to be 12 h each will result in predicting sepsis onset 24 h early, which is well within the ranges explored from prior works. Our score assignment is as follows, in time zone A, we assign a score of ‘+1’ to TP, ‘−0.5’ to FN, in zone B, we assign a score of ‘+0.5’ to TP and ‘0’ to FN, in zone C, we assign a score of ‘−0.05’ to FP and ‘+1’ to TN, respectively. The rationale for this score assignment is that the sepsis model is trained to predict the likelihood of sepsis and hence we expect it to capture more TP than FN in zones A/B. In time zone C, we wanted to penalize FP and yet be numerically distinct from TP score assignment. The number of time zones and their durations, A and B, are problem-specific and can be configured.

Predictive AMOC CEF

In this section, we describe the predictive Activity Monitoring Characteristic Curve (AMOC) CEF. The AMOC was originally introduced as a way to evaluate and compare disease outbreak detection systems [16]. The AMOC curve depicts the tradeoff between the speed of detection and the false alarm rate produced by a detection system. In detection settings, an AMOC curve demonstrates the tradeoff between the elapsed time between the event onset and detection of event (i.e., time to detection) and the false alarmrate, both of which are desired to be minimized. The AMOC curve which originally focused on evaluating detection algorithms [16], needs to be reframed to evaluate predictive ML models that provide continuous score-based predictions. Applied to prediction tasks, predictive AMOC curves depict tradeoff between anticipation time (elapsed time between a positive prediction and the event onset) and the false alarm rate. In detection setting, the detection time is desired to be minimized whereas in prediction settings, the anticipation time needs to be maximized. With this modification, predictive AMOC curves can be generated to evaluate and compare ML models that provide continuous predictions, and act as a guide to configure predictive algorithms to produce appropriate alarms in clinical settings.

First, we determine a range and resolution for a set of risk thresholds, Z, to sweep through. For example, sepsis models that produce probability of sepsis onset, the range and an example resolution would be [0,1] and 0.01, respectively.

Second, we sweep through the set of risk thresholds Z and compute anticipation time, or the earliest time a positive prediction occurred within a pre-specified window W hours prior to the event onset. A positive prediction is produced whenever r_t⁽ⁿ⁾ > z, z ∈ Z, where r_t⁽ⁿ⁾ represents the event risk score of the n^th patient stay at time stamp t. Note that the anticipation time is computed only for cases. For example, if n^th patient stay has sepsis at time t_sepsis⁽ⁿ⁾ and W = 12 h, the first positive prediction within W = 12 h prior to t_sepsis⁽ⁿ⁾ is taken as the anticipation time for the sepsis patient stay. Alternatively, W = ∞, considers the first positive prediction (without temporal window limitations) prior to t_sepsis⁽ⁿ⁾ as the anticipation time for the n^th sepsis patient stay. See Fig. 2 for an example of risk threshold crossings and anticipation time for W = ∞. For cases that do not have a positive prediction within W hours of event onset, we set the anticipation time to be zero, meaning the prediction model failed to make a positive prediction and hence the time available for clinicians to react is zero. A distribution of anticipation times for the events is available at each risk threshold Z and summarized using the median.

Fig. 2.

Sample case with ten individual risk threshold crossings and three notifications, all prior to onset of event (t_onset). The anticipation time is defined as the time between the first risk threshold crossing and t_onset

At each risk threshold, z, we also compute the false alarm rate. A false alarm occurs when any positive prediction occurs for a control patient stay or a positive prediction is produced outside of pre-specified window W hours prior to event onset for cases. For example, if W = 12 and a patient stay has sepsis at time t_sepsis⁽ⁿ⁾, any positive prediction that has occurred greater than 12 h prior to t_sepsis⁽ⁿ⁾ is a false alarm. We compute the false alarm rate by [1] adding any positive predictions for control stays and positive predictions outside the temporal window W for cases and [2] dividing this sum by the total number of opportunities available to make a false prediction. The false alarm rate produced by the ML model is available for each risk threshold in Z.

Lastly, we make the predictive AMOC curve by plotting the false alarm rate on the x-axis and the anticipation time on the y-axis. For each risk threshold in Z, a false alarm rate and anticipation time will generate a point along the predictive AMOC curve. For simplicity, we create predictive AMOC curves assuming that the clinician would respond to the very first threshold crossing.

Predictive AMOC CEF Settings for the problem of Sepsis Prediction

For the problem of sepsis prediction, we define risk thresholds Z as follows. The LGBM is a regression model that produces an event risk score that could be negative. We sweep through Z in the range of [min value, 0] in 1e − 4 resolution and [0, max value] in 0.1 resolution. We observed that we get more operating points on the AMOC curve when we sweep through Z at a finer resolution. The SIRS model is a probabilistic classifier hence we sweep through the probability scores between [0,1] in 0.0025 resolution. We experimented with four windows, W = [6, 12, 24, ∞] hours, where W = ∞ implies there is no temporal window and considers positive predictions occurring anytime between unit admission and sepsis onset. We chose W to be between 6 and 24 h based on prior works on sepsis prediction [4, 12–15].

Notification-based CEF

Clinical ML models will run continuously over time and make predictions at a fixed frequency (e.g., sepsis model makes hourly predictions). For a single patient stay, the generated event risk scores can repeatedly cross the risk threshold and generate multiple consecutive positive predictions. However, these multiple consecutive positive predictions may not be clinically relevant since they are typically muted after an initial risk threshold crossing. Therefore, when continuously evaluating model performance, we consolidate multiple consecutive positive predictions to form a single notification. See Fig. 2 for an example of individual risk threshold crossings and notifications.

In clinical practice, the model output is often communicated to the user in the form of notifications, which are used to alert staff about changes in patient status. By treating each notification as one sample for performance evaluation, the resulting evaluation metrics can better capture the real-world model performance over time. We define a notification to start when the event risk score first exceeds the risk threshold and end when the event risk score drops below the risk threshold.

Assuming the ideal lead time of early prediction for the clinical event is T hours. We take the T-hour time block immediately preceding the event onset and assign a positive label. We define this as a positive-label time block. The duration of the positive-label time block can be shorter than T hours if the event starts within the first T hours of unit stay. The time preceding the start of the positive-label time block is divided into consecutive T hour time blocks and they are assigned negative labels. These blocks are defined as the negative-label time blocks. For a control, the positive-label time block is absent due to the absence of event, and the entire unit stay is divided into consecutive T-hour negative-label time blocks. Figure 3 shows for a case (top) the positive-label time block (Time Block S1) and negative-label time blocks (Time Block S2, S3, S4) and for a control (bottom) negative-label time blocks (Time Block L1 to L5).

Fig. 3.

We partition a patient stay into time blocks of length ≤ T hours. For cases the time blocks are partitioned from event onset to unit admission and for controls they are partitioned from unit admission to discharge. For cases, the block prior to the event of interest is labeled ‘Time Block S1’. Note that few time blocks may be shorter than T hours (e.g., time blocks S4 and L5). We overlay the notifications over these time blocks. Notifications are also capped at a max of T hours. When notifications extend beyond T hours, they are divided into multiple adjacent notifications e.g., notifications I and II. The space between notifications is labeled as ‘No Notifications’ and they are capped at T hours as well

The notifications represent positive predictions. When the notification is on, i.e., the time stamp is within one of those notifications, the clinical staff (intended users of the prediction model) should expect the onset of the event within the next T hours. Conversely, when the notification is off, the users will not expect the onset of the event within the next T hours. Therefore, the time period in between notifications will form negative predictions. Given the above interpretation of a positive prediction (notification) and negative prediction (no notifications), we divide the notifications longer than T hours into consecutive T-hour notifications. We refer to each ≤ T hour notification block as positive-prediction time block. Similarly, those negative predictions (time between two notifications) longer than T hours are also truncated into consecutive T-hour blocks and is referred to as negative-prediction time block. See Fig. 3 for an example of positive-prediction time block (Notification I, II, III for the case) and negative-prediction time blocks (No Notification blocks).

Given the positive-label time block, negative-label time block, positive-prediction time block, and negative-prediction time block, we can now define continuous-time metrics. Continuous-time Recall (C-Recall) is defined as among all events, the fraction having positive-prediction time blocks overlapping with the positive-label time block. Continuous-time Precision (C-Precision) is defined as among all positive-prediction time blocks, the fraction having event onset within the next T hours. Continuous-time Specificity (C-Specificity) is defined as among all negative-label time blocks, the fraction of those do not overlap with positive-prediction time blocks. Continuous-time negative predicted value (C-NPV) is defined as among all negative-prediction time blocks, the fraction of those do not overlap with positive-label time blocks.

Notification-based CEF Settings for the problem of Sepsis Prediction

For the problem of sepsis prediction, we set the length of time blocks T to be 6 h. For simplicity, we assume that both label blocks and prediction blocks are of the same length, T. But this can be set to different lengths for different problems of study.

Results

First, we present sepsis prediction dataset statistics, followed by the train-test setup for LGBM and SIRS models. Next, we report baseline metrics for evaluating clinical ML models, and finally, results from the three CEFs.

Sepsis onset in both datasets follows sepsis-3 guidelines. The Emory dataset comprises 20,000 patient stays, with 1,142 (5.7%) sepsis patients showing a median (IQR) onset time of 34.5 (10.25, 80) hours post-ICU admission. The MIMIC-IV dataset includes 58,694 ICU stays, with 5,531 (9.4%) sepsis patients having a median (IQR) onset time of 39 (14, 84) hours. Both datasets are split into training and testing sets using a 70%−30% stratified method. For more data statistics on clinical parameters, see Appendix Table 5.

We train the sepsis prediction models on the train partitions. For both the LGBM and SIRS models we need to pick a risk threshold to convert event risk scores into positive predictions. We follow the same steps as the sepsis challenge winner to pick a risk threshold [11]. For the LGBM regression model, the risk thresholds were − 0.01 and 0.01 for the Emory and MIMIC-IV datasets, respectively. For the SIRS model, the risk thresholds were 0.64 and 0.63 for the Emory and MIMIC-IV datasets, respectively. For details on the sepsis challenge, sepsis-3 guidelines, datasets, modeling specifics, and evaluation criteria, see Appendix A.

Baseline Continuous Evaluation Metrics

We compute single time slice AUROC – which is computed using sepsis event risk scores and labels. For cases, we use the sepsis event risk score available H hours prior to sepsis onset. For controls a random time stamp is chosen as pseudo-onset. We report single time slice AUROC for H between 1 and 6 hours prior to sepsis onset. We present AUROC from single time slice evaluations in Fig. 4. As expected, both models exhibit a decrease in AUROC as we move farther away from the event. Based on the performance of AUROC ≥ 0.8, up to three hours prior to event onset, the LGBM model is considered to have acceptable performance [17].

Fig. 4.

Single time slice AUROC for LGBM, SIRS models on Emory and MIMIC-IV datasets

We also report performance from Wong et al.‘s Epic sepsis model study [8], using standard metrics, e.g., AUROC and PPV. These metrics are at hospitalization level, with one sample per stay at the very first risk threshold crossing, assuming clinical response at that time. At time horizon level, each stay has multiple samples. A 12-hour horizon labels samples positive if sepsis occurs within 12 h otherwise negative. This results in non-independent samples and underrepresentation of sepsis due to many negative samples.

From Table 1, overall, the LGBM model is performing better than the SIRS model with higher AUROC, PPV, sensitivity and specificity at both levels. The AUROC for LGBM model at a time horizon level is 0.80, this metric is particularly problematic since it is computed on non-independent samples and is impacted by longer length of stays. Additionally, it does not reflect clinical practice since multiple, repeated positive predictions for the same sepsis event is not clinically useful. While the hospitalization level metrics are immune to these issues it does not differentiate between patients with short (< 24 h) and long (> 7 days) stays since both stays contribute equally.

Table 1.

Sepsis model performance when evaluated similar to the ESM paper [8]

Dataset	Model	Level	Sepsis incidence	AUROC	PPV	Sensitivity	Lead time (Median (IQR)); No. of patient stays
Emory	LGBM	Hospitalization level	5.73%	0.83	0.23	0.59	44 (10–88); 204
		Time horizon 12 h	1.99%	0.80	0.10	0.50
	SIRS	Hospitalization level	5.73%	0.74	0.12	0.69	35.5 (7–86); 236
		Time horizon 12 h	1.99%	0.75	0.06	0.44

Lead time and contributing patient counts are available only at the hospitalization level. In Table 1, metrics are computed at both the stay and hourly sample levels, complicating direct comparison. Still, the LGBM model is considered acceptable with a hospitalization-level AUROC > 0.8 [17]. Similar metrics for MIMIC-IV dataset are presented in the Appendix.

Zone-based CEF

We report the mean continuous score (MCS) in Table 2 for different patient groupings in the test partition. We compare the performance of the LGBM to the SIRS models in both datasets. The positive predictions from each model receive the same scores in each zone as described in Sect. "Zone-based CEF Settings for the problem of Sepsis Prediction". The best MCS is + 1 indicating all TP in time zones A, B and all TN in time zone C i.e. the predictions match the best possible outcomes. The MCS can take on negative values when predictions are directly opposite to the best possible outcomes.

Table 2.

Mean continuous score for LGBM and SIRS sepsis models

Dataset	Test partition patient grouping	# Test patients	LGBM	SIRS
Emory	All patients	6,000	0.92	0.86
	All Non-sepsis Patients	5,656	0.97	0.90
	All Sepsis Patients	344	0.17	0.24
	Sepsis Patients with Zones A, B, C (late sepsis onset)	199	0.46	0.54
	Sepsis Patients with Zones A, B (medium sepsis onset)	39	−0.24	−0.16
	Sepsis Patients with Zone A (early sepsis onset)	106	−0.21	−0.17
MIMIC-IV	All patients	17,609	0.88	0.84
	All Non-sepsis Patients	15,949	0.94	0.90
	All Sepsis Patients	1,660	0.33	0.29
	Sepsis Patients with Zones A, B, C (late sepsis onset)	1,037	0.53	0.54
	Sepsis Patients with Zones A, B (medium sepsis onset)	274	0.06	−0.02
	Sepsis Patients with Zone A (early sepsis onset)	349	−0.07	−0.21

In the Emory dataset’s test partition of 6,000 patients, the LGBM model achieves a MCS of 0.92, outperforming the SIRS model’s 0.86. For non-sepsis patients, the LGBM performs exceptionally well, with a MCS of 0.97 compared to SIRS. Recall these non-sepsis patients only have time zone C, with TNs receiving a score of ‘+1’ and FPs ‘−0.05’. For sepsis patients, the LGBM model performs poorly, with a MCS of 0.17 compared to SIRS. Results for sepsis patients are further categorized by early (t_sepsis ≤ 12 hours), medium (12 < t_sepsis ≤ 24 hours), and late (t_sepsis >24 hours) sepsis onset. This grouping will determine what time zones are available to compute MCS for each patient. For example, a patient with late sepsis onset will have contributions from all three zones when compared to a patient with early sepsis onset who has contributions only from zone A. For late onset patients, LGBM performs slightly worse than SIRS (0.46 vs. 0.54). LGBM also underperforms SIRS in medium and early onset groups. The negative MCS scores for these groups indicate that majority of the predictions are FNs, which contribute a score of ‘−0.5’.

The low MCS scores for early and medium sepsis onset patients, in the LGBM model relative to the SIRS model, is due to the over reliance on the input feature: number of hours in the ICU. In contrast, the SIRS model, which is largely physiology driven, does not take the number of hours in ICU into account when making positive predictions. Additionally, the LGBM model relies on input features which are missing more often for early and medium onset stays when compared to the smaller set of SIRS inputs that are more frequently available for the same cohort.

In the MIMIC-IV dataset, the LGBM model outperforms the SIRS model for sepsis patients, particularly in the early and medium onset cohort. This difference could be due to less missing data in LGBM inputs compared to SIRS and the threshold choice in the MIMIC-IV dataset.

The zone-based CEF makes it possible to compare clinical ML models in different time zones to determine the real utility and performance of these models when running continuously in real-world settings. At a dataset level, it appears that both models perform well with a high MCS score, similar to AUROCs reported in Fig. 4; Table 2. But, when we present the results for different patient groupings, we notice that the model performance is largely driven by non-sepsis patients and by patients with late sepsis onset. This inflated performance fallacy exists in many clinical ML models with large class imbalances. Classical metrics like AUROC cannot be applied to specific patient groupings (e.g., only on cases with early sepsis onset) with a single label. The zone-based continuous evaluation is one approach to reporting continuous performance on patient sub-populations with single labels.

Predictive AMOC CEF

Predictive AMOC curves produced for the LGBM and SIRS models for window size W = ∞ and W = 12 h are shown in Fig. 5. For a window size W = ∞, all positive predictions prior to sepsis onset are true alarms.

Fig. 5.

Predictive AMOC curves for LGBM and SIRS models using W = ∞ and W = 12 for both datasets

In the Emory dataset, for false alarm rates < 0.25 (i.e., 1 false alarm every 4 hours) and W = ∞, the LGBM model provides true alarms earlier (i.e., longer median anticipation time) than the SIRS model. For false alarm rates between 0.25 and 0.35 (i.e., 1 false alarm every 3 hours) and W = ∞, the LGBM and SIRS models provide similar (median) anticipation time. The SIRS model provided true alarms earlier than the LGBM model for false alarm rates > 0.35.

With a window size W = ∞, all alarms ahead of t_sepsis⁽ⁿ⁾ are deemed to be clinically actionable alarms associated with the future sepsis event. Alternatively, predictive AMOC curves can be recreated considering the case that alarms are only clinically actionable within a window W = 12 h prior to sepsis onset. This judgement is similar to [4], where only positive predictions within W = 12 h prior to t_sepsis⁽ⁿ⁾ were given a positive utility function value. For false alarm rates < 0.4 (i.e., 1 false alarm every 2.5 hours) and W = 12 h, the LGBM model provides true alarms earlier (i.e., longer median anticipation time) than the SIRS model. At false alarm rates ≥ 0.4 and W = 12, the LGBM and SIRS model provide equivalent median anticipation time.

For the MIMIC-IV dataset and W = ∞, the crossover happens much later at a false alarm rate of 0.5. The SIRS model had longer median anticipation times between false alarm rates of 0.5 to 0.75. For W = 12 h, we observe very similar trend as the Emory dataset. Results for W = 6 and W = 24 h can be found in Appendix A with similar interpretations.

Since alarms that occur > 12 h ahead of sepsis events are deemed false alarms for window W = 12 h, the anticipation time is lower for W = 12 vs. W = ∞ at a given false alarm rate. To illustrate, if we choose a false alarm rate of 1 per 16 h, in the Emory dataset the SIRS model provides median anticipation times of 0 h and 4 h for W = 12 and W = ∞, respectively, while the LGBM model provides a median anticipation time of 0.5 and 5 h, respectively. At that expected false alarm rate and a defined actionable time window of W = 12, the SIRS model fails to provide clinicians any lead time to reassess patients.

The median anticipation time for W = ∞ shown on the y-axis can be compared to the lead time shown in Table 1. In the MIMIC-IV dataset, for the LGBM model, off the 1,660 sepsis patient stays, 1,112 patient stays have a positive prediction (chosen threshold 0.01) with a median lead time of 27 h. In comparison, from Fig. 5B, at a threshold of 0 the median anticipation time is 12 h at a false alert rate of 0.1. This median anticipation time is much lower than the median lead time since for sepsis patients who did not have a single positive prediction (i.e. FN) the anticipation time is set to 0 h, thus lowering the median.

Notification-based CEF

In the Emory dataset, the total number of positive-label blocks and negative-label time blocks are 344 and 41,245 respectively. In MIMIC-IV dataset, the total number of positive label blocks and negative-label blocks are 1,660 and 170,450 respectively.

The continuous-time metrics for both datasets (Emory, MIMIC-IV) and both prediction models (LGBM, SIRS) are shown in Table 3. For LGBM on Emory, C-Recall is 0.54 means among all 344 sepsis patients, 54% of them would make the pager go off within the 6-hour window preceding sepsis onset; C-Precision is 0.07 means that if the pager goes off, the probability the patient will develop sepsis in the next 6 hours is 7%; C-Specificity is 0.89 means during the time periods the patient will not develop sepsis, within the next 6 h, there is a 89% probability that the pager will not fire a sepsis alert; C-NPV is 0.99 means during the time when the pager does not fire sepsis alert, there’s 99% probability that the patient will not develop sepsis within the next 6 hours.

Table 3.

We show C-Recall, C-Precision, C-Specificity, and C-NPV for both datasets (Emory or MIMIC-IV), both models (LGBM, SIRS)

Dataset	Model	Risk threshold	C-Recall	C-Precision	C-Specificity	C-NPV
Emory	LGBM	−0.01	0.54	0.07	0.89	0.99
	SIRS	0.64	0.61	0.04	0.79	0.99
MIMIC-IV	LGBM	0.01	0.55	0.06	0.89	0.99
	SIRS	0.63	0.57	0.04	0.80	0.99

For both models on both datasets, the C-Precision of both models is consistently below 0.1, indicating that more than 10 positive-prediction blocks need to be evaluated before a TP prediction within the next 6 hours. This highlights the major challenge of using these models in practice: large number of FPs. Compared to the conventional precision – fraction of positive predictions that are TPs, C-Precision needs special interpretation where we replace positive predictions with notifications and replace TPs with notifications having event onset within the next hours. The reason for lower C-Precision is because predictions are made at every time point in the continuous setting resulting in much lower prevalence of positive-label-time-blocks (0.96% in MIMIC-IV).

Note that the performance evaluation at a single snapshot time point using conventional precision cannot identify the issues of low C-precision as the sample taken from the control is more likely to have lower event risk scores thus better AUROCs like shown in Fig. 4. However, since the model will run continuously over the entire control patient stay, any high-risk score will contribute to FPs. Our proposed C-Precision can reliably capture the effects of these FP on the model usability. When compared to the hospitalization-level metrics presented in Table 1, where each patient stay only contributes one risk score for the evaluation (PPV = 0.23 on Emory), our proposed C-Precision (0.07 on Emory) can better capture the effects of FPs caused by multiple high-risk scores from the same patient stay. When compared to the 12-hour time horizon level metrics presented in Table 1, where multiple consecutive threshold crossings are treated as independent samples (PPV = 0.10 on Emory), our proposed evaluation metrics aim to consolidate multiple consecutive threshold crossings into a single notification.

Discussion

We presented three novel CEFs to evaluate ML models continuously on retrospective datasets. We demonstrated the usefulness and generalizability of these CEFs in the context of sepsis prediction in two datasets. There are few similarities between the three frameworks. First, all three frameworks take control patients into account when computing performance (via FP and TN). Second, all three frameworks have options to distinguish between alerts right before the event from alerts that are too early and deemed unactionable. Third, all three frameworks are agnostic to the frequency of predictions and are less influenced by the length of unit stays for both cases and controls.

We created Table 4 to compare the three CEFs to elucidate its strengths and weaknesses. We further discuss these traits for each CEF below.

Table 4.

Comparing the three CEFs to elucidate its strengths and weakness

	Strengths	Weaknesses
Zone-Based CEF	▪ Handles multiple predictions for the target event ▪ Can be evaluated during and after event of interest	▪ Evaluated on individual predictions ▪ Threshold will need to be pre-selected ▪ The MCS is partially interpretable
AMOC CEF	▪ Metrics are interpretable ▪ Can enable threshold selection	▪ Evaluated on individual predictions ▪ Handles only the very first positive prediction within window W ▪ Cannot be evaluated during or after event of interest
Notification-Based CEF	▪ Evaluated on notifications ▪ Handles multiple notifications for the target event ▪ Metrics are interpretable	▪ Threshold will need to be pre-selected ▪ Cannot be evaluated during or after event of interest

The zone-based CEF, succinctly captures the performance in a single metric, namely the MCS, the MCS by itself is difficult to interpret. MCS score’s interpretability improves when reported by zone with appropriate choice of scores. For instance, the Emory-LGBM model for early sepsis onset group had a negative MCS score of ‘−0.21’ (see Table 2). Recall these patients only had zone A with a TP score of ‘+1’ and FN score of ‘−0.5’. Hence a negative MCS score implies that the LGBM model is producing roughly more than 2 times the number of FNs when compared to TPs. In contrast among the non-sepsis patients, with only zone C, the MCS score represents a weight-balanced proportion of TN among all controls. The relatively high MCS score in non-sepsis patients (both models, both datasets had MCS ≥ 0.9 out of possible 1.0) indicates that in this group both models generate very few false alerts. Thus, a clinical practitioner considering deploying the LGBM or SIRS model can expect both models at their respective chosen thresholds to: [1] generate few alerts, a big proportion of which would be true alerts; [2] generate very few false alerts in non-sepsis patients. These are insights that are practical and useful when considering deployment of ML models.

Zone-based CEF offers flexibility in defining the length of zone A, B and C based on the clinical problem and user needs. Additionally, the weights for TP, TN, FP, FN can be adjusted to reward/penalize model performance based on clinical goals. For a different clinical condition with a faster onset, a shorter length should be chosen. We also chose a small penalty for FNs to be ‘−0.05’ keeping clinical workflow in mind. A sepsis prediction model like the ones presented in this paper would be expected to provide an early indication of sepsis and it would be used alongside standard sepsis monitoring protocols. If the intention is to use a sepsis prediction model solely to trigger assessment for sepsis, a higher penalty for FNs should be chosen. In this way zone-based CEF can be customized to clinical workflows to provide a meaningful assessment of predictive model performance over time and to ensure adoption of zone-based metrics by clinical practitioners.

The predictive AMOC curve on the other hand has good interpretability since it is presented along two dimensions that are easily interpretable. Clinical practitioners benefit the most from AMOC curves when using it to select a risk threshold – which can be used to convert risk scores into positive predictions. Traditional approaches typically use single time slice data to determine the threshold that maximizes composite scores such as F1 or breakeven precision-recall but in reality, these scores are not indicative of model behavior during deployment. For example, for the LGBM-Emory model, we utilize the data one hour prior to sepsis onset for cases and a random sample for controls. Note that this dataset resulted in an AUROC of 0.87 in Fig. 4. To deploy this model, we would choose the threshold that maximized breakeven precision-recall. At a chosen threshold of ‘−0.009749’ resulted in breakeven precision-recall of 0.51 and specificity of 0.96. But at this same threshold in the AMOC curve the median anticipation time is 5 h and the false alert rate is 0.05. This indicates that traditional approaches to threshold selection are simply inadequate when it comes to continuous evaluation.

A clinician could use the AMOC CEF to choose a risk threshold such that it results in a longer median anticipation time and a low false alert rate. One benefit of using anticipation time compared to the lead time is that the former accounts for both detected cases (TPs) and missed cases (FNs) while the latter only uses detected cases (TPs). As the number of missed cases increases, the lead time remains high while the anticipation time decreases reflecting the clinical reality that for many cases the model provided no advance warning.

The longer median anticipation time will lead to early detection of disease which gives clinicians adequate time to intervene. Early detection and intervention have generally been shown to improve patient outcomes like shorter length of hospital stay, lower risk for mortality and lower risk of hospital re-admission [18–20]. The lower false alert rate will lead to improved clinician trust in the predictive ML algorithm and less likely for clinical staff to experience alarm fatigue and alarm desensitization. These are some measurable clinical benefits when using the AMOC for continuous evaluation.

Both zone-based and predictive AMOC CEFs were evaluated on multiple positive predictions rather than notifications. The notification-based framework extends the standard confusion matrix-based metrics to continuous settings. These metrics better characterize the user’s perception of model performance when deployed at the bedside.

A major challenge with notification-based CEF is the number of FPs. We propose mitigation plans to reduce the number of FPs. A low C-Precision is caused by the flawed nature of the trained prediction model. For example, we found the sepsis LGBM model will always output high risk scores for patient stays that exceed 59 h in the ICU. A mitigation measure would be to drop the number of hours in ICU from the input features as it introduces clinical bias. A low C-Precision indicates that most notifications are FP which makes the user to ignore model predictions due to alarm fatigue. Therefore, the prediction model should be configured in such a way that the C-Precision should exceed a minimal acceptable threshold tolerable to the user.

Given the tradeoff between C-Precision and C-Recall, increasing C-Precision to reduce alarm fatigue will naturally decrease C-Recall. Low C-Recall will benefit a small subset of sepsis patients at the expense of missing most sepsis patients, resulting in worse overall patient outcomes. Therefore, besides the prediction model, the user should be advised to make use of other monitoring parameters to detect patient conditions. In this workflow, it is important to have high C-Precision to reduce false alerts at the expense of C-Recall. Additionally, C-Precision will never reach 100% so the user should be advised to use the model predictions only as triggers for reassessment of the patient condition, and importantly model prediction should never directly trigger medical interventions without further reassessment.

In an alternate workflow, if the prediction model replaces the existing standard of care, then a low C-Recall is a risk to patient safety and therefore should be improved. Improving C-Recall will lead to a lower C-Precision and consequently more false alerts. All these tradeoffs can be evaluated inexpensively on retrospective datasets by varying the risk threshold (higher risk threshold will increase C-Precision) and the notification definition (requiring multiple consecutive threshold crossings to start the notification which will lead to increased C-Precision).

There are some limitations in this work pertaining to training the sepsis prediction models and the framework metrics. Firstly, on sepsis model training, using number of hours in ICU as an input variable, introduces clinical bias into the LGBM sepsis models. We retained this input since our goal was to replicate the sepsis model previously published. Moreover, the SIRS model does not use this feature and therefore is not impacted. Second limitation of the model training is that we used the sepsis-3 criteria to determine the onset of sepsis, and we did not explore any alternate label definitions. However, as the goal of this work was to present CEFs, we used sepsis definition and previously published prediction models. On the CEFs, we have not reported confidence intervals for the proposed evaluation metrics since we compute and report the metrics on a single test partition for both datasets. This is very similar to a single AUROC often reported on test partitions or external dataset validations.

In the future, we would like to use these CEFs to evaluate other clinical risk prediction models such as hypotension prediction, early prediction of general deterioration, hospital-acquired infection and more. Evaluating the actions and clinical benefits from the model requires conducting a prospective clinical study where the model is deployed on the bedside and clinicians’ responses are collected and analyzed in parallel. These CEFs can be extended to evaluate the performance in prospective studies. The framework specific parameters can be selected to align with the clinical workflow. Additionally, model fairness can be evaluated using the proposed CEFs on different patient subgroups, including those vulnerable and historically marginalized patient populations. The CEFs can be used to evaluate already deployed ML models. These metrics provide additional information on model performance over time, and the impact of dataset shift in the deployed population. Updated model performance can be compared to the original model by these metrics. By retrospective evaluation of models using the CEFs, potential poor performance of models when running continuously can be identified earlier. This will prevent unnecessary prospective studies on models having good retrospective performance as measured by standard metrics, while exhibiting poor performance when evaluated continuously over time. The ultimate goal of the CEFs is to enable deploying an ML for early detection with lower false alarm rates. We hypothesize that this will lead to measurable clinical benefits with improved patient outcomes [18–20]. The evidence supporting the above hypothesis will need to be generated from randomized controlled trials.

In conclusion, we believe these frameworks provide a more realistic understanding of clinical ML model’s utility and behavior. All CEFs and the proposed metrics are impacted by their respective settings. Understanding of the effect of these settings on continuous model performance will enable clinical decision-making and to build trust. The three proposed frameworks complement each other and provide a wholistic view of the behavior of clinical ML models when run continuously. We encourage ML practitioners to utilize these frameworks to perform continuous evaluations on retrospective datasets and report performance along with the single time slice evaluations.

Appendix A

A.1 Early Prediction of Sepsis Challenge

Sepsis is a life-threatening condition that occurs when the body’s response to infection causes tissue damage, organ failure, and/or death [21]. The CDC reports that in the US, at least 1.7 million adults develop sepsis with close to 350,000 deaths. 1 in 3 patients who die in the hospital had sepsis during their hospital stay. Early detection and treatment of sepsis is critical to improving sepsis outcomes, where each hour of delayed treatment has been associated with roughly 4 − 8% increase in mortality [22, 23]. The 2019 computing in cardiology challenge was to detect sepsis using physiological variables. Specifically, the goal of this challenge was to develop ML models to predict sepsis six hours in advance. Worldwide, 104 teams participated in the challenge.

The challenge dataset comes from two different hospital systems: Beth Israel Deaconess Medical Center (BIDMC) and Emory university hospital. We only utilized the data from the Emory university hospital system since the BIDMC dataset may overlap with our second dataset (described next), which also comes from BIDMC.

A.2 Emory Dataset

This dataset was collected over the past decade with approval from the appropriate Institutional Review Board. There are a total of 20,000 patient stays in this dataset. In total, 1,142 patient stays have sepsis with a prevalence of 5.7%. The Emory dataset consists of hourly summaries of 8 vital signs, 26 laboratory variables, and 6 demographic variables. All variables were pre-processed and made available at one hour resolution during patient ICU stays. The Emory dataset also consists of sepsis labels that we describe next.

A.3 Emory Dataset - Onset of Sepsis (t_sepsis)

The clinical onset of sepsis was defined according to the Sepsis-3 guidelines [21], which we briefly describe here,

1. t_suspicion: The time of clinical suspicion is the time of antibiotics when followed by blood cultures within 24 h or the time of blood cultures when followed by antibiotics within 72 h. In either case, the antibiotics should have been administered continuously for at least 72 h.

2. t_SOFA: The time when the sequential organ failure assessment (SOFA) score dropped by 2 or more units within a 24-hour period.

3. t_sepsis: The time of clinical onset of sepsis. Note that t_SOFA will need to occur within 24 h from t_suspicion or 12 h after t_suspicion (i.e., t_suspicion − 24 ≤ t_SOFA ≤ t_suspicion + 12). When this criterion is satisfied, the clinical onset time of sepsis is the minimum of [t_SOFA,t_suspicion].

The median (IQR) of sepsis onset time for the 1,142 sepsis patients is 34.5 (10.25, 80) hours following ICU admission, with a minimum and maximum of 7 and 337 h, respectively. The sepsis label is binary and is set to one at times t ≥ t_sepsis(n) − 6 h and zero at times t < t_sepsis(n) − 6 h for the n_th sepsis patient. For example, if a patient’s sepsis onset is hour 50 then this binary label would be set to one starting at hour 44 to the end of ICU stay. For all non-sepsis patients, the sepsis label is set to zero for all times t. We refer to this label as ‘shifted sepsis label’ since it is set to 1 six hours prior to sepsis onset. The shifted sepsis label was created by the challenge organizers to enable early prediction of sepsis giving clinicians a buffer time of six hours to react.

A.4 MIMIC IV Dataset

The Medical Information Mart for Intensive Care (MIMIC-IV) database is a comprehensive collection of deidentified medical records from patients admitted to the Beth Israel Deaconess Medical Center’s emergency department and intensive care unit in Boston, MA. The database, encompassing data from over 65,000 ICU patients and 200,000 emergency department visits, aims to enhance patient care through knowledge discovery and algorithm development [10].

MIMIC-IV is sourced from two in-hospital database systems: a custom hospital wide EHR (hosp module) and an ICU-specific clinical information system (icu module). A total of 364,627 unique individuals is in MIMIC-IV v3.1, each represented by a unique subject id. These individuals had 546,028 hospitalizations and 94,458 unique ICU stays. In this research, we used the icu module to extract model inputs and sepsis labels. In total, 5,531 patient stays have sepsis with a prevalence of 9.4%. All input variables were pre-processed and made available at one hour resolution, similar to the Emory dataset.

A.5 MIMIC IV Dataset-Onset of Sepsis (t_sepsis)

The creators of the MIMIC-IV dataset have also provided a derived table with sepsis onset times using Sepsis-3 guidelines [21, 24]. The logic to extract onset time of sepsis in MIMIC-IV followed the same logic as Emory dataset with one difference in the occurrence of the t_SOFA score (i.e., t_suspicion − 48 ≤ t_SOFA ≤ t_suspicion + 24). The window to look for t_SOFA score is slightly longer when compared to the Emory dataset. Here, we implicitly assume that the baseline SOFA score was 0 before admission to the ICU for patients not known to have pre-existing organ dysfunction. We excluded sepsis patient stays with sepsis onset within 6 h of ICU admission to align with the Emory dataset from the Sepsis challenge. The median (IQR) of sepsis onset time for the 5,531 sepsis patients is 39 (14, 84) hours following ICU admission, with a minimum and maximum of 7 and 1,646 h, respectively. Similar to the Emory dataset, we created a binary ‘shifted sepsis label’ for the MIMIC-IV dataset. All other details are similar to the Emory dataset.

A.6 Dataset Statistics

In Table 5, two datasets, Emory and MIMIC-IV, are examined to provide statistics stratified by the presence or absence of sepsis events among the patient stays. The median value, along with the interquartile range, was reported 1 h prior to sepsis onset for sepsis patient stays, and at a random time for non-sepsis patient stays. The input features in both datasets are fill-forwarded indefinitely before calculating the 1-hour prior statistics to avoid missingness due to different chart times. This is the same protocol followed by the challenge winner.

Table 5.

Dataset statistics stratified by sepsis and non-sepsis events

	Emory Dataset (N = 20,000)		MIMIC-IV Dataset (N = 58,694)
Feature	Sepsis (N = 1,142; Prev.=5.7%)	Non-sepsis	Sepsis (N = 5,531; Prev.=9.4%)	Non-sepsis
Age (years)	62 [50, 72]	62 [50, 72]	65 [53, 75]	64 [52, 75]
ICU LOS (hours)	37 [13, 82]	39 [24, 46]	42 [17, 87]	38 [23, 67]
Heart Rate (bpm)	90 [78, 104]	81 [70, 94]	88 [75, 102]	81 [70, 93]
Systolic BP (mmHg)	119 [103, 139]	124 [110, 141]	118 [103, 135]	119 [106, 135]
Temperature (C)	37 [36.4, 37.8]	36.7 [36.4, 37.2]	37 [36.7, 37.6]	36.8 [36.6, 37]
Respiratory Rate (bpm)	19 [16, 23]	18 [16, 20]	20 [16, 24]	18 [15, 21]
WBC (103/µL)	11.4 [8.2, 15.5]	9.3 [6.9, 12.3]	11.5 [8.3, 15.7]	9.6 [7.2, 13]
HCT (%)	30 [26, 35]	32 [28, 37]	30 [26, 35]	32 [28, 37]
Lactate (mg/dL)	1.6 [1.1, 2.3]	1.6 [1.2, 2.3]	1.5 [1.1, 2.2]	1.6 [1.1, 2.2]
Creatinine (mg/dL)	1.2 [0.8, 2]	0.9 [0.7, 1.4]	1 [0.7, 1.7]	0.9 [0.7, 1.2]

Across both datasets, patient stays with sepsis generally exhibit greater variations from the normal clinical parameters. Notably, stays with sepsis show elevated heart rates, white blood cell counts, alongside extended ICU stays, compared to non-sepsis patient stays. Additionally, the data suggest slightly older ages among sepsis patient stays, particularly within the MIMIC-IV cohort.

A.7 Challenge Utility Score

For the challenge, a utility score was designed that rewards sepsis predictions six hours prior to the onset of sepsis and penalizes late predictions as well as false alarms. Refer to Figure 6. They define a score U(n, t) for each prediction, i.e.,for each patient n and each time interval t, as,

Fig. 6.

Sepsis (above) and non-sepsis (below) utility function from the 2019 computing in cardiology challenge. Figure from (4)

• U_{TP(n, t)} - positive prediction at time t for sepsis patient n

• U_{FN(n, t)} - negative prediction at time t for sepsis patient n

• U_{FP(n, t)} - positive prediction at time t for non-sepsis patient n

• U_{TN(n, t)} - negative prediction at time t for non-sepsis patient n

The utility is summed over all N patients, all T(n) timestamps from the n^th patient, to get,

A1

Lastly, the utility score is normalized to be between 1 (best possible score) and 0 (completely inactive classifier) as, 

A2

A.8 Sepsis Challenge Winner

The challenge winner developed a light gradient boosted tree model (LGBM) to predict sepsis [11]. Two novelties in this work include signature transformation of inputs and framing the problem of sepsis prediction as a regression problem. To train the LGBM regressor, the challenge winners created a modified utility score. The modified utility score, for a sepsis patient, is the difference between true positive and false negative (U_TP−U_FN). This modified utility score is set to −0.05 from ICU admission to t_early (Figure A1) and is set to 0 from t_late to ICU discharge. In between t_early and t_late the modified utility score is the difference between the U_TP and U_FN, like shown in Figure A1.The modified utility score, for a non-sepsis patient, is the difference between false positive and true negative (U_TP−U_FN) and is set to a constant of −0.05 throughout patient stay. The LGBM model is trained to minimize the mean absolute error between predicted utility score and modified utility score. The challenge winner achieved a best utility score of 0.36 across all three datasets (two public and one not released private dataset).

A.9 Train-Test Partition

We partition both Emory and MIMIC-IV datasets into 70 − 30% train-test split. For the Emory dataset, the train partition contains data from 14,000 patient stays of which 798 (5.7%) have sepsis. The test partition contains data from 6,000 patient stays of which 344 (5.7%) have sepsis. For the MIMIV-IV dataset, the train partition contains data from 41,085 patient stays of which 3,871 (9.4%) have sepsis. The test partition contains data from 17,609 patient stays of which 1,660 (9.4%) have sepsis. At a sample level i.e. hourly summaries, both datasets, both partitions have ∼ 1.4% to ∼ 1.6% of sepsis samples.

A.10 Hyperparameter Tuning

In this section, we describe our 5-fold CV setup for hyperparameter tuning for both sepsis prediction models (described in Sections A.10, A.11). We utilize the train partition to train the sepsis models. We perform five-fold CV within the train partition to pick hyperparameters. Within the train partition all hourly samples from all patient stays are partitioned into five folds. Hence, adjacent, correlated samples (samples from hours 1, 2, 3, 4, 5, etc.) from a single patient stay could be present across the five folds leading to data leakage. The pros of this CV approach are that long patient stays (e.g., >5 days in ICU) which contribute many hourly samples (e.g., 10 days x 24 h = 240 hourly samples) will be uniformly distributed across five folds. The probability that all hourly samples from a single patient stay will occur in a single fold is very low therefore these long patient stays will have less influence on the choice of hyperparameters.

A.11 LGBM model

In this work, we utilized the sepsis prediction model from the challenge winner. The LGBM is a light gradient boosted tree model which was the winner of the 2019 Physionet challenge [11]. The LGBM model is a regression model that is trained on 40 inputs to produce a continuous score. We retrain the sepsis prediction model using the train partition. We do not use the pretrained sepsis prediction model from the challenge winner since is it trained on the complete Emory dataset and a large portion of the MIMIC-IV dataset. Using this pretrained sepsis prediction model for continuous evaluation on test partitions will result in inflated performance due to data leakage. We used the light gradient boosted tree (LGBM) algorithm as described by the challenge winner [11].

We followed the preprocessing steps of the challenge winner by forward filling the 40 clinical variables within each patient and setting missing variables to ‘NaN’. We trained the LGBM regression model on the modified utility score (described in section A.7 above) using the 70% train partition of both datasets. We performed hyperparameter tuning via 5-fold CV (described in Section A.9) using the train partition. We experimented with learning rate from [1e − 4,1e − 3,1e − 2,0.1] and the number of estimators from [500,750,1000]. We chose the best hyperparameters by maximizing the negative mean absolute error. All other settings were set to their default.

The predicted LGBM regression scores range from − 0.03 to 1.59 in the train partition. Like the challenge winner, we convert these predicted regression scores to sepsis binary labels by selecting a regression threshold that maximizes the challenge utility score. We sweep through the min and max regression scores in 0.01 increments.

Lastly, we performed an analysis to ensure that we have trained a sepsis prediction model similar to that of the challenge winner. We reproduced the challenge utility score by training a LGBM model in a double nested 5-fold CV on the complete Emory dataset. We obtained a mean challenge utility score of 0.32 ± 0.01. The challenge winner reported a mean utility score of 0.39, on both Emory and BIDMC datasets, when utilizing the original 40 features [11].

A.12 SIRS Model

We utilize a systemic inflammatory response syndrome (SIRS) as a baseline sepsis prediction model. SIRS is evoked when there is a life-threatening condition like infection (e.g., sepsis), trauma or a worsening health condition [25]. Clinically, SIRS is identified by two or more symptoms including fever or hypothermia, tachycardia, tachypnoea and change in blood leucocyte count [26]. All these variables are available in Emory and MIMIC-IV datasets making it possible to compute SIRS score, but not all patients have all variables available at all times. We developed a predictive SIRS model that serves as a baseline sepsis ML model when compared to LGBM. We trained a XGBoost model using the same set of inputs as the SIRS criteria and binary ‘shifted sepsis label’, on the train partition of both datasets. We performed hyperparameter tuning using 5fold CV on the train partition for the number of decision trees from [50,100,200,500] and learning rate from [0.01,0.2,0.5,1.0]. We set the decision trees to be of depth 1. All other settings were default. For each test patient stay, we get a sepsis event risk score, hourly from both LGBM, SIRS models.

A.13 LGBM and SIRS Threshold Selection

For both the LGBM and SIRS models we need to pick a risk threshold to make positive predictions. We follow the same steps as the sepsis challenge winner to pick a risk threshold [11]. For each dataset, we choose the risk threshold using the train partition and use the same risk threshold on its respective test partition. For both models, we sweep through the min and max sepsis event risk scores in 0.01 resolution to compute the sepsis challenge utility score [4]. We choose the risk threshold that maximized the challenge utility score. For the LGBM model, the risk thresholds were − 0.01 and 0.01 for the Emory and MIMIC-IV datasets, respectively. For the SIRS model, the risk thresholds were 0.64 and 0.63 for the Emory and MIMIC-IV datasets, respectively. Note that the LGBM model is a regression model and the predictions can be negative. In Figure 7, we show a plot of the utility score as a function of risk thresholds for both models, both datasets. These risk thresholds are used for all performance metrics except for AMOC curves in which we sweep through multiple risk thresholds to plot them.

Fig. 7.

Challenge utility score for (A) LGBM and (B) SIRS models as a function of threshold

A.14 Epic Sepsis Model Paper Performance Metrics

In Table 6, we show the continuous evaluation metrics from prior work, specifically the performance reported by Wong et al., when evaluating the Epic sepsis model [8].

Table 6.

Sepsis model performance when evaluated similar to the ESM paper [8]

Dataset	Model	Level	Sepsis incidence	AUROC	PPV	Sensitivity	Specificity	NPV	No. needed to evaluate	Lead time (Median (IQR)); No. of patient stays
Emory	LGBM	Hospitalization level	5.73%	0.83	0.23	0.59	0.88	0.97	4.28	44 (10–88); 204
		Time horizon 12 h	1.99%	0.80	0.10	0.50	0.91	0.99	9.58
	SIRS	Hospitalization level	5.73%	0.74	0.12	0.69	0.70	0.97	8.19	35.5 (7–86); 236
		Time horizon 12 h	1.99%	0.75	0.06	0.44	0.87	0.99	15.67
MIMIC-IV	LGBM	Hospitalization level	9.42%	0.72	0.33	0.67	0.78	0.76	4.22	27 (8–72); 1112
		Time horizon 12 h	2.40%	0.76	0.10	0.38	0.91	0.98	10.14
	SIRS	Hospitalization level	9.42%	0.68	0.17	0.68	0.66	0.95	5.73	33 (10–75); 1129
		Time horizon 12 h	2.40%	0.72	0.07	0.38	0.88	0.98	13.92

A.15 AMOC CEF-Results from Windows W = 6 and W = 12

See Figures. 8, 9, 10 and 11.

Fig. 8.

Emory Dataset, predictive AMOC curves, W = 6 h

Fig. 9.

Emory Dataset, predictive AMOC curves, W = 24 h

Fig. 10.

MIMIC-IV Dataset, predictive AMOC curves, W = 6 h

Fig. 11.

MIMIC-IV Dataset, predictive AMOC curves, W = 24 h

Author contributions

AN, YC, BM conducted the research, interpreted the results and contributed to writing; LA extracted and processed the datasets; EG conceived the idea, supervised the research, contributed to reviewing and to writing. All authors reviewed the manuscript.

Funding

None.

Data availability

Information on the two publicly available datasets is provided within the manuscript or supplementary information files.

Declarations

Ethics approval and consent to participate

Not Applicable.

Consent for publication

We give consent for publication.

Competing interests

The authors declare no competing interests.