A novel, standardised approach to balancing effectiveness, efficiency and utility of surveillance AI prediction models for hospitalised patients using sepsis prediction as an exemplar

Abstract

Objective

To introduce a novel, standardised approach to evaluating AI prediction models in balancing effectiveness, efficiency and utility, using a sepsis prediction model case study.

Materials and Methods

Retrospective patient data from electronic medical records of 7 public hospitals was used to retrain and evaluate a machine learning sepsis prediction model. Four conventional metrics—area under the receiver operating curve (AUROC), sensitivity, positive predictive value, and specificity—were compared with a novel graphical display integrating metrics of predictive accuracy (effectiveness), alert burden (efficiency) and lead time of alerts relative to clinical events (utility) for different alert thresholds.

Results

The dataset comprised 977,506 inpatient admissions. The novel methodology produced a plot of four vertically aligned graphs that enables decision-makers to identify an alert threshold that optimally balances effectiveness, efficiency and utility (EEU) at the level of an entire admission, and which differs from that derived using conventional metrics.

Discussion

Conventional evaluation metrics do not consider alert timing relative to clinical events and are often applied to different evaluation datasets (sample and admission level), introducing bias and confusion. In contrast, the EEU methodology (i) generates admission level evaluations at different alert thresholds; (ii) measures alert timing relative to clinical events; and (iii) provides a visual display that enables identification of the alert threshold that optimally balances EEU factors.

Conclusion

Evaluations of prediction models for adverse events in hospitalised patients should incorporate the EEU approach in assessing model suitability and selecting alert thresholds.

Introduction

Artificial intelligence-(AI) based models that use real-time patient surveillance data in hospitals to predict the onset of adverse events, such as sepsis or clinical deterioration, have been extensively researched with some now widely used in clinical practice.^1–3 However, such models, when deployed, have demonstrated mixed or no improvements in patient care, ^4–6 the reasons for which are often under-investigated.^2,7 Efferent arm shortcomings such as poor model integration into clinical workflows or inadequate clinician training or response to model outputs may be partly responsible. Alternatively, clinicians may not adopt these models if they: (i) are ineffective at identifying patients at risk of the adverse event, (ii) are inefficient in the use of resources and clinician time or (iii) have little utility for clinicians in improving care beyond what is achievable using current methods. Measuring effectiveness, efficiency and utility (hereafter termed EEU factors) prior to implementation is essential for any prediction model in optimising user acceptance, as embodied in both the Technology Acceptance Model (TAM) and the Unified Theory of Acceptance and Use of Technology (UTAUT).^8–10

Prior to implementation, AI prediction models are usually evaluated both retrospectively, using static datasets, and prospectively, using dynamic real-time data feeds, consistent with reporting guidelines such as TRIPOD-AI¹¹. In two recent systematic reviews of implemented sepsis and deterioration prediction models,^1,2 the most commonly cited performance metrics were: (i) Discrimination statistics: Area Under the Receiver Operating Curve (AUROC) and Area Under the Precision Recall Curve (AUPRC); and (ii) Alert threshold dependent operating metrics: sensitivity, positive predictive value (PPV), median time between first alert and onset of the adverse event (MTBFAA) and variants of mean alert count per day.

These metrics map to one or more EEU factors (see Table 1), and are used to select between different models and to select an optimal alert threshold, ie, the probability output of the AI model, above which an alert will be triggered. However, for both functions, they demonstrate considerable shortcomings. First, MTBFAA is a poor metric for clinician utility because an alert is of little or no use if it arrives after clinicians suspect or have responded to the event being predicted. ^7,12 In the case of sepsis, an alert arriving 3 hours before the patient becomes overtly septic with organ dysfunction (ie, meets Sepsis-3 criteria¹³) but 2 hours after clinicians have initiated antibiotics, is of little value. Second, neither AUROC nor AUPRC consider alert timing, in that prediction model A with identical AUROC to model B may deliver alerts on average 4 hours after model B when clinicians have already identified the event. Third, none of the metrics consider all three EEU factors simultaneously, hindering clinicians and other decision-makers from selecting the most optimal model and alert threshold that provides acceptable sensitivity, without incurring alert fatigue (low PPV), and where the first alert usually arrives before clinicians suspect or confirm the event. Decision curve analysis (DCA) has been used to determine a probability output threshold for a particular model, above which clinician response is likely to confer benefit, but which is derived from static, retrospective model evaluations and does not consider alert timing relative to other clinical events when applied to dynamic, real-time data input.¹⁴ DCA is also difficult to understand and interpret for clinicians.¹⁵

Table 1.

Factors impacting translation of AI-based surveillance predictive models to care improvement (column 1), the corresponding model alert-threshold dependent AI evaluation metrics (column 2) and the relationship between Area Under the Receiver Operating Curve (AUROC) and Area Under the Precision Recall Curve (AUPRC) (columns 3,4) with the factors and the alert-threshold dependent metrics.

(EEU) Factors	Common evaluation metrics: Alert-threshold dependent	Common evaluation Metrics: Statistical discrimination
		AUROC	AUPRC
Effectiveness: at identifying target population	Sensitivity: proportion of target population identified	Non-linear relationship	Non-linear relationship
Efficiency: of clinician time	Positive Predictive Value (PPV): an indication of wasted clinical time assessing false positives, ie, the proportion of alerts that are correct (ie, true positives)	Not directly related	Non-linear relationship
	Mean Alert Count per Day (MACD): an indication of the overall alert burden for clinicians	Not directly related	Not directly related
Utility: benefit to clinicians over existing care methods	Median Time Between first Alert and the Adverse event (MTBFAA): An indication of the number of hours of advance warning provided to clinicians	Not directly related	Not directly related

EEU = Effectiveness, efficiency and utility. Non-linear relationship means that the metric heading the column (columns 3 or 4) does not have a linear relationship with the metric specified in column 2; Not directly related means that it is not possible to derive the metric specified in column 2 with the metric specified in the column header (columns 3 or 4).

These conventional performance metrics differ from one-another but the nature of trade-offs when comparing these metrics can be complex and offers no clear optimal choice as to model suitability or alert threshold for particular clinical settings. These limitations were identified during the design of a real-time prospective silent evaluation of an AI-based sepsis prediction model for a state-wide hospital network.¹⁶ In response, a novel evaluation process needed to be developed, in collaboration with clinician end-users, that integrated each EEU factor in ways that trade-offs between these factors could be easily appreciated in assessing model suitability and selection of optimal alert thresholds.

Objective

Using a case-study of a sepsis-prediction model, to develop a novel, standardised methodology for evaluating and balancing EEU factors that enables decision-makers to (i) decide whether an AI model is suitable for use in clinical practice and (ii) select an optimal alert threshold for its use.

Background and significance

Other researchers have recognised the limitations in using AUROC or AUPRC alone and have incorporated some but not all elements of the EEU trade-offs within their model evaluations. For example, for sepsis prediction models, Wong et al. evaluated the Epic Sepsis Model (ESM) and graphed AUROC, sensitivity and specificity at 3 different time horizons relative to the sepsis event to try to understand how model effectiveness changes with time. ¹⁷ Topiwala et al. considered clinical utility, defining it as the tool both correctly alerting and alerting before clinical suspicion (as documented in clinical notes), each metric being derived independently.¹⁸ Similarly, Giannini et al. reported the time to antibiotic administration rather than just the time to the predicted sepsis event. ¹⁹ Kamran et al. re-evaluated AUROC on the basis of alerts arriving before a range of clinical actions to try to understand the trade-off between model performance and utility.¹² Finally, although Brown et al. attempted to select an optimal alert threshold using a plot of sensitivity, specificity, PPV and false positive rate at multiple alert thresholds, alert timing was not considered, meaning utility could not be assessed simultaneously.²⁰ Similar limitations apply to studies of implemented models predicting other non-sepsis events such as COVID-19, cardiac arrests and clinical deterioration.^21–28

A methodology that provides hospitals with a transparent and integrated approach to evaluating AI model suitability in terms of effectiveness, efficiency and utility and with a standardised means to balance these factors using the alert threshold, could prevent unsuitable models from being implemented, promote better clinician adoption and improve the benefits of implemented models.

Materials and methods

This section is divided into 2 parts: (i) The case study setup describing the retrospective patient dataset used to train and evaluate the AI model plus a brief overview of dataset preparation, feature variables and model configuration; and (ii) A detailed description of how the novel methodology was derived and compared to conventional metrics in assessing model performance and selection of a clinically optimal alert threshold.

Case study setup

Participants and setting

Patient data was extracted from adult inpatient admissions between 1 January 2021 and 31 December 2024 at 7 digital hospitals (three tertiary, four secondary; five metropolitan, two regional) with integrated electronic medical records (EMRs) within the Queensland Health public hospital system in Australia. Excluded admissions were those without an emergency department (ED) and ward stay or without at least one entry of heart rate, systolic blood pressure and oxygen saturation. At the patient level, as our focus was on ED or general ward patients, we also excluded data pertaining to care episodes closely monitored within intensive care units (ICUs), operating theatres and post-anaesthetic care units. This study was approved by the Metro South Hospital and Health Service Human Research Ethics Committee (HREC/2022/QMS/88742) and was granted a waiver of patient consent with Public Health Act approval. TRIPOD+AI guidance for AI-based model prediction studies was followed for study reporting.¹¹

Reference standard for defining sepsis

A clinical expert committee consisting of 8 senior clinicians (VC, RW, VB, PL, KW, SW, AI, IAS (chair)) representing the specialties of intensive care, emergency medicine, oncology, infectious diseases, internal medicine and paediatric medicine formulated a modified Sepsis-3 definition¹³ (see Table 2 and further detail in Supplementary Appendix 1) using available EMR data which incorporated multi-organ dysfunction combined with a clinical suspicion of sepsis, as evidenced by the ordering of body fluid collection (BFC) and antibiotics by treating clinicians.^13,29

Table 2.

Sepsis-3 definition.

Elements of definition	Criteria of definition
Suspected or confirmed infection (within specified time limit)	Antimicrobial ordera PLUS Microbiological Specimen order b (If Microbiological Specimen order first, then Abx within 72 hours, if antimicrobial order first, then Microbiological Specimen order within 24 hours)
	AND
Evidence of Organ Dysfunction	Multi-organ dysfunction: A total SOFA score of ≥ 2 and from 2 or more organ systems OR Administration of IV vasoactive medication c (A SOFA score as defined above, indicating organ dysfunction, will count towards the Sep-3 label if it occurs within the time period: 48 hours before suspected infection to 24 hours after infection first suspected)

^aAppropriate antimicrobial orders that indicate a suspicion of infection are listed below in Appendix 1.1.

^bAppropriate Pathology specimen orders that indicate a suspicion of infection are listed below in Appendix 1.2.

^cIV vasoactive (inotrope/vasopressor) drug administration as an independent indicator of significant organ dysfunction. See Appendix 1, Table 2: QSA Modified SOFA criteria. To reflect original SOFA and current practise, vasoactive drugs include: dopamine, dobutamine, adrenaline, noradrenaline, metaraminol, phenylephrine and vasopressin. See Appendix 1.3 for vasoactive drug administration codes. In this setting vasoactive administration is suggestive of septic shock.³⁰

AI model, variables and dataset preparation

We refined the open-source light gradient boosting machine learning model (LightGBM)³¹ developed and described in prior work.¹⁶ In total, 40 variables were used as 66 features for the model, with missing variable values filled using last-one-carried-forward and missing values occurring before the first data entry given a null value (see Supplementary Appendix 2 for full details).

As per prior work,¹⁶ selected features were binned into aggregated (averaged) hourly time intervals (called samples) within each included admission, commencing with the patient’s first recorded measurement and ending with the last hour of available data. The one-hour time intervals reflected the intended update (or refresh) frequency for the deployed model, as determined by the clinician expert committee. For each hourly sample, a binary flag was assigned, set to true if Sepsis-3 criteria was met during or after that hour, or otherwise set to false.

The entire dataset of samples for all included admissions was divided into a Train dataset, comprising all samples from patient admissions between January 1, 2021 and December 31, 2023, and a Test dataset comprising remaining samples for all patient admissions in calendar year 2024. During model training, all hourly samples were removed after Sepsis-3 criteria were first met.

Model training and evaluation

The model was trained on the Train dataset using five-fold cross validation, with the same process used for hyper-parameter optimisation (see Supplementary Appendix 3). The best model identified in cross-fold validation was used for final external evaluation and alert threshold selection using the Test dataset. For purposes of comparison, we report two sets of results, conventional and novel, as follows.

Conventional evaluation and alert threshold selection

We documented retrospective and prospective model evaluation metrics (see Supplementary Appendix 4) for six implemented AI-based sepsis prediction tools identified in a recent systematic review:² (Epic sepsis model¹⁷, InSight^18,32,33, SepsisWatch^34,35, TrewScore³⁶, Sepsis Sniffer³⁷ and other²⁰). Where at least 2 studies reported a given metric, we selected them for our conventional evaluation; these included metrics for both a fixed alert threshold (sensitivity, specificity, PPV, MTBFAA) and across all thresholds (AUROC). The fixed alert threshold selected corresponded with the median reported sensitivity of 0.8 (range 0.75–0.86).^{20,32,33,35,36} We also identified two different evaluation dataset configurations from the studies: sample level (either stated¹⁷ or assumed ^21,25,26 if there was no mention of a dataset consolidation method) and admission level. Referring to the illustrative example in Figure 1, sample level datasets (ie, hourly samples, Test dataset) comprise all prediction samples generated by the model within each patient admission, for all patient admissions. Admission level datasets (herein called ‘Test-adm’) consolidate to a single sample per admission by selecting one sample within each patient admission, usually the one with the maximum model score,^35,38–40 for example, ‘Sample 10’ with model output = 0.7 in Figure 1.

Figure 1.

Graph demonstrating model output samples (decimal values between 0 and 1) contained within a single patient admission, reported hourly after admission (hourly samples). The alert threshold represents the model output, above which a positive alert for sepsis is generated.

Proposed novel model evaluation and alert threshold selection

The methodology comprises four steps.

In Step 1, a precision-recall curve (PR-curve) that plots PPV versus sensitivity is generated for the model using the sample level Test dataset across the range of threshold values. We used increments of 0.02 across the full threshold range (0 to 1).

In Step 2, the sample level PR-curve is transposed to a new graph that plots admission level metrics PPV, sensitivity and proportion of patient admissions alerted. To achieve this, an admission level dataset is built from the sample level dataset and metrics evaluated for every threshold used in the sample-level PR-curve in Step 1. To build each admission level dataset for each threshold, the first hourly sample with a model output at or above that threshold is selected for each admission, or if no output is above the threshold a random sample is taken (see Supplementary Appendix 5 for detailed method).

In Step 3, the true positive samples generated at each threshold in Step 2 are related to real-world clinical actions by reporting the time interval between the first sepsis alert and specific actions, diagnostic or therapeutic, that indicate clinicians are aware of or suspect sepsis. The clinical expert committee identified 3 clinical events: (i) ordering of a relevant microbiological test; (ii) ordering of antibiotics; and (iii) patient review triggered by a score of 4 on the existing rule-based Queensland Adult Deterioration Detection System (Q-ADDS),⁴¹ a deterioration detection tool similar to the National Early Warning Score.⁴² The median time from the first alert to (i), (ii), (i) or (ii), (iii) and to when Sepsis-3 criteria are first met is graphed for the same thresholds and with the same x-axis (sample level sensitivity) as the graph in Step 2. Similarly, a new graph is generated that plots the proportion of first alerts that occur before these same clinical events.

In Step 4, the four plots described above are arranged vertically with the same x-axis scaling. The clinician committee were then asked to seek consensus on a vertical line through all four graphs to select the alert threshold at which they considered optimal balance existed between admission level sensitivity, PPV, proportion of all patients alerted, and the median timing of alerts prior to each of the three clinical events. The decision-making process could conclude with consensus, or final committee chair (IAS) decision where consensus could not be achieved.

Results

Patient characteristics

Of 2,012,304 original admissions, 994,232 with no time spent in ED or a ward (ie, day patients) and 16,726 with no vital signs (heart rate, systolic blood pressure, oxygen saturation) were excluded, leaving 977,506 admissions for analysis, with 38,688,058 hourly data samples (see Table 3).

Table 3.

Patient and admission characteristics for the Train and Test datasets

	Train dataset (2021–2023)	Test dataset (2024)
Patient count	376,783	158,394
Admission level statistics
- Admission count	732,974	244,532
- Percent female	53.7%	53.6%
- Median Age (IQR)	54 (35,72)	56 (35,73)
- Median length of stay in hours (IQR)	29.5 (9.4, 90)	35 (12,96)
- Mean length of stay in hours	98	112
- Median SOFA score	0	0
- Met Sepsis-3 criteria (% total)	61,273 (8.4%)	23,098 (9.5%)
Sample level statistics
- Count of hourly samples	28,188,955	10,499,103
- Met Sepsis-3 criteria (% of total)	7,258,941 (26%)	2,862,561 (27%)

Conventional metric evaluation

The results for model evaluation using conventional metrics for the Test dataset, including both sample and admission level reporting, are provided in columns 1 and 2 of Table 4. The results change considerably when reported by admission level versus sample level (see Figure 2), including 13% higher AUROC, 37% higher specificity and 63% lower median hours between first alert and when Sepsis-3 criteria are first met. For sensitivity and PPV, between group differences remain within 2 percentage points. Results by site are provided in Supplementary Appendix 6.

Table 4.

Evaluation results for the sepsis prediction model on the Test dataset.

	Test dataset
	1 Sample level (Conventional)	2 Admission level (Conventional)		3 Admission level (Novel)
Sample count	10,499,103
Admission count		244,532		244,532
Sepsis-3 count (prevalence)	2,862,561 (27%)	23,098 (9.5%)		23,098 (9.5%)
Area under the receiver operating curve (AUROC)	0.8219	0.9317		Not applicable
Threshold dependent metrics:
Selected model threshold (For sensitivity of 0.8 set by conventional methods)	0.06	0.25		0.12
Conventional metrics		Actual	Reported
Sensitivity	77%	79%	80%	94%
Positive predictive value (PPV)	50%	48%	47%	26%
Number needed to evaluate (NNE)*	2.0	2.1	2.1	3.8
Specificity	71%	97%	91%	71%
Median time between first alert and adverse event (MTBFAA)**; (% of first alerts before adverse event)	14 hours (99%)	5.2 hours (87%)	− 0.5 hours (45%)	11 hours (97%)
Additional novel metrics:
Median time from 1st alert to blood culture order (% first alerts before this event)	2.6 hours (91%)	0.3 hours (57%)	− 8.7 hours (20%)	1.5 hours (79%)
Median time from 1st alert to antibiotics order (% first alerts before this event)	4.4 hours (92%)	1 hour (65%)	− 6.9 hours (27%)	3 hours (82%)
Median time from 1st alert to either antibiotics or blood culture order (% first alerts before either event)	1.9 hours (87%)	0 hours (49%)	− 11.5 hours (15%)	1 hour (73%)
Median time from 1st alert to Q-ADDS score of 4 reached (% first alerts before this event)	4 hours (74%)	0 hours (49%)	− 6 hours (8 %)	1 hour (66%)

The conventional evaluation metrics are presented at both the sample level (column 1) and the admission level Test-adm dataset (column 2) using an alert threshold value that yields a model sensitivity as close to 80% as possible. Note that the conventional metrics at admission level are documented for what would be ‘reported’, if using the admission level dataset derived from the maximum model output per admission or what the ‘actual’ results would be if the first score above the threshold is used to aggregate the dataset (ie, same method as for novel evaluation). Conventional metrics are also shown for the alert threshold setting identified through the novel evaluation method (column 3). Novel evaluation metrics located at the bottom of the table are generated for both the conventional alert threshold settings and the alert threshold identified through the novel evaluation method.

^*NNE is the inverse of PPV and represents the number of patients with alerts who need to be evaluated to identify one with true sepsis, and gives an indication of potential alert fatigue.

^**Adverse event = Sepsis-3 criteria are met.

Figure 2.

Receiver operating curves for the model evaluated on the sample level Test dataset (orange line, square markers) and the admission level Test-adm dataset (blue line, dot marker). ROC=receiver operating curve.

Proposed novel evaluation

Figure 3 demonstrates how the proposed evaluation plots integrate with one another, using sample level sensitivity as the common x-axis. This allows visualisation of the impact of the full range of model alert thresholds on model effectiveness (graph B, sensitivity), efficiency (graph B, PPV and percentage of patients alerted) and utility (graph C, median hours of first alert to clinical event and graph D, percentage of first alerts arriving before the clinical events). By drawing a vertical line across the 4 plots, the full range of model performance possibilities across each EEU factor and the trade-offs required between each can be ascertained.

Figure 3.

Novel standardised model evaluation graph set, all with the same x-axis (sample level sensitivity). From the top: (A) Precision-recall curve evaluated at the sample level; (B) Admission level sensitivity (dotted blue line, dot markers), positive predictive value (blue line, dot markers) and proportion of patients alerted (dashed blue line, dot markers); (C) Median hours from first alert to: Sepsis-3 criteria met (red crosses), blood culture order (blue dot), antibiotics ordered (orange dots), blood culture or antibiotics order (black dots) and Q-ADDS system reaching a score of 4, mandating doctor review of patient (yellow dots); and (D) the proportion of first alerts occurring before the same set of clinical events as for (C) using the same coloured markers. Abbreviation: Q-ADDS = Queensland Adult Deterioration Detection Score.

In the sepsis prediction case study, the clinician committee used this integrated visual display to:

1. identify that the model was unsuitable to use in clinical practice at an alert threshold that conferred a sensitivity of 80% (admission level) because most first alerts would arrive at the same time clinicians were already suspecting sepsis (reflected in the ordering of blood cultures or antibiotics), or at the same time that the existing Q-ADDS system was triggering a patient review (see Figure 4, Threshold A and Table 4);

2. choose an alert threshold that optimally balanced all three EEU factors ie, clinically acceptable sensitivity, PPV/NNE and alert lead time (See Figure 4, Threshold B).

Figure 4.

The same graphs B, C and D from Figure 3, but now showing the selection of two model alert threshold selections (thick vertical black lines). The original threshold selection (Threshold A) was based on an admission-level sensitivity of 0.8 using conventional methods. The balanced threshold selection (Threshold B) shows how effectiveness (Graph B), efficiency (Graph B) and utility (Graphs C and D) were balanced to select a threshold that generated first alerts prior to either antibiotics or blood culture orders and prior to Q-ADDS level 4 being reached requiring clinician review.

Accordingly, in this case study, the alert threshold of 0.12 was deemed optimal by consensus of the clinician committee (see Table 4 column 3).

Discussion

A key objective of evaluating AI prediction models that continuously monitor patient data is to identify the best performing model and its alert threshold that optimally balances effectiveness, efficiency and utility when applied to clinical practice. Using conventional discriminative metrics, like AUROC^1,2,43 and AUPRC to select models and using DCA to set alert thresholds do not address these objectives because they were designed to evaluate single-use risk assessment tools where timing of alerts in relation to clinical events was not a variable, despite its importance when assessing the clinical utility of any model-based tool. Alerts arriving after clinicians are aware of the event being predicted are distracting, waste time, worsen alert fatigue and erode tool trust and adoption.^1,2,8–10 Our proposed methodology is the first standardised and easily visualised approach that enables clinicians and healthcare organisations to systematically evaluate prediction models for their ability to balance all three EEU factors across the full range of alert thresholds.

Limitations of existing methods for evaluating AI-based prediction models

Conventional methods for evaluating AI-based prediction models using patient surveillance data are not standardised and often ambiguous, invalid or insufficient for clinicians to properly assess such models. We found the use of different evaluation datasets (sample and admission level) is common practice but confusing and misleading. Sample level datasets contain all predictions generated throughout each admission (eg, all hourly predictions in our case study), whereas admission level datasets contain just one selected sample, usually that with the maximum model probability output. Sample level datasets are often biased towards the sickest patients who spend more time in hospital (ie, more hourly samples), which explains why in our case study, Sepsis-3 prevalence in the sample level Test dataset was 2.8 times higher than in the admission level dataset. On the other hand, use of an admission level evaluation dataset with a single selected sample based on highest probability of sepsis, which in our study yielded a 13.4% higher AUROC than for the sample level dataset, is misleading and should not be used. The reason is that all discriminative metrics, like AUROC, measure true and false positive rates at each alert threshold between 0 and 1 and, by selecting a single sample for each admission, an admission level dataset omits all other threshold samples, thus providing true and false positive values for a fraction of the model outputs, in our study, just 2% of all samples. Also, if the sample with the maximum model probability output is selected for each admission, then the admission level dataset is biased towards higher alert thresholds (ie, higher likelihood of the predicted event), explaining the higher AUROC. For these reasons, conventional metrics (see Table 4, column 3 and ‘reported’ sub-column) calculated from a single admission level dataset are invalid as, compared to sample level dataset metrics, they do not reflect future performance when deployed in real-world clinical settings.

Conventional metrics also have other serious limitations. Firstly, when implementing AI-based surveillance models, most hospitals will introduce an alert suppression strategy to minimise alert fatigue, usually suppressing alerts after the first one which, in most cases, has elicited a clinician response,^32,44–49 which would equal removal of 98% of all alerts in our study. Therefore, evaluating a model on an all-alert-basis is discordant with how it will be used. Secondly, knowing the PPV, specificity and sensitivity of the model on a sample basis, rather than on an admission basis, is contrary to how clinicians interpret and apply these metrics. For model selection, most clinicians want to know what proportion of sepsis events are identified by the model (sensitivity) in a timely manner (alert lead time) and how many additional patients are wrongly alerted (reflected in PPV and NNE). The relationships between sensitivity and PPV are different between sample and admission level evaluations. For example, in our study, PPV was 50% at the sample level versus 15% at the admission level (ie, 1 in 2 versus 1 in 7 alerted patients were correct). We conclude that using sample level datasets directly to select the best model or alert threshold would be misleading and likely non-optimal.

Advantages of proposed novel, standardised method for model evaluation

Our proposed methodology for synthesising and presenting prediction model evaluations enables clinicians and other decision-makers to transparently assess the EEU factors of models across all alert thresholds. This approach overcomes the limitations of current conventional methods in three novel ways: (i) The sample level dataset is used as the basis for model evaluation but admission level metrics are generated for each threshold, which overcomes the bias and other limitations mentioned previously and which more truly reflects the as-implemented performance of the model for all thresholds; (ii) The measurement of alert timing relative to specific clinical actions, rather than just the predicted event itself (ie, meeting Sepsis-3 criteria), and ascertaining the proportion of first alerts arriving before each action, indicate more realistically how much and how often the alert can provide advance warning to clinicians, by which they can better appraise model utility; and (iii) Model evaluation is visually presented in a way that enables clinicians to directly view the interplay between all EEU factors and to consider different scenarios in which these factors will interact differently (see Figure 4), easily appreciated by vertically aligning a graph for each metric using a common, identically scaled x-axis, and then drawing a vertical line at any point along the x-axis.

In our sepsis case study, this novel evaluation methodology led to the transparent selection of an optimal model threshold very different to the one chosen using conventional metrics. Our clinicians regarded utility (ie, the ability to identify and treat patients with evolving sepsis earlier than clinical intuition or existing early warning system would allow) was the factor most attractive of clinician use of the AI tool. Accordingly, model efficiency (ie, ability to minimise false alerts) was sacrificed (see Figure 4) until the model could provide at least 1 hour, on average, of advance warning before clinicians themselves suspected sepsis and initiated a response. This corresponded to an alert threshold of 0.12 whereas conventional evaluation generated a very different threshold of 0.25, with a much higher efficiency (ie, higher PPV) but with most alerts providing no advance warning to clinicians.

Implications for research and practice

Based on our findings, we suggest conventional evaluation metrics, such as AUROC, AUPRC, Matthews correlation coefficient,⁵⁰ model calibration¹⁴ and decision curves¹⁵ are insufficient for either appraising performance of AI prediction models or selecting the optimal alert threshold for real-world continuous patient surveillance. Although contrary to current practice,¹⁴ conventional metrics do not incorporate alert timing relative to clinician intuition and actions within their calculations, despite this being a key utility factor affecting adoption and care outcomes. A high performing (high AUROC) model in our study was not seen as clinically useful, and many other deployed models may have similar limitations, which may partly explain why reported impacts of implemented prediction tools on care outcomes are mixed or negative.⁴

We recommend model developers, working with clinician users, adopt our proposed methodology for model evaluations. It can be accessed through an open-source python library we have created for this purpose (https://github.com/esalient/pysalient), making it easier for data scientists to execute consistent model evaluations and empowering clinicians to decide on model suitability and threshold selection in preparation for prospective evaluations and pilot trials.

We also recommend that research reporting guidelines for AI prediction models, such as TRIPOD-AI,¹¹ request authors to clearly report how and why sample or admission level datasets were used in model evaluations in reducing the risk of reporting misleading or invalid results.

Using the proposed methodology requires that clinicians are trained and take additional time to review model evaluation and alert threshold selection. This is a more complex process than simply comparing AUROC values and may incur additional time and cost. However, we contend that incorporating clinicians at this early stage, prior to model implementation, may prevent inadequate models from being implemented or setting model thresholds that provide little utility for clinicians.

Strengths and limitations

Our methodology for considering all EEU factors in selecting models and their alert thresholds can also be customised in several ways. Firstly, certain metrics can be replaced in that, while we utilised proportion of patients alerted as one efficiency metric, other metrics could be incorporated, such as mean alert count per day. Secondly, new plots could be added vertically, for example in evaluating potential bias within patient sub-groups. Adding effectiveness and efficiency graphs for a potentially biased racial sub-group could visually suggest the use of different alert thresholds for those groups, or at least highlight the impact of using a single model for that sub-group.¹⁶ Finally, while our case study was directed at sepsis prediction, the evaluation methodology is applicable to other prediction tasks for which different clinical events may serve as better indicators of clinical utility than the predicted event.

The use of the proposed methodology requires clinician training and validation studies are required in assessing its downstream impact on clinical care and patient outcomes.

Conclusion

Although many AI-based surveillance prediction tools have been implemented, evidence of improved care remains limited, which in part may be due to poor performance and utility of implemented models with consequent low clinician adoption. Conventional methods for evaluating model performance are limited and, depending on the type of evaluation dataset used, can be biased and misleading. A novel EEU evaluation methodology overcomes these limitations and provides a standardised way for decision makers and clinicians to assess model suitability for their clinical environment and to choose an optimal alert threshold prior to prospective evaluations and pilot trials.

Author contributions

Conceptualisation: AHV, VKC, IAS, MR ADI, SMM, BV, PJL; Data curation: AHV, AA-H, TMI; Formal analysis: AHV, RJS, IAS, AA-H; Funding acquisition: IAS, MR, SMM, BV, PJL, AHV; Investigation: AHV, VKC, AA-H, TMI; Methodology: AHV, RJS, VKC, IAS, SFW, AA-H, TMI, SMM, BV, PJL, RW, KMW, AS; Project administration: IAS, MR, TMI, SMM; Software: AHV, RJS, AA-H; Supervision: AHV, IAS, MR, SMM, BV, PJL; Validation: AHV, VKC, IAS, SFW, AA-H, TMI, BV, PJL, RW, KMW; Visualisation: AHV, VKC, TMI; Writing—original and draft: AHV, IAS, VKC; Writing—review and editing: All authors.

Anton H. van der Vegt (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing—original draft, Writing—review & editing), Victoria Campbell(Investigation, Methodology, Validation, Visualization, Writing—original draft, Writing—review & editing), Robert Webb(Investigation, Methodology, Validation, Writing—review & editing), Balasubramanian Venkatesh(Conceptualization, Funding acquisition, Methodology, Supervision, Validation, Writing—review & editing), Paul J Lane(Conceptualization, Funding acquisition, Methodology, Supervision, Validation), Kathryn Wilks(Methodology, Validation, Writing—review & editing), Steven McPhail(Funding acquisition, Methodology, Project administration, Supervision, Writing—review & editing), Michael Rice(Funding acquisition, Project administration, Supervision, Writing—review & editing), Tara Isaacs(Investigation, Methodology, Project administration, Validation, Visualization, Writing—review & editing), Stephen Whebell(Methodology, Validation, Writing—review & editing), Adam Irwin(Conceptualization, Validation, Writing—review & editing), Amith Shetty(Methodology, Writing—review & editing), Ahmad Abdel-Hafez(Formal analysis, Investigation, Methodology, Software, Validation, Writing—review & editing), and Rudolf Schnetler(Formal analysis, Methodology, Software, Writing—review & editing), Ian A Scott(Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing—original draft, Writing—review & editing).

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was funded by Clinical Excellence Queensland; and the Queensland Technology Future Fund, Department of Health, Queensland Government. A.H.V. was funded through a Queensland Government, Advanced Queensland Industry Research Fellowship grant.

Conflicts of interest

None declared.

Data availability

The data underlying this article cannot be shared publicly due to ethical and legal reasons. The data will be archived and accessible to research groups through Queensland Health after all ethical governance requirements have been met and data custodian approval received.