Improving End-of-Life Screening in the Emergency Department with Collaborative Artificial Intelligence

Abstract

Study Objectives:

To compare end-of-life predictions as measured by the physician-answered Surprise Question (SQ, “would you be surprised if this patient died in the next 6 months?”), the Geriatric End-of-Life Screening Tool (GEST) artificial intelligence (AI) model, and a new collaborative GEST+SQ model for predicting 6-month mortality in older emergency department (ED) patients.

Methods:

This was a single-site prospective cohort study (Nov 2022 - June 2023) at a tertiary academic ED of patients aged ≥65 years. Answers to the SQ were collected within the electronic health record (EHR) at ED disposition and GEST scores were calculated from available records using lab, vital sign, demographic and historical data. Six-month mortality was adjudicated via EHR and state records. SQ and GEST were compared using sensitivity and specificity. A new logistic regression model was developed combining SQ and GEST (GEST+SQ) and compared to GEST alone using area under receiver-operating characteristic curves (ROC-AUC) for discrimination and expected calibration error (ECE) for calibration. We modeled a sequential screening pathway where low- and high-risk patients received only GEST screening while intermediate risk patients received both GEST and SQ, reporting the proportion of patients for whom adding the SQ to GEST would change a theoretical referral to intervention.

Results:

From 9,256 eligible patients, 3,479 had SQ responses (37.6%), with 13.3% 6-month mortality. When matching GEST sensitivity to SQ (83.8%), GEST had greater specificity than the SQ (61.5% [56.7–67.1] vs. 50.8% [49.1–52.6]). At matching specificity (50.8%), GEST sensitivity (90.0% [87.0–92.7]) exceeded the SQ (83.8% [80.3–87.0]). GEST had an ROC-AUC of 0.79 (0.77–0.81), while the GEST+SQ model had ROC-AUC of 0.80 (0.78–0.82). The GEST+SQ model had significantly improved ECE of 0.01 (0.01–0.02) for GEST+SQ vs. 0.042 (0.03–0.05) for GEST alone. In a sequential screening pathway, as few as 5% of patients required SQ screening following GEST risk scoring.

Conclusions:

GEST modestly outperformed the SQ for predicting 6-month mortality. A GEST+SQ collaborative model did not improve discrimination (ROC-AUC) over GEST alone, but improved calibration. Sequential screening using GEST and then the SQ for intermediate risk patients could decrease physician screening burden by 95% relative to manual, SQ-only screening. Collaborative approaches integrating automated tools with targeted physician input may enhance ED mortality risk assessment while reducing clinician effort.

Introduction:

Mortality risk assessment is implicit in much of emergency medical care. Near-term (i.e., ≤ 30 day) mortality risk factors heavily into workup intensity, disposition, and level-of-care decision-making. For older adults, 75% of whom are seen in the emergency department (ED) within the last 6 months of life,^1,2 assessment of longer-term mortality risk (≥ 6 months) is a cornerstone of evidence-based tools for shared decision-making³, ED-based serious illness communication⁴, palliative care referral⁵, comprehensive geriatric assessments⁶, and transitions of care⁷, among other diverse processes. Similarly, mortality risk is tightly linked to patient complexity and multimorbidity⁸, which connects to critical themes in ED quality including diagnostic quality⁹ and confidence¹⁰, workup intensity¹¹, and operational throughput.^12,13 Taken together, mortality risk is a latent variable highly relevant to the care of older adults in the ED. Many of the implications of mortality risk relate to broader goals of care. in an ideal setting this would be comprehensively discussed by the patient’s outpatient team before an ED visit, but in practice, basic goals of care documentation such as advance directives are available less than 50% of the time in an ED encounter, even in patients with chronic comorbidities, nursing facility residents, or older patients.¹⁴ Thus, ED physicians often must start from a blank slate when addressing issues related to mortality and goals of care.

Given the implicit importance of longer-term mortality risk among older adults in the ED, several tools have been developed to measure ED physician gestalt for mortality risk. We consider the best studied of these tools – the surprise question (SQ). The SQ, “would you be surprised if this patient died in the next six months” is validated in the ED, among other settings, where it predicts both mortality risk and resource utilization¹⁵, but, like other manual screening tools, requires clinician participation at each encounter for each eligible patient.^16–21 Nevertheless, health systems are now systematically incorporating this screening tool into ED operations.^19,22

To reduce screening burden on clinicians while improving mortality risk assessment for older adults in the ED, we recently developed and externally validated¹an automatable electronic health record (EHR) machine learning algorithm called the Geriatric End-of-life Screening Tool (GEST). GEST, an artificial intelligence (AI) logistic regression algorithm, uses commonly available EHR data including age, vital signs, and select labs to predict mortality and does not require clinician input.^23,24 While predictive models, including AI models, are increasingly prevalent in emergency medicine, comparison to usual care or clinician impression are often lacking.^21,25–28 To address this gap, the objective of this study is to compare end-of-life predictions by the SQ, GEST, and by a new combined model incorporating SQ and GEST (GEST+SQ), both in head-to-head performance analysis as well as in a hypothetical sequential screening application where GEST is used to pre-screen for SQ focusing on prognostic performance and potential reductions in clinician bedside screening effort.

Methods:

Study Design:

This is a single site prospective cohort study conducted between 11/1/2022 and 06/30/2023 at Beth Israel Deaconess Medical Center, a tertiary care academic ED in Boston, MA. During the study period, the SQ question was incorporated into the ED disposition activity for all patients ≥65 years of age. Due to institutional policies, the SQ was implemented as a “soft-stop” at the time of patient admission or discharge and could be skipped without interrupting the clinical workflow.²⁹ The SQ was only recorded only once per ED encounter and was not linked with further interventions. Patients with SQ responses were included in this study. Patients with a home address outside of Massachusetts were excluded due to unreliable mortality data. Only the index visit (first encounter in the study period) for each patient was included. Patient race, ethnicity, and sex were sourced from the EHR as collected during the registration process. Patients who died within 24 hours of ED arrival were excluded in alignment with the GEST derivation and validation studies. We used the TRIPOD+AI reporting guideline.³⁰ The study was approved by the BIDMC Institutional Review Board (2022P000795).

GEST scoring:

GEST scores were calculated using EHR data four hours into the patient encounter using parameters from the previously published method including age, complete blood count and basic metabolic panel data, need for supplemental oxygen in the ED, and data about ED diagnosis, prior diagnoses, and admissions.^23,24 Data were centered and missing data mean-imputed according to the prespecified GEST model. Mortality was assessed using EHR and Massachusetts death records which were matched using previously described methods.²⁴ Last names, birth dates, and sex required exact matches, while first names were matched probabilistically. The GEST score is an estimate of the patient’s six-month mortality risk (0–100%).

Methods of comparison between GEST and SQ:

Summary statistics of the patient cohort as well as the SQ-positive and SQ-negative subgroups were calculated. We report demographics and mortality of groups who had the SQ answered and not answered, as well as the SQ responses.^31–33 Factors associated with the presence or absence of SQ response were analyzed using logistic regression.

We calculated sensitivity and specificity for six-month mortality for SQ responses and for GEST. GEST sensitivity was reported at the operating point where its specificity was equal to the SQ, and vice versa for specificity. Sensitivity, sensitivity and F1 score (the harmonic mean of precision and recall) were calculated for each user. Factors associated with improved F1 score were analyzed using logistic regression. Calibration curves were plotted using a sigmoid curve fit. Population-weighted net reclassification improvement (NRI) was also used to measure the number of cases correctly reclassified by GEST relative to SQ.³⁴ Confidence intervals were calculated using 10,000 bootstraps. A sensitivity analysis was performed by responding provider type (attending or resident). SQ inter-rater reliability (IRR) was assessed separately using Krippendorff’s alpha³⁵ in a cohort of patients with repeat encounters containing SQ responses, allowing within-patient comparison across encounters. As described above, repeat encounters were not included in any other analyses. IRR bootstraps used 100 replicates.

Combined model training and evaluation:

A logistic regression model was trained using ten-fold cross-validation to predict six-month mortality in our cohort using either the SQ alone, GEST alone, or GEST+SQ. Receiver-operating characteristic curves (ROC) and calibration curves were calculated for each model and summarized with area under the ROC curve (ROC-AUC), measuring discrimination and expected calibration error (ECE) respectively. ECE, a standard calibration metric in machine learning, measures how well predicted probabilities match true event rates.³⁶ We used ECE rather than the Brier score because the Brier score is a composite of discrimination and calibration, rather than isolated calibration.³⁷ The final model used in sequential screening modeling resulted from averaging coefficients across all folds. Code was written in Python version (v3.12.3), models were implemented in pure Python and scikit-learn (v1.6.1)³⁸, and statistical analysis was performed with the Scipy (v1.15.2)³⁹ and Statsmodels (v0.14.4)⁴⁰ packages.

Sequential GEST+SQ screening pathway modeling:

We modeled the integration of GEST into an ED-based screening workflow using the cohort of patients with SQ responses.²⁴ In the baseline approach, all patients aged ≥65 years are screened using the SQ, as is current practice at some hospitals.^15,19 In contrast, our sequential screening strategy involves identifying all patients above a pre-specified mortality risk threshold - we chose 20%, 30%, and 40% thresholds, in keeping with prior work.^23,41,42 In this analysis, we used the GEST+SQ model to assign each patient into a low, intermediate, or high-risk group. A patient was low risk if their GEST score was sufficiently low such that a SQ response of “no” would not reassign them above the mortality risk threshold, while for the high-risk group, the GEST score was sufficiently high such that the SQ response of “yes” would not reassign them to the low-risk group. A patient was intermediate risk if crossing the pre-specified mortality risk threshold depended on the SQ response. In a sequential screening scenario, only intermediate-risk patients required clinician SQ screening.

Results:

During the study period, there were a total of 12,939 encounters by 9,256 patients aged ≥65 years; 3,479 patients (37.6%) had SQ responses after excluding patients who expired within 24 hours (Supplementary Figure 1). The mean age of this SQ cohort was 77.8 (SD 8.3) years, 54.5% of patients were female, 7.0% were Hispanic, and 66.9% were white (Table 1). The proportion of admissions was 62.2%, the average GEST score across the entire cohort was 11.4%, and 462 (13.3%) patients died within 6-months. Comparison with the GEST derivation demographics is shown in Supplementary Table S1.

Table 1:

Study demographics. Statistics are shown for the subset of the population with “Yes” and “No” answers to the surprise question. For each categorical variable, n and N indicate the number of patients in a given category and the total number of patients with data available for that variable, respectively.

		Surprise “No”	Surprise “Yes”
Total n		1870	1609
Sex, n/N (%)	Female	994/1870 (53.2)	903/1609 (56.1)
	Male	876/1870 (46.8)	706/1609 (43.9)
Hispanic ethnicity , n/N (%)		114/1795 (6.4)	131/1572 (8.3)
Race, n/N (%)	AI/AN	1/1809 (0.1)	2/1592 (0.1)
	Asian	111/1809 (6.1)	76/1592 (4.8)
	Black	331/1809 (18.3)	395/1592 (24.8)
	Native Hawaiian or other Pacific Islander	1/1809 (0.1)	0/1592 (0.0)
	Other	74/1809 (4.1)	82/1592 (5.2)
	Unknown	61/1809 (3.4)	17/1592 (1.1)
	White	1291/1809 (71.3)	1037/1592 (65.1)
Admitted, n/N (%)		1433/1870 (76.6)	730/1609 (45.4)
Age, mean (SD)		79.6 (8.7)	75.7 (7.3)
GEST Score, mean (SD)		15.6 (16.6)	6.6 (8.8)
Six month mortality, n/N (%)		387/1870 (20.7)	75/1609 (4.7)

“GEST Score” is the output of the model studied in this paper. It is an estimate of the probability of patient mortality in the next 6 months, ranging from 0–100%, with greater values indicating higher risk of mortality. Prior literature has used estimated mortality of roughly 30% as a clinically meaningful threshold for palliative referral.⁴⁰

Among included encounters, 387 of the 1,870 patients with a SQ response of “No” (would not be surprised) experienced 6-month mortality (20.7%), as compared to 75 of the 1,609 patients with a response of “Yes” (4.7%). Patients for whom the SQ was answered were older, less likely to be Hispanic or white, and had higher GEST scores and mortality than those without the SQ (Supplementary Table S2). Of SQ responses, 91.3% were from resident physicians (Supplementary Table S3). Patients who had attending responses were younger, had lower GEST scores, lower admission rates, and lower mortality rates than those seen by residents (Supplementary Table S3). The median number of SQ responses per physician was 4 (IQR: 2–12.5). The treating physician’s SQ response rate was more strongly associated with SQ response than patient age, sex, race, ethnicity, disposition, GEST score, or whether the physician was a resident or attending (Supplementary Figure S2).

The SQ had a sensitivity of 83.8% (80.3–87.0%) and specificity of 50.8% (49.1–52.6). This performance is similar to other studies of the SQ.¹⁶ At a matching sensitivity, GEST had a specificity of 61.5% (56.7–67.1) while at matching specificity, GEST had sensitivity of 90.0% (87.0–92.7) (Figure 1). GEST outperformed resident physician SQ responses (GEST sensitivity 0.90 [0.87–0.93] and specificity 0.61 [0.56–0.67] vs resident SQ sensitivity 0.84 [0.80–0.87] and specificity 0.50 [0.48–0.52]) but confidence intervals overlapped with attendings’ performance in predicting mortality (GEST sensitivity 0.83 [0.67–0.96] and specificity 0.58 [0.06–0.79] vs attending SQ sensitivity 0.87 [0.71–1.0] and specificity 0.63 [0.57–0.68]) (Supplementary Figure S3). NRI for GEST versus the SQ ranged from 0.28 (0.25–0.30) to 0.32 (0.29–0.34), favoring GEST (Supplementary Figure S4). The strongest predictor of physician accuracy was average GEST score of their patients (log odds-ratio 0.51 [−0.029–1.05]), but the confidence interval included zero, as it did for resident vs attending status, rate of SQ response, and patient age, disposition, race, ethnicity, and sex (Supplementary Table S4). Inter-rater reliability was assessed separately using the subset of 400 patients with repeat SQ evaluations during the study period – these repeat encounters were not used in the other analyses. Krippendorff’s alpha was 0.190 (0.113–0.292).

Figure 1:

ROC-AUC of GEST vs clinician gestalt using the Surprise Question

ROC curve for GEST model achieves higher sensitivity and specificity than clinicians answering the Surprise Question (SQ). GEST is shown as an ROC curve while clinicians, with binary answers, are shown as points. GEST sensitivity and specificity at operating points that match the SQ are shown with horizontal (sensitivity at matching specificity) and vertical (specificity at matching sensitivity) lines representing a 95% confidence interval on the corresponding value. The SQ is marked with 95% confidence intervals as well. There is considerable variability in the sensitivity and specificity of individual clinicians’ answers to the SQ.

The combined GEST+SQ model (Supplementary Figure S5) had an ROC-AUC of 0.80 (0.78–0.82) compared to the cross-validated ROC-AUC of 0.79 (0.77–0.81) for GEST. The combined model had an expected calibration error of 0.01 (0.01–0.02) compared to the GEST error of 0.04 (0.03–0.05) (Figure 2, Supplementary Figure S6).

Figure 2:

Calibration of GEST and the combined GEST+SQ model

A model combining the GEST score with clinician gestalt represented by the Surprise Question (SQ) achieves better calibration, in that the predicted p(mortality) from the model more closely estimates the observed mortality rate. A calibration curve matching the dashed line would have perfect performance, i.e., its predicted and empiric mortality rates are equal. In this figure, observed mortality rates are smoothed as described in the methods. Binned probabilities are shown in Supplemental Figure S5.

Using mortality risk cutoffs of 20%, 30%, and 40%, sequential screening models indicated 81.2%, 89.6%, and 93.7% of patients had sufficiently low GEST scores to not require SQ screening because even a positive response would not cross the mortality risk cutoff, while 4.7%, 2.3%, and 1.3% of patients had scores that were sufficiently high that even a negative SQ screen would not cross the mortality risk threshold (Figure 3). This left 14.1%, 8.1%, or 5.0% of patients assigned intermediate-risk and requiring SQ screening.

Figure 3:

Modeling sequential palliative needs screening using GEST to automatically rule-in and out high- and low-risk patients, respectively and then SQ for intermediate-risk patients.

At proposed palliative referral thresholds of 20%, 30%, and 40% mortality risk, the combined-model referral strategy involves prompting clinicians for input (i.e., provide a SQ response) on 14.1%, 8.1%, and 5.0% of patients, respectively. The intermediate-risk range is highlighted in the purple and indicates patients for whom, if the SQ answer is “no”, the GEST+SQ model would indicate mortality risk above the referral threshold (horizontal gray bar), while if it is ”yes”, the model would indicate risk below the threshold. For all other patients, the GEST+SQ predicted risk would be either below (left, light blue shading) or above (right, light orange shading) the referral threshold regardless of the SQ answer. A histogram showing the percentage of patients with each given GEST score is shown for reference.

Discussion:

In this single-site, prospective cohort study, we found that GEST, an automated EHR-integrated algorithm, had slightly superior prognostic performance to treating clinicians in identifying older adults with 6-month mortality risk. When combining GEST with SQ, model calibration, the reliability of the assigned risk values, improved, but discrimination, the accurate ranking of predictions, did not. We note that in clinical decision support, calibration may be more important than discrimination because calibration focuses on predicting accurate risk for the current patient rather than ranking them relative to the whole population.⁴³

These data suggest that SQ, GEST, or the collaborative clinician-AI GEST+SQ model are reasonable approaches for mortality prediction in the ED. Moreover, we performed a modeling analysis where a sequential screening approach using AI (i.e., GEST) pre-screened patients, ruling in and out high risk and low risk patients, respectively, saving clinician input (i.e., SQ) for the intermediate cases. We found that as many as 95% of patients would not require clinician input with this approach. Given the observed performance equipoise between GEST and SQ, the clinician effort savings of GEST+SQ may be preferable for ED teams implementing mortality risk assessment tools for older adults.

Automated mortality risk screening with GEST has several potential applications. First, mortality risk assessment has operational implications in anticipating resource use and screening for care transition needs.⁷ Second, GEST may be a useful quality assurance tool, both for screening for high-risk cases for adverse outcomes and for risk-adjustment in analyses of clinician- and site-level care variation. Finally, GEST can be used to trigger palliative care referral or ED team-driven serious illness communication, depending on department resources. ^1,4,5,44

There are numerous ED screening tools for the older adult ED population, many of which have partially overlapping target outcomes (e.g., mortality, frailty, functional decline, ED revisit), but systematic implementation of these tools is limited, even among accredited geriatric emergency departments.⁴⁵ A critical step to filling this gap is transitioning from clinician-driven screening (i.e., physicians, nurses, social workers), to automated systems tightly integrated into electronic health record (EHR) workflows.⁴⁶ While these automated tools have proliferated, few have been compared to physician judgement, and, therefore, their clinical applicability and adequacy is unknown.^27,28 While GEST was not trained to predict other adverse ED outcomes for older adults, the strong associations between mortality risk and frailty, functional decline, and revisits, introduce the possibility that GEST may be able to augment or replace multiple ED screening tools, further streamlining ED processes.¹⁴

While not explicitly analyzed in this work, the advent of large language models (LLMs) raises the potential for future iterations of tools like GEST with significantly improved performance. LLMs alone typically do not show clear benefit over classical machine learning methods for the type of probabilistic risk stratification discussed here, but they are able to leverage a new source of data in the form of unstructured text.⁴⁷ We anticipate that LLM-based feature extraction will provide new, highly informative features for models like GEST resulting in even higher predictive performance.⁴⁸

Limitations of this work include data from a single academic medical center and the exclusion of patients with primary residence outside of MA due to restrictions on mortality capture. The high admission rate (62.2%) limits generalizability. A large proportion of responses were by resident physicians; patients with SQ responses by resident physicians were older and higher-mortality than those with attending responses, both of which limit head-to-head SQ and GEST comparison. An important limitation is missingness in responses to the SQ, which was primarily driven by provider response rate, rather than patient attributes. This aligns with prior literature on clinician-level variation in EHR use patterns and while this missingness would be addressed by mandatory responses, also called hard-stops, prior work has suggested that these can lead to misleading responses.^29,49 The low response rate highlights the difficulty of operationalizing mortality screening in the ED but supports the goal of using models like GEST+SQ to reduce the physician burden of screening tools.

Taken together, we argue that the approach of using AI to pre-screen for clearly high- or low-risk patients while relying on clinicians for intermediate cases has potential broad applicability to ED processes. EDs increasingly support broad individual and population health screening objectives including substance use, suicide, housing instability, elder safety, and others.^50–52 While, individually, these may be feasible, the cumulative effects of screening programs can burden ED teams. We propose that this sequential, collaborative AI-clinician approach may be applied to many other ED screening activities, saving both clinician effort and enabling a wider array of interventions.

Data Sharing Agreement:

Data are not available for sharing due to PHI.

Code Sharing:

Code is available upon request.