Development and exploratory external assessment of a machine learning model for early delirium prediction in non–mechanically ventilated patients with sepsis

Abstract

Delirium is a common and severe complication of sepsis in critically ill patients and is associated with increased mortality and prolonged hospitalization. Early identification of high-risk individuals remains challenging, particularly among non–mechanically ventilated patients who are less frequently represented in prediction studies. Using routinely collected electronic health record data, we performed a retrospective multicenter study to establish and validate an early prediction model for delirium in non–mechanically ventilated adults with sepsis. Patients from the MIMIC-IV database were used for model development, and an independent cohort from the eICU Collaborative Research Database was used for exploratory external assessment. A total of 3,015 patients in the development cohort and 1,119 patients in the exploratory test cohort were included. Among the evaluated algorithms, XGBoost achieved the best discrimination, with an area under the receiver operating characteristic curve of 0.807 in internal validation. The eICU cohort contained only 10 delirium cases (incidence < 1%), forming a sparse-event dataset; although the XGBoost model yielded an AUC of 0.833 in this cohort, the calibration and decision curve analysis results should be interpreted with extreme caution due to limited positive events. Lower Glasgow Coma Scale scores, higher illness-severity indices, exposure to sedative or analgesic agents, and metabolic disturbances such as elevated anion gap and abnormalities in hematocrit and calcium were the strongest contributors to prediction. These predictors align with recognized features of sepsis-associated brain dysfunction. This externally validated model may support early risk stratification and targeted preventive strategies in this vulnerable population.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1007/s10238-026-02306-0.

Introduction

Delirium, a manifestation of acute brain dysfunction, has emerged as one of the strongest predictors of long-term cognitive impairment, institutionalization, and post-discharge mortality among intensive care unit (ICU) survivors. In patients with sepsis, sepsis‑associated delirium (SAD) reflects a convergence of neuroinflammation, blood–brain barrier injury, cerebral perfusion abnormalities, and neurotransmitter dysregulation, and is increasingly recognized as a distinct clinical entity within the spectrum of sepsis‑associated encephalopathy [1]. Beyond its in‑hospital impact, delirium is linked to increased early post‑discharge mortality and healthcare utilization, underscoring the need for proactive risk identification before overt cognitive failure ensues [2].

Contemporary critical‑care guidelines call for routine bedside screening and prevention of delirium using validated instruments—most commonly the Confusion Assessment Method for the ICU (CAM‑ICU) and the Intensive Care Delirium Screening Checklist (ICDSC) [3, 4]. Recent psychometric work also emphasizes delirium severity quantification (e.g., CAM‑ICU‑7), facilitating severity‑aware prognostication and outcome tracking [5]. However, despite standardized screening and prevention frameworks, real‑world implementation remains variable, and high‑risk patients can still be missed early in their ICU course, when prevention may be most effective [3–5].

Several prediction models for ICU delirium have been proposed, including both traditional regression-based approaches and more recent machine learning (ML) methods [6]. Although these models demonstrate variable discrimination, many were developed in heterogeneous ICU populations without distinguishing mechanically ventilated from non–mechanically ventilated patients [6–8]. Mechanical ventilation and deep sedation are themselves strong contributors to delirium risk, and pooling these populations may obscure risk patterns specific to patients who are not ventilated.

The increasing availability of large-scale electronic health record (EHR) databases has enabled the development of data-driven prediction models using routinely collected clinical variables [9]. ML algorithms are capable of capturing nonlinear relationships and interactions among predictors and may improve risk estimation compared with conventional statistical approaches [8, 10]. However, for clinical adoption, models must demonstrate stable performance across independent datasets and provide interpretable outputs that align with real-world decision-making workflows.

Therefore, the objective of this study was to develop a machine learning model for early prediction of delirium in non–mechanically ventilated adults with sepsis using multicenter EHR data, with additional exploratory cross-database assessment on an independent multicenter cohort. By focusing on a clearly defined patient subgroup and performing internal testing alongside exploratory cross-cohort analysis, we aimed to provide a framework for early delirium risk stratification.

Method

Study design

This was a retrospective, multicenter study based on two publicly accessible critical care databases: the MIMIC-IV (version 3.1) and the eICU CRD (version 2.0). The study aimed to develop and validate a ML–based model to predict delirium in non-mechanically ventilated patients with sepsis. The overall workflow included data extraction, cohort selection, data preprocessing, feature selection, model development, and performance evaluation.

Data sources

The MIMIC-IV database contains detailed clinical data of adult patients admitted to the intensive care units of Beth Israel Deaconess Medical Center (Boston, MA, USA) between 2008 and 2022. The eICU Collaborative Research Database comprises de-identified data from more than 200 hospitals across the United States, serving as an independent cohort for exploratory cross-cohort assessment. Notably, substantial heterogeneity exists between MIMIC-IV and eICU. MIMIC-IV originates from a single large tertiary academic ICU with standardized universal delirium screening and unified sedation protocols. In contrast, eICU integrates data from over 200 hospitals across the US, including a large number of community and rural ICUs, where delirium screening is not universally implemented, and sedation strategies, daily ICU management practices vary substantially across centers. These differences are the primary causes of the huge discrepancy in delirium incidence between the two databases. The researcher (Bingjie Xiang) obtained authorized access to the databases after successfully completing the data use training provided by the collaborating institution (Certificate No. 67051510). All datasets are fully de-identified in accordance with the Health Insurance Portability and Accountability Act (HIPAA). Consequently, the study was exempt from institutional review board approval and informed consent requirements.

Participants

Patients were enrolled according to the following inclusion criteria: those who fulfilled the Sepsis 3.0 diagnostic criteria, were aged between 18 and 80 years, had an ICU stay of at least 24 h, and were admitted to the ICU for the first time during the hospitalization. Patients were excluded if they had received MV at any point during the ICU stay, had a documented history of psychiatric disorders, or had incomplete delirium assessment or missing data exceeding 30% of key variables.

Delirium was assessed using the CAM-ICU, which evaluates four core features: (1) acute onset with a fluctuating course, (2) inattention, (3) disorganized thinking, and (4) altered level of consciousness. Delirium was defined as the presence of features (1) and (2) together with either feature (3) or (4), or an ICDSC score ≥ 4.

Data extraction

For each eligible patient, demographic variables, laboratory results, comorbidities, severity scores, and therapeutic interventions were obtained from the initial evaluation during the ICU stay. The extracted variables included: (1) Demographic variables — age, gender, weight, and height; (2) Severity scores — Sequential Organ Failure Assessment (SOFA), Simplified Acute Physiology Score II (SAPS II), Oxford Acute Severity of Illness Score (OASIS), Glasgow Coma Scale (GCS), and Acute Physiology and Chronic Health Evaluation II (APACHE II); (3) Laboratory parameters — complete blood count (white blood cells, monocytes, lymphocytes, red blood cells, platelets, hematocrit, hemoglobin, red cell distribution width), liver and renal function tests (alanine aminotransferase, aspartate aminotransferase, bilirubin, urea nitrogen, creatinine, albumin), electrolytes and blood gases (sodium, potassium, calcium, chloride, anion gap, lactate, pH, PCO₂, PO₂), coagulation indicators (prothrombin time [PT], thromboplastin Time [PTT], fibrinogen), and other biomarkers (creatine kinase, lactate dehydrogenase, glucose, free calcium); (4) Comorbidities — hypertension, diabetes mellitus (type 1 and type 2), chronic kidney disease, acute kidney injury, liver cirrhosis, hepatitis, stroke, coronary heart disease, myocardial infarction (MI), heart failure, cancer, chronic obstructive pulmonary disease (COPD), and hyperlipidemia; (5) Therapeutic interventions — use of sedative or analgesic agents (Sa), vasopressors, and neuromuscular blockers; (6) Clinical outcomes — length of hospital stay, ICU stay, and in-hospital mortality.

Data preprocessing and feature selection

Continuous variables were winsorized at the 1st and 99th percentiles to reduce the influence of extreme outliers. Variables with more than 20% missingness were excluded from further analysis. To address the issue of class imbalance in the dataset, a hybrid sampling strategy combining SMOTE and undersampling was employed [11–13]. Initially, the Synthetic Minority Oversampling Technique (SMOTE) was applied to generate synthetic samples for the minority class, thereby increasing its representation. Subsequently, a controlled undersampling of the majority class was performed to further balance the overall class distribution. This combined approach not only mitigates the bias towards the majority class but also enhances the model’s predictive accuracy for minority instances and improves generalization to unseen data.

The MIMIC-IV dataset was split into a training set (70%) and internal validation set (30%) using stratified random sampling, with delirium status as the stratification variable. A random seed of 123 was fixed to guarantee reproducible data partitioning, which balanced the proportion of delirium events across the two subsets and reduced partition bias. Feature selection for both logistic regression and ML models was performed on the training dataset. To ensure robust and unbiased feature selection, both Least Absolute Shrinkage and Selection Operator (LASSO) regression (lambda.1se criterion) and the Boruta algorithm were applied to the training dataset. Variables showing significant group differences and identified by both methods were retained as final predictors for model development.

Model development and testing

A traditional predictive model was constructed using logistic regression. A series of ML algorithms were developed to construct predictive models, including Decision Tree, Gradient Boosting Decision Tree (GBDT), AdaBoost, LightGBM, Logistic Classifier, Random Forest (RF), Naïve Bayes (NB), CatBoost, Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), Multilayer Perceptron (MLP), and K-Nearest Neighbors (KNN). All models were trained using the MIMIC-IV training dataset, internally validated on the MIMIC-IV validation dataset, and subjected to exploratory cross-cohort assessment on the eICU Collaborative Research Database. Model performance was evaluated using the area under the receiver operating characteristic curve (AUC), accuracy, precision, recall, F1-score, specificity, and false negative rate (FNR). Calibration performance and clinical applicability were further assessed using calibration plots and decision curve analysis (DCA), respectively. To interpret the final model, SHapley Additive exPlanations (SHAP) values were calculated to quantify the contribution of each feature to the model’s prediction. The SHAP importance plot and beeswarm plot were used to visualize feature importance and the direction of influence, while the SHAP force plot illustrated how individual features contributed to specific predictions.

Statistical analysis

Continuous variables were presented as mean ± standard deviation (SD) or median (interquartile range, IQR) depending on data distribution, and compared using the t-test or Mann–Whitney U test as appropriate. Categorical variables were summarized as counts and percentages, and compared using the χ² test or Fisher’s exact test. A p value < 0.05 was considered statistically significant. All analyses were performed using DecisionLinnc1.0 software and R (version 4.3.1).

Results

Patient characteristics

After applying the inclusion and exclusion criteria, a total of 3,015 non-mechanically ventilated sepsis patients from the MIMIC-IV database and 1,119 from the eICU database were included in the final analysis (Fig. 1). Among them, 1,145 (37.98%) and 10 (0.89%) patients developed delirium, respectively. This large discrepancy is mainly attributed to different hospital settings, delirium screening strategies and sedation practices between the two databases, as described in the Methods section. Baseline characteristics are summarized in Table 1 and supplementary Table 1. Notably, the eICU cohort only contained 10 delirium patients, forming a sparse-event dataset with limited statistical power for robust external validation. In the MIMIC-IV cohort, delirious patients were older (64 [54–71] vs. 62 [52–71] years, P = 0.01) with comparable gender distribution (P = 0.835). They showed higher severity across all scoring systems, including SOFA (5 [3–8] vs. 4 [2–6], P < 0.001), SAPS II (37 [29–46] vs. 32 [24–40], P < 0.001), OASIS (31.00 [26.00–36.00] vs. 26.00 [22.00–31.00], P < 0.001), and APACHE II (19.00 [14.00–24.00] vs. 16.00 [12.00–20.00], P < 0.001), along with a lower GCS (13.50 [11.00–15.00] vs. 15.00 [14.00–15.00], P < 0.001). Similar but nonsignificant trends were observed in the eICU cohort (P > 0.05), except for GCS (14.00 [11.25–14.00] vs. 15.00 [14.00–15.00], P < 0.001).

Fig. 1.

Flowchart of patient selection

Table 1.

Baseline characteristics of patients with and without delirium among non-mechanically ventilated sepsis patients in the MIMIC-IV and eICU databases

	MIMIC-IV			eICU
	Non-Delirium (N = 1,870)	Delirium (N = 1,145)	P	Non-Delirium (N = 1,109)	Delirium (N = 10)	P
Age	62 (52–71)	64 (54–71)	0.010	63.00 (53.00–71.00)	59.50 (54.50–66.50)	0.838
BMI	27.42 (23.50–32.85)	28.04 (23.41–33.40)	0.658	28.36 (23.42–34.31)	23.63 (22.84–28.03)	0.193
Gender, n (%)			0.835			0.754
Female	811.00 (43.37%)	501.00 (43.76%)		572(51.58%)	6(60%)
Male	1,059.00 (56.63%)	644.00 (56.24%)		537(48.42%)	4(40%)
SOFA	4.00 (2.00–6.00)	5.00 (3.00–8.00)	< 0.001	5.00 (3.00–7.00)	5.50 (4.25–6.75)	0.528
SAPSII	32.00 (24.00–40.00)	37.00 (29.00–46.00)	< 0.001	35.21 (14.11)	36.40 (14.55)	0.873
OASIS	26.00 (22.00–31.00)	31.00 (26.00–36.00)	< 0.001	26.00 (21.00–32.00)	32.50 (26.00-39.50)	0.059
GCS	15.00 (14.00–15.00)	13.50 (11.00–15.00)	< 0.001	15.00 (14.00–15.00)	14.00 (11.25-14.00)	< 0.001
APACHEII	16.00 (12.00–20.00)	19.00 (14.00–24.00)	< 0.001	61.00 (46.00–77.00)	61.50 (44.75-67)	0.880
Hematocrit (%a, %b)	30.88 (6.47)	32.15 (6.96)	< 0.001	32.80 (29.30-37.08)	34.50 (33.05–35.38)	0.245
Aniongap (mEq/La)	14.00 (12.00–17.00)	15.00 (13.00–18.00)	< 0.001	14.00 (11.00–17.00)	13.00 (10.50–14.00)	0.445
Calcium total (mg/dLa, mg/dLb)	8.20 (7.70–8.70)	8.45 (7.80–8.90)	< 0.001	8.60 (8.20–9.10)	9.20 (8.58–9.38)	0.072
Hypertension, n (%)			0.015			0.78
No	1,013.00 (68.58%)	511.00 (63.56%)		965.00 (87.02%)	9.00 (90.00%)
Yes	464.00 (31.42%)	293.00 (36.44%)		144.00 (12.98%)	1.00 (10.00%)
Acute kidney injury, n (%)			0.114			0.694
No	839.00 (56.80%)	429.00 (53.36%)		197.00 (17.76%)	2.00 (20.00%)
Yes	638.00 (43.20%)	375.00 (46.64%)		912.00 (82.24%)	8.00 (80.00%)
Liver cirrhosis, n (%)			< 0.001			1.000
No	1,270.00 (85.99%)	647.00 (80.47%)		1,030.00 (92.88%)	10.00 (100.00%)
Yes	207.00 (14.01%)	157.00 (19.53%)		79.00 (7.12%)	0.00 (0.00%)
Hepatitis, n (%)			0.001			1.000
No	1,365.00 (92.42%)	711.00 (88.43%)		1,094.00 (98.65%)	10.00 (100.00%)
Yes	112.00 (7.58%)	93.00 (11.57%)		15.00 (1.35%)	0.00 (0.00%)
Stroke, n (%)			0.051			0.016
No	1,378.00 (93.30%)	732.00 (91.04%)		1,088.00 (98.11%)	8.00 (80.00%)
Yes	393	341		21.00 (1.89%)	2.00 (20.00%)
Chronic kidney disease, n (%)			0.071			1.000
No	1,202.00 (81.38%)	629.00 (78.23%)		1,035.00 (93.33%)	10.00 (100.00%)
Yes	275.00 (18.62%)	175.00 (21.77%)		74.00 (6.67%)	0.00 (0.00%)
Cancer, n (%)			0.009			1.000
No	1,208.00 (81.79%)	692.00 (86.07%)		1,088.00 (98.11%)	10.00 (100.00%)
Yes	269.00 (18.21%)	112.00 (13.93%)		21.00 (1.89%)	0.00 (0.00%)
T2DM, n (%)			< 0.001			0.099
No	1,091.00 (73.87%)	534.00 (66.42%)		1,050.00 (94.68%)	8.00 (80.00%)
Yes	386.00 (26.13%)	270.00 (33.58%)		59.00 (5.32%)	2.00 (20.00%)
T1DM, n (%)			0.650			1.000
No	1,420.00 (96.14%)	776.00 (96.52%)		1,104.00 (99.55%)	10.00 (100.00%)
Yes	57.00 (3.86%)	28.00 (3.48%)		5	0
Hyperlipidemia, n (%)			0.842			0.217
No	1,031.00 (69.80%)	558.00 (69.40%)		1,083.00 (97.66%)	9.00 (90.00%)
Yes	446.00 (30.20%)	246.00 (30.60%)		26.00 (2.34%)	1.00 (10.00%)
Heart failure, n (%)			0.394			1.000
No	1,194.00 (80.84%)	638.00 (79.35%)		242.00 (21.82%)	2.00 (20.00%)
Yes	283.00 (19.16%)	166.00 (20.65%)		867.00 (78.18%)	8.00 (80.00%)
Myocardial infarction, n (%)			0.014			0.086
No	1,404.00 (95.06%)	744.00 (92.54%)		1,055.00 (95.13%)	8.00 (80.00%)
Yes	73.00 (4.94%)	60.00 (7.46%)		54.00 (4.87%)	2.00 (20.00%)
Coronary heart disease, n (%)			0.040			0.254
No	1,125.00 (76.17%)	581.00 (72.26%)		1,002.00 (90.35%)	8.00 (80.00%)
Yes	352.00 (23.83%)	223.00 (27.74%)		107.00 (9.65%)	2.00 (20.00%)
COPD, n (%)			0.061			0.590
No	1,321.00 (89.44%)	698.00 (86.82%)		1,015.00 (91.52%)	9.00 (90.00%)
Yes	156.00 (10.56%)	106.00 (13.18%)		94.00 (8.48%)	1.00 (10.00%)
Sedative analgesic, n (%)			< 0.001			0.129
No	1,497.00 (80.05%)	756.00 (66.03%)		847.00 (76.38%)	10.00 (100.00%)
Yes	373.00 (19.95%)	389.00 (33.97%)		262.00 (23.62%)	0.00 (0.00%)
Vasopressor, n (%)			0.854			0.380
No	1,332.00 (71.23%)	812.00 (70.92%)		928.00 (83.68%)	10.00 (100.00%)
Yes	538.00 (28.77%)	333.00 (29.08%)		181.00 (16.32%)	0.00 (0.00%)
Neuromuscular blocker, n (%)			0.034			1.000
No	1,857.00 (99.30%)	1,128.00 (98.52%)		1,095.00 (98.74%)	10.00 (100.00%)
Yes	13.00 (0.70%)	17.00 (1.48%)		14.00 (1.26%)	0.00 (0.00%)
Hospital day	8.00 (4.86–15.74)	10.13 (6.00–18.89)	< 0.001	7.22 (4.52–12.27)	13.54 (8.83–25.59)	0.023
ICU day	2.10 (1.52–3.49)	3.04 (1.88–6.09)	< 0.001	2.46 (1.71–3.83)	4.15 (2.73–7.71)	0.021
Death in current admisson, n (%)			< 0.001			0.78
No	1,739.00 (92.99%)	981.00 (85.68%)		1,004.00 (92.36%)	9.00 (90.00%)
Yes	131.00 (7.01%)	164.00 (14.32%)		83.00 (7.64%)	1.00 (10.00%)

BMI, Body Mass Index; SOFA, Sequential Organ Failure Assessment; SAPS II, Simplified Acute Physiology Score II; OASIS, Oxford Acute Severity of Illness Score; GCS, Glasgow Coma Scale; APACHE II, Acute Physiology and Chronic Health Evaluation II; RDW, Red Cell Distribution Width; PCO2, Partial Pressure of Carbon Dioxide; PH, Potential Hydrogen; PO2, Partial Pressure of Oxygen; PT, Prothrombin Time; PTT, Partial Thromboplastin Time; T2DM, Type 2 Diabetes Mellitus; T1DM, Type 1 Diabetes Mellitus; COPD, Chronic Obstructive Pulmonary Disease. Continuous variables are presented as mean (standard deviation) or median (interquartile range), as appropriate. a Measurement unit applied to the MIMIC-IV cohort. b Measurement unit applied to the eICU cohort

Delirious patients exhibited significant abnormalities in several laboratory parameters in MIMIC-IV, including higher red blood cells (3.50 [3.01–4.07] vs. 3.42 [2.85–3.92] m/uL, P = 0.009), hematocrit (32.15 ± 6.96 vs. 30.88 ± 6.47%, P < 0.001), hemoglobin (10.53 ± 2.38 vs. 10.17 ± 2.21 g/dL, P = 0.002), urea nitrogen (22.00 [13.00–39.50] vs. 20.00 [12.00–34.00] mg/dL, P = 0.003), lactate dehydrogenase (277.00 [201.00–405.00] vs. 242.00 [176.00–368.00] IU/L, P < 0.001), albumin (3.00 [2.50–3.50] vs. 2.90 [2.50–3.30] g/dL, P = 0.011), glucose ( 125.00 [103.00–177.00] vs. 122.00 [101.00–158.00] mg/dL, P = 0.04), lactate (1.90 [1.20–2.80] vs. 1.70 [1.20–2.50] mmol/L, P = 0.017), anion gap (15.00 [13.00–18.00] vs. 14.00 [12.00–17.00] mEq/L, P < 0.001), calcium (8.45 [7.80–8.90] vs. 8.20 [7.70–8.70] mg/dL, P < 0.001), and sodium (138.00 [134.00–142.00] vs. 137.00 [134.00–140.00] mEq/L, P < 0.001). In the eICU cohort, PCO2 (66.6 [45.70-71.15] vs. 41.25 [33.75–50.88] mmHg, p < 0.032), and sodium (144.00 [142.25-155.75] vs. 141.00 [138.00-144.00] mmol/L, p = 0.029) were higher in delirious patients, while other parameters showed no significant difference.

In the MIMIC-IV cohort, patients with delirium had a higher proportion of hypertension (36.44% vs. 31.42%, P = 0.015), liver cirrhosis (19.53% vs. 14.01%, P < 0.001), hepatitis (11.57% vs. 7.58%, P = 0.001), type 2 diabetes mellitus (33.58% vs. 26.13%, P < 0.001), myocardial infarction (7.46% vs. 4.94%, P = 0.014), and coronary heart disease (27.74% vs. 23.83%, P = 0.040), whereas cancer was less frequent in this group (13.93% vs. 18.21%, P = 0.009). In the eICU cohort, delirious patients had higher rates of stroke (20.00% vs. 1.89%, P < 0.001), while other comorbidities did not differ significantly between groups (P > 0.05).

In the MIMIC-IV cohort, patients with delirium were more likely to receive sedative or analgesic agents (33.97% vs. 19.95%, P < 0.001) and neuromuscular blockers (1.48% vs. 0.70%, P = 0.034), whereas the use of vasopressors was comparable between groups (P > 0.05). Similar but nonsignificant trends were observed in the eICU cohort (P > 0.05).

Delirium was also accompanied by prolonged hospitalization and worse outcomes in the MIMIC-IV cohort, including a longer ICU stay (3.04 [1.88–6.09] vs. 2.10 [1.52–3.49] days, P < 0.001), longer hospital stay (10.13 [6.00–18.89] vs. 8.00 [4.86–15.74] days, P < 0.001), and higher in-hospital mortality (14.32% vs. 7.01%, P < 0.001). Comparable trends were identified in the eICU cohort with respect to ICU and hospital length of stay: prolonged ICU stay (4.15 [2.73–7.71] vs. 2.46 [1.71–3.83] days, P = 0.021) and extended hospital stay (13.54 [8.83–25.59] vs. 7.22 [4.52–12.27] days, P = 0.023). However, no statistically significant difference in in-hospital mortality was detected between groups (P = 0.78).

Selected predictors

Using the lambda.1se criterion, Lasso regression identified nine predictive variables: GCS, OASIS, Sa, APACHE II, anion gap, hematocrit, calcium, SOFA, and type 2 diabetes mellitus (T2DM) (Fig. 2A-B). The Boruta algorithm yielded the same set of features, confirming their consistent importance in relation to the study outcome (Fig. 2C).

Fig. 2.

Feature selection by LASSO and Boruta. (A) Mean cross-validation error (CVM) as a function of the LASSO regularization parameter (λ). (B) LASSO coefficient paths across λ. (C) Feature selection using the Boruta algorithm

Model construction and testing

A traditional logistic regression model and 12 ML models were developed to predict delirium in non-mechanically ventilated patients with sepsis. The logistic regression model was expressed as:

logit(p) = − 2.617 + 1.071×Sa + 0.241×calcium + 0.043×SOFA + 0.058×OASIS − 0.373×GCS + 0.043×APACHE II + 0.044×hematocrit + 0.020×anion gap + 0.475×T2DM. In the training dataset, the logistic regression model achieved an AUC of 0.843 (Fig. 3A), while the internal validation AUC was 0.804 (Fig. 3B), indicating acceptable discrimination and good internal consistency. However, in the exploratory eICU test cohort, the model’s performance declined (AUC = 0.569) (Fig. 3C), suggesting limited generalizability.

Fig. 3.

Model discrimination across datasets. (A) ROC curve for logistic regression in the training set (AUC = 0.843); (B) ROC curve for logistic regression in the internal validation set (AUC = 0.804); (C) ROC curve for logistic regression in the eICU cohort (AUC = 0.569); (D) ROC curves of 12 machine-learning models in the internal validation set; (E) ROC curves of 6 machine-learning models in the internal validation set

Among the 12 ML models, the Random Forest (AUC = 0.838), CatBoost (AUC = 0.8298), XGBoost (AUC = 0.8066) and LightGBM (AUC = 0.8014) algorithms showed superior discrimination in the internal validation dataset (Table 2; Fig. 3D). In the exploratory external analysis based on the sparse-event eICU cohort, the XGBoost and LightGBM yielded relatively higher AUC values of 0.8334 and 0.8321, respectively (Table 3; Fig. 3E).

Table 2.

Predictive performance of machine learning models in the internal validation cohorts

Model	Accuracy	Prevalence	Recall	F1-Score	AUROC	Presicion	Specificity	FNR
Logistic Classifier	0.7115	0.3636	0.7880	0.6651	0.8055	0.5754	0.6677	0.2120
GBDT	0.7273	0.3636	0.6848	0.6462	0.7291	0.6117	0.7516	0.3152
KNN	0.6640	0.3636	0.7228	0.6101	0.6952	0.5278	0.6304	0.2772
MLP	0.7292	0.3636	0.5978	0.6162	0.7812	0.6358	0.8043	0.4022
CatBoost	0.7708	0.3636	0.7228	0.6963	0.8298	0.6717	0.7981	0.2772
SVM	0.6838	0.3636	0.7880	0.6444	0.7642	0.5451	0.6242	0.2120
XGBoost	0.7668	0.3636	0.7011	0.6862	0.8066	0.6719	0.8043	0.2989
NB	0.6660	0.3636	0.6359	0.5806	0.7141	0.5342	0.6832	0.3641
LightGBM	0.7569	0.3636	0.6793	0.6702	0.8014	0.6614	0.8012	0.3207
AdaBoost	0.7075	0.3636	0.6848	0.63	0.7026	0.5833	0.7205	0.3152
RF	0.7846	0.3636	0.6630	0.6912	0.8380	0.7219	0.8540	0.3370
DesicionTree	0.7826	0.3636	0.6630	0.6893	0.7765	0.7176	0.8509	0.3370

Table 3.

Predictive performance of machine learning models in the eICU cohort

Model	Accuracy	Prevalence	Recall	F1-Score	AUROC	Precision	Specificity	FNR
DecisionTree	0.6376	0.5425	0.5342	0.6153	0.8025	0.7253	0.7601	0.4658
RF	0.6368	0.5425	0.5342	0.6148	0.8161	0.7239	0.7584	0.4658
CatBoost	0.7566	0.5425	0.8063	0.7823	0.8145	0.7597	0.6976	0.1937
LightGBM	0.8192	0.5425	1.0	0.8571	0.8321	0.75	0.6047	0.0
XGBoost	0.8068	0.5425	0.9302	0.8393	0.8334	0.7646	0.6605	0.0698
Logistic Classifier	0.6252	0.5425	1.0	0.7433	0.6502	0.5914	0.1807	0.0

Calibration and decision curve analysis

To further evaluate the reliability and clinical applicability of this model, calibration analysis and DCA were performed. Calibration plots showed good agreement between the predicted and observed probabilities for several ML models in the internal validation dataset, such as the XGBoost and CatBoost models (Fig. 4A), indicating that their predicted risks closely matched actual outcomes. The DCA results from the large-sample internal validation confirmed that the XGBoost model provided the greatest net clinical benefit across a wide range of threshold probabilities (Fig. 4B). In the exploratory eICU test cohort, only 10 patients developed delirium, representing a typical sparse-event scenario. Although the calibration curve of the XGBoost model appeared to align with the ideal reference line visually (Fig. 4C), this apparent good calibration may be a statistical artifact caused by the extremely low disease prevalence, rather than solid predictive performance. For the same reason, the results of DCA in the external cohort (Fig. 4D) have limited reference value and cannot fully reflect the real clinical net benefit of the model. For the above reasons, the eICU cohort is only used for exploratory cross-setting observation in this study, rather than formal confirmatory validation.

Fig. 4.

Calibration and decision-curve analysis (DCA). (A) Calibration curves in the internal validation set; (B) DCA in the internal validation set; (C) Calibration curves in the eICU cohort; (D) DCA in the eICU cohort

Model interpretation

To enhance interpretability and identify the main contributors to delirium prediction, SHAP analysis was applied to the final XGBoost model. The SHAP results revealed that the most influential predictors were the GCS, OASIS score, and the use of Sa (Fig. 5A-B). Patients who received Sa and those with lower GCS or higher OASIS values tended to have a higher predicted probability of delirium (Fig. 5C). Variables with moderate contributions included the APACHE II score, anion gap, hematocrit, and serum calcium, reflecting metabolic and physiological instability. In contrast, the SOFA score and T2DM exhibited relatively low SHAP importance values (Fig. 5A-B), indicating limited incremental predictive effects once other severity indices were considered. Consistent SHAP value distributions in both the internal testing dataset and exploratory eICU test cohort demonstrated the model’s interpretability across datasets (Fig. 5). The SHAP force plot provided individualized insights into prediction mechanisms by visualizing how each feature influenced the model’s output (Fig. 5C and G). In this plot, red bars represent variables that increase the predicted probability of delirium, while blue bars indicate those that decrease it. For example, in a representative patient from the internal validation cohort, all major features—OASIS = 20, GCS = 15, anion gap = 11, Sa = 0, and APACHE II = 11—collectively lowering the model’s predicted delirium risk (Fig. 5C). In contrast, for a patient from the exploratory eICU test cohort, Sa = 1 and hematocrit = 35.9 were identified as the main risk-enhancing factors, whereas GCS = 15, OASIS = 26, and anion gap = 12 acted as protective contributors, reducing the overall predicted probability of delirium (Fig. 5G).

Fig. 5.

Model interpretability with SHAP. (A) SHAP Importance Plot in the internal validation set.; (B) SHAP Bees Plot in the internal validation set; (C) SHAP Heat Force Plot in the internal validation set; (D) SHAP Importance Plot in the eICU cohort; (E) SHAP Bees Plot in the eICU cohort; (F) SHAP Heat Force Plot in the eICU cohort

Discussion

Principal findings

In this multicenter retrospective study, we developed and internally validated a machine learning model for early prediction of delirium in non–mechanically ventilated patients with sepsis using routinely collected EHR data, with exploratory cross-cohort assessment conducted on the independent eICU cohort. The XGBoost algorithm showed robust discrimination and favorable calibration in the large MIMIC-IV internal validation cohort. While we tested the model on an independent eICU cohort for exploratory cross-setting observation, the sparse-event nature of the eICU dataset limits the reliability of its calibration and clinical net benefit metrics. Importantly, the model relied on a limited set of routinely available bedside variables, supporting its potential feasibility for integration into clinical workflows.

Our focus on sepsis-associated delirium aligns with current concepts of acute brain dysfunction as a major driver of long-term morbidity among ICU survivors [1, 2]. Contemporary guidelines emphasize routine delirium screening, severity tracking, and bundled prevention strategies such as the ABCDEF framework [3–5]. However, these approaches depend on timely identification of patients at greatest risk. Our model complements existing bedside screening by enabling early, data-driven risk stratification before delirium becomes clinically apparent, thereby supporting more targeted and efficient deployment of preventive resources.

Comparison with prior work

Previous machine-learning models for ICU delirium or sepsis-associated delirium have generally been developed in mixed ICU populations that combine mechanically ventilated and non-mechanically ventilated patients [6, 8]. Because mechanical ventilation and deep sedation are themselves strong and often dominant drivers of delirium, this design limits clinical interpretability and obscures intrinsic vulnerability to brain dysfunction. In contrast, we specifically examined non-mechanically ventilated patients with sepsis—a subgroup that is frequently overlooked but represents a key target for delirium prevention. In this population, patients are typically awake or lightly sedated, allowing implementation of mobilization, sleep promotion, and sedation adjustment. Internal testing within MIMIC-IV yielded stable predictive results, while exploratory cross-cohort observation in eICU offered preliminary reference for cross-institutional applicability, despite differences in case mix and documentation practices between institutions [9, 10, 14].

The most influential predictors identified by SHAP analysis—level of consciousness (GCS), overall illness severity (OASIS), and exposure to sedative or analgesic agents—provide strong face validity and clinical insight. These variables capture both acute brain dysfunction and global physiologic stress, which are central to the pathophysiology of sepsis-associated delirium [1]. The prominence of sedative and analgesic exposure further underscores the importance of sedation stewardship even in non-ventilated patients, while metabolic indices such as anion gap, hematocrit, and calcium suggest that systemic hypoperfusion and metabolic stress may represent modifiable contributors to delirium risk [3, 4, 15, 16].

Clinical application

From a practical perspective, the model enables automated early risk estimation using routinely available ICU EHR variables. Rather than replacing bedside screening tools, it complements existing workflows by identifying patients who may benefit from intensified monitoring or preventive strategies. Currently, most ICUs adopt universal delirium prevention protocols for all critically ill patients. While these simple interventions are easy to implement, one-size-fits-all care is not optimal. Mechanically ventilated patients usually require continuous deep sedation, which is a well-established risk factor for delirium. In comparison, non-mechanically ventilated patients are typically lightly sedated or even awake, with distinct risk profiles and clinical management needs. For this relatively less severely ill subgroup, universal intensive assessments and interventions bring extra burden to medical staff and may even disturb patients’ sleep and rest. With this prediction model, we can distinguish patients at high delirium risk from low-risk counterparts. Targeted intensified interventions are delivered to high-risk groups, whereas standard care is sufficient for low-risk patients. This refined strategy optimizes resource allocation and improves the overall efficiency of delirium prevention.

Integration into electronic decision support systems requires local calibration and prospective evaluation. It should be noted that the eICU cohort had an extremely low delirium rate with only a small number of positive cases. This sparse-event feature undermines the reliability of metrics derived from this exploratory test cohort, including calibration and DCA, which should be considered when interpreting and applying our findings.

Limitations

Several limitations should be acknowledged. First, the two databases exhibited prominent heterogeneity in patient case mix, delirium screening protocols, sedation regimens and routine ICU practices, which resulted in a remarkable difference in delirium incidence. The exploratory eICU test cohort had an extremely low delirium rate (< 1%) with merely 10 positive cases. Under such a sparse-event setting, the calibration curve and DCA are prone to bias; the visually favorable calibration result is likely induced by uniformly low risk prediction for all patients under low prevalence, and these exploratory assessment metrics cannot reliably verify the model’s generalizability. Furthermore, we failed to find additional publicly available databases with complete delirium labels and sufficient positive events for secondary external validation. Prospective multicenter validation using real clinical cohorts with adequate delirium cases will be performed in our future work. Second, delirium ascertainment depended on routine clinical documentation, and differences in screening intensity between databases likely introduced label noise. Exposure to sedative and analgesic agents was captured without detailed information on dose or depth of sedation. Third, although hybrid resampling was used to address class imbalance, synthetic sampling may introduce artifacts, and alternative approaches such as cost-sensitive learning warrant further study. Finally, the use of U.S.-based databases may limit generalizability to other health-care systems, emphasizing the need for prospective validation in diverse settings with adequate delirium events in future research.

Conclusions

We developed a machine learning model for early delirium prediction in non–mechanically ventilated patients with sepsis using multicenter EHR data. The XGBoost model exhibited robust discrimination and calibration in the large MIMIC-IV derivation and internal validation cohort. The eICU cohort was only utilized for exploratory cross-hospital observation due to its extremely low delirium incidence and limited positive events. This model relies on routine clinical indicators and offers a reproducible framework for clinical decision support. Large prospective multicenter studies with sufficient delirium cases are required to further verify its real-world predictive value and clinical benefit.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

B.X. conceived, designed, and drafted the manuscript; B.X. also retrieved and organized the data. Y.G. contributed to the translation and compilation of the literature. B.X., Y.G., and Y.R. jointly performed the statistical analyses. X.Z. revised and critically checked the whole manuscript and acquired the funding support.J.L. and Y.Z. supervised and provided overall guidance for the research plan and process.

Funding

This research was funded by Natural Science Foundation of Hunan Province (grant number No.2024JJ6571).

Data availability

The data supporting the findings of this study are publicly available from PhysioNet (https://physionet.org/), including the MIMIC-IV and eICU Collaborative Research Database.

Declarations

Competing interests

The authors declare no competing interests.

Consent for publication

Not applicable.

Ethical approval and consent to participate

The data used in this study were obtained from the Medical Information Mart for Intensive Care IV (MIMIC-IV) and the eICU Collaborative Research Database (eICU-CRD), both of which are publicly available, de-identified databases. The establishment of the MIMIC-IV database was approved by the Institutional Review Boards of the Massachusetts Institute of Technology and Beth Israel Deaconess Medical Center, and the eICU database was approved by the Institutional Review Board of Philips Healthcare. Informed consent was waived because all data were de-identified. Access to the databases was granted after completion of the required training and credentialing through PhysioNet.