Early prediction of postoperative delirium in major surgery patients using multimodal preoperative EHR and medical notes: a retrospective cohort study in the Indiana network for patient care

Abstract

Introduction

This study assesses the utility of integrating multimodal preoperative electronic health records (EHRs), including unstructured medical notes, for early postoperative delirium prediction. We aim to demonstrate the feasibility of this data integration to facilitate future expansion of the methodology towards risk stratification.

Methods

Transformer-based models were trained and evaluated using EHR data (2013–2021) from the Indiana Network for Patient Care for patients undergoing major surgeries such as laparotomies, bowel resections and cardiac procedures. Delirium was assessed retrospectively using (1) International Classification of Diseases, 10th revision codes and (2) clinician/nurse assessments from both structured and unstructured documentation.

Results

From an initial cohort of 35 128 control patients, postoperative delirium was identified in 3872 (assessment 1) and 15 351 (assessment 2) cases. Prevalent background comorbidities included pre-existing conditions such as hypertension, diabetes, hypothyroidism and chronic pulmonary disease. Following data balancing, our late fusion multimodal transformer model achieved an area under the receiver operating characteristic curve of 0.80. The model demonstrated increased sensitivity to patients with high disease burden, highlighting the potential of advanced architectures for using unstructured clinical data.

Conclusions

Using up to 4 years of preoperative EHR data offers the significant advantage of earlier risk identification, facilitating enhanced preoperative surgical patient care. By successfully unlocking the nuanced clinical signals within preoperative unstructured data, this study provides a robust foundation for incorporating medical notes into early postoperative delirium prediction.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Admission-specific structured data, particularly information from the immediate preoperative period and intraoperative electronic health record (EHR), along with a restricted set of machine learning techniques, represent a primary focus of research in delirium prediction modelling.

• Unstructured clinical notes remain underused partly due to a lack of robust fusion methodologies with other modalities.

WHAT THIS STUDY ADDS

• This study demonstrates that a multimodal fusion transformer-based model can successfully integrate preadmission unstructured medical notes with structured preoperative EHR data to predict delirium risk, shifting the prediction timeline earlier.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• This research provides the foundation for an early warning system that refines risk stratification, thereby enabling a proactive approach to patient care without introducing additional data collection or manual scoring burdens at the time of hospitalisation or surgery.

Introduction

Postoperative delirium, a neurocognitive complication following major surgery, affects approximately 50% of older adults and is characterised by acute attentional and cognitive deficits.¹ Beyond its acute presentation, delirium is associated with significant morbidity, including prolonged hospitalisation, increased mortality and long-term cognitive and psychiatric sequelae.² Notably, it is recognised as an independent risk factor for Alzheimer’s disease and related dementias.³ Though there are known risk factors for postoperative delirium, these risk factors are often unrecognised in the preoperative period, precluding reliable risk stratification.

Machine learning methodologies, including logistic regression (LR), eXtreme Gradient Boosting (XGBoost), least absolute shrinkage and selection operator regression, random forests (RF), support vector machines (SVM) and neural networks (NN), have been extensively applied for electronic health record (EHR)-based delirium prediction in diverse patient populations, reporting area under the curve (AUC) scores from 0.50 to 0.92.4,6 A notable characteristic of many existing delirium prediction models is their dependence on data ascertained through structured clinical investigations, subsequent to hospital admission (such as within the intensive care unit (ICU) environment), or from a singular temporal assessment. Therefore, a critical unmet need exists for prehospital delirium prediction models to enable proactive risk stratification and preventative interventions.

Integrating textual and structured data has demonstrated superior performance in clinical prediction models compared with unimodal approaches.⁷ Despite the inherent complexities of unstructured clinical narratives, including physician notes, progress reports and radiology reports, arising from their volume, heterogeneity and sensitive information, these sources offer temporal and contextual information which is often absent in structured data.

Transformer models are state-of-the-art machine learning models for feature extraction.⁸ While tree-based models traditionally dominate tabular data modelling, advancements like inter-sample attention and improved embeddings have significantly enhanced the efficacy of the transformer architecture in handling tabular data.9,11 Transformers can effectively handle multimodal clinical data, capturing intricate relationships within these data and offering an efficient contextual encoding mechanism for medical notes.¹² Additionally, transformer-based language models (LMs), such as BioBERT and Bioformer, are specifically tailored for biomedical text.^{13 14}

This investigation explores the application of contemporary transformer architectures to improve the prediction of postoperative delirium. Using a multimodal late fusion model, we combine transformer-based text embeddings from preadmission medical notes with tabular EHR data to characterise delirium risk in patients undergoing major surgery, thereby facilitating the timely development of targeted care plans.

Materials and methods

The present study introduces models that employ multimodal late fusion to achieve preoperative prediction of postoperative delirium risk. These models are developed and evaluated retrospectively using 30 000 patients selected from a cohort of nearly 55 000 major surgery patients drawn from a statewide health information exchange, the Indiana Network for Patient Care (INPC).¹⁵

Retrospective cohort selection

The study period is from 2013 to 2021 with respect to the surgery index date. Delirium cases with matching controls are included in the study according to specific inclusion criteria. Specifically, patients must be 50 years or older because cognitive problems, including delirium, tend to impact older adults. They must also have undergone a major surgery at one of the fifteen Indiana University Health facilities requiring an inpatient stay of at least 1 day, and have at least one encounter per year within the INPC for the 4 years prior to surgery (ie, preoperative observation period).

Patients with severe mental illness, significant sensory impairment (auditory or visual), alcohol intoxication or related delirium, pre-existing dementia or delirium at admission or immediately preoperatively were excluded from the study cohort. These patients would have already been identified at risk following established preventative protocols within the standard clinical workflow, irrespective of the outcome of the proposed delirium risk prediction model.

A patient could potentially undergo multiple procedures during a single admission. The date of the earliest procedure is designated as the index date. The outcome of interest is postoperative delirium, which is assessed in two ways:

• Assessment I: International Classification of Diseases, 10th revision (ICD-10) diagnosis codes.

• Assessment II: Clinical documentation from nurses and healthcare providers.

While ICD known underidentification of delirium (1.5%–18%) is likely influenced by institutional practices and clinician awareness,^{16 17} this variability was mitigated in our study through the use of expert-validated ICD delirium codes for case definition. Control subjects were defined as patients with documented comorbidities but without any delirium diagnoses or clinical evidence of delirium in their EHR.

EHR records were extracted and processed across four primary domains. First, medical notes from the year preceding the index date were analysed, with a focus on sections pertaining to indication, findings, impression, procedure, and diagnosis. To standardise note length distributions across patient cohorts and mitigate computational constraints, only the K most recent notes, capped at K=20, were retained. The aggregate length of medical notes within the observation window can serve as a proxy for both disease burden (extensive documentation) and potential EHR deficiencies (incomplete entries or outliers with excessively long records). The threshold of 20 notes was empirically determined to preserve comparable note length distributions and quartile ranges across cohorts (table 1), while concurrently serving as a criterion for identifying these data characteristics. A sensitivity analysis using an unmatched, prevalence-representative cohort was conducted to evaluate model performance under natural clinical distributions, addressing potential confounding from note-length matching.

Table 1. Quartiles of the combined document length in number of words of the patients’ medical notes for delirium cases (assessment II) and controls cohorts when the maximum number of notes is set to K=20.

	Controls	Delirium
Q1	442	490
Q2	890	1046
Q3	1594	1751

Second, demographics consist of age, sex, race and ethnicity of the patient. The numerical value of the patient’s age at the index date is considered. Sex was coded as a binary categorical variable (ie, male/female). Race and ethnicity were also coded as binary categorical variables according to the majority class (ie, White/Non-White, Not Hispanic/Others). Third, diagnosis ICD codes were extracted from the medical records of the patients over an observation period of 4 years prior to the index date and summed along the 31 Elixhauser disease categories.¹⁸ This extended 4-year period ensured the inclusion of a comprehensive range of pre-existing chronic disease conditions. Finally, illness severity was operationalised using prior delirium, Charlson Comorbidity Index (CCI) score,¹⁹ American Society of Anesthesiologists (ASA) physical status classification²⁰ and emergency procedure indicator. While excluding patients with active dementia or delirium at admission, prior delirium (4.8% controls vs 12.43% delirium cases, table 2) was included as a relevant episodic exposure variable within the 4-year preindex period, representing a potential vulnerability factor for postoperative delirium distinct from our exclusion criteria. ASA physical status was encoded as an ordinal ‘ASA’ variable, and emergency procedure indicator as a binary ‘emergency’ variable.

Table 2. Baseline characteristics and summary statistics for the entire patient cohort identified from INPC.

Variable		Controls		Delirium cases (Assessment II)		SMD	P value
Age (mean, SD)		66.7 (9.9)		69.8 (11.2)		0.296	<0.01
		Patients	Percentage	Patients	Percentage
		35,128	100%	15,351	100%
Sex	Female	19,265	54.84	7,804	50.83	0.072	<0.01
	Male	15,853	45.13	7,547	49.90
Ethnicity	Not Hispanic or Latino	34,571	98.41	15,085	98.27	0.009	<0.01
	Others	557	1.58	266	1.73
Race	White	31,612	89.99	13,573	88.41	0.050	<0.01
	Others	3,516	10	1,778	11.58
Diseases	Congestive heart failure	4,777	13.6	3,532	23.00	0.201	<0.01
	Cardiac arrhythmias	9,013	25.65	5,209	33.93	0.141	<0.01
	Valvular disease	4,202	11.96	2,815	18.33	0.149	<0.01
	Pulmonary circulation disorders	2,027	5.77	1,531	9.97	0.128	<0.01
	Peripheral vascular disorders	5,916	16.84	3,722	24.24	0.162	<0.01
	Hypertension, uncomplicated	24,195	68.87	11,257	73.33	0.075	<0.01
	Hypertension, complicated	6,273	17.85	4,177	27.20	0.181	<0.01
	Paralysis	585	1.67	502	3.27	0.087	<0.01
	Other neurological disorders	2,293	6.52	2,119	13.80	0.208	<0.01
	Chronic pulmonary disease	9,815	27.94	5,447	35.48	0.135	<0.01
	Diabetes, uncomplicated	10,545	30.01	5,581	36.35	0.100	<0.01
	Diabetes, complicated	5,314	15.12	3,308	21.54	0.123	<0.01
	Hypothyroidism	7,334	20.87	3,673	23.92	0.064	<0.01
	Renal failure	6,619	18.84	4,215	27.45	0.160	<0.01
	Liver disease	3,705	10.54	1,930	12.57	0.053	<0.01
	Peptic ulcer disease excluding bleeding	983	2.8	588	3.83	0.049	<0.01
	AIDS/HIV	129	0.36	76	0.50	0.021	0.23
	Lymphoma	710	2.02	400	2.60	0.029	<0.01
	Metastatic cancer	2,034	5.79	920	5.99	0.011	0.33
	Solid tumor without metastasis	9,310	26.5	3,878	25.26	0.020	<0.01
	RA/collagen vascular disease	2,930	8.34	1,310	8.53	0.012	0.03
	Coagulopathy	2,556	7.27	1,832	11.93	0.131	<0.01
	Obesity	9,043	25.74	3,758	24.48	0.050	0.04
	Weight loss	2,675	7.61	1,949	12.69	0.146	0.05
	Fluid and electrolyte disorders	9,704	27.62	5,900	38.42	0.189	<0.01
	Blood loss anaemia	733	2.08	483	3.14	0.058	<0.01
	Deficiency anaemia	6,073	17.28	3,678	23.95	0.140	<0.01
	Alcohol abuse	1,668	4.75	1,150	7.49	0.103	<0.01
	Drug abuse	1,399	3.98	847	5.51	0.058	<0.01
	Psychoses	1,068	3.04	766	4.98	0.094	<0.01
	Depression	8,924	25.4	4,545	29.60	0.067	<0.01
Severity	Previous delirium	1,691	4.8	1,909	12.43	0.201	<0.01
	Charlson Comorbidity Index (CCI)	23,108	65.78	11,347	73.91	0.143	<0.01
	ASA physical status classification	26,867	76.48	13,526	88.11	0.264	<0.01
	Emergency surgery	2,734	7.78	2,988	19.46	0.297	<0.01

P values are derived using an independent samples t-test for the continuous variable (age) and the χ2 test of independence to assess the relationship between exposure variables and the binary outcome labels: (a) Controls versus (b) Delirium cases as defined by assessment II. SMD is reported to quantify the magnitude of baseline imbalances between the cohorts, with an SMD >0.10 indicating a clinically meaningful difference. Features with statistically significant p values (<0.01) are denoted in bold.

ASA, American Society of Anesthesiologists; INPC, Indiana Network for Patient Care; RA, rheumatoid arthritis; SMD, standardised mean difference.

To balance study cohorts for delirium prediction, all delirium cases were matched with controls based on age group, sex, race, ethnicity and aggregate medical note length within the 1-year observation period. This note length matching aimed to mitigate model bias towards extensively documented patients. The consort diagram of eligible patients from the INPC is detailed in figure 1. Matching on note length could alter real-world class separation; to address this, a sensitivity analysis on a non-matched or prevalence-representative cohort is conducted. The findings are detailed in the Results section to demonstrate clinical utility within a real-world distribution.

Figure 1. Selection of the case/control patients from eligible patients in INPC. CAM, Confusion Assessment Method; ICD, International Classification of Diseases; INPC, Indiana Network for Patient Care.

Model development and validation

The risk prediction model employs a late fusion strategy across four distinct modalities (figure 2). The first modality uses the encoder of a transformer LM to translate the medical notes into a unified vector embedding. Our preliminary evaluation indicated that the Bioformer model exhibited superior performance compared with BioBERT, BERT and TinyBERT.^{13 14 21} The three remaining tabular modalities (demographics, diagnoses and severity indicators) are encoded using stacked multilayer NN architectures. Standard scaling was applied to all tabular variables prior to embedding. To ensure a balanced feature representation during late fusion, all modalities were projected into a uniform 300-dimensional space and subsequently passed through independent layer normalisation. This empirically selected embedding size and normalisation strategy strictly prevents high-dimensional or high-variance textual embeddings from disproportionately dominating the network’s weight updates during backpropagation. Finally, these normalised embeddings are aggregated through summation and processed by a 4-layer feed-forward NN classifier with a softmax output layer. Alternative configurations, such as concatenation and early fusion, were explored but did not outperform this optimised late fusion architecture.

Figure 2. Late fusion architecture of the delirium risk prediction models with an example input from each modality. [CLS] denotes the special classification token used in transformer architectures. ASA, American Society of Anesthesiologists; CCI, Charlson Comorbidity Index; NN, neural network.

Distinct delirium models (referred to as D_I and D_II models) were trained for each delirium assessment (assessment I and assessment II). Ablation studies were conducted, starting with all available modalities and removing them until only medical notes remained, that is, (1) all modalities, (2) demographics, diagnosis, severity and (3) medical notes. To ensure a comprehensive assessment of structured-data predictive limits prior to multimodal fusion, RF, XGBoost, SVM and LR were employed as baseline models for the tabular data.

Statistical analysis

Descriptive statistics were used to summarise baseline patient characteristics across demographics, disease conditions and illness severity. The χ² test of independence was used to examine the association between categorical variables (controls vs cases), where delirium is defined according to assessment II. For comorbidities and emergency variables, the test evaluated the presence or absence of the condition. For risk scores, a non-zero threshold was applied for the CCI and a threshold >2 was used for the ASA physical status classification. For the continuous variable (age), an independent samples t-test was employed. Additionally, the standardised mean difference (SMD) was calculated to quantify the magnitude of baseline imbalances between the cohorts, with an SMD >0.10 indicating a clinically meaningful difference. Prior to modelling, continuous variables were standardised using zero-mean scaling, and missing structured EHR data were handled via complete case analysis. Statistical analyses and predictive modelling were performed in Python, using Scikit-learn, SciPy and PyTorch.

Patient and public involvement

This research is a retrospective machine learning study using preadmission, deidentified EHR data (2013–2021) from the INPC. Therefore, patients and the public were not involved in the design, conduct, reporting or dissemination plans of this study.

Results

Summary statistics for the initial INPC patient cohort are presented in table 2. Age was significantly associated with delirium occurrence (p<0.01), with an SMD of 0.296. Several Elixhauser comorbidity groups demonstrated a statistically significant association with the binary outcome labels (p<0.01), including congestive heart failure, neurological disorders, renal failure and drug abuse.

Risk model performance

Prediction performance varied depending on the specific delirium assessment definition used. To streamline the reporting of our results, we established a standardised model nomenclature. Each model is denoted by its base algorithm: D for the transformer-based deep learning model, RF for Random Forest, SVM for SVM, XGB for XGBoost and LR for Logistic Regression. A subscript (I or II) indicates the specific delirium assessment used to derive the outcome labels. A superscript indicates the input modalities: m for medical notes only, d for structured tabular data only (demographics, diagnoses and severity) and a for all modalities combined (eg, an all-modality transformer model for assessment I is denoted as D_I^a). Table 3 details the distribution of patients across the 80/20 training and testing split for both assessments. Of the 80% allocated to the training set, 10% was reserved as an internal validation split. Furthermore, table 3 highlights that the patient cohort defined by assessment II contains approximately four times the volume of available records compared with assessment I.

Table 3. Number of patients in training and test datasets for delirium models developed using risk prediction according to delirium assessments I and II.

	Delirium assessment I			Delirium assessment II
	Controls	Delirium cases	Total	Controls	Delirium cases	Total
Train	3091	3104	6195	12 255	12 303	24 558
Test	781	768	1549	3093	3048	6141
Total	3872	3872	7744	15 348	15 351	30 699

Late fusion models integrating all modalities (ie, D_I^a and D_II^a) achieved the highest performance for both assessments as shown in table 4. Despite access to a smaller training dataset (table 3), model D_I^a (AUC-receiver operating characteristic (ROC) 0.80) outperformed D_II^a (AUC-ROC 0.73). The reduced performance of assessment II can be attributed to its broader definition, encompassing milder and subsyndromal delirium with inherent diagnostic ambiguity due to overlapping or fluctuating clinical presentations. This complexity in identifying subtle and transient delirium likely impeded the model’s ability to learn clear decision boundaries, resulting in diminished discriminatory performance. To comprehensively assess clinical utility beyond discrimination, the best-performing multimodal framework, D_I^a was further evaluated using additional threshold-based metrics with a positive predictive value (PPV) of 0.71, negative predictive value (NPV) of 0.74 and calibration statistics, that is, Brier score of 0.186, calibration slope of 0.9, calibration-in-the-large of 0.025 and calibration plot is shown in figure 3.

Table 4. AUC-ROC, sensitivity and specificity of the delirium risk models.

Delirium	Model	Modality	AUC-ROC (95% CI)	Sensitivity	Specificity
Assessment I	Demographics, diagnosis, severity	RFId	0.76 (0.73 to 0.78)	0.66	0.70
		DId	0.75 (0.72 to 0.77)	0.70	0.70
		SVMId	0.76 (0.71 to 0.78)	0.69	0.72
		XGBId	0.77 (0.72 to 0.79)	0.67	0.76
		LRId	0.76 (0.73 to 0.77)	0.63	0.77
	Medical notes	DIm	0.72 (0.69 to 0.74)	0.65	0.69
	All modalities	DIa	0.80 (0.77 to 0.83)	0.72	0.77
Assessment II	Demographics, diagnosis, severity	RFIId	0.71 (0.69 to 0.72)	0.65	0.66
		DIId	0.71 (0.69 to 0.72)	0.64	0.67
		SVMIId	0.70 (0.67 to 0.72)	0.60	0.69
		XGBIId	0.72 (0.69 to 0.73)	0.62	0.71
		LRIId	0.68 (0.67 to 0.70)	0.60	0.69
	Medical notes	DIIm	0.66 (0.64 to 0.67)	0.60	0.65
	All modalities	DIIa	0.73 (0.70 to 0.74)	0.64	0.71

The study and results are conducted according to the Transparent Reporting of a multivariable model for Individual Prognosis Or Diagnosis Statement.28 The bold entries represent the highest performance achieved across the evaluated models.

AUC-ROC, area under the receiver operating characteristic curve; LR, logistic regression; RF, random forest; SVM, support vector machine; XGB, eXtreme Gradient Boosting.

Figure 3. Calibration plot for the late fusion multimodal prediction model D_I^a evaluated on the Assessment I test set. The x-axis represents the mean predicted probability of delirium, and the y-axis represents the observed fraction of true delirium cases.

Shapley Additive Explanations analysis for model D_I^a

Shapley Additive Explanation (SHAP)²² analysis was conducted to enhance the model interpretability by providing feature rankings and attributions at the individual instance level. To represent the unstructured text modality for this analysis, the transformer layer’s output was passed through a linear layer, yielding distinct weights for the positive and negative classes, which were treated as two discrete features. Consequently, a total of 41 features were analysed: 2 representing the medical notes modality, 31 for diagnoses, 4 for demographics and 4 for illness severity.

Global feature importance, evaluated via a SHAP summary plot (figure 4), revealed that advanced age emerged as the most impactful positive driver of postoperative delirium risk. Furthermore, high clinical severity markers, including elevated ASA physical status, emergency surgical status and higher CCI scores, consistently drove risk predictions upward. Text-derived logits also ranked among the top predictive features alongside these structured variables.

Figure 4. Top 10 predictors identified using the Shapley Additive Explanations (SHAP) for the late fusion multimodal delirium model D_I^a. ASA, American Society of Anesthesiologists.

To further investigate the role of age, a SHAP dependence plot was generated to visualise its interaction with the CCI (online supplemental figure 1). The plot demonstrates a clear positive relationship: as age increases, its contribution to delirium risk increases. Furthermore, among older patients, those with lower comorbidity burdens (blue data points) exhibit slightly higher isolated SHAP values for age compared with those with higher comorbidity burdens (red data points).

Sensitivity analysis

To analyse the relationship between the model’s top predictors and its classification patterns, we examined the feature distributions across true positives, true negatives, false positives and false negatives (online supplemental figure 2). In general, both true positives and false positives exhibited higher overall disease burdens. To further investigate this potential comorbidity-related bias, we examined the CCI score quartiles for the test set predictions (online supplemental table 1). This analysis revealed a clear trend: false negatives were associated with lower CCI scores (Q1–Q3: 0, 1, 3), suggesting reduced model sensitivity among patients with lower comorbidity burdens. Conversely, false positives were associated with higher CCI scores (Q1–Q3: 2, 3, 5), suggesting reduced specificity in patients with higher comorbidities.

To evaluate the clinical relevance of these high-comorbidity false positives, an expert physician review was conducted. The review confirmed that these misclassified patients were, in fact, at an extremely high clinical risk for delirium. This cohort frequently presented with an ASA physical status ≥3 and preindex clinical notes documenting significant cardiovascular pathology (eg, non-ischaemic dilated cardiomyopathy, mitral valve disease), major surgical interventions (eg, spinal) and chronic systemic conditions (eg, chronic pain, degenerative disc disease, chronic obstructive pulmonary disease, chronic kidney disease progressing to end-stage renal disease). These findings suggest that the model is capable of identifying patients with a high clinical likelihood of delirium, even in cases where formal diagnostic coding criteria may not have been fully satisfied.

Finally, because note-length matching can artificially alter real-world class separation, a formal sensitivity analysis was conducted using the unmatched, prevalence-representative cohort (n=3602; 1864 controls and 1738 cases). For this specific analysis, delirium cases were identified using a composite outcome of either a positive Confusion Assessment Method²³ screen or the presence of delirium-specific ICD-10 codes. This composite definition was chosen to establish a rigorous, clinically validated baseline to test the real-world generalisability of models trained on standard EHR definitions. When evaluated on this real-world distribution, the optimised multimodal model (D_I^a) achieved an AUC-ROC of 0.71, a PPV of 0.66 and a NPV of 0.65. Calibration statistics included a Brier score of 0.225, a calibration slope of 0.612 and a calibration-in-the-large of 0.066. The calibration plot is shown in online supplemental figure 3. While this represents an expected decrease from the matched cohort performance (AUC-ROC 0.80), an AUC-ROC of 0.71 demonstrates that the multimodal framework retains meaningful predictive performance and clinical utility under real-world data conditions.

Discussion

This study aimed to establish a multimodal framework integrating unstructured medical notes and structured preoperative EHR data to identify postoperative delirium risk in major surgery patients, thereby facilitating timely development of care plans.

The SHAP analysis (figure 4) demonstrates that the model’s feature attributions align heavily with established clinical expectations.²⁴ Advanced age, elevated ASA physical status and high CCI scores, comorbidities like renal failure, other neurological disorders and chronic pulmonary disease were confirmed as the primary drivers of postoperative delirium risk. Notably, our analysis revealed that while age independently contributes to delirium risk across the population, its relative marginal contribution is somewhat attenuated among patients with greater baseline comorbidity (online supplemental figure 1), where other health factors also contribute substantially to the prediction. Crucially, text-derived logits ranked among the top predictive features alongside these established physiological markers. This confirms that unstructured narrative data successfully captures nuanced risk signals that complement the structured baseline, underscoring the critical value of medical notes as a distinct, information-rich modality for preoperative risk modelling.

Baseline models (RF, XGBoost, SVM and LR) demonstrated strong efficacy on structured tabular data, aligning with prior literature indicating that tree-based models perform comparably to deep learning on purely tabular datasets.⁹ However, our findings demonstrate that this strong baseline performance is significantly enhanced through multimodal integration, as deep learning architectures are uniquely capable of capturing complex nuances in both data types. While unstructured narratives present inherent modelling challenges when used in isolation as reflected in the comparatively lower performance of our text-only models (D_I^m and D_II^m), they remain an invaluable source of subtle clinical indicators. By employing a deep learning late-fusion architecture, the model seamlessly integrates these high-dimensional medical notes with baseline physiological parameters, maximising the predictive value of both data types.

Furthermore, performance discrepancies between assessment I and II models likely stem from the absence of a definitive delirium ground truth. Assessment II’s broader definition, which includes milder or borderline cases, introduces diagnostic ambiguity compared with the strict ICD-10 disease codification used in assessment I.

As anticipated, the model’s AUC-ROC decreased from 0.80 in the matched test cohort to 0.71 in the unmatched, prevalence-representative cohort. This shift indicates that aggregate note length potentially acts as a proxy for underlying disease burden. However, maintaining an AUC-ROC of 0.71 demonstrates that while cohort matching was necessary to mitigate training bias towards extensively documented patients, the optimised multimodal model retains robust discriminative ability and clinical utility when deployed across highly imbalanced, real-world data distributions.

A primary strength of our approach is its prioritisation of clinical practicality. The model uses 40 routinely accessible preadmission features, including diagnoses and medical notes. We excluded patients with pre-existing cognitive impairment (eg, dementia), acknowledging their distinct preoperative management due to an already elevated delirium risk.²⁵ Our analysis supported this exclusion, which showed that including patients with dementia (a common practice in other studies) only provided a marginal 2% performance increase but carried the risk of confounding. This exclusion emphasises a key strength: our aim to model delirium risk in a general surgical population, avoiding the potential performance inflation seen in studies enriched with known high-risk patients.

Furthermore, while our model’s performance is lower than some works using extensive, real-time inpatient EHR features,⁴ it successfully tackles the far more challenging task of predicting high-risk delirium patients using data collected over up to 4 years prior to hospitalisation. The complexity and limited availability of highly accurate, current exposure variables used in previous studies may hinder widespread clinical adoption. By designing our model for deployment at the time of surgical scheduling or the preoperative evaluation visit (typically 1 to 4 weeks prior to surgery), we shift risk stratification significantly earlier. Flagging high-risk patients at this juncture enables proactive interventions, such as medication optimisation and preemptive geriatric co-management. While standard admission-based screening tools are clinically valuable, they often require manual bedside data entry and offer a narrow temporal window that precludes extensive preoperative intervention. In contrast, our automated multimodal framework intends to mitigate the burden of manual scoring, complementing admission-based scores and initiating the preventative measures before the patient is admitted to the hospital.

In evaluating the clinical utility of predictive models, the operational trade-off between sensitivity and specificity is critical. For postoperative delirium, high sensitivity is prioritised to ensure vulnerable patients are not missed. Our optimised multimodal architecture directly supports this clinical priority. Compared with the highest-performing structured-only baseline (XGB_I^d), the integration of unstructured medical notes primarily drove improvements in model sensitivity (increasing from 0.67 to 0.72 in assessment I while preserving strong specificity (0.77). Maintaining this concurrently high specificity is equally vital for EHR-integrated clinical decision support tools to prevent false positives. While the default classification threshold of 0.5 demonstrates a strong, balanced predictive profile, future implementation research will incorporate Decision Curve Analysis. This will allow health systems to mathematically weigh the net clinical benefit of true positives against the resource costs of false positives, enabling dynamic threshold calibration tailored to specific institutional capacities.

While multimodal architectures inherently introduce higher computational complexity compared with traditional tabular classifiers, this integration offers distinct operational benefits. Structured EHR data frequently lacks the granularity to capture real-time, nuanced indicators of patient vulnerability, such as subtle baseline cognitive impairment, frailty or specific functional declines. Medical notes natively capture this critical context. By automating the extraction of these unstructured features, our multimodal model identifies high-risk surgical patients who might otherwise remain undetected by purely structured algorithms, delivering this enriched predictive value passively without introducing additional documentation burden on clinical workflows.

To further maximise the integration of unstructured data, future work should leverage ongoing NLP advancements, such as exploring advanced transformer models designed for longer clinical sequences (eg, ModernBERT²⁶) to enhance prediction accuracy, particularly for Assessment II models which are challenged by the inclusion of broader criteria of delirium with milder and subsyndromal cases. Additionally, while our exploration of attention-based Multimodal Adaptation Gating (MAG)²⁷ during early prototyping yielded negligible performance improvements compared with our optimised additive fusion, we hypothesise that incorporating longitudinal time-series data rather than our current static preoperative variables might be necessary to fully leverage the MAG architecture in future iterations.

Ultimately, a pragmatic clinical implementation strategy, building on our late fusion architecture, could involve initially deploying a robust tabular model using extended preoperative data. This baseline risk could be subsequently augmented with recent clinical notes over a shorter preoperative window, refining predictions while strategically mitigating the computational and processing demands associated with extensive unstructured data.

Conclusions

This study demonstrates the value of multimodal machine learning frameworks for predicting postoperative delirium risk using routinely collected, preadmission EHR data. By successfully integrating unstructured medical notes with structured parameters, our late-fusion architecture outperformed single-modality baselines, highlighting the synergistic value of unstructured narrative data in capturing nuanced patient vulnerabilities. Ultimately, this approach enables the early identification of high-risk surgical patients weeks before their procedure. This critical temporal window allows for the proactive development of individualised care plans, which can contribute to enhanced care quality and mitigate severe postsurgical complications.

Data availability statement

Data are available on reasonable request.