Large language model-augmented offline reinforcement learning framework for sepsis management in critical care

Abstract

Sepsis is a life-threatening organ dysfunction caused by a dysregulated host response to infection, making optimal management critical. Existing Reinforcement Learning (RL) approaches for sepsis management have mainly relied on structured data (e.g., vital signs, laboratory results), lacking contextual information needed for comprehensive patient understanding. In this work, we propose a Multimodal Offline REinforcement learning for Clinical notes Leveraged Enhanced stAte Representation (MORE-CLEAR) framework for sepsis management. MORE-CLEAR employs large language models (LLMs) to facilitate the extraction of rich semantic representations from clinical notes, preserving clinical context and improving patient state representation. Gated fusion and cross-modal attention allow dynamic weight adjustment and the effective integration of multimodal data. Cross-validation using two public (MIMIC-III, MIMIC-IV) and one tertiary ICU dataset (SNUH) showed that MORE-CLEAR significantly improved the estimated survival rate and policy performance compared to single-modal RL. This approach could expedite sepsis management by enabling RL models to propose effective actions.

Introduction

Sepsis, characterized by a dysregulated host response to infection, leads to life-threatening organ dysfunction and remains a major cause of death among critically ill patients^1,2. Early identification and prompt intervention in the intensive care unit (ICU) are pivotal for improving outcomes in sepsis, emphasizing the need for timely and effective therapeutic strategies. However, timely recognition of sepsis remains challenging because of its diverse etiologies and rapid clinical progression^3,4. Although vasopressors and fluid therapy are essential for resuscitating patients with septic shock, the optimal timing and dosage of these interventions remain controversial^5–9.

In recent years, reinforcement learning (RL) has emerged as a promising approach for identifying optimal treatment strategies^10–12. Hemodynamic management in sepsis is inherently a sequential decision-making problem, in which treatment decisions are repeatedly adjusted as patient states evolve. Although supervised learning (SL) and causal effect estimation have been applied to sepsis-related tasks^13,14, SL formulations typically treat each decision independently and thus struggle to capture temporal dependencies inherent to sequential decision-making. In contrast, by formulating patient care as a sequential decision-making problem consisting of states, actions, and rewards, RL facilitates the development of data-driven, personalized treatment policies. RL research in sepsis management relies predominantly on structured data, such as laboratory results and vital signs^15–18. While these data are valuable from a clinical perspective, they are often characterized by substantial missingness, noise, and irregular sampling, which collectively undermine the reliability of state representations and hinder effective policy learning¹⁹. Unlike RL studies, which rely heavily on structured data, multimodal learning has gained increasing attention in medical applications by integrating heterogeneous data sources to obtain richer patient representations^20–22. Multimodal fusion strategies have demonstrated clear benefits across a wide range of supervised predictive tasks, including sepsis prediction^23,24. Despite these encouraging results, existing multimodal approaches are largely confined to static classification or regression tasks, leaving their integration into RL frameworks insufficiently explored. This gap motivates approaches that leverage multimodal patient information to optimize RL-based treatment policies for clinical decision support.

In clinical practice, decisions regarding treatment planning and medication administration are often guided by clinical notes, which provide nuanced contextual information, including medical history, symptom evolution, and responses to interventions^25,26. Incorporating clinical notes can compensate for the limitations of structured data and improve patient state representation. However, in RL training, the irregular timing and uneven informational density of clinical notes make effective multimodal integration challenging, and addressing these issues is essential for robust policy learning. Therefore, the key consideration in implementing a multimodal artificial intelligence model that integrates clinical notes with structured data for deep RL-based sepsis management is to derive a robust state representation that enables effective policy optimization.

To address these challenges, we developed MORE-CLEAR (Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced state Representation), a large language model (LLM)-augmented multimodal offline RL framework for sepsis management in the ICU. We hypothesized that integrating clinical notes with structured ICU data could provide a more comprehensive representation of patient states, thereby improving policy learning and predictive performance. This study aimed to evaluate whether the proposed framework enhances treatment policy estimations and generalizability across multiple ICU databases. Extensive cross-evaluations on MIMIC-III, MIMIC-IV, and a tertiary institutional dataset from Seoul National University Hospital (SNUH) demonstrated that MORE-CLEAR consistently outperforms unimodal frameworks.

Results

Study design

For the development of the RL models, we utilized three independent databases: two publicly available datasets from the United States, MIMIC-III and MIMIC-IV, and one tertiary institutional dataset collected from SNUH in South Korea. MIMIC-III, MIMIC-IV, and SNUH included 11,114, 10,203 and 599 patients with sepsis, respectively. Each dataset was randomly split into 80% for model training and 20% for internal validation. We independently trained the MORE-CLEAR on each dataset and evaluated performance using internal validation. A detailed description of the MORE-CLEAR architecture is provided in the “Methods” section. Additionally, cross-dataset validation was performed on the remaining datasets that were not used for training. We used off-policy evaluation (OPE) and the survival rate estimation metric (BDESR) for policy evaluation. OPE includes offline policy evaluation with re-weighted aggregates (OPERA)²⁷, doubly robust (DR)²⁸, fitted Q-evaluation (FQE)²⁹, and weighted importance sampling (WIS)³⁰. Further details about these metrics can be found in the subsection “Evaluation metrics” in the “Methods” section. Figure 1 illustrates the overall study design.

Fig. 1. Study design and workflow.

Sepsis cohorts were independently curated from MIMIC-III (training n = 8891; test n = 2223), MIMIC-IV (training n = 8162; test n = 2041), and SNUH (training n = 479; test n = 120). RL agents were trained separately within each training cohort. Both internal validation on the corresponding test cohort and external cross-dataset validation were conducted using OPE and BDESR.

Performance comparison across modalities

The OPE and BDESR performances for each modality are reported in Tables 1 and 2, respectively. Table 1 shows that the MORE-CLEAR generally outperforms unimodal baselines across a broad range of evaluation metrics. The multimodal models, BERT + Conservative Q-Learning (CQL)³¹, Clinical Bert (CB) + CQL, and Llama 3.1-8B (Llama) + CQL, outperform both structured data-only and text-only baselines, particularly in terms of OPERA (0.450 to 3.382 in Llama + CQL), DR (0.546 to 3.007 in Llama + CQL), and FQE (0.320 to 1.382). Dataset-specific trends further reinforce the advantage of the MORE-CLEAR framework. In MIMIC-III, Llama + CQL and Bert + CQL show the highest scores on OPERA (3.382), DR (3.007), and FQE (1.382), respectively. Furthermore, Llama + CQL achieves the best outcomes for WIS (0.766), whereas CB + CQL dominates the outstanding performance on the FQE score of 7.522 in MIMIC-IV. In the SNUH dataset, the superiority of multimodal models remains evident. Across all settings, CB + CQL emerges as the most robust combination, with Bert + CQL also exhibiting strong and consistent results. Combining text representations with conventional structured data-based RL algorithms provides clear performance benefits. These results also provide substantial evidence supporting the efficacy of the MORE-CLEAR framework, which offers a meaningful synergy of multimodal data.

Table 1.

OPE metric performance across all modalities

		Structured data		Text			Multimodal (MORE-CLEAR)
Dataset	Metric	BCQ	CQL	Bert	CB	Llama	Bert + CQL	CB + CQL	Llama + CQL
MIMIC-III	↑OPERA	0.889 $\pm$0.02	2.925$\pm$0.03	0.512$\pm$0.09	0.607$\pm$0.09	0.450$\pm$0.09	3.335$\pm$0.14	3.329$\pm$0.10	3.382 $\pm$0.14
	↑DR	0.917$\pm$0.04	2.419$\pm$0.05	0.574$\pm$0.08	0.587$\pm$0.07	0.546 $\pm$0.10	2.968$\pm$0.10	2.975$\pm$0.06	3.007 $\pm$0.09
	↑FQE	0.472$\pm$0.12	0.622$\pm$0.07	0.320$\pm$0.28	0.485$\pm$0.32	0.181 $\pm$0.08	1.382$\pm$0.74	1.346$\pm$0.38	0.899 $\pm$0.22
	↑WIS	0.563$\pm$0.25	0.682$\pm$0.03	0.721$\pm$0.08	0.738$\pm$0.02	0.728 $\pm$0.08	0.693$\pm$0.02	0.702$\pm$0.04	0.683 $\pm$0.03
MIMIC-IV	↑OPERA	0.885$\pm$0.09	3.862$\pm$0.04	0.436$\pm$0.07	0.461$\pm$0.07	0.714 $\pm$0.36	3.861$\pm$0.16	3.877$\pm$0.16	3.810 $\pm$0.14
	↑DR	0.963$\pm$0.05	3.791$\pm$0.03	0.521$\pm$0.24	0.866$\pm$0.18	0.750 $\pm$0.29	3.648$\pm$0.10	3.677$\pm$0.09	3.596 $\pm$0.12
	↑FQE	0.466$\pm$0.10	1.243$\pm$0.13	0.406$\pm$0.22	0.210$\pm$0.05	0.535 $\pm$0.24	2.597$\pm$2.07	7.522$\pm$3.33	1.322 $\pm$0.35
	↑WIS	0.731$\pm$0.08	0.753$\pm$0.03	0.566$\pm$0.31	0.736 $\pm$0.12	0.739 $\pm$0.12	0.734$\pm$0.04	0.758$\pm$0.04	0.766 $\pm$0.04
SNUH	↑OPERA	0.782$\pm$0.05	2.390$\pm$0.22	0.602$\pm$0.10	0.599$\pm$0.12	0.717 $\pm$0.22	2.781$\pm$0.17	2.930$\pm$0.14	1.969 $\pm$0.28
	↑DR	0.952$\pm$0.07	2.487$\pm$0.20	0.752$\pm$0.09	0.601 $\pm$0.09	0.689 $\pm$0.06	2.790$\pm$0.23	2.739$\pm$0.09	1.893 $\pm$0.27
	↑FQE	0.591$\pm$0.13	1.290$\pm$0.28	0.692$\pm$0.15	0.510$\pm$0.33	0.585 $\pm$0.07	1.517$\pm$0.71	1.681$\pm$0.96	1.811 $\pm$0.42
	↑WIS	0.675$\pm$0.26	0.685$\pm$0.02	0.422$\pm$0.22	0.413$\pm$0.28	0.464 $\pm$0.40	0.742$\pm$0.01	0.729$\pm$0.04	0.627 $\pm$0.25

Policy performance on MIMIC-III, MIMIC-IV, and SNUH evaluated using four OPE metrics: OPERA, DR, FQE, and WIS, where higher values (↑) indicate better performance. Results are reported as mean $\pm$ standard deviation over five random seeds. Models include structured-data baselines, namely BCQ (Batch-Constrained Q-learning) and CQL (Conservative Q-learning); text-only baselines based on BERT, Clinical BERT (CB), and Llama3.1-8B (Llama); and multimodal models (MORE-CLEAR) that combine structured data and text using CQL (Bert + CQL, CB + CQL, and Llama + CQL). Best and second-best results for each metric and dataset are underlined.

Table 2.

Performance of the BDESR metric across all modalities and learning paradigms

			MIMIC-III		MIMIC-IV		SNUH
ML	Modality	Model	Low-BDESR	High-BDESR	Low-BDESR	High-BDESR	Low-BDESR	High-BDESR
Human	-	Clinician	0.783a		0.741a		0.755a
SL	Structured data	MLP	0.835 $\pm$0.01	0.626 $\pm$0.01	0.776 $\pm$0.01	0.575 $\pm$0.01	0.749 $\pm$0.09	0.569 $\pm$0.06
	Text	CB	0.831 $\pm$0.01	0.615 $\pm$0.01	0.720 $\pm$0.01	0.562 $\pm$0.00	0.818 $\pm$0.00	0.666 $\pm$0.00
		Llama	0.812 $\pm$0.01	0.613 $\pm$0.01	0.704 $\pm$0.01	0.539 $\pm$0.01	0.860 $\pm$0.04	0.591 $\pm$0.04
	Multimodal	CB + MLP	0.808 $\pm$0.01	0.638 $\pm$0.01	0.763 $\pm$0.02	0.575 $\pm$0.01	0.843 $\pm$0.05	0.510 $\pm$0.05
		Llama+MLP	0.825 $\pm$0.02	0.626 $\pm$0.02	0.759 $\pm$0.01	0.567 $\pm$0.01	0.865 $\pm$0.04	0.643 $\pm$0.08
RL	Structured data	BCQ	0.848 $\pm$0.04	0.744 $\pm$0.06	0.843 $\pm$0.05	0.733 $\pm$0.04	0.889 $\pm$0.08	0.722 $\pm$0.13
		CQL	0.847 $\pm$0.03	0.633 $\pm$0.02	0.851 $\pm$0.01	0.635 $\pm$0.02	0.893 $\pm$0.06	0.606 $\pm$0.05
	Text	Bert	0.834 $\pm$0.03	0.682 $\pm$0.06	0.827 $\pm$0.02	0.688$\pm$0.02	0.856 $\pm$0.06	0.636 $\pm$0.12
		CB	0.815 $\pm$0.03	0.660 $\pm$0.06	0.839 $\pm$0.02	0.700 $\pm$0.02	0.901 $\pm$0.00	0.578 $\pm$0.14
		Llama	0.836 $\pm$0.01	0.722 $\pm$0.08	0.790 $\pm$0.03	0.560 $\pm$0.02	0.930 $\pm$0.05	0.644 $\pm$0.09
	Multimodal (MORE-CLEAR)	Bert+CQL	0.832 $\pm$0.02	0.631 $\pm$0.01	0.840 $\pm$0.01	0.631 $\pm$0.03	0.810 $\pm$0.09	0.644 $\pm$0.09
		CB + CQL	0.861 $\pm$0.03	0.619 $\pm$0.02	0.856 $\pm$0.02	0.618 $\pm$0.03	0.921 $\pm$0.12	0.614 $\pm$0.06
		Llama+CQL	0.862 $\pm$0.02	0.614 $\pm$0.01	0.837 $\pm$0.02	0.569 $\pm$0.01	0.891 $\pm$0.04	0.711 $\pm$0.06

Estimated survival rates are reported for Low-BDESR (trajectories closely aligned with the learned policy) and High-BDESR (trajectories with large deviations) groups across MIMIC-III, MIMIC-IV, and SNUH datasets. Results include clinician-derived outcomes, SL baselines, and RL-based policies under structured, textual, and multimodal settings. The means and standard deviations of the results across five seeds are reported. The top two results are underlined for the Low-BDESR metric. The parameter $p$ is set to 20.

^aAs a baseline, clinician results are reported on the full BDESR cohort and are not stratified into Low/High BDESR.

Table 2 presents BDESR-stratified estimated survival rates, comparing the low-BDESR and high-BDESR groups within each cohort: those closely aligned with the learned policy’s recommended actions (Low-BDESR) and those that deviate significantly (High-BDESR). In addition to RL-based policies, SL baselines and clinician-derived survival rates are reported for comprehensive comparison. In the SNUH dataset, these clinician results reflect routine care delivered across specialized medical and surgical ICUs in a tertiary academic hospital by multidisciplinary intensivist-led teams; the supervising attending intensivists were board-certified in their primary specialties and in critical care medicine, represented multiple disciplines including internal medicine, surgery, anesthesiology, neurology/neurosurgery, and thoracic surgery, and had at least 5 years of clinical experience in critical care medicine during the study period. Across all datasets, survival rates for the Low-BDESR group were consistently higher than those for the High-BDESR group, indicating that the policy has been trained in a clinically beneficial direction. This trend was more pronounced for RL-based approaches than for SL baselines. While SL models achieved reasonable performance, RL based models generally yielded higher survival rates, particularly in multimodal settings. Within the Low-BDESR group, multimodal RL configurations demonstrated superior survival rates across all datasets, with CB + CQL consistently achieving the highest rates of 0.861, 0.856, and 0.921 in MIMIC-III, MIMIC-IV, and the SNUH dataset, respectively. This finding suggests that integrating clinical notes and structured data meaningfully contributes to the identification of effective treatment strategies. To further examine the statistical relationship between the actions recommended by MORE-CLEAR and patient survival, we conducted a Spearman rank correlation analysis (Supplementary Table 5). The results indicate that the action patterns of MORE-CLEAR are predominantly associated with trajectories linked to survival, supporting that the learned policy tends to enact treatment strategies aligned with patient survival.

Sensitivity analysis of contextual representation performance

We evaluate four strategies for integrating clinical notes: (1) raw note embedding (raw), (2) simple token imputation (impute), (3) concatenation within a fixed time window (stack), and (4) encoding the initial note as a static context vector (context). In raw note embedding, the note is utilized as input if it exists within the time frame (4 h); otherwise, no textual input is provided. Simple token imputation involves forward-filling the note if it does not exist in the current time frame. Concatenation within a fixed time window stacks notes of the previous 12 h, thereby incorporating recent notes (window size(W) = 3; see Supplementary Table 1 for more details). Figure 2a shows experimental results across OPE metrics. OPERA scores consistently increase from raw to context in MIMIC-III ($\approx 2.6\to \approx 3.3$) and MIMIC-IV ($\approx 3.3\to \approx 3.8$). On the SNUH dataset, context achieves the highest OPERA score ($\approx 2.7$). Context also yields the best performance on the DR metric, particularly in the SNUH dataset. FQE differences are negligible in MIMIC-III but favor context in MIMIC-IV, albeit with high variance. This variability is expected because FQE relies on a learned Q-function for the evaluation policy, making it more sensitive to limited state-action coverage and distribution shift between the behavior data and the evaluation policy. By contrast, OPERA and DR tend to mitigate such instability through estimator aggregation or doubly robust correction. In SNUH, the stack method attains the highest FQE score. WIS scores remain comparable across all approaches, with context performing slightly better. These results indicate that leveraging the initial clinical note as a context vector, combined with our context-aware gated fusion, improves policy performance, though some metrics remain sensitive to high variance.

Fig. 2. Sensitivity, generalization, and reliability analyses of the MORE-CLEAR framework.

a Sensitivity analysis of contextual representation strategies. OPE results for four clinical note integration strategies: raw note usage (raw), forward-filling imputation (impute), temporal stacking within a fixed window (stack), and injecting the initial note as a persistent contextual vector (context). Bars indicate the mean over five random seeds; error bars denote the standard error. b DR estimator performance across training iterations under cross-dataset validation. DR estimates are tracked over training steps for each target cohort, comparing different train→test transfer settings. Solid lines represent the mean and shaded bands indicate the standard error over five seeds. c Cross-dataset validation with bootstrap uncertainty. For each train→test setting, OPERA and DR are estimated on the test set using bootstrap resampling (n = 1000) to compute the bootstrap mean and 95% confidence intervals (horizontal error bars). d Effect of clinical note summarization. Radar plots comparing OPE metrics for policies trained using raw clinical notes versus LLM-summarized notes. e Value-estimation reliability. Kernel density estimates of Bellman residuals ($\delta$) for multimodal, structured-only, and text-only policies. CB + CQL was used for all experiments across MIMIC-III, MIMIC-IV, and SNUH.

External validation

Table 3 reports cross-dataset validation results of the MORE-CLEAR framework in conjunction with CB + CQL. When evaluated on MIMIC-IV, the policy trained on MIMIC-III yields OPERA 3.004, FQE 2.116, and WIS 0.689, while the policy trained using SNUH achieves a slightly higher OPERA, a substantially lower FQE, and a comparable WIS. In the reverse setting, the MIMIC-III-based policy demonstrates suboptimal performance on SNUH, while the MIMIC-IV-based policy exhibits marginally superior generalization. On MIMIC-III, the policy trained with SNUH yields OPERA 2.830, FQE 1.150, and WIS 0.601, while the policy trained with MIMIC-IV attains the highest OPERA score of 3.974 across all settings. Overall, the policy trained using MIMIC-IV demonstrates consistent performance in terms of high expected returns and low-variance estimates across datasets, thereby indicating superior generalization. Conversely, the MIMIC-III-based policy demonstrates a high degree of sensitivity to distributional shifts, resulting in performance degradation in the external dataset.

Table 3.

Cross-dataset validation results of the MORE-CLEAR framework

Train→Test	OPERA	DR	FQE	WIS
MIMIC-III → MIMIC-IV	3.004 $\pm$0.15	2.380 $\pm$0.14	2.116 $\pm$1.29	0.689 $\pm$0.03
SNUH → MIMIC-IV	3.107 $\pm$0.30	1.749 $\pm$0.21	1.188 $\pm$0.11	0.677 $\pm$0.01
MIMIC-III → SNUH	2.011 $\pm$0.16	2.966 $\pm$0.24	0.835 $\pm$0.36	0.731 $\pm$0.07
MIMIC-IV → SNUH	2.909 $\pm$0.15	3.859 $\pm$0.11	0.897 $\pm$0.40	0.744 $\pm$0.05
SNUH → MIMIC-III	2.830 $\pm$0.09	3.538 $\pm$0.10	1.150 $\pm$0.02	0.601$\pm$0.04
MIMIC-IV → MIMIC-III	3.974 $\pm$0.12	2.041 $\pm$0.28	1.488 $\pm$0.58	0.649 $\pm$0.02

Cross-dataset OPE performance of the CB + CQL model evaluated using OPERA, DR, FQE, and WIS. Values denote mean $\pm$ standard deviation over five seeds.

In order to assess whether policies improve consistently across different datasets, the DR estimates are further visualized across training iterations in Fig. 2b. Our observations indicate that policy learning exhibits a positive reinforcement trend across all datasets. It is noteworthy that the model trained on MIMIC-IV consistently attains the highest performance when evaluated on other datasets, followed by models trained on MIMIC-III and SNUH, respectively. This finding indicates that the data distribution of MIMIC-IV effectively captures the underlying characteristics of both MIMIC-III and SNUH. To quantify uncertainty in cross-dataset evaluation, we additionally report bootstrap confidence intervals. As shown in Fig. 2c, most external validation results exhibit relatively narrow confidence intervals, indicating consistent generalization performance. All external validation results demonstrated robust performance compared with the internal validation outcomes (Table 1). Through a comprehensive external evaluation across three databases, we confirmed that the model’s generalizability and distribution robustness depend on the characteristics of the training database.

Summarization performance

The MORE-CLEAR framework employs LLM-based summarization to mitigate the irregular distribution of unstructured information during sampling. Figure 2d presents the results of OPE metrics for policies trained on either raw clinical notes or structured summarization notes. The policies leveraging summarized notes exhibit consistently larger envelopes in the radar plots. The OPERA scores extend further toward the outer grid, indicating a higher expected return under the learned policy. The DR also improves, reflecting more accurate value estimation. Improvement in FQE and WIS further suggests that utilizing summarized notes leads to stable estimation of both bias and variance. These results indicate that structured summarization of clinical notes enables more consistent learning than raw clinical notes.

Overestimation analysis

Figure 2e presents the estimated probability density functions of the Bellman residuals for three policies: multimodal (blue), structured data-only (orange), and text-only (green). It directly compares overestimation bias and estimation stability across datasets. The multimodal policy exhibits a distribution tightly concentrated around zero with minimal variance, indicating low bias and high reliability in value estimation. By contrast, the density of structured data-only policies is shifted to the right and substantially wider, reflecting a pronounced positive bias (overestimation) and unstable predictions. The text-only policy shows a sharp peak at zero but with a narrow spread, suggesting overconfident estimates that may under-represent actual estimation error. These results demonstrate that integrating structured data with clinical notes via the MORE-CLEAR framework substantially reduces both bias and variance in Bellman residuals, yielding more stable and trustworthy value-function estimates than either unimodal strategy.

Severity-stratified action distributions

Lactate-stratified action distributions of the RL and clinician policies are compared in Fig. 3. Lactate was used because it is a clinically established marker of illness severity. In the low-lactate stratum, the RL policy shows a lower-intensity vasopressor pattern than the clinician policy, placing more probability mass on lower discrete action bins across the trajectory. In the high-lactate stratum, the RL policy remains slightly lower-intensity than the clinician policy, but the difference is less pronounced. Notably, in MIMIC-III and MIMIC-IV, the RL policy assigns higher frequencies to high-intensity vasopressor actions (levels 2–4) compared to the clinician policy. Across datasets, the RL policy demonstrates increased vasopressor administration in the high-lactate cohort relative to the low-lactate cohort, indicating severity-dependent modulation of action selection. In the SNUH cohort, differences between the RL and clinician policies are small across both lactate strata. This suggests that the learned RL policy closely reflects the treatment patterns present in the underlying dataset. For IV fluid administration, action distributions are similar between the RL and clinician policies across all datasets and lactate groups, with no pronounced differences observed.

Fig. 3. Severity-stratified action distributions of vasopressor and IV fluid of RL and clinician policy.

Heatmaps show the distribution of vasopressor and IV fluid actions over the treatment trajectory for the RL (MORE-CLEAR) policy and the clinician policy across MIMIC-III, MIMIC-IV, and SNUH cohorts. Patients were stratified into low- and high-lactate groups using a lactate threshold of 2 mmol/L. The x-axis denotes the step index within each trajectory, and the y-axis denotes the discretized action level. Color intensity represents the log-scaled frequency of action selection.

Discussion

In this study, we developed and validated MORE-CLEAR, a multimodal offline RL framework that integrates structured ICU data with unstructured clinical notes to improve state representation and treatment policy learning for sepsis management. Our key findings are as follows: First, multimodal policies consistently outperformed structured-only and text-only baselines on OPE and BDESR, while LLM-based summarization and multimodal fusion improved estimation stability and reduced overestimation bias. Second, policy performance depended on how text was incorporated: using the initial note as a static context vector with context-aware gated fusion yielded the most consistent gains, whereas stacking recent notes provided smaller or less reliable improvements. Overall, these findings indicate that robust multimodal offline RL for sepsis depends not only on adding clinical text, but on leveraging clinically meaningful context in a way that stabilizes learning and supports generalization across heterogeneous settings.

To the best of our knowledge, this study presents a significant step toward optimizing decision-making in ICU settings, addressing a gap largely unexplored in previous research. Prior work on RL for sepsis management has primarily focused on optimizing vasopressor and intravenous fluid dosages^16,17,32,33, or designing strategies for antibiotic administration¹⁸ using various off-policy and Offline RL algorithms. They sometimes struggle with reliability issues, such as overdosing, and have had to rely on manually designed rule-based constraints to mitigate safety concerns^18,33. In addition, some studies have explored directly employing LLMs as policies to solve sequential decision-making problems^34,35. Chase et al.³⁴ showed that LLM-generated medication recommendations were potentially life-threatening in up to 57.1% of cases, demonstrating the risks of using LLMs as independent policy models in clinical settings without reference to quantitative indicators. MORE-CLEAR advances prior work by enabling reliable and clinically grounded treatment policy learning without manual rule-based constraints or unsafe reliance on LLMs as standalone policies. Moreover, comparative testing of multiple LLMs (Supplementary Tables 2, 3) informs appropriate model selection for clinical-text summarization and interpretation, demonstrating the adaptability of MORE-CLEAR across computational environments. Across cross-dataset validation, robustness analyses, and ablations, MORE-CLEAR showed consistent gains and more stable value estimation. This pattern suggests that leveraging clinically meaningful context, rather than simply adding more text, improves reliability under sparse and heterogeneous documentation. Together, these findings offer practical design guidance for multimodal offline RL systems intended for ICU decision support.

Clinically, the benefit of multimodal fusion is plausible because structured ICU time-series provide objective measurements of physiologic status, whereas clinical notes encode higher-level contextual information (e.g., baseline comorbidities, diagnostic reasoning, treatment intent, and evolving clinical impressions) that is not consistently captured by vitals and laboratory data. This complementarity is expected to be most pronounced when structured measurements are sparse or noisy and when notes provide high-signal context. Consistent with our evaluations, multimodal models outperformed unimodal baselines, whereas text-only settings were more sensitive to documentation missingness and heterogeneity. These results align with real-world clinical practice, where clinicians evaluate a patient’s objective status using laboratory results and vital signs while also referring to clinical notes that provide contextual details such as medical history, treatment progression, and admission status. Notably, models based on structured data still outperform their text-only counterparts, which can be interpreted from two complementary perspectives. First, structured data may provide more stable and informative representations of patient states than clinical notes. Second, clinical notes often contain substantial missingness, which can limit their effectiveness as primary features for policy learning, as illustrated in Supplementary Fig. 1. This observation is further supported by Fig. 2a, which shows that raw status data with greater missingness yield lower performance than both the imputed and stacked approaches. Collectively, these findings highlight that while clinical notes enrich contextual understanding, structured data remain indispensable for ensuring stable and reliable state representations in RL–based sepsis management.

The BDESR analysis further underscores the clinical relevance of the proposed policy. Consistent with our overall results, multimodal models achieved higher estimated survival rates than their unimodal counterparts. Furthermore, patients whose treatment trajectories closely aligned with the RL-suggested policy showed higher survival rates than those with greater deviations, suggesting that the RL policy captures actionable and clinically meaningful treatment patterns. Notably, survival rates in the High-BDESR group declined across all modalities, likely because this group included a greater proportion of trajectories that ended in mortality. In addition, because High-BDESR is defined by a large policy-clinician discrepancy, it may overrepresent clinically complex trajectories and rare or atypical state-action regions with limited data support. In such low-coverage regimes, the learned policy is more likely to deviate from clinician actions due to extrapolation, and the observed survival can appear lower and more variable due to both case-mix severity and coverage effects. Nevertheless, these overall findings suggest that the RL policy tends to recommend actions associated with favorable outcomes, reflecting its ability to avoid detrimental treatment patterns and align with evidence-based clinical decision-making principles. This interpretation is consistent with the results of the Spearman correlation analysis. In addition, the observation that RL policies achieved higher survival rates than SL policies suggests that, beyond improved representation learning, explicitly modeling sequential decision-making through RL provides additional benefits in capturing sequential dynamics.

Across all evaluation metrics, the context-based strategy achieved the best overall performance, underscoring the effectiveness of encoding the initial clinical note as a contextual vector. The stacking strategies in our sensitivity analysis incorporate recent clinical notes into state estimation. Although recent notes can capture clinically important events, clinical documentation is often cumulative, and notes recorded close in time frequently overlap in content. By contrast, the initial clinical note provides richer baseline context (e.g., comorbidities, presenting symptoms, and the clinical rationale for ICU admission) that is not fully restated in subsequent notes. Preserving this baseline context supports more reliable state estimation and has a more meaningful impact on RL decision-making than repeatedly incorporating overlapping recent notes. Furthermore, although MIMIC-IV contains only two categories of clinical notes, radiology reports (including X-ray, computed tomography, magnetic resonance imaging, and ultrasound) and discharge summaries (Supplementary Fig. 1), it demonstrated superior performance, suggesting that radiology reports meaningfully contribute to contextual understanding. These implications are consistent with the findings in Supplementary Fig. 5, which indicate that physician notes play a relatively more important role in performance improvement than nursing notes. In clinical practice, such imaging-based reports support both diagnostic assessment and therapeutic planning; for example, echocardiography during initial resuscitation can guide fluid administration and hemodynamic optimization⁵. In MIMIC-III and SNUH, which include a broader range of clinical note types, model performance improved further, indicating that integrating diverse note categories alongside radiology reports can more comprehensively enhance patient state representation.

The model trained on MIMIC-IV demonstrated consistent performance across cross-dataset evaluations. This can be attributed to the characteristics of MIMIC-IV, which underwent a more refined data collection process than MIMIC-III and contains higher-quality, radiology-based clinical notes. In addition, MIMIC-IV, collected from 2008 to 2022, likely reflects a more standardized sepsis treatment trajectory following the revision of international sepsis guidelines⁵. These observations emphasize that the quality and category of both structured data and clinical notes should be carefully considered when developing decision-support models. Moreover, because offline RL is inherently sensitive to distributional shifts, rigorous external validation across heterogeneous cohorts remains essential before clinical implementation. In this context, the relatively smaller size of the SNUH dataset can be regarded as another source of inter-cohort variation, and we further assessed whether the available data were adequate for RL training (Supplementary Fig. 4). SNUH exhibited meaningful performance even when only 60% of the subsampled data were used, indicating that the dataset size is not insufficient for effective RL learning. The consistent cross-dataset performance of MORE-CLEAR, spanning different collection periods and institutions, supports its potential generalizability to real-world critical care environments. Finally, the severity-stratified treatment patterns observed in this study suggest that the learned policy may capture severity-aware treatment behaviors, modulating treatment intensity according to physiologic acuity. Although the model was trained using binary rewards assigned only upon final survival, our proposed framework showed that it suggests actions requiring actual clinical intervention rather than superficial correlations (e.g., rejecting high drug doses because they are highly associated with high mortality rates) by leveraging the rich information in clinical notes. The relatively small divergence between the RL and clinician policies in the SNUH cohort further suggests that the learned actions are influenced by local practice patterns, underscoring the importance of site-specific validation.

From a methodological perspective, MORE-CLEAR confines the LLM to structured summarization and representation, thereby mitigating irregular and heterogeneous documentation and alleviating unsafe reliance on LLMs as standalone decision policies. However, the benefit of text augmentation is bounded by documentation quality and cross-site domain shift: sparse notes, differing note-writing styles, and local practice variation can attenuate gains, reinforcing the need for site-specific validation. From a clinical implementation perspective, the learned policy should be deployed as clinician-in-the-loop decision support (e.g., discrete action-bin suggestions accompanied by contextual explanations and safety checks) rather than as prescriptive orders. Prospective evaluation that incorporates monitoring for distribution shift and unintended treatment intensification or avoidance (overtreatment or undertreatment) will be essential before real-world adoption. Given that the model’s training data includes both nursing and physician notes, evaluating the causal relationship between its inputs and outputs would ideally be conducted by board-certified intensivists or physicians with formal critical care training and sepsis management experience. Against this background, our SHAP analysis showed that the directionality of SHAP values for clinical variables generally aligns with established clinical interpretations, suggesting that the model can provide clinically meaningful insights into clinician decision-making (Supplementary Fig. 6).

This study has several inherent limitations. Although we evaluated the model across datasets from different countries, this study is retrospective and includes data from only a limited number of institutions. In addition, MIMIC-III and MIMIC-IV were collected from the same institution, which limits their interpretation as fully independent external cohorts. However, the two datasets span largely non-overlapping periods of approximately a decade and differ substantially in data collection practices, preprocessing pipelines, and overall data quality. These factors should be taken into account when interpreting the results of this study. Additionally, the survival-based outcome design may not fully reflect the nuanced trajectory of patient recovery, potentially overlooking intermediate therapeutic effects. In future work, we plan to incorporate time-varying severity measures such as SOFA scores and serum lactate into the reward design to better capture intermediate treatment responses. Finally, the multimodal architecture may introduce challenges in interpretability and real-time deployment efficiency; continued efforts toward transparent modeling and lightweight optimization will be essential for trustworthy ICU integration. In addition, modeling with finer-grained time steps is necessary to better reflect rapid clinical decision-making in emergency settings.

In conclusion, we developed and validated MORE-CLEAR, a multimodal offline RL framework that integrates structured data with clinical notes to support clinical decision-making. By effectively fusing heterogeneous modalities, the model achieved more expressive patient state representations and improved policy learning performance. In sepsis management, MORE-CLEAR demonstrated enhanced robustness and generalizability compared to conventional approaches, suggesting the potential of MORE-CLEAR as a clinical decision support tool.

Methods

Dataset

We leveraged three ICU databases for the experiments: MIMIC-III³⁶, MIMIC-IV³⁷, and SNUH (the tertiary institutional dataset). The MIMIC-III v1.4 database is a publicly available and de-identified critical care dataset comprising 61,532 ICU admissions from 2001 to 2012 at the Beth Israel Deaconess Medical Center in Boston, Massachusetts, USA. The MIMIC-IV v2.2 database expands both temporal coverage and clinical representation by including 73,181 ICU admissions recorded between 2008 and 2019, along with improved data standardization and the adoption of modern coding systems that better reflect contemporary critical care practice. The SNUH was retrieved from the prospective registry of ICU patients at Seoul National University Hospital in the Republic of Korea from April 2022 to March 2025 via the clinical data warehouse. The collection of data and subsequent analysis were approved by the institutional review board of Seoul National University Hospital (IRB-E-2506-002-1644), with a waiver for written informed consent due to the study’s retrospective design and data anonymity. The feature statistics are described in Supplementary Table 4 and Supplementary Fig. 2, and the data preprocessing procedures are provided in Supplementary Note 1.

Multimodal Markov decision process

RL provides a principled framework for addressing sequential decision-making problems, typically formalized as a Markov decision process (MDP)³⁸. In this study, we model the progression of patient states as an MDP with multiple modalities of observation. The multimodal MDP is specified by the tuple $\left(S,A,P,r,\mathrm{\gamma }\right)$, with $S={\prod }_{i=1}^d{\mathcal{O}}^{M_i}$, which is the joint observation space formed by $d$ modalities, where ${\mathcal{O}}^{M_i}$ denotes the observation space of the $i$-th modality. $A$ denotes the set of all possible actions. The transition probability function $P:S\times A\times S\to \left[\mathrm{0,1}\right]$ is defined such that, for any joint observation $o_t=\left(o_t^{M_1},o_t^{M_2},\dots ,o_t^{M_d}\right)$ and action $a_t$, the probability $P\left(o_{t+1},|,o_t,a_t\right)$ represents the likelihood of transitioning to the next joint observation $o_{t+1}=\left(o_{t+1}^{M_1},o_{t+1}^{M_2},\dots ,o_{t+1}^{M_d}\right)$. The function $\mathrm{r}:\mathrm{S}\times \mathrm{A}\to \mathbb{R}$ denotes a reward function providing the expected reward. $\mathrm{\gamma }\in \left[\mathrm{0,1}\right)$ is the discount factor. The goal of an RL is to find a policy $\mathrm{\pi }:S\times A\to \left[\mathrm{0,1}\right]$ that maximizes the expected cumulative discounted return ${\sum }_{t=0}^{\infty }{\mathrm{\gamma }}^t{r}_t$. We conduct multimodal RL in an offline setting³⁹, where the policy $\mathrm{\pi }$ is learned from a fixed dataset $\mathcal{D}$ collected under a behavior policy ${\mathrm{\pi }}_{\beta }$, without any additional online interaction.

Problem formulation for sepsis management

In this section, we formalize the problem setting of multimodal RL for optimizing sepsis treatment. The patient state $S={\mathcal{O}}^{M_l}\times {\mathcal{O}}^{M_n}$ comprises observations from two distinct modalities: structured data, i.e., laboratory results and vital signs, ${\mathcal{O}}^{M_l}$ and unstructured clinical notes ${\mathcal{O}}^{M_n}$. The action space $A$ is defined as a discrete set of treatment options, specifically representing dosage levels of vasopressors and intravenous fluids. Each action $a_t\in A$ denotes the treatment administered at time step $t$. The reward function $r\left(s_t,a_t\right)$ quantifies the clinical outcome of applying $a_t$ in state $s_t$, which follows the primary outcome of this study. The transition function $P$ specifies the evolution of the patient’s state over time, and an episode $\mathrm{\tau }$ represents a sequence of clinician–patient interactions. The treatment-optimization objective is to identify a policy $\mathrm{\pi }$ that maximizes the expected discounted cumulative clinical benefit: ${\text{argmax}}_{\mathrm{\pi }}{\mathbb{E}}_{\mathrm{\tau }\mathrm{\sim }\mathrm{\pi }}\left[{\sum }_{\left(\mathrm{s},\mathrm{a}\right)\mathrm{\in }\mathrm{\tau }}\gamma \mathrm{r}\left(\mathrm{s},\mathrm{a}\right)\right]$.

State space, action space and reward function

The patient state $s_t$ is defined at four-hour intervals and consists of two primary modalities: structured data ${\mathcal{O}}^{M_l}$, which comprises 42 clinical variables, and clinical notes ${\mathcal{O}}^{M_n}$, which encompass nursing records, physician documentation, discharge summaries, and other information. Each episode spans from 24 h prior to 48 h after the suspected onset of sepsis, capturing the critical period for therapeutic intervention. The action $a_t$ denotes a joint decision over intravenous (IV) fluid and vasopressor administration, each discretized into five levels, yielding 25 unique treatment combinations. The reward $r_t$ is defined such that all intermediate time steps are assigned a reward of 0, whereas the terminal step $\left(T\right)$ yields a reward of +1 if the patient survives for 90 days and -1 otherwise.

MORE-CLEAR framework

In RL-based sepsis management models, two key components can be distinguished: (1) state representation and (2) policy optimization, where the latter involves learning an optimal decision policy from state representation through RL algorithms. The MORE-CLEAR framework is specifically designed to address the state representation component, with a particular emphasis on capturing clinically salient information from multimodal data to enhance downstream policy optimization.

MORE-CLEAR (Fig. 4) integrates structured data (laboratory results and vital signs) and unstructured clinical notes to construct an enriched and comprehensive state representation for offline RL. Clinical notes are first summarized and encoded into dense embedding vectors using a pre-trained LLM. Motivated by clinical reasoning, in which early patient information guides the interpretation of subsequent trajectories, the initial-time note is modeled as a context vector injected at each decision step to mitigate information sparsity and preserve temporal coherence. Both the context and observation components of the note embeddings are independently encoded and fused through a gated fusion mechanism. In parallel, structured data are processed by a refined MLP-based encoder inspired by the MLP-Mixer⁴⁰ architecture to handle missing values and complex feature interactions. Finally, a bidirectional cross-modal attention module integrates salient information from the textual and structured modalities, forming a unified state representation that facilitates robust RL policy learning. The ablation study for each component is presented in Supplementary Table 6.

Fig. 4. MORE-CLEAR framework.

MORE-CLEAR integrates LLM-based clinical note summarization with multimodal offline RL for sepsis treatment optimization. Raw clinical notes are summarized by a frozen LLM, and summarized observation notes are embedded using a frozen LLM encoder. The observation note embedding is combined with the initial context note through a context-aware gated fusion module. This representation then interacts with structured EMR embeddings, encoded by an MLP, via bidirectional cross-modal attention. The resulting multimodal state is used for policy learning, where discrete actions correspond to five-level dose bins for IV fluids and vasopressors, optimized via offline RL with Bellman updates.

Our framework is motivated by CQL³¹, an offline RL algorithm that has demonstrated strong performance in the sepsis treatment task^32,41,42. CQL modifies the standard Bellman objective by adding a conservative regularization term to mitigate the tendency of conventional DQN to overestimate Q-values.

$${\mathcal{L}}_{CQL}\left(\theta \right)=\frac{1}{2}{\mathbb{E}}_{\left(s,a,r,s^{\prime }\right)\mathcal{~}\mathcal{D}}\left[{\left(Q_{\theta }\right(s,a)-(r+\gamma {\mathbb{E}}_{a^{\prime }~\pi }\left[Q_{\overline{\theta }}\right(s^{\prime },a^{\prime }\left)\right])}^2\right]+\alpha \left({\mathbb{E}}_{s\mathcal{~}\mathcal{D},a~\mu }\left[Q_{\theta }\left(s,a\right)\right]-{\mathbb{E}}_{\left(s,a\right)\mathcal{~}\mathcal{D}}\left[Q_{\theta }\left(s,a\right)\right]\right).$$ 1

$\mathrm{\mu }$ denotes a distribution over actions intended to represent OOD behaviors at state $s$. By suppressing the Q-values of actions sampled from $\mu$, CQL prevents the policy from overestimating the value of actions that are not well represented in the offline dataset. Conversely, by encouraging higher Q-values for state–action pairs observed in the dataset $\mathcal{D}$, the learned Q-function is guided to assign sufficiently high values to actions taken in the data. The hyperparameter $\mathrm{\alpha }>0$ controls the strength of the conservative regularization. Larger $\mathrm{\alpha }$ intensifies the OOD penalty, promoting more rigorous suppression of unobserved action values and enhancing robustness. In contrast, excessively small $\alpha$ diminishes the conservative effect, potentially reviving the overoptimistic bias commonly observed in standard DQN formulations. Consequently, $\alpha$ plays a crucial role in balancing OOD-action suppression and learning stability.

Our method constructs sequential trajectories by concatenating all notes recorded within fixed time intervals. This process often produces long sequences containing a mixture of diverse information, which complicates the identification of key content and occasionally results in inputs that exceed the token limits. To address this, we employ an open-source LLM to summarize raw clinical notes and generate meaningful note embeddings. A structured prompting guides LLM to summarize with organized content into clinically meaningful sections: reason for encounter or admission, patient history and context, key findings, assessment and diagnosis, treatments and interventions, and care plan or recommendations (see Supplementary Fig. 3 for more details). Gemma-3-27B-it⁴³ is employed for the note summarization, as it demonstrated the best performance in bilingual (English/Korean) medical QA and summarization tasks (see Supplementary Table 3). Subsequent to the summarization, the structured text is fed into the LLMs, which employ average pooling on the hidden state of the final layer to extract the latent representation.

We introduce a context-aware gated fusion mechanism to extract enriched patient state representations from clinical notes ${\mathcal{O}}^{M_n}$. Henceforth, the initial note ${\mathrm{o}}_0^{M_n}$ refers the context note and $o_t^{M_n}$ refers to the event observation note at time $t$. The context note often offers implicit contextual cues that critically inform subsequent clinical decision-making, while the observation note captures local, time-specific information, including the patient’s evolving condition and treatment progress. Therefore, context-aware gated fusion is designed to maintain contextual notes to retain information about the patient’s underlying medical conditions, history, presenting complaints, and the cause of hospitalization, while dynamically integrating them with event observation notes from each time frame to yield more expressive patient state representations. In our context-aware gated fusion mechanism, $o_0^{M_n}$ and $o_t^{M_n}$ are projected into a high-dimensional vector space by a pre-trained LLM encoder $\varphi$.

$$f_c=\varphi \left(o_0^{M_n}\right)\in {\mathbb{R}}^{d_n},f_e=\varphi \left(o_t^{M_n}\right)\in {\mathbb{R}}^{d_n}.$$ 2

$d_n$ denotes the dimensionality of the note embedding. The final note representation $n_t\in {\mathbb{R}}^d$ is computed as an adaptive, weighted combination of the context vector $f_c$ and the event observation vector $f_e$. A gating mechanism modulates this combination. ${\mathrm{\psi }}_t$ is parameterized by a learnable weight $W\in {\mathbb{R}}^{d\times 2d}$ and bias $b\in {\mathbb{R}}^d$.

$${\psi }_t=\mathrm{\sigma }\left(W\left[{\mathrm{f}}_{\mathrm{c}};{\mathrm{f}}_{\mathrm{e}}\right]+\mathrm{b}\right)\in {\left(0,1\right)}^d,n_t={\mathrm{\psi }}_t\odot f_c+\left(1-{\mathrm{\psi }}_t\right)\odot f_e.$$ 3

Here, $\mathrm{\sigma }\left(\cdot \right)$ is the sigmoid function, $\left[\cdot ;\cdot \right]$ denotes concatenation, and $\odot$ is the element-wise product. The context-aware representation $n_t$ selectively integrates episode-level context with time-specific information, enabling enriched state representations for RL optimization.

To obtain embeddings for structured data, we employ an MLP-based encoder inspired by the MLP-Mixer⁴⁰ architecture, adapting its token and channel mixing paradigm from image patches to structured feature tokens. Let $o_t^{M_l}\in {\mathbb{R}}^s$ denote the structured observation (laboratory results and vital signs) at time $t$, where $s$ is the number of structured features. In the sepsis task, $s=42$. The MLP encoder produces a structured embedding ${\mathcal{l}}_t=\theta \left(o_t^{M_l}\right)$. We first tokenize each scalar feature via a learnable feature-wise affine transformation:

$$T_{t,j}=b_j+o_{t,j}^{M_l}{W}_j,W_j\in {\mathbb{R}}^d,b_j\in {\mathbb{R}}^d,$$ 4

and stack them to obtain $T_t\in {\mathbb{R}}^{s\times d}$. We then refine $T_t$ using $L$ mixer blocks with pre-norm residual connections, where layer normalization is applied before each MLP layer. Each block alternates between (i) token-mixing MLPs that mix information across the feature axis $s$, parameterized by a tunable hidden dimension $D_s$ via a two-layer mapping ${\mathbb{R}}^s\to {\mathbb{R}}^{D_s}\to {\mathbb{R}}^s$, and (ii) channel-mixing MLPs that apply feature-wise transformations over the embedding axis $d$, parameterized by a tunable hidden dimension $D_c$ via a two-layer mapping ${\mathbb{R}}^d\to {\mathbb{R}}^{D_c}\to {\mathbb{R}}^d$. After the $L$ blocks, we obtain ${\tilde{T}}_t\in {\mathbb{R}}^{s\times d}$ and compute the structured embedding by mean pooling over feature tokens:

$${\mathcal{l}}_t=\frac{1}{s}\sum _{j=1}^s{\tilde{T}}_{t,j}\in {\mathbb{R}}^d.$$ 5

The outcome ${\mathcal{l}}_t$ serves as the structured representation fed into the subsequent bidirectional cross-modal attention module.

Clinical notes offer therapeutic context and insights, while also encompassing diverse content such as auxiliary procedures, clinician–patient communications, and detailed patient behaviors. In contrast, structured data, i.e., laboratory results and vital signs, provide quantitative measurements of a patient’s physiological state at specific time points, though often limited by missing values and irregular sampling. In clinical practice, accurate patient assessment depends on the integration of these two complementary modalities^44,45. Motivated by this insight, we introduce a bidirectional cross-modal attention module that captures the interaction between the two modalities to selectively extract clinically meaningful information. Let $n,\mathcal{l}\in {\mathbb{R}}^d$ denote the clinical note embedding from gated fusion and structured data embedding vectors. We employ a bidirectional cross-modal attention mechanism to capture both $\left(l,v\right)\to \mathit{note}$ and $\mathit{note}\to \left(l,v\right)$ interactions. Here, $\mathit{landv}$ denotes laboratory results and vital signs, respectively, and $\left(l,v\right)$ represent structured data, while $\mathit{note}$ refers to clinical notes. Each modality is first assembled into a two-token memory, and then projected into the key and value spaces, while two learnable router queries are projected into the query space. Specifically, we stack the $\mathcal{l}\mathit{andn}$ as $\mathcal{M}=\mathcal{\left[}\mathcal{l}n\right]\in {\mathrm{ℝ}}^{2\times d}$. Key and value matrices from $\mathcal{M}$ as defined below:

$$\mathcal{K}=\mathcal{M}{W}^{\mathcal{K}},\mathcal{V}=\mathcal{M}{W}^{\mathcal{V}},$$ 6

Two learnable router query tokens $q_{\mathcal{l}}$ and $q_n$ is defined to capture the two directions $\left(l,v\right)\to \mathit{note}$ and $\mathit{note}\to \left(l,v\right)$, respectively:

$${\mathcal{Q}}^{\left(l,v\right)\to \mathit{note}}=q_{\mathcal{l}}{W}^{\mathcal{Q}},{\mathcal{Q}}^{\mathit{note}\to \left(l,v\right)}=q_n{W}^{\mathcal{Q}}.$$ 7

Here, $W^{\cdot }$ are learnable projection matrices and $d_k$ denotes the attention dimensionality. For each direction $u\in \{\left(l,v\right)\to \mathit{note},\mathit{note}\to \left(l,v\right)\}$, attention weights ${\mathrm{\alpha }}^{\left(u\right)}$ over two tokens and the corresponding attention-weighted feature ${\mathcal{A}}^{\left(u\right)}$ are computed as

$${\mathrm{\alpha }}^{\left(u\right)}=\text{softmax}\left(\frac{{\mathcal{Q}}^{\left(u\right)}{\left(\mathcal{K}\right)}^{\top }}{\sqrt{d_k}}\right)\in {\mathrm{ℝ}}^{1\times 2},{\mathcal{A}}^{\left(u\right)}={\mathrm{\alpha }}^{\left(u\right)}\mathcal{V}\in {\mathrm{ℝ}}^{1\times d}.$$ 8

Next, each original feature $(n,\mathcal{l}\mathcal{)}$ is concatenated with its corresponding attention vector and then passed through a linear transformation.

$$\tilde{\mathcal{l}}=W_{\mathcal{l}}\left[\mathcal{l};{\mathcal{A}}^{\mathit{note}\to \left(l,v\right)}\right]\in {\mathrm{ℝ}}^d,\tilde{n}=W_n\left[n;{\mathcal{A}}^{\left(l,v\right)\to \mathit{note}}\right]\in {\mathrm{ℝ}}^d,$$ 9

where $W_{\mathcal{l}},W_n\in R^{d\times \mathbb{2}d}$. Finally, the fused state representation is calculated as:

$$s=\left[\tilde{\mathcal{l}};\tilde{n}\right]\in {\mathbb{R}}^{2d}.$$ 10

Evaluation metrics

A rigorous evaluation of the MORE-CLEAR’s policy was conducted by employing four off-policy evaluation (OPE) metrics, including weighted importance sampling (WIS)³⁰, doubly robust (DR) estimator²⁸, fitted Q-evaluation (FQE)²⁹, and offline policy evaluation with re-weighted aggregates (OPERA)²⁷. From a methodological perspective, WIS re-weights observed returns using importance ratios and can exhibit high variance under long horizons or distribution mismatch, often yielding conservative estimates after weight normalization. FQE instead learns a Q-function for the evaluation policy and may become sensitive to limited state–action coverage or extrapolation beyond the behavior data, which can increase estimator variance or introduce bias. The DR estimator combines importance sampling with a learned outcome model to reduce variance, although model misspecification can still lead to bias. OPERA adaptively aggregates multiple OPE estimators via re-weighting to balance the bias–variance trade-off. Each OPE metric has distinct bias and variance characteristics. Accordingly, in offline RL, model performance is assessed by considering consistent improvements across multiple complementary metrics, rather than relying on a single estimator^46,47. Importantly, the reported OPE values in this study are used only for relative model comparison within the same experimental setting, and should not be interpreted as guaranteeing absolute performance advantages or as exact estimates of true expected return. Beyond these quantitative measures, we further examined the policy’s clinical relevance by introducing the Behavioral Discrepancy Estimated Survival Rate (BDESR), a metric designed to capture alignment between RL policy actions and clinician actions.

We define the BDESR to estimate the clinical effectiveness of a decision-making model, including both RL and ML approaches. This metric quantifies the degree of discrepancy between actions administered by the model and clinician actions recorded in the batch dataset, with the corresponding survival rate estimated based on behavioral inconsistency. For each patient episode $i$, divergences between the model actions and clinician actions over time are computed separately for the two actions: IV fluid and vasopressor administration.

$$m_{\text{iv}}^{\left(i\right)}=\frac{1}{T_i}\sum _{t=1}^{T_i}|{\hat{A}}_{\text{iv},t}^{\left(i\right)}-A_{\text{iv},t}^{\left(i\right)}|,m_{\text{vaso}}^{\left(i\right)}=\frac{1}{T_i}\sum _{t=1}^{T_i}|{\hat{A}}_{\text{vaso},t}^{\left(i\right)}-A_{\text{vaso},t}^{\left(i\right)}|,$$ 11

where $\hat{A}$ denotes the actions recommended by the model and $A$ denotes the actions taken by the clinician. A weighted average of $m_{\text{iv}}^{\left(i\right)}$ and $m_{\text{vaso}}^{\left(i\right)}$ is used to compute overall discrepancy:

$$m^{\left(i\right)}=\alpha m_{\text{iv}}^{\left(i\right)}+\beta m_{\text{vaso}}^{\left(i\right)},\alpha +\beta =1,\alpha ,\beta \ge 0.$$ 12

To evaluate the clinical impact of adherence to the model, we compare survival rates between episodes with low and high policy discrepancies. We identify the most extreme $N\%$ of episodes by sorting the discrepancy scores $m^{\left(i\right)}$ and computing the $p=N/2$ and $\left(100-p\right)$ percentiles, denoted ${\xi }_p$ and ${\xi }_{100-p}$, where $p$ is a hyperparameter. The low and high-discrepancy cohort is defined as:

$$L=\left\{i,|,m^{\left(i\right)}\le {\xi }_p\right\},H=\left\{i,|,m^{\left(i\right)}\ge {\xi }_{100-p}\right\}.$$ 13

These cohorts represent the lower and upper extremes of the discrepancy distribution. Survival status for each episode is encoded as a binary indicator $S^{\left(i\right)}\in \left\{\mathrm{0,1}\right\}$, where 0 indicates mortality. Survival rate is defined as follows:

$$\text{High}-\text{BDESR}=\frac{1}{\mathit{\left|}H\mathit{\right|}}\sum _{i\in H}1\{S^{\left(i\right)}=1\},\text{Low}-\text{BDESR}=\frac{1}{\mathit{\left|}L\mathit{\right|}}\sum _{i\in L}1\{S^{\left(i\right)}=1\}.$$ 14

BDESR facilitates a more precise evaluation of the clinical effectiveness of a model by enabling a comparison of survival between patients with well-behaved (Low-BDESR) and significantly deviant (High-BDESR) behaviors.

Implementation details

All experiments were conducted with fixed hyperparameter settings for consistency across datasets and random seeds. We trained all models using a batch size of 256, a learning rate of 0.0001, and the Adam optimizer (weight decay = 0.01). The discount factor was set to $\gamma =0.98,$ and gradient clipping with a maximum norm of 1.0 was applied during training. For offline RL, the Batch-Constrained Q-learning (BCQ)⁴⁸ threshold was set to 0.3, and the CQL regularization coefficient was fixed at 2.0. Both BCQ and CQL employed a Dueling DQN architecture, with a target network updated every 10 steps using a hard update scheme. The replay buffer size was set to 200,000, and each model was trained for a total of 1500 update steps.

BCQ and CQL were used as structured data-only RL baselines. We employed a neural network consisting of three fully connected layers with 1024 hidden units each, using ReLU activations and batch normalization. Llama 3.1-8B⁴⁹, BERT⁵⁰, and ClinicalBERT⁵¹ were used for text data-only RL baselines. Llama 3.1-8B is a decoder-only Transformer large language model pretrained on large-scale general-domain corpora for next-token prediction, providing strong general-purpose text representations. BERT is a bidirectional Transformer pretrained for masked language modeling, and ClinicalBERT further adapts this architecture to the clinical domain by additionally pretraining on large-scale clinical notes, improving coverage of clinical terminology and documentation styles. The text-based model employed language models as a frozen embedding extractor, integrated with a Dueling Q-network. Textual embeddings were obtained via mean pooling over the last hidden layer of the LLM. The embedding dimension was 4096 for Llama-based models and 768 for BERT-based models. In the multimodal setting, for the MLP encoder, we set the embedding dimension to 1024 and stack $L=2$ mixer blocks. In each block, the token-mixing MLP used a token expansion ratio $r_s=2$, and the channel-mixing MLP used a channel expansion ratio $r_c=2$. We used the same hyperparameter settings for the language model as in the text-only baselines. Structured features and textual embeddings were fused via gated fusion and cross-modal attention with a hidden dimensionality of 1024. During training, all LLM parameters were frozen, and only the downstream RL networks were updated. All baseline SL methods used the same architecture as the RL settings and were trained using cross-entropy loss.

For clinical note summarization, the LLM inference was performed using fixed decoding parameters (temperature = 0.2, top-p = 0.95, maximum tokens = 2048, repetition penalty = 1.0). The experiments were conducted in a computing environment equipped with four NVIDIA A6000 GPUs, using CUDA version 13.0 and PyTorch version 2.1. The largest parameter component of the framework is the LLM used for clinical note summarization (Gemma-3-27B-it). Since the LLM is used in a frozen manner, training does not require backpropagation through the language model. This design choice was motivated by the consideration that jointly optimizing the internal parameters of the LLM during RL training can introduce non-stationarity into Q-value estimation, thereby substantially complicating the learning process (Supplementary Fig. 7). For deployment, MORE-CLEAR can be operated with two NVIDIA A6000 GPUs under BF16/FP16 inference settings. Furthermore, with 8-bit or 4-bit quantized inference, deployment is feasible even on a single A6000 GPU, making the framework practical for real-world clinical decision support environments via on-premises solutions.

Author contributions

B.J., H.L., and H.L.Y. contributed to the conception and design of the study. Y.L., S.A.P. and J.L. collected the data. Y.L. and J.L. curated the data and performed the analyses. All authors contributed to the interpretation of the analyzed results. Y.L., B.J., H.L. and H.L.Y. drafted the original manuscript. H.L. and H.L.Y. revised it for important intellectual content. All authors reviewed and approved the final version of the manuscript.

Data availability

The MIMIC datasets are publicly available at https://physionet.org/content/mimiciii/1.4/ and https://physionet.org/content/mimiciv/3.1/. Due to ethical restrictions imposed by the IRB, the SNUH dataset is available upon reasonable request.

Code availability

The source code of the MORE-CLEAR framework can be found at: https://github.com/Doldolee/MORE-CLEAR.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-026-02611-8.