HRRT: hierarchical reinforcement learning for renal replacement therapy decision support

Abstract

Renal replacement therapy (RRT) is a critical intervention for patients with acute kidney injury (AKI). However, clinical decision-making regarding the timing of initiation, modality selection, optimal ultrafiltration rate, and weaning criteria remains highly complex and exhibits significant practice variation. Current RCTs fail to guide adaptive strategies, leaving dynamic decision-making heavily reliant on empirical experience. Consequently, a substantial gap persists in dynamically adjusting treatment in response to patient-specific clinical progression. We conducted a retrospective multi-center study to develop Hierarchical Reinforcement Learning for Renal Replacement Therapy (HRRT), a holistic clinical decision support system (CDSS) that covers the full decision-making process in RRT. 2467 Intensive Care Unit (ICU) stays of 1439 patients from a US hospital were used for training and internal testing. The model’s performance was evaluated on two external validation sets, where we selected 1085 ICU stays of 1085 patients from Netherlands and 1230 stays of 845 patients from China. The estimated mortality rate decreased by 6.1 percentage points from 47.7% (95% CI: 45.2–50.0) to 41.6% (95% CI: 35.6–47.2) compared to clinician-led outcomes. These findings imply that an AI-driven, holistic approach to RRT can reduce inappropriate practice variation and lower ICU mortality by recommending personalized treatment adjustments.

Introduction

Acute Kidney Injury (AKI) is common in critically ill patients and often triggers the need for Renal Replacement Therapy (RRT, also termed kidney replacement therapy). It is widely known that up to half of patients in the ICU develop AKI, and mortality rates can reach 80% in those with Stage 3 AKI ^1–3. While prior work has largely focused on predicting the onset of AKI, determining the optimal time to initiate RRT, and selecting the treatment modality, it has mostly produced mixed clinical benefits ^4–7. Early initiation of RRT has not been shown to consistently improve survival, but delayed initiation can lead to fluid overload and metabolic complications, affecting morbidity^8–11. Moreover, conventional biochemical and urine output monitoring does not reflect the complex and dynamic physiology of critically ill patients, limiting their ability to guide discontinuation of therapy⁷.

Beyond decisions on treatment initiation, weaning timing, and modality selection, the debate surrounding the optimal dosing intensity for RRT remains particularly contentious. Large randomized controlled trials such as the Veterans Affairs/National Institutes of Health (VA/NIH) Acute Renal Failure Trial Network¹²and The Randomized Evaluation of Normal vs Augmentin Level (RENAL) Replacement Therapy Study¹³ both showed that more intensive RRT regimens do not reduce mortality or accelerate renal recovery compared with less-intensive strategies¹⁴. More recently, Faria et al.¹⁵ demonstrated on the real-world ICU data that across a wide range of fluid removal rates (5–40 mL/kg/hr), there was no strong association with mortality. These findings collectively suggest that fixed ultrafiltration targets—whether high or low—do not significantly influence patient outcomes. Given the dynamic clinical course of critically ill patients, the ideal RRT dose is likely not a static value but a variable target that must be continuously adapted in response to evolving hemodynamic and metabolic conditions¹⁶. The challenge lies in the cognitive burden that complex decision-making places on clinicians, who must continuously synthesize high-dimensional physiological data to make precise adjustments without robust, evidence-based guidelines.

Incorporating artificial intelligence (AI) for RRT offers great potential for improving AKI management through individualized treatment strategies¹⁷. Among current AI technologies, Reinforcement Learning (RL) is especially suitable for the healthcare domain due to its ability for sequential decision-making^18,19. While previous studies have explored the use of RL for recommending RRT modalities, they have not adequately addressed the dynamic adjustment of appropriate ultrafiltration targets tailored to individual patient needs²⁰. Hierarchical Reinforcement Learning (HRL) is uniquely designed to decompose complex problems into multi-level decision processes and has been successfully applied to manage conditions such as multi-organ dysfunction²¹. In this study, we leverage HRL to develop a holistic decision-support framework that captures the entire therapeutic process of RRT. This process requires clinicians to sequentially determine when to initiate therapy, select the appropriate modality, establish suitable ultrafiltration goals and fluid removal rates, adjust these parameters based on the patient’s evolving condition, and ultimately decide when to discontinue treatment. We hypothesize that this comprehensive, multidimensional, and individualized approach will outperform standard clinician-driven protocols, moving beyond a “one-size-fits-all” paradigm to improve mortality outcomes in critically ill patients with AKI.

Results

Study population

The study included a total of 4782 patient stays that received RRT, derived from three datasets (Fig. 1, and Table 1). The derivation cohort, sourced from the Medical Information Mart for Intensive Care (MIMIC-IV) dataset²², comprised 2467 stays. Patients in this cohort had a median age of 66 years (IQR 56–75), and 59.8% were male; the ICU mortality rate was 46.5%. The external validation cohort consisted of two datasets: AmsterdamUMCdb²³ and Dr.ECC. The AmsterdamUMCdb dataset included 1085 patient stays, with a similar median age of 65 years (IQR 55–75), a higher percentage of male patients (67.0%) and an ICU mortality rate 44.4%. The Dr.ECC dataset contained 1230 stays, with an older median age of 69 years (IQR 56–78), less male patients (29.9%) and a lower ICU mortality rate of 10.5%.

Fig. 1. Flowchart of dataset construction.

We excluded non-RRT patients and those with extremely long ICU stay (>60 days).

Table 1.

Patient characteristics at ICU admission levels with statistical comparison between derivation and validation cohorts

Characteristic	Derivation	External (Amsterdam)		External (Dr.ECC)
	MIMIC-IV	Value	P-value	Value	P-value
Summary of datasets
Enrollment period	2008–2022	2003–2016	–	2017–2024	–
No. of included ICU stays	2467	1085	–	1230	–
Location	Boston	Amsterdam	–	Hangzhou	–
Patient characteristics
Age, median (IQR), y	66 (56–75)	65 (55 – 75)	0.06	69 (56 – 78)	< 0.001
Weight, mean (SD), kg	82.2 (15.8)	78.7 (18.4)	< 0.001	73.9 (1.6)	< 0.001
Gender, n (%)
Male	1476 (59.8)	727 (67.0)	0.67	368 (29.9)	< 0.001
Female	991 (40.2)	358 (33.0)	0.67	862 (70.1)	< 0.001
Race, n (%)
White	1392 (56.4)	NAb	–	NAb	–
Black	424 (17.2)	NAb	–	NAb	–
Asian	61 (2.5)	NAb	–	NAb	–
Othersa	590 (23.9%)	NAb	–	NAb	–
Dialysis type, n (%)
IHD only	750 (30.4)	40 (3.7)	< 0.001	0 (0)	< 0.001
CRRT only	1330 (53.9)	981 (90.4)	< 0.001	1230 (100)	< 0.001
Both IHD and CRRT	387 (15.7)	64 (5.9)	< 0.001	0 (0)	< 0.001
AKI stage at ICU admission, n (%)
Stage 1	53 (2.1)	NAb	–	NAb	–
Stage 2	30 (1.2)	NAb	–	NAb	–
Stage 3	2384 (96.6)	NAb	–	NAb	–
Systolic Blood Pressure, median (IQR), mmHg	110.0 (99.0–125.0)	127.0 (111.0–147.0)	0.33	128.0 (110.0–140.0)	< 0.001
Mean Blood Pressure, median (IQR), mmHg	71.8 (64.0–81.0)	81.3 (72.3–92.3)	0.09	82.0 (70.0–90.0)	< 0.001
Diastolic Blood Pressure, median (IQR), mmHg	55.0 (47.5–64.0)	58.0 (51.0–67.0)	< 0.001	62.0 (50.0–70.0)	< 0.001
SpO2, median (IQR), %	98.0 (96.0–100.0)	98.0 (96.0–99.0)	< 0.001	96.0 (94.0–100.0)	< 0.001
Urea, median (IQR), mmol/L	13.2 (7.9–21.1)	15.0 (10.2–22.8)	0.009	15.0 (8.0–25.0)	< 0.001
Creatinine, median (IQR), μmol/L	327.1 (176.8–521.6)	170.0 (112.0–348.0)	< 0.001	172.7 (108.0–275.3)	< 0.001
pH, median (IQR)	7.4 (7.3–7.4)	7.4 (7.3 – 7.5)	< 0.001	7.3 (7.2–7.4)	< 0.001
Heart Rate, median (IQR), bpm	90.2 (78.2–103.0)	90.1 (78.3–103.4)	< 0.001	90.2 (80.0–110.0)	< 0.001
Bicarbonate, median (IQR), mmol/L	25.7 (22.4–28.9)	25.7 (22.4–28.9)	< 0.001	23.5 (18.0–25.0)	< 0.001
Potassium, median (IQR), mmol/L	4.2 (3.9–4.5)	4.2 (3.9–4.5)	0.44	4.1 (3.8–4.6)	0.12
Sodium, median (IQR), mmol/L	137.0 (134.0–140.0)	138.0 (135.0–143.0)	< 0.001	138.0 (135.0–142.0)	0.08
PaO2/FiO2 ratio, median (IQR)	216.0 (145.0–292.5)	217.7 (171.1–274.1)	< 0.001	287.0 (180.0–450.0)	< 0.001
Urine Output, median (IQR), mL	205.0 (75.0–465.0)	185.0 (35.0–595.0)	< 0.001	487.5 (400.0–679.2)	< 0.001
Patient Outcome, n(%)
ICU mortality	1148 (46.5)	483 (44.4)	< 0.001	129 (10.5)	< 0.001

^aIncludes Asian, Native American, Pacific Islander, or missing data.

^bData not recorded in the dataset.

Race was incorporated as a feature to enhance comparability. The MIMIC-IV dataset includes diverse ethnicities, whereas the validation dataset lacks this information, hence we assumed a single ethnicity due to data limitations. The external validation cohorts show slightly higher median blood pressure readings compared to the derivation cohort. The Dr.ECC cohort shows comparably healthier features, such as a higher PaO₂/FiO₂ ratio, urine outputs, and a much lower mortality rate. The derivation cohort contains more utilization of intermittent hemodialysis (IHD) only treatment compared to the other two datasets, while AmsterdamUMCdb and Dr.ECC contain significantly more continuous renal replacement therapy (CRRT) only patients.

HRRT shows superior performance on modality decision-making over clinicians

To evaluate the HRRT policy, we conducted a series of offline analyses comparing its performance to the historical clinician policy, as summarized in Fig. 3. First, to evaluate the policy’s performance without deploying it in a live clinical setting, we employed off-line policy evaluation (OPE) using the Weighted Importance Sampling (WIS) method (Fig. 3A). The WIS metric consistently confirms HRRT performance superiority across both internal and external cohorts. In the MIMIC-IV internal validation cohort, the HRRT policy achieved a substantially higher expected return of −0.21 (95% CI: −0.56 to 0.15), clearly separating its performance from the baseline clinician −0.74 (95% CI: −1.01 to −0.46) and random policies −0.78 (95% CI: −1.08 to −0.48). This superior performance is robustly validated in the AmsterdamUMCdb external cohort. Since Dr.ECC dataset does not contain information about IHD, we only used AmsterdamUMCdb for external validation. The HRRT policy obtained a mean WIS return of −0.47 (95% CI: −0.64 to −0.29), which represents a marked improvement over the clinician policy, measured at −0.65 (95% CI: −0.83 to −0.46), and the random baseline, which performed similarly at −0.67 (95% CI: −0.83 to −0.49). The distinct separation of the HRRT confidence interval from the confidence intervals of both baseline policies provides strong evidence that the learned policy successfully generalizes to unseen patient populations and consistently achieves higher expected long-term outcomes than the standard of care.

Fig. 3. Performance comparison on the modality selection task for internal (MIMIC-IV) and external (AmsterdamUMCdb) validation set.

A OPE via WIS for clinician, random, and HRRT policy. B Box plots of Q-Values where the decisions disagree between clinicians and HRRT. C Mortality vs. Q-Value Analysis. The blue line is the clinician policy value and its corresponding mortality rate. The mean Q-value of the HRRT policy (red dashed line) maps to an estimated mortality rate. D The modality comparison between clinician and HRRT policy on trajectory-level and transition-level and the bar plot of modality distributions.

We further analyzed the agent’s learned Q-values (Fig. 3B), which represent its belief in the long-term value of a given action. The box plot compares the Q-values of the actions chosen by the clinician versus those chosen by the HRRT policy for the exact same states where the two policies disagree. This plot reveals that in cases of disagreement, the HRRT policy consistently assigns a higher Q-value to its own recommended action despite the imitation constraint. It shows that even within the safe set of actions that the clinician also used, the agent learned to successfully distinguish the high-value safe actions from the low-value safe actions.

To validate that these Q-values are clinically meaningful, we plotted the actual mortality rate against the Q-values of the clinician’s actions (Fig. 3C). This plot demonstrates a strong inverse correlation, confirming that higher Q-values are indeed associated with lower mortality. We then calculated the mean Q-value for the HRRT policy and mapped it to this curve, resulting in an estimated mortality of 41.6% (95% CI: 35.6–47.2) for MIMIC-IV compared to the mortality under clinicians’ action 47.7% (95% CI: 45.2–50.0) for the internal validation set, averaging across 5 seeds. For the external validation set, we achieved an estimated mortality rate of 39.0% (95% CI: 36.7–45.1) compared to 44.4% (95% CI: 38.4–46.3). This provides strong evidence that the HRRT policy’s superior Q-values translate to tangible improvements in predicted patient survival.

Finally, we analyzed the behavioral differences between the policies (Fig. 3D), examining trajectory-level and transition-level concordance, and aggregate modality distributions. First, the Trajectory-Level RRT Modality matrices (Fig. 3D Left) reveal how the agent manages entire patient episodes compared to the clinician. On the MIMIC-IV training set, there is notable concordance in the CRRT-only (211 trajectories) and IHD-only (95 trajectories) categories, indicating the agent frequently agrees with the clinician’s choice of modality consistency. On the unseen AmsterdamUMCdb validation set, the agent demonstrates robust generalization by maintaining high concordance on the most critical cases. The policy validates the clinician’s decision in the majority of CRRT-only trajectories (531 cases). In instances where the policy differs, it exercises prudent judgment by recommending IHD only (137 trajectories) or No RRT (166 trajectories), effectively identifying opportunities to minimize continuous invasive therapy and avoid potential over-treatment in an out-of-distribution (OOD) setting. This de-escalation is further detailed in the Transition-Level matrices (Fig. 3D Middle). Transition-level analysis was performed by comparing the actions given by the clinicians and RL policy for every state transition (t to t + 1).In the AmsterdamUMCdb cohort, the agent demonstrates a robust, safety-driven policy shift. This indicates that while the agent may agree with the need for dialysis at the trajectory level, it often recommends pausing or terminating CRRT sessions earlier than the clinician. The Modality Distribution bar charts (Fig. 3D Right) summarize these aggregate effects. In the MIMIC-IV training set, the HRRT policy is characterized by a distinct preference for IHD, showing a significantly higher volume of IHD steps compared to the clinician. In the AmsterdamUMCdb domain, the policy modulation is evident: the agent’s distribution is dominated by No RRT, effectively reducing the total volume of RRT compared to the clinician to avoid over-optimistic interventions on OOD patients.

HRRT shows superior performance on fluid management over clinicians

The IHD agent demonstrates a reasonable alignment with the clinician’s discrete ultrafiltration volume levels on the MIMIC-IV training set (Fig. 4A). As IHD is not always documented in ICU Electronic Health Records (EHRs) and the treatment strategy varies between different regions, validation was restricted to the internal dataset. We discretized the ultrafiltration volumes into four levels based on the observed data distribution: <1.4L, 1.4–2.0L, 2.0–3.0L, and >3.0L. The confusion matrix reveals that the agent’s policy is concentrated around the intermediate volume levels (Level 2 and 3: 1.4L to 3.0L), which corresponds to common clinical practice. While the agent and clinician do not always select the exact same level, the off-diagonal elements are clustered tightly around the diagonal, indicating that the agent’s deviations are generally conservative and within one level of the clinician’s choice. The mortality analysis (Fig. 4B) further supports the safety of this policy: the mortality rate remains stable when the absolute difference between the agent’s and clinician’s action level is small. A rise in mortality is only observed at the largest deviation (a difference of 3), suggesting the agent has successfully learned to mimic the low-mortality actions from the expert distribution.

Fig. 4. Comparison of treatment fluid decisions made by clinicians vs. HRRT on internal validation sets for IHD agent π_IHD.

A Heatmap showing the alignment between clinician-selected IHD ultrafiltration volumes (y-axis) and HRRT-recommended ultrafiltration levels (x-axis). For IHD fluid management, ultrafiltration volume was chosen as the action variable and discretized into four levels based on the observed data distribution: <1.4 L, 1.4–2.0 L, 2.0–3.0 L, and >3.0 L. Each cell represents the number of treatment instances corresponding to that pair. B Mortality rate (y-axis) as a function of the absolute difference in ultrafiltration level between clinician practice and the HRRT policy (x-axis). Larger action differences indicate greater deviation from the HRRT-recommended ultrafiltration volume. Shaded regions denote the 95% confidence interval.

The continuous-action CRRT agent shows excellent policy generalization across multiple domains. We discretized the continuous effluent rates into levels by dividing 10, where rates <10 mL/kg/hr or >60 mL/kg/hr were categorized into extreme classes. The confusion matrices for effluent rate levels (Fig. 5A) display a strong diagonal correlation not only on the MIMIC-IV training set but also on the external validation sets (AmsterdamUMCdb and Dr.ECC). This confirms that the policy successfully identifies and recommends appropriate dosage levels for unseen patient states in varying hospital systems. The policy exhibits a distinct preference for Levels 2 and 3 (corresponding to 20–30 mL/kg/h), aligning with standard clinical guidelines for safety and efficacy. Notably, the distribution of recommended effluent rates for the Dr.ECC cohort is shifted toward slightly higher levels (Levels 3 and 4) compared to MIMIC-IV and AmsterdamUMCdb. This distribution shift is likely attributable to the lower acuity of the Dr.ECC population. The policy’s tendency to recommend higher effluent rates in the Dr.ECC dataset reflects the agent’s sensitivity to the underlying patient severity. As the Dr.ECC cohort represents a comparatively healthier population with better physiological stability, the agent identifies that these patients can tolerate higher effluent flows for more efficient solute clearance, whereas it correctly constrains dosages in the higher-mortality MIMIC-IV and AmsterdamUMCdb cohorts to moderate solute shifts and preserve hemodynamic stability. The mortality analysis (Fig. 5B) provides a critical safety insight that holds true across all three datasets: patient mortality is consistently at its lowest when the absolute difference between the HRRT policy and the clinician’s action is small (0–1 levels). As the agent’s recommended action diverges significantly from the clinician’s (differences >2), the mortality rate trends upward. This mortality curve suggests an association between the learned policy and improved outcomes.

Fig. 5. Comparison of treatment fluid decisions made by clinicians vs. HRRT on internal and external validation sets for CRRT agent π_CRRT.

A Heatmaps showing the agreement between clinician-chosen CRRT effluent rates (y-axis) and HRRT-recommended effluent rates (x-axis) across three datasets (MIMIC-IV, AmsterdamUMCdb, Dr.ECC). Continuous effluent rates were discretized into 5 levels; values < 10 mL/kg/hr or > 60 mL/kg/hr were grouped into extreme bins. Each cell indicates the number of treatment decisions falling into the corresponding (clinician level, HRRT level) pair. B Relationship between the absolute action difference between clinician and HRRT policies (x-axis) and the observed mortality rate (y-axis). The x-axis reflects how far clinical practice deviates from the HRRT-recommended effluent rate level; the y-axis shows the corresponding mortality among those treatment instances. Shaded regions represent 95% confidence intervals.

Time of initiation and cessation shift between clinicians and HRRT

To evaluate the temporal characteristics of our learned HRRT policy, we compared its recommendations for dialysis initiation and cessation against the recorded clinician policy on the internal validation set. The results are visualized in Fig. 6. In Fig. 6A, the distribution for the clinician policy (top) demonstrates a peak of approximately 150 patients within the first few days post-ICU admission, accompanied by a broader distribution and a longer tail extending out to 30 days. In contrast, the HRRT policy (middle) is significantly more concentrated in the immediate post-admission period (0–2 days), peaking sharply at roughly 250 patients. This discrepancy is explicitly quantified in the bottom plot; the distribution is clearly shifted to the left of zero, with a mean in the negative values. This demonstrates that our learned policy is systematically more aggressive, tending to initiate RRT 1–3 days sooner than the clinician policy. Conversely, Fig. 6B reveals that the HRRT policy tends to maintain RRT for a longer duration than observed in standard clinical practice. While both policies exhibit a right-skewed distribution for the time from ICU admission to the last dialysis stop, the HRRT policy displays a broader distribution in the later stages of care. The bottom plot, which illustrates the distribution of the stop time difference (HRRT - Clinician), confirms this observation: the distribution is notably right-shifted, with the peak and majority of trajectories falling into positive territory. This indicates that the HRRT agent consistently recommends stopping RRT later than clinicians, suggesting a more conservative approach to weaning that may prioritize physiological stability and renal recovery over early cessation. Figure 6C reveals distinct behavioral differences between the two policies for active RRT duration. The clinician policy is characterized by a dominant spike at durations of less than 2 days with a long tail extending into longer durations, indicating a high prevalence of very short treatment courses. In contrast, the HRRT policy yields a smoother, broader distribution of treatment duration. The agent demonstrates a reduced tendency for immediate cessation, instead maintaining active therapy more consistently across the 2- to 10-day window, suggesting a learned strategy that prioritizes sustained physiological stabilization prior to weaning. The time of initiation and cessation shows a similar pattern on the external validation set; we report the results in Supplementary Fig. 2.

Fig. 6. Temporal validation of RRT initiation and termination on the internal MIMIC-IV cohort.

A Time from ICU admission to the initiation of the first RRT session. The histograms compare the temporal distributions of the Clinician (top) and HRRT policy (middle). The difference distribution (bottom) shows the trajectory-level time difference (t_HRRT − t_Clinician), where negative values indicate the HRRT policy initiates treatment earlier than the clinician. B Time from ICU admission to the termination of the last RRT session. C Distribution of active RRT duration (in days) under the clinician policy (left) and the proposed HRRT policy (right). The histograms illustrate the total time patients spent receiving renal replacement therapy before successful cessation or trajectory end.

Subgroup analysis based on length of ICU stay

To assess the policy’s effect on different patient cohorts, we performed a subgroup analysis based on the length of ICU stay (Fig. 7). We interpret the length of ICU stay as a proxy for underlying comorbidity burden.

Fig. 7. Subgroup analysis of mean ΔeGFR based on ICU stay length.

A Short Stay cohort. B Long Stay cohort. Blue and orange lines represent patients with low and high policy agreement, respectively. Shaded areas represent the 95% confidence intervals.

In the Short Stay cohort (Fig. 7A), representing patients with a lower comorbidity burden, a clear divergence in renal function is observed. Patients with high policy agreement demonstrate consistent improvement, with the mean change in estimated glomerular filtration rate (ΔeGFR) becoming positive within the first few days and stabilizing at approximately +10 mL/min/1.73m². Conversely, the group with low agreement shows an initial slight improvement followed by a steady decline after day 5. This suggests that for patients with lower comorbidity, adherence to the policy is associated with renal recovery, while non-adherence is linked to deterioration.

The Long Stay cohort (Fig. 7B) shows a different temporal pattern reflective of a higher comorbidity burden. The high-agreement group maintains stable renal function, with the mean ΔeGFR hovering around zero for the first 40 days, followed by a trend toward improvement in later stages. The low-agreement group mirrors this stability for the first 40–43 days; however, after this point, they experience a precipitous drop in eGFR, signifying rapid and severe worsening of renal function. For these high-comorbidity patients, the protective effect of policy adherence becomes most evident in the very late stages of critical illness.

Across both subgroups, high agreement with the learned HRRT policy is consistently associated with superior renal function outcomes—either rapid improvement (Short Stay) or maintenance followed by late recovery (Long Stay). In contrast, low policy agreement is consistently associated with renal function deterioration, a decline that is particularly abrupt and severe in the long-stay cohort.

Sensitivity analysis

To evaluate the robustness of our reinforcement learning framework, we performed a sensitivity analysis regarding the discretization of the decision time step. Our primary analysis utilizes a 6-h interval, a choice motivated by the granularity of urine output measurements in the MIMIC-IV dataset and the physiological reality that IHD sessions frequently exceed 4 h. We excluded a 1-h interval as it necessitates an unrealistically high frequency of re-evaluation for RRT, which is practically infeasible in a clinical setting. To test for stability, we retrained and evaluated the policy using a 4-h interval (Supplementary Fig. 3). The results indicate that the policy’s performance is robust to this variation. The WIS for the 4-h interval confirms that the HRRT policy (−0.27, 95% CI:−0.57 to 0.03) continues to significantly outperform the clinician policy (−1.10, 95% CI:−1.36 to −0.85). Furthermore, the behavioral characteristics—including the strong negative correlation between Q-values and mortality (Supplementary Fig. 3C) and the preference for IHD in the modality distribution (Supplementary Fig. 3D) remain consistent with the 6-h baseline, demonstrating that the learned policy is stable and not an artifact of the specific temporal discretization.

Reward function design calibration

Since the reward signal implicitly defines the agent’s objective, its careful design is essential for effectively guiding the policy learning toward clinically optimal strategies. Previous sensitivity analyzes in medical RL have highlighted that learned policies can be dangerously sensitive to reward formulations²⁴. Past papers on treatment recommendation typically consider two primary settings: terminal(long-term) rewards based solely on patient outcomes²⁵, or combined with intermittent (short-term) rewards that integrate patient state changes^19,26. Therefore, we conducted experiments that compare mortality only rewards R_mort and a mixed reward R_mix using Sequential Organ Failure Assessment (SOFA)²⁷ score. We selected the SOFA score as the basis for the intermediate reward because it provides a holistic evaluation of patient status across multiple organ systems, reflecting genuine physiological improvement rather than simple solute clearance. Critically, we excluded direct filtration markers such as urea and creatinine from the reward function. Since the serum concentrations of these small solutes are mechanically controlled by the CRRT settings, including them would introduce a spurious feedback loop. This analysis allows us to understand how nuanced treatment effects and outcomes are related to the applicability of our results. The reward functions are defined as follows:

$$R_{mort}=\left\{\begin{array}{cc}+a,\hfill & \mathrm{if}{s}_{t+1}\in S_T\cap S_{\mathrm{sur}}\hfill \\ -a,\hfill & \mathrm{if}{s}_{t+1}\in S_T\cap S_{\mathrm{dec}}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right)$$ 1

$$R_{mix}=\left\{\begin{array}{cc}+a,\hfill & \mathrm{if}{s}_{t+1}\in S_T\cap S_{\mathrm{sur}}\hfill \\ -a,\hfill & \mathrm{if}{s}_{t+1}\in S_T\cap S_{\mathrm{dec}}\hfill \\ R_{\mathrm{inter}}\left(s_t,s_{t+1}\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right)$$ 2

$$R_{\mathrm{inter}}=-c\cdot \left({\mathrm{SOFA}}_{t+1}-{\mathrm{SOFA}}_t\right)-b\cdot \mathbb{I}\left({\mathrm{SOFA}}_{t+1}={\mathrm{SOFA}}_t>0\right)$$ 3

where S_T means set of terminal states, S_sur means the set of survived patients and S_dec for deceased patients. a, b, c are hyperparameters of weights for each component. In our analysis, we explored various scenarios to assess the impact of different reward structures

• The effect of parameter a on R_mort.

• The effect of parameters b, c on R_mix.

• The comparison between R_mort and R_mix.

First, we assessed the effect of the terminal reward magnitude a on R_mort by comparing R_mort with a = 10 and a = 100. As shown in Table 2, the WIS values for the static baselines (Clinician and Random) scale linearly with the change in the reward parameter (exactly 10 times), as these policies are fixed evaluations of the test data. In contrast, the HRRT policy scales approximately, but not linearly (ratio ≈ 8.6). This divergence occurs because HRRT is a learned policy parameterized by a neural network; increasing the reward scale alters the magnitude of the Q-values during training. Due to the non-linear Softmax activation in the BCQ imitation head, larger Q-values result in a sharper, more deterministic probability distribution compared to the a = 10 setting. Consequently, the learned policy π_θ varies slightly between the two settings. Despite this numerical variance in scaling, the policy’s relative advantage remains robust. The a = 10 setting achieved a 19.09% relative improvement in WIS, while the a = 100 setting achieved a 2.03% improvement. Importantly, both reward scales drove the agent toward a similar mortality reduction (13.42% and 11.74%, respectively), confirming that the optimal policy is stable regardless of the absolute magnitude of the terminal reward a. Our results suggest that for a mortality-only objective, the HRRT policy is robust to the absolute scale of the terminal reward signal.

Table 2.

Analysis on different reward function weights with relative OPE metric changes (%)

Reward	Method	WIS (%↑)	Mortality (%↓)
Rmort(a = 100)	Clinician	5.91	47.7
	Random	3.73	–
	HRRT	6.03 (2.03)	42.1 (11.74)
Rmort(a = 10)	Clinician	0.59	47.7
	Random	0.37	–
	HRRT	0.70 (18.64)	41.3 (13.42)
Rmix(a = 10, b = 1, c = 1)	Clinician	−27.13	47.7
	Random	−22.62	–
	HRRT	−16.27 (40.03)	45.7 (4.19)
Rmix(a = 10, b = 0.033, c = 0.25)	Clinician	−0.74	47.7
	Random	−0.78	–
	HRRT	−0.21 (71.62)	41.6 (12.79)

The percentage for WIS is the relative improvement over the clinician, while for Mortality it is the relative reduction.

We further analyzed the relative weights of the intermediate reward parameters b and c (Table 2). Operating under the prerequisite that the terminal reward must significantly outweigh cumulative intermediate signals to prioritize patient survival, we first tested the uncalibrated setting where b = c = 1. As expected, this equal-weight configuration proved misaligned with the clinical objective. The large intermediate weights create a cumulative penalty that effectively overpowers the terminal reward signal (a = 10). Consequently, the agent pathologically optimizes for these intermediate proxies at the expense of the primary survival goal. This misalignment explains the discrepancy in our results: while the policy achieved a substantial relative WIS improvement (+40.03%), this optimization translated into a negligible reduction in mortality (4.19%). Subsequently, we calibrated the parameters to set the expected total intermediate reward ∣R_intermediate∣ ≈ 2.0, utilizing a theoretical analysis based on average trajectory lengths (Supplementary Section 4). By ensuring the intermediate signal remains secondary to the survival goal, we achieved the most robust policy performance. This configuration yielded the highest relative WIS gain (+71.62%) and, crucially, the most significant reduction in patient mortality (12.79%). These findings confirm that calibrated intermediate rewards effectively guide the agent toward safer, survival-oriented strategies without distracting from the ultimate therapeutic objective.

Finally, we evaluated the trade-off between pure outcome optimization (R_mort, a = 10) and holistic patient management (R_mix, b = 0.033, c = 0.25) (Table 2). Theoretically, the policy optimizing exclusively for the terminal signal should achieve the absolute minimum mortality rate. Our empirical results align with this theory: the R_mort policy achieved a marginally lower estimated mortality (41.3%) compared to the optimal R_mix (41.6%). Despite this slight numerical advantage for the mortality-only policy, we posit that the calibrated R_mix represents the superior clinical strategy. In healthcare, the therapeutic objective is not merely survival, but the maintenance of patient well-being and the minimization of organ failure during the ICU stay. The calibrated R_mix achieves a mortality reduction comparable to the theoretical optimum (12.79% vs 13.42%) but does so by actively minimizing the SOFA score at every time step. This aligns with recent findings that intermediate physiological guidance can produce more robust policies by ensuring the agent maintains the patient in a stable condition throughout the trajectory, rather than potentially allowing physiological degradation to satisfy a sparse terminal signal²⁸.

Model interpretability

To explain the decision-making logic of the HRRT agent compared to human experts, we performed a feature importance analysis using SHAP (Shapley Additive Explanations) values (Fig. 8). The clinician policy (Fig. 8A) is characterized by a heavy reliance on immediate fluid status and hemodynamic markers, with the 6-h urine output rate and serum sodium acting as the dominant drivers for RRT initiation. This aligns with traditional clinical heuristics that prioritize oliguria and volume management. In contrast, the HRRT policy (Fig. 8B) exhibits a paradigm shift toward metabolic and solute-based indicators. The agent identifies creatinine, urea, and bicarbonate as the most critical features, assigning them significantly higher importance than urine output. This indicates that the reinforcement learning policy has moved beyond reactive volume management to a more proactive, solute-oriented strategy, prioritizing the clearance of uremic toxins and the correction of metabolic acidosis over simple urine volume metrics. Furthermore, while both policies recognize the importance of hemodynamic stability, the HRRT agent’s specific focus on filtration biomarkers suggests a more precise targeting of renal metabolic demand rather than just hemodynamic support.

Fig. 8. Shapley Additive Explanations (SHAP) values.

A Feature importance for clinician policy. B Feature importance for HRRT policy.

Discussion

In this multicohort retrospective validation study spanning three international ICUs, we developed and evaluated a HRL framework to provide end-to-end management and optimization of RRT for ICU patients with AKI, including initiation and weaning timing, modality selection, and dynamic adjustment of ultrafiltration. The model was trained and internally validated using the MIMIC-IV database, with external validation performed on two independent cohorts. Together, these three datasets represented 4782 stays of more than 3000 critically ill patients across different countries and health systems, enabling robust evaluation of model generalizability. As seen in Figs. 3–5, HRRT achieved better performance compared to the clinician policy on both the internal and external validation sets for all the tasks.

Using the HRRT framework to provide a customized dose of RRT can improve outcomes through several distinct mechanisms. Firstly, the temporal analysis (Fig. 6 and Supplementary Fig. 2) reveals a distinct shift in treatment duration. The HRRT policy, akin to other AI-based predictive models¹⁷, demonstrates a capacity to integrate multidimensional data to identify subtle trajectories of worsening AKI earlier than standard clinical heuristics. This allows the agent to initiate therapy empirically earlier in specific high-risk phenotypes, adopting a proactive approach to prevent toxic accumulation and cascading metabolic failure before they become critical. Conversely, the policy tends to terminate treatment later, ensuring that renal support is maintained until physiological stability is fully consolidated, thereby reducing the risk of premature weaning and subsequent relapse. Secondly, the policy demonstrates a context-aware selection of modality (Fig. 3). Rather than adhering to a rigid protocol, the agent promotes the use of IHD when efficient clearance is feasible while strictly limiting CRRT in sensitive or OOD scenarios to maximize patient safety. Thirdly, as shown in Fig. 5, HRRT generally matches clinical effluent dosing, and significant deviations from the model’s recommended dosage are associated with increased mortality, supporting the clinical safety of the learned policy. Finally, our subgroup analysis (Fig. 7) highlights the critical role of cumulative management consistency in patients with a high comorbidity burden. During the early acute phase of critical illness, overwhelming severity often obscures the isolated benefit of concordant RRT decisions. However, as patients transition into a prolonged ICU stay (e.g., beyond 43 days), their physiological reserve becomes severely depleted by chronic comorbidities and cumulative ICU-related stressors, such as secondary nosocomial infections or prolonged exposure to nephrotoxic agents. In this fragile state, prolonged policy divergence—reflecting inconsistent treatment goals or repeated shifts in care philosophy—inflicts intolerable physiological stress, leading to the precipitous late-stage renal deterioration observed in our analysis. High agreement with the HRRT policy mitigates this risk by providing a consistent, data-driven management strategy throughout extended stays.

Beyond optimizing treatment timing and selecting therapy modalities, our findings highlight the potential of RL to move beyond the long-standing debate of the optimal dose of RRT towards individualized and dynamic treatment strategies. Large randomized controlled trials such as the VA/NIH acute renal failure trial¹² and RENAL study¹³ have already established that higher-intensity dosing did not improve mortality compared with standard therapy. More recently, real-world analysis did not demonstrate a consistent dose-response relationship across a range of effluent intensities. These results demonstrate that there is no universal dosing threshold and underscore the importance of individualized titration. However, current practice is heterogeneous, usually guided by clinician experience, institutional protocols, or even isolated laboratory values rather than the illness trajectory. The HRRT framework aims to address this gap by learning sequential treatment policies that account for evolving patient physiology, with the goal of aligning the dose delivery of RRT closer to the principles of individualized care. This study reports the first use of applying RL to RRT dosing to sharpen clinical decision-making for ICU patients with AKI.

The HRRT model employs a hierarchical decision-making approach that integrates the identification of key decision nodes with personalized adjustments based on the dynamic physiological evolution of patients during RRT. This framework addresses a critical limitation in most clinical AI applications: the fragmentation of decision support. AI systems that provide decision support only at isolated points in the continuously evolving patient journey may undermine trust among clinicians and patients. In contrast, HRRT offers an end-to-end coverage, evaluating all potentially eligible patients and guiding every relevant decision phase. Consequently, our model advances beyond narrow-scope applications to establish a comprehensive, full-spectrum clinical decision support system (CDSS) for continuous patient management.

A notable strength of this study is the comprehensive validation using three independent datasets across North America, Europe, and Asia. Despite significant heterogeneity in baseline demographics, practice patterns, and resource availability, the HRRT policy consistently outperformed clinician-observed policies in off-policy evaluations (OPEs). The model demonstrated robust transferability, effectively generalizing across cohorts with distinct modality distributions: a balanced mix of IHD and CRRT in the MIMIC-IV training set versus a predominantly CRRT-driven practice in the AmsterdamUMCdb and Dr.ECC cohort. Consequently, while local calibration and contextual adaptation remain important for implementation, the underlying decision-making framework is transferable across diverse global populations.

The interpretability analysis using SHAP (Fig. 8) further validates the clinical relevance of the HRRT policy’s decision-making logic. While the clinician’s policy heavily emphasizes fluid balance metrics such as urine output rate and serum sodium, reflecting traditional indicators for volume management, the HRRT agent demonstrates a sophisticated prioritization of metabolic markers. Specifically, the agent identifies creatinine, urea, and bicarbonate as the most critical features driving intervention. This prioritization aligns with the pathophysiological goals of RRT: the clearance of uremic toxins and the correction of metabolic acidosis. By elevating these solute-based indicators, the reinforcement learning model effectively targets the underlying metabolic derangements that necessitate renal support, suggesting that its policy is grounded in robust, physiologically sound signals essential for patient survival and recovery.

The primary limitation of this study lies in its retrospective design based on offline data. We used historical clinician actions as a comparative baseline that are subject to human error and practice variation; this is largely addressed by RL, which learns optimal policies through experience rather than merely imitating historical behavior. Although we employed robust OPE methods to estimate policy value, these techniques cannot definitively establish efficacy. Secondly, while our model demonstrated robust generalizability across three large international databases, the findings may not seamlessly translate to smaller, community-based hospitals. Validation in these settings is currently restricted by the scarcity of high-resolution EHR data from smaller centers, which represents a crucial avenue for future research. Furthermore, dataset-inherent constraints introduced specific analytical challenges such as relatively shorter follow-up time and the severe class imbalance in RRT modalities. The imbalance restricted external validation and necessitated the oversampling of IHD interventions to prevent policy collapse, which artificially alters the state-action distribution and may introduce algorithmic bias. Consequently, future work is essential to bridge the gap between algorithmic validation and clinical deployment. Our next steps focus on prospective validation through RCTs to rigorously confirm the safety and tangible clinical benefits of the HRRT model in real-time decision-making²⁹.

In conclusion, HRRT represents the first end-to-end clinical decision support system utilizing HRL to provide comprehensive management of RRT. HRRT policy consistently identifies optimized therapeutic windows, advocating for earlier initiation and a more conservative, later cessation of therapy compared to standard empirical practice. Furthermore, the model’s prioritization of metabolic and physiological markers over static indicators aligns with pathophysiological goals to potentially reduce ICU mortality and enhance renal recovery. RRT modality selection and timing of initiation are influenced not only by physiologic parameters of the patient, but also by institutional resources, staffing as well as patient preferences. Our model does not explicitly incorporate these operational or contextual factors and, therefore, should be interpreted as a decision-support tool rather than a prescription. Implementation will require local calibration and clinician oversight to ensure alignment with institutional workflows and patient-centered care.

Methods

Database and cohort construction

For the derivation cohort, the data for training and internal validation were sourced from the public MIMIC-IV v3.1 database²², a comprehensive, de-identified health dataset for 94,458 stays of 364,627 patients collected from the ICUs at the Beth Israel Deaconess Medical Center from 2008 to 2022. Ethics approval for the use of the MIMIC-IV dataset was granted by the Institutional Review Boards (IRB) of the Massachusetts Institute of Technology and Beth Israel Deaconess Medical Center. We included adult patients (≥18 years old) who received IHD or CRRT treatment during their ICU stay (length of stay should be less than 60 days) and mortality was documented. For external validation, we used 2 datasets, AmsterdamUMCdb²³ and Dr.ECC. AmsterdamUMCdb contains 23,106 admissions of 20,109 patients admitted to Amsterdam University Medical Center from 2003 to 2016. AmsterdamUMCdb is a public critical care database comprising de-identified health data from patients admitted to the ICUs of the Amsterdam University Medical Centers in the Netherlands. Access to AmsterdamUMCdb was approved by the Amsterdam University Medical Centers’ ethics committee. Dr.ECC is a non-public database of emergency critically ill patients led and established by Sir Run Run Shaw Hospital that includes more than 90,000 patients with emergency visit records from 2017 to 2024, covering data of patients from outpatient and emergency visits, hospitalization, to intensive care unit treatment. For the Dr.ECC dataset, ethics approval and IRB clearance were obtained from the Institutional Review Board of Sir Run Run Shaw Hospital, Zhejiang University School of Medicine (Reference No. 2025-Research-1320). For all three datasets, the requirement for individual patient informed consent was waived by the respective ethics committees due to the retrospective nature of the study and the use of de-identified clinical data.

Data extraction and preprocessing

Details of the cohort selection process are documented in Fig. 1. 133,139 ICU stays from three datasets were evaluated for potential inclusion. After applying the exclusion criteria, 4782 unique patient stays were included in our study. There were 2467 stays for MIMIC-IV dataset where we used a 8:2 split for training and an internal validation set. 2315 stays were included for external validation in total, where 1085 stays were from AmsterdamUMCdb, and 1230 stays were included in Dr.ECC. Each stay was considered as a single patient trajectory and we used 6-h interval to discretize the data. The detailed sensitivity analysis about the binning interval is detailed in “Sensitivity analysis”. Each patient's stay is described by 18 clinical features (Supplementary Table 1), which encompass patient demographics, vital signs, lab test results, and urine outputs. For IHD fluid management, we selected the ultrafiltration volume as the action variable and discretized it into four levels based on the data distribution (<1.4 L, 1.4–2.0 L, 2.0–3.0 L, and >3.0 L). For CRRT fluid management, we calculated the effluent rate based on CRRT subtypes. The definition of effluent rate for different subtypes is in Supplementary Table 2. The missing rates for features are reported in Supplementary Table 3. To handle missing features, we employed forward and backward filling for general features within the same ICU stay. For fluid dosages, we imputed values based on RRT sessions, defining a new session whenever the gap between two dialysis records exceeded 8 h.

HRRT framework

We propose the HRRT (Fig. 2), a two-step hierarchical decision-making framework designed to optimize RRT decisions. Fundamentally different from monolithic models that struggle with complex hybrid action spaces, HRRT decomposes the problem into a Hierarchical Markov Decision Process (HMDP). This enables HRRT to integrate specialized offline RL algorithms at each stage, thereby effectively addressing the distinct challenges of the hybrid action space associated with modality selection and fluid management task.

Fig. 2. Overview of HRRT workflow.

A We extracted patient states and clinician actions from MIMIC-IV dataset, the IHD modality data are oversampled to mitigate the data imbalance issue for modality agent training. B Overview of the hierarchical training process. Step 1 is the modality selection that uses BCQ to choose whether to initiate dialysis and which modality. It learns the feature representations through the baseline networks, and then predicts the q values through the q head and the i head will imitate the clinicians' actions to mask out the less possible actions. In Step 2 Fluid Management the respective IHD or CRRT agent will learn to recommend exact fluid dosage or rate. C Evaluation of HRRT framework.

HRRT takes patient clinical data and historical clinician actions from the EHR database as input (Fig. 2A). The patient data include demographics, vital signs, and lab test results, which form the state. Clinician action data include whether the patient is on dialysis for each time step, and if so, which modality. The action data also include details on IHD ultrafiltrate volumes (discrete actions) and CRRT effluent rates (continuous actions). HRRT treats the patient's state trajectories as the environment. The high-level action is selecting a modality (No RRT, IHD, or CRRT), and the subsequent low-level action is selecting a specific fluid management dosage. Thus, RRT optimization is framed as a HMDP of high-level modality selection followed by low-level dosage determination.

Central to HRRT is a two-level policy represented by specialized neural networks (Fig. 2B). The first step, Modality Selection (Fig. 2B left), involves a high-level policy (π_modality) that selects one of three discrete options. To train this policy, we employ discrete Batch-Constrained Q-Learning (BCQ)³⁰ to mitigate extrapolation error, a critical failure mode in offline RL where the agent overestimates the value of OOD actions. It achieves this by using an imitation head (I) to learn the clinician’s behavior, which then creates a plausibility mask. This mask constrains the Q head (Q), forcing it to only select the best action from a safe subset of actions that are well-supported by the data, thus preventing a policy collapse.

Upon selection of a modality, HRRT activates one of two specialized, low-level agents for Step 2. Fluid Management (Fig. 2B right). For the discrete-action IHD task that selects an ultrafiltrate volume, we utilize Conservative Q-Learning (CQL)³¹. CQL addresses overestimation by adding a pessimistic penalty term to the Bellman loss, pushing down the Q-values of unseen actions and stabilizing the policy. For the continuous-action CRRT task to determine an effluent rate, we employ Implicit Q-Learning (IQL)³². IQL avoids extrapolation error by implicitly constraining the policy to actions well-supported by the observed data. By applying the well-chosen algorithm to each component, HRRT learns a robust, multi-level policy from the static offline clinical dataset.

In the evaluation phase (Fig. 2C), the performance of the full HRRT policy is compared against the clinician baseline using multiple evaluation methods, including OPE via WIS, mortality vs. Q-value, and analyzes of renal function and treatment timing.

Hierarchical Markov decision process

Standard Markov Decision Process (MDP) model tasks satisfying the Markov property³³ can struggle with long-horizon problems or complex, multi-stage decisions. To address this, we formulate our problem as a HMDP, which decomposes the decision-making process into multiple levels of policies³⁴. In our two-level hierarchy, a top-level policy, π_h, does not select primitive actions. Instead, it selects from a set of options or sub-policies, $o\in \mathcal{O}$. An option is a complete policy in itself that runs for multiple time steps until it reaches a termination condition. Once an option o is selected, a lower-level policy, π_o(a∣s), executes the primitive actions a at each subsequent time step. This framework allows the agent to reason at a higher level of abstraction. The top-level policy learns the long-term strategy of which sub-policy to invoke, while the lower-level policies learn the fine-grained, step-by-step actions required to implement that sub-policy. Formally, the environment’s dynamics are still defined by the primitive actions, where $p\left(s\prime ,r\mid s,a\right)$ is the probability of transitioning to state $s\prime$ with reward r after taking primitive action a in state s:

$$p\left(s^{\prime },r\mid s,a\right)=\mathrm{Pr}\left\{S_{t+1}=s^{\prime },R_{t+1}=r\mid S_t=s,A_t=a\right\}.$$ 4

However, the agent’s policies operate at different levels of this hierarchy to manage the complexity of the task. The MDP components for our study are defined as follows:

• Policy π: A policy defines the learning agent’s behavior by mapping the perceived environmental state to a specific action. Given the complexity of determining both the therapeutic modality and the precise dosage, we formulated the task using a hierarchical structure. The decision-making burden is distributed across three distinct sub-policies: a high-level modality policy π_modality, and two low-level policies π_IHD and π_CRRT, respectively.

• State S: At each time step t, the agent receives a state vector s_t representing the patient’s clinical condition. To facilitate the optimization of a dynamic treatment regime, this representation incorporates patient demographics, vital signs, laboratory results, and other relevant biomarkers^24,35. The complete list of features is detailed in Table 1, this feature set remains consistent across all hierarchical subtasks.

• Action A: At each time t, the agent executes an action a. We utilize a hybrid action space: actions for the modality and IHD policies are discrete, whereas the CRRT policy action is continuous. Specifically, the modality policy selects from {No RRT, IHD, CRRT}; the IHD policy selects from four discrete levels of ultrafiltration volume; and the CRRT policy outputs a continuous effluent flow rate.

• Reward r: The reward signal provides feedback to the agent, defining positive and negative therapeutic events. Our framework incorporates both short-term rewards, which provide immediate feedback on physiological state changes, and long-term rewards associated with patient outcomes. The methodology for calibrating this reward function is elaborated in Section “Reward function design calibration”.

Analysis on oversampling

A significant challenge in the dataset was the severe class imbalance within the dialysis action space. In the original data, IHD was recorded in only 2% of observed actions. This sparsity presents a major obstacle for offline RL algorithms like BCQ, which rely on sufficient data coverage to estimate values and constrain the policy to the behavioral distribution. When trained on the original imbalanced data (Fig. 9 No-Oversampling), the model failed to learn the value of IHD entirely, resulting in a degenerate policy that never recommended IHD. To address this, we employed an oversampling strategy to incrementally increase the prevalence of IHD samples (4%, 6%, 8%, and 10%) in the training buffer. At lower ratios (4% and 6%), the agent began to identify IHD but remained hesitant. For example, at 6%, the agent matched the clinician’s IHD decision in only 83 cases. Conversely, aggressive oversampling at 10% caused the policy to deviate significantly from clinical norms, recommending IHD in 3350 instances where the clinician deemed no RRT necessary, an unrealistic and potentially unsafe escalation of care. Since the objective of BCQ is to optimize decisions within a safe, clinically plausible range, we identified 8% as the optimal configuration. In this setting, the agent effectively learned the relative value of IHD, achieving a balanced distribution that improved recognition of IHD needs without the excessive false positives observed at higher ratios. This balanced policy yielded the highest estimated WIS and a significantly improved estimated mortality rate.

Fig. 9.

Impact of IHD oversampling on policy action distribution.

Offline reinforcement learning

A primary challenge in developing a clinical decision-support agent from existing medical records is the static, offline nature of the data. Standard online reinforcement learning algorithms, such as Double Dueling Deep Q-Networks (D3QN)³⁶, are not designed for this setting. They are exploration-based since usually they can interact with the environment to correct mistaken beliefs. When applied to an offline dataset, they are highly susceptible to extrapolation error: the agent learns to exploit OOD actions that were rare in the dataset but which, due to spurious correlations or function approximation errors, appear to have unrealistically high Q-values. This exact failure mode was observed when a standard D3QN agent was first applied to our high-level modality task. The policy quickly collapsed, learning to recommend IHD almost exclusively. The root cause is a spurious correlation in the data: IHD, while the rarest modality, is associated with a less severe patient cohort and thus lower mortality. The D3QN agent incorrectly learned that the IHD action caused this low mortality and developed an explosive, over-optimistic Q-value for it, leading to a total policy collapse. To mitigate this critical extrapolation error, our HRRT framework employs a unified hierarchy of specialized offline algorithms. Each algorithm is specifically chosen to address the unique data challenges, such as class imbalance, data sparsity, or action space complexity, at its stage of the hierarchy.

For the high-level modality agent, we selected BCQ³⁰. The modality task is characterized by severe class imbalance, with IHD actions being significantly rarer than CRRT or No RRT. BCQ solves the overestimation problem by using a hard constraint. It has an imitation head to learn the clinician’s policy, and the RL agent is only permitted to select actions that fall within this learned, safe distribution. This directly prevents the agent from exploiting the IHD action and ensures its high-level decisions remain robust and safe. For the low-level IHD agent, a similar issue was observed. Even when the training data was balanced, the total number of IHD samples was still small, and a standard D3QN again collapsed. We therefore employed CQL³¹. Instead of a hard constraint, CQL introduces a soft constraint by adding a conservative penalty to the loss function. This penalty pushes down the Q-values of all OOD actions, forcing the agent to be pessimistic about any action not well-supported by the data, thereby achieving stability. Finally, for the low-level CRRT agent, we need to handle the continuous action space of the effluent rate. For this, we employed IQL³², a state-of-the-art offline algorithm that is particularly effective for continuous control. Unlike BCQ’s hard constraint or CQL’s soft penalty, IQL works by implicitly constraining the policy. It learns the Q-function by performing regression only on actions observed in the dataset, which prevents the agent from querying or overestimating the value of OOD actions. This approach provides robust stability for the continuous control task and completes our fully offline framework. Please refer to Supplementary Section 1 for the details about each algorithm and why we chose a decoupled method instead of end-to-end training.

Evaluation of reinforcement learning model

We conducted a comprehensive evaluation of the trained HRRT agent using various qualitative and quantitative metrics designed to assess both model performance and clinical safety without risks to the patients. To estimate policy value, we employed WIS (Fig. 3A), which re-weights observed retrospective returns to calculate the expected cumulative reward of the HRRT policy relative to clinician and random baselines. This was complemented by an analysis of Q-values (Fig. 3C), where we constructed a curve linking clinician values to observed ICU mortality, thereby validating the reward function as a proxy for survival and allowing us to estimate the agent’s potential mortality benefit. To understand the agent’s strategic behavior, we utilized confusion matrices (Fig. 3D) at both the trajectory level, to assess overall modality agreement, and the transition level, to distinguish between fundamental disagreements in necessity versus temporal shifts in initiation. Safety was further rigorously evaluated for the low-level IHD and CRRT agents by plotting observed mortality against the absolute deviation between the agent’s recommended dosage and the clinician’s action (Figs. 4 and 5), verifying that the policy operates within a low-mortality therapeutic window. We also validated the temporal plausibility of the policy by comparing the distributions of RRT initiation and cessation times against clinical practice to ensure the agent respects physiological constraints (Fig. 6, and Supplementary Fig. 2). Finally, we analyzed the changes in renal functional outcomes. We stratified patients from the internal validation cohort into short stay (<20 days, Fig. 7A) and long stay (>20 days, Fig. 7B) groups. For each group, we evaluated ΔeGFR³⁷ over time. We plotted the ΔeGFR against time for the trajectories with high versus low agreement.

Statistical analysis

Python 3.11.0 (Python Software Foundation, Wilmington, DE, USA) was used for signal preprocessing, model development and validation, statistical testing, and visualization. Continuous variables were described as mean(SD) or median (IQR) and categorical variables as counts (percentage). For global outcome metrics such as mortality where patient-level variance is aggregated into a single policy value, uncertainty was quantified across 5 independent experimental runs initialized with different random seeds.

Author contributions

Q.X., K.C.S., M.F., Z.T. and M.G. conceived the idea and study design. Y.H. and Z.Z. coordinated data acquisition and provided essential resources. Q.X., P.C., J.Y. and C.H. extracted and curated data. Q.X. and F.W. performed methodology design, data analysis and provided the figures/tables. Q.X. and Z.Y.C.T drafted the manuscript, and M.F., K.C.S. and M.G. revised it critically for important intellectual content. M.F., K.C.S., Y.H. and Z.Z. supervised the project or provided clinical interpretation. All authors have read and approved the final manuscript.

Data availability

MIMIC-IV is publicly available via https://physionet.org/content/mimiciv/3.1/ and AmsterdamUMCdb is publicly available via https://amsterdammedicaldatascience.nl/. Dr.ECC dataset is not publicly available due to institutional restrictions on data privacy.

Code availability

The codes used for generating the results of this study are available at: https://github.com/xuqianyi/RL-RRT.

Competing interests

The authors declare no competing financial or non-financial interests. Author Z.Z. is Associate Editor of npj digital medicine. Z.Z. was not involved in the journal’s review of, or decisions related to, this manuscript.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02900-2.