A Reinforcement Learning (RL)-Motivated Simulation Framework for Evaluating Vancomycin Dosing Strategies

Abstract

Achieving and maintaining the therapeutic range in vancomycin treatment is important for optimal outcomes. While guidelines and best practices based on empirical studies exist, the theoretical best dosing strategies under various conditions remain illusive. We developed an RL-based simulation framework using a deep learning two-compartment pharmacokinetic model (PK-RNN-2CM) and introduced the area under the time-concentration curve (AUC) reward score, which translates clinical guidelines into an RL reward. Ground truth time-concentration curves were generated from patient-specific data, and simulated curves were produced under different dosing strategies with optional noise perturbations to mimic real-world settings. Evaluation metrics included 24-hour AUC assessments and RMSE. Results indicated that while the low-dosing AUC target (low-doser) and the high-dosing AUC target (high-doser) performed comparably in noise-free conditions, the low-doser achieved slightly higher AUC reward scores under noisy conditions, whereas the high-doser exhibited greater stability. This framework opens new approaches for optimizing vancomycin dosing.

Introduction

Optimizing vancomycin dosing to rapidly achieve the therapeutic range is an important clinical challenge, given its importance in ensuring effective treatment while minimizing toxicity^1–3. Therapeutic drug monitoring (TDM) plays a central role in personalizing vancomycin therapy, but there remains uncertainty in identifying the best dosing and measurement strategies to reliably reach the therapeutic range early in treatment^2,4,5. The most updated vancomycin clinical guidelines recommend targeting a ratio of area under the curve over 24 hours to minimum inhibitory concentration (AUC/MIC) of ≥400, with a further recommendation of an AUC range of 400 to 600 mg·h/L (assuming a MIC of 1 mg/L) to both optimize efficacy and avoid acute kidney injury³. Currently, Bayesian models have been widely used in hospitals for dosing recommendations because they integrate populational pharmacokinetic (PK) parameters with patient-specific information to predict vancomycin concentrations and then guide dosing decisions^4,5. Recent advancements in deep learning have led to the development of the PK-RNN series models PK-RNN-1CM⁶ and PK-RNN-2CM⁷, which dynamically predict vancomycin concentration over time with higher accuracy than a Bayesian vancomycin therapeutic drug monitoring model (VTDM) introduced by Lim et al. (2014)⁵. However, while these two deep learning models provide more accurate concentration prediction, they lack the dosing recommendation component in clinically used Bayesian models⁴. This project is important because it seeks to evaluate different dosing strategies, addressing the current gap where advanced deep learning models have not yet been fully integrated into dosing decision frameworks. Inspired by previous studies using reinforcement learning (RL) to optimize dosing recommendation^8,9, we conceptualized our simulation approach to mimic an RL environment: a dosing agent interacts with a simulated patient PK system called oracle (the environment), makes decisions for dosing prescriptions (actions), and receives feedback (rewards) based on how well the predicted AUC meets therapeutic targets. Although this implementation does not include a fully autonomous RL agent, these RL principles guided the design for future dynamic dosing strategies, distinguishing between static dosing strategies and those that adapt based on predicted concentration changes over time, which represents a promising direction for future improvements of PK-RNN models.

Previous work focusing on vancomycin PK dosing strategies has largely aimed to develop predictive models and static dosing recommendations based on Bayesian methods^2,4,5,10. These approaches have improved dosing decisions by using patient-specific information, they typically rely on priors from a specific patient population and may not fully capture the complexity of real-time, adaptive decision-making. Deep learning models PK-RNN-1CM and PK-RNN-2CM have the ability to dynamically predict the vancomycin concentration based on time step during the treatment, but they can’t provide dosing recommendations. Moreover, some simulation-based studies have explored various dosing strategies to optimize the AUC within the therapeutic range^8,11,12, there remains a gap in developing more advanced, data-driven approaches that can both predict concentrations and potentially offer dynamic dosing recommendations. This gap highlights the need for a comprehensive framework that not only compares the performance of established Bayesian models with newly developed deep-learning approaches but also sets the stage for future developments in dynamic dosing.

To address this gap, we developed a simulation framework that comprises two key components: an oracle and a dosing agent. The oracle, based on the pre-trained PK-RNN-2CM model, generates ground truth time-concentration curves using different dosing strategies without real measurements. In contrast, the dosing agent simulates clinical decision-making by receiving a single concentration measurement from the oracle and then using the PK-RNN-2CM model along with real patient-specific data available up to the first dose to generate subsequent time-concentration curves. The agent’s performance is evaluated by comparing its simulated curves with the oracle’s ground truth curves, using metrics such as AUC and root mean square error (RMSE). This study thus not only compares static dosing strategies between Bayesian and deep learning models but also lays the foundation for future dynamic, RL-inspired dosing systems that can continuously refine therapeutic decisions.

Methods

Data source and preprocessing

This study used the same real-world dataset as the development of PK-RNN models^6,7, which was extracted from an electronic health record (EHR) data warehouse of the Memorial Hermann Health System (MHHS) in Houston, Texas. The cohort included adult patients (aged >18 years) who received at least one dose of vancomycin between August 2019 and March 2020. Patients with renal replacement therapy or with vancomycin level measurements taken at inappropriate times were excluded. All patient data were de-identified to ensure privacy. The start time of each encounter was defined as the earliest recorded timestamp, and the end time was determined by the time of the last vancomycin concentration measurement. These time points, including the vancomycin administration time, the vancomycin level, and the end-of-day timestamp, were used as discrete time steps to update the parameters of the PKRNN-2CM model.

In the MHHS dataset, the duration of vancomycin infusion varied with the dosage: doses ≤1000 mg were infused over 1 hour, 1001-1500 mg over 1.5 hours, 1501-2000 mg over 2 hours, and doses exceeding 2000 mg over 2.5 hours. For simplicity, a uniform infusion rate of 1 g/hour was assumed, and measurements were not taken during the infusion period. Missing values were imputed using the most recent available measurement under the assumption of clinical stability; if no values were available for a given patient, the overall mean was used. Continuous variables were standardized using Z-scores.

Simulation framework

An RL-motivated simulation framework was developed to evaluate vancomycin dosing strategies. The framework includes an oracle simulation that generates ground truth PK trajectories and an agent simulation that generates time-concentration curves based on different dosing prescriptions and the information from the oracle. Together, these simulations provide a systematic basis for comparing and optimizing dosing prescriptions.

Figure 1 shows the simulation project framework used in this study. The process begins with the dosing agent transmitting the initial loading dose, which is derived from the patient’s original first dose in the MHHS dataset to the oracle. The oracle then employs the pre-trained PKRNN-2CM model to extract patient-specific PK parameters and generate the time-concentration curve for the first dosing cycle. Based on this curve, the oracle provides a simulation measurement, defined as the trough vancomycin concentration before the second dose, to the dosing agent. The dosing agent uses this measurement to update the patient’s PK parameters, which are then input into the PK estimator (PKRNN-2CM) to generate estimated time-concentration curves for all candidate dosing prescriptions. After calculating the estimated AUCs for each candidate curve, the dosing agent forwards these values to the dosing AUC target module. This module selects the best-estimated dosing prescription based on different dosing AUC targets, and the dosing agent submits the chosen prescription to the oracle. Finally, the oracle rolls out the remaining time-concentration curve for subsequent dosing cycles, provides an evaluation measurement, and computes the oracle AUCs. These results are used to assess the performance of the selected dosing prescription through the primary AUC reward score and a secondary point-wise RMSE evaluation.

Figure 1.

Simulation project framework

A figure depicts the RL-based simulation framework that integrates an oracle and a dosing agent to generate, update, and evaluate vancomycin time-concentration curves.

Oracle simulation

The oracle simulation is designed to produce a ground truth concentration-time curve for each patient. Using the pre-trained PKRNN-2CM model, the oracle generates the PK trajectory based on complete patient-specific information extracted from the MHHS dataset and the dosing prescription provided by the dosing agent. The oracle provides two types of measurements to the dosing agent based on the dosing prescription. The dosing agent uses the simulation measurement to update its internal time-concentration curve, while the evaluation measurement serves to assess the accuracy of the agent’s estimated trajectory. This simulation does not include feedback from measured vancomycin levels in the real data and serves as the gold standard for evaluating dosing prescriptions.

Dosing agent simulation

The dosing agent functions as the decision-making component, relying exclusively on patient data and information provided by the oracle. It receives a single simulation measurement from the oracle to update its representation of the patient’s time-concentration curve. The dosing agent has two key components. The first component is the PK estimator, which uses the deep learning model PKRNN-2CM to compute the patient’s PK parameters based on the original first dose and the simulation measurement from the oracle. The second component is the dosing AUC target, implemented in two variants, the high-doser and the low-doser, that select the dosing prescription that best meets the predefined AUC targets. The agent operates on a finite set of dosing prescriptions defined by specific dose values (500 mg, 750 mg, 1 g, 1.25 g, 1.5 g, 1.75 g, and 2 g) and dosing intervals (8 hours, 12 hours, and 24 hours). Furthermore, the evaluation measurement is used at two time points: the peak concentration after the second dose or the trough concentration before the fourth dose.

For the dosing agent with a PK estimator PKRNN-2CM, the dosing algorithm is implemented by a forward rollout procedure. First, the patient receives the initial dose with the original value and time from the MHHS dataset, and the oracle provides a simulation measurement along with all relevant patient information up to that time. This measurement is then processed by the PK estimator to derive an estimate of the patient’s PK parameters. With these estimates, the dosing agent performs a forward simulation of the PK trajectory under every candidate dosing prescription. For each dosing prescription, the agent calculates the estimated AUC over three subsequent intervals: 0-24 hours, 24-48 hours, and 48-72 hours following the second dose. The prescription that provides the best estimated AUC across these intervals is selected as the best dosing prescription. This selected dosing prescription is submitted to the oracle, which generates the corresponding ground truth PK trajectory. The final performance of the dosing agent is assessed by comparing its estimated time-concentration curve with the oracle’s ground truth curve to compute an AUC reward score.

Incorporation of noise

To more accurately reflect the variability in clinical settings, the simulation framework includes an optional noise setting that introduces perturbations into the agent simulation. Noise can be applied either to the simulation measurement or to the first dose of the patient, both of which are used for generating and updating the time-concentration curve. Two types of noise are considered: value noise and time noise. When noise is applied to the value of the simulation measurement, the modified measurement is randomly set to either 0.9 or 1.1 times the original value. Alternatively, if noise is added to the value of the first dose, then for a dose value less than 1g, a fixed increment of 0.25 is added; for a dose value equal to or greater than 1, a random perturbation of plus or minus 0.25 is applied. In addition to value noise, time noise can also be introduced. When time noise is implemented, the dosing or measurement time is randomly adjusted by plus or minus 10 minutes. Value noise and time noise capture different types of clinical variability. Value noise captures inaccuracies in the amount of drug administered or the concentration measured, such as rounding errors or recording mistakes. In contrast, time noise reflects uncertainties in the recorded timing of events, which may occur because of workflow delays or documentation errors in real clinical settings. By modeling both value and time noise independently, we aim to extract the effects of quantity-based and time-based variability, providing a more detailed simulation of real-world clinical data. These noise configurations enable the simulation framework to capture the uncertainties present in real-world dosing and measurement scenarios.

Dosing prescription selection

The best estimated dosing prescription is selected based on the estimated AUC. Clinical guidelines recommend targeting an AUC/MIC of 400–600 mg·h/L per 24-hour period, and for simplicity, we assume a MIC of 1 mg/L. The dosing agent uses a dosing AUC target to evaluate and compare different dosing prescriptions. For each patient, the agent calculates estimated AUC values over the intervals 0-24 hours, 24-48 hours, and 48-72 hours following the second dose. The best dosing prescription is defined as one in which all three AUC values fall within the target range of 400 mg·h/L to 600 mg·h/L. When multiple dosing prescriptions meet this criterion, different dosing AUC targets are used to select the best option: the low-doser approach selects the prescription with AUC values closest to 450 mg·h/L, while the high-doser approach selects the one with AUC values closest to 550 mg·h/L. As a result, each patient may have two best dosing prescriptions selected, although in some cases they may be the same. It is important to note that low-doser and high-doser refer to dosing prescription selection strategies, not categories of prescriptions. They are not used to classify prescriptions or patients, but rather to compare different prescription options that already meet the clinical target range. For each patient, both strategies may provide a valid prescription and are not treated as mutually exclusive categories.

PK-RNN-2CM

The PK-RNN-2CM model represents a novel deep learning-based approach for vancomycin therapeutic drug monitoring. Recognizing the limitations of one-compartment models, which are commonly used due to their simplicity, the PK-RNN-2CM model integrates recurrent neural network-driven PK parameter estimation with a two-compartment PK model. This approach leverages both simulated data and real-world electronic health record data to predict vancomycin concentration trajectories. Experimental results demonstrated that PK-RNN-2CM⁷ significantly outperforms the simpler PK-RNN-1CM6 model, as evidenced by lower RMSE values on both simulated and real data⁷. In this study, the PKRNN-2CM model was chosen as the foundation of the simulation framework due to its previously demonstrated accuracy and ability to handle real-world clinical variability. While other established TDM models, such as Bayesian approaches, are commonly used in clinical practice^6,7, we focused exclusively on PK-RNN-2CM to evaluate the effects of different dosing prescription strategies. The goal of this work is not to benchmark predictive models, but to improve the process of dosing decision-making under various clinical scenarios.

Evaluation metrics

The evaluation is conducted to compare the best dosing prescription’s performance selected from the dosing agent against the oracle’s ground truth. The best estimated dosing prescription selected by the agent is submitted to the oracle, which then generates the corresponding time-concentration curve. The overall performance is quantified using two metrics. The first metric is the AUC reward score, whereby each of the three AUC intervals is assigned a score of +1 if its value lies within the range of 400 to 600, and 0 if it falls outside this range. The second metric is the point-wise RMSE calculated between the evaluation measurement provided by the oracle and the corresponding values from the agent-generated time-concentration curve. Together, these metrics provide a comprehensive assessment of the dosing agent’s ability to both select the best dosing prescription and accurately predict the patient’s PK trajectory.

Implementation details

Patients were divided into training, test, and validation sets based on patient identification in a ratio of 70:15:15. Both the oracle and dosing agent simulations, which use the PKRNN-2CM model, were trained and validated using real MHHS data, while simulation experiments were conducted on the test set. Hyperparameters for model training were maintained as described by Nigo et al. (2022)⁶ and Mao et al. (2024)⁷. In the simulation, the peak measurement was defined as the concentration observed 2 hours after a dose if the dose was less than or equal to 1g, or 3 hours after a dose if it exceeded 1g. The trough measurement was defined as the concentration observed 1 hour before the dose. The study was implemented based on PyTorch 1.9.0 using Python 3.8 (Python Software Foundation).

Results

Data descriptive analysis

This study includes 5,483 patients from the MHHS dataset with 8,689 encounters, as shown in Table 1. The median weight of the patients was 82.9 kg, and the median height was 172 cm. The demographic analysis included age, gender, and race/ethnicity. 3,069 (55%) patients were male, with a median age of 61 years. The largest racial group was White, comprising 36.4% of the population, followed by African Americans (less than 20%) and Asians (less than 2%). Additionally, 783 (14.2%) patients identified as Hispanic.

Table 1.

Descriptive analysis of the study cohort

Characteristics	Number (%) or median (IQR)
Basic Characteristics
Patients	5,483
Encounters	8,689
Weight (kg)	82.9 (65.5 – 101.6)
Height (cm)	172 (165.1 – 181.1)
Demographics
Age	61 (48-73)
Gender
Male	3,069 (55%)
Race and ethnicity
White	2,003 (36.4%)
African American	1,069 (19.5%)
Asian	83 (1.5%)
Non-Hispanic	3,905 (71.2%)
Hispanic	783 (14.2%)

IQR: Interquartile range.

Primary results for AUC-related evaluation metrics

The AUC-related results highlight how low- and high-dosers perform under noise-free conditions across different time intervals, offering insights into median AUC alignment with target values and variability among patients, as shown in Figure 2. The boxplot compares the oracle-derived AUC distributions, with the x-axis representing the time intervals of 0-24, 24-48, and 48-72 hours after the second dose, and the y-axis showing the average AUCs over 3 repeats. Two boxplots are shown per time interval, corresponding to the two dosing AUC targets (low-doser and high-doser). The yellow-shaded region marks the target AUC therapeutic range of 400-600 mg·h/L, and the red dashed lines represent the high-doser target of 550 mg·h/L while the low-doser target of 450 mg·h/L. From Figure 2, the low-doser provides a median average AUC always close to 450 mg·h/L for the 0-24 and 24-48 hour intervals, whereas the high-doser centers around 550 mg·h/L with a larger distance. During the 48-72 hours, the median average AUC of the high-doser is closer to the target, suggesting the differences in earlier intervals decrease over time. Regarding variability, the interquartile ranges (IQRs) provide insight into how tightly or broadly the average AUCs cluster around their respective medians. In particular, the 0-24 hour window generally has a smaller IQR for both dosing AUC targets, indicating more consistent AUCs shortly after the initial dose, while the 48-72 hour window shows wider IQRs, reflecting greater variability in later treatment stages. Additionally, the boxplots show that the number of outliers is highest for the 0-24 hour interval, suggesting that some patients experience AUCs well above or below the central distribution, where fewer outliers appear in the 48-72 hour interval, even though the variability is larger overall.

Figure 2.

A boxplot of the distributions for the oracle average AUC without noise

The yellow shaded area indicates the therapeutic AUC range from the clinical guidelines.

Tables 2 and 3 present the findings related to AUC evaluation metrics. When comparing the average AUC reward scores between the low-doser (targeting an AUC of 450 mg·h/L) and high-doser (targeting an AUC of 550 mg·h/L), no statistically significant difference was observed under the noise-free condition (p = 4.61e-01, unpaired t-test).

Table 2.

Results table of AUC-related evaluation metrics for low-doser

Noise type	Avg. AUC reward score (STD)	Avg. correct AUC%(STD)	Avg. AUC (0-24) (STD)	Avg. AUC (24-48) (STD)	Avg. AUC (48-72) (STD)
Without noise	2393.33 (60.73)	59.58 (1.51)	462.94 (7.32)	463.95 (11.86)	423.32 (28.28)
Measurement value	2284.67 (156.55)	56.88 (3.90)	487.15 (21.03)	484.55 (13.98)	439.13 (36.03)
Measurement time	2285.33 (156.51)	56.89 (3.90)	487.11 (21.16)	484.57 (13.96)	439.20 (36.04)
Dose value	2288.33 (145.43)	56.97 (3.62)	486.74 (23.09)	486.11 (14.41)	439.06 (34.74)
Dose time	2267.0 (141.88)	56.44 (3.53)	486.8 (21.60)	485.53 (14.50)	436.42 (38.53)

AUC: the area under the time-concentration curve; STD: standard deviation over 3 repeats.

Table 3.

Results table of AUC-related evaluation metrics for high-doser

Noise type	Avg. AUC reward score (STD)	Avg. correct AUC%(STD)	Avg. AUC (0-24) (STD)	Avg. AUC (24-48) (STD)	Avg. AUC (48-72) (STD)
Without noise	2430.00 (19.20)	60.49 (0.48)	524.85 (7.45)	514.18 (16.07)	518.14 (34.78)
Measurement value	2279.67 (72.64)	56.75 (1.81)	552.24 (29.18)	537.12 (16.79)	531.03 (29.66)
Measurement time	2280.33 (71.70)	56.76 (1.79)	552.12 (29.08)	537.09 (16.80)	530.97 (29.65)
Dose value	2285.33 (86.15)	56.89 (2.14)	554.03 (28.00)	537.95 (17.70)	532.19 (32.32)
Dose time	2243.67 (53.85)	55.86 (1.34)	550.60 (28.80)	537.25 (16.97)	529.39 (32.46)

AUC: the area under the time-concentration curve; STD: standard deviation over 3 repeats.

When different types of noise are introduced, while small differences between AUC reward scores remain consistent, the low-doser consistently achieves higher AUC reward scores across all noise types. This observation suggests that the dosing AUC target aimed at achieving an AUC of 450 mg·h/L may lead to a greater proportion of dosing prescriptions within the therapeutic range of 400-600 mg·h/L, especially for real-world clinical scenarios with noise. In addition, analysis of the standard deviations (STDs) across 3 repeats indicates that the high-doser produces lower variability compared to the low-doser. Given the small difference between the AUC reward scores, the high-doser could be a better dosing AUC target for stability.

The percentage of correct AUC values, which is defined as the proportion of AUCs falling within the therapeutic range, followed a similar pattern to the AUC reward scores. Both low- and high-dosers achieved approximately 60% correct AUCs under noise-free conditions, which decreased to around 56% when noise was present. Moreover, the effects of adding measurement value noise and measurement time noise were similar for both dosing AUC targets, as well as the difference compared to the noise introduced by measurement value and dose value. Notably, noise added to the dose time consistently resulted in lower AUC reward scores and a reduced percentage of correct AUC values for both dosing AUC targets.

Both tables also display the average AUC values calculated for the time intervals of 0-24 hours, 24-48 hours, and 48-72 hours across 3 repeats. For the low-doser, the average AUC values were consistently close to the target of 450 mg·h/L with the values for the 0-24 hour and 24-48 hour intervals generally higher than 450, while those for the 48-72 hour interval were lower. The AUCs from the no-noise scenario show small differences within the 3 AUCs, indicating a steady patient condition maintained by the selected best-estimated dosing prescription from the dosing agent. The average AUC values for the high-doser show a similar trend with the AUCs around the target value of 550. Interestingly, when comparing the median AUCs from Figure 2 and the average AUCs from Tables 2 and 3, the median AUC 48-72 hour is higher than the median AUCs 0-24 hour and 24-48 hour for both low- and high-dosers, while the average AUCs for 48-72 hours showing lower than the other two. This indicates that the asymmetric distribution of AUCs from the selected best dosing prescriptions is left-skewed. As shown in Figure 2, the interquartile range remains consistent across time intervals, but there are more extreme high outliers than low ones. However, because the mean is sensitive to extreme values, even a few very low AUCs have a stronger impact on lowering the average, whereas the median remains robust to these outliers. This pattern shows that while most patients maintain stable or increasing AUCs over time, a subset experiences AUCs much lower than 400, likely contributing to the observed divergence between the mean and median.

Figure 3 provides a summary of the average AUC versus the percentage of correct AUC for both low- and high-dosers across various scenarios. As in Figure 2, the yellow area indicates the therapeutic AUC range, while the red dashed lines represent the target AUCs for low- and high-dosers. Filled shapes correspond to AUCs from the low-doser, while unfilled shapes represent AUCs from the high-doser. The different shapes denote the AUC time intervals: circles for AUC (0-24), squares for AUC (24-48), and triangles for AUC (48-72). The colors of the shapes differentiate between experimental scenarios: no noise, measurement value noise, measurement time noise, dose value noise, and dose time noise.

Figure 3.

Average AUC vs. Percentage of correct AUC for different dosing AUC targets

Secondary results for RMSE-related evaluation metrics

Table 4 presents the secondary results, which focus on the RMSEs calculated between the oracle’s ground truth and the best estimated dosing prescription selected by the dosing agent. In these analyses, two RMSE metrics were evaluated: the point-wise RMSE, which compares individual concentration measurements, and the RMSE of AUCs, which assesses the overall accuracy of the AUC predictions. Both low-doser and high-doser exhibited comparable average point-wise RMSEs across all scenarios, with the high-doser consistently achieving slightly lower point-wise RMSEs. In contrast, the low-doser provided lower RMSE of AUCs in all scenarios, indicating a superior ability to predict AUCs that are closer to the oracle’s ground truth.

Table 4.

Results table of RMSE-related evaluation metrics

	Low-doser		High-doser
Noise type	Avg. RMSE (STD)	Avg. RMSE of AUC (STD)	Avg. RMSE (STD)	Avg. RMSE of AUC (STD)
Without noise	2.47 (0.19)	50.89 (3.39)	2.38 (0.12)	59.36 (3.74)
Measurement value	2.49 (0.11)	61.80 (10.85)	2.37 (0.29)	70.53 (11.53)
Measurement time	2.50 (0.13)	61.83 (10.88)	2.39 (0.27)	70.51 (11.46)
Dose value	2.57 (0.06)	62.19 (10.52)	2.35 (0.19)	70.55 (11.75)
Dose time	2.43 (0.15)	62.56 (10.41)	2.35 (0.29)	70.91 (11.49)

RMSE: root mean square error; STD: standard deviation over 3 repeats.

Furthermore, the introduction of noise led to an increase in the RMSE of AUCs. While different types of noise did not produce statistically significant differences in the RMSE of AUCs overall, noise added to the dosing parameters resulted in slightly higher RMSE of AUCs compared to noise introduced in the measurement process. Interestingly, the average RMSE in the scenario of no noise was not the lowest for either dosing AUC target, although the differences were minor. This result may be attributable to the fact that the point-wise RMSE was based on a single evaluation measurement, potentially leading to less stable error estimates.

Discussion

In this study, we evaluated vancomycin dosing strategies using an RL-based simulation framework. The primary results based on AUC-related evaluation metrics demonstrated that the average AUC reward scores were comparable between the low- and high-dosers without noise, with no statistically significant differences observed. The boxplot further demonstrated that the low-doser provided median average AUCs closer to its target for the 0-24 and 24-48 hour intervals, with smaller IQR for all time intervals when compared with the high-doser. After introducing noise, the low-doser consistently achieved slightly higher AUC reward scores than the high-doser across all noise scenarios, suggesting that targeting an AUC of 450 mg·h/L may result in more AUCs within the therapeutic range of 400-600 mg·h/L throughout the 3 days after the second dose compared to targeting an AUC of 550, particularly in real-world clinical settings where noise can be common. Despite this, the standard deviation of AUC reward scores was consistently lower for the high-doser, indicating that targeting an AUC of 550 mg·h/L may provide greater stability. This difference in variability is unlikely to be caused by any instability in the underlying PK-RNN-2CM model, which remains consistent across both dosing targets. Instead, it likely arises from interactions between the fixed set of predefined dosing prescriptions and patient-specific PK profiles. For some patients, there may be more candidate prescriptions falling near 550 mg·h/L than near 450 mg·h/L, leading to greater prescription selection variability for the low-doser. The percentage of correct AUC values followed a similar trend, remaining around 60% for both dosing AUC targets without noise and decreasing slightly to 56% across all noise scenarios. It is interesting that noise added to dose time had the largest impact, leading to lower AUC reward scores and a reduced percentage of correct AUC values compared to other noise types. The average AUC values across 0-24 hours, 24-48 hours, and 48-72 hours showed that the low-doser consistently achieved AUC values around the target of 450 mg·h/L, with a slightly decreasing trend over time, whereas the high-doser exhibited a similar pattern around their target of 550 mg·h/L. The secondary results, based on RMSE-related evaluation metrics, provided further insights into the predictive performance of the two dosing AUC targets. Both low- and high-dosers exhibited similar point-wise RMSEs, with the high-doser achieving slightly lower values, suggesting its advantage in predicting concentrations. However, when considering the RMSE of AUCs, the low-doser consistently outperformed the high-doser, indicating a closer prediction to the oracle’s ground truth of AUCs. Adding noise caused an expected increase in RMSE values, though no statistically significant differences were observed between different noise types.

However, noise added to the dose resulted in slightly higher RMSE of AUCs compared to measurement noise, suggesting that inaccuracies in dosing have a greater impact on AUC predictions than measurement. Interestingly, the RMSE values without noise were not the lowest for either dosing AUC target, possibly due to the single measurement used for evaluation, which may have introduced variability.

The major contribution of this study is the development of an RL framework that incorporates a novel evaluation metric, the AUC reward score. While previous studies have explored RL models for dosing recommendations8,9, our work is the first to introduce an RL framework focusing on vancomycin dosing strategies evaluation. We defined the evaluation metric AUC reward score, which effectively translates the clinical guideline of targeting an AUC/MIC ratio of 400-600 mg·h/L into an RL reward. By doing so, the framework bridges the gap between clinical dosing recommendations and informatics, allowing for the systematic selection of optimal dosing prescriptions based on simulated patient responses.

Another key contribution of this work is the implementation of two dosing AUC target strategies. Based on the therapeutic range of 400-600 mg·h/L, the low-doser approach targets an AUC of 450, and the high-doser targets an AUC of 550. Compared with directly using the AUC therapeutic range from the clinical guidelines, several literature related to vancomycin dosing recommendations mentioned a further target of 450¹³ or 550¹⁴, but to the best of our knowledge, we are the first research to compare the difference between those two dosing AUC targets. This dual-target strategy provides a robust comparison and evaluation of different dosing prescriptions, offering insights into how subtle variations in target AUC can impact the proportion of dosing prescriptions that fall within the therapeutic range.

Furthermore, this study incorporates different types of noise into the simulation framework. One of the limitations of the PK-RNN-2CM model is that there’s no noise introduced during model development, while noise commonly appears in real-world datasets⁷. In this work, we tested several different types of noise, including measurement value noise, measurement time noise, dose value noise, and dose time noise. This addition makes the simulation framework more reflective of real-world clinical conditions and paves the way for future directions of adding noise into deep learning models like PK-RNN-2CM. By accounting for these factors, our approach enhances the robustness and practical applicability of the dosing recommendations.

Despite these advances, our study has several limitations. First, we only tested a finite set of dosing prescriptions, which may partly explain why the best-selected dosing prescription resulted in only approximately 60% of AUC values falling within the therapeutic range. Second, the point-wise RMSE was evaluated based on a single measurement, potentially leading to less stable error estimates. Third, our analysis was conducted using retrospective data from a single healthcare system, which may limit the generalizability of our findings to other populations or clinical settings. Fourth, the simulation framework was developed using the PKRNN-2CM model and was not evaluated with alternative TDM models such as Bayesian approaches. It remains to be evaluated whether the RL-based framework would provide similar results when connected to different underlying PK models. Future work should focus on extending the framework and conducting more comprehensive comparisons across multiple TDM models.

In conclusion, our RL-based simulation framework, featuring the novel evaluation metric AUC reward score and the incorporation of various types of noise, provides a promising approach for optimizing vancomycin dosing strategies. The results suggest that targeting a lower AUC may improve the likelihood of achieving the therapeutic range to improve patient outcomes, even under noisy conditions. Future work should first focus on adapting the RL framework into an autonomous system capable of selecting and adjusting dosing prescriptions without relying on predefined options. Subsequent steps include expanding the set of candidate dosing prescriptions to improve flexibility and personalization, refining evaluation metrics to better capture clinical relevance, and validating the framework in diverse clinical settings to ensure robustness and generalizability. In addition, advancing toward a fully autonomous RL agent will require incorporating real-time patient data streams, dynamic learning from ongoing treatment outcomes, and prospective clinical validation to ensure safety and effectiveness. This study represents a step toward more personalized and data-driven clinical decision-making in vancomycin therapeutic drug monitoring.