AI‐assisted warfarin dose optimisation with CURATE.AI for clinical impact: Retrospective data analysis

Abstract

Background

Standard‐of‐care for warfarin dose titration is conventionally based on physician‐guided drug dosing. This may lead to frequent deviations from target international normalized ratio (INR) due to inter‐ and intra‐patient variability and may potentially result in adverse events including recurrent thromboembolism and life‐threatening hemorrhage.

Objectives

We aim to employ CURATE.AI, a small‐data, artificial intelligence‐derived platform that has been clinically validated in a range of indications, to optimize and guide warfarin dosing.

Patients/methods

A personalized CURATE.AI response profile was generated using warfarin dose (inputs) and corresponding change in INR between two consecutive days (phenotypic outputs) and used to identify and recommend an optimal dose to achieve target treatment outcomes. CURATE.AI's predictive performance was then evaluated with a set of metrics that assessed both technical performance and clinical relevance.

Results and conclusions

In this retrospective study of 127 patients, CURATE.AI fared better in terms of Percentage Absolute Prediction Error and Percentage Prediction Error of 20% compared to other models in the literature. It also had negligible underprediction bias, potentially translating into lower bleeding risk. Modeled potential time in therapeutic range with CURATE.AI was not significantly different from physician‐guided dosing, so it is on‐par yet provides a systematic approach to warfarin dosing, easing the mental‐burden on guesswork by physicians.

This study lays the groundwork for the prospective study of CURATE.AI as a clinical decision support system. CURATE.AI may facilitate the effective use of affordable warfarin with a well‐established safety profile, without the need for costly, new oral anticoagulants. This can have significant impact both on the individual and public health.

Translational Impact Statement.

Physician‐guided warfarin dosing lead to frequent deviations from INR and adverse events. CURATE.AI, an artificial intelligence‐derived platform, has potential as a Clinician Decision Support System to improve accuracy of warfarin dosing decisions compared to physician‐guided dosing. This can facilitate the effective use of affordable warfarin with a well‐established safety profile, without the need for costly, new oral anticoagulants. This can have significant impact both on the individual and public health.

1. BACKGROUND

Warfarin is one of the most widely prescribed oral anticoagulant ¹ , ² , ³ that is FDA‐approved for the prophylaxis and treatment of thromboembolic conditions, including deep venous thrombosis, pulmonary embolisms, atrial fibrillation, cardiac valve replacement and dilated cardiomyopathies. ⁴ , ⁵ , ⁶ These conditions affect a large proportion of the population and contribute to significant morbidity and mortality. ⁷ Pulmonary embolism alone results in 250,000 hospitalizations each year and approximately 200,000 deaths. ⁸ Approximately 200,000 strokes each year can be attributed to thromboembolism. ⁹ There are 10 million cases of venous thromboembolism occurring every year worldwide ¹⁰ , ¹¹ and over 7 million cases of ischemic stroke. ¹²

Preventing and treating thromboembolism is thus of major public health importance. Standard‐of‐care (SOC) for warfarin dose titration is conventionally based on physician‐guided drug dosing where physicians make educated guesses about factors causing deviation from the target range and the amount by which to adjust the dose in response. ³ The complexity of inter‐ and intra‐patient variability in response to drug dosage lead to frequent deviations from the target international normalized ratio (INR) ranges. ¹³ INR is a measure derived from prothrombin time that reflects the coagulability of blood. INR is recognized as the standard for laboratory monitoring of warfarin therapy as it is convenient to measure and is an established means for clinicians to monitor anticoagulation therapy and use for warfarin dose adjustment. ¹⁴ , ¹⁵ , ¹⁶ Keeping a patient's INR within target therapeutic range is critical to the maintenance of anticoagulation in patients at risk of venous thromboembolism. ¹⁷ , ¹⁸ Given warfarin's narrow therapeutic index, over‐ and under‐coagulation may potentially result in adverse events including recurrent thromboembolism and life‐threatening hemorrhage. ¹⁹ , ²⁰ , ²¹ This forms a major challenge in warfarin therapy causing significant medical and economic consequences and diminishing the patient's quality of life. ²²

This has led to a recent shift toward the use of expensive New Oral Anticoagulants (NOACs) like Apixaban, Rivaroxaban, and Dabigatran. ²³ However, there remain indications and comorbidities for which warfarin is preferred, such as prosthetic valves, ²⁴ , ²⁵ , ²⁶ , ²⁷ antiphospholipid syndrome, ²⁸ and a high risk of gastrointestinal bleeding. ²⁸ , ²⁹ , ³⁰ , ³¹ Furthermore, warfarin is cost‐effective, has a well‐established safety profile, ³² can be managed by patients themselves with PT/INR self‐testing devices ³³ and can be easily reversed in case of over‐coagulation whereas, specific reversal agents for NOACs face issues of clinical availability and affordability. ³⁴ Warfarin thus maintains a stronghold in the armamentarium of anticoagulation. ³⁵

In an attempt to better manage warfarin therapy, various dosing algorithms have been developed. ³⁶ , ³⁷ However, these algorithms are not without limitations. First, they often require parameters ranging from environmental to genetic factors known to affect warfarin pharmacokinetics (PK) and pharmacodynamics (PD). ³⁷ , ³⁸ , ³⁹ The incorporation of such extensive clinical information makes them cumbersome and resource‐demanding. For example, dosing algorithms that require input of genetic information necessitates the availability of rapid‐turnaround genotyping which may not be feasible especially in resource‐poor environments. ⁷ Second, most of them are population‐based and hence require population‐derived values or require training on extensive population data, with a limited ability to account for inter‐ and intra‐individual variability. ⁴⁰ There thus remains a need for a truly personalized yet implementable approach for managing warfarin dosing.

CURATE.AI is an artificial intelligence (AI)‐derived, small data platform that has been clinically validated for dose optimisation in oncology and immunosuppression, among other indication areas. ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ CURATE.AI follows a similar essence to the personalized construction and use of digital twins to potentially tailor treatments for patients. ⁵⁰ CURATE.AI correlates patient‐specific drug and dose (inputs) to the corresponding efficacy or toxicity (phenotypic outputs) to generate a N‐of‐1 (personalized) CURATE.AI response profile. ⁴⁰ Based on the profile, CURATE.AI identifies and recommends an optimal dose to achieve the target treatment outcomes for the individual patient whose data were provided. As more patient dose and response data are collected along the course of treatment, the CURATE.AI profile can evolve with the patient's response across dose range and time, resulting in a robust and sustainable dose optimisation process. ⁴⁰ Of note, population‐based big data sets are not required or utilized in CURATE.AI. ⁴⁹ Another key feature of CURATE.AI is the optimisation based on phenotypic output, rather than modeling the complex system that determines drug effects: drug bioavailability, PK/PD, drug–drug interactions, medication adherence or patient comorbidities among many others. In CURATE.AI, these phenomena are inherently incorporated into the input–output relationship and, as such, CURATE.AI is both disease mechanism‐independent and indication‐agnostic. ⁵⁰ Taken together, CURATE.AI is a promising algorithm for warfarin therapy.

In this study, we aimed to employ CURATE.AI to retrospectively model the optimisation of warfarin dosing for individual patients, based on each patient's data only. We explored both quadratic and linear relationships as the basis for CURATE.AI dose optimisation. ²⁰ , ⁴⁰ CURATE.AI's performance was then evaluated with a set of metrics that assessed both technical and clinical performance to demonstrate CURATE.AI's robustness and support the potential application of CURATE.AI as a clinical decision support system (CDSS) in the prospective management of warfarin dosing.

2. METHODS

2.1. Data collection

This retrospective study was approved by the National Healthcare Group Domain Specific Review Board, NHG DSRB 2020/00019. The study analyzed the clinical records of 127 patients who underwent warfarin therapy for deep venous thrombosis, pulmonary embolism and atrial fibrillation in the year of 2018 in the National University Hospital (NUH), a tertiary hospital in Singapore. De‐identified data containing patient demographics, prescribed warfarin doses, corresponding observed INR and date of INR measurements was obtained from Discovery.AI, a platform that collates NUH patient data in an anonymized fashion. Data on genetic factors were not required in the CURATE.AI dosing optimisation process and were not collected.

2.2. Personalized dosing optimisation process with CURATE.AI

The process of CURATE.AI dose optimisation using second order (CURATE.AI Quadratic) and first order functions (CURATE.AI Linear) is demonstrated below using simulated patient data.

2.2.1. Simulated CURATE.AI quadratic process

Figure 1a,b demonstrate the CURATE.AI Quadratic dosing optimisation process using simulated data. In this simulation, warfarin dose was modulated against the change in the phenotypic output—change in INR between two consecutive days ($∆$ INR). Using a minimum of three uniquely modulated doses, the administered warfarin dose (input), represented by $W$, and the corresponding $∆$ INR response (output) from days 1, 2, and 3 were mapped via a second order response equation $∆\mathit{INR}\left(W\right)=W^2-6.5W+9.5$ (Figure 1a) where the equation coefficients [1; −0.65; 9.5] were specific to the simulated case at the given timepoint. This equation represents the patient's profile. The $∆$ INR and INR predictions for day 4 are computed from this initial profile with the next recorded dose of 4 mg.

FIGURE 1.

CURATE.AI quadratic and linear process with simulated data. Simulated records of patient's prescribed warfarin doses and their corresponding INR measurements were used. CURATE.AI Quadratic (a, b) The patient's initial profile (blue) was calibrated using dose–response data pairs on days 1 to 3. The profile was used to predict the patient's ∆INR and INR response to the warfarin dose given on day 4. Numbers within the circles correspond to the dosing days with the given warfarin dose. (a) Initial calibrated profile (blue) using the dose–response data pairs from days 1 to 3. (b) Assuming no systemic and regimen changes, the profile evolves to include the observed dose–response data pair on day 4. CURATE.AI Linear (c and d): The patient's initial profile (blue) was calibrated using dose response data pairs on days 1 and 2. The profile was used to predict the patient's ∆INR and INR response to the warfarin dose given on day 3. Numbers within the circles correspond to the dosing days with the given warfarin dose. (c) Initial calibrated profile (blue) using the dose–response data pairs from days 1 and 2. (d) With no systemic and regimen changes, the profile evolves to include the observed dose–response data pair on day 3.

With no systemic or regimen changes, the observed dose–response data pair on day 4 was included into the patient's profile, represented by the evolved response equation $\mathit{INR}\left(W\right)=-0.24W^2+0.73W+0.073$ (Figure 1b) with new coefficients [−0.24; 0.73; 0.073] specific to the simulated case at the new timepoint. Predictions for $∆$ INR and INR for day 5 are computed and the process is repeated for the remaining timepoints.

2.2.2. Simulated CURATE.AI linear process

The CURATE.AI Linear process is similar to that of CURATE.AI Quadratic. It requires a minimum of 2, instead of 3, uniquely modulated doses and the corresponding phenotypic output to calibrate a profile. The modulated warfarin doses and $∆$ INR response from days 1 and 2 are mapped via a first order response equation represented by $∆\mathit{INR}\left(W\right)=-W+2.5$ (Figure 1c). From this initial calibrated profile and the next recorded dose of 4 mg, the predictions for $∆$ INR and INR for day 3 are computed.

With no systemic or regimen changes on day 3, the observed dose response data pair on day 3 is included in the profile, yielding a response equation $\mathit{INR}\left(W\right)=-0.62W+1.62$ (Figure 1d). Predictions for $\mathit{∆}$ INR and INR for day 4 are computed and the process repeats for the remaining timepoints.

2.3. Evaluation of CURATE.AI's predictive performance

In order to evaluate CURATE.AI's INR predictive performance, six metrics were identified: three technical performance metrics and three clinically relevant performance metrics (Table 1).

TABLE 1.

Table of metrics.

Group	Metrics	Formula	Reference
Technical performance metrics	Percentage absolute prediction error (PAPE)	$\frac{\mathrm{abs}\left(\text{Predicted}\mathrm{INR}-\text{Observed}\mathrm{INR}\right)}{\text{Observed}\mathrm{INR}}\%$	Vadher et al. 51
	Percentage prediction error (PPE)	$\frac{\text{Predicted}\mathrm{INR}-\text{Observed}\mathrm{INR}}{\text{Observed}\mathrm{INR}}\%$	Asiimwe et al. 37
	PPE 20%	$-20\%<\mathrm{PPE}<20\%$	Xue et al. 35
Clinically relevant performance metrics	INR clinical prediction error (CPE)	$\text{Lower bound}<\text{Predicted}\mathrm{INR}-\text{Observed}\mathrm{INR}<\text{Upper bound}$	Newly devised
	Underprediction bias	$\text{Predicted}\mathrm{INR}-\text{Observed}\mathrm{INR}<0$
	Modeled potential time in therapeutic range (TTR)	$\frac{\mathrm{No}.\text{of prediction events marked}\mathrm{as}\text{success}}{\text{Total}\mathrm{no}.\text{of predictions made}}\%$

2.3.1. Technical performance metrics

Three metrics were selected from literature review to perform cross model comparison for models used for serial INR predictions. ³⁵ , ³⁷ , ⁵¹ At the time of analysis, there were only a limited number of models in the literature that used INR as the predicted response. These were shortlisted and compared against CURATE.AI using metrics reported in their respective analysis.

First, percentage absolute prediction error (PAPE) was selected from Vadher et al.'s evaluation of their population‐based pharmacokinetic/pharmacodynamic (PKPD) model. The model uses an empirical model which describes the relationship between vitamin K dependent clotting factors and INR, and the Bayesian method for parameter estimation. ⁵¹ CURATE.AI's PAPE was compared with Vadher's Bayesian PKPD model ⁴² by identifying patients with sufficient data for at least four predictions to match the number of predictions between the two models. Statistical comparison between the PAPE for CURATE.AI and Vadher's model was not able to be performed because of the limited retrospective data set in this study and the lack of availability of raw data from these models in the literature.

Second, percentage prediction error (PPE) was chosen to benchmark CURATE.AI's performance against the performance of Xue et al.'s dose–response model, Xue et al.'s PKPD model and Hamberg et al.'s PKPD model reported for an independent dataset. The mean/median percentage PE for those models was obtained by summing each category across the five hospitals' data they were externally evaluated against. ² , ³⁵ Of note, all those three models require population‐based estimates of parameters alongside patient specific parameters like genetic makeup, observed warfarin dose and INR response, whereas CURATE.AI only requires 2 to 3 longitudinal dose–response data pairs of the treated patient to generate its first recommendation.

Lastly, INR PPE of 20% (PPE 20%) was selected from Xue et al.'s model evaluation. ³⁵ PPE 20% considers INR prediction error within ±20% as ideal. INR predictions with an error <−20% and >20% are considered as underprediction and overprediction, respectively. The authors did not provide substantiated reasoning behind the choice of 20% as an acceptable threshold but it forms a useful technical comparison metric to compare the predictive performance of CURATE.AI with other models in the literature. CURATE.AI's results were only compared against Xue et al.'s findings with respect to this metric due to the absence of relevant data required to compare against Vadher et al.'s model.

2.3.2. Clinically relevant performance metrics

Three metrics were newly devised to assess the clinical relevance of CURATE.AI (Table 1). A literature review revealed a lack of clinically relevant metrics to evaluate serial INR prediction performance. ³⁴ With the goal of CURATE.AI developed as a CDSS in the prospective management of warfarin dosing, it is essential that clinically relevant metrics are part of the criteria for performance evaluation.

First, INR clinical prediction error (CPE), assesses the percentage of the predictions whose CPE was within prespecified, clinically motivated error threshold. INR therapeutic range for most patients is 2.0–3.0. With the risk of bleeding increases significantly with INR > 5.0 and risk of thromboembolism with INR < 1.5, ¹⁴ , ¹⁵ , ¹⁶ , ⁵² we considered 2.0 and 0.5 error thresholds as clinically relevant and included them in the CURATE.AI performance assessment.

Second, underprediction bias, focuses on negative prediction error. Underprediction bias increases the risk of overdosing warfarin, in turn increasing the patient's risk of bleeding that is not easily reversed.

Lastly, modeled potential time in therapeutic range (TTR) was generated to estimate CURATE.AI's capability to support warfarin dosing in prolonging INR within the therapeutic range. Each prediction instance was labeled as success or non‐success (Figure 2). To be labeled as success—leading to INR within therapeutic range—the dosing instance had to fulfill the following:

1. Both the corresponding predicted INR and the observed INR were in therapeutic range—CURATE.AI correctly predicted the INR to be in target range

2. Predicted INR and the observed INR were both below or both above the therapeutic range but there exist an actionable dose—a dose that CURATE.AI would have potentially recommended that lies within 0 to 14 mg and is not the observed dose, that would achieve target INR.

FIGURE 2.

Decision flowchart of modeled potential TTR. Each prediction instance is first determined if they count as a success instance using an algorithm.

The CURATE.AI modeled potential TTR for a single patient was then calculated by the percentage of success instances over the total number of predictions. This modeled potential TTR was compared against physician‐guided TTR which was calculated by the percentage of prediction events resulting in an INR in the therapeutic range over the total number of INR measurements, beginning from the first CURATE.AI prediction.

2.4. Statistical analysis

Data analysis was performed using scripts written in Python (Version 3.11.0). Normality test was performed with Shapiro–Wilk test at α = 0.05. Statistical comparison were performed with Wilcoxon signed‐rank test at α = 0.05.

3. RESULTS

3.1. Patient population

The study analyzed the clinical records of 127 patients who underwent warfarin therapy for deep venous thrombosis, pulmonary embolism and atrial fibrillation in the year of 2018 in the National University Hospital (NUH). The demographic characteristics of these patients are listed in Table 2.

TABLE 2.

Patient demographics.

Variables	Patients (N = 127)
Sex
Male	76 (59.8%)
Female	51 (40.2%)
Age [years]	58.5 [18–99] a
Ethnicity
Chinese	83
Malay	18
Indian	10
Others	16

^a Mean (range).

Patients with sufficient dose–response data to generate at least one second‐order dose–response profile and were able to generate a minimum of one INR prediction were identified. A total of N = 92 patients and N = 118 patients were identified to have met the requirement for CURATE.AI Quadratic and CURATE.AI Linear analysis, respectively (Figure 3). The target range of INR was assumed 2–3 for all patients.

FIGURE 3.

Internal validation data screening flow. A total of N = 127 de‐identified patient data were retrieved from the Discovery.AI Platform. N = 92 and N = 118 patient data met the requirements of a minimum of 3 and 2 modulated doses and corresponding response readout for CURATE.AI Quadratic and CURATE.AI Linear profile analysis, respectively. N = 9 patient data did not meet the requirement and were excluded from the analysis. N = number of patients.

3.2. Out‐of‐bounds predictions

It is possible for models to generate predictions outside of physiological and clinical quantitation means. Out‐of‐Bounds (OOB) predictions refer to INR predictions that are impossible in the physiological context (negative INR) and values beyond the limit of quantitation (INR > 15). ⁵³ As CURATE.AI is intended as CDSS with physician oversight, the OOB predictions will be actively managed should they occur. 1.15% (n = 9 out of 786 predictions) of INR predictions from CURATE.AI Quadratic and 0.88% (n = 9 out of 1025 predictions) of INR predictions from CURATE.AI Linear model were considered OOB and removed from the subsequent analyses.

3.3. INR distribution

The boxplots of INR values of the dataset, together with the predicted INR values using CURATE.AI Quadratic and CURATE.AI Linear are shown in Figure 4. The distribution of INR values that are within and out of the assumed therapeutic range of INR 2–3 are shown in Table 3.

FIGURE 4.

INR distribution. Raw and predicted INR boxplots for INR measured in patients in standard of care (N = 118, n = 1507) and predicted using CURATE.AI Quadratic (N = 92, n = 777) and CURATE.AI Linear (N = 118, n = 1016). Interquartile ranges are 0.98, 0.95, and 0.94 for standard of care, CURATE.AI Quadratic and CURATE.AI Linear, respectively. The lower bar and upper bar indicate median ± 1.5*IQR. Dashed lines represent the assumed 2–3 therapeutic range. N = number of patients, n = total number of measured events or predicted events.

TABLE 3.

INR distribution (number [%]).

Data/Model	INR < 2	Therapeutic INR	INR > 3
Standard of care (n = 1507)	810 (53.7%)	559 (37.1%)	138 (9.16%)
CURATE.AI Quadratic (n = 777)	340 (43.8%)	335 (43.1%)	102 (13.1%)
CURATE.AI Linear (n = 1016)	451 (44.4%)	443 (43.6%)	122 (12%)

Note: Counts of observed and predicted INR from the dataset and CURATE.AI, respectively. n: number of measured events or predicted events.

3.4. CURATE.AI technical performance metrics

3.4.1. Percentage absolute prediction error

The median Percentage absolute prediction error (PAPE) across all predictions for CURATE.AI Quadratic and CURATE.AI Linear are shown in Table 4.

TABLE 4.

CURATE.AI PAPE.

Model	PAPE
	Median [95% CI]
CURATE.AI quadratic (n = 777)	14.9 [13.7–17.0]
CURATE.AI linear (n = 1016)	14.3 [13.3–15.5]

Note: Values are reported as median [95% CI]. n: total number of prediction events.

To compare CURATE.AI's performance with Vadher's Bayesian PKPD model, patients with sufficient data for at least four predictions were identified for this analysis (CURATE.AI Quadratic: N = 62, CURATE.AI Linear: N = 75).

Overall trend in median PAPE for both CURATE.AI Quadratic and CURATE.AI Linear were similar with Vadher's Bayesian PKPD model—the median PAPE was worse after the first prediction, which subsequently gets better.

However, for Vadher et al.'s model a two‐fold worsening in PAPE was observed after the first prediction. Each day‐to‐day differences in PAPE for CURATE.AI Quadratic and CURATE.AI Linear on the other hand were not statistically significant (Figure 5).

FIGURE 5.

Serial Percentage Absolute Prediction Error (PAPE). Median PAPE for the first four serial INR predictions for CURATE.AI Quadratic (N = 62) CURATE.AI Linear (N = 75) and Vadher et al.'s Bayesian PKPD model (N = 74). The error bar represents 95% confidence interval. Comparison rests on the assumption that the underlying samples validated for each model are representative of a sample of patient population. N = number of patients.

The median PAPE for CURATE.AI Quadratic and Linear were similar (Quadratic: 17.0%, Linear: 13.7%) with Vadher et al.'s model (15.3%) for the fourth prediction when the Vadher et al.'s model was expected to perform the best.

3.4.2. PPE and PPE 20%

The median PPE across all predictions generated by CURATE.AI Quadratic (n = 777) and CURATE.AI Linear (n = 1016) was 1.2 and 1.3, with the 95% confidence interval [−0.6 to 2.6] and [−0.2–2.9], respectively.

The PPE 20% results for CURATE.AI Quadratic and Linear are shown in Table 4 and the distribution box plots are shown in Figure 6. 61.0% (n = 474) and 62.3% (n = 633) of INR predictions generated by CURATE.AI Quadratic and CURATE.AI Linear, respectively, were considered ideal (INR prediction within ±20%). Both CURATE.AI models fared better in their predictive performance compared to Xue et al.'s dose–response model (57.9% ideal INR predictions), Xue et al.'s PKPD model (29.0% ideal INR predictions) and Hamberg et al.'s PKPD model (50.7% ideal INR predictions) (Table 5).

FIGURE 6.

Percentage prediction error (PPE). INR PPE for both CURATE.AI quadratic (n = 777) and linear (n = 1016). Interquartile ranges (IQR) are 31.2 and 29.7 for CURATE.AI quadratic and linear, respectively. The lower bar and upper bar indicate median ± 1.5*IQR. Dashed lines represent the ±20% threshold for ideal predictions. n = total number of prediction events.

TABLE 5.

INR PPE 20% from CURATE.AI and literature (number (percentage)).

Model	Under prediction	Ideal prediction	Over prediction
CURATE.AI Quadratic (n = 777)	122 (15.7%)	474 (61.0%)	181 (23.3%)
CURATE.AI Linear (n = 1016)	156 (15.4%)	633 (62.3%)	227 (22.3%)
Xue et al. Xue et al. Dose–response model 32	532 (20.2%)	1525 (57.9%)	576 (21.9%)
Xue et al. PKPD model 51	1390 (52.8%)	764 (29.0%)	479 (18.2%)
Hamberg et al. PKPD model 2	714 (27.1%)	1336 (50.7%)	583 (22.1%)

Note: Categories (under prediction, ideal prediction and over prediction) are based on a 20% threshold determined from literature. Comparisons between CURATE.AI and Xue et al.'s dose–response model, PKPD model and Hamberg et al.'s PKPD model rests on the assumption that the underlying samples validated for each model are representative of a sample of patient population. n: total number of prediction events.

3.5. CURATE.AI clinically relevant performance metrics

3.5.1. Clinical prediction error

The Clinical prediction error (CPE) results for CURATE.AI are shown in Table 5. For the CPE threshold of 2.0, 97.0% (n = 754) and 96.7% (n = 982) of predictions from CURATE.AI Quadratic and Linear were considered ideal, respectively. For the CPE threshold of 0.5, 66.3% (n = 515) and 68.0% (n = 691) of the predictions were considered ideal (Table 6 and Figure 7a).

TABLE 6.

Clinical prediction error from internal evaluation of CURATE.AI (Number (Percentage)).

Threshold	Model	Under prediction	Ideal prediction	Over prediction
2.0	CURATE.AI Quadratic (n = 777)	8 (1.03%)	754 (97.0%)	15 (1.93%)
	CURATE.AI Linear (n = 1016)	11 (1.08%)	982 (96.7%)	23 (2.26%)
0.5	CURATE.AI Quadratic (n = 777)	114 (14.7%)	515 (66.3%)	148 (19.0%)
	CURATE.AI Linear (n = 1016)	142 (14.0%)	691 (68.0%)	183 (18.0%)

Note: Categories (under prediction, ideal prediction and over prediction) are based on 2.0 and 0.5 unit threshold. n: total number of prediction events.

FIGURE 7.

(a) INR clinical prediction error (CPE). CPE, in terms of INR units for both CURATE.AI quadratic (n = 614) and CURATE.AI linear (n = 774). Interquartile ranges (IQR) are 0.64 and 0.61 for CURATE.AI quadratic and linear, respectively. The lower bar and upper bar indicate median ± 1.5*IQR. Dashed lines represent the ±2.0 and ±0.5 threshold for ideal predictions. (b) Bias (INR units). Bias, expressed as prediction error for CURATE.AI quadratic (n = 614) and CURATE.AI linear (n = 774). The lower bar and upper bar represent the lower and upper range of 95% CI, respectively. (c, d) Modeled potential TTR (%). Modeled potential TTR expressed as percentage, for both (c) CURATE.AI quadratic (N = 91) and (d) CURATE.AI linear (N = 118) against physician‐guided dosing. The error bar represents the lower and upper range of 95% confidence interval, respectively. Normality testing performed using Shapiro–Wilk at 0.05 level of significance. Statistical comparison was performed with Wilcoxon signed rank test at 0.05 level of significance. N = number of patients. n = total number of prediction events.

3.5.2. Underprediction bias

The bias for both CURATE.AI Quadratic and CURATE.AI Linear are shown in Figure 7b. The median bias for both CURATE.AI Quadratic and CURATE.AI Linear were 0.025 and 0.03, respectively, and the upper and lower bound values for the 95% confidence interval were clinically insignificant. This suggests that CURATE.AI predictions have negligible underprediction bias.

3.5.3. Modeled potential time in therapeutic range (TTR)

The bar plots of modeled potential TTR for both CURATE.AI Quadratic and CURATE.AI Linear against physician‐guided dosing are shown in Figure 7c,d. Differences in modeled potential TTR between CURATE.AI and physician‐guided dosing were not statistically significant.

3.6. Comparison Between CURATE.AI Quadratic and CURATE.AI Linear

For additional evaluation for the more appropriate CURATE.AI model, pairwise comparison between CURATE.AI Quadratic and CURATE.AI Linear for the same prediction events (n = 777) across both technical performance and clinical relevance was performed and the results are shown in Figure 8. CURATE.AI Linear's predictions are significantly more accurate compared to CURATE.AI Quadratic (p < 0.001) (Figure 8a). No differences were observed for bias metrics percentage PE and modeled potential TTR (Figure 8b,c).

FIGURE 8.

Bar plots comparing between CURATE.AI quadratic and linear in terms of precision, bias and modeled potential TTR (%). (a) PAPE for both CURATE.AI quadratic and linear (n = 777). (b) PPE for both CURATE.AI quadratic and linear (n = 777). (c) Modeled potential TTR (%) for CURATE.AI Quadratic and Linear (N = 91). The lower bar and upper error bar represents 95% confidence interval. Comparison rests on the assumption that the underlying samples validated for each model are representative of a sample of patient population. No statistically significant difference was detected between the conditions t with Wilcoxon signed rank test at α = 0.05. N = number of patients. n = total number of prediction events.

3.6.1. Five‐fold cross validation

The five‐fold cross validation Median PAPE for both the training set and test sets were similar at 15.19 ± 0.46 and 15.49 ± 1.33, respectively, for CURATE.AI Quadratic; and 14.3 ± 0.16 and 14.62 ± 0.74, respectively, for CURATE.AI Linear. This suggests that factors such as patient demographics, clinical characteristics and genetic factors present in the used data set do not affect the predictive performance of both CURATE.AI Quadratic and CURATE.AI Linear and that the CURATE.AI models may be generalizable to other data sets of shared characteristics.

4. DISCUSSION

CURATE.AI successfully identified personalized dose–response relationships between warfarin and INR in 92 patients using CURATE.AI Quadratic and 118 patients using CURATE.AI Linear. CURATE.AI's robustness and potential application as a CDSS was then demonstrated with six metrics that assessed both technical performance and clinical relevance.

In comparison with existing models developed for INR response prediction, our models achieved predictions with precision considered as ideal (PPE <20%) more frequently as compared to Xue et al.'s dose–response model, ³² Xue et al.'s PKPD model ⁵⁴ and Hamberg et al.'s PKPD model ² (Table 4). CURATE.AI Quadratic and CURATE.AI Linear demonstrated comparable technical performance to Vadher et al.'s model. ⁵¹ CURATE.AI.'s median PAPE was within ±1.7% difference from Vadher et al.'s model by the fourth prediction.

Furthermore, there were fluctuations in median PAPE for Vadher's Bayesian PKPD model, notably after the first prediction. Although predictions improved in subsequent predictions, CURATE.AI's performance was consistent along the treatment timeline (Figure 5). This consistent performance despite common environmental fluctuations from day‐to‐day may be explained by the personalized approach of CURATE.AI that focuses on using the individual patient's data only, and its dynamic ability to evolve the generated profile by incorporating newly observed dose–response data pairs. This aligns with the hypothesis by Vadher et al that attributed their worsening performance over time to model misspecification ⁵¹ which may result from the use of population‐based parameter estimates that may not be applicable to every patient.

In addition, this study highlights the utility of a small data approach in facilitating a more robust implementation of CDSS into a routine clinic use. Models that require more parameters or require training on large datasets suffer a trade‐off between comprehensive modeling and practicality for clinical deployment. ⁵⁵ While a comprehensive modeling approach that accounts for many predictors may demonstrate high predictive performance, it may be challenging for implementation as a clinical solution considering that large amounts of resources are required to obtain information on the predictors from every patient and to set up and/or train the model. Conversely, CURATE.AI achieved potentially superior performance while requiring significantly fewer parameters as algorithm inputs, making it attractive for deployment into a clinical practice.

Besides exploring the conventional second‐order polynomial (CURATE.AI Quadratic) as the backbone of CURATE.AI response predictions this analysis also explored the first‐order variant—CURATE.AI Linear. CURATE.AI Linear had comparable bias and superior precision to CURATE.AI Quadratic, albeit a slightly worse modeled potential TTR. With a potential to translate into reaching the optimal INR range sooner, as it requires one less temporal data pair, CURATE.AI Linear can potentially reduce clinic visits, blood tests and warfarin related emergency room visits without compromising on performance, thereby easing the burden on healthcare resources and improving the quality of life of patients. ²²

4.1. Importance of defining clinically relevant performance metrics for AI model evaluation

In this study, metrics were also newly devised based on both clinical experience and literature review. In the literature, only 7% of developed algorithms were assessed for clinical relevance at least once ³⁷ despite an increasing interest in bridging the gap between healthcare and artificial intelligence solutions for the development of clinically relevant artificial intelligence platforms. ⁵⁶ Given that the goal of CURATE.AI is to be applied as CDSS, it was essential to not just evaluate CURATE.AI's performance on technical aspects but also on clinical relevance.

CURATE.AI showed good performance in terms of all clinically relevant performance metrics. INR CPE demonstrated CURATE.AI being able to generate predictions considered ideal for nearly 97% and 70% of prediction events with the thresholds of 2.0 and 0.5, respectively. This is the first study to use INR CPE as an evaluation metric and future studies and clinical practice are needed to validate the ideal INR prediction error threshold. In the evaluation of the prediction bias, both CURATE.AI Quadratic and CURATE.AI Linear demonstrated negligible underprediction bias, translating to a lower risk of overdosing warfarin and warfarin‐induced bleeding. This metric was chosen because clinical tolerance for warfarin‐induced bleeding is lower than thromboembolism and more emphasis should thus be placed on minimizing warfarin overdose. Bleeding is the most common side effect of warfarin and occurs in up to 41% of patients, ⁷ , ⁵⁷ , ⁵⁸ with the frequency of warfarin‐induced bleeding reported at 15% to 20% per year and severe bleeding episodes ranging between 1% and 3%. ⁵⁹ , ⁶⁰ While reversal agents for warfarin‐induced bleeding are available, there exist no optimal therapy for anticoagulant overdose, especially in emergency settings where time is of essence. ⁶¹ Even minor bleeding can lead to withdrawal of therapy, thus depriving patients of the most effective, and often only, therapy available to prevent thromboembolism and leads to repeated office and sometimes emergency room visits. ⁷ Conversely, risk of thromboembolism with patients on oral anticoagulants is relatively less significant. This is illustrated in a prospective, observational cohort study on 1024 patients receiving long term anticoagulation who experienced episodes of warfarin therapy interruptions reported an associated low risk of thromboembolism when interruptions are short (≤5 days). ⁶²

TTR is the most widely accepted method for determining the quality of warfarin therapy ¹¹ , ¹⁶ , ⁶³ , ⁶⁴ defined as the percentage of time that patients' INR was within target range. Of the three commonly accepted methods for calculating TTR, the Rosendaal method measures the duration that a patient remains in therapeutic range, ⁶⁵ while the traditional method measures the ratio of measured therapeutic events over all measured events. ⁶⁶ Without running a prospective study on CURATE.AI in anticoagulation therapy, a TTR for CURATE.AI cannot be determined. Thus, the modeled potential TTR approach was devised as a proxy measure to provide a preliminary understanding on clinical efficacy using retrospective data. Enhancing time in therapeutic range is a critical determinant to maximizing efficacy and safety of warfarin ⁶³ , ⁶⁷ , ⁶⁸ , ⁶⁹ , ⁷⁰ because one of the major challenges in warfarin therapy is its narrow therapeutic index and the large inter‐individual variability in patient response to the drug, ⁷ leading to fluctuations outside the therapeutic range and adverse events such as bleeding or recurrent thromboembolism. ⁶⁷ , ⁷¹ , ⁷²

Results from the ROCKET AF trial showed that Sweden had the highest mean overall TTR at 75% compared to 64% in Singapore. ⁷³ A possible explanation would be the use of a computer based system with dosing algorithm—Sweden's national quality register for atrial fibrillation and oral anticoagulation (AuriculA) which has been shown to improve TTR compared to manual dosing. ⁷⁴ Similar to CURATE.AI, AuriculA provides a clinical decision tool aiding warfarin dosing using an algorithm requiring the patient's last two INR results and suggest the dosage of warfarin that can be accepted or manually changed. ⁷⁵ The results from this study saw no significant differences in modeled potential TTR between CURATE.AI and physician‐guided dosing. This could mean that CURATE.AI is an algorithm that is on‐par with physician‐guided dosing and yet provides a systematic approach to warfarin dosing which would ease the mental‐burden on guesswork by physicians.

4.2. CURATE.AI in a clinical workflow

CURATE.AI is envisioned as a CDSS that aims to make personalized dosing recommendations to the treating physician using its novel small‐data, personalized approach.

Since warfarin dose titration already involves regular INR testing and dose adjustments with physician oversight, CURATE.AI can be easily integrated into the standard clinical workflow. With self‐monitoring of oral anticoagulation gaining traction, ⁷⁶ there is also a potential for stable patients to utilize CURATE.AI for self‐management of oral anticoagulation with suitable healthcare support as backup (Figure 9). In addition, CURATE.AI has the ability to dynamically evolve with the patient's status as newly measured dose–response data pairs are included into the profile, ensuring up‐to‐date recommendations with the latest patient status.

FIGURE 9.

Proposed CURATE.AI Integration into a Clinical Workflow. Standard of care: Warfarin dose and corresponding INR response for a patient on anticoagulation therapy are charted and dose decision by the clinician is heavily relied on clinical experience. CURATE.AI: CURATE.AI maps the warfarin (inputs) dose and corresponding INR response (outputs) for that individual to calibrate a personalized dose–response profile. From this profile, CURATE.AI will recommend optimal doses to achieve the target INR to the clinician and, as a future possibility, potentially to patients for self‐management of oral anticoagulation.

4.3. Limitations and future directions

The primary objective of this study was to test the potential of applying CURATE.AI in personalized dosing optimisation in anticoagulation therapy with warfarin. The optimisation analysis conducted in this study resulted in favorable prediction outcomes that support further prospective validation. However, this study is not without limitations. The analysis used dose and INR as parameters and INR has it's intrinsic limitations in being correlated with exact plasma warfarin concentrations as it may not detect the factors causing fluctuations in plasma warfarin concentrations immediately but INR is used as it is the most clinically relevant and established means of measuring the dose–response effect of warfarin. ¹⁴ , ¹⁵ , ¹⁶ , ⁶⁴ The analysis broadly assumed an INR target range of 2–3 for all patients, while some patients might have a different INR target range which would have influenced the results of the modeled potential TTR. In addition, because of the retrospective nature of this study, we were not able to select doses of warfarin that are best suited for CURATE.AI dose optimisation subsequently, which would have influenced the dose recommendations coming from CURATE.AI. The sequential INR measurements in the dataset were not regularly spaced as well. Another limitation was that like‐for‐like comparison on technical performance metrics was unable to be performed between Curate.AI and the models selected from the literature because of the limited retrospective data set in this study and the lack of availability of raw data from these models in the literature. This in turn made certain statistical comparisons between CURATE.AI and these models not possible. Also, the newly devised metrics in this study have yet to be independently clinically validated. These new metrics were motivated by the lack of robust clinically relevant metrics in the literature to evaluate serial INR prediction performance. Our metrics were devised based on both clinical experience and literature review, making this study the first‐of‐its‐kind and suggesting these metrics for future application in studies on warfarin dosing algorithms.

CURATE.AI approach in itself is also not without limitations. The requirement for time and technical consistency between successive INR measurements may pose practical challenges. Before broad clinical deployment, CURATE.AI should be prospectively tested in an appropriately sized and controlled trial, with a study design that will not only answer fundamental questions of CURATE.AI technical performance, but also the pragmatic considerations of CURATE.AI's implementation into future patient care, toward improving personalized medicine in warfarin therapy and beyond.

5. CONCLUSIONS

After over 60 years of use, warfarin remains one of the most cost‐effective and widely prescribed medications for the prophylaxis and treatment of thromboembolic conditions. Still, warfarin management requires significant efforts in terms of frequent dose adjustments and can be challenging to use in the clinical setting. Fluctuations in INR can lead to adverse events including recurrent thromboembolism and life‐threatening hemorrhage. This results in significant medical and economic consequences and diminishes patient's quality of life.

CURATE.AI is a small data AI‐derived platform that aims to streamline the dose and drug optimisation in a personalized fashion. This retrospective study demonstrated that CURATE.AI may improve accuracy of warfarin dosing decisions compared to decisions made by unaided clinicians. CURATE.AI performed superiorly in terms of technical performance, illustrating CURATE.AI's robustness and supporting the potential application of CURATE.AI as a CDSS in the prospective management of warfarin dosing. The use of CURATE.AI in the clinical setting may facilitate the continued use of an established and cost‐effective drug like warfarin. This can have a significant impact both on the individual patient and public health.

AUTHOR CONTRIBUTIONS

Tiffany Rui Xuan Gan: Conceptualization; methodology; data curation; investigation; writing – original draft; writing – review and editing; formal analysis. Lester W. J. Tan: Methodology; data curation; writing – original draft; formal analysis; writing – review and editing. Mathias Egermark: Writing – review and editing. Anh T. L. Truong: Methodology; formal analysis. Kirthika Kumar: Formal analysis. Shi‐Bei Tan: Formal analysis. Sarah Tang: Data curation. Agata Blasiak: Conceptualization; methodology; formal analysis; writing – review and editing. Boon Cher Goh: Conceptualization; writing – review and editing. Kee Yuan Ngiam: Conceptualization; writing – review and editing. Dean Ho: Conceptualization; writing – review and editing.

CONFLICT OF INTEREST STATEMENT

Mathias Egermark is an employee of Roche Diagnostics and shareholder in F. Hoffmann‐La Roche Lester W. J. Tan, Anh T. L. Truong, Kirthika Kumar, Shi‐Bei Tan, Agata Blasiak and Dean Ho are co‐inventors of previously filed pending patents on artificial intelligence‐based therapy development. Dean Ho is a co‐founder and shareholder of KYAN Therapeutics, which has licensed intellectual property pertaining to AI‐based oncology drug development. The rest of the authors declare no conflict of interests.

ETHICS STATEMENT

No animal or human subjects were involved in this study. Only retrospective data was collected for this study.

Gan TRX, Tan LWJ, Egermark M, et al. AI‐assisted warfarin dose optimisation with CURATE.AI for clinical impact: Retrospective data analysis. Bioeng Transl Med. 2025;10(3):e10757. doi: 10.1002/btm2.10757

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.