New artificial intelligence prediction model using serial prothrombin time international normalized ratio measurements in atrial fibrillation patients on vitamin K antagonists: GARFIELD-AF

Abstract

Aims

Most clinical risk stratification models are based on measurement at a single time-point rather than serial measurements. Artificial intelligence (AI) is able to predict one-dimensional outcomes from multi-dimensional datasets. Using data from Global Anticoagulant Registry in the Field (GARFIELD)-AF registry, a new AI model was developed for predicting clinical outcomes in atrial fibrillation (AF) patients up to 1 year based on sequential measures of prothrombin time international normalized ratio (PT-INR) within 30 days of enrolment.

Methods and results

Patients with newly diagnosed AF who were treated with vitamin K antagonists (VKAs) and had at least three measurements of PT-INR taken over the first 30 days after prescription were analysed. The AI model was constructed with multilayer neural network including long short-term memory and one-dimensional convolution layers. The neural network was trained using PT-INR measurements within days 0–30 after starting treatment and clinical outcomes over days 31–365 in a derivation cohort (cohorts 1–3; n = 3185). Accuracy of the AI model at predicting major bleed, stroke/systemic embolism (SE), and death was assessed in a validation cohort (cohorts 4–5; n = 1523). The model’s c-statistic for predicting major bleed, stroke/SE, and all-cause death was 0.75, 0.70, and 0.61, respectively.

Conclusions

Using serial PT-INR values collected within 1 month after starting VKA, the new AI model performed better than time in therapeutic range at predicting clinical outcomes occurring up to 12 months thereafter. Serial PT-INR values contain important information that can be analysed by computer to help predict adverse clinical outcomes.

Introduction

In chronic diseases such as atrial fibrillation (AF) risk stratification using prediction models is useful for clinical decision-making. Several models predict clinical events such as stroke and bleeding.^1–3 The CHA₂DS₂-VASc and HAS-BLED scores are widely used to select suitable AF patients for oral anticoagulation (OAC).^4–6 However, some of the variables in these scoring systems are not consistently related to outcomes.⁷ Novel machine learning technology has facilitated the development of more accurate models such as the Global Anticoagulant Registry in the Field (GARFIELD)-AF risk model.⁸ However, these models incorporate data obtained at a single time-point, baseline. Although computers can process multi-dimensional data such as changes of variables over time, few models have used these inputs to predict future clinical events.^9,10

Vitamin K antagonists (VKAs) continue to be prescribed for the prevention of stroke in patients with AF, despite the more recent introduction of non-VKA oral anticoagulants (NOACs).^11,12 The VKAs are the only recommended choice of OAC for AF patients with haemodynamically overt mitral stenosis and mechanical heart valve. Clinicians adjust the dose of VKA based on an individual patient’s prothrombin time international normalized ratio (PT-INR) at each visit. Time in therapeutic range (TTR) is widely used to standardize the effects of VKA therapy over periods beyond 6 months.^13–17 Various bleeding risk scores feature a TTR component to enhance accuracy,¹⁸ and TTR has predictive power for thrombotic and bleeding events.^19,20 However, information on serial changes in PT-INR during early-phase VKA therapy, which may reflect many occult clinical characteristics of patients such as genotype,^21,22 concomitant medications,²³ and lifestyle,²⁴ were not included in these TTR-based models.

Advances in artificial intelligence (AI) using recurrent neural networks allow the identification and translation of multi-dimensional data including time-series data directly into meaningful models.²⁵ Herein, we describe a new AI model for predicting clinical outcomes over 31–365 days after patient enrolment. The model evaluates serially measured PT-INR within the first 30 days of treatment only without other clinical parameters, using data from the largest multinational prospective registry in AF, GARFIELD-AF. The predictive accuracy of the AI model was compared with that of TTR. The working hypothesis was to test whether serially measured PT-INR in early phase can provide information to predict future clinical events.

Methods

Design

The AI model was derived from prospective GARFIELD-AF data gathered in adults with newly diagnosed AF.²⁶ Three independent AI models were developed with the same composite of neural network structure with multi-dimensional patient-level PT-INR values obtained within the first 30 days after starting treatment. The model tabulated the clinical events of major bleed, ischaemic stroke/systemic embolism (SE), and death occurring within days 31–365.

Registry population

The GARFIELD-AF is an ongoing, international, prospective registry of newly diagnosed patients with AF at risk of stroke. The study design, baseline characteristics, and main results have been published.^26–29 Eligible patients were adults aged >18 years who had been diagnosed with non-valvular AF within the previous 6 weeks and had at least one risk factor for stroke as judged by the investigator. Risk factors were not pre-specified in the protocol. Any use of antithrombotic agents was shared decision between clinicians and patients only. Patients with a transient reversible cause of AF and those for whom follow-up was not envisaged were excluded. The present analysis was conducted in patients enrolled in GARFIELD-AF cohorts 1–5 between March 2010 and August 2016. Data were extracted from the study database in November 2017.

Study population

Patients who received anticoagulation therapy with VKA and had three or more PT-INR measurements within the first 30 days after enrolment were included in the model. Patients were excluded if they had experienced any outcome events such as serious bleeding or stroke or died within the first 30 days. In this analysis, day of first visit was set as day 0. Patients from cohorts 1–3 (recruited between March 2010 and October 2014) were included in the derivation cohort whereas those in cohort 4–5 (recruited March 2014 to August 2016) comprised the validation cohort. This study design was considered stringent because each GARFIELD-AF cohort exhibited substantial differences in terms of participating countries, use of anticoagulants, and outcomes.¹¹

Follow-up

Collection of follow-up data occurred at 4 monthly intervals based on medical records and, sometimes, telephone interviews up to 24 months. The incidence of ischaemic stroke, transient ischaemic attack (TIA), SE, acute coronary syndrome, hospitalization, death (cardiovascular and non-cardiovascular), chronic heart failure (CHF; occurrence or worsening), and bleeding (severity and location) was documented. An audit and quality control programme was applied, and data were examined for completeness and accuracy by the co-ordinating centre (TRI, London, UK). By design, 20% of all electronic case report forms in the GARFIELD-AF registry were monitored against source documentation at sites over the 8 years of recruitment and follow-up.

Outcomes

Outcome measures used in this analysis were major bleeding, stroke/SE, and all-cause death occurring between days 31 and 365. Major bleed was classified by investigators according to International Society on Thrombosis and Haemostasis definition.³⁰ Stroke/SE was defined as a combined Endpoint of ischaemic stroke, SE, and TIA.

Artificial intelligence model

The structure of neural networks for the AI model is shown in Figure 1A. To deal with serial data on raw PT-INR measurements, the AI model was constructed by stacking multiple layers of special neurons that can deal with time-dependent data, namely one-dimensional convolution layer and long short-term memory (LSTM) layer. The LSTM layer transfer rectified data to each neighbouring neuron.³¹ This structure allows the layer to learn time-dependent data in sequential order.

Figure 1.

Structure of the neural network and input data for the model. Schematic illustration of the neural network model (A). Vector of 30 prothrombin time international normalized ratio measurements for 4708 patients used as input to the model (B). The element at nth dimension of the vector holds the prothrombin time international normalized ratio value measured on day n. Un-measured data-points were filled with 0. These pairs of prothrombin time international normalized ratio vectors labelled with outcome for each patient were used for model training (C). 1D convolution, one-dimensional convolution; PT-INR, prothrombin time international normalized ratio; LSTM, long short-term memory.

The neural network model was trained independently for each outcome event. For training, PT-INR measurement patterns for each individual patient were converted to a 30 dimensional PT-INR vector as shown in Figure 1B. All PT-INR measurements obtained within the first 30 days were input to the model. The measured PT-INR value was inserted into nth element of the 30 dimensional vector, where n is the number of days after starting VKA. Un-measured data-points were filled with 0. Each vector for patients was labelled with the occurrence of outcome (0 for no event and 1 for event for all three outcome measures) within days 31–365. The neural networks were trained with the multi-dimensional dataset of the PT-INR vector and outcome label as shown in Figure 1C.

Model training

The process of model training is shown in Figure 2. The training was performed using only patient data from the derivation cohort. The derivation cohort was further split into training (70%) and testing (30%) datasets. The training was performed for 500 epochs and each training epoch included a mini-batch of 455 patients randomly selected from the training dataset. Conceptually, the performance of the model is designed to improve by training with longer epochs. However, this approach can also result in overfitting. To avoid this pitfall and select the model with best performance, the model was evaluated using the testing dataset at the end of each epoch. The final model was that which performed best with the testing dataset. The performance was measured by calculating the c-statistics of the prediction model for all the data in testing dataset. No data from validation cohort were used for training.

Figure 2.

Model training process. Schematic illustration of the training process. Datasets are indicated with yellow, processes indicated with blue, and models derived in each epoch indicated with light red. The derivation cohort was further split into training dataset and testing dataset. The training was performed by mini-batch method using 455 as batch size. For each epoch, the derived model was compared with the best model derived with previous epochs using the testing datasets. The training was performed for 500 epochs and the best model using the testing dataset selected.

Model validation

The derived models were validated by inputting the 30 day PT-INR vector and obtaining prediction scores for each outcome. Predicted outcomes were compared with the actual clinical course for each individual patient in the validation cohort. Receiver operating characteristic (ROC) curves were drawn to evaluate the predictive value of the model. The threshold to achieve overall best accuracy for the model was determined and the model’s sensitivity and specificity calculated at that threshold. To test the ability of the model to discriminate between high- and low-risk patients for each event, three sets of Kaplan–Meier plots were drawn for event rates stratified as high and low risk with the threshold.

Ethics

All trial protocols were approved by independent ethics committee and hospital-based institutional review board. The registry was conducted in accordance with the Declaration of Helsinki, local regulatory requirements, and International Conference on Harmonisation-Good Pharmacoepidemiological and Clinical Practice guidelines. All patients provided written informed consent to participate.

Statistical analysis

The neural network was constructed and trained using Keras framework version 2.1.6 (https://keras.io) and TensorFlow version 1.8.0 as backend.³² The neural network was trained using the back-propagation supervised training algorithm. The loss function that was previously reported to reflect the c-statistics³³ was minimized using the RMSprop optimizer.

The c-statistics, the threshold to achieve best accuracy and corresponding accuracy, sensitivity, and specificity of the model at the threshold with its 95% confidence intervals (95% CI) were calculated by bootstrap procedure with 2000 bootstrap rounds using the pROC package of R version 3.5.1.³⁴ Comparison of ROC curve between the AI model and TTR was performed similarly with pROC package. Kaplan–Meier plots were produced using the survival package of R. All P-values were calculated by log-rank test; a P-value <0.05 was considered statistically significant.

Results

Patients

The flowchart of patient selection is shown in Figure 3. Of 14 437 de novo AF patients treated with VKA, 4806 had at least three PT-INR measurements within the first 30 days and were included in the analysis. Ninety-eight patients were excluded (92 with an outcome event within the first 30 days and 6 with missing information). Of the remainder, 3185 were eligible for inclusion in the derivation cohort and 1523 in validation cohort. Baseline characteristics are displayed in Table 1. There was no substantial intergroup difference in terms of patients’ sex, age, body mass index, and left ventricular ejection fraction. Patients with at least three PT-INR measurements during initial 30 days were slightly less likely to have paroxysmal AF and CHF than those with fewer than three INR measurements. No difference in baseline characteristics was noted between derivation and validation cohorts.

Figure 3.

Patient selection. Schematic illustration of patient selection process. PT-INR, prothrombin time international normalized ratio; VKA, vitamin K antagonist.

Table 1.

Patients’ baseline demographics and clinical characteristics

	≥3 PT-INRs (N = 4708)	0–2 PT-INRs (N = 9630)	≥3 PT-INRs subgroup
			Derivation (N = 3185)	Validation (N = 1523)
Sex, n (%)
Female	2085 (44.3)	4330 (45.0)	1420 (44.6)	665 (43.7)
Male	2623 (55.7)	5300 (55.0)	1765 (55.4)	858 (56.3)
Age at dx, years	72.1 (9.9)	70.0 (10.7)	72.2 (9.7)	72.0 (10.2)
BMI, kg/m2	28.7 (5.9)	28.1 (5.7)	28.6 (5.7)	29.0 (6.1)
LVEF, %	53.7 (12.9)	55.7 (12.7)	53.2 (13.2)	54.7 (12.2)
Type of AF, n (%)
New	2409 (51.2)	4087 (42.4)	1706 (53.6)	703 (46.2)
Paroxysmal	798 (16.9)	2207 (22.9)	567 (17.8)	231 (15.2)
Permanent	877 (18.6)	1514 (15.7)	487 (15.3)	390 (25.6)
Persistent	624 (13.3)	1822 (18.9)	425 (13.3)	199 (13.1)
CHF, n (%)	721 (15.3)	2149 (22.3)	466 (14.6)	255 (16.7)
CAD, n (%)	878 (18.6)	1896 (19.7)	511 (16.0)	367 (24.1)
ACS	461 (9.8)	872 (9.1)	292 (9.2)	169 (11.1)
CHA2DS2-VASc	3.4 (1.5)	3.3 (1.5)	34 (1.5)	33 (1.4)
HAS-BLED	1.4 (0.9)	1.4 (0.9)	15 (0.9)	14 (0.9)

Values are mean (SD) unless specified otherwise.

ACS, acute coronary syndromes; AF, atrial fibrillation; BMI, body mass index; CAD, coronary artery disease; CHF, congestive heart failure; LVEF, left ventricular ejection fraction.

Predictive value of artificial intelligence model

The ROC curve compiled for the validation cohort (Figure 4A) revealed that the AI model had a statistically higher predictive value compared with TTR with c-statistics for major bleeding and all-cause death 0.75 and 0.61, respectively (both P = 0.01 vs. TTR). A similar trend albeit nonsignificant was observed for stroke, with a c-statistic 0.70 (P = 0.08 vs. TTR). Forest plots of 95% CI for AUC of ROC curves (Figure 4B) show that the AI model performed better than random; c-statistics for major bleed, stroke, and all-cause death were 0.75 (95% CI, 0.62–0.87), 0.70 (95% CI, 0.56–0.83), and 0.61 (95% CI, 0.54–0.67), respectively, whereas TTR was not significantly different compared with random (for same outcomes: 0.47 [95% CI, 0.32–0.61], 0.47 [95% CI, 0.31–0.64], and 0.48 [95% CI, 0.42–0.54], respectively). Table 2 shows the accuracies, sensitivities, and specificities for the best thresholds derived from the ROC curve for major bleed, stroke, and all-cause death. The model showed good predictive accuracy for major bleeding with a sensitivity 0.79 and specificity 0.78. These results were similar for the training dataset (Supplementary material online, Table 1 and Figure 1A, B).

Figure 4.

ROC analysis of the artificial intelligence model. Comparison of receiver operating characteristic curves compiled from artificial intelligence model and time in therapeutic range (A). Comparisons were performed using stratified bootstrap method with 2000 bootstrap rounds. Forest plots of area under curve of the receiver operating characteristic curve for each outcome (B). The 95% confidence intervals were calculated by stratified bootstrap method with 2000 bootstrap rounds. AUC, area under curve; CI, confidence interval; ROC, receiver operating characteristic.

Table 2.

Best predictive accuracies and corresponding sensitivities and specificities (95% CIs) for validation cohort

AI	Accuracy	Sensitivity	Specificity
Major bleed	0.78 (0.40–0.92)	0.79 (0.50–1.00)	0.78 (0.39–0.93)
Stroke	0.53 (0.24–0.98)	0.85 (0.31–1.00)	0.53 (0.23–0.99)
All-cause death	0.64 (0.51–0.69)	0.63 (0.50–0.76)	0.65 (0.50–0.70)

AI, artificial intelligence.

Survival analysis

Kaplan–Meier plots stratified by risk determined from the AI model are shown in Figure 5. The threshold of prediction score was calculated for each event to achieve the best accuracy according to the ROC curve. The best thresholds for major bleed, stroke/SE, and all-cause death were 0.27, 0.44, and 0.49, respectively. Note that these output values from our model are arbitrary numbers related to risk of future events but not actual probabilities. Patients who had a model output higher than or equal to threshold were classified as high risk and the remainder were low risk. Among 1523 patients in the validation cohort, 354 were classified high risk for major bleeding, 738 for stroke, and 560 for death. High-risk patients had higher cumulative event rates (major bleed, stroke/SE, and all-cause death) compared with low-risk patients.

Figure 5.

Cumulative event rates according to risk stratified by artificial intelligence model. Kaplan–Meier curves stratified with risk according to artificial intelligence model for major bleed (A), stroke/systemic embolism (B), and all-cause death (C). The threshold for risk stratification was made by that which gave the best accuracy in the validation cohort for each outcome according to the receiver operating characteristic curve. The P-values were calculated by log-rank test. AUC, area under curve; ROC, receiver operating characteristic.

The same analysis was performed for the derivation dataset. No threshold calculations were performed for the derivation dataset and the same thresholds obtained from the validation dataset were used for this analysis. The results were similar, supporting the robustness of the threshold (Supplementary material online, Figure 2).

Discussion

We created a new method to convert early time-series measurements of PT-INR to a long-term prediction model. The novel AI model, constructed with neural networks including one-dimensional convolution and LSTM, garnered useful information from the raw PT-INR values, and measurement dates over 30 days after VKA initiation and converted this to predict major bleeding events over the next 11 months. Although TTR is widely used to standardize VKA therapy over the long term,^14–17 its accuracy for predicting future thrombotic/bleeding events is low.³⁵ A previous report showed that GARFIELD-AF patients with 1 year TTR <65% had worse outcomes than those with greater values.³⁶ Within 30 days, TTR has low predictive power because early-phase PT-INR values vary greatly due to a number of influencing factors including genetics,^21,22 choice of commercial thromboplastin and coagulometer device,^37–39 and patients’ lifestyles.⁴⁰ With the use of AI, we show here the presence of important information in raw PT-INR patterns over first 30 days that can predict clinical events occurring from days 31 to 365.

Multiple useful models exist to predict clinical outcomes in patients with AF.^{1–3,8,41,42} However, most use single time-point data. HAS-BLED score, on the other hand, does include time-series data on PT-INR in the guise of labile PT-INR, which is expressed by TTR.³ Our AI model using time-series PT-INR values has better predictive power than TTR for major clinical events, at least in the early phase of VKA initiation. Even with multi-dimensional data including 31 datasets our AI model output is a prediction score given as a single value. Thus, output of the new model may be included in conventional scoring models by introducing a cut-off, similarly to integration of TTR.¹⁸ The ROC analysis in validation cohort revealed that our AI model has modest predictive power with a best c-statistic 0.78, for major bleeding. However, the model could be usefully incorporated into previous models and thereby improve their accuracy, as has been done with TTR.¹⁸ Our prediction model could expand to automatic prediction of clinical outcomes from multi-dimensional data when incorporated into electrical patient recording systems, for example. Since our models are able to predict clinical outcomes in the early phase of treatment, they may discriminate patients who are unsuitable for VKA therapy and suggest switching them to NOACs, which are associated with lower bleeding risk compared with VKA.

Our novel AI model comprising a neural network can efficiently connect multiple time-dependent measurements to clinical outcomes to form a prediction model. Although in this study, the network was used only to learn PT-INR patterns as specific target, the same structure may have the ability to convert other multi-dimensional time-dependent measurements to prediction models. Therefore, the network may provide a new means to incorporate time-dependent data in prediction models.

Output values from our AI model are related to risk of future events but not their probability. Therefore, calibration of the model with typical Hosmer-Lemeshow goodness-of-fit (GOF) test is not feasible.

Study limitations

Several limitations of this analysis should be noted. First, validation of the AI models was performed using datasets derived from the GARFIELD-AF registry. External validations of the AI model were not conducted. Thus, validity of this model beyond GARFIELD-AF patients is unknown. On the other hand, large dissimilarities between cohorts 1–3 and 4 and 5 were noted, suggesting that our model is sufficiently robust to apply in daily clinical practice. Prothrombin time international normalized ratio within 30 days may be influenced by concomitant dosing with parenteral anticoagulants. However, our model attempted to account for all influencing factors beyond the effects of VKA. We hope that other researchers will test our model’s performance in external datasets.

Second, the AI model was trained only with PT-INR data and did not include other information such as sex, age, biomarkers, concomitant drugs, or other serially measured values. Although consecutive patient data were analysed, unrecognized confounders may exist. Many other known risk factors for adverse outcome events were not considered in our models.

Third, by selecting only patients with >3 PT-INR measurements within 30 days, two-thirds of the entire cohort were excluded, which could introduce selection bias. Furthermore, patients do not necessarily remain stable after day 31 and our model cannot capture changes at time-points later than day 31. Future studies will examine the impact of time periods beyond 30 days in relation to AI risk prediction.

Fourth, although our results suggest the presence of crucial information within the PT-INR measurement pattern to predict patients’ clinical course, the nature of that information is unknown. It might be present in the target PT-INR value, PT-INR fluctuations, PT-INR measurement frequency, or elsewhere.

Fifth, the c-statistics, sensitivity, and specificity of our models are far from perfect. Further studies to improve predictive accuracy possibly by adding other clinical characteristics and measurements are necessary.

Sixth, statistical significance was not achieved in either the derivation or validation cohort in comparison with TTR for prediction of all-cause death and stroke. This could be explained by low numbers of events limiting statistical power. Moreover, even though the number of deaths observed was not low, they could have been caused by factors not related to anticoagulation. Validation of the model in larger cohorts with higher numbers of events may demonstrate better predictive power for death and stroke. Despite these limitations, our results suggest that serially measured PT-INR values within 30 days contain information enabling us to predict serious bleeding outcomes up to 1 year. Trained AI may thus be able to detect individuals at high risk for major adverse cardiac events early in the treatment course.

Conclusions

In AF patients treated with VKA, we developed new AI models to predict all-cause death, stroke, and major bleeding events occurring between months 2 and 12. The models’ predictive accuracy was greatest for major bleeding, followed by all-cause mortality and stroke/SE. Our results imply that AI can capture important information to predict future outcomes from early-phase PT-INR measurements.