Abstract
Background
Conventional bleeding risk scores after percutaneous coronary intervention (PCI) have limited discrimination and external validation, so we developed and validated a machine learning (ML)-based bleeding risk score using real-world data.
Methods and Results
The primary outcome was major bleeding. In the Clinical Deep Data Accumulation System (CLIDAS) cohort, bleeding events occurring >30 days after PCI were identified by chart review and classified as moderate or severe bleeding by the Global Utilization of Streptokinase and TPA for Occluded Coronary Arteries criteria. In the Health, Clinic, and Education Information Evaluation Institute (HCEI) cohort, major bleeding was identified using ICD-10 codes and Diagnosis Procedure Combination (DPC) data. We analyzed 2,502 acute coronary syndrome patients undergoing PCI from CLIDAS and externally validated the model in 10,928 patients from HCEI. A total of 5 ML algorithms were trained. Key predictors were age, body mass index, Btype natriuretic peptide, and hemoglobin. Logistic regression demonstrated the best performance (area under the curve [AUC] 0.748) and was used to derive the CLIDAS bleeding risk score. The CLIDAS score outperformed the binary J-HBR (AUC 0.74 vs. 0.63, P=0.007) and binary ARC-HBR (AUC 0.74 vs. 0.62, P=0.011), but in external validation the CLIDAS score demonstrated limited discrimination (AUC 0.64), lower than continuous conventional scores.
Conclusions
Our novel ML-based bleeding risk score showed better discrimination than conventional binary classifications in the derivation/internal validation, but external performance was limited.
Key Words: Clinical Deep Data Accumulation System (CLIDAS), High bleeding risk, Machine learning, Percutaneous coronary intervention
Central Figure.
Antithrombotic therapy is essential after percutaneous coronary intervention (PCI), but an increased incidence of bleeding complications can worsen prognosis and offset its anti-ischemic benefit.1–4 Accurate stratification of patients at high bleeding risk (HBR), such as by the Academic Research Consortium for High Bleeding Risk (ARC-HBR), is therefore crucial.4–8 However, to address the unique characteristics of the Japanese population, the Japanese Circulation Society established the Japanese High Bleeding Risk (J-HBR) criteria.9–11 Although these criteria exhibit high sensitivity, the criteria classify approximately 70% of patients as HBR, leading to a high rate of false positives,9 which highlights the need for more precise assessment of bleeding risk.
Recent studies have focused on the application of real-world data and machine learning (ML) methods. By utilizing large-scale, multifactorial data derived from electronic health records and applying ML methods, it is possible to develop predictive models that account for non-linear and high-dimensional variables.12,13 A previous study has applied ML to predict clinical outcomes in PCI patients, and have reported improved predictive performance compared to conventional risk scores.14 However, the observed improvements are generally modest, and results across studies have been inconsistent.14,15 Thus, it is important to perform external validation and assess the feasibility of implementing such models in clinical practice.
In this study we aimed to develop a ML-based prediction model for bleeding complications in patients with coronary artery disease using the Clinical Deep Data Accumulation System (CLIDAS) database, a real-world multicenter clinical database in Japan, and to validate it using an external large-scale real-world dataset.
Methods
Data Sources and Study Population
The CLIDAS is a multicenter database established by collecting clinical information from 7 tertiary care hospitals in Japan. It was designed to directly extract data related to a wide range of medical procedures and therapies, including those for patients undergoing PCI, from electronic medical records.16,17 Comprehensive information was gathered, including patient demographics (age, sex), physiological parameters (body weight, body mass index (BMI), blood pressure, pulse rate), laboratory data including hemoglobin (Hb), creatinine, estimated glomerular filtration rate (eGFR), hemoglobin A1c (HbA1c), platelet count, B-type natriuretic peptide (BNP), prescribed medications including antiplatelet agents, anticoagulants, angiotensin-converting enzyme inhibitors, angiotensin II receptor blockers, β-blockers, statins, ezetimibe, proprotein convertase subtilisin/kexin type 9 inhibitors, and proton pump inhibitors, left ventricular ejection fraction measured by echocardiogram, and details of cardiac catheterization and PCI procedures including target lesion and multivessel intervention. All data were systematically extracted from the Standardized Structured Medical Information eXchange Standard Storage and Extended Storage. Additional clinical information was recorded, such as urgency of presentation (e.g., acute coronary syndrome [ACS]), medical history including hypertension, diabetes mellitus (DM), dyslipidemia, chronic kidney disease (CKD), hemodialysis, history of PCI, prior coronary artery bypass grafting, previous myocardial infarction, history of stroke, hospitalization for heart failure (HF), atrial fibrillation, peripheral artery disease, and long-term outcomes. The current study was registered in the UMIN Clinical Trials Registry (UMIN000057338).
The CLIDAS-PCI database includes 9,690 patients who underwent PCI for ACS or chronic coronary syndrome (CCS) between April 2013 and March 2019.16,18–21 After excluding 1,803 patients with incomplete follow-up, 1,425 patients with a post-PCI observation period <30 days, and 3,955 patients with CCS, 2,502 ACS patients were finally analyzed (Figure 1). A total of 64 candidate variables including demographic and clinical characteristics, medical history, vital signs and laboratory values at admission, as well as PCI-related procedural information and prescribed medications, were used in the ML development. The Health, Clinic, and Education Information Evaluation Institute (HCEI) database is a large-scale, electronic health records-derived database constructed from medical records collected across Japan. As of March 2023, it contained structured clinical data from 229 medical institutions, representing approximately 26 million patients. The dataset includes basic demographic information, admission and discharge records, diagnostic codes (International Classification of Diseases, 10th Revision [ICD-10]), prescriptions, laboratory data, and details on medical procedures and surgeries. For external validation, we extracted patients from the HCEI database who underwent stent implantation for ACS and received antiplatelet therapy, using the same inclusion criteria as for the CLIDAS cohort. The data were provided in January 2023. A total of 10,928 patients were identified and used as the external validation cohort (Figure 1).
Figure 1.
Study flowchart for machine learning (ML) model development and validation. Patient selection and ML workflow from the CLIDAS database for bleeding risk prediction model development. The cohort was divided into derivation and internal validation sets, followed by feature selection, model training, and external validation using the HCEI database. CLIDAS, Clinical Deep Data Accumulation System; HCEI, Health, Clinic, and Education Information Evaluation Institute; PCI, percutaneous coronary intervention; SMOTE-NC, Synthetic Minority Over-sampling Technique for Nominal and Continuous.
Outcomes
The primary outcome was major bleeding events. In the CLIDAS database, bleeding events occurring >30 days after PCI were identified based on medical chart review, including date and anatomical site of bleeding, and classified in accordance with the predefined Global Utilization of Streptokinase and TPA for Occluded Coronary Arteries (GUSTO) moderate/severe criteria. In the HCEI database, major bleeding events were identified using ICD-10 codes and Diagnosis Procedure Combination (DPC) data according to a previously reported hospitalization-based coding definition.22 No additional source-document or chart-based validation of the HCEI-defined endpoint was performed in the present study.
Feature Selection and Preprocessing
The CLIDAS dataset was divided into training and internal validation cohorts in a 7 : 3 ratio using stratified sampling based on the outcome, followed by imputation for missing candidate variables. Variables with >20% missing data (n=8) were excluded from the analysis. For the remaining variables, missing values were imputed using the multivariate imputation method, IterativeImputer, which estimates missing values based on regression relationships with other variables. Missing data in the HCEI database were also imputed, using the missForest algorithm, a non-parametric random forest-based imputation method. To identify the most important features for model construction, the Boruta algorithm was applied. Boruta evaluates the importance of each variable by comparing it to shadow features generated from shuffled data and statistically tests whether each feature is significantly more informative than random noise.23 The algorithm was applied to the development cohort to determine the optimal set of features for modeling. Because of the relatively low incidence of bleeding events and concerns regarding class imbalance, oversampling was performed to increase the representation of the minority class. To address the presence of categorical variables in the dataset, the Synthetic Minority Over-sampling Technique for Nominal and Continuous (SMOTE-NC) method was used to generate synthetic minority class samples within the development cohort.24 SMOTE-NC improves class balance by identifying neighboring samples of each minority class instance and creating new synthetic data points through interpolation between them.24
Model Construction and Hyperparameter Tuning
We used 5 classification algorithms (logistic regression, support vector machine, random forest, Gaussian naive Bayes, and eXtreme Gradient Boosting) to develop the prediction model. To reduce overfitting and improve generalizability, hyperparameter tuning was performed using cross-validation. Specifically, RandomizedSearchCV was used to randomly sample combinations from predefined hyperparameter ranges, and model performance was evaluated using k-fold cross-validation. The optimal set of hyperparameters was selected for each algorithm, and final model performance was evaluated on the internal validation set. The final CLIDAS bleeding risk score was derived from the model selected based on overall performance and clinical interpretability.
Model Evaluation and Statistical Analysis
Model discrimination was assessed using the area under the receiver operating characteristic curve (AUC). In addition, performance metrics, including precision, recall, F1 score, specificity, false-positive rate (FPR), and false-negative rate, were calculated to compare model balance. Based on the trained models, a risk scoring system was developed. For the determination of cutoff values, we compared 3 different scoring approaches: (1) mathematically identifying inflection points at which the risk associated with each selected continuous variable increased nonlinearly and dichotomizing the variables at these points; (2) dichotomizing variables at these cutoff points and assigning points according to the odds ratios (ORs) for each variable; and (3) setting cutoff values based on clinical considerations. The scoring method that yielded the highest AUC was selected for the final model. Continuous variables selected by the Boruta algorithm (age, BMI, BNP, and Hb) were binarized using cutoff points derived from spline analysis, which identified inflection points associated with increased risk: age ≥74 years, BNP ≥81.59 pg/mL, BMI <23.2, and Hb <12.38 g/dL. Score points were assigned based on ORs: 3 points each for age ≥74 (OR 3.22) and BNP ≥81.59 (OR 3.01), 2 points for Hb <12.38 (OR 2.41), and 1 point for BMI <23.2 (OR 1.41). Alternative scoring strategies were also explored, including binarization based on clinically meaningful thresholds (e.g., age 80 years, BMI 18.5, Hb 10/12 g/dL, and BNP 100/200 pg/mL). To quantify the performance difference between the CLIDAS bleeding risk score and conventional risk scores (e.g., J-HBR, ARC-HBR), we calculated the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). Each AUC was compared using the DeLong test. All analyses were conducted using Python (v3.11.11) and R (v4.0.5). Packages used included scikit-learn, statsmodels, and the survivalROC package in R.
Results
Development and Internal Validation of the Predictive Model
The baseline characteristics of 2,502 ACS patients from the CLIDAS database are summarized in Table 1. Mean age was 71 years, 78% were male, 86% had hypertension, 38% had DM, 80% had dyslipidemia, and 37% had CKD. Except for the proportion of PCI performed for right coronary artery lesions, there were no significant differences between the training and internal validation cohorts. The incidence of major bleeding complications was 3.5% (87 cases), comprising 30 cases of intracranial hemorrhage, 36 cases of gastrointestinal bleeding, and 21 cases of other types. Using the training dataset, the Boruta algorithm identified 4 features as important predictors: age, BMI, BNP, and Hb (Figure 2A). Comparison of model performance among the 5 ML methods showed that logistic regression (AUC 0.748) and random forest (AUC 0.744) demonstrated the best discriminative abilities (Figure 2B). The recall and specificity of logistic regression were 0.808 and 0.655, respectively, capturing approximately 81% of bleeding cases with a FPR of approximately 35% (Figure 2C). In contrast, random forest yielded a slightly lower recall of 0.577 but a higher specificity of 0.724 (Figure 2C). Based on these findings, we derived the final CLIDAS bleeding risk score from the logistic regression model.
Table 1.
Baseline Characteristics of Patients in the CLIDAS Database
| CLIDAS | P value | |||
|---|---|---|---|---|
| Overall (N=2,502) |
Derivation (N=1,751) |
Validation (N=751) |
||
| Age, years | 71 (62, 78) | 71 (62, 78) | 70 (62, 78) | 0.913 |
| Male, n (%) | 1,945 (78%) | 1,358 (78%) | 587 (78%) | 0.738 |
| Body weight, kg | 64 (56, 71) | 64 (55, 71) | 63 (56, 71) | 0.681 |
| BMI, kg/m2 | 23.9 (21.8, 26.3) | 24.0 (21.9, 26.3) | 23.9 (21.8, 26.3) | 0.751 |
| Family history of CVD, n (%) | 485 (19%) | 334 (19%) | 151 (20%) | 0.55 |
| SBP at admission, mmHg | 132 (116, 149) | 132 (117, 149) | 131 (115, 148) | 0.206 |
| SBP at discharge, mmHg | 115 (103, 126) | 115 (103, 126) | 115 (104, 126) | 0.752 |
| DBP at admission, mmHg | 75 (64, 87) | 75 (64, 87) | 75 (64, 86) | 0.513 |
| DBP at discharge, mmHg | 64 (57, 72) | 64 (57, 72) | 64 (57, 71) | 0.839 |
| PR at admission, beats/min | 75 (65, 86) | 75 (65, 86) | 75 (65, 85) | 0.785 |
| PR at discharge, beats/min | 67 (60, 75) | 67 (60, 75) | 68 (60, 76) | 0.511 |
| Hypertension, n (%) | 2,143 (86%) | 1,502 (86%) | 641 (85%) | 0.78 |
| Diabetes, n (%) | 958 (38%) | 661 (38%) | 297 (40%) | 0.397 |
| Dyslipidemia, n (%) | 2,005 (80%) | 1,411 (81%) | 594 (79%) | 0.393 |
| CKD, n (%) | 919 (37%) | 643 (37%) | 276 (37%) | 0.989 |
| Hemodialysis, n (%) | 118 (4.7%) | 85 (4.9%) | 33 (4.4%) | 0.619 |
| Prior HF hospitalization, n (%) | 112 (4.5%) | 76 (4.3%) | 36 (4.8%) | 0.615 |
| Prior PCI, n (%) | 355 (14%) | 252 (14%) | 103 (14%) | 0.657 |
| Prior CABG, n (%) | 98 (3.9%) | 68 (3.9%) | 30 (4.0%) | 0.895 |
| Prior MI, n (%) | 411 (16%) | 290 (17%) | 121 (16%) | 0.781 |
| Prior stroke, n (%) | 256 (10%) | 180 (10%) | 76 (10%) | 0.904 |
| Atrial fibrillation, n (%) | 84 (3.4%) | 59 (3.4%) | 25 (3.3%) | 0.959 |
| Malignancy, n (%) | 199 (8.0%) | 141 (8.1%) | 58 (7.7%) | 0.921 |
| Peripheral artery disease, n (%) | 96 (3.8%) | 70 (4.0%) | 26 (3.5%) | 0.523 |
| LMT, n (%) | 171 (6.8%) | 122 (7.0%) | 49 (6.5%) | 0.687 |
| LAD, n (%) | 1,717 (69%) | 1,208 (69%) | 509 (68%) | 0.549 |
| LCX, n (%) | 984 (39%) | 707 (40%) | 277 (37%) | 0.101 |
| RCA, n (%) | 1,262 (50%) | 854 (49%) | 408 (54%) | 0.011 |
| Multivessel disease, n (%) | 1,209 (48%) | 844 (48%) | 365 (49%) | 0.854 |
| BNP, pg/mL | 101 (36, 259) | 99 (35, 259) | 105 (37, 256) | 0.508 |
| Hb, g/dL | 12.60 (11.23, 13.77) | 12.63 (11.22, 13.78) | 12.60 (11.25, 13.75) | 0.601 |
| Uric acid, mg/dL | 5.53 (4.65, 6.47) | 5.53 (4.63, 6.47) | 5.53 (4.70, 6.49) | 0.543 |
| Platelet count, ×103/μL | 207 (171, 250) | 207 (171, 250) | 208 (171, 247) | 0.874 |
| Creatinine, mg/dL | 0.87 (0.71, 1.13) | 0.87 (0.71, 1.14) | 0.86 (0.72, 1.11) | 0.881 |
| HbA1c, % | 6.03 (5.70, 6.80) | 6.03 (5.70, 6.80) | 6.05 (5.70, 6.80) | 0.891 |
| eGFR | 66 (51, 77) | 66 (51, 77) | 67 (51, 77) | 0.849 |
| EF, % | 56 (48, 65) | 56 (48, 65) | 56 (49, 65) | 0.162 |
| EF ≥50, n (%) | 1,590 (64%) | 1,106 (63%) | 484 (64%) | 0.541 |
| EF ≥40, <50, n (%) | 359 (14%) | 256 (15%) | 103 (14%) | 0.554 |
| EF <40, n (%) | 246 (9.8%) | 179 (10%) | 67 (8.9%) | 0.316 |
| ACEi, n (%) | 1,386 (55%) | 960 (55%) | 426 (57%) | 0.381 |
| ARB, n (%) | 649 (26%) | 453 (26%) | 196 (26%) | 0.905 |
| ACEi or ARB, n (%) | 1,945 (78%) | 1,351 (77%) | 594 (79%) | 0.285 |
| β-blocker, n (%) | 1,783 (71%) | 1,233 (70%) | 550 (73%) | 0.153 |
| Diuretics, n (%) | 651 (26%) | 448 (26%) | 203 (27%) | 0.45 |
| Statin, n (%) | 2,206 (88%) | 1,550 (89%) | 656 (87%) | 0.406 |
| Ezetimibe, n (%) | 95 (3.8%) | 65 (3.7%) | 30 (4.0%) | 0.735 |
| PCSK9i, n (%) | 1 (<0.1%) | 0 (0%) | 1 (0.1%) | 0.3 |
| OAC, n (%) | 298 (12%) | 209 (12%) | 89 (12%) | 0.952 |
| Antiplatelet therapy, n (%) | 2,475 (99%) | 1,733 (99%) | 742 (99%) | 0.705 |
| DAPT, n (%) | 2,227 (89%) | 1,558 (89%) | 669 (89%) | 0.94 |
| Aspirin, n (%) | 2,340 (94%) | 1,648 (94%) | 692 (92%) | 0.066 |
| P2Y12i, n (%) | 2,292 (92%) | 1,601 (91%) | 691 (92%) | 0.633 |
| Prasugrel, n (%) | 794 (32%) | 562 (32%) | 232 (31%) | 0.553 |
| PPI, n (%) | 2,272 (91%) | 1,591 (91%) | 681 (91%) | 0.884 |
| PCAB, n (%) | 165 (6.6%) | 122 (7.0%) | 43 (5.7%) | 0.251 |
| Bleeding event, n (%) | 87 (3.5%) | 61 (3.5%) | 26 (3.5%) | 0.978 |
Data are presented as median (interquartile range) for continuous variables and as number (percentage) for categorical variables. The overall cohort (N=2,502) was randomly divided into derivation (N=1,751, 70%) and validation (N=751, 30%) cohorts for machine learning model development and internal validation. No significant differences were observed between the 2 cohorts for baseline characteristics (all P>0.05), indicating appropriate randomization for model development and validation. ACEi, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; BMI, body mass index; BNP, B-type natriuretic peptide; CABG, coronary artery bypass grafting; CKD, chronic kidney disease; CLIDAS, Clinical Deep Data Accumulation System; CVD, cardiovascular disease; DAPT, dual antiplatelet therapy; DBP, diastolic blood pressure; eGFR, estimated glomerular filtration rate; EF, ejection fraction; Hb, hemoglobin; HbA1c, glycated hemoglobin; HF, heart failure; LAD, left anterior descending artery; LCX, left circumflex artery; LMT, left main trunk; MI, myocardial infarction; OAC, oral anticoagulant; P2Y12i, P2Y12 receptor inhibitor; PCAB, potassium-competitive acid blocker; PCI, percutaneous coronary intervention; PCSK9i, proprotein convertase subtilisin/kexin type 9 inhibitor; PPI, proton pump inhibitor; PR, pulse rate; RCA, right coronary artery; SBP, systolic blood pressure.
Figure 2.
Variable selection and performance evaluation of machine learning algorithms for bleeding risk prediction. (A) Feature importance of four variables selected by the Boruta algorithm. (B) ROC curves showing discriminative performance of five algorithms. Logistic regression (AUC 0.748) and random forest (AUC 0.744) demonstrated the highest discriminative performance. (C) Performance metrics comparison. AUC, area under the curve; ROC, receiver operating characteristic.
Construction of the Risk Score and Comparison With Existing Risk Scores
Among our 3 different scoring approaches, the scoring method based on clinically determined cutoff values demonstrated the highest discriminative ability (AUC 0.74) and was therefore selected for further comparison with existing risk scores: the J-HBR score and the ARC-HBR score (Table 2). When compared as continuous scores, the CLIDAS risk score showed higher AUCs than the J-HBR score (0.74 vs. 0.70; P=0.38) and the ARC-HBR score (0.74 vs. 0.69; P=0.38), although these differences did not reach statistical significance. In analyses using binary categorizations (J-HBR-positive [≥1 point] and ARC-HBR-positive [≥1 point]), the CLIDAS score showed significantly higher AUCs (0.74 vs. 0.63, P=0.007; and 0.74 vs. 0.62, P=0.011, respectively). Although NRI and IDI values were positive in all comparisons, statistical significance was observed only when comparing the CLIDAS score against binary J-HBR-positive and ARC-HBR-positive definitions (NRI: 0.05 [P=0.60] vs. continuous J-HBR; 0.08 [P=0.44] vs. continuous ARC-HBR; IDI: 0.02 [P=0.034] and 0.02 [P=0.016], respectively) (Table 3). No significant NRI or IDI improvements were detected in direct comparisons with the continuous J-HBR or ARC-HBR scores.
Table 2.
CLIDAS Risk Score Point Weighting Systems and ROC-AUC Performance
| OR (95% CI) | Point | AUC (95% CI) | |
|---|---|---|---|
| A. Binarization using spline curve inflection points as cutoffs | 0.71 (0.60, 0.81) | ||
| Age ≥74 years | – | 1 | |
| BNP ≥81.59 | – | 1 | |
| BMI <23.2 | – | 1 | |
| Hb <12.38 | – | 1 | |
| B. Weighting by ORs using curve inflection points as cutoffs | 0.69 (0.58, 0.80) | ||
| Age ≥74 years | 3.22 (1.90–5.66) | 3 | |
| BNP ≥81.59 | 3.01 (1.67–5.83) | 3 | |
| BMI <23.2 | 1.41 (0.84–2.35) | 1 | |
| Hb <12.38 | 2.41 (1.42–4.19) | 2 | |
| C. Weighting by ORs using clinically-based cutoffs | 0.74 (0.64, 0.83) | ||
| Age <80 years | Ref. | 1 | |
| ≥80 years | 3.88 (2.30–6.51) | 4 | |
| BNP <100 | Ref. | 1 | |
| ≥100, <200 | 2.67 (1.32–5.38) | 2 | |
| ≥200 | 2.72 (1.49–5.11) | 3 | |
| BMI ≥18.5 | Ref. | 1 | |
| <18.5 | 3.66 (1.64–7.40) | 4 | |
| Hb ≥12 | Ref. | 1 | |
| ≥10, <12 | 2.54 (1.43–4.56) | 2 | |
| <10 | 4.07 (1.96–8.08) | 4 |
Of the 3 scoring approaches compared, Model C demonstrated the highest discriminative performance (AUC 0.74, 95% CI: 0.64–0.83). AUC, area under the curve; CI, confidence interval; OR, odds ratio; Ref, reference category. Other abbreviations as in Table 1.
Table 3.
Discriminative Performance Comparison of CLIDAS Risk Score With Conventional Bleeding Risk Scores
| AUC (95% CI) | DeLong, P value | NRI (95% CI), P value | IDI (95% CI), P value | |
|---|---|---|---|---|
| CLIDAS risk score | 0.74 (0.64, 0.83) | Ref. | Ref. | Ref. |
| vs. J-HBR score | 0.70 (0.61, 0.80) | 0.38 | 0.08 (−0.14, 0.29), 0.48 | 0.01 (−0.003, 0.02), 0.14 |
| vs. J-HBR positive | 0.63 (0.56, 0.70) | 0.007 | 0.05 (−0.13, 0.23), 0.60 | 0.02 (0.001, 0.03), 0.034 |
| vs. ARC-HBR score | 0.69 (0.60, 0.79) | 0.38 | 0.04 (−0.17, 0.26), 0.71 | 0.01 (−0.01, 0.02), 0.24 |
| vs. ARC-HBR positive | 0.62 (0.53, 0.70) | 0.011 | 0.08 (−0.12, 0.28), 0.44 | 0.02 (0.003, 0.03), 0.016 |
Predictive performance of the CLIDAS risk score was compared against conventional bleeding risk scores. AUC comparisons were performed using the DeLong test. NRI and IDI assess the incremental predictive value of the CLIDAS score. Binary classifications used established cutoff thresholds: J-HBR positive (≥1 point) and ARC-HBR positive (≥1 point). The CLIDAS score demonstrated superior discrimination compared to binary risk classifications but showed comparable performance to continuous risk scores. ARC-HBR, Academic Research Consortium High Bleeding Risk; AUC, area under the curve; CI, confidence interval; IDI, integrated discrimination improvement; J-HBR, Japanese High Bleeding Risk; NRI, net reclassification improvement; Ref, reference.
External Validation Assessment
The baseline characteristics of the external validation cohort (HCEI database, 10,928 ACS patients) are summarized in Table 4. Compared with the CLIDAS cohort, there were no significant differences in age or sex, but the external cohort had a lower BMI and higher rates of chronic HF. In contrast, histories of stroke, malignancy, and hemodialysis were less frequent. Hb and platelet counts were higher, and BNP levels were lower in the external validation cohort (Table 4). The proportion of J-HBR positivity was 56%, similar to the CLIDAS cohort, but ARC-HBR positivity was lower at 52%, and the incidence of major bleeding events was also lower at 1.5%. The discriminative ability of the CLIDAS risk score in the external validation cohort was lower than in the internal validation, with an AUC of 0.64 (95% CI: 0.60–0.69) (Figure 3). Although the predictive abilities of J-HBR positivity (AUC 0.61) and ARC-HBR positivity (AUC 0.62) were comparable to the CLIDAS risk score, the continuous J-HBR score (AUC 0.67) and ARC-HBR score (AUC 0.68) showed better performance.
Table 4.
Baseline Characteristics Comparison Between CLIDAS and HCEI Databases
| Overall (N=13,430) |
External validation (HCEI) (N=10,928) |
Derivation and internal validation (CLIDAS) (N=2,502) |
P value | |
|---|---|---|---|---|
| Age, years | 70 (61, 78) | 70 (61, 78) | 71 (62, 78) | 0.829 |
| Male, n (%) | 10,340 (77%) | 8,395 (77%) | 1,945 (78%) | 0.326 |
| Body weight, kg | 23.8 (21.7, 26.2) | 23.8 (21.6, 26.2) | 23.9 (21.8, 26.3) | 0.043 |
| Hemodialysis, n (%) | 122 (0.9%) | 4 (<0.1%) | 118 (4.7%) | <0.001 |
| Heart failure, n (%) | 1,576 (12%) | 1,464 (13%) | 112 (4.5%) | <0.001 |
| Prior stroke, n (%) | 727 (5.4%) | 471 (4.3%) | 256 (10%) | <0.001 |
| Malignancy, n (%) | 988 (7.4%) | 789 (7.2%) | 199 (8.0%) | <0.001 |
| Peripheral artery disease, n (%) | 706 (5.3%) | 610 (5.6%) | 96 (3.8%) | <0.001 |
| Hemoglobin, g/dL | 13.50 (12.00, 14.80) | 13.70 (12.20, 15.00) | 12.60 (11.23, 13.77) | <0.001 |
| Platelet count, ×103/μL | 211 (174, 256) | 212 (174, 257) | 207 (171, 250) | <0.001 |
| BNP, pg/mL | 70 (24, 205) | 65 (22, 191) | 101 (36, 259) | <0.001 |
| eGFR | 66 (51, 79) | 65 (51, 80) | 66 (51, 77) | 0.017 |
| OAC, n (%) | 1,731 (13%) | 1,433 (13%) | 298 (12%) | 0.105 |
| CLIDAS bleeding score | 5.00 (4.00, 7.00) | 5.00 (4.00, 7.00) | 5.00 (4.00, 7.00) | <0.001 |
| Age, years | 0.014 | |||
| <80, n (%) | 10,533 (78%) | 8,525 (78%) | 2,008 (80%) | |
| ≥80, n (%) | 2,897 (22%) | 2,403 (22%) | 494 (20%) | |
| BMI, kg/m2 | 0.602 | |||
| ≥18.5, n (%) | 12,720 (95%) | 10,345 (95%) | 2,375 (95%) | |
| <18.5, n (%) | 710 (5.3%) | 583 (5.3%) | 127 (5.1%) | |
| BNP, pg/mL | <0.001 | |||
| <100, n (%) | 7,916 (59%) | 6,667 (61%) | 1,249 (50%) | |
| ≥100, <200, n (%) | 2,100 (16%) | 1,635 (15%) | 465 (19%) | |
| ≥200, n (%) | 3,414 (25%) | 2,626 (24%) | 788 (31%) | |
| Hemoglobin, g/dL | <0.001 | |||
| ≥12, n (%) | 10,072 (75%) | 8,496 (78%) | 1,576 (63%) | |
| ≥10, <12, n (%) | 2,408 (18%) | 1,728 (16%) | 680 (27%) | |
| <10, n (%) | 950 (7.1%) | 704 (6.4%) | 246 (9.8%) | |
| Japan-HBR score | 1.00 (0.00, 2.00) | 1.00 (0.00, 1.50) | 1.00 (0.00, 2.00) | 0.041 |
| ARC-HBR score | 1.00 (0.00, 1.50) | 1.00 (0.00, 1.50) | 1.00 (0.00, 2.00) | <0.001 |
| Japan-HBR-positive, n (%) | 7,529 (56%) | 6,124 (56%) | 1,405 (56%) | 0.916 |
| ARC-HBR-positive, n (%) | 7,066 (53%) | 5,690 (52%) | 1,376 (55%) | 0.008 |
| Bleeding events, n (%) | 251 (1.9%) | 164 (1.5%) | 87 (3.5%) | <0.001 |
Data are presented as median (interquartile range) for continuous variables and as number (percentage) for categorical variables. P values represent comparisons between the two databases. ARC, Academic Research Consortium; HBR, high bleeding risk; HCEI, Health, Clinic, and Education Information Evaluation Institute. Other abbreviations as in Table 1.
Figure 3.
ROC-AUC of risk scores in the external validation cohort. ROC curves comparing the CLIDAS bleeding risk score with conventional risk scores in the external validation cohort (n=10,928 patients from the HCEI database). The CLIDAS score showed limited discrimination in the external validation cohort (AUC 0.64), with lower AUC values than the continuous J-HBR and ARC-HBR scores (AUC 0.67 and 0.68, respectively), although no statistically significant differences were observed by DeLong test. ARC-HBR, Academic Research Consortium for High Bleeding Risk; AUC, area under the curve; CLIDAS, Clinical Deep Data Accumulation System; HCEI, Health, Clinic, and Education Information Evaluation Institute; J-HBR, Japan-High Bleeding Risk; ROC, receiver operating characteristic.
Discussion
In this study we developed a bleeding risk prediction model using ML algorithms based on a real-world, multicenter database. The CLIDAS bleeding risk score showed good discrimination in the derivation/internal validation setting and outperformed the binary J-HBR and ARC-HBR definitions. However, its discrimination decreased in the external validation cohort and did not outperform the continuous J-HBR or ARC-HBR scores. Therefore, the present findings should be interpreted cautiously, and the proposed score should be regarded as an exploratory and pragmatic real-world prediction tool rather than a definitive clinically established model.
Notably, the selection of BNP as a predictor in the risk model was an important and novel finding. BNP reflects the severity of HF and has been suggested to be associated with an increased risk of bleeding and major adverse coronary events.20,25 In the existing HBR criteria, HF is included as a disease category and considered as a bleeding risk factor. However, there is no specific definition of HF, and incorporating a quantitative marker such as BNP, which reflects HF severity, may improve the accuracy of risk stratification. Consistent with this view, a previous study from the same CLIDAS database has shown that higher BNP levels predict subsequent bleeding events and are associated with reduced thrombogenicity.25 Regarding the utility of the ML methods, previous studies have shown that ML models may provide only modest improvements in AUC by a few percentage points over conventional risk scores.14 Similarly, in our study, the difference in AUC was approximately 0.04, and the introduction of ML did not result in a dramatic improvement in predictive performance. Notably, by simplifying the scoring system and incorporating BNP, we improved the interpretability of the model; however, further refinement is needed before routine clinical implementation. The observed decline in external validity may be attributed to differences in patients’ backgrounds and disease severity between the CLIDAS cohort (mainly from university hospitals) and the HCEI cohort (primarily from community hospitals). For example, the patients in the HCEI cohort had lower BNP levels than those in the CLIDAS-PCI cohort (mean BNP: 65 pg/mL in HCEI vs. 101 pg/mL in CLIDAS) which may have limited the applicability of the model. In addition, the incidence of major bleeding complications was 3.5% in the CLIDAS cohort vs. 1.5% in the HCEI cohort, and this discrepancy may have influenced the results. Another possible explanation is the difference in endpoint ascertainment between the 2 datasets. In the CLIDAS cohort, bleeding events occurring >30 days after PCI were identified through chart review and classified according to predefined GUSTO criteria, whereas in the HCEI cohort, major bleeding was identified using ICD-10 and DPC-based definitions according to a hospitalization-based coding algorithm. Therefore, the 2 endpoints were not completely identical in operational definition, and this difference may have contributed to the lower external performance. These factors suggest the possibility of model overfitting or the presence of unmeasured confounders, highlighting the need for additional training and periodic model refinement through multicenter collaborative studies. From a clinical perspective, it is important that the model not only warns of bleeding risk but also contributes to individualized antithrombotic therapy strategies. Although the ARC-HBR and Japan-HBR (J-HBR) scores are suited for broad screening of HBR patients, our model, with its higher specificity compared to existing risk scores, may enable more precise identification of truly high-risk patients, thereby clarifying candidates for short-duration dual antiplatelet therapy (DAPT) or de-escalation strategies.26,27 Recent data in Asian patients with HBR undergoing PCI suggest that abbreviated DAPT followed by singleantiplatelet therapy may be a feasible strategy.28 In patient populations with concomitant high thrombotic and bleeding risks, applying this model for risk stratification may provide a valuable decision-support tool in clinical practice.
Study Limitations
First, it is necessary to consider not only bleeding risk but also thrombotic risk in a comprehensive manner in future risk assessments. For patients classified as HBR who also have a high risk of thrombosis, shortening the duration of DAPT alone may not be sufficient; thus there is a need to develop integrated models that can balance both bleeding and thrombotic risks. Furthermore, the limited external validity of our single ML score (AUC 0.64) suggests that future research should prioritize using clustering methods to identify distinct phenotypic subgroups for truly individualized antithrombotic therapy.29 Second, because this study was based on retrospective observational data, the influence of unmeasured confounders cannot be completely excluded. Third, to ensure external validity, further validation of the model in diverse regions and multiple centers, as well as prospective studies, will be required.
Conclusions
We developed a ML-based bleeding risk score for patients after PCI, using a multicenter real-world database. The score, based on 4 readily available clinical variables, outperformed conventional binary classifications. However, its performance was limited in the external validation cohort and did not exceed that of continuous conventional scores. Therefore, the model should be interpreted as an exploratory real-world prediction tool and requires further refinement and validation before clinical implementation.
Future research directions include continuous retraining of the model using additional external cohort data, development and prospective evaluation of clinical decision-support systems designed for model implementation, and construction of hybrid models that simultaneously consider both bleeding and thrombotic risks. The integration of real-world data and ML-based precise risk prediction is expected to serve as an essential technological foundation for the advancement of personalized antithrombotic therapy.
Disclosures
T.M. received research grants from Amgen and honoraria from Abbott Medical and Bayer. H.M. received research grants from Mistubishi Electric Co., Ltd., and honoraria from Daiichi Sankyo Co., Ltd. T Kabutoya received scholarship funding from Abbott Medical and the Joint Research Fund of Novartis Pharma and Bayer. Y.I. received honoraria from Daiichi Sankyo and Toa Eiyo, Japan. K.K. received research grants and honoraria from Sanwa Kagaku and Kenkyusho. A.K. received honoraria from AstraZeneca, Eli Lilly, and Sumitomo Pharma. Y.N. received research grants and consultations from Bayer. K.T. received remuneration for lectures from Abbott Medical Co., Ltd., Amgen K.K., Bayer Yakuhin, Ltd., Daiichi Sankyo Co., Ltd., Kowa Pharmaceutical Co. Ltd., Nippon Boehringer Ingelheim Co., Ltd., Novartis Pharma K.K., Otsuka Pharmaceutical Co., Ltd., Pfizer Japan Inc., Takeda Pharmaceutical Co., Ltd., and TERUMO Co., Ltd.; trust research/joint research funds from Bayer Yakuhin, Ltd., Bristol-Myers K.K., Daiichi Sankyo Co., Ltd., MOCHIDA PHARMACEUTICAL CO., LTD., EA Pharma Co., Ltd., TAUNS Laboratories, Inc., Novo Nordisk Pharma Ltd., and PRA Health Sciences; received scholarship funds from Abbott Medical Co., Ltd., ITI Co.,Ltd., Boehringer Ingelheim Japan, Otsuka Pharmaceutical Co., Ltd., and Boston Scientific Japan K.K; and was affiliated with endowed departments at Abbott Japan Co., Ltd., Boston Scientific Japan K.K., Fides-one, Inc., GM Medical Co., Ltd., ITI Co., Ltd., Kaneka Medix Co., Ltd., NIPRO CORPORATION, TERUMO Co., Ltd., Philips Japan Ltd., Getinge Group Japan K.K., Orbusneich Medical K. K., Abbott Medical Co., Ltd., BIOTRONIK JAPAN, Inc., Boston Scientific Japan K.K., Fukuda Denshi Co., Ltd., Japan Lifeline Co., Ltd., Medtronic Japan Co., Ltd., Nippon Boehringer Ingelheim Co., Ltd., and NIPRO CORPORATION. H.S. received stock and stock options from Precision Inc. H.F. received consulting fees from Mehergen Group Holdings and honoraria from Novartis Pharma and Otsuka Pharmaceutical. R.N. received joint research funding from Novartis Pharma and Bayer. The other authors declare no conflicts of interest.
IRB Information
This study was approved by the Ethics Committee of Kumamoto University (Protocol Nos. 2406 and 2554). Because anonymized data were used, informed consent was deemed unnecessary.
Author Contributions
All authors contributed to the manuscript. MI performed the statistical analyses. M.I., T.N., K.T., T.M., T. Kohro, T. Kabutoya, N.A., Y.N., Y.O., H.M., S.K., T.I., K.K., Y.I., A.K., Y. Mizuno, K.N., M.N., Y. Miyamoto, H.S., H.F., and R.N. acquired the CLIDAS data. T.T., M.I., S.I., T.N., K.T., H.M. and S.K. conceived and designed the study. T.T. drafted the manuscript. M.I., T.N., and K.T. critically revised the manuscript for key intellectual content. T.M., H.F., and R.N. supervised the study and prepared the manuscript.
Acknowledgments
The authors thank Kowa Company for funding the development of CLIDAS. The authors appreciate the contributions of all CLIDAS research group members. We thank Yuri Matoba (Precision Inc., Tokyo, Japan) for helping us to integrate the data. We used ChatGPT to assist with the English language editing of the manuscript. K.T., T.M., and Y. Miyamoto are members of Circulation Reports’ Editorial Team.
Funding Statement
Funding Source: This work was supported by Kowa Company, Ltd. and a Health Labour Sciences Research Grant (Grant Number: 22FA1016), the Cross-ministerial Strategic Innovation Promotion Program on Integrated Health Care System (Grant Number: JPJ012425), and the Taiju Life Social Welfare Foundation.
Data Availability
The deidentified participant data will not be shared.
References
- 1. Kaikita K, Yasuda S, Akao M, Ako J, Matoba T, Nakamura M, et al.. Bleeding and subsequent cardiovascular events and death in atrial fibrillation with stable coronary artery disease: Insights from the AFIRE trial. Circ Cardiovasc Interv 2021; 14: e010476. [DOI] [PubMed] [Google Scholar]
- 2. Kwok CS, Rao SV, Myint PK, Keavney B, Nolan J, Ludman PF, et al.. Major bleeding after percutaneous coronary intervention and risk of subsequent mortality: A systematic review and meta-analysis. Open Heart 2014; 1: e000021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Piccolo R, Oliva A, Avvedimento M, Franzone A, Windecker S, Valgimigli M, et al.. Mortality after bleeding versus myocardial infarction in coronary artery disease: A systematic review and meta-analysis. EuroIntervention 2021; 17: 550–560. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Galli M, Gragnano F, Berteotti M, Marcucci R, Gargiulo G, Calabro P, et al.. Antithrombotic therapy in high bleeding risk, Part I: Percutaneous cardiac interventions. JACC Cardiovasc Interv 2024; 17: 2197–2215. [DOI] [PubMed] [Google Scholar]
- 5. Ueki Y, Bar S, Losdat S, Otsuka T, Zanchin C, Zanchin T, et al.. Validation of the Academic Research Consortium for High Bleeding Risk (ARC-HBR) criteria in patients undergoing percutaneous coronary intervention and comparison with contemporary bleeding risk scores. EuroIntervention 2020; 16: 371–379. [DOI] [PubMed] [Google Scholar]
- 6. Urban P, Mehran R, Colleran R, Angiolillo DJ, Byrne RA, Capodanno D, et al.. Defining high bleeding risk in patients undergoing percutaneous coronary intervention. Circulation 2019; 140: 240–261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Liu J, He H, Su H, Hou J, Luo Y, Chen Q, et al.. The predictive value of the ARC-HBR criteria for in-hospital bleeding risk following percutaneous coronary intervention in patients with acute coronary syndrome. Int J Cardiol Heart Vasc 2024; 55: 101527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Tsukizawa T, Fujihara M.. Relationship between in-hospital event rates and high bleeding risk score in patients undergoing primary percutaneous coronary intervention for acute myocardial infarction. Cardiovasc Interv Ther 2022; 37: 490–496. [DOI] [PubMed] [Google Scholar]
- 9. Shimizu T, Sakuma Y, Kurosawa Y, Muto Y, Sato A, Abe S, et al.. Validation of Japanese bleeding risk criteria in patients after percutaneous coronary intervention and comparison with contemporary bleeding risk criteria. Circ Rep 2022; 4: 230–238. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Nakanishi N, Kaikita K, Ishii M, Kuyama N, Tabata N, Ito M, et al.. Japanese high bleeding risk criteria status predicts low thrombogenicity and bleeding events in patients undergoing percutaneous coronary intervention. Cardiovasc Interv Ther 2023; 38: 299–308. [DOI] [PubMed] [Google Scholar]
- 11. Tsukada YT, Aoki-Kamiya C, Mizuno A, Nakayama A, Ide T, Aoyama R, et al.. JCS/JCC/JACR/JATS 2024 Guideline on cardiovascular practice with consideration for diversity, equity, and inclusion. Circ J 2025; 89: 658–739. [DOI] [PubMed] [Google Scholar]
- 12. Jee J, Fong C, Pichotta K, Tran TN, Luthra A, Waters M, et al.. Automated real-world data integration improves cancer outcome prediction. Nature 2024; 636: 728–736. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Feuerriegel S, Frauen D, Melnychuk V, Schweisthal J, Hess K, Curth A, et al.. Causal machine learning for predicting treatment outcomes. Nat Med 2024; 30: 958–968. [DOI] [PubMed] [Google Scholar]
- 14. Mortazavi BJ, Bucholz EM, Desai NR, Huang C, Curtis JP, Masoudi FA, et al.. Comparison of machine learning methods with national cardiovascular data registry models for prediction of risk of bleeding after percutaneous coronary intervention. JAMA Netw Open 2019; 2: e196835. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Zaka A, Mutahar D, Gorcilov J, Gupta AK, Kovoor JG, Stretton B, et al.. Machine learning approaches for risk prediction after percutaneous coronary intervention: A systematic review and meta-analysis. Eur Heart J Digit Health 2025; 6: 23–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Oba Y, Kabutoya T, Kohro T, Imai Y, Kario K, Sato H, et al.. Relationships among heart rate, beta-blocker dosage, and prognosis in patients with coronary artery disease in a real-world database using a multimodal data acquisition system. Circ J 2023; 87: 336–344. [DOI] [PubMed] [Google Scholar]
- 17. Matoba T, Kohro T, Fujita H, Nakayama M, Kiyosue A, Miyamoto Y, et al.. Architecture of the Japan Ischemic Heart Disease Multimodal Prospective Data Acquisition for Precision Treatment (J-IMPACT) system. Int Heart J 2019; 60: 264–270. [DOI] [PubMed] [Google Scholar]
- 18. Tokai T, Ishii M, Otsuka Y, Ikebe S, Nakamura T, Tsujita K, et al.. Proton pump inhibitor use, gastrointestinal bleeding reduction, and long-term prognosis after percutaneous coronary intervention. Am J Cardiol 2025; 252: 78–87. [DOI] [PubMed] [Google Scholar]
- 19. Otsuka Y, Ishii M, Ikebe S, Tokai T, Nakamura T, Tsujita K, et al.. Long-term bleeding events post-percutaneous coronary intervention in patients with malignancy with and without anticoagulant therapy. Cardiovasc Interv Ther 2025; 40: 796–806. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Ikebe S, Ishii M, Otsuka Y, Nakamura T, Tsujita K, Matoba T, et al.. Impact of heart failure severity and major bleeding events after percutaneous coronary intervention on subsequent major adverse cardiac events. Int J Cardiol Cardiovasc Risk Prev 2023; 18: 200193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Matoba T, Katsuki S, Nakano Y, Kawahara T, Kimura M, Hino R, et al.. Efficacy and safety of high-intensity statins in japanese patients after percutaneous coronary intervention: Insights from the Clinical Deep Data Accumulation System (CLIDAS((R))). Circ J 2025; 89: 1204–1215. [DOI] [PubMed] [Google Scholar]
- 22. Lamberts M, Staerk L, Olesen JB, Fosbol EL, Hansen ML, Harboe L, et al.. Major bleeding complications and persistence with oral anticoagulation in non-valvular atrial fibrillation: Contemporary findings in real-life Danish patients. J Am Heart Assoc 2017; 6: e004517. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Kursa MB, Rudnicki RW.. Feature selection with the Boruta Package. J Stat Software 2010; 36: 1–13. [Google Scholar]
- 24. Chawla NV BK, Hall LO, Kegelmeyer WP.. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res 2002; 16: 321–357. [Google Scholar]
- 25. Otsuka Y, Ishii M, Ikebe S, Nakamura T, Tsujita K, Kaikita K, et al.. BNP level predicts bleeding event in patients with heart failure after percutaneous coronary intervention. Open Heart 2023; 10: e002489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Kogame N, Nakagawa Y, Kozuma K, Iijima R, Tsutsui A, Murakami Y, et al.. Sex-based differences in the effect of short dual antiplatelet therapy followed by prasugrel monotherapy in high-bleeding-risk patients after percutaneous coronary intervention. Circ Rep 2026; 8: 554–563. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Han JK, Lee SD, Hwang D, Park SH, Kang J, Yang HM, et al.. Renal function-stratified comparison of short- and long-term dual antiplatelet therapy in patients undergoing percutaneous coronary intervention with third-generation drug-eluting stents: Post hoc analysis from the HOST-IDEA randomized clinical trial. Circ J 2025; 89: 281–291. [DOI] [PubMed] [Google Scholar]
- 28. Kang J, Abdul Ghapar AK, Selvaraj K, Hur SH, Tam CC, Jang Y, et al.. One-month dual antiplatelet therapy in high bleeding risk Asian patients undergoing percutaneous coronary intervention: Onyx ONE clear 2-Year results. Circ Rep 2024; 6: 333–340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. De Filippo O, Cammann VL, Pancotti C, Di Vece D, Silverio A, Schweiger V, et al.. Machine learning-based prediction of in-hospital death for patients with takotsubo syndrome: The InterTAK-ML model. Eur J Heart Fail 2023; 25: 2299–2311. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The deidentified participant data will not be shared.