Table 2.

Comparison of validation results of machine learning models.

Models	Accuracy	AUC	Recall	Precision	F1 value
CatBoost	0.89	0.87	0.33	0.78	0.44
RF	0.89	0.88	0.26	0.82	0.38
XGBoost	0.90	0.83	0.41	0.81	0.51
LR	0.89	0.82	0.38	0.63	0.46
KNN	0.88	0.75	0.21	0.61	0.31
Model with oversampling (SMOTEENN)
CatBoost	0.96	0.99	0.98	0.95	0.97
RF	0.95	0.99	0.98	0.94	0.96
XGBoost	0.94	0.98	0.98	0.92	0.95
LR	0.91	0.95	0.92	0.92	0.92
KNN	0.92	0.96	0.98	0.88	0.93
Tradition risk score model
GRACE score	0.84	0.80	0.46	0.59	0.51

AUC and F1 score: the higher, the better. XGBoost: Extreme Gradient Boosting; RF: random forest; LR: logistic regression; KNN: K-nearest neighbors.