Table 3.
Performance evaluation of prediction models on nonselection and after feature selection.
| Method | Accuracy | Kappa | Sensitivity | Specificity | AUC | |
|---|---|---|---|---|---|---|
| Overall (72 variables) |
LGR | 0.7198 | 0.4427 | 0.6711 | 0.7939 | 0.7926 |
| RF | 0.7077 | 0.3965 | 0.7355 | 0.6655 | 0.7784 | |
| MARS | 0.7104 | 0.4294 | 0.6444 | 0.8108 | 0.7890 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.7225 | 0.4394 | 0.7044 | 0.7500 | 0.7934 | |
| LGR selection (17 variables) |
LGR | 0.6179 | 0.2752 | 0.4888 | 0.8141 | 0.6981 |
| RF | 0.6260 | 0.2829 | 0.5177 | 0.7905 | 0.6912 | |
| MARS | 0.6219 | 0.2771 | 0.5088 | 0.7939 | 0.6917 | |
| CART | 0.5911 | 0.2292 | 0.4533 | 0.8006 | 0.6576 | |
| XGBoost | 0.6246 | 0.2845 | 0.5044 | 0.8074 | 0.6977 | |
| RF selection (11 variables) |
LGR | 0.6876 | 0.3960 | 0.5866 | 0.8412 | 0.7784 |
| RF | 0.6916 | 0.3937 | 0.6244 | 0.7939 | 0.7637 | |
| MARS | 0.6890 | 0.3817 | 0.6444 | 0.7567 | 0.7675 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.6983 | 0.4161 | 0.5977 | 0.8513 | 0.7790 | |
| CART selection (9 variables) |
LGR | 0.7091 | 0.4009 | 0.7311 | 0.6756 | 0.7624 |
| RF | 0.6554 | 0.3464 | 0.5200 | 0.8614 | 0.7557 | |
| MARS | 0.7091 | 0.3954 | 0.7488 | 0.6486 | 0.7653 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.7131 | 0.4062 | 0.7444 | 0.6655 | 0.7652 | |
| MARS selection (7 variables) |
LGR | 0.6876 | 0.3960 | 0.5866 | 0.8412 | 0.7784 |
| RF | 0.6916 | 0.3937 | 0.6244 | 0.7939 | 0.7637 | |
| MARS | 0.6890 | 0.3817 | 0.6444 | 0.7567 | 0.7675 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.6983 | 0.4161 | 0.5977 | 0.8513 | 0.7790 | |
| XGBoost selection (7 variables) |
LGR | 0.7184 | 0.4186 | 0.7444 | 0.6790 | 0.7739 |
| RF | 0.6903 | 0.3800 | 0.6600 | 0.7364 | 0.7453 | |
| MARS | 0.7131 | 0.4096 | 0.7333 | 0.6824 | 0.7683 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.7104 | 0.4212 | 0.6733 | 0.7668 | 0.7763 | |
| XGBoost selection and 3 risk factors (10 variables) |
LGR | 0.6890 | 0.3937 | 0.6044 | 0.8175 | 0.7807 |
| RF | 0.7037 | 0.4008 | 0.6911 | 0.7229 | 0.7727 | |
| MARS | 0.7225 | 0.4233 | 0.7600 | 0.6665 | 0.7831 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.6970 | 0.4069 | 0.6200 | 0.8141 | 0.7845 | |
| MARS selection and 3 risk factors (10 variables) |
LGR | 0.6916 | 0.3964 | 0.6155 | 0.8074 | 0.7780 |
| RF | 0.6836 | 0.3806 | 0.6088 | 0.7972 | 0.7629 | |
| MARS | 0.7024 | 0.3998 | 0.6844 | 0.7297 | 0.7722 | |
| CART | 0.6930 | 0.3360 | 0.8111 | 0.5135 | 0.7031 | |
| XGBoost | 0.7077 | 0.4190 | 0.6600 | 0.7804 | 0.7806 |
Abbreviations: LGR: logistic regression; RF: random forest; CART: classification and regression tree; MARS: multivariate adaptive regression splines; AUC: area under the curve; XGBoost: extreme gradient boosting.