Table III.
Prediction performance of the models, a) data, b) with downsampling, c) with SMOTE. LR: logistic regression, GAM: generalized additive model, RF: random forests, SVM: support vector machine, NN: neural network, XGB: extreme gradient boosting, NB: Naïve Bayes. AUC: Area under the ROC Curve. CI: Confidence interval.
| a) | ||||
| Model | Accuracy (CI 0.05, 0.95) | AUC (CI 0.05, 0.95) | Sensitivity (CI 0.05, 0.95) | Specificity (CI 0.05, 0.95) |
| LR | 0.77 (0.68,0.83) | 0.86 (0.85,0.87) | 0.78 (0.70,0.88) | 0.77 (0.67,0.83) |
| GAM | 0.81 (0.76,0.83) | 0.86 (0.84,0.88) | 0.75 (0.69,0.83) | 0.81 (0.76,0.83) |
| RF | 0.80 (0.75,0.87) | 0.85 (0.83,0.86) | 0.81 (0.75,0.88) | 0.73 (0.58,0.80) |
| SVM | 0.84 (0.71,0.89) | 0.71 (0.62,0.75) | 0.48 (0.31,0.62) | 0.85 (0.72,0.91) |
| NN | 0.57 (0.48,0.71) | 0.85 (0.83,0.87) | 0.56 (0.47,0.70) | 0.90 (0.81,0.95) |
| XGB | 0.73 (0.49,0.77) | 0.85 (0.84,0.86) | 0.73 (0.47,0.77) | 0.82 (0.76,0.96) |
| NB | 0.75 (0.65,0.77) | 0.79 (0.77,0.82) | 0.71 (0.66,0.81) | 0.75 (0.64,0.77) |
| b) | ||||
| Model | Accuracy (CI 0.05, 0.95) | AUC (CI 0.05, 0.95) | Sensitivity (CI 0.05, 0.95) | Specificity (CI 0.05, 0.95) |
| LR | 0.81 (0.72,0.83) | 0.86 (0.84,0.87) | 0.74 (0.67,0.85) | 0.82 (0.72,0.84) |
| GAM | 0.74 (0.72,0.75) | 0.86 (0.84,0.87) | 0.82 (0.79,0.86) | 0.73 (0.72,0.75) |
| RF | 0.78 (0.76,0.80) | 0.86 (0.84,0.87) | 0.78 (0.75,0.80) | 0.76 (0.70,0.81) |
| SVM | 0.73 (0.69,0.77) | 0.85 (0.83,0.87) | 0.82 (0.76,0.89) | 0.71 (0.69,0.77) |
| NN | 0.86 (0.03,0.92) | 0.85 (0.83,0.87) | 0.87 (0,0.94) | 0.60 (0.39,1) |
| XGB | 0.64 (0.60,0.75) | 0.85 (0.83,0.86) | 0.63 (0.59,0.75) | 0.88 (0.79,0.92) |
| NB | 0.68 (0.66,0.71) | 0.79 (0.77,0.81) | 0.79 (0.74,0.84) | 0.68 (0.65,0.71) |
| c) | ||||
| Model | Accuracy (CI 0.05, 0.95) | AUC (CI 0.05, 0.95) | Sensitivity (CI 0.05, 0.95) | Specificity (CI 0.05, 0.95) |
| LR | 0.75 (0.73, 0.81) | 0.85 (0.83, 0.86) | 0.80 (0.70, 0.85) | 0.75 (0.72, 0.81) |
| GAM | 0.69 (0.62, 0.74) | 0.85 (0.84, 0.87) | 0.86 (0.81, 0.91) | 0.68 (0.61, 0.74) |
| RF | 0.81 (0.68, 0.89) | 0.85 (0.84, 0.87) | 0.81 (0.67, 0.90) | 0.73 (0.56, 0.85) |
| SVM | 0.70 (0.65,0.74) | 0.84 (0.82,0.87) | 0.84 (0.77,0.90) | 0.69 (0.66,0.75) |
| NN | 0.46 (0.03, 0.76) | 0.84 (0.75, 0.86) | 0.97 (0.77, 1) | 0.44 (0, 0.76) |
| XGB | 0.60 (0.10, 0.71) | 0.84 (0.81, 0.85) | 0.59 (0.07, 0.71) | 0.90 (0.79, 1) |
| NB | 0.79 (0.78, 0.90) | 0.80 (0.77, 0.81) | 0.75 (0.68, 0.79) | 0.71 (0.70, 0.76) |