Table 2.
Model performance in the internal and prospective validation sets.
| Model | AUROC | Best cutoff | Gray zone | Values in gray zone | Youden index (%) | Sensitivity (%) | Specificity (%) | PPV (%) | NPV (%) |
|---|---|---|---|---|---|---|---|---|---|
| Internal validation | |||||||||
| CatBoost | 0.84 (0.82–0.85) | 0.148 | 0.07–0.24 | 1,276 (39.60%) | 50 | 72 (68–76) | 78 (76–79) | 41 (38–44) | 93 (92–94) |
| LightGBM | 0.83 (0.81–0.85) | 0.147 | 0.06–0.24 | 1,269 (39.39%) | 49 | 70 (66–74) | 79 (77–80) | 41 (38–44) | 93 (92–94) |
| XGBoost | 0.83 (0.81–0.85) | 0.156 | 0.04–0.23 | 1182 (36.69%) | 47 | 64 (60–68) | 84 (82–85) | 45 (42–49) | 92 (91–93) |
| GBDT | 0.82 (0.80–0.84) | 0.144 | 0.08–0.25 | 1380 (42.62%) | 50 | 76 (72–79) | 74 (73–76) | 38 (36–41) | 93 (92–95) |
| Random forest | 0.82 (0.80–0.84) | 0.183 | 0.08–0.29 | 1472 (45.46%) | 49 | 73 (70–77) | 75 (74–77) | 39 (36–42) | 93 (92–94) |
| AdaBoost | 0.80 (0.78–0.82) | 0.493 | 0.49–0.50 | 1046 (32.30%) | 45 | 61 (57–65) | 84 (83–86) | 45 (41–49) | 91 (90–92) |
| MLP | 0.78 (0.76–0.80) | 0.173 | 0.02–0.35 | 1737 (53.64%) | 43 | 63 (59–67) | 80 (79–82) | 40 (37–43) | 91 (90–92) |
| SVM | 0.78 (0.76–0.80) | 0.142 | 0.09–0.16 | 2004 (61.89%) | 46 | 60 (56–64) | 86 (85–87) | 47 (44–51) | 91 (90–92) |
| LR | 0.77 (0.75–0.80) | 0.179 | 0.06–0.25 | 1840 (56.83%) | 44 | 64 (60–68) | 80 (79–81) | 40 (37–43) | 91 (90–92) |
| NaiveBayes | 0.77 (0.75–0.79) | 0.058 | 0.00–0.49 | 2711 (83.72%) | 41 | 65 (62–70) | 75 (74–77) | 36 (33–39) | 91 (90–92) |
| KNN | 0.77 (0.74–0.79) | 0.188 | 0.05–0.21 | 1428 (44.10%) | 40 | 55 (51–59) | 85 (84–86) | 44 (40–47) | 90 (89–91) |
| Prospective validation | |||||||||
| CatBoost | 0.80 (0.74–0.86) | 0.049 | 0.04–0.09 | 198 (39.36%) | 48 | 85 (74–93) | 64 (59–68) | 21 (15–26) | 97 (95–99) |
| LR | 0.77 (0.70–0.84) | 0.834 | 0.37–0.88 | 246 (48.91%) | 38 | 51 (37–65) | 87 (84–90) | 31 (21–42) | 94 (92–96) |
| LightGBM | 0.77 (0.70–0.84) | 0.053 | 0.04–0.10 | 260 (51.69%) | 44 | 81 (69–91) | 63 (59–68) | 20 (15–26) | 97 (95–99) |
| XGBoost | 0.77 (0.71–0.82) | 0.045 | 0.03–0.13 | 217 (43.14%) | 48 | 83 (71–93) | 65 (61–70) | 21 (15–27) | 97 (95–99) |
| SVM | 0.74 (0.67–0.82) | 0.956 | 0.33–0.85 | 254 (50.50%) | 38 | 41 (28–55) | 97 (95–98) | 60 (43–77) | 94 (91–96) |
| NaiveBayes | 0.74 (0.66–0.80) | 0.377 | 0.42–0.87 | 230 (45.73%) | 35 | 96 (90–100) | 39 (34–43) | 15 (12–19) | 99 (97–100) |
| GBDT | 0.72 (0.64–0.79) | 0.495 | 0.34–0.85 | 261 (51.89%) | 30 | 81 (68–91) | 49 (44–54) | 15 (11–19) | 96 (93–98) |
| MLP | 0.71 (0.64–0.78) | 0.781 | 0.37–0.90 | 275 (54.67%) | 31 | 55 (42–69) | 76 (72–80) | 20 (14–27) | 94 (91–96) |
| KNN | 0.71 (0.65–0.78) | 0.63 | 0.42–0.88 | 239 (47.51%) | 33 | 69 (55–81) | 65 (60–69) | 18 (13–24) | 95 (92–97) |
| AdaBoost | 0.70 (0.62–0.78) | 0.992 | 0.34–0.88 | 271 (53.88%) | 30 | 31 (19–44) | 98 (97–100) | 70 (50–88) | 93 (90–95) |
| Random forest | 0.69 (0.62–0.77) | 0.64 | 0.32–0.85 | 278 (55.27%) | 33 | 48 (31–58) | 85 (74–92) | 60 (49–72) | 93 (91–95) |
Models are ordered according to their areas under receiver operating characteristic curves. Youden index was defined as sensitivity + specificity – 1. The bold values indicate the best performance of the 10 models in the internal or prospective validation. XGBOOST, eXtremely Gradient Boosting; GBDT, Gradient Boosting Decision Tree; KNN, K-Nearest Neighbor; SVM, Support Vector Machine; MLP, Multi-Layer Perceptron; LR, Logistic Regression; PPV, Positive Predictive Value; NPV, Negative Predictive Value.