Table 2.
Model performance in the internal and prospective validation sets
| Model | AUC | ACC (%) | Best cutoff | Gray zone | Values in gray zone | Youden index (%) | Sensitivity (%) | Specificity (%) | F1 Score | PPV (%) | NPV (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Internal validation | |||||||||||
| CatBoost | 0.87 (0.82–0.92) | 76 (71–81) | 0.045 | 0.04–0.08 | 64 (22.94%) | 64 | 89 (75–100) | 75 (70–80) | 0.41 (0.29–0.52) | 27 (18–36) | 98 (96–100) |
| Random Forest | 0.85 (0.78–0.92) | 73 (68–78) | 0.077 | 0.08–0.14 | 51 (18.28%) | 64 | 92 (80–100) | 72 (66–77) | 0.39 (0.27–0.50) | 25 (16–34) | 99 (97–100) |
| XGBoost | 0.83 (0.76–0.89) | 71 (66–76) | 0.007 | 0.00–0.04 | 241 (86.38%) | 57 | 89 (76–100) | 69 (64–74) | 0.36 (0.25–0.47) | 23 (14–31) | 98 (96–100) |
| KNN | 0.82 (0.75–0.89) | 75 (70–80) | 0.05 | 0.04–0.09 | 85 (30.47%) | 62 | 89 (74–100) | 74 (68–79) | 0.40 (0.28–0.50) | 25 (17–35) | 98 (96–100) |
| GBDT | 0.82 (0.73–0.90) | 76 (71–81) | 0.03 | 0.01–0.06 | 161 (57.71%) | 52 | 77 (60–92) | 76 (70–81) | 0.37 (0.25–0.48) | 24 (15–34) | 97 (94–99) |
| NaiveBayes | 0.81 (0.72–0.88) | 66 (61–71) | 0.013 | 0.01–0.25 | 97 (34.77%) | 55 | 93 (80–100) | 63 (57–69) | 0.33 (0.24–0.43) | 20 (14–28) | 99 (97–100) |
| LightGBM | 0.80 (0.71–0.87) | 74 (69–79) | 0.004 | 0.00–0.03 | 242 (86.74%) | 54 | 81 (64–95) | 74 (68–79) | 0.37 (0.25–0.48) | 24 (15–33) | 97 (95–99) |
| LR | 0.80 (0.70–0.89) | 73 (67–78) | 0.055 | 0.02–0.12 | 155 (55.56%) | 56 | 85 (69–97) | 72 (66–77) | 0.36 (0.25–0.47) | 23 (15–32) | 98 (96–99) |
| SVM | 0.79 (0.72–0.86) | 63 (57–68) | 0.066 | 0.07–0.12 | 80 (28.67%) | 52 | 93 (80–100) | 60 (54–66) | 0.32 (0.22–0.41) | 19 (12–26) | 99 (97–100) |
| AdaBoost | 0.77 (0.66–0.86) | 85 (81–89) | 0.486 | 0.47–0.49 | 118 (42.29%) | 45 | 58 (38–76) | 88 (83–92) | 0.42 (0.27–0.55) | 33 (20–47) | 95 (92–98) |
| COX | 0.75 (0.64–0.84) | 71 (66–76) | 0.242 | 0.15–0.43 | 135 (48.39%) | 47 | 77 (59–93) | 70 (65–76) | 0.32 (0.22–0.43) | 21 (13–29) | 97 (94–99) |
| MLP | 0.73 (0.62–0.83) | 66 (60–71) | 0.001 | 0.00–0.04 | 241 (86.38%) | 38 | 74 (55–91) | 65 (59–71) | 0.28 (0.18–0.38) | 17 (11–25) | 96 (93–99) |
| Prospective validation | |||||||||||
| CatBoost | 0.85 (0.80–0.89) | 72 (68–77) | 0.062 | 0.06–0.11 | 85 (20.29%) | 62 | 92 (84–98) | 70 (65–74) | 0.44 (0.35–0.52) | 29 (22–36) | 98 (97–100) |
| XGBoost | 0.81 (0.76–0.86) | 67 (63–72) | 0.014 | 0.01–0.15 | 149 (35.56%) | 53 | 88 (78–96) | 64 (60–69) | 0.40 (0.31–0.47) | 26 (19–32) | 98 (95–99) |
| GBDT | 0.81 (0.76–0.85) | 66 (61–70) | 0.047 | 0.04–0.18 | 150 (33.94%) | 51 | 88 (78–96) | 63 (58–68) | 0.36 (0.28–0.44) | 23 (17–29) | 98 (96–99) |
| Random Forest | 0.80 (0.75–0.85) | 74 (70–78) | 0.167 | 0.11–0.23 | 164 (37.10%) | 50 | 76 (64–88) | 74 (70–78) | 0.40 (0.31–0.49) | 27 (20–35) | 96 (94–98) |
| COX | 0.76 (0.70–0.81) | 67 (62–71) | 0.38 | 0.37–0.60 | 151 (34.16%) | 52 | 88 (79–96) | 64 (59–69) | 0.37 (0.30–0.45) | 24 (18–30) | 98 (96–99) |
| LightGBM | 0.74 (0.67–0.80) | 68 (63–72) | 0.013 | 0.00–0.09 | 358 (85.44%) | 34 | 67 (54–80) | 68 (63–72) | 0.33 (0.25–0.41) | 22 (16–29) | 94 (91–97) |
| AdaBoost | 0.72 (0.64–0.79) | 67 (63–72) | 0.483 | 0.47–0.49 | 275 (62.22%) | 41 | 74 (61–86) | 66 (62–71) | 0.34 (0.26–0.42) | 22 (16–29) | 95 (92–98) |
| KNN | 0.70 (0.63–0.77) | 68 (64–72) | 0.039 | 0.01–0.07 | 272 (61.54%) | 31 | 62 (49–75) | 69 (64–73) | 0.30 (0.22–0.38) | 20 (14–27) | 93 (90–96) |
| LR | 0.68 (0.60–0.76) | 79 (75–83) | 0.085 | 0.02–0.11 | 273 (61.76%) | 34 | 52 (37–65) | 82 (78–86) | 0.36 (0.25–0.45) | 27 (18–36) | 93 (90–96) |
| NaiveBayes | 0.67 (0.59–0.74) | 65 (61–70) | 0.021 | 0.00–0.15 | 387 (87.56%) | 28 | 62 (50–76) | 66 (61–70) | 0.29 (0.21–0.36) | 19 (14–25) | 93 (90–96) |
| MLP | 0.66 (0.58–0.74) | 58 (54–63) | 0.005 | 0.00–0.27 | 391 (88.46%) | 32 | 76 (64–88) | 56 (51–61) | 0.29 (0.22–0.36) | 18 (13–23) | 95 (92–98) |
| SVM | 0.65 (0.58–0.72) | 51 (46–55) | 0.055 | 0.04–0.14 | 305 (69.00%) | 32 | 86 (75–95) | 46 (41–51) | 0.28 (0.21–0.35) | 17 (12–22) | 96 (93–99) |
Models are ordered according to the area under the receiver operating characteristic curve. The Youden index was defined as sensitivity + specificity − 1.
XGBOOST, eXtremely Gradient Boosting; GBDT, Gradient Boosting Decision Tree; KNN, K-Nearest Neighbor; SVM, Support Vector Machine; MLP, Multi-Layer Perceptron; LR, Logistic Regression; PPV, positive predictive value; NPV, negative predictive value