Table 2.
Performance for mortality risk prediction of models in validation cohorts.
| AUC (95% CI) | Accuracy (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | NPV (95% CI) | F1 | Kappa | Brier | |
|---|---|---|---|---|---|---|---|---|---|
| Internal validation cohort (SFV) | |||||||||
| MRPMC | 0.9621 (0.9464–0.9778) | 92.4% (90.1–94.4%) | 57.3% (46.4–67.7%) | 98.3% (96.8–99.2%) | 85.0% (73.4–92.9%) | 93.2% (90.8–95.2%) | 0.685 | 0.644 | 0.051 |
| SVM | 0.9594 (0.9424–0.9764) | 92.4% (90.1–94.4%) | 60.7% (49.8–70.9%) | 97.8% (96.1–98.8%) | 81.8% (70.4–90.2%) | 93.7% (91.4–95.6%) | 0.697 | 0.655 | 0.052 |
| GBDT | 0.9454 (0.9246–0.9662) | 91.5% (89.0–93.6%) | 60.7% (49.8–70.9%) | 96.6% (94.7–98.0%) | 75.0% (63.4–84.5%) | 93.6% (91.3–95.5%) | 0.696 | 0.643 | 0.066 |
| LR | 0.9614 (0.9456–0.9772) | 92.1% (89.7–94.1%) | 56.2% (45.3–66.7%) | 98.1% (96.6–99.1%) | 83.3% (71.5–91.7%) | 93.1% (90.6–95.0%) | 0.671 | 0.628 | 0.051 |
| NN | 0.9615 (0.9456–0.9774) | 92.1% (89.7–94.1%) | 51.7% (40.8–62.4%) | 98.9% (97.6–99.6%) | 88.5% (76.6–95.7%) | 92.5% (90.0–94.5%) | 0.653 | 0.612 | 0.051 |
| External validation cohort (OV) | |||||||||
| MRPMC | 0.9760 (0.9613–0.9906) | 95.5% (93.8–96.8%) | 45.0% (32.1–58.4%) | 99.6% (98.8–99.9%) | 90.0% (73.5–97.9%) | 95.7% (94.0–97.0%) | 0.600 | 0.579 | 0.029 |
| SVM | 0.9774 (0.9640–0.9908) | 95.8% (94.1–97.0%) | 50.0% (36.8–63.2%) | 99.5% (98.6–99.9%) | 88.2% (72.6–96.7%) | 96.1% (94.5–97.4%) | 0.638 | 0.618 | 0.028 |
| GBDT | 0.9536 (0.9279–0.9793) | 94.8% (93.0–96.2%) | 48.3% (35.2–61.6%) | 98.5% (97.4–99.3%) | 72.5% (56.1–85.4%) | 95.9% (94.3–97.2%) | 0.580 | 0.553 | 0.039 |
| LR | 0.9721 (0.9568–0.9875) | 95.4% (93.7–96.7%) | 45.0% (32.1–58.4%) | 99.5% (98.6–99.9%) | 87.1% (70.2–96.4%) | 95.7% (94.0–97.0%) | 0.593 | 0.572 | 0.031 |
| NN | 0.9754 (0.9602–0.9906) | 95.6% (94.0–96.9%) | 46.7% (33.7–60.0%) | 99.6% (98.8–99.9%) | 90.3% (74.3–98.0%) | 95.8% (94.2–97.1%) | 0.615 | 0.595 | 0.028 |
| External validation cohort (CHWH) | |||||||||
| MRPMC | 0.9246 (0.8763–0.9729) | 87.9% (80.6–93.2%) | 42.1% (20.3–66.5%) | 96.9% (91.2–99.4%) | 72.7% (39.0–94.0%) | 89.5% (82.0–94.7%) | 0.533 | 0.470 | 0.083 |
| SVM | 0.9067 (0.8482–0.9652) | 88.8% (81.6–93.9%) | 57.9% (33.5–79.8%) | 94.6% (88.4–98.3%) | 68.8% (41.3–89.0%) | 92.0% (84.8–96.5%) | 0.629 | 0.563 | 0.090 |
| GBDT | 0.9021 (0.8347–0.9694) | 87.9% (80.6–93.2%) | 31.6% (12.6–56.6%) | 99.0% (94.4–100.0%) | 85.7% (42.1–99.6%) | 88.1% (80.5–93.5%) | 0.462 | 0.410 | 0.089 |
| LR | 0.9213 (0.8710–0.9717) | 87.1% (79.6–92.6%) | 36.8% (16.3–61.6%) | 96.9% (91.2–99.4%) | 70.0% (34.8–93.3%) | 88.7% (81.1–94.0%) | 0.483 | 0.417 | 0.091 |
| NN | 0.9202 (0.8700–0.9705) | 88.8% (81.6–93.9%) | 47.4% (24.5–71.1%) | 96.9% (91.2–99.4%) | 75.0% (42.8–94.5%) | 90.4% (83.0–95.3%) | 0.581 | 0.520 | 0.083 |
SFV internal validation cohort of Sino-French New City Campus of Tongji Hospital, OV Optical Valley Campus of Tongji Hospital, CHWH The Central Hospital of Wuhan, MRPMC mortality risk prediction model for COVID-19, SVM support vector machine, GBDT gradient boosted decision tree, LR logistic regression, NN neural network, AUC area under the receiver operating characteristics curve, PPV positive predictive value, NPV negative predictive value, 95% CI 95% confidence interval.