Table 3.
Difference-in-difference (DiD) metrics for each outcome. Means are based on 1000 bootstrapped iterations with 95% CIs. A positive DiD indicates that the grouped model resulted in a reduced drop in performance compared to the baseline model.
| Outcome, metric | Top baseline algorithm | Top grouped algorithm | Baseline internal validation, mean (95% CI) | Baseline external validation, mean (95% CI) | Grouped internal validation, mean (95% CI) | Grouped external validation, mean (95% CI) | DiD, mean (95% CI) | P value | |
| SSIa | SVMb | LRc |
|
||||||
|
|
AUCd |
|
|
0.906 (0.904-0.908) | 0.763 (0.762-0.764) | 0.904 (0.903-0.906) | 0.833 (0.833-0.834) | 0.072 (0.070-0.074) | <.001 |
|
|
F1-score |
|
|
0.501 (0.499-0.503) | 0.300 (0.299-0.302) | 0.476 (0.474-0.478) | 0.376 (0.375-0.376) | 0.100 (0.097-0.103) | <.001 |
| Pneumonia | LR | SVM |
|
||||||
|
|
AUC |
|
|
0.953 (0.949-0.957) | 0.683 (0.682-0.685) | 0.994 (0.994-0.995) | 0.973 (0.973-0.974) | 0.250 (0.247-0.252) | <.001 |
|
|
F1-score |
|
|
0.504 (0.498-0.509) | 0.302 (0.299-0.305) | 0.456 (0.452-0.461) | 0.467 (0.465-0.468) | 0.212 (0.206-0.218) | <.001 |
| Sepsis | LR | RFe |
|
||||||
|
|
AUC |
|
|
0.964 (0.963-0.964) | 0.890 (0.889-0.891) | 0.948 (0.946-0.949) | 0.883 (0.883-0.884) | 0.008 (0.007-0.010) | <.001 |
|
|
F1-score |
|
|
0.469 (0.467-0.472) | 0.050 (0.050-0.050) | 0.419 (0.416-0.422) | 0.092 (0.092-0.093) | 0.091 (0.089-0.093) | <.001 |
| UTIf | SVM | LR |
|
||||||
|
|
AUC |
|
|
0.898 (0.895-0.900) | 0.886 (0.885-0.887) | 0.936 (0.934-0.939) | 0.929 (0.928-0.930) | 0.006 (0.002-0.009) | .002 |
|
|
F1-score |
|
|
0.153 (0.148-0.158) | 0.063 (0.061-0.064) | 0.244 (0.241-0.246) | 0.225 (0.224-0.226) | 0.073 (0.068-0.077) | <.001 |
aSSI: surgical site infection.
bSVM: support vector machine.
cLR: logistic regression.
dAUC: area under the receiver operating characteristic curve.
eRF: random forest.
fUTI: urinary tract infection.