Table 4.
Class prediction performance of each machine learning algorithm with imbalance adjustment in the validation cohort.
| Matching ratio and model | BAa (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) | F1 score | PLRb (95% CI) | NLRc (95% CI) | ||||||
| 0.5% (Real world) | ||||||||||||
|
|
MEWSd | 0.75 (0.74-0.76) | 0.72 (0.70-0.73) | 0.78 (0.78-0.79) | 0.096 | 3.31 (3.24-3.38) | 0.36 (0.34-0.38) | |||||
|
|
LRe | 0.76 (0.76-0.77) | 0.75 (0.75-0.78) | 0.76 (0.76-0.76) | 0.093 | 3.21 (3.15-3.27) | 0.31 (0.29-0.33) | |||||
|
|
RNNf | 0.84 (0.84-0.85) | 0.85 (0.84-0.86) | 0.84 (0.83-0.84) | 0.143 | 5.17 (5.09-5.26) | 0.18 (0.17-0.19) | |||||
|
|
RFg | 0.88 (0.88-0.89) | 0.88 (0.87-0.89) | 0.89 (0.88-0.89) | 0.198 | 7.72 (7.61-7.85) | 0.13 (0.12-0.14) | |||||
| 1% | ||||||||||||
|
|
MEWS | 0.74 (0.73-0.74) | 0.72 (0.70-0.73) | 0.76 (0.76-0.76) | 0.148 | 2.97 (2.90-3.03) | 0.37 (0.36-0.39) | |||||
|
|
LR | 0.81 (0.80-0.81) | 0.78 (0.76-0.79) | 0.84 (0.84-0.84) | 0.218 | 4.77 (4.67-4.88) | 0.27 (0.25-0.28) | |||||
|
|
RNN | 0.84 (0.83-0.85) | 0.87 (0.85-0.88) | 0.81 (0.81-0.82) | 0.218 | 4.67 (4.59-4.76) | 0.17 (0.15-0.18) | |||||
|
|
RF | 0.88 (0.87-0.88) | 0.90 (0.89-0.91) | 0.86 (0.86-0.86) | 0.278 | 6.49 (6.38-6.60) | 0.12 (0.11-0.13) | |||||
| 5% | ||||||||||||
|
|
MEWS | 0.72 (0.71-0.73) | 0.72 (0.70-0.73) | 0.72 (0.72-0.73) | 0.348 | 2.57 (2.50-2.63) | 0.39 (0.37-0.41) | |||||
|
|
LR | 0.85 (0.84-0.85) | 0.83 (0.82-0.84) | 0.87 (0.86-0.87) | 0.555 | 6.15 (5.97-6.34) | 0.20 (0.18-0.21) | |||||
|
|
RNN | 0.87 (0.87-0.88) | 0.89 (0.88-0.90) | 0.85 (0.85-0.85) | 0.562 | 5.96 (5.80-6.15) | 0.12 (0.11-0.14) | |||||
|
|
RF | 0.90 (0.90-0.91) | 0.92 (0.91-0.93) | 0.89 (0.88-0.89) | 0.639 | 8.23 (7.97-8.49) | 0.09 (0.08-0.10) | |||||
| 10% | ||||||||||||
|
|
MEWS | 0.70 (0.69-0.71) | 0.72 (0.70-0.73) | 0.69 (0.68-0.69) | 0.419 | 2.29 (2.23-2.35) | 0.41 (0.39-0.43) | |||||
|
|
LR | 0.87 (0.86-0.87) | 0.86 (0.85-0.87) | 0.87 (0.87-0.88) | 0.675 | 6.80 (6.54-7.07) | 0.16 (0.15-0.17) | |||||
|
|
RNN | 0.89 (0.89-0.90) | 0.93 (0.92-0.94) | 0.85 (0.85-0.86) | 0.681 | 6.32 (6.11-6.54) | 0.08 (0.07-0.09) | |||||
|
|
RF | 0.92 (0.92-0.92) | 0.94 (0.94-0.95) | 0.90 (0.89-0.90) | 0.756 | 9.31 (8.95-9.69) | 0.06 (0.06-0.07) | |||||
aBA: balanced accuracy.
bPLR: positive likelihood ratio.
cNLR: negative likelihood ratio.
dMEWS: modified early warning score.
eLR: logistic regression.
fRNN: recurrent neural network.
gRF: random forest.