Table 3.
Overall predictive performance for each machine learning algorithm with imbalance adjustment in the development and validation cohorts.
| Matching ratio and model |
Development cohort | Validation cohort | |||||||||||||
| AUROCa
(95% CI) |
AUPRCb
(95% CI) |
ICIc | Calibration slope (95% CI) |
AUROC (95% CI) |
AUPRC (95% CI) |
ICI | Calibration slope (95% CI) |
||||||||
| 0.5% (Real world) | |||||||||||||||
|
|
MEWSd | 0.77 (0.77-0.77) | 0.09 (0.08-0.09) | 0.013 | 3.69 (3.64-3.74) | 0.80 (0.80-0.81) | 0.11 (0.10-0.12) | 0.016 | 4.19 (4.09-4.29) | ||||||
|
|
LRe | 0.82 (0.82-0.83) | 0.08 (0.08-0.09) | 0.003 | 1.09 (1.08-1.10) | 0.82 (0.81-0.83) | 0.09 (0.09-0.10) | 0.004 | 1.12 (1.09-1.15) | ||||||
|
|
RNNf | 0.96 (0.96-0.97) | 0.47 (0.46-0.48) | 0.002 | 1.13 (1.12-1.15) | 0.91 (0.90-0.91) | 0.17 (0.16-0.18) | 0.006 | 0.70 (0.69-0.72) | ||||||
|
|
RFg | 1.00 (1.00-1.00) | 1.00 (1.00-1.00) | 0.007 | 6.71 (6.18-7.24) | 0.94 (0.94-0.95) | 0.37 (0.35-0.39) | 0.003 | 1.09 (1.06-1.13) | ||||||
| 1% | |||||||||||||||
|
|
MEWS | 0.76 (0.76-0.77) | 0.12 (0.12-0.12) | 0.022 | 3.46 (3.41-3.51) | 0.79 (0.79-0.80) | 0.16 (0.15-0.17) | 0.025 | 4.09 (3.97-4.20) | ||||||
|
|
LR | 0.88 (0.88-0.89) | 0.26 (0.25-0.26) | 0.008 | 1.07 (1.06-1.09) | 0.88 (0.87-0.88) | 0.28 (0.27-0.30) | 0.007 | 1.09 (1.06-1.12) | ||||||
|
|
RNN | 0.96 (0.96-0.96) | 0.52 (0.51-0.53) | 0.003 | 1.03 (1.01-1.04) | 0.91 (0.91-0.92) | 0.33 (0.32-0.35) | 0.010 | 0.79 (0.78-0.81) | ||||||
|
|
RF | 1.00 (1.00-1.00) | 1.00 (1.00-1.00) | 0.010 | 7.51 (6.86-8.15) | 0.94 (0.93-0.94) | 0.47 (0.45-0.49) | 0.003 | 1.14 (1.11-1.18) | ||||||
| 5% | |||||||||||||||
|
|
MEWS | 0.73 (0.72-0.73) | 0.25 (0.25-0.26) | 0.052 | 3.08 (3.01-3.14) | 0.77 (0.77-0.78) | 0.35 (0.34-0.37) | 0.066 | 4.22 (4.08-4.37) | ||||||
|
|
LR | 0.91 (0.90-0.91) | 0.59 (0.58-0.60) | 0.034 | 1.00 (0.99-1.02) | 0.90 (0.89-0.90) | 0.61 (0.60-0.63) | 0.028 | 1.02 (0.99-1.04) | ||||||
|
|
RNN | 0.97 (0.97-0.97) | 0.79 (0.79-0.80) | 0.003 | 1.04 (1.02-1.06) | 0.94 (0.93-0.94) | 0.68 (0.66-0.69) | 0.015 | 0.82 (0.80-0.84) | ||||||
|
|
RF | 1.00 (1.00-1.00) | 1.00 (1.00-1.00) | 0.025 | 9.88 (8.54-11.22) | 0.96 (0.96-0.96) | 0.78 (0.76-0.79) | 0.012 | 1.21 (1.17-1.25) | ||||||
| 10% | |||||||||||||||
|
|
MEWS | 0.70 (0.70-0.71) | 0.29 (0.29-0.30) | 0.018 | 1.14 (1.11-1.17) | 0.76 (0.75-0.77) | 0.42 (0.41-0.44) | 0.043 | 1.68 (1.62-1.75) | ||||||
|
|
LR | 0.93 (0.92-0.93) | 0.71 (0.70-0.71) | 0.043 | 1.00 (0.99-1.01) | 0.92 (0.91-0.92) | 0.72 (0.71-0.74) | 0.039 | 0.98 (0.95-1.01) | ||||||
|
|
RNN | 0.98 (0.97-0.98) | 0.87 (0.87-0.88) | 0.002 | 1.02 (1.00-1.04) | 0.95 (0.95-0.96) | 0.82 (0.81-0.83) | 0.015 | 0.81 (0.79-0.84) | ||||||
|
|
RF | 1.00 (1.00-1.00) | 1.00 (1.00-1.00) | 0.025 | 10.19 (8.73-11.65) | 0.97 (0.97-0.97) | 0.86 (0.84-0.87) | 0.014 | 1.14 (1.09-1.18) | ||||||
aAUROC: area under the receiver operating characteristic curve.
bAUPRC: area under the precision recall curve.
cICI: integrated calibration index.
dMEWS: modified early warning score.
eLR: logistic regression.
fRNN: recurrent neural network.
gRF: random forest.