Table 2.
Five-fold cross-validation result comparison.
| Model | Sensitivity, mean (SD) | Specificity, mean (SD) | Accuracy, mean (SD) | Balanced accuracy, mean (SD) | AUROCa, mean (SD) | ||||||
| 921 features (including ICD-10b) | |||||||||||
|
|
AdaBoostc | 0.9713 (0.0060) | 0.9890 (0.0040) | 0.9801 (0.0035) | 0.9801 (0.0035) | 0.9973 (0.0005) | |||||
|
|
XGBoostd | 0.9674 (0.0035) | 0.9897 (0.0034) | 0.9786 (0.0016) | 0.9786 (0.0016) | 0.9968 (0.0004) | |||||
|
|
LightGBMe | 0.9678 (0.0034) | 0.9898 (0.0034) | 0.9788 (0.0015) | 0.9788 (0.0015) | 0.9968 (0.0005) | |||||
|
|
GBMf | 0.7952 (0.0088) | 0.9475 (0.0074) | 0.8713 (0.0059) | 0.8713 (0.0059) | 0.9319 (0.0054) | |||||
|
|
ERTg | 0.8944 (0.0106) | 0.9248 (0.0110) | 0.9095 (0.0063) | 0.9095 (0.0063) | 0.9542 (0.0051) | |||||
|
|
LRh | 0.9514 (0.0052) | 0.9933 (0.0030) | 0.9723 (0.0026) | 0.9723 (0.0026) | 0.9717 (0.0027) | |||||
|
|
RFi | 0.9310 (0.0116) | 0.9667 (0.0086) | 0.9488 (0.0206) | 0.9488 (0.0116) | 0.9872 (0.0034) | |||||
|
|
DNNj | 0.9708 (0.0058) | 0.9847 (0.0048) | 0.9778 (0.0038) | 0.9778 (0.0038) | 0.9944 (0.0012) | |||||
|
|
AdaBoost+XGBoost | 0.9675 (0.0036) | 0.9899 (0.0034) | 0.9787 (0.0016) | 0.9787 (0.0016) | 0.9970 (0.0005) | |||||
|
|
AdaBoost+LightGBM | 0.9681 (0.0034) | 0.9900 (0.0033) | 0.9790 (0.0014) | 0.9790 (0.0014) | 0.9970 (0.0005) | |||||
|
|
XGBoost+LigtGBM | 0.9675 (0.0036) | 0.9899 (0.0034) | 0.9787 (0.0016) | 0.9787 (0.0016) | 0.9968 (0.0004) | |||||
|
|
AdaBoost+XGBoost+LightGBM | 0.9675 (0.0036) | 0.9899 (0.0034) | 0.9787 (0.0016) | 0.9787 (0.0016) | 0.9970 (0.0005) | |||||
| 878 features (ICD-10 only) | |||||||||||
|
|
AdaBoost | 0.8261 (0.0073) | 0.9429 (0.0070) | 0.8845 (0.0053) | 0.8845 (0.0053) | 0.9448 (0.0056) | |||||
|
|
XGBoost | 0.6801 (0.0172) | 0.9722 (0.0065) | 0.8261 (0.0095) | 0.8261 (0.0095) | 0.8929 (0.0051) | |||||
|
|
LightGBM | 0.6877 (0.0140) | 0.9717 (0.0071) | 0.8297 (0.0072) | 0.8297 (0.0072) | 0.8939 (0.0056) | |||||
|
|
GBM | 0.7952 (0.0088) | 0.9475 (0.0074) | 0.8713 (0.0059) | 0.8713 (0.0059) | 0.9319 (0.0054) | |||||
|
|
ERT | 0.8944 (0.0106) | 0.9248 (0.0110) | 0.9096 (0.0063) | 0.9096 (0.0063) | 0.9542 (0.0051) | |||||
|
|
LR | 0.7535 (0.0110) | 0.9540 (0.0060) | 0.8537 (0.0055) | 0.8537 (0.0054) | 0.9401 (0.0066) | |||||
|
|
RF | 0.6615 (0.0424) | 0.9724 (0.0125) | 0.8169 (0.0185) | 0.8169 (0.0185) | 0.9265 (0.0070) | |||||
|
|
DNN | 0.9329 (0.0158) | 0.9788 (0.0126) | 0.9559 (0.0059) | 0.9559 (0.0059) | 0.9867 (0.0023) | |||||
|
|
AdaBoost+XGBoost | 0.6931 (0.0101) | 0.9719 (0.0068) | 0.8325 (0.0060) | 0.8325 (0.0059) | 0.9408 (0.0047) | |||||
|
|
AdaBoost+LightGBM | 0.6960 (0.0124) | 0.9715 (0.0070) | 0.8337 (0.0068) | 0.8337 (0.0068) | 0.9408 (0.0048) | |||||
|
|
XGBoost+LigtGBM | 0.6824 (0.0150) | 0.9719 (0.0068) | 0.8271 (0.0089) | 0.8271 (0.0089) | 0.8939 (0.0055) | |||||
|
|
AdaBoost+XGBoost+LightGBM | 0.6908 (0.0104) | 0.9718 (0.0070) | 0.8312 (0.0063) | 0.8313 (0.0063) | 0.9405 (0.0048) | |||||
| 43 features (excluding ICD-10) | |||||||||||
|
|
AdaBoost | 0.9707 (0.0050) | 0.9854 (0.0062) | 0.9781 (0.0020) | 0.9781 (0.0020) | 0.9965 (0.0007) | |||||
|
|
XGBoost | 0.9658 (0.0040) | 0.9889 (0.0039) | 0.9773 (0.0014) | 0.9773 (0.0014) | 0.9960 (0.0005) | |||||
|
|
LightGBM | 0.9661 (0.0040) | 0.9887 (0.0041) | 0.9774 (0.0013) | 0.9774 (0.0013) | 0.9961 (0.0004) | |||||
|
|
GBM | 0.9729 (0.0036) | 0.9858 (0.0054) | 0.9793 (0.0021) | 0.9793 (0.0021) | 0.9965 (0.0006) | |||||
|
|
ERT | 0.9712 (0.0041) | 0.9828 (0.0052) | 0.9770 (0.0024) | 0.9770 (0.0024) | 0.9937 (0.0011) | |||||
|
|
LR | 0.9448 (0.0053) | 0.9921 (0.0029) | 0.9685 (0.0023) | 0.9685 (0.0023) | 0.9941 (0.0009) | |||||
|
|
RF | 0.9079 (0.0089) | 0.9503 (0.0107) | 0.9291 (0.0061) | 0.9291 (0.0062) | 0.9818 (0.0018) | |||||
|
|
DNN | 0.8805 (0.0482) | 0.8903 (0.0465) | 0.8854 (0.0104) | 0.8854 (0.0104) | 0.9424 (0.0050) | |||||
|
|
AdaBoost+XGBoost | 0.9660 (0.0039) | 0.9888 (0.0040) | 0.9774 (0.0013) | 0.9774 (0.0013) | 0.9962 (0.0005) | |||||
|
|
AdaBoost+LightGBM | 0.9661 (0.0039) | 0.9890 (0.0041) | 0.9775 (0.0012) | 0.9775 (0.0012) | 0.9962 (0.0005) | |||||
|
|
XGBoost+LigtGBM | 0.9659 (0.0039) | 0.9889 (0.0040) | 0.9774 (0.0012) | 0.9774 (0.0012) | 0.9960 (0.0005) | |||||
|
|
AdaBoost+XGBoost+LightGBM | 0.9661 (0.0039) | 0.9891 (0.0041) | 0.9776 (0.0013) | 0.9776 (0.0013) | 0.9961 (0.0005) | |||||
| Traditional methods | |||||||||||
|
|
Inclusive SRRk | 0.9271 | 0.8867 | 0.8867 | 0.9069 | 0.9345 | |||||
|
|
Exclusive SRR | 0.9250 | 0.9100 | 0.9100 | 0.9175 | 0.9554 | |||||
|
|
KTASl | 0.9461 | 0.9778 | 0.9778 | 0.9619 | 0.9372 | |||||
aAUROC: area under the receiver operating characteristic curve.
bICD-10: International Classification of Disease 10th Revision.
cAdaBoost: adaptive boosting.
dXGBoost: extreme gradient boosting.
eLightGBM: light gradient boosting machine.
fGBM: gradient boosting machine.
gERT: extremely random trees.
hLR: logistic regression.
iRF: random forest.
jDNN: deep neural network.
kSRR: survival risk ratio.
lKTAS: Korean Triage and Acuity Scale.