Table 4.
Performance metrics, including confidence intervals for area under the receiver operater characteristics curve (ROC-AUC), for all mortality models using trauma score and injury severity score (TRISS) as baseline, comparing models trained on either danish trauma dataset (DTD), the American trauma quality Improvement program dataset (TQIPD) or a mixed traning dataset consisting of a random forest-selected subset of TQIPD and DTD training data (mixed). Neural network with transfer learning was retrained on DTD. When a resampling method was applied on training data either Synthetic Minority Over-Sampling Technique (SMOTE) or SMOTE with down-sampling removing Tomek links was used (SMOTETomek). When class weighting was applied on a neural network loss function, weights are denoted first by the negative class and secondly for the postive class.
| Model | Training data source | Resampling method | Weights | ROC-AUC | Lower CI ROC-AUC | Upper CI ROC-AUC | F2 score | Precision | Recall |
|---|---|---|---|---|---|---|---|---|---|
| TRISS | DTD | None | None | 0.853 | 0.751 | 0.955 | 0.581 | 0.251 | 0.867 |
| RF | DTD | None | None | 0.971 | 0.922 | 1.0 | 0.746 | 0.654 | 0.773 |
| SMOTE | None | 0.971 | 0.922 | 1.0 | 0.752 | 0.680 | 0.773 | ||
| SMOTETomek | None | 0.966 | 0.913 | 1.0 | 0.783 | 0.667 | 0.818 | ||
| Mixed | None | None | 0.986 | 0.951 | 1.0 | 0.756 | 0.581 | 0.818 | |
| SMOTE | None | 0.973 | 0.926 | 1.0 | 0.721 | 0.696 | 0.727 | ||
| SMOTETomek | None | 0.977 | 0.933 | 1.0 | 0.763 | 0.600 | 0.818 | ||
| TQIPD | None | None | 0.985 | 0.949 | 1.0 | 0.861 | 0.618 | 0.955 | |
| SMOTE | None | 0.967 | 0.915 | 1.0 | 0.734 | 0.382 | 0.955 | ||
| SMOTETomek | None | 0.964 | 0.909 | 1.0 | 0.732 | 0.514 | 0.818 | ||
| AdaBoost | DTD | None | None | 0.969 | 0.918 | 1.0 | 0.642 | 0.667 | 0.636 |
| SMOTE | None | 0.968 | 0.916 | 1.0 | 0.746 | 0.654 | 0.773 | ||
| SMOTETomek | None | 0.963 | 0.908 | 1.0 | 0.826 | 0.704 | 0.864 | ||
| Mixed | None | None | 0.982 | 0.943 | 1.0 | 0.734 | 0.762 | 0.727 | |
| SMOTE | None | 0.977 | 0.933 | 1.0 | 0.746 | 0.654 | 0.773 | ||
| SMOTETomek | None | 0.977 | 0.932 | 1.0 | 0.738 | 0.529 | 0.818 | ||
| TQIPD | None | None | 0.983 | 0.944 | 1.0 | 0.766 | 0.739 | 0.773 | |
| SMOTE | None | 0.970 | 0.921 | 1.0 | 0.692 | 0.429 | 0.818 | ||
| SMOTETomek | None | 0.971 | 0.922 | 1.0 | 0.736 | 0.463 | 0.864 | ||
| XGB | DTD | None | None | 0.937 | 0.866 | 1.0 | 0.642 | 0.667 | 0.636 |
| SMOTE | None | 0.960 | 0.903 | 1.0 | 0.697 | 0.500 | 0.773 | ||
| SMOTETomek | None | 0.952 | 0.889 | 1.0 | 0.759 | 0.708 | 0.773 | ||
| Mixed | None | None | 0.979 | 0.937 | 1.0 | 0.776 | 0.643 | 0.818 | |
| SMOTE | None | 0.985 | 0.950 | 1.0 | 0.812 | 0.655 | 0.864 | ||
| SMOTETomek | None | 0.981 | 0.940 | 1.0 | 0.750 | 0.562 | 0.818 | ||
| TQIPD | None | None | 0.975 | 0.929 | 1.0 | 0.769 | 0.621 | 0.818 | |
| SMOTE | None | 0.978 | 0.934 | 1.0 | 0.784 | 0.457 | 0.955 | ||
| SMOTETomek | None | 0.975 | 0.928 | 1.0 | 0.748 | 0.487 | 0.864 | ||
| EBM | DTD | None | None | 0.951 | 0.887 | 1.0 | 0.733 | 0.607 | 0.773 |
| SMOTE | None | 0.933 | 0.859 | 1.0 | 0.690 | 0.571 | 0.727 | ||
| SMOTETomek | None | 0.952 | 0.890 | 1.0 | 0.750 | 0.562 | 0.818 | ||
| Mixed | None | None | 0.980 | 0.938 | 1.0 | 0.739 | 0.630 | 0.773 | |
| SMOTE | None | 0.984 | 0.946 | 1.0 | 0.800 | 0.541 | 0.909 | ||
| SMOTETomek | None | 0.984 | 0.946 | 1.0 | 0.756 | 0.581 | 0.818 | ||
| TQIPD | None | None | 0.982 | 0.942 | 1.0 | 0.812 | 0.655 | 0.864 | |
| SMOTE | None | 0.964 | 0.910 | 1.0 | 0.664 | 0.425 | 0.773 | ||
| SMOTETomek | None | 0.960 | 0.902 | 1.0 | 0.714 | 0.422 | 0.864 | ||
| Neural network | DTD | None | 0.5439, 6.1970 | 0.974 | 0.928 | 1.0 | 0.745 | 0.396 | 0.955 |
| SMOTE | None | 0.906 | 0.821 | 0.991 | 0.732 | 0.514 | 0.818 | ||
| SMOTETomek | None | 0.867 | 0.769 | 0.964 | 0.678 | 0.533 | 0.727 | ||
| Mixed | None | 0.5503, 5.4705 | 0.986 | 0.951 | 1.0 | 0.821 | 0.478 | 1.0 | |
| SMOTE | None | 0.930 | 0.855 | 1.0 | 0.709 | 0.462 | 0.818 | ||
| SMOTETomek | None | 0.930 | 0.855 | 1.0 | 0.714 | 0.422 | 0.864 | ||
| TQIPD | None | None | 0.974 | 0.928 | 1.0 | 0.775 | 0.488 | 0.909 | |
| SMOTE | None | 0.979 | 0.936 | 1.0 | 0.753 | 0.379 | 1.0 | ||
| SMOTETomek | None | 0.967 | 0.914 | 1.0 | 0.697 | 0.500 | 0.773 | ||
| Neural network (Transfer learning) | Mixed | None | 0.5503, 5.4705 | 0.988 | 0.955 | 1.0 | 0.866 | 0.564 | 1.000 |
| SMOTE | None | 0.963 | 0.908 | 1.0 | 0.720 | 0.432 | 0.864 | ||
| SMOTETomek | None | 0.972 | 0.923 | 1.0 | 0.755 | 0.412 | 0.955 | ||
| TQIPD | None | None | 0.981 | 0.940 | 1.0 | 0.814 | 0.512 | 0.955 | |
| SMOTE | None | 0.976 | 0.931 | 1.0 | 0.787 | 0.513 | 0.909 | ||
| SMOTETomek | None | 0.978 | 0.935 | 1.0 | 0.789 | 0.467 | 0.955 |