Table 3.
Comparison between the ensemble models and three immunologists in each sub-category
| Method | Category | Accuracy (%) | Precision | Recall | F1–score | Kappa | Time(s) | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| w/o | w | w/o | w | w/o | w | w/o | w | w/o | w | - | ||
| Imm-1 | (-) | 84.4 | 88.7 ↑ | 0.965 | 0.989 ↑ | 0.631 | 0.723 ↑ | 0.763 | 0.835 ↑ | - | - | - |
| (1 +) | 75.5 | 80.7 ↑ | 0.146 | 0.327 ↑ | 0.152 | 0.348 ↑ | 0.149 | 0.337 ↑ | - | - | - | |
| (2 +) | 86.5 | 93.3 ↑ | 0.059 | 0.000 | 0.143 | 0.000 | 0.084 | - | - | - | - | |
| (3 +) | 81.3 | 87.1 ↑ | 0.254 | 0.382 ↑ | 0.692 | 1.000 ↑ | 0.372 | 0.553 ↑ | - | - | - | |
| (4 +) | 91.4 | 96.9 ↑ | 0.966 | 0.972 ↑ | 0.773 | 0.936 ↑ | 0.859 | 0.954 ↑ | - | - | - | |
| avg | 83.8 | 89.3 ↑ | 0.478 | 0.534 ↑ | 0.478 | 0.601 ↑ | 0.478 | 0.566 ↑ | 0.469 | 0.637 ↑ | 5.084 | |
| Imm-2 | (-) | 83.1 | 99.1 ↑ | 0.838 | 0.985 ↑ | 0.715 | 0.992 ↑ | 0.772 | 0.988 ↑ | - | - | - |
| (1 +) | 81.3 | 98.2 ↑ | 0.174 | 0.935 ↑ | 0.087 | 0.935 ↑ | 0.116 | 0.935 ↑ | - | - | - | |
| (2 +) | 89.6 | 97.2 ↑ | 0.045 | 0.778 ↑ | 0.071 | 0.500 ↑ | 0.055 | 0.609 ↑ | - | - | - | |
| (3 +) | 83.4 | 94.5 ↑ | 0.274 | 0.611 ↑ | 0.654 | 0.846 ↑ | 0.386 | 0.710 ↑ | - | - | - | |
| (4 +) | 90.2 | 95.1 ↑ | 0.861 | 0.952 ↑ | 0.845 | 0.900 ↑ | 0.853 | 0.925 ↑ | - | - | - | |
| avg | 85.5 | 96.8 ↑ | 0.438 | 0.852 ↑ | 0.474 | 0.835 ↑ | 0.456 | 0.843 ↑ | 0.500 | 0.886 ↑ | 7.5 | |
| Imm-3 | (-) | 96.6 | 100 ↑ | 0.922 | 1.000 ↑ | 1.000 | 1.000 | 0.959 | 1.000 ↑ | - | - | - |
| (1 +) | 95.1 | 98.5 ↑ | 1.000 | 1.000 | 0.652 | 0.891 ↑ | 0.789 | 0.942 ↑ | - | - | - | |
| (2 +) | 97.9 | 97.9 | 0.769 | 0.769 | 0.714 | 0.714 | 0.740 | 0.740 | - | - | - | |
| (3 +) | 93.3 | 93.3 | 0.542 | 0.542 | 1.000 | 1.000 | 0.703 | 0.703 | - | - | - | |
| (4 +) | 95.1 | 95.1 | 1.000 | 1.000 | 0.855 | 0.855 | 0.922 | 0.922 | - | - | - | |
| avg | 95.6 | 97.0 ↑ | 0.847 | 0.862 ↑ | 0.844 | 0.892 ↑ | 0.845 | 0.861 ↑ | 0.844 | 0.892 ↑ | 4 | |
| Imm-avg | (-) | 88 | 95.9 ↑ | 0.908 | 0.991 ↑ | 0.782 | 0.905 ↑ | 0.831 | 0.941 ↑ | - | - | - |
| (1 +) | 84 | 92.5 ↑ | 0.440 | 0.754 ↑ | 0.297 | 0.725 ↑ | 0.351 | 0.738 ↑ | - | - | - | |
| (2 +) | 91.3 | 96.1 ↑ | 0.291 | 0.516 ↑ | 0.309 | 0.405 ↑ | 0.293 | 0.675 ↑ | - | - | - | |
| (3 +) | 86 | 91.6 ↑ | 0.357 | 0.512 ↑ | 0.782 | 0.949 ↑ | 0.487 | 0.655 ↑ | - | - | - | |
| (4 +) | 92.2 | 95.7 ↑ | 0.942 | 0.959 ↑ | 0.824 | 0.912 ↑ | 0.878 | 0.935 ↑ | - | - | - | |
| avg | 88.3 | 94.4 ↑ | 0.588 | 0.749 ↑ | 0.599 | 0.776↑ | 0.593 | 0.757 ↑ | 0.604 | 0.805 ↑ | 5.528 | |
| Model | (-) | 100 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | - | - | - |
| (1 +) | 99.1 | 99.4↑ | 1 | 1 | 0.935 | 0.957↑ | 0.966 | 0.978↑ | - | - | - | |
| (2 +) | 99.7 | 99.4 | 0.933 | 0.875 | 1 | 1 | 0.965 | 0.933 | - | - | - | |
| (3 +) | 99.3 | 100↑ | 0.929 | 1↑ | 1 | 1 | 0.963 | 1↑ | - | - | - | |
| (4 +) | 100 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | - | - | - | |
| avg | 99.6 | 99.8 ↑ | 0.972 | 0.975 ↑ | 0.987 | 0.991 ↑ | 0.979 | 0.983 ↑ | 0.987 | 0.991 ↑ | 0.094 | |
Notes: Imm-n denotes Immunologist-1, Immunologist-2, Immunologist-3, and Immunologist-avg; w/o represents the immunologist without model assistance; w represents the immunologist with model assistance; “↑” indicates that the result of the immunologist with model assistance is better than that of the immunologists