Table 3.
Performance metrics for the different bioinformatics assays for the blast+-, KMA- and SRST2-based detection methods
|
Bioinformatics assay |
Reference standard |
Detection method |
TP |
FN |
TN |
FP |
Accuracy (%) |
Precision (%) |
Sensitivity (%) |
Specificity (%) |
Repeatability (%) |
Reproducibility (%) |
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
AMR prediction |
Database |
blast+ |
80 |
3 |
196 |
4 |
97.53 |
95.24 |
96.39 |
98.00 |
100.00 |
100.00 |
|
|
KMA |
79 |
4 |
196 |
4 |
97.17 |
95.18 |
95.18 |
98.00 |
100.00 |
100.00 |
|||
|
SRST2 |
80 |
3 |
196 |
4 |
97.53 |
95.24 |
96.39 |
98.00 |
100.00 |
100.00 |
|||
|
Virulence gene detection |
stx genes |
Database |
blast+ |
138 |
3 |
121 |
0 |
98.85 |
100.00 |
97.87 |
100.00 |
100.00 |
100.00 |
|
KMA |
141 |
0 |
120 |
1* |
99.62 |
99.30 |
100.00 |
99.17 |
100.00 |
100.00 |
|||
|
SRST2 |
141 |
0 |
120 |
1* |
99.62 |
99.30 |
100.00 |
99.17 |
100.00 |
100.00 |
|||
|
Other genes |
Database |
blast+ |
883 |
20 |
718 |
0 |
98.77 |
100.00 |
97.79 |
100.00 |
100.00 |
100.00 |
|
|
KMA |
903 |
0 |
717 |
1 |
99.94 |
99.89 |
100.00 |
99.86 |
100.00 |
100.00 |
|||
|
SRST2 |
902 |
1 |
718 |
0 |
99.94 |
100.00 |
99.89 |
100.00 |
100.00 |
100.00 |
|||
|
Serotype determination |
Database |
blast+ |
103 |
3 |
10 |
0 |
97.41 |
100.00 |
97.17 |
100.00 |
100.00 |
100.00 |
|
|
KMA |
104 |
2 |
10 |
0 |
98.28 |
100.00 |
98.11 |
100.00 |
100.00 |
100.00 |
|||
|
SRST2 |
104 |
2 |
10 |
0 |
98.28 |
100.00 |
98.11 |
100.00 |
100.00 |
100.00 |
|||
|
Plasmid replicon detection |
Tool |
blast+ |
333 (333)† |
0 (15)† |
12 505 (12 490)† |
0 (0)† |
100.00 (99.88)† |
100.00 (100.00)† |
100.00 (95.69)† |
100.00 (100.00)† |
100.00 |
100.00 |
|
|
KMA |
292 (299)† |
41 (49)† |
12 496 (12 490)† |
9 (0)† |
99.61 (99.62)† |
97.01 (100.00)† |
87.69 (85.92)† |
99.93 (100.00)† |
100.00 |
100.00 |
|||
|
SRST2 |
319 (334)† |
14 (14)† |
12 490 (12 490)† |
15 (0)† |
99.77 (99.89)† |
95.51 (100.00)† |
95.80 (95.98†) |
99.88 (100.00)† |
100.00 |
100.00 |
|||
|
Sequence typing (cgMLST) |
Tool |
blast+ |
328 329 |
874 |
24 595 |
535 |
99.60 |
99.84 |
99.73 |
97.87 |
100.00 |
100.00 |
|
|
KMA |
327 387 |
1816 |
24 595 |
535 |
99.34 |
99.84 |
99.45 |
97.87 |
100.00 |
100.00 |
|||
|
SRST2‡ |
– |
– |
– |
– |
– |
– |
– |
– |
– |
– |
|||
*Probably caused by low-level within-species contamination of sample EH2038.
†Updated results when FP results that could be traced back to algorithmic issues with blast+ are considered as TP.
‡cgMLST using SRST2 could not be evaluated due to the long running times.
TP, true positive; FP, false positive; FN, false negative;TN, true negative.