Table 1. Benchmarks using simulated sequence data.
Public complete genomes of three species, including (a) S. aureus, (b) N. meningitidis and (c) E. coli, were used for the benchmark. Identity, sequence identity between the root and reference isolates; positive predictive value (PPV), ratio of true-positive detected SNP sites to all detected SNP sites; Sensitivity, ratio of true-positive detected SNP sites to all true SNP sites; Called-sites, ratio of sites where a nucleotide was determined unambiguously in all isolates to the reference-genome size. All stats are averaged values among ten duplications. Called-sites ratio was not calculated for Cortex and CFSAN and was represented as ‘–’, because they do not output information for non-variant regions.
| (a) | Identity (%) | Cortex | Freebayes | GATK | SAMtools | VarScan | Snippy | CFSAN | NASP | PHEnix | BactSNP |
|---|---|---|---|---|---|---|---|---|---|---|---|
| PPV (%) | 99.9 | 99.07 | 74.35 | 73.04 | 93.36 | 96.27 | 58.05 | 99.78 | 100.00 | 99.94 | 100.00 |
| 99 | 98.16 | 38.69 | 26.75 | 63.68 | 78.22 | 7.13 | 95.45 | 100.00 | 99.89 | 100.00 | |
| 98 | 98.59 | 31.71 | 25.18 | 53.33 | 72.04 | 3.31 | 96.22 | 100.00 | 99.67 | 100.00 | |
| 97 | 98.37 | 27.55 | 21.17 | 46.73 | 71.34 | 2.13 | 95.34 | 99.94 | 99.44 | 100.00 | |
| Common-region PPV (%) |
99.9 | 100.00 | 100.00 | 98.79 | 99.91 | 100.00 | 94.24 | 100.00 | 100.00 | 100.00 | 100.00 |
| 99 | 99.89 | 98.05 | 93.55 | 98.76 | 98.50 | 48.47 | 100.00 | 100.00 | 100.00 | 100.00 | |
| 98 | 100.00 | 98.46 | 95.34 | 97.63 | 99.48 | 33.57 | 100.00 | 100.00 | 100.00 | 100.00 | |
| 97 | 100.00 | 99.17 | 89.87 | 96.88 | 98.60 | 24.31 | 99.84 | 100.00 | 100.00 | 100.00 | |
| Sensitivity (%) | 99.9 | 95.37 | 99.15 | 99.71 | 99.83 | 99.39 | 99.66 | 99.04 | 97.81 | 99.83 | 99.55 |
| 99 | 85.63 | 94.83 | 99.56 | 99.56 | 99.45 | 98.62 | 97.29 | 97.79 | 99.45 | 99.06 | |
| 98 | 77.55 | 87.97 | 98.04 | 98.99 | 98.59 | 96.98 | 90.85 | 96.74 | 98.76 | 98.54 | |
| 97 | 73.24 | 81.09 | 97.60 | 98.82 | 97.60 | 95.42 | 81.25 | 94.97 | 98.49 | 97.71 | |
| Called-sites (%) | 99.9 | – | 88.92 | 95.36 | 64.40 | 95.96 | 94.19 | – | 91.91 | 94.78 | 93.69 |
| 99 | – | 84.31 | 90.11 | 61.24 | 89.88 | 87.80 | – | 87.16 | 89.37 | 88.25 | |
| 98 | – | 84.96 | 90.40 | 61.86 | 89.97 | 86.69 | – | 87.27 | 89.42 | 88.30 | |
| 97 | – | 84.38 | 89.65 | 61.71 | 88.90 | 84.66 | – | 86.39 | 88.39 | 87.12 |
| (b) | Identity (%) | Cortex | Freebayes | GATK | SAMtools | VarScan | Snippy | CFSAN | NASP | PHEnix | BactSNP |
|---|---|---|---|---|---|---|---|---|---|---|---|
| PPV (%) | 99.9 | 96.08 | 94.94 | 96.21 | 98.61 | 98.91 | 47.46 | 99.56 | 100.00 | 100.00 | 100.00 |
| 99 | 96.38 | 50.35 | 41.42 | 71.33 | 80.49 | 5.77 | 96.10 | 100.00 | 99.66 | 100.00 | |
| 98 | 96.57 | 27.24 | 19.41 | 46.44 | 61.13 | 2.72 | 91.34 | 99.83 | 99.89 | 100.00 | |
| 97 | 96.81 | 23.68 | 16.88 | 43.20 | 64.76 | 2.12 | 91.29 | 100.00 | 99.66 | 100.00 | |
| Common-region PPV (%) |
99.9 | 99.38 | 100.00 | 99.89 | 99.89 | 100.00 | 97.47 | 100.00 | 100.00 | 100.00 | 100.00 |
| 99 | 99.41 | 99.44 | 98.04 | 99.18 | 99.79 | 81.17 | 99.78 | 100.00 | 100.00 | 100.00 | |
| 98 | 99.35 | 98.64 | 93.38 | 98.11 | 99.67 | 69.99 | 99.38 | 100.00 | 100.00 | 100.00 | |
| 97 | 99.51 | 98.67 | 89.41 | 98.14 | 99.74 | 62.64 | 99.70 | 100.00 | 100.00 | 100.00 | |
| Sensitivity (%) | 99.9 | 95.43 | 99.16 | 99.55 | 99.49 | 99.33 | 98.61 | 97.33 | 96.09 | 98.94 | 97.32 |
| 99 | 89.44 | 94.30 | 99.17 | 98.95 | 98.89 | 98.17 | 93.26 | 96.30 | 98.78 | 98.01 | |
| 98 | 82.86 | 88.95 | 98.41 | 98.69 | 97.97 | 96.04 | 84.91 | 95.43 | 98.30 | 97.36 | |
| 97 | 74.35 | 84.05 | 97.54 | 97.70 | 96.53 | 94.26 | 76.42 | 94.40 | 97.65 | 96.65 | |
| Called-sites (%) | 99.9 | – | 91.19 | 97.74 | 66.25 | 99.30 | 95.21 | – | 88.05 | 96.47 | 91.41 |
| 99 | – | 88.64 | 94.50 | 64.33 | 96.14 | 90.67 | – | 85.89 | 92.95 | 89.20 | |
| 98 | – | 86.79 | 91.86 | 63.25 | 92.30 | 85.30 | – | 82.38 | 89.29 | 85.60 | |
| 97 | – | 81.61 | 86.06 | 59.54 | 86.04 | 78.85 | – | 78.48 | 83.62 | 80.90 |
| (c) | Identity (%) | Cortex | Freebayes | GATK | SAMtools | VarScan | Snippy | CFSAN | NASP | PHEnix | BactSNP |
|---|---|---|---|---|---|---|---|---|---|---|---|
| PPV (%) | 99.9 | 98.60 | 35.81 | 35.04 | 89.98 | 83.32 | 56.27 | 99.89 | 100.00 | 100.00 | 100.00 |
| 99 | 98.43 | 28.60 | 21.72 | 74.45 | 76.59 | 5.38 | 96.78 | 100.00 | 99.94 | 100.00 | |
| 98 | 98.70 | 15.03 | 9.95 | 42.77 | 52.34 | 2.39 | 94.90 | 100.00 | 99.78 | 100.00 | |
| 97 | 99.05 | 23.64 | 18.70 | 70.21 | 81.90 | 1.54 | 94.24 | 99.94 | 99.83 | 100.00 | |
| Common-region PPV (%) |
99.9 | 100.00 | 99.30 | 94.66 | 99.89 | 99.89 | 96.48 | 100.00 | 100.00 | 100.00 | 100.00 |
| 99 | 99.76 | 98.76 | 89.64 | 99.48 | 99.78 | 76.46 | 99.76 | 100.00 | 100.00 | 100.00 | |
| 98 | 100.00 | 96.56 | 78.04 | 98.63 | 98.84 | 54.01 | 99.74 | 100.00 | 100.00 | 100.00 | |
| 97 | 100.00 | 97.35 | 84.23 | 99.31 | 99.86 | 52.38 | 98.98 | 100.00 | 100.00 | 100.00 | |
| Sensitivity (%) | 99.9 | 97.47 | 99.22 | 99.72 | 99.67 | 99.50 | 99.33 | 98.39 | 98.78 | 99.50 | 99.33 |
| 99 | 93.67 | 95.12 | 99.66 | 99.49 | 98.87 | 98.48 | 97.08 | 98.09 | 99.61 | 99.16 | |
| 98 | 87.30 | 88.01 | 98.95 | 99.17 | 98.73 | 98.21 | 91.78 | 97.10 | 99.33 | 99.00 | |
| 97 | 83.21 | 78.37 | 98.70 | 99.21 | 98.20 | 96.79 | 80.94 | 95.31 | 98.93 | 98.81 | |
| Called-sites (%) | 99.9 | – | 88.78 | 95.09 | 64.12 | 96.55 | 93.01 | – | 90.19 | 94.02 | 92.40 |
| 99 | – | 82.60 | 88.20 | 59.80 | 88.39 | 86.07 | – | 85.56 | 87.39 | 86.44 | |
| 98 | – | 78.74 | 83.60 | 56.95 | 83.49 | 79.77 | – | 79.59 | 82.19 | 81.00 | |
| 97 | – | 74.93 | 79.52 | 54.62 | 80.10 | 76.24 | – | 77.28 | 78.68 | 77.89 |