Table 3.
test set validation statistical significance using different scores. A. Rank normalized score (pairwise; Wilcoxon matched-pairs signed rank test), B. Rank normalized score (Unpaired Comparison; Mann-Whitney U test), C. ΔRNS (Unpaired Comparison; Mann-Whitney U test). Assay Central (AC); AdaBoost (ada); Bernoulli Naïve-Bayes (bnb); k-nearest neighbors (knn); Random Forest (rf); support vector classification (svc); Deep Neural Networks (DL).
A. | ||||||||
---|---|---|---|---|---|---|---|---|
ada | bnb | DL | knn | rf | svc | Consensus | AC | |
ada | N/A | 0.0006*** | 0.1011 | 0.5457 | 0.1688 | 0.0894 | 0.0340* | 0.0164* |
bnb | 0.0006*** | N/A | 0.0004*** | 0.0491* | 0.0035** | 0.0008*** | <0.0001**** | 0.9888 |
DL | 0.1011 | 0.0004*** | N/A | 0.0291* | 0.0425* | 0.5088 | 0.9218 | 0.0002*** |
knn | 0.5457 | 0.0491* | 0.0291* | N/A | 0.2768 | 0.0526 | 0.0457* | 0.0526 |
rf | 0.1688 | 0.0425* | 0.0035** | 0.2768 | N/A | 0.0039** | 0.0011** | 0.2076 |
svc | 0.0894 | 0.0008*** | 0.5088 | 0.0526 | 0.0039** | N/A | 0.5457 | 0.0008*** |
Consensus | 0.0340* | <0.0001**** | 0.9218 | 0.0457* | 0.0011** | 0.5457 | N/A | <0.0001**** |
AC | 0.0164* | 0.9888 | 0.0002*** | 0.0526 | 0.2076 | 0.0008*** | <0.0001**** | N/A |
B. | ||||||||
ada | bnb | DL | knn | rf | svc | Consensus | AC | |
ada | N/A | 0.2912 | 0.4071 | 0.6165 | 0.4803 | 0.3956 | 0.4803 | 0.2382 |
bnb | 0.2912 | N/A | 0.0355* | 0.3842 | 0.4932 | 0.0287* | 0.0483* | 0.8944 |
DL | 0.4071 | 0.0355* | N/A | 0.1153 | 0.1106 | >0.9999 | 0.8143 | 0.0587 |
knn | 0.6165 | 0.3842 | 0.1153 | N/A | 0.9430 | 0.1106 | 0.2221 | 0.6753 |
rf | 0.4803 | 0.4932 | 0.1106 | 0.9430 | N/A | 0.0890 | 0.1305 | 0.6456 |
svc | 0.3956 | 0.0287* | >0.9999 | 0.1106 | 0.0890 | N/A | 0.9593 | 0.0459* |
Consensus | 0.4803 | 0.0483* | 0.8143 | 0.2221 | 0.1305 | 0.9593 | N/A | 0.0533 |
AC | 0.2382 | 0.8944 | 0.0587 | 0.6753 | 0.6456 | 0.0459* | 0.0533 | N/A |
C. | ||||||||
ada | bnb | DL | knn | rf | svc | Consensus | AC | |
ada | N/A | 0.0025** | 0.1448 | 0.6444 | 0.1151 | 0.0756 | 0.1524 | 0.0205* |
bnb | 0.0025** | N/A | <0.0001**** | 0.0217* | 0.2818 | <0.0001**** | <0.0001**** | 0.6456 |
DL | 0.1448 | <0.0001**** | N/A | 0.0349* | 0.0023** | 0.4499 | 0.6159 | <0.0001**** |
knn | 0.6444 | 0.0217* | 0.0349* | N/A | 0.2008 | 0.0263* | 0.0534 | 0.0820 |
rf | 0.1151 | 0.2818 | 0.0023** | 0.2008 | N/A | 0.0012** | 0.0022** | 0.5364 |
svc | 0.0756 | <0.0001**** | 0.4499 | 0.0263* | 0.0012** | N/A | 0.3187 | <0.0001**** |
Consensus | 0.1524 | <0.0001**** | 0.6159 | 0.0534 | 0.0022** | 0.3187 | N/A | 0.0001*** |
AC | 0.0205* | 0.6456 | <0.0001**** | 0.0820 | 0.5364 | <0.0001**** | 0.0001*** | N/A |