Table 3.
Accuracy metrics for antagonist activity evaluation with different numbers of activity sources per chemical
| # sources | # chemicals | Model and parameters | Sensitivity | Specificity | Bal Accuracy | Accuracy | ROC AUC | Score |
|---|---|---|---|---|---|---|---|---|
| 1 | 6533 | CERAPP consensus | 0.15 | 0.91 | 0.53 | 0.88 | 0.55 | 0.26 |
| 1 | 6533 | Morgan kNN arithm k = 3 | 0.04 | 0.99 | 0.52 | 0.95 | 0.53 | 0.26 |
| 1 | 6533 | Morgan kNN geom k = 3 | 0.00 | 1.00 | 0.50 | 0.96 | 0.51 | 0.24 |
| 1 | 6533 | Morgan kNN exp k = 3 X = 1.5 | 0.04 | 0.99 | 0.52 | 0.95 | 0.53 | 0.26 |
| 1 | 6533 | Indigo kNN arithm k = 10 | 0.04 | 0.99 | 0.52 | 0.95 | 0.57 | 0.28 |
| 1 | 6533 | Indigo kNN geom k = 10 | 0.00 | 1.00 | 0.50 | 0.96 | 0.50 | 0.24 |
| 1 | 6533 | Indigo kNN exp k = 10 X = 1.5 | 0.05 | 0.99 | 0.52 | 0.95 | 0.57 | 0.28 |
| 1 | 6533 | Indigo GkNN k = 10 X = 3 Y = 7 | 0.10 | 0.98 | 0.54 | 0.94 | 0.57 | 0.29 |
| 1 | 6533 | Indigo GkNN k = 10 X = 5 Y = 15 | 0.10 | 0.98 | 0.54 | 0.94 | 0.57 | 0.29 |
| 3 | 1707 | CERAPP consensus | 0.17 | 0.90 | 0.53 | 0.87 | 0.58 | 0.27 |
| 3 | 1707 | Morgan kNN arithm k = 3 | 0.09 | 0.99 | 0.54 | 0.95 | 0.57 | 0.29 |
| 3 | 1707 | Morgan kNN geom k = 3 | 0.00 | 1.00 | 0.50 | 0.95 | 0.53 | 0.25 |
| 3 | 1707 | Morgan kNN exp k = 3 X = 1.5 | 0.10 | 1.00 | 0.55 | 0.96 | 0.57 | 0.30 |
| 3 | 1707 | Indigo kNN arithm k = 10 | 0.12 | 1.00 | 0.56 | 0.96 | 0.65 | 0.35 |
| 3 | 1707 | Indigo kNN geom k = 10 | 0.00 | 1.00 | 0.50 | 0.95 | 0.50 | 0.24 |
| 3 | 1707 | Indigo kNN exp k = 10 X = 1.5 | 0.14 | 1.00 | 0.57 | 0.96 | 0.65 | 0.36 |
| 3 | 1707 | Indigo GkNN k = 10 X = 3 Y = 7 | 0.18 | 0.99 | 0.58 | 0.95 | 0.65 | 0.36 |
| 3 | 1707 | Indigo GkNN k = 10 X = 5 Y = 15 | 0.18 | 0.99 | 0.58 | 0.95 | 0.65 | 0.36 |
| 5 | 431 | CERAPP consensus | 0.24 | 0.89 | 0.56 | 0.84 | 0.67 | 0.32 |
| 5 | 431 | Morgan kNN arithm k = 3 | 0.14 | 0.99 | 0.56 | 0.93 | 0.61 | 0.32 |
| 5 | 431 | Morgan kNN geom k = 3 | 0.00 | 1.00 | 0.50 | 0.93 | 0.52 | 0.24 |
| 5 | 431 | Morgan kNN exp k = 3 X = 1.5 | 0.17 | 1.00 | 0.58 | 0.94 | 0.61 | 0.34 |
| 5 | 431 | Indigo kNN arithm k = 10 | 0.10 | 1.00 | 0.55 | 0.94 | 0.65 | 0.33 |
| 5 | 431 | Indigo kNN geom k = 10 | 0.00 | 1.00 | 0.50 | 0.93 | 0.50 | 0.23 |
| 5 | 431 | Indigo kNN exp k = 10 X = 1.5 | 0.10 | 1.00 | 0.55 | 0.94 | 0.65 | 0.33 |
| 5 | 431 | Indigo GkNN k = 10 X = 3 Y = 7 | 0.17 | 0.99 | 0.58 | 0.93 | 0.65 | 0.35 |
| 5 | 431 | Indigo GkNN k = 10 X = 5 Y = 15 | 0.17 | 0.99 | 0.58 | 0.93 | 0.65 | 0.35 |
| 7 | 103 | CERAPP consensus | 0.31 | 0.91 | 0.61 | 0.84 | 0.67 | 0.34 |
| 7 | 103 | Morgan kNN arithm k = 3 | 0.23 | 0.98 | 0.60 | 0.88 | 0.68 | 0.36 |
| 7 | 103 | Morgan kNN geom k = 3 | 0.00 | 1.00 | 0.50 | 0.87 | 0.54 | 0.24 |
| 7 | 103 | Morgan kNN exp k = 3 X = 1.5 | 0.23 | 1.00 | 0.62 | 0.90 | 0.68 | 0.38 |
| 7 | 103 | Indigo kNN arithm k = 10 | 0.08 | 1.00 | 0.54 | 0.88 | 0.79 | 0.38 |
| 7 | 103 | Indigo kNN geom k = 10 | 0.00 | 1.00 | 0.50 | 0.87 | 0.50 | 0.22 |
| 7 | 103 | Indigo kNN exp k = 10 X = 1.5 | 0.15 | 0.98 | 0.57 | 0.87 | 0.80 | 0.39 |
| 7 | 103 | Indigo GkNN k = 10 X = 3 Y = 7 | 0.23 | 0.98 | 0.60 | 0.88 | 0.80 | 0.43 |
| 7 | 103 | Indigo GkNN k = 10 X = 5 Y = 15 | 0.31 | 0.99 | 0.65 | 0.90 | 0.80 | 0.47 |
| 9 | 46 | CERAPP consensus | 0.40 | 1.00 | 0.70 | 0.87 | 0.73 | 0.44 |
| 9 | 46 | Morgan kNN arithm k = 3 | 0.30 | 0.97 | 0.64 | 0.83 | 0.73 | 0.38 |
| 9 | 46 | Morgan kNN geom k = 3 | 0.00 | 1.00 | 0.50 | 0.78 | 0.55 | 0.22 |
| 9 | 46 | Morgan kNN exp k = 3 X = 1.5 | 0.30 | 1.00 | 0.65 | 0.85 | 0.73 | 0.40 |
| 9 | 46 | Indigo kNN arithm k = 10 | 0.10 | 1.00 | 0.55 | 0.80 | 0.79 | 0.35 |
| 9 | 46 | Indigo kNN geom k = 10 | 0.00 | 1.00 | 0.50 | 0.78 | 0.50 | 0.20 |
| 9 | 46 | Indigo kNN exp k = 10 X = 1.5 | 0.20 | 0.97 | 0.59 | 0.80 | 0.79 | 0.37 |
| 9 | 46 | Indigo GkNN k = 10 X = 3 Y = 7 | 0.30 | 0.97 | 0.64 | 0.83 | 0.80 | 0.42 |
| 9 | 46 | Indigo GkNN k = 10 X = 5 Y = 15 | 0.40 | 1.00 | 0.70 | 0.87 | 0.80 | 0.49 |
“kNN arithm”, “kNN geom”, and “kNN exp” indicate the kNN models with the arithmetic, geometric, and exponential averaging, respectively. The cumulative score shown in the last column is the product of balanced accuracy, accuracy, and ROC AUC. Italic font indicates accuracy metric values that exceed those for the CERAPP consensus model