Table 1.
Dependency on the model quality parameter λ using models generated from datasets with 100 actives (n) on 10,000 compounds in total (N)
| Metric | λ | χ (%) | ||||
|---|---|---|---|---|---|---|
| 2 | 5 | 10 | 20 | 40 | ||
| PM | 0.51 ± 0.35 | 0.74 ± 0.24 | 0.89 ± 0.09 | 0.95 ± 0.02 | 0.98 ± 0.01 | 0.5 |
| ROCE | 2.35 ± 2.18 | 5.13 ± 3.39 | 10.46 ± 4.99 | 22.34 ± 7.86 | 49.96 ± 14.35 | |
| EF | 2.28 ± 2.06 | 4.83 ± 3.03 | 9.38 ± 4.04 | 18.08 ± 5.17 | 32.94 ± 6.22 | |
| REF | 2.28 ± 2.06 | 4.83 ± 3.03 | 9.38 ± 4.04 | 18.08 ± 5.17 | 32.94 ± 6.22 | |
| CCR | 0.50 ± 0.01 | 0.51 ± 0.01 | 0.52 ± 0.01 | 0.54 ± 0.01 | 0.58 ± 0.02 | |
| MCC | 0.01 ± 0.01 | 0.03 ± 0.02 | 0.06 ± 0.03 | 0.12 ± 0.04 | 0.23 ± 0.04 | |
| CKC | 0.01 ± 0.01 | 0.03 ± 0.02 | 0.06 ± 0.03 | 0.11 ± 0.03 | 0.21 ± 0.04 | |
| SEN | 0.01 ± 0.01 | 0.02 ± 0.02 | 0.05 ± 0.02 | 0.09 ± 0.03 | 0.16 ± 0.03 | |
| SPE | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | |
| PRE | 0.02 ± 0.02 | 0.05 ± 0.03 | 0.09 ± 0.04 | 0.18 ± 0.05 | 0.33 ± 0.06 | |
| ACC | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | |
| PM | 0.61 ± 0.23 | 0.80 ± 0.11 | 0.90 ± 0.04 | 0.95 ± 0.01 | 0.98 ± 0.00 | 1 |
| ROCE | 2.32 ± 1.55 | 5.07 ± 2.31 | 10.19 ± 3.34 | 20.97 ± 5.08 | 44.00 ± 8.22 | |
| EF | 2.26 ± 1.48 | 4.83 ± 2.09 | 9.25 ± 2.75 | 17.33 ± 3.46 | 30.54 ± 3.95 | |
| REF | 2.26 ± 1.48 | 4.83 ± 2.09 | 9.25 ± 2.75 | 17.33 ± 3.46 | 30.54 ± 3.95 | |
| CCR | 0.51 ± 0.01 | 0.52 ± 0.01 | 0.54 ± 0.01 | 0.58 ± 0.02 | 0.65 ± 0.02 | |
| MCC | 0.01 ± 0.02 | 0.04 ± 0.02 | 0.08 ± 0.03 | 0.17 ± 0.03 | 0.30 ± 0.04 | |
| CKC | 0.01 ± 0.02 | 0.04 ± 0.02 | 0.08 ± 0.03 | 0.17 ± 0.03 | 0.30 ± 0.04 | |
| SEN | 0.02 ± 0.01 | 0.05 ± 0.02 | 0.09 ± 0.03 | 0.17 ± 0.03 | 0.31 ± 0.04 | |
| SPE | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 | |
| PRE | 0.02 ± 0.01 | 0.05 ± 0.02 | 0.09 ± 0.03 | 0.17 ± 0.03 | 0.31 ± 0.04 | |
| ACC | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.99 ± 0.00 | |
| PM | 0.66 ± 0.13 | 0.82 ± 0.06 | 0.90 ± 0.02 | 0.95 ± 0.01 | 0.97 ± 0.00 | 2 |
| ROCE | 2.30 ± 1.08 | 4.91 ± 1.56 | 9.69 ± 2.18 | 18.75 ± 3.08 | 35.21 ± 4.06 | |
| EF | 2.26 ± 1.03 | 4.70 ± 1.43 | 8.88 ± 1.82 | 15.87 ± 2.19 | 26.17 ± 2.23 | |
| REF | 4.52 ± 2.07 | 9.40 ± 2.85 | 17.76 ± 3.65 | 31.74 ± 4.38 | 52.34 ± 4.45 | |
| CCR | 0.51 ± 0.01 | 0.54 ± 0.01 | 0.58 ± 0.02 | 0.65 ± 0.02 | 0.75 ± 0.02 | |
| MCC | 0.02 ± 0.01 | 0.05 ± 0.02 | 0.11 ± 0.03 | 0.21 ± 0.03 | 0.36 ± 0.03 | |
| CKC | 0.02 ± 0.01 | 0.05 ± 0.02 | 0.11 ± 0.02 | 0.20 ± 0.03 | 0.34 ± 0.03 | |
| SEN | 0.05 ± 0.02 | 0.09 ± 0.03 | 0.18 ± 0.04 | 0.32 ± 0.04 | 0.52 ± 0.04 | |
| SPE | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.98 ± 0.00 | 0.99 ± 0.00 | |
| PRE | 0.02 ± 0.01 | 0.05 ± 0.01 | 0.09 ± 0.02 | 0.16 ± 0.02 | 0.26 ± 0.02 | |
| ACC | 0.97 ± 0.00 | 0.97 ± 0.00 | 0.97 ± 0.00 | 0.98 ± 0.00 | 0.98 ± 0.00 | |
Metric abbreviations are given in the Methods section. All metrics are dependent on the model quality, but in case of the ROCE, EF, REF, MCC, CKC, SEN and PRE metrics there is at least a tenfold increase when moving from a bad model (λ = 2) to a good model (λ = 40), while for the PM metric there is a doubling of the value. The accuracy ACC and specificity SPE metrics are not dependent on the quality of model, while the correct classification rate metric (CCR) shifts from 0.5 in the case of a bad model to a maximum of 0.75 for the best model. Good models have a PM of >0.9; for good models this value is largely independent on the applied cutoff value χ (see Table 3 as well)