Table 3.
Comparative assessment for the residue-level disorder predictions on the CAID dataset. We sort predictors by the area under receiver operating characteristic curve (AUC) values. The “Threshold” column provides the cut-off values that we use to convert the real-valued propensities into the binary scores. We assess statistical significance of the differences when compared with the top-ranked flDPnn, AF2-RSA, and AF2-pLDDT, which we highlight using bold font; we show results next to the measured metric using the x|y|z format, where x denotes that flDPnn is significantly better (+), worse (-), and not different (=) than the results from a given predictor at p-value = 0.05; y compares against AF2-RSA; and z compares against AF2-pLDDT. Acronyms: area under the precision-recall curve (AUPRC) and Matthew correlation coefficient (MCC).
| Predictor | Threshold | Coverage (%) | AUC | AUPRC | MCC | F1 |
|---|---|---|---|---|---|---|
| flDPnn | 0.337 | 100 | 0.814 /-/- | 0.475 /-/- | 0.358 /-/- | 0.462 /-/- |
| flDPlr | 0.417 | 100 | 0.793 + /-/- | 0.422 + /-/- | 0.323 + /- /- | 0.433 + /-/- |
| AF2-RSAoptimized-rank | 0.857 | 98 | 0.785 + /-/- | 0.357 + /-/- | 0.248 + /= /- | 0.380 + /= /- |
| RawMSA | 0.683 | 100 | 0.780 + /= /- | 0.414 + /-/- | 0.288 + /= /- | 0.404 + /-/- |
| Espritz-D | 0.477 | 100 | 0.774 + /= /- | 0.410 + /-/- | 0.289 + /-/- | 0.406 + /-/- |
| AF2-RSA | 0.847 | 98 | 0.768 + / /- | 0.325 + / /- | 0.203 + / /- | 0.343 + / /- |
| DisoMine | 0.563 | 100 | 0.765 + /= /- | 0.388 + /- /- | 0.244 + /= /- | 0.367 + /= /- |
| SPOT-Disorder2 | 0.824 | 94 | 0.760 + /= /- | 0.340 + /= /- | 0.200 + /= /- | 0.351 + /= /- |
| AUCpred | 1.000 | 100 | 0.757 + /= /- | 0.479 = /-/- | 0.258 + /= /- | 0.399 + /-/- |
| SPOT-Disorder-Single | 0.764 | 100 | 0.757 + /= /- | 0.318 + /= /- | 0.221 + /= /- | 0.348 + /= /- |
| IDPseq2seq | 0.976 | 100 | 0.754 + /= /- | 0.322 + /= /- | 0.209 + /= /- | 0.339 + /= /- |
| AUCpred-np | 1.000 | 100 | 0.751 + /+ /- | 0.428 + /-/- | 0.226 + /= /- | 0.349 + /= /- |
| Metapredict | 0.615 | 100 | 0.746 + /+ /- | 0.340 + /= /- | 0.241 + /= /- | 0.365 + /= /- |
| SPOT-Disorder1 | 0.945 | 100 | 0.744 + /+ /- | 0.268 + /+ /= | 0.143 + /+ /= | 0.284 + /= /= |
| IUPred-short | 0.613 | 100 | 0.739 + /+ /= | 0.311 + /= /- | 0.221 + /= /= | 0.349 + /= /- |
| AF2-pLDDToptimized-rank | 0.657 | 99 | 0.737 + /+ /= | 0.289 + /= /= | 0.160 + /= /= | 0.290 + /= /= |
| IUPred-long | 0.719 | 100 | 0.737 + /+ /= | 0.298 + /= /= | 0.218 + /= /- | 0.346 + /= /- |
| ODiNPred | 0.996 | 100 | 0.734 + /+ /= | 0.314 + /+ /= | 0.207 + /= /- | 0.330 + /= /- |
| VSL2B | 0.905 | 100 | 0.732 + /+ /= | 0.301 + /= /- | 0.203 + /= /- | 0.333 + /= /- |
| AF2-pLDDT | 0.628 | 99 | 0.722 + /+ / | 0.272 + /+ / | 0.137 + /+ / | 0.278 + /+ / |
| RFPR | 1.000 | 100 | 0.721 + /+ /= | 0.338 + /= /- | 0.109 + /+ /= | 0.219 + /+ /+ |
| DISOPRED3 | 0.965 | 100 | 0.701 + /+ /+ | 0.290 + /= /= | 0.122 + /+ /= | 0.263 + /+ /= |
| DisEMBL-465 | 0.533 | 100 | 0.685 + /+ /+ | 0.283 + /+ /= | 0.196 + /+ /- | 0.328 + /= /- |
| DisEMBL-HL | 0.131 | 100 | 0.654 + /+ /+ | 0.274 + /+ /= | 0.170 + /+ /= | 0.302 + /= /= |