Table 3.
The evaluation of cross-predictions and over-predictions for the 10 structure-trained and disorder-trained predictors of binding residues and the new hybridDBRpred meta-predictor using the sampled test dataset
| Dataset | Type of methods | Predictors | CPRratio at 0.1 FPR | AUCPC | OPRratio at 0.1 FPR | AUOPC |
|---|---|---|---|---|---|---|
| Structure-annotated proteins from benchmark dataset | ||||||
| Structure-trained | TargetS | 0.404 ± 0.125+/+ | 0.570 ± 0.033+/+ | 1.062 ± 0.354+/+ | 0.341 ± 0.027+/+ | |
| TargetDNA | 2.431 ± 0.406+/+ | 0.328 ± 0.032+/+ | 4.129 ± 0.531+/+ | 0.245 ± 0.027+/+ | ||
| BindN+ | 2.218 ± 0.457+/+ | 0.325 ± 0.034+/+ | 3.587 ± 0.332+/+ | 0.250 ± 0.018+/+ | ||
| DNAPred | 2.988 ± 0.525 /+ | 0.276 ± 0.030 /+ | 5.319 ± 0.540 / = | 0.180 ± 0.020 /+ | ||
| DNAgenie | 2.140 ± 0.781+/+ | 0.330 ± 0.065+/+ | 3.355 ± 0.633+/+ | 0.249 ± 0.035+/+ | ||
| Disorder-trained | fMoRFpred | 0.785 ± 0.180+/+ | 0.556 ± 0.023+/+ | 0.624 ± 0.105+/+ | 0.569 ± 0.018+/+ | |
| ANCHOR2 | 1.531 ± 1.306+/+ | 0.441 ± 0.063+/+ | 0.288 ± 0.243+/+ | 0.536 ± 0.033+/+ | ||
| DeepDISObind | 1.013 ± 0.904+/+ | 0.404 ± 0.052+/+ | 0.149 ± 0.149+/+ | 0.515 ± 0.035+/+ | ||
| MoRFchibi | 1.199 ± 0.383+/+ | 0.462 ± 0.039+/+ | 1.240 ± 0.308+/+ | 0.467 ± 0.027+/+ | ||
| DisoRDPbind | 2.420 ± 0.671+/+ | 0.353 ± 0.036+/+ | 1.879 ± 0.367+/+ | 0.380 ± 0.031+/+ | ||
| Baseline meta-predictors | Average-based | 3.178 ± 0.781=/+ | 0.254 ± 0.033–/+ | 5.033 ± 0.486–/= | 0.182 ± 0.017=/+ | |
| Logistic regression | 3.259 ± 0.747–/+ | 0.274 ± 0.027=/+ | 4.151 ± 0.548+/+ | 0.218 ± 0.019+/+ | ||
| Deep learning meta-predictor | hybridDBRpred | 4.004 ± 1.207–/ | 0.210 ± 0.038–/ | 5.156 ± 0.442=/ | 0.172 ± 0.017–/ | |
| Disorder- annotated proteins from benchmark dataset | ||||||
| Structure-trained | TargetS | 1.447 ± 0.414+/+ | 0.463 ± 0.046+/+ | 1.927 ± 0.259+/+ | 0.440 ± 0.029+/+ | |
| TargetDNA | 1.204 ± 0.342+/+ | 0.541 ± 0.037+/+ | 1.882 ± 0.359+/+ | 0.458 ± 0.031+/+ | ||
| BindN+ | 1.591 ± 0.505+/+ | 0.475 ± 0.046+/+ | 2.240 ± 0.426+/+ | 0.427 ± 0.035+/+ | ||
| DNAPred | 1.586 ± 0.447+/+ | 0.496 ± 0.040+/+ | 1.999 ± 0.289+/+ | 0.459 ± 0.031+/+ | ||
| DNAgenie | 3.077 ± 1.268 /+ | 0.323 ± 0.071 /+ | 3.415 ± 0.877 /+ | 0.316 ± 0.058 /+ | ||
| Disorder-trained | fMoRFpred | 0.947 ± 0.158+/+ | 0.515 ± 0.020+/+ | 1.255 ± 0.128+/+ | 0.483 ± 0.016+/+ | |
| ANCHOR2 | 1.044 ± 0.568+/+ | 0.571 ± 0.076+/+ | 1.874 ± 0.675+/+ | 0.386 ± 0.057+/+ | ||
| DeepDISObind | 2.615 ± 3.071+/+ | 0.454 ± 0.085+/+ | 1.794 ± 0.689+/+ | 0.342 ± 0.067+/+ | ||
| MoRFchibi | 1.615 ± 0.515+/+ | 0.428 ± 0.043+/+ | 2.347 ± 0.395+/+ | 0.362 ± 0.027+/+ | ||
| DisoRDPbind | 2.037 ± 0.587+/+ | 0.389 ± 0.046+/+ | 2.491 ± 0.398+/+ | 0.364 ± 0.028+/+ | ||
| Baseline meta-predictors | Average-based | 2.618 ± 0.843+/+ | 0.371 ± 0.051+/+ | 3.200 ± 0.526=/+ | 0.341 ± 0.037+/+ | |
| Logistic regression | 2.111 ± 0.734+/+ | 0.354 ± 0.052+/+ | 3.110 ± 0.457+/+ | 0.309 ± 0.026=/+ | ||
| Deep learning meta-predictor | hybridDBRpred | 4.864 ± 2.327–/ | 0.237 ± 0.064–/ | 3.979 ± 0.883–/ | 0.234 ± 0.049–/ | |
| Entire from benchmark dataset | ||||||
| Structure-trained | TargetS | 0.952 ± 0.207+/+ | 0.469 ± 0.040+/+ | 1.586 ± 0.236+/+ | 0.401 ± 0.022+/+ | |
| TargetDNA | 1.771 ± 0.366+/+ | 0.448 ± 0.032+/+ | 2.875 ± 0.340+/+ | 0.361 ± 0.024+/+ | ||
| BindN+ | 1.926 ± 0.477+/+ | 0.400 ± 0.038+/+ | 2.827 ± 0.291+/+ | 0.351 ± 0.024+/+ | ||
| DNAPred | 2.472 ± 0.533+/+ | 0.387 ± 0.035+/+ | 3.452 ± 0.338+/+ | 0.336 ± 0.025+/+ | ||
| DNAgenie | 2.988 ± 1.003 /+ | 0.287 ± 0.053 /+ | 3.349 ± 0.483 /+ | 0.298 ± 0.036 /+ | ||
| Disorder-trained | fMoRFpred | 0.765 ± 0.137+/+ | 0.548 ± 0.018+/+ | 0.956 ± 0.103+/+ | 0.520 ± 0.014+/+ | |
| ANCHOR2 | 0.660 ± 0.257+/+ | 0.630 ± 0.046+/+ | 1.754 ± 0.444+/+ | 0.447 ± 0.033+/+ | ||
| DeepDISObind | 1.379 ± 0.762+/+ | 0.578 ± 0.047+/+ | 1.357 ± 0.422+/+ | 0.417 ± 0.033+/+ | ||
| MoRFchibi | 1.424 ± 0.427+/+ | 0.439 ± 0.035+/+ | 1.687 ± 0.224+/+ | 0.413 ± 0.021+/+ | ||
| DisoRDPbind | 1.844 ± 0.428+/+ | 0.395 ± 0.037+/+ | 2.212 ± 0.269+/+ | 0.371 ± 0.021+/+ | ||
| Baseline meta-predictors | Average-based | 3.242 ± 0.864–/+ | 0.293 ± 0.038=/+ | 4.008 ± 0.338–/+ | 0.271 ± 0.022–/+ | |
| Logistic regression | 2.516 ± 0.772+/+ | 0.315 ± 0.042+/+ | 3.552 ± 0.350–/+ | 0.272 ± 0.017–/+ | ||
| Deep learning meta-predictor | hybridDBRpred | 5.413 ± 1.918–/ | 0.201 ± 0.042–/ | 4.275 ± 0.380–/ | 0.216 ± 0.023–/ | |
We report averages and the corresponding standard deviations over the 100 subsets (see ‘Assessment metrics and statistical analysis’ section for details). The best results for a given dataset and for each column are shown in bold font. We report results from the statistical significance test using superscript in the ‘x/y’ format where x indicates comparison against the current method with the highest AUC and y stands for the comparison against the new hybridDBRpred meta-predictor; +, =, and – denote that the best current predictor or hybridDBRpred is significantly better, not significantly different, significantly worse than another method at P-value < 0.01.