Table 3.
Performance evaluation reported for each method, highlighting possible biases.
| Method | Validation | Training data | Test Data | Correlation (ρ) | Anti-symmetry (ρ_dir-inv) | Sequence Identity/homologs |
|---|---|---|---|---|---|---|
| FoldX [18] | none | S339 | S625 | 0.82 | Biased* (−0.38) | not declared |
| MUpro [51] | 20‐fold CV and LOO | S1615 | S1615, S388 | 0.13–0.76 | Biased* (−0.02) | Homologs removed from S1615 (SR1496, SR1135, SR1023, SR1539) |
| CUPSAT [36] | 3/4/5-fold CV | S1538, S1603 | S1538, S1603 | 0.55–0.78 | Biased* (−0.54) | not declared |
| I-Mutant(3.0/2.0) [26], [40] | 10/20/30-fold CV | S1948 | S1948 | 0.62–0.71 | Biased* (0.02) | not declared |
| iPTREE-STAB [25] | 4/10/20-fold CV | S1859 | S1859 | 0.7 | not evaluated | not declared |
| AUTO-MUTE (2.0) [69] | 20‐fold CV | Subsets of S1948, S1615, S388,S1791, S1396, S2204 | Subsets of S1948, S1615, S388,S1791, S1396 | 0.74–0.79 | Biased* (−0.06) | not declared |
| Prethermut [29] | 10-fold CV | S3366, S2156 | S3366, S2156 | 0.67–0.72 | not evaluated | not declared |
| POPMUSIC(3.1/2.1) [28] | 5-fold CV | S2648 | S2648 | 0.63–0.79 | Biased* (−0.29); unbiased version POPMUSICsym (−0.77) | not declared |
| Pro-Maya [52] | 5/10-fold CV and LOO | S2648, S2156 | S2648, S2156 | 0.59–0.8 | not evaluated | <30% identity and keeping information on the mutation site |
| PROTS-RF [51] | 5/10-fold CV | S2156 | S2156 + D180, D140 (27 and 19 proteins) | 0.62–0.86 | Unbiased | <30% identity in CV |
| iStable(2.0) [30], [53] | 5-fold CV v2.0: 10-fold CV | S2648, S1948 v2.0: S3528 | M1311, M1820 (from S2648, S1948, no redundancies) v2.0: S630 | 0.85 v2.0: 0.67–0.71 | Biased* (−0.05) v2.0: not evaluated | Meta predictor combining several predictors and using the same protein variant to train the combined model |
| NeEMO [54] | 10-fold CV | S2399 (113 proteins) | IM_631 (from S2399) | 0.5–0.79 | Biased* (0.09) | Evaluated but not used in CV |
| DUET [41] | S350, p53 | S2648 | p53,S350 | 0.71–0.82 | Biased* (−0.21) | not declared |
| mCSM [20] | 5/10/20-fold CV | S2648, S1925 | S350 | 0.51–0.82 | Biased* (−0.26) | 5-fold cross-validation separating by protein (Pearson = 0.51) |
| EASE-MM [24] | 10-fold CV and blind | 1676 mutations from S1948 | S543, S236 | 0.51–0.59 | not evaluated | max 25% sequence identity between folds and train/test sets |
| INPS(3D) [43], [55] | 5-fold CV | S2648, p53 | S2648, p53 | 0.53–0.71 | Unbiased for sequence-only* (−0.99 1D, −0.86 3D) | 5-fold cross-validation separating by protein as in mCSM |
| STRUM [22] | 5-fold CV | Q3421 | S2648, S350, Q306 (subset of S2648) | 0.4–0.8 | Biased* (0.34) | Q306 as test, with sequence identity < 60% |
| ELASPIC [56] | 20‐fold CV and LOO | S3463 (159 proteins) | S2636 (134 proteins), S2104 (79 proteins) | 0.77 | not evaluated | 90% sequence identity redundancy reduction |
| SAAFEC [39] | 5-fold CV | 983 mutations from Protherm (42 proteins) | 983 mutations from Protherm (42 proteins) | 0.61 | not evaluated | not declared |
| MAESTRO(web) [57] | 5/10/20-fold CV | S2648, S1925 (from S1948), S1765, S2244 | S2648, S350 | 0.63–0.76 | Biased* (−0.34) | Lowest correlation (0.63) when separating by protein in 5-fold CV as in mCSM |
| SDM(2) [46] | None | None | S2648, S350, p53, S140 | 0.52–0.63 | Biased* (−0.75) | not declared |
| TML-MP [58] | 5-fold cross validation | S2648 S350, M233 | 2648 S350, M233 | 0.54–0.82 | not evaluated | not declared |
| ThreeFoil [16] | 2-fold CV with 1000 reshuffling | Broom2017 | Broom2017 | 0.73 | not evaluated | not declared |
| DynaMut [59] | 10-fold CV and blind TS | S2648 | S351 | 0.58–0,70 | Biased | Homology included, use in input Duet, trained on the same S2640 |
| DDGun [21] | S2648, Ssym, p53, Myoglobin | Untrained | S2648, Ssym, p53, Myoglobin | 0.45–0.71 (0.54–0.68) | Unbiased* (−0.99) | not applicable |
| DeepDDG [17] | Cao Train | Cao Test | 0.56 | not evaluated | Homologs (>25%) removed from test set | |
| ProTstab [60] | 10-fold CV | Varibench | Varibench | 0.79 | not evaluated | not declared |
Possible biases can be due to anti-symmetry and lack of checking for sequence identity on training/test data. CV = cross-validation, LOO = leave-one-out, TS = test-set, MCC = Matthews Correlation Coefficient, ρ = correlation, ρ_dir-inv = correlation between direct and inverse variation. *Evaluated on Ssym database [11].