Table 1.
Performance evaluation based on ClinVar benchmark data
| Methods | Missing | PPV (%) | NPV (%) | Specificity (%) | FPR (%) | Sensitivity (%) | FNR (%) | Accuracy (%) | MCC | AUC | hser-AUC | hspr-AUC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Class one: function prediction methods | ||||||||||||
| FATHMM | 195 | 63.48 | 67.35 | 77.84 | 22.16 | 50.52 | 49.48 | 66.02 | 0.296 | 0.694 | 0.536 | 0.586 |
| fitCons | 299 | 51.61 | 64.47 | 62.23 | 37.77 | 54.01 | 45.99 | 58.72 | 0.162 | 0.611 | 0.505 | 0.509 |
| LRT | 995 | 72.28 | 87.52 | 67.63 | 32.37 | 89.75 | 10.25 | 78.35 | 0.586 | 0.789 | 0.631 | 0.521 |
| MutationAssessor | 357 | 68.42 | 83.15 | 69.66 | 30.34 | 82.32 | 17.68 | 75.28 | 0.518 | 0.850 | 0.584 | 0.668 |
| MutationTaster | 34 | 62.01 | 94.93 | 55.56 | 44.44 | 96.07 | 3.93 | 72.99 | 0.542 | 0.610 | 0.560 | 0.504 |
| PolyPhen2-HDIV | 52 | 61.84 | 90.26 | 56.58 | 43.42 | 92.02 | 7.98 | 71.93 | 0.503 | 0.839 | 0.644 | 0.551 |
| PolyPhen2-HVAR | 52 | 68.50 | 88.47 | 68.97 | 31.03 | 88.24 | 11.76 | 77.32 | 0.571 | 0.865 | 0.657 | 0.627 |
| PROVEAN | 134 | 73.03 | 83.45 | 76.99 | 23.01 | 80.32 | 19.68 | 78.45 | 0.569 | 0.858 | 0.617 | 0.628 |
| SIFT | 190 | 67.11 | 86.78 | 67.90 | 32.10 | 86.36 | 13.64 | 75.86 | 0.541 | 0.860 | 0.633 | 0.597 |
| VEST3 | 16 | 77.95 | 90.33 | 81.03 | 18.97 | 88.55 | 11.45 | 84.27 | 0.689 | 0.929 | 0.729 | 0.756 |
| Class two: conservation methods | ||||||||||||
| GERP++ | 16 | 52.48 | 91.12 | 34.80 | 65.20 | 95.50 | 4.50 | 60.90 | 0.364 | 0.739 | 0.598 | 0.507 |
| phastCons | 0 | 63.60 | 87.58 | 61.86 | 38.14 | 88.37 | 11.63 | 73.26 | 0.507 | 0.767 | 0.642 | 0.517 |
| phyloP | 0 | 61.13 | 88.96 | 56.51 | 43.49 | 90.71 | 9.29 | 71.21 | 0.486 | 0.848 | 0.619 | 0.594 |
| SiPhy | 17 | 63.65 | 82.12 | 64.75 | 35.25 | 81.40 | 18.60 | 71.93 | 0.460 | 0.776 | 0.625 | 0.515 |
| Class three: ensemble methods | ||||||||||||
| CADD | 1 | 63.96 | 93.19 | 59.99 | 40.01 | 94.18 | 5.82 | 74.69 | 0.556 | 0.877 | 0.683 | 0.584 |
| DANN | 1 | 64.47 | 83.32 | 65.71 | 34.29 | 82.55 | 17.45 | 72.95 | 0.480 | 0.807 | 0.649 | 0.544 |
| Eigen | 299 | 65.05 | 91.97 | 62.88 | 37.12 | 92.64 | 7.36 | 75.59 | 0.563 | 0.871 | 0.669 | 0.595 |
| FATHMM-MKL | 1 | 56.54 | 93.70 | 44.36 | 55.64 | 96.04 | 3.96 | 66.57 | 0.451 | 0.822 | 0.663 | 0.552 |
| GenoCanyon | 0 | 56.10 | 80.99 | 50.22 | 49.78 | 84.37 | 15.63 | 64.90 | 0.358 | 0.683 | 0.561 | 0.509 |
| M-CAP | 2141 | 80.51 | 60.64 | 35.00 | 65.00 | 92.20 | 7.80 | 77.58 | 0.335 | 0.814 | 0.562 | 0.622 |
| MetaLR | 31 | 80.28 | 75.98 | 88.21 | 11.79 | 63.24 | 36.76 | 77.44 | 0.538 | 0.874 | 0.703 | 0.670 |
| MetaSVM | 31 | 83.03 | 77.09 | 89.95 | 10.05 | 64.77 | 35.23 | 79.09 | 0.574 | 0.858 | 0.547 | 0.693 |
| REVEL | 30 | 81.12 | 87.07 | 85.27 | 14.73 | 83.33 | 16.67 | 84.43 | 0.684 | 0.920 | 0.689 | 0.756 |
AUC, area under the curve; FNR, false negative rate; FPR, false positive rate; hser-AUC, high-sensitivity regional area under the curve; hspr-AUC, high-specificity regional area under the curve; MCC, Mathew correlation coefficient; NPV, negative predictive value; PPV, positive predictive value. ClinVar is an open database that aggregates clinically observed genetic variants. We obtained 4880 missense variants from this database, including 2098 pathogenic variants and 2782 benign variants.