TABLE 1.
Performance of VREfm prediction models in terms of k-fold CV, timewise validation, and external validation
| Evaluation metrics | Machine learning models: |
||
|---|---|---|---|
| RF model | SVM model | KNN model | |
| AUROC (95% CI) | |||
| 5-fold CV | 0.8495 (0.8397–0.8594) | 0.8367 (0.8264–0.8471) | 0.7908 (0.7792–0.8024) |
| 10-fold CV | 0.8491 (0.8392–0.8589) | 0.8338 (0.8234–0.8442) | 0.7589 (0.7468–0.7710) |
| Timewise validation | 0.8463 (0.8273–0.8654) | 0.8368 (0.8169–0.8566) | 0.7908 (0.7690–0.8127) |
| External validation | 0.8553 (0.8399–0.8706) | 0.8407 (0.8246–0.8569) | 0.8050 (0.7872–0.8227) |
| Accuracy (95% CI) | |||
| 5-fold CV | 0.7769 (0.7660–0.7878) | 0.7610 (0.7499–0.7721) | 0.7248 (0.7131–0.7364) |
| 10-fold CV | 0.7789 (0.7608–0.7827) | 0.7587 (0.7476–0.7699) | 0.6906 (0.6786–0.7027) |
| Timewise validation | 0.7840 (0.7640–0.8039) | 0.7815 (0.7615–0.8016) | 0.7228 (0.7011–0.7445) |
| External validation | 0.7855 (0.7687–0.8024) | 0.7781 (0.7610–0.7951) | 0.7355 (0.7174–0.7536) |
| Sensitivity (95% CI) | |||
| 5-fold CV | 0.8054 (0.7951–0.8517) | 0.7826 (0.7719–0.7934) | 0.7873 (0.7767–0.7980) |
| 10-fold CV | 0.7863 (0.7756–0.7969) | 0.8192 (0.8091–0.8292) | 0.7096 (0.6978–0.7214) |
| Timewise validation | 0.8153 (0.7965–0.8341) | 0.8415 (0.8238–0.8592) | 0.7491 (0.7281–0.7702) |
| External validation | 0.7791 (0.7620–0.7961) | 0.7954 (0.7789–0.8120) | 0.8044 (0.7881–0.8207) |
| Specificity (95% CI) | |||
| 5-fold CV | 0.7497 (0.7384–0.7609) | 0.7403 (0.7289–0.7517) | 0.6649 (0.6526–0.6772) |
| 10-fold CV | 0.7789 (0.7680–0.7897) | 0.7009 (0.6890–0.7128) | 0.6725 (0.6603–0.6848) |
| Timewise validation | 0.7477 (0.7266–0.7688) | 0.7120 (0.6900–0.7340) | 0.6922 (0.6698–0.7146) |
| External validation | 0.7930 (0.7764–0.8096) | 0.7580 (0.7405–0.7756) | 0.6560 (0.6365–0.6755) |