Table 2.
Performance indicators of machine learning and statistical models in overall cohort
| Model | Model performance*: C statistic (95% range†) | Average absolute change in model performance: % (95% range†) |
|---|---|---|
| Logistic (Caret) | 0.879 (0.879 to 0.879) | 0.00 (−0.03 to 0.04) |
| Random forest (Caret) | 0.869 (0.867 to 0.869) | −1.20 (−1.33 to −1.10%) |
| Neural network (Caret) | 0.878 (0.867 to 0.880) | −0.15 (−1.35 to 0.06) |
| Statistic logistic model | 0.879 (0.879 to 0.879) | 0.01 (−0.02 to 0.04) |
| QRISK3 | 0.879 | Reference model |
| Framingham | 0.865 | −1.66 (−1.66 to −1.66) |
| Local Cox model | 0.877 (0.877 to 0.878) | −0.22 (−0.28 to −0.17) |
| Parametric survival model (Weibull) | 0.877 (0.876 to 0.877) | −0.29 (−0.35 to −0.24) |
| Parametric survival model (Gaussian) | 0.876 (0.876 to 0.877) | −0.33 (−0.39 to −0.29) |
| Parametric survival model (Logistic) | 0.876 (0.875 to 0.876) | −0.36 (−0.43 to −0.31) |
| Logistic (Sklearn) | 0.879 (0.879 to 0.879) | 0.00 (−0.05 to 0.03) |
| Random forest (Sklearn) | 0.872 (0.871 to 0.873) | −0.80 (−0.89 to −0.71) |
| Neural network (Sklearn) | 0.872 (0.832 to 0.879) | −0.85 (−5.39 to −0.03) |
| Gradient boosting classifier (Sklearn) | 0.878 (0.877 to 0.878) | −0.17 (−0.29 to −0.08) |
| extra-trees (Sklearn) | 0.863 (0.861 to 0.864) | −1.89 (−2.05 to −1.76) |
| Logistic (h2o) | 0.879 (0.878 to 0.879) | −0.06 (−0.10 to −0.02) |
| Random forest (h2o) | 0.877 (0.877 to 0.878) | −0.22 (−0.29 to −0.17) |
| Neural network (h2o) | 0.875 (0.870 to 0.879) | −0.45 (−1.09 to −0.04) |
| autoML (h2o) | 0.879 (0.879 to 0.880) | −0.00 (−0.07 to 0.06) |
*
Model performance was calculated in binary framework. Threshold 7.5% was used to calculate precision and recall for all models.
†
95% range (2.5-97.5%) of model performance derived from 100 random samples.