TABLE 6.
Cross-validated area under the receiver-operating curve (95% CI) for all algorithms based on Atrius 2016 data
| Algorithm | cv-AUC | Algorithm | cv-AUC | Algorithm | cv-AUC |
|---|---|---|---|---|---|
| ridge10,dev | 0.915(0.884, 0.946) | xgb50,4,25 | 0.862(0.824, 0.899) | xgb20,4,10 | 0.805(0.762, 0.847) |
| ridge20,dev | 0.909(0.877, 0.941) | xgb20,2,10 | 0.858(0.820, 0.896) | lasso20,pre,dev | 0.803(0.761, 0.846) |
| lasso20,dev | 0.905(0.873, 0.938) | glmStep50 | 0.857(0.818, 0.895) | lasso10,pre,dev | 0.801(0.758, 0.844) |
| lasso20,auc | 0.898(0.865, 0.932) | nn50,5 | 0.855(0.817, 0.893) | SL42 | 0.800(0.758, 0.843) |
| xgb20,4,50 | 0.895(0.861, 0.929) | xgb10,2,10 | 0.855(0.816, 0.893) | nn20,2 | 0.799(0.756, 0.841) |
| xgb20,2,50 | 0.895(0.861, 0.929) | glm10,pre | 0.848(0.809, 0.887) | glm20,pre,wt | 0.798(0.755, 0.841) |
| ridge20,auc | 0.895(0.861, 0.929) | glmStep50,pre | 0.843(0.803, 0.882) | lasso20,pre,auc | 0.790(0.747, 0.834) |
| xgb50,2,50 | 0.893(0.859, 0.927) | glm20 | 0.839(0.799, 0.879) | svm50,pre | 0.783(0.739, 0.827) |
| xgb50,4,50 | 0.893(0.859, 0.927) | glm10,pre,wt | 0.837(0.797, 0.877) | glm20,wt | 0.747(0.702, 0.793) |
| lasso10,dev | 0.892(0.858, 0.926) | glm50,pre,wt | 0.836(0.796, 0.876) | glm10 | 0.739(0.693, 0.785) |
| xgb10,2,50 | 0.890(0.856, 0.925) | glm50,pre | 0.834(0.794, 0.875) | xgb10,4,10 | 0.735(0.689, 0.781) |
| lasso10,auc | 0.890(0.856, 0.924) | SL.xgboost | 0.825(0.784, 0.866) | glm20,pre | 0.692(0.644, 0.739) |
| xgb10,4,50 | 0.890(0.855, 0.924) | ridge20,pre,auc | 0.823(0.782, 0.864) | svm50 | 0.680(0.632, 0.728) |
| ridge10,auc | 0.888(0.853, 0.922) | ridge10,pre,auc | 0.819(0.777, 0.860) | rForest10 | 0.664(0.616, 0.712) |
| xgb50,2,25 | 0.882(0.847, 0.917) | nn10,2 | 0.819(0.777, 0.860) | rForest20 | 0.657(0.609, 0.706) |
| xgb20,2,25 | 0.880(0.845, 0.916) | glm50 | 0.818(0.777, 0.860) | rForest20,pre | 0.643(0.594, 0.691) |
| xgb10,4,25 | 0.878(0.842, 0.914) | nn10,1 | 0.817(0.776, 0.859) | rForest10,pre | 0.626(0.578, 0.674) |
| SL69 | 0.876(0.839, 0.912) | nn20,5 | 0.817(0.776, 0.859) | rForest50 | 0.569(0.521, 0.617) |
| nn50,2 | 0.873(0.836, 0.909) | nn20,1 | 0.817(0.775, 0.858) | rForest50,pre | 0.569(0.520, 0.617) |
| xgb20,4,25 | 0.870(0.833, 0.907) | lasso10,pre,auc | 0.816(0.775, 0.858) | glm10,wt | 0.515(0.468, 0.562) |
| xgb50,2,10 | 0.867(0.830, 0.904) | ridge20,pre,dev | 0.815(0.773, 0.856) | nnet10,5h,pre | 0.500(0.453, 0.547) |
| nn50,1 | 0.864(0.827, 0.902) | glm50,wt | 0.812(0.770, 0.854) | nnet10,10h,pre | 0.500(0.453, 0.547) |
| xgb10,2,25 | 0.862(0.825, 0.900) | nn10,5 | 0.810(0.768, 0.852) | nnet20,5h,pre | 0.500(0.453, 0.547) |
| ridge10,pre,dev | 0.808(0.766, 0.850) | nnet20,10h,pre | 0.500(0.453, 0.547) |
Subscript key: (50, 20,10): # controls per case, pre: 23 pre-selected covariates, auc: neg AUC loss.
dev: deviance loss, wt: weighted regression, 5h, 10h: # nodes in hidden layer.
xgb subscript key: (a, b, c): a = # controls per case, b=depth, c = min obs per node.
nn subscript key: (a, b): a = # controls per case, b= # nodes in hidden layer.
95% confidence interval method calculated using Hanley’s method.48 Method of LeDell et al (2015) is not applicable to single external validation set.