Table 3.
The mean (95% bootstrap confidence interval) of the optimism-adjusted performance results in the MIMIC-III example for each estimation method. Main study corresponds to model fitting on the whole MIMIC-III dataset, while subset corresponds to the sub-analysis on the minimum required sample size.
| Study | Model | Calibration-in-the-large | Calibration slope | AUC | Brier score |
|---|---|---|---|---|---|
| Main | MLE | 0.00 (−0.04, 0.04) | 0.99 (0.94, 1.03) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Uniform closed-form | 0.00 (−0.04, 0.04) | 1.00 (0.95, 1.04) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Uniform bootstrap | 0.00 (−0.04, 0.04) | 1.00 (0.96, 1.05) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Firths | 0.00 (−0.04, 0.04) | 0.99 (0.95, 1.03) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | LASSO | 0.00 (−0.04, 0.04) | 0.99 (0.95, 1.04) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Repeat CV LASSO | 0.00 (−0.04, 0.04) | 0.99 (0.95, 1.04) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Ridge | 0.00 (−0.04, 0.04) | 1.05 (1.00, 1.10) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | Repeat CV ridge | 0.00 (−0.04, 0.04) | 1.05 (1.00, 1.10) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.09) |
| Main | LASSO1SE | 0.00 (−0.04, 0.04) | 1.21 (1.15, 1.27) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.10) |
| Main | Ridge1SE | 0.00 (−0.04, 0.04) | 1.26 (1.20, 1.32) | 0.74 (0.73, 0.75) | 0.09 (0.09, 0.10) |
| Subset | MLE | −0.01 (−0.14, 0.12) | 0.90 (0.74, 1.05) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Uniform closed-form | −0.01 (−0.13, 0.11) | 0.99 (0.82, 1.16) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Uniform bootstrap | −0.01 (−0.13, 0.11) | 1.02 (0.84, 1.19) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Firths | −0.01 (−0.13, 0.11) | 0.92 (0.78, 1.06) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | LASSO | −0.01 (−0.13, 0.11) | 0.98 (0.83, 1.12) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Repeat CV LASSO | −0.01 (−0.13, 0.11) | 0.99 (0.84, 1.14) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Ridge | −0.01 (−0.13, 0.11) | 1.06 (0.91, 1.21) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | Repeat CV ridge | −0.01 (−0.13, 0.11) | 1.04 (0.89, 1.19) | 0.74 (0.71, 0.76) | 0.10 (0.09, 0.10) |
| Subset | LASSO1SE | −0.01 (−0.12, 0.10) | 1.40 (1.14, 1.67) | 0.73 (0.71, 0.76) | 0.10 (0.09, 0.11) |
| Subset | Ridge1SE | −0.01 (−0.12, 0.10) | 1.83 (1.53, 2.13) | 0.74 (0.71, 0.77) | 0.10 (0.09, 0.11) |