Table 5.
Prediction performance in entire validation data for suicide attempt in the 30 days following an outpatient visit; 95% confidence intervals (CIs) constructed using 10,000 bootstrap samples.
| Prediction model | AUC (95% CI) | Brier score (95% CI) | F-score of 99th percentile (95% CI) | Sensitivity of 99th percentile (95% CI) | Specificity of 99th percentile (95% CI) | PPV of 99th percentile (95% CI) |
|---|---|---|---|---|---|---|
| 30-day suicide attempt following mental health specialty visits. | ||||||
| OP |
0.857 (0.853, 0.860) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.078 (0.075, 0.082) |
0.183 (0.176, 0.190) |
0.990 (0.990, 0.991) |
0.050 (0.048, 0.052) |
| LR‡ |
0.858 (0.855, 0.862) |
2.7 × 10−3 (2.7,2.8) x10−3 |
0.081 (0.078, 0.084) |
0.224 (0.217, 0.232) |
0.988 (0.988, 0.988) |
0.049 (0.048, 0.051) |
| RF |
0.855 (0.853, 0.859) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.084 (0.081, 0.087) |
0.201 (0.195, 0.208) |
0.990 (0.990, 0.990) |
0.053 (0.051, 0.055) |
| ANN |
0.860 (0.857, 0.863) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.088 (0.085, 0.092) |
0.207 (0.200, 0.214) |
0.990 (0.990, 0.991) |
0.056 (0.054, 0.058) |
| Ensemble: LR/RF |
0.866 (0.863, 0.869) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.087 (0.084, 0.090) |
0.227 (0.220, 0.235) |
0.989 (0.989, 0.989) |
0.054 (0.052, 0.055) |
| Ensemble: RF/ANN |
0.866 (0.863, 0.869) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.089 (0.086, 0.092) |
0.210 (0.203, 0.217) |
0.990 (0.990, 0.991) |
0.057 (0.055, 0.059) |
| Ensemble: LR/ANN |
0.863 (0.862, 0.866) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.086 (0.083, 0.089) |
0.225 (0.218, 0.232) |
0.989 (0.989, 0.989) |
0.053 (0.052, 0.055) |
| Ensemble: LR/RF/ANN |
0.867 (0.864, 0.870) |
2.7 × 10−3 (2.6,2.7) x10−3 |
0.088 (0.085, 0.091) |
0.225 (0.218, 0.232) |
0.989 (0.989, 0.990) |
0.055 (0.053, 0.057) |
| 30-day suicide attempt following mental health visits to a general medical provider. | ||||||
| OP | 0.842 (0.836, 0.849) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.060 (0.057, 0.063) | 0.241 (0.228, 0.253) |
0.990 (0.990, 0.990) |
0.034 (0.032, 0.036) |
| LR |
0.839 (0.832, 0.845) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.058 (0.055, 0.061) |
0.263 (0.251, 0.276) |
0.989 (0.988, 0.989) |
0.033 (0.031, 0.035) |
| RF |
0.838 (0.832, 0.845) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.058 (0.055, 0.061) |
0.247 (0.234, 0.259) |
0.989 (0.989, 0.990) |
0.033 (0.031, 0.035) |
| ANN |
0.843 (0.836, 0.849) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.066 (0.062, 0.069) |
0.259 (0.247, 0.272) |
0.990 (0.990, 0.990) |
0.038 (0.035, 0.040) |
| Ensemble: LR/RF |
0.847 (0.841, 0.853) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.060 (0.056, 0.063) |
0.258 (0.246, 0.271) |
0.989 (0.989, 0.989) |
0.034 (0.032, 0.036) |
| Ensemble: RF/ANN |
0.846 (0.840, 0.852) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.064 (0.060, 0.067) |
0.261 (0.249, 0.274) |
0.990 (0.990, 0.990) |
0.036 (0.034, 0.038) |
| Ensemble: LR/ANN |
0.844 (0.838, 0.851) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.063 (0.060, 0.066) |
0.265 (0.252, 0.278) |
0.989 (0.989, 0.990) |
0.036 (0.034, 0.038) |
| Ensemble: LR/RF/ANN |
0.848 (0.842, 0.854) |
1.4 × 10−3 (1.4,1.5) x10−3 |
0.062 (0.059, 0.066) |
0.262 (0.249, 0.275) |
0.990 (0.989, 0.990) |
0.035 (0.033, 0.037) |
AUC area under the receiver operating curve, PPV positive predicted value, OP original parsimonious, LR logistic regression with lasso variable selection, RF random forest, ANN artificial neural network.