Table 8.
Cross validation of models with 95% confidence intervals and significance tests comparing the performance in the two samples
| Development Dataset | Validation Dataset | P Value for the Difference | |
|---|---|---|---|
| Opioid‐related overdose | |||
| Sensitivity (95% CI) | 97.9 (96.0‐99.0) | 97.2 (95.5‐98.4) | 0.493 |
| Specificity (95% CI) | 88.9 (85.6‐91.6) | 84.6 (81.3‐87.5) | 0.028 |
| PPV (95% CI) | 89.2 (86.4‐91.5) | 87.4 (85.0‐89.3) | 0.3421 |
| NPV (95% CI) | 97.8 (95.9‐98.8) | 96.5 (94.5‐97.8) | 0.205 |
| F‐score | 0.93 | 0.92 | |
| OODa classified as heroin involved | |||
| Sensitivity (95% CI) | 94.0 (83.5‐98.8) | 91.8 (84.6‐96.4) | 0.635 |
| Specificity (95% CI) | 96.8 (94.5‐98.3) | 99.0 (97.7‐99.7) | 0.018 |
| PPV (95% CI) | 79.7 (69.1‐87.3) | 94.7 (88.3‐97.7) | 0.004 |
| NPV (95% CI) | 99.2 (97.6‐99.7) | 98.4 (97.0‐99.2) | 0.330 |
| F‐score | 0.86 | 0.93 | |
| OOD classified as suicide/suicide attempt | |||
| Sensitivity (95% CI) | 77.4 (70.0‐83.7) | 70.5 (63.8‐76.7) | 0.142 |
| Specificity (95% CI) | 88.1 (83.6‐91.7) | 90.2 (86.9‐92.9) | 0.380 |
| PPV (95% CI) | 79.0 (82.8‐84.0) | 78.9 (73.3‐83.6) | 0.995 |
| NPV (95% CI) | 87.1 (83.4‐90.1) | 85.5 (82.6‐87.9) | 0.551 |
| F‐score | 0.78 | 0.74 | |
| OOD classified as abuse involved | |||
| Sensitivity (95% CI) | 82.0 (73.6‐88.6) | 75.3 (68.2‐81.5) | 0.184 |
| Specificity (95% CI) | 83.4 (76.7‐88.9) | 79.5 (73.6‐84.6) | 0.329 |
| PPV (95% CI) | 77.8 (70.9‐83.4) | 74.0 (68.5‐78.9) | 0.462 |
| NPV (95% CI) | 86.8 (81.4‐90.7) | 80.5 (76.0‐84.4) | 0.117 |
| F‐score | 0.80 | 0.75 | |
| Overdose classified as suicide/suicide attempt using NLP‐enhanced algorithm | |||
| Sensitivity (95% CI) | 88.4 (82.3‐93.0) | 78.7 (72.5‐84.1) | 0.016 |
| Specificity (95% CI) | 91.8 (87.8‐94.8) | 91.0 (87.7‐93.6) | 0.707 |
| PPV (95% CI) | 86.2 (80.6‐90.3) | 81.9 (76.7‐86.2) | 0.278 |
| NPV (95% CI) | 93.2 (89.8‐95.5) | 89.2 (86.3‐91.5) | 0.079 |
| F‐score | 0.87 | 0.80 | |
| Overdose classified as involving abuse using NLP‐enhanced algorithm | |||
| Sensitivity (95% CI) | 90.1 (83.0‐95.0) | 80.5 (73.8‐86.1) | 0.030 |
| Specificity (95% CI) | 79.6 (72.5‐85.6) | 76.3 (70.2‐81.8) | 0.449 |
| PPV (95% CI) | 75.8 (69.5‐81.1) | 72.5 (67.4‐77.2) | 0.517 |
| NPV (95% CI) | 91.9 (86.6‐95.2) | 83.4 (78.7‐87.3) | 0.023 |
| F‐score | 0.82 | 0.76 | |
Note. NLP: natural language processing; OOD: opioid‐related overdose.
a
Opioid‐related overdose.