Table 5.
Evaluation of NLP application using portability dataset
| Portability dataset (KPW), n = 435 | ||||||
|---|---|---|---|---|---|---|
| NLP‐only Classification | Performance Compared with Gold Standard Chart Review na = 305 | |||||
| na | Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | NPV (95% CI) | ||
| Event type, irrespective of substance involved | Intentional overdose | 31 | .65 (.45‐.80) | .99 (.96‐1.0) | .83 (.62‐.95) | .96 (.93‐.98) |
| Unintentional overdose | 43 | .53 (.38‐.69) | .96 (.92‐.98) | .68 (.49‐.82) | .93 (.89‐.95) | |
| Overdose of any type | 74 | .68 (.56‐.78) | .97 (.93‐.98) | .86 (.74‐.93) | .90 (.86‐.94) | |
| Adverse drug reaction—ADR | 5 | 0.0 (0.0‐.54) | .97 (.94‐.98) | 0.0 (0.0‐.34) | .98 (.96‐.99) | |
| Substance involved in overdose or ADR | Heroin | 5 | .20 (.01‐.70) | .99 (.97‐1.0) | .33 (.02‐.87) | .99 (.96‐1.0) |
| Opioid only (excludes heroin) | 26 | .38 (.21‐.59) | .94 (.90‐.96) | .37 (.20‐.58) | .94 (.91‐.97) | |
| Polysubstance including opioid (excludes heroin) | 42 | .48 (.32‐.63) | .97 (.94‐.99) | .71 (.51‐.86) | .92 (.88‐.95) | |
| Any opioid (excludes heroin, includes polysubstance) | 68 | .69 (.57‐.79) | .97 (.93‐.98) | .85 (.88‐.95) | .92 (.88‐.95) | |
| Substance abuse involved in opioid‐related overdose | Prescription med abuse (whether prescribed or not) | 18 | .28 (.11‐.54) | .98 (.95‐.99) | .45 (.18‐.75) | .96 (.92‐.98) |
| Substance abuse (including alcohol abuse or presence) | 18 | .44 (.22‐.69) | .98 (.95‐.99) | .53 (.27‐.78) | .97 (.94‐.98) | |
| Illicit drug abuse | 25 | .28 (.13‐.50) | .99 (.97‐1.0) | .70 (.35‐.92) | .94 (.90‐.96) | |
| Any substance abuse | 44 | .52 (.37‐.67) | .98 (.95‐.99) | .82 (.62‐.93) | .92 (.88‐.95) | |
| Patient error in opioid‐related overdose or ADR | Patient error | 5 | 0.0 (0.0‐.54) | .99 (.97‐1.0) | 0.0 (0.0‐.80) | .98 (.96‐.99) |
Note. Bold values indicate measures that are significantly different (P < .05, chi‐square test) from results using the development dataset (see Table 3).
Abbreviations: KPNW, Kaiser Permanente Northwest; NLP, natural language processing; NPV, negative predictive value; PPV, positive predictive value.
a
n is the count as determined by gold standard chart review, includes only events for which EHR source data were available.