Table 3.

Evaluation of NLP application using development dataset

Development Dataset (KPNW) N = 1006
NLP‐only Classification		na	Performance Compared with Gold Standard Chart Review na = 627
			Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Event type, irrespective of substance involved	Intentional overdose	74	.81 (.70‐.89)	.98 (.96‐.99)	.83 (.72‐.91)	.97 (.96‐.99)
	Unintentional overdose	158	.71 (.63‐.78)	.94 (.91‐.96)	.79 (.71‐.85)	.91 (.87‐.93)
	Overdose of any type	232	.80 (.74‐.85)	.93 (.90‐.95)	.87 (.81‐.91)	.89 (.85‐.92)
	Adverse drug reaction—ADR	79	.24 (.15‐.35)	.93 (.90‐.95)	.32 (.21‐.45)	.89 (.87‐.92)
Substance involved in overdose or ADR	Heroin	19	.84 (.60‐.96)	1.0 (.99‐1.0)	.80 (.56‐.93)	1.0 (.99‐1.0)
	Opioid only (excludes heroin)	98	.37 (.27‐.47)	.94 (.92‐.95)	.39 (.29‐.50)	.93 (.91‐.95)
	Polysubstance including opioid (excludes heroin)	175	.57 (.49‐.64)	.96 (.94‐.97)	.75 (.67‐.82)	.91 (.89‐.93)
	Any opioid (excludes heroin, includes polysubstance)	273	.72 (.66‐.77)	.96 (.94‐.97)	.88 (.83‐.92)	.90 (.87‐.92)
Substance abuse involved in opioid‐related overdose	Prescription med abuse (whether prescribed or not)	56	.38 (.25‐.51)	.96 (.94‐.98)	.50 (.34‐.66)	.94 (.92‐.96)
	Substance abuse (including alcohol abuse or presence)	45	.62 (.47‐.76)	.97 (.95‐.98)	.58 (.43‐.72)	.97 (.95‐.98)
	Illicit drug abuse	84	.46 (.36‐.58)	.97 (.97‐.98)	.68 (.55‐.80)	.92 (.90‐.94)
	Any substance abuse	129	.67 (.59‐.75)	.96 (.94‐.97)	.81 (.72‐.87)	.92 (.89‐.94)
Patient error in opioid‐related overdose or ADR	Patient error	21	.33 (.15‐.57)	.98 (.96‐.99)	.33 (.15‐.57)	.98 (.96‐.99)

Abbreviations: KPNW, Kaiser Permanente Northwest; NLP, natural language processing; NPV, negative predictive value; PPV, positive predictive value.

^an is the count as determined by gold standard chart review, includes only events for which EHR source data were available.