TABLE 2.

Accuracy measurements of the NLP system in identifying vaccine-related anaphylaxis after removing cases without notes

Site	Reference standard (n/N)	TP	TN	FN	FP	Sensitivity CI (%)	Specificity CI (%)	PPV CI (%)	NPV CI (%)	F-score CI
KPSC	14/421	12	401	2	6	85.7 57.2–98.2	98.5 96.8–99.5	66.7 41.0–86.7	99.5 98.2–99.9	0.83 0.98–1.00
Site 2	1/46	0	42	1	3	0.0	93.3 81.7–98.6	0.0	97.7 87.7–99.9	0.00
Site 3	0/104	0	102	0	2	NA	98.1 93.2–99.8	0.0	100.0 97.1–100	NA
Site 4	1/75	0	72	1	2	0.0	97.3 90.6–99.7	0.0	98.6 92.6–99.9	0.00
Site 5	1/67	0	62	1	4	0.0	93.9 85.2–98.3	0.0	98.4 91.5–99.9	0.00

Note: 95% confidence intervals are displayed beneath point estimates. Reference standard: 2009 to 2011 anaphylaxis cases abstracted and adjudicated as part of McNeil et al study.

Abbreviations: F-score: $F=\frac{\left({\beta }^2+1\right)\ast PPV\ast \text{sensitivity}}{{\beta }^2\ast PPV+\text{sensitivity}}$, with β = 3; FN, false negative—NLP negative, reference standard positive; FP, false positive—NLP positive, reference standard negative; n, number of positive vaccine-related anaphylaxis cases; N, number of cases reviewed; NPV, negative predictive value; TN, true negative—NLP negative, reference standard negative; TP, true positive—NLP positive, reference standard positive; PPV, positive predictive value.