TABLE 3.
Accuracy measurements of the NLP system in identifying anaphylaxis regardless of cause, including cases without notes
| Site | Reference standard (n/N) | TP | TN | FN | FP | Sensitivity CI (%) | Specificity CI (%) | PPV CI (%) | NPV CI (%) | F-score CI |
|---|---|---|---|---|---|---|---|---|---|---|
| KPSC | 26/423 | 22 | 388 | 4 | 9 | 84.6 65.1–95.6 |
97.7 95.7–98.9 |
71.0 52.0–85.8 |
99.0 97.4–99.7 |
0.83 0.40–0.99 |
| Site 2 | 4/46 | 3 | 33 | 1 | 9 | 75.0 19.4–99.4 |
78.6 63.2–89.7 |
25.0 5.4–57.2 |
97.1 84.7–99.9 |
0.63 0.09–0.99 |
| Site 3 | 2/104 | 1 | 97 | 1 | 5 | 50.0 1.2–98.7 |
95.1 88.9–98.4 |
16.7 0.4–64.1 |
99.0 94.4–99.9 |
0.42 0.01–0.99 |
| Site 4 | 9/83 | 4 | 69 | 5 | 5 | 44.4 13.7–78.8 |
93.2 84.9–97.8 |
44.4 13.7–78.8 |
93.2 84.9–97.8 |
0.44 0.06–0.90 |
| Site 5 | 5/75 | 2 | 61 | 3 | 9 | 40.0 5.2–85.3 |
87.1 76.9–93.9 |
18.2 2.2–51.8 |
95.3 86.9–99.0 |
0.36 0.01–0.99 |
Note: 95% confidence intervals are displayed beneath point estimates. Reference standard: 2009 to 2011 anaphylaxis cases abstracted and adjudicated as part of McNeil et al study.
Abbreviations: F-score: , with β = 3; FN, false negative—NLP negative, reference standard positive; FP, false positive—NLP positive, reference standard negative; n, number of positive vaccine-related anaphylaxis cases; N, number of cases reviewed; NPV, negative predictive value; TN, true negative—NLP negative, reference standard negative; TP, true positive—NLP positive, reference standard positive; PPV, positive predictive value.