Table 3.
Performance characteristics of automated surveillance systems vs reference standards
| References | Patient population | Number of patients | nvHAP/HAP | Algorithm ID number. a | Reference type ID numberb | Manual part of surveillance | Reference method | Sensitivity | Specificity | Positive predictive value | Negative predictive value |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 1 |
CDC-NHS PNU1 both reviewers one reviewer |
– | – |
42% (39–46) 67% (64–70) |
– | |
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 2 |
Clinical nvHAP (treating physician) both reviewers one reviewer |
– | – |
42% (39–46) 60% (57–63) |
– | |
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 6 |
Presence of clinical deterioration both reviewers one reviewer |
– | – |
87% (85–89) 98% (98–99) |
– | |
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 4 |
Discharge summary nvHAP both reviewers one reviewer |
– | – |
35% (32–38) 49% (46–52) |
– | |
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 5 |
Expert reviewer nvHAP both reviewers one reviewer |
– | – |
50% (46–53) 71% (68–74) |
– | |
| Stern [22] | Patients who met surveillance criteria | 250 | nvHAP | 1 | 7 | VA claims based definition of nvHAP | – | – | 7.9% (4–12) | – | |
| Ji [23] | Patients with impaired oxygenation | 120 | nvHAP | 8 | 2 | Clinical nvHAP (treating physician) | 56% (40–70) | 77% (68–86) | 60% (47–71) | – | |
| Jones [25] | Patients who met surveillance criteria | 250 | nvHAP | 8 | 2 or 4 or 5 | Clinical nvHAP (treating physician) OR discharge summary nvHAP OR expert reviewer | – | – | 81% | – | |
| Ramirez-Batlle [24] | Patients with impaired oxygenation | 120 | nvHAP | 10 | 1 | CDC-NHSN PNEU | 59% (39–77) | 73% (62–81) | 41% (30–52) | 85% (78–90) | |
| Ramirez-Batlle [24] | Patients with impaired oxygenation | 120 | nvHAP | 10 | 2 | Clinical nvHAP (treating physician) | 56% (40–70) | 77% (66–86) | 60% (47–71) | 74% (67–80 | |
| Ramirez-Batlle [24] | Patients with impaired oxygenation | 120 | nvHAP | 10 | 3 | “True pneumonia” (clinical plus AB-treatment) | 71% (51–87) | 76% (66–84) | 48% (37–58) | 90% (83–94) | |
| Valentine [28] | Patients with hemato-oncolic disease | 151 | HAP post-EMR:34% pre-EMR:13% | 12 | 1 | CDC-NHSN PNU3 | – | – | 18% (12–25) | ||
| Wolfensberger [21] | Random sample plus patients with ICD10 | 2091 | nvHAP | 15 | 1 | x | ECDC nvHAP | 98% (93–99) or higher | (100% not mentioned in the paper, but per definition) | (100% not mentioned in the paper, but per definition) | 99% (98–100) or higher |
| Wolfensberger [27] | Patients from 3 surgical and 1 medical department | 747 (590 random sample, 157 with ICD10) | HAP | 16 | 1 | HELICS/ECDC nvHAP | 59% (48–69) (extrapolated) | 98% (98–99) (extrapolated) | 35% (27–42) | 99% (98–100) | |
| FitzHenry [31] | Procedures with surgical procedures | 7743 patients with 8186 procedures (4098 in development and 4088 in validation cohort) | HAP (postoperative pneumonia) | 17 | 1 | VASQIP | 79% (no CI) | – | – | – | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 18 | 1 | HELICS | 99% (98–100) | 58% (56–61) | 22% (19–25) | 100% (99–100) | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 19 | 1 | HELICS | 86% (81–91) | 89% (88–91) | 49% (43–55) | 98% (97–99) | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 20 | 1 | HELICS | 93% (89–97) | 64% (61–66) | 23% (20–27) | 99% (98–99) | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 21 | 1 | HELICS | 40% (33–48) | 92% (90–93) | 36% (29–43) | 93% (91–94) | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 22 | 1 | HELICS | 99% (97–100) | 61% (58–63) | 23% (20–26) | 100% (99–100) | |
| Bouzbid [29] | Patients > 48 h on ICU | 1499 | HAP (on ICU) | 23 | 1 | HELICS | 81% (74–86) | 92% (91–94) | 55% (48–61) | 98% (97–98) | |
| Haas [19] and Mendonca [20] | NICU | 1688 or 1692 | HAP (on Neo) | 24 | 1 | NNIS-criteria | 71% (no CI) | 95% (no CI) | 8% (no CI) | 99.8% (no CI) |
CDC-NHSN, Centre for Disease Control and Prevention—National Healthcare Safety Network; CI, confidence interval; ECDC, European Centre for Disease Prevention and Control; HAP, hospital-acquired pneumonia; HELICS, Hospitals in Europe Link for Infection Control through Surveillance; ICU, intensive care unit; ID, identification; NNIS, National Nosocomial Infections Surveillance; nvHAP, non-ventilator-associated hospital-acquired pneumonia; Neo, neonatology; PNEU, pneumonia; VASQIP, Veterans Affairs Surgical Quality Improvement Program
aAlgorithm ID number: please refer to Table 2 to see the components of the specific algorithm
bReference Type ID numbers: 1, well established surveillance definitions from CDC, ECDC, HELICS, NNIS or VASQIP; 2, clinical HAP diagnosed by treating physician; 3, “true pneumonia” based on clinical diagnosis and antibiotic treatment; 4, HAP according to discharge summary; 5, HAP according to expert reviewer; 6, presence of clinical deterioration; 7, claims based HAP