Automated Surveillance for Ventilator-Associated Events

Abstract

BACKGROUND:

The US Centers for Disease Control and Prevention has implemented a new, multitiered definition for ventilator-associated events (VAEs) to replace their former definition of ventilator-associated pneumonia (VAP). We hypothesized that the new definition could be implemented in an automated, efficient, and reliable manner using the electronic health record and that the new definition would identify different patients than those identified under the previous definition.

METHODS:

We conducted a retrospective cohort analysis using an automated algorithm to analyze all patients admitted to the ICU at a single urban, tertiary-care hospital from 2008 to 2013.

RESULTS:

We identified 26,466 consecutive admissions to the ICU, 10,998 (42%) of whom were mechanically ventilated and 675 (3%) of whom were identified as having any VAE. Any VAE was associated with an adjusted increased risk of death (OR, 1.91; 95% CI, 1.53-2.37; P < .0001). The automated algorithm was reliable (sensitivity of 93.5%, 95% CI, 77.2%-98.8%; specificity of 100%, 95% CI, 98.8%-100% vs a human abstractor). Comparison of patients with a VAE and with the former VAP definition yielded little agreement (κ = 0.06).

CONCLUSIONS:

A fully automated method of identifying VAEs is efficient and reliable within a single institution. Although VAEs are strongly associated with worse patient outcomes, additional research is required to evaluate whether and which interventions can successfully prevent VAEs.

In 2013, the US Centers for Disease Control and Prevention (CDC) put forth new definitions for ventilator-associated events (VAEs), the result of a collaboration of the Critical Care Societies Collaborative, the American Association for Respiratory Care, the Association of Professionals in Infection Control and Epidemiology, the Council of State and Territorial Epidemiologists, the Healthcare Infection Control Practices Advisory Committee, the Infectious Diseases Society of America, and the Society for Healthcare Epidemiology of America, among others.¹ The new definition replaced the previous National Healthcare Safety Network (NHSN) definitions for ventilator-associated pneumonia (VAP) in adults.

The previous definitions were criticized for their lack of reliability and validity^2‐6 primarily because of the subjective nature of several of the necessary elements, such as “change in character of sputum” and radiology interpretation.⁵ These made the prior VAP definition difficult to use in surveillance, in research, and as a measure for pay-for-performance metrics and hospital assessment. Given the substantial mortality, morbidity, and cost attributed to the clinical entity of VAP,^7‐11 however, there was considerable clinical, public health, and governmental interest¹² to measure and report VAP as a hospital benchmark.

The new NHSN definition corrects many of the shortcomings of the earlier definition. First, it creates a taxonomy of iatrogenic ventilator complications, differentiating between all iatrogenic ventilator-related injuries and infectious ones. Second, the new definition relies on concrete, discrete changes in vital signs, ventilator settings, and culture data, making it possible to automate the surveillance process. Third, the CDC removed subjective and problematic components of the previous definition, including the evaluation of radiology and change in the character of sputum, among others.

First, we hypothesized that an automated assessment of the new NHSN definition could be reliably implemented using existing hospital databases. Second, we sought to compare patients with VAEs to those patients who did not develop these events. Finally, we anticipated that the patients who had met the previous definition for VAP would be different from those patients identified under the new definition.

Materials and Methods

Setting

The study was performed at the Beth Israel Deaconess Medical Center, a tertiary care, urban hospital in Boston, Massachusetts, with > 70 intensive care beds in nine ICUs. The study was reviewed by the hospital’s institutional review board and was granted a waiver of informed consent (protocol number 2013-P000062).

Study Design and Data Sources

All patients aged ≥ 18 years admitted to any of the hospital’s nine ICUs from July 1, 2008, to March 31, 2013, were included in the study. We extracted prospectively collected patient-level data from the detailed electronic medical record at our institution. We extracted age, race, sex, comorbidities defined using the Elixhauser method,¹³ patient-level case mix,¹⁴ severity of illness measured using the Sepsis-Related Organ Failure Assessment (SOFA),¹⁵ medication use using pharmacy charges, ventilator use based on electronic medical record documentation, admission source (same-day surgery, ED, or other), emergent admissions, hospital disposition (home with or without services vs any other disposition), length of stay (discharge date minus admission date plus one), and in-hospital mortality (defined as any in-hospital death, including those associated with do-not-resuscitate orders or aggressive comfort care).

Primary Independent Variable

The primary independent variable of interest, VAE, was defined using CDC’s NHSN new definitions.¹ We developed electronic algorithms to extract each of the four levels of the new definition: ventilator-associated condition (VAC), infection-related VAE (IVAC), possible VAP, and probable VAP. The algorithm by which patients were identified and the data abstracted are outlined in e-Figure 1 ^{(370.5KB, pdf)} , as is greater detail on the VAE definitions (e-Fig 2 ^{(370.5KB, pdf)} ). The algorithm assigned each patient as having one of the four categories of VAE or no VAE. We then validated the output of the algorithm against cases that were manually categorized by a nurse with > 5 years’ experience in abstracting NHSN VAP cases (J. G.) using a convenience sample of months; the human reviewer used the CDC calculator for VAE¹⁶ to ensure categorization consistent with the federal surveillance definition.

Outcomes

The primary outcome of interest was in-hospital mortality. Secondary outcomes included hospital length of stay, ICU length of stay, and likelihood of returning home rather than dying or going to a rehabilitation or extended-care facility. The outcomes of patients with VAE were compared with patients who were mechanically ventilated for at least 4 days but who did not develop VAE. This comparison group was chosen because patients who are ventilated for < 4 days cannot, by definition, have a VAE. They must have 2 days of mechanical ventilation followed by a worsening in positive end-expiratory pressure or Fio₂ sustained for 2 days, for 4 total days of mechanical ventilation. We also assessed the relationship between VAE and the former definition of VAP (the pneumonia [PNEU] definition) over this same time period. As part of routine ICU operations, we had previously prospectively identified VAP using NHSN’s former definitions. For logistical reasons, this surveillance included only 7 months of each calendar year and only four ICUs. As a result, we included only these ICUs and months in the analysis comparing VAE and the former definition of VAP. VAC (or any VAE) was chosen as the comparator group to patients who met the PNEU surveillance definition of VAP, as these rates are reported to the CDC currently.

Statistical Analysis

The unit of analysis was hospital admission. Estimates of the validity of the electronic algorithm as compared with the human abstractor are presented in terms of the sensitivity, specificity, and accuracy of the algorithm. When the algorithm’s result differed from the human abstractor’s, we performed chart review to evaluate the reason. Cohen’s κ was used to compare agreement between patients identified as having VAE and as having VAP under the prior federal definition. Estimates of κ range from 0 to 1, where 0 represents no agreement and 1 represents complete agreement.

Unadjusted comparisons were made between patients with VAE and patients without VAE who were ventilated for ≥ 4 days using Student t test, χ², or Fisher exact test, as appropriate. To assess the independent relationship of developing VAE to a patient’s risk of death, we constructed multivariable logistic regression models adjusting for all variables with > 0.1 significance using a stepwise selection process (e-Table 1 ^{(370.5KB, pdf)} ). All analyses were conducted using SAS, version 9.3 (SAS Institute Inc).

Results

A total of 26,466 consecutive hospital admissions were included for analysis. Of these, 10,998 (42%) required mechanical ventilation, with an average duration of mechanical ventilation of 4 days (median, 2 days; interquartile range, 1-5 days) and a total number of 46,850 ventilated days. There were 3,302 patients ventilated for ≥ 4 days continuously.

Patient Population

The algorithm identified 675 cases of VAE (3% of all admissions, 6% of all patients who required mechanical ventilation). On average, when compared with patients who received mechanical ventilation for ≥ 4 days, patients with VAE were younger and more likely to be men. There were no significant differences in comorbidities between groups, with the exception of iron-deficiency anemia and hypertension, both of which were less prevalent among patients with VAEs, and chronic weight loss, which was more prevalent among patients with VAEs (Table 1). Patients with VAEs were more likely to die (38% vs 24%; relative risk, 1.57; P < .0001) and had longer ICU and hospital stays (18 days vs 11 days, P < .0001; and 24 days vs 18 days, P < .0001, respectively) and were less likely to go home or home with services (10% vs 17%, P < .0001). Patients with VAEs had a significantly higher case-mix index than patients without VAEs and higher SOFA scores on the day of admission (Table 1). In adjusted analyses, patients with VAE continued to have a higher likelihood of death (OR, 1.91; 95% CI, 1.53-2.37; P < .0001) (e-Table 1 ^{(370.5KB, pdf)} ).

TABLE 1 ] .

Univariable Comparison Between Patients With a VAE and Patients on Mechanical Ventilation for ≥ 4 Days Without a VAE

Characteristic	VAE	No VAE	Test of Significance
No.	675	2,730	...
Demographics
Age, mean (SD), y	62 (17)	64 (16)	.004
% Female	39	45	.002
% Nonwhite	33	32	.63
Comorbidities
Congestive heart failure	19	20	.49
Pulmonary circulation disorders	6	6	.86
Peripheral vascular disease	7	8	.50
Paralysis	8	7	.29
Other neurologic disorders	11	11	.67
Chronic lung disease	15	17	.13
Diabetes	13	14	.36
Diabetes with complications	3	4	.16
Hypothyroidism	5	7	.06
Renal failure	12	14	.13
Metastatic cancer	3	4	.54
Solid tumor without metastasis	3	3	.94
Rheumatoid arthritis/collagen vascular disease	2	2	.58
Obesity	6	6	.83
Weight loss	13	10	.03
Chronic blood loss anemia	< 1	< 1	.9
Deficiency anemias	8	12	.01
Alcohol abuse	7	7	.56
Drug abuse	3	4	.12
Psychiatric disease	3	3	.65
Depression	4	6	.12
Chronic hypertension	34	40	.004
ICU of admission
MICU	42	43	.04
SICU	40	36	...
% Admissions emergent	67	68	.88
% Admissions from ED	66	67	.69
% Admissions from same-day surgery	4	6	.06
Case-mix index	8.2	6.0	< .0001
SOFA score on first ICU day	6.0	5.6	.01
Risk of in-hospital death	38	24	< .0001
Length of stay, mean (SD), d	24 (17)	18 d (14 d)	< .0001
ICU length of stay, mean (SD), d	18 (12)	11 d (8 d)	< .0001
Likelihood of going home or home with services	10	17	< .0001

Data are presented as % unless otherwise noted. Tests of significance were performed using χ², Fisher exact, or Student t test where appropriate. MICU = medical ICU; SICU = surgical ICU; SOFA = Sepsis-Related Organ Failure Assessment; VAE = ventilator-associated event.

Algorithm Validation

We compared the output of the electronic algorithm with a trained human abstractor. The abstractor reviewed 1,229 cases, of whom 426 were ventilated and 31 had VAEs. The electronic algorithm correctly identified 29 cases, for a net sensitivity of 93.5% (95% CI, 77.2%-98.8%), specificity of 100% (95% CI, 98.8%-100%), and an accuracy of 99.5% compared with the human reviewer (Table 2) (comparison of validation cohort with other vented patients available in e-Table 2 ^{(370.5KB, pdf)} ). There were no cases identified as positive by the algorithm but negative by the reviewer. On chart review of the two cases not identified by the algorithm, we found that in both cases no endotracheal tube or tracheostomy or set tidal volume was charted at all in the electronic medical record, and, therefore, the algorithm classified the patients as receiving noninvasive ventilation (e-Fig 1 ^{(370.5KB, pdf)} ). The algorithm required 60 total min to extract and analyze the entire cohort of 26,466 patients (0.16 s per patient in the ICU) as compared with between 17 and 30 min per patient in the ICU for VAE surveillance for the human (P < .0001).

TABLE 2 ] .

Review of 6 Mo of All Patients for Six ICUs

	Human Reviewer
Electronic Algorithm	VAC	IVAC	Possible VAP	Probable VAP	Not Identified
VAC	19	...	...	...	...
IVAC	1	3	...	...	...
Possible VAP	...	...	6	...	...
Probable VAP	...	...	...	0	...
Not identified	2a	...	...	...	...

Months reviewed were February 2009, March 2009, July 2009, July 2012, February 2013, and March 2013. A total of 1,229 patients admitted during these months reviewed with 426 of these patients on mechanical ventilation. Sensitivity of 93.5%, specificity of 100%, and accuracy of 99.5%. IVAC = infection-related ventilator-associated complicaton; VAC = ventilator-associated condition; VAP = ventilator-associated pneumonia.

^aIndividual not charted as using either endotracheal tube or tracheostomy for > 48 h.

Comparison With Previous NHSN Definitions

Comparison of VAE classifications vs CDC’s previous NHSN VAP (PNEU) definition is presented in Table 3. Patients with VAE and VAP under the former federal definition were at increased risk of death, prolonged length of stay, and ICU length of stay and at decreased risk of returning home with or without services as compared with patients with neither illness (Table 3). However, only nine of 30 patients (30%) who met CDC’s former VAP definition fulfilled VAE criteria (nine VACs, three IVACs, three possible pneumonias, zero probable pneumonias). The κ statistic ranged from 0.06 for VACs to 0 for probable VAP, suggesting no agreement.

TABLE 3 ] .

Univariable Comparison Between VAEs Under the New NHSN Definition, NHSN VAP Under the Former PNEU Definition, and All Other Patients

Outcome	VAE	NHSN-PNEU VAP	Neither VAE nor NHSN-PNEU VAP	Test of Difference
No.	246	30	6,585	...
Risk of in-hospital death, %	39	37	11	< .0001
Length of stay, mean (SD), d	25 (19)	23 (14)	9 (9)	< .0001
ICU length of stay, mean (SD), d	18 (13)	16 (10)	3 (4)	< .0001
Likelihood of going home or home with services, %	8	3	48	< .0001

Patients limited to 6,852 total patients who were admitted to the four ICUs reviewed for NHSN VAP during the 6 mo they were reviewed between 2008 and 2012. Nine patients were defined as having both VAE and NHSN VAP. Tests of significance were performed across VAE, NHSN VAP, both, and neither, using χ², Fisher exact, or Student t test where appropriate. NHSN-PNEU = National Healthcare Safety Network-Pneumonia; PNEU = pneumonia. See Table 1 and 2 legends for expansion of other abbreviations.

Discussion

Our results show that, in a large cohort of consecutive ICU admissions, the CDC’s new definition for VAEs identifies patients with increased risk of death, longer lengths of stay, and decreased likelihood of returning home following hospitalization. Our study also demonstrates the feasibility of complete automation of screening for VAE, potentially saving thousands of hours of staff time spent in chart review. Finally, although patients were at increased risk of death when identified under either the former definition of VAP or the new VAE definition, these were two minimally overlapping cohorts of patients, suggesting they may represent different disease entities and opening important new lines of investigation.

The previous NHSN definition was already under significant criticism as failing both tests of reliability and validity,^5,17 which made its use as a benchmark of hospital quality and performance problematic.^2,18 The new definition of VAE eliminated several areas of subjectivity, such as chest radiograph interpretation, that were believed to be an important source of the flawed performance of the definition as a surveillance metric.² Our study served as a proof of concept: Surveillance of VAE can be implemented with reliability using completely automated methods. However, further research is necessary to determine whether VAE definitions are reliable across hospitals, particularly when different electronic medical records are used. Further, it is unclear what additional shortcomings may emerge from the new definition. For example, might hospitals lower possible and probable VAP rates by never sending respiratory cultures, thereby categorizing all patients as IVAC who may have VAPs (an important reason CDC has pursued reporting VAC and IVAC rates rather than VAP rates)? Among our population, only four of 361 total patients who were considered to have IVAC had no respiratory cultures sent during their admission (and zero of 18 patients with IVAC met this criterion in the validation sample).

Previous evaluations of definitions similar to VAE demonstrated similar associations with patient-level outcomes.^19,20 Our data confirmed that the new definition identified patients at greater risks of death and longer lengths of stay, further supporting its capacity to identify patients at risk for poor outcomes. However, both our analysis and the literature suggested the previous definition of VAP also predicted poor outcomes.^8,11,21 Additional research is needed to determine whether VAE can be prevented or if it serves solely as a marker of severity of illness not captured in risk adjustment. Furthermore, it is unknown how well the four-level VAE definition distinguishes between patients with true ventilator-associated iatrogenesis vs those with worsening disease states; greater research is necessary to understand what disease states are described by each level of the VAE taxonomies.

Our study did have several strengths. We studied a large cohort of consecutive patients in the ICU, assessing VAE in all patients in our ICUs, thereby eliminating any selection bias in the study population. We validated our algorithm against hand-extracted chart data entered into the online CDC calculator for VAE by a nurse highly experienced in NSHN VAP abstraction, and we were able to compare VAE rates with PNEU-defined VAP rates collected at the same institution over the same time period.

Our study has a number of limitations. First, we identified comorbidities using administrative data. Although we used a validated algorithm to transform discharge diagnoses to comorbidities,¹³ manual review of charts might have revealed additional comorbidities. To mitigate this limitation, we additionally adjusted for the SOFA score,¹⁵ although residual confounding from unmeasured severity of illness may remain . Second, although our study demonstrated reliable extraction of VAE over time, the reliability was in large part due to the automated process. In the absence of an electronic medical record in the ICU that permits similar automation, subjectivity in data abstraction could threaten the major advantage of this new definition, its decreased subjectivity compared with prior VAP definitions. Third, we made use of a single abstractor to validate our electronic algorithm rather than multiple human comparators. However, we attempted to minimize the effect of this shortcoming by having the human extractor validate her conclusions against the online CDC VAE calculator. Fourth, because we considered the unit of analysis to be the hospital admission, we treated patients as either “having VAE” or “not having VAE.” Further research could address temporal factors related to VAE, since patients have VAE on some but not all of their ICU days. Fifth, although we sought to describe the epidemiology of the new surveillance definition of VAE, we did not address how many episodes of VAE might be preventable (ie, what proportion of VAE represents true iatrogenesis and what proportion is acute lung injury from preexisting systemic disease). This is a critical next direction for research to measure and improve the care of patients with iatrogenic lung injury. Finally, this remains a single-institution study. Further research is necessary to establish the between-institution reliability of the current definition.

Conclusions

In a large cohort of consecutive ICU admissions, the CDC’s new definition for VAEs effectively identifies patients at greater risk of poor outcomes. The new surveillance definition for VAEs put forth by the CDC represents a substantial step forward toward generating a reliable method of identifying iatrogenic complications from mechanical ventilation. However, it is not yet clear whether the new definition identifies an iatrogenic cause of patient illness that can be intervened upon or, instead, identifies a marker of severity of illness for patients undergoing mechanical ventilation, or some combination of both. Until VAE can be established as a target on which we can intervene to reduce patient harm, we continue to recommend against using VAE for pay-for-performance metrics or legislated mandates.

Supplementary Material

Online Supplement

ABBREVIATIONS

CDC

Centers for Disease Control and Prevention

IVAC

infection-related ventilator-associated complication

NHSN

National Healthcare Safety Network

PNEU

pneumonia

SOFA

Sepsis-Related Organ Failure Assessment

VAC

ventilator-associated condition

VAE

ventilator-associated event

VAP

ventilator-associated pneumonia