A Method to Explore Variations of Ventilator-Associated Event Surveillance Definitions in Large Critical Care Databases in the United States

OBJECTIVES:

The Centers for Disease Control has well-established surveillance programs to monitor preventable conditions in patients supported by mechanical ventilation (MV). The aim of the study was to develop a data-driven methodology to examine variations in the first tier of the ventilator-associated event surveillance definition, described as a ventilator-associated condition (VAC). Further, an interactive tool was designed to illustrate the effect of changes to the VAC surveillance definition, by applying different ventilator settings, time-intervals, demographics, and selected clinical criteria.

DESIGN:

Retrospective, multicenter, cross-sectional analysis.

SETTING:

Three hundred forty critical care units across 209 hospitals, comprising 261,910 patients in both the electronic Intensive Care Unit Clinical Research Database and Medical Information Mart for Intensive Care III databases.

PATIENTS:

A total of 14,517 patients undergoing MV for 4 or more days.

MEASUREMENTS AND MAIN RESULTS:

We designed a statistical analysis framework, complemented by a custom interactive data visualization tool to depict how changes to the VAC surveillance definition alter its prognostic performance, comparing patients with and without VAC. This methodology and tool enable comparison of three clinical outcomes (hospital mortality, hospital length-of-stay, and ICU length-of-stay) and provide the option to stratify patients by six criteria in two categories: patient population (dataset and ICU type) and clinical features (minimum Fio₂, minimum positive end-expiratory pressure, early/late VAC, and worst first-day respiratory Sequential Organ Failure Assessment score). Patient population outcomes were depicted by heatmaps with mortality odds ratios. In parallel, outcomes from ventilation setting variations and clinical features were depicted with Kaplan-Meier survival curves.

CONCLUSIONS:

We developed a method to examine VAC using information extracted from large electronic health record databases. Building upon this framework, we developed an interactive tool to visualize and quantify the implications of variations in the VAC surveillance definition in different populations, across time and critical care settings. Data for patients with and without VAC was used to illustrate the effect of the application of this method and visualization tool.

The Centers for Disease Control (CDC) uses surveillance definitions to monitor preventable conditions that affect patient outcomes. To improve clinical outcomes for patients receiving mechanical ventilation (MV), the CDC adopted the ventilator-associated pneumonia (VAP) definition in 1994 (1). Almost 2 decades later, in 2013, the CDC convened an expert panel to address shortcomings of the VAP definition (2). The working group developed the concept of ventilator-associated events (VAEs) to extend the definition beyond infection and identify both infectious and noninfectious nosocomial respiratory deterioration. The VAE surveillance definition provides more specific criteria for detecting events of interest using a three-tier hierarchy. An event in the first tier is a ventilator-associated condition (VAC). The second tier is an infection-related ventilator-associated complication. The third tier is a possible/probable VAP (3–5). The VAC tier includes all events in subsequent tiers, thereby identifying patients at risk of preventable negative clinical outcomes.

One of the CDC panel’s intentions in proposing revisions was to use specific clinical criteria that could be readily extracted from the electronic health record (EHR). Retrospective analyses demonstrated that components of the VAE definition can be effectively implemented in existing EHR databases (6, 7). Automated surveillance models have been used to implement VAE surveillance in real time (4, 8, 9). Surveillance definitions may need to be revised and calibrated to improve capture of patients at risk or to improve implementation of health policy. Low event frequency, small population sizes, and different care settings complicate surveillance definition calibration. The availability of large publicly available databases, representing a heterogeneous patient population in diverse critical care settings and with documented patient outcomes, allows further exploration of definitions and their associations with patient outcomes.

Surveillance definitions aim to detect patients at risk of adverse clinical outcomes and are especially relevant for conditions with relatively low incidence but detrimental repercussions on patient status. This is the case for VAE, which have an incidence of 2.0-7.9/1000 ventilation-days and are associated with increased mortality (10). Most hospitals use considerable human resources to support VAE surveillance, although automated computer surveillance programs can be developed to provide real-time VAE monitoring (9–11).

This study proposes a methodology and offers an interactive data visualization tool to test variations in the VAC component of the VAE surveillance definition, using EHR data from publicly available U.S. databases. In this proof-of-concept study, the impact of changes to the VAC definition is assessed based on how they stratify patient outcomes.

MATERIALS AND METHODS

We extracted clinical data from two different EHR datasets, selected patients receiving MV, and developed a visual tool to analyze different subpopulations and the incidence of VAC across combinations of clinical criteria. The corresponding code for the proposed framework to frequently reevaluate VAC definitions can be found on GitHub at https://github.com/SCCM2020-team7/VAC-paper01. This dashboard will be hosted live for several years after publication at https://vacdefinitionexplorer.duhs.duke.edu/.

Data Sources

eICU Clinical Research Database (CRD) and Medical Information Mart for Intensive Care III (MIMIC-III) provided the data source for these analyses (12–15). We extracted data for all patients receiving MV for at least 4 calendar days (eTable 2, http://links.lww.com/CCX/B84), in accordance with the CDC criteria for VAC:

• 1) Baseline: MV for 2 calendar days, with at least one measurement of positive end-expiratory pressure (PEEP) and fractional inspired oxygen (Fio₂) recorded for greater than or equal to 1 hr/d; and

• 2) Instability: MV for 2 calendar days immediately following the baseline period, with at least one measurement of increased Fio₂ and/or of PEEP recorded for greater than or equal to 1 hr/d.

Exclusion criteria were:

• 1) age less than 18 years,

• 2) missing Fio₂ data, or

• 3) missing PEEP data.

Missing Fio₂ and PEEP data were defined as no data within a day period. Only the first MV episode in each ICU stay was included to avoid confounding. Hospital and ICU length of stay (LOS) were calculated for survivors and nonsurvivors separately. In addition to statistical analyses stratified by dataset, patient populations were combined across eICU and MIMIC-III to maximize power. To support pooled analysis, ICU type was harmonized across eICU and MIMIC-III, resulting in the four following categories: medical, surgical, mixed medical-surgical, and neurological ICUs.

Data in the Medical Information Mart for Intensive Care III (MIMIC-III) and the eICU-CRD have been previously deidentified, and the institutional review boards of the Massachusetts Institute of Technology (No. 0403000206) and Beth Israel Deaconess Medical Center (2001-P-001699/14) both approved the use of the databases for research.

CDC Definitions

The CDC VAC definition requires an initial baseline period of 2 calendar days of MV stability, followed by 2 calendar days of MV instability, defined by worsening oxygenation (eFig. 1, http://links.lww.com/CCX/B84)(3).

CDC-Calendar Daytime Interval.

If a patient receives MV during any part of a calendar day, they are considered to have received MV for the whole “CDC-Calendar Day” time interval. Per the CDC definition, once an MV session crosses the midnight boundary, a new day is counted. This creates a situation where 2 calendar days can range from 25 to 48 hours (eFigs. 2–4, http://links.lww.com/CCX/B84). With similar variability in both stability and instability periods, the total minimum MV duration for CDC-calendar day criteria is between 50 and 96 hours. More details are provided on the CDC website (3).

CDC Oxygenation Definition.

Worsening oxygenation after the baseline stability period is defined as an increase above minimum Fio₂ (ΔminFiO₂) of greater than or equal to 20% or an increase above minimum PEEP (ΔminPEEP) of greater than or equal to 3 cm H₂O—relative to the patient’s baseline oxygenation parameters—maintained for longer than 1 hour (3).

Comparative Definitions

In this study, comparative definitions for worsening oxygenation were explored. Furthermore, an interactive visualization tool was built and made publicly available to explore the influence of varying selection criteria on sample size, VAC incidence rates, and patient outcomes—within and across the EHR databases used. Through the interactive tool, distributions of patient outcomes can be used to evaluate different VAC definition parameters.

Comparative 24-Hour Period Time Interval.

The baseline stability period was established using 24-hour time periods, from intubation onwards. Each subsequent “day” began 24 hours after intubation. All patients spent an equal amount of time in the stability phase (≥48 hr). Time spent in the subsequent 2-day instability period was also identical for all patients (≥48 hr), resulting in a minimum of 96 hours. This alternative definition of time periods is described in eFigures 2–4 (http://links.lww.com/CCX/B84).

Comparative Oxygenation Definition.

Worsening oxygenation after the baseline stability period was defined by different combinations of change in ΔminFiO₂ and ΔminPEEP. From baseline values, ΔminFiO₂ values were modified using 10% increments (10–70%); similarly, ΔminPEEP values were modified using 1 cm H₂O increments (1–7 cm H₂O).

Statistical Comparisons.

The datasets were analyzed with R Core Team, Auckland, New Zealand 3.6.3 and Python 3.6 Centrum Wiskunde & Informatica (CWI), Netherlands. In both the eICU-CRD and MIMIC-III databases, comparisons between the characteristics of patients with and without VAC were conducted using two-sided t tests (for numeric variables) or chi-square tests (for categorical variables). Kaplan-Meier separation used log-rank tests.

Interactive Visualization

An interactive visualization was created to further explore the association of selected criteria (dataset, time interval definition, and ICU type) with patient outcomes. Specifically, this tool features heatmaps as well as Kaplan-Meier curves characterizing inhospital mortality among patients with and without VAC. Furthermore, the effect of varying thresholds for ventilation settings (e.g., ΔminPEEP, ΔminFiO₂) can be tested empirically.

Heatmaps

Heatmaps showcasing differences in outcomes between VAC and non-VAC patients were generated. Odds ratios were used as evaluation metrics for inhospital mortality, whereas mean time-to-event measures were used for hospital LOS and ICU LOS). To facilitate comparing outcome differences resulting from varying time intervals (calendar day and 24-hr periods), distinct heatmaps were provided for each time interval definition. Within a given heatmap, the relative influence of ΔminPEEP and ΔminFiO₂ criteria is rendered through color-coded matrix cells, where color intensity increases with the magnitude of adverse outcomes, that is, VAC mortality odds ratio and the prolongation of the hospital/ICU LOS.

Kaplan-Meier Step Functions

Kaplan-Meier step functions were generated to illustrate differences in 28-day inhospital mortality, measured from the start of MV, between patient subgroups. If the patient survived their hospitalization, the mortality outcome was considered right-censored on hospital discharge. In contrast, if the patient died during their hospitalization, the length of time between the start of MV and death was considered.

VAC Incidence

VAC incidence was computed as the number of VAC per 1,000 person-days of MV (3).

RESULTS

Across the two ICU databases, 14,517 patients met eligibility criteria for this study (Fig. 1). The eICU-CRD database contains 200,859 patients, and 8,210 (4.0%) met inclusion criteria. The MIMIC-III database contains 61,051 patients, and 6,307 (10.3%) met inclusion criteria. In total, when applying the CDC criteria, 1,366 out of 14,517 patients (9.41%) met the CDC criteria for VAC (eICU-CRD: 600/8,210 [7.3%], MIMIC-III: 766/6,307 [10.5%]).

Figure 1.

Flow diagram. eICU-CRD = electronic ICU - Clinical Research Database, MIMIC-III = Medical Information Mart for Intensive Care III, MV = mechanical ventilation, PEEP = positive end-expiratory pressure, VAC = ventilator-associated condition.

Clinical Outcomes for Patients With and Without VAC

In both databases, patients meeting the CDC criteria for VAC (ΔminFiO₂ ≥ 20%, ΔminPEEP ≥ 3, 4 ventilated calendar days) had higher mortality than their counterparts without VAC (34.4% vs 25.9%; p < 0.001), a longer duration of MV (11.5 ± 7.9 vs 7.7 ± 5.4 d; p < 0.001), and a longer ICU-LOS (15.6 ± 11.3 vs 11.74 ± 9.3 d; p < 0.001). Patient characteristics are further described in Tables 1 and 2.

TABLE 1.

Patient Characteristics, Stratified by Ventilator-Associated Condition Versus No Ventilator-Associated Condition Status

Variable	VAC	%	No VAC	%	All	p
Patients	1,366	9.41	13,151	90.59	14,517
Race and ethnicity
Asian	21	1.54	228	1.74	249	0.3995
Black	143	10.47	1,296	9.86	1,439
Hispanic	63	4.61	520	3.96	583
Native_American	8	0.59	50	0.38	58
White	979	71.67	9,696	73.79	10,675
Missing	152	11.13	1,359	10.33	10.41%
Sex
Female	536	39.24	5,873	44.7	6,409	< 0.001
Male	830	60.76	7,271	55.33	8,101
Missing	0	0.0	0	0.0	0.0%
ICU type
Med-surg	368	26.94	4,727	35.97	5,095	< 0.001
Medical	476	34.85	4,294	32.68	4,770
Neuro	58	4.25	630	4.79	688
Surgical	464	33.97	3,498	26.62	3,962
Missing	0	0.0	0	0.0	0.0%
Inhospital mortality?
Survivors	892	65.3	10,179	77.47	11,071	< 0.001
Nonsurvivors	470	34.41	2,881	21.93	3,351
Missing	4	0.29	82	0.62	0.59%
Age (yr, mean ± sd)	58.96 ± 15.86		62.57 ± 16.46		62.23 ± 16.44	< 0.001
Body mass index (kg/m2, mean ± sd)	31.94 ± 11.3		29.82 ± 9.54		30.01 ± 9.74	< 0.001
Worst Sequential Organ Failure Assessment (first 24 hr, mean ± sd)	10.42 ± 4.0		9.28 ± 3.75		9.39 ± 3.79	< 0.001
Mechanical ventilation duration (d, mean ± sd)	11.51 ± 7.89		7.73 ± 5.35		8.09 ± 5.74	< 0.001
Outcomes by survivorship status
ICU LOS (d)
Survivors	17.47 ± 11.54		11.8 ± 9.09		12.26 ± 9.44	< 0.001
Nonsurvivors	11.92 ± 9.89		11.56 ± 9.95		11.61 ± 9.94	0.93
Hospital LOS (d)
Survivors	26.85 ± 18.12		21.05 ± 17.35		21.52 ± 17.48	< 0.001
Nonsurvivors	15.99 ± 14.59		17.16 ± 17.39		17.0 ± 17.03	0.0386

LOS = length of stay, VAC = ventilator-associated condition.

The table characterizes patients in the pooled sample, emanating from the combination of the eICU Clinical Research Database and Medical Information Mart for Intensive Care III electronic health record critical care databases. Missingness rates are provided, when applicable. VAC labels are computed from Centers for Disease Control criteria (minimum Fio₂ ≥ 20%, minimum positive end-expiratory pressure ≥ 3, calendar day time intervals).

TABLE 2.

Patient Stratification and Proposed Analysis Paradigm to Evaluate Alternative (Ventilator-Associated Condition) Surveillance Definitions

Patient Population (Underlying Characteristics)	Clinical Criteria (Ventilation Parameter Settings)	Patient Outcomes (Evaluation Metrics)
Data source (i.e., Medical Information Mart for Intensive Care III or eICU Clinical Research Database)	Tier 1 of VAC criteria	Inhospital mortality
	Minimum FiO2	ICU LOS
	Minimum positive end-expiratory pressure
		Hospital LOS
Type of ICU (i.e., medical, surgical, and neuro	Calendar day vs 24-hr day
	Early vs late VAC (<7 vs ≥7 d)

eICU = electronic ICU, LOS = length of stay, VAC = ventilator-associated condition.

The levels of analysis triggering alternative VAC surveillance definitions (see clinical criteria) across patient subpopulations (based on data source and ICU type) are described. Further, three distinct outcomes were used to compare/contrast the performance of varying definitions.

Interactive Tool

The interactive tool supports open exploration of variations in VAC criteria (Fig. 2). Heatmaps showcasing variability in mortality odds ratios and time-to-event outcomes (for ICU LOS and hospital LOS) as part of the interactive visualization tool (Figs. 2 and 3) offer perspective on the effects of jointly modifying the VAC criteria. Additionally, a closer look at stratified patient demographics (i.e., both the prevalence of VAC vs no VAC but also differential inhospital mortality rates) clarifies the combined role of the time definition, ΔminFiO₂, ΔminPEEP, VAC timing, ICU types, and respiratory Sequential Organ Failure Assessment (SOFA) score in identifying a wide range of patients at risk for adverse clinical outcomes.

Figure 2.

Illustration of the interactive visualization tool for a select set of ventilation criteria: minimum Fio₂ (ΔminFio₂) greater than or equal to 20%, minimum positive end-expiratory pressure (ΔminPEEP) greater than or equal to 3 (n = 14,517; ventilator-associated condition [VAC] = 1,366; no VAC = 13,151). On each heatmap, the cell surrounded by a blue box corresponds to the criteria selected in this example (ΔminFio₂ ≥ 20%, ΔminPEEP ≥ 3). The upper left graph represents the Kaplan-Meier survival curves among patients with VAC versus patients without VAC (“no VAC”) resulting from the selected set of clinical criteria, whereas the upper right panel illustrates variability in mortality odds ratios across changes in ΔminFio₂ and ΔminPEEP. In addition, the lower panels showcase differences in ICU (left) and hospital (right) length of stay outcomes. Although the estimated probabilities of survival clearly distinguish VAC patients from others until the 15th day following the start of mechanical ventilation, they fully overlap from the 16th day onward (p = 0.063).

Figure 3.

Illustration of the interactive visualization tool for a select set of ventilation criteria: minimum Fio₂ (ΔminFio₂) greater than or equal to 20%, minimum positive end-expiratory pressure (ΔminPEEP) greater than or equal to 6 (n = 14,517; ventilator-associated condition [VAC] = 274; no ventilator-associated condition [VAC] = 14,243). On each heatmap, the cell surrounded by a blue box corresponds to the criteria selected in this example (ΔminFio₂ ≥ 20, ΔminPEEP ≥ 6). Of note, the detrimental effects of VAC occurrence during the ICU stay are more pronounced than in the preceding example (ΔminFio₂ ≥ 20, ΔminPEEP ≥ 3)—across all three outcomes. The upper left plot represents the Kaplan-Meier survival curves among patients with VAC versus patients without VAC (non-VAC) resulting from the selected set of clinical criteria, whereas the upper right heatmap illustrates variability in mortality odds ratios across changes in PEEP and Fio₂. In addition, the lower panels showcase differences in ICU (left) and hospital (right) length of stay outcomes. Although the estimated Kaplan-Meier survival curve pertaining to VAC patients presents wider confidence intervals than in the first/previous/preceding example, the estimated probabilities of survival for VAC and non-VAC patients clearly separate until the end of follow-up (i.e. 28 d, p < 0.001).

These impacts are illustrated using two examples.

• 1) In this database, using the CDC definition of ΔminFiO₂ greater than or equal to 20% and ΔminPEEP greater than or equal to 3 presented in Figure 2 yields 1,366 patients (9.7%) with VAC, that is, an incidence of 11.63 VAC/ 1,000 ventilator-days. In contrast with the significant difference in overall mortality (34.4% vs 25.9%; p < 0.001), the Kaplan-Meier curve suggests no statistically significant difference in survival curves between patients with and without VAC by log-rank test (p = 0.06). Patients with VAC were on average younger (59.0 vs 62.6), had higher BMIs (31.94 vs 29.82), had higher SOFA scores (10.42 vs 9.28), and remained mechanically ventilated longer (11.51 vs 7.73 d). Notably, ICU LOS was similar between survivors and nonsurvivors.

• 2) In contrast, when applying an increased threshold of ΔminPEEP greater than or equal to 6 while keeping a ΔminFiO₂ greater than or equal to 20%, the Kaplan-Meier curve presents a statistically significant difference in survival between patients with and without VAC (p < 0.005), despite using the same database as in the first example.

When converting from CDC-Calendar Day to 24-hour day, there were 28.1% (10,436 vs 14,517) fewer patients examined and demonstrated a slightly higher mortality rate (25.2% vs 23.2%). Further examples are provided in eFigure 5 (http://links.lww.com/CCX/B84).

DISCUSSION

In this study, we examined the implications of varying definitions of VAC using two publicly available critical care databases from the United States. To facilitate such an analysis, we developed a dynamic interactive tool to graphically visualize the impacts of changes in patient populations, clinical criteria, and clinical outcomes. Each of these can be adjusted in the interactive tool.

Selecting a patient population allows examination both by dataset (i.e., selecting either both available databases through a pooled analysis or a single database) and by VAC status (VAC vs non-VAC). The heatmaps and initial cohort characterizations (e.g., descriptive statistics) provide context as to how individual subpopulations compare in terms of inhospital mortality when selecting varying combinations of VAC criteria.

The selection of various VAC criteria allows nuanced examination of definition standards for ΔminFiO₂, ΔminPEEP, or calendar versus 24-hour day. Different combinations of these parameters affect the number of patients labeled with VAC. The interactive tool provides descriptive statistics of patient cohort composition, showcases differences in the distributions of LOS outcomes in patients with and without VAC, and characterizes the overlap between Kaplan-Meier curves across distinct subpopulations. This analysis framework presents several advantages. First, Kaplan-Meier curves are visual tools that are well-understood by clinical care teams and allow detection of striking differences between patient outcomes as time progresses. Second, their use does not involve the formulation of any parametric assumptions.

The interactive tool can easily be modified to fit other clinical definitions. In this capacity, it would have the potential to examine additional surveillance definitions from the CDC and other regulatory entities (e.g., Food and Drug Administration [FDA]/European Medicines Agency [EMA]). Furthermore, this tool also allows for examination of approaches to VAE detection in specific patient subpopulations, including among trauma and neurocritical care patients who have been identified as high risk for VAC (16–18).

In other data-driven studies, the previous definition of VAP established by the CDC was used to identify a sample of patients with VAC (7, 19, 20). In these studies, patients with VAP had worse outcomes (LOS and mortality) than their counterparts without VAP (8, 11). In recent years, both the original VAC surveillance definitions and modified versions have been implemented in retrospective observational studies and yielded similar results (4, 6, 16) (see eTable 1, http://links.lww.com/CCX/B84). The question of whether the CDC-National Healthcare Safety Network (NHSN) definitions could be improved was explored by Klompas et al (11), demonstrating different performances when clinical variables were modified.

VAC, as currently defined (events per 1,000 MV days), is a relatively rare event. Although studies published to date come from a variety of clinical settings and countries, in many cases, the low VAC incidence requires a national surveillance effort to achieve a sufficient sample size for analysis. Databases integrate data spanning multiple years and often predate the creation of the VAC definition. This presents distributional shifts in patient characteristics and temporal changes in clinical practice to facilitate statistical reevaluation (4, 7, 19). In this context, although most large databases have these limitations, the advent of sizable databases, involving more than one center and large patient populations, opens a new avenue to interrogate VAE definitions and identify patients with worse clinical outcomes under multiple clinical and time interval criteria.

Our study has some limitations. Despite evaluating the impact of varying clinical criteria, we only explored the first tier of VAE. As such, the current study lacks granularity on the etiology of the VAE; examining the implications of alternative surveillance definitions in subsequent tiers will be the object of future research. Additionally, because we examined datasets comprising adult patients only, our results are not generalizable to the pediatric population. Further, our statistical analyses of patient outcomes, although already accounting for age, sex, ICU type, and early/late VAC, do not yet address other influential clinical criteria (e.g., patient comorbidities). Future work would thus test the importance of these potential confounders and incorporate them as needed. Finally, future research will build upon the associations derived in this study, effectively linking patient characteristics, clinical criteria, and time-interval criteria, as well as VAC incidence and long-term outcomes, and further leverage causal analysis to estimate the relative contribution of each individual factor toward patient morbidity and mortality.

The main objective of this research study was not to demonstrate the causal effects of VAE on patient outcomes, but to instead provide a principled methodology implemented in two distinct and sizable databases to retrospectively explore the implementation of various surveillance definitions in different critical care patient cohorts. The appeal of our proposed framework, beyond the opportunity to visually and interactively investigate VAE surveillance definitions through a customizable interface, is the potential for replication in other areas of public health and medicine.

A strength of this study derives from the heterogeneity and complementarity of the two EHR datasets analyzed. Although the eICU-CRD database presents temporal homogeneity (2014–2015) but geographic heterogeneity, the MIMIC-III database is notable for geographic homogeneity but temporal heterogeneity. As generalizability across time and geographies are key considerations for the elaboration of desired nationwide surveillance definitions, this tool provides the opportunity to examine both “overall” performance and “dataset-specific” performance. Going forward, we recommend the CDC-NHSN, in their role as a national surveillance organization, and foster the use of open-access and expert-validated databases to unlock a new era of clinical research to develop future definitions (21).

More importantly, surveillance priorities may change in the years to come, requiring the dynamic reassessment of existing definitions to facilitate health policy decision-making, with an eye toward improving clinical outcomes in patient populations with evolving conditions and needs. For example, the performance of existing VAC criteria will almost undoubtedly change from before the COVID pandemic through the COVID pandemic. Additional applications include the near real-time processing of EHR data, enabling the effective comparison of VAC incidence at a more granular resolution (e.g., surgical ICUs at academic institutions located in different geographic areas). Such an endeavor would allow testing for equal performance of surveillance definitions among vulnerable patient populations and signal the need to refine criteria to increase clinical impact.

CONCLUSIONS

This study presents a methodology to evaluate variations on surveillance definitions for VAC in large publicly available critical care databases. We present a visual interactive tool that illustrates differences in patient outcomes induced by definition changes. The proposed framework utilizes heatmaps to illustrate the effect of comparative VAC definitions on patient outcomes (mortality, ICU LOS, and hospital LOS). This framework can be used to reevaluate VAC definitions in additional datasets, as well as for emerging conditions (e.g., COVID-19) and evolving patient populations. Further, national and international surveillance agencies can invest in the deployment of user-friendly VAC surveillance tools that allow critical care teams to learn from not only their patients, but other sites as well.