Abstract
Rationale: Estimating the impact of ventilator-associated pneumonia (VAP) from routinely collected intensive care unit (ICU) data is methodologically challenging.
Objectives: We aim to replicate earlier findings of limited VAP-attributable ICU mortality in an independent cohort. By refining statistical analyses, we gradually tackle different sources of bias.
Methods: Records of 2,720 adult patients admitted to Ghent University Hospital ICUs (2013–2017) and receiving mechanical ventilation within 48 hours after admission were extracted from linked Intensive Care Information System and Computer-based Surveillance and Alerting of Nosocomial Infections, Antimicrobial Resistance, and Antibiotic Consumption in the ICU databases. The VAP-attributable fraction of ICU mortality was estimated using a competing risk analysis that is restricted to VAP-free patients (approach 1), accounts for VAP onset by treating it as either a competing (approach 2) or censoring event (approach 3), or additionally adjusts for time-dependent confounding via inverse probability weighting (approach 4).
Results: A total of 210 patients (7.7%) acquired VAP. Based on benchmark approach 4, we estimated that (compared with current preventive measures) hypothetical eradication of VAP would lead to a relative ICU mortality reduction of 1.7% (95% confidence interval, −1.3 to 4.6) by Day 10 and of 3.6% (95% confidence interval, 0.7 to 6.5) by Day 60. Approaches 1–3 produced estimates ranging from −0.7% to 2.5% by Day 10 and from 5.2% to 5.5% by Day 60.
Conclusions: In line with previous studies using appropriate methodology, we found limited VAP-attributable ICU mortality given current state-of-the-art VAP prevention measures. Our study illustrates that inappropriate accounting of the time dependency of exposure and confounding of its effects may misleadingly suggest protective effects of early-onset VAP and systematically overestimate attributable mortality.
Keywords: hospital mortality, survival analysis, confounding factors (epidemiology), causality, ventilator-associated pneumonia
Hospital-acquired infections (HAIs) form a major public health problem in developed countries, as they are associated with increased morbidity, mortality, and health-related costs (1–3). Given their critical illness and exposure to invasive treatments, intensive care unit (ICU) patients are particularly prone to acquiring HAIs, especially ventilator-associated pneumonia (VAP).
Appropriate quantification of the impact and burden of unprevented VAP is imperative to understanding its severity and the importance of additional preventive measures and timely treatment. Although its occurrence is reported to be associated with increased ICU mortality, the well-known mantra “correlation does not imply causation” dictates that a considerable number of intubated patients die with, but not necessarily from, VAP. Estimation of the causal impact of HAIs, however, remains subtle and controversial because such assessments are based on either observational data or randomized trials with varying levels of preventive effectiveness (4). Given the multitude of potential sources of variation, past observational studies produced highly variable findings, with excess risk estimates ranging from 0% to 50% (5).
Multistate model (MSM) approaches for estimating mortality and prolonged stay due to HAIs have been widely advocated in recent years, as they aid in avoiding common types of bias (5–7). Their focus on a specific effect measure, the time-dependent population-attributable fraction (PAF), may moreover reduce variability in findings because of various definitions of excess risk. Even so, widely used MSM approaches (8) produce results that cannot be causally interpreted, even in the absence of confounding (9, 10). Moreover, MSM approaches are ill equipped to tackle bias due to group imbalances in prognostic factors that may accrue over the course of time. They may therefore misinform clinical practice. This latter shortcoming, although repeatedly highlighted in the literature (5, 11, 12), remains underappreciated.
The aim of this article is twofold. First, we aim to replicate earlier findings from Bekaert and colleagues (13), who provided the first and (to our knowledge) only study to appropriately address time-dependent confounding in the estimation of VAP-attributable ICU mortality. We assess whether their findings of limited VAP-attributable ICU mortality, which are based on the French multicenter Outcomerea database (n = 4,479) (1997–2008), generalize to an independent cohort of 2,720 mechanically ventilated patients. To maximally reduce bias and ensure between-study comparability, we use identical estimation approaches for causal inference. Second, we illustrate the importance of appropriately accounting for the time-dependent nature of events under study and of their confounding factors by comparing results with those produced by MSM approaches. In doing so, we gradually refine our analysis to demonstrate, in each subsequent step, how different types of bias can be eliminated or reduced.
Methods
Study Population
VAP-attributable ICU mortality was estimated on the basis of records of a cohort of adult patients (aged 18 or older) admitted to the Ghent University Hospital (medical and surgical) ICUs between January 2013 and November 2017, who stayed at the ICU for at least 48 hours and received mechanical ventilation within 48 hours after admission. In the case of ICU readmissions, only the first episode was included.
Data Collection
Admission characteristics were extracted from the Intensive Care Information System database (GE Healthcare Centricity Critical Care) and consisted of demographic data (sex, age, and weight), admission category (medicine, emergency surgery, or scheduled surgery), and severity of illness and comorbidities as captured by the APACHE (Acute Physiology and Chronic Health Evaluation) II score and the updated Charlson comorbidity index (14), respectively. In addition, ICU admission and discharge dates, ICU survival, and data on daily interventions and treatments were extracted, including enteral feeding, corticosteroids (>0.5 mg/kg), mechanical ventilation, use of vasoactive agents, hemodialysis, tracheotomy tube, and treatment limitation decisions (codes 0–4). Measurements of daily disease severity and organ function, as captured by the Sequential Organ Failure Assessment score, were extracted from the Computer-based Surveillance and Alerting of Nosocomial Infections, Antimicrobial Resistance, and Antibiotic Consumption in the ICU (COSARA) (15–17) database.
Following De Bus and colleagues (18), VAP was defined as “hospital-acquired pneumonia diagnosed in patients under mechanical ventilation for 48 hours or longer, or in patients who had been extubated for less than 48 hours after mechanical ventilation for at least 2 days” to exclude community-acquired pneumonia or other types of hospital-acquired pneumonia unrelated to mechanical ventilation. Daily indicators of the acquisition of VAP and other infections (fungal infections and bacterial infections, including abdominal, catheter-related, respiratory, and urinary tract infections), and administered antibiotic treatments were extracted from the COSARA database. A more detailed list of extracted variables, their definitions, and (re)coding are included in the online supplement.
The Ghent University Hospital Ethics Committee approved the study (registration number B670201732106) and waived informed consent because all analyses were performed retrospectively on pseudonymized records. Unique patient and ICU admission identifiers allowed us to link the extracted records.
Statistical Analysis
In this article, we focus on estimation of the time-dependent PAF of ICU mortality due to VAP, which expresses the proportion of preventable death cases in the ICU in the absence of VAP as a function of time since admission. This (inherently causal) effect measure involves a comparison of the observed cumulative incidence of ICU death, which can readily be estimated using a standard competing risk (CR) analysis (that treats ICU discharge as a competing event), with the nonobservable counterfactual cumulative incidence of ICU death that would have been observed if, counter to the fact, VAP had been avoided in all considered patients. Because of its hypothetical nature, the estimation of this second quantity is more challenging. We revisit and compare four proposed approaches for estimating the time-dependent PAF (within the first 60 d after ICU admission), which differ only in the estimation of this counterfactual (VAP-free) cumulative incidence. Each of these approaches apportions weights to VAP-free events that are inversely proportional to the amount of selection of VAP-free patients in the analysis to “reconstruct” the original cohort in the absence of VAP. However, their respective weighting schemes differ in terms of how well they respect the temporal ordering of events and take into account differential selection of VAP-free patients over time. In a methodological companion paper (19), we illustrate how each approach involves a refinement with respect to another, thereby enabling us to gradually tackle and assess different sources of bias. In the discussion (and in the online supplement), we provide more intuition into these refinements.
Approach 1 approximates this quantity by the cumulative incidence of ICU death in patients who remained VAP-free until death or discharge, as estimated by a standard CR analysis (using the Aalen-Johansen estimator), treating ICU discharge as a competing event. This approach ignores VAP onset time or, equivalently, VAP-free patient-time of infected patients.
Approach 2 is based on a CR analysis that treats VAP-free ICU discharge and VAP onset as competing events for VAP-free ICU death. The counterfactual cumulative incidence of ICU death is estimated as a function of the respective cumulative incidences of these competing events that takes into account VAP onset. This approach corresponds with estimation of the PAF by a particular MSM (i.e., the progressive disability model), and was originally proposed by Schumacher and colleagues (8).
Approach 3, also based on a CR analysis, accounts for VAP onset by treating it as a censoring event rather than a competing event for both ICU discharge and death. By censoring infected patients as soon as they acquire VAP, this approach aims to recover the cumulative incidence of ICU death had all patients remained without VAP. This alternative MSM-based approach (20) was originally proposed in another setting and corresponds with an inverse probability (IP) of censoring weighted Aalen-Johansen estimator of VAP-free ICU mortality, which weighs each VAP-free event by the probability of a patient having remained VAP-free while hospitalized (21).
Approach 4 is an extension of approach 3 in which informative censoring of infected patients (or, equivalently, time-dependent confounding of the effect of VAP) is accounted for by incorporating patient (covariate) history into the calculation of the IP weights. This approach is identical to previously suggested causal inference techniques based on IP weighting (13, 22–25) and involves a more elaborate modeling component (because of adjustment for time-dependent covariates). In particular, a Cox proportional hazards model was fitted for the daily probability of acquiring VAP in function of the available covariate history, including admission characteristics and time-dependent factors as listed in Data Collection. For each VAP-free patient-day at the ICU, weights were calculated from the fitted probabilities from the final Cox model. These time-dependent, patient-specific weights express the reciprocal of the probability of a patient having remained VAP-free while hospitalized and the patient’s observed covariate profile up to that day (for more details on causal and modeling assumptions, the set of adjusted covariates, obtained balance across these covariates, and the distribution of the IP weights, see online supplement and Figures E1–E5).
We used nonparametric bootstrapping based on 1,000 samples to calculate percentile-based 95% confidence intervals (CIs) for the time-dependent PAF as estimated by each of the four approaches. All analyses were conducted in R (26) (version 4.0.2) using the ipw (27) and survival (28) R packages.
The STROBE (strengthening the reporting of observational studies in epidemiology) (29) and RECORD (the reporting of studies conducted using observational routinely-collected health data) (30) guidelines were followed.
Results
Descriptives
For 2,729 patients, the first ICU episode that fulfilled the inclusion criteria was included (see flow diagram in Figure 1). Nine episodes were excluded from final analyses because either ICU survival (one episode) or APACHE II score (eight episodes) was missing. During the 32,526 patient-days of follow-up in the remaining 2,720 patients, 210 patients (7.7%) developed VAP, resulting in 29,091 VAP-free patient-days and 3,471 VAP-infected patient-days (note that, for the purpose of this analysis, every patient-day after VAP onset is considered a VAP-infected patient-day). Half of patients infected with VAP were diagnosed with VAP within 5 days of admission. Patients who acquired VAP had a median ICU stay of 21 days and received mechanical ventilation for a median duration of 15 days. In contrast, patients who did not acquire VAP had a median ICU stay of only 7 days and received mechanical ventilation for a median duration of 3 days. Patient characteristics on admission and crude mortality rates are summarized in Table 1.
Figure 1.
Study flow diagram. APACHE = Acute Physiology and Chronic Health Evaluation; ICU = intensive care unit; VAP = ventilator-associated pneumonia.
Table 1.
Characteristics and crude mortality rates for patients (mechanically ventilated within 48 h after admission) with and without VAP
| Patients with VAP (n = 210) | Patients without VAP (n = 2,510) | All patients (n = 2,720) | |
|---|---|---|---|
| Sex, M, n (%) | 146 (69.5) | 1,569 (62.5) | 1,715 (63.1) |
| Age, mean (SD), yr | 56.1 (16.7) | 60.5 (15.7) | 60.2 (15.9) |
| ICU length of stay, median (Q1–Q3), d | 21 (12–31) | 7 (5–13) | 8 (5–15) |
| Ventilation, median (Q1–Q3), d | 15 (9–23) | 3 (2–8) | 4 (2–9) |
| APACHE II score, mean (SD) | 27.1 (6.8) | 27.1 (6.7) | 27.1 (6.7) |
| SOFA score on admission, mean (SD) | 9.5 (3.7) | 9.1 (3.7) | 9.1 (3.7) |
| Respiratory, mean (SD) | 2.2 (1.3) | 2.1 (1.2) | 2.1 (1.2) |
| Coagulation, mean (SD) | 0.7 (1.1) | 0.6 (1.0) | 0.6 (1.0) |
| Liver, mean (SD) | 0.3 (0.7) | 0.4 (0.9) | 0.4 (0.9) |
| Cardio, mean (SD) | 3.1 (1.5) | 2.7 (1.6) | 2.8 (1.6) |
| Central nervous system, mean (SD) | 2.6 (1.8) | 2.8 (1.7) | 2.8 (1.7) |
| Renal, mean (SD) | 0.5 (0.9) | 0.5 (0.9) | 0.5 (0.9) |
| Admission category, n (%) | |||
| Medicine | 70 (33.3) | 956 (38.1) | 1,026 (37.7) |
| Emergency surgery | 111 (52.9) | 1,023 (40.8) | 1,134 (41.7) |
| Scheduled surgery | 29 (13.8) | 531 (21.2) | 560 (20.6) |
| Charlson comorbidity index (updated), mean (SD) | 1.2 (1.7) | 2.1 (2.4) | 2.0 (2.3) |
| Myocardial infarction, n (%) | 9 (4.3) | 126 (5.0) | 135 (5.0) |
| Congestive heart failure, n (%) | 33 (15.7) | 486 (19.4) | 519 (19.1) |
| Peripheral vascular disease, n (%) | 28 (13.3) | 324 (12.9) | 352 (12.9) |
| Cerebrovascular disease, n (%) | 12 (5.7) | 141 (5.6) | 153 (5.6) |
| Dementia, n (%) | 1 (0.5) | 30 (1.2) | 31 (1.1) |
| Chronic pulmonary disease, n (%) | 26 (12.4) | 373 (14.9) | 399 (14.7) |
| Mild liver disease, n (%) | 8 (3.8) | 38 (1.5) | 46 (1.7) |
| Diabetes without chronic complications, n (%) | 20 (9.5) | 364 (14.5) | 384 (14.1) |
| Diabetes with chronic complications, n (%) | 5 (2.4) | 54 (2.2) | 59 (2.2) |
| Hemiplegia or paraplegia, n (%) | 5 (2.4) | 67 (2.7) | 72 (2.6) |
| Renal disease, n (%) | 21 (10.0) | 423 (16.9) | 444 (16.3) |
| Any malignancy, incl leukemia and lymphoma, n (%) | 23 (11.0) | 418 (16.7) | 441 (16.2) |
| Moderate or severe liver disease, n (%) | 10 (4.8) | 259 (10.3) | 269 (9.9) |
| Metastatic solid tumor, n (%) | 2 (1.0) | 187 (7.5) | 189 (6.9) |
| AIDS/HIV, n (%) | 3 (1.4) | 13 (0.5) | 16 (0.6) |
| Crude mortality rates | |||
| 30-d ICU mortality, n (%) | 60 (28.6) | 451 (18.0) | 511 (18.8) |
| 60-d ICU mortality, n (%) | 69 (32.9) | 470 (18.7) | 539 (19.8) |
| Global ICU mortality, n (%) | 69 (32.9) | 473 (18.8) | 542 (19.9) |
Definition of abbreviations: AIDS = acquired immunodeficiency syndrome; APACHE = Acute Physiology and Chronic Health Evaluation; HIV = human immunodeficiency virus; ICU = intensive care unit; incl = including; Q1 = first quartile or 25th percentile; Q3 = third quartile or 75th percentile; SD = standard deviation; SOFA = Sequential Organ Failure Assessment; VAP = ventilator-associated pneumonia.
VAP-Attributable Fraction of ICU Mortality
Figure 2 and Table 2 provide a comprehensive comparison of the results obtained from the different estimation approaches under study. The observed 10-day, 30-day, and 60-day ICU mortality risk in mechanically ventilated patients was 12.5% (340 ICU deaths of 2,720 patients), 18.9% (513/2720), and 19.9% (540/2720), respectively (t-day ICU mortality risk corresponds with the proportion of all admitted patients who died in the ICU within t days after admission).
Figure 2.
Results of the four different CR analysis approaches for estimating the time-dependent population-attributable fraction (PAF) of intensive care unit (ICU) mortality due to ventilator-associated pneumonia (VAP): a CR analysis restricted to VAP-free patients (approach 1; panel A), a CR analysis that treats VAP as a competing event (approach 2; panel B), a CR analysis that treats VAP as a censoring event (approach 3; panel C), and a CR analysis adjusted for time-dependent confounding (approach 4; panel D). Upper panels: observed cumulative incidence of ICU mortality (black curves) and estimated counterfactual VAP-free cumulative incidence of ICU mortality (gray curves). Lower panels: estimated PAF of ICU death due to VAP (solid lines) and 95% pointwise confidence intervals (shaded areas). CR = competing risk.
Table 2.
Comparison of estimates of the PAF of ICU death due to VAP as obtained by four different competing risk analyses
| 10 Days since ICU Admission | 30 Days since ICU Admission | 60 Days since ICU Admission | |
|---|---|---|---|
| Patients with VAP infection, n (%) | 160 (5.9) | 208 (7.6) | 210 (7.7) |
| VAP-free ICU deaths, n (%) | 316 (11.6) | 452 (16.6) | 471 (17.3) |
| ICU deaths, n (%) | 340 (12.5) | 513 (18.9) | 540 (19.9) |
| Approach 1: competing risk analysis restricted to patients who remain VAP-free until end of follow-up | |||
| Estimated deaths had VAP been eradicated, n (%)* | 342.4 (12.6) | 489.8 (18.0) | 510.4 (18.8) |
| Estimated PAF, % (95% CI) | −0.7 (−3.6 to 0.2) | 4.5 (1.8 to 7.4) | 5.5 (2.9 to 8.2) |
| Approach 2: competing risk analysis that treats VAP acquisition as a competing event | |||
| Estimated deaths had VAP been eradicated, n (%)* | 335.8 (12.3) | 489.4 (18.0) | 510.4 (18.8) |
| Estimated PAF, % (95% CI) | 1.3 (−1.5 to 3.7) | 4.6 (1.9 to 7.4) | 5.5 (2.9 to 8.2) |
| Approach 3: competing risk analysis that treats VAP acquisition as a censoring event | |||
| Estimated deaths had VAP been eradicated, n (%)* | 331.6 (12.2) | 488.6 (18.0) | 512.2 (18.8) |
| Estimated PAF, % (95% CI) | 2.5 (−0.3 to 5.1) | 4.8 (2.0 to 7.7) | 5.2 (2.6 to 7.8) |
| Approach 4: competing risk analysis that adjusts for time-dependent confounding by IP weighting | |||
| Estimated deaths had VAP been eradicated, n (%)* | 333.8 (12.3) | 496.2 (18.2) | 520.1 (19.1) |
| Estimated PAF, % (95% CI) | 1.8 (−1.1 to 4.8) | 3.3 (0.2 to 6.3) | 3.7 (0.8 to 6.6) |
Definition of abbreviations: CI = confidence interval; ICU = intensive care unit; IP = inverse probability; PAF = population-attributable fraction; VAP = ventilator-associated pneumonia.
The counterfactual risk of ICU death by Day t had VAP been prevented for all is estimated by weighing each VAP-free ICU death before or at Day t by a factor that captures the degree of depletion of patients with VAP infection by the end of study follow-up (approach 1), by Day t (approach 2), by the corresponding time of ICU death (approach 3), or with a similar observed covariate history by the corresponding time of ICU death (approach 4). See the online supplement for more details.
Among patients who did not acquire VAP by the end of follow-up (approach 1), the 10-day ICU mortality risk was 12.6% (316/2510). Extrapolated to the original population, it was estimated that if all patients had remained without VAP, approximately 342 patients would have died in the ICU by Day 10. In other words, approach 1 estimated an excess of two deaths by Day 10 in the absence of VAP (PAF = −0.7%; 95% CI, −3.6% to 0.2%). In contrast, it was estimated that 23 ICU deaths could have been prevented by Day 30 (PAF = 4.5%; 95% CI, 1.8–7.4%) and approximately 30 deaths could have been prevented by Day 60 (PAF = 5.5%; 95% CI, 2.9–8.2%) if all patients had remained without VAP.
Approach 2 produced a highly similar estimate to approach 1 at Day 30 and an identical one at Day 60 (because no patients acquired VAP after Day 60), but a positive estimate of the counterfactual risk at Day 10. Among patients who remained without VAP (at least) up to Day 10, the 10-day ICU mortality risk was 12.3% (316/2560). This approach thus suggests that had all patients remained without VAP (at least) up to Day 10, nearly 336 would have died at the ICU by Day 10, corresponding with four prevented deaths by Day 10 (PAF = 1.3%; 95% CI, −1.5% to 3.7%).
Approach 3 estimated that almost eight deaths could have been prevented by Day 10 (PAF = 2.5; 95% CI, −0.3% to 5.1%). At Days 30 and 60, approach 3 produced similar estimates to those of approaches 1 and 2.
Finally, according to approach 4, the 10-day, 30-day, and 60-day ICU mortality risks had all patients remained without VAP were estimated to be 12.3%, 18.2%, and 19.1%, respectively. These estimated risks correspond with considerably lower estimates of the number of preventable cases at longer follow-up as compared with the other approaches: 6.2 by Day 10 (PAF = 1.8%; 95% CI, −1.1% to 4.8%), 16.8 by Day 30 (PAF = 3.3%; 95% CI, 0.2–6.3%), and 19.9 by Day 60 (PAF = 3.7%; 95% CI, 0.8–6.6%).
In sum, the favored benchmark approach 4 demonstrates that ICU mortality attributable to VAP is very limited (approximately 3–4% by Day 60). Shortly after ICU admission (within the first 2 wk), approach 1 indicates a modest but nonnegligible protective effect of VAP, which is not corroborated by more refined analyses (approaches 2–4). When considering larger time windows, approach 4 indicates that the attributable ICU mortality due to VAP may be even smaller compared with estimates obtained from less refined analyses (approaches 1–3).
Discussion
In this study, we aimed to replicate earlier findings of limited VAP-attributable ICU mortality (13) and, in doing so, illustrate the importance of appropriately accounting for the time-dependent nature of the events under study and their confounding factors.
Main Findings
We estimated that by Day 30, 3.3% of deaths in ventilated patients could have been avoided by successful VAP prevention. By Day 60, the proportion of preventable cases increased to 3.7%. Although largely in line with the modest impact reported by Bekaert and colleagues (13) (4.4% on Day 30 and 5.9% on Day 60), we found an even more limited impact. Between-study variability in estimates may be related to a host of factors, such as differences in the patient case mix, diagnostic procedures and definitions, characteristics of infection, timeliness and appropriateness of treatment, application of different (and often suboptimal) statistical methods, and use of various definitions of excess risk (5, 13). In keeping with Bekaert and colleagues (13), we have chosen to focus on the PAF because of its intuitively appealing interpretation and its central role as a target of inference of predominant MSM-based approaches (8). Unlike other effect measures, such as the absolute risk reduction, it implicitly captures VAP incidence because not only lower incidences of ICU mortality but also higher incidences of infection translate into larger PAFs. A differential incidence ratio may explain discrepancies between our findings, based on a cohort in which 7.7% of patients acquired VAP and 19.9% died in the ICU by Day 60 (incidence ratio = 0.39), and those of Bekaert and colleagues (13), who reported incidences of 15.3% and 25.6% respectively (incidence ratio = 0.60). Remarkably, despite lower incidences, our cohort had worse prognosis at ICU admission (Simplified Acute Physiology Score II interquartile range, 59–80 versus 28–53). Our smaller estimates may also, in part, reflect a smaller excess risk of death in patients with a poorer prognosis (13, 31, 32).
Comparison with Findings from Conventional Statistical Approaches
Our focus on the time-dependent PAF facilitated both between-study comparisons and within-study comparisons of alternative estimation approaches targeting the same effect measure. Together with a recent study by von Cube and colleagues (9), our study is among the first to compare different MSM approaches for estimating the time-dependent PAF of ICU mortality due to VAP. Although their comparison did not include an analysis that adjusts for time-dependent confounding (approach 4) or a naive CR analysis (approach 1), we chose to also include these for two reasons. First, we believe that proper adjustment for available time-dependent factors may considerably reduce confounding bias and therefore produce the most reliable benchmark. Second, under certain assumptions, the PAF can be interpreted as the relative mortality reduction in an RCT that randomly assigns eligible patients to receive either a fully effective bundle of preventive measures or standard of care (for more details, see the online supplement). A recent characterization reveals that the compared approaches can be organized hierarchically with respect to how well they emulate this hypothetical prevention trial using observational data (19). To the best of our knowledge, this is the first study to directly compare these four approaches in a hierarchical fashion.
Approach 1 compares the original population with patients who remained VAP-free until death or discharge. This comparison is known to produce immortal time bias because unlike patients who eventually acquired VAP, these patients may not have survived long enough to acquire VAP and, accordingly, did not get apportioned protective “immortal time.” This time-dependent bias is apparent during the first 2 weeks and gradually attenuates, being practically nonexistent at 4 weeks (after which only a few patients acquire VAP) (Figure 2A).
Approach 2 accounts for the time-dependent nature of VAP onset by comparing the original population with patients who had remained VAP-free by each consecutive time wave, which eliminates immortal time bias (8). However, it fails to fully respect the temporal ordering of events because deceased patients receive weights that incorporate information on future events while ignoring information on past events (for more details, see the online supplement). The (modest) protective negative “bump” during the first 10 days (Figure 2B) may partly be explained by this residual, more subtle form of time-dependent bias. In contrast to earlier reports (8), results obtained by this approach can therefore not be causally interpreted, even given sufficient confounding adjustment (9).
Approach 3 (Figure 2C), a less familiar MSM-based approach closer in spirit to causal inference methods for longitudinal data (10, 20, 33–35), fully eliminates all forms of time-dependent bias because it apportions weights that do not incorporate information on future events and exploit additional available information on past events. The residual time-dependent bias produced by approach 2, which can be readily quantified by comparing the results of approaches 2 and 3, is clearest within the first days and weeks. Although it seems to become negligible toward the end of follow-up, bias may remain more pronounced in settings with different temporal dynamics of the considered events (9). In general, residual time-dependent bias is expected to be relatively small whenever exposure prevalence is low, as in our study (although see reference 19 for an example with a higher exposure prevalence).
Finally, approach 4 assigns weights to deceased patients that not only depend on their time of death (as in approach 3), but also on their individual characteristics and (evolution of) disease severity. As in approach 3, this approach compares the original population with the same population in a hypothetical world in which VAP is eradicated, but it no longer naively assumes that on any given day, incident VAP cases are exchangeable with hospitalized VAP-free patients. Comparing results from the third and the last (benchmark) analysis (Figures 2C and 2D) indicates that once measured imbalances over time are adjusted for, the estimated counterfactual cumulative incidence curve more closely matches the observed cumulative incidence curve or, in other words, that a considerable share of patients who die with VAP do not die of VAP. In line with recent findings (9), adjusting only for baseline confounders produces results that are almost identical to an unadjusted analysis (approach 3) (see Figure E5 in the online supplement). This suggests that, in acute settings such as the ICU, imbalances at baseline may often be negligible but accrue over time. Although observational studies based on MSMs have similarly indicated a relatively modest excess risk of ICU death due to VAP (9, 31, 32), these estimates may still have been upwardly biased because of failure to adjust for time-dependent confounding.
Limitations
Although our final estimates may be considered more reliable than those obtained by other approaches, they are also prone to bias. First, residual confounding bias cannot be ruled out because certain confounders may have been either unknown or simply missing from available databases, or because of potential misspecification of the Cox model fitted to calculate IP weights. However, these concerns may be rather limited compared with other studies, as, in accordance with Bekaert and colleagues (13), our analysis adjusted for a much richer set of available time-varying confounders. Second, measurement error in VAP diagnosis as well as timing of its onset (because of an incubation effect) may have affected our analysis. Missed VAP cases may have resulted in underestimation of VAP incidence and, consequently, also the VAP-attributable fraction. Underestimation may also be likely in case of ascertainment bias (e.g., when missed VAP cases would be more common among severely ill patients with a treatment limitation decision; see online supplement for details on how we attempted to tackle this). Finally, as pointed out elsewhere (13), we emphasize that our analysis estimated the fraction of ICU mortality attributable to diagnosed and treated VAP that could not be prevented by current state-of-the-art prevention efforts. As such, we caution against interpreting our findings as an indication to minimize the importance of prevention measures (as current measures may likely have prevented VAP and considerably shortened hospitalization in many patients), accurate diagnosis, or adequate treatment, because these are inherently captured by our estimates.
Conclusions
In conclusion, this study replicates earlier findings of limited ICU mortality attributable to treated VAP given current state-of-the-art VAP prevention measures (13) in an independent cohort using the same causal modeling techniques for estimation. In addition, it provides a compelling illustration that 1) failure to (properly) account for the time-dependent nature of events may misleadingly indicate survival advantages of infected patients shortly after ICU admission and 2) failure to account for the time-dependent nature of confounding leads to systematic overestimation of the PAF, mostly toward end of follow-up.
Supplementary Material
Acknowledgments
Acknowledgment
The authors thank Christian Danneels for the many years of dedicated work. Without him, they could not have realized this study. They also thank two anonymous referees for their valuable feedback and suggestions.
Footnotes
Author Contributions: D.D.B. and J.D. conceived the idea of this replication study. J.S. conceived the idea of a comparison of statistical methods, performed the data analysis, and wrote the first draft. L.D.B., P.D., D.D.B., and J.D. collected the data. B.G. performed data extraction. S.V. provided support for the data analysis. Interpretation of the results was done by J.S., S.V., P.D., D.D.B., and J.D. All authors critically revised the manuscript and approved the final version.
This article has an online supplement, which is accessible from this issue’s table of contents at www.atsjournals.org
Author disclosures are available with the text of this article at www.atsjournals.org.
References
- 1.Graves N, Weinhold D, Tong E, Birrell F, Doidge S, Ramritu P, et al. Effect of healthcare-acquired infection on length of hospital stay and cost. Infect Control Hosp Epidemiol. 2007;28:280–292. doi: 10.1086/512642. [DOI] [PubMed] [Google Scholar]
- 2.De Angelis G, Murthy A, Beyersmann J, Harbarth S. Estimating the impact of healthcare-associated infections on length of stay and costs. Clin Microbiol Infect. 2010;16:1729–1735. doi: 10.1111/j.1469-0691.2010.03332.x. [DOI] [PubMed] [Google Scholar]
- 3.Zimlichman E, Henderson D, Tamir O, Franz C, Song P, Yamin CK, et al. Health care-associated infections: a meta-analysis of costs and financial impact on the US health care system. JAMA Intern Med. 2013;173:2039–2046. doi: 10.1001/jamainternmed.2013.9763. [DOI] [PubMed] [Google Scholar]
- 4.Melsen WG, Rovers MM, Koeman M, Bonten MJM. Estimating the attributable mortality of ventilator-associated pneumonia from randomized prevention studies. Crit Care Med. 2011;39:2736–2742. doi: 10.1097/CCM.0b013e3182281f33. [DOI] [PubMed] [Google Scholar]
- 5.Timsit J-F, Zahar J-R, Chevret S. Attributable mortality of ventilator-associated pneumonia. Curr Opin Crit Care. 2011;17:464–471. doi: 10.1097/MCC.0b013e32834a5ae9. [DOI] [PubMed] [Google Scholar]
- 6.Schumacher M, Allignol A, Beyersmann J, Binder N, Wolkewitz M. Hospital-acquired infections: appropriate statistical treatment is urgently needed! Int J Epidemiol. 2013;42:1502–1508. doi: 10.1093/ije/dyt111. [DOI] [PubMed] [Google Scholar]
- 7.Lee J, Bittner EA. Improving the state of use and understanding of multistate models in critical care. Crit Care Med. 2018;46:1191–1192. doi: 10.1097/CCM.0000000000003162. [DOI] [PubMed] [Google Scholar]
- 8.Schumacher M, Wangler M, Wolkewitz M, Beyersmann J. Attributable mortality due to nosocomial infections: a simple and useful application of multistate models. Methods Inf Med. 2007;46:595–600. [PubMed] [Google Scholar]
- 9.von Cube M, Schumacher M, Bailly S, Timsit J, Lepape A, Savey A, et al. The population-attributable fraction for time-dependent exposures and competing risks-a discussion on estimands. Stat Med. 2019;38:3880–3895. doi: 10.1002/sim.8208. [DOI] [PubMed] [Google Scholar]
- 10.von Cube M, Schumacher M, Wolkewitz M. Causal inference with multistate models—estimands and estimators of the population attributable fraction. J R Stat Soc A. 2020;183:1479–1500. [Google Scholar]
- 11.Bekaert M, Benoit DD, Decruyenaere J, Vansteelandt S. Reply to Wolkewitz et al. Intensive Care Med. 2010;36:550. [Google Scholar]
- 12.Steen J, Vansteelandt S, Benoit DD, Decruyenaere J. Multistate models in critical care: two steps forward, one step back. Crit Care Med. 2019;47:e376. doi: 10.1097/CCM.0000000000003580. [DOI] [PubMed] [Google Scholar]
- 13.Bekaert M, Timsit J-F, Vansteelandt S, Depuydt P, Vésin A, Garrouste-Orgeas M, et al. Outcomerea Study Group. Attributable mortality of ventilator-associated pneumonia: a reappraisal using causal analysis. Am J Respir Crit Care Med. 2011;184:1133–1139. doi: 10.1164/rccm.201105-0867OC. [DOI] [PubMed] [Google Scholar]
- 14.Quan H, Li B, Couris CM, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173:676–682. doi: 10.1093/aje/kwq433. [DOI] [PubMed] [Google Scholar]
- 15.Steurbaut K, Colpaert K, Gadeyne B, Depuydt P, Vosters P, Danneels C, et al. COSARA: integrated service platform for infection surveillance and antibiotic management in the ICU. J Med Syst. 2012;36:3765–3775. doi: 10.1007/s10916-012-9849-8. [DOI] [PubMed] [Google Scholar]
- 16.De Bus L, Diet G, Gadeyne B, Leroux-Roels I, Claeys G, Steurbaut K, et al. Validity analysis of a unique infection surveillance system in the intensive care unit by analysis of a data warehouse built through a workflow-integrated software application. J Hosp Infect. 2014;87:159–164. doi: 10.1016/j.jhin.2014.03.010. [DOI] [PubMed] [Google Scholar]
- 17.De Bus L, Gadeyne B, Steen J, Boelens J, Claeys G, Benoit D, et al. A complete and multifaceted overview of antibiotic use and infection diagnosis in the intensive care unit: results from a prospective four-year registration. Crit Care. 2018;22:241. doi: 10.1186/s13054-018-2178-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.De Bus L, Saerens L, Gadeyne B, Boelens J, Claeys G, De Waele JJ, et al. Development of antibiotic treatment algorithms based on local ecology and respiratory surveillance cultures to restrict the use of broad-spectrum antimicrobial drugs in the treatment of hospital-acquired pneumonia in the intensive care unit: a retrospective analysis. Crit Care. 2014;18:R152. doi: 10.1186/cc13990. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Steen J, Morzywolek P, Van Biesen W, Decruyenaere J, Vansteelandt S.Handling time-dependent exposures and confounders when estimating attributable fractions -- bridging the gap between multistate and counterfactual modeling [preprint]arXiv2020[accessed 2020 Nov 9]. Available from: https://arxiv.org/abs/2011.04833v1
- 20.Keiding N, Klein JP, Horowitz MM. Multi-state models and outcome prediction in bone marrow transplantation. Stat Med. 2001;20:1871–1885. doi: 10.1002/sim.810. [DOI] [PubMed] [Google Scholar]
- 21.Satten GA, Datta S. The Kaplan-Meier estimator as an inverse-probability-of-censoring weighted average. Am Stat. 2001;55:207–210. doi: 10.1198/000313001317098185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Bekaert M, Vansteelandt S, Mertens K. Adjusting for time-varying confounding in the subdistribution analysis of a competing risk. Lifetime Data Anal. 2010;16:45–70. doi: 10.1007/s10985-009-9130-8. [DOI] [PubMed] [Google Scholar]
- 23.Satten GA, Datta S, Robins JM. Estimating the marginal survival function in the presence of time dependent covariates. Stat Probab Lett. 2001;54:397–403. [Google Scholar]
- 24.Robins J, Hernan M.Estimation of the causal effects of time-varying exposures Fitzmaurice G, Davidian M, Verbeke G, Molenberghs G.editors. Longitudinal data analysis.Boca Raton, FL: CRC press; 2008553–599. [Google Scholar]
- 25.Bailly S, Pirracchio R, Timsit J-F. What’s new in the quantification of causal effects from longitudinal cohort studies: a brief introduction to marginal structural models for intensivists. Intensive Care Med. 2016;42:576–579. doi: 10.1007/s00134-015-3919-6. [DOI] [PubMed] [Google Scholar]
- 26.R Core Team. Vienna, Austria: R Foundation for Statistical Computing; 2019. R: a language and environment for statistical computing. [Google Scholar]
- 27.van der Wal WM, Geskus RB. ipw: an R package for inverse probability weighting. J Stat Softw. 2011;43 [Google Scholar]
- 28.Therneau TM.A package for survival analysis in R 2020. Available from: https://cran.r-project.org/package=survival
- 29.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP STROBE Initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335:806–808. doi: 10.1136/bmj.39335.541782.AD. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. RECORD Working Committee. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015;12:e1001885. doi: 10.1371/journal.pmed.1001885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Nguile-Makao M, Zahar JR, Français A, Tabah A, Garrouste-Orgeas M, Allaouchiche B, et al. Attributable mortality of ventilator-associated pneumonia: respective impact of main characteristics at ICU admission and VAP onset using conditional logistic regression and multi-state models. Intensive Care Med. 2010;36:781–789. doi: 10.1007/s00134-010-1824-6. [DOI] [PubMed] [Google Scholar]
- 32.Coeurjolly JF, Nguile-Makao M, Timsit J-F, Liquet B. Attributable risk estimation for adjusted disability multistate models: application to nosocomial infections. Biom J. 2012;54:600–616. doi: 10.1002/bimj.201100222. [DOI] [PubMed] [Google Scholar]
- 33.Arjas E, Eerola M. On predictive causality in longitudinal studies. J Stat Plan Inference. 1993;34:361–386. [Google Scholar]
- 34.Keiding N, Filiberti M, Esbjerg S, Robins JM, Jacobsen N. The graft versus leukemia effect after bone marrow transplantation: a case study using structural nested failure time models. Biometrics. 1999;55:23–28. doi: 10.1111/j.0006-341x.1999.00023.x. [DOI] [PubMed] [Google Scholar]
- 35.Klein JP, Keiding N, Copelan EA. Plotting summary predictions in multistate survival models: probabilities of relapse and death in remission for bone marrow transplantation patients. Stat Med. 1993;12:2315–2332. doi: 10.1002/sim.4780122408. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.