Abstract
Objectives:
Mechanically ventilated emergency department (ED) patients experience high morbidity and mortality. In a prior trial at our center, ED-based lung-protective ventilation was associated with improved care delivery and outcomes. Whether this strategy has persisted in the years after the trial remains unclear. The objective was to assess practice change and clinical outcomes associated with ED lung-protective ventilation.
Design:
Secondary analysis of individual patient-level data from prior clinical trials and cohort studies.
Setting:
ED and ICUs of a single academic center.
Patients:
Mechanically ventilated adults.
Interventions:
A lung-protective ventilator protocol used as the default approach in the ED.
Measurements and Main Results:
The primary ventilator-related outcome was tidal volume, and the primary clinical outcome was hospital mortality. Secondary outcomes included ventilator-, hospital-, and ICU-free days. Multivariable logistic regression, propensity score (PS)-adjustment, and multiple a priori subgroup analyses were used to evaluate outcome as a function of the intervention. A total of 1,796 patients in the pre-intervention period and 1,403 patients in the intervention period were included. In the intervention period, tidal volume was reduced from 8.2 mL/kg PBW (7.3 – 9.1) to 6.5 mL/kg PBW (6.1 – 7.1), and low tidal volume ventilation increased from 46.8% to 96.2%, P < 0.01. The intervention period was associated with lower mortality, (35.9% versus 19.1%), remaining significant after multivariable logistic regression analysis (aOR 0.43; 95% CI 0.35 – 0.53; P < 0.01). Similar results were seen after PS adjustment and in subgroups. The intervention group had more ventilator- [18.8 (10.1) vs. 14.1 (11.9), P <0.01], hospital- [12.2 (9.6) vs. 9.4 (9.5), P <0.01], and ICU-free days [16.6 (10.1) vs. 13.1 (11.1), P <0.01].
Conclusions:
ED lung-protective ventilation has persisted in the years since implementation and was associated with improved outcomes. These data suggest the use of ED-based lung-protective ventilation as a means to improve outcome.
Keywords: emergency department, lung-protective ventilation, mechanical ventilation, ventilator-associated lung injury, implementation
INTRODUCTION
It has been decades since pre-clinical data demonstrated that the mechanical ventilator can injure the lungs [i.e. ventilator-associated lung-injury (VALI)] [1–3]. Subsequently, clinical data showed that a lung-protective ventilator strategy improves outcome in patients with acute respiratory distress syndrome (ARDS), and there is increased realization of benefit in patients without ARDS as well [4–8].
The need for mechanical ventilation in the emergency department (ED) is common and these patients are at high risk of death [9]. In response to the lack of mechanical ventilation data from the ED domain, our research team previously conducted single- and multicenter cohort studies to characterize ventilator practices in the ED [10, 11]. The principal findings were that: 1) the use of high tidal volume was common [e.g. 8.8 (7.8 – 10.0) mL/kg predicted body weight (PBW)]; 2) ARDS was present in a significant proportion of patients in the ED or within 1 to 2 days of intensive care unit (ICU) admission; and 3) ED ventilator settings held influence over early ICU settings. Owing to the clinical outcome data associated with lung-protective ventilation, the time-sensitive nature of VALI (i.e. can occur in hours), and data demonstrating poor adherence to lung-protective ventilation in the ICU, we hypothesized that ED-based lung-protective ventilation could improve both clinical outcomes and adherence to lung protection in the ICU [12]. In 2014, we therefore made a lung-protective ventilation strategy the default approach in our ED, and we studied its impact with a propensity-matched, before-after clinical trial, the Lung-Protective Ventilation Initiated in the Emergency Department (LOV-ED) Trial, which demonstrated improved adherence to lung-protective ventilation in both the ED and ICU and better clinical outcomes [13–15].
Since that time, lung-protective ventilation in our ED has remained the recommended default approach, in collaboration between respiratory therapists and ED clinicians. The objectives of this study were to evaluate whether this default approach has persisted over time and its clinical impact. We hypothesized that an initial default lung-protective ventilation approach in the ED would be associated with persistent practice change and improved clinical outcomes.
METHODS
Study Setting and Design
This was a single-center, secondary cohort analysis of individual patient-level data from prior clinical trials and cohort studies, each of which has been previously published [10, 11, 14–21], and conducted at a tertiary, academic medical center. The pre-intervention group was comprised of patients managed before ED-based lung-protective ventilation was instituted [10, 11, 14–16]. The intervention group consisted of patients managed after ED-based lung-protective ventilation, which began during the intervention period of the LOV-ED Trial (October 2014) [14, 15, 17–20]. All mechanically ventilated patients were enrolled consecutively, and each study was approved by the Human Research Protection Office with waiver of requirement of written informed consent (see Supplemental Digital Content 1 for each approval date and number). The procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation and with the Helsinki Declaration of 1975, as most recently amended. These results are reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement (Supplemental Digital Content 2) [22].
Participants
The cohorts comprising each study were similar in that the inclusion criteria for each were: 1) adult patients age ≥ 18 years; and 2) mechanical ventilation in the ED via an endotracheal tube. Two studies were restricted to patients with sepsis [10, 16]. The LOV-ED Trial included patients without ARDS in the ED; the a priori planned sub-study involved patients with ARDS [14, 15]. The ED-AWARENESS Study required survival to extubation in its eligibility criteria [18]. The inclusion and exclusion criteria for each study are in Supplemental Digital Content 3.
Treatment Interventions
The ED-based default lung-protective ventilation strategy was implemented as part of the LOV-ED Trial, and consisted of a low tidal volume ventilation (LTVV) approach, more appropriate setting of positive end-expiratory pressure (PEEP), and limitation of hyperoxia [13, 14]. The overarching goal was to establish mechanical ventilator settings as an important priority in the ED, standardize care delivery and reduce unnecessary practice variability in order to improve care and outcomes. During the trial, ED respiratory therapists obtained an accurate height with a tape measure, tidal volume was indexed to PBW, and ventilator settings were established per protocol. Since that time, this approach has been protocolized through Respiratory Therapy, and in collaboration with the ED clinical staff.
Assessments and Outcome Measures
Baseline demographics, comorbid conditions, vital signs, laboratory variables, illness severity [i.e. sequential organ failure assessment (SOFA) score], and indication for endotracheal intubation were collected [23]. Process of care variables in the ED included length of stay, duration of ED mechanical ventilation, intubation location, administration of blood products, central venous catheter placement, antibiotics, and vasopressor use.
Mechanical ventilator settings in the ED, airway pressures, pulmonary mechanics, and gas exchange variables were collated. When more than one value was present (e.g. tidal volume), median values were used. The calculated ventilator variables included driving pressure (plateau pressure – PEEP) and static compliance of the respiratory system [tidal volume/ (plateau pressure – PEEP)]. LTVV was defined as the use of tidal volume of ≤ 8 mL/kg PBW, as this was the upper limit of tidal volume allowed by previous investigations in ARDS [5]. Patients were followed until hospital discharge or death.
The primary ventilator-related outcome with respect to practice change was tidal volume and LTVV. The primary clinical outcome was hospital mortality. Secondary outcomes included ventilator-, hospital-, and ICU-free days.
Statistical Analysis
Participants were divided into two cohorts: 1) a pre-intervention group (prior to implementation of ED lung-protective ventilation); and 2) an intervention group (after implementation of ED lung-protective ventilation). Data was analyzed at the individual patient level (versus aggregated study-level data) in order to increase the ability to detect differential treatment effects across the individuals that comprised each study and for better adjustment for confounders [24]. Descriptive statistics, including mean (standard deviation [SD]), median (interquartile range [IQR]), and frequency distributions were used to assess patient characteristics, ventilator variables, and clinical outcomes.
With respect to ventilator settings, data from all included studies were analyzed. Categorical characteristics were compared using the chi-square test, including a comparison of LTVV between the two groups. Continuous characteristics were compared using the independent samples t-test or Mann-Whitney U test. Since data suggest that ED tidal volume has decreased over time, and because before-after analyses are susceptible to temporal changes and random fluctuations over time, we conducted an interrupted time series analysis (ITSA) for the primary ventilator-related outcome of LTVV. This included ten time points (five time points before and after the intervention). Segmented regression of time-series data was used to compare LTVV before and after the intervention, taking into account autocorrelation among the data. The models included an intercept, main intervention effect, linear time for the entire study (time points 1 to 10), and another linear time after the intervention (starting from 6th time point).
With respect to clinical outcomes, the analysis compared outcomes between patients in each intervention period. Data from the ED-AWARENESS Study was excluded from outcome analyses, as survival to extubation was a key inclusion criteria of that study [18, 25]. This resulted in a comparatively less critically ill cohort, which would bias clinical outcome results in favor of the intervention period. Categorical characteristics were compared using the chi-square test. Continuous characteristics were compared using the independent samples t-test or Mann-Whitney U test. A multivariable logistic regression model was used to evaluate hospital mortality (the primary clinical outcome) as a function of the intervention period. Variables known to influence outcome in mechanically ventilated ED patients were chosen a priori for inclusion in the model and included age, illness severity, ED length of stay, and the presence of sepsis [26–31]. Diabetes mellitus and malignancy were also included in the model after demonstration of statistical differences in these variables between the groups, which were also felt to be clinically significant. In addition, since a higher proportion of pre-intervention patients had diabetes mellitus and malignancy, both of which could potentially account for worse outcomes seen in the pre-intervention group, we elected to err conservatively and include those variables in the model. Variables for inclusion or exclusion from the model were selected in sequential fashion and based on the significance level of 0.10 for entry and 0.10 for removal. Adjusted odds ratios (ORs) and corresponding 95% confidence intervals (CI) are reported for the multivariable model, adjusted for all variables in the model.
Given the nonrandomized treatment assignment, multiple pre-planned secondary analyses were conducted to more fully explore the impact of ED-based mechanical ventilation on outcome. First, patient characteristics and outcomes were compared between patients receiving LTVV in the ED, regardless of the intervention period, and the multivariable logistic regression model was repeated, adjusting for LTVV. To screen for heterogeneous treatment effects, a priori subgroups were analyzed according to age (cutoff 60 years), ED length of stay (cutoff 6 hours), illness severity (SOFA score cutoff 4), sepsis, and trauma. To further explore differences according to age and illness severity, a post hoc decision was made to analyze hospital mortality stratified by age and SOFA quartiles.
To balance the covariate distribution between cohorts, we calculated the average treatment effect for the intervention compared to pre-intervention using propensity score-matched analysis. The propensity that a patient was in the intervention group (i.e. propensity score) was modeled using logistic regression as a function of the following: age, severity of illness (total SOFA score), malignancy (yes/no), and sepsis (yes/no). We then performed nearest neighbor matching with 0.10 set as the largest absolute difference compatible with a valid match (i.e. caliper width). We evaluated for covariate balance between the intervention and pre-intervention groups by plotting the standardized percent bias for each covariable for the unmatched and matched groups. We also calculated Rubins’ B (absolute standardized difference of the means of the linear index of the propensity score in the intervention and pre-intervention groups) and Rubin’s R (ratio of the intervention and pre-intervention variances of the propensity score index). A Rubins’ B < 0.25 and a R between 0.5–2 is generally considered sufficiently balanced [32].
Assuming baseline hospital mortality of 30% in the pre-intervention group, the sample size available for analysis would have a 99% power to detect an absolute difference in survival of 5% between groups. Thus, our fixed sample size was large enough to assess for clinically important differences in mortality and conduct multivariable logistic modeling with at least 10:1 event per covariable ratio [33]. All tests were two-tailed, and P value <0.05 was considered statistically significant.
RESULTS
Study population
Supplemental Digital Content 3 shows characteristics of included studies according to intervention group. Five studies were included in the pre-intervention period (n= 1,796) [10, 11, 14–16], and six studies in the intervention period (n= 1,403) [14, 15, 17–20].
Baseline characteristics are shown in Table 1. Missing data existed for sex, race, alcohol abuse, lactate, intubation location, and vasopressor use; these are also shown in Table 1. The pre-intervention group was older, more frequently female, and more often with history of diabetes mellitus, malignancy, and sepsis.
Table 1.
Characteristics of mechanically ventilated patients according to study group.
| Baseline characteristics | Pre-intervention (n= 1796) | Intervention Group (n= 1020) | P value |
|---|---|---|---|
|
| |||
| Age (yr) | 60.2 (16.6) | 55.7 (18.4) | <0.01 |
|
| |||
| Sex, n= 2557 | |||
| Female, n (%) | 715 (46.5) | 416 (40.8) | <0.01 |
|
| |||
| Body mass index (kg/m2) | 28.8 (10.3) | 28.6 (8.9) | 0.54 |
|
| |||
| Race, n= 2557 | <0.01 | ||
| White, n (%) | 662 (43.1) | 432 (42.4) | |
| Black, n (%) | 851 (55.4) | 561 (55.0) | |
| Hispanic, n (%) | 20 (1.3) | 5 (0.5) | |
|
| |||
| Comorbidities, n (%) | |||
| Diabetes mellitus | 634 (35.3) | 315 (30.9) | 0.02 |
| Cirrhosis | 134 (7.5) | 67 (6.6) | 0.38 |
| CHF | 386 (21.5) | 232 (22.7) | 0.44 |
| ESRD/Dialysis | 142 (7.9) | 71 (7.0) | 0.36 |
| COPD | 419 (23.3) | 232 (22.7) | 0.72 |
| Immunosuppression | 221 (12.3) | 114 (11.2) | 0.37 |
| Malignancy | 358 (19.9) | 129 (12.6) | <0.01 |
| Alcohol abuse, n= 2557 | 214 (13.9) | 134 (13.1) | 0.57 |
|
| |||
| Mean arterial pressure (mmHg) | 84.8 (26.3) | 93.6 (34.8) | <0.01 |
|
| |||
| Lactate (mmol/L), n= 2712 | 2.6 (1.6 – 4.8) | 2.7 (1.6 – 5.0) | 0.23 |
|
| |||
| SOFA** | 5.4 (3.2) | 4.8 (3.1) | <0.01 |
|
| |||
| Reason for mechanical ventilation, n (%) | |||
| Sepsis | 771 (42.9) | 299 (29.3) | <0.01 |
| Trauma | 284 (15.8) | 239 (23.4) | |
| COPD | 104 (5.8) | 84 (8.2) | |
| CHF/pulmonary edema | 94 (5.2) | 49 (4.8) | |
| Cardiac arrest | 115 (6.4) | 53 (5.2) | |
| Asthma | 32 (1.8) | 15 (1.5) | |
| Drug overdose | 77 (4.3) | 46 (4.5) | |
| Other | 319 (17.8) | 235 (23.0) | |
|
| |||
| Process of Care Variables | |||
|
| |||
| ED length of stay (hours) | 6.5 (3.8) | 5.8 (3.7) | <0.01 |
|
| |||
| Duration ED mechanical ventilation (hours) | 5.3 (4.0) | 4.4 (2.8) | <0.01 |
|
| |||
| Intubation location, n (%); n= 2462 | <0.01 | ||
| ED | 1167 (80.9) | 766 (75.1) | |
| Transferring facility | 153 (10.6) | 156 (15.3) | |
| Prehospital | 122 ( 8.5) | 98 (9.6) | |
|
| |||
| Vasopressor use, n (%); n= 2811 | 511 (28.5) | 347 (34.2) | <0.01 |
CHF: congestive heart failure; ESRD: end-stage renal disease; COPD: chronic obstructive pulmonary disease; SOFA: sequential organ failure assessment score; ED: emergency department
Continuous variables are reported as mean (standard deviation) and median (interquartile range).
schizophrenia, bipolar disorder, major depression, anxiety
modified score, which excludes Glasgow Coma Scale
for presumed infection (i.e. not for prophylaxis)
Ventilator characteristics
Table 2 shows ED ventilator variables according to study group. The intervention period was associated with significant changes in tidal volume, PEEP, fraction of inspired oxygen (FiO2), and driving pressure (P < 0.01 for all). Tidal volume was reduced from 8.2 mL/kg PBW (7.3 – 9.1) to 6.5 mL/kg PBW (6.1 – 7.1), and LTVV increased from 46.8% to 96.2%, P < 0.01. Figure 1 shows distribution of ED tidal volume according to study group, and Supplemental Digital Content 4 shows ED tidal volume trends over time. Results of the ITSA, and its interpretation, for LTVV are in Figure 2 and Supplemental Digital Content 5. LTVV at the start of the study was 32.7%, then there was a slight linear increase of 4.6% per time interval in the pre-intervention group. There was a significant jump in the provision of LTVV of 46.3% due to the intervention (p < 0.01), and a linear decrease of 7.1% after the intervention. The slope of the time trend after the intervention was −2.5% (−0.0706 + 0.0455), which is lower than that in the pre-intervention group (p < 0.01). Overall, these results suggest the intervention increased the level of LTVV, and decreased the slope of the time trend, as opposed to natural trends over time.
Table 2.
Ventilator variables in the emergency department according to study group.
| Ventilator Variable | Pre-intervention (n= 1,796) | Intervention Group (n= 1403) | * Odds Ratio or Between-Group Difference (95% CI) | P value |
|---|---|---|---|---|
|
| ||||
| Tidal volume, mL | ||||
| Median (IQR) | 500 (500 – 550) | 450 (400 – 500) | ||
| Mean (SD) | 518.6 (71.7) | 435.1 (66.2) | −83.6 (−88.4 to −78.8) | <0.01 |
|
| ||||
| Tidal volume, mL/kg PBW | ||||
| Median (IQR) | 8.2 (7.3 – 9.1) | 6.5 (6.1 – 7.1) | ||
| Mean (SD) | 8.3 (1.6) | 6.6 (1.0) | −1.7 (−1.8 to −1.5) | <0.01 |
|
| ||||
| PEEP | ||||
| Median (IQR) | 5 (5 – 5) | 5 (5 – 8) | ||
| Mean (SD) | 5.5 (1.6) | 6.6 (2.8) | 1.1 (0.9 to 1.3) | <0.01 |
|
| ||||
| FiO2 | ||||
| Median (IQR) | 80 (60 – 100) | 40 (40 – 70) | ||
| Mean (SD) | 76.9 (23.6) | 57.2 (23.8) | −19.7 (−21.3 to −18.0) | <0.01 |
|
| ||||
| Low tidal volume ventilation, n (%) | 826 (46.8) | 1296 (96.2) | 14.9 (11.9 – 18.7) | <0.01 |
|
| ||||
| Ventilator Mode, n (%) | ||||
| VC-AC | 1594 (89.2) | 1249 (89.0) | 0.9 (0.8 – 1.2) | 0.84 |
| PC-AC | 76 (4.3) | 27 (1.9) | 0.4 (0.3 – 0.7) | <0.01 |
| VC-SIMV | 42 (2.4) | 2 (0.1) | 0.1 (0.01 – 0.2) | <0.01 |
| PRVC-AC | 55 (3.1) | 74 (5.3) | 1.8 (1.2 – 2.5) | <0.01 |
|
| ||||
| Plateau pressure, cmH2O | ||||
| Median (IQR) | 19 (16 – 24) | 18 (15 – 23) | ||
| Mean (SD) | 20.5 (5.9) | 19.5 (5.9) | −1.0 (−1.5 to −0.5) | <0.01 |
|
| ||||
| Compliance respiratory system (mL/cm H2O) | ||||
| Median (IQR) | 35.7 (27.8 – 46.2) | 35.7 (26.7 – 45.5) | ||
| Mean (SD) | 40.0 (20.8) | 38.1 (17.8) | −1.9 (−3.4 to −0.5) | 0.01 |
|
| ||||
| Driving Pressure (cm H2O) | ||||
| Median (IQR) | 14 (11 – 18) | 12 (9 – 15) | ||
| Mean (SD) | 15.2 (5.9) | 13.1 (5.3) | −2.1 (−2.5 to −1.6) | <0.01 |
|
| ||||
| PaO2 | ||||
| Median (IQR) | 153 (96 – 235) | 119 (79 – 175) | ||
| Mean (SD) | 182.0 (107.7) | 138.4 (83.0) | −43.6 (−51.2 to −36.0) | <0.01 |
|
| ||||
| PaO2:FiO2 | ||||
| Median (IQR) | 219 (125 – 323) | 238 (140 – 363) | ||
| Mean (SD) | 234.3 (127.8) | 264.5 (156.8) | 30.2 (18.4 to 42.0) | <0.01 |
|
| ||||
| SpO2:FiO2 | ||||
| Median (IQR) | 157 (97 – 240) | 185 (100 – 248) | ||
| Mean (SD) | 161.2 (70.5) | 180.3 (70.9) | 19.1 (8.3 to 30.0) | <0.01 |
IQR: interquartile range; SD: standard deviation; PBW: predicted body weight; PEEP: positive end-expiratory pressure; FiO2: fraction of inspired oxygen; VC: volume control; AC: assist control; PC: pressure control; SIMV: synchronized intermittent mandatory ventilation; PRVC: pressure regulated volume control; PaO2: partial pressure of arterial oxygen; PaCO2: partial pressure of arterial carbon dioxide; SpO2: peripheral oxygen saturation; CI: confidence interval
Odds ratio is presented for binary data and between-group difference is presented as the difference in means for the continuous data.
Figure 1.

Distribution of emergency department tidal volume according to study group.
There was an increase in low-tidal volume ventilation in the ED after implementation of a default lung-protective ventilator protocol (46.8% to 96.2%, P < 0.01).
ED: emergency department; PBW: predicted body weight
Figure 2.

Interrupted time series analysis for the outcome of low tidal volume ventilation (LTVV). LTVV at the start of the study was 32.7%, then there was a slight linear increase of 4.6% per time interval in the pre-intervention group. There was a significant jump in the provision of LTVV of 46.3% due to the intervention (p < 0.01), and a linear decrease of 7.1% after the intervention. The slope of the time trend after the intervention was −2.5% (−0.0706 + 0.0455), which is lower than that in the pre-intervention group (p < 0.01).
AR= autoregressive
Clinical Outcomes
Table 3 shows clinical outcomes by study group. The intervention group had more ventilator- [18.8 (10.1) vs. 14.1 (11.9), P <0.01], hospital- [12.2 (9.6) vs. 9.4 (9.5), P <0.01], and ICU-free days [16.6 (10.1) vs. 13.1 (11.1), P <0.01]. The intervention period was associated with lower mortality, decreased from 35.9% to 19.1%, which remained significant after multivariable logistic regression analysis (aOR 0.43; 95% CI 0.35 – 0.53; P < 0.01) (Table 3). Supplemental Digital Content 6 shows variables included in the adjusted analysis.
Table 3.
Results of outcome analyses according to study group.
| Outcome variable | Pre-intervention (n= 1796) | Intervention Group (n= 1020) | aOR or Between-Group Difference (95% CI) | P value* |
|---|---|---|---|---|
| Mortality, n (%) | 644 (35.9) | 195 (19.1) | 0.43 (0.35 – 0.53) | <0.01 |
| Ventilator-free days | 14.1 (11.9) | 18.8 (10.4) | 4.71 (3.87 to 5.56) | <0.01 |
| Hospital-free days | 9.4 (9.5) | 12.2 (9.6) | 2.81 (2.07 to 3.54) | <0.01 |
| ICU-free days | 13.1 (11.1) | 16.6 (10.1) | 3.48 (2.67 to 4.28) | <0.01 |
aOR: adjusted odds ratio; CI: confidence interval; ICU: intensive care unit
p value for the primary outcome (hospital mortality) was a Wald test estimated using a logistic regression model accounting for age, illness severity, emergency department length of stay, sepsis, diabetes mellitus, malignancy, and the intervention period. p values for the secondary outcomes are from the independent sample t test.
Secondary Analyses
ED ventilator variables according to receipt of LTVV are shown in Supplemental Digital Content 7. The LTVV group had significantly lower tidal volume 6.7 (6.2 – 7.3) mL/kg PBW versus 9.1 mL/kg PBW, P <0.01. Significant differences were also seen for plateau pressure, compliance of the respiratory system, and driving pressure.
Patient characteristics and clinical outcomes according to receipt of LTVV in the ED (irrespective of intervention period) are in Supplemental Digital Content 8 and Supplemental Digital Content 9. The LTVV group had more ventilator- [16.7 (11.4) vs. 14.4 (11.7), P <0.01], hospital- [11.0 (9.7) vs. 9.4 (9.5), P <0.01], and ICU-free days [15.1 (10.7) vs. 13.3 (11.1), P <0.01]. Patients managed with LTVV in the ED had lower mortality (26.5% vs. 34.9%, P <0.01) (Supplemental Digital Content 9), which remained significant after multivariable logistic regression (aOR 0.66; 95% CI 0.55 – 0.79; P < 0.01). Supplemental Digital Content 10 shows the variables included in the adjusted analysis.
Subgroup analyses are in Supplemental Digital Content 11. The primary outcome favored LTVV across all subgroups of age, ED length of stay, illness severity, sepsis, and trauma. The intervention group also had lower mortality across quartiles of age (Supplemental Digital Content 12) and illness severity (Supplemental Digital Content 13).
In the propensity-matched analysis, modeled with logistic regression as a function of age, malignancy, SOFA score, and sepsis, Rubin’s B was 0.10 in the matched cohort, decreased from 0.39 in the unmatched cohort, and Rubin’s R was 1.03. These indicate sufficient balance and that matching worked well. Standardized bias was reduced from >20% to < 10% for all four variables (mean bias decrease from 23.3% to 4.6%; Supplemental Digital Content 14). In this propensity-matched analysis, the intervention group had an average effect size of −0.11 (95% CI −0.15 to −0.07), suggesting the intervention was associated with a decreased probability of hospital mortality of 11%.
DISCUSSION
The ED could be an optimal arena to initiate lung-protective ventilation for several reasons. ED visits for the critically ill are increasing and prolonged lengths of stay are common and associated with worse outcomes [34]. These prolonged lengths of stay provide a window in which VALI, which is time-sensitive and can occur within hours, can be mitigated or propagated. Clinical data support this premise. Multiple studies have demonstrated an association between non-protective ventilation and worsening lung injury, with an increase in ARDS incidence that occurs typically early after admission from the ED (i.e. ICU day 1 or 2) [6, 8, 35–39]. Lung-protective ventilation in the operating room, with similar ventilation duration as that of an ED stay, has been shown to decrease major pulmonary complications and hospital length of stay, and is recommended as the default starting approach to improve outcome [7, 40]. Finally, non-adherence to lung-protective ventilation in the ICU is common and negatively impacts outcome [12, 41, 42]. However, lung-protective ventilation initiated in the ED has been shown to increase adherence in the ICU, suggesting that attention to ED-based mechanical ventilation will increase overall adherence to lung protection [14, 15, 43].
Given our prior work, this cohort study with a before-after implementation design which spanned approximately a decade, was conducted to assess the impact of ED-based lung-protective ventilation. A principal finding was that an ED-based default lung-protective ventilation approach was associated with persistent practice change over time. Tidal volume was lower by close to 2.0 mL/kg PBW, the proportion of patients managed with LTVV increased from 46.8% to 96.2%, and median FiO2 was reduced by 40%. In addition, the intervention was associated with better clinical outcomes.
There are multiple examples demonstrating a lack of real-world sustainability of efficacious and cost-effective interventions, including lung-protective ventilation [12, 44]. Given this fact, along with our prior research focused on mechanically ventilated ED patients, we were in a unique position to examine the sustainability of ED-based lung-protective ventilation over time in order to better inform practicing clinicians and researchers [14]. These data extend the findings of our prior work and provide incremental evidence to suggest that the implementation of ED-based lung-protective ventilation can be sustained over time, and could improve clinical outcomes. We believe our results are secondary to multiple factors, including ongoing education, our local efficacy data and opinion leaders, monitoring over time, resources (e.g. a dedicated ED-based respiratory therapist), and buy-in and collaboration between respiratory therapy and ED clinicians. Most importantly, the intervention has been operationalized with respiratory therapists as the primary participating stakeholders, and has been implemented as the default approach, with intermittent audit and feedback. Default options are those put in place when the alternative is not chosen and they highly influence clinician decisions [45]. These types of “nudges” (i.e. changes to choice architecture without restricting choice) are powerful behavior modifiers, are comparatively easily implemented in order to increase adherence to best evidence, and can be quite effective in domains prone to decision fatigue, such as the ED [45–48].
The intervention was associated with observed clinical outcome benefit, including lower mortality, greater ventilator-, ICU-, and hospital-free days. These findings were stable after multiple analytic approaches, including PS adjustment and subgroup analyses. These data are congruent with recent work and suggest the use of lung-protective ventilation in the ED as the default approach could improve clinical outcome [49]. In addition, additive benefit may have been observed by reducing the delivered FiO2 in the intervention group, though significant uncertainty exists regarding oxygen targets in among the critically ill [50, 51].
There are several limitations to consider. While the sample size is large, it is from only a single center. Therefore, these findings with respect to both practice patterns and outcome may not be generalizable to other sites. The data from each study included a consecutive sample and was obtained during routine care, which is pragmatic and more consistent with real-world clinical care. However, an important point that we stress is that the observational nature of the data limits our ability to assess causation, and the results should only be viewed as associations. Also, combining data from nonrandomized studies increases risk of confounding and statistical heterogeneity [52]. In addition, the study cohort was comprised of only those patients previously enrolled in a clinical study and does not include all mechanically ventilated patients in our ED. We therefore do not have a comparison of the overall proportion of mechanically ventilated patients in each group, giving rise to the potential for selection bias as a contributor to the results. There was also cohort imbalance and likely unmeasured confounders, or other interventions that could account for our observed results. More studies in the pre-intervention group were retrospective, increasing the chance of bias and potential overestimation of the observed effect size. Because of these facts, multiple pre-planned analyses were conducted, each demonstrating consistent results, which enhances causation and internal consistency. However, the likelihood that residual confounding persists is high. In addition, while all analyses were consistent in their direction of effect (i.e. better outcome in the intervention effect), the observed effect size for mortality reduction is unexpectedly high for a tidal volume reduction of 1.7 mL/kg PBW. This likely reflects the single center approach and observational design, and the true effect size is likely quite smaller. The before-after study design can be influenced by changes in care and outcomes that occur over time. The time series analyses point to an impact of the intervention, versus natural trends over time, as the greatest change in clinical practice occurred in the intervention period. Finally, it is recommended that confounders for causal inference be chosen a priori when dealing with observational data [53]. This was our initial approach, and subsequently diabetes mellitus and malignancy were then included in the multivariable model. While this did not change the direction or effect size of the association, it does represent a deviation in our a priori plan.
CONCLUSIONS
This before-after study of mechanically ventilated patients demonstrates that lung-protective ventilation in the ED is associated with significant improvements in the delivery of LTVV ventilation and clinical outcome.
Supplementary Material
KEY POINTS.
Question:
What is the clinical impact associated with using ED-based lung-protective ventilation as the default approach for mechanically ventilated patients?
Findings:
In this secondary analysis of individual patient-level data from prior clinical trials and cohort studies, ED lung-protective ventilation was associated with lower mortality and more ventilator-, ICU-, and hospital-free days.
Meaning:
These findings support the use of lung-protective ventilation in the ED as a means to improve clinical outcome
ACKNOWLEDGMENTS
This work was performed at Washington University School of Medicine in St. Louis.
Footnotes
Conflicts of Interest and Sources of Funding
MHK is supported by the Barnes-Jewish Hospital Foundation. BMF is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) under award number R34HL150404. CRC is supported by the National Institute of Aging under award numbers R21/R33AG058926 and R61/R33 AG069822. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Funders played no role in the following features of the study: study design, data collection, data management, data analysis, data interpretation, writing of the manuscript or decision to submit the manuscript for publication.
All authors report no conflicts of interest.
Copyright Form Disclosure: Dr. Fuller’s institution received funding from the National Heart, Lung, and Blood Institute. Drs. Fuller, Yan, and Roberts received support for article research from the National Institutes of Health. Dr. Mohr’s institution received funding from ESSOS, Inc. The remaining authors have disclosed that they do not have any potential conflicts of interest.
Contributor Information
Brian M. Fuller, Departments of Emergency Medicine and Anesthesiology, Division of Critical Care, Washington University School of Medicine in St. Louis, St. Louis, MO 63110.
Nicholas M. Mohr, Departments of Emergency Medicine and Anesthesiology, Division of Critical Care, Roy J. and Lucille A. Carver College of Medicine, University of Iowa, 200 Hawkins Drive, 1008 RCP, Iowa City, IA 52242.
Enyo Ablordeppey, Departments of Emergency Medicine and Anesthesiology, Division of Critical Care, Washington University School of Medicine in St. Louis, St. Louis, MO 63110.
Olivia Roman, Department of Emergency Medicine, Washington University School of Medicine in St. Louis, St. Louis, MO 63110.
Dylan Mittauer, Washington University School of Medicine in St. Louis, St. Louis, MO 63110.
Yan Yan, Division of Public Health Sciences, Department of Surgery, Division of Biostatistics, Washington University School of Medicine, 418E, 2nd floor, 600 South Taylor Ave., St. Louis, MO 63110.
Marin H. Kollef, Department of Medicine, Division of Pulmonary and Critical Care Medicine, Washington University School of Medicine in St. Louis.
Christopher R. Carpenter, Department of Emergency Medicine, Washington University in St. Louis School of Medicine, 660 South Euclid Avenue, St. Louis, MO 63110.
Brian W. Roberts, Department of Emergency Medicine, Cooper University Hospital, One Cooper Plaza, K152, Camden, NJ.
REFERENCES
- 1.Webb HH, Tierney DF. Experimental pulmonary edema due to intermittent positive pressure ventilation with high inflation pressures. Protection by positive end-expiratory pressure. Amer Rev Respir Dis 1974, 110(5):556–565. [DOI] [PubMed] [Google Scholar]
- 2.Dreyfuss D, Soler P, Basset G, Saumon G. High inflation pressure pulmonary edema: respective effects of high airway pressure, high tidal volume, and positive end-expiratory pressure. Amer Rev Respir Dis 1988, 137(5):1159–1164. [DOI] [PubMed] [Google Scholar]
- 3.Muscedere J, Mullen J, Gan K, Slutsky A. Tidal ventilation at low airway pressures can augment lung injury. Amer J Resp Crit Care Med 1994, 149(5):1327–1334. [DOI] [PubMed] [Google Scholar]
- 4.Hickling K, Henderson S, Jackson R. Low mortality associated with low volume pressure limited ventilation with permissive hypercapnia in severe adult respiratory distress syndrome. Intensive care med 1990, 16(6):372–377. [DOI] [PubMed] [Google Scholar]
- 5.Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. The Acute Respiratory Distress Syndrome Network. N Engl J Med 2000, 342(18):1301–1308. [DOI] [PubMed] [Google Scholar]
- 6.Neto AS, Cardoso SO, Manetta JA, Pereira VGM, Espósito DC, Pasqualucci MdOP et al. Association between use of lung-protective ventilation with lower tidal volumes and clinical outcomes among patients without acute respiratory distress syndrome: a meta-analysis. JAMA 2012, 308(16):1651–1659. [DOI] [PubMed] [Google Scholar]
- 7.Futier E, Constantin J-M, Paugam-Burtz C, Pascal J, Eurin M, Neuschwander A et al. A trial of intraoperative low-tidal-volume ventilation in abdominal surgery. N Engl J Med 2013, 369(5):428–437. [DOI] [PubMed] [Google Scholar]
- 8.Determann RM, Royakkers A, Wolthuis EK, Vlaar AP, Choi G, Paulus F et al. Ventilation with lower tidal volumes as compared with conventional tidal volumes for patients without acute lung injury: a preventive randomized controlled trial. Crit Care 2010, 14(1):1–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Easter BD, Fischer C, Fisher J. The use of mechanical ventilation in the ED. The American Journal of Emergency Medicine 2012, 30(7):1183–1188. [DOI] [PubMed] [Google Scholar]
- 10.Fuller BM, Mohr NM, Dettmer M, Kennedy S, Cullison K, Bavolek R et al. Mechanical ventilation and acute lung injury in emergency department patients with severe sepsis and septic shock: an observational study. Acad Emerg Med 2013, 20(7):659–669. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Fuller BM, Mohr NM, Miller CN, Deitchman AR, Levine BJ, Castagno N et al. Mechanical Ventilation and ARDS in the ED: A Multicenter, Observational, Prospective, Cross-sectional Study. Chest 2015, 148(2):365–374. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Needham DM, Colantuoni E, Mendez-Tellez PA, Dinglas VD, Sevransky JE, Himmelfarb CRD et al. Lung protective mechanical ventilation and two year survival in patients with acute lung injury: prospective cohort study. BMJ 2012, 344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Fuller BM, Ferguson I, Mohr NM, Stephens RJ, Briscoe CC, Kolomiets AA et al. Lung-protective ventilation initiated in the emergency department (LOV-ED): a study protocol for a quasi-experimental, before-after trial aimed at reducing pulmonary complications. BMJ Open 2016, 6(4):e010991. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Fuller BM, Ferguson IT, Mohr NM, Drewry AM, Palmer C, Wessman BT et al. Lung-protective ventilation initiated in the emergency department (LOV-ED): a quasi-experimental, before-after trial. Ann Emerg Med 2017, 70(3):406–418. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Fuller BM, Ferguson IT, Mohr NM, Drewry AM, Palmer C, Wessman BT et al. A quasi-experimental, before-after trial examining the impact of an emergency department mechanical ventilator protocol on clinical outcomes and lung-protective ventilation in acute respiratory distress syndrome. Crit Care Med 2017, 45(4):645–652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dettmer MR, Mohr NM, Fuller BM. Sepsis-associated pulmonary complications in emergency department patients monitored with serial lactate: an observational cohort study. Journal of critical care 2015, 30(6):1163–1168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fuller BM, Roberts BW, Mohr NM, Knight IV WA, Adeoye O, Pappal RD et al. The ED-SED Study: A Multicenter, Prospective Cohort Study of Practice Patterns and Clinical Outcomes Associated With Emergency Department SEDation for Mechanically Ventilated Patients. Critical Care Medicine 2019, 47(11):1539–1548. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Pappal RD, Roberts BW, Mohr NM, Ablordeppey E, Wessman BT, Drewry AM et al. The ED-AWARENESS Study: A Prospective, Observational Cohort Study of Awareness With Paralysis in Mechanically Ventilated Patients Admitted From the Emergency Department. Annals of Emergency Medicine 2021, 77(5):532–544. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Fuller BM, Roberts BW, Mohr NM, Pappal RD, Stephens RJ, Yan Y et al. A study protocol for a multicentre, prospective, before-and-after trial evaluating the feasibility of implementing targeted SEDation after initiation of mechanical ventilation in the emergency department (The ED-SED Pilot Trial). BMJ Open 2020, 10(12):e041987. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Stephens R, Evans E, Pajor M, Pappal R, Kollef MH, Roberts BW, Mohr NM, Fuller BM. THE IMPACT OF EARLY DEEP SEDATION IN MECHANICALLY VENTILATED PATIENTS DURING THE COVID-19 PANDEMIC. Crit Care Med 2022, 50(1):126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Fuller BM, Roberts BW, Mohr NM, Faine B, Drewry AM, Wessman BT et al. The Feasibility of Implementing Targeted SEDation in Mechanically Ventilated Emergency Department Patients: The ED-SED Pilot Trial. Critical Care Medicine 2022, 50(8): 1224–1235. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. Annals of Internal Medicine 2007, 147(8):573–577. [DOI] [PubMed] [Google Scholar]
- 23.Vincent J, De Mendonça A, Cantraine F, et al. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Critical care medicine 1998, 26(11):1793–1800. [DOI] [PubMed] [Google Scholar]
- 24.Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. Bmj 2010, 340. [DOI] [PubMed] [Google Scholar]
- 25.Pappal RD, Roberts BW, Mohr NM, Ablordeppey E, Wessman BT, Drewry AM et al. Protocol for a prospective, observational cohort study of awareness in mechanically ventilated patients admitted from the emergency department: the ED-AWARENESS Study. BMJ Open 2019, 9(10):e033379. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Sudarsanam T, Jeyaseelan L, Thomas K, John G. Predictors of mortality in mechanically ventilated patients. Postgraduate medical journal 2005, 81(962):780–783. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Hersch M, Izbicki G, Dahan D, Breuer GS, Nesher G, Einav S. Predictors of mortality of mechanically ventilated patients in internal medicine wards. Journal of Critical Care 2012, 27(6):694–701. [DOI] [PubMed] [Google Scholar]
- 28.Lai C-C, Shieh J-M, Chiang S-R, Chiang K-H, Weng S-F, Ho C-H et al. The outcomes and prognostic factors of patients requiring prolonged mechanical ventilation. Scientific Reports 2016, 6(1):1–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Esteban A, Anzueto A, Frutos F, Alía I, Brochard L, Stewart TE et al. Characteristics and outcomes in adult patients receiving mechanical ventilation: a 28-day international study. Jama 2002, 287(3):345–355. [DOI] [PubMed] [Google Scholar]
- 30.Angotti LB, Richards JB, Fisher DF, Sankoff JD, Seigel TA, Al Ashry HS et al. Duration of mechanical ventilation in the emergency department. Western Journal of Emergency Medicine 2017, 18(5):972. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.García-Gigorro R, de la Cruz Vigo F, Andrés-Esteban E, Chacón-Al S, Varas GM, Sánchez-Izquierdo J et al. Impact on patient outcome of emergency department length of stay prior to ICU admission. Medicina Intensiva (English Edition) 2017, 41(4):201–208. [DOI] [PubMed] [Google Scholar]
- 32.Rubin DB. Using propensity scores to help design observational studies: application to the tobacco litigation. Health Services and Outcomes Research Methodology 2001, 2(3):169–188. [Google Scholar]
- 33.Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epi 1996, 49(12):1373–1379. [DOI] [PubMed] [Google Scholar]
- 34.Mohr NM, Wessman BT, Bassin B, Elie-Turenne MC, Ellender T, Emlet LL et al. Boarding of critically ill patients in the emergency department. Journal of the American College of Emergency Physicians Open 2020, 1(4):423–431. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Gajic O, Frutos-Vivar F, Esteban A, Hubmayr RD, Anzueto A. Ventilator settings as a risk factor for acute respiratory distress syndrome in mechanically ventilated patients. Intensive care medicine 2005, 31(7):922–926. [DOI] [PubMed] [Google Scholar]
- 36.Jia X, Malhotra A, Saeed M, Mark RG, Talmor D. Risk factors for ARDS in patients receiving mechanical ventilation for> 48 h*. Chest 2008, 133(4):853–861. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Mascia L, Zavala E, Bosma K, et al. High tidal volume is associated with the development of acute lung injury after severe brain injury: An international observational study*. Critical care medicine 2007, 35(8):1815–1820. [DOI] [PubMed] [Google Scholar]
- 38.Pasero D, Davi A, Guerriero F, et al. High tidal volume as an independent risk factor for acute lung injury after cardiac surgery. Intensive care medicine 2008, 34(Supplement 1):0398. [Google Scholar]
- 39.Yilmaz M, Keegan M, Iscimen R, et al. Toward the prevention of acute lung injury: Protocol-guided limitation of large tidal volume ventilation and inappropriate transfusion*. Critical care medicine 2007, 35(7):1660–1666. [DOI] [PubMed] [Google Scholar]
- 40.Young CC, Harris EM, Vacchiano C, Bodnar S, Bukowy B, Elliott RRD et al. Lung-protective ventilation for the surgical patient: international expert panel-based consensus recommendations. British Journal of Anaesthesia 2019, 123(6):898–913. [DOI] [PubMed] [Google Scholar]
- 41.Needham DM, Yang T, Dinglas VD, Mendez-Tellez PA, Shanholtz C, Sevransky JE et al. Timing of low tidal volume ventilation and intensive care unit mortality in acute respiratory distress syndrome. A prospective cohort study. Am J Respir Crit Care Med 2015, 191(2):177–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Bellani G, Laffey JG, Pham T, Fan E, Brochard L, Esteban A et al. Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 countries. JAMA 2016, 315(8):788–800. [DOI] [PubMed] [Google Scholar]
- 43.De Monnin K, Terian E, Yaegar LH, Pappal RD, Mohr NM, Roberts BW et al. Low Tidal Volume Ventilation for Emergency Department Patients: A Systematic Review and Meta-Analysis on Practice Patterns and Clinical Impact. Critical Care Medicine 2022. [DOI] [PubMed] [Google Scholar]
- 44.Seow H, Phillips CO, Rich MW, Spertus JA, Krumholz HM, Lynn J. Isolation of health services research from practice and policy: the example of chronic heart failure management. Journal of the American Geriatrics Society 2006, 54(3):535–540. [DOI] [PubMed] [Google Scholar]
- 45.Halpern SD. Using default options and other nudges to improve critical care. Critical care medicine 2018, 46(3):460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Courtright KR, Dress EM, Singh J, Bayes BA, Chowdhury M, Small DS et al. Prognosticating Outcomes and Nudging Decisions with Electronic Records in the Intensive Care Unit Trial Protocol. Annals of the American Thoracic Society 2021, 18(2):336–346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Last BS, Buttenheim AM, Timon CE, Mitra N, Beidas RS. Systematic review of clinician-directed nudges in healthcare contexts. BMJ open 2021, 11(7):e048801. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Wallace DJ, Coleman Drake DM. Nudges in the Care of the Critically Ill. Critical care medicine 2019, 47(8):1150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Fernando SM, Fan E, Rochwerg B, Burns KE, Brochard LJ, Cook DJ et al. Lung-protective ventilation and associated outcomes and costs among patients receiving invasive mechanical ventilation in the ED. Chest 2021, 159(2):606–618. [DOI] [PubMed] [Google Scholar]
- 50.Cumpstey AF, Oldman AH, Martin DS, Smith A, Grocott MP. Oxygen targets during mechanical ventilation in the ICU: a systematic review and meta-analysis. Critical Care Explorations 2022, 4(4). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Page D, Ablordeppey E, Wessman BT, Mohr NM, Trzeciak S, Kollef MH et al. Emergency department hyperoxia is associated with increased mortality in mechanically ventilated patients: a cohort study. Critical Care 2018, 22(1):1–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Reeves B, Deeks J, Higgins J, Wells G. on behalf of the Cochrane Non-Randomised Studies Methods Group. Chapter 13: including non-randomized studies. Cochrane handbook for systematic reviews of interventions version 2008, 5(0). [Google Scholar]
- 53.Maslove DM, Leisman DE. Causal inference from observational data: New guidance from pulmonary, critical care, and sleep journals. Crit Care Med 2019, 47(1):1–2. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
