Key Points
Question
Were the 2013 New York State regulations mandating the use of protocols for sepsis recognition and treatment associated with in-hospital mortality differences compared with states that did not implement sepsis regulations?
Findings
In this retrospective cohort study of 1 012 410 hospitalized adults with sepsis, mandated protocolized sepsis care in New York State was associated with a significantly greater decline in risk-adjusted mortality in New York compared with a group of control states that did not implement mandated protocolized sepsis care. By the 10th quarter after implementation of the regulations, the adjusted absolute mortality was 3.2% lower than expected in New York State relative to the control states.
Meaning
The New York State sepsis regulations were associated with significantly reduced sepsis mortality, but whether broader adoption of state-level sepsis mandates in other states would lead to further reductions in sepsis-related mortality is unknown.
Abstract
Importance
Beginning in 2013, New York State implemented regulations mandating that hospitals implement evidence-based protocols for sepsis management, as well as report data on protocol adherence and clinical outcomes to the state government. The association between these mandates and sepsis outcomes is unknown.
Objective
To evaluate the association between New York State sepsis regulations and the outcomes of patients hospitalized with sepsis.
Design, Setting, and Participants
Retrospective cohort study of adult patients hospitalized with sepsis in New York State and in 4 control states (Florida, Maryland, Massachusetts, and New Jersey) using all-payer hospital discharge data (January 1, 2011-September 30, 2015) and a comparative interrupted time series analytic approach.
Exposures
Hospitalization for sepsis before (January 1, 2011-March 31, 2013) vs after (April 1, 2013-September 30, 2015) implementation of the 2013 New York State sepsis regulations.
Main Outcomes and Measures
The primary outcome was 30-day in-hospital mortality. Secondary outcomes were intensive care unit admission rates, central venous catheter use, Clostridium difficile infection rates, and hospital length of stay.
Results
The final analysis included 1 012 410 sepsis admissions to 509 hospitals. The mean age was 69.5 years (SD, 16.4 years) and 47.9% were female. In New York State and in the control states, 139 019 and 289 225 patients, respectively, were admitted before implementation of the sepsis regulations and 186 767 and 397 399 patients, respectively, were admitted after implementation of the sepsis regulations. Unadjusted 30-day in-hospital mortality was 26.3% in New York State and 22.0% in the control states before the regulations, and was 22.0% in New York State and 19.1% in the control states after the regulations. Adjusting for patient and hospital characteristics as well as preregulation temporal trends and season, mortality after implementation of the regulations decreased significantly in New York State relative to the control states (P = .02 for the joint test of the comparative interrupted time series estimates). For example, by the 10th quarter after implementation of the regulations, adjusted absolute mortality was 3.2% (95% CI, 1.0% to 5.4%) lower than expected in New York State relative to the control states (P = .004). The regulations were associated with no significant differences in intensive care unit admission rates (P = .09) (10th quarter adjusted difference, 2.8% [95% CI, −1.7% to 7.2%], P = .22), a significant relative decrease in hospital length of stay (P = .04) (10th quarter adjusted difference, 0.50 days [95% CI, −0.47 to 1.47 days], P = .31), a significant relative decrease in the C difficile infection rate (P < .001) (10th quarter adjusted difference, −1.8% [95% CI, −2.6% to −1.0%], P < .001), and a significant relative increase in central venous catheter use (P = .02) (10th quarter adjusted difference, 4.8% [95% CI, 2.3% to 7.4%], P < .001).
Conclusions and Relevance
In New York State, mandated protocolized sepsis care was associated with a greater decrease in sepsis mortality compared with sepsis mortality in control states that did not implement sepsis regulations. Because baseline mortality rates differed between New York and comparison states, it is uncertain whether these findings are generalizable to other states.
This cohort study uses hospital discharge data to compare 30-day in-hospital mortality of patients with sepsis in New York State before vs after implementation of regulations mandating rapid administration of antibiotics and fluid resuscitation and compares those data with mortality in 4 comparison states (Florida, Maryland, Massachusetts, and New Jersey) without such a mandate.
Introduction
Sepsis is a leading cause of morbidity and mortality in the United States.1 Several treatments are of proven effectiveness in this population, including timely administration of antibiotics and early resuscitation with intravenous fluids.2 However, many patients with sepsis do not receive these evidence-based practices, leading to excess morbidity and mortality.3,4,5 To address this problem, policy makers are increasingly turning to regulatory mechanisms designed to mandate sepsis performance improvement in the form of care protocols for early recognition and treatment.6
A pioneering example of these mandates is the regulations issued by the New York State Department of Health during May 2013, known as Rory’s Regulations after a 12-year-old boy who died of sepsis.7 These regulations require all acute care hospitals in the state to develop and implement protocols for timely recognition and treatment of sepsis, including administration of antibiotics by 3 hours and resuscitation with intravenous fluid by 6 hours for patients with signs of hypoperfusion. The regulations also require hospitals to routinely train their staff in protocol implementation and report both protocol adherence and clinical outcomes to the state’s department of health.
Although timely sepsis treatment is supported by robust observational and clinical trial data,8,9,10 the role of governmental mandates as a strategy to enforce the use of sepsis protocols remains controversial.11 Sepsis mandates could encourage uptake of evidence-based care practices, leading to reduced mortality, but could also encourage overuse of intravenous fluids and antibiotics, leading to adverse consequences.7 The goal of this study was to examine sepsis outcomes before and after implementation of the sepsis regulations in New York State, comparing these changes with the outcomes in other states that did not implement sepsis regulations during this time.
Methods
Study Design
We performed a retrospective cohort study of hospitalized patients with sepsis. The study was approved by the University of Pittsburgh Human Research Protection Office, which deemed the study exempt from human subjects review because it was a secondary analysis of existing data and waived the need for informed consent (PRO17110272).
We used a comparative interrupted time series study design, comparing New York State with the 4 control states of Florida, Maryland, Massachusetts, and New Jersey. These control states were chosen because they have similar demographic characteristics to New York and, except for Florida, they are geographically proximal to New York. A comparative interrupted time series study compares the longitudinal outcome changes between an intervention group and a control group, thereby subtracting underlying secular trends and any other changes that may have occurred in both groups. By using this approach, we were able make inferences about the regulations that would not be possible using New York State data alone.12
To support the rigor and reproducibility of our results, all analyses were prespecified prior to receipt of the final data set and a detailed statistical analysis plan (Supplement 1) was published online13 (additional details appear in the eMethods in Supplement 2). Deviations from this plan due to unforeseen circumstances are noted as post hoc, and a rationale for all deviations appears in the eMethods in Supplement 2.
Data Sources
Our primary data source was the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project State Inpatient Database. The State Inpatient Database contains patient-level administrative data for all hospitalizations in participating states. We used the State Inpatient Database to identify hospitalizations that occurred from January 1, 2011, through September 30, 2015. We linked the data from the State Inpatient Database to hospital-level data from the 2015 Centers for Medicare & Medicaid Healthcare Cost Reporting Information System to obtain hospital characteristics such as hospital type, number of beds, and academic status; and used the 2010 US Census to obtain data on each hospital’s metropolitan statistical area population.
Patients and Hospitals
We identified hospital admissions with sepsis using validated International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes for infection and organ failure.14 This strategy, known as the Dombrovskiy strategy, is less specific but more sensitive than approaches that rely solely on the explicit ICD-9-CM codes for sepsis,15 and captures a slightly larger patient population than is identified by retrospective chart review.16 In choosing a broad sepsis identification strategy, we sought to account for the fact that many patients with sepsis may be missed by chart review yet are still eligible for evidence-based practices. We excluded admissions for patients younger than 18 years, admissions to hospitals that could not be identified in the Healthcare Cost Reporting Information System, and admissions with missing data for key covariates. We further excluded hospitals that were not classified as short-stay acute care hospitals by the Healthcare Cost Reporting Information System, hospitals with no sepsis admissions, and, to create a more homogenous sample, hospital types that were not shared across New York State and the control states both before and after the introduction of the regulations (more detail on this process appears in the eMethods in Supplement 2).
Outcomes
The primary outcome variable was 30-day in-hospital mortality. We also examined 4 secondary outcome variables that reflected potential adverse unintended consequences of the regulations: intensive care unit (ICU) admission rate, hospital length of stay, central venous catheter use, and Clostridium difficile infection rate. The ICU admission rates were examined as a marker for health care intensity because data suggest that protocolized sepsis treatment may increase ICU admissions.17 Hospital length of stay was examined as a proxy for resource use because data suggest that protocolized sepsis treatment may increase hospital costs.18 Rates for central venous catheter use were examined based on the hypothesis that the sepsis mandate could lead to an increase in invasive central catheter insertion for monitoring and resuscitation.11 C difficile infection rates were examined based on the hypothesis that the sepsis mandate may encourage antibiotic overuse, leading to an increase in cases of C difficile infection.11
Additional Variables
Variables for case-mix adjustment were based on a previously published risk-adjustment model for sepsis,19 and included age; sex; race and ethnicity; admission through the emergency department; transfer from an acute care hospital; cases of organ failure present at admission, which were defined similar to the study by Elias et al20; sepsis infection categories, which were defined similar to the study by Ames et al21; and Elixhauser chronic comorbid conditions.22 We included race as a potential confounder based on prior studies demonstrating an independent relationship between race and sepsis outcomes.23 Race and ethnicity were obtained directly from the hospital discharge record, which is based on patient self-report either directly or via the patient’s primary insurer. A full list of variables and their definitions appears in eTable 1 in Supplement 2.
Statistical Analysis
Primary Analysis
The hospital characteristics were compared between New York State and the control states using the χ2 test. We examined patient characteristics between New York State and the control states before and after implementation of the regulations, but did not formally test for differences because the large sample size made it likely that all tests would be significant. For cases in which unexpected differences were found between New York State and the control states, additional post hoc comparisons were performed to better understand and provide additional context for the results.
A comparative interrupted time series analysis was performed to test the relationship between the New York State sepsis regulations and outcomes.24,25 We performed the analysis separately for each outcome variable described above. In specifying the models, we accounted for the possibility that the association between the regulations and the outcomes might change over time due to their staged implementation (eTable 2 in Supplement 2 contains a complete policy timeline). Accordingly, rather than specifying a single postimplementation temporal trend, we fit a model with indicators for each postimplementation quarter.26,27 The preimplementation period was defined as hospital discharge from January 1, 2011, through March 31, 2013, before the filing of the regulations. The postimplementation period was defined as hospital discharge from April 1, 2013, through September 30, 2015.
All models were fit using linear regression with robust standard errors clustered at the hospital level. For binary outcomes, this approach corresponds to a linear probability model and the coefficients represent the between-group absolute risk differences. All models controlled for the patient and hospital characteristics listed above, as well as seasonality based on calendar quarter.28 We also controlled for preregulation temporal trends using a continuous time variable, implemented as quarters, as well as a treatment indicator × continuous time variable interaction term. This approach accounts for the fact that sepsis outcomes were generally improving over time,29 as well as the possibility that preimplementation temporal trends might differ between New York State and the control states.26,27
To test the association between the regulations and the patient outcomes, we included a postimplementation quarter × intervention group interaction term (ie, New York State vs the control states). The estimates for these interaction terms are interpreted as the difference in the deviations from the counterfactual preregulation trend between New York State and the control states during that quarter; or, more simply, the estimated association between the regulations and the patient outcomes during the given quarter. The primary test of the association between the regulations and the patient outcomes was a joint test that all of the quarter-specific estimates were equal to zero. To understand the direction and magnitude of any observed overall associations, we also calculated point estimates and 95% CIs for each individual interaction term. The results from the 10th quarter after implementation (ie, during 2015, quarter 3) are highlighted herein as a representative example of the quarter-specific associations.
The comparative interrupted time series model can be simplified to a traditional difference-in-differences model if trends for the outcomes were parallel in New York State and in the control states prior to the regulations. We directly examined for this possibility by fitting a model containing a treatment indicator, a continuous time variable, the interaction of these 2 variables, and all patient- and hospital-level covariates, restricted to the preregulation period. We did this separately for each outcome. We considered parallel trends as being present if the interaction term from this model was not significant. In cases in which there were parallel trends, we simplified the comparative interrupted time series model to a difference-in-differences model by excluding the term for the interaction of the treatment indicator with the continuous time variable.26,27 The P values from the tests of parallel trends and the respective models used are presented alongside the model results.
Secondary and Sensitivity Analyses
A concern regarding the use of administrative codes to identify sepsis is that the regulations could have changed sepsis coding patterns, potentially biasing the results. To understand this issue, a secondary analysis was performed in which we fit a similar model as described above, except with all adult hospital admissions as the population and an indicator for sepsis as the primary dependent variable. A negative test of the interaction terms would indicate that the regulations were not associated with changes in administrative coding for sepsis.
A number of prespecified sensitivity analyses were also performed to examine the robustness of the results to our design decisions. Specifically, we repeated the primary analysis, limiting the sample to patients with severe sepsis and septic shock as defined using the ICD-9-CM codes that explicitly identify sepsis and septic shock15; expanding the sample to patients with sepsis according to a broader definition that includes additional organ failures15; excluding New York City hospitals that had participated in an earlier Greater New York Hospital Association sepsis quality improvement initiative30; and shifting the preimplementation period back in time by 2 quarters to account for the possibility that hospitals began implementing the regulations when they were first announced. One post hoc sensitivity analysis was performed, restricting the control states to those with preregulation temporal trends that were most similar to New York to account for the possibility that the results were driven by unmeasured differences between New York State and the control states.
In addition, we performed subgroup analyses on our primary model based on age, number of comorbidities, number of organ failures, emergency department use, hospital size, hospital academic status, and hospital sepsis volume. For each subgroup, we tested for heterogeneity of the association between the regulations and patient outcomes using 3-way interaction terms and applying the Bonferroni method to correct for multiple comparisons.
Because of the potential for type I error due to multiple comparisons, findings for the analyses of the secondary end points should be interpreted as exploratory. Additional details about the modeling strategy appear in Supplement 1 and in the eMethods in Supplement 2. Statistical analyses were performed using Stata version 15.0 (StataCorp). All tests were 2-sided and a P value of .05 or less was considered significant.
Results
Patients and Hospitals
A patient flow diagram appears in the eFigure in Supplement 2. The final analysis included 1 012 410 sepsis admissions to 509 hospitals. After patient and hospital exclusions, there were 325 786 sepsis admissions to 163 hospitals in New York State, and 686 624 sepsis admissions to 346 hospitals in the control states. The mean age was 69.5 years (SD, 16.4 years) and 47.9% were female. In New York State and in the control states, 139 019 and 289 225 patients, respectively, were admitted before implementation of the sepsis regulations and 186 767 and 397 399 patients, respectively, were admitted after implementation of the sepsis regulations. Compared with hospitals in the control states, the hospitals in New York State were more likely to be teaching hospitals and tended to have smaller ICUs (Table 1). The characteristics of patients with sepsis were generally similar in New York State and in the control states both before and after implementation of the sepsis regulations (Table 2).
Table 1. Characteristics of Study Hospitals.
| Characteristic | No. (%)a | P Value | |
|---|---|---|---|
| New York State (n = 163) |
Control States: FL, MA, MD, and NJ (n = 346) |
||
| Size and teaching statusb | |||
| Small teaching | 28 (17.2) | 84 (24.3) | <.001 |
| Large teaching | 59 (36.2) | 57 (16.5) | |
| Nonteaching | 76 (46.6) | 205 (59.2) | |
| No. of hospital beds | |||
| <100 | 47 (28.8) | 65 (18.8) | .02 |
| 100-250 | 54 (33.1) | 146 (42.2) | |
| >250 | 62 (38.0) | 135 (39.0) | |
| No. of intensive care unit beds | |||
| ≤10 | 60 (36.8) | 78 (22.5) | .001 |
| 11-25 | 34 (20.9) | 113 (32.7) | |
| >25 | 69 (42.3) | 155 (44.8) | |
| Metropolitan statistical area population sizec | |||
| <100 000 | 25 (15.3) | 33 (9.5) | .15 |
| 100 000-1 million | 41 (25.2) | 88 (25.4) | |
| ≥1 million | 97 (59.5) | 225 (65.0) | |
| Sepsis case volume, cases/quarterd | |||
| <51 | 65 (39.9) | 104 (30.1) | .08 |
| ≥51-<125 | 47 (28.8) | 123 (35.5) | |
| ≥125 | 51 (31.3) | 119 (34.4) | |
Data are from 2015 unless otherwise indicated.
Teaching status was obtained from the Centers for Medicare & Medicaid Healthcare Cost Report and Information System and size was defined using the full-time resident-to-bed ratio. A large teaching hospital had a full-time resident-to-bed ratio of 0.2 or greater; small, greater than 0 and less than 0.2; nonteaching, 0.
Based on the population of the metropolitan statistical area containing the hospital ZIP code.
Data are based on mean per quarter volume during quarters with any observations.
Table 2. Admission Characteristics, Discharge Disposition, and Unadjusted Study Outcomes for Patients With Sepsis in the Primary Analysis.
| New York State | Control States: FL, MA, MD, and NJ | |||
|---|---|---|---|---|
| Preregulation (n = 139 019) |
Postregulation (n = 186 767) |
Preregulation (n = 289 225) |
Postregulation (n = 397 399) |
|
| Admission Characteristics | ||||
| Age, mean (SD), y | 71.1 (16.1) | 70.4 (16.4) | 69.2 (16.4) | 68.8 (16.4) |
| Sex, No. (%) | ||||
| Male | 70 557 (50.8) | 95 845 (51.3) | 150 511 (52.0) | 210 120 (52.9) |
| Female | 68 462 (49.2) | 90 922 (48.7) | 138 714 (48.0) | 187 279 (47.1) |
| Race/ethnicity, No. (%) | ||||
| White | 84 555 (60.8) | 113 139 (60.6) | 203 462 (70.3) | 278 063 (70.0) |
| Black | 23 718 (17.1) | 30 606 (16.4) | 51 724 (17.9) | 66 490 (16.7) |
| Hispanic | 13 705 (9.9) | 17 622 (9.4) | 24 626 (8.5) | 38 575 (9.7) |
| Othera | 17 041 (12.3) | 25 400 (13.6) | 9413 (3.3) | 14 271 (3.6) |
| Admission, No. (%) | ||||
| Via emergency department | 118 707 (85.4) | 163 066 (87.3) | 256 806 (88.8) | 356 233 (89.6) |
| Transfer from another hospital | 6504 (4.7) | 8754 (4.7) | 11 620 (4.0) | 16 202 (4.1) |
| Comorbidities, No. (%)b | ||||
| 0-1 | 9735 (7.0) | 10 265 (5.5) | 15 898 (5.5) | 20 303 (5.1) |
| 2-3 | 47 738 (34.3) | 51 371 (27.5) | 75 644 (26.2) | 103 075 (25.9) |
| ≥4 | 81 546 (58.7) | 125 131 (67.0) | 197 683 (68.3) | 274 021 (69.0) |
| No. of failed organs at admission, No. (%)b | ||||
| 0 | 37 798 (27.2) | 46 352 (24.8) | 68 663 (23.7) | 82 655 (20.8) |
| 1 | 62 382 (44.9) | 83 957 (45.0) | 123 555 (42.7) | 171 597 (43.2) |
| 2-3 | 35 667 (25.7) | 51 460 (27.6) | 87 668 (30.3) | 128 720 (32.4) |
| ≥4 | 3172 (2.3) | 4998 (2.7) | 9339 (3.2) | 14 427 (3.6) |
| Discharge Disposition, No. (%) | ||||
| Home | 41 758 (30.0) | 66 861 (35.8) | 92 115 (31.8) | 143 121 (36.0) |
| Transfer to another acute care hospital | 4112 (3.0) | 5171 (2.8) | 12 510 (4.3) | 16 721 (4.2) |
| Transfer to a postacute care facilityc | 50 922 (36.6) | 67 692 (36.2) | 114 260 (39.5) | 153 465 (38.6) |
| Died | 41 403 (29.8) | 45 704 (24.5) | 68 176 (23.6) | 80 363 (20.2) |
| Other | 824 (0.6) | 1339 (0.7) | 2164 (0.7) | 3729 (0.9) |
| Unadjusted Study Outcomes | ||||
| 30-d In-hospital mortality, No. (%) | 36 536 (26.3) | 41 108 (22.0) | 63 725 (22.0) | 75 872 (19.1) |
| ICU admission, No. (%) | 82 345 (59.2) | 104 846 (56.1) | 221 082 (76.4) | 297 776 (74.9) |
| Hospital length of stay, median (IQR), d | 10 (5-19) | 9 (5-17) | 8 (4-15) | 8 (4-14) |
| Central venous catheter use, No. (%) | 51 814 (37.3) | 66 420 (35.6) | 138 906 (48.0) | 171 702 (43.2) |
| Clostridium difficile infection, No. (%) | 13 347 (9.6) | 13 872 (7.4) | 23 852 (8.2) | 27 916 (7.0) |
Abbreviations: ICU, intensive care unit; IQR, interquartile range.
Asian or Pacific Islander, Native American, or missing.
Ascertained from International Classification of Diseases, Ninth Revision, Clinical Modification billing codes in the administrative record.20,22
Skilled nursing facility, inpatient rehabilitation facility, or long-term acute care hospital.
Unadjusted mortality was modestly higher in New York State compared with the control states both before and after implementation of the sepsis regulations (Table 2). Unadjusted mortality was 26.3% in New York State and 22.0% in the control states before implementation of the sepsis regulations, and was 22.0% in New York State and 19.1% in the control states after implementation of the sepsis regulations. The ICU admission rates were substantially lower in New York State compared with the control states (preregulation period: 59.2% vs 76.4%, respectively); a post hoc analysis indicated that these differences were related to differences in ICU bed supply (eTable 3 in Supplement 2), with similar ICU admission rates at hospitals with similar numbers of beds (eg, preregulation ICU admission rates at hospitals with <10 beds: 52.3% in New York State vs 65.4% in the control states). Central venous catheter use was also substantially lower in New York State compared with the control states (preregulation period: 37.3% vs 48.0%, respectively); a post hoc analysis indicated that these differences were related to differences in ICU admission rates (eTable 4 in Supplement 2), with similar rates of central venous catheter use conditional on admission to the ICU (eg, preregulation central venous catheter use in patients admitted to the ICU: 49.0% in New York State vs 55.0% in the control states).
Primary and Secondary Outcome Analyses
The results of the primary and secondary outcome analyses appear in Table 3. Controlling for patient characteristics, hospital characteristics, seasonality, and preregulation temporal trends, the introduction of the regulations was associated with a significant relative decrease in the adjusted risk of 30-day in-hospital mortality (primary outcome; P = .02 for the joint test of the comparative interrupted time series estimates). The association was consistent across all periods. For the 10 postregulation quarters, all point estimates were negative and 7 were statistically significantly different than 0.
Table 3. Adjusted Quarter-Specific Estimates of the Association Between the New York State Sepsis Regulations and Patient Outcomes From the Analysis for the Primary Outcome and the 4 Secondary Outcomes.
| Primary Outcome | Secondary Outcomes | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 30-d In-hospital Mortality, % (95% CI)a | P Value | ICU Admission Rate, % (95% CI)a |
P Value | Hospital Length of Stay, d (95% CI)a |
P Value | Central Venous Catheter Use, % (95% CI)a | P Value |
Clostridium difficile Infection Rate, % (95% CI)a |
P Value | |
| Preregulationb | NA | NA | NA | NA | NA | |||||
| 2013 | ||||||||||
| Quarter 2 | −2.0 (−3.2 to −0.7) | .002 | 1.2 (−0.5 to 3.0) | .16 | 0.87 (0.29 to 1.45) | .003 | −0.2 (−1.9 to 1.6) | .86 | −0.3 (−1.1 to 0.5) | .45 |
| Quarter 3 | −1.4 (−2.8 to 0) | .05 | 0.9 (−1.2 to 3.1) | .41 | 0.38 (−0.22 to 0.98) | .21 | 1.2 (−0.5 to 2.9) | .16 | −1.3 (−2.1 to −0.6) | <.001 |
| Quarter 4 | −1.6 (−2.9 to −0.3) | .02 | 0.5 (−1.9 to 3.0) | .66 | 0.08 (−0.48 to 0.64) | .78 | 1.2 (−0.4 to 2.9) | .14 | −0.4 (−1.1 to 0.3) | .26 |
| 2014 | ||||||||||
| Quarter 1 | −2.0 (−3.4 to −0.6) | .005 | 0.6 (−2.2 to 3.5) | .67 | −0.24 (−0.86 to 0.37) | .43 | 1.8 (0 to 3.6) | .05 | −0.7 (−1.4 to 0.1) | .08 |
| Quarter 2 | −1.8 (−3.6 to 0) | .05 | −0.9 (−4.3 to 2.4) | .58 | 0.75 (−0.11 to 1.60) | .09 | 1.2 (−0.9 to 3.3) | .26 | −0.6 (−1.4 to 0.2) | .13 |
| Quarter 3 | −1.6 (−3.3 to 0.1) | .06 | 1.5 (−2.2 to 5.2) | .42 | 0.54 (−0.22 to 1.31) | .16 | 2.3 (0.5 to 4.2) | .01 | −1.3 (−2.2 to −0.4) | .003 |
| Quarter 4 | −2.3 (−3.9 to −0.6) | .008 | 1.6 (−2.0 to 5.3) | .38 | 0.19 (−0.54 to 0.93) | .61 | 2.8 (0.8 to 4.9) | .006 | −0.7 (−1.5 to 0.1) | .07 |
| 2015 | ||||||||||
| Quarter 1 | −2.4 (−4.5 to −0.3) | .02 | 1.6 (−2.3 to 5.5) | .42 | 0.33 (−0.58 to 1.24) | .48 | 3.0 (1.0 to 5.0) | .003 | −1.9 (−2.7 to −1.1) | <.001 |
| Quarter 2 | −3.7 (−6.0 to −1.5) | .001 | 2.3 (−2.0 to 6.6) | .29 | 0.61 (−0.38 to 1.61) | .23 | 2.4 (0 to 4.9) | .05 | −1.1 (−2.0 to −0.3) | .009 |
| Quarter 3 | −3.2 (−5.4 to −1.0) | .004 | 2.8 (−1.7 to 7.2) | .22 | 0.50 (−0.47 to 1.47) | .31 | 4.8 (2.3 to 7.4) | <.001 | −1.8 (−2.6 to −1.0) | <.001 |
| Test of parallel trendsc | .01 | .04 | .004 | .80 | .38 | |||||
| Model typec | CITS | CITS | CITS | DID | DID | |||||
| Joint test of significanced | .02 | .09 | .04 | .02 | <.001 | |||||
Abbreviations: CITS, comparative interrupted time series; DID, difference-in-differences; ICU, intensive care unit; NA, data not available.
Adjusted for all patient and hospital characteristics as well as preregulation temporal trends and season. Estimates are interpreted as the difference in that quarter between the adjusted outcome and the adjusted counterfactual trend in New York State compared with the 4 control states.
The preregulation period includes hospitalizations from January 1, 2011, through March 31, 2013.
Tested using a model containing a treatment indicator, a continuous time variable, the interaction of these 2 variables, and all patient- and hospital-level covariates, restricted to the preregulation period. The reported P values are from the test of significance for the interaction term from this model. In cases in which the interaction term was statistically significant, a CITS model was used. Otherwise, it was assumed that there were parallel temporal trends during the preregulation period and a DID model was used. These model decisions are outlined in the prespecified statistical analysis plan in Supplement 1 and described further in the eMethods in Supplement 2.
Examines whether the differences between the counterfactual vs actual outcomes are different between New York State and the 4 control states. This is the primary test of the association between the regulations and the patient outcomes per the prespecified statistical analysis plan.
For example, during the last study quarter after implementation of the regulations (July 2015 through September 2015), the adjusted absolute mortality was 3.2% (95% CI, 1.0%-5.4%) lower than expected in New York State relative to the control states (P = .004). This estimate represents the absolute reduction in mortality associated with the regulations during this quarter. These results, along with unadjusted mortality trends before and after implementation of the sepsis regulations are shown graphically in Figure 1. The figure suggests that baseline risk-adjusted mortality was higher in New York State compared with the control states, declined over time in both groups both before and after implementation of the sepsis regulations, and declined more rapidly over time in New York State after implementation, although not to the point that it was level with the control states. Quarterly absolute adjusted mortality rates for New York State and the control states appear in eTable 5 in Supplement 2.
Figure 1. Quarter-Specific Estimates of the Primary Outcome.
The preregulation period includes hospitalizations from January 1, 2011, through March 31, 2013. The postregulation period includes hospitalizations from April 1, 2013, through September 30, 2015.
In A, the circles and squares represent the per-quarter estimates along with fitted ordinary least-squares lines within the preperiod and postperiod. In B, comparative interrupted time series model results are shown. During the preregulation period, the dotted lines represent the risk-adjusted temporal trends in outcome from the fully adjusted model. These lines are extended into the postregulation period, where they represent the counterfactual adjusted outcomes had the preregulation trends continued. The circles and squares represent the adjusted quarter-specific postintervention estimates from the fully adjusted model. Values above the dotted line indicate that adjusted values for that quarter are higher than the counterfactual, and values below the dotted line indicate that adjusted values are lower than the counterfactual. Estimates were adjusted for age, sex, race and ethnicity, admission through the emergency department, transfer from an acute care hospital, number of organ failures present at hospital admission, sepsis infection categories, chronic comorbid conditions, hospital characteristics, season, and preregulation temporal trends.
The results for the secondary outcomes appear in Table 3 and in Figure 2 (with the quarterly absolute adjusted outcomes for New York State and the control states in eTables 6-9 in Supplement 2). The sepsis regulations were associated with no significant differences in ICU admission rates (P = .09 for the joint test of significance) (10th quarter [labeled 2015, quarter 3, in Table 3] adjusted difference, 2.8% [95% CI, −1.7% to 7.2%], P = .22), a significant relative decrease in C difficile infection rates (P < .001 for the joint test of significance) (10th quarter adjusted difference, −1.8% [95% CI, −2.6% to −1.0%], P < .001), and a significant relative increase in central venous catheter use (P = .02 for the joint test significance) (10th quarter adjusted difference, 4.8% [95% CI, 2.3% to 7.4%], P < .001). Although the overall estimates for hospital length of stay were significant (P = .04 for the joint test of significance), these results appeared to be related to early differences with no significant differences later in the study period (10th quarter adjusted difference, 0.50 days [95% CI, −0.47 to 1.47 days], P = .31).
Figure 2. Quarter-Specific Estimates of the 4 Secondary Outcomes.
In A, C, E, and G, the circles and squares represent the per-quarter estimates along with fitted ordinary least-squares lines within the preperiod and postperiod. In B, D, F, and H, the circles and squares represent quarter-specific postintervention estimates from the fully adjusted model. During the preregulation period, the dotted lines represent the risk-adjusted temporal trends in outcome from the fully adjusted model. These lines are extended into the postregulation period, where they represent the counterfactual adjusted outcomes had the preregulation trends continued.
Sensitivity Analyses
There was no significant association between the regulations and administrative coding of sepsis, indicating that the results are unlikely to be related to changes in coding practices (eTable 10 in Supplement 2). Repeating the analysis of 30-day in-hospital mortality using 2 different administrative definitions for sepsis, excluding hospitals that participated in an earlier Greater New York Hospital Association sepsis quality improvement initiative,30 and shifting the preimplementation period back by 2 quarters yielded similar results to the primary analysis (eTables 11-14 in Supplement 2).
Repeating the primary analysis in states with preregulation temporal trends that were not significantly different than New York State also yielded similar estimates; however, the results were not statistically significant in light of the smaller sample size and risk estimates that were slightly closer to the null (eTable 15 in Supplement 2). The observed associations were not significantly different among any prespecified subgroups (eTable 16 in Supplement 2).
Discussion
A health policy mandate requiring protocolized sepsis care in New York State was associated with a statistically significant reduction in risk-adjusted mortality among adults hospitalized with sepsis compared with 4 control states that had not adopted such regulations.
The mechanism of this finding is unknown. It is likely related to a combination of factors, including increased use of early antibiotics and intravenous fluids and staff education as mandated by the regulations. It may also be reflective of a postregulation organizational culture shift that emphasizes continuous quality improvement for acutely ill patients.
There was no evidence of a change in ICU use associated with the regulations, indicating no large-scale changes in treatment intensity. However, there was a significant relative increase in central venous catheter use, which might occur if the regulations were associated with more insertions of central venous catheters to facilitate intensive physiological monitoring. It is notable that this difference appeared to be related to a steeper decline in use of central venous catheters in the 4 control states compared with New York State. This finding may be related to the publication of a large clinical trial suggesting the lack of a role for early-goal directed sepsis treatment, which occurred about the same time as the New York State sepsis regulations.10 It is likely that this trial’s publication was associated with a general decrease in central venous catheter use that was offset by an increase in New York State under the regulations.
The regulations were also associated with a significant relative increase in hospital length of stay; however, this finding was not consistent across the postregulation periods. In addition, implementation of the sepsis regulations was associated with a significant relative decrease in C difficile infection rates. Although the hypothesis was that implementation of the sepsis regulations would be associated with more antibiotic use and thus more C difficile infection, it is possible that better sepsis treatment was associated with decreased cases of organ failure, which lowered the risk for C difficile. Because the analysis was limited to patients with sepsis, it is still possible that the regulations were associated with antibiotic use and C difficile infection in the general population.
These results extend the findings from recent studies8,31 using clinical data reported to the New York State Department of Health under the regulations. These studies showed that early use of antibiotics was associated with lower mortality,8 and that sepsis outcomes in New York State were improving over time,31 observations that are consistent with others in the sepsis literature.9,29 Although these studies demonstrate the value of early sepsis treatment, they could not directly examine the regulations because sepsis outcomes are known to be improving over time.29,32 The present study overcomes that limitation by using data from before implementation of the sepsis regulations and data from 4 control states that did not implement the sepsis regulations.
Taken together, these results provide support for the New York State sepsis policy and others like it. To our knowledge, this policy is the first example of a government-issued sepsis policy designed to incentivize quality improvement by mandating evidence-based care.7 Similar policies were later implemented in Illinois and New Jersey, and a number of other states have sepsis policies under development. At the same time, the observed patterns of sepsis mortality raise concerns about the generalizability of the findings outside of New York State. Specifically, baseline sepsis mortality was higher in New York State compared with the 4 control states, and mortality was declining less rapidly in New York State compared with the control states prior to implementation of the sepsis regulations. Thus, the regulations may have served primarily to correct relatively poor-quality sepsis care in New York State. States with high-quality sepsis care may not see similar results if they adopt sepsis regulations.
Limitations
This study has several limitations. First, by identifying sepsis using administrative codes it is possible that some patients in the study did not have sepsis or that some patients with sepsis were missed. However, there was no evidence that the regulations were associated with sepsis coding patterns in a way that would cause bias, and the primary coding schema yielded a patient population similar to that identified through retrospective chart review, mitigating this concern.19 It is also possible that other changes in coding patterns (such as differential upcoding of suspected sepsis diagnoses, or better documentation of comorbidities in New York State vs the control states) could have influenced the results, but the similarities between the adjusted and unadjusted findings make this unlikely.
Second, the study did not examine postdischarge outcomes such as postdischarge mortality, long-term mortality, or functional status. However, there was no evidence of an increase in postacute care use in New York State after implementation of the regulations, suggesting that the observed mortality findings are not simply due to earlier discharge to postacute care (Table 2).
Third, the study was unable to directly test the relationship between the sepsis protocol and outcome. Rather, the study was designed to examine the policy as a whole, rather than sepsis protocols, which are just 1 part of the larger policy.
Fourth, the results could be sensitive to the modeling approach. However, the results were robust to several sensitivity analyses addressing that possibility, and the act of prepublishing a statistical analysis plan reduced the likelihood that the findings are an artifact of the statistical methods.33
Conclusions
In New York State, mandated protocolized sepsis care was associated with a greater decrease in sepsis mortality compared with sepsis mortality in control states that did not implement sepsis regulations. Because baseline mortality rates differed between New York and comparison states, it is uncertain whether these findings are generalizable to other states.
Statistical analysis plan
eMethods.
eTable 1. Full list of variables and their definitions
eTable 2. Complete policy timeline
eTable 3. Post hoc analysis of ICU admission rates
eTable 4. Post hoc analysis of central line insertion rates
eTable 5. Primary analysis of in-hospital mortality-by 30-days
eTable 6. Primary analysis of intensive care unit admission rates
eTable 7. Primary analysis of hospital length of stay
eTable 8. Primary analysis of central venous catheter rates
eTable 9. Primary analysis of C. difficile rates
eTable 10. Supplementary analysis of sepsis coding
eTable 11. Sensitivity analysis defining sepsis using explicit sepsis codes
eTable 12. Sensitivity analysis defining sepsis using the modified Angus codes
eTable 13. Sensitivity analysis excluding hospitals in greater New York city
eTable 14. Sensitivity analysis shifting the pre-regulation period back in time
eTable 15. Sensitivity analysis varying the control states
eTable 16. Subgroup analyses
eFigure. Patient flow diagram
References
- 1.Singer M, Deutschman CS, Seymour CW, et al. . The third internatonal consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):801-810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Dellinger RP, Levy MM, Rhodes A, et al. ; Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup . Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 2013;39(2):165-228. doi: 10.1007/s00134-012-2769-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Levy MM, Dellinger RP, Townsend SR, et al. . The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010;36(2):222-231. doi: 10.1007/s00134-009-1738-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Levy MM, Rhodes A, Phillips GS, et al. . Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study. Crit Care Med. 2015;43(1):3-12. doi: 10.1097/CCM.0000000000000723 [DOI] [PubMed] [Google Scholar]
- 5.Rhodes A, Phillips G, Beale R, et al. . The Surviving Sepsis Campaign bundles and outcome: results from the International Multicentre Prevalence Study on Sepsis (the IMPreSS study). Intensive Care Med. 2015;41(9):1620-1628. doi: 10.1007/s00134-015-3906-y [DOI] [PubMed] [Google Scholar]
- 6.Cooke CR, Iwashyna TJ. Sepsis mandates: improving inpatient care while advancing quality improvement. JAMA. 2014;312(14):1397-1398. doi: 10.1001/jama.2014.11350 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hershey TB, Kahn JM. State sepsis mandates—a new era for regulation of hospital quality. N Engl J Med. 2017;376(24):2311-2313. doi: 10.1056/NEJMp1611928 [DOI] [PubMed] [Google Scholar]
- 8.Seymour CW, Gesten F, Prescott HC, et al. . Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med. 2017;376(23):2235-2244. doi: 10.1056/NEJMoa1703058 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Liu VX, Fielding-Singh V, Greene JD, et al. . The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med. 2017;196(7):856-863. doi: 10.1164/rccm.201609-1848OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yealy DM, Kellum JA, Huang DT, et al. ; ProCESS Investigators . A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683-1693. doi: 10.1056/NEJMoa1401602 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Rhee C, Gohil S, Klompas M. Regulatory mandates for sepsis care—reasons for caution. N Engl J Med. 2014;370(18):1673-1676. doi: 10.1056/NEJMp1400276 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Dimick JB, Ryan AM. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA. 2014;312(22):2401-2402. doi: 10.1001/jama.2014.16153 [DOI] [PubMed] [Google Scholar]
- 13.Kahn JM. Quantitative evaluation of the effects the 2013 New York State sepsis regulations. https://osf.io/jcwdv/. Accessed May 18, 2018.
- 14.Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007;35(5):1244-1250. doi: 10.1097/01.CCM.0000261890.41311.E9 [DOI] [PubMed] [Google Scholar]
- 15.Iwashyna TJ, Odden A, Rohde J, et al. . Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis. Med Care. 2014;52(6):e39-e43. doi: 10.1097/MLR.0b013e318268ac86 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Prescott HC, Cope TM, Gesten FC, et al. . Reporting of sepsis cases for performance measurement versus for reimbursement in New York State. Crit Care Med. 2018;46(5):666-673. doi: 10.1097/CCM.0000000000003005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Angus DC, Barnato AE, Bell D, et al. . A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators. Intensive Care Med. 2015;41(9):1549-1560. doi: 10.1007/s00134-015-3822-1 [DOI] [PubMed] [Google Scholar]
- 18.Rowan KM, Angus DC, Bailey M, et al. ; PRISM Investigators . Early, goal-directed therapy for septic shock—a patient-level meta-analysis. N Engl J Med. 2017;376(23):2223-2234. doi: 10.1056/NEJMoa1701380 [DOI] [PubMed] [Google Scholar]
- 19.Darby JL, Davis BS, Barbash IJ, Kahn JM. An administrative model for benchmarking hospitals on their 30-day sepsis mortality. BMC Health Serv Res. 2019;19(1):221. doi: 10.1186/s12913-019-4037-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Elias KM, Moromizato T, Gibbons FK, Christopher KB. Derivation and validation of the acute organ failure score to predict outcome in critically ill patients: a cohort study. Crit Care Med. 2015;43(4):856-864. doi: 10.1097/CCM.0000000000000858 [DOI] [PubMed] [Google Scholar]
- 21.Ames SG, Davis BS, Angus DC, Carcillo JA, Kahn JM. Hospital variation in risk-adjusted pediatric sepsis mortality. Pediatr Crit Care Med. 2018;19(5):390-396. doi: 10.1097/PCC.0000000000001502 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Quan H, Sundararajan V, Halfon P, et al. . Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130-1139. doi: 10.1097/01.mlr.0000182534.19832.83 [DOI] [PubMed] [Google Scholar]
- 23.Mayr FB, Yende S, Linde-Zwirble WT, et al. . Infection rate and acute organ dysfunction risk as explanations for racial differences in severe sepsis. JAMA. 2010;303(24):2495-2503. doi: 10.1001/jama.2010.851 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Abadie A. Semiparametric difference-in-differences estimators. Rev Econ Stud. 2005;72(1):1-19. doi: 10.1111/0034-6527.00321 [DOI] [Google Scholar]
- 25.Imbens GW, Wooldridge JM. Recent developments in the econometrics of program evaluation. J Econ Lit. 2009;47(1):5-86. doi: 10.1257/jel.47.1.5 [DOI] [Google Scholar]
- 26.Jacob R, Somers M-A, Zhu P, Bloom H. The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Eval Rev. 2016;40(3):167-198. doi: 10.1177/0193841X16663414 [DOI] [PubMed] [Google Scholar]
- 27.St Clair T, Hallberg K, Cook TD. The validity and precision of the comparative interrupted time-series design: three within-study comparisons. J Educ Behav Stat. 2016;41(3):269-299. doi: 10.3102/1076998616636854 [DOI] [Google Scholar]
- 28.Danai PA, Sinha S, Moss M, Haber MJ, Martin GS. Seasonal variation in the epidemiology of sepsis. Crit Care Med. 2007;35(2):410-415. doi: 10.1097/01.CCM.0000253405.17038.43 [DOI] [PubMed] [Google Scholar]
- 29.Kaukonen K-M, Bailey M, Suzuki S, Pilcher D, Bellomo R. Mortality related to severe sepsis and septic shock among critically ill patients in Australia and New Zealand, 2000-2012. JAMA. 2014;311(13):1308-1316. doi: 10.1001/jama.2014.2637 [DOI] [PubMed] [Google Scholar]
- 30.Greater New York Hospital Association STOP sepsis. https://www.gnyha.org/topic/sepsis/. Accessed May 18, 2018.
- 31.Levy MM, Gesten FC, Phillips GS, et al. . Mortality changes associated with mandated public reporting for sepsis: results of the New York State initiative. Am J Respir Crit Care Med. 2018;198(11):1406-1412. doi: 10.1164/rccm.201712-2545OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Rhee C, Dantes R, Epstein L, et al. ; CDC Prevention Epicenter Program . Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA. 2017;318(13):1241-1249. doi: 10.1001/jama.2017.13836 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Thomas L, Peterson ED. The value of statistical analysis plans in observational research: defining high-quality research from the start. JAMA. 2012;308(8):773-774. doi: 10.1001/jama.2012.9502 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Statistical analysis plan
eMethods.
eTable 1. Full list of variables and their definitions
eTable 2. Complete policy timeline
eTable 3. Post hoc analysis of ICU admission rates
eTable 4. Post hoc analysis of central line insertion rates
eTable 5. Primary analysis of in-hospital mortality-by 30-days
eTable 6. Primary analysis of intensive care unit admission rates
eTable 7. Primary analysis of hospital length of stay
eTable 8. Primary analysis of central venous catheter rates
eTable 9. Primary analysis of C. difficile rates
eTable 10. Supplementary analysis of sepsis coding
eTable 11. Sensitivity analysis defining sepsis using explicit sepsis codes
eTable 12. Sensitivity analysis defining sepsis using the modified Angus codes
eTable 13. Sensitivity analysis excluding hospitals in greater New York city
eTable 14. Sensitivity analysis shifting the pre-regulation period back in time
eTable 15. Sensitivity analysis varying the control states
eTable 16. Subgroup analyses
eFigure. Patient flow diagram