Mortality Changes Associated with Mandated Public Reporting for Sepsis. The Results of the New York State Initiative

Abstract

Rationale: In 2013, the New York State Department of Health (NYSDOH) began a mandatory state-wide initiative to improve early recognition and treatment of severe sepsis and septic shock.

Objectives: This study examines protocol initiation, 3-hour and 6-hour sepsis bundle completion, and risk-adjusted hospital mortality among adult patients with severe sepsis and septic shock.

Methods: Cohort analysis included all patients from all 185 hospitals in New York State reported to the NYSDOH from April 1, 2014, to June 30, 2016. A total of 113,380 cases were submitted to NYSDOH, of which 91,357 hospitalizations from 183 hospitals met study inclusion criteria. NYSDOH required all hospitals to submit and follow evidence-informed protocols (including elements of 3-h and 6-h sepsis bundles: lactate measurement, early blood cultures and antibiotic administration, fluids, and vasopressors) for early identification and treatment of severe sepsis or septic shock.

Measurements and Main Results: Compliance with elements of the sepsis bundles and risk-adjusted mortality were studied. Of 91,357 patients, 74,293 (81.3%) had the sepsis protocol initiated. Among these individuals, 3-hour bundle compliance increased from 53.4% to 64.7% during the study period (P < 0.001), whereas among those eligible for the 6-hour bundle (n = 35,307) compliance increased from 23.9% to 30.8% (P < 0.001). Risk-adjusted mortality decreased from 28.8% to 24.4% (P < 0.001) in patients among whom a sepsis protocol was initiated. Greater hospital compliance with 3-hour and 6-hour bundles was associated with shorter length of stay and lower risk and reliability-adjusted mortality.

Conclusions: New York’s statewide initiative increased compliance with sepsis-performance measures. Risk-adjusted sepsis mortality decreased during the initiative and was associated with increased hospital-level compliance.

At a Glance Commentary

Scientific Knowledge on the Subject

The New York State initiative was the first mandated public reporting initiative for sepsis, and it has been followed by similar initiatives by the U.S. Centers for Medicare and Medicaid Services and other states. However, there remain limited and mixed data on the impact of mandated public reporting programs generally and, in particular, for sepsis.

What This Study Adds to the Field

This study reports on the protocol initiation, 3-hour and 6-hour sepsis bundle completion, and risk-adjusted hospital mortality among adult patients with severe sepsis and septic shock over a 2-year period after implementation of the New York State initiative. The study adds important insights into the role mandated reporting may play in driving clinician behavior.

Sepsis is a common, lethal, and costly illness (1–4). In 2013, New York State Department of Health (NYSDOH) began a state-wide initiative to improve the early recognition and treatment of patients with severe sepsis and septic shock (5). It was motivated in part by the tragic case in 2012 of Rory Staunton, a previously healthy adolescent, who died of septic shock. The regulations required all hospitals in the state to develop, and submit for review and approval by the department, “evidence informed” protocols to recognize and treat patients with severe sepsis and septic shock. Reporting of patient-specific data to NYSDOH to evaluate sepsis process and mortality outcomes began April 1, 2014. Neither financial penalties nor incentives were associated with the program.

Over the past 30 years, there has been substantial growth in performance measurement and public reporting programs in health care. These approaches are intended to improve in health care quality and cost-effectiveness (6, 7). However, there is substantial uncertainty regarding the effectiveness of these approaches and for sepsis care it was explicitly decried as premature by some (8). The efficacy of many of the common components of early sepsis therapy remained disputed (8–10). Concerns centered around medical effectiveness of bundle elements, potential unintended consequences, and uncertainty if hospital policies would translate into sustained meaningful outcome improvements (5–7). Unintended negative consequences for patients of such state interventions have been documented, including the unnecessary administration of antibiotics to patients who are not infected, the development of antibiotic resistance, distraction of care from other disease states and important bedside activities, and ultimately protocol and metric fatigue (10–15).

Therefore, we sought to evaluate the initial 2 years of Rory’s Regulations, as these NYSDOH initiatives came to be called in the popular press. As part of the evaluation we asked: to what extent and when were the newly instituted sepsis protocols activated? How did this change over the early life of the program and between hospitals? Were the changes in protocolized behavior associated with changes in risk-adjusted inpatient mortality among sepsis patients included in the protocol and, in comparison, among patients not included in the protocol?

Some of the results of these studies have been previously reported in the form of an abstract (16).

Methods

New York Sepsis Regulation

The New York State sepsis initiative originated with the New York State Executive Office in collaboration with the NYSDOH. NYSDOH sought input from expert clinicians, hospital association representatives, the state’s Quality Improvement Organization (IPRO) (17), and peer-reviewed literature to inform the new sepsis regulations. In early 2013, NYSDOH issued amendments to existing public health regulations requiring hospitals to submit and follow evidence-informed protocols for early identification and treatment of severe sepsis or septic shock (5). Although protocols could be tailored to specific hospitals, they were required to include both of the following:

• 3-hour bundle: administration of antibiotics within 3 hours of patient identification, drawing blood cultures before administering those antibiotics, and measuring of blood lactate levels within 3 hours;

• 6-hour bundle: for patients with hypotension (systolic blood pressure <90 mm Hg) or lactate ≥4 mmol/L the administration of a 30 ml/kg bolus, vasopressors for refractory hypotension, and remeasurement of lactate within 6 hours of bundle initiation.

Hospitals varied in their sepsis identification strategies; institutional triggers for sepsis protocol initiation included: 1) sepsis screening by clinical assessment only, 2) clinical screening and abnormal laboratory studies (i.e., serum lactate and white blood cell count), 3) clinical screening and a “code sepsis or rapid response,” and 4) assessment for systemic inflammatory response syndrome criteria indicators (see Table E2 in the online supplement). Regardless of identification strategy, all cases identified had severe sepsis or septic shock. The regulations permitted hospitals to have flexibility in case identification to facilitate broader adoption.

Reporting of Sepsis Cases

Because the initiative was introduced in 2013, the recommended criteria for prospectively identifying severe sepsis and septic shock were based on the 2003 consensus sepsis definitions (“Sepsis 2”) (18), not the Third International Consensus definitions (“Sepsis 3”) (19). Hospitals submitted the data on sepsis cases quarterly through a secure, online portal. To promote accurate data collection and reporting, a Data Dictionary for Severe Sepsis and Septic Shock was provided to hospitals (20). Hospital chief executive officers and chief medical officers were also required by the state to confirm compliance and institutional support for the hospital protocol and regulatory requirements.

Patients

Patients were reported from 185 hospitals in New York State between April 1, 2014 and June 30, 2016 (see Figure E1). Patients excluded a priori were those with advanced directives that limited treatment with sepsis care interventions in hospital protocols, all interhospital transfers, and those who declined interventions. For patients with multiple eligible sepsis hospitalizations, each hospitalization was included.

Although there was no required method for identifying severe sepsis and septic shock cases, NYSDOH strongly encouraged hospitals to use both clinical and administrative data, and prospective and retrospective approaches, to ensure complete reporting. Moreover, NYSDOH took several additional approaches to encourage complete reporting. First, NYSDOH screened the state-wide discharge database to assess underreporting. Hospitals were notified of potential missed cases identified in the discharge database, and were provided the opportunity to review and submit these cases as appropriate. Second, IPRO nurse reviewers audited a 10% random sample of all submitted cases each quarter to assess the accuracy of reported variables. Hospitals received quarterly feedback on each relevant measure compared with statewide averages, and their performance trends over time. Percent protocol initiation and raw mortality percents were also tracked and presented to hospitals quarterly.

Statistical Analysis

Hospital and patient characteristics are presented as frequencies and percentages for categorical variables, and median and the interquartile range for continuous variables. To assess variability across hospitals during the initiative, multilevel logistic regression models with patients nested within hospitals were used to estimate the probability and reliability-adjusted percent of protocol initiation. Based on these percents, hospitals were then categorized by quartiles. Multilevel models were also used to rank a hospital’s compliance with 3-hour and 6-hour sepsis treatment bundles. We compared the temporal trends in protocol initiation, bundle compliance among patients with a sepsis protocol initiated, and risk-adjusted in-hospital mortality over the study period using maximum likelihood logistic regression with a robust standard error clustering on patient. Fractional polynomials were used to determine whether changes over time were linear in the logit. Mortality was adjusted for illness severity using a multivariable logistic regression model incorporating patient data from the first full year of the NYSDOH initiative. This model was developed elsewhere to evaluate New York State hospital performance. The model is described in Table E3, and in a separate paper describing the model development and validation (21). The final risk adjustment model had an area under the receiver operator characteristic curve (C statistic) of 0.77 in internal validation data.

We used separate multivariable logistic regression models to examine the association among 1) protocol initiation, 2) 3-hour bundle compliance, 3) 6-hour bundle compliance, and 4) individual bundle elements (first serum lactate reported within 3 h, blood cultures obtained before antibiotics, broad-spectrum antibiotics within 3 h, completion of intravenous fluids for patients with hypotension or elevated serum lactate within 6 h, vasopressors given for refractory hypotension within 6 h, and serum lactate reordered if missing or elevated lactate within 6 h) with in-hospital mortality. We calculated adjusted odds ratios (ORs) for in-hospital mortality by study month, by protocol initiation, by bundle compliance, and by individual bundle elements. Because hospitals were required to report bundle compliance for only those patients with a sepsis protocol initiated, analyses of bundle compliance are restricted to patients with a sepsis protocol initiated.

Institutional review board approval (exemption) was obtained by NYSDOH. All analyses were run using Stata 14.2 (StataCorp). P values less than or equal to 0.05 were considered statistically significant. Analyses were conducted by independent statisticians at The Ohio State University to minimize risks of bias by interests of NYSDOH. This manuscript was prepared for publication using the SQUIRE 2.0 guidelines for reporting quality improvement (22).

Results

Patient Characteristics

Of 113,380 severe sepsis and septic shock cases submitted to NYSDOH from all 185 nonfederal hospitals in New York State during the study period, 91,357 hospitalizations from 183 hospitals met study inclusion criteria (see Figure E1). Of these, 47,778 (52.3%) had severe sepsis, and 43,579 (47.7%) had septic shock (Table 1). Median age was 71 years, and the most common sites of infection were respiratory, urinary, and gastrointestinal. A total of 22.8%, 26.6%, and 30.3% received mechanical ventilation within 6, 12, and 24 hours of protocol initiation. In-hospital mortality was 26.7%. In bivariate analyses, patients who died were older (median age, 75 vs. 70 yr; P < 0.001), had more comorbid disease, and were less likely to have a sepsis protocol initiated in the emergency department (57.0% vs. 66.6%; P < 0.001). Patient characteristics by sepsis identification strategy are presented in Table E2. The proportion of patients with septic shock and mechanical ventilation before protocol initiation was stable across the study period (see Figures E5 and E7), as were mean initial lactate and comorbidity burden (see Figures E6 and E7).

Table 1.

Patient Demographics and Clinical Characteristics by In-Hospital Mortality

Characteristic	Alive (n = 66,941)	Died (n = 24,416)	Total (n = 91,357)	P Value
Median (IQR) age, yr	70 (58–81)	75 (63–85)	71 (59–82)	<0.001
Sex
Male	34,396 (51.4)	12,628 (51.7)	47,024 (51.5)	0.357
Race
White	42,792 (63.9)	15,487 (63.4)	58,279 (63.8)	0.004
Black	12,188 (18.2)	4,648 (19.0)	16,836 (18.4)
Native American	121 (0.2)	36 (0.1)	157 (0.2)
Asian	2,499 (3.7)	936 (3.8)	3,435 (3.8)
Pacific Islander	94 (0.1)	29 (0.1)	123 (0.1)
Multiracial	1,351 (2.0)	546 (2.2)	1,897 (2.1)
Other	7,896 (11.8)	2,734 (11.2)	10,630 (11.6)
Ethnicity
Spanish/Hispanic origin	7,395 (11.0)	2,353 (9.6)	9,748 (10.7)	<0.001
Not Spanish/Hispanic	52,570 (78.5)	19,239 (78.8)	71,809 (78.6)
Unknown	6,955 (10.4)	2,815 (11.5)	9,770 (10.7)
Multiethnic	21 (0.0)	9 (0.0)	30 (0.0)
Protocol initiated	54,658 (81.7)	19,635 (80.4)	74,293 (81.3)	<0.001
Place of protocol initiation
No	12,283 (18.3)	4,781 (19.6)	17,064 (18.7)	<0.001
ER	44,566 (66.6)	13,908 (57.0)	58,474 (64.0)
Floor	5,960 (8.9)	2,730 (11.2)	8,690 (9.5)
ICU	4,132 (6.2)	2,997 (12.3)	7,129 (7.8)
Type of sepsis
Severe sepsis	40,461 (60.4)	7,317 (30.0)	47,778 (52.3)	<0.001
Septic shock	26,480 (39.6)	17,099 (70.0)	43,579 (47.7)
Site of infection
Urinary	18,643 (27.8)	3,603 (14.8)	22,246 (24.4)	<0.001
Respiratory	24,307 (36.3)	10,979 (45.0)	35,286 (38.6)
Gastrointestinal	7,547 (11.3)	3,323 (13.6)	10,870 (11.9)
Skin	4,952 (7.4)	1,171 (4.8)	6,123 (6.7)
Central nervous system	363 (0.5)	91 (0.4)	454 (0.5)
Other	5,608 (8.4)	2,157 (8.8)	7,765 (8.5)
Unknown	5,521 (8.2)	3,092 (12.7)	8,613 (9.4)
Mechanical ventilation before protocol initiation	5,299 (7.9)	4,165 (17.1)	9,464 (10.4)	<0.001
Thrombocytopenia	13,659 (20.4)	7,013 (28.7)	20,672 (22.6)	<0.001
Bandemia	16,661 (24.9)	7,080 (29.0)	23,741 (26.0)	<0.001
Lower respiratory infection	29,863 (44.6)	13,846 (56.7)	43,709 (47.8)	<0.001
Altered mental status	26,579 (39.7)	13,822 (56.6)	40,401 (44.2)	<0.001
Admitted to ICU	37,695 (56.3)	19,038 (78.0)	56,733 (62.1)	<0.001
Chronic respiratory failure	7,000 (10.5)	4,272 (17.5)	11,272 (12.3)	<0.001
AIDS/HIV disease	1,698 (2.5)	541 (2.2)	2,239 (2.5)	0.006
Metastatic cancer	6,001 (9.0)	4,023 (16.5)	10,024 (11.0)	<0.001
Lymphoma/leukemia/multiple myeloma	2,950 (4.4)	1,785 (7.3)	4,735 (5.2)	<0.001
Immune-modifying medications	11,154 (16.7)	5,051 (20.7)	16,205 (17.7)	<0.001
Congestive heart failure	13,684 (20.4)	6,809 (27.9)	20,493 (22.4)	<0.001
Chronic renal failure	6,866 (10.3)	3,818 (15.6)	10,684 (11.7)	<0.001
Chronic liver disease	3,534 (5.3)	2,607 (10.7)	6,141 (6.7)	<0.001
Diabetes	24,670 (36.9)	8,647 (35.4)	33,317 (36.5)	<0.001
Organ transplant	1,387 (2.1)	586 (2.4)	1,973 (2.2)	0.003
Median (IQR) number of comorbidities	2 (1–3)	3 (2–4)	2 (1–4)	<0.001

Definition of abbreviations: ER = emergency room; IQR = interquartile range.

Data are shown as n (%) unless otherwise indicated.

Sepsis Protocol Initiation

During the study period, 74,293 (81.3%) sepsis cases had a sepsis protocol initiated. (Patient characteristics by protocol initiation status are reported in Table E4. The percent of cases with a protocol initiated increased from 74% to 86% over the course of the study (see Figure E2). Sepsis protocol initiation varied from less than 20% to nearly 100% across hospitals (see Figure E3), and was 90.4% at the median hospital. Among cases with a protocol initiated, about 80% were initiated in the emergency department, 10% were initiated in a hospital ward, and 10% were initiated in an ICU (see Figure E4). Hospitals with a higher percentage of protocol initiation were more likely to be nonprofit, teaching facilities, located in metropolitan areas, with a higher number of certified beds (see Table E5). An analysis of patient-level data revealed that there were similarities among patient characteristics across the four quartiles of protocol initiation (see Table E6).

Sepsis Bundle Compliance

Among 74,293 cases with a sepsis protocol initiated, overall compliance (all eligible elements completed) with the 3-hour and 6-hour sepsis bundles increased over the study period by 0.43% per month (95% confidence interval [CI], 0.37–0.49; P < 0.001) and 0.54% per month (95% CI, 0.49–0.58; P < 0.001), respectively (Figure 1). The 3-hour bundle compliance increased from 53.4% to 64.7% (P < 0.001), whereas among 35,307 hypotensive (and therefore eligible) patients, 6-hour bundle compliance increased from 23.9% to 30.8% (P < 0.001). These are standardized difference of the means equal to 7.98 and 3.69 for the 3-hour and the 6-hour bundle, respectively. Compliance with individual elements in the 3-hour and 6-hour bundles also improved over time (see Figure E8).

Figure 1.

Compliance with the 3-hour bundle and the 6-hour bundle over time. The regression lines for bundle compliance are based on individual unadjusted logistic regression models with time entered as a square root expression (3-h model) and a quadratic expression (6-h model). Using the 27 monthly observations, 3-hour bundle compliance and 6-hour bundle compliance increase 0.43% per month (95% confidence interval, 0.37–0.49%; P < 0.001) and 0.54% per month (95% confidence interval, 0.49–0.58%; P < 0.001), respectively.

In-Hospital Mortality

Among cases with a sepsis protocol initiated, risk-adjusted absolute mortality declined from 28.8% to 24.4% (P < 0.001), and decreased by 0.168% per month (95% CI, 0.167–0.169; P < 0.001) over the course of the study (Figure 2). This is a standardized difference of the risk-adjusted means equal to −8.67. Odds of in-hospital mortality declined by 1.1% each month in patients with a sepsis protocol initiated (n = 74,293; OR, 0.989 per additional month; P < 0.001; 95% CI, 0.987–0.992), whereas odds of in-hospital mortality were stable in patients without a sepsis protocol initiated (n = 17,064; OR, 1.00 per additional month; P = 0.25; 95% CI, 1.00–1.01) (Figure 3; see Figures E9 and E10 and Table E7). Overall risk-adjusted hospital mortality in the entire population (both with and without a protocol initiated) also decreased over time, by 1% per month (OR, 0.991; 95% CI, 0.989–0.994; P < 0.001) (see Figure E11).

Figure 2.

Change in risk-adjusted mortality over time. Time is entered into the risk-adjusted hospital mortality model as a linear expression. Using the 27 monthly observations, risk-adjusted mortality decreases 0.17% per month (95% confidence interval, 0.167–0.169; P < 0.001).

Figure 3.

Risk-adjusted hospital mortality over time by protocol initiation status. Risk-adjusted mortality improved in patients with a sepsis protocol initiated throughout the study period, but it was stable for patients without a protocol initiated. The difference in mortality between patients treated with and without a sepsis protocol first became significant (P = 0.019) during the third month of the study.

Association between Sepsis Bundle Compliance and Mortality

Hospitals with greater 3-hour and 6-hour compliance had lower risk-adjusted mortality and median hospital length of stay (P < 0.001 for each comparison) (Table 2). For example, hospitals in the lowest quartile of 3-hour bundle compliance had a risk-adjusted mortality of 29.8%, compared with 23.5% risk-adjusted mortality in hospitals in the highest quartile of 3-hour bundle compliance.

Table 2.

Patient Outcomes by Hospital Quartile of Compliance with the 3-Hour and 6-Hour Bundles

	First (Lowest)	Second	Third	Fourth (Highest)	P Value*
Quartiles of 3-h bundle compliance†
Patients, n	18,915	19,634	17,232	18,512
Risk-adjusted hospital mortality, % (95% CI)	29.8 (29.2–30.4)	26.2 (25.6–26.8)	25.9 (25.3–26.5)	23.5 (22.9–24.1)	<0.001
Median (IQR) hospital LOS for those that survived, d	263 (158–463)	257 (148–453)	232 (142–401)	199 (124–336)	<0.001
Quartiles of 6-h bundle compliance†
Patients, n	19,038	18,377	18,441	18,437
Risk-adjusted hospital mortality, % (95% CI)	28.4 (27.8–29.0)	27.7 (27.1–28.3)	25.9 (25.3–26.4)	23.4 (22.9–24.0)	<0.001
Median (IQR) hospital LOS for those that survived, d	247 (149–432)	259 (151–455)	220 (136–383)	217 (130–377)	<0.001

Definition of abbreviations: CI = confidence interval; IQR = interquartile range; LOS = length of stay.

^*Risk-adjusted hospital mortality is based on chi-square test of trend, and hospital LOS is based on the nonparametric equality-of-medians test.

^†The quartiles of probability of bundle compliance are based on two individual unadjusted random-effects logistic regression models where hospital is the random term. Only patients with a sepsis protocol initiated were included in the model (n = 72,293).

Completion of individual bundle elements (including administration of fluids) were each associated with reduced odds of in-hospital mortality, with the exception of vasopressors for refractory hypotension (see Table E8). Risk-adjusted mortality decreased by 5% (OR, 0.95; P < 0.001; 95% CI, 0.94–0.96) and 6% (OR, 0.94; P < 0.001; 95% CI, 0.93–0.95) for each 10% increase in hospital compliance with the 3-hour and 6-hour sepsis bundles, respectively (see Table E9).

Discussion

New York State introduced regulations (Rory’s Regulations) in 2013 to improve state-wide sepsis care. The regulations mandated the development and implementation of sepsis protocols in each hospital, and the reporting of patient-level treatment and outcomes. In this study, we examined New York’s experience during the first 2 years of the ongoing initiative. Results of the initiative have also been reported on the NYSDOH website. However, the public report, designed for hospital administrators and public at large, provides just unadjusted aggregate results and each hospital’s quintile ranking for protocol initiation, 3-hour bundle compliance, and 6-hour bundle compliance.

By contrast, we examined aggregate trends in risk-adjusted hospital mortality over the first 27 months of the initiative, and the relationship between bundle compliance and outcomes. A study focused on patients in this database admitted through the emergency department who had completion of the 3-hour bundle was also recently published (23).

During the first 27 months of Rory’s Regulations, there was substantial but not universal implementation of sepsis protocols, with increasing compliance across the first 2 years of the program. Patients treated under the protocol experienced a risk-adjusted 4.4% absolute (15% relative) reduction in risk-adjusted mortality over the study period, which correlated with the improved bundle compliance at the hospital-level. All comers (patients with and without a sepsis protocol initiated) experienced a 3.6% absolute (12.2% relative) reduction in risk-adjusted mortality over the study period.

Although we cannot prove that the improvement in risk-adjusted mortality among sepsis patients was a direct result of the regulations, there is reason to believe that this may be the case. First, there were aggregate increases in protocol initiation, 3- and 6-hour bundle completion, and individual bundle element completion over the study period, all of which were correlated with improved outcomes at the patient and hospital level (except vasopressors for refractory hypotension, which we suspect is more strongly confounded by indication). Second, there was a drop in the risk-adjusted mortality, suggesting that the improvements were not merely the result of stage migration or changes in coding that may confound sepsis trends measured in administrative data. Rather, in addition to finding a decline in risk-adjusted mortality, we found no evidence that less severely ill patients were increasingly identified over the study period, because median lactate, proportion with septic shock, and proportion with mechanical ventilation before protocol initiation remained stable (see Figures E5–E7).

It is important to consider the context of New York’s sepsis regulations when assessing their implementation and outcomes. On the one hand, there were no formal financial incentives associated with New York’s regulations. However, the regulations garnered substantial public attention, including commentary from the governor and coverage in the New York Times (24). They also came at the same time that multiple advocacy groups and the U.S. Centers for Medicare and Medicaid Services were particularly active in their efforts to increase public awareness of sepsis care (25–27). Public calls to professionalism and the attention of state leaders may be powerful drivers of quality improvement in sepsis care, perhaps complementing public reporting and financial incentive approaches, and a model that could potentially serve as a blueprint to implement similar strategies in other states.

Although protocol initiation, bundle completion, and risk adjusted mortality all improved over the first 27 months of the initiative, implementation was not perfect. Although all hospitals had a state-approved sepsis protocol in place with a structure for audit and feedback, many did not achieve 100% initiation of the protocol and, even after 2 years, fully one-third of patients were not receiving the minimum package of blood cultures, antibiotics, and serum lactate measurement as quickly as desired. This emphasizes the important implementation gaps that remain to achieve standardization of care (28).

Strengths of our study include the evaluation of a first-in-the-nation, focused effort to improve sepsis care via an innovative state-wide mechanism, where we can measure both implementation and patient-related outcomes. Furthermore, NYSDOH’s detailed audits provide quality control on both case ascertainment and individual data elements. Further strengths include our ability to conduct detailed physiologic risk-adjustment as a result of the detailed patient-level data collection, limiting bias from stage migration.

The study has several limitations. First, as with the evaluation of most quality improvement initiatives, it is difficult to prove a causal relationship between the intervention and reduced mortality from sepsis. Because this was a broad, state-wide, mandated initiative, randomized assignment of interventions at a hospital-level did not occur. Second, the methods used by individual hospitals to identify patients with sepsis varied, and there was no mechanism for centralized case finding or adjudication. As such, the effects here represent the effects on sepsis-as-recognized across the state in an array of hospitals, not sepsis according to the standard of select efficacy trials. This is an extension of the widely understood efficacy/effectiveness trade-off in planning clinical trials. At the outset of the state-wide initiative, hospitals were given flexibility to develop their own protocols for guiding the interventions included in the 3- and 6-hour bundle; attempting to disentangle which triggers work best for which hospitals was beyond the scope of this project given confounding by other unmeasured hospital processes.

Third, a potential limitation of this design is that we cannot distinguish the general effects of knowing that a hospital will be monitored for public reporting with specific effects of the hospital protocols. This might seem to leave the study vulnerable to the so-called Hawthorne effect. However, in contrast, we believe the increased attention to patients with sepsis caused by public reporting is a desired part of the mechanism of Rory’s Regulations, not an unintended side effect to be minimized; an analysis that removed such an effect would inappropriately underestimate the public health effect of the Regulations.

Fourth, there were limits in the data accuracy for chart documentation of timing. This is consistent with the clinical reality of the gap between identifying a patient with sepsis, initiating treatment, and timing of documentation in the clinical record. In particular, we could not evaluate the impact of care on patients who did not have sepsis or, if they had sepsis, it was never recognized. Fifth, because complete data collection was not required for patients in whom a protocol was not initiated, compliance with the 3-hour and 6-hour bundle could not be determined in these patients. Lastly, we could not evaluate whether there were spill-over effects, either positive (because of streamlining care for a common condition) or negative (in that the care of patients with sepsis might have distracted from the care of other patients).

In conclusion, this study reports the results of the New York State initiative for sepsis, which demonstrates improved care for patients with sepsis as evidenced by increased compliance with performance metrics and decreased risk-adjusted mortality over the first 2 years of the ongoing initiative. A state-wide initiative using regulations and nonfinancial incentives seems to have substantially changed care.