Abstract
Background:
Medicare requires that hospitals report on their adherence to the Severe Sepsis and Septic Shock Early Management Bundle (SEP-1).
Objective:
To evaluate the effect of SEP-1 on treatment patterns and patient outcomes.
Design:
Longitudinal study of hospitals using repeated cross-sectional cohorts of patients.
Setting:
11 hospitals within an integrated health system.
Patients:
54 225 encounters between January 2013 and December 2017 for adults with sepsis who were hospitalized through the emergency department.
Intervention:
Onset of the SEP-1 reporting requirement in October 2015.
Measurements:
Changes in SEP-1–targeted processes, including antibiotic administration, lactate measurement, and fluid administration at 3 hours from sepsis onset; repeated lactate and vasopressor administration for hypotension within 6 hours of sepsis onset; and sepsis outcomes, including risk-adjusted intensive care unit (ICU) admission, in-hospital mortality, and home discharge among survivors.
Results:
Two years after its implementation, SEP-1 was associated with variable changes in process measures, with the greatest effect being an increase in lactate measurement within 3 hours of sepsis onset (absolute increase, 23.7 percentage points [95% CI, 20.7 to 26.7 percentage points]; P < 0.001). There were small increases in antibiotic administration (absolute increase, 4.7 percentage points [CI, 1.9 to 7.6 percentage points]; P = 0.001) and fluid administration of 30 mL/kg of body weight within 3 hours of sepsis onset (absolute increase, 3.4 percentage points [CI, 1.5 to 5.2 percentage points]; P < 0.001). There was no change in vasopressor administration. There was a small increase in ICU admissions (absolute increase, 2.0 percentage points [CI, 0 to 4.0 percentage points]; P = 0.055) and no changes in mortality (absolute change, 0.1 percentage points [CI, −0.9 to 1.1 percentage points]; P = 0.87) or discharge to home.
Limitation:
Data are from a single health system.
Conclusion:
Implementation of the SEP-1 mandatory reporting program was associated with variable changes in process measures, without improvements in clinical outcomes. Revising the measure may optimize its future effect.
Primary Funding Source:
Agency for Healthcare Research and Quality.
Sepsis is a common, deadly, and costly disease (1–3). Despite the development and dissemination of evidence-based sepsis guidelines, many patients do not receive timely treatment, resulting in preventable morbidity and mortality (4, 5). Therefore, government agencies are increasingly turning to public policy as a means of achieving broad-based improvements in sepsis care. In New York State, regulations requiring hospitals to develop, implement, and report their adherence to sepsis protocols were associated with decreases in sepsis mortality (6). At the federal level, the Centers for Medicare & Medicaid Services (CMS) implemented the Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) in October 2015. This policy requires hospitals to collect and report data on their adherence to a multicomponent sepsis treatment bundle, including blood cultures, early antibiotics, serial lactate measurement, intravenous (IV) fluids, vasopressors for refractory hypotension, and documentation of a patient’s response to treatment (7).
The SEP-1 program differs from the New York State regulations in several ways that create uncertainty as to its ultimate effect. First, it is a prescriptive, “all-or-none” measure, meaning that treatment for a given patient is considered adherent only if all elements are completed. In contrast, the New York State regulations give hospitals flexibility with respect to specific elements of the protocols and report adherence separately for the 3- and 6-hour elements. Second, whereas the New York State regulations emphasize education and implementation through mandatory protocols, SEP-1 focuses only on data reporting—providing less information on how hospitals should actually implement sepsis treatment bundles. Third, New York State reports both processes and risk-adjusted sepsis mortality, whereas SEP-1 does not include an outcome measure. Given these differences, substantial debate remains about the overall value of SEP-1 (7–11).
Early results from the SEP-1 program have not resolved this uncertainty. Performance of SEP-1 varies across hospitals, and patient-level adherence to SEP-1 is not consistently associated with improved survival (12, 13). However, these results do not provide insight into longitudinal changes in sepsis treatment and outcomes associated with SEP-1 implementation—a necessary component of a rigorous policy evaluation that asks not whether the elements of SEP-1 are effective treatments, but rather whether the SEP-1 policy was itself effective at improving treatment patterns and outcomes (6). We evaluated the association between SEP-1 implementation and longitudinal changes in sepsis treatment and outcomes. We used electronic health record (EHR) data from a large, multihospital health system, which serves as a model case for the effect of SEP-1 across the country.
Methods
Overview and Rationale for Study Design
We used EHR data from patients in 11 hospitals in the University of Pittsburgh Medical Center (UPMC) Health System from January 2013 to December 2017 to do a multicenter longitudinal study of hospitals using repeated cross-sectional patient cohorts. We analyzed changes in sepsis treatment processes and outcomes that occurred in association with the October 2015 implementation of SEP-1. The UPMC Health System is a large, regional health care provider that includes hospitals of different sizes, urbanicity, academic status, and publicly reported SEP-1 performance (Supplement, available at Annals.org). Hospitals within the UPMC Health System responded to SEP-1 with several strategies that were common in hospitals across the United States (14), including sepsis alerts, electronic order sets, and clinical documentation reminders. These elements make the UPMC Health System a useful model case for evaluating the effect of SEP-1.
We chose an EHR-based analysis rather than using administrative claims for 2 reasons. First, time-stamped data on sepsis treatment processes are missing from administrative data sets, precluding an analysis of processes and outcomes. Second, Medicare’s transition from International Classification of Diseases, Ninth Revision (ICD-9), to ICD, 10th Revision (ICD-10), occurred at the same time as the implementation of SEP-1, meaning that any evaluation using ICD codes to identify patients with sepsis would be biased by a contemporaneous change in the denominator (15).
Data Source
We obtained data from the UPMC EHR (Cerner PowerChart, Cerner Corporation) from an existing registry of hospital encounters; earlier versions of these data were used to develop and validate current sepsis definitions (16). These data include demographic characteristics, vital signs, laboratory values, medication administration, microbiology results, ventilator settings, and diagnosis and procedure codes.
Patients
We included patients aged 18 years or older who were hospitalized with community-onset sepsis, defined as the combination of suspected infection (blood, urine, respiratory, or other body fluid culture obtained) and organ failure (2 or more Sequential Organ Failure Assessment [SOFA] points) within 6 hours of arrival to the emergency department (ED) (17). We focused on these patients for 2 reasons. First, they represent most patients admitted with sepsis (18). Second, because timing of treatment is a key component of SEP-1, limiting the analysis to those with community-onset sepsis allowed us to improve our precision in defining the time of sepsis onset. In our primary analyses, we defined “time zero” as the time the body fluid culture was ordered. Because antibiotic therapy was a component of the SEP-1 bundle, we did not include the receipt of antibiotics as part of the definition of suspected infection.
This definition differs from the definition of sepsis used by CMS in SEP-1, which is based on administrative diagnosis codes. As noted earlier, we intentionally chose not to use administrative codes because the change from ICD-9 to ICD-10 is likely to have influenced the denominator. Therefore, although our patient population is different from the population measured in SEP-1, it is less biased for the purposes of evaluating SEP-1 than approaches using administrative codes.
A complete list of exclusion criteria is available in the Supplement. Notable exclusions included transfers, very short (<24 hours) and very long (>30 days) hospital stays, early comfort care, and repeated encounters.
Variables
Our dependent variables of interest included both processes and outcomes. We created indicators for the following processes: SEP-1–adherent IV antibiotic therapy, lactate measurement, and crystalloid IV fluid administration within 3 hours of suspected infection and repeated lactate measurement and administration of a vasopressor within 6 hours of suspected infection. The full details of how we operationalized these process measures are in the Supplement. Outcomes included admission to the intensive care unit, in-hospital mortality, and discharge to home among survivors.
The primary independent variable was the quarter of hospital admission. We excluded patients in the fourth quarter of 2015, representing the 3 months after SEP-1 implementation, to allow for potential “run-in” effects.
Additional patient-level variables used for risk adjustment included patient age, selected Elixhauser comorbidities (19), SOFA score within 6 hours of presentation to the ED, and a hierarchical variable for source of infection based on positive results in clinical microbiology data (Supplement).
We assumed that missing data for process measures indicated that the process was not completed because the data derive from date- and time-stamped events from charting in the EHR—that is, if lactate was not measured, the values for the lactate date and time stamps are missing. No data were missing for the outcome variables.
Analysis
We compared patient characteristics before and after SEP-1 implementation using standard summary statistics.
For all process and outcome variables, we did a longitudinal study of hospitals using repeated cross-sectional patient cohorts to assess the effect of SEP-1; full details of our model specification are in the Supplement. The primary independent variable was the quarter of hospital admission. We modeled this independent variable through a continuous quarterly time variable centered at the first quarter of 2016, an indicator for the period after SEP-1, and the interaction of time-by-period. This model generates coefficients for both a one-time shift and a change in trend for the dependent variable. To illustrate the clinical effect of the changes associated with SEP-1, we used these coefficients to estimate the expected value of the dependent variable in the fourth quarter of 2017 in the presence of SEP-1 and the expected value in the absence of SEP-1 (from projected baseline trends). The difference between these 2 expected values estimates the effect of SEP-1 on the dependent variable. This approach relies on a strong assumption that the pre-SEP-1 trend would have continued unchanged in the absence of SEP-1, which cannot be tested in the absence of a control group.
In all models, we included a hospital fixed effect to account for changes within hospitals over time. We used logistic regression for binary outcomes and generalized linear models with a log-g distribution for continuous outcomes, with robust SEs. We did not do risk adjustment in models for which the dependent variable was a process measure. Where the dependent variable was an outcome measure, we fit a model with risk adjustment to account for possible changes in case mix over time using patient-level variables, including age (as linear splines), indicators for Elixhauser comorbidities, SOFA score within 6 hours of ED presentation, an indicator to account for seasonality, and source of infection (Supplement).
We created graphs to show the effects of SEP-1 by plotting quarterly values for the dependent variable of interest, along with trend lines from the regression models. To show the clinical effect of the changes associated with SEP-1, we included in these graphs the marginal estimates in quarter 4 of 2017 as observed with SEP-1 and as expected in the absence of SEP-1; the difference between these values estimates the effect of SEP-1 by the end of the study period.
Subgroup Analyses
We repeated the analysis in the subgroup of patients with septic shock, who may be more likely to benefit from early treatment than those without shock (20). We defined shock in 2 prespecified ways: hypotension at admission, defined as a mean arterial pressure of less than 65 mm Hg within 6 hours of ED presentation, and receipt of vasopressor therapy within the first 2 days of hospital admission. We also analyzed hospital-level changes in lactate, antibiotic, and fluid adherence and in-hospital mortality to assess the consistency of the effects across hospitals.
Sensitivity Analyses
We did several sensitivity analyses, which are detailed in the Supplement. First, we analyzed antibiotic and lactate adherence using the time of ED presentation as time zero. Second, we analyzed changes in process measures as continuous rather than dichotomized outcomes. Third, we repeated the analysis of mortality, including hospice discharges. Fourth, we used data on SEP-1 reporting to generate a cohort of patients more closely resembling the patients whose data were actually reported as part of the SEP-1 measure. Fifth, we examined the possibility of ascertainment bias by analyzing changes in quarterly case counts, number of blood cultures ordered, and cases with hypotension, all indexed to quarterly ED encounters. Finally, we did an analysis of the potential influence of unmeasured confounding.
We used Stata, version 16.1 (StataCorp), for all analyses. We considered results to be significant at a P value of 0.05. This research was approved by the University of Pittsburgh Human Research Protection Office (PRO 17010375).
Role of Funding Source
This study was funded by grants from the National Institutes of Health and the Agency for Healthcare Research and Quality. Neither agency had any role in the design, conduct, or analysis of the study or the decision to submit for publication.
Results
We identified 54 225 unique patients meeting criteria for suspected infection and organ dysfunction within 6 hours of arrival to the ED, 2415 of whom were excluded in the washout period (Figure 1). Patient characteristics, including age, comorbidity counts, and SOFA scores, were similar before and after implementation of SEP-1 (Table 1). Compared with patients in the pre-SEP-1 period, those in the post–SEP-1 period were slightly less likely to have presenting hypotension (23.4% vs. 21.1%) and to receive vasopressors (4.8% vs. 4.2%). Table 1 also provides details on characteristics of subgroups of patients with indicators of shock.
Figure 1.
Patient selection flow diagram.
ED = emergency department; LOS = length of stay; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle; SOFA = Sequential Organ Failure Assessment.
Table 1.
Patient Characteristics
| Variable | Before SEP-1 | After SEP-1 |
|---|---|---|
| All patients, n | 29 051 | 22 759 |
| Median age (IQR), y | 72 (59 to 83) | 71 (59 to 83) |
| Female, n (%) | 15 401 (53.0) | 11 758 (51.7) |
| White, n (%) | 24 949 (85.9) | 19 423 (85.3) |
| Median comorbidity count (IQR), n | 4 (3 to 6) | 4 (3 to 6) |
| Median SOFA score within first 6 h (IQR) | 3 (2 to 4) | 3 (2 to 4) |
| Mean arterial pressure <65 mm Hg within first 6 h, n (%) | 6809 (23.4) | 4801 (21.1) |
| Mechanical ventilation, n (%) | 2472 (8.5) | 1673 (7.4) |
| Vasopressors, n (%) | 1401 (4.8) | 963 (4.2) |
| Received SEP-1–adherent antibiotics within 3 h of suspected infection, n (%) | 12 418 (42.7) | 10719 (47.1) |
| Median time to antibiotic administration from suspected infection (IQR), h | 1.9 (0.9 to 3.9) | 1.8 (0.9 to 3.5) |
| Lactate checked within 3 h of suspected infection, n (%) | 7711 (26.5) | 13 551 (59.5) |
| Median time to lactate measurement from suspected infection (IQR), h | 0.2 (−0.1 to 1.0) | 0.2 (0 to 0.5) |
| Received 30 mL/kg of body weight of IV fluids within 3 h of suspected infection, n (%) | 2250 (7.7) | 2807 (12.3) |
| Median fluid volume received within 3 h of suspected infection (IQR), mL/kg | 4.2 (0.0 to 14.2) | 9.0 (0.0 to 17.3) |
| Admission to the ICU, n (%) | 7205 (24.8) | 5031 (22.1) |
| In-hospital mortality, n (%) | 1419 (4.9) | 947 (4.2) |
| Patients with a mean arterial pressure <65 mm Hg within 6 h, n | 6809 | 4801 |
| Median age (IQR), y | 72 (60 to 83) | 70 (59 to 82) |
| Median comorbidity count (IQR), n | 4 (3 to 6) | 5 (3 to 6) |
| Median SOFA score within first 6 h (IQR) | 4 (3 to 5) | 4 (3 to 6) |
| Mechanical ventilation, n (%) | 1090 (16.0) | 726 (15.1) |
| Vasopressors, n (%) | 1123 (16.5) | 794 (16.5) |
| Received SEP-1–adherent antibiotics within 3 h of suspected infection, n (%) | 3286 (48.3) | 2554 (53.2) |
| Median time to antibiotic administration from suspected infection (IQR), h | 1.9 (1.0 to 3.6) | 1.8 (0.9 to 3.3) |
| Lactate checked within 3 h of suspected infection, n (%) | 2871 (42.2) | 3629 (75.6) |
| Median time to lactate measurement from suspected infection (IQR), h | 0.2 (0 to 1.2) | 0.2 (0 to 0.4) |
| Received 30 mL/kg of body weight of IV fluids within 3 h of suspected infection, n (%) | 1261 (18.5) | 1419 (29.6) |
| Median fluid volume received within 3 h of suspected infection (IQR), mL/kg | 11.7 (1.4 to 22.3) | 16.4 (6.9 to 29.7) |
| Admission to the ICU, n (%) | 3162 (46.4) | 2184 (45.5) |
| In-hospital mortality, n (%) | 641 (9.4) | 429 (8.9) |
| Patients requiring vasopressors, n | 1401 | 963 |
| Median age (IQR), y | 71 (60 to 82) | 70 (59 to 80) |
| Median comorbidity count (IQR), n | 5 (4 to 7) | 5 (4 to 7) |
| Median SOFA score within first 6 h (IQR) | 5 (4 to 7) | 6 (4 to 8) |
| Mechanical ventilation, n (%) | 717 (51.2) | 452 (46.9) |
| Vasopressors, n (%) | 2131 (100.0) | 1456 (100.0) |
| Received SEP-1–adherent antibiotics within 3 h of suspected infection, n (%) | 860 (61.4) | 639 (66.4) |
| Median time to antibiotic administration from suspected infection (IQR), h | 1.9 (1.1 to 3.4) | 1.7 (0.9 to 3.0) |
| Lactate checked within 3 h of suspected infection, n (%) | 940 (67.1) | 840 (87.2) |
| Median time to lactate measurement from suspected infection (IQR), h | 0.3 (0 to 1.8) | 0.1 (−0.1 to 0.4) |
| Received 30 mL/kg of body weight of IV fluids within 3 h of suspected infection, n (%) | 410 (29.3) | 417 (43.3) |
| Median fluid volume within 3 h of suspected infection (IQR), mL/kg | 16.5 (3.7 to 30.0) | 23.4 (10.7 to 36.7) |
| Admission to the ICU, n (%) | 1358 (96.9) | 935 (97.1) |
| In-hospital mortality, n (%) | 424 (30.3) | 268 (27.8) |
ICU = intensive care unit; IQR = interquartile range; IV = intravenous; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle; SOFA = Sequential Organ Failure Assessment.
Two years after its implementation, SEP-1 was associated with variable changes in sepsis process measures (Table 2 and Figure 2; model coefficients for level and slope shown in Supplement Tables 4 and 5). Most notably, SEP-1 was associated with a large absolute increase in adherence to 3-hour lactate measurement (23.7 percentage points [95% CI, 20.7 to 26.7 percentage points]; P < 0.001). There were increases in the rates of adherence to antibiotic administration within 3 hours (absolute increase, 4.7 percentage points [CI, 1.9 to 7.6 percentage points]; P = 0.001) and adherence to administering 30 mL/kg of body weight of IV fluids within 3 hours (absolute increase, 3.4 percentage points [CI, 1.5 to 5.2 percentage points]; P < 0.001), although the magnitude of these changes is of uncertain clinical effect. There were no meaningful differences in the rate of vasopressor administration within 6 hours (absolute decrease, −1.2 percentage points [CI, −5.9 to 3.5 percentage points]; P = 0.61).
Table 2.
Differences in Unadjusted Process Measures After SEP-1*
| Variable | Q3 2015 (Before SEP-1), Estimate (95% CI), % | Q4 2017 | Estimated Difference Between Expected and Observed in Q4 2017 (Effect of SEP-1) | ||
|---|---|---|---|---|---|
| Expected (Without SEP-1), Estimate (95% CI), % | Observed (With SEP-1), Estimate (95% CI), % | ||||
| Estimate (95% CI), percentage points | P Value | ||||
| All patients | |||||
| Antibiotics within 3 h | 43.6 (42.5 to 44.7) | 45.1 (42.5 to 47.7) | 49.8 (48.6 to 51.0) | 4.7 (1.9 to 7.6) | 0.001 |
| Lactate within 3 h | 33.1 (32.0 to 34.1) | 46.5 (43.6 to 49.3) | 70.2 (69.1 to 71.2) | 23.7 (20.7 to 26.7) | <0.001 |
| 30 mL/kg IV fluid within 3 h | 8.5 (7.9 to 9.1) | 9.9 (8.2 to 11.6) | 13.2 (12.4 to 14.0) | 3.4 (1.5 to 5.2) | <0.001 |
| Repeat lactate within 6 h† | 9.4 (8.4 to 10.5) | 9.6 (6.9 to 12.3) | 27.0 (25.7 to 28.2) | 17.4 (14.4 to 20.4) | <0.001 |
| Vasopressors within 6 h‡ | 17.1 (15.4 to 18.8) | 18.2 (14.0 to 22.5) | 17.0 (15.1 to 19.0) | −1.2 (−5.9 to 3.5) | 0.61 |
| Patients with a mean arterial pressure <65 mm Hg in first 6 h | |||||
| Antibiotics within 3 h | 50.4 (48.1 to 52.7) | 54.1 (48.7 to 59.5) | 57.8 (55.3 to 60.4) | 3.7 (−2.2 to 9.7) | 0.22 |
| Lactate within 3 h | 50.7 (48.4 to 52.9) | 65.0 (60.0 to 70.0) | 84.0 (82.2 to 85.8) | 19.0 (13.7 to 24.2) | <0.001 |
| 30 mL/kg IV fluid within 3 h | 20.7 (18.9 to 22.6) | 24.9 (19.8 to 30.0) | 32.1 (29.6 to 34.5) | 7.2 (1.5 to 12.8) | 0.013 |
| Repeat lactate within 6 h† | 15.7 (13.4 to 18.0) | 17.6 (11.4 to 23.7) | 40.1 (37.2 to 42.9) | 22.5 (15.7 to 29.3) | <0.001 |
| Vasopressors within 6 h‡ | 17.1 (15.4 to 18.8) | 18.2 (14.0 to 22.5) | 17.0 (15.1 to 19.0) | −1.2 (−5.9 to 3.5) | 0.61 |
| Patients requiring vasopressors | |||||
| Antibiotics within 3 h | 65.7 (61.0 to 70.4) | 72.7 (62.7 to 82.7) | 71.0 (65.8 to 76.1) | −1.7 (−13.0 to 9.5) | 0.76 |
| Lactate within 3 h | 75.1 (70.9 to 79.3) | 85.0 (78.3 to 91.8) | 91.5 (88.4 to 94.5) | 6.4 (−1.0 to 13.8) | 0.088 |
| 30 mL/kg IV fluid within 3 h | 34.0 (29.2 to 38.9) | 42.6 (29.8 to 55.4) | 45.6 (40.0 to 51.2) | 3.0 (−11.0 to 16.9) | 0.68 |
| Repeat lactate within 6 h† | 31.3 (26.1 to 36.5) | 40.2 (26.0 to 54.5) | 60.5 (54.8 to 66.1) | 20.2 (4.9 to 35.6) | 0.009 |
IV = intravenous; Q = quarter; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.
The table shows the effect of SEP-1 on processes of care, illustrated as projected differences in expected rates in Q4 2017, two years after SEP-1 implementation.
Indicates a repeat lactate measurement within 6 h of suspected infection, when the initial lactate level is greater than 2 mmol/L.
Indicates vasopressor initiation within 6 h of initial hypotension. The results are identical between the overall cohort and the hypotensive cohort because patients without hypotension were by definition excluded in the regression.
Figure 2.
Changes in unadjusted process measures in overall cohort.
The solid black circles represent unadjusted quarterly rates of 3 process measures: antibiotics (top), lactate (middle), and IV fluids (bottom). The hollow circle is the washout quarter. The vertical dashed line is the time of SEP-1 implementation in Q4 2015. The solid gray line is baseline trend without SEP-1, fitted in the baseline period and projected into the post-SEP-1 period. The solid black line is the fitted line for the trend in the post-SEP-1 period. The black vertical error bars in Q4 2017 represent 95% CIs surrounding the point estimate for the expected value of the process measure from the fitted line in the presence of SEP-1. The gray vertical error bars in Q4 2017 represent 95% CIs surrounding the point estimate for the expected value of the process measure from the fitted line in the pre-SEP-1 baseline, projected into the post–SEP-1 period. The difference between these expected values is the effect estimate for SEP-1 in Q4 2017. IV = intravenous; Q = quarter; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.
Implementation of SEP-1 was not associated with statistically significant or clinically meaningful changes in outcomes (Table 3 and Figure 3). Changes in intensive care unit admissions were not statistically significant and of uncertain clinical significance (absolute increase, 2.0 percentage points [CI, 0 to 4.0 percentage points]; P = 0.055) (Table 3). Mortality was decreasing in the period before SEP-1, and there was no significant difference in the change in in-hospital mortality after SEP-1, although the CIs were wide and cannot exclude the possibility of a clinically meaningful effect (absolute change, 0.1 percentage points [CI, −0.9 to 1.1 percentage points]; P = 0.87).
Table 3.
Differences in Risk-Adjusted Outcome Measures After SEP-1*
| Variable | Q3 2015 (Before SEP-1), Estimate (95% CI), % | Q4 2017 | Estimated Difference Between Expected and Observed in Q4 2017 (Effect of SEP-1) | ||
|---|---|---|---|---|---|
| Expected (Without SEP-1), Estimate (95% CI), % | Observed (With SEP-1), Estimate (95% CI), % | ||||
| Estimate (95% CI), percentage points | P Value | ||||
| All patients | |||||
| Admission to the ICU | 23.1 (22.3 to 24.0) | 20.4 (18.6 to 22.3) | 22.4 (21.5 to 23.2) | 2.0 (0 to 4.0) | 0.055 |
| In-hospital mortality | 4.5 (4.0 to 4.9) | 3.9 (3.0 to 4.8) | 4.0 (3.5 to 4.4) | 0.1 (−0.9 to 1.1) | 0.87 |
| Discharged hornet | 66.5 (65.6 to 67.5) | 69.8 (67.6 to 71.9) | 68.0 (67.0 to 69.0) | −1.8 (−4.1 to 0.6) | 0.145 |
| Patients with a mean arterial pressure <65 mm Hg in first 6 h | |||||
| Admission to the ICU | 45.4 (43.4 to 47.5) | 42.7 (37.8 to 47.5) | 45.4 (43.1 to 47.8) | 2.7 (−2.6 to 8.1) | 0.31 |
| In-hospital mortality | 8.8 (7.6 to 10.1) | 7.9 (5.2 to 10.6) | 8.8 (7.4 to 10.2) | 0.9 (−2.1 to 3.9) | 0.56 |
| Discharged home† | 59.6 (57.5 to 61.7) | 65.0 (60.3 to 69.8) | 59.0 (56.5 to 61.5) | −6.0 (−11.3 to −0.1) | 0.026 |
| Patients requiring vasopressors | |||||
| Admission to the ICU | 94.3 (91.6 to 97.1) | 90.5 (80.2 to 100) | 97.7 (96.1 to 99.4) | 7.2 (−3.1 to 17.6) | 0.172 |
| In-hospital mortality | 28.8 (24.2 to 33.2) | 25.8 (15.6 to 36.0) | 28.1 (23.1 to 33.2) | 2.3 (−8.8 to 13.4) | 0.68 |
| Discharged home† | 34.3 (29.1 to 39.5) | 32.4 (20.0 to 44.7) | 31.4 (24.6 to 37.1) | −1.0 (−14.4 to 12.3) | 0.88 |
ICU = intensive care unit; Q = quarter; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.
The table shows the effect of SEP-1 on clinical outcomes, illustrated as projected differences in expected rates in Q4 2017, two years after SEP-1 implementation. All outcomes were adjusted for age, Elixhauser comorbidities, admission Sequential Organ Failure Assessment score, source of infection, and seasonality.
Discharge to home only evaluated among patients who survived to hospital discharge.
Figure 3.
Changes in risk-adjusted outcome measures in overall cohort.
The solid black circles represent risk-adjusted quarterly rates of 3 outcome measures: ICU admission (top), in-hospital mortality (middle), and discharge home among survivors (bottom). The hollow circle is the washout quarter. The vertical dashed line is the time of SEP-1 implementation in Q4 2015. The solid gray line is baseline trend without SEP-1, fitted in the baseline period and projected into the post–SEP-1 period. The solid black line is the fitted line for the trend in the post–SEP-1 period. The black vertical error bars in Q4 2017 represent 95% CIs surrounding the point estimate for the expected value of the outcome measure from the fitted line in the presence of SEP-1. The gray vertical error bars in Q4 2017 represent 95% CIs surrounding the point estimate for the expected value of the outcome measure from the fitted line in the pre-SEP-1 baseline, projected into the post–SEP-1 period. The difference between these expected values is the effect estimate for SEP-1 in Q4 2017. ICU = intensive care unit; Q = quarter; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.
The results were similar in the subgroup analyses of patients with shock (Tables 2 and 3). The full details of the hospital-level results are in the Supplement (Supplement Figure 3). These results show some variation in process changes across hospitals (for example, 8 of 11 hospitals had an increase in adherence to lactate measurement); hospital-level changes in mortality were clustered around zero for 9 of 11 hospitals, which is consistent with the overall null effect.
Results of the sensitivity analyses are provided in the Supplement and support the primary findings. Process measure changes were similar when defined in reference to ED registration (Supplement Table 6). Process and outcome changes were similar in the SEP-1 reporting subgroup (Supplement Tables 12 and 13). There was some evidence for ascertainment bias from an increase in culture ordering (Supplement Table 14). The bias analysis showed an E-value of 2.4 for the change in lactate, meaning that a confounder would have to be associated with lactate adherence to an odds ratio of 2.4, and 2.4 times as likely in the period after SEP-1, to explain this effect (Supplement, page 24).
Discussion
In a study of 54 225 patients across 11 hospitals, we found that implementation of the SEP-1 program was associated with variable changes in processes of care related to sepsis resuscitation. The greatest change was a substantial increase in the rates of lactate measurement. There were also small and clinically less important increases in the rates of administration of early antibiotics and IV fluid. Notably, these changes in process measures were not associated with meaningful changes in clinical outcomes.
Our findings provide important insights into the use of health policy to improve the care of patients with sepsis and other forms of acute illness. The SEP-1 program was associated with increases in adherence to individual process measures, showing that pay-for-reporting programs can induce behavior change at the bedside. However, the overall effect was modest and largely limited to the element of SEP-1 with indirect effect on patient outcomes (lactate measurement), rather than the element most tightly linked to better outcomes (early antibiotics) (13). There are several potential explanations for this finding. First, it may simply be easier to order a laboratory test than to coordinate timely administration of antibiotics and IV fluids. Second, clinicians exercising discretion on the basis of concern for potential patient harm may be more likely to limit antibiotic or fluid therapy than lactate measurement (7,10). Overall, our findings highlight that, although the measure itself may be all-or-none, SEP-1 affected some processes more than others.
Our results contrast with evaluations of the New York State sepsis regulations, which suggest that some sepsis policies did improve outcomes (5, 6). The New York State sepsis regulations included requirements that hospitals develop and implement sepsis bundles, report adherence to the measures, and report clinical outcomes. The linking of structure, process, and outcome within an implementation framework meant that New York State told hospitals both what to do (process measures) and how to do it (protocol development and implementation) while holding them accountable for patient outcomes (21). This broad-based approach is much different than the narrower, process-only approach taken in the SEP-1 program.
It is also possible that the response to SEP-1 within study hospitals was insufficient to induce behavior changes large enough to translate into improvements in outcomes. That is, SEP-1 may have been more effective in other hospitals, either because they did more to improve sepsis quality in response to SEP-1 or because they had lower quality at baseline. However, there is no reason to suspect that UPMC’s efforts were different from those nationally, nor was baseline adherence high in our study, and the median publicly reported SEP-1 adherence in UPMC hospitals in fiscal year 2017 was 38%, which together do not suggest a ceiling effect.
Our results should be interpreted in the context of health care performance measurement on a national scale. Although CMS recently began publicly reporting the results of SEP-1 on Hospital Compare, initially SEP-1 was purely a pay-for-reporting program. Neither pay for reporting or public reporting are as effective as pay for performance, which imposes financial penalties based on formulas that include the actual level of performance (22). That said, there is limited evidence that pay-for-performance programs consistently improve care in the inpatient setting (23). Given that we saw some improvements in process measures related to SEP-1 implementation suggests that rather than having no effect, SEP-1 as a pay-for-reporting measure may simply not have achieved its intended, broad-based improvements in sepsis care.
Overall, our findings raise questions about the value of SEP-1. Hospitals are investing tremendous time and energy in responding to SEP-1, including many hours of data abstraction (14). Indeed, 1 cost analysis suggested that a single academic hospital invested more than $150 000 per month in the SEP-1 response (24). If SEP-1 incentivized hospitals to dedicate quality improvement resources to increasing adherence with processes that did not improve survival, it may have failed to achieve its intended goal. The CMS argued early in the course of SEP-1 development that although there could be some unintended consequences from unnecessary broad-spectrum antibiotics and IV fluids, SEP-1 would ultimately do more good than harm. Although the CIs for changes in mortality leave some uncertainty, our findings call that conclusion into question—if there were groups of patients whose outcomes improved, they seem to be at least balanced out by harm in other patients, meaning that averaged outcomes across the population were unchanged. Therefore, it is worth considering revisions to the measure that make it more targeted and simpler, such as focusing on early antibiotics for patients with septic shock, in which the benefits may be greatest (10, 11). Another important revision may be incorporating a sepsis outcome measure—the development of which is already under way (25).
Sepsis is a complex syndrome without a gold-standard definition, and our study highlights the strength of using granular EHR data in sepsis research. We defined sepsis on the basis of suspected infection and organ dysfunction, which is consistent with current consensus definitions but differs from the case definition for the SEP-1 measure (7). However, our approach avoided biases inherent in using diagnosis codes, which changed at the same time SEP-1 was introduced (15). Our approach also should validly capture the effects of a policy like SEP-1 on a population of patients in whom clinicians may reasonably have suspected sepsis at the point of care, rather than on a group of patients that data abstractors determined retrospectively to have had sepsis. Different case definitions inherently capture cohorts with different illness severity—and potentially different susceptibility to ascertainment bias in the face of external policy incentives (5, 6, 24). Because our case definition included the ordering of body fluid cultures, increases in blood culture ordering associated with SEP-1 could have included less severely ill patients after SEP-1, creating a bias in favor of lower mortality after SEP-1, which we did not observe. Ultimately, defining a complex syndrome like sepsis in a way that operates consistently across studies and is not susceptible to ascertainment bias remains an ongoing challenge for the field (26).
Our study has several other potential limitations. First, our population had relatively low overall mortality, increasing the possibility that the patients were not sick enough to benefit from small increases in the quality of care. However, our results were consistent in multiple subgroups with higher baseline mortality. Second, we used the time of suspected infection as the time of sepsis onset in our primary analysis. The issue of sepsis time zero is controversial and without a gold standard. It is reassuring that the results were similar in a sensitivity analysis using the time of ED arrival as the time of onset. Third, although there seems to be a clear shift in practice in association with SEP-1, we cannot rule out the possibility that other sepsis awareness campaigns influenced care over time, that there were anticipation effects related to the announcement of SEP-1 in the years before implementation, or that a longer observation period would have revealed changes in outcomes. Fourth, our analysis lacked a control group to illustrate trends during the post–SEP-1 period, so our conclusions rely on the assumption that the pre-SEP-1 trends continued unchanged into the post–SEP-1 period; changes in the trend absent SEP-1 could alter our conclusions about the effects of SEP-1. Fifth, we did not specifically examine changes in the use or effect of broad-spectrum antibiotics; this is an important area for future work. Finally, although we included data from several hospitals of varying size and location, these hospitals were all part of a single health system. It is possible that the local context in other health systems would yield different findings.
Overall, our findings suggest that SEP-1 implementation had variable effects on its targeted process measures and that either there was no associated change in outcomes or that any benefits to patients may be counterbalanced by harm in other patients from the fixed application of a sepsis treatment bundle. Policymakers could consider revisions to the measure that simplify the components, permitting clinicians some discretion on the basis of patient characteristics, and allowing clinicians to focus on the aspects of sepsis care that most directly drive improvements in patient outcomes.
Supplementary Material
Acknowledgment:
The authors thank the University of Pittsburgh’s Clinical and Translational Science Institute and the Department of Critical Care Medicine’s Biostatistical and Data Management Core for assistance in preparation of the raw data for this study.
Grant Support:
By grant K08HS025455 from the Agency for Healthcare Research and Quality (Dr. Barbash) and grants K24HL133444 (Dr. Kahn) and UL1TR001857 (Clinical and Translational Science Institute) from the National Institutes of Health.
Footnotes
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M20-5043.
Reproducible Research Statement: Study protocol and statistical code: Available from Dr. Barbash (barbashij@upmc.edu). Data set: Not available.
References
- 1.Rhee C, Dantes R, Epstein L, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009–2014. JAMA. 2017;318:1241–1249. doi: 10.1001/jama.2017.13836 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Liu V, Escobar GJ, Greene JD, et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA. 2014;312:90–2. [DOI] [PubMed] [Google Scholar]
- 3.Buchman TG, Simpson SQ, Sciarretta KL, et al. Sepsis among Medicare beneficiaries: 1. The burdens of sepsis, 2012–2018. Crit Care Med. 2020;48:276–288. doi: 10.1097/CCM.0000000000004224 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Seymour CW, Gesten F, Prescott HC, et al. Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med. 2017;376:2235–2244. doi: 10.1056/NEJMoa1703058 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Levy MM, Gesten FC, Phillips GS, et al. Mortality changes associated with mandated public reporting for sepsis: the results of the New York State initiative. Am J Respir Crit Care Med. 2018;198:1406–1412. doi: 10.1164/rccm.201712-2545OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kahn JM, Davis BS, Yabes JG, et al. Association between state-mandated protocolized sepsis care and in-hospital mortality among adults with sepsis. JAMA. 2019;322:240–250. doi: 10.1001/jama.2019.9021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Barbash IJ, Kahn JM, Thompson BT. Opening the debate on the new sepsis definition. Medicare’s sepsis reporting program: two steps forward, one step back. Am J Respir Crit Care Med. 2016;194:139–41. doi: 10.1164/rccm.201604-0723ED [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Klompas M, Rhee C. The CMS sepsis mandate: right disease, wrong measure. Ann Intern Med. 2016;165:517–518. doi: 10.7326/M16-0588 [DOI] [PubMed] [Google Scholar]
- 9.Pepper DJ, Jaswal D, Sun J, et al. Evidence underpinning the Centers for Medicare & Medicaid Services’ Severe Sepsis and Septic Shock Management Bundle (SEP-1): a systematic review. Ann Intern Med. 2018;168:558–568. doi: 10.7326/M17-2947 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Rhee C, Chiotos K, Cosgrove SE, et al. Infectious Diseases Society of America position paper: recommended revisions to the national Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) sepsis quality measure. Clin Infect Dis. 2020. doi: 10.1093/cid/ciaa059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Rhee C, Strich JR, Klompas M, et al. SEP-1 has brought much needed attention to improving sepsis care…but now is the time to improve SEP-1. Crit Care Med. 2020;48:779–782. doi: 10.1097/CCM.0000000000004305 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Barbash IJ, Davis B, Kahn JM. National performance on the Medicare SEP-1 sepsis quality measure. Crit Care Med. 2019;47:1026–1032. doi: 10.1097/CCM.0000000000003613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Rhee C, Filbin MR, Massaro AF, et al. Compliance with the national SEP-1 quality measure and association with sepsis outcomes: a multicenter retrospective cohort study. Crit Care Med. 2018;46:1585–1591. doi: 10.1097/CCM.0000000000003261 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Barbash IJ, Rak KJ, Kuza CC, et al. Hospital perceptions of Medicare’s sepsis quality reporting initiative. J Hosp Med. 2017;12:963–968. doi: 10.12788/jhm.2929 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kempker AJ, Rudd KE, Wang HE, et al. Sepsis epidemiology across the International Classification of Diseases, 9th edition, to International Classification of Diseases, 10th edition, chasm—a direct application of the Institute for Health Metrics and Evaluation case definition to hospital discharge data. Crit Care Med. 2020;48: 1881–1884. doi: 10.1097/CCM.0000000000004577 [DOI] [PubMed] [Google Scholar]
- 16.Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315:762–74. doi: 10.1001/jama.2016.0288 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315:801–10. doi: 10.1001/jama.2016.0287 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Rhee C, Wang R, Zhang Z, et al. Epidemiology of hospital-onset versus community-onset sepsis in U.S. hospitals and association with mortality: a retrospective analysis using electronic clinical data. Crit Care Med. 2019;47:1169–1176. doi: 10.1097/CCM.0000000000003817 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care. 1998;36:8–27. [DOI] [PubMed] [Google Scholar]
- 20.Liu VX, Fielding-Singh V, Greene JD, et al. The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med. 2017;196:856–863. doi: 10.1164/rccm.201609-1848OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Donabedian A The quality of care: how can it be assessed. JAMA. 1988;260:1743–8. [DOI] [PubMed] [Google Scholar]
- 22.Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007;356:486–96. [DOI] [PubMed] [Google Scholar]
- 23.Mendelson A, Kondo K, Damberg C, et al. The effects of pay-for-performance programs on health, health care use, and processes of care: a systematic review. Ann Intern Med. 2017;166:341–353. doi: 10.7326/M16-1881 [DOI] [PubMed] [Google Scholar]
- 24.Afshar M, Arain E, Ye C, et al. Patient outcomes and cost-effectiveness of a sepsis care quality improvement program in a health system. Crit Care Med. 2019;47:1371–1379. doi: 10.1097/CCM.0000000000003919 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Centers for Medicare & Medicaid Services. Technical expert panel composition (membership) list. Accessed at www.cms.gov/files/document/patient-safety-measure-development-and-maintenance-sepsis-outcome-measure-tep-composition-list.pdf on 5 December 2020.
- 26.Angus DC. Opening the debate on the new sepsis definition defining sepsis: a case of bounded rationality and fuzzy thinking. Am J Respir Crit Care Med. 2016;194:14–5. doi: 10.1164/rccm.201604-0879ED [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.