Compliance with the National SEP-1 Quality Measure and Association with Sepsis Outcomes: A Multicenter Retrospective Cohort Study

Abstract

Objectives

Many septic patients receive care that fails the Centers for Medicare and Medicaid Services’ SEP-1 measure, but it is unclear whether this reflects meaningful lapses in care, differences in clinical characteristics, or excessive rigidity of the “all-or-nothing” measure. We compared outcomes in cases that passed versus failed SEP-1 during the first 2 years after the measure was implemented.

Design

Retrospective cohort study.

Setting

Seven U.S. hospitals.

Patients

Adult patients included in SEP-1 reporting between October 2015 and September 2017.

Interventions

None.

Measurements and Main Results

Of 851 sepsis cases in the cohort, 281 (33%) passed SEP-1 and 570 (67%) failed. SEP-1 failures had higher rates of septic shock (20% vs 9%, p<0.001), hospital-onset sepsis (11% vs 4%, p=0.001), and vague presenting symptoms (46% vs 30%, p<0.001). The most common reasons for failure were omission of 3-hour and 6-hour lactate measurements (228/570 failures, 40%). Only 86/570 failures (15.1%) had >3-hour delays until broad-spectrum antibiotics. Cases that failed SEP-1 had higher in-hospital mortality rates (18.4% vs 11.0%, OR 1.82, 95% CI 1.19-2.80, p=0.006) but this association was no longer significant after adjusting for differences in clinical characteristics and severity-of-illness (adjusted OR 1.36, 95% CI 0.85-2.18, p=0.205). Delays of >3-hours until antibiotics were significantly associated with death (adjusted OR 1.94, 95% CI 1.04-3.62, p=0.038) while failing SEP-1 for any other reason was not (adjusted OR 1.10, 95% CI 0.70-1.72, p=0.674).

Conclusions

Crude mortality rates were higher in sepsis cases that failed versus passed SEP-1 but there was no difference after adjusting for clinical characteristics and severity-of-illness. Delays in antibiotic administration were associated with higher mortality but only accounted for a small fraction of SEP-1 failures. SEP-1 may not clearly differentiate between high versus low-quality care and detailed risk adjustment is necessary to properly interpret associations between SEP-1 compliance and mortality.

In October 2015, the Centers for Medicare and Medicaid Services (CMS) began requiring U.S. hospitals to report compliance rates with the “SEP-1” core sepsis measure. The severe sepsis bundle requires lactate measurements, blood cultures, and broad-spectrum antibiotics within 3 hours of sepsis onset, with repeat lactate measurements within 6 hours if the initial lactate is >2.0mmol/L.[1] The septic shock bundle also requires 30 cc/kg of intravenous fluids within 3 hours, vasopressors within 6 hours for persistent hypotension, and a repeat volume assessment exam within 6 hours.[1]

Preliminary data from CMS indicate that the majority of SEP-1 cases nationally fail the measure and cases that fail have higher mortality rates than cases that pass.[2] It is unclear, however, whether failures are due to clinically meaningful lapses in care or whether the measure is overly prescriptive. CMS imposes very strict conditions to pass SEP-1, including detailed documentation of volume status, repeat lactate measurements regardless of patients’ clinical appearance, and little flexibility to accommodate relative contraindications to aggressive fluid resuscitation.[3, 4] It is also unclear if higher mortality rates for cases that fail SEP-1 are due to inferior care or higher severity-of-illness. For example, SEP-1 has more requirements for septic shock compared to severe sepsis alone, which may make SEP-1 failure more likely and inflate its apparent impact on mortality.[5]

In addition, the evidence supporting each of the components included in SEP-1 is variable. Some measures, such as time to antibiotic administration, are relatively well supported whereas lactate measurements, volume reassessments, and how much fluids to give patients are more controversial [6–11]. As an “all-or-nothing measure” that requires perfect performance to pass, SEP-1 gives equal weight to all of these components.

Given the substantial resources being devoted by hospitals to SEP-1 compliance and reporting, we evaluated the association between SEP-1 compliance and patient outcomes taking into account patient’s clinical characteristics. We examined sepsis cases reported by 7 academic and community hospitals to CMS during the first 2 years after SEP-1 implementation.

METHODS

Study Design, Patients, and Setting

This was a retrospective cohort study of sepsis cases submitted by 7 hospitals to CMS for the SEP-1 measure from October 1^st, 2015 – when SEP-1 went into effect – through September 31^st, 2017. SEP-1 adherence was measured by quality staff at each hospital who reviewed 20 randomly selected cases per month with discharge ICD-10 codes for sepsis, as per CMS requirements. Quality staff assessed whether patients met CMS criteria for severe sepsis (i.e., documentation of suspected infection, ≥2 systemic inflammatory response syndrome criteria, and organ dysfunction), when “time zero” occurred, and whether sepsis bundles were completed [1](see Online Supplement, Appendix A and B for a summary of SEP-1 criteria). CMS exclusion criteria included transfer from outside facilities, documented goals of care precluding sepsis care, or hospital length-of-stay greater than 120 days. We also excluded cases transferred out of study hospitals to other acute care hospitals since their vital status at final discharge could not be ascertained.

The primary study sites included 2 academic referral hospitals in Boston, MA (Massachusetts General Hospital and Brigham and Women’s Hospital) and 3 community hospitals in Eastern Massachusetts (Brigham and Women’s Faulkner Hospital, North Shore Medical Center, and Newton Wellesley Hospital). In addition, Barnes-Jewish Hospital in St. Louis, MO and Duke University Hospital in Durham, NC (both academic referral hospitals) each contributed 30 randomly selected cases from quarters 3 or 4 of 2016 that met inclusion criteria. The study was approved by the Institutional Review Boards at Harvard Pilgrim Health Care Institute, Partners Healthcare, Washington University School of Medicine, and Duke University Health System.

Outcome and Variables

The primary outcome was in-hospital mortality. The primary exposure was failing SEP-1 (on any bundle component). Covariates from SEP-1 reporting included age, sex, race, specialty of discharging physician (medical, surgical, or other), and presence of septic shock (defined by initial lactate ≥4 mmol/L or persistent hypotension despite a fluid bolus of ≥30 cc/kg, as per CMS criteria [1]). Study investigators also reviewed medical records to assess organ dysfunction at severe sepsis time zero, body site of infection (pulmonary, urinary, intra-abdominal, or other), positive blood cultures (within ± 48 hours of time zero, excluding common skin contaminants), and ICU admission and discharge dates. We calculated comorbidities and a weighted comorbidity score using the Elixhauser method for International Statistical Classification of Diseases and Related Health Problems, 10^th-revision discharge diagnosis codes.[12–14] Hospital-onset sepsis was defined as time zero occurring more than 48 hours after admission.

SEP-1 reporting requirements allow abstractors to stop once any bundle component is determined to be non-compliant; for example, if a patient failed an initial lactate check, hospital quality officers did not routinely assess whether care teams passed or failed all subsequent components. Study investigators manually reviewed all cases, however, to identify the time of administration of intravenous broad-spectrum antibiotics. “Broad-spectrum” antibiotics were defined per CMS SEP-1 criteria, which requires monotherapy with broad-spectrum beta-lactams or fluoroquinolones, or combination therapy with two narrower-spectrum antibiotics.[1]

We also reviewed medical records for documentation of explicit infectious symptoms versus vague symptoms at the time of presentation to the emergency department for sepsis present-on-admission or within the 24 hours before hospital-onset sepsis, since certain symptoms may increase the likelihood that clinicians recognize and treat sepsis.[15] Explicit infectious symptoms were defined as fever (including fever at triage), sweats, chills, rigors, productive cough, dysuria, overt skin/soft tissue changes (e.g., unilateral limb erythema, abscess, or draining wound), or referral from an outside provider for documented infection (e.g., positive blood cultures), while vague infectious symptoms included altered mental status, weakness, fatigue, malaise, focal neurologic symptoms, abdominal pain, nausea, vomiting, diarrhea, hypotension, shortness of breath, dry cough, hypoxemia, or unexplained laboratory abnormalities without explicit infectious symptoms.[15]

Statistical Analysis

We compared characteristics of cases that passed versus failed SEP-1 using the Wilcoxon rank sum test for continuous variables and the chi-squared statistic for categorical variables. We used univariate logistic regression to assess associations between individual covariates and in-hospital death. We included the year of hospitalization (year 2 vs 1 of the study) as a covariate to account for possible temporal changes in SEP-1 compliance and minor specification changes that CMS introduced after the first year. Multivariate logistic regression was used to assess associations between SEP-1 failure and death. Age, sex, and race were included in the multivariable model a priori given their known association with sepsis outcomes.[16, 17] Additional variables were chosen by first including all covariates with univariate p-values ≤0.20. We then removed all covariates with adjusted p-values >0.10 from the multivariate model. The c-statistic was calculated to assess the discriminatory performance of the final multivariate model.

Time-to-antibiotics was not included as a separate covariate due to collinearity with the SEP-1 measure. In a sensitivity analysis, however, we replaced SEP-1 failure with one variable for time-to-antibiotics >3-hours (which was assessed for all study patients, including those that failed SEP-1 earlier in the bundle pathway) and one variable for SEP-1 failure due to any reason other than time-to-antibiotics. All analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). We considered p<0.05 to be statistically significant and used two-tail tests.

RESULTS

Patient Characteristics and Reasons for SEP-1 Failure

A flowchart demonstrating the study cohort derivation and exclusions is shown in Figure 1. Of the 851 sepsis patients available for analysis, 281 (33.0%) passed SEP-1 while 570 (67.0%) failed. SEP-1 compliance rates were higher in the second year of the study versus the first (36.2% vs 29.6%, p=0.002).

Figure 1.

Flowchart for study cohort derivation and exclusions

Cases that failed SEP-1 were similar to those that passed in terms of age, sex, race, and comorbidity burden, but were significantly different with respect to other clinical characteristics (Table 1). Notably, SEP-1 failures were more likely to have septic shock, hospital-onset sepsis, vague rather than explicit infectious symptoms, and non-pulmonary infections compared to cases that passed.

Table 1.

Characteristics and Outcomes of Sepsis Patients that Passed versus Failed SEP-1

Clinical Characteristics	Pass (n=281)	Fail (n=570)	p-value
Median Age (IQR)	68 (57-81)	67 (57-80)	0.319
Male Sex	155 (55.2%)	303 (53.2%)	0.582
White Race	223 (79.4%)	446 (78.3%)	0.710
Median Elixhauser Score (IQR)	11 (5-16)	11 (5-17)	0.608
Academic vs Community Hospital	144 (51.3%)	301 (52.8%)	0.668
Discharged in Study Year 2 (vs Year 1)	158 (56.2%)	278 (48.8%)	0.041*
Discharging Service
Medical	206 (73.3%)	407 (71.4%)	0.560
Surgical	4 (1.4%)	37 (6.5%)	0.001*
Other	71 (25.3%)	125 (21.9%)	0.277
Sepsis Onset in Emergency Department	232 (82.6%)	421 (73.9%)	0.005*
Hospital-Onset Sepsis (>48 hours from presentation)	12 (4.3%)	63 (11.1%)	0.001*
Initial Sepsis Organ Dysfunction
Hypotension	87 (31.0%)	189 (33.2%)	0.520
Lactate >2 and <4	80 (28.5%)	138 (24.2%)	0.181
Lactate ≥4	18 (6.4%)	72 (12.6%)	0.006*
Respiratory Failure	13 (4.6%)	37 (6.5%)	0.277
Creatinine >2	20 (7.1%)	36 (6.3%)	0.658
Bilirubin >2	8 (2.9%)	13 (2.3%)	0.617
Platelets <100	10 (3.6%)	15 (2.6%)	0.452
INR >1.5 or PTT >60	4 (1.4%)	9 (1.6%)	0.862
MD documentation of severe sepsis/septic shock	41 (14.6%)	61 (10.7%)	0.101
Septic Shock (Persistent Hypotension or Lactate ≥4)	25 (8.9%)	112 (19.7%)	<0.001*
Positive blood cultures	75 (26.7%)	160 (28.1%)	0.672
Explicit Infectious Symptoms at Presentation	197 (70.1%)	310 (54.4%)	<0.001*
Body Site Source of Infection
Pneumonia	113 (40.2%)	188 (33.0%)	0.038*
Urinary Tract Infection	66 (23.5%)	137 (24.0%)	0.860
Intra-abdominal Infection	50 (17.8%)	105 (18.4%)	0.824
Other	52 (18.5%)	140 (24.6%)	0.047*
Outcomes
Required ICU Stay	142 (50.5%)	299 (52.5%)	0.598
Median ICU Length of Stay (IQR)	3 (2-6)	4 (2-9)	0.030*
Median Hospital LOS	7 (5-12)	8 (5-13)	0.132
In-Hospital Death	31 (11.0%)	105 (18.4%)	0.006*

^*Indicates statistically significant variables at p<0.05.

The reasons that cases failed SEP-1 are shown in Table 2. Failure to draw an initial lactate or repeat lactate within 6 hours accounted for 40% of failures. Among all 570 cases that failed (including those that failed to have initial lactate or blood cultures drawn), only 86 (15.1%) patients had delays of >3 hours until broad-spectrum antibiotic administration.

Table 2.

Reasons for SEP-1 Failure

Bundle Failure Reason	Number of Failures (%)* (Total N=570)
Initial Lactate Not Drawn Within 3 Hours	112 (19.7%)
Blood Cultures Within 3 Hours (Not Drawn, or Drawn After Antibiotics)	86 (15.1%)
Antibiotics Within 3 Hours
Not Given	77 (13.5%)
Inappropriate Selection	12 (2.1%)
Repeat Lactate Not Drawn Within 6 Hours	116 (20.4%)
Crystalloids (Inadequate Amount or Not Given within 3 Hours)	104 (18.3%)
Persistent Hypotension Not Assessed After Crystalloid Fluids	4 (0.7%)
Vasopressors Not Given Within 6 Hours of Persistent Hypotension	8 (1.4%)
Volume Assessment Not Done within 6 Hours of Septic Shock	42 (7.4%)

^*The distribution includes only the first component of the SEP-1 bundle that failed in each case.

SEP-1 Compliance and Mortality

Of the 851 sepsis patients, 136 (16.0%) died in-hospital. Sepsis mortality was similar in the first versus second year of the study (68/415, 16.4% versus 68/368, 15.6%, p=0.441). The results of the univariate screen and multivariate analysis are shown in Table 3. Unadjusted mortality rates were higher for SEP-1 failures (18.4% vs 11.0%, OR 1.82, 95% CI 1.19, 2.80, p=0.006) but this difference was no longer significant after adjusting for patients’ clinical characteristics (adjusted OR 1.36, 95% CI 0.85, 2.18, p=0.205). Variables significantly associated with an increased odds of death on multivariate analysis included age, non-white race, higher Elixhauser score, hospital-onset sepsis, septic shock, non-urinary source of infection, and vague presenting symptoms. The model’s c-statistic was 0.79.

Table 3.

Univariate and Multivariate Models Examining Factors Associated with Death

Covariates	Univariate Screen		Multivariate Model
Age (Continuous)*	1.01 [1.00, 1.02]	0.057	1.02 [1.00, 1.03]	0.016
Male Sex*	1.03 [0.71, 1.49]	0.880		0.256
White Race*	0.78 [0.51, 1.20]	0.263	0.60 [0.37, 0.96]	0.035
Elixhauser Score* (Continuous)	1.06 [1.04, 1.09]	<0.001	1.05 [1.03, 1.08]	<0.001
Academic Hospital (vs Community)*	1.64 [1.13, 2.40]	0.010
Study Year 2 vs Year 1	0.94 [0.65, 1.36]	0.754
Discharging Service		0.239
Medical	REFERENCE
Surgical	1.41 [0.63, 3.15]
Other	1.40 [0.92, 2.13]
Hospital-Onset Sepsis*	5.13 [3.11, 8.47]	<0.001	4.61 [2.62, 8.10]	<0.001
Hypotension at Sepsis Onset	1.21 [0.83, 1.78]	0.329
Septic Shock (Persistent Hypotension or Lactate ≥4 mmol/L)*	1.70 [1.08, 2.66]	0.022	1.89 [1.14, 3.12]	0.014
Respiratory Failure at Sepsis Onset*	2.95 [1.59, 5.47]	<0.001	2.00 [0.98, 4.06]	0.056
Vague Symptoms*	3.16 [2.16, 4.64]	<0.001	2.36 [1.53, 3.62]	<0.001
Body Site of Infection*		<0.001		<0.001
Urinary	REFERENCE		REFERENCE
Pulmonary	3.49 [1.86, 6.55]		3.23 [1.64, 6.38]
Abdominal	2.55 [1.25, 5.21]		2.24 [1.04, 4.84]
Other	4.09 [2.12, 7.90]		4.20 [2.06, 8.58]
Positive Blood Cultures	1.11 [0.74, 1.66]	0.609
Failing SEP-1 (All-or-Nothing)	1.82 [1.19, 2.80]	0.006	1.36 [0.85, 2.18]	0.205

^*Indicates variables that were included in the multivariate model, based on significance at p<0.20 on univariate screen or a priori decision to include (age, sex, race, and failing SEP-1). Academic hospital was dropped in the intermediate model as its p-value was >0.10.

On sensitivity analysis, time-to-antibiotics of >3 hours was significantly associated with death (adjusted OR 1.94, 95% CI 1.04, 3.62, p=0.038) while failing SEP-1 for any reason other than time-to-antibiotics was not (adjusted OR 1.10, 95% CI 0.70, 1.72, p=0.674). Findings were consistent for patients with severe sepsis alone versus those with septic shock and patients with community- versus hospital-onset sepsis; however, both SEP-1 failure and >3 hour delays in antibiotics were associated with higher mortality in patients with explicit infectious signs but not those with vague presenting complaints (eTable in the Supplement).

DISCUSSION

Most sepsis patients in this multicenter cohort received care that was non-compliant with the national SEP-1 measure. Mortality rates were higher in cases that failed SEP-1 compared to those that passed, but SEP-1 failures were more likely to have septic shock, hospital-onset sepsis, and vague infectious presenting symptoms. There was no significant difference in mortality between SEP-1 passes versus failures after adjusting for these differences. Delays in broad spectrum antibiotics were associated with higher mortality rates but only accounted for a fraction of SEP-1 failures.

Our findings of similar adjusted outcomes in cases that failed versus passed SEP-1 may reflect the overly rigid nature of the measure rather than ineffectiveness of timely sepsis care. In particular, SEP-1 does not allow partial credit for completing some bundle components nor does it prioritize any bundle components over others. The most common reasons for failure in our cohort were not measuring initial or repeat lactate levels. Although lactate levels may help risk stratify patients [18–20], there is limited evidence that measuring lactate improves patient outcomes.[10] Many cases also failed because clinicians administered inadequate volumes of crystalloid fluids or neglected to document a repeat volume assessment exam. Only 15% of failures were due to delays >3 hours in administering antibiotics, the one bundle component that was associated with higher mortality on multivariate analysis. This mortality association is consistent with prior studies suggesting that timely antibiotics are the most important component of sepsis bundles, particularly in patients with septic shock.[7, 8, 21–23] In contrast, there is little evidence to support the fluid bundle component or the other SEP-1 hemodynamic interventions.[7, 11]

In our cohort, SEP-1 failures were more common amongst patients with septic shock, presumably because this requires more steps to be performed and documented to pass. SEP-1 failures were also more common in hospital-onset sepsis, which tends to occur in more severely ill patients and is associated with worse outcomes than community-onset sepsis.[24] Previous studies have also demonstrated that delays in sepsis recognition and management are more common on hospital wards compared to emergency departments, where sepsis awareness and protocolized care tends to be more common.[5, 25, 26]

We found that explicit infectious symptoms were strongly associated with SEP-1 compliance, timely antibiotics, and survival rates. Previous studies have documented that fever is associated with faster sepsis recognition [27–29], but this study and a companion analysis [15] extend this observation to include other obvious signs of infection. Our findings also suggests that presenting symptoms may be an important unmeasured confounder in other observational studies that have suggested lower mortality rates with rapid sepsis bundle application.[15, 30–35] Conversely, patients with vague presenting symptoms may suffer worse outcomes because of delays in recognition and care or more frequent comorbid conditions. In addition, the lack of benefit of sepsis bundles and timely antibiotics in patients with vague symptoms may be because true infections are less common in this population.

Our study has several limitations. First, our findings may not be generalizable to other healthcare systems. However, our rate of SEP-1 compliance is similar to what has been reported nationwide [2], and our hospitals included both academic and community hospitals from 3 different states. Second, it is possible that our study was underpowered to detect a statistically significant association of failing SEP-1 with mortality. However, our sensitivity analyses demonstrated the significance of time-to-antibiotics, and the effect estimate was close to one for all SEP-1 component failures other than timely antibiotics. Third, as with all observational studies, we cannot rule out the possibility of residual confounding. Fourth, CMS introduced minor changes in the SEP-1 specification in the second year of SEP-1. However, study year had no influence in our model. Lastly, aside from antibiotic administration time, we were unable to measure the relative contributions of different components of the SEP-1 bundle or percentage of total bundle compliance to patients’ outcomes, since data on each component was not available in patients who failed the measure. This also means that our reported failure rates for individual SEP-1 bundle components may underestimate their true failure rates.

In conclusion, our early experience with SEP-1 demonstrates a high rate of SEP-1 failures and higher crude mortality rates in sepsis cases that failed versus passed, but no difference in mortality after adjusting for clinical characteristics and severity-of-illness. The all-or-nothing nature of SEP-1 fails to differentiate between vital factors, such as early antibiotic administration, versus secondary factors, such as measuring lactates and documenting volume status. In addition, sophisticated risk adjustment is necessary to interpret differences in outcomes between SEP-1 passes versus failures. These findings call into question the utility of SEP-1 as currently structured and suggest possible ways to improve the measure.