The Effect of Severe Sepsis and Septic Shock Management Bundle (SEP-1) Compliance and Implementation on Mortality Among Patients With Sepsis

Abstract

Background:

The Centers for Medicare & Medicaid Services (CMS) Severe Sepsis and Septic Shock Management Bundle (SEP-1) is now included in the Hospital Value-Based Purchasing (VBP) Program.

Purpose:

To assess the evidence supporting SEP-1 compliance or SEP-1 implementation in improving sepsis mortality.

Data Sources:

PubMed, Web of Science, EMBASE, CINAHL Complete, and Cochrane Library from inception to 26 November 2024.

Study Selection:

Studies of adults with sepsis that included 3- or 6-hour sepsis bundles defined by SEP-1 specifications.

Data Extraction:

Article screening, full-text review, data extraction, and risk-of-bias assessment were independently performed by 2 authors. Level of evidence was determined using GRADE (Grading of Recommendations Assessment, Development and Evaluation) criteria and National Quality Forum criteria.

Data Synthesis:

A total of 4403 unique references were screened, and 17 studies were included. Twelve studies assessed the relationship between SEP-1 compliance and mortality; 5 showed statistically significant benefit, whereas 7 did not. Among studies showing benefit, 1 did not adjust for confounders, 1 found benefit only among patients with severe sepsis, 1 included only patients with septic shock, and 1 included only Medicare beneficiaries. Five studies assessed the relationship between SEP-1 implementation and sepsis mortality; only 1 showed significant benefit, but it did not adjust for mortality trends before SEP-1 implementation. All 17 studies were observational, and none had low risk of bias.

Limitations:

The conclusions are limited by the underlying quality of the available studies, as all were observational. Because there was considerable methodologic heterogeneity among the included studies, a meta-analysis was not performed as the results could have been misleading.

Conclusion:

This review found no moderate- or high-level evidence to support that compliance with or implementation of SEP-1 was associated with sepsis mortality. CMS should reconsider the addition of SEP-1 to the Hospital VBP Program.

Primary Funding Source:

None. (PROSPERO: CRD-42023482787)

Sepsis, a dysregulated host response to infection, is one of the most common, deadly, and costly conditions in the United States, with more than 1.7 million cases and more than 350 000 deaths each year (1). Because sepsis is a heterogeneous syndrome, protocolized sepsis management (“sepsis bundles”), although operationally convenient on a population level, may not account for individual patient response to therapy (2–6). In 2015, the Centers for Medicare & Medicaid Services (CMS) implemented the Severe Sepsis and Septic Shock Management Bundle (SEP-1) as a pay-for-reporting measure (Supplement Section A, available at Annals.org) (7–10).

Multiple professional medical societies and patient advocacy groups support the SEP-1 quality metric, citing increased awareness of sepsis care and improved outcomes (11, 12). In contrast, in May 2020, the Infectious Diseases Society of America (IDSA) published a position paper that outlined recommended changes to SEP-1 aimed at addressing issues related to sepsis overdiagnosis and downstream sequelae, including antibiotic overuse and antibiotic resistance (13–15). In August 2023, CMS announced it would include SEP-1 as a pay-for-performance measure by incorporating it into the Hospital Value-Based Purchasing (VBP) Program, citing data suggesting that antibiotic and fluid-focused sepsis bundles were associated with improvement in patient-centered outcomes (16–19). In October 2023, IDSA published a joint position paper with 5 other professional organizations that recommended retiring SEP-1 in favor of outcome-based metrics (10).

The inclusion of SEP-1 in the Hospital VBP Program will directly affect hospital Medicare reimbursements. Thus, assessing the effect of SEP-1 compliance on patient-centered outcomes has important implications for health care policy, administration, economics, and clinical practice. A 2019 systematic review of observational studies found that bundles that focused on antibiotic delivery and fluid resuscitation showed mortality benefit, but this effect was seen regardless of antibiotic timing and fluid volume (19). Of note, none of these studies were assessed to have low risk of bias and there was substantial heterogeneity among them, limiting the interpretability of the meta-analysis (19, 20). Another systematic review found no moderate- or high-level evidence to support the hemodynamic components of SEP-1, but at the time, only 1 study had been published that assessed SEP-1 in totality (21).

In this study, we performed a systematic review to assess whether there was any moderate- or high-level evidence to suggest that SEP-1 compliance or SEP-1 implementation was associated with an improvement in sepsis mortality.

Methods

We conducted this systematic review according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement (Supplement Section B, available at Annals.org) and registered it in the PROSPERO database (CRD42023482787) on 26 November 2023; the protocol was updated on 13 November 2024. The systematic review was conducted using the Covidence software tool (Veritas Health Innovation) (22). Conceptually, there were 2 principal research questions this review sought to address: 1) Is there evidence that SEP-1 compliance is associated with improvement in sepsis mortality, and 2) is there evidence that SEP-1 implementation is associated with improvement in sepsis mortality? The first question is a direct assessment of the utility of SEP-1, given that the effect of bundle compliance is assessed at the individual patient level. The second question is an indirect assessment of the clinical utility of SEP-1, as bundle compliance is assessed at the system level, where individual-level bundle compliance may not be known.

Data Sources and Search

We searched PubMed, Web of Science, EMBASE, CINAHL Complete, and the Cochrane Library, without language restrictions, between inception and 26 November 2024 (the date of the search) (Supplement Section C, available at Annals.org). Previous studies and systematic reviews have thoroughly assessed the effects of individual components on mortality, and we did not include these articles in our review (21). The search was constructed in consultation with a research librarian (P.T.). Our search terms were informed by a review of search terms from a previously conducted systematic review (21).

Study Selection

We included all studies of adults (aged ≥18 years) with sepsis, severe sepsis, or septic shock. We made an exception to the age criterion for 1 study that included patients aged 16 years or older but had a median age of 45 years (23). Because SEP-1 is an “all-or-nothing” metric, we specifically looked for studies assessing the entire 3-hour bundle and/or the entire 6-hour bundle (with the exception of the volume reassessment), even if the bundle was not explicitly named “SEP-1.” We included studies that compared mortality between patients who received each bundle and those who did not (effect of compliance) as well as studies that compared sepsis mortality between a pre–SEP-1 implementation period and a post–SEP-1 implementation period (effect of implementation). We excluded studies with bundles that were not sufficiently similar to SEP-1. For example, the New York State (NYS) Department of Health sepsis bundle prescribes a 30-cc/kg bolus (for patients with hypotension or a lactate level ≥4 mmol/L) within 6 hours, whereas SEP-1 prescribes the same bolus within 3 hours (24). In addition, the time frames for the Surviving Sepsis Campaign 2008 (1-hour, 6-hour, and 24-hour components) and 2018 (1-hour components) sepsis management bundles were too discordant from SEP-1 (25, 26). Finally, in addition to our systematic review, we also manually reviewed all studies included in a 2019 systematic review on sepsis bundles; none of these studies were sufficiently similar to SEP-1 to be included (Supplement Section D, available at Annals.org) (19).

Each article’s title and abstract were independently screened for inclusion by 2 authors (J.S.F. and G.W.), and disagreements were resolved by discussion between them. Next, these 2 authors performed independent, full-text review of articles that met initial screening criteria to obtain consensus on final inclusion. When uncertainty remained, a third author (A.M.) served as a tiebreaker.

Data Extraction and Quality Assessment

Three authors (J.S.F., J.C.M., and M.H.) independently extracted data using a standardized tool (Supplement Section E, available at Annals.org), and 2 authors (J.S.F. and G.W.) independently checked the data for accuracy. Extracted data included information related to study design, baseline information about the study population (age, proportions of genders, proportions of comorbidities), raw proportions of primary outcomes (when available), and unadjusted and adjusted measures of association for primary outcomes (when available, stratified by treatment group). Because the SEP-1 metric has undergone various revisions since 2015, we used SEP-1 compliance reported by each individual study rather than predefined criteria.

Two authors (J.S.F. and G.W.) independently assessed each study for potential confounders and risk of bias, and disagreements were resolved by consensus. All included studies were observational, and we used the Newcastle-Ottawa Scale (NOS) for risk-of-bias assessment (27, 28). The NOS has separate instruments for cohort and case–control studies, but both assess 3 domains of bias (selection, comparability, and ascertainment of outcome or exposure), awarding “stars” within each domain when risk of bias is assessed to be low (Supplement Section F, available at Annals.org) (27, 28). We modeled our NOS criteria on previously conducted systematic reviews that assessed the evidence underpinning sepsis bundles (19, 21). When assessing comparability bias, we required that a study adjust mortality estimates for severity of illness, comorbidities, age, and site of infection. In addition to 28-day mortality, we considered in-hospital mortality to be an acceptable mortality outcome as this is a widely used metric to assess the quality of sepsis care. If studies used codes from the ninth or 10th revision of the International Classification of Diseases but did not perform manual validation of diagnoses, we determined that the representativeness of the study sample was “unknown,” as using explicit sepsis codes alone has been found to have low sensitivity in identifying cases of sepsis compared with clinical criteria (29–31). Given that before–after cohort studies are particularly susceptible to confounding from secular trends, we required that this study design adjust for seasonality as well as existing mortality trends in the preimplementation period to be considered to have low risk of bias (32). We assessed outcome bias according to whether the study assessed mortality blindly or from record linkage and also required authors to comment on data missingness or adequacy of follow-up; when such data were not explicitly stated, we classified this domain as “no description.” If a study had high risk of bias, unknown risk of bias, or “no description” for any NOS domain, it was judged as “not low risk” for bias.

The primary outcome was mortality. There was considerable heterogeneity with regard to which mortality outcome was reported (for example, in-hospital mortality or 28-day mortality), which measure of association was reported (for example, odds ratio [OR], adjusted risk reduction), and whether the measure of association was unadjusted or adjusted. Thus, we present the outcome and measure of associations reported by each study, without standardization. When unadjusted differences in mortality were the only measure of association reported, we manually calculated the corresponding 95% CI to help define the precision of the estimate of effect. Due to high methodologic heterogeneity of the studies, we did not perform a meta-analysis because of concerns that the result could be misleading by creating a biased, pooled estimate of effect (20, 33).

Data Synthesis and Analysis

We determined the level of evidence for the SEP-1 bundle by author consensus (J.S.F., G.W., and A.M.) using GRADE (Grading of Recommendations Assessment, Development and Evaluation) criteria and the National Quality Forum’s system for evaluating process measures (Supplement Sections G and H, available at Annals.org) (34–37). We chose the GRADE criteria because this system is endorsed by CMS and is most closely aligned with the Agency for Healthcare Research and Quality recommendations (35–37). We used a modified GRADE approach optimized for when a single estimate of effect (meta-analysis) is not calculated (38). For the primary outcome (mortality), measures of association and their 95% CIs are presented.

Role of the Funding Source

This study received no funding.

Results

The literature search identified 6922 total (4403 unique) references. Forty-six references passed screening and underwent full-text review. Of these, 17 met inclusion criteria (Figure 1). No randomized controlled trials (RCTs) were identified; all studies were observational (11 concurrent cohort studies [23, 39–48], 1 case–control study [49], and 5 before–after cohort studies [50–54]) (Table 1). Thirteen studies assessed both the 3-hour and 6-hour SEP-1 bundles, and 4 studies assessed only the 3-hour SEP-1 bundle. There was substantial heterogeneity in terms of study sample size (range, 158 to 252 599), number of study centers (range, 1 to 3241), the sepsis definition used (CMS/Sepsis-2, Sepsis-3 [Supplement Section I, available at Annals.org]), and the patient population studied (including Medicare beneficiaries and patients with septic shock) (Tables 1 and 2) (55, 56). There was also considerable heterogeneity with regard to which mortality outcome was reported (in-hospital mortality, 28-day in-hospital mortality), the measure of association reported (OR, mortality difference), and adjustment for potential confounding.

Figure 1.

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram.

Table 1.

Results of Studies Assessing the Association Between SEP-1 Compliance or SEP-1 Implementation and Mortality

Study, Year (Reference)	Study Dates	SEP-1 Bundle Studied	Primary Outcome Reported	Raw Mortality, % (n/N)		Unadjusted Measure of Association (95% CI)	Adjusted Measure of Association (95% CI)
				Intervention Group*	Control Group*
Concurrent cohort studies and cross-sectional studies (SEP-1 compliance)
Baghdadi et al, 2020 (39)	October 2014–October 2016	3-h/6-h†	In-hospital mortality	16.6 (321/1928)	19.5 (895/4476)	Mortality difference‡: 2.9% (−1.9% to 7.7%)	Rebalanced mortality difference§: 0.1% (−2.4% to 2.6%)
Bauer et al, 2020 (40)	October 2015–March 2019	3-h	In-hospital mortality	5.9 (90/1526)	11.7 (266/2273)	Mortality difference‡: 5.8% (4.0% to 7.6%)||	aOR for interaction (severe sepsis)¶: 0.44 (0.32 to 0.61)||
							aOR for interaction (septic shock)¶: 0.93 (0.60 to 1.46)
Chen et al, 2023 (41)	September 2017–February 2018	3-h	In-hospital mortality	35.7 (153/428)	24.8 (136/549)	uOR: 1.61 (1.21 to 2.10)	aOR**: 0.72 (0.47 to 1.10)
Choi et al, 2021 (42)	March 2008–December 2017	3-h/6-h†	28-d mortality	37.1 (211/569)	53.6 (218/407)	uOR: 0.51 (0.40 to 0.66)||	aOR††: 0.61 (0.40 to 0.91)||
Dierkes et al, 2022 (43)	January 2017–December 2017	3-h/6-h	30-d mortality	NR‡‡	NR‡‡	uOR: 0.98 (0.97 to 1.00)	aOR§§: 0.98 (0.97 to 1.00)
Green et al, 2019 (23)	January 2010–December 2014	3-h	In-hospital mortality	14.5 (55/380)	10.4 (31/298)	Mortality difference‡: 4.1% (−0.9% to 13.9%)	NA
Lawrence et al, 2024 (44)	January 2018–December 2019	3-h/6-h	In-hospital mortality	7.9 (36/457)	7.5 (9/120)	Mortality difference‡: 0.4% (−4.9% to 5.7%)	aOR||||: 1.32 (0.58 to 3.22)
Rhee et al, 2018 (45)	October 2015–September 2017	3-h/6-h	In-hospital mortality	11 (31/281)	18.4 (105/570)	uOR¶¶: 0.55 (0.36 to 0.84)	aOR***: 0.74 (0.46 to 1.18)
Sloan et al, 2022 (46)	January 2019–June 2020	3-h/6-h	In-hospital mortality	14.8 (29/195)	27.7 (67/242	Mortality difference (all): 12.8% (5.3% to 20.3%)||	NA
						Mortality difference (severe sepsis)‡: 0.54% (−0.6% to 13.8%)
						Mortality difference (septic shock): 16.5% (3.3% to 29.7%)||
Townsend et al, 2022 (47)	October 2015–March 2017	3-h/6-h	30-d mortality	21.7 (30 444/140 504)	30.3 (58 554/193 266)	Mortality difference: 8.6% (8.0% to 9.2%)‡||	Standard match, ARR†††: 5.67% (5.33% to 6.00%)||
							Stringent match, ARR†††: 4.06% (3.70% to 4.41%)||
							Septic shock 6-h bundle conditional aOR: 1.05 (0.96 to 1.15)
Utariani et al, 2023 (48)	January 2019–December 2019	3-h/6-h	In-hospital mortality	42.2 (30/71)	89.2 (83/93)	Mortality difference: 47.0% (33.9% to 60.1%)‡||	NA
Case–control studies (SEP-1 compliance)
August et al, 2022 (49)	July 2017–December 2019	3-h	28-d in-hospital mortality	10.2 (33/325)	9.8 (32/325)	uOR: 1.035 (0.620 to 1.728)	aOR‡‡‡: 1.04 (0.72 to 1.50)
Before–after cohort studies (SEP-1 implementation)
Afshar et al, 2019 (50)	March 2014–July 2017	3-h/6-h	In-hospital mortality	3.9 (306/7770)	5.1 (311/6107)	Overall raw mortality difference between periods‡: 1.2% (0.5% to 1.9%)||	aOR§§§: 0.70 (0.57 to 0.86)||
Anderson et al, 2022 (51)	Pre: October 2014–September 2015	3-h/6-h	In-hospital mortality or discharge to hospice	6.8 (1225/18 044)	7.1 (920/12 970)	Overall raw mortality difference between periods‡: 0.3% (−1.8% to 2.5%)	NA
	Post: November 2015–October 2016
Barbash et al, 2021 (52)	Pre: January 2013–September 2015	3-h/6-h	In-hospital mortality	4.2 (947/22 759)	4.9 (1419/29 051)	Overall raw mortality difference between periods‡: 0.7% (−2.5% to 3.9%)	Expected mortality change between pre- and post-SEP-1 periods: 3.9%
	Post: January 2016–December 2017						Observed mortality change between pre- and post-SEP-1 periods: 4.0%
							Difference in expected vs. observed mortality after SEP-1 implementation-||||||: 0.1% (−0.9% to 1.1%)
Ramsdell et al, 2017 (53)	Pre: April 2015–September 2015	3-h/6-h	In-hospital mortality	14.5 (16/110)	27.1 (13/48)	Overall raw mortality difference between periods: 12.6% (−1.6% to 26.8%)‡	NA
	Post: October 2015–February 2016
Rhee et al, 2021 (54)	Pre: October 2013–September 2015	3-h/6-h	Short-term mortality ¶¶¶	20.4 (11 449/56 129)	20.3 (12 461/61 381)	uOR for overall mortality between periods: 1.01 (0.98 to 1.04)	aOR for level change¶¶¶: 0.94 (0.7 to 1.29)
	Post: January 2016–December 2017						aOR for trend change¶¶¶: 1.00 (0.97 to 1.04)

aOR = adjusted odds ratio; ARR = absolute risk reduction; NA = not applicable; NR = not reported; Q4 = fourth quarter; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle; SOFA = Sequential Organ Failure Assessment; uOR = unadjusted odds ratio.

^*For longitudinal cohort studies and case–control studies, the intervention group represents those who received the full SEP-1 bundle, and the control group represents those who did not receive the full bundle. For before–after cohort studies, the intervention group represents patients in the postimplementation period, and the control group represents patients in the preimplementation period.

^†This 6-hour bundle excluded volume reassessment.

^‡We calculated the 95% CI for this unadjusted difference in mortality using data from the study; the CI was not reported in the study.

^§Propensity-matched multiple probit regression with adjustment for comorbidities and severity of illness was used to rebalance groups and calculate coefficients. Regression output was then used to calculate average mortality using regression coefficients. The adjusted mortality difference is the difference between these 2 rebalanced groups.

^||The outcome showed significant mortality benefit.

^¶Multiple logistic regression with interaction term for SEP-1 compliance and presence/absence of septic shock, adjusted for age, sex, race, facility, and comorbidities.

^**Multiple logistic regression model adjusted for age, sex, race, APACHE III (Acute Physiology, Age, Chronic Health Evaluation) score, SOFA score, highest lactate level, hours in the hospital before onset of sepsis-induced hypotension, location of hypotension onset, intensive care unit types, and secondary contributors to shock (intoxication, history of immunosuppression, cancer, cumulative fluid, use of any vasopressor, use of mechanical ventilation in the 24 hours after onset of sepsis-induced hypotension).

^††Multiple logistic regression model adjusted for age, sex, mechanical ventilation, positive blood culture results, various laboratory studies, various interventions, and SOFA score but not comorbidities.

^‡‡Unable to obtain data from author group or patient-level data not available.

^§§Multiple logistic regression adjusted for hospital-level characteristics and patient-level characteristics (sex, Elixhauser comorbidities, severity of illness).

^||||Multiple logistic regression model adjusted for age, sex, race, ethnicity, septic shock, community- vs. hospital-acquired sepsis, source of infection, time zero, and time of arrival but not comorbidities.

^¶¶The authors reported the aOR in terms of SEP-1 failure (1.8 [95% CI, 1.2 to 2.8]). We used the inverse of the point estimate and 95% CI bounds to report in terms of SEP-1 compliance.

^***Multiple logistic regression adjusted for age, SEP-1 compliance, Elixhauser score, severity of illness, and other variables. The authors reported the aOR in terms of SEP-1 failure (1.36 [95% CI, 0.85 to 2.18]). We used the inverse of the point estimate and 95% CI bounds to report in terms of SEP-1 compliance.

^†††Propensity-matched analysis using hierarchical generalized linear model, adjusted for comorbidities, infection source, severity of illness, and other variables.

^‡‡‡Propensity-matched multiple regression adjusted for age, severity of illness, and some but not all important comorbidities.

^§§§Multiple logistic regression model adjusted for age, sex, race, insurance status, Elixhauser comorbidity score, service type, calendar month, and SOFA score at admission.

^||||||Interrupted time-series analysis using multiple logistic regression models for mortality and adjusted for patient factors (age, Elixhauser comorbidities, severity of illness, source of infection), seasonality, and hospital fixed effects was used to calculate adjusted mortality in Q4 2017 with SEP-1 implementation (using trend from postimplementation period) and expected mortality in Q4 2017 if SEP-1 had not been implemented (using trend for pre–SEP-1 period). The difference between expected mortality and observed mortality was not statistically significant.

^¶¶¶Interrupted time-series analysis using multiple logistic regression models adjusted for baseline characteristics, presence of septic shock, age but not comorbidities, and in-hospital death or discharge to hospice.

Table 2.

Characteristics of Included Studies

Study, Year (Reference)	Sepsis Type	Sepsis Definition	Total Patients, n	Study Country	Study Sites, n	Location of Sepsis Diagnosis	Outcomes Studied
Concurrent cohort studies and cross-sectional studies (SEP-1 compliance)
Baghdadi et al, 2020 (39)	Severe sepsis/septic shock	Sepsis-3	6404	United States	4	ED, wards, ICU	In-hospital mortality, vasopressor support
Bauer et al, 2020 (40)	Sepsis/severe sepsis/septic shock	CMS/Sepsis-2 ICD-10 codes*	3799	United States	12	ED, wards, ICU	In-hospital mortality, hospital LOS, 30-d readmission
Chen et al, 2023 (41)	Sepsis with hypotension†	Sepsis-3	977	United States	34	ED	In-hospital mortality, lactate level, SOFA score, mechanical ventilation, renal replacement therapy, hospital LOS, ICU LOS, ventilation-free days, vasopressor-free days
Choi et al, 2021 (42)	Septic shock	Sepsis-3	976	South Korea	1	All	28-d mortality, in-hospital mortality, ICU admission
Dierkes et al, 2022 (43)	Sepsis/severe sepsis/septic shock	Sepsis diagnosis at admission‡	252 599	United States	537	ED	30-/60-d mortality, 7-/30-/60-d readmissions, hospital LOS, ICU admission
Green et al, 2019 (23)§	Surgical sepsis||	Modified Sepsis-2||	677	South Africa	1	ED, wards, ICU	In-hospital mortality, ICU admission, hospital LOS
Lawrence et al, 2024 (44)	Severe sepsis/septic shock	Sepsis-3	577	United States	1	ED, wards, ICU	In-hospital mortality
Rhee et al, 2018 (45)	Severe sepsis/septic shock	CMS/Sepsis-2 ICD-10 codes*	851	United States	7	ED, wards, ICU	In-hospital mortality
Sloan et al, 2022 (46)	Severe sepsis/septic shock	CMS/Sepsis-2 ICD-10 codes‡	437	United States	1	ED, wards, ICU	In-hospital mortality
Townsend et al, 2022 (47)	Severe sepsis/septic shock	CMS/Sepsis-2 ICD-10 codes*	245 740	United States	3241	ED, wards, ICU	30-d mortality, hospital LOS
Utariani et al, 2023 (48)	Sepsis/severe sepsis/septic shock	Sepsis-3	164	Indonesia	1	ED, ICU	In-hospital mortality
Case–control studies (SEP-1 compliance)
August et al, 2022 (49)	Severe sepsis	CMS/Sepsis-2 ICD-10 codes*	650	United States	1	ED, wards, ICU	28-d in-hospital mortality, individual 3-h bundle element compliance, progression to septic shock
Before–after cohort studies (SEP-1 implementation)
Afshar et al, 2019 (50)	Suspected sepsis¶	Suspected infection, CMS/Sepsis-2, Sepsis-3¶	13 877	United States	1	ED, wards, ICU	In-hospital mortality, 28-d hospital-free days, 28-d ICU-free days, time to first antibiotic, ICU utilization, inflation-adjusted hospital charges, incremental costeffectiveness ratio
Anderson et al, 2022 (51)	Suspected sepsis**	Suspected sepsis**	31 014††	United States	10††	ED, wards, ICU	In-hospital mortality, days of antimicrobial therapy
Barbash et al, 2021 (52)‡‡	Severe sepsis/septic shock	Sepsis-3	51 810	United States	11	ED	In-hospital mortality, admission to ICU, discharge to home, antibiotic adherence, lactate adherence, fluid adherence
Ramsdell et al, 2017 (53)	Severe sepsis/septic shock	CMS/Sepsis-2 ICD-9 (pre) and ICD-10 (post)*	158	United States	1	ED, wards, ICU	In-hospital mortality, bundle compliance, hospital LOS, ICU LOS
Rhee et al, 2021 (54)‡‡	Severe sepsis/septic shock	CMS/Sepsis-2 ICD-10 codes*	117 510	United States	114	ED, wards, ICU	Short-term mortality (in-hospital death or discharge to hospice), lactate measurements, anti-MRSA or antipseudomonal antibiotic use

CMS = Centers for Medicare & Medicaid Services; ED = emergency department; ICD-9 = International Classification of Diseases, Ninth Revision; ICD-10 = International Classification of Diseases, 10th Revision; ICU = intensive care unit; LOS = length of stay; MRSA = methicillin-resistant Staphylococcus aureus; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle; SOFA = Sequential Organ Failure Assessment.

^*Manual validation of sepsis diagnoses was performed.

^†Defined by mean arterial pressure ≤65 mm Hg, systolic blood pressure ≤90 mm Hg, or requirement for vasopressor.

^‡No manual validation of sepsis diagnoses reported.

^§Included patients aged ≥16 years (median age, 45 years).

^||Defined as documented surgical source of infection and ≥2 of the following: fever or hypothermia, heart rate >90 beats/min or >2 SDs above the normal value for age, respiratory rate >20 breaths/min, altered mental status.

^¶The study assessed multiple sepsis definitions. Suspected infection was defined as antibiotics delivered <24 hours before or <72 hours after body fluid collection.

^**Suspected sepsis defined as patient with ≥1 blood culture who was still receiving broad-spectrum antibacterial agents 48–72 hours after blood culture collection.

^††Subanalysis that included patients with suspected sepsis from 10 sites.

^‡‡Interrupted time-series analysis.

Effect of SEP-1 Compliance on Mortality

Overall Effect

Eleven concurrent cohort studies and 1 case–control study assessed the effect of SEP-1 compliance on mortality (23, 39–48). Of these, 5 studies showed a significant mortality benefit in at least 1 subgroup (40, 42, 46–48), and 7 found no association between SEP-1 compliance and mortality (23, 39, 41, 43–45, 49).

Studies Showing Mortality Benefit With SEP-1 Compliance

Sloan and colleagues conducted a retrospective, single-center, concurrent cohort study (n = 437) in which an unadjusted analysis showed that mortality was lower in the group that was compliant with the 3-hour and 6-hour bundles compared with the noncompliant group; an analysis stratified by illness severity showed that the survival benefit was seen only in the septic shock group (46). Choi and colleagues performed a single-center study of patients with septic shock (n = 976) and found that SEP-1 compliance was associated with mortality (OR, 0.61 [95% CI, 0.40 to 0.91]) (42). Bauer and colleagues performed a multicenter (n = 12) study of patients with sepsis (n = 3799) that assessed for a statistical interaction between SEP-1 compliance and the presence of septic shock using a multivariate model. SEP-1 compliance in patients without septic shock was associated with a lower odds of death (adjusted OR, 0.44 [CI, 0.32 to 0.61]), but compliance in patients with septic shock was not associated with mortality (adjusted OR, 0.93 [CI, 0.60 to 1.46]) (40). Townsend and colleagues performed a retrospective multicenter study of Medicare beneficiaries (n = 245 740) that used a propensity-matched, adjusted analysis that reported an adjusted absolute risk reduction of 5.7% (CI, 5.3% to 6.0%) among patients who were compliant with the SEP-1 bundle (47). Utariani and colleagues performed a single-center study (n = 164) that found that mortality was lower in patients receiving the full bundle (48).

Studies Showing No Mortality Benefit With SEP-1 Compliance

Chen and colleagues performed a multicenter (n = 34) study of patients with sepsis-induced hypotension (n = 977) and found no difference in mortality between those who were compliant and those who were noncompliant with the 3-hour SEP-1 bundle (41). Green and colleagues performed a single-center study of patients with surgical sources of sepsis (n = 677) and found no difference in mortality between those who were compliant and those who were noncompliant with the 3-hour SEP-1 bundle (23). Baghdadi and colleagues performed a multicenter (n = 4) study of patients with sepsis (n = 6404) that used a propensity-matched, adjusted analysis and found no association between 3-hour and 6-hour SEP-1 compliance and mortality among the overall population and among populations with hospital-acquired and community-acquired sepsis (39). Dierkes and colleagues performed a multicenter (n = 537) study of Medicare beneficiaries (n = 252 599) that adjusted for system-level factors, such as nurse staffing, and found no association between 30-day or 60-day mortality and hospital-level 3-hour or 6-hour SEP-1 compliance (43). Studies that assessed all patients with sepsis without age restrictions found no difference in mortality (39, 44, 45).

Effect of SEP-1 Implementation on Mortality

Five before–after cohort studies reported mortality outcomes after SEP-1 implementation. One study showed a survival benefit and 4 showed no association between SEP-1 implementation and mortality (50–54). Afshar and colleagues performed a single-center pre–post analysis showing that mortality decreased by 1.2% (CI, 0.5% to 1.9%) after the SEP-1 mandate was implemented in parallel with a sepsis quality improvement (QI) program (50). Ramsdell and colleagues performed a single-center, unadjusted before–after study of patients with sepsis (n = 158) that showed no significant decrease in overall mortality after SEP-1 implementation (53). Barbash and colleagues performed a multicenter (n = 11) study of patients with sepsis (n = 51 810) with an interrupted time-series (ITS) analysis that controlled for multiple patient-level characteristics, hospital-level fixed effects, and seasonality and found no difference between pre–SEP-1 and post–SEP-1 implementation mortality trends (expected mortality difference, 0.1% [CI, −0.9% to 1.1%]) (52). In another multicenter (n = 114) study of patients with sepsis (n = 117 510), Rhee and colleagues performed an ITS analysis and found no significant immediate change in the adjusted odds of death as well as no change in the odds of death over time after SEP-1 implementation (54). Anderson and colleagues performed a multicenter (n = 10) before–after unadjusted subanalysis of patients with suspected sepsis (n = 31 014) and found no difference in mortality between pre–SEP-1 and post–SEP-1 implementation periods (51).

Risk of Bias and Level of Evidence

No study was assessed as having low risk of bias (Figures 2 to 4). Studies were given a “not low risk” designation most frequently for failing to adjust for important confounders or failing to report data missingness or loss to follow-up. We scored the overall level of evidence as “low” according to the GRADE and National Quality Forum criteria (Tables 3 and 4).

Figure 2. Newcastle-Ottawa quality assessment for longitudinal cohort studies.

ICD-10 = International Classification of Diseases, 10th Revision; ND = no description.

* Study meets quality assessment for domain category.

† Included patients with ICD-10 codes for severe sepsis and septic shock with no manual validation of ICD-10 codes reported.

‡ Included only patients with sepsis-induced hypotension.

§ Adjusted for age and some comorbidities but not site of infection.

|| Included only patients with septic shock.

¶ Adjusted for age but not comorbidities or site of infection.

** Included patients with a diagnosis of sepsis at admission but did not report manual validation of diagnosis or how diagnosis was identified (e.g., ICD-10 codes).

†† Included only Medicare beneficiaries.

‡‡ Included only surgical sources of sepsis.

§§ Adjusted for age and site of infection but not comorbidities.

|||| Measured mortality at 48 hours.

Figure 4. Newcastle-Ottawa quality assessment for case–control studies..

SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle..

* Study meets quality assessment for domain category..

† Includes patients with severe sepsis only..

‡ Adjusted for age and some comorbidities but not site of infection.

Table 3.

Assessment of Level of Evidence by GRADE Criteria

GRADE Domain	Judgment	Assessment of Domain (Effect on Evidence Level)
Limitations (“risk of bias”)	No studies were considered to have low risk of bias according to the Newcastle-Ottawa Scale. Most studies were downgraded for failing to adjust for important confounders or failing to comment on data missingness.	Borderline (no change in evidence level)
Indirectness	Twelve studies (23, 39–48) provide direct evidence (SEP-1 compliance) and 5 studies provide indirect evidence (SEP-1 implementation) on the clinical question at hand. However, 2 indirect studies (52, 54) used acceptable methods for assessing a system-wide intervention.	Not serious (no change in evidence level)
Imprecision	A total of 712 626 patients were included across all studies. There was a mix of small and large studies, with larger studies generally reporting effects with tighter confidence intervals.	Not serious (no change in evidence level)
Inconsistency	The direction and magnitude of effect varied across studies. Six studies (40, 42, 46–48, 50) showed benefit and 11 studies (39, 41–45, 49, 51–54) showed no benefit with regard to mortality.	Borderline (no change in evidence level)
Publication bias	We assessed publication bias to be unlikely given that both positive and negative studies were published and we performed a comprehensive search of the literature.	Not serious
Factors that can increase certainty of evidence:	A single observational study (47) showed a large positive effect with adequate precision. We did not believe that this was sufficient to increase the certainty of the evidence. A dose-response gradient was not relevant to our research question. We found no evidence for confounders masking benefit.	Not substantial (no change in evidence level)
Large effect
Dose-response gradient
Confounders masking benefit
Overall GRADE assessment	Low certainty of the evidence	–

GRADE = Grading of Recommendations Assessment, Development and Evaluation; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.

Table 4.

Assessment of Level of Evidence per the NQF Criteria

Quantity of Body of Evidence	Quality of Body of Evidence	Consistency of Results	Overall NQF Assessment
There were 17 observational studies that assessed the association between mortality and SEP-1 compliance (23, 39–49) or SEP-1 implementation (50–54).	There was a single large, multicenter, observational study with adequate control for confounders and precise estimates of effect that showed benefit. However, this study did not have low risk of bias and included only Medicare beneficiaries (47). One small single-center study showing benefit had high risk of bias, did not adjust for confounders, and had imprecise effect estimates (46). A subanalysis of this study showed that the benefit was limited to patients with septic shock. One study assessed for an interaction between SEP-1 compliance and the presence of septic shock and found that SEP-1 compliance was associated with improved mortality among patients without septic shock but not among those with septic shock (39). A single-center study of patients with septic shock showed a mortality benefit, but this study did not have low risk of bias (42). A single-center before–after study showed that mortality decreased after SEP-1 implementation; however, SEP-1 was implemented alongside a robust QI program, making it difficult to differentiate the effect of SEP-1 implementation from the effect of the QI program. Furthermore, this study did not adjust for existing mortality trends present in the preimplementation period.	Five studies (40, 42, 46–48) showed potential mortality benefits with SEP-1 compliance in various sepsis subpopulations, and 7 (23, 39, 41, 43–45) showed no benefit. One study (50) assessing SEP-1 implementation showed a mortality benefit, and 4 studies (51–54) showed no benefit. Overall, estimates of effect were not consistent in magnitude or direction.	Low certainty of the evidence

NQF = National Quality Forum; QI = quality improvement; SEP-1 = Severe Sepsis and Septic Shock Early Management Bundle.

Discussion

Our systematic review is, to our knowledge, the first to evaluate the level of evidence for both the effect of SEP-1 compliance (patient level) and the effect of SEP-1 implementation (system level) on sepsis mortality. We found 12 studies that assessed the relationship between SEP-1 compliance and sepsis mortality; 5 showed benefit, and 7 showed no benefit. Five studies assessed the relationship between SEP-1 implementation and sepsis mortality, and only 1 showed significant benefit. No studies had low risk of bias. We found no moderate- or high-level evidence that compliance with or implementation of the entire 3- or 6-hour SEP-1 bundle is associated with sepsis mortality. Our findings are important because SEP-1 compliance is now linked to financial incentives via CMS’s Hospital VBP Program.

Accounting for factors associated with the primary study exposure is critical in conducting well-controlled observational studies because failing to do so leads to confounding by indication (57). For example, patients with sepsis who present with classic symptoms may be more likely than patients with vague or subtle presentations to receive an intervention (such as the SEP-1 bundle). Previous research has found that patients with sepsis with vague symptoms have higher mortality, even when delays in care are accounted for (58). Rhee and colleagues specifically accounted for this by adjusting for “vague symptoms,” whereas all studies showing a mortality benefit did not (45, 46, 59). Clinicians often strongly disagree about a patient’s likelihood of having sepsis, especially in cases with vague symptoms (60, 61). Thus, it is unsurprising that patients who do not have all elements of the SEP-1 bundle completed are more likely to present with vague symptoms, given that diagnostic certainty is likely to influence whether a patient receives the complete SEP-1 bundle (45). Townsend and colleagues argue that adjusting for vague symptoms is “unnecessary because SEP-1 abstractors set time zero only after physicians document suspicion of infection” (47). However, abstractors disagree on “time zero” in almost 2 out of 3 sepsis cases that are reviewed, demonstrating a wide discrepancy between written SEP-1 policy and real-world implementation and highlighting the need to control for presentations that induce diagnostic uncertainty (62).

Townsend and colleagues (47) provided the most compelling study to date in support of SEP-1. Their study was methodologically robust, showed mortality benefit among Medicare beneficiaries, and used a well-controlled propensity-matched primary analysis as well as various sophisticated sensitivity analyses with similar results. Of note, there were substantial differences in compliance among individual bundle components between the severe sepsis and septic shock groups. Despite the use of propensity score matching, patients who received noncompliant care were much more likely to have septic shock, including persistent hypotension and lactate levels of 4 mmol/L or higher. This highlights how complying with all components of an all-or-nothing bundle becomes increasingly difficult in complex patients, and the added components in the septic shock bundle may convey a higher probability of noncompliance. Thus, there is a risk that bundle noncompliance may merely serve as an inconspicuous proxy for severity of illness in these patients, which portends an inferior outcome (63). In a subgroup restricted to patients with septic shock, mortality rates were similar between those who were compliant with the 6-hour bundle and those who were not (38% vs. 35%), which suggests that the primary analysis had residual confounding. In addition, the study by Townsend and colleagues (47) did not report the proportions of patients with community- versus hospital-acquired sepsis and did not match groups by this variable or adjust for this variable as part of its models. This is important because mortality is known to be as much as 2-fold higher in patients with hospital-acquired sepsis (64–66). Moreover, Medicare beneficiaries may not be representative of the overall sepsis population, which limits the generalizability of these findings. Similar to the study by Townsend and colleagues, Baghdadi and colleagues (39) performed a well-executed propensity-matched analysis using a large multicenter cohort of adults aged 18 years or older, but they found no association between bundle compliance and mortality. They also performed a stratified analysis comparing patients with community-acquired versus hospital-acquired sepsis and found no difference in bundle compliance and mortality.

Before–after studies that use ITS analyses are important for evaluating the real-world effect of SEP-1 implementation because patient mix tends to be stable over time and this design can account for secular trends, ensuring that the observed effect is not merely a continuation of a trend that existed before implementation (67, 68). Notably, the one before–after study that showed mortality benefit adjusted for seasonality but did not account for mortality trends in the preimplementation period (50). Moreover, SEP-1 was implemented alongside a robust sepsis QI program, making it difficult to differentiate the effect of SEP-1 implementation from the effect of the QI program (50). Two of the 4 before–after studies we included used an ITS analysis, and both found no difference in mortality after SEP-1 implementation (52, 54). Moreover, these 2 studies used objective clinical criteria rather than sepsis diagnoses to identify their populations, making their results less susceptible to ascertainment bias.

Several studies assessed important secondary outcomes. Two concurrent cohort studies showed an association between SEP-1 compliance and a decrease in hospital length of stay (43, 47), and 1 study showed no association between bundle compliance and intensive care unit (ICU) length of stay (69). Two studies found no association between SEP-1 compliance and vasopressor days (39, 52). One study found no association between bundle compliance and 7-day, 30-day, or 60-day hospital readmissions, and 1 study showed that bundle compliance was associated with lower 30-day readmissions (40, 43). In studies with before–after cohort designs, SEP-1 implementation was associated with increased adherence to process measures, including ordering lactate tests, administering broad-spectrum antibiotics, and overall SEP-1 compliance (52–54). SEP-1 implementation was associated with no change in hospital or ICU length of stay (53) and actually led to an increase in ICU admissions in 1 study (52).

This systematic review was designed to capture studies that assessed the effect of SEP-1 or similar bundles on mortality. Notably, studies that included data from the NYS Sepsis Care Improvement Initiative were omitted because the NYS bundle prescribes a 30-cc/kg bolus within 6 hours (for patients with hypotension or lactate level ≥4 mmol/L), whereas SEP-1 prescribes the same bolus within 3 hours (70). One such study from NYS performed an intergroup analysis of patients who received the complete bundle within 12 hours and found that mortality increased for each hour of delay in bundle completion. However, this study did not include a true control group, as patients who did not receive the full bundle were excluded (24).

A systematic review and meta-analysis published in 2019 identified 17 studies that assessed the association between sepsis bundle use and patient survival (19). Nine of the 17 studies showed a statistically significant improvement in survival (19). All included studies were observational, there was substantial heterogeneity as well as significant variability in fluid and antibiotic bundle components among them, and none had low risk of bias. Moreover, of the 9 studies showing benefit, 3 did not adjust for any confounders, and the other 4 partially adjusted for age, comorbidities, acuity of illness, or infection site (19). We reviewed each study included in this systematic review and found that none of the bundles that they studied were similar enough to SEP-1 to make a fair comparison; thus, we excluded all of these studies from our review. This review found no RCTs that studied sepsis bundles. Of note, a recent stepped-wedge cluster RCT conducted in 23 emergency departments in France and Spain found no survival benefit between implementation of a 1-hour sepsis bundle and usual care (71).

Protocolized sepsis bundles such as SEP-1, when applied at the population level, are operationally efficient and have been shown to improve sepsis care delivery. Of the individual sepsis bundle components, timely antibiotic administration has the most robust effect on mortality; the effect of other components (such as fluid resuscitation) on mortality is less definitive (19, 72). However, the overall effect of sepsis bundle compliance on mortality remains uncertain, as most of the existing evidence is observational and disentangling confounding from the true effect of SEP-1 has proved difficult. Sepsis is a heterogeneous condition, and evidence exists that patients may respond differently to treatment on the basis of both host and pathogen factors (2, 73). Some argue that SEP-1’s rigidity as an all-or-nothing mandate may disempower clinicians from providing personalized care based on an individual patient’s clinical picture and physiology (74, 75). The potential benefits of mandated standardization of sepsis treatment must be weighed against the potential harm to patients and the greater health care system. For example, pressure to comply with SEP-1’s time requirements in patients with noninfectious sepsis mimics, such as methamphetamine intoxication or pancreatitis, may lead to increased broad-spectrum antimicrobial use, which may increase risk for adverse events, such as acute kidney injury, delirium, or contracting a nosocomial or multidrug-resistant organism (18, 76–79). Finally, sepsis bundle overuse may lead to increased health system resource utilization and antimicrobial resistance (10).

Our conclusions are limited by the underlying quality of the available studies, as all were observational. Most included studies were U.S.-based; thus, caution in extrapolating these results to other countries is advised. In addition, because we aimed to study SEP-1, we excluded studies that examined sepsis bundles that were not similar enough to SEP-1 to make a fair comparison. Because there was considerable methodologic heterogeneity among included studies, we did not perform a meta-analysis as the results could have been misleading (20). Because studies were conducted during different time frames, different iterations of SEP-1 were in effect during different studies, and clinician practice may have varied between institutions and evolved over time, making it difficult to standardize and compare studies (80).

Overall, we found no high- or moderate-level evidence to suggest that SEP-1 compliance was associated with improved mortality; however, there may be a signal of mortality benefit in certain populations, such as Medicare beneficiaries and patients with septic shock. Until higher-quality evidence supporting SEP-1 is available, the addition of SEP-1 into the Hospital VBP Program should be reconsidered.

Figure 3. Modified Newcastle-Ottawa quality assessment for before–after cohort studies.

ND = no description.

* Study meets quality assessment for domain category.

† Adjusted for age and comorbidities but not site of infection.

‡ Adjusted for seasonality but not preexisting trends in outcome in preimplementation period.

§ Sepsis was defined as patient with ≥1 blood culture who was still receiving broad-spectrum antibacterial agents 48–72 hours after blood culture collection.

|| Adjusted for age but not comorbidities or site of infection.