Driving blind: instituting SEP-1 without high quality outcomes data

Abstract

In 2015, the Centers for Medicare and Medicaid Services (CMS) instituted an all-or-none sepsis performance measure bundle (SEP-1) to promote high-quality, cost-effective care. Systematic reviews demonstrated only low-quality evidence supporting most of SEP-1’s interventions. CMS has removed some but not all of these unproven components. The current SEP-1 version requires patients with suspected sepsis have a lactate level, blood cultures, broad-spectrum antibiotics and, if hypotensive, a fixed 30 mL/kg fluid infusion within 3 hours, and a repeat lactate if initially elevated within 6 hours. Experts have continued to raise concerns that SEP-1 remains overly prescriptive, lacks a sound scientific basis and presents risks (overuse of antibiotics and inappropriate fluids not titrated to need). To incentivize compliance with SEP-1, CMS now publicly publishes how often hospitals complete all interventions in individual patients. However, compliance measured across hospitals (5 studies, 48–2,851 hospitals) or patients (three studies, 110–851 patients) has been low (approximately 50%) which is not surprising given SEP-1’s lack of scientific basis. The largest observational study (1,738 patients) reporting survival rates employing SEP-1 found they were not significantly improved with the measure (P=0.53) as did the next largest study (851 patients, adjusted survival odds ratio 1.36, 95% CI, 0.85 to 2.18). Two smaller observational studies (158 and 450 patients) reported SEP-1 improved unadjusted survival (P≤0.05) but were confounded either by baseline imbalances or by simultaneous introduction of a code sepsis protocol to improve compliance. Regardless, retrospective studies have well known biases related to non-randomized designs, uncontrolled data collection and failure to adjust for unrecognized influential variables. Such low-quality science should not be the basis for a national mandate compelling care for a rapidly lethal disease with a high mortality rate. Instead, SEP-1 should be based on high quality reproducible evidence from randomized controlled trials (RCT) demonstrating its benefit and thereby safety. Otherwise we risk not only doing harm but standardizing it.

Introduction

Sepsis afflicts around 1.7 million people annually in the United States and has an in-hospital mortality rate of 15.6% and estimated cost of over 20 billion USD (1,2). Most US hospitals participate in the federal government’s Medicare/Medicaid health insurance program administered by the Centers for Medicare and Medicaid Services (CMS). This program pays 63% of all inpatient medical costs making it the largest single payer in the US (1). CMS develops and institutes performance measures (PM) for illnesses having a major impact on the health of the US population with the goal of promoting quality and cost-effective care nationally. Given the significant health and financial burdens of sepsis, CMS instituted the Severe Sepsis and Septic Shock Performance Measure bundle (termed SEP-1) in 2015 (Table 1) (3).

Table 1. Components included in the 2015 and 2018 versions of SEP-1.

Sepsis category	Time requirement	2015 SEP-1 bundle	2018 SEP-1 bundle
Severe sepsis (diagnostic criteria)    • 2/4 SIRS criteria • One of the following     -End organ dysfunction* OR -Lactate ≥2 mmol/L	3-hour bundle	1. Lactate (initial)	1. Lactate (initial)
		2. Blood cultures prior to antibiotics	2. Blood cultures prior to antibiotics
		3. Antibiotic administration	3. Antibiotic administration
			4. 30 mL/kg IV fluid (if initial hypotension)
	6-hour bundle	1. Repeat lactate (if initial lactate elevated)	1. Repeat lactate (if initial lactate elevated)
Septic shock (diagnostic criteria)    • 2/4 SIRS criteria • One of the following     -Systolic BP <90 mmHg OR -Lactate ≥4 mmol/L	3-hour bundle	1. Lactate (initial)	1. Lactate (initial)
		2. Blood cultures prior to antibiotics	2. Blood cultures prior to antibiotics
		3. Antibiotic administration	3. Antibiotic administration
		4. 30 mL/kg IV fluid administration	4. 30 mL/kg IV fluid administration
	6-hour bundle	1. Repeat lactate (if initial lactate elevated)	1. Repeat lactate (if initial lactate elevated)
		2. Vasopressors (if SBP <90 after 30 mL/kg IVF)	2. Vasopressors (if SBP <90 after 30 mL/kg IVF)
		If either: SBP <90 mmHg after 30 mL/kg IVF OR initial lactate ≥4 mmol/L
		1. Perform a volume status or perfusion exam	1. Perform a volume status or perfusion exam (not otherwise specified)
		(a) Option 1: focused 5-point physical exam
		• Vital signs
		• Cardiopulmonary exam
		• Capillary refill test
		• Peripheral pulse exam
		• Skin exam
		(b) Option 2: choose two of the following
		• Central venous pressure measurement
		• Central venous O2 saturation measurement
		• Cardiac POCUS
		• Fluid administration or passive leg raise

*, end organ dysfunction as defined by CMS: lactate >2 mmol/L, INR > 1.5, aPTT >60 seconds, total bilirubin >2 mg/dL, creatinine >2 mg/dL, urine output <0.5 mL/kg/hour for 2 hours, respiratory failure requiring invasive or non-invasive ventilation, systolic BP <90 mmHg or mean arterial pressure <65 mmHg or systolic BP drop >40 mmHg from baseline. IVF, intravenous fluids; mmHg, millimeters of mercury; POCUS, point of care ultrasound; SBP, systolic blood pressure; SEP-1, severe sepsis and septic shock early management bundle; SIRS, systemic inflammation response syndrome.

CMS employs stepwise enforcement programs that progressively incentivize hospitals to comply with its PMs. Initially, to receive any reimbursement from CMS, US hospitals must participate in CMS’s Hospital Inpatient Quality Reporting Program (IQR) (4,5). Enrolled hospitals are only required to report their compliance data to not jeopardize CMS payment. The next step is for a PM to become part of the Hospital Compare program in which CMS publicly reports hospital’s compliance with a measure (6,7). This program is intended to stimulate a hospital to follow PMs based on its concerns that poor compliance data will cause consumers to spend health care dollars elsewhere. The final step CMS employs to promote PM compliance, is to include the measure in the Value-Based Purchasing program. Here, CMS bases hospital reimbursement directly on whether the measure was followed for patients (8,9). In 2018, using data collected during 2017, SEP-1 became part of CMS Hospital Compare program (6).

The fact that SEP-1’s reporting requirements and impact on reimbursement may be forcing hospital-based clinicians to change their practice for septic patients, has engendered strong opinions. Hospitals and health-care workers have reported that SEP-1 is much more complex and requires many more hours of valuable health worker time than most other approved PMs (10-12). Concerns have also been raised that SEP-1’s rigid requirements do not allow clinicians to adjust their practice sufficiently for septic patients, a group that differs greatly in underlying comorbidities and sepsis severity. In fact, sepsis experts have noted that SEP-1 interventions lack data to support benefit and when rigidly applied may harm some patients (9,12-17). Consistent with this latter concern, SEP-1 has been revised twice since it was originally proposed and then instituted, and several interventions have been removed. Unfortunately, SEP-1 continues to mandate care encompassing unproven components that may be ineffective or harmful to some patients (18). Yet for this unproven care, hospitals have had to expend valuable resources to comply with and report on the measure (14,19,20).

This review will examine the development of SEP-1, the measure’s potential risks in its current form, and initial data from the last four years regarding compliance with it and its possible effects on morbidity and mortality. Unfortunately, only uncontrolled and potentially biased observational data are presently available to assess these latter parameters. While compliance data may be measurable, SEP-1’s effects on morbidity and mortality can only be speculated about. Without reproducible high quality outcomes data from randomized controlled trials (RCT) comparing SEP-1’s individual components and overall combined requirements vs. clinician’s usual sepsis care, the safety and benefit of SEP-1 will remain unknown.

SEP-1 and the Surviving Sepsis Campaign sepsis bundles

Developers of the version of SEP-1 introduced in 2015 included several past presidents and members of the Society of Critical Care Medicine (SCCM) (21). The PM was closely patterned after the 2012 Surviving Sepsis Campaign’s (SSC) guideline sepsis bundle that some of these individuals were also closely involved with (21,22). Therefore, understanding the missteps that have been taken with the SSC bundles is helpful in understanding the problems that have occurred with SEP-1.

The SSC introduced two sepsis bundles in 2004, each directing a group of interventions be completed within specified time periods (23). These SSC bundles, like SEP-1 and other CMS endorsed PMs, were based on the idea that administering a group of evidence-based interventions together for an illness would result in significantly better outcomes than if these interventions were individually instituted (24). Unfortunately, the initial 2004 SSC sepsis bundles included several controversial interventions lacking strong and reproducible supporting evidence including activated protein C (APC), intense glucose control, corticosteroids and early goal directed therapy (EGDT). All of these interventions were proven ineffective or harmful in subsequent RCTs and over time removed from later SSC bundle versions (25-30).

No RCTs were ever done testing any of the SSC bundles either before or after their implementation, to provide high or even moderate quality evidence to support their use. Instead, ongoing promotion of the SSC bundles was based on low quality observational studies reporting that survival was improved in septic patients during a period following vs. before institution of a bundle or in groups of concurrent septic patients that did vs. did not complete a bundle (21). However, these observational bundle studies included many of the same interventions that were subsequently disproven in one or more RCTs. With the benefit of hindsight, the SSC bundles should have been supported with high-quality data showing consistent overall benefit before they were implemented. In a rapidly lethal disease with a high mortality rate like sepsis, the decision to publish guidelines based on low quality data was questionable at best and has required multiple guideline bundle revisions following studies disproving components. Although the SSC sepsis bundles have now evolved into a single bundle with fewer components, it unfortunately continues to include controversial interventions that lack strong supporting evidence and which experts report concerns about (31). Consistent with these concerns, a recent large analysis of experience with the SSC bundles demonstrated that close to 75% of patients had not been administered all components (32). Fortunately though, unlike the SEP-1 mandate that can be tied to hospital reimbursement and compel clinicians to use it, the SSC sepsis bundles have been part of a guideline that clinicians can institute or not based on their experience and judgement.

Development and institution of SEP-1

The National Quality Forum (NQF) initially evaluates PMs for CMS (21). The NQF is largely funded by grants from the US government and membership is made up of experts from industry, healthcare groups and academia. If the NQF finds sufficient criteria have been met, it will endorse a measure for adoption by CMS. In 2008, Henry Ford Hospital presented a sepsis bundle PM to the NQF for endorsement. There was a requirement in the submitted bundle to employ central venous pressure (CVP) and central venous oxygen saturation (ScVO₂) measures to guide hemodynamic support (21). Henry Ford Hospital and one of the PM’s authors had held the patent on a continuous ScVO₂ catheter that would provide these measures and allow this stipulation in the PM to be met (21,33). Ultimately, the NQF did not endorse the use of CVP and ScVO₂ measures in 2008 but did endorse the other submitted bundle components, although CMS did not formally adopt these at the time (21). The 2008 sepsis bundle then came up for re-endorsement by the NQF in 2012. For this re-endorsement, Henry Ford Hospital submitted its originally proposed PM and inexplicably again inserted the requirement for CVP and ScVO₂ measures for persistent hypotension that had been previously excluded by the NQF. On this second evaluation, the NQF reversed itself and endorsed the entire measure including the need for CVP and ScVO₂ guided hemodynamic support. The requirement for these measures was approved despite the following knowledge. First, rather than consensus, there was marked debate in the literature as to whether use of CVP and ScVO₂ measures benefited sepsis outcomes (21). Second, no new high-quality data was presented to justify including the hemodynamic measures in the PM five years after they had been originally rejected (21). Third, the experimental nature of these measures for sepsis still existed and three RCTs specifically testing their benefits were nearing completion (26,29,30). Finally, multiple professional medical societies had strongly encouraged NQF not to endorse the measures (21). Later examination of this review and endorsement process noted not only the potential financial conflicts related to Henry Ford Hospital holding the patent for a catheter specifically designed for CVP and ScVO₂ measures, but also the presence of troubling relationships between an NQF committee member, a trade association, a manufacturer of CVP and ScVO₂ catheters, and developers of the PM (21,33). These potentially conflicted relationships weaken the credibility of the NQFs final endorsement of SEP-1 (21).

CMS formally adopted SEP-1 in early 2014 (21). Shortly after this announcement, the first of the three RCTs testing EGDT (ProCESS) reported that CVP and ScVO₂ directed hemodynamic support for early sepsis was no better than usual care (29). These findings were confirmed by the two subsequently reported RCTs (26,30). CMS then delayed adoption of SEP-1 and revised the measure. Despite these three large RCTs demonstrating no clinical improvement with the use of ScVO₂ and CVP measures, CMS in 2015 adopted a revised version of SEP-1 that still included them (Table 1) (21,34). Specifically, this 2015 version of SEP-1 required that septic patients with hypotension 6 hours after presentation have either a five-component volume status and tissue perfusion exam (including vital sign, cardiopulmonary, capillary refill, peripheral pulse and skin exam) or two of the following: CVP measure, ScVO₂ measure, cardiac ultrasound or volume challenge test (Table 1).

The initial 2015 version of SEP-1 drew criticism from clinicians caring for septic patients (9,12,13,35). SEP-1 required considerable resources to document and comply with and was one of the most complex PMs ever adopted by CMS. Despite being of no proven survival benefit in addition to increasing costs, CVP and ScVO₂ measures were options included in this version. Other required hemodynamic interventions such as the five-component physical exam and cardiac ultrasound also appeared to lack sufficient supporting evidence to warrant inclusion in a nationally mandated PM. SEP-1 required administration of an inflexible 30 mL/kg fluid infusion for all hypotensive patients regardless of co-morbidities and pre-infusion volume status. In a patient fluid overloaded at the onset of septic shock, or in one with heart failure or end stage renal disease, this large volume could be harmful. Furthermore, SEP-1 required serial lactate measurements to guide hemodynamic support. However, lactate is not a direct or indirect measure of volume status and could be misleading (17,36). Lastly, SEP-1 required clinicians to administer broad-spectrum antibiotics within 3 hours of presentation in all patients with suspected sepsis. In stable patients, administering broad-spectrum antibiotics within 3 hours before infection is confirmed, could result in excessive and potentially harmful antibiotic use in those who after a more careful and extended examination might be proven not to have infection (9,16,37).

We were not familiar with any high-quality reproducible data supporting the hemodynamic interventions that CMS was mandating in 2015 version of SEP-1. We therefore performed a systematic review to objectively examine the strength of the evidence underlying the survival benefit of these required interventions (CVP and ScVO₂ measures were not re-examined since they had already been shown with high quality evidence to not be beneficial) (34). Serial lactate measures and 30 mL/kg fluid infusions were only supported with low quality evidence consisting of observational studies that did not control for confounders or RCTs with bias or that did not show benefit. No study had assessed the survival benefit of the five-component volume status and tissue perfusion exam or bedside cardiovascular ultrasound in septic patients. Three RCTs had examined the fluid challenge techniques in septic patients but had not found them to be beneficial. Only one low quality observational study had actually examined the 2015 version of SEP-1 in its entirety (38).

The findings from this analysis confirmed concerns that the hemodynamic interventions mandated in the 2015 SEP-1 version lacked any strong supportive scientific evidence to justify their implementation as the standard care for every suspected septic patient. CMS revised SEP-1 in 2018 and adopted a version that no longer stipulated how clinicians should direct hemodynamic support for persistent hypotension (Table 1). However, this simplified bundle still required that all patients presenting with suspected sepsis receive antibiotics within 3 hours and have a 30 mL/kg fluid infusion if meeting septic shock criteria and have a second lactate level measured if the initial lactate was ≥2 mmol/L. Criticism continued that mandating even this pared down but still not scientifically based and inflexible version of SEP-1 posed a risk if uniformly applied for all potentially septic patients (15,17,39).

We then performed another systematic review and meta-analysis and found that only low quality evidence from 17 observational studies overall supported potentially improved survival with sepsis bundles focusing primarily on antibiotic and fluid support (18). The timing of antibiotics and amounts of fluid required in these bundles varied. Only 1 of 17 studies examined a bundle that required serial lactate measures. None of the studies examined the adverse effects of implementing SEP-1 including excessive antibiotic treatment and fluid overload. Our systematic review of this low quality evidence concluded that while a sepsis PM could facilitate care of the septic patient, “without high-quality evidence for its safety and benefit, it should not preempt a qualified clinician’s judgement or hinder the clinician’s medical practice” (18).

Basis for potential harm with SEP-1

Consumers can decide with SEP-1’s compliance data now publicly available via CMS’s Hospital Compare program, where they will spend their healthcare dollars (6,7). Because SEP-1 is an all-or-none PM, these circumstances may result in pressure for some clinicians to alter how they typically treat sepsis no matter what hospital they work at. There is presently no high-quality data to inform clinicians as to the risks and benefits of SEP-1’s mandated interventions when administered uniformly to all potentially septic patients. The possibility that SEP-1 is harmful for some patients is of equal weight and importance to the possibility that it is beneficial for others.

SEP-1 requires all patients with suspected sepsis to be administered broad-spectrum antibiotics within 3 hours of the time a possible infection and 2 of 4 systemic inflammatory response syndrome (SIRS) criteria are documented. No one would disagree that antibiotic treatment is essential as early as possible for patients presenting with septic shock or a clear source of infection and studies support this approach (40-42). However, rapid antibiotic administration in many potentially septic but stable patients with unconfirmed infection may not be as urgently needed (42,43). The signs and symptoms used to identify patients with sepsis syndrome are neither specific nor diagnostic. Many patients with conditions not caused by infections can present with the same 2 of 4 SIRS criteria and an elevated lactate used to diagnose sepsis (e.g., cirrhosis, pancreatitis, heart failure, or asthma exacerbation to name a few). These non-infected patients would not benefit and might be harmed by antibacterial therapy. In a retrospective analysis of 2,579 patients admitted to ICUs in the Netherlands with a diagnosis of sepsis, 40% were subsequently found to have a low or no likelihood of infection based on CDC and International Sepsis Forum criteria (44). A retrospective analysis comparing patients with severe sepsis who progressed to septic shock versus those that did not, found that the median time to antibiotic administration in the 2945 patients not developing shock ranged up to almost 5 hours [median 2.76 h (IQR 1.60–4.82)] (43). Furthermore, in the other 984 patients, delaying antibiotics for as long as 5 hours did not appear to increase the risk of shock. While the limited data available now suggests that introduction of SEP-1 has been associated with increased antibiotic use, whether it has caused more patients without infection to receive antibiotics is unknown (14). However, looking at a prior similar experience when CMS reduced the required time for antibiotic administration for community acquired pneumonia (CAP) from 8 to 4 hours, diagnostic accuracy for the condition decreased from 74% to 66% in EDs and inappropriate antibiotic administration increased (45,46). In a survey, 42% of physicians noted that in order to comply with the CMS CAP guidelines, they prescribed antibiotics to patients they did not think had pneumonia (46). Inappropriate antibiotic administration has clearly documented risks. When broad spectrum antibiotics were prescribed longer and without a clear indication in a large group of post-operative patients, it resulted in higher rates of C. difficile infection and acute kidney injury (47). In 1,488 hospitalized general medical patients, 20% of adverse drug effects were due to antibiotics that were not clinically indicated (48). Finally, even one day of anti-pseudomonas β-lactam antibiotic coverage was shown to increase the risk of developing antibiotic resistance to anti-pseudomonas agents (49). Pressure to complete SEP-1 within a limited time period will likely lead to wider prescribing of broad-spectrum antibiotics (which include anti-pseudomonas coverage) and will have unintended and potentially harmful consequences. Of note, the Infectious Disease Society of America (IDSA) did not endorse the 2016 SSC guidelines, which are similar to SEP-1 due in part to their concern that these recommendations would result in excessive and harmful antibiotic use (37).

SEP-1 still requires administration of a fixed 30 mL/kg fluid volume within 3 hours for patients presenting with initial hypotension or meeting the measure’s definition of septic shock [i.e., 2 of 4 SIRS criteria with either systolic blood pressure <90 mmHg or a lactate ≥4 mmol/L]. This requirement appears to be based on recommendations in the SSC’s 2012 and 2016 guidelines (22,50). Once again, no one would disagree that a patient presenting with sepsis and hypotension needs prompt hemodynamic support to limit organ injury and this usually includes early fluid resuscitation. However, the actual volume a patient requires or tolerates will vary greatly based on comorbidities, presenting fluid status and the severity and type of infection. The fluid requirement for an obese 80-year-old patient with a history of congestive heart failure and chronic renal dysfunction presenting with sepsis due to a urinary tract infection would likely be very different than an otherwise healthy but dehydrated 40-year-old with a ruptured appendix. Because of such differences, there is substantial concern that SEP-1’s mandated 30 mL/kg fluid volume will be harmful, being too much for some patients or too little for others (13,15,35). While the 2016 SSC guideline based this 30 mL/kg on the average pre-randomization fluid volumes usual care patients received in the ProCESS, ARISE and ProMISe trials testing EGDT, these amounts varied greatly as can be seen by the large standard deviations or intra-quartile ranges (IQR) in the fluids actually administered: ProCESS—mean 28 mL/kg, SD ±21 mL/kg; ARISE—mean 34.7 mL/kg, SD ±20.1 mL/kg; and ProMISe—median 1,790 mL, IQR 1,000–2,500 mL] (26,29,30,50). Similar variability in the fluid volumes septic patients initially received was reported in an analysis of 3,686 patients from 9 hospitals in New York state between 2014 and 2016 (mean 27 mL/kg, SD ±20 mL/kg) (51). This variability is consistent with clinicians titrating fluids in hypotensive patients with smaller challenges that are discontinued when blood pressure rises, vasopressors are started, or there is an inadequate response to fluids and concern for fluid overload. In an observational study by the European Society of Intensive Care Medicine in 2013, the median (IQR) volume employed in fluid challenges across 2,213 critically ill patients with hypotension or other evidence of organ hypoperfusion and including 595 (27%) with sepsis, was 500 (500–999 mL) (52). In a worldwide survey of academic intensivists with 3,138 responses, only 20% reported a fluid bolus should be greater than 750 mL (53). The total fluid volume administered to patient’s pre-randomization in the Restricted Fluid Resuscitation in Sepsis-associated Hypotension (REFRESH) trial was only 1,000 mL while the first challenge after enrollment was to be 1,000 mL followed by 500 mL boluses as needed (54). Taken together, these data indicate clinicians use smaller volumes to assess volume responsiveness and titrate fluid administration to effect instead of indiscriminately administering a large fixed volume (i.e., 30 mL/kg) to all patients with suspected septic shock regardless of their underlying comorbidities. Finally, the SSC guidelines appropriately grade the evidence supporting use of a 30 mL/kg fluid challenge for every septic shock patient as weak, a grade consistent with findings from our systematic review (34,50). Standardization of this rigid unproven fluid requirement in a nationally mandated performance measure may have unintended consequences and result in net harm in many health care centers. In the absence of high quality RCTs to support it as usual care, it is difficult to justify this SEP-1 component for a rapidly lethal disease with a high mortality rate (34).

Finally, SEP-1 requires a follow-up lactate by 6 hours if an initial one is ≥2 mmol/L, presumably to be used as an indication for continued hemodynamic resuscitation and support. Increased lactate levels and their persistence may be a predictor of poor outcome (55). However, in RCTs conducted in the US, use of hemodynamic support protocols targeting reductions in lactate levels either in only septic patients (two studies) or in septic and other critically ill patients (one study) did not improve or had unclear effects on survival (56-58). Failure to show benefit with this approach is probably because lactate elevation is complex and has many etiologies not necessarily associated with septic shock, including; increased adrenergic stimulation during states of stress; thiamine deficiency; reduced lactate clearance due to renal/hepatic failure; infarcted tissue; impaired tissue oxygen consumption; and shock states causing very low cardiac output (17,36,59,60). Administering fluids based on lactate elevations seems unfounded since it is not a measure of volume status per se and doing so risks the development of fluid overload (especially in renal failure) and tissue edema that may itself interfere with tissue oxygen extraction (36). The 2016 SSC guidelines grade both the recommendation and evidence supporting the use of serial lactate levels for hemodynamic decisions as weak yet it must be measured at least once and sometimes serially in SEP-1 (50).

Experience with SEP-1: hospital compliance

At least eight studies are now available describing compliance with SEP-1 based on data collected over periods between October 2015 and March 2018 and that would pertain primarily to compliance with the 2015 SEP-1 version (38,61-67) (Figure 1, Table 2). Overall compliance rates (i.e., the percentage of patients within a hospital reportedly receiving all SEP-1 components) varied widely across hospitals in these studies from 31 to 80% (Table 2) (38,61-67). One study noted a compliance of 31% before and 71% after introduction of a code sepsis protocol (67). Overall however, 75% of studies reported overall mean or median compliance rates that were ≤54%. These findings indicate that to date, compliance with SEP-1 has been low. Several hospital and patient characteristics associated with the reported compliance rates are outlined in Table 2. However, given the concerns and questions that have been raised about SEP-1, there are important reasons justifying non-compliance including: uncertainty about the diagnosis of sepsis in a particular patient, the presence of contraindications for required bundle interventions, need for additional time to confirm an infection diagnosis and avoidance of unnecessary antibiotic administration in a minimally ill patient, and clinician concern that required interventions are unproven (e.g., survival benefit of performing serial lactate measures). Although an option may be available to withhold bundle components, how often clinicians have employed this when needed and for what reasons are unknown. Thus far, little data is available to determine the reasons for SEP-1 noncompliance in patients. However, a study in 1,027 patients from two hospitals in New York which looked specifically at compliance with the 30 mL/kg fluid infusion requirement (present in a New York State Department of Health mandate and SEP-1), found this compliance was significantly lower in patients with CHF or chronic kidney disease and contraindications to excessive fluid administration (15).

Figure 1.

The median or mean (designated by * or ** respectively) percentages of either hospitals or patients reportedly compliant with the Centers for Medicare and Medicaid Services sepsis bundle performance measure (SEP-1) in eight studies. Interquartile ranges (IQR) or standard deviations (±) are shown where they were provided. At the bottom of the figure are shown the total number of hospitals or patients these percentage compliances were measured across. For one study by Barbash et al. that compared compliances in safety-net versus non-safety-net hospitals and the study by Whitfield et al. that compared compliances before (pre) and after (post) implementation of a “sepsis code” hospital protocol, two bars are shown.

Table 2. Summary of compliance data and other results from eight studies examining experience with the CMS sepsis bundle performance measure (SEP-1).

Study	Time period	Database		Overall SEP-1 compliance*	Other results
		Source	Hospital number
Ramsdell [2017] (38)	4/2015–2/2016	Individual chart review	1 (110 cases)	51.80%	Compliance with individual components:
					• Lactate measured: 88.2%
					• Blood cultures obtained: 88.2%
					• Antibiotics initiated: 90.0%
					• 30 mL/kg bolus
					• Repeat lactate: 80.9%
					• Vasopressor started: 71.1%
					• Volume status/tissue perfusion assessment: 34.8%
Greenwood-Ericksen [2019] (64)	10/2016–12/2016	E-QUAL**#	205 (EDs)	80% (IQR 62–95%)	Rural vs. urban hospital SEP-1 compliance:
					• Median 80% (IQR: 67–100%) vs. 80% (51–90%)
					• Mean 79% vs. 71% (P=0.049)
					• Initial lactate: 89 vs. 92% (P=0.28)
					• Initial blood cultures: 83 vs. 87% (P=0.21)
					• Antibiotics; 60 vs. 74% (P≤0.0001)
					• Fluid administration: 84 vs. 91% (P≤0.0001)
Rhee [2018] (62)	10/2015–9/2017	Individual chart review	7 (851 cases)	33%	Noncompliance within individual patients associated (P≤0.04) with:
					• Presence septic shock
					• Hospital onset sepsis
					• Vague infectious symptoms
					• Non-pulmonary infections
					40% noncompliant cases due to absence of initial or follow-up lactate level
Venkatesh [2018] (61)	10/2015–9/2016	E-QUAL**	50 (EDs)	54% (IQR 30–75%)	92% of EDs implemented QI projects to improve SEP-1 compliance
					Compliance improved from 39% to 57% during reporting year
					SEP-1 components with lowest compliance: IVF, repeat lactate, vasopressors
Barbash [2019] (66)	10/2016–9/2017	HCP**	2,851	48.9% (SD ±19.4%)	In multi-variate analysis, higher SEP-1 compliance associated with:
					• Larger reported SEP-1 case volumes (P<0.0001)
					• For profit hospitals (P<0.0001)
					• Non-teaching hospitals (P=0.04)
					• Smaller hospital size (P≤0.003)
					SEP-1 performance associated with performance on other time directed CMS measures including head CT interpretation for stroke patients, aspirin for chest pain, and ECG for chest pain patients (P<0.0001 for each)
Barbash [2019] (65)	1/2017–9/2017	HCP**	2,827	Safety-net hospitals## 46.9% (n=703)^; non-safety-net hospitals 49.2% (n=2,124)^	In adjusted analysis, decreased SEP-1 compliance in safety-net hospitals associated with:
					• Hospitals not affiliated with health system (P<0.01 for interaction)
					• Smaller hospitals (P=0.03 for interaction)
Liao [2019] (63)	4/2017–3/2018	HCP**	48	52.7% (SD ±14.8%)	In multi-variate analysis, SEP-1 compliance greater in:
					• For profit hospitals (P<0.0001)
					• Smaller hospitals with less beds (P=0.003)
Whitfield [2019] (67)	12/2016–2/2018	Individual chart review	1 (450 cases)	Pre-implementation: 30.7%; post-implementation: 71.3%	Implementation of code sepsis protocol, aiming to expedite recognition, diagnosis and treatment of patients with sepsis or septic shock
					Multidisciplinary team included emergency room physician, physician extender, triage nurse, charge nurse, clinical pharmacist, phlebotomist, and house supervisor

*, SEP-1 compliance, percentage of patients a hospital reported that received all SEP-1 components; **, E-QUAL, HCP, IQRP provided hospital level data; ^#, ACEP provided patient level data; ^##, safety-net hospitals were the top quartile of hospitals based on the proportion of Medicare days occupied by patients receiving SSI payments; ^, adjusted difference (95% CI) comparing safety-net to non-safety-net hospitals −2.3% (−4.0% to −0.6%), P<0.01. CMS, Center for Medicare and Medicaid services; ED, emergency department; E-QUAL, Emergency Quality Network (initiated launched by American College of Emergency Physicians); HCP, Hospital Compare Program; IQR, interquartile range; IQRP, Inpatient Quality Reporting Program; IVF, intravenous fluid amount administered; QI, quality improvement; SD, standard deviation.

Experience with SEP-1: impact on survival

At least four observational studies have examined the impact of SEP-1 on mortality (Table 3) (14,38,62,67). While the unadjusted odds ratio of survival (95% CI) in the only multi-center study was significantly increased in compliant patients [OR: 1.82 (95% CI: 1.19, 2.80), P=0.006], statistical significance was lost after adjustment for other variables impacting survival [OR: 1.36 (95% CI: 0.85, 2.18), P=0.21] (62). In the largest study, mortality rates did not differ significantly comparing patients from the first three months of 2015 before introduction of SEP-1 versus patients from similar time periods in either 2016 or 2017 after the measure’s introduction [86 of 615 (14%) vs. either 87 of 591 (14.7%) or 91 of 596 (15.3%), P≥0.53 for each comparison] (14). The two remaining studies reported unadjusted survival was increased either after introduction of SEP-1 [35 survivors of 48 total (72.9%) before vs. 94 of 110 (85.4%) after] or a code sepsis protocol designed to increase SEP-1 compliance [288 of 300 (96%) before vs. 150 of 150 (100%) after] (38,67). Reported p-values for these changes were 0.05 and 0.01 respectively. But these two retrospective studies were confounded. The former study reported baseline imbalances favoring SEP-1 patients (i.e., decreased hypotension and creatinine levels) and the latter introduced a code sepsis protocol to improve SEP-1 compliance that potentially itself improved survival. Furthermore, in the study which reported baseline imbalances, our calculations suggest that the odds ratio of survival (95% CI) may cross the no effect line [2.18 (0.95, 5.00)] and P=0.06 (Chi-square) (38).

Table 3. Summary of outcome data and other results from four studies examining experience with the CMS sepsis bundle performance measure (SEP-1).

Study	Time period	Study design	Number		Comparisons	Mortality	Other results
			Hosp.	Cases
Ramsdell [2017] (38)	4/2015–2/2016	RC	1	158	Compared patients before versus after SEP-1 introduction	In hospital mortality rates: before: 13 of 48 died: 27.1%; after: 16 of 110 died: 14.5%; unadjusted P=0.05	Significantly greater number of before patients had hypotension and increased creatinine levels at presentation (P≤0.01) in the after group
							3 h and 6 h bundles and overall SEP-1 compliance were increased in after group compared to before group (P<0.01)
Rhee [2018] (62)	11/2015–9/2017	RC	7	851	Compared compliant	In hospital mortality rates: noncompliant: 105 of 570 died: 18.4%; compliant: 31 of 281 died: 11.0%; odds ratio survival (95% CI): unadjusted: 1.82 (1.19-2.80), P=0.006; adjusted: 1.36 (0.85–2.18), P=0.21	Variables associated with survival*: age, non-white race, higher Elixhauser score, hospital-onset sepsis, septic shock, non-urinary infection source, vague presenting symptoms
					versus non-compliant patients		Time to antibiotics >3 h associated with death (adjusted OR 1.94, 95% CI: 1.04–3.62, P=0.038)
							Noncompliance with SEP-1 for any other reason not associated with death (1.10, 0.70–1.72, P=0.674)
Esposito [2018] (14)	1/2015–3/2015#	RC	1	1,802	Compared patients in the first 3 months of 2015 before and in the first 3 months of 2016 and of 2017 after SEP-1 introduction	Mortality rates: before [2015]: 86 of 615 died: 14.0%; after [2016]: 87 of 591 died: 14.7%; after [2017]: 91 of 596 died: 15.3%; unadjusted P≥0.53 for 2016 or 2017 vs. 2015	Over the 3-y period of study:
							• Pip/tazo use increased 24.7% (P=0.001)
							• Vanco use increased 24.5% (P=0.001)
							• Lactate orders increased 773% (P<0.001)
							• BC orders increased 41% (P<0.001)
							Approximate additional cost from 2015 to 2017:
							• Lactate levels: $23,661
							• Blood cultures: $10,040
							• Pip/tazo: $2.241
							• Vancomycin: $1.086
Whitfield [2019] (67)	12/2016–2/2018	RC	1	450	Compared patients before and after instituting an ACSP designed to increase SEP-1 compliance	In hospital mortality rates: before: 12 of 300 died: 4%; after: 0 of 150 died: 0%; unadjusted P=0.01	Before vs. after:
							• SEP-1 compliance: 31 vs. 71%, P<0.001
							• Median (IQR) total cost/case ($): 9,222 (6,216, 15,153) vs. 10,195 (6,499, 17,340) (P=0.11)
							• Median (IQR) direct variable cost/case ($): 4,906 (3,344, 8,300) vs. 5,436 (3,610, 9,871) (P=0.09)

*, based on multivariate analysis; ^#, compared data obtained during first three months of 2015 vs. 2016 vs. 2017. ACSP, Adult Code Sepsis Protocol; Hosp, hospitals; NYS, New York State; Pip/tazo, piperacillin/tazobactam; RC, retrospective cohort; Vanco, vancomycin.

In the limited data to date, it is unclear whether SEP-1 has had a substantial impact on mortality. CMS will presumably publish the much larger overall experience with the measure and provide analysis of it. However, without prospective high quality data controlling for bias such as with RCTs, it will be unknown whether any apparent reduction in mortality with SEP-1 was due to the measure’s components or other factors. As noted above, while repeated observational studies suggested that both the 6- and 24-hour SSC sepsis bundles improved survival, subsequent RCTs showed that costly components in the bundles were ineffective or harmful. A striking example was a retrospective analysis of a 7.5-year experience with the SSC bundles (32). This analysis included almost 30,000 patients that had unrefereed data submitted to the SSC in a manner analogous to the data collected for SEP-1. This analysis reported that after adjustment for multiple variables, the components directing hemodynamic support with CVP and ScVO₂ measures were significantly (P<0.001) associated with improved survival. Yet three large well-conducted RCTs then showed this practice was no more beneficial than usual care but increased cost (26,29,30). The SSC analysis also reported that policies directing administration of corticosteroids or APC were associated with improved risk-adjusted survival either significantly or in a strong trend (P=0.03 and 0.06 respectively) but RCTs have also not supported these results. Multiple factors will confound any observational analysis of the benefits or harm from SEP-1 or its individual components including; ascertainment bias; the possibility that one highly beneficial component could mask non-beneficial or possibly harmful ones; uncontrolled data collection; use of adjunctive aids with SEP-1 that impact survival; and failure to adequately adjust for unrecognized but influential variables during survival analysis.

Conclusion: do not drive blind—obtain high quality data

Sepsis is a syndrome caused by many different of types of infection and is a major US health problem with a high morbidity and mortality. A PM to assess and help guide hospitals’ diagnosis and treatment of septic patients may have benefit. However, sepsis is a complex syndrome. First, it has signs and symptoms that overlap with other serious conditions and that make confirmation of its initial diagnosis difficult. Second, the severity and course of sepsis varies greatly among patients based on the type of infection producing it and underlying comorbidities complicating it. There is no question that patients presenting with sepsis require prompt diagnosis, antibiotic therapy, and when needed—hemodynamic support. However, experts agree that these necessary therapies have risks if administered indiscriminately and are not adjusted based on the likelihood and severity of the underlying infection and on complicating comorbid conditions (9,12,13,15,17,37).

There have been ongoing concerns that SEP-1 mandates an inflexible “one size fits all” therapeutic approach for sepsis that lacks high or even moderate level evidence demonstrating its benefit and defining its risks in the highly diverse group of patients it is directed at. While the source of the low compliance reported so far with SEP-1 can be from many etiologies, it may very well reflect these concerns and clinicians’ need to individualize care in patterns not consistent with the measure. Without high quality evidence based on reproducible RCT, the true benefits and risks associated with SEP-1 are unknown. Before standardizing therapies in a rapidly lethal disease with a high mortality rate, we must hold to the dictum “do no harm” and obtain high quality evidence proving benefit and thereby safety. Our patients deserve no less.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.