Variation in the Contents of Sepsis Bundles and Quality Measures. A Systematic Review

Abstract

Rationale: Sepsis contributes to one in every two to three inpatient hospital deaths. Early recognition and treatment are instrumental in reducing mortality, yet there are substantial quality gaps. Sepsis bundles containing quality metrics are often used in efforts to improve outcomes. Several prominent organizations have published their own bundles, but there are few head-to-head comparisons of content.

Objectives: We sought to determine the degree of agreement on component elements of sepsis bundles and the associated timing goals for completion of each element. We additionally sought to evaluate the amount of variation between metrics associated with bundles.

Methods: We reviewed the components of and level of agreement among several sepsis resuscitation and management bundles. We compared the individual bundle elements, together with their associated goals and metrics. We performed a systematic review (PubMed 2008–2015) and searched publically available online content, supplemented by interviews with key informants, to identify eight distinct bundles. Bundles are presented as current as of April 2015.

Measurements and Main Results: Broadly, elements of care covered early resuscitation and short-term management. Bundles varied from 6 to 10 elements, and there were 12 distinct elements listed across all bundles. Only lactate collection and broad-spectrum antibiotics were common to all eight bundles, although there were seven elements included in at least 75% of the bundles. Timing goals for the collection of lactate and antibiotic administration varied among bundles from within 1 to 6 hours of diagnosis or admission. Notably, no bundle included metrics evaluating timeliness or completeness of sepsis recognition.

Conclusions: There is a lack of consensus on component elements and timing goals across highly recognized sepsis bundles. These differences highlight an urgent need for comparative effectiveness research to guide future implementation and for metrics to evaluate progress. None of the widely instituted bundles include metrics to evaluate sepsis recognition or diagnostic accuracy.

Sepsis, the body’s inflammatory response to severe infection, contributes to one in every two to three inpatient hospital deaths, and the incidence appears to be increasing (1–3). In 2011, aggregate hospital costs for sepsis hospitalizations totaled more than $20 billion; the National Center for Health Statistics deemed it the most expensive condition treated in hospitals in the United States (4, 5). Early recognition and treatment are essential in reducing mortality in patients with severe sepsis and septic shock (6), yet there are substantial documented quality gaps in hospitals in the United States (1, 7–9). Although uncertainty remains regarding specific details of resuscitation targets and optimal management (10–12), there are increasing efforts to improve sepsis care.

Resuscitation and management bundles are often used to systematically improve sepsis outcomes. The Surviving Sepsis Campaign (SSC) distilled its guidelines into bundles to streamline adoption of evidence-based practices (13), and these bundles have been implemented widely (9, 14, 15). Since the SSC initiated the movement toward sepsis care bundles, several other prominent organizations have created their own bundles (16–22). There are few head-to-head comparisons of the content of these bundles and no comparative effectiveness research to guide the selection of which quality metrics may be most appropriate for any given sepsis care improvement effort.

We reviewed the components of and level of agreement among sepsis resuscitation and management bundles, together with their associated quality metrics. Some of the results of this study have been reported previously in the form of an abstract (23).

Methods

Search Strategy

We conducted a literature review by searching Ovid Medline for articles published in 17 relevant critical care journals (Table 1 based on list of journals generated by Harhay and colleagues [24]) between 2008 and April 2015 (see online supplement for search strategy). To capture bundles that are currently in use but have not been published in peer-reviewed journals, we searched Google using the keywords “sepsis bundle” OR “sepsis quality improvement.” After identifying organizations with sepsis bundles, we sought to gather associated metrics via websites and PubMed-index articles, and, if none were available, we contacted the organization via e-mail to request information pertaining to any metrics associated with the bundle. If an organization had published its bundle before 2012, we attempted to contact the organization via e-mail to request information on any changes made since it had been published. When an organization had more than one bundle during the time point, we report on the most recent bundle obtained from that organization; citations are to published work regarding that organization’s bundle. For example, we analyze Intermountain Healthcare’s sepsis bundle current as of December 2014, which has been updated modestly from the published evaluation of their bundle (16).

Table 1.

Journals searched

Peer-reviewed Journals
Critical Care Medicine
Intensive Care Medicine
Journal of the American Medical Association
New England Journal of Medicine
American Journal of Respiratory and Critical Care Medicine
The Lancet
Chest
Annals of Internal Medicine
Archives of (now JAMA) Internal Medicine
Canadian Medical Association Journal
British Medical Journal
Journal of Critical Care
American Journal of Emergency Medicine
Lung
Resuscitation
Annals of Emergency Medicine
Shock

For the purpose of this report, we defined a “bundle” as a list of care elements derived from evidence-based recommendations for resuscitation and management of patients with sepsis up to 6 hours after diagnosis. Articles were excluded if they did not report the use of a specific bundle or if they indicated that another organization’s bundle was being used. We noted that several organizations published algorithms for resuscitation; these algorithms differ from bundles in that they are step-by-step guides for making care decisions rather than discretized checklists of elements. Published treatment algorithms without explicit bundles were excluded on the basis of this definition. This decision was made a priori because algorithms often introduce a level of detail that cannot be covered realistically in a concise bundle; review of all treatment algorithms may be of interest but would be a distinct project.

We summarized and compared the components within each bundle, including the timing and resuscitation goals associated with the individual elements that were most commonly included across bundles. Several bundles were published with additional care guidelines, but for the purposes of this study we looked only at elements of care included in the bundles.

We defined a “metric” as any measure proposed by an organization to assess bundle compliance or related outcomes. For the organizations that provided quality metrics to assess core bundle compliance, we examined phrasing of the measures to identify common measurement goals. Distinctions were made between metrics that evaluated whether an action listed in the bundle was completed or a resuscitation or time goal was achieved.

Results

Our systematic literature review identified 158 unique documents (Figure 1), including 36 articles that stated they were using any year’s version of the SSC bundle. Bundles were collected from nine organizations, including the SSC (13), two American national organizations (the National Quality Forum [NQF] (20) and the Centers for Medicare and Medicaid [25]), one national campaign in the United Kingdom (the Sepsis Six [19]), one state-level collaborative (MHA Keystone [21]), three private healthcare systems (Intermountain Healthcare, Kaiser Permanente Northern California, and Loma Linda [16–18]), and one public hospital (Advocate Christ Medical Center [22]). The Centers for Medicare and Medicaid Services and NQF bundles are grouped because the Centers for Medicare and Medicaid Services state they are collecting data on the NQF 500 bundle as modified in 2015.

Figure 1.

Study flow diagram. SSC = Surviving Sepsis Campaign.

Elements of care covered early resuscitation and short-term management by design. Bundles included from 6 to 10 elements, and there were 12 distinct elements listed across all bundles. Only lactate collection and broad-spectrum antibiotics were common to all eight distinct bundles, although there were seven elements included in at least 75% of the bundles (Table 2).

Table 2.

Elements included in each organization’s sepsis resuscitation and management bundle

Bundle Element	Surviving Sepsis 2012	KPNC 2014	Intermountain Healthcare 2014	NQF/CMS 2014	Loma Linda STOP Sepsis Bundle 2009	Sepsis Six 2012	MHA Keystone 2014	Advocate Christ Medical Center 2010
Lactate	X	X	X	X	X	X	X	X
Blood cultures	X	X	X	X		X	X	X
Antibiotics	X	X	X	X	X	X	X	X
Fluid	X	X	X	X		X	X	X
Maintain MAP	X	X	X	X	X		X	X
Maintain/measure CVP/ ScvO2		X	X		X		X	X
Repeat lactate	X	X	X	X	X		X
Give steroids			X		X			X
Maintain glucose			X
Ventilator target			X
Oxygen						X		X
Measure urine output						X

Definition of abbreviations: CMS = Centers for Medicare and Medicaid; CVP = central venous pressure; KPNC = Kaiser Permanente Northern California; MAP = mean arterial pressure; MHA = Michigan Health & Hospital Association; NQF = National Quality Forum; Scv_O2 = central venous oxygen saturation; STOP = Strategies to Timely Obviate the Progression of Sepsis.

Although there appears to be moderate consensus around core bundle elements, and all parties agree that timing of treatment matters, organizations disagreed by at least 2 hours about important timing goals for six of seven core bundle elements (Table 3). Timing goals for the collection of lactate and antibiotic administration, the only elements agreed on by all organizations, varied among bundles from within 1 to within 6 hours. Time zero used for each of these goals also varied within and among bundles. Of the organizations that included maintenance of central venous pressure (CVP) as a bundle goal, all agreed that it should be performed at or within 6 hours, but there was discrepancy about whether time zero should be emergency department triage time, time of intensive care unit admission, or time when the patient first met sepsis criteria or sepsis was first documented (Figure 2). There was, however, relatively little variation in resuscitation targets. Mean arterial pressure targets varied by only 5 mm Hg among bundles (from ≥60 to ≥65 mm Hg).

Table 3.

Phrasing (or paraphrasing, if official phrasing was unavailable) used by each organization to describe how the metric should be measured to assess bundle compliance

Metric	Surviving Sepsis 2012	KPNC 2014	Intermountain Healthcare 2014	NQF 2014	Loma Linda 2009	MHA 2014
Lactate measured	“Serum lactate measured within 3 h of presentation”	Number of patients with lactate collected within 1 h	Serum lactate measured	Requires a response of “yes” to the question: “Was a lactate level obtained within 3 h of time of presentation?”	Percentage of patients who had lactate measured	“Percentage of patients who had a lactate level measured”
Blood cultures collected	“Blood cultures collected before broad-spectrum antibiotics administered”		Blood cultures obtained prior to antibiotic administration	Requires a response of “yes” to the question: “Were blood cultures obtained prior to antibiotic administration and within 3 h of time of presentation?”	Percentage of patients with cultures obtained prior to antibiotics	“Percentage of patients who obtained blood cultures prior to administration of antibiotics”
Antibiotics given	“Broad spectrum antibiotics administered within 3 h of admission”	(Combined Y/N completion measure with antibiotics, fluid administration and repeat lactate)	Broad-spectrum antibiotics administered within 3 h from ED arrival	Requires a response of “yes” to the question: “Were broad spectrum antibiotics administered within 3 h of the time of presentation?”	Percentage of patients with antibiotics within 4 h of meeting bundle criteria	“Percentage of patients who received broad spectrum antibiotics within one hour of diagnosis of severe sepsis or septic shock” *Bundle  = within 3 h
Antibiotics appropriate						“Percentage of patients who received appropriate antibiotic based on culture and sensitivity”
Fluid given	“30 ml/kg crystalloid fluid bolus delivered” *Bundle = within 3 h, metric is Y/N within 24 h	(Combined Y/N completion measure with antibiotics, fluid administration and repeat lactate)	Fluid bolus of 30 ml/kg PBW of crystalloid IV administered for hypotension (MAP) <65 or SBP <90 and/or lactate ≥4 mmol/L	Requires a response of “yes” or “not applicable” to the question: “Were 30 ml/kg of crystalloid administered for hypotension or lactate ≥4 mmol/L within 6 h of the time of presentation?”		“Percentage of patients who received 30 mL/kg crystalloid for hypotension or lactate ≥4 mmol/L”
Vasopressors given for hypotension	“Apply vasopressors for hypotension to maintain MAP ≥ 65”		Vasopressors applied to maintain MAP ≥60 mm Hg if hypotension does not respond to initial 30 ml/kg PBW fluid	Requires a response of “yes” or “not applicable” to the question: “Were vasopressors applied within 6 h of the time of presentation for hypotension that did not respond to initial fluid resuscitation to maintain a mean arterial pressure ≥ 65 mm Hg?”
MAP target achieved		Patients at MAP target at 6 h after time zero			Percentage of patients with MAP ≥ 65 mm Hg or SBP ≥90 mm Hg within 6 h	“Percentage of patients for whom a goal of MAP ≥65 mm Hg was achieved within six hours”
CVP measured	CVP measured within 6 h of presentation
CVP target achieved	CVP ≥8 achieved within 6 h of presentation	CVP within target at 6 h after time zero	If, after fluids, MAP < 60 and/or most recent lactate ≥ 4, was either A or B achieved: A) CVP ≥ 8 AND ScvO2 ≥ 70% B) If using NICOM protocol, until patient is either no longer fluid responsive or becomes stable off vasopressors AND until cardiac index is ≥2.5 L/min/m2		Percentage of patients with CVP ≥8 mm Hg within 6 h	“Percentage of patients for whom a goal of CVP ≥8 mm Hg was achieved within six hours”
ScvO2 measured	ScvO2 measured within 6 h of presentation
ScvO2 target achieved	ScvO2 70% (or SvO2 65%) achieved within 6 h of presentation	ScvO2 within target at 6 h	If, after fluids, MAP < 60 and/or most recent lactate ≥ 4, was either A or B achieved: A) CVP ≥ 8 AND ScvO2 ≥ 70% B) If using NICOM protocol, until patient is either no longer fluid responsive or becomes stable off vasopressors AND until cardiac index ≥ 2.5 L/min/m2		Percentage of patients with ScvO2 ≥ 70% within 6 h	Percentage of patients for whom a goal of ScvO2≥70% was achieved within six hours
Repeat lactate	Remeasure lactate	Combined Y/N completion measure with antibiotics, fluid administration, and repeat lactate	If initial lactate ≥ 2 mmol/L, repeat measurement within 6 h of ED arrival	Requires a response of “yes” or “not applicable” to the question: “Was serum lactate re-measured if initially elevated within 6 h of presentation”
Lactate clearance		Intermediate and high lactate clearance within 12 h of index lactate				Percentage of patients who had a normalization of lactate within six hours

Definition of abbreviations: CVP = central venous pressure; ED = emergency department; IV = intravenous; KPNC = Kaiser Permanente Northern California; MAP = mean arterial pressure; MHA = Michigan Health & Hospital Association; NICOM = noninvasive cardiac output monitoring; NQF = National Quality Forum; PBW=predicted body weight; SBP = systolic blood pressure; Scv_O2 = central venous oxygen saturation; Sv_O2 = venous oxygen saturation; Y/N = binary yes/no scoring.

If statement is not described as a percentage, it is a yes (all parts of the statement are true) or no question.

^*Indicates discrepancies between timing goals within bundle and timing measured by metric.

Figure 2.

Variation in timing recommended for completion of primary bundle elements. The red asterisk denotes the Surviving Sepsis Campaign bundle recommendation. CVP = central venous pressure; MAP = mean arterial pressure.

Seven organizations published or provided quality metrics for assessing bundle compliance. Several metrics highlighted a difference in methodology by tracking whether a step of the protocol was completed (i.e., CVP measured) vs. tracking whether a therapeutic goal was achieved (CVP target achieved) (Table 4). Notably, only one organization suggested an outcome metric for tracking sepsis recognition time in the emergency department setting, and no bundle included metrics for evaluating timeliness or completeness of recognition of sepsis.

Table 4.

Components of bundle elements that are measured by associated quality metrics

Metric	Surviving Sepsis 2012	KPNC 2014	Intermountain Healthcare 2014	NQF/CMS 2014	Loma Linda 2009	MHA 2014	Advocate Christ Medical Center 2010
Lactate measured	X	X	X	X	X	X	X
Blood cultures collected	X		X	X	X	X	X
Antibiotics given	X	X	X	X	X	X	X
Antibiotics appropriate						X
Fluid given	X	X	X	X		X	X
Vasopressors given for hypotension	X		X	X			X
MAP target achieved		X	X		X	X
CVP measured	X						X
CVP target achieved	X	X	X		X	X	X
ScvO2 measured	X						X
ScvO2 target achieved	X	X	X		X	X
Repeat lactate	X	X	X	X
Lactate clearance		X				X

Definition of abbreviations: CMS = Centers for Medicare and Medicaid; CVP = central venous pressure; KPNC = Kaiser Permanente Northern California; MAP = mean arterial pressure; MHA = Michigan Health & Hospital Association; NQF = National Quality Forum; Scv_O2 = central venous oxygen saturation.

Discussion

Although movement toward bundled sepsis care gained prominence from a single source (the SSC), widely available bundles for early sepsis treatment show disagreement regarding critical aspects of their construction and implementation. Organizations agree that early resuscitation is critical for these patients, but consensus does not exist as to what must be included in that resuscitation. Organizations disagree about the timing goals for every core bundle element. Quality metrics associated with bundles demonstrate significantly different methodologies for tracking bundle compliance. To what extent these differences would result in differential ranking of hospitals or changes in practice patterns is an empirical question that urgently needs to be answered.

Our review highlights a gap in existing efforts to improve care for patients with sepsis. None of these organizations appear to have integrated metrics to track rates of missed diagnosis or overdiagnosis of sepsis in their care bundles. Several organizations use a screening tool to aid in the diagnosis of sepsis, but there is an absence of tools that can be used to assess how well we are doing at making the diagnosis in an accurate and timely manner. Yet we know that for sepsis, diagnosis is a critical challenge, with frequent misses and delays detected on chart review (9). Risks of overdiagnosis, with subsequent illusory improvements in metrics (26), are also not accounted for in existing measures. An important priority for next-generation efforts to improve sepsis care must be to develop feasible ways to assess the completeness and timeliness of sepsis recognition (27), evaluating both missed cases and the risks of overdiagnosis.

The disagreements among bundles may represent reasonable disagreements about priorities. Although all were created to improve sepsis outcomes, some prioritize actions that can be done quickly by any healthcare professional, such as the Sepsis Six bundle’s focus on providing oxygen and measuring urine output. At the other end of the spectrum are bundles that are more comprehensive prioritizations of evidence-based practice recommendations for resuscitation and early management, such as the Intermountain bundle. There are no clear grounds to simply recommend one set of bundles and metrics over another. Recent evidence from the ProCESS trial, corroborated by ARISE and ProMISe (11, 12), has led organizations such as the SSC and the NQF to make changes within their bundle, further highlighting the need for comparative effectiveness research to inform which bundle and metrics should be recommended. Such research should consider the improvements in sepsis outcomes as well as the reporting burden, implementation priorities, and sustainability of alternative approaches. Until then, appeals to authority, conformity, or convenience may hold sway.

Bundles exist as a non–evidence-based way to prioritize evidence-based recommendations for sepsis care. The dissonance among these efforts highlights a gap in our understanding of how to move from evidence to quality improvement interventions. Figure 3 presents one conceptual model of such a progression and the tools that have been proposed to help at each step. Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) has emerged as a dominant method for moving from the scientific evidence to a specific set of guidelines and recommendations for practice (19). However, for guidelines to be effective, their recommendations must be implemented. Variation in the degree to which guidelines facilitate the translation of their recommendations into action is one important barrier to implementation. The Appraisal of Guidelines for Research and Evaluation (AGREE) tool has been developed to evaluate the quality of guidelines and their feasibility of implementation (20). However, both GRADE and AGREE fail to guide the process through which individual recommendations can or even should be integrated into bundles.

Figure 3.

Conceptual model of alternative approaches to move from evidence to intervention to improve practice. Horizontal arrows indicate bodies of evidence-based work that can be used to support a given step. AGREE = Appraisal of Guidelines for Research and Evaluation; GRADE = Grading of Recommendations, Assessment, Development, and Evaluations; Q. I. = Quality Improvement.

In addition, metrics are frequently developed by health systems to manage the implementation of bundles and recommendations. Such metrics should be evaluated for their reliability, openness to gaming, and propensity to cause unintended effects, but the state of the art here is still unformalized and largely unstudied (28). There is some emerging work on the best way to report metrics to decision-makers (21), and, once a deficit has been reported, a growing body of work in implementation science will guide how to craft an intervention.

We note that there are also useful frameworks for doing a full evaluation as to whether a given set of interventions has, in fact, made things better. The Promoting Action on Research Implementation in Health Services (PARiHS) framework can guide the evaluation of candidate bundle elements for implementation (29) and the sorts of questions it may be useful to ask in customizing a bundle to its institutional context. After intervention, the Reach Effectiveness-Adoption Implementation Maintenance (RE-AIM) framework can be used to evaluate implementation outcomes (30). Neither of these frameworks can yet provide robust evidence-based prescriptions for those desiring to lead an implementation in sepsis, although they provide valuable guidance on the factors that ought to be included when any organization seeks to study its own experience.

This work is not without limitations. We evaluated the bundles and metrics for sepsis care from organizations gathered via systematic review of the literature published since 2008 and aided by key informants. We did not conduct an exhaustive survey of individual hospitals’ practices. Such an evaluation may be possible in the near future as a result of reporting efforts such as those from New York’s Rory’s Regulations, but was not feasible at the time of this writing. Furthermore, we evaluated the documents that were available as of April 2015, supplemented by key informant interviews. Our intention was to collect data on bundles and associated metrics that are currently in use, and we acknowledge that the bundles used in internal quality improvement efforts may not be reflected in the published literature. We note that not all organizations may have had the chance to respond to recently published evidence and may yet choose to update their bundles accordingly. Future work should evaluate the relationship between the documents reported and actual clinical practice in sites that have adopted these practices, as well as trace the expansion of these bundles and metrics as more attention is focused on the care of patients with sepsis.

Conclusions

In summary, diverse approaches have emerged to improve the care of severe sepsis. Although there is moderate agreement across bundles, such as on the importance of antibiotics, fluid resuscitation, and evaluation for hypoperfusion, this agreement is incomplete. An evidence base to guide the general problem of developing bundles and metrics from recommendations, and to guide the selection of specific sepsis bundles and metrics from the many options, is urgently needed.