Keeping It Simple in Sepsis Measures

“I didn’t have time to write a short letter, so I wrote a long one instead.”

-Mark Twain

Sepsis is a logical target for quality measures. Specifically, sepsis represents the perfect storm of immense public health burden^1–3 combined with unexplained practice^4–6 and outcomes⁷ variation. Thus, it is not surprising that in October 2015, the Centers of Medicare and Medicaid Services (CMS) adopted a sepsis quality measure.⁸ More surprising were the complex contents of the CMS Sepsis Core Measure “SEP-1” quality measure.⁹ CMS had written a “long letter.”

The multiple processes targeted with the CMS SEP-1 quality measure can best be understood with a brief account of history. SEP-1 arose from the National Quality Forum’s (NQF) project #0500: “Severe Sepsis and Septic Shock: Management Bundle,” a measure based upon Rivers et al.’s¹⁰ single-center, randomized, controlled trial of early goal-directed therapy (EGDT) for severe sepsis. EGDT was an intervention that consisted of fluid resuscitation and hemodynamic management based upon fulfilling specific targets of central venous pressure, superior vena cava oxygen saturation (or lactic acid), and hemoglobin and mean arterial pressures.¹¹ The large mortality benefits, physiological rationale, and algorithmic responses to a variety of abnormal clinical values provided an appealing protocol to critical care and emergency physicians trained to normalize measured values, as well as policy makers looking for quality measures. Observational studies consistently showed associations between adoption of guideline-based “sepsis bundles” and improved patient outcomes,^12–14 setting the stage for the transition of NQF #0500 into SEP-1.

However, the transition from EGDT-based NQF #0500 to SEP-1 has been tumultuous. Soon after adoption of SEP-1, the consensus definitions of sepsis changed markedly. Sepsis went from being defined as the presence of infection with concomitant systemic inflammatory response syndrome (sepsis), organ dysfunction (severe sepsis), and/or shock,¹⁵ to being defined as a dysregulated response to infection resulting in life-threatening organ dysfunction (sepsis) and/or fluid-resistant hypotension requiring vasopressors and lactate greater than 2 mmol/L.¹⁶ As the study by Barbash et al.¹⁷ in this issue clearly outlines, conflicting definitions of “sepsis” have left clinicians confused as to whom the SEP-1 measure should apply. At the same time, results of 3 large, international, randomized trials investigating the efficacy of EGDT were published, providing strong evidence that EGDT did not provide improved patient outcomes over usual care.¹⁸ SEP-1 adapted with the evolving evidence base, adding putative “usual care” processes such as evaluation of skin and peripheral pulses, and use of dynamic measures of fluid responsiveness, as quality measures.¹⁹ However, as Barbash et al. also outline, the resulting process measure was incredibly complex, with potentially more than 50 data elements collected over 6 hours in the initial management of sepsis.

In addition to its unprecedented complexity, SEP-1 received criticism for the weak evidence base of its individual components. The general concepts behind SEP-1 are well-accepted tenets of sepsis management: rapid recognition, assessment and treatment of underlying infection, and institution of intravenous fluids and vasopressor support for septic shock. However, the “all or none” prescriptive nature of the SEP-1 bundle was based on a somewhat arbitrary set of measures and targets. For example, patients with septic shock must receive 30 cc/kg of intravenous fluids to be “SEP-1 compliant.” The value “30 cc/kg” was taken from the average volume of fluids reported in prior sepsis trials, essentially an expert opinion to very low level of evidence.²⁰ The strict 30 cc/kg cutoff did not take into account that “the median isn’t the message”²¹ in fluid management: optimal resuscitation targets are unclear,²² and selecting the median as a target ignores the fact that 50% of patients enrolled in international trials of EGDT received less than 30 cc/kg of initial fluid resuscitation (the interquartile range was 16–42 cc/kg).¹⁸ Thus, most participants in trials upon which the SEP-1 fluid measure was based would ironically not have met the SEP-1 measure. Mandates for physical exam and physiological measures were based on similarly low levels of evidence.

Into this context, Barbash et al. use a representative sample of US hospitals to explore the opinions of hospital quality leaders regarding the SEP-1 measure. First, the qualitative methods used by Barbash et al. warrant some explanation. Much of biomedical research is characterized by hypothesis-driven, deductive reasoning: theories are tested using observations. In contrast, the methods of Barbash et al. use inductive reasoning: observations are used to develop theories within a systematic approach called “grounded theory” that explores common themes emerging from structured interviews.²³ Inductive reasoning can later inform deductive reasoning, feeding theories into testable hypotheses. However, qualitative, inductive research is not meant to test hypotheses and is not subject to typical notions of “power and sample size” often expected of quantitative statistical analyses. Qualitative studies reach sufficient sample size when no further themes emerge, a situation called “thematic saturation”; the sample size here of 29 participants rests comfortably in the range of participants commonly needed for thematic saturation.²³

Barbash et al. identified common themes in opinions of quality leaders regarding SEP-1. Namely, the complexity of SEP-1 necessitated a major resource investment into sepsis care and data collection. The major infrastructure investments needed to comply with SEP-1 also bred frustration regarding lack of perceived fairness around the “all or none” nature of the measure and raised multiple additional challenges including lack of clinician buy-in and resistance to protocolized care. Prior qualitative studies evaluating hospital quality leaders’ opinions on performance measures identified similar concerns about lack of “fairness,”²⁴ but the implementation of SEP-1 has raised additional concern regarding the large burdens of instituting major infrastructure changes to monitor processes of care required to report on this measure. Despite the major challenges of responding to SEP-1, quality leaders expressed optimism that increased attention to sepsis would ultimately lead to better patient outcomes.

How might future sepsis quality measures achieve the adequate balance between focusing attention on improving care processes for high-impact diseases, without imposing additional burdens on the healthcare system? Lessons from Barbash et al. help us move forward. First, rather than taxing hospitals with administratively complex process measures, initial attempts at quality measures should start simply. Policy makers should consider moving forward into new areas of quality measurement in 2 ways: (1) pursue 1 or 2 processes with strong etiological links to important patient outcomes (eg, timely antibiotics in septic shock),^25–28 and/or (2) use risk-adjusted outcomes and allow individual hospitals to adopt processes that improve local patient outcomes. Evidence suggests that the introduction of a quality measure may result in improved outcomes regardless of adoption of specific target processes,²⁹ although results are mixed.^30,31 In either case, complex “all or none” measures based upon weak evidence run a high risk of inciting clinician resentment and paradoxically perpetuating poor quality by increasing healthcare costs (decreased efficiency) without gains in safety, effectiveness, timeliness, or equity.³² It has been estimated that hospitals spend on average $2 million to implement SEP-1,³³ with unclear return on the investment. The experience of SEP-1 is a reminder that, as evidence evolves, quality measures must adapt lest they become irrelevant. However, it is also a reminder that quality measures should not sit precariously on the edge of evidence. Withdrawal of process-based measures due to a changing evidence landscape breeds mistrust and impairs future attempts to improve quality.

Sepsis quality measures face additional challenges. If recent experience with interpretation of sepsis definitions can serve as a guide, variable uptake of newer sepsis definitions between/across hospitals will impair the ability to risk-adjust outcome measures and increase bias in identifying outlier hospitals.³⁴ In addition, recent studies have already raised skepticism regarding the effectiveness of individual SEP-1 bundle components, confirming suspicions that the 30 cc/kg fluid bolus is not a magic quality target. Rather, the effectiveness of prior sepsis bundles has likely been driven by improved time to antibiotics, a process unstudied in sepsis trials, but driven by increased attention to the importance of early sepsis recognition and timely management.²⁸ Timeliness of antibiotics can act as an effect modifier for more complex sepsis therapies, with quicker time to antibiotics associated with reversal of previously described effectiveness of activated protein C,³⁵ and EGDT.²⁸

Sepsis has a legacy in which improving simple processes (ie, time to antibiotics) obviates the need for more complex interventions (eg, activated protein C, EGDT). To the extent that CMS remains committed to using process-based measures of quality, those focused on sepsis are likely to be most effective when pared down to the simplest and strongest evidence base—improved recognition³⁶ and timely antibiotics (for patients with infection-induced organ dysfunction and shock). Taking the time to start simply may best serve our current patients and preserve stakeholder buy-in for quality initiatives likely to benefit our future patients.