Association between Medicare’s sepsis reporting policy (SEP-1) and the documentation of a sepsis diagnosis in the clinical record

Abstract

BACKGROUND:

The role of health policy in the effort to improve sepsis diagnosis remains unclear.

OBJECTIVE:

To determine the association between the implementation of Medicare’s sepsis reporting measure (SEP-1) and sepsis diagnosis rates as assessed in clinical documentation.

DESIGN:

Interrupted time series analysis of a retrospective, electronic health record cohort

SUBJECTS:

Adult patients hospitalized with suspected infection and organ dysfunction within 6 hours of presentation to the emergency department, admitted to one of 11 hospitals in a multi-hospital health system from January 2013 to December 2017.

MEASURES:

Clinician-diagnosed sepsis, as reflected by the inclusion of the terms “sepsis” or “septic” in the text of clinical notes in the first two calendar days following presentation.

RESULTS:

Among 44,074 adult sepsis patients admitted to 11 hospitals over 5 years, the proportion with sepsis documentation was 32.2% just prior to the implementation of SEP-1 in the third quarter of 2015 and increased to 37.3% by the fourth quarter of 2017. Of the nine post-SEP-1 quarters, eight had odds ratios for a sepsis diagnosis greater than 1 (overall range 0.98 – 1.26; p-value for a joint test of statistical significance 0.005). The effects were clinically modest, with a maximum effect of an absolute increase of 4.2% (95% confidence interval: 0.9 – 7.8) at the end of the study period. The effect was greater in patients who did not require vasopressors compared to patients who required vasopressors (p-value for test of interaction 0.02).

CONCLUSIONS:

SEP-1 implementation was associated with modest increases in sepsis diagnosis rates, primarily among patients who did not require vasoactive medications.

INTRODUCTION

Timely and accurate diagnosis is an essential element of high-quality health care (1–3). Sepsis is a common and deadly disease that illustrates the importance of diagnosis, in that early treatment saves lives, but only if clinicians accurately and promptly recognize sepsis (4, 5). In the last decade policy makers enacted multicomponent regulations to improve sepsis treatment and outcomes, in part by promoting early sepsis diagnosis. Such policies include Rory’s Regulations in New York State and the Centers for Medicare and Medicaid Services (CMS) Severe Sepsis and Septic Shock Management Bundle, commonly known as SEP-1 (6–8). By incentivizing hospitals to invest in staff education, electronic prompts, and other strategies to improve sepsis recognition, policies such as these could be an important front in the effort to improve diagnostic quality (9, 10).

Sepsis policies have had mixed effects on clinical outcomes, with some studies suggesting substantial improvements in sepsis survival (6) and others suggesting little or no change (8, 11). Whether these policies directly affect sepsis diagnosis rates is unknown. To better understand this issue, we analyzed changes in clinical documentation that occurred in association with the October 2015 implementation of SEP-1, a major federal policy initiative in the United States. We used electronic health record (EHR) data from a large, multi-hospital health system, providing both the granularity needed to analyze clinical documentation and the breadth needed to make generalizable inferences on the policy’s effects across the nation.

METHODS

Data source

We performed an interrupted time series analysis of multi-hospital EHR data to determine the association between SEP-1 implementation and changes in sepsis diagnosis, as reflected in bedside clinicians’ documentation. We studied 11 general, short-stay, acute care hospitals at UPMC—a multi-hospital health system based primarily in western Pennsylvania—from January 1, 2013 to December 31, 2017 (see supplement for characteristics). This time period represents 11 quarters prior to and 9 quarters following the implementation of SEP-1 in October 2015. We obtained data from the UPMC inpatient EHR (Cerner PowerChart, Cerner Corporation, Kansas City, MO) via an existing registry of hospital encounters. Earlier versions of these data were used to develop and validate current sepsis definitions (12). We collected EHR data and information from discharge coding from hospital admissions (inclusive of the preceding emergency department care for patients admitted through the ED); these data included unique patient and visit identifiers, demographics, Elixhauser comorbidities (13), vital signs, laboratory values, medications, procedures, discharge disposition, and clinical notes (14).

Patient cohort

We included patients aged 18 or older who were hospitalized with community-onset sepsis, defined as the combination of suspected infection (blood, urine, respiratory, or other body fluid culture obtained AND a dose of intravenous antibiotics) and organ failure (two or more Sequential Organ Failure Assessment (SOFA) points) within 6 hours of arrival to the emergency department (15). We focused on these patients for two reasons. First, limiting the analysis to those with community-onset sepsis allowed us to improve our precision in defining the time of sepsis onset, given that we were most interested in how SEP-1 affected clinicians’ suspicion of sepsis early in the hospital course. Second, they represent the majority of patients admitted with sepsis, making them the most relevant from a health policy perspective (16).

To further narrow the cohort and create a relatively homogenous patient group we excluded interhospital transfers, patients with very short hospital stays (<24 hours), patients with very long hospital stays (>30 days), patients who had orders for comfort measures within 24 hours of admission, and repeated encounters for individual patients via random selection.

Primary outcome

The dependent variable of interest was an indicator for whether sepsis was documented by the treating provider in the first two calendar days following presentation to the ED. We examined only electronic clinical notes from the ED and the history and physical, because we were interested in whether sepsis was diagnosed early in the treatment course. Relatedly, we excluded notes with date stamps after the first two calendar days of the admission, even if they were ED or history and physical notes, since it was possible that the note contents reflected the provider’s perspective later in the patient’s course.

We used the De-ID software package to deidentify the clinical notes prior to analysis (University of Pittsburgh, Pittsburgh PA) (17). We then used a de novo Python script to parse the notes and select only the section of the note representing what clinical providers (e.g. physicians, students, advanced practice providers) documented as the “assessment,” “impression,” or “medical decision making”, in order to avoid misclassification of sepsis in the remainder of the note. We then searched this section for the terms “sepsis” or “septic” as a clinician diagnosis of sepsis. We hypothesized that this approach would identify not only free text documentation but also diagnosis codes the clinician assigned in the EHR that populated the end of the note, both of which would reasonably reflect that the clinician considered sepsis as a potential diagnosis. Each note was assigned an indicator for whether sepsis was documented, which was then rolled up to the encounter-level, creating an indicator for whether sepsis was documented in any eligible note during the first two calendar days.

Additional variables

The primary exposure of interest was the quarter of hospital admission. Additional variables included age, sex, self-reported race (Black, white, other), Elixhauser comorbidities (13), sequential organ failure assessment (SOFA) score within 6 hours of presentation (15), admission to the intensive care unit or requirement for invasive mechanical ventilation during the hospital stay, requirement for vasopressors within 24 hours of suspected infection (e.g. administration of intravenous norepinephrine, phenylephrine, vasopressin, epinephrine, dopamine), receipt of SEP-1 compliant antibiotics, administration of 30cc/kg of intravenous fluids, serum lactate measurement as defined previously (8), the presence of an abnormal white blood cell count within 6 hours of ED presentation (above 12 ×10⁹/L, below 5 ×10⁹/L, or greater than 10% band forms on differential), and the presence of abnormal body temperature within 6 hours of ED presentation (above 37.5 or below 36 degrees Celsius).

Statistical Analyses

First, we analyzed differences between patients hospitalized before and after SEP-1; and between patients in whom sepsis was documented and those in whom sepsis was not documented, using standard summary statistics as appropriate. Second, we performed an interrupted time series analysis of changes in sepsis documentation in response to the introduction of SEP-1. This approach estimates changes in the dependent variable at a specific time point, accounting for any pre-existing temporal trends occurring prior to that time point. The primary dependent variable was the indicator for whether sepsis was documented, and the primary independent variable was the quarter of hospital admission. We modeled quarter of admission as a continuous time variable centered in quarter 4 of 2015 (the time of SEP-1 implementation) and as a quarterly indicator variable for each of the post-implementation quarters. These indicators estimate the direction, magnitude, and significance of the deviation from the pre-implementation trend carried forward into the post-implementation period. We chose this specification rather than simply modeling the pre-implementation and post-implementation trends because it allows for a non-linear trend change—for example, a gradual increase or an initial increase followed by a decrease in the outcome of interest (6). In a logistic model, the individual indicators estimate the odds ratio of the effect; we also used these parameters to calculate the absolute magnitude of the effect differences within each post-implementation quarter. To understand the overall effect of the policy, we performed a joint test of significance for these post-implementation quarterly indicators as the primary test of significance for the outcome.

We included hospital fixed effects to account for within-hospital changes over time. We used logistic regression models with robust standard errors. We reported effects as odds ratios. See supplementary material for additional detail.

Subgroup analyses

We examined differential changes in several prespecified subgroups. We chose subgroups that represented patients with high severity of illness and explicit signs of infection with the hypothesis that sepsis documentation might change differently under SEP-1 conditional on the index of suspicion from the treating clinician. The subgroups included, i.) the requirement for vasopressors within 24 hours of suspected infection; ii.) the presence of an abnormal white blood cell count within 6 hours of ED presentation; and iii.) the presence of abnormal body temperature within 6 hours of ED presentation.

We modeled these subgroups by including an interaction term between the subgroup indicator and the continuous time variable and post-implementation quarterly time indicators. This approach allowed baseline trends to differ between subgroups, and the subgroup-by-quarter indicator variables estimate the differential effect associated with SEP-1 implementation between the subgroups. The joint test of significance for these indicator interactions tests whether there was a difference in the association between SEP-1 implementation and sepsis documentation depending on the subgroup.

Sensitivity Analyses

We conducted two sensitivity analyses to assess the robustness of our findings to study assumptions. First, to assess whether changes in the documentation of the words “sepsis” or “septic” were also capturing instances in which the clinician did not believe the patient had sepsis, we labeled patients as “not sepsis” if the clinician included negating words (e.g. no, not, don’t, unlikely) within the three words before or after the term “sepsis” or “septic”—after first eliminating common words such as “it”, “is” and “and”. Second, to assess whether our approach of restricting the analysis to the end of the clinical note impression missed important information, we re-ran the analysis capturing sepsis documentation throughout the entire text of the clinical note. Finally, to better understand the context in which sepsis was or was not explicitly documented, we conducted a random chart audit of clinical notes from 10 patients with sepsis documented and 10 patients without sepsis documented. Because a full semantic and quantitative analysis of this chart audit was beyond the scope of the current manuscript, we present these results as a series of key quotations and diagnoses in the supplement to contextualize the results of our primary analysis.

All analyses were performed using Stata 17.0 (StataCorp, College Station, TX) and Python. We considered results to be significant at a p-value of 0.05. This research was approved by the University of Pittsburgh Human Research Protection Office (PRO17010375). The data were obtained with a waiver of informed consent.

RESULTS

Study population and sepsis documentation

We analyzed data from 44,074 unique patients admitted to 11 hospitals, of which 23,512 were admitted before and 20,562 admitted after SEP-1 implementation in October 2015. Sepsis was explicitly documented in 28.4% of patients prior to SEP-1 and 33.9% of patients after SEP-1 (Table 1). Patient characteristics were otherwise similar over time, with no clinically meaningful differences in age, sex, comorbidity counts, and organ failure scores. Compared to patients in whom clinicians did not document sepsis (Table 2), those in whom sepsis was documented were more likely to have an abnormal temperature (33.6% vs. 54.0%, p<0.001) and an abnormal WBC count (48.8% vs. 66.7%, p<0.001). As might be expected, patients in whom sepsis was explicitly documented were more likely to receive SEP-1 compliant sepsis care processes, such as blood cultures, antibiotics, intravenous fluids, and lactate measurement. Patients in whom sepsis was documented were also more severely ill, as indicated by the fact they were more likely to require vasopressors, mechanical ventilation, and ICU admission. The in-hospital mortality rate among patients with sepsis documented was 9.6% vs. 3.2% among those in whom sepsis was not documented.

Table 1.

Patient Characteristics by Period

		Before SEP-1	After SEP-1	p-value
No.		23512	20562
Patient Characteristics
Age, years median (IQR)		72 (59, 83)	71 (59, 83)	<0.001
Female Sex, no. (%)		12428 (52.9%)	10568 (51.4%)	0.002
Race, no. (%)
Black		2709 (11.5%)	2326 (11.3%)	0.010
Othera		493 (2.1%)	519 (2.5%)
White		20310 (86.4%)	17717 (86.2%)
No. of Elixhauser Comorbidities, median (IQR)		4 (3, 6)	4 (3, 6)	<0.001
SOFA score, median (IQR)		3 (2, 4)	3 (2, 4)	0.11
Sepsis treatment processes
Antibiotics 3 Hours from ED Arrival, no. (%)		9355 (39.8%)	8941 (43.5%)	<0.001
Lactate 3 Hours from ED Arrival, no. (%)		6320 (26.9%)	12105 (58.9%)	<0.001
Blood cultures obtained, no. (%)		20292 (86.3%)	17923 (87.2%)	0.008
Received 30cc/kg IV fluids within 3 Hours, no (%)		1981 (8.4%)	2728 (13.3%)	<0.001
Markers of infection
Abnormal Temperature, no. (%)		8457 (40.2%)	8153 (39.7%)	0.22
Abnormal WBC Count, no. (%)		12736 (54.2%)	11214 (54.5%)	0.44
Markers of illness severity
Mechanical Ventilation, no. (%)		2281 (9.7%)	1712 (8.3%)	<0.001
Vasopressors, no. (%)		1425 (6.1%)	1111 (5.4%)	0.003
ICU Admission, no (%)		6382 (27.1%)	5081 (24.7%)	<0.001
In-Hospital Mortality, no. (%)		1311 (5.6%)	966 (4.7%)	<0.001
Primary outcome
Sepsis Documented, no. (%)		6674 (28.4%)	6977 (33.9%)	<0.001

^aOther race corresponds to Chinese, Filipino, Hawaiian, American Indian/Alaskan, Asian, Hawaiian/Other Pacific Islander, Middle Eastern, Native American, or Pacific Islander

Abbreviations: IQR=interquartile range; SOFA=sequential organ failure assessment score; ED=emergency department; ICU=intensive care unit

Table 2.

Patient Characteristics by Sepsis Documentation

	Sepsis Documented
	No	Yes	p-value
No.	30423	13651
Patient Characteristics
Age, years median (IQR)	72 (60, 84)	71 (59, 82)	<0.001
Female Sex, no. (%)	16187 (53.2%)	6810 (49.9%)	<0.001
Race, no. (%)
Black	3253 (10.7%)	1782 (13.1%)	<0.001
Othera	657 (2.2%)	355 (2.6%)
White	26513 (87.1%)	11514 (84.3%)
No. of Elixhauser Comorbidities, median (IQR)	4 (3, 5)	5 (3, 6)	<0.001
SOFA score, median (IQR)	3 (2, 4)	4 (2, 5)	<0.001
Sepsis treatment processes
Antibiotics 3 Hours from ED Arrival, no. (%)	11924 (39.2%)	6372 (46.7%)	<0.001
Lactate 3 Hours from ED Arrival, no. (%)	10363 (34.1%)	8062 (59.1%)	<0.001
Blood cultures obtained, no. (%)	25039 (82.3%)	13176 (96.5%)	<0.001
Received 30cc/kg IV fluids within 3 Hours, no (%)	1880 (6.2%)	2829 (20.7%)	<0.001
Markers of infection
Abnormal Temperature, no. (%)	10235 (33.6%)	7375 (54.0%)	<0.001
Abnormal WBC Count, no. (%)	14847 (48.8%)	9103 (66.7%)	<0.001
Markers of illness severity
Mechanical Ventilation, no. (%)	1858 (6.1%)	2135 (15.6%)	<0.001
Vasopressors, no. (%)	415 (1.4%)	2121 (15.5%)	<0.001
ICU Admission, no (%)	5113 (16.8%)	6350 (46.5%)	<0.001
In-Hospital Mortality, no. (%)	965 (3.2%)	1312 (9.6%)	<0.001

^aOther race corresponds to Chinese, Filipino, Hawaiian, American Indian/Alaskan, Asian, Hawaiian/Other Pacific Islander, Middle Eastern, Native American, or Pacific Islander

Abbreviations: IQR=interquartile range; SOFA=sequential organ failure assessment score; ED=emergency department; ICU=intensive care unit

Primary analysis

In the primary analysis accounting for pre-existing temporal trends of increasing sepsis documentation, sepsis documentation was more common following SEP-1 (p-value for the joint test of significance <0.001), with rates increasing from 32.2% in the third quarter of 2015 to 37.3% in the fourth quarter of 2017. (Figure 1 and Table 3). The magnitudes of the changes were clinically modest. Eight out of nine post-SEP-1 quarters had odds ratios above 1 (OR range 0.98 – 1.26), but only two of these quarters had odds ratios above 1.2, which translated into less than a ten percent absolute difference in sepsis documentation rates compared with those expected from pre-SEP-1 trends.

Figure 1. Change in Sepsis Diagnosis.

Black dots are point estimates for quarterly rates of sepsis diagnosis. Grey shaded area represents 95% confidence interval for predicted values based on pre-SEP-1 temporal trends. Vertical dashed line indicates SEP-1 implementation.

Table 3.

Changes in Sepsis Documentation after SEP-1

Primary Analysis
Post Quarter	Expected without SEP-1, %	Observed with SEP-1, %	Absolute difference, %	95% CI for difference	OR	95% CI for OR
Quarter 4 2015	30.5	30.0	−0.4	−2.8 – 2.0	0.98	0.88 – 1.10
Quarter 1 2016	30.7	31.6	0.9	−1.5 – 3.3	1.04	0.93 – 1.17
Quarter 2 2016	31.0	32.5	1.6	−0.9 – 4.1	1.07	0.97 – 1.21
Quarter 3 2016	31.3	36.5	5.2	2.5 – 7.9	1.26	1.12 – 1.43
Quarter 4 2016	31.6	33.2	1.6	−1.2 – 4.4	1.08	0.94 – 1.22
Quarter 1 2017	32.0	32.6	0.6	−2.3 – 3.5	1.03	0.90 – 1.18
Quarter 2 2017	32.3	34.8	2.5	−0.6 – 5.6	1.12	0.97 – 1.29
Quarter 3 2017	32.6	35.6	3.0	−.03 – 6.2	1.14	0.99 – 1.33
Quarter 4 2017	33.0	37.3	4.4	0.9 – 7.8	1.22	1.04 – 1.42
Joint test of significance					0.005
Subgroup Analyses	Abnormal WBC Interaction Term		Abnormal Temperature Interaction Term		Vasopressors Interaction Term
Post Quarter	OR	95% CI	OR	95% CI	OR	95% CI
Quarter 4 2015	0.93	0.73 – 1.18	0.77	0.61 – 0.98	0.78	0.44 – 1.37
Quarter 1 2016	0.91	0.72 – 1.15	0.97	0.77 – 1.22	0.85	0.47 – 1.57
Quarter 2 2016	0.89	0.69 – 1.13	0.91	0.71 – 1.15	0.72	0.37 – 1.40
Quarter 3 2016	0.91	0.70 – 1.17	0.85	0.66 – 1.09	0.57	0.29 – 1.10
Quarter 4 2016	0.92	0.70 – 1.21	0.91	0.69 – 1.19	0.89	0.42 – 1.86
Quarter 1 2017	0.84	0.64 – 1.12	0.82	0.62 – 1.08	1.24	0.56 – 2.76
Quarter 2 2017	0.97	0.72 – 1.31	0.92	0.69 – 1.23	0.46	0.22 – 0.97
Quarter 3 2017	0.77	0.56 – 1.04	0.71	0.52 – 0.96	0.94	0.38 – 2.34
Quarter 4 2017	0.68	0.49 – 0.93	0.74	0.54 – 1.02	0.32	0.14 – 0.70
Joint test of significance	0.31		0.26		0.02

Subgroup analyses

In subgroup analyses, the increase in sepsis documentation was greater in patients not treated with vasopressors compared to those treated with vasopressors (p-value for interaction between vasopressor administration and post-SEP-1 quarter 0.02; Figure 2 and Table 3). Similarly, an increase in sepsis documentation appeared more common in patients without an abnormal WBC count or abnormal temperature within 6 hours of ED presentation, although these differences in effects were not statistically significant (p-values for test of interaction = 0.31 and 0.26, respectively).

Figure 2. Change in Sepsis Documentation by Subgroup.

For each indicator of abnormal white blood cell count (Panel A), abnormal temperature (Panel B), and vasopressor administration (Panel C), the black dots are the quarterly rates for patients with the subgroup characteristic and the grey dots are the rates for those without the subgroup characteristic. The solid lines are the predicted values from the pre-SEP-1 quarterly trends.

Sensitivity analyses

In sensitivity analyses, when recategorizing notes in which sepsis was surrounded by negating words (N=1356 patients), the differences after SEP-1 were attenuated in magnitude (Table s1). When analyzing the entire note text rather than just the assessment or impression, the overall rates of sepsis diagnosis were higher, but changes in association with SEP-1 were similar to those from the primary analysis (Table s2). Results of the chart audits are provided in supplementary Tables s3 and s4. The results of the “sepsis positive” audit do not suggest we are identifying many false-positives with a sepsis flag. The audit of the “sepsis negative” charts highlights the challenge of capturing sepsis diagnosis in the context of other conditions (such as respiratory failure from exacerbations of congestive heart failure and chronic obstructive pulmonary disease), and implicit rather than explicit documentation.

DISCUSSION

In a study of more than 40,000 patients admitted to 11 hospitals, we found that sepsis diagnoses as reflected in explicit documentation in clinical notes increased following the implementation of the SEP-1 program. However, the overall effects were modest—whereas about a third of patients had sepsis documented prior to SEP-1, sepsis was documented for still fewer than half of patients by the end the study period, with the SEP-1 policy accounting for less than a 10 percent absolute increase in sepsis documentation. In addition, the effects of SEP-1 appeared to be limited to patients who were less severely ill or who had fewer explicit signs of infection at presentation to the hospital; these patients may represent previously undiagnosed, subtle presentations of sepsis, or alternatively may be less likely to benefit from early and aggressive sepsis treatment (4).

These findings have several important implications for ongoing efforts to use policy mandates to improve acute care delivery. First, our results provide mechanistic insight into why sepsis policy mandates like SEP-1 may fail to achieve their intended impact. In order to initiate the life-saving treatment protocols that such policies espouse, clinicians must be aware of and recognize sepsis in a given patient. Our findings suggest that SEP-1 had only a modest impact on the number of patients in whom the treating clinicians diagnosed sepsis at the time of presentation. Moreover, SEP-1 appeared to increase clinicians’ suspicion for sepsis in less severely ill patients and patients who may have been less likely to have a bacterial infection—and therefore less likely to derive meaningful benefit from sepsis treatment bundles. This pattern would be consistent with observations that SEP-1 appears to have had small effects on clinical processes of care, without meaningful changes in patient outcomes (18).

Second, our findings highlight the need for health care policies to directly incentivize diagnostic accuracy as well as evidence-based treatment. The vast majority of health care policies around evidence-based practices focus on treatment, with relatively few policies focusing on diagnosis(2). Our findings suggest that a measure focused primarily on the processes that occur downstream from diagnostic decision making may fail to improve treatment when clinicians do not initiate timely sepsis treatment due to under-diagnosis. Diagnostic accuracy is gaining increasing attention by funding agencies and policy makers—our results support continued work in that space for sepsis and other acute illnesses (1–3).

Finally, our findings underscore the major ongoing challenges to diagnosing sepsis at the bedside. We found a sepsis diagnosis documented in less than half of patients with organ dysfunction who received antibiotics and had a body fluid culture obtained; this highlights the inherent difficulty of making a clinical diagnosis of sepsis. This is not a new phenomenon—diagnosing sepsis has been a challenge for decades, across multiple iterations of consensus clinical definitions (19, 20). Clinicians need better tools to help them accurately diagnose sepsis early in its course. These might include targeted education, more accurate and actionable electronic alerts, precision diagnostics that better differentiate severe illness caused by infection from that caused by non-infectious diseases, or other as-yet-undiscovered strategies (21–23). If the precision medicine revolution is to be successful in uncovering more effective sepsis treatments, the impact of current and novel treatments will remain limited until we can bring similar innovations to how we diagnose sepsis and other forms of acute, severe illness.

Our study has several important strengths. First, we used EHR data from a large, multi-hospital health system, allowing us to analyze data from many patients and hospitals and increase the generalizability of our findings. Second, by using EHR data we could analyze clinical note text to gain insight into clinicians’ thought processes as reflected in clinical documentation—which allowed us to capture suspicion for sepsis at the point of care rather than based on retrospective assignment of diagnosis codes after discharge. Third, results of a sensitivity analysis suggested we were not simply observing an increase in the inclusion of sepsis terms with negating phrases that would indicate another diagnosis is more likely. Finally, our analysis is grounded in and provides empirical support for a conceptual model that yields mechanistic insight into the difficulty of achieving broad-based improvements in treatment and outcomes using sepsis policy mandates.

Our findings should be interpreted in the context of several limitations. First, our use of the terms “sepsis” and “septic” in clinical documentation is only a proxy for whether a clinician diagnosed the patient with sepsis. Our approach does not reflect undocumented thought processes and cannot account for the possibility that a clinician might include sepsis in a differential diagnosis while ultimately settling on and treating an alternative diagnosis, or even settling on and treating sepsis without documenting it—indeed we found evidence for this phenomenon in our chart audit. These are important areas for future work to refine our ability to analyze unstructured data. Second, we did not attempt to determine the specificity or predictive value of clinician-documented sepsis, because sepsis lacks a gold standard against which the accuracy of clinician diagnosis can be judged (24). In addition, because we were seeking a mechanistic explanation for apparently low levels of bundle compliance, we were most interested in the problem of underdiagnosis among patients in which a sepsis diagnosis might reasonably have been considered—i.e. patients with organ dysfunction, body fluid cultures, and antibiotics within 6 hours of arrival to the emergency department. Third, while we included a large number of patients across 11 hospitals, the data come from a single health system with a single EHR; it is possible that an analysis from other hospitals using other documentation systems would yield different results. Fourth, because the SEP-1 policy was implemented in all US hospitals at the same time, our analysis could not contain a control group. We cannot rule out the possibility of a contemporaneous change unrelated to SEP-1 that affected sepsis documentation—although we are not aware of any such policy change nationally or within the hospitals we studied. Finally, we used a clinical definition of sepsis that combined suspected infection and organ dysfunction rather than one based on diagnosis codes—but this was an intentional decision designed to allow us to understand the impact of SEP-1 on sepsis awareness in a population with less explicit signs of sepsis, and to overcome well-described epidemiologic biases associated with sepsis diagnosis codes (25, 26).

Conclusion

The implementation of SEP-1 was associated with clinically modest effects on clinicians’ documentation of sepsis in clinical notes, with effects concentrated in less severely ill patients. Future efforts to improve health policy and quality improvement should account for clinicians’ awareness of the targeted conditions in order to more optimally target opportunities to improve care.