Objective Sepsis Surveillance Using Electronic Clinical Data

Abstract

OBJECTIVE

To compare the accuracy of surveillance of severe sepsis using electronic health record clinical data vs claims and to compare incidence and mortality trends using both methods.

DESIGN

We created an electronic health record–based surveillance definition for severe sepsis using clinical indicators of infection (blood culture and antibiotic orders) and concurrent organ dysfunction (vasopressors, mechanical ventilation, and/or abnormal laboratory values). We reviewed 1,000 randomly selected medical charts to characterize the definition’s accuracy and stability over time compared with a claims-based definition requiring infection and organ dysfunction codes. We compared incidence and mortality trends from 2003–2012 using both methods.

SETTING

Two US academic hospitals.

PATIENTS

Adult inpatients.

RESULTS

The electronic health record–based clinical surveillance definition had stable and high sensitivity over time (77% in 2003–2009 vs 80% in 2012, P=.58) whereas the sensitivity of claims increased (52% in 2003–2009 vs 67% in 2012, P=.02). Positive predictive values for claims and clinical surveillance definitions were comparable (55% vs 53%, P=.65) and stable over time. From 2003 to 2012, severe sepsis incidence imputed from claims rose by 72% (95% CI, 57%–88%) and absolute mortality declined by 5.4% (95% CI, 4.6%–6.7%). In contrast, incidence using the clinical surveillance definition increased by 7.7% (95% CI, −1.1% to 17%) and mortality declined by 1.7% (95% CI, 1.1%–2.3%).

CONCLUSIONS

Sepsis surveillance using clinical data is more sensitive and more stable over time compared with claims and can be done electronically. This may enable more reliable estimates of sepsis burden and trends.

Multiple studies have reported a 2- to 3-fold rise in severe sepsis incidence over the past several decades, accompanied by substantial decreases in case fatality rates.^1–6 Almost all of these estimates are based upon claims data, however, and may therefore be biased by increasingly vigilant diagnosis and coding practices.^7–9 Indeed, we previously demonstrated that the sensitivity of sepsis codes for capturing the most overt form of sepsis, bacteremia with concurrent vasopressors or lactic acidosis, has increased significantly over time, and that improving documentation of acute organ dysfunction is also likely biasing estimates of changing sepsis severity and burden.^10,11

Given the questionable reliability of administrative claims to track severe sepsis incidence and outcomes, we developed a surveillance definition that uses clinical data instead of diagnosis codes and is potentially applicable using electronic health record (EHR) data. Our aim was to characterize the accuracy and stability of this definition over time and compare it with claims-based definitions, using manual medical chart reviews with the international consensus definition as the reference standard. We then estimated and compared changes in severe sepsis incidence and mortality rates using the clinical vs claims-based surveillance definitions.

METHODS

This was a retrospective cohort study at Massachusetts General Hospital and Brigham and Women’s Hospital in Boston, Massachusetts, involving all patients at least 18 years old hospitalized from January 1, 2003, through December 31, 2012. The study was approved by the Partners Healthcare Institutional Review Board.

Surveillance Definitions Based on EHR Clinical Data

Our surveillance definition for severe sepsis required clinical indicators of suspected infection and acute organ dysfunction (Table 1). “Suspected infection” was defined as a blood culture order and at least 4 consecutive days of antibiotics (or <4 days if antibiotics were continued until at least 1 day prior to death or discharge), with the first day of antibiotics required to be a new parenteral agent. Four days was chosen as a minimum duration of antibiotic therapy because empirical antibiotics are often stopped after 48–72 hours when cultures return negative and the patient’s condition is deemed noninfectious. We required the antibiotic start date and at least 1 organ dysfunction criterion to occur within ±2 calendar days of the blood culture order. Organ dysfunction and hypoperfusion were defined using criteria and thresholds adapted from the Surviving Sepsis Guidelines, but modified to take into account baseline levels of organ dysfunction.¹²

TABLE 1.

Comparison of the International Consensus Definition of Severe Sepsis and Surveillance Definition Based on Electronic Health Record (EHR) Clinical Data

	Severe Sepsis Consensus Definitiona	EHR Clinical Surveillance Definition
Sepsis	1Suspected or documented infection + 2≥2 systemic inflammatory response syndrome criteria: Temperature >38.3°C or <36° C Heart rate >90 beats/min Respiratory rate >20 breaths/min or mechanical ventilation or pCO2 <32mm Hg White blood cell count >12,000 or <4,000 μL, or >10% bands +	1Blood culture order + 2New intravenous antibiotic followed by any systemic antibioticb every day for total ≥4 antibiotic days (or until 1 day prior to discharge or death) +
Organ Dysfunction	3Any of the following thought to be due to sepsis: Hypotension (systolic blood pressure <90mm Hg or drop by >40mm Hg, or mean arterial pressure <65mm Hg) Serum lactate ≥2.0 mmol/L Acute lung injury with PaO2/FiO2 <250 in the absence of pneumonia as infection source, or <200 in the presence of pneumonia as infection source Urine output <0.5 mL/kg/hr for more than 2 hours despite adequate fluid resuscitation Creatinine increase by ≥0.5 mg/dL from baseline Total bilirubin ≥2 mg/dL Platelet count <100,0000 μL Coagulopathy (INR >1.5	3≥ 1 organ dysfunction marker: Any vasopressorc Initiation of mechanical ventilation, continued for ≥2 continuous calendar days (or 1 day if death occurs that day) Creatinine increase by ≥0.5 mg/dL from baselined (excluding patients with end-stage renal diseasee) Total bilirubin ≥2.0 mg/dL and increase by 100% from baselined Platelet count <100 μL and ≥50% decline from baselined INR >1.5 and ≥0.5 increase from baselined (excluding patients on warfarinf) (Serum Lactate ≥2.0 mmol/L)g EHR Clinical Surveillance Definition fulfilled if criteria 2 and 3 occur within ±2 days of blood culture order

NOTE. INR, international normalized ratio.

^aAdapted from the 1991 American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference Committee, 2001 Society of Critical Care Medicine/ European Society of Intensive Care Medicine/American College of Chest Physicians/ American Thoracic Society/ Surgical Infection Society International Sepsis Definitions Conference, and 2012 Surviving Sepsis Campaign Guidelines.

^bNew intravenous antibiotic defined by washout period of 2 days. Any subsequent oral antibiotic also requires washout period of 2 days.

^cVasopressor=norepinephrine, dopamine, vasopressin, epinephrine, phenylephrine.

^dBaseline values are defined as lowest for creatinine, bilirubin, and INR from day −30 of hospitalization admission to day of discharge, or the highest for platelets from day −30 of admission to day of discharge.

^eEnd-stage renal disease defined by International Classification of Diseases, Ninth Revision, Clinical Modification code of 585.6.

^fAny order for warfarin from day −30 of admission to day of hospital discharge.

^gSerum lactate was not included in the primary surveillance definition, but was included in a secondary definition.

We excluded lactate levels from our primary definition because the use of serum lactate orders has increased dramatically over the past decade.^13–16 We performed a sensitivity analysis, however, to assess whether and how the inclusion of lactate (with a level ≥2.0mmol/L) would affect the perceived incidence of severe sepsis over time. We also explored the utility of a parsimonious “simplified surveillance definition” that required only evidence of suspected infection and vasopressors, initiation of mechanical ventilation for at least 2 days, or an acute rise in creatinine—reflecting the most common forms of organ dysfunction associated with sepsis.^17–19 We reasoned that if a simpler definition can mirror trends detected with a comprehensive definition then it might facilitate future public health surveillance efforts across a broader array of hospitals.

Surveillance Definitions Based on Claims Data

We identified patients with claims-based indicators of severe sepsis on hospital discharge using the method of Angus et al²⁰ as modified by Iwashyna et al.²¹ This “implicit” definition flags patients with concurrent International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for infection and organ dysfunction, or “explicit” ICD-9-CM codes for severe sepsis or septic shock (995.92, 785.52). Secondarily, we examined just the patients with explicit severe sepsis or septic shock codes.

Assessment of Surveillance Definition Accuracy

We compared the accuracy of our clinical and claims-based definitions by reviewing 1,000 randomly selected medical charts of patients with at least 1 blood culture order while hospitalized. We reasoned that blood culture orders were a simple marker that would capture the great majority of patients with severe sepsis. We drew 600 medical charts from hospitalizations from 2003–2009 and 400 from 2012 in order to be able to assess for changes in the sensitivity and positive predictive value (PPV) of our various definitions over time while retaining precision of our estimates for the current era. An intensivist (C.R.) systematically reviewed each patient’s progress notes, discharge summaries, nursing flow sheets, medication records, and microbiology, laboratory, and radiology findings using a standardized data collection tool in REDCap²² to determine whether the patient met criteria for severe sepsis using the international consensus definition.²³ A second intensivist (S.K.) independently reviewed 60 randomly selected medical charts (split evenly between those initially classified as severe sepsis, septic shock, and non-severe sepsis/septic shock). Each reviewer was masked to the other’s findings as well as to patients’ ICD-9-CM codes and whether patients were positive with respect to the electronic clinical surveillance definitions. Interobserver agreement was assessed using the kappa statistic. After all medical chart reviews were complete and surveillance definitions applied, we examined discrepant cases to understand reasons for false-positives and false-negatives.

Incidence and Mortality Trends

We applied all surveillance definitions to all patients hospitalized at Massachusetts General Hospital and Brigham and Women’s Hospital in 2003–2012 and calculated annual incidence and in-hospital mortality rates for patients flagged by each definition.

Data Source

Patients’ demographic characteristics, ICD-9-CM codes, medications, laboratory results, and dates of admission, discharge, and death were retrieved from the Partners Research Patient Data Registry, a centralized clinical data warehouse that has been in full production since February 2002 and is populated with data extracted from Partners’ home-built EHR system.²⁴ We obtained blood culture data from the clinical microbiology laboratories. The dates of initiation and discontinuation of mechanical ventilation of all hospitalized patients were obtained from the respiratory therapy departments for the Massachusetts General Hospital cohort for the entire study period and from Brigham and Women’s Hospital for the years 2005–2012. We used ICD-9-CM codes (96.7x) or Current Procedural Terminology codes (94002, 94003, or 94004) to identify mechanical ventilation in the Brigham and Women’s Hospital population for the years 2003–2004.

Statistical Analyses

Exact 95% binominal CIs were calculated for sensitivity and PPV. Differences in sensitivity and PPV in 2012 vs 2003–2009, and between the clinical and claims definitions in 2012, were analyzed using the z test for 2 proportions. Ten-year incidence and mortality trends were assessed by fitting time series models with linear trends to the observed annual rates. The 10-year fitted percent change for incidence was calculated as the ratio between the fitted absolute annual change multiplied by 10 and the observed baseline incidence rate in 2003. Trends imputed from clinical and claims data were compared through the z score by dividing the difference between each slope by the square root of the sum of the variance of each fitted trend line. We considered P<.05 to be statistically significant and used 2-sided tests. All analyses were performed using SAS, version 9.4 (SAS Institute).

RESULTS

For 2003–2012, there were a total of 901,466 hospitalizations at both hospitals. Of these, 64,199 (7.1%) flagged the EHR clinical surveillance definition for severe sepsis, 69,075 (7.7%) met Angus criteria, and 11,096 (1.2%) had explicit severe sepsis or septic shock codes.

Accuracy of Surveillance Definitions

Of the 1,000 medical charts reviewed, 220 met criteria for severe sepsis. Agreement between the 2 chart reviewers was very good (kappa, 0.8). The characteristics of severe sepsis and non-severe sepsis patients are summarized in Table 2. The most common types of sepsis-induced organ dysfunction were hypotension (72.3% of cases), acute kidney injury (55.9%), and acute respiratory distress syndrome (24.1%). Septic shock was present in 88 (40%) of the 220 patients with severe sepsis. Lactate levels were measured in an increasing fraction of severe sepsis patients over time: 16 (29.6%) of 54 cases in 2003 and 71 (77.2%) of 92 cases in 2012 (P< .01). Forty-eight (21.8%) of the 220 patients with severe sepsis died in the hospital, compared with 15 (1.9%) of the 780 patients without severe sepsis (P < .01) (Table 2).

TABLE 2.

Characteristics of Patients With Severe Sepsis Determined by Medical Record Review

Variable	Severe Sepsis (n= 220)	No Severe Sepsis (n= 780)	P value
Age, median (IQR), y	65.0 (54.5–76.0)	59.5 (44.5–72.0)	<.01
Male sex	121 (55.0)	396 (50.8)	.27
Nonwhite race	51 (23.2)	219 (28.1)	.15
Primary hospital service
Medical	180 (81.8)	528 (67.7)	<.01
Surgical	32 (14.6)	165 (21.2)	.03
Other	8 (3.6)	87 (11.2)	<.01
Comorbidities (Elixhauser)
Cancer	48 (21.8)	179 (22.9)	.72
Diabetes mellitus	34 (15.5)	167 (21.4)	.05
Heart failure	54 (24.5)	100 (12.8)	<.01
Liver disease	13 (5.9)	35 (4.5)	.39
Lung disease	28 (12.7)	121 (15.5)	.30
Renal disease	35 (15.9)	89 (11.4)	.08
Primary source of infection
Pulmonary	87 (39.5)	170 (21.8)	<.01
Urinary	32 (14.5)	68 (8.7)	.01
Skin and soft tissue	10 (4.5)	74 (9.5)	.02
Gastrointestinal/intra-abdominal	44 (20.0)	74 (9.5)	<.01
Central nervous system	4 (1.8)	9 (1.2)	.44
Other	43 (19.6)	93 (11.9)	<.01
No infection	0 (0)	292 (37.4)	NA
Positive blood culture	58 (26.4)	41 (5.3)	<.01
Hospital-onset severe sepsis	62 (28.2)	NA	–
Organ dysfunction attributable to sepsis
Hypotension	159 (72.3)	NA	–
Acute kidney injury/oliguria	123 (55.9)	NA	–
Lactate ≥2.0 mmol/L	74 (33.6)	NA	–
Acute respiratory distress syndrome	53 (24.1)	NA	–
Thrombocytopenia	37 (16.8)	NA	–
Coagulopathy	33 (15.0)	NA	–
Hepatic injury	28 (12.7)	NA	–
Outcomes
Hospital length of stay, median (IQR), d	11 (6–20.5)	6 (4–11)	<.01
Required intensive care unit	124 (56.4)	152 (19.5)	<.01
Death in hospital	48 (21.8)	15 (1.9)	<.01

NOTE. Data are no. (%) of patients, unless otherwise specified. IQR, interquartile range.

The accuracy of all surveillance definitions is summarized in Table 3. The sensitivity of the primary EHR clinical surveillance definition was stable between 2003–2009 and 2012 (77.3% vs 80.4%, P = .58), as was the PPV (51.0% vs 52.9%, P= .74). In contrast, the sensitivity of Angus codes increased from 51.6% in 2003–2009 to 67.4% in 2012 (P = .02) and sensitivity of explicit severe sepsis/septic shock codes increased from 9.4% in 2003–2009 to 26.1% in 2012 (P< .01). The PPVs of both claims-based definitions were stable across time. Reasons for false-negatives and false-positives for the clinical and Angus claims surveillance definitions after review of discrepant cases are detailed in Table 4.

TABLE 3.

Accuracy of Surveillance Definitions Based on Electronic Health Record (EHR) Clinical and Claims Data for Identifying Hospitalizations With Severe Sepsis Determined by Medical Record Review in 2012 vs 2003–2009

Variable	Sensitivity			Positive predictive value
	2003–2009 n (%) [95% CI]	2012 n (%) [95% CI]	P value	2003–2009 n (%) [95% CI]	2012 n (%) [95% CI]	P value
EHR clinical surveillance definitions
Primary definition:	99/128 (77.3) [69.1–84.3]	74/92 (80.4) [70.9–88.0]	.58	99/194 (51.0) [43.8–58.3]	74/140 (52.9) [44.2–61.3]	.74
Blood culture + antibiotic ≥ 4 days + (vasopressor or mechanical ventilation × 2 days or creatinine increase ≥ 0.5 or INR >1.5 or total bilirubin ≥2.0 or platelet count <100) within ±2 days
Definition with lactate:	102/128 (79.7) [71.7–86.3]	79/92 (85.9) [77.1–92.3]	.24	102/197 (51.8) [44.6–58.9]	79/146 (54.1) [45.7–62.4]	.67
Blood culture + antibiotic ≥ 4 days + (vasopressor or lactate ≥ 2.0 or mechanical ventilation × 2 days or creatinine increase ≥ 0.5 or INR >1.5 or total bilirubin ≥2.0 or platelet count <100) within ±2 days
Simplified definition:	95/128 (74.2) [65.7–81.5]	65/92 (70.7) [60.2–79.7]	.56	95/153 (62.1) [53.9–69.8]	65/106 (61.3) [51.4–70.6]	.90
Blood culture + antibiotic ≥ 4 days + (vasopressor or mechanical ventilation × 2 days or creatinine increase ≥ 0.5) within ±2 days
Claims surveillance definitions
Modified Angus codes:	66/128 (51.6) [42.6–60.5]	62/92 (67.4) [56.8–76.8]	.02a	67/128 (52.3) [43.3–61.2]	62/112 (55.4) [45.7–64.8]	.64
1 of 1,286 infection ICD-9-CM codes and 1 of 13 organ dysfunction codes, or explicit severe sepsis (995.92) or septic shock (785.52) code
Explicit severe sepsis or septic shock codes:	12/128 (9.4) [4.9–15.8]	24/92 (26.1) [17.5–36.3]	<.01a	13/13 (100) [75.3–100]	24/25 (96.0) [79.7–99.9]	.47
995.92 or 785.52 ICD-9-CM code

NOTE. ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; INR, international normalized ratio.

^aIndicates statistically significant differences.

TABLE 4.

Reasons for False-Negatives and False-Positives for Surveillance Definitions Relative to Severe Sepsis Determined by Medical Record Review

	False-negatives	False-positives
EHR clinical surveillance definition	N=47a (of 220 chart review-confirmed severe sepsis cases)	N= 162a (of 334 EHR definition-positive cases)
	Sepsis-induced hypotension without vasopressors or other organ dysfunction (n =22) Sepsis episode not captured by EHR antibiotic criteria (n =18): Antibiotics not given on each day for ≥4 day course (n= 6) Did not meet required ≥4 day antibiotic course duration (n =4) Parenteral antibiotic given was not a “new” agent (n =4) Antibiotic administration not captured by EHR (n= 2) Only given oral antibiotics (n =1) Not given antibiotics due to being made comfort care (n =1) Lactate ≥2.0 mmol as sole manifestation of organ dysfunction (n =4) EHR rule for baseline organ dysfunction not consistent with reviewer’s assessment (n =4) Sepsis-induced organ dysfunction that occurred outside ±2 day infection window (n =2)	No evidence of infection despite blood culture order and course of antibiotics (n =33) Infection present, but organ dysfunction flagged by definition deemed not to be attributable to sepsis (n =129): Creatinine rise (n= 46) Thrombocytopenia (n =44) Vasopressors (n =25) Mechanical ventilation (n= 24) Bilirubin rise (n =20) INR rise (n= 14)
Claims (modified Angus codes)	N=91a (of 220 chart review-confirmed severe sepsis cases)	N= 111a (of 240 Angus claims-positive cases)
	Not coded for infection (n =33) Coded for infection but not coded for sepsis-induced organ dysfunction (n = 56): Hypotension or shock (n= 42) Acute kidney failure (n =24) Thrombocytopenia (n =3) Respiratory failure (n =2) Liver dysfunction (n= 2) Lactate as sole manifestation of sepsis-induced organ dysfunction so not captured by coding scheme (n =2)	Coded for organ dysfunction that was deemed not to be attributable to sepsis (n =111): Acute kidney failure (n =53) Ventilator use (n =22) Hypotension or shock (n =20) Thrombocytopenia (n =17) Delirium/encephalopathy (n =12) Liver dysfunction (n =3) Coagulopathy (n =1)

NOTE. EHR, electronic health record; INR, international normalized ratio.

^aNumbers in each cell do not necessarily add up to total N because several cases had multiple reasons for false-negative or false-positive status.

In 2012, the EHR clinical surveillance definition had superior sensitivity compared with the Angus definition (80.4% vs 67.4%, P = .04) and the explicit severe sepsis/septic shock codes (80.4% vs 26.1%, P < .01). PPV for the clinical surveillance definition (52.9%) was similar to the Angus definition (55.4%, P =.69) but less than the PPV for explicit severe sepsis/septic shock codes (96.0%, P < .01).

On sensitivity analysis, adding lactate at least 2.0 mmol/L as a criterion increased sensitivity and PPV, whereas the simplified definition (suspected infection and vasopressors, mechanical ventilation for ≥2 days, or creatinine increase of ≥0.5 mg/dL) had slightly lower sensitivity and higher PPV. Both definitions also had stable performance over time.

Of the 48 chart review–confirmed severe sepsis patients who died, the EHR clinical surveillance definition flagged 41 (85.4%), the definition with lactates included flagged 42 (87.5%), and the simplified definition flagged 40 (83.3%). In contrast, the Angus method flagged 29 deaths (60.4%) and explicit severe sepsis/septic shock codes flagged 14 (29.2%) (P< .01 for all clinical definitions vs either claims definition).

Incidence and Mortality Trends

There were substantial discrepancies between incidence trends derived from surveillance using clinical vs claims data (Figure 1). The incidence of severe sepsis using Angus codes increased from 519 per 10,000 hospitalizations in 2003 to 867 in 2012 (fitted 10-year increase of 72.4%[95% CI, 57.1%–87.6%], P<.01 for linear trend). With explicit severe sepsis/septic shock codes, the incidence increased from 21 to 160 per 10,000 hospitalizations (572% increase, [95% CI, 395%–750%], P<.01). In contrast, there was only a modest trend toward more cases using the EHR-based clinical surveillance definition, changing from 709 to 757 cases per 10,000 hospitalizations (7.7% increase [95% CI, −1.1% to +16.5%], P=.14). When including lactate at least 2.0mmol/L as an additional criterion, however, there was a significant increase in incidence, from 712 to 824 cases per 10,000 hospitalizations (18.6% increase [95% CI, 6.5%–30.6%], P=.02). Annual incidence using the simplified definition was stable, changing from 597 to 574 cases per 10,000 hospitalizations (−4.7% change [95% CI, −15.3% to +5.9%], P=.42).

FIGURE 1.

Severe sepsis incidence trends using surveillance definitions based on electronic health record (EHR) clinical data versus claims data, 2003–2012. Percentages next to each method refer to fitted 10-year change relative to 2003, with associated 95% confidence limits.

Annual changes in hospital mortality rates varied similarly (Figure 2). There was a substantial mortality decrease using Angus codes from 17.6% in 2003 to 12.6% in 2012 (fitted absolute 10-year decline of 5.4% [95% CI, 4.1%–6.7%], P< .01), whereas mortality using explicit severe sepsis/septic shock codes decreased from 50.0% to 32.2% (17.3% decline [95% CI, 15.2%–19.5%], P < .01). In contrast, mortality rates decreased more modestly amongst patients meeting the EHR definition, from 15.8% in 2003 to 14.2% in 2012 (1.7% decline [95% CI, 1.1%–2.3%], P < .01). The mortality rate associated with the definition including lactates decreased from 15.8% to 13.6% (2.6% decline [95% CI, 2.0%–3.2%], P <.01), while mortality associated with the simplified definition was stable: 17.9% in 2003 vs 17.5% in 2012 (0.5% decline [95% CI, 0.4%–1.4%], P = .30). All incidence and mortality trends imputed from clinical data were significantly different than trends from claims data (P <.01 for all comparisons).

FIGURE 2.

Severe sepsis mortality trends using surveillance definitions based on electronic health record (EHR) clinical data versus claims data, 2003–2012. Percentages next to each method refer to fitted absolute 10-year change, with associated 95% confidence limits.

DISCUSSION

We found that a surveillance definition based on EHR clinical data had superior sensitivity for identifying hospitalizations with severe sepsis compared with claims-based methods. The PPV of this definition was similar to the widely used modified Angus method but lower than explicit severe sepsis/septic shock codes. However, the clinical surveillance definition demonstrated stable performance over time, whereas the sensitivity of both claims methods increased, particularly the explicit severe sepsis/septic shock codes. On applying the definition to all hospitalized patients over 10 years, we found a much more modest increase in severe sepsis incidence and decline in hospital mortality compared with claims-based estimates.

The increasing sensitivity of claims over the past decade suggests that clinicians and coders are becoming better at appropriately recognizing and coding for sepsis and/or organ dysfunction over time. This is likely due to increasing awareness of sepsis, financial incentives to code for higher patient complexity, and the introduction of new diagnosis codes.^7–11 The larger decrease in mortality rates of patients with sepsis codes compared with that of patients fulfilling our clinical surveillance definition suggests that sepsis codes are being assigned to less ill patients over time. The stable PPV for these codes, however, suggests that they are nonetheless appropriate. Our findings do, however, underscore the need for more credible ways to measure sepsis rates over time, which is particularly important in light of the profusion of sepsis prevention initiatives and new governmental regulations compelling hospitals to adopt sepsis protocols and report adherence to quality measures.⁹ The EHR-based clinical surveillance method had stable performance over time, suggesting that it may give more reliable estimates of changes in disease incidence and mortality than claims.

Our analysis of the true severe sepsis cases missed by the EHR clinical surveillance definition revealed that most were either sepsis manifested solely by transient hypotension (a less severely ill group whose importance of capturing for surveillance can be debated) or due to failure of the antibiotic criteria to capture septic patients. Reassuringly, only 2 cases were missed because organ dysfunction occurred outside the ±2 day infection window period, suggesting that this is a reasonable timeframe for surveillance. Unsurprisingly, most false-positive cases with the EHR definition were due to the lack of a causal association attributed by the reviewer between infection and organ dysfunction. However, for purposes of standardizing surveillance, removing this potentially subjective assessment is likely to be beneficial.

Our study supports the feasibility of large-scale, automated, objective sepsis surveillance using EHR clinical data. At present there are relatively few large healthcare networks with complete electronic data spanning many years; however, the increasing adoption of EHR systems is making automatable public surveillance increasingly feasible. We note that surveillance using a parsimonious set of organ dysfunction criteria (vasopressors, mechanical ventilation, and rise in creatinine) had slightly lower sensitivity, and slightly higher PPV, compared with our full surveillance definition, with similar observed trends in incidence over time. This type of definition, with its relatively simple criteria, may facilitate surveillance across a broader array of facilities. In addition, the EHR-based clinical surveillance definitions captured more of the severe sepsis deaths compared with claims. Resources spent on surveillance should ideally capture events associated with the greatest risk of adverse outcomes in order to help target and monitor prevention efforts most efficiently.

When lactate was added as a surveillance criterion, there was a significant increase in sepsis incidence, though still less than that imputed with claims. This is likely due to the increasing use of lactate testing over time and extension to patients without evidence of other organ dysfunction.¹⁶ This may reflect the impact of the Surviving Sepsis Campaign guidelines and numerous publications suggesting that measuring lactate levels is beneficial.^12,25–28 Our findings suggest that although lactate measurement may be important for the clinical care of individual patients, it is less suited to helping estimate changes in population-level sepsis burden over time because of significant increases in testing rates.

Our study has several important limitations. First, we included data from only 2 academic hospitals, which may limit the generalizability of our findings. However, the incidence, clinical characteristics, and mortality rate of our severe sepsis cohort resembles that reported in other studies,^29–31 as do the performance characteristics of the claims methods we examined and their associated incidence and mortality trends.^21,32,33 Second, a single reviewer classified cases as severe sepsis using international consensus definitions. Cases were reviewed using structured forms, however, and there was excellent agreement with a second masked reviewer on a random subset of medical charts. Third, we limited our surveillance to patients with blood culture orders. It is unlikely, however, that many patients with severe sepsis did not have blood cultures ordered since the threshold for ordering blood cultures is very low in severely ill patients.^29,34 Fourth, several components of our surveillance definition rely on therapeutic maneuvers that may have varying thresholds between clinicians and institutions, such as initiation of antibiotics, vasopressors, and mechanical ventilation. However, claims-based methods also rely on capturing treatment modalities including mechanical ventilation.³¹ Fifth, the modified Angus definition includes codes for neurologic dysfunction, but we did not include this as one of the clinical criteria. We felt this was justified because altered mental status was not included in the most recent Surviving Sepsis Guidelines’ definition of severe sepsis.¹² Sixth, our data do not directly speak to the performance of ICD-10 codes, which are now being implemented in the United States. However, having a surveillance measure that is stable over time will only become more important as clinicians become more comfortable and proficient with the new coding scheme. Moreover, the clinical forces leading clinicians to be increasingly more vigilant about detecting and treating sepsis will continue independently of the introduction of ICD-10. Notably, a recent systematic review of the validity of sepsis claims found that the sensitivity of ICD-10 codes in other countries ranged from 5.9% to 52.5%—well below the sensitivity of our EHR definitions.³⁵ Lastly, examination of trends using clinical data cannot be viewed as definitive since both sensitivity and PPV of the surveillance definitions were imperfect. Nonetheless, the stable sensitivity and PPV of the clinical surveillance definitions across time support their use as proxies for severe sepsis over the entire decade.

In conclusion, a surveillance definition for severe sepsis based upon routinely collected EHR clinical data was more sensitive and resistant to changes in performance over time compared with claims-based methods. When we applied the surveillance definition to 10 years of data from 2 large hospitals, we found that severe sepsis incidence has been rising more modestly and mortality declining less rapidly compared with claims-based estimates. The use of EHR clinical data warrants further exploration for routine public health surveillance.