A comparative analysis of sepsis identification methods in an electronic database

Abstract

Objective

To evaluate the relative validity of criteria for the identification of sepsis in an intensive care unit database.

Design

Retrospective cohort study of adult ICU admissions from 2008–2012.

Setting

Tertiary teaching hospital in Boston, Massachusetts.

Patients

Initial admission of all adult patients to non-cardiac surgical ICUs.

Interventions

Comparison of five different algorithms for retrospectively identifying sepsis, including the Sepsis-3 criteria.

Measurements and Main Results

11,791 of 23,620 ICU admissions (49.9%) met criteria for the study. Within this subgroup, 59.9% were suspected of infection on ICU admission, 75.2% of admissions had SOFA ≥2, and 49.1% had both suspicion of infection and SOFA ≥2 thereby meeting the Sepsis-3 criteria. The AUROC of SOFA (0.74) for hospital mortality was consistent with previous studies of the Sepsis-3 criteria. The CDC, Angus, Martin, CMS and explicit coding methods for identifying sepsis revealed respective sepsis incidences of 31.9%, 28.6%, 14.7%, 11.0%, and 9.0%. In-hospital mortality increased with decreasing cohort size, ranging from 30.1% (explicit codes) to 14.5% (sepsis-3 criteria). Agreement among the criteria was acceptable (Cronbach alpha 0.40–0.62).

Conclusions

The new organ dysfunction based Sepsis-3 criteria have been proposed as a clinical method for identifying sepsis. These criteria identified a larger, less severely ill cohort than that identified by previously utilized administrative definitions. The Sepsis-3 criteria have several advantages over prior methods, including less susceptibility to coding practices changes, provision of temporal context, and possession of high construct validity. However, the Sepsis-3 criteria also present new challenges, especially when calculated retrospectively. Future studies on sepsis should recognize the differences in outcome incidence among identification methods, and contextualize their findings according to the different cohorts identified.

Introduction

Sepsis is a major and economically significant disease in the Intensive Care Unit (ICU), costing over $20 billion in the US in 2011 (5.2% of all US hospital costs) [1], with costs growing to over $23 billion in 2013 (6.2% of all US hospital costs) [2]. The European Society of Intensive Care Medicine/Society of Critical Care Medicine Third International Consensus Definitions for Sepsis and Septic Shock task force (the Sepsis-3 task force) recently defined sepsis as a “life-threatening organ dysfunction caused by a dysregulated host response to infection” [3]. Analyzing retrospective databases, the authors proposed and evaluated new clinical criteria for detection of sepsis: an increase of ≥2 Sequential Organ Failure Assessment (SOFA) score points in a defined temporal context of suspected infection [3, 4]. These new criteria were further validated in a dataset of 184,875 adults in ICUs across Australia and New Zealand [5]. While the utility of the Sepsis-3 criteria for clinical care is still being deliberated upon [6], to date, little research has focused on application of the new criteria to identify septic patients in electronic health records (EHRs).

The penetration of EHRs has dramatically increased in the US, from 9.4% in 2008 to 83.8% in 2015, a 9-fold increase [7]. Research using EHRs is becoming progressively more important, and has the potential for making decision-making more precise, more robust, and more personal. Past criteria for sepsis using EHRs mainly utilized administratively assigned billing codes, with the research focused on the epidemiology of sepsis [8, 9]. The criteria as proposed by the Sepsis-3 task force offer an attractive operational definition of sepsis in retrospective observational research because they are objectively quantifiable; incorporate an approximation for the start time of clinical concern as opposed to classifying entire hospitalizations; and are based directly on the physiologic data rather than captured indirectly via administrative codes. Our study aims to examine previously employed administrative criteria for retrospective identification of patients with sepsis and compare these with the new Sepsis-3 criteria.

Material and Methods

Study Population

Study data were acquired from the Medical Information Mart in Intensive Care (MIMIC)-III database v1.4 [10]. MIMIC-III is a large, openly available de-identified dataset comprised of patients admitted to the Beth Israel Deaconess Medical Center (BIDMC, Boston Massachusetts, USA). The database encompasses admissions between 2001 and 2012. Use of the MIMIC-III database was approved by Institutional Review Boards of BIDMC and MIT. Data extraction adhered to the original sepsis-3 study as closely as possible [3, 4]. We focused on ICU admissions from years 2008–2012 for three reasons: antibiotic prescriptions are only recorded from 2003 onward, explicit sepsis codes were introduced at BIDMC in 2004, and the group of admissions between 2008–2012 are easily identifiable in the database (Supplemental Table 1 and Supplemental Figure 1).

A total of 23,620 ICU admissions were analyzed; of these, we excluded 3 non-adults, 7,536 secondary (or greater) admissions for patients to avoid repeated measures, 2,298 admissions to the cardiothoracic surgical service since their post-operative physiologic derangements do not translate to the same mortality risk as the other ICU patients, and 18 admissions with missing data. We excluded patients suspected of infection more than 24 hours before ICU admission as MIMIC-III only contains ICU data (excludes 1,250 patients), and more than 24 hours after ICU admission as we chose to focus the majority of patients who are admitted to the ICU with sepsis (excludes 824 patients). The final cohort contained 11,791 patients.

Outcomes

As in Seymour et al. [4], our primary outcome was hospital mortality and the secondary outcome was a composite of hospital mortality and/or prolonged (≥3 days) ICU length of stay (LOS).

Variables

In our study, we precisely replicated the Sepsis-3 task force [3, 4] definition of suspected infection as the acquisition of a body fluid culture temporally contiguous to administration of antibiotics. Other data extracted included patient demographics and all necessary variables for calculating SOFA scores [11], which were calculated using data from the first 24 hours of the ICU stay. The sepsis-3 criteria for sepsis were extracted as suspected infection with associated organ dysfunction (SOFA≥2). Five other definitions of sepsis were extracted: (1) explicit criteria: the presence of at least one of the two proposed International Classification of Diseases 9^th Revision (ICD-9) codes explicitly mentioning sepsis (995.92, severe sepsis and 785.52, septic shock); (2) Angus methodology: ICD-9 codes for sepsis as proposed by Angus et al. [8]; (3) Martin methodology: ICD-9 codes proposed by Martin et al. [9]; (4) the Centers for Medicare & Medicaid Services (CMS) criteria: an adaptation of the CMS Severe Sepsis and Septic Shock Management Bundle (NQF #0500) which uses a combination of diagnostic ICD-9 codes, SIRS criteria, and specific thresholds for organ dysfunction [12]; and (5) the Center for Disease Control (CDC) complete surveillance criteria, which utilize suspicion of infection criteria that are identical to Sepsis-3 along with organ dysfunction criteria that are similar (but not identical) to SOFA [13].

Analysis

Demographics for the cohort were extracted. The cohort was also grouped by survival at hospital discharge and statistical comparison between these groups was done using the two sample t-test, Pearson’s X2 test, or the Mann-Whitney-Wilcoxon U test, as indicated. We evaluated SOFA against primary and secondary outcomes. The discrimination of SOFA was evaluated using the area under the receiver operator characteristic curve (AUROC). We compared the population identified by the Sepsis-3 criteria with other populations identified by three methods: visually, using Cronbach alpha, and via their relationship to the primary and secondary outcomes. Statistical significance was set at the 0.001 level as in Seymour et al. [4].

As MIMIC-III is open to the public our study is completely accessible, reproducible, and available online [14].

Results

Demographics of the population studied are provided (Table 1). Of the 11,791 patients, 75.2% (9,323) had a SOFA score ≥2 during their first ICU day. Median age was 64.5 years (Q1–Q3: 51.1–78.5) and mean BMI was 28.7 (SD: 8.4) kg/m². Median ICU LOS was 1.9 days (Q1–Q3: 1.1–3.5), and hospital mortality was 10.8%. All demographic items had significant differences between survivors/non-survivors except gender and BMI.

Table 1.

Demographics of the cohort for all 11,791 ICU stays, as well as demographics for stays when grouped into survival and non-survival at hospital discharge. If not otherwise specified, data is represented as the median with the 25th and 75th percentiles in square brackets. BMI: body mass index, SIRS: Systemic Inflammatory Response Syndrome (score ranges between 0 and 4), SOFA: Sequential Organ Failure Assessment (score ranges between 0 and 24), Elixhauser index: summarizes the degree of comorbid burden for the patient with higher scores indicate higher levels of comorbidity (score ranges between −36 and 51).

Variables	All patients (N = 11,791)	Survivors (N = 10,514)	Non-Survivors (N = 1,277)	P-value
Age (y) [Q1–Q3]	64.5 [51.1, 78.5]	63.3 [50.0, 77.5]	74.9 [61.8, 83.7]	<0.001
Male, n (%)	6478 (54.9%)	5,795 (55.1%)	683 (53.5%)	0.28
BMI (kg/m2), mean ± SD	28.7 ± 8.4	28.7 ± 8.2	28.1 ± 10.1	0.17
Race, n (%)				<0.001
White	8497 (72.1%)	7,630 (72.6%)	867 (67.9%)
Black	1110 (9.4%)	1,036 (9.9%)	74 (5.8%)
Hispanic	457 (3.9%)	424 (4.0%)	33 (2.6%)
Elixhauser index	1 [−1, 6]	0 [−1, 6]	5 [0, 10]	<0.001
SIRS	3 [2, 3]	3 [2, 3]	3 [3, 4]	<0.001
SOFA	3 [2, 5]	3 [1, 5]	6 [4, 10]	<0.001
Mechanical ventilation, n (%)	4149 (35.2%)	3,273 (31.1%)	876 (68.6%)	<0.001
ICU length-of-stay (d)	1.9 [1.1, 3.5]	1.9 [1.1, 3.2]	2.4 [1.1, 5.6]	<0.001
30 day mortality, n (%)	1619 (13.7%)	375 (3.6%)	1,244 (97.4%)	<0.001
Hospital mortality, n (%)	1277 (10.8%)	0 (0)	1277 (100%)	<0.001

Hospital mortality was higher for patients with SOFA ≥2 (13.2%) than those with SOFA <2 (3.6%); the secondary outcome, a composite of in-hospital mortality and/or ICU length of stay ≥3 days, occurred in two fifths (41.2%) of patients. Patients suspected of infection had higher hospital mortality (12.5% vs 8.3%). Examining only these patients, SOFA had an AUROC of 0.74 (95% CI [0.72 – 0.76], primary outcome) and 0.69 (95% CI [0.68 – 0.70], secondary outcome).

The Sepsis-3 criteria identified the largest cohort of patients (49.1%, 5784 cases), followed by CDC (31.9%, 3761), Angus (28.6%, 3368), Martin (14.7%, 1734), CMS (11.0%, 1302), and Explicit (9.0%, 1062). The in-hospital mortality rate was highest in Explicit (31.4%), followed by CMS (27.2%), Martin (23.4%), CDC (18.6%), Angus (17.7%), and Sepsis-3 (14.7%). Figure 1 shows these trends graphically. The rankings for the composite outcome of mortality and long ICU LOS were similar, with the exception of the CDC and Martin criteria being transposed (Table 2.).

Figure 1.

Percentage of patients detected by the criteria are shown in blue, and the mortality rate of these patients is shown in red. Actual values are available in Supplemental Table 1.

Table 2.

Percentage of patients identified by the various sepsis criteria and the outcome frequency for the subgroups identified. The composite outcome is defined as in-hospital mortality and/or ICU length of stay ≥ 3 days.

Criteria	Patients who satisfy criteria (N, %)	In-hospital mortality for positive cases	In-hospital mortality for negative cases	Composite outcome for positive cases	Composite outcome for negative cases
SOFA≥2	8869, 75.2%	13.20%	3.60%	41.20%	19.20%
Suspected of infection	7061, 59.9%	12.50%	8.30%	46.30%	19.90%
Sepsis-3	5784, 49.1%	14.50%	7.30%	50.00%	21.90%
CDC	3761, 31.9%	18.60%	7.20%	61.10%	23.80%
Angus	3368, 28.6%	17.90%	8.00%	61.20%	25.50%
Martin	1734, 14.7%	22.70%	8.80%	60.10%	31.50%
CMS	1302, 11.0%	27.20%	8.80%	64.70%	32.10%
Explicit	1062, 9.0%	30.10%	8.90%	70.70%	32.20%

Figure 2 shows 3-set Venn diagrams for the criteria assessed. Figure 2a compares Sepsis-3, Angus, and Martin. All three criteria were satisfied by 1420 patients (12%), while 5488 patients (46.2%) did not satisfy any. Very few patients were identified by Martin alone (0.5%, 58), and similarly few for Angus (3.4%, 399). Figure 2b compares Sepsis-3, CDC, and CMS: 51.5% satisfied at least one criteria, 9.0% satisfied all three criteria. The majority of patients who satisfied CDC also satisfied Sepsis-3 (3521 patients, 93.6%), and similarly those who satisfied CMS usually also satisfied Sepsis-3 (1228 patients, 94.3). The Explicit criteria, not shown in Figure 2, were entirely subsumed by Angus criteria and almost entirely subsumed by Martin criteria (Supplemental Figure 2). Cronbach alpha for Sepsis-3 varied from acceptable (0.40–0.49 vs Explicit, CMS, Martin) to good (0.62 vs Angus and 0.76 vs CDC) (Supplemental Table 2).

Figure 2.

Venn diagrams comparing overlap in populations identified by criteria presented. (a) Martin, Angus, and Sepsis-3 criteria: 6,343 (53.8%) patients were captured by at least one criteria, while 1,420 patients (12.0%) satisfied all criteria. (b) CMS, CDC, and Sepsis-3 criteria: 6,076 patients (51.5%) satisfied at least one of the criteria, while 1,062 patients (9.0%) satisfied all criteria.

Discussion

Current large-scale EHR sepsis identification frequently rests on administrative coding primarily done for billing. In contrast, the Sepsis-3 criteria primarily relies on the assembly of contributory data elements based on physiology, via the SOFA score, and clinical practice, via the definition of suspicion of infection.

We calculated the AUROC of SOFA for both the primary outcome (hospital mortality) and the secondary outcome (composite outcome of ICU length of stay ≥3 days or hospital mortality). Our reported AUROC of 0.74 for SOFA against in-hospital mortality is similar to that of Seymour et al. (0.74) [4] and Raith et al. (0.75) [5], though it is worth noting that our AUROC of SOFA against the secondary outcome (0.69) was lower than that of Raith et al. (0.74) [5]. These results give confidence in our replication of the Sepsis-3 criteria.

We found important disparities in the identification of sepsis using the various approaches. When examining different methodologies for retrospectively identifying patients with sepsis, cohort sizes varied from small (explicit: 1062 patients, 9.0% of the entire cohort) to almost half of all patients (Sepsis-3: 5784 patients, 49.1% of the entire cohort). Among purely administrative definitions (Angus, Martin, explicit), we found similar disparities (Figure 1). Iwashyna et al. assessed variance in cohort sizes for only these administrative criteria and further performed an expert chart review of a subset of these records [15]. The authors found that the explicit criteria identified a pure cohort (100% PPV) but missed the vast majority of septic patients (9.2% sensitivity). Iwashyna et al. also found that the Angus methodology identified a larger population of septic patients (50.3% sensitivity) but at a cost of fidelity (70.7% PPV). Our results appear consistent with the conclusions of Iwashyna et al. in that mortality rate ran roughly in reverse order of cohort size, and we are able to extend their results to the CMS, CDC, and Sepsis-3 criteria. This could imply that the more restrictive cohorts represent sicker, higher risk segments of the population. Sepsis-3 identified the largest cohort in our study, and this cohort mostly encapsulated those identified by other criteria (Figure 2). Only 4.8% of patients were identified by the approaches of Angus and Martin but not by Sepsis-3 (Figure 2a), and only 2.2% were identified only by the methodologies from CMS and CDC (Figure 2b). We posit that Sepsis-3, in general, identifies a larger and likely less “pure” cohort of septic patients, but one that still remains at higher risk of mortality (14.5% vs. 7.3%) and higher risk of composite mortality/excess length of stay (50.0% vs 21.9%).

Another advantage of the Sepsis-3 criterion is the temporal context it provides. All other criteria utilized billing (ICD-9) codes which are typically assigned on hospital discharge and are not time stamped within the stay. Consequently, these administrative criteria are only capable of identifying sepsis for entire hospitalizations and cannot be used to assess the time course of the disease. In contrast, the algorithm for Sepsis-3 requires the delineation of a time point at which the patient may be septic (suspected of infection with associated organ failure). This time point could be useful in retrospective assessment of the trajectory of the patient’s illness. It is worth noting that this defined onset time may occur later in the course of the illness than optimally desirable for clinical detection [6], and alternative criteria may be necessary depending on the desired application.

Lastly, Sepsis-3 is also advantageous as it better aligns with the contemporary understanding of the pathophysiology of sepsis. Angus et al. [16] have proposed a framework to assess sepsis criteria, and Seymour et al. [13] provided a case study using this framework. Briefly, the sepsis-3 criteria for sepsis demonstrate content validity (agreement with contemporary understanding of sepsis), construct validity (agreement with similar previously utilized definitions), and criterion validity (identification of a cohort at risk of death). We provide a more detailed assessment in the Supplemental Material, and we refer the interested reader there.

Overall, Sepsis-3 appears to present usable and viable criteria for retrospectively identifying septic patients in EHRs for the three reasons discussed: (1) it is consistent with other criteria, (2) it is timely, and (3) it satisfies many forms of validity. However, there are some limitations to the Sepsis-3 criteria. Both the Sepsis-3 and the CDC criteria rely on treatments as surrogates for organ failure. More importantly for Sepsis-3, the retrospective definition of suspicion of infection is entirely dependent the actions of the clinician. As a result, the test lacks meta-reliability, that is, it is susceptible to changes unrelated to the biology of the patient.

Organ failure as captured by SOFA (and utilized by Sepsis-3) also has limitations. The neurological component utilizes the Glasgow Coma Scale, which has known issues regarding inter-rater reliability and use/scoring in intubated and/or sedated patients [17]. The respiratory component requires an arterial blood gas, and uses a low PaO2/FiO2 ratio as a marker of severity of illness. This measurement requires a known, specialized source of oxygen for accurate measurement of FiO2 and thus is variably accurate across different treatment regimens [18]. Finally, the cardiovascular component is primarily determined by the type and rate of vasopressor administration, and not on the degree of organ failure, thus scoring is susceptible to the clinician’s propensity for certain interventions. The cardiovascular component of SOFA is scored as 2 if a patient is administered low-dose dopamine, though this is infrequently done in contemporary clinical practice.. All of these issues are rooted in the inherent difficulty of quantifying the level of organ dysfunction when patients are intensively treated. In the absence of advances in direct quantification of organ function, carefully conceived simplifications of current criteria could improve robustness to variation in clinical practice and may improve construct validity. For example, instead of quantifying the level of organ dysfunction based on the type and dose of vasopressor (as is done in SOFA), criteria could be simplified to use of any vasopressor (such as in the CDC definition).

Our study has several limitations. First, our results are limited to a single tertiary medical center. Second, we excluded patients suspected of infection more than 24 hours before or after ICU admission, and our results are limited to patients admitted to the ICU with sepsis. We did not address the use of the SIRS criteria for sepsis identification as these criteria are not intended to independently identify septic patients [19]. Finally, knowledge of sepsis continues to develop, and the evaluation in this work rests vulnerably upon universal agreement of what sepsis is, how it is defined clinically, and precisely how the applicable terminologies are documented.

Conclusion

Current identification of sepsis within the U.S., outside of individual chart review, frequently relies on proxies such as administrative coding primarily done for billing. The advent of large EHR databases allows for finer grained classification of sepsis using new methods based on patient physiology. Taking advantage of a publicly available EHR, we have demonstrated that administrative and physiology based approaches result in cohorts of severely ill patients with variable outcome frequencies. Among methods assessed here, the Sepsis-3 criteria identified the largest, healthiest cohort. As more clinical research is performed on routinely collected patient data, it becomes progressively more important to develop standardized criteria that can identify sepsis in a consistent, reliable, and usable manner.