Validation of Adult Sepsis Event and Epidemiologic Analysis of Sepsis Prevalence and Mortality Using Adult Sepsis Event’s Electronic Health Records-Based Sequential Organ Failure Assessment Criteria: A Single-Center Study in South Korea*

Abstract

OBJECTIVES:

In 2018, the Centers for Disease Control and Prevention introduced the Adult Sepsis Event (ASE) definition, using electronic health records (EHRs) data for surveillance and sepsis quality improvement. However, data regarding ASE outside the United States remain limited. We therefore aimed to validate the diagnostic accuracy of the ASE and to assess the prevalence and mortality of sepsis using ASE.

DESIGN:

Retrospective cohort study.

SETTING:

A single center in South Korea, with 2732 beds including 221 ICU beds.

PATIENTS:

During the validation phase, adult patients who were hospitalized or visiting the emergency department between November 5 and November 11, 2019, were included. In the subsequent phase of epidemiologic analysis, we included adult patients who were admitted from January to December 2020.

INTERVENTIONS:

None.

MEASUREMENTS AND MAIN RESULTS:

ASE had a sensitivity of 91.6%, a specificity of 98.3%, a positive predictive value (PPV) of 57.4%, and a negative predictive value of 99.8% when compared with the Sepsis-3 definition. Of 126,998 adult patient hospitalizations in 2020, 6,872 cases were diagnosed with sepsis based on the ASE (5.4% per year), and 893 patients were identified as having sepsis according to the International Classification of Diseases, 10th Edition (ICD-10) (0.7% per year). Hospital mortality rates were 16.6% (ASE) and 23.5% (ICD-10-coded sepsis). Monthly sepsis prevalence and hospital mortality exhibited less variation when diagnosed using ASE compared with ICD-10 coding (coefficient of variation [CV] for sepsis prevalence: 0.051 vs. 0.163, Miller test p < 0.001; CV for hospital mortality: 0.087 vs. 0.261, p = 0.001).

CONCLUSIONS:

ASE demonstrated high sensitivity and a moderate PPV compared with the Sepsis-3 criteria in a Korean population. The prevalence of sepsis, as defined by ASE, was 5.4% per year and was similar to U.S. estimates. The prevalence of sepsis by ASE was eight times higher and exhibited less monthly variability compared with that based on the ICD-10 code.

KEY POINTS.

Question: Can the Centers for Disease Control and Prevention’s Adult Sepsis Event (ASE) be effectively applied in real-world hospital settings outside the United States, and what are the prevalence and mortality rates of ASEs in these settings?

Findings: In this retrospective cohort study, the prevalence of sepsis as determined by the ASE criteria was 5.4% annually, whereas it was only 0.7% according to the International Classification of Diseases, 10th Edition (ICD-10) code. Hospital mortality rates were 16.6% for ASE cases and 23.5% for those identified with ICD-10-coded sepsis.

Meaning: Compared with ICD-10-coded sepsis, sepsis according to ASE criteria appears approximately eight times more frequent and exhibits lower mortality.

Sepsis is characterized as life-threatening organ dysfunction resulting from a dysregulated host response to infection (1). In 2017, the global prevalence of sepsis was estimated at 48.9 million cases, culminating in 11.0 million deaths, which represented approximately 20% of all worldwide deaths (2). Given the profound implications of the disease and its consequent healthcare burden, surveillance and quality improvement initiatives related to sepsis are imperative for healthcare practitioners and policymakers. However, achieving precise estimates of sepsis and mortality remains challenging due to discrepancies in its definitions, diagnostic criteria, and coding practices (3).

In 1991, sepsis was initially defined based on the criteria of the systemic inflammatory response syndrome (4). This definition was expanded upon during the International Sepsis Definitions Conference in 2001 (5). In 2016, the Third International Consensus Definitions for Sepsis and Septic Shock, known as Sepsis-3, were introduced. According to this consensus, sepsis is diagnosed when a patient has a suspected infection, and the Sequential Organ Failure Assessment (SOFA) score increases by two points or more as a result of the infection (6). However, the SOFA score is complex and cannot be calculated automatically. This complexity makes the identification of sepsis difficult. Consequently, researchers often resort to manual reviews of electronic health records (EHRs) or use varied combinations of infection and organ dysfunction codes from administrative data (3, 7–9).

In 2018, the Centers for Disease Control and Prevention (CDC) introduced a new definition for sepsis called the Adult Sepsis Event (ASE), which leverages EHR data for sepsis surveillance (10). Rhee et al (11) posited that the EHR-based ASE criteria offer a more objective assessment of sepsis prevalence and mortality compared with claims-based data. However, only a handful of studies have evaluated the diagnostic accuracy of the ASE criteria against the Sepsis-3 criteria. Furthermore, data regarding sepsis prevalence and outcomes using the ASE criteria outside the United States remain limited (12–14). Similarly, in Korea, comprehensive data on sepsis prevalence and mortality are sparse. Previous studies have predominantly based their sepsis diagnoses on the International Classification of Diseases, 10th Edition (ICD-10) code.

In this study, we aimed to validate the diagnostic accuracy of the CDC’s ASE criteria compared with the Sepsis-3 criteria and to assess the prevalence and mortality of sepsis using the ASE definition.

MATERIALS AND METHODS

Study Design and Patients

This was a retrospective cohort study conducted at a tertiary teaching hospital in South Korea, which has 2732 beds including 221 ICU beds. The study was structured in two distinct phases. In the initial phase, our focus was on the diagnostic validation of ASE. We included patients 19 years old or older who were hospitalized or visited the emergency department between November 5 and 11, 2019. In the subsequent phase, we concentrated on evaluating the prevalence and outcomes of ASE. In this phase, we included patients 19 years old or older who were admitted to the hospital between January and December 2020. We excluded individuals who were not admitted, as well as those who were discharged, transferred, or died in the emergency department.

Definitions of Sepsis

We used multiple criteria to diagnose sepsis, including Sepsis-3, the ASE, and the ICD-10 code for sepsis. We adhered to the original definitions of the Sepsis-3 and ASE criteria (6, 10), which are described in Appendices 1 and 2 (http://links.lww.com/CCM/H525). We defined ICD-10-coded sepsis when patients had a sepsis code for claims, such as R572 for septic shock, and A418 for other specified sepsis (Appendix 3, http://links.lww.com/CCM/H525).

Phase I: Validation of ASE

We validated the diagnostic accuracy of the ASE criteria compared with the Sepsis-3 criteria. In this phase, a total of 6186 patients were included and two study personnel thoroughly reviewed the complete medical records of all patients and identified cases of sepsis using the Sepsis-3 criteria. Concurrently, we identified ASE cases for the same period. We evaluated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of ASE in comparison to the Sepsis-3 definition and identified the factors contributing to both false positives and false negatives in ASE. We also identified cases of sepsis coded using ICD-10 during the same period.

Phase II: Epidemiologic Analysis of Sepsis

After evaluating the diagnostic efficacy of the ASE in our cohort of Korean patients, we assessed the prevalence and mortality of sepsis cases diagnosed using ASE criteria. A total of 126,988 patients were included in this period, among whom cases that met the ASE criteria (Appendix 4, http://links.lww.com/CCM/H525) or ICD-10-coded sepsis were extracted. We assessed the monthly prevalence and mortality of sepsis using the ASE criteria and compared them with ICD-10-coded sepsis cases. Additionally, we explored factors linked to hospital mortality in ASE cases. The variables collected are presented in Appendix 5 (http://links.lww.com/CCM/H525).

Statistical Analysis

Variables are presented either means with sds or medians with interquartile ranges, based on their distribution. For comparing continuous variables, we used the Student t test. The chi-square test or Fisher exact test was chosen for categorical variables. To compare the coefficients of variations (CVs), we used Miller test (15). Risk factors for hospital mortality were pinpointed through multivariate logistic regression analysis. In the univariate analysis, variables with p values of less than 0.10 were incorporated, and a backward elimination approach was used for the multivariate analysis. All p values were two-tailed, with the threshold for statistical significance set at a p value of less than 0.05. All statistical analyses and graphs were conducted using R (open-source software version 4.2.1).

The ethical approval and consent to participate are presented in Appendix 6 (http://links.lww.com/CCM/H525).

RESULTS

Validation of ASE

Between November 5 and November 11, 155 cases of sepsis were diagnosed based on the Sepsis-3 criteria, whereas 247 cases were identified using the ASE criteria from 6186 patients (Supplemental eTable 1, http://links.lww.com/CCM/H525). ASE had a sensitivity of 91.6% (95% CI, 86.1–95.5%), a specificity of 98.3% (95% CI, 97.9–98.6%), a PPV of 57.4% (95% CI, 52.6–62.1%), and a NPV of 99.8% (95% CI, 99.6–99.9%) when compared with the Sepsis-3 definition. The correlation between ICD-10-coded sepsis and Sepsis-3 observed during the same period is shown in Supplemental eTables 2 and 3 (http://links.lww.com/CCM/H525). The reasons behind the false positives and false negatives for ASE are detailed in Supplemental eTables 4 and 5 (http://links.lww.com/CCM/H525).

Epidemiologic Analysis of Sepsis Prevalence and Mortality Using the ASE Criteria

From the 126,998 patients admitted to the hospital between January and December 2020, 6,872 were diagnosed with ASE. The annual incidence rate for ASE was 5.41% (5,411 per 100,000 admissions), whereas the rate for ICD-10-coded sepsis stood at 0.70% (703 per 100,000 admissions). We examined the monthly incidence rates for both ASE and ICD-10-coded sepsis, and their CV were 0.051 and 0.163, respectively (Fig. 1). ASE cases exhibited less month-to-month variability in prevalence compared with ICD-10-coded sepsis, as evidenced by Miller test (p < 0.001). The hospital mortality rates were 16.6% for ASE cases and 23.5% for ICD-10-coded sepsis. Likewise, the monthly hospital mortality rates showed reduced variability in ASE cases compared with ICD-10-coded sepsis (0.087 vs. 0.261, p = 0.001) (Fig. 2).

Figure 1.

Monthly trend of the prevalence of Adult Sepsis Event (ASE) and International Classification of Diseases, 10th Edition (ICD-10)-coded sepsis patients in 2020.

Figure 2.

Monthly trends of the hospital mortality of Adult Sepsis Event (ASE) and International Classification of Diseases, 10th Edition (ICD-10)-coded sepsis patients in 2020.

Table 1 presents the characteristics of patients diagnosed with ASE. The median age of patients was 65 years (IQR, 56–73), with 62.3% being male. Community-onset sepsis accounted for 62.3%, whereas hospital-onset sepsis constituted 37.7%. A total of 71.4% of patients were admitted to the medical department. The most common EHRs- based SOFA (eSOFA) criteria met were hyperlactatemia (60.3%), thrombocytopenia (32.7%), and hepatic injury (27.7%). The average number of organ dysfunctions was 1.8 ± 1.2. The ICU admission rate was 27.2%. Hospitalized ASE patients who died exhibited the following characteristics: they were older than survivors (66.0 vs. 64.0 yr, p < 0.001); a higher proportion were admitted to the medical department (85.1% vs. 68.7%, p < 0.001); they had higher rates of vasopressor use (41.3% vs. 22.4%, p < 0.001), mechanical ventilation use (26.2% vs. 13.8%, p < 0.001), hyperlactatemia (83.9% vs. 55.6%, p < 0.001), and acute kidney injury (28.6% vs. 21.4%, p < 0.001). The number of organ dysfunctions (2.2 ± 1.4 vs. 1.8 ± 1.2, p < 0.001) and the proportion of ICU admissions (40.2% vs. 24.6%, p < 0.001) were higher in patients who died. The hospital length of stay was shorter for those who died (11.0 vs. 14.0 d, p < 0.001). Supplemental eTable 6 (http://links.lww.com/CCM/H525) presents laboratory findings of lactate, creatinine, bilirubin values, and platelet counts.

TABLE 1.

Characteristics of Adult Sepsis Event Patients According to Survival Outcome

	Total (n = 6,872)	Survived (n = 5729)	Died (n = 1143)	p
Age, median (IQR)	65.0 (56.0; 73.0)	64.0 (56.0; 73.0)	66.0 (58.0; 75.0)	< 0.001
Male sex, n (%)	4279 (62.3)	3576 (62.4)	703 (61.5)	0.583
Community-onset sepsis, n (%)	4279 (62.3)	3543 (61.8)	736 (64.4)	0.112
Hospital-onset sepsis, n (%)	2593 (37.7)	2186 (38.2)	407 (35.6)	0.112
Medical department, n (%)	4909 (71.4)	3936 (68.7)	973 (85.1)	< 0.001
Type of organ dysfunction, n (%)
Vasopressor initiation	1757 (25.6)	1285 (22.4)	472 (41.3)	< 0.001
Mechanical ventilation	1091 (15.9)	792 (13.8)	299 (26.2)	< 0.001
Hyperlactatemia	4144 (60.3)	3185 (55.6)	959 (83.9)	< 0.001
Acute kidney injury	1553 (22.6)	1226 (21.4)	327 (28.6)	< 0.001
Hepatic injury	1902 (27.7)	1717 (30.0)	185 (16.2)	< 0.001
Thrombocytopenia	2250 (32.7)	1962 (34.2)	288 (25.2)	< 0.001
Number of organ dysfunction, mean ± sd	1.85 ± 1.21	1.77 ± 1.17	2.2 ± 1.4	< 0.001
ICU admission, n (%)	1870 (27.2)	1411 (24.6)	459 (40.2)	< 0.001
Hospital length of stay after Adult Sepsis Event diagnosis, d, median (IQR)	14.0 (8.0; 25.0)	14.0 (9.0; 25.0)	11.0 (4.0; 22.0)	< 0.001

IQR = interquartile range.

We conducted a multivariate logistic regression analysis to identify factors associated with hospital mortality among ASE patients (Fig. 3). For every 1-year increase in age, the odds of mortality increased by 1% (odds ratio [OR] 1.01, p = 0.003). Sepsis patients admitted to the surgical department had lower mortality than those in the medical department (OR 0.35, p < 0.001). Hospital-onset sepsis was associated with higher mortality compared with community-onset sepsis (OR 1.17, p < 0.038). Among the types of organ dysfunction (eSOFA criteria met), initiation of mechanical ventilation (OR 1.46, p < 0.001), initiation of vasopressors (OR 1.77, p < 0.001), hyperlactatemia (OR 3.18, p < 0.001), and acute kidney injury (OR 1.32, p < 0.001) were associated with higher mortality. Conversely, hepatic injury (OR 0.63, p < 0.001) and thrombocytopenia (OR 0.69, p < 0.001) were associated with lower mortality.

Figure 3.

Forest plot of factors associated with hospital mortality in patients with Adult Sepsis Event in multivariate logistic regression analysis.

In the study, 56.7% of patients met one eSOFA criterion, 20.1% met two criteria, 11.4% met three, 6.9% met four, 3.8% met five, and 1.2% met all six criteria. The mortality rates for these groups were 12.3%, 19.7%, 23.1%, 26.7%, 21.4%, and 36.6%, respectively. The number of organ failures showed a linear association with the mortality rate (Fig. 4). We observed various combinations of organ failure, and Supplemental eFigure 1 (http://links.lww.com/CCM/H525) depicts the number of patients in each group alongside their respective mortality rates. Patients who required vasopressors exhibited hyperlactatemia, suffered acute kidney injury, and used mechanical ventilation had the highest mortality rate at 43.0%.

Figure 4.

Number of electronic health records-based Sequential Organ Failure Assessment criteria met (organ failures) and associated hospital mortality in patients with Adult Sepsis Event (ASE).

Supplemental eTable 7 (http://links.lww.com/CCM/H525) shows the monthly admissions of COVID-19 patients and monthly counts of ASE and ICD-10-coded sepsis among these patients. Among 6182 patients with ASE, 33 were admitted due to COVID-19, among whom 5 had concomitant ICD-10-coded sepsis.

DISCUSSION

In this study, we assessed the diagnostic performance of ASE criteria against the Sepsis-3 criteria using Korean subjects. Following this validation, we examined a cohort of patients admitted over 1 year, evaluating the prevalence, mortality, and characteristics of sepsis as diagnosed by ASE definition. We then compared these findings to sepsis cases coded by ICD-10. Our research is notable as few studies have validated the ASE using in-depth reviews of medical records outside the United States. Additionally, the prevalence and mortality rates of sepsis, based on ASE definitions, were sourced from real-world EHR data. Given that the ASE was developed for sepsis surveillance and quality improvement, our study serves as a pioneering example of its practical application. Furthermore, our findings underscore the feasibility of implementing ASE criteria in actual hospital environments for monitoring sepsis.

The incidence rate of sepsis diagnosed by ASE criteria in our study was 5.4%. This is eight times higher than the incidence rate of sepsis identified by ICD-10 codes. However, the hospital mortality rate for ICD-10-coded sepsis was higher than that of ASE. This disparity suggests that many cases of sepsis went undetected by ICD-10 coding. The underreporting could be due to several reasons: physicians might not have recognized sepsis, they could have been unaware of the revised sepsis diagnostic criteria, or the intricacies of the criteria might have been too complex for prompt bedside diagnosis (3, 16, 17). Furthermore, sepsis that developed during hospital stays often went unrecorded, especially if the initial reasons for admission were not related to sepsis. Physicians typically enter diagnosis codes at admission or discharge, not during the hospitalization. Thus, it appears that primarily the more severe cases of sepsis were identified with the ICD-10 sepsis code. This results in a lower incidence rate, yet a higher in-hospital mortality rate when compared with sepsis pinpointed by ASE criteria. Notably, we observed greater monthly variability in both prevalence and hospital mortality rates for ICD-10-coded sepsis than for ASE cases. This implies that ICD-10 coding for sepsis might not be entirely reliable for epidemiologic purposes and could introduce inaccuracies and bias compared with the ASE criteria. Additionally, determining the exact onset date of sepsis is challenging for patients diagnosed via ICD-10 codes. In the study by Rhee et al (11), the EHR-based sepsis prevalence was 6.0%, with an in-hospital mortality rate of 15.0%. Our results were in line with these findings, showing an incidence rate of 5.6% and a mortality rate of 16.6%. However, our current findings differ markedly from earlier reports in Korea. Specifically, Oh et al (18) cited a sepsis prevalence of 233.6 per 100,000 and a mortality rate of 22.6% in 2016 using ICD-10 codes from claims data. The variance between prior studies and ours suggests that research based on ICD-10 or manual screening of medical records might underestimate sepsis incidence while overestimating its mortality.

In our study, ASE showed robust diagnostic accuracy, displaying a high sensitivity (91.6%), specificity (98.3%), and NPV (99.8%). However, the PPV was moderate at 57.4%, which is lower than the 70.4% reported by Rhee et al (11) in their study. Of the 105 patients with false positives in our review, over half (n = 55) neither had definite nor possible infections. The majority of these patients had undergone blood culture tests and were preemptively treated with antibiotics due to shocks unrelated to septic shock. These shocks included hemorrhagic and cardiogenic shocks, cardiac arrest, and adrenal insufficiency. Some exhibited systemic inflammation due to drugs such as antithymocyte globulin or procedures like transarterial chemoembolization. Patients who had experienced a stroke or showed B symptoms from lymphoma, or had solid tumors and were assessed by physicians as having cancer fever, developed a fever and were treated with antibiotics. However, their medical records did not confirm the presence of any infection. The difference in PPV might also be attributed to our inclusion of hyperlactatemia in the eSOFA criteria. Sixteen cases presented with hyperlactatemia, but their SOFA score did not increase by two points. Prior studies have indicated that excluding lactate from the eSOFA criteria could reduce the prevalence of ASE, decrease sensitivity, and increase PPV. Nevertheless, whether lactate was included or excluded, there was no significant change in the area under the receiver operating characteristic curve (AUROC) for in-hospital mortality, as evidenced by a previous study (14). Because it is standard procedure to measure lactate levels in deteriorating patients at our center, we opted to include lactate in the eSOFA criteria. Some studies propose that the eSOFA criteria possess a lower sensitivity compared with Sepsis-3 due to the exclusion of patients not on mechanical ventilation or those exhibiting mental changes (12, 14). However, our study demonstrated a higher sensitivity with only 13 cases of false negatives. Merely six cases had hypoxemia without the need for mechanical ventilation. We relied solely on the Pao₂/Fio₂ ratio to determine the SOFA score for Sepsis-3, not using the Sao₂/Fio₂ ratio. Consequently, patients with mild hypoxemia who did not undergo arterial blood gas analysis were excluded from both the Sepsis-3 and ASE groups (1). Therefore, ASE demonstrated high sensitivity but low PPV in our research, and our results indicate that the “presumed infection” criteria may have inherent limitations, as cases without any confirmed infection constituted over half of the total false positives. Simply relying on blood culture tests and antibiotic treatments may not effectively discern patients with potential or confirmed infections in a clinical setting. Further modifications and research in this area could enhance the diagnostic accuracy of the ASE in the future.

Like the SOFA score, the eSOFA criteria were not originally developed to predict mortality (19, 20). Nonetheless, multiple studies have indicated that an elevation in the SOFA score can serve as a dependable predictor of sepsis prognosis, even if that was not its initial intent (21–23). Rhee et al (14) found that the eSOFA surpassed the SOFA when assessing the AUROC for in-hospital mortality among patients with presumed infections. The eSOFA criteria, being more concise than the SOFA criteria, have scores that range from 0 to 6. Our data revealed that patients exhibiting failures across all six organs had a 36.6% mortality rate, whereas those with only one organ failure faced a 12.3% mortality rate. Consequently, multiple organ failure, as assessed by the eSOFA criteria, typically suggests a more adverse outcome compared with patients diagnosed with a single organ failure. Factors such as the need for mechanical ventilation or vasopressors, as well as the presence of hyperlactatemia or acute kidney injury, correlated with increased mortality. On the other hand, hepatic injury and thrombocytopenia did not show a significant association with increased mortality in eSOFA-diagnosed sepsis patients. Several studies have investigated the link between SOFA subscores and mortality. Jentzer et al determined that, within the SOFA criteria, the cardiovascular, respiratory, renal, and CNS subscores were distinct predictors of in-hospital mortality. However, the coagulation and liver subscores did not markedly affect in-hospital mortality in a cardiac ICU environment (24). These results align closely with ours.

This study has several limitations. First, the research was conducted at a single tertiary center known for its high severity level among hospitals in Korea. Our center treated a particularly large group of patients with hematologic oncological conditions, solid organ transplantations, and multiple comorbidities. In such a patient demographic, the observed severity and mortality rates of sepsis might be higher compared with a general hospital setting. Additionally, we operate a 24-hour, physician-led rapid response team and routinely monitor lactate levels when patient conditions deteriorate. We observed that lactic acidosis was reported in 60% of ASE cases at our center, but the results might differ significantly in other hospitals. Consequently, the outcomes and characteristics of ASE observed in our study might not be representative of hospitals in Korea. Second, we analyzed variations in monthly prevalence over a 1-year period and compared this to ICD-10 sepsis diagnoses. This timeframe might be insufficient to evaluate the seasonal fluctuations in sepsis epidemiology. Third, although examining hospital mortality, we did not account for variations in patients’ performance or comorbidities within the ASE group. Such baseline performance and comorbidities might have influenced the observed hospital mortality rates. Fourth, this study revealed that the prevalence of ICD-10-coded sepsis in Korea was only 0.7%, which is significantly lower than the U.S. claim-based sepsis prevalence reported by Rhee et al (25). In the United States, the implementation of Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) and the introduction of financial incentives, such as enhanced reimbursement rates, have motivated clinicians and hospitals to be more meticulous in documenting sepsis codes (25, 26). However, in Korea, there are currently no policy-based incentives or disincentives for sepsis coding. The disparity in healthcare policy and coding incentives between the United States. and Korea likely accounts for these differences. The epidemiologic estimates of sepsis based on ICD-10 coding may be underreported in Korea. Fifth, despite the potential for instability in sepsis coding due to the COVID-19 pandemic, the low number of COVID-19 cases and hospitalizations in 2020 led us to believe that our data remains significant, even when considering the impact of COVID-19. In 2020, 82 COVID-19 patients were admitted to our center. Among these, 33 were ASE cases, with 5 also identified as having sepsis according to ICD-10. Due to the relatively small number of COVID-19 hospitalizations, we believe that the pandemic had minimal impact on our study’s results.

CONCLUSIONS

The ASE exhibited high sensitivity and a moderate positive predictive value in comparison to the Sepsis-3 criteria. The prevalence of sepsis, as delineated by ASE, was 5.4% per year, which aligns with U.S. estimates. The prevalence of sepsis by ASE criteria was roughly eight times greater and showed less monthly fluctuation than that defined by the ICD-10 code. Mortality of sepsis based on ASE was lower than that based on the ICD-10 code.