Abstract
Background
Sepsis remains a leading cause of death for burn patients, yet the condition is hard to spot early. Hospitals generally rely on the International Classification of Diseases (ICD) codes for surveillance, but these codes are assigned late and often miss active cases.
Objectives
We developed and validated a scalable, electronic health record (EHR)-based digital phenotype that improves identification of burn-related sepsis compared with ICD codes alone.
Methods
We performed a retrospective cohort study of adult burn inpatients ( n = 1,371) admitted to Loyola University Medical Center (2007–2021). Structured EHR data and clinical notes were extracted from a research database. Sequential Organ Failure Assessment (SOFA) scores were calculated every 4 hours; natural language processing parsed vasopressor doses and culture alerts. Four algorithms were evaluated: (1) ICD codes; (2) SOFA increase ≥2 + broad-spectrum antibiotics; (3) SOFA increase ≥2 + positive blood culture; (4) SOFA increase ≥2 + any positive culture.
Results
ICD coding alone classified 79 encounters (5.8%) as sepsis. EHR-enhanced algorithms identified more cases: 123 via SOFA + antibiotics (9.0%), 21 via SOFA + blood culture (1.5%), and 53 via SOFA + any culture (3.9%). The rule count score (0–4) achieved the highest performance (area under the curve [AUC] 0.92), outperforming ICD codes alone (AUC 0.77).
Conclusion
Our multimodal digital phenotype doubled sepsis detection compared to ICD-based surveillance. The tiered risk assessment approach showed excellent discrimination with increasing positive predictive value as more criteria were met. This phenotype can be implemented using routine EHR data, supporting early warning tools for this high-risk population.
Keywords: digital phenotyping, clinical NLP, EHR data pipelines, clinical decision support (CDS), multimodal data integration, disease surveillance, model evaluation, data pseudonymization & de-identification
Background and Significance
Sepsis, defined as life-threatening organ dysfunction caused by a dysregulated host response to infection, remains a leading cause of morbidity and mortality worldwide. 1 2 The Third International Consensus Definitions (Sepsis-3) introduced standardized criteria that tie sepsis to an acute increase in Sequential Organ Failure Assessment (SOFA) score, a composite measure that evaluates dysfunction across six organ systems: Respiration, coagulation, liver, cardiovascular, central nervous system, and renal function, with each scored from 0 to 4 (total range 0–24). Under the Sepsis-3 framework, an acute SOFA increase of ≥2 points from baseline signifies organ dysfunction consistent with sepsis. 1 3 While this standardization has improved diagnostic uniformity, sepsis remains a heterogeneous syndrome lacking a definitive gold standard. 4 This challenge is particularly pronounced in specialized populations with unique physiologic responses, notably burn patients. 4 5
Burn patients present distinctive diagnostic complexities. Severe thermal injury causes substantial skin barrier loss and triggers a profound systemic inflammatory response that closely mimics sepsis. 2 5 These patients face heightened susceptibility to invasive infections due to compromised integumentary defenses, extensive wound surfaces, and intensive interventions required during care. 2 5 The pathophysiology of burn sepsis differs significantly from other critically ill populations, with alternating hyperinflammatory and immunosuppressive phases that can either mask sepsis or generate false positives. 2 6 Despite advances in acute burn care, mortality among burn patients who develop sepsis remains approximately one in three—substantially higher than general critical care populations. 6 7
No widely accepted burn-specific sepsis definition exists. Various specialized diagnostic criteria have been proposed, yet none conclusively outperform general sepsis definitions in this population. 8 Sepsis-3 criteria identify more burn sepsis cases than legacy burn criteria, but with reduced specificity, frequently flagging patients without true sepsis. 8 Consequently, burn specialists often rely on expert judgment rather than standardized criteria, introducing variability in care and research. 8 Furthermore, retrospective identification using administrative coding (International Classification of Diseases [ICD] codes) yields suboptimal sensitivity, capturing only about 35% of sepsis cases in general populations and likely even fewer in burn patients with overlapping inflammatory signs. 9 10 11 Effective clinical informatics research requires electronic health record (EHR) data that capture the complexity of clinical decision-making, and recent approaches combining clinical knowledge with machine learning have shown promise for identifying high-need patients. 12 13
The limitations of current approaches create a clear gap: Burn centers lack reliable tools for both real-time sepsis surveillance and accurate retrospective case identification. This gap affects quality improvement, research, and ultimately patient outcomes. Recent advances in computational methods and EHR data science offer promising solutions, but most sepsis detection algorithms were developed in general critical care populations, systematically excluding burn patients. 6 14 The poor performance of generic sepsis algorithms in burn patients reflects the unique pathophysiology of thermal injury. Generic sepsis algorithms perform poorly in specialized populations; for example, a widely deployed commercial sepsis prediction model demonstrated disappointing external validation (area under the curve [AUC] ∼0.63) and missed two-thirds of clinician-diagnosed cases. 11
Our study addresses this critical need by developing a novel EHR-based digital phenotype specifically designed for burn sepsis. The innovation of our approach lies in the integration of multiple data modalities: Structured physiological measurements, sequential organ dysfunction scores, antimicrobial administration patterns, and microbiological findings, supplemented with information extracted from unstructured clinical notes via natural language processing. This multimodal approach is particularly valuable in burn care, where subtle clinical indicators are often documented in notes rather than structured fields. 1 15 By capturing these complementary signals of infection and organ dysfunction, our phenotype aims to overcome the limitations of both ICD-based surveillance and generic sepsis algorithms.
The potential effect of this work extends beyond improved surveillance. A reliable burn sepsis digital phenotype could support more accurate real-time detection, facilitate consistent research cohort identification, and potentially serve as the foundation for early-warning clinical decision support tools tailored to this high-risk population.
Objectives
The goals of this study were to (1) develop an automated sepsis phenotype for burn in-patients that merges (a) a ≥2-point rise in SOFA score with (b) evidence of infection from various sources; (2) develop and validate a cumulative risk score (0–4) based on the number of sepsis criteria met; (3) compare its case-capture rate with ICD-based surveillance in a 15-year retrospective burn cohort; and (4) assess the discriminatory ability of this tiered approach versus individual definitions.
Methods
Study Setting
This study was approved by the Loyola University Chicago Health Sciences Campus Institutional Review Board (IRB217770) and was designed as a retrospective cohort analysis conducted at the Burn and Shock Trauma Institute, a specialized tertiary care center within Loyola University Medical Center (LUMC) in Maywood, IL. The center serves a diverse patient population, including a high volume of patients requiring acute and intensive burn care. We leveraged data from the Clinical Research Database (CRDB), a comprehensive, longitudinal repository containing over 15 years (2007–2021) of clinical data on more than 2 million patients. The CRDB integrates both structured and unstructured data elements from the EHR system, encompassing changes in clinical practice patterns and coding conventions over time.
Cohort Identification and Data Sources
From the CRDB, we initially extracted a broad cohort of 9,235 inpatient encounters, representing 8,842 unique patients. We included adult patients (≥18 years old) admitted to the burn unit at LUMC with a length of stay between 5 and 30 days, as this window allowed adequate time to capture the development of sepsis following a burn injury. We excluded patients who were readmitted within 30 days of their index admission to reduce confounding related to recurrent hospitalizations.
Among the patients meeting these criteria, we identified those with ICD-9 or ICD-10 codes indicative of burn injuries. This identification step ensured that the final cohort was focused on individuals at risk for developing burn sepsis. After applying burn-specific ICD codes to the 3,210 patients meeting inclusion criteria, 1,371 patients constituted our final analysis cohort. The cohort selection process is summarized in Fig. 1 , showing progression from initial 9,235 encounters to final 1,371 burn patients analyzed. The EHR data available included demographic information, medication administration records, laboratory tests, microbiology culture results, clinical notes, and flowsheet data documenting vital signs and interventions. All identifying information was pseudonymized prior to analysis.
Fig. 1.
Patient flow diagram. ICD, International Classification of Diseases; SOFA, Sequential Organ Failure Assessment.
Preprocessing and Data Standardization
The raw data underwent extensive preprocessing and pseudonymization to ensure patient privacy while maintaining data integrity. Medical record numbers and hospital account IDs were replaced with unique pseudonymized identifiers using a consistent mapping system. All temporal data, including admission dates, discharge dates, and laboratory timestamps, were offset by a random time interval to further protect patient privacy while preserving the relative timing between events. The final cleaned dataset integrated multiple data sources, including patient demographics, laboratory results, medication administration records, clinical notes, diagnostic codes, and flowsheet data. All date–time fields were standardized to a consistent format, and the relationships between different data tables were preserved through the pseudonymized identifiers.
Sepsis Detection Algorithm
We developed and compared four sepsis detection algorithms, hereafter referred to by standardized labels: Algorithm 01 (ICD), based on ICD diagnostic codes alone; Algorithm 02 (SOFA + ABx), combining a ≥2-point SOFA increase with broad-spectrum antibiotic initiation; Algorithm 03 (SOFA + BCx), combining a ≥2-point SOFA increase with a positive blood culture; and Algorithm 04 (SOFA + Cx), combining a ≥2-point SOFA increase with any positive culture. We also evaluated a combined “any rule” approach (positive if ≥1 algorithms were met) and a cumulative “rule count” score (0–4), defined as the total number of the four algorithms met by each patient.
International Classification of Diseases Diagnostic Codes
The first algorithm (Algorithm 01: ICD) identified cases of potential burn sepsis using ICD-9 and ICD-10 diagnostic codes. Patients were classified as burn patients if their primary or secondary diagnoses included burn-related codes ( Tables 1 and 2 ). Sepsis determination relied on codes corresponding to sepsis, severe sepsis, or septic shock ( Tables 3 and 4 ). While ICD codes are widely used for administrative and epidemiological purposes, they often lack the temporal resolution and clinical nuance necessary for precise sepsis phenotyping. This ICD-only approach served as a comparator to assess the value added by our more comprehensive methods.
Table 1. ICD-10 codes for burn-related illnesses.
| ICD-10 code | Description |
|---|---|
| T20 | Burn and corrosion of head, face, and neck |
| T21 | Burn and corrosion of trunk |
| T22 | Burn and corrosion of shoulder and upper limb, except wrist and hand |
| T23 | Burn and corrosion of wrist and hand |
| T24 | Burn and corrosion of lower limb, except ankle and foot |
| T25 | Burn and corrosion of ankle and foot |
| T30 | Burn and corrosion, body region unspecified |
| T31 | Burns classified according to extent of body surface involved |
| T32 | Corrosions classified according to extent of body surface involved |
Abbreviation: ICD-10, International Classification of Diseases, Tenth Edition.
Table 2. ICD-9 codes for burn-related illnesses.
| ICD-9 code | Description |
|---|---|
| 940 | Burn confined to eye and adnexa |
| 941 | Burn of face head and neck |
| 942 | Burn of trunk |
| 943 | Burn of upper limb except wrist and hand |
| 944 | Burn of wrist(s) and hand(s) |
| 945 | Burn of lower limb(s) |
| 946 | Burns of multiple specified sites |
| 947 | Burn of internal organs |
| 948 | Burns classified according to extent of body surface involved |
| 949 | Burn unspecified site |
Abbreviation: ICD-9, International Classification of Diseases, Ninth Edition.
Table 3. ICD-10 codes for sepsis-related illnesses.
| ICD code | Description |
|---|---|
| A02.1 | Salmonella sepsis |
| A40.8 | Other streptococcal sepsis |
| A40.9 | Streptococcal sepsis, unspecified |
| A41.50 | Gram-negative sepsis, unspecified |
| A41.59 | Other gram-negative sepsis |
| A41.89 | Other specified sepsis |
| B37.7 | Candidal sepsis |
| O85 | Puerperal sepsis |
| R65.20 | Severe sepsis without septic shock |
| R65.21 | Severe sepsis with septic shock |
| T81.12XA | Postprocedural septic shock, initial encounter |
Abbreviation: ICD, International Classification of Diseases.
Table 4. ICD-9 codes for sepsis-related illnesses.
| ICD code | Description |
|---|---|
| 038.0 | Streptococcal septicemia |
| 038.10 | Staphylococcal septicemia, unspecified |
| 038.11 | Methicillin-susceptible Staphylococcus aureus septicemia |
| 038.12 | Methicillin-resistant Staphylococcus aureus septicemia |
| 038.19 | Other staphylococcal septicemia |
| 038.2 | Pneumococcal septicemia ( Streptococcus pneumoniae septicemia) |
| 038.49 | Other septicemia due to gram-negative organisms |
| 038.8 | Other specified septicemias |
| 038.9 | Unspecified septicemia |
| 670.24 | Puerperal sepsis, postpartum condition or complication |
| 995.92 | Severe sepsis |
| 998.02 | Postoperative shock, septic |
Abbreviation: ICD, International Classification of Diseases.
It is important to note that ICD codes represent retrospective billing classifications assigned by medical coders, which may be incomplete or delayed. In contrast, our gold standard—clinician chart review—represents a comprehensive clinical assessment considering laboratory trends, vital sign trajectories, microbiological timing, clinical documentation, and treatment response. The chart reviewers determined sepsis presence based on the totality of medical record evidence, not merely billing code assignment.
Electronic Health Record-Based Algorithms
To improve detection and specificity beyond ICD-based criteria, we developed a suite of EHR-based definitions that leveraged both structured and unstructured clinical data. These algorithms integrated physiological measures via the SOFA score, evidence of infection (positive microbiological results), and treatment patterns (e.g., antibiotic administration, vasopressors). In addition, we applied Natural Language Processing (NLP) techniques within certain data extraction steps—specifically, to identify vasopressor information from free-text medication administration records and to detect positive culture alerts in clinical notes—enhancing the completeness and accuracy of our phenotype.
Sequential Organ Failure Assessment Score Calculation
The SOFA score was central to our EHR-based approach, as it quantitatively reflects organ dysfunction and is widely used in sepsis research. The SOFA score evaluates six organ systems—respiration, coagulation, liver, cardiovascular, central nervous system, and renal—each scored from 0 to 4 points ( Supplementary Table S1 [available in the online version only]). 1 For each of the six organ systems evaluated, we developed specific data processing pipelines to ensure accurate score calculation. Raw laboratory and clinical data were extracted from multiple database tables and preprocessed for analysis. For laboratory values (PaO 2 , bilirubin, platelet count, and creatinine), we standardized measurement units and timestamps. Vital signs (mean arterial pressure, Glasgow Coma Scale) were extracted from the flowsheet data with their corresponding timestamp information. 11
A particularly complex aspect of the data processing involved the calculation of vasopressor doses. We implemented an NLP approach using regular expressions to extract medication names and concentrations from the medication administration records. The algorithm identified key information to parse medication names and concentrations. All vasopressor doses were converted to norepinephrine-equivalent doses (μg/kg/min) using established conversion factors (epinephrine 1:1, norepinephrine 1:1, dopamine 15:0.1, phenylephrine 10:1). 16
The conversion calculation accounted for drug concentration, infusion rate, and a standardized patient weight of 70 kg.
Medication administration data in our EHR exist in a semistructured format where drug name, dose, concentration, and infusion rate are stored as concatenated text strings rather than discrete structured fields. We developed regular expression patterns to parse these elements, extracting vasopressor drug names (norepinephrine, epinephrine, dopamine, phenylephrine), concentration values, and infusion rates. The extraction patterns were manually validated against a subset of medication records to confirm accurate parsing. Similarly, positive culture results were identified by detecting standardized “ALERT-ABNORMAL” notification patterns in the laboratory results.
While the specific regular expression patterns are institution-specific, we provide representative examples in Supplementary Table S2 (available in the online version only) to illustrate our extraction approach. For vasopressor parsing, the pattern captured drug name, concentration (mg/mL), and infusion rate (mL/h) from concatenated medication strings of the general form [Drug Name] [Concentration] mg/[Volume] mL [Infusion Rate] mL/h. For culture alerts, the pattern identified records containing the standardized ALERT-ABNORMAL flag in laboratory result notes. Institutions implementing this phenotype would need to adapt these patterns to their local EHR documentation format.
For each time point in a patient's stay, we developed a function to retrieve the most relevant clinical values by finding the closest recorded measurement to the time of interest. The SOFA score was then calculated according to standard criteria ( Supplementary Table S1 [available in the online version only]) 1 : Respiratory system (PaO 2 /FiO 2 ratio), cardiovascular system (mean arterial pressure and vasopressor requirements), hepatic system (bilirubin), coagulation (platelets), renal system (creatinine), and neurological system (Glasgow Coma Scale). Each system was scored from 0 to 4 points based on the severity of dysfunction. The complete SOFA score calculation algorithm, including organ system thresholds, EHR data sources, and vasopressor conversion formulas, is provided in Supplementary Table S1 (available in the online version only).
To identify acute organ dysfunction, we implemented an algorithm that calculated SOFA scores at 4-hour intervals throughout each patient's hospital stay. For patients stays shorter than 4 hours, scores were calculated at admission and discharge. The algorithm tracked the minimum SOFA score and systematically searched for any subsequent increase of ≥2 points, which defined significant organ dysfunction. When such an increase was identified, both the timestamp of the minimum SOFA score and the timestamp of the first ≥2-point increase were recorded. This approach allowed us to precisely identify both the timing of the initial organ dysfunction and the subsequent deterioration.
The entire analysis pipeline was implemented using Python within a Jupyter Notebook environment, employing the Pandas library for data manipulation and leveraging Python's built-in date–time functions for temporal analysis. The code was optimized for computational efficiency, particularly in handling large datasets with multiple time points and complex temporal relationships between different clinical measurements. 17 18 19
Combined Sequential Organ Failure Assessment and Culture-Based Algorithms
We evaluated three EHR-based phenotypes that combined SOFA-defined organ dysfunction with evidence of infection.
SOFA + antibiotics (Algorithm 02: +ABx): Patients were classified as having burn sepsis if they had a ≥2-point increase in their SOFA score in conjunction with the initiation or escalation of broad-spectrum antibiotic therapy ( Table 5 ).
SOFA + positive blood culture (Algorithm 03: +BCx): This definition required both a significant SOFA score increase and the presence of at least one positive blood culture result. Positive results identified through NLP-detected “ALERT-ABNORMAL” notes were included.
SOFA + any positive culture (Algorithm 04: +Cx): Recognizing that burn-related infections may arise from multiple sources, this more inclusive definition allowed any positive culture type—blood or otherwise—to qualify as evidence of infection when combined with a ≥2 point SOFA increase ( Fig. 2 ).
Table 5. List of broad-spectrum antibiotics by generic and trade name.
| Drug class | Generic name | Trade name |
|---|---|---|
| Antifungals | Fluconazole | Diflucan |
| Micafungin | Mycamine | |
| Carbapenems | Meropenem | Merrem |
| Imipenem–Cilastatin | Primaxin | |
| Cephalosporins (first generation) | Cefazolin | Ancef |
| Cefotaxime | Claforan | |
| Cephalosporins (third generation) | Ceftazidime | Fortaz |
| Ceftriaxone | Rocephin | |
| Cephalosporins (fourth generation) | Cefepime | Maxipime |
| Extended-spectrum penicillins | Piperacillin–Tazobactam | Zosyn |
| Fluoroquinolones | Ciprofloxacin | Cipro |
| Glycopeptides | Vancomycin | Vancocin |
| Lincosamides | Clindamycin | Cleocin |
| Monobactams | Aztreonam | Azactam |
| Nitroimidazoles | Metronidazole | Flagyl |
| Penicillins | Ampicillin | Principen, Omnipen |
| Ampicillin–Sulbactam | Unasyn |
Fig. 2.
Sepsis case definition algorithm flowchart 1. ICD, International Classification of Diseases; SOFA, Sequential Organ Failure Assessment.
Stratified Sampling and Chart Review
To validate our EHR-based definitions of burn sepsis, we conducted a physician-led chart review using stratified sampling to account for variability in burn sepsis prevalence across different patient groups. We used proportional allocation-stratified sampling, in which patients were stratified by the number of the four sepsis detection algorithms (Algorithms 01–04) they met (0, 1, 2, 3, or 4). Sample sizes within each stratum were allocated proportionally to stratum size to ensure that the validation cohort reflected the distribution of algorithmic classifications in the full cohort. Given the variability in reported sepsis prevalence among burn patients (approximately 10–30%), we conservatively estimated a prevalence of 10% for sample size calculations to ensure adequate power and precision.
Using a margin of error of ±5% at a 95% confidence level, we calculated the required total sample size of patients for chart review. We applied proportional allocation sampling, resulting in 122 patients who met no criteria, 12 who met one criterion, 4 who met two criteria, 3 who met three criteria, and 1 patient who met all four criteria.
A tiered chart review process was implemented, with clinician reviewer consensus serving as the gold standard for sepsis determination. The 142 sampled records were randomly divided into four groups of approximately equal size (36, 35, 36, and 35 records). Each group was independently reviewed by two of four physician reviewers using a structured abstraction form, with reviewers blinded to algorithmic classifications. For each record, if both primary reviewers agreed on sepsis status, that classification was accepted. In cases of disagreement, a third physician served as the consensus adjudicator. All data capture was conducted through REDCap. 20
Validation and Statistical Analysis
Algorithm performance was assessed using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Clinician chart review of sepsis identification served as the reference standard for comparison. McNemar's test evaluated differences in case detection between ICD and EHR-based approaches. Interrater reliability for NLP-derived keywords was assessed via Cohen's κ on a 142-note subsample annotated by two clinicians. Multivariate logistic regression analyzed associations between sepsis definitions and clinical outcomes (e.g., mortality, ventilator days), adjusting for age, burn severity, and comorbidities. Statistical significance was set at p < 0.05.
To integrate the four algorithms into a single summary measure, we defined a “rule count” score ranging from 0 to 4, equal to the total number of algorithms (Algorithms 01–04) that classified a patient as having sepsis. A rule count of 0 indicates that no algorithm detected sepsis, whereas a rule count of 4 indicates that all four algorithms concurrently identified the patient as septic. We additionally evaluated a binary “any rule” threshold, defined as a rule count ≥1 (i.e., the union of all four algorithms). Performance of each “individual rule”—that is, each algorithm evaluated independently—was compared against the combined approaches. The rule count was treated as an ordinal variable for receiver operating characteristic (ROC) analysis to assess the discriminatory value of cumulative evidence.
Results
Cohort Characteristics
From the initial cohort of 9,235 inpatient encounters, 3,210 met the inclusion criteria. After applying burn-specific ICD codes, 1,371 patients were included in the final analysis. We identified 1,198 patients without any sepsis detection criteria met and 173 patients who met at least one of the four criteria, out of which 110 met one criterion, 32 met two criteria, 22 met three criteria, and 9 met all four criteria ( Fig. 1 ).
Case Counts by Algorithm (Full Cohort, n = 1,371)
When applied to the full analysis cohort, Algorithm 01 (ICD) classified 79 patients (5.8%) as having sepsis. The EHR-based algorithms identified additional cases: Algorithm 02 (SOFA + ABx) flagged 123 patients (9.0%), Algorithm 03 (SOFA + BCx) flagged 21 (1.5%), and Algorithm 04 (SOFA + Cx) flagged 53 (3.9%). Under the “any rule” threshold (rule count ≥1), 173 patients (12.6%) met at least one criterion, representing a 119% increase in case detection compared with ICD coding alone.
The overlap among the four algorithms is shown in Supplementary Fig. S1 (available in the online version only). Of the 173 patients meeting at least one criterion, 110 (63.6%) met exactly one, 32 (18.5%) met two, 22 (12.7%) met three, and 9 (5.2%) met all four.
To validate these algorithmic classifications, 142 patients were selected through stratified sampling for clinician chart review (see “Stratified Sampling and Chart Review” section). Of these 142 patients, 9 (6.3%) were classified as sepsis-positive by the reviewing clinicians, serving as the reference standard. The confusion matrices for each algorithm against this gold standard are presented in Table 6 . The overlap between the four phenotyping algorithms is illustrated in Supplementary Fig. S1 (available in the online version only). Among the 173 patients meeting at least one criterion, 110 met exactly one criterion, while 63 met two or more criteria, demonstrating substantial but incomplete overlap between detection methods.
Table 6. Confusion matrices for sepsis phenotyping algorithms versus clinician chart review.
| ICD codes (Algorithm 01) | |||
|---|---|---|---|
| Sepsis + | Sepsis − | Total | |
| Algorithm positive | 5 | 3 | 8 |
| Algorithm negative | 4 | 130 | 134 |
| Total | 9 | 133 | 142 |
| SOFA + ABx (Algorithm 02) | |||
| Sepsis + | Sepsis − | Total | |
| Algorithm positive | 5 | 11 | 16 |
| Algorithm negative | 4 | 122 | 126 |
| Total | 9 | 133 | 142 |
| SOFA + BCx (Algorithm 03) | |||
| Sepsis + | Sepsis − | Total | |
| Algorithm positive | 2 | 1 | 3 |
| Algorithm negative | 7 | 132 | 139 |
| Total | 9 | 133 | 142 |
| SOFA + Cx (Algorithm 04) | |||
| Sepsis + | Sepsis − | Total | |
| Algorithm positive | 5 | 1 | 6 |
| Algorithm negative | 4 | 132 | 136 |
| Total | 9 | 133 | 142 |
| Any rule (≥1 criterion met) | |||
| Sepsis + | Sepsis − | Total | |
| Algorithm positive | 8 | 12 | 20 |
| Algorithm negative | 1 | 121 | 122 |
| Total | 9 | 133 | 142 |
Abbreviations: ICD, International Classification of Diseases; SOFA, Sequential Organ Failure Assessment.
Notes: Each panel presents the confusion matrix for one phenotyping algorithm (Algorithms 01–04 and the combined “any rule” threshold) versus the clinician chart-review reference standard.
Columns: Chart-review-confirmed sepsis (Sepsis + , n = 9) versus non-sepsis (Sepsis − , n = 133).
Rows: Classification by the row's algorithm (Algorithm Positive = algorithm flagged the patient as having sepsis). Cell values are patient counts. N = 142 stratified-sampled validation cohort drawn from the 1,371-patient full cohort (see “Stratified Sampling and Chart Review” section for sampling design).
Performance of Sepsis Identification Algorithms
The rule count score (0–4) achieved the highest discriminatory performance with an AUC of 0.92, significantly outperforming any individual algorithm (AUC range: 0.61–0.77) and Algorithm 01 (ICD) alone (AUC: 0.77). The rule count demonstrated a clear gradation of sepsis probability: The PPV increased systematically with higher scores—25% (3/12) for patients meeting one criterion, 50% (2/4) for two criteria, 67% (2/3) for three criteria, and 100% (1/1) for all four criteria.
When using the “any rule” threshold (rule count ≥1), the combined approach achieved 89% sensitivity and 91% specificity against clinician chart review, with an NPV of 0.99, indicating excellent rule-out capability.
The individual algorithms showed varying performance characteristics ( Table 7 ). Algorithm 01 (ICD) demonstrated high specificity (0.98) but limited sensitivity (0.56). Algorithm 03 (SOFA + BCx) achieved the highest specificity (0.99) but the lowest sensitivity (0.22), reflecting its stringent requirement for a positive blood culture. Algorithm 04 (SOFA + Cx) maintained both high specificity (0.99) and the highest PPV (0.83) among individual algorithms, with moderate sensitivity (0.56). Algorithm 02 (SOFA + ABx) showed balanced performance with sensitivity of 0.56 and specificity of 0.92, though its PPV was lower (0.31), likely reflecting the high baseline rate of empiric antibiotic use in burn patients. The combined “any rule” approach substantially increased sensitivity to 0.89 while maintaining good specificity (0.91), achieving the best overall balance for screening purposes with an NPV of 0.99.
Table 7. Performance metrics for each individual definition and the combined approach (any rule).
| Algorithm | Sensitivity | Specificity | PPV | NPV | F1 |
|---|---|---|---|---|---|
| ICD | 0.56 | 0.98 | 0.63 | 0.97 | 0.59 |
| SOFA + ABx | 0.56 | 0.92 | 0.31 | 0.97 | 0.40 |
| SOFA + BCx | 0.22 | 0.99 | 0.67 | 0.95 | 0.33 |
| SOFA + Cx | 0.56 | 0.99 | 0.83 | 0.97 | 0.67 |
| Any rule (union) | 0.89 | 0.91 | 0.40 | 0.99 | 0.55 |
Abbreviations: ICD, International Classification of Diseases; NPV, negative predictive value; PPV, positive predictive value; SOFA, Sequential Organ Failure Assessment.
When evaluating algorithm performance by the number of criteria met, we observed that cases meeting multiple criteria were more likely to be confirmed as sepsis by clinician review. Specifically,
Among cases meeting only one criterion, 25% (3/12) were clinically confirmed as sepsis.
Among cases meeting two criteria, 50% (2/4) were confirmed.
Among cases meeting three criteria, 67% (2/3) were confirmed.
The single case meeting all four criteria was confirmed as sepsis.
This progressive increase in PPV with an increasing number of criteria met supports the value of a tiered risk assessment approach.
Receiver Operating Characteristic Analysis
The continuous rule count (0–4) achieved the highest AUC of 0.92, demonstrating excellent discriminatory ability. This significantly outperformed any individual rule (AUC range: 0.61–0.77) and traditional ICD coding (AUC: 0.77). The SOFA + culture algorithm demonstrated the next best discrimination (AUC: 0.77), followed by SOFA + antibiotics (AUC: 0.74) and SOFA + blood culture (AUC: 0.61; Fig. 3 ).
Fig. 3.
Overlaid ROC curves for each algorithm (ICD, culture, blood culture, antibiotics) plus the continuous “number of rules met” score (0–4), with AUC annotations. AUC, area under the curve; ICD, International Classification of Diseases; ROC, receiver operating characteristic.
The superior performance of the rule count metric suggests that considering the cumulative weight of evidence across multiple sepsis indicators provides more reliable identification than any single definition alone, supporting a “tiered alert” approach to sepsis detection in burn patients.
Discussion
Principal Findings
Our study demonstrates that a multidimensional, EHR-based digital phenotype for burn sepsis substantially outperforms traditional ICD-based surveillance. The key findings include (1) our tiered approach achieved excellent discrimination (AUC 0.92) for burn sepsis identification compared with expert clinician review; (2) the digital phenotype increased case detection by 119% compared to ICD codes alone (12.6% vs. 5.8% prevalence); and (3) cases identified by multiple detection methods had progressively higher likelihood of representing true sepsis, with PPVs ranging from 25% for a single criterion to 100% for all four criteria.
This graduated approach addresses a fundamental challenge in burn sepsis detection—the complex, often subtle presentation that frequently confounds single-threshold detection methods. By integrating multiple streams of clinical evidence (physiologic deterioration, antimicrobial treatment, microbiology findings), our phenotype captures the multifaceted nature of sepsis while maintaining high specificity.
The choice between our combined approach and ICD-based surveillance depends critically on the intended clinical application. While Table 7 shows that ICD codes achieve a higher F1 score (0.59 vs. 0.55) and PPV (0.63 vs. 0.40) than the “any rule” approach, these metrics prioritize different clinical goals. For screening and surveillance applications, where missing sepsis cases carry severe consequences, including delayed treatment and increased mortality, the “any rule” approach's superior sensitivity (0.89 vs. 0.56) and excellent NPV (0.99) make it the preferred choice. The approach detects nearly twice as many true sepsis cases while maintaining high specificity (0.91). Conversely, for resource-intensive interventions where false positives carry significant costs, the higher PPV of individual algorithms or higher rule counts may be more appropriate. Our tiered rule count approach (AUC 0.92) provides a framework for matching clinical response intensity to the strength of evidence, with PPV increasing from 25% for one criterion to 100% for all four criteria.
Clinical Implications
Our findings have several important practical applications: First, the progressive increase in sepsis likelihood with higher rule counts (25% with one criterion to 100% with four) naturally supports a tiered clinical response system. This approach enables graduated alerting where increasing evidence triggers progressively more urgent responses, enhancing monitoring for score 1, nursing assessment for score 2, provider notification for score 3, and rapid intervention for score 4—effectively balancing sensitivity with alert fatigue concerns.
Second, our digital phenotype significantly enhances surveillance capabilities. Unlike retrospective ICD codes that miss over half of sepsis cases, our algorithm can be executed automatically using standard EHR data, supporting both quality improvement initiatives and potentially earlier clinical intervention.
Third, by establishing a reproducible, validated phenotype for burn sepsis, our approach could standardize case definitions across research studies and registries, addressing a persistent challenge in burn research where varying definitions have complicated cross-study comparisons.
Limitations
The challenges of applying externally developed phenotyping algorithms to local EHR data are well-documented; external validation remains essential for establishing generalizability. Our phenotype could potentially support clinical decision support tools, similar to disease phenotype detection approaches implemented in other academic settings. 21 22 23 In addition, our study has several limitations that should be acknowledged. First, it represents a single-center experience, potentially limiting generalizability to centers with different patient populations or documentation practices. 24 Besides, our validation relied on a moderate sample size ( n = 142) for chart review, which, while statistically adequate, may limit precision in subgroup analyses. Possible misclassification of borderline cases remains a concern, as sepsis diagnosis lacks a perfect gold standard even with expert review. Fourth, as with any EHR-based study, we relied on data collected during routine care, which may be missing non-randomly, particularly for patients with less severe presentations or shorter stays. Finally, without external validation at another burn center, we cannot definitively assess how our phenotype would perform across different EHR implementations and clinical practices.
Conclusion
The burn-specific sepsis digital phenotype developed in this study more than doubles case detection compared to traditional ICD-based surveillance while maintaining excellent specificity. By combining organ dysfunction metrics with evidence of infection from multiple sources, our approach better captures the complex presentations of sepsis in burn patients. The progressive increase in PPV with increasing number of criteria met supports a tiered approach to sepsis alerts that can balance sensitivity with clinical workflow considerations.
Conflict of Interest The authors declare that they have no conflict of interest.
Contributors' Statement N.D.S.: conceptualization, funding acquisition, investigation, project administration, resources, supervision. L.L.: formal analysis, software, validation, visualization, writing—original draft, writing—review and editing. A.A.P.: conceptualization, data curation, methodology, resources. J.C.K.: conceptualization, data curation, resources, validation.
Protection of Human and Animal Subjects
The study was approved and reviewed by the Loyola University Chicago Health Sciences Campus Institutional Review Board (approval no.: IRB217770). The data were accessed, processed, and analyzed in a secure research environment. All patient information was pseudonymized before analysis, ensuring compliance with the Health Insurance Portability and Accountability Act (HIPAA) and institutional policies. Research staff involved in data handling and analysis received training in data security and confidentiality measures, and only aggregated results were reported.
Clinical Relevance Statement.
Automating a burn-specific sepsis phenotype within the EHR more than doubles case detection compared with ICD-code surveillance. The tiered approach enables graduated clinical responses based on the number of criteria met, balancing sensitivity with alert fatigue concerns.
Multiple-Choice Questions
-
Which detection strategy showed the highest overall discrimination (AUC) against clinician chart review in this study?
ICD codes alone
Cumulative rule-count score (0–4 rules met)
SOFA + broad-spectrum antibiotics
SOFA + positive blood culture
Correct Answer: The correct answer is option b. The rule-count score achieved an AUC = 0.92, surpassing all individual definitions.
-
When a threshold of “ ≥ 1 rule met” was used (the union of all four criteria), what sensitivity was achieved during validation?
0.22
0.89
0.56
0.74
Correct Answer: The correct answer is option b. The “any-rule” approach detected eight out of nine true-positive cases, giving 89% sensitivity.
Supplementary Material
References
- 1.Singer M, Deutschman C S, Seymour C W et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3) JAMA. 2016;315(08):801–810. doi: 10.1001/jama.2016.0287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Boehm D, Menke H. Sepsis in burns—lessons learnt from developments in the management of septic shock. Medicina (Kaunas) 2021;58(01):26–26. doi: 10.3390/medicina58010026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Evans L. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2021. Crit Care Med. 2021;49(11):1063–1143. doi: 10.1097/CCM.0000000000005337. [DOI] [PubMed] [Google Scholar]
- 4.Shappell C N, Klompas M, Rhee C.Surveillance strategies for tracking sepsis incidence and outcomes J Infect Dis 2020222(2 suppl 2):S74–S83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dvorak J E, Ladhani H A, Claridge J A. Review of sepsis in burn patients: in 2020. Surg Infect (Larchmt) 2021;22(01):37–43. doi: 10.1089/sur.2020.367. [DOI] [PubMed] [Google Scholar]
- 6.Zhang P, Zou B, Liou Y C, Huang C. The pathogenesis and diagnosis of sepsis post burn injury. Burns Trauma. 2021;9:tkaa047. doi: 10.1093/burnst/tkaa047. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Belba M K, Petrela E Y, Belba A G. Epidemiology and outcome analysis of sepsis and organ dysfunction/failure after burns. Burns. 2017;43(06):1335–1347. doi: 10.1016/j.burns.2017.02.017. [DOI] [PubMed] [Google Scholar]
- 8.Yoon J, Kym D, Hur J et al. Comparative usefulness of sepsis-3, burn sepsis, and conventional sepsis criteria in major burns. Crit Care Med. 2018;46(07):e656–e662. doi: 10.1097/CCM.0000000000003144. [DOI] [PubMed] [Google Scholar]
- 9.Liu B, Hadzi-Tosev M, Liu Y et al. Accuracy of International Classification of Diseases, 10th Revision codes for identifying sepsis: a systematic review and meta-analysis. Crit Care Explor. 2022;4(11):e0788. doi: 10.1097/CCE.0000000000000788. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Moor M, Rieck B, Horn M, Jutzeler C R, Borgwardt K. Early prediction of sepsis in the ICU using machine learning: A systematic review. Front Med (Lausanne) 2021;8:607952–607952. doi: 10.3389/fmed.2021.607952. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wong A, Otles E, Donnelly J P et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(08):1065–1070. doi: 10.1001/jamainternmed.2021.2626. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kennell T I, Jr, Willig J H, Cimino J J. Clinical informatics researcher's desiderata for the data content of the next generation electronic health record. Appl Clin Inform. 2017;8(04):1159–1172. doi: 10.4338/ACI-2017-06-R-0101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Hewner S, Smith E, Sullivan S S. Identifying high-need primary care patients using nursing knowledge and machine learning methods. Appl Clin Inform. 2023;14(03):408–417. doi: 10.1055/a-2048-7343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Al-Sultani Z, Inglis T J, McFadden B, Thomas E, Reynolds M. Sepsis in silico: definition, development and application of an electronic phenotype for sepsis. J Med Microbiol. 2025;74(03):1986. doi: 10.1099/jmm.0.001986. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Greenhalgh D G. Sepsis in the burn patient: a different problem than sepsis in the general population. Burns Trauma. 2017;5:23–23. doi: 10.1186/s41038-017-0089-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.ATHOS-3 Investigators . Khanna A, English S W, Wang X S et al. Angiotensin II for the treatment of vasodilatory shock. N Engl J Med. 2017;377(05):419–430. doi: 10.1056/NEJMoa1704154. [DOI] [PubMed] [Google Scholar]
- 17.Python Software Foundation. The Python Language Reference. Python Software Foundation. Accessed 9 June 2026 at:https://www.python.org/
- 18.Kluyver T, Ragan-Kelley B, Pérez Fet al. Jupyter Notebooks—a publishing format for reproducible computational workflowsPositioning and Power in Academic Publishing: Players, Agents and Agendas 2016; 87–90
- 19.McKinney W.Data Structures for Statistical Computing in PythonPaper presented in: Proceedings of the 9th Python in Science Conference 2010; 56–61
- 20.Harris P A, Taylor R, Thielke R, Payne J, Gonzalez N, Conde J G. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(02):377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Barron A G, Fenick A M, Maciejewski K R, Turer C B, Sharifi M. External validation of an electronic phenotyping algorithm detecting attention to high body mass index in pediatric primary care. Appl Clin Inform. 2024;15(04):700–708. doi: 10.1055/s-0044-1787975. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Samal L, Wu E, Aaron S et al. Refining clinical phenotypes to improve clinical decision support and reduce alert fatigue: a feasibility study. Appl Clin Inform. 2023;14(03):528–537. doi: 10.1055/s-0043-1768994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Joseph A, Mullett C, Lilly C et al. Coronary artery disease phenotype detection in an academic hospital system setting. Appl Clin Inform. 2021;12(01):10–16. doi: 10.1055/s-0040-1721012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Garduno Rapp N E, Herzberg S D, Ong H H et al. Application of an externally developed algorithm to identify research cases and controls from electronic health record data: failures and successes. Appl Clin Inform. 2025;16(02):314–326. doi: 10.1055/a-2524-5216. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.