Skip to main content
Open Forum Infectious Diseases logoLink to Open Forum Infectious Diseases
. 2026 Jul 24;13(8):ofag466. doi: 10.1093/ofid/ofag466

Interobserver Agreement for Classifying Infections in Emergency Department Patients Using Modified Centers for Disease Control and Prevention Criteria

Oren Turgman 1,#, Evelien Reijnders 2,#, Sebastiaan C M Joosten 3, Susanne E Doeleman 4, Aryna Kolodyazhna 5, Floris M C de Vries 6, Renée A Douma 7, Hazra S Moeniralam 8, Hessel Peters-Sengers 9,10, Stephanie Popping 11,12, Tom van der Poll 13,14, W Joost Wiersinga 15,16,✉,3
PMCID: PMC13428169  PMID: 42542778

Abstract

Background

To develop and evaluate a standardized emergency department (ED) adapted framework based on Centers for Disease Control (CDC) surveillance definitions and modified SOFA assessment for reproducible infection and sepsis classification in patients with suspected infection on the ED.

Methods

As part of the prospective cohort study BIOSEP, adults presenting to the ED with suspected infection and a Modified Early Warning Score ≥2 were consecutively enrolled between 2022 and 2026 at 4 Dutch hospitals. All enrolled patients in this study were retrospectively adjudicated through post hoc review of their electronic health records. Each case was independently assessed by 2 blinded physicians for plausibility of infection, infection diagnosis, causative pathogen, and sepsis occurrence. Disagreements were resolved by a third blinded reviewer. Interobserver agreement was quantified using Cohen's kappa and Gwet's agreement coefficients (gAC).

Results

In total, 1062 ED patients were included with 1145 infection episodes. Interobserver agreement for infection diagnosis was 88.9%, and agreement statistics were classified as very good with Cohen's κ of 0.87 (95% CI 0.85–0.89) and a gAC1 of 0.89 (0.87–0.91). Agreement for causative pathogens was 90.1% (κ 0.89 [0.87–0.92]; gAC1 0.97 [0.95–0.98]), and for sepsis occurrence 93.4% (κ 0.80 [0.76–0.85]; gAC1 0.92 [0.90–0.94]). Agreement for plausibility of infection was also high (87.9%; weighted κ 0.87 [0.85–0.89]; gAC2 0.92 [0.90–0.93]).

Conclusions

Our ED-adapted adjudication framework based on CDC surveillance definitions yields high interobserver agreement for infection and sepsis classification across a heterogeneous ED population, supporting its use in large-scale observational research.

Keywords: classification, diagnosis, emergency department, infectious diseases, interobserver variability


A standardized diagnostic handbook, based on CDC/NHSN infection definitions and modified SOFA criteria, demonstrated very good interobserver agreement for infection and sepsis classification in emergency department patients with suspected infection and supports its use in observational research.


Standardized and reproducible definitions of infections and sepsis are essential for valid and comparable clinical research. However, no universally accepted reference standard exists to retrospectively classify infections at emergency department (ED) presentation. In 1988, the Centers for Disease Control and Prevention (CDC) introduced surveillance criteria for specific nosocomial infections [1]. Subsequent annual revisions by the CDC and the National Healthcare Safety Network (NHSN) have resulted in surveillance-based infection definitions that are widely evaluated in research on individual infection entities [2–4]. Nevertheless, their applicability as a standardized framework for all-cause infection at ED presentation remains uncertain. In addition, these definitions include complex and subjective criteria that may further limit applicability in real-world settings.

In 2013, these CDC surveillance definitions were translated into a structured adjudication framework and systematically evaluated in a mixed population of patients on the intensive care unit (ICU) [5]. Interobserver agreement for classifying infection sources using CDC criteria was excellent overall, although full concordance between independent observers was rare for some infection types [5]. Whether such an operationalized framework is applicable in the ED, where case mix, diagnostic availability, and infection severity differ substantially, requires systematic evaluation.

Sepsis is defined as life-threatening organ dysfunction due to a dysregulated host response to infection and is therefore a critical classification domain alongside infection itself [6]. Its diagnosis relies on quantification of organ dysfunction usually defined as an acute increase of ≥2 Sequential Organ Failure Assessment (SOFA) points [6]. However, key SOFA components, most notably the arterial oxygen tension to inspired oxygen fraction ratio (PaO2/FiO2), are frequently unavailable in an ED setting. This remains a limitation with the recently updated SOFA-2 criteria, which include noninvasive respiratory support and extracorporeal membrane oxygenation (ECMO) but continue to rely on PaO2/FiO2 ratios for respiratory scoring [7]. To address this limitation, the modified SOFA (mSOFA) has been proposed and validated for the ED [8, 9], which substitutes the PaO2/FiO2 ratio with the more accessible oxygen saturation to inspired oxygen fraction ratio (SpO2/FiO2). In addition to issues of availability, several SOFA components demonstrate only moderate interobserver agreement in ICU settings, highlighting the need to assess this agreement when applied in the ED [10].

The present study evaluates whether an updated, ED-adapted version of the 2023 CDC/NHSN surveillance definitions, supplemented by structured organ-dysfunction assessment using the mSOFA, provides a reproducible reference standard for retrospective classification of infection and sepsis among ED patients with suspected infection, with supporting evidence for construct and predictive validity.

METHODS

Study Design and Setting

This study was performed within the framework of the Towards novel BIOmarkers for SEPsis (BIOSEP) project aimed at the identification and evaluation of novel diagnostics for sepsis in the ED (clinicaltrials.gov registration number NCT06178822) [11]. In this prospective multicenter observational cohort study, patients (≥18 years) presenting to the ED with suspected infection, as determined by the responsible ED physician at the time of presentation, and a Modified Early Warning Score (MEWS) ≥ 2 were included between 2022 and 2026 at 2 academic medical centers (Amsterdam University Medical Centers, location AMC, and location VUmc) and 2 teaching hospitals (Flevoziekenhuis [Flevo Hospital], Almere, and St. Antonius Hospital, Nieuwegein) [11].

Adjudication Procedure

For the present analysis, we retrospectively adjudicated the first consecutively enrolled patients during the initial phase of the study to establish a robust method for classifying infection and sepsis. Adjudication was performed according to the criteria specified in the diagnostic handbook (Supplementary File 1) and included assessment of infection plausibility, infection diagnosis, causative pathogens, and occurrence of sepsis during hospital stay and complications related to primary infections. This was based on a post hoc review of all relevant clinical data available from the electronic patient file (Epic Systems, Verona, WI) after at least 1 month of follow-up, such as vital parameters, laboratory and microbiological results, and imaging reports. This follow-up period allowed adjudicators to also consider pending diagnostic results that became available after the index ED presentation, such as blood culture results. When available, information from outpatient visits, repeat ED visits, or hospital admissions shortly after the index ED presentation was also considered. This was particularly relevant for patients discharged directly from the ED, for whom clinical data from hospital admission were not available.

A rotating panel of 6 physicians with clinical experience in internal medicine, ranging from several months to 1 year, was used, with adjudicator combinations varying across cases. Before the start of the adjudication process, all adjudicators were trained in the use of the diagnostic handbook. Training consisted of review of the handbook, discussion of the classification rules, and adjudication of example cases followed by consensus discussion. Each patient was then independently and blindly adjudicated by 2 physicians. Disagreements were resolved by a third blinded physician. When the third adjudicator did not agree with one of the prior adjudicators, the case was evaluated by an attending infectiologist or clinical microbiologist. Verbal and written informed consent was obtained from all BIOSEP participants or their legal representatives. The study was approved by the Amsterdam University Medical Centre Ethics Committee (No. 2022.0279).

Measurements and Outcomes: Diagnostic Handbook and Definitions

Infection diagnoses and definitions were based on the 2023 CDC/NHSN Surveillance Definitions for Specific Types of Infections [12] and categorized into 4 levels reflecting likelihood of infection: none, possible, probable and definite (Supplementary File 1). These included infections of the lower respiratory tract, upper respiratory tract, cardiovascular system, central nervous system, urinary tract, skin and soft tissue, gastrointestinal tract, reproductive system, bones and joints, eye, ear, and oral cavity. Plausibility of infection was categorized into none, possible, probable and definite, based on relevant available clinical information categorized in specific terms for each infection (Supplementary File 1). For each patient, occurrence of sepsis related to the primary infection during hospital stay was assessed, based on a change of 2 or higher in the mSOFA. Adjudication of causative pathogens was performed using a list of commonly found causative versus commensal pathogens in patients presenting to the ED with suspected infections and/or sepsis [13]. Complications related to infection were recorded in a case report form (Supplementary File 1) using consensus definitions where possible and included: acute respiratory distress syndrome (ARDS; updated 2024 Berlin criteria) [14], acute kidney injury (AKI; 2004 RIFLE criteria) [15], myocardial infarction (2018 Fourth Universal Definition of Myocardial Infarction) [16], stroke (2013 An Updated Definition of Stroke for the 21st Century) [17], deep venous thrombosis and pulmonary embolism (2019 ESC criteria) [18], delirium (2018 DSM-5 criteria) [19], and secondary infections (Supplementary File 1). These complications were defined as being associated with the primary infection and could be present at presentation to the ED or develop during hospital admission.

Data Analysis

For all infection episodes we assessed interobserver agreement for infection diagnosis by the first 2 adjudicators. Partial agreement was defined as both adjudicators choosing the same tract (eg, urinary tract) where infection occurred, while full agreement was defined as the same specific adjudicated diagnosis chosen by both adjudicators (eg, upper urinary system infection). Only for cases wherein both adjudicators agreed on the infection diagnosis, we calculated agreement statistics for plausibility of infection, causative pathogens, and sepsis diagnosis. Partial agreement for plausibility was defined as adjudication with the category infection not present (plausibility none and possible) or infection present (plausibility probable and definite). Full agreement was defined as the same adjudicated level of plausibility. To correct for agreement by chance, we calculated Cohen's kappa (κ) for categorical variables and a weighted κ for ordinal variables (ie, plausibility of infection) [20]. We classified κ > 0.80 as very good, 0.60 < κ < 0.80 as good, 0.40 < κ < 0.60 as moderate, 0.20 < κ < 0.40 as fair, and κ < 0.20 as poor [21]. To further correct for skewed distribution of infection diagnoses, Gwet's agreement coefficient 1 (gAC1) was calculated for categorical variables and a weighted gAC2 for ordinal variables (ie, plausibility of infection) [22]. For both Cohen's kappa and Gwet's AC statistics, a 95% confidence interval was calculated using a 1000× bootstrap procedure. For all descriptive figures, tables and statistics we reported adjudicated final diagnoses (eg, a case of urinary tract infection confirmed by 2 adjudicators), while we used single observations for all analyses pertaining to agreement (eg, agreement rates and kappa values). As the study was of a descriptive nature, a formal sample size calculation was not performed. Data were analyzed using R version 4.5.2 (Vienna, Austria) and RStudio version 2026.01.0 (Posit Software, Boston, MA). ChatGPT (OpenAI) was used for grammar and language refinement.

RESULTS

Patient and Cohort Characteristics

A total of 1062 consecutively enrolled patients were included with 1145 episodes of suspected infection. For 978 patients (92.1%) only 1 episode of suspected infection was recorded, while 79 (7.4%) had 2 simultaneous infections and 3 patients (0.3%) had 3 suspected infections at ED presentation. Since each episode was initially observed by 2 adjudicators, 2290 observations were evaluated in the agreement analyses (Figure 1). For 299 episodes (26.1%) a third adjudicator assessed the case. A total of 307 observations (13.4%) were classified as having no infection, despite suspected infection at ED presentation. A majority of patients were male (56.3%) and presented to the ED of an academic hospital (62.4%; Table 1). Median age was 66 years (interquartile range [IQR] 54, 75) and median MEWS was 3.0 (IQR 2.0, 4.0). Following ED presentation, 679 patients (63.9%) were admitted to hospital with a median length of hospital stay of 4 days (IQR 2, 7). Of all patients, 199 (18.7%) presented with sepsis or developed sepsis during hospital admission. ICU admission occurred in 50 patients (4.7%) and 30-day mortality was 4% in the overall cohort (Table 1).

Figure 1.

Diagnostic classification flow diagram for patients presenting to the emergency department with suspected infection and a Modified Early Warning Score of two or higher. The figure shows allocation of observations to major infection categories and specific diagnoses.

Descriptive diagram showing the diagnostic classification of patients presenting at the emergency room with suspected infection and a MEWS ≥ 2. The total number of observations (n = 2290) is double the number of infections (n = 1145), as each patient was independently assessed by 2 observers. Allocation to the main infection categories was determined based on the diagnoses of these 2 assessors; in cases of disagreement, the diagnosis made by a third observer was used. Partial diagnostic agreement was defined as concordance at the level of the organ system (eg, respiratory tract), while full diagnostic agreement was defined as concordance on specific diagnostic terms (eg, community-acquired pneumonia). Abbreviations: CAP, community-acquired pneumonia; LRT, lower respiratory tract; HAP, hospital-acquired pneumonia; UTI, urinary tract infection; SSTI, skin and soft tissue infections; SSI, surgical site infections; BSI, bloodstream infection; CRBI, catheter related bloodstream infection; CNS, central nervous system.

Table 1.

Baseline Characteristics of Patients at Emergency Department Presentation and Clinical Outcomes

ED Patients (n = 1062)
Demographics
Inclusion site, n (%)
 Amsterdam UMC, location AMC 315 (29.7)
 Amsterdam UMC, location VUmc 347 (32.7)
 Flevo Hospital 243 (22.9)
 Sint Antonius Hospital 157 (14.8)
Sex, male, n (%) 598 (56.3)
Age, y (median [IQR]) 66.0 [54.0, 75.0]
Body mass index (mean (SD)) 26.5 (5.9)
Days of symptoms (median [IQR]) 3.0 [1.0, 7.0]
Comorbidities, n (%)
Diabetes mellitus 236 (22.2)
Cardiovascular comorbidities 562 (52.9)
Pulmonary comorbidities 323 (30.4)
Active malignancy 119 (11.2)
Immunosuppressive disease 58 (5.5)
Chronic kidney disease 137 (12.9)
Vital parameters at baseline
Heart rate, beats per minute (mean (SD)) 98.1 (21.6)
Systolic blood pressure, mmHg (mean (SD)) 128 (26.0)
Diastolic blood pressure, mmHg (mean (SD)) 77.0 (17.4)
Respiratory rate, per minute (median [IQR]) 20.0 [16.0, 25.0]
Temperature, °C (median [IQR]) 37.5 [36.5, 38.3]
Oxygen saturation, % (median [IQR]) 96.0 [93.0, 98.0]
Supplementary oxygen, yes, n (%) 219 (20.6)
Laboratory values at baseline
C-reactive protein, mg/L (median [IQR]) 83.0 [29.0, 176.5]
Hemoglobin, mmol/L (mean (SD)) 7.8 (1.4)
Platelets, 109/L (median [IQR]) 246.0 [183.0, 318.5]
Leukocytes, 109/L (median [IQR]) 11.0 [7.8, 14.9]
Creatinine, µmol/L (median [IQR]) 83.0 [66.0, 112.0]
eGFR, mL/min/1.73 m2 (median [IQR]) 75.0 [52.0, 90.0]
Urea, mmol/L (median [IQR]) 6.3 [4.5, 9.1]
Bilirubin, µmol/L (median [IQR]) 10.0 [7.0, 16.0]
Complications
AKI, n (%) 91 (8.6)
ARDS, n (%) 5 (0.5)
Delirium, n (%) 46 (4.3)
Sepsis, n (%)a 199 (18.7)
Disease severity and outcomes
MEWS (median [IQR]) 3.0 [2.0, 4.0]
Ward length of stay, days (median [IQR]) 4.0 [2.0, 7.0]
ICU length of stay, days (median [IQR]) 2.0 [1.0, 4.0]
Hospital admission, n (%) 679 (63.9)
ICU admission, n (%) 50 (4.7)
30-days mortality, n (%) 43 (4.0)
90-days mortality, n (%) 81 (7.6)

Continuous data are presented as median [interquartile range] and categorical data as number (percentage).

Abbreviations: eGFR, estimated glomerular filtration rate; ICU, intensive care unit; IQR, interquartile range.

aSepsis at admission or developed during admission.

Adjudicated Infections and Infection-related Complications

Most commonly adjudicated diagnoses were community-acquired pneumonia (CAP; n = 330, 28.8% of all observations), pharyngitis/laryngitis (134; 11.7%), urinary tract infection (133; 11.6%), and skin and soft tissue infections (105; 9.2%) (Figure 2). Most common infection sites were the lower respiratory tract (345; 31.2%), urinary tract (165; 15.1%) and gastrointestinal tract (132; 12.4%) (Figure 2). Most prevalent adjudicated complications related to the primary infection were AKI (n = 92; 8.7%), delirium (n = 46; 4.3%) and secondary infections (n = 28; 2.6%) (Supplementary Table 1). A majority (65.4%) of these infection-related complications were already present at presentation to the ED. Complications that developed during hospital admission were less common and occurred after a median of 3 days (IQR 1, 7).

Figure 2.

Pie chart and bar chart summarizing adjudicated infection sources and diagnoses. Lower respiratory tract infections represent the largest infection category, followed by urinary tract and gastrointestinal infections. Community-acquired pneumonia is the most frequent specific diagnosis, with urinary tract infection, pharyngitis/laryngitis, and skin and soft tissue infection also commonly represented.

Pie and bar chart showing the distribution and agreement rates of all sources of infections. Distributions of adjudicated tract of source of infection (pie chart) and specific infection diagnoses (bar chart). Tracts with few observations (<5) are aggregated in the category Rare. Abbreviations: BSI, bloodstream infection.

Interobserver Agreements

Interobserver agreement statistics were very good for infection diagnosis (88.9%; κ 0.87 [0.85–0.89] and gAC1 0.89 [0.87–0.91]), causative pathogens (90.1%; κ 0.89 [0.87–0.92]; gAC1 0.97 [0.95–0.98]) and occurrence of sepsis (93.4%; κ 0.80 [0.76–0.85]; gAC1 0.92 [0.90–0.94]). Partial agreement on infection diagnosis (91.1%; κ 0.89 [0.87–0.91]; gAC1 0.90 [0.89–0.92]) was higher than full agreement mentioned above. Full agreement statistics for plausibility of infection were also classified as very good (87.9%; weighted κ 0.87 [0.85–0.89]; gAC2 0.92 [0.90–0.93]) with partial agreement resulting in slightly higher statistics (96%; κ 0.88 [0.84–0.91]; gAC1 0.94 [0.92–0.96]) (Figure 3; Table 2).

Figure 3.

Three-panel figure showing interobserver agreement statistics for infection classification. Agreement was very good for infection diagnosis, plausibility of infection, causative pathogens, and sepsis occurrence. Agreement was highest for lower respiratory tract, urinary tract, and gastrointestinal infections, with community-acquired pneumonia among the diagnoses showing the strongest concordance between observers.

Interobserver agreement statistics of major categories and sources of infection. A, Three agreement statistics are displayed for full agreement on all major recorded variables. 95% confidence intervals are displayed using whiskers. Statistics for all variables are unweighted, except for those related to plausibility as this was adjudicated on an ordinal 4-point scale. B, Agreement statistics for tracts of source of infection. C, Agreement statistics for specific infection diagnoses. Abbreviation: BSI, bloodstream infection.

Table 2.

Interobserver Agreement for Infection-related Variables Using Agreement Rate, Cohen's Kappa, and Gwet's AC

Agreement Rate (%) Cohen's Kappa Gwet's AC
Diagnosis
 Partial 91.1% 0.89 (0.87–0.91) 0.90 (0.89–0.92)
 Full 88.9% 0.87 (0.85–0.89) 0.89 (0.87–0.91)
Plausibility of infection
 Partial 96.0% 0.88 (0.84–0.91) 0.94 (0.92–0.96)
 Full 87.9% 0.87 (0.85–0.89) 0.92 (0.90–0.93)
Pathogen
 Full 90.1% 0.89 (0.87–0.92) 0.97 (0.95–0.98)
Sepsis
 Full 93.4% 0.80 (0.76–0.85) 0.92 (0.90–0.94)

Partial agreement for diagnosis was defined as agreement on infection site (eg, urinary tract), and full agreement as agreement on the specific diagnosis (eg, upper urinary tract infection). For plausibility of infection, partial agreement was defined as agreement on absence (none/possible) versus presence (probable/definite) of infection, and full agreement as the same category of plausibility. Agreement for plausibility of infection, pathogen and sepsis was only assessed in cases with full agreement on diagnosis; partial agreement was not defined for pathogen and sepsis adjudication.

Sites of infections with high interobserver agreement and high observation frequency were the lower respiratory tract (718 observations, 95.8% agreement, κ 0.94 [0.92–0.96]), urinary tract (344 observations, 94.2% agreement, κ 0.93 [0.90–0.96]) and gastrointestinal tract (285 observations, 95.4% agreement, κ 0.95 [0.92–0.97]) (Figure 3; Supplementary Table 2). Diagnoses with high interobserver agreement and highest number of observations were CAP (665 observations, 95% agreement, κ 0.93 [0.91–0.95]), urinary tract infection (268 observations, 89.6% agreement, κ 0.88 [0.83–0.92]) and pharyngitis/laryngitis (258 observations, 86.8% agreement, κ 0.85 [0.8–0.9]) (Figure 3; Supplementary Table 3). Misclassification of the 6 largest tracts and diagnosis categories was also analyzed and displayed in Sankey diagrams (Supplementary Figures 1–12). With regard to the upper respiratory tract, the most common discordantly adjudicated tract was the lower respiratory tract. For all other tracts, discordant adjudication most often resulted in classification as no infection. At the diagnostic level, discordant adjudication was mostly within the same tract or as no infection. The highest misclassification rate, defined as disagreement between the first 2 adjudicators, occurred when cases were classified as no infection by the first adjudicator (disagreement rate 21.8%). In contrast, disagreement was lowest for the gastrointestinal tract (4.6% disagreement) and CAP (5% disagreement). Interobserver agreement statistics were very high for the classification of infection and sepsis in ED patients.

DISCUSSION

In this study, we developed and evaluated a standardized adjudication framework incorporating CDC/NHSN surveillance definitions with mSOFA assessment for reproducible classification of infection and sepsis on the ED. Using systematic medical record review with independent multireviewer assessment, we observed very good interobserver agreement for infection diagnosis, plausibility of infection, causative pathogens, and sepsis occurrence. These findings demonstrate that our standardized adjudication framework can be reliably applied for research purposes to achieve diagnosis-level infection classification in heterogeneous ED populations.

Consistent and well-defined infection diagnoses are essential for valid interpretation and comparability of observational research [23]. In studies performed on the ED, adjudicated reference standards have previously been applied in specific contexts, such as patients with pulmonary complaints [24], differentiation between bacterial and viral infection [13, 25], or assignment to broad anatomical tracts [13]. However, universally accepted standardized definitions for retrospective, diagnosis-level infection classification across heterogeneous ED populations are lacking. Consequently, many observational studies rely on physician diagnosis or expert panels, approaches prone to interobserver variability and, in consensus settings, groupthink bias [26]. Even when structured handbooks are used, expert adjudication remains time-consuming; adjudication of ∼2400 patients required over 700 hours of specialist time in one study, yet the same work showed that, using a standardized handbook, trained students or residents achieved 83% agreement with experts [24].

Previous work in critically ill patients on the ICU reported high interobserver agreement when applying CDC surveillance definitions for infection diagnosis [5]. In the present study, we extended these findings by applying a comparable adjudication framework adapted to the ED, with independent dual review and third review in case of disagreement. This approach resulted in very good interobserver agreement for infection diagnosis and causative pathogens. Agreement was generally higher than reported in the prior ICU study, possibly reflecting differences in case mix and diagnostic complexity. Another explanation for the high agreement observed for plausibility of infection, causative pathogens and occurrence of sepsis may be that these variables were adjudicated only in cases with concordant infection diagnoses. Cases with discordant infection diagnoses may have been more prone to disagreement across these additional components.

In our cohort, CAP accounted for approximately one-third of adjudicated infections, followed by upper respiratory, urinary tract, and skin and soft tissue infections. Comparable distributions have been reported in ED cohorts from Denmark, France and the United States [27–29], suggesting that our study population reflects a typical ED case mix.

Beyond classifying the source of infection, our framework also assessed sepsis occurrence. Diagnosing sepsis according to Sepsis-3 in the ED is challenging because PaO2/FiO2 ratios required for SOFA are often unavailable at presentation. We therefore applied a validated mSOFA score based on routinely available data [8]. Nevertheless, the mSOFA requires clinical judgment to determine whether organ function represents an acute change from baseline and whether it is attributable to infection. Prior work has shown poor interobserver agreement among intensivists when attributing organ dysfunction to infection [30]. Despite this inherent subjectivity, interobserver agreement for sepsis occurrence was very good in our study, supporting the reliability of a retrospectively adjudicated mSOFA-based sepsis classification in the ED.

A key strength of this study is the development and application of a standardized adjudication handbook based on updated CDC/NHSN surveillance definitions, adapted to the ED setting, enabling consistent research-based infection and sepsis classification. A second strength is the use of a 3-adjudicator panel, aligning with prior infectious disease research showing that this panel size optimizes reproducibility without additional benefit from larger groups [25]. Third, the framework was successfully applied in a clinically heterogeneous ED population reflective of routine practice. Several limitations should be acknowledged. Adjudication relied on retrospective electronic health records, which may be incomplete or subject to reporting bias, particularly for assessment of sepsis development among patients discharged directly from the ED; however, because all reviewers assessed identical information, this is unlikely to have affected agreement between reviewers. Second, the diagnostic handbook was adapted to and validated within the Dutch healthcare context, which may limit generalizability to healthcare systems with different clinical practices. Furthermore, as CDC/NHSN surveillance definitions are periodically updated, the criteria applied in this study may not fully reflect the most recent revisions, although these updates generally involve minor refinements. Finally, rare or nonendemic infections are not comprehensively captured by CDC definitions and were classified as “other,” and categorized as definite only when the diagnosis was confirmed with certainty.

In conclusion, we developed a valid and reproducible approach for the retrospective classification of infection diagnoses in patients presenting to the ED for research purposes. High interobserver agreement was observed across a broad range of infection diagnoses and causative pathogens, despite the heterogeneity and diagnostic uncertainty inherent to the ED setting. We believe this approach provides a robust method for consistently classifying patients with infections in large-scale observational ED studies, both in the Netherlands and globally.

Supplementary Material

ofag466_Supplementary_Data

Notes

Authors contributions. O.T. and E.R. contributed to study design, data collection, adjudication, analysis and interpretation of the data, and manuscript drafting. S.C.M.J., S.E.D., A.K., and F.M.Cd.V. contributed to data collection and adjudication. H.P.S. provided methodological input. S.P. contributed microbiological expertise. R.A.D. and H.S.M. served as local investigators and contributed to data collection, W.J.W. and Tvd.P. served as principal investigators and contributed to study design, interpretation of results, and manuscript preparation. All authors reviewed the results and approved the final version of the manuscript.

Acknowledgments. We gratefully acknowledge all patients who participated in this study.

Financial support. Parts of this study were supported by a Eurostars grant (EUROSTARS Project SEP-RAID E!30) and a Top Consortium for Knowledge and Innovation (TKI) grant (Early Detection of Sepsis by Host Response Profiling [DETECT-SEPSIS]), with additional support from Deepull Diagnostics (Barcelona, Spain) and Inflammatix (Sunnyvale, CA). The funders had no role in the conduct or reporting of the study.

Data availability. The datasets and R scripts used and analyzed during the current study are available from the corresponding author upon reasonable request.

Statement regarding artificial intelligence. During the preparation of this manuscript the authors used ChatGPT (OpenAI) to assist with grammar and language refinement After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Patient consent statement . Verbal and written informed consent was obtained from all BIOSEP participants or their legal representatives. The study was approved by the Amsterdam University Medical Centre Ethics Committee (No. 2022.0279).

Contributor Information

Oren Turgman, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Evelien Reijnders, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Sebastiaan C M Joosten, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Susanne E Doeleman, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Aryna Kolodyazhna, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Floris M C de Vries, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands.

Renée A Douma, Department of Internal Medicine, Flevo Hospital, Almere, The Netherlands.

Hazra S Moeniralam, Department of Internal Medicine, St. Antonius Hospital, Nieuwegein, The Netherlands.

Hessel Peters-Sengers, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands; Department of Epidemiology and Data Science, Amsterdam UMC, Location Vrije Universiteit, Amsterdam, The Netherlands.

Stephanie Popping, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands; Department of Medical Microbiology and Infection Prevention, Amsterdam UMC, Amsterdam, The Netherlands.

Tom van der Poll, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands; Department of Medicine, Division of Infectious Diseases, Amsterdam UMC, Location Academic Medical Centre, Amsterdam, The Netherlands.

W Joost Wiersinga, Centre for Infection and Molecular Medicine, University of Amsterdam, Amsterdam Institute for Infection and Immunity, Amsterdam UMC, Amsterdam, The Netherlands; Department of Medicine, Division of Infectious Diseases, Amsterdam UMC, Location Academic Medical Centre, Amsterdam, The Netherlands.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

References

  • 1. Garner  JS, Jarvis  WR, Emori  TG, et al.  CDC definitions for nosocomial infections. Am J Infect Control  1988; 16:128–40. [DOI] [PubMed] [Google Scholar]
  • 2. Henriksen  NA, Meyhoff  CS, Wetterslev  J, et al.  Clinical relevance of surgical site infection as defined by the criteria of the Centers for Disease Control and Prevention. J Hosp Infect  2010; 75:173–7. [DOI] [PubMed] [Google Scholar]
  • 3. Stern  SE, Christensen  MA, Nevers  MR, et al.  Electronic surveillance criteria for non-ventilator-associated hospital-acquired pneumonia: assessment of reliability and validity. Infect Control Hosp Epidemiol  2023; 44:1769–1775. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Kim  JJ, Kang  H, Khan  AN, et al.  A comparison of National Healthcare Safety Network surveillance definitions to physician review for central-line-associated bloodstream infections. Infect Control Hosp Epidemiol  2023; 44:2062–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Klein Klouwenberg  PM, Ong  DS, Bos  LD, et al.  Interobserver agreement of Centers for Disease Control and Prevention criteria for classifying infections in critically ill patients. Crit Care Med  2013; 41:2373–8. [DOI] [PubMed] [Google Scholar]
  • 6. Singer  M, Deutschman  CS, Seymour  CW, et al.  The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA  2016; 315:801–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Ranzani  OT, Singer  M, Salluh  JIF, et al.  Development and validation of the sequential organ failure assessment (SOFA)-2 score. JAMA  2025; 334:2090–103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Grissom  CK, Brown  SM, Kuttler  KG, et al.  A modified sequential organ failure assessment score for critical care triage. Disaster Med Public Health Prep  2010; 4:277–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Rahmatinejad  Z, Reihani  H, Tohidinezhad  F, et al.  Predictive performance of the SOFA and mSOFA scoring systems for predicting in-hospital mortality in the emergency department. Am J Emerg Med  2019; 37:1237–41. [DOI] [PubMed] [Google Scholar]
  • 10. Perez-Torres  D, Merino-Garcia  PA, Canas-Perez  I, et al.  Real-world inter-observer variability of the Sequential Organ Failure Assessment (SOFA) score in intensive care medicine: the time has come for an update. Crit Care.  2023; 27:160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Reijnders  E, Turgman  O, Joosten  SCM, et al.  Towards novel BIOmarkers to diagnose SEPsis (BIOSEP) in the emergency room: a protocol for a multicentre, prospective cohort study. BMJ Open  2025; 15:e103138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. National Center for Emerging and Zoonotic Infectious Diseases, Division of Healthcare Quality Promotion, National Healthcare Safety Network . CDC/NHSN surveillance definitions for specific types of infections. Atlanta, GA: Centers for Disease Control and Prevention; 2023. [Google Scholar]
  • 13. Whitfield  NN, Hogan  CA, Chenoweth  J, et al.  A standardized protocol using clinical adjudication to define true infection status in patients presenting to the emergency department with suspected infections and/or sepsis. Diagn Microbiol Infect Dis  2024; 110:116382. [DOI] [PubMed] [Google Scholar]
  • 14. Matthay  MA, Arabi  Y, Arroliga  AC, et al.  A new global definition of acute respiratory distress syndrome. Am J Respir Crit Care Med  2024; 209:37–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Bellomo  R, Ronco  C, Kellum  JA, et al.  Acute renal failure—definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) group. Crit Care  2004; 8:R204–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Thygesen  K, Alpert  JS, Jaffe  AS, et al.  Fourth universal definition of myocardial infarction (2018). Circulation  2018; 138:e618–51. [DOI] [PubMed] [Google Scholar]
  • 17. Sacco  RL, Kasner  SE, Broderick  JP, et al.  An updated definition of stroke for the 21st century: a statement for healthcare professionals from the American Heart Association/American Stroke Association. Stroke  2013; 44:2064–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Konstantinides  SV, Meyer  G, Becattini  C, et al.  2019 ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS). Eur Heart J  2020; 41:543–603. [DOI] [PubMed] [Google Scholar]
  • 19. Regier  DA, Kuhl  EA, Kupfer  DJ. The DSM-5: classification and criteria changes. World Psychiatry  2013; 12:92–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Cohen  J. A coefficient of agreement for nominal scales. Educ Psychol Meas  1960; 20:37–46. [Google Scholar]
  • 21. Landis  JR, Koch  GG. The measurement of observer agreement for categorical data. Biometrics.  1977; 33:159–74. [PubMed] [Google Scholar]
  • 22. Gwet  K. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol  2008; 61:29–48. [DOI] [PubMed] [Google Scholar]
  • 23. Patel  R, Tsalik  EL, Evans  S, et al.  Clinically adjudicated reference standards for evaluation of infectious diseases diagnostics. Clin Infect Dis  2023; 76:938–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. van Engelen  TSR, Kanglie  M, van den Berk  IAH, et al.  Classifying the diagnosis of study participants in clinical trials: a structured and efficient approach. Eur Radiol Exp  2020; 4:44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. van Houten  CB, Naaktgeboren  CA, Ashkenazi-Hoffnung  L, et al.  Expert panel diagnosis demonstrated high reproducibility as reference standard in infectious diseases. J Clin Epidemiol  2019; 112:20–7. [DOI] [PubMed] [Google Scholar]
  • 26. Bankier  AA, Levine  D, Halpern  EF, et al.  Consensus interpretation in imaging research: is there a better way?  Radiology.  2010; 257:14–7. [DOI] [PubMed] [Google Scholar]
  • 27. Skjot-Arkil  H, Cartuliares  MB, Heltborg  A, et al.  Clinical characteristics and diagnostic accuracy of preliminary diagnoses in adults with infections in Danish emergency departments: a multicentre combined cross-sectional and diagnostic study. BMJ Open  2024; 14:e090259. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Oberlin  M, Balen  F, Bertrand  L, et al.  Sepsis prevalence among patients with suspected infection in emergency department: a multicenter prospective cohort study. Eur J Emerg Med  2020; 27:373–8. [DOI] [PubMed] [Google Scholar]
  • 29. Hooper  GA, Klippel  CJ, McLean  SR, et al.  Concordance between initial presumptive and final adjudicated diagnoses of infection among patients meeting sepsis-3 criteria in the emergency department. Clin Infect Dis  2023; 76:2047–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Rhee  C, Kadri  SS, Danner  RL, et al.  Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes. Crit Care.  2016; 20:89. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ofag466_Supplementary_Data

Articles from Open Forum Infectious Diseases are provided here courtesy of Oxford University Press

RESOURCES