Accuracy of the modified Global Burden of Disease International Classification of Diseases coding methods for identifying sepsis: a prospective multicentre cohort study

Abstract

Background

This study assessed the accuracy of three International Classification of Diseases (ICD) codes methods derived from Global Burden of Disease (GBD) sepsis study (modified GBD method) in identifying sepsis, compared to the Angus method. Sources of errors in these methods were also reported.

Methods

Prospective multicentre, observational, study. Emergency Department patients aged ≥ 16 years with high sepsis risk from nine hospitals in NSW, Australia were screened for clinical sepsis using Sepsis 3 criteria and coded as having sepsis or not using the modified GBD and Angus methods. The three modified GBD methods were: Explicit—sepsis-specific ICD code recorded; Implicit—sepsis-specific code or infection as primary ICD code plus organ dysfunction code; Implicit plus—as for Implicit but infection as primary or secondary ICD code. Agreement between clinical sepsis and ICD coding methods was assessed using Cronbach alpha (α). For false positive cases (ICD-coded sepsis but not clinically diagnosed), the ICD codes leading to those errors were documented. For false negatives (clinically diagnosed sepsis but ICD-coded), uncoded sources of infection and organ dysfunction were documented.

Results

Of 6869 screened patients, 450 (median age 72.4 years, 48.9% females) met inclusion criteria. Clinical sepsis was diagnosed in 215/450 (47.8%). The explicit, implicit, implicit plus and Angus methods identified sepsis in 108/450 (24.0%), 175/450 (38.9%), 222/450 (49.3%) and 170/450 (37.8%), respectively. Sensitivity was 41.4%, 58.1%, 67.4% and 55.8%, and specificity 91.9%, 78.7%, 67.2% and 79.1%, respectively. Agreement between clinical sepsis and all ICD coding methods was low (α = 0.52–0.56). False positives were 19, 50, and 77, while false negatives were 126, 90, and 70 for the explicit, implicit, and implicit plus methods, respectively. For false positive cases, unspecified urinary tract infection, hypotension and acute kidney failure were commonly assigned infection and organ dysfunction codes. About half (44.3%-55.6%) of the false negative cases didn’t have a pathogen documented.

Conclusion

The modified GBD method demonstrated low accuracy in identifying sepsis; with the implicit plus method being the most accurate. Errors in identifying sepsis using ICD codes arise mostly from coding for unspecified urinary infections and associated organ dysfunction.

Trial registration

The study was registered at the ANZCTR (ACTRN12621000333819) on 24 March 2021.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-025-05448-x.

Background

Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection and is estimated to affect around 50 million people each year with 13.66 million associated deaths [1–3]. Accurate measurement of the burden of sepsis is crucial for healthcare professionals, researchers, funders, and policymakers but remains challenging despite the increased visibility of sepsis and increasing awareness of the global health challenge it poses.

Sepsis epidemiology can be studied using retrospective chart review [4–6], prospective inception cohort studies [7, 8], analysis of high-quality databases [4–6], electronic medical records [6, 9–11] and routinely collected administrative or billing data. Each of these has shortcomings; retrospective chart review is notoriously inaccurate, prospective inception cohort studies are less subject to missing data and recall bias but are resource intensive and impractical to conduct at scale or repeatedly. Analysing International Classification of Diseases (ICD) coding currently offers the most practical method of estimating sepsis incidence and outcomes at scale and over time [9] and was the basis of the Global Burden of Diseases (GBD) study of global sepsis epidemiology [2]. A number of high-profile studies have used different combinations of ICD codes and methods to estimate sepsis epidemiology in high income countries [12, 13]. The two major approaches to use ICD codes to estimate sepsis epidemiology are: explicit, which requires that a sepsis-specific code is listed in hospital separation or billing data, and implicit, which additionally counts hospitalizations where codes for infection and organ dysfunction are both listed during a single hospitalisation episode.

In 2020, the GBD sepsis project modelled sepsis epidemiology between 1990 to 2017 using ICD coding of cause of death data and derived incidence by dividing sepsis related deaths by estimated case fatality rates [2]. A modification of the ICD codes used in the GBD sepsis project (modified GBD method) was subsequently used to analyse an administrative dataset in Australia [14, 15]. The modified GBD method included sepsis-specific, and a combination of infection and organ dysfunction ICD-10th Revision-Australian Modification (ICD-10-AM) codes.

Most studies evaluating the accuracy of estimates of sepsis epidemiology using ICD coding methods compared ICD coding with retrospective chart review and reported wide variation in sepsis estimates [5, 12, 16–19], and in diagnostic accuracy [20].

Using ICD codes results in considerable variation in disease estimates when comparing coding data with clinical reference standards. This has been shown for a variety of clinical conditions [21–23]. Specifically for sepsis, only a few studies have attempted to evaluate sources of error in the ICD coding methods [21, 24], an essential step in improving their performance.

We conducted a prospective cohort study to evaluate the diagnostic accuracy of three modified GBD methods by comparing them with prospective clinical diagnosis of sepsis (the gold standard) and comparing their diagnostic accuracy with the Angus ICD coding method. Additionally, we describe the sources of false positive and false negative errors in the modified GBD methods.

Methods

Study design

This was a prospective multicentre, observational, diagnostic accuracy study conducted at nine hospitals in New South Wales, Australia. Due to disruption by the SARS-CoV-2 pandemic, participant enrolment took longer than anticipated and was completed between December 2020 and January 2023.

Study population and sample size calculation

Sample size calculation was based on considerations for the width of a two-sided 95% confidence interval (CI) of reported sensitivity of explicit sepsis ICD coding methods of between 7 and 16% [13, 25]. Optimal statistical efficiency is achieved if 50% of the enrolled patients have a clinical diagnosis of sepsis. For this reason and the low incidence of sepsis in the general hospital population, eligible patients were those presenting to the Emergency Department with a high risk of having or developing sepsis. This was based on two previous studies which reported 32.2–50.45% of patients with a diagnosed infection and two or more positive quick Sequential Organ Failure Assessment (qSOFA) criteria developed clinical sepsis [26, 27]. Based on these data and anticipating about half of our patient cohort would have a clinical diagnosis of sepsis; a sample size of 500 was selected to provide about 250 patients with clinical sepsis.

Based on pre-existing data, the following eligibility criteria were adopted:

• Patients aged 16 years or more presenting to the emergency department

• Expected duration of hospital stay more than 24 h

• An order for culture of body fluid plus oral or intravenous administration of an antibiotic indicating suspected infection

• Presence of at least two of the following qSOFA criteria [1]

• Altered mental status (Glasgow Coma Scale < 15)

• Respiratory rate (≥ 22/min)

• Systolic blood pressure ≤ 100 mmHg

To examine the predictive ability of our eligibility criteria, we conducted a preliminary analysis of the first 100 patients with a provision to modify study eligibility criteria if the 95% confidence interval (CI) of the proportion did not include 50%.

Patients were assessed as having clinical sepsis if they met the Sepsis-3 criteria [1] of the presence of suspected or presumed infection plus an increase of two or more in Sequential Organ Failure Assessment (SOFA) [28] score from baseline. For the follow-up SOFA score, if the total score was recorded in the clinical notes, it was used. Otherwise, it was calculated using the most recent variables within 24 h of obtaining the culture, using a SOFA calculation sheet. If data for one or more SOFA domain were missing or incomplete that domain was assigned a score of zero, and the total SOFA score was calculated from domains for which data were available.

For patients with admission SOFA score of two or more, those assigned one of the explicit sepsis codes were classified as having clinical sepsis whereas those assigned one of the infection codes as primary diagnosis were independently reviewed by two members of the research team to determine if the increased SOFA can be attributed to any chronic health condition. Any disagreement between two reviewers were resolved by a third reviewer. For patients for whom increased SOFA couldn’t be attributed to underlying chronic health condition they were classified as having clinical sepsis.

Study procedures

All patients admitted through emergency departments were screened by a member of the research team for eligibility. Demographic characteristics and hospital admission details of the enrolled patients were recorded at the time of admission. Patients were followed up twice after admission.

First, at 96 h post admission, to assess the occurrence of clinical sepsis. When no data was available to calculate SOFA score between admission and 96 h follow-up, the SOFA score at admission was used to determine the presence of clinical sepsis. Two investigators reviewed the pre-existing health history of patients who had a SOFA score of two or more at admission to determine if the SOFA score was explained by pre-existing organ dysfunction, if not they were assigned a clinical diagnosis of sepsis.

Second, on day 60 after admission to collect outcome data including duration of hospital admission, alive or dead at hospital discharge, and cause of death where relevant. The medical records of patients who did not meet the criteria for clinical sepsis at 96 h follow-up were reviewed to determine if they met the clinical sepsis criteria between 96 h and hospital discharge. Primary and secondary diagnosis ICD-10-AM codes, that had been assigned by trained coders were obtained from hospital databases to determine whether the codes assigned to the patient satisfied the ICD coding method criteria to be classified as having sepsis. For further details of ICD coding in Australia, see the Australian Coding Standards [29]. Patients still in the hospital at the final follow-up were excluded due to lack of ICD coding data.

Data analysis

Following ICD coding methods were evaluated: (Additional File 1; Supplementary Tables 1, 2, 3).

1. Modified GBD methods

2. Explicit: presence of one of the explicit sepsis ICD-10-AM codes as the primary or secondary diagnosis.

3. Implicit: presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as a secondary diagnosis from the modified GBD codes OR one of the explicit sepsis codes.

4. Implicit plus: Presence of an infection code as primary or secondary diagnosis and “organ dysfunction code” from the modified GBD codes OR one of the explicit sepsis codes.

5. Angus [13]: Presence of an infection code and an “organ dysfunction code” from the Angus codes (Additional File1; Supplementary Tables 2 and 3) or R57.2 or R65.1 from the explicit sepsis codes (Additional File1; Supplementary Table 1)

The following two-by-two contingency table was created for each ICD coding method:

Sepsis as per ICD coding method	Sepsis by clinical diagnosis
	Yes	No
Yes	True positive (TP)	False positive (FP)
No	False negative (FN)	True negative (TN)

Standard diagnostic accuracy parameters were calculated as follows:

• Sensitivity = TP/TP + FN × 100

• Specificity = TN/TN + FP × 100

• Positive Predictive Value (PPV) = TP/ (TP + FP) × 100

• Negative Predictive Value (NPV) = TN/ (TN + FN) × 100

• Positive likelihood ratio (LR +) = Se/(1-Sp)

• Negative likelihood ratio (LR-) = (1-Se)/Sp

• Diagnostic Odds Ratio (DOR) = (TP x TN)/(FP x FN)

Clinical diagnosis of sepsis, the presence of suspected or presumed infection plus an increase of 2 or more in SOFA score, the Sepsis 3 criteria [1], was used as the reference standard.

For false positive cases, those classified as sepsis by ICD coding but not clinically, we document the ICD codes leading to those errors, and for false negatives, those diagnosed with clinical sepsis but not by ICD coding, we document the uncoded sources of infection and organ dysfunction.

Statistical analysis

For descriptive analyses, continuous variables are reported either as mean with standard deviation (SD) or median with interquartile range; (IQR) as appropriate. Proportions are presented as a percentage with 95% confidence intervals (CIs). Diagnostic accuracy parameters are presented as percentages and ratios along with 95% CI. All tests of significance were two-tailed and a P value < 0.05 was considered as statistically significant; P values are not corrected for multiplicity of testing. Comparisons of sensitivity and specificity were performed using McNemar test [30, 31], while predictive values were compared using generalised score statistics [32]. Agreement between clinical sepsis and ICD coding methods was assessed using Cronbach’s alpha (α) coefficient [33], a measure of internal consistency with value more than 0.7 considered as satisfactory [34]. A descriptive analysis of false positive and false negative cases was done to report the sources of errors in each modified GBD method.

Analyses were performed using SPSS v28.0, SAS v9.4 and Microsoft Excel (Microsoft Corporation 2018; https://office.microsoft.com/excel).

The results are reported as per STARD [35] and STROBE [36] guidelines for reporting diagnostic accuracy and observational studies (Refer Additional Files 3 and 4).

Results

Of the initial 100 patients 42 [42% (95% CI, 32.3–51.8%)] had a clinical diagnosis of sepsis, consequently, we continued the study using the original eligibility criteria.

Of 6869 screened patients, 450 were included in the analysis (Fig. 1). Of these, 215 [47.8%; (95% CI, 43.2%-52.4%)] patients were assigned a clinical diagnosis of sepsis, which was considered acceptable for valid statistical analysis. The number (%; 95% CI) coded for sepsis by different ICD coding methods were modified GBD-explicit 108 (24.0%; 20.3–28.2%), modified GBD- implicit 175 (38.9%;34.5–43.5%), modified GBD- implicit plus 222 (49.3%; 44.7–53.9%) and Angus 170 (37.8%; 33.4–42.3%) (Fig. 1).

Fig. 1.

Study flow. Clinical sepsis: Number of patients who met Sepsis-3 criteria. *Presence of one of the explicit sepsis ICD-10-AM codes (Additional File 1; Supplementary Table 1) as the primary or secondary diagnosis. **Presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as secondary diagnosis from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1). ^$Presence of an infection code and an “organ dysfunction code” from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1). ***Presence of an infection code and an “organ dysfunction code” from the Angus codes (Additional File 1; Supplementary Tables 2 and 3) or R57.2 or R65.1 from the explicit sepsis codes (Additional File 1; Supplementary Table 1). GBD, Global Burden of Disease; ICD-10-AM, International Classification of Diseases-10th Revision-Australian Modification. Note: The number of patients in various groups are not mutually exclusive

Patient characteristics

In the overall population, the mean (SD) age was 72.4 (18.3) years, with 327 (72.7%) patients aged 65 years or more and 217 (48.2%) being female. Medical admissions accounted for 398 (88.4%) patients and 82 (18.2%) patients were admitted to intensive care unit (ICU). (Additional File 2; Supplementary Table 1). Demographic characteristics were comparable between clinical sepsis and sepsis ICD coding groups, except for a significantly higher ICU admission rate in the explicit modified GBD group compared to the clinical sepsis cohort (43/108, 39.8% versus 62/215, 28.8%, difference 11.0% [95% CI, 0.17%-21.8%; p = 0.01]).

The median (IQR) length of hospital stay for patients diagnosed with clinical sepsis was 9.5 [6–17] days. In-hospital mortality, censored at 60 days, was 24/215 (11.2%) in clinical sepsis patients, 20/108 (18.5%), 21/175 (12.0%) and 23/222 (10.4%) in patients designated as having sepsis using the explicit, implicit, and implicit plus modified GBD methods, respectively and 19/170 (11.2%) of those identified using the Angus method. Length of hospital stay, and mortality rate were similar between clinical sepsis and ICD coding methods groups (Additional File 2; Supplementary Fig. 1).

Details of the clinical characteristics of patients in various sepsis groups are given in Additional File 2; Supplementary Table 2 while details of the pathogen and antimicrobials profile of clinical sepsis patients are given in Additional File 2; Supplementary Tables 4 and 5.

Diagnostic accuracy of the modified GBD method

The explicit modified GBD method correctly classified 89 of 215 patients diagnosed with clinical sepsis: sensitivity 41.4% (95% CI, 34.8–48.8%), specificity 91.9% (95% CI, 88.4–95.4%), PPV 82.4% (95% CI, 75.2–89.6%), and NPV 63.2% (95% CI, 58.1%-68.3%).

The implicit modified GBD method correctly classified 125 of 215 patients diagnosed with clinical sepsis: sensitivity 58.1% (95% CI, 51.5–64.7%), specificity 78.7% (95% CI, 73.5–84.0%), PPV 71.4% (95% CI, 64.7–78.1%), and NPV 67.3% (95% CI, 61.7–72.8%).

The implicit plus modified GBD method correctly classified 145 of 215 patients diagnosed with clinical sepsis: sensitivity 67.4% (95% CI, 61.2–73.7%), specificity 67.2% (95% CI, 61.2%-73.2%), PPV 65.3% (95% CI, 59.1%-71.6%), and NPV 69.3% (95% CI, 63.3%-75.3%) (Table 1 and Additional File 2; Supplementary Table 3).

Table 1.

Diagnostic accuracy parameters of modified GBD and Angus methods

Diagnostic accuracy parameters	Modified GBD			Angus value (95% CI)	P value*	P value**
	Explicit value (95% CI)	Implicit value (95% CI)	Implicit plus value (95% CI)
Sensitivity	41.4% (34.8–48.8%)	58.1% (51.5–64.7%)	67.4% (61.2–73.7%)	55.8% (49.2–62.4%)	0.46	< 0.0001
Specificity	91.9% (88.4–95.4%)	78.7% (73.5–84.0%)	67.2% (61.2–73.2%)	79.1% (74.0–84.3%)	1.00	< 0.0001
PPV	82.4% (75.2–89.6%)	71.4% (64.7–78.1%)	65.3% (59.1–71.6%)	71.0% (64.2–77.8%)	0.77	0.01
NPV	63.2% (58.1–68.3%)	67.3% (61.7–72.8%)	69.3% (63.3–75.3%)	66.1% (60.3–71.7%)	0.50	0.04
Positive LR	5.1 (3.2–8.1)	2.7 (2.1–3.6)	2.1 (1.7–2.5)	2.7 (2.0–3.5)	–	–
Negative LR	0.64 (0.57–0.72)	0.53 (0.45–0.63)	0.48 (0.39–0.60)	0.56 (0.47–0.66)	–	–
DOR	8.0 (4.7–13.8)	5.1 (3.4–7.8)	4.8 (3.1–7.3)	4.3 (2.9–6.3)	–	–

^*Implicit versus Angus; **Implicit plus versus Angus

CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio; DOR, diagnostic Odd Ratio

Comparison of sensitivity and specificity were performed using McNemar test whereas positive and negative predictive values were compared using Generalised score statistics

Explicit: Presence of one of the explicit sepsis ICD-10-AM codes (Additional File 1; Supplementary Table 1) as the primary or secondary diagnosis

Implicit: Presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as secondary diagnosis from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1)

Implicit plus: Presence of an infection code and an “organ dysfunction code” from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1)

Angus: Presence of an infection code and an “organ dysfunction code” from the Angus codes (Additional File 1; Supplementary Tables 2 and 3) or R57.2 or R65.1 from the explicit sepsis codes (Additional File 1; Supplementary Table 1)

DOR: Diagnostic Odds Ratio, GBD: Global Burden of Disease; ICD: International Classification of Diseases-10th Revision- Australian Modification

Details of other diagnostic accuracy parameters are provided in Table 1.

Diagnostic accuracy of the Angus method

The Angus method correctly classified 120 of 215 patients diagnosed with clinical sepsis: sensitivity 55.8% (95% CI, 49.2–62.4%), specificity 79.1% (95% CI, 74.0–84.3%), PPV 71.0% (95% CI, 64.2–77.8%), and NPV 66.1% (95% CI, 60.3–71.7%).

The diagnostic accuracy parameters of the Angus method were comparable to the implicit modified GBD method. In comparison to the implicit plus modified GBD method the angus method had lower sensitivity (P < 0.0001) and negative predictive value (P = 0.04) but higher specificity (P < 0.0001) and positive predictive value (P = 0.01) (Table 1 and Additional File 2; Supplementary Table 3).

Agreement between clinical sepsis and ICD coding methods

Of the three modified GBD methods, the explicit method did not identify 126/215 (58.6%) patients with clinical sepsis while the implicit and implicit plus methods did not identify 90/215 (41.9%) and 70/215 (32.6%) of patients with clinical sepsis, respectively. All methods showed low agreement with clinical sepsis (α = 0.51–0.56) (Fig. 2).

Fig. 2.

Agreement between clinical sepsis and various ICD coding methods. Note: Size of circle is proportional to number of patients in a group; overlapped area indicates degree of agreement. Numbers in the overlapping areas indicate the number of patients satisfying multiple criteria. Clinical sepsis: Number of patients who met Sepsis-3 criteria. Explicit: Presence of one of the explicit sepsis ICD-10-AM codes (Additional File 1; Supplementary Table 1) as the primary or secondary diagnosis. Implicit: Presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as secondary diagnosis from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1). Implicit plus: Presence of an infection code and an “organ dysfunction code” from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1). Angus: Presence of an infection code and an “organ dysfunction code” from the Angus codes (Additional File 1; Supplementary Tables 2 and 3) or R57.2 or R65.1 from the explicit sepsis codes (Additional File 1; Supplementary Table 1). GBD, Global Burden of Disease; ICD-10-AM, International Classification of Diseases-10th Revision- Australian Modification

Sources of errors

False positives—explicit

In the 19 cases that were wrongly identified as having sepsis by the explicit modified GBD method, A41.9 (Unspecified sepsis) (11; 57.9%) was the most common ICD-10-AM code followed by A41.5 (Sepsis due to other and unspecified gram-negative organisms) (4; 21.1%). The false positive cases were evenly listed as primary or secondary diagnosis except A41.9 (Unspecified sepsis) which was a secondary diagnosis in 10/19 (90.9%) of cases (Table 2).

Table 2.

False positive cases for explicit modified GBD method (N = 19)

Explicit sepsis ICD-10-AM code	Number	Primary diagnosis	Secondary diagnosis
A41.9 Unspecified sepsis	11 (57.9%)	1 (9.1%)	10 (90.9%)
A41.5 Sepsis due to other and unspecified Gram-negative organisms	4 (21.1%)	2 (50.0%)	2 (50.0%)
A40 Streptococcal sepsis	3 (15.8%)	1 (33.3%)	2 (66.7%)
A41.8 Other specified sepsis	2 (10.5%)	1 (50.0%)	1 (50.0%)

One patient had two explicit sepsis ICD-10-AM codes

Explicit: Presence of one of the explicit sepsis ICD-10-AM codes as the primary or secondary diagnosis (Refer Additional File 1; Supplementary Table 1)

ICD-10-AM, international classification of disease; GBD, Global Burden of Disease

False positives—implicit

Of the 50 false positive cases, 19 (38.0%) had one of the explicit sepsis ICD-10-AM codes. The most common infection code was N39 (Urinary tract infection, site not specified), noted in 13 (26.0%) patients, whereas the most common organ dysfunction code was N17.9 (Acute kidney failure, unspecified), noted in 26 (52.0%) patients. Common pairs of infection and organ dysfunction ICD-10-AM codes were N39 (Urinary tract infection, site not specified), J12.8 (Viral pneumonia) and B97 (Viral agents as the cause of diseases classified elsewhere) in combination with N17.9 (Acute kidney failure, unspecified), and J96 (Respiratory failure, not elsewhere classified), each noted in four (8.0%) patients (Fig. 3A).

Fig. 3.

Heat map of combinations of infection codes and organ dysfunction ICD-10-AM codes in false positive cases. A Implicit modified GBD method (N = 50). Note: More than one infection and/or organ dysfunction code was present per patient. ICD-10-AM, international classification of disease-10th revision-Australian modification; GBD, Global Burden of Disease. Implicit: Presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as secondary diagnosis from the modified GBD codes (Additional File 1, Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1). B Implicit plus modified GBD method (N = 77). Note: More than one infection and/or organ dysfunction code was present per patient. ICD-10-AM: International Classification of Disease-10th Revision- Australian Modification; GBD: Global Burden of Disease. Implicit plus: Presence of an infection code and an “organ dysfunction code” from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1)

False positives—implicit plus

Out of 77 false positive cases, 19 (24.7%) had one of the explicit sepsis codes. The most common infection codes was N39 (Urinary tract infection, site not specified) in 20 patients (26.0%), whereas the most common organ dysfunction code was N17.9 (Acute kidney failure, unspecified), noted in 49 (63.6%) patients. The most common pair of infection and organ dysfunction ICD-10-AM codes was N39 (Urinary tract infection, site not specified) and N17.9 (Acute kidney failure, unspecified) noted in 9 patients (11.7%) (Fig. 3B).

False negatives—explicit

There were 126 false negative cases for modified GBD explicit method (cases with clinically diagnosed sepsis but not allocated an explicit sepsis code). Of these, 57 (45.2%) patients did not have a causative pathogen identified in the medical record. Amongst those with a documented causative organism, E. coli (17; 13.5%) was the most common pathogen. The most common pair of pathogen and infection site was E. coli and renal/genitourinary infection, seen in 11 (8.7%) cases (Additional File 2; Supplementary Fig. 3A).

False negatives—implicit

Sepsis was clinically diagnosed but the modified GBD implicit method was not satisfied in 90 patients. Of those, 24 (26.7%) had only an organ dysfunction code recorded, 37 (41.1%) had only an infection code recorded, and 29 (32.2%) cases had neither recorded (Fig. 4).

Fig. 4.

Distribution of infection and organ dysfunction ICD-10-AM codes in false negative cases in implicit and implicit plus modified GBD methods. ICD-10-AM, international classification of disease; OD, organ dysfunction; GBD, Global Burden of Disease. Implicit: Presence of an infection code listed as the primary diagnosis and an “organ dysfunction code” listed as secondary diagnosis from the modified GBD codes OR one of the explicit sepsis codes. Implicit plus: Presence of an infection code and an “organ dysfunction code” from the modified GBD codes (Additional File 1; Supplementary Tables 2 and 3) OR one of the explicit sepsis codes (Additional File 1; Supplementary Table 1)

Fifty (55.6%) patients did not have a pathogen documented. When documented, E. coli (14 cases; 15.6%) was the most common pathogen and renal/genitourinary 27 (30.0%) and pulmonary 17 (18.9%) were the most common infection sites with their combination being the most common pair (8; 8.9%) (Additional File 2; Supplementary Fig. 3B).

False negatives—implicit plus

In 70 cases, sepsis was clinically diagnosed, but the modified GBD implicit plus method was not satisfied. Of those, nine (12.9%) had only an organ dysfunction code recorded, 37 (52.9%) had only an infection code recorded, and 24 (34.3%) cases had neither recorded (Fig. 4). Thirty-one (44.3%) patients had no documented pathogen. Where documented, E. coli was the most common pathogen (8 cases; 11.4%), and renal/genitourinary was the most common site of infection (18 cases; 25.7%) with their combination being the most common pair (6; 8.6%). (Additional File 2; Supplementary Fig. 3C).

Across all GBD modified methods, N39 (Urinary tract infection, site not specified) and J18.9 (Pneumonia, unspecified) were commonly assigned infection codes whereas F05 (Delirium, not induced by alcohol and other psychoactive Substances) and N17.9 (Acute Kidney Failure, unspecified) were common organ dysfunction codes (Additional File 2; Supplementary Table 6).

Discussion

Summary of key findings

In this prospective cohort study, we assessed the accuracy of the four methods using ICD coding, in identifying sepsis using clinically diagnosed sepsis as the reference standard. None of the evaluated methods showed an optimal combination of sensitivity and specificity, defined as more than or equal to 80%, in identifying sepsis. The explicit modified GBD method significantly undercounted sepsis, similar to previous studies of explicit methods conducted in Australia [25], and other countries [21, 37]. Amongst various implicit methods, the GBD implicit plus method produced a sepsis count that was numerically closest to the count using clinical diagnosis. However, this resulted from a similar number of false positive and false negative designations which is reflected in the calculated sensitivity and specificity and the low level of agreement between clinical sepsis and various ICD coding methods. This result is similar to that seen with the Electronic Health Record method evaluated by Rhee et al. [10]. Of the methods we assessed, the GBD implicit plus method provided the most accurate estimate of the number of sepsis-related deaths. The Angus method provided the least accurate estimates of the number of deaths even though its diagnostic accuracy parameters were similar to the implicit modified GBD method.

The sensitivity of implicit modified GBD and Angus method in our study was comparable to estimates of the Angus method in previous studies [10, 19, 25]. In our study, using a broader implicit approach (implicit Plus) resulted in an increase in sensitivity and negative predictive value but a reduction in specificity and positive predictive value which is similar to a previous study [21].

For all modified GBD methods, most false positive cases occurred either when an ICD-10-AM code of unspecified sepsis was assigned or unspecified ICD-10-AM infection code was recorded in combination with an ICD-10-AM code for unspecified acute kidney failure, hypotension, or respiratory failure. Whereas most false negative cases were noted when clinicians made a clinical diagnosis of sepsis, but a causative organism was not identified. Moreover, one-third of false negative cases had neither infection nor organ dysfunction codes recorded. The absence of information on isolated organism in majority of false negative cases and a considerable proportion of common infection and organ dysfunction ICD-10-AM codes being ‘unspecific’ suggest inconsistent or poor clinical documentation. This mirrors findings in previous sepsis trials where ~ 30% of the included patients did not have a positive culture [10, 11, 38]. Increased interaction between coders and clinical staff, and clear clinical notes can potentially reduce overall errors in the ICD coding methods; this has also been noted in previous studies [39, 40].

Strengths and limitations

The strengths of this study include its prospective design and use of clinical diagnosis made using the contemporaneous Sepsis-3 definition as the reference standard. The prospective design allowed the clinical diagnosis of sepsis to be clinically adjudicated by an intensive care physician in cases of doubt. As prospective cohort studies are likely to produce the most accurate sepsis estimates [9], the diagnostic accuracy data generated from this study should be robust.

In terms of limitations, although we could not achieve the target sample size of 500 due to a significant delay in patients’ enrolment due to the SARS-CoV-2 pandemic, we were able to complete the study with 450 patients in a challenging environment of the pandemic. As this study was conducted in one healthcare system in NSW, Australia, the applicability of the results to other healthcare systems is unknown, particularly low-and -middle income countries where endemic pathogens and disease patterns are very different from those where our study was conducted. Lastly, variations in the coding practices and regulations and inter-rater variability in clinical sepsis diagnosis may have impacted sepsis estimates [6, 10, 41–43]. Lastly, as the study was conducted in patients at high risk of sepsis, the positive and negative predictive values may differ in populations where sepsis prevalence is different.

Significance and implications

This study provides the first evidence of the accuracy of a new set of ICD-10-AM codes derived from those used in the GBD sepsis study, currently considered as the most authoritative estimate of sepsis globally, in identifying sepsis in hospital settings. Like other ICD coding methods, the modified GBD implicit method undercounts sepsis cases. That most methods underestimated the number of cases of sepsis as well as the number of associated deaths has significant implications for healthcare providers, funders, and policymakers. Future research should examine false positives and negative cases to identify sources of errors in ICD coding methods and, as recommended by the World Health Assembly Resolution, seek to improve and strengthen methods of using ICD coding to accurately document the global epidemiology of sepsis [44].

Demonstrating sources of error which are common across healthcare systems would allow adjustment or correction of coding methods to provide more standardised estimates of sepsis epidemiology. Findings from our analysis highlight that educating healthcare workers on the importance of clear documentation of sepsis, infection, and organ dysfunction in clinical notes should be a high priority so that coders are able to assign appropriate codes. Moreover, studies are needed in other countries, particularly low- and middle-income countries to generate more representative data on the sources of error in sepsis ICD coding. That would help achieve consensus to derive a standardised method to adjust sepsis estimates using calibrated ICD coding methods.

Conclusion

ICD-10-AM codes adapted from the GBD sepsis study demonstrated a low accuracy in identifying clinically diagnosed sepsis cases using Sepsis 3 criteria. Of the methods assessed, the modified GBD implicit plus method produced the most reliable estimates of sepsis incidence and mortality. All ICD coding methods showed poor agreement with clinical diagnosis of sepsis. Unspecified sepsis, infection, and organ dysfunction codes along with incomplete documentation of causative microorganisms and organ dysfunction contributes significantly to inaccuracies in using the modified GBD codes to identify sepsis.

Author contributions

A.K., S.F., N.H., B.V., A.D., and L.B. contributed to the study conception and design. Data collection was performed by A.K., C.M., K.T., M.S., C.Y., P.M., A.A., F.H., D.B., C.B., G.R., S.R., F.B., B.S., L.T, S.N., S.L., R.K., A.B., A.Z., S.S., D.H., S.F., M.I., T.H., G.C., A.K., B.A., R.S., R.M., D.I., M.L., M.S. and G.F. Data analysis was performed by A.K., S.S. and Y.L. The first draft of the manuscript was written by A.K., and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

Internal funding from The George Institute for Global Health, Professors Finfer and Venkatesh are supported by Leadership Fellowships and Dr Hammond by an Emerging Leader Fellowship from the Australian National Health and Medical Research Council (NHMRC).

Availability of data and materials

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

The study was performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and was approved by the Southwestern Sydney Local Health District Human Research Ethics Committee (approval number 2020/ETH00180, dated 14 May 2020). This was a low/negligible risk observational study using demographic, physiological and patient data which is measured routinely as part of clinical care. Moreover, there was no clinical risk associated for participants as the study was observational in nature and has no potential to interfere with standard treatment. There was no risk to the rights, privacy or professional reputation of carers, health professionals and/or institutions as the study solely concerns with analysis of clinical data collected as part of standard clinical care. Hence, ethical approval was obtained with a waiver of individual patient consent in keeping with local guidelines on the conduct of research in humans and complying with state and Federal privacy laws.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.