SEVERE SEPSIS COHORTS DERIVED FROM CLAIMS-BASED STRATEGIES APPEAR TO BE BIASED TOWARDS A MORE SEVERELY ILL PATIENT POPULATION

Abstract

Objective

The epidemiology of severe sepsis is derived from administrative databases that rely on ICD-9-CM codes to select cases. We compared the sensitivity of two code-abstraction methods in identifying severe sepsis cases using a severe sepsis registry.

Design

Single center retrospective cohort study

Setting

Tertiary care, Academic, University Hospital

Patients

1735 patients with severe sepsis or septic shock

Interventions

None

Measurements

Proportion identified as severe sepsis using two code-abstraction methods, (1) the new specific ICD-9 codes for severe sepsis and septic shock, and (2) a validated method requiring two ICD-9 codes for infection and end-organ dysfunction. Multivariable logistic regression was performed to determine sociodemographics and clinical characteristics associated with documentation and coding accuracy.

Main Results

The strategy combining a code for infection and end-organ dysfunction was more sensitive in identifying cases than the method requiring specific ICD-9 codes for severe sepsis or septic shock (47% vs. 21%). Elevated serum lactate level, (p<0.001), ICU admission (p<0.001), presence of shock (p<0.001), bacteremia as the source of sepsis (p=0.02), and increased APACHE II score (p<0.001) were independently associated with being appropriately documented and coded. The 28-day mortality was significantly higher in those who were accurately documented/coded (41%, compared to 14% in those who were not, p<0.001), reflective of a more severe presentation on admission.

Conclusions

Patients admitted with severe sepsis and septic shock were incompletely documented and under-coded, using either ICD-9 code abstracting method. Documentation of severe sepsis using the new sepsis codes was more common in more severely ill patients. These findings are important when evaluating current national estimates and when interpreting epidemiologic studies of severe sepsis as cohorts derived from claims-based strategies appear to be biased towards a more severely ill patient population.

Introduction

Severe sepsis is a frequently fatal condition, and is currently estimated to be the tenth leading cause of death in the United States (1). Recent epidemiologic estimates of severe sepsis incidence have used retrospective analyses of large administrative databases (2–6) identifying cases based on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Prior to 2003, there were no specific codes that matched the 1992 ACCP/SCCM consensus criteria (7) for severe sepsis and septic shock. Thus, cases were selected based on the abstraction of a combination of two codes representing an infection and an acute end-organ dysfunction as a proxy for the severe sepsis consensus criteria (the Angus coding method) (2). However, the specific methodology employed to abstract these codes has not been standardized across epidemiologic studies. In 2005, a Swiss group abstracted three different cohorts of severe sepsis from a single national database using one of three code abstraction methods (2–3,8) and found that incidence varied from 0.13 per 1000 to 0.43 per 1000 depending on the specific coding strategy (6).

Recognizing the potential limitations of the Angus coding method, in 2003 the Center for Medicare and Medicaid Services (CMS) created criteria-specific codes for the diagnoses of severe sepsis (995.92) and septic shock (785.52), to more accurately identify cases. Although these new criteria-specific codes may improve the accuracy of case selection, the performance of this new method for identifying cases for epidemiologic purposes remains largely unknown.

Based on observations of physician documentation in chart reviews done at our institution, and suggestions from recent, small validation studies (9–10), we hypothesized that severe sepsis and septic shock would be inadequately documented and therefore under-coded. As recent studies suggested that specificity approaches 100% with the new criteria-specific ICD-9 codes (9–10), we focused our study on the sensitivity of these two different coding strategies. Because important, recent studies have relied on claims-based definitions of severe sepsis (10–12), it is critical to understand the limitations of these approaches to inform our understanding of the epidemiology of severe sepsis and to aid in our interpretation of studies which rely on such approaches.

Our primary objective was to determine the sensitivity of the new criteria-specific, severe sepsis codes and to compare them to the Angus coding method. In addition, since physician documentation determines whether codes are assigned, we hypothesized that variability in the accuracy of coding of severe sepsis and septic shock may result from differences in patient demographics or clinical characteristics. Similarly, we hypothesized that patient outcome may be associated with variability in code assignments. To test these hypotheses, we compared the clinical characteristics and patient outcomes of those who were accurately coded and those who were not.

Materials and Methods

The Institutional Review Board of the University of Pennsylvania approved the study with an informed consent waiver.

Study Population

We conducted a retrospective cohort study of patients admitted through the Emergency Department (ED) at the Hospital of the University of Pennsylvania (HUP) with the diagnosis of severe sepsis or septic shock. The details of our case selection strategy have been described previously (13). In brief, we identified all patients admitted from the HUP ED who met established consensus criteria for severe sepsis or septic shock (7,14) from January 1, 2005 through April, 2009. We focused the present study on patients admitted through the ED given evidence that 67 – 79% of patients present to the hospital with severe sepsis (10, 15). Furthermore, this approach permitted us to identify medical and surgical cases as well as ward and ICU admissions in a systematic fashion from their initial point of contact with the healthcare system. As such, the opportunity to recognize, document, and subsequently code cases was similar across cases, beginning from the time of admission. We opted not to augment the severe sepsis cohort with de novo cases of severe sepsis from the general ward or ICUs as there is no established gold standard for ascertaining de novo cases of severe sepsis during the hospitalization and because de novo cases (e.g., procedure-related, hospital-acquired infections) are substantially different than admitted severe sepsis cases. Further, because de novo cases were admitted for an alternative explanation, it is plausible that recognition and coding differ significantly between cases admitted with severe sepsis and de novo cases.

We defined sepsis as suspected infection associated with two out of four systemic inflammatory response syndrome (SIRS) criteria, severe sepsis as sepsis associated with hypotension, hypoperfusion, or organ dysfunction, and septic shock as sepsis associated with refractory hypotension, in accord with established definitions (7,14). We excluded repeat patient visit(s), patients discharged from the ED, patients who left against medical advice, trauma patients, and patients who failed to meet criteria for severe sepsis. Regarding the latter exclusion, we reviewed details from the hospitalization, including the discharge summary, to ensure that the criteria for severe sepsis were sustained beyond the ED. For the present study, we reviewed 100 randomly selected charts to validate severe sepsis during the hospitalization. In 2 of the 100 charts, an alternative diagnosis was considered at discharge to explain the systemic inflammatory response syndrome and acute end-organ dysfunction (a patient with new-onset seizure and respiratory failure and a patient with delirium and shock attributed to heat stroke). We maintained these patients in the cohort as each received antibiotics for the majority of their hospitalization for a suspected infection. Each of the remaining 98 cases were treated during the hospitalization as an identified infection.

Data Collection

Trained investigators, blinded to the ICD-9 codes assigned, abstracted clinical data using a pre-drafted case report form. The case report form included sociodemographic data, comorbid conditions, ED vital signs and laboratory results, interventions received in the ED to gauge intensity of care (e.g., use of mechanical ventilation, use of vasoactive agents, initiation of early goal-directed therapy (EGDT (16)), admission location (ward or intensive care unit (ICU) and service (medical or surgical), and outcomes (28 and 60-day mortality). As such, clinical characteristics reflected those which were present upon admission. Mortality status was assessed using the hospital record and the Social Security Death Index (17).

Exposures

We a priori hypothesized that sociodemographics, certain comorbidities, etiology of infection, and illness severity would be associated with an increased likelihood of being coded with an ICD-9 code specific for severe sepsis or septic shock (see Table 1). Our candidate risk factors were selected based on the reasoning that patients who are more likely to be recognized as fulfilling severe sepsis criteria will be more likely to be subsequently documented and consequently coded with an ICD-9 code specific for severe sepsis or septic shock.

Table 1.

Patient characteristics for the overall severe sepsis cohort, by whether patients were assigned ICD-9 specific codes for severe sepsis or septic shock, and by whether patients were assigned end-organ dysfunction and infection code assignments (the Angus coding method (2)).

Variables	All Patients (n=1735)	ICD-9 Specific Code Assignments (995.92 and 785.52)			Combining End-Organ Dysfunction and Infection Code Assignments
Patient Characteristics at Admission		Assigned (n=373)	Not Assigned (n=1,362)	p-value	Assigned (n=818)	Not Assigned (n=917)	p-value
Age, years	58 (46 – 71)	60 (50 – 72)	58 (45 – 70)	0.004	59 (49–73)	57 (44 – 69)	<0.001
Gender (female)	45.1%	47.4%	44.5%	0.31	45.8%	44.5%	0.57
Race
White	35.8%	32.7%	36.7%	0.22	33.5%	38.0%	0.14
Black	45.5%	46.1%	45.4%		47.7%	43.6%
Other	18.6%	21.2%	17.9%		18.8%	18.4%
Apache II baseline	16 (12 – 20)	20 (16 – 25)	15 (11 – 19)	<0.001	18 (14 – 23)	14 (10 – 18)	<0.001
Lactate Values, mmol/L	2.8 (2.0 – 4.3)	4.4 (2.7 – 6.4)	2.6 (1.9 – 3.7)	<0.001	3.3 (2.1 – 5.3)	2.6 (1.9 – 3.5)	<0.001
Nadir Systolic Blood Pressure, mmHg	96 (82 – 114)	83 (72 – 99)	100 (86 – 117)	<0.001	89 (77 – 105)	103 (89 – 118)	<0.001
Presence of Shock*	18.5% (n=321)	44.8%	11.3%	<0.001	29.5%	8.7%	<0.001
ICU Admission	51.7%	86.6%	42.1%	<0.001	69.9%	35.4%	<0.001
Underlying Comorbidity
Oncology	32.8%	36.5%	31.8%	0.09	33.6%	32.1%	0.49
Immunosuppression	38.4%	42.1%	37.4%	0.10	39.0%	38.0%	0.65
Chronic kidney disease	15.0%	14.5%	15.2%	0.73	16.3%	14.0%	0.18
Congestive heart failure	10.9%	11.9%	10.6%	0.49	13.3%	8.8%	0.003
COPD	7.0%	8.6%	6.5%	0.37	7.8%	6.2%	0.45
End-stage liver disease	5.4%	8.8%	4.5%	0.001	7.1%	3.9%	0.004
Diabetes mellitus	27.5%	24.9%	28.2%	0.21	28.4%	26.7%	0.44
Site of Infection **
Bacteremia	14.2%	19.0%	12.8%	0.002	14.7%	13.7%	0.58
Respiratory Infection	30.2%	28.7%	30.7%	0.56	30.0%	30.5%	0.93
Urinary Tract Infection	23.6%	23.3%	23.6%	0.89	27.0%	20.5%	0.003
GI Infection	16.0%	19.0%	15.2%	0.07	17.2%	14.9%	0.19
Soft-Tissue Infection	10.3%	9.9%	10.4%	0.92	9.7%	10.9%	0.39
Medical Admission	85.5%	87.5%	85.0%	0.21	88.9%	82.5%	<0.001
Interventions at Admission
Use of mechanical ventilation	10.3%	25.5%	6.2%	<0.001	15.8%	5.4%	<0.001
Use of vasoactive agents	7.8%	23.0%	3.6%	<0.001	13.9%	2.3%	<0.001
EGDT Initiated in ED	38.8% (261 of 672 eligible)	51.5% (141 of 274 eligible)	30.2% (120 of 398 eligible)	<0.001	44.3% (193 of 436 eligible)	28.8% (68 of 236 eligible)	<0.001
Outcomes
28-Day Mortality	19.5%	41.0%	13.6%	<0.001	29.0%	11.0%	<0.001
60-Day Mortality	22.6%	45.8%	16.3%	<0.001	33.0%	13.4%	<0.001

Continuous variables are presented as median and interquartile range. Categorical data are presented as proportions.

^*Shock was defined as systolic blood pressure less than 90 mm Hg after fluid resuscitation (1500 mL) or use of vasoactive agents (13–14).

^**Site of infection was not mutually exclusive, as a patient could have multiple infectious sources identified.

Given well-established disparities in the incidence and mortality of sepsis based on age, male gender, and non-white race (3), we considered age, gender, and race/ethnicity as potential factors which would be associated with being correctly coded. We hypothesized that immunocompromised patients (oncology, HIV/AIDS, and organ transplantation) would be more likely to be coded as severe sepsis or septic shock (18). We hypothesized that respiratory infections and blood-stream infections, given their association with greater morbidity and mortality, compared to urinary infectious source, would be more likely to be coded correctly. We used the following variables as markers of illness severity: APACHE II scores (19) calculated based on data at ED presentation, admission to an ICU, patients with septic shock and increased serum lactate levels, intensity of care, and those with higher mortality. Last, we considered whether the frequency of accurate coding has increased over time in 1-year intervals.

Outcomes

Our primary aim was to determine the sensitivity of two code-abstraction methods to identify severe sepsis and septic shock. In the first method, we categorized patients as correctly coded if they received specific ICD-9 codes for severe sepsis (ICD-9 995.92) or septic shock (ICD-9 785.52). In the second, we used the Angus coding method (2).

ICD-9 codes were recorded for each patient using the Horizon Performance Manager (McKesson Information Solutions, Alpharetta, GA), a hospital administrative database. Consistent with established coding guidelines, ICD-9 code assignments are based on all available physician and physician extender documentation from the entire patient encounter, including documentation in the ED, during the hospitalization, and in the discharge summary (20). At HUP, an Electronic Medical Record (EMR) is used in the ED and in the Surgical ICUs. Neither the medical or surgical wards, nor the Medical ICU, use an EMR. In those locales where an EMR is not used, paper templates are used to support physician billing, but not to support hospital coding. Coding at HUP is assigned by certified coders. In the 100 randomly selected charts reviewed to validate severe sepsis during the hospitalization, we also reviewed clinician documentation to confirm the accuracy of the coding assignments.

Our secondary aim was to identify patient-level factors associated with being correctly coded as having severe sepsis or septic shock. For these analyses, the outcome was whether specific ICD-9 codes for severe sepsis (ICD-9 995.92) or septic shock (ICD-9 785.52) were assigned. We performed separate analyses where the outcome was whether severe sepsis was assigned using the Angus coding method.

Statistical Analysis

For our primary aim, we calculated the sensitivity for both code abstraction methods and present them as proportions. For our secondary aim, we performed univariate analyses to identify which exposures were statistically associated with each of the code abstraction methods. Comparisons between patients who were and were not accurately coded were tested using the Student’s t-test for continuous variables and the chi-squared statistic or Fisher’s exact test for categorical variables. To determine whether the frequency of accurate coding has increased over time, we used the non-parametric test for trend (21).

We conducted multivariable logistic regression analyses to adjust for potential confounding in the association between clinical characteristics and whether a patient was assigned a specific ICD-9 code for severe sepsis or septic shock and present the results as adjusted odds ratios (OR) with 95% confidence intervals (CI). We tested for multicollinearity using variance inflation factors. We included each candidate risk factor and potential confounder in the multivariable model, which was associated with accurate coding at a significance level <0.20 in univariate analyses. We assessed model discrimination by calculating the area under the receiver operating characteristic curve (AUC). We assess model calibration using the Hosmer-Lemeshow test statistic.

We conducted two sensitivity analyses. First, we forced each clinical variable in a separate multivariable model to determine if we identified the same risk factors as in our primary analyses. Then, in a backward elimination, we removed the non-significant (p>0.05) variables from both multivariable models to determine if we identified the same risk factors in more parsimonious models. Analyses were conducted using Stata 11.0 software (Stata Datacorp, College Station, TX) (22).

Results

From 2005 to April 2009, 1735 patients were admitted with severe sepsis or septic shock. The mean age of the cohort was 58 years (range 19 to 93) and 54.9% (n=952) were male (see Table 1). The majority (51.7%) of patients were admitted to an ICU and 85.5% were admitted to a medical service. Of all admissions, 18.5% presented with septic shock in the ED. Overall, the 28-day mortality was 19.5%; mortality was significantly higher in the patients admitted with septic shock, compared to those with severe sepsis (36.4% vs. 15.6%, p<0.001).

Of 1735 patients with severe sepsis, 373 (21.5%) received an ICD-9 code for severe sepsis or septic shock. As shown in Table 2, the ICD-9 code for severe sepsis was assigned to 20.5% (n=355) of all subjects. The sensitivity was higher for the 321 subjects presenting with septic shock; wherein 49.5% (n=159) of the septic shock cases received an ICD-9 code for severe sepsis. The ICD-9 code for septic shock had a sensitivity of 42.4% (n=136) for identifying the 321 subjects presenting with septic shock. In contrast, the Angus coding method had the highest sensitivity: 47.0% for all subjects, and 74.9% for those presenting with septic shock. There was limited incremental value of a combined approach wherein severe sepsis cases were identified if either the ICD-9 specific codes (995.92, 785.52) or the Angus coding method were assigned. Specifically, there were 818 cases identified through the Angus method (see Table 1) and only an additional 9 patients (N=827) were identified through such a combined approach.

Table 2.

Sensitivities of two different code abstraction methods for identifying cases of severe sepsis and septic shock determined by patient-level data.

Code Abstraction Method	Sensitivity to Identify Severe Sepsis Cases, n=1735*	(95% CI)
1. Severe Sepsis (ICD-9 specific coding method, 995.92)	20.5%	(18.6% – 22.4%)
2. Combining End-Organ Dysfunction and Infection Codes (the Angus Coding Method)	47.2%	(44.8% – 49.5%)
	Sensitivity to Identify Septic Shock Cases, n=321*	(95% CI)
1. Severe Sepsis (ICD-9 specific coding method, 995.92)	49.5%	(44.0% – 55.0%)
2. Septic Shock (ICD-9 specific coding method, 785.52)	42.4%	(37.0% – 47.8%)
3. Combining End-Organ Dysfunction and Infection Codes (the Angus Coding Method)	75.1%	(70.4% – 79.8%)

Categorical data are presented as proportions.

^*Cases of septic shock (n=321) were encompassed within the severe sepsis (n=1735) population.

In our chart review of 100 randomly selected charts of patients with severe sepsis for accuracy of both ICD-9 coding and clinician documentation, we found that the low proportion of cases assigned a code for severe sepsis is explained primarily by deficiencies in clinician documentation. Specifically, of the 27 (27%) cases coded as severe sepsis or septic shock all (100%) were correctly documented as such in the medical record. Whereas, out of the remaining 73 (73%) cases that were not coded as severe sepsis, 3 (4%) were documented as such.

To further explain these results, we determined the patient and clinical factors associated with being assigned an ICD-9 specific code for severe sepsis or septic shock. As shown in Table 1, patients who received an ICD-9 specific code for severe sepsis or septic shock (n=373) were older (p=0.004), more severely ill, as measured by higher baseline APACHE II scores (p<0.001), had higher initial serum lactate levels (p<0.001), lower systolic blood pressure measurements at presentation (p<0.001), a greater incidence of shock (p<0.001), and an increased likelihood of being admitted to an ICU (p<0.001). Regarding the latter finding, whereas 36.0% (n=323) of the 897 patients admitted to an ICU received an ICD-9 specific code for severe sepsis and/or septic shock, only 6.0% (n=50) of the 838 ward admissions did. Furthermore, consistent with the severity of illness observations, we found that patients assigned an ICD-9 specific code for severe sepsis or septic shock received greater intensity of care upon admission and experienced worse outcomes (Table 1). Of note, there was no increase in the frequency of accurate coding over time (p=0.33). A comparison of those assigned ICD-9 severe sepsis codes based on the Angus coding method revealed similar findings to the comparison for whether an ICD-9 specific code for severe sepsis or septic shock was assigned (Table 1).

In a multivariable logistic regression model (Table 4), we found that higher baseline APACHE II scores (p<0.001), the presence of shock at admission (<0.001), higher initial serum lactate levels (<0.001), bacteremia as the source of infection (p=0.02), and admission to an ICU (p<0.001) remained independently associated with receiving an ICD-9 code specific for severe sepsis or septic shock. The model discriminated patients who were assigned an ICD-9 code specific for severe sepsis and/or septic shock with an AUC of 0.83 and the model was well calibrated (p=0.14).

Table 4.

Sensitivity Analyses: Multivariable logistic regression model of adjusted odds ratios for being assigned an ICD-9 specific code for severe sepsis or septic shock (995.92 and 785.52) where each candidate variable was forced into the model.

Complete Model (n=1721)*	Adjusted Odds Ratio (95% CI)	p-value
Gender (male)	0.74 (0.56 – 0.98)	0.04
Apache II Score (baseline) **	1.06 (1.04 – 1.09)	<0.001
Presence of Shock **	2.94 (2.16 – 4.00)	<0.001
Serum Lactate **	1.13 (1.07 – 1.18)	<0.001
Etiology of Sepsis:
Bacteremia	1.73 (1.19 – 2.53)	0.004
ICU Admission	4.86 (3.38 – 7.00)	<0.001
Time (year) **	1.17 (1.04 – 1.30)	0.006
Sensitivity Analysis †	Adjusted Odds Ratio (95% CI)	p-value
Gender (male)	0.77 (0.59 – 1.00)	0.052
Apache II Score (baseline) **	1.06 (1.04 – 1.08)	<0.001
Presence of Shock **	3.02 (2.23 – 4.09)	<0.001
Serum Lactate **	1.14 (1.09 – 1.19)	<0.001
Etiology of Sepsis:
Bacteremia	1.59 (1.11 – 2.28)	0.01
ICU Admission	4.44 (3.12 – 6.32)	<0.001
Time (year)	1.20 (1.09 – 1.33)	<0.001

^*Age, systolic blood pressure, and admission type (medical vs. surgical) were found to be collinear with Apache II and were therefore not included in the complete model. The remaining variables included in the model (race, comorbidities, and other sites of infection) were not significantly associated with being assigned an ICD9 code for severe sepsis or septic shock.

^**For each 1-unit increase in illness severity measure, serum lactate, or time in years. Shock was defined as systolic blood pressure less than 90 mm Hg after fluid resuscitation (1500 mL) or use of vasoactive agents (13–14).

^†In the sensitivity analyses, a more parsimonious model was created wherein variables that were not significantly associated with being assigned an ICD-9 specific code for severe sepsis or septic shock were removed.

Our sensitivity analyses provide additional support that severity of illness was significantly associated with being assigned an ICD-9 code specific for severe sepsis or septic shock. Specifically, in the more parsimonious model for the primary analyses, each of the clinical variables associated with being assigned an ICD-9 specific code for severe sepsis or septic shock remained significant (Table 3). In the multivariable models which included all candidate variables, we identified the same clinical variables as were found to be associated with being assigned an ICD-9 specific code for severe sepsis or septic shock in our primary analyses (Table 4). In addition, we identified that females were more likely to be assigned an ICD-9 specific code for severe sepsis or septic shock and, over time, the frequency of assignment was more common; however, in the more parsimonious models, the association between gender and code assignment was no longer statistically significant by conventional standards.

Table 3.

Multivariable logistic regression model of adjusted odds ratios for being assigned an ICD-9 specific code for severe sepsis or septic shock (995.92 and 785.52).

Primary Analysis (n=1735)	Adjusted Odds Ratio (95% CI)	p-value
Apache II Score (baseline) *	1.06 (1.04 – 1.09)	<0.001
Presence of Shock *	2.81 (2.08 – 3.79)	<0.001
Serum Lactate (mmol/L) *	1.14 (1.08 – 1.19)	<0.001
Etiology of Sepsis:
Bacteremia	1.51 (1.06 – 2.17)	0.02
Gastrointestinal	1.35 (0.94 – 1.93)	0.10
Comorbid Conditions:
Immunosuppression **	1.22 (0.89 – 1.66)	0.21
Oncology **	1.05 (0.77 – 1.45)	0.74
End-stage liver disease	1.10 (0.64 – 1.89)	0.72
ICU Admission	4.14 (2.92 – 5.87)	<0.001
Sensitivity Analysis †	Adjusted Odds Ratio (95% CI)	p-value
Apache II Score (baseline) *	1.06 (1.04 – 1.09)	<0.001
Presence of Shock	2.87 (2.13 – 3.86)	<0.001
Serum Lactate *	1.14 (1.09 – 1.19)	<0.001
Etiology of Sepsis:
Bacteremia	1.48 (1.04 – 2.12)	0.03
ICU Admission	4.02 (2.84 – 5.70)	<0.001

^*For each 1-unit increase in illness severity measure or serum lactate (mmol/L). Shock was defined as systolic blood pressure less than 90 mm Hg after fluid resuscitation (1500 mL) or use of vasoactive agents (13–14).

^**Neither immunosuppression, nor oncology, were significantly associated with being assigned an ICD9 code for severe sepsis or septic shock when the other variable was removed from the model.

^†In the sensitivity analysis, a more parsimonious model was created wherein variables that were not significantly associated with being assigned an ICD-9 specific code for severe sepsis or septic shock were removed.

Discussion

We found that patients who were admitted to the hospital with a confirmed diagnosis of severe sepsis or septic shock were infrequently assigned one of the new ICD-9 specific codes for these diagnoses. We determined that this deficiency in coding is attributable to inadequacies in clinician documentation. Specifically, clinicians infrequently document the presence of severe sepsis. In addition, regardless of the coding approach used, clinicians were less likely to document severe sepsis or septic shock when patients were less severely ill, reflected by lower Apache II scores, absence of shock or need for an ICU bed, lower serum lactate levels, and lesser intensity of care at the time of admission; while other clinical factors including comorbidities did not have an impact.

Epidemiology studies are vital to determine disease characteristics, the types and numbers of patients affected, and clinical outcomes. Incidence and outcome data are critical for clinicians caring for such patients and important in determining healthcare resource allocation. Since the measurement of sepsis incidence through prospective patient identification is challenging and often limited to single-center studies, epidemiologic estimates of sepsis and its related outcomes have been made largely from administrative data sets (2–6, 10–12).

Our data suggest that if the specific severe sepsis and septic shock ICD-9 codes are used for epidemiologic purposes, they may underestimate disease incidence and overestimate morbidity and mortality as cohorts derived from this claims-based approach appear to be biased towards a more severely ill patient population. Although the traditional Angus coding method for identifying cases of severe sepsis was more sensitive in identifying these patients, it did not identify half the severe sepsis cases and was also biased towards a more severely ill patient population. As a strategy which relies on documentation of a code for an infection in combination with an organ failure, a plausible explanation for this discrepancy, which requires validation, is that physicians do not document organ dysfunction adequately either. Since this traditional coding method (2) has been the strategy employed in several landmark studies of severe sepsis epidemiology, the incidence of severe sepsis reported previously may represent a gross underestimate of the true value. Similarly, this limitation may impact the conclusions that can be drawn from epidemiologic studies that rely on the traditional coding strategy when describing clinical characteristics, processes of care, or outcomes associated with severe sepsis (10–12).

A major strength of our study is that we used patient-level data to identify cases of severe sepsis and to build a robust cohort of severe sepsis. We then tested the performance of two different claims-based approaches to identify severe sepsis within our cohort. We found that only a fraction of the true severe sepsis cases were documented appropriately by clinicians. Clinicians are notoriously poor at documenting completely and accurately. This is similar to what we and others have found with clinician documentation of acute respiratory distress syndrome (ARDS), which likely relates to difficulty recognizing clinical syndromes that require multiple complex criteria to identify. In addition, the finding that the severe sepsis documentation deficiencies were seen in patients with a much lower acuity is also not surprising and is consistent with what we found in a prior study of patients with ALI/ARDS (23). The more severely ill a patient is, the more likely a clinician will describe the problem and document the etiology in the medical record. This has important implications for future epidemiology studies of severe sepsis, as severe sepsis cohorts derived from claims-based strategies appear to be biased towards a more severely ill patient population.

Another intriguing hypothesis generated by our results, is that a lack of documentation reflects a lack of recognition. If this notion is correct, then perhaps patients with severe sepsis who are not recognized as such receive inadequate care by their clinicians. However, we found better clinical outcomes in those patients who were not documented appropriately; this discordance is likely explained by the fact that this group has a much lower acuity. It is possible that outcomes would be even better in the non-documented sepsis cases had they been appropriately recognized and managed. Future studies are required to determine whether opportunities exist to improve care in the undocumented severe sepsis cases.

Our findings also have important implications for hospital reimbursement as well as individual clinician and institutional performance and quality ratings. The failure to appropriately document severe sepsis may significantly impact adversely on reimbursement because the diagnosis related group assigned may reflect a much less complex diagnosis that warrants lower payment. Furthermore, in the last decade hospitals are increasingly being compared and rated using poorly validated measures of quality and performance. Two such metrics include the overall mortality rate and sepsis specific mortality rate (24–27). With both metrics, to control for patient acuity the measurement is normalized by expressing the overall mortality or sepsis mortality as a ratio of the observed to expected values. The expected mortality is calculated using ICD-9 codes assigned at discharge. Since these codes are derived from what the clinician documents, performance and ratings will be driven, in part, by how accurate clinicians document their patients’ clinical problems, e.g. severe sepsis.

There are several important limitations to our study. First, our study was designed to test the sensitivity of two claims-based strategies to identify severe sepsis. Our study design does not permit us to comment on other test characteristics of these strategies; however, our approach is reasonable given recent studies which suggest that specificity approaches 100% (9–10). In addition, since our search strategy requires the presence of two or more systemic inflammatory response syndrome (SIRS) criteria, we may have underestimated the number of patients with severe sepsis admitted from the ED during the study period. For example, Shapiro et al demonstrated that in a cohort of 3102 visits for sepsis, that 34% of patients with severe sepsis and 24% of patients with septic shock did not meet SIRS criteria. However, the absence of these patients in all likelihood biases toward an overestimate of the number of patients correctly coded with sepsis ICD-9 codes (28). Second, as a study conducted at a single academic center with a focus on cases admitted with severe sepsis, we acknowledge that our findings may not generalize to de novo cases of severe sepsis or to hospitals that utilize different documentation practices (e.g., standardized electronic physician documentation). As such, our study warrants confirmation in a multi-center study and further study is necessary to examine sepsis recognition across the continuum of care. Third, our study design focused on clinical characteristics present upon admission. As such, it is unclear how processes of care or other variables (length of stay) during the hospitalization influence documentation and coding. Fourth, as we did not abstract details at the physician level, important questions about whether different types of physicians (hospitalist, intensivist, etc.) are more likely to document severe sepsis remain unanswered and it is unclear whether or how such data would have altered our results.

Conclusion

We found that patients admitted with severe sepsis and septic shock were incompletely documented and under-coded. The cases identified through either of the claims-based strategies studied were the most severely ill. This study raises an important question about the accuracy of current estimates of severe sepsis incidence and may inform the interpretation of epidemiologic studies of severe sepsis as cohorts derived from claims-based strategies appear to be biased towards a more severely ill patient population.