Identifying Patients with Severe Sepsis Using Administrative Claims: Patient-Level Validation of the Angus Implementation of the International Consensus Conference Definition of Severe Sepsis

Abstract

Background

Severe sepsis is a common and costly problem. Although consistently defined clinically by consensus conference since 1991, there have been several different implementations of the severe sepsis definition using ICD-9-CM codes for research. We conducted a single center, patient-level validation of one common implementation of the severe sepsis definition, the so-called “Angus” implementation.

Methods

Administrative claims for all hospitalizations for patients initially admitted to general medical services from an academic medical center in 2009–2010 were reviewed. On the basis of ICD-9-CM codes, hospitalizations were sampled for review by three internal medicine-trained hospitalists. Chart reviews were conducted with a structured instrument, and the gold standard was the hospitalists’ summary clinical judgment on whether the patient had severe sepsis.

Results

3,146 (13.5%) hospitalizations met ICD-9-CM criteria for severe sepsis by the Angus implementation (“Angus-positive”) and 20,142 (86.5%) were Angus-negative. Chart reviews were performed for 92 randomly-selected Angus-positive and 19 randomly-selected Angus-negative hospitalizations. Reviewers had a kappa of 0.70. The Angus implementation’s positive predictive value (PPV) was 70.7% (95%CI: 51.2%, 90.5%). The negative predictive value was 91.5% (95%CI: 79.0%, 100%). The sensitivity was 50.4% (95%CI: 14.8%, 85.7%). Specificity was 96.3% (95%CI: 92.4%, 100%). Two alternative ICD-9-CM implementations had high PPVs but sensitivities of less than 20%.

Conclusions

The Angus implementation of the international consensus conference definition of severe sepsis offers a reasonable but imperfect approach to identifying patients with severe sepsis when compared with a gold standard of structured review of the medical chart by trained hospitalists.

INTRODUCTION

Severe sepsis is a common cause of hospitalization, likely more common than acute myocardial infarction. ^1,2 The incidence of severe sepsis increases sharply with age, leading it to be termed “a quintessential disease of the aged”. ³ Not only is severe sepsis the most common non-cardiac cause for intensive care unit (ICU) use, it has emerged as a major driver of hospital costs in the United States. ⁴ Severe sepsis is a condition associated with high inpatient mortality, ¹ and also enduring effects on patient mortality, ⁵ health care spending, ^6,7 disability, ⁸ cognitive function, ⁸ and quality of life. ^9,10 Despite its importance, guidance on how to study severe sepsis using administrative databases is lacking.

Severe sepsis was defined by a 1991 consensus conference as a syndrome that occurs when proven or suspected infection leads to organ dysfunction. ¹¹ This definition intentionally encompasses a wide range of common reasons for hospitalization, from vasopressor-dependent septic shock in the ICU to pneumonia with hypoxemia or a urinary tract infection causing acute renal failure. The fundamental definitions, presented in Table 1, were reaffirmed in a 2001 consensus conference. ¹² The consensus definition emphasizes the common host response rather than particular inciting infections, ¹³ in accordance with contemporary mechanistic biologic research which indicates that much of the damage of severe sepsis comes not from direct attack by microorganisms, but rather by a poorly moderated immunologic and coagulopathic response to those organisms. ^14–16 Therapeutic research is focused primarily on moderation of this host response.^17,18

Table 1. International Consensus Conference Distinctions in the Definition of Severe Sepsis.

This 2001 Conference was sponsored by the Society for Critical Care Medicine, the European Society of Intensive Care Medicine, the American College of Chest Physicians, the American Thoracic Society, and the Surgical Infection Society. ^12,32

SIRS: Systemic Inflammatory Response Syndrome (a body-wide inflammatory response)

Sepsis: SIRS caused by suspected or proven infection

Severe Sepsis: Sepsis that causes acute organ dysfunction

Septic Shock: Severe sepsis where the acute organ dysfunction leads to tissue hypoperfusion.

The international consensus conference definition has been used to define enrollment criteria for clinical trials and is integral to evidence-based bedside management. ¹³ This definition has proved useful for epidemiologic studies. ^19–21 Given the limits of prospective case ascertainment, as in other disease states and comorbidity scores, ^22–26 administrative implementations of the international consensus conference have been published using ICD-9-CM codes.

Among the most common administrative implementations for severe sepsis is the so-called “Angus” implementation. ^{6–8,27–29} This implementation has been cited more than 2,000 times as of December 2011 (Web of Science). This implementation was validated by demonstrating that it identifies a population of patients similar in aggregate to one identified by nursing-led prospective assessment, but not that the same patients are so identified. ^19,30 Despite this large number of citations, we are not aware of any patient-level validation comparing the Angus implementation to a gold-standard of physician review. We therefore conducted such a validation at a large, tertiary care medical center in the United States.

METHODS

Hospitalizations

We examined all hospitalizations of adult patients (≥ 18 years) who were initially admitted to non-ICU medical services at the University of Michigan Health System during 2009–2010. Transfers from other hospitals were excluded. Hospitalizations were sampled at random for review. Hospitalizations meeting the Angus implementation of severe sepsis (defined below) were proportionately sampled from a 2×2 classification based on any need for ICU care after admission, and length of stay greater or less than 6 days. This proportionate sampling ensured that potential sources of variability in coding accuracy are appropriately represented in the analysis sample as compared to the broader patient population. We also sampled a smaller number of patients who were Angus-definition-negative. Sample sizes were selected combining considerations of the likely confidence interval of the positive predictive values and considerations of the resources available to conduct the study. All analyses of measurement characteristics adjust for differential sampling weights.

Implementations of Severe Sepsis Definition

A SAS algorithm for the Angus implementation of severe sepsis is presented in Appendix 2. The text labels for the codes are provided in the Appendix in the original description of the implementation. ¹ In this implementation, there are two ways for a hospitalization to become “Angus-positive”. If explicit codes for severe sepsis (995.92) or septic shock (785.52) are found, then the hospitalization is Angus-positive. If not, all ICD-9-CM diagnosis codes are reviewed for an infection code. If an infection code is present, then the ICD-9-CM diagnoses and procedure codes are examined for codes for acute organ dysfunction. If both an infection and an acute organ dysfunction code are found, then the patient is Angus-positive. Otherwise, the hospitalization is Angus-negative.

As secondary analyses, we considered the measurement characteristics of two alternative approaches. The “explicit diagnosis implementation” identified hospitalizations with severe sepsis or septic shock (codes 995.92 or 785.52 in any position). The “Martin implementation” ³¹ labeled as severe sepsis those hospitalizations with infection defined as any of septicemia (038), septicemic (020.0), bacteremia (790.7), disseminated fungal infection (117.9), disseminated candida infection (112.5), or disseminated fungal endocarditis (112.81) – if those hospitalizations also had an organ dysfunction code using the same dysfunction list as Angus. Alternatively, a hospitalization would be “Martin-positive” if it was coded with either of the explicit diagnosis codes.

In our analysis, we compared algorithms that used all diagnosis codes available in the internal claims records (more than 20), or those that mimicked the Medicare files UB-92 restriction to only 10 diagnoses, and found that they classified all patients in precisely the same way.

Chart Review for Gold Standard

All chart reviews were performed by practicing hospitalists using a structured instrument. (See Appendix 1 for instrument.) Reviewers first identified whether the patient was infected. They then reviewed the chart for evidence of each organ failure as defined in the 2001 international consensus conference in Table 1 of the report, ³² and also determined the extent to which those organ failures were likely caused by the response to the infection versus another mechanism. Having reviewed each component, the reviewer was then asked whether, in his/her clinical judgment, this patient had severe sepsis during this hospitalization. Initially, 27 charts were reviewed with discrepancies reconciled item by item by all three reviewers. These 27 training patients were Angus-positive in order to insure a consistent interpretation of the international consensus conference definition. An additional 30 charts were then reviewed by two reviewers each and again all discrepancies were reconciled among all three reviewers. After the initial training charts, reviewers were blinded to the status of the hospitalizations, which included a mix of Angus-negative and Angus-positive charts.

Analyses

The measurement characteristics were defined in the standard ways, as illustrated in Table 2. ³³ Initial data extraction was done in SAS 9.1 (SAS Institute, Cary, NC), and all analyses were conducted in Stata 12.0 (StataCorp LP, College Station, TX). Measurement characteristics were calculated using the survey functions to adjust for survey weights, and binomial confidence intervals provided, truncated at 0% or 100%. 95% confidence intervals are provided for estimated proportions. This project was approved by the University of Michigan Institutional Review Board.

Table 2.

Definitions Used in the Study

		Medical Record Abstraction Gold Standard
		Sepsis	Not-Sepsis
Claims-Based Implementation	Sepsis	A	B
	Not-Sepsis	C	D

Sensitivity: A/(A+C)

Specificity: D/(B+D)

Positive Predictive Value: A/(A+B)

Negative Predictive Value: D/(D+C)

RESULTS

We examined 23,288 hospitalizations, of which 3,146 (13.5%) were Angus-positive and 20,142 (86.5%) were Angus-negative. Among these, a sample of 111 eligible hospitalizations was identified (92 Angus-positive, 19 Angus-negative). After training with 27 hospitalizations, reviewers had a kappa of 0.70 when examined over the next 30 consecutive chart-reviews. Training hospitalizations had been randomly sampled and were included in this analysis after adjudication. Of the 111 patients whose hospitalizations were examined, 48 (43.2%) patients were men, and the mean age was 61.4 years (SD: 17.8); 18.0% involved care in an ICU during their stay. Fourteen hospitalizations (12.6% of the total, 15.2% of Angus-positive patients) had an explicit ICD-9-CM code for severe sepsis or septic shock. After review by the hospitalists, 63/111 (57%) hospitalizations were judged to have had severe sepsis.

The Angus implementation had a positive predictive value (PPV) of 70.7% (95%CI: 51.2%, 90.5%) in analyses taking into account sampling weights. The negative predictive value (NPV) was 91.5% (95%CI: 79.0%, 100%). The sensitivity was 50.4% (95%CI: 14.8%, 85.7%), with 3 of 19 Angus-negative cases being found to have severe sepsis by hospitalist review. Specificity was 96.3% (95%CI: 92.4%, 100%). The measurement characteristics were qualitatively similar for patients who were and were not ever admitted to the ICU, but with substantial imprecision in these subgroup analyses. For example, the PPV was 70.1% (95%CI: 48.3%, 91.9%) among those never admitted to the ICU and 76.4% (95%CI: 53.9%, 98.9%) in those patients who did require ICU care.

Measurement characteristics of other implementations are summarized in Table 3. The explicit diagnosis of severe sepsis or septic shock had a PPV of 100% (95%CI: 76.8%, 100%), with 9.2% sensitivity (95%CI: 0%, 19.2%). The Martin-implementation had a PPV of 97.6% (95%CI: 92.4%, 100%) with 16.9% sensitivity (95%CI: 1.6%, 32.2%).

Table 3.

Measurement Characteristics of Alternative Definitions of Severe Sepsis Against a Gold Standard of Hospitalist Structured Implicit Review

	Angus Implementation	Explicit Diagnosis Code	Martin Implementation
PPV	70.7%	100%	97.6%
95% CI	51.1%, 90.4%	76.8%, 100%	92.4%, 100%
NPV	91.5%	86.0%	87.0%
95% CI	79.0%, 100%	73.5%, 98.4%	74.7%, 99.3%
Sensitivity	50.3%	9.3%	16.8%
95% CI	14.8%, 85.7%	0%, 19.3%	1.6%, 32.2%
Specificity	96.3%	100%	99.9%
95% CI	92.4%, 100%	92.6%, 100%	99.8%, 100%

Narrative Analysis of False Positives & False Negatives of Angus Implementation

Three charts were Angus-negative but had severe sepsis. In the first case, the patient had cholangitis with resulting acute renal failure; cholangitis is not on the implemented list of infections. In the second, a liver transplant recipient developed a urinary tract infection and pseudomonal infection (both of which were coded and met the infection criteria). This resulted in cardiac and renal failure (and thereby severe sepsis), but neither was coded. In the third, a patient with underlying interstitial lung disease presented with hypoxemia and was treated for pneumonia. Although hypoxemic, the patient did not require mechanical ventilation, and so no acute organ dysfunction was coded.

Thirty-two charts were Angus-positive but did not have severe sepsis based on clinical review. Most of these patients did not have severe sepsis because the organ dysfunction identified was felt to have a cause other than response to the infection. In many cases this stemmed from the complicated course of the patient, as in a patient with atrial fibrillation and a gastrointestinal bleed leading to hypotension who also had a urinary tract infection and Clostridium difficile infection, or a patient with a bursa infection and chronic thrombocytopenia. In some cases, the decision involved more clinical judgment, as in a young woman with gastroenteritis whose acute renal dysfunction was judged to be caused by her dehydration from diarrhea rather than from a systemic inflammatory response. Across organ dysfunctions, no single set of organ dysfunction seemed disproportionately more common in the false positives of the Angus-implementation relative to the true positives. (Table 4.)

Table 4. Prevalence of Organ Dysfunction ICD-9-CM Codes Among True Positive and False Positive Hospitalizations Meeting the Angus Implementation of Severe Sepsis.

These values incorporate sampling weights.

	True Positives (n=60)	False Positives (n=32)
Cardiovascular	27.6%	20.0%
95% CI	8.6%, 46.4%	0.8%, 39.3%
Neurological	26.4%	37.9%
95% CI	0.2%, 52.7%	0.0%, 87.4%
Hematologic	11.9%	18.0%
95% CI	0.0%, 27.2%	0.2%, 35.8%
Hepatic	0.8%	2.0%
95% CI	0.0%, 2.5%	0.0%, 6.4%
Renal	82.6%	32.1%
95% CI	71.4%, 93.8%	4.0%, 60.1%
Respiratory	9.9%	0.0%
95% CI	2.5%, 17.3%	0.0%, 10.9%

DISCUSSION

As severe sepsis emerges as a major driver of cost, mortality and morbidity, the need to precisely understand what is being measured in various health services research studies has become increasingly urgent. In this patient-level validation of the 2001 Angus implementation, claims had a 71% PPV for severe sepsis among patients initially admitted to a non-ICU medical service. The sensitivity was estimated at 50%, but with broad confidence intervals. Other coding implementations for severe sepsis had high PPV, but quite poor sensitivity. This suggests that the Angus implementation does identify a population predominantly comprised of patients with severe sepsis, though not a pure sample. Such a population can be of clear value in research, but the limitations must be acknowledged and appropriately considered.

In light of the limitations of the Angus implementation, it is useful to consider alternative implementations of the international consensus conference definition of severe sepsis. The most prominent alternative coding implementation defined sepsis “by the presence of any of the following ICD-9-CM codes: 038 (septicemia), 020.0 (septicemic), 790.7 (bacteremia), 117.9 (disseminated fungal infection), 112.5 (disseminated candida infection), and 112.81 (disseminated fungal endocarditis).” ³¹ At a single center, the 038 code was reported to have a PPV of 88.9% and NPV of 80% against a gold standard of the 1991 consensus definition as implemented in a case-control study.³¹ Lagu and colleagues use the same ICD-9-CM codes, while also requiring that the patient received antibiotics and underwent blood cultures in their unique multi-center database. ^34–36 The usefulness of this 038-centric implementation hinges on the definition of septicemia. The 1991 consensus conference was skeptical of value of the term “septicemia,” ¹¹ and the word does not appear in the 2001 document. As both the 1991 document and American Medical Association’s 2009 ICD-9-CM coding guidelines³⁷ note, “septicemia” requires the presence of pathological microorganisms or their toxins in the blood; informal physician usage of the term may be less precise. ¹³ In contemporary clinical trials of severe sepsis only 1 in 3 enrolled patients had positive blood cultures ^15,17,18; the international consensus conference requires neither a positive culture nor a positive blood culture for a diagnosis of severe sepsis. As such, the 038-centric definition—focused on hematogenous spread of microorganisms—poorly aligns with contemporary clinical practice for severe sepsis as it may inappropriately exclude non-bacteremic patients. To avoid confusion, we suggest that research that nonetheless uses an 038-centric implementation use the term “septicemia” rather than “severe sepsis”—as was recently done in an important study. ⁴ It is essential that there be close and transparent alignment between terminology and objects of study in health services research and those in biological and clinical research.

An alternative is to rely only on cases in which physicians explicitly use the words and coders explicitly code severe sepsis (995.92) or septic shock (785.52). A single center validation of 995.92 against blinded physician chart review reported 52% sensitivity and 98% specificity for the international consensus conference definition. ³⁸ An abstract reported a distinct single center validation of 785.52, with 46% sensitivity with 99% specificity. ³⁹ In our data, the combination of 995.92 or 785.52 had very high specificity and PPV but low sensitivity. For research questions (or sensitivity analyses) in which purity of sample is essential, this may be an excellent option. External generalizability will be limited by the nonrandom selection process resulting from such labeling.

For research with any of these implementations—or even prospective case finding—it is important to recognize that the syndromic definition of severe sepsis involves, inevitably, some clinical judgment. The levels of agreement of the hospitalist abstractors were acceptable but imperfect, despite their extensive training together and shared practice environment. This same heterogeneity plagues not only health services research, but basic research and clinical trials in severe sepsis. ⁴⁰ While greater protocolization of the gold standard used for diagnosing severe sepsis might improve agreement in case ascertainment, it is not clear that it will improve accuracy in terms of recognizing the true physiologic derangments in the patients.

Our study has a number of limitations that must be kept in mind. This is a single center study, and other single center studies for the 995.92 and 785.52 codes (reviewed above) suggest cross-center variability. All patients were admitted to medical wards, in contrast to the ICU, and the measurement characteristics may vary in different patient-populations. Our gold standard was retrospective assessment by highly trained hospitalists, rather than prospective assessment with the ability to directly examine the patient or order tests. As our primary analytic interest was the PPV of the Angus implementation, we sampled relatively few Angus-negative charts, leading to large confidence intervals in our estimates of the sensitivity of the definition and in the performance characteristics of alternative sepsis implementations.

In conclusion, the Angus ICD-9-CM implementation of the 1991 and 2001 international consensus conference definition of severe sepsis offers reasonable but imperfect PPV for identifying a cohort of patients with severe sepsis. Efforts to enhance the implementation might include a more complete list of infections. In addition, a multi-center approach to the validation of any new implementation may be beneficial, to help assess the extent of heterogeneity across institutions. For some research purposes, a more restrictive implementation using only explicit documentation and coding of severe sepsis and septic shock may offer excellent PPV but at a cost of greatly reduced sensitivity.