Billing diagnoses do not accurately identify out-of-hospital cardiac arrest patients: an analysis of a regional healthcare system

Abstract

Background

International Classification of Diseases 9^th Edition’s Clinical Modification (ICD-9CM) codes are frequently used in health services research. We tested the operating characteristics of ICD-9CM codes for identifying out-of-hospital cardiac arrest (OHCA) subjects.

Methods

We used ICD-9CM codes to generate an “administrative cohort” of subjects treated after possible OHCA at one of six emergency departments (EDs) between January 2010 and April 2014. We performed a structured chart review to determine proportion of this administrative cohort with actual OHCA (true positive rate for the ICD-9CM-based search method). The largest study site maintains a prospective registry of consecutive OHCA subjects, which we used to construct a “registry cohort”. We used this cohort to calculate the sensitivity of the ICD-9CM-based search strategy at this site, and compared in-hospital mortality and discharge dispositions between the two cohorts using Chi-square tests.

Results

ICD-9CM codes identified 2461 subjects that comprised the administrative cohort. Of these, the true positive rate for actual OHCA on chart review was 40%. ICD9-CM code sensitivity was 100% for subjects coded as dead on arrival and 19% for subjects coded as surviving to ED disposition. There were 609 OHCA subjects in the registry cohort and 268 subjects in the administrative cohort who presented to registry site. Only 26 subjects appeared in both cohorts. In-hospital mortality was significantly higher in the administrative cohort than the registry cohort (91% vs. 61%, P < 0.001), and hospital discharge disposition of survivors was less favorable (P < 0.001). Neither difference persisted after excluding subjects surviving <6h.

Conclusion

Compared to a prospective registry, ICD-9CM codes are an insensitive method to identify OHCA subjects. Moreover, ICD-9CM codes identify a biased sample of the OHCA population with higher mortality.

Introduction

Health services research frequently uses administrative data to identify cohorts of patients with similar diagnoses. In the United States, diagnosis and procedure billing is commonly based on the International Classification of Diseases 9^th Edition’s Clinical Modification (ICD-9CM) coding system. Linkage of ICD-9CM codes to census information has been used to describe the epidemiology of many time-sensitive medical emergencies, including adult¹ and pediatric² sepsis, in-hospital cardiac arrest³ and stroke,⁴ and traumatic brain injury.⁵ Risks of ICD-9CM-based methods for case identification in large populations include misclassification and sampling bias.⁶ For example, in sepsis research, ICD-9CM codes are less sensitive and specific compared to chart review,⁷ and may identify cases with more severe illness, creating a threat to generalizability.

Out-of-hospital cardiac arrest (OHCA) is among the most common treatable causes of death in the United States, with substantial variation in outcomes between regions.⁸ Thus, large population-based studies are a public health priority, and administrative data might provide an efficient means to accomplish these studies. Unfortunately, ICD-9CM-based selection methods have only moderate specificity for identifying patients with cardiac arrest in the Medicare population.⁹ The performance characteristics of ICD-9CM codes for identifying patients with OHCA are unknown.

Our regional health system has 650,000 annual emergency department (ED) encounters with a common electronic medical record, offering an opportunity to compare the characteristics of ICD-9CM-based OHCA patient identification to manual chart review in a large population. Additionally, we maintain a prospective registry of patients treated after cardiac arrest at our tertiary care referral hospital, creating a secondary method to identify and verify OHCA cases.^10-14 We tested the hypotheses that ICD-9CM codes would not identify all patients in the prospective registry, and that ICD-9CM codes would identify some cases that had not had an actual OHCA based on chart review. Secondarily, we hypothesized that clinical characteristics would differ between subjects identified using ICD-9CM codes, retrospective chart review, and the prospective registry.

Methods

The University of Pittsburgh Institutional Review Board approved all aspects of the study with a Waiver of Informed Consent. We included subjects treated between 1/1/2010 and 4/30/2014 within a large healthcare network in southwestern Pennsylvania. We compared the characteristics of two separate cohorts: an “administrative cohort” constructed from an ICD-9CM-based search strategy of billing data associated with encounter codes and a “registry cohort” generated by querying the prospective cardiac arrest registry. We used survival to hospital discharge and hospital discharge disposition to compare disease severity between the cohorts, because these are readily obtainable from administrative databases and strongly associated with validated metrics of disease severity after OHCA.¹⁵ Finally, we reviewed the entire medical record of a random subset of cases in the administrative cohort to determine if each case represented a true OHCA.

Administrative cohort

We identified patients who may have sustained an OHCA using ICD-9-CM codes for cardiac arrest (Cardiac arrest: 427.5; Ventricular fibrillation: 427.41; Paroxysmal ventricular tachycardia: 427.1; Ventricular flutter: 427.42; Respiratory Arrest: 799.1) assigned to ED encounters. We selected these codes based on prior work^{3, 9, 16} and included 799.1 to maximize sensitivity. We searched all diagnoses associated with the ED encounter, rather than restricting our search to the primary diagnosis. We categorized hospital discharge disposition as death, hospice, long-term acute care facility, skilled nursing facility, acute rehabilitation, home or unknown. Trained coders at a centralized service coded the final diagnoses for all ED encounters using ICD-9CM codes according to American College of Emergency Physician guidelines. For hospitalized subjects, the diagnoses for the ED encounter are assigned at the time of hospital discharge after review of the inpatient record. Ongoing quality assurance measures include regular audits by a separate coding group and review and feedback based on any discrepancies or other concerns. We restricted our query to ED encounters at six hospitals between January 1^st, 2010 and April 30^th, 2014. We excluded traumatic or surgical causes of arrest and cases in which electronic records from the index visit were missing.

Registry cohort

We maintain a prospective cardiac arrest registry of consecutive patients treated by our Post-Cardiac Arrest Service (PCAS) at our largest, regional referral center.^10-14 In this registry, we define cardiac arrest as a confirmed pulseless episode and chest compressions delivered by a healthcare professional The PCAS registry includes all patients for whom the local PCAS are consulted.¹⁷ The on-call PCAS attending physician accept all inter-facility post-arrest transfers and are systematically consulted by the ED attending on patients that have ROSC for >20 minutes and by either the ED or Cardiac Care Unit attending for patients directly transferred to the cardiac catheterization laboratory. An initial aim of this study was to be surveillance to ensure the PCAS registry catchment.

We queried this registry for patients who presented from January 1^st, 2010 to April 30^th, 2014 after OHCA. We excluded patients with surgical etiology of arrest (e.g. trauma, intracerebral hemorrhage or exsanguination) and patients treated after in-hospital cardiac arrest. The PCAS registry includes subjects who transition to comfort measure only or re-arrest without successful resuscitation within 6 hours of hospital arrival, and these patients are prospectively identified. To parallel our methods in the administrative cohort, we also queried our registry for each subject’s discharge disposition. In the PCAS registry, hospital discharge disposition is coded as home, acute rehabilitation, skilled nursing facility, long-term acute care, hospice, deceased, or unknown.

Chart review

We performed a structured chart review of a subset of cases in the administrative cohort to determine if each was a true OHCA. We defined true OHCA as documentation of chest compressions from a health care professional, rescue shocks from an automated external defibrillator, or an ED attending physician clinical impression of cardiac arrest. We also recorded each subject’s discharge disposition from the hospital (categorized as above). We reviewed the charts for all subjects coded as surviving their ED visit, as well as a randomly selected 20% of subjects with disposition identified as dead on arrival, dead in the emergency department, or dead in the operating room (all of which we considered “dead on arrival” (DOA)). To allow direct comparison to the registry cohort, which only includes patients treated by the PCAS at our highest volume hospital, we reviewed all subjects that were treated at that hospital. We identified cases where there was a transition to comfort measure only, re-arrest without successful resuscitation within 6 hours of hospital arrival, and pre-existing advance directive that precluded intensive care. We repeated our analyses both with and without including these subjects to allow direct comparison to the registry cohort.

Statistical Methods

We used descriptive statistics to summarize population characteristics for the administrative and registry cohorts. We used the registry cohort as a gold-standard methodology to which we compared the ICD-9CM-based search strategy, and calculated the sensitivity of the ICD-9CM search based on this. We used manual chart review as the reference to determine the proportion of the overall administrative cohort with true OHCA (the true positive rate the ICD-9CM-bases search strategy). We compared survival to hospital discharge and hospital discharge destinations between the administrative cohort and the registry cohort using a two group tests of proportions Chi-square. We repeated the same analysis but excluded subjects that died within 6 hours of hospital arrival.

Results

The ICD-9CM-based search identified 2516 potential subjects, of which, 28 (1%) had missing hospital records and 27 (1%) had a surgical etiology of arrest, leaving 2461 subjects in the final administrative cohort (Figure 1). Overall, 626 (34%) subjects in the administrative cohort we coded as surviving to ED disposition, and 1206 (66%) were coded as DOA, so we completed the structured chart review for 852 subjects (626 + (1206*20%)). On this chart review, 343 subjects were determined to have suffered an actual OHCA (true positive rate 40%). This true positive rate differed between those coded as DOA (100% true positive) and those coded as surviving to hospital admission (19% true positives) (P<0.001). Extrapolating these data to the entire administrative cohort, an estimated 1323 subjects suffered an actual OHCA (true positive rate 72%). Non-arrest related cardiac diagnoses were the most common final diagnoses among those without an actual OHCA (Table 1). Administratively coded ED discharge disposition was discordant for a majority subjects coded as survivors yet subjects coded as non-survivors had largely accurate discharge disposition as compared to manual chart review (Supplemental Tables 1 and 2).

Figure 1.

Patient accrual.

Table 1.

Subjects reported as discharge alive/cardiac arrest vs. abstracted diagnosis

Diagnosis by chart review	Coded as survived (n = 626)	Coded as DOA@ (n = 226)
Out-of-hospital cardiac arrest	117 (19%)	226 (100%)
Non-arrest diagnosis	509 (81%)	0 (0%)
Cardiac	286 (56%)	-
AICD fired or malfunction	47 (16%)	-
Acute MI	12 (4%)	-
Dysrhythmia	91 (32%)	-
Atrial fibrillation	39 (14%)	-
Supraventricular tachycardia	7 (2%)	-
Ventricular tachycardia	39 (14%)	-
Ventricular fibrillation	1 (<1%)	-
Not Specified arrhythmia	5 (2%)	-
Chest pain	103 (36%)	-
Palpitations	20 (7%)	-
Other*	13 (5%)	-
Non-Cardiac	223 (44%)	-
Abdominal pain	12 (5%)	-
Infection	19 (9%)	-
Respiratory distress or failure	15 (7%)	-
Musculoskeletal issue	16 (7%)	-
Dizziness or weakness	13 (5%)	-
SOB	16 (7%)	-
Trauma	22 (10%)	-
CVA	9 (4%)	-
Syncope	24 (11%)	-
Other#	77 (35%)	-

Percentages are shown as percent of diagnosis subheading.

^@a random sample of 20% of charts for subjects coded as DOA in the administrative cohort were reviewed

^*other cardiac diagnosis were: congestive heart failure exacerbation, hypertension, and aortic dissection.

^#other non-cardiac diagnosis were: acute kidney injury, allergic reaction, AMS, anxiety attack, allergic reaction, ascites, fistula/catheter/feeding tube complication, Dehydration, Depression, Dermatitis, dissection aortic aneurysm, dyspepsia, dysphagia, edema, esophageal obstruction, seizure, fever, flank pain, GI bleed, gout, headache, hypertension, hypoglycemia, hyponatremia, medication refill, menorrhagia, missed dialysis, nausea, pleural effusion, psychiatric issue, overdose, scabies, unresponsive, and well patient encounter.

Abbreviations: DOA – Dead on arrival, in the emergency department or in the operating room prior to admission; AICD- automatic implanted cardiac defibrillator; MI- myocardial infarction; SOB- shortness of breath; CVA- cerebral vascular accident

The PCAS registry identified 937 potential subjects, of which 284 (30%) were in-hospital cardiac arrests and 44 (5%) had a surgical etiology of arrest, leaving 609 subjects in the final registry cohort (Figure 1). Overall, a subgroup of 268 subjects in the administrative cohort (11% of the total administrative cohort) presented to the PCAS registry site. Structured chart review performed for all 268 of these subjects showed that 159 subjects suffered OHCA (true positive rate for ICD-9CM-based search strategy 40%). A total of 26 subjects (4% of registry patients, 10% of the administrative cohort subgroup) were identified in both cohorts (Figure 2). Thus, the overall sensitivity of the ICD-9CM search strategy compared to the gold standard registry cohort was 4% (Figure 3). After excluding subjects who survived <6 hours, the ICD-9CM search strategy had sensitivity of 3%(15/562) for identifying registry patients.

Figure 2.

Venn diagram depicting the registry and administrative cohorts and their overlap

Figure 3.

Comparison of subjects identified by registry and administrative for all subjects treated that the registry hospital

In-hospital mortality was higher in the administrative cohort than in the registry cohort (mortality 91% vs. 61%, p < 0.001), and discharge disposition of survivors was less favorable (p < 0.001) (Table 2). When comparing only subjects who survived > 6 hours after ROSC, mortality (68% vs. 57% p=0.067) and discharge disposition (p=0.09) did not differ between groups (Table 2).

Table 2.

Comparison of ICD-9CM administrative vs. prospective registry cohorts

Disposition	Administrative cohort N = 159	Registry cohort N = 609	P value
Survived to discharge	14 (9%)	223 (37%)	-
Home	10 (6%)	108 (18%)
Acute rehabilitation	1 (1%)	39 (7%)
Skilled nursing facility	2 (1%)	51 (8%)
Long term acute care	1 (1%)	19 (3%)
Hospice	0 (0%)	9 (1%)
Died in hospital	145 (91%)	371 (61%)
Unknown	0 (0%)	11 (2%)
Died within 6hrs (n)	116	62
Data for 26 subjects identified by both methods appear in both columns

After excluding subjects surviving <6 hours
Disposition	N = 45	N = 562	P value
Survived to discharge	14 (31%)	223 (41%)	-
Home	10 (22%)	108 (20%)
Acute rehabilitation	1 (2%)	39 (7%)
Skilled nursing facility	2 (5%)	51 (9%)
Long term acute care	1 (2%)	19 (3%)
Hospice	0 (0%)	9 (2%)
Died in hospital	31 (69%)	309 (57%)
Unknown	0 (0%)	11 (2%)

Data for 15 subjects identified by both methods appear in both columns

Discussion

We found that querying the ICD-9CM codes in billing data is not a sensitive approach to identifying OHCA when using a prospective hospital-based registry as the gold standard for surveillance In addition, a manual chart review of cases identified by ICD-9CM codes revealed that many were not OHCA, and those that were had significantly higher mortality than a confirmed OHCA population.

Our findings differ from those reported by DeBruin, et al., who performed a similar study in a cohort of Medicare beneficiaries from five states. The ICD-9CM codes selected by the investigators to identify possible OHCA subjects differed somewhat from those used in our study, and included: paroxysmal ventricular tachycardia (427.1), ventricular fibrillation and/or flutter (427.4*), cardiac arrest (427.5), paroxysmal tachycardia, unspecified (427.2), premature beats (427.6*), other cardiac dysrhythmias (427.8), and cardiac dysrhythmias, unspecified (427.9). Of the 34,374 subjects identified by ICD-9CM codes, the authors reviewed 235 medical records. The positive predictive value (PPV) of ICD-9CM codes was 79.7% (95%CI 68.3–88.4) for inpatient claims and 93.6% (95%CI 82.5–98.7) for ED claims. In contrast to DeBruin, we queried only claims generated from ED encounters, searched for a narrower range of arrhythmia ICD-9CM codes (427.1,427.41, 427.42) and included respiratory arrest (799.1) in our search strategy,

One explanation for the differences between Debruin and our cohorts is that “cardiac arrest” is a common final pathway for many heterogeneous disease states. As such, some “miscoding” may reflect heterogenous nature of the disease and suggest that cardiac arrest requires a different evaluation method than a standard administrative (ICD-9CM) approach. In addition, differences between hospital coding systems may further invite inaccuracies into administrative data. Finally, cardiac arrest survivors either arrest out of the hospital and are initially treated in the ED or arrest in hospital. The differences in how and by whom these subjects are initially coded may have important implications for data accuracy.¹⁸

We used survival to hospital discharge and hospital discharge disposition as surrogate markers of severity of illness, and found that our administrative cohort had greater severity of illness compared to the registry cohort. This effect was primarily in the large proportion of subjects who died within 6 hours of presentation, who may not have been included in the registry cohort. Moreover, the performance characteristics of ICD-9CM coding significantly differed between subjects who were DOA and those who survived to disposition from the ED. Wang described similar sampling bias in a study of community-acquired sepsis ⁷ The authors used two strategies to identify subjects with sepsis: ICD-9CM codes assigned at hospital discharge and medical record review of the subjects in the REASONS Stroke study.¹⁹ Mortality in the ICD-9CM cohort was significantly higher than subjects identified by chart review of the REASONS cohort for the diagnosis of sepsis (25.5% vs. 10.3%; p=0.006) and severe sepsis (25.5% vs. 11.5%; p=0.002) despite both cohorts having similar admission characteristics. These results are consistent with our own and suggest that cohorts constructed using ICD-9CM codes may represent a biased sample of the true population of interest.

Our findings have several implications for OHCA research. Using ICD-9CM codes associated with ED encounters to identify OHCA subjects may overestimate incidence of OHCA, and underestimate in-hospital survival. Hospital discharge coding as investigated by DeBruin⁹ may offer improved performance characteristics but do not identify subjects that are transferred, die prior to admission, or otherwise do not have an inpatient code assigned, which likely represent a large majority of OHCA cases in the United States.²⁰ If ICD-9CM codes are used to identify cardiac arrests, strict data quality assessment measures such as case-by-case chart review is imperative to ensure data quality, and failure to perform these quality checks may lead to biased or entirely erroneous results.

Better methods to identify and track OCHA patients are urgently needed. As shown in our study, administrative data and a single-hospital registry of admitted patients identified unique subgroups of OHCA patients, leaving the true OHCA cohort uncertain. Linkage of existing prehospital OHCA with hospital administrative data is a potential tool but presents unique challenges with limited linkage capability.²¹ Prospectively maintained ED-based or EMS-based registries are probably the best methods for compete identification of OHCA subjects.²²

Our work has several limitations. Our administrative cohort drew on ED encounters from 6 hospitals a single health network and a single centralized coding system. Therefore, the generalizability of our findings to other health systems with different coding methods is unknown. Additionally, different eligibility criteria for inclusion in the administrative and registry cohorts (e.g. all patients coded as suffering cardiac arrest vs only those patients with ROSC) may account for at least some of the observed differences between the cohort. However, even other studies also highlight critical deficiencies of claims-based identification of OHCA subjects.⁹ Combined, these studies suggest ICD-9CM coding for OHCA is more prone to inaccuracy and bias than for other diseases. Another potential limitation of our work is our selection of the gold standards against which we calculated the performance characteristics of ICD-9CM-based search strategies. We performed a detailed, structured retrospective chart review to determine true positive rate of ICD-9CM coding, but such retrospective strategies are imperfect. Given that OHCA is a major medical event that is unlikely to not be noted by providers, we believe adjudication of OHCA by retrospective chart review has good face validity. Our use of a prospective registry to test the sensitivity of ICD-9CM coding is also imperfect, although no better comprehensive surveillance is available in our care system. While it is clear that different search methods yielded distinct cohorts, our absolute capture through these methods is still unknown without a true “gold standard.” We believe that a prospective registry likely reflects the best available method, but is nevertheless imperfect and future efforts will need to address this limitation.

In conclusion, we found that ICD-9CM codes associated with ED encounters are insensitive for identification of OHCA subjects, have a low true positive rate, and identify a biased sample of the overall OHCA population with higher mortality. These factors must be considered when using administrative data to identify OHCA cohorts for research purposes, and prospective registries are needed to provide accurate epidemiological data.

Appendix

The Post Cardiac Arrest Service researchers are;

Jon C. Rittenberger, MD, MS

Clifton W. Callaway, MD, PhD

Francis X. Guyette, MD, MPH

Ankur A. Doshi, MD

Cameron Dezfulian, MD

Jonathan Elmer, MD

Bradley J. Molyneaux, MD, PhD

Lillian Emlet, MD, MS