Position matters: Validation of medicare hospital claims for myocardial infarction against medical record review in the atherosclerosis risk in communities study

Abstract

Purpose:

The objectives of this study were to investigate sensitivity and specificity of myocardial infarction (MI) case definitions using multiple discharge code positions and multiple diagnosis codes when comparing administrative data to hospital surveillance data.

Methods:

Hospital surveillance data for ARIC Study cohort participants with matching participant ID and service dates to Centers for Medicare and Medicaid Services (CMS) hospitalization records for hospitalizations occurring between 2001 and 2013 were included in this study. Classification of Definite or Probable MI from ARIC medical record review defined “gold standard” comparison for validation measures. In primary analyses, an MI was defined with ICD9 code 410 from CMS records. Secondary analyses defined MI using code 410 in combination with additional codes.

Results:

A total of 25 549 hospitalization records met study criteria. In primary analysis, specificity was at least 0.98 for all CMS definitions by discharge code position. Sensitivity ranged from 0.48 for primary position only to 0.63 when definition included any discharge code position. The sensitivity of definitions including codes 410 and 411.1 were higher than sensitivity observed when using code 410 alone. Specificity of these alternate definitions was higher for women (0.98) than for men (0.96).

Conclusion:

Algorithms that rely exclusively on primary discharge code position will miss approximately 50% of all MI cases due to low sensitivity of this definition. We recommend defining MI by code 410 in any of first 5 discharge code positions overall and by codes 410 and 411.1 in any of first 3 positions for sensitivity analyses of women.

1 |. INTRODUCTION

Administrative data present a convenient source for investigators to study factors associated with the risk of myocardial infarction (MI) or post-discharge MI care. The validity of the measurement of the outcome and the representativeness of the study population rely on the proper identification of disease. Recent studies examining the ability of International Classification of Diseases, Clinical Modification 9th revision (ICD9) discharge codes to identify MI cases were more likely to report the positive predictive value (PPV) of definitions^1–4 rather than their sensitivity and specificity. Published studies reporting sensitivity and specificity were either conducted over 10 years ago^5,6 or used selected populations^7,8 that may not generalize to the current US population. It is also important to investigate the effect of relaxing the standard restriction of discharge code position in the MI case definition³ because a validation study of MI in an HIV population showed evidence of higher sensitivity when removing these restrictions.⁷

The goal of this study was to investigate the validity of discharge codes in Centers for Medicare and Medicaid Services (CMS) insurance claims data to identify MI cases. The first aim of this study was to investigate if defining the presence of MI using more than the first 2 discharge code positions would lead to higher sensitivity with negligibly lower specificity in comparison with algorithms that use the first and second discharge positions only. To investigate whether less severe MI cases might be recorded with alternate diagnostic codes, the second aim investigated the validity of multiple cardiac related discharge codes to identify MI cases. Lastly, we explored if MI validity measures were different by age (<75 years, ≥75 years) or between men and women.⁹ To achieve study objectives, we compared CMS defined MI cases to adjudicated hospital record events for Medicare beneficiaries in the Atherosclerosis Risk in Communities (ARIC) Study.

2 |. METHODS

2.1 |. Study population

2.1.1 |. The ARIC study cohort

The ARIC study, designed to examine the etiology of cardiovascular disease, recruited a prospective cohort of 15 792 individuals 45 to 64 years of age at baseline through probabilistic sampling from 4 United States regions: Forsyth County, North Carolina; Jackson, Mississippi; Minneapolis, Minnesota; and Washington County, Maryland. Only African Americans were sampled in the Jackson study center. Cohort participants have been followed since baseline examination visit (1987–1989) for surveillance of cardiovascular events as well as prospective examinations and annual follow-up interviews.¹⁰

2.2 |. Linkage with CMS Medicare claims

Data for ARIC cohort participants were linked with the health care encounter data from the CMS using a finder file that included participants’ social security numbers, sex, and birthdate. In ARIC, 14 899 cohort study participants were Medicare eligible and available for linkage. The crosswalk file between the ARIC finder file and CMS Medicare Beneficiary Summary file yielded 14 702 ARIC cohort IDs (98.7% match), 13 746 of which included a perfect match on all 3 finder file variables, 952 of which were additionally matched on just the social security number and sex, and 4 of which were additionally matched on just the social security number and birthdate.

Medicare is a governmental sponsored health insurance program for people over the age of 65, those under age 65 with specific medical conditions, and patients with kidney failure requiring dialysis or kidney transplant. The CMS Medical Provider Analysis and Review (MedPAR) file contains summary records of inpatient hospital stays for beneficiaries with fee-for-service coverage and Medicare Advantage (MA) coverage. Hospitals submit information only (non-payment) claims for MA covered patients to get inpatient stay credit toward the Medicare Supplemental Security Income program.¹¹ Because the diagnosis of MI should not differ by insurance coverage, we did not exclude hospitalizations occurring during MA coverage. We restricted our analyses to all hospitalization records that matched between the 2 data sources during the observation period (2001-2013) as described next.

2.3 |. Matched sample of medical encounters between ARIC and CMS Medicare

To assess the validity of CMS Medicare claims to identify MI cases, ARIC surveillance records were matched to MedPAR records by cohort ID and dates of service occurring between 2001 and 2013. Initially, 13 809 records were matched using ARIC ID, admission date, and discharge date (Figure 1). Because a considerable number of ARIC surveillance records did not include an admission date, an additional 9456 records were matched using only ARIC ID and discharge date. Because the dates on MedPAR records might not exactly match event dates as recorded by ARIC, we matched 2284 records using ARIC ID and either admission date or discharge date allowing up to 7 days difference between the service dates in ARIC and service dates in MedPAR. The final analysis dataset contained 25 549 matched records without including 12 785 unmatched MedPAR records and 9190 unmatched ARIC records (Figure 1). One observation per hospitalization defined the unit of analysis in this study so we did not exclude records if a patient had recurrent MI events.

FIGURE 1.

Flow diagram of Medicare to ARIC record matching

2.4 |. Event identification

ARIC study protocols were used to classify MI events according to information on the presence of chest pain, electrocardiogram evidence, and biomarker findings that were abstracted from the hospital medical records of ARIC cohort participants. Events are initially classified as Definite MI, Probable MI, Suspect MI, and No MI according to a computer-generated algorithm.¹² A panel of ARIC physician reviewers reevaluates computer-generated MI classifications on an ongoing basis to correct any inconsistent findings between hospital records and computer classifications. For the purposes of this study, the ARIC classification of Definite or Probable MI was considered the gold standard when calculating concordance of the presence of MI between matched MedPAR and ARIC hospitalization records.

We investigated several definitions of MI using ICD9 cardiac-related diagnostic and procedure codes (Table 1) from the MedPAR records. First, we defined an MI event as ICD9 code 410 (Acute MI) excluding code 410.x2 (subsequent episode of care) in any discharge code position on an inpatient Medicare record. All analyses in this study excluded 410.x2 codes from algorithms as we sought to identify records where an event occurred and not the process of MI care. Because each hospitalization record in the MedPAR dataset had up to 10 discharge code positions, we also investigated the validity of definitions that included code 410 occurring in a series of discharge code positions (1 only, ≤2, ≤3, …, ≤10). For aim 2, we defined an MI by inpatient records with a combination of codes 410,411.1 (Intermediate coronary syndrome), or 413 (angina pectoris) by discharge code position. To increase the specificity of these definitions, we also investigated algorithms that required the presence of procedure codes related to revascularization on the same record as codes 411.1 and 413 (Table 1). A MedPAR record that did not include the a priori defined ICD9 diagnostic/procedure codes was classified as not an MI for each definition separately.

TABLE 1.

Description of ICD9 codes used in CMS algorithms to classify MI events

ICD9 Codes	Description
Diagnosis:
410.xx	Acute myocardial infarction
411.1	Intermediate coronary syndrome
413.xx	Angina pectoris
Procedure:
00.66	Percutaneous transluminal coronary angioplasty
88.5x	Angiocardiography
37.2x	Diagnostic procedures on heart and pericardium
36.0x	Removal of coronary artery obstruction and insertion of stent (starting 10/2007)
36.1x	Bypass anastomosis for heart revascularization (starting 10/2007)

2.5 |. Covariate ascertainment

Demographic information (age at hospitalization, sex, and race), ARIC Study center, and teaching hospital status (teaching, non-teaching) were obtained from information gathered according to ARIC study protocols. Baseline information on history of smoking, alcohol intake, and body mass index were obtained from the fourth ARIC clinical examination visit that occurred between 1996 and 1998, approximately 4 years prior to the start of the study observation period in 2001. History of MI, diabetes, and hypertension were also ascertained at this ARIC visit and from inpatient and outpatient CMS records occurring between 1996 and 2001.

2.6 |. Statistical analyses

The demographic and baseline clinical characteristics of the study population were summarized using proportions. A cross tabulation of events from the matched sample was used to calculate sensitivity, specificity, and PPV (Appendix Table 1). Because an individual may have contributed more than 1 hospitalization record in the analysis, generalized estimating equations were used to compute 95% confidence intervals.

3 |. RESULTS

3.1 |. Study population

Of the 25 549 hospitalizations matched on ARIC participant ID and dates of service, 1139 records (4.4%) were classified as definite or probable MI according to ARIC MI classification criteria. The mean age at the time of hospitalization was 74.8 (SD 6.1) years for the 7995 beneficiaries who contributed an average of 3.2 hospitalizations per participant in the matched records dataset (Table 2). Women accounted for 57% of hospitalizations. Caucasian participants primarily residing near Minneapolis, MN and Washington County, MD study centers comprised 74% of the study population. Less than 1% of participants from these areas were non-Caucasian. All of the participants near the Jackson, MI study center (1,791) were African-American per ARIC study design. At baseline, 63% of the beneficiaries had a history of hypertension but only a small proportion (10%) had a history of MI. At the fourth ARIC examination visit occurring between 1996 and 1999, 60% of participants reported being a current or former smoker, 76% reported being a current or former drinker, and 77% had a BMI ≥ 25 kg/m². A total of 20 914 (82%) hospital records occurred during periods of fee-for-service coverage.

TABLE 2.

Characteristics of medical encounters between January 2001 and December 2013 by data source. The Atherosclerosis Risk in Communities (ARIC) cohort

Characteristic	Medicare	ARIC	Matched Sample
Hospital records	38 334	34 739	25 549
Age group
Aged <75 yrs.	18 177 (47.4%)	17 424 (50.2%)	12 532 (49.1%)
Aged ≥75 yrs.	20 157 (52.6%)	17 315 (49.8%)	13 017 (50.9%)
Sex
Female	22 415 (58.5%)	19 250 (55.4%)	14 607 (57.2%)
Male	15 919 (41.5%)	15 489 (44.6%)	10 942 (42.8%)
Center
Forsyth County	8175 (21.3%)	8665 (24.9%)	5779 (22.6%)
Jackson	10 193 (26.6%)	9130 (26.3%)	6682 (26.2%)
Minneapolis	9698 (25.3%)	7724 (22.2%)	5538 (21.7%)
Washington County	10 268 (26.8%)	9220 (26.5%)	7550 (29.6%)
Hospital type
Non-teaching		21 025 (61.4%)	16 083 (64.0%)
Teaching		9431 (27.5%)	6580 (26.2%)
Out of area		3797 (11.1%)	2475 (9.8%)
Missing		486	411
Unique beneficiaries	8905	9523	7995
Mean(SD) # of hospitalizations	4.3 (4.66)	3.6 (3.58)	3.2 (3.21)

3.2 |. Validity of MI classifications

In the overall population, MI defined by ICD9 code 410 alone had high specificity (>0.98) and low to moderate sensitivity (0.48 to 0.63) when compared with the gold standard definition (Figure 2). With an increasing number of discharge code positions included in the algorithm, the PPV decreased from 0.75 (95%CI: 0.71, 0.78) for ICD9 code 410 in the primary position only to 0.67 (95%CI:0.64, 0.70) for ICD9 code 410 in any discharge code position on the record (data not shown). In the Figure 2 plot of sensitivity and specificity estimates from MI algorithms by discharge code position, the slope of the sensitivity curve is the greatest between discharge code positions 1 and 3. The slope of the sensitivity curve attenuates after position 3 and is negligible after position 5 (Figure 2). The difference between the sensitivity of the MI algorithm that included the first 5 discharge code positions 0.62 (95%CI: 0.59, 0.66) and all positions 0.63 (95%CI: 0.60, 0.66) was minimal. Specificity decreased minimally with the inclusion of additional positions in ICD9 definition of MI.

FIGURE 2.

Sensitivity and specificity of Medicare ICD9 code 410 (excluding 410.x2) to identify an MI when compared with ARIC classification of definite or probable MI by discharge code position [Colour figure can be viewed at wileyonlinelibrary.com]

In analyses stratified separately by sex and age, we observed that sensitivity of ICD9 code 410 was greater in men compared with women and greater among beneficiaries less than 75 years old compared with those at least 75 years old at the time of their hospitalization (Figure 3). The greatest difference in sensitivity across each stratified analysis was evident when MI was defined from ICD9 codes in the first or second discharge code positions only (Figure 3). The 5 most frequent diagnoses among hospitalizations with a definite or probable MI according to ARIC protocols were acute MI, other forms of chronic ischemic heart disease, heart failure, cardiac dysrhythmias, and essential hypertension. A summary of these codes present in the first and second discharge code positions only was summarized overall and by sex (Appendix Table 2). In MI algorithms defined by ICD9 code 410 in any discharge code position, sensitivity and PPV were similar within age strata and sex strata. Specificity, at least 0.982 for all stratified algorithms, was negligibly higher among women and younger (<75 years old) beneficiaries compared with men and older (≥75 years old) beneficiaries, respectively.

FIGURE 3.

Sensitivity of Medicare hospitalization records containing ICD9 discharge code 410.xx (excluding 410.x2) to identify an MI when compared with ARIC classification of definite or probable MI stratified by sex, age at hospitalization, and discharge code position

We also investigated whether adding other cardiovascular codes to the MI algorithm based upon ICD9 discharge code 410 alone would improve sensitivity without significantly decreasing the high specificity (>0.98). The MI algorithm using diagnostic codes for intermediate coronary syndrome (411.1), angina pectoris (413), or acute MI (410) in any discharge position had the highest sensitivity but lowest specificity (Table 3). The sensitivity of algorithms that included codes 410 or 411.1 was higher than sensitivity observed when including code 410 alone (Table 3). The sensitivity of algorithms including either codes 410 or 411.1 with cardiac procedure codes on the 411.1 record ranged from 0.65 (95% CI: 0.62, 0.68) when using any of first 3 discharge code positions to 0.69 (95% CI: 0.66, 0.72) when using any discharge code position. MI definitions that included code 413 had lower specificity and similar sensitivity compared with MI definitions that included codes 411.1 or 410. MI algorithm specificity was higher when limiting the number of discharge code positions used in the case definitions that included the additional codes (Table 3). We did not observe statistical differences in the sensitivity of ICD9 definitions using multiple cardiac-related discharge codes when stratifying by age or sex (Appendix Table 3). While the absolute difference in sensitivity of ICD9 MI definitions including codes 410 or 411.1 compared with code 410 alone was similar for men and women, the decrease in specificity from inclusion of the additional code was higher for men than women.

TABLE 3.

Sensitivity, specificity, and positive predictive value (PPV) of Medicare hospitalization records containing ICD9 discharge code 410 (excluding 410.x2), 411.1, or 413 in selected discharge code positions to identify an MI when compared with the ARIC classification of definite or probable MI

	Definite or Probable MI
	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	Kappa (95%CI)
410 or 411.1 or 413 (any)	0.73 (0.70, 0.76)	0.946 (0.943, 0.950)	0.39 (0.36, 0.41)	0.48 (0.45, 0.50)
410 or 411.1 (any)	0.71 (0.68, 0.74)	0.961 (0.957, 0.964)	0.46 (0.43, 0.48)	0.53 (0.51, 0.55)
410 or 411.1+ or 413+ (any)	0.69 (0.66, 0.72)	0.964 (0.961, 0.967)	0.47 (0.44, 0.50)	0.54 (0.51, 0.56)
410 or 411.1+ (any)	0.69 (0.66, 0.72)	0.969 (0.967, 0.972)	0.51 (0.48, 0.54)	0.56 (0.54, 0.59)
410 or 411.1+ or 413+ (≤5)	0.68 (0.65, 0.71)	0.966 (0.963, 0.969)	0.48 (0.46, 0.51)	0.54 (0.52, 0.57)
410 or 411.1+ (≤5)	0.68 (0.65, 0.71)	0.971 (0.968, 0.973)	0.52 (0.49, 0.55)	0.57 (0.54, 0.59)
410 or 411.1+ or 413+ (≤3)	0.65 (0.62, 0.68)	0.970 (0.967, 0.972)	0.50 (0.47, 0.53)	0.55 (0.52, 0.57)
410 or 411.1+ (≤3)	0.65 (0.62, 0.68)	0.974 (0.971, 0.976)	0.54 (0.51, 0.57)	0.57 (0.54, 0.59)
410 or 413+ (any)	0.64 (0.61, 0.67)	0.980 (0.978, 0.982)	0.60 (0.57, 0.63)	0.60 (0.57, 0.62)
410 (any)	0.63 (0.60, 0.66)	0.985 (0.984, 0.987)	0.67 (0.64, 0.70)	0.63 (0.61, 0.66)
410 (≤5)	0.62 (0.59, 0.66)	0.986 (0.985, 0.988)	0.68 (0.65, 0.71)	0.63 (0.61, 0.66)

Note: Discharge code positions are in parentheses. Codes 411.1⁺ and 413⁺ indicate that revascularization procedure codes (00.66, 36.0, 36.1, 88.5, 99.10, or 37.2) were included on the same record as discharge codes 411.1 or 413, respectively. ICD Codes: 410–Acute myocardial infarction, 411–Other acute and subacute forms of ischemic heart disease, 413–Angina Pectoris

4 |. DISCUSSION

In this study, we investigated how well MI case definitions based on ICD9 diagnostic codes would compare with classifications of MI available from medical record review in patients with matching hospitalization records. Leveraging the linkage of CMS Medicare records with data from the ARIC study, we observed high specificity of algorithms based on ICD9 diagnosis code 410. The high specificity of diagnostic codes to detect an MI provides confidence that MI cases identified are true cases. However, algorithms that rely on the first discharge code position alone will miss approximately 50% of all MI cases due to the low sensitivity of this definition. The sensitivity of algorithms to identify MI cases increased as we expanded the number of discharge code positions for inclusion in our case definition beyond the primary position. MI algorithms defined by ICD9 code 410 occurring in any discharge code position had the highest sensitivity compared with algorithms restricted by discharge code position. Sensitivity of ICD9 definitions of MI compared with the gold standard was consistently lower for women than men suggesting that a sex difference exists in the coding of MI-related hospitalizations for Medicare aged adults. We observed that this difference was attenuated when all discharge code positions were included in the definition of MI. This difference in coding may reflect sex differences in the presentation MI and sex differences in the attributable risk of heart disease symptoms on outcomes.^13,14

Our second aim was to explore the effect of the inclusion of additional diagnostic codes on the sensitivity and specificity of the MI algorithm compared with ICD9 code 410 alone. As expected, we found that MI case definitions with multiple ICD9 diagnostic codes increased sensitivity but decreased specificity in comparison with an algorithm based on code 410 alone. Despite the decrease in specificity, we recommend that researchers consider an MI algorithm with either code 410 or 411.1 in the first 3 discharge code positions requiring cardiac-related procedure codes on the same record as code 411.1 to capture less severe MI cases when appropriate. While the sensitivity/specificity trade-off may not be adequate when studying a Medicare aged male population, this alternate MI case definition should be included as a secondary or sensitivity analysis when defining MI for Medicare aged women.

This study is important because the design provides an opportunity to report both measures of validity (sensitivity and specificity) as well as reliability (positive and negative predictive value). Most previous studies comparing diagnostic code 410 to chart review reported only PPV because they did not collect data to report sensitivity and specificity. These previous studies comparing diagnosis code 410 to chart review in the US reported higher PPV (88.4% to 94.1%) than that observed in our study (70.6%).^1,2,4,15 This difference in PPV is likely related to differences in MI prevalence in each study. For example, the MI prevalence reported in the Pladevall study (12.1%) was higher than that observed in our study (4.4%).

This study adds to the literature by highlighting the importance of discharge code position in MI case definitions based on administrative data. Brouwer et al reported the sensitivity and specificity of algorithms with ICD9 codes in the first 2 discharge code positions or in any position among Medicaid beneficiaries from an HIV cohort.⁷ Our study outcomes augment the results from this earlier study by providing results from a broader subset of the US population and detailed results from additional discharge code positions. While it was unclear which discharge code positions were used in the Pladevall study to define cases, the low sensitivity reported for code 410 only was consistent with the low sensitivity we observed in the current study when restricting to the first discharge code position only.⁵ Although the higher sensitivity and specificity in the current study compared with the Pladevall study may be due to changes in coding and MI biomarker evidence over the last 20 years, we suspect discharge code position is a significant contributing factor to these differences.

Using the ARIC study data linked to CMS Medicare data in the validation of ICD9 codes to identify MI cases had both strengths and weaknesses. One of the greatest strengths of this study is the use of adjudicated ARIC classification of MI events as the gold standard. Second, the ARIC study collected information on all hospitalizations occurring among cohort participants; therefore, unlike studies that just collected information on MI events, we were able to calculate sensitivity and specificity of MI algorithms. One limitation of this study was the inability to match all of the ARIC hospitalizations to hospitalizations identified from the CMS Medicare data¹⁶ so our estimates may overestimate the truth. Also, because the available CMS data included only the first 10 discharge code positions of the 25 available currently in hospital records, we were unable to investigate significance of discharge codes present in any of these latter positions. Another limitation of this study was the relatively low MI occurrence in ARIC which limits applicability of PPV results to external populations.¹⁷ The prevalence of MI in Americans’ aged 60 to 80 was recently estimated to be 11% in men and 5.2% in women.¹⁸ Our study results are relevant to the discussion of the validity of administrative codes to identify MI cases even though ARIC cohort participants may not be representative of all US patients because studies measuring sensitivity and specificity are rarely performed due in part to the laborious and expensive nature of chart review. This study also contributes to the literature on the definition of MI in women and individuals over the age of 75.

In conclusion, discharge code position matters in MI case definitions. Although sex differences exist in the sensitivity of MI case definitions that include diagnosis codes in the first discharge code position only, accurate identification of MI in administrative data for both sexes can be achieved by including at least the first 5 discharge code positions in the ICD9 definition when using code 410 only. Investigators should also consider including ICD9 code 411.1 in combination with procedure codes to the 410 code definition when investigating MI in Medicare aged women. These definitions can be used in research for ascertainment of outcomes and comorbid cardiovascular disease.

KEY POINTS.

• Case definition that rely exclusively on the primary discharge code position will miss approximately 50% of all MI cases due to low sensitivity of this definition.

• Accurate identification of MI in administrative data can be achieved by including at least the first 5 discharge code positions in the definition of MI when using ICD9 410 code only.

• Plausible alternate diagnostic code algorithms to identify MI include ICD9 code 411.1 in a limited number of discharge code positions or code 410 in any position when investigating MI in women.