Validation of Acute Myocardial Infarction in the Food and Drug Administration’s Mini-Sentinel program

Abstract

Purpose

To validate an algorithm based upon International Classification of Diseases, 9^th revision, Clinical Modification (ICD-9-CM) codes for acute myocardial infarction (AMI) documented within the Mini-Sentinel Distributed Database (MSDD).

Methods

Using an ICD-9-CM-based algorithm (hospitalized patients with 410.x0 or 410.x1 in primary position), we identified a random sample of potential cases of AMI in 2009 from 4 Data Partners participating in the Mini-Sentinel Program. Cardiologist reviewers used information abstracted from hospital records to assess the likelihood of an AMI diagnosis based on criteria from the joint European Society of Cardiology and American College of Cardiology Global Task Force. Positive predictive values (PPVs) of the ICD-9-based algorithm were calculated.

Results

Of the 153 potential cases of AMI identified, hospital records for 143 (93%) were retrieved and abstracted. Overall, the PPV was 86.0% (95% confidence interval; 79.2%, 91.2%). PPVs ranged from 76.3% to 94.3% across the 4 Data Partners.

Conclusions

The overall PPV of potential AMI cases, as identified using an ICD-9-CM-based algorithm, may be acceptable for safety surveillance; however, PPVs do vary across Data Partners. This validation effort provides a contemporary estimate of the reliability of this algorithm for use in future surveillance efforts conducted using the FDA’s MSDD.

INTRODUCTION

Through the Sentinel Initiative launched in May 2008, the U.S. Food and Drug Administration (FDA) is developing a national electronic system for postmarket risk identification and analysis of medical product safety that will use automated healthcare data to complement its existing surveillance systems.(1, 2) The Mini-Sentinel pilot (www.mini-sentinel.org) – an FDA-funded project designed to inform development of the Sentinel Initiative – is a collaborative effort between FDA and a number of Academic and Data Partners.(3) The success of the Sentinel Initiative’s surveillance efforts depends in part on the ability to accurately identify health outcomes of interest using automated healthcare databases.

Automated healthcare databases are frequently used to conduct epidemiologic research,(4-6) but the accuracy of a diagnosis from these data sources can be uncertain. In recognition of this uncertainty, FDA has requested that the Mini-Sentinel develop procedures for conducting outcome validation and use these procedures to validate the diagnostic criteria for several important health outcomes, beginning with acute myocardial infarction (AMI).

AMI was selected as the first health outcome to be validated because AMI is an important adverse outcome for FDA to be able to monitor. Drug-induced myocardial infarction has been difficult to identify through existing passive surveillance mechanisms.(1) Validation of an AMI algorithm within the Mini-Sentinel Distributed Database (MSDD) would enhance the FDA’s ability to conduct needed surveillance efforts. The primary goal of this validation activity was to assess the positive predictive value (PPV) of an algorithm based upon International Classification of Diseases, 9^th revision, Clinical Modification (ICD-9-CM) codes for AMI documented within the MSDD. The MSDD currently comprises the administrative and claims data of 17 Data Partners formatted into a common data model.(7) An additional goal of this project was to develop the procedures needed to conduct an outcome validation in the setting of public health surveillance. A recent paper(8) describes the development of those procedures in detail.

METHODS

We studied patients enrolled in health plans represented by 4 large and geographically diverse Data Partners: the HMO Research Network, Humana, HealthCore, Inc. and Kaiser Permanente Center for Effectiveness and Safety Research (CESR). Due to the project’s inclusion in the FDA’s Sentinel Initiative, it was not under the purview of the Office of Human Research Protection and, therefore, did not require Institutional Review Board (IRB) approval (8-11). Each Data Partner was provided with a privacy packet prepared by the Mini-Sentinel Privacy Panel. This packet included: 1) the Mini-Sentinel Privacy Panel White Paper describing data privacy issues in Mini-Sentinel (9); 2) a letter from the Department of Health and Human Services Office for Human Research Protections (OHRP) to the FDA stating that the regulations OHRP administers do not apply to the Sentinel Initiative (OHRP oversees all IRBs); and 3) a letter from the FDA to the Mini-Sentinel Principal Investigator explaining FDA’s legal authority to obtain data for use in its Sentinel and Mini-Sentinel activities. The privacy packet described the legal basis for determining that the work of the Mini-Sentinel pilot is not under the purview of IRBs.

Source Population

The source population included patients hospitalized for electronically identified AMI between January 1, 2009 and December 31, 2009. All patients were required to be enrollees of the health plan for the entire duration of hospitalization (no patient was excluded due to this criterion). To increase generalizability, there were no restrictions on age, sex, or other patient characteristics. There was no baseline period required and there were no exclusions based on previous disease.

Case identification

Patients with claims evidence of AMI were identified by selecting those with an ICD-9-CM code for AMI (410.x0, 410.x1) in the principal or primary position on facility claims for hospitalizations. Participating Data Partners executed a distributed SAS program (SAS version 9.1, SAS Institute, Cary, North Carolina) designed to query their own locally maintained administrative and claims databases to identify a random sample of AMI cases and the hospitals in which they received care. Using this program, we achieved our goal of identifying a random sample of at least 100 AMI cases for validation. A sample of 100 cases would allow determination of the PPV of the diagnostic coding algorithm with a 95% confidence interval (CI) of ± 0.10, assuming a PPV of 50%.

Data Partners subsequently retrieved and redacted records of potential AMI cases, including hospital transfer records where available. These redacted records were made available for full text medical chart review by trained nurse abstractors.

Since this was the first validation activity to be conducted using the FDA MSDD, there was no precedent for our expected retrieval rate of AMI cases. A rate of 66% was projected as a conservative estimate based on the diversity of hospital sites and the anticipated complexities; a minimum of 150 charts requested was therefore deemed sufficient to obtain at least 100 cases of AMI. In order to distribute chart requests equally among 4 sites, 38 charts were requested from each Data Partner with the exception of one which was splitting its chart retrieval among 3 sites (13 charts per site) and therefore provided 39 in total. In sum, a total of 153 charts with a diagnosis of AMI were requested from participating Data Partners.

Case Confirmation

Two trained nurse abstractors reviewed records provided by the 4 Data Partners and completed a standardized data abstraction form (Appendix A) which had been developed to provide information on sex, age, race, length of hospital stay and transfer to or from another hospital, as well as clinical information pertinent to the diagnosis of AMI. Abstracted clinical information included EKG images, cardiac biomarkers, information on ischemic symptoms and results of cardiac diagnostic tests (Appendix A). Both abstractors gathered data from the first 10 cases. This abstracted information was reviewed together by both nurse abstractors to ensure high inter-rater reliability on items critical for the adjudication process. In consultation with FDA staff and individuals with clinical and epidemiologic expertise relevant to cardiovascular disease, the lead team created an adjudication protocol (Appendix B) based on standardized criteria from the joint European Society of Cardiology and American College of Cardiology Global Task Force(12, 13) and based on the literature on troponin standardization.(14)

Abstracted information was reviewed independently by two cardiologists. Cardiologist adjudicators were provided with copies of EKGs and copies of all cardiac tests and procedure reports of all likely AMI cases. The adjudicators did not review discharge summaries but were instead provided with abstracted case information. They then classified cases of possible AMI as either: (1) definite MI; (2) probable MI; (3) no MI; or (4) unable to determine. The adjudicators were provided criteria for AMI (Appendix B) but were not instructed on specific criteria for distinguishing definite from probable cases of AMI. They were asked to use their clinical judgment to make this determination.

An AMI was considered to be present if both cardiologist adjudicators categorized the case as definite or probable. When the adjudicators disagreed on the classification of a case (i.e. if one adjudicator indicated definite or probable and one indicated no MI or unable to determine), they met and reached consensus; consensus was reached in all cases. The initial assessment of the adjudicators was compared and inter-rater reliability was calculated using Cohen’s kappa statistic.(15)

A positive predictive value (PPV) was calculated as the percentage of confirmed cases (definite or probable) of AMI among all hospitalizations identified. The PPVs (overall and according to Data Partner, patient demographics [sex, age, race], and characteristics of hospital stay [length of hospital stay, transfer to or from another hospital]) were estimated. Because specific identification of Data Partners was not necessary for the purpose of the analysis, we present PPVs stratified by Data Partner (table 1) but do not identify Data Partners by name.

Table 1.

Calculated positive predictive value (PPV) and 95% confidence interval for acute myocardial infarction (AMI) by Data Partner.

Data Partner	Number of charts requested	Charts obtained (%)	AMI confirmed	AMI not confirmed	Insufficient information in medical chart	PPV (%)	95 % CI
Data Partner 1	38	32 (84.2)	26	4	2	81.3	63.6, 92.8
Data Partner 2	38	38 (100)	29	1	8	76.3	59.8, 88.6
Data Partner 3	38	35 (92.1)	33	0	2	94.3	80.1, 99.3
Data Partner 4	39	38 (97.4)	35	1	2	92.1	78.6, 98.3
Overall	153	143 (93.5)	123	6	14	86.0	79.2, 91.2

RESULTS

Of the 153 potential cases of AMI identified from health plan administrative data, hospital records for 143 (93%) were available for review. Medical record information for 7 cases could not be obtained due privacy concerns. Hospital IRBs in these cases required a patient signature to release charts. This occurred despite the information provided to Data Partners indicating that the FDA Sentinel Initiative activities are not under the purview of the OHRP and do not require IRB approval. Medical records for 3 potential cases could not be located. Retrieval rates across the Data Partners ranged from 84% to 100% of cases identified from health plan data.

The mean age of patients for whom records were abstracted was 69.5 years (range 27-94) and 47% were female. Of the 143 cases, 14 cases required joint review by the two cardiologists; they reached consensus in all cases and ultimately determined that 123 cases were either definite or probable AMIs (118 were classified by at least one cardiologist as definite; 5 were characterized by both as probable).

There were 20 cases that were judged not to be consistent with a definite or probable AMI, either because one or both cardiologists felt that there was insufficient information available to confirm or deny the presence of an AMI (14 cases) or because both cardiologists agreed that there was sufficient information available to indicate that the case was not an AMI (6 cases). The kappa score for inter-rater reliability, based on initial assessment of the cardiologist adjudicators, was found to be 0.60 (95% CI: 0.42, 0.78), which indicates moderate agreement.

Overall, the PPV was 86.0% (95% CI: 79.2%, 91.2%). PPVs ranged from 76.3% to 94.3% across the Data Partners (Table 1). Several different subgroups defined a priori were examined (Table 2), but PPV estimates were imprecise due to the small sample size.

Table 2.

The positive predictive value and 95% confidence interval of acute myocardial infarction (AMI) by patient characteristics

	Total (n)	AMI present (n)	No MI (n)	Unable to determine (n)	Positive predictive value (%)	95 % Confidence Interval
Total	143	123	6	14	86.0	79.2 to 91.2
Age <75 years	74	70	2	2	94.6	86.7 to 98.5
Age 75+	53	42	3	8	79.3	65.9 to 89.2
Age Unavailable	16	11	1	4	68.8	41.3 to 89.0
Male	76	71	2	3	93.4	85.3 to 97.8
<75	45	43	1	1	95.6	84.9 to 99.5
75+	26	23	1	2	88.5	69.9 to 97.6
Age unavailable	5	5	0	0	100
Female	67	52	4	11	77.6	65.8 to 86.9
<75	29	27	1	1	93.1	77.2 to 99.2
75+	27	19	2	6	70.4	49.8 to 86.3
Age unavailable	11	6	1	4	54.6	23.4 to 83.3
White	73	64	4	5	87.7	77.9 to 94.2
Nonwhite	14	11	1	2	78.6	49.2 to 95.3
Length of stay <3 days	15	13	1	1	86.7	59.5 to 98.3
Length of stay ≥3 days	115	97	5	13	84.4	76.4 to 90.5
Transferred from or to another hospital	62	54	1	7	87.1	76.2 to 94.3
Not transferred	81	69	5	7	85.2	75.6 to 92.1

DISCUSSION

Our results suggest that AMI can be reliably identified in the MSDD. Rates of record retrieval were higher than expected across all Data Partners. This was attributed in part to successful relationship building with hospital partners, distribution of comprehensive informational packets and frequent follow-up on all chart requests. The project’s status as an FDA Sentinel Initiative effort may also have contributed to high rates of chart return. We had the opportunity to review over 90% of possible cases, demonstrating that the MSDD was capable of yielding a retrieval rate comparable to that of previous AMI validation efforts.(17, 19, 23, 31)

Previous validation studies(16-33) have found AMI to be a reliably coded event.(34) Published AMI validation studies frequently report PPV’s of 90% or greater,(19, 21, 24, 26, 27, 29, 31-33) although some studies report values lower than this range.(18, 23) With a slightly lower PPV than past validation studies, our project provides an update to previously published studies in several important ways. We drew on a broader patient population, validating cases through medical record review and using a contemporary definition of AMI.(12, 13) We found that the PPV rate varied across the 4 Data Partners. Our small sample size leads us to interpret all between-group comparisons with caution.

A number of prior studies validating the diagnosis of AMI in different patient samples have been limited to a single state or province,(18, 19, 21, 29, 33) although several studies conducted outside the U.S. incorporated a broader segment of the study country’s population. (24, 27, 32) In addition, other studies used exclusion criteria based on age(29) or included only Medicaid,(19) Medicare(21) or Veterans Administration(33) populations. We drew potential cases from multiple Data Partners across the U.S. and we did not exclude any cases based on age or other patient characteristics. With the exception of one study by Yeh et al.,(29) our validation approach differed from those of previous studies due to our decision to conduct chart reviews and due to the validation criteria we used. Several prior studies did not conduct chart reviews but relied instead on linking data to a cardiac registry database(18, 24) or on questionnaires sent to patients’ primary providers for confirmation of the identified AMI.(32) Where chart reviews were conducted, validation criteria included World Health Organization criteria,(21, 26, 35) Women’s Health Initiative criteria (19, 36) and 2003 criteria from the American Heart Association.(27, 37) Yeh et al. used similar validation criteria to ours as well as an identical case definition and found a PPV of 96.7% for MIs occurring in members of a single health plan, Kaiser Permanente Northern California, between 1999 and 2008. Our inclusion of a broader population may provide a more generalizable PPV estimate for future FDA surveillance efforts nationwide.

Limitations of our validation activity include our small sample size. While we sampled from a broad segment of the insured U.S. population, our small overall number of records makes it difficult to draw conclusions about PPV in various patient subgroups. The study population also consisted only of patients with health insurance.

This validation project demonstrated high rates of record retrieval from hospitals nationwide and employed vigorous validation criteria, including the use of expert, board-certified cardiologist reviewers to adjudicate using updated criteria for the definition of AMI. Our PPV for AMI was somewhat lower than had been previously suggested by the literature, but provides a contemporary estimate of the reliability of this ICD-9-CM-based algorithm for use in future surveillance efforts conducted in the MSDD.

CONCLUSIONS

The overall PPV of potential AMI cases, as identified using an ICD-9-CM-based algorithm, may be acceptable for safety surveillance purposes; however, PPVs varied across the 4 Data Partners included in this validation activity. Our investigation provides a contemporary estimate of the reliability of this ICD-9-based algorithm for use in future surveillance efforts conducted in the Mini-Sentinel program.

Key points.

• Acute Myocardial Infarction (AMI) can be reliably identified for medical product safety surveillance in the Food and Drug Administration’s (FDA’s) Mini-Sentinel Distributed Database (MSDD), using an ICD9-CM-based algorithm and contemporary validation criteria.

• The positive predictive value (PPV) of this AMI algorithm in the MSDD is 86%, slightly lower than observed in other published studies.

• This validation activity demonstrates high medical record retrieval rates in the setting of an FDA-funded public health surveillance effort conducted in a distributed system of electronic healthcare databases.

Appendix A

Mini-Sentinel acute myocardial infarction validation: Abstraction form

Appendix B

Mini-Sentinel acute myocardial infarction validation: Adjudication for