Design for Validation of Acute Myocardial Infarction Cases in Mini-Sentinel

Abstract

Purpose

To describe the Acute Myocardial Infarction (AMI) Validation project, a test case for health outcome validation within the FDA-funded Mini-Sentinel pilot program.

Methods

The project consisted of four parts: (1) case identification: developing an ICD9-based algorithm to identify hospitalized AMI patients within the Mini-Sentinel Distributed Database; (2) chart retrieval: establishing procedures that ensured patient privacy (collection and transfer of minimum necessary amount of information, and redaction of direct identifiers to validate potential cases of AMI; (3) abstraction and adjudication: trained nurse abstractors gathered key data using a standardized form with cardiologist adjudication; and (4) calculation of the positive predictive value of the constructed algorithm.

Results

Key decision points included: (1) breadth of the AMI algorithm; (2) centralized vs. distributed abstraction; and (3) approaches to maintaining patient privacy and to obtaining charts for public health purposes. We used an algorithm limited to ICD9 codes 410.x0-410.x1. Centralized data abstraction was performed due to the modest number of charts requested (<155). The project’s public health status accelerated chart retrieval in most instances.

Conclusions

We have established a process to validate AMI within Mini-Sentinel, which may be used for other health outcomes. Challenges include: (1) ensuring that only minimum necessary data is transmitted by Data Partners for centralized chart review; (2) establishing procedures to maintain data privacy while still allowing for timely access to medical charts; and (3) securing access to charts for public health uses that do not require IRB approval while maintaining patient privacy.

Introduction

In 2007, the U.S. Congress passed the FDA Amendments Act (FDAAA) mandating the FDA to establish a postmarket risk identification and analysis system to link and analyze safety data from multiple sources.¹ In May 2008, in response to the Congressional mandate, the FDA launched the Sentinel Initiative, a long-term program designed to create a national electronic monitoring system for medical product safety (the Sentinel System). The Sentinel System is being developed and implemented in stages and, when fully functional, will complement FDA’s existing postmarket safety surveillance systems.

The Mini-Sentinel pilot, a contract awarded by FDA to Harvard Pilgrim Health Care Institute (HPHCI) to develop the scientific operations needed for the eventual Sentinel System, is being conducted as a collaborative effort between FDA and a consortium of institutions led by HPHCI.^{2, 3} Because accurate and timely identification of health outcomes is an essential component of active safety surveillance, Mini-Sentinel convened a workgroup to establish a process for identification and validation of a selected health outcome: acute myocardial infarction (AMI). This is the first health outcome to be validated under Mini-Sentinel. In addition to developing and validating an algorithm to identify hospitalized AMI cases within the Mini-Sentinel Distributed Database,⁴ another goal of the workgroup was to design an efficient validation process that could be used as a model for future validation efforts of other health outcomes of interest.

This paper describes how the AMI Validation Workgroup developed the Mini-Sentinel validation process for AMI and discusses the barriers encountered. Additional information can be found in our final report.⁵

Overview of Design for the AMI Validation Process

The Mini-Sentinel AMI Validation project was a collaboration among multiple entities. The FDA’s Sentinel Initiative team contributed input and oversight throughout the validation effort. The Mini-Sentinel Operations Center, comprised of researchers and staff from HPHCI, provided a scientific, analytic and administrative infrastructure. The Operations Center facilitated communication between collaborators, designed programs for chart retrieval and coordinated the retrieval effort. Academic researchers from Meyers Primary Care Institute and University of Massachusetts Medical School were charged with designing the approach to chart identification, identifying necessary chart components, and performing abstraction and adjudication. Four Data Partners participated in this project, including HealthCore, Inc.; Humana; three member health plans within the Kaiser Permanente Center for Effectiveness and Safety Research (CESR); and two health plans in the HMO Research Network (HMORN). Data Partners implemented computer programs written in SAS by the Operations Center to identify likely AMI cases, retrieved, copied, de-identified and transmitted selected healthcare data to the lead team through the Operations Center via a secure web portal.

The AMI validation process consisted of four parts: (1) an approach to case identification with the goal of producing an algorithm that would reliably identify AMI cases ; (2) a protocol for case retrieval from the Data Partners, which outlined necessary chart components to confirm the AMI diagnosis and established effective approaches to obtaining and de-identifying chart information; (3) a parsimonious data abstraction form including relevant elements derived from the medical chart components and completed by trained nurse abstractors; and (4) an adjudication protocol for confirmation of the AMI diagnosis by cardiologist adjudicators. The culmination of this effort will be a determination of the positive predictive value (PPV) of the algorithm. The following sections focus on the first three parts of this process.

Case Identification

The overarching goals of this project were to validate the diagnostic codes used to identify likely AMI cases and to design an efficient validation process that could be used for future validation of other health outcomes in Mini-Sentinel. It was determined that 100 charts would be sufficient to obtain a reasonable PPV and establish the validation process. To obtain more contemporary findings, we decided to include only patients who were hospitalized for AMI between January 1, 2009 and December 31, 2009 for whom there were records in the Mini-Sentinel Distributed Database, which currently comprises administrative and claims data formatted locally into a common data model.⁴ Hospital stays of less than 24 hours and observation stays were included. There were no restrictions on age, sex, other diagnoses, or other patient characteristics, but patients were required to be enrollees of the respective health plan for the entire duration of hospitalization.

We had the opportunity to consult with a concurrent Mini-Sentinel workgroup that was charged with developing an active surveillance protocol for AMI,⁶ which informed decision-making in the early stages. The two workgroups began by reviewing the literature and examining prior completed reviews^{7, 8} to identify previously used algorithm components, with a focus on those yielding the highest PPVs (Table 1). We also consulted with cardiologists, cardiovascular researchers, and FDA review staff with expertise in cardiovascular disease. We considered the types of data that would likely be available from the medical records relating to the patient’s index hospitalization, as well as the likelihood of access to information both prior to the hospitalization and following hospital discharge for AMI survivors. We reviewed the pathophysiology of AMI and acute coronary syndrome, and discussed whether to create a strict definition of AMI or a definition that more broadly captured cardiovascular disease states as part of a continuum.

Table 1.

Critical information gathered from review of previous acute myocardial infarction (AMI) validation studies of electronic health data

Data types	Algorithm components	Algorithm structure	Algorithm performance metrics
Registry Hospital Administrative Data Claims data Other (Electronic health record, survey data)	ICD-8, ICD-9, ICD-10 discharge codes Position of this code (primary position; first position; second position; “Most responsible diagnosis”) Diagnosis-related Group (DRG) codes Other criteria: length of hospital stay (3-180 days); transfer to or from outside hospital; death during hospital stay; previous AMI in past 8 weeks;	Combination of codes by type (ICD-8, 9, 10, DRG) using OR vs. AND Combination by position (primary only vs. primary OR first vs. primary OR first OR second, etc.) Combination of code with other criteria (example: ICD code AND length of stay >3 days).	Sensitivity Specificity Positive predictive value (PPV)

In reviewing the literature, we found a wide range of ICD-9 codes in use, with a few studies assessing ICD-8 or ICD-10 codes and several studies combining ICD codes with other criteria.^7-20 We identified ICD-9 code 410 as the code most frequently yielding PPVs in the mid to high 90% range, and we also identified the need to specify the ICD-9 code using 2 decimal places. Since the number 2 in the second place after the decimal (i.e., 410.x2) indicates a past MI, we limited our sample to 410.x0 or 410.x1. Our algorithm therefore identified patients with ICD-9 hospital discharge codes (a principal or primary discharge code only^9,15,17,19) of 410.x0 and 410.x1. If a Data Partner does not have a diagnosis designated as principal or primary, we used the first-listed discharge diagnosis. Although we reviewed previously studied algorithms that incorporated hospital length of stay, we did not find that this reliably increased the PPV and, therefore, did not include this requirement in the final algorithm.^{10, 12, 16, 17}

The two workgroups held additional meetings with members of both teams in order to reach consensus on a common AMI definition for both studies. We discussed including in our definition deaths occurring within one day of an emergency department visit for acute ischemic heart disease (ICD-9 code: 411.1, 411.8, 413.x), but decided against including these additional ICD rubrics due to concerns regarding the adequacy of information that would be available to adjudicate these cases. Codes including 412 (old myocardial infarct) and 414 (chronic ischemic heart disease) were excluded in an effort to focus on acute events. Our final algorithm to identify AMI patients identified patients with ICD-9 principal (or first-listed) discharge codes 410.x0 and 410.x1.

Based on this algorithm, the Operations Center developed a SAS program, tested it with two Data Partners for accuracy, and then distributed it to all Data Partners participating in the project. To obtain 100 cases for adjudication, efforts were made to identify approximately 150 cases (assuming 67% retrieval) across all participating Data Partners, with each Data Partner pursuing an equal number of cases. In order to identify a random sample of likely AMI cases and the hospitals in which they received care, participating Data Partners executed the SAS program to query their own locally maintained administrative and claims data (see following section of this report).

Case Retrieval

Centralized vs. distributed chart abstraction

The workgroup developed a protocol for retrieving medical chart information. In order to proceed with chart retrieval, we needed to: 1) decide whether chart abstraction would take place centrally or in a distributed manner (i.e., each Data Partner abstracts its own charts); and 2) establish protocols for ensuring patient privacy and data security, and for helping Data Partners comply with their regulatory responsibilities when supplying data for public health surveillance, which does not require individual consent or privacy authorization.²¹ Because the abstraction process would have major implications in terms of the amount of information transferred, the workgroup held multiple meetings to address the question of centralized vs. distributed data abstraction. Before selecting an approach to pursue, the workgroup discussed why a centralized versus distributed approach might be preferred for the purpose of this validation activity and as a model for future Mini-Sentinel validations. Specifically, the workgroup discussed the following issues:

1. Capacity to maintain patient de-identification: Individually identifiable information other than dates of service in the charts was intended to be fully redacted prior to transmission from Data Partner sites to the Operations Center. Data Partners noted, however, that some individually identifiable information might be overlooked, especially in extremely large charts, and this might pose a risk to patient privacy.

2. Existing infrastructure within the Data Partners to perform medical chart abstraction by trained abstractors: Some Data Partners advocated for a distributed approach since they had available experienced abstractors who could be trained to perform the required abstraction tasks. Others did not have experienced abstractors on site.

3. Quality of data abstraction: Given the modest number of medical records to be reviewed and the relatively high number of sites, it would be challenging to train abstractors to perform only a handful of abstractions and still maintain adequate quality and reliability. Additionally, some Data Partners would not be using nurses or other individuals with relevant healthcare experience as abstractors, leading to an increased likelihood of variation in abstraction quality.

4. Short-term efficiency: A distributed approach would require abstractor training and evaluation at multiple sites, potentially impacting negatively the timeline for the overall abstraction effort.

5. Long-term efficiency: Efficiency of future Mini-Sentinel validation projects was also a consideration. In the future, when another health outcome needs to be validated, a centralized approach would require training of a limited number of abstractors instead of periodically retraining numerous abstractors across multiple Data Partners. A centralized approach could maximize resources and minimize the amount of time required to abstract necessary information.

After careful consideration, a centralized approach was ultimately pursued and selected components of medical records were extracted. All charts were redacted of direct identifiers, but retained dates of service to reduce the amount of information transmitted to the minimum amount necessary to accomplish the public health purpose of the project. The redacted redcords were then securely transferred via the Mini-Sentinel Secure Portal to the lead team at Meyers Primary Care Institute for centralized abstraction.

Determination of chart components

Once a centralized abstraction approach was chosen, the lead team proposed a list of critical chart components and other information they considered important for the validation. This initial list was developed broadly and then narrowed down based on input from Data Partners, the Operations Center, the FDA, and individuals with clinical and epidemiologic expertise relevant to cardiovascular disease.

Abstraction tools from various validation studies were reviewed to inform decisions on the list of critical chart components and other information to be extracted.^{10, 20, 22} In response to Data Partners’ concerns over the amount of information to be extracted and transferred, the lead team further excluded several chart components (e.g., medications and patient vital signs) in order to lessen the amount of information to be transferred.

The Operations Center reviewed the revised list in relation to the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule’s minimum necessary standard and confirmed that the critical chart components and other information requested constituted the minimum amount of information for the activity. The chart components that were requested are summarized in Table 2.

Table 2.

Components extracted from each chart of patients with suspected acute myocardial infarction (AMI)

○ Admission history and physical

○ Discharge summary

○ Transfer records

○ Cardiology consult notes

○ Autopsy reports/ Death notes

○ EMT/Ambulance notes

○ Emergency Department notes

○ Copies of all 12 lead electrocardiograms

○ Laboratory reports

○ Cardiac catheterization report

○ Percutaneous coronary intervention reports

○ Cardiac bypass surgery report

○ Cardiac stress tests/nuclear stress tests/reports

Certain items requested remained broad in scope. For example, copies of all laboratory results were requested. This was done in order to obtain cardiac biomarker information.

Cardiac biomarkers are one of the critical items of interest for AMI validation, but only represent a subset of all laboratory results. This decision was made in order to avoid the need for a highly trained individual at each site capable of determining which specific pages of the laboratory report section of the medical record were required.

Obtaining chart information

After the list of chart components was finalized, Data Partners proceeded to execute the SAS program locally, identifying a random sample of likely AMI cases whose medical records were to be located. Data Partners then asked source data holders (e.g., individual hospital medical records departments) for access to these patients’ records. Source data holders either sent the medical records to vendors commissioned to extract, copy, and redact the requested information, or allowed Data Partners direct access to records for extraction, copying and redaction. Redacted chart data were sent to the Operations Center via the Mini-Sentinel Secure Portal.

The Operations Center provided each Data Partner with a privacy packet prepared by the Mini-Sentinel Privacy Panel. This packet included: 1) the Mini-Sentinel Privacy Panel White Paper describing data privacy issues in Mini-Sentinel;²³ 2) a letter from the Department of Health and Human Services Office for Human Research Protections (OHRP) to the FDA stating that the regulations OHRP administers do not apply to the Sentinel Initiative (OHRP oversees all IRBs); and 3) a letter from the FDA to the Mini-Sentinel Principal Investigator explaining FDA’s legal authority to obtain data for use in its Sentinel and Mini-Sentinel activities. The privacy packet described the legal basis for determining that the work of the Mini-Sentinel pilot constitutes a public health activity not under the purview of IRBs. Data Partners were strongly encouraged to disseminate the privacy packet to their IRBs and Privacy Boards as well as any other relevant entities.

The Operations Center also supplied a customizable letter template to provider sites, an instructional flowchart and a list of frequently asked questions. The letter (addressed to each provider site from specific Data Partners) explained the purpose of the project and what was being requested. Letters also explained that the request was being carried out on behalf of Mini-Sentinel and the FDA. The flowchart outlined the array of possible scenarios for chart retrieval and detailed the steps for chart redaction and data transmission. The frequently asked questions included a list of anticipated questions, such as “How were provider sites selected to participate in this activity?”, “Where will the abstracted data go”, and “How can I contact my Data Partner directly?”

Redaction of individually identifiable information was performed in accordance with HIPAA’s provisions for a ‘limited dataset,’ which is an alternative to using fully de-identified information. Under HIPAA, creation of a limited dataset requires removal of 16 direct identifiers, but allows for the inclusion of dates, geographic location (not as specific as street address), and any other code or characteristic not explicitly excluded.²¹ Redaction was completed before chart components were transferred to the Operations Center. Each Data Partner assigned a new, de-identified ID unique to each redacted chart, and maintained a crosswalk between the newly assigned IDs and the original IDs. The Operations Center did not receive this crosswalk for IDs; it was maintained by locally by each Data Partner.

Admission and discharge dates as well as dates corresponding to EKGs, laboratory results, procedures and tests were not redacted. This information was considered crucial for determining whether available EKGs and test results corresponded to the hospital stay of interest and therefore whether an AMI occurred during the identified hospital stay. In addition, for certain tests (EKGs, cardiac biomarkers), the results needed to be assessed by cardiologist adjudicators in chronological order. We considered assigning reference values for every date. Ultimately we opted not to pursue this approach since we felt that this would substantially increase workload and introduce multiple opportunities for error.

Data Partners were provided with credentials to login to the Mini-Sentinel Secure Portal for transferring, managing, or retrieving chart data. Security was managed within the folder structure of the site; the Secure Portal contains private folders accessible only to specified members within each Data Partner site and authorized Operations Center staff, as well as common folders defined for all users. Data Partners electronically uploaded redacted charts to their site-specific private folders. The Operations Center verified that all charts¹ were redacted thoroughly and then moved all files to a separate private folder, allowing the lead team access to the data.

Retrieval of charts requested

A total of 153 charts were requested, of which 143 were successfully retrieved. Approximately 80% of the 143 charts were obtained within three months following the request. The total time it took to received all 143 charts was approximately five months.

Challenges encountered during the chart retrieval process

Regarding level of burden, Data Partners were initially concerned that they would be required to obtain information from multiple sources, including outpatient medical records. However, clinical information relevant to the present validation study was to be extracted from medical records relating to only a single hospitalization.

Regarding privacy issues and IRB concerns, Data Partners described several challenges they encountered during the chart retrieval process. Though sometimes causing delays, most source data holder IRBs allowed charts to be located and retrieved after being provided with the privacy packet containing letters and documents that clarified the status of this validation project as a public health surveillance activity undertaken under the auspices of the FDA. However, despite the privacy packet, seven charts requested were not obtained due to IRB concerns and insistence on patient consent prior to releasing medical records.

Several other issues were brought to the workgroup’s attention by the Data Partners. Some redacted charts were sent by mail as opposed to electronic transmission, which led to delays in transferring data. One Data Partner found that including a list of frequently asked questions and answers along with each chart request led to improved turnaround times. Frequent inquiries concerning the disposition of charts and relationship building with hospital staff processing the request were also helpful in obtaining charts more quickly.

Abstraction and Adjudication

Development of abstraction and adjudication form

The lead team identified and reviewed a number of AMI abstraction forms and manuals used in past AMI validation activities. The team also consulted individuals with clinical and epidemiological expertise relevant to cardiovascular disease, reviewed the American Heart Association (AHA)’s Universal Definition of Myocardial Infarction^{24, 25} and the literature on troponin standardization,²⁶ and communicated with directors of laboratories on percentile cutoffs for what were considered to be “positive” troponin values. Based on clinical consultation and literature review, the lead team created a 36-item abstraction form that included: 1) general demographic information; 2) brief medical history; 3) cardiac biomarker information; 4) copies of electrocardiograms; 5) cardiac testing, procedure, and intervention information; and 6) information on disposition at the time of hospital discharge. The lead team trained two nurse abstractors to enter abstracted information into a Microsoft Access database and provided an accompanying instruction manual. Both abstractors gathered data from the first 10 cases. These abstractions were reviewed together with both nurse abstractors to ensure high inter-rater reliability on items critical for the adjudications.

In consultation with the FDA and individuals with clinical and epidemiologic expertise relevant to cardiovascular disease, the lead team created an adjudication protocol based on the AHA Universal Definition of Myocardial Infarction. In addition to abstracted data from the form described above, adjudicating cardiologists were provided with copies of electrocardiograms and copies of all cardiac test and procedure reports.

Access to redacted charts

Redacted components of the medical record were sent to the Operations Center via the Mini-Sentinel Secure Portal and then made available through this secure site to the lead team for data abstraction and case adjudication.

Challenges encountered during the chart abstraction and adjudication process

One of the more challenging issues that emerged in the design of the abstraction and adjudication forms related to differences in cardiac biomarker reference standards among different hospitals. It was essential to design abstraction materials that could adequately capture both biomarker results and reference standards, even when presented in a variety of ways from different hospital sources.

The workgroup was also challenged with reconciling the biomarker standards described in the published AHA definition of AMI²⁴ with laboratory values likely to be available in hospital records. While the published definition we were using defined abnormal biomarker values as falling “above the 99^th percentile of the upper reference limit”,²⁴ preliminary reviews of several charts showed that hospital laboratories did not routinely report percentile cut-offs. Through communication with the director of one hospital laboratory, we also found that the reported reference values did not always correspond to this 99^th percentile cut-off. We opted to capture any and all available information on reference standards from charts (i.e., from printed laboratory reports) but did not contact laboratory directors at each individual site for any further information.

Summary of lessons learned

We believe the following lessons learned will assist us in developing best practices when conducting similar Mini-Sentinel validation activities in the future:

1. Protocol Development: Engaging the FDA from the initial phase of the project facilitated development of the scope of work and setting of project goals.

2. Workgroup: A workgroup that includes the FDA, the Mini-Sentinel Operations Center, and the academic and Data Partners provided a platform for effective communication during protocol development and allowed us to identify potential challenges quickly. Regular workgroup meetings also allowed for achieving consensus among the workgroup members with regard to project timelines, requested chart components, and deliverables.

3. Chart Request: The Operations Center provided each Data Partner with a privacy packet to disseminate to source data holders. These documents outlined the activity as public health surveillance and detailed privacy and confidentiality standards used in the activity. Data Partners distributed these documents when making the initial chart request to source data holders, and we believe that this resource helped expedite the chart retrieval process.

4. Privacy Packet: One Data Partner mentioned that they were asked for contract information showing that the Data Partner was part of Mini-Sentinel. In future work, consideration should be given to adding this information to the privacy packet.

5. Chart Retrieval: The workgroup planned for multiple chart component extraction scenarios (i.e., vendors vs. non-vendor processes) which provided Data Partners with additional options and increased flexibility when retrieving charts.

6. Some Data Partners preferred to employ third-party vendors to retrieve charts. This process did briefly delay the project timeline as Data Partners faced challenges in finding reasonably priced vendors. Although each chart retrieval scenario required additional resource planning, this hybrid approach made for an efficient overall chart retrieval process.

7. Transferring of information: The Operation Center provided Data Partners access to the Mini-Sentinel Secure Portal for transferring, managing, and retrieving chart data. The Portal is a secure and immediate pathway for uploading data. The Operations Center was also able to track all data and provide abstractors access to charts through this environment.

8. Overall Timeline: Approximately seven to eight months were required to develop a protocol, request charts from Data Partner sites and complete abstraction and adjudication.

9. Considerations for future validation efforts: Future efforts to validate health outcomes on a national scale should pay particular attention to abstractor training and data capture for laboratory values (or other tests) whose reference ranges differ by hospital. Laboratory values or tests in which serial values must be captured can pose a challenge for patients who are transferred in from an outside hospital, due to the presence of multiple data formats and multiple reference ranges.

Conclusions

The AMI validation project has established a process for validating medical outcomes within Mini-Sentinel that can serve as a model for future surveillance validation activities. The project has provided important insights into the challenges inherent in conducting health outcome validation across public, academic, and private entities. Key issues identified include: 1) the need to determine the scope of health outcome definitions (broad vs. more focused); 2) the need for early assessment regarding centralized vs. distributed approaches to chart abstraction; and 3) the need to have a policy and systematic approach for maintaining patient privacy, data security, and addressing regulatory compliance issues under HIPAA and the Common Rule. In addition, it will be important for future validation projects to anticipate between-hospital differences in laboratory reference standards and between-hospital variations in how these data will be presented to the adjudicators.

Key points.

• Key decision points in the Mini-Sentinel acute myocardial infarction (AMI) validation process included: (1) determining the scope of the ICD-9 based AMI algorithm; (2) determining whether to pursue centralized vs. distributed abstraction; and (3) approaches to maintaining patient privacy and to addressing the project’s status as a public health activity that does not come under the purview of Institutional Review Boards (IRBs).

• We used an algorithm limited to ICD9 codes 410.x0-410.x1. Centralized data abstraction was performed due to the modest number of medical charts requested (<155 in total). The project’s public health status accelerated chart retrieval in most instances.