Validation of physician billing and hospitalization data to identify patients with ischemic heart disease using data from the Electronic Medical Record Administrative data Linked Database (EMRALD)

Abstract

BACKGROUND:

Reporting of ischemic heart disease (IHD) prevalence in Canada has been based on self-report or patients presenting to hospital. However, IHD often presents and can be managed in the outpatient setting.

OBJECTIVES:

To determine whether the combination of hospital data and physician billings could accurately identify patients with IHD.

METHODS:

A random sample of 969 adult patients from the Electronic Medical Record Administrative data Linked Database (EMRALD) – an electronic medical record database of primary care physicians in Ontario linked to administrative data for the province of Ontario – was used. A number of combinations of physician billing and hospital discharge abstracts were tested to determine the accuracy of using administrative data to identify IHD patients.

RESULTS:

Two physician billings within a one-year period (with one of the billings by a specialist or a family physician in a hospital or emergency room setting) or a hospital discharge abstract gave a sensitivity of 77.0% (95% CI 68.2% to 85.9%), a specificity of 98.0% (95% CI 97.0% to 98.9%), a positive predictive value of 78.8% (95% CI 70.1% to 87.5%), a negative predictive value of 97.7% (95% CI 96.8% to 98.7%) and a kappa of 0.76 (95% CI 0.68 to 0.83).

CONCLUSIONS:

A combination of physician billing and hospital discharge abstracts can be used to identify patients with IHD. Population prevalence of IHD can be measured using administrative data.

Ischemic heart disease (IHD) is one of the leading causes of mortality and morbidity in Canada (1,2). IHD accounts for a significant proportion of all acute hospitalizations and costs the Canadian economy an estimated 18.6 billion dollars a year (3).

The accuracy of administrative data in the identification of IHD patients has been validated in previous studies against self-report population surveys (4,5) and against inpatient hospital data (6–8). Compared with self-report surveys, the sensitivity of administrative data for identifying IHD has ranged from 29% to 68% (4,9). However, this poor sensitivity may not be a result of the inability of administrative data to identify patients with this disease but, rather, inaccuracies of self-report (10). Previous validation of inpatient hospital data against hospital discharge abstracts have reported a sensitivity ranging from 45% to 83% (8,11,12). However, these studies have been limited to hospitalized patients and have limited their analysis to one code such as IHD or unstable angina alone. Using a single code may not capture all patients with acute coronary syndrome, and patients who have had a percutaneous coronary intervention (PCI) or coronary artery bypass graft surgery (CABG) by definition also have IHD. Furthermore, IHD patients may not initially present to hospital and would not be detected when using hospital-based administrative data only, thereby excluding patients diagnosed and managed in the outpatient setting.

Patients with IHD have been identified using data from general practitioner electronic medical records (EMRs) in the United Kingdom (UK) (13–15). However, the UK has a long history of computerization within general practitioner offices, with well-organized research databases (16,17) that have been coded by the general practitioner, making identification of patients with disease conditions relatively straightforward. Thus, it follows that assessments of the capture rate of IHD within the EMR in the UK have been determined to be high (18).

In Canada, the uptake of EMRs in the primary care setting is increasing, which has the potential to allow for assessment of the prevalence of IHD in the community and not just the acute setting. To overcome the limited scope of previous Canadian studies, we used a comprehensive list of IHD codes, and both hospital discharge abstracts and physician billing data to assess the validity and reliability of administrative data in identifying patients who have IHD. We used physician-diagnosed IHD contained within family physician EMRs in Ontario as the reference standard to improve on the inaccuracies of self-reporting, and to include both in- and outpatient information.

METHODS

As part of a study for the Canadian Cardiovascular Outcomes Research Team, a convenience sample of 17 physicians using Practice Solutions EMR (Practice Solutions, Canada) contributed their data to create an EMR Administrative data Linked Database (EMRALD). Participating physicians were required to have a minimum of two years of data on their EMR system before being included in the study. This criterion was established to ensure that a reasonable amount of health information would be recorded in the physician EMRs. Physician time on the EMR was calculated from the date the data were obtained. Data from the EMR were extracted between June and December 2007. The duration that data were recorded varied from a maximum of 18 years to a minimum of two years.

There were 19,376 active adult patients (20 years of age or older as of December 31, 2007) with valid health card numbers. An active patient was defined as having at least two visits in the three years before the date the data were downloaded. A 5% random sample of these patients (n=969) was taken, and three trained chart abstractors reviewed all entries in the patient records and scored each entry to indicate the presence or absence of an IHD diagnosis. Entries included the cumulative patient profile (including a problem list and health history), progress notes (generated at each family physician visit), diagnostic tests, specialist letters, operating room reports, emergency room records and hospital discharge summaries. The diagnosis of IHD was classified as ‘hard’ or ‘soft’ IHD. ‘Hard’ IHD was defined as evidence of IHD in a coronary catheterization report or specialist letter. ‘Soft’ IHD was defined as evidence of IHD only in the family physician record such as the cumulative patient profile or progress notes. Patients who had documented evidence of a myocardial infarction (MI), PCI or CABG were also included as IHD patients. Patients were considered to have IHD if one or more EMR entries were scored as depicting presence of the disease.

The measurement of intraobserver reliability resulted in kappa values exceeding 0.80, which indicated very good agreement. When measuring interobserver reliability, the kappa values exceeded 0.85 for all comparisons among the three abstractors, which also indicated very good agreement.

Patient health card numbers were replaced with a unique identification number and anonymously linked to the administrative data holdings for the province of Ontario housed at the Institute for Clinical Evaluative Sciences (ICES) – including the Canadian Institute for Health Information hospital discharge abstracts database, which records the most responsible diagnosis for a hospital admission and up to 15 comorbid conditions using The International Classification of Diseases and Related Health Problems, 9th Revision (before fiscal year 2002 [410–414]) and 10th Revision (beginning at fiscal year 2002 [I20–I25]) developed by the WHO. The most responsible diagnosis or any one of the secondary diagnostic codes was used to indicate that a patient had IHD. In addition, procedure codes for PCI and CABG were included. The Canadian Classification of Health Interventions (CCI) coding system was used beginning in 2002, and the Canadian Classification of Diagnostic, Therapeutic and Surgical Procedures (CCP) coding system was used before 2002. The CCP codes for PCI were 4802, 4803 and 4809, and the CCI codes were 1IJ50 and 1IJ57GQxx. The CCP code for CABG was 481 and the CCI code was 1IJ76.

The Ontario Health Insurance Plan (OHIP) physician billing database records over 95% of the outpatient visits for the residents of Ontario (19). Codes 410, 412 and 413 were used to denote IHD in the OHIP physician claims database. Fee codes R742 and R743 denoted CABG, and fee codes Z434 and G298 denoted PCI. In addition, OHIP records the type of physician (type 00 indicates general practitioner or family physician) submitting the billing encounter and the location of the encounter. Using the EMR chart-abstracted results for IHD, various case definitions were compared using combinations of Canadian Institute for Health Information and OHIP to determine whether there was an administrative data algorithm that accurately identified patients with IHD. The present study also looked at whether requiring the OHIP billing to be performed by a specialist or a general practitioner/family physician in a hospital or emergency room setting improved the accuracy of the results.

The sensitivity of the administrative data was calculated as the proportion of patients with IHD identified by the administrative data algorithm who had IHD according to the ‘reference standard’ of manual EMR abstraction. Specificity was calculated in the same manner, except that it was based on individuals without IHD. Positive predictive value (PPV) was defined as the proportion of IHD patients identified by the administrative data algorithm that were confirmed by the ‘reference standard.’ Negative predictive value was defined similarly for patients who did not have IHD according to the manual EMR abstraction. Kappa statistics for agreement between the EMR abstracted data and the administrative data were also calculated. All analyses and 95% CIs for these proportions were calculated using the binomial approximation method in SAS version 9.1 (SAS Institute Inc, USA).

Various combinations of administrative data algorithms were validated against an IHD cohort that included ‘hard’ IHD, MI, PCI or CABG with and without ‘soft’ IHD against the 5% random sample (n=969). The present project received ethics approval through the Sunnybrook Health Sciences Centre (Toronto, Ontario) Research Ethics Board.

RESULTS

Physicians were mainly men (70.6%) and in urban practice (58.3%). All but one of the physicians was in group practice. The mean (± SD) period of time in practice was 20.5±10.2 years and the average length of time on the EMR was 7.4±7.3 years. The adult patient cohort (20 years of age or older) was 46.3% men and the mean age was 49.0±17.2 years.

Using only one hospital discharge abstract had high specificity, PPV and negative predictive value, but sensitivity was low. Adding one physician billing diagnosis or a hospital discharge abstract increased the sensitivity but dropped the PPV to unacceptably low levels when including both hard and soft definitions for IHD (Table 1). Using a case definition of two physician billing codes in one year alone or with a hospital discharge abstract increased the sensitivity but dropped the PPV when compared with hospital discharge abstract alone. Allowing for a two- or three-year window for the second physician billing code had little impact, while requiring a third physician billing code within a one-year period resulted in a 5% drop in sensitivity with only a 1% increase in PPV. Requiring one of the physician billing codes to be billed by a specialist or general practitioner/family physician in a hospital or emergency room setting provided the most optimal combination of sensitivity and PPV (Table 1).

TABLE 1.

Comparison of manually abstracted occurrences of ischemic heart disease (IHD; including ‘hard’ or ‘soft’ IHD, myocardial infarction, percutaneous coronary intervention or coronary artery bypass grafting) from primary care electronic medical records versus administrative data

Rule label	Total agreement, %	Sensitivity, %	Specificity, %	PPV, %	NPV, %	Kappa
1 CIHI	96.1 (94.9–97.3)	62.1 (51.9–72.3)	99.4 (98.9–99.9)	91.5 (84.4–98.6)	96.4 (95.2–97.6)	0.72 (0.64–0.80)
1 OHIP or CIHI	92.0 (90.2–93.7)	85.1 (77.6–92.5)	92.6 (90.9–94.4)	53.2 (44.9–61.5)	98.4 (97.6–99.3)	0.61 (0.53–0.67)
2 OHIP in 1 year	95.4 (94.0–96.7)	72.4 (63.0–81.8)	97.6 (96.6–98.6)	75.0 (65.7–84.3)	97.3 (96.2–98.4)	0.71 (0.63–0.79)
2 OHIP in 1 year or CIHI	95.7 (94.4–97.0)	77.0 (68.2–85.9)	97.5 (96.5–98.5)	75.3 (66.3–84.2)	97.7 (96.7–98.7)	0.73 (0.66–0.81)
2 OHIP in 2 years or CIHI	95.6 (94.3–96.9)	77.0 (68.2–85.9)	97.4 (96.3–98.4)	74.4 (65.4–83.5)	97.7 (96.7–98.7)	0.73 (0.66–0.81)
2 OHIP in 3 years or CIHI	95.7 (94.4–97.0)	78.2 (69.5–86.8)	97.4 (96.3–98.4)	74.7 (65.8–83.7)	97.8 (96.9–98.8)	0.74 (0.67–0.82)
3 OHIP in 1 year or CIHI	95.6 (94.3–96.9)	72.4 (63.0–81.8)	97.8 (96.9–98.8)	76.8 (67.7–86.0)	97.3 (96.2–98.4)	0.72 (0.64–0.80)
All specialist, or GP in hospital or emergency – 2 OHIP in 1 year or CIHI	95.8 (94.5–97.0)	73.6 (64.3–82.8)	98.0 (97.0–98.9)	78.0 (69.1–87.0)	97.4 (96.4–98.5)	0.73 (0.66–0.81)
At least one specialist, or GP in hospital or emergency – 2 OHIP in 1 year or CIHI	96.1 (94.9–97.3)	77.0 (68.2–85.9)	98.0 (97.0–98.9)	78.8 (70.1–87.5)	97.7 (96.8–98.7)	0.76 (0.68–0.83)

Data in parentheses represent the 95% CI for each point estimate. Gold standard is the electronic medical records chart-abstracted IHD (n=87); total n=969 and IHD prevalence of 9.0%. CIHI Canadian Institute for Health Information Discharge Abstract Database; GP General practitioner; NPV Negative predictive value; OHIP Ontario Health Insurance Plan Physician Claims Database; PPV Positive predictive value

When using only ‘hard’ IHD in the abstraction definition of denoting presence of the disease, the relative patterns of the various combinations of physician billing and hospital discharge abstracts were similar compared with when ‘soft’ IHD was included; however, sensitivity was higher and PPV was lower when only evidence of ‘hard’ IHD, MI, PCI or CABG was used as the reference standard (Table 2).

TABLE 2.

Comparison of manually abstracted occurrences of ischemic heart disease (IHD; including ‘hard’ IHD only, myocardial infarction, percutaneous coronary intervention or coronary artery bypass grafting) from primary care electronic medical records versus administrative data

Rule label	Total agreement, %	Sensitivity, %	Specificity, %	PPV, %	NPV, %	Kappa
1 CIHI	96.7 (95.6–97.8)	70.1 (59.2–81.1)	98.7 (97.9–99.4)	79.7 (69.4–89.9)	97.8 (96.8–98.8)	0.73 (0.64–0.82)
1 OHIP or CIHI	91.1 (89.3–92.9)	89.6 (82.2–96.9)	91.2 (89.4–93.1)	43.2 (34.9–51.4)	99.2 (98.5–99.8)	0.54 (0.46–0.62)
2 OHIP in 1 year	95.6 (94.3–96.9)	80.6 (71.1–90.1)	96.7 (95.5–97.8)	64.3 (54.0–74.5)	98.5 (97.7–99.3)	0.69 (0.60–0.78)
2 OHIP in 1 year or CIHI	95.7 (94.4–97.0)	85.1 (76.5–93.6)	96.5 (95.2–97.7)	64.0 (54.1–74.0)	98.9 (98.2–99.6)	0.71 (0.62–0.79)
2 OHIP in 2 years or CIHI	95.6 (94.3–96.9)	85.1 (76.5–93.6)	96.3 (95.1–97.6)	63.3 (53.4–73.3)	98.9 (98.2–99.6)	0.70 (0.62–0.78)
2 OHIP in 3 years or CIHI	95.7 (94.4–97.0)	86.6 (78.4–94.7)	96.3 (95.1–97.6)	63.7 (53.9–73.6)	99.0 (98.3–99.6)	0.71 (0.63–0.79)
3 OHIP in 1 year or CIHI	96.0 (94.7–97.2)	82.1 (72.9–91.3)	97.0 (95.9–98.1)	67.1 (56.9–77.2)	98.6 (97.9–99.4)	0.72 (0.63–0.80)
All specialist, or GP in hospital or emergency – 2 OHIP in 1 year or CIHI	96.0 (94.7–97.2)	82.1 (72.9–91.3)	97.0 (95.9–98.1)	67.1 (56.9–77.2)	98.6 (97.9–99.4)	0.72 (0.63–0.80)
At least one specialist, or GP in hospital or emergency – 2 OHIP in 1 year or CIHI	96.1 (94.9–97.3)	85.1 (76.5–93.6)	96.9 (95.8–98.0)	67.1 (57.1–77.1)	98.9 (98.2–99.6)	0.73 (0.65–0.81)

Data in parentheses represent the 95% CI for each point estimate. Gold standard is the electronic medical records chart-abstracted IHD (n=67); total n=969 and IHD prevalence of 6.9%. CIHI Canadian Institute for Health Information Discharge Abstract Database; GP General practitioner; NPV Negative predictive value; OHIP Ontario Health Insurance Plan Physician Claims Database; PPV Positive predictive value

DISCUSSION

The present study showed that an administrative data algorithm of two physician billing codes (with one of the physician billing codes being from a specialist or general practitioner/family physician in a hospital or emergency room setting) or one hospital discharge abstract can identify patients with IHD with reasonable validity. We did, however, find that depending on the definition of what constitutes evidence for the presence of IHD, the false-positive rate varied considerably. This highlights the occasional inconsistency between diagnosis of IHD by a specialist compared with a family physician. Not including family physician diagnosis or recording of angina or silent MI, which may be managed exclusively in the outpatient setting by a family physician, may lead to under-reporting of the population prevalence of IHD. Admittedly, there are cases of presumed and even treated IHD by the family physician, which may subsequently not be confirmed after further testing; however, the false positives captured with administrative data may be counter balanced with the lack of 100% sensitivity.

It is difficult to compare our findings with previous validation studies because they used a reference standard derived from either hospital charts, clinical patient registries based on hospitalization, or self-report surveys. Our study had substantially higher sensitivity than that found using self-report as a reference standard (4). This likely reflects the higher quality of using physician records and notes as a reference standard, and using data abstracted by individuals with clinical expertise and, thereby, not subject to lay interpretation of what constitutes IHD. Other studies in Ontario have reported variable sensitivities when comparing hospital discharge abstracts with re-abstracted hospital charts (8) and the Fastrak II inpatient registry (20). However, these studies assessed a narrower definition of IHD. It is not surprising that our findings show that the use of physician billing data increases the sensitivity of identifying patients with IHD because it is a chronic disease condition that can often present to the family physician and does not necessarily require hospitalization. In addition, our administrative data algorithms included procedure codes for PCI or CABG, which may have resulted in more complete and accurate capture of patients with IHD.

Limitations

There are limitations to the present study that need to be acknowledged. First, we only used a convenience sample of family physicians on one EMR system and, thus, the generalizability of our results may be questioned. However, prevalence of IHD should not be affected by whether the physician is on an EMR or which EMR they are on. Indeed, the prevalence of IHD in our randomly selected patients was similar to the reported prevalence in other industrialized countries (21). Second, varying our definition of IHD using hard and soft evidence for the presence of the disease found 20 additional cases when the family physician record alone was sufficient evidence. While some of these cases represent new unsubstantiated diagnoses, many were due to the short duration of the EMR record or the lack of recording of externally generated letters in the EMR. Of note, the requirement of one of the physician billings to be by a specialist or family physician in a hospital setting slightly reduced the rate of false positives. Third, using this administrative data algorithm may slightly underestimate the true population prevalence of IHD because this type of study and using administrative data require the patient to have sought medical care and to have had their IHD diagnosed, recorded in the EMR and coded in the billing data.

While prescriptions for nitrates were not included in the present validation because, through administrative data, nitrate prescriptions for patients in Ontario include only those 65 years of age or older, the EMR has prescribing data for all ages and, thus, could include nitrate prescriptions in a case definition. This warrants further investigation.

Improved efforts to standardize coding by coder, clinicians, administrators and governments will further improve the quality, efficiency and usefulness of administrative data for surveillance of diseases in Ontario and across the country. Nonetheless, our results indicate that the presence of IHD can be accurately captured by administrative data using a combination of physician billing data and hospital discharge abstracts. Thus, prevalence of IHD is measurable over a large population with administrative data, allowing for further evaluation of patterns and outcomes of care.