Abstract
Background
Electronic health data are routinely used to conduct studies of cardiovascular disease in the setting of the Veterans Health Administration (VA). Previous studies have estimated the positive predictive value (PPV) of International Classification of Disease, Ninth Revision (ICD-9) codes for acute myocardial infarction (MI), but the sensitivity of these codes for all true events and the accuracy of coding algorithms for prevalent disease status at baseline are largely unknown.
Methods
We randomly sampled 180 Veterans from the VA Puget Sound Health Care System who initiated diabetes treatment. The full electronic medical record was reviewed to identify prevalent conditions at baseline and acute MI events during follow up. The accuracy of various coding algorithms was assessed.
Results
Algorithms for previous acute events at baseline had high PPV (previous MI: 97%; previous stroke: 81%) but low sensitivity (previous MI: 38%; previous stroke: 52%). Algorithms for chronic conditions at baseline had high PPV (heart failure: 72%; coronary heart disease [CHD]: 85%) and high sensitivity (heart failure: 90%, CHD: 84%). For current smoking status at baseline, ICD-9 codes with pharmacy data had a PPV of 77% and sensitivity of 73%. The coding algorithm for acute MI events during follow up had high PPV (80%) and sensitivity (89%)
Conclusions
ICD-9 codes for acute MI events during follow up had high PPV and sensitivity. The sensitivity of ICD-9 codes for previous acute events at baseline was low, but a composite variable for baseline CHD had good accuracy.
Keywords: validation, electronic health data, diabetes mellitus, myocardial infarction, smoking
INTRODUCTION
The availability of healthcare databases has made possible the conduct of pharmacoepidemiologic studies of cardiovascular disease in large populations at low cost.1 However, studies that rely on administrative data may fail to detect true associations when events during follow up are missed.2 Validation studies typically sample subjects with diagnosis codes for outcomes of interest to estimate positive predictive values (PPVs),3–5 but do not account for missed events. Missed events may be problematic in the setting of the Veterans Health Administration (VA), where acute conditions such as myocardial infarction (MI) are often hospitalized at non-VA facilities and may not be captured fully in healthcare databases. Thus, while diagnosis codes for acute MI events have high PPV,3,4 they may have low sensitivity.
The presence of prevalent cardiovascular disease and cardiovascular risk factors, assessed to address confounding, may also be misclassified. Information on prevalent health conditions at baseline is typically collected during a time window of 12 months or less.6,7 This approach may accurately capture chronic conditions that result in frequent healthcare encounters, but may fail to identify previous acute events or lifestyle characteristics such as smoking, which can be important predictors of future events. The misclassification of prevalent cardiovascular disease at baseline has the potential to cause substantial bias in estimates of drug associations,8 the direction of which may be uncertain.2
To evaluate the accuracy of VA electronic health data for cardiovascular disease, we randomly sampled subjects from an ongoing cohort study of Veterans with treated diabetes, reviewed the full VA electronic medical record to identify prevalent conditions at baseline and acute MI events during follow up, and estimated the accuracy of various coding algorithms during three time periods: a 12-month baseline window, a 24-month baseline window, and during follow-up for MI events. Our hypothesis was that algorithms for acute cardiovascular conditions (MI, stroke) would have high PPV but low sensitivity, while algorithms for chronic conditions (angina, CHF) would have both high PPV and high sensitivity.
METHODS
Study population
The study population included 1,563 Veterans with type 2 diabetes who initiated insulin therapy from January 1 2004 to December 31 2009 while receiving regular care from the VA Puget Sound Health Care System, defined as having at least two primary care visits at a VA facility and filling at least two prescriptions at a VA pharmacy in the 12 month baseline period prior to the first insulin prescription. This study population is part of a larger national cohort study of Veterans initiating therapy for type 2 diabetes.9 180 subjects were randomly selected for chart review, including an oversampling of subjects who had inpatient and outpatient encounters with International Classification of Disease, Ninth Revision (ICD-9) codes for acute MI (410) and old MI (412) during follow up through December 31 2010. We estimated that this sample size would allow us to determine the sensitivity of the ICD-9 code for acute MI events with 95% confidence intervals (CIs) of ± 10%, assuming a sensitivity of approximately 50%. Information on health care encounters at VA and non-VA facilities and information on filled prescriptions from VA pharmacies was obtained from databases housed at the VA Austin Information Technology Center.
Coding algorithms and validation
Algorithms for prevalent conditions and smoking status used ICD-9 codes from inpatient and outpatient encounters during 12- and 24-month baseline periods (codes listed in Appendix table 1). The algorithm for coronary heart disease (CHD) included ICD-9 codes for previous MI, angina, and previous revascularization. Two algorithms for current smoking status were evaluated: one that included only ICD-9 codes, and one that included either ICD-9 codes or a filled prescription for nicotine replacement therapy or the smoking cessation drug varenicline during the baseline period. Acute MI events during follow up were identified from hospitalizations with an ICD-9 code for acute MI (410).
On a training set of 15 subjects, two physicians (JSF and MB) reviewed the full VA electronic medical record and achieved consensus on the validation of prevalent conditions at baseline and acute MI events during follow up. For the remaining subjects, the full VA electronic medical record was reviewed by one physician to validate prevalent conditions at baseline and acute MI events during follow up. The validation of prevalent conditions required physician documentation of the condition. Adapting criteria from previous studies,10,11 the validation of acute MI events required (1) physician documentation of acute MI, (2) presence of ischemic symptoms, and (3) troponin levels greater than 2x the upper limit of normal or an electrocardiogram with ST-segment elevation or new left bundle branch block. Validation criteria for each algorithm are listed in Appendix table 1.
Statistical analysis
Analyses were conducted using Stata, version 11.0. For each coding algorithm, we estimated the PPV, negative predictive value (NPV), sensitivity, and specificity, along with the 95% CIs for each of these measures, by using the “svy” command, which accounts for the stratified sampling.12 Incidence rates for acute MI events and 95% CIs were estimated using Poisson regression. Secondary analyses evaluated acute MI algorithms that excluded recurrent episodes of care (410.x2) and subjects with a previous MI during the baseline period. This study was approved by the VA Puget Sound Health Care System Institutional Review Board.
RESULTS
Out of 180 subjects sampled, access to the VA electronic medical record was restricted for 8 subjects, leaving 172 available for chart abstraction; 97% were male, 88% were white, and the mean age was 63 years (range 41–93).
Prevalent cardiovascular disease
Using a 12-month window for baseline data, algorithms for previous acute cardiovascular events had high PPV but low sensitivity (Table 1). For example, for previous MI the PPV was 97% (95% CI: 82–100%) but the sensitivity only 38% (95% CI: 19–60%). Of the 27 previous MI events missed by ICD-9 codes, 25 (93%) occurred prior to the 12-month baseline window, and the mean time before cohort entry for previous MI events was 10 years.
Table 1.
Validity of ICD-9 codes for prevalent health conditions.
| Condition | Validated events |
12-month window for ICD-9 codes | 24-month window for ICD-9 codes | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PPV (95% CI) |
NPV (95% CI) |
Sensitivity (95% CI) |
Specificity (95% CI) |
PPV (95% CI) |
NPV (95% CI) |
Sensitivity (95% CI) |
Specificity (95% CI) |
||
| Previous MI | 40 | 97% (82–100%) | 87% (79–92%) | 38% (19–60%) | 100% (98–100%) | 98% (85–100%) | 88% (80–93%) | 44% (25–66%) | 100% (98–100%) |
| Angina | 26 | 83% (36–98%) | 96% (90–98%) | 57% (29–81%) | 99% (91–100%) | 85% (44–98%) | 97% (93–99%) | 74% (46–90%) | 99% (92–100%) |
| Coronary heart disease | 88 | 85% (70–93%) | 89% (78–95%) | 84% (69–92%) | 90% (78–96%) | 85% (71–93%) | 93% (82–97%) | 90% (77–96%) | 89% (78–95%) |
| Congestive heart failure | 32 | 72% (47–86%) | 99% (91–100%) | 90% (57–98%) | 95% (88–98%) | 69% (46–86%) | 98% (91–100%) | 90% (57–98%) | 94% (88–98%) |
| Previous stroke | 26 | 81% (45–95%) | 93% (85–97%) | 52% (26–76%) | 98% (92–100%) | 77% (45–93%) | 93% (85–97%) | 52% (27–76%) | 98% (93–99%) |
| Current smoking (ICD-9 codes only) | 45 | 74% (52–88%) | 84% (73–91%) | 62% (43–78%) | 90% (80–96%) | 69% (51–83%) | 89% (79–95%) | 77% (59–89%) | 85% (74–92%) |
| Current smoking (ICD-9 codes + medication use) | 45 | 77% (57–90%) | 88% (78–94%) | 73% (54–86%) | 90% (80–96%) | 71% (54–84%) | 93% (83–97%) | 77% (59–89%) | 85% (74–92%) |
International Classification of Disease, Ninth Revision (ICD-9) codes from inpatient and outpatient encounters were used to identify prevalent health conditions. Specific ICD-9 codes and validation criteria are listed in Supplemental Table 1. CI = confidence interval, MI = myocardial infarction, NPV = negative predictive value, PPV = positive predictive value.
In contrast, algorithms for chronic conditions generally had high PPV and high sensitivity. For example, the PPV for CHF was 72% (95% CI: 47–86%) and the sensitivity was 90% (95% CI: 57–98%). For current smoking status, the algorithm that used both ICD-9 codes and pharmacy data had slightly higher PPV and sensitivity than the algorithm that used only ICD-9 codes.
For all baseline conditions, increasing the time window from 12 to 24 months resulted in modest gains in sensitivity but little impact on PPV (Table 1).
Acute MI events
Out of 46 subjects with an ICD-9 code for acute MI during follow up, 37 had a validated MI event (36 were hospitalized at a VA facility) and 9 did not. Among the 9 who did not have a validated MI event, 6 were hospitalized for other acute cardiac disorders (unstable angina, atypical chest pain, and CHF) and 2 had elective revascularization procedures. Among the 126 subjects who did not have ICD-9 codes for acute MI during follow up, 4 validated events occurred, all of which were hospitalized at non-VA facilities. Among the 41 total validated events, 34 (83%) were non-ST-segment elevation MIs, 7 (17%) were ST-segment elevation MIs, and 40 (98%) had elevated troponin levels.
During a mean follow up of 3.8 years, the incidence rate for acute MI events was 1.3% per year (95% CI: 1.2–1.5%) using ICD-9 codes and 1.2% per year (95% CI: 1.0–1.4%) using validated events. In primary analyses that included all acute MI codes, the PPV (80%; 95% CI, 66–90%) and the sensitivity (89%; 95% CI, 78–96%) were high (Table 2). Excluding 410.x2 codes and restricting to subjects with no previous MI had little impact on PPV and sensitivity.
Table 2.
Validity of ICD-9 codes for acute myocardial infarction events during follow up.
| Coding algorithm | Validated events | PPV (95% CI) | NPV (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) |
|---|---|---|---|---|---|
| All 410.xx codes | 41 | 80% (66–90%) | 99% (98–100%) | 90% (78–96%) | 99% (98–100%) |
| Exclude 410.x2 codes | 41 | 80% (66–89%) | 99% (98–100%) | 87% (75–94%) | 99% (98–100%) |
| Exclude subjects with previous MI at baseline | 40 | 80% (66–89%) | 99% (98–100%) | 91% (81–96%) | 99% (98–100%) |
Validation criteria for acute myocardial infarction are listed in Supplemental Table 1. CI = confidence interval, NPV = negative predictive value, PPV = positive predictive value.
DISCUSSION
In this validation study of electronic health data among regular users of the VA Puget Sound Health Care System, ICD-9 codes for acute MI during follow up had high PPV and missed few events. The sensitivity of ICD-9 codes for previous acute events at baseline was low, but chronic conditions such as CHF were identified with good accuracy. A composite variable for prevalent CHD at baseline, which incorporated ICD-9 codes for both acute events and chronic conditions, performed well.
The high PPV estimate for acute MI in our study agrees with previous literature estimates from another VA study (PPV 90%; 95% CI, 84–93%), from a Mini-Sentinel validation study (PPV 86%; 95% CI, 79–91%), and from the Women’s Health Initiative (WHI) (PPV 78%; 95% CI, 75–80%).3,4,13 The WHI also conducted active surveillance for acute cardiovascular events during follow up; the sensitivity of hospitalization diagnosis codes for all validated acute MI events was 80% (95% CI 77–83%), consistent with our results.13
The low sensitivity for previous acute events at baseline in our study appears to be explained by the fact that most of these events occurred prior to the 12-month baseline window. Increasing this time window to 24 months did not substantially improve the sensitivity. In contrast, algorithms for chronic cardiovascular conditions had both high PPV and high sensitivity, consistent with previous findings. For example, in a validation study of outpatient-only VA data and prevalent health conditions, the sensitivity and specificity of ICD-9 codes for CHD were 87% and 92% respectively.14 Our algorithm for current smoking status at baseline had a higher sensitivity (73%) than what has been reported previously (range: 32–56%).4,14,15 This may reflect the restriction of our study population to Veterans who received regular primary care from VA facilities, and thus were more likely to have smoking status documented and coded from health care encounters.
There are often tradeoffs between different measures of accuracy for diagnostic algorithms that use electronic health data, and researchers may prioritize them differently depending on the goals of the study.2 For example, sensitivity may be prioritized when the complete identification of a condition is important, such as in a surveillance study of drug adverse events that can later be confirmed with additional information.16 On the other hand, high PPV may be prioritized when events must be identified efficiently but without the opportunity for validation. These tradeoffs may be less important when diagnostic algorithms capture nearly all events with good accuracy, as was the case in our study for acute MI events during follow up.
Our study had limitations. Results from this study may not generalize to Veterans who do not have diabetes, who do not receive regular care from the VA healthcare system, or who live in geographic regions where acute MI events are usually hospitalized at VA facilities. The estimate for the sensitivity of the coding algorithm for acute MI events could be influenced, in either direction, by random error in the sampling of subjects without acute MI codes who had genuine events during follow up. Strengths of this study include the detailed reviews of the entire VA electronic medical record to identify events missed by ICD-9 codes and the use of case definitions and quality control measures to improve the reliability of the medical record reviews.
In conclusion, this study suggests that electronic health data can accurately identify new acute MI events among regular users of the VA healthcare system. Prevalent cardiovascular disease can also be identified with moderate accuracy, although previous acute events are often missed using typical periods of baseline data collection. Studies that use these data should acknowledge these limitations and the potential for misclassification and bias.
Supplementary Material
KEY POINTS.
Previous validation studies of administrative data have found that ICD-9 codes for acute cardiovascular events have high positive predictive value (PPV), but the sensitivity of these codes for previous events and for new events during follow up is largely unknown.
Among a cohort of Veterans with diabetes who received regular care from the VA Puget Sound Health Care System, ICD-9 codes for previous acute MI and previous acute stroke events had high PPV (97% and 81%) but low sensitivity (38% and 52%)
ICD-9 codes for coronary heart disease, congestive heart failure, and smoking status at baseline had high PPV and sensitivity.
ICD-9 codes for acute MI events during follow up also had high PPV (80%) and sensitivity (90%).
VA electronic health data can be used to identify cardiovascular disease with moderate to good accuracy, but the limitations of these data should be acknowledged.
Acknowledgments
FUNDING
Dr. Floyd was supported by grant K08HL116640 from the National, Heart, Lung, and Blood Institute. The VA Puget Sound Health Care System provided support for Drs. Moore, Boyko, and Smith in this research.
Footnotes
CONFLICTS OF INTEREST
The authors have no conflicts of interest to report.
REFERENCES CITED
- 1.Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58(4):323–337. doi: 10.1016/j.jclinepi.2004.10.012. [DOI] [PubMed] [Google Scholar]
- 2.Chubak J, Pocobelli G, Weiss NS. Tradeoffs between accuracy measures for electronic health care data algorithms. J Clin Epidemiol. 2012;65(3):343–349. e342. doi: 10.1016/j.jclinepi.2011.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Cutrona SL, Toh S, Iyer A, et al. Validation of acute myocardial infarction in the Food and Drug Administration’s Mini-Sentinel program. Pharmacoepidemiol Drug Saf. 2013;22(1):40–54. doi: 10.1002/pds.3310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Niesner K, Murff HJ, Griffin MR, et al. Validation of VA administrative data algorithms for identifying cardiovascular disease hospitalization. Epidemiology. 2013;24(2):334–335. doi: 10.1097/EDE.0b013e3182821e75. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Niu B, Forde KA, Goldberg DS. Coding algorithms for identifying patients with cirrhosis and hepatitis B or C virus using administrative data. Pharmacoepidemiol Drug Saf. 2015;24(1):107–111. doi: 10.1002/pds.3721. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Graham DJ, Ouellet-Hellstrom R, MaCurdy TE, et al. Risk of acute myocardial infarction, stroke, heart failure, and death in elderly Medicare patients treated with rosiglitazone or pioglitazone. JAMA. 2010;304(4):411–418. doi: 10.1001/jama.2010.920. [DOI] [PubMed] [Google Scholar]
- 7.Roumie CL, Hung AM, Greevy RA, et al. Comparative effectiveness of sulfonylurea and metformin monotherapy on cardiovascular events in type 2 diabetes mellitus: a cohort study. Ann Intern Med. 2012;157(9):601–610. doi: 10.7326/0003-4819-157-9-201211060-00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Psaty BM, Koepsell TD, Siscovick D, et al. An approach to several problems in using large databases for population-based case-control studies of the therapeutic efficacy and safety of anti-hypertensive medicines. Stat Med. 1991;10(4):653–662. doi: 10.1002/sim.4780100416. [DOI] [PubMed] [Google Scholar]
- 9.Wheeler S, Moore K, Forsberg CW, et al. Mortality among veterans with type 2 diabetes initiating metformin, sulfonylurea or rosiglitazone monotherapy. Diabetologia. 2013;56(9):1934–1943. doi: 10.1007/s00125-013-2958-1. [DOI] [PubMed] [Google Scholar]
- 10.Ives DG, Fitzpatrick AL, Bild DE, et al. Surveillance and ascertainment of cardiovascular events. The Cardiovascular Health Study. Ann Epidemiol. 1995;5(4):278–285. doi: 10.1016/1047-2797(94)00093-9. [DOI] [PubMed] [Google Scholar]
- 11.Psaty BM, Heckbert SR, Koepsell TD, et al. The risk of myocardial infarction associated with antihypertensive drug therapies. JAMA : the journal of the American Medical Association. 1995;274(8):620–625. [PubMed] [Google Scholar]
- 12.West BT, Berglund P, Heeringa SG. A closer examination of subpopulation analysis of complex-sample survey data. The Stata Journal. 2008;8(4):520–531. [Google Scholar]
- 13.Heckbert SR, Kooperberg C, Safford MM, et al. Comparison of self-report, hospital discharge codes, and adjudication of cardiovascular events in the Women’s Health Initiative. Am J Epidemiol. 2004;160(12):1152–1158. doi: 10.1093/aje/kwh314. [DOI] [PubMed] [Google Scholar]
- 14.Borzecki AM, Wong AT, Hickey EC, Ash AS, Berlowitz DR. Identifying hypertension-related comorbidities from administrative data: what’s the optimal approach? Am J Med Qual. 2004;19(5):201–206. doi: 10.1177/106286060401900504. [DOI] [PubMed] [Google Scholar]
- 15.Wiley LK, Shah A, Xu H, Bush WS. ICD-9 tobacco use codes are effective identifiers of smoking status. Journal of the American Medical Informatics Association : JAMIA. 2013;20(4):652–658. doi: 10.1136/amiajnl-2012-001557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Floyd JS, Heckbert SR, Weiss NS, Carrell DS, Psaty BM. Use of administrative data to estimate the incidence of statin-related rhabdomyolysis. JAMA : the journal of the American Medical Association. 2012;307(15):1580–1582. doi: 10.1001/jama.2012.489. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.