Visual Abstract
Keywords: cardiovascular disease, cardiovascular events, clinical epidemiology, coronary artery disease, epidemiology and outcomes, heart failure, hospitalization, epidemiologic methods
Abstract
Key Points
International Classification of Diseases (ICD) codes had high positive predictive value compared with physician adjudication for cardiovascular events in patients with kidney disease.
Sensitivities of ICD codes for cardiovascular events were lower compared with physician adjudication.
ICD code usage provides opportunities for studies such as large epidemiologic studies and clinical trials, but has some limitations.
Background
The risk of cardiovascular disease is higher in individuals with CKD, and cardiovascular disease events are common and important end points for research studies in CKD. Adjudication by a central committee is considered the most rigorous approach of ascertaining cardiovascular disease outcomes; however, it is resource intensive. There are limited data to determine the accuracy of International Classification of Diseases (ICD) code–ascertained outcomes compared with physician adjudication for cardiovascular disease events in CKD and kidney failure.
Methods
Using data from the Chronic Renal Insufficiency Cohort (CRIC), we evaluated hospitalization events in participants with CKD and kidney failure to determine the accuracy of ICD-9 and 10 codes compared with physician adjudication of the cardiovascular disease outcomes: heart failure, myocardial infarction, stroke, and atrial fibrillation. For ICD codes, we determined the positive predictive value (PPV), negative predictive value, sensitivity, and specificity for each cardiovascular disease outcome on the basis of primary and secondary diagnosis codes. Association of known cardiovascular disease risk factors with incident cardiovascular disease outcomes was determined for ICD codes versus physician adjudication.
Results
Comparing primary ICD-9 or 10 discharge codes with physician adjudication for 3464 participants, we found PPVs of 79% for heart failure, 77% for myocardial infarction, 77% for ischemic stroke, and 85% for atrial fibrillation for individuals with CKD and kidney failure. Negative predictive values ranged from 94% to 99%. Specificities were high at 99%–100%. Sensitivities were much lower at 15%–48%. The associations between cardiovascular disease risk factors and comorbidities (including age, diabetes, eGFR) were similar for ICD code–identified and physician adjudication–identified events, with r values ranging from 0.82 to 0.98.
Conclusions
PPV was near 80% for heart failure, myocardial infarction, stroke, and atrial fibrillation for primary ICD codes versus physician adjudication; however, sensitivity was lower. ICD code usage in medical research may allow greater efficiency with limited resources for outcome ascertainment.
Introduction
Cardiovascular disease and cardiovascular disease events in persons with CKD and kidney failure are important end points for observational studies and clinical trials because of their high prevalence, morbidity, and mortality.1,2 Ascertaining cardiovascular disease outcomes through adjudication by a central or physician committee is considered the most rigorous approach, but is resource intensive. International Classification of Diseases (ICD) codes provide a standardized system for medical coding of diseases for billing and clinical documentation. ICD code usage in medical research provides opportunities for pragmatic studies, including epidemiologic research and clinical trials, greater efficiency with limited resources, and expanded participant populations.
However, accuracy of clinically used ICD codes for research outcome identification requires an understanding of their strengths and limitations in comparison with gold-standard adjudication. CKD may pose unique challenges for accurate diagnosis of cardiovascular disease by ICD codes, especially in heart failure and myocardial infarction, where CKD can affect biomarker levels that are frequently used to make the diagnoses (e.g., troponin for myocardial infarction and brain natriuretic peptide in heart failure). Furthermore, CKD and cardiovascular disease may have overlapping clinical symptoms, which contribute to this challenge. Using ICD codes for trial outcomes could streamline outcome ascertainment, but there are limited data examining the accuracy of cardiovascular disease ICD-ascertained versus physician-adjudicated outcomes in CKD and kidney failure populations to help guide researchers. The aim of our study was to compare the accuracy of ICD code–ascertained versus physician adjudication for outcome identification in hospitalizations for heart failure, myocardial infarction, stroke, and atrial fibrillation in the Chronic Renal Insufficiency Cohort (CRIC).
Methods
Study Population
This analysis used National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) public repository data for the CRIC study, a multicenter prospective cohort study of adults with CKD recruited from seven clinical centers (with 13 enrolling sites) through years 2003 to 2023.3,4 CRIC was established to examine risk factors for the progression of CKD and the development and worsening of cardiovascular disease in patients with CKD.3 Adults with CKD age 21–74 years were eligible to participate if they met the following age‐specific eGFR criteria: 20–70 ml/min per 1.73 m2 for age 21–44 years, 20–60 ml/min per 1.73 m2 for age 45–64 years, and 20–50 ml/min per 1.73 m2 for age 65–74 years using the four-variable Modification of Diet in Renal Disease equation. The study followed participants prospectively and included participants who progressed to kidney failure requiring dialysis or kidney transplant. Exclusion criteria included heart failure with New York Heart Association functional class 3 or 4, polycystic kidney disease, cirrhosis, renal cancer, recent chemotherapy, dialysis within the past month, or history of organ transplant at study recruitment. Participants with a history of heart failure (as long as not New York Heart Association class 3 or 4 at study recruitment), myocardial infarction, stroke, or atrial fibrillation were included in all analyses.
Participants had annual in-person visits and 6-month phone contacts. At the annual in-person visits, participants completed questionnaires updating interim medical events and medication use and underwent laboratory testing. Institutional review board approval was required from all participating institutions, and informed consent to participate in CRIC was obtained from all participants for all parts of the study: screening, baseline testing, subsequent follow-up visits, and hospital record review.5
Physician-Adjudicated Cardiovascular Hospitalizations
Study participants were interviewed every 6 months during alternating in-person and telephone visits about hospitalizations, experiencing a possible cardiovascular disease event, or undergoing certain diagnostic tests or procedures. Participants could have recurrent hospitalizations, and all cardiovascular disease events were categorized by their CKD or kidney failure status at the time of each specific hospitalization. On the basis of participant report, from 2003 to 2013, all hospital records with ICD codes related to CRIC cardiovascular disease outcomes were retrieved for review. After 2013, all records, regardless of hospitalization reason, were retrieved. For every hospitalization, ICD-9 and ICD-10 codes were collected. Of note, the number of ICD codes collected varied by year: Between 2003 and mid-2007, the first five codes were collected. This increased to the first 30 codes in mid-2007 and then to 50 codes beginning in early 2015.
Outcomes for heart failure, myocardial infarction, atrial fibrillation, and stroke were all adjudicated by two independent CRIC study physicians. Heart failure events were determined by Framingham and Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial criteria on the basis of clinical symptoms, radiographic studies, physical examination records, and central venous hemodynamic monitoring data and echocardiographic imaging when available.6,7
Diagnosis of adjudicated probable or definite myocardial infarction was made on the basis of symptoms consistent with acute ischemia, cardiac biomarker levels, and electrocardiograms as recommended by a consensus statement on the universal definition of myocardial infarction.8
Outcomes of probable and definite ischemic stroke were made after review by two neurologists of all hospitalizations and emergency department visits suggestive of stroke. Diagnosis of definite stroke was determined on the basis of autopsy findings or sudden onset of neurologic symptoms supported with computed tomography or magnetic resonance imaging demonstration of infarction in a territory where an injury or infarction would be expected to create those symptoms. Diagnosis of probable stroke was defined as sudden or rapid onset of one major or two minor neurologic signs or symptoms within a cerebrovascular distribution lasting >24 hours or until the patient died, without imaging evidence within 24 hours, but with no alternative explanation.9
Atrial fibrillation and other arrhythmias underwent centralized adjudicated review by at least two study physicians with manual review of relevant medical records and electrocardiograms if available using standardized criteria. Outcomes for all event types were confirmed when both reviewers agreed on a probable or definite occurrence.3,10
ICD-Identified Outcomes
Heart failure, myocardial infarction, stroke, and atrial fibrillation events for this study were determined by ICD-9 or ICD-10 codes on the basis of previous literature (Supplemental Table 1).11–13 We evaluated ICD codes in two ways: using primary ICD code for the hospitalization or using primary code plus secondary diagnosis code positions. We primarily focused on primary ICD codes to focus on major events. Between 2003 and mid-2007, the first five codes were collected. This increased to the first 30 codes in mid-2007 and then to 50 codes beginning in early 2015. During our study period of 2003–2023, there was a mix of hospitalizations using ICD-9 and ICD-10 events, with the transition happening nationally in 2015. We performed a secondary analysis in which we examined the accuracy of ICD-9 and ICD-10 codes separately.
Covariates
Systolic and diastolic BP, heart rate, height, weight, and urine/blood specimens were collected during annual in-person visits. Demographic information and medical history were obtained by self-report. Self-reported history of cardiovascular disease included myocardial infarction or revascularization, heart failure, and stroke or peripheral vascular disease. eGFR for analysis was calculated from serum creatinine using the 2021 CKD Epidemiology Collaboration equation without race coefficient, with serum creatinine measured using an enzymatic method on an Ortho Vitros 950 at the CRIC central laboratory and standardized to isotope dilution mass spectrometry–traceable values. Diabetes mellitus was defined as a fasting glucose >126 mg/dl, a nonfasting glucose >200 mg/dl, or use of insulin or other diabetes medication. Hypertension was defined as systolic BP ≥140 mm Hg and/or diastolic BP ≥90 mm Hg and/or self-reported use of antihypertensive therapy. BP was taken using a standardized protocol with three measurements at a study visit after 5 minutes rest.14
Statistical Analysis
We determined the positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity of ICD codes to detect the gold standard of physician-adjudicated diagnosis for each cardiovascular disease outcome. We also calculated a Cohen kappa coefficient with bootstrapped confidence intervals (CIs) at the level of the individual to account for correlation within person for each cardiovascular disease outcome and assess for agreement between the two event identification types. We evaluated ICD codes in two ways: using primary ICD code for the hospitalization or using primary code plus all secondary diagnosis code positions available during the study year. We evaluated these operating characteristics for all participants and stratified by CKD or kidney failure (defined as receipt of dialysis or kidney transplant) status at the time of hospitalization. Given the transition from ICD-9 to ICD-10 codes during the study period, we also compared PPVs for hospitalizations that used ICD-9 versus ICD-10 codes for physician-adjudicated outcomes. Finally, to test whether the association of known cardiovascular disease risk factors (age, sex, race/ethnicity, current smoking, diabetes, history of cardiovascular disease, body mass index, systolic BP, LDL cholesterol, and eGFR, all evaluated at the time of hospitalization) with incident cardiovascular disease outcomes differed when determined by each type of ascertainment, ICD codes versus physician-adjudicated, we used Cox regression with Wei et al. SEMs to estimate the association.15 Pearson correlation coefficients were used to assess the strength of linear relationship between hazard ratios (HRs) when using ICD code–based and physician-adjudicated–based outcome definitions. A two-sided P value < 0.05 was considered statistically significant. All analyses were conducted with R version 4.2.3 (R Foundation for Statistical Computing, Vienna, Austria).
Results
Study Participants
Using CRIC repository data with follow-up through January 2023, 3464 participants of 3939 total CRIC participants had at least one hospitalization. We evaluated 33,082 hospitalization events. The mean age at first hospitalization was 60 (SD 11), and 55% were male. The median follow-up time was 9.7 (interquartile range, 5.5–13.3) years, and the median time from cohort entry to first hospitalization was 1.6 (interquartile range, 0.6–3.6) years. The mean eGFR (SD) at first hospitalization was 41 (17) ml/min per 1.73 m2. Four percent (n=133) of participants had progressed to kidney failure by the time of their first hospitalization. Twenty-four percent of individuals had a history of myocardial infarction or coronary artery revascularization. History of atrial fibrillation was reported for 18% of participants, 10% reported a history of heart failure, and 10% reported a history of stroke. Medication usage was fairly high with 70% of participants taking angiotensin-converting enzyme inhibitors or angiotensin receptor blockers and 52% taking β blockers at the time of first hospitalization (Table 1).
Table 1.
Characteristics of participants with CKD at first cardiovascular disease hospitalization (N=3464)
| Characteristic | All Participants (N=3464) | Participants with CKD at First Hospitalization (n=3331) | Participants with Kidney Failure at First Hospitalization (n=133) |
|---|---|---|---|
| Time from cohort entry to first hospitalization, yr, median (IQR) | 1.6 (0.6–3.6) | 1.5 (0.6–3.5) | 3.8 (1.9–7.1) |
| Age, yr | 60 (11) | 60 (11) | 56 (13) |
| Male, No. (%) | 1896 (55) | 1808 (54) | 88 (66) |
| Race/ethnicity, No. (%) | |||
| Hispanic | 412 (12) | 373 (11) | 39 (29) |
| Non-Hispanic Black | 1461 (42) | 1415 (42) | 46 (35) |
| Non-Hispanic White | 1462 (42) | 1426 (43) | 36 (27) |
| BMI, kg/m2 | 32.4 (8.0) | 32.5 (8.1) | 30.3 (7.3) |
| Diabetes, No. (%) | 1797 (52) | 1728 (52) | 69 (52) |
| Hypertension, No. (%) | 3137 (91) | 3008 (90) | 129 (97) |
| Current smoking, No. (%) | 440 (13) | 426 (13) | 14 (11) |
| Systolic BP, mm Hg | 129 (23) | 129 (22) | 139 (23) |
| LDL cholesterol, mg/dl | 101 (35) | 101 (35) | 108 (44) |
| Self-reported heart failure, No. (%) | 340 (10) | 326 (10) | 14 (11) |
| Self-reported heart failure or EF <50%, No. (%) | 782 (23) | 726 (22) | 56 (42) |
| Cardiovascular disease, No. (%) | |||
| History MI or revascularization | 823 (24) | 801 (24) | 22 (17) |
| History of stroke | 356 (10) | 345 (10) | 11 (8) |
| History of peripheral vascular disease | 244 (7) | 232 (7) | 12 (9) |
| History of atrial fibrillation | 633 (18) | 616 (18) | 17 (13) |
| eGFR, ml/min per 1.73 m 2 , No. (%) | 41 (17) | 42 (16) | 16 (10) |
| ≥60 | 476 (14) | 475 (14) | 1 (1) |
| 45–59 | 892 (26) | 891 (27) | 1 (1) |
| 30–44 | 1169 (34) | 1162 (35) | 7 (5) |
| 15–29 | 779 (22) | 726 (22) | 53 (40) |
| <15 | 148 (4) | 77 (2) | 71 (53) |
| 24-h protein, g, median (IQR) | 0.2 (0.1–1.1) | 0.2 (0.1–0.9) | 3.0 (1.1–5.2) |
| <1 | 2487 (72) | 2459 (74) | 28 (21) |
| 1–3 | 465 (13) | 433 (13) | 32 (24) |
| >3 | 409 (12) | 346 (10) | 63 (47) |
| Missing | 103 (3) | 93 (3) | 10 (8) |
| Medications, No. (%) | |||
| ACEi/ARB | 2434 (70) | 2359 (71) | 75 (56) |
| β blockers | 1789 (52) | 1716 (52) | 73 (55) |
| Diuretics | 2094 (60) | 2017 (61) | 77 (58) |
Entries are mean (SD) for continuous variables and No. (%) for categorical variables, except as noted. ACEi, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; BMI, body mass index; EF, ejection fraction; IQR, interquartile range; MI, myocardial infarction.
Heart Failure Events
There were 1507 heart failure hospitalizations identified by primary ICD codes and 2477 definite or probable events by physician adjudication. Examination of primary diagnosis codes using ICD-9 or 10 codes compared with physician adjudication had a PPV of 79% (95% CI, 77% to 81%), NPV of 96% (95% CI, 96% to 96%), sensitivity of 48% (95% CI, 46% to 50%), and specificity of 99% (95% CI, 99% to 99%). PPV and sensitivity (75% [95% CI, 71% to 83%) and 34% [95% CI, 31% to 37%], respectively) were slightly lower for events occurring in participants with kidney failure (Table 2 and Supplemental Table 2A). With the inclusion of secondary diagnosis codes in addition to primary, there was a decrease in PPV by over half to 33% (Figure 1). The sensitivity and specificity were 91% and 85%, respectively, when using all available secondary ICD codes (Figure 2).
Table 2.
Test performance of primary International Classification of Diseases code–based versus physician-adjudicated cardiovascular disease outcomes in CKD
| Event Type | Physician-Adjudicated Events (No.) | ICD Code–Based Events (No.) | PPV (95% CI) | NPV (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) |
|---|---|---|---|---|---|---|
| Heart failure | ||||||
| All events | 2477 | 1507 | 79% (77% to 81%) | 96% (96% to 96%) | 48% (46% to 50%) | 99% (99% to 99%) |
| CKD participant events | 1664 | 1140 | 80% (78% to 83%) | 97% (96% to 97%) | 55% (53% to 57%) | 99% (99% to 99%) |
| Kidney failure participant events | 813 | 367 | 75% (71% to 80%) | 95% (94% to 95%) | 34% (31% to 37%) | 99% (99% to 99%) |
| Myocardial infarction | ||||||
| All events | 640 | 396 | 77% (73% to 81%) | 99% (99% to 99%) | 48% (44% to 52%) | 100% (100% to 100%) |
| CKD participant events | 405 | 274 | 75% (70% to 80%) | 99% (99% to 99%) | 51% (46% to 55%) | 100% (100% to 100%) |
| Kidney failure participant events | 235 | 122 | 82% (75% to 89%) | 99% (98% to 99%) | 43% (36% to 49%) | 100% (100% to 100%) |
| Stroke | ||||||
| All events | 338 | 209 | 77% (71% to 82%) | 99% (99% to 100%) | 47% (42% to 53%) | 100% (100% to 100%) |
| CKD participant events | 251 | 166 | 73% (66% to 80%) | 99% (99% to 100%) | 48% (42% to 54%) | 100% (100% to 100%) |
| Kidney failure participant events | 87 | 43 | 91% (82% to 99%) | 100% (99% to 100%) | 45% (34% to 55%) | 100% (100% to 100%) |
| Atrial fibrillation | ||||||
| All events | 2239 | 385 | 85% (82% to 89%) | 94% (94% to 94%) | 15% (13% to 16%) | 100% (100% to 100%) |
| CKD participant events | 1448 | 269 | 83% (79% to 88%) | 95% (94% to 95%) | 15% (14% to 17%) | 100% (100% to 100%) |
| Kidney failure participant events | 791 | 116 | 85% (79% to 92%) | 93% (93% to 94%) | 13% (10% to 15%) | 100% (100% to 100%) |
CI, confidence interval; ICD, International Classification of Diseases; NPV, negative predictive value; PPV, positive predictive value.
Figure 1.
PPV and NPV of code-adjudicated outcomes by number of primary and secondary ICD codes used. ICD, International Classification of Diseases; NPV, negative predictive value; PPV, positive predictive value.
Figure 2.
Sensitivity and specificity of code-adjudicated outcomes by number of primary and secondary ICD codes used.
A majority of hospitalizations used ICD-9 codes, with a total of 1230 ICD-9–documented heart failure events and 277 ICD-10–documented heart failure events. PPVs for participants with CKD were 84% (95% CI, 82% to 86%) for ICD-9 and 60% (95% CI, 53% to 68%) for ICD-10. For participants with kidney failure, the ICD-9 PPV was 77% (95% CI, 72% to 82%) compared with 71% (95% CI, 62% to 80%) for ICD-10 (Table 3).
Table 3.
Summary of code-based (first position only) versus adjudicated heart failure, myocardial infarction, stroke, and atrial fibrillation in Chronic Renal Insufficiency Cohort, comparing International Classification of Diseases-9 versus International Classification of Diseases-10 code–based events
| Code-Based Heart Failure | Total Code-Based Events | Definite+Probable Adjudicated Heart Failure | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|
| Total | Yes | No | |||
| All events | |||||
| ICD-9 events | 1230 | 1014 | 216 | 82% (80% to 85%) | 96% (96% to 96%) |
| ICD-10 events | 277 | 178 | 99 | 64% (59% to 70%) | 96% (96% to 97%) |
| CKD participant events | |||||
| ICD-9 events | 963 | 808 | 155 | 84% (82% to 86%) | 96% (96% to 97%) |
| ICD-10 events | 177 | 107 | 70 | 60% (53% to 68%) | 98% (98% to 98%) |
| Kidney failure participant events | |||||
| ICD-9 events | 267 | 206 | 61 | 77% (72% to 82%) | 95% (94% to 95%) |
| ICD-10 events | 100 | 71 | 29 | 71% (62% to 80%) | 93% (92% to 94%) |
| Code-Based Myocardial Infarction | Total Code-Based Events | Definite+Probable Adjudicated Myocardial Infarction | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|
| Total | Yes | No | |||
| All events | |||||
| ICD-9 events | 312 | 256 | 56 | 82% (78% to 86%) | 99% (99% to 99%) |
| ICD-10 events | 84 | 49 | 35 | 58% (48% to 69%) | 99% (99% to 99%) |
| CKD participant events | |||||
| ICD-9 events | 221 | 174 | 47 | 79% (73% to 84%) | 99% (99% to 99%) |
| ICD-10 events | 53 | 31 | 22 | 58% (45% to 72%) | 99% (99% to 100%) |
| Kidney failure participant events | |||||
| ICD-9 events | 91 | 82 | 9 | 90% (84% to 96%) | 99% (98% to 99%) |
| ICD-10 events | 31 | 18 | 13 | 58% (41% to 75%) | 98% (97% to 99%) |
| Code-Based Stroke | Total Code-Based Events | Definite+Probable Adjudicated Stroke | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|
| Total | Yes | No | |||
| All events | |||||
| ICD-9 events | 159 | 133 | 26 | 84% (78% to 89%) | 100% (99% to 100%) |
| ICD-10 events | 50 | 27 | 23 | 54% (40% to 68%) | 100% (99% to 100%) |
| CKD participant events | |||||
| ICD-9 events | 128 | 104 | 24 | 81% (74% to 88%) | 99% (99% to 100%) |
| ICD-10 events | 38 | 17 | 21 | 45% (29% to 61%) | 100% (99% to 100%) |
| Kidney failure participant events | |||||
| ICD-9 events | 31 | 29 | 2 | 94% (85% to 100%) | 100% (99% to 100%) |
| ICD-10 events | 12 | 10 | 2 | 83% (62% to 100%) | 100% (99% to 100%) |
| Code-Based Atrial Fibrillation | Total Code-Based Events | Definite+Probable Adjudicated Atrial Fibrillation | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|
| Total | Yes | No | |||
| All events | |||||
| ICD-9 events | 310 | 279 | 31 | 90% (87% to 93%) | 94% (94% to 94%) |
| ICD-10 events | 75 | 49 | 26 | 65% (55% to 76%) | 94% (94% to 95%) |
| CKD events | |||||
| ICD-9 events | 211 | 187 | 24 | 89% (84% to 93%) | 94% (94% to 95%) |
| ICD-10 events | 58 | 37 | 21 | 64% (51% to 76%) | 95% (95% to 96%) |
| Kidney failure participant events | |||||
| ICD-9 events | 99 | 92 | 7 | 93% (88% to 98%) | 93% (92% to 94%) |
| ICD-10 events | 17 | 12 | 5 | 71% (49% to 92%) | 92% (91% to 93%) |
CI, confidence interval; ICD, International Classification of Diseases; NPV, negative predictive value; PPV, positive predictive value.
Myocardial Infarction Events
There were 396 primary ICD-coded myocardial infarction events and 640 physician-adjudicated definite or probable events. ICD codes provided a PPV of 77% (95% CI, 73% to 81%), NPV of 99% (95% CI, 99% to 99%), sensitivity of 48% (95% CI, 44% to 52%), and specificity of 100% (95% CI, 100% to 100%) (Table 2 and Supplemental Table 2B). Similar to heart failure, the PPV for myocardial infarction was higher for ICD-9 codes compared with ICD-10 (82% [95% CI, 78% to 86%] versus 58% [95% CI, 48% to 69%) (Table 3). PPV decreased to 50% with the addition of all available secondary ICD codes (Figure 1).
Stroke Events
With a total of 209 total primary ICD code–identified and 338 physician-adjudicated definite or probable stroke events, we found a PPV of 77% (95% CI, 73% to 81%), NPV of 99% (95% CI, 99% to 99%), sensitivity of 47% (95% CI, 42% to 53%), and specificity of 100% (95% CI, 100% to 100%). PPV was higher for hospitalizations among participants with kidney failure at 91% (95% CI, 82% to 99%) compared with 73% (95% CI, 66% to 80%) for participants with CKD (Table 2 and Supplemental Table 2C). Sensitivity remained low at 61% even when using both primary and secondary diagnosis codes (Figure 2).
Atrial Fibrillation Events
There were a total of 385 atrial fibrillation primary ICD code–identified events and 2239 physician-adjudicated definite or probable atrial fibrillation events. Atrial fibrillation primary ICD code–identified event sensitivity was only 15% (95% CI, 13% to 16%). PPV was 85% (95% CI, 82% to 89%), NPV 94% (95% CI, 94% to 94%), and specificity 100% (95% CI, 100% to 100%) (Table 2). PPV was lower for ICD-10 codes (65% [95% CI, 55% to 76%]) compared with ICD-9 codes (90% [95% CI, 87% to 93%]) (Table 3).
Correspondence of ICD Code–Based and Adjudicated Events
Assessing agreement between code-based and adjudicated events, Cohen kappa coefficient with bootstrapped CIs were 0.57 (95% CI, 0.55 to 0.60) for heart failure, 0.58 (95% CI, 0.54 to 0.62) for myocardial infarction, 0.59 (95% CI, 0.54 to 0.64) for stroke, and 0.23 (95% CI, 0.21 to 0.26) for atrial fibrillation when accounting for correlation within person for each cardiovascular disease outcome. The HRs of cardiovascular disease risk factors including older age, smoking, body mass index, and eGFR were similar for ICD code–ascertained and physician-adjudicated events for all four cardiovascular disease outcomes. Although atrial fibrillation had large differences in HR for Hispanic ethnicity and history of cardiovascular disease, the directionality was the same, and CIs were wide and overlapping (Supplemental Table 3). Pearson correlation coefficients between ICD code and adjudicated event unadjusted HRs for 10 different cardiovascular disease risk factors were r=0.98 for heart failure, r=0.83 for myocardial infarction, r=0.97 for stroke, and r=0.82 for atrial fibrillation (Figure 3).
Figure 3.
Correspondence of unadjusted associations of risk factors with code-based and adjudication-based events. Variables used: age (per 10-year increment), sex, race/ethnicity, smoking, diabetes, history of CVD, BMI (per 5 kg/m2 increment), SBP (per 10 mm Hg increment), LDL (per 20 mg/dl decrement), eGFR (per 15 ml/min per 1.73 m2 decrement). AF, atrial fibrillation; BMI, body mass index; CI, confidence interval; CVD, cardiovascular disease; HF, heart failure; HR, hazard ratio; Hx, history; MI, myocardial infarction; SBP, systolic BP.
Discussion
In a large prospective research cohort of participants with CKD and kidney failure, we evaluated cardiovascular disease outcome ascertainment by hospital ICD codes versus physician adjudication for four important cardiovascular disease outcomes: heart failure, myocardial infarction, stroke, and atrial fibrillation. When using primary ICD-9 or 10 codes to determine cardiovascular disease outcomes, PPV was near 80% for heart failure, myocardial infarction, stroke, and atrial fibrillation. PPVs were lower for ICD-10 codes compared with ICD-9 codes, and this was true for both participants with CKD and kidney failure. Sensitivity was much lower (15%–48%) than PPV, indicating that using ICD codes may lead to significant missed identification of cases. The associations between established cardiovascular disease risk factors and comorbidities (including age, diabetes, eGFR) and cardiovascular disease outcomes were similar for cardiovascular disease events identified by ICD codes and those that were identified by physician adjudication, suggesting that in some studies, use of ICD codes may be comparable with physician adjudication. These data may inform the approach to cardiovascular disease outcome ascertainment in future studies of patients with kidney disease, requiring a study design to balance low sensitivities with high PPV with ICD code use.
Of the four cardiovascular disease outcomes in this study, heart failure identification using ICD codes is the only outcome previously studied specifically in patients with CKD. A previous study using a population-based cohort of individuals with CKD (mean eGFR 49 ml/min per 1.73 m2) in a single health system found a PPV for heart failure primary diagnosis code of 92% for hospitalizations.16 PPVs near 90% were also described in two meta-analyses and qualitative reviews of heart failure in general populations in North American databases.17,18 We found a heart failure primary code PPV of 79%, slightly lower than these published reports. We did see a lower heart failure PPV for participants with kidney failure (75%) compared with CKD (80%), indicating that participants with kidney failure were more likely to be incorrectly coded for heart failure in the primary code position than individuals with CKD not on dialysis. Reasons for our lower observed PPV may include having a wider range of kidney function in our study population, including those with kidney failure. Given the known added challenges of heart failure diagnosis in CKD, our study provides important data on the accuracy of ICD codes for heart failure for a range of persons with CKD, including those with kidney failure.
For myocardial infarction, we found a PPV of 77%. This is similar to ICD-9 codes PPV from a study using the Atherosclerosis Risk in Communities (ARIC) cohort that had a 70%–80% PPV for definite or probable acute myocardial infarction.19 Two other previous studies used ICD-10 codes to examine PPV in myocardial infarction, one in Western Australia with PPV for ST-elevation myocardial infarction of 78% and non–ST-elevation myocardial infarction of 91% and one in a single center in Japan with PPV of 82.5%.20,21 Although our overall PPV is fairly similar to these studies, when using ICD-10–only events, our observed PPV was much lower at 58%. It is worthwhile to note that these ICD-10 codes were adopted in 2015, and in 2017, the first highly sensitive cardiac troponin assay was approved by the US Food and Drug Administration.22 Because CKD is associated with increased troponin levels, which are used in diagnosing myocardial infarction, it reasonable to expect a CKD population to have unique diagnostic challenges, potentially leading to the lower PPVs we observed.
For stroke, we found a PPV of 77%, similar to an analysis in the ARIC cohort that reported a PPV of 81% for primary code position and 75% primary and secondary ICD-9 codes for ischemic stroke.23 Another prior analysis using ICD-10 codes in the ARIC and Regards studies reported PPVs of 65.2% and 64.7% for Medicare claims for stroke but also included hemorrhagic strokes and stroke syndrome ICD codes.12 Our analysis of ICD-10 compared with ICD-9 codes was limited because of a low event rate of only 50 ICD-10–identified events. Given that these two analyses of the ARIC cohort for ICD-9 codes and ICD-10 codes varied by >15% in these different studies, as well as our low sample size of stroke outcomes, additional evaluation of ICD-10 codes is warranted for individuals with CKD. This could likely eventually be completed in the CRIC cohort in the future once event numbers rise to a level for greater precision of estimates.
For atrial fibrillation, we report a PPV of 85% for primary ICD codes and 53% for combined primary and secondary ICD codes compared with physician adjudication. Previous literature in atrial fibrillation is limited even outside of CKD populations, but a multiyear study in Olmstead County reported a PPV 64% for prevalent atrial fibrillation using one inpatient or outpatient ICD code.13 Their analysis did not include hospital-only analysis or evaluation of primary and secondary codes separately, which limits our comparisons. Atrial fibrillation is often misidentified in clinical settings, but in longitudinal studies such as ours, individuals may present to the hospital with long-established or stable atrial fibrillation. This also likely explains why we found that PPV when including secondary codes for atrial fibrillation remains near 53%, whereas heart failure was much lower at 33%. In our study, we focused on primary diagnoses codes of atrial fibrillation during a hospitalization to minimize capturing chronic, stable atrial fibrillation. Cohen kappa coefficient's between ICD code and physician adjudication showed only fair agreement for atrial fibrillation compared with moderate agreement for heart failure, myocardial infarction, and stroke.24 This lower level of agreement was likely due to chronic or stable atrial fibrillation identified by physician adjudication that was not placed at the primary ICD code by the treating clinician.
There was a consistent trend that the PPVs for ICD-10 codes did not perform as well as the ICD-9 codes. In 2015, the CRIC dataset collection of ICD codes increased from up to 30–50 ICD codes. However, this would not explain the differences we saw between ICD-9 and ICD-10 accuracy in our analysis of primary only ICD codes. Given that ICD-10 codes were meant to improve coding accuracy, this is something that warrants additional investigation, especially in patients with CKD and kidney failure, to determine whether this is truly a limitation of ICD codes or instead is related to changes in clinical diagnosis and management, e.g., with the addition of improved imaging and biomarkers in the past decade.
Within our CKD and kidney failure population, we did not identify any specific individual characteristics that appeared to influence the accuracy of ICD codes. The associations between cardiovascular disease risk factors and comorbidities including age, diabetes, prior cardiovascular disease, and eGFR were similar in ICD code–identified events and physician adjudication–identified events. This lack of identified bias provides reassurance that ICD-based outcomes may be acceptable in diverse study populations with CKD.
On the basis of our findings, investigators can carefully consider the use of ICD codes versus physician adjudication for cardiovascular disease events in the design of studies. Sensitivities for all cardiovascular disease events types were low in our study. This analysis focused on primary ICD codes in which both increases calculated PPV and decreases sensitivity. Owing to the limitations of data collection, we are unable to determine whether these missed diagnoses by ICD codes are due to incorrect ICD code usage or if missed clinically by the treating patient clinician. Using ICD codes for outcome ascertainment would likely be most beneficial in studies where high specificity or NPV is desired, such as identifying patients who have not had prior cardiovascular disease admissions or events, which could be helpful for determining inclusion in a pragmatic clinical trial. With PPVs near 80% for heart failure, myocardial infarction, stroke, and atrial fibrillation using primary ICD codes, studies with large enough sample sizes might be able to assess interventions or risk factors. Very large studies would also likely be the most cost-prohibitive to run with adjudication committees. ICD code analysis may increase the number and potentially the diversity of individuals that can be included as these studies. When including secondary codes, PPV, NPV, sensitivity, and specificity minimally changed with the addition of more than the first five ICD codes. Although individual institutions or clinicians might vary in ICD-coding practices, including 5–10 ICD codes instead of all ICD codes might be sufficient for future studies' research questions. Future researchers should consider these trade-offs as the plan or interpret future studies.
Our study has several strengths. We evaluated a relatively large, multicenter, longitudinal, diverse cohort of individuals with CKD. Participants were followed longitudinally, and cardiovascular disease events were physician-adjudicated using standardized diagnostic criteria for all participants. Our study spans the usage of ICD-9 and ICD-10 codes, which allows comparison with previous literature, although this lowered our power for ICD-10 code analysis alone.
This study also had limitations. This study included research volunteers which CKD may not be fully generalizable to all adults with CKD in the United States. Although the CRIC study had relatively few exclusion criteria, compared with many large real-world data sources of hospitalizations, the study participants were recruited from CKD clinics and did have to be healthy enough to complete outpatient appointments and study. It is also possible that these research participants were more likely to engage in routine outpatient and preventative care and medications. For hospitalizations before 2013, records were only adjudicated if participants' hospital ICD codes were related to a prespecified cardiovascular outcome of interest. Although there were a range of outcomes beyond what is included in this study, this does limit our interpretation of NPV, sensitivity, and specificity for events before 2013. Our analysis identified individuals with kidney failure, and some of them received a kidney transplant as their first modality of treatment. This number was small overall, and we were unable to examine the kidney transplant group separately. Finally, we were only able to capture the first mode of KRT.
In conclusion, the findings of this cohort study suggest that primary ICD-9 and 10 codes for heart failure, myocardial infarction, stroke, and atrial fibrillation provide near 80% PPV in patients with CKD and kidney failure but have low sensitivity for physician-adjudicated diagnoses. Although ICD code usage in medical research provides opportunities for studies such as large epidemiologic studies and clinical trials and allows greater efficiency with limited resources, there may be some limitations such as lower sensitivities in some cases. However, the relatively high PPVs would allow creation of cohorts with high likelihood of the cardiovascular disease event. Ultimately, investigators will need to weigh the importance of decreased sensitivity of identifying these events against the cost and resource utilization of adjudication committees.
Supplementary Material
Acknowledgments
This article was not prepared in collaboration with investigators of the CRIC study and does not necessarily reflect the opinions or views of the CRIC study investigators, the NIDDK central repository (NIDDK-CR), or the NIDDK. The data from the CRIC reported here were supplied by the NIDDK-CR.
Footnotes
See related letter to the editor, “Critique of ICD Code Accuracy in Identifying Cardiovascular Events in CKD and Kidney Failure Populations,” and reply, “Authors' Reply: Critique of ICD Code Accuracy in Identifying Cardiovascular Events in CKD and Kidney Failure Populations,” on pages 651 and 653, respectively.
Disclosures
Disclosure forms, as provided by each author, are available with the online version of the article at http://links.lww.com/JSN/F456.
Author Contributions
Conceptualization: Nisha Bansal, Ian H. de Boer, Alan S. Go, Bryan Kestenbaum, Leila R. Zelnick, Anna M. Zemke.
Data curation: Leila R. Zelnick.
Formal analysis: Leila R. Zelnick.
Investigation: Nisha Bansal, Anna M. Zemke.
Methodology: Nisha Bansal, Ian H. de Boer, Bryan Kestenbaum, Leila R. Zelnick, Anna M. Zemke.
Resources: Nisha Bansal.
Supervision: Nisha Bansal.
Writing – original draft: Anna M. Zemke.
Writing – review & editing: Nisha Bansal, Ian H. de Boer, Alan S. Go, Bryan Kestenbaum, Leila R. Zelnick, Anna M. Zemke.
Funding
A.M. Zemke: National Institute of Diabetes and Digestive and Kidney Diseases (T32DK00746 and TL1DK143270). N. Bansal: National Institute of Diabetes and Digestive and Kidney Diseases (K26DK138333).
Data Availability Statements
Original data generated for the study are available in a repository subject to controlled access. Data Type: Clinical Trial Data. Repository Name: CRIC. Reason for Restriction: Data from the CRIC study reported here are available for request at the NIDDK-CR Website, Resources for Research, https://repository.niddk.nih.gov. The data are deidentified, and access criteria is determined by the NIDDK-CR.
Supplemental Material
This article contains the following supplemental material online at http://links.lww.com/JSN/F457.
Supplemental Table 1A. Heart failure ICD-9 and ICD-10 codes.
Supplemental Table 1B. Myocardial infarction ICD-9 and ICD-10 codes.
Supplemental Table 1C. Stroke ICD-9 and ICD-10 codes.
Supplemental Table 1D. Atrial fibrillation ICD-9 and ICD-10 codes.
Supplemental Table 2A. Summary of code-based (first position only) versus adjudicated heart failure in CRIC.
Supplemental Table 2B. Summary of code-based (first position only) versus adjudicated myocardial infarction in CRIC.
Supplemental Table 2C. Summary of code-based (first position only) versus adjudicated stroke in CRIC.
Supplemental Table 2D. Summary of code-based (first position only) versus adjudicated atrial fibrillation in CRIC.
Supplemental Table 3. Associations of traditional cardiovascular disease risk factors with code-based versus adjudicated cardiovascular events.
References
- 1.Collister D Pyne L Bhasin AA, et al. Heart failure events in randomized controlled trials for adults receiving maintenance dialysis: a meta-epidemiologic study. Nephrol Dial Transplant. 2025;40(2):371–384. doi: 10.1093/ndt/gfae156 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Tong A Manns B Wang AYM, et al. Implementing core outcomes in kidney disease: report of the standardized outcomes in nephrology (SONG) implementation workshop. Kidney Int. 2018;94(6):1053–1068. doi: 10.1016/j.kint.2018.08.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Feldman HI Appel LJ Chertow GM, et al.; Chronic Renal Insufficiency Cohort (CRIC) Study Investigators. The Chronic Renal Insufficiency Cohort (CRIC) study: design and methods. J Am Soc Nephrol. 2003;14(7 suppl 2):S148–S153. doi: 10.1097/01.ASN.0000070149.78399.ce [DOI] [PubMed] [Google Scholar]
- 4.Lash JP Go AS Appel LJ, et al. Chronic Renal Insufficiency Cohort (CRIC) study: baseline characteristics and associa- tions with kidney function. Clin J Am Soc Nephrol. 2009;4(8):1302–1311. doi: 10.2215/CJN.00070109 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Chronic Renal Insufficiency Cohort (CRIC) Study. Final Protocol. University of Pennsylvania; 2003. Accessed July 10, 2025. https://repository.niddk.nih.gov/media/studies/cric/Protocol/CRIC%20Study_Protocol_V1_20030113.pdf [Google Scholar]
- 6.McKee PA, Castelli WP, McNamara PM, Kannel WB. The natural history of congestive heart failure: the Framingham study. N Engl J Med. 1971;285(26):1441–1446. doi: 10.1056/NEJM197112232852601 [DOI] [PubMed] [Google Scholar]
- 7.Einhorn PT Davis BR Massie BM, et al. The antihypertensive and lipid lowering treatment to prevent heart attack trial (ALLHAT) heart failure validation study: diagnosis and prognosis. Am Heart J. 2007;153(1):42–53. doi: 10.1016/j.ahj.2006.10.012 [DOI] [PubMed] [Google Scholar]
- 8.Thygesen K Alpert JS White HD, et al.; Joint ESC/ACCF/AHA/WHF Task Force for the Redefinition of Myocardial Infarction. Universal definition of myocardial infarction. Circulation. 2007;116(22):2634–2653. doi: 10.1161/CIRCULATIONAHA.107.187397 [DOI] [PubMed] [Google Scholar]
- 9.Rosamond WD Folsom AR Chambless LE, et al. Stroke incidence and survival among middle-aged adults: 9-year follow-up of the Atherosclerosis Risk in Communities (ARIC) cohort. Stroke. 1999;30(4):736–743. doi: 10.1161/01.str.30.4.736 [DOI] [PubMed] [Google Scholar]
- 10.Bansal N, Zelnick LR, Scherzer R, Estrella M, Shlipak MG. Risk factors and outcomes associated with heart failure with preserved and reduced ejection fraction in people with chronic kidney disease. Circ Heart Fail. 2024;17(5):e011173. doi: 10.1161/CIRCHEARTFAILURE.123.011173 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Pocobelli G Ichikawa L Yu O, et al. Validation of International Classification of Diseases, tenth revision, clinical modification diagnosis codes for heart failure subtypes. Pharmacoepidemiol Drug Saf. 2022;31(9):992–997. doi: 10.1002/pds.5489 [DOI] [PubMed] [Google Scholar]
- 12.Columbo JA Daya N Colantonio LD, et al. Derivation and validation of ICD-10 codes for identifying incident stroke. JAMA Neurol. 2024;81(8):875–881. doi: 10.1001/jamaneurol.2024.2044 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chamberlain AM Roger VL Noseworthy PA, et al. Identification of incident atrial fibrillation from electronic medical records. J Am Heart Assoc. 2022;11(7):e023237. doi: 10.1161/JAHA.121.023237 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Muntner P Anderson A Charleston J, et al. Hypertension awareness, treatment, and control in adults with CKD: results from the Chronic Renal Insufficiency Cohort (CRIC) study. Am J Kidney Dis. 2010;55(3):441–451. doi: 10.1053/j.ajkd.2009.09.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Wei LJ, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. J Am Stat Assoc. 1989;84(408):1065–1073. doi: 10.2307/2290084 [DOI] [Google Scholar]
- 16.Floyd JS Wellman R Fuller S, et al. Use of electronic health data to estimate heart failure events in a population-based cohort with CKD. Clin J Am Soc Nephrol. 2016;11(11):1954–1961. doi: 10.2215/CJN.03900416 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.McCormick N, Lacaille D, Bhole V, Avina-Zubieta JA. Validity of heart failure diagnoses in administrative databases: a systematic review and meta-analysis. PLoS One. 2014;9(8):e104519. doi: 10.1371/journal.pone.0104519 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Saczynski JS Andrade SE Harrold LR, et al. A systematic review of validated methods for identifying heart failure using administrative data. Pharmacoepidemiol Drug Saf. 2012;21 Suppl 1(0 1):129–140. doi: 10.1002/pds.2313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Rosamond WD Chambless LE Sorlie PD, et al. Trends in the sensitivity, positive predictive value, false-positive rate, and comparability ratio of hospital discharge diagnosis codes for acute myocardial infarction in four US communities, 1987-2000. Am J Epidemiol. 2004;160(12):1137–1146. doi: 10.1093/aje/kwh341 [DOI] [PubMed] [Google Scholar]
- 20.Ando T Ooba N Mochizuki M, et al. Positive predictive value of ICD-10 codes for acute myocardial infarction in Japan: a validation study at a single center. BMC Health Serv Res. 2018;18(1):895. doi: 10.1186/s12913-018-3727-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Nedkoff L Lopez D Hung J, et al. Validation of ICD-10-AM coding for myocardial infarction subtype in hospitalisation data. Heart Lung Circ. 2022;31(6):849–858. doi: 10.1016/j.hlc.2021.11.014 [DOI] [PubMed] [Google Scholar]
- 22.Bhatia PM, Daniels LB. Highly sensitive cardiac troponins: the evidence behind sex-specific cutoffs. J Am Heart Assoc. 2020;9(10):e015272. doi: 10.1161/JAHA.119.015272 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Jones SA, Gottesman RF, Shahar E, Wruck L, Rosamond WD. Validity of hospital discharge diagnosis codes for stroke: the Atherosclerosis Risk in Communities study. Stroke. 2014;45(11):3219–3225. doi: 10.1161/STROKEAHA.114.006316 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. doi: 10.2307/2529310 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Original data generated for the study are available in a repository subject to controlled access. Data Type: Clinical Trial Data. Repository Name: CRIC. Reason for Restriction: Data from the CRIC study reported here are available for request at the NIDDK-CR Website, Resources for Research, https://repository.niddk.nih.gov. The data are deidentified, and access criteria is determined by the NIDDK-CR.