Assessment of ICD-10-CM code assignment validity for case finding of outpatient anticoagulant-related bleeding among Medicare beneficiaries

Abstract

Purpose:

To assess performance of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) code assignments for identifying bleeding events resulting in emergency department visits and hospitalizations among outpatient Medicare beneficiaries prescribed anticoagulants.

Methods:

Performance of 206 ICD-10-CM code assignments indicative of bleeding, five anticoagulant adverse effect/poisoning codes, and five coagulopathy codes (according to Medicare Parts A and B claims) as assessed among Medicare fee-for-service beneficiaries prescribed anticoagulants between October 1, 2015 and September 30, 2016 (according to Part D claims). Structured medical record review was the gold standard for validating the presence of anticoagulant-related bleeding. Sensitivity was adjusted to correct for partial verification bias due to sampling design.

Results:

Based on the study sample of 1166 records (583 cases, 583 controls), 57 of 206 codes yielded the optimal performance for anticoagulant-related bleeding (diagnostic odds ratio, 51; positive predictive value (PPV), 75.7% [95% CI, 72.0%–79.1%]; adjusted sensitivity, 70.0% [95% CI, 63.2%–77.7%]). Codes for intracranial bleeding demonstrated the highest PPV (85.0%) and adjusted sensitivity (91.0%). Bleeding codes in the primary position demonstrated high PPV (86.9%), but low adjusted sensitivity (36.0%). The adjusted sensitivity improved to 69.5% when codes in a secondary position were added. Only one adverse effect/poisoning code was used, appearing in 7.8% of cases and controls (PPV, 71.4% and adjusted sensitivity, 6.8%).

Conclusions:

Performance of ICD-10-CM code assignments for bleeding among patients prescribed anticoagulants varied by bleed type and code position. Adverse effect/poisoning codes were not commonly used and would have missed over 90% of anticoagulant-related bleeding cases.

1 |. INTRODUCTION

Anticoagulants are essential for the prevention and treatment of thromboembolic disorders but are also leading causes of medication-related harm, including emergency department (ED) visits and hospital admissions for adverse drug events (ADEs) among older Americans.^1–6 Administrative claims data are becoming increasingly utilized in postmarketing surveillance of drug safety.⁷ With five additional oral anticoagulants, other than warfarin, approved in the United States since 2010,⁸ new anticoagulant management metrics in physician payment models,⁹ and a focus on anticoagulants in Centers for Medicare & Medicaid Services (CMS) nationwide quality improvement initiatives,^10,11 administrative claims data will be increasingly important for assessing anticoagulant safety. Although using administrative claims is an efficient way to monitor ADEs and measure quality improvement, very little is known about the validity of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) for identifying medication-related harm.^12,13 Reliance on a limited set of ICD-9-CM codes,^14–17 such as external cause of injury and poisoning codes (“E” codes), may identify only 10% of anticoagulant ADEs¹⁸ and no published studies using U.S. data have assessed validity of ICD-10-CM, which took effect in the United States in 2015.¹⁹

To advance public health and postmarketing surveillance and quality improvement efforts, we sought to assess the validity of ICD-10-CM code assignment for identifying anticoagulant-related bleeding events that resulted in acute care encounters.

2 |. METHODS

2.1 |. Study design and setting

This study was a multicenter, retrospective evaluation utilizing medical record review for validation of a prospectively derived list of ICD-10-CM codes. The study population included Medicare fee-for-service beneficiaries who received care in five hospitals in three states (California, Florida, and Ohio) and were prescribed anticoagulants between October 1, 2015 and September 30, 2016, identified using linked Medicare Parts A, B, and D administrative claims.

Anticoagulant exposure started on the fill date of the first outpatient anticoagulant prescription within the study period or on October 1 if the patient had a supply from an anticoagulant prescription that overlapped the beginning of the study period. Exposure ended 10 days after the fill date plus the days’ supply of the last anticoagulant prescription, with a gap of less than 30 days between prescription fills allowed. Anticoagulants included any oral anticoagulant, unfractionated heparin, low-molecular-weight heparin, or fondaparinux.

Cases were defined as patients prescribed anticoagulants who had acute care encounters (ie, emergency department [ED] visit, observation stay, or hospitalization) with at least one ICD-10-CM code indicative of bleeding, as identified on Parts B and A data. Controls were patients prescribed anticoagulants who had acute care encounters with no ICD-10-CM codes indicative of bleeding during their anticoagulant exposure period. Cases were matched with an equal number of controls by presenting hospital, type of encounter, year of discharge, and length of anticoagulant exposure relative to the index acute care encounter. For cases with more than one match, one control was randomly selected from potential matches.

2.2 |. Administrative data: Identification of ICD-10-CM bleeding codes

We compiled a list of ICD-9-CM and ICD-10-CM codes that previously have been used to validate anticoagulant-related bleeding and used fiscal year 2016 general equivalence mappings to map ICD-9-CM to ICD-10-CM codes.^20–25 The list of ICD-10-CM codes was reviewed by four clinical experts in cardiology, anticoagulation management, and medical coding. To identify bleeding events that were most likely related to or exacerbated by anticoagulant therapy, experts removed codes indicative of perioperative bleeding and codes indicative of traumatic intracranial bleeding with a severe head injury, such as skull fracture, brain injury, or crushing injury of the head. To identify clinically significant bleeding events, experts did not include codes for microscopic hematuria or for contusions or lacerations. The resulting list included 206 ICD-10-CM codes, consisting of 95 codes for intracranial bleeding, 60 codes for gastrointestinal (GI) bleeding, and 51 codes for other types of bleeds (eg, genitourinary bleeds; Tables S1 and S2).

Acute care encounter types were ED visits (ED treat-and-release and observation encounters with a bleeding code in any diagnosis position), hospitalizations with a bleeding code in the primary diagnosis position (Position 1), or hospitalizations with a bleeding code in a secondary diagnosis position (Positions 2 through 25). All cases with a bleeding code in the primary position were included; a random subset of hospitalization cases with bleeding codes in a secondary position and of ED visits were then chosen. For hospitalization encounters, cases were retained for analysis only if the present-on-admission indicator corresponding to that bleeding diagnosis was equal to “yes” (ie, the event was present at the time of admission or occurred during an outpatient visit prior to admission).

In addition to bleeding codes, we evaluated performance of 10 other ICD-10-CM codes: five codes indicating anticoagulant adverse effects or poisoning and five codes indicating coagulopathy.

2.3 |. Clinical data: Validation of bleeding events

Medical record review served as the gold standard reference for validating anticoagulant-related bleeding events identified from administrative claims. Medical records were reviewed by nine clinician reviewers specializing in internal medicine, emergency medicine, and/or pharmacotherapy, with five reviewers assigned to each hospital’s records. The reviewers were blinded to case and control assignment. Each record was reviewed for the presence of a diagnosis of bleeding by the treating clinician, supporting evidence (eg, laboratory values, endoscopy results, transfusions, or other treatments) for bleeding, and attribution of bleeding to outpatient anticoagulant use. For attribution assessments, we used the Liverpool Adverse Drug Reaction Causality Assessment Tool, which classifies the association between the drug and adverse event into four causality categories: “definite,” “probable,” “possible,” and “unlikely.”²⁶ We added a category of “unable to determine” for records that did not contain sufficient information. A gold standard reference for a bleed required two criteria to be met: (a) diagnosis of bleeding present in the medical record or objective evidence of bleeding and (b) definite, probable, or possible attribution of bleeding to an anticoagulant. Bleeding events were further assessed for severity using a grade of major or nonmajor bleeding based on the International Society on Thrombosis and Haemostasis criteria.²⁷ To ensure consistency in medical record abstraction, reviewers collected data using a structured abstraction form and followed an abstraction protocol on which they received training. We assessed inter-rater reliability (IRR) by calculating the mean of pairwise kappa statistics using a sample of records randomly selected from the five participating hospitals. The pairwise kappa was calculated for each of 10 pairs among five possible reviewers across 29 unique records for which reviews were completed. Additional review of false positive and false negative records was undertaken by one study investigator (T.H.Y.) to classify sources of disagreement between claims data and medical record review.

2.4 |. Statistical analysis

Statistical analyses were performed using SAS (SAS Institute Inc., Cary, NC). Two-sided Fisher’s exact test and chi-square test were used for comparisons of baseline characteristics between cases and controls. We assessed ICD-10-CM codes for positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity. Since the medical record review sample was enriched to 1-to-1 case-to-control ratio, calculation of performance attributes would be affected by partial verification bias (overestimation of sensitivity and underestimation of specificity). We used a previously described correction,²⁸ which requires estimation of the prevalence of the bleeding codes in the original sample (before selection) to calculate adjusted sensitivity. We estimated this prevalence using the combined claims data from the five-hospital cohort. Confidence intervals for the corrected statistics were estimated by cluster bootstrap resampling of the selected cases nested in matched pairs, followed by application of the correction, assuming a constant bleed code prevalence in the original sample. The 2.5% and 97.5% quantiles of the distribution resulting from 2000 bootstrap samples were taken as the lower and upper confidence limits, respectively. Diagnostic odds ratios (ORs) were calculated to provide a summary measure of the performance. This statistic was estimated from the PPV and NPV as PPV × NPV / ((1−PPV) × (1−NPV)).²⁹ To derive a select code set that optimized PPV and NPV, codes were ordered by the performance of each single code (PPV then NPV). Overall performance was assessed for sequential code sets formed by adding each code in list from best to worst performing codes.

3 |. RESULTS

A total of 1166 records (583 cases, 583 controls) were utilized to assess performance of the 206 ICD-10-CM codes (Figure 1). Cases were slightly older than controls (mean age, 73.8 years ± 13.8 years vs 72.9 years ± 12.9 years), but differences between cases and controls in mean age, sex, or race were not statistically significant (Table 1). Warfarin was the most commonly prescribed anticoagulant among cases and controls (63.4%), followed by the direct oral anticoagulants (41.1%; rivaroxaban, apixaban, and dabigatran). Most (61.4%) cases were hospitalizations; in 43.4% of cases, bleeding codes were in the primary position and in 18.0% bleeding codes were in the secondary position. Of the cases with a confirmed bleed on medical record review (N = 499), attribution of the bleeding event to an anticoagulant was categorized as “definite” (10.4%), “probable” (39.7%), or “possible” (37.3%). Approximately, one half (51.7%) of confirmed cases involved nonmajor bleeding and one half (48.3%) involved major bleeding.

FIGURE 1.

Identification of cases and controls for validation of ICD-10-CM diagnosis codes for bleeding among Medicare beneficiaries prescribed anticoagulants in a five-hospital cohort. Abbreviations: ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification, POA, Present on admission. ^aAcute care encounters consisted of emergency department visits (including observation stays) and hospital admissions. ^bRefer to Table S1 for all codes included in the study. ^cOf these 1500 encounters, 29 encounters (N = 24 cases, N = 5 controls) were randomly selected to calculate inter-rater reliability. ^dFor cases with multiple acute care encounters with bleeding events, a hierarchical rule was applied and the bleeding event in the highest-ranking acute care encounter was retained (hospital admissions with diagnosis code for bleeding in the primary position were considered highest ranking and emergency department visits or observation stays were considered lowest ranking). For patients with multiple bleeding events resulting in the same type of acute care encounter, a single event was randomly selected and retained. Thus, each patient contributed only one encounter. ^eCases and controls were matched by presenting hospital, type of encounter, year of discharge, and length of anticoagulant exposure relative to the index acute care encounter. For cases with more than one match, one control was randomly selected from potential matches

TABLE 1.

Patients prescribed anticoagulants with ICD-10-CM diagnosis codes for bleeding and matched controls, by demographic and clinical characteristics (N = 1166)^a

Patient Characteristics	Cases, N (%)	Controls, N (%)	P value
Age, y			0.039
21–64	99 (17.0)	122 (20.9)
65–74	161 (27.6)	181 (31.0)
75–84	197 (33.8)	160 (27.4)
≥85	126 (21.6)	120 (20.6)
Sex			0.204
Women	273 (46.8)	295 (50.6)
Men	310 (53.2)	288 (49.4)
Race			0.586
White	472 (81.0)	484 (83.0)
Black	73 (12.5)	68 (11.7)
Other or Unknown	38 (6.5)	31 (5.3)
Presenting hospital			N/A
Hospital A	101 (17.3)	101 (17.3)
Hospital B	111 (19.0)	111 (19.0)
Hospital C	133 (22.8)	133 (22.8)
Hospital D	130 (22.3)	130 (22.3)
Hospital E	108 (18.5)	108 (18.5)
Anticoagulant prescribedb
Warfarin	391 (67.1)	348 (59.7)	0.016
Apixaban	94 (16.1)	127 (21.8)	0.017
Rivaroxaban	112 (19.2)	104 (17.8)	0.598
Dabigatran	25 (4.3)	42 (7.2)	0.043
Enoxaparin	100 (17.1)	101 (17.3)	1.000
Unfractionated heparin	4 (0.7)	5 (0.9)	1.000
Fondaparinux	3 (0.5)	1 (0.2)	0.624
Dalteparin	2 (0.3)	0	0.500
Type of encounterc
ED visit (including observation stays)	225 (38.6)	225 (38.6)	N/A
Hospitalization - bleeding code in primary position	253 (43.4)	253 (43.4)
Hospitalization - bleeding code in secondary position	105 (18.0)	105 (18.0)
Year of encounter
2015	131 (22.5)	131 (22.5)	N/A
2016	452 (77.5)	452 (77.5)
Total	583	583	N/A

Abbreviations: ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; N/A, not applicable.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016). Cases were patients with at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period. Controls were patients prescribed anticoagulants who had acute care encounters with no ICD-10-CM codes indicative of bleeding during their anticoagulant exposure period. Controls were matched by presenting hospital, type of encounter, year of discharge, and length of anticoagulant exposure relative to the index acute care encounter. P values are not shown for variables on which cases and controls were matched (N/A).

^bTotal exceeds 583 for each column as there were 264 (22.6%) of 1166 patients with prescriptions for more than one anticoagulant during the study period.

^c“Bleeding code in primary position” refers to encounters with ICD-10-CM codes indicative of bleeding that appeared only in the primary (first) diagnosis position. “Bleeding code in secondary position” refers to encounters with ICD-10-CM codes indicative of bleeding that appeared only in the secondary (2nd through 25th) diagnosis position. Matched controls did not have ICD-10-CM codes indicative of bleeding, but were selected in 1:1 ratio for each case under each acute care encounter.

Among all cases, a total of 66 (32.0%) of 206 codes potentially indicative of bleeding were used in claims data, consisting of 29 codes for intracranial bleeding, 22 codes for GI bleeding, and 15 codes for other bleeding (Table S1). The most commonly identified codes, which comprised approximately two thirds of all cases, were R040, “epistaxis” (15.3%), K921, “melena” (14.8%), K922, “gastrointestinal hemorrhage, unspecified” (14.1%), R319, “hematuria, unspecified” (12.2%), R042, “hemoptysis” (6.3%), and R310, “gross hematuria” (5.0%; Table 2). Among ED visit cases, four codes for intracranial bleeding (I609, I610, I618, and I629), three codes for GI bleeding (K264, K5731, and K661), and three codes for other bleeding (M25012, N950, and R233) demonstrated 100% PPV. Among hospitalization cases with a bleeding code in the primary or secondary positions, the majority of codes demonstrated 100% PPV. Of the 66 codes used in claims data, nine codes (S065X8A, S066X8A, S06358, S064X9A, S066X3A, K51911, K5791, I312, and H05232) failed to identify any bleeding cases (PPV, 0%).

TABLE 2.

Frequency and positive predictive value of ICD-10-CM diagnosis codes for identifying bleeding among patients prescribed anticoagulants, by encounter type and code position^a

ICD-10-CM Codeb	Code Description	Frequency	%	PPVc (%)
Any acute care encounter, code in any positiond
R040	Epistaxis	89	15.3	87.6
K921	Melena	86	14.8	70.9
K922	Gastrointestinal hemorrhage, unspecified	82	14.1	72.0
R319	Hematuria, unspecified	71	12.2	54.9
R042	Hemoptysis	37	6.3	51.4
R310	Gross hematuria	29	5.0	82.8
K5731	Diverticulosis of large intestine without perforation or abscess with bleeding	21	3.6	100.0
K625	Hemorrhage of anus and rectum	20	3.4	80.0
S065X0A	Traumatic subdural hemorrhage without loss of consciousness, initial encounter	20	3.4	90.0
K31811	Angiodysplasia of stomach and duodenum with bleeding	14	2.4	100.0
K920	Hematemesis	13	2.2	53.8
N939	Abnormal uterine and vaginal bleeding, unspecified	11	1.9	36.4
R58	Hemorrhage, not elsewhere classified	9	1.5	66.7
I615	Nontraumatic intracerebral hemorrhage, intraventricular	9	1.5	100.0
S066X0A	Traumatic subarachnoid hemorrhage without loss of consciousness, initial encounter	9	1.5	100.0
I6201	Nontraumatic acute subdural hemorrhage	7	1.2	100.0
K2971	Gastritis, unspecified, with bleeding	6	1.0	83.3
K661	Hemoperitoneum	6	1.0	83.3
I618	Other nontraumatic intracerebral hemorrhage	6	1.0	83.3
N938	Other specified abnormal uterine and vaginal bleeding	6	1.0	66.7
K5521	Angiodysplasia of colon with hemorrhage	5	0.9	100.0
K264	Chronic or unspecified duodenal ulcer with hemorrhage	5	0.9	80.0
I609	Nontraumatic subarachnoid hemorrhage, unspecified	5	0.9	100.0
I610	Nontraumatic intracerebral hemorrhage in hemisphere, subcortical	5	0.9	100.0
I629	Nontraumatic intracranial hemorrhage, unspecified	5	0.9	80.0
S065X9A	Traumatic subdural hemorrhage with loss of consciousness of unspecified duration, initial encounter	5	0.9	100.0
K2211	Ulcer of esophagus with bleeding	5	0.9	100.0
I614	Nontraumatic intracerebral hemorrhage in cerebellum	4	0.7	75.0
I619	Nontraumatic intracerebral hemorrhage, unspecified	4	0.7	100.0
K254	Chronic or unspecified gastric ulcer with hemorrhage	4	0.7	100.0
I611	Nontraumatic intracerebral hemorrhage in hemisphere, cortical	3	0.5	100.0
K2981	Duodenitis with bleeding	3	0.5	100.0
S066X9A	Traumatic subarachnoid hemorrhage with loss of consciousness of unspecified duration, initial encounter	3	0.5	33.3
K274	Chronic or unspecified peptic ulcer, site unspecified, with hemorrhage	2	0.3	50.0
I312	Hemopericardium, not elsewhere classified	2	0.3	0.0
I6200	Nontraumatic subdural hemorrhage, unspecified	2	0.3	100.0
I6202	Nontraumatic subacute subdural hemorrhage	2	0.3	50.0
I6203	Nontraumatic chronic subdural hemorrhage	2	0.3	50.0
K226	Gastro-esophageal laceration-hemorrhage syndrome	2	0.3	100.0
K2901	Acute gastritis with bleeding	2	0.3	100.0
K3182	Dieulafoy lesion (hemorrhagic) of stomach and duodenum	2	0.3	100.0
K6381	Dieulafoy lesion of intestine	2	0.3	100.0
N950	Postmenopausal bleeding	2	0.3	50.0
R0489	Hemorrhage from other sites in respiratory passages	2	0.3	50.0
R233	Spontaneous ecchymoses	2	0.3	100.0
ED visits, code in any positione
R040	Epistaxis	70	31.1	90.0
R319	Hematuria, unspecified	55	24.4	65.5
K921	Melena	21	9.3	47.6
K922	Gastrointestinal hemorrhage, unspecified	17	7.6	52.9
R042	Hemoptysis	16	7.1	50.0
R310	Gross hematuria	14	6.2	71.4
K625	Hemorrhage of anus and rectum	12	5.3	75.0
N939	Abnormal uterine and vaginal bleeding, unspecified	9	4.0	33.3
R58	Hemorrhage, not elsewhere classified	7	3.1	57.1
K920	Hematemesis	6	2.7	50.0
I629	Nontraumatic intracranial hemorrhage, unspecified	2	0.9	100.0
R233	Spontaneous ecchymoses	2	0.9	100.0
Hospitalizations, code in primary positionf
K922	Gastrointestinal hemorrhage, unspecified	48	13.4	87.5
K921	Melena	43	12.0	83.7
S065X0A	Traumatic subdural hemorrhage without loss of consciousness, initial encounter	19	5.3	89.5
K5731	Diverticulosis of large intestine without perforation or abscess with bleeding	18	5.0	100.0
K31811	Angiodysplasia of stomach and duodenum with bleeding	13	3.6	100.0
R040	Epistaxis	11	3.1	100.0
R310	Gross hematuria	7	2.0	100.0
K625	Hemorrhage of anus and rectum	6	1.7	83.3
I6201	Nontraumatic acute subdural hemorrhage	6	1.7	100.0
I615	Nontraumatic intracerebral hemorrhage, intraventricular	5	1.4	100.0
S066X0A	Traumatic subarachnoid hemorrhage without loss of consciousness, initial encounter	5	1.4	100.0
S065X9A	Traumatic subdural hemorrhage with loss of consciousness of unspecified duration, initial encounter	5	1.4	100.0
K2211	Ulcer of esophagus with bleeding	5	1.4	100.0
K264	Chronic or unspecified duodenal ulcer with hemorrhage	4	1.1	75.0
K2971	Gastritis, unspecified, with bleeding	4	1.1	100.0
K920	Hematemesis	4	1.1	75.0
K5521	Angiodysplasia of colon with hemorrhage	3	0.8	100.0
I611	Nontraumatic intracerebral hemorrhage in hemisphere, cortical	3	0.8	100.0
I610	Nontraumatic intracerebral hemorrhage in hemisphere, subcortical	3	0.8	100.0
I619	Nontraumatic intracerebral hemorrhage, unspecified	3	0.8	100.0
I614	Nontraumatic intracerebral hemorrhage in cerebellum	2	0.6	100.0
I618	Other nontraumatic intracerebral hemorrhage	2	0.6	100.0
K661	Hemoperitoneum	2	0.6	100.0
I6202	Nontraumatic subacute subdural hemorrhage	2	0.6	50.0
K226	Gastro-esophageal laceration-hemorrhage syndrome	2	0.6	100.0
K254	Chronic or unspecified gastric ulcer with hemorrhage	2	0.6	100.0
K2901	Acute gastritis with bleeding	2	0.6	100.0
K2981	Duodenitis with bleeding	2	0.6	100.0
K3182	Dieulafoy lesion (hemorrhagic) of stomach and duodenum	2	0.6	100.0
S066X9A	Traumatic subarachnoid hemorrhage with loss of consciousness of unspecified duration, initial encounter	2	0.6	50.0
N938	Other specified abnormal uterine and vaginal bleeding	2	0.6	100.0
Hospitalizations, code in secondary positionf
K921	Melena	22	6.1	68.2
R042	Hemoptysis	20	5.6	50.0
K922	Gastrointestinal hemorrhage, unspecified	17	4.7	47.1
R319	Hematuria, unspecified	15	4.2	20.0
R040	Epistaxis	8	2.2	50.0
R310	Gross hematuria	8	2.2	87.5
I615	Nontraumatic intracerebral hemorrhage, intraventricular	4	1.1	100.0
S066X0A	Traumatic subarachnoid hemorrhage without loss of consciousness, initial encounter	4	1.1	100.0
I609	Nontraumatic subarachnoid hemorrhage, unspecified	3	0.8	100.0
K920	Hematemesis	3	0.8	33.3
K661	Hemoperitoneum	3	0.8	66.7
I618	Other nontraumatic intracerebral hemorrhage	3	0.8	66.7
N938	Other specified abnormal uterine and vaginal bleeding	3	0.8	66.7
N939	Abnormal uterine and vaginal bleeding, unspecified	2	0.6	50.0
K5521	Angiodysplasia of colon with hemorrhage	2	0.6	100.0
K274	Chronic or unspecified peptic ulcer, site unspecified, with hemorrhage	2	0.6	50.0
K5731	Diverticulosis of large intestine without perforation or abscess with bleeding	2	0.6	100.0
K2971	Gastritis, unspecified, with bleeding	2	0.6	50.0
I312	Hemopericardium, not elsewhere classified	2	0.6	0.0
K254	Chronic or unspecified gastric ulcer with hemorrhage	2	0.6	100.0
K625	Hemorrhage of anus and rectum	2	0.6	100.0
I614	Nontraumatic intracerebral hemorrhage in cerebellum	2	0.6	50.0
I629	Nontraumatic intracranial hemorrhage, unspecified	2	0.6	50.0
I6200	Nontraumatic subdural hemorrhage, unspecified	2	0.6	100.0
I6203	Nontraumatic chronic subdural hemorrhage	2	0.6	50.0
R0489	Hemorrhage from other sites in respiratory passages	2	0.6	50.0

Abbreviations: ED, emergency department; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; PPV, positive predictive value.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016) and at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period.

^bCodes are shown only if they occurred with N > 1 frequency for each encounter type. Cases may have greater than one ICD-10-CM code for bleeding from the table present.

^cPPV refers to the percentage of cases with that ICD-10-CM code that were confirmed as a bleed of any type on medical record review.

^dFrequency percentages for any acute care encounters are calculated out of a denominator of all cases with an ICD-10-CM code for any type of bleed (N = 583).

^eFrequency percentages for ED visits are calculated out of a denominator of all ED visit cases with an ICD-10-CM code for any type of bleed (N = 225).

^fFrequency percentages for hospitalizations (diagnosis code in primary or secondary position) are calculated out of a denominator of all hospitalization cases with an ICD-10-CM code for any type of bleed (N = 358).

Coding performance varied by code position (Table 3). When in the primary position, PPV of codes for any bleeding was high (86.9%), but sensitivity was low (36.0%). Addition of secondary position codes reduced PPV to 74.7% but increased sensitivity to 69.5%. This pattern was consistent across the various types of bleeding. When codes in any position were included, intracranial bleeding codes had the highest PPV (85.0%), sensitivity (91.0%), and diagnostic OR (2986), followed by GI bleeding (PPV, 75.4%; sensitivity, 90.7%; diagnostic OR, 539) and other bleeds (PPV, 68.6%; sensitivity, 53.0%; diagnostic OR, 47).

TABLE 3.

Performance attributes of ICD-10-CM diagnosis codes for identifying bleeding among patients prescribed anticoagulants, by type of bleed and code position^a

Type of Bleed, Code Positionb	Frequency	%c	PPV, 95% CId	Adjusted Sensitivity, 95% CIe	Diagnostic OR
Any bleeding
Primary position	397	69.9	86.9% (83.2%–90.1%)	36.0% (33.5%–41.3%)	37
Any position	568	100.0	74.7% (70.9%–78.2%)	69.5% (63.0%–76.7%)	46
Intracranial bleeding
Primary position	69	12.1	91.3% (82.0%–96.7%)	71.2% (59.4%–89.9%)	1590
Any position	80	14.1	85.0% (75.3%–92.0%)	91.0% (79.6%–100.0%)	2986
GI bleeding
Primary position	191	33.6	84.8% (78.9%–89.6%)	53.4% (45.8%–62.5%)	154
Any position	252	44.4	75.4% (69.6%–80.6%)	90.7% (83.0%–98.0%)	539
Other bleeding
Primary position	137	24.1	85.4% (78.4%–90.9%)	23.9% (20.4%–28.5%)	63
Any position	236	41.5	68.6% (62.3%–74.5%)	53.0% (45.6%–61.6%)	47

Abbreviations: CI, confidence interval; GI, gastrointestinal; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; OR, odds ratio; PPV, positive predictive value.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016). Cases were selected from patients with at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period. Controls were matched by presenting hospital, type of encounter, year of discharge, and length of anticoagulant exposure relative to the acute care encounter.

^bRefer to Table S1 for a list of codes pertaining to each type of bleed.

^cFrequency percentages are calculated out of a denominator of 568 cases with an ICD-10-CM code for any type of bleed. Cases with a diagnosis code indicative of more than one type of bleeding (eg, a code for both intracranial bleeding and a code for other bleeding) were excluded from this analysis (N = 15).

^dPPV refers to the percentage of cases with ICD-10-CM code in that position that were confirmed as a bleed of that type on medical record review.

^eSensitivity refers to the sensitivity of a primary or secondary position code for identifying a bleed of that type. Sensitivity adjusted to account for verification bias due to sampling design.²⁸

Of the five anticoagulant adverse effect/poisoning codes, only one code (T45515A, “adverse effect of anticoagulants, initial encounter”) was used, appearing in 7.8% of cases and controls (PPV, 71.4%; sensitivity 6.8%; Table 4). With the exception of the code Z7901, “long term (current) use of anticoagulants”, codes indicative of coagulopathy (rather than bleeding) were also used infrequently. Although most of these codes demonstrated moderately high PPV, sensitivity was low for all adverse effect/poisoning and coagulopathy codes, except for code Z7901, which had sensitivity of 64.7% but a PPV of only 48.0%. Together, all the adverse effect/poisoning and coagulopathy codes demonstrated a PPV of 49.1% and sensitivity of 71.0%; when code Z7901 was excluded, these codes demonstrated a PPV of 71.5% (95% CI, 64.0%–78.3%) and sensitivity of 17.0% (95% CI, 15.3%–18.8%).

TABLE 4.

Performance attributes of adverse effect, poisoning, and coagulopathy ICD-10-CM diagnosis codes for identifying bleeding among patients prescribed anticoagulants, by code^a

ICD-10-CM Code	Code Description	Frequency (%)b	PPV, 95% CIc	Adjusted Sensitivity, 95% CId	Bleeding Code Concurrently Present, Frequency (%)e
Adverse effect and poisoning codes
T45515A	Adverse effect of anticoagulants, initial encounter	91 (7.8%)	71.4 (61.0%–80.4%)	6.8% (6.0%–7.8%)	77 (84.6%)
T45511A	Poisoning by anticoagulants, accidental (unintentional), initial encounter	0
T45514A	Poisoning by anticoagulants, undetermined, initial encounter	0
T45515S	Adverse effect of anticoagulants, sequela	0
T45521A	Poisoning by antithrombotic drugs, accidental (unintentional), initial encounter	0
Coagulopathy codes
D689	Coagulation defect, unspecified	19 (1.6%)	84.2% (60.4%–96.6%)	2.8% (2.2%–3.3%)	16 (84.2%)
D6832	Hemorrhagic disorder due to extrinsic circulating anticoagulants	12 (1.0%)	83.3% (51.6%–97.9%)	0.9% (0.6%–l.l%)	12 (100.0%)
D688	Other specified coagulation defects	6 (0.5%)	66.7% (22.3%–95.7%)	0.4% (0.1%–0.7%)	5 (83.3%)
R791	Abnormal coagulation profile	88 (7.5%)	65.9% (55.0%–75.7%)	9.2% (7.8%–10.5%)	63 (71.6%)
Z7901	Long term (current) use of anticoagulants	640 (54.9%)	48.0% (44.0%–51.9%)	64.7% (61.1%–68.0%)	354 (55.3%)
All codes above
All	Any of the codes above	696 (59.7%)	49.1% (45.4%–52.9%)	71.0% (67.5%–74.3 %)	395 (56.8%)

Abbreviations: ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; PPV, positive predictive value.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016). Cases were patients with at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period. Controls were patients prescribed anticoagulants who had acute care encounters with no ICD-10-CM codes indicative of bleeding during their anticoagulant exposure period.

^bTotal exceeds the sum of all code frequencies as some encounters had greater than one adverse effect/poisoning or coagulopathy codes present. Frequency percentages are calculated out of a denominator of all cases (1166) with an ICD-10-CM code for any type of bleed (N = 583) and all controls (N = 583).

^cPPV refers to the percentage of cases with that ICD-10-CM code that were confirmed as a bleed of any type on medical record review.

^dSensitivity refers to the sensitivity of that code for identifying a bleed of any type. Sensitivity adjusted to account for verification bias due to sampling design.²⁸

^eFrequency percentages are calculated out of a denominator of medical records with that specific ICD-10-CM code; eg, 77 (84.6%) of 91 records with code T45515A present also had an ICD-10-CM code for bleeding present.

Performance attributes were evaluated across various sets of codes to identify an optimized code set (Table 5). Together, all study codes demonstrated moderately high PPV (74.8%) and moderately high sensitivity (69.8%). The code set optimizing PPV and NPV consisted of 57 codes and demonstrated similar PPV and sensitivity, but with only a marginal improvement in diagnostic OR from 48 to 51, indicating that all bleeding codes identified in claims data contribute to overall performance. Addition of the adverse effect/poisoning code, T45515A, did not substantially impact the performance of either the original or optimized code set. Addition of all the adverse effect, poisoning, and coagulopathy codes improved sensitivity of the optimized code set from 70.0% to 95.4% but lowered the PPV from 75.7% to 52.0% and halved the diagnostic OR (51 vs 25). Figure 2 demonstrates the progressive increase in sensitivity and NPV and the progressive decrease in specificity and PPV with addition of more bleeding codes. Performance attributes were also evaluated for all codes, irrespective of the criterion of causal attribution to the anticoagulant; the resultant PPV was 85.6% (95% CI, 82.5%–88.3%) and adjusted sensitivity was 60.0% (95% CI, 54.6%–66.4%).

TABLE 5.

Performance attributes of various code sets for identifying bleeding among patients prescribed anticoagulants^a

Code Set	Number of Codes in Set	Number of Cases Where Codes were Used	PPV, 95% CI	Adjusted Sensitivity, 95% CI	Diagnostic OR
All study codesb	66	583	74.8% (71.1%–78.3%)	69.8% (62.9%–77.6%)	48
Optimized code set (optimizing PPV and NPV)c	57	576	75.7% (72.0%–79.1%)	70.0% (63.2%–77.7%)	51
All study codes, combined with addition of adverse effect/poisoning codesd	67	597	73.2% (69.5%–76.7%)	72.1% (65.5%–79.4%)	44
Optimized code set with addition of adverse effect/poisoning codesd	58	591	73.9% (70.2%–77.4%)	72.3% (65.7%–79.5%)	47
Optimized code set with addition of adverse effect/poisoning codes and coagulopathy codesd	63	880	52.0% (48.7%–55.4%)	95.4% (92.8%–97.7%)	25

Abbreviations: CI, confidence interval; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; PPV, positive predictive value; OR, odds ratio.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016). Cases were patients with at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period. Controls were patients prescribed anticoagulants who had acute care encounters with no ICD-10-CM codes indicative of bleeding during their anticoagulant exposure period.

^bThe initial number of codes identified for the study consisted of 206 codes, of which 66 codes were identified in administrative claims data. Refer to Table S1 for a list of all codes included in the study.

^cThe nine codes that are not included in the optimized code set are: S065X8A, S066X8A, S06358A, K51911, K5791, S064X9A, S066X3A, H05232, and I31.2. Refer to Table S1 for descriptions of all codes included in the study.

^dThe initial number of adverse effect/poisoning codes identified for the study consisted of five codes, of which one code was identified in administrative claims data, and five coagulopathy codes, of which all were identified in the administrative claims data. Refer to Table 4 for a list of all adverse effect/poisoning and coagulopathy codes included in the study.

FIGURE 2.

Change in overall performance of ICD-10-CM diagnosis codes for identifying bleeding among patients prescribed anticoagulants, with addition of each code evaluated.^a Abbreviations: ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification, NPV, negative predictive value, PPV, positive predictive value. ^aThe initial number of codes identified for the study consisted of 206 codes, of which 66 codes were identified in administrative claims data. Codes were ordered by the performance of each single code (PPV then NPV). The performance statistic value is a summary measure of code performance with subsequent addition of each code in list from best to worst performing. Refer to Table S1 for all codes included in the study and the order in which they were added [Colour figure can be viewed at wileyonlinelibrary.com]

There were 164 false positive cases identified, where ICD-10-CM coding indicated a bleed, but medical record review did not. Most false positive cases had ICD-10-CM codes for other bleeding (48.8% of all false positive cases) and for GI bleeding (43.3%; Table 6). In most (74.4%) false positive cases, discordance resulted from uncertain attribution of bleeding to the anticoagulant (eg, bleeding possibly associated with recent surgery or patient reported not using anticoagulant at the time of the bleeding event) and from lack of objective evidence to confirm the bleeding event (eg, initial ED visit work-up included a diagnosis of melena, but subsequent fecal occult blood tests or endoscopy tests were negative). There were 34 false negative cases, where an ICD-10-CM code for any bleed type was absent, but medical record review indicated a bleed had occurred. Among these false negative cases, adverse effect, poisoning, and coagulopathy codes other than Z7901 (“long term [current] use of anticoagulants”) appeared in only 4 (11.8%) of 34 false negative cases in the absence of any bleeding code.

TABLE 6.

Reasons for discordance between ICD-10-CM diagnosis codes and gold standard medical record review for identifying bleeding among patients prescribed anticoagulants, by type of bleed and reason for discordance^a

	Number (%) of Casesb
Reason for Discordance	Intracranial Bleeding (N = 13)	GI Bleeding (N = 71)	Other Bleeding (N = 80)	Selected Examples
Attribution of bleeding to anticoagulant could not be ascertained	11 (84.6%)	11 (15.5%)	35 (43.8%)	• Patient involved in motor vehicle accident admitted for intracranial bleeding. Unable to ascertain attribution, bleeding could have been related to recent accident. • Patient with recent history of genitourinary surgery admitted for blood in urine. INR sub-therapeutic. Unable to ascertain attribution, bleeding could have been related to recent surgery. • Patient admitted for ischemic stroke and experienced hemorrhagic transformation. • Patient reported stopping the anticoagulant for a recent surgery.
Bleeding event could not be confirmed due to lack of supporting objective evidence or unclear documentation	0 (0%)	40 (56.3%)	25 (31.3%)	• Patient reported blood in stool, guaiac test was negative, and no further testing was documented. • Patient admitted for upper GI bleeding, guaiac test was positive, but endoscopy was negative for bleeding. • Patient reported blood in emesis, but no further testing or documentation of bleeding in the medical record. • Hemoptysis was noted on admission, but patient was a transfer from another facility and there was no description of the previous stay in the medical record.
Bleeding event not present in the medical record	0 (0%)	11 (15.5%)	15 (18.8%)	• No clinician diagnosis or objective evidence of a bleeding event present-on-admission documented in the medical record.
Other	2 (15.4%)	9 (12.7%)	5 (6.3%)	• ICD-10-CM code for GI bleed and other bleed (hemoptysis) present on claims data, but reviewer only identified GI bleed in medical record. • ICD-10-CM code for other bleed (hemorrhage, unspecified), but reviewer identified the bleed as a GI bleed.

Abbreviations: GI, gastrointestinal; INR, International Normalized Ratio; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification.

^aPatients are Medicare fee-for-service beneficiaries with outpatient prescriptions for at least one anticoagulant during the study period (October 1, 2015 to September 30, 2016). Cases were patients with at least one acute care encounter (ie, ED visit, observation stay, or hospitalization) with one or more ICD-10-CM codes indicative of bleeding during the anticoagulant exposure period. Controls were patients prescribed anticoagulants who had acute care encounters with no ICD-10-CM codes indicative of bleeding during their anticoagulant exposure period. Only false positive cases are shown here; ie, cases where ICD-10-CM coding indicated the presence of a bleed, however, medical record review determination was in disagreement.

^bPercentages are calculated out of all false positive cases in that bleed type category. Refer to Table S1 for a list of codes pertaining to each type of bleed.

IRR was calculated for 29 unique medical records on bleed occurrence determination. The mean kappa across all pairs of reviewers was 0.76, indicating substantial agreement.

4 |. DISCUSSION

Among Medicare beneficiaries prescribed anticoagulants, 66 (32.0%) of the 206 ICD-10-CM codes indicative of bleeding were used in administrative claims for acute care encounters. Using medical record review of 1166 records with a pre-specified definition of bleeding as the gold standard reference for validation, 57 of those 66 codes resulted in optimal performance as reflected by a diagnostic OR of 51; this optimized code set reliably identified anticoagulant-related bleeding approximately 76% of the time with a sensitivity of 70%.

Our findings have important implications for using ICD-10-CM codes in administrative claims to identify anticoagulant-related bleeding. First, performance of individual codes varied widely, with codes for intracranial bleeding yielding the highest PPV and sensitivity, followed by codes for GI bleeding and other bleeding, suggesting that selection of diagnostic codes for identifying anticoagulant-related bleeding may need to be tailored based on the intended use. For example, studies requiring high PPV (ie, maximizing the probability that cases identified by diagnostic codes are truly anticoagulant-related bleeding) could identify cases using intracranial and GI bleeding codes in any position, but this approach would miss other types of clinically significant bleeding. Of the 206 ICD-10-CM codes identified by clinical experts for inclusion in the study, 140 codes (68%) were never used in the claims data sampled in this study and therefore could not be included in the set of validated codes. Given that these 140 codes have face validity, studies requiring high sensitivity (ie, maximizing the number of bleeding events identified) could include them as it is possible these codes will be present in larger cohorts of administrative claims. Utilizing a broad definition of the gold standard in which the criterion of causal attribution of the bleeding event was not used, yielded a higher PPV (86%), but lower sensitivity (60%).

Second, as identified in previous validation studies of diagnostic codes for bleeding and other conditions,^20,30–32 code performance was substantially impacted by code position. Bleeding codes in the primary position demonstrated high PPV (87%), but low sensitivity (36%). The sensitivity improved to 70% when secondary position codes were added, which suggests that although reliable identification of true positive anticoagulant-related bleeding cases relied heavily on primary position codes; both primary and secondary position codes are necessary to achieve adequate identification of bleeding events.

Third, we found that the sensitivity of adverse effect/poisoning codes to detect anticoagulant-related bleeding was poor. Only one of the five adverse effect/poisoning codes evaluated was used among cases and controls. This is important as previous studies that have relied solely on a similar subset of codes (“E” codes in ICD-9-CM) to characterize the burden of anticoagulant-related harm would have significantly underestimated the number of bleeding events.^14–17 In our study, reliance on adverse effect or poisoning codes alone would have missed approximately 93% of bleeding cases. With the exception of Z7901 (“long-term [current] use of anticoagulants”), coagulopathy codes demonstrated moderate PPV and low sensitivity and did not substantially improve the diagnostic OR of the original or optimized code set.

Comparison of our findings with those of other studies is limited in that previous studies utilized ICD-9-CM, evaluated only PPV or a very limited set of codes, or did not attribute bleeding to anticoagulant exposure.^{18,20–25,33} However, similar to our study, moderately high to high PPV has been demonstrated for certain bleeding codes.^20–22,24 A preliminary validation study similar to this one in an all-payor population yielded comparable results,³⁴ suggesting that the results of this study among Medicare beneficiaries may be generalizable to other populations.

Use of ICD-10-CM took effect in October 2015 in the United States, thus accuracy of ICD-10-CM codes may evolve. Establishing how codes are presently being used is necessary to allow for monitoring of trends.^35–37 There were 226 cases (15.0%) among the initial 1500 patients eligible for the study that were not available for review; however, because cases were randomly assigned to reviewers, we would not expect this to bias the study findings. Given the large study sample size and enrichment of the study cohort for bleeding cases, we were able to quantify the accuracy of common codes for bleeding; however, some codes did not appear in claims data and thus could be not be evaluated. We did not evaluate the contribution of procedure codes to the performance of the code sets. Procedure codes were available only for hospitalizations and were not associated with a present-on-admission indicator, making it challenging to identify if the procedure was for an admitting diagnosis or a complication of hospitalization. Lastly, our study was limited to clinically significant bleeding events and may not represent the full spectrum of anticoagulant-related harm (eg, changes in laboratory coagulation parameters or minor bleeding events such as contusions or lacerations). By limiting to cases that were most likely caused or exacerbated by anticoagulants, we also may have excluded types of bleeds worsened by anticoagulation (ie, perioperative events and head injuries).

This assessment of ICD-10-CM code performance for identifying bleeding among patients prescribed anticoagulants can help optimize identification of an important, common, and clinically significant consequence of medication-related harm for drug safety research and quality improvement.

Supplementary Material

SUPPORTING INFORMATION

Additional supporting information may be found online in the Supporting Information section at the end of the article.

KEY POINTS.

• In a multicenter, chart validation study among Medicare beneficiaries performance of ICD-10-CM code assignments for identifying cases of outpatient anticoagulant-related bleeding among Medicare beneficiaries varied by type of bleed and code position.

• Codes for adverse effects/poisoning were uncommon and had poor sensitivities. Approximately, 93% of anticoagulant-related bleeding events would have been missed if only adverse effect/poisoning codes were used.

• A code set optimizing PPV and negative predictive value (NPV) was identified, consisting of 57 bleeding codes (diagnostic odds ratio, 51, PPV, 76%, adjusted sensitivity, 70%).