Validation of an algorithm to identify fractures among patients within the Veterans Health Administration

Abstract

Objective:

To validate an algorithm that identifies fractures using billing codes from the International Classification of Diseases Ninth Revision (ICD-9) and Tenth Revision (ICD-10) for inpatient, outpatient, and emergency department visits in a population of patients.

Methods:

We identified and reviewed a random sample of 543 encounters for adults receiving care within a single Veterans Health Administration healthcare system and had a first fracture episode between 2010 and 2019. To determine if an encounter represented a true incident fracture, we performed chart abstraction and assessed the type of fracture and mechanism. We calculated the positive predictive value (PPV) for the overall algorithm and each component diagnosis code along with 95% confidence intervals. Inverse probabilities of selection sampling weights were used to reflect the underlying study population.

Results:

The algorithm had an initial PPV of 73.5% (Confidence interval [CI] 69.5, 77.1), with low performance when weighted to reflect the full population [PPV 66.3% (CI 58.8, 73.1)]. The modified algorithm was restricted to diagnosis codes with PPVs >50% and outpatient codes were restricted to the first outpatient position, with the exception of one high performing code. The resulting unweighted PPV improved to 90.1% (CI 86.2, 93.0) and weighted PPV of 91.3% (CI 86.8, 94.4). A confirmation sample demonstrated verified performance with PPV of 87.3% (76.0, 93.7). PPVs by location of care (inpatient, emergency department and outpatient) remained greater than 85% in the modified algorithm.

Conclusions:

The modified algorithm, which included primary billing codes for inpatient, outpatient, and emergency department visits, demonstrated excellent PPV for identification of fractures among a cohort of patients within the Veterans Health Administration system.

Introduction

The use of medical claims data for observational research is well described.^1,2 Algorithms are necessary when extrapolating research data from clinical documentation.³ Various algorithms are valid and were developed to identify fractures in medical administrative claims data. Ray et al used 1986 Medicare data to develop a novel algorithm for anatomic sites most frequently associated with osteoporotic fracture. ⁴ The algorithm performed best for hip fractures but notably did not capture vertebral compression fractures, which is another common fracture secondary to bone loss.^4,5 A new algorithm utilizing the REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort included vertebral and recurrent fractures of the same site.⁶

Neither of these prior algorithms have been validated within the Veterans Healthcare Administration (VHA) which is the largest healthcare provider and often provides fracture care for patients after an acute fracture event or surgical repair occurs in a community-based hospital. Furthermore, neither of the prior two algorithms included validation of International Classification of Diseases (ICD) 10 codes which include fracture laterality and timing with separate codes for “initial” versus “subsequent” visits.

In this study, our aim was to understand and validate the accuracy of an expanded fracture algorithm which included all ICD 9 and 10 fracture codes and to evaluate the algorithm performance stratified by location of care (inpatient, outpatient, emergency department) and by ICD 9 versus 10 code performance.

Methods

Study Design and Population

This was a validation study of a fracture algorithm applied to a cohort of veterans receiving care within the VHA. The study was approved by the VHA - Tennessee Valley Healthcare System (TVHS) Institutional Review Board. We used existing data and a waiver of informed consent. The underlying study population was a national observational cohort of veterans who were aged 18 years and older and regular users of VHA care. The sample for this study were patients who utilized VHA TVHS between 10/1/2000 through 09/30/2020 and were identified as having a fracture event (below) between 2010 and 2019.

Fracture Events

The algorithm was adapted from two separate validated fracture detection algorithms. The first, developed by Ray et al., used ICD 9 codes. The second, developed by Wright et al., expanded the Ray algorithm to include vertebral compression fractures and recurrent fracture events.^2,4

Each case event was identified based on fracture International Classification of Diseases Ninth Revision (ICD-9) and Tenth Revision (ICD-10) codes in the first two positions from inpatient or outpatient claims. Fracture locations included: skull/facial, cervical/neck, rib/sternum/thoracic, lumbar/pelvic shoulder/arm, forearm, wrist/hand, femur, leg/ankle, foot/toe. ICD-9 and ICD-10 codes were used in combination such that the earliest event was detected if there were multiple codes for a fracture (Appendix Figure 1). We required a new fracture by excluding patient visits if there was the same fracture code within the VHA in the two years prior. A fracture event was identified by looking for the fracture codes in the first two positions.

Events were then grouped into 6 strata for chart abstraction: 1) Inpatient event, diagnosis code in primary position 2) Inpatient event, diagnosis code in secondary position 3) Emergency Department event, diagnosis code in primary position 4) Emergency Department event, diagnosis code in secondary position 5) Outpatient event, diagnosis code in primary position 6) Outpatient event, diagnosis code in secondary position. An emergency department (ED) event was identified by the additional presence of CPT codes 99281, 99282, 99283, 99284, and 99285 indicating emergency care.

Data Collection and Chart review

We systematically selected a sample of events for chart review and data abstraction stratifying by year and ICD-9 or ICD-10 codes. If an individual had multiple fracture events documented during the study period, the first date of a diagnosis code for a facture was selected for review and was noted to be the index VHA event date (Appendix Figure 1). Using a secure VA REDCap instrument (Appendix), clinical data was abstracted on fracture events from the VA Computerized Patient Record System (CPRS), VISTA, Joint Legacy Viewer (JLV) and all scanned documentation of care provided outside of the VHA as care in the community. This abstraction was performed by two physicians (TH and TR), who were blinded to the fracture ICD 9 or 10 codes. To ensure data accuracy and consistency with abstraction, each reviewer independently reviewed a 5% sample of records. Overall agreement between reviewers was excellent (kappa > 85%).

The presence or absence of fracture symptoms, signs, and radiologic features including pathologic fracture and fall history were abstracted from documentation. Data reviewed included the history and physical, outpatient notes, specialty consultation, and radiology reports from the index event date of the fracture and up to 90 days prior. This time constraint was used to understand signs or symptoms of fracture associated with the initial event.

An event was determined to be a true fracture when considering mechanism of injury, location, and radiologic features. Fractures were categorized as true fractures episodes if the timing was unknown or occurred within the 90 day look back time frame prior to the index event date and there was a radiologic confirmation of fracture.

Statistical Analysis

Descriptive statistics were used to characterize the study sample. Using the chart review as the reference standard, we calculated the positive predictive value (PPV—true positive fracture events divided by algorithm-positive fracture events detected). The PPV was calculated for the overall algorithm and for the 6 strata. Positive predictive values were then calculated separately for each distinct ICD-9 or ICD 10 code. We modified the algorithm to remove individual ICD 9 or 10 codes where the accuracy was suboptimal (PPV ≤ 50%) both overall and for the diagnosis position. We then revalidated the new algorithm in a confirmation sample of 60 patient charts to verify performance in a second convenience sample.

To more accurately reflect the underlying study population from which the sample for review was selected, we calculated the PPV using inverse probability of selection sampling weights. To create 95% CIs, a weighted Poisson Regression model was used to calculate standard errors. Chart review classifications were treated as statistically independent. A sensitivity analysis evaluated the algorithm performance to those with a prior diagnosis of diabetes. Statistical analyses were performed using R Statistical Software.⁷

Results

Chart selection and Patient Characteristics

There were 4682 eligible fracture events from 3451 patients in TVHS between 2010 and 2019. A sample of 543 fracture events were selected. From the sample, 22 events were excluded for patients seen outside of TVHS for most care or there were insufficient data for fracture determination. Review of 521 charts were amenable to determination of a fracture event. The mean patient age in the sample was 65.8 years (standard deviation [SD] 11.3). Patients were overwhelmingly male (93.1%); 82.0% were White and 14.6% were Black. Ninety percent of patients had a diagnosis of type 2 diabetes at the time of index fracture event reflecting the underlying cohort of patients with diabetes. Forty percent of events were minimally traumatic or atraumatic while 22% were considered traumatic events including falls from a height and motor vehicle accidents. The location of the majority of the patient’s care for the fracture episode is also noted in Table 1.

Table 1:

Characteristics of sampled patients from Electronic Medical Record

Characteristic	N=521
Age in years, Mean (Standard deviation [SD])*	65.8 (11.3)
Age groups, n (%)*
< 55 years old	67 (12.9)
55 - 64 years old	171 (32.8)
65 - 74 years old	182 (34.9)
≥ 75 years old	99 (19.0)
Sex, Male n (%)	485 (93.0)
Race, n (%)
White	427 (81.9)
Black	76 (14.6)
Other†	18 (3.5)
Type 2 Diabetes, n (%)	469 (90.0)
Discharged Setting of Medical Care‡ n(%)
Outpatient	216 (41.4)
Emergency Department	200 (38.4)
Hospitalization/ Hospital transfer	88 (16.9)
Uncertain/ Not available	17 (3.3)
Primary reason for visit n(%)
Fracture follow-up care	68 (13.1)
Suspected fracture or confirmed new fracture	197 (37.8)
Injury	136 (26.1)
Other (visit with a pain complaint)	98 (18.8)
Unknown	22 (4.2)
Mechanism n (%)
Minimally traumatic, atraumatic, fall from same level (slip or trip), or overuse injury	212 (40.7)
Traumatic fall from mid height: bed, chair, toilet	12 (2.3)
Traumatic fall from great height: major trauma, car accident	116 (22.3)
Pathologic fracture	16 (3.0)
Unknown mechanism or circumstances (patient found down)	27 (5.2)
No fracture	138 (26.5)
Distribution of Codes
ICD 9	338 (64.9)
ICD 10	183 (35.1)

^*age not available in 2 charts

^†Race other includes patients who self report their race as unknown (n=12); American Indian (n=1); Hawaiian Pacific Islander (n=1) White non Hispanic (n=1) and Hispanic ethnicity (n=3)

^‡Includes patients who received majority of care in this setting. For example a patient seen in ED but admitted to hospital would be included in the Hospitalization group even if the first code for fracture validation appeared in the Emergency room

Positive Predictive Value of Fracture Algorithm

Of 521 fracture events reviewed, 383 fulfilled criteria for a fracture (radiologic confirmation) yielding a PPV of 73.5% (69.5, 77.1). The PPV varied by location and code position (Table 2 top). The highest PPV occurred for inpatient codes in the secondary position (100%) followed by emergency department codes in the primary and secondary position, with a PPV of 87.2% (79.7, 92.2) and 88% (75.2, 94.7), respectively. The PPV of outpatient codes was low with a PPV of 74.9% (67.6, 80.9) for codes in the primary position and 57.5% (49.9, 64.7) for the secondary position. In the weighted analysis that reflects the performance of the algorithm in the underlying study population, the overall weighted PPV for fractures was 66.3% (58.8, 73.1). Stratification of the algorithm by ICD 9 or 10 in the original and modified algorithm demonstrates a consistent performance of both ICD9 and 10 codes.

Table 2:

The unweighted and weighted positive predictive values (PPV) and 95% Confidence intervals (95% CI) for true fracture events. The unweighted PPV are stratified by location of care code position; and ICD code.

Original Algorithm
	N charts detected by algorithm	True Fracture Episode	PPV (95% CI)*
Unweighted PPV	521	383	73.5 (69.5,77.1)
Weighted PPV	521	345	66.3 (58.8,73.1)
Location of Care and Code Position
Inpatient Primary position	5	4	80.0 (11.1,99.2)
Inpatient Second position	8	8	100.0 (100.0,100.0)
Emergency Department Primary position	117	102	87.2 (79.7,92.2)
Emergency Department Second position	50	44	88.0 (75.2,94.7)
Outpatient Primary position	167	125	74.9 (67.6,80.9)
Outpatient Second position	174	100	57.5 (49.9,64.7)
ICD code
ICD9 Unweighted PPV	338	255	75.4 (70.6, 79.8)
ICD9 Weighted PPV	338	205	60.4 (51.2, 69.4)
ICD10 Unweighted PPV	183	128	70.0 (62.9, 76.2)
ICD10 Weighted PPV	183	146	79.9 (72.4, 85.8)
Modified Algorithm after removal of select codes
Unweighted PPV	313	282	90.1 (86.2,93.0)
Weighted PPV	313	286	91.3 (86.87,94.4)
Location of Care and Code Position
Inpatient Primary position	4	4	100 (100, 100)
Inpatient Second position	8	8	100 (100, 100)
Emergency Department Primary position	106	100	94.3 (87.8,97.5)
Emergency Department Second position	46	41	89.1 (75.8,95.6)
Outpatient Primary position	138	118	85.5 (78.5,90.5)
Outpatient Second position	11	11	100 (100,100)
ICD code
ICD9 Unweighted PPV	208	186	89.4 (84.4, 93.0)
ICD9 Weighted PPV	208	190	91.1 (85.0, 94.9)
ICD10 Unweighted PPV	105	96	91.4 (84.2, 95.5)
ICD10 Weighted PPV	105	96	91.7 (83.1, 96.2)

^*Confidence intervals calculated using Poisson regression

We evaluated the accuracy of specific ICD-9 or 10 codes in the algorithm and calculated the PPV for each ICD-9 or 10 diagnosis code (Table 3). There was significant variability in the performance of each ICD code. The following ICD-9 codes had a PPV greater than 90%: 804, 807, 810, 812, 813, 816, 817, 826, and E887; ICD-10 codes with the highest PPV were M484, M80, S12, S22, S52, S72, and S82. The ICD-9 codes with the lowest PPV were 733.9, 801, 819, and 827; the ICD-10 codes with the lowest PPV were S49, S59, S79, S89, and S99.

Table 3:

The unweighted positive predictive value (PPV) for individual ICD 9 and ICD10 fracture codes Highlighted ICD codes were removed in modified algorithm

Code	Diagnosis	N charts by algorithm	True Fracture Episode	PPV (95% CI)*
ICD 9
733.1	Pathologic Fracture	14	12	86 (59,97)
733.8	Malunion and nonunion of fracture	16	2	12 (2,38)
801	Fracture of base of skull.	2	1	50 (10,90)
802	Fracture of face bones	17	14	82 (58,94)
804	Multiple fractures involving skull or face with other bones	1	1	100 (17,100)
805	Fracture of vertebral column without mention of spinal cord injury	12	9	75 (46,91)
806	Fracture of vertebral column with spinal cord injury	13	7	54 (29,77)
807	Fracture of rib(s) sternum larynx and trachea	15	14	93 (68,100)
808	Fracture of pelvis	16	9	56 (33,77)
810	Fracture of clavicle	16	15	94 (69,100)
811	Fracture of scapula	12	9	75 (46,91)
812	Fracture of humerus	15	14	93 (68,100)
813	Fracture of radius and ulna	16	15	94 (69,100)
814	Fracture of carpal bone(s)	16	12	75 (50,90)
815	Fracture of metacarpal bone(s)	13	11	85 (56,97)
816	Fracture of one or more phalanges of hand	16	15	94 (69,100)
817	Multiple fractures of hand bones	1	1	100 (17,100)
819	Multiple fractures involving both upper limbs and upper limb with rib(s) and sternum	1	0	0 (0,83)
820	Fracture of neck of femur	15	9	60 (36,80)
821	Fracture of other and unspecified parts of femur	13	11	85 (56,97)
822	Fracture of patella	14	10	71 (45,88)
823	Fracture of tibia and fibula	14	9	64 (39,84)
824	Fracture of ankle	12	8	67 (39,86)
825	Fracture of one or more tarsal and metatarsal bones	17	12	71 (47,87)
826	Fracture of one or more phalanges of foot	13	12	92 (64,100)
827	Other multiple and ill-defined fractures of lower limb	2	1	50 (10,90)
829	Fracture of unspecified bones	15	11	73 (48,89)
E887**	Fracture, cause unspecified.	11	11	100 (69,100)
ICD 10
M484	Fatigue fracture of vertebra	3	3	100 (38,100)
M80	Osteoporosis with pathological fracture	5	5	100 (51,100)
M84	Disorder of continuity of bone	12	8	67 (39,86)
S02	Fracture of skull and facial bones	9	6	67 (35,88)
S12	Fracture of cervical vertebra and other parts of neck	10	9	90 (57,100)
S22	Fracture of rib(s), sternum and thoracic spine	15	14	93 (68,100)
S32	Fracture of lumbar spine and pelvis	14	10	71 (45,88)
S42	Fracture of shoulder and upper arm	14	11	79 (52,93)
S49	Other and unspecified injuries of shoulder and upper arm	9	2	22 (6,56)
S52	Fracture of forearm	12	11	92 (62,100)
S59	Other and unspecified injuries of elbow and forearm	11	5	45 (21,72)
S62	Fracture at wrist and hand level	12	9	75 (46,91)
S72	Fracture of femur	12	11	92 (62,100)
S79	Other and unspecified injuries of hip and thigh	2	0	0 (0,71)
S82	Fracture of lower leg, including ankle	10	10	100 (67,100)
S89	Other and unspecified injuries of lower leg	10	3	30 (11,61)
S92	Fracture of foot and toe, except ankle	10	8	80 (48,95)
S99	Other and unspecified injuries of ankle and foot	13	3	23 (8,51)

^*Wilson confidence intervals were used for the positive predictive values confidence intervals.

^**Although high PPV all were in Outpatient second position, this code was selectively added back to modified algorithm

Accuracy of the Modified Fracture Algorithm

The modified algorithm removed individual ICD 9 and ICD 10 all codes with a PPV less than or equal to 50%. We restricted the use of ICD 9 code 808 (fracture of pelvis) and ICD 10 code S32 (fracture of lumbar spine and pelvis) to inpatient or ED location. All outpatient ICD 9 or 10 codes in the secondary position without a preceding ED visit, were also removed except ICD 9 code E887 (Fracture, cause unspecified), which remained as part of the modified algorithm. In this modified algorithm, the overall unweighted PPV was 90.1% (86.2, 93.0); the overall weighted PPV was 91.3% (86.8, 94.4) with no difference in performance when stratified by ICD 9 or 10 codes. The PPV for the modified algorithm varied by location and code position (Table 2 bottom). The highest PPVs occurred for inpatient codes (100%). The emergency department codes in the primary and secondary position also had high PPV, with a PPV of 94.3% (87.8, 97.5) and 89.1 (75.8, 95.6), respectively. The PPV of outpatient care in the primary position was the lowest with a PPV of 85.5% (78.5.6, 90.5). The only outpatient code allowed in the secondary position, E887, had a PPV of 100%.

In a sensitivity analysis that restricted to the subgroup of patients with diabetes (n=282), the modified algorithm performance demonstrated an unweighted PPV of 91.5 (87.6, 94.2) and weighted PPV of 93.5 (89.7, 95.9).

In the confirmation sample of 60 patient charts, 5 charts were excluded for insufficient information in the medical record to determine if a fracture was present; thus 55 were reviewed to determine presence or absence of fracture and 48 of which had a true fracture event PPV of 87.3% (76.0, 93.7).

Nationally the current algorithm detected 46,295 cases while the original algorithm detected 53,478 cases (7,183 more). This decrease of 13% of cases decreased sensitivity but yielded an improved specificity.

Discussion

This study validated an algorithm to identify patients with a fracture seeking care in VHA. VHA is a system that often provides aftercare for emergent events where the index episode of care is provided in the community. We were able to modify existing algorithms and expand upon the prior work of Ray and Wright to validate ICD10 codes and care provided in multiple clinical locations. We found that we could improve the accuracy and PPV by removal of low performing codes and restricting to the first outpatient position for all outpatient events (except for code E887). Through these modifications we improved the weighted PPV from 66.3% to 91.3%. The inclusion of outpatient codes also increases the capture of events where care episodes were begun in the community and continued to be provided by the VHA.

This algorithm demonstrate that specific codes such as hip fractures had very high PPV in Ray and Wright, but were lower in this study. Hip fractures are often emergent, and Veterans will be transported to the nearest hospital rather than a VHA hospital. The episodes we detected often were outpatient and occurred after the event and significant treatment such as surgical repair occurred. Often this care may be paid for by Medicare or paid for by VHA. For the chart review and validation, we can consistently evaluate chart information for those patients once they are seen within the VHA. Information on care received outside of the VA within the community care paid for by VHA or Medicare may or may not be available within each chart. The algorithm does differ from the Wright algorithm in that we were able to achieve relatively high PPV through the application of only ICD9 and 10 codes without the addition of fracture repair codes and spine imaging codes. The use of only ICD9 and 10 codes in different settings of care can be a simplified method for fracture detection particularly within the VHA with very good accuracy.

We also demonstrate high accuracy in our sensitivity analysis which restricts the algorithm to patients with diabetes. We found that the algorithm performance had a weighted PPV of 93.5 (89.7, 95.9). No other algorithm evaluated performance restricted to patients who are at higher fracture risk due to diabetic bone disease. This validation serves as a methodologic foundation for future studies which will enable researchers to utilize this algorithm for assessment of fracture outcomes among a cohort of patients with diabetes who are at risk for metabolic bone complications of diabetes. The importance of a high PPV is crucial for such studies. This algorithm’s high PPV ensures that this algorithm can detect true fracture events accurately when conducting observational research.

Our study does have several limitations. Data abstraction by chart review may be subject to error due to missing information or differences in interpretation.⁸ We utilized a standardized chart abstraction process and a case definition. Both reviewers independently reviewed a 5% sample of records allowing for more consistency. Overall agreement among reviewers was excellent (kappa > 85%), however each chart abstractor is dependent upon information that is documented in the electronic health record. If data was missing due to provision of fracture care outside of the VHA healthcare system, then it could not be abstracted. It should be noted that this location of care often did not align with the index ICD9 or 10 code location for the event. For example, a patient may have presented to the emergency department with a suspected or known fracture (index code detected), but the majority of the episode of care reviewed was inpatient. Furthermore, there is no universal case definition for a fracture event therefore clinical judgement must be used when interpreting the data and applying our case definition. Additionally, this algorithm was designed to be highly specific in capturing fracture events that were likely the primary reason for care (in the first or second position). Patients with numerous medical issues who had a fracture coded after the second position would be missed (lower sensitivity). The modified algorithm demonstrated a decrease of 13% in the overall number of cases but improved specificity. The algorithm required that the index fracture code not be present within the VHA for 2 years prior, which could miss an event that was a reinjury and repeat fracture of the same bone/ site. We did not select a random sample of fracture negative patients for review. Therefore, the negative predictive value (NPV) of the algorithm cannot be ascertained. Finally, this study was limited to a sample of Veterans receiving care within a single VHA healthcare system and the sample was predominantly older, white men. This may limit the generalizability of the study findings to other settings and there may be variation in the PPV by subgroups and different locations.

In conclusion, this study validated a modified algorithm for identifying fractures among a cohort of patients in the VHA. The algorithm demonstrated excellent PPV for identification of fractures using both ICD9 and 10 codes and in multiple settings of care, inpatient, emergency department and outpatient.

Key Points.

• The use of claims data for medical research is well described; various algorithms have been developed to identify clinical outcomes based on billing data

• We modified a fracture algorithm to use within the Veterans Health Administration system which is an integrated system that often sees patients after the acute fracture event may have occurred. We also expanded the algorithm to incorporate use of ICD 10 codes.

• Among a cohort of veterans, a novel algorithm to identify fractures using ICD 9 and 10 codes in a variety of settings including the inpatient, emergency department and outpatient had a positive predictive value (PPV) of 90.1% (CI 86.2, 93.0)

Plain Language Summary.

This study sought to identify fracture outcomes among Veterans using billing codes from the medical record. Researchers selected a random sample of 543 encounters identified as having a fracture based on the billing code at a single Veterans healthcare system. Researchers reviewed these medical records to assess the fracture type and mechanism to determine the accuracy for the algorithm. The algorithm was modified to reach a positive predictive value of 90%. This algorithm will inform future research identifying fractures among a large cohort of patients with the Veterans Health Administration System.

Abbreviations:

PPV

Positive Predictive Value

T2DM

Type 2 Diabetes Mellitus

TVHS

Tennessee Valley Healthcare System

VHA

Veterans Healthcare Administration