Overcoming barriers to population-based injury research: development and validation of an ICD-10–to–AIS algorithm

Abstract

Background

Hospital administrative databases are a useful source of population-level data on injured patients; however, these databases use the International Classification of Diseases (ICD) system, which does not provide a direct means of estimating injury severity. We created and validated a crosswalk to derive Abbreviated Injury Scale (AIS) scores from injury-related diagnostic codes in the tenth revision of the ICD (ICD-10).

Methods

We assessed the validity of the crosswalk using data from the Ontario Trauma Registry Comprehensive Data Set (OTR-CDS). The AIS and Injury Severity Scores (ISS) derived using the algorithm were compared with those assigned by expert abstractors. We evaluated the ability of the algorithm to identify patients with AIS scores of 3 or greater. We used κ and intraclass correlation coefficients (ICC) as measures of concordance.

Results

In total, 10 431 patients were identified in the OTR-CDS. The algorithm accurately identified patients with at least 1 AIS score of 3 or greater (κ 0.65), as well as patients with a head AIS score of 3 or greater (κ 0.78). Mapped and abstracted ISS were similar; ICC across the entire cohort was 0.83 (95% confidence interval 0.81–0.84), indicating good agreement. When comparing mapped and abstracted ISS, the difference between scores was 10 or less in 87% of patients. Concordance between mapped and abstracted ISS was similar across strata of age, mechanism of injury and mortality.

Conclusion

Our ICD-10–to–AIS algorithm produces reliable estimates of injury severity from data available in administrative databases. This algorithm can facilitate the use of administrative data for population-based injury research in jurisdictions using ICD-10.

In the context of injury research, a reliable and accurate means of estimating injury severity is essential to meaningful evaluations of patient outcomes, intervention effectiveness or hospital performance. The Injury Severity Score (ISS) and the Abbreviated Injury Scale (AIS) from which it is derived have been the most widely used approach to injury severity scoring.^1–3 Abbreviated Injury Scale coding is standardized, with oversight and education provided by the Association for the Advancement of Automotive Medicine (AAAM). Its dependent score, the ISS, has been shown to be closely associated with risk of death following injury.³ These measures of injury severity are most frequently captured by experienced trauma registrars and thus are typically only available in trauma centres. This limited availability poses major challenges in any population-based analyses where only administrative discharge data are available.

Hospital administrative databases have been identified as a potentially useful source of population-level data on injured patients. However, rather than relying on injury-specific injury scores these databases generally use the International Classification of Diseases (ICD) system, published by the World Health Organization (WHO), to index hospital records. Although widely available, ICD diagnostic codes do not provide a direct means of estimating injury severity.

Recognizing the usefulness of administrative data sources for the purposes of injury-related research, an automated algorithm to convert ICD diagnostic codes to AIS scores has previously been developed for the clinical modification of the ninth revision of the ICD (ICD-9-CM).⁴ This algorithm has been validated and has been of demonstrable utility in a variety of settings focusing on population-based trauma care.^5–8

In 1990, the development of the tenth revision of the ICD (ICD-10) was endorsed; this modification substantially broadened the scope of the classification system.⁹ The ICD-10 includes expanded injury codes and greater specificity in code assignment.¹⁰ Versions of the ICD-10 have been adopted in a number of jurisdictions, including Australia, the United Kingdom, Germany and Canada. Given the substantial modifications in coding in ICD-10 compared with ICD-9, we developed an ICD-10–to–AIS algorithm to convert ICD-10 codes to AIS scores, and we report on the validation of this algorithm.

Methods

The development of the ICD-10–to–AIS algorithm comprised 2 components: algorithm development and validation using data from the Ontario Trauma Registry. The final algorithm is an automated computer algorithm. This study was approved by the Research Ethics Board of St. Michael’s Hospital. Because data used for this study were derived from administrative databases, waiver of consent was sought and obtained from the Research Ethics Board of St. Michael’s Hospital.

Data sources

We tested the validity of the algorithm using the Ontario Trauma Registry Comprehensive Data Set (OTR-CDS, admission years 2002–2004). The OTR-CDS includes patients managed at a designated trauma centre with moderate to severe injuries (ISS ≥ 12) in the province of Ontario, Canada. The OTR-CDS includes AIS scores and ISS calculated by expert abstractors by means of chart abstraction. The data set also includes ICD-10 injury diagnostic codes and external cause of injury codes for each patient.

Crosswalk development

In consultation with a member of the AIS faculty of the AAAM and a certified AIS specialist, each injury-related diagnostic code in the ICD-10 lexicon (codes ranging from S00 to T79.0) was assigned to 1 of 9 AIS body regions and 1 of 6 ISS body regions. Each ICD-10 diagnostic code was also assigned to an appropriate AIS severity code based on the 1998 update of the AIS system.¹ The ICD-10 diagnostic codes related to foreign bodies (T15–T19), burns and corrosion injury (T20–T32), poisoning (T36–T65) and environmental exposure (T33–T35, T66–78) were excluded. Complications of medical care and late effects of injury (T80–T98) were also excluded. We used the Canadian Modification (ICD-10-CA)¹¹ to define AIS codes. There are no significant differences between ICD-10-CA and other versions of the ICD-10 in the relevant injury code ranges.

Each ICD-10 diagnosis was assigned the lowest or most conservative AIS score with which it could be accurately associated. In cases where an ICD-10-CA diagnostic code described injuries to more than 1 organ, each with a different AIS score but within the same ISS body region, the lowest AIS score was assigned. The ICD-10 diagnoses that could not be assigned an accurate AIS score were assigned an AIS score of 9 and were not included in ISS calculation. Specifically, an AIS score of 9 was assigned when an ICD-10-CA diagnostic code described injuries to multiple organs in different ISS body regions, injuries to an unidentified body region or multiple injuries of more than 1 type to a body region (e.g., S09.7 — multiple injuries of head).

Algorithm validation

Injury mechanism was assigned to an external cause of injury code (E-code) according to the ICD-10 injury mortality diagnosis matrix.¹² Patients with a primary mechanism of injury of burn, poisoning, drowning, exposure, suffocation, overexertion, hanging or submersion were excluded from the cohort. Patients with missing E-codes were also excluded. Using the algorithm, we derived AIS and ISS scores from the ICD-10 codes in the OTR-CDS.

Validity of the algorithm was assessed in 2 ways. First we determined the extent of agreement between the AIS scores in the OTR-CDS (considered the gold standard) with those derived from the algorithm. In addition, we determined the agreement between the ISS derived from the 2 methods. Concordance between maximum AIS score (AIS_max) by body region obtained from the algorithm and from registry data was measured as a percentage of exact agreement. Patient data were further stratified by age group (< 15 yr and ≥ 15 yr) and injury mechanism (blunt, penetrating) to assess algorithm performance across patient groups. Concordance across mapped and abstracted scores in detecting severe injury (AIS score ≥ 3) by body region was measured using exact agreement and the κ coefficient.

We analyzed the degree of concordance between mapped and abstracted ISS using 2 approaches. First, we compared the absolute difference between the 2 scores to estimate the percentage of patients for whom the difference was 10 or less. We also evaluated concordance using the intraclass correlation coefficient (ICC). Ranging from 0 to 1.0, ICC accounts for 2 sources of variation in ISS scores: variations in scores across different patients and variations owing to a lack of concordance across methods (algorithm and abstraction).¹³ A high ICC suggests high concordance between methods. The 95% confidence interval (CI) for ICC was computed using bootstrapping methodology.¹⁴ In the estimation of ICC, we allowed a margin of error of more or less than 3 for ISS, given the known inter-rater reliability of chart abstractors in assignment of an ISS.¹⁵ Analyses were also stratified by age group (< 15 yr and ≥ 15 yr), injury mechanism (blunt, penetrating) and survival status (in-hospital death, alive at discharge).

Statistical analysis

We calculated descriptive statistics for demographic and injury data in the OTR-CDS. Means and standard deviations or medians and interquartile ranges were calculated for continuous variables. We measured absolute and relative frequencies for discrete variables. In all statistical analyses, we considered results to be significant at p < 0.05. Data were analyzed using SAS software, version 9.1.

Results

Algorithm development

We included a total of 1542 ICD-10 diagnostic codes in the range S00–T79.0. Twenty-nine ICD-10 diagnostic codes (1.8%) could not be assigned an AIS body region. An additional 196 could not be assigned an AIS score and therefore received a score of 9 and were not included in ISS calculation. Of these, 27% were in the abdominal region and 26% were in the lower extremities. Overall, 85% of ICD-10 diagnoses were assigned an AIS body region and severity.

Validation of the ICD-10–to–AIS algorithm

There were 10 431 patients who met our inclusion criteria who were listed in the OTR-CDS in 2002–2004 (Table 1). Most patients (71.6%) were men, and the mean age of patients was 44 years. The most frequent mechanisms of injury were motor vehicle collisions (51.6%) and falls (34.3%). Crude in-hospital mortality in this cohort was 10.8%. Overall, 20 patients had ICD-10 diagnoses that could not be translated into an ISS.

Table 1.

Characteristics of patients in the Ontario Trauma Registry, Comprehensive Data Set

Characteristic	No. (%)*
No.	10 431
Male sex	7469 (71.6)
Age, mean (SD) yr	43.7 (23.2)
Mechanism of injury
Blunt
Falls	3574 (34.3)
MVC	5388 (51.6)
Other blunt	776 (7.4)
Other	193 (1.8)
Penetrating	500 (4.8)
Stab/impalement	289 (2.8)
Firearm	211 (2.0)
Injury severity
ISS, median (IQR) [range]	22 (16–27) [12–75]
Mortality	1129 (10.8)

IQR = interquartile range; ISS = Injury Severity Score;² MVC = motor vehicle collision; SD = standard deviation.

^*Unless indicated otherwise.

Concordance of algorithm and abstracted AIS scores

To provide a greater understanding of the strengths and weakness of the ICD-10–to–AIS algorithm, we evaluated the concordance of AIS_max across all body regions, as well as the ability of the algorithm to identify the presence or absence of a severe injury (AIS score ≥ 3) in each body region. The AIS_max for each patient identified using the algorithm and from chart abstraction were in exact agreement in 57% of patients (Table 2). This degree of agreement is similar to that previously observed in studies of inter-rater reliability among AIS abstractors.¹⁵ The body regions with the highest proportion of agreement for AIS_max were the face (68%), neck (71%) and upper extremities (72%). When comparing AIS_max obtained from the algorithm and by abstraction, exact agreement was lowest for head injuries (52%), chest injuries (51%) and lower extremity injuries (51%). When patients were stratified by age and mechanism of injury, overall agreement between abstracted and algorithm AIS_max was lowest for patients with penetrating injuries (48%).

Table 2.

Percentage of exact agreement for maximum AIS score, by body region, derived from abstracted and mapped scores

Body region	Overall	Age < 15	Age ≥ 15	Blunt injuries	Penetrating injuries
All	57	59	57	58	48
Head	52	47	52	52	55
Face	68	74	67	68	58
Neck	71	89	70	74	66
Chest	51	44	52	53	31
Abdomen	62	74	61	65	39
Spine	58	59	58	58	60
Upper extremity	72	82	71	72	64
Lower extremity	51	59	50	50	71

AIS = Abbreviated Injury Scale.¹

Because, in the context of risk adjustment, the AIS is most commonly used as a means of identifying the presence or absence of severe injury in a particular body region, we examined the ability of the algorithm to identify severe injury (AIS score ≥ 3) in each body region (Table 3). The algorithm demonstrated highest concordance with mapped AIS scores in identifying patients with severe head injuries (κ 0.78, 95% CI 0.76–0.80). Conversely, the algorithm performed least well in identifying severe injuries to the face (κ 0.14, 95% CI 0.10–0.17).

Table 3.

Agreement in identifying patients with AIS score ≥ 3, by body region, derived from mapped and abstracted scores

Body region	AIS score ≥ 3		Exact agreement,%	κ
	Mapped scores, %	Abstracted scores, %
All	50	62	83	0.65
Head	80	85	94	0.78
Face	2	20	82	0.14
Neck	15	21	85	0.51
Chest	88	84	92	0.67
Abdomen	40	53	79	0.59
Spine	30	57	66	0.35
Upper extremity	14	30	79	0.40
Lower extremity	37	62	68	0.39

AIS = Abbreviated Injury Scale.¹

Concordance of algorithm and abstracted ISS

Overall, mapped and abstracted ISS were similar (Table 4). The difference between ISS obtained using the algorithm and abstracted scores was 10 or less in 87% of patients. When patients were stratified by age, mechanism of injury and discharge status, ISS obtained from the algorithm and from chart abstraction were similar within each stratum. The greatest variation in the differences between mapped and abstracted ISS was observed among patients with penetrating injuries, where the difference between the 2 scores was 10 or less among 82% of patients.

Table 4.

Comparison of ISS in the Ontario Trauma Registry, Comprehensive Data Set (algorithm v. abstracted)

Category	No. (%)	ICC (95% CI)	Difference between ISS (mapped) and ISS (abstracted) ≤ 10, %
Overall	10 431	0.83 (0.81–0.84)	87
Age, yr
< 15	896 (8.6)	0.79 (0.75–0.83)	88
≥ 15	9 535 (91.4)	0.83 (0.81–0.84)	87
Mechanism
Penetrating	500 (4.8)	0.64 (0.57–0.71)	82
Blunt	9 931 (95.2)	0.83 (0.82–0.84)	88
Discharge status
Alive	9 302 (89.2)	0.82 (0.81–0.83)	87
Dead	1 129 (10.8)	0.79 (0.74–0.83)	87

CI = confidence interval; ICC = intraclass correlation coefficient; ISS = Injury Severity Score.²

The concordance of ISS across methods was also evaluated using the ICC. Across the entire patient cohort, the ICC demonstrated excellent agreement between ISS obtained by mapping of ICD-10 diagnoses and by chart abstraction (ICC 0.83, 95% CI 0.81–0.84). Although ISS concordance across methods was excellent among patients with blunt injuries, patients aged 15 years and older and patients discharged alive (Table 4), ICC ranged from 0.64 to 0.79 among pediatric patients, patients with penetrating injuries and patients who died in hospital. All ICCs were consistent with substantial to excellent agreement, indicating that the algorithm was internally valid and a reliable means of obtaining ISS from ICD-10 diagnostic codes.

Discussion

Introduced in 1971 and 1974, respectively, the AIS and ISS systems have proven to be accurate and replicable means of estimating the degree of anatomic injury among trauma patients.³ Although a number of alternative scoring systems aimed at improving on the predictive performance of ISS have been proposed, including the ICD-based Injury Severity Score (ICISS), the Anatomic Profile (AP) and the Trauma and Injury Severity Score (TRISS), the AIS and ISS systems remain the most widely used means of identifying injury severity and of risk-adjusting trauma-related outcomes for variations in injury severity. Despite the availability of these injury scores in trauma registries and specialized trauma databases, AIS scores and ISS are not typically available from administrative and other non-specialized data sets.

We developed a novel ICD-10–to–AIS conversion algorithm, which will allow ISS to be derived from administrative data sets employing ICD-10. By examining the concordance of AIS scores and ISS derived from ICD-10 mapping to scores provided by expert abstractors in the OTR-CDS, we have demonstrated that mapped scores are accurate compared with those provided by chart abstraction.

The 1998 version of the AIS was used in this study. Although the 2005 version of the AIS is in the process of being adopted in many jurisdictions, we felt that an algorithm using the 1998 version of the AIS remained relevant. First, it will take a number of years of data collection before sufficient data exist for the AIS 2005 to be used in a meaningful way. In the interim, there is a large volume of legacy data employing ICD-10 that has never been analyzed. In addition, particularly in Canada, the AIS 2005 is not being implemented in all institutions at the same rate.¹⁶ As such, the 1998 version of the AIS remains a useful reference point.

Alternative methods of estimating injury severity from administrative data have been proposed, with the ICISS system being the most prominent. The ICISS system relies on estimates of the survival risk ratios associated with each ICD diagnosis.¹⁷ The ICISS scores are derived empirically from each data set and have been shown in some studies to be more accurate than algorithm-derived ISS.^18–20 The accuracy of ICISS, however, may vary across data sets and may be inaccurate for rare diagnoses when derived from small data sets.²¹ Because our ICD-10–to–AIS algorithm is based on consensus definitions rather than empirically derived data, algorithm performance should be minimally dependent on data sources. Moreover, previous work has shown that, when incorporated into a risk adjustment model that includes other predictors of injury-related death, algorithm-derived ISS and ICISS produced risk-adjustment models with similar performance.²²

More recently, an injury severity model (the trauma mortality prediction model based on ICD-9; TMPM-ICD9) based on regression modelling of mortality and ICD has been developed using data from the National Trauma Data Bank and ICD-9 diagnostic codes.²³ This model has been shown to be more accurate than ICISS, and the methodology could theoretically be replicated with ICD-10 diagnostic codes. However, once again, because this method is dependent on the composition and size of the data source, model validity would vary across data sets, with smaller data sources likely to produce less accurate models.

Limitations

The ICD-10–to–AIS algorithm described in this study has a number of limitations. The ability of the algorithm to identify severe injury varied by body region. Whereas the algorithm reliably identified severe injuries in the head, chest and abdomen, agreement between the algorithm and abstracted scores was low in other body regions. These differences are likely related to variations in the degree of granularity of ICD-10 diagnostic codes across body regions. However, body regions where the algorithm had lower performance were also body regions where injuries were less common in the population. As a result, the impact of algorithm error on risk adjustment of mortality in large data sets or population analyses is likely to be small.

Concordance between ISS calculated by expert abstractors and those calculated using the algorithm was also not perfect. This may be a reflection of the quality of ICD-10 diagnostic coding in the database used. Nevertheless, the concordance between scores obtained by mapping and abstractions was very similar or better than concordance observed between human abstractors. In this study, we demonstrated an overall ICC of 0.83 for ISS obtained using the algorithm and from abstracted data. A previous study demonstrated an ICC of 0.83 for physician abstractors and an ICC of 0.66 for ISS scores obtained by medical record technicians and research assistants.¹⁵

Data from trauma registries that provide ISS are only reliable if AIS scores and ISS are assigned by trained abstractors who accurately capture all patient injuries. Similarly, the accuracy of ISS derived from the algorithm will be dependent on the quality of ICD-10 coding data available in the database to which it is being applied. Researchers using the algorithm must ensure that ICD-10 diagnoses are reliably coded in the database that will be used; the algorithm cannot accurately assign ISS if the ICD-10 codes are not entered accurately.

Conclusion

We have developed an algorithm that reliably and accurately maps ICD-10 diagnoses to ISS, and in this study we have demonstrated the validity of the instrument using multiple approaches. Whereas the current version of the algorithm may be useful primarily for evaluating legacy data, we are currently developing an algorithm that employs the 2005 version of the AIS. This algorithm will facilitate the use of administrative data to study injury outcomes and will ensure that trauma-related health services research captures the outcomes of patients that currently are not adequately characterized in specialized trauma registries.

Investigators interested in using this algorithm should contact the corresponding author.